2021-04-09T18:04:22.017 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-09T18:04:22.018 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/failover} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-04-09_17:56:50-fs-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6031712 --name yuriw-2021-04-09_17:56:50-fs-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi -- /home/teuthworker/archive/yuriw-2021-04-09_17:56:50-fs-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6031712/orig.config.yaml 2021-04-09T18:04:22.063 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-04-09T18:04:22.136 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-04-09_17:56:50-fs-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6031712 branch: wip-yuri2-testing-2021-04-09-0846-nautilus description: fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/failover} email: ceph-qa@ceph.io first_in_suite: false job_id: '6031712' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2021-04-09_17:56:50-fs-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '7.9' overrides: admin_socket: branch: wip-yuri2-testing-2021-04-09-0846-nautilus ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(TOO_FEW_PGS\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - not responding, replacing - \(MDS_INSUFFICIENT_STANDBY\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: disabled: true install: ceph: sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2021-04-09-0846-nautilus sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b owner: scheduled_yuriw@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b sleep_before_teardown: 0 suite: fs suite_branch: wip-yuri2-testing-2021-04-09-0846-nautilus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d23cd2d5f486487b0b80655c4e33344f0f97b82b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b targets: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDujfJq+j0JX9Rzb2kuSh2IZbYDMq9JMv3AOKD8r1QVfJqzOO6LpTyrHoh8smN/f5VT9aogti+FIJ4lfoJtfTGOW/ZC1ynPjqlHCPXQAQU3HMzdLXgEd/I/36vFU6fVzGxCh0UFq1lFZMnZFAu/jIPQkGV4HBU9Ek0tmcXm3dZY+GoJxaDCSJkabJC1WOg0TQ1kFLJCLRMych4FAg6CTsWk/ywsUs42UzmZ7EFe53r+Xi9f/Q41sRJqPoTrAUH+ANrnPBmkKN0wSuEUn8cAYJhc4A1Y/cTrDnq2+y9dTaurM3bPCfjfAkpMAClav/N+Ufv1/vhLOfKyU+Yw2AzZx63X smithi086.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCogA8ikymy+H5SbjPokhU/7Hk4dZ64hMy9e+2+A0iS9JeFAqfiavATu959P0kxZqnkvaNIDJixrpIZbP2Ekr7J5Dmwn6FoKAfP7WkW+AgU76b0AnS7w2wPMwBr05gytG+2/5l684BL6ebwOdqv2eoQ0q9ElB1XByvUNnOzZIEE3enw1voggnafNFgaYge8ShAmDiqOc9AON6i/Our357ueXcQr7hEsidHoxyn81S0Jczp5qSqpZX8JW6y0IhD1kq1trQUFUQ41zO/TbD92AEZzTSyIgVCLkWW8RiWfenTf18y67MLVsq1jArScgU2E1c/azpstkFv1I4ji/22dbfQd tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_failover teuthology_branch: master teuthology_sha1: 85d61eae4759f46ce21e9a37cd816a7a1a66c9d5 timestamp: 2021-04-09_17:56:50 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23935 2021-04-09T18:04:22.136 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d23cd2d5f486487b0b80655c4e33344f0f97b82b/qa; will attempt to use it 2021-04-09T18:04:22.137 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d23cd2d5f486487b0b80655c4e33344f0f97b82b/qa/tasks 2021-04-09T18:04:22.137 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-04-09T18:04:22.139 INFO:teuthology.task.internal:Checking packages... 2021-04-09T18:04:22.159 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'd23cd2d5f486487b0b80655c4e33344f0f97b82b' 2021-04-09T18:04:22.160 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-04-09T18:04:22.160 INFO:teuthology.packaging:ref: None 2021-04-09T18:04:22.160 INFO:teuthology.packaging:tag: None 2021-04-09T18:04:22.161 INFO:teuthology.packaging:branch: wip-yuri2-testing-2021-04-09-0846-nautilus 2021-04-09T18:04:22.161 INFO:teuthology.packaging:sha1: d23cd2d5f486487b0b80655c4e33344f0f97b82b 2021-04-09T18:04:22.162 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-yuri2-testing-2021-04-09-0846-nautilus 2021-04-09T18:04:22.349 INFO:teuthology.task.internal:Found packages for ceph version 14.2.19-116.gd23cd2d5 2021-04-09T18:04:22.350 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-04-09T18:04:22.364 INFO:teuthology.task.internal:no buildpackages task found 2021-04-09T18:04:22.364 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-04-09T18:04:22.576 INFO:teuthology.task.internal:Saving configuration 2021-04-09T18:04:22.597 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-04-09T18:04:22.618 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-09T18:04:22.658 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-09 17:57:54.405621', 'description': '/home/teuthworker/archive/yuriw-2021-04-09_17:56:50-fs-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6031712', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOWp1ZngRngwVbI+qTv2seIAviCjx74wZoVT0zgcBTh6NSapPOIpRHFMCkU10Qi7CzGKqHbQpSMa80yASuVyLm0=', 'os_version': '7.9', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi032.front.sepia.ceph.com'} 2021-04-09T18:04:22.681 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-09 17:57:54.405600', 'description': '/home/teuthworker/archive/yuriw-2021-04-09_17:56:50-fs-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6031712', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEB+wuT9UPvP/T7n5HRrswbUTodLqkqugHM2b9bb2YsM9koOQJWle3Z61M04Cs+0RoW4i7g1727Hb2iC5OeC/cU=', 'os_version': '7.9', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi086.front.sepia.ceph.com'} 2021-04-09T18:04:22.681 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-04-09T18:04:22.700 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-04-09T18:04:22.700 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2021-04-09T18:04:22.700 INFO:teuthology.run_tasks:Running task console_log... 2021-04-09T18:04:22.817 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f26de744c80>, signals=[15]) 2021-04-09T18:04:22.818 INFO:teuthology.run_tasks:Running task internal.connect... 2021-04-09T18:04:22.829 INFO:teuthology.task.internal:Opening connections... 2021-04-09T18:04:22.829 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2021-04-09T18:04:22.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-09T18:04:22.909 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2021-04-09T18:04:22.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-09T18:04:22.982 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-04-09T18:04:22.994 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2021-04-09T18:04:23.124 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2021-04-09T18:04:23.124 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2021-04-09T18:04:23.166 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Red Hat Enterprise Linux Server" 2021-04-09T18:04:23.166 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="7.9 (Maipo)" 2021-04-09T18:04:23.167 INFO:teuthology.orchestra.run.smithi032.stdout:ID="rhel" 2021-04-09T18:04:23.167 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="fedora" 2021-04-09T18:04:23.167 INFO:teuthology.orchestra.run.smithi032.stdout:VARIANT="Server" 2021-04-09T18:04:23.167 INFO:teuthology.orchestra.run.smithi032.stdout:VARIANT_ID="server" 2021-04-09T18:04:23.168 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="7.9" 2021-04-09T18:04:23.168 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Employee SKU" 2021-04-09T18:04:23.168 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2021-04-09T18:04:23.168 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-04-09T18:04:23.169 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.redhat.com/" 2021-04-09T18:04:23.169 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-09T18:04:23.169 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:04:23.169 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-04-09T18:04:23.170 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-04-09T18:04:23.170 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-09T18:04:23.170 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-04-09T18:04:23.171 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-04-09T18:04:23.186 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2021-04-09T18:04:23.247 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2021-04-09T18:04:23.247 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2021-04-09T18:04:23.290 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Red Hat Enterprise Linux Server" 2021-04-09T18:04:23.291 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="7.9 (Maipo)" 2021-04-09T18:04:23.291 INFO:teuthology.orchestra.run.smithi086.stdout:ID="rhel" 2021-04-09T18:04:23.291 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="fedora" 2021-04-09T18:04:23.291 INFO:teuthology.orchestra.run.smithi086.stdout:VARIANT="Server" 2021-04-09T18:04:23.291 INFO:teuthology.orchestra.run.smithi086.stdout:VARIANT_ID="server" 2021-04-09T18:04:23.292 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="7.9" 2021-04-09T18:04:23.292 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Employee SKU" 2021-04-09T18:04:23.292 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2021-04-09T18:04:23.292 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-04-09T18:04:23.293 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.redhat.com/" 2021-04-09T18:04:23.293 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-09T18:04:23.293 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:04:23.293 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-04-09T18:04:23.293 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-04-09T18:04:23.294 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-09T18:04:23.294 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-04-09T18:04:23.294 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2021-04-09T18:04:23.311 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-04-09T18:04:23.322 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-04-09T18:04:23.368 INFO:teuthology.task.internal:Checking for old test directory... 2021-04-09T18:04:23.368 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2021-04-09T18:04:23.372 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2021-04-09T18:04:23.411 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-04-09T18:04:23.422 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-04-09T18:04:23.423 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2021-04-09T18:04:23.453 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2021-04-09T18:04:23.515 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-04-09T18:04:23.621 INFO:teuthology.run_tasks:Running task kernel... 2021-04-09T18:04:23.647 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'mds.c': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'mds.d': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}} 2021-04-09T18:04:23.649 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-04-09T18:04:23.649 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2021-04-09T18:04:23.693 INFO:teuthology.orchestra.run.smithi032.stdout:3.10.0-1160.24.1.el7.x86_64 2021-04-09T18:04:23.694 INFO:teuthology.task.kernel:Running kernel on smithi032: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T18:04:23.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2021-04-09T18:04:24.807 INFO:teuthology.orchestra.run.smithi032.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T18:04:24.808 INFO:teuthology.orchestra.run.smithi032.stdout: : disabled-repos, subscription-manager 2021-04-09T18:04:24.975 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:04:24.975 INFO:teuthology.orchestra.run.smithi032.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T18:04:24.975 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:04:25.003 INFO:teuthology.orchestra.run.smithi032.stdout:Loading mirror speeds from cached hostfile 2021-04-09T18:04:25.017 INFO:teuthology.orchestra.run.smithi032.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T18:04:26.299 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do 2021-04-09T18:04:26.319 DEBUG:teuthology.orchestra.run.smithi032:> echo no | sudo yum reinstall kernel || true 2021-04-09T18:04:26.532 INFO:teuthology.orchestra.run.smithi032.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T18:04:26.532 INFO:teuthology.orchestra.run.smithi032.stdout: : disabled-repos, subscription-manager 2021-04-09T18:04:26.596 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:04:26.596 INFO:teuthology.orchestra.run.smithi032.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T18:04:26.596 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:04:26.680 INFO:teuthology.orchestra.run.smithi032.stderr:Skipping the running kernel: kernel-3.10.0-1160.24.1.el7.x86_64 2021-04-09T18:04:26.703 INFO:teuthology.orchestra.run.smithi032.stdout:Loading mirror speeds from cached hostfile 2021-04-09T18:04:26.703 INFO:teuthology.orchestra.run.smithi032.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T18:04:26.708 INFO:teuthology.orchestra.run.smithi032.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-04-09T18:04:26.709 INFO:teuthology.orchestra.run.smithi032.stderr:Error: Nothing to do 2021-04-09T18:04:26.725 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel --last | head -n 1 2021-04-09T18:04:26.942 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-3.10.0-1160.24.1.el7.x86_64 Thu 08 Apr 2021 06:20:34 PM UTC 2021-04-09T18:04:26.943 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T18:04:26.943 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-04-09T18:04:26.944 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2021-04-09T18:04:26.991 INFO:teuthology.orchestra.run.smithi086.stdout:3.10.0-1160.24.1.el7.x86_64 2021-04-09T18:04:26.991 INFO:teuthology.task.kernel:Running kernel on smithi086: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T18:04:26.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2021-04-09T18:04:28.008 INFO:teuthology.orchestra.run.smithi086.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T18:04:28.008 INFO:teuthology.orchestra.run.smithi086.stdout: : disabled-repos, subscription-manager 2021-04-09T18:04:28.182 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:04:28.183 INFO:teuthology.orchestra.run.smithi086.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T18:04:28.183 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:04:28.213 INFO:teuthology.orchestra.run.smithi086.stdout:Loading mirror speeds from cached hostfile 2021-04-09T18:04:28.226 INFO:teuthology.orchestra.run.smithi086.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T18:04:29.362 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do 2021-04-09T18:04:29.380 DEBUG:teuthology.orchestra.run.smithi086:> echo no | sudo yum reinstall kernel || true 2021-04-09T18:04:29.579 INFO:teuthology.orchestra.run.smithi086.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T18:04:29.579 INFO:teuthology.orchestra.run.smithi086.stdout: : disabled-repos, subscription-manager 2021-04-09T18:04:29.655 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:04:29.656 INFO:teuthology.orchestra.run.smithi086.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-04-09T18:04:29.656 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:04:29.731 INFO:teuthology.orchestra.run.smithi086.stderr:Skipping the running kernel: kernel-3.10.0-1160.24.1.el7.x86_64 2021-04-09T18:04:29.733 INFO:teuthology.orchestra.run.smithi086.stdout:Loading mirror speeds from cached hostfile 2021-04-09T18:04:29.733 INFO:teuthology.orchestra.run.smithi086.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T18:04:29.738 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-04-09T18:04:29.738 INFO:teuthology.orchestra.run.smithi086.stderr:Error: Nothing to do 2021-04-09T18:04:29.754 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel --last | head -n 1 2021-04-09T18:04:29.939 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-3.10.0-1160.24.1.el7.x86_64 Thu 08 Apr 2021 06:20:34 PM UTC 2021-04-09T18:04:29.940 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.24.1.el7.x86_64 2021-04-09T18:04:29.940 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-04-09T18:04:29.941 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-04-09T18:04:29.941 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-04-09T18:04:30.010 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2021-04-09T18:04:30.022 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-04-09T18:04:30.023 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-04-09T18:04:30.094 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2021-04-09T18:04:30.105 INFO:teuthology.run_tasks:Running task internal.base... 2021-04-09T18:04:30.114 INFO:teuthology.task.internal:Creating test directory... 2021-04-09T18:04:30.114 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-04-09T18:04:30.118 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-04-09T18:04:30.161 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-04-09T18:04:30.173 INFO:teuthology.run_tasks:Running task internal.archive... 2021-04-09T18:04:30.208 INFO:teuthology.task.internal:Creating archive directory... 2021-04-09T18:04:30.209 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-04-09T18:04:30.213 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-04-09T18:04:30.261 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-04-09T18:04:30.272 INFO:teuthology.task.internal:Enabling coredump saving... 2021-04-09T18:04:30.272 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-04-09T18:04:30.276 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-04-09T18:04:30.332 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-04-09T18:04:30.348 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-04-09T18:04:30.349 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-04-09T18:04:30.361 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-04-09T18:04:30.362 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-04-09T18:04:30.372 INFO:teuthology.task.internal:Configuring sudo... 2021-04-09T18:04:30.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-04-09T18:04:30.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-04-09T18:04:30.431 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-04-09T18:04:30.442 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-04-09T18:04:30.442 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-04-09T18:04:30.446 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-04-09T18:04:30.486 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-04-09T18:04:30.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-04-09T18:04:30.673 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-04-09T18:04:30.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-04-09T18:04:30.812 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2021-04-09T18:04:30.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-04-09T18:04:30.870 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-04-09T18:04:30.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-04-09T18:04:31.006 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-04-09T18:04:31.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-04-09T18:04:31.139 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2021-04-09T18:04:31.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-04-09T18:04:31.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2021-04-09T18:04:31.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2021-04-09T18:04:31.290 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-04-09T18:04:31.299 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-04-09T18:04:31.339 INFO:teuthology.run_tasks:Running task internal.timer... 2021-04-09T18:04:31.349 INFO:teuthology.task.internal:Starting timer... 2021-04-09T18:04:31.350 INFO:teuthology.run_tasks:Running task pcp... 2021-04-09T18:04:31.401 INFO:teuthology.run_tasks:Running task selinux... 2021-04-09T18:04:31.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2021-04-09T18:04:31.530 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: [ OK ] 2021-04-09T18:04:31.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2021-04-09T18:04:31.619 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: [ OK ] 2021-04-09T18:04:31.620 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-04-09T18:04:31.621 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2021-04-09T18:04:31.667 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2021-04-09T18:04:31.667 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2021-04-09T18:04:31.718 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2021-04-09T18:04:31.719 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2021-04-09T18:04:31.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\)' 2021-04-09T18:04:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:04:31.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\)' 2021-04-09T18:04:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:04:31.832 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-04-09T18:04:31.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2021-04-09T18:04:31.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2021-04-09T18:04:31.942 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-04-09T18:04:31.954 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-04-09T18:04:31.955 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-04-09T18:04:31.969 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-04-09T18:04:31.971 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-04-09T18:04:32.302 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-04-09T18:04:33.106 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-04-09T18:04:35.238 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-04-09T18:04:35.261 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:35 +0000 (0:00:00.039) 0:00:00.039 ********** 2021-04-09T18:04:35.272 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:35 +0000 (0:00:00.010) 0:00:00.049 ********** 2021-04-09T18:04:36.785 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi032.front.sepia.ceph.com] 2021-04-09T18:04:36.814 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] 2021-04-09T18:04:36.839 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:36 +0000 (0:00:01.567) 0:00:01.616 ********** 2021-04-09T18:04:36.875 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:36 +0000 (0:00:00.036) 0:00:01.653 ********** 2021-04-09T18:04:37.139 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-04-09T18:04:37.141 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-04-09T18:04:37.161 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:37 +0000 (0:00:00.285) 0:00:01.938 ********** 2021-04-09T18:04:37.166 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-04-09T18:04:37.188 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:37 +0000 (0:00:00.026) 0:00:01.965 ********** 2021-04-09T18:04:37.710 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-04-09T18:04:37.712 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-04-09T18:04:37.732 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:37 +0000 (0:00:00.544) 0:00:02.510 ********** 2021-04-09T18:04:37.749 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-04-09T18:04:37.771 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:37 +0000 (0:00:00.038) 0:00:02.548 ********** 2021-04-09T18:04:37.797 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-04-09T18:04:37.799 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:37.819 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:37 +0000 (0:00:00.047) 0:00:02.596 ********** 2021-04-09T18:04:37.834 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:37.856 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:37 +0000 (0:00:00.036) 0:00:02.633 ********** 2021-04-09T18:04:38.141 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-04-09T18:04:38.142 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-09T18:04:38.164 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:38 +0000 (0:00:00.308) 0:00:02.941 ********** 2021-04-09T18:04:38.170 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-09T18:04:38.193 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:38 +0000 (0:00:00.028) 0:00:02.970 ********** 2021-04-09T18:04:39.068 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-04-09T18:04:39.069 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} 2021-04-09T18:04:39.091 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:39 +0000 (0:00:00.898) 0:00:03.869 ********** 2021-04-09T18:04:39.109 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:39 +0000 (0:00:00.017) 0:00:03.886 ********** 2021-04-09T18:04:39.570 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-04-09T18:04:39.572 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-04-09T18:04:39.573 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-04-09T18:04:39.593 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-04-09T18:04:39.594 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-04-09T18:04:39.667 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-04-09T18:04:39.670 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-04-09T18:04:39.830 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:39 +0000 (0:00:00.721) 0:00:04.607 ********** 2021-04-09T18:04:39.847 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:39 +0000 (0:00:00.017) 0:00:04.625 ********** 2021-04-09T18:04:40.413 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi032.front.sepia.ceph.com] 2021-04-09T18:04:40.443 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] 2021-04-09T18:04:40.471 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-04-09T18:04:40.472 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:40 +0000 (0:00:00.623) 0:00:05.248 ********** 2021-04-09T18:04:40.512 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:40 +0000 (0:00:00.041) 0:00:05.290 ********** 2021-04-09T18:04:41.076 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-04-09T18:04:41.078 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.156060", "end": "2021-04-09 18:04:41.051219", "rc": 0, "start": "2021-04-09 18:04:40.895159", "stderr": "", "stderr_lines": [], "stdout": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "stdout_lines": ["5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232"]} 2021-04-09T18:04:41.098 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.585) 0:00:05.875 ********** 2021-04-09T18:04:41.104 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.162702", "end": "2021-04-09 18:04:41.071643", "rc": 0, "start": "2021-04-09 18:04:40.908941", "stderr": "", "stderr_lines": [], "stdout": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "stdout_lines": ["5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232"]} 2021-04-09T18:04:41.126 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.028) 0:00:05.904 ********** 2021-04-09T18:04:41.308 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-04-09T18:04:41.310 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.022328", "end": "2021-04-09 18:04:41.286331", "failed_when_result": false, "rc": 0, "start": "2021-04-09 18:04:41.264003", "stderr": "", "stderr_lines": [], "stdout": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "stdout_lines": ["5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232"]} 2021-04-09T18:04:41.332 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.205) 0:00:06.110 ********** 2021-04-09T18:04:41.339 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011561", "end": "2021-04-09 18:04:41.315217", "failed_when_result": false, "rc": 0, "start": "2021-04-09 18:04:41.303656", "stderr": "", "stderr_lines": [], "stdout": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "stdout_lines": ["5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232"]} 2021-04-09T18:04:41.372 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.039) 0:00:06.149 ********** 2021-04-09T18:04:41.410 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-04-09T18:04:41.412 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2021-04-09T18:04:41.434 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.061) 0:00:06.211 ********** 2021-04-09T18:04:41.440 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2021-04-09T18:04:41.463 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.029) 0:00:06.241 ********** 2021-04-09T18:04:41.500 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-04-09T18:04:41.500 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.522 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.059) 0:00:06.300 ********** 2021-04-09T18:04:41.549 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.571 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.048) 0:00:06.348 ********** 2021-04-09T18:04:41.586 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-04-09T18:04:41.588 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.609 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.037) 0:00:06.386 ********** 2021-04-09T18:04:41.625 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.647 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.038) 0:00:06.424 ********** 2021-04-09T18:04:41.673 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-04-09T18:04:41.675 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.696 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.048) 0:00:06.473 ********** 2021-04-09T18:04:41.712 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.735 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.039) 0:00:06.512 ********** 2021-04-09T18:04:41.762 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-04-09T18:04:41.764 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.787 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.051) 0:00:06.564 ********** 2021-04-09T18:04:41.803 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.826 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.039) 0:00:06.604 ********** 2021-04-09T18:04:41.853 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-04-09T18:04:41.855 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.876 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.049) 0:00:06.654 ********** 2021-04-09T18:04:41.903 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.925 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:41 +0000 (0:00:00.048) 0:00:06.702 ********** 2021-04-09T18:04:41.963 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-04-09T18:04:41.965 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.976 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.978 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd 2021-04-09T18:04:41.980 INFO:teuthology.task.ansible.out:0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.989 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.990 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:41.992 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.003 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.004 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.006 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.016 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.017 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.019 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.021 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.031 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.032 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.034 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e1 2021-04-09T18:04:42.036 INFO:teuthology.task.ansible.out:2dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.046 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.047 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.049 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.050 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.050 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.051 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jc 2021-04-09T18:04:42.053 INFO:teuthology.task.ansible.out:aratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31 2021-04-09T18:04:42.055 INFO:teuthology.task.ansible.out:edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.064 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.066 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.066 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.067 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilk 2021-04-09T18:04:42.069 INFO:teuthology.task.ansible.out:in@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c 2021-04-09T18:04:42.071 INFO:teuthology.task.ansible.out:01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.081 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.082 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.084 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.086 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.097 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.098 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.100 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.101 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467c 2021-04-09T18:04:42.103 INFO:teuthology.task.ansible.out:cdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.112 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.114 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.115 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.117 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.119 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.130 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.132 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.134 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.144 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.146 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.147 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.148 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 4715 2021-04-09T18:04:42.150 INFO:teuthology.task.ansible.out:38eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.150 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 2021-04-09T18:04:42.152 INFO:teuthology.task.ansible.out: 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.163 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.165 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.166 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.167 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.178 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.179 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.181 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.182 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.184 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.194 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.196 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.197 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang 2021-04-09T18:04:42.198 INFO:teuthology.task.ansible.out:", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.207 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.209 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.220 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.221 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.223 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.223 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.234 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.235 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.236 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.236 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK 2021-04-09T18:04:42.238 INFO:teuthology.task.ansible.out:28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.248 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.249 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.250 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.260 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.262 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.262 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.263 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item 2021-04-09T18:04:42.264 INFO:teuthology.task.ansible.out:": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.274 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.275 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.276 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.277 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.287 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.289 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.290 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.290 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn 2021-04-09T18:04:42.292 INFO:teuthology.task.ansible.out:": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.301 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.303 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.314 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.316 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.317 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.317 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": " 2021-04-09T18:04:42.319 INFO:teuthology.task.ansible.out:swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.329 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.331 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.332 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.333 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.333 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "ch 2021-04-09T18:04:42.335 INFO:teuthology.task.ansible.out:anged": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.345 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.346 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.347 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": fa 2021-04-09T18:04:42.349 INFO:teuthology.task.ansible.out:lse, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.358 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.360 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.360 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@hom 2021-04-09T18:04:42.362 INFO:teuthology.task.ansible.out:e BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.371 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.373 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.374 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.374 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa 2021-04-09T18:04:42.376 INFO:teuthology.task.ansible.out:@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.385 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.387 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.388 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.388 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.389 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856 2021-04-09T18:04:42.391 INFO:teuthology.task.ansible.out:c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.400 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.402 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.403 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.403 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name 2021-04-09T18:04:42.405 INFO:teuthology.task.ansible.out:': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.415 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.417 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.418 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.420 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.431 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.433 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.443 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.445 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.446 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.457 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.459 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.459 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.460 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb" 2021-04-09T18:04:42.462 INFO:teuthology.task.ansible.out:, "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.471 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.473 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.474 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.476 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.487 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.488 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.489 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.489 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.490 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", 2021-04-09T18:04:42.491 INFO:teuthology.task.ansible.out:"ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.500 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.502 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.513 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.515 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.515 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AK 2021-04-09T18:04:42.517 INFO:teuthology.task.ansible.out:NOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.526 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.528 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.529 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.co 2021-04-09T18:04:42.531 INFO:teuthology.task.ansible.out:m] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.540 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.542 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.542 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_l 2021-04-09T18:04:42.543 INFO:teuthology.task.ansible.out:oop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.553 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.554 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.555 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.555 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": " 2021-04-09T18:04:42.557 INFO:teuthology.task.ansible.out:vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.566 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.568 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.579 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.581 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.582 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.582 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sep 2021-04-09T18:04:42.584 INFO:teuthology.task.ansible.out:ia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.593 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.595 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.595 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.597 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.606 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.607 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.608 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.608 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.608 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.637 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.638 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.639 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.639 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansib 2021-04-09T18:04:42.641 INFO:teuthology.task.ansible.out:le_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.662 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:42 +0000 (0:00:00.736) 0:00:07.439 ********** 2021-04-09T18:04:42.668 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.670 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.671 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.719 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:42 +0000 (0:00:00.056) 0:00:07.496 ********** 2021-04-09T18:04:42.724 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-04-09T18:04:42.748 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:42 +0000 (0:00:00.028) 0:00:07.525 ********** 2021-04-09T18:04:42.796 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:42 +0000 (0:00:00.048) 0:00:07.574 ********** 2021-04-09T18:04:42.812 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-04-09T18:04:42.814 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.839 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:42 +0000 (0:00:00.043) 0:00:07.617 ********** 2021-04-09T18:04:42.857 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.886 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:42 +0000 (0:00:00.046) 0:00:07.663 ********** 2021-04-09T18:04:42.902 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-04-09T18:04:42.904 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.925 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:42 +0000 (0:00:00.039) 0:00:07.702 ********** 2021-04-09T18:04:42.951 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:42.977 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:42 +0000 (0:00:00.052) 0:00:07.754 ********** 2021-04-09T18:04:42.994 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-09T18:04:42.996 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.017 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:43 +0000 (0:00:00.039) 0:00:07.794 ********** 2021-04-09T18:04:43.043 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.070 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:43 +0000 (0:00:00.053) 0:00:07.848 ********** 2021-04-09T18:04:43.089 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-09T18:04:43.091 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.114 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:43 +0000 (0:00:00.044) 0:00:07.892 ********** 2021-04-09T18:04:43.141 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.163 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:43 +0000 (0:00:00.049) 0:00:07.941 ********** 2021-04-09T18:04:43.180 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-04-09T18:04:43.182 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.202 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:43 +0000 (0:00:00.038) 0:00:07.980 ********** 2021-04-09T18:04:43.228 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.251 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:43 +0000 (0:00:00.048) 0:00:08.029 ********** 2021-04-09T18:04:43.288 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-09T18:04:43.290 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.300 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.300 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.302 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.312 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.314 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.324 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.325 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.336 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.337 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.338 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.348 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.350 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.351 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.352 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.363 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.364 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.365 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.365 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.375 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.377 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.378 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.388 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.390 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.390 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.401 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.402 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.403 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.403 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.403 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.404 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.415 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.416 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.417 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.417 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.428 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.430 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.430 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.441 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.442 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.443 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.443 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.444 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.454 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.456 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.457 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.467 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.469 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.470 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.470 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.481 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.483 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.487 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.510 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:43 +0000 (0:00:00.259) 0:00:08.288 ********** 2021-04-09T18:04:43.516 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.516 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.518 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.520 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2021-04-09T18:04:43.546 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:43 +0000 (0:00:00.035) 0:00:08.324 ********** 2021-04-09T18:04:44.051 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-04-09T18:04:44.052 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "53b31205b3cc9729fea6ce5ceab6366ba2fdabc0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2021-04-09T18:04:44.072 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.526) 0:00:08.850 ********** 2021-04-09T18:04:44.120 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "53b31205b3cc9729fea6ce5ceab6366ba2fdabc0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2021-04-09T18:04:44.143 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.070) 0:00:08.921 ********** 2021-04-09T18:04:44.149 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-04-09T18:04:44.150 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => { "msg": "Host smithi086.front.sepia.ceph.com is running RedHat 7.9 (Maipo)" } 2021-04-09T18:04:44.176 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.032) 0:00:08.954 ********** 2021-04-09T18:04:44.203 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => { "msg": "Host smithi032.front.sepia.ceph.com is running RedHat 7.9 (Maipo)" } 2021-04-09T18:04:44.225 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.048) 0:00:09.003 ********** 2021-04-09T18:04:44.243 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-04-09T18:04:44.245 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2021-04-09T18:04:44.266 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.040) 0:00:09.044 ********** 2021-04-09T18:04:44.293 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2021-04-09T18:04:44.316 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.049) 0:00:09.094 ********** 2021-04-09T18:04:44.353 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-04-09T18:04:44.355 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:44.376 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.060) 0:00:09.154 ********** 2021-04-09T18:04:44.382 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:44.405 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.028) 0:00:09.183 ********** 2021-04-09T18:04:44.670 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-04-09T18:04:44.672 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.083158", "end": "2021-04-09 18:04:44.652134", "rc": 0, "start": "2021-04-09 18:04:44.568976", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-04-09T18:04:44.692 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.287) 0:00:09.470 ********** 2021-04-09T18:04:44.698 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.048664", "end": "2021-04-09 18:04:44.663143", "rc": 0, "start": "2021-04-09 18:04:44.614479", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-04-09T18:04:44.721 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.028) 0:00:09.499 ********** 2021-04-09T18:04:44.758 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-04-09T18:04:44.759 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:44.780 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.058) 0:00:09.557 ********** 2021-04-09T18:04:44.796 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:44.818 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.038) 0:00:09.596 ********** 2021-04-09T18:04:44.845 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-04-09T18:04:44.847 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:44.867 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.048) 0:00:09.644 ********** 2021-04-09T18:04:44.882 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:44.905 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.037) 0:00:09.682 ********** 2021-04-09T18:04:44.931 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-04-09T18:04:44.933 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:44.954 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:44 +0000 (0:00:00.048) 0:00:09.731 ********** 2021-04-09T18:04:44.990 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:45.012 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.058) 0:00:09.790 ********** 2021-04-09T18:04:45.028 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-04-09T18:04:45.030 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:45.053 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.040) 0:00:09.831 ********** 2021-04-09T18:04:45.079 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:45.101 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.048) 0:00:09.879 ********** 2021-04-09T18:04:45.118 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-04-09T18:04:45.119 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:45.141 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.039) 0:00:09.918 ********** 2021-04-09T18:04:45.168 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:45.191 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.049) 0:00:09.968 ********** 2021-04-09T18:04:45.220 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-04-09T18:04:45.222 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-04-09T18:04:45.243 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.052) 0:00:10.020 ********** 2021-04-09T18:04:45.291 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-04-09T18:04:45.313 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.070) 0:00:10.091 ********** 2021-04-09T18:04:45.329 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-04-09T18:04:45.331 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-09T18:04:45.352 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.039) 0:00:10.130 ********** 2021-04-09T18:04:45.400 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-09T18:04:45.401 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-09T18:04:45.425 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.072) 0:00:10.202 ********** 2021-04-09T18:04:45.430 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-04-09T18:04:45.432 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-04-09T18:04:45.455 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.030) 0:00:10.232 ********** 2021-04-09T18:04:45.502 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-04-09T18:04:45.524 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:45 +0000 (0:00:00.069) 0:00:10.302 ********** 2021-04-09T18:04:47.077 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-04-09T18:04:47.078 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1617991486.14-rEokyj/katello-ca-consumer-latest.noarchWs_AML.rpm is already installed"]} 2021-04-09T18:04:47.100 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:47 +0000 (0:00:01.575) 0:00:11.877 ********** 2021-04-09T18:04:47.126 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1617991486.17-RPyycJ/katello-ca-consumer-latest.noarchuxeEhS.rpm is already installed"]} 2021-04-09T18:04:47.149 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:47 +0000 (0:00:00.049) 0:00:11.926 ********** 2021-04-09T18:04:47.631 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-04-09T18:04:47.632 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "4a68bf238bc0f5c253db5c3e544b53f300b954a5", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/yum/vars/releasever", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 3, "state": "file", "uid": 0} 2021-04-09T18:04:47.653 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:47 +0000 (0:00:00.504) 0:00:12.431 ********** 2021-04-09T18:04:47.721 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "4a68bf238bc0f5c253db5c3e544b53f300b954a5", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/yum/vars/releasever", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 3, "state": "file", "uid": 0} 2021-04-09T18:04:47.744 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:47 +0000 (0:00:00.090) 0:00:12.522 ********** 2021-04-09T18:04:48.345 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-04-09T18:04:48.346 INFO:teuthology.task.ansible.out:fatal: [smithi086.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-09T18:04:48.348 INFO:teuthology.task.ansible.out:...ignoring 2021-04-09T18:04:48.369 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:48 +0000 (0:00:00.625) 0:00:13.147 ********** 2021-04-09T18:04:48.447 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-04-09T18:04:48.449 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-04-09T18:04:48.470 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:48 +0000 (0:00:00.100) 0:00:13.248 ********** 2021-04-09T18:04:48.479 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** fatal: [smithi032.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-04-09T18:04:48.500 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:48 +0000 (0:00:00.029) 0:00:13.277 ********** 2021-04-09T18:04:48.567 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-04-09T18:04:48.569 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-04-09T18:04:48.589 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:48 +0000 (0:00:00.089) 0:00:13.367 ********** 2021-04-09T18:04:54.697 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-04-09T18:04:54.699 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-04-09T18:04:54.719 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:54 +0000 (0:00:06.129) 0:00:19.497 ********** 2021-04-09T18:04:54.787 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-04-09T18:04:54.789 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-04-09T18:04:54.810 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:54 +0000 (0:00:00.090) 0:00:19.587 ********** 2021-04-09T18:04:55.356 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-04-09T18:04:55.358 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.315567", "end": "2021-04-09 18:04:55.333799", "failed_when_result": false, "rc": 0, "start": "2021-04-09 18:04:55.018232", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-04-09T18:04:55.386 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:55 +0000 (0:00:00.576) 0:00:20.164 ********** 2021-04-09T18:04:55.393 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-04-09T18:04:55.395 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-04-09T18:04:55.421 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:55 +0000 (0:00:00.034) 0:00:20.198 ********** 2021-04-09T18:04:55.459 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-04-09T18:04:55.461 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:55.481 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:55 +0000 (0:00:00.060) 0:00:20.258 ********** 2021-04-09T18:04:55.497 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-04-09T18:04:55.499 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-04-09T18:04:55.520 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:55 +0000 (0:00:00.039) 0:00:20.298 ********** 2021-04-09T18:04:55.701 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 09 April 2021 18:04:55 +0000 (0:00:00.057) 0:00:20.355 ********** TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Friday 09 April 2021 18:04:55 +0000 (0:00:00.090) 0:00:20.445 ********** 2021-04-09T18:04:55.735 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-04-09T18:04:55.737 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:55.757 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:55 +0000 (0:00:00.089) 0:00:20.535 ********** 2021-04-09T18:04:56.076 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-04-09T18:04:56.078 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.308298", "end": "2021-04-09 18:04:56.050706", "failed_when_result": false, "rc": 0, "start": "2021-04-09 18:04:55.742408", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-04-09T18:04:56.099 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:56 +0000 (0:00:00.341) 0:00:20.877 ********** 2021-04-09T18:04:56.169 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-04-09T18:04:56.171 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:56.202 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:56 +0000 (0:00:00.102) 0:00:20.979 ********** 2021-04-09T18:04:56.282 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-04-09T18:04:56.284 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:56.304 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:56 +0000 (0:00:00.102) 0:00:21.082 ********** 2021-04-09T18:04:56.363 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-04-09T18:04:56.364 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-09T18:04:56.386 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:56 +0000 (0:00:00.082) 0:00:21.164 ********** 2021-04-09T18:04:56.560 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-04-09T18:04:56.562 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:56.583 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:56 +0000 (0:00:00.196) 0:00:21.361 ********** 2021-04-09T18:04:57.511 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-04-09T18:04:57.513 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:01.514207", "end": "2021-04-09 18:04:57.493683", "rc": 0, "start": "2021-04-09 18:04:55.979476", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2021-04-09T18:04:57.532 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:57 +0000 (0:00:00.949) 0:00:22.310 ********** 2021-04-09T18:04:57.611 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-04-09T18:04:57.611 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2021-04-09T18:04:57.632 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:57 +0000 (0:00:00.099) 0:00:22.410 ********** 2021-04-09T18:04:57.699 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-04-09T18:04:57.701 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:57.724 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:57 +0000 (0:00:00.091) 0:00:22.501 ********** 2021-04-09T18:04:57.822 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-04-09T18:04:57.824 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-04-09T18:04:57.843 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:57 +0000 (0:00:00.119) 0:00:22.621 ********** 2021-04-09T18:04:57.941 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-04-09T18:04:57.943 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-04-09T18:04:57.962 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:57 +0000 (0:00:00.118) 0:00:22.740 ********** 2021-04-09T18:04:58.617 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-04-09T18:04:58.619 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:01.698757", "end": "2021-04-09 18:04:58.595087", "rc": 0, "start": "2021-04-09 18:04:56.896330", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2021-04-09T18:04:58.641 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:58 +0000 (0:00:00.678) 0:00:23.418 ********** 2021-04-09T18:04:58.710 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-04-09T18:04:58.712 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2021-04-09T18:04:58.732 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:58 +0000 (0:00:00.091) 0:00:23.510 ********** 2021-04-09T18:04:58.800 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-04-09T18:04:58.802 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:04:58.822 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:58 +0000 (0:00:00.090) 0:00:23.600 ********** 2021-04-09T18:04:59.005 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-04-09T18:04:59.007 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-04-09T18:04:59.027 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:59 +0000 (0:00:00.204) 0:00:23.805 ********** 2021-04-09T18:04:59.199 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-04-09T18:04:59.220 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:04:59 +0000 (0:00:00.192) 0:00:23.997 ********** 2021-04-09T18:05:00.190 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-04-09T18:05:00.192 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-04-09T18:05:00.212 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:00 +0000 (0:00:00.991) 0:00:24.989 ********** 2021-04-09T18:05:01.857 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-04-09T18:05:01.877 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:01 +0000 (0:00:01.665) 0:00:26.655 ********** 2021-04-09T18:05:03.090 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-04-09T18:05:03.092 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-7-server-rpms", "--enable=rhel-7-server-optional-rpms", "--enable=rhel-7-server-extras-rpms", "--enable=rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:02.562990", "end": "2021-04-09 18:05:03.071825", "rc": 0, "start": "2021-04-09 18:05:00.508835", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-optional-rpms' is enabled for this system.\nRepository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2021-04-09T18:05:03.112 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:03 +0000 (0:00:01.234) 0:00:27.890 ********** 2021-04-09T18:05:03.336 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-04-09T18:05:03.339 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-04-09T18:05:03.358 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:03 +0000 (0:00:00.246) 0:00:28.136 ********** 2021-04-09T18:05:03.581 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-04-09T18:05:03.583 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-09T18:05:03.603 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:03 +0000 (0:00:00.244) 0:00:28.380 ********** 2021-04-09T18:05:04.179 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-04-09T18:05:04.181 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-04-09T18:05:04.648 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-04-09T18:05:04.671 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:04 +0000 (0:00:01.068) 0:00:29.448 ********** 2021-04-09T18:05:04.956 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-04-09T18:05:04.958 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-7-server-rpms", "--enable=rhel-7-server-optional-rpms", "--enable=rhel-7-server-extras-rpms", "--enable=rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:02.841770", "end": "2021-04-09 18:05:04.940182", "rc": 0, "start": "2021-04-09 18:05:02.098412", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-optional-rpms' is enabled for this system.\nRepository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2021-04-09T18:05:04.979 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:04 +0000 (0:00:00.308) 0:00:29.757 ********** 2021-04-09T18:05:05.224 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-04-09T18:05:05.226 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} TASK [common : Remove old apt-mirror repository definition.] ******************* ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-04-09T18:05:05.248 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:05 +0000 (0:00:00.268) 0:00:30.025 ********** 2021-04-09T18:05:05.482 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-04-09T18:05:05.484 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-09T18:05:05.504 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:05 +0000 (0:00:00.256) 0:00:30.281 ********** 2021-04-09T18:05:05.689 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-04-09T18:05:05.690 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2021-04-09T18:05:05.725 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:05 +0000 (0:00:00.221) 0:00:30.503 ********** 2021-04-09T18:05:05.805 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-04-09T18:05:05.807 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:05.827 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:05 +0000 (0:00:00.101) 0:00:30.604 ********** 2021-04-09T18:05:05.894 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-04-09T18:05:05.896 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:05.916 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:05 +0000 (0:00:00.089) 0:00:30.694 ********** 2021-04-09T18:05:05.983 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-04-09T18:05:05.985 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.005 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:06 +0000 (0:00:00.088) 0:00:30.782 ********** 2021-04-09T18:05:06.072 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-04-09T18:05:06.074 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.094 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:06 +0000 (0:00:00.088) 0:00:30.871 ********** 2021-04-09T18:05:06.100 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-04-09T18:05:06.102 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-04-09T18:05:06.165 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-04-09T18:05:06.167 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.176 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.178 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.202 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:06 +0000 (0:00:00.108) 0:00:30.980 ********** 2021-04-09T18:05:06.271 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-04-09T18:05:06.272 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.293 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:06 +0000 (0:00:00.090) 0:00:31.070 ********** 2021-04-09T18:05:06.360 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-04-09T18:05:06.362 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.386 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:06 +0000 (0:00:00.093) 0:00:31.164 ********** 2021-04-09T18:05:06.455 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-04-09T18:05:06.455 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.486 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:06 +0000 (0:00:00.099) 0:00:31.264 ********** 2021-04-09T18:05:06.556 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.578 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:06 +0000 (0:00:00.091) 0:00:31.356 ********** 2021-04-09T18:05:06.605 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-04-09T18:05:06.607 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-04-09T18:05:06.641 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:06 +0000 (0:00:00.062) 0:00:31.418 ********** 2021-04-09T18:05:06.647 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-04-09T18:05:06.649 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:06.671 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:06 +0000 (0:00:00.030) 0:00:31.448 ********** 2021-04-09T18:05:07.218 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-04-09T18:05:07.220 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} 2021-04-09T18:05:07.521 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-04-09T18:05:07.523 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.661607", "end": "2021-04-09 18:05:07.500327", "rc": 0, "start": "2021-04-09 18:05:06.838720", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:05:07.543 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:07 +0000 (0:00:00.872) 0:00:32.321 ********** 2021-04-09T18:05:07.729 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-04-09T18:05:07.731 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2021-04-09T18:05:07.765 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:07 +0000 (0:00:00.221) 0:00:32.542 ********** 2021-04-09T18:05:07.844 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-04-09T18:05:07.846 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:07.865 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:07 +0000 (0:00:00.100) 0:00:32.643 ********** 2021-04-09T18:05:07.933 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-04-09T18:05:07.935 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:07.955 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:07 +0000 (0:00:00.089) 0:00:32.732 ********** 2021-04-09T18:05:08.022 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-04-09T18:05:08.024 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.045 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.090) 0:00:32.823 ********** 2021-04-09T18:05:08.113 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-04-09T18:05:08.114 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.134 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.088) 0:00:32.912 ********** 2021-04-09T18:05:08.203 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-04-09T18:05:08.205 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.214 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.215 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.237 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.102) 0:00:33.015 ********** 2021-04-09T18:05:08.244 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-04-09T18:05:08.245 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-50.el7.x86_64 providing krb5-workstation is already installed"]} 2021-04-09T18:05:08.268 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.031) 0:00:33.046 ********** 2021-04-09T18:05:08.307 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-04-09T18:05:08.309 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.337 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.068) 0:00:33.115 ********** 2021-04-09T18:05:08.343 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-04-09T18:05:08.345 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.369 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.031) 0:00:33.146 ********** 2021-04-09T18:05:08.397 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-04-09T18:05:08.399 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.420 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.051) 0:00:33.198 ********** 2021-04-09T18:05:08.437 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-04-09T18:05:08.439 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.460 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.039) 0:00:33.238 ********** 2021-04-09T18:05:08.487 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-04-09T18:05:08.488 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.510 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.050) 0:00:33.288 ********** 2021-04-09T18:05:08.538 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-04-09T18:05:08.540 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.561 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.050) 0:00:33.338 ********** 2021-04-09T18:05:08.587 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-04-09T18:05:08.589 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.610 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.049) 0:00:33.387 ********** 2021-04-09T18:05:08.627 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-04-09T18:05:08.629 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.650 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.039) 0:00:33.427 ********** 2021-04-09T18:05:08.688 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-04-09T18:05:08.690 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:08.711 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:08 +0000 (0:00:00.061) 0:00:33.489 ********** 2021-04-09T18:05:09.191 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-04-09T18:05:09.193 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-04-09T18:05:09.218 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:09 +0000 (0:00:00.506) 0:00:33.995 ********** 2021-04-09T18:05:09.517 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-04-09T18:05:09.518 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.618981", "end": "2021-04-09 18:05:09.500336", "rc": 0, "start": "2021-04-09 18:05:08.881355", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:05:09.539 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:09 +0000 (0:00:00.321) 0:00:34.316 ********** 2021-04-09T18:05:10.344 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-04-09T18:05:10.346 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-50.el7.x86_64 providing krb5-workstation is already installed"]} 2021-04-09T18:05:10.365 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:10 +0000 (0:00:00.826) 0:00:35.143 ********** 2021-04-09T18:05:10.433 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-04-09T18:05:10.434 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:10.454 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:10 +0000 (0:00:00.089) 0:00:35.232 ********** 2021-04-09T18:05:10.522 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-04-09T18:05:10.524 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:10.543 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:10 +0000 (0:00:00.088) 0:00:35.321 ********** 2021-04-09T18:05:10.611 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-04-09T18:05:10.613 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:10.632 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:10 +0000 (0:00:00.088) 0:00:35.409 ********** 2021-04-09T18:05:10.700 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-04-09T18:05:10.702 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:10.722 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:10 +0000 (0:00:00.089) 0:00:35.499 ********** 2021-04-09T18:05:11.286 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-04-09T18:05:11.288 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-04-09T18:05:11.308 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:11 +0000 (0:00:00.586) 0:00:36.086 ********** 2021-04-09T18:05:44.144 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-04-09T18:05:44.146 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2021-04-09T18:05:44.167 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:44 +0000 (0:00:32.858) 0:01:08.944 ********** 2021-04-09T18:05:44.740 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-04-09T18:05:44.742 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-04-09T18:05:45.216 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-04-09T18:05:45.671 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-04-09T18:05:45.695 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:45 +0000 (0:00:01.528) 0:01:10.472 ********** 2021-04-09T18:05:45.866 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-04-09T18:05:45.867 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-04-09T18:05:45.888 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:45 +0000 (0:00:00.192) 0:01:10.665 ********** 2021-04-09T18:05:46.421 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-04-09T18:05:46.423 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-04-09T18:05:46.877 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-04-09T18:05:47.343 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-04-09T18:05:47.366 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:47 +0000 (0:00:01.478) 0:01:12.143 ********** 2021-04-09T18:05:47.433 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-04-09T18:05:47.434 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:47.454 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:47 +0000 (0:00:00.088) 0:01:12.232 ********** 2021-04-09T18:05:47.521 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-04-09T18:05:47.523 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-04-09T18:05:47.543 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:47 +0000 (0:00:00.088) 0:01:12.320 ********** 2021-04-09T18:05:47.765 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-04-09T18:05:47.766 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-04-09T18:05:47.785 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:47 +0000 (0:00:00.242) 0:01:12.563 ********** 2021-04-09T18:05:47.862 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-04-09T18:05:47.864 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:47.873 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:47.884 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:47.886 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:47.906 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:47 +0000 (0:00:00.120) 0:01:12.683 ********** 2021-04-09T18:05:49.741 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-04-09T18:05:49.743 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2021-04-09T18:05:49.764 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:49 +0000 (0:00:01.857) 0:01:14.541 ********** 2021-04-09T18:05:50.359 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-04-09T18:05:50.361 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-04-09T18:05:50.867 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-04-09T18:05:51.352 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-04-09T18:05:51.375 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:51 +0000 (0:00:01.611) 0:01:16.152 ********** 2021-04-09T18:05:51.557 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-04-09T18:05:51.559 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-04-09T18:05:51.579 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:51 +0000 (0:00:00.203) 0:01:16.356 ********** 2021-04-09T18:05:52.132 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-04-09T18:05:52.134 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-04-09T18:05:52.610 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-04-09T18:05:53.098 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-04-09T18:05:53.121 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:53 +0000 (0:00:01.542) 0:01:17.899 ********** 2021-04-09T18:05:53.178 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-04-09T18:05:53.180 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:53.201 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:53 +0000 (0:00:00.079) 0:01:17.978 ********** 2021-04-09T18:05:53.268 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-04-09T18:05:53.269 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-04-09T18:05:53.290 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:53 +0000 (0:00:00.089) 0:01:18.068 ********** 2021-04-09T18:05:53.512 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-04-09T18:05:53.514 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-04-09T18:05:53.534 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:53 +0000 (0:00:00.243) 0:01:18.311 ********** 2021-04-09T18:05:53.611 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-04-09T18:05:53.613 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:53.622 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:53.624 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:53.633 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:53.655 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:53 +0000 (0:00:00.121) 0:01:18.433 ********** 2021-04-09T18:05:54.239 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-04-09T18:05:54.240 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2021-04-09T18:05:54.261 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:54 +0000 (0:00:00.605) 0:01:19.038 ********** 2021-04-09T18:05:54.328 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-04-09T18:05:54.329 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:54.349 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:54 +0000 (0:00:00.088) 0:01:19.127 ********** 2021-04-09T18:05:54.882 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-04-09T18:05:54.884 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-04-09T18:05:54.904 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:54 +0000 (0:00:00.554) 0:01:19.682 ********** 2021-04-09T18:05:55.459 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-04-09T18:05:55.461 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-04-09T18:05:55.481 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:55 +0000 (0:00:00.576) 0:01:20.258 ********** 2021-04-09T18:05:55.548 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-04-09T18:05:55.550 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:55.559 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-04-09T18:05:55.580 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:55 +0000 (0:00:00.099) 0:01:20.358 ********** 2021-04-09T18:05:55.803 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-04-09T18:05:55.804 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-09T18:05:55.825 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:55 +0000 (0:00:00.244) 0:01:20.602 ********** 2021-04-09T18:05:56.068 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-04-09T18:05:56.069 INFO:teuthology.task.ansible.out:fatal: [smithi086.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-04-09T18:05:56.090 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:56 +0000 (0:00:00.265) 0:01:20.868 ********** 2021-04-09T18:05:56.157 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-04-09T18:05:56.159 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:05:56.180 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:56 +0000 (0:00:00.090) 0:01:20.958 ********** 2021-04-09T18:05:56.723 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-04-09T18:05:56.725 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-04-09T18:05:56.745 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:56 +0000 (0:00:00.564) 0:01:21.523 ********** 2021-04-09T18:05:57.412 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-04-09T18:05:57.413 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-04-09 18:03:45 UTC", "ActiveEnterTimestampMonotonic": "20907470", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target nss-lookup.target network-online.target time-sync.target local-fs.target var-run.mount network.target system.slice systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-04-09 18:03:45 UTC", "AssertTimestampMonotonic": "20903553", "Before": "shutdown.target multi-user.target getty@tty1.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-04-09 18:03:45 UTC", "ConditionTimestampMonotonic": "20903552", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1115", "ExecMainStartTimestamp": "Fri 2021-04-09 18:03:45 UTC", "ExecMainStartTimestampMonotonic": "20907359", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c / 2021-04-09T18:05:57.414 INFO:teuthology.task.ansible.out:etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Fri 2021-04-09 18:03:45 UTC] ; stop_time=[n/a] ; pid=1115 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-04-09 18:03:45 UTC", "InactiveExitTimestampMonotonic": "20907470", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "1115", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice", "Restart": "no 2021-04-09T18:05:57.416 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-04-09 18:03:45 UTC", "WatchdogTimestampMonotonic": "20907411", "WatchdogUSec": "0"}} 2021-04-09T18:05:57.435 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:57 +0000 (0:00:00.689) 0:01:22.212 ********** 2021-04-09T18:05:57.648 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-04-09T18:05:57.649 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001935", "end": "2021-04-09 18:05:57.632619", "rc": 0, "start": "2021-04-09 18:05:57.630684", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-04-09T18:05:57.669 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:57 +0000 (0:00:00.234) 0:01:22.447 ********** 2021-04-09T18:05:58.338 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-04-09T18:05:58.340 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-34.el7.x86_64 providing policycoreutils-python is already installed"]} 2021-04-09T18:05:58.360 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:05:58 +0000 (0:00:00.690) 0:01:23.137 ********** 2021-04-09T18:06:00.316 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-04-09T18:06:00.318 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2021-04-09T18:06:00.339 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:00 +0000 (0:00:01.979) 0:01:25.117 ********** 2021-04-09T18:06:00.408 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-04-09T18:06:00.410 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:00.429 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:00 +0000 (0:00:00.090) 0:01:25.207 ********** 2021-04-09T18:06:00.994 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-04-09T18:06:00.996 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-04-09T18:06:01.016 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:01 +0000 (0:00:00.586) 0:01:25.793 ********** 2021-04-09T18:06:01.580 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-04-09T18:06:01.582 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-04-09T18:06:01.602 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:01 +0000 (0:00:00.586) 0:01:26.380 ********** 2021-04-09T18:06:01.669 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-04-09T18:06:01.671 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-04-09T18:06:01.692 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:01 +0000 (0:00:00.089) 0:01:26.469 ********** 2021-04-09T18:06:01.915 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-04-09T18:06:01.916 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-09T18:06:01.937 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:01 +0000 (0:00:00.245) 0:01:26.714 ********** 2021-04-09T18:06:02.294 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-04-09T18:06:02.295 INFO:teuthology.task.ansible.out:fatal: [smithi032.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-04-09T18:06:02.315 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:02 +0000 (0:00:00.378) 0:01:27.093 ********** 2021-04-09T18:06:02.383 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-04-09T18:06:02.385 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:02.405 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:02 +0000 (0:00:00.089) 0:01:27.182 ********** 2021-04-09T18:06:02.967 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-04-09T18:06:02.969 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-04-09T18:06:02.989 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:02 +0000 (0:00:00.584) 0:01:27.767 ********** 2021-04-09T18:06:03.295 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-04-09T18:06:03.296 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-04-09 18:02:26 UTC", "ActiveEnterTimestampMonotonic": "20317194", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target var-run.mount basic.target system.slice nss-lookup.target network.target network-online.target time-sync.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-04-09 18:02:26 UTC", "AssertTimestampMonotonic": "20276924", "Before": "multi-user.target shutdown.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-04-09 18:02:26 UTC", "ConditionTimestampMonotonic": "20276923", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1068", "ExecMainStartTimestamp": "Fri 2021-04-09 18:02:26 UTC", "ExecMainStartTimestampMonotonic": "20317123", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c / 2021-04-09T18:06:03.298 INFO:teuthology.task.ansible.out:etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Fri 2021-04-09 18:02:26 UTC] ; stop_time=[n/a] ; pid=1068 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-04-09 18:02:26 UTC", "InactiveExitTimestampMonotonic": "20317194", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127535", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127535", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "1068", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice basic.target", "Restart": "no 2021-04-09T18:06:03.299 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-04-09 18:02:26 UTC", "WatchdogTimestampMonotonic": "20317167", "WatchdogUSec": "0"}} 2021-04-09T18:06:03.318 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:03 +0000 (0:00:00.328) 0:01:28.095 ********** 2021-04-09T18:06:03.539 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-04-09T18:06:03.541 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.002289", "end": "2021-04-09 18:06:03.523703", "rc": 0, "start": "2021-04-09 18:06:03.521414", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-04-09T18:06:03.561 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:03 +0000 (0:00:00.243) 0:01:28.339 ********** 2021-04-09T18:06:04.231 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-04-09T18:06:04.232 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2021-04-09T18:06:04.252 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:04 +0000 (0:00:00.691) 0:01:29.030 ********** 2021-04-09T18:06:04.423 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-04-09T18:06:04.425 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-34.el7.x86_64 providing policycoreutils-python is already installed"]} 2021-04-09T18:06:04.446 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:04 +0000 (0:00:00.193) 0:01:29.223 ********** 2021-04-09T18:06:05.443 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-04-09T18:06:05.444 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-04-09T18:06:05.465 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:05 +0000 (0:00:01.019) 0:01:30.243 ********** 2021-04-09T18:06:10.580 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-04-09T18:06:10.582 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2021-04-09T18:06:10.604 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:10 +0000 (0:00:05.138) 0:01:35.381 ********** 2021-04-09T18:06:11.559 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-04-09T18:06:11.561 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-04-09T18:06:11.581 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:11 +0000 (0:00:00.976) 0:01:36.358 ********** 2021-04-09T18:06:13.258 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-04-09T18:06:13.260 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:07.581013", "end": "2021-04-09 18:06:13.239154", "failed_when_result": false, "rc": 0, "start": "2021-04-09 18:06:05.658141", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-04-09T18:06:13.281 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:13 +0000 (0:00:01.700) 0:01:38.059 ********** 2021-04-09T18:06:13.866 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-04-09T18:06:13.868 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-04-09T18:06:13.888 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:13 +0000 (0:00:00.606) 0:01:38.666 ********** 2021-04-09T18:06:14.111 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-04-09T18:06:14.113 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013193", "end": "2021-04-09 18:06:14.091841", "rc": 0, "start": "2021-04-09 18:06:14.078648", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2021-04-09T18:06:14.132 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:14 +0000 (0:00:00.243) 0:01:38.910 ********** 2021-04-09T18:06:14.354 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-04-09T18:06:14.356 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.011988", "end": "2021-04-09 18:06:14.336708", "rc": 0, "start": "2021-04-09 18:06:14.324720", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:06:14.375 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:14 +0000 (0:00:00.243) 0:01:39.153 ********** 2021-04-09T18:06:20.167 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-04-09T18:06:20.169 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:08.360001", "end": "2021-04-09 18:06:20.142327", "failed_when_result": false, "rc": 0, "start": "2021-04-09 18:06:11.782326", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-04-09T18:06:20.190 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:20 +0000 (0:00:05.814) 0:01:44.967 ********** 2021-04-09T18:06:20.777 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-04-09T18:06:20.778 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-04-09T18:06:20.802 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:20 +0000 (0:00:00.611) 0:01:45.579 ********** 2021-04-09T18:06:21.066 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-04-09T18:06:21.068 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.014856", "end": "2021-04-09 18:06:21.051310", "rc": 0, "start": "2021-04-09 18:06:21.036454", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2021-04-09T18:06:21.088 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:21 +0000 (0:00:00.286) 0:01:45.865 ********** 2021-04-09T18:06:21.320 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-04-09T18:06:21.322 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.009887", "end": "2021-04-09 18:06:21.304976", "rc": 0, "start": "2021-04-09 18:06:21.295089", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:06:21.342 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:21 +0000 (0:00:00.254) 0:01:46.120 ********** 2021-04-09T18:06:22.124 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-04-09T18:06:22.126 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:07.527692", "end": "2021-04-09 18:06:22.098886", "rc": 0, "start": "2021-04-09 18:06:14.571194", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:06:22.148 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:22 +0000 (0:00:00.806) 0:01:46.926 ********** 2021-04-09T18:06:22.393 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-04-09T18:06:22.395 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-04-09T18:06:22.418 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:22 +0000 (0:00:00.269) 0:01:47.196 ********** 2021-04-09T18:06:22.486 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-04-09T18:06:22.488 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2021-04-09T18:06:22.508 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:22 +0000 (0:00:00.090) 0:01:47.286 ********** 2021-04-09T18:06:22.576 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-04-09T18:06:22.578 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-04-09T18:06:22.598 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:22 +0000 (0:00:00.089) 0:01:47.376 ********** 2021-04-09T18:06:22.676 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-04-09T18:06:22.678 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "podman", "podman-docker", "python3"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2021-04-09T18:06:22.700 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:22 +0000 (0:00:00.101) 0:01:47.477 ********** 2021-04-09T18:06:22.778 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-04-09T18:06:22.779 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-04-09T18:06:22.800 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:22 +0000 (0:00:00.100) 0:01:47.577 ********** 2021-04-09T18:06:22.981 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-04-09T18:06:22.982 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-04-09T18:06:23.002 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:22 +0000 (0:00:00.202) 0:01:47.780 ********** 2021-04-09T18:06:23.224 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-04-09T18:06:23.225 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-04-09T18:06:23.245 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:23 +0000 (0:00:00.243) 0:01:48.023 ********** 2021-04-09T18:06:23.313 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-04-09T18:06:23.315 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:23.339 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:23 +0000 (0:00:00.093) 0:01:48.116 ********** 2021-04-09T18:06:23.398 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-04-09T18:06:23.400 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:23.428 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:23 +0000 (0:00:00.089) 0:01:48.206 ********** 2021-04-09T18:06:23.674 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-04-09T18:06:23.676 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-04-09T18:06:23.696 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:23 +0000 (0:00:00.267) 0:01:48.473 ********** 2021-04-09T18:06:24.302 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-04-09T18:06:24.325 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:24 +0000 (0:00:00.629) 0:01:49.103 ********** 2021-04-09T18:06:24.393 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-04-09T18:06:24.394 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi086"}, "changed": false} 2021-04-09T18:06:24.414 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:24 +0000 (0:00:00.088) 0:01:49.192 ********** 2021-04-09T18:06:25.126 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-04-09T18:06:25.128 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi086.front.sepia.ceph.com", "ansible_hostname": "smithi086", "ansible_nodename": "smithi086"}, "changed": false, "name": "smithi086"} 2021-04-09T18:06:25.147 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:25 +0000 (0:00:00.732) 0:01:49.924 ********** 2021-04-09T18:06:25.673 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-04-09T18:06:25.674 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-04-09T18:06:25.695 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:25 +0000 (0:00:00.547) 0:01:50.472 ********** 2021-04-09T18:06:26.241 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-04-09T18:06:26.243 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-04-09T18:06:26.263 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:26 +0000 (0:00:00.567) 0:01:51.040 ********** 2021-04-09T18:06:26.887 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-04-09T18:06:26.888 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-04-09T18:06:26.909 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:26 +0000 (0:00:00.646) 0:01:51.687 ********** 2021-04-09T18:06:27.446 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-04-09T18:06:27.448 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-04-09T18:06:27.467 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:27 +0000 (0:00:00.558) 0:01:52.245 ********** 2021-04-09T18:06:27.691 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-04-09T18:06:27.692 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-04-09T18:06:27.712 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:27 +0000 (0:00:00.245) 0:01:52.490 ********** 2021-04-09T18:06:27.924 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-04-09T18:06:27.926 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2021-04-09T18:06:27.945 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:27 +0000 (0:00:00.232) 0:01:52.723 ********** 2021-04-09T18:06:28.191 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-04-09T18:06:28.193 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-04-09T18:06:28.221 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:28 +0000 (0:00:00.275) 0:01:52.999 ********** 2021-04-09T18:06:28.773 INFO:teuthology.task.ansible.out: 2021-04-09T18:06:28.775 INFO:teuthology.task.ansible.out:TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-04-09T18:06:28.804 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:28 +0000 (0:00:00.582) 0:01:53.581 ********** 2021-04-09T18:06:29.352 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-04-09T18:06:29.354 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-04-09T18:06:29.374 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:29 +0000 (0:00:00.569) 0:01:54.151 ********** 2021-04-09T18:06:29.909 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-04-09T18:06:29.911 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:08.327540", "end": "2021-04-09 18:06:29.883870", "rc": 0, "start": "2021-04-09 18:06:21.556330", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:06:29.931 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:29 +0000 (0:00:00.557) 0:01:54.708 ********** 2021-04-09T18:06:30.187 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-04-09T18:06:30.189 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-04-09T18:06:30.209 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:30 +0000 (0:00:00.277) 0:01:54.986 ********** 2021-04-09T18:06:30.277 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-04-09T18:06:30.279 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2021-04-09T18:06:30.301 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:30 +0000 (0:00:00.092) 0:01:55.078 ********** 2021-04-09T18:06:30.370 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-04-09T18:06:30.372 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-04-09T18:06:30.403 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:30 +0000 (0:00:00.101) 0:01:55.180 ********** 2021-04-09T18:06:30.473 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-04-09T18:06:30.475 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "podman", "podman-docker", "python3"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2021-04-09T18:06:30.507 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:30 +0000 (0:00:00.104) 0:01:55.284 ********** 2021-04-09T18:06:30.578 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-04-09T18:06:30.580 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-04-09T18:06:30.610 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:30 +0000 (0:00:00.103) 0:01:55.388 ********** 2021-04-09T18:06:30.796 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-04-09T18:06:30.798 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-04-09T18:06:30.819 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:30 +0000 (0:00:00.209) 0:01:55.597 ********** 2021-04-09T18:06:31.043 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-04-09T18:06:31.045 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-04-09T18:06:31.064 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:31 +0000 (0:00:00.244) 0:01:55.842 ********** 2021-04-09T18:06:31.132 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-04-09T18:06:31.133 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:31.156 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:31 +0000 (0:00:00.092) 0:01:55.934 ********** 2021-04-09T18:06:31.224 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-04-09T18:06:31.226 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:31.246 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:31 +0000 (0:00:00.088) 0:01:56.023 ********** 2021-04-09T18:06:31.406 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-04-09T18:06:31.408 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.819674", "end": "2021-04-09 18:06:31.386162", "rc": 0, "start": "2021-04-09 18:06:29.566488", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:06:31.430 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:31 +0000 (0:00:00.184) 0:01:56.207 ********** 2021-04-09T18:06:31.497 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-04-09T18:06:31.499 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:31.519 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:31 +0000 (0:00:00.088) 0:01:56.296 ********** 2021-04-09T18:06:31.524 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-04-09T18:06:31.526 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-04-09T18:06:31.548 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:31 +0000 (0:00:00.028) 0:01:56.325 ********** 2021-04-09T18:06:32.073 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-04-09T18:06:32.074 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item={'key': 'rhel-7-fcgi-ceph', 'value': {'name': 'RHEL 7 Local fastcgi Repo', 'baseurl': 'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2021-04-09T18:06:32.178 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-04-09T18:06:32.180 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-04-09T18:06:32.204 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:32 +0000 (0:00:00.656) 0:01:56.981 ********** 2021-04-09T18:06:32.272 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-04-09T18:06:32.273 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi032"}, "changed": false} 2021-04-09T18:06:32.301 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:32 +0000 (0:00:00.097) 0:01:57.078 ********** 2021-04-09T18:06:32.548 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-04-09T18:06:32.550 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2021-04-09T18:06:32.574 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:32 +0000 (0:00:00.272) 0:01:57.351 ********** 2021-04-09T18:06:32.642 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-04-09T18:06:32.664 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:32 +0000 (0:00:00.090) 0:01:57.442 ********** 2021-04-09T18:06:32.671 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-04-09T18:06:32.673 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi032", "ansible_hostname": "smithi032", "ansible_nodename": "smithi032"}, "changed": false, "name": "smithi032"} 2021-04-09T18:06:32.696 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:32 +0000 (0:00:00.032) 0:01:57.474 ********** 2021-04-09T18:06:32.724 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-04-09T18:06:32.747 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:32 +0000 (0:00:00.050) 0:01:57.524 ********** 2021-04-09T18:06:32.815 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-04-09T18:06:32.817 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:32.839 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:32 +0000 (0:00:00.092) 0:01:57.617 ********** 2021-04-09T18:06:32.918 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-04-09T18:06:32.919 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi086.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-04-09T18:06:32.940 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:32 +0000 (0:00:00.100) 0:01:57.718 ********** 2021-04-09T18:06:33.009 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:33.031 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:33 +0000 (0:00:00.090) 0:01:57.809 ********** 2021-04-09T18:06:33.100 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:06:33.101 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:33.123 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:33 +0000 (0:00:00.091) 0:01:57.901 ********** 2021-04-09T18:06:33.254 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-04-09T18:06:33.255 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-04-09T18:06:33.277 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:33 +0000 (0:00:00.153) 0:01:58.054 ********** 2021-04-09T18:06:33.792 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-04-09T18:06:33.794 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-54.el7_8.noarch providing yum-utils is already installed"]} 2021-04-09T18:06:33.816 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:33 +0000 (0:00:00.538) 0:01:58.593 ********** 2021-04-09T18:06:33.832 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-04-09T18:06:33.833 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-04-09T18:06:33.856 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:33 +0000 (0:00:00.040) 0:01:58.633 ********** 2021-04-09T18:06:34.426 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-04-09T18:06:34.428 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-04-09T18:06:34.448 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:34 +0000 (0:00:00.592) 0:01:59.225 ********** 2021-04-09T18:06:34.797 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-04-09T18:06:34.799 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.754054", "end": "2021-04-09 18:06:34.774934", "rc": 0, "start": "2021-04-09 18:06:34.020880", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: download-cc-rdu01.fedoraproject.org", "No unfinished transactions left."]} 2021-04-09T18:06:34.820 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:34 +0000 (0:00:00.372) 0:01:59.598 ********** 2021-04-09T18:06:35.017 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-04-09T18:06:35.018 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-04-09T18:06:35.050 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:35 +0000 (0:00:00.230) 0:01:59.828 ********** 2021-04-09T18:06:35.070 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-04-09T18:06:35.071 INFO:teuthology.task.ansible.out:fatal: [smithi086.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.023359", "end": "2021-04-09 18:06:35.042787", "msg": "non-zero return code", "rc": 1, "start": "2021-04-09 18:06:35.019428", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-04-09T18:06:35.095 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:35 +0000 (0:00:00.044) 0:01:59.872 ********** 2021-04-09T18:06:35.165 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-04-09T18:06:35.166 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:35.188 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:35 +0000 (0:00:00.092) 0:01:59.965 ********** 2021-04-09T18:06:35.290 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-04-09T18:06:35.292 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-04-09T18:06:35.314 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:35 +0000 (0:00:00.125) 0:02:00.091 ********** 2021-04-09T18:06:35.552 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-04-09T18:06:35.553 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2021-04-09T18:06:35.574 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:35 +0000 (0:00:00.260) 0:02:00.351 ********** 2021-04-09T18:06:35.839 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-04-09T18:06:35.841 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-04-09T18:06:35.860 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:35 +0000 (0:00:00.286) 0:02:00.638 ********** 2021-04-09T18:06:36.426 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-04-09T18:06:36.428 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-04-09T18:06:36.450 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:36 +0000 (0:00:00.589) 0:02:01.227 ********** 2021-04-09T18:06:37.015 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-04-09T18:06:37.017 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-04-09T18:06:37.038 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:37 +0000 (0:00:00.588) 0:02:01.815 ********** 2021-04-09T18:06:39.227 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-04-09T18:06:39.229 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.963939", "end": "2021-04-09 18:06:39.204900", "rc": 0, "start": "2021-04-09 18:06:37.240961", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:06:39.249 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:39 +0000 (0:00:02.211) 0:02:04.027 ********** 2021-04-09T18:06:39.318 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:39.346 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:39 +0000 (0:00:00.096) 0:02:04.123 ********** 2021-04-09T18:06:39.935 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-04-09T18:06:39.937 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item={'key': 'rhel-7-fcgi-ceph', 'value': {'name': 'RHEL 7 Local fastcgi Repo', 'baseurl': 'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2021-04-09T18:06:40.439 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2021-04-09T18:06:40.464 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:40 +0000 (0:00:01.118) 0:02:05.241 ********** 2021-04-09T18:06:40.532 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-04-09T18:06:40.553 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:40 +0000 (0:00:00.088) 0:02:05.330 ********** 2021-04-09T18:06:40.610 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-04-09T18:06:40.631 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:40 +0000 (0:00:00.078) 0:02:05.409 ********** 2021-04-09T18:06:40.698 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-04-09T18:06:40.700 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:40.720 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:40 +0000 (0:00:00.088) 0:02:05.497 ********** 2021-04-09T18:06:40.798 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-04-09T18:06:40.800 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-04-09T18:06:40.821 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:40 +0000 (0:00:00.101) 0:02:05.599 ********** 2021-04-09T18:06:40.891 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-04-09T18:06:40.896 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:40.914 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:40 +0000 (0:00:00.092) 0:02:05.691 ********** 2021-04-09T18:06:40.992 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:06:40.993 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:41.013 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:41 +0000 (0:00:00.099) 0:02:05.791 ********** 2021-04-09T18:06:41.747 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-04-09T18:06:41.749 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-54.el7_8.noarch providing yum-utils is already installed"]} 2021-04-09T18:06:41.769 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:41 +0000 (0:00:00.755) 0:02:06.547 ********** 2021-04-09T18:06:42.846 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-04-09T18:06:42.847 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.839686", "end": "2021-04-09 18:06:42.827681", "rc": 0, "start": "2021-04-09 18:06:41.987995", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: download-cc-rdu01.fedoraproject.org", "No unfinished transactions left."]} 2021-04-09T18:06:42.869 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:42 +0000 (0:00:01.099) 0:02:07.646 ********** 2021-04-09T18:06:43.133 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-04-09T18:06:43.135 INFO:teuthology.task.ansible.out:fatal: [smithi032.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.025813", "end": "2021-04-09 18:06:43.111712", "msg": "non-zero return code", "rc": 1, "start": "2021-04-09 18:06:43.085899", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-04-09T18:06:43.157 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:43 +0000 (0:00:00.287) 0:02:07.934 ********** 2021-04-09T18:06:43.225 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-04-09T18:06:43.226 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:06:43.247 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:43 +0000 (0:00:00.089) 0:02:08.024 ********** 2021-04-09T18:06:46.919 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-04-09T18:06:46.920 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-175.el7_9.3.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7 2021-04-09T18:06:46.921 INFO:teuthology.task.ansible.out:-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-175.el7_9.3 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages (+3 Dependent packages)\n\nInstalled size: 22 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 1/6 \n Erasing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 2/6 \n Erasing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 3/6 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/6 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 6/6 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/6 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/6 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 5/6 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 6/6 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-175.el7_9.3 2021-04-09T18:06:46.923 INFO:teuthology.task.ansible.out: \n qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-04-09T18:06:46.942 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:46 +0000 (0:00:03.695) 0:02:11.720 ********** 2021-04-09T18:06:52.335 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-04-09T18:06:52.337 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "leveldb", "xmlstarlet", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be erased\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-28.el7 @rhel-7-server-rpms 24 k\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages\n\nInstalled size: 655 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/3 \n Erasing : boost-random-1.53.0-28.el7.x86_64 2/3 \n Erasing : leveldb-1.12.0-11.el7.x86_64 2021-04-09T18:06:52.338 INFO:teuthology.task.ansible.out: 3/3 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/3 \n Verifying : boost-random-1.53.0-28.el7.x86_64 2/3 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 3/3 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-28.el7 leveldb.x86_64 0:1.12.0-11.el7 \n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-04-09T18:06:52.357 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:52 +0000 (0:00:05.414) 0:02:17.134 ********** 2021-04-09T18:06:54.789 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-04-09T18:06:54.791 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-175.el7_9.3.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7 2021-04-09T18:06:54.793 INFO:teuthology.task.ansible.out:-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-175.el7_9.3 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages (+3 Dependent packages)\n\nInstalled size: 22 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 1/6 \n Erasing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 2/6 \n Erasing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 3/6 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/6 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 6/6 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/6 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/6 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 5/6 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 6/6 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-175.el7_9.3 2021-04-09T18:06:54.794 INFO:teuthology.task.ansible.out: \n qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-04-09T18:06:54.812 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:06:54 +0000 (0:00:02.454) 0:02:19.589 ********** 2021-04-09T18:07:00.378 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-04-09T18:07:00.379 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "leveldb", "xmlstarlet", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be erased\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-28.el7 @rhel-7-server-rpms 24 k\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages\n\nInstalled size: 655 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/3 \n Erasing : boost-random-1.53.0-28.el7.x86_64 2/3 \n Erasing : leveldb-1.12.0-11.el7.x86_64 2021-04-09T18:07:00.381 INFO:teuthology.task.ansible.out: 3/3 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/3 \n Verifying : boost-random-1.53.0-28.el7.x86_64 2/3 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 3/3 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-28.el7 leveldb.x86_64 0:1.12.0-11.el7 \n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-04-09T18:07:00.400 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:00 +0000 (0:00:05.588) 0:02:25.178 ********** 2021-04-09T18:07:12.914 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-09T18:07:12.916 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-54.el7_8.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-19.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-120.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-23.el7_8.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-44.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-28.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-22.el7.x86_64 providing xfsprogs is already installed", "xfsprogs-devel-4.5.0-22.el7.x86_64 providing xfsprogs-devel is already installed", "gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "parted-3.1-32.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "7:lvm2-2.02.187-6.el7.x86_64 providing lvm2 is already installed", "1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-65.el7_9.1.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.15.0-11.el7.x86_64 providing valgrind is already installed", "python-n 2021-04-09T18:07:12.918 INFO:teuthology.task.ansible.out:ose-1.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "lsof-4.87-6.el7.x86_64 providing lsof is already installed", "iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-3.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-65.el7_9.1.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-15.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-97.el7_9.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-97.el7_9.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-97.el7_9.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-97.el7_9.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.68.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing 2021-04-09T18:07:12.919 INFO:teuthology.task.ansible.out:ncurses-devel is already installed", "python-devel-2.7.5-90.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-4.el7_8.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-299.el7_9.noarch providing perl-CPAN is already installed", "podman-1.6.4-29.el7_9.x86_64 providing podman is already installed", "podman-docker-1.6.4-29.el7_9.noarch providing podman-docker is already installed", "python3-3.6.8-18.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n========================================================================= 2021-04-09T18:07:12.921 INFO:teuthology.task.ansible.out:=======\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-28.el7 rhel-7-server-rpms 39 k\n librados2 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 704 k\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 440 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 26 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-28.el7.x86_64 1/7 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/7 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/7 \n Installing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 4/7 \n Installing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 5/7 \n Installing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 7/7 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 2021-04-09T18:07:12.923 INFO:teuthology.task.ansible.out: 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/7 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/7 \n Verifying : boost-random-1.53.0-28.el7.x86_64 4/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 7/7 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-28.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-04-09T18:07:12.938 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:12 +0000 (0:00:12.538) 0:02:37.716 ********** 2021-04-09T18:07:18.554 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-04-09T18:07:18.556 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["leveldb", "xmlstarlet"]}, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages\n\nTotal download size: 223 k\nInstalled size: 631 k\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 4.5 MB/s | 223 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 2021-04-09T18:07:18.558 INFO:teuthology.task.ansible.out: 1/2 \n Installing : leveldb-1.12.0-11.el7.x86_64 2/2 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/2 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 2/2 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-04-09T18:07:18.576 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:18 +0000 (0:00:05.637) 0:02:43.354 ********** 2021-04-09T18:07:19.314 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-04-09T18:07:19.316 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2021-04-09T18:07:19.336 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:19 +0000 (0:00:00.759) 0:02:44.114 ********** 2021-04-09T18:07:19.404 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-09T18:07:19.406 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:19.426 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:19 +0000 (0:00:00.089) 0:02:44.203 ********** 2021-04-09T18:07:19.713 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-04-09T18:07:19.715 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service system.slice dbus.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "iptables.service ip6tables.service shutdown.target ipset.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "Ig 2021-04-09T18:07:19.717 INFO:teuthology.task.ansible.out:noreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", 2021-04-09T18:07:19.719 INFO:teuthology.task.ansible.out: "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-04-09T18:07:19.737 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:19 +0000 (0:00:00.310) 0:02:44.514 ********** 2021-04-09T18:07:20.014 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-04-09T18:07:20.016 INFO:teuthology.task.ansible.out:fatal: [smithi086.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-04-09T18:07:20.038 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:20 +0000 (0:00:00.301) 0:02:44.815 ********** 2021-04-09T18:07:20.700 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-04-09T18:07:20.702 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-04-09T18:07:20.722 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:20 +0000 (0:00:00.684) 0:02:45.499 ********** 2021-04-09T18:07:20.790 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-04-09T18:07:20.791 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:20.811 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:20 +0000 (0:00:00.088) 0:02:45.588 ********** 2021-04-09T18:07:20.878 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-04-09T18:07:20.880 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:20.899 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:20 +0000 (0:00:00.088) 0:02:45.677 ********** 2021-04-09T18:07:20.967 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-04-09T18:07:20.969 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:20.990 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:20 +0000 (0:00:00.090) 0:02:45.768 ********** 2021-04-09T18:07:21.058 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-04-09T18:07:21.060 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.080 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.089) 0:02:45.857 ********** 2021-04-09T18:07:21.138 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-04-09T18:07:21.140 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.160 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.080) 0:02:45.938 ********** 2021-04-09T18:07:21.230 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-04-09T18:07:21.232 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.252 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.092) 0:02:46.030 ********** 2021-04-09T18:07:21.310 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-04-09T18:07:21.331 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.078) 0:02:46.109 ********** 2021-04-09T18:07:21.388 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-04-09T18:07:21.390 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.410 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.078) 0:02:46.188 ********** 2021-04-09T18:07:21.478 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-04-09T18:07:21.480 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.501 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.090) 0:02:46.279 ********** 2021-04-09T18:07:21.569 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-04-09T18:07:21.571 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.581 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.602 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.101) 0:02:46.380 ********** 2021-04-09T18:07:21.670 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-04-09T18:07:21.672 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.692 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.089) 0:02:46.469 ********** 2021-04-09T18:07:21.770 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-04-09T18:07:21.791 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.099) 0:02:46.569 ********** 2021-04-09T18:07:21.859 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-04-09T18:07:21.861 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.883 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.091) 0:02:46.660 ********** 2021-04-09T18:07:21.950 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-04-09T18:07:21.952 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:21.973 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:21 +0000 (0:00:00.090) 0:02:46.751 ********** 2021-04-09T18:07:22.041 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-04-09T18:07:22.042 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.065 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.091) 0:02:46.842 ********** 2021-04-09T18:07:22.122 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-09T18:07:22.124 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.144 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.079) 0:02:46.922 ********** 2021-04-09T18:07:22.212 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-04-09T18:07:22.214 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.234 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.090) 0:02:47.012 ********** 2021-04-09T18:07:22.294 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-09T18:07:22.295 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-54.el7_8.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-19.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-120.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-23.el7_8.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-44.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-28.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-22.el7.x86_64 providing xfsprogs is already installed", "xfsprogs-devel-4.5.0-22.el7.x86_64 providing xfsprogs-devel is already installed", "gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "parted-3.1-32.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "7:lvm2-2.02.187-6.el7.x86_64 providing lvm2 is already installed", "1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-65.el7_9.1.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.15.0-11.el7.x86_64 providing valgrind is already installed", "python-n 2021-04-09T18:07:22.297 INFO:teuthology.task.ansible.out:ose-1.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "lsof-4.87-6.el7.x86_64 providing lsof is already installed", "iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-3.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-65.el7_9.1.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-15.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-97.el7_9.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-97.el7_9.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-97.el7_9.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-97.el7_9.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.68.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing 2021-04-09T18:07:22.298 INFO:teuthology.task.ansible.out:ncurses-devel is already installed", "python-devel-2.7.5-90.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-4.el7_8.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-299.el7_9.noarch providing perl-CPAN is already installed", "podman-1.6.4-29.el7_9.x86_64 providing podman is already installed", "podman-docker-1.6.4-29.el7_9.noarch providing podman-docker is already installed", "python3-3.6.8-18.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n========================================================================= 2021-04-09T18:07:22.300 INFO:teuthology.task.ansible.out:=======\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-28.el7 rhel-7-server-rpms 39 k\n librados2 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 704 k\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 440 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 24 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-28.el7.x86_64 1/7 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/7 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/7 \n Installing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 4/7 \n Installing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 5/7 \n Installing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 7/7 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 2021-04-09T18:07:22.302 INFO:teuthology.task.ansible.out: 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/7 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/7 \n Verifying : boost-random-1.53.0-28.el7.x86_64 4/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 7/7 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-28.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-04-09T18:07:22.322 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.087) 0:02:47.099 ********** 2021-04-09T18:07:22.329 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-09T18:07:22.331 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.362 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.040) 0:02:47.139 ********** 2021-04-09T18:07:22.433 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-04-09T18:07:22.435 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.455 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.092) 0:02:47.232 ********** 2021-04-09T18:07:22.534 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-04-09T18:07:22.536 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.561 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.106) 0:02:47.338 ********** 2021-04-09T18:07:22.631 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-04-09T18:07:22.633 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.656 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.094) 0:02:47.433 ********** 2021-04-09T18:07:22.723 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-04-09T18:07:22.725 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.745 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.089) 0:02:47.523 ********** 2021-04-09T18:07:22.814 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-04-09T18:07:22.816 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.844 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.099) 0:02:47.622 ********** 2021-04-09T18:07:22.913 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-04-09T18:07:22.915 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:22.936 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:22 +0000 (0:00:00.091) 0:02:47.713 ********** 2021-04-09T18:07:23.006 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-04-09T18:07:23.008 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.028 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.091) 0:02:47.805 ********** 2021-04-09T18:07:23.096 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-04-09T18:07:23.098 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.118 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.089) 0:02:47.895 ********** 2021-04-09T18:07:23.186 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-04-09T18:07:23.188 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.211 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.093) 0:02:47.989 ********** 2021-04-09T18:07:23.279 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-04-09T18:07:23.281 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.301 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.089) 0:02:48.078 ********** 2021-04-09T18:07:23.369 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-04-09T18:07:23.371 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.390 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.089) 0:02:48.168 ********** 2021-04-09T18:07:23.469 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-04-09T18:07:23.471 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.490 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.100) 0:02:48.268 ********** 2021-04-09T18:07:23.559 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-04-09T18:07:23.561 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.581 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.090) 0:02:48.359 ********** 2021-04-09T18:07:23.640 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-04-09T18:07:23.642 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.662 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.080) 0:02:48.439 ********** 2021-04-09T18:07:23.741 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-04-09T18:07:23.743 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.765 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.102) 0:02:48.542 ********** 2021-04-09T18:07:23.853 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-04-09T18:07:23.855 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.875 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.110) 0:02:48.653 ********** 2021-04-09T18:07:23.953 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-04-09T18:07:23.955 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:23.979 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:23 +0000 (0:00:00.103) 0:02:48.757 ********** 2021-04-09T18:07:24.048 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:24.050 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:24.075 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:24 +0000 (0:00:00.095) 0:02:48.852 ********** 2021-04-09T18:07:24.164 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-04-09T18:07:24.165 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:24.185 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:24 +0000 (0:00:00.110) 0:02:48.963 ********** 2021-04-09T18:07:24.256 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-09T18:07:24.258 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:24.279 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:24 +0000 (0:00:00.093) 0:02:49.056 ********** 2021-04-09T18:07:24.347 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-09T18:07:24.349 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:24.377 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:24 +0000 (0:00:00.098) 0:02:49.154 ********** 2021-04-09T18:07:24.456 INFO:teuthology.task.ansible.out: 2021-04-09T18:07:24.458 INFO:teuthology.task.ansible.out:TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:24.479 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:24 +0000 (0:00:00.102) 0:02:49.257 ********** 2021-04-09T18:07:24.548 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-09T18:07:24.550 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:24.570 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:24 +0000 (0:00:00.091) 0:02:49.348 ********** 2021-04-09T18:07:24.650 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-09T18:07:24.652 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:24.672 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:24 +0000 (0:00:00.102) 0:02:49.450 ********** 2021-04-09T18:07:24.740 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-09T18:07:24.742 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:24.762 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:24 +0000 (0:00:00.090) 0:02:49.540 ********** 2021-04-09T18:07:25.326 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-04-09T18:07:25.327 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2021-04-09T18:07:25.359 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:25 +0000 (0:00:00.596) 0:02:50.136 ********** 2021-04-09T18:07:25.654 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-04-09T18:07:25.655 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Fri 2021-04-09 18:03:46 UTC", "ActiveEnterTimestampMonotonic": "21285194", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target system.slice systemd-journald.socket rpc-gssd.service nfs-mountd.service rpc-statd.service network-online.target proc-fs-nfsd.mount nfs-config.service -.mount nfs-idmapd.service gssproxy.service rpcbind.socket tmp.mount", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-04-09 18:03:46 UTC", "AssertTimestampMonotonic": "21065646", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-idmapd.service nfs-mountd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-04-09 18:03:46 UTC", "ConditionTimestampMonotonic": "21065645", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Fri 2021-04-09 18:03:46 UTC", "ExecMainExitTimestampMonotonic": "21249556", "ExecMainPID": "1169", "ExecMainStartTimestamp": "Fri 2021-04-09 18:03:46 UTC", "ExecMainStartTimestampMonotonic": "21085546", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=ye 2021-04-09T18:07:25.657 INFO:teuthology.task.ansible.out:s ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-04-09 18:03:46 UTC", "InactiveExitTimestampMonotonic": "21066817", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "l 2021-04-09T18:07:25.659 INFO:teuthology.task.ansible.out:oaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "network.target system.slice -.mount nfs-mountd.service proc-fs-nfsd.mount", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "rpc-statd-notify.service nfs-config.service auth-rpcgss-module.service rpc-statd.service network-online.target rpcbind.socket nfs-idmapd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-04-09T18:07:25.679 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:25 +0000 (0:00:00.320) 0:02:50.457 ********** 2021-04-09T18:07:25.759 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-04-09T18:07:25.761 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:25.783 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:25 +0000 (0:00:00.103) 0:02:50.560 ********** 2021-04-09T18:07:25.852 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-09T18:07:25.854 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:25.879 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:25 +0000 (0:00:00.096) 0:02:50.657 ********** 2021-04-09T18:07:25.948 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-09T18:07:25.949 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:25.970 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:25 +0000 (0:00:00.090) 0:02:50.747 ********** 2021-04-09T18:07:26.050 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-09T18:07:26.052 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.073 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.103) 0:02:50.851 ********** 2021-04-09T18:07:26.142 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-09T18:07:26.144 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.168 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.094) 0:02:50.945 ********** 2021-04-09T18:07:26.226 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-09T18:07:26.227 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.249 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.081) 0:02:51.027 ********** 2021-04-09T18:07:26.317 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-09T18:07:26.319 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.339 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.089) 0:02:51.116 ********** 2021-04-09T18:07:26.397 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-04-09T18:07:26.398 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.418 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.079) 0:02:51.196 ********** 2021-04-09T18:07:26.487 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-04-09T18:07:26.489 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.509 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.090) 0:02:51.286 ********** 2021-04-09T18:07:26.588 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-04-09T18:07:26.589 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.610 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.100) 0:02:51.387 ********** 2021-04-09T18:07:26.678 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-04-09T18:07:26.680 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.700 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.089) 0:02:51.477 ********** 2021-04-09T18:07:26.768 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-04-09T18:07:26.770 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.790 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.090) 0:02:51.567 ********** 2021-04-09T18:07:26.858 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-04-09T18:07:26.859 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.880 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.090) 0:02:51.658 ********** 2021-04-09T18:07:26.948 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-04-09T18:07:26.950 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:26.975 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:26 +0000 (0:00:00.094) 0:02:51.752 ********** 2021-04-09T18:07:27.043 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-04-09T18:07:27.045 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.065 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:27 +0000 (0:00:00.090) 0:02:51.842 ********** 2021-04-09T18:07:27.135 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-04-09T18:07:27.137 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.156 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:27 +0000 (0:00:00.091) 0:02:51.934 ********** 2021-04-09T18:07:27.224 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-04-09T18:07:27.226 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.248 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:27 +0000 (0:00:00.091) 0:02:52.026 ********** 2021-04-09T18:07:27.316 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-04-09T18:07:27.318 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.338 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:27 +0000 (0:00:00.089) 0:02:52.115 ********** 2021-04-09T18:07:27.395 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.422 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:27 +0000 (0:00:00.084) 0:02:52.199 ********** 2021-04-09T18:07:27.490 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-04-09T18:07:27.492 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.513 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:27 +0000 (0:00:00.090) 0:02:52.290 ********** 2021-04-09T18:07:27.582 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:27.583 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-04-09T18:07:27.609 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:27 +0000 (0:00:00.096) 0:02:52.387 ********** 2021-04-09T18:07:27.689 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-04-09T18:07:27.691 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi086.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-04-09T18:07:27.701 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.703 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.712 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.714 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.724 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:27.749 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:27 +0000 (0:00:00.139) 0:02:52.526 ********** 2021-04-09T18:07:27.786 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-04-09T18:07:27.788 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["leveldb", "xmlstarlet"]}, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages\n\nTotal download size: 223 k\nInstalled size: 631 k\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 4.2 MB/s | 223 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 2021-04-09T18:07:27.790 INFO:teuthology.task.ansible.out: 1/2 \n Installing : leveldb-1.12.0-11.el7.x86_64 2/2 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/2 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 2/2 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-04-09T18:07:27.815 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:27 +0000 (0:00:00.066) 0:02:52.592 ********** 2021-04-09T18:07:28.011 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-04-09T18:07:28.013 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.002888", "end": "2021-04-09 18:07:27.988381", "rc": 0, "start": "2021-04-09 18:07:27.985493", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-lock"]} 2021-04-09T18:07:28.038 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:28 +0000 (0:00:00.223) 0:02:52.816 ********** 2021-04-09T18:07:28.118 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-04-09T18:07:28.120 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:28.142 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:28 +0000 (0:00:00.104) 0:02:52.920 ********** 2021-04-09T18:07:28.221 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-04-09T18:07:28.223 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:28.244 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:28 +0000 (0:00:00.101) 0:02:53.022 ********** 2021-04-09T18:07:28.313 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-04-09T18:07:28.315 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:28.335 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:28 +0000 (0:00:00.090) 0:02:53.113 ********** 2021-04-09T18:07:28.403 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-04-09T18:07:28.405 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:28.426 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:28 +0000 (0:00:00.091) 0:02:53.204 ********** 2021-04-09T18:07:28.621 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-04-09T18:07:28.623 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2021-04-09T18:07:28.648 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:28 +0000 (0:00:00.222) 0:02:53.426 ********** 2021-04-09T18:07:28.727 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-09T18:07:28.729 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:28.750 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:28 +0000 (0:00:00.101) 0:02:53.527 ********** 2021-04-09T18:07:29.057 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-04-09T18:07:29.057 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service dbus.service basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "iptables.service ebtables.service ipset.service shutdown.target ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "Ig 2021-04-09T18:07:29.059 INFO:teuthology.task.ansible.out:noreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127535", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127535", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", 2021-04-09T18:07:29.061 INFO:teuthology.task.ansible.out: "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-04-09T18:07:29.079 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.329) 0:02:53.857 ********** 2021-04-09T18:07:29.116 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-04-09T18:07:29.187 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.170-6.el7.x86_64 providing device-mapper is already installed"]} Friday 09 April 2021 18:07:29 +0000 (0:00:00.060) 0:02:53.917 ********** 2021-04-09T18:07:29.222 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-04-09T18:07:29.224 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item={'block_used': 13831461, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226493023, 'size_available': 927715422208, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90206, 'block_size': 4096, 'inode_available': 60964770}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226493023, "block_size": 4096, "block_total": 240324484, "block_used": 13831461, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964770, "inode_total": 61054976, "inode_used": 90206, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 927715422208, "size_total": 984369086464, "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}} 2021-04-09T18:07:29.249 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.109) 0:02:54.027 ********** 2021-04-09T18:07:29.365 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-04-09T18:07:29.366 INFO:teuthology.task.ansible.out:fatal: [smithi032.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-04-09T18:07:29.390 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.140) 0:02:54.168 ********** 2021-04-09T18:07:29.471 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-04-09T18:07:29.473 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005799", "end": "2021-04-09 18:07:29.448555", "rc": 0, "start": "2021-04-09 18:07:29.442756", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:29.494 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.103) 0:02:54.271 ********** 2021-04-09T18:07:29.574 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-04-09T18:07:29.576 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'block_used': 13831461, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226493023, 'size_available': 927715422208, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90206, 'block_size': 4096, 'inode_available': 60964770}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226493023, "block_size": 4096, "block_total": 240324484, "block_used": 13831461, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964770, "inode_total": 61054976, "inode_used": 90206, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 927715422208, "size_total": 984369086464, "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:29.603 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.109) 0:02:54.381 ********** 2021-04-09T18:07:29.672 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-04-09T18:07:29.674 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-04-09T18:07:29.704 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.100) 0:02:54.482 ********** 2021-04-09T18:07:29.728 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.023) 0:02:54.505 ********** 2021-04-09T18:07:29.767 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-04-09T18:07:29.769 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:29.793 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.065) 0:02:54.570 ********** 2021-04-09T18:07:29.811 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:29.813 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGQ8', 'wwn-0x5000c500918e8385'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGQ8-part1', 'wwn-0x5000c500918e8385-part1'], 'uuids': ['7884d11b-1a98-41be-b51a-86bec1e24d7d']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e8385', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LGQ8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGQ8", "wwn-0x5000c500918e8385"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGQ8-part1", "wwn-0x5000c500918e8385-part1"], "labels": [], "masters": [], "uuids": ["7884d11b-1a98-41be-b51a-86bec1e24d7d"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LGQ8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e8385"}}, "skip 2021-04-09T18:07:29.815 INFO:teuthology.task.ansible.out:_reason": "Conditional result was False"} 2021-04-09T18:07:29.855 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-04-09T18:07:29.856 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:29.876 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.083) 0:02:54.654 ********** 2021-04-09T18:07:29.945 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-04-09T18:07:29.947 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:29.969 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:29 +0000 (0:00:00.092) 0:02:54.746 ********** 2021-04-09T18:07:30.037 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-04-09T18:07:30.039 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.062 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.093) 0:02:54.839 ********** 2021-04-09T18:07:30.133 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-04-09T18:07:30.135 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.155 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.093) 0:02:54.933 ********** 2021-04-09T18:07:30.224 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-04-09T18:07:30.226 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.246 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.090) 0:02:55.023 ********** 2021-04-09T18:07:30.305 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-04-09T18:07:30.329 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.083) 0:02:55.107 ********** 2021-04-09T18:07:30.400 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-04-09T18:07:30.401 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.421 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.092) 0:02:55.199 ********** 2021-04-09T18:07:30.500 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-04-09T18:07:30.502 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.522 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.100) 0:02:55.299 ********** 2021-04-09T18:07:30.590 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-04-09T18:07:30.592 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.602 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.624 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.101) 0:02:55.401 ********** 2021-04-09T18:07:30.692 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-04-09T18:07:30.693 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.719 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.095) 0:02:55.497 ********** 2021-04-09T18:07:30.792 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-04-09T18:07:30.818 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.098) 0:02:55.596 ********** 2021-04-09T18:07:30.887 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-04-09T18:07:30.889 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.909 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.090) 0:02:55.687 ********** 2021-04-09T18:07:30.977 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-04-09T18:07:30.978 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:30.999 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:30 +0000 (0:00:00.089) 0:02:55.776 ********** 2021-04-09T18:07:31.067 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-04-09T18:07:31.069 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.088 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.089) 0:02:55.866 ********** 2021-04-09T18:07:31.158 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-09T18:07:31.159 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.185 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.096) 0:02:55.963 ********** 2021-04-09T18:07:31.243 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-04-09T18:07:31.244 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.265 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.079) 0:02:56.042 ********** 2021-04-09T18:07:31.333 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-09T18:07:31.335 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.361 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.096) 0:02:56.138 ********** 2021-04-09T18:07:31.421 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.442 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.081) 0:02:56.220 ********** 2021-04-09T18:07:31.480 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:31.482 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-Cr5RHZ-jSgc-Waf5-leBr-ihVA-TmOX-IBbjZx', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400UL400BGN', 'nvme-nvme.8086-50484654363230343030554c34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEDMD400G4', 'serial': 'PHFT620400UL400BGN', 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.493678", "end": "2021-04-09 18:07:31.449266", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Cr5RHZ-jSgc-Waf5-leBr-ihVA-TmOX-IBbjZx", "nvme-INTEL_SSDPEDMD400G4_PHFT620400UL400BGN", "nvme-nvme.8086-50484654363230343030554c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:29.955588", "stderr": "", "stderr_lines": [], "stdo 2021-04-09T18:07:31.483 INFO:teuthology.task.ansible.out:ut": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:31.513 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-04-09T18:07:31.515 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.535 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.092) 0:02:56.313 ********** 2021-04-09T18:07:31.604 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-04-09T18:07:31.606 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.627 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.091) 0:02:56.404 ********** 2021-04-09T18:07:31.696 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-04-09T18:07:31.698 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.719 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.092) 0:02:56.497 ********** 2021-04-09T18:07:31.721 INFO:teuthology.task.ansible.out: 2021-04-09T18:07:31.789 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-04-09T18:07:31.791 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.811 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.090) 0:02:56.588 ********** 2021-04-09T18:07:31.900 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-04-09T18:07:31.901 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:31.924 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:31 +0000 (0:00:00.113) 0:02:56.701 ********** 2021-04-09T18:07:32.003 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-04-09T18:07:32.005 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.026 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.102) 0:02:56.804 ********** 2021-04-09T18:07:32.096 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-04-09T18:07:32.098 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.123 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.097) 0:02:56.901 ********** 2021-04-09T18:07:32.203 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.235 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.111) 0:02:57.012 ********** 2021-04-09T18:07:32.305 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-04-09T18:07:32.307 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.335 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.099) 0:02:57.112 ********** 2021-04-09T18:07:32.404 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-04-09T18:07:32.405 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.431 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.096) 0:02:57.209 ********** 2021-04-09T18:07:32.502 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-04-09T18:07:32.503 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.529 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.097) 0:02:57.306 ********** 2021-04-09T18:07:32.588 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-04-09T18:07:32.590 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.610 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.080) 0:02:57.387 ********** 2021-04-09T18:07:32.688 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-04-09T18:07:32.691 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.711 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.101) 0:02:57.489 ********** 2021-04-09T18:07:32.718 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:32.720 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuINL5wOl2IcKVFpSae8oElQAX1PVhzvcw'], 'uuids': ['35992020-1f6b-427b-84ca-e76f6e4500c3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.046913", "end": "2021-04-09 18:07:32.677617", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuINL5wOl2IcKVFpSae8oElQAX1PVhzvcw"], "labels": [], "masters": [], "uuids": ["35992020-1f6b-427b-84ca-e76f6e4500c3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:31.630704", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:32.773 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-04-09T18:07:32.775 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.795 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.083) 0:02:57.572 ********** 2021-04-09T18:07:32.884 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-04-09T18:07:32.886 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:32.907 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:32 +0000 (0:00:00.112) 0:02:57.684 ********** 2021-04-09T18:07:32.986 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-04-09T18:07:32.988 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:33.008 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:33 +0000 (0:00:00.101) 0:02:57.786 ********** 2021-04-09T18:07:33.077 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:33.079 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:33.101 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:33 +0000 (0:00:00.092) 0:02:57.878 ********** 2021-04-09T18:07:33.181 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-04-09T18:07:33.183 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:33.203 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:33 +0000 (0:00:00.102) 0:02:57.980 ********** 2021-04-09T18:07:33.275 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-09T18:07:33.277 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:33.297 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:33 +0000 (0:00:00.094) 0:02:58.075 ********** 2021-04-09T18:07:33.370 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-09T18:07:33.372 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:33.392 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:33 +0000 (0:00:00.094) 0:02:58.169 ********** 2021-04-09T18:07:33.461 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-04-09T18:07:33.463 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:33.493 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:33 +0000 (0:00:00.100) 0:02:58.270 ********** 2021-04-09T18:07:33.564 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-09T18:07:33.566 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:33.601 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:33 +0000 (0:00:00.107) 0:02:58.378 ********** 2021-04-09T18:07:33.671 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-09T18:07:33.672 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:33.697 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:33 +0000 (0:00:00.096) 0:02:58.474 ********** 2021-04-09T18:07:33.766 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-09T18:07:33.767 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:33.792 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:33 +0000 (0:00:00.095) 0:02:58.570 ********** 2021-04-09T18:07:33.937 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:33.939 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUu9Mh2z2u8GxGXcwc2pDlAK9VCdVRkR8QJ'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.058470", "end": "2021-04-09 18:07:33.911275", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUu9Mh2z2u8GxGXcwc2pDlAK9VCdVRkR8QJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:32.852805", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:34.371 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-04-09T18:07:34.373 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2021-04-09T18:07:34.394 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:34 +0000 (0:00:00.601) 0:02:59.172 ********** 2021-04-09T18:07:34.706 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-04-09T18:07:34.707 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Fri 2021-04-09 18:02:27 UTC", "ActiveEnterTimestampMonotonic": "20685675", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network-online.target system.slice tmp.mount rpc-gssd.service rpc-statd.service local-fs.target nfs-mountd.service nfs-config.service -.mount nfs-idmapd.service proc-fs-nfsd.mount rpcbind.socket gssproxy.service systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-04-09 18:02:27 UTC", "AssertTimestampMonotonic": "20492329", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-mountd.service nfs-idmapd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-04-09 18:02:27 UTC", "ConditionTimestampMonotonic": "20492329", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Fri 2021-04-09 18:02:27 UTC", "ExecMainExitTimestampMonotonic": "20652294", "ExecMainPID": "1138", "ExecMainStartTimestamp": "Fri 2021-04-09 18:02:27 UTC", "ExecMainStartTimestampMonotonic": "20536609", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=ye 2021-04-09T18:07:34.708 INFO:teuthology.task.ansible.out:s ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-04-09 18:02:27 UTC", "InactiveExitTimestampMonotonic": "20493391", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127535", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127535", "LimitSTACK": "18446744073709551615", "LoadState": "l 2021-04-09T18:07:34.710 INFO:teuthology.task.ansible.out:oaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "network.target nfs-mountd.service -.mount proc-fs-nfsd.mount system.slice", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "rpc-statd-notify.service network-online.target nfs-config.service rpc-statd.service rpcbind.socket auth-rpcgss-module.service nfs-idmapd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-04-09T18:07:34.729 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:34 +0000 (0:00:00.334) 0:02:59.506 ********** 2021-04-09T18:07:34.797 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-04-09T18:07:34.799 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:34.820 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:34 +0000 (0:00:00.090) 0:02:59.597 ********** 2021-04-09T18:07:34.889 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-09T18:07:34.891 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:34.917 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:34 +0000 (0:00:00.096) 0:02:59.694 ********** 2021-04-09T18:07:34.985 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-09T18:07:34.987 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.010 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.092) 0:02:59.787 ********** 2021-04-09T18:07:35.091 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-09T18:07:35.093 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.115 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.105) 0:02:59.893 ********** 2021-04-09T18:07:35.165 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:35.167 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuR4lWKhEA4aE1DKJ84Q7CsfzVOtM30UpV'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.053566", "end": "2021-04-09 18:07:35.140908", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuR4lWKhEA4aE1DKJ84Q7CsfzVOtM30UpV"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:34.087342", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:35.177 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.198 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.082) 0:02:59.975 ********** 2021-04-09T18:07:35.277 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-09T18:07:35.279 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.299 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.100) 0:03:00.076 ********** 2021-04-09T18:07:35.373 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-09T18:07:35.375 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.406 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.107) 0:03:00.184 ********** 2021-04-09T18:07:35.475 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.498 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.091) 0:03:00.275 ********** 2021-04-09T18:07:35.567 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-04-09T18:07:35.569 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.599 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.101) 0:03:00.377 ********** 2021-04-09T18:07:35.668 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-04-09T18:07:35.670 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.689 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.090) 0:03:00.467 ********** 2021-04-09T18:07:35.758 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.779 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.089) 0:03:00.557 ********** 2021-04-09T18:07:35.858 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-04-09T18:07:35.861 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.880 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.100) 0:03:00.657 ********** 2021-04-09T18:07:35.948 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-04-09T18:07:35.950 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:35.972 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:35 +0000 (0:00:00.091) 0:03:00.749 ********** 2021-04-09T18:07:36.034 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-04-09T18:07:36.036 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.057 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:36 +0000 (0:00:00.084) 0:03:00.834 ********** 2021-04-09T18:07:36.126 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-04-09T18:07:36.128 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.149 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:36 +0000 (0:00:00.091) 0:03:00.926 ********** 2021-04-09T18:07:36.232 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-04-09T18:07:36.234 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.254 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:36 +0000 (0:00:00.105) 0:03:01.031 ********** 2021-04-09T18:07:36.323 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-04-09T18:07:36.325 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.345 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:36 +0000 (0:00:00.091) 0:03:01.123 ********** 2021-04-09T18:07:36.394 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:36.396 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUusznML3vBwc6koi3JL3g4lRj8MbFVeHPq'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.058328", "end": "2021-04-09 18:07:36.370356", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUusznML3vBwc6koi3JL3g4lRj8MbFVeHPq"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:35.312028", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:36.440 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.467 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:36 +0000 (0:00:00.121) 0:03:01.245 ********** 2021-04-09T18:07:36.548 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-04-09T18:07:36.550 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.572 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:36 +0000 (0:00:00.105) 0:03:01.350 ********** 2021-04-09T18:07:36.642 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-04-09T18:07:36.644 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.673 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:36 +0000 (0:00:00.101) 0:03:01.451 ********** 2021-04-09T18:07:36.755 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:36.757 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-04-09T18:07:36.777 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:36 +0000 (0:00:00.103) 0:03:01.554 ********** 2021-04-09T18:07:36.856 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-04-09T18:07:36.858 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.868 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-04-09T18:07:36.870 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.880 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.881 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.892 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.903 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:36.940 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:36 +0000 (0:00:00.163) 0:03:01.717 ********** 2021-04-09T18:07:37.167 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-04-09T18:07:37.169 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003281", "end": "2021-04-09 18:07:37.147192", "rc": 0, "start": "2021-04-09 18:07:37.143911", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-lock"]} 2021-04-09T18:07:37.191 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:37 +0000 (0:00:00.251) 0:03:01.969 ********** 2021-04-09T18:07:37.259 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-04-09T18:07:37.261 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:37.286 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:37 +0000 (0:00:00.094) 0:03:02.063 ********** 2021-04-09T18:07:37.359 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:37.385 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:37 +0000 (0:00:00.098) 0:03:02.162 ********** 2021-04-09T18:07:37.468 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-04-09T18:07:37.470 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:37.493 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:37 +0000 (0:00:00.107) 0:03:02.270 ********** 2021-04-09T18:07:37.562 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-04-09T18:07:37.565 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:37.594 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:37 +0000 (0:00:00.101) 0:03:02.371 ********** 2021-04-09T18:07:37.633 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:37.634 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuxcMZMoNNFVsL8ibqzGdnRZoOzdDTTLQe'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.060182", "end": "2021-04-09 18:07:37.605673", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuxcMZMoNNFVsL8ibqzGdnRZoOzdDTTLQe"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:36.545491", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:37.663 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:37 +0000 (0:00:00.069) 0:03:02.441 ********** 2021-04-09T18:07:37.743 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-09T18:07:37.745 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGQ8', 'wwn-0x5000c500918e8385'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGQ8-part1', 'wwn-0x5000c500918e8385-part1'], 'uuids': ['7884d11b-1a98-41be-b51a-86bec1e24d7d']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e8385', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LGQ8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGQ8", "wwn-0x5000c500918e8385"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGQ8-part1", "wwn-0x5000c500918e8385-part1"], "labels": [], "masters": [], "uuids": ["7884d11b-1a98-41be-b51a-86bec1e24d7d"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LGQ8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e8385"}}, "skip 2021-04-09T18:07:37.746 INFO:teuthology.task.ansible.out:_reason": "Conditional result was False"} 2021-04-09T18:07:37.955 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-Cr5RHZ-jSgc-Waf5-leBr-ihVA-TmOX-IBbjZx', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400UL400BGN', 'nvme-nvme.8086-50484654363230343030554c34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEDMD400G4', 'serial': 'PHFT620400UL400BGN', 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039094", "end": "2021-04-09 18:07:37.929614", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Cr5RHZ-jSgc-Waf5-leBr-ihVA-TmOX-IBbjZx", "nvme-INTEL_SSDPEDMD400G4_PHFT620400UL400BGN", "nvme-nvme.8086-50484654363230343030554c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:37.890520", "stderr": "110+0 records in\n110+0 records out\n115343 2021-04-09T18:07:37.957 INFO:teuthology.task.ansible.out:360 bytes (115 MB) copied, 0.0362408 s, 3.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0362408 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:38.271 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuINL5wOl2IcKVFpSae8oElQAX1PVhzvcw'], 'uuids': ['35992020-1f6b-427b-84ca-e76f6e4500c3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.148984", "end": "2021-04-09 18:07:38.246195", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuINL5wOl2IcKVFpSae8oElQAX1PVhzvcw"], "labels": [], "masters": [], "uuids": ["35992020-1f6b-427b-84ca-e76f6e4500c3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:38.097211", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.145944 s, 790 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.145944 s, 790 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:38.326 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-04-09T18:07:38.327 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.170-6.el7.x86_64 providing device-mapper is already installed"]} 2021-04-09T18:07:38.355 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:38 +0000 (0:00:00.692) 0:03:03.133 ********** 2021-04-09T18:07:38.436 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-04-09T18:07:38.438 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item={'block_used': 13831463, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226493021, 'size_available': 927715414016, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90208, 'block_size': 4096, 'inode_available': 60964768}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226493021, "block_size": 4096, "block_total": 240324484, "block_used": 13831463, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964768, "inode_total": 61054976, "inode_used": 90208, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 927715414016, "size_total": 984369086464, "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}} 2021-04-09T18:07:38.460 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:38 +0000 (0:00:00.104) 0:03:03.237 ********** 2021-04-09T18:07:38.581 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-09T18:07:38.583 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUu9Mh2z2u8GxGXcwc2pDlAK9VCdVRkR8QJ'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.144641", "end": "2021-04-09 18:07:38.553519", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUu9Mh2z2u8GxGXcwc2pDlAK9VCdVRkR8QJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:38.408878", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.141803 s, 813 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.141803 s, 813 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:38.688 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-04-09T18:07:38.689 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.006255", "end": "2021-04-09 18:07:38.672930", "rc": 0, "start": "2021-04-09 18:07:38.666675", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:38.711 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:38 +0000 (0:00:00.251) 0:03:03.489 ********** 2021-04-09T18:07:38.793 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-04-09T18:07:38.795 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'block_used': 13831463, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226493021, 'size_available': 927715414016, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90208, 'block_size': 4096, 'inode_available': 60964768}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226493021, "block_size": 4096, "block_total": 240324484, "block_used": 13831463, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964768, "inode_total": 61054976, "inode_used": 90208, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 927715414016, "size_total": 984369086464, "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:38.820 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:38 +0000 (0:00:00.108) 0:03:03.597 ********** 2021-04-09T18:07:38.882 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-09T18:07:38.882 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuR4lWKhEA4aE1DKJ84Q7CsfzVOtM30UpV'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.144312", "end": "2021-04-09 18:07:38.856307", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuR4lWKhEA4aE1DKJ84Q7CsfzVOtM30UpV"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:38.711995", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.141524 s, 815 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.141524 s, 815 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:38.895 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-04-09T18:07:38.919 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:38 +0000 (0:00:00.098) 0:03:03.696 ********** 2021-04-09T18:07:39.009 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:39.010 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2LL', 'wwn-0x5000c500871b79da'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2LL-part1', 'wwn-0x5000c500871b79da-part1'], 'uuids': ['7884d11b-1a98-41be-b51a-86bec1e24d7d']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b79da', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J2LL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2LL", "wwn-0x5000c500871b79da"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2LL-part1", "wwn-0x5000c500871b79da-part1"], "labels": [], "masters": [], "uuids": ["7884d11b-1a98-41be-b51a-86bec1e24d7d"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2LL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b79da"}}, "skip 2021-04-09T18:07:39.012 INFO:teuthology.task.ansible.out:_reason": "Conditional result was False"} 2021-04-09T18:07:39.192 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-09T18:07:39.193 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUusznML3vBwc6koi3JL3g4lRj8MbFVeHPq'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.146896", "end": "2021-04-09 18:07:39.162733", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUusznML3vBwc6koi3JL3g4lRj8MbFVeHPq"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:39.015837", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.144091 s, 800 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.144091 s, 800 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:39.512 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuxcMZMoNNFVsL8ibqzGdnRZoOzdDTTLQe'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHFT620400UL400BGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.146517", "end": "2021-04-09 18:07:39.480415", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EMbpmKv7phBUFD1tFvJXQ8YeMyiM9kUuxcMZMoNNFVsL8ibqzGdnRZoOzdDTTLQe"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400UL400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:39.333898", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.14364 s, 803 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.14364 s, 803 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:39.542 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:39 +0000 (0:00:00.623) 0:03:04.320 ********** 2021-04-09T18:07:39.844 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-04-09T18:07:39.846 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.068403", "end": "2021-04-09 18:07:39.762818", "rc": 0, "start": "2021-04-09 18:07:39.694415", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:39.866 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:39 +0000 (0:00:00.323) 0:03:04.643 ********** 2021-04-09T18:07:40.075 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-04-09T18:07:40.077 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.032550", "end": "2021-04-09 18:07:40.054037", "rc": 0, "start": "2021-04-09 18:07:40.021487", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:40.101 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.234) 0:03:04.878 ********** 2021-04-09T18:07:40.173 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-04-09T18:07:40.234 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.132) 0:03:05.011 ********** 2021-04-09T18:07:40.261 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-04-09T18:07:40.263 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.286 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.052) 0:03:05.064 ********** 2021-04-09T18:07:40.314 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-04-09T18:07:40.316 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.341 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.054) 0:03:05.118 ********** 2021-04-09T18:07:40.369 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-04-09T18:07:40.371 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.395 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.054) 0:03:05.173 ********** 2021-04-09T18:07:40.434 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-04-09T18:07:40.435 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.456 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.060) 0:03:05.233 ********** 2021-04-09T18:07:40.641 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-04-09T18:07:40.643 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-09T18:07:40.663 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.207) 0:03:05.441 ********** 2021-04-09T18:07:40.680 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:40.682 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-sSXTBf-1com-VL4q-25wl-0dwV-Zlk4-gNCCly', 'nvme-INTEL_SSDPED1D480GA_PHMB747402DN480DGN', 'nvme-nvme.8086-50484d42373437343032444e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPED1D480GA', 'serial': 'PHMB747402DN480DGN', 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.483975", "end": "2021-04-09 18:07:40.646167", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-sSXTBf-1com-VL4q-25wl-0dwV-Zlk4-gNCCly", "nvme-INTEL_SSDPED1D480GA_PHMB747402DN480DGN", "nvme-nvme.8086-50484d42373437343032444e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:39.162192", "stderr": "", "stderr_lines": [], "stdout": "Creating 2021-04-09T18:07:40.683 INFO:teuthology.task.ansible.out: new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:40.703 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-04-09T18:07:40.705 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'block_used': 13831461, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226493023, 'size_available': 927715422208, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90206, 'block_size': 4096, 'inode_available': 60964770}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226493023, "block_size": 4096, "block_total": 240324484, "block_used": 13831461, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964770, "inode_total": 61054976, "inode_used": 90206, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 927715422208, "size_total": 984369086464, "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.725 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.061) 0:03:05.503 ********** 2021-04-09T18:07:40.751 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-04-09T18:07:40.753 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.775 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.050) 0:03:05.553 ********** 2021-04-09T18:07:40.802 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:40.804 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.825 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.049) 0:03:05.602 ********** 2021-04-09T18:07:40.852 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:40.854 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.874 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.048) 0:03:05.651 ********** 2021-04-09T18:07:40.900 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-04-09T18:07:40.902 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.923 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.048) 0:03:05.700 ********** 2021-04-09T18:07:40.950 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-04-09T18:07:40.952 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:40.974 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:40 +0000 (0:00:00.050) 0:03:05.751 ********** 2021-04-09T18:07:41.000 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-04-09T18:07:41.002 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:41.023 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:41 +0000 (0:00:00.049) 0:03:05.801 ********** 2021-04-09T18:07:41.647 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-04-09T18:07:41.649 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-04-09T18:07:41.670 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:41 +0000 (0:00:00.646) 0:03:06.448 ********** 2021-04-09T18:07:41.931 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:41.932 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4O4n1r3vt3oCIpwX7PYjTsxjwC1GiZGBiC'], 'uuids': ['fc98dc74-5bca-4c68-a2bd-c7a6e4546e10']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.056368", "end": "2021-04-09 18:07:41.898374", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4O4n1r3vt3oCIpwX7PYjTsxjwC1GiZGBiC"], "labels": [], "masters": [], "uuids": ["fc98dc74-5bca-4c68-a2bd-c7a6e4546e10"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:40.842006", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:42.268 INFO:teuthology.task.ansible.out: 2021-04-09T18:07:42.269 INFO:teuthology.task.ansible.out:TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:42.531 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:42.805 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:43.077 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:43.174 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:43.176 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4OM5F1OARmxLegQI3BWeI8Rz3Jk1JCdyJk'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.047439", "end": "2021-04-09 18:07:43.148401", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4OM5F1OARmxLegQI3BWeI8Rz3Jk1JCdyJk"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:42.100962", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:43.343 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-04-09T18:07:43.345 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:43.370 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:43 +0000 (0:00:01.699) 0:03:08.147 ********** 2021-04-09T18:07:43.556 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-04-09T18:07:43.558 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-04-09T18:07:43.583 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:43 +0000 (0:00:00.213) 0:03:08.361 ********** 2021-04-09T18:07:43.779 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-04-09T18:07:43.781 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-09T18:07:43.938 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-09T18:07:44.108 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-09T18:07:44.280 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-09T18:07:44.293 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:44.322 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:44 +0000 (0:00:00.739) 0:03:09.100 ********** 2021-04-09T18:07:44.349 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-04-09T18:07:44.351 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:44.376 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:44 +0000 (0:00:00.053) 0:03:09.153 ********** 2021-04-09T18:07:44.404 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-04-09T18:07:44.406 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:44.435 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:44 +0000 (0:00:00.059) 0:03:09.213 ********** 2021-04-09T18:07:44.442 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:44.442 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4OmI5B0xj9DxEMI0VRdSe707IbpOjREbHK'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.053624", "end": "2021-04-09 18:07:44.390606", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4OmI5B0xj9DxEMI0VRdSe707IbpOjREbHK"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:43.336982", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:44.622 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-04-09T18:07:44.625 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-04-09T18:07:44.645 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:44 +0000 (0:00:00.210) 0:03:09.423 ********** 2021-04-09T18:07:44.673 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:44.697 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:44 +0000 (0:00:00.051) 0:03:09.474 ********** 2021-04-09T18:07:45.260 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-04-09T18:07:45.262 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true} 2021-04-09T18:07:45.287 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:45 +0000 (0:00:00.589) 0:03:10.064 ********** 2021-04-09T18:07:45.654 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:45.656 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4O2KrQhsbFmV5xG0oknzonWOCM0NfQlCpo'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.047501", "end": "2021-04-09 18:07:45.631789", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4O2KrQhsbFmV5xG0oknzonWOCM0NfQlCpo"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:44.584288", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:46.828 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-04-09T18:07:46.830 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-04-09T18:07:46.856 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:46 +0000 (0:00:01.569) 0:03:11.634 ********** 2021-04-09T18:07:46.883 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-04-09T18:07:46.884 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:46.908 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:46 +0000 (0:00:00.052) 0:03:11.686 ********** 2021-04-09T18:07:46.916 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-09T18:07:46.917 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4ONYzNJVZkgqcnv4pXvpQyqYCnjUfMot3I'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.050261", "end": "2021-04-09 18:07:46.874491", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4ONYzNJVZkgqcnv4pXvpQyqYCnjUfMot3I"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:45.824230", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-09T18:07:46.952 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:46 +0000 (0:00:00.043) 0:03:11.729 ********** 2021-04-09T18:07:46.959 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-04-09T18:07:46.962 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:46.988 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:46 +0000 (0:00:00.036) 0:03:11.766 ********** 2021-04-09T18:07:46.995 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-09T18:07:46.998 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2LL', 'wwn-0x5000c500871b79da'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2LL-part1', 'wwn-0x5000c500871b79da-part1'], 'uuids': ['7884d11b-1a98-41be-b51a-86bec1e24d7d']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b79da', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J2LL', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2LL", "wwn-0x5000c500871b79da"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2LL-part1", "wwn-0x5000c500871b79da-part1"], "labels": [], "masters": [], "uuids": ["7884d11b-1a98-41be-b51a-86bec1e24d7d"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2LL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b79da"}}, "skip 2021-04-09T18:07:46.999 INFO:teuthology.task.ansible.out:_reason": "Conditional result was False"} 2021-04-09T18:07:47.019 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:47.042 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:47 +0000 (0:00:00.053) 0:03:11.820 ********** 2021-04-09T18:07:47.070 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-04-09T18:07:47.072 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:47.092 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:47 +0000 (0:00:00.050) 0:03:11.870 ********** 2021-04-09T18:07:47.120 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:47.141 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:47 +0000 (0:00:00.048) 0:03:11.919 ********** 2021-04-09T18:07:47.167 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-04-09T18:07:47.169 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:47.190 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:47 +0000 (0:00:00.048) 0:03:11.967 ********** 2021-04-09T18:07:47.216 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-04-09T18:07:47.218 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:47.249 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:47 +0000 (0:00:00.059) 0:03:12.026 ********** 2021-04-09T18:07:47.259 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-09T18:07:47.260 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-sSXTBf-1com-VL4q-25wl-0dwV-Zlk4-gNCCly', 'nvme-INTEL_SSDPED1D480GA_PHMB747402DN480DGN', 'nvme-nvme.8086-50484d42373437343032444e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPED1D480GA', 'serial': 'PHMB747402DN480DGN', 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.038338", "end": "2021-04-09 18:07:47.211622", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-sSXTBf-1com-VL4q-25wl-0dwV-Zlk4-gNCCly", "nvme-INTEL_SSDPED1D480GA_PHMB747402DN480DGN", "nvme-nvme.8086-50484d42373437343032444e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:47.173284", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 2021-04-09T18:07:47.262 INFO:teuthology.task.ansible.out: MB) copied, 0.0350447 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0350447 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:47.273 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-04-09T18:07:47.275 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:47.298 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:47 +0000 (0:00:00.048) 0:03:12.075 ********** 2021-04-09T18:07:47.327 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-04-09T18:07:47.329 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:47.350 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:47 +0000 (0:00:00.051) 0:03:12.127 ********** 2021-04-09T18:07:47.376 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-04-09T18:07:47.378 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:47.400 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:47 +0000 (0:00:00.050) 0:03:12.178 ********** 2021-04-09T18:07:47.439 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-04-09T18:07:47.441 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:47.461 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:47 +0000 (0:00:00.060) 0:03:12.239 ********** 2021-04-09T18:07:47.510 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-09T18:07:47.512 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4O4n1r3vt3oCIpwX7PYjTsxjwC1GiZGBiC'], 'uuids': ['fc98dc74-5bca-4c68-a2bd-c7a6e4546e10']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.098155", "end": "2021-04-09 18:07:47.486031", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4O4n1r3vt3oCIpwX7PYjTsxjwC1GiZGBiC"], "labels": [], "masters": [], "uuids": ["fc98dc74-5bca-4c68-a2bd-c7a6e4546e10"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:47.387876", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0950781 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0950781 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:47.787 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4OM5F1OARmxLegQI3BWeI8Rz3Jk1JCdyJk'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.099242", "end": "2021-04-09 18:07:47.759009", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4OM5F1OARmxLegQI3BWeI8Rz3Jk1JCdyJk"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:47.659767", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0961084 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0961084 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:48.059 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4OmI5B0xj9DxEMI0VRdSe707IbpOjREbHK'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.098859", "end": "2021-04-09 18:07:48.036387", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4OmI5B0xj9DxEMI0VRdSe707IbpOjREbHK"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:47.937528", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0956715 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0956715 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:48.111 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-04-09T18:07:48.113 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7_8.2.x86_64 providing ntp is already installed"]} 2021-04-09T18:07:48.135 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:48 +0000 (0:00:00.673) 0:03:12.912 ********** 2021-04-09T18:07:48.376 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-04-09T18:07:48.378 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:48.397 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:48 +0000 (0:00:00.262) 0:03:13.175 ********** 2021-04-09T18:07:48.403 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-09T18:07:48.405 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4O2KrQhsbFmV5xG0oknzonWOCM0NfQlCpo'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.099021", "end": "2021-04-09 18:07:48.307148", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4O2KrQhsbFmV5xG0oknzonWOCM0NfQlCpo"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:48.208127", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0958827 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0958827 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:48.427 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-04-09T18:07:48.429 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:48.454 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:48 +0000 (0:00:00.056) 0:03:13.231 ********** 2021-04-09T18:07:48.609 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-09T18:07:48.611 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4ONYzNJVZkgqcnv4pXvpQyqYCnjUfMot3I'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHMB747402DN480DGN', 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.097406", "end": "2021-04-09 18:07:48.586371", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-MmpECBBvGasW9yUPtRtf4RUMYiOFSe4ONYzNJVZkgqcnv4pXvpQyqYCnjUfMot3I"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB747402DN480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-09 18:07:48.488965", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0943318 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0943318 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:48.654 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:48 +0000 (0:00:00.199) 0:03:13.431 ********** 2021-04-09T18:07:48.918 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-04-09T18:07:48.920 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.071788", "end": "2021-04-09 18:07:48.890730", "rc": 0, "start": "2021-04-09 18:07:48.818942", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:48.949 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:48 +0000 (0:00:00.295) 0:03:13.727 ********** 2021-04-09T18:07:49.001 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-04-09T18:07:49.003 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2021-04-09T18:07:49.035 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.085) 0:03:13.812 ********** 2021-04-09T18:07:49.065 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:49.087 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.052) 0:03:13.865 ********** 2021-04-09T18:07:49.179 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-04-09T18:07:49.181 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.032530", "end": "2021-04-09 18:07:49.159653", "rc": 0, "start": "2021-04-09 18:07:49.127123", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:49.204 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.116) 0:03:13.982 ********** 2021-04-09T18:07:49.231 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-04-09T18:07:49.267 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.062) 0:03:14.044 ********** 2021-04-09T18:07:49.307 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-04-09T18:07:49.309 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:49.332 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.065) 0:03:14.110 ********** 2021-04-09T18:07:49.360 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-04-09T18:07:49.362 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:49.385 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.053) 0:03:14.163 ********** 2021-04-09T18:07:49.415 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-04-09T18:07:49.417 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure ntpd is running.] *********************************** 2021-04-09T18:07:49.419 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "tmp.mount ntpdate.service syslog.target systemd-journald.socket basic.target sntp.service system.slice -.mount", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target chronyd.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "Lim 2021-04-09T18:07:49.421 INFO:teuthology.task.ansible.out:itAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice -.mount basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "Timeo 2021-04-09T18:07:49.423 INFO:teuthology.task.ansible.out:utStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-04-09T18:07:49.457 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.071) 0:03:14.235 ********** 2021-04-09T18:07:49.492 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.034) 0:03:14.269 ********** 2021-04-09T18:07:49.501 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-04-09T18:07:49.503 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:49.535 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.043) 0:03:14.313 ********** 2021-04-09T18:07:49.546 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:49.548 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-04-09T18:07:49.577 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.041) 0:03:14.354 ********** 2021-04-09T18:07:49.628 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:49.630 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:49.652 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.074) 0:03:14.429 ********** 2021-04-09T18:07:49.742 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-04-09T18:07:49.745 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-09T18:07:49.772 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.119) 0:03:14.549 ********** 2021-04-09T18:07:49.821 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-04-09T18:07:49.823 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'block_used': 13831463, 'uuid': '7884d11b-1a98-41be-b51a-86bec1e24d7d', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226493021, 'size_available': 927715414016, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90208, 'block_size': 4096, 'inode_available': 60964768}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226493021, "block_size": 4096, "block_total": 240324484, "block_used": 13831463, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964768, "inode_total": 61054976, "inode_used": 90208, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 927715414016, "size_total": 984369086464, "uuid": "7884d11b-1a98-41be-b51a-86bec1e24d7d"}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:49.846 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.074) 0:03:14.623 ********** 2021-04-09T18:07:49.876 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-04-09T18:07:49.878 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:49.899 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.052) 0:03:14.676 ********** 2021-04-09T18:07:49.937 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:49.939 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:49.970 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:49 +0000 (0:00:00.071) 0:03:14.747 ********** 2021-04-09T18:07:50.010 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:50.012 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:50.037 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:50 +0000 (0:00:00.067) 0:03:14.815 ********** 2021-04-09T18:07:50.089 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-04-09T18:07:50.091 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:50.113 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:50 +0000 (0:00:00.075) 0:03:14.890 ********** 2021-04-09T18:07:50.150 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-04-09T18:07:50.152 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:50.181 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:50 +0000 (0:00:00.068) 0:03:14.958 ********** 2021-04-09T18:07:50.208 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-04-09T18:07:50.210 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:50.233 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:50 +0000 (0:00:00.052) 0:03:15.010 ********** 2021-04-09T18:07:50.591 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-04-09T18:07:50.592 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-04-09T18:07:50.625 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:50 +0000 (0:00:00.391) 0:03:15.402 ********** 2021-04-09T18:07:50.773 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-14.el7.noarch providing python-pip is already installed"]} 2021-04-09T18:07:50.804 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:50 +0000 (0:00:00.179) 0:03:15.582 ********** 2021-04-09T18:07:50.835 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:50.865 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:50 +0000 (0:00:00.060) 0:03:15.643 ********** 2021-04-09T18:07:50.892 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-04-09T18:07:50.894 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:50.923 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:50 +0000 (0:00:00.058) 0:03:15.701 ********** 2021-04-09T18:07:50.973 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-04-09T18:07:50.975 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:51.113 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-04-09T18:07:51.115 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-04-09T18:07:51.135 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:51 +0000 (0:00:00.211) 0:03:15.912 ********** 2021-04-09T18:07:51.372 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-04-09T18:07:51.374 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:51.646 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-04-09T18:07:51.648 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-04-09T18:07:51.669 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:51 +0000 (0:00:00.534) 0:03:16.447 ********** 2021-04-09T18:07:51.677 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-04-09T18:07:51.679 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:51.951 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:52.226 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-09T18:07:52.262 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:52 +0000 (0:00:00.592) 0:03:17.039 ********** 2021-04-09T18:07:52.460 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-04-09T18:07:52.462 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-04-09T18:07:52.494 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:52 +0000 (0:00:00.232) 0:03:17.272 ********** 2021-04-09T18:07:52.710 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-04-09T18:07:52.712 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-09T18:07:52.743 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-04-09T18:07:52.744 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-04-09T18:07:52.767 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:52 +0000 (0:00:00.272) 0:03:17.544 ********** 2021-04-09T18:07:52.797 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:52.823 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:52 +0000 (0:00:00.055) 0:03:17.600 ********** 2021-04-09T18:07:52.860 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-04-09T18:07:52.862 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-04-09T18:07:52.890 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:52 +0000 (0:00:00.067) 0:03:17.668 ********** 2021-04-09T18:07:52.902 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-04-09T18:07:52.904 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-09T18:07:53.064 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-04-09T18:07:53.065 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-04-09T18:07:53.089 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.198) 0:03:17.867 ********** 2021-04-09T18:07:53.097 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-04-09T18:07:53.097 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-09T18:07:53.121 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-04-09T18:07:53.122 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.144 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.054) 0:03:17.921 ********** 2021-04-09T18:07:53.171 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-04-09T18:07:53.172 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-04-09T18:07:53.201 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.057) 0:03:17.978 ********** 2021-04-09T18:07:53.229 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-04-09T18:07:53.230 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.260 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.058) 0:03:18.037 ********** 2021-04-09T18:07:53.269 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-04-09T18:07:53.270 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-09T18:07:53.272 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.312 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.052) 0:03:18.089 ********** 2021-04-09T18:07:53.339 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-04-09T18:07:53.341 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.361 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.049) 0:03:18.139 ********** 2021-04-09T18:07:53.388 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-04-09T18:07:53.390 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.412 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.051) 0:03:18.190 ********** 2021-04-09T18:07:53.440 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-04-09T18:07:53.442 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-04-09T18:07:53.464 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.051) 0:03:18.242 ********** 2021-04-09T18:07:53.613 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-04-09T18:07:53.644 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.180) 0:03:18.422 ********** 2021-04-09T18:07:53.662 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-04-09T18:07:53.665 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-04-09T18:07:53.691 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.046) 0:03:18.468 ********** 2021-04-09T18:07:53.698 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-04-09T18:07:53.700 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.725 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.034) 0:03:18.503 ********** 2021-04-09T18:07:53.736 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-04-09T18:07:53.737 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.763 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.037) 0:03:18.540 ********** 2021-04-09T18:07:53.812 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-04-09T18:07:53.813 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.815 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.829 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.838 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.840 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.862 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.098) 0:03:18.639 ********** 2021-04-09T18:07:53.900 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-04-09T18:07:53.903 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.913 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.915 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.926 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.928 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:53.950 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.088) 0:03:18.727 ********** 2021-04-09T18:07:53.966 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-04-09T18:07:53.968 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true} 2021-04-09T18:07:53.990 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:53 +0000 (0:00:00.040) 0:03:18.768 ********** 2021-04-09T18:07:53.997 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:53.999 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.029 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.033) 0:03:18.802 ********** 2021-04-09T18:07:54.051 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-04-09T18:07:54.053 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.076 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.051) 0:03:18.853 ********** 2021-04-09T18:07:54.114 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-04-09T18:07:54.116 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.152 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.075) 0:03:18.929 ********** 2021-04-09T18:07:54.182 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-04-09T18:07:54.184 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.218 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.066) 0:03:18.995 ********** 2021-04-09T18:07:54.247 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-04-09T18:07:54.249 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.277 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.059) 0:03:19.055 ********** 2021-04-09T18:07:54.284 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-04-09T18:07:54.286 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-04-09T18:07:54.312 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.034) 0:03:19.089 ********** 2021-04-09T18:07:54.320 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-04-09T18:07:54.322 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.349 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.036) 0:03:19.126 ********** 2021-04-09T18:07:54.356 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.383 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.034) 0:03:19.160 ********** 2021-04-09T18:07:54.392 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-04-09T18:07:54.394 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.425 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.041) 0:03:19.202 ********** 2021-04-09T18:07:54.431 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-04-09T18:07:54.433 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.459 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.034) 0:03:19.236 ********** 2021-04-09T18:07:54.466 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-04-09T18:07:54.468 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.497 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.038) 0:03:19.275 ********** 2021-04-09T18:07:54.504 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-04-09T18:07:54.506 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.529 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.031) 0:03:19.307 ********** 2021-04-09T18:07:54.568 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-04-09T18:07:54.570 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.594 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.064) 0:03:19.371 ********** 2021-04-09T18:07:54.621 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-04-09T18:07:54.622 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.652 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.057) 0:03:19.429 ********** 2021-04-09T18:07:54.680 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-04-09T18:07:54.681 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.715 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.062) 0:03:19.492 ********** 2021-04-09T18:07:54.724 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-04-09T18:07:54.725 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.006631", "end": "2021-04-09 18:07:54.677138", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-04-09 18:07:54.670507", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:54.747 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-04-09T18:07:54.749 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.771 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.056) 0:03:19.548 ********** 2021-04-09T18:07:54.810 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-04-09T18:07:54.812 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.837 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.066) 0:03:19.615 ********** 2021-04-09T18:07:54.865 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-04-09T18:07:54.866 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.891 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.053) 0:03:19.669 ********** 2021-04-09T18:07:54.900 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-04-09T18:07:54.902 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003316", "end": "2021-04-09 18:07:54.854033", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-04-09 18:07:54.850717", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:54.922 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-04-09T18:07:54.924 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:54.950 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:54 +0000 (0:00:00.059) 0:03:19.728 ********** 2021-04-09T18:07:54.978 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-04-09T18:07:54.980 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:55.012 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:55 +0000 (0:00:00.061) 0:03:19.789 ********** 2021-04-09T18:07:55.054 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi086.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003090", "end": "2021-04-09 18:07:55.018566", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-04-09 18:07:55.015476", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:55.235 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.019646", "end": "2021-04-09 18:07:55.204543", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-04-09 18:07:55.184897", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:07:55.263 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:55 +0000 (0:00:00.251) 0:03:20.041 ********** 2021-04-09T18:07:55.290 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-04-09T18:07:55.319 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:55 +0000 (0:00:00.055) 0:03:20.097 ********** 2021-04-09T18:07:55.517 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-04-09T18:07:55.519 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-04-09T18:07:55.715 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-04-09T18:07:55.716 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7_8.2.x86_64 providing ntp is already installed"]} 2021-04-09T18:07:55.736 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:55 +0000 (0:00:00.416) 0:03:20.513 ********** 2021-04-09T18:07:55.764 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-04-09T18:07:55.766 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:55.795 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:55 +0000 (0:00:00.058) 0:03:20.572 ********** 2021-04-09T18:07:55.822 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-04-09T18:07:55.823 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:55.844 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:55 +0000 (0:00:00.049) 0:03:20.622 ********** 2021-04-09T18:07:56.383 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-04-09T18:07:56.384 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2021-04-09T18:07:56.404 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:56 +0000 (0:00:00.560) 0:03:21.182 ********** 2021-04-09T18:07:56.430 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-04-09T18:07:56.432 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:56.452 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:56 +0000 (0:00:00.047) 0:03:21.229 ********** 2021-04-09T18:07:56.772 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-04-09T18:07:56.773 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "tmp.mount sntp.service system.slice systemd-journald.socket syslog.target basic.target -.mount ntpdate.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "multi-user.target shutdown.target chronyd.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "Lim 2021-04-09T18:07:56.775 INFO:teuthology.task.ansible.out:itAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127535", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127535", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "Timeo 2021-04-09T18:07:56.777 INFO:teuthology.task.ansible.out:utStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-04-09T18:07:56.795 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:56 +0000 (0:00:00.342) 0:03:21.572 ********** 2021-04-09T18:07:56.821 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:56.823 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-04-09T18:07:56.843 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:56 +0000 (0:00:00.048) 0:03:21.621 ********** 2021-04-09T18:07:56.882 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:07:56.884 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:56.906 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:56 +0000 (0:00:00.062) 0:03:21.683 ********** 2021-04-09T18:07:58.078 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-04-09T18:07:58.080 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-14.el7.noarch providing python-pip is already installed"]} 2021-04-09T18:07:58.106 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:58 +0000 (0:00:01.199) 0:03:22.883 ********** 2021-04-09T18:07:58.133 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-04-09T18:07:58.135 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:58.158 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:58 +0000 (0:00:00.052) 0:03:22.935 ********** 2021-04-09T18:07:58.185 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-04-09T18:07:58.187 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:07:58.216 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:58 +0000 (0:00:00.058) 0:03:22.994 ********** 2021-04-09T18:07:58.413 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-04-09T18:07:58.415 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-04-09T18:07:58.437 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:58 +0000 (0:00:00.220) 0:03:23.215 ********** 2021-04-09T18:07:58.990 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-04-09T18:07:58.992 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-04-09T18:07:59.013 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:07:59 +0000 (0:00:00.575) 0:03:23.790 ********** 2021-04-09T18:07:59.993 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-04-09T18:07:59.995 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-04-09T18:08:00.024 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:00 +0000 (0:00:01.011) 0:03:24.801 ********** 2021-04-09T18:08:00.052 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-04-09T18:08:00.054 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:00.075 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:00 +0000 (0:00:00.051) 0:03:24.852 ********** 2021-04-09T18:08:00.102 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-04-09T18:08:00.104 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-04-09T18:08:00.129 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:00 +0000 (0:00:00.053) 0:03:24.906 ********** 2021-04-09T18:08:00.317 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-04-09T18:08:00.319 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-04-09T18:08:00.343 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:00 +0000 (0:00:00.213) 0:03:25.120 ********** 2021-04-09T18:08:00.372 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-04-09T18:08:00.374 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:00.399 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:00 +0000 (0:00:00.056) 0:03:25.177 ********** 2021-04-09T18:08:00.429 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-04-09T18:08:00.431 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-04-09T18:08:00.455 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:00 +0000 (0:00:00.055) 0:03:25.233 ********** 2021-04-09T18:08:00.494 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-04-09T18:08:00.495 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:00.517 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:00 +0000 (0:00:00.062) 0:03:25.295 ********** 2021-04-09T18:08:00.721 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-04-09T18:08:00.723 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-04-09T18:08:00.744 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:00 +0000 (0:00:00.226) 0:03:25.522 ********** 2021-04-09T18:08:00.938 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-04-09T18:08:00.941 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-04-09T18:08:00.960 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:00 +0000 (0:00:00.216) 0:03:25.738 ********** 2021-04-09T18:08:00.989 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-04-09T18:08:00.990 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.013 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.052) 0:03:25.790 ********** 2021-04-09T18:08:01.051 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-04-09T18:08:01.053 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.063 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.074 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.076 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.102 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.089) 0:03:25.880 ********** 2021-04-09T18:08:01.143 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-04-09T18:08:01.145 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi032.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.156 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.157 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.159 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.197 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.094) 0:03:25.975 ********** 2021-04-09T18:08:01.225 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-09T18:08:01.227 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.252 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.054) 0:03:26.029 ********** 2021-04-09T18:08:01.279 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-04-09T18:08:01.280 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.304 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.052) 0:03:26.082 ********** 2021-04-09T18:08:01.332 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-04-09T18:08:01.334 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.356 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.052) 0:03:26.134 ********** 2021-04-09T18:08:01.384 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-04-09T18:08:01.386 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.411 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.054) 0:03:26.188 ********** 2021-04-09T18:08:01.439 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-04-09T18:08:01.441 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.465 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.054) 0:03:26.243 ********** 2021-04-09T18:08:01.492 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-04-09T18:08:01.494 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.524 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.058) 0:03:26.301 ********** 2021-04-09T18:08:01.564 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-04-09T18:08:01.566 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.594 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.070) 0:03:26.372 ********** 2021-04-09T18:08:01.622 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-04-09T18:08:01.624 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:01.648 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:01 +0000 (0:00:00.053) 0:03:26.425 ********** 2021-04-09T18:08:01.864 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-04-09T18:08:01.866 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.007272", "end": "2021-04-09 18:08:01.838057", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-04-09 18:08:01.830785", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:08:02.051 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003642", "end": "2021-04-09 18:08:02.016618", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-04-09 18:08:02.012976", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:08:02.230 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003483", "end": "2021-04-09 18:08:02.202695", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-04-09 18:08:02.199212", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-04-09T18:08:02.412 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.005321", "end": "2021-04-09 18:08:02.386463", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-04-09 18:08:02.381142", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:08:02.437 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:02 +0000 (0:00:00.789) 0:03:27.215 ********** 2021-04-09T18:08:02.467 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-04-09T18:08:02.490 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:02 +0000 (0:00:00.053) 0:03:27.268 ********** 2021-04-09T18:08:02.696 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-04-09T18:08:02.698 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-04-09T18:08:02.785 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-04-09T18:08:02.815 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:02 +0000 (0:00:00.324) 0:03:27.592 ********** 2021-04-09T18:08:02.842 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:02 +0000 (0:00:00.026) 0:03:27.619 ********** 2021-04-09T18:08:03.419 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi032.front.sepia.ceph.com] 2021-04-09T18:08:03.448 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] 2021-04-09T18:08:03.473 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:03 +0000 (0:00:00.631) 0:03:28.250 ********** 2021-04-09T18:08:03.515 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:03 +0000 (0:00:00.041) 0:03:28.292 ********** 2021-04-09T18:08:03.523 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:03.551 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:03 +0000 (0:00:00.035) 0:03:28.328 ********** 2021-04-09T18:08:03.558 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:03.583 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:03 +0000 (0:00:00.032) 0:03:28.361 ********** 2021-04-09T18:08:03.590 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-04-09T18:08:03.591 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-04-09T18:08:03.619 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:03 +0000 (0:00:00.035) 0:03:28.396 ********** 2021-04-09T18:08:03.628 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-04-09T18:08:03.661 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:03 +0000 (0:00:00.042) 0:03:28.439 ********** 2021-04-09T18:08:03.669 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-04-09T18:08:03.669 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-04-09T18:08:03.695 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:03 +0000 (0:00:00.033) 0:03:28.473 ********** 2021-04-09T18:08:03.705 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-04-09T18:08:03.732 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:03 +0000 (0:00:00.036) 0:03:28.509 ********** 2021-04-09T18:08:09.571 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-04-09T18:08:09.573 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing podman are up to date", "All packages providing podman-docker are up to date", ""]} 2021-04-09T18:08:09.604 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:09 +0000 (0:00:05.871) 0:03:34.381 ********** 2021-04-09T18:08:09.641 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-04-09T18:08:09.643 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-04-09T18:08:09.664 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:09 +0000 (0:00:00.060) 0:03:34.441 ********** 2021-04-09T18:08:09.702 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-04-09T18:08:09.704 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:09.726 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:09 +0000 (0:00:00.062) 0:03:34.503 ********** 2021-04-09T18:08:09.955 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-04-09T18:08:09.957 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:cert_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-04-09T18:08:09.988 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-04-09T18:08:09.990 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing podman are up to date", "All packages providing podman-docker are up to date", ""]} 2021-04-09T18:08:10.011 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:10 +0000 (0:00:00.285) 0:03:34.788 ********** 2021-04-09T18:08:10.049 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-04-09T18:08:10.051 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-04-09T18:08:10.072 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:10 +0000 (0:00:00.060) 0:03:34.849 ********** 2021-04-09T18:08:10.100 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-04-09T18:08:10.102 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:10.127 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:10 +0000 (0:00:00.054) 0:03:34.904 ********** 2021-04-09T18:08:10.178 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-04-09T18:08:10.380 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} Friday 09 April 2021 18:08:10 +0000 (0:00:00.092) 0:03:34.996 ********** 2021-04-09T18:08:10.409 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:cert_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-04-09T18:08:10.612 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-04-09T18:08:10.637 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:10 +0000 (0:00:00.417) 0:03:35.414 ********** 2021-04-09T18:08:10.813 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-04-09T18:08:10.815 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-04-09T18:08:11.209 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-04-09T18:08:11.294 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-04-09T18:08:11.319 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:11 +0000 (0:00:00.682) 0:03:36.096 ********** 2021-04-09T18:08:11.726 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-04-09T18:08:11.751 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:11 +0000 (0:00:00.431) 0:03:36.528 ********** 2021-04-09T18:08:13.612 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-04-09T18:08:13.613 INFO:teuthology.task.ansible.out:ok: [smithi086.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-d1avg3-build\nRequirement already up-to-date: toml in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: typing in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-d1avg3-build", "Requirement already up-to-date: toml in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: typing in /usr/lib/python2.7/site-packages (from registries- 2021-04-09T18:08:13.615 INFO:teuthology.task.ansible.out:conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-04-09T18:08:13.638 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:13 +0000 (0:00:01.887) 0:03:38.416 ********** 2021-04-09T18:08:13.666 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-04-09T18:08:13.668 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:13.689 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:13 +0000 (0:00:00.051) 0:03:38.467 ********** 2021-04-09T18:08:13.726 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-04-09T18:08:13.727 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:13.748 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:13 +0000 (0:00:00.058) 0:03:38.526 ********** 2021-04-09T18:08:14.036 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-04-09T18:08:14.038 INFO:teuthology.task.ansible.out:changed: [smithi086.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.104107", "end": "2021-04-09 18:08:14.012103", "rc": 0, "start": "2021-04-09 18:08:13.907996", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:08:14.061 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:14 +0000 (0:00:00.312) 0:03:38.838 ********** 2021-04-09T18:08:14.068 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-04-09T18:08:14.069 INFO:teuthology.task.ansible.out:ok: [smithi032.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-N3mJpI-build\nRequirement already up-to-date: toml in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: typing in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-N3mJpI-build", "Requirement already up-to-date: toml in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/lib/python2.7/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: typing in /usr/lib/python2.7/site-packages (from registries- 2021-04-09T18:08:14.071 INFO:teuthology.task.ansible.out:conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-04-09T18:08:14.095 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:14 +0000 (0:00:00.034) 0:03:38.873 ********** 2021-04-09T18:08:14.103 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-04-09T18:08:14.105 INFO:teuthology.task.ansible.out:skipping: [smithi086.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:14.153 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-04-09T18:08:14.155 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:14.179 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:14 +0000 (0:00:00.083) 0:03:38.956 ********** 2021-04-09T18:08:14.208 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-04-09T18:08:14.210 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:14.235 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:14 +0000 (0:00:00.055) 0:03:39.012 ********** 2021-04-09T18:08:14.527 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-04-09T18:08:14.529 INFO:teuthology.task.ansible.out:changed: [smithi032.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.098737", "end": "2021-04-09 18:08:14.508585", "rc": 0, "start": "2021-04-09 18:08:14.409848", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-09T18:08:14.553 INFO:teuthology.task.ansible.out:Friday 09 April 2021 18:08:14 +0000 (0:00:00.318) 0:03:39.330 ********** 2021-04-09T18:08:14.592 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-04-09T18:08:14.594 INFO:teuthology.task.ansible.out:skipping: [smithi032.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-09T18:08:14.660 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-04-09T18:08:14.661 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2021-04-09T18:08:14.663 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2021-04-09T18:08:14.665 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* 2021-04-09T18:08:14.665 INFO:teuthology.task.ansible.out:smithi032.front.sepia.ceph.com : ok=136 changed=35 unreachable=0 failed=0 skipped=176 rescued=0 ignored=4 smithi086.front.sepia.ceph.com : ok=136 changed=35 unreachable=0 failed=0 skipped=176 rescued=0 ignored=4 Friday 09 April 2021 18:08:14 +0000 (0:00:00.112) 0:03:39.443 ********** =============================================================================== 2021-04-09T18:08:14.669 INFO:teuthology.task.ansible.out:common : Make sure smartmontools is installed -------------------------- 32.86s testnode : Install packages -------------------------------------------- 12.54s 2021-04-09T18:08:14.671 INFO:teuthology.task.ansible.out:common : Register with subscription-manager. ---------------------------- 6.13s container-host : Install container packages ----------------------------- 5.87s testnode : Ensure ceph dependency packages are not present. ------------- 5.59s testnode : Ensure ceph packages are not present. ------------------------ 3.70s testnode : Rebuild rpmdb ------------------------------------------------ 2.21s container-host : Install registries-conf-ctl ---------------------------- 1.89s common : nrpe - Remove SELinux policy package --------------------------- 1.70s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.61s common : Install CA Cert from Satellite Server -------------------------- 1.58s Gathering Facts --------------------------------------------------------- 1.57s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.54s common : Enable necessary rhsm repos ------------------------------------ 1.23s testnode : Install python-pip on rpm based systems. --------------------- 1.20s testnode : Configure common additional repos in /etc/yum.repos.d/ ------- 1.12s testnode : Removing saved yum transactions ------------------------------ 1.10s common : nrpe - Ensure SELinux policy is up to date --------------------- 1.02s testnode : Install packages via pip ------------------------------------- 1.01s common : Disable all rhsm repos ----------------------------------------- 0.99s 2021-04-09T18:08:15.001 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2021-04-09T18:08:15.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-09T18:08:15.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-09T18:08:15.115 DEBUG:teuthology.orchestra.run.smithi032:> true 2021-04-09T18:08:15.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-09T18:08:15.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-09T18:08:15.498 DEBUG:teuthology.orchestra.run.smithi086:> true 2021-04-09T18:08:15.765 INFO:teuthology.run_tasks:Running task clock... 2021-04-09T18:08:15.781 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-04-09T18:08:15.782 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-04-09T18:08:15.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-04-09T18:08:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-04-09T18:08:22.919 INFO:teuthology.orchestra.run.smithi032.stdout:ntpd: time slew +0.001338s 2021-04-09T18:08:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit not found. 2021-04-09T18:08:22.999 INFO:teuthology.orchestra.run.smithi032.stdout: remote refid st t when poll reach delay offset jitter 2021-04-09T18:08:23.000 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================== 2021-04-09T18:08:23.000 INFO:teuthology.orchestra.run.smithi032.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-09T18:08:23.000 INFO:teuthology.orchestra.run.smithi032.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-09T18:08:23.001 INFO:teuthology.orchestra.run.smithi032.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-09T18:08:23.001 INFO:teuthology.orchestra.run.smithi032.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-09T18:08:23.001 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-04-09T18:08:23.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-04-09T18:08:23.072 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-04-09T18:08:30.126 INFO:teuthology.orchestra.run.smithi086.stdout:ntpd: time slew +0.003450s 2021-04-09T18:08:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit not found. 2021-04-09T18:08:30.205 INFO:teuthology.orchestra.run.smithi086.stdout: remote refid st t when poll reach delay offset jitter 2021-04-09T18:08:30.205 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================== 2021-04-09T18:08:30.206 INFO:teuthology.orchestra.run.smithi086.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-09T18:08:30.206 INFO:teuthology.orchestra.run.smithi086.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-09T18:08:30.206 INFO:teuthology.orchestra.run.smithi086.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-09T18:08:30.206 INFO:teuthology.orchestra.run.smithi086.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-09T18:08:30.207 INFO:teuthology.run_tasks:Running task install... 2021-04-09T18:08:30.216 DEBUG:teuthology.task.install:project ceph 2021-04-09T18:08:30.217 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b'}} 2021-04-09T18:08:30.217 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b'} 2021-04-09T18:08:30.217 INFO:teuthology.task.install:Using flavor: basic 2021-04-09T18:08:30.222 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2021-04-09T18:08:30.222 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2021-04-09T18:08:30.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo lsb_release -is 2021-04-09T18:08:30.316 INFO:teuthology.orchestra.run.smithi032.stdout:RedHatEnterpriseServer 2021-04-09T18:08:30.317 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-04-09T18:08:30.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo lsb_release -is 2021-04-09T18:08:30.319 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b', 'tag': None, 'wait_for_package': False} 2021-04-09T18:08:30.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo lsb_release -is 2021-04-09T18:08:30.394 INFO:teuthology.orchestra.run.smithi032.stdout:RedHatEnterpriseServer 2021-04-09T18:08:30.395 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-04-09T18:08:30.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=d23cd2d5f486487b0b80655c4e33344f0f97b82b 2021-04-09T18:08:30.401 INFO:teuthology.orchestra.run.smithi086.stdout:RedHatEnterpriseServer 2021-04-09T18:08:30.401 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-04-09T18:08:30.402 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b', 'tag': None, 'wait_for_package': False} 2021-04-09T18:08:30.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo lsb_release -is 2021-04-09T18:08:30.470 INFO:teuthology.orchestra.run.smithi086.stdout:RedHatEnterpriseServer 2021-04-09T18:08:30.472 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-04-09T18:08:30.472 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=d23cd2d5f486487b0b80655c4e33344f0f97b82b 2021-04-09T18:08:30.546 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2021-04-09-0846-nautilus/d23cd2d5f486487b0b80655c4e33344f0f97b82b/centos/7/flavors/default/ 2021-04-09T18:08:30.547 INFO:teuthology.task.install.rpm:Package version is 14.2.19-116.gd23cd2d5 2021-04-09T18:08:30.617 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2021-04-09-0846-nautilus/d23cd2d5f486487b0b80655c4e33344f0f97b82b/centos/7/flavors/default/ 2021-04-09T18:08:30.618 INFO:teuthology.task.install.rpm:Package version is 14.2.19-116.gd23cd2d5 2021-04-09T18:08:30.670 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2021-04-09-0846-nautilus/d23cd2d5f486487b0b80655c4e33344f0f97b82b/centos/7/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2021-04-09-0846-nautilus/d23cd2d5f486487b0b80655c4e33344f0f97b82b/centos/7/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2021-04-09-0846-nautilus/d23cd2d5f486487b0b80655c4e33344f0f97b82b/centos/7/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-04-09T18:08:30.671 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2021-04-09T18:08:30.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-04-09T18:08:30.730 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2021-04-09T18:08:30.731 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d23cd2d5f486487b0b80655c4e33344f0f97b82b/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-04-09T18:08:30.734 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2021-04-09-0846-nautilus/d23cd2d5f486487b0b80655c4e33344f0f97b82b/centos/7/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2021-04-09-0846-nautilus/d23cd2d5f486487b0b80655c4e33344f0f97b82b/centos/7/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2021-04-09-0846-nautilus/d23cd2d5f486487b0b80655c4e33344f0f97b82b/centos/7/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-04-09T18:08:30.735 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2021-04-09T18:08:30.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-04-09T18:08:30.788 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2021-04-09T18:08:30.789 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d23cd2d5f486487b0b80655c4e33344f0f97b82b/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-04-09T18:08:30.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-04-09T18:08:30.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-04-09T18:08:30.902 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-04-09T18:08:30.945 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-04-09T18:08:30.959 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2021-04-09T18:08:30.961 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2021-04-09T18:08:30.996 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2021-04-09T18:08:30.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2021-04-09T18:08:31.173 INFO:teuthology.orchestra.run.smithi032.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T18:08:31.174 INFO:teuthology.orchestra.run.smithi032.stdout: : disabled-repos, subscription-manager 2021-04-09T18:08:31.191 INFO:teuthology.orchestra.run.smithi086.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T18:08:31.192 INFO:teuthology.orchestra.run.smithi086.stdout: : disabled-repos, subscription-manager 2021-04-09T18:08:31.549 INFO:teuthology.orchestra.run.smithi086.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2021-04-09T18:08:31.550 INFO:teuthology.orchestra.run.smithi086.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2021-04-09T18:08:31.550 INFO:teuthology.orchestra.run.smithi086.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2021-04-09T18:08:31.559 INFO:teuthology.orchestra.run.smithi032.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2021-04-09T18:08:31.560 INFO:teuthology.orchestra.run.smithi032.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2021-04-09T18:08:31.560 INFO:teuthology.orchestra.run.smithi032.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2021-04-09T18:08:31.708 INFO:teuthology.orchestra.run.smithi086.stdout:Cleaning up list of fastest mirrors 2021-04-09T18:08:31.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2021-04-09T18:08:31.774 INFO:teuthology.orchestra.run.smithi032.stdout:Cleaning up list of fastest mirrors 2021-04-09T18:08:31.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2021-04-09T18:08:31.922 INFO:teuthology.orchestra.run.smithi086.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T18:08:31.922 INFO:teuthology.orchestra.run.smithi086.stdout: : disabled-repos, subscription-manager 2021-04-09T18:08:32.002 INFO:teuthology.orchestra.run.smithi032.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-04-09T18:08:32.002 INFO:teuthology.orchestra.run.smithi032.stdout: : disabled-repos, subscription-manager 2021-04-09T18:08:32.245 INFO:teuthology.orchestra.run.smithi086.stdout:Determining fastest mirrors 2021-04-09T18:08:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T18:08:32.374 INFO:teuthology.orchestra.run.smithi032.stdout:Determining fastest mirrors 2021-04-09T18:08:35.379 INFO:teuthology.orchestra.run.smithi032.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-04-09T18:09:05.464 INFO:teuthology.orchestra.run.smithi086.stdout:29 packages excluded due to repository priority protections 2021-04-09T18:09:11.587 INFO:teuthology.orchestra.run.smithi032.stdout:29 packages excluded due to repository priority protections 2021-04-09T18:09:32.188 INFO:teuthology.orchestra.run.smithi086.stdout:Resolving Dependencies 2021-04-09T18:09:32.189 INFO:teuthology.orchestra.run.smithi086.stdout:--> Running transaction check 2021-04-09T18:09:32.189 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2021-04-09T18:09:32.197 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.199 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: ceph-osd = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.200 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: ceph-mds = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.202 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: ceph-mon = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.204 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-fuse.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.218 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-mgr.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.218 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: ceph-base = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.219 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.221 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.221 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.222 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.223 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.224 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.227 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-mgr-dashboard-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:32.228 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:32.228 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.228 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.229 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:32.232 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.233 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:32.233 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.233 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:32.233 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-radosgw.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.236 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: librgw2 = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.243 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.243 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.244 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.244 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-test.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.247 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: ceph-common = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.248 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.249 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.251 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.252 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-5.el7 will be installed 2021-04-09T18:09:32.260 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-5.el7.x86_64 2021-04-09T18:09:32.261 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2021-04-09T18:09:32.261 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package libcephfs-devel.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.262 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: librados-devel = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:libcephfs-devel-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.262 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package libcephfs2.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.263 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2021-04-09T18:09:32.263 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package librados2.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be an update 2021-04-09T18:09:32.266 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.266 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2021-04-09T18:09:32.266 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package librbd1.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be an update 2021-04-09T18:09:32.267 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-21.el7_9 will be installed 2021-04-09T18:09:32.267 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-21.el7_9 for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-04-09T18:09:32.296 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-04-09T18:09:32.297 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-ceph-compat.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.297 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-cephfs = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.300 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-rgw = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.300 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-rbd = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.301 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-rados = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.301 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python3-cephfs.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.302 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python3-ceph-argparse = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python3-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.303 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python3-rados = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python3-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.304 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package rbd-fuse.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.304 INFO:teuthology.orchestra.run.smithi086.stdout:--> Running transaction check 2021-04-09T18:09:32.305 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-base.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.306 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-common.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.308 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.308 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.311 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.311 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.313 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:32.313 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.314 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-mds.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.314 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-mon.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.314 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-osd.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.317 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package ceph-selinux.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.338 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package jq.x86_64 0:1.6-2.el7 will be installed 2021-04-09T18:09:32.339 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libonig.so.5()(64bit) for package: jq-1.6-2.el7.x86_64 2021-04-09T18:09:32.339 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package krb5-devel.x86_64 0:1.15.1-50.el7 will be installed 2021-04-09T18:09:32.339 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-04-09T18:09:32.339 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-04-09T18:09:32.341 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-04-09T18:09:32.347 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-04-09T18:09:32.349 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2021-04-09T18:09:32.349 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package librabbitmq.x86_64 0:0.8.0-3.el7 will be installed 2021-04-09T18:09:32.350 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package librados-devel.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.351 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package libradosstriper1.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.352 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package librgw2.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.352 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2021-04-09T18:09:32.353 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-04-09T18:09:32.353 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-04-09T18:09:32.354 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be updated 2021-04-09T18:09:32.357 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-1.0.2k-19.el7.x86_64 2021-04-09T18:09:32.358 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-04-09T18:09:32.358 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-cephfs.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.358 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2021-04-09T18:09:32.373 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2021-04-09T18:09:32.374 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:32.374 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:32.374 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:32.376 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:32.377 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:32.378 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-rados.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.378 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-rbd.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.379 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-requests.noarch 0:2.6.0-10.el7 will be installed 2021-04-09T18:09:32.379 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-10.el7.noarch 2021-04-09T18:09:32.385 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-rgw.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.385 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2021-04-09T18:09:32.385 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2021-04-09T18:09:32.386 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2021-04-09T18:09:32.386 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2021-04-09T18:09:32.386 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2021-04-09T18:09:32.386 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-kubernetes.noarch 1:11.0.0-6.el7 will be installed 2021-04-09T18:09:32.387 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-certifi for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-04-09T18:09:32.388 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-google-auth for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-04-09T18:09:32.388 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-requests-oauthlib for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-04-09T18:09:32.388 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-websocket-client for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-04-09T18:09:32.397 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2021-04-09T18:09:32.398 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2021-04-09T18:09:32.398 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python3-ceph-argparse.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.398 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python3-rados.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.399 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2021-04-09T18:09:32.399 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2021-04-09T18:09:32.400 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-04-09T18:09:32.401 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-04-09T18:09:32.402 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2021-04-09T18:09:32.403 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package zlib-devel.x86_64 0:1.2.7-19.el7_9 will be installed 2021-04-09T18:09:32.403 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: zlib = 1.2.7-19.el7_9 for package: zlib-devel-1.2.7-19.el7_9.x86_64 2021-04-09T18:09:32.409 INFO:teuthology.orchestra.run.smithi086.stdout:--> Running transaction check 2021-04-09T18:09:32.410 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2021-04-09T18:09:32.410 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2021-04-09T18:09:32.411 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-19.el7 will be installed 2021-04-09T18:09:32.411 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2021-04-09T18:09:32.411 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package libselinux-devel.x86_64 0:2.5-15.el7 will be installed 2021-04-09T18:09:32.413 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-15.el7.x86_64 2021-04-09T18:09:32.414 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-15.el7.x86_64 2021-04-09T18:09:32.414 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-15.el7.x86_64 2021-04-09T18:09:32.449 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2021-04-09T18:09:32.450 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2021-04-09T18:09:32.450 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package oniguruma.x86_64 0:6.8.2-1.el7 will be installed 2021-04-09T18:09:32.451 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be updated 2021-04-09T18:09:32.451 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package openssl.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-04-09T18:09:32.452 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:32.452 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2021-04-09T18:09:32.453 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2021-04-09T18:09:32.453 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2021-04-09T18:09:32.457 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2021-04-09T18:09:32.457 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2021-04-09T18:09:32.458 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2021-04-09T18:09:32.458 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2021-04-09T18:09:32.458 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2021-04-09T18:09:32.459 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2021-04-09T18:09:32.459 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2021-04-09T18:09:32.460 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2021-04-09T18:09:32.460 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2021-04-09T18:09:32.461 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2021-04-09T18:09:32.461 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2021-04-09T18:09:32.461 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2021-04-09T18:09:32.462 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-04-09T18:09:32.462 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-04-09T18:09:32.463 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-04-09T18:09:32.463 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2021-04-09T18:09:32.463 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2021-04-09T18:09:32.464 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2021-04-09T18:09:32.464 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2021-04-09T18:09:32.464 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-04-09T18:09:32.464 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-04-09T18:09:32.465 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2021-04-09T18:09:32.465 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package zlib.x86_64 0:1.2.7-18.el7 will be updated 2021-04-09T18:09:32.465 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package zlib.x86_64 0:1.2.7-19.el7_9 will be an update 2021-04-09T18:09:32.465 INFO:teuthology.orchestra.run.smithi086.stdout:--> Running transaction check 2021-04-09T18:09:32.466 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2021-04-09T18:09:32.466 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2021-04-09T18:09:32.467 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2021-04-09T18:09:32.467 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2021-04-09T18:09:32.467 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2021-04-09T18:09:32.468 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2021-04-09T18:09:32.468 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-apipkg.noarch 0:1.5-5.el7 will be installed 2021-04-09T18:09:32.468 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2021-04-09T18:09:32.468 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2021-04-09T18:09:32.469 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2021-04-09T18:09:32.469 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python2-rsa.noarch 0:3.4.2-2.el7 will be installed 2021-04-09T18:09:32.469 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2021-04-09T18:09:32.469 INFO:teuthology.orchestra.run.smithi086.stdout:--> Running transaction check 2021-04-09T18:09:32.470 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2021-04-09T18:09:32.470 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2021-04-09T18:09:32.470 INFO:teuthology.orchestra.run.smithi086.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2021-04-09T18:09:32.471 INFO:teuthology.orchestra.run.smithi086.stdout:--> Running transaction check 2021-04-09T18:09:32.471 INFO:teuthology.orchestra.run.smithi086.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2021-04-09T18:09:33.234 INFO:teuthology.orchestra.run.smithi086.stdout:--> Finished Dependency Resolution 2021-04-09T18:09:33.316 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:09:33.316 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies Resolved 2021-04-09T18:09:33.328 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:09:33.329 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2021-04-09T18:09:33.329 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2021-04-09T18:09:33.329 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2021-04-09T18:09:33.330 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2021-04-09T18:09:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2021-04-09T18:09:33.330 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 2.0 k 2021-04-09T18:09:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 572 k 2021-04-09T18:09:33.331 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 1.8 M 2021-04-09T18:09:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 3.9 M 2021-04-09T18:09:33.332 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-cloud 2021-04-09T18:09:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 83 k 2021-04-09T18:09:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local 2021-04-09T18:09:33.333 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 1.1 M 2021-04-09T18:09:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 21 k 2021-04-09T18:09:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-ssh noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 21 k 2021-04-09T18:09:33.334 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 5.9 M 2021-04-09T18:09:33.335 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 34 M 2021-04-09T18:09:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel 2021-04-09T18:09:33.336 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 0.176-5.el7 rhel-7-server-rpms 40 k 2021-04-09T18:09:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2021-04-09T18:09:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 17 k 2021-04-09T18:09:33.337 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 500 k 2021-04-09T18:09:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.5 M 2021-04-09T18:09:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: python-ceph-compat x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 2.2 k 2021-04-09T18:09:33.338 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 151 k 2021-04-09T18:09:33.339 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 69 k 2021-04-09T18:09:33.339 INFO:teuthology.orchestra.run.smithi086.stdout:Updating: 2021-04-09T18:09:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 3.4 M 2021-04-09T18:09:33.340 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 1.7 M 2021-04-09T18:09:33.340 INFO:teuthology.orchestra.run.smithi086.stdout:Installing for dependencies: 2021-04-09T18:09:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 5.5 M 2021-04-09T18:09:33.341 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 19 M 2021-04-09T18:09:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards 2021-04-09T18:09:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 19 k 2021-04-09T18:09:33.342 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 1.9 M 2021-04-09T18:09:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 4.3 M 2021-04-09T18:09:33.343 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 16 M 2021-04-09T18:09:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 20 k 2021-04-09T18:09:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-2.el7 epel 167 k 2021-04-09T18:09:33.344 INFO:teuthology.orchestra.run.smithi086.stdout: keyutils-libs-devel 2021-04-09T18:09:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2021-04-09T18:09:33.345 INFO:teuthology.orchestra.run.smithi086.stdout: krb5-devel x86_64 1.15.1-50.el7 rhel-7-server-rpms 273 k 2021-04-09T18:09:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2021-04-09T18:09:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: libcom_err-devel x86_64 1.42.9-19.el7 rhel-7-server-rpms 32 k 2021-04-09T18:09:33.346 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2021-04-09T18:09:33.355 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.8.0-3.el7 rhel-7-server-rpms 37 k 2021-04-09T18:09:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 86 k 2021-04-09T18:09:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 364 k 2021-04-09T18:09:33.356 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2021-04-09T18:09:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 5.2 M 2021-04-09T18:09:33.357 INFO:teuthology.orchestra.run.smithi086.stdout: libselinux-devel x86_64 2.5-15.el7 rhel-7-server-rpms 187 k 2021-04-09T18:09:33.360 INFO:teuthology.orchestra.run.smithi086.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2021-04-09T18:09:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2021-04-09T18:09:33.361 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2021-04-09T18:09:33.362 INFO:teuthology.orchestra.run.smithi086.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2021-04-09T18:09:33.363 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.8.2-1.el7 epel 181 k 2021-04-09T18:09:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2021-04-09T18:09:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2021-04-09T18:09:33.364 INFO:teuthology.orchestra.run.smithi086.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2021-04-09T18:09:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: python-ceph-argparse 2021-04-09T18:09:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 35 k 2021-04-09T18:09:33.365 INFO:teuthology.orchestra.run.smithi086.stdout: python-cephfs x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 113 k 2021-04-09T18:09:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2021-04-09T18:09:33.366 INFO:teuthology.orchestra.run.smithi086.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2021-04-09T18:09:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2021-04-09T18:09:33.367 INFO:teuthology.orchestra.run.smithi086.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2021-04-09T18:09:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2021-04-09T18:09:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2021-04-09T18:09:33.368 INFO:teuthology.orchestra.run.smithi086.stdout: rhel-7-server-rpms 866 k 2021-04-09T18:09:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2021-04-09T18:09:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2021-04-09T18:09:33.369 INFO:teuthology.orchestra.run.smithi086.stdout: python-rados x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 199 k 2021-04-09T18:09:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: python-rbd x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 194 k 2021-04-09T18:09:33.370 INFO:teuthology.orchestra.run.smithi086.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2021-04-09T18:09:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: python-requests noarch 2.6.0-10.el7 rhel-7-server-rpms 95 k 2021-04-09T18:09:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: python-rgw x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 77 k 2021-04-09T18:09:33.371 INFO:teuthology.orchestra.run.smithi086.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2021-04-09T18:09:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: python-simplegeneric 2021-04-09T18:09:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 0.8-7.el7 epel 12 k 2021-04-09T18:09:33.372 INFO:teuthology.orchestra.run.smithi086.stdout: python-singledispatch 2021-04-09T18:09:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2021-04-09T18:09:33.373 INFO:teuthology.orchestra.run.smithi086.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2021-04-09T18:09:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2021-04-09T18:09:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2021-04-09T18:09:33.374 INFO:teuthology.orchestra.run.smithi086.stdout: 202 k 2021-04-09T18:09:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: python-websocket-client 2021-04-09T18:09:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2021-04-09T18:09:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: 66 k 2021-04-09T18:09:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2021-04-09T18:09:33.376 INFO:teuthology.orchestra.run.smithi086.stdout: 102 k 2021-04-09T18:09:33.377 INFO:teuthology.orchestra.run.smithi086.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2021-04-09T18:09:33.380 INFO:teuthology.orchestra.run.smithi086.stdout: 562 k 2021-04-09T18:09:33.381 INFO:teuthology.orchestra.run.smithi086.stdout: python2-apipkg noarch 1.5-5.el7 epel 17 k 2021-04-09T18:09:33.381 INFO:teuthology.orchestra.run.smithi086.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2021-04-09T18:09:33.381 INFO:teuthology.orchestra.run.smithi086.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2021-04-09T18:09:33.382 INFO:teuthology.orchestra.run.smithi086.stdout: python2-google-auth 2021-04-09T18:09:33.382 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 1:1.1.1-5.el7 epel 76 k 2021-04-09T18:09:33.383 INFO:teuthology.orchestra.run.smithi086.stdout: python2-kubernetes noarch 1:11.0.0-6.el7 epel 1.1 M 2021-04-09T18:09:33.383 INFO:teuthology.orchestra.run.smithi086.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2021-04-09T18:09:33.384 INFO:teuthology.orchestra.run.smithi086.stdout: python2-pyasn1-modules 2021-04-09T18:09:33.384 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2021-04-09T18:09:33.385 INFO:teuthology.orchestra.run.smithi086.stdout: 59 k 2021-04-09T18:09:33.385 INFO:teuthology.orchestra.run.smithi086.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2021-04-09T18:09:33.385 INFO:teuthology.orchestra.run.smithi086.stdout: python2-requests-oauthlib 2021-04-09T18:09:33.388 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2021-04-09T18:09:33.388 INFO:teuthology.orchestra.run.smithi086.stdout: python2-rsa noarch 3.4.2-2.el7 epel 68 k 2021-04-09T18:09:33.388 INFO:teuthology.orchestra.run.smithi086.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2021-04-09T18:09:33.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse 2021-04-09T18:09:33.389 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 35 k 2021-04-09T18:09:33.390 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 317 k 2021-04-09T18:09:33.390 INFO:teuthology.orchestra.run.smithi086.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2021-04-09T18:09:33.391 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2021-04-09T18:09:33.391 INFO:teuthology.orchestra.run.smithi086.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2021-04-09T18:09:33.391 INFO:teuthology.orchestra.run.smithi086.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2021-04-09T18:09:33.392 INFO:teuthology.orchestra.run.smithi086.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2021-04-09T18:09:33.392 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 50 k 2021-04-09T18:09:33.393 INFO:teuthology.orchestra.run.smithi086.stdout:Updating for dependencies: 2021-04-09T18:09:33.393 INFO:teuthology.orchestra.run.smithi086.stdout: openssl x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 493 k 2021-04-09T18:09:33.393 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-libs x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.2 M 2021-04-09T18:09:33.394 INFO:teuthology.orchestra.run.smithi086.stdout: zlib x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 90 k 2021-04-09T18:09:33.397 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:09:33.397 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2021-04-09T18:09:33.397 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2021-04-09T18:09:33.400 INFO:teuthology.orchestra.run.smithi086.stdout:Install 19 Packages (+70 Dependent packages) 2021-04-09T18:09:33.413 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages (+ 3 Dependent packages) 2021-04-09T18:09:33.413 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:09:33.414 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 130 M 2021-04-09T18:09:33.414 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading packages: 2021-04-09T18:09:33.415 INFO:teuthology.orchestra.run.smithi086.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2021-04-09T18:09:38.676 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2021-04-09T18:09:38.676 INFO:teuthology.orchestra.run.smithi086.stdout:Total 24 MB/s | 130 MB 00:05 2021-04-09T18:09:38.721 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2021-04-09T18:09:38.896 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2021-04-09T18:09:39.189 INFO:teuthology.orchestra.run.smithi032.stdout:Resolving Dependencies 2021-04-09T18:09:39.190 INFO:teuthology.orchestra.run.smithi032.stdout:--> Running transaction check 2021-04-09T18:09:39.190 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2021-04-09T18:09:39.200 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.200 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: ceph-osd = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.209 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: ceph-mds = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.210 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: ceph-mon = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.210 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-fuse.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.215 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-mgr.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.219 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: ceph-base = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.220 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.227 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.232 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.233 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.233 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.234 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.234 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-mgr-dashboard-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:39.235 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:39.235 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.235 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.236 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:39.238 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.239 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:39.250 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.251 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.19-116.gd23cd2d5.el7.noarch 2021-04-09T18:09:39.251 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-radosgw.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.276 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: librgw2 = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.277 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.277 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.278 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.278 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-test.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.279 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: ceph-common = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.279 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.280 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.280 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.280 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-5.el7 will be installed 2021-04-09T18:09:39.281 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-5.el7.x86_64 2021-04-09T18:09:39.291 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2021-04-09T18:09:39.297 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package libcephfs-devel.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.298 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: librados-devel = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:libcephfs-devel-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.298 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package libcephfs2.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.299 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2021-04-09T18:09:39.299 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package librados2.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be an update 2021-04-09T18:09:39.300 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.304 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2021-04-09T18:09:39.304 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package librbd1.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be an update 2021-04-09T18:09:39.305 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-21.el7_9 will be installed 2021-04-09T18:09:39.306 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-21.el7_9 for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-04-09T18:09:39.310 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-04-09T18:09:39.311 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-ceph-compat.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.311 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-cephfs = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.312 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-rgw = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.312 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-rbd = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.313 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-rados = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.325 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python3-cephfs.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.325 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python3-ceph-argparse = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python3-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.326 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python3-rados = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:python3-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.326 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package rbd-fuse.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.326 INFO:teuthology.orchestra.run.smithi032.stdout:--> Running transaction check 2021-04-09T18:09:39.327 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-base.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.364 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-common.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.367 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.19-116.gd23cd2d5.el7 for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.368 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.368 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.368 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.369 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 2021-04-09T18:09:39.370 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.371 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-mds.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.371 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-mon.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.372 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-osd.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.372 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package ceph-selinux.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.373 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package jq.x86_64 0:1.6-2.el7 will be installed 2021-04-09T18:09:39.373 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libonig.so.5()(64bit) for package: jq-1.6-2.el7.x86_64 2021-04-09T18:09:39.374 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package krb5-devel.x86_64 0:1.15.1-50.el7 will be installed 2021-04-09T18:09:39.374 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-04-09T18:09:39.375 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-04-09T18:09:39.375 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-04-09T18:09:39.375 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-04-09T18:09:39.376 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2021-04-09T18:09:39.376 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package librabbitmq.x86_64 0:0.8.0-3.el7 will be installed 2021-04-09T18:09:39.376 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package librados-devel.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.376 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package libradosstriper1.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.377 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package librgw2.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.377 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2021-04-09T18:09:39.378 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-04-09T18:09:39.379 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-04-09T18:09:39.381 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be updated 2021-04-09T18:09:39.381 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-1.0.2k-19.el7.x86_64 2021-04-09T18:09:39.381 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-04-09T18:09:39.382 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-cephfs.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.382 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2021-04-09T18:09:39.382 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2021-04-09T18:09:39.382 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:39.383 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:39.383 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:39.385 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:39.385 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2021-04-09T18:09:39.385 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-rados.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.386 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-rbd.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.386 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-requests.noarch 0:2.6.0-10.el7 will be installed 2021-04-09T18:09:39.387 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-10.el7.noarch 2021-04-09T18:09:39.389 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-rgw.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.389 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2021-04-09T18:09:39.389 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2021-04-09T18:09:39.389 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2021-04-09T18:09:39.390 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2021-04-09T18:09:39.390 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2021-04-09T18:09:39.390 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-kubernetes.noarch 1:11.0.0-6.el7 will be installed 2021-04-09T18:09:39.391 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-certifi for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-04-09T18:09:39.392 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-google-auth for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-04-09T18:09:39.392 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-requests-oauthlib for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-04-09T18:09:39.393 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-websocket-client for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-04-09T18:09:39.394 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded 2021-04-09T18:09:39.394 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2021-04-09T18:09:39.397 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2021-04-09T18:09:39.398 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2021-04-09T18:09:39.399 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python3-ceph-argparse.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.399 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python3-rados.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.399 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2021-04-09T18:09:39.405 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2021-04-09T18:09:39.405 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-04-09T18:09:39.407 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-04-09T18:09:39.407 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2021-04-09T18:09:39.409 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package zlib-devel.x86_64 0:1.2.7-19.el7_9 will be installed 2021-04-09T18:09:39.409 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: zlib = 1.2.7-19.el7_9 for package: zlib-devel-1.2.7-19.el7_9.x86_64 2021-04-09T18:09:39.413 INFO:teuthology.orchestra.run.smithi032.stdout:--> Running transaction check 2021-04-09T18:09:39.415 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2021-04-09T18:09:39.415 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2021-04-09T18:09:39.423 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-19.el7 will be installed 2021-04-09T18:09:39.424 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2021-04-09T18:09:39.424 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package libselinux-devel.x86_64 0:2.5-15.el7 will be installed 2021-04-09T18:09:39.428 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-15.el7.x86_64 2021-04-09T18:09:39.428 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-15.el7.x86_64 2021-04-09T18:09:39.428 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-15.el7.x86_64 2021-04-09T18:09:39.436 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2021-04-09T18:09:39.436 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2021-04-09T18:09:39.436 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package oniguruma.x86_64 0:6.8.2-1.el7 will be installed 2021-04-09T18:09:39.437 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be updated 2021-04-09T18:09:39.437 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package openssl.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-04-09T18:09:39.437 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.19-116.gd23cd2d5.el7 will be installed 2021-04-09T18:09:39.437 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2021-04-09T18:09:39.438 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2021-04-09T18:09:39.438 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2021-04-09T18:09:39.438 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2021-04-09T18:09:39.438 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2021-04-09T18:09:39.438 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2021-04-09T18:09:39.439 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2021-04-09T18:09:39.439 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2021-04-09T18:09:39.439 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2021-04-09T18:09:39.439 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2021-04-09T18:09:39.440 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2021-04-09T18:09:39.440 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2021-04-09T18:09:39.452 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2021-04-09T18:09:39.452 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2021-04-09T18:09:39.452 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2021-04-09T18:09:39.453 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-04-09T18:09:39.453 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-04-09T18:09:39.457 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-04-09T18:09:39.458 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2021-04-09T18:09:39.458 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2021-04-09T18:09:39.458 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2021-04-09T18:09:39.458 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2021-04-09T18:09:39.460 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-04-09T18:09:39.461 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-04-09T18:09:39.461 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2021-04-09T18:09:39.461 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package zlib.x86_64 0:1.2.7-18.el7 will be updated 2021-04-09T18:09:39.465 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package zlib.x86_64 0:1.2.7-19.el7_9 will be an update 2021-04-09T18:09:39.466 INFO:teuthology.orchestra.run.smithi032.stdout:--> Running transaction check 2021-04-09T18:09:39.471 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2021-04-09T18:09:39.472 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2021-04-09T18:09:39.473 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2021-04-09T18:09:39.474 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2021-04-09T18:09:39.478 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2021-04-09T18:09:39.478 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2021-04-09T18:09:39.479 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-apipkg.noarch 0:1.5-5.el7 will be installed 2021-04-09T18:09:39.479 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2021-04-09T18:09:39.480 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2021-04-09T18:09:39.480 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2021-04-09T18:09:39.481 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python2-rsa.noarch 0:3.4.2-2.el7 will be installed 2021-04-09T18:09:39.481 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2021-04-09T18:09:39.481 INFO:teuthology.orchestra.run.smithi032.stdout:--> Running transaction check 2021-04-09T18:09:39.482 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2021-04-09T18:09:39.482 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2021-04-09T18:09:39.483 INFO:teuthology.orchestra.run.smithi032.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2021-04-09T18:09:39.511 INFO:teuthology.orchestra.run.smithi032.stdout:--> Running transaction check 2021-04-09T18:09:39.511 INFO:teuthology.orchestra.run.smithi032.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2021-04-09T18:09:40.195 INFO:teuthology.orchestra.run.smithi086.stdout: Updating : zlib-1.2.7-19.el7_9.x86_64 1/99 2021-04-09T18:09:40.333 INFO:teuthology.orchestra.run.smithi032.stdout:--> Finished Dependency Resolution 2021-04-09T18:09:40.424 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:09:40.425 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies Resolved 2021-04-09T18:09:40.439 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:09:40.440 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2021-04-09T18:09:40.440 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2021-04-09T18:09:40.441 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2021-04-09T18:09:40.441 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2021-04-09T18:09:40.442 INFO:teuthology.orchestra.run.smithi032.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2021-04-09T18:09:40.442 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 2.0 k 2021-04-09T18:09:40.443 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 572 k 2021-04-09T18:09:40.443 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 1.8 M 2021-04-09T18:09:40.443 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 3.9 M 2021-04-09T18:09:40.444 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-cloud 2021-04-09T18:09:40.447 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 83 k 2021-04-09T18:09:40.455 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local 2021-04-09T18:09:40.456 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 1.1 M 2021-04-09T18:09:40.456 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 21 k 2021-04-09T18:09:40.460 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-ssh noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 21 k 2021-04-09T18:09:40.461 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 5.9 M 2021-04-09T18:09:40.462 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 34 M 2021-04-09T18:09:40.462 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel 2021-04-09T18:09:40.463 INFO:teuthology.orchestra.run.smithi032.stdout: x86_64 0.176-5.el7 rhel-7-server-rpms 40 k 2021-04-09T18:09:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2021-04-09T18:09:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 17 k 2021-04-09T18:09:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 500 k 2021-04-09T18:09:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.5 M 2021-04-09T18:09:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: python-ceph-compat x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 2.2 k 2021-04-09T18:09:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 151 k 2021-04-09T18:09:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 69 k 2021-04-09T18:09:40.467 INFO:teuthology.orchestra.run.smithi032.stdout:Updating: 2021-04-09T18:09:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 3.4 M 2021-04-09T18:09:40.473 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 1.7 M 2021-04-09T18:09:40.473 INFO:teuthology.orchestra.run.smithi032.stdout:Installing for dependencies: 2021-04-09T18:09:40.474 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 5.5 M 2021-04-09T18:09:40.474 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 19 M 2021-04-09T18:09:40.474 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards 2021-04-09T18:09:40.475 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 2:14.2.19-116.gd23cd2d5.el7 ceph-noarch 19 k 2021-04-09T18:09:40.475 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 1.9 M 2021-04-09T18:09:40.476 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 4.3 M 2021-04-09T18:09:40.476 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 16 M 2021-04-09T18:09:40.477 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 20 k 2021-04-09T18:09:40.477 INFO:teuthology.orchestra.run.smithi032.stdout: jq x86_64 1.6-2.el7 epel 167 k 2021-04-09T18:09:40.478 INFO:teuthology.orchestra.run.smithi032.stdout: keyutils-libs-devel 2021-04-09T18:09:40.478 INFO:teuthology.orchestra.run.smithi032.stdout: x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2021-04-09T18:09:40.478 INFO:teuthology.orchestra.run.smithi032.stdout: krb5-devel x86_64 1.15.1-50.el7 rhel-7-server-rpms 273 k 2021-04-09T18:09:40.479 INFO:teuthology.orchestra.run.smithi032.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2021-04-09T18:09:40.479 INFO:teuthology.orchestra.run.smithi032.stdout: libcom_err-devel x86_64 1.42.9-19.el7 rhel-7-server-rpms 32 k 2021-04-09T18:09:40.489 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2021-04-09T18:09:40.490 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.8.0-3.el7 rhel-7-server-rpms 37 k 2021-04-09T18:09:40.495 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 86 k 2021-04-09T18:09:40.496 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 364 k 2021-04-09T18:09:40.497 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2021-04-09T18:09:40.498 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 5.2 M 2021-04-09T18:09:40.498 INFO:teuthology.orchestra.run.smithi032.stdout: libselinux-devel x86_64 2.5-15.el7 rhel-7-server-rpms 187 k 2021-04-09T18:09:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2021-04-09T18:09:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2021-04-09T18:09:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2021-04-09T18:09:40.500 INFO:teuthology.orchestra.run.smithi032.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2021-04-09T18:09:40.500 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma x86_64 6.8.2-1.el7 epel 181 k 2021-04-09T18:09:40.501 INFO:teuthology.orchestra.run.smithi032.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2021-04-09T18:09:40.501 INFO:teuthology.orchestra.run.smithi032.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2021-04-09T18:09:40.501 INFO:teuthology.orchestra.run.smithi032.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2021-04-09T18:09:40.502 INFO:teuthology.orchestra.run.smithi032.stdout: python-ceph-argparse 2021-04-09T18:09:40.502 INFO:teuthology.orchestra.run.smithi032.stdout: x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 35 k 2021-04-09T18:09:40.503 INFO:teuthology.orchestra.run.smithi032.stdout: python-cephfs x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 113 k 2021-04-09T18:09:40.503 INFO:teuthology.orchestra.run.smithi032.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2021-04-09T18:09:40.503 INFO:teuthology.orchestra.run.smithi032.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2021-04-09T18:09:40.504 INFO:teuthology.orchestra.run.smithi032.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2021-04-09T18:09:40.506 INFO:teuthology.orchestra.run.smithi032.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2021-04-09T18:09:40.507 INFO:teuthology.orchestra.run.smithi032.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2021-04-09T18:09:40.508 INFO:teuthology.orchestra.run.smithi032.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2021-04-09T18:09:40.508 INFO:teuthology.orchestra.run.smithi032.stdout: rhel-7-server-rpms 866 k 2021-04-09T18:09:40.508 INFO:teuthology.orchestra.run.smithi032.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2021-04-09T18:09:40.509 INFO:teuthology.orchestra.run.smithi032.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2021-04-09T18:09:40.509 INFO:teuthology.orchestra.run.smithi032.stdout: python-rados x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 199 k 2021-04-09T18:09:40.510 INFO:teuthology.orchestra.run.smithi032.stdout: python-rbd x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 194 k 2021-04-09T18:09:40.510 INFO:teuthology.orchestra.run.smithi032.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2021-04-09T18:09:40.510 INFO:teuthology.orchestra.run.smithi032.stdout: python-requests noarch 2.6.0-10.el7 rhel-7-server-rpms 95 k 2021-04-09T18:09:40.511 INFO:teuthology.orchestra.run.smithi032.stdout: python-rgw x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 77 k 2021-04-09T18:09:40.511 INFO:teuthology.orchestra.run.smithi032.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2021-04-09T18:09:40.511 INFO:teuthology.orchestra.run.smithi032.stdout: python-simplegeneric 2021-04-09T18:09:40.512 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 0.8-7.el7 epel 12 k 2021-04-09T18:09:40.512 INFO:teuthology.orchestra.run.smithi032.stdout: python-singledispatch 2021-04-09T18:09:40.513 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2021-04-09T18:09:40.513 INFO:teuthology.orchestra.run.smithi032.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2021-04-09T18:09:40.513 INFO:teuthology.orchestra.run.smithi032.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2021-04-09T18:09:40.514 INFO:teuthology.orchestra.run.smithi032.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2021-04-09T18:09:40.514 INFO:teuthology.orchestra.run.smithi032.stdout: 202 k 2021-04-09T18:09:40.515 INFO:teuthology.orchestra.run.smithi032.stdout: python-websocket-client 2021-04-09T18:09:40.517 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2021-04-09T18:09:40.517 INFO:teuthology.orchestra.run.smithi032.stdout: 66 k 2021-04-09T18:09:40.518 INFO:teuthology.orchestra.run.smithi032.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2021-04-09T18:09:40.519 INFO:teuthology.orchestra.run.smithi032.stdout: 102 k 2021-04-09T18:09:40.519 INFO:teuthology.orchestra.run.smithi032.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2021-04-09T18:09:40.520 INFO:teuthology.orchestra.run.smithi032.stdout: 562 k 2021-04-09T18:09:40.520 INFO:teuthology.orchestra.run.smithi032.stdout: python2-apipkg noarch 1.5-5.el7 epel 17 k 2021-04-09T18:09:40.520 INFO:teuthology.orchestra.run.smithi032.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2021-04-09T18:09:40.521 INFO:teuthology.orchestra.run.smithi032.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2021-04-09T18:09:40.521 INFO:teuthology.orchestra.run.smithi032.stdout: python2-google-auth 2021-04-09T18:09:40.521 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 1:1.1.1-5.el7 epel 76 k 2021-04-09T18:09:40.522 INFO:teuthology.orchestra.run.smithi032.stdout: python2-kubernetes noarch 1:11.0.0-6.el7 epel 1.1 M 2021-04-09T18:09:40.522 INFO:teuthology.orchestra.run.smithi032.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2021-04-09T18:09:40.523 INFO:teuthology.orchestra.run.smithi032.stdout: python2-pyasn1-modules 2021-04-09T18:09:40.523 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2021-04-09T18:09:40.524 INFO:teuthology.orchestra.run.smithi032.stdout: 59 k 2021-04-09T18:09:40.524 INFO:teuthology.orchestra.run.smithi032.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2021-04-09T18:09:40.524 INFO:teuthology.orchestra.run.smithi032.stdout: python2-requests-oauthlib 2021-04-09T18:09:40.525 INFO:teuthology.orchestra.run.smithi032.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2021-04-09T18:09:40.525 INFO:teuthology.orchestra.run.smithi032.stdout: python2-rsa noarch 3.4.2-2.el7 epel 68 k 2021-04-09T18:09:40.530 INFO:teuthology.orchestra.run.smithi032.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2021-04-09T18:09:40.531 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse 2021-04-09T18:09:40.531 INFO:teuthology.orchestra.run.smithi032.stdout: x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 35 k 2021-04-09T18:09:40.532 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:14.2.19-116.gd23cd2d5.el7 ceph 317 k 2021-04-09T18:09:40.532 INFO:teuthology.orchestra.run.smithi032.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2021-04-09T18:09:40.537 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2021-04-09T18:09:40.538 INFO:teuthology.orchestra.run.smithi032.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2021-04-09T18:09:40.538 INFO:teuthology.orchestra.run.smithi032.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2021-04-09T18:09:40.539 INFO:teuthology.orchestra.run.smithi032.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2021-04-09T18:09:40.539 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 50 k 2021-04-09T18:09:40.540 INFO:teuthology.orchestra.run.smithi032.stdout:Updating for dependencies: 2021-04-09T18:09:40.540 INFO:teuthology.orchestra.run.smithi032.stdout: openssl x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 493 k 2021-04-09T18:09:40.541 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.2 M 2021-04-09T18:09:40.542 INFO:teuthology.orchestra.run.smithi032.stdout: zlib x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 90 k 2021-04-09T18:09:40.542 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:09:40.543 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2021-04-09T18:09:40.543 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2021-04-09T18:09:40.545 INFO:teuthology.orchestra.run.smithi032.stdout:Install 19 Packages (+70 Dependent packages) 2021-04-09T18:09:40.545 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 2 Packages (+ 3 Dependent packages) 2021-04-09T18:09:40.546 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:09:40.546 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 130 M 2021-04-09T18:09:40.547 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading packages: 2021-04-09T18:09:40.549 INFO:teuthology.orchestra.run.smithi032.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2021-04-09T18:09:40.561 INFO:teuthology.orchestra.run.smithi086.stdout: Updating : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 2/99 2021-04-09T18:09:40.891 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/99 2021-04-09T18:09:41.146 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.8.0-3.el7.x86_64 4/99 2021-04-09T18:09:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : zlib-devel-1.2.7-19.el7_9.x86_64 5/99 2021-04-09T18:09:41.731 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python2-six-1.9.0-0.el7.noarch 6/99 2021-04-09T18:09:42.102 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 7/99 2021-04-09T18:09:42.438 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 8/99 2021-04-09T18:09:42.808 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-webob-1.2.3-7.el7.noarch 9/99 2021-04-09T18:09:43.082 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 10/99 2021-04-09T18:09:43.367 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-requests-2.6.0-10.el7.noarch 11/99 2021-04-09T18:09:43.621 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:python-ceph-argparse-14.2.19-116.gd23cd2d5.el7.x86_64 12/99 2021-04-09T18:09:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 13/99 2021-04-09T18:09:44.400 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 14/99 2021-04-09T18:09:44.758 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 15/99 2021-04-09T18:09:45.045 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 16/99 2021-04-09T18:09:45.325 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.8.2-1.el7.x86_64 17/99 2021-04-09T18:09:45.618 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-2.el7.x86_64 18/99 2021-04-09T18:09:46.019 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 19/99 2021-04-09T18:09:46.346 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 20/99 2021-04-09T18:09:46.625 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 21/99 2021-04-09T18:09:46.885 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 22/99 2021-04-09T18:09:47.251 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 23/99 2021-04-09T18:09:47.625 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 24/99 2021-04-09T18:09:47.853 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-grafana-dashboards-14.2.19-116.gd23cd2d5.el7.noar 25/99 2021-04-09T18:09:48.144 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 26/99 2021-04-09T18:09:48.446 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/99 2021-04-09T18:09:48.703 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcom_err-devel-1.42.9-19.el7.x86_64 28/99 2021-04-09T18:09:49.006 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 29/99 2021-04-09T18:09:49.307 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 30/99 2021-04-09T18:09:49.641 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:python3-ceph-argparse-14.2.19-116.gd23cd2d5.el7.x86_64 31/99 2021-04-09T18:09:49.937 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 32/99 2021-04-09T18:09:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 33/99 2021-04-09T18:09:50.520 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 34/99 2021-04-09T18:09:50.815 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 35/99 2021-04-09T18:09:51.068 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 36/99 2021-04-09T18:09:51.386 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-04-09T18:09:52.511 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/99 2021-04-09T18:09:52.851 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 1:python2-kubernetes-11.0.0-6.el7.noarch 39/99 2021-04-09T18:09:53.123 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 40/99 2021-04-09T18:09:53.386 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 41/99 2021-04-09T18:09:53.611 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 42/99 2021-04-09T18:09:53.949 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 43/99 2021-04-09T18:09:54.578 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 44/99 2021-04-09T18:09:54.905 INFO:teuthology.orchestra.run.smithi086.stdout: Updating : 2:librados2-14.2.19-116.gd23cd2d5.el7.x86_64 45/99 2021-04-09T18:09:55.308 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:libcephfs2-14.2.19-116.gd23cd2d5.el7.x86_64 46/99 2021-04-09T18:09:55.618 INFO:teuthology.orchestra.run.smithi086.stdout: Updating : 2:librbd1-14.2.19-116.gd23cd2d5.el7.x86_64 47/99 2021-04-09T18:09:56.340 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:python-rados-14.2.19-116.gd23cd2d5.el7.x86_64 48/99 2021-04-09T18:09:56.574 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:librgw2-14.2.19-116.gd23cd2d5.el7.x86_64 49/99 2021-04-09T18:09:56.870 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:python-rgw-14.2.19-116.gd23cd2d5.el7.x86_64 50/99 2021-04-09T18:09:57.132 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:python-rbd-14.2.19-116.gd23cd2d5.el7.x86_64 51/99 2021-04-09T18:09:57.409 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:python-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 52/99 2021-04-09T18:09:58.235 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2021-04-09T18:09:58.235 INFO:teuthology.orchestra.run.smithi032.stdout:Total 7.3 MB/s | 130 MB 00:17 2021-04-09T18:09:58.284 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2021-04-09T18:09:58.472 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2021-04-09T18:09:58.926 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded 2021-04-09T18:09:58.927 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2021-04-09T18:09:59.668 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:libradosstriper1-14.2.19-116.gd23cd2d5.el7.x86_64 53/99 2021-04-09T18:09:59.943 INFO:teuthology.orchestra.run.smithi032.stdout: Updating : zlib-1.2.7-19.el7_9.x86_64 1/99 2021-04-09T18:10:00.293 INFO:teuthology.orchestra.run.smithi032.stdout: Updating : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 2/99 2021-04-09T18:10:00.564 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 54/99 2021-04-09T18:10:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/99 2021-04-09T18:10:00.831 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.8.0-3.el7.x86_64 4/99 2021-04-09T18:10:01.010 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-base-14.2.19-116.gd23cd2d5.el7.x86_64 55/99 2021-04-09T18:10:01.149 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : zlib-devel-1.2.7-19.el7_9.x86_64 5/99 2021-04-09T18:10:01.465 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python2-six-1.9.0-0.el7.noarch 6/99 2021-04-09T18:10:01.893 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 7/99 2021-04-09T18:10:02.200 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 8/99 2021-04-09T18:10:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-webob-1.2.3-7.el7.noarch 9/99 2021-04-09T18:10:02.828 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 10/99 2021-04-09T18:10:03.086 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-requests-2.6.0-10.el7.noarch 11/99 2021-04-09T18:10:03.318 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:python-ceph-argparse-14.2.19-116.gd23cd2d5.el7.x86_64 12/99 2021-04-09T18:10:03.566 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 13/99 2021-04-09T18:10:03.904 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 14/99 2021-04-09T18:10:04.246 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 15/99 2021-04-09T18:10:04.588 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 16/99 2021-04-09T18:10:04.864 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : oniguruma-6.8.2-1.el7.x86_64 17/99 2021-04-09T18:10:05.130 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : jq-1.6-2.el7.x86_64 18/99 2021-04-09T18:10:05.532 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 19/99 2021-04-09T18:10:05.834 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 20/99 2021-04-09T18:10:06.119 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 21/99 2021-04-09T18:10:06.371 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 22/99 2021-04-09T18:10:06.754 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 23/99 2021-04-09T18:10:07.162 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 24/99 2021-04-09T18:10:07.390 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-grafana-dashboards-14.2.19-116.gd23cd2d5.el7.noar 25/99 2021-04-09T18:10:07.707 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 26/99 2021-04-09T18:10:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/99 2021-04-09T18:10:08.248 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcom_err-devel-1.42.9-19.el7.x86_64 28/99 2021-04-09T18:10:08.528 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 29/99 2021-04-09T18:10:08.809 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 30/99 2021-04-09T18:10:09.141 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:python3-ceph-argparse-14.2.19-116.gd23cd2d5.el7.x86_64 31/99 2021-04-09T18:10:09.439 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 32/99 2021-04-09T18:10:09.698 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 33/99 2021-04-09T18:10:10.002 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 34/99 2021-04-09T18:10:10.303 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 35/99 2021-04-09T18:10:10.581 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 36/99 2021-04-09T18:10:10.921 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-04-09T18:10:11.223 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-selinux-14.2.19-116.gd23cd2d5.el7.x86_64 56/99 2021-04-09T18:10:11.934 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-osd-14.2.19-116.gd23cd2d5.el7.x86_64 57/99 2021-04-09T18:10:12.081 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/99 2021-04-09T18:10:12.388 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-mon-14.2.19-116.gd23cd2d5.el7.x86_64 58/99 2021-04-09T18:10:12.396 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 1:python2-kubernetes-11.0.0-6.el7.noarch 39/99 2021-04-09T18:10:12.660 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 40/99 2021-04-09T18:10:12.734 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-mds-14.2.19-116.gd23cd2d5.el7.x86_64 59/99 2021-04-09T18:10:12.905 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 41/99 2021-04-09T18:10:13.032 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:librados-devel-14.2.19-116.gd23cd2d5.el7.x86_64 60/99 2021-04-09T18:10:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 42/99 2021-04-09T18:10:13.255 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:python3-rados-14.2.19-116.gd23cd2d5.el7.x86_64 61/99 2021-04-09T18:10:13.428 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 43/99 2021-04-09T18:10:13.591 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 62/99 2021-04-09T18:10:13.877 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-routes-1.13-2.el7.noarch 63/99 2021-04-09T18:10:14.100 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 44/99 2021-04-09T18:10:14.444 INFO:teuthology.orchestra.run.smithi032.stdout: Updating : 2:librados2-14.2.19-116.gd23cd2d5.el7.x86_64 45/99 2021-04-09T18:10:14.608 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 64/99 2021-04-09T18:10:14.889 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:libcephfs2-14.2.19-116.gd23cd2d5.el7.x86_64 46/99 2021-04-09T18:10:14.986 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-mako-0.8.1-2.el7.noarch 65/99 2021-04-09T18:10:15.193 INFO:teuthology.orchestra.run.smithi032.stdout: Updating : 2:librbd1-14.2.19-116.gd23cd2d5.el7.x86_64 47/99 2021-04-09T18:10:15.562 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 66/99 2021-04-09T18:10:15.915 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:python-rados-14.2.19-116.gd23cd2d5.el7.x86_64 48/99 2021-04-09T18:10:15.965 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 67/99 2021-04-09T18:10:16.192 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:librgw2-14.2.19-116.gd23cd2d5.el7.x86_64 49/99 2021-04-09T18:10:16.305 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python2-apipkg-1.5-5.el7.noarch 68/99 2021-04-09T18:10:16.443 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:python-rgw-14.2.19-116.gd23cd2d5.el7.x86_64 50/99 2021-04-09T18:10:16.688 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 69/99 2021-04-09T18:10:17.024 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 70/99 2021-04-09T18:10:17.035 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:python-rbd-14.2.19-116.gd23cd2d5.el7.x86_64 51/99 2021-04-09T18:10:17.267 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:python-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 52/99 2021-04-09T18:10:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 71/99 2021-04-09T18:10:17.634 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 72/99 2021-04-09T18:10:17.942 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libselinux-devel-2.5-15.el7.x86_64 73/99 2021-04-09T18:10:18.227 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : krb5-devel-1.15.1-50.el7.x86_64 74/99 2021-04-09T18:10:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:libradosstriper1-14.2.19-116.gd23cd2d5.el7.x86_64 53/99 2021-04-09T18:10:19.823 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 75/99 2021-04-09T18:10:20.556 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : scipy-0.12.1-6.el7.x86_64 76/99 2021-04-09T18:10:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 54/99 2021-04-09T18:10:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-base-14.2.19-116.gd23cd2d5.el7.x86_64 55/99 2021-04-09T18:10:21.154 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.19-116.gd23cd2d5.el 77/99 2021-04-09T18:10:21.463 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 78/99 2021-04-09T18:10:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-mgr-ssh-14.2.19-116.gd23cd2d5.el7.noarch 79/99 2021-04-09T18:10:22.610 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 80/99 2021-04-09T18:10:23.003 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-mgr-dashboard-14.2.19-116.gd23cd2d5.el7.noarch 81/99 2021-04-09T18:10:23.286 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-mgr-rook-14.2.19-116.gd23cd2d5.el7.noarch 82/99 2021-04-09T18:10:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.19-116.gd23cd2d5.el 83/99 2021-04-09T18:10:23.829 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:python3-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 84/99 2021-04-09T18:10:24.591 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:libcephfs-devel-14.2.19-116.gd23cd2d5.el7.x86_64 85/99 2021-04-09T18:10:28.314 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 86/99 2021-04-09T18:10:28.565 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 87/99 2021-04-09T18:10:28.814 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 88/99 2021-04-09T18:10:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:rbd-fuse-14.2.19-116.gd23cd2d5.el7.x86_64 89/99 2021-04-09T18:10:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : 2:ceph-fuse-14.2.19-116.gd23cd2d5.el7.x86_64 90/99 2021-04-09T18:10:30.279 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : elfutils-libelf-devel-0.176-5.el7.x86_64 91/99 2021-04-09T18:10:30.664 INFO:teuthology.orchestra.run.smithi086.stdout: Updating : 1:openssl-1.0.2k-21.el7_9.x86_64 92/99 2021-04-09T18:10:31.124 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flex-2.5.37-6.el7.x86_64 93/99 2021-04-09T18:10:31.424 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : bison-3.0.4-2.el7.x86_64 94/99 2021-04-09T18:10:31.726 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : 1:openssl-1.0.2k-19.el7.x86_64 95/99 2021-04-09T18:10:31.744 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-selinux-14.2.19-116.gd23cd2d5.el7.x86_64 56/99 2021-04-09T18:10:32.065 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : 1:openssl-libs-1.0.2k-19.el7.x86_64 96/99 2021-04-09T18:10:32.318 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 97/99 2021-04-09T18:10:32.614 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 98/99 2021-04-09T18:10:32.643 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-osd-14.2.19-116.gd23cd2d5.el7.x86_64 57/99 2021-04-09T18:10:33.096 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : zlib-1.2.7-18.el7.x86_64 99/99 2021-04-09T18:10:33.096 INFO:teuthology.orchestra.run.smithi086.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2021-04-09T18:10:33.097 INFO:teuthology.orchestra.run.smithi086.stdout: : manager 2021-04-09T18:10:33.105 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-mon-14.2.19-116.gd23cd2d5.el7.x86_64 58/99 2021-04-09T18:10:33.413 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-mds-14.2.19-116.gd23cd2d5.el7.x86_64 59/99 2021-04-09T18:10:33.678 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:librados-devel-14.2.19-116.gd23cd2d5.el7.x86_64 60/99 2021-04-09T18:10:33.844 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-fuse-14.2.19-116.gd23cd2d5.el7.x86_64 1/99 2021-04-09T18:10:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 2/99 2021-04-09T18:10:33.953 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:python3-rados-14.2.19-116.gd23cd2d5.el7.x86_64 61/99 2021-04-09T18:10:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:libradosstriper1-14.2.19-116.gd23cd2d5.el7.x86_64 3/99 2021-04-09T18:10:34.186 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-2.el7.x86_64 4/99 2021-04-09T18:10:34.293 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 62/99 2021-04-09T18:10:34.295 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:python-ceph-argparse-14.2.19-116.gd23cd2d5.el7.x86_64 5/99 2021-04-09T18:10:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-routes-1.13-2.el7.noarch 6/99 2021-04-09T18:10:34.565 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-routes-1.13-2.el7.noarch 63/99 2021-04-09T18:10:34.890 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:librgw2-14.2.19-116.gd23cd2d5.el7.x86_64 7/99 2021-04-09T18:10:34.891 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 64/99 2021-04-09T18:10:34.999 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 8/99 2021-04-09T18:10:35.120 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 9/99 2021-04-09T18:10:35.216 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 10/99 2021-04-09T18:10:35.245 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-mako-0.8.1-2.el7.noarch 65/99 2021-04-09T18:10:35.319 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 11/99 2021-04-09T18:10:35.443 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 12/99 2021-04-09T18:10:35.551 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-osd-14.2.19-116.gd23cd2d5.el7.x86_64 13/99 2021-04-09T18:10:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 14/99 2021-04-09T18:10:35.777 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:librados2-14.2.19-116.gd23cd2d5.el7.x86_64 15/99 2021-04-09T18:10:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 66/99 2021-04-09T18:10:35.900 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:openssl-1.0.2k-21.el7_9.x86_64 16/99 2021-04-09T18:10:36.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python2-apipkg-1.5-5.el7.noarch 17/99 2021-04-09T18:10:36.128 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 18/99 2021-04-09T18:10:36.199 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 67/99 2021-04-09T18:10:36.233 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 19/99 2021-04-09T18:10:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 20/99 2021-04-09T18:10:36.488 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 21/99 2021-04-09T18:10:36.557 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python2-apipkg-1.5-5.el7.noarch 68/99 2021-04-09T18:10:36.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 22/99 2021-04-09T18:10:36.714 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:librados-devel-14.2.19-116.gd23cd2d5.el7.x86_64 23/99 2021-04-09T18:10:36.832 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-requests-2.6.0-10.el7.noarch 24/99 2021-04-09T18:10:36.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 25/99 2021-04-09T18:10:37.066 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 26/99 2021-04-09T18:10:37.106 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 69/99 2021-04-09T18:10:37.166 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:python2-kubernetes-11.0.0-6.el7.noarch 27/99 2021-04-09T18:10:37.276 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 28/99 2021-04-09T18:10:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.19-116.gd23cd2d5.el 29/99 2021-04-09T18:10:37.415 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 70/99 2021-04-09T18:10:37.509 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 30/99 2021-04-09T18:10:37.627 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 31/99 2021-04-09T18:10:37.663 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 71/99 2021-04-09T18:10:37.744 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 32/99 2021-04-09T18:10:37.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 33/99 2021-04-09T18:10:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:python-rbd-14.2.19-116.gd23cd2d5.el7.x86_64 34/99 2021-04-09T18:10:37.993 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 72/99 2021-04-09T18:10:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 35/99 2021-04-09T18:10:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 36/99 2021-04-09T18:10:38.299 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libselinux-devel-2.5-15.el7.x86_64 73/99 2021-04-09T18:10:38.338 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 37/99 2021-04-09T18:10:38.455 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:libcephfs2-14.2.19-116.gd23cd2d5.el7.x86_64 38/99 2021-04-09T18:10:38.573 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 39/99 2021-04-09T18:10:38.602 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : krb5-devel-1.15.1-50.el7.x86_64 74/99 2021-04-09T18:10:38.692 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python2-rsa-3.4.2-2.el7.noarch 40/99 2021-04-09T18:10:38.807 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 41/99 2021-04-09T18:10:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 42/99 2021-04-09T18:10:39.024 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 43/99 2021-04-09T18:10:39.125 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-libelf-devel-0.176-5.el7.x86_64 44/99 2021-04-09T18:10:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.19-116.gd23cd2d5.el7.noarch 45/99 2021-04-09T18:10:39.376 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-base-14.2.19-116.gd23cd2d5.el7.x86_64 46/99 2021-04-09T18:10:39.493 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 47/99 2021-04-09T18:10:39.643 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:libcephfs-devel-14.2.19-116.gd23cd2d5.el7.x86_64 48/99 2021-04-09T18:10:39.774 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:python-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 49/99 2021-04-09T18:10:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:python-rgw-14.2.19-116.gd23cd2d5.el7.x86_64 50/99 2021-04-09T18:10:39.988 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.8.0-3.el7.x86_64 51/99 2021-04-09T18:10:40.125 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libselinux-devel-2.5-15.el7.x86_64 52/99 2021-04-09T18:10:40.246 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-1.2.7-19.el7_9.x86_64 53/99 2021-04-09T18:10:40.294 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 75/99 2021-04-09T18:10:40.363 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-mon-14.2.19-116.gd23cd2d5.el7.x86_64 54/99 2021-04-09T18:10:40.480 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 55/99 2021-04-09T18:10:40.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 56/99 2021-04-09T18:10:40.715 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:python3-ceph-argparse-14.2.19-116.gd23cd2d5.el7.x86_64 57/99 2021-04-09T18:10:40.832 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 58/99 2021-04-09T18:10:40.932 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 59/99 2021-04-09T18:10:41.047 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : scipy-0.12.1-6.el7.x86_64 76/99 2021-04-09T18:10:41.060 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:rbd-fuse-14.2.19-116.gd23cd2d5.el7.x86_64 60/99 2021-04-09T18:10:41.192 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-devel-1.2.7-19.el7_9.x86_64 61/99 2021-04-09T18:10:41.304 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 62/99 2021-04-09T18:10:41.420 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : krb5-devel-1.15.1-50.el7.x86_64 63/99 2021-04-09T18:10:41.543 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-mgr-rook-14.2.19-116.gd23cd2d5.el7.noarch 64/99 2021-04-09T18:10:41.664 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 65/99 2021-04-09T18:10:41.671 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.19-116.gd23cd2d5.el 77/99 2021-04-09T18:10:41.778 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcom_err-devel-1.42.9-19.el7.x86_64 66/99 2021-04-09T18:10:41.895 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-mgr-ssh-14.2.19-116.gd23cd2d5.el7.noarch 67/99 2021-04-09T18:10:42.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 68/99 2021-04-09T18:10:42.154 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 69/99 2021-04-09T18:10:42.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 70/99 2021-04-09T18:10:42.353 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 78/99 2021-04-09T18:10:42.372 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 71/99 2021-04-09T18:10:42.483 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 72/99 2021-04-09T18:10:42.592 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 73/99 2021-04-09T18:10:42.598 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-mgr-ssh-14.2.19-116.gd23cd2d5.el7.noarch 79/99 2021-04-09T18:10:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 74/99 2021-04-09T18:10:42.817 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.19-116.gd23cd2d5.el7.noar 75/99 2021-04-09T18:10:42.959 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:python3-rados-14.2.19-116.gd23cd2d5.el7.x86_64 76/99 2021-04-09T18:10:43.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 77/99 2021-04-09T18:10:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-selinux-14.2.19-116.gd23cd2d5.el7.x86_64 78/99 2021-04-09T18:10:43.320 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:librbd1-14.2.19-116.gd23cd2d5.el7.x86_64 79/99 2021-04-09T18:10:43.437 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 80/99 2021-04-09T18:10:43.533 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 80/99 2021-04-09T18:10:43.545 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 81/99 2021-04-09T18:10:43.662 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 82/99 2021-04-09T18:10:43.780 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.19-116.gd23cd2d5.el 83/99 2021-04-09T18:10:43.877 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-mgr-dashboard-14.2.19-116.gd23cd2d5.el7.noarch 81/99 2021-04-09T18:10:43.922 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:python-rados-14.2.19-116.gd23cd2d5.el7.x86_64 84/99 2021-04-09T18:10:44.039 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : bison-3.0.4-2.el7.x86_64 85/99 2021-04-09T18:10:44.153 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-mgr-rook-14.2.19-116.gd23cd2d5.el7.noarch 82/99 2021-04-09T18:10:44.156 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 86/99 2021-04-09T18:10:44.273 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 87/99 2021-04-09T18:10:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 88/99 2021-04-09T18:10:44.432 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.19-116.gd23cd2d5.el 83/99 2021-04-09T18:10:44.483 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 89/99 2021-04-09T18:10:44.677 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:python3-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 84/99 2021-04-09T18:10:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:libcephfs-devel-14.2.19-116.gd23cd2d5.el7.x86_64 85/99 2021-04-09T18:10:45.555 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flex-2.5.37-6.el7.x86_64 90/99 2021-04-09T18:10:45.676 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 91/99 2021-04-09T18:10:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:ceph-mds-14.2.19-116.gd23cd2d5.el7.x86_64 92/99 2021-04-09T18:10:45.902 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.8.2-1.el7.x86_64 93/99 2021-04-09T18:10:46.021 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 2:python3-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 94/99 2021-04-09T18:10:46.022 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 95/99 2021-04-09T18:10:46.023 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-1.2.7-18.el7.x86_64 96/99 2021-04-09T18:10:46.025 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 97/99 2021-04-09T18:10:46.025 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 98/99 2021-04-09T18:10:46.366 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 99/99 2021-04-09T18:10:46.367 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:10:46.367 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2021-04-09T18:10:46.368 INFO:teuthology.orchestra.run.smithi086.stdout: bison.x86_64 0:3.0.4-2.el7 2021-04-09T18:10:46.368 INFO:teuthology.orchestra.run.smithi086.stdout: ceph.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.368 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.369 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.370 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.370 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.370 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.371 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.371 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-ssh.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.372 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.373 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.373 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel.x86_64 0:0.176-5.el7 2021-04-09T18:10:46.373 INFO:teuthology.orchestra.run.smithi086.stdout: flex.x86_64 0:2.5.37-6.el7 2021-04-09T18:10:46.375 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.375 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.376 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel.x86_64 1:1.0.2k-21.el7_9 2021-04-09T18:10:46.376 INFO:teuthology.orchestra.run.smithi086.stdout: python-ceph-compat.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.376 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.377 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.377 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:10:46.378 INFO:teuthology.orchestra.run.smithi086.stdout:Dependency Installed: 2021-04-09T18:10:46.378 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.378 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.379 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.379 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.380 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.391 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.391 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.392 INFO:teuthology.orchestra.run.smithi086.stdout: jq.x86_64 0:1.6-2.el7 2021-04-09T18:10:46.392 INFO:teuthology.orchestra.run.smithi086.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2021-04-09T18:10:46.393 INFO:teuthology.orchestra.run.smithi086.stdout: krb5-devel.x86_64 0:1.15.1-50.el7 2021-04-09T18:10:46.394 INFO:teuthology.orchestra.run.smithi086.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2021-04-09T18:10:46.395 INFO:teuthology.orchestra.run.smithi086.stdout: libcom_err-devel.x86_64 0:1.42.9-19.el7 2021-04-09T18:10:46.396 INFO:teuthology.orchestra.run.smithi086.stdout: liboath.x86_64 0:2.6.2-1.el7 2021-04-09T18:10:46.396 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq.x86_64 0:0.8.0-3.el7 2021-04-09T18:10:46.398 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.398 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.399 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2021-04-09T18:10:46.399 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.400 INFO:teuthology.orchestra.run.smithi086.stdout: libselinux-devel.x86_64 0:2.5-15.el7 2021-04-09T18:10:46.400 INFO:teuthology.orchestra.run.smithi086.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2021-04-09T18:10:46.402 INFO:teuthology.orchestra.run.smithi086.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2021-04-09T18:10:46.402 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2021-04-09T18:10:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2021-04-09T18:10:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma.x86_64 0:6.8.2-1.el7 2021-04-09T18:10:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: pcre-devel.x86_64 0:8.32-17.el7 2021-04-09T18:10:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: python-beaker.noarch 0:1.5.4-10.el7 2021-04-09T18:10:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2021-04-09T18:10:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: python-ceph-argparse.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: python-cephfs.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2021-04-09T18:10:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: python-execnet.noarch 0:1.7.1-3.el7 2021-04-09T18:10:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2021-04-09T18:10:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: python-mako.noarch 0:0.8.1-2.el7 2021-04-09T18:10:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2021-04-09T18:10:46.417 INFO:teuthology.orchestra.run.smithi086.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2021-04-09T18:10:46.417 INFO:teuthology.orchestra.run.smithi086.stdout: python-pecan.noarch 0:0.4.5-2.el7 2021-04-09T18:10:46.417 INFO:teuthology.orchestra.run.smithi086.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2021-04-09T18:10:46.418 INFO:teuthology.orchestra.run.smithi086.stdout: python-rados.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.418 INFO:teuthology.orchestra.run.smithi086.stdout: python-rbd.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.419 INFO:teuthology.orchestra.run.smithi086.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2021-04-09T18:10:46.419 INFO:teuthology.orchestra.run.smithi086.stdout: python-requests.noarch 0:2.6.0-10.el7 2021-04-09T18:10:46.420 INFO:teuthology.orchestra.run.smithi086.stdout: python-rgw.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.420 INFO:teuthology.orchestra.run.smithi086.stdout: python-routes.noarch 0:1.13-2.el7 2021-04-09T18:10:46.420 INFO:teuthology.orchestra.run.smithi086.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2021-04-09T18:10:46.421 INFO:teuthology.orchestra.run.smithi086.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2021-04-09T18:10:46.426 INFO:teuthology.orchestra.run.smithi086.stdout: python-tempita.noarch 0:0.5.1-6.el7 2021-04-09T18:10:46.427 INFO:teuthology.orchestra.run.smithi086.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2021-04-09T18:10:46.427 INFO:teuthology.orchestra.run.smithi086.stdout: python-webob.noarch 0:1.2.3-7.el7 2021-04-09T18:10:46.428 INFO:teuthology.orchestra.run.smithi086.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2021-04-09T18:10:46.428 INFO:teuthology.orchestra.run.smithi086.stdout: python-webtest.noarch 0:1.3.4-6.el7 2021-04-09T18:10:46.431 INFO:teuthology.orchestra.run.smithi086.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2021-04-09T18:10:46.434 INFO:teuthology.orchestra.run.smithi086.stdout: python2-apipkg.noarch 0:1.5-5.el7 2021-04-09T18:10:46.434 INFO:teuthology.orchestra.run.smithi086.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2021-04-09T18:10:46.435 INFO:teuthology.orchestra.run.smithi086.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2021-04-09T18:10:46.435 INFO:teuthology.orchestra.run.smithi086.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2021-04-09T18:10:46.436 INFO:teuthology.orchestra.run.smithi086.stdout: python2-kubernetes.noarch 1:11.0.0-6.el7 2021-04-09T18:10:46.438 INFO:teuthology.orchestra.run.smithi086.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2021-04-09T18:10:46.439 INFO:teuthology.orchestra.run.smithi086.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2021-04-09T18:10:46.439 INFO:teuthology.orchestra.run.smithi086.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2021-04-09T18:10:46.440 INFO:teuthology.orchestra.run.smithi086.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2021-04-09T18:10:46.440 INFO:teuthology.orchestra.run.smithi086.stdout: python2-rsa.noarch 0:3.4.2-2.el7 2021-04-09T18:10:46.440 INFO:teuthology.orchestra.run.smithi086.stdout: python2-six.noarch 0:1.9.0-0.el7 2021-04-09T18:10:46.441 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.441 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.442 INFO:teuthology.orchestra.run.smithi086.stdout: scipy.x86_64 0:0.12.1-6.el7 2021-04-09T18:10:46.442 INFO:teuthology.orchestra.run.smithi086.stdout: socat.x86_64 0:1.7.3.2-2.el7 2021-04-09T18:10:46.443 INFO:teuthology.orchestra.run.smithi086.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2021-04-09T18:10:46.443 INFO:teuthology.orchestra.run.smithi086.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2021-04-09T18:10:46.443 INFO:teuthology.orchestra.run.smithi086.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2021-04-09T18:10:46.444 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel.x86_64 0:1.2.7-19.el7_9 2021-04-09T18:10:46.444 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:10:46.445 INFO:teuthology.orchestra.run.smithi086.stdout:Updated: 2021-04-09T18:10:46.445 INFO:teuthology.orchestra.run.smithi086.stdout: librados2.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.445 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:10:46.446 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:10:46.446 INFO:teuthology.orchestra.run.smithi086.stdout:Dependency Updated: 2021-04-09T18:10:46.447 INFO:teuthology.orchestra.run.smithi086.stdout: openssl.x86_64 1:1.0.2k-21.el7_9 openssl-libs.x86_64 1:1.0.2k-21.el7_9 2021-04-09T18:10:46.448 INFO:teuthology.orchestra.run.smithi086.stdout: zlib.x86_64 0:1.2.7-19.el7_9 2021-04-09T18:10:46.472 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:10:46.474 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2021-04-09T18:10:46.560 DEBUG:teuthology.parallel:result is None 2021-04-09T18:10:49.214 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 86/99 2021-04-09T18:10:49.487 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 87/99 2021-04-09T18:10:49.694 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 88/99 2021-04-09T18:10:49.992 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:rbd-fuse-14.2.19-116.gd23cd2d5.el7.x86_64 89/99 2021-04-09T18:10:50.249 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : 2:ceph-fuse-14.2.19-116.gd23cd2d5.el7.x86_64 90/99 2021-04-09T18:10:50.600 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : elfutils-libelf-devel-0.176-5.el7.x86_64 91/99 2021-04-09T18:10:50.977 INFO:teuthology.orchestra.run.smithi032.stdout: Updating : 1:openssl-1.0.2k-21.el7_9.x86_64 92/99 2021-04-09T18:10:51.376 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flex-2.5.37-6.el7.x86_64 93/99 2021-04-09T18:10:51.703 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : bison-3.0.4-2.el7.x86_64 94/99 2021-04-09T18:10:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : 1:openssl-1.0.2k-19.el7.x86_64 95/99 2021-04-09T18:10:53.385 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : 1:openssl-libs-1.0.2k-19.el7.x86_64 96/99 2021-04-09T18:10:53.642 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 97/99 2021-04-09T18:10:53.875 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 98/99 2021-04-09T18:10:54.426 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : zlib-1.2.7-18.el7.x86_64 99/99 2021-04-09T18:10:54.426 INFO:teuthology.orchestra.run.smithi032.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2021-04-09T18:10:54.426 INFO:teuthology.orchestra.run.smithi032.stdout: : manager 2021-04-09T18:10:55.258 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-fuse-14.2.19-116.gd23cd2d5.el7.x86_64 1/99 2021-04-09T18:10:55.375 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 2/99 2021-04-09T18:10:55.501 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:libradosstriper1-14.2.19-116.gd23cd2d5.el7.x86_64 3/99 2021-04-09T18:10:55.643 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : jq-1.6-2.el7.x86_64 4/99 2021-04-09T18:10:55.768 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:python-ceph-argparse-14.2.19-116.gd23cd2d5.el7.x86_64 5/99 2021-04-09T18:10:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-routes-1.13-2.el7.noarch 6/99 2021-04-09T18:10:56.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:librgw2-14.2.19-116.gd23cd2d5.el7.x86_64 7/99 2021-04-09T18:10:56.128 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 8/99 2021-04-09T18:10:56.229 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 9/99 2021-04-09T18:10:56.329 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 10/99 2021-04-09T18:10:56.430 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 11/99 2021-04-09T18:10:56.539 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 12/99 2021-04-09T18:10:56.656 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-osd-14.2.19-116.gd23cd2d5.el7.x86_64 13/99 2021-04-09T18:10:56.773 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-mgr-14.2.19-116.gd23cd2d5.el7.x86_64 14/99 2021-04-09T18:10:56.890 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:librados2-14.2.19-116.gd23cd2d5.el7.x86_64 15/99 2021-04-09T18:10:57.015 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:openssl-1.0.2k-21.el7_9.x86_64 16/99 2021-04-09T18:10:57.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python2-apipkg-1.5-5.el7.noarch 17/99 2021-04-09T18:10:57.308 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 18/99 2021-04-09T18:10:57.425 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 19/99 2021-04-09T18:10:57.567 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 20/99 2021-04-09T18:10:57.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:python-ceph-compat-14.2.19-116.gd23cd2d5.el7.x86_64 21/99 2021-04-09T18:10:57.811 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 22/99 2021-04-09T18:10:57.936 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:librados-devel-14.2.19-116.gd23cd2d5.el7.x86_64 23/99 2021-04-09T18:10:58.061 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-requests-2.6.0-10.el7.noarch 24/99 2021-04-09T18:10:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 25/99 2021-04-09T18:10:58.271 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 26/99 2021-04-09T18:10:58.363 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:python2-kubernetes-11.0.0-6.el7.noarch 27/99 2021-04-09T18:10:58.463 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 28/99 2021-04-09T18:10:58.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.19-116.gd23cd2d5.el 29/99 2021-04-09T18:10:58.689 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 30/99 2021-04-09T18:10:58.815 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 31/99 2021-04-09T18:10:58.958 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 32/99 2021-04-09T18:10:59.082 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 33/99 2021-04-09T18:10:59.208 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:python-rbd-14.2.19-116.gd23cd2d5.el7.x86_64 34/99 2021-04-09T18:10:59.333 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-14.2.19-116.gd23cd2d5.el7.x86_64 35/99 2021-04-09T18:10:59.461 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 36/99 2021-04-09T18:10:59.562 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 37/99 2021-04-09T18:10:59.662 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:libcephfs2-14.2.19-116.gd23cd2d5.el7.x86_64 38/99 2021-04-09T18:10:59.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 39/99 2021-04-09T18:10:59.855 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python2-rsa-3.4.2-2.el7.noarch 40/99 2021-04-09T18:10:59.964 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 41/99 2021-04-09T18:11:00.081 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 42/99 2021-04-09T18:11:00.198 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 43/99 2021-04-09T18:11:00.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-devel-0.176-5.el7.x86_64 44/99 2021-04-09T18:11:00.441 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.19-116.gd23cd2d5.el7.noarch 45/99 2021-04-09T18:11:00.591 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-base-14.2.19-116.gd23cd2d5.el7.x86_64 46/99 2021-04-09T18:11:00.717 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 47/99 2021-04-09T18:11:00.842 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:libcephfs-devel-14.2.19-116.gd23cd2d5.el7.x86_64 48/99 2021-04-09T18:11:00.967 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:python-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 49/99 2021-04-09T18:11:01.135 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:python-rgw-14.2.19-116.gd23cd2d5.el7.x86_64 50/99 2021-04-09T18:11:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.8.0-3.el7.x86_64 51/99 2021-04-09T18:11:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libselinux-devel-2.5-15.el7.x86_64 52/99 2021-04-09T18:11:01.486 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : zlib-1.2.7-19.el7_9.x86_64 53/99 2021-04-09T18:11:01.570 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-mon-14.2.19-116.gd23cd2d5.el7.x86_64 54/99 2021-04-09T18:11:01.663 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 55/99 2021-04-09T18:11:01.780 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 56/99 2021-04-09T18:11:01.897 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:python3-ceph-argparse-14.2.19-116.gd23cd2d5.el7.x86_64 57/99 2021-04-09T18:11:02.014 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 58/99 2021-04-09T18:11:02.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 59/99 2021-04-09T18:11:02.282 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:rbd-fuse-14.2.19-116.gd23cd2d5.el7.x86_64 60/99 2021-04-09T18:11:02.416 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : zlib-devel-1.2.7-19.el7_9.x86_64 61/99 2021-04-09T18:11:03.033 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 62/99 2021-04-09T18:11:03.159 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : krb5-devel-1.15.1-50.el7.x86_64 63/99 2021-04-09T18:11:03.288 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-mgr-rook-14.2.19-116.gd23cd2d5.el7.noarch 64/99 2021-04-09T18:11:03.410 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 65/99 2021-04-09T18:11:03.535 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcom_err-devel-1.42.9-19.el7.x86_64 66/99 2021-04-09T18:11:03.644 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-mgr-ssh-14.2.19-116.gd23cd2d5.el7.noarch 67/99 2021-04-09T18:11:03.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-test-14.2.19-116.gd23cd2d5.el7.x86_64 68/99 2021-04-09T18:11:03.865 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 69/99 2021-04-09T18:11:03.971 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 70/99 2021-04-09T18:11:04.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 71/99 2021-04-09T18:11:04.214 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-common-14.2.19-116.gd23cd2d5.el7.x86_64 72/99 2021-04-09T18:11:04.331 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 73/99 2021-04-09T18:11:04.448 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 74/99 2021-04-09T18:11:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.19-116.gd23cd2d5.el7.noar 75/99 2021-04-09T18:11:04.699 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:python3-rados-14.2.19-116.gd23cd2d5.el7.x86_64 76/99 2021-04-09T18:11:04.835 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 77/99 2021-04-09T18:11:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-selinux-14.2.19-116.gd23cd2d5.el7.x86_64 78/99 2021-04-09T18:11:05.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:librbd1-14.2.19-116.gd23cd2d5.el7.x86_64 79/99 2021-04-09T18:11:05.209 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 80/99 2021-04-09T18:11:05.330 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 81/99 2021-04-09T18:11:05.444 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 82/99 2021-04-09T18:11:05.570 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.19-116.gd23cd2d5.el 83/99 2021-04-09T18:11:05.653 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:python-rados-14.2.19-116.gd23cd2d5.el7.x86_64 84/99 2021-04-09T18:11:05.754 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : bison-3.0.4-2.el7.x86_64 85/99 2021-04-09T18:11:05.896 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 86/99 2021-04-09T18:11:06.013 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 87/99 2021-04-09T18:11:06.131 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 88/99 2021-04-09T18:11:06.248 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-radosgw-14.2.19-116.gd23cd2d5.el7.x86_64 89/99 2021-04-09T18:11:06.373 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flex-2.5.37-6.el7.x86_64 90/99 2021-04-09T18:11:06.499 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 91/99 2021-04-09T18:11:06.624 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:ceph-mds-14.2.19-116.gd23cd2d5.el7.x86_64 92/99 2021-04-09T18:11:06.750 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : oniguruma-6.8.2-1.el7.x86_64 93/99 2021-04-09T18:11:06.875 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 2:python3-cephfs-14.2.19-116.gd23cd2d5.el7.x86_64 94/99 2021-04-09T18:11:06.876 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 95/99 2021-04-09T18:11:06.876 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : zlib-1.2.7-18.el7.x86_64 96/99 2021-04-09T18:11:06.877 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 97/99 2021-04-09T18:11:06.877 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 98/99 2021-04-09T18:11:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 99/99 2021-04-09T18:11:07.266 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:07.266 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2021-04-09T18:11:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: bison.x86_64 0:3.0.4-2.el7 2021-04-09T18:11:07.267 INFO:teuthology.orchestra.run.smithi032.stdout: ceph.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.268 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.269 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.270 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-ssh.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.271 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel.x86_64 0:0.176-5.el7 2021-04-09T18:11:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: flex.x86_64 0:2.5.37-6.el7 2021-04-09T18:11:07.272 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.273 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel.x86_64 1:1.0.2k-21.el7_9 2021-04-09T18:11:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: python-ceph-compat.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.274 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.275 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:07.275 INFO:teuthology.orchestra.run.smithi032.stdout:Dependency Installed: 2021-04-09T18:11:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.276 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards.noarch 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.277 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.278 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: jq.x86_64 0:1.6-2.el7 2021-04-09T18:11:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2021-04-09T18:11:07.304 INFO:teuthology.orchestra.run.smithi032.stdout: krb5-devel.x86_64 0:1.15.1-50.el7 2021-04-09T18:11:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2021-04-09T18:11:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: libcom_err-devel.x86_64 0:1.42.9-19.el7 2021-04-09T18:11:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: liboath.x86_64 0:2.6.2-1.el7 2021-04-09T18:11:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq.x86_64 0:0.8.0-3.el7 2021-04-09T18:11:07.305 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2021-04-09T18:11:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.306 INFO:teuthology.orchestra.run.smithi032.stdout: libselinux-devel.x86_64 0:2.5-15.el7 2021-04-09T18:11:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2021-04-09T18:11:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2021-04-09T18:11:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2021-04-09T18:11:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2021-04-09T18:11:07.307 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma.x86_64 0:6.8.2-1.el7 2021-04-09T18:11:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: pcre-devel.x86_64 0:8.32-17.el7 2021-04-09T18:11:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: python-beaker.noarch 0:1.5.4-10.el7 2021-04-09T18:11:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2021-04-09T18:11:07.308 INFO:teuthology.orchestra.run.smithi032.stdout: python-ceph-argparse.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: python-cephfs.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2021-04-09T18:11:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: python-execnet.noarch 0:1.7.1-3.el7 2021-04-09T18:11:07.309 INFO:teuthology.orchestra.run.smithi032.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2021-04-09T18:11:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: python-mako.noarch 0:0.8.1-2.el7 2021-04-09T18:11:07.310 INFO:teuthology.orchestra.run.smithi032.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2021-04-09T18:11:07.337 INFO:teuthology.orchestra.run.smithi032.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2021-04-09T18:11:07.338 INFO:teuthology.orchestra.run.smithi032.stdout: python-pecan.noarch 0:0.4.5-2.el7 2021-04-09T18:11:07.338 INFO:teuthology.orchestra.run.smithi032.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2021-04-09T18:11:07.339 INFO:teuthology.orchestra.run.smithi032.stdout: python-rados.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.339 INFO:teuthology.orchestra.run.smithi032.stdout: python-rbd.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.340 INFO:teuthology.orchestra.run.smithi032.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2021-04-09T18:11:07.340 INFO:teuthology.orchestra.run.smithi032.stdout: python-requests.noarch 0:2.6.0-10.el7 2021-04-09T18:11:07.341 INFO:teuthology.orchestra.run.smithi032.stdout: python-rgw.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.341 INFO:teuthology.orchestra.run.smithi032.stdout: python-routes.noarch 0:1.13-2.el7 2021-04-09T18:11:07.341 INFO:teuthology.orchestra.run.smithi032.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2021-04-09T18:11:07.342 INFO:teuthology.orchestra.run.smithi032.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2021-04-09T18:11:07.342 INFO:teuthology.orchestra.run.smithi032.stdout: python-tempita.noarch 0:0.5.1-6.el7 2021-04-09T18:11:07.343 INFO:teuthology.orchestra.run.smithi032.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2021-04-09T18:11:07.343 INFO:teuthology.orchestra.run.smithi032.stdout: python-webob.noarch 0:1.2.3-7.el7 2021-04-09T18:11:07.346 INFO:teuthology.orchestra.run.smithi032.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2021-04-09T18:11:07.346 INFO:teuthology.orchestra.run.smithi032.stdout: python-webtest.noarch 0:1.3.4-6.el7 2021-04-09T18:11:07.346 INFO:teuthology.orchestra.run.smithi032.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2021-04-09T18:11:07.347 INFO:teuthology.orchestra.run.smithi032.stdout: python2-apipkg.noarch 0:1.5-5.el7 2021-04-09T18:11:07.349 INFO:teuthology.orchestra.run.smithi032.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2021-04-09T18:11:07.381 INFO:teuthology.orchestra.run.smithi032.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2021-04-09T18:11:07.381 INFO:teuthology.orchestra.run.smithi032.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2021-04-09T18:11:07.381 INFO:teuthology.orchestra.run.smithi032.stdout: python2-kubernetes.noarch 1:11.0.0-6.el7 2021-04-09T18:11:07.384 INFO:teuthology.orchestra.run.smithi032.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2021-04-09T18:11:07.384 INFO:teuthology.orchestra.run.smithi032.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2021-04-09T18:11:07.385 INFO:teuthology.orchestra.run.smithi032.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2021-04-09T18:11:07.385 INFO:teuthology.orchestra.run.smithi032.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2021-04-09T18:11:07.395 INFO:teuthology.orchestra.run.smithi032.stdout: python2-rsa.noarch 0:3.4.2-2.el7 2021-04-09T18:11:07.395 INFO:teuthology.orchestra.run.smithi032.stdout: python2-six.noarch 0:1.9.0-0.el7 2021-04-09T18:11:07.396 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.396 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.396 INFO:teuthology.orchestra.run.smithi032.stdout: scipy.x86_64 0:0.12.1-6.el7 2021-04-09T18:11:07.397 INFO:teuthology.orchestra.run.smithi032.stdout: socat.x86_64 0:1.7.3.2-2.el7 2021-04-09T18:11:07.397 INFO:teuthology.orchestra.run.smithi032.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2021-04-09T18:11:07.398 INFO:teuthology.orchestra.run.smithi032.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2021-04-09T18:11:07.398 INFO:teuthology.orchestra.run.smithi032.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2021-04-09T18:11:07.398 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel.x86_64 0:1.2.7-19.el7_9 2021-04-09T18:11:07.399 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:07.399 INFO:teuthology.orchestra.run.smithi032.stdout:Updated: 2021-04-09T18:11:07.400 INFO:teuthology.orchestra.run.smithi032.stdout: librados2.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.400 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1.x86_64 2:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.400 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:07.401 INFO:teuthology.orchestra.run.smithi032.stdout:Dependency Updated: 2021-04-09T18:11:07.401 INFO:teuthology.orchestra.run.smithi032.stdout: openssl.x86_64 1:1.0.2k-21.el7_9 openssl-libs.x86_64 1:1.0.2k-21.el7_9 2021-04-09T18:11:07.402 INFO:teuthology.orchestra.run.smithi032.stdout: zlib.x86_64 0:1.2.7-19.el7_9 2021-04-09T18:11:07.402 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:07.402 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2021-04-09T18:11:07.473 DEBUG:teuthology.parallel:result is None 2021-04-09T18:11:07.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo lsb_release -is 2021-04-09T18:11:07.572 INFO:teuthology.orchestra.run.smithi032.stdout:RedHatEnterpriseServer 2021-04-09T18:11:07.575 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-04-09T18:11:07.576 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=d23cd2d5f486487b0b80655c4e33344f0f97b82b 2021-04-09T18:11:07.732 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-04-09T18:11:07.801 INFO:teuthology.orchestra.run.smithi032.stdout:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.802 INFO:teuthology.packaging:The installed version of ceph is 14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:07.802 INFO:teuthology.task.install:The correct ceph version 14.2.19-116.gd23cd2d5 is installed. 2021-04-09T18:11:07.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo lsb_release -is 2021-04-09T18:11:07.911 INFO:teuthology.orchestra.run.smithi086.stdout:RedHatEnterpriseServer 2021-04-09T18:11:07.913 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-04-09T18:11:07.914 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=d23cd2d5f486487b0b80655c4e33344f0f97b82b 2021-04-09T18:11:08.093 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-04-09T18:11:08.171 INFO:teuthology.orchestra.run.smithi086.stdout:14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:08.171 INFO:teuthology.packaging:The installed version of ceph is 14.2.19-116.gd23cd2d5.el7 2021-04-09T18:11:08.172 INFO:teuthology.task.install:The correct ceph version 14.2.19-116.gd23cd2d5 is installed. 2021-04-09T18:11:08.173 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-04-09T18:11:08.173 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2021-04-09T18:11:08.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-04-09T18:11:08.239 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2021-04-09T18:11:08.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-04-09T18:11:08.295 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-04-09T18:11:08.296 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2021-04-09T18:11:08.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2021-04-09T18:11:08.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-04-09T18:11:08.451 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2021-04-09T18:11:08.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2021-04-09T18:11:08.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-04-09T18:11:08.599 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-04-09T18:11:08.600 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2021-04-09T18:11:08.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2021-04-09T18:11:08.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-04-09T18:11:08.756 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2021-04-09T18:11:08.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2021-04-09T18:11:08.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-04-09T18:11:08.907 INFO:teuthology.run_tasks:Running task ceph... 2021-04-09T18:11:09.147 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-04-09T18:11:09.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2021-04-09T18:11:09.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2021-04-09T18:11:09.210 INFO:tasks.ceph:Disabling ceph logrotate... 2021-04-09T18:11:09.211 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/logrotate.d/ceph 2021-04-09T18:11:09.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2021-04-09T18:11:09.269 INFO:tasks.ceph:Creating extra log directories... 2021-04-09T18:11:09.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-04-09T18:11:09.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-04-09T18:11:09.329 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2021-04-09T18:11:09.330 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2021-04-09T18:11:09.330 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2021-04-09T18:11:09.332 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2021-04-09T18:11:09.332 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-04-09T18:11:09.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-04-09T18:11:25.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2021-04-09T18:11:25.669 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2021-04-09T18:11:25.669 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-04-09T18:11:25.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-04-09T18:11:26.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2021-04-09T18:11:26.412 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-04-09T18:11:26.412 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(TOO_FEW_PGS\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'not responding, replacing', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-04-09T18:11:26.413 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2021-04-09_17:56:50-fs-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi/6031712', 'branch': 'wip-yuri2-testing-2021-04-09-0846-nautilus', 'description': 'fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/failover}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '6031712', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2021-04-09_17:56:50-fs-wip-yuri2-testing-2021-04-09-0846-nautilus-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '7.9', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2021-04-09-0846-nautilus'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(TOO_FEW_PGS\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'not responding, replacing', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'disabled': True}, 'install': {'ceph': {'sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri2-testing-2021-04-09-0846-nautilus', 'sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-yuri2-testing-2021-04-09-0846-nautilus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d23cd2d5f486487b0b80655c4e33344f0f97b82b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b', 'targets': {'smithi032.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDujfJq+j0JX9Rzb2kuSh2IZbYDMq9JMv3AOKD8r1QVfJqzOO6LpTyrHoh8smN/f5VT9aogti+FIJ4lfoJtfTGOW/ZC1ynPjqlHCPXQAQU3HMzdLXgEd/I/36vFU6fVzGxCh0UFq1lFZMnZFAu/jIPQkGV4HBU9Ek0tmcXm3dZY+GoJxaDCSJkabJC1WOg0TQ1kFLJCLRMych4FAg6CTsWk/ywsUs42UzmZ7EFe53r+Xi9f/Q41sRJqPoTrAUH+ANrnPBmkKN0wSuEUn8cAYJhc4A1Y/cTrDnq2+y9dTaurM3bPCfjfAkpMAClav/N+Ufv1/vhLOfKyU+Yw2AzZx63X', 'smithi086.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCogA8ikymy+H5SbjPokhU/7Hk4dZ64hMy9e+2+A0iS9JeFAqfiavATu959P0kxZqnkvaNIDJixrpIZbP2Ekr7J5Dmwn6FoKAfP7WkW+AgU76b0AnS7w2wPMwBr05gytG+2/5l684BL6ebwOdqv2eoQ0q9ElB1XByvUNnOzZIEE3enw1voggnafNFgaYge8ShAmDiqOc9AON6i/Our357ueXcQr7hEsidHoxyn81S0Jczp5qSqpZX8JW6y0IhD1kq1trQUFUQ41zO/TbD92AEZzTSyIgVCLkWW8RiWfenTf18y67MLVsq1jArScgU2E1c/azpstkFv1I4ji/22dbfQd'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'sha1': 'd23cd2d5f486487b0b80655c4e33344f0f97b82b'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_failover']}}], 'teuthology_branch': 'master', 'teuthology_sha1': '85d61eae4759f46ce21e9a37cd816a7a1a66c9d5', 'timestamp': '2021-04-09_17:56:50', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23935'} 2021-04-09T18:11:26.414 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-04-09T18:11:26.419 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-04-09T18:11:26.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2021-04-09T18:11:26.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2021-04-09T18:11:26.565 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2021-04-09T18:11:26.566 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2021-04-09T18:11:26.614 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-04-09T18:11:26.615 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-04-09T18:11:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-0’ 2021-04-09T18:11:26.697 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-09T18:11:26.698 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 47943 Links: 1 2021-04-09T18:11:26.698 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-09T18:11:26.698 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2021-04-09T18:11:26.699 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2021-04-09 18:10:45.648753741 +0000 2021-04-09T18:11:26.699 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2021-04-09 18:07:50.940416487 +0000 2021-04-09T18:11:26.700 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2021-04-09 18:07:50.940416487 +0000 2021-04-09T18:11:26.700 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2021-04-09T18:11:26.712 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2021-04-09T18:11:26.714 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2021-04-09T18:11:26.715 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes (512 B) copied, 0.000148524 s, 3.4 MB/s 2021-04-09T18:11:26.717 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-04-09T18:11:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-1’ 2021-04-09T18:11:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-09T18:11:26.799 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 59466 Links: 1 2021-04-09T18:11:26.799 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-09T18:11:26.799 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2021-04-09T18:11:26.800 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2021-04-09 18:10:45.648753741 +0000 2021-04-09T18:11:26.800 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2021-04-09 18:07:51.344407538 +0000 2021-04-09T18:11:26.800 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2021-04-09 18:07:51.344407538 +0000 2021-04-09T18:11:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2021-04-09T18:11:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2021-04-09T18:11:26.814 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2021-04-09T18:11:26.814 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes (512 B) copied, 0.000153922 s, 3.3 MB/s 2021-04-09T18:11:26.816 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-04-09T18:11:26.899 INFO:teuthology.orchestra.run.smithi032.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2021-04-09T18:11:26.899 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-09T18:11:26.899 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 56781 Links: 1 2021-04-09T18:11:26.900 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-09T18:11:26.900 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2021-04-09T18:11:26.901 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2021-04-09 18:10:45.648753741 +0000 2021-04-09T18:11:26.901 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2021-04-09 18:07:51.650400758 +0000 2021-04-09T18:11:26.901 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2021-04-09 18:07:51.650400758 +0000 2021-04-09T18:11:26.902 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2021-04-09T18:11:26.915 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2021-04-09T18:11:26.915 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2021-04-09T18:11:26.916 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes (512 B) copied, 0.000147878 s, 3.5 MB/s 2021-04-09T18:11:26.917 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-04-09T18:11:27.000 INFO:teuthology.orchestra.run.smithi032.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-3’ 2021-04-09T18:11:27.001 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-09T18:11:27.001 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 55257 Links: 1 2021-04-09T18:11:27.002 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-09T18:11:27.002 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2021-04-09T18:11:27.003 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2021-04-09 18:10:45.648753741 +0000 2021-04-09T18:11:27.003 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2021-04-09 18:07:51.929394577 +0000 2021-04-09T18:11:27.003 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2021-04-09 18:07:51.929394577 +0000 2021-04-09T18:11:27.004 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2021-04-09T18:11:27.017 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2021-04-09T18:11:27.017 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2021-04-09T18:11:27.018 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes (512 B) copied, 0.000158874 s, 3.2 MB/s 2021-04-09T18:11:27.019 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:27.020 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2021-04-09T18:11:27.021 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2021-04-09T18:11:27.078 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-04-09T18:11:27.078 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-04-09T18:11:27.159 INFO:teuthology.orchestra.run.smithi086.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-0’ 2021-04-09T18:11:27.159 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-09T18:11:27.159 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 51748 Links: 1 2021-04-09T18:11:27.160 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-09T18:11:27.160 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2021-04-09T18:11:27.160 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2021-04-09 18:10:24.980990844 +0000 2021-04-09T18:11:27.160 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2021-04-09 18:07:42.244278146 +0000 2021-04-09T18:11:27.161 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2021-04-09 18:07:42.244278146 +0000 2021-04-09T18:11:27.161 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2021-04-09T18:11:27.171 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2021-04-09T18:11:27.171 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2021-04-09T18:11:27.172 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes (512 B) copied, 0.000285009 s, 1.8 MB/s 2021-04-09T18:11:27.174 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-04-09T18:11:27.253 INFO:teuthology.orchestra.run.smithi086.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-1’ 2021-04-09T18:11:27.253 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-09T18:11:27.253 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 59614 Links: 1 2021-04-09T18:11:27.254 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-09T18:11:27.254 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2021-04-09T18:11:27.254 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2021-04-09 18:10:24.980990844 +0000 2021-04-09T18:11:27.254 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2021-04-09 18:07:42.511272569 +0000 2021-04-09T18:11:27.255 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2021-04-09 18:07:42.511272569 +0000 2021-04-09T18:11:27.255 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2021-04-09T18:11:27.265 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2021-04-09T18:11:27.266 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2021-04-09T18:11:27.266 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes (512 B) copied, 0.000259456 s, 2.0 MB/s 2021-04-09T18:11:27.268 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-04-09T18:11:27.350 INFO:teuthology.orchestra.run.smithi086.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2021-04-09T18:11:27.350 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-09T18:11:27.351 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 58779 Links: 1 2021-04-09T18:11:27.351 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-09T18:11:27.351 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2021-04-09T18:11:27.351 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2021-04-09 18:10:24.980990844 +0000 2021-04-09T18:11:27.352 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2021-04-09 18:07:42.782266908 +0000 2021-04-09T18:11:27.352 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2021-04-09 18:07:42.782266908 +0000 2021-04-09T18:11:27.352 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2021-04-09T18:11:27.359 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2021-04-09T18:11:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2021-04-09T18:11:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes (512 B) copied, 0.000217654 s, 2.4 MB/s 2021-04-09T18:11:27.362 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-04-09T18:11:27.442 INFO:teuthology.orchestra.run.smithi086.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-3’ 2021-04-09T18:11:27.442 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-09T18:11:27.442 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 57030 Links: 1 2021-04-09T18:11:27.443 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-09T18:11:27.443 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2021-04-09T18:11:27.443 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2021-04-09 18:10:24.980990844 +0000 2021-04-09T18:11:27.443 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2021-04-09 18:07:43.059261121 +0000 2021-04-09T18:11:27.443 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2021-04-09 18:07:43.059261121 +0000 2021-04-09T18:11:27.444 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2021-04-09T18:11:27.454 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2021-04-09T18:11:27.454 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2021-04-09T18:11:27.454 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes (512 B) copied, 0.000263361 s, 1.9 MB/s 2021-04-09T18:11:27.456 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:27.456 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-04-09T18:11:27.457 INFO:tasks.ceph:Generating config... 2021-04-09T18:11:27.458 INFO:tasks.ceph:[client] client mount timeout = 600 2021-04-09T18:11:27.458 INFO:tasks.ceph:[client] debug client = 20 2021-04-09T18:11:27.459 INFO:tasks.ceph:[client] debug ms = 1 2021-04-09T18:11:27.459 INFO:tasks.ceph:[mds] debug mds = 20 2021-04-09T18:11:27.459 INFO:tasks.ceph:[mds] debug ms = 1 2021-04-09T18:11:27.460 INFO:tasks.ceph:[mds] mds bal frag = True 2021-04-09T18:11:27.460 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2021-04-09T18:11:27.460 INFO:tasks.ceph:[mds] mds bal merge size = 5 2021-04-09T18:11:27.461 INFO:tasks.ceph:[mds] mds bal split bits = 3 2021-04-09T18:11:27.461 INFO:tasks.ceph:[mds] mds bal split size = 100 2021-04-09T18:11:27.461 INFO:tasks.ceph:[mds] mds debug frag = True 2021-04-09T18:11:27.461 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2021-04-09T18:11:27.462 INFO:tasks.ceph:[mds] mds op complaint time = 180 2021-04-09T18:11:27.462 INFO:tasks.ceph:[mds] mds verify scatter = True 2021-04-09T18:11:27.462 INFO:tasks.ceph:[mds] osd op complaint time = 180 2021-04-09T18:11:27.463 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-04-09T18:11:27.463 INFO:tasks.ceph:[mgr] debug ms = 1 2021-04-09T18:11:27.463 INFO:tasks.ceph:[mon] debug mon = 20 2021-04-09T18:11:27.464 INFO:tasks.ceph:[mon] debug ms = 1 2021-04-09T18:11:27.465 INFO:tasks.ceph:[mon] debug paxos = 20 2021-04-09T18:11:27.466 INFO:tasks.ceph:[mon] mon op complaint time = 120 2021-04-09T18:11:27.466 INFO:tasks.ceph:[osd] bdev async discard = True 2021-04-09T18:11:27.466 INFO:tasks.ceph:[osd] bdev enable discard = True 2021-04-09T18:11:27.466 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2021-04-09T18:11:27.467 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-04-09T18:11:27.467 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-04-09T18:11:27.467 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-04-09T18:11:27.468 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-04-09T18:11:27.468 INFO:tasks.ceph:[osd] debug ms = 1 2021-04-09T18:11:27.468 INFO:tasks.ceph:[osd] debug osd = 20 2021-04-09T18:11:27.468 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-04-09T18:11:27.469 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-04-09T18:11:27.469 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-04-09T18:11:27.487 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-04-09T18:11:27.492 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-04-09T18:11:27.492 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-04-09T18:11:27.492 INFO:tasks.ceph:[osd] osd op complaint time = 180 2021-04-09T18:11:27.493 INFO:tasks.ceph:Setting up mon.a... 2021-04-09T18:11:27.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-04-09T18:11:27.602 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.keyring 2021-04-09T18:11:27.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-04-09T18:11:27.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-04-09T18:11:27.761 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.32'), ('mon.b', '172.21.15.86'), ('mon.c', '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]')] 2021-04-09T18:11:27.762 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.32 --add b 172.21.15.86 --addv c '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:27.854 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:27.855 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: generated fsid cf69fbbc-37e8-4833-99b4-5ecbe4444cfb 2021-04-09T18:11:27.856 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 0 2021-04-09T18:11:27.856 INFO:teuthology.orchestra.run.smithi032.stdout:fsid cf69fbbc-37e8-4833-99b4-5ecbe4444cfb 2021-04-09T18:11:27.856 INFO:teuthology.orchestra.run.smithi032.stdout:last_changed 2021-04-09 18:11:27.836937 2021-04-09T18:11:27.857 INFO:teuthology.orchestra.run.smithi032.stdout:created 2021-04-09 18:11:27.836937 2021-04-09T18:11:27.857 INFO:teuthology.orchestra.run.smithi032.stdout:min_mon_release 0 (unknown) 2021-04-09T18:11:27.858 INFO:teuthology.orchestra.run.smithi032.stdout:0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2021-04-09T18:11:27.858 INFO:teuthology.orchestra.run.smithi032.stdout:1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2021-04-09T18:11:27.858 INFO:teuthology.orchestra.run.smithi032.stdout:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2021-04-09T18:11:27.859 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-04-09T18:11:27.862 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cf69fbbc-37e8-4833-99b4-5ecbe4444cfb... 2021-04-09T18:11:27.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-04-09T18:11:27.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-04-09T18:11:27.959 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2021-04-09T18:11:27.959 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2021-04-09T18:11:27.960 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-04-09T18:11:27.960 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2021-04-09T18:11:27.960 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.960 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2021-04-09T18:11:27.961 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.961 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2021-04-09T18:11:27.961 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.961 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2021-04-09T18:11:27.961 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2021-04-09T18:11:27.962 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.962 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2021-04-09T18:11:27.962 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2021-04-09T18:11:27.962 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.963 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2021-04-09T18:11:27.963 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2021-04-09T18:11:27.965 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.965 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2021-04-09T18:11:27.972 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.973 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2021-04-09T18:11:27.973 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2021-04-09T18:11:27.974 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2021-04-09T18:11:27.974 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2021-04-09T18:11:27.974 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2021-04-09T18:11:27.975 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2021-04-09T18:11:27.975 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2021-04-09T18:11:27.976 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-04-09T18:11:27.976 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2021-04-09T18:11:27.976 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.977 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-04-09T18:11:27.979 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.980 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2021-04-09T18:11:27.980 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.981 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2021-04-09T18:11:27.981 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.982 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2021-04-09T18:11:27.989 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2021-04-09T18:11:27.989 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2021-04-09T18:11:27.991 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.992 INFO:teuthology.orchestra.run.smithi032.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-04-09T18:11:27.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon_client_directed_command_retry = 4 2021-04-09T18:11:27.992 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790]" 2021-04-09T18:11:27.993 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = cf69fbbc-37e8-4833-99b4-5ecbe4444cfb 2021-04-09T18:11:27.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:27.994 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2021-04-09T18:11:27.994 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2021-04-09T18:11:27.994 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.016 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2021-04-09T18:11:28.017 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2021-04-09T18:11:28.017 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.018 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2021-04-09T18:11:28.018 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2021-04-09T18:11:28.019 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.019 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2021-04-09T18:11:28.020 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2021-04-09T18:11:28.020 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2021-04-09T18:11:28.020 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.021 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2021-04-09T18:11:28.021 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2021-04-09T18:11:28.022 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.022 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2021-04-09T18:11:28.023 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.054 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2021-04-09T18:11:28.055 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.055 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2021-04-09T18:11:28.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.061 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2021-04-09T18:11:28.062 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2021-04-09T18:11:28.062 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.062 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2021-04-09T18:11:28.063 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2021-04-09T18:11:28.063 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2021-04-09T18:11:28.068 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2021-04-09T18:11:28.068 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = bitmap 2021-04-09T18:11:28.073 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2021-04-09T18:11:28.074 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2021-04-09T18:11:28.074 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 20 2021-04-09T18:11:28.075 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 20 2021-04-09T18:11:28.075 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2021-04-09T18:11:28.075 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2021-04-09T18:11:28.119 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 10 2021-04-09T18:11:28.119 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2021-04-09T18:11:28.120 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2021-04-09T18:11:28.120 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2021-04-09T18:11:28.121 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2021-04-09T18:11:28.121 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2021-04-09T18:11:28.122 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2021-04-09T18:11:28.122 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.122 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2021-04-09T18:11:28.123 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2021-04-09T18:11:28.123 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2021-04-09T18:11:28.124 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2021-04-09T18:11:28.124 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2021-04-09T18:11:28.125 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2021-04-09T18:11:28.125 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2021-04-09T18:11:28.125 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.126 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2021-04-09T18:11:28.126 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2021-04-09T18:11:28.127 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2021-04-09T18:11:28.159 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2021-04-09T18:11:28.159 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2021-04-09T18:11:28.160 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2021-04-09T18:11:28.160 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2021-04-09T18:11:28.160 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2021-04-09T18:11:28.161 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2021-04-09T18:11:28.161 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2021-04-09T18:11:28.161 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2021-04-09T18:11:28.162 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2021-04-09T18:11:28.162 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:11:28.163 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2021-04-09T18:11:28.163 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2021-04-09T18:11:28.164 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2021-04-09T18:11:28.164 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2021-04-09T18:11:28.165 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-04-09T18:11:28.165 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-04-09T18:11:28.165 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2021-04-09T18:11:28.231 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2021-04-09T18:11:28.232 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2021-04-09T18:11:28.232 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2021-04-09T18:11:28.233 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2021-04-09T18:11:28.233 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2021-04-09T18:11:28.234 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2021-04-09T18:11:28.234 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2021-04-09T18:11:28.234 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2021-04-09T18:11:28.235 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2021-04-09T18:11:28.235 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2021-04-09T18:11:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2021-04-09T18:11:28.236 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2021-04-09T18:11:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: mds bal frag = True 2021-04-09T18:11:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: mds bal fragment size max = 10000 2021-04-09T18:11:28.237 INFO:teuthology.orchestra.run.smithi032.stdout: mds bal merge size = 5 2021-04-09T18:11:28.238 INFO:teuthology.orchestra.run.smithi032.stdout: mds bal split bits = 3 2021-04-09T18:11:28.268 INFO:teuthology.orchestra.run.smithi032.stdout: mds bal split size = 100 2021-04-09T18:11:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2021-04-09T18:11:28.269 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2021-04-09T18:11:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2021-04-09T18:11:28.270 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2021-04-09T18:11:28.271 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2021-04-09T18:11:28.297 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2021-04-09T18:11:28.297 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2021-04-09T18:11:28.298 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-04-09T18:11:28.344 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2021-04-09T18:11:28.345 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.345 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2021-04-09T18:11:28.346 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.346 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2021-04-09T18:11:28.347 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.361 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2021-04-09T18:11:28.362 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2021-04-09T18:11:28.363 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2021-04-09T18:11:28.385 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2021-04-09T18:11:28.386 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.386 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2021-04-09T18:11:28.386 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2021-04-09T18:11:28.386 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.387 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2021-04-09T18:11:28.387 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.387 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2021-04-09T18:11:28.387 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2021-04-09T18:11:28.388 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2021-04-09T18:11:28.388 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2021-04-09T18:11:28.388 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2021-04-09T18:11:28.388 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2021-04-09T18:11:28.389 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2021-04-09T18:11:28.401 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-04-09T18:11:28.401 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2021-04-09T18:11:28.402 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.402 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-04-09T18:11:28.403 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.403 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2021-04-09T18:11:28.403 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.404 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2021-04-09T18:11:28.404 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.405 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2021-04-09T18:11:28.405 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2021-04-09T18:11:28.405 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2021-04-09T18:11:28.406 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.406 INFO:teuthology.orchestra.run.smithi086.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-04-09T18:11:28.407 INFO:teuthology.orchestra.run.smithi086.stdout: mon_client_directed_command_retry = 4 2021-04-09T18:11:28.439 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = "172.21.15.32,172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790]" 2021-04-09T18:11:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = cf69fbbc-37e8-4833-99b4-5ecbe4444cfb 2021-04-09T18:11:28.440 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.440 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2021-04-09T18:11:28.441 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2021-04-09T18:11:28.451 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.452 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2021-04-09T18:11:28.457 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2021-04-09T18:11:28.457 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.458 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2021-04-09T18:11:28.458 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2021-04-09T18:11:28.459 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.459 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2021-04-09T18:11:28.460 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2021-04-09T18:11:28.460 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2021-04-09T18:11:28.461 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.461 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2021-04-09T18:11:28.462 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2021-04-09T18:11:28.462 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.462 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2021-04-09T18:11:28.463 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.463 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2021-04-09T18:11:28.464 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.464 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2021-04-09T18:11:28.465 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.465 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2021-04-09T18:11:28.500 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2021-04-09T18:11:28.502 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.502 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2021-04-09T18:11:28.503 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2021-04-09T18:11:28.503 INFO:teuthology.orchestra.run.smithi086.stdout: bdev async discard = True 2021-04-09T18:11:28.504 INFO:teuthology.orchestra.run.smithi086.stdout: bdev enable discard = True 2021-04-09T18:11:28.504 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore allocator = bitmap 2021-04-09T18:11:28.504 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2021-04-09T18:11:28.505 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2021-04-09T18:11:28.505 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 20 2021-04-09T18:11:28.506 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 20 2021-04-09T18:11:28.506 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2021-04-09T18:11:28.507 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2021-04-09T18:11:28.507 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 10 2021-04-09T18:11:28.508 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2021-04-09T18:11:28.508 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2021-04-09T18:11:28.508 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2021-04-09T18:11:28.509 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2021-04-09T18:11:28.509 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2021-04-09T18:11:28.510 INFO:teuthology.orchestra.run.smithi086.stdout: osd op complaint time = 180 2021-04-09T18:11:28.510 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.511 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2021-04-09T18:11:28.544 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2021-04-09T18:11:28.544 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2021-04-09T18:11:28.551 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2021-04-09T18:11:28.552 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2021-04-09T18:11:28.552 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2021-04-09T18:11:28.552 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2021-04-09T18:11:28.553 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.553 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2021-04-09T18:11:28.554 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2021-04-09T18:11:28.554 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2021-04-09T18:11:28.555 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2021-04-09T18:11:28.574 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2021-04-09T18:11:28.575 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2021-04-09T18:11:28.585 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2021-04-09T18:11:28.585 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2021-04-09T18:11:28.586 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2021-04-09T18:11:28.586 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2021-04-09T18:11:28.586 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2021-04-09T18:11:28.587 INFO:teuthology.orchestra.run.smithi086.stdout: mon op complaint time = 120 2021-04-09T18:11:28.587 INFO:teuthology.orchestra.run.smithi086.stdout: 2021-04-09T18:11:28.588 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2021-04-09T18:11:28.615 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2021-04-09T18:11:28.616 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2021-04-09T18:11:28.616 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2021-04-09T18:11:28.616 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-04-09T18:11:28.616 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-04-09T18:11:28.617 INFO:teuthology.orchestra.run.smithi086.stdout: client mount timeout = 600 2021-04-09T18:11:28.617 INFO:teuthology.orchestra.run.smithi086.stdout: debug client = 20 2021-04-09T18:11:28.617 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2021-04-09T18:11:28.634 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2021-04-09T18:11:28.634 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.b] 2021-04-09T18:11:28.635 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.c] 2021-04-09T18:11:28.635 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.a] 2021-04-09T18:11:28.635 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.c] 2021-04-09T18:11:28.636 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.b] 2021-04-09T18:11:28.636 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.d] 2021-04-09T18:11:28.636 INFO:teuthology.orchestra.run.smithi086.stdout:[mds] 2021-04-09T18:11:28.637 INFO:teuthology.orchestra.run.smithi086.stdout: debug mds = 20 2021-04-09T18:11:28.637 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2021-04-09T18:11:28.637 INFO:teuthology.orchestra.run.smithi086.stdout: mds bal frag = True 2021-04-09T18:11:28.637 INFO:teuthology.orchestra.run.smithi086.stdout: mds bal fragment size max = 10000 2021-04-09T18:11:28.638 INFO:teuthology.orchestra.run.smithi086.stdout: mds bal merge size = 5 2021-04-09T18:11:28.638 INFO:teuthology.orchestra.run.smithi086.stdout: mds bal split bits = 3 2021-04-09T18:11:28.638 INFO:teuthology.orchestra.run.smithi086.stdout: mds bal split size = 100 2021-04-09T18:11:28.638 INFO:teuthology.orchestra.run.smithi086.stdout: mds debug frag = True 2021-04-09T18:11:28.638 INFO:teuthology.orchestra.run.smithi086.stdout: mds debug scatterstat = True 2021-04-09T18:11:28.639 INFO:teuthology.orchestra.run.smithi086.stdout: mds op complaint time = 180 2021-04-09T18:11:28.639 INFO:teuthology.orchestra.run.smithi086.stdout: mds verify scatter = True 2021-04-09T18:11:28.639 INFO:teuthology.orchestra.run.smithi086.stdout: osd op complaint time = 180 2021-04-09T18:11:28.640 INFO:tasks.ceph:Creating admin key on mon.a... 2021-04-09T18:11:28.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-04-09T18:11:28.754 INFO:tasks.ceph:Copying monmap to all nodes... 2021-04-09T18:11:28.818 DEBUG:teuthology.orchestra.remote:smithi032:/etc/ceph/ceph.keyring is 206B 2021-04-09T18:11:28.913 DEBUG:teuthology.orchestra.remote:smithi032:/home/ubuntu/cephtest/ceph.monmap is 373B 2021-04-09T18:11:28.961 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2021-04-09T18:11:28.962 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2021-04-09T18:11:28.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.keyring 2021-04-09T18:11:28.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-04-09T18:11:29.038 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2021-04-09T18:11:29.039 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:29.120 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2021-04-09T18:11:29.120 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2021-04-09T18:11:29.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2021-04-09T18:11:29.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-04-09T18:11:29.195 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2021-04-09T18:11:29.196 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:29.278 INFO:tasks.ceph:Setting up mon nodes... 2021-04-09T18:11:29.279 INFO:tasks.ceph:Setting up mgr nodes... 2021-04-09T18:11:29.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2021-04-09T18:11:29.390 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2021-04-09T18:11:29.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-04-09T18:11:29.515 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-04-09T18:11:29.520 INFO:tasks.ceph:Setting up mds nodes... 2021-04-09T18:11:29.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2021-04-09T18:11:29.637 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2021-04-09T18:11:29.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2021-04-09T18:11:29.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2021-04-09T18:11:29.807 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2021-04-09T18:11:29.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2021-04-09T18:11:29.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2021-04-09T18:11:29.968 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2021-04-09T18:11:29.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2021-04-09T18:11:30.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2021-04-09T18:11:30.121 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2021-04-09T18:11:30.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2021-04-09T18:11:30.179 INFO:tasks.ceph_client:Setting up client nodes... 2021-04-09T18:11:30.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-04-09T18:11:30.272 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.client.0.keyring 2021-04-09T18:11:30.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-04-09T18:11:30.383 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.1.keyring 2021-04-09T18:11:30.398 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-04-09T18:11:30.398 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-04-09T18:11:30.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-04-09T18:11:30.463 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:30.464 INFO:tasks.ceph:role: osd.0 2021-04-09T18:11:30.464 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com 2021-04-09T18:11:30.465 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-04-09T18:11:30.558 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2021-04-09T18:11:30.559 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-04-09T18:11:30.559 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-09T18:11:30.559 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=0, sparse=0 2021-04-09T18:11:30.559 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-09T18:11:30.560 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2021-04-09T18:11:30.560 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-09T18:11:30.560 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-09T18:11:30.560 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-09T18:11:30.561 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-09T18:11:30.562 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2021-04-09T18:11:30.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-04-09T18:11:30.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-04-09T18:11:30.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-04-09T18:11:30.766 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:30.768 INFO:tasks.ceph:role: osd.1 2021-04-09T18:11:30.768 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com 2021-04-09T18:11:30.769 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-04-09T18:11:30.864 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2021-04-09T18:11:30.865 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-04-09T18:11:30.865 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-09T18:11:30.866 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=0, sparse=0 2021-04-09T18:11:30.866 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-09T18:11:30.866 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2021-04-09T18:11:30.866 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-09T18:11:30.866 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-09T18:11:30.867 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-09T18:11:30.867 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-09T18:11:30.873 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2021-04-09T18:11:30.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-04-09T18:11:30.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-04-09T18:11:31.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-04-09T18:11:31.063 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:31.064 INFO:tasks.ceph:role: osd.2 2021-04-09T18:11:31.064 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com 2021-04-09T18:11:31.065 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-04-09T18:11:31.162 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2021-04-09T18:11:31.162 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-04-09T18:11:31.162 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-09T18:11:31.162 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=0, sparse=0 2021-04-09T18:11:31.163 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-09T18:11:31.163 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2021-04-09T18:11:31.163 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-09T18:11:31.163 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-09T18:11:31.164 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-09T18:11:31.164 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-09T18:11:31.165 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2021-04-09T18:11:31.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-04-09T18:11:31.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-04-09T18:11:31.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-04-09T18:11:31.381 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:31.382 INFO:tasks.ceph:role: osd.3 2021-04-09T18:11:31.382 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com 2021-04-09T18:11:31.383 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-04-09T18:11:31.480 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2021-04-09T18:11:31.481 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2021-04-09T18:11:31.481 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-09T18:11:31.481 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=0, sparse=0 2021-04-09T18:11:31.482 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-09T18:11:31.482 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2021-04-09T18:11:31.482 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-09T18:11:31.482 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-09T18:11:31.482 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-09T18:11:31.483 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-09T18:11:31.484 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2021-04-09T18:11:31.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2021-04-09T18:11:31.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-04-09T18:11:31.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:31.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:31.701 7ff28d66ca80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-04-09T18:11:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:31.701 7ff28d66ca80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-04-09T18:11:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:31.702 7ff28d66ca80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-04-09T18:11:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:31.702 7ff28d66ca80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-04-09T18:11:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:31.702 7ff28d66ca80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-04-09T18:11:31.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:31.702 7ff28d66ca80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-04-09T18:11:34.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-04-09T18:11:34.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:34.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:34.669 7f12b0dd9a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-04-09T18:11:34.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:34.670 7f12b0dd9a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-04-09T18:11:34.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:34.670 7f12b0dd9a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-04-09T18:11:34.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:34.670 7f12b0dd9a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-04-09T18:11:34.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:34.670 7f12b0dd9a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-04-09T18:11:34.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:34.670 7f12b0dd9a80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-04-09T18:11:37.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-04-09T18:11:37.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:37.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:37.630 7f45231bda80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-04-09T18:11:37.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:37.630 7f45231bda80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-04-09T18:11:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:37.630 7f45231bda80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-04-09T18:11:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:37.630 7f45231bda80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-04-09T18:11:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:37.630 7f45231bda80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-04-09T18:11:37.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:37.630 7f45231bda80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-04-09T18:11:40.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-04-09T18:11:40.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:40.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:40.579 7fad9b844a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-04-09T18:11:40.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:40.579 7fad9b844a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-04-09T18:11:40.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:40.579 7fad9b844a80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-04-09T18:11:40.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:40.579 7fad9b844a80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-04-09T18:11:40.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:40.579 7fad9b844a80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-04-09T18:11:40.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:11:40.580 7fad9b844a80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-04-09T18:11:43.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-04-09T18:11:43.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2021-04-09T18:11:43.525 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:43.526 INFO:tasks.ceph:role: osd.4 2021-04-09T18:11:43.526 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2021-04-09T18:11:43.527 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-04-09T18:11:44.072 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2021-04-09T18:11:44.073 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-04-09T18:11:44.073 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-09T18:11:44.074 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=0, sparse=0 2021-04-09T18:11:44.074 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-09T18:11:44.075 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2021-04-09T18:11:44.082 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-09T18:11:44.082 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-09T18:11:44.083 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-09T18:11:44.083 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-09T18:11:44.084 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2021-04-09T18:11:44.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2021-04-09T18:11:44.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2021-04-09T18:11:44.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2021-04-09T18:11:44.328 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:44.328 INFO:tasks.ceph:role: osd.5 2021-04-09T18:11:44.329 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2021-04-09T18:11:44.329 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-04-09T18:11:44.841 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2021-04-09T18:11:44.842 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-04-09T18:11:44.843 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-09T18:11:44.843 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=0, sparse=0 2021-04-09T18:11:44.844 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-09T18:11:44.844 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2021-04-09T18:11:44.844 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-09T18:11:44.845 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-09T18:11:44.845 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-09T18:11:44.846 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-09T18:11:44.859 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2021-04-09T18:11:44.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2021-04-09T18:11:44.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2021-04-09T18:11:45.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2021-04-09T18:11:45.076 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:45.077 INFO:tasks.ceph:role: osd.6 2021-04-09T18:11:45.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2021-04-09T18:11:45.078 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-04-09T18:11:45.631 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2021-04-09T18:11:45.632 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-04-09T18:11:45.632 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-09T18:11:45.633 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=0, sparse=0 2021-04-09T18:11:45.633 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-09T18:11:45.633 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2021-04-09T18:11:45.634 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-09T18:11:45.634 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-09T18:11:45.635 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-09T18:11:45.635 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-09T18:11:45.638 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2021-04-09T18:11:45.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2021-04-09T18:11:45.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2021-04-09T18:11:45.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2021-04-09T18:11:45.880 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-04-09T18:11:45.880 INFO:tasks.ceph:role: osd.7 2021-04-09T18:11:45.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com 2021-04-09T18:11:45.882 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-04-09T18:11:46.446 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2021-04-09T18:11:46.447 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2021-04-09T18:11:46.447 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-09T18:11:46.448 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=0, sparse=0 2021-04-09T18:11:46.448 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-09T18:11:46.448 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2021-04-09T18:11:46.449 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-09T18:11:46.449 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-09T18:11:46.450 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-09T18:11:46.450 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-09T18:11:46.453 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2021-04-09T18:11:46.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2021-04-09T18:11:46.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2021-04-09T18:11:46.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:46.723 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:46.704 7f845084ca80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2021-04-09T18:11:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:46.704 7f845084ca80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2021-04-09T18:11:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:46.704 7f845084ca80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-04-09T18:11:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:46.704 7f845084ca80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-04-09T18:11:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:46.704 7f845084ca80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-04-09T18:11:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:46.705 7f845084ca80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2021-04-09T18:11:49.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-04-09T18:11:49.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:49.648 7f8ab3da1a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2021-04-09T18:11:49.667 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:49.648 7f8ab3da1a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2021-04-09T18:11:49.667 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:49.648 7f8ab3da1a80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-04-09T18:11:49.667 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:49.648 7f8ab3da1a80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-04-09T18:11:49.667 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:49.648 7f8ab3da1a80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-04-09T18:11:49.668 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:49.649 7f8ab3da1a80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2021-04-09T18:11:52.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-04-09T18:11:52.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:52.594 7f714224ea80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2021-04-09T18:11:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:52.595 7f714224ea80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2021-04-09T18:11:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:52.595 7f714224ea80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-04-09T18:11:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:52.595 7f714224ea80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-04-09T18:11:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:52.595 7f714224ea80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-04-09T18:11:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:52.595 7f714224ea80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2021-04-09T18:11:55.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2021-04-09T18:11:55.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:11:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:55.540 7f81e9fa9a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2021-04-09T18:11:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:55.541 7f81e9fa9a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2021-04-09T18:11:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:55.541 7f81e9fa9a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-04-09T18:11:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:55.541 7f81e9fa9a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-04-09T18:11:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:55.541 7f81e9fa9a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-04-09T18:11:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:11:55.542 7f81e9fa9a80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2021-04-09T18:11:56.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:11:56.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:11:58.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2021-04-09T18:11:58.411 INFO:tasks.ceph:Reading keys from all nodes... 2021-04-09T18:11:58.412 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:11:58.464 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.xfXvG1BV2r 2021-04-09T18:11:58.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.xfXvG1BV2r 2021-04-09T18:11:58.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.xfXvG1BV2r 2021-04-09T18:11:58.683 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.xfXvG1BV2r is 56B 2021-04-09T18:11:58.736 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.xfXvG1BV2r 2021-04-09T18:11:58.806 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:11:58.883 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.tNORIjJ9Bu 2021-04-09T18:11:58.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.tNORIjJ9Bu 2021-04-09T18:11:58.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.tNORIjJ9Bu 2021-04-09T18:11:59.098 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.tNORIjJ9Bu is 56B 2021-04-09T18:11:59.161 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.tNORIjJ9Bu 2021-04-09T18:11:59.234 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:11:59.311 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.PHVlcLZMGi 2021-04-09T18:11:59.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/mds/ceph-c/keyring /tmp/tmp.PHVlcLZMGi 2021-04-09T18:11:59.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.PHVlcLZMGi 2021-04-09T18:11:59.529 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.PHVlcLZMGi is 56B 2021-04-09T18:11:59.598 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.PHVlcLZMGi 2021-04-09T18:11:59.645 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:11:59.725 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.iW4xuIcXg8 2021-04-09T18:11:59.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.iW4xuIcXg8 2021-04-09T18:11:59.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.iW4xuIcXg8 2021-04-09T18:11:59.932 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.iW4xuIcXg8 is 56B 2021-04-09T18:12:00.004 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.iW4xuIcXg8 2021-04-09T18:12:00.051 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:12:00.130 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.sRKxRlJIE9 2021-04-09T18:12:00.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.sRKxRlJIE9 2021-04-09T18:12:00.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.sRKxRlJIE9 2021-04-09T18:12:00.338 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.sRKxRlJIE9 is 56B 2021-04-09T18:12:00.406 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.sRKxRlJIE9 2021-04-09T18:12:00.450 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:12:00.531 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.3ZXVGTCbXe 2021-04-09T18:12:00.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.3ZXVGTCbXe 2021-04-09T18:12:00.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.3ZXVGTCbXe 2021-04-09T18:12:00.744 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.3ZXVGTCbXe is 56B 2021-04-09T18:12:00.796 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.3ZXVGTCbXe 2021-04-09T18:12:00.838 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:12:00.922 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.8y98AYGckd 2021-04-09T18:12:00.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.8y98AYGckd 2021-04-09T18:12:01.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.8y98AYGckd 2021-04-09T18:12:01.168 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.8y98AYGckd is 56B 2021-04-09T18:12:01.260 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.8y98AYGckd 2021-04-09T18:12:01.322 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:01.371 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.SPnYSMX6qI 2021-04-09T18:12:01.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.SPnYSMX6qI 2021-04-09T18:12:01.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.SPnYSMX6qI 2021-04-09T18:12:01.549 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.SPnYSMX6qI is 56B 2021-04-09T18:12:01.619 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.SPnYSMX6qI 2021-04-09T18:12:01.674 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.sbhuLyE0YN 2021-04-09T18:12:01.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.sbhuLyE0YN 2021-04-09T18:12:01.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.sbhuLyE0YN 2021-04-09T18:12:01.942 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.sbhuLyE0YN is 56B 2021-04-09T18:12:02.023 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.sbhuLyE0YN 2021-04-09T18:12:02.068 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.cT5QpEW6Li 2021-04-09T18:12:02.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/mds/ceph-d/keyring /tmp/tmp.cT5QpEW6Li 2021-04-09T18:12:02.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.cT5QpEW6Li 2021-04-09T18:12:02.358 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.cT5QpEW6Li is 56B 2021-04-09T18:12:02.467 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.cT5QpEW6Li 2021-04-09T18:12:02.518 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.FbYqWccVlq 2021-04-09T18:12:02.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.FbYqWccVlq 2021-04-09T18:12:02.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.FbYqWccVlq 2021-04-09T18:12:02.796 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.FbYqWccVlq is 56B 2021-04-09T18:12:02.865 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.FbYqWccVlq 2021-04-09T18:12:02.920 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.GsLE4nNmN3 2021-04-09T18:12:02.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.GsLE4nNmN3 2021-04-09T18:12:03.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.GsLE4nNmN3 2021-04-09T18:12:03.187 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.GsLE4nNmN3 is 56B 2021-04-09T18:12:03.257 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.GsLE4nNmN3 2021-04-09T18:12:03.304 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:03.380 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.D2apBvYCMH 2021-04-09T18:12:03.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.D2apBvYCMH 2021-04-09T18:12:03.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.D2apBvYCMH 2021-04-09T18:12:03.580 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.D2apBvYCMH is 56B 2021-04-09T18:12:03.632 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.D2apBvYCMH 2021-04-09T18:12:03.678 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.qxoMpHaQ5t 2021-04-09T18:12:03.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.qxoMpHaQ5t 2021-04-09T18:12:03.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.qxoMpHaQ5t 2021-04-09T18:12:03.956 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.qxoMpHaQ5t is 56B 2021-04-09T18:12:04.035 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.qxoMpHaQ5t 2021-04-09T18:12:04.147 DEBUG:teuthology.orchestra.remote:smithi032:/etc/ceph/ceph.client.0.keyring is 59B 2021-04-09T18:12:04.247 DEBUG:teuthology.orchestra.remote:smithi086:/etc/ceph/ceph.client.1.keyring is 59B 2021-04-09T18:12:04.307 INFO:tasks.ceph:Adding keys to all mons... 2021-04-09T18:12:04.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo tee -a /etc/ceph/ceph.keyring 2021-04-09T18:12:04.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2021-04-09T18:12:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.y] 2021-04-09T18:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg== 2021-04-09T18:12:04.372 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.a] 2021-04-09T18:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w== 2021-04-09T18:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.c] 2021-04-09T18:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ== 2021-04-09T18:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2021-04-09T18:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ== 2021-04-09T18:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2021-04-09T18:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg== 2021-04-09T18:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2021-04-09T18:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw== 2021-04-09T18:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.3] 2021-04-09T18:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw== 2021-04-09T18:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2021-04-09T18:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA== 2021-04-09T18:12:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.b] 2021-04-09T18:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ== 2021-04-09T18:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.d] 2021-04-09T18:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg== 2021-04-09T18:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.4] 2021-04-09T18:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg== 2021-04-09T18:12:04.451 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.5] 2021-04-09T18:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q== 2021-04-09T18:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.6] 2021-04-09T18:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg== 2021-04-09T18:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.7] 2021-04-09T18:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg== 2021-04-09T18:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2021-04-09T18:12:04.456 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw== 2021-04-09T18:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2021-04-09T18:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag== 2021-04-09T18:12:04.459 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.y] 2021-04-09T18:12:04.459 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg== 2021-04-09T18:12:04.459 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2021-04-09T18:12:04.460 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w== 2021-04-09T18:12:04.460 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2021-04-09T18:12:04.461 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ== 2021-04-09T18:12:04.461 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.0] 2021-04-09T18:12:04.543 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ== 2021-04-09T18:12:04.543 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.1] 2021-04-09T18:12:04.544 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg== 2021-04-09T18:12:04.547 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.2] 2021-04-09T18:12:04.548 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw== 2021-04-09T18:12:04.548 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.3] 2021-04-09T18:12:04.548 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw== 2021-04-09T18:12:04.549 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.x] 2021-04-09T18:12:04.549 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA== 2021-04-09T18:12:04.550 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2021-04-09T18:12:04.550 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ== 2021-04-09T18:12:04.551 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2021-04-09T18:12:04.572 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg== 2021-04-09T18:12:04.572 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.4] 2021-04-09T18:12:04.573 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg== 2021-04-09T18:12:04.595 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.5] 2021-04-09T18:12:04.597 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q== 2021-04-09T18:12:04.597 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.6] 2021-04-09T18:12:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg== 2021-04-09T18:12:04.598 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.7] 2021-04-09T18:12:04.598 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg== 2021-04-09T18:12:04.599 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2021-04-09T18:12:04.599 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw== 2021-04-09T18:12:04.600 INFO:teuthology.orchestra.run.smithi032.stdout:[client.1] 2021-04-09T18:12:04.600 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag== 2021-04-09T18:12:04.601 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-04-09T18:12:04.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-04-09T18:12:04.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-04-09T18:12:04.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-04-09T18:12:04.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-04-09T18:12:04.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-04-09T18:12:04.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.208 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-04-09T18:12:05.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-04-09T18:12:05.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-04-09T18:12:05.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-04-09T18:12:05.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-04-09T18:12:05.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-04-09T18:12:05.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:05.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:06.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:06.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-09T18:12:06.162 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-09T18:12:06.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-09T18:12:06.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-09T18:12:06.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-09T18:12:06.382 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-04-09T18:12:06.383 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-04-09T18:12:06.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-04-09T18:12:06.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-04-09T18:12:06.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-04-09T18:12:06.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-04-09T18:12:06.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-04-09T18:12:06.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-04-09T18:12:06.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-04-09T18:12:07.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-04-09T18:12:07.057 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:12:07.087 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-04-09T18:12:07.128 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-04-09T18:12:07.129 INFO:tasks.ceph.mon.a:Restarting daemon 2021-04-09T18:12:07.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-04-09T18:12:07.171 INFO:tasks.ceph.mon.a:Started 2021-04-09T18:12:07.172 INFO:tasks.ceph.mon.b:Restarting daemon 2021-04-09T18:12:07.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-04-09T18:12:07.176 INFO:tasks.ceph.mon.b:Started 2021-04-09T18:12:07.176 INFO:tasks.ceph.mon.c:Restarting daemon 2021-04-09T18:12:07.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-04-09T18:12:07.179 INFO:tasks.ceph.mon.c:Started 2021-04-09T18:12:07.179 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-04-09T18:12:07.180 INFO:tasks.ceph.mgr.y:Restarting daemon 2021-04-09T18:12:07.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2021-04-09T18:12:07.186 INFO:tasks.ceph.mgr.y:Started 2021-04-09T18:12:07.186 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-04-09T18:12:07.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-04-09T18:12:07.199 INFO:tasks.ceph.mgr.x:Started 2021-04-09T18:12:07.200 INFO:tasks.ceph:Setting crush tunables to default 2021-04-09T18:12:07.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd crush tunables default 2021-04-09T18:12:12.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.344 7f49edf5a700 1 Processor -- start 2021-04-09T18:12:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.345 7f49edf5a700 1 -- start start 2021-04-09T18:12:12.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.345 7f49edf5a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.345 7f49edf5a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e80d7290 con 0x7f49e80d6c90 2021-04-09T18:12:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.345 7f49edf5a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e80d8400 con 0x7f49e80d7810 2021-04-09T18:12:12.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.345 7f49edf5a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e80d7090 con 0x7f49e80cffb0 2021-04-09T18:12:12.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.346 7f49e77fe700 1 -- >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 msgr2=0x7f49e80d2450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2021-04-09T18:12:12.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.346 7f49e7fff700 1 -- >> v1:172.21.15.32:6789/0 conn(0x7f49e80d6c90 legacy=0x7f49e810cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2021-04-09T18:12:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.346 7f49e77fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:12.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.346 7f49e6ffd700 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f49e80d7810 legacy=0x7f49e80d7c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2021-04-09T18:12:12.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.547 7f49e77fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.547 7f49e7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f49e80d6c90 0x7f49e810cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50570/0 (socket says 172.21.15.32:50570) 2021-04-09T18:12:12.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.547 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 learned_addr learned my addr 172.21.15.32:0/2826225891 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:12.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.547 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 msgr2=0x7f49e80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-04-09T18:12:12.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.547 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 msgr2=0x7f49e80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.547 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.547 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-04-09T18:12:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.948 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.948 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 msgr2=0x7f49e80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-04-09T18:12:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.948 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 msgr2=0x7f49e80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.948 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:07.948 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-04-09T18:12:12.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:08.749 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:08.749 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 msgr2=0x7f49e80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-04-09T18:12:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:08.749 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 msgr2=0x7f49e80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:08.749 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:08.749 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-04-09T18:12:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e57fa700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 msgr2=0x7f49e80d2450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-04-09T18:12:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e57fa700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49e80cffb0 0x7f49e80d2450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e57fa700 1 -- 172.21.15.32:0/2826225891 >> v1:172.21.15.86:6789/0 conn(0x7f49e80d7810 legacy=0x7f49e80d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:12.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e57fa700 1 -- 172.21.15.32:0/2826225891 >> v1:172.21.15.32:6789/0 conn(0x7f49e80d6c90 legacy=0x7f49e810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:12.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e57fa700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:12.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e57fa700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e57fa700 1 -- 172.21.15.32:0/2826225891 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d40508b0 con 0x7f49d4005d10 2021-04-09T18:12:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e57fa700 1 -- 172.21.15.32:0/2826225891 --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d4051b90 con 0x7f49d4009ba0 2021-04-09T18:12:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e57fa700 1 -- 172.21.15.32:0/2826225891 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4051d80 con 0x7f49d4003150 2021-04-09T18:12:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 msgr2=0x7f49d400c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2021-04-09T18:12:12.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 msgr2=0x7f49d400c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 msgr2=0x7f49d4005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-04-09T18:12:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:12.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 msgr2=0x7f49d4005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.346 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:12.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 msgr2=0x7f49d400c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-04-09T18:12:12.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 msgr2=0x7f49d400c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-04-09T18:12:12.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 msgr2=0x7f49d4005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2021-04-09T18:12:12.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 msgr2=0x7f49d4005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.547 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-04-09T18:12:12.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 msgr2=0x7f49d400c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-04-09T18:12:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 msgr2=0x7f49d400c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-04-09T18:12:12.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 msgr2=0x7f49d4005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2021-04-09T18:12:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 msgr2=0x7f49d4005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:10.948 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-04-09T18:12:12.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 msgr2=0x7f49d400c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-04-09T18:12:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 msgr2=0x7f49d400c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-04-09T18:12:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:12.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 msgr2=0x7f49d4005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2021-04-09T18:12:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e77fe700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 msgr2=0x7f49d4005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:11.749 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-04-09T18:12:12.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.529 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206110955 0 0) 0x7f49d0003310 con 0x7f49d4005d10 2021-04-09T18:12:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c8002fe0 con 0x7f49d4005d10 2021-04-09T18:12:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1467008119 0 0) 0x7f49d0003520 con 0x7f49d4005d10 2021-04-09T18:12:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 msgr2=0x7f49d4005560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-04-09T18:12:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d4003150 0x7f49d4005560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 msgr2=0x7f49d400c090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-04-09T18:12:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 --2- 172.21.15.32:0/2826225891 >> v2:172.21.15.32:3300/0 conn(0x7f49d4009ba0 0x7f49d400c090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:12.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d4052020 con 0x7f49d4005d10 2021-04-09T18:12:12.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f49d0003960 con 0x7f49d4005d10 2021-04-09T18:12:12.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 >> v1:172.21.15.32:6789/0 conn(0x7f49d4005d10 legacy=0x7f49d40081a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49c8003a70 0x7f49c8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49c8006800 0x7f49c8008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:12.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49c80095a0 0x7f49c800ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:12.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c800c2e0 con 0x7f49c8006800 2021-04-09T18:12:13.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c800c500 con 0x7f49c8003a70 2021-04-09T18:12:13.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c800c720 con 0x7f49c80095a0 2021-04-09T18:12:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e6ffd700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49c8006800 0x7f49c8008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49c8003a70 0x7f49c8005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:13.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49c80095a0 0x7f49c800ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:13.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49c80095a0 msgr2=0x7f49c800ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2021-04-09T18:12:13.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49c80095a0 msgr2=0x7f49c800ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:13.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49c80095a0 0x7f49c800ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:13.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.530 7f49e7fff700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49c80095a0 0x7f49c800ba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:13.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e6ffd700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49c80095a0 msgr2=0x7f49c800ba90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-04-09T18:12:13.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e6ffd700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49c80095a0 0x7f49c800ba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:13.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e6ffd700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49c8003a70 msgr2=0x7f49c8005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:13.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e6ffd700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49c8003a70 0x7f49c8005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:13.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e7fff700 1 -- 172.21.15.32:0/2826225891 reap_dead start 2021-04-09T18:12:13.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e6ffd700 1 -- 172.21.15.32:0/2826225891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c800c9c0 con 0x7f49c8006800 2021-04-09T18:12:13.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e77fe700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49c8003a70 0x7f49c8005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:13.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e6ffd700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49c8006800 0x7f49c8008d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f49d0009550 tx=0x7f49d0009c50).ready entity=mon.0 client_cookie=524f7cedf7e941d3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:13.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49d0006350 con 0x7f49c8006800 2021-04-09T18:12:13.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d0006610 con 0x7f49c8006800 2021-04-09T18:12:13.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49e67fc700 1 -- 172.21.15.32:0/2826225891 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49d0003500 con 0x7f49c8006800 2021-04-09T18:12:13.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49edf5a700 1 -- 172.21.15.32:0/2826225891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49c8006800 msgr2=0x7f49c8008d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:13.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.531 7f49edf5a700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49c8006800 0x7f49c8008d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f49d0009550 tx=0x7f49d0009c50).stop 2021-04-09T18:12:13.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.532 7f49edf5a700 1 -- 172.21.15.32:0/2826225891 shutdown_connections 2021-04-09T18:12:13.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.532 7f49edf5a700 1 --2- 172.21.15.32:0/2826225891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49c8006800 0x7f49c8008d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.532 7f49edf5a700 1 -- 172.21.15.32:0/2826225891 >> 172.21.15.32:0/2826225891 conn(0x7f49e80ce5d0 msgr2=0x7f49e80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.532 7f49edf5a700 1 -- 172.21.15.32:0/2826225891 shutdown_connections 2021-04-09T18:12:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.532 7f49edf5a700 1 -- 172.21.15.32:0/2826225891 wait complete. 2021-04-09T18:12:13.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.553 7f49edf5a700 1 Processor -- start 2021-04-09T18:12:13.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.554 7f49edf5a700 1 -- start start 2021-04-09T18:12:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.554 7f49edf5a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49e808d210 0x7f49e808fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:13.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.554 7f49edf5a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e80903f0 0x7f49e81a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:13.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.554 7f49edf5a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e81a6080 0x7f49e81a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:13.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.554 7f49edf5a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e808d690 con 0x7f49e808d210 2021-04-09T18:12:13.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.554 7f49edf5a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e80b0fd0 con 0x7f49e81a6080 2021-04-09T18:12:13.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.554 7f49edf5a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e81a6480 con 0x7f49e80903f0 2021-04-09T18:12:13.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49e808d210 0x7f49e808fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:13.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49e808d210 0x7f49e808fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41056/0 (socket says 172.21.15.32:41056) 2021-04-09T18:12:13.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e77fe700 1 -- 172.21.15.32:0/3646822149 learned_addr learned my addr 172.21.15.32:0/3646822149 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:13.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e6ffd700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e80903f0 0x7f49e81a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:13.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e7fff700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e81a6080 0x7f49e81a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:13.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e6ffd700 1 -- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e80903f0 msgr2=0x7f49e81a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-04-09T18:12:13.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e6ffd700 1 -- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e80903f0 msgr2=0x7f49e81a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-04-09T18:12:13.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e6ffd700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e80903f0 0x7f49e81a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:13.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e6ffd700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e80903f0 0x7f49e81a5a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:13.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e77fe700 1 -- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e80903f0 msgr2=0x7f49e81a5a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-04-09T18:12:13.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e77fe700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e80903f0 0x7f49e81a5a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:13.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e77fe700 1 -- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e81a6080 msgr2=0x7f49e81a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:13.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e77fe700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e81a6080 0x7f49e81a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:13.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e77fe700 1 -- 172.21.15.32:0/3646822149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e808d8b0 con 0x7f49e808d210 2021-04-09T18:12:13.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e7fff700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e81a6080 0x7f49e81a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:13.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49e77fe700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49e808d210 0x7f49e808fdd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f49e000a850 tx=0x7f49e000abf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:13.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49dffff700 1 -- 172.21.15.32:0/3646822149 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49e00050d0 con 0x7f49e808d210 2021-04-09T18:12:13.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49dffff700 1 -- 172.21.15.32:0/3646822149 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e0003430 con 0x7f49e808d210 2021-04-09T18:12:13.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49edf5a700 1 -- 172.21.15.32:0/3646822149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e808d8b0 con 0x7f49e808d210 2021-04-09T18:12:13.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.555 7f49dffff700 1 -- 172.21.15.32:0/3646822149 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49e0003910 con 0x7f49e808d210 2021-04-09T18:12:13.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.556 7f49edf5a700 1 -- 172.21.15.32:0/3646822149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e808d8b0 con 0x7f49e808d210 2021-04-09T18:12:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.556 7f49dffff700 1 -- 172.21.15.32:0/3646822149 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) v1 ==== 190+0+0 (secure 0 0 0) 0x7f49e0005030 con 0x7f49e808d210 2021-04-09T18:12:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.556 7f49dffff700 1 -- 172.21.15.32:0/3646822149 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f49e0003980 con 0x7f49e808d210 2021-04-09T18:12:13.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.602 7f49edf5a700 1 -- 172.21.15.32:0/3646822149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e81a52b0 con 0x7f49e808d210 2021-04-09T18:12:13.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.605 7f49dffff700 1 -- 172.21.15.32:0/3646822149 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+66016 (secure 0 0 0) 0x7f49e00079b0 con 0x7f49e808d210 2021-04-09T18:12:13.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.703 7f49edf5a700 1 -- 172.21.15.32:0/3646822149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"profile": "default", "prefix": "osd crush tunables"} v 0) v1 -- 0x7f49e81a52b0 con 0x7f49e808d210 2021-04-09T18:12:13.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.706 7f49dffff700 1 -- 172.21.15.32:0/3646822149 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"profile": "default", "prefix": "osd crush tunables"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f49e0003430 con 0x7f49e808d210 2021-04-09T18:12:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2021-04-09T18:12:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.707 7f49edf5a700 1 -- 172.21.15.32:0/3646822149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49e808d210 msgr2=0x7f49e808fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.707 7f49edf5a700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49e808d210 0x7f49e808fdd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f49e000a850 tx=0x7f49e000abf0).stop 2021-04-09T18:12:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.707 7f49edf5a700 1 -- 172.21.15.32:0/3646822149 shutdown_connections 2021-04-09T18:12:13.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.707 7f49edf5a700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e81a6080 0x7f49e81a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:13.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.707 7f49edf5a700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e80903f0 0x7f49e81a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:13.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.707 7f49edf5a700 1 --2- 172.21.15.32:0/3646822149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49e808d210 0x7f49e808fdd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:13.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.707 7f49edf5a700 1 -- 172.21.15.32:0/3646822149 >> 172.21.15.32:0/3646822149 conn(0x7f49e81a8f50 msgr2=0x7f49e8188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:13.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.708 7f49edf5a700 1 -- 172.21.15.32:0/3646822149 shutdown_connections 2021-04-09T18:12:13.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:12.708 7f49edf5a700 1 -- 172.21.15.32:0/3646822149 wait complete. 2021-04-09T18:12:13.215 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-04-09T18:12:13.216 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:12:13.269 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.9UCLWiIQbk 2021-04-09T18:12:13.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.9UCLWiIQbk 2021-04-09T18:12:13.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.9UCLWiIQbk 2021-04-09T18:12:13.478 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.9UCLWiIQbk is 37B 2021-04-09T18:12:13.540 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.9UCLWiIQbk 2021-04-09T18:12:13.605 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:12:13.674 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.xPrXXLGnll 2021-04-09T18:12:13.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.xPrXXLGnll 2021-04-09T18:12:13.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.xPrXXLGnll 2021-04-09T18:12:13.878 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.xPrXXLGnll is 37B 2021-04-09T18:12:13.930 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.xPrXXLGnll 2021-04-09T18:12:13.976 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:12:14.057 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.IUteZfMyT7 2021-04-09T18:12:14.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.IUteZfMyT7 2021-04-09T18:12:14.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.IUteZfMyT7 2021-04-09T18:12:14.269 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.IUteZfMyT7 is 37B 2021-04-09T18:12:14.321 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.IUteZfMyT7 2021-04-09T18:12:14.367 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2021-04-09T18:12:14.451 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.EEEHUr1J0d 2021-04-09T18:12:14.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.EEEHUr1J0d 2021-04-09T18:12:14.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.EEEHUr1J0d 2021-04-09T18:12:14.667 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.EEEHUr1J0d is 37B 2021-04-09T18:12:14.741 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.EEEHUr1J0d 2021-04-09T18:12:14.786 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:14.828 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.zSSGmIbZfe 2021-04-09T18:12:14.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.zSSGmIbZfe 2021-04-09T18:12:14.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.zSSGmIbZfe 2021-04-09T18:12:15.022 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.zSSGmIbZfe is 37B 2021-04-09T18:12:15.072 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.zSSGmIbZfe 2021-04-09T18:12:15.115 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:15.195 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.KWFBaQ6izO 2021-04-09T18:12:15.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.KWFBaQ6izO 2021-04-09T18:12:15.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.KWFBaQ6izO 2021-04-09T18:12:15.391 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.KWFBaQ6izO is 37B 2021-04-09T18:12:15.449 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.KWFBaQ6izO 2021-04-09T18:12:15.492 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:15.570 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.CHUXE6t2hR 2021-04-09T18:12:15.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.CHUXE6t2hR 2021-04-09T18:12:15.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.CHUXE6t2hR 2021-04-09T18:12:15.761 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.CHUXE6t2hR is 37B 2021-04-09T18:12:15.809 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.CHUXE6t2hR 2021-04-09T18:12:15.858 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2021-04-09T18:12:15.931 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.u4NKX9Lm12 2021-04-09T18:12:15.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.u4NKX9Lm12 2021-04-09T18:12:16.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.u4NKX9Lm12 2021-04-09T18:12:16.129 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.u4NKX9Lm12 is 37B 2021-04-09T18:12:16.181 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.u4NKX9Lm12 2021-04-09T18:12:16.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 8f892062-dd3f-413b-bbd2-6cfd9d640f69 0 2021-04-09T18:12:16.426 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.384 7f745a8ac700 1 Processor -- start 2021-04-09T18:12:16.428 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f745a8ac700 1 -- start start 2021-04-09T18:12:16.428 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f745a8ac700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f74540d6c90 0x7f74540d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.428 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f745a8ac700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74540d2450 0x7f745410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.429 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f745a8ac700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74540d7290 con 0x7f74540cffb0 2021-04-09T18:12:16.429 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f745a8ac700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74540d7090 con 0x7f74540d2450 2021-04-09T18:12:16.430 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f745a8ac700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74540d91f0 con 0x7f74540d6c90 2021-04-09T18:12:16.430 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f744bfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f74540d6c90 0x7f74540d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:16.431 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f7453fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f74540cffb0 0x7f74540d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42788/0 (socket says 172.21.15.86:42788) 2021-04-09T18:12:16.432 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f7453fff700 1 -- 172.21.15.86:0/1721729309 learned_addr learned my addr 172.21.15.86:0/1721729309 (peer_addr_for_me v1:172.21.15.86:0/0) 2021-04-09T18:12:16.432 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.385 7f7458ec9700 1 --2- 172.21.15.86:0/1721729309 >> v2:172.21.15.32:3300/0 conn(0x7f74540d2450 0x7f745410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:16.433 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f74537fe700 1 -- 172.21.15.86:0/1721729309 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945177071 0 0) 0x7f7444000ce0 con 0x7f74540cffb0 2021-04-09T18:12:16.433 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f74537fe700 1 -- 172.21.15.86:0/1721729309 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7440002fe0 con 0x7f74540cffb0 2021-04-09T18:12:16.433 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f744bfff700 1 -- 172.21.15.86:0/1721729309 >> v2:172.21.15.32:3300/0 conn(0x7f74540d2450 msgr2=0x7f745410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:16.434 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f744bfff700 1 --2- 172.21.15.86:0/1721729309 >> v2:172.21.15.32:3300/0 conn(0x7f74540d2450 0x7f745410cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.434 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f744bfff700 1 -- 172.21.15.86:0/1721729309 >> v1:172.21.15.86:6789/0 conn(0x7f74540cffb0 legacy=0x7f74540d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:16.435 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f744bfff700 1 -- 172.21.15.86:0/1721729309 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74540d2850 con 0x7f74540d6c90 2021-04-09T18:12:16.435 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f7458ec9700 1 --2- 172.21.15.86:0/1721729309 >> v2:172.21.15.32:3300/0 conn(0x7f74540d2450 0x7f745410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:16.435 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f744bfff700 1 --2- 172.21.15.86:0/1721729309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540d6c90 0x7f74540d8000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f743c006650 tx=0x7f743c009a80).ready entity=mon.1 client_cookie=66fe836002cdcba4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:16.436 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f74537fe700 1 -- 172.21.15.86:0/1721729309 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f743c0009b0 con 0x7f74540d6c90 2021-04-09T18:12:16.443 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f74537fe700 1 -- 172.21.15.86:0/1721729309 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743c00c0c0 con 0x7f74540d6c90 2021-04-09T18:12:16.443 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f74537fe700 1 -- 172.21.15.86:0/1721729309 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f743c00c510 con 0x7f74540d6c90 2021-04-09T18:12:16.444 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f745a8ac700 1 -- 172.21.15.86:0/1721729309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540d6c90 msgr2=0x7f74540d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:16.444 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.386 7f745a8ac700 1 --2- 172.21.15.86:0/1721729309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540d6c90 0x7f74540d8000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f743c006650 tx=0x7f743c009a80).stop 2021-04-09T18:12:16.445 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.387 7f745a8ac700 1 -- 172.21.15.86:0/1721729309 shutdown_connections 2021-04-09T18:12:16.461 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.387 7f745a8ac700 1 --2- 172.21.15.86:0/1721729309 >> v2:172.21.15.32:3300/0 conn(0x7f74540d2450 0x7f745410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.461 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.387 7f745a8ac700 1 --2- 172.21.15.86:0/1721729309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540d6c90 0x7f74540d8000 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.387 7f745a8ac700 1 -- 172.21.15.86:0/1721729309 >> 172.21.15.86:0/1721729309 conn(0x7f74540ce5d0 msgr2=0x7f74540cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.387 7f745a8ac700 1 -- 172.21.15.86:0/1721729309 shutdown_connections 2021-04-09T18:12:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.387 7f745a8ac700 1 -- 172.21.15.86:0/1721729309 wait complete. 2021-04-09T18:12:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f745a8ac700 1 Processor -- start 2021-04-09T18:12:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f745a8ac700 1 -- start start 2021-04-09T18:12:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f745a8ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74540a8a90 0x7f74541b8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f745a8ac700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74541b8c30 0x7f745408d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f745a8ac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540a8e90 0x7f745408dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f745a8ac700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745408d420 con 0x7f74540a8a90 2021-04-09T18:12:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f745a8ac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745417d3f0 con 0x7f74540a8e90 2021-04-09T18:12:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f745a8ac700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74541a0b10 con 0x7f74541b8c30 2021-04-09T18:12:16.491 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f7458ec9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540a8e90 0x7f745408dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f7458ec9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540a8e90 0x7f745408dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50752/0 (socket says 172.21.15.86:50752) 2021-04-09T18:12:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f7458ec9700 1 -- 172.21.15.86:0/3579970022 learned_addr learned my addr 172.21.15.86:0/3579970022 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.406 7f744bfff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74541b8c30 0x7f745408d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f744bfff700 1 -- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74541b8c30 msgr2=0x7f745408d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-04-09T18:12:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f744bfff700 1 -- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74541b8c30 msgr2=0x7f745408d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-04-09T18:12:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f744bfff700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74541b8c30 0x7f745408d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:16.512 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f744bfff700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74541b8c30 0x7f745408d9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:16.512 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f7453fff700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74540a8a90 0x7f74541b8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f7458ec9700 1 -- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74541b8c30 msgr2=0x7f745408d9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-04-09T18:12:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f7458ec9700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74541b8c30 0x7f745408d9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f7458ec9700 1 -- 172.21.15.86:0/3579970022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74540a8a90 msgr2=0x7f74541b8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f7458ec9700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74540a8a90 0x7f74541b8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f7458ec9700 1 -- 172.21.15.86:0/3579970022 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745408c630 con 0x7f74540a8e90 2021-04-09T18:12:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f7458ec9700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540a8e90 0x7f745408dfc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f744c009980 tx=0x7f744c00bcd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f74517fa700 1 -- 172.21.15.86:0/3579970022 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f744c00c2f0 con 0x7f74540a8e90 2021-04-09T18:12:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f74517fa700 1 -- 172.21.15.86:0/3579970022 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c006790 con 0x7f74540a8e90 2021-04-09T18:12:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745408c630 con 0x7f74540a8e90 2021-04-09T18:12:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f74517fa700 1 -- 172.21.15.86:0/3579970022 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f744c00c7e0 con 0x7f74540a8e90 2021-04-09T18:12:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.407 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745408c630 con 0x7f74540a8e90 2021-04-09T18:12:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.408 7f74517fa700 1 -- 172.21.15.86:0/3579970022 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f744c0011f0 con 0x7f74540a8e90 2021-04-09T18:12:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.408 7f74517fa700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.408 7f74517fa700 1 -- 172.21.15.86:0/3579970022 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f744c031ca0 con 0x7f74540a8e90 2021-04-09T18:12:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.408 7f7453fff700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.408 7f7453fff700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.431 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74541b7120 con 0x7f74540a8e90 2021-04-09T18:12:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.434 7f74517fa700 1 -- 172.21.15.86:0/3579970022 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f744c00be10 con 0x7f74540a8e90 2021-04-09T18:12:16.638 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.620 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", "id": 0} v 0) v1 -- 0x7f74540d7290 con 0x7f74540a8e90 2021-04-09T18:12:16.644 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.626 7f74517fa700 1 -- 172.21.15.86:0/3579970022 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f744c006670 con 0x7f74540a8e90 2021-04-09T18:12:16.645 INFO:teuthology.orchestra.run.smithi086.stdout:0 2021-04-09T18:12:16.648 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 msgr2=0x7f743801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:16.649 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.649 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540a8e90 msgr2=0x7f745408dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:16.649 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540a8e90 0x7f745408dfc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f744c009980 tx=0x7f744c00bcd0).stop 2021-04-09T18:12:16.649 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 shutdown_connections 2021-04-09T18:12:16.650 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.650 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74540a8e90 0x7f745408dfc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.650 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74541b8c30 0x7f745408d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.650 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 --2- 172.21.15.86:0/3579970022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74540a8a90 0x7f74541b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.650 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 >> 172.21.15.86:0/3579970022 conn(0x7f74541905e0 msgr2=0x7f74541b7350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:16.651 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 shutdown_connections 2021-04-09T18:12:16.651 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.630 7f745a8ac700 1 -- 172.21.15.86:0/3579970022 wait complete. 2021-04-09T18:12:16.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 5f094ec4-4fc7-4ed9-a278-f51fab1f1727 1 2021-04-09T18:12:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.778 7fc2e78f1700 1 Processor -- start 2021-04-09T18:12:16.830 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.778 7fc2e78f1700 1 -- start start 2021-04-09T18:12:16.830 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e78f1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc2e00cffb0 0x7fc2e00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.830 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e78f1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 0x7fc2e010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.831 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e78f1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e00d7290 con 0x7fc2e00d7810 2021-04-09T18:12:16.831 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e78f1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e00d7090 con 0x7fc2e00cffb0 2021-04-09T18:12:16.832 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e78f1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e00d91f0 con 0x7fc2e00d6c90 2021-04-09T18:12:16.833 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e5f0e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 0x7fc2e010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:16.833 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e570d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc2e00cffb0 0x7fc2e00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:16.846 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e570d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc2e00cffb0 0x7fc2e00d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50756/0 (socket says 172.21.15.86:50756) 2021-04-09T18:12:16.847 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e5f0e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 0x7fc2e010cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51972/0 (socket says 172.21.15.86:51972) 2021-04-09T18:12:16.847 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e570d700 1 -- 172.21.15.86:0/1116461569 learned_addr learned my addr 172.21.15.86:0/1116461569 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:16.847 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e4f0c700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc2e00d7810 0x7fc2e00d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42806/0 (socket says 172.21.15.86:42806) 2021-04-09T18:12:16.861 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e5f0e700 1 -- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 msgr2=0x7fc2e010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-04-09T18:12:16.862 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e5f0e700 1 -- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 msgr2=0x7fc2e010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:16.862 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e5f0e700 1 --2- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 0x7fc2e010cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:16.863 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e5f0e700 1 --2- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 0x7fc2e010cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:16.863 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e570d700 1 -- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 msgr2=0x7fc2e010cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-04-09T18:12:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e570d700 1 --2- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 0x7fc2e010cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.875 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e570d700 1 -- 172.21.15.86:0/1116461569 >> v1:172.21.15.86:6789/0 conn(0x7fc2e00d7810 legacy=0x7fc2e00d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:16.876 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2e570d700 1 -- 172.21.15.86:0/1116461569 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e00d8400 con 0x7fc2e00cffb0 2021-04-09T18:12:16.877 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.779 7fc2d7fff700 1 -- 172.21.15.86:0/1116461569 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265232315 0 0) 0x7fc2d00009b0 con 0x7fc2e00d7810 2021-04-09T18:12:16.877 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2e570d700 1 --2- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e00cffb0 0x7fc2e00d2450 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc2dc006a80 tx=0x7fc2dc009e80).ready entity=mon.1 client_cookie=7155eb543a43bec7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2d7fff700 1 -- 172.21.15.86:0/1116461569 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2dc00b510 con 0x7fc2e00cffb0 2021-04-09T18:12:16.894 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2d7fff700 1 -- 172.21.15.86:0/1116461569 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc00b7d0 con 0x7fc2e00cffb0 2021-04-09T18:12:16.933 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2d7fff700 1 -- 172.21.15.86:0/1116461569 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2dc00bcb0 con 0x7fc2e00cffb0 2021-04-09T18:12:16.938 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2e78f1700 1 -- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e00cffb0 msgr2=0x7fc2e00d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:16.939 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2e78f1700 1 --2- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e00cffb0 0x7fc2e00d2450 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc2dc006a80 tx=0x7fc2dc009e80).stop 2021-04-09T18:12:16.939 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2e78f1700 1 -- 172.21.15.86:0/1116461569 shutdown_connections 2021-04-09T18:12:16.940 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2e78f1700 1 --2- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e00d6c90 0x7fc2e010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.940 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2e78f1700 1 --2- 172.21.15.86:0/1116461569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e00cffb0 0x7fc2e00d2450 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:16.940 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.780 7fc2e78f1700 1 -- 172.21.15.86:0/1116461569 >> 172.21.15.86:0/1116461569 conn(0x7fc2e00ce5d0 msgr2=0x7fc2e00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:16.941 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.781 7fc2e78f1700 1 -- 172.21.15.86:0/1116461569 shutdown_connections 2021-04-09T18:12:16.941 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.781 7fc2e78f1700 1 -- 172.21.15.86:0/1116461569 wait complete. 2021-04-09T18:12:16.942 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.799 7fc2e78f1700 1 Processor -- start 2021-04-09T18:12:16.942 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.799 7fc2e78f1700 1 -- start start 2021-04-09T18:12:16.991 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e78f1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e01b8380 0x7fc2e01b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.992 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e78f1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 0x7fc2e008cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.992 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e78f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e008d1d0 0x7fc2e01805b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:16.993 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e78f1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e0181a30 con 0x7fc2e008d1d0 2021-04-09T18:12:16.993 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e78f1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e008c230 con 0x7fc2e01b8380 2021-04-09T18:12:16.993 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e78f1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e01b8da0 con 0x7fc2e008c7b0 2021-04-09T18:12:16.994 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e4f0c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 0x7fc2e008cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:16.994 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e570d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e01b8380 0x7fc2e01b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.004 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e570d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e01b8380 0x7fc2e01b8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50764/0 (socket says 172.21.15.86:50764) 2021-04-09T18:12:17.005 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e4f0c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 0x7fc2e008cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51980/0 (socket says 172.21.15.86:51980) 2021-04-09T18:12:17.007 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e570d700 1 -- 172.21.15.86:0/2161579952 learned_addr learned my addr 172.21.15.86:0/2161579952 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:17.007 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e5f0e700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e008d1d0 0x7fc2e01805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.007 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e4f0c700 1 -- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 msgr2=0x7fc2e008cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-04-09T18:12:17.013 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e4f0c700 1 -- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 msgr2=0x7fc2e008cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-04-09T18:12:17.013 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e4f0c700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 0x7fc2e008cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:17.016 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e4f0c700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 0x7fc2e008cbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:17.017 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e570d700 1 -- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 msgr2=0x7fc2e008cbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-04-09T18:12:17.017 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e570d700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 0x7fc2e008cbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.018 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e570d700 1 -- 172.21.15.86:0/2161579952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e008d1d0 msgr2=0x7fc2e01805b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.036 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e570d700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e008d1d0 0x7fc2e01805b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.039 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.800 7fc2e570d700 1 -- 172.21.15.86:0/2161579952 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e01b8f90 con 0x7fc2e01b8380 2021-04-09T18:12:17.039 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.801 7fc2e570d700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e01b8380 0x7fc2e01b8780 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc2dc00b910 tx=0x7fc2dc00bcb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.801 7fc2d5ffb700 1 -- 172.21.15.86:0/2161579952 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2dc006030 con 0x7fc2e01b8380 2021-04-09T18:12:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.801 7fc2d5ffb700 1 -- 172.21.15.86:0/2161579952 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc0062b0 con 0x7fc2e01b8380 2021-04-09T18:12:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.801 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e01b8f90 con 0x7fc2e01b8380 2021-04-09T18:12:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.801 7fc2d5ffb700 1 -- 172.21.15.86:0/2161579952 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2dc005850 con 0x7fc2e01b8380 2021-04-09T18:12:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.801 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e01b8f90 con 0x7fc2e01b8380 2021-04-09T18:12:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.801 7fc2d5ffb700 1 -- 172.21.15.86:0/2161579952 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc2dc005660 con 0x7fc2e01b8380 2021-04-09T18:12:17.111 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.801 7fc2d5ffb700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c0018930 0x7fc2c001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.112 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.801 7fc2d5ffb700 1 -- 172.21.15.86:0/2161579952 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 857+0+0 (secure 0 0 0) 0x7fc2dc031cc0 con 0x7fc2e01b8380 2021-04-09T18:12:17.131 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.802 7fc2e4f0c700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c0018930 0x7fc2c001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.131 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.802 7fc2e4f0c700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c0018930 0x7fc2c001ae10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:17.146 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.825 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e01b7180 con 0x7fc2e01b8380 2021-04-09T18:12:17.147 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:16.828 7fc2d5ffb700 1 -- 172.21.15.86:0/2161579952 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc2dc00bdf0 con 0x7fc2e01b8380 2021-04-09T18:12:17.147 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.014 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", "id": 1} v 0) v1 -- 0x7fc2e008c230 con 0x7fc2e01b8380 2021-04-09T18:12:17.147 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.019 7fc2d5ffb700 1 -- 172.21.15.86:0/2161579952 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc2dc005660 con 0x7fc2e01b8380 2021-04-09T18:12:17.161 INFO:teuthology.orchestra.run.smithi086.stdout:1 2021-04-09T18:12:17.179 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c0018930 msgr2=0x7fc2c001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.179 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c0018930 0x7fc2c001ae10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.179 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e01b8380 msgr2=0x7fc2e01b8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.180 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e01b8380 0x7fc2e01b8780 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc2dc00b910 tx=0x7fc2dc00bcb0).stop 2021-04-09T18:12:17.180 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 shutdown_connections 2021-04-09T18:12:17.181 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c0018930 0x7fc2c001ae10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.181 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e008d1d0 0x7fc2e01805b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.182 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e008c7b0 0x7fc2e008cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.182 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 --2- 172.21.15.86:0/2161579952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e01b8380 0x7fc2e01b8780 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 >> 172.21.15.86:0/2161579952 conn(0x7fc2e0189800 msgr2=0x7fc2e01b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:17.199 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 shutdown_connections 2021-04-09T18:12:17.199 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.024 7fc2e78f1700 1 -- 172.21.15.86:0/2161579952 wait complete. 2021-04-09T18:12:17.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 7a25e453-ac85-4d4e-8a06-5a18f990be19 2 2021-04-09T18:12:17.397 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.350 7f8926119700 1 Processor -- start 2021-04-09T18:12:17.398 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.350 7f8926119700 1 -- start start 2021-04-09T18:12:17.398 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f8926119700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f89200cffb0 0x7f89200d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.398 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f8926119700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 0x7f892010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.399 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f8926119700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200d7290 con 0x7f89200d7810 2021-04-09T18:12:17.399 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f8926119700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89200d7090 con 0x7f89200cffb0 2021-04-09T18:12:17.400 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f8926119700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89200d91f0 con 0x7f89200d6c90 2021-04-09T18:12:17.400 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891f7fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f89200cffb0 0x7f89200d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.400 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891ffff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 0x7f892010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.401 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891ffff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 0x7f892010cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51986/0 (socket says 172.21.15.86:51986) 2021-04-09T18:12:17.417 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891f7fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f89200cffb0 0x7f89200d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50770/0 (socket says 172.21.15.86:50770) 2021-04-09T18:12:17.418 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891ffff700 1 -- 172.21.15.86:0/202228414 learned_addr learned my addr 172.21.15.86:0/202228414 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:17.418 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891effd700 1 --1- 172.21.15.86:0/202228414 >> v1:172.21.15.32:6789/0 conn(0x7f89200d7810 0x7f89200d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.86:36266/0 (socket says 172.21.15.86:36266) 2021-04-09T18:12:17.436 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891ffff700 1 -- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 msgr2=0x7f892010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-04-09T18:12:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891ffff700 1 -- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 msgr2=0x7f892010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-04-09T18:12:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891ffff700 1 --2- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 0x7f892010cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.351 7f891ffff700 1 --2- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 0x7f892010cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f891f7fe700 1 -- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 msgr2=0x7f892010cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-04-09T18:12:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f891f7fe700 1 --2- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 0x7f892010cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f891e7fc700 1 -- 172.21.15.86:0/202228414 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909144269 0 0) 0x7f89140009b0 con 0x7f89200d7810 2021-04-09T18:12:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f891f7fe700 1 -- 172.21.15.86:0/202228414 >> v1:172.21.15.32:6789/0 conn(0x7f89200d7810 legacy=0x7f89200d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f891f7fe700 1 -- 172.21.15.86:0/202228414 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200d8400 con 0x7f89200cffb0 2021-04-09T18:12:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f891f7fe700 1 --2- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89200cffb0 0x7f89200d2450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f89100069d0 tx=0x7f8910009e40).ready entity=mon.1 client_cookie=b0470b22cf3ee320 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f891e7fc700 1 -- 172.21.15.86:0/202228414 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f891000b510 con 0x7f89200cffb0 2021-04-09T18:12:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f891e7fc700 1 -- 172.21.15.86:0/202228414 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891000b890 con 0x7f89200cffb0 2021-04-09T18:12:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f891e7fc700 1 -- 172.21.15.86:0/202228414 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f891000bd70 con 0x7f89200cffb0 2021-04-09T18:12:17.462 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f8926119700 1 -- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89200cffb0 msgr2=0x7f89200d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.463 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.352 7f8926119700 1 --2- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89200cffb0 0x7f89200d2450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f89100069d0 tx=0x7f8910009e40).stop 2021-04-09T18:12:17.463 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.353 7f8926119700 1 -- 172.21.15.86:0/202228414 shutdown_connections 2021-04-09T18:12:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.353 7f8926119700 1 --2- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89200d6c90 0x7f892010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.353 7f8926119700 1 --2- 172.21.15.86:0/202228414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89200cffb0 0x7f89200d2450 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.353 7f8926119700 1 -- 172.21.15.86:0/202228414 >> 172.21.15.86:0/202228414 conn(0x7f89200ce5d0 msgr2=0x7f89200cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.353 7f8926119700 1 -- 172.21.15.86:0/202228414 shutdown_connections 2021-04-09T18:12:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.353 7f8926119700 1 -- 172.21.15.86:0/202228414 wait complete. 2021-04-09T18:12:17.485 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.371 7f8926119700 1 Processor -- start 2021-04-09T18:12:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f8926119700 1 -- start start 2021-04-09T18:12:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f8926119700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f892008a620 0x7f892008b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f8926119700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892008b6d0 0x7f8920180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.493 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f8926119700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8920180c50 0x7f8920181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.494 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f8926119700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892008e660 con 0x7f892008a620 2021-04-09T18:12:17.494 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f8926119700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892008ab30 con 0x7f8920180c50 2021-04-09T18:12:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f8926119700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892008bad0 con 0x7f892008b6d0 2021-04-09T18:12:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f891ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8920180c50 0x7f8920181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f891effd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892008b6d0 0x7f8920180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f891ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8920180c50 0x7f8920181050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50780/0 (socket says 172.21.15.86:50780) 2021-04-09T18:12:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f891ffff700 1 -- 172.21.15.86:0/872415077 learned_addr learned my addr 172.21.15.86:0/872415077 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f891f7fe700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f892008a620 0x7f892008b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f891effd700 1 -- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892008b6d0 msgr2=0x7f8920180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-04-09T18:12:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f891effd700 1 -- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892008b6d0 msgr2=0x7f8920180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-04-09T18:12:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f891effd700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892008b6d0 0x7f8920180630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-04-09T18:12:17.503 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.372 7f891effd700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892008b6d0 0x7f8920180630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-04-09T18:12:17.503 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f891ffff700 1 -- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892008b6d0 msgr2=0x7f8920180630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-04-09T18:12:17.504 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f891ffff700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892008b6d0 0x7f8920180630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.504 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f891ffff700 1 -- 172.21.15.86:0/872415077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f892008a620 msgr2=0x7f892008b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.505 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f891ffff700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f892008a620 0x7f892008b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.505 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f891ffff700 1 -- 172.21.15.86:0/872415077 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892008e1f0 con 0x7f8920180c50 2021-04-09T18:12:17.505 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f891f7fe700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f892008a620 0x7f892008b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:17.552 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f891ffff700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8920180c50 0x7f8920181050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f891800dfc0 tx=0x7f891800b890).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:17.555 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f8907fff700 1 -- 172.21.15.86:0/872415077 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8918006a80 con 0x7f8920180c50 2021-04-09T18:12:17.556 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f8907fff700 1 -- 172.21.15.86:0/872415077 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891800bc80 con 0x7f8920180c50 2021-04-09T18:12:17.556 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f8907fff700 1 -- 172.21.15.86:0/872415077 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89180022a0 con 0x7f8920180c50 2021-04-09T18:12:17.556 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f8926119700 1 -- 172.21.15.86:0/872415077 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892008e1f0 con 0x7f8920180c50 2021-04-09T18:12:17.557 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f8926119700 1 -- 172.21.15.86:0/872415077 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892008e1f0 con 0x7f8920180c50 2021-04-09T18:12:17.557 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.373 7f8907fff700 1 -- 172.21.15.86:0/872415077 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8918000b50 con 0x7f8920180c50 2021-04-09T18:12:17.557 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.374 7f8907fff700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8900018930 0x7f890001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.558 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.374 7f8907fff700 1 -- 172.21.15.86:0/872415077 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 960+0+0 (secure 0 0 0) 0x7f89180317f0 con 0x7f8920180c50 2021-04-09T18:12:17.558 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.374 7f891f7fe700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8900018930 0x7f890001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.558 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.374 7f891f7fe700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8900018930 0x7f890001ae10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:17.572 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.397 7f8926119700 1 -- 172.21.15.86:0/872415077 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89201b7180 con 0x7f8920180c50 2021-04-09T18:12:17.572 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.400 7f8907fff700 1 -- 172.21.15.86:0/872415077 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8918000940 con 0x7f8920180c50 2021-04-09T18:12:17.607 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.589 7f8926119700 1 -- 172.21.15.86:0/872415077 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", "id": 2} v 0) v1 -- 0x7f89200d7290 con 0x7f8920180c50 2021-04-09T18:12:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.593 7f8907fff700 1 -- 172.21.15.86:0/872415077 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8918000b50 con 0x7f8920180c50 2021-04-09T18:12:17.615 INFO:teuthology.orchestra.run.smithi086.stdout:2 2021-04-09T18:12:17.621 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 -- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8900018930 msgr2=0x7f890001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.622 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8900018930 0x7f890001ae10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.622 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 -- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8920180c50 msgr2=0x7f8920181050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.623 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8920180c50 0x7f8920181050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f891800dfc0 tx=0x7f891800b890).stop 2021-04-09T18:12:17.623 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 -- 172.21.15.86:0/872415077 shutdown_connections 2021-04-09T18:12:17.623 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8900018930 0x7f890001ae10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.636 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8920180c50 0x7f8920181050 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.636 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892008b6d0 0x7f8920180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.637 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 --2- 172.21.15.86:0/872415077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f892008a620 0x7f892008b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.666 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.598 7f8926119700 1 -- 172.21.15.86:0/872415077 >> 172.21.15.86:0/872415077 conn(0x7f8920189800 msgr2=0x7f89201b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:17.667 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.599 7f8926119700 1 -- 172.21.15.86:0/872415077 shutdown_connections 2021-04-09T18:12:17.667 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.599 7f8926119700 1 -- 172.21.15.86:0/872415077 wait complete. 2021-04-09T18:12:17.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new fc7ef9cf-5cd2-4fd2-8112-245a5141a327 3 2021-04-09T18:12:17.844 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3eea8b9700 1 Processor -- start 2021-04-09T18:12:17.848 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3eea8b9700 1 -- start start 2021-04-09T18:12:17.848 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3eea8b9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ee40d6c90 0x7f3ee40d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.849 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3eea8b9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3ee40d2450 0x7f3ee410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.849 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3eea8b9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee40d7290 con 0x7f3ee40cffb0 2021-04-09T18:12:17.851 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3eea8b9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee40d7090 con 0x7f3ee40d2450 2021-04-09T18:12:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3eea8b9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee40d91f0 con 0x7f3ee40d6c90 2021-04-09T18:12:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3edbfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ee40d6c90 0x7f3ee40d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3ee3fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3ee40cffb0 0x7f3ee40d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42830/0 (socket says 172.21.15.86:42830) 2021-04-09T18:12:17.855 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.797 7f3ee3fff700 1 -- 172.21.15.86:0/278241678 learned_addr learned my addr 172.21.15.86:0/278241678 (peer_addr_for_me v1:172.21.15.86:0/0) 2021-04-09T18:12:17.855 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3ee8ed6700 1 --2- 172.21.15.86:0/278241678 >> v2:172.21.15.32:3300/0 conn(0x7f3ee40d2450 0x7f3ee410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.856 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3edbfff700 1 -- 172.21.15.86:0/278241678 >> v2:172.21.15.32:3300/0 conn(0x7f3ee40d2450 msgr2=0x7f3ee410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:17.856 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3edbfff700 1 --2- 172.21.15.86:0/278241678 >> v2:172.21.15.32:3300/0 conn(0x7f3ee40d2450 0x7f3ee410cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.856 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3edbfff700 1 -- 172.21.15.86:0/278241678 >> v1:172.21.15.86:6789/0 conn(0x7f3ee40cffb0 legacy=0x7f3ee40d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.862 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3edbfff700 1 -- 172.21.15.86:0/278241678 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40d2850 con 0x7f3ee40d6c90 2021-04-09T18:12:17.866 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3ee37fe700 1 -- 172.21.15.86:0/278241678 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625930787 0 0) 0x7f3ed4000ce0 con 0x7f3ee40cffb0 2021-04-09T18:12:17.866 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3edbfff700 1 --2- 172.21.15.86:0/278241678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee40d6c90 0x7f3ee40d8000 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f3ecc006650 tx=0x7f3ecc009a00).ready entity=mon.1 client_cookie=c511cb1892159937 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:17.869 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3ee37fe700 1 -- 172.21.15.86:0/278241678 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ecc009df0 con 0x7f3ee40d6c90 2021-04-09T18:12:17.869 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3ee37fe700 1 -- 172.21.15.86:0/278241678 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ecc0009b0 con 0x7f3ee40d6c90 2021-04-09T18:12:17.870 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3ee37fe700 1 -- 172.21.15.86:0/278241678 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ecc00c2b0 con 0x7f3ee40d6c90 2021-04-09T18:12:17.870 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3eea8b9700 1 -- 172.21.15.86:0/278241678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee40d6c90 msgr2=0x7f3ee40d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.870 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.798 7f3eea8b9700 1 --2- 172.21.15.86:0/278241678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee40d6c90 0x7f3ee40d8000 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f3ecc006650 tx=0x7f3ecc009a00).stop 2021-04-09T18:12:17.870 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.799 7f3eea8b9700 1 -- 172.21.15.86:0/278241678 shutdown_connections 2021-04-09T18:12:17.876 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.799 7f3eea8b9700 1 --2- 172.21.15.86:0/278241678 >> v2:172.21.15.32:3300/0 conn(0x7f3ee40d2450 0x7f3ee410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.877 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.799 7f3eea8b9700 1 --2- 172.21.15.86:0/278241678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee40d6c90 0x7f3ee40d8000 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.877 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.799 7f3eea8b9700 1 -- 172.21.15.86:0/278241678 >> 172.21.15.86:0/278241678 conn(0x7f3ee40ce5d0 msgr2=0x7f3ee40cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:17.877 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.799 7f3eea8b9700 1 -- 172.21.15.86:0/278241678 shutdown_connections 2021-04-09T18:12:17.877 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.799 7f3eea8b9700 1 -- 172.21.15.86:0/278241678 wait complete. 2021-04-09T18:12:17.878 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3eea8b9700 1 Processor -- start 2021-04-09T18:12:17.878 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3eea8b9700 1 -- start start 2021-04-09T18:12:17.878 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3eea8b9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ee41a7dc0 0x7f3ee41a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.878 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3eea8b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee4184080 0x7f3ee4184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.879 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3eea8b9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee4184aa0 0x7f3ee408f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.879 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3eea8b9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee41a8b40 con 0x7f3ee4184080 2021-04-09T18:12:17.879 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3eea8b9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee41a7870 con 0x7f3ee4184aa0 2021-04-09T18:12:17.879 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3eea8b9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee41a87e0 con 0x7f3ee41a7dc0 2021-04-09T18:12:17.879 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3ee3fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ee41a7dc0 0x7f3ee41a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.888 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3ee8ed6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee4184aa0 0x7f3ee408f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.888 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3ee3fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ee41a7dc0 0x7f3ee41a81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52006/0 (socket says 172.21.15.86:52006) 2021-04-09T18:12:17.889 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.818 7f3ee3fff700 1 -- 172.21.15.86:0/375384832 learned_addr learned my addr 172.21.15.86:0/375384832 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:17.889 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3edbfff700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee4184080 0x7f3ee4184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.891 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee8ed6700 1 -- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ee41a7dc0 msgr2=0x7f3ee41a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.891 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee8ed6700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ee41a7dc0 0x7f3ee41a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.892 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee8ed6700 1 -- 172.21.15.86:0/375384832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee4184080 msgr2=0x7f3ee4184480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:17.895 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee8ed6700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee4184080 0x7f3ee4184480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:17.895 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee8ed6700 1 -- 172.21.15.86:0/375384832 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee408f810 con 0x7f3ee4184aa0 2021-04-09T18:12:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee3fff700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ee41a7dc0 0x7f3ee41a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:17.896 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3edbfff700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee4184080 0x7f3ee4184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:17.897 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee8ed6700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee4184aa0 0x7f3ee408f1f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f3edc00b8f0 tx=0x7f3edc00bc90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:17.897 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee17fa700 1 -- 172.21.15.86:0/375384832 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3edc0011f0 con 0x7f3ee4184aa0 2021-04-09T18:12:17.897 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee17fa700 1 -- 172.21.15.86:0/375384832 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc003030 con 0x7f3ee4184aa0 2021-04-09T18:12:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee408f810 con 0x7f3ee4184aa0 2021-04-09T18:12:17.968 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3ee17fa700 1 -- 172.21.15.86:0/375384832 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3edc0017b0 con 0x7f3ee4184aa0 2021-04-09T18:12:17.968 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.819 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee408f810 con 0x7f3ee4184aa0 2021-04-09T18:12:17.969 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.820 7f3ee17fa700 1 -- 172.21.15.86:0/375384832 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3edc0014b0 con 0x7f3ee4184aa0 2021-04-09T18:12:17.969 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.820 7f3ee17fa700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc0188d0 0x7f3ebc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:17.969 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.820 7f3ee17fa700 1 -- 172.21.15.86:0/375384832 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1063+0+0 (secure 0 0 0) 0x7f3edc031b00 con 0x7f3ee4184aa0 2021-04-09T18:12:17.970 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.820 7f3ee3fff700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc0188d0 0x7f3ebc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:17.970 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.820 7f3ee3fff700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc0188d0 0x7f3ebc01add0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:17.972 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.843 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee4182e80 con 0x7f3ee4184aa0 2021-04-09T18:12:17.987 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:17.846 7f3ee17fa700 1 -- 172.21.15.86:0/375384832 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3edc00bdd0 con 0x7f3ee4184aa0 2021-04-09T18:12:18.054 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.033 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", "id": 3} v 0) v1 -- 0x7f3ee40d7290 con 0x7f3ee4184aa0 2021-04-09T18:12:18.056 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.037 7f3ee17fa700 1 -- 172.21.15.86:0/375384832 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3edc001430 con 0x7f3ee4184aa0 2021-04-09T18:12:18.060 INFO:teuthology.orchestra.run.smithi086.stdout:3 2021-04-09T18:12:18.062 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc0188d0 msgr2=0x7f3ebc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.068 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc0188d0 0x7f3ebc01add0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.069 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee4184aa0 msgr2=0x7f3ee408f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.069 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee4184aa0 0x7f3ee408f1f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f3edc00b8f0 tx=0x7f3edc00bc90).stop 2021-04-09T18:12:18.070 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 shutdown_connections 2021-04-09T18:12:18.070 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc0188d0 0x7f3ebc01add0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.070 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee4184080 0x7f3ee4184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.071 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ee4184aa0 0x7f3ee408f1f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.071 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 --2- 172.21.15.86:0/375384832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ee41a7dc0 0x7f3ee41a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.072 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 >> 172.21.15.86:0/375384832 conn(0x7f3ee41a8f50 msgr2=0x7f3ee4188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:18.072 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 shutdown_connections 2021-04-09T18:12:18.072 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.042 7f3eea8b9700 1 -- 172.21.15.86:0/375384832 wait complete. 2021-04-09T18:12:18.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 122503dc-3cb6-4959-b287-cbae5ce53e7d 4 2021-04-09T18:12:18.290 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48cef8a700 1 Processor -- start 2021-04-09T18:12:18.292 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48cef8a700 1 -- start start 2021-04-09T18:12:18.292 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48cef8a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c80d6c90 0x7f48c80d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.293 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48cef8a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f48c80d2450 0x7f48c810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.293 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48cef8a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c80d7290 con 0x7f48c80cffb0 2021-04-09T18:12:18.294 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48cef8a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c80d7090 con 0x7f48c80d2450 2021-04-09T18:12:18.294 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48cef8a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c80d91f0 con 0x7f48c80d6c90 2021-04-09T18:12:18.294 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48c7fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c80d6c90 0x7f48c80d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.295 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48cd5a7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f48c80d2450 0x7f48c810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.295 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48c7fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c80d6c90 0x7f48c80d8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52014/0 (socket says 172.21.15.86:52014) 2021-04-09T18:12:18.296 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48cd5a7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f48c80d2450 0x7f48c810cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50798/0 (socket says 172.21.15.86:50798) 2021-04-09T18:12:18.297 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.244 7f48c7fff700 1 -- 172.21.15.86:0/805105570 learned_addr learned my addr 172.21.15.86:0/805105570 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:18.326 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48c7fff700 1 -- 172.21.15.86:0/805105570 >> v2:172.21.15.86:3300/0 conn(0x7f48c80d2450 msgr2=0x7f48c810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:18.327 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48c7fff700 1 --2- 172.21.15.86:0/805105570 >> v2:172.21.15.86:3300/0 conn(0x7f48c80d2450 0x7f48c810cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.327 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48c7fff700 1 -- 172.21.15.86:0/805105570 >> v1:172.21.15.32:6789/0 conn(0x7f48c80cffb0 legacy=0x7f48c80d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.327 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48c77fe700 1 -- 172.21.15.86:0/805105570 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640724582 0 0) 0x7f48b0000c60 con 0x7f48c80cffb0 2021-04-09T18:12:18.327 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48c7fff700 1 -- 172.21.15.86:0/805105570 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c80d2850 con 0x7f48c80d6c90 2021-04-09T18:12:18.328 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48cd5a7700 1 --2- 172.21.15.86:0/805105570 >> v2:172.21.15.86:3300/0 conn(0x7f48c80d2450 0x7f48c810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:18.328 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48c7fff700 1 --2- 172.21.15.86:0/805105570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c80d6c90 0x7f48c80d8000 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f48b8006a80 tx=0x7f48b8009f00).ready entity=mon.2 client_cookie=e5f045466aead7e0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:18.334 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48c77fe700 1 -- 172.21.15.86:0/805105570 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b800b5c0 con 0x7f48c80d6c90 2021-04-09T18:12:18.334 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48c77fe700 1 -- 172.21.15.86:0/805105570 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b800b5c0 con 0x7f48c80d6c90 2021-04-09T18:12:18.334 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48c77fe700 1 -- 172.21.15.86:0/805105570 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b800b8c0 con 0x7f48c80d6c90 2021-04-09T18:12:18.335 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48cef8a700 1 -- 172.21.15.86:0/805105570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c80d6c90 msgr2=0x7f48c80d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.335 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.245 7f48cef8a700 1 --2- 172.21.15.86:0/805105570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c80d6c90 0x7f48c80d8000 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f48b8006a80 tx=0x7f48b8009f00).stop 2021-04-09T18:12:18.335 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.246 7f48cef8a700 1 -- 172.21.15.86:0/805105570 shutdown_connections 2021-04-09T18:12:18.335 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.246 7f48cef8a700 1 --2- 172.21.15.86:0/805105570 >> v2:172.21.15.86:3300/0 conn(0x7f48c80d2450 0x7f48c810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.335 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.246 7f48cef8a700 1 --2- 172.21.15.86:0/805105570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c80d6c90 0x7f48c80d8000 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.336 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.246 7f48cef8a700 1 -- 172.21.15.86:0/805105570 >> 172.21.15.86:0/805105570 conn(0x7f48c80ce5d0 msgr2=0x7f48c80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:18.336 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.246 7f48cef8a700 1 -- 172.21.15.86:0/805105570 shutdown_connections 2021-04-09T18:12:18.336 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.246 7f48cef8a700 1 -- 172.21.15.86:0/805105570 wait complete. 2021-04-09T18:12:18.336 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.265 7f48cef8a700 1 Processor -- start 2021-04-09T18:12:18.337 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.265 7f48cef8a700 1 -- start start 2021-04-09T18:12:18.337 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48cef8a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c81a7dc0 0x7f48c81a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.337 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48cef8a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c8184080 0x7f48c8184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.337 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48cef8a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c8184aa0 0x7f48c808f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.337 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48cef8a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81a8b40 con 0x7f48c8184aa0 2021-04-09T18:12:18.338 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48cef8a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81a7870 con 0x7f48c81a7dc0 2021-04-09T18:12:18.338 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48cef8a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81a87e0 con 0x7f48c8184080 2021-04-09T18:12:18.338 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48c7fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c8184080 0x7f48c8184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.338 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48ccda6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c81a7dc0 0x7f48c81a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.359 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48c7fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c8184080 0x7f48c8184480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52020/0 (socket says 172.21.15.86:52020) 2021-04-09T18:12:18.360 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48c7fff700 1 -- 172.21.15.86:0/3957467204 learned_addr learned my addr 172.21.15.86:0/3957467204 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:18.360 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48cd5a7700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c8184aa0 0x7f48c808f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.361 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48ccda6700 1 -- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c8184080 msgr2=0x7f48c8184480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.361 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48ccda6700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c8184080 0x7f48c8184480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.361 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48ccda6700 1 -- 172.21.15.86:0/3957467204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c8184aa0 msgr2=0x7f48c808f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.361 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48ccda6700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c8184aa0 0x7f48c808f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.361 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48ccda6700 1 -- 172.21.15.86:0/3957467204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c808f810 con 0x7f48c81a7dc0 2021-04-09T18:12:18.362 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48c7fff700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c8184080 0x7f48c8184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:18.362 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48ccda6700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c81a7dc0 0x7f48c81a81c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f48b000b510 tx=0x7f48b000b8b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:18.368 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48c57fa700 1 -- 172.21.15.86:0/3957467204 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b000bca0 con 0x7f48c81a7dc0 2021-04-09T18:12:18.371 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48c57fa700 1 -- 172.21.15.86:0/3957467204 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b00066d0 con 0x7f48c81a7dc0 2021-04-09T18:12:18.372 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c808f810 con 0x7f48c81a7dc0 2021-04-09T18:12:18.372 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48c57fa700 1 -- 172.21.15.86:0/3957467204 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b0001510 con 0x7f48c81a7dc0 2021-04-09T18:12:18.372 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.266 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c808f810 con 0x7f48c81a7dc0 2021-04-09T18:12:18.375 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.267 7f48c57fa700 1 -- 172.21.15.86:0/3957467204 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f48b0000a50 con 0x7f48c81a7dc0 2021-04-09T18:12:18.385 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.267 7f48c57fa700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48ac018930 0x7f48ac01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.386 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.267 7f48c57fa700 1 -- 172.21.15.86:0/3957467204 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1166+0+0 (secure 0 0 0) 0x7f48b0031800 con 0x7f48c81a7dc0 2021-04-09T18:12:18.386 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.267 7f48c7fff700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48ac018930 0x7f48ac01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.386 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.268 7f48c7fff700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48ac018930 0x7f48ac01ae10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:18.388 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.290 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c8182e80 con 0x7f48c81a7dc0 2021-04-09T18:12:18.388 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.293 7f48c57fa700 1 -- 172.21.15.86:0/3957467204 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f48b0000a50 con 0x7f48c81a7dc0 2021-04-09T18:12:18.499 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.481 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", "id": 4} v 0) v1 -- 0x7f48c80d7290 con 0x7f48c81a7dc0 2021-04-09T18:12:18.502 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.484 7f48c57fa700 1 -- 172.21.15.86:0/3957467204 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f48b00066d0 con 0x7f48c81a7dc0 2021-04-09T18:12:18.506 INFO:teuthology.orchestra.run.smithi086.stdout:4 2021-04-09T18:12:18.509 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48ac018930 msgr2=0x7f48ac01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48ac018930 0x7f48ac01ae10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c81a7dc0 msgr2=0x7f48c81a81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c81a7dc0 0x7f48c81a81c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f48b000b510 tx=0x7f48b000b8b0).stop 2021-04-09T18:12:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 shutdown_connections 2021-04-09T18:12:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48ac018930 0x7f48ac01ae10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c8184aa0 0x7f48c808f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c8184080 0x7f48c8184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 --2- 172.21.15.86:0/3957467204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c81a7dc0 0x7f48c81a81c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 >> 172.21.15.86:0/3957467204 conn(0x7f48c81a8f50 msgr2=0x7f48c8188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 shutdown_connections 2021-04-09T18:12:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.490 7f48cef8a700 1 -- 172.21.15.86:0/3957467204 wait complete. 2021-04-09T18:12:18.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 097269c3-c880-4660-a0c2-12cd6fc7bcf9 5 2021-04-09T18:12:18.686 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.640 7fc42c2df700 1 Processor -- start 2021-04-09T18:12:18.686 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.640 7fc42c2df700 1 -- start start 2021-04-09T18:12:18.687 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42c2df700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4240d6c90 0x7fc4240d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.687 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42c2df700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc4240d2450 0x7fc42410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.689 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42c2df700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4240d7290 con 0x7fc4240cffb0 2021-04-09T18:12:18.689 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42c2df700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4240d7090 con 0x7fc4240d2450 2021-04-09T18:12:18.689 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42c2df700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240d91f0 con 0x7fc4240d6c90 2021-04-09T18:12:18.691 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42a8fc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc4240d2450 0x7fc42410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.691 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc4298fa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4240d6c90 0x7fc4240d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.692 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42a8fc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc4240d2450 0x7fc42410cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50812/0 (socket says 172.21.15.86:50812) 2021-04-09T18:12:18.692 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc4298fa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4240d6c90 0x7fc4240d8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52030/0 (socket says 172.21.15.86:52030) 2021-04-09T18:12:18.693 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42a8fc700 1 -- 172.21.15.86:0/3432506784 learned_addr learned my addr 172.21.15.86:0/3432506784 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:18.693 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42a8fc700 1 -- 172.21.15.86:0/3432506784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4240d6c90 msgr2=0x7fc4240d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:18.694 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42a8fc700 1 --2- 172.21.15.86:0/3432506784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4240d6c90 0x7fc4240d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.694 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.641 7fc42a8fc700 1 -- 172.21.15.86:0/3432506784 >> v1:172.21.15.32:6789/0 conn(0x7fc4240cffb0 legacy=0x7fc4240d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.695 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.642 7fc42a8fc700 1 -- 172.21.15.86:0/3432506784 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4240d2850 con 0x7fc4240d2450 2021-04-09T18:12:18.717 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.642 7fc4298fa700 1 --2- 172.21.15.86:0/3432506784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4240d6c90 0x7fc4240d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:18.720 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.642 7fc42a8fc700 1 --2- 172.21.15.86:0/3432506784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4240d2450 0x7fc42410cfe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fc41c006b80 tx=0x7fc41c009b90).ready entity=mon.1 client_cookie=e3e9be577dfb87b6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:18.723 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.642 7fc4290f9700 1 -- 172.21.15.86:0/3432506784 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc41c0022e0 con 0x7fc4240d2450 2021-04-09T18:12:18.723 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.642 7fc4290f9700 1 -- 172.21.15.86:0/3432506784 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c002620 con 0x7fc4240d2450 2021-04-09T18:12:18.724 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.642 7fc4290f9700 1 -- 172.21.15.86:0/3432506784 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc41c002b00 con 0x7fc4240d2450 2021-04-09T18:12:18.724 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.642 7fc42c2df700 1 -- 172.21.15.86:0/3432506784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4240d2450 msgr2=0x7fc42410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.724 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.642 7fc42c2df700 1 --2- 172.21.15.86:0/3432506784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4240d2450 0x7fc42410cfe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fc41c006b80 tx=0x7fc41c009b90).stop 2021-04-09T18:12:18.725 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.643 7fc42c2df700 1 -- 172.21.15.86:0/3432506784 shutdown_connections 2021-04-09T18:12:18.764 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.643 7fc42c2df700 1 --2- 172.21.15.86:0/3432506784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4240d2450 0x7fc42410cfe0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.765 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.643 7fc42c2df700 1 --2- 172.21.15.86:0/3432506784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4240d6c90 0x7fc4240d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.765 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.643 7fc42c2df700 1 -- 172.21.15.86:0/3432506784 >> 172.21.15.86:0/3432506784 conn(0x7fc4240ce5d0 msgr2=0x7fc4240cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:18.796 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.643 7fc42c2df700 1 -- 172.21.15.86:0/3432506784 shutdown_connections 2021-04-09T18:12:18.799 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.643 7fc42c2df700 1 -- 172.21.15.86:0/3432506784 wait complete. 2021-04-09T18:12:18.799 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.661 7fc42c2df700 1 Processor -- start 2021-04-09T18:12:18.808 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.661 7fc42c2df700 1 -- start start 2021-04-09T18:12:18.813 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42c2df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc42408a620 0x7fc42408b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.813 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42c2df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc42408b6d0 0x7fc424180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.815 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42c2df700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc424180c50 0x7fc424181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.815 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42c2df700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42408e660 con 0x7fc42408a620 2021-04-09T18:12:18.819 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42c2df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42408ab30 con 0x7fc42408b6d0 2021-04-09T18:12:18.820 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42c2df700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42408bad0 con 0x7fc424180c50 2021-04-09T18:12:18.831 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42a8fc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc424180c50 0x7fc424181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.832 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc4298fa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc42408b6d0 0x7fc424180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.832 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42a8fc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc424180c50 0x7fc424181050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52034/0 (socket says 172.21.15.86:52034) 2021-04-09T18:12:18.832 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42a8fc700 1 -- 172.21.15.86:0/4138899774 learned_addr learned my addr 172.21.15.86:0/4138899774 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:18.833 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc42a0fb700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc42408a620 0x7fc42408b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.833 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc4298fa700 1 -- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc424180c50 msgr2=0x7fc424181050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.834 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc4298fa700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc424180c50 0x7fc424181050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.834 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc4298fa700 1 -- 172.21.15.86:0/4138899774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc42408a620 msgr2=0x7fc42408b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.843 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc4298fa700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc42408a620 0x7fc42408b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.844 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.662 7fc4298fa700 1 -- 172.21.15.86:0/4138899774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42408e1f0 con 0x7fc42408b6d0 2021-04-09T18:12:18.844 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.663 7fc4298fa700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc42408b6d0 0x7fc424180630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc414009540 tx=0x7fc41400b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:18.844 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.663 7fc41affd700 1 -- 172.21.15.86:0/4138899774 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc41400bcd0 con 0x7fc42408b6d0 2021-04-09T18:12:18.845 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.663 7fc41affd700 1 -- 172.21.15.86:0/4138899774 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc414006740 con 0x7fc42408b6d0 2021-04-09T18:12:18.845 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.663 7fc41affd700 1 -- 172.21.15.86:0/4138899774 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc414002220 con 0x7fc42408b6d0 2021-04-09T18:12:18.855 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.663 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42408e1f0 con 0x7fc42408b6d0 2021-04-09T18:12:18.856 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.663 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42408e1f0 con 0x7fc42408b6d0 2021-04-09T18:12:18.856 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.664 7fc41affd700 1 -- 172.21.15.86:0/4138899774 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc414009fe0 con 0x7fc42408b6d0 2021-04-09T18:12:18.857 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.664 7fc41affd700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc40c018930 0x7fc40c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:18.857 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.664 7fc41affd700 1 -- 172.21.15.86:0/4138899774 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1269+0+0 (secure 0 0 0) 0x7fc414031890 con 0x7fc42408b6d0 2021-04-09T18:12:18.858 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.664 7fc42a0fb700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc40c018930 0x7fc40c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:18.858 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.664 7fc42a0fb700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc40c018930 0x7fc40c01adc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:18.873 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.687 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4241b7180 con 0x7fc42408b6d0 2021-04-09T18:12:18.874 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.690 7fc41affd700 1 -- 172.21.15.86:0/4138899774 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc414006740 con 0x7fc42408b6d0 2021-04-09T18:12:18.897 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.875 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", "id": 5} v 0) v1 -- 0x7fc4240d7290 con 0x7fc42408b6d0 2021-04-09T18:12:18.905 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.879 7fc41affd700 1 -- 172.21.15.86:0/4138899774 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc414000c90 con 0x7fc42408b6d0 2021-04-09T18:12:18.906 INFO:teuthology.orchestra.run.smithi086.stdout:5 2021-04-09T18:12:18.909 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc40c018930 msgr2=0x7fc40c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.909 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc40c018930 0x7fc40c01adc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc42408b6d0 msgr2=0x7fc424180630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc42408b6d0 0x7fc424180630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc414009540 tx=0x7fc41400b8e0).stop 2021-04-09T18:12:18.911 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 shutdown_connections 2021-04-09T18:12:18.911 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc40c018930 0x7fc40c01adc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.912 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc424180c50 0x7fc424181050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc42408b6d0 0x7fc424180630 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 --2- 172.21.15.86:0/4138899774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc42408a620 0x7fc42408b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:18.962 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 >> 172.21.15.86:0/4138899774 conn(0x7fc424189800 msgr2=0x7fc4241b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:18.962 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 shutdown_connections 2021-04-09T18:12:18.963 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:18.885 7fc42c2df700 1 -- 172.21.15.86:0/4138899774 wait complete. 2021-04-09T18:12:18.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 2a3e3f80-b45d-4425-8527-c07a38cea770 6 2021-04-09T18:12:19.180 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.135 7f1e39044700 1 Processor -- start 2021-04-09T18:12:19.181 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.135 7f1e39044700 1 -- start start 2021-04-09T18:12:19.181 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e39044700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1e340cffb0 0x7f1e340d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.181 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e39044700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e340d6c90 0x7f1e3410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.182 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e39044700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e340d7290 con 0x7f1e340d7810 2021-04-09T18:12:19.182 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e39044700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e340d7090 con 0x7f1e340cffb0 2021-04-09T18:12:19.183 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e39044700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340d91f0 con 0x7f1e340d6c90 2021-04-09T18:12:19.183 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e3359e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e340d6c90 0x7f1e3410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.184 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e32d9d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1e340cffb0 0x7f1e340d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.184 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e32d9d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1e340cffb0 0x7f1e340d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50824/0 (socket says 172.21.15.86:50824) 2021-04-09T18:12:19.185 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e3359e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e340d6c90 0x7f1e3410cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52040/0 (socket says 172.21.15.86:52040) 2021-04-09T18:12:19.185 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e32d9d700 1 -- 172.21.15.86:0/126248259 learned_addr learned my addr 172.21.15.86:0/126248259 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:19.186 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e3359e700 1 -- 172.21.15.86:0/126248259 >> v2:172.21.15.86:3300/0 conn(0x7f1e340cffb0 msgr2=0x7f1e340d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:19.186 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e3359e700 1 --2- 172.21.15.86:0/126248259 >> v2:172.21.15.86:3300/0 conn(0x7f1e340cffb0 0x7f1e340d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.187 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e3359e700 1 -- 172.21.15.86:0/126248259 >> v1:172.21.15.32:6789/0 conn(0x7f1e340d7810 legacy=0x7f1e340d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.187 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e3359e700 1 -- 172.21.15.86:0/126248259 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e340d8400 con 0x7f1e340d6c90 2021-04-09T18:12:19.188 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e31d9b700 1 -- 172.21.15.86:0/126248259 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283344949 0 0) 0x7f1e1c001330 con 0x7f1e340d7810 2021-04-09T18:12:19.188 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.136 7f1e32d9d700 1 --2- 172.21.15.86:0/126248259 >> v2:172.21.15.86:3300/0 conn(0x7f1e340cffb0 0x7f1e340d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:19.188 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.137 7f1e3359e700 1 --2- 172.21.15.86:0/126248259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e340d6c90 0x7f1e3410cfe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1e2c006ab0 tx=0x7f1e2c009b40).ready entity=mon.2 client_cookie=15442e18094fcb3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:19.188 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.137 7f1e31d9b700 1 -- 172.21.15.86:0/126248259 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e2c0022e0 con 0x7f1e340d6c90 2021-04-09T18:12:19.194 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.137 7f1e31d9b700 1 -- 172.21.15.86:0/126248259 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2c002660 con 0x7f1e340d6c90 2021-04-09T18:12:19.195 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.137 7f1e31d9b700 1 -- 172.21.15.86:0/126248259 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e2c002b40 con 0x7f1e340d6c90 2021-04-09T18:12:19.195 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.137 7f1e39044700 1 -- 172.21.15.86:0/126248259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e340d6c90 msgr2=0x7f1e3410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.195 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.137 7f1e39044700 1 --2- 172.21.15.86:0/126248259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e340d6c90 0x7f1e3410cfe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1e2c006ab0 tx=0x7f1e2c009b40).stop 2021-04-09T18:12:19.195 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.138 7f1e39044700 1 -- 172.21.15.86:0/126248259 shutdown_connections 2021-04-09T18:12:19.196 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.138 7f1e39044700 1 --2- 172.21.15.86:0/126248259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e340d6c90 0x7f1e3410cfe0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.196 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.138 7f1e39044700 1 --2- 172.21.15.86:0/126248259 >> v2:172.21.15.86:3300/0 conn(0x7f1e340cffb0 0x7f1e340d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.196 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.138 7f1e39044700 1 -- 172.21.15.86:0/126248259 >> 172.21.15.86:0/126248259 conn(0x7f1e340ce5d0 msgr2=0x7f1e340cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:19.196 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.138 7f1e39044700 1 -- 172.21.15.86:0/126248259 shutdown_connections 2021-04-09T18:12:19.197 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.138 7f1e39044700 1 -- 172.21.15.86:0/126248259 wait complete. 2021-04-09T18:12:19.197 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.156 7f1e39044700 1 Processor -- start 2021-04-09T18:12:19.197 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.156 7f1e39044700 1 -- start start 2021-04-09T18:12:19.197 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e39044700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3408a620 0x7f1e3408b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.197 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e39044700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3408b6d0 0x7f1e34180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.198 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e39044700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e34180c50 0x7f1e34181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.198 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e39044700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3408e660 con 0x7f1e3408b6d0 2021-04-09T18:12:19.199 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e39044700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3408ab30 con 0x7f1e34180c50 2021-04-09T18:12:19.199 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e39044700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3408bad0 con 0x7f1e3408a620 2021-04-09T18:12:19.199 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e3359e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e34180c50 0x7f1e34181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.199 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e32d9d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3408a620 0x7f1e3408b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.200 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e3359e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e34180c50 0x7f1e34181050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50834/0 (socket says 172.21.15.86:50834) 2021-04-09T18:12:19.201 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e3359e700 1 -- 172.21.15.86:0/1098845571 learned_addr learned my addr 172.21.15.86:0/1098845571 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:19.201 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e3259c700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3408b6d0 0x7f1e34180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.201 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e32d9d700 1 -- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e34180c50 msgr2=0x7f1e34181050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.201 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e32d9d700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e34180c50 0x7f1e34181050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.201 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e32d9d700 1 -- 172.21.15.86:0/1098845571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3408b6d0 msgr2=0x7f1e34180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.202 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e32d9d700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3408b6d0 0x7f1e34180630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.202 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e32d9d700 1 -- 172.21.15.86:0/1098845571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3408e1f0 con 0x7f1e3408a620 2021-04-09T18:12:19.202 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e32d9d700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3408a620 0x7f1e3408b0b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1e24006ed0 tx=0x7f1e2400b870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:19.202 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e2b7fe700 1 -- 172.21.15.86:0/1098845571 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e240009b0 con 0x7f1e3408a620 2021-04-09T18:12:19.209 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e2b7fe700 1 -- 172.21.15.86:0/1098845571 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e24006b70 con 0x7f1e3408a620 2021-04-09T18:12:19.209 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e2b7fe700 1 -- 172.21.15.86:0/1098845571 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e240052e0 con 0x7f1e3408a620 2021-04-09T18:12:19.209 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e39044700 1 -- 172.21.15.86:0/1098845571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3408e1f0 con 0x7f1e3408a620 2021-04-09T18:12:19.210 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.157 7f1e39044700 1 -- 172.21.15.86:0/1098845571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e34181960 con 0x7f1e3408a620 2021-04-09T18:12:19.210 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.158 7f1e2b7fe700 1 -- 172.21.15.86:0/1098845571 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1e24000c90 con 0x7f1e3408a620 2021-04-09T18:12:19.211 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.158 7f1e2b7fe700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188d0 0x7f1e1801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.211 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.158 7f1e2b7fe700 1 -- 172.21.15.86:0/1098845571 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1372+0+0 (secure 0 0 0) 0x7f1e240318c0 con 0x7f1e3408a620 2021-04-09T18:12:19.215 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.158 7f1e3259c700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188d0 0x7f1e1801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.215 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.158 7f1e3259c700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188d0 0x7f1e1801add0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:19.219 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.182 7f1e39044700 1 -- 172.21.15.86:0/1098845571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e341b7180 con 0x7f1e3408a620 2021-04-09T18:12:19.219 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.185 7f1e2b7fe700 1 -- 172.21.15.86:0/1098845571 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1e24006b70 con 0x7f1e3408a620 2021-04-09T18:12:19.390 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.372 7f1e39044700 1 -- 172.21.15.86:0/1098845571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", "id": 6} v 0) v1 -- 0x7f1e340d7290 con 0x7f1e3408a620 2021-04-09T18:12:19.393 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.375 7f1e2b7fe700 1 -- 172.21.15.86:0/1098845571 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1e24000c90 con 0x7f1e3408a620 2021-04-09T18:12:19.397 INFO:teuthology.orchestra.run.smithi086.stdout:6 2021-04-09T18:12:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 -- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188d0 msgr2=0x7f1e1801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188d0 0x7f1e1801add0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 -- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3408a620 msgr2=0x7f1e3408b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3408a620 0x7f1e3408b0b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1e24006ed0 tx=0x7f1e2400b870).stop 2021-04-09T18:12:19.404 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 -- 172.21.15.86:0/1098845571 shutdown_connections 2021-04-09T18:12:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188d0 0x7f1e1801add0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3408b6d0 0x7f1e34180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.406 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e34180c50 0x7f1e34181050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.406 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 --2- 172.21.15.86:0/1098845571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3408a620 0x7f1e3408b0b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.407 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 -- 172.21.15.86:0/1098845571 >> 172.21.15.86:0/1098845571 conn(0x7f1e34189800 msgr2=0x7f1e341b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:19.407 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 -- 172.21.15.86:0/1098845571 shutdown_connections 2021-04-09T18:12:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.381 7f1e39044700 1 -- 172.21.15.86:0/1098845571 wait complete. 2021-04-09T18:12:19.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 526aa5a2-5c73-4473-a78b-240711b2015c 7 2021-04-09T18:12:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.531 7fc29190c700 1 Processor -- start 2021-04-09T18:12:19.570 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.531 7fc29190c700 1 -- start start 2021-04-09T18:12:19.570 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.531 7fc29190c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc28c0cffb0 0x7fc28c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.570 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.531 7fc29190c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0d7290 con 0x7fc28c0d7810 2021-04-09T18:12:19.570 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.531 7fc29190c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0d8400 con 0x7fc28c0d6c90 2021-04-09T18:12:19.571 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.531 7fc29190c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c0d7090 con 0x7fc28c0cffb0 2021-04-09T18:12:19.571 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.531 7fc28bd75700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc28c0d6c90 0x7fc28c10cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42886/0 (socket says 172.21.15.86:42886) 2021-04-09T18:12:19.571 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.531 7fc28bd75700 1 -- 172.21.15.86:0/613584428 learned_addr learned my addr 172.21.15.86:0/613584428 (peer_addr_for_me v1:172.21.15.86:0/0) 2021-04-09T18:12:19.571 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.531 7fc28b574700 1 --2- 172.21.15.86:0/613584428 >> v2:172.21.15.32:3300/0 conn(0x7fc28c0cffb0 0x7fc28c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.572 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28a572700 1 -- 172.21.15.86:0/613584428 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520542395 0 0) 0x7fc284000d30 con 0x7fc28c0d6c90 2021-04-09T18:12:19.572 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28a572700 1 -- 172.21.15.86:0/613584428 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc278002fe0 con 0x7fc28c0d6c90 2021-04-09T18:12:19.572 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28a572700 1 -- 172.21.15.86:0/613584428 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698639098 0 0) 0x7fc274000ff0 con 0x7fc28c0d7810 2021-04-09T18:12:19.572 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28a572700 1 -- 172.21.15.86:0/613584428 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2780042c0 con 0x7fc28c0d7810 2021-04-09T18:12:19.572 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28b574700 1 -- 172.21.15.86:0/613584428 >> v1:172.21.15.86:6789/0 conn(0x7fc28c0d6c90 legacy=0x7fc28c10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.573 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28b574700 1 -- 172.21.15.86:0/613584428 >> v1:172.21.15.32:6789/0 conn(0x7fc28c0d7810 legacy=0x7fc28c0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.573 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28b574700 1 -- 172.21.15.86:0/613584428 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0d91f0 con 0x7fc28c0cffb0 2021-04-09T18:12:19.573 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28b574700 1 --2- 172.21.15.86:0/613584428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0cffb0 0x7fc28c0d2450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fc27c006650 tx=0x7fc27c009a80).ready entity=mon.0 client_cookie=1ed58e7bb1a50520 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28a572700 1 -- 172.21.15.86:0/613584428 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc27c0009b0 con 0x7fc28c0cffb0 2021-04-09T18:12:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28a572700 1 -- 172.21.15.86:0/613584428 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27c00c0c0 con 0x7fc28c0cffb0 2021-04-09T18:12:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc28a572700 1 -- 172.21.15.86:0/613584428 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc27c00c510 con 0x7fc28c0cffb0 2021-04-09T18:12:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc29190c700 1 -- 172.21.15.86:0/613584428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0cffb0 msgr2=0x7fc28c0d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.532 7fc29190c700 1 --2- 172.21.15.86:0/613584428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0cffb0 0x7fc28c0d2450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fc27c006650 tx=0x7fc27c009a80).stop 2021-04-09T18:12:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.533 7fc29190c700 1 -- 172.21.15.86:0/613584428 shutdown_connections 2021-04-09T18:12:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.533 7fc29190c700 1 --2- 172.21.15.86:0/613584428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0cffb0 0x7fc28c0d2450 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.533 7fc29190c700 1 -- 172.21.15.86:0/613584428 >> 172.21.15.86:0/613584428 conn(0x7fc28c0ce5d0 msgr2=0x7fc28c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.533 7fc29190c700 1 -- 172.21.15.86:0/613584428 shutdown_connections 2021-04-09T18:12:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.533 7fc29190c700 1 -- 172.21.15.86:0/613584428 wait complete. 2021-04-09T18:12:19.577 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.551 7fc29190c700 1 Processor -- start 2021-04-09T18:12:19.577 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc29190c700 1 -- start start 2021-04-09T18:12:19.590 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc29190c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc28c1814c0 0x7fc28c092d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.590 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc29190c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc28c093390 0x7fc28c0abbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.591 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc29190c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0ac200 0x7fc28c1a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.591 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc29190c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c093790 con 0x7fc28c0ac200 2021-04-09T18:12:19.592 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc29190c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c1818c0 con 0x7fc28c093390 2021-04-09T18:12:19.602 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc29190c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0ac600 con 0x7fc28c1814c0 2021-04-09T18:12:19.603 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc28ad73700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc28c093390 0x7fc28c0abbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.603 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc28ad73700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc28c093390 0x7fc28c0abbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50844/0 (socket says 172.21.15.86:50844) 2021-04-09T18:12:19.604 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc28b574700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc28c1814c0 0x7fc28c092d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.620 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc28ad73700 1 -- 172.21.15.86:0/1839839331 learned_addr learned my addr 172.21.15.86:0/1839839331 (peer_addr_for_me v2:172.21.15.86:0/0) 2021-04-09T18:12:19.622 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.552 7fc28bd75700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0ac200 0x7fc28c1a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.623 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc28b574700 1 -- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc28c093390 msgr2=0x7fc28c0abbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.627 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc28b574700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc28c093390 0x7fc28c0abbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.628 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc28b574700 1 -- 172.21.15.86:0/1839839331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0ac200 msgr2=0x7fc28c1a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.629 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc28b574700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0ac200 0x7fc28c1a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.629 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc28b574700 1 -- 172.21.15.86:0/1839839331 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0ac820 con 0x7fc28c1814c0 2021-04-09T18:12:19.629 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc28ad73700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc28c093390 0x7fc28c0abbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:19.630 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc28bd75700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0ac200 0x7fc28c1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:19.630 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc28b574700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc28c1814c0 0x7fc28c092d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc27c0066b0 tx=0x7fc27c00b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:19.631 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc283fff700 1 -- 172.21.15.86:0/1839839331 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc27c000c90 con 0x7fc28c1814c0 2021-04-09T18:12:19.648 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc283fff700 1 -- 172.21.15.86:0/1839839331 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27c002030 con 0x7fc28c1814c0 2021-04-09T18:12:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc283fff700 1 -- 172.21.15.86:0/1839839331 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc27c002510 con 0x7fc28c1814c0 2021-04-09T18:12:19.655 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc29190c700 1 -- 172.21.15.86:0/1839839331 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c0ac820 con 0x7fc28c1814c0 2021-04-09T18:12:19.665 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.553 7fc29190c700 1 -- 172.21.15.86:0/1839839331 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c0ac820 con 0x7fc28c1814c0 2021-04-09T18:12:19.665 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.554 7fc283fff700 1 -- 172.21.15.86:0/1839839331 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc27c000940 con 0x7fc28c1814c0 2021-04-09T18:12:19.670 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.554 7fc283fff700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc270018930 0x7fc27001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:19.670 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.554 7fc283fff700 1 -- 172.21.15.86:0/1839839331 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1475+0+0 (secure 0 0 0) 0x7fc27c031980 con 0x7fc28c1814c0 2021-04-09T18:12:19.671 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.554 7fc28ad73700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc270018930 0x7fc27001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:19.671 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.554 7fc28ad73700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc270018930 0x7fc27001ae10 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:19.693 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.577 7fc29190c700 1 -- 172.21.15.86:0/1839839331 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c1b7110 con 0x7fc28c1814c0 2021-04-09T18:12:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.580 7fc283fff700 1 -- 172.21.15.86:0/1839839331 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc27c000c90 con 0x7fc28c1814c0 2021-04-09T18:12:19.788 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.769 7fc29190c700 1 -- 172.21.15.86:0/1839839331 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", "id": 7} v 0) v1 -- 0x7fc28c0d7290 con 0x7fc28c1814c0 2021-04-09T18:12:19.792 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.774 7fc283fff700 1 -- 172.21.15.86:0/1839839331 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc27c000940 con 0x7fc28c1814c0 2021-04-09T18:12:19.798 INFO:teuthology.orchestra.run.smithi086.stdout:7 2021-04-09T18:12:19.800 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 -- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc270018930 msgr2=0x7fc27001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.801 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc270018930 0x7fc27001ae10 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.801 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 -- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc28c1814c0 msgr2=0x7fc28c092d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:19.801 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc28c1814c0 0x7fc28c092d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc27c0066b0 tx=0x7fc27c00b8e0).stop 2021-04-09T18:12:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 -- 172.21.15.86:0/1839839331 shutdown_connections 2021-04-09T18:12:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc270018930 0x7fc27001ae10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.803 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc28c0ac200 0x7fc28c1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.803 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc28c093390 0x7fc28c0abbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.804 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 --2- 172.21.15.86:0/1839839331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc28c1814c0 0x7fc28c092d70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:19.804 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 -- 172.21.15.86:0/1839839331 >> 172.21.15.86:0/1839839331 conn(0x7fc28c198770 msgr2=0x7fc28c1ac0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:19.805 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 -- 172.21.15.86:0/1839839331 shutdown_connections 2021-04-09T18:12:19.805 INFO:teuthology.orchestra.run.smithi086.stderr:2021-04-09 18:12:19.779 7fc29190c700 1 -- 172.21.15.86:0/1839839331 wait complete. 2021-04-09T18:12:19.809 INFO:tasks.ceph.osd.0:Restarting daemon 2021-04-09T18:12:19.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-04-09T18:12:19.825 INFO:tasks.ceph.osd.0:Started 2021-04-09T18:12:19.826 INFO:tasks.ceph.osd.1:Restarting daemon 2021-04-09T18:12:19.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-04-09T18:12:19.856 INFO:tasks.ceph.osd.1:Started 2021-04-09T18:12:19.862 INFO:tasks.ceph.osd.2:Restarting daemon 2021-04-09T18:12:19.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-04-09T18:12:19.893 INFO:tasks.ceph.osd.2:Started 2021-04-09T18:12:19.899 INFO:tasks.ceph.osd.3:Restarting daemon 2021-04-09T18:12:19.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-04-09T18:12:19.906 INFO:tasks.ceph.osd.3:Started 2021-04-09T18:12:19.907 INFO:tasks.ceph.osd.4:Restarting daemon 2021-04-09T18:12:19.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2021-04-09T18:12:19.930 INFO:tasks.ceph.osd.4:Started 2021-04-09T18:12:19.931 INFO:tasks.ceph.osd.5:Restarting daemon 2021-04-09T18:12:19.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2021-04-09T18:12:19.940 INFO:tasks.ceph.osd.5:Started 2021-04-09T18:12:19.941 INFO:tasks.ceph.osd.6:Restarting daemon 2021-04-09T18:12:19.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2021-04-09T18:12:19.945 INFO:tasks.ceph.osd.6:Started 2021-04-09T18:12:19.946 INFO:tasks.ceph.osd.7:Restarting daemon 2021-04-09T18:12:19.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2021-04-09T18:12:19.962 INFO:tasks.ceph.osd.7:Started 2021-04-09T18:12:19.966 INFO:tasks.ceph:Waiting for OSDs to come up 2021-04-09T18:12:20.482 INFO:tasks.ceph.osd.0.smithi032.stderr:2021-04-09 18:12:20.460 7fab2da0ea80 -1 Falling back to public interface 2021-04-09T18:12:20.502 INFO:tasks.ceph.osd.1.smithi032.stderr:2021-04-09 18:12:20.481 7fb490daba80 -1 Falling back to public interface 2021-04-09T18:12:20.544 INFO:tasks.ceph.osd.2.smithi032.stderr:2021-04-09 18:12:20.525 7f65b5e6aa80 -1 Falling back to public interface 2021-04-09T18:12:20.559 INFO:tasks.ceph.osd.3.smithi032.stderr:2021-04-09 18:12:20.540 7fae56f8ba80 -1 Falling back to public interface 2021-04-09T18:12:20.562 INFO:tasks.ceph.osd.4.smithi086.stderr:2021-04-09 18:12:20.543 7fa397321a80 -1 Falling back to public interface 2021-04-09T18:12:20.583 INFO:tasks.ceph.osd.5.smithi086.stderr:2021-04-09 18:12:20.565 7f89ad007a80 -1 Falling back to public interface 2021-04-09T18:12:20.589 INFO:tasks.ceph.osd.6.smithi086.stderr:2021-04-09 18:12:20.571 7fbfdcb23a80 -1 Falling back to public interface 2021-04-09T18:12:20.607 INFO:tasks.ceph.osd.7.smithi086.stderr:2021-04-09 18:12:20.589 7fb38561da80 -1 Falling back to public interface 2021-04-09T18:12:20.776 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-04-09T18:12:20.940 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.899 7f2109924700 1 Processor -- start 2021-04-09T18:12:20.941 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.899 7f2109924700 1 -- start start 2021-04-09T18:12:20.941 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2109924700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f210410da00 0x7f210410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:20.941 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2109924700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210410d480 con 0x7f210410d010 2021-04-09T18:12:20.942 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2109924700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21040d1f70 con 0x7f21040d7ca0 2021-04-09T18:12:20.942 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2109924700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21040d8d30 con 0x7f210410da00 2021-04-09T18:12:20.942 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2103574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f210410d010 0x7f21040d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50688/0 (socket says 172.21.15.32:50688) 2021-04-09T18:12:20.942 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2103574700 1 -- 172.21.15.32:0/1556314876 learned_addr learned my addr 172.21.15.32:0/1556314876 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:20.942 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2103d75700 1 --2- 172.21.15.32:0/1556314876 >> v2:172.21.15.86:3300/0 conn(0x7f210410da00 0x7f210410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:20.943 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2102572700 1 -- 172.21.15.32:0/1556314876 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992259567 0 0) 0x7f20f4000ce0 con 0x7f210410d010 2021-04-09T18:12:20.944 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2102572700 1 -- 172.21.15.32:0/1556314876 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ec002fe0 con 0x7f210410d010 2021-04-09T18:12:20.944 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2102572700 1 -- 172.21.15.32:0/1556314876 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215657130 0 0) 0x7f20f8001080 con 0x7f21040d7ca0 2021-04-09T18:12:20.944 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2102572700 1 -- 172.21.15.32:0/1556314876 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ec003210 con 0x7f21040d7ca0 2021-04-09T18:12:20.944 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2102572700 1 -- 172.21.15.32:0/1556314876 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2989772117 0 0) 0x7f20f40010e0 con 0x7f210410d010 2021-04-09T18:12:20.945 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2103d75700 1 -- 172.21.15.32:0/1556314876 >> v1:172.21.15.86:6789/0 conn(0x7f21040d7ca0 legacy=0x7f21040d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:20.945 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2103d75700 1 -- 172.21.15.32:0/1556314876 >> v1:172.21.15.32:6789/0 conn(0x7f210410d010 legacy=0x7f21040d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:20.946 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.900 7f2103d75700 1 -- 172.21.15.32:0/1556314876 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21040d9220 con 0x7f210410da00 2021-04-09T18:12:20.947 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.901 7f2103d75700 1 --2- 172.21.15.32:0/1556314876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210410da00 0x7f210410de00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f20fc006e50 tx=0x7f20fc009df0).ready entity=mon.1 client_cookie=c68258de152aa648 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:20.947 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.901 7f2102572700 1 -- 172.21.15.32:0/1556314876 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20fc00b510 con 0x7f210410da00 2021-04-09T18:12:20.947 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.901 7f2102572700 1 -- 172.21.15.32:0/1556314876 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc003030 con 0x7f210410da00 2021-04-09T18:12:20.947 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.901 7f2102572700 1 -- 172.21.15.32:0/1556314876 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20fc000f90 con 0x7f210410da00 2021-04-09T18:12:20.948 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.901 7f2109924700 1 -- 172.21.15.32:0/1556314876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210410da00 msgr2=0x7f210410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:20.948 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.901 7f2109924700 1 --2- 172.21.15.32:0/1556314876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210410da00 0x7f210410de00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f20fc006e50 tx=0x7f20fc009df0).stop 2021-04-09T18:12:20.948 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.902 7f2109924700 1 -- 172.21.15.32:0/1556314876 shutdown_connections 2021-04-09T18:12:20.948 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.902 7f2109924700 1 --2- 172.21.15.32:0/1556314876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210410da00 0x7f210410de00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:20.949 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.902 7f2109924700 1 -- 172.21.15.32:0/1556314876 >> 172.21.15.32:0/1556314876 conn(0x7f21040ce5f0 msgr2=0x7f21040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:20.949 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.902 7f2109924700 1 -- 172.21.15.32:0/1556314876 shutdown_connections 2021-04-09T18:12:20.949 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.902 7f2109924700 1 -- 172.21.15.32:0/1556314876 wait complete. 2021-04-09T18:12:20.949 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.922 7f2109924700 1 Processor -- start 2021-04-09T18:12:20.950 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.922 7f2109924700 1 -- start start 2021-04-09T18:12:20.951 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2109924700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f210408f040 0x7f21040b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:20.951 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2109924700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21040b75e0 0x7f21041a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:20.967 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2109924700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21041a99e0 0x7f2104185ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:21.026 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2109924700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21041a9de0 con 0x7f210408f040 2021-04-09T18:12:21.027 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2109924700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21041864e0 con 0x7f21040b75e0 2021-04-09T18:12:21.027 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2109924700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2104186700 con 0x7f21041a99e0 2021-04-09T18:12:21.027 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2103574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f210408f040 0x7f21040b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:21.027 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2102d73700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21040b75e0 0x7f21041a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:21.028 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2103574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f210408f040 0x7f21040b6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41102/0 (socket says 172.21.15.32:41102) 2021-04-09T18:12:21.028 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2103574700 1 -- 172.21.15.32:0/3733247156 learned_addr learned my addr 172.21.15.32:0/3733247156 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:21.028 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2103d75700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21041a99e0 0x7f2104185ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:21.028 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2102d73700 1 -- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21041a99e0 msgr2=0x7f2104185ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:21.029 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2102d73700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21041a99e0 0x7f2104185ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:21.053 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2102d73700 1 -- 172.21.15.32:0/3733247156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f210408f040 msgr2=0x7f21040b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:21.053 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2102d73700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f210408f040 0x7f21040b6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:21.053 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2102d73700 1 -- 172.21.15.32:0/3733247156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2104186920 con 0x7f21040b75e0 2021-04-09T18:12:21.054 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.923 7f2103574700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f210408f040 0x7f21040b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:21.054 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.924 7f2102d73700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21040b75e0 0x7f21041a93c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f20f800b510 tx=0x7f20f800b8b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:21.054 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.924 7f20ebfff700 1 -- 172.21.15.32:0/3733247156 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20f8006240 con 0x7f21040b75e0 2021-04-09T18:12:21.055 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.924 7f20ebfff700 1 -- 172.21.15.32:0/3733247156 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f80054f0 con 0x7f21040b75e0 2021-04-09T18:12:21.055 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.924 7f2109924700 1 -- 172.21.15.32:0/3733247156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2104186920 con 0x7f21040b75e0 2021-04-09T18:12:21.055 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.924 7f20ebfff700 1 -- 172.21.15.32:0/3733247156 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20f8005a10 con 0x7f21040b75e0 2021-04-09T18:12:21.055 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.924 7f2109924700 1 -- 172.21.15.32:0/3733247156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2104186920 con 0x7f21040b75e0 2021-04-09T18:12:21.056 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.925 7f20ebfff700 1 -- 172.21.15.32:0/3733247156 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f20f8000c60 con 0x7f21040b75e0 2021-04-09T18:12:21.056 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.925 7f20ebfff700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20e40188d0 0x7f20e401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:21.056 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.925 7f20ebfff700 1 -- 172.21.15.32:0/3733247156 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1578+0+0 (secure 0 0 0) 0x7f20f80319c0 con 0x7f21040b75e0 2021-04-09T18:12:21.056 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.925 7f2103574700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20e40188d0 0x7f20e401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:21.056 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.925 7f2103574700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20e40188d0 0x7f20e401add0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:21.057 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.944 7f2109924700 1 -- 172.21.15.32:0/3733247156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210408f4d0 con 0x7f21040b75e0 2021-04-09T18:12:21.067 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:20.950 7f20ebfff700 1 -- 172.21.15.32:0/3733247156 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f20f8005730 con 0x7f21040b75e0 2021-04-09T18:12:21.277 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.259 7f2109924700 1 -- 172.21.15.32:0/3733247156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f210408f4d0 con 0x7f21040b75e0 2021-04-09T18:12:21.279 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.259 7f20ebfff700 1 -- 172.21.15.32:0/3733247156 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+5464 (secure 0 0 0) 0x7f20f8000c60 con 0x7f21040b75e0 2021-04-09T18:12:21.279 INFO:teuthology.misc.health.smithi032.stdout: 2021-04-09T18:12:21.280 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:19.771129","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:12:21.284 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 -- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20e40188d0 msgr2=0x7f20e401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:21.284 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20e40188d0 0x7f20e401add0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:21.286 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 -- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21040b75e0 msgr2=0x7f21041a93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:21.286 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21040b75e0 0x7f21041a93c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f20f800b510 tx=0x7f20f800b8b0).stop 2021-04-09T18:12:21.286 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 -- 172.21.15.32:0/3733247156 shutdown_connections 2021-04-09T18:12:21.287 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20e40188d0 0x7f20e401add0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:21.287 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21041a99e0 0x7f2104185ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:21.288 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21040b75e0 0x7f21041a93c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:21.288 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 --2- 172.21.15.32:0/3733247156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f210408f040 0x7f21040b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:21.288 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 -- 172.21.15.32:0/3733247156 >> 172.21.15.32:0/3733247156 conn(0x7f210408fb70 msgr2=0x7f210419ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:21.289 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 -- 172.21.15.32:0/3733247156 shutdown_connections 2021-04-09T18:12:21.289 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:21.262 7f2109924700 1 -- 172.21.15.32:0/3733247156 wait complete. 2021-04-09T18:12:21.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2021-04-09T18:12:21.766 INFO:tasks.ceph.osd.0.smithi032.stderr:2021-04-09 18:12:21.747 7fab2da0ea80 -1 osd.0 0 log_to_monitors {default=true} 2021-04-09T18:12:21.792 INFO:tasks.ceph.osd.1.smithi032.stderr:2021-04-09 18:12:21.773 7fb490daba80 -1 osd.1 0 log_to_monitors {default=true} 2021-04-09T18:12:21.837 INFO:tasks.ceph.osd.2.smithi032.stderr:2021-04-09 18:12:21.818 7f65b5e6aa80 -1 osd.2 0 log_to_monitors {default=true} 2021-04-09T18:12:21.852 INFO:tasks.ceph.osd.4.smithi086.stderr:2021-04-09 18:12:21.834 7fa397321a80 -1 osd.4 0 log_to_monitors {default=true} 2021-04-09T18:12:21.853 INFO:tasks.ceph.osd.3.smithi032.stderr:2021-04-09 18:12:21.834 7fae56f8ba80 -1 osd.3 0 log_to_monitors {default=true} 2021-04-09T18:12:21.879 INFO:tasks.ceph.osd.5.smithi086.stderr:2021-04-09 18:12:21.861 7f89ad007a80 -1 osd.5 0 log_to_monitors {default=true} 2021-04-09T18:12:21.887 INFO:tasks.ceph.osd.6.smithi086.stderr:2021-04-09 18:12:21.869 7fbfdcb23a80 -1 osd.6 0 log_to_monitors {default=true} 2021-04-09T18:12:21.903 INFO:tasks.ceph.osd.7.smithi086.stderr:2021-04-09 18:12:21.885 7fb38561da80 -1 osd.7 0 log_to_monitors {default=true} 2021-04-09T18:12:23.495 INFO:tasks.ceph.osd.1.smithi032.stderr:2021-04-09 18:12:23.475 7fb477424700 -1 osd.1 0 waiting for initial osdmap 2021-04-09T18:12:23.496 INFO:tasks.ceph.osd.3.smithi032.stderr:2021-04-09 18:12:23.475 7fae3d604700 -1 osd.3 0 waiting for initial osdmap 2021-04-09T18:12:23.497 INFO:tasks.ceph.osd.0.smithi032.stderr:2021-04-09 18:12:23.475 7fab14087700 -1 osd.0 0 waiting for initial osdmap 2021-04-09T18:12:23.504 INFO:tasks.ceph.osd.4.smithi086.stderr:2021-04-09 18:12:23.485 7fa37d99a700 -1 osd.4 0 waiting for initial osdmap 2021-04-09T18:12:23.505 INFO:tasks.ceph.osd.5.smithi086.stderr:2021-04-09 18:12:23.485 7f8993680700 -1 osd.5 0 waiting for initial osdmap 2021-04-09T18:12:23.508 INFO:tasks.ceph.osd.7.smithi086.stderr:2021-04-09 18:12:23.485 7fb36bc96700 -1 osd.7 0 waiting for initial osdmap 2021-04-09T18:12:23.508 INFO:tasks.ceph.osd.6.smithi086.stderr:2021-04-09 18:12:23.485 7fbfc319c700 -1 osd.6 0 waiting for initial osdmap 2021-04-09T18:12:23.510 INFO:tasks.ceph.osd.2.smithi032.stderr:2021-04-09 18:12:23.484 7f659c4e3700 -1 osd.2 0 waiting for initial osdmap 2021-04-09T18:12:26.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:12:26.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:12:28.101 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-04-09T18:12:28.269 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.228 7f57983fd700 1 Processor -- start 2021-04-09T18:12:28.269 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.228 7f57983fd700 1 -- start start 2021-04-09T18:12:28.270 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.228 7f57983fd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57900d7ca0 0x7f57900d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.270 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.228 7f57983fd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579010d480 con 0x7f579010d010 2021-04-09T18:12:28.270 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.228 7f57983fd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57900d86c0 con 0x7f579010da00 2021-04-09T18:12:28.270 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.228 7f57983fd700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57900d9220 con 0x7f57900d7ca0 2021-04-09T18:12:28.271 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5795a18700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57900d7ca0 0x7f57900d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.271 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5796219700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f579010d010 0x7f57900d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50916/0 (socket says 172.21.15.32:50916) 2021-04-09T18:12:28.271 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5796219700 1 -- 172.21.15.32:0/2159832207 learned_addr learned my addr 172.21.15.32:0/2159832207 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:28.271 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5795a18700 1 -- 172.21.15.32:0/2159832207 >> v1:172.21.15.86:6789/0 conn(0x7f579010da00 legacy=0x7f579010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.272 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5795a18700 1 -- 172.21.15.32:0/2159832207 >> v1:172.21.15.32:6789/0 conn(0x7f579010d010 legacy=0x7f57900d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.272 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5795217700 1 -- 172.21.15.32:0/2159832207 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624271280 0 0) 0x7f578c001060 con 0x7f579010d010 2021-04-09T18:12:28.272 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5795a18700 1 -- 172.21.15.32:0/2159832207 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57900d1f70 con 0x7f57900d7ca0 2021-04-09T18:12:28.272 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5795a18700 1 --2- 172.21.15.32:0/2159832207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900d7ca0 0x7f57900d80a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5780006650 tx=0x7f5780009a00).ready entity=mon.0 client_cookie=2125c514e8c51ab9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:28.273 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5795217700 1 -- 172.21.15.32:0/2159832207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5780009df0 con 0x7f57900d7ca0 2021-04-09T18:12:28.273 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5795217700 1 -- 172.21.15.32:0/2159832207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57800009b0 con 0x7f57900d7ca0 2021-04-09T18:12:28.273 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f5795217700 1 -- 172.21.15.32:0/2159832207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f578000c2b0 con 0x7f57900d7ca0 2021-04-09T18:12:28.273 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f57983fd700 1 -- 172.21.15.32:0/2159832207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900d7ca0 msgr2=0x7f57900d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.274 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.229 7f57983fd700 1 --2- 172.21.15.32:0/2159832207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900d7ca0 0x7f57900d80a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5780006650 tx=0x7f5780009a00).stop 2021-04-09T18:12:28.274 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.230 7f57983fd700 1 -- 172.21.15.32:0/2159832207 shutdown_connections 2021-04-09T18:12:28.274 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.230 7f57983fd700 1 --2- 172.21.15.32:0/2159832207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900d7ca0 0x7f57900d80a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.275 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.230 7f57983fd700 1 -- 172.21.15.32:0/2159832207 >> 172.21.15.32:0/2159832207 conn(0x7f57900ce5f0 msgr2=0x7f57900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:28.275 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.230 7f57983fd700 1 -- 172.21.15.32:0/2159832207 shutdown_connections 2021-04-09T18:12:28.275 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.230 7f57983fd700 1 -- 172.21.15.32:0/2159832207 wait complete. 2021-04-09T18:12:28.281 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f57983fd700 1 Processor -- start 2021-04-09T18:12:28.281 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f57983fd700 1 -- start start 2021-04-09T18:12:28.281 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f57983fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900bd890 0x7f57900bdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.282 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f57983fd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57900beb40 0x7f57900bef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.282 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f57983fd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57901a59d0 0x7f57901a5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.282 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f57983fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57900c38e0 con 0x7f57900bd890 2021-04-09T18:12:28.283 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f57983fd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57900be5c0 con 0x7f57901a59d0 2021-04-09T18:12:28.283 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f57983fd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57900bcae0 con 0x7f57900beb40 2021-04-09T18:12:28.290 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f5796219700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900bd890 0x7f57900bdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.291 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f5796219700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900bd890 0x7f57900bdc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41332/0 (socket says 172.21.15.32:41332) 2021-04-09T18:12:28.291 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f5796219700 1 -- 172.21.15.32:0/265885901 learned_addr learned my addr 172.21.15.32:0/265885901 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:28.291 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f5795a18700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57900beb40 0x7f57900bef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.292 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.251 7f5796a1a700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57901a59d0 0x7f57901a5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.295 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f5796219700 1 -- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57900beb40 msgr2=0x7f57900bef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.295 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f5796219700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57900beb40 0x7f57900bef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.296 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f5796219700 1 -- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57901a59d0 msgr2=0x7f57901a5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.296 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f5796219700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57901a59d0 0x7f57901a5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.296 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f5796219700 1 -- 172.21.15.32:0/265885901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57900be2b0 con 0x7f57900bd890 2021-04-09T18:12:28.296 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f5796219700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900bd890 0x7f57900bdc90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f578c009fc0 tx=0x7f578c00bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:28.296 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f5786ffd700 1 -- 172.21.15.32:0/265885901 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f578c006080 con 0x7f57900bd890 2021-04-09T18:12:28.297 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f5786ffd700 1 -- 172.21.15.32:0/265885901 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c006400 con 0x7f57900bd890 2021-04-09T18:12:28.297 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f5786ffd700 1 -- 172.21.15.32:0/265885901 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f578c0049a0 con 0x7f57900bd890 2021-04-09T18:12:28.297 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f57983fd700 1 -- 172.21.15.32:0/265885901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57900be2b0 con 0x7f57900bd890 2021-04-09T18:12:28.297 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.252 7f57983fd700 1 -- 172.21.15.32:0/265885901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57900be2b0 con 0x7f57900bd890 2021-04-09T18:12:28.298 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.253 7f5786ffd700 1 -- 172.21.15.32:0/265885901 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f578c004720 con 0x7f57900bd890 2021-04-09T18:12:28.298 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.253 7f5786ffd700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57780188c0 0x7f577801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.298 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.253 7f5786ffd700 1 -- 172.21.15.32:0/265885901 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4404+0+0 (secure 0 0 0) 0x7f578c0326f0 con 0x7f57900bd890 2021-04-09T18:12:28.298 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.253 7f5795a18700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57780188c0 0x7f577801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.298 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.253 7f5795a18700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57780188c0 0x7f577801adc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:28.321 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.272 7f57983fd700 1 -- 172.21.15.32:0/265885901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57901a47d0 con 0x7f57900bd890 2021-04-09T18:12:28.333 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.276 7f5786ffd700 1 -- 172.21.15.32:0/265885901 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f578c004930 con 0x7f57900bd890 2021-04-09T18:12:28.482 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.463 7f57983fd700 1 -- 172.21.15.32:0/265885901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5790112070 con 0x7f57900bd890 2021-04-09T18:12:28.483 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.464 7f5786ffd700 1 -- 172.21.15.32:0/265885901 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+9772 (secure 0 0 0) 0x7f578c004720 con 0x7f57900bd890 2021-04-09T18:12:28.484 INFO:teuthology.misc.health.smithi032.stdout: 2021-04-09T18:12:28.484 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":13,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:24.480980","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:12:28.487 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 -- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57780188c0 msgr2=0x7f577801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.487 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57780188c0 0x7f577801adc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.487 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 -- 172.21.15.32:0/265885901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900bd890 msgr2=0x7f57900bdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.488 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900bd890 0x7f57900bdc90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f578c009fc0 tx=0x7f578c00bc40).stop 2021-04-09T18:12:28.488 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 -- 172.21.15.32:0/265885901 shutdown_connections 2021-04-09T18:12:28.488 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57780188c0 0x7f577801adc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.488 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57901a59d0 0x7f57901a5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.489 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57900beb40 0x7f57900bef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.489 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 --2- 172.21.15.32:0/265885901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57900bd890 0x7f57900bdc90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.489 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 -- 172.21.15.32:0/265885901 >> 172.21.15.32:0/265885901 conn(0x7f57900bcd80 msgr2=0x7f579008dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:28.489 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 -- 172.21.15.32:0/265885901 shutdown_connections 2021-04-09T18:12:28.490 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:28.466 7f57983fd700 1 -- 172.21.15.32:0/265885901 wait complete. 2021-04-09T18:12:28.515 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-04-09T18:12:28.515 INFO:tasks.ceph:Creating RBD pool 2021-04-09T18:12:28.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd pool create rbd 8 2021-04-09T18:12:28.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.654 7f9e21771700 1 Processor -- start 2021-04-09T18:12:28.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.654 7f9e21771700 1 -- start start 2021-04-09T18:12:28.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.654 7f9e21771700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9e1c10d010 0x7f9e1c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e21771700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c10d480 con 0x7f9e1c0d1da0 2021-04-09T18:12:28.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e21771700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c10e1f0 con 0x7f9e1c0d8490 2021-04-09T18:12:28.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e21771700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1c0d8d30 con 0x7f9e1c10d010 2021-04-09T18:12:28.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1ad73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9e1c0d1da0 0x7f9e1c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50932/0 (socket says 172.21.15.32:50932) 2021-04-09T18:12:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1ad73700 1 -- 172.21.15.32:0/20988210 learned_addr learned my addr 172.21.15.32:0/20988210 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1b574700 1 --2- 172.21.15.32:0/20988210 >> v2:172.21.15.86:3300/0 conn(0x7f9e1c10d010 0x7f9e1c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798035546 0 0) 0x7f9e04000ce0 con 0x7f9e1c0d1da0 2021-04-09T18:12:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e08002fe0 con 0x7f9e1c0d1da0 2021-04-09T18:12:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3327798782 0 0) 0x7f9e04000ed0 con 0x7f9e1c0d1da0 2021-04-09T18:12:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 >> v2:172.21.15.86:3300/0 conn(0x7f9e1c10d010 msgr2=0x7f9e1c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 --2- 172.21.15.32:0/20988210 >> v2:172.21.15.86:3300/0 conn(0x7f9e1c10d010 0x7f9e1c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 >> v1:172.21.15.86:6789/0 conn(0x7f9e1c0d8490 legacy=0x7f9e1c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c0d9220 con 0x7f9e1c0d1da0 2021-04-09T18:12:28.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f9e04001600 con 0x7f9e1c0d1da0 2021-04-09T18:12:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 >> v1:172.21.15.32:6789/0 conn(0x7f9e1c0d1da0 legacy=0x7f9e1c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08003a40 0x7f9e08005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e08006790 0x7f9e08008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e08009590 0x7f9e0800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0800c300 con 0x7f9e08003a40 2021-04-09T18:12:28.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0800c520 con 0x7f9e08006790 2021-04-09T18:12:28.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1b574700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08003a40 0x7f9e08005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1a572700 1 -- 172.21.15.32:0/20988210 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08050750 con 0x7f9e08009590 2021-04-09T18:12:28.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1bd75700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e08009590 0x7f9e0800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.655 7f9e1ad73700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e08006790 0x7f9e08008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1b574700 1 -- 172.21.15.32:0/20988210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e08009590 msgr2=0x7f9e0800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:28.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1b574700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e08009590 0x7f9e0800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1b574700 1 -- 172.21.15.32:0/20988210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e08006790 msgr2=0x7f9e08008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:28.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1b574700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e08006790 0x7f9e08008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1b574700 1 -- 172.21.15.32:0/20988210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e080509b0 con 0x7f9e08003a40 2021-04-09T18:12:28.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1bd75700 1 -- 172.21.15.32:0/20988210 reap_dead start 2021-04-09T18:12:28.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1b574700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08003a40 0x7f9e08005ef0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9e14006fc0 tx=0x7f9e14009060).ready entity=mon.0 client_cookie=2b21ae1933d3cdc4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:28.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1a572700 1 -- 172.21.15.32:0/20988210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e1400c420 con 0x7f9e08003a40 2021-04-09T18:12:28.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1a572700 1 -- 172.21.15.32:0/20988210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e14009a70 con 0x7f9e08003a40 2021-04-09T18:12:28.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e1a572700 1 -- 172.21.15.32:0/20988210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e1400c720 con 0x7f9e08003a40 2021-04-09T18:12:28.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e21771700 1 -- 172.21.15.32:0/20988210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08003a40 msgr2=0x7f9e08005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.656 7f9e21771700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08003a40 0x7f9e08005ef0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9e14006fc0 tx=0x7f9e14009060).stop 2021-04-09T18:12:28.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.657 7f9e21771700 1 -- 172.21.15.32:0/20988210 shutdown_connections 2021-04-09T18:12:28.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.657 7f9e21771700 1 --2- 172.21.15.32:0/20988210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08003a40 0x7f9e08005ef0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.657 7f9e21771700 1 -- 172.21.15.32:0/20988210 >> 172.21.15.32:0/20988210 conn(0x7f9e1c0ce5f0 msgr2=0x7f9e1c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:28.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.657 7f9e21771700 1 -- 172.21.15.32:0/20988210 shutdown_connections 2021-04-09T18:12:28.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.657 7f9e21771700 1 -- 172.21.15.32:0/20988210 wait complete. 2021-04-09T18:12:28.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.677 7f9e21771700 1 Processor -- start 2021-04-09T18:12:28.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.677 7f9e21771700 1 -- start start 2021-04-09T18:12:28.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.677 7f9e21771700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e1c0a0110 0x7f9e1c18c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.677 7f9e21771700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e1c0a0510 0x7f9e1c0c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.677 7f9e21771700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e1c0c36e0 0x7f9e1c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.677 7f9e21771700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c18cf60 con 0x7f9e1c0a0110 2021-04-09T18:12:28.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.677 7f9e21771700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c199ea0 con 0x7f9e1c0a0510 2021-04-09T18:12:28.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.677 7f9e21771700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c187d70 con 0x7f9e1c0c36e0 2021-04-09T18:12:28.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e1c0a0110 0x7f9e1c18c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e1c0a0110 0x7f9e1c18c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41352/0 (socket says 172.21.15.32:41352) 2021-04-09T18:12:28.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1b574700 1 -- 172.21.15.32:0/3534146248 learned_addr learned my addr 172.21.15.32:0/3534146248 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:28.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1bd75700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e1c0c36e0 0x7f9e1c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1ad73700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e1c0a0510 0x7f9e1c0c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1b574700 1 -- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e1c0c36e0 msgr2=0x7f9e1c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1b574700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e1c0c36e0 0x7f9e1c0c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1b574700 1 -- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e1c0a0510 msgr2=0x7f9e1c0c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:28.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1b574700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e1c0a0510 0x7f9e1c0c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:28.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1b574700 1 -- 172.21.15.32:0/3534146248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c18d4e0 con 0x7f9e1c0a0110 2021-04-09T18:12:28.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1ad73700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e1c0a0510 0x7f9e1c0c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e1b574700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e1c0a0110 0x7f9e1c18c750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f9e14009a10 tx=0x7f9e1400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:28.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e13fff700 1 -- 172.21.15.32:0/3534146248 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e140094e0 con 0x7f9e1c0a0110 2021-04-09T18:12:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e13fff700 1 -- 172.21.15.32:0/3534146248 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1400c2f0 con 0x7f9e1c0a0110 2021-04-09T18:12:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e21771700 1 -- 172.21.15.32:0/3534146248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c18d4e0 con 0x7f9e1c0a0110 2021-04-09T18:12:28.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e13fff700 1 -- 172.21.15.32:0/3534146248 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e1400c7c0 con 0x7f9e1c0a0110 2021-04-09T18:12:28.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.678 7f9e21771700 1 -- 172.21.15.32:0/3534146248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c18d4e0 con 0x7f9e1c0a0110 2021-04-09T18:12:28.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.679 7f9e13fff700 1 -- 172.21.15.32:0/3534146248 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9e14001010 con 0x7f9e1c0a0110 2021-04-09T18:12:28.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.679 7f9e13fff700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e000188c0 0x7f9e0001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:28.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.679 7f9e13fff700 1 -- 172.21.15.32:0/3534146248 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4404+0+0 (secure 0 0 0) 0x7f9e140326e0 con 0x7f9e1c0a0110 2021-04-09T18:12:28.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.680 7f9e1ad73700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e000188c0 0x7f9e0001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:28.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.680 7f9e1ad73700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e000188c0 0x7f9e0001adc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:28.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.699 7f9e21771700 1 -- 172.21.15.32:0/3534146248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c10d480 con 0x7f9e1c0a0110 2021-04-09T18:12:28.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.702 7f9e13fff700 1 -- 172.21.15.32:0/3534146248 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9e140091f0 con 0x7f9e1c0a0110 2021-04-09T18:12:28.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:28.889 7f9e21771700 1 -- 172.21.15.32:0/3534146248 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"} v 0) v1 -- 0x7f9e1c10d480 con 0x7f9e1c0a0110 2021-04-09T18:12:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.307 7f9e13fff700 1 -- 172.21.15.32:0/3534146248 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7f9e14001010 con 0x7f9e1c0a0110 2021-04-09T18:12:29.373 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'rbd' created 2021-04-09T18:12:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 -- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e000188c0 msgr2=0x7f9e0001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e000188c0 0x7f9e0001adc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 -- 172.21.15.32:0/3534146248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e1c0a0110 msgr2=0x7f9e1c18c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e1c0a0110 0x7f9e1c18c750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f9e14009a10 tx=0x7f9e1400bd10).stop 2021-04-09T18:12:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 -- 172.21.15.32:0/3534146248 shutdown_connections 2021-04-09T18:12:29.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e000188c0 0x7f9e0001adc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:29.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e1c0c36e0 0x7f9e1c0c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:29.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e1c0a0510 0x7f9e1c0c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:29.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 --2- 172.21.15.32:0/3534146248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e1c0a0110 0x7f9e1c18c750 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:29.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 -- 172.21.15.32:0/3534146248 >> 172.21.15.32:0/3534146248 conn(0x7f9e1c0b93a0 msgr2=0x7f9e1c0b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:29.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 -- 172.21.15.32:0/3534146248 shutdown_connections 2021-04-09T18:12:29.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.356 7f9e21771700 1 -- 172.21.15.32:0/3534146248 wait complete. 2021-04-09T18:12:29.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2021-04-09T18:12:29.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.532 7f3a2a66e700 1 Processor -- start 2021-04-09T18:12:29.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.532 7f3a2a66e700 1 -- start start 2021-04-09T18:12:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a2a66e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2410d020 0x7f3a240d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a2a66e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2410d490 con 0x7f3a240d84a0 2021-04-09T18:12:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a2a66e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2410e200 con 0x7f3a240d1db0 2021-04-09T18:12:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a2a66e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2410e790 con 0x7f3a2410d020 2021-04-09T18:12:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a28c8b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3a240d84a0 0x7f3a2410da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50952/0 (socket says 172.21.15.32:50952) 2021-04-09T18:12:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a28c8b700 1 -- 172.21.15.32:0/953553867 learned_addr learned my addr 172.21.15.32:0/953553867 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:29.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a23fff700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2410d020 0x7f3a240d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:29.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a237fe700 1 -- 172.21.15.32:0/953553867 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495485893 0 0) 0x7f3a1c000d30 con 0x7f3a240d84a0 2021-04-09T18:12:29.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a237fe700 1 -- 172.21.15.32:0/953553867 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a10002fe0 con 0x7f3a240d84a0 2021-04-09T18:12:29.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a237fe700 1 -- 172.21.15.32:0/953553867 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525107385 0 0) 0x7f3a0c000c60 con 0x7f3a240d1db0 2021-04-09T18:12:29.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a237fe700 1 -- 172.21.15.32:0/953553867 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a10003210 con 0x7f3a240d1db0 2021-04-09T18:12:29.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a237fe700 1 -- 172.21.15.32:0/953553867 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2432149089 0 0) 0x7f3a1c001460 con 0x7f3a240d84a0 2021-04-09T18:12:29.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a237fe700 1 -- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2410d020 msgr2=0x7f3a240d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:29.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a237fe700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2410d020 0x7f3a240d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:29.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.533 7f3a237fe700 1 -- 172.21.15.32:0/953553867 >> v1:172.21.15.86:6789/0 conn(0x7f3a240d1db0 legacy=0x7f3a240d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:29.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 -- 172.21.15.32:0/953553867 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a240d8d40 con 0x7f3a240d84a0 2021-04-09T18:12:29.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 -- 172.21.15.32:0/953553867 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f3a1c001980 con 0x7f3a240d84a0 2021-04-09T18:12:29.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 -- 172.21.15.32:0/953553867 >> v1:172.21.15.32:6789/0 conn(0x7f3a240d84a0 legacy=0x7f3a2410da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:29.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a10003b20 0x7f3a100065b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:29.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a10004100 0x7f3a10008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:29.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a10009500 0x7f3a1000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:29.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 -- 172.21.15.32:0/953553867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100502e0 con 0x7f3a10003b20 2021-04-09T18:12:29.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a23fff700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a10003b20 0x7f3a100065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:29.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 -- 172.21.15.32:0/953553867 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100504c0 con 0x7f3a10009500 2021-04-09T18:12:29.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 -- 172.21.15.32:0/953553867 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100506e0 con 0x7f3a10004100 2021-04-09T18:12:29.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a28c8b700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a10009500 0x7f3a1000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:29.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a1bfff700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a10004100 0x7f3a10008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:29.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a23fff700 1 -- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a10004100 msgr2=0x7f3a10008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:29.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a23fff700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a10004100 0x7f3a10008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:29.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a23fff700 1 -- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a10009500 msgr2=0x7f3a1000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:29.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a23fff700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a10009500 0x7f3a1000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:29.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a23fff700 1 -- 172.21.15.32:0/953553867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a10050980 con 0x7f3a10003b20 2021-04-09T18:12:29.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a28c8b700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a10009500 0x7f3a1000ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:29.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a23fff700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a10003b20 0x7f3a100065b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f3a14006ac0 tx=0x7f3a14009ab0).ready entity=mon.0 client_cookie=13682fb66c8883d0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:29.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 -- 172.21.15.32:0/953553867 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a1400c0c0 con 0x7f3a10003b20 2021-04-09T18:12:29.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 -- 172.21.15.32:0/953553867 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1400c3c0 con 0x7f3a10003b20 2021-04-09T18:12:29.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a237fe700 1 -- 172.21.15.32:0/953553867 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a140009b0 con 0x7f3a10003b20 2021-04-09T18:12:29.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a2a66e700 1 -- 172.21.15.32:0/953553867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a10003b20 msgr2=0x7f3a100065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a2a66e700 1 --2- 172.21.15.32:0/953553867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a10003b20 0x7f3a100065b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f3a14006ac0 tx=0x7f3a14009ab0).stop 2021-04-09T18:12:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.534 7f3a28c8b700 1 -- 172.21.15.32:0/953553867 reap_dead start 2021-04-09T18:12:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.535 7f3a2a66e700 1 -- 172.21.15.32:0/953553867 shutdown_connections 2021-04-09T18:12:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.535 7f3a2a66e700 1 -- 172.21.15.32:0/953553867 >> 172.21.15.32:0/953553867 conn(0x7f3a240ce600 msgr2=0x7f3a240d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.535 7f3a2a66e700 1 -- 172.21.15.32:0/953553867 shutdown_connections 2021-04-09T18:12:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.535 7f3a2a66e700 1 -- 172.21.15.32:0/953553867 wait complete. 2021-04-09T18:12:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.555 7f3a2a66e700 1 Processor -- start 2021-04-09T18:12:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.555 7f3a2a66e700 1 -- start start 2021-04-09T18:12:29.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a2a66e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a240ba4f0 0x7f3a240ba8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:29.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a2a66e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2409fe20 0x7f3a240a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:29.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a2a66e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a240a0840 0x7f3a241a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:29.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a2a66e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240bf0d0 con 0x7f3a240ba4f0 2021-04-09T18:12:29.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a2a66e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a241a13e0 con 0x7f3a240a0840 2021-04-09T18:12:29.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a2a66e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a241a15c0 con 0x7f3a2409fe20 2021-04-09T18:12:29.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a23fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a240ba4f0 0x7f3a240ba8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:29.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a23fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a240ba4f0 0x7f3a240ba8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41370/0 (socket says 172.21.15.32:41370) 2021-04-09T18:12:29.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a23fff700 1 -- 172.21.15.32:0/3373213695 learned_addr learned my addr 172.21.15.32:0/3373213695 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:29.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a23fff700 1 -- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2409fe20 msgr2=0x7f3a240a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:29.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a23fff700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2409fe20 0x7f3a240a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:12:29.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a23fff700 1 -- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a240a0840 msgr2=0x7f3a241a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:29.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a23fff700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a240a0840 0x7f3a241a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:12:29.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a23fff700 1 -- 172.21.15.32:0/3373213695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a241a17e0 con 0x7f3a240ba4f0 2021-04-09T18:12:29.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a23fff700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a240ba4f0 0x7f3a240ba8f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f3a140066c0 tx=0x7f3a1400b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:29.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.556 7f3a217fa700 1 -- 172.21.15.32:0/3373213695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a1400bdb0 con 0x7f3a240ba4f0 2021-04-09T18:12:29.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.557 7f3a217fa700 1 -- 172.21.15.32:0/3373213695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a14000940 con 0x7f3a240ba4f0 2021-04-09T18:12:29.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.557 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a241a17e0 con 0x7f3a240ba4f0 2021-04-09T18:12:29.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.557 7f3a217fa700 1 -- 172.21.15.32:0/3373213695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a14002030 con 0x7f3a240ba4f0 2021-04-09T18:12:29.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.557 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a241a17e0 con 0x7f3a240ba4f0 2021-04-09T18:12:29.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.557 7f3a217fa700 1 -- 172.21.15.32:0/3373213695 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3a14009360 con 0x7f3a240ba4f0 2021-04-09T18:12:29.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.557 7f3a217fa700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a080188c0 0x7f3a0801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:29.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.557 7f3a217fa700 1 -- 172.21.15.32:0/3373213695 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4746+0+0 (secure 0 0 0) 0x7f3a14032410 con 0x7f3a240ba4f0 2021-04-09T18:12:29.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.558 7f3a1bfff700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a080188c0 0x7f3a0801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:29.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.558 7f3a1bfff700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a080188c0 0x7f3a0801adc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:29.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.577 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2410d490 con 0x7f3a240ba4f0 2021-04-09T18:12:29.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.580 7f3a217fa700 1 -- 172.21.15.32:0/3373213695 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3a1402cf30 con 0x7f3a240ba4f0 2021-04-09T18:12:29.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:29.756 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3a2410d490 con 0x7f3a240ba4f0 2021-04-09T18:12:30.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.308 7f3a217fa700 1 -- 172.21.15.32:0/3373213695 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3a1402cf30 con 0x7f3a240ba4f0 2021-04-09T18:12:30.340 INFO:teuthology.orchestra.run.smithi032.stderr:enabled application 'rbd' on pool 'rbd' 2021-04-09T18:12:30.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a080188c0 msgr2=0x7f3a0801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:30.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a080188c0 0x7f3a0801adc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a240ba4f0 msgr2=0x7f3a240ba8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:30.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a240ba4f0 0x7f3a240ba8f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f3a140066c0 tx=0x7f3a1400b9c0).stop 2021-04-09T18:12:30.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 shutdown_connections 2021-04-09T18:12:30.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a080188c0 0x7f3a0801adc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a240a0840 0x7f3a241a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:12:30.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2409fe20 0x7f3a240a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:12:30.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 --2- 172.21.15.32:0/3373213695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a240ba4f0 0x7f3a240ba8f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 >> 172.21.15.32:0/3373213695 conn(0x7f3a24097d10 msgr2=0x7f3a24091330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:30.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 shutdown_connections 2021-04-09T18:12:30.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.323 7f3a2a66e700 1 -- 172.21.15.32:0/3373213695 wait complete. 2021-04-09T18:12:30.376 INFO:tasks.ceph:Setting up CephFS filesystem... 2021-04-09T18:12:30.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:12:30.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.549 7f0df76b3700 1 Processor -- start 2021-04-09T18:12:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.549 7f0df76b3700 1 -- start start 2021-04-09T18:12:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df76b3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0df010d010 0x7f0df00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df76b3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00d8490 0x7f0df010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:30.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df76b3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df010d480 con 0x7f0df00d1da0 2021-04-09T18:12:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df76b3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df00d8d30 con 0x7f0df010d010 2021-04-09T18:12:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df76b3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00d9220 con 0x7f0df00d8490 2021-04-09T18:12:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df54cf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0df010d010 0x7f0df00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:30.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df54cf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0df010d010 0x7f0df00d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41378/0 (socket says 172.21.15.32:41378) 2021-04-09T18:12:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df54cf700 1 -- 172.21.15.32:0/1768323112 learned_addr learned my addr 172.21.15.32:0/1768323112 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df5cd0700 1 --2- 172.21.15.32:0/1768323112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00d8490 0x7f0df010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df54cf700 1 -- 172.21.15.32:0/1768323112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00d8490 msgr2=0x7f0df010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df54cf700 1 --2- 172.21.15.32:0/1768323112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00d8490 0x7f0df010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df54cf700 1 -- 172.21.15.32:0/1768323112 >> v1:172.21.15.86:6789/0 conn(0x7f0df00d1da0 legacy=0x7f0df00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df54cf700 1 -- 172.21.15.32:0/1768323112 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df00d09c0 con 0x7f0df010d010 2021-04-09T18:12:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0de3fff700 1 -- 172.21.15.32:0/1768323112 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365856004 0 0) 0x7f0de4000c60 con 0x7f0df00d1da0 2021-04-09T18:12:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0df54cf700 1 --2- 172.21.15.32:0/1768323112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df010d010 0x7f0df00d1780 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0de8006a80 tx=0x7f0de8009e80).ready entity=mon.0 client_cookie=abb4c7b424759359 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:30.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.550 7f0de3fff700 1 -- 172.21.15.32:0/1768323112 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de800b510 con 0x7f0df010d010 2021-04-09T18:12:30.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.551 7f0de3fff700 1 -- 172.21.15.32:0/1768323112 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de800b7d0 con 0x7f0df010d010 2021-04-09T18:12:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.551 7f0de3fff700 1 -- 172.21.15.32:0/1768323112 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de800bcb0 con 0x7f0df010d010 2021-04-09T18:12:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.551 7f0df76b3700 1 -- 172.21.15.32:0/1768323112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df010d010 msgr2=0x7f0df00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:30.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.551 7f0df76b3700 1 --2- 172.21.15.32:0/1768323112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df010d010 0x7f0df00d1780 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0de8006a80 tx=0x7f0de8009e80).stop 2021-04-09T18:12:30.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.551 7f0df76b3700 1 -- 172.21.15.32:0/1768323112 shutdown_connections 2021-04-09T18:12:30.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.551 7f0df76b3700 1 --2- 172.21.15.32:0/1768323112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00d8490 0x7f0df010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.551 7f0df76b3700 1 --2- 172.21.15.32:0/1768323112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df010d010 0x7f0df00d1780 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.551 7f0df76b3700 1 -- 172.21.15.32:0/1768323112 >> 172.21.15.32:0/1768323112 conn(0x7f0df00ce5f0 msgr2=0x7f0df00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:30.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.552 7f0df76b3700 1 -- 172.21.15.32:0/1768323112 shutdown_connections 2021-04-09T18:12:30.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.552 7f0df76b3700 1 -- 172.21.15.32:0/1768323112 wait complete. 2021-04-09T18:12:30.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.572 7f0df76b3700 1 Processor -- start 2021-04-09T18:12:30.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.572 7f0df76b3700 1 -- start start 2021-04-09T18:12:30.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.572 7f0df76b3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df01a5210 0x7f0df01a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:30.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.572 7f0df76b3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df01a6340 0x7f0df01a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:30.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.572 7f0df76b3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00abaa0 0x7f0df00abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:30.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.572 7f0df76b3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df01a6d60 con 0x7f0df01a6340 2021-04-09T18:12:30.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.572 7f0df76b3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00ac4c0 con 0x7f0df01a5210 2021-04-09T18:12:30.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.572 7f0df76b3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00ac6e0 con 0x7f0df00abaa0 2021-04-09T18:12:30.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df4cce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df01a6340 0x7f0df01a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:30.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df4cce700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df01a6340 0x7f0df01a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41386/0 (socket says 172.21.15.32:41386) 2021-04-09T18:12:30.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df4cce700 1 -- 172.21.15.32:0/4267556664 learned_addr learned my addr 172.21.15.32:0/4267556664 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:30.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df5cd0700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00abaa0 0x7f0df00abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:30.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df54cf700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df01a5210 0x7f0df01a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:30.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df4cce700 1 -- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00abaa0 msgr2=0x7f0df00abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:30.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df4cce700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00abaa0 0x7f0df00abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df4cce700 1 -- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df01a5210 msgr2=0x7f0df01a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:30.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df4cce700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df01a5210 0x7f0df01a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df4cce700 1 -- 172.21.15.32:0/4267556664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df017a720 con 0x7f0df01a6340 2021-04-09T18:12:30.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df5cd0700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00abaa0 0x7f0df00abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:30.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df4cce700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df01a6340 0x7f0df01a6740 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0de4006f20 tx=0x7f0de400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:30.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0de1ffb700 1 -- 172.21.15.32:0/4267556664 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de400bd80 con 0x7f0df01a6340 2021-04-09T18:12:30.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0de1ffb700 1 -- 172.21.15.32:0/4267556664 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de4001080 con 0x7f0df01a6340 2021-04-09T18:12:30.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df017a720 con 0x7f0df01a6340 2021-04-09T18:12:30.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0de1ffb700 1 -- 172.21.15.32:0/4267556664 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de4001550 con 0x7f0df01a6340 2021-04-09T18:12:30.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.573 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df017a720 con 0x7f0df01a6340 2021-04-09T18:12:30.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.574 7f0de1ffb700 1 -- 172.21.15.32:0/4267556664 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0de4001260 con 0x7f0df01a6340 2021-04-09T18:12:30.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.574 7f0de1ffb700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0dd0018930 0x7f0dd001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:30.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.574 7f0de1ffb700 1 -- 172.21.15.32:0/4267556664 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f0de4032710 con 0x7f0df01a6340 2021-04-09T18:12:30.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.574 7f0df54cf700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0dd0018930 0x7f0dd001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:30.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.574 7f0df54cf700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0dd0018930 0x7f0dd001ae10 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.593 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df01a6d60 con 0x7f0df01a6340 2021-04-09T18:12:30.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.597 7f0de1ffb700 1 -- 172.21.15.32:0/4267556664 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0de4006bf0 con 0x7f0df01a6340 2021-04-09T18:12:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.784 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0df01a6d60 con 0x7f0df01a6340 2021-04-09T18:12:30.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.785 7f0de1ffb700 1 -- 172.21.15.32:0/4267556664 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11155 (secure 0 0 0) 0x7f0de4001080 con 0x7f0df01a6340 2021-04-09T18:12:30.806 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:30.806 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:30.306715","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:12:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0dd0018930 msgr2=0x7f0dd001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0dd0018930 0x7f0dd001ae10 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df01a6340 msgr2=0x7f0df01a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:30.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df01a6340 0x7f0df01a6740 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0de4006f20 tx=0x7f0de400b990).stop 2021-04-09T18:12:30.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 shutdown_connections 2021-04-09T18:12:30.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0dd0018930 0x7f0dd001ae10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0df01a6340 0x7f0df01a6740 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0df00abaa0 0x7f0df00abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 --2- 172.21.15.32:0/4267556664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df01a5210 0x7f0df01a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:30.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 >> 172.21.15.32:0/4267556664 conn(0x7f0df01a99c0 msgr2=0x7f0df017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:30.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 shutdown_connections 2021-04-09T18:12:30.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:30.787 7f0df76b3700 1 -- 172.21.15.32:0/4267556664 wait complete. 2021-04-09T18:12:30.864 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:12:30.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:12:31.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f9216a0d700 1 Processor -- start 2021-04-09T18:12:31.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f9216a0d700 1 -- start start 2021-04-09T18:12:31.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f9216a0d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f921010d010 0x7f92100d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f9216a0d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92100d1da0 0x7f92100d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f9216a0d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f92100d82c0 0x7f921010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f9216a0d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92100d8d30 con 0x7f92100d82c0 2021-04-09T18:12:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f9216a0d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92100d9220 con 0x7f921010d010 2021-04-09T18:12:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f9216a0d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921010d480 con 0x7f92100d1da0 2021-04-09T18:12:31.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f921502a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f92100d82c0 0x7f921010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f921502a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f92100d82c0 0x7f921010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41394/0 (socket says 172.21.15.32:41394) 2021-04-09T18:12:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f921502a700 1 -- 172.21.15.32:0/3197528693 learned_addr learned my addr 172.21.15.32:0/3197528693 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f920ffff700 1 --2- 172.21.15.32:0/3197528693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92100d1da0 0x7f92100d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.007 7f9214829700 1 --2- 172.21.15.32:0/3197528693 >> v2:172.21.15.86:3300/0 conn(0x7f921010d010 0x7f92100d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f921502a700 1 -- 172.21.15.32:0/3197528693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92100d1da0 msgr2=0x7f92100d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:31.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f921502a700 1 --2- 172.21.15.32:0/3197528693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92100d1da0 0x7f92100d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f921502a700 1 -- 172.21.15.32:0/3197528693 >> v2:172.21.15.86:3300/0 conn(0x7f921010d010 msgr2=0x7f92100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:31.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f921502a700 1 --2- 172.21.15.32:0/3197528693 >> v2:172.21.15.86:3300/0 conn(0x7f921010d010 0x7f92100d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f921502a700 1 -- 172.21.15.32:0/3197528693 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92100d86c0 con 0x7f92100d82c0 2021-04-09T18:12:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f9214829700 1 --2- 172.21.15.32:0/3197528693 >> v2:172.21.15.86:3300/0 conn(0x7f921010d010 0x7f92100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f921502a700 1 --2- 172.21.15.32:0/3197528693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100d82c0 0x7f921010da00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9208006f00 tx=0x7f9208009f10).ready entity=mon.0 client_cookie=8f10c119edaa7dcf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:31.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f920f7fe700 1 -- 172.21.15.32:0/3197528693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920800b3b0 con 0x7f92100d82c0 2021-04-09T18:12:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f920f7fe700 1 -- 172.21.15.32:0/3197528693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920800b730 con 0x7f92100d82c0 2021-04-09T18:12:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f920f7fe700 1 -- 172.21.15.32:0/3197528693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920800bc10 con 0x7f92100d82c0 2021-04-09T18:12:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f9216a0d700 1 -- 172.21.15.32:0/3197528693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100d82c0 msgr2=0x7f921010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.008 7f9216a0d700 1 --2- 172.21.15.32:0/3197528693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100d82c0 0x7f921010da00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9208006f00 tx=0x7f9208009f10).stop 2021-04-09T18:12:31.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.009 7f9216a0d700 1 -- 172.21.15.32:0/3197528693 shutdown_connections 2021-04-09T18:12:31.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.009 7f9216a0d700 1 --2- 172.21.15.32:0/3197528693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100d82c0 0x7f921010da00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.009 7f9216a0d700 1 --2- 172.21.15.32:0/3197528693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92100d1da0 0x7f92100d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.009 7f9216a0d700 1 --2- 172.21.15.32:0/3197528693 >> v2:172.21.15.86:3300/0 conn(0x7f921010d010 0x7f92100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.009 7f9216a0d700 1 -- 172.21.15.32:0/3197528693 >> 172.21.15.32:0/3197528693 conn(0x7f92100ce5f0 msgr2=0x7f92100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:31.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.009 7f9216a0d700 1 -- 172.21.15.32:0/3197528693 shutdown_connections 2021-04-09T18:12:31.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.009 7f9216a0d700 1 -- 172.21.15.32:0/3197528693 wait complete. 2021-04-09T18:12:31.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.029 7f9216a0d700 1 Processor -- start 2021-04-09T18:12:31.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.029 7f9216a0d700 1 -- start start 2021-04-09T18:12:31.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9216a0d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100c3c00 0x7f921009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9216a0d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9210095240 0x7f92101a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9216a0d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92101a1270 0x7f92100b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9216a0d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921009cda0 con 0x7f92100c3c00 2021-04-09T18:12:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9216a0d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92100c3680 con 0x7f92101a1270 2021-04-09T18:12:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9216a0d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9210095640 con 0x7f9210095240 2021-04-09T18:12:31.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9214829700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100c3c00 0x7f921009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9214829700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100c3c00 0x7f921009c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41398/0 (socket says 172.21.15.32:41398) 2021-04-09T18:12:31.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9214829700 1 -- 172.21.15.32:0/2078868654 learned_addr learned my addr 172.21.15.32:0/2078868654 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:31.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f920ffff700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9210095240 0x7f92101a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9214829700 1 -- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9210095240 msgr2=0x7f92101a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9214829700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9210095240 0x7f92101a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9214829700 1 -- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92101a1270 msgr2=0x7f92100b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9214829700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92101a1270 0x7f92100b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:12:31.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9214829700 1 -- 172.21.15.32:0/2078868654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92101a1670 con 0x7f92100c3c00 2021-04-09T18:12:31.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9214829700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100c3c00 0x7f921009c600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f9200006f40 tx=0x7f920000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f920d7fa700 1 -- 172.21.15.32:0/2078868654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920000bd80 con 0x7f92100c3c00 2021-04-09T18:12:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f920d7fa700 1 -- 172.21.15.32:0/2078868654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9200000c90 con 0x7f92100c3c00 2021-04-09T18:12:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92101a1670 con 0x7f92100c3c00 2021-04-09T18:12:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.030 7f920d7fa700 1 -- 172.21.15.32:0/2078868654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9200002220 con 0x7f92100c3c00 2021-04-09T18:12:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.031 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92101a1670 con 0x7f92100c3c00 2021-04-09T18:12:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.031 7f920d7fa700 1 -- 172.21.15.32:0/2078868654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f920000a000 con 0x7f92100c3c00 2021-04-09T18:12:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.031 7f920d7fa700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91f0018930 0x7f91f001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.031 7f920d7fa700 1 -- 172.21.15.32:0/2078868654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f92000326b0 con 0x7f92100c3c00 2021-04-09T18:12:31.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.031 7f920ffff700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91f0018930 0x7f91f001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.032 7f920ffff700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91f0018930 0x7f91f001ae10 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.051 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9210093d50 con 0x7f92100c3c00 2021-04-09T18:12:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.055 7f920d7fa700 1 -- 172.21.15.32:0/2078868654 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9200000c90 con 0x7f92100c3c00 2021-04-09T18:12:31.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.235 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f921010d480 con 0x7f92100c3c00 2021-04-09T18:12:31.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.236 7f920d7fa700 1 -- 172.21.15.32:0/2078868654 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9200006b00 con 0x7f92100c3c00 2021-04-09T18:12:31.256 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:12:31.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91f0018930 msgr2=0x7f91f001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91f0018930 0x7f91f001ae10 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100c3c00 msgr2=0x7f921009c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100c3c00 0x7f921009c600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f9200006f40 tx=0x7f920000b990).stop 2021-04-09T18:12:31.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 shutdown_connections 2021-04-09T18:12:31.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91f0018930 0x7f91f001ae10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92101a1270 0x7f92100b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:12:31.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9210095240 0x7f92101a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 --2- 172.21.15.32:0/2078868654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92100c3c00 0x7f921009c600 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 >> 172.21.15.32:0/2078868654 conn(0x7f92100975f0 msgr2=0x7f9210096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 shutdown_connections 2021-04-09T18:12:31.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.238 7f9216a0d700 1 -- 172.21.15.32:0/2078868654 wait complete. 2021-04-09T18:12:31.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:12:31.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.471 7fb7a9dc5700 1 Processor -- start 2021-04-09T18:12:31.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.471 7fb7a9dc5700 1 -- start start 2021-04-09T18:12:31.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.472 7fb7a9dc5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb7a410d010 0x7fb7a40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.472 7fb7a9dc5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7a40d8490 0x7fb7a410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.472 7fb7a9dc5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a410d480 con 0x7fb7a40d1da0 2021-04-09T18:12:31.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.472 7fb7a9dc5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a40d8d30 con 0x7fb7a40d8490 2021-04-09T18:12:31.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.472 7fb7a9dc5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a40d9220 con 0x7fb7a410d010 2021-04-09T18:12:31.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.472 7fb7a3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7a40d8490 0x7fb7a410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.472 7fb7a3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7a40d8490 0x7fb7a410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41408/0 (socket says 172.21.15.32:41408) 2021-04-09T18:12:31.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.472 7fb7a3fff700 1 -- 172.21.15.32:0/804215524 learned_addr learned my addr 172.21.15.32:0/804215524 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:31.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.472 7fb7a37fe700 1 --2- 172.21.15.32:0/804215524 >> v2:172.21.15.86:3300/0 conn(0x7fb7a410d010 0x7fb7a40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a3fff700 1 -- 172.21.15.32:0/804215524 >> v2:172.21.15.86:3300/0 conn(0x7fb7a410d010 msgr2=0x7fb7a40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:31.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a3fff700 1 --2- 172.21.15.32:0/804215524 >> v2:172.21.15.86:3300/0 conn(0x7fb7a410d010 0x7fb7a40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a3fff700 1 -- 172.21.15.32:0/804215524 >> v1:172.21.15.86:6789/0 conn(0x7fb7a40d1da0 legacy=0x7fb7a40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a3fff700 1 -- 172.21.15.32:0/804215524 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a410e020 con 0x7fb7a40d8490 2021-04-09T18:12:31.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a37fe700 1 --2- 172.21.15.32:0/804215524 >> v2:172.21.15.86:3300/0 conn(0x7fb7a410d010 0x7fb7a40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:31.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a3fff700 1 --2- 172.21.15.32:0/804215524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40d8490 0x7fb7a410da00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb79c006b80 tx=0x7fb79c009b90).ready entity=mon.0 client_cookie=fcba1b1bc9aeff77 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:31.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a27fc700 1 -- 172.21.15.32:0/804215524 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb79c0022e0 con 0x7fb7a40d8490 2021-04-09T18:12:31.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a27fc700 1 -- 172.21.15.32:0/804215524 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c002620 con 0x7fb7a40d8490 2021-04-09T18:12:31.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a27fc700 1 -- 172.21.15.32:0/804215524 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb79c002b00 con 0x7fb7a40d8490 2021-04-09T18:12:31.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a9dc5700 1 -- 172.21.15.32:0/804215524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40d8490 msgr2=0x7fb7a410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.473 7fb7a9dc5700 1 --2- 172.21.15.32:0/804215524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40d8490 0x7fb7a410da00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb79c006b80 tx=0x7fb79c009b90).stop 2021-04-09T18:12:31.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.474 7fb7a9dc5700 1 -- 172.21.15.32:0/804215524 shutdown_connections 2021-04-09T18:12:31.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.474 7fb7a9dc5700 1 --2- 172.21.15.32:0/804215524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a40d8490 0x7fb7a410da00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.474 7fb7a9dc5700 1 --2- 172.21.15.32:0/804215524 >> v2:172.21.15.86:3300/0 conn(0x7fb7a410d010 0x7fb7a40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.474 7fb7a9dc5700 1 -- 172.21.15.32:0/804215524 >> 172.21.15.32:0/804215524 conn(0x7fb7a40ce5f0 msgr2=0x7fb7a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.474 7fb7a9dc5700 1 -- 172.21.15.32:0/804215524 shutdown_connections 2021-04-09T18:12:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.474 7fb7a9dc5700 1 -- 172.21.15.32:0/804215524 wait complete. 2021-04-09T18:12:31.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.494 7fb7a9dc5700 1 Processor -- start 2021-04-09T18:12:31.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.494 7fb7a9dc5700 1 -- start start 2021-04-09T18:12:31.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.494 7fb7a9dc5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7a40ac0f0 0x7fb7a40ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.494 7fb7a9dc5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a41a6c20 0x7fb7a41a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.494 7fb7a9dc5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7a41a56c0 0x7fb7a41a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.494 7fb7a9dc5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a41a76c0 con 0x7fb7a41a6c20 2021-04-09T18:12:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.494 7fb7a9dc5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a417c7e0 con 0x7fb7a41a56c0 2021-04-09T18:12:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.494 7fb7a9dc5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a417ca00 con 0x7fb7a40ac0f0 2021-04-09T18:12:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a41a6c20 0x7fb7a41a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a41a6c20 0x7fb7a41a7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41416/0 (socket says 172.21.15.32:41416) 2021-04-09T18:12:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a37fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7a40ac0f0 0x7fb7a40ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a2ffd700 1 -- 172.21.15.32:0/2847175875 learned_addr learned my addr 172.21.15.32:0/2847175875 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a3fff700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7a41a56c0 0x7fb7a41a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a2ffd700 1 -- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7a40ac0f0 msgr2=0x7fb7a40ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a2ffd700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7a40ac0f0 0x7fb7a40ac4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a2ffd700 1 -- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7a41a56c0 msgr2=0x7fb7a41a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a2ffd700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7a41a56c0 0x7fb7a41a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a2ffd700 1 -- 172.21.15.32:0/2847175875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a41a60e0 con 0x7fb7a41a6c20 2021-04-09T18:12:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a2ffd700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a41a6c20 0x7fb7a41a7020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb79800b8e0 tx=0x7fb79800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb78bfff700 1 -- 172.21.15.32:0/2847175875 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb79800c2f0 con 0x7fb7a41a6c20 2021-04-09T18:12:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb78bfff700 1 -- 172.21.15.32:0/2847175875 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79800c570 con 0x7fb7a41a6c20 2021-04-09T18:12:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a41a60e0 con 0x7fb7a41a6c20 2021-04-09T18:12:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb78bfff700 1 -- 172.21.15.32:0/2847175875 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb79800c9d0 con 0x7fb7a41a6c20 2021-04-09T18:12:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.495 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a41a60e0 con 0x7fb7a41a6c20 2021-04-09T18:12:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.496 7fb78bfff700 1 -- 172.21.15.32:0/2847175875 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb798001110 con 0x7fb7a41a6c20 2021-04-09T18:12:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.496 7fb78bfff700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb784018930 0x7fb78401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.496 7fb78bfff700 1 -- 172.21.15.32:0/2847175875 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fb798032aa0 con 0x7fb7a41a6c20 2021-04-09T18:12:31.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.496 7fb7a37fe700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb784018930 0x7fb78401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.496 7fb7a37fe700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb784018930 0x7fb78401ae10 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.516 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a41b7050 con 0x7fb7a41a6c20 2021-04-09T18:12:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.520 7fb78bfff700 1 -- 172.21.15.32:0/2847175875 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb79800bdc0 con 0x7fb7a41a6c20 2021-04-09T18:12:31.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.699 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb7a410d480 con 0x7fb7a41a6c20 2021-04-09T18:12:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.700 7fb78bfff700 1 -- 172.21.15.32:0/2847175875 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+64 (secure 0 0 0) 0x7fb798001090 con 0x7fb7a41a6c20 2021-04-09T18:12:31.722 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:31.722 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:12:31.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:31.723 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:12:31.723 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:12:31.723 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:31.724 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:12:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb784018930 msgr2=0x7fb78401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb784018930 0x7fb78401ae10 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a41a6c20 msgr2=0x7fb7a41a7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a41a6c20 0x7fb7a41a7020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb79800b8e0 tx=0x7fb79800bc80).stop 2021-04-09T18:12:31.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 shutdown_connections 2021-04-09T18:12:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb784018930 0x7fb78401ae10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7a41a6c20 0x7fb7a41a7020 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7a41a56c0 0x7fb7a41a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 --2- 172.21.15.32:0/2847175875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7a40ac0f0 0x7fb7a40ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.702 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 >> 172.21.15.32:0/2847175875 conn(0x7fb7a4199680 msgr2=0x7fb7a41a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.703 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 shutdown_connections 2021-04-09T18:12:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.703 7fb7a9dc5700 1 -- 172.21.15.32:0/2847175875 wait complete. 2021-04-09T18:12:31.736 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:12:31.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:12:31.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.871 7f63c7201700 1 Processor -- start 2021-04-09T18:12:31.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.871 7f63c7201700 1 -- start start 2021-04-09T18:12:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c7201700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f63c00d7ca0 0x7f63c00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c7201700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c010da00 0x7f63c010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c7201700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c010d480 con 0x7f63c010d010 2021-04-09T18:12:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c7201700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c00d9220 con 0x7f63c00d7ca0 2021-04-09T18:12:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c7201700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c00d86c0 con 0x7f63c010da00 2021-04-09T18:12:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c481c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f63c00d7ca0 0x7f63c00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c501d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f63c010d010 0x7f63c00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51010/0 (socket says 172.21.15.32:51010) 2021-04-09T18:12:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c501d700 1 -- 172.21.15.32:0/720332002 learned_addr learned my addr 172.21.15.32:0/720332002 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c581e700 1 --2- 172.21.15.32:0/720332002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c010da00 0x7f63c010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63b7fff700 1 -- 172.21.15.32:0/720332002 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153522020 0 0) 0x7f63bc000ce0 con 0x7f63c010d010 2021-04-09T18:12:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63b7fff700 1 -- 172.21.15.32:0/720332002 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac002fe0 con 0x7f63c010d010 2021-04-09T18:12:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c481c700 1 -- 172.21.15.32:0/720332002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c010da00 msgr2=0x7f63c010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c481c700 1 --2- 172.21.15.32:0/720332002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c010da00 0x7f63c010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c481c700 1 -- 172.21.15.32:0/720332002 >> v1:172.21.15.32:6789/0 conn(0x7f63c010d010 legacy=0x7f63c00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c481c700 1 -- 172.21.15.32:0/720332002 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c00d1f70 con 0x7f63c00d7ca0 2021-04-09T18:12:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.872 7f63c481c700 1 --2- 172.21.15.32:0/720332002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00d7ca0 0x7f63c00d80a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f63b0006650 tx=0x7f63b0009a80).ready entity=mon.0 client_cookie=a99e3bd5805e4764 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:31.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.873 7f63b7fff700 1 -- 172.21.15.32:0/720332002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b00009b0 con 0x7f63c00d7ca0 2021-04-09T18:12:31.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.873 7f63b7fff700 1 -- 172.21.15.32:0/720332002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b000c0c0 con 0x7f63c00d7ca0 2021-04-09T18:12:31.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.873 7f63b7fff700 1 -- 172.21.15.32:0/720332002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b000c510 con 0x7f63c00d7ca0 2021-04-09T18:12:31.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.873 7f63c7201700 1 -- 172.21.15.32:0/720332002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00d7ca0 msgr2=0x7f63c00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:31.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.873 7f63c7201700 1 --2- 172.21.15.32:0/720332002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00d7ca0 0x7f63c00d80a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f63b0006650 tx=0x7f63b0009a80).stop 2021-04-09T18:12:31.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.873 7f63c7201700 1 -- 172.21.15.32:0/720332002 shutdown_connections 2021-04-09T18:12:31.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.873 7f63c7201700 1 --2- 172.21.15.32:0/720332002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c010da00 0x7f63c010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.873 7f63c7201700 1 --2- 172.21.15.32:0/720332002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00d7ca0 0x7f63c00d80a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:31.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.873 7f63c7201700 1 -- 172.21.15.32:0/720332002 >> 172.21.15.32:0/720332002 conn(0x7f63c00ce5f0 msgr2=0x7f63c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:31.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.874 7f63c7201700 1 -- 172.21.15.32:0/720332002 shutdown_connections 2021-04-09T18:12:32.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.874 7f63c7201700 1 -- 172.21.15.32:0/720332002 wait complete. 2021-04-09T18:12:32.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c7201700 1 Processor -- start 2021-04-09T18:12:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c7201700 1 -- start start 2021-04-09T18:12:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c7201700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c00b1680 0x7f63c00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c7201700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c009e330 0x7f63c00b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:32.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c7201700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00b2190 0x7f63c009f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:32.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c7201700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c009e730 con 0x7f63c00b2190 2021-04-09T18:12:32.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c7201700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c009f6c0 con 0x7f63c009e330 2021-04-09T18:12:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c7201700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c009f8e0 con 0x7f63c00b1680 2021-04-09T18:12:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c581e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00b2190 0x7f63c009f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c581e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00b2190 0x7f63c009f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41428/0 (socket says 172.21.15.32:41428) 2021-04-09T18:12:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.894 7f63c581e700 1 -- 172.21.15.32:0/1365228655 learned_addr learned my addr 172.21.15.32:0/1365228655 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c501d700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c00b1680 0x7f63c00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c481c700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c009e330 0x7f63c00b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c581e700 1 -- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c00b1680 msgr2=0x7f63c00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c581e700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c00b1680 0x7f63c00b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c581e700 1 -- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c009e330 msgr2=0x7f63c00b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:32.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c581e700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c009e330 0x7f63c00b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c581e700 1 -- 172.21.15.32:0/1365228655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c009fb00 con 0x7f63c00b2190 2021-04-09T18:12:32.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c481c700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c009e330 0x7f63c00b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:12:32.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c581e700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00b2190 0x7f63c009f0a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f63b800b910 tx=0x7f63b800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:32.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63b5ffb700 1 -- 172.21.15.32:0/1365228655 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b8006aa0 con 0x7f63c00b2190 2021-04-09T18:12:32.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63b5ffb700 1 -- 172.21.15.32:0/1365228655 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8003030 con 0x7f63c00b2190 2021-04-09T18:12:32.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c7201700 1 -- 172.21.15.32:0/1365228655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c009fb00 con 0x7f63c00b2190 2021-04-09T18:12:32.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63b5ffb700 1 -- 172.21.15.32:0/1365228655 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b8001470 con 0x7f63c00b2190 2021-04-09T18:12:32.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.895 7f63c7201700 1 -- 172.21.15.32:0/1365228655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c009fb00 con 0x7f63c00b2190 2021-04-09T18:12:32.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.896 7f63b5ffb700 1 -- 172.21.15.32:0/1365228655 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f63b8000ec0 con 0x7f63c00b2190 2021-04-09T18:12:32.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.896 7f63b5ffb700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63a0018930 0x7f63a001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:32.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.896 7f63b5ffb700 1 -- 172.21.15.32:0/1365228655 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f63b8032ab0 con 0x7f63c00b2190 2021-04-09T18:12:32.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.896 7f63c501d700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63a0018930 0x7f63a001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:32.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.896 7f63c501d700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63a0018930 0x7f63a001adc0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:32.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.916 7f63c7201700 1 -- 172.21.15.32:0/1365228655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c010d480 con 0x7f63c00b2190 2021-04-09T18:12:32.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:31.919 7f63b5ffb700 1 -- 172.21.15.32:0/1365228655 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f63b800bdf0 con 0x7f63c00b2190 2021-04-09T18:12:32.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.101 7f63c7201700 1 -- 172.21.15.32:0/1365228655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f63c010d480 con 0x7f63c00b2190 2021-04-09T18:12:32.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.322 7f63b5ffb700 1 -- 172.21.15.32:0/1365228655 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v17) v1 ==== 133+0+0 (secure 0 0 0) 0x7f63b8000ec0 con 0x7f63c00b2190 2021-04-09T18:12:32.391 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:12:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 -- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63a0018930 msgr2=0x7f63a001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:32.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63a0018930 0x7f63a001adc0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 -- 172.21.15.32:0/1365228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00b2190 msgr2=0x7f63c009f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00b2190 0x7f63c009f0a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f63b800b910 tx=0x7f63b800bcb0).stop 2021-04-09T18:12:32.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 -- 172.21.15.32:0/1365228655 shutdown_connections 2021-04-09T18:12:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63a0018930 0x7f63a001adc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c00b2190 0x7f63c009f0a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c009e330 0x7f63c00b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 --2- 172.21.15.32:0/1365228655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c00b1680 0x7f63c00b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 -- 172.21.15.32:0/1365228655 >> 172.21.15.32:0/1365228655 conn(0x7f63c00c4290 msgr2=0x7f63c00c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:32.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 -- 172.21.15.32:0/1365228655 shutdown_connections 2021-04-09T18:12:32.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.368 7f63c7201700 1 -- 172.21.15.32:0/1365228655 wait complete. 2021-04-09T18:12:32.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:12:32.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.566 7fe258ee2700 1 Processor -- start 2021-04-09T18:12:32.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.566 7fe258ee2700 1 -- start start 2021-04-09T18:12:32.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe258ee2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2540d6ca0 0x7fe2540d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:32.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe258ee2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2540d72a0 con 0x7fe2540cffc0 2021-04-09T18:12:32.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe258ee2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2540d2860 con 0x7fe2540d2460 2021-04-09T18:12:32.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe258ee2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2540d70a0 con 0x7fe2540d6ca0 2021-04-09T18:12:32.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe252312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2540d6ca0 0x7fe2540d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe252b13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe2540cffc0 0x7fe2540d7820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51024/0 (socket says 172.21.15.32:51024) 2021-04-09T18:12:32.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe252b13700 1 -- 172.21.15.32:0/3871922319 learned_addr learned my addr 172.21.15.32:0/3871922319 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:32.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe252312700 1 -- 172.21.15.32:0/3871922319 >> v1:172.21.15.86:6789/0 conn(0x7fe2540d2460 legacy=0x7fe25410cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:32.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe252312700 1 -- 172.21.15.32:0/3871922319 >> v1:172.21.15.32:6789/0 conn(0x7fe2540cffc0 legacy=0x7fe2540d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:32.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe252312700 1 -- 172.21.15.32:0/3871922319 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2540d9200 con 0x7fe2540d6ca0 2021-04-09T18:12:32.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe251b11700 1 -- 172.21.15.32:0/3871922319 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881782837 0 0) 0x7fe244000ce0 con 0x7fe2540cffc0 2021-04-09T18:12:32.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.567 7fe252312700 1 --2- 172.21.15.32:0/3871922319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540d6ca0 0x7fe2540d8010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fe248006650 tx=0x7fe248009a80).ready entity=mon.0 client_cookie=e7b6354db7f5d0b7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:32.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.568 7fe251b11700 1 -- 172.21.15.32:0/3871922319 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2480009b0 con 0x7fe2540d6ca0 2021-04-09T18:12:32.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.568 7fe251b11700 1 -- 172.21.15.32:0/3871922319 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24800c0c0 con 0x7fe2540d6ca0 2021-04-09T18:12:32.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.568 7fe251b11700 1 -- 172.21.15.32:0/3871922319 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe24800c510 con 0x7fe2540d6ca0 2021-04-09T18:12:32.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.568 7fe258ee2700 1 -- 172.21.15.32:0/3871922319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540d6ca0 msgr2=0x7fe2540d8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:32.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.568 7fe258ee2700 1 --2- 172.21.15.32:0/3871922319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540d6ca0 0x7fe2540d8010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fe248006650 tx=0x7fe248009a80).stop 2021-04-09T18:12:32.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.568 7fe258ee2700 1 -- 172.21.15.32:0/3871922319 shutdown_connections 2021-04-09T18:12:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.568 7fe258ee2700 1 --2- 172.21.15.32:0/3871922319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540d6ca0 0x7fe2540d8010 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.568 7fe258ee2700 1 -- 172.21.15.32:0/3871922319 >> 172.21.15.32:0/3871922319 conn(0x7fe2540ce5e0 msgr2=0x7fe2540cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:32.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.568 7fe258ee2700 1 -- 172.21.15.32:0/3871922319 shutdown_connections 2021-04-09T18:12:32.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.569 7fe258ee2700 1 -- 172.21.15.32:0/3871922319 wait complete. 2021-04-09T18:12:32.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe258ee2700 1 Processor -- start 2021-04-09T18:12:32.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe258ee2700 1 -- start start 2021-04-09T18:12:32.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe258ee2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2540c0de0 0x7fe2540c1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:32.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe258ee2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540c20d0 0x7fe2540b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:32.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe258ee2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2540b94d0 0x7fe2540b98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:32.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe258ee2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2540c15b0 con 0x7fe2540c20d0 2021-04-09T18:12:32.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe258ee2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2540c2510 con 0x7fe2540c0de0 2021-04-09T18:12:32.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe258ee2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2540c11e0 con 0x7fe2540b94d0 2021-04-09T18:12:32.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe252312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540c20d0 0x7fe2540b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:32.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe252312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540c20d0 0x7fe2540b8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41444/0 (socket says 172.21.15.32:41444) 2021-04-09T18:12:32.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe252312700 1 -- 172.21.15.32:0/2352180202 learned_addr learned my addr 172.21.15.32:0/2352180202 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:32.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe253314700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2540b94d0 0x7fe2540b98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:32.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.589 7fe252b13700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2540c0de0 0x7fe2540c1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:32.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe252312700 1 -- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2540b94d0 msgr2=0x7fe2540b98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:32.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe252312700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2540b94d0 0x7fe2540b98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe252312700 1 -- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2540c0de0 msgr2=0x7fe2540c1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:32.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe252312700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2540c0de0 0x7fe2540c1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe252312700 1 -- 172.21.15.32:0/2352180202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2540934c0 con 0x7fe2540c20d0 2021-04-09T18:12:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe252312700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540c20d0 0x7fe2540b8eb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fe248009650 tx=0x7fe24800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe2437fe700 1 -- 172.21.15.32:0/2352180202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2480093a0 con 0x7fe2540c20d0 2021-04-09T18:12:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe2437fe700 1 -- 172.21.15.32:0/2352180202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe248006070 con 0x7fe2540c20d0 2021-04-09T18:12:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2540934c0 con 0x7fe2540c20d0 2021-04-09T18:12:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe2437fe700 1 -- 172.21.15.32:0/2352180202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2480052e0 con 0x7fe2540c20d0 2021-04-09T18:12:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.590 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2540934c0 con 0x7fe2540c20d0 2021-04-09T18:12:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.591 7fe2437fe700 1 -- 172.21.15.32:0/2352180202 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe248009300 con 0x7fe2540c20d0 2021-04-09T18:12:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.591 7fe2437fe700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2380188c0 0x7fe23801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.591 7fe2437fe700 1 -- 172.21.15.32:0/2352180202 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5111+0+0 (secure 0 0 0) 0x7fe2480325b0 con 0x7fe2540c20d0 2021-04-09T18:12:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.591 7fe252b13700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2380188c0 0x7fe23801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.592 7fe252b13700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2380188c0 0x7fe23801adc0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.611 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25408dba0 con 0x7fe2540c20d0 2021-04-09T18:12:32.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.615 7fe2437fe700 1 -- 172.21.15.32:0/2352180202 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe248000940 con 0x7fe2540c20d0 2021-04-09T18:12:32.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:32.797 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fe25408dba0 con 0x7fe2540c20d0 2021-04-09T18:12:33.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.324 7fe2437fe700 1 -- 172.21.15.32:0/2352180202 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v18) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe248009300 con 0x7fe2540c20d0 2021-04-09T18:12:33.381 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:12:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2380188c0 msgr2=0x7fe23801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2380188c0 0x7fe23801adc0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540c20d0 msgr2=0x7fe2540b8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540c20d0 0x7fe2540b8eb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fe248009650 tx=0x7fe24800b8e0).stop 2021-04-09T18:12:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 shutdown_connections 2021-04-09T18:12:33.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2380188c0 0x7fe23801adc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2540c20d0 0x7fe2540b8eb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2540b94d0 0x7fe2540b98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 --2- 172.21.15.32:0/2352180202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2540c0de0 0x7fe2540c1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 >> 172.21.15.32:0/2352180202 conn(0x7fe2540a3640 msgr2=0x7fe25408ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 shutdown_connections 2021-04-09T18:12:33.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.364 7fe258ee2700 1 -- 172.21.15.32:0/2352180202 wait complete. 2021-04-09T18:12:33.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:12:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.539 7f3759590700 1 Processor -- start 2021-04-09T18:12:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.539 7f3759590700 1 -- start start 2021-04-09T18:12:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3759590700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37540d7cb0 0x7f37540d80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3759590700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f375410da10 0x7f375410de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3759590700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375410d490 con 0x7f375410d020 2021-04-09T18:12:33.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3759590700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37540d9230 con 0x7f37540d7cb0 2021-04-09T18:12:33.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3759590700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37540d86d0 con 0x7f375410da10 2021-04-09T18:12:33.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3752abe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37540d7cb0 0x7f37540d80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:33.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3752abe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37540d7cb0 0x7f37540d80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41452/0 (socket says 172.21.15.32:41452) 2021-04-09T18:12:33.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3752abe700 1 -- 172.21.15.32:0/2374626686 learned_addr learned my addr 172.21.15.32:0/2374626686 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3753ac0700 1 --2- 172.21.15.32:0/2374626686 >> v2:172.21.15.86:3300/0 conn(0x7f375410da10 0x7f375410de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:33.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3752abe700 1 -- 172.21.15.32:0/2374626686 >> v2:172.21.15.86:3300/0 conn(0x7f375410da10 msgr2=0x7f375410de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:33.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3752abe700 1 --2- 172.21.15.32:0/2374626686 >> v2:172.21.15.86:3300/0 conn(0x7f375410da10 0x7f375410de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3752abe700 1 -- 172.21.15.32:0/2374626686 >> v1:172.21.15.86:6789/0 conn(0x7f375410d020 legacy=0x7f37540d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3752abe700 1 -- 172.21.15.32:0/2374626686 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37540d1f80 con 0x7f37540d7cb0 2021-04-09T18:12:33.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.540 7f3752abe700 1 --2- 172.21.15.32:0/2374626686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540d7cb0 0x7f37540d80b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f373c006700 tx=0x7f373c009b80).ready entity=mon.0 client_cookie=17f5b44e284aa46c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:33.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f37522bd700 1 -- 172.21.15.32:0/2374626686 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f373c00c0c0 con 0x7f37540d7cb0 2021-04-09T18:12:33.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f37522bd700 1 -- 172.21.15.32:0/2374626686 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c00c440 con 0x7f37540d7cb0 2021-04-09T18:12:33.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f37522bd700 1 -- 172.21.15.32:0/2374626686 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f373c00c920 con 0x7f37540d7cb0 2021-04-09T18:12:33.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f3759590700 1 -- 172.21.15.32:0/2374626686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540d7cb0 msgr2=0x7f37540d80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f3759590700 1 --2- 172.21.15.32:0/2374626686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540d7cb0 0x7f37540d80b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f373c006700 tx=0x7f373c009b80).stop 2021-04-09T18:12:33.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f3759590700 1 -- 172.21.15.32:0/2374626686 shutdown_connections 2021-04-09T18:12:33.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f3759590700 1 --2- 172.21.15.32:0/2374626686 >> v2:172.21.15.86:3300/0 conn(0x7f375410da10 0x7f375410de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f3759590700 1 --2- 172.21.15.32:0/2374626686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37540d7cb0 0x7f37540d80b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f3759590700 1 -- 172.21.15.32:0/2374626686 >> 172.21.15.32:0/2374626686 conn(0x7f37540ce600 msgr2=0x7f37540d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:33.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f3759590700 1 -- 172.21.15.32:0/2374626686 shutdown_connections 2021-04-09T18:12:33.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.541 7f3759590700 1 -- 172.21.15.32:0/2374626686 wait complete. 2021-04-09T18:12:33.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.561 7f3759590700 1 Processor -- start 2021-04-09T18:12:33.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f3759590700 1 -- start start 2021-04-09T18:12:33.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f3759590700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754095190 0x7f37540948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:33.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f3759590700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3754092f30 0x7f375409a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:33.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f3759590700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3754093330 0x7f375409acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f3759590700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375409b360 con 0x7f3754095190 2021-04-09T18:12:33.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f3759590700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375409b570 con 0x7f3754093330 2021-04-09T18:12:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f3759590700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37540c0bc0 con 0x7f3754092f30 2021-04-09T18:12:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f37532bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754095190 0x7f37540948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f37532bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754095190 0x7f37540948b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41454/0 (socket says 172.21.15.32:41454) 2021-04-09T18:12:33.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f37532bf700 1 -- 172.21.15.32:0/1362196268 learned_addr learned my addr 172.21.15.32:0/1362196268 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f3752abe700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3754092f30 0x7f375409a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f3753ac0700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3754093330 0x7f375409acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f37532bf700 1 -- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3754092f30 msgr2=0x7f375409a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.562 7f37532bf700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3754092f30 0x7f375409a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f37532bf700 1 -- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3754093330 msgr2=0x7f375409acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f37532bf700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3754093330 0x7f375409acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f37532bf700 1 -- 172.21.15.32:0/1362196268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375409bac0 con 0x7f3754095190 2021-04-09T18:12:33.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f3752abe700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3754092f30 0x7f375409a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f3753ac0700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3754093330 0x7f375409acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f37532bf700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754095190 0x7f37540948b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f374400b5b0 tx=0x7f374400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f374bfff700 1 -- 172.21.15.32:0/1362196268 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f374400bd40 con 0x7f3754095190 2021-04-09T18:12:33.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f374bfff700 1 -- 172.21.15.32:0/1362196268 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3744006240 con 0x7f3754095190 2021-04-09T18:12:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f3759590700 1 -- 172.21.15.32:0/1362196268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375409bac0 con 0x7f3754095190 2021-04-09T18:12:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f374bfff700 1 -- 172.21.15.32:0/1362196268 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3744005690 con 0x7f3754095190 2021-04-09T18:12:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f3759590700 1 -- 172.21.15.32:0/1362196268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375409bac0 con 0x7f3754095190 2021-04-09T18:12:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.563 7f374bfff700 1 -- 172.21.15.32:0/1362196268 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f37440054a0 con 0x7f3754095190 2021-04-09T18:12:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.564 7f374bfff700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37380188d0 0x7f373801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:33.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.564 7f374bfff700 1 -- 172.21.15.32:0/1362196268 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f37440329c0 con 0x7f3754095190 2021-04-09T18:12:33.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.564 7f3752abe700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37380188d0 0x7f373801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:33.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.564 7f3752abe700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37380188d0 0x7f373801add0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:33.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.584 7f3759590700 1 -- 172.21.15.32:0/1362196268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37541b7120 con 0x7f3754095190 2021-04-09T18:12:33.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.588 7f374bfff700 1 -- 172.21.15.32:0/1362196268 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3744000c60 con 0x7f3754095190 2021-04-09T18:12:33.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.770 7f3759590700 1 -- 172.21.15.32:0/1362196268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f375410d490 con 0x7f3754095190 2021-04-09T18:12:33.790 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:12:33.771 7f51150fb700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:12:33.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.774 7f374bfff700 1 -- 172.21.15.32:0/1362196268 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 187+0+0 (secure 0 0 0) 0x7f3744006240 con 0x7f3754095190 2021-04-09T18:12:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 2 and data pool 3 2021-04-09T18:12:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 -- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37380188d0 msgr2=0x7f373801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37380188d0 0x7f373801add0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 -- 172.21.15.32:0/1362196268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754095190 msgr2=0x7f37540948b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754095190 0x7f37540948b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f374400b5b0 tx=0x7f374400b950).stop 2021-04-09T18:12:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 -- 172.21.15.32:0/1362196268 shutdown_connections 2021-04-09T18:12:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37380188d0 0x7f373801add0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3754093330 0x7f375409acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3754092f30 0x7f375409a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 --2- 172.21.15.32:0/1362196268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3754095190 0x7f37540948b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 -- 172.21.15.32:0/1362196268 >> 172.21.15.32:0/1362196268 conn(0x7f375408c120 msgr2=0x7f3754099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:33.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 -- 172.21.15.32:0/1362196268 shutdown_connections 2021-04-09T18:12:33.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.780 7f3759590700 1 -- 172.21.15.32:0/1362196268 wait complete. 2021-04-09T18:12:33.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:12:33.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.945 7ff6b82da700 1 Processor -- start 2021-04-09T18:12:33.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.945 7ff6b82da700 1 -- start start 2021-04-09T18:12:33.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b82da700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff6b010d010 0x7ff6b00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:33.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b82da700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff6b00d8490 0x7ff6b010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:33.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b82da700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b010d480 con 0x7ff6b00d1da0 2021-04-09T18:12:33.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b82da700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b00d8d30 con 0x7ff6b010d010 2021-04-09T18:12:33.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b82da700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b00d9220 con 0x7ff6b00d8490 2021-04-09T18:12:33.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b60f6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff6b010d010 0x7ff6b00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:33.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b60f6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff6b010d010 0x7ff6b00d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41462/0 (socket says 172.21.15.32:41462) 2021-04-09T18:12:33.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b60f6700 1 -- 172.21.15.32:0/1940175173 learned_addr learned my addr 172.21.15.32:0/1940175173 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:33.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b68f7700 1 --2- 172.21.15.32:0/1940175173 >> v2:172.21.15.86:3300/0 conn(0x7ff6b00d8490 0x7ff6b010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:33.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b60f6700 1 -- 172.21.15.32:0/1940175173 >> v2:172.21.15.86:3300/0 conn(0x7ff6b00d8490 msgr2=0x7ff6b010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:33.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b60f6700 1 --2- 172.21.15.32:0/1940175173 >> v2:172.21.15.86:3300/0 conn(0x7ff6b00d8490 0x7ff6b010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b60f6700 1 -- 172.21.15.32:0/1940175173 >> v1:172.21.15.86:6789/0 conn(0x7ff6b00d1da0 legacy=0x7ff6b00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b60f6700 1 -- 172.21.15.32:0/1940175173 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b010e020 con 0x7ff6b010d010 2021-04-09T18:12:33.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b50f4700 1 -- 172.21.15.32:0/1940175173 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841736127 0 0) 0x7ff6a4000c60 con 0x7ff6b00d1da0 2021-04-09T18:12:33.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.946 7ff6b60f6700 1 --2- 172.21.15.32:0/1940175173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b010d010 0x7ff6b00d1780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff6ac006700 tx=0x7ff6ac009b00).ready entity=mon.0 client_cookie=30530e39784a3dfd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:33.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.947 7ff6b50f4700 1 -- 172.21.15.32:0/1940175173 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6ac00c0c0 con 0x7ff6b010d010 2021-04-09T18:12:33.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.947 7ff6b50f4700 1 -- 172.21.15.32:0/1940175173 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac00c400 con 0x7ff6b010d010 2021-04-09T18:12:33.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.947 7ff6b50f4700 1 -- 172.21.15.32:0/1940175173 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6ac0009b0 con 0x7ff6b010d010 2021-04-09T18:12:33.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.947 7ff6b82da700 1 -- 172.21.15.32:0/1940175173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b010d010 msgr2=0x7ff6b00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:33.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.947 7ff6b82da700 1 --2- 172.21.15.32:0/1940175173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b010d010 0x7ff6b00d1780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff6ac006700 tx=0x7ff6ac009b00).stop 2021-04-09T18:12:33.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.948 7ff6b82da700 1 -- 172.21.15.32:0/1940175173 shutdown_connections 2021-04-09T18:12:33.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.948 7ff6b82da700 1 --2- 172.21.15.32:0/1940175173 >> v2:172.21.15.86:3300/0 conn(0x7ff6b00d8490 0x7ff6b010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.948 7ff6b82da700 1 --2- 172.21.15.32:0/1940175173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b010d010 0x7ff6b00d1780 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:33.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.948 7ff6b82da700 1 -- 172.21.15.32:0/1940175173 >> 172.21.15.32:0/1940175173 conn(0x7ff6b00ce5f0 msgr2=0x7ff6b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.948 7ff6b82da700 1 -- 172.21.15.32:0/1940175173 shutdown_connections 2021-04-09T18:12:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.948 7ff6b82da700 1 -- 172.21.15.32:0/1940175173 wait complete. 2021-04-09T18:12:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b82da700 1 Processor -- start 2021-04-09T18:12:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b82da700 1 -- start start 2021-04-09T18:12:34.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b82da700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b00a90c0 0x7ff6b01a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b82da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b01a76d0 0x7ff6b01a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b82da700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b017bce0 0x7ff6b017c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b82da700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01a7c60 con 0x7ff6b01a76d0 2021-04-09T18:12:34.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b82da700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01a8e20 con 0x7ff6b00a90c0 2021-04-09T18:12:34.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b82da700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b017db80 con 0x7ff6b017bce0 2021-04-09T18:12:34.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b58f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b01a76d0 0x7ff6b01a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b58f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b01a76d0 0x7ff6b01a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41472/0 (socket says 172.21.15.32:41472) 2021-04-09T18:12:34.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.968 7ff6b58f5700 1 -- 172.21.15.32:0/950590867 learned_addr learned my addr 172.21.15.32:0/950590867 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:34.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b68f7700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b017bce0 0x7ff6b017c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b60f6700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b00a90c0 0x7ff6b01a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b58f5700 1 -- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b017bce0 msgr2=0x7ff6b017c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b58f5700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b017bce0 0x7ff6b017c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b58f5700 1 -- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b00a90c0 msgr2=0x7ff6b01a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b58f5700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b00a90c0 0x7ff6b01a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b58f5700 1 -- 172.21.15.32:0/950590867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b0179610 con 0x7ff6b01a76d0 2021-04-09T18:12:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b58f5700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b01a76d0 0x7ff6b01a8800 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff6a400b580 tx=0x7ff6a400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:34.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6a2ffd700 1 -- 172.21.15.32:0/950590867 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6a400bd10 con 0x7ff6b01a76d0 2021-04-09T18:12:34.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6a2ffd700 1 -- 172.21.15.32:0/950590867 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a4001080 con 0x7ff6b01a76d0 2021-04-09T18:12:34.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b82da700 1 -- 172.21.15.32:0/950590867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b0179610 con 0x7ff6b01a76d0 2021-04-09T18:12:34.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6a2ffd700 1 -- 172.21.15.32:0/950590867 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6a4001550 con 0x7ff6b01a76d0 2021-04-09T18:12:34.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.969 7ff6b82da700 1 -- 172.21.15.32:0/950590867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b0179610 con 0x7ff6b01a76d0 2021-04-09T18:12:34.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.970 7ff6a2ffd700 1 -- 172.21.15.32:0/950590867 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff6a4001260 con 0x7ff6b01a76d0 2021-04-09T18:12:34.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.970 7ff6a2ffd700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6980188c0 0x7ff69801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.970 7ff6a2ffd700 1 -- 172.21.15.32:0/950590867 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff6a4032870 con 0x7ff6b01a76d0 2021-04-09T18:12:34.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.970 7ff6b60f6700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6980188c0 0x7ff69801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.970 7ff6b60f6700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6980188c0 0x7ff69801adc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.990 7ff6b82da700 1 -- 172.21.15.32:0/950590867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b01b7010 con 0x7ff6b01a76d0 2021-04-09T18:12:34.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:33.993 7ff6a2ffd700 1 -- 172.21.15.32:0/950590867 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff6a4006bf0 con 0x7ff6b01a76d0 2021-04-09T18:12:34.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.180 7ff6b82da700 1 -- 172.21.15.32:0/950590867 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6b010d480 con 0x7ff6b01a76d0 2021-04-09T18:12:34.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.181 7ff6a2ffd700 1 -- 172.21.15.32:0/950590867 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14006 (secure 0 0 0) 0x7ff6a4001260 con 0x7ff6b01a76d0 2021-04-09T18:12:34.202 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:34.202 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":19,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:33.771991","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:12:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 -- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6980188c0 msgr2=0x7ff69801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6980188c0 0x7ff69801adc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 -- 172.21.15.32:0/950590867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b01a76d0 msgr2=0x7ff6b01a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b01a76d0 0x7ff6b01a8800 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff6a400b580 tx=0x7ff6a400b920).stop 2021-04-09T18:12:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 -- 172.21.15.32:0/950590867 shutdown_connections 2021-04-09T18:12:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6980188c0 0x7ff69801adc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6b01a76d0 0x7ff6b01a8800 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b017bce0 0x7ff6b017c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 --2- 172.21.15.32:0/950590867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b00a90c0 0x7ff6b01a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 -- 172.21.15.32:0/950590867 >> 172.21.15.32:0/950590867 conn(0x7ff6b01acae0 msgr2=0x7ff6b017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:34.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 -- 172.21.15.32:0/950590867 shutdown_connections 2021-04-09T18:12:34.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.183 7ff6b82da700 1 -- 172.21.15.32:0/950590867 wait complete. 2021-04-09T18:12:34.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:12:34.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.386 7f7e38064700 1 Processor -- start 2021-04-09T18:12:34.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.386 7f7e38064700 1 -- start start 2021-04-09T18:12:34.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e38064700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e300d7ca0 0x7f7e300d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e38064700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3010d480 con 0x7f7e3010d010 2021-04-09T18:12:34.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e38064700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e300d86c0 con 0x7f7e3010da00 2021-04-09T18:12:34.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e38064700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e300d9220 con 0x7f7e300d7ca0 2021-04-09T18:12:34.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e3567f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e300d7ca0 0x7f7e300d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e35e80700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7e3010d010 0x7f7e300d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51066/0 (socket says 172.21.15.32:51066) 2021-04-09T18:12:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e35e80700 1 -- 172.21.15.32:0/930972176 learned_addr learned my addr 172.21.15.32:0/930972176 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:34.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e34e7e700 1 -- 172.21.15.32:0/930972176 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184915652 0 0) 0x7f7e28000d30 con 0x7f7e3010da00 2021-04-09T18:12:34.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e34e7e700 1 -- 172.21.15.32:0/930972176 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e1c002fe0 con 0x7f7e3010da00 2021-04-09T18:12:34.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e3567f700 1 -- 172.21.15.32:0/930972176 >> v1:172.21.15.86:6789/0 conn(0x7f7e3010da00 legacy=0x7f7e3010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e34e7e700 1 -- 172.21.15.32:0/930972176 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197464287 0 0) 0x7f7e2c000ce0 con 0x7f7e3010d010 2021-04-09T18:12:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e3567f700 1 -- 172.21.15.32:0/930972176 >> v1:172.21.15.32:6789/0 conn(0x7f7e3010d010 legacy=0x7f7e300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.387 7f7e3567f700 1 -- 172.21.15.32:0/930972176 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e300d1f70 con 0x7f7e300d7ca0 2021-04-09T18:12:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.388 7f7e3567f700 1 --2- 172.21.15.32:0/930972176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e300d7ca0 0x7f7e300d80a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7e24006650 tx=0x7f7e24009a80).ready entity=mon.0 client_cookie=24c60af678dbf7c6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.388 7f7e34e7e700 1 -- 172.21.15.32:0/930972176 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e240009b0 con 0x7f7e300d7ca0 2021-04-09T18:12:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.388 7f7e34e7e700 1 -- 172.21.15.32:0/930972176 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2400c0c0 con 0x7f7e300d7ca0 2021-04-09T18:12:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.388 7f7e34e7e700 1 -- 172.21.15.32:0/930972176 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e2400c510 con 0x7f7e300d7ca0 2021-04-09T18:12:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.388 7f7e38064700 1 -- 172.21.15.32:0/930972176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e300d7ca0 msgr2=0x7f7e300d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.388 7f7e38064700 1 --2- 172.21.15.32:0/930972176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e300d7ca0 0x7f7e300d80a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7e24006650 tx=0x7f7e24009a80).stop 2021-04-09T18:12:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.389 7f7e38064700 1 -- 172.21.15.32:0/930972176 shutdown_connections 2021-04-09T18:12:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.389 7f7e38064700 1 --2- 172.21.15.32:0/930972176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e300d7ca0 0x7f7e300d80a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.389 7f7e38064700 1 -- 172.21.15.32:0/930972176 >> 172.21.15.32:0/930972176 conn(0x7f7e300ce5f0 msgr2=0x7f7e300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.389 7f7e38064700 1 -- 172.21.15.32:0/930972176 shutdown_connections 2021-04-09T18:12:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.389 7f7e38064700 1 -- 172.21.15.32:0/930972176 wait complete. 2021-04-09T18:12:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.409 7f7e38064700 1 Processor -- start 2021-04-09T18:12:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.409 7f7e38064700 1 -- start start 2021-04-09T18:12:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.409 7f7e38064700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e30099920 0x7f7e30179f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.409 7f7e38064700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3017a560 0x7f7e301b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.409 7f7e38064700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e301b85a0 0x7f7e301b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.409 7f7e38064700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3017a960 con 0x7f7e3017a560 2021-04-09T18:12:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.409 7f7e38064700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e30099d20 con 0x7f7e301b85a0 2021-04-09T18:12:34.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.409 7f7e38064700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e30179610 con 0x7f7e30099920 2021-04-09T18:12:34.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e3567f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3017a560 0x7f7e301b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e3567f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3017a560 0x7f7e301b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41484/0 (socket says 172.21.15.32:41484) 2021-04-09T18:12:34.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e3567f700 1 -- 172.21.15.32:0/907562387 learned_addr learned my addr 172.21.15.32:0/907562387 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e35e80700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e30099920 0x7f7e30179f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e36681700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e301b85a0 0x7f7e301b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e3567f700 1 -- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e30099920 msgr2=0x7f7e30179f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e3567f700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e30099920 0x7f7e30179f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e3567f700 1 -- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e301b85a0 msgr2=0x7f7e301b89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e3567f700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e301b85a0 0x7f7e301b89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e3567f700 1 -- 172.21.15.32:0/907562387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e30180be0 con 0x7f7e3017a560 2021-04-09T18:12:34.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e3567f700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3017a560 0x7f7e301b7f80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f7e24009650 tx=0x7f7e2400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:34.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e227fc700 1 -- 172.21.15.32:0/907562387 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e240093a0 con 0x7f7e3017a560 2021-04-09T18:12:34.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e227fc700 1 -- 172.21.15.32:0/907562387 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2400bd50 con 0x7f7e3017a560 2021-04-09T18:12:34.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e227fc700 1 -- 172.21.15.32:0/907562387 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e240052e0 con 0x7f7e3017a560 2021-04-09T18:12:34.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e38064700 1 -- 172.21.15.32:0/907562387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e30180be0 con 0x7f7e3017a560 2021-04-09T18:12:34.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.410 7f7e38064700 1 -- 172.21.15.32:0/907562387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e30180be0 con 0x7f7e3017a560 2021-04-09T18:12:34.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.411 7f7e227fc700 1 -- 172.21.15.32:0/907562387 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7e24006030 con 0x7f7e3017a560 2021-04-09T18:12:34.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.411 7f7e227fc700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e180188d0 0x7f7e1801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.411 7f7e227fc700 1 -- 172.21.15.32:0/907562387 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7e240329e0 con 0x7f7e3017a560 2021-04-09T18:12:34.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.411 7f7e35e80700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e180188d0 0x7f7e1801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.412 7f7e35e80700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e180188d0 0x7f7e1801add0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:34.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.431 7f7e38064700 1 -- 172.21.15.32:0/907562387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e301b7070 con 0x7f7e3017a560 2021-04-09T18:12:34.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.434 7f7e227fc700 1 -- 172.21.15.32:0/907562387 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7e240009c0 con 0x7f7e3017a560 2021-04-09T18:12:34.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.622 7f7e38064700 1 -- 172.21.15.32:0/907562387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e3010d480 con 0x7f7e3017a560 2021-04-09T18:12:34.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.623 7f7e227fc700 1 -- 172.21.15.32:0/907562387 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f7e24006030 con 0x7f7e3017a560 2021-04-09T18:12:34.712 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:34.713 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":19,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:33.771991","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:12:34.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 -- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e180188d0 msgr2=0x7f7e1801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e180188d0 0x7f7e1801add0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 -- 172.21.15.32:0/907562387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3017a560 msgr2=0x7f7e301b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3017a560 0x7f7e301b7f80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f7e24009650 tx=0x7f7e2400b960).stop 2021-04-09T18:12:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 -- 172.21.15.32:0/907562387 shutdown_connections 2021-04-09T18:12:34.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e180188d0 0x7f7e1801add0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e3017a560 0x7f7e301b7f80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e301b85a0 0x7f7e301b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 --2- 172.21.15.32:0/907562387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e30099920 0x7f7e30179f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 -- 172.21.15.32:0/907562387 >> 172.21.15.32:0/907562387 conn(0x7f7e301aa2b0 msgr2=0x7f7e301a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:34.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 -- 172.21.15.32:0/907562387 shutdown_connections 2021-04-09T18:12:34.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.625 7f7e38064700 1 -- 172.21.15.32:0/907562387 wait complete. 2021-04-09T18:12:34.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:12:34.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.923 7fef1b3be700 1 Processor -- start 2021-04-09T18:12:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef1b3be700 1 -- start start 2021-04-09T18:12:34.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef1b3be700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fef1410d010 0x7fef140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef1b3be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140d8490 0x7fef1410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:34.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef1b3be700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1410d480 con 0x7fef140d1da0 2021-04-09T18:12:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef1b3be700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef140d8d30 con 0x7fef1410d010 2021-04-09T18:12:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef1b3be700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef140d9220 con 0x7fef140d8490 2021-04-09T18:12:34.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef191da700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fef1410d010 0x7fef140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef199db700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140d8490 0x7fef1410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:34.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef189d9700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fef140d1da0 0x7fef140d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:49010/0 (socket says 172.21.15.32:49010) 2021-04-09T18:12:34.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.924 7fef189d9700 1 -- 172.21.15.32:0/3003532232 learned_addr learned my addr 172.21.15.32:0/3003532232 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef191da700 1 -- 172.21.15.32:0/3003532232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140d8490 msgr2=0x7fef1410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef191da700 1 --2- 172.21.15.32:0/3003532232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140d8490 0x7fef1410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef191da700 1 -- 172.21.15.32:0/3003532232 >> v1:172.21.15.86:6789/0 conn(0x7fef140d1da0 legacy=0x7fef140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef191da700 1 -- 172.21.15.32:0/3003532232 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1410e020 con 0x7fef1410d010 2021-04-09T18:12:34.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef07fff700 1 -- 172.21.15.32:0/3003532232 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836192056 0 0) 0x7fef080010c0 con 0x7fef140d1da0 2021-04-09T18:12:34.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef199db700 1 --2- 172.21.15.32:0/3003532232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140d8490 0x7fef1410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:34.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef191da700 1 --2- 172.21.15.32:0/3003532232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef1410d010 0x7fef140d1780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fef10006650 tx=0x7fef10009a80).ready entity=mon.1 client_cookie=35a828fc8f95c8d3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:34.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef07fff700 1 -- 172.21.15.32:0/3003532232 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef100009b0 con 0x7fef1410d010 2021-04-09T18:12:34.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef07fff700 1 -- 172.21.15.32:0/3003532232 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1000c0c0 con 0x7fef1410d010 2021-04-09T18:12:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef07fff700 1 -- 172.21.15.32:0/3003532232 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef1000c510 con 0x7fef1410d010 2021-04-09T18:12:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef1b3be700 1 -- 172.21.15.32:0/3003532232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef1410d010 msgr2=0x7fef140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:34.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.925 7fef1b3be700 1 --2- 172.21.15.32:0/3003532232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef1410d010 0x7fef140d1780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fef10006650 tx=0x7fef10009a80).stop 2021-04-09T18:12:34.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.926 7fef1b3be700 1 -- 172.21.15.32:0/3003532232 shutdown_connections 2021-04-09T18:12:34.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.926 7fef1b3be700 1 --2- 172.21.15.32:0/3003532232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140d8490 0x7fef1410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.926 7fef1b3be700 1 --2- 172.21.15.32:0/3003532232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef1410d010 0x7fef140d1780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:34.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.926 7fef1b3be700 1 -- 172.21.15.32:0/3003532232 >> 172.21.15.32:0/3003532232 conn(0x7fef140ce5f0 msgr2=0x7fef140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:34.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.926 7fef1b3be700 1 -- 172.21.15.32:0/3003532232 shutdown_connections 2021-04-09T18:12:34.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.926 7fef1b3be700 1 -- 172.21.15.32:0/3003532232 wait complete. 2021-04-09T18:12:34.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.946 7fef1b3be700 1 Processor -- start 2021-04-09T18:12:35.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.946 7fef1b3be700 1 -- start start 2021-04-09T18:12:35.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef1b3be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140a0110 0x7fef1418c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:35.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef1b3be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef140a0510 0x7fef140c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:35.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef1b3be700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef140c36e0 0x7fef140c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:35.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef1b3be700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1418cf60 con 0x7fef140a0510 2021-04-09T18:12:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef1b3be700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef14199ea0 con 0x7fef140c36e0 2021-04-09T18:12:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef1b3be700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef14187d70 con 0x7fef140a0110 2021-04-09T18:12:35.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef189d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef140a0510 0x7fef140c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:35.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef191da700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140a0110 0x7fef1418c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:35.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef189d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef140a0510 0x7fef140c30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41496/0 (socket says 172.21.15.32:41496) 2021-04-09T18:12:35.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef199db700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef140c36e0 0x7fef140c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef189d9700 1 -- 172.21.15.32:0/2947027566 learned_addr learned my addr 172.21.15.32:0/2947027566 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef191da700 1 -- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef140c36e0 msgr2=0x7fef140c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef191da700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef140c36e0 0x7fef140c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef191da700 1 -- 172.21.15.32:0/2947027566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef140a0510 msgr2=0x7fef140c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:35.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef191da700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef140a0510 0x7fef140c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:35.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef191da700 1 -- 172.21.15.32:0/2947027566 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1418d4e0 con 0x7fef140a0110 2021-04-09T18:12:35.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.947 7fef189d9700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef140a0510 0x7fef140c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:35.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.948 7fef191da700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140a0110 0x7fef1418c750 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fef1000bfc0 tx=0x7fef10008890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:35.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.948 7fef05ffb700 1 -- 172.21.15.32:0/2947027566 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef10008c80 con 0x7fef140a0110 2021-04-09T18:12:35.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.948 7fef05ffb700 1 -- 172.21.15.32:0/2947027566 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef10002030 con 0x7fef140a0110 2021-04-09T18:12:35.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.948 7fef05ffb700 1 -- 172.21.15.32:0/2947027566 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef10002510 con 0x7fef140a0110 2021-04-09T18:12:35.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.948 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1418d4e0 con 0x7fef140a0110 2021-04-09T18:12:35.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.948 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1418d4e0 con 0x7fef140a0110 2021-04-09T18:12:35.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.948 7fef05ffb700 1 -- 172.21.15.32:0/2947027566 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fef10006030 con 0x7fef140a0110 2021-04-09T18:12:35.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.948 7fef05ffb700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feef40188c0 0x7feef401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:35.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.948 7fef05ffb700 1 -- 172.21.15.32:0/2947027566 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fef100329b0 con 0x7fef140a0110 2021-04-09T18:12:35.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.949 7fef189d9700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feef40188c0 0x7feef401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:35.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.949 7fef189d9700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feef40188c0 0x7feef401adc0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.968 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1410d480 con 0x7fef140a0110 2021-04-09T18:12:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:34.972 7fef05ffb700 1 -- 172.21.15.32:0/2947027566 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fef10002030 con 0x7fef140a0110 2021-04-09T18:12:35.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.166 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fef1410d480 con 0x7fef140a0110 2021-04-09T18:12:35.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.784 7fef05ffb700 1 -- 172.21.15.32:0/2947027566 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7fef10006030 con 0x7fef140a0110 2021-04-09T18:12:35.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feef40188c0 msgr2=0x7feef401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:35.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feef40188c0 0x7feef401adc0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:35.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140a0110 msgr2=0x7fef1418c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:35.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140a0110 0x7fef1418c750 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fef1000bfc0 tx=0x7fef10008890).stop 2021-04-09T18:12:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 shutdown_connections 2021-04-09T18:12:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feef40188c0 0x7feef401adc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:35.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef140a0510 0x7fef140c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:35.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef140c36e0 0x7fef140c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 --2- 172.21.15.32:0/2947027566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef140a0110 0x7fef1418c750 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:35.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 >> 172.21.15.32:0/2947027566 conn(0x7fef140b93a0 msgr2=0x7fef140b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:35.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 shutdown_connections 2021-04-09T18:12:35.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:35.833 7fef1b3be700 1 -- 172.21.15.32:0/2947027566 wait complete. 2021-04-09T18:12:35.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:12:36.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.018 7f8225888700 1 Processor -- start 2021-04-09T18:12:36.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.018 7f8225888700 1 -- start start 2021-04-09T18:12:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.018 7f8225888700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f822010d010 0x7f82200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f8225888700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82200d8490 0x7f822010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f8225888700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822010d480 con 0x7f82200d1da0 2021-04-09T18:12:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f8225888700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82200d8d30 con 0x7f822010d010 2021-04-09T18:12:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f8225888700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82200d9220 con 0x7f82200d8490 2021-04-09T18:12:36.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821ed73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f82200d1da0 0x7f82200d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51096/0 (socket says 172.21.15.32:51096) 2021-04-09T18:12:36.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821ed73700 1 -- 172.21.15.32:0/2417807703 learned_addr learned my addr 172.21.15.32:0/2417807703 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:36.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821f574700 1 --2- 172.21.15.32:0/2417807703 >> v2:172.21.15.86:3300/0 conn(0x7f822010d010 0x7f82200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821fd75700 1 --2- 172.21.15.32:0/2417807703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82200d8490 0x7f822010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821e572700 1 -- 172.21.15.32:0/2417807703 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241768666 0 0) 0x7f8214000ce0 con 0x7f82200d1da0 2021-04-09T18:12:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821e572700 1 -- 172.21.15.32:0/2417807703 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8208002fe0 con 0x7f82200d1da0 2021-04-09T18:12:36.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821e572700 1 -- 172.21.15.32:0/2417807703 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1637036385 0 0) 0x7f8214000ed0 con 0x7f82200d1da0 2021-04-09T18:12:36.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821f574700 1 -- 172.21.15.32:0/2417807703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82200d8490 msgr2=0x7f822010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:36.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821f574700 1 --2- 172.21.15.32:0/2417807703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82200d8490 0x7f822010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821f574700 1 -- 172.21.15.32:0/2417807703 >> v1:172.21.15.32:6789/0 conn(0x7f82200d1da0 legacy=0x7f82200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821f574700 1 -- 172.21.15.32:0/2417807703 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822010e020 con 0x7f822010d010 2021-04-09T18:12:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821f574700 1 --2- 172.21.15.32:0/2417807703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822010d010 0x7f82200d1780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f8210006650 tx=0x7f8210009a80).ready entity=mon.1 client_cookie=433407580ccd72f8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.019 7f821e572700 1 -- 172.21.15.32:0/2417807703 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82100009b0 con 0x7f822010d010 2021-04-09T18:12:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f821e572700 1 -- 172.21.15.32:0/2417807703 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821000c0c0 con 0x7f822010d010 2021-04-09T18:12:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f821e572700 1 -- 172.21.15.32:0/2417807703 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f821000c510 con 0x7f822010d010 2021-04-09T18:12:36.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f8225888700 1 -- 172.21.15.32:0/2417807703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822010d010 msgr2=0x7f82200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f8225888700 1 --2- 172.21.15.32:0/2417807703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822010d010 0x7f82200d1780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f8210006650 tx=0x7f8210009a80).stop 2021-04-09T18:12:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f8225888700 1 -- 172.21.15.32:0/2417807703 shutdown_connections 2021-04-09T18:12:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f8225888700 1 --2- 172.21.15.32:0/2417807703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82200d8490 0x7f822010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f8225888700 1 --2- 172.21.15.32:0/2417807703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822010d010 0x7f82200d1780 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f8225888700 1 -- 172.21.15.32:0/2417807703 >> 172.21.15.32:0/2417807703 conn(0x7f82200ce5f0 msgr2=0x7f82200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:36.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f8225888700 1 -- 172.21.15.32:0/2417807703 shutdown_connections 2021-04-09T18:12:36.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.020 7f8225888700 1 -- 172.21.15.32:0/2417807703 wait complete. 2021-04-09T18:12:36.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.040 7f8225888700 1 Processor -- start 2021-04-09T18:12:36.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f8225888700 1 -- start start 2021-04-09T18:12:36.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f8225888700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82201aca20 0x7f82201ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f8225888700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822018bd40 0x7f822018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f8225888700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822008fba0 0x7f822008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f8225888700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822018c7e0 con 0x7f82201aca20 2021-04-09T18:12:36.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f8225888700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822018ca30 con 0x7f822018bd40 2021-04-09T18:12:36.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f8225888700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82200905f0 con 0x7f822008fba0 2021-04-09T18:12:36.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f821f574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82201aca20 0x7f82201ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f821f574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82201aca20 0x7f82201ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41508/0 (socket says 172.21.15.32:41508) 2021-04-09T18:12:36.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.041 7f821f574700 1 -- 172.21.15.32:0/1981849998 learned_addr learned my addr 172.21.15.32:0/1981849998 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f821fd75700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822008fba0 0x7f822008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f821ed73700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822018bd40 0x7f822018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f821f574700 1 -- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822008fba0 msgr2=0x7f822008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f821f574700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822008fba0 0x7f822008ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f821f574700 1 -- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822018bd40 msgr2=0x7f822018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f821f574700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822018bd40 0x7f822018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f821f574700 1 -- 172.21.15.32:0/1981849998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82200b4300 con 0x7f82201aca20 2021-04-09T18:12:36.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f821ed73700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822018bd40 0x7f822018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:12:36.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f821f574700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82201aca20 0x7f82201ace20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f8210009630 tx=0x7f821000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:36.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f8207fff700 1 -- 172.21.15.32:0/1981849998 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f821000bd70 con 0x7f82201aca20 2021-04-09T18:12:36.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f8207fff700 1 -- 172.21.15.32:0/1981849998 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8210000960 con 0x7f82201aca20 2021-04-09T18:12:36.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f8225888700 1 -- 172.21.15.32:0/1981849998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82200b4300 con 0x7f82201aca20 2021-04-09T18:12:36.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f8207fff700 1 -- 172.21.15.32:0/1981849998 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82100054d0 con 0x7f82201aca20 2021-04-09T18:12:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.042 7f8225888700 1 -- 172.21.15.32:0/1981849998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82200b4300 con 0x7f82201aca20 2021-04-09T18:12:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.043 7f8207fff700 1 -- 172.21.15.32:0/1981849998 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f821000a000 con 0x7f82201aca20 2021-04-09T18:12:36.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.043 7f8207fff700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8200018930 0x7f820001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.043 7f8207fff700 1 -- 172.21.15.32:0/1981849998 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f82100329c0 con 0x7f82201aca20 2021-04-09T18:12:36.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.043 7f821ed73700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8200018930 0x7f820001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.043 7f821ed73700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8200018930 0x7f820001ae10 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:36.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.063 7f8225888700 1 -- 172.21.15.32:0/1981849998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8220182d70 con 0x7f82201aca20 2021-04-09T18:12:36.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.066 7f8207fff700 1 -- 172.21.15.32:0/1981849998 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f821001f700 con 0x7f82201aca20 2021-04-09T18:12:36.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.258 7f8225888700 1 -- 172.21.15.32:0/1981849998 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f822018c760 con 0x7f82201aca20 2021-04-09T18:12:36.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.258 7f8207fff700 1 -- 172.21.15.32:0/1981849998 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1415 (secure 0 0 0) 0x7f8210000940 con 0x7f82201aca20 2021-04-09T18:12:36.355 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 3 2021-04-09T18:12:36.367 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:36.368 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:12:33.772139","modified":"2021-04-09 18:12:35.782368","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-04-09T18:12:36.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 -- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8200018930 msgr2=0x7f820001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8200018930 0x7f820001ae10 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 -- 172.21.15.32:0/1981849998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82201aca20 msgr2=0x7f82201ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82201aca20 0x7f82201ace20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f8210009630 tx=0x7f821000b980).stop 2021-04-09T18:12:36.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 -- 172.21.15.32:0/1981849998 shutdown_connections 2021-04-09T18:12:36.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8200018930 0x7f820001ae10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822008fba0 0x7f822008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f822018bd40 0x7f822018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 --2- 172.21.15.32:0/1981849998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82201aca20 0x7f82201ace20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 -- 172.21.15.32:0/1981849998 >> 172.21.15.32:0/1981849998 conn(0x7f822019b460 msgr2=0x7f822019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:36.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 -- 172.21.15.32:0/1981849998 shutdown_connections 2021-04-09T18:12:36.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.261 7f8225888700 1 -- 172.21.15.32:0/1981849998 wait complete. 2021-04-09T18:12:36.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:12:36.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.575 7fb0bd117700 1 Processor -- start 2021-04-09T18:12:36.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.575 7fb0bd117700 1 -- start start 2021-04-09T18:12:36.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0bd117700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b80d7ca0 0x7fb0b80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0bd117700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b810d480 con 0x7fb0b810d010 2021-04-09T18:12:36.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0bd117700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b80d86c0 con 0x7fb0b810da00 2021-04-09T18:12:36.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0bd117700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b80d9220 con 0x7fb0b80d7ca0 2021-04-09T18:12:36.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b6d9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb0b810d010 0x7fb0b80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51106/0 (socket says 172.21.15.32:51106) 2021-04-09T18:12:36.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b6d9d700 1 -- 172.21.15.32:0/2750237361 learned_addr learned my addr 172.21.15.32:0/2750237361 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:36.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b659c700 1 --2- 172.21.15.32:0/2750237361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b80d7ca0 0x7fb0b80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b5d9b700 1 -- 172.21.15.32:0/2750237361 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514195102 0 0) 0x7fb0a8000ce0 con 0x7fb0b810d010 2021-04-09T18:12:36.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b5d9b700 1 -- 172.21.15.32:0/2750237361 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a0002fe0 con 0x7fb0b810d010 2021-04-09T18:12:36.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b5d9b700 1 -- 172.21.15.32:0/2750237361 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322630893 0 0) 0x7fb0b0000d30 con 0x7fb0b810da00 2021-04-09T18:12:36.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b5d9b700 1 -- 172.21.15.32:0/2750237361 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a0003210 con 0x7fb0b810da00 2021-04-09T18:12:36.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b659c700 1 -- 172.21.15.32:0/2750237361 >> v1:172.21.15.86:6789/0 conn(0x7fb0b810da00 legacy=0x7fb0b810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b659c700 1 -- 172.21.15.32:0/2750237361 >> v1:172.21.15.32:6789/0 conn(0x7fb0b810d010 legacy=0x7fb0b80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.576 7fb0b659c700 1 -- 172.21.15.32:0/2750237361 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b80d1f70 con 0x7fb0b80d7ca0 2021-04-09T18:12:36.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.577 7fb0b659c700 1 --2- 172.21.15.32:0/2750237361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b80d7ca0 0x7fb0b80d80a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb0ac006650 tx=0x7fb0ac009a00).ready entity=mon.2 client_cookie=48a0714098560f3d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:36.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.577 7fb0b5d9b700 1 -- 172.21.15.32:0/2750237361 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0ac009df0 con 0x7fb0b80d7ca0 2021-04-09T18:12:36.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.577 7fb0b5d9b700 1 -- 172.21.15.32:0/2750237361 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac0009b0 con 0x7fb0b80d7ca0 2021-04-09T18:12:36.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.577 7fb0b5d9b700 1 -- 172.21.15.32:0/2750237361 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0ac00c2b0 con 0x7fb0b80d7ca0 2021-04-09T18:12:36.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.577 7fb0bd117700 1 -- 172.21.15.32:0/2750237361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b80d7ca0 msgr2=0x7fb0b80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.577 7fb0bd117700 1 --2- 172.21.15.32:0/2750237361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b80d7ca0 0x7fb0b80d80a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fb0ac006650 tx=0x7fb0ac009a00).stop 2021-04-09T18:12:36.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.578 7fb0bd117700 1 -- 172.21.15.32:0/2750237361 shutdown_connections 2021-04-09T18:12:36.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.578 7fb0bd117700 1 --2- 172.21.15.32:0/2750237361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b80d7ca0 0x7fb0b80d80a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.578 7fb0bd117700 1 -- 172.21.15.32:0/2750237361 >> 172.21.15.32:0/2750237361 conn(0x7fb0b80ce5f0 msgr2=0x7fb0b80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:36.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.578 7fb0bd117700 1 -- 172.21.15.32:0/2750237361 shutdown_connections 2021-04-09T18:12:36.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.578 7fb0bd117700 1 -- 172.21.15.32:0/2750237361 wait complete. 2021-04-09T18:12:36.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0bd117700 1 Processor -- start 2021-04-09T18:12:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0bd117700 1 -- start start 2021-04-09T18:12:36.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0bd117700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8185600 0x7fb0b8193e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0bd117700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0b8185a00 0x7fb0b808b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0bd117700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b808bf60 0x7fb0b808a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0bd117700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b808c360 con 0x7fb0b8185600 2021-04-09T18:12:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0bd117700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b8194460 con 0x7fb0b8185a00 2021-04-09T18:12:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0bd117700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b8194680 con 0x7fb0b808bf60 2021-04-09T18:12:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0b6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8185600 0x7fb0b8193e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0b6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8185600 0x7fb0b8193e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41522/0 (socket says 172.21.15.32:41522) 2021-04-09T18:12:36.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.598 7fb0b6d9d700 1 -- 172.21.15.32:0/2108798137 learned_addr learned my addr 172.21.15.32:0/2108798137 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:36.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b659c700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0b8185a00 0x7fb0b808b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b759e700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b808bf60 0x7fb0b808a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b6d9d700 1 -- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b808bf60 msgr2=0x7fb0b808a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b6d9d700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b808bf60 0x7fb0b808a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b6d9d700 1 -- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0b8185a00 msgr2=0x7fb0b808b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b6d9d700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0b8185a00 0x7fb0b808b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b6d9d700 1 -- 172.21.15.32:0/2108798137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b808ac70 con 0x7fb0b8185600 2021-04-09T18:12:36.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b759e700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b808bf60 0x7fb0b808a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:12:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b659c700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0b8185a00 0x7fb0b808b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:12:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0b6d9d700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8185600 0x7fb0b8193e40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb0a8001490 tx=0x7fb0a800a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0a77fe700 1 -- 172.21.15.32:0/2108798137 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0a800ad40 con 0x7fb0b8185600 2021-04-09T18:12:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0a77fe700 1 -- 172.21.15.32:0/2108798137 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a8000990 con 0x7fb0b8185600 2021-04-09T18:12:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b808ac70 con 0x7fb0b8185600 2021-04-09T18:12:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0a77fe700 1 -- 172.21.15.32:0/2108798137 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0a8004590 con 0x7fb0b8185600 2021-04-09T18:12:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.599 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b808ac70 con 0x7fb0b8185600 2021-04-09T18:12:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.600 7fb0a77fe700 1 -- 172.21.15.32:0/2108798137 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb0a80043a0 con 0x7fb0b8185600 2021-04-09T18:12:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.600 7fb0a77fe700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb09c018930 0x7fb09c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.600 7fb0a77fe700 1 -- 172.21.15.32:0/2108798137 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb0a80329a0 con 0x7fb0b8185600 2021-04-09T18:12:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.600 7fb0b659c700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb09c018930 0x7fb09c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.600 7fb0b659c700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb09c018930 0x7fb09c01ae10 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.620 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b81b7080 con 0x7fb0b8185600 2021-04-09T18:12:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.623 7fb0a77fe700 1 -- 172.21.15.32:0/2108798137 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb0a8000990 con 0x7fb0b8185600 2021-04-09T18:12:36.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.812 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0b810d480 con 0x7fb0b8185600 2021-04-09T18:12:36.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.813 7fb0a77fe700 1 -- 172.21.15.32:0/2108798137 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fb0a8000990 con 0x7fb0b8185600 2021-04-09T18:12:36.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:36.834 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:34.773743","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:12:36.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb09c018930 msgr2=0x7fb09c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb09c018930 0x7fb09c01ae10 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8185600 msgr2=0x7fb0b8193e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:36.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8185600 0x7fb0b8193e40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb0a8001490 tx=0x7fb0a800a950).stop 2021-04-09T18:12:36.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 shutdown_connections 2021-04-09T18:12:36.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb09c018930 0x7fb09c01ae10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0b808bf60 0x7fb0b808a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0b8185a00 0x7fb0b808b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 --2- 172.21.15.32:0/2108798137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8185600 0x7fb0b8193e40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:36.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 >> 172.21.15.32:0/2108798137 conn(0x7fb0b8190550 msgr2=0x7fb0b81b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:36.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 shutdown_connections 2021-04-09T18:12:36.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:36.815 7fb0bd117700 1 -- 172.21.15.32:0/2108798137 wait complete. 2021-04-09T18:12:36.855 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2021-04-09T18:12:36.856 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:12:36.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:12:36.904 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:12:36.905 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:12:36.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:12:36.955 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:12:36.956 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:12:36.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:12:36.976 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:12:36.977 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:12:36.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:12:37.001 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:12:37.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:12:37.046 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:12:37.103 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:12:37.115 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:12:37.138 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:12:37.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.143 7f387f466700 1 Processor -- start 2021-04-09T18:12:37.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.143 7f387f466700 1 -- start start 2021-04-09T18:12:37.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387f466700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387810d010 0x7f38780d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387f466700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38780d1da0 0x7f38780d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387f466700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f38780d82c0 0x7f387810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387f466700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387810e780 con 0x7f38780d1da0 2021-04-09T18:12:37.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387f466700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38780d8d30 con 0x7f38780d82c0 2021-04-09T18:12:37.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387f466700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38780d9220 con 0x7f387810d010 2021-04-09T18:12:37.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387ca81700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38780d1da0 0x7f38780d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387ca81700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38780d1da0 0x7f38780d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41558/0 (socket says 172.21.15.32:41558) 2021-04-09T18:12:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387ca81700 1 -- 172.21.15.32:0/443663619 learned_addr learned my addr 172.21.15.32:0/443663619 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387d282700 1 --2- 172.21.15.32:0/443663619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387810d010 0x7f38780d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.144 7f387da83700 1 --2- 172.21.15.32:0/443663619 >> v2:172.21.15.86:3300/0 conn(0x7f38780d82c0 0x7f387810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f387ca81700 1 -- 172.21.15.32:0/443663619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387810d010 msgr2=0x7f38780d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:37.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f387ca81700 1 --2- 172.21.15.32:0/443663619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387810d010 0x7f38780d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f387ca81700 1 -- 172.21.15.32:0/443663619 >> v2:172.21.15.86:3300/0 conn(0x7f38780d82c0 msgr2=0x7f387810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:37.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f387ca81700 1 --2- 172.21.15.32:0/443663619 >> v2:172.21.15.86:3300/0 conn(0x7f38780d82c0 0x7f387810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f387ca81700 1 -- 172.21.15.32:0/443663619 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387810d480 con 0x7f38780d1da0 2021-04-09T18:12:37.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f387ca81700 1 --2- 172.21.15.32:0/443663619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38780d1da0 0x7f38780d7ca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f3868006700 tx=0x7f3868009b80).ready entity=mon.0 client_cookie=782c8338040559b3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:37.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f386ffff700 1 -- 172.21.15.32:0/443663619 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f386800c0c0 con 0x7f38780d1da0 2021-04-09T18:12:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f386ffff700 1 -- 172.21.15.32:0/443663619 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386800c440 con 0x7f38780d1da0 2021-04-09T18:12:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f386ffff700 1 -- 172.21.15.32:0/443663619 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f386800c920 con 0x7f38780d1da0 2021-04-09T18:12:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f387f466700 1 -- 172.21.15.32:0/443663619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38780d1da0 msgr2=0x7f38780d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.145 7f387f466700 1 --2- 172.21.15.32:0/443663619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38780d1da0 0x7f38780d7ca0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f3868006700 tx=0x7f3868009b80).stop 2021-04-09T18:12:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.146 7f387f466700 1 -- 172.21.15.32:0/443663619 shutdown_connections 2021-04-09T18:12:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.146 7f387f466700 1 --2- 172.21.15.32:0/443663619 >> v2:172.21.15.86:3300/0 conn(0x7f38780d82c0 0x7f387810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.146 7f387f466700 1 --2- 172.21.15.32:0/443663619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38780d1da0 0x7f38780d7ca0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.146 7f387f466700 1 --2- 172.21.15.32:0/443663619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387810d010 0x7f38780d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.146 7f387f466700 1 -- 172.21.15.32:0/443663619 >> 172.21.15.32:0/443663619 conn(0x7f38780ce5f0 msgr2=0x7f38780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:37.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.146 7f387f466700 1 -- 172.21.15.32:0/443663619 shutdown_connections 2021-04-09T18:12:37.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.146 7f387f466700 1 -- 172.21.15.32:0/443663619 wait complete. 2021-04-09T18:12:37.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.166 7f387f466700 1 Processor -- start 2021-04-09T18:12:37.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.166 7f387f466700 1 -- start start 2021-04-09T18:12:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387f466700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38781aca20 0x7f38781ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387f466700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818bd40 0x7f387818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387f466700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387808fba0 0x7f387808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387f466700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387818c7e0 con 0x7f38781aca20 2021-04-09T18:12:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387f466700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387818ca30 con 0x7f387818bd40 2021-04-09T18:12:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387f466700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38780905f0 con 0x7f387808fba0 2021-04-09T18:12:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387d282700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38781aca20 0x7f38781ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387d282700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38781aca20 0x7f38781ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41560/0 (socket says 172.21.15.32:41560) 2021-04-09T18:12:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387d282700 1 -- 172.21.15.32:0/530594970 learned_addr learned my addr 172.21.15.32:0/530594970 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387da83700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387808fba0 0x7f387808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387ca81700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818bd40 0x7f387818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387d282700 1 -- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387808fba0 msgr2=0x7f387808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387d282700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387808fba0 0x7f387808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387d282700 1 -- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818bd40 msgr2=0x7f387818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387d282700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818bd40 0x7f387818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387d282700 1 -- 172.21.15.32:0/530594970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38780b4300 con 0x7f38781aca20 2021-04-09T18:12:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387d282700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38781aca20 0x7f38781ace20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f387400b560 tx=0x7f387400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f386dffb700 1 -- 172.21.15.32:0/530594970 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f387400bcf0 con 0x7f38781aca20 2021-04-09T18:12:37.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f386dffb700 1 -- 172.21.15.32:0/530594970 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3874000c90 con 0x7f38781aca20 2021-04-09T18:12:37.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f386dffb700 1 -- 172.21.15.32:0/530594970 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3874002220 con 0x7f38781aca20 2021-04-09T18:12:37.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387f466700 1 -- 172.21.15.32:0/530594970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38780b4300 con 0x7f38781aca20 2021-04-09T18:12:37.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.167 7f387f466700 1 -- 172.21.15.32:0/530594970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38780b4300 con 0x7f38781aca20 2021-04-09T18:12:37.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.168 7f386dffb700 1 -- 172.21.15.32:0/530594970 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f38740052e0 con 0x7f38781aca20 2021-04-09T18:12:37.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.168 7f386dffb700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38580188c0 0x7f385801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.168 7f386dffb700 1 -- 172.21.15.32:0/530594970 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3874032800 con 0x7f38781aca20 2021-04-09T18:12:37.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.168 7f387ca81700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38580188c0 0x7f385801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.169 7f387ca81700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38580188c0 0x7f385801adc0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:37.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.188 7f387f466700 1 -- 172.21.15.32:0/530594970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3878182d70 con 0x7f38781aca20 2021-04-09T18:12:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.192 7f386dffb700 1 -- 172.21.15.32:0/530594970 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3874006b00 con 0x7f38781aca20 2021-04-09T18:12:37.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.380 7f387f466700 1 -- 172.21.15.32:0/530594970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f387818c760 con 0x7f38781aca20 2021-04-09T18:12:37.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.380 7f386dffb700 1 -- 172.21.15.32:0/530594970 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f38740052e0 con 0x7f38781aca20 2021-04-09T18:12:37.430 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:37.430 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:34.773743","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:12:37.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.382 7f387f466700 1 -- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38580188c0 msgr2=0x7f385801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.382 7f387f466700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38580188c0 0x7f385801adc0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.382 7f387f466700 1 -- 172.21.15.32:0/530594970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38781aca20 msgr2=0x7f38781ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.382 7f387f466700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38781aca20 0x7f38781ace20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f387400b560 tx=0x7f387400b900).stop 2021-04-09T18:12:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.383 7f387f466700 1 -- 172.21.15.32:0/530594970 shutdown_connections 2021-04-09T18:12:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.383 7f387f466700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38580188c0 0x7f385801adc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.383 7f387f466700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f387808fba0 0x7f387808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.383 7f387f466700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818bd40 0x7f387818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.383 7f387f466700 1 --2- 172.21.15.32:0/530594970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38781aca20 0x7f38781ace20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.383 7f387f466700 1 -- 172.21.15.32:0/530594970 >> 172.21.15.32:0/530594970 conn(0x7f387819b460 msgr2=0x7f387819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:37.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.383 7f387f466700 1 -- 172.21.15.32:0/530594970 shutdown_connections 2021-04-09T18:12:37.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.383 7f387f466700 1 -- 172.21.15.32:0/530594970 wait complete. 2021-04-09T18:12:37.457 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:12:32.136283', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:12:32.831693', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:12:37.458 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:12:37.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.613 7f427659a700 1 Processor -- start 2021-04-09T18:12:37.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.613 7f427659a700 1 -- start start 2021-04-09T18:12:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f427659a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f427010d010 0x7f42700d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f427659a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f42700d8490 0x7f427010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f427659a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427010d480 con 0x7f42700d1da0 2021-04-09T18:12:37.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f427659a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42700d8d30 con 0x7f42700d8490 2021-04-09T18:12:37.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f427659a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42700d9220 con 0x7f427010d010 2021-04-09T18:12:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f4274bb7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f42700d8490 0x7f427010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f4267fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f42700d1da0 0x7f42700d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51158/0 (socket says 172.21.15.32:51158) 2021-04-09T18:12:37.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f4267fff700 1 -- 172.21.15.32:0/1023389486 learned_addr learned my addr 172.21.15.32:0/1023389486 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f426ffff700 1 --2- 172.21.15.32:0/1023389486 >> v2:172.21.15.86:3300/0 conn(0x7f427010d010 0x7f42700d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f426f7fe700 1 -- 172.21.15.32:0/1023389486 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227125317 0 0) 0x7f4258000ce0 con 0x7f42700d1da0 2021-04-09T18:12:37.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f426f7fe700 1 -- 172.21.15.32:0/1023389486 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f425c002fe0 con 0x7f42700d1da0 2021-04-09T18:12:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f4274bb7700 1 -- 172.21.15.32:0/1023389486 >> v2:172.21.15.86:3300/0 conn(0x7f427010d010 msgr2=0x7f42700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:37.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.614 7f4274bb7700 1 --2- 172.21.15.32:0/1023389486 >> v2:172.21.15.86:3300/0 conn(0x7f427010d010 0x7f42700d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.615 7f4274bb7700 1 -- 172.21.15.32:0/1023389486 >> v1:172.21.15.32:6789/0 conn(0x7f42700d1da0 legacy=0x7f42700d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.615 7f4274bb7700 1 -- 172.21.15.32:0/1023389486 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427010e020 con 0x7f42700d8490 2021-04-09T18:12:37.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.615 7f4274bb7700 1 --2- 172.21.15.32:0/1023389486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42700d8490 0x7f427010da00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4268006e30 tx=0x7f4268009e60).ready entity=mon.0 client_cookie=5d0cf3c7067eb291 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:37.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.615 7f426f7fe700 1 -- 172.21.15.32:0/1023389486 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4268006a50 con 0x7f42700d8490 2021-04-09T18:12:37.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.615 7f426f7fe700 1 -- 172.21.15.32:0/1023389486 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426800b300 con 0x7f42700d8490 2021-04-09T18:12:37.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.615 7f426f7fe700 1 -- 172.21.15.32:0/1023389486 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f426800b750 con 0x7f42700d8490 2021-04-09T18:12:37.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.615 7f427659a700 1 -- 172.21.15.32:0/1023389486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42700d8490 msgr2=0x7f427010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.615 7f427659a700 1 --2- 172.21.15.32:0/1023389486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42700d8490 0x7f427010da00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4268006e30 tx=0x7f4268009e60).stop 2021-04-09T18:12:37.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.616 7f427659a700 1 -- 172.21.15.32:0/1023389486 shutdown_connections 2021-04-09T18:12:37.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.616 7f427659a700 1 --2- 172.21.15.32:0/1023389486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42700d8490 0x7f427010da00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.616 7f427659a700 1 --2- 172.21.15.32:0/1023389486 >> v2:172.21.15.86:3300/0 conn(0x7f427010d010 0x7f42700d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.616 7f427659a700 1 -- 172.21.15.32:0/1023389486 >> 172.21.15.32:0/1023389486 conn(0x7f42700ce5f0 msgr2=0x7f42700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:37.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.616 7f427659a700 1 -- 172.21.15.32:0/1023389486 shutdown_connections 2021-04-09T18:12:37.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.616 7f427659a700 1 -- 172.21.15.32:0/1023389486 wait complete. 2021-04-09T18:12:37.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.636 7f427659a700 1 Processor -- start 2021-04-09T18:12:37.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.636 7f427659a700 1 -- start start 2021-04-09T18:12:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.636 7f427659a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42700c3c00 0x7f427009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.636 7f427659a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4270095240 0x7f42701a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.636 7f427659a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42701a1270 0x7f42700b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.636 7f427659a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427009cda0 con 0x7f4270095240 2021-04-09T18:12:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.636 7f427659a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42700c3680 con 0x7f42700c3c00 2021-04-09T18:12:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.636 7f427659a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4270095640 con 0x7f42701a1270 2021-04-09T18:12:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.636 7f4267fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4270095240 0x7f42701a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f4267fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4270095240 0x7f42701a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41572/0 (socket says 172.21.15.32:41572) 2021-04-09T18:12:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f4267fff700 1 -- 172.21.15.32:0/4221866127 learned_addr learned my addr 172.21.15.32:0/4221866127 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f4274bb7700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42701a1270 0x7f42700b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f426ffff700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42700c3c00 0x7f427009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f4267fff700 1 -- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42701a1270 msgr2=0x7f42700b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f4267fff700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42701a1270 0x7f42700b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f4267fff700 1 -- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42700c3c00 msgr2=0x7f427009c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f4267fff700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42700c3c00 0x7f427009c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f4267fff700 1 -- 172.21.15.32:0/4221866127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42701a1670 con 0x7f4270095240 2021-04-09T18:12:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f4267fff700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4270095240 0x7f42701a0c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4258006f00 tx=0x7f425800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f426d7fa700 1 -- 172.21.15.32:0/4221866127 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f425800bd30 con 0x7f4270095240 2021-04-09T18:12:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f426d7fa700 1 -- 172.21.15.32:0/4221866127 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4258006030 con 0x7f4270095240 2021-04-09T18:12:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f426d7fa700 1 -- 172.21.15.32:0/4221866127 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4258004590 con 0x7f4270095240 2021-04-09T18:12:37.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f427659a700 1 -- 172.21.15.32:0/4221866127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42701a1670 con 0x7f4270095240 2021-04-09T18:12:37.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.637 7f427659a700 1 -- 172.21.15.32:0/4221866127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42701a1670 con 0x7f4270095240 2021-04-09T18:12:37.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.638 7f426d7fa700 1 -- 172.21.15.32:0/4221866127 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f42580043a0 con 0x7f4270095240 2021-04-09T18:12:37.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.638 7f426d7fa700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42540188c0 0x7f425401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:37.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.638 7f426d7fa700 1 -- 172.21.15.32:0/4221866127 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4258032790 con 0x7f4270095240 2021-04-09T18:12:37.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.638 7f426ffff700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42540188c0 0x7f425401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:37.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.638 7f426ffff700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42540188c0 0x7f425401adc0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.658 7f427659a700 1 -- 172.21.15.32:0/4221866127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4270093d50 con 0x7f4270095240 2021-04-09T18:12:37.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.661 7f426d7fa700 1 -- 172.21.15.32:0/4221866127 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f42580009b0 con 0x7f4270095240 2021-04-09T18:12:37.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.841 7f427659a700 1 -- 172.21.15.32:0/4221866127 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f427010d480 con 0x7f4270095240 2021-04-09T18:12:37.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.842 7f426d7fa700 1 -- 172.21.15.32:0/4221866127 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f42580043a0 con 0x7f4270095240 2021-04-09T18:12:37.864 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:12:37.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 -- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42540188c0 msgr2=0x7f425401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42540188c0 0x7f425401adc0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 -- 172.21.15.32:0/4221866127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4270095240 msgr2=0x7f42701a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:37.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4270095240 0x7f42701a0c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4258006f00 tx=0x7f425800b940).stop 2021-04-09T18:12:37.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 -- 172.21.15.32:0/4221866127 shutdown_connections 2021-04-09T18:12:37.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42540188c0 0x7f425401adc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4270095240 0x7f42701a0c50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42701a1270 0x7f42700b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 --2- 172.21.15.32:0/4221866127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42700c3c00 0x7f427009c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:37.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 -- 172.21.15.32:0/4221866127 >> 172.21.15.32:0/4221866127 conn(0x7f42700975f0 msgr2=0x7f4270096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:37.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.844 7f427659a700 1 -- 172.21.15.32:0/4221866127 shutdown_connections 2021-04-09T18:12:37.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:37.845 7f427659a700 1 -- 172.21.15.32:0/4221866127 wait complete. 2021-04-09T18:12:37.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:12:38.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa8668a8700 1 Processor -- start 2021-04-09T18:12:38.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa8668a8700 1 -- start start 2021-04-09T18:12:38.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa8668a8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86010d010 0x7fa8600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa8668a8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8600d8490 0x7fa86010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa8668a8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86010d480 con 0x7fa8600d1da0 2021-04-09T18:12:38.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa8668a8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86010e780 con 0x7fa8600d8490 2021-04-09T18:12:38.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa8668a8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600d8d30 con 0x7fa86010d010 2021-04-09T18:12:38.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa864ec5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8600d8490 0x7fa86010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa864ec5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8600d8490 0x7fa86010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41582/0 (socket says 172.21.15.32:41582) 2021-04-09T18:12:38.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa864ec5700 1 -- 172.21.15.32:0/3229198047 learned_addr learned my addr 172.21.15.32:0/3229198047 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.027 7fa85ffff700 1 --2- 172.21.15.32:0/3229198047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86010d010 0x7fa8600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa85f7fe700 1 -- 172.21.15.32:0/3229198047 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393568504 0 0) 0x7fa848000c60 con 0x7fa8600d1da0 2021-04-09T18:12:38.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa85f7fe700 1 -- 172.21.15.32:0/3229198047 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa84c002fe0 con 0x7fa8600d1da0 2021-04-09T18:12:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa864ec5700 1 -- 172.21.15.32:0/3229198047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86010d010 msgr2=0x7fa8600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:38.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa864ec5700 1 --2- 172.21.15.32:0/3229198047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86010d010 0x7fa8600d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa864ec5700 1 -- 172.21.15.32:0/3229198047 >> v1:172.21.15.86:6789/0 conn(0x7fa8600d1da0 legacy=0x7fa8600d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa864ec5700 1 -- 172.21.15.32:0/3229198047 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8600d9220 con 0x7fa8600d8490 2021-04-09T18:12:38.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa85ffff700 1 --2- 172.21.15.32:0/3229198047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86010d010 0x7fa8600d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:38.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa864ec5700 1 --2- 172.21.15.32:0/3229198047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600d8490 0x7fa86010da00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa858006f00 tx=0x7fa858009f10).ready entity=mon.0 client_cookie=2899b192624bf8cb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:38.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa85f7fe700 1 -- 172.21.15.32:0/3229198047 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa85800b3b0 con 0x7fa8600d8490 2021-04-09T18:12:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa85f7fe700 1 -- 172.21.15.32:0/3229198047 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85800b730 con 0x7fa8600d8490 2021-04-09T18:12:38.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa85f7fe700 1 -- 172.21.15.32:0/3229198047 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa85800bc10 con 0x7fa8600d8490 2021-04-09T18:12:38.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa8668a8700 1 -- 172.21.15.32:0/3229198047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600d8490 msgr2=0x7fa86010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.028 7fa8668a8700 1 --2- 172.21.15.32:0/3229198047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600d8490 0x7fa86010da00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa858006f00 tx=0x7fa858009f10).stop 2021-04-09T18:12:38.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.029 7fa8668a8700 1 -- 172.21.15.32:0/3229198047 shutdown_connections 2021-04-09T18:12:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.029 7fa8668a8700 1 --2- 172.21.15.32:0/3229198047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600d8490 0x7fa86010da00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.029 7fa8668a8700 1 --2- 172.21.15.32:0/3229198047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86010d010 0x7fa8600d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.029 7fa8668a8700 1 -- 172.21.15.32:0/3229198047 >> 172.21.15.32:0/3229198047 conn(0x7fa8600ce5f0 msgr2=0x7fa8600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:38.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.029 7fa8668a8700 1 -- 172.21.15.32:0/3229198047 shutdown_connections 2021-04-09T18:12:38.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.029 7fa8668a8700 1 -- 172.21.15.32:0/3229198047 wait complete. 2021-04-09T18:12:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.049 7fa8668a8700 1 Processor -- start 2021-04-09T18:12:38.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.049 7fa8668a8700 1 -- start start 2021-04-09T18:12:38.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa8668a8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8600b1e50 0x7fa8600b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa8668a8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8600b9fd0 0x7fa8600b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa8668a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600b5510 0x7fa8600b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa8668a8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600b2250 con 0x7fa8600b5510 2021-04-09T18:12:38.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa8668a8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600a1220 con 0x7fa8600b9fd0 2021-04-09T18:12:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa8668a8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600b6b30 con 0x7fa8600b1e50 2021-04-09T18:12:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa864ec5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600b5510 0x7fa8600b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa864ec5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600b5510 0x7fa8600b5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41590/0 (socket says 172.21.15.32:41590) 2021-04-09T18:12:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa864ec5700 1 -- 172.21.15.32:0/432449773 learned_addr learned my addr 172.21.15.32:0/432449773 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa85ffff700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8600b1e50 0x7fa8600b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa857fff700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8600b9fd0 0x7fa8600b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa864ec5700 1 -- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8600b1e50 msgr2=0x7fa8600b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa864ec5700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8600b1e50 0x7fa8600b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa864ec5700 1 -- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8600b9fd0 msgr2=0x7fa8600b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa864ec5700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8600b9fd0 0x7fa8600b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa864ec5700 1 -- 172.21.15.32:0/432449773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8600b5f30 con 0x7fa8600b5510 2021-04-09T18:12:38.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa864ec5700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600b5510 0x7fa8600b5910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa85800b8e0 tx=0x7fa85800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa85d7fa700 1 -- 172.21.15.32:0/432449773 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa858003030 con 0x7fa8600b5510 2021-04-09T18:12:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa85d7fa700 1 -- 172.21.15.32:0/432449773 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85800bdc0 con 0x7fa8600b5510 2021-04-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa8668a8700 1 -- 172.21.15.32:0/432449773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8600b5f30 con 0x7fa8600b5510 2021-04-09T18:12:38.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.050 7fa85d7fa700 1 -- 172.21.15.32:0/432449773 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa85800c570 con 0x7fa8600b5510 2021-04-09T18:12:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.051 7fa8668a8700 1 -- 172.21.15.32:0/432449773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8600b5f30 con 0x7fa8600b5510 2021-04-09T18:12:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.051 7fa85d7fa700 1 -- 172.21.15.32:0/432449773 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa858005820 con 0x7fa8600b5510 2021-04-09T18:12:38.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.051 7fa85d7fa700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8440188c0 0x7fa84401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.051 7fa85d7fa700 1 -- 172.21.15.32:0/432449773 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa858032b40 con 0x7fa8600b5510 2021-04-09T18:12:38.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.051 7fa85ffff700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8440188c0 0x7fa84401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.052 7fa85ffff700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8440188c0 0x7fa84401adc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:38.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.071 7fa8668a8700 1 -- 172.21.15.32:0/432449773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8601b7040 con 0x7fa8600b5510 2021-04-09T18:12:38.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.075 7fa85d7fa700 1 -- 172.21.15.32:0/432449773 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa858006c80 con 0x7fa8600b5510 2021-04-09T18:12:38.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.254 7fa8668a8700 1 -- 172.21.15.32:0/432449773 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa8601b7040 con 0x7fa8600b5510 2021-04-09T18:12:38.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.254 7fa85d7fa700 1 -- 172.21.15.32:0/432449773 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa858005820 con 0x7fa8600b5510 2021-04-09T18:12:38.275 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:12:38.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 -- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8440188c0 msgr2=0x7fa84401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8440188c0 0x7fa84401adc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 -- 172.21.15.32:0/432449773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600b5510 msgr2=0x7fa8600b5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600b5510 0x7fa8600b5910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa85800b8e0 tx=0x7fa85800bc80).stop 2021-04-09T18:12:38.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 -- 172.21.15.32:0/432449773 shutdown_connections 2021-04-09T18:12:38.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8440188c0 0x7fa84401adc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600b5510 0x7fa8600b5910 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8600b9fd0 0x7fa8600b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 --2- 172.21.15.32:0/432449773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8600b1e50 0x7fa8600b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 -- 172.21.15.32:0/432449773 >> 172.21.15.32:0/432449773 conn(0x7fa8600b27e0 msgr2=0x7fa8601b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:38.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 -- 172.21.15.32:0/432449773 shutdown_connections 2021-04-09T18:12:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.257 7fa8668a8700 1 -- 172.21.15.32:0/432449773 wait complete. 2021-04-09T18:12:38.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:12:38.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.424 7f3077082700 1 Processor -- start 2021-04-09T18:12:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.424 7f3077082700 1 -- start start 2021-04-09T18:12:38.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.425 7f3077082700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f307010d010 0x7f30700d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.425 7f3077082700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f30700d1da0 0x7f30700d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.425 7f3077082700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307010d480 con 0x7f30700d82c0 2021-04-09T18:12:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.425 7f3077082700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30700d8d30 con 0x7f307010d010 2021-04-09T18:12:38.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.425 7f3077082700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30700d9220 con 0x7f30700d1da0 2021-04-09T18:12:38.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.425 7f3074e9e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f307010d010 0x7f30700d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.425 7f307569f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f30700d82c0 0x7f307010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51186/0 (socket says 172.21.15.32:51186) 2021-04-09T18:12:38.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.425 7f307569f700 1 -- 172.21.15.32:0/2485363422 learned_addr learned my addr 172.21.15.32:0/2485363422 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:38.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f306ffff700 1 --2- 172.21.15.32:0/2485363422 >> v2:172.21.15.86:3300/0 conn(0x7f30700d1da0 0x7f30700d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f3074e9e700 1 -- 172.21.15.32:0/2485363422 >> v2:172.21.15.86:3300/0 conn(0x7f30700d1da0 msgr2=0x7f30700d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f3074e9e700 1 --2- 172.21.15.32:0/2485363422 >> v2:172.21.15.86:3300/0 conn(0x7f30700d1da0 0x7f30700d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f3074e9e700 1 -- 172.21.15.32:0/2485363422 >> v1:172.21.15.32:6789/0 conn(0x7f30700d82c0 legacy=0x7f307010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f3074e9e700 1 -- 172.21.15.32:0/2485363422 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30700d86f0 con 0x7f307010d010 2021-04-09T18:12:38.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f306f7fe700 1 -- 172.21.15.32:0/2485363422 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457928240 0 0) 0x7f3068000d30 con 0x7f30700d82c0 2021-04-09T18:12:38.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f3074e9e700 1 --2- 172.21.15.32:0/2485363422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f307010d010 0x7f30700d1780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3060006650 tx=0x7f3060009a00).ready entity=mon.0 client_cookie=4ca3c110f322d720 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f306f7fe700 1 -- 172.21.15.32:0/2485363422 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3060009df0 con 0x7f307010d010 2021-04-09T18:12:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f306f7fe700 1 -- 172.21.15.32:0/2485363422 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30600009b0 con 0x7f307010d010 2021-04-09T18:12:38.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f306f7fe700 1 -- 172.21.15.32:0/2485363422 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f306000c2b0 con 0x7f307010d010 2021-04-09T18:12:38.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f3077082700 1 -- 172.21.15.32:0/2485363422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f307010d010 msgr2=0x7f30700d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.426 7f3077082700 1 --2- 172.21.15.32:0/2485363422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f307010d010 0x7f30700d1780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3060006650 tx=0x7f3060009a00).stop 2021-04-09T18:12:38.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.427 7f3077082700 1 -- 172.21.15.32:0/2485363422 shutdown_connections 2021-04-09T18:12:38.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.427 7f3077082700 1 --2- 172.21.15.32:0/2485363422 >> v2:172.21.15.86:3300/0 conn(0x7f30700d1da0 0x7f30700d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.427 7f3077082700 1 --2- 172.21.15.32:0/2485363422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f307010d010 0x7f30700d1780 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.427 7f3077082700 1 -- 172.21.15.32:0/2485363422 >> 172.21.15.32:0/2485363422 conn(0x7f30700ce5f0 msgr2=0x7f30700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:38.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.427 7f3077082700 1 -- 172.21.15.32:0/2485363422 shutdown_connections 2021-04-09T18:12:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.427 7f3077082700 1 -- 172.21.15.32:0/2485363422 wait complete. 2021-04-09T18:12:38.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.447 7f3077082700 1 Processor -- start 2021-04-09T18:12:38.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.447 7f3077082700 1 -- start start 2021-04-09T18:12:38.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3077082700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30700c3c00 0x7f307009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3077082700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3070095240 0x7f30701a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3077082700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701a1270 0x7f30700b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3077082700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307009cda0 con 0x7f30700c3c00 2021-04-09T18:12:38.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3077082700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30700c3680 con 0x7f30701a1270 2021-04-09T18:12:38.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3077082700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3070095640 con 0x7f3070095240 2021-04-09T18:12:38.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3074e9e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30700c3c00 0x7f307009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3074e9e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30700c3c00 0x7f307009c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41600/0 (socket says 172.21.15.32:41600) 2021-04-09T18:12:38.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3074e9e700 1 -- 172.21.15.32:0/1542813242 learned_addr learned my addr 172.21.15.32:0/1542813242 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:38.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f306ffff700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3070095240 0x7f30701a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f307569f700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701a1270 0x7f30700b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3074e9e700 1 -- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3070095240 msgr2=0x7f30701a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3074e9e700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3070095240 0x7f30701a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3074e9e700 1 -- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701a1270 msgr2=0x7f30700b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3074e9e700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701a1270 0x7f30700b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3074e9e700 1 -- 172.21.15.32:0/1542813242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30701a1670 con 0x7f30700c3c00 2021-04-09T18:12:38.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3074e9e700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30700c3c00 0x7f307009c600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f30600096a0 tx=0x7f306000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:38.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f306d7fa700 1 -- 172.21.15.32:0/1542813242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30600093a0 con 0x7f30700c3c00 2021-04-09T18:12:38.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f306d7fa700 1 -- 172.21.15.32:0/1542813242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30600067f0 con 0x7f30700c3c00 2021-04-09T18:12:38.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f306d7fa700 1 -- 172.21.15.32:0/1542813242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30600022b0 con 0x7f30700c3c00 2021-04-09T18:12:38.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3077082700 1 -- 172.21.15.32:0/1542813242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30701a1670 con 0x7f30700c3c00 2021-04-09T18:12:38.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.448 7f3077082700 1 -- 172.21.15.32:0/1542813242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30701a1670 con 0x7f30700c3c00 2021-04-09T18:12:38.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.449 7f306d7fa700 1 -- 172.21.15.32:0/1542813242 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f30600052e0 con 0x7f30700c3c00 2021-04-09T18:12:38.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.449 7f306d7fa700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f30500188d0 0x7f305001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.449 7f306d7fa700 1 -- 172.21.15.32:0/1542813242 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f30600329f0 con 0x7f30700c3c00 2021-04-09T18:12:38.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.449 7f306ffff700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f30500188d0 0x7f305001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.450 7f306ffff700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f30500188d0 0x7f305001add0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:38.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.469 7f3077082700 1 -- 172.21.15.32:0/1542813242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3070093d50 con 0x7f30700c3c00 2021-04-09T18:12:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.473 7f306d7fa700 1 -- 172.21.15.32:0/1542813242 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f306001f730 con 0x7f30700c3c00 2021-04-09T18:12:38.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.652 7f3077082700 1 -- 172.21.15.32:0/1542813242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f307010d480 con 0x7f30700c3c00 2021-04-09T18:12:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.652 7f306d7fa700 1 -- 172.21.15.32:0/1542813242 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f30600052e0 con 0x7f30700c3c00 2021-04-09T18:12:38.692 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:12:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 -- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f30500188d0 msgr2=0x7f305001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f30500188d0 0x7f305001add0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 -- 172.21.15.32:0/1542813242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30700c3c00 msgr2=0x7f307009c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30700c3c00 0x7f307009c600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f30600096a0 tx=0x7f306000b8e0).stop 2021-04-09T18:12:38.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 -- 172.21.15.32:0/1542813242 shutdown_connections 2021-04-09T18:12:38.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f30500188d0 0x7f305001add0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701a1270 0x7f30700b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3070095240 0x7f30701a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 --2- 172.21.15.32:0/1542813242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30700c3c00 0x7f307009c600 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 -- 172.21.15.32:0/1542813242 >> 172.21.15.32:0/1542813242 conn(0x7f30700975f0 msgr2=0x7f3070096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:38.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 -- 172.21.15.32:0/1542813242 shutdown_connections 2021-04-09T18:12:38.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.655 7f3077082700 1 -- 172.21.15.32:0/1542813242 wait complete. 2021-04-09T18:12:38.728 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-04-09T18:12:38.734 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2021-04-09T18:12:38.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2021-04-09T18:12:38.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.869 7f2f9d3e8700 1 Processor -- start 2021-04-09T18:12:38.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.869 7f2f9d3e8700 1 -- start start 2021-04-09T18:12:38.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.869 7f2f9d3e8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f980d7ca0 0x7f2f980d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.869 7f2f9d3e8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9810d480 con 0x7f2f9810d010 2021-04-09T18:12:38.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.869 7f2f9d3e8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f980d86c0 con 0x7f2f9810da00 2021-04-09T18:12:38.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.869 7f2f9d3e8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f980d9220 con 0x7f2f980d7ca0 2021-04-09T18:12:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9707d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2f9810d010 0x7f2f980d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51198/0 (socket says 172.21.15.32:51198) 2021-04-09T18:12:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9707d700 1 -- 172.21.15.32:0/1625392125 learned_addr learned my addr 172.21.15.32:0/1625392125 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:38.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9687c700 1 --2- 172.21.15.32:0/1625392125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f980d7ca0 0x7f2f980d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9607b700 1 -- 172.21.15.32:0/1625392125 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159255613 0 0) 0x7f2f88000ce0 con 0x7f2f9810d010 2021-04-09T18:12:38.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9607b700 1 -- 172.21.15.32:0/1625392125 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f80002fe0 con 0x7f2f9810d010 2021-04-09T18:12:38.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9607b700 1 -- 172.21.15.32:0/1625392125 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782467503 0 0) 0x7f2f90000d30 con 0x7f2f9810da00 2021-04-09T18:12:38.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9607b700 1 -- 172.21.15.32:0/1625392125 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f80003210 con 0x7f2f9810da00 2021-04-09T18:12:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9687c700 1 -- 172.21.15.32:0/1625392125 >> v1:172.21.15.86:6789/0 conn(0x7f2f9810da00 legacy=0x7f2f9810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9687c700 1 -- 172.21.15.32:0/1625392125 >> v1:172.21.15.32:6789/0 conn(0x7f2f9810d010 legacy=0x7f2f980d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9607b700 1 -- 172.21.15.32:0/1625392125 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3312336747 0 0) 0x7f2f88000ed0 con 0x7f2f9810d010 2021-04-09T18:12:38.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9687c700 1 -- 172.21.15.32:0/1625392125 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f980d1f70 con 0x7f2f980d7ca0 2021-04-09T18:12:38.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9687c700 1 --2- 172.21.15.32:0/1625392125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f980d7ca0 0x7f2f980d80a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f2f8c006650 tx=0x7f2f8c009a00).ready entity=mon.2 client_cookie=fcef505a934fa0e4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:38.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9607b700 1 -- 172.21.15.32:0/1625392125 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8c009df0 con 0x7f2f980d7ca0 2021-04-09T18:12:38.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9607b700 1 -- 172.21.15.32:0/1625392125 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8c0009b0 con 0x7f2f980d7ca0 2021-04-09T18:12:38.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.870 7f2f9607b700 1 -- 172.21.15.32:0/1625392125 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8c00c2b0 con 0x7f2f980d7ca0 2021-04-09T18:12:38.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.871 7f2f9d3e8700 1 -- 172.21.15.32:0/1625392125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f980d7ca0 msgr2=0x7f2f980d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.871 7f2f9d3e8700 1 --2- 172.21.15.32:0/1625392125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f980d7ca0 0x7f2f980d80a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f2f8c006650 tx=0x7f2f8c009a00).stop 2021-04-09T18:12:38.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.871 7f2f9d3e8700 1 -- 172.21.15.32:0/1625392125 shutdown_connections 2021-04-09T18:12:38.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.871 7f2f9d3e8700 1 --2- 172.21.15.32:0/1625392125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f980d7ca0 0x7f2f980d80a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.871 7f2f9d3e8700 1 -- 172.21.15.32:0/1625392125 >> 172.21.15.32:0/1625392125 conn(0x7f2f980ce5f0 msgr2=0x7f2f980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:38.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.871 7f2f9d3e8700 1 -- 172.21.15.32:0/1625392125 shutdown_connections 2021-04-09T18:12:38.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.871 7f2f9d3e8700 1 -- 172.21.15.32:0/1625392125 wait complete. 2021-04-09T18:12:38.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.891 7f2f9d3e8700 1 Processor -- start 2021-04-09T18:12:38.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.891 7f2f9d3e8700 1 -- start start 2021-04-09T18:12:38.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9d3e8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981ac380 0x7f2f9818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9d3e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f981ac780 0x7f2f9818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9d3e8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98183520 0x7f2f98183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9d3e8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f981ace10 con 0x7f2f981ac780 2021-04-09T18:12:38.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9d3e8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f98189cf0 con 0x7f2f981ac380 2021-04-09T18:12:38.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9d3e8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f98183f40 con 0x7f2f98183520 2021-04-09T18:12:38.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9687c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f981ac780 0x7f2f9818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9687c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f981ac780 0x7f2f9818a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41614/0 (socket says 172.21.15.32:41614) 2021-04-09T18:12:38.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9687c700 1 -- 172.21.15.32:0/4230145264 learned_addr learned my addr 172.21.15.32:0/4230145264 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:38.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9707d700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981ac380 0x7f2f9818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9787e700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98183520 0x7f2f98183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9687c700 1 -- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98183520 msgr2=0x7f2f98183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9687c700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98183520 0x7f2f98183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9687c700 1 -- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981ac380 msgr2=0x7f2f9818a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9687c700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981ac380 0x7f2f9818a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.892 7f2f9687c700 1 -- 172.21.15.32:0/4230145264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f98184160 con 0x7f2f981ac780 2021-04-09T18:12:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.893 7f2f9687c700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f981ac780 0x7f2f9818a850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2f8c0096a0 tx=0x7f2f8c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.893 7f2f7ffff700 1 -- 172.21.15.32:0/4230145264 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8c00bd80 con 0x7f2f981ac780 2021-04-09T18:12:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.893 7f2f7ffff700 1 -- 172.21.15.32:0/4230145264 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8c006030 con 0x7f2f981ac780 2021-04-09T18:12:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.893 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f98184160 con 0x7f2f981ac780 2021-04-09T18:12:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.893 7f2f7ffff700 1 -- 172.21.15.32:0/4230145264 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8c005560 con 0x7f2f981ac780 2021-04-09T18:12:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.893 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f98184160 con 0x7f2f981ac780 2021-04-09T18:12:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.894 7f2f7ffff700 1 -- 172.21.15.32:0/4230145264 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2f8c006030 con 0x7f2f981ac780 2021-04-09T18:12:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.894 7f2f7ffff700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f780188c0 0x7f2f7801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.894 7f2f7ffff700 1 -- 172.21.15.32:0/4230145264 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2f8c0327f0 con 0x7f2f981ac780 2021-04-09T18:12:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.894 7f2f9707d700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f780188c0 0x7f2f7801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.895 7f2f9707d700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f780188c0 0x7f2f7801adc0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:39.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.914 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f98189cf0 con 0x7f2f981ac780 2021-04-09T18:12:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:38.917 7f2f7ffff700 1 -- 172.21.15.32:0/4230145264 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2f8c0009c0 con 0x7f2f981ac780 2021-04-09T18:12:39.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.154 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f2f98189cf0 con 0x7f2f981ac780 2021-04-09T18:12:39.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.156 7f2f7ffff700 1 -- 172.21.15.32:0/4230145264 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v3) v1 ==== 74+0+70604 (secure 0 0 0) 0x7f2f8c006030 con 0x7f2f981ac780 2021-04-09T18:12:39.178 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f780188c0 msgr2=0x7f2f7801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:39.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f780188c0 0x7f2f7801adc0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:39.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f981ac780 msgr2=0x7f2f9818a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f981ac780 0x7f2f9818a850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2f8c0096a0 tx=0x7f2f8c00b990).stop 2021-04-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 shutdown_connections 2021-04-09T18:12:39.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f780188c0 0x7f2f7801adc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:39.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f981ac780 0x7f2f9818a850 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:39.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98183520 0x7f2f98183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:39.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 --2- 172.21.15.32:0/4230145264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981ac380 0x7f2f9818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:39.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.159 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 >> 172.21.15.32:0/4230145264 conn(0x7f2f9819b890 msgr2=0x7f2f981a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.160 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 shutdown_connections 2021-04-09T18:12:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:39.160 7f2f9d3e8700 1 -- 172.21.15.32:0/4230145264 wait complete. 2021-04-09T18:12:39.202 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":3,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":23869},{"type":"v1","addr":"172.21.15.86:6801","nonce":23869}]},"active_addr":"172.21.15.86:6801/23869","active_change":"2021-04-09 18:12:14.249236","available":true,"standbys":[{"gid":4110,"name":"y","available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"]}} 2021-04-09T18:12:39.235 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2021-04-09T18:12:40.042 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-04-09T18:12:40.218 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.168 7f8bd4b99700 1 Processor -- start 2021-04-09T18:12:40.219 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.168 7f8bd4b99700 1 -- start start 2021-04-09T18:12:40.219 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bd4b99700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8bd010d010 0x7f8bd00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.219 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bd4b99700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd010d480 con 0x7f8bd00d1da0 2021-04-09T18:12:40.220 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bd4b99700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd010e1f0 con 0x7f8bd00d8490 2021-04-09T18:12:40.220 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bd4b99700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd00d8d30 con 0x7f8bd010d010 2021-04-09T18:12:40.221 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bce05d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8bd00d1da0 0x7f8bd00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51214/0 (socket says 172.21.15.32:51214) 2021-04-09T18:12:40.221 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bce05d700 1 -- 172.21.15.32:0/689567246 learned_addr learned my addr 172.21.15.32:0/689567246 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:40.222 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bce85e700 1 --2- 172.21.15.32:0/689567246 >> v2:172.21.15.86:3300/0 conn(0x7f8bd010d010 0x7f8bd00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.222 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bcd85c700 1 -- 172.21.15.32:0/689567246 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191210921 0 0) 0x7f8bc4000ce0 con 0x7f8bd00d1da0 2021-04-09T18:12:40.223 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bcd85c700 1 -- 172.21.15.32:0/689567246 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb8002fe0 con 0x7f8bd00d1da0 2021-04-09T18:12:40.223 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bcd85c700 1 -- 172.21.15.32:0/689567246 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014274210 0 0) 0x7f8bc8000d30 con 0x7f8bd00d8490 2021-04-09T18:12:40.224 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.169 7f8bcd85c700 1 -- 172.21.15.32:0/689567246 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb8003210 con 0x7f8bd00d8490 2021-04-09T18:12:40.224 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bce85e700 1 -- 172.21.15.32:0/689567246 >> v1:172.21.15.86:6789/0 conn(0x7f8bd00d8490 legacy=0x7f8bd010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.224 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bce85e700 1 -- 172.21.15.32:0/689567246 >> v1:172.21.15.32:6789/0 conn(0x7f8bd00d1da0 legacy=0x7f8bd00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.225 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bcd85c700 1 -- 172.21.15.32:0/689567246 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2340079983 0 0) 0x7f8bc40010e0 con 0x7f8bd00d1da0 2021-04-09T18:12:40.225 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bce85e700 1 -- 172.21.15.32:0/689567246 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd00d9220 con 0x7f8bd010d010 2021-04-09T18:12:40.226 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bce85e700 1 --2- 172.21.15.32:0/689567246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd010d010 0x7f8bd00d1780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8bc0006650 tx=0x7f8bc0009a00).ready entity=mon.1 client_cookie=4321bc27ab09e560 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:40.226 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bcd85c700 1 -- 172.21.15.32:0/689567246 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bc0009df0 con 0x7f8bd010d010 2021-04-09T18:12:40.244 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bcd85c700 1 -- 172.21.15.32:0/689567246 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc00009b0 con 0x7f8bd010d010 2021-04-09T18:12:40.244 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bcd85c700 1 -- 172.21.15.32:0/689567246 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bc000c2b0 con 0x7f8bd010d010 2021-04-09T18:12:40.245 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bd4b99700 1 -- 172.21.15.32:0/689567246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd010d010 msgr2=0x7f8bd00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.245 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.170 7f8bd4b99700 1 --2- 172.21.15.32:0/689567246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd010d010 0x7f8bd00d1780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8bc0006650 tx=0x7f8bc0009a00).stop 2021-04-09T18:12:40.255 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.171 7f8bd4b99700 1 -- 172.21.15.32:0/689567246 shutdown_connections 2021-04-09T18:12:40.256 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.171 7f8bd4b99700 1 --2- 172.21.15.32:0/689567246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd010d010 0x7f8bd00d1780 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.256 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.171 7f8bd4b99700 1 -- 172.21.15.32:0/689567246 >> 172.21.15.32:0/689567246 conn(0x7f8bd00ce5f0 msgr2=0x7f8bd00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:40.257 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.171 7f8bd4b99700 1 -- 172.21.15.32:0/689567246 shutdown_connections 2021-04-09T18:12:40.257 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.171 7f8bd4b99700 1 -- 172.21.15.32:0/689567246 wait complete. 2021-04-09T18:12:40.257 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.191 7f8bd4b99700 1 Processor -- start 2021-04-09T18:12:40.258 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.191 7f8bd4b99700 1 -- start start 2021-04-09T18:12:40.258 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bd4b99700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd0185060 0x7f8bd0185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.259 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bd4b99700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bd0186c00 0x7f8bd0187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.281 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bd4b99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8bd0191360 0x7f8bd0191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.282 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bd4b99700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd01863a0 con 0x7f8bd0191360 2021-04-09T18:12:40.282 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bd4b99700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0185a60 con 0x7f8bd0185060 2021-04-09T18:12:40.283 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bd4b99700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd0187620 con 0x7f8bd0186c00 2021-04-09T18:12:40.283 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bcf05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8bd0191360 0x7f8bd0191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.284 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bcf05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8bd0191360 0x7f8bd0191760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41628/0 (socket says 172.21.15.32:41628) 2021-04-09T18:12:40.284 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bcf05f700 1 -- 172.21.15.32:0/2863382579 learned_addr learned my addr 172.21.15.32:0/2863382579 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:40.284 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bce05d700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bd0186c00 0x7f8bd0187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.309 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bce85e700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd0185060 0x7f8bd0185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.310 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bcf05f700 1 -- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bd0186c00 msgr2=0x7f8bd0187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.310 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bcf05f700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bd0186c00 0x7f8bd0187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.310 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bcf05f700 1 -- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd0185060 msgr2=0x7f8bd0185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.310 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bcf05f700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd0185060 0x7f8bd0185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.311 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bcf05f700 1 -- 172.21.15.32:0/2863382579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd0185460 con 0x7f8bd0191360 2021-04-09T18:12:40.311 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bce85e700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd0185060 0x7f8bd0185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:12:40.311 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bcf05f700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8bd0191360 0x7f8bd0191760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8bc800b8e0 tx=0x7f8bc800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:40.311 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bbf7fe700 1 -- 172.21.15.32:0/2863382579 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bc8000d30 con 0x7f8bd0191360 2021-04-09T18:12:40.312 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bbf7fe700 1 -- 172.21.15.32:0/2863382579 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc800c2f0 con 0x7f8bd0191360 2021-04-09T18:12:40.312 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd0185460 con 0x7f8bd0191360 2021-04-09T18:12:40.312 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bbf7fe700 1 -- 172.21.15.32:0/2863382579 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bc800c790 con 0x7f8bd0191360 2021-04-09T18:12:40.312 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.192 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd0185460 con 0x7f8bd0191360 2021-04-09T18:12:40.312 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.193 7f8bbf7fe700 1 -- 172.21.15.32:0/2863382579 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8bc8000cc0 con 0x7f8bd0191360 2021-04-09T18:12:40.313 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.193 7f8bbf7fe700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bb40188c0 0x7f8bb401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.313 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.193 7f8bbf7fe700 1 -- 172.21.15.32:0/2863382579 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8bc8032c00 con 0x7f8bd0191360 2021-04-09T18:12:40.313 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.194 7f8bce85e700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bb40188c0 0x7f8bb401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.313 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.194 7f8bce85e700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bb40188c0 0x7f8bb401adc0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:40.314 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.213 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd01866d0 con 0x7f8bd0191360 2021-04-09T18:12:40.314 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.216 7f8bbf7fe700 1 -- 172.21.15.32:0/2863382579 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8bc800bdc0 con 0x7f8bd0191360 2021-04-09T18:12:40.424 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.405 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bd01866d0 con 0x7f8bd0191360 2021-04-09T18:12:40.425 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.406 7f8bbf7fe700 1 -- 172.21.15.32:0/2863382579 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f8bc8000cc0 con 0x7f8bd0191360 2021-04-09T18:12:40.426 INFO:teuthology.misc.health.smithi032.stdout: 2021-04-09T18:12:40.426 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":20,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:34.773743","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:12:40.429 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bb40188c0 msgr2=0x7f8bb401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.429 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bb40188c0 0x7f8bb401adc0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.430 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8bd0191360 msgr2=0x7f8bd0191760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.430 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8bd0191360 0x7f8bd0191760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8bc800b8e0 tx=0x7f8bc800bc80).stop 2021-04-09T18:12:40.431 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 shutdown_connections 2021-04-09T18:12:40.432 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bb40188c0 0x7f8bb401adc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.432 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8bd0191360 0x7f8bd0191760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.433 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bd0186c00 0x7f8bd0187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.433 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 --2- 172.21.15.32:0/2863382579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bd0185060 0x7f8bd0185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.433 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 >> 172.21.15.32:0/2863382579 conn(0x7f8bd01837d0 msgr2=0x7f8bd0195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:40.434 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 shutdown_connections 2021-04-09T18:12:40.434 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:12:40.408 7f8bd4b99700 1 -- 172.21.15.32:0/2863382579 wait complete. 2021-04-09T18:12:40.442 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-04-09T18:12:40.443 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:12:40.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.581 7f5e3da90700 1 Processor -- start 2021-04-09T18:12:40.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.581 7f5e3da90700 1 -- start start 2021-04-09T18:12:40.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.581 7f5e3da90700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e3810d010 0x7f5e380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e3da90700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3810d480 con 0x7f5e380d8490 2021-04-09T18:12:40.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e3da90700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3810e1f0 con 0x7f5e380d1da0 2021-04-09T18:12:40.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e3da90700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e380d8d30 con 0x7f5e3810d010 2021-04-09T18:12:40.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e377fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e3810d010 0x7f5e380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e37fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5e380d8490 0x7f5e3810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51226/0 (socket says 172.21.15.32:51226) 2021-04-09T18:12:40.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e37fff700 1 -- 172.21.15.32:0/2658515232 learned_addr learned my addr 172.21.15.32:0/2658515232 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:40.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e377fe700 1 -- 172.21.15.32:0/2658515232 >> v1:172.21.15.86:6789/0 conn(0x7f5e380d1da0 legacy=0x7f5e380d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:40.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e377fe700 1 -- 172.21.15.32:0/2658515232 >> v1:172.21.15.32:6789/0 conn(0x7f5e380d8490 legacy=0x7f5e3810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e377fe700 1 -- 172.21.15.32:0/2658515232 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e380d9220 con 0x7f5e3810d010 2021-04-09T18:12:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e367fc700 1 -- 172.21.15.32:0/2658515232 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475103251 0 0) 0x7f5e30000d30 con 0x7f5e380d8490 2021-04-09T18:12:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e377fe700 1 --2- 172.21.15.32:0/2658515232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3810d010 0x7f5e380d1780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5e28006650 tx=0x7f5e28009a80).ready entity=mon.0 client_cookie=66e4bfb24c39313d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:40.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e367fc700 1 -- 172.21.15.32:0/2658515232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e280009b0 con 0x7f5e3810d010 2021-04-09T18:12:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e367fc700 1 -- 172.21.15.32:0/2658515232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e28006740 con 0x7f5e3810d010 2021-04-09T18:12:40.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e367fc700 1 -- 172.21.15.32:0/2658515232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2800c0c0 con 0x7f5e3810d010 2021-04-09T18:12:40.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e3da90700 1 -- 172.21.15.32:0/2658515232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3810d010 msgr2=0x7f5e380d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.582 7f5e3da90700 1 --2- 172.21.15.32:0/2658515232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3810d010 0x7f5e380d1780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5e28006650 tx=0x7f5e28009a80).stop 2021-04-09T18:12:40.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.583 7f5e3da90700 1 -- 172.21.15.32:0/2658515232 shutdown_connections 2021-04-09T18:12:40.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.583 7f5e3da90700 1 --2- 172.21.15.32:0/2658515232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e3810d010 0x7f5e380d1780 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.583 7f5e3da90700 1 -- 172.21.15.32:0/2658515232 >> 172.21.15.32:0/2658515232 conn(0x7f5e380ce5f0 msgr2=0x7f5e380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:40.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.583 7f5e3da90700 1 -- 172.21.15.32:0/2658515232 shutdown_connections 2021-04-09T18:12:40.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.583 7f5e3da90700 1 -- 172.21.15.32:0/2658515232 wait complete. 2021-04-09T18:12:40.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.603 7f5e3da90700 1 Processor -- start 2021-04-09T18:12:40.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.603 7f5e3da90700 1 -- start start 2021-04-09T18:12:40.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e3da90700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e380c10d0 0x7f5e380bdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e3da90700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380be590 0x7f5e381c2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e3da90700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e381c3530 0x7f5e381a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e3da90700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380bd9f0 con 0x7f5e381c3530 2021-04-09T18:12:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e3da90700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380c14d0 con 0x7f5e380be590 2021-04-09T18:12:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e3da90700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380be990 con 0x7f5e380c10d0 2021-04-09T18:12:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e381c3530 0x7f5e381a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e381c3530 0x7f5e381a5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41644/0 (socket says 172.21.15.32:41644) 2021-04-09T18:12:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e37fff700 1 -- 172.21.15.32:0/1092369609 learned_addr learned my addr 172.21.15.32:0/1092369609 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e377fe700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e380c10d0 0x7f5e380bdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e36ffd700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380be590 0x7f5e381c2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e37fff700 1 -- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e380c10d0 msgr2=0x7f5e380bdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e37fff700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e380c10d0 0x7f5e380bdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e37fff700 1 -- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380be590 msgr2=0x7f5e381c2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e37fff700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380be590 0x7f5e381c2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.604 7f5e37fff700 1 -- 172.21.15.32:0/1092369609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e380c3780 con 0x7f5e381c3530 2021-04-09T18:12:40.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.605 7f5e37fff700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e381c3530 0x7f5e381a5680 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5e30007f90 tx=0x7f5e3000abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:40.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.605 7f5e1ffff700 1 -- 172.21.15.32:0/1092369609 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e30000cc0 con 0x7f5e381c3530 2021-04-09T18:12:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.605 7f5e1ffff700 1 -- 172.21.15.32:0/1092369609 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e300070d0 con 0x7f5e381c3530 2021-04-09T18:12:40.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.605 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e380c3780 con 0x7f5e381c3530 2021-04-09T18:12:40.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.605 7f5e1ffff700 1 -- 172.21.15.32:0/1092369609 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e30007520 con 0x7f5e381c3530 2021-04-09T18:12:40.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.605 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e380c3780 con 0x7f5e381c3530 2021-04-09T18:12:40.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.606 7f5e1ffff700 1 -- 172.21.15.32:0/1092369609 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5e300072b0 con 0x7f5e381c3530 2021-04-09T18:12:40.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.606 7f5e1ffff700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e18018930 0x7f5e1801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:40.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.606 7f5e1ffff700 1 -- 172.21.15.32:0/1092369609 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5e30032d30 con 0x7f5e381c3530 2021-04-09T18:12:40.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.606 7f5e377fe700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e18018930 0x7f5e1801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:40.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.606 7f5e377fe700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e18018930 0x7f5e1801ae10 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:40.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.626 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e380bcb40 con 0x7f5e381c3530 2021-04-09T18:12:40.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.629 7f5e1ffff700 1 -- 172.21.15.32:0/1092369609 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5e30007d00 con 0x7f5e381c3530 2021-04-09T18:12:40.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.818 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e380bcb40 con 0x7f5e381c3530 2021-04-09T18:12:40.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.819 7f5e1ffff700 1 -- 172.21.15.32:0/1092369609 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f5e300070d0 con 0x7f5e381c3530 2021-04-09T18:12:40.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:40.840 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:34.773743","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:12:40.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e18018930 msgr2=0x7f5e1801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e18018930 0x7f5e1801ae10 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e381c3530 msgr2=0x7f5e381a5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:40.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e381c3530 0x7f5e381a5680 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5e30007f90 tx=0x7f5e3000abd0).stop 2021-04-09T18:12:40.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 shutdown_connections 2021-04-09T18:12:40.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e18018930 0x7f5e1801ae10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e381c3530 0x7f5e381a5680 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380be590 0x7f5e381c2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 --2- 172.21.15.32:0/1092369609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e380c10d0 0x7f5e380bdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:40.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 >> 172.21.15.32:0/1092369609 conn(0x7f5e381acbe0 msgr2=0x7f5e3808dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 shutdown_connections 2021-04-09T18:12:40.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:40.821 7f5e3da90700 1 -- 172.21.15.32:0/1092369609 wait complete. 2021-04-09T18:12:40.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-04-09T18:12:41.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.047 7fe57efc2700 1 Processor -- start 2021-04-09T18:12:41.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.047 7fe57efc2700 1 -- start start 2021-04-09T18:12:41.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57efc2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5780cffb0 0x7fe5780d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57efc2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5780d7810 0x7fe5780d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57efc2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780d7290 con 0x7fe5780d5c80 2021-04-09T18:12:41.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57efc2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780d8590 con 0x7fe5780d7810 2021-04-09T18:12:41.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57efc2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5780d91f0 con 0x7fe5780cffb0 2021-04-09T18:12:41.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe577fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5780d7810 0x7fe5780d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57d5df700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe5780d5c80 0x7fe5780d6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51238/0 (socket says 172.21.15.32:51238) 2021-04-09T18:12:41.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57d5df700 1 -- 172.21.15.32:0/1760475578 learned_addr learned my addr 172.21.15.32:0/1760475578 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:41.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57cdde700 1 --2- 172.21.15.32:0/1760475578 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5780cffb0 0x7fe5780d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe577fff700 1 -- 172.21.15.32:0/1760475578 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5780cffb0 msgr2=0x7fe5780d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:41.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe577fff700 1 --2- 172.21.15.32:0/1760475578 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5780cffb0 0x7fe5780d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe577fff700 1 -- 172.21.15.32:0/1760475578 >> v1:172.21.15.32:6789/0 conn(0x7fe5780d5c80 legacy=0x7fe5780d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe577fff700 1 -- 172.21.15.32:0/1760475578 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5780d8230 con 0x7fe5780d7810 2021-04-09T18:12:41.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe5777fe700 1 -- 172.21.15.32:0/1760475578 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257362262 0 0) 0x7fe570000d30 con 0x7fe5780d5c80 2021-04-09T18:12:41.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57cdde700 1 --2- 172.21.15.32:0/1760475578 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5780cffb0 0x7fe5780d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:12:41.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe577fff700 1 --2- 172.21.15.32:0/1760475578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5780d7810 0x7fe5780d7c10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fe568006650 tx=0x7fe568009a80).ready entity=mon.0 client_cookie=e08a5914b396f5e2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:41.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe5777fe700 1 -- 172.21.15.32:0/1760475578 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5680009b0 con 0x7fe5780d7810 2021-04-09T18:12:41.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe5777fe700 1 -- 172.21.15.32:0/1760475578 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56800c0c0 con 0x7fe5780d7810 2021-04-09T18:12:41.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe5777fe700 1 -- 172.21.15.32:0/1760475578 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe56800c510 con 0x7fe5780d7810 2021-04-09T18:12:41.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57efc2700 1 -- 172.21.15.32:0/1760475578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5780d7810 msgr2=0x7fe5780d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.048 7fe57efc2700 1 --2- 172.21.15.32:0/1760475578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5780d7810 0x7fe5780d7c10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fe568006650 tx=0x7fe568009a80).stop 2021-04-09T18:12:41.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.049 7fe57efc2700 1 -- 172.21.15.32:0/1760475578 shutdown_connections 2021-04-09T18:12:41.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.049 7fe57efc2700 1 --2- 172.21.15.32:0/1760475578 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5780d7810 0x7fe5780d7c10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.049 7fe57efc2700 1 --2- 172.21.15.32:0/1760475578 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5780cffb0 0x7fe5780d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.049 7fe57efc2700 1 -- 172.21.15.32:0/1760475578 >> 172.21.15.32:0/1760475578 conn(0x7fe5780ce5d0 msgr2=0x7fe5780cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:41.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.049 7fe57efc2700 1 -- 172.21.15.32:0/1760475578 shutdown_connections 2021-04-09T18:12:41.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.049 7fe57efc2700 1 -- 172.21.15.32:0/1760475578 wait complete. 2021-04-09T18:12:41.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.069 7fe57efc2700 1 Processor -- start 2021-04-09T18:12:41.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.069 7fe57efc2700 1 -- start start 2021-04-09T18:12:41.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe57efc2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe57808a620 0x7fe57808b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe57efc2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe57808b6d0 0x7fe578180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe57efc2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe578180c50 0x7fe578181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe57efc2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57808e660 con 0x7fe57808b6d0 2021-04-09T18:12:41.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe57efc2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57808ab30 con 0x7fe57808a620 2021-04-09T18:12:41.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe57efc2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57808bad0 con 0x7fe578180c50 2021-04-09T18:12:41.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe577fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe57808b6d0 0x7fe578180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe577fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe57808b6d0 0x7fe578180630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41654/0 (socket says 172.21.15.32:41654) 2021-04-09T18:12:41.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe577fff700 1 -- 172.21.15.32:0/3812050046 learned_addr learned my addr 172.21.15.32:0/3812050046 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:41.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe57d5df700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe578180c50 0x7fe578181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe57cdde700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe57808a620 0x7fe57808b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe577fff700 1 -- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe578180c50 msgr2=0x7fe578181050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe577fff700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe578180c50 0x7fe578181050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe577fff700 1 -- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe57808a620 msgr2=0x7fe57808b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.070 7fe577fff700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe57808a620 0x7fe57808b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.071 7fe577fff700 1 -- 172.21.15.32:0/3812050046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57808e1f0 con 0x7fe57808b6d0 2021-04-09T18:12:41.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.071 7fe577fff700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe57808b6d0 0x7fe578180630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe568009630 tx=0x7fe56800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:41.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.071 7fe5757fa700 1 -- 172.21.15.32:0/3812050046 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5680093a0 con 0x7fe57808b6d0 2021-04-09T18:12:41.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.071 7fe5757fa700 1 -- 172.21.15.32:0/3812050046 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56800bcf0 con 0x7fe57808b6d0 2021-04-09T18:12:41.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.071 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57808e1f0 con 0x7fe57808b6d0 2021-04-09T18:12:41.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.071 7fe5757fa700 1 -- 172.21.15.32:0/3812050046 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5680054d0 con 0x7fe57808b6d0 2021-04-09T18:12:41.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.071 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57808e1f0 con 0x7fe57808b6d0 2021-04-09T18:12:41.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.071 7fe5757fa700 1 -- 172.21.15.32:0/3812050046 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe5680054f0 con 0x7fe57808b6d0 2021-04-09T18:12:41.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.072 7fe5757fa700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe55c018930 0x7fe55c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.072 7fe5757fa700 1 -- 172.21.15.32:0/3812050046 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe568032a10 con 0x7fe57808b6d0 2021-04-09T18:12:41.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.072 7fe57cdde700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe55c018930 0x7fe55c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.072 7fe57cdde700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe55c018930 0x7fe55c01ae10 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:41.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.091 7fe57efc2700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7fe578062a70 0x7fe578062e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.091 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 --> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe57808e1f0 con 0x7fe578062a70 2021-04-09T18:12:41.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.092 7fe57d5df700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7fe578062a70 0x7fe578062e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.092 7fe57d5df700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7fe578062a70 0x7fe578062e70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:41.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.092 7fe5757fa700 1 -- 172.21.15.32:0/3812050046 <== osd.0 v2:172.21.15.32:6800/24886 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fe570004720 con 0x7fe578062a70 2021-04-09T18:12:41.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.099 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 --> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe5781b7180 con 0x7fe578062a70 2021-04-09T18:12:41.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.100 7fe5757fa700 1 -- 172.21.15.32:0/3812050046 <== osd.0 v2:172.21.15.32:6800/24886 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe570007b10 con 0x7fe578062a70 2021-04-09T18:12:41.210 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2021-04-09T18:12:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7fe578062a70 msgr2=0x7fe578062e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7fe578062a70 0x7fe578062e70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe55c018930 msgr2=0x7fe55c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe55c018930 0x7fe55c01ae10 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe57808b6d0 msgr2=0x7fe578180630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 --2- 172.21.15.32:0/3812050046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe57808b6d0 0x7fe578180630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe568009630 tx=0x7fe56800b900).stop 2021-04-09T18:12:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57d5df700 1 -- 172.21.15.32:0/3812050046 reap_dead start 2021-04-09T18:12:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 shutdown_connections 2021-04-09T18:12:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 >> 172.21.15.32:0/3812050046 conn(0x7fe578189800 msgr2=0x7fe5781b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:41.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 shutdown_connections 2021-04-09T18:12:41.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.101 7fe57efc2700 1 -- 172.21.15.32:0/3812050046 wait complete. 2021-04-09T18:12:41.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-04-09T18:12:41.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.383 7f39768e0700 1 Processor -- start 2021-04-09T18:12:41.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.383 7f39768e0700 1 -- start start 2021-04-09T18:12:41.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.383 7f39768e0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39700d6c80 0x7f39700d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.383 7f39768e0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39700d2440 0x7f397010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.383 7f39768e0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39700d7280 con 0x7f39700cffa0 2021-04-09T18:12:41.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.383 7f39768e0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39700d7080 con 0x7f39700d2440 2021-04-09T18:12:41.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.383 7f39768e0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39700d91e0 con 0x7f39700d6c80 2021-04-09T18:12:41.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f396ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f39700cffa0 0x7f39700d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51252/0 (socket says 172.21.15.32:51252) 2021-04-09T18:12:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f3974efd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39700d2440 0x7f397010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f396ffff700 1 -- 172.21.15.32:0/684136086 learned_addr learned my addr 172.21.15.32:0/684136086 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:41.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f3967fff700 1 --2- 172.21.15.32:0/684136086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39700d6c80 0x7f39700d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f396f7fe700 1 -- 172.21.15.32:0/684136086 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805978869 0 0) 0x7f3960000ce0 con 0x7f39700cffa0 2021-04-09T18:12:41.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f396f7fe700 1 -- 172.21.15.32:0/684136086 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f395c002fe0 con 0x7f39700cffa0 2021-04-09T18:12:41.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f3974efd700 1 -- 172.21.15.32:0/684136086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39700d6c80 msgr2=0x7f39700d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:41.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f3974efd700 1 --2- 172.21.15.32:0/684136086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39700d6c80 0x7f39700d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f3974efd700 1 -- 172.21.15.32:0/684136086 >> v1:172.21.15.32:6789/0 conn(0x7f39700cffa0 legacy=0x7f39700d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f3974efd700 1 -- 172.21.15.32:0/684136086 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39700d2840 con 0x7f39700d2440 2021-04-09T18:12:41.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f3974efd700 1 --2- 172.21.15.32:0/684136086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39700d2440 0x7f397010cfd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f3968006e30 tx=0x7f3968009e60).ready entity=mon.0 client_cookie=16fcb9f478363768 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:41.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f396f7fe700 1 -- 172.21.15.32:0/684136086 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3968006a50 con 0x7f39700d2440 2021-04-09T18:12:41.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f396f7fe700 1 -- 172.21.15.32:0/684136086 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396800b300 con 0x7f39700d2440 2021-04-09T18:12:41.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f396f7fe700 1 -- 172.21.15.32:0/684136086 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f396800b750 con 0x7f39700d2440 2021-04-09T18:12:41.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f39768e0700 1 -- 172.21.15.32:0/684136086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39700d2440 msgr2=0x7f397010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.384 7f39768e0700 1 --2- 172.21.15.32:0/684136086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39700d2440 0x7f397010cfd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f3968006e30 tx=0x7f3968009e60).stop 2021-04-09T18:12:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.385 7f39768e0700 1 -- 172.21.15.32:0/684136086 shutdown_connections 2021-04-09T18:12:41.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.385 7f39768e0700 1 --2- 172.21.15.32:0/684136086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39700d2440 0x7f397010cfd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.385 7f39768e0700 1 --2- 172.21.15.32:0/684136086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39700d6c80 0x7f39700d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.385 7f39768e0700 1 -- 172.21.15.32:0/684136086 >> 172.21.15.32:0/684136086 conn(0x7f39700ce5c0 msgr2=0x7f39700cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:41.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.385 7f39768e0700 1 -- 172.21.15.32:0/684136086 shutdown_connections 2021-04-09T18:12:41.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.385 7f39768e0700 1 -- 172.21.15.32:0/684136086 wait complete. 2021-04-09T18:12:41.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.405 7f39768e0700 1 Processor -- start 2021-04-09T18:12:41.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.405 7f39768e0700 1 -- start start 2021-04-09T18:12:41.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f39768e0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39701a7db0 0x7f39701a81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f39768e0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3970184070 0x7f3970184470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f39768e0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3970184a90 0x7f397008f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f39768e0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39701a8b30 con 0x7f3970184a90 2021-04-09T18:12:41.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f39768e0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39701a7860 con 0x7f39701a7db0 2021-04-09T18:12:41.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f39768e0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39701a87d0 con 0x7f3970184070 2021-04-09T18:12:41.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3974efd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3970184a90 0x7f397008f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3974efd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3970184a90 0x7f397008f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41672/0 (socket says 172.21.15.32:41672) 2021-04-09T18:12:41.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3974efd700 1 -- 172.21.15.32:0/1344797497 learned_addr learned my addr 172.21.15.32:0/1344797497 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:41.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3967fff700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3970184070 0x7f3970184470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f396ffff700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39701a7db0 0x7f39701a81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3974efd700 1 -- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3970184070 msgr2=0x7f3970184470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3974efd700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3970184070 0x7f3970184470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3974efd700 1 -- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39701a7db0 msgr2=0x7f39701a81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3974efd700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39701a7db0 0x7f39701a81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3974efd700 1 -- 172.21.15.32:0/1344797497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397008f810 con 0x7f3970184a90 2021-04-09T18:12:41.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f396ffff700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39701a7db0 0x7f39701a81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:41.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f3974efd700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3970184a90 0x7f397008f1f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f3968001070 tx=0x7f396800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:41.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.406 7f396d7fa700 1 -- 172.21.15.32:0/1344797497 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3968001270 con 0x7f3970184a90 2021-04-09T18:12:41.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.407 7f396d7fa700 1 -- 172.21.15.32:0/1344797497 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3968006df0 con 0x7f3970184a90 2021-04-09T18:12:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.407 7f39768e0700 1 -- 172.21.15.32:0/1344797497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397008f810 con 0x7f3970184a90 2021-04-09T18:12:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.407 7f396d7fa700 1 -- 172.21.15.32:0/1344797497 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39680016e0 con 0x7f3970184a90 2021-04-09T18:12:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.407 7f39768e0700 1 -- 172.21.15.32:0/1344797497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397008f810 con 0x7f3970184a90 2021-04-09T18:12:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.407 7f396d7fa700 1 -- 172.21.15.32:0/1344797497 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f39680093b0 con 0x7f3970184a90 2021-04-09T18:12:41.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.407 7f396d7fa700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f39540188d0 0x7f395401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.407 7f396d7fa700 1 -- 172.21.15.32:0/1344797497 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3968032ea0 con 0x7f3970184a90 2021-04-09T18:12:41.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.407 7f396ffff700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f39540188d0 0x7f395401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.408 7f396ffff700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f39540188d0 0x7f395401add0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.427 7f39768e0700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7f3970062a70 0x7f3970062e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.427 7f39768e0700 1 -- 172.21.15.32:0/1344797497 --> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3970182e70 con 0x7f3970062a70 2021-04-09T18:12:41.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.427 7f3967fff700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7f3970062a70 0x7f3970062e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.427 7f3967fff700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7f3970062a70 0x7f3970062e70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:41.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.428 7f396d7fa700 1 -- 172.21.15.32:0/1344797497 <== osd.1 v2:172.21.15.32:6808/24901 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f3958006a70 con 0x7f3970062a70 2021-04-09T18:12:41.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.435 7f39768e0700 1 -- 172.21.15.32:0/1344797497 --> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3970182e70 con 0x7f3970062a70 2021-04-09T18:12:41.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.435 7f396d7fa700 1 -- 172.21.15.32:0/1344797497 <== osd.1 v2:172.21.15.32:6808/24901 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3958006e10 con 0x7f3970062a70 2021-04-09T18:12:41.575 INFO:teuthology.orchestra.run.smithi032.stdout:55834574854 2021-04-09T18:12:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 -- 172.21.15.32:0/1344797497 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7f3970062a70 msgr2=0x7f3970062e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7f3970062a70 0x7f3970062e70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 -- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f39540188d0 msgr2=0x7f395401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f39540188d0 0x7f395401add0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 -- 172.21.15.32:0/1344797497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3970184a90 msgr2=0x7f397008f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 --2- 172.21.15.32:0/1344797497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3970184a90 0x7f397008f1f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f3968001070 tx=0x7f396800bd40).stop 2021-04-09T18:12:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f3974efd700 1 -- 172.21.15.32:0/1344797497 reap_dead start 2021-04-09T18:12:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 -- 172.21.15.32:0/1344797497 shutdown_connections 2021-04-09T18:12:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 -- 172.21.15.32:0/1344797497 >> 172.21.15.32:0/1344797497 conn(0x7f39701a8f40 msgr2=0x7f3970188050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 -- 172.21.15.32:0/1344797497 shutdown_connections 2021-04-09T18:12:41.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.436 7f39768e0700 1 -- 172.21.15.32:0/1344797497 wait complete. 2021-04-09T18:12:41.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-04-09T18:12:41.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.724 7f001115c700 1 Processor -- start 2021-04-09T18:12:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.724 7f001115c700 1 -- start start 2021-04-09T18:12:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f001115c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f000c10e7e0 0x7f000c0d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f001115c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c0d7290 con 0x7f000c0cffb0 2021-04-09T18:12:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f001115c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c0d2450 con 0x7f000c0d6c90 2021-04-09T18:12:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f001115c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000c0d7090 con 0x7f000c10e7e0 2021-04-09T18:12:41.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f000ad9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f000c0cffb0 0x7f000c0d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51268/0 (socket says 172.21.15.32:51268) 2021-04-09T18:12:41.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f000ad9d700 1 -- 172.21.15.32:0/762500077 learned_addr learned my addr 172.21.15.32:0/762500077 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f000b59e700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f000c10e7e0 0x7f000c0d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068734123 0 0) 0x7efffc000ce0 con 0x7f000c0cffb0 2021-04-09T18:12:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff0002fe0 con 0x7f000c0cffb0 2021-04-09T18:12:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673526081 0 0) 0x7efff8001080 con 0x7f000c0d6c90 2021-04-09T18:12:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff0003210 con 0x7f000c0d6c90 2021-04-09T18:12:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2043835560 0 0) 0x7efffc0010e0 con 0x7f000c0cffb0 2021-04-09T18:12:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f000c10e7e0 msgr2=0x7f000c0d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f000c10e7e0 0x7f000c0d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 >> v1:172.21.15.86:6789/0 conn(0x7f000c0d6c90 legacy=0x7f000c10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c0d05b0 con 0x7f000c0cffb0 2021-04-09T18:12:41.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7efffc001600 con 0x7f000c0cffb0 2021-04-09T18:12:41.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 >> v1:172.21.15.32:6789/0 conn(0x7f000c0cffb0 legacy=0x7f000c0d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efff0003af0 0x7efff0005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff0006780 0x7efff0008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efff0009580 0x7efff000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff00502d0 con 0x7efff0006780 2021-04-09T18:12:41.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff00504b0 con 0x7efff0003af0 2021-04-09T18:12:41.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f0009d9b700 1 -- 172.21.15.32:0/762500077 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff00506d0 con 0x7efff0009580 2021-04-09T18:12:41.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f000a59c700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff0006780 0x7efff0008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f000ad9d700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efff0003af0 0x7efff0005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.725 7f000b59e700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efff0009580 0x7efff000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f000a59c700 1 -- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efff0009580 msgr2=0x7efff000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:41.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f000a59c700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efff0009580 0x7efff000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f000a59c700 1 -- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efff0003af0 msgr2=0x7efff0005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:41.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f000a59c700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efff0003af0 0x7efff0005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f000a59c700 1 -- 172.21.15.32:0/762500077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff0050970 con 0x7efff0006780 2021-04-09T18:12:41.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f000a59c700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff0006780 0x7efff0008c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7efff8006ad0 tx=0x7efff8009b40).ready entity=mon.0 client_cookie=39a14580af2fd8b4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:41.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f0009d9b700 1 -- 172.21.15.32:0/762500077 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff8000c10 con 0x7efff0006780 2021-04-09T18:12:41.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f0009d9b700 1 -- 172.21.15.32:0/762500077 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff800b2e0 con 0x7efff0006780 2021-04-09T18:12:41.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f0009d9b700 1 -- 172.21.15.32:0/762500077 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff800b730 con 0x7efff0006780 2021-04-09T18:12:41.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f001115c700 1 -- 172.21.15.32:0/762500077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff0006780 msgr2=0x7efff0008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f001115c700 1 --2- 172.21.15.32:0/762500077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efff0006780 0x7efff0008c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7efff8006ad0 tx=0x7efff8009b40).stop 2021-04-09T18:12:41.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.726 7f000b59e700 1 -- 172.21.15.32:0/762500077 reap_dead start 2021-04-09T18:12:41.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.727 7f001115c700 1 -- 172.21.15.32:0/762500077 shutdown_connections 2021-04-09T18:12:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.727 7f001115c700 1 -- 172.21.15.32:0/762500077 >> 172.21.15.32:0/762500077 conn(0x7f000c0ce5d0 msgr2=0x7f000c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.727 7f001115c700 1 -- 172.21.15.32:0/762500077 shutdown_connections 2021-04-09T18:12:41.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.727 7f001115c700 1 -- 172.21.15.32:0/762500077 wait complete. 2021-04-09T18:12:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.747 7f001115c700 1 Processor -- start 2021-04-09T18:12:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.747 7f001115c700 1 -- start start 2021-04-09T18:12:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f001115c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f000c08d210 0x7f000c08fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f001115c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f000c0903f0 0x7f000c1a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f001115c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f000c1a6080 0x7f000c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:41.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f001115c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c08d690 con 0x7f000c0903f0 2021-04-09T18:12:41.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f001115c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c0b0fd0 con 0x7f000c1a6080 2021-04-09T18:12:41.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f001115c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000c1a6480 con 0x7f000c08d210 2021-04-09T18:12:41.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f000c0903f0 0x7f000c1a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f000c0903f0 0x7f000c1a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41692/0 (socket says 172.21.15.32:41692) 2021-04-09T18:12:41.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000a59c700 1 -- 172.21.15.32:0/3346526322 learned_addr learned my addr 172.21.15.32:0/3346526322 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000ad9d700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f000c08d210 0x7f000c08fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000b59e700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f000c1a6080 0x7f000c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000a59c700 1 -- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f000c08d210 msgr2=0x7f000c08fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000a59c700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f000c08d210 0x7f000c08fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000a59c700 1 -- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f000c1a6080 msgr2=0x7f000c1a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000a59c700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f000c1a6080 0x7f000c1a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000a59c700 1 -- 172.21.15.32:0/3346526322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c08d8b0 con 0x7f000c0903f0 2021-04-09T18:12:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f000a59c700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f000c0903f0 0x7f000c1a5a60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efff8009670 tx=0x7efff800dba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f00037fe700 1 -- 172.21.15.32:0/3346526322 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff8000c10 con 0x7f000c0903f0 2021-04-09T18:12:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f00037fe700 1 -- 172.21.15.32:0/3346526322 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff8009030 con 0x7f000c0903f0 2021-04-09T18:12:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f00037fe700 1 -- 172.21.15.32:0/3346526322 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff8002210 con 0x7f000c0903f0 2021-04-09T18:12:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f001115c700 1 -- 172.21.15.32:0/3346526322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000c08d8b0 con 0x7f000c0903f0 2021-04-09T18:12:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.748 7f001115c700 1 -- 172.21.15.32:0/3346526322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c08d8b0 con 0x7f000c0903f0 2021-04-09T18:12:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.749 7f00037fe700 1 -- 172.21.15.32:0/3346526322 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efff8007030 con 0x7f000c0903f0 2021-04-09T18:12:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.749 7f00037fe700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7effe80188c0 0x7effe801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.749 7f00037fe700 1 -- 172.21.15.32:0/3346526322 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efff80349b0 con 0x7f000c0903f0 2021-04-09T18:12:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.749 7f000ad9d700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7effe80188c0 0x7effe801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.750 7f000ad9d700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7effe80188c0 0x7effe801adc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.769 7f001115c700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x7f000c184080 0x7f000c184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.769 7f001115c700 1 -- 172.21.15.32:0/3346526322 --> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f000c0b0fd0 con 0x7f000c184080 2021-04-09T18:12:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.769 7f000b59e700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x7f000c184080 0x7f000c184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.769 7f000b59e700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x7f000c184080 0x7f000c184480 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:42.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.769 7f00037fe700 1 -- 172.21.15.32:0/3346526322 <== osd.2 v2:172.21.15.32:6816/24934 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f00040056b0 con 0x7f000c184080 2021-04-09T18:12:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.776 7f001115c700 1 -- 172.21.15.32:0/3346526322 --> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f000c0b0fd0 con 0x7f000c184080 2021-04-09T18:12:42.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.777 7f00037fe700 1 -- 172.21.15.32:0/3346526322 <== osd.2 v2:172.21.15.32:6816/24934 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0004006930 con 0x7f000c184080 2021-04-09T18:12:42.070 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2021-04-09T18:12:42.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 -- 172.21.15.32:0/3346526322 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x7f000c184080 msgr2=0x7f000c184480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x7f000c184080 0x7f000c184480 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:42.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 -- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7effe80188c0 msgr2=0x7effe801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7effe80188c0 0x7effe801adc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:42.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 -- 172.21.15.32:0/3346526322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f000c0903f0 msgr2=0x7f000c1a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 --2- 172.21.15.32:0/3346526322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f000c0903f0 0x7f000c1a5a60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efff8009670 tx=0x7efff800dba0).stop 2021-04-09T18:12:42.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f000b59e700 1 -- 172.21.15.32:0/3346526322 reap_dead start 2021-04-09T18:12:42.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 -- 172.21.15.32:0/3346526322 shutdown_connections 2021-04-09T18:12:42.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 -- 172.21.15.32:0/3346526322 >> 172.21.15.32:0/3346526322 conn(0x7f000c1a8f50 msgr2=0x7f000c188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:42.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 -- 172.21.15.32:0/3346526322 shutdown_connections 2021-04-09T18:12:42.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:41.778 7f001115c700 1 -- 172.21.15.32:0/3346526322 wait complete. 2021-04-09T18:12:42.153 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-04-09T18:12:42.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.309 7f022fae3700 1 Processor -- start 2021-04-09T18:12:42.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.309 7f022fae3700 1 -- start start 2021-04-09T18:12:42.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022fae3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f022810e7e0 0x7f02280d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022fae3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02280d7290 con 0x7f02280d6c90 2021-04-09T18:12:42.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022fae3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02280d2450 con 0x7f02280cffb0 2021-04-09T18:12:42.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022fae3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02280d7090 con 0x7f022810e7e0 2021-04-09T18:12:42.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022d0fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f02280d6c90 0x7f022810cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51306/0 (socket says 172.21.15.32:51306) 2021-04-09T18:12:42.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022d0fe700 1 -- 172.21.15.32:0/2825677767 learned_addr learned my addr 172.21.15.32:0/2825677767 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:42.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022e100700 1 --2- 172.21.15.32:0/2825677767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f022810e7e0 0x7f02280d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022c8fd700 1 -- 172.21.15.32:0/2825677767 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528764682 0 0) 0x7f0218000ce0 con 0x7f02280d6c90 2021-04-09T18:12:42.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022c8fd700 1 -- 172.21.15.32:0/2825677767 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0214002fe0 con 0x7f02280d6c90 2021-04-09T18:12:42.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022c8fd700 1 -- 172.21.15.32:0/2825677767 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124486560 0 0) 0x7f0224000c60 con 0x7f02280cffb0 2021-04-09T18:12:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022c8fd700 1 -- 172.21.15.32:0/2825677767 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02140042c0 con 0x7f02280cffb0 2021-04-09T18:12:42.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022e100700 1 -- 172.21.15.32:0/2825677767 >> v1:172.21.15.86:6789/0 conn(0x7f02280cffb0 legacy=0x7f02280d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022e100700 1 -- 172.21.15.32:0/2825677767 >> v1:172.21.15.32:6789/0 conn(0x7f02280d6c90 legacy=0x7f022810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022c8fd700 1 -- 172.21.15.32:0/2825677767 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2132010298 0 0) 0x7f02180010e0 con 0x7f02280d6c90 2021-04-09T18:12:42.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022e100700 1 -- 172.21.15.32:0/2825677767 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02281120d0 con 0x7f022810e7e0 2021-04-09T18:12:42.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.310 7f022e100700 1 --2- 172.21.15.32:0/2825677767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f022810e7e0 0x7f02280d8000 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f0220006e50 tx=0x7f0220009df0).ready entity=mon.2 client_cookie=7ac1a0d5114742bb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:42.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.311 7f022c8fd700 1 -- 172.21.15.32:0/2825677767 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f022000b510 con 0x7f022810e7e0 2021-04-09T18:12:42.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.311 7f022c8fd700 1 -- 172.21.15.32:0/2825677767 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0220003030 con 0x7f022810e7e0 2021-04-09T18:12:42.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.311 7f022c8fd700 1 -- 172.21.15.32:0/2825677767 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f022000b7d0 con 0x7f022810e7e0 2021-04-09T18:12:42.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.311 7f022fae3700 1 -- 172.21.15.32:0/2825677767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f022810e7e0 msgr2=0x7f02280d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.311 7f022fae3700 1 --2- 172.21.15.32:0/2825677767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f022810e7e0 0x7f02280d8000 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f0220006e50 tx=0x7f0220009df0).stop 2021-04-09T18:12:42.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.311 7f022fae3700 1 -- 172.21.15.32:0/2825677767 shutdown_connections 2021-04-09T18:12:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.311 7f022fae3700 1 --2- 172.21.15.32:0/2825677767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f022810e7e0 0x7f02280d8000 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.311 7f022fae3700 1 -- 172.21.15.32:0/2825677767 >> 172.21.15.32:0/2825677767 conn(0x7f02280ce5d0 msgr2=0x7f02280cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.312 7f022fae3700 1 -- 172.21.15.32:0/2825677767 shutdown_connections 2021-04-09T18:12:42.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.312 7f022fae3700 1 -- 172.21.15.32:0/2825677767 wait complete. 2021-04-09T18:12:42.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022fae3700 1 Processor -- start 2021-04-09T18:12:42.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022fae3700 1 -- start start 2021-04-09T18:12:42.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022fae3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0228184020 0x7f0228197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022fae3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0228184420 0x7f02281aa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022fae3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0228184820 0x7f02281aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022fae3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0228184c20 con 0x7f0228184420 2021-04-09T18:12:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022fae3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02281aa260 con 0x7f0228184020 2021-04-09T18:12:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022fae3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02281ab3f0 con 0x7f0228184820 2021-04-09T18:12:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022d0fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0228184420 0x7f02281aa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022d0fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0228184420 0x7f02281aa7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41720/0 (socket says 172.21.15.32:41720) 2021-04-09T18:12:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022d0fe700 1 -- 172.21.15.32:0/4039712186 learned_addr learned my addr 172.21.15.32:0/4039712186 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022e100700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0228184820 0x7f02281aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.332 7f022d8ff700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0228184020 0x7f0228197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f022d0fe700 1 -- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0228184820 msgr2=0x7f02281aadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f022d0fe700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0228184820 0x7f02281aadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:42.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f022d0fe700 1 -- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0228184020 msgr2=0x7f0228197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f022d0fe700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0228184020 0x7f0228197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:42.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f022d0fe700 1 -- 172.21.15.32:0/4039712186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02281a9ef0 con 0x7f0228184420 2021-04-09T18:12:42.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f022d0fe700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0228184420 0x7f02281aa7b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0218000f90 tx=0x7f021800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f021e7fc700 1 -- 172.21.15.32:0/4039712186 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0218001a30 con 0x7f0228184420 2021-04-09T18:12:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f021e7fc700 1 -- 172.21.15.32:0/4039712186 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02180043a0 con 0x7f0228184420 2021-04-09T18:12:42.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f021e7fc700 1 -- 172.21.15.32:0/4039712186 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02180047f0 con 0x7f0228184420 2021-04-09T18:12:42.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f022fae3700 1 -- 172.21.15.32:0/4039712186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02281a9ef0 con 0x7f0228184420 2021-04-09T18:12:42.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.333 7f022fae3700 1 -- 172.21.15.32:0/4039712186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02281a9ef0 con 0x7f0228184420 2021-04-09T18:12:42.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.334 7f021e7fc700 1 -- 172.21.15.32:0/4039712186 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0218004580 con 0x7f0228184420 2021-04-09T18:12:42.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.334 7f021e7fc700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02100188d0 0x7f021001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.334 7f021e7fc700 1 -- 172.21.15.32:0/4039712186 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f02180329e0 con 0x7f0228184420 2021-04-09T18:12:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.334 7f022d8ff700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02100188d0 0x7f021001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.334 7f022d8ff700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02100188d0 0x7f021001add0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:42.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.354 7f022fae3700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f0228062a70 0x7f0228062e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.354 7f022fae3700 1 -- 172.21.15.32:0/4039712186 --> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0228184c20 con 0x7f0228062a70 2021-04-09T18:12:42.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.354 7f022e100700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f0228062a70 0x7f0228062e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.354 7f022e100700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f0228062a70 0x7f0228062e70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.354 7f021e7fc700 1 -- 172.21.15.32:0/4039712186 <== osd.3 v2:172.21.15.32:6824/24942 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f0220009390 con 0x7f0228062a70 2021-04-09T18:12:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.361 7f022fae3700 1 -- 172.21.15.32:0/4039712186 --> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0228182e20 con 0x7f0228062a70 2021-04-09T18:12:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.362 7f021e7fc700 1 -- 172.21.15.32:0/4039712186 <== osd.3 v2:172.21.15.32:6824/24942 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0220009050 con 0x7f0228062a70 2021-04-09T18:12:42.669 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2021-04-09T18:12:42.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 -- 172.21.15.32:0/4039712186 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f0228062a70 msgr2=0x7f0228062e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f0228062a70 0x7f0228062e70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:42.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 -- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02100188d0 msgr2=0x7f021001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02100188d0 0x7f021001add0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:42.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 -- 172.21.15.32:0/4039712186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0228184420 msgr2=0x7f02281aa7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 --2- 172.21.15.32:0/4039712186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0228184420 0x7f02281aa7b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0218000f90 tx=0x7f021800a940).stop 2021-04-09T18:12:42.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022e100700 1 -- 172.21.15.32:0/4039712186 reap_dead start 2021-04-09T18:12:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 -- 172.21.15.32:0/4039712186 shutdown_connections 2021-04-09T18:12:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 -- 172.21.15.32:0/4039712186 >> 172.21.15.32:0/4039712186 conn(0x7f02281838a0 msgr2=0x7f02281a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 -- 172.21.15.32:0/4039712186 shutdown_connections 2021-04-09T18:12:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.363 7f022fae3700 1 -- 172.21.15.32:0/4039712186 wait complete. 2021-04-09T18:12:42.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2021-04-09T18:12:42.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.926 7fccabdae700 1 Processor -- start 2021-04-09T18:12:42.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.926 7fccabdae700 1 -- start start 2021-04-09T18:12:42.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccabdae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca40cffb0 0x7fcca40d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccabdae700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcca40d6c90 0x7fcca410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:42.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccabdae700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca40d7290 con 0x7fcca40d7810 2021-04-09T18:12:42.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccabdae700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca40d7090 con 0x7fcca40d6c90 2021-04-09T18:12:42.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccabdae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40d8590 con 0x7fcca40cffb0 2021-04-09T18:12:42.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccaa3cb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcca40d6c90 0x7fcca410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fcca93c9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcca40d7810 0x7fcca40d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51324/0 (socket says 172.21.15.32:51324) 2021-04-09T18:12:42.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fcca93c9700 1 -- 172.21.15.32:0/3013450797 learned_addr learned my addr 172.21.15.32:0/3013450797 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:42.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fcca9bca700 1 --2- 172.21.15.32:0/3013450797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca40cffb0 0x7fcca40d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:42.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccaa3cb700 1 -- 172.21.15.32:0/3013450797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca40cffb0 msgr2=0x7fcca40d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:42.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccaa3cb700 1 --2- 172.21.15.32:0/3013450797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca40cffb0 0x7fcca40d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:42.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccaa3cb700 1 -- 172.21.15.32:0/3013450797 >> v1:172.21.15.32:6789/0 conn(0x7fcca40d7810 legacy=0x7fcca40d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:42.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fcca8bc8700 1 -- 172.21.15.32:0/3013450797 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824467557 0 0) 0x7fcc940009b0 con 0x7fcca40d7810 2021-04-09T18:12:42.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fccaa3cb700 1 -- 172.21.15.32:0/3013450797 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40d91f0 con 0x7fcca40d6c90 2021-04-09T18:12:42.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.927 7fcca9bca700 1 --2- 172.21.15.32:0/3013450797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca40cffb0 0x7fcca40d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:42.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.928 7fccaa3cb700 1 --2- 172.21.15.32:0/3013450797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca40d6c90 0x7fcca410cfe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fcc9c006e30 tx=0x7fcc9c009e60).ready entity=mon.0 client_cookie=85331fbbaa0672df server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:42.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.928 7fcca8bc8700 1 -- 172.21.15.32:0/3013450797 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc9c006a50 con 0x7fcca40d6c90 2021-04-09T18:12:43.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.928 7fcca8bc8700 1 -- 172.21.15.32:0/3013450797 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c00b300 con 0x7fcca40d6c90 2021-04-09T18:12:43.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.928 7fcca8bc8700 1 -- 172.21.15.32:0/3013450797 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc9c00b750 con 0x7fcca40d6c90 2021-04-09T18:12:43.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.928 7fccabdae700 1 -- 172.21.15.32:0/3013450797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca40d6c90 msgr2=0x7fcca410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.928 7fccabdae700 1 --2- 172.21.15.32:0/3013450797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca40d6c90 0x7fcca410cfe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fcc9c006e30 tx=0x7fcc9c009e60).stop 2021-04-09T18:12:43.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.929 7fccabdae700 1 -- 172.21.15.32:0/3013450797 shutdown_connections 2021-04-09T18:12:43.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.929 7fccabdae700 1 --2- 172.21.15.32:0/3013450797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca40d6c90 0x7fcca410cfe0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.929 7fccabdae700 1 --2- 172.21.15.32:0/3013450797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca40cffb0 0x7fcca40d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.929 7fccabdae700 1 -- 172.21.15.32:0/3013450797 >> 172.21.15.32:0/3013450797 conn(0x7fcca40ce5d0 msgr2=0x7fcca40cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:43.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.929 7fccabdae700 1 -- 172.21.15.32:0/3013450797 shutdown_connections 2021-04-09T18:12:43.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.929 7fccabdae700 1 -- 172.21.15.32:0/3013450797 wait complete. 2021-04-09T18:12:43.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.949 7fccabdae700 1 Processor -- start 2021-04-09T18:12:43.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.949 7fccabdae700 1 -- start start 2021-04-09T18:12:43.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.949 7fccabdae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca4188cf0 0x7fcca418d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.949 7fccabdae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca41890f0 0x7fcca418dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.949 7fccabdae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcca418c180 0x7fcca418c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.949 7fccabdae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca418d0b0 con 0x7fcca4188cf0 2021-04-09T18:12:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.949 7fccabdae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca418cba0 con 0x7fcca418c180 2021-04-09T18:12:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.949 7fccabdae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca418cd80 con 0x7fcca41890f0 2021-04-09T18:12:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.949 7fcca9bca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca4188cf0 0x7fcca418d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcca9bca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca4188cf0 0x7fcca418d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41740/0 (socket says 172.21.15.32:41740) 2021-04-09T18:12:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcca9bca700 1 -- 172.21.15.32:0/2403568298 learned_addr learned my addr 172.21.15.32:0/2403568298 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcca93c9700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca41890f0 0x7fcca418dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fccaa3cb700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcca418c180 0x7fcca418c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcca9bca700 1 -- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca41890f0 msgr2=0x7fcca418dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcca9bca700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcca41890f0 0x7fcca418dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcca9bca700 1 -- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcca418c180 msgr2=0x7fcca418c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcca9bca700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcca418c180 0x7fcca418c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcca9bca700 1 -- 172.21.15.32:0/2403568298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca4183d10 con 0x7fcca4188cf0 2021-04-09T18:12:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcca9bca700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca4188cf0 0x7fcca418d630 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fcca0006f20 tx=0x7fcca000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcc9a7fc700 1 -- 172.21.15.32:0/2403568298 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcca000bd80 con 0x7fcca4188cf0 2021-04-09T18:12:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcc9a7fc700 1 -- 172.21.15.32:0/2403568298 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca0000c90 con 0x7fcca4188cf0 2021-04-09T18:12:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fcc9a7fc700 1 -- 172.21.15.32:0/2403568298 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcca00022b0 con 0x7fcca4188cf0 2021-04-09T18:12:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fccabdae700 1 -- 172.21.15.32:0/2403568298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca4183d10 con 0x7fcca4188cf0 2021-04-09T18:12:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.950 7fccabdae700 1 -- 172.21.15.32:0/2403568298 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca4183d10 con 0x7fcca4188cf0 2021-04-09T18:12:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.951 7fcc9a7fc700 1 -- 172.21.15.32:0/2403568298 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcca000a000 con 0x7fcca4188cf0 2021-04-09T18:12:43.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.951 7fcc9a7fc700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc8c0188c0 0x7fcc8c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.951 7fcc9a7fc700 1 -- 172.21.15.32:0/2403568298 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcca0032980 con 0x7fcca4188cf0 2021-04-09T18:12:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.951 7fcca93c9700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc8c0188c0 0x7fcc8c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.952 7fcca93c9700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc8c0188c0 0x7fcc8c01adc0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.971 7fccabdae700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x7fcca41a2eb0 0x7fcca4062a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.971 7fccabdae700 1 -- 172.21.15.32:0/2403568298 --> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcca4182e20 con 0x7fcca41a2eb0 2021-04-09T18:12:43.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.971 7fccaa3cb700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x7fcca41a2eb0 0x7fcca4062a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.971 7fccaa3cb700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x7fcca41a2eb0 0x7fcca4062a70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.972 7fcc9a7fc700 1 -- 172.21.15.32:0/2403568298 <== osd.4 v2:172.21.15.86:6802/25232 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fcc9c0093d0 con 0x7fcca41a2eb0 2021-04-09T18:12:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.981 7fccabdae700 1 -- 172.21.15.32:0/2403568298 --> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcca4182e20 con 0x7fcca41a2eb0 2021-04-09T18:12:43.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.981 7fcc9a7fc700 1 -- 172.21.15.32:0/2403568298 <== osd.4 v2:172.21.15.86:6802/25232 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcc9c009380 con 0x7fcca41a2eb0 2021-04-09T18:12:43.102 INFO:teuthology.orchestra.run.smithi032.stdout:55834574854 2021-04-09T18:12:43.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 -- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x7fcca41a2eb0 msgr2=0x7fcca4062a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x7fcca41a2eb0 0x7fcca4062a70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 -- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc8c0188c0 msgr2=0x7fcc8c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc8c0188c0 0x7fcc8c01adc0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 -- 172.21.15.32:0/2403568298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca4188cf0 msgr2=0x7fcca418d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 --2- 172.21.15.32:0/2403568298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca4188cf0 0x7fcca418d630 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fcca0006f20 tx=0x7fcca000b990).stop 2021-04-09T18:12:43.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccaa3cb700 1 -- 172.21.15.32:0/2403568298 reap_dead start 2021-04-09T18:12:43.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 -- 172.21.15.32:0/2403568298 shutdown_connections 2021-04-09T18:12:43.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 -- 172.21.15.32:0/2403568298 >> 172.21.15.32:0/2403568298 conn(0x7fcca41a36a0 msgr2=0x7fcca4185120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:43.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 -- 172.21.15.32:0/2403568298 shutdown_connections 2021-04-09T18:12:43.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:42.982 7fccabdae700 1 -- 172.21.15.32:0/2403568298 wait complete. 2021-04-09T18:12:43.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2021-04-09T18:12:43.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.257 7f17471c7700 1 Processor -- start 2021-04-09T18:12:43.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.258 7f17471c7700 1 -- start start 2021-04-09T18:12:43.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.258 7f17471c7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f17400cffb0 0x7f17400d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.258 7f17471c7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17400d6c90 0x7f174010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.258 7f17471c7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17400d7290 con 0x7f17400d7810 2021-04-09T18:12:43.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.258 7f17471c7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17400d7090 con 0x7f17400cffb0 2021-04-09T18:12:43.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.258 7f17471c7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17400d91f0 con 0x7f17400d6c90 2021-04-09T18:12:43.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.258 7f173ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f17400d7810 0x7f17400d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51340/0 (socket says 172.21.15.32:51340) 2021-04-09T18:12:43.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.258 7f173ffff700 1 -- 172.21.15.32:0/1630568463 learned_addr learned my addr 172.21.15.32:0/1630568463 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f17457e4700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17400d6c90 0x7f174010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f1744fe3700 1 --2- 172.21.15.32:0/1630568463 >> v2:172.21.15.86:3300/0 conn(0x7f17400cffb0 0x7f17400d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897725004 0 0) 0x7f17340009b0 con 0x7f17400d7810 2021-04-09T18:12:43.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1728002fe0 con 0x7f17400d7810 2021-04-09T18:12:43.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3258583996 0 0) 0x7f1734000e20 con 0x7f17400d7810 2021-04-09T18:12:43.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17400d6c90 msgr2=0x7f174010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:43.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17400d6c90 0x7f174010cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 >> v2:172.21.15.86:3300/0 conn(0x7f17400cffb0 msgr2=0x7f17400d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:43.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 --2- 172.21.15.32:0/1630568463 >> v2:172.21.15.86:3300/0 conn(0x7f17400cffb0 0x7f17400d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17400d8400 con 0x7f17400d7810 2021-04-09T18:12:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f1734001530 con 0x7f17400d7810 2021-04-09T18:12:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f1744fe3700 1 --2- 172.21.15.32:0/1630568463 >> v2:172.21.15.86:3300/0 conn(0x7f17400cffb0 0x7f17400d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 >> v1:172.21.15.32:6789/0 conn(0x7f17400d7810 legacy=0x7f17400d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1728047aa0 0x7f1728049f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f172804a790 0x7f172804cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f172804d520 0x7f172804fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17280502f0 con 0x7f1728047aa0 2021-04-09T18:12:43.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1728050510 con 0x7f172804d520 2021-04-09T18:12:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1728050730 con 0x7f172804a790 2021-04-09T18:12:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f1744fe3700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1728047aa0 0x7f1728049f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f173ffff700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f172804a790 0x7f172804cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f17457e4700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f172804d520 0x7f172804fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f1744fe3700 1 -- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f172804a790 msgr2=0x7f172804cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:43.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f1744fe3700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f172804a790 0x7f172804cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f1744fe3700 1 -- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f172804d520 msgr2=0x7f172804fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:43.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f1744fe3700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f172804d520 0x7f172804fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.259 7f1744fe3700 1 -- 172.21.15.32:0/1630568463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17280509d0 con 0x7f1728047aa0 2021-04-09T18:12:43.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f1744fe3700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1728047aa0 0x7f1728049f10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1730006b00 tx=0x7f173000ce20).ready entity=mon.0 client_cookie=270e1a9f01b3ba52 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1730009310 con 0x7f1728047aa0 2021-04-09T18:12:43.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17300068d0 con 0x7f1728047aa0 2021-04-09T18:12:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f173f7fe700 1 -- 172.21.15.32:0/1630568463 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1730009890 con 0x7f1728047aa0 2021-04-09T18:12:43.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f17471c7700 1 -- 172.21.15.32:0/1630568463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1728047aa0 msgr2=0x7f1728049f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f17471c7700 1 --2- 172.21.15.32:0/1630568463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1728047aa0 0x7f1728049f10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1730006b00 tx=0x7f173000ce20).stop 2021-04-09T18:12:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f17457e4700 1 -- 172.21.15.32:0/1630568463 reap_dead start 2021-04-09T18:12:43.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f17471c7700 1 -- 172.21.15.32:0/1630568463 shutdown_connections 2021-04-09T18:12:43.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f17471c7700 1 -- 172.21.15.32:0/1630568463 >> 172.21.15.32:0/1630568463 conn(0x7f17400ce5d0 msgr2=0x7f17400cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:43.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f17471c7700 1 -- 172.21.15.32:0/1630568463 shutdown_connections 2021-04-09T18:12:43.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.260 7f17471c7700 1 -- 172.21.15.32:0/1630568463 wait complete. 2021-04-09T18:12:43.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f17471c7700 1 Processor -- start 2021-04-09T18:12:43.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f17471c7700 1 -- start start 2021-04-09T18:12:43.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f17471c7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1740184ec0 0x7f1740198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f17471c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17401abf00 0x7f17401accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f17471c7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17401ac300 0x7f1740184020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f17471c7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17401ac750 con 0x7f17401abf00 2021-04-09T18:12:43.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f17471c7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17401ad2f0 con 0x7f17401ac300 2021-04-09T18:12:43.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f17471c7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1740188cb0 con 0x7f1740184ec0 2021-04-09T18:12:43.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f173ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17401abf00 0x7f17401accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f173ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17401abf00 0x7f17401accd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41758/0 (socket says 172.21.15.32:41758) 2021-04-09T18:12:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.281 7f173ffff700 1 -- 172.21.15.32:0/2960053641 learned_addr learned my addr 172.21.15.32:0/2960053641 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:43.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f1744fe3700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1740184ec0 0x7f1740198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f17457e4700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17401ac300 0x7f1740184020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f173ffff700 1 -- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1740184ec0 msgr2=0x7f1740198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f173ffff700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1740184ec0 0x7f1740198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f173ffff700 1 -- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17401ac300 msgr2=0x7f1740184020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f173ffff700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17401ac300 0x7f1740184020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f173ffff700 1 -- 172.21.15.32:0/2960053641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1740184640 con 0x7f17401abf00 2021-04-09T18:12:43.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f1744fe3700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1740184ec0 0x7f1740198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:43.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f17457e4700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17401ac300 0x7f1740184020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:43.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f173ffff700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17401abf00 0x7f17401accd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f17340016b0 tx=0x7f173400a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f173d7fa700 1 -- 172.21.15.32:0/2960053641 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1734001310 con 0x7f17401abf00 2021-04-09T18:12:43.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f173d7fa700 1 -- 172.21.15.32:0/2960053641 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173400ad60 con 0x7f17401abf00 2021-04-09T18:12:43.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f17471c7700 1 -- 172.21.15.32:0/2960053641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1740184640 con 0x7f17401abf00 2021-04-09T18:12:43.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f173d7fa700 1 -- 172.21.15.32:0/2960053641 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17340043c0 con 0x7f17401abf00 2021-04-09T18:12:43.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.282 7f17471c7700 1 -- 172.21.15.32:0/2960053641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1740184640 con 0x7f17401abf00 2021-04-09T18:12:43.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.283 7f173d7fa700 1 -- 172.21.15.32:0/2960053641 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1734004140 con 0x7f17401abf00 2021-04-09T18:12:43.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.283 7f173d7fa700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1720018930 0x7f172001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.283 7f173d7fa700 1 -- 172.21.15.32:0/2960053641 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1734032a10 con 0x7f17401abf00 2021-04-09T18:12:43.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.283 7f1744fe3700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1720018930 0x7f172001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.283 7f1744fe3700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1720018930 0x7f172001ae10 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.302 7f17471c7700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f1740062a70 0x7f1740062e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.302 7f17471c7700 1 -- 172.21.15.32:0/2960053641 --> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f17401ac750 con 0x7f1740062a70 2021-04-09T18:12:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.302 7f17457e4700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f1740062a70 0x7f1740062e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.303 7f17457e4700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f1740062a70 0x7f1740062e70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.303 7f173d7fa700 1 -- 172.21.15.32:0/2960053641 <== osd.5 v2:172.21.15.86:6810/25237 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f1738006bf0 con 0x7f1740062a70 2021-04-09T18:12:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.312 7f17471c7700 1 -- 172.21.15.32:0/2960053641 --> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f17401ad2f0 con 0x7f1740062a70 2021-04-09T18:12:43.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.312 7f173d7fa700 1 -- 172.21.15.32:0/2960053641 <== osd.5 v2:172.21.15.86:6810/25237 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1738009140 con 0x7f1740062a70 2021-04-09T18:12:43.655 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2021-04-09T18:12:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.313 7f17471c7700 1 -- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f1740062a70 msgr2=0x7f1740062e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.313 7f17471c7700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f1740062a70 0x7f1740062e70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.313 7f17471c7700 1 -- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1720018930 msgr2=0x7f172001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.314 7f17471c7700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1720018930 0x7f172001ae10 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.314 7f17471c7700 1 -- 172.21.15.32:0/2960053641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17401abf00 msgr2=0x7f17401accd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.314 7f17471c7700 1 --2- 172.21.15.32:0/2960053641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17401abf00 0x7f17401accd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f17340016b0 tx=0x7f173400a970).stop 2021-04-09T18:12:43.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.314 7f17457e4700 1 -- 172.21.15.32:0/2960053641 reap_dead start 2021-04-09T18:12:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.314 7f17471c7700 1 -- 172.21.15.32:0/2960053641 shutdown_connections 2021-04-09T18:12:43.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.314 7f17471c7700 1 -- 172.21.15.32:0/2960053641 >> 172.21.15.32:0/2960053641 conn(0x7f17401838a0 msgr2=0x7f17401a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.314 7f17471c7700 1 -- 172.21.15.32:0/2960053641 shutdown_connections 2021-04-09T18:12:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.314 7f17471c7700 1 -- 172.21.15.32:0/2960053641 wait complete. 2021-04-09T18:12:43.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2021-04-09T18:12:43.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.880 7eff0b572700 1 Processor -- start 2021-04-09T18:12:43.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.880 7eff0b572700 1 -- start start 2021-04-09T18:12:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0b572700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7eff040cffb0 0x7eff040d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0b572700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7eff040d6c90 0x7eff0410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0b572700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff040d7290 con 0x7eff040d7810 2021-04-09T18:12:43.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0b572700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff040d7090 con 0x7eff040cffb0 2021-04-09T18:12:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0b572700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff040d91f0 con 0x7eff040d6c90 2021-04-09T18:12:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0938e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7eff040cffb0 0x7eff040d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0938e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7eff040cffb0 0x7eff040d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41768/0 (socket says 172.21.15.32:41768) 2021-04-09T18:12:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0938e700 1 -- 172.21.15.32:0/1168630799 learned_addr learned my addr 172.21.15.32:0/1168630799 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff09b8f700 1 --2- 172.21.15.32:0/1168630799 >> v2:172.21.15.86:3300/0 conn(0x7eff040d6c90 0x7eff0410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0938e700 1 -- 172.21.15.32:0/1168630799 >> v2:172.21.15.86:3300/0 conn(0x7eff040d6c90 msgr2=0x7eff0410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0938e700 1 --2- 172.21.15.32:0/1168630799 >> v2:172.21.15.86:3300/0 conn(0x7eff040d6c90 0x7eff0410cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0938e700 1 -- 172.21.15.32:0/1168630799 >> v1:172.21.15.86:6789/0 conn(0x7eff040d7810 legacy=0x7eff040d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.881 7eff0938e700 1 -- 172.21.15.32:0/1168630799 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff040d8400 con 0x7eff040cffb0 2021-04-09T18:12:43.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.882 7eff0938e700 1 --2- 172.21.15.32:0/1168630799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff040cffb0 0x7eff040d2450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7eff00006700 tx=0x7eff00009b80).ready entity=mon.0 client_cookie=1367808592afcca5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.882 7efefbfff700 1 -- 172.21.15.32:0/1168630799 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff0000c0c0 con 0x7eff040cffb0 2021-04-09T18:12:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.882 7efefbfff700 1 -- 172.21.15.32:0/1168630799 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0000c440 con 0x7eff040cffb0 2021-04-09T18:12:43.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.882 7efefbfff700 1 -- 172.21.15.32:0/1168630799 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff0000c920 con 0x7eff040cffb0 2021-04-09T18:12:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.882 7eff0b572700 1 -- 172.21.15.32:0/1168630799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff040cffb0 msgr2=0x7eff040d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.882 7eff0b572700 1 --2- 172.21.15.32:0/1168630799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff040cffb0 0x7eff040d2450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7eff00006700 tx=0x7eff00009b80).stop 2021-04-09T18:12:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.882 7eff0b572700 1 -- 172.21.15.32:0/1168630799 shutdown_connections 2021-04-09T18:12:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.882 7eff0b572700 1 --2- 172.21.15.32:0/1168630799 >> v2:172.21.15.86:3300/0 conn(0x7eff040d6c90 0x7eff0410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.882 7eff0b572700 1 --2- 172.21.15.32:0/1168630799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff040cffb0 0x7eff040d2450 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.883 7eff0b572700 1 -- 172.21.15.32:0/1168630799 >> 172.21.15.32:0/1168630799 conn(0x7eff040ce5d0 msgr2=0x7eff040cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.883 7eff0b572700 1 -- 172.21.15.32:0/1168630799 shutdown_connections 2021-04-09T18:12:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.883 7eff0b572700 1 -- 172.21.15.32:0/1168630799 wait complete. 2021-04-09T18:12:43.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.903 7eff0b572700 1 Processor -- start 2021-04-09T18:12:43.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.903 7eff0b572700 1 -- start start 2021-04-09T18:12:43.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0b572700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff0408d210 0x7eff0408fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0b572700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff040903f0 0x7eff041a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0b572700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff041a6080 0x7eff041a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0b572700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0408d690 con 0x7eff0408d210 2021-04-09T18:12:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0b572700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff040b0fd0 con 0x7eff041a6080 2021-04-09T18:12:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0b572700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff041a6480 con 0x7eff040903f0 2021-04-09T18:12:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0938e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff0408d210 0x7eff0408fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0938e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff0408d210 0x7eff0408fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41774/0 (socket says 172.21.15.32:41774) 2021-04-09T18:12:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0938e700 1 -- 172.21.15.32:0/201381504 learned_addr learned my addr 172.21.15.32:0/201381504 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff09b8f700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff041a6080 0x7eff041a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff08b8d700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff040903f0 0x7eff041a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0938e700 1 -- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff040903f0 msgr2=0x7eff041a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0938e700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff040903f0 0x7eff041a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0938e700 1 -- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff041a6080 msgr2=0x7eff041a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0938e700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff041a6080 0x7eff041a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0938e700 1 -- 172.21.15.32:0/201381504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0408d8b0 con 0x7eff0408d210 2021-04-09T18:12:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0938e700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff0408d210 0x7eff0408fdd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7eff0000b580 tx=0x7eff0000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7efef9ffb700 1 -- 172.21.15.32:0/201381504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff0000bd10 con 0x7eff0408d210 2021-04-09T18:12:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7efef9ffb700 1 -- 172.21.15.32:0/201381504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff00000c90 con 0x7eff0408d210 2021-04-09T18:12:43.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0b572700 1 -- 172.21.15.32:0/201381504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0408d8b0 con 0x7eff0408d210 2021-04-09T18:12:43.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7efef9ffb700 1 -- 172.21.15.32:0/201381504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff00002220 con 0x7eff0408d210 2021-04-09T18:12:43.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.904 7eff0b572700 1 -- 172.21.15.32:0/201381504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0408d8b0 con 0x7eff0408d210 2021-04-09T18:12:43.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.905 7efef9ffb700 1 -- 172.21.15.32:0/201381504 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7eff00009f90 con 0x7eff0408d210 2021-04-09T18:12:43.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.905 7efef9ffb700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018930 0x7efee401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.905 7efef9ffb700 1 -- 172.21.15.32:0/201381504 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7eff000329f0 con 0x7eff0408d210 2021-04-09T18:12:43.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.905 7eff08b8d700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018930 0x7efee401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.906 7eff08b8d700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018930 0x7efee401ae10 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.925 7eff0b572700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7eff04062a70 0x7eff04062e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:43.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.925 7eff0b572700 1 -- 172.21.15.32:0/201381504 --> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff040b0fd0 con 0x7eff04062a70 2021-04-09T18:12:43.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.925 7eff09b8f700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7eff04062a70 0x7eff04062e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:43.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.925 7eff09b8f700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7eff04062a70 0x7eff04062e70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:43.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.926 7efef9ffb700 1 -- 172.21.15.32:0/201381504 <== osd.6 v2:172.21.15.86:6818/25238 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7efefc00c0c0 con 0x7eff04062a70 2021-04-09T18:12:43.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.935 7eff0b572700 1 -- 172.21.15.32:0/201381504 --> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7eff041a52b0 con 0x7eff04062a70 2021-04-09T18:12:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.935 7efef9ffb700 1 -- 172.21.15.32:0/201381504 <== osd.6 v2:172.21.15.86:6818/25238 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efefc006e10 con 0x7eff04062a70 2021-04-09T18:12:43.998 INFO:teuthology.orchestra.run.smithi032.stdout:55834574853 2021-04-09T18:12:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 -- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7eff04062a70 msgr2=0x7eff04062e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7eff04062a70 0x7eff04062e70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 -- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018930 msgr2=0x7efee401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018930 0x7efee401ae10 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 -- 172.21.15.32:0/201381504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff0408d210 msgr2=0x7eff0408fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 --2- 172.21.15.32:0/201381504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff0408d210 0x7eff0408fdd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7eff0000b580 tx=0x7eff0000b920).stop 2021-04-09T18:12:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff09b8f700 1 -- 172.21.15.32:0/201381504 reap_dead start 2021-04-09T18:12:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 -- 172.21.15.32:0/201381504 shutdown_connections 2021-04-09T18:12:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 -- 172.21.15.32:0/201381504 >> 172.21.15.32:0/201381504 conn(0x7eff041a8f50 msgr2=0x7eff04188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:44.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 -- 172.21.15.32:0/201381504 shutdown_connections 2021-04-09T18:12:44.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:43.936 7eff0b572700 1 -- 172.21.15.32:0/201381504 wait complete. 2021-04-09T18:12:44.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2021-04-09T18:12:44.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.144 7f36c3167700 1 Processor -- start 2021-04-09T18:12:44.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.144 7f36c3167700 1 -- start start 2021-04-09T18:12:44.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c3167700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f36bc0cffa0 0x7f36bc0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c3167700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc0d6c80 0x7f36bc10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c3167700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0d7280 con 0x7f36bc0d7800 2021-04-09T18:12:44.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c3167700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36bc0d7080 con 0x7f36bc0cffa0 2021-04-09T18:12:44.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c3167700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc0d91e0 con 0x7f36bc0d6c80 2021-04-09T18:12:44.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c1784700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc0d6c80 0x7f36bc10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c0f83700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f36bc0cffa0 0x7f36bc0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c1784700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc0d6c80 0x7f36bc10cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:49012/0 (socket says 172.21.15.32:49012) 2021-04-09T18:12:44.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c0f83700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f36bc0cffa0 0x7f36bc0d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:33542/0 (socket says 172.21.15.32:33542) 2021-04-09T18:12:44.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36c1784700 1 -- 172.21.15.32:0/1791822986 learned_addr learned my addr 172.21.15.32:0/1791822986 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:44.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.145 7f36bbfff700 1 --1- 172.21.15.32:0/1791822986 >> v1:172.21.15.86:6789/0 conn(0x7f36bc0d7800 0x7f36bc0d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:49300/0 (socket says 172.21.15.32:49300) 2021-04-09T18:12:44.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36bb7fe700 1 -- 172.21.15.32:0/1791822986 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356632710 0 0) 0x7f36a8000c60 con 0x7f36bc0d7800 2021-04-09T18:12:44.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36c1784700 1 -- 172.21.15.32:0/1791822986 >> v2:172.21.15.86:3300/0 conn(0x7f36bc0cffa0 msgr2=0x7f36bc0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:44.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36c1784700 1 --2- 172.21.15.32:0/1791822986 >> v2:172.21.15.86:3300/0 conn(0x7f36bc0cffa0 0x7f36bc0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36c1784700 1 -- 172.21.15.32:0/1791822986 >> v1:172.21.15.86:6789/0 conn(0x7f36bc0d7800 legacy=0x7f36bc0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36c1784700 1 -- 172.21.15.32:0/1791822986 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc0d83f0 con 0x7f36bc0d6c80 2021-04-09T18:12:44.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36c0f83700 1 --2- 172.21.15.32:0/1791822986 >> v2:172.21.15.86:3300/0 conn(0x7f36bc0cffa0 0x7f36bc0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:44.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36c1784700 1 --2- 172.21.15.32:0/1791822986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc0d6c80 0x7f36bc10cfd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f36b4006aa0 tx=0x7f36b4009f20).ready entity=mon.2 client_cookie=60ddbd2d77e8619c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36bb7fe700 1 -- 172.21.15.32:0/1791822986 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b400b5c0 con 0x7f36bc0d6c80 2021-04-09T18:12:44.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36bb7fe700 1 -- 172.21.15.32:0/1791822986 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b4003030 con 0x7f36bc0d6c80 2021-04-09T18:12:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36bb7fe700 1 -- 172.21.15.32:0/1791822986 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b400bb40 con 0x7f36bc0d6c80 2021-04-09T18:12:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36c3167700 1 -- 172.21.15.32:0/1791822986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc0d6c80 msgr2=0x7f36bc10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.146 7f36c3167700 1 --2- 172.21.15.32:0/1791822986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc0d6c80 0x7f36bc10cfd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f36b4006aa0 tx=0x7f36b4009f20).stop 2021-04-09T18:12:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.147 7f36c3167700 1 -- 172.21.15.32:0/1791822986 shutdown_connections 2021-04-09T18:12:44.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.147 7f36c3167700 1 --2- 172.21.15.32:0/1791822986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc0d6c80 0x7f36bc10cfd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.147 7f36c3167700 1 --2- 172.21.15.32:0/1791822986 >> v2:172.21.15.86:3300/0 conn(0x7f36bc0cffa0 0x7f36bc0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.147 7f36c3167700 1 -- 172.21.15.32:0/1791822986 >> 172.21.15.32:0/1791822986 conn(0x7f36bc0ce5c0 msgr2=0x7f36bc0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:44.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.147 7f36c3167700 1 -- 172.21.15.32:0/1791822986 shutdown_connections 2021-04-09T18:12:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.147 7f36c3167700 1 -- 172.21.15.32:0/1791822986 wait complete. 2021-04-09T18:12:44.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.167 7f36c3167700 1 Processor -- start 2021-04-09T18:12:44.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.167 7f36c3167700 1 -- start start 2021-04-09T18:12:44.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.167 7f36c3167700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36bc18e870 0x7f36bc1812b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.167 7f36c3167700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc180330 0x7f36bc1b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c3167700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36bc1b8c80 0x7f36bc0adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c3167700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc180d30 con 0x7f36bc1b8c80 2021-04-09T18:12:44.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c3167700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc1894e0 con 0x7f36bc18e870 2021-04-09T18:12:44.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c3167700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc18ec70 con 0x7f36bc180330 2021-04-09T18:12:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c1784700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36bc1b8c80 0x7f36bc0adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c0f83700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36bc18e870 0x7f36bc1812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c1784700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36bc1b8c80 0x7f36bc0adf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41792/0 (socket says 172.21.15.32:41792) 2021-04-09T18:12:44.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36bbfff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc180330 0x7f36bc1b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c1784700 1 -- 172.21.15.32:0/304482833 learned_addr learned my addr 172.21.15.32:0/304482833 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:44.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c0f83700 1 -- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc180330 msgr2=0x7f36bc1b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c0f83700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc180330 0x7f36bc1b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c0f83700 1 -- 172.21.15.32:0/304482833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36bc1b8c80 msgr2=0x7f36bc0adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c0f83700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36bc1b8c80 0x7f36bc0adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c0f83700 1 -- 172.21.15.32:0/304482833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc1818d0 con 0x7f36bc18e870 2021-04-09T18:12:44.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36bbfff700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36bc180330 0x7f36bc1b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:44.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c1784700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36bc1b8c80 0x7f36bc0adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:44.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c0f83700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36bc18e870 0x7f36bc1812b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f36b0009540 tx=0x7f36b000b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:44.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36b97fa700 1 -- 172.21.15.32:0/304482833 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b000bcd0 con 0x7f36bc18e870 2021-04-09T18:12:44.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36b97fa700 1 -- 172.21.15.32:0/304482833 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b0006740 con 0x7f36bc18e870 2021-04-09T18:12:44.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c3167700 1 -- 172.21.15.32:0/304482833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36bc1818d0 con 0x7f36bc18e870 2021-04-09T18:12:44.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36b97fa700 1 -- 172.21.15.32:0/304482833 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b0002220 con 0x7f36bc18e870 2021-04-09T18:12:44.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.168 7f36c3167700 1 -- 172.21.15.32:0/304482833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc1818d0 con 0x7f36bc18e870 2021-04-09T18:12:44.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.169 7f36b97fa700 1 -- 172.21.15.32:0/304482833 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f36b0009fe0 con 0x7f36bc18e870 2021-04-09T18:12:44.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.169 7f36b97fa700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f369c018930 0x7f369c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.169 7f36b97fa700 1 -- 172.21.15.32:0/304482833 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f36b0032960 con 0x7f36bc18e870 2021-04-09T18:12:44.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.169 7f36bbfff700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f369c018930 0x7f369c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.170 7f36bbfff700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f369c018930 0x7f369c01adc0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:44.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.189 7f36c3167700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6826/25248,v1:172.21.15.86:6827/25248] conn(0x7f36bc18f840 0x7f36bc062a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.189 7f36c3167700 1 -- 172.21.15.32:0/304482833 --> [v2:172.21.15.86:6826/25248,v1:172.21.15.86:6827/25248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36bc1818d0 con 0x7f36bc18f840 2021-04-09T18:12:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.189 7f36c1784700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6826/25248,v1:172.21.15.86:6827/25248] conn(0x7f36bc18f840 0x7f36bc062a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.189 7f36c1784700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6826/25248,v1:172.21.15.86:6827/25248] conn(0x7f36bc18f840 0x7f36bc062a70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.190 7f36b97fa700 1 -- 172.21.15.32:0/304482833 <== osd.7 v2:172.21.15.86:6826/25248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f36b4009310 con 0x7f36bc18f840 2021-04-09T18:12:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.199 7f36c3167700 1 -- 172.21.15.32:0/304482833 --> [v2:172.21.15.86:6826/25248,v1:172.21.15.86:6827/25248] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f36bc180d30 con 0x7f36bc18f840 2021-04-09T18:12:44.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.199 7f36b97fa700 1 -- 172.21.15.32:0/304482833 <== osd.7 v2:172.21.15.86:6826/25248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f36b4006bb0 con 0x7f36bc18f840 2021-04-09T18:12:44.414 INFO:teuthology.orchestra.run.smithi032.stdout:55834574854 2021-04-09T18:12:44.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 -- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6826/25248,v1:172.21.15.86:6827/25248] conn(0x7f36bc18f840 msgr2=0x7f36bc062a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6826/25248,v1:172.21.15.86:6827/25248] conn(0x7f36bc18f840 0x7f36bc062a70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 -- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f369c018930 msgr2=0x7f369c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f369c018930 0x7f369c01adc0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 -- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36bc18e870 msgr2=0x7f36bc1812b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 --2- 172.21.15.32:0/304482833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36bc18e870 0x7f36bc1812b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f36b0009540 tx=0x7f36b000b8e0).stop 2021-04-09T18:12:44.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c1784700 1 -- 172.21.15.32:0/304482833 reap_dead start 2021-04-09T18:12:44.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 -- 172.21.15.32:0/304482833 shutdown_connections 2021-04-09T18:12:44.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 -- 172.21.15.32:0/304482833 >> 172.21.15.32:0/304482833 conn(0x7f36bc1897f0 msgr2=0x7f36bc1b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:44.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 -- 172.21.15.32:0/304482833 shutdown_connections 2021-04-09T18:12:44.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.200 7f36c3167700 1 -- 172.21.15.32:0/304482833 wait complete. 2021-04-09T18:12:44.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-04-09T18:12:44.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.628 7f38e103e700 1 Processor -- start 2021-04-09T18:12:44.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.628 7f38e103e700 1 -- start start 2021-04-09T18:12:44.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.629 7f38e103e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc10e7e0 0x7f38dc0d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.629 7f38e103e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38dc0d7290 con 0x7f38dc0cffb0 2021-04-09T18:12:44.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.629 7f38e103e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38dc0d2450 con 0x7f38dc0d6c90 2021-04-09T18:12:44.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.629 7f38e103e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc0d7090 con 0x7f38dc10e7e0 2021-04-09T18:12:44.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.629 7f38dad9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f38dc0cffb0 0x7f38dc0d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51388/0 (socket says 172.21.15.32:51388) 2021-04-09T18:12:44.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.629 7f38dad9d700 1 -- 172.21.15.32:0/3428245519 learned_addr learned my addr 172.21.15.32:0/3428245519 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:44.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.629 7f38db59e700 1 --2- 172.21.15.32:0/3428245519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc10e7e0 0x7f38dc0d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38d9d9b700 1 -- 172.21.15.32:0/3428245519 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628955628 0 0) 0x7f38cc000ce0 con 0x7f38dc0cffb0 2021-04-09T18:12:44.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38d9d9b700 1 -- 172.21.15.32:0/3428245519 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c4002fe0 con 0x7f38dc0cffb0 2021-04-09T18:12:44.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38d9d9b700 1 -- 172.21.15.32:0/3428245519 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564883491 0 0) 0x7f38d0000c60 con 0x7f38dc0d6c90 2021-04-09T18:12:44.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38db59e700 1 -- 172.21.15.32:0/3428245519 >> v1:172.21.15.86:6789/0 conn(0x7f38dc0d6c90 legacy=0x7f38dc10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38db59e700 1 -- 172.21.15.32:0/3428245519 >> v1:172.21.15.32:6789/0 conn(0x7f38dc0cffb0 legacy=0x7f38dc0d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38db59e700 1 -- 172.21.15.32:0/3428245519 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc0d2670 con 0x7f38dc10e7e0 2021-04-09T18:12:44.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38db59e700 1 --2- 172.21.15.32:0/3428245519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc10e7e0 0x7f38dc0d8000 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f38d4006e50 tx=0x7f38d4009e70).ready entity=mon.2 client_cookie=5397630da173e6be server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:44.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38d9d9b700 1 -- 172.21.15.32:0/3428245519 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38d4006a20 con 0x7f38dc10e7e0 2021-04-09T18:12:44.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38d9d9b700 1 -- 172.21.15.32:0/3428245519 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d4006790 con 0x7f38dc10e7e0 2021-04-09T18:12:44.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38d9d9b700 1 -- 172.21.15.32:0/3428245519 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38d400b700 con 0x7f38dc10e7e0 2021-04-09T18:12:44.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38e103e700 1 -- 172.21.15.32:0/3428245519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc10e7e0 msgr2=0x7f38dc0d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.630 7f38e103e700 1 --2- 172.21.15.32:0/3428245519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc10e7e0 0x7f38dc0d8000 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f38d4006e50 tx=0x7f38d4009e70).stop 2021-04-09T18:12:44.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.631 7f38e103e700 1 -- 172.21.15.32:0/3428245519 shutdown_connections 2021-04-09T18:12:44.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.631 7f38e103e700 1 --2- 172.21.15.32:0/3428245519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc10e7e0 0x7f38dc0d8000 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.631 7f38e103e700 1 -- 172.21.15.32:0/3428245519 >> 172.21.15.32:0/3428245519 conn(0x7f38dc0ce5d0 msgr2=0x7f38dc0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.631 7f38e103e700 1 -- 172.21.15.32:0/3428245519 shutdown_connections 2021-04-09T18:12:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.631 7f38e103e700 1 -- 172.21.15.32:0/3428245519 wait complete. 2021-04-09T18:12:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.651 7f38e103e700 1 Processor -- start 2021-04-09T18:12:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.651 7f38e103e700 1 -- start start 2021-04-09T18:12:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38e103e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38dc08a620 0x7f38dc08b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38e103e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc08b6d0 0x7f38dc180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38e103e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38dc180c50 0x7f38dc181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38e103e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc08e660 con 0x7f38dc08a620 2021-04-09T18:12:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38e103e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc08ab30 con 0x7f38dc180c50 2021-04-09T18:12:44.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38e103e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc08bad0 con 0x7f38dc08b6d0 2021-04-09T18:12:44.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38dad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38dc08a620 0x7f38dc08b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38dad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38dc08a620 0x7f38dc08b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41804/0 (socket says 172.21.15.32:41804) 2021-04-09T18:12:44.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38dad9d700 1 -- 172.21.15.32:0/3609614940 learned_addr learned my addr 172.21.15.32:0/3609614940 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:44.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38da59c700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc08b6d0 0x7f38dc180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38db59e700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38dc180c50 0x7f38dc181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38dad9d700 1 -- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc08b6d0 msgr2=0x7f38dc180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38dad9d700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc08b6d0 0x7f38dc180630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38dad9d700 1 -- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38dc180c50 msgr2=0x7f38dc181050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38dad9d700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38dc180c50 0x7f38dc181050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38dad9d700 1 -- 172.21.15.32:0/3609614940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc08e1f0 con 0x7f38dc08a620 2021-04-09T18:12:44.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38dad9d700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38dc08a620 0x7f38dc08b0b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f38cc001400 tx=0x7f38cc00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38cb7fe700 1 -- 172.21.15.32:0/3609614940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38cc000ef0 con 0x7f38dc08a620 2021-04-09T18:12:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38cb7fe700 1 -- 172.21.15.32:0/3609614940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38cc00acf0 con 0x7f38dc08a620 2021-04-09T18:12:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.652 7f38e103e700 1 -- 172.21.15.32:0/3609614940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38dc08e1f0 con 0x7f38dc08a620 2021-04-09T18:12:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.653 7f38cb7fe700 1 -- 172.21.15.32:0/3609614940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38cc004590 con 0x7f38dc08a620 2021-04-09T18:12:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.653 7f38e103e700 1 -- 172.21.15.32:0/3609614940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38dc08e1f0 con 0x7f38dc08a620 2021-04-09T18:12:44.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.653 7f38cb7fe700 1 -- 172.21.15.32:0/3609614940 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f38cc0043a0 con 0x7f38dc08a620 2021-04-09T18:12:44.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.653 7f38cb7fe700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38c0018930 0x7f38c001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:44.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.653 7f38cb7fe700 1 -- 172.21.15.32:0/3609614940 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f38cc032a10 con 0x7f38dc08a620 2021-04-09T18:12:44.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.653 7f38da59c700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38c0018930 0x7f38c001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:44.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.654 7f38da59c700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38c0018930 0x7f38c001adc0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:44.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.673 7f38e103e700 1 -- 172.21.15.32:0/3609614940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38dc1b7180 con 0x7f38dc08a620 2021-04-09T18:12:44.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.677 7f38cb7fe700 1 -- 172.21.15.32:0/3609614940 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f38cc0009b0 con 0x7f38dc08a620 2021-04-09T18:12:44.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.875 7f38e103e700 1 -- 172.21.15.32:0/3609614940 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f38dc0d7290 con 0x7f38dc08a620 2021-04-09T18:12:44.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.875 7f38cb7fe700 1 -- 172.21.15.32:0/3609614940 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f38cc00acf0 con 0x7f38dc08a620 2021-04-09T18:12:44.899 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2021-04-09T18:12:44.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 -- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38c0018930 msgr2=0x7f38c001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38c0018930 0x7f38c001adc0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 -- 172.21.15.32:0/3609614940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38dc08a620 msgr2=0x7f38dc08b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:44.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38dc08a620 0x7f38dc08b0b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f38cc001400 tx=0x7f38cc00a900).stop 2021-04-09T18:12:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 -- 172.21.15.32:0/3609614940 shutdown_connections 2021-04-09T18:12:44.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38c0018930 0x7f38c001adc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38dc180c50 0x7f38dc181050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38dc08b6d0 0x7f38dc180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 --2- 172.21.15.32:0/3609614940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38dc08a620 0x7f38dc08b0b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:44.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 -- 172.21.15.32:0/3609614940 >> 172.21.15.32:0/3609614940 conn(0x7f38dc189800 msgr2=0x7f38dc1b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:44.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 -- 172.21.15.32:0/3609614940 shutdown_connections 2021-04-09T18:12:44.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:44.878 7f38e103e700 1 -- 172.21.15.32:0/3609614940 wait complete. 2021-04-09T18:12:44.943 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2021-04-09T18:12:44.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-04-09T18:12:45.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.107 7fe205b60700 1 Processor -- start 2021-04-09T18:12:45.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.107 7fe205b60700 1 -- start start 2021-04-09T18:12:45.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe205b60700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2000cffb0 0x7fe2000d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe205b60700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe2000d7810 0x7fe2000d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe205b60700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2000d7290 con 0x7fe2000d8230 2021-04-09T18:12:45.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe205b60700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2000d91f0 con 0x7fe2000cffb0 2021-04-09T18:12:45.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe205b60700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20010dff0 con 0x7fe2000d7810 2021-04-09T18:12:45.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe1ff7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2000cffb0 0x7fe2000d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe1feffd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe2000d7810 0x7fe2000d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe1fffff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe2000d8230 0x7fe2000d5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:49330/0 (socket says 172.21.15.32:49330) 2021-04-09T18:12:45.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe1ff7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe2000cffb0 0x7fe2000d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41812/0 (socket says 172.21.15.32:41812) 2021-04-09T18:12:45.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.108 7fe1fffff700 1 -- 172.21.15.32:0/4051048631 learned_addr learned my addr 172.21.15.32:0/4051048631 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:45.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1ff7fe700 1 -- 172.21.15.32:0/4051048631 >> v2:172.21.15.86:3300/0 conn(0x7fe2000d7810 msgr2=0x7fe2000d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:45.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1ff7fe700 1 --2- 172.21.15.32:0/4051048631 >> v2:172.21.15.86:3300/0 conn(0x7fe2000d7810 0x7fe2000d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1ff7fe700 1 -- 172.21.15.32:0/4051048631 >> v1:172.21.15.86:6789/0 conn(0x7fe2000d8230 legacy=0x7fe2000d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1ff7fe700 1 -- 172.21.15.32:0/4051048631 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20010e210 con 0x7fe2000cffb0 2021-04-09T18:12:45.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1fe7fc700 1 -- 172.21.15.32:0/4051048631 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423479973 0 0) 0x7fe1f8001440 con 0x7fe2000d8230 2021-04-09T18:12:45.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1feffd700 1 --2- 172.21.15.32:0/4051048631 >> v2:172.21.15.86:3300/0 conn(0x7fe2000d7810 0x7fe2000d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:45.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1ff7fe700 1 --2- 172.21.15.32:0/4051048631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000cffb0 0x7fe2000d2450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fe1f0006650 tx=0x7fe1f0009a80).ready entity=mon.0 client_cookie=4b35756e988cd12f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:45.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1fe7fc700 1 -- 172.21.15.32:0/4051048631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f00009b0 con 0x7fe2000cffb0 2021-04-09T18:12:45.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1fe7fc700 1 -- 172.21.15.32:0/4051048631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f000c0c0 con 0x7fe2000cffb0 2021-04-09T18:12:45.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe1fe7fc700 1 -- 172.21.15.32:0/4051048631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f000c510 con 0x7fe2000cffb0 2021-04-09T18:12:45.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe205b60700 1 -- 172.21.15.32:0/4051048631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000cffb0 msgr2=0x7fe2000d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.109 7fe205b60700 1 --2- 172.21.15.32:0/4051048631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000cffb0 0x7fe2000d2450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fe1f0006650 tx=0x7fe1f0009a80).stop 2021-04-09T18:12:45.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.110 7fe205b60700 1 -- 172.21.15.32:0/4051048631 shutdown_connections 2021-04-09T18:12:45.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.110 7fe205b60700 1 --2- 172.21.15.32:0/4051048631 >> v2:172.21.15.86:3300/0 conn(0x7fe2000d7810 0x7fe2000d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.110 7fe205b60700 1 --2- 172.21.15.32:0/4051048631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000cffb0 0x7fe2000d2450 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.110 7fe205b60700 1 -- 172.21.15.32:0/4051048631 >> 172.21.15.32:0/4051048631 conn(0x7fe2000ce5d0 msgr2=0x7fe2000cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.110 7fe205b60700 1 -- 172.21.15.32:0/4051048631 shutdown_connections 2021-04-09T18:12:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.110 7fe205b60700 1 -- 172.21.15.32:0/4051048631 wait complete. 2021-04-09T18:12:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.130 7fe205b60700 1 Processor -- start 2021-04-09T18:12:45.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.130 7fe205b60700 1 -- start start 2021-04-09T18:12:45.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe205b60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000b8470 0x7fe2000b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe205b60700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe20018f050 0x7fe20018f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe205b60700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe200190180 0x7fe200190580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe205b60700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe200193120 con 0x7fe2000b8470 2021-04-09T18:12:45.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe205b60700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20018fc00 con 0x7fe20018f050 2021-04-09T18:12:45.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe205b60700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe200190ba0 con 0x7fe200190180 2021-04-09T18:12:45.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1ff7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000b8470 0x7fe2000b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1ff7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000b8470 0x7fe2000b8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41818/0 (socket says 172.21.15.32:41818) 2021-04-09T18:12:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1ff7fe700 1 -- 172.21.15.32:0/3790879914 learned_addr learned my addr 172.21.15.32:0/3790879914 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:45.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1fffff700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe200190180 0x7fe200190580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1feffd700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe20018f050 0x7fe20018f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1ff7fe700 1 -- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe200190180 msgr2=0x7fe200190580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1ff7fe700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe200190180 0x7fe200190580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1ff7fe700 1 -- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe20018f050 msgr2=0x7fe20018f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1ff7fe700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe20018f050 0x7fe20018f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1ff7fe700 1 -- 172.21.15.32:0/3790879914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2001a9e60 con 0x7fe2000b8470 2021-04-09T18:12:45.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1ff7fe700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000b8470 0x7fe2000b8870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe1f0009650 tx=0x7fe1f000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:45.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1e7fff700 1 -- 172.21.15.32:0/3790879914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f00093a0 con 0x7fe2000b8470 2021-04-09T18:12:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1e7fff700 1 -- 172.21.15.32:0/3790879914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f000bd50 con 0x7fe2000b8470 2021-04-09T18:12:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe205b60700 1 -- 172.21.15.32:0/3790879914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2001a9e60 con 0x7fe2000b8470 2021-04-09T18:12:45.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.131 7fe1e7fff700 1 -- 172.21.15.32:0/3790879914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f00052e0 con 0x7fe2000b8470 2021-04-09T18:12:45.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.132 7fe205b60700 1 -- 172.21.15.32:0/3790879914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2001a9e60 con 0x7fe2000b8470 2021-04-09T18:12:45.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.132 7fe1e7fff700 1 -- 172.21.15.32:0/3790879914 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe1f0006030 con 0x7fe2000b8470 2021-04-09T18:12:45.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.132 7fe1e7fff700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1e0018930 0x7fe1e001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.132 7fe1e7fff700 1 -- 172.21.15.32:0/3790879914 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe1f00329e0 con 0x7fe2000b8470 2021-04-09T18:12:45.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.132 7fe1feffd700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1e0018930 0x7fe1e001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.133 7fe1feffd700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1e0018930 0x7fe1e001ae10 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:45.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.152 7fe205b60700 1 -- 172.21.15.32:0/3790879914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe200182e10 con 0x7fe2000b8470 2021-04-09T18:12:45.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.156 7fe1e7fff700 1 -- 172.21.15.32:0/3790879914 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe1f00009c0 con 0x7fe2000b8470 2021-04-09T18:12:45.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.356 7fe205b60700 1 -- 172.21.15.32:0/3790879914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe2000d7290 con 0x7fe2000b8470 2021-04-09T18:12:45.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.356 7fe1e7fff700 1 -- 172.21.15.32:0/3790879914 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe1f000bd50 con 0x7fe2000b8470 2021-04-09T18:12:45.450 INFO:teuthology.orchestra.run.smithi032.stdout:55834574854 2021-04-09T18:12:45.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 -- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1e0018930 msgr2=0x7fe1e001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1e0018930 0x7fe1e001ae10 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 -- 172.21.15.32:0/3790879914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000b8470 msgr2=0x7fe2000b8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000b8470 0x7fe2000b8870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe1f0009650 tx=0x7fe1f000b960).stop 2021-04-09T18:12:45.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 -- 172.21.15.32:0/3790879914 shutdown_connections 2021-04-09T18:12:45.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1e0018930 0x7fe1e001ae10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe200190180 0x7fe200190580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe20018f050 0x7fe20018f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 --2- 172.21.15.32:0/3790879914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2000b8470 0x7fe2000b8870 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 -- 172.21.15.32:0/3790879914 >> 172.21.15.32:0/3790879914 conn(0x7fe200183890 msgr2=0x7fe20019b900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:45.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 -- 172.21.15.32:0/3790879914 shutdown_connections 2021-04-09T18:12:45.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.359 7fe205b60700 1 -- 172.21.15.32:0/3790879914 wait complete. 2021-04-09T18:12:45.580 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2021-04-09T18:12:45.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-04-09T18:12:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.721 7f21678cc700 1 Processor -- start 2021-04-09T18:12:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.721 7f21678cc700 1 -- start start 2021-04-09T18:12:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.721 7f21678cc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f21600cffb0 0x7f21600d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f21678cc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21600d7810 0x7f21600d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f21678cc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21600d7290 con 0x7f21600d8230 2021-04-09T18:12:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f21678cc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21600d91f0 con 0x7f21600d7810 2021-04-09T18:12:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f21678cc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216010dff0 con 0x7f21600cffb0 2021-04-09T18:12:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2164ee7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21600d7810 0x7f21600d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2165ee9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f21600d8230 0x7f21600d5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51416/0 (socket says 172.21.15.32:51416) 2021-04-09T18:12:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2165ee9700 1 -- 172.21.15.32:0/889954027 learned_addr learned my addr 172.21.15.32:0/889954027 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:45.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f21656e8700 1 --2- 172.21.15.32:0/889954027 >> v2:172.21.15.86:3300/0 conn(0x7f21600cffb0 0x7f21600d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2164ee7700 1 -- 172.21.15.32:0/889954027 >> v2:172.21.15.86:3300/0 conn(0x7f21600cffb0 msgr2=0x7f21600d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2164ee7700 1 --2- 172.21.15.32:0/889954027 >> v2:172.21.15.86:3300/0 conn(0x7f21600cffb0 0x7f21600d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2164ee7700 1 -- 172.21.15.32:0/889954027 >> v1:172.21.15.32:6789/0 conn(0x7f21600d8230 legacy=0x7f21600d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2164ee7700 1 -- 172.21.15.32:0/889954027 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216010e210 con 0x7f21600d7810 2021-04-09T18:12:45.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2157fff700 1 -- 172.21.15.32:0/889954027 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769591169 0 0) 0x7f2158000d30 con 0x7f21600d8230 2021-04-09T18:12:45.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2164ee7700 1 --2- 172.21.15.32:0/889954027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21600d7810 0x7f21600d7c10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f2150006650 tx=0x7f2150009a00).ready entity=mon.0 client_cookie=8d7fae433d17050d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:45.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2157fff700 1 -- 172.21.15.32:0/889954027 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2150009df0 con 0x7f21600d7810 2021-04-09T18:12:45.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2157fff700 1 -- 172.21.15.32:0/889954027 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21500009b0 con 0x7f21600d7810 2021-04-09T18:12:45.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f2157fff700 1 -- 172.21.15.32:0/889954027 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f215000c2b0 con 0x7f21600d7810 2021-04-09T18:12:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f21678cc700 1 -- 172.21.15.32:0/889954027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21600d7810 msgr2=0x7f21600d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.722 7f21678cc700 1 --2- 172.21.15.32:0/889954027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21600d7810 0x7f21600d7c10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f2150006650 tx=0x7f2150009a00).stop 2021-04-09T18:12:45.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.723 7f21678cc700 1 -- 172.21.15.32:0/889954027 shutdown_connections 2021-04-09T18:12:45.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.723 7f21678cc700 1 --2- 172.21.15.32:0/889954027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21600d7810 0x7f21600d7c10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.723 7f21678cc700 1 --2- 172.21.15.32:0/889954027 >> v2:172.21.15.86:3300/0 conn(0x7f21600cffb0 0x7f21600d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.723 7f21678cc700 1 -- 172.21.15.32:0/889954027 >> 172.21.15.32:0/889954027 conn(0x7f21600ce5d0 msgr2=0x7f21600cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.723 7f21678cc700 1 -- 172.21.15.32:0/889954027 shutdown_connections 2021-04-09T18:12:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.723 7f21678cc700 1 -- 172.21.15.32:0/889954027 wait complete. 2021-04-09T18:12:45.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.743 7f21678cc700 1 Processor -- start 2021-04-09T18:12:45.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f21678cc700 1 -- start start 2021-04-09T18:12:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f21678cc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21601a7dc0 0x7f21601a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f21678cc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2160184080 0x7f2160184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f21678cc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2160184aa0 0x7f216008f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f21678cc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21601a8b40 con 0x7f2160184aa0 2021-04-09T18:12:45.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f21678cc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21601a7870 con 0x7f21601a7dc0 2021-04-09T18:12:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f21678cc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21601a87e0 con 0x7f2160184080 2021-04-09T18:12:45.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f2165ee9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2160184aa0 0x7f216008f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f2165ee9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2160184aa0 0x7f216008f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41830/0 (socket says 172.21.15.32:41830) 2021-04-09T18:12:45.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f2165ee9700 1 -- 172.21.15.32:0/3338256854 learned_addr learned my addr 172.21.15.32:0/3338256854 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:45.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f2164ee7700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2160184080 0x7f2160184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.744 7f21656e8700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21601a7dc0 0x7f21601a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f2165ee9700 1 -- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2160184080 msgr2=0x7f2160184480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f2165ee9700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2160184080 0x7f2160184480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f2165ee9700 1 -- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21601a7dc0 msgr2=0x7f21601a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f2165ee9700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21601a7dc0 0x7f21601a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:45.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f2165ee9700 1 -- 172.21.15.32:0/3338256854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216008f810 con 0x7f2160184aa0 2021-04-09T18:12:45.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f2165ee9700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2160184aa0 0x7f216008f1f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2158007f90 tx=0x7f215800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:45.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f2155ffb700 1 -- 172.21.15.32:0/3338256854 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2158000cc0 con 0x7f2160184aa0 2021-04-09T18:12:45.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f2155ffb700 1 -- 172.21.15.32:0/3338256854 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2158007020 con 0x7f2160184aa0 2021-04-09T18:12:45.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f21678cc700 1 -- 172.21.15.32:0/3338256854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216008f810 con 0x7f2160184aa0 2021-04-09T18:12:45.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f2155ffb700 1 -- 172.21.15.32:0/3338256854 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2158007500 con 0x7f2160184aa0 2021-04-09T18:12:45.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.745 7f21678cc700 1 -- 172.21.15.32:0/3338256854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216008f810 con 0x7f2160184aa0 2021-04-09T18:12:45.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.746 7f2155ffb700 1 -- 172.21.15.32:0/3338256854 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2158007200 con 0x7f2160184aa0 2021-04-09T18:12:45.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.746 7f2155ffb700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2140018930 0x7f214001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:45.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.746 7f2155ffb700 1 -- 172.21.15.32:0/3338256854 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2158032d40 con 0x7f2160184aa0 2021-04-09T18:12:45.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.746 7f21656e8700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2140018930 0x7f214001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:45.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.746 7f21656e8700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2140018930 0x7f214001ae10 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:45.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.766 7f21678cc700 1 -- 172.21.15.32:0/3338256854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2160182e80 con 0x7f2160184aa0 2021-04-09T18:12:45.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.769 7f2155ffb700 1 -- 172.21.15.32:0/3338256854 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2158007d00 con 0x7f2160184aa0 2021-04-09T18:12:46.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.968 7f21678cc700 1 -- 172.21.15.32:0/3338256854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f21600d7290 con 0x7f2160184aa0 2021-04-09T18:12:46.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.968 7f2155ffb700 1 -- 172.21.15.32:0/3338256854 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2158007020 con 0x7f2160184aa0 2021-04-09T18:12:46.008 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2021-04-09T18:12:46.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 -- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2140018930 msgr2=0x7f214001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2140018930 0x7f214001ae10 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 -- 172.21.15.32:0/3338256854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2160184aa0 msgr2=0x7f216008f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2160184aa0 0x7f216008f1f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2158007f90 tx=0x7f215800ac50).stop 2021-04-09T18:12:46.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 -- 172.21.15.32:0/3338256854 shutdown_connections 2021-04-09T18:12:46.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2140018930 0x7f214001ae10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2160184aa0 0x7f216008f1f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2160184080 0x7f2160184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 --2- 172.21.15.32:0/3338256854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21601a7dc0 0x7f21601a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 -- 172.21.15.32:0/3338256854 >> 172.21.15.32:0/3338256854 conn(0x7f21601a8f50 msgr2=0x7f2160188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:46.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 -- 172.21.15.32:0/3338256854 shutdown_connections 2021-04-09T18:12:46.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:45.971 7f21678cc700 1 -- 172.21.15.32:0/3338256854 wait complete. 2021-04-09T18:12:46.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2021-04-09T18:12:46.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2021-04-09T18:12:46.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.188 7f9a55706700 1 Processor -- start 2021-04-09T18:12:46.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.188 7f9a55706700 1 -- start start 2021-04-09T18:12:46.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.188 7f9a55706700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffb0 0x7f9a500d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.188 7f9a55706700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a500d7810 0x7f9a500d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.188 7f9a55706700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a500d7290 con 0x7f9a500d5c80 2021-04-09T18:12:46.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.188 7f9a55706700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a500d8590 con 0x7f9a500d7810 2021-04-09T18:12:46.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.188 7f9a55706700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a500d91f0 con 0x7f9a500cffb0 2021-04-09T18:12:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4eabe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a500d7810 0x7f9a500d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4f2bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffb0 0x7f9a500d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4fac0700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9a500d5c80 0x7f9a500d6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:49356/0 (socket says 172.21.15.32:49356) 2021-04-09T18:12:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4eabe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a500d7810 0x7f9a500d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:33600/0 (socket says 172.21.15.32:33600) 2021-04-09T18:12:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4f2bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffb0 0x7f9a500d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:49064/0 (socket says 172.21.15.32:49064) 2021-04-09T18:12:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4fac0700 1 -- 172.21.15.32:0/3332582654 learned_addr learned my addr 172.21.15.32:0/3332582654 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4eabe700 1 -- 172.21.15.32:0/3332582654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffb0 msgr2=0x7f9a500d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:46.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4eabe700 1 --2- 172.21.15.32:0/3332582654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffb0 0x7f9a500d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4eabe700 1 -- 172.21.15.32:0/3332582654 >> v1:172.21.15.86:6789/0 conn(0x7f9a500d5c80 legacy=0x7f9a500d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4eabe700 1 -- 172.21.15.32:0/3332582654 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a500d8230 con 0x7f9a500d7810 2021-04-09T18:12:46.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4f2bf700 1 --2- 172.21.15.32:0/3332582654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffb0 0x7f9a500d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:46.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4e2bd700 1 -- 172.21.15.32:0/3332582654 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436175873 0 0) 0x7f9a48001440 con 0x7f9a500d5c80 2021-04-09T18:12:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4eabe700 1 --2- 172.21.15.32:0/3332582654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a500d7810 0x7f9a500d7c10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9a3c006650 tx=0x7f9a3c009a80).ready entity=mon.1 client_cookie=f46b293f7d1937b4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:46.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4e2bd700 1 -- 172.21.15.32:0/3332582654 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a3c0009b0 con 0x7f9a500d7810 2021-04-09T18:12:46.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4e2bd700 1 -- 172.21.15.32:0/3332582654 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c00c0c0 con 0x7f9a500d7810 2021-04-09T18:12:46.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.189 7f9a4e2bd700 1 -- 172.21.15.32:0/3332582654 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a3c00c510 con 0x7f9a500d7810 2021-04-09T18:12:46.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.190 7f9a55706700 1 -- 172.21.15.32:0/3332582654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a500d7810 msgr2=0x7f9a500d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.190 7f9a55706700 1 --2- 172.21.15.32:0/3332582654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a500d7810 0x7f9a500d7c10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9a3c006650 tx=0x7f9a3c009a80).stop 2021-04-09T18:12:46.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.190 7f9a55706700 1 -- 172.21.15.32:0/3332582654 shutdown_connections 2021-04-09T18:12:46.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.190 7f9a55706700 1 --2- 172.21.15.32:0/3332582654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a500d7810 0x7f9a500d7c10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.190 7f9a55706700 1 --2- 172.21.15.32:0/3332582654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffb0 0x7f9a500d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.190 7f9a55706700 1 -- 172.21.15.32:0/3332582654 >> 172.21.15.32:0/3332582654 conn(0x7f9a500ce5d0 msgr2=0x7f9a500cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:46.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.190 7f9a55706700 1 -- 172.21.15.32:0/3332582654 shutdown_connections 2021-04-09T18:12:46.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.190 7f9a55706700 1 -- 172.21.15.32:0/3332582654 wait complete. 2021-04-09T18:12:46.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.210 7f9a55706700 1 Processor -- start 2021-04-09T18:12:46.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.210 7f9a55706700 1 -- start start 2021-04-09T18:12:46.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a55706700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a50190100 0x7f9a50190d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a55706700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a501913a0 0x7f9a500b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a55706700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a500b8330 0x7f9a500b8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a55706700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a50190500 con 0x7f9a501913a0 2021-04-09T18:12:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a55706700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a501917a0 con 0x7f9a500b8330 2021-04-09T18:12:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a55706700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a50193ab0 con 0x7f9a50190100 2021-04-09T18:12:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a4eabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a501913a0 0x7f9a500b7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a4eabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a501913a0 0x7f9a500b7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41846/0 (socket says 172.21.15.32:41846) 2021-04-09T18:12:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a4eabe700 1 -- 172.21.15.32:0/3614578136 learned_addr learned my addr 172.21.15.32:0/3614578136 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:46.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a4fac0700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a500b8330 0x7f9a500b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a4f2bf700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a50190100 0x7f9a50190d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a4eabe700 1 -- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a50190100 msgr2=0x7f9a50190d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a4eabe700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a50190100 0x7f9a50190d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.211 7f9a4eabe700 1 -- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a500b8330 msgr2=0x7f9a500b8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.212 7f9a4eabe700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a500b8330 0x7f9a500b8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.212 7f9a4eabe700 1 -- 172.21.15.32:0/3614578136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a501ac2e0 con 0x7f9a501913a0 2021-04-09T18:12:46.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.212 7f9a4eabe700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a501913a0 0x7f9a500b7d10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9a3c009630 tx=0x7f9a3c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:46.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.212 7f9a47fff700 1 -- 172.21.15.32:0/3614578136 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a3c0093a0 con 0x7f9a501913a0 2021-04-09T18:12:46.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.212 7f9a47fff700 1 -- 172.21.15.32:0/3614578136 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c00bcf0 con 0x7f9a501913a0 2021-04-09T18:12:46.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.212 7f9a55706700 1 -- 172.21.15.32:0/3614578136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a501ac2e0 con 0x7f9a501913a0 2021-04-09T18:12:46.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.212 7f9a47fff700 1 -- 172.21.15.32:0/3614578136 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a3c0054d0 con 0x7f9a501913a0 2021-04-09T18:12:46.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.212 7f9a55706700 1 -- 172.21.15.32:0/3614578136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a501ac2e0 con 0x7f9a501913a0 2021-04-09T18:12:46.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.213 7f9a47fff700 1 -- 172.21.15.32:0/3614578136 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9a3c0054f0 con 0x7f9a501913a0 2021-04-09T18:12:46.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.213 7f9a47fff700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a2c0188c0 0x7f9a2c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.213 7f9a47fff700 1 -- 172.21.15.32:0/3614578136 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9a3c0327a0 con 0x7f9a501913a0 2021-04-09T18:12:46.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.213 7f9a4f2bf700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a2c0188c0 0x7f9a2c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.214 7f9a4f2bf700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a2c0188c0 0x7f9a2c01adc0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:46.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.233 7f9a55706700 1 -- 172.21.15.32:0/3614578136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a50182e80 con 0x7f9a501913a0 2021-04-09T18:12:46.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.236 7f9a47fff700 1 -- 172.21.15.32:0/3614578136 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9a3c0052e0 con 0x7f9a501913a0 2021-04-09T18:12:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.434 7f9a55706700 1 -- 172.21.15.32:0/3614578136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f9a500d7290 con 0x7f9a501913a0 2021-04-09T18:12:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.435 7f9a47fff700 1 -- 172.21.15.32:0/3614578136 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a3c0054f0 con 0x7f9a501913a0 2021-04-09T18:12:46.538 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2021-04-09T18:12:46.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 -- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a2c0188c0 msgr2=0x7f9a2c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a2c0188c0 0x7f9a2c01adc0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 -- 172.21.15.32:0/3614578136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a501913a0 msgr2=0x7f9a500b7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a501913a0 0x7f9a500b7d10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9a3c009630 tx=0x7f9a3c00b900).stop 2021-04-09T18:12:46.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 -- 172.21.15.32:0/3614578136 shutdown_connections 2021-04-09T18:12:46.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a2c0188c0 0x7f9a2c01adc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a501913a0 0x7f9a500b7d10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a500b8330 0x7f9a500b8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 --2- 172.21.15.32:0/3614578136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a50190100 0x7f9a50190d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 -- 172.21.15.32:0/3614578136 >> 172.21.15.32:0/3614578136 conn(0x7f9a50184a80 msgr2=0x7f9a501a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:46.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 -- 172.21.15.32:0/3614578136 shutdown_connections 2021-04-09T18:12:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.437 7f9a55706700 1 -- 172.21.15.32:0/3614578136 wait complete. 2021-04-09T18:12:46.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2021-04-09T18:12:46.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2021-04-09T18:12:46.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.798 7f9b46e22700 1 Processor -- start 2021-04-09T18:12:46.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.798 7f9b46e22700 1 -- start start 2021-04-09T18:12:46.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b46e22700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9b400cffb0 0x7f9b400d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b46e22700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b400d7810 0x7f9b400d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b46e22700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b400d7290 con 0x7f9b400d8230 2021-04-09T18:12:46.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b46e22700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b400d91f0 con 0x7f9b400d7810 2021-04-09T18:12:46.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b46e22700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b4010dff0 con 0x7f9b400cffb0 2021-04-09T18:12:46.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b3ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9b400d7810 0x7f9b400d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b4543f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9b400d8230 0x7f9b400d5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51442/0 (socket says 172.21.15.32:51442) 2021-04-09T18:12:46.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b4543f700 1 -- 172.21.15.32:0/201012864 learned_addr learned my addr 172.21.15.32:0/201012864 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:46.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b44c3e700 1 --2- 172.21.15.32:0/201012864 >> v2:172.21.15.86:3300/0 conn(0x7f9b400cffb0 0x7f9b400d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b3ffff700 1 -- 172.21.15.32:0/201012864 >> v2:172.21.15.86:3300/0 conn(0x7f9b400cffb0 msgr2=0x7f9b400d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:46.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b3ffff700 1 --2- 172.21.15.32:0/201012864 >> v2:172.21.15.86:3300/0 conn(0x7f9b400cffb0 0x7f9b400d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b3ffff700 1 -- 172.21.15.32:0/201012864 >> v1:172.21.15.32:6789/0 conn(0x7f9b400d8230 legacy=0x7f9b400d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b3ffff700 1 -- 172.21.15.32:0/201012864 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4010e210 con 0x7f9b400d7810 2021-04-09T18:12:46.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b3f7fe700 1 -- 172.21.15.32:0/201012864 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184665595 0 0) 0x7f9b380009b0 con 0x7f9b400d8230 2021-04-09T18:12:46.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.799 7f9b3ffff700 1 --2- 172.21.15.32:0/201012864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b400d7810 0x7f9b400d7c10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9b300069d0 tx=0x7f9b30009dc0).ready entity=mon.0 client_cookie=4f33566eaa12df47 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:46.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b3f7fe700 1 -- 172.21.15.32:0/201012864 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b3000b510 con 0x7f9b400d7810 2021-04-09T18:12:46.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b3f7fe700 1 -- 172.21.15.32:0/201012864 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b30006ac0 con 0x7f9b400d7810 2021-04-09T18:12:46.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b3f7fe700 1 -- 172.21.15.32:0/201012864 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b30000d30 con 0x7f9b400d7810 2021-04-09T18:12:46.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b46e22700 1 -- 172.21.15.32:0/201012864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b400d7810 msgr2=0x7f9b400d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b46e22700 1 --2- 172.21.15.32:0/201012864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b400d7810 0x7f9b400d7c10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9b300069d0 tx=0x7f9b30009dc0).stop 2021-04-09T18:12:46.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b46e22700 1 -- 172.21.15.32:0/201012864 shutdown_connections 2021-04-09T18:12:46.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b46e22700 1 --2- 172.21.15.32:0/201012864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b400d7810 0x7f9b400d7c10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b46e22700 1 --2- 172.21.15.32:0/201012864 >> v2:172.21.15.86:3300/0 conn(0x7f9b400cffb0 0x7f9b400d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b46e22700 1 -- 172.21.15.32:0/201012864 >> 172.21.15.32:0/201012864 conn(0x7f9b400ce5d0 msgr2=0x7f9b400cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b46e22700 1 -- 172.21.15.32:0/201012864 shutdown_connections 2021-04-09T18:12:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.800 7f9b46e22700 1 -- 172.21.15.32:0/201012864 wait complete. 2021-04-09T18:12:46.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.821 7f9b46e22700 1 Processor -- start 2021-04-09T18:12:46.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.821 7f9b46e22700 1 -- start start 2021-04-09T18:12:46.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b46e22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b401b8380 0x7f9b401b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b46e22700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b4008c7b0 0x7f9b4008cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b46e22700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b4008d1d0 0x7f9b401805b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:46.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b46e22700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40181a30 con 0x7f9b401b8380 2021-04-09T18:12:46.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b46e22700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4008c230 con 0x7f9b4008d1d0 2021-04-09T18:12:46.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b46e22700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b401b8da0 con 0x7f9b4008c7b0 2021-04-09T18:12:46.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b44c3e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b401b8380 0x7f9b401b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b44c3e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b401b8380 0x7f9b401b8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41858/0 (socket says 172.21.15.32:41858) 2021-04-09T18:12:46.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b44c3e700 1 -- 172.21.15.32:0/369482321 learned_addr learned my addr 172.21.15.32:0/369482321 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:46.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b4543f700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b4008d1d0 0x7f9b401805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b3ffff700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b4008c7b0 0x7f9b4008cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:46.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b44c3e700 1 -- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b4008c7b0 msgr2=0x7f9b4008cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b44c3e700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b4008c7b0 0x7f9b4008cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b44c3e700 1 -- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b4008d1d0 msgr2=0x7f9b401805b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:46.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b44c3e700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b4008d1d0 0x7f9b401805b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:46.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b44c3e700 1 -- 172.21.15.32:0/369482321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b401b8f90 con 0x7f9b401b8380 2021-04-09T18:12:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b44c3e700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b401b8380 0x7f9b401b8780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9b28006f60 tx=0x7f9b2800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:47.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b3d7fa700 1 -- 172.21.15.32:0/369482321 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2800bd60 con 0x7f9b401b8380 2021-04-09T18:12:47.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b3d7fa700 1 -- 172.21.15.32:0/369482321 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b28000c90 con 0x7f9b401b8380 2021-04-09T18:12:47.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b46e22700 1 -- 172.21.15.32:0/369482321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b401b8f90 con 0x7f9b401b8380 2021-04-09T18:12:47.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b3d7fa700 1 -- 172.21.15.32:0/369482321 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b28002220 con 0x7f9b401b8380 2021-04-09T18:12:47.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.822 7f9b46e22700 1 -- 172.21.15.32:0/369482321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b401b8f90 con 0x7f9b401b8380 2021-04-09T18:12:47.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.823 7f9b3d7fa700 1 -- 172.21.15.32:0/369482321 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9b280052e0 con 0x7f9b401b8380 2021-04-09T18:12:47.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.823 7f9b3d7fa700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b24018930 0x7f9b2401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.823 7f9b3d7fa700 1 -- 172.21.15.32:0/369482321 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9b280329d0 con 0x7f9b401b8380 2021-04-09T18:12:47.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.823 7f9b3ffff700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b24018930 0x7f9b2401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.824 7f9b3ffff700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b24018930 0x7f9b2401ae10 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.843 7f9b46e22700 1 -- 172.21.15.32:0/369482321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b401b7180 con 0x7f9b401b8380 2021-04-09T18:12:47.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:46.847 7f9b3d7fa700 1 -- 172.21.15.32:0/369482321 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9b28000c90 con 0x7f9b401b8380 2021-04-09T18:12:47.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.046 7f9b46e22700 1 -- 172.21.15.32:0/369482321 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9b4008c230 con 0x7f9b401b8380 2021-04-09T18:12:47.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.046 7f9b3d7fa700 1 -- 172.21.15.32:0/369482321 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9b28006b00 con 0x7f9b401b8380 2021-04-09T18:12:47.071 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2021-04-09T18:12:47.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 -- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b24018930 msgr2=0x7f9b2401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b24018930 0x7f9b2401ae10 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 -- 172.21.15.32:0/369482321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b401b8380 msgr2=0x7f9b401b8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b401b8380 0x7f9b401b8780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9b28006f60 tx=0x7f9b2800b970).stop 2021-04-09T18:12:47.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 -- 172.21.15.32:0/369482321 shutdown_connections 2021-04-09T18:12:47.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b24018930 0x7f9b2401ae10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b4008d1d0 0x7f9b401805b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b4008c7b0 0x7f9b4008cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 --2- 172.21.15.32:0/369482321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b401b8380 0x7f9b401b8780 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 -- 172.21.15.32:0/369482321 >> 172.21.15.32:0/369482321 conn(0x7f9b40189800 msgr2=0x7f9b401b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 -- 172.21.15.32:0/369482321 shutdown_connections 2021-04-09T18:12:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.049 7f9b46e22700 1 -- 172.21.15.32:0/369482321 wait complete. 2021-04-09T18:12:47.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.4 2021-04-09T18:12:47.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2021-04-09T18:12:47.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.259 7f723b295700 1 Processor -- start 2021-04-09T18:12:47.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.259 7f723b295700 1 -- start start 2021-04-09T18:12:47.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f723b295700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f72340d6c90 0x7f72340d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f723b295700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72340d2450 0x7f723410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f723b295700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72340d7290 con 0x7f72340cffb0 2021-04-09T18:12:47.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f723b295700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72340d7090 con 0x7f72340d2450 2021-04-09T18:12:47.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f723b295700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72340d91f0 con 0x7f72340d6c90 2021-04-09T18:12:47.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f72398b2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72340d2450 0x7f723410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f72398b2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72340d2450 0x7f723410cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41868/0 (socket says 172.21.15.32:41868) 2021-04-09T18:12:47.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f72398b2700 1 -- 172.21.15.32:0/1140811755 learned_addr learned my addr 172.21.15.32:0/1140811755 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:47.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f72388b0700 1 --2- 172.21.15.32:0/1140811755 >> v2:172.21.15.86:3300/0 conn(0x7f72340d6c90 0x7f72340d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f72398b2700 1 -- 172.21.15.32:0/1140811755 >> v2:172.21.15.86:3300/0 conn(0x7f72340d6c90 msgr2=0x7f72340d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:47.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f72398b2700 1 --2- 172.21.15.32:0/1140811755 >> v2:172.21.15.86:3300/0 conn(0x7f72340d6c90 0x7f72340d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f72398b2700 1 -- 172.21.15.32:0/1140811755 >> v1:172.21.15.86:6789/0 conn(0x7f72340cffb0 legacy=0x7f72340d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f72398b2700 1 -- 172.21.15.32:0/1140811755 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72340d2850 con 0x7f72340d2450 2021-04-09T18:12:47.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.260 7f72398b2700 1 --2- 172.21.15.32:0/1140811755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340d2450 0x7f723410cfe0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f722c006f00 tx=0x7f722c009e90).ready entity=mon.0 client_cookie=99eaad4794841697 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:47.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.261 7f722bfff700 1 -- 172.21.15.32:0/1140811755 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f722c00b300 con 0x7f72340d2450 2021-04-09T18:12:47.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.261 7f722bfff700 1 -- 172.21.15.32:0/1140811755 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c00b640 con 0x7f72340d2450 2021-04-09T18:12:47.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.261 7f722bfff700 1 -- 172.21.15.32:0/1140811755 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f722c00bb20 con 0x7f72340d2450 2021-04-09T18:12:47.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.261 7f723b295700 1 -- 172.21.15.32:0/1140811755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340d2450 msgr2=0x7f723410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.261 7f723b295700 1 --2- 172.21.15.32:0/1140811755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340d2450 0x7f723410cfe0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f722c006f00 tx=0x7f722c009e90).stop 2021-04-09T18:12:47.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.261 7f723b295700 1 -- 172.21.15.32:0/1140811755 shutdown_connections 2021-04-09T18:12:47.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.261 7f723b295700 1 --2- 172.21.15.32:0/1140811755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340d2450 0x7f723410cfe0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.261 7f723b295700 1 --2- 172.21.15.32:0/1140811755 >> v2:172.21.15.86:3300/0 conn(0x7f72340d6c90 0x7f72340d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.261 7f723b295700 1 -- 172.21.15.32:0/1140811755 >> 172.21.15.32:0/1140811755 conn(0x7f72340ce5d0 msgr2=0x7f72340cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:47.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.262 7f723b295700 1 -- 172.21.15.32:0/1140811755 shutdown_connections 2021-04-09T18:12:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.262 7f723b295700 1 -- 172.21.15.32:0/1140811755 wait complete. 2021-04-09T18:12:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f723b295700 1 Processor -- start 2021-04-09T18:12:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f723b295700 1 -- start start 2021-04-09T18:12:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f723b295700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f723408d210 0x7f723408fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f723b295700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340903f0 0x7f72341a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f723b295700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72341a6080 0x7f72341a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f723b295700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723408d690 con 0x7f72340903f0 2021-04-09T18:12:47.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f723b295700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72340b0fd0 con 0x7f72341a6080 2021-04-09T18:12:47.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f723b295700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72341a6480 con 0x7f723408d210 2021-04-09T18:12:47.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f72388b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340903f0 0x7f72341a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f72388b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340903f0 0x7f72341a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41872/0 (socket says 172.21.15.32:41872) 2021-04-09T18:12:47.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.282 7f72388b0700 1 -- 172.21.15.32:0/1698672566 learned_addr learned my addr 172.21.15.32:0/1698672566 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:47.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f72390b1700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f723408d210 0x7f723408fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f72398b2700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72341a6080 0x7f72341a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f72388b0700 1 -- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f723408d210 msgr2=0x7f723408fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f72388b0700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f723408d210 0x7f723408fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f72388b0700 1 -- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72341a6080 msgr2=0x7f72341a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f72388b0700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72341a6080 0x7f72341a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f72388b0700 1 -- 172.21.15.32:0/1698672566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723408d870 con 0x7f72340903f0 2021-04-09T18:12:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f72398b2700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72341a6080 0x7f72341a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:12:47.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f72388b0700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340903f0 0x7f72341a5a60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f7224006f60 tx=0x7f722400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:47.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f7229ffb700 1 -- 172.21.15.32:0/1698672566 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f722400bd60 con 0x7f72340903f0 2021-04-09T18:12:47.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f7229ffb700 1 -- 172.21.15.32:0/1698672566 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7224000c90 con 0x7f72340903f0 2021-04-09T18:12:47.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f7229ffb700 1 -- 172.21.15.32:0/1698672566 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7224002220 con 0x7f72340903f0 2021-04-09T18:12:47.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f723b295700 1 -- 172.21.15.32:0/1698672566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723408d870 con 0x7f72340903f0 2021-04-09T18:12:47.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.283 7f723b295700 1 -- 172.21.15.32:0/1698672566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723408d870 con 0x7f72340903f0 2021-04-09T18:12:47.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.284 7f7229ffb700 1 -- 172.21.15.32:0/1698672566 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f72240052e0 con 0x7f72340903f0 2021-04-09T18:12:47.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.284 7f7229ffb700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72140188c0 0x7f721401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.284 7f7229ffb700 1 -- 172.21.15.32:0/1698672566 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7224032770 con 0x7f72340903f0 2021-04-09T18:12:47.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.284 7f72390b1700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72140188c0 0x7f721401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.284 7f72390b1700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72140188c0 0x7f721401adc0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:47.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.304 7f723b295700 1 -- 172.21.15.32:0/1698672566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72341a52b0 con 0x7f72340903f0 2021-04-09T18:12:47.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.307 7f7229ffb700 1 -- 172.21.15.32:0/1698672566 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7224006b00 con 0x7f72340903f0 2021-04-09T18:12:47.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.508 7f723b295700 1 -- 172.21.15.32:0/1698672566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f72341a52b0 con 0x7f72340903f0 2021-04-09T18:12:47.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.508 7f7229ffb700 1 -- 172.21.15.32:0/1698672566 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f72240052e0 con 0x7f72340903f0 2021-04-09T18:12:47.529 INFO:teuthology.orchestra.run.smithi032.stdout:55834574856 2021-04-09T18:12:47.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 -- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72140188c0 msgr2=0x7f721401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72140188c0 0x7f721401adc0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 -- 172.21.15.32:0/1698672566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340903f0 msgr2=0x7f72341a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340903f0 0x7f72341a5a60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f7224006f60 tx=0x7f722400b970).stop 2021-04-09T18:12:47.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 -- 172.21.15.32:0/1698672566 shutdown_connections 2021-04-09T18:12:47.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72140188c0 0x7f721401adc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72340903f0 0x7f72341a5a60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72341a6080 0x7f72341a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 --2- 172.21.15.32:0/1698672566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f723408d210 0x7f723408fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 -- 172.21.15.32:0/1698672566 >> 172.21.15.32:0/1698672566 conn(0x7f72341a8f50 msgr2=0x7f7234188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:47.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 -- 172.21.15.32:0/1698672566 shutdown_connections 2021-04-09T18:12:47.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.511 7f723b295700 1 -- 172.21.15.32:0/1698672566 wait complete. 2021-04-09T18:12:47.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.5 2021-04-09T18:12:47.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2021-04-09T18:12:47.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.735 7fe38f1b8700 1 Processor -- start 2021-04-09T18:12:47.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe38f1b8700 1 -- start start 2021-04-09T18:12:47.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe38f1b8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3880cffb0 0x7fe3880d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe38f1b8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe3880d6c90 0x7fe38810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe38f1b8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3880d7290 con 0x7fe3880d7810 2021-04-09T18:12:47.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe38f1b8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3880d7090 con 0x7fe3880cffb0 2021-04-09T18:12:47.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe38f1b8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3880d91f0 con 0x7fe3880d6c90 2021-04-09T18:12:47.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe38cfd4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe3880cffb0 0x7fe3880d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe387fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe3880d7810 0x7fe3880d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51470/0 (socket says 172.21.15.32:51470) 2021-04-09T18:12:47.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe387fff700 1 -- 172.21.15.32:0/964472400 learned_addr learned my addr 172.21.15.32:0/964472400 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:47.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.736 7fe38d7d5700 1 --2- 172.21.15.32:0/964472400 >> v2:172.21.15.86:3300/0 conn(0x7fe3880d6c90 0x7fe38810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe3877fe700 1 -- 172.21.15.32:0/964472400 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332987655 0 0) 0x7fe37c000ce0 con 0x7fe3880d7810 2021-04-09T18:12:47.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe3877fe700 1 -- 172.21.15.32:0/964472400 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe370002fe0 con 0x7fe3880d7810 2021-04-09T18:12:47.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe38cfd4700 1 -- 172.21.15.32:0/964472400 >> v2:172.21.15.86:3300/0 conn(0x7fe3880d6c90 msgr2=0x7fe38810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:47.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe38cfd4700 1 --2- 172.21.15.32:0/964472400 >> v2:172.21.15.86:3300/0 conn(0x7fe3880d6c90 0x7fe38810cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe38cfd4700 1 -- 172.21.15.32:0/964472400 >> v1:172.21.15.32:6789/0 conn(0x7fe3880d7810 legacy=0x7fe3880d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe38cfd4700 1 -- 172.21.15.32:0/964472400 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3880d8400 con 0x7fe3880cffb0 2021-04-09T18:12:47.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe38cfd4700 1 --2- 172.21.15.32:0/964472400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3880cffb0 0x7fe3880d2450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe378006650 tx=0x7fe378009a80).ready entity=mon.0 client_cookie=536a8eed23f1f3ab server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:47.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe3877fe700 1 -- 172.21.15.32:0/964472400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3780009b0 con 0x7fe3880cffb0 2021-04-09T18:12:47.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe3877fe700 1 -- 172.21.15.32:0/964472400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37800c0c0 con 0x7fe3880cffb0 2021-04-09T18:12:47.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe3877fe700 1 -- 172.21.15.32:0/964472400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3780009b0 con 0x7fe3880cffb0 2021-04-09T18:12:47.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe38f1b8700 1 -- 172.21.15.32:0/964472400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3880cffb0 msgr2=0x7fe3880d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.737 7fe38f1b8700 1 --2- 172.21.15.32:0/964472400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3880cffb0 0x7fe3880d2450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe378006650 tx=0x7fe378009a80).stop 2021-04-09T18:12:47.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.738 7fe38f1b8700 1 -- 172.21.15.32:0/964472400 shutdown_connections 2021-04-09T18:12:47.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.738 7fe38f1b8700 1 --2- 172.21.15.32:0/964472400 >> v2:172.21.15.86:3300/0 conn(0x7fe3880d6c90 0x7fe38810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.738 7fe38f1b8700 1 --2- 172.21.15.32:0/964472400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3880cffb0 0x7fe3880d2450 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.738 7fe38f1b8700 1 -- 172.21.15.32:0/964472400 >> 172.21.15.32:0/964472400 conn(0x7fe3880ce5d0 msgr2=0x7fe3880cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:47.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.738 7fe38f1b8700 1 -- 172.21.15.32:0/964472400 shutdown_connections 2021-04-09T18:12:47.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.738 7fe38f1b8700 1 -- 172.21.15.32:0/964472400 wait complete. 2021-04-09T18:12:47.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.758 7fe38f1b8700 1 Processor -- start 2021-04-09T18:12:47.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.758 7fe38f1b8700 1 -- start start 2021-04-09T18:12:47.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe38f1b8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe38808a620 0x7fe38808b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe38f1b8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe38808b6d0 0x7fe388180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe38f1b8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe388180c50 0x7fe388181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe38f1b8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38808e660 con 0x7fe38808b6d0 2021-04-09T18:12:47.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe38f1b8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38808ab30 con 0x7fe388180c50 2021-04-09T18:12:47.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe38f1b8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38808bad0 con 0x7fe38808a620 2021-04-09T18:12:47.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe387fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe38808b6d0 0x7fe388180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe387fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe38808b6d0 0x7fe388180630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41886/0 (socket says 172.21.15.32:41886) 2021-04-09T18:12:47.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe387fff700 1 -- 172.21.15.32:0/2134950394 learned_addr learned my addr 172.21.15.32:0/2134950394 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:47.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe38cfd4700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe38808a620 0x7fe38808b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe38d7d5700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe388180c50 0x7fe388181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe387fff700 1 -- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe38808a620 msgr2=0x7fe38808b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe387fff700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe38808a620 0x7fe38808b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe387fff700 1 -- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe388180c50 msgr2=0x7fe388181050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe387fff700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe388180c50 0x7fe388181050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe387fff700 1 -- 172.21.15.32:0/2134950394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38808e1f0 con 0x7fe38808b6d0 2021-04-09T18:12:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe387fff700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe38808b6d0 0x7fe388180630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe37c00a4f0 tx=0x7fe37c00a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe3857fa700 1 -- 172.21.15.32:0/2134950394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe37c000ef0 con 0x7fe38808b6d0 2021-04-09T18:12:47.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe3857fa700 1 -- 172.21.15.32:0/2134950394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c00ac80 con 0x7fe38808b6d0 2021-04-09T18:12:47.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.759 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38808e1f0 con 0x7fe38808b6d0 2021-04-09T18:12:47.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.760 7fe3857fa700 1 -- 172.21.15.32:0/2134950394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe37c004590 con 0x7fe38808b6d0 2021-04-09T18:12:47.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.760 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38808e1f0 con 0x7fe38808b6d0 2021-04-09T18:12:47.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.760 7fe3857fa700 1 -- 172.21.15.32:0/2134950394 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe37c0043a0 con 0x7fe38808b6d0 2021-04-09T18:12:47.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.760 7fe3857fa700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe368018900 0x7fe36801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:47.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.760 7fe3857fa700 1 -- 172.21.15.32:0/2134950394 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe37c032a20 con 0x7fe38808b6d0 2021-04-09T18:12:47.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.761 7fe38cfd4700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe368018900 0x7fe36801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:47.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.761 7fe38cfd4700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe368018900 0x7fe36801adc0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:47.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.780 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3881b7180 con 0x7fe38808b6d0 2021-04-09T18:12:47.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.784 7fe3857fa700 1 -- 172.21.15.32:0/2134950394 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe37c0009b0 con 0x7fe38808b6d0 2021-04-09T18:12:48.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.983 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe3880d7290 con 0x7fe38808b6d0 2021-04-09T18:12:48.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.984 7fe3857fa700 1 -- 172.21.15.32:0/2134950394 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe37c0043a0 con 0x7fe38808b6d0 2021-04-09T18:12:48.004 INFO:teuthology.orchestra.run.smithi032.stdout:55834574854 2021-04-09T18:12:48.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe368018900 msgr2=0x7fe36801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe368018900 0x7fe36801adc0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe38808b6d0 msgr2=0x7fe388180630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe38808b6d0 0x7fe388180630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe37c00a4f0 tx=0x7fe37c00a890).stop 2021-04-09T18:12:48.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 shutdown_connections 2021-04-09T18:12:48.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe368018900 0x7fe36801adc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe38808b6d0 0x7fe388180630 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe388180c50 0x7fe388181050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 --2- 172.21.15.32:0/2134950394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe38808a620 0x7fe38808b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 >> 172.21.15.32:0/2134950394 conn(0x7fe388189800 msgr2=0x7fe3881b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 shutdown_connections 2021-04-09T18:12:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:47.986 7fe38f1b8700 1 -- 172.21.15.32:0/2134950394 wait complete. 2021-04-09T18:12:48.016 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.6 2021-04-09T18:12:48.017 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2021-04-09T18:12:48.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.150 7f2a3c34c700 1 Processor -- start 2021-04-09T18:12:48.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.150 7f2a3c34c700 1 -- start start 2021-04-09T18:12:48.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3c34c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a340cffb0 0x7f2a340d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3c34c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a340d6c90 0x7f2a3410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3c34c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a340d7290 con 0x7f2a340d7810 2021-04-09T18:12:48.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3c34c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a340d7090 con 0x7f2a340d6c90 2021-04-09T18:12:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3c34c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a340d8590 con 0x7f2a340cffb0 2021-04-09T18:12:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3a969700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a340d6c90 0x7f2a3410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3a969700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a340d6c90 0x7f2a3410cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41894/0 (socket says 172.21.15.32:41894) 2021-04-09T18:12:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3a969700 1 -- 172.21.15.32:0/3375067326 learned_addr learned my addr 172.21.15.32:0/3375067326 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3a168700 1 --2- 172.21.15.32:0/3375067326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a340cffb0 0x7f2a340d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3a969700 1 -- 172.21.15.32:0/3375067326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a340cffb0 msgr2=0x7f2a340d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3a969700 1 --2- 172.21.15.32:0/3375067326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a340cffb0 0x7f2a340d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3a969700 1 -- 172.21.15.32:0/3375067326 >> v1:172.21.15.86:6789/0 conn(0x7f2a340d7810 legacy=0x7f2a340d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3a969700 1 -- 172.21.15.32:0/3375067326 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a340d91f0 con 0x7f2a340d6c90 2021-04-09T18:12:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3a969700 1 --2- 172.21.15.32:0/3375067326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a340d6c90 0x7f2a3410cfe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2a2c006f00 tx=0x7f2a2c009f10).ready entity=mon.0 client_cookie=77a3cdd767b089c2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a39166700 1 -- 172.21.15.32:0/3375067326 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a2c00b3b0 con 0x7f2a340d6c90 2021-04-09T18:12:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a39166700 1 -- 172.21.15.32:0/3375067326 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2c00b730 con 0x7f2a340d6c90 2021-04-09T18:12:48.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a39166700 1 -- 172.21.15.32:0/3375067326 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a2c00bc10 con 0x7f2a340d6c90 2021-04-09T18:12:48.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3c34c700 1 -- 172.21.15.32:0/3375067326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a340d6c90 msgr2=0x7f2a3410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.151 7f2a3c34c700 1 --2- 172.21.15.32:0/3375067326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a340d6c90 0x7f2a3410cfe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2a2c006f00 tx=0x7f2a2c009f10).stop 2021-04-09T18:12:48.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.152 7f2a3c34c700 1 -- 172.21.15.32:0/3375067326 shutdown_connections 2021-04-09T18:12:48.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.152 7f2a3c34c700 1 --2- 172.21.15.32:0/3375067326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a340d6c90 0x7f2a3410cfe0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.152 7f2a3c34c700 1 --2- 172.21.15.32:0/3375067326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a340cffb0 0x7f2a340d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.152 7f2a3c34c700 1 -- 172.21.15.32:0/3375067326 >> 172.21.15.32:0/3375067326 conn(0x7f2a340ce5d0 msgr2=0x7f2a340cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:48.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.152 7f2a3c34c700 1 -- 172.21.15.32:0/3375067326 shutdown_connections 2021-04-09T18:12:48.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.152 7f2a3c34c700 1 -- 172.21.15.32:0/3375067326 wait complete. 2021-04-09T18:12:48.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.172 7f2a3c34c700 1 Processor -- start 2021-04-09T18:12:48.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.172 7f2a3c34c700 1 -- start start 2021-04-09T18:12:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a3c34c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a34190210 0x7f2a341a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a3c34c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a341aa0b0 0x7f2a341923f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a3c34c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a34192a10 0x7f2a340926e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a3c34c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a340a5f30 con 0x7f2a341aa0b0 2021-04-09T18:12:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a3c34c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a34192e10 con 0x7f2a34192a10 2021-04-09T18:12:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a3c34c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a34092d00 con 0x7f2a34190210 2021-04-09T18:12:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a39967700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a341aa0b0 0x7f2a341923f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a39967700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a341aa0b0 0x7f2a341923f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41900/0 (socket says 172.21.15.32:41900) 2021-04-09T18:12:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a39967700 1 -- 172.21.15.32:0/2153758585 learned_addr learned my addr 172.21.15.32:0/2153758585 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a3a168700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a34190210 0x7f2a341a9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a3a969700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a34192a10 0x7f2a340926e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a39967700 1 -- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a34190210 msgr2=0x7f2a341a9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a39967700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a34190210 0x7f2a341a9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a39967700 1 -- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a34192a10 msgr2=0x7f2a340926e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a39967700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a34192a10 0x7f2a340926e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.173 7f2a39967700 1 -- 172.21.15.32:0/2153758585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a34092f20 con 0x7f2a341aa0b0 2021-04-09T18:12:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.174 7f2a39967700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a341aa0b0 0x7f2a341923f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2a2400b5b0 tx=0x7f2a2400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.174 7f2a2affd700 1 -- 172.21.15.32:0/2153758585 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a2400bd40 con 0x7f2a341aa0b0 2021-04-09T18:12:48.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.174 7f2a2affd700 1 -- 172.21.15.32:0/2153758585 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a24000e20 con 0x7f2a341aa0b0 2021-04-09T18:12:48.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.174 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a34092f20 con 0x7f2a341aa0b0 2021-04-09T18:12:48.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.174 7f2a2affd700 1 -- 172.21.15.32:0/2153758585 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a240054a0 con 0x7f2a341aa0b0 2021-04-09T18:12:48.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.174 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a34092f20 con 0x7f2a341aa0b0 2021-04-09T18:12:48.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.174 7f2a2affd700 1 -- 172.21.15.32:0/2153758585 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2a24006030 con 0x7f2a341aa0b0 2021-04-09T18:12:48.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.175 7f2a2affd700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a1c0188d0 0x7f2a1c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.175 7f2a2affd700 1 -- 172.21.15.32:0/2153758585 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2a24032a20 con 0x7f2a341aa0b0 2021-04-09T18:12:48.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.175 7f2a3a168700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a1c0188d0 0x7f2a1c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.175 7f2a3a168700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a1c0188d0 0x7f2a1c01add0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:48.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.195 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a340a5f30 con 0x7f2a341aa0b0 2021-04-09T18:12:48.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.198 7f2a2affd700 1 -- 172.21.15.32:0/2153758585 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2a2400bd40 con 0x7f2a341aa0b0 2021-04-09T18:12:48.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.400 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2a340a5f30 con 0x7f2a341aa0b0 2021-04-09T18:12:48.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.400 7f2a2affd700 1 -- 172.21.15.32:0/2153758585 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2a24000e20 con 0x7f2a341aa0b0 2021-04-09T18:12:48.421 INFO:teuthology.orchestra.run.smithi032.stdout:55834574855 2021-04-09T18:12:48.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a1c0188d0 msgr2=0x7f2a1c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a1c0188d0 0x7f2a1c01add0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a341aa0b0 msgr2=0x7f2a341923f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a341aa0b0 0x7f2a341923f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2a2400b5b0 tx=0x7f2a2400b950).stop 2021-04-09T18:12:48.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 shutdown_connections 2021-04-09T18:12:48.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a1c0188d0 0x7f2a1c01add0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a341aa0b0 0x7f2a341923f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a34192a10 0x7f2a340926e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 --2- 172.21.15.32:0/2153758585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a34190210 0x7f2a341a9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 >> 172.21.15.32:0/2153758585 conn(0x7f2a341ad170 msgr2=0x7f2a341b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 shutdown_connections 2021-04-09T18:12:48.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.403 7f2a3c34c700 1 -- 172.21.15.32:0/2153758585 wait complete. 2021-04-09T18:12:48.474 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.7 2021-04-09T18:12:48.475 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2021-04-09T18:12:48.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-04-09T18:12:48.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.631 7ff13e9df700 1 Processor -- start 2021-04-09T18:12:48.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.631 7ff13e9df700 1 -- start start 2021-04-09T18:12:48.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.632 7ff13e9df700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff13810da00 0x7ff13810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.632 7ff13e9df700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13810d480 con 0x7ff1380d7ca0 2021-04-09T18:12:48.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.632 7ff13e9df700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1380d1f70 con 0x7ff13810d010 2021-04-09T18:12:48.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.632 7ff13e9df700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1380d8d30 con 0x7ff13810da00 2021-04-09T18:12:48.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.632 7ff12ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff1380d7ca0 0x7ff1380d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51496/0 (socket says 172.21.15.32:51496) 2021-04-09T18:12:48.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.632 7ff12ffff700 1 -- 172.21.15.32:0/1179473654 learned_addr learned my addr 172.21.15.32:0/1179473654 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:48.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.632 7ff13cffc700 1 --2- 172.21.15.32:0/1179473654 >> v2:172.21.15.86:3300/0 conn(0x7ff13810da00 0x7ff13810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.632 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507807682 0 0) 0x7ff120000ce0 con 0x7ff1380d7ca0 2021-04-09T18:12:48.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.632 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff124002fe0 con 0x7ff1380d7ca0 2021-04-09T18:12:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394827014 0 0) 0x7ff128000c60 con 0x7ff13810d010 2021-04-09T18:12:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff124003210 con 0x7ff13810d010 2021-04-09T18:12:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2753728156 0 0) 0x7ff120000ed0 con 0x7ff1380d7ca0 2021-04-09T18:12:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 >> v2:172.21.15.86:3300/0 conn(0x7ff13810da00 msgr2=0x7ff13810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:48.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 --2- 172.21.15.32:0/1179473654 >> v2:172.21.15.86:3300/0 conn(0x7ff13810da00 0x7ff13810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 >> v1:172.21.15.86:6789/0 conn(0x7ff13810d010 legacy=0x7ff1380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1380d9220 con 0x7ff1380d7ca0 2021-04-09T18:12:48.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7ff120001600 con 0x7ff1380d7ca0 2021-04-09T18:12:48.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff13cffc700 1 --2- 172.21.15.32:0/1179473654 >> v2:172.21.15.86:3300/0 conn(0x7ff13810da00 0x7ff13810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:48.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 >> v1:172.21.15.32:6789/0 conn(0x7ff1380d7ca0 legacy=0x7ff1380d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff124003b90 0x7ff124005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1240067a0 0x7ff124008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1240095a0 0x7ff12400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff137fff700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff124003b90 0x7ff124005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff124050320 con 0x7ff124003b90 2021-04-09T18:12:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff124050540 con 0x7ff1240067a0 2021-04-09T18:12:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff124050760 con 0x7ff1240095a0 2021-04-09T18:12:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff13cffc700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1240095a0 0x7ff12400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff12ffff700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1240067a0 0x7ff124008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff137fff700 1 -- 172.21.15.32:0/1179473654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1240095a0 msgr2=0x7ff12400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:48.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff137fff700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1240095a0 0x7ff12400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff137fff700 1 -- 172.21.15.32:0/1179473654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1240067a0 msgr2=0x7ff124008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:48.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff137fff700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1240067a0 0x7ff124008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff137fff700 1 -- 172.21.15.32:0/1179473654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff124050a00 con 0x7ff124003b90 2021-04-09T18:12:48.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff13cffc700 1 -- 172.21.15.32:0/1179473654 reap_dead start 2021-04-09T18:12:48.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff137fff700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff124003b90 0x7ff124005fa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7ff128001040 tx=0x7ff128009a20).ready entity=mon.0 client_cookie=f663780c1f31e17 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:48.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff128000c10 con 0x7ff124003b90 2021-04-09T18:12:48.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1280066e0 con 0x7ff124003b90 2021-04-09T18:12:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff1377fe700 1 -- 172.21.15.32:0/1179473654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff128000c10 con 0x7ff124003b90 2021-04-09T18:12:48.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff13e9df700 1 -- 172.21.15.32:0/1179473654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff124003b90 msgr2=0x7ff124005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.633 7ff13e9df700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff124003b90 0x7ff124005fa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7ff128001040 tx=0x7ff128009a20).stop 2021-04-09T18:12:48.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.634 7ff13e9df700 1 -- 172.21.15.32:0/1179473654 shutdown_connections 2021-04-09T18:12:48.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.634 7ff13e9df700 1 --2- 172.21.15.32:0/1179473654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff124003b90 0x7ff124005fa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.634 7ff13e9df700 1 -- 172.21.15.32:0/1179473654 >> 172.21.15.32:0/1179473654 conn(0x7ff1380ce5f0 msgr2=0x7ff1380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:48.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.634 7ff13e9df700 1 -- 172.21.15.32:0/1179473654 shutdown_connections 2021-04-09T18:12:48.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.634 7ff13e9df700 1 -- 172.21.15.32:0/1179473654 wait complete. 2021-04-09T18:12:48.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff13e9df700 1 Processor -- start 2021-04-09T18:12:48.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff13e9df700 1 -- start start 2021-04-09T18:12:48.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff13e9df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1381ac380 0x7ff13818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff13e9df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381ac780 0x7ff13818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff13e9df700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff138183520 0x7ff138183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:48.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff13e9df700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1381ace10 con 0x7ff1381ac780 2021-04-09T18:12:48.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff13e9df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff138189cf0 con 0x7ff1381ac380 2021-04-09T18:12:48.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff13e9df700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff138183f40 con 0x7ff138183520 2021-04-09T18:12:48.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff12ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381ac780 0x7ff13818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff12ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381ac780 0x7ff13818a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41918/0 (socket says 172.21.15.32:41918) 2021-04-09T18:12:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff12ffff700 1 -- 172.21.15.32:0/1803913106 learned_addr learned my addr 172.21.15.32:0/1803913106 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff137fff700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1381ac380 0x7ff13818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.655 7ff13cffc700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff138183520 0x7ff138183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:48.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff12ffff700 1 -- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff138183520 msgr2=0x7ff138183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff12ffff700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff138183520 0x7ff138183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:48.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff12ffff700 1 -- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1381ac380 msgr2=0x7ff13818a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:48.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff12ffff700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1381ac380 0x7ff13818a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff12ffff700 1 -- 172.21.15.32:0/1803913106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138184160 con 0x7ff1381ac780 2021-04-09T18:12:49.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff12ffff700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381ac780 0x7ff13818a850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7ff120001170 tx=0x7ff12000a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:49.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff1357fa700 1 -- 172.21.15.32:0/1803913106 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff120001c20 con 0x7ff1381ac780 2021-04-09T18:12:49.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff1357fa700 1 -- 172.21.15.32:0/1803913106 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff120007280 con 0x7ff1381ac780 2021-04-09T18:12:49.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff138184160 con 0x7ff1381ac780 2021-04-09T18:12:49.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff1357fa700 1 -- 172.21.15.32:0/1803913106 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1200043a0 con 0x7ff1381ac780 2021-04-09T18:12:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.656 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff138184160 con 0x7ff1381ac780 2021-04-09T18:12:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.657 7ff1357fa700 1 -- 172.21.15.32:0/1803913106 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff1200043a0 con 0x7ff1381ac780 2021-04-09T18:12:49.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.657 7ff1357fa700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff11c0188c0 0x7ff11c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:49.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.657 7ff1357fa700 1 -- 172.21.15.32:0/1803913106 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff120032760 con 0x7ff1381ac780 2021-04-09T18:12:49.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.657 7ff137fff700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff11c0188c0 0x7ff11c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:49.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.658 7ff137fff700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff11c0188c0 0x7ff11c01adc0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:49.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.676 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff138189cf0 con 0x7ff1381ac780 2021-04-09T18:12:49.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.679 7ff1357fa700 1 -- 172.21.15.32:0/1803913106 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff120004750 con 0x7ff1381ac780 2021-04-09T18:12:49.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.871 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 --> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7ff138189cf0 con 0x7ff11c0188c0 2021-04-09T18:12:49.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.874 7ff1357fa700 1 -- 172.21.15.32:0/1803913106 <== mgr.4102 v2:172.21.15.86:6800/23869 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+77159 (crc 0 0 0) 0x7ff128006980 con 0x7ff11c0188c0 2021-04-09T18:12:49.169 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2021-04-09T18:12:49.201 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:49.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff11c0188c0 msgr2=0x7ff11c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:49.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff11c0188c0 0x7ff11c01adc0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381ac780 msgr2=0x7ff13818a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:49.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381ac780 0x7ff13818a850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7ff120001170 tx=0x7ff12000a8a0).stop 2021-04-09T18:12:49.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 shutdown_connections 2021-04-09T18:12:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff11c0188c0 0x7ff11c01adc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381ac780 0x7ff13818a850 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff138183520 0x7ff138183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 --2- 172.21.15.32:0/1803913106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1381ac380 0x7ff13818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 >> 172.21.15.32:0/1803913106 conn(0x7ff13819b890 msgr2=0x7ff1381a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 shutdown_connections 2021-04-09T18:12:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:48.876 7ff13e9df700 1 -- 172.21.15.32:0/1803913106 wait complete. 2021-04-09T18:12:49.230 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2021-04-09 18:12:48.257404","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8390336,"kb_used_data":1664,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584384,"statfs":{"total":773094113280,"available":764502409216,"internally_reserved":8589934592,"allocated":1703936,"data_stored":554676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1526,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000994"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:34.452606","last_change":"2021-04-09 18:12:30.319071","last_active":"2021-04-09 18:12:34.452606","last_peered":"2021-04-09 18:12:34.452606","last_clean":"2021-04-09 18:12:34.452606","last_became_active":"2021-04-09 18:12:30.318784","last_became_peered":"2021-04-09 18:12:30.318784","last_unstale":"2021-04-09 18:12:34.452606","last_undegraded":"2021-04-09 18:12:34.452606","last_fullsized":"2021-04-09 18:12:34.452606","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.808546","last_change":"2021-04-09 18:12:33.331423","last_active":"2021-04-09 18:12:41.808546","last_peered":"2021-04-09 18:12:41.808546","last_clean":"2021-04-09 18:12:41.808546","last_became_active":"2021-04-09 18:12:33.331079","last_became_peered":"2021-04-09 18:12:33.331079","last_unstale":"2021-04-09 18:12:41.808546","last_undegraded":"2021-04-09 18:12:41.808546","last_fullsized":"2021-04-09 18:12:41.808546","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.783821","last_change":"2021-04-09 18:12:33.781125","last_active":"2021-04-09 18:12:33.783821","last_peered":"2021-04-09 18:12:33.783821","last_clean":"2021-04-09 18:12:33.783821","last_became_active":"2021-04-09 18:12:33.780825","last_became_peered":"2021-04-09 18:12:33.780825","last_unstale":"2021-04-09 18:12:33.783821","last_undegraded":"2021-04-09 18:12:33.783821","last_fullsized":"2021-04-09 18:12:33.783821","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:34.452588","last_change":"2021-04-09 18:12:30.316459","last_active":"2021-04-09 18:12:34.452588","last_peered":"2021-04-09 18:12:34.452588","last_clean":"2021-04-09 18:12:34.452588","last_became_active":"2021-04-09 18:12:30.316115","last_became_peered":"2021-04-09 18:12:30.316115","last_unstale":"2021-04-09 18:12:34.452588","last_undegraded":"2021-04-09 18:12:34.452588","last_fullsized":"2021-04-09 18:12:34.452588","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"20'6","reported_seq":"16","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.811071","last_change":"2021-04-09 18:12:33.333374","last_active":"2021-04-09 18:12:41.811071","last_peered":"2021-04-09 18:12:41.811071","last_clean":"2021-04-09 18:12:41.811071","last_became_active":"2021-04-09 18:12:33.333035","last_became_peered":"2021-04-09 18:12:33.333035","last_unstale":"2021-04-09 18:12:41.811071","last_undegraded":"2021-04-09 18:12:41.811071","last_fullsized":"2021-04-09 18:12:41.811071","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779703","last_change":"2021-04-09 18:12:33.779202","last_active":"2021-04-09 18:12:33.779703","last_peered":"2021-04-09 18:12:33.779703","last_clean":"2021-04-09 18:12:33.779703","last_became_active":"2021-04-09 18:12:33.778916","last_became_peered":"2021-04-09 18:12:33.778916","last_unstale":"2021-04-09 18:12:33.779703","last_undegraded":"2021-04-09 18:12:33.779703","last_fullsized":"2021-04-09 18:12:33.779703","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.781925","last_change":"2021-04-09 18:12:33.781366","last_active":"2021-04-09 18:12:33.781925","last_peered":"2021-04-09 18:12:33.781925","last_clean":"2021-04-09 18:12:33.781925","last_became_active":"2021-04-09 18:12:33.781061","last_became_peered":"2021-04-09 18:12:33.781061","last_unstale":"2021-04-09 18:12:33.781925","last_undegraded":"2021-04-09 18:12:33.781925","last_fullsized":"2021-04-09 18:12:33.781925","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"11","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779941","last_change":"2021-04-09 18:12:30.316199","last_active":"2021-04-09 18:12:33.779941","last_peered":"2021-04-09 18:12:33.779941","last_clean":"2021-04-09 18:12:33.779941","last_became_active":"2021-04-09 18:12:30.315813","last_became_peered":"2021-04-09 18:12:30.315813","last_unstale":"2021-04-09 18:12:33.779941","last_undegraded":"2021-04-09 18:12:33.779941","last_fullsized":"2021-04-09 18:12:33.779941","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"20'3","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.810882","last_change":"2021-04-09 18:12:33.333344","last_active":"2021-04-09 18:12:41.810882","last_peered":"2021-04-09 18:12:41.810882","last_clean":"2021-04-09 18:12:41.810882","last_became_active":"2021-04-09 18:12:33.332949","last_became_peered":"2021-04-09 18:12:33.332949","last_unstale":"2021-04-09 18:12:41.810882","last_undegraded":"2021-04-09 18:12:41.810882","last_fullsized":"2021-04-09 18:12:41.810882","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.778748","last_change":"2021-04-09 18:12:33.777893","last_active":"2021-04-09 18:12:33.778748","last_peered":"2021-04-09 18:12:33.778748","last_clean":"2021-04-09 18:12:33.778748","last_became_active":"2021-04-09 18:12:33.777554","last_became_peered":"2021-04-09 18:12:33.777554","last_unstale":"2021-04-09 18:12:33.778748","last_undegraded":"2021-04-09 18:12:33.778748","last_fullsized":"2021-04-09 18:12:33.778748","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"12","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.776357","last_change":"2021-04-09 18:12:30.313110","last_active":"2021-04-09 18:12:33.776357","last_peered":"2021-04-09 18:12:33.776357","last_clean":"2021-04-09 18:12:33.776357","last_became_active":"2021-04-09 18:12:30.312615","last_became_peered":"2021-04-09 18:12:30.312615","last_unstale":"2021-04-09 18:12:33.776357","last_undegraded":"2021-04-09 18:12:33.776357","last_fullsized":"2021-04-09 18:12:33.776357","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":"17","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.813860","last_change":"2021-04-09 18:12:33.333727","last_active":"2021-04-09 18:12:41.813860","last_peered":"2021-04-09 18:12:41.813860","last_clean":"2021-04-09 18:12:41.813860","last_became_active":"2021-04-09 18:12:33.333326","last_became_peered":"2021-04-09 18:12:33.333326","last_unstale":"2021-04-09 18:12:41.813860","last_undegraded":"2021-04-09 18:12:41.813860","last_fullsized":"2021-04-09 18:12:41.813860","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779698","last_change":"2021-04-09 18:12:30.317416","last_active":"2021-04-09 18:12:33.779698","last_peered":"2021-04-09 18:12:33.779698","last_clean":"2021-04-09 18:12:33.779698","last_became_active":"2021-04-09 18:12:30.317112","last_became_peered":"2021-04-09 18:12:30.317112","last_unstale":"2021-04-09 18:12:33.779698","last_undegraded":"2021-04-09 18:12:33.779698","last_fullsized":"2021-04-09 18:12:33.779698","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'5","reported_seq":"15","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.808651","last_change":"2021-04-09 18:12:33.329808","last_active":"2021-04-09 18:12:41.808651","last_peered":"2021-04-09 18:12:41.808651","last_clean":"2021-04-09 18:12:41.808651","last_became_active":"2021-04-09 18:12:33.329428","last_became_peered":"2021-04-09 18:12:33.329428","last_unstale":"2021-04-09 18:12:41.808651","last_undegraded":"2021-04-09 18:12:41.808651","last_fullsized":"2021-04-09 18:12:41.808651","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.781717","last_change":"2021-04-09 18:12:33.781025","last_active":"2021-04-09 18:12:33.781717","last_peered":"2021-04-09 18:12:33.781717","last_clean":"2021-04-09 18:12:33.781717","last_became_active":"2021-04-09 18:12:33.780701","last_became_peered":"2021-04-09 18:12:33.780701","last_unstale":"2021-04-09 18:12:33.781717","last_undegraded":"2021-04-09 18:12:33.781717","last_fullsized":"2021-04-09 18:12:33.781717","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779385","last_change":"2021-04-09 18:12:30.315339","last_active":"2021-04-09 18:12:33.779385","last_peered":"2021-04-09 18:12:33.779385","last_clean":"2021-04-09 18:12:33.779385","last_became_active":"2021-04-09 18:12:30.314975","last_became_peered":"2021-04-09 18:12:30.314975","last_unstale":"2021-04-09 18:12:33.779385","last_undegraded":"2021-04-09 18:12:33.779385","last_fullsized":"2021-04-09 18:12:33.779385","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":"11","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:34.787036","last_change":"2021-04-09 18:12:33.329984","last_active":"2021-04-09 18:12:34.787036","last_peered":"2021-04-09 18:12:34.787036","last_clean":"2021-04-09 18:12:34.787036","last_became_active":"2021-04-09 18:12:33.329565","last_became_peered":"2021-04-09 18:12:33.329565","last_unstale":"2021-04-09 18:12:34.787036","last_undegraded":"2021-04-09 18:12:34.787036","last_fullsized":"2021-04-09 18:12:34.787036","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.788904","last_change":"2021-04-09 18:12:33.784198","last_active":"2021-04-09 18:12:33.788904","last_peered":"2021-04-09 18:12:33.788904","last_clean":"2021-04-09 18:12:33.788904","last_became_active":"2021-04-09 18:12:33.783816","last_became_peered":"2021-04-09 18:12:33.783816","last_unstale":"2021-04-09 18:12:33.788904","last_undegraded":"2021-04-09 18:12:33.788904","last_fullsized":"2021-04-09 18:12:33.788904","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.778194","last_change":"2021-04-09 18:12:33.330844","last_active":"2021-04-09 18:12:33.778194","last_peered":"2021-04-09 18:12:33.778194","last_clean":"2021-04-09 18:12:33.778194","last_became_active":"2021-04-09 18:12:33.330206","last_became_peered":"2021-04-09 18:12:33.330206","last_unstale":"2021-04-09 18:12:33.778194","last_undegraded":"2021-04-09 18:12:33.778194","last_fullsized":"2021-04-09 18:12:33.778194","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779586","last_change":"2021-04-09 18:12:30.316686","last_active":"2021-04-09 18:12:33.779586","last_peered":"2021-04-09 18:12:33.779586","last_clean":"2021-04-09 18:12:33.779586","last_became_active":"2021-04-09 18:12:30.316397","last_became_peered":"2021-04-09 18:12:30.316397","last_unstale":"2021-04-09 18:12:33.779586","last_undegraded":"2021-04-09 18:12:33.779586","last_fullsized":"2021-04-09 18:12:33.779586","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.780687","last_change":"2021-04-09 18:12:33.779614","last_active":"2021-04-09 18:12:33.780687","last_peered":"2021-04-09 18:12:33.780687","last_clean":"2021-04-09 18:12:33.780687","last_became_active":"2021-04-09 18:12:33.778867","last_became_peered":"2021-04-09 18:12:33.778867","last_unstale":"2021-04-09 18:12:33.780687","last_undegraded":"2021-04-09 18:12:33.780687","last_fullsized":"2021-04-09 18:12:33.780687","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.811298","last_change":"2021-04-09 18:12:33.332835","last_active":"2021-04-09 18:12:41.811298","last_peered":"2021-04-09 18:12:41.811298","last_clean":"2021-04-09 18:12:41.811298","last_became_active":"2021-04-09 18:12:33.332485","last_became_peered":"2021-04-09 18:12:33.332485","last_unstale":"2021-04-09 18:12:41.811298","last_undegraded":"2021-04-09 18:12:41.811298","last_fullsized":"2021-04-09 18:12:41.811298","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.776726","last_change":"2021-04-09 18:12:30.316233","last_active":"2021-04-09 18:12:33.776726","last_peered":"2021-04-09 18:12:33.776726","last_clean":"2021-04-09 18:12:33.776726","last_became_active":"2021-04-09 18:12:30.315922","last_became_peered":"2021-04-09 18:12:30.315922","last_unstale":"2021-04-09 18:12:33.776726","last_undegraded":"2021-04-09 18:12:33.776726","last_fullsized":"2021-04-09 18:12:33.776726","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.784266","last_change":"2021-04-09 18:12:33.783881","last_active":"2021-04-09 18:12:33.784266","last_peered":"2021-04-09 18:12:33.784266","last_clean":"2021-04-09 18:12:33.784266","last_became_active":"2021-04-09 18:12:33.783671","last_became_peered":"2021-04-09 18:12:33.783671","last_unstale":"2021-04-09 18:12:33.784266","last_undegraded":"2021-04-09 18:12:33.784266","last_fullsized":"2021-04-09 18:12:33.784266","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":69489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048792,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323048,"statfs":{"total":96636764160,"available":95562801152,"internally_reserved":1073741824,"allocated":212992,"data_stored":69329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048804,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323036,"statfs":{"total":96636764160,"available":95562788864,"internally_reserved":1073741824,"allocated":225280,"data_stored":70449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.185}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.223}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":70289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]}]}} 2021-04-09T18:12:49.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-04-09T18:12:49.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.404 7fb1bd54e700 1 Processor -- start 2021-04-09T18:12:49.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.404 7fb1bd54e700 1 -- start start 2021-04-09T18:12:49.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.404 7fb1bd54e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b810d010 0x7fb1b80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:49.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.404 7fb1bd54e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb1b80d1da0 0x7fb1b80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:49.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1bd54e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b810d480 con 0x7fb1b80d82c0 2021-04-09T18:12:49.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1bd54e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1b810e780 con 0x7fb1b80d1da0 2021-04-09T18:12:49.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1bd54e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b80d8d30 con 0x7fb1b810d010 2021-04-09T18:12:49.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b7ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb1b80d82c0 0x7fb1b810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51516/0 (socket says 172.21.15.32:51516) 2021-04-09T18:12:49.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b7ac0700 1 -- 172.21.15.32:0/2365405842 learned_addr learned my addr 172.21.15.32:0/2365405842 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:12:49.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b6abe700 1 --2- 172.21.15.32:0/2365405842 >> v2:172.21.15.86:3300/0 conn(0x7fb1b80d1da0 0x7fb1b80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:49.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b72bf700 1 --2- 172.21.15.32:0/2365405842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b810d010 0x7fb1b80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:49.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b62bd700 1 -- 172.21.15.32:0/2365405842 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940811147 0 0) 0x7fb1b00009b0 con 0x7fb1b80d82c0 2021-04-09T18:12:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b62bd700 1 -- 172.21.15.32:0/2365405842 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a4002fe0 con 0x7fb1b80d82c0 2021-04-09T18:12:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b6abe700 1 -- 172.21.15.32:0/2365405842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b810d010 msgr2=0x7fb1b80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b6abe700 1 --2- 172.21.15.32:0/2365405842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b810d010 0x7fb1b80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b6abe700 1 -- 172.21.15.32:0/2365405842 >> v1:172.21.15.32:6789/0 conn(0x7fb1b80d82c0 legacy=0x7fb1b810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b6abe700 1 -- 172.21.15.32:0/2365405842 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b80d9220 con 0x7fb1b80d1da0 2021-04-09T18:12:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b72bf700 1 --2- 172.21.15.32:0/2365405842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b810d010 0x7fb1b80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b6abe700 1 --2- 172.21.15.32:0/2365405842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1b80d1da0 0x7fb1b80d7ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb1a80069d0 tx=0x7fb1a8009e40).ready entity=mon.1 client_cookie=543e2b812a3cf34c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:49.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b62bd700 1 -- 172.21.15.32:0/2365405842 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1a800b510 con 0x7fb1b80d1da0 2021-04-09T18:12:49.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b62bd700 1 -- 172.21.15.32:0/2365405842 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a800b890 con 0x7fb1b80d1da0 2021-04-09T18:12:49.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.405 7fb1b62bd700 1 -- 172.21.15.32:0/2365405842 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1a800bd70 con 0x7fb1b80d1da0 2021-04-09T18:12:49.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.406 7fb1bd54e700 1 -- 172.21.15.32:0/2365405842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1b80d1da0 msgr2=0x7fb1b80d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:49.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.406 7fb1bd54e700 1 --2- 172.21.15.32:0/2365405842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1b80d1da0 0x7fb1b80d7ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb1a80069d0 tx=0x7fb1a8009e40).stop 2021-04-09T18:12:49.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.406 7fb1bd54e700 1 -- 172.21.15.32:0/2365405842 shutdown_connections 2021-04-09T18:12:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.406 7fb1bd54e700 1 --2- 172.21.15.32:0/2365405842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1b80d1da0 0x7fb1b80d7ca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.406 7fb1bd54e700 1 --2- 172.21.15.32:0/2365405842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b810d010 0x7fb1b80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.406 7fb1bd54e700 1 -- 172.21.15.32:0/2365405842 >> 172.21.15.32:0/2365405842 conn(0x7fb1b80ce5f0 msgr2=0x7fb1b80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.406 7fb1bd54e700 1 -- 172.21.15.32:0/2365405842 shutdown_connections 2021-04-09T18:12:49.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.406 7fb1bd54e700 1 -- 172.21.15.32:0/2365405842 wait complete. 2021-04-09T18:12:49.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.426 7fb1bd54e700 1 Processor -- start 2021-04-09T18:12:49.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.426 7fb1bd54e700 1 -- start start 2021-04-09T18:12:49.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1bd54e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8186480 0x7fb1b808c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1bd54e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b8186880 0x7fb1b808c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1bd54e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1b808d680 0x7fb1b808da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:49.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1bd54e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b808bc60 con 0x7fb1b8186480 2021-04-09T18:12:49.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1bd54e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b808e0a0 con 0x7fb1b808d680 2021-04-09T18:12:49.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1bd54e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b808e2c0 con 0x7fb1b8186880 2021-04-09T18:12:49.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b72bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8186480 0x7fb1b808c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:49.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b72bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8186480 0x7fb1b808c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41930/0 (socket says 172.21.15.32:41930) 2021-04-09T18:12:49.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b72bf700 1 -- 172.21.15.32:0/3887741341 learned_addr learned my addr 172.21.15.32:0/3887741341 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:49.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b6abe700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b8186880 0x7fb1b808c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:49.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b7ac0700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1b808d680 0x7fb1b808da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:49.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b72bf700 1 -- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b8186880 msgr2=0x7fb1b808c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:49.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b72bf700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b8186880 0x7fb1b808c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b72bf700 1 -- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1b808d680 msgr2=0x7fb1b808da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b72bf700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1b808d680 0x7fb1b808da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.427 7fb1b72bf700 1 -- 172.21.15.32:0/3887741341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b808e6e0 con 0x7fb1b8186480 2021-04-09T18:12:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.428 7fb1b72bf700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8186480 0x7fb1b808c1e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb1a000b560 tx=0x7fb1a000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:49.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.428 7fb1affff700 1 -- 172.21.15.32:0/3887741341 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1a000bcf0 con 0x7fb1b8186480 2021-04-09T18:12:49.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.428 7fb1affff700 1 -- 172.21.15.32:0/3887741341 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a0000c90 con 0x7fb1b8186480 2021-04-09T18:12:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.428 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b808e6e0 con 0x7fb1b8186480 2021-04-09T18:12:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.428 7fb1affff700 1 -- 172.21.15.32:0/3887741341 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1a0002220 con 0x7fb1b8186480 2021-04-09T18:12:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.428 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b808e6e0 con 0x7fb1b8186480 2021-04-09T18:12:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.429 7fb1affff700 1 -- 172.21.15.32:0/3887741341 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb1a000bcf0 con 0x7fb1b8186480 2021-04-09T18:12:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.429 7fb1affff700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb19c0188c0 0x7fb19c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:49.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.429 7fb1affff700 1 -- 172.21.15.32:0/3887741341 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb1a00349a0 con 0x7fb1b8186480 2021-04-09T18:12:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.429 7fb1b6abe700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb19c0188c0 0x7fb19c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.430 7fb1b6abe700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb19c0188c0 0x7fb19c01adc0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:49.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.449 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b8186c80 con 0x7fb1b8186480 2021-04-09T18:12:49.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.452 7fb1affff700 1 -- 172.21.15.32:0/3887741341 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb1a0006b00 con 0x7fb1b8186480 2021-04-09T18:12:49.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.641 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 --> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7fb1b8186c80 con 0x7fb19c0188c0 2021-04-09T18:12:49.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.644 7fb1affff700 1 -- 172.21.15.32:0/3887741341 <== mgr.4102 v2:172.21.15.86:6800/23869 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+77159 (crc 0 0 0) 0x7fb1a8000cc0 con 0x7fb19c0188c0 2021-04-09T18:12:49.730 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2021-04-09T18:12:49.732 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:49.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb19c0188c0 msgr2=0x7fb19c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:49.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb19c0188c0 0x7fb19c01adc0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8186480 msgr2=0x7fb1b808c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:49.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8186480 0x7fb1b808c1e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb1a000b560 tx=0x7fb1a000b900).stop 2021-04-09T18:12:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 shutdown_connections 2021-04-09T18:12:49.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb19c0188c0 0x7fb19c01adc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1b808d680 0x7fb1b808da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1b8186880 0x7fb1b808c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 --2- 172.21.15.32:0/3887741341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8186480 0x7fb1b808c1e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:49.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 >> 172.21.15.32:0/3887741341 conn(0x7fb1b8195e70 msgr2=0x7fb1b81a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:49.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 shutdown_connections 2021-04-09T18:12:49.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:49.646 7fb1bd54e700 1 -- 172.21.15.32:0/3887741341 wait complete. 2021-04-09T18:12:49.868 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2021-04-09 18:12:48.257404","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8390336,"kb_used_data":1664,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584384,"statfs":{"total":773094113280,"available":764502409216,"internally_reserved":8589934592,"allocated":1703936,"data_stored":554676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1526,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000994"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:34.452606","last_change":"2021-04-09 18:12:30.319071","last_active":"2021-04-09 18:12:34.452606","last_peered":"2021-04-09 18:12:34.452606","last_clean":"2021-04-09 18:12:34.452606","last_became_active":"2021-04-09 18:12:30.318784","last_became_peered":"2021-04-09 18:12:30.318784","last_unstale":"2021-04-09 18:12:34.452606","last_undegraded":"2021-04-09 18:12:34.452606","last_fullsized":"2021-04-09 18:12:34.452606","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.808546","last_change":"2021-04-09 18:12:33.331423","last_active":"2021-04-09 18:12:41.808546","last_peered":"2021-04-09 18:12:41.808546","last_clean":"2021-04-09 18:12:41.808546","last_became_active":"2021-04-09 18:12:33.331079","last_became_peered":"2021-04-09 18:12:33.331079","last_unstale":"2021-04-09 18:12:41.808546","last_undegraded":"2021-04-09 18:12:41.808546","last_fullsized":"2021-04-09 18:12:41.808546","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.783821","last_change":"2021-04-09 18:12:33.781125","last_active":"2021-04-09 18:12:33.783821","last_peered":"2021-04-09 18:12:33.783821","last_clean":"2021-04-09 18:12:33.783821","last_became_active":"2021-04-09 18:12:33.780825","last_became_peered":"2021-04-09 18:12:33.780825","last_unstale":"2021-04-09 18:12:33.783821","last_undegraded":"2021-04-09 18:12:33.783821","last_fullsized":"2021-04-09 18:12:33.783821","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:34.452588","last_change":"2021-04-09 18:12:30.316459","last_active":"2021-04-09 18:12:34.452588","last_peered":"2021-04-09 18:12:34.452588","last_clean":"2021-04-09 18:12:34.452588","last_became_active":"2021-04-09 18:12:30.316115","last_became_peered":"2021-04-09 18:12:30.316115","last_unstale":"2021-04-09 18:12:34.452588","last_undegraded":"2021-04-09 18:12:34.452588","last_fullsized":"2021-04-09 18:12:34.452588","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"20'6","reported_seq":"16","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.811071","last_change":"2021-04-09 18:12:33.333374","last_active":"2021-04-09 18:12:41.811071","last_peered":"2021-04-09 18:12:41.811071","last_clean":"2021-04-09 18:12:41.811071","last_became_active":"2021-04-09 18:12:33.333035","last_became_peered":"2021-04-09 18:12:33.333035","last_unstale":"2021-04-09 18:12:41.811071","last_undegraded":"2021-04-09 18:12:41.811071","last_fullsized":"2021-04-09 18:12:41.811071","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779703","last_change":"2021-04-09 18:12:33.779202","last_active":"2021-04-09 18:12:33.779703","last_peered":"2021-04-09 18:12:33.779703","last_clean":"2021-04-09 18:12:33.779703","last_became_active":"2021-04-09 18:12:33.778916","last_became_peered":"2021-04-09 18:12:33.778916","last_unstale":"2021-04-09 18:12:33.779703","last_undegraded":"2021-04-09 18:12:33.779703","last_fullsized":"2021-04-09 18:12:33.779703","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.781925","last_change":"2021-04-09 18:12:33.781366","last_active":"2021-04-09 18:12:33.781925","last_peered":"2021-04-09 18:12:33.781925","last_clean":"2021-04-09 18:12:33.781925","last_became_active":"2021-04-09 18:12:33.781061","last_became_peered":"2021-04-09 18:12:33.781061","last_unstale":"2021-04-09 18:12:33.781925","last_undegraded":"2021-04-09 18:12:33.781925","last_fullsized":"2021-04-09 18:12:33.781925","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"11","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779941","last_change":"2021-04-09 18:12:30.316199","last_active":"2021-04-09 18:12:33.779941","last_peered":"2021-04-09 18:12:33.779941","last_clean":"2021-04-09 18:12:33.779941","last_became_active":"2021-04-09 18:12:30.315813","last_became_peered":"2021-04-09 18:12:30.315813","last_unstale":"2021-04-09 18:12:33.779941","last_undegraded":"2021-04-09 18:12:33.779941","last_fullsized":"2021-04-09 18:12:33.779941","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"20'3","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.810882","last_change":"2021-04-09 18:12:33.333344","last_active":"2021-04-09 18:12:41.810882","last_peered":"2021-04-09 18:12:41.810882","last_clean":"2021-04-09 18:12:41.810882","last_became_active":"2021-04-09 18:12:33.332949","last_became_peered":"2021-04-09 18:12:33.332949","last_unstale":"2021-04-09 18:12:41.810882","last_undegraded":"2021-04-09 18:12:41.810882","last_fullsized":"2021-04-09 18:12:41.810882","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.778748","last_change":"2021-04-09 18:12:33.777893","last_active":"2021-04-09 18:12:33.778748","last_peered":"2021-04-09 18:12:33.778748","last_clean":"2021-04-09 18:12:33.778748","last_became_active":"2021-04-09 18:12:33.777554","last_became_peered":"2021-04-09 18:12:33.777554","last_unstale":"2021-04-09 18:12:33.778748","last_undegraded":"2021-04-09 18:12:33.778748","last_fullsized":"2021-04-09 18:12:33.778748","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"12","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.776357","last_change":"2021-04-09 18:12:30.313110","last_active":"2021-04-09 18:12:33.776357","last_peered":"2021-04-09 18:12:33.776357","last_clean":"2021-04-09 18:12:33.776357","last_became_active":"2021-04-09 18:12:30.312615","last_became_peered":"2021-04-09 18:12:30.312615","last_unstale":"2021-04-09 18:12:33.776357","last_undegraded":"2021-04-09 18:12:33.776357","last_fullsized":"2021-04-09 18:12:33.776357","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":"17","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.813860","last_change":"2021-04-09 18:12:33.333727","last_active":"2021-04-09 18:12:41.813860","last_peered":"2021-04-09 18:12:41.813860","last_clean":"2021-04-09 18:12:41.813860","last_became_active":"2021-04-09 18:12:33.333326","last_became_peered":"2021-04-09 18:12:33.333326","last_unstale":"2021-04-09 18:12:41.813860","last_undegraded":"2021-04-09 18:12:41.813860","last_fullsized":"2021-04-09 18:12:41.813860","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779698","last_change":"2021-04-09 18:12:30.317416","last_active":"2021-04-09 18:12:33.779698","last_peered":"2021-04-09 18:12:33.779698","last_clean":"2021-04-09 18:12:33.779698","last_became_active":"2021-04-09 18:12:30.317112","last_became_peered":"2021-04-09 18:12:30.317112","last_unstale":"2021-04-09 18:12:33.779698","last_undegraded":"2021-04-09 18:12:33.779698","last_fullsized":"2021-04-09 18:12:33.779698","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'5","reported_seq":"15","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.808651","last_change":"2021-04-09 18:12:33.329808","last_active":"2021-04-09 18:12:41.808651","last_peered":"2021-04-09 18:12:41.808651","last_clean":"2021-04-09 18:12:41.808651","last_became_active":"2021-04-09 18:12:33.329428","last_became_peered":"2021-04-09 18:12:33.329428","last_unstale":"2021-04-09 18:12:41.808651","last_undegraded":"2021-04-09 18:12:41.808651","last_fullsized":"2021-04-09 18:12:41.808651","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.781717","last_change":"2021-04-09 18:12:33.781025","last_active":"2021-04-09 18:12:33.781717","last_peered":"2021-04-09 18:12:33.781717","last_clean":"2021-04-09 18:12:33.781717","last_became_active":"2021-04-09 18:12:33.780701","last_became_peered":"2021-04-09 18:12:33.780701","last_unstale":"2021-04-09 18:12:33.781717","last_undegraded":"2021-04-09 18:12:33.781717","last_fullsized":"2021-04-09 18:12:33.781717","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779385","last_change":"2021-04-09 18:12:30.315339","last_active":"2021-04-09 18:12:33.779385","last_peered":"2021-04-09 18:12:33.779385","last_clean":"2021-04-09 18:12:33.779385","last_became_active":"2021-04-09 18:12:30.314975","last_became_peered":"2021-04-09 18:12:30.314975","last_unstale":"2021-04-09 18:12:33.779385","last_undegraded":"2021-04-09 18:12:33.779385","last_fullsized":"2021-04-09 18:12:33.779385","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":"11","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:34.787036","last_change":"2021-04-09 18:12:33.329984","last_active":"2021-04-09 18:12:34.787036","last_peered":"2021-04-09 18:12:34.787036","last_clean":"2021-04-09 18:12:34.787036","last_became_active":"2021-04-09 18:12:33.329565","last_became_peered":"2021-04-09 18:12:33.329565","last_unstale":"2021-04-09 18:12:34.787036","last_undegraded":"2021-04-09 18:12:34.787036","last_fullsized":"2021-04-09 18:12:34.787036","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.788904","last_change":"2021-04-09 18:12:33.784198","last_active":"2021-04-09 18:12:33.788904","last_peered":"2021-04-09 18:12:33.788904","last_clean":"2021-04-09 18:12:33.788904","last_became_active":"2021-04-09 18:12:33.783816","last_became_peered":"2021-04-09 18:12:33.783816","last_unstale":"2021-04-09 18:12:33.788904","last_undegraded":"2021-04-09 18:12:33.788904","last_fullsized":"2021-04-09 18:12:33.788904","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.778194","last_change":"2021-04-09 18:12:33.330844","last_active":"2021-04-09 18:12:33.778194","last_peered":"2021-04-09 18:12:33.778194","last_clean":"2021-04-09 18:12:33.778194","last_became_active":"2021-04-09 18:12:33.330206","last_became_peered":"2021-04-09 18:12:33.330206","last_unstale":"2021-04-09 18:12:33.778194","last_undegraded":"2021-04-09 18:12:33.778194","last_fullsized":"2021-04-09 18:12:33.778194","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.779586","last_change":"2021-04-09 18:12:30.316686","last_active":"2021-04-09 18:12:33.779586","last_peered":"2021-04-09 18:12:33.779586","last_clean":"2021-04-09 18:12:33.779586","last_became_active":"2021-04-09 18:12:30.316397","last_became_peered":"2021-04-09 18:12:30.316397","last_unstale":"2021-04-09 18:12:33.779586","last_undegraded":"2021-04-09 18:12:33.779586","last_fullsized":"2021-04-09 18:12:33.779586","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.780687","last_change":"2021-04-09 18:12:33.779614","last_active":"2021-04-09 18:12:33.780687","last_peered":"2021-04-09 18:12:33.780687","last_clean":"2021-04-09 18:12:33.780687","last_became_active":"2021-04-09 18:12:33.778867","last_became_peered":"2021-04-09 18:12:33.778867","last_unstale":"2021-04-09 18:12:33.780687","last_undegraded":"2021-04-09 18:12:33.780687","last_fullsized":"2021-04-09 18:12:33.780687","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-04-09 18:12:41.811298","last_change":"2021-04-09 18:12:33.332835","last_active":"2021-04-09 18:12:41.811298","last_peered":"2021-04-09 18:12:41.811298","last_clean":"2021-04-09 18:12:41.811298","last_became_active":"2021-04-09 18:12:33.332485","last_became_peered":"2021-04-09 18:12:33.332485","last_unstale":"2021-04-09 18:12:41.811298","last_undegraded":"2021-04-09 18:12:41.811298","last_fullsized":"2021-04-09 18:12:41.811298","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:32.318453","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:32.318453","last_clean_scrub_stamp":"2021-04-09 18:12:32.318453","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.776726","last_change":"2021-04-09 18:12:30.316233","last_active":"2021-04-09 18:12:33.776726","last_peered":"2021-04-09 18:12:33.776726","last_clean":"2021-04-09 18:12:33.776726","last_became_active":"2021-04-09 18:12:30.315922","last_became_peered":"2021-04-09 18:12:30.315922","last_unstale":"2021-04-09 18:12:33.776726","last_undegraded":"2021-04-09 18:12:33.776726","last_fullsized":"2021-04-09 18:12:33.776726","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:29.305185","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:29.305185","last_clean_scrub_stamp":"2021-04-09 18:12:29.305185","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-04-09 18:12:33.784266","last_change":"2021-04-09 18:12:33.783881","last_active":"2021-04-09 18:12:33.784266","last_peered":"2021-04-09 18:12:33.784266","last_clean":"2021-04-09 18:12:33.784266","last_became_active":"2021-04-09 18:12:33.783671","last_became_peered":"2021-04-09 18:12:33.783671","last_unstale":"2021-04-09 18:12:33.784266","last_undegraded":"2021-04-09 18:12:33.784266","last_fullsized":"2021-04-09 18:12:33.784266","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-09 18:12:33.321936","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-09 18:12:33.321936","last_clean_scrub_stamp":"2021-04-09 18:12:33.321936","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":69489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048792,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323048,"statfs":{"total":96636764160,"available":95562801152,"internally_reserved":1073741824,"allocated":212992,"data_stored":69329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048804,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323036,"statfs":{"total":96636764160,"available":95562788864,"internally_reserved":1073741824,"allocated":225280,"data_stored":70449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.185}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.223}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":70289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]}]}} 2021-04-09T18:12:49.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2021-04-09T18:12:50.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.027 7f7e15ef2700 1 Processor -- start 2021-04-09T18:12:50.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.027 7f7e15ef2700 1 -- start start 2021-04-09T18:12:50.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e15ef2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e100cffa0 0x7f7e100d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:50.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e15ef2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7e100d7800 0x7f7e100d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:50.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e15ef2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100d8220 0x7f7e100d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:50.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e15ef2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e100d91e0 con 0x7f7e100cffa0 2021-04-09T18:12:50.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e15ef2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e100d7280 con 0x7f7e100d7800 2021-04-09T18:12:50.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e15ef2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100d6290 con 0x7f7e100d8220 2021-04-09T18:12:50.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e100cffa0 0x7f7e100d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:50.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e100cffa0 0x7f7e100d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41938/0 (socket says 172.21.15.32:41938) 2021-04-09T18:12:50.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0f7fe700 1 -- 172.21.15.32:0/2167635971 learned_addr learned my addr 172.21.15.32:0/2167635971 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0ffff700 1 --2- 172.21.15.32:0/2167635971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100d8220 0x7f7e100d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0effd700 1 --2- 172.21.15.32:0/2167635971 >> v2:172.21.15.86:3300/0 conn(0x7f7e100d7800 0x7f7e100d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0f7fe700 1 -- 172.21.15.32:0/2167635971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100d8220 msgr2=0x7f7e100d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0f7fe700 1 --2- 172.21.15.32:0/2167635971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100d8220 0x7f7e100d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:50.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0f7fe700 1 -- 172.21.15.32:0/2167635971 >> v2:172.21.15.86:3300/0 conn(0x7f7e100d7800 msgr2=0x7f7e100d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:12:50.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0f7fe700 1 --2- 172.21.15.32:0/2167635971 >> v2:172.21.15.86:3300/0 conn(0x7f7e100d7800 0x7f7e100d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:50.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0f7fe700 1 -- 172.21.15.32:0/2167635971 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100d6480 con 0x7f7e100cffa0 2021-04-09T18:12:50.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0ffff700 1 --2- 172.21.15.32:0/2167635971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100d8220 0x7f7e100d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:50.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.028 7f7e0f7fe700 1 --2- 172.21.15.32:0/2167635971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e100cffa0 0x7f7e100d2440 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f7e00006700 tx=0x7f7e00009b80).ready entity=mon.0 client_cookie=83306829b14daa9b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:50.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e0e7fc700 1 -- 172.21.15.32:0/2167635971 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e0000c0c0 con 0x7f7e100cffa0 2021-04-09T18:12:50.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e0e7fc700 1 -- 172.21.15.32:0/2167635971 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0000c440 con 0x7f7e100cffa0 2021-04-09T18:12:50.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e0e7fc700 1 -- 172.21.15.32:0/2167635971 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e0000c920 con 0x7f7e100cffa0 2021-04-09T18:12:50.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e15ef2700 1 -- 172.21.15.32:0/2167635971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e100cffa0 msgr2=0x7f7e100d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:50.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e15ef2700 1 --2- 172.21.15.32:0/2167635971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e100cffa0 0x7f7e100d2440 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f7e00006700 tx=0x7f7e00009b80).stop 2021-04-09T18:12:50.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e15ef2700 1 -- 172.21.15.32:0/2167635971 shutdown_connections 2021-04-09T18:12:50.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e15ef2700 1 --2- 172.21.15.32:0/2167635971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100d8220 0x7f7e100d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e15ef2700 1 --2- 172.21.15.32:0/2167635971 >> v2:172.21.15.86:3300/0 conn(0x7f7e100d7800 0x7f7e100d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e15ef2700 1 --2- 172.21.15.32:0/2167635971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e100cffa0 0x7f7e100d2440 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:50.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.029 7f7e15ef2700 1 -- 172.21.15.32:0/2167635971 >> 172.21.15.32:0/2167635971 conn(0x7f7e100ce5c0 msgr2=0x7f7e100cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:12:50.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.030 7f7e15ef2700 1 -- 172.21.15.32:0/2167635971 shutdown_connections 2021-04-09T18:12:50.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.030 7f7e15ef2700 1 -- 172.21.15.32:0/2167635971 wait complete. 2021-04-09T18:12:50.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e15ef2700 1 Processor -- start 2021-04-09T18:12:50.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e15ef2700 1 -- start start 2021-04-09T18:12:50.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e15ef2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100b4250 0x7f7e1009a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:50.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e15ef2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e1018e280 0x7f7e10192370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:50.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e15ef2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10184d80 0x7f7e10191250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:50.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e15ef2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10192a10 con 0x7f7e10184d80 2021-04-09T18:12:50.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e15ef2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10191df0 con 0x7f7e1018e280 2021-04-09T18:12:50.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e15ef2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10191870 con 0x7f7e100b4250 2021-04-09T18:12:50.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e0ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10184d80 0x7f7e10191250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:50.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e0ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10184d80 0x7f7e10191250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41948/0 (socket says 172.21.15.32:41948) 2021-04-09T18:12:50.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.050 7f7e0ffff700 1 -- 172.21.15.32:0/3630373061 learned_addr learned my addr 172.21.15.32:0/3630373061 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:12:50.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e0f7fe700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100b4250 0x7f7e1009a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:50.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e0effd700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e1018e280 0x7f7e10192370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:50.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e0ffff700 1 -- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100b4250 msgr2=0x7f7e1009a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:50.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e0ffff700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100b4250 0x7f7e1009a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:50.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e0ffff700 1 -- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e1018e280 msgr2=0x7f7e10192370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:12:50.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e0ffff700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e1018e280 0x7f7e10192370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:12:50.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e0ffff700 1 -- 172.21.15.32:0/3630373061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10191a90 con 0x7f7e10184d80 2021-04-09T18:12:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e0f7fe700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100b4250 0x7f7e1009a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:12:50.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e0ffff700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10184d80 0x7f7e10191250 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f7e0800a940 tx=0x7f7e0800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:50.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7df7fff700 1 -- 172.21.15.32:0/3630373061 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e08001400 con 0x7f7e10184d80 2021-04-09T18:12:50.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7df7fff700 1 -- 172.21.15.32:0/3630373061 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0800ae20 con 0x7f7e10184d80 2021-04-09T18:12:50.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e10191a90 con 0x7f7e10184d80 2021-04-09T18:12:50.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7df7fff700 1 -- 172.21.15.32:0/3630373061 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e080018b0 con 0x7f7e10184d80 2021-04-09T18:12:50.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.051 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e10191a90 con 0x7f7e10184d80 2021-04-09T18:12:50.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.052 7f7df7fff700 1 -- 172.21.15.32:0/3630373061 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7e080011f0 con 0x7f7e10184d80 2021-04-09T18:12:50.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.052 7f7df7fff700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7df0018930 0x7f7df001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:12:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.052 7f7df7fff700 1 -- 172.21.15.32:0/3630373061 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7e08032d70 con 0x7f7e10184d80 2021-04-09T18:12:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.052 7f7e0f7fe700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7df0018930 0x7f7df001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:12:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.052 7f7e0f7fe700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7df0018930 0x7f7df001ae10 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:12:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.072 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e10182d60 con 0x7f7e10184d80 2021-04-09T18:12:50.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.076 7f7df7fff700 1 -- 172.21.15.32:0/3630373061 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7e080016a0 con 0x7f7e10184d80 2021-04-09T18:12:50.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.264 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7e10182d60 con 0x7f7e10184d80 2021-04-09T18:12:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.267 7f7df7fff700 1 -- 172.21.15.32:0/3630373061 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+211750 (secure 0 0 0) 0x7f7e080011f0 con 0x7f7e10184d80 2021-04-09T18:12:50.518 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:12:50.521 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:12:50.522 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:12:50.540 INFO:teuthology.orchestra.run.smithi032.stdout: "health": { 2021-04-09T18:12:50.564 INFO:teuthology.orchestra.run.smithi032.stdout: "checks": {}, 2021-04-09T18:12:50.565 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "HEALTH_OK" 2021-04-09T18:12:50.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:50.575 INFO:teuthology.orchestra.run.smithi032.stdout: "election_epoch": 8, 2021-04-09T18:12:50.577 INFO:teuthology.orchestra.run.smithi032.stdout: "quorum": [ 2021-04-09T18:12:50.577 INFO:teuthology.orchestra.run.smithi032.stdout: 0, 2021-04-09T18:12:50.577 INFO:teuthology.orchestra.run.smithi032.stdout: 1, 2021-04-09T18:12:50.577 INFO:teuthology.orchestra.run.smithi032.stdout: 2 2021-04-09T18:12:50.623 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:50.642 INFO:teuthology.orchestra.run.smithi032.stdout: "quorum_names": [ 2021-04-09T18:12:50.642 INFO:teuthology.orchestra.run.smithi032.stdout: "a", 2021-04-09T18:12:50.643 INFO:teuthology.orchestra.run.smithi032.stdout: "b", 2021-04-09T18:12:50.643 INFO:teuthology.orchestra.run.smithi032.stdout: "c" 2021-04-09T18:12:50.644 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:50.644 INFO:teuthology.orchestra.run.smithi032.stdout: "quorum_age": 36, 2021-04-09T18:12:50.644 INFO:teuthology.orchestra.run.smithi032.stdout: "monmap": { 2021-04-09T18:12:50.645 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 1, 2021-04-09T18:12:50.645 INFO:teuthology.orchestra.run.smithi032.stdout: "min_mon_release_name": "14", 2021-04-09T18:12:50.680 INFO:teuthology.orchestra.run.smithi032.stdout: "num_mons": 3 2021-04-09T18:12:50.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:50.680 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap": { 2021-04-09T18:12:50.680 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap": { 2021-04-09T18:12:50.681 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 20, 2021-04-09T18:12:50.681 INFO:teuthology.orchestra.run.smithi032.stdout: "num_osds": 8, 2021-04-09T18:12:50.681 INFO:teuthology.orchestra.run.smithi032.stdout: "num_up_osds": 8, 2021-04-09T18:12:50.681 INFO:teuthology.orchestra.run.smithi032.stdout: "num_in_osds": 8, 2021-04-09T18:12:50.682 INFO:teuthology.orchestra.run.smithi032.stdout: "num_remapped_pgs": 0 2021-04-09T18:12:50.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:50.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:50.682 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap": { 2021-04-09T18:12:50.683 INFO:teuthology.orchestra.run.smithi032.stdout: "pgs_by_state": [ 2021-04-09T18:12:50.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:50.683 INFO:teuthology.orchestra.run.smithi032.stdout: "state_name": "active+clean", 2021-04-09T18:12:50.683 INFO:teuthology.orchestra.run.smithi032.stdout: "count": 24 2021-04-09T18:12:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: "num_pgs": 24, 2021-04-09T18:12:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: "num_pools": 3, 2021-04-09T18:12:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: "num_objects": 21, 2021-04-09T18:12:50.685 INFO:teuthology.orchestra.run.smithi032.stdout: "data_bytes": 2286, 2021-04-09T18:12:50.685 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_used": 8591704064, 2021-04-09T18:12:50.685 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_avail": 764502409216, 2021-04-09T18:12:50.685 INFO:teuthology.orchestra.run.smithi032.stdout: "bytes_total": 773094113280, 2021-04-09T18:12:50.685 INFO:teuthology.orchestra.run.smithi032.stdout: "write_bytes_sec": 767, 2021-04-09T18:12:50.686 INFO:teuthology.orchestra.run.smithi032.stdout: "read_op_per_sec": 0, 2021-04-09T18:12:50.730 INFO:teuthology.orchestra.run.smithi032.stdout: "write_op_per_sec": 2 2021-04-09T18:12:50.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:50.731 INFO:teuthology.orchestra.run.smithi032.stdout: "fsmap": { 2021-04-09T18:12:50.731 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 6, 2021-04-09T18:12:50.731 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 1, 2021-04-09T18:12:50.742 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:12:50.743 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:12:50.745 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 1, 2021-04-09T18:12:50.746 INFO:teuthology.orchestra.run.smithi032.stdout: "by_rank": [ 2021-04-09T18:12:50.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:50.747 INFO:teuthology.orchestra.run.smithi032.stdout: "filesystem_id": 1, 2021-04-09T18:12:50.756 INFO:teuthology.orchestra.run.smithi032.stdout: "rank": 0, 2021-04-09T18:12:50.756 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "a", 2021-04-09T18:12:50.757 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "up:active", 2021-04-09T18:12:50.757 INFO:teuthology.orchestra.run.smithi032.stdout: "gid": 4314 2021-04-09T18:12:50.758 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:50.775 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:50.775 INFO:teuthology.orchestra.run.smithi032.stdout: "up:standby": 3 2021-04-09T18:12:50.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:50.776 INFO:teuthology.orchestra.run.smithi032.stdout: "mgrmap": { 2021-04-09T18:12:50.776 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 3, 2021-04-09T18:12:50.777 INFO:teuthology.orchestra.run.smithi032.stdout: "active_gid": 4102, 2021-04-09T18:12:50.777 INFO:teuthology.orchestra.run.smithi032.stdout: "active_name": "x", 2021-04-09T18:12:50.778 INFO:teuthology.orchestra.run.smithi032.stdout: "active_addrs": { 2021-04-09T18:12:50.778 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:12:50.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:50.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:12:50.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6800", 2021-04-09T18:12:50.857 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 23869 2021-04-09T18:12:50.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:50.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:50.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:12:50.858 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6801", 2021-04-09T18:12:50.859 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 23869 2021-04-09T18:12:50.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:50.876 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:12:50.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:50.898 INFO:teuthology.orchestra.run.smithi032.stdout: "active_addr": "172.21.15.86:6801/23869", 2021-04-09T18:12:50.899 INFO:teuthology.orchestra.run.smithi032.stdout: "active_change": "2021-04-09 18:12:14.249236", 2021-04-09T18:12:50.899 INFO:teuthology.orchestra.run.smithi032.stdout: "available": true, 2021-04-09T18:12:50.900 INFO:teuthology.orchestra.run.smithi032.stdout: "standbys": [ 2021-04-09T18:12:50.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:50.901 INFO:teuthology.orchestra.run.smithi032.stdout: "gid": 4110, 2021-04-09T18:12:50.901 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "y", 2021-04-09T18:12:50.922 INFO:teuthology.orchestra.run.smithi032.stdout: "available_modules": [ 2021-04-09T18:12:50.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:50.945 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "alerts", 2021-04-09T18:12:50.945 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:50.948 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:50.948 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:50.949 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:12:50.949 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:12:50.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:50.960 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:50.960 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:50.961 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2021-04-09T18:12:50.961 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:50.962 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:50.962 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:50.962 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How frequently to reexamine health status", 2021-04-09T18:12:50.977 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:50.978 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:50.978 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:50.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:50.979 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_destination": { 2021-04-09T18:12:50.980 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_destination", 2021-04-09T18:12:50.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:50.995 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:50.996 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email address to send alerts to", 2021-04-09T18:12:50.997 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.002 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.002 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.003 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_from_name": { 2021-04-09T18:12:51.003 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_from_name", 2021-04-09T18:12:51.003 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.004 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.004 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.004 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "Ceph", 2021-04-09T18:12:51.004 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.005 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.005 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.005 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email From: name", 2021-04-09T18:12:51.006 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.007 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.007 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.007 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_host": { 2021-04-09T18:12:51.008 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_host", 2021-04-09T18:12:51.008 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.008 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.008 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.009 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:51.009 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.009 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.009 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.009 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP server", 2021-04-09T18:12:51.010 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.010 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.010 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.040 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_password": { 2021-04-09T18:12:51.040 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_password", 2021-04-09T18:12:51.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.041 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.041 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.041 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:51.041 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.041 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.042 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.042 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Password to authenticate with", 2021-04-09T18:12:51.042 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.042 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.043 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.043 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_port": { 2021-04-09T18:12:51.043 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_port", 2021-04-09T18:12:51.044 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:51.044 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.044 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.044 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "465", 2021-04-09T18:12:51.044 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.045 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.045 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.045 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP port", 2021-04-09T18:12:51.045 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.045 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.046 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.046 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_sender": { 2021-04-09T18:12:51.046 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_sender", 2021-04-09T18:12:51.046 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.047 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.123 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.124 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:51.124 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.124 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.124 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.125 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP envelope sender", 2021-04-09T18:12:51.125 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.125 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.126 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.140 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_ssl": { 2021-04-09T18:12:51.140 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_ssl", 2021-04-09T18:12:51.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:51.140 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.140 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.141 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:51.141 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.141 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.141 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.141 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use SSL to connect to SMTP server", 2021-04-09T18:12:51.151 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.151 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.151 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.152 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_user": { 2021-04-09T18:12:51.158 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_user", 2021-04-09T18:12:51.158 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.158 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.158 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.158 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:51.159 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.159 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.159 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.159 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "User to authenticate as", 2021-04-09T18:12:51.160 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.160 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.160 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.160 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:51.161 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:51.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:51.161 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ansible", 2021-04-09T18:12:51.161 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:51.162 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:51.162 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:51.162 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2021-04-09T18:12:51.162 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2021-04-09T18:12:51.175 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.175 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.176 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:51.180 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:51.180 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.180 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.181 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.181 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:51.181 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.181 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.181 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.182 INFO:teuthology.orchestra.run.smithi032.stdout: "server_url": { 2021-04-09T18:12:51.182 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_url", 2021-04-09T18:12:51.182 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.182 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.183 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:51.183 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:51.183 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.236 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.242 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.243 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:51.243 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.244 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.244 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.252 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2021-04-09T18:12:51.258 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2021-04-09T18:12:51.259 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.259 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.259 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:51.260 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:51.260 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.261 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.261 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.262 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:51.262 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.262 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.263 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.286 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_server": { 2021-04-09T18:12:51.286 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_server", 2021-04-09T18:12:51.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.287 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.287 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:51.288 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:51.288 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.288 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.289 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.289 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:51.290 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.290 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.290 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:51.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:51.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:51.307 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "balancer", 2021-04-09T18:12:51.308 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:51.308 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:51.309 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:51.309 INFO:teuthology.orchestra.run.smithi032.stdout: "active": { 2021-04-09T18:12:51.309 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "active", 2021-04-09T18:12:51.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:51.310 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.343 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.344 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:51.358 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.359 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.359 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.359 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically balance PGs across cluster", 2021-04-09T18:12:51.360 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.360 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.361 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.361 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_time": { 2021-04-09T18:12:51.375 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_time", 2021-04-09T18:12:51.376 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.377 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.377 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.391 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0000", 2021-04-09T18:12:51.391 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.392 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.392 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.405 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "beginning time of day to automatically balance", 2021-04-09T18:12:51.405 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-04-09T18:12:51.406 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.406 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.407 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_weekday": { 2021-04-09T18:12:51.407 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_weekday", 2021-04-09T18:12:51.408 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2021-04-09T18:12:51.408 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.409 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.409 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2021-04-09T18:12:51.409 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2021-04-09T18:12:51.410 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "7", 2021-04-09T18:12:51.410 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.411 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2021-04-09T18:12:51.435 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-04-09T18:12:51.437 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.437 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.438 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_max_iterations": { 2021-04-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_max_iterations", 2021-04-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2021-04-09T18:12:51.439 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.440 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.440 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "25", 2021-04-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2021-04-09T18:12:51.441 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "250", 2021-04-09T18:12:51.442 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.442 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum number of iterations to attempt optimization", 2021-04-09T18:12:51.442 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.443 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.466 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.467 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_metrics": { 2021-04-09T18:12:51.467 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_metrics", 2021-04-09T18:12:51.469 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.469 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.470 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.470 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "pgs,objects,bytes", 2021-04-09T18:12:51.470 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.470 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.475 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.475 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "metrics with which to calculate OSD utilization", 2021-04-09T18:12:51.475 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2021-04-09T18:12:51.476 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.477 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.477 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_step": { 2021-04-09T18:12:51.477 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_step", 2021-04-09T18:12:51.478 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2021-04-09T18:12:51.478 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.479 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.479 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.5", 2021-04-09T18:12:51.479 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0.001", 2021-04-09T18:12:51.479 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "0.999", 2021-04-09T18:12:51.479 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.480 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "aggressiveness of optimization", 2021-04-09T18:12:51.480 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2021-04-09T18:12:51.480 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.480 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.481 INFO:teuthology.orchestra.run.smithi032.stdout: "end_time": { 2021-04-09T18:12:51.481 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_time", 2021-04-09T18:12:51.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.481 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.482 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.482 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2400", 2021-04-09T18:12:51.482 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.482 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.483 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.483 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ending time of day to automatically balance", 2021-04-09T18:12:51.483 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-04-09T18:12:51.548 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: "end_weekday": { 2021-04-09T18:12:51.550 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_weekday", 2021-04-09T18:12:51.551 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2021-04-09T18:12:51.556 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.557 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7", 2021-04-09T18:12:51.563 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2021-04-09T18:12:51.564 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "7", 2021-04-09T18:12:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.565 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2021-04-09T18:12:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-04-09T18:12:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.598 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "min_score": { 2021-04-09T18:12:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_score", 2021-04-09T18:12:51.599 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2021-04-09T18:12:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2021-04-09T18:12:51.600 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "minimum score, below which no optimization is attempted", 2021-04-09T18:12:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.601 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2021-04-09T18:12:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2021-04-09T18:12:51.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "none", 2021-04-09T18:12:51.644 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2021-04-09T18:12:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-compat", 2021-04-09T18:12:51.645 INFO:teuthology.orchestra.run.smithi032.stdout: "none", 2021-04-09T18:12:51.656 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap" 2021-04-09T18:12:51.657 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:51.657 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Balancer mode", 2021-04-09T18:12:51.657 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.675 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.676 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.677 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_ids": { 2021-04-09T18:12:51.677 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_ids", 2021-04-09T18:12:51.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:51.711 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.712 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.712 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:51.713 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.713 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.713 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.750 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "pools which the automatic balancing will be limited to", 2021-04-09T18:12:51.750 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.751 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.751 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.782 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:12:51.783 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:12:51.783 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:51.784 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.784 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.785 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2021-04-09T18:12:51.785 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.785 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.786 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.786 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and attempt optimization", 2021-04-09T18:12:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_deviation": { 2021-04-09T18:12:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_deviation", 2021-04-09T18:12:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2021-04-09T18:12:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2021-04-09T18:12:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "deviation below which no optimization is attempted", 2021-04-09T18:12:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2021-04-09T18:12:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_iterations": { 2021-04-09T18:12:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_iterations", 2021-04-09T18:12:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2021-04-09T18:12:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2021-04-09T18:12:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum upmap optimization iterations", 2021-04-09T18:12:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crash", 2021-04-09T18:12:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "retain_interval": { 2021-04-09T18:12:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retain_interval", 2021-04-09T18:12:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31536000", 2021-04-09T18:12:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain crashes before pruning them", 2021-04-09T18:12:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_recent_interval": { 2021-04-09T18:12:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_recent_interval", 2021-04-09T18:12:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1209600", 2021-04-09T18:12:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "time interval in which to warn about recent crashes", 2021-04-09T18:12:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dashboard", 2021-04-09T18:12:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_HOST": { 2021-04-09T18:12:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_HOST", 2021-04-09T18:12:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_ENABLED": { 2021-04-09T18:12:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_ENABLED", 2021-04-09T18:12:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_LOG_PAYLOAD": { 2021-04-09T18:12:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2021-04-09T18:12:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "ENABLE_BROWSABLE_API": { 2021-04-09T18:12:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ENABLE_BROWSABLE_API", 2021-04-09T18:12:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_cephfs": { 2021-04-09T18:12:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_cephfs", 2021-04-09T18:12:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_iscsi": { 2021-04-09T18:12:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_iscsi", 2021-04-09T18:12:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_mirroring": { 2021-04-09T18:12:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_mirroring", 2021-04-09T18:12:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_rbd": { 2021-04-09T18:12:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_rbd", 2021-04-09T18:12:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_rgw": { 2021-04-09T18:12:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_rgw", 2021-04-09T18:12:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2021-04-09T18:12:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2021-04-09T18:12:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_PASSWORD": { 2021-04-09T18:12:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_PASSWORD", 2021-04-09T18:12:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2021-04-09T18:12:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_SSL_VERIFY": { 2021-04-09T18:12:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2021-04-09T18:12:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_URL": { 2021-04-09T18:12:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_URL", 2021-04-09T18:12:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_USERNAME": { 2021-04-09T18:12:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_USERNAME", 2021-04-09T18:12:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2021-04-09T18:12:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_FRONTEND_API_URL": { 2021-04-09T18:12:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2021-04-09T18:12:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2021-04-09T18:12:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2021-04-09T18:12:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "ISCSI_API_SSL_VERIFICATION": { 2021-04-09T18:12:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2021-04-09T18:12:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_HOST": { 2021-04-09T18:12:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_HOST", 2021-04-09T18:12:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "REST_REQUESTS_TIMEOUT": { 2021-04-09T18:12:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "REST_REQUESTS_TIMEOUT", 2021-04-09T18:12:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "45", 2021-04-09T18:12:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ACCESS_KEY": { 2021-04-09T18:12:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ACCESS_KEY", 2021-04-09T18:12:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ADMIN_RESOURCE": { 2021-04-09T18:12:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2021-04-09T18:12:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2021-04-09T18:12:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_HOST": { 2021-04-09T18:12:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_HOST", 2021-04-09T18:12:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_PORT": { 2021-04-09T18:12:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_PORT", 2021-04-09T18:12:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "80", 2021-04-09T18:12:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SCHEME": { 2021-04-09T18:12:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SCHEME", 2021-04-09T18:12:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "http", 2021-04-09T18:12:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SECRET_KEY": { 2021-04-09T18:12:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SECRET_KEY", 2021-04-09T18:12:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SSL_VERIFY": { 2021-04-09T18:12:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SSL_VERIFY", 2021-04-09T18:12:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_USER_ID": { 2021-04-09T18:12:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_USER_ID", 2021-04-09T18:12:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "crt_file": { 2021-04-09T18:12:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crt_file", 2021-04-09T18:12:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "debug": { 2021-04-09T18:12:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "debug", 2021-04-09T18:12:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:52.947 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:52.948 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:52.948 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:52.949 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable/disable debug options", 2021-04-09T18:12:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "jwt_token_ttl": { 2021-04-09T18:12:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "jwt_token_ttl", 2021-04-09T18:12:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "28800", 2021-04-09T18:12:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2021-04-09T18:12:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2021-04-09T18:12:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2021-04-09T18:12:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2021-04-09T18:12:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2021-04-09T18:12:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2021-04-09T18:12:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2021-04-09T18:12:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2021-04-09T18:12:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2021-04-09T18:12:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8080", 2021-04-09T18:12:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2021-04-09T18:12:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2021-04-09T18:12:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl_server_port": { 2021-04-09T18:12:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl_server_port", 2021-04-09T18:12:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8443", 2021-04-09T18:12:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2021-04-09T18:12:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2021-04-09T18:12:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "redirect", 2021-04-09T18:12:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2021-04-09T18:12:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2021-04-09T18:12:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect" 2021-04-09T18:12:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.242 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.242 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.242 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.254 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2021-04-09T18:12:53.257 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2021-04-09T18:12:53.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2021-04-09T18:12:53.259 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2021-04-09T18:12:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2021-04-09T18:12:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.260 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: "url_prefix": { 2021-04-09T18:12:53.261 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url_prefix", 2021-04-09T18:12:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.262 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.324 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.338 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:53.338 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.338 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.339 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.342 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.346 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.346 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.346 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.392 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2021-04-09T18:12:53.398 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2021-04-09T18:12:53.398 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.398 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.399 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.399 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:53.402 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.402 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.402 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.425 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.425 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.426 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.426 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:53.436 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:53.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:53.443 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "deepsea", 2021-04-09T18:12:53.443 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:53.443 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:53.443 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:53.447 INFO:teuthology.orchestra.run.smithi032.stdout: "salt_api_eauth": { 2021-04-09T18:12:53.451 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "salt_api_eauth", 2021-04-09T18:12:53.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.451 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.452 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.452 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "sharedsecret", 2021-04-09T18:12:53.452 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.452 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.453 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.453 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.453 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.453 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.453 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.454 INFO:teuthology.orchestra.run.smithi032.stdout: "salt_api_password": { 2021-04-09T18:12:53.454 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "salt_api_password", 2021-04-09T18:12:53.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.454 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.455 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.455 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:53.455 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.455 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.456 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.456 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.456 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.456 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.456 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.457 INFO:teuthology.orchestra.run.smithi032.stdout: "salt_api_url": { 2021-04-09T18:12:53.457 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "salt_api_url", 2021-04-09T18:12:53.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.457 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.458 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.458 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:53.458 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.458 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.459 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.459 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.504 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.507 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.507 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.512 INFO:teuthology.orchestra.run.smithi032.stdout: "salt_api_username": { 2021-04-09T18:12:53.512 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "salt_api_username", 2021-04-09T18:12:53.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.513 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.513 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:53.513 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:53.513 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.514 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.514 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.514 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:53.514 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.515 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.515 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.515 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:53.589 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2021-04-09T18:12:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:53.599 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:53.600 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_monitoring": { 2021-04-09T18:12:53.612 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_monitoring", 2021-04-09T18:12:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.613 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:53.619 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.620 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.634 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.635 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "monitor device health metrics", 2021-04-09T18:12:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.636 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "mark_out_threshold": { 2021-04-09T18:12:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mark_out_threshold", 2021-04-09T18:12:53.705 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.706 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:53.707 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2419200", 2021-04-09T18:12:53.729 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.729 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.730 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.730 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically mark OSD if it may fail before this long", 2021-04-09T18:12:53.730 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.731 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.731 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.732 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": { 2021-04-09T18:12:53.732 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_name", 2021-04-09T18:12:53.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:53.733 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.733 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:53.763 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "device_health_metrics", 2021-04-09T18:12:53.764 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.764 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.767 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.768 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name of pool in which to store device health metrics", 2021-04-09T18:12:53.768 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.803 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.803 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.804 INFO:teuthology.orchestra.run.smithi032.stdout: "retention_period": { 2021-04-09T18:12:53.805 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retention_period", 2021-04-09T18:12:53.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:53.805 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.806 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:53.806 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15552000", 2021-04-09T18:12:53.807 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.828 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.828 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.829 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain device health metrics", 2021-04-09T18:12:53.829 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.829 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.830 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.857 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_frequency": { 2021-04-09T18:12:53.857 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_frequency", 2021-04-09T18:12:53.857 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:53.858 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:53.858 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2021-04-09T18:12:53.858 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.859 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.868 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.868 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to scrape device health metrics", 2021-04-09T18:12:53.869 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.869 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.883 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.884 INFO:teuthology.orchestra.run.smithi032.stdout: "self_heal": { 2021-04-09T18:12:53.928 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "self_heal", 2021-04-09T18:12:53.929 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:53.929 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.929 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:53.930 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:53.930 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:53.931 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:53.931 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:53.931 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "preemptively heal cluster around devices that may fail", 2021-04-09T18:12:53.932 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:53.932 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:53.933 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:53.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:53.933 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:12:53.933 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:12:53.934 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:53.963 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:53.972 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:54.049 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2021-04-09T18:12:54.049 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.050 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.050 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.050 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and check device health", 2021-04-09T18:12:54.051 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.051 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.051 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.052 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_threshold": { 2021-04-09T18:12:54.052 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_threshold", 2021-04-09T18:12:54.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:54.053 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.053 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:54.053 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7257600", 2021-04-09T18:12:54.053 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.054 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.054 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.054 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise health warning if OSD may fail before this long", 2021-04-09T18:12:54.054 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.054 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.055 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:54.094 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:54.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:54.094 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_cloud", 2021-04-09T18:12:54.095 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:54.095 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:54.095 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:54.096 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_cert_context": { 2021-04-09T18:12:54.096 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_cert_context", 2021-04-09T18:12:54.096 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.096 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.097 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.107 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:54.108 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.108 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.108 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.108 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.109 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.109 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.109 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.109 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_default_authority": { 2021-04-09T18:12:54.110 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_default_authority", 2021-04-09T18:12:54.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.129 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.129 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.129 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "localhost", 2021-04-09T18:12:54.129 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.130 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.131 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.131 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.132 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.132 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.132 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.133 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_password": { 2021-04-09T18:12:54.133 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_password", 2021-04-09T18:12:54.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.133 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.134 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.134 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:54.134 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.134 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.135 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.141 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.141 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.141 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.141 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.151 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_port": { 2021-04-09T18:12:54.151 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_port", 2021-04-09T18:12:54.152 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.152 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.155 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.155 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31400", 2021-04-09T18:12:54.156 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.156 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.156 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.156 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.157 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.157 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.157 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.158 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_retrieve_prediction_interval": { 2021-04-09T18:12:54.158 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_retrieve_prediction_interval", 2021-04-09T18:12:54.158 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.158 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.190 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.190 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "43200", 2021-04-09T18:12:54.190 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.196 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.197 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.197 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.198 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.198 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.198 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.206 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_server": { 2021-04-09T18:12:54.206 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_server", 2021-04-09T18:12:54.207 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.207 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.207 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.207 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:54.208 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.208 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.208 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.208 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.209 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.209 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.209 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.209 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_ssl_target_name_override": { 2021-04-09T18:12:54.210 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_ssl_target_name_override", 2021-04-09T18:12:54.210 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.210 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.210 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.210 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "localhost", 2021-04-09T18:12:54.211 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.211 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.211 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.211 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.211 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.212 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.212 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.212 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_upload_metrics_interval": { 2021-04-09T18:12:54.213 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_upload_metrics_interval", 2021-04-09T18:12:54.213 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.213 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.213 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.214 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2021-04-09T18:12:54.214 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.214 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.214 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.214 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.215 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.215 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.218 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.218 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_upload_smart_interval": { 2021-04-09T18:12:54.219 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_upload_smart_interval", 2021-04-09T18:12:54.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.219 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.219 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.220 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "43200", 2021-04-09T18:12:54.220 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.220 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.220 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.222 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.222 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.222 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.222 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.223 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_user": { 2021-04-09T18:12:54.223 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_user", 2021-04-09T18:12:54.223 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.223 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.224 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:54.224 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.224 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.224 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.225 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.225 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.225 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.226 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.226 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:12:54.227 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:12:54.227 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.228 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.228 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.229 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2021-04-09T18:12:54.229 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.229 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.230 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.230 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.231 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.308 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.308 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:54.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:54.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:54.310 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_local", 2021-04-09T18:12:54.310 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:54.311 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:54.311 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:54.312 INFO:teuthology.orchestra.run.smithi032.stdout: "predict_interval": { 2021-04-09T18:12:54.312 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predict_interval", 2021-04-09T18:12:54.312 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.312 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.313 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.313 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2021-04-09T18:12:54.314 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.314 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.314 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.315 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.339 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.339 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.340 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.341 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:12:54.341 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:12:54.342 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.463 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.464 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.464 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2021-04-09T18:12:54.464 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.464 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.465 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.465 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.465 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.465 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.466 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:54.481 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:54.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:54.482 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "influx", 2021-04-09T18:12:54.482 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": false, 2021-04-09T18:12:54.483 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "influxdb python module not found", 2021-04-09T18:12:54.483 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:54.513 INFO:teuthology.orchestra.run.smithi032.stdout: "batch_size": { 2021-04-09T18:12:54.514 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "batch_size", 2021-04-09T18:12:54.514 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.515 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.515 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.540 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5000", 2021-04-09T18:12:54.541 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.541 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.542 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.542 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.543 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.543 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.544 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.544 INFO:teuthology.orchestra.run.smithi032.stdout: "database": { 2021-04-09T18:12:54.545 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "database", 2021-04-09T18:12:54.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.545 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.546 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.613 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "ceph", 2021-04-09T18:12:54.613 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.614 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.614 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.615 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.615 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.615 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.639 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.656 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": { 2021-04-09T18:12:54.658 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hostname", 2021-04-09T18:12:54.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.660 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.660 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.661 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:12:54.661 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.661 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.662 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.662 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.662 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.663 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.663 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.719 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:12:54.719 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:12:54.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.720 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.721 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.721 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "30", 2021-04-09T18:12:54.721 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.722 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.722 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.737 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.738 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.738 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.738 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.740 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2021-04-09T18:12:54.740 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2021-04-09T18:12:54.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.742 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.743 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.743 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:12:54.747 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.763 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.764 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.764 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.766 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.881 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.881 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.882 INFO:teuthology.orchestra.run.smithi032.stdout: "port": { 2021-04-09T18:12:54.882 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "port", 2021-04-09T18:12:54.882 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.882 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.883 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8086", 2021-04-09T18:12:54.883 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.883 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.884 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.884 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.884 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.884 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.884 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2021-04-09T18:12:54.942 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2021-04-09T18:12:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.943 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.944 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "false", 2021-04-09T18:12:54.945 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.957 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.958 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.959 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.960 INFO:teuthology.orchestra.run.smithi032.stdout: "threads": { 2021-04-09T18:12:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threads", 2021-04-09T18:12:54.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:54.962 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2021-04-09T18:12:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:54.963 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:54.964 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:54.965 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2021-04-09T18:12:54.966 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2021-04-09T18:12:54.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:54.999 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.001 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:12:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.003 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_ssl": { 2021-04-09T18:12:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_ssl", 2021-04-09T18:12:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.005 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "true", 2021-04-09T18:12:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.006 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.007 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.008 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "insights", 2021-04-09T18:12:55.010 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.022 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:12:55.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.063 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "iostat", 2021-04-09T18:12:55.080 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.086 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.087 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:12:55.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.090 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "localpool", 2021-04-09T18:12:55.193 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.194 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.194 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:55.194 INFO:teuthology.orchestra.run.smithi032.stdout: "failure_domain": { 2021-04-09T18:12:55.194 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "failure_domain", 2021-04-09T18:12:55.195 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.195 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.195 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:55.195 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "host", 2021-04-09T18:12:55.211 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.211 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.212 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.212 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "failure domain for any created local pool", 2021-04-09T18:12:55.212 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.212 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.213 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.213 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": { 2021-04-09T18:12:55.213 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_size", 2021-04-09T18:12:55.213 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:55.214 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.214 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:55.214 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.214 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.215 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.215 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.226 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default min_size for any created local pool", 2021-04-09T18:12:55.226 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.226 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.226 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.231 INFO:teuthology.orchestra.run.smithi032.stdout: "num_rep": { 2021-04-09T18:12:55.231 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "num_rep", 2021-04-09T18:12:55.232 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:55.233 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.233 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:55.233 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3", 2021-04-09T18:12:55.234 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.234 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.234 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.234 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default replica count for any created local pool", 2021-04-09T18:12:55.235 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.235 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.235 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.236 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": { 2021-04-09T18:12:55.236 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_num", 2021-04-09T18:12:55.236 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:55.236 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.245 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:55.246 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "128", 2021-04-09T18:12:55.246 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.246 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.246 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.247 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default pg_num for any created local pool", 2021-04-09T18:12:55.247 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.247 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.257 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.257 INFO:teuthology.orchestra.run.smithi032.stdout: "prefix": { 2021-04-09T18:12:55.258 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prefix", 2021-04-09T18:12:55.261 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.262 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.262 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:55.263 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.267 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.268 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.268 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.268 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name prefix for any created local pool", 2021-04-09T18:12:55.269 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.269 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.269 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.270 INFO:teuthology.orchestra.run.smithi032.stdout: "subtree": { 2021-04-09T18:12:55.270 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "subtree", 2021-04-09T18:12:55.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.270 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.343 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:55.343 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "rack", 2021-04-09T18:12:55.343 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.344 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.344 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.344 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "CRUSH level for which to create a local pool", 2021-04-09T18:12:55.344 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.345 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.345 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.346 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.346 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator_cli", 2021-04-09T18:12:55.347 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.347 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.347 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:55.347 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator": { 2021-04-09T18:12:55.348 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2021-04-09T18:12:55.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.348 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.348 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.348 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.349 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.349 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.349 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.349 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.350 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.350 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.350 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.394 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_autoscaler", 2021-04-09T18:12:55.394 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.395 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.395 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:55.410 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:12:55.410 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:12:55.411 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.411 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.415 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2021-04-09T18:12:55.438 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.439 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.449 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.450 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.450 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.451 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.478 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.480 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "progress", 2021-04-09T18:12:55.480 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.481 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.481 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:55.481 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2021-04-09T18:12:55.481 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2021-04-09T18:12:55.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:55.482 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.482 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.494 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:55.494 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.494 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.495 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.495 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.495 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.495 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.496 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.506 INFO:teuthology.orchestra.run.smithi032.stdout: "max_completed_events": { 2021-04-09T18:12:55.506 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_completed_events", 2021-04-09T18:12:55.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:55.507 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.507 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:55.507 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "50", 2021-04-09T18:12:55.508 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.508 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.508 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.508 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "number of past completed events to remember", 2021-04-09T18:12:55.509 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.509 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.509 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.509 INFO:teuthology.orchestra.run.smithi032.stdout: "persist_interval": { 2021-04-09T18:12:55.510 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "persist_interval", 2021-04-09T18:12:55.510 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:55.517 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.523 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:55.524 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2021-04-09T18:12:55.524 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.526 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.526 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.526 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to persist completed events", 2021-04-09T18:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.551 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus", 2021-04-09T18:12:55.559 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.578 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.578 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:55.579 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools": { 2021-04-09T18:12:55.579 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools", 2021-04-09T18:12:55.580 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.580 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.580 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.581 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.581 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.582 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.582 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.582 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.611 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.612 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.615 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.617 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools_refresh_interval": { 2021-04-09T18:12:55.617 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools_refresh_interval", 2021-04-09T18:12:55.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:55.626 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.627 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.627 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "300", 2021-04-09T18:12:55.628 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.628 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.628 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.629 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.629 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.630 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.631 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.668 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_interval": { 2021-04-09T18:12:55.669 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_interval", 2021-04-09T18:12:55.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.669 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.670 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.670 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.679 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.679 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.699 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.722 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.722 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.723 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.753 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2021-04-09T18:12:55.753 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2021-04-09T18:12:55.753 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.754 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.754 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.755 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.755 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.755 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.756 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.756 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.757 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.758 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2021-04-09T18:12:55.759 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2021-04-09T18:12:55.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.774 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.774 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.774 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.775 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.775 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.776 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.776 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.777 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.828 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.829 INFO:teuthology.orchestra.run.smithi032.stdout: "stale_cache_strategy": { 2021-04-09T18:12:55.829 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stale_cache_strategy", 2021-04-09T18:12:55.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.830 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.830 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.832 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.833 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.833 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.833 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.834 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.834 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.835 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.835 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.837 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2021-04-09T18:12:55.838 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.839 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.839 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:12:55.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.843 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "restful", 2021-04-09T18:12:55.843 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.843 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.844 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:55.844 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2021-04-09T18:12:55.845 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2021-04-09T18:12:55.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.845 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.846 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.846 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.847 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.847 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.848 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.848 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.849 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.851 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.852 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.853 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2021-04-09T18:12:55.853 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2021-04-09T18:12:55.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.855 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.856 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.857 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.857 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.858 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.858 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.858 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.860 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.862 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2021-04-09T18:12:55.863 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2021-04-09T18:12:55.863 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.864 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.864 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.865 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.866 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.874 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rook", 2021-04-09T18:12:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.876 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:12:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "selftest", 2021-04-09T18:12:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.877 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "roption1": { 2021-04-09T18:12:55.878 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption1", 2021-04-09T18:12:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.879 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.880 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "roption2": { 2021-04-09T18:12:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption2", 2021-04-09T18:12:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.882 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "xyz", 2021-04-09T18:12:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.883 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.884 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption1": { 2021-04-09T18:12:55.885 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption1", 2021-04-09T18:12:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.886 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.887 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.888 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.889 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption2": { 2021-04-09T18:12:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption2", 2021-04-09T18:12:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:55.890 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.893 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.894 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.895 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.896 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption3": { 2021-04-09T18:12:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption3", 2021-04-09T18:12:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2021-04-09T18:12:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.897 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.898 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.899 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.903 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption4": { 2021-04-09T18:12:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption4", 2021-04-09T18:12:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.905 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.906 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.907 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption5": { 2021-04-09T18:12:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption5", 2021-04-09T18:12:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:55.908 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.909 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.910 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.911 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.912 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.912 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.912 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption6": { 2021-04-09T18:12:55.912 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption6", 2021-04-09T18:12:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.913 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:55.914 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.915 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "testkey": { 2021-04-09T18:12:55.917 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testkey", 2021-04-09T18:12:55.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.949 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.949 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.949 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.949 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.950 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.950 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.950 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.951 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.951 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.951 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.955 INFO:teuthology.orchestra.run.smithi032.stdout: "testlkey": { 2021-04-09T18:12:55.956 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testlkey", 2021-04-09T18:12:55.956 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.956 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.957 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.957 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.957 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.957 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.958 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.958 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.962 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.962 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.962 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.962 INFO:teuthology.orchestra.run.smithi032.stdout: "testnewline": { 2021-04-09T18:12:55.963 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testnewline", 2021-04-09T18:12:55.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.963 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.964 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.964 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.964 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.964 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.964 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.965 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.965 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.965 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.965 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.974 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh", 2021-04-09T18:12:55.974 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.975 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.975 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:55.975 INFO:teuthology.orchestra.run.smithi032.stdout: "inventory_cache_timeout_min": { 2021-04-09T18:12:55.975 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "inventory_cache_timeout_min", 2021-04-09T18:12:55.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.976 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.978 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.979 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.987 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.988 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.988 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.988 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.989 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.989 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.989 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.990 INFO:teuthology.orchestra.run.smithi032.stdout: "ssh_config_file": { 2021-04-09T18:12:55.990 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh_config_file", 2021-04-09T18:12:55.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:55.990 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:55.990 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:55.991 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:55.992 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:55.992 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:55.992 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:55.993 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:55.993 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:55.994 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:55.994 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:55.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:55.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.995 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "status", 2021-04-09T18:12:55.995 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.996 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.996 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:12:55.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:55.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:55.996 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telegraf", 2021-04-09T18:12:55.997 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:55.997 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:55.997 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:56.004 INFO:teuthology.orchestra.run.smithi032.stdout: "address": { 2021-04-09T18:12:56.004 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "address", 2021-04-09T18:12:56.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.005 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.017 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.017 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2021-04-09T18:12:56.017 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.018 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.018 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.018 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.021 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.021 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.021 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.022 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:12:56.022 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:12:56.022 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:56.022 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.022 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.035 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15", 2021-04-09T18:12:56.036 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.036 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.036 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.037 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.037 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.037 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.037 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.038 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.038 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:56.042 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telemetry", 2021-04-09T18:12:56.043 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:56.044 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:56.047 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:56.047 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_basic": { 2021-04-09T18:12:56.048 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_basic", 2021-04-09T18:12:56.049 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:56.049 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.049 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.049 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:56.049 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.050 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.050 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.050 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share basic cluster information (size, version)", 2021-04-09T18:12:56.050 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.051 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.117 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.118 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_crash": { 2021-04-09T18:12:56.121 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_crash", 2021-04-09T18:12:56.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:56.122 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.122 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.122 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:56.123 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.123 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.124 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.124 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.125 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.125 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.125 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.143 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_device": { 2021-04-09T18:12:56.144 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_device", 2021-04-09T18:12:56.144 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:56.145 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.145 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.145 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:56.146 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.146 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.147 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.147 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.288 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.289 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.289 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.290 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_ident": { 2021-04-09T18:12:56.290 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_ident", 2021-04-09T18:12:56.291 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:56.300 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.305 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.306 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:56.306 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.309 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.313 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.313 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.322 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.323 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.323 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.324 INFO:teuthology.orchestra.run.smithi032.stdout: "contact": { 2021-04-09T18:12:56.325 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "contact", 2021-04-09T18:12:56.325 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.335 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.335 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.343 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:12:56.343 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.344 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.347 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.347 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.348 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.348 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.355 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.356 INFO:teuthology.orchestra.run.smithi032.stdout: "description": { 2021-04-09T18:12:56.356 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "description", 2021-04-09T18:12:56.357 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.357 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.358 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.358 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:12:56.359 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.371 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.372 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.372 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.373 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.373 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.373 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.374 INFO:teuthology.orchestra.run.smithi032.stdout: "device_url": { 2021-04-09T18:12:56.374 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_url", 2021-04-09T18:12:56.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.375 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.375 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.376 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/device", 2021-04-09T18:12:56.376 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.376 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.377 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.377 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.378 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.397 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.397 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.398 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2021-04-09T18:12:56.398 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2021-04-09T18:12:56.398 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:56.398 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.399 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.423 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:56.424 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.424 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.424 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.425 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.425 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.426 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.426 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.427 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:12:56.498 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:12:56.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:56.499 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.499 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.499 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "24", 2021-04-09T18:12:56.500 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "8", 2021-04-09T18:12:56.500 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.501 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.501 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.502 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.502 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.545 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.546 INFO:teuthology.orchestra.run.smithi032.stdout: "last_opt_revision": { 2021-04-09T18:12:56.547 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "last_opt_revision", 2021-04-09T18:12:56.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:56.548 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.548 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.549 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1", 2021-04-09T18:12:56.549 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.549 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.550 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.550 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.551 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.576 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.576 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.577 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard": { 2021-04-09T18:12:56.578 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard", 2021-04-09T18:12:56.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:56.578 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.579 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.579 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:56.580 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.580 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.581 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.581 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.603 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.603 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.604 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.605 INFO:teuthology.orchestra.run.smithi032.stdout: "organization": { 2021-04-09T18:12:56.605 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "organization", 2021-04-09T18:12:56.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.606 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.616 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.617 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:12:56.617 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.645 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.646 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.663 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.664 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.664 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.664 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.668 INFO:teuthology.orchestra.run.smithi032.stdout: "proxy": { 2021-04-09T18:12:56.668 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "proxy", 2021-04-09T18:12:56.668 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.669 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.669 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.670 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:12:56.680 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.680 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.681 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.681 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.681 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.682 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.691 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.699 INFO:teuthology.orchestra.run.smithi032.stdout: "url": { 2021-04-09T18:12:56.705 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url", 2021-04-09T18:12:56.706 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.707 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.712 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.713 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/report", 2021-04-09T18:12:56.713 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.714 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.728 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.728 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.731 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.732 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.738 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:56.743 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "test_orchestrator", 2021-04-09T18:12:56.746 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:56.746 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:56.746 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:12:56.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:56.752 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2021-04-09T18:12:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:56.753 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:56.765 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_clones": { 2021-04-09T18:12:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_clones", 2021-04-09T18:12:56.766 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.767 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4", 2021-04-09T18:12:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.768 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Number of asynchronous cloner threads", 2021-04-09T18:12:56.769 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.770 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.771 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix", 2021-04-09T18:12:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:56.814 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "identifier": { 2021-04-09T18:12:56.815 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "identifier", 2021-04-09T18:12:56.856 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.856 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.856 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:56.857 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.857 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.857 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.899 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.901 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.901 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.901 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.902 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:12:56.903 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:12:56.903 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:56.903 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.904 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2021-04-09T18:12:56.905 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.905 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.905 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.906 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.906 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.907 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.927 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.928 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_host": { 2021-04-09T18:12:56.929 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_host", 2021-04-09T18:12:56.929 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.930 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.930 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.930 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:12:56.931 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.941 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.942 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.942 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.949 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.949 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.950 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.950 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_port": { 2021-04-09T18:12:56.950 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_port", 2021-04-09T18:12:56.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:56.951 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.951 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.951 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10051", 2021-04-09T18:12:56.954 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.955 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.955 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.955 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.956 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.956 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.956 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:56.964 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_sender": { 2021-04-09T18:12:56.965 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_sender", 2021-04-09T18:12:56.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:56.965 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:56.965 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:56.965 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/usr/bin/zabbix_sender", 2021-04-09T18:12:56.966 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:56.966 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:56.966 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:56.966 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:56.967 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:56.967 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:56.967 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:56.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.968 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.968 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:12:56.968 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:56.978 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:56.978 INFO:teuthology.orchestra.run.smithi032.stdout: "modules": [ 2021-04-09T18:12:56.978 INFO:teuthology.orchestra.run.smithi032.stdout: "iostat", 2021-04-09T18:12:56.979 INFO:teuthology.orchestra.run.smithi032.stdout: "restful" 2021-04-09T18:12:56.979 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:57.107 INFO:teuthology.orchestra.run.smithi032.stdout: "available_modules": [ 2021-04-09T18:12:57.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:57.109 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "alerts", 2021-04-09T18:12:57.109 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:57.110 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:57.110 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:57.110 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:12:57.111 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:12:57.111 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:57.112 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.112 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.113 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2021-04-09T18:12:57.113 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.113 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.114 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.114 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How frequently to reexamine health status", 2021-04-09T18:12:57.115 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.154 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.154 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.155 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_destination": { 2021-04-09T18:12:57.155 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_destination", 2021-04-09T18:12:57.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.163 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.163 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.163 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:57.164 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.164 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.164 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.164 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email address to send alerts to", 2021-04-09T18:12:57.164 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.165 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.165 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.165 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_from_name": { 2021-04-09T18:12:57.166 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_from_name", 2021-04-09T18:12:57.166 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.166 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.166 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.166 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "Ceph", 2021-04-09T18:12:57.167 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.167 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.167 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.167 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email From: name", 2021-04-09T18:12:57.167 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.168 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.196 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.196 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_host": { 2021-04-09T18:12:57.202 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_host", 2021-04-09T18:12:57.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.202 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.202 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.203 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:57.203 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.203 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.203 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.204 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP server", 2021-04-09T18:12:57.204 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.204 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.205 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.205 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_password": { 2021-04-09T18:12:57.205 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_password", 2021-04-09T18:12:57.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.206 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.206 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.206 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:57.206 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.206 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.207 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.207 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Password to authenticate with", 2021-04-09T18:12:57.207 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.207 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.208 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.208 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_port": { 2021-04-09T18:12:57.208 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_port", 2021-04-09T18:12:57.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:57.209 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.215 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "465", 2021-04-09T18:12:57.215 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.216 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.216 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.216 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP port", 2021-04-09T18:12:57.221 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.221 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.222 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_sender": { 2021-04-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_sender", 2021-04-09T18:12:57.223 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.224 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.224 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:57.227 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.227 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.227 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.233 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP envelope sender", 2021-04-09T18:12:57.234 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.235 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.236 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.237 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_ssl": { 2021-04-09T18:12:57.237 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_ssl", 2021-04-09T18:12:57.237 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:57.238 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.238 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.238 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:57.239 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.273 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.274 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.274 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use SSL to connect to SMTP server", 2021-04-09T18:12:57.274 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.275 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.283 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.284 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_user": { 2021-04-09T18:12:57.285 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_user", 2021-04-09T18:12:57.285 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.285 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.286 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.286 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:57.287 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.300 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.301 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.310 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "User to authenticate as", 2021-04-09T18:12:57.311 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.311 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.312 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:57.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:57.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:57.314 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ansible", 2021-04-09T18:12:57.315 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:57.315 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:57.316 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:57.316 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2021-04-09T18:12:57.316 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2021-04-09T18:12:57.317 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.335 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.335 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:57.336 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:57.336 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.337 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.358 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.360 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:57.360 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.360 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.361 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.364 INFO:teuthology.orchestra.run.smithi032.stdout: "server_url": { 2021-04-09T18:12:57.372 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_url", 2021-04-09T18:12:57.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.412 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.412 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:57.412 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:57.413 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.413 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.414 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.414 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:57.414 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.415 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.430 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.431 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2021-04-09T18:12:57.432 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2021-04-09T18:12:57.432 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.433 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.433 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:57.433 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:57.438 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.439 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.439 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.440 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:57.440 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.478 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.479 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.493 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_server": { 2021-04-09T18:12:57.494 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_server", 2021-04-09T18:12:57.494 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.494 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.497 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:57.498 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:57.499 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.499 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.500 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.500 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:57.501 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.501 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.501 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.502 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:57.502 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:57.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:57.576 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "balancer", 2021-04-09T18:12:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "active": { 2021-04-09T18:12:57.665 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "active", 2021-04-09T18:12:57.668 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:57.669 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.670 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically balance PGs across cluster", 2021-04-09T18:12:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.705 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.707 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_time": { 2021-04-09T18:12:57.725 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_time", 2021-04-09T18:12:57.726 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.726 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.727 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.727 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0000", 2021-04-09T18:12:57.756 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.757 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "beginning time of day to automatically balance", 2021-04-09T18:12:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-04-09T18:12:57.758 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.781 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_weekday": { 2021-04-09T18:12:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_weekday", 2021-04-09T18:12:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2021-04-09T18:12:57.782 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.783 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2021-04-09T18:12:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2021-04-09T18:12:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "7", 2021-04-09T18:12:57.784 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2021-04-09T18:12:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-04-09T18:12:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.820 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_max_iterations": { 2021-04-09T18:12:57.824 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_max_iterations", 2021-04-09T18:12:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2021-04-09T18:12:57.825 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.826 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.826 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "25", 2021-04-09T18:12:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2021-04-09T18:12:57.827 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "250", 2021-04-09T18:12:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum number of iterations to attempt optimization", 2021-04-09T18:12:57.828 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:57.829 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_metrics": { 2021-04-09T18:12:57.869 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_metrics", 2021-04-09T18:12:57.883 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.888 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "pgs,objects,bytes", 2021-04-09T18:12:57.889 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.890 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.890 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.891 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "metrics with which to calculate OSD utilization", 2021-04-09T18:12:57.954 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2021-04-09T18:12:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.955 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.956 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_step": { 2021-04-09T18:12:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_step", 2021-04-09T18:12:57.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2021-04-09T18:12:57.964 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.965 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.5", 2021-04-09T18:12:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0.001", 2021-04-09T18:12:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "0.999", 2021-04-09T18:12:57.966 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "aggressiveness of optimization", 2021-04-09T18:12:57.967 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2021-04-09T18:12:57.968 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.970 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "end_time": { 2021-04-09T18:12:57.972 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_time", 2021-04-09T18:12:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.973 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2400", 2021-04-09T18:12:57.974 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:57.975 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ending time of day to automatically balance", 2021-04-09T18:12:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-04-09T18:12:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:57.976 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:57.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "end_weekday": { 2021-04-09T18:12:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_weekday", 2021-04-09T18:12:57.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2021-04-09T18:12:57.991 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:57.992 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7", 2021-04-09T18:12:57.993 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2021-04-09T18:12:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "7", 2021-04-09T18:12:58.001 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2021-04-09T18:12:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-04-09T18:12:58.002 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.003 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: "min_score": { 2021-04-09T18:12:58.006 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_score", 2021-04-09T18:12:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2021-04-09T18:12:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2021-04-09T18:12:58.008 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.009 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "minimum score, below which no optimization is attempted", 2021-04-09T18:12:58.017 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.018 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.019 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2021-04-09T18:12:58.021 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2021-04-09T18:12:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.022 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:58.023 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "none", 2021-04-09T18:12:58.031 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.032 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2021-04-09T18:12:58.033 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-compat", 2021-04-09T18:12:58.075 INFO:teuthology.orchestra.run.smithi032.stdout: "none", 2021-04-09T18:12:58.079 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap" 2021-04-09T18:12:58.083 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:58.083 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Balancer mode", 2021-04-09T18:12:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.084 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_ids": { 2021-04-09T18:12:58.085 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_ids", 2021-04-09T18:12:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.086 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:58.110 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.111 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.116 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "pools which the automatic balancing will be limited to", 2021-04-09T18:12:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.117 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.118 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:12:58.149 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:12:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:58.156 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.157 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:58.165 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2021-04-09T18:12:58.169 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.174 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.174 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.175 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and attempt optimization", 2021-04-09T18:12:58.175 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.175 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.198 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.198 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_deviation": { 2021-04-09T18:12:58.199 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_deviation", 2021-04-09T18:12:58.199 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:58.200 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.200 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:58.201 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2021-04-09T18:12:58.201 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2021-04-09T18:12:58.251 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.251 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.252 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "deviation below which no optimization is attempted", 2021-04-09T18:12:58.253 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2021-04-09T18:12:58.253 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.270 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.274 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_iterations": { 2021-04-09T18:12:58.274 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_iterations", 2021-04-09T18:12:58.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2021-04-09T18:12:58.277 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.277 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:58.277 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2021-04-09T18:12:58.278 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.278 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.278 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.290 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum upmap optimization iterations", 2021-04-09T18:12:58.290 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.290 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.305 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:58.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:58.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:58.311 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crash", 2021-04-09T18:12:58.312 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:58.312 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:58.320 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:58.321 INFO:teuthology.orchestra.run.smithi032.stdout: "retain_interval": { 2021-04-09T18:12:58.321 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retain_interval", 2021-04-09T18:12:58.330 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:58.330 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.331 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:58.331 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31536000", 2021-04-09T18:12:58.381 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.382 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.382 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.382 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain crashes before pruning them", 2021-04-09T18:12:58.383 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.383 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.384 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.431 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_recent_interval": { 2021-04-09T18:12:58.431 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_recent_interval", 2021-04-09T18:12:58.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:12:58.432 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.432 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:12:58.433 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1209600", 2021-04-09T18:12:58.433 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.433 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.434 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.441 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "time interval in which to warn about recent crashes", 2021-04-09T18:12:58.441 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.441 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.442 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:58.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:12:58.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:12:58.443 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dashboard", 2021-04-09T18:12:58.473 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:12:58.473 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:12:58.474 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:12:58.474 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_HOST": { 2021-04-09T18:12:58.475 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_HOST", 2021-04-09T18:12:58.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:58.475 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.476 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.476 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:58.476 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.569 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.569 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.573 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.574 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.574 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.574 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.575 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_ENABLED": { 2021-04-09T18:12:58.605 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_ENABLED", 2021-04-09T18:12:58.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:58.608 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.617 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.619 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:58.620 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.628 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.629 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.636 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.637 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.637 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.638 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.639 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_LOG_PAYLOAD": { 2021-04-09T18:12:58.639 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2021-04-09T18:12:58.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:58.640 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.640 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.641 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:58.702 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.707 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.708 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.708 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.709 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.709 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.710 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.711 INFO:teuthology.orchestra.run.smithi032.stdout: "ENABLE_BROWSABLE_API": { 2021-04-09T18:12:58.711 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ENABLE_BROWSABLE_API", 2021-04-09T18:12:58.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:58.758 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.758 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.758 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:58.759 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.759 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.760 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.776 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.776 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.777 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.777 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.785 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_cephfs": { 2021-04-09T18:12:58.785 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_cephfs", 2021-04-09T18:12:58.786 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:58.786 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.788 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.788 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:58.790 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.791 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.791 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.792 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.792 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.793 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.793 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.800 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_iscsi": { 2021-04-09T18:12:58.801 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_iscsi", 2021-04-09T18:12:58.801 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:58.802 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.802 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.856 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:58.856 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.856 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.857 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.857 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.857 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.857 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.858 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.858 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_mirroring": { 2021-04-09T18:12:58.858 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_mirroring", 2021-04-09T18:12:58.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:58.859 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.859 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.859 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:58.859 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.860 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.860 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.860 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.861 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.877 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.877 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.916 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_rbd": { 2021-04-09T18:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_rbd", 2021-04-09T18:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:58.926 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.927 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:58.927 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.927 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.927 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.928 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.928 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.928 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.928 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.940 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_rgw": { 2021-04-09T18:12:58.940 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_rgw", 2021-04-09T18:12:58.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:58.940 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.941 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.941 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:58.941 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.941 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.942 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.942 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.973 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.974 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.977 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.978 INFO:teuthology.orchestra.run.smithi032.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2021-04-09T18:12:58.978 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2021-04-09T18:12:58.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:58.979 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.979 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.979 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:58.979 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:58.980 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:58.980 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:58.980 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:58.980 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:58.980 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:58.981 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:58.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:58.981 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_PASSWORD": { 2021-04-09T18:12:58.981 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_PASSWORD", 2021-04-09T18:12:58.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:58.982 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:58.982 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:58.982 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2021-04-09T18:12:59.010 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.012 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.012 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.018 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.018 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.018 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.021 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.031 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_SSL_VERIFY": { 2021-04-09T18:12:59.031 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2021-04-09T18:12:59.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:59.037 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.039 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.039 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:59.046 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.046 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.047 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.047 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.048 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.049 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.049 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.050 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_URL": { 2021-04-09T18:12:59.050 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_URL", 2021-04-09T18:12:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.056 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.058 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.060 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.086 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.086 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.086 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.087 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.087 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.087 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_USERNAME": { 2021-04-09T18:12:59.088 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_USERNAME", 2021-04-09T18:12:59.088 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.088 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.088 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.089 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2021-04-09T18:12:59.089 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.089 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.089 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.090 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.090 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.090 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.090 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.091 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_FRONTEND_API_URL": { 2021-04-09T18:12:59.091 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2021-04-09T18:12:59.091 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.101 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.101 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.101 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.101 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.102 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.104 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.105 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.105 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.106 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.106 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.106 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2021-04-09T18:12:59.107 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2021-04-09T18:12:59.107 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:59.107 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.108 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.108 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:59.108 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.108 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.126 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.126 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.126 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.126 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.127 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.132 INFO:teuthology.orchestra.run.smithi032.stdout: "ISCSI_API_SSL_VERIFICATION": { 2021-04-09T18:12:59.136 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2021-04-09T18:12:59.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:59.136 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.156 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.157 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:59.157 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.157 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.157 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.158 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.158 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.164 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.164 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.165 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_HOST": { 2021-04-09T18:12:59.166 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_HOST", 2021-04-09T18:12:59.166 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.168 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.168 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.169 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.169 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.170 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.170 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.170 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.171 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.171 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.179 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.180 INFO:teuthology.orchestra.run.smithi032.stdout: "REST_REQUESTS_TIMEOUT": { 2021-04-09T18:12:59.181 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "REST_REQUESTS_TIMEOUT", 2021-04-09T18:12:59.181 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:59.182 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.183 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.183 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "45", 2021-04-09T18:12:59.184 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.196 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.197 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.197 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.198 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.204 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.204 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.211 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ACCESS_KEY": { 2021-04-09T18:12:59.212 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ACCESS_KEY", 2021-04-09T18:12:59.212 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.213 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.213 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.214 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.215 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.215 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.216 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.216 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.226 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.226 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.227 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.250 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ADMIN_RESOURCE": { 2021-04-09T18:12:59.259 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2021-04-09T18:12:59.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.261 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.282 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.283 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2021-04-09T18:12:59.301 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.302 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.302 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.302 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.303 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.304 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.304 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.305 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_HOST": { 2021-04-09T18:12:59.349 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_HOST", 2021-04-09T18:12:59.351 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.352 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.358 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.358 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.361 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.362 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.373 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.373 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.376 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.376 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.399 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.405 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_PORT": { 2021-04-09T18:12:59.405 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_PORT", 2021-04-09T18:12:59.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:59.410 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.443 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.444 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "80", 2021-04-09T18:12:59.444 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.444 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.450 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.452 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.456 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.456 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.457 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.461 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SCHEME": { 2021-04-09T18:12:59.461 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SCHEME", 2021-04-09T18:12:59.469 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.469 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.470 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.470 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "http", 2021-04-09T18:12:59.471 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.471 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.472 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.472 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.472 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.479 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.480 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.481 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SECRET_KEY": { 2021-04-09T18:12:59.482 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SECRET_KEY", 2021-04-09T18:12:59.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.483 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.483 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.484 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.484 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.485 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.485 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.485 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.486 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.487 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.488 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.489 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SSL_VERIFY": { 2021-04-09T18:12:59.489 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SSL_VERIFY", 2021-04-09T18:12:59.489 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:59.490 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.493 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.494 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:59.494 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.494 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.495 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.495 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.496 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.496 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.496 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.508 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_USER_ID": { 2021-04-09T18:12:59.508 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_USER_ID", 2021-04-09T18:12:59.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.509 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.509 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.510 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.510 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.511 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.511 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.512 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.512 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.512 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.513 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.514 INFO:teuthology.orchestra.run.smithi032.stdout: "crt_file": { 2021-04-09T18:12:59.514 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crt_file", 2021-04-09T18:12:59.514 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.515 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.515 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.515 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.516 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.517 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.517 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.518 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.518 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.518 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.519 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.520 INFO:teuthology.orchestra.run.smithi032.stdout: "debug": { 2021-04-09T18:12:59.520 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "debug", 2021-04-09T18:12:59.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:59.521 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.521 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.522 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:12:59.522 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.523 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.524 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable/disable debug options", 2021-04-09T18:12:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.646 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.646 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.647 INFO:teuthology.orchestra.run.smithi032.stdout: "jwt_token_ttl": { 2021-04-09T18:12:59.647 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "jwt_token_ttl", 2021-04-09T18:12:59.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:59.648 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.653 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "28800", 2021-04-09T18:12:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.660 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.662 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.701 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.702 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.702 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.703 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.703 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2021-04-09T18:12:59.704 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2021-04-09T18:12:59.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.705 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.705 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.705 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.706 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.706 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.707 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.707 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.707 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.708 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.708 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.709 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2021-04-09T18:12:59.709 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2021-04-09T18:12:59.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.710 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.710 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.792 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.792 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.792 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.792 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.793 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.793 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.793 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.793 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.794 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2021-04-09T18:12:59.794 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2021-04-09T18:12:59.794 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.794 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.794 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.795 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2021-04-09T18:12:59.795 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.795 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.795 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.796 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.796 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.796 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.796 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.797 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2021-04-09T18:12:59.797 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2021-04-09T18:12:59.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:59.797 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.798 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.798 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8080", 2021-04-09T18:12:59.798 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.798 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.798 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.799 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.799 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.827 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.827 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.828 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2021-04-09T18:12:59.838 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2021-04-09T18:12:59.840 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:12:59.840 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.840 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.841 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:12:59.841 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.841 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.852 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.853 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.856 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.857 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.857 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.857 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl_server_port": { 2021-04-09T18:12:59.858 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl_server_port", 2021-04-09T18:12:59.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:59.858 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.858 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8443", 2021-04-09T18:12:59.859 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.859 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.859 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.859 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.860 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.860 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.860 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.870 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2021-04-09T18:12:59.870 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2021-04-09T18:12:59.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.871 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.871 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.871 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "redirect", 2021-04-09T18:12:59.871 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.871 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.872 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2021-04-09T18:12:59.872 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2021-04-09T18:12:59.872 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect" 2021-04-09T18:12:59.874 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:12:59.875 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.875 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.876 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.876 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.877 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2021-04-09T18:12:59.877 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2021-04-09T18:12:59.877 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:12:59.878 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.878 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.878 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2021-04-09T18:12:59.878 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2021-04-09T18:12:59.879 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2021-04-09T18:12:59.879 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.879 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.879 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.883 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.883 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.883 INFO:teuthology.orchestra.run.smithi032.stdout: "url_prefix": { 2021-04-09T18:12:59.911 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url_prefix", 2021-04-09T18:12:59.911 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:12:59.920 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:12:59.920 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:12:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:12:59.921 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:12:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:12:59.922 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:12:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:12:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:12:59.923 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:12:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:12:59.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:12:59.925 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2021-04-09T18:12:59.999 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2021-04-09T18:13:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.000 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.018 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.047 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.047 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.048 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.048 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.049 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.049 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:00.049 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:00.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:00.051 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "deepsea", 2021-04-09T18:13:00.071 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:00.071 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:00.071 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:00.072 INFO:teuthology.orchestra.run.smithi032.stdout: "salt_api_eauth": { 2021-04-09T18:13:00.072 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "salt_api_eauth", 2021-04-09T18:13:00.072 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.072 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.075 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.075 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "sharedsecret", 2021-04-09T18:13:00.076 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.076 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.077 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.077 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.077 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.077 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.081 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.082 INFO:teuthology.orchestra.run.smithi032.stdout: "salt_api_password": { 2021-04-09T18:13:00.082 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "salt_api_password", 2021-04-09T18:13:00.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.083 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.083 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.083 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:00.084 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.084 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.084 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.084 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.090 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.091 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.091 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.092 INFO:teuthology.orchestra.run.smithi032.stdout: "salt_api_url": { 2021-04-09T18:13:00.092 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "salt_api_url", 2021-04-09T18:13:00.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.092 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.093 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.093 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:00.093 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.093 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.094 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.094 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.094 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.095 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.095 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.100 INFO:teuthology.orchestra.run.smithi032.stdout: "salt_api_username": { 2021-04-09T18:13:00.102 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "salt_api_username", 2021-04-09T18:13:00.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.103 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.104 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.104 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:00.105 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.105 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.105 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.107 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.110 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.110 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.111 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.162 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:00.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:00.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:00.165 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2021-04-09T18:13:00.174 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:00.175 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:00.175 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:00.176 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_monitoring": { 2021-04-09T18:13:00.176 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_monitoring", 2021-04-09T18:13:00.177 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:00.177 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.177 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:00.191 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:13:00.191 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.192 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.192 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.193 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "monitor device health metrics", 2021-04-09T18:13:00.193 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.193 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.193 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.220 INFO:teuthology.orchestra.run.smithi032.stdout: "mark_out_threshold": { 2021-04-09T18:13:00.221 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mark_out_threshold", 2021-04-09T18:13:00.221 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:13:00.221 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.222 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:00.222 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2419200", 2021-04-09T18:13:00.250 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.250 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.250 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.250 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically mark OSD if it may fail before this long", 2021-04-09T18:13:00.251 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.251 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.251 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.252 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": { 2021-04-09T18:13:00.252 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_name", 2021-04-09T18:13:00.252 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.252 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.252 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:00.253 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "device_health_metrics", 2021-04-09T18:13:00.277 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.280 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.280 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.281 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name of pool in which to store device health metrics", 2021-04-09T18:13:00.281 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.281 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.282 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.337 INFO:teuthology.orchestra.run.smithi032.stdout: "retention_period": { 2021-04-09T18:13:00.337 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retention_period", 2021-04-09T18:13:00.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:13:00.338 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.338 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:00.339 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15552000", 2021-04-09T18:13:00.339 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.340 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.340 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.340 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain device health metrics", 2021-04-09T18:13:00.341 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.402 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.403 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.404 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_frequency": { 2021-04-09T18:13:00.404 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_frequency", 2021-04-09T18:13:00.404 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:13:00.405 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.428 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:00.428 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2021-04-09T18:13:00.429 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.429 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.430 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.430 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to scrape device health metrics", 2021-04-09T18:13:00.436 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.436 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.452 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.453 INFO:teuthology.orchestra.run.smithi032.stdout: "self_heal": { 2021-04-09T18:13:00.453 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "self_heal", 2021-04-09T18:13:00.453 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:00.454 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.454 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:00.471 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:13:00.475 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.475 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.476 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.476 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "preemptively heal cluster around devices that may fail", 2021-04-09T18:13:00.502 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.503 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.517 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:13:00.518 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:13:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:13:00.519 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:00.520 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2021-04-09T18:13:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.521 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and check device health", 2021-04-09T18:13:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.522 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.532 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_threshold": { 2021-04-09T18:13:00.533 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_threshold", 2021-04-09T18:13:00.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:13:00.537 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:00.542 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7257600", 2021-04-09T18:13:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.543 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.544 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise health warning if OSD may fail before this long", 2021-04-09T18:13:00.560 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.561 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:00.562 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:00.563 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_cloud", 2021-04-09T18:13:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:00.564 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:00.565 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_cert_context": { 2021-04-09T18:13:00.623 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_cert_context", 2021-04-09T18:13:00.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.623 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.624 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.624 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:00.624 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.624 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.625 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.625 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.625 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.625 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.626 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.626 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_default_authority": { 2021-04-09T18:13:00.626 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_default_authority", 2021-04-09T18:13:00.627 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.632 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.633 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.633 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "localhost", 2021-04-09T18:13:00.633 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.633 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.634 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.634 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.634 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.634 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.635 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.635 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_password": { 2021-04-09T18:13:00.636 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_password", 2021-04-09T18:13:00.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.636 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.636 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.637 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:00.637 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.637 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.638 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.638 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.638 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.639 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.639 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.639 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_port": { 2021-04-09T18:13:00.640 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_port", 2021-04-09T18:13:00.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.640 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.640 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.641 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31400", 2021-04-09T18:13:00.641 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.642 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.642 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.642 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.642 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.642 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.643 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.643 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_retrieve_prediction_interval": { 2021-04-09T18:13:00.643 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_retrieve_prediction_interval", 2021-04-09T18:13:00.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.644 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.654 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "43200", 2021-04-09T18:13:00.654 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.654 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.657 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.657 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.657 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.661 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.663 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.663 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_server": { 2021-04-09T18:13:00.667 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_server", 2021-04-09T18:13:00.668 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.668 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.668 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.668 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:00.669 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.669 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.670 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.670 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.670 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.671 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.671 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.673 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_ssl_target_name_override": { 2021-04-09T18:13:00.673 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_ssl_target_name_override", 2021-04-09T18:13:00.673 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.674 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.674 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.674 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "localhost", 2021-04-09T18:13:00.674 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.675 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.675 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.675 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.675 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.676 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.676 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.677 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_upload_metrics_interval": { 2021-04-09T18:13:00.677 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_upload_metrics_interval", 2021-04-09T18:13:00.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.677 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.677 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.678 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2021-04-09T18:13:00.678 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.678 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.678 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.679 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.679 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.679 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.679 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.680 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_upload_smart_interval": { 2021-04-09T18:13:00.680 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_upload_smart_interval", 2021-04-09T18:13:00.680 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.681 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.681 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.681 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "43200", 2021-04-09T18:13:00.681 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.682 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.684 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.684 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.685 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.726 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.737 INFO:teuthology.orchestra.run.smithi032.stdout: "diskprediction_user": { 2021-04-09T18:13:00.738 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_user", 2021-04-09T18:13:00.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.745 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.746 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.750 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.806 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.808 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:13:00.809 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:13:00.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.866 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.866 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.867 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2021-04-09T18:13:00.867 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:00.867 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:00.868 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:00.868 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:00.869 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:00.869 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:00.869 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:00.870 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:00.870 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:00.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:00.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:00.888 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_local", 2021-04-09T18:13:00.891 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:00.894 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:00.894 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:00.898 INFO:teuthology.orchestra.run.smithi032.stdout: "predict_interval": { 2021-04-09T18:13:00.901 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predict_interval", 2021-04-09T18:13:00.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:00.901 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:00.902 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:00.998 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2021-04-09T18:13:00.999 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.001 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.001 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.002 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.002 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.041 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.041 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.042 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:13:01.042 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:13:01.042 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.042 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.043 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.043 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2021-04-09T18:13:01.043 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.043 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.044 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.044 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.044 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.047 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.047 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.047 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.048 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:01.048 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "influx", 2021-04-09T18:13:01.049 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": false, 2021-04-09T18:13:01.049 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "influxdb python module not found", 2021-04-09T18:13:01.049 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:01.049 INFO:teuthology.orchestra.run.smithi032.stdout: "batch_size": { 2021-04-09T18:13:01.050 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "batch_size", 2021-04-09T18:13:01.050 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.050 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.050 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.051 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5000", 2021-04-09T18:13:01.061 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.061 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.061 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.061 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.062 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.062 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.062 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.063 INFO:teuthology.orchestra.run.smithi032.stdout: "database": { 2021-04-09T18:13:01.063 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "database", 2021-04-09T18:13:01.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.063 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.064 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.064 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "ceph", 2021-04-09T18:13:01.065 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.065 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.065 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.074 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.078 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.078 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.079 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.080 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": { 2021-04-09T18:13:01.080 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hostname", 2021-04-09T18:13:01.081 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.081 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.082 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.082 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:13:01.083 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.083 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.084 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.105 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.105 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.105 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.130 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.130 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:13:01.130 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:13:01.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.131 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.140 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "30", 2021-04-09T18:13:01.141 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.142 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.153 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.154 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.154 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.161 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.162 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.162 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2021-04-09T18:13:01.163 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2021-04-09T18:13:01.163 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.163 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.163 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.164 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:13:01.164 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.164 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.164 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.168 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.169 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.169 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.169 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.170 INFO:teuthology.orchestra.run.smithi032.stdout: "port": { 2021-04-09T18:13:01.171 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "port", 2021-04-09T18:13:01.171 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.171 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.172 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.172 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8086", 2021-04-09T18:13:01.172 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.172 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.173 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.173 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.173 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.173 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2021-04-09T18:13:01.174 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2021-04-09T18:13:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.175 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "false", 2021-04-09T18:13:01.176 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.177 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.177 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.177 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.215 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.216 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.217 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.218 INFO:teuthology.orchestra.run.smithi032.stdout: "threads": { 2021-04-09T18:13:01.218 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threads", 2021-04-09T18:13:01.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.229 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.230 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.230 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2021-04-09T18:13:01.230 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.231 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.231 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.232 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.232 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.233 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.233 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.250 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2021-04-09T18:13:01.250 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2021-04-09T18:13:01.250 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.251 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.253 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:13:01.253 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.254 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.254 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.254 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.254 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_ssl": { 2021-04-09T18:13:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_ssl", 2021-04-09T18:13:01.256 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.256 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.256 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.256 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "true", 2021-04-09T18:13:01.256 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.257 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.257 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.257 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.257 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:01.259 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "insights", 2021-04-09T18:13:01.259 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:01.259 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:13:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "iostat", 2021-04-09T18:13:01.261 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:01.261 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:01.263 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:13:01.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:01.264 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "localpool", 2021-04-09T18:13:01.264 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:01.264 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:01.264 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:01.264 INFO:teuthology.orchestra.run.smithi032.stdout: "failure_domain": { 2021-04-09T18:13:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "failure_domain", 2021-04-09T18:13:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "host", 2021-04-09T18:13:01.266 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.266 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.266 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.266 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "failure domain for any created local pool", 2021-04-09T18:13:01.267 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.267 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.267 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.268 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": { 2021-04-09T18:13:01.268 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_size", 2021-04-09T18:13:01.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:01.269 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.269 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:01.269 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:01.269 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.269 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.270 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.270 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default min_size for any created local pool", 2021-04-09T18:13:01.270 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.270 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.270 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: "num_rep": { 2021-04-09T18:13:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "num_rep", 2021-04-09T18:13:01.271 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3", 2021-04-09T18:13:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.273 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.274 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default replica count for any created local pool", 2021-04-09T18:13:01.274 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.275 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.275 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.276 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": { 2021-04-09T18:13:01.276 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_num", 2021-04-09T18:13:01.277 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "128", 2021-04-09T18:13:01.280 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.280 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.280 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.281 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default pg_num for any created local pool", 2021-04-09T18:13:01.281 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.281 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.282 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.282 INFO:teuthology.orchestra.run.smithi032.stdout: "prefix": { 2021-04-09T18:13:01.286 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prefix", 2021-04-09T18:13:01.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.287 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.287 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:01.288 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:01.288 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.289 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.290 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.290 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name prefix for any created local pool", 2021-04-09T18:13:01.290 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.291 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.293 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.294 INFO:teuthology.orchestra.run.smithi032.stdout: "subtree": { 2021-04-09T18:13:01.294 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "subtree", 2021-04-09T18:13:01.295 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.295 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.296 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:01.296 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "rack", 2021-04-09T18:13:01.297 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.297 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.297 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "CRUSH level for which to create a local pool", 2021-04-09T18:13:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.299 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.299 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.299 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:01.301 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator_cli", 2021-04-09T18:13:01.301 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:01.301 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:01.302 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:01.302 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator": { 2021-04-09T18:13:01.303 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2021-04-09T18:13:01.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.528 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.528 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.531 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:01.532 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.532 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.533 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.533 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.535 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:01.572 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_autoscaler", 2021-04-09T18:13:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2021-04-09T18:13:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2021-04-09T18:13:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2021-04-09T18:13:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "progress", 2021-04-09T18:13:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2021-04-09T18:13:01.631 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2021-04-09T18:13:01.632 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:01.632 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.633 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.633 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:13:01.634 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.634 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.635 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.635 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.637 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.637 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.638 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.645 INFO:teuthology.orchestra.run.smithi032.stdout: "max_completed_events": { 2021-04-09T18:13:01.646 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_completed_events", 2021-04-09T18:13:01.651 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:01.671 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.672 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:01.673 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "50", 2021-04-09T18:13:01.673 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.673 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.674 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.674 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "number of past completed events to remember", 2021-04-09T18:13:01.675 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.675 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.675 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.676 INFO:teuthology.orchestra.run.smithi032.stdout: "persist_interval": { 2021-04-09T18:13:01.676 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "persist_interval", 2021-04-09T18:13:01.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:13:01.677 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.732 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2021-04-09T18:13:01.733 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.734 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to persist completed events", 2021-04-09T18:13:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.735 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.744 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus", 2021-04-09T18:13:01.746 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:01.747 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools": { 2021-04-09T18:13:01.748 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools", 2021-04-09T18:13:01.771 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.772 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.772 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.773 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:01.773 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.781 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.781 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.782 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.782 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.783 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.783 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.784 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools_refresh_interval": { 2021-04-09T18:13:01.786 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools_refresh_interval", 2021-04-09T18:13:01.786 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:01.786 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.788 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.789 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "300", 2021-04-09T18:13:01.790 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.790 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.791 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.792 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.792 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.793 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.793 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.794 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_interval": { 2021-04-09T18:13:01.794 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_interval", 2021-04-09T18:13:01.795 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.795 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.796 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.796 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:01.796 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.805 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.806 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.810 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.810 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.811 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.813 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.816 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2021-04-09T18:13:01.816 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2021-04-09T18:13:01.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.860 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.860 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.861 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:01.861 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.862 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.862 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.862 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.863 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.863 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.863 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:01.864 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2021-04-09T18:13:01.865 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2021-04-09T18:13:01.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:01.866 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:01.945 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:01.952 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:01.953 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:01.953 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:01.953 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:01.954 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:01.954 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:01.954 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:01.955 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:01.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.003 INFO:teuthology.orchestra.run.smithi032.stdout: "stale_cache_strategy": { 2021-04-09T18:13:02.004 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stale_cache_strategy", 2021-04-09T18:13:02.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.007 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.022 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.022 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.023 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.023 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.023 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.024 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.024 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.025 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.025 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.094 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:02.095 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:02.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:02.095 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2021-04-09T18:13:02.121 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:02.121 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:02.122 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:13:02.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:02.123 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "restful", 2021-04-09T18:13:02.123 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:02.124 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:02.160 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:02.160 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2021-04-09T18:13:02.171 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2021-04-09T18:13:02.171 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.172 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.211 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.211 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.212 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.213 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.213 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.214 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.284 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2021-04-09T18:13:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2021-04-09T18:13:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.286 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.287 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.288 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2021-04-09T18:13:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2021-04-09T18:13:02.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.291 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.292 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:02.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:02.294 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rook", 2021-04-09T18:13:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:13:02.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:02.296 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "selftest", 2021-04-09T18:13:02.310 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:02.311 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:02.319 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "roption1": { 2021-04-09T18:13:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption1", 2021-04-09T18:13:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.322 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.323 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.356 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.357 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "roption2": { 2021-04-09T18:13:02.358 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption2", 2021-04-09T18:13:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.359 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.362 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "xyz", 2021-04-09T18:13:02.363 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.364 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.365 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption1": { 2021-04-09T18:13:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption1", 2021-04-09T18:13:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.366 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.367 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.368 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.369 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption2": { 2021-04-09T18:13:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption2", 2021-04-09T18:13:02.370 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:02.371 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.372 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.373 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.374 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.375 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.376 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption3": { 2021-04-09T18:13:02.377 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption3", 2021-04-09T18:13:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2021-04-09T18:13:02.395 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.396 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.401 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.402 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.403 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.404 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption4": { 2021-04-09T18:13:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption4", 2021-04-09T18:13:02.427 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.428 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.448 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.449 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.449 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.449 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.450 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.450 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.451 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.451 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.451 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.452 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption5": { 2021-04-09T18:13:02.452 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption5", 2021-04-09T18:13:02.453 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:02.453 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.454 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.454 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.521 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.522 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.528 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.528 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.528 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.538 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption6": { 2021-04-09T18:13:02.538 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption6", 2021-04-09T18:13:02.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:02.539 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.544 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.545 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:13:02.545 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.545 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.545 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.546 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.546 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.546 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.546 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.547 INFO:teuthology.orchestra.run.smithi032.stdout: "testkey": { 2021-04-09T18:13:02.547 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testkey", 2021-04-09T18:13:02.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.547 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.549 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.549 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.549 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.549 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.550 INFO:teuthology.orchestra.run.smithi032.stdout: "testlkey": { 2021-04-09T18:13:02.550 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testlkey", 2021-04-09T18:13:02.550 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.660 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "testnewline": { 2021-04-09T18:13:02.662 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testnewline", 2021-04-09T18:13:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.663 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.695 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.700 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.701 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.710 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:02.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:02.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:02.721 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh", 2021-04-09T18:13:02.721 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:02.722 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:02.722 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:02.722 INFO:teuthology.orchestra.run.smithi032.stdout: "inventory_cache_timeout_min": { 2021-04-09T18:13:02.723 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "inventory_cache_timeout_min", 2021-04-09T18:13:02.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.724 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.724 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.724 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.733 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.734 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.734 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.734 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.736 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.738 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.739 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.741 INFO:teuthology.orchestra.run.smithi032.stdout: "ssh_config_file": { 2021-04-09T18:13:02.744 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh_config_file", 2021-04-09T18:13:02.745 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.745 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.745 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.746 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:02.749 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.750 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.750 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.753 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.753 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.754 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.754 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:02.755 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:02.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:02.765 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "status", 2021-04-09T18:13:02.765 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:02.766 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:02.766 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:13:02.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:02.857 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telegraf", 2021-04-09T18:13:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:02.865 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "address": { 2021-04-09T18:13:02.866 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "address", 2021-04-09T18:13:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:02.867 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2021-04-09T18:13:02.868 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.869 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.870 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.870 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.871 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.925 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:13:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:13:02.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:13:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:02.933 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15", 2021-04-09T18:13:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:02.935 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:03.056 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:03.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:03.057 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telemetry", 2021-04-09T18:13:03.058 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:03.061 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:03.061 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:03.062 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_basic": { 2021-04-09T18:13:03.062 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_basic", 2021-04-09T18:13:03.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:03.063 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.077 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.078 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:13:03.078 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.078 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.079 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.082 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share basic cluster information (size, version)", 2021-04-09T18:13:03.082 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.082 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.083 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.084 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_crash": { 2021-04-09T18:13:03.084 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_crash", 2021-04-09T18:13:03.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:03.095 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.096 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.096 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:13:03.096 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.097 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.177 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.177 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.178 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.178 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.194 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.195 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_device": { 2021-04-09T18:13:03.195 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_device", 2021-04-09T18:13:03.195 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:03.195 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.196 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.196 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2021-04-09T18:13:03.196 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.196 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.197 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.197 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.197 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.197 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.197 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.198 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_ident": { 2021-04-09T18:13:03.198 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_ident", 2021-04-09T18:13:03.198 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:03.199 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.199 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.199 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:13:03.199 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.200 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.200 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.200 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.200 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.201 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.201 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.201 INFO:teuthology.orchestra.run.smithi032.stdout: "contact": { 2021-04-09T18:13:03.234 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "contact", 2021-04-09T18:13:03.234 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:03.234 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.240 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.241 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:13:03.241 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.241 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.241 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.242 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.242 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.242 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.244 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.245 INFO:teuthology.orchestra.run.smithi032.stdout: "description": { 2021-04-09T18:13:03.245 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "description", 2021-04-09T18:13:03.245 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:13:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.246 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.247 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.250 INFO:teuthology.orchestra.run.smithi032.stdout: "device_url": { 2021-04-09T18:13:03.251 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_url", 2021-04-09T18:13:03.294 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:03.331 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.332 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/device", 2021-04-09T18:13:03.333 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.334 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.335 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.336 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2021-04-09T18:13:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2021-04-09T18:13:03.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.338 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.339 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:13:03.382 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.382 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.382 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.383 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.383 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.384 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.384 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.385 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:13:03.385 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:13:03.405 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:03.406 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.406 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.406 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "24", 2021-04-09T18:13:03.407 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "8", 2021-04-09T18:13:03.419 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.421 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.421 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.422 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.422 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.422 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.431 INFO:teuthology.orchestra.run.smithi032.stdout: "last_opt_revision": { 2021-04-09T18:13:03.432 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "last_opt_revision", 2021-04-09T18:13:03.432 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:03.433 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.467 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.467 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1", 2021-04-09T18:13:03.468 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.468 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.469 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.469 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.469 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.470 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.470 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.479 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard": { 2021-04-09T18:13:03.480 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard", 2021-04-09T18:13:03.480 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2021-04-09T18:13:03.480 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.481 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.481 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2021-04-09T18:13:03.519 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.520 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.520 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.520 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.521 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.522 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.522 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.523 INFO:teuthology.orchestra.run.smithi032.stdout: "organization": { 2021-04-09T18:13:03.523 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "organization", 2021-04-09T18:13:03.527 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:03.590 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.590 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.590 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:13:03.590 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.591 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.591 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.591 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.591 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.605 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.607 INFO:teuthology.orchestra.run.smithi032.stdout: "proxy": { 2021-04-09T18:13:03.610 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "proxy", 2021-04-09T18:13:03.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:03.614 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.615 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.615 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:13:03.615 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.615 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.616 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.616 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.616 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.616 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.616 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.617 INFO:teuthology.orchestra.run.smithi032.stdout: "url": { 2021-04-09T18:13:03.617 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url", 2021-04-09T18:13:03.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:03.632 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.633 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.633 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/report", 2021-04-09T18:13:03.633 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.633 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.633 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.634 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.634 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.634 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.634 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:03.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:03.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:03.635 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "test_orchestrator", 2021-04-09T18:13:03.636 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:03.666 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:03.666 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": {} 2021-04-09T18:13:03.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:03.671 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2021-04-09T18:13:03.671 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:03.671 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:03.677 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:03.679 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_clones": { 2021-04-09T18:13:03.680 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_clones", 2021-04-09T18:13:03.680 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:03.681 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.681 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.682 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4", 2021-04-09T18:13:03.682 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.693 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.701 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.701 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Number of asynchronous cloner threads", 2021-04-09T18:13:03.702 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.702 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.703 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:03.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:03.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:03.717 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix", 2021-04-09T18:13:03.728 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2021-04-09T18:13:03.728 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2021-04-09T18:13:03.734 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2021-04-09T18:13:03.734 INFO:teuthology.orchestra.run.smithi032.stdout: "identifier": { 2021-04-09T18:13:03.735 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "identifier", 2021-04-09T18:13:03.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:03.764 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.764 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.769 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2021-04-09T18:13:03.771 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.777 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.778 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.784 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.784 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.784 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.785 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.794 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2021-04-09T18:13:03.795 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2021-04-09T18:13:03.795 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2021-04-09T18:13:03.798 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.806 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.806 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2021-04-09T18:13:03.815 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.816 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.816 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.817 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.823 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.824 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.824 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.841 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_host": { 2021-04-09T18:13:03.842 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_host", 2021-04-09T18:13:03.842 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:03.843 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.843 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.843 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "None", 2021-04-09T18:13:03.844 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.844 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.845 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.845 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.845 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.867 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.868 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.879 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_port": { 2021-04-09T18:13:03.879 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_port", 2021-04-09T18:13:03.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2021-04-09T18:13:03.891 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.892 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.892 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10051", 2021-04-09T18:13:03.893 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.893 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.893 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.910 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.911 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:03.916 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_sender": { 2021-04-09T18:13:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_sender", 2021-04-09T18:13:03.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2021-04-09T18:13:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2021-04-09T18:13:03.924 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2021-04-09T18:13:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/usr/bin/zabbix_sender", 2021-04-09T18:13:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2021-04-09T18:13:03.937 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2021-04-09T18:13:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2021-04-09T18:13:03.938 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2021-04-09T18:13:03.955 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2021-04-09T18:13:03.956 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2021-04-09T18:13:03.957 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2021-04-09T18:13:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:03.958 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:03.959 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:13:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "services": {}, 2021-04-09T18:13:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "always_on_modules": { 2021-04-09T18:13:03.960 INFO:teuthology.orchestra.run.smithi032.stdout: "nautilus": [ 2021-04-09T18:13:03.961 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2021-04-09T18:13:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2021-04-09T18:13:03.993 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2021-04-09T18:13:04.028 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator_cli", 2021-04-09T18:13:04.028 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2021-04-09T18:13:04.029 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2021-04-09T18:13:04.029 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2021-04-09T18:13:04.030 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2021-04-09T18:13:04.030 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:04.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:04.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:04.031 INFO:teuthology.orchestra.run.smithi032.stdout: "servicemap": { 2021-04-09T18:13:04.031 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 1, 2021-04-09T18:13:04.032 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "0.000000", 2021-04-09T18:13:04.049 INFO:teuthology.orchestra.run.smithi032.stdout: "services": {} 2021-04-09T18:13:04.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:04.050 INFO:teuthology.orchestra.run.smithi032.stdout: "progress_events": {} 2021-04-09T18:13:04.050 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:13:04.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:13:04.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7df0018930 msgr2=0x7f7df001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7df0018930 0x7f7df001ae10 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10184d80 msgr2=0x7f7e10191250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10184d80 0x7f7e10191250 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f7e0800a940 tx=0x7f7e0800ace0).stop 2021-04-09T18:13:04.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 shutdown_connections 2021-04-09T18:13:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7df0018930 0x7f7df001ae10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e10184d80 0x7f7e10191250 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e1018e280 0x7f7e10192370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 --2- 172.21.15.32:0/3630373061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e100b4250 0x7f7e1009a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.273 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 >> 172.21.15.32:0/3630373061 conn(0x7f7e10090a90 msgr2=0x7f7e101832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:04.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.274 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 shutdown_connections 2021-04-09T18:13:04.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:12:50.274 7f7e15ef2700 1 -- 172.21.15.32:0/3630373061 wait complete. 2021-04-09T18:13:04.186 INFO:tasks.ceph.ceph_manager.ceph:{'fsid': 'cf69fbbc-37e8-4833-99b4-5ecbe4444cfb', 'health': {'checks': {}, 'status': 'HEALTH_OK'}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 36, 'monmap': {'epoch': 1, 'min_mon_release_name': '14', 'num_mons': 3}, 'osdmap': {'osdmap': {'epoch': 20, 'num_osds': 8, 'num_up_osds': 8, 'num_in_osds': 8, 'num_remapped_pgs': 0}}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 24}], 'num_pgs': 24, 'num_pools': 3, 'num_objects': 21, 'data_bytes': 2286, 'bytes_used': 8591704064, 'bytes_avail': 764502409216, 'bytes_total': 773094113280, 'write_bytes_sec': 767, 'read_op_per_sec': 0, 'write_op_per_sec': 2}, 'fsmap': {'epoch': 6, 'id': 1, 'up': 1, 'in': 1, 'max': 1, 'by_rank': [{'filesystem_id': 1, 'rank': 0, 'name': 'a', 'status': 'up:active', 'gid': 4314}], 'up:standby': 3}, 'mgrmap': {'epoch': 3, 'active_gid': 4102, 'active_name': 'x', 'active_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6800', 'nonce': 23869}, {'type': 'v1', 'addr': '172.21.15.86:6801', 'nonce': 23869}]}, 'active_addr': '172.21.15.86:6801/23869', 'active_change': '2021-04-09 18:12:14.249236', 'available': True, 'standbys': [{'gid': 4110, 'name': 'y', 'available_modules': [{'name': 'alerts', 'can_run': True, 'error_string': '', 'module_options': {'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'How frequently to reexamine health status', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_destination': {'name': 'smtp_destination', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email address to send alerts to', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_from_name': {'name': 'smtp_from_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'Ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email From: name', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_host': {'name': 'smtp_host', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_password': {'name': 'smtp_password', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Password to authenticate with', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_port': {'name': 'smtp_port', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '465', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP port', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_sender': {'name': 'smtp_sender', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP envelope sender', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_ssl': {'name': 'smtp_ssl', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Use SSL to connect to SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_user': {'name': 'smtp_user', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'User to authenticate as', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ansible', 'can_run': True, 'error_string': '', 'module_options': {'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_url': {'name': 'server_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_server': {'name': 'verify_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'balancer', 'can_run': True, 'error_string': '', 'module_options': {'active': {'name': 'active', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically balance PGs across cluster', 'long_desc': '', 'tags': [], 'see_also': []}, 'begin_time': {'name': 'begin_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '0000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'beginning time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'begin_weekday': {'name': 'begin_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to this day of the week or later', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'crush_compat_max_iterations': {'name': 'crush_compat_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '25', 'min': '1', 'max': '250', 'enum_allowed': [], 'desc': 'maximum number of iterations to attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'crush_compat_metrics': {'name': 'crush_compat_metrics', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'pgs,objects,bytes', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'metrics with which to calculate OSD utilization', 'long_desc': 'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', 'tags': [], 'see_also': []}, 'crush_compat_step': {'name': 'crush_compat_step', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0.5', 'min': '0.001', 'max': '0.999', 'enum_allowed': [], 'desc': 'aggressiveness of optimization', 'long_desc': '.99 is very aggressive, .01 is less aggressive', 'tags': [], 'see_also': []}, 'end_time': {'name': 'end_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '2400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'ending time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'end_weekday': {'name': 'end_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '7', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to days of the week earlier than this', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'min_score': {'name': 'min_score', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'minimum score, below which no optimization is attempted', 'long_desc': '', 'tags': [], 'see_also': []}, 'mode': {'name': 'mode', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'none', 'min': '', 'max': '', 'enum_allowed': ['crush-compat', 'none', 'upmap'], 'desc': 'Balancer mode', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_ids': {'name': 'pool_ids', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'pools which the automatic balancing will be limited to', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'upmap_max_deviation': {'name': 'upmap_max_deviation', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '1', 'max': '', 'enum_allowed': [], 'desc': 'deviation below which no optimization is attempted', 'long_desc': 'If the number of PGs are within this count then no optimization is attempted', 'tags': [], 'see_also': []}, 'upmap_max_iterations': {'name': 'upmap_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '10', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'maximum upmap optimization iterations', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'crash', 'can_run': True, 'error_string': '', 'module_options': {'retain_interval': {'name': 'retain_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '31536000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain crashes before pruning them', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_recent_interval': {'name': 'warn_recent_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '1209600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'time interval in which to warn about recent crashes', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'dashboard', 'can_run': True, 'error_string': '', 'module_options': {'ALERTMANAGER_API_HOST': {'name': 'ALERTMANAGER_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_ENABLED': {'name': 'AUDIT_API_ENABLED', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_LOG_PAYLOAD': {'name': 'AUDIT_API_LOG_PAYLOAD', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ENABLE_BROWSABLE_API': {'name': 'ENABLE_BROWSABLE_API', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_cephfs': {'name': 'FEATURE_TOGGLE_cephfs', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_iscsi': {'name': 'FEATURE_TOGGLE_iscsi', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_mirroring': {'name': 'FEATURE_TOGGLE_mirroring', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rbd': {'name': 'FEATURE_TOGGLE_rbd', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rgw': {'name': 'FEATURE_TOGGLE_rgw', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {'name': 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_PASSWORD': {'name': 'GRAFANA_API_PASSWORD', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_SSL_VERIFY': {'name': 'GRAFANA_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_URL': {'name': 'GRAFANA_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_USERNAME': {'name': 'GRAFANA_API_USERNAME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_FRONTEND_API_URL': {'name': 'GRAFANA_FRONTEND_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_UPDATE_DASHBOARDS': {'name': 'GRAFANA_UPDATE_DASHBOARDS', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ISCSI_API_SSL_VERIFICATION': {'name': 'ISCSI_API_SSL_VERIFICATION', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'PROMETHEUS_API_HOST': {'name': 'PROMETHEUS_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'REST_REQUESTS_TIMEOUT': {'name': 'REST_REQUESTS_TIMEOUT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '45', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ACCESS_KEY': {'name': 'RGW_API_ACCESS_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ADMIN_RESOURCE': {'name': 'RGW_API_ADMIN_RESOURCE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_HOST': {'name': 'RGW_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_PORT': {'name': 'RGW_API_PORT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '80', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SCHEME': {'name': 'RGW_API_SCHEME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'http', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SECRET_KEY': {'name': 'RGW_API_SECRET_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SSL_VERIFY': {'name': 'RGW_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_USER_ID': {'name': 'RGW_API_USER_ID', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'crt_file': {'name': 'crt_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'debug': {'name': 'debug', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Enable/disable debug options', 'long_desc': '', 'tags': [], 'see_also': []}, 'jwt_token_ttl': {'name': 'jwt_token_ttl', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '28800', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '::', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8080', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl_server_port': {'name': 'ssl_server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8443', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_behaviour': {'name': 'standby_behaviour', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'redirect', 'min': '', 'max': '', 'enum_allowed': ['error', 'redirect'], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_error_status_code': {'name': 'standby_error_status_code', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '500', 'min': '400', 'max': '599', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url_prefix': {'name': 'url_prefix', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'deepsea', 'can_run': True, 'error_string': '', 'module_options': {'salt_api_eauth': {'name': 'salt_api_eauth', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'sharedsecret', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_password': {'name': 'salt_api_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_url': {'name': 'salt_api_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_username': {'name': 'salt_api_username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'devicehealth', 'can_run': True, 'error_string': '', 'module_options': {'enable_monitoring': {'name': 'enable_monitoring', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'monitor device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'mark_out_threshold': {'name': 'mark_out_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '2419200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically mark OSD if it may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_name': {'name': 'pool_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'device_health_metrics', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name of pool in which to store device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'retention_period': {'name': 'retention_period', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '15552000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_frequency': {'name': 'scrape_frequency', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to scrape device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'self_heal': {'name': 'self_heal', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'preemptively heal cluster around devices that may fail', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and check device health', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_threshold': {'name': 'warn_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '7257600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'raise health warning if OSD may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_cloud', 'can_run': True, 'error_string': '', 'module_options': {'diskprediction_cert_context': {'name': 'diskprediction_cert_context', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_default_authority': {'name': 'diskprediction_default_authority', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_password': {'name': 'diskprediction_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_port': {'name': 'diskprediction_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '31400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_retrieve_prediction_interval': {'name': 'diskprediction_retrieve_prediction_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_server': {'name': 'diskprediction_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_ssl_target_name_override': {'name': 'diskprediction_ssl_target_name_override', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_metrics_interval': {'name': 'diskprediction_upload_metrics_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_smart_interval': {'name': 'diskprediction_upload_smart_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_user': {'name': 'diskprediction_user', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_local', 'can_run': True, 'error_string': '', 'module_options': {'predict_interval': {'name': 'predict_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'influx', 'can_run': False, 'error_string': 'influxdb python module not found', 'module_options': {'batch_size': {'name': 'batch_size', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'database': {'name': 'database', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'hostname': {'name': 'hostname', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '30', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'port': {'name': 'port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '8086', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'false', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'threads': {'name': 'threads', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_ssl': {'name': 'verify_ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'true', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'insights', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'iostat', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'localpool', 'can_run': True, 'error_string': '', 'module_options': {'failure_domain': {'name': 'failure_domain', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'host', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'failure domain for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'min_size': {'name': 'min_size', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default min_size for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'num_rep': {'name': 'num_rep', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '3', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default replica count for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'pg_num': {'name': 'pg_num', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '128', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default pg_num for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'prefix': {'name': 'prefix', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name prefix for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'subtree': {'name': 'subtree', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'rack', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'CRUSH level for which to create a local pool', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'orchestrator_cli', 'can_run': True, 'error_string': '', 'module_options': {'orchestrator': {'name': 'orchestrator', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'pg_autoscaler', 'can_run': True, 'error_string': '', 'module_options': {'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'progress', 'can_run': True, 'error_string': '', 'module_options': {'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'max_completed_events': {'name': 'max_completed_events', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '50', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'number of past completed events to remember', 'long_desc': '', 'tags': [], 'see_also': []}, 'persist_interval': {'name': 'persist_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to persist completed events', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'prometheus', 'can_run': True, 'error_string': '', 'module_options': {'rbd_stats_pools': {'name': 'rbd_stats_pools', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rbd_stats_pools_refresh_interval': {'name': 'rbd_stats_pools_refresh_interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '300', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_interval': {'name': 'scrape_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'stale_cache_strategy': {'name': 'stale_cache_strategy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rbd_support', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'restful', 'can_run': True, 'error_string': '', 'module_options': {'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rook', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'selftest', 'can_run': True, 'error_string': '', 'module_options': {'roption1': {'name': 'roption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'roption2': {'name': 'roption2', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'xyz', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption1': {'name': 'rwoption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption2': {'name': 'rwoption2', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption3': {'name': 'rwoption3', 'type': 'float', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption4': {'name': 'rwoption4', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption5': {'name': 'rwoption5', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption6': {'name': 'rwoption6', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testkey': {'name': 'testkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testlkey': {'name': 'testlkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testnewline': {'name': 'testnewline', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ssh', 'can_run': True, 'error_string': '', 'module_options': {'inventory_cache_timeout_min': {'name': 'inventory_cache_timeout_min', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssh_config_file': {'name': 'ssh_config_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'status', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'telegraf', 'can_run': True, 'error_string': '', 'module_options': {'address': {'name': 'address', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'unixgram:///tmp/telegraf.sock', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '15', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'telemetry', 'can_run': True, 'error_string': '', 'module_options': {'channel_basic': {'name': 'channel_basic', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Share basic cluster information (size, version)', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_crash': {'name': 'channel_crash', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_device': {'name': 'channel_device', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_ident': {'name': 'channel_ident', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'contact': {'name': 'contact', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'description': {'name': 'description', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'device_url': {'name': 'device_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/device', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '24', 'min': '8', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'last_opt_revision': {'name': 'last_opt_revision', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '1', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'leaderboard': {'name': 'leaderboard', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'organization': {'name': 'organization', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'proxy': {'name': 'proxy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url': {'name': 'url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/report', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'test_orchestrator', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'volumes', 'can_run': True, 'error_string': '', 'module_options': {'max_concurrent_clones': {'name': 'max_concurrent_clones', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '4', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Number of asynchronous cloner threads', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'zabbix', 'can_run': True, 'error_string': '', 'module_options': {'identifier': {'name': 'identifier', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_host': {'name': 'zabbix_host', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_port': {'name': 'zabbix_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '10051', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_sender': {'name': 'zabbix_sender', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '/usr/bin/zabbix_sender', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}]}], 'modules': ['iostat', 'restful'], 'available_modules': [{'name': 'alerts', 'can_run': True, 'error_string': '', 'module_options': {'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'How frequently to reexamine health status', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_destination': {'name': 'smtp_destination', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email address to send alerts to', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_from_name': {'name': 'smtp_from_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'Ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email From: name', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_host': {'name': 'smtp_host', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_password': {'name': 'smtp_password', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Password to authenticate with', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_port': {'name': 'smtp_port', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '465', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP port', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_sender': {'name': 'smtp_sender', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP envelope sender', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_ssl': {'name': 'smtp_ssl', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Use SSL to connect to SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_user': {'name': 'smtp_user', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'User to authenticate as', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ansible', 'can_run': True, 'error_string': '', 'module_options': {'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_url': {'name': 'server_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_server': {'name': 'verify_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'balancer', 'can_run': True, 'error_string': '', 'module_options': {'active': {'name': 'active', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically balance PGs across cluster', 'long_desc': '', 'tags': [], 'see_also': []}, 'begin_time': {'name': 'begin_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '0000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'beginning time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'begin_weekday': {'name': 'begin_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to this day of the week or later', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'crush_compat_max_iterations': {'name': 'crush_compat_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '25', 'min': '1', 'max': '250', 'enum_allowed': [], 'desc': 'maximum number of iterations to attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'crush_compat_metrics': {'name': 'crush_compat_metrics', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'pgs,objects,bytes', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'metrics with which to calculate OSD utilization', 'long_desc': 'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', 'tags': [], 'see_also': []}, 'crush_compat_step': {'name': 'crush_compat_step', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0.5', 'min': '0.001', 'max': '0.999', 'enum_allowed': [], 'desc': 'aggressiveness of optimization', 'long_desc': '.99 is very aggressive, .01 is less aggressive', 'tags': [], 'see_also': []}, 'end_time': {'name': 'end_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '2400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'ending time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'end_weekday': {'name': 'end_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '7', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to days of the week earlier than this', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'min_score': {'name': 'min_score', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'minimum score, below which no optimization is attempted', 'long_desc': '', 'tags': [], 'see_also': []}, 'mode': {'name': 'mode', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'none', 'min': '', 'max': '', 'enum_allowed': ['crush-compat', 'none', 'upmap'], 'desc': 'Balancer mode', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_ids': {'name': 'pool_ids', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'pools which the automatic balancing will be limited to', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'upmap_max_deviation': {'name': 'upmap_max_deviation', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '1', 'max': '', 'enum_allowed': [], 'desc': 'deviation below which no optimization is attempted', 'long_desc': 'If the number of PGs are within this count then no optimization is attempted', 'tags': [], 'see_also': []}, 'upmap_max_iterations': {'name': 'upmap_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '10', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'maximum upmap optimization iterations', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'crash', 'can_run': True, 'error_string': '', 'module_options': {'retain_interval': {'name': 'retain_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '31536000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain crashes before pruning them', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_recent_interval': {'name': 'warn_recent_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '1209600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'time interval in which to warn about recent crashes', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'dashboard', 'can_run': True, 'error_string': '', 'module_options': {'ALERTMANAGER_API_HOST': {'name': 'ALERTMANAGER_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_ENABLED': {'name': 'AUDIT_API_ENABLED', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_LOG_PAYLOAD': {'name': 'AUDIT_API_LOG_PAYLOAD', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ENABLE_BROWSABLE_API': {'name': 'ENABLE_BROWSABLE_API', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_cephfs': {'name': 'FEATURE_TOGGLE_cephfs', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_iscsi': {'name': 'FEATURE_TOGGLE_iscsi', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_mirroring': {'name': 'FEATURE_TOGGLE_mirroring', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rbd': {'name': 'FEATURE_TOGGLE_rbd', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rgw': {'name': 'FEATURE_TOGGLE_rgw', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {'name': 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_PASSWORD': {'name': 'GRAFANA_API_PASSWORD', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_SSL_VERIFY': {'name': 'GRAFANA_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_URL': {'name': 'GRAFANA_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_USERNAME': {'name': 'GRAFANA_API_USERNAME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_FRONTEND_API_URL': {'name': 'GRAFANA_FRONTEND_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_UPDATE_DASHBOARDS': {'name': 'GRAFANA_UPDATE_DASHBOARDS', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ISCSI_API_SSL_VERIFICATION': {'name': 'ISCSI_API_SSL_VERIFICATION', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'PROMETHEUS_API_HOST': {'name': 'PROMETHEUS_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'REST_REQUESTS_TIMEOUT': {'name': 'REST_REQUESTS_TIMEOUT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '45', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ACCESS_KEY': {'name': 'RGW_API_ACCESS_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ADMIN_RESOURCE': {'name': 'RGW_API_ADMIN_RESOURCE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_HOST': {'name': 'RGW_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_PORT': {'name': 'RGW_API_PORT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '80', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SCHEME': {'name': 'RGW_API_SCHEME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'http', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SECRET_KEY': {'name': 'RGW_API_SECRET_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SSL_VERIFY': {'name': 'RGW_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_USER_ID': {'name': 'RGW_API_USER_ID', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'crt_file': {'name': 'crt_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'debug': {'name': 'debug', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Enable/disable debug options', 'long_desc': '', 'tags': [], 'see_also': []}, 'jwt_token_ttl': {'name': 'jwt_token_ttl', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '28800', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '::', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8080', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl_server_port': {'name': 'ssl_server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8443', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_behaviour': {'name': 'standby_behaviour', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'redirect', 'min': '', 'max': '', 'enum_allowed': ['error', 'redirect'], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_error_status_code': {'name': 'standby_error_status_code', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '500', 'min': '400', 'max': '599', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url_prefix': {'name': 'url_prefix', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'deepsea', 'can_run': True, 'error_string': '', 'module_options': {'salt_api_eauth': {'name': 'salt_api_eauth', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'sharedsecret', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_password': {'name': 'salt_api_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_url': {'name': 'salt_api_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_username': {'name': 'salt_api_username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'devicehealth', 'can_run': True, 'error_string': '', 'module_options': {'enable_monitoring': {'name': 'enable_monitoring', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'monitor device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'mark_out_threshold': {'name': 'mark_out_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '2419200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically mark OSD if it may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_name': {'name': 'pool_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'device_health_metrics', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name of pool in which to store device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'retention_period': {'name': 'retention_period', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '15552000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_frequency': {'name': 'scrape_frequency', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to scrape device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'self_heal': {'name': 'self_heal', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'preemptively heal cluster around devices that may fail', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and check device health', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_threshold': {'name': 'warn_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '7257600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'raise health warning if OSD may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_cloud', 'can_run': True, 'error_string': '', 'module_options': {'diskprediction_cert_context': {'name': 'diskprediction_cert_context', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_default_authority': {'name': 'diskprediction_default_authority', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_password': {'name': 'diskprediction_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_port': {'name': 'diskprediction_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '31400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_retrieve_prediction_interval': {'name': 'diskprediction_retrieve_prediction_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_server': {'name': 'diskprediction_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_ssl_target_name_override': {'name': 'diskprediction_ssl_target_name_override', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_metrics_interval': {'name': 'diskprediction_upload_metrics_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_smart_interval': {'name': 'diskprediction_upload_smart_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_user': {'name': 'diskprediction_user', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_local', 'can_run': True, 'error_string': '', 'module_options': {'predict_interval': {'name': 'predict_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'influx', 'can_run': False, 'error_string': 'influxdb python module not found', 'module_options': {'batch_size': {'name': 'batch_size', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'database': {'name': 'database', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'hostname': {'name': 'hostname', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '30', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'port': {'name': 'port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '8086', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'false', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'threads': {'name': 'threads', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_ssl': {'name': 'verify_ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'true', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'insights', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'iostat', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'localpool', 'can_run': True, 'error_string': '', 'module_options': {'failure_domain': {'name': 'failure_domain', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'host', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'failure domain for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'min_size': {'name': 'min_size', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default min_size for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'num_rep': {'name': 'num_rep', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '3', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default replica count for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'pg_num': {'name': 'pg_num', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '128', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default pg_num for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'prefix': {'name': 'prefix', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name prefix for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'subtree': {'name': 'subtree', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'rack', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'CRUSH level for which to create a local pool', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'orchestrator_cli', 'can_run': True, 'error_string': '', 'module_options': {'orchestrator': {'name': 'orchestrator', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'pg_autoscaler', 'can_run': True, 'error_string': '', 'module_options': {'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'progress', 'can_run': True, 'error_string': '', 'module_options': {'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'max_completed_events': {'name': 'max_completed_events', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '50', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'number of past completed events to remember', 'long_desc': '', 'tags': [], 'see_also': []}, 'persist_interval': {'name': 'persist_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to persist completed events', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'prometheus', 'can_run': True, 'error_string': '', 'module_options': {'rbd_stats_pools': {'name': 'rbd_stats_pools', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rbd_stats_pools_refresh_interval': {'name': 'rbd_stats_pools_refresh_interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '300', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_interval': {'name': 'scrape_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'stale_cache_strategy': {'name': 'stale_cache_strategy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rbd_support', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'restful', 'can_run': True, 'error_string': '', 'module_options': {'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rook', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'selftest', 'can_run': True, 'error_string': '', 'module_options': {'roption1': {'name': 'roption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'roption2': {'name': 'roption2', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'xyz', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption1': {'name': 'rwoption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption2': {'name': 'rwoption2', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption3': {'name': 'rwoption3', 'type': 'float', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption4': {'name': 'rwoption4', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption5': {'name': 'rwoption5', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption6': {'name': 'rwoption6', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testkey': {'name': 'testkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testlkey': {'name': 'testlkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testnewline': {'name': 'testnewline', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ssh', 'can_run': True, 'error_string': '', 'module_options': {'inventory_cache_timeout_min': {'name': 'inventory_cache_timeout_min', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssh_config_file': {'name': 'ssh_config_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'status', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'telegraf', 'can_run': True, 'error_string': '', 'module_options': {'address': {'name': 'address', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'unixgram:///tmp/telegraf.sock', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '15', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'telemetry', 'can_run': True, 'error_string': '', 'module_options': {'channel_basic': {'name': 'channel_basic', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Share basic cluster information (size, version)', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_crash': {'name': 'channel_crash', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_device': {'name': 'channel_device', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_ident': {'name': 'channel_ident', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'contact': {'name': 'contact', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'description': {'name': 'description', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'device_url': {'name': 'device_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/device', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '24', 'min': '8', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'last_opt_revision': {'name': 'last_opt_revision', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '1', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'leaderboard': {'name': 'leaderboard', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'organization': {'name': 'organization', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'proxy': {'name': 'proxy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url': {'name': 'url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/report', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'test_orchestrator', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'volumes', 'can_run': True, 'error_string': '', 'module_options': {'max_concurrent_clones': {'name': 'max_concurrent_clones', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '4', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Number of asynchronous cloner threads', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'zabbix', 'can_run': True, 'error_string': '', 'module_options': {'identifier': {'name': 'identifier', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_host': {'name': 'zabbix_host', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_port': {'name': 'zabbix_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '10051', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_sender': {'name': 'zabbix_sender', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '/usr/bin/zabbix_sender', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}], 'services': {}, 'always_on_modules': {'nautilus': ['balancer', 'crash', 'devicehealth', 'orchestrator_cli', 'progress', 'rbd_support', 'status', 'volumes']}}, 'servicemap': {'epoch': 1, 'modified': '0.000000', 'services': {}}, 'progress_events': {}} 2021-04-09T18:13:04.225 INFO:tasks.ceph.ceph_manager.ceph:clean! 2021-04-09T18:13:04.230 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-04-09T18:13:04.231 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph health 2021-04-09T18:13:04.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:13:04.417 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.351 7f4e0ab56700 1 Processor -- start 2021-04-09T18:13:04.417 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.351 7f4e0ab56700 1 -- start start 2021-04-09T18:13:04.418 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.352 7f4e0ab56700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4e040cffa0 0x7f4e040d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.418 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.352 7f4e0ab56700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4e040d7800 0x7f4e040d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.419 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.352 7f4e0ab56700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040d7280 con 0x7f4e040d8220 2021-04-09T18:13:04.430 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.352 7f4e0ab56700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e040d91e0 con 0x7f4e040cffa0 2021-04-09T18:13:04.431 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.352 7f4e0ab56700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0410dfe0 con 0x7f4e040d7800 2021-04-09T18:13:04.434 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.352 7f4e08972700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4e040cffa0 0x7f4e040d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:04.435 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.352 7f4e08972700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4e040cffa0 0x7f4e040d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41952/0 (socket says 172.21.15.32:41952) 2021-04-09T18:13:04.435 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.352 7f4e08972700 1 -- 172.21.15.32:0/183365579 learned_addr learned my addr 172.21.15.32:0/183365579 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:04.436 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.352 7f4e03fff700 1 --2- 172.21.15.32:0/183365579 >> v2:172.21.15.86:3300/0 conn(0x7f4e040d7800 0x7f4e040d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:04.436 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e08972700 1 -- 172.21.15.32:0/183365579 >> v2:172.21.15.86:3300/0 conn(0x7f4e040d7800 msgr2=0x7f4e040d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:04.437 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e08972700 1 --2- 172.21.15.32:0/183365579 >> v2:172.21.15.86:3300/0 conn(0x7f4e040d7800 0x7f4e040d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.437 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e08972700 1 -- 172.21.15.32:0/183365579 >> v1:172.21.15.86:6789/0 conn(0x7f4e040d8220 legacy=0x7f4e040d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.437 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e08972700 1 -- 172.21.15.32:0/183365579 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0410e200 con 0x7f4e040cffa0 2021-04-09T18:13:04.438 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e08972700 1 --2- 172.21.15.32:0/183365579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040cffa0 0x7f4e040d2440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f4dec006700 tx=0x7f4dec009b80).ready entity=mon.0 client_cookie=d7617c6ea4de3281 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:04.438 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e037fe700 1 -- 172.21.15.32:0/183365579 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dec00c0c0 con 0x7f4e040cffa0 2021-04-09T18:13:04.439 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e037fe700 1 -- 172.21.15.32:0/183365579 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dec00c440 con 0x7f4e040cffa0 2021-04-09T18:13:04.439 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e037fe700 1 -- 172.21.15.32:0/183365579 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dec00c920 con 0x7f4e040cffa0 2021-04-09T18:13:04.439 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e0ab56700 1 -- 172.21.15.32:0/183365579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040cffa0 msgr2=0x7f4e040d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.473 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.353 7f4e0ab56700 1 --2- 172.21.15.32:0/183365579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040cffa0 0x7f4e040d2440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f4dec006700 tx=0x7f4dec009b80).stop 2021-04-09T18:13:04.474 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.354 7f4e0ab56700 1 -- 172.21.15.32:0/183365579 shutdown_connections 2021-04-09T18:13:04.477 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.354 7f4e0ab56700 1 --2- 172.21.15.32:0/183365579 >> v2:172.21.15.86:3300/0 conn(0x7f4e040d7800 0x7f4e040d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.479 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.354 7f4e0ab56700 1 --2- 172.21.15.32:0/183365579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e040cffa0 0x7f4e040d2440 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.479 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.354 7f4e0ab56700 1 -- 172.21.15.32:0/183365579 >> 172.21.15.32:0/183365579 conn(0x7f4e040ce5c0 msgr2=0x7f4e040cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:04.487 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.354 7f4e0ab56700 1 -- 172.21.15.32:0/183365579 shutdown_connections 2021-04-09T18:13:04.488 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.354 7f4e0ab56700 1 -- 172.21.15.32:0/183365579 wait complete. 2021-04-09T18:13:04.488 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.374 7f4e0ab56700 1 Processor -- start 2021-04-09T18:13:04.489 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.374 7f4e0ab56700 1 -- start start 2021-04-09T18:13:04.489 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e0ab56700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e041acbb0 0x7f4e041acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.577 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e0ab56700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e04187a40 0x7f4e04187e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.577 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e0ab56700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e041ab080 0x7f4e041ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.578 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e0ab56700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04187540 con 0x7f4e04187a40 2021-04-09T18:13:04.578 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e0ab56700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04188460 con 0x7f4e041ab080 2021-04-09T18:13:04.578 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e0ab56700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e041abaa0 con 0x7f4e041acbb0 2021-04-09T18:13:04.579 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e03fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e04187a40 0x7f4e04187e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:04.579 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e03fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e04187a40 0x7f4e04187e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41958/0 (socket says 172.21.15.32:41958) 2021-04-09T18:13:04.579 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e03fff700 1 -- 172.21.15.32:0/3959359842 learned_addr learned my addr 172.21.15.32:0/3959359842 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:04.580 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e08972700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e041acbb0 0x7f4e041acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:04.581 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e09173700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e041ab080 0x7f4e041ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:04.581 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e03fff700 1 -- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e041acbb0 msgr2=0x7f4e041acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.581 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e03fff700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e041acbb0 0x7f4e041acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.582 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e03fff700 1 -- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e041ab080 msgr2=0x7f4e041ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.582 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e03fff700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e041ab080 0x7f4e041ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.583 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e03fff700 1 -- 172.21.15.32:0/3959359842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e041abcc0 con 0x7f4e04187a40 2021-04-09T18:13:04.583 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e08972700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e041acbb0 0x7f4e041acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:04.610 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e09173700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e041ab080 0x7f4e041ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:04.610 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.375 7f4e03fff700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e04187a40 0x7f4e04187e40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4df4006f60 tx=0x7f4df400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:04.611 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.376 7f4e017fa700 1 -- 172.21.15.32:0/3959359842 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4df400bd60 con 0x7f4e04187a40 2021-04-09T18:13:04.611 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.376 7f4e017fa700 1 -- 172.21.15.32:0/3959359842 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df4000c90 con 0x7f4e04187a40 2021-04-09T18:13:04.615 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.376 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e041abcc0 con 0x7f4e04187a40 2021-04-09T18:13:04.616 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.376 7f4e017fa700 1 -- 172.21.15.32:0/3959359842 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4df4002220 con 0x7f4e04187a40 2021-04-09T18:13:04.616 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.376 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e041abcc0 con 0x7f4e04187a40 2021-04-09T18:13:04.616 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.377 7f4e017fa700 1 -- 172.21.15.32:0/3959359842 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4df4000c90 con 0x7f4e04187a40 2021-04-09T18:13:04.616 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.377 7f4e017fa700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4de80188c0 0x7f4de801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.616 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.377 7f4e017fa700 1 -- 172.21.15.32:0/3959359842 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4df4032770 con 0x7f4e04187a40 2021-04-09T18:13:04.617 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.377 7f4e08972700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4de80188c0 0x7f4de801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:04.617 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.378 7f4e08972700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4de80188c0 0x7f4de801adc0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:04.617 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.396 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e04188460 con 0x7f4e04187a40 2021-04-09T18:13:04.618 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.400 7f4e017fa700 1 -- 172.21.15.32:0/3959359842 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4df4006b00 con 0x7f4e04187a40 2021-04-09T18:13:04.618 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.588 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7f4e04188460 con 0x7f4e04187a40 2021-04-09T18:13:04.619 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.588 7f4e017fa700 1 -- 172.21.15.32:0/3959359842 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+10 (secure 0 0 0) 0x7f4df400bd60 con 0x7f4e04187a40 2021-04-09T18:13:04.619 INFO:teuthology.misc.health.smithi032.stdout:HEALTH_OK 2021-04-09T18:13:04.652 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4de80188c0 msgr2=0x7f4de801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.653 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4de80188c0 0x7f4de801adc0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.700 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e04187a40 msgr2=0x7f4e04187e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.700 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e04187a40 0x7f4e04187e40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4df4006f60 tx=0x7f4df400b970).stop 2021-04-09T18:13:04.701 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 shutdown_connections 2021-04-09T18:13:04.701 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4de80188c0 0x7f4de801adc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.701 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e04187a40 0x7f4e04187e40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.701 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e041ab080 0x7f4e041ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.701 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 --2- 172.21.15.32:0/3959359842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e041acbb0 0x7f4e041acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.702 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 >> 172.21.15.32:0/3959359842 conn(0x7f4e04183810 msgr2=0x7f4e0419b900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:04.702 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 shutdown_connections 2021-04-09T18:13:04.702 INFO:teuthology.misc.health.smithi032.stderr:2021-04-09 18:13:04.591 7f4e0ab56700 1 -- 172.21.15.32:0/3959359842 wait complete. 2021-04-09T18:13:04.707 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2021-04-09T18:13:04.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.837 7f8dd6ee9700 1 Processor -- start 2021-04-09T18:13:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.837 7f8dd6ee9700 1 -- start start 2021-04-09T18:13:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd6ee9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8dd010d010 0x7f8dd00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd6ee9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8dd00d8490 0x7f8dd010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd6ee9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd010d480 con 0x7f8dd00d1da0 2021-04-09T18:13:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd6ee9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd00d8d30 con 0x7f8dd00d8490 2021-04-09T18:13:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd6ee9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd00d9220 con 0x7f8dd010d010 2021-04-09T18:13:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd5506700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8dd00d8490 0x7f8dd010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd5506700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8dd00d8490 0x7f8dd010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41966/0 (socket says 172.21.15.32:41966) 2021-04-09T18:13:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd5506700 1 -- 172.21.15.32:0/210071060 learned_addr learned my addr 172.21.15.32:0/210071060 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd4d05700 1 --2- 172.21.15.32:0/210071060 >> v2:172.21.15.86:3300/0 conn(0x7f8dd010d010 0x7f8dd00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd5506700 1 -- 172.21.15.32:0/210071060 >> v2:172.21.15.86:3300/0 conn(0x7f8dd010d010 msgr2=0x7f8dd00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd5506700 1 --2- 172.21.15.32:0/210071060 >> v2:172.21.15.86:3300/0 conn(0x7f8dd010d010 0x7f8dd00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.838 7f8dd5506700 1 -- 172.21.15.32:0/210071060 >> v1:172.21.15.86:6789/0 conn(0x7f8dd00d1da0 legacy=0x7f8dd00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.839 7f8dd5506700 1 -- 172.21.15.32:0/210071060 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd010e020 con 0x7f8dd00d8490 2021-04-09T18:13:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.839 7f8dcf7fe700 1 -- 172.21.15.32:0/210071060 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584393354 0 0) 0x7f8dc4000c60 con 0x7f8dd00d1da0 2021-04-09T18:13:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.839 7f8dd4d05700 1 --2- 172.21.15.32:0/210071060 >> v2:172.21.15.86:3300/0 conn(0x7f8dd010d010 0x7f8dd00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.839 7f8dd5506700 1 --2- 172.21.15.32:0/210071060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd00d8490 0x7f8dd010da00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f8dc8006f00 tx=0x7f8dc8009f10).ready entity=mon.0 client_cookie=7824c5f68bf0203c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.839 7f8dcf7fe700 1 -- 172.21.15.32:0/210071060 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dc800b3b0 con 0x7f8dd00d8490 2021-04-09T18:13:04.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.839 7f8dcf7fe700 1 -- 172.21.15.32:0/210071060 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc800b730 con 0x7f8dd00d8490 2021-04-09T18:13:04.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.839 7f8dcf7fe700 1 -- 172.21.15.32:0/210071060 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dc8000f90 con 0x7f8dd00d8490 2021-04-09T18:13:04.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.839 7f8dd6ee9700 1 -- 172.21.15.32:0/210071060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd00d8490 msgr2=0x7f8dd010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:04.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.839 7f8dd6ee9700 1 --2- 172.21.15.32:0/210071060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd00d8490 0x7f8dd010da00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f8dc8006f00 tx=0x7f8dc8009f10).stop 2021-04-09T18:13:04.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.840 7f8dd6ee9700 1 -- 172.21.15.32:0/210071060 shutdown_connections 2021-04-09T18:13:04.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.840 7f8dd6ee9700 1 --2- 172.21.15.32:0/210071060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd00d8490 0x7f8dd010da00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.840 7f8dd6ee9700 1 --2- 172.21.15.32:0/210071060 >> v2:172.21.15.86:3300/0 conn(0x7f8dd010d010 0x7f8dd00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:04.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.840 7f8dd6ee9700 1 -- 172.21.15.32:0/210071060 >> 172.21.15.32:0/210071060 conn(0x7f8dd00ce5f0 msgr2=0x7f8dd00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:04.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.840 7f8dd6ee9700 1 -- 172.21.15.32:0/210071060 shutdown_connections 2021-04-09T18:13:04.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.840 7f8dd6ee9700 1 -- 172.21.15.32:0/210071060 wait complete. 2021-04-09T18:13:04.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.860 7f8dd6ee9700 1 Processor -- start 2021-04-09T18:13:04.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.860 7f8dd6ee9700 1 -- start start 2021-04-09T18:13:04.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.860 7f8dd6ee9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dd0185060 0x7f8dd0185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.860 7f8dd6ee9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd0186c00 0x7f8dd0187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.860 7f8dd6ee9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dd0191360 0x7f8dd0191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:04.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.860 7f8dd6ee9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd01863a0 con 0x7f8dd0186c00 2021-04-09T18:13:04.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.860 7f8dd6ee9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0185a60 con 0x7f8dd0191360 2021-04-09T18:13:04.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.860 7f8dd6ee9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0187620 con 0x7f8dd0185060 2021-04-09T18:13:05.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd0186c00 0x7f8dd0187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd0186c00 0x7f8dd0187000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41970/0 (socket says 172.21.15.32:41970) 2021-04-09T18:13:05.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcffff700 1 -- 172.21.15.32:0/2570807544 learned_addr learned my addr 172.21.15.32:0/2570807544 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:05.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dd4d05700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dd0185060 0x7f8dd0185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dd5506700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dd0191360 0x7f8dd0191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcffff700 1 -- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dd0185060 msgr2=0x7f8dd0185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcffff700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dd0185060 0x7f8dd0185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcffff700 1 -- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dd0191360 msgr2=0x7f8dd0191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcffff700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dd0191360 0x7f8dd0191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcffff700 1 -- 172.21.15.32:0/2570807544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0185460 con 0x7f8dd0186c00 2021-04-09T18:13:05.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dd5506700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dd0191360 0x7f8dd0191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:05.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcffff700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd0186c00 0x7f8dd0187000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f8dc400b580 tx=0x7f8dc400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:05.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcd7fa700 1 -- 172.21.15.32:0/2570807544 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dc400bd10 con 0x7f8dd0186c00 2021-04-09T18:13:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcd7fa700 1 -- 172.21.15.32:0/2570807544 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc4001080 con 0x7f8dd0186c00 2021-04-09T18:13:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dcd7fa700 1 -- 172.21.15.32:0/2570807544 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dc4001550 con 0x7f8dd0186c00 2021-04-09T18:13:05.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd0185460 con 0x7f8dd0186c00 2021-04-09T18:13:05.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.861 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd0185460 con 0x7f8dd0186c00 2021-04-09T18:13:05.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.862 7f8dcd7fa700 1 -- 172.21.15.32:0/2570807544 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8dc400a000 con 0x7f8dd0186c00 2021-04-09T18:13:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.862 7f8dcd7fa700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8db0018930 0x7f8db001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.862 7f8dcd7fa700 1 -- 172.21.15.32:0/2570807544 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8dc4032a50 con 0x7f8dd0186c00 2021-04-09T18:13:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.862 7f8dd4d05700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8db0018930 0x7f8db001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.863 7f8dd4d05700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8db0018930 0x7f8db001adc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.882 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd01866d0 con 0x7f8dd0186c00 2021-04-09T18:13:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:04.886 7f8dcd7fa700 1 -- 172.21.15.32:0/2570807544 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8dc4006bf0 con 0x7f8dd0186c00 2021-04-09T18:13:05.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.076 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dd01866d0 con 0x7f8dd0186c00 2021-04-09T18:13:05.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.077 7f8dcd7fa700 1 -- 172.21.15.32:0/2570807544 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f8dc400a000 con 0x7f8dd0186c00 2021-04-09T18:13:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:05.108 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:34.773743","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:05.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8db0018930 msgr2=0x7f8db001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8db0018930 0x7f8db001adc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd0186c00 msgr2=0x7f8dd0187000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd0186c00 0x7f8dd0187000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f8dc400b580 tx=0x7f8dc400b920).stop 2021-04-09T18:13:05.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 shutdown_connections 2021-04-09T18:13:05.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8db0018930 0x7f8db001adc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dd0186c00 0x7f8dd0187000 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dd0191360 0x7f8dd0191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 --2- 172.21.15.32:0/2570807544 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dd0185060 0x7f8dd0185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 >> 172.21.15.32:0/2570807544 conn(0x7f8dd01837d0 msgr2=0x7f8dd0195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:05.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 shutdown_connections 2021-04-09T18:13:05.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.079 7f8dd6ee9700 1 -- 172.21.15.32:0/2570807544 wait complete. 2021-04-09T18:13:05.146 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:12:32.136283', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:12:32.831693', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:13:05.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:13:05.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.283 7f0bcb613700 1 Processor -- start 2021-04-09T18:13:05.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.283 7f0bcb613700 1 -- start start 2021-04-09T18:13:05.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bcb613700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0bc40d6ca0 0x7f0bc40d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bcb613700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0bc40d2460 0x7f0bc410cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bcb613700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc40d72a0 con 0x7f0bc40cffc0 2021-04-09T18:13:05.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bcb613700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc40d70a0 con 0x7f0bc40d2460 2021-04-09T18:13:05.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bcb613700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc40d9200 con 0x7f0bc40d6ca0 2021-04-09T18:13:05.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bc9c30700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0bc40d2460 0x7f0bc410cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bc942f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0bc40cffc0 0x7f0bc40d7820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51568/0 (socket says 172.21.15.32:51568) 2021-04-09T18:13:05.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bc942f700 1 -- 172.21.15.32:0/291785078 learned_addr learned my addr 172.21.15.32:0/291785078 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:05.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bc8c2e700 1 --2- 172.21.15.32:0/291785078 >> v2:172.21.15.86:3300/0 conn(0x7f0bc40d6ca0 0x7f0bc40d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bc9c30700 1 -- 172.21.15.32:0/291785078 >> v2:172.21.15.86:3300/0 conn(0x7f0bc40d6ca0 msgr2=0x7f0bc40d8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bc9c30700 1 --2- 172.21.15.32:0/291785078 >> v2:172.21.15.86:3300/0 conn(0x7f0bc40d6ca0 0x7f0bc40d8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bc9c30700 1 -- 172.21.15.32:0/291785078 >> v1:172.21.15.32:6789/0 conn(0x7f0bc40cffc0 legacy=0x7f0bc40d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bbbfff700 1 -- 172.21.15.32:0/291785078 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486668750 0 0) 0x7f0bc0000ce0 con 0x7f0bc40cffc0 2021-04-09T18:13:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bc9c30700 1 -- 172.21.15.32:0/291785078 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc40d2860 con 0x7f0bc40d2460 2021-04-09T18:13:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.284 7f0bc9c30700 1 --2- 172.21.15.32:0/291785078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40d2460 0x7f0bc410cff0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0bbc006e30 tx=0x7f0bbc009de0).ready entity=mon.0 client_cookie=d621df2fbb038003 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:05.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bbbfff700 1 -- 172.21.15.32:0/291785078 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bbc00b300 con 0x7f0bc40d2460 2021-04-09T18:13:05.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bbbfff700 1 -- 172.21.15.32:0/291785078 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc00b680 con 0x7f0bc40d2460 2021-04-09T18:13:05.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bbbfff700 1 -- 172.21.15.32:0/291785078 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bbc00bb60 con 0x7f0bc40d2460 2021-04-09T18:13:05.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bcb613700 1 -- 172.21.15.32:0/291785078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40d2460 msgr2=0x7f0bc410cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bcb613700 1 --2- 172.21.15.32:0/291785078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40d2460 0x7f0bc410cff0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0bbc006e30 tx=0x7f0bbc009de0).stop 2021-04-09T18:13:05.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bcb613700 1 -- 172.21.15.32:0/291785078 shutdown_connections 2021-04-09T18:13:05.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bcb613700 1 --2- 172.21.15.32:0/291785078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40d2460 0x7f0bc410cff0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bcb613700 1 --2- 172.21.15.32:0/291785078 >> v2:172.21.15.86:3300/0 conn(0x7f0bc40d6ca0 0x7f0bc40d8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bcb613700 1 -- 172.21.15.32:0/291785078 >> 172.21.15.32:0/291785078 conn(0x7f0bc40ce5e0 msgr2=0x7f0bc40cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:05.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.285 7f0bcb613700 1 -- 172.21.15.32:0/291785078 shutdown_connections 2021-04-09T18:13:05.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.286 7f0bcb613700 1 -- 172.21.15.32:0/291785078 wait complete. 2021-04-09T18:13:05.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bcb613700 1 Processor -- start 2021-04-09T18:13:05.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bcb613700 1 -- start start 2021-04-09T18:13:05.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bcb613700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bc4098cc0 0x7f0bc409b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bcb613700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40990c0 0x7f0bc409bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bcb613700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4094d40 0x7f0bc4095140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bcb613700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40994c0 con 0x7f0bc40990c0 2021-04-09T18:13:05.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bcb613700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40997e0 con 0x7f0bc4094d40 2021-04-09T18:13:05.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bcb613700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc409aff0 con 0x7f0bc4098cc0 2021-04-09T18:13:05.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bc8c2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40990c0 0x7f0bc409bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bc8c2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40990c0 0x7f0bc409bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41986/0 (socket says 172.21.15.32:41986) 2021-04-09T18:13:05.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.306 7f0bc8c2e700 1 -- 172.21.15.32:0/730704866 learned_addr learned my addr 172.21.15.32:0/730704866 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:05.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bc942f700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bc4098cc0 0x7f0bc409b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bc9c30700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4094d40 0x7f0bc4095140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bc8c2e700 1 -- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bc4098cc0 msgr2=0x7f0bc409b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bc8c2e700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bc4098cc0 0x7f0bc409b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bc8c2e700 1 -- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4094d40 msgr2=0x7f0bc4095140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bc8c2e700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4094d40 0x7f0bc4095140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bc8c2e700 1 -- 172.21.15.32:0/730704866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc409c1b0 con 0x7f0bc40990c0 2021-04-09T18:13:05.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bc8c2e700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40990c0 0x7f0bc409bb90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f0bb4006f00 tx=0x7f0bb400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:05.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bb9ffb700 1 -- 172.21.15.32:0/730704866 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bb400bd30 con 0x7f0bc40990c0 2021-04-09T18:13:05.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bb9ffb700 1 -- 172.21.15.32:0/730704866 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb4002030 con 0x7f0bc40990c0 2021-04-09T18:13:05.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bcb613700 1 -- 172.21.15.32:0/730704866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc409c1b0 con 0x7f0bc40990c0 2021-04-09T18:13:05.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bb9ffb700 1 -- 172.21.15.32:0/730704866 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bb4002590 con 0x7f0bc40990c0 2021-04-09T18:13:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.307 7f0bcb613700 1 -- 172.21.15.32:0/730704866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc409c1b0 con 0x7f0bc40990c0 2021-04-09T18:13:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.308 7f0bb9ffb700 1 -- 172.21.15.32:0/730704866 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0bb4000c90 con 0x7f0bc40990c0 2021-04-09T18:13:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.308 7f0bb9ffb700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ba40188d0 0x7f0ba401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.308 7f0bb9ffb700 1 -- 172.21.15.32:0/730704866 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0bb4032a40 con 0x7f0bc40990c0 2021-04-09T18:13:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.308 7f0bc942f700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ba40188d0 0x7f0ba401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.309 7f0bc942f700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ba40188d0 0x7f0ba401add0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.328 7f0bcb613700 1 -- 172.21.15.32:0/730704866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc4099790 con 0x7f0bc40990c0 2021-04-09T18:13:05.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.331 7f0bb9ffb700 1 -- 172.21.15.32:0/730704866 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0bb401f780 con 0x7f0bc40990c0 2021-04-09T18:13:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.509 7f0bcb613700 1 -- 172.21.15.32:0/730704866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f0bc4099790 con 0x7f0bc40990c0 2021-04-09T18:13:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.509 7f0bb9ffb700 1 -- 172.21.15.32:0/730704866 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0bb400bd30 con 0x7f0bc40990c0 2021-04-09T18:13:05.616 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:05.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 -- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ba40188d0 msgr2=0x7f0ba401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ba40188d0 0x7f0ba401add0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 -- 172.21.15.32:0/730704866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40990c0 msgr2=0x7f0bc409bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40990c0 0x7f0bc409bb90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f0bb4006f00 tx=0x7f0bb400b940).stop 2021-04-09T18:13:05.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 -- 172.21.15.32:0/730704866 shutdown_connections 2021-04-09T18:13:05.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ba40188d0 0x7f0ba401add0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bc40990c0 0x7f0bc409bb90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bc4094d40 0x7f0bc4095140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 --2- 172.21.15.32:0/730704866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bc4098cc0 0x7f0bc409b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 -- 172.21.15.32:0/730704866 >> 172.21.15.32:0/730704866 conn(0x7f0bc409fa00 msgr2=0x7f0bc40c3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 -- 172.21.15.32:0/730704866 shutdown_connections 2021-04-09T18:13:05.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.512 7f0bcb613700 1 -- 172.21.15.32:0/730704866 wait complete. 2021-04-09T18:13:05.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:13:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.810 7fb70cc2a700 1 Processor -- start 2021-04-09T18:13:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.810 7fb70cc2a700 1 -- start start 2021-04-09T18:13:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70cc2a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7080cffc0 0x7fb7080d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70cc2a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb7080d7820 0x7fb7080d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70cc2a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080d8240 0x7fb7080d5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70cc2a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7080d9200 con 0x7fb7080cffc0 2021-04-09T18:13:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70cc2a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7080d72a0 con 0x7fb7080d7820 2021-04-09T18:13:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70cc2a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080d62b0 con 0x7fb7080d8240 2021-04-09T18:13:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70685e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7080cffc0 0x7fb7080d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70685e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7080cffc0 0x7fb7080d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41992/0 (socket says 172.21.15.32:41992) 2021-04-09T18:13:05.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70685e700 1 -- 172.21.15.32:0/2176261893 learned_addr learned my addr 172.21.15.32:0/2176261893 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70705f700 1 --2- 172.21.15.32:0/2176261893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080d8240 0x7fb7080d5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.811 7fb70605d700 1 --2- 172.21.15.32:0/2176261893 >> v2:172.21.15.86:3300/0 conn(0x7fb7080d7820 0x7fb7080d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70685e700 1 -- 172.21.15.32:0/2176261893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080d8240 msgr2=0x7fb7080d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70685e700 1 --2- 172.21.15.32:0/2176261893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080d8240 0x7fb7080d5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70685e700 1 -- 172.21.15.32:0/2176261893 >> v2:172.21.15.86:3300/0 conn(0x7fb7080d7820 msgr2=0x7fb7080d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70685e700 1 --2- 172.21.15.32:0/2176261893 >> v2:172.21.15.86:3300/0 conn(0x7fb7080d7820 0x7fb7080d7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70685e700 1 -- 172.21.15.32:0/2176261893 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7080d64a0 con 0x7fb7080cffc0 2021-04-09T18:13:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70605d700 1 --2- 172.21.15.32:0/2176261893 >> v2:172.21.15.86:3300/0 conn(0x7fb7080d7820 0x7fb7080d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70705f700 1 --2- 172.21.15.32:0/2176261893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080d8240 0x7fb7080d5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:05.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70685e700 1 --2- 172.21.15.32:0/2176261893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080cffc0 0x7fb7080d2460 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb6f8006700 tx=0x7fb6f8009b80).ready entity=mon.0 client_cookie=3e24dde0d4330915 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:05.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70585c700 1 -- 172.21.15.32:0/2176261893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f800c0c0 con 0x7fb7080cffc0 2021-04-09T18:13:05.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70585c700 1 -- 172.21.15.32:0/2176261893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f800c440 con 0x7fb7080cffc0 2021-04-09T18:13:05.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70585c700 1 -- 172.21.15.32:0/2176261893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f800c920 con 0x7fb7080cffc0 2021-04-09T18:13:05.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70cc2a700 1 -- 172.21.15.32:0/2176261893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080cffc0 msgr2=0x7fb7080d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.812 7fb70cc2a700 1 --2- 172.21.15.32:0/2176261893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080cffc0 0x7fb7080d2460 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb6f8006700 tx=0x7fb6f8009b80).stop 2021-04-09T18:13:05.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.813 7fb70cc2a700 1 -- 172.21.15.32:0/2176261893 shutdown_connections 2021-04-09T18:13:05.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.813 7fb70cc2a700 1 --2- 172.21.15.32:0/2176261893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080d8240 0x7fb7080d5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.813 7fb70cc2a700 1 --2- 172.21.15.32:0/2176261893 >> v2:172.21.15.86:3300/0 conn(0x7fb7080d7820 0x7fb7080d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.813 7fb70cc2a700 1 --2- 172.21.15.32:0/2176261893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080cffc0 0x7fb7080d2460 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.813 7fb70cc2a700 1 -- 172.21.15.32:0/2176261893 >> 172.21.15.32:0/2176261893 conn(0x7fb7080ce5e0 msgr2=0x7fb7080cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:05.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.813 7fb70cc2a700 1 -- 172.21.15.32:0/2176261893 shutdown_connections 2021-04-09T18:13:05.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.813 7fb70cc2a700 1 -- 172.21.15.32:0/2176261893 wait complete. 2021-04-09T18:13:05.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.833 7fb70cc2a700 1 Processor -- start 2021-04-09T18:13:05.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.833 7fb70cc2a700 1 -- start start 2021-04-09T18:13:05.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70cc2a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080b89a0 0x7fb7080c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70cc2a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080b8da0 0x7fb7080be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70cc2a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7080b91a0 0x7fb7080bead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:05.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70cc2a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080c39b0 con 0x7fb7080b89a0 2021-04-09T18:13:05.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70cc2a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080c0d60 con 0x7fb7080b91a0 2021-04-09T18:13:05.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70cc2a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080bf0f0 con 0x7fb7080b8da0 2021-04-09T18:13:05.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080b89a0 0x7fb7080c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080b89a0 0x7fb7080c31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41998/0 (socket says 172.21.15.32:41998) 2021-04-09T18:13:05.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70685e700 1 -- 172.21.15.32:0/1136833571 learned_addr learned my addr 172.21.15.32:0/1136833571 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:05.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70605d700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080b8da0 0x7fb7080be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70705f700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7080b91a0 0x7fb7080bead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:05.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70685e700 1 -- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080b8da0 msgr2=0x7fb7080be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70685e700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080b8da0 0x7fb7080be4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70685e700 1 -- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7080b91a0 msgr2=0x7fb7080bead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:05.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70685e700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7080b91a0 0x7fb7080bead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:05.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70685e700 1 -- 172.21.15.32:0/1136833571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7080bf310 con 0x7fb7080b89a0 2021-04-09T18:13:05.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70685e700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080b89a0 0x7fb7080c31a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb6f800b580 tx=0x7fb6f800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:05.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb6f77fe700 1 -- 172.21.15.32:0/1136833571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f800bd10 con 0x7fb7080b89a0 2021-04-09T18:13:05.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb6f77fe700 1 -- 172.21.15.32:0/1136833571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f8002030 con 0x7fb7080b89a0 2021-04-09T18:13:05.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb6f77fe700 1 -- 172.21.15.32:0/1136833571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f8002550 con 0x7fb7080b89a0 2021-04-09T18:13:05.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7080bf310 con 0x7fb7080b89a0 2021-04-09T18:13:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.834 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7080bf310 con 0x7fb7080b89a0 2021-04-09T18:13:06.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.835 7fb6f77fe700 1 -- 172.21.15.32:0/1136833571 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb6f8002420 con 0x7fb7080b89a0 2021-04-09T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.835 7fb6f77fe700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6ec018930 0x7fb6ec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.835 7fb6f77fe700 1 -- 172.21.15.32:0/1136833571 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb6f8032a40 con 0x7fb7080b89a0 2021-04-09T18:13:06.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.835 7fb70605d700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6ec018930 0x7fb6ec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.836 7fb70605d700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6ec018930 0x7fb6ec01ae10 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:06.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.855 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7081b70c0 con 0x7fb7080b89a0 2021-04-09T18:13:06.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:05.859 7fb6f77fe700 1 -- 172.21.15.32:0/1136833571 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb6f8002010 con 0x7fb7080b89a0 2021-04-09T18:13:06.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.037 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb7080d72a0 con 0x7fb7080b89a0 2021-04-09T18:13:06.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.037 7fb6f77fe700 1 -- 172.21.15.32:0/1136833571 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb6f8002220 con 0x7fb7080b89a0 2021-04-09T18:13:06.062 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:06.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6ec018930 msgr2=0x7fb6ec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6ec018930 0x7fb6ec01ae10 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080b89a0 msgr2=0x7fb7080c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080b89a0 0x7fb7080c31a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb6f800b580 tx=0x7fb6f800b920).stop 2021-04-09T18:13:06.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 shutdown_connections 2021-04-09T18:13:06.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6ec018930 0x7fb6ec01ae10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7080b91a0 0x7fb7080bead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7080b8da0 0x7fb7080be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 --2- 172.21.15.32:0/1136833571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7080b89a0 0x7fb7080c31a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 >> 172.21.15.32:0/1136833571 conn(0x7fb7080a06a0 msgr2=0x7fb70808fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 shutdown_connections 2021-04-09T18:13:06.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.040 7fb70cc2a700 1 -- 172.21.15.32:0/1136833571 wait complete. 2021-04-09T18:13:06.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:13:06.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.247 7fdb1daa0700 1 Processor -- start 2021-04-09T18:13:06.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.247 7fdb1daa0700 1 -- start start 2021-04-09T18:13:06.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.248 7fdb1daa0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdb1810da00 0x7fdb1810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.248 7fdb1daa0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb1810d480 con 0x7fdb1810d010 2021-04-09T18:13:06.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.248 7fdb1daa0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb180d1f70 con 0x7fdb180d7ca0 2021-04-09T18:13:06.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.248 7fdb1daa0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb180d8d30 con 0x7fdb1810da00 2021-04-09T18:13:06.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.248 7fdb177fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdb1810d010 0x7fdb180d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51596/0 (socket says 172.21.15.32:51596) 2021-04-09T18:13:06.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.248 7fdb177fe700 1 -- 172.21.15.32:0/3450114995 learned_addr learned my addr 172.21.15.32:0/3450114995 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:06.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.248 7fdb17fff700 1 --2- 172.21.15.32:0/3450114995 >> v2:172.21.15.86:3300/0 conn(0x7fdb1810da00 0x7fdb1810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.248 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853040135 0 0) 0x7fdb08000ce0 con 0x7fdb1810d010 2021-04-09T18:13:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdafc002fe0 con 0x7fdb1810d010 2021-04-09T18:13:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2125523365 0 0) 0x7fdb08000ed0 con 0x7fdb1810d010 2021-04-09T18:13:06.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 >> v2:172.21.15.86:3300/0 conn(0x7fdb1810da00 msgr2=0x7fdb1810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 --2- 172.21.15.32:0/3450114995 >> v2:172.21.15.86:3300/0 conn(0x7fdb1810da00 0x7fdb1810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 >> v1:172.21.15.86:6789/0 conn(0x7fdb180d7ca0 legacy=0x7fdb180d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb180d9220 con 0x7fdb1810d010 2021-04-09T18:13:06.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fdb08001600 con 0x7fdb1810d010 2021-04-09T18:13:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 >> v1:172.21.15.32:6789/0 conn(0x7fdb1810d010 legacy=0x7fdb180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc003a40 0x7fdafc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc006790 0x7fdafc008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdafc009590 0x7fdafc00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc00c300 con 0x7fdafc006790 2021-04-09T18:13:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc00c520 con 0x7fdafc003a40 2021-04-09T18:13:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc050750 con 0x7fdafc009590 2021-04-09T18:13:06.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb16ffd700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc006790 0x7fdafc008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb177fe700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc003a40 0x7fdafc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb17fff700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdafc009590 0x7fdafc00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb16ffd700 1 -- 172.21.15.32:0/3450114995 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdafc009590 msgr2=0x7fdafc00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:06.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb16ffd700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdafc009590 0x7fdafc00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb16ffd700 1 -- 172.21.15.32:0/3450114995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc003a40 msgr2=0x7fdafc005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:06.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb16ffd700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc003a40 0x7fdafc005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb17fff700 1 -- 172.21.15.32:0/3450114995 reap_dead start 2021-04-09T18:13:06.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb16ffd700 1 -- 172.21.15.32:0/3450114995 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdafc0509b0 con 0x7fdafc006790 2021-04-09T18:13:06.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb177fe700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc003a40 0x7fdafc005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.249 7fdb16ffd700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc006790 0x7fdafc008c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fdb04006ab0 tx=0x7fdb04009a90).ready entity=mon.0 client_cookie=9d7ba68fcbfc42b5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb040066a0 con 0x7fdafc006790 2021-04-09T18:13:06.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb040022e0 con 0x7fdafc006790 2021-04-09T18:13:06.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb167fc700 1 -- 172.21.15.32:0/3450114995 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb04002740 con 0x7fdafc006790 2021-04-09T18:13:06.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb1daa0700 1 -- 172.21.15.32:0/3450114995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc006790 msgr2=0x7fdafc008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb1daa0700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc006790 0x7fdafc008c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fdb04006ab0 tx=0x7fdb04009a90).stop 2021-04-09T18:13:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb1daa0700 1 -- 172.21.15.32:0/3450114995 shutdown_connections 2021-04-09T18:13:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb1daa0700 1 --2- 172.21.15.32:0/3450114995 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdafc006790 0x7fdafc008c90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb1daa0700 1 -- 172.21.15.32:0/3450114995 >> 172.21.15.32:0/3450114995 conn(0x7fdb180ce5f0 msgr2=0x7fdb180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb1daa0700 1 -- 172.21.15.32:0/3450114995 shutdown_connections 2021-04-09T18:13:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.250 7fdb1daa0700 1 -- 172.21.15.32:0/3450114995 wait complete. 2021-04-09T18:13:06.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.270 7fdb1daa0700 1 Processor -- start 2021-04-09T18:13:06.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb1daa0700 1 -- start start 2021-04-09T18:13:06.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb1daa0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb1809e530 0x7fdb1809b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb1daa0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb1809e930 0x7fdb1809f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb1daa0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb1809f990 0x7fdb18094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb1daa0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1809edf0 con 0x7fdb1809e930 2021-04-09T18:13:06.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb1daa0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1809fd90 con 0x7fdb1809e530 2021-04-09T18:13:06.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb1daa0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1809bd40 con 0x7fdb1809f990 2021-04-09T18:13:06.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb16ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb1809e930 0x7fdb1809f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb16ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb1809e930 0x7fdb1809f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42022/0 (socket says 172.21.15.32:42022) 2021-04-09T18:13:06.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb16ffd700 1 -- 172.21.15.32:0/3926493344 learned_addr learned my addr 172.21.15.32:0/3926493344 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:06.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb17fff700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb1809f990 0x7fdb18094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.271 7fdb177fe700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb1809e530 0x7fdb1809b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb16ffd700 1 -- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb1809f990 msgr2=0x7fdb18094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb16ffd700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb1809f990 0x7fdb18094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb16ffd700 1 -- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb1809e530 msgr2=0x7fdb1809b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb16ffd700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb1809e530 0x7fdb1809b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb16ffd700 1 -- 172.21.15.32:0/3926493344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb18095290 con 0x7fdb1809e930 2021-04-09T18:13:06.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb16ffd700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb1809e930 0x7fdb1809f370 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fdb04000bf0 tx=0x7fdb0400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:06.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb0ffff700 1 -- 172.21.15.32:0/3926493344 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb040093e0 con 0x7fdb1809e930 2021-04-09T18:13:06.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb0ffff700 1 -- 172.21.15.32:0/3926493344 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb04006560 con 0x7fdb1809e930 2021-04-09T18:13:06.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb18095290 con 0x7fdb1809e930 2021-04-09T18:13:06.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb0ffff700 1 -- 172.21.15.32:0/3926493344 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb04002470 con 0x7fdb1809e930 2021-04-09T18:13:06.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.272 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb18095290 con 0x7fdb1809e930 2021-04-09T18:13:06.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.273 7fdb0ffff700 1 -- 172.21.15.32:0/3926493344 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdb04009380 con 0x7fdb1809e930 2021-04-09T18:13:06.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.273 7fdb0ffff700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdaf4018930 0x7fdaf401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.273 7fdb0ffff700 1 -- 172.21.15.32:0/3926493344 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdb04032a10 con 0x7fdb1809e930 2021-04-09T18:13:06.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.273 7fdb177fe700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdaf4018930 0x7fdaf401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.273 7fdb177fe700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdaf4018930 0x7fdaf401ae10 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:06.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.292 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb18095b10 con 0x7fdb1809e930 2021-04-09T18:13:06.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.296 7fdb0ffff700 1 -- 172.21.15.32:0/3926493344 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdb04009020 con 0x7fdb1809e930 2021-04-09T18:13:06.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.475 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fdb18095b10 con 0x7fdb1809e930 2021-04-09T18:13:06.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.475 7fdb0ffff700 1 -- 172.21.15.32:0/3926493344 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fdb04000ef0 con 0x7fdb1809e930 2021-04-09T18:13:06.497 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:06.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdaf4018930 msgr2=0x7fdaf401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdaf4018930 0x7fdaf401ae10 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb1809e930 msgr2=0x7fdb1809f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb1809e930 0x7fdb1809f370 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fdb04000bf0 tx=0x7fdb0400b970).stop 2021-04-09T18:13:06.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 shutdown_connections 2021-04-09T18:13:06.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdaf4018930 0x7fdaf401ae10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb1809e930 0x7fdb1809f370 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb1809f990 0x7fdb18094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 --2- 172.21.15.32:0/3926493344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb1809e530 0x7fdb1809b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 >> 172.21.15.32:0/3926493344 conn(0x7fdb180c4290 msgr2=0x7fdb180c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:06.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 shutdown_connections 2021-04-09T18:13:06.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.478 7fdb1daa0700 1 -- 172.21.15.32:0/3926493344 wait complete. 2021-04-09T18:13:06.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:06.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.644 7f40d7d00700 1 Processor -- start 2021-04-09T18:13:06.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.644 7f40d7d00700 1 -- start start 2021-04-09T18:13:06.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.645 7f40d7d00700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f40d010d010 0x7f40d00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.645 7f40d7d00700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d010d480 con 0x7f40d00d8490 2021-04-09T18:13:06.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.645 7f40d7d00700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d010e1f0 con 0x7f40d00d1da0 2021-04-09T18:13:06.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.645 7f40d7d00700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d00d8d30 con 0x7f40d010d010 2021-04-09T18:13:06.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.645 7f40d631d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f40d00d8490 0x7f40d010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51618/0 (socket says 172.21.15.32:51618) 2021-04-09T18:13:06.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.645 7f40d631d700 1 -- 172.21.15.32:0/2805735158 learned_addr learned my addr 172.21.15.32:0/2805735158 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:06.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.645 7f40d5b1c700 1 --2- 172.21.15.32:0/2805735158 >> v2:172.21.15.86:3300/0 conn(0x7f40d010d010 0x7f40d00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.645 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762951943 0 0) 0x7f40c8000d30 con 0x7f40d00d8490 2021-04-09T18:13:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.645 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40bc002fe0 con 0x7f40d00d8490 2021-04-09T18:13:06.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 469229054 0 0) 0x7f40c8001460 con 0x7f40d00d8490 2021-04-09T18:13:06.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 >> v2:172.21.15.86:3300/0 conn(0x7f40d010d010 msgr2=0x7f40d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:06.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 --2- 172.21.15.32:0/2805735158 >> v2:172.21.15.86:3300/0 conn(0x7f40d010d010 0x7f40d00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 >> v1:172.21.15.86:6789/0 conn(0x7f40d00d1da0 legacy=0x7f40d00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d00d9220 con 0x7f40d00d8490 2021-04-09T18:13:06.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f40c8001980 con 0x7f40d00d8490 2021-04-09T18:13:06.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 >> v1:172.21.15.32:6789/0 conn(0x7f40d00d8490 legacy=0x7f40d010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40bc003ae0 0x7f40bc005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40bc006810 0x7f40bc008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40bc009630 0x7f40bc00bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc00c3a0 con 0x7f40bc003ae0 2021-04-09T18:13:06.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc00c5c0 con 0x7f40bc006810 2021-04-09T18:13:06.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0507f0 con 0x7f40bc009630 2021-04-09T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d5b1c700 1 --2- 172.21.15.32:0/2805735158 >> v2:172.21.15.86:3300/0 conn(0x7f40d010d010 0x7f40d00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:06.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d631d700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40bc009630 0x7f40bc00bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d531b700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40bc006810 0x7f40bc008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d5b1c700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40bc003ae0 0x7f40bc005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d5b1c700 1 -- 172.21.15.32:0/2805735158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40bc009630 msgr2=0x7f40bc00bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:06.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d5b1c700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40bc009630 0x7f40bc00bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d5b1c700 1 -- 172.21.15.32:0/2805735158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40bc006810 msgr2=0x7f40bc008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:06.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d5b1c700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40bc006810 0x7f40bc008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d5b1c700 1 -- 172.21.15.32:0/2805735158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc050a50 con 0x7f40bc003ae0 2021-04-09T18:13:06.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d631d700 1 -- 172.21.15.32:0/2805735158 reap_dead start 2021-04-09T18:13:06.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d5b1c700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40bc003ae0 0x7f40bc005f70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f40cc000b80 tx=0x7f40cc009e80).ready entity=mon.0 client_cookie=59023c93dfa6a0ae server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:06.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40cc006b40 con 0x7f40bc003ae0 2021-04-09T18:13:06.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc006e00 con 0x7f40bc003ae0 2021-04-09T18:13:06.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d4b1a700 1 -- 172.21.15.32:0/2805735158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40cc007260 con 0x7f40bc003ae0 2021-04-09T18:13:06.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d7d00700 1 -- 172.21.15.32:0/2805735158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40bc003ae0 msgr2=0x7f40bc005f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.646 7f40d7d00700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40bc003ae0 0x7f40bc005f70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f40cc000b80 tx=0x7f40cc009e80).stop 2021-04-09T18:13:06.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.647 7f40d7d00700 1 -- 172.21.15.32:0/2805735158 shutdown_connections 2021-04-09T18:13:06.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.647 7f40d7d00700 1 --2- 172.21.15.32:0/2805735158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40bc003ae0 0x7f40bc005f70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.647 7f40d7d00700 1 -- 172.21.15.32:0/2805735158 >> 172.21.15.32:0/2805735158 conn(0x7f40d00ce5f0 msgr2=0x7f40d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:06.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.647 7f40d7d00700 1 -- 172.21.15.32:0/2805735158 shutdown_connections 2021-04-09T18:13:06.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.647 7f40d7d00700 1 -- 172.21.15.32:0/2805735158 wait complete. 2021-04-09T18:13:06.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.667 7f40d7d00700 1 Processor -- start 2021-04-09T18:13:06.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.667 7f40d7d00700 1 -- start start 2021-04-09T18:13:06.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d7d00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40d01aca20 0x7f40d01ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d7d00700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40d018bd40 0x7f40d018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d7d00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d008fba0 0x7f40d008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d7d00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d018c7e0 con 0x7f40d008fba0 2021-04-09T18:13:06.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d7d00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d018ca30 con 0x7f40d01aca20 2021-04-09T18:13:06.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d7d00700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d00905f0 con 0x7f40d018bd40 2021-04-09T18:13:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d631d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d008fba0 0x7f40d008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d631d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d008fba0 0x7f40d008ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42042/0 (socket says 172.21.15.32:42042) 2021-04-09T18:13:06.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d631d700 1 -- 172.21.15.32:0/2071972345 learned_addr learned my addr 172.21.15.32:0/2071972345 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:06.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d5b1c700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40d01aca20 0x7f40d01ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d531b700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40d018bd40 0x7f40d018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d631d700 1 -- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40d018bd40 msgr2=0x7f40d018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d631d700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40d018bd40 0x7f40d018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d631d700 1 -- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40d01aca20 msgr2=0x7f40d01ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d631d700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40d01aca20 0x7f40d01ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d631d700 1 -- 172.21.15.32:0/2071972345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d00b4300 con 0x7f40d008fba0 2021-04-09T18:13:06.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.668 7f40d631d700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d008fba0 0x7f40d008ffa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f40c80079e0 tx=0x7f40c800ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:06.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.669 7f40c27fc700 1 -- 172.21.15.32:0/2071972345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40c8007020 con 0x7f40d008fba0 2021-04-09T18:13:06.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.669 7f40c27fc700 1 -- 172.21.15.32:0/2071972345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c80072e0 con 0x7f40d008fba0 2021-04-09T18:13:06.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.669 7f40c27fc700 1 -- 172.21.15.32:0/2071972345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40c8004720 con 0x7f40d008fba0 2021-04-09T18:13:06.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.669 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d00b4300 con 0x7f40d008fba0 2021-04-09T18:13:06.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.669 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d00b4300 con 0x7f40d008fba0 2021-04-09T18:13:06.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.669 7f40c27fc700 1 -- 172.21.15.32:0/2071972345 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f40c8001d50 con 0x7f40d008fba0 2021-04-09T18:13:06.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.669 7f40c27fc700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f40b8018930 0x7f40b801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:06.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.669 7f40c27fc700 1 -- 172.21.15.32:0/2071972345 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f40c8032d70 con 0x7f40d008fba0 2021-04-09T18:13:06.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.670 7f40d5b1c700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f40b8018930 0x7f40b801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:06.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.670 7f40d5b1c700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f40b8018930 0x7f40b801ae10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:06.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.689 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d0182d70 con 0x7f40d008fba0 2021-04-09T18:13:06.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.693 7f40c27fc700 1 -- 172.21.15.32:0/2071972345 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f40c8001b40 con 0x7f40d008fba0 2021-04-09T18:13:06.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.884 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40d018c760 con 0x7f40d008fba0 2021-04-09T18:13:06.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.885 7f40c27fc700 1 -- 172.21.15.32:0/2071972345 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2769 (secure 0 0 0) 0x7f40c80072e0 con 0x7f40d008fba0 2021-04-09T18:13:06.933 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 6 2021-04-09T18:13:06.934 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:06.934 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4323,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3379802756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3379802756},{"type":"v1","addr":"172.21.15.32:6835","nonce":3379802756}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4},{"gid":4333,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3413992045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3413992045},{"type":"v1","addr":"172.21.15.86:6835","nonce":3413992045}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4},{"gid":4342,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2501526335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2501526335},{"type":"v1","addr":"172.21.15.86:6837","nonce":2501526335}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:12:33.772139","modified":"2021-04-09 18:12:42.800036","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4314":{"gid":4314,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6833/3268604338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3268604338},{"type":"v1","addr":"172.21.15.32:6833","nonce":3268604338}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-04-09T18:13:06.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.887 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f40b8018930 msgr2=0x7f40b801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.887 7f40d7d00700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f40b8018930 0x7f40b801ae10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.887 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d008fba0 msgr2=0x7f40d008ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:06.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.887 7f40d7d00700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d008fba0 0x7f40d008ffa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f40c80079e0 tx=0x7f40c800ac80).stop 2021-04-09T18:13:06.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.887 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 shutdown_connections 2021-04-09T18:13:06.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.887 7f40d7d00700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f40b8018930 0x7f40b801ae10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.887 7f40d7d00700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40d008fba0 0x7f40d008ffa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.887 7f40d7d00700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40d018bd40 0x7f40d018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.887 7f40d7d00700 1 --2- 172.21.15.32:0/2071972345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40d01aca20 0x7f40d01ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:06.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.888 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 >> 172.21.15.32:0/2071972345 conn(0x7f40d019b460 msgr2=0x7f40d019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:06.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.888 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 shutdown_connections 2021-04-09T18:13:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:06.888 7f40d7d00700 1 -- 172.21.15.32:0/2071972345 wait complete. 2021-04-09T18:13:07.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:12:33.772139', 'modified': '2021-04-09 18:12:42.800036', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4314}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4314': {'gid': 4314, 'name': 'a', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6833/3268604338', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3268604338}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3268604338}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:13:07.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:13:07.349 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:13:07.496 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:13:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:13:07.497 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:13:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4314, 2021-04-09T18:13:07.498 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:13:07.499 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:13:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 25.677529046, 2021-04-09T18:13:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 6, 2021-04-09T18:13:07.501 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 20, 2021-04-09T18:13:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 20, 2021-04-09T18:13:07.502 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 30.449235567999999 2021-04-09T18:13:07.503 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:13:07.504 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 4314, "want_state": "up:active", "state": "up:active", "rank_uptime": 25.677529046, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 30.449235567999999 } 2021-04-09T18:13:07.507 INFO:teuthology.run_tasks:Running task ceph-fuse... 2021-04-09T18:13:07.661 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2021-04-09T18:13:07.662 INFO:tasks.ceph_fuse:config is None 2021-04-09T18:13:07.662 INFO:tasks.ceph_fuse:new config is {'client.0': None, 'client.1': None, 'disabled': True} 2021-04-09T18:13:07.663 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2021-04-09T18:13:07.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:07.664 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:13:07.664 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:13:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:13:07.745 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:13:07.751 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:13:07.752 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:07.768 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:13:07.769 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:13:07.843 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:13:07.845 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:13:07.846 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2021-04-09T18:13:07.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.054 7f54a571c700 1 Processor -- start 2021-04-09T18:13:08.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.054 7f54a571c700 1 -- start start 2021-04-09T18:13:08.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.054 7f54a571c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a00d7ca0 0x7f54a00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.054 7f54a571c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54a010da00 0x7f54a010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.054 7f54a571c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a010d480 con 0x7f54a010d010 2021-04-09T18:13:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.054 7f54a571c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a00d9220 con 0x7f54a010da00 2021-04-09T18:13:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.054 7f54a571c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00d86c0 con 0x7f54a00d7ca0 2021-04-09T18:13:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549fac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54a010da00 0x7f54a010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549fac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54a010da00 0x7f54a010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42048/0 (socket says 172.21.15.32:42048) 2021-04-09T18:13:08.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549fac0700 1 -- 172.21.15.32:0/1723135915 learned_addr learned my addr 172.21.15.32:0/1723135915 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549eabe700 1 --2- 172.21.15.32:0/1723135915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a00d7ca0 0x7f54a00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549fac0700 1 -- 172.21.15.32:0/1723135915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a00d7ca0 msgr2=0x7f54a00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:08.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549fac0700 1 --2- 172.21.15.32:0/1723135915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a00d7ca0 0x7f54a00d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549fac0700 1 -- 172.21.15.32:0/1723135915 >> v1:172.21.15.86:6789/0 conn(0x7f54a010d010 legacy=0x7f54a00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549fac0700 1 -- 172.21.15.32:0/1723135915 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a00d1f70 con 0x7f54a010da00 2021-04-09T18:13:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549fac0700 1 --2- 172.21.15.32:0/1723135915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a010da00 0x7f54a010de00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5498006b80 tx=0x7f5498009b10).ready entity=mon.0 client_cookie=51318a90d23b9acc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549e2bd700 1 -- 172.21.15.32:0/1723135915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54980022e0 con 0x7f54a010da00 2021-04-09T18:13:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549e2bd700 1 -- 172.21.15.32:0/1723135915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5498002620 con 0x7f54a010da00 2021-04-09T18:13:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f549e2bd700 1 -- 172.21.15.32:0/1723135915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5498002b00 con 0x7f54a010da00 2021-04-09T18:13:08.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f54a571c700 1 -- 172.21.15.32:0/1723135915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a010da00 msgr2=0x7f54a010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.055 7f54a571c700 1 --2- 172.21.15.32:0/1723135915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a010da00 0x7f54a010de00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5498006b80 tx=0x7f5498009b10).stop 2021-04-09T18:13:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.056 7f54a571c700 1 -- 172.21.15.32:0/1723135915 shutdown_connections 2021-04-09T18:13:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.056 7f54a571c700 1 --2- 172.21.15.32:0/1723135915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a010da00 0x7f54a010de00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.056 7f54a571c700 1 --2- 172.21.15.32:0/1723135915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a00d7ca0 0x7f54a00d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.056 7f54a571c700 1 -- 172.21.15.32:0/1723135915 >> 172.21.15.32:0/1723135915 conn(0x7f54a00ce5f0 msgr2=0x7f54a00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:08.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.056 7f54a571c700 1 -- 172.21.15.32:0/1723135915 shutdown_connections 2021-04-09T18:13:08.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.056 7f54a571c700 1 -- 172.21.15.32:0/1723135915 wait complete. 2021-04-09T18:13:08.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.076 7f54a571c700 1 Processor -- start 2021-04-09T18:13:08.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.076 7f54a571c700 1 -- start start 2021-04-09T18:13:08.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f54a571c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a01a5210 0x7f54a0185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f54a571c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54a0185a70 0x7f54a00acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f54a571c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a0185e70 0x7f54a00ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f54a571c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a018f7b0 con 0x7f54a01a5210 2021-04-09T18:13:08.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f54a571c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00ad7f0 con 0x7f54a0185a70 2021-04-09T18:13:08.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f54a571c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00ada10 con 0x7f54a0185e70 2021-04-09T18:13:08.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a01a5210 0x7f54a0185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a01a5210 0x7f54a0185450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42052/0 (socket says 172.21.15.32:42052) 2021-04-09T18:13:08.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549f2bf700 1 -- 172.21.15.32:0/3711276406 learned_addr learned my addr 172.21.15.32:0/3711276406 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:08.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549fac0700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a0185e70 0x7f54a00ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549eabe700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54a0185a70 0x7f54a00acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549f2bf700 1 -- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a0185e70 msgr2=0x7f54a00ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549f2bf700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a0185e70 0x7f54a00ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549f2bf700 1 -- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54a0185a70 msgr2=0x7f54a00acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549f2bf700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54a0185a70 0x7f54a00acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549f2bf700 1 -- 172.21.15.32:0/3711276406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a0179610 con 0x7f54a01a5210 2021-04-09T18:13:08.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549fac0700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a0185e70 0x7f54a00ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:08.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549eabe700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54a0185a70 0x7f54a00acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f549f2bf700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a01a5210 0x7f54a0185450 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f549000b5b0 tx=0x7f549000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.077 7f5487fff700 1 -- 172.21.15.32:0/3711276406 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f549000bd40 con 0x7f54a01a5210 2021-04-09T18:13:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.078 7f5487fff700 1 -- 172.21.15.32:0/3711276406 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5490006240 con 0x7f54a01a5210 2021-04-09T18:13:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.078 7f54a571c700 1 -- 172.21.15.32:0/3711276406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a0179610 con 0x7f54a01a5210 2021-04-09T18:13:08.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.078 7f5487fff700 1 -- 172.21.15.32:0/3711276406 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5490005690 con 0x7f54a01a5210 2021-04-09T18:13:08.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.078 7f54a571c700 1 -- 172.21.15.32:0/3711276406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a0179610 con 0x7f54a01a5210 2021-04-09T18:13:08.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.078 7f5487fff700 1 -- 172.21.15.32:0/3711276406 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f54900054a0 con 0x7f54a01a5210 2021-04-09T18:13:08.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.078 7f5487fff700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54800188d0 0x7f548001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.078 7f5487fff700 1 -- 172.21.15.32:0/3711276406 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5490032a30 con 0x7f54a01a5210 2021-04-09T18:13:08.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.079 7f549eabe700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54800188d0 0x7f548001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.079 7f549eabe700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54800188d0 0x7f548001add0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:08.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.099 7f54a571c700 1 -- 172.21.15.32:0/3711276406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a018f7b0 con 0x7f54a01a5210 2021-04-09T18:13:08.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.102 7f5487fff700 1 -- 172.21.15.32:0/3711276406 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5490000c60 con 0x7f54a01a5210 2021-04-09T18:13:08.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.290 7f54a571c700 1 -- 172.21.15.32:0/3711276406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54a018f7b0 con 0x7f54a01a5210 2021-04-09T18:13:08.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.290 7f5487fff700 1 -- 172.21.15.32:0/3711276406 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f54900054a0 con 0x7f54a01a5210 2021-04-09T18:13:08.326 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:08.327 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:34.773743","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:08.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.292 7f54a571c700 1 -- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54800188d0 msgr2=0x7f548001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.292 7f54a571c700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54800188d0 0x7f548001add0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.292 7f54a571c700 1 -- 172.21.15.32:0/3711276406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a01a5210 msgr2=0x7f54a0185450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.292 7f54a571c700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a01a5210 0x7f54a0185450 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f549000b5b0 tx=0x7f549000b950).stop 2021-04-09T18:13:08.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.293 7f54a571c700 1 -- 172.21.15.32:0/3711276406 shutdown_connections 2021-04-09T18:13:08.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.293 7f54a571c700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54800188d0 0x7f548001add0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.293 7f54a571c700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54a0185e70 0x7f54a00ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.293 7f54a571c700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54a0185a70 0x7f54a00acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.293 7f54a571c700 1 --2- 172.21.15.32:0/3711276406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54a01a5210 0x7f54a0185450 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.293 7f54a571c700 1 -- 172.21.15.32:0/3711276406 >> 172.21.15.32:0/3711276406 conn(0x7f54a01a99c0 msgr2=0x7f54a017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:08.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.293 7f54a571c700 1 -- 172.21.15.32:0/3711276406 shutdown_connections 2021-04-09T18:13:08.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.293 7f54a571c700 1 -- 172.21.15.32:0/3711276406 wait complete. 2021-04-09T18:13:08.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:12:32.136283', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:12:32.831693', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:13:08.425 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:13:08.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.564 7fc8c35d7700 1 Processor -- start 2021-04-09T18:13:08.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.564 7fc8c35d7700 1 -- start start 2021-04-09T18:13:08.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8c35d7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc8bc10d010 0x7fc8bc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8c35d7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc10d480 con 0x7fc8bc0d1da0 2021-04-09T18:13:08.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8c35d7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc10e1f0 con 0x7fc8bc0d8490 2021-04-09T18:13:08.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8c35d7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8bc0d8d30 con 0x7fc8bc10d010 2021-04-09T18:13:08.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8c0bf2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc8bc0d1da0 0x7fc8bc0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51654/0 (socket says 172.21.15.32:51654) 2021-04-09T18:13:08.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8c0bf2700 1 -- 172.21.15.32:0/1049522072 learned_addr learned my addr 172.21.15.32:0/1049522072 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:08.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8c13f3700 1 --2- 172.21.15.32:0/1049522072 >> v2:172.21.15.86:3300/0 conn(0x7fc8bc10d010 0x7fc8bc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8b3fff700 1 -- 172.21.15.32:0/1049522072 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261417603 0 0) 0x7fc8ac0009b0 con 0x7fc8bc0d1da0 2021-04-09T18:13:08.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8b3fff700 1 -- 172.21.15.32:0/1049522072 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a8002fe0 con 0x7fc8bc0d1da0 2021-04-09T18:13:08.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.565 7fc8b3fff700 1 -- 172.21.15.32:0/1049522072 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898141344 0 0) 0x7fc8b4000d30 con 0x7fc8bc0d8490 2021-04-09T18:13:08.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8b3fff700 1 -- 172.21.15.32:0/1049522072 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a8003210 con 0x7fc8bc0d8490 2021-04-09T18:13:08.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8c13f3700 1 -- 172.21.15.32:0/1049522072 >> v1:172.21.15.86:6789/0 conn(0x7fc8bc0d8490 legacy=0x7fc8bc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8c13f3700 1 -- 172.21.15.32:0/1049522072 >> v1:172.21.15.32:6789/0 conn(0x7fc8bc0d1da0 legacy=0x7fc8bc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8c13f3700 1 -- 172.21.15.32:0/1049522072 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc0d9220 con 0x7fc8bc10d010 2021-04-09T18:13:08.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8c13f3700 1 --2- 172.21.15.32:0/1049522072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8bc10d010 0x7fc8bc0d1780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc8b8006650 tx=0x7fc8b8009a80).ready entity=mon.1 client_cookie=c45e6acd13f8340 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:08.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8b3fff700 1 -- 172.21.15.32:0/1049522072 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b80009b0 con 0x7fc8bc10d010 2021-04-09T18:13:08.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8b3fff700 1 -- 172.21.15.32:0/1049522072 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b800c0c0 con 0x7fc8bc10d010 2021-04-09T18:13:08.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8b3fff700 1 -- 172.21.15.32:0/1049522072 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b800c510 con 0x7fc8bc10d010 2021-04-09T18:13:08.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8c35d7700 1 -- 172.21.15.32:0/1049522072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8bc10d010 msgr2=0x7fc8bc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.566 7fc8c35d7700 1 --2- 172.21.15.32:0/1049522072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8bc10d010 0x7fc8bc0d1780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc8b8006650 tx=0x7fc8b8009a80).stop 2021-04-09T18:13:08.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.567 7fc8c35d7700 1 -- 172.21.15.32:0/1049522072 shutdown_connections 2021-04-09T18:13:08.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.567 7fc8c35d7700 1 --2- 172.21.15.32:0/1049522072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8bc10d010 0x7fc8bc0d1780 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.567 7fc8c35d7700 1 -- 172.21.15.32:0/1049522072 >> 172.21.15.32:0/1049522072 conn(0x7fc8bc0ce5f0 msgr2=0x7fc8bc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:08.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.567 7fc8c35d7700 1 -- 172.21.15.32:0/1049522072 shutdown_connections 2021-04-09T18:13:08.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.567 7fc8c35d7700 1 -- 172.21.15.32:0/1049522072 wait complete. 2021-04-09T18:13:08.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.587 7fc8c35d7700 1 Processor -- start 2021-04-09T18:13:08.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.587 7fc8c35d7700 1 -- start start 2021-04-09T18:13:08.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c35d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8bc0b8de0 0x7fc8bc0c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c35d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8bc0c0640 0x7fc8bc1936d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c35d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8bc193cf0 0x7fc8bc1940f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c35d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc0c0a40 con 0x7fc8bc0c0640 2021-04-09T18:13:08.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c35d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc0b89a0 con 0x7fc8bc193cf0 2021-04-09T18:13:08.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c35d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc1b7b70 con 0x7fc8bc0b8de0 2021-04-09T18:13:08.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c0bf2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8bc0c0640 0x7fc8bc1936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c0bf2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8bc0c0640 0x7fc8bc1936d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42068/0 (socket says 172.21.15.32:42068) 2021-04-09T18:13:08.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c0bf2700 1 -- 172.21.15.32:0/3381993379 learned_addr learned my addr 172.21.15.32:0/3381993379 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:08.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c13f3700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8bc0b8de0 0x7fc8bc0c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c1bf4700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8bc193cf0 0x7fc8bc1940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c0bf2700 1 -- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8bc0b8de0 msgr2=0x7fc8bc0c0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c0bf2700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8bc0b8de0 0x7fc8bc0c0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c0bf2700 1 -- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8bc193cf0 msgr2=0x7fc8bc1940f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c0bf2700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8bc193cf0 0x7fc8bc1940f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c0bf2700 1 -- 172.21.15.32:0/3381993379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc0bb2d0 con 0x7fc8bc0c0640 2021-04-09T18:13:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c0bf2700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8bc0c0640 0x7fc8bc1936d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc8ac001c60 tx=0x7fc8ac00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8b1ffb700 1 -- 172.21.15.32:0/3381993379 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8ac00acc0 con 0x7fc8bc0c0640 2021-04-09T18:13:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8b1ffb700 1 -- 172.21.15.32:0/3381993379 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac004140 con 0x7fc8bc0c0640 2021-04-09T18:13:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8bc0bb2d0 con 0x7fc8bc0c0640 2021-04-09T18:13:08.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8b1ffb700 1 -- 172.21.15.32:0/3381993379 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8ac004620 con 0x7fc8bc0c0640 2021-04-09T18:13:08.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.588 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8bc0bb2d0 con 0x7fc8bc0c0640 2021-04-09T18:13:08.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.589 7fc8b1ffb700 1 -- 172.21.15.32:0/3381993379 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc8ac0047c0 con 0x7fc8bc0c0640 2021-04-09T18:13:08.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.589 7fc8b1ffb700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc89c018930 0x7fc89c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:08.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.589 7fc8b1ffb700 1 -- 172.21.15.32:0/3381993379 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc8ac0329e0 con 0x7fc8bc0c0640 2021-04-09T18:13:08.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.589 7fc8c13f3700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc89c018930 0x7fc89c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:08.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.590 7fc8c13f3700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc89c018930 0x7fc89c01ae10 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:08.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.609 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8bc1b7040 con 0x7fc8bc0c0640 2021-04-09T18:13:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.612 7fc8b1ffb700 1 -- 172.21.15.32:0/3381993379 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc8ac0045b0 con 0x7fc8bc0c0640 2021-04-09T18:13:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.793 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc8bc10d480 con 0x7fc8bc0c0640 2021-04-09T18:13:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.793 7fc8b1ffb700 1 -- 172.21.15.32:0/3381993379 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc8ac0047c0 con 0x7fc8bc0c0640 2021-04-09T18:13:08.872 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.795 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc89c018930 msgr2=0x7fc89c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc89c018930 0x7fc89c01ae10 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8bc0c0640 msgr2=0x7fc8bc1936d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:08.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8bc0c0640 0x7fc8bc1936d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc8ac001c60 tx=0x7fc8ac00a8d0).stop 2021-04-09T18:13:08.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 shutdown_connections 2021-04-09T18:13:08.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc89c018930 0x7fc89c01ae10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8bc0c0640 0x7fc8bc1936d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8bc193cf0 0x7fc8bc1940f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 --2- 172.21.15.32:0/3381993379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8bc0b8de0 0x7fc8bc0c0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:08.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 >> 172.21.15.32:0/3381993379 conn(0x7fc8bc092a00 msgr2=0x7fc8bc1b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:08.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 shutdown_connections 2021-04-09T18:13:08.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:08.796 7fc8c35d7700 1 -- 172.21.15.32:0/3381993379 wait complete. 2021-04-09T18:13:08.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:13:09.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.134 7fe4242d2700 1 Processor -- start 2021-04-09T18:13:09.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.134 7fe4242d2700 1 -- start start 2021-04-09T18:13:09.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4242d2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c10d010 0x7fe41c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4242d2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe41c0d8490 0x7fe41c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4242d2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41c10d480 con 0x7fe41c0d1da0 2021-04-09T18:13:09.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4242d2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe41c10e780 con 0x7fe41c0d8490 2021-04-09T18:13:09.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4242d2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0d8d30 con 0x7fe41c10d010 2021-04-09T18:13:09.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4228ef700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe41c0d8490 0x7fe41c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4228ef700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe41c0d8490 0x7fe41c10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42076/0 (socket says 172.21.15.32:42076) 2021-04-09T18:13:09.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4228ef700 1 -- 172.21.15.32:0/2475603591 learned_addr learned my addr 172.21.15.32:0/2475603591 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:09.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4220ee700 1 --2- 172.21.15.32:0/2475603591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c10d010 0x7fe41c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4228ef700 1 -- 172.21.15.32:0/2475603591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c10d010 msgr2=0x7fe41c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:09.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4228ef700 1 --2- 172.21.15.32:0/2475603591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c10d010 0x7fe41c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4228ef700 1 -- 172.21.15.32:0/2475603591 >> v1:172.21.15.86:6789/0 conn(0x7fe41c0d1da0 legacy=0x7fe41c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:09.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4228ef700 1 -- 172.21.15.32:0/2475603591 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0d9220 con 0x7fe41c0d8490 2021-04-09T18:13:09.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4220ee700 1 --2- 172.21.15.32:0/2475603591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c10d010 0x7fe41c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:09.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.135 7fe4228ef700 1 --2- 172.21.15.32:0/2475603591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0d8490 0x7fe41c10da00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe414006f00 tx=0x7fe414009f10).ready entity=mon.0 client_cookie=16fb55c6c18bf9c0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.136 7fe4210ec700 1 -- 172.21.15.32:0/2475603591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41400b3b0 con 0x7fe41c0d8490 2021-04-09T18:13:09.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.136 7fe4210ec700 1 -- 172.21.15.32:0/2475603591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41400b730 con 0x7fe41c0d8490 2021-04-09T18:13:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.136 7fe4210ec700 1 -- 172.21.15.32:0/2475603591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41400bc10 con 0x7fe41c0d8490 2021-04-09T18:13:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.136 7fe4242d2700 1 -- 172.21.15.32:0/2475603591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0d8490 msgr2=0x7fe41c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:09.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.136 7fe4242d2700 1 --2- 172.21.15.32:0/2475603591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0d8490 0x7fe41c10da00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fe414006f00 tx=0x7fe414009f10).stop 2021-04-09T18:13:09.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.137 7fe4242d2700 1 -- 172.21.15.32:0/2475603591 shutdown_connections 2021-04-09T18:13:09.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.137 7fe4242d2700 1 --2- 172.21.15.32:0/2475603591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0d8490 0x7fe41c10da00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.137 7fe4242d2700 1 --2- 172.21.15.32:0/2475603591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c10d010 0x7fe41c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.137 7fe4242d2700 1 -- 172.21.15.32:0/2475603591 >> 172.21.15.32:0/2475603591 conn(0x7fe41c0ce5f0 msgr2=0x7fe41c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:09.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.137 7fe4242d2700 1 -- 172.21.15.32:0/2475603591 shutdown_connections 2021-04-09T18:13:09.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.137 7fe4242d2700 1 -- 172.21.15.32:0/2475603591 wait complete. 2021-04-09T18:13:09.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.157 7fe4242d2700 1 Processor -- start 2021-04-09T18:13:09.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.157 7fe4242d2700 1 -- start start 2021-04-09T18:13:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.157 7fe4242d2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c0b1e50 0x7fe41c0b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.157 7fe4242d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0b9fd0 0x7fe41c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.157 7fe4242d2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0b5510 0x7fe41c0b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.157 7fe4242d2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0b2250 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.157 7fe4242d2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0a1220 con 0x7fe41c0b5510 2021-04-09T18:13:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.157 7fe4242d2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0b6b30 con 0x7fe41c0b1e50 2021-04-09T18:13:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.157 7fe4218ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0b9fd0 0x7fe41c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4218ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0b9fd0 0x7fe41c0b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42080/0 (socket says 172.21.15.32:42080) 2021-04-09T18:13:09.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4218ed700 1 -- 172.21.15.32:0/963539491 learned_addr learned my addr 172.21.15.32:0/963539491 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:09.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4220ee700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c0b1e50 0x7fe41c0b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4228ef700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0b5510 0x7fe41c0b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4218ed700 1 -- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c0b1e50 msgr2=0x7fe41c0b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:09.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4218ed700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c0b1e50 0x7fe41c0b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4218ed700 1 -- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0b5510 msgr2=0x7fe41c0b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:09.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4218ed700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0b5510 0x7fe41c0b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4218ed700 1 -- 172.21.15.32:0/963539491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0b5f30 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4218ed700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0b9fd0 0x7fe41c0b4ef0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe40c00b5b0 tx=0x7fe40c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:09.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe412ffd700 1 -- 172.21.15.32:0/963539491 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe40c00bd40 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe412ffd700 1 -- 172.21.15.32:0/963539491 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40c006240 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4242d2700 1 -- 172.21.15.32:0/963539491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c0b5f30 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe412ffd700 1 -- 172.21.15.32:0/963539491 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe40c005690 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.158 7fe4242d2700 1 -- 172.21.15.32:0/963539491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c0b5f30 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.159 7fe412ffd700 1 -- 172.21.15.32:0/963539491 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe40c0054a0 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.159 7fe412ffd700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe404018930 0x7fe40401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.159 7fe412ffd700 1 -- 172.21.15.32:0/963539491 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe40c032a00 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.159 7fe4220ee700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe404018930 0x7fe40401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.159 7fe4220ee700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe404018930 0x7fe40401adc0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:09.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.179 7fe4242d2700 1 -- 172.21.15.32:0/963539491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41c1b7040 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.182 7fe412ffd700 1 -- 172.21.15.32:0/963539491 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe40c0009b0 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.362 7fe4242d2700 1 -- 172.21.15.32:0/963539491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe41c1b7040 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.362 7fe412ffd700 1 -- 172.21.15.32:0/963539491 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe40c0054a0 con 0x7fe41c0b9fd0 2021-04-09T18:13:09.495 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:09.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 -- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe404018930 msgr2=0x7fe40401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:09.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe404018930 0x7fe40401adc0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 -- 172.21.15.32:0/963539491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0b9fd0 msgr2=0x7fe41c0b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:09.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0b9fd0 0x7fe41c0b4ef0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe40c00b5b0 tx=0x7fe40c00b950).stop 2021-04-09T18:13:09.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 -- 172.21.15.32:0/963539491 shutdown_connections 2021-04-09T18:13:09.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe404018930 0x7fe40401adc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe41c0b9fd0 0x7fe41c0b4ef0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41c0b5510 0x7fe41c0b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 --2- 172.21.15.32:0/963539491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41c0b1e50 0x7fe41c0b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 -- 172.21.15.32:0/963539491 >> 172.21.15.32:0/963539491 conn(0x7fe41c0b27e0 msgr2=0x7fe41c1b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:09.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 -- 172.21.15.32:0/963539491 shutdown_connections 2021-04-09T18:13:09.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.365 7fe4242d2700 1 -- 172.21.15.32:0/963539491 wait complete. 2021-04-09T18:13:09.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:13:09.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.781 7f9b39b2c700 1 Processor -- start 2021-04-09T18:13:09.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.781 7f9b39b2c700 1 -- start start 2021-04-09T18:13:09.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.781 7f9b39b2c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340d7ca0 0x7f9b340d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b39b2c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9b3410da00 0x7f9b3410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b39b2c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3410d480 con 0x7f9b3410d010 2021-04-09T18:13:09.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b39b2c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b340d9220 con 0x7f9b3410da00 2021-04-09T18:13:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b39b2c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340d86c0 con 0x7f9b340d7ca0 2021-04-09T18:13:09.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b337fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9b3410d010 0x7f9b340d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51678/0 (socket says 172.21.15.32:51678) 2021-04-09T18:13:09.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b337fe700 1 -- 172.21.15.32:0/1385111201 learned_addr learned my addr 172.21.15.32:0/1385111201 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:09.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b32ffd700 1 --2- 172.21.15.32:0/1385111201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340d7ca0 0x7f9b340d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b33fff700 1 --2- 172.21.15.32:0/1385111201 >> v2:172.21.15.86:3300/0 conn(0x7f9b3410da00 0x7f9b3410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b327fc700 1 -- 172.21.15.32:0/1385111201 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206455150 0 0) 0x7f9b24000ce0 con 0x7f9b3410d010 2021-04-09T18:13:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b327fc700 1 -- 172.21.15.32:0/1385111201 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b1c002fe0 con 0x7f9b3410d010 2021-04-09T18:13:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b32ffd700 1 -- 172.21.15.32:0/1385111201 >> v2:172.21.15.86:3300/0 conn(0x7f9b3410da00 msgr2=0x7f9b3410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b32ffd700 1 --2- 172.21.15.32:0/1385111201 >> v2:172.21.15.86:3300/0 conn(0x7f9b3410da00 0x7f9b3410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b32ffd700 1 -- 172.21.15.32:0/1385111201 >> v1:172.21.15.32:6789/0 conn(0x7f9b3410d010 legacy=0x7f9b340d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b32ffd700 1 -- 172.21.15.32:0/1385111201 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b340d1f70 con 0x7f9b340d7ca0 2021-04-09T18:13:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b33fff700 1 --2- 172.21.15.32:0/1385111201 >> v2:172.21.15.86:3300/0 conn(0x7f9b3410da00 0x7f9b3410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b32ffd700 1 --2- 172.21.15.32:0/1385111201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340d7ca0 0x7f9b340d80a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9b28006650 tx=0x7f9b28009a80).ready entity=mon.2 client_cookie=abcb9e9adb07e737 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.782 7f9b327fc700 1 -- 172.21.15.32:0/1385111201 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b280009b0 con 0x7f9b340d7ca0 2021-04-09T18:13:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b327fc700 1 -- 172.21.15.32:0/1385111201 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2800c0c0 con 0x7f9b340d7ca0 2021-04-09T18:13:09.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b327fc700 1 -- 172.21.15.32:0/1385111201 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2800c510 con 0x7f9b340d7ca0 2021-04-09T18:13:09.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b39b2c700 1 -- 172.21.15.32:0/1385111201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340d7ca0 msgr2=0x7f9b340d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:09.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b39b2c700 1 --2- 172.21.15.32:0/1385111201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340d7ca0 0x7f9b340d80a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9b28006650 tx=0x7f9b28009a80).stop 2021-04-09T18:13:09.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b39b2c700 1 -- 172.21.15.32:0/1385111201 shutdown_connections 2021-04-09T18:13:09.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b39b2c700 1 --2- 172.21.15.32:0/1385111201 >> v2:172.21.15.86:3300/0 conn(0x7f9b3410da00 0x7f9b3410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b39b2c700 1 --2- 172.21.15.32:0/1385111201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340d7ca0 0x7f9b340d80a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b39b2c700 1 -- 172.21.15.32:0/1385111201 >> 172.21.15.32:0/1385111201 conn(0x7f9b340ce5f0 msgr2=0x7f9b340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:09.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b39b2c700 1 -- 172.21.15.32:0/1385111201 shutdown_connections 2021-04-09T18:13:09.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.783 7f9b39b2c700 1 -- 172.21.15.32:0/1385111201 wait complete. 2021-04-09T18:13:09.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.803 7f9b39b2c700 1 Processor -- start 2021-04-09T18:13:09.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b39b2c700 1 -- start start 2021-04-09T18:13:09.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b39b2c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b340a0cb0 0x7f9b34092550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b39b2c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340932e0 0x7f9b3408a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b39b2c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340936e0 0x7f9b3408ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b39b2c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340b34f0 con 0x7f9b340936e0 2021-04-09T18:13:09.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b39b2c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b34092d60 con 0x7f9b340a0cb0 2021-04-09T18:13:09.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b39b2c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340900a0 con 0x7f9b340932e0 2021-04-09T18:13:09.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b33fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340936e0 0x7f9b3408ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b33fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340936e0 0x7f9b3408ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42094/0 (socket says 172.21.15.32:42094) 2021-04-09T18:13:09.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b33fff700 1 -- 172.21.15.32:0/2728413853 learned_addr learned my addr 172.21.15.32:0/2728413853 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:09.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b32ffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340932e0 0x7f9b3408a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.804 7f9b337fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b340a0cb0 0x7f9b34092550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b33fff700 1 -- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340932e0 msgr2=0x7f9b3408a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:09.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b33fff700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340932e0 0x7f9b3408a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b33fff700 1 -- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b340a0cb0 msgr2=0x7f9b34092550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b33fff700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b340a0cb0 0x7f9b34092550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b33fff700 1 -- 172.21.15.32:0/2728413853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b340a10b0 con 0x7f9b340936e0 2021-04-09T18:13:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b337fe700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b340a0cb0 0x7f9b34092550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:09.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b33fff700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340936e0 0x7f9b3408ac40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f9b2c00b910 tx=0x7f9b2c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:09.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b1bfff700 1 -- 172.21.15.32:0/2728413853 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2c006aa0 con 0x7f9b340936e0 2021-04-09T18:13:09.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b1bfff700 1 -- 172.21.15.32:0/2728413853 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c003030 con 0x7f9b340936e0 2021-04-09T18:13:09.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b340a10b0 con 0x7f9b340936e0 2021-04-09T18:13:09.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b1bfff700 1 -- 172.21.15.32:0/2728413853 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2c001470 con 0x7f9b340936e0 2021-04-09T18:13:09.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.805 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b340a10b0 con 0x7f9b340936e0 2021-04-09T18:13:09.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.806 7f9b1bfff700 1 -- 172.21.15.32:0/2728413853 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9b2c0011f0 con 0x7f9b340936e0 2021-04-09T18:13:09.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.806 7f9b1bfff700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b140188d0 0x7f9b1401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:09.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.806 7f9b1bfff700 1 -- 172.21.15.32:0/2728413853 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9b2c032d60 con 0x7f9b340936e0 2021-04-09T18:13:09.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.806 7f9b337fe700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b140188d0 0x7f9b1401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:09.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.806 7f9b337fe700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b140188d0 0x7f9b1401add0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:09.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.826 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3410d480 con 0x7f9b340936e0 2021-04-09T18:13:09.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:09.829 7f9b1bfff700 1 -- 172.21.15.32:0/2728413853 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9b2c00bdf0 con 0x7f9b340936e0 2021-04-09T18:13:10.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.007 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f9b3410d480 con 0x7f9b340936e0 2021-04-09T18:13:10.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.007 7f9b1bfff700 1 -- 172.21.15.32:0/2728413853 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9b2c000df0 con 0x7f9b340936e0 2021-04-09T18:13:10.030 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.009 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b140188d0 msgr2=0x7f9b1401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.009 7f9b39b2c700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b140188d0 0x7f9b1401add0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.009 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340936e0 msgr2=0x7f9b3408ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.009 7f9b39b2c700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340936e0 0x7f9b3408ac40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f9b2c00b910 tx=0x7f9b2c00bcb0).stop 2021-04-09T18:13:10.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.010 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 shutdown_connections 2021-04-09T18:13:10.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.010 7f9b39b2c700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9b140188d0 0x7f9b1401add0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.010 7f9b39b2c700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b340936e0 0x7f9b3408ac40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.010 7f9b39b2c700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b340932e0 0x7f9b3408a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.010 7f9b39b2c700 1 --2- 172.21.15.32:0/2728413853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b340a0cb0 0x7f9b34092550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.010 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 >> 172.21.15.32:0/2728413853 conn(0x7f9b340b2fa0 msgr2=0x7f9b340a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:10.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.010 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 shutdown_connections 2021-04-09T18:13:10.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.010 7f9b39b2c700 1 -- 172.21.15.32:0/2728413853 wait complete. 2021-04-09T18:13:10.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.226 7f0e68f6e700 1 Processor -- start 2021-04-09T18:13:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.226 7f0e68f6e700 1 -- start start 2021-04-09T18:13:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e68f6e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e640d7ca0 0x7f0e640d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e68f6e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0e6410da00 0x7f0e6410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e68f6e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6410d480 con 0x7f0e6410d010 2021-04-09T18:13:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e68f6e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e640d9220 con 0x7f0e6410da00 2021-04-09T18:13:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e68f6e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640d86c0 con 0x7f0e640d7ca0 2021-04-09T18:13:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e63314700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0e6410da00 0x7f0e6410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e62b13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0e6410d010 0x7f0e640d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51690/0 (socket says 172.21.15.32:51690) 2021-04-09T18:13:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e62b13700 1 -- 172.21.15.32:0/4015544384 learned_addr learned my addr 172.21.15.32:0/4015544384 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e62312700 1 --2- 172.21.15.32:0/4015544384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e640d7ca0 0x7f0e640d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e61b11700 1 -- 172.21.15.32:0/4015544384 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260961888 0 0) 0x7f0e54001060 con 0x7f0e6410d010 2021-04-09T18:13:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e63314700 1 -- 172.21.15.32:0/4015544384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e640d7ca0 msgr2=0x7f0e640d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:10.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e63314700 1 --2- 172.21.15.32:0/4015544384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e640d7ca0 0x7f0e640d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e63314700 1 -- 172.21.15.32:0/4015544384 >> v1:172.21.15.32:6789/0 conn(0x7f0e6410d010 legacy=0x7f0e640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.227 7f0e63314700 1 -- 172.21.15.32:0/4015544384 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640d1f70 con 0x7f0e6410da00 2021-04-09T18:13:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.228 7f0e63314700 1 --2- 172.21.15.32:0/4015544384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e6410da00 0x7f0e6410de00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f0e5c006ab0 tx=0x7f0e5c009b40).ready entity=mon.0 client_cookie=72c9fb1009dd809d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.228 7f0e61b11700 1 -- 172.21.15.32:0/4015544384 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e5c0022e0 con 0x7f0e6410da00 2021-04-09T18:13:10.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.228 7f0e61b11700 1 -- 172.21.15.32:0/4015544384 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c002660 con 0x7f0e6410da00 2021-04-09T18:13:10.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.228 7f0e61b11700 1 -- 172.21.15.32:0/4015544384 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e5c002b40 con 0x7f0e6410da00 2021-04-09T18:13:10.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.228 7f0e68f6e700 1 -- 172.21.15.32:0/4015544384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e6410da00 msgr2=0x7f0e6410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.228 7f0e68f6e700 1 --2- 172.21.15.32:0/4015544384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e6410da00 0x7f0e6410de00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f0e5c006ab0 tx=0x7f0e5c009b40).stop 2021-04-09T18:13:10.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.229 7f0e68f6e700 1 -- 172.21.15.32:0/4015544384 shutdown_connections 2021-04-09T18:13:10.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.229 7f0e68f6e700 1 --2- 172.21.15.32:0/4015544384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e6410da00 0x7f0e6410de00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.229 7f0e68f6e700 1 --2- 172.21.15.32:0/4015544384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e640d7ca0 0x7f0e640d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.229 7f0e68f6e700 1 -- 172.21.15.32:0/4015544384 >> 172.21.15.32:0/4015544384 conn(0x7f0e640ce5f0 msgr2=0x7f0e640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:10.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.229 7f0e68f6e700 1 -- 172.21.15.32:0/4015544384 shutdown_connections 2021-04-09T18:13:10.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.229 7f0e68f6e700 1 -- 172.21.15.32:0/4015544384 wait complete. 2021-04-09T18:13:10.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.249 7f0e68f6e700 1 Processor -- start 2021-04-09T18:13:10.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.249 7f0e68f6e700 1 -- start start 2021-04-09T18:13:10.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.249 7f0e68f6e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e641ac380 0x7f0e6418a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.249 7f0e68f6e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e641ac780 0x7f0e6418a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.249 7f0e68f6e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e64183520 0x7f0e64183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.249 7f0e68f6e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e641ace10 con 0x7f0e641ac380 2021-04-09T18:13:10.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.249 7f0e68f6e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e64189cf0 con 0x7f0e64183520 2021-04-09T18:13:10.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.249 7f0e68f6e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e64183f40 con 0x7f0e641ac780 2021-04-09T18:13:10.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.249 7f0e62b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e641ac380 0x7f0e6418a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e62b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e641ac380 0x7f0e6418a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42106/0 (socket says 172.21.15.32:42106) 2021-04-09T18:13:10.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e62b13700 1 -- 172.21.15.32:0/1705016685 learned_addr learned my addr 172.21.15.32:0/1705016685 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:10.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e62312700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e641ac780 0x7f0e6418a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e63314700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e64183520 0x7f0e64183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e62b13700 1 -- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e641ac780 msgr2=0x7f0e6418a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e62b13700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e641ac780 0x7f0e6418a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e62b13700 1 -- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e64183520 msgr2=0x7f0e64183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e62b13700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e64183520 0x7f0e64183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e62b13700 1 -- 172.21.15.32:0/1705016685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e64184160 con 0x7f0e641ac380 2021-04-09T18:13:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e62b13700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e641ac380 0x7f0e6418a230 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0e54009980 tx=0x7f0e5400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:10.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e537fe700 1 -- 172.21.15.32:0/1705016685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e54006220 con 0x7f0e641ac380 2021-04-09T18:13:10.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e537fe700 1 -- 172.21.15.32:0/1705016685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e540065a0 con 0x7f0e641ac380 2021-04-09T18:13:10.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e64184160 con 0x7f0e641ac380 2021-04-09T18:13:10.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e537fe700 1 -- 172.21.15.32:0/1705016685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e54004910 con 0x7f0e641ac380 2021-04-09T18:13:10.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.250 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e64184160 con 0x7f0e641ac380 2021-04-09T18:13:10.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.251 7f0e537fe700 1 -- 172.21.15.32:0/1705016685 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0e54004720 con 0x7f0e641ac380 2021-04-09T18:13:10.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.251 7f0e537fe700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0e48018930 0x7f0e4801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.251 7f0e537fe700 1 -- 172.21.15.32:0/1705016685 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0e54032dd0 con 0x7f0e641ac380 2021-04-09T18:13:10.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.251 7f0e62312700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0e48018930 0x7f0e4801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.251 7f0e62312700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0e48018930 0x7f0e4801adc0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.271 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e64182d10 con 0x7f0e641ac380 2021-04-09T18:13:10.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.275 7f0e537fe700 1 -- 172.21.15.32:0/1705016685 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0e5400be10 con 0x7f0e641ac380 2021-04-09T18:13:10.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.465 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e64182d10 con 0x7f0e641ac380 2021-04-09T18:13:10.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.466 7f0e537fe700 1 -- 172.21.15.32:0/1705016685 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f0e54004720 con 0x7f0e641ac380 2021-04-09T18:13:10.498 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:10.499 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:34.773743","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:10.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0e48018930 msgr2=0x7f0e4801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0e48018930 0x7f0e4801adc0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e641ac380 msgr2=0x7f0e6418a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e641ac380 0x7f0e6418a230 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0e54009980 tx=0x7f0e5400bcd0).stop 2021-04-09T18:13:10.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 shutdown_connections 2021-04-09T18:13:10.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0e48018930 0x7f0e4801adc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e64183520 0x7f0e64183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e641ac780 0x7f0e6418a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 --2- 172.21.15.32:0/1705016685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e641ac380 0x7f0e6418a230 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 >> 172.21.15.32:0/1705016685 conn(0x7f0e6419b890 msgr2=0x7f0e641a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:10.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 shutdown_connections 2021-04-09T18:13:10.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.468 7f0e68f6e700 1 -- 172.21.15.32:0/1705016685 wait complete. 2021-04-09T18:13:10.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:12:32.136283', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:12:32.831693', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:13:10.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:13:10.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.714 7f74b51a0700 1 Processor -- start 2021-04-09T18:13:10.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.714 7f74b51a0700 1 -- start start 2021-04-09T18:13:10.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74b51a0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f74b00d7ca0 0x7f74b00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74b51a0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b010d480 con 0x7f74b010da00 2021-04-09T18:13:10.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74b51a0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00d86c0 con 0x7f74b010d010 2021-04-09T18:13:10.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74b51a0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b00d9220 con 0x7f74b00d7ca0 2021-04-09T18:13:10.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74af59e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f74b010da00 0x7f74b010de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51706/0 (socket says 172.21.15.32:51706) 2021-04-09T18:13:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74af59e700 1 -- 172.21.15.32:0/2690000026 learned_addr learned my addr 172.21.15.32:0/2690000026 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74ae59c700 1 --2- 172.21.15.32:0/2690000026 >> v2:172.21.15.86:3300/0 conn(0x7f74b00d7ca0 0x7f74b00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74add9b700 1 -- 172.21.15.32:0/2690000026 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870843847 0 0) 0x7f74a8000d30 con 0x7f74b010da00 2021-04-09T18:13:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74add9b700 1 -- 172.21.15.32:0/2690000026 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7494002fe0 con 0x7f74b010da00 2021-04-09T18:13:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74add9b700 1 -- 172.21.15.32:0/2690000026 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2121680522 0 0) 0x7f74a8001460 con 0x7f74b010da00 2021-04-09T18:13:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74ae59c700 1 -- 172.21.15.32:0/2690000026 >> v1:172.21.15.86:6789/0 conn(0x7f74b010d010 legacy=0x7f74b00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74ae59c700 1 -- 172.21.15.32:0/2690000026 >> v1:172.21.15.32:6789/0 conn(0x7f74b010da00 legacy=0x7f74b010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74ae59c700 1 -- 172.21.15.32:0/2690000026 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b0112100 con 0x7f74b00d7ca0 2021-04-09T18:13:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.715 7f74ae59c700 1 --2- 172.21.15.32:0/2690000026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b00d7ca0 0x7f74b00d80a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f749c006650 tx=0x7f749c009a00).ready entity=mon.1 client_cookie=881841f3c0c5856 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.716 7f74add9b700 1 -- 172.21.15.32:0/2690000026 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f749c009df0 con 0x7f74b00d7ca0 2021-04-09T18:13:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.716 7f74add9b700 1 -- 172.21.15.32:0/2690000026 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749c0009b0 con 0x7f74b00d7ca0 2021-04-09T18:13:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.716 7f74add9b700 1 -- 172.21.15.32:0/2690000026 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f749c00c2b0 con 0x7f74b00d7ca0 2021-04-09T18:13:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.716 7f74b51a0700 1 -- 172.21.15.32:0/2690000026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b00d7ca0 msgr2=0x7f74b00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.716 7f74b51a0700 1 --2- 172.21.15.32:0/2690000026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b00d7ca0 0x7f74b00d80a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f749c006650 tx=0x7f749c009a00).stop 2021-04-09T18:13:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.716 7f74b51a0700 1 -- 172.21.15.32:0/2690000026 shutdown_connections 2021-04-09T18:13:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.717 7f74b51a0700 1 --2- 172.21.15.32:0/2690000026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b00d7ca0 0x7f74b00d80a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.717 7f74b51a0700 1 -- 172.21.15.32:0/2690000026 >> 172.21.15.32:0/2690000026 conn(0x7f74b00ce5f0 msgr2=0x7f74b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.717 7f74b51a0700 1 -- 172.21.15.32:0/2690000026 shutdown_connections 2021-04-09T18:13:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.717 7f74b51a0700 1 -- 172.21.15.32:0/2690000026 wait complete. 2021-04-09T18:13:10.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74b51a0700 1 Processor -- start 2021-04-09T18:13:10.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74b51a0700 1 -- start start 2021-04-09T18:13:10.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74b51a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b017f2d0 0x7f74b00b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74b51a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74b00b5140 0x7f74b00be9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74b51a0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74b00befd0 0x7f74b0194b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74b51a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00b45a0 con 0x7f74b00b5140 2021-04-09T18:13:10.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74b51a0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00b5540 con 0x7f74b017f2d0 2021-04-09T18:13:10.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74b51a0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00b02a0 con 0x7f74b00befd0 2021-04-09T18:13:10.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74ae59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74b00b5140 0x7f74b00be9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74ae59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74b00b5140 0x7f74b00be9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42124/0 (socket says 172.21.15.32:42124) 2021-04-09T18:13:10.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74ae59c700 1 -- 172.21.15.32:0/2278781883 learned_addr learned my addr 172.21.15.32:0/2278781883 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:10.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74aed9d700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b017f2d0 0x7f74b00b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.737 7f74af59e700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74b00befd0 0x7f74b0194b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74ae59c700 1 -- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74b00befd0 msgr2=0x7f74b0194b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74ae59c700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74b00befd0 0x7f74b0194b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74ae59c700 1 -- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b017f2d0 msgr2=0x7f74b00b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74ae59c700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b017f2d0 0x7f74b00b4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74ae59c700 1 -- 172.21.15.32:0/2278781883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b00bf3d0 con 0x7f74b00b5140 2021-04-09T18:13:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74aed9d700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b017f2d0 0x7f74b00b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74ae59c700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74b00b5140 0x7f74b00be9b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f749c0096a0 tx=0x7f749c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74a77fe700 1 -- 172.21.15.32:0/2278781883 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f749c00bd80 con 0x7f74b00b5140 2021-04-09T18:13:10.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74a77fe700 1 -- 172.21.15.32:0/2278781883 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749c006030 con 0x7f74b00b5140 2021-04-09T18:13:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b00bf3d0 con 0x7f74b00b5140 2021-04-09T18:13:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74a77fe700 1 -- 172.21.15.32:0/2278781883 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f749c005560 con 0x7f74b00b5140 2021-04-09T18:13:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.738 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b00bf3d0 con 0x7f74b00b5140 2021-04-09T18:13:10.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.739 7f74a77fe700 1 -- 172.21.15.32:0/2278781883 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f749c0052e0 con 0x7f74b00b5140 2021-04-09T18:13:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.739 7f74a77fe700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f748c018930 0x7f748c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.739 7f74a77fe700 1 -- 172.21.15.32:0/2278781883 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f749c0329f0 con 0x7f74b00b5140 2021-04-09T18:13:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.739 7f74aed9d700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f748c018930 0x7f748c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.739 7f74aed9d700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f748c018930 0x7f748c01ae10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.759 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b010d480 con 0x7f74b00b5140 2021-04-09T18:13:10.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.762 7f74a77fe700 1 -- 172.21.15.32:0/2278781883 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f749c0009c0 con 0x7f74b00b5140 2021-04-09T18:13:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.943 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f74b010d480 con 0x7f74b00b5140 2021-04-09T18:13:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.943 7f74a77fe700 1 -- 172.21.15.32:0/2278781883 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f749c006030 con 0x7f74b00b5140 2021-04-09T18:13:10.965 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:10.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f748c018930 msgr2=0x7f748c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f748c018930 0x7f748c01ae10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74b00b5140 msgr2=0x7f74b00be9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:10.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74b00b5140 0x7f74b00be9b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f749c0096a0 tx=0x7f749c00b990).stop 2021-04-09T18:13:10.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 shutdown_connections 2021-04-09T18:13:10.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f748c018930 0x7f748c01ae10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74b00b5140 0x7f74b00be9b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74b00befd0 0x7f74b0194b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 --2- 172.21.15.32:0/2278781883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b017f2d0 0x7f74b00b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 >> 172.21.15.32:0/2278781883 conn(0x7f74b01b9710 msgr2=0x7f74b01b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 shutdown_connections 2021-04-09T18:13:10.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:10.946 7f74b51a0700 1 -- 172.21.15.32:0/2278781883 wait complete. 2021-04-09T18:13:10.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:13:11.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.129 7fecc1aea700 1 Processor -- start 2021-04-09T18:13:11.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.129 7fecc1aea700 1 -- start start 2021-04-09T18:13:11.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecc1aea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc10d010 0x7fecbc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecc1aea700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fecbc0d8490 0x7fecbc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecc1aea700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecbc10d480 con 0x7fecbc0d1da0 2021-04-09T18:13:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecc1aea700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecbc10e780 con 0x7fecbc0d8490 2021-04-09T18:13:11.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecc1aea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc0d8d30 con 0x7fecbc10d010 2021-04-09T18:13:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbaffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fecbc0d1da0 0x7fecbc0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51722/0 (socket says 172.21.15.32:51722) 2021-04-09T18:13:11.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbaffd700 1 -- 172.21.15.32:0/2509743693 learned_addr learned my addr 172.21.15.32:0/2509743693 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:11.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbb7fe700 1 --2- 172.21.15.32:0/2509743693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc10d010 0x7fecbc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbbfff700 1 --2- 172.21.15.32:0/2509743693 >> v2:172.21.15.86:3300/0 conn(0x7fecbc0d8490 0x7fecbc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecba7fc700 1 -- 172.21.15.32:0/2509743693 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905009435 0 0) 0x7fecb0000ce0 con 0x7fecbc0d1da0 2021-04-09T18:13:11.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecba7fc700 1 -- 172.21.15.32:0/2509743693 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca4002fe0 con 0x7fecbc0d1da0 2021-04-09T18:13:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbbfff700 1 -- 172.21.15.32:0/2509743693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc10d010 msgr2=0x7fecbc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecba7fc700 1 -- 172.21.15.32:0/2509743693 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2840735470 0 0) 0x7fecb00010e0 con 0x7fecbc0d1da0 2021-04-09T18:13:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbbfff700 1 --2- 172.21.15.32:0/2509743693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc10d010 0x7fecbc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbbfff700 1 -- 172.21.15.32:0/2509743693 >> v1:172.21.15.32:6789/0 conn(0x7fecbc0d1da0 legacy=0x7fecbc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbbfff700 1 -- 172.21.15.32:0/2509743693 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecbc0d9220 con 0x7fecbc0d8490 2021-04-09T18:13:11.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbb7fe700 1 --2- 172.21.15.32:0/2509743693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc10d010 0x7fecbc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.130 7fecbbfff700 1 --2- 172.21.15.32:0/2509743693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0d8490 0x7fecbc10da00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fecb4006e50 tx=0x7fecb4009e70).ready entity=mon.1 client_cookie=98a1c85be84d66b3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecba7fc700 1 -- 172.21.15.32:0/2509743693 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecb4006a20 con 0x7fecbc0d8490 2021-04-09T18:13:11.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecba7fc700 1 -- 172.21.15.32:0/2509743693 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb4006790 con 0x7fecbc0d8490 2021-04-09T18:13:11.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecba7fc700 1 -- 172.21.15.32:0/2509743693 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecb400b700 con 0x7fecbc0d8490 2021-04-09T18:13:11.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecc1aea700 1 -- 172.21.15.32:0/2509743693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0d8490 msgr2=0x7fecbc10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecc1aea700 1 --2- 172.21.15.32:0/2509743693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0d8490 0x7fecbc10da00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fecb4006e50 tx=0x7fecb4009e70).stop 2021-04-09T18:13:11.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecc1aea700 1 -- 172.21.15.32:0/2509743693 shutdown_connections 2021-04-09T18:13:11.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecc1aea700 1 --2- 172.21.15.32:0/2509743693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0d8490 0x7fecbc10da00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecc1aea700 1 --2- 172.21.15.32:0/2509743693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc10d010 0x7fecbc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecc1aea700 1 -- 172.21.15.32:0/2509743693 >> 172.21.15.32:0/2509743693 conn(0x7fecbc0ce5f0 msgr2=0x7fecbc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:11.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecc1aea700 1 -- 172.21.15.32:0/2509743693 shutdown_connections 2021-04-09T18:13:11.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.131 7fecc1aea700 1 -- 172.21.15.32:0/2509743693 wait complete. 2021-04-09T18:13:11.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.151 7fecc1aea700 1 Processor -- start 2021-04-09T18:13:11.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecc1aea700 1 -- start start 2021-04-09T18:13:11.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecc1aea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0c3c00 0x7fecbc09c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecc1aea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc095240 0x7fecbc1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecc1aea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecbc1a1270 0x7fecbc0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecc1aea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc09cda0 con 0x7fecbc1a1270 2021-04-09T18:13:11.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecc1aea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc0c3680 con 0x7fecbc0c3c00 2021-04-09T18:13:11.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecc1aea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecbc095640 con 0x7fecbc095240 2021-04-09T18:13:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecbbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecbc1a1270 0x7fecbc0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecbbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecbc1a1270 0x7fecbc0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42136/0 (socket says 172.21.15.32:42136) 2021-04-09T18:13:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.152 7fecbbfff700 1 -- 172.21.15.32:0/2020173915 learned_addr learned my addr 172.21.15.32:0/2020173915 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:11.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbaffd700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc095240 0x7fecbc1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbb7fe700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0c3c00 0x7fecbc09c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbbfff700 1 -- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc095240 msgr2=0x7fecbc1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbbfff700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc095240 0x7fecbc1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbbfff700 1 -- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0c3c00 msgr2=0x7fecbc09c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbbfff700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0c3c00 0x7fecbc09c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbbfff700 1 -- 172.21.15.32:0/2020173915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecbc1a1670 con 0x7fecbc1a1270 2021-04-09T18:13:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbaffd700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc095240 0x7fecbc1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:11.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbb7fe700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0c3c00 0x7fecbc09c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:11.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecbbfff700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecbc1a1270 0x7fecbc0b0cf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fecb4006e60 tx=0x7fecb400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:11.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7feca3fff700 1 -- 172.21.15.32:0/2020173915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecb4009130 con 0x7fecbc1a1270 2021-04-09T18:13:11.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7feca3fff700 1 -- 172.21.15.32:0/2020173915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb4000df0 con 0x7fecbc1a1270 2021-04-09T18:13:11.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecbc1a1670 con 0x7fecbc1a1270 2021-04-09T18:13:11.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7feca3fff700 1 -- 172.21.15.32:0/2020173915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecb4001390 con 0x7fecbc1a1270 2021-04-09T18:13:11.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.153 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecbc1a1670 con 0x7fecbc1a1270 2021-04-09T18:13:11.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.154 7feca3fff700 1 -- 172.21.15.32:0/2020173915 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fecb4009130 con 0x7fecbc1a1270 2021-04-09T18:13:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.154 7feca3fff700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c018930 0x7fec9c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.154 7feca3fff700 1 -- 172.21.15.32:0/2020173915 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fecb4032d80 con 0x7fecbc1a1270 2021-04-09T18:13:11.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.154 7fecbb7fe700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c018930 0x7fec9c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.155 7fecbb7fe700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c018930 0x7fec9c01ae10 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.174 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecbc093d50 con 0x7fecbc1a1270 2021-04-09T18:13:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.177 7feca3fff700 1 -- 172.21.15.32:0/2020173915 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fecb400be30 con 0x7fecbc1a1270 2021-04-09T18:13:11.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.356 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fecbc10d480 con 0x7fecbc1a1270 2021-04-09T18:13:11.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.356 7feca3fff700 1 -- 172.21.15.32:0/2020173915 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fecb4000df0 con 0x7fecbc1a1270 2021-04-09T18:13:11.458 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:11.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c018930 msgr2=0x7fec9c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c018930 0x7fec9c01ae10 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecbc1a1270 msgr2=0x7fecbc0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecbc1a1270 0x7fecbc0b0cf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fecb4006e60 tx=0x7fecb400bcf0).stop 2021-04-09T18:13:11.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 shutdown_connections 2021-04-09T18:13:11.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c018930 0x7fec9c01ae10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecbc1a1270 0x7fecbc0b0cf0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecbc095240 0x7fecbc1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 --2- 172.21.15.32:0/2020173915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecbc0c3c00 0x7fecbc09c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 >> 172.21.15.32:0/2020173915 conn(0x7fecbc0975f0 msgr2=0x7fecbc096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:11.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 shutdown_connections 2021-04-09T18:13:11.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.359 7fecc1aea700 1 -- 172.21.15.32:0/2020173915 wait complete. 2021-04-09T18:13:11.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:13:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.621 7fc1525c8700 1 Processor -- start 2021-04-09T18:13:11.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.621 7fc1525c8700 1 -- start start 2021-04-09T18:13:11.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.621 7fc1525c8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc14c0d7ca0 0x7fc14c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.621 7fc1525c8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc14c10da00 0x7fc14c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.621 7fc1525c8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14c10d480 con 0x7fc14c10d010 2021-04-09T18:13:11.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.621 7fc1525c8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14c0d9220 con 0x7fc14c10da00 2021-04-09T18:13:11.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.621 7fc1525c8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14c0d86c0 con 0x7fc14c0d7ca0 2021-04-09T18:13:11.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.621 7fc150be5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc14c10da00 0x7fc14c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc150be5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc14c10da00 0x7fc14c10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42144/0 (socket says 172.21.15.32:42144) 2021-04-09T18:13:11.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc150be5700 1 -- 172.21.15.32:0/1763052588 learned_addr learned my addr 172.21.15.32:0/1763052588 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:11.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc143fff700 1 --2- 172.21.15.32:0/1763052588 >> v2:172.21.15.86:3300/0 conn(0x7fc14c0d7ca0 0x7fc14c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc150be5700 1 -- 172.21.15.32:0/1763052588 >> v2:172.21.15.86:3300/0 conn(0x7fc14c0d7ca0 msgr2=0x7fc14c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:11.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc150be5700 1 --2- 172.21.15.32:0/1763052588 >> v2:172.21.15.86:3300/0 conn(0x7fc14c0d7ca0 0x7fc14c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc150be5700 1 -- 172.21.15.32:0/1763052588 >> v1:172.21.15.86:6789/0 conn(0x7fc14c10d010 legacy=0x7fc14c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc150be5700 1 -- 172.21.15.32:0/1763052588 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c0d1f70 con 0x7fc14c10da00 2021-04-09T18:13:11.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc150be5700 1 --2- 172.21.15.32:0/1763052588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c10da00 0x7fc14c10de00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc144006f00 tx=0x7fc144009f10).ready entity=mon.0 client_cookie=2bdec41aacbeef75 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:11.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc14b7fe700 1 -- 172.21.15.32:0/1763052588 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc14400b3b0 con 0x7fc14c10da00 2021-04-09T18:13:11.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc14b7fe700 1 -- 172.21.15.32:0/1763052588 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14400b730 con 0x7fc14c10da00 2021-04-09T18:13:11.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc14b7fe700 1 -- 172.21.15.32:0/1763052588 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc14400bc10 con 0x7fc14c10da00 2021-04-09T18:13:11.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc1525c8700 1 -- 172.21.15.32:0/1763052588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c10da00 msgr2=0x7fc14c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.622 7fc1525c8700 1 --2- 172.21.15.32:0/1763052588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c10da00 0x7fc14c10de00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc144006f00 tx=0x7fc144009f10).stop 2021-04-09T18:13:11.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.623 7fc1525c8700 1 -- 172.21.15.32:0/1763052588 shutdown_connections 2021-04-09T18:13:11.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.623 7fc1525c8700 1 --2- 172.21.15.32:0/1763052588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c10da00 0x7fc14c10de00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.623 7fc1525c8700 1 --2- 172.21.15.32:0/1763052588 >> v2:172.21.15.86:3300/0 conn(0x7fc14c0d7ca0 0x7fc14c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.623 7fc1525c8700 1 -- 172.21.15.32:0/1763052588 >> 172.21.15.32:0/1763052588 conn(0x7fc14c0ce5f0 msgr2=0x7fc14c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:11.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.623 7fc1525c8700 1 -- 172.21.15.32:0/1763052588 shutdown_connections 2021-04-09T18:13:11.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.623 7fc1525c8700 1 -- 172.21.15.32:0/1763052588 wait complete. 2021-04-09T18:13:11.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.643 7fc1525c8700 1 Processor -- start 2021-04-09T18:13:11.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.643 7fc1525c8700 1 -- start start 2021-04-09T18:13:11.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1525c8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14c19e2c0 0x7fc14c0a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1525c8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc14c1797d0 0x7fc14c179bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1525c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c09d350 0x7fc14c09d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1525c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c17a1f0 con 0x7fc14c09d350 2021-04-09T18:13:11.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1525c8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c09dd70 con 0x7fc14c19e2c0 2021-04-09T18:13:11.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1525c8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c09df90 con 0x7fc14c1797d0 2021-04-09T18:13:11.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc150be5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c09d350 0x7fc14c09d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc150be5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c09d350 0x7fc14c09d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42152/0 (socket says 172.21.15.32:42152) 2021-04-09T18:13:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc150be5700 1 -- 172.21.15.32:0/3249765364 learned_addr learned my addr 172.21.15.32:0/3249765364 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc143fff700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc14c1797d0 0x7fc14c179bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc14bfff700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14c19e2c0 0x7fc14c0a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc150be5700 1 -- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc14c1797d0 msgr2=0x7fc14c179bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc150be5700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc14c1797d0 0x7fc14c179bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc150be5700 1 -- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14c19e2c0 msgr2=0x7fc14c0a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc150be5700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14c19e2c0 0x7fc14c0a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc150be5700 1 -- 172.21.15.32:0/3249765364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c09e1b0 con 0x7fc14c09d350 2021-04-09T18:13:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc150be5700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c09d350 0x7fc14c09d750 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc14400b9b0 tx=0x7fc14400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1497fa700 1 -- 172.21.15.32:0/3249765364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc144005820 con 0x7fc14c09d350 2021-04-09T18:13:11.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1497fa700 1 -- 172.21.15.32:0/3249765364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc144005aa0 con 0x7fc14c09d350 2021-04-09T18:13:11.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c09e1b0 con 0x7fc14c09d350 2021-04-09T18:13:11.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.644 7fc1497fa700 1 -- 172.21.15.32:0/3249765364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc144006030 con 0x7fc14c09d350 2021-04-09T18:13:11.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.645 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14c09e1b0 con 0x7fc14c09d350 2021-04-09T18:13:11.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.645 7fc1497fa700 1 -- 172.21.15.32:0/3249765364 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc144006030 con 0x7fc14c09d350 2021-04-09T18:13:11.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.645 7fc1497fa700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc130018930 0x7fc13001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:11.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.645 7fc1497fa700 1 -- 172.21.15.32:0/3249765364 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc144032d60 con 0x7fc14c09d350 2021-04-09T18:13:11.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.645 7fc14bfff700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc130018930 0x7fc13001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:11.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.646 7fc14bfff700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc130018930 0x7fc13001ae10 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:11.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.665 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14c0a1a10 con 0x7fc14c09d350 2021-04-09T18:13:11.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.669 7fc1497fa700 1 -- 172.21.15.32:0/3249765364 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc144000f90 con 0x7fc14c09d350 2021-04-09T18:13:11.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.846 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fc14c0a1a10 con 0x7fc14c09d350 2021-04-09T18:13:11.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.846 7fc1497fa700 1 -- 172.21.15.32:0/3249765364 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc144006030 con 0x7fc14c09d350 2021-04-09T18:13:11.914 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:11.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.848 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc130018930 msgr2=0x7fc13001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.848 7fc1525c8700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc130018930 0x7fc13001ae10 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c09d350 msgr2=0x7fc14c09d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:11.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c09d350 0x7fc14c09d750 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc14400b9b0 tx=0x7fc14400bd50).stop 2021-04-09T18:13:11.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 shutdown_connections 2021-04-09T18:13:11.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc130018930 0x7fc13001ae10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc14c09d350 0x7fc14c09d750 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc14c1797d0 0x7fc14c179bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 --2- 172.21.15.32:0/3249765364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14c19e2c0 0x7fc14c0a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 >> 172.21.15.32:0/3249765364 conn(0x7fc14c0a22e0 msgr2=0x7fc14c094c80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 shutdown_connections 2021-04-09T18:13:12.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:11.849 7fc1525c8700 1 -- 172.21.15.32:0/3249765364 wait complete. 2021-04-09T18:13:12.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:12.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.233 7f489b4a2700 1 Processor -- start 2021-04-09T18:13:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.233 7f489b4a2700 1 -- start start 2021-04-09T18:13:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.234 7f489b4a2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48940d7ca0 0x7f48940d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.234 7f489b4a2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f489410da00 0x7f489410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.234 7f489b4a2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489410d480 con 0x7f489410d010 2021-04-09T18:13:12.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.234 7f489b4a2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48940d9220 con 0x7f489410da00 2021-04-09T18:13:12.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.234 7f489b4a2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48940d86c0 con 0x7f48940d7ca0 2021-04-09T18:13:12.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.234 7f4899abf700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f489410da00 0x7f489410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.234 7f4898abd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48940d7ca0 0x7f48940d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.234 7f48992be700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f489410d010 0x7f48940d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:49672/0 (socket says 172.21.15.32:49672) 2021-04-09T18:13:12.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.234 7f48992be700 1 -- 172.21.15.32:0/995830133 learned_addr learned my addr 172.21.15.32:0/995830133 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:12.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f4899abf700 1 -- 172.21.15.32:0/995830133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48940d7ca0 msgr2=0x7f48940d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:12.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f4899abf700 1 --2- 172.21.15.32:0/995830133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48940d7ca0 0x7f48940d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f4899abf700 1 -- 172.21.15.32:0/995830133 >> v1:172.21.15.86:6789/0 conn(0x7f489410d010 legacy=0x7f48940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f4899abf700 1 -- 172.21.15.32:0/995830133 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48940d1f70 con 0x7f489410da00 2021-04-09T18:13:12.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f4899abf700 1 --2- 172.21.15.32:0/995830133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f489410da00 0x7f489410de00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f488c006ad0 tx=0x7f488c009ad0).ready entity=mon.1 client_cookie=88f87d894013bca4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:12.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f488bfff700 1 -- 172.21.15.32:0/995830133 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c0022e0 con 0x7f489410da00 2021-04-09T18:13:12.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f488bfff700 1 -- 172.21.15.32:0/995830133 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c006410 con 0x7f489410da00 2021-04-09T18:13:12.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f488bfff700 1 -- 172.21.15.32:0/995830133 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c002620 con 0x7f489410da00 2021-04-09T18:13:12.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f489b4a2700 1 -- 172.21.15.32:0/995830133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f489410da00 msgr2=0x7f489410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.235 7f489b4a2700 1 --2- 172.21.15.32:0/995830133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f489410da00 0x7f489410de00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f488c006ad0 tx=0x7f488c009ad0).stop 2021-04-09T18:13:12.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.236 7f489b4a2700 1 -- 172.21.15.32:0/995830133 shutdown_connections 2021-04-09T18:13:12.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.236 7f489b4a2700 1 --2- 172.21.15.32:0/995830133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f489410da00 0x7f489410de00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.236 7f489b4a2700 1 --2- 172.21.15.32:0/995830133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48940d7ca0 0x7f48940d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.236 7f489b4a2700 1 -- 172.21.15.32:0/995830133 >> 172.21.15.32:0/995830133 conn(0x7f48940ce5f0 msgr2=0x7f48940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:12.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.236 7f489b4a2700 1 -- 172.21.15.32:0/995830133 shutdown_connections 2021-04-09T18:13:12.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.236 7f489b4a2700 1 -- 172.21.15.32:0/995830133 wait complete. 2021-04-09T18:13:12.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.256 7f489b4a2700 1 Processor -- start 2021-04-09T18:13:12.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.256 7f489b4a2700 1 -- start start 2021-04-09T18:13:12.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f489b4a2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48940ac0f0 0x7f48940ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f489b4a2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48941a6c20 0x7f48941a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f489b4a2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48941a56c0 0x7f48941a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f489b4a2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48941a76c0 con 0x7f48940ac0f0 2021-04-09T18:13:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f489b4a2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489417c7e0 con 0x7f48941a56c0 2021-04-09T18:13:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f489b4a2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489417ca00 con 0x7f48941a6c20 2021-04-09T18:13:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f48992be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48940ac0f0 0x7f48940ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f48992be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48940ac0f0 0x7f48940ac4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42162/0 (socket says 172.21.15.32:42162) 2021-04-09T18:13:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f48992be700 1 -- 172.21.15.32:0/2116940307 learned_addr learned my addr 172.21.15.32:0/2116940307 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f4899abf700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48941a56c0 0x7f48941a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f4898abd700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48941a6c20 0x7f48941a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f48992be700 1 -- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48941a6c20 msgr2=0x7f48941a7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f48992be700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48941a6c20 0x7f48941a7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f48992be700 1 -- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48941a56c0 msgr2=0x7f48941a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f48992be700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48941a56c0 0x7f48941a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f48992be700 1 -- 172.21.15.32:0/2116940307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48941a60e0 con 0x7f48940ac0f0 2021-04-09T18:13:12.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.257 7f48992be700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48940ac0f0 0x7f48940ac4f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f4890007fc0 tx=0x7f489000ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:12.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.258 7f4889ffb700 1 -- 172.21.15.32:0/2116940307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4890007020 con 0x7f48940ac0f0 2021-04-09T18:13:12.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.258 7f4889ffb700 1 -- 172.21.15.32:0/2116940307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48900072e0 con 0x7f48940ac0f0 2021-04-09T18:13:12.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.258 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48941a60e0 con 0x7f48940ac0f0 2021-04-09T18:13:12.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.258 7f4889ffb700 1 -- 172.21.15.32:0/2116940307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48900046a0 con 0x7f48940ac0f0 2021-04-09T18:13:12.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.258 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48941a60e0 con 0x7f48940ac0f0 2021-04-09T18:13:12.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.258 7f4889ffb700 1 -- 172.21.15.32:0/2116940307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f48900046a0 con 0x7f48940ac0f0 2021-04-09T18:13:12.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.259 7f4889ffb700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4874018930 0x7f487401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.259 7f4889ffb700 1 -- 172.21.15.32:0/2116940307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4890032d90 con 0x7f48940ac0f0 2021-04-09T18:13:12.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.259 7f4898abd700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4874018930 0x7f487401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.259 7f4898abd700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4874018930 0x7f487401ae10 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.278 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48941b7050 con 0x7f48940ac0f0 2021-04-09T18:13:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.281 7f4889ffb700 1 -- 172.21.15.32:0/2116940307 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f48900048b0 con 0x7f48940ac0f0 2021-04-09T18:13:12.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.470 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48941b7050 con 0x7f48940ac0f0 2021-04-09T18:13:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.471 7f4889ffb700 1 -- 172.21.15.32:0/2116940307 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f48900046a0 con 0x7f48940ac0f0 2021-04-09T18:13:12.493 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:12.494 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:34.773743","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:12.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.473 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4874018930 msgr2=0x7f487401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.473 7f489b4a2700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4874018930 0x7f487401ae10 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.473 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48940ac0f0 msgr2=0x7f48940ac4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.473 7f489b4a2700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48940ac0f0 0x7f48940ac4f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f4890007fc0 tx=0x7f489000ac80).stop 2021-04-09T18:13:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.474 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 shutdown_connections 2021-04-09T18:13:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.474 7f489b4a2700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4874018930 0x7f487401ae10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.474 7f489b4a2700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48941a56c0 0x7f48941a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.474 7f489b4a2700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48941a6c20 0x7f48941a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.474 7f489b4a2700 1 --2- 172.21.15.32:0/2116940307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48940ac0f0 0x7f48940ac4f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.474 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 >> 172.21.15.32:0/2116940307 conn(0x7f4894199680 msgr2=0x7f48941a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:12.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.474 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 shutdown_connections 2021-04-09T18:13:12.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.474 7f489b4a2700 1 -- 172.21.15.32:0/2116940307 wait complete. 2021-04-09T18:13:12.507 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:12:32.136283', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:12:32.831693', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:13:12.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:13:12.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.673 7f86aff41700 1 Processor -- start 2021-04-09T18:13:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.674 7f86aff41700 1 -- start start 2021-04-09T18:13:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.674 7f86aff41700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a80d6ca0 0x7f86a80d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.674 7f86aff41700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86a80d2460 0x7f86a810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.674 7f86aff41700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a80d72a0 con 0x7f86a80cffc0 2021-04-09T18:13:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.674 7f86aff41700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a80d70a0 con 0x7f86a80d2460 2021-04-09T18:13:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.674 7f86aff41700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80d9200 con 0x7f86a80d6ca0 2021-04-09T18:13:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ad55c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a80d6ca0 0x7f86a80d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ae55e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86a80d2460 0x7f86a810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ad55c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a80d6ca0 0x7f86a80d8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:49400/0 (socket says 172.21.15.32:49400) 2021-04-09T18:13:12.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ad55c700 1 -- 172.21.15.32:0/624282540 learned_addr learned my addr 172.21.15.32:0/624282540 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:12.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ad55c700 1 -- 172.21.15.32:0/624282540 >> v2:172.21.15.86:3300/0 conn(0x7f86a80d2460 msgr2=0x7f86a810cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:12.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ad55c700 1 --2- 172.21.15.32:0/624282540 >> v2:172.21.15.86:3300/0 conn(0x7f86a80d2460 0x7f86a810cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ad55c700 1 -- 172.21.15.32:0/624282540 >> v1:172.21.15.86:6789/0 conn(0x7f86a80cffc0 legacy=0x7f86a80d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86acd5b700 1 -- 172.21.15.32:0/624282540 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148667211 0 0) 0x7f86a4000c60 con 0x7f86a80cffc0 2021-04-09T18:13:12.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ad55c700 1 -- 172.21.15.32:0/624282540 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a80d2860 con 0x7f86a80d6ca0 2021-04-09T18:13:12.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ae55e700 1 --2- 172.21.15.32:0/624282540 >> v2:172.21.15.86:3300/0 conn(0x7f86a80d2460 0x7f86a810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:12.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86ad55c700 1 --2- 172.21.15.32:0/624282540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a80d6ca0 0x7f86a80d8010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f8698006700 tx=0x7f8698009b80).ready entity=mon.2 client_cookie=20092405f013de3f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:12.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86acd5b700 1 -- 172.21.15.32:0/624282540 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f869800c0c0 con 0x7f86a80d6ca0 2021-04-09T18:13:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86acd5b700 1 -- 172.21.15.32:0/624282540 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869800c440 con 0x7f86a80d6ca0 2021-04-09T18:13:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86acd5b700 1 -- 172.21.15.32:0/624282540 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f869800c920 con 0x7f86a80d6ca0 2021-04-09T18:13:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86aff41700 1 -- 172.21.15.32:0/624282540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a80d6ca0 msgr2=0x7f86a80d8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.675 7f86aff41700 1 --2- 172.21.15.32:0/624282540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a80d6ca0 0x7f86a80d8010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f8698006700 tx=0x7f8698009b80).stop 2021-04-09T18:13:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.676 7f86aff41700 1 -- 172.21.15.32:0/624282540 shutdown_connections 2021-04-09T18:13:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.676 7f86aff41700 1 --2- 172.21.15.32:0/624282540 >> v2:172.21.15.86:3300/0 conn(0x7f86a80d2460 0x7f86a810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.676 7f86aff41700 1 --2- 172.21.15.32:0/624282540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a80d6ca0 0x7f86a80d8010 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.676 7f86aff41700 1 -- 172.21.15.32:0/624282540 >> 172.21.15.32:0/624282540 conn(0x7f86a80ce5e0 msgr2=0x7f86a80cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:12.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.676 7f86aff41700 1 -- 172.21.15.32:0/624282540 shutdown_connections 2021-04-09T18:13:12.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.676 7f86aff41700 1 -- 172.21.15.32:0/624282540 wait complete. 2021-04-09T18:13:12.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.696 7f86aff41700 1 Processor -- start 2021-04-09T18:13:12.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.696 7f86aff41700 1 -- start start 2021-04-09T18:13:12.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86aff41700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a81833a0 0x7f86a80a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86aff41700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a80a2550 0x7f86a80a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86aff41700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a8094d00 0x7f86a8095100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86aff41700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80a1fd0 con 0x7f86a80a2550 2021-04-09T18:13:12.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86aff41700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80a2f70 con 0x7f86a81833a0 2021-04-09T18:13:12.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86aff41700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a8095720 con 0x7f86a8094d00 2021-04-09T18:13:12.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ad55c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a80a2550 0x7f86a80a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ad55c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a80a2550 0x7f86a80a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42178/0 (socket says 172.21.15.32:42178) 2021-04-09T18:13:12.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ad55c700 1 -- 172.21.15.32:0/2877654753 learned_addr learned my addr 172.21.15.32:0/2877654753 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ae55e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a8094d00 0x7f86a8095100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86add5d700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a81833a0 0x7f86a80a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ad55c700 1 -- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a8094d00 msgr2=0x7f86a8095100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ad55c700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a8094d00 0x7f86a8095100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ad55c700 1 -- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a81833a0 msgr2=0x7f86a80a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ad55c700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a81833a0 0x7f86a80a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ad55c700 1 -- 172.21.15.32:0/2877654753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a8095940 con 0x7f86a80a2550 2021-04-09T18:13:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.697 7f86ad55c700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a80a2550 0x7f86a80a2950 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f869800b5e0 tx=0x7f869800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.698 7f869e7fc700 1 -- 172.21.15.32:0/2877654753 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f869800bd70 con 0x7f86a80a2550 2021-04-09T18:13:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.698 7f869e7fc700 1 -- 172.21.15.32:0/2877654753 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8698002030 con 0x7f86a80a2550 2021-04-09T18:13:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.698 7f86aff41700 1 -- 172.21.15.32:0/2877654753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a8095940 con 0x7f86a80a2550 2021-04-09T18:13:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.698 7f869e7fc700 1 -- 172.21.15.32:0/2877654753 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8698002550 con 0x7f86a80a2550 2021-04-09T18:13:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.698 7f86aff41700 1 -- 172.21.15.32:0/2877654753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a8095940 con 0x7f86a80a2550 2021-04-09T18:13:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.698 7f869e7fc700 1 -- 172.21.15.32:0/2877654753 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8698000c90 con 0x7f86a80a2550 2021-04-09T18:13:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.698 7f869e7fc700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8690018930 0x7f869001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:12.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.698 7f869e7fc700 1 -- 172.21.15.32:0/2877654753 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8698032a50 con 0x7f86a80a2550 2021-04-09T18:13:12.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.699 7f86add5d700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8690018930 0x7f869001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:12.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.699 7f86add5d700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8690018930 0x7f869001ae10 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:12.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.718 7f86aff41700 1 -- 172.21.15.32:0/2877654753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a8095be0 con 0x7f86a80a2550 2021-04-09T18:13:12.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.722 7f869e7fc700 1 -- 172.21.15.32:0/2877654753 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8698006020 con 0x7f86a80a2550 2021-04-09T18:13:12.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.905 7f86aff41700 1 -- 172.21.15.32:0/2877654753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f86a8093df0 con 0x7f86a80a2550 2021-04-09T18:13:12.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.906 7f869e7fc700 1 -- 172.21.15.32:0/2877654753 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8698000c90 con 0x7f86a80a2550 2021-04-09T18:13:12.935 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 -- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8690018930 msgr2=0x7f869001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8690018930 0x7f869001ae10 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 -- 172.21.15.32:0/2877654753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a80a2550 msgr2=0x7f86a80a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:12.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a80a2550 0x7f86a80a2950 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f869800b5e0 tx=0x7f869800b980).stop 2021-04-09T18:13:12.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 -- 172.21.15.32:0/2877654753 shutdown_connections 2021-04-09T18:13:12.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8690018930 0x7f869001ae10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86a80a2550 0x7f86a80a2950 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a8094d00 0x7f86a8095100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 --2- 172.21.15.32:0/2877654753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a81833a0 0x7f86a80a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 -- 172.21.15.32:0/2877654753 >> 172.21.15.32:0/2877654753 conn(0x7f86a809fa00 msgr2=0x7f86a80b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 -- 172.21.15.32:0/2877654753 shutdown_connections 2021-04-09T18:13:12.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:12.908 7f86aff41700 1 -- 172.21.15.32:0/2877654753 wait complete. 2021-04-09T18:13:12.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:13:13.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.115 7fcb0ef1b700 1 Processor -- start 2021-04-09T18:13:13.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.115 7fcb0ef1b700 1 -- start start 2021-04-09T18:13:13.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0ef1b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcb0810d010 0x7fcb080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0ef1b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080d1da0 0x7fcb080d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0ef1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb080d82c0 0x7fcb0810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0ef1b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb080d8d30 con 0x7fcb080d82c0 2021-04-09T18:13:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0ef1b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb080d9220 con 0x7fcb0810d010 2021-04-09T18:13:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0ef1b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0810d480 con 0x7fcb080d1da0 2021-04-09T18:13:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0d538700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb080d82c0 0x7fcb0810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0d538700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb080d82c0 0x7fcb0810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42188/0 (socket says 172.21.15.32:42188) 2021-04-09T18:13:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0d538700 1 -- 172.21.15.32:0/468711355 learned_addr learned my addr 172.21.15.32:0/468711355 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb07fff700 1 --2- 172.21.15.32:0/468711355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080d1da0 0x7fcb080d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0cd37700 1 --2- 172.21.15.32:0/468711355 >> v2:172.21.15.86:3300/0 conn(0x7fcb0810d010 0x7fcb080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0d538700 1 -- 172.21.15.32:0/468711355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080d1da0 msgr2=0x7fcb080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0d538700 1 --2- 172.21.15.32:0/468711355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080d1da0 0x7fcb080d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0d538700 1 -- 172.21.15.32:0/468711355 >> v2:172.21.15.86:3300/0 conn(0x7fcb0810d010 msgr2=0x7fcb080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:13.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0d538700 1 --2- 172.21.15.32:0/468711355 >> v2:172.21.15.86:3300/0 conn(0x7fcb0810d010 0x7fcb080d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0d538700 1 -- 172.21.15.32:0/468711355 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb080d86c0 con 0x7fcb080d82c0 2021-04-09T18:13:13.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0cd37700 1 --2- 172.21.15.32:0/468711355 >> v2:172.21.15.86:3300/0 conn(0x7fcb0810d010 0x7fcb080d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:13.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.116 7fcb0d538700 1 --2- 172.21.15.32:0/468711355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb080d82c0 0x7fcb0810da00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fcb00006b80 tx=0x7fcb00009b90).ready entity=mon.0 client_cookie=e79286c26b543b65 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:13.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb077fe700 1 -- 172.21.15.32:0/468711355 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb000022e0 con 0x7fcb080d82c0 2021-04-09T18:13:13.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb077fe700 1 -- 172.21.15.32:0/468711355 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00002620 con 0x7fcb080d82c0 2021-04-09T18:13:13.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb077fe700 1 -- 172.21.15.32:0/468711355 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb00002b00 con 0x7fcb080d82c0 2021-04-09T18:13:13.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb0ef1b700 1 -- 172.21.15.32:0/468711355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb080d82c0 msgr2=0x7fcb0810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:13.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb0ef1b700 1 --2- 172.21.15.32:0/468711355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb080d82c0 0x7fcb0810da00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fcb00006b80 tx=0x7fcb00009b90).stop 2021-04-09T18:13:13.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb0ef1b700 1 -- 172.21.15.32:0/468711355 shutdown_connections 2021-04-09T18:13:13.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb0ef1b700 1 --2- 172.21.15.32:0/468711355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb080d82c0 0x7fcb0810da00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb0ef1b700 1 --2- 172.21.15.32:0/468711355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080d1da0 0x7fcb080d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb0ef1b700 1 --2- 172.21.15.32:0/468711355 >> v2:172.21.15.86:3300/0 conn(0x7fcb0810d010 0x7fcb080d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb0ef1b700 1 -- 172.21.15.32:0/468711355 >> 172.21.15.32:0/468711355 conn(0x7fcb080ce5f0 msgr2=0x7fcb080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb0ef1b700 1 -- 172.21.15.32:0/468711355 shutdown_connections 2021-04-09T18:13:13.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.117 7fcb0ef1b700 1 -- 172.21.15.32:0/468711355 wait complete. 2021-04-09T18:13:13.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.137 7fcb0ef1b700 1 Processor -- start 2021-04-09T18:13:13.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb0ef1b700 1 -- start start 2021-04-09T18:13:13.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb0ef1b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080c3c00 0x7fcb0809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb0ef1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb08095240 0x7fcb081a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb0ef1b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb081a1270 0x7fcb080b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb0ef1b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0809cda0 con 0x7fcb08095240 2021-04-09T18:13:13.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb0ef1b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb080c3680 con 0x7fcb081a1270 2021-04-09T18:13:13.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb0ef1b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb08095640 con 0x7fcb080c3c00 2021-04-09T18:13:13.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb07fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb08095240 0x7fcb081a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb07fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb08095240 0x7fcb081a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42192/0 (socket says 172.21.15.32:42192) 2021-04-09T18:13:13.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.138 7fcb07fff700 1 -- 172.21.15.32:0/3507311078 learned_addr learned my addr 172.21.15.32:0/3507311078 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:13.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb0d538700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb081a1270 0x7fcb080b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb0cd37700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080c3c00 0x7fcb0809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb07fff700 1 -- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080c3c00 msgr2=0x7fcb0809c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:13.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb07fff700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080c3c00 0x7fcb0809c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb07fff700 1 -- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb081a1270 msgr2=0x7fcb080b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:13.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb07fff700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb081a1270 0x7fcb080b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb07fff700 1 -- 172.21.15.32:0/3507311078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb081a1670 con 0x7fcb08095240 2021-04-09T18:13:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb0cd37700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080c3c00 0x7fcb0809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:13.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb07fff700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb08095240 0x7fcb081a0c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fcaf8009980 tx=0x7fcaf800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:13.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb057fa700 1 -- 172.21.15.32:0/3507311078 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaf800c2f0 con 0x7fcb08095240 2021-04-09T18:13:13.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb057fa700 1 -- 172.21.15.32:0/3507311078 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf800c5b0 con 0x7fcb08095240 2021-04-09T18:13:13.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb057fa700 1 -- 172.21.15.32:0/3507311078 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaf800ca90 con 0x7fcb08095240 2021-04-09T18:13:13.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb081a1670 con 0x7fcb08095240 2021-04-09T18:13:13.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.139 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb081a1670 con 0x7fcb08095240 2021-04-09T18:13:13.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.140 7fcb057fa700 1 -- 172.21.15.32:0/3507311078 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcaf8001010 con 0x7fcb08095240 2021-04-09T18:13:13.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.140 7fcb057fa700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcaec018930 0x7fcaec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.140 7fcb057fa700 1 -- 172.21.15.32:0/3507311078 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcaf8032dc0 con 0x7fcb08095240 2021-04-09T18:13:13.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.140 7fcb0cd37700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcaec018930 0x7fcaec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.140 7fcb0cd37700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcaec018930 0x7fcaec01ae10 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:13.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.160 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb08093d50 con 0x7fcb08095240 2021-04-09T18:13:13.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.164 7fcb057fa700 1 -- 172.21.15.32:0/3507311078 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcaf800be10 con 0x7fcb08095240 2021-04-09T18:13:13.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.344 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fcb0810d480 con 0x7fcb08095240 2021-04-09T18:13:13.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.344 7fcb057fa700 1 -- 172.21.15.32:0/3507311078 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcaf8001010 con 0x7fcb08095240 2021-04-09T18:13:13.709 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:13.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcaec018930 msgr2=0x7fcaec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:13.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcaec018930 0x7fcaec01ae10 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb08095240 msgr2=0x7fcb081a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:13.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb08095240 0x7fcb081a0c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fcaf8009980 tx=0x7fcaf800bcd0).stop 2021-04-09T18:13:13.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 shutdown_connections 2021-04-09T18:13:13.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcaec018930 0x7fcaec01ae10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb08095240 0x7fcb081a0c50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb081a1270 0x7fcb080b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 --2- 172.21.15.32:0/3507311078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb080c3c00 0x7fcb0809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 >> 172.21.15.32:0/3507311078 conn(0x7fcb080975f0 msgr2=0x7fcb08096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:13.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 shutdown_connections 2021-04-09T18:13:13.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.347 7fcb0ef1b700 1 -- 172.21.15.32:0/3507311078 wait complete. 2021-04-09T18:13:13.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:13:13.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.901 7fe111887700 1 Processor -- start 2021-04-09T18:13:13.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.901 7fe111887700 1 -- start start 2021-04-09T18:13:13.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe111887700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe10c10d010 0x7fe10c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe111887700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe10c0d1da0 0x7fe10c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe111887700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c0d82c0 0x7fe10c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe111887700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c0d8d30 con 0x7fe10c0d1da0 2021-04-09T18:13:13.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe111887700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c0d9220 con 0x7fe10c10d010 2021-04-09T18:13:13.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe111887700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c10d4b0 con 0x7fe10c0d82c0 2021-04-09T18:13:13.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10ad73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe10c0d1da0 0x7fe10c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10ad73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe10c0d1da0 0x7fe10c0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42202/0 (socket says 172.21.15.32:42202) 2021-04-09T18:13:13.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10ad73700 1 -- 172.21.15.32:0/2764186972 learned_addr learned my addr 172.21.15.32:0/2764186972 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:13.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10bd75700 1 --2- 172.21.15.32:0/2764186972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c0d82c0 0x7fe10c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10b574700 1 --2- 172.21.15.32:0/2764186972 >> v2:172.21.15.86:3300/0 conn(0x7fe10c10d010 0x7fe10c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10ad73700 1 -- 172.21.15.32:0/2764186972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c0d82c0 msgr2=0x7fe10c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:13.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10ad73700 1 --2- 172.21.15.32:0/2764186972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c0d82c0 0x7fe10c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10ad73700 1 -- 172.21.15.32:0/2764186972 >> v2:172.21.15.86:3300/0 conn(0x7fe10c10d010 msgr2=0x7fe10c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:13.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10ad73700 1 --2- 172.21.15.32:0/2764186972 >> v2:172.21.15.86:3300/0 conn(0x7fe10c10d010 0x7fe10c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10ad73700 1 -- 172.21.15.32:0/2764186972 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c0d86c0 con 0x7fe10c0d1da0 2021-04-09T18:13:13.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10ad73700 1 --2- 172.21.15.32:0/2764186972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0d1da0 0x7fe10c0d7ca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe100006700 tx=0x7fe100009b80).ready entity=mon.0 client_cookie=5bcc99573b9d7e8e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:13.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.902 7fe10a572700 1 -- 172.21.15.32:0/2764186972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe10000c0c0 con 0x7fe10c0d1da0 2021-04-09T18:13:13.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe10a572700 1 -- 172.21.15.32:0/2764186972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10000c440 con 0x7fe10c0d1da0 2021-04-09T18:13:13.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe10a572700 1 -- 172.21.15.32:0/2764186972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe10000c920 con 0x7fe10c0d1da0 2021-04-09T18:13:13.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe111887700 1 -- 172.21.15.32:0/2764186972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0d1da0 msgr2=0x7fe10c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:13.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe111887700 1 --2- 172.21.15.32:0/2764186972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0d1da0 0x7fe10c0d7ca0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe100006700 tx=0x7fe100009b80).stop 2021-04-09T18:13:13.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe111887700 1 -- 172.21.15.32:0/2764186972 shutdown_connections 2021-04-09T18:13:13.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe111887700 1 --2- 172.21.15.32:0/2764186972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c0d82c0 0x7fe10c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe111887700 1 --2- 172.21.15.32:0/2764186972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0d1da0 0x7fe10c0d7ca0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe111887700 1 --2- 172.21.15.32:0/2764186972 >> v2:172.21.15.86:3300/0 conn(0x7fe10c10d010 0x7fe10c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:13.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe111887700 1 -- 172.21.15.32:0/2764186972 >> 172.21.15.32:0/2764186972 conn(0x7fe10c0ce5f0 msgr2=0x7fe10c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:13.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe111887700 1 -- 172.21.15.32:0/2764186972 shutdown_connections 2021-04-09T18:13:13.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.903 7fe111887700 1 -- 172.21.15.32:0/2764186972 wait complete. 2021-04-09T18:13:13.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.923 7fe111887700 1 Processor -- start 2021-04-09T18:13:13.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe111887700 1 -- start start 2021-04-09T18:13:13.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe111887700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0a0a10 0x7fe10c0bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe111887700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10c18ad00 0x7fe10c19fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe111887700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c1a03f0 0x7fe10c095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe111887700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c18b100 con 0x7fe10c0a0a10 2021-04-09T18:13:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe111887700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c095860 con 0x7fe10c18ad00 2021-04-09T18:13:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe111887700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c095a80 con 0x7fe10c1a03f0 2021-04-09T18:13:13.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe10b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0a0a10 0x7fe10c0bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:13.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe10b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0a0a10 0x7fe10c0bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42204/0 (socket says 172.21.15.32:42204) 2021-04-09T18:13:13.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe10b574700 1 -- 172.21.15.32:0/684096653 learned_addr learned my addr 172.21.15.32:0/684096653 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:14.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe10bd75700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c1a03f0 0x7fe10c095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.924 7fe10ad73700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10c18ad00 0x7fe10c19fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe10b574700 1 -- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c1a03f0 msgr2=0x7fe10c095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe10b574700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c1a03f0 0x7fe10c095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe10b574700 1 -- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10c18ad00 msgr2=0x7fe10c19fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe10b574700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10c18ad00 0x7fe10c19fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe10b574700 1 -- 172.21.15.32:0/684096653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c189250 con 0x7fe10c0a0a10 2021-04-09T18:13:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe10ad73700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10c18ad00 0x7fe10c19fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe10b574700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0a0a10 0x7fe10c0bf870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fe0fc00b530 tx=0x7fe0fc00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe0f3fff700 1 -- 172.21.15.32:0/684096653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0fc00bcc0 con 0x7fe10c0a0a10 2021-04-09T18:13:14.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe0f3fff700 1 -- 172.21.15.32:0/684096653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc002030 con 0x7fe10c0a0a10 2021-04-09T18:13:14.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe111887700 1 -- 172.21.15.32:0/684096653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10c189250 con 0x7fe10c0a0a10 2021-04-09T18:13:14.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe0f3fff700 1 -- 172.21.15.32:0/684096653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0fc002480 con 0x7fe10c0a0a10 2021-04-09T18:13:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe111887700 1 -- 172.21.15.32:0/684096653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10c189250 con 0x7fe10c0a0a10 2021-04-09T18:13:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.925 7fe0f3fff700 1 -- 172.21.15.32:0/684096653 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe0fc002210 con 0x7fe10c0a0a10 2021-04-09T18:13:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.926 7fe0f3fff700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0ec018930 0x7fe0ec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.926 7fe0f3fff700 1 -- 172.21.15.32:0/684096653 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe0fc0329f0 con 0x7fe10c0a0a10 2021-04-09T18:13:14.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.926 7fe10ad73700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0ec018930 0x7fe0ec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.926 7fe10ad73700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0ec018930 0x7fe0ec01adc0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:14.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.946 7fe111887700 1 -- 172.21.15.32:0/684096653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10c093d50 con 0x7fe10c0a0a10 2021-04-09T18:13:14.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:13.949 7fe0f3fff700 1 -- 172.21.15.32:0/684096653 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe0fc002240 con 0x7fe10c0a0a10 2021-04-09T18:13:14.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.128 7fe111887700 1 -- 172.21.15.32:0/684096653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fe10c10d480 con 0x7fe10c0a0a10 2021-04-09T18:13:14.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.128 7fe0f3fff700 1 -- 172.21.15.32:0/684096653 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe0fc002030 con 0x7fe10c0a0a10 2021-04-09T18:13:14.148 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:14.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 -- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0ec018930 msgr2=0x7fe0ec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0ec018930 0x7fe0ec01adc0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 -- 172.21.15.32:0/684096653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0a0a10 msgr2=0x7fe10c0bf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0a0a10 0x7fe10c0bf870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fe0fc00b530 tx=0x7fe0fc00b8d0).stop 2021-04-09T18:13:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 -- 172.21.15.32:0/684096653 shutdown_connections 2021-04-09T18:13:14.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0ec018930 0x7fe0ec01adc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10c1a03f0 0x7fe10c095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10c18ad00 0x7fe10c19fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 --2- 172.21.15.32:0/684096653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe10c0a0a10 0x7fe10c0bf870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.130 7fe111887700 1 -- 172.21.15.32:0/684096653 >> 172.21.15.32:0/684096653 conn(0x7fe10c0975f0 msgr2=0x7fe10c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.131 7fe111887700 1 -- 172.21.15.32:0/684096653 shutdown_connections 2021-04-09T18:13:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.131 7fe111887700 1 -- 172.21.15.32:0/684096653 wait complete. 2021-04-09T18:13:14.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:14.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.300 7fcb52a81700 1 Processor -- start 2021-04-09T18:13:14.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.300 7fcb52a81700 1 -- start start 2021-04-09T18:13:14.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb52a81700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcb4c0cffa0 0x7fcb4c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb52a81700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb4c0d7800 0x7fcb4c0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb52a81700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c0d7280 con 0x7fcb4c0d8220 2021-04-09T18:13:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb52a81700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4c0d91e0 con 0x7fcb4c0d7800 2021-04-09T18:13:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb52a81700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4c10dfe0 con 0x7fcb4c0cffa0 2021-04-09T18:13:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb4bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb4c0d7800 0x7fcb4c0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb4bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb4c0d7800 0x7fcb4c0d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42212/0 (socket says 172.21.15.32:42212) 2021-04-09T18:13:14.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb4bfff700 1 -- 172.21.15.32:0/2037200363 learned_addr learned my addr 172.21.15.32:0/2037200363 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:14.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb5089d700 1 --2- 172.21.15.32:0/2037200363 >> v2:172.21.15.86:3300/0 conn(0x7fcb4c0cffa0 0x7fcb4c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb4bfff700 1 -- 172.21.15.32:0/2037200363 >> v2:172.21.15.86:3300/0 conn(0x7fcb4c0cffa0 msgr2=0x7fcb4c0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb4bfff700 1 --2- 172.21.15.32:0/2037200363 >> v2:172.21.15.86:3300/0 conn(0x7fcb4c0cffa0 0x7fcb4c0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb4bfff700 1 -- 172.21.15.32:0/2037200363 >> v1:172.21.15.86:6789/0 conn(0x7fcb4c0d8220 legacy=0x7fcb4c0d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb4bfff700 1 -- 172.21.15.32:0/2037200363 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c10e200 con 0x7fcb4c0d7800 2021-04-09T18:13:14.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb4bfff700 1 --2- 172.21.15.32:0/2037200363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c0d7800 0x7fcb4c0d7c00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fcb34006700 tx=0x7fcb34009b80).ready entity=mon.0 client_cookie=149552d7dc8ed134 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:14.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.301 7fcb4b7fe700 1 -- 172.21.15.32:0/2037200363 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3400c0c0 con 0x7fcb4c0d7800 2021-04-09T18:13:14.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb4b7fe700 1 -- 172.21.15.32:0/2037200363 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3400c440 con 0x7fcb4c0d7800 2021-04-09T18:13:14.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb4b7fe700 1 -- 172.21.15.32:0/2037200363 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3400c920 con 0x7fcb4c0d7800 2021-04-09T18:13:14.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb52a81700 1 -- 172.21.15.32:0/2037200363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c0d7800 msgr2=0x7fcb4c0d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb52a81700 1 --2- 172.21.15.32:0/2037200363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c0d7800 0x7fcb4c0d7c00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fcb34006700 tx=0x7fcb34009b80).stop 2021-04-09T18:13:14.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb52a81700 1 -- 172.21.15.32:0/2037200363 shutdown_connections 2021-04-09T18:13:14.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb52a81700 1 --2- 172.21.15.32:0/2037200363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c0d7800 0x7fcb4c0d7c00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb52a81700 1 --2- 172.21.15.32:0/2037200363 >> v2:172.21.15.86:3300/0 conn(0x7fcb4c0cffa0 0x7fcb4c0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb52a81700 1 -- 172.21.15.32:0/2037200363 >> 172.21.15.32:0/2037200363 conn(0x7fcb4c0ce5c0 msgr2=0x7fcb4c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:14.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb52a81700 1 -- 172.21.15.32:0/2037200363 shutdown_connections 2021-04-09T18:13:14.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.302 7fcb52a81700 1 -- 172.21.15.32:0/2037200363 wait complete. 2021-04-09T18:13:14.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.322 7fcb52a81700 1 Processor -- start 2021-04-09T18:13:14.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.322 7fcb52a81700 1 -- start start 2021-04-09T18:13:14.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb52a81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c1acff0 0x7fcb4c1847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb52a81700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb4c183fc0 0x7fcb4c1a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb52a81700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb4c186ff0 0x7fcb4c1873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb52a81700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c188c30 con 0x7fcb4c1acff0 2021-04-09T18:13:14.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb52a81700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c1843c0 con 0x7fcb4c186ff0 2021-04-09T18:13:14.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb52a81700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c1a5c20 con 0x7fcb4c183fc0 2021-04-09T18:13:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb5089d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c1acff0 0x7fcb4c1847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb5089d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c1acff0 0x7fcb4c1847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42222/0 (socket says 172.21.15.32:42222) 2021-04-09T18:13:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb5089d700 1 -- 172.21.15.32:0/749031782 learned_addr learned my addr 172.21.15.32:0/749031782 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:14.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb4bfff700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb4c183fc0 0x7fcb4c1a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.323 7fcb5109e700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb4c186ff0 0x7fcb4c1873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb5089d700 1 -- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb4c183fc0 msgr2=0x7fcb4c1a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb5089d700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb4c183fc0 0x7fcb4c1a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb5089d700 1 -- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb4c186ff0 msgr2=0x7fcb4c1873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb5089d700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb4c186ff0 0x7fcb4c1873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb5089d700 1 -- 172.21.15.32:0/749031782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c187a10 con 0x7fcb4c1acff0 2021-04-09T18:13:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb4bfff700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb4c183fc0 0x7fcb4c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:14.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb5089d700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c1acff0 0x7fcb4c1847a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fcb3c0099e0 tx=0x7fcb3c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:14.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb497fa700 1 -- 172.21.15.32:0/749031782 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3c00c2f0 con 0x7fcb4c1acff0 2021-04-09T18:13:14.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb497fa700 1 -- 172.21.15.32:0/749031782 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c00c630 con 0x7fcb4c1acff0 2021-04-09T18:13:14.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb52a81700 1 -- 172.21.15.32:0/749031782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c187a10 con 0x7fcb4c1acff0 2021-04-09T18:13:14.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb497fa700 1 -- 172.21.15.32:0/749031782 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3c00cb10 con 0x7fcb4c1acff0 2021-04-09T18:13:14.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.324 7fcb52a81700 1 -- 172.21.15.32:0/749031782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c187a10 con 0x7fcb4c1acff0 2021-04-09T18:13:14.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.325 7fcb497fa700 1 -- 172.21.15.32:0/749031782 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcb3c00a020 con 0x7fcb4c1acff0 2021-04-09T18:13:14.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.325 7fcb497fa700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb300188c0 0x7fcb3001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.325 7fcb497fa700 1 -- 172.21.15.32:0/749031782 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcb3c032b40 con 0x7fcb4c1acff0 2021-04-09T18:13:14.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.325 7fcb4bfff700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb300188c0 0x7fcb3001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.325 7fcb4bfff700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb300188c0 0x7fcb3001adc0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.345 7fcb52a81700 1 -- 172.21.15.32:0/749031782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c182dc0 con 0x7fcb4c1acff0 2021-04-09T18:13:14.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.348 7fcb497fa700 1 -- 172.21.15.32:0/749031782 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcb3c00c2f0 con 0x7fcb4c1acff0 2021-04-09T18:13:14.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.538 7fcb52a81700 1 -- 172.21.15.32:0/749031782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb4c0d7280 con 0x7fcb4c1acff0 2021-04-09T18:13:14.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.539 7fcb497fa700 1 -- 172.21.15.32:0/749031782 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fcb3c009fe0 con 0x7fcb4c1acff0 2021-04-09T18:13:14.615 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:14.615 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:12:34.773743","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:12:32.136283","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-04-09 18:12:32.831693","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 -- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb300188c0 msgr2=0x7fcb3001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb300188c0 0x7fcb3001adc0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 -- 172.21.15.32:0/749031782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c1acff0 msgr2=0x7fcb4c1847a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c1acff0 0x7fcb4c1847a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fcb3c0099e0 tx=0x7fcb3c00bd40).stop 2021-04-09T18:13:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 -- 172.21.15.32:0/749031782 shutdown_connections 2021-04-09T18:13:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb300188c0 0x7fcb3001adc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb4c186ff0 0x7fcb4c1873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb4c183fc0 0x7fcb4c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 --2- 172.21.15.32:0/749031782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb4c1acff0 0x7fcb4c1847a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.541 7fcb52a81700 1 -- 172.21.15.32:0/749031782 >> 172.21.15.32:0/749031782 conn(0x7fcb4c09abb0 msgr2=0x7fcb4c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.542 7fcb52a81700 1 -- 172.21.15.32:0/749031782 shutdown_connections 2021-04-09T18:13:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.542 7fcb52a81700 1 -- 172.21.15.32:0/749031782 wait complete. 2021-04-09T18:13:14.639 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:12:32.136283', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:12:32.831693', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:13:14.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:13:14.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.797 7f543734c700 1 Processor -- start 2021-04-09T18:13:14.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.797 7f543734c700 1 -- start start 2021-04-09T18:13:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f543734c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f543010d010 0x7f54300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f543734c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54300d8490 0x7f543010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f543734c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543010d480 con 0x7f54300d1da0 2021-04-09T18:13:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f543734c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f543010e780 con 0x7f54300d8490 2021-04-09T18:13:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f543734c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54300d8d30 con 0x7f543010d010 2021-04-09T18:13:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5435969700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54300d8490 0x7f543010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5434967700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f54300d1da0 0x7f54300d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51820/0 (socket says 172.21.15.32:51820) 2021-04-09T18:13:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5434967700 1 -- 172.21.15.32:0/984239815 learned_addr learned my addr 172.21.15.32:0/984239815 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:14.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5435168700 1 --2- 172.21.15.32:0/984239815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f543010d010 0x7f54300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5427fff700 1 -- 172.21.15.32:0/984239815 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655576235 0 0) 0x7f54200009b0 con 0x7f54300d1da0 2021-04-09T18:13:14.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5427fff700 1 -- 172.21.15.32:0/984239815 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f541c002fe0 con 0x7f54300d1da0 2021-04-09T18:13:14.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5435969700 1 -- 172.21.15.32:0/984239815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f543010d010 msgr2=0x7f54300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5435969700 1 --2- 172.21.15.32:0/984239815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f543010d010 0x7f54300d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5435969700 1 -- 172.21.15.32:0/984239815 >> v1:172.21.15.32:6789/0 conn(0x7f54300d1da0 legacy=0x7f54300d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5435969700 1 -- 172.21.15.32:0/984239815 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54300d9220 con 0x7f54300d8490 2021-04-09T18:13:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.798 7f5435969700 1 --2- 172.21.15.32:0/984239815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300d8490 0x7f543010da00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5428006e30 tx=0x7f5428009de0).ready entity=mon.0 client_cookie=e8ef36f9be4b51b0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.799 7f5427fff700 1 -- 172.21.15.32:0/984239815 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f542800b300 con 0x7f54300d8490 2021-04-09T18:13:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.799 7f5427fff700 1 -- 172.21.15.32:0/984239815 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542800b680 con 0x7f54300d8490 2021-04-09T18:13:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.799 7f5427fff700 1 -- 172.21.15.32:0/984239815 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f542800bb60 con 0x7f54300d8490 2021-04-09T18:13:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.799 7f543734c700 1 -- 172.21.15.32:0/984239815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300d8490 msgr2=0x7f543010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.799 7f543734c700 1 --2- 172.21.15.32:0/984239815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300d8490 0x7f543010da00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5428006e30 tx=0x7f5428009de0).stop 2021-04-09T18:13:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.799 7f543734c700 1 -- 172.21.15.32:0/984239815 shutdown_connections 2021-04-09T18:13:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.799 7f543734c700 1 --2- 172.21.15.32:0/984239815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300d8490 0x7f543010da00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.799 7f543734c700 1 --2- 172.21.15.32:0/984239815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f543010d010 0x7f54300d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.799 7f543734c700 1 -- 172.21.15.32:0/984239815 >> 172.21.15.32:0/984239815 conn(0x7f54300ce5f0 msgr2=0x7f54300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.800 7f543734c700 1 -- 172.21.15.32:0/984239815 shutdown_connections 2021-04-09T18:13:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.800 7f543734c700 1 -- 172.21.15.32:0/984239815 wait complete. 2021-04-09T18:13:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.820 7f543734c700 1 Processor -- start 2021-04-09T18:13:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.820 7f543734c700 1 -- start start 2021-04-09T18:13:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.820 7f543734c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54300a0110 0x7f543018c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.820 7f543734c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300a0510 0x7f54300c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.820 7f543734c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54300c36e0 0x7f54300c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.820 7f543734c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543018cf60 con 0x7f54300a0510 2021-04-09T18:13:14.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.820 7f543734c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5430199ea0 con 0x7f54300c36e0 2021-04-09T18:13:14.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.820 7f543734c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5430187d70 con 0x7f54300a0110 2021-04-09T18:13:14.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.820 7f5434967700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300a0510 0x7f54300c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5435168700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54300a0110 0x7f543018c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5434967700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300a0510 0x7f54300c30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42234/0 (socket says 172.21.15.32:42234) 2021-04-09T18:13:14.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5434967700 1 -- 172.21.15.32:0/3666582900 learned_addr learned my addr 172.21.15.32:0/3666582900 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:14.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5435969700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54300c36e0 0x7f54300c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5434967700 1 -- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54300a0110 msgr2=0x7f543018c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5434967700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54300a0110 0x7f543018c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5434967700 1 -- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54300c36e0 msgr2=0x7f54300c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5434967700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54300c36e0 0x7f54300c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5434967700 1 -- 172.21.15.32:0/3666582900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543018d4e0 con 0x7f54300a0510 2021-04-09T18:13:14.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5434967700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300a0510 0x7f54300c30c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f542000a4c0 tx=0x7f542000a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:14.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5425ffb700 1 -- 172.21.15.32:0/3666582900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f542000ac50 con 0x7f54300a0510 2021-04-09T18:13:14.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5425ffb700 1 -- 172.21.15.32:0/3666582900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5420004140 con 0x7f54300a0510 2021-04-09T18:13:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f543734c700 1 -- 172.21.15.32:0/3666582900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543018d4e0 con 0x7f54300a0510 2021-04-09T18:13:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f5425ffb700 1 -- 172.21.15.32:0/3666582900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5420004620 con 0x7f54300a0510 2021-04-09T18:13:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.821 7f543734c700 1 -- 172.21.15.32:0/3666582900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543018d4e0 con 0x7f54300a0510 2021-04-09T18:13:14.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.822 7f5425ffb700 1 -- 172.21.15.32:0/3666582900 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5420004320 con 0x7f54300a0510 2021-04-09T18:13:14.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.822 7f5425ffb700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5410018930 0x7f541001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:14.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.822 7f5425ffb700 1 -- 172.21.15.32:0/3666582900 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f54200329d0 con 0x7f54300a0510 2021-04-09T18:13:14.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.822 7f5435168700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5410018930 0x7f541001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:14.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.822 7f5435168700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5410018930 0x7f541001ae10 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:14.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.842 7f543734c700 1 -- 172.21.15.32:0/3666582900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543010d480 con 0x7f54300a0510 2021-04-09T18:13:14.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:14.845 7f5425ffb700 1 -- 172.21.15.32:0/3666582900 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f54200009b0 con 0x7f54300a0510 2021-04-09T18:13:15.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.024 7f543734c700 1 -- 172.21.15.32:0/3666582900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f543010d480 con 0x7f54300a0510 2021-04-09T18:13:15.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.024 7f5425ffb700 1 -- 172.21.15.32:0/3666582900 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5420004140 con 0x7f54300a0510 2021-04-09T18:13:15.045 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:15.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.026 7f543734c700 1 -- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5410018930 msgr2=0x7f541001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.026 7f543734c700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5410018930 0x7f541001ae10 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.026 7f543734c700 1 -- 172.21.15.32:0/3666582900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300a0510 msgr2=0x7f54300c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.026 7f543734c700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300a0510 0x7f54300c30c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f542000a4c0 tx=0x7f542000a860).stop 2021-04-09T18:13:15.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.027 7f543734c700 1 -- 172.21.15.32:0/3666582900 shutdown_connections 2021-04-09T18:13:15.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.027 7f543734c700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5410018930 0x7f541001ae10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.027 7f543734c700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54300a0510 0x7f54300c30c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.027 7f543734c700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54300c36e0 0x7f54300c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.027 7f543734c700 1 --2- 172.21.15.32:0/3666582900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54300a0110 0x7f543018c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.027 7f543734c700 1 -- 172.21.15.32:0/3666582900 >> 172.21.15.32:0/3666582900 conn(0x7f54300b93a0 msgr2=0x7f54300b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:15.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.027 7f543734c700 1 -- 172.21.15.32:0/3666582900 shutdown_connections 2021-04-09T18:13:15.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.027 7f543734c700 1 -- 172.21.15.32:0/3666582900 wait complete. 2021-04-09T18:13:15.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:13:15.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.229 7f9dafe36700 1 Processor -- start 2021-04-09T18:13:15.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.229 7f9dafe36700 1 -- start start 2021-04-09T18:13:15.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.230 7f9dafe36700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da810d010 0x7f9da80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.230 7f9dafe36700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9da80d1da0 0x7f9da80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.230 7f9dafe36700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da810d480 con 0x7f9da80d82c0 2021-04-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.230 7f9dafe36700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da810e780 con 0x7f9da80d1da0 2021-04-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.230 7f9dafe36700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80d8d30 con 0x7f9da810d010 2021-04-09T18:13:15.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.230 7f9dad451700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9da80d1da0 0x7f9da80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.230 7f9dae453700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9da80d82c0 0x7f9da810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51832/0 (socket says 172.21.15.32:51832) 2021-04-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.230 7f9dae453700 1 -- 172.21.15.32:0/499428889 learned_addr learned my addr 172.21.15.32:0/499428889 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.230 7f9dadc52700 1 --2- 172.21.15.32:0/499428889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da810d010 0x7f9da80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dacc50700 1 -- 172.21.15.32:0/499428889 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122981355 0 0) 0x7f9da0000d30 con 0x7f9da80d82c0 2021-04-09T18:13:15.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dadc52700 1 -- 172.21.15.32:0/499428889 >> v2:172.21.15.32:3300/0 conn(0x7f9da80d1da0 msgr2=0x7f9da80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dadc52700 1 --2- 172.21.15.32:0/499428889 >> v2:172.21.15.32:3300/0 conn(0x7f9da80d1da0 0x7f9da80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dadc52700 1 -- 172.21.15.32:0/499428889 >> v1:172.21.15.32:6789/0 conn(0x7f9da80d82c0 legacy=0x7f9da810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dadc52700 1 -- 172.21.15.32:0/499428889 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da80d9220 con 0x7f9da810d010 2021-04-09T18:13:15.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dad451700 1 --2- 172.21.15.32:0/499428889 >> v2:172.21.15.32:3300/0 conn(0x7f9da80d1da0 0x7f9da80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dadc52700 1 --2- 172.21.15.32:0/499428889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da810d010 0x7f9da80d1780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f9da4006650 tx=0x7f9da4009a80).ready entity=mon.2 client_cookie=b057cef9684e42b2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dacc50700 1 -- 172.21.15.32:0/499428889 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9da40009b0 con 0x7f9da810d010 2021-04-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dacc50700 1 -- 172.21.15.32:0/499428889 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da400c0c0 con 0x7f9da810d010 2021-04-09T18:13:15.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dacc50700 1 -- 172.21.15.32:0/499428889 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9da400c510 con 0x7f9da810d010 2021-04-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dafe36700 1 -- 172.21.15.32:0/499428889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da810d010 msgr2=0x7f9da80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.231 7f9dafe36700 1 --2- 172.21.15.32:0/499428889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da810d010 0x7f9da80d1780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f9da4006650 tx=0x7f9da4009a80).stop 2021-04-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.232 7f9dafe36700 1 -- 172.21.15.32:0/499428889 shutdown_connections 2021-04-09T18:13:15.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.232 7f9dafe36700 1 --2- 172.21.15.32:0/499428889 >> v2:172.21.15.32:3300/0 conn(0x7f9da80d1da0 0x7f9da80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.232 7f9dafe36700 1 --2- 172.21.15.32:0/499428889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da810d010 0x7f9da80d1780 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.232 7f9dafe36700 1 -- 172.21.15.32:0/499428889 >> 172.21.15.32:0/499428889 conn(0x7f9da80ce5f0 msgr2=0x7f9da80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.232 7f9dafe36700 1 -- 172.21.15.32:0/499428889 shutdown_connections 2021-04-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.232 7f9dafe36700 1 -- 172.21.15.32:0/499428889 wait complete. 2021-04-09T18:13:15.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.252 7f9dafe36700 1 Processor -- start 2021-04-09T18:13:15.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.252 7f9dafe36700 1 -- start start 2021-04-09T18:13:15.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dafe36700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da81a4380 0x7f9da81a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dafe36700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da80bdfa0 0x7f9da80bfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dafe36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da80c0480 0x7f9da80be500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dafe36700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80bbb00 con 0x7f9da80c0480 2021-04-09T18:13:15.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dafe36700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80bb6b0 con 0x7f9da81a4380 2021-04-09T18:13:15.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dafe36700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80c0880 con 0x7f9da80bdfa0 2021-04-09T18:13:15.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dae453700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da80c0480 0x7f9da80be500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dae453700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da80c0480 0x7f9da80be500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42250/0 (socket says 172.21.15.32:42250) 2021-04-09T18:13:15.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dae453700 1 -- 172.21.15.32:0/2164401481 learned_addr learned my addr 172.21.15.32:0/2164401481 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dadc52700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da81a4380 0x7f9da81a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dad451700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da80bdfa0 0x7f9da80bfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dae453700 1 -- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da80bdfa0 msgr2=0x7f9da80bfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dae453700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da80bdfa0 0x7f9da80bfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dae453700 1 -- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da81a4380 msgr2=0x7f9da81a4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dae453700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da81a4380 0x7f9da81a4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dae453700 1 -- 172.21.15.32:0/2164401481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da80c0aa0 con 0x7f9da80c0480 2021-04-09T18:13:15.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dadc52700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da81a4380 0x7f9da81a4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:15.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dae453700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da80c0480 0x7f9da80be500 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f9da0007f90 tx=0x7f9da000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9d9a7fc700 1 -- 172.21.15.32:0/2164401481 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9da0000cc0 con 0x7f9da80c0480 2021-04-09T18:13:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9d9a7fc700 1 -- 172.21.15.32:0/2164401481 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da0007020 con 0x7f9da80c0480 2021-04-09T18:13:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da80c0aa0 con 0x7f9da80c0480 2021-04-09T18:13:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.253 7f9d9a7fc700 1 -- 172.21.15.32:0/2164401481 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9da0007500 con 0x7f9da80c0480 2021-04-09T18:13:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.254 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da80c0aa0 con 0x7f9da80c0480 2021-04-09T18:13:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.254 7f9d9a7fc700 1 -- 172.21.15.32:0/2164401481 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9da0007200 con 0x7f9da80c0480 2021-04-09T18:13:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.254 7f9d9a7fc700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9d90018930 0x7f9d9001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.254 7f9d9a7fc700 1 -- 172.21.15.32:0/2164401481 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9da0032d50 con 0x7f9da80c0480 2021-04-09T18:13:15.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.254 7f9dadc52700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9d90018930 0x7f9d9001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.255 7f9dadc52700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9d90018930 0x7f9d9001ae10 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:15.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.274 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da8093d50 con 0x7f9da80c0480 2021-04-09T18:13:15.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.278 7f9d9a7fc700 1 -- 172.21.15.32:0/2164401481 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9da0007d00 con 0x7f9da80c0480 2021-04-09T18:13:15.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.457 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f9da8093d50 con 0x7f9da80c0480 2021-04-09T18:13:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.458 7f9d9a7fc700 1 -- 172.21.15.32:0/2164401481 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9da0006f00 con 0x7f9da80c0480 2021-04-09T18:13:15.480 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.460 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9d90018930 msgr2=0x7f9d9001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.460 7f9dafe36700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9d90018930 0x7f9d9001ae10 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.460 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da80c0480 msgr2=0x7f9da80be500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.460 7f9dafe36700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da80c0480 0x7f9da80be500 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f9da0007f90 tx=0x7f9da000ac50).stop 2021-04-09T18:13:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.460 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 shutdown_connections 2021-04-09T18:13:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.460 7f9dafe36700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9d90018930 0x7f9d9001ae10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.460 7f9dafe36700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da80c0480 0x7f9da80be500 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.461 7f9dafe36700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9da80bdfa0 0x7f9da80bfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.461 7f9dafe36700 1 --2- 172.21.15.32:0/2164401481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9da81a4380 0x7f9da81a4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.461 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 >> 172.21.15.32:0/2164401481 conn(0x7f9da8189860 msgr2=0x7f9da8094e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:15.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.461 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 shutdown_connections 2021-04-09T18:13:15.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.461 7f9dafe36700 1 -- 172.21.15.32:0/2164401481 wait complete. 2021-04-09T18:13:15.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:13:15.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.634 7efdfc0ee700 1 Processor -- start 2021-04-09T18:13:15.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.634 7efdfc0ee700 1 -- start start 2021-04-09T18:13:15.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfc0ee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40d7ca0 0x7efdf40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfc0ee700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efdf410da00 0x7efdf410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfc0ee700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf410d480 con 0x7efdf410d010 2021-04-09T18:13:15.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfc0ee700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf40d9220 con 0x7efdf410da00 2021-04-09T18:13:15.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfc0ee700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40d86c0 con 0x7efdf40d7ca0 2021-04-09T18:13:15.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfa70b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efdf410da00 0x7efdf410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdf9709700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40d7ca0 0x7efdf40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfa70b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efdf410da00 0x7efdf410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42256/0 (socket says 172.21.15.32:42256) 2021-04-09T18:13:15.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfa70b700 1 -- 172.21.15.32:0/3287953062 learned_addr learned my addr 172.21.15.32:0/3287953062 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:15.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdf9f0a700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efdf410d010 0x7efdf40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:49770/0 (socket says 172.21.15.32:49770) 2021-04-09T18:13:15.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdf9709700 1 --2- 172.21.15.32:0/3287953062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40d7ca0 0x7efdf40d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:49484/0 (socket says 172.21.15.32:49484) 2021-04-09T18:13:15.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfa70b700 1 -- 172.21.15.32:0/3287953062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40d7ca0 msgr2=0x7efdf40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:15.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdf8f08700 1 -- 172.21.15.32:0/3287953062 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284327095 0 0) 0x7efdf0000c60 con 0x7efdf410d010 2021-04-09T18:13:15.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfa70b700 1 --2- 172.21.15.32:0/3287953062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40d7ca0 0x7efdf40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfa70b700 1 -- 172.21.15.32:0/3287953062 >> v1:172.21.15.86:6789/0 conn(0x7efdf410d010 legacy=0x7efdf40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfa70b700 1 -- 172.21.15.32:0/3287953062 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4112100 con 0x7efdf410da00 2021-04-09T18:13:15.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdf9709700 1 --2- 172.21.15.32:0/3287953062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40d7ca0 0x7efdf40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:15.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.635 7efdfa70b700 1 --2- 172.21.15.32:0/3287953062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf410da00 0x7efdf410de00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7efdec006f00 tx=0x7efdec009f10).ready entity=mon.0 client_cookie=6ab885c6c5ccf497 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:15.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.636 7efdf8f08700 1 -- 172.21.15.32:0/3287953062 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdec00b3b0 con 0x7efdf410da00 2021-04-09T18:13:15.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.636 7efdf8f08700 1 -- 172.21.15.32:0/3287953062 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec00b730 con 0x7efdf410da00 2021-04-09T18:13:15.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.636 7efdf8f08700 1 -- 172.21.15.32:0/3287953062 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdec00bc10 con 0x7efdf410da00 2021-04-09T18:13:15.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.636 7efdfc0ee700 1 -- 172.21.15.32:0/3287953062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf410da00 msgr2=0x7efdf410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.636 7efdfc0ee700 1 --2- 172.21.15.32:0/3287953062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf410da00 0x7efdf410de00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7efdec006f00 tx=0x7efdec009f10).stop 2021-04-09T18:13:15.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.636 7efdfc0ee700 1 -- 172.21.15.32:0/3287953062 shutdown_connections 2021-04-09T18:13:15.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.636 7efdfc0ee700 1 --2- 172.21.15.32:0/3287953062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf410da00 0x7efdf410de00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.636 7efdfc0ee700 1 --2- 172.21.15.32:0/3287953062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40d7ca0 0x7efdf40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.636 7efdfc0ee700 1 -- 172.21.15.32:0/3287953062 >> 172.21.15.32:0/3287953062 conn(0x7efdf40ce5f0 msgr2=0x7efdf40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.637 7efdfc0ee700 1 -- 172.21.15.32:0/3287953062 shutdown_connections 2021-04-09T18:13:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.637 7efdfc0ee700 1 -- 172.21.15.32:0/3287953062 wait complete. 2021-04-09T18:13:15.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdfc0ee700 1 Processor -- start 2021-04-09T18:13:15.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdfc0ee700 1 -- start start 2021-04-09T18:13:15.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdfc0ee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40a0cb0 0x7efdf4092550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdfc0ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf40932e0 0x7efdf408a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdfc0ee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdf40936e0 0x7efdf408ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdfc0ee700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40b34f0 con 0x7efdf40932e0 2021-04-09T18:13:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdfc0ee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4092d60 con 0x7efdf40936e0 2021-04-09T18:13:15.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdfc0ee700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40900a0 con 0x7efdf40a0cb0 2021-04-09T18:13:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdf9709700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf40932e0 0x7efdf408a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdf9709700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf40932e0 0x7efdf408a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42260/0 (socket says 172.21.15.32:42260) 2021-04-09T18:13:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdf9709700 1 -- 172.21.15.32:0/1519113172 learned_addr learned my addr 172.21.15.32:0/1519113172 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:15.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdf9f0a700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40a0cb0 0x7efdf4092550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.657 7efdfa70b700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdf40936e0 0x7efdf408ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efdf9709700 1 -- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40a0cb0 msgr2=0x7efdf4092550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efdf9709700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40a0cb0 0x7efdf4092550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efdf9709700 1 -- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdf40936e0 msgr2=0x7efdf408ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efdf9709700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdf40936e0 0x7efdf408ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efdf9709700 1 -- 172.21.15.32:0/1519113172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf40a10b0 con 0x7efdf40932e0 2021-04-09T18:13:15.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efdfa70b700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdf40936e0 0x7efdf408ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:15.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efdf9709700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf40932e0 0x7efdf408a620 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7efde8006f40 tx=0x7efde800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:15.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efde67fc700 1 -- 172.21.15.32:0/1519113172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efde800bd80 con 0x7efdf40932e0 2021-04-09T18:13:15.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efde67fc700 1 -- 172.21.15.32:0/1519113172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde8000c90 con 0x7efdf40932e0 2021-04-09T18:13:15.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf40a10b0 con 0x7efdf40932e0 2021-04-09T18:13:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efde67fc700 1 -- 172.21.15.32:0/1519113172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efde8002220 con 0x7efdf40932e0 2021-04-09T18:13:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.658 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf40a10b0 con 0x7efdf40932e0 2021-04-09T18:13:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.659 7efde67fc700 1 -- 172.21.15.32:0/1519113172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efde800a000 con 0x7efdf40932e0 2021-04-09T18:13:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.659 7efde67fc700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efddc018930 0x7efddc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:15.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.659 7efde67fc700 1 -- 172.21.15.32:0/1519113172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efde8032a10 con 0x7efdf40932e0 2021-04-09T18:13:15.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.659 7efdf9f0a700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efddc018930 0x7efddc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:15.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.659 7efdf9f0a700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efddc018930 0x7efddc01ae10 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.679 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf410d480 con 0x7efdf40932e0 2021-04-09T18:13:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.682 7efde67fc700 1 -- 172.21.15.32:0/1519113172 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efde8006b00 con 0x7efdf40932e0 2021-04-09T18:13:15.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.861 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7efdf410d480 con 0x7efdf40932e0 2021-04-09T18:13:15.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.862 7efde67fc700 1 -- 172.21.15.32:0/1519113172 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7efde800a000 con 0x7efdf40932e0 2021-04-09T18:13:15.885 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:15.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efddc018930 msgr2=0x7efddc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efddc018930 0x7efddc01ae10 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf40932e0 msgr2=0x7efdf408a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:15.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf40932e0 0x7efdf408a620 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7efde8006f40 tx=0x7efde800b990).stop 2021-04-09T18:13:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 shutdown_connections 2021-04-09T18:13:15.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efddc018930 0x7efddc01ae10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdf40932e0 0x7efdf408a620 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdf40936e0 0x7efdf408ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 --2- 172.21.15.32:0/1519113172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdf40a0cb0 0x7efdf4092550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 >> 172.21.15.32:0/1519113172 conn(0x7efdf40b2fa0 msgr2=0x7efdf40a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 shutdown_connections 2021-04-09T18:13:15.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:15.864 7efdfc0ee700 1 -- 172.21.15.32:0/1519113172 wait complete. 2021-04-09T18:13:16.122 INFO:tasks.cephfs_test_runner:Starting test: test_down (tasks.cephfs.test_failover.TestClusterResize) 2021-04-09T18:13:16.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down' 2021-04-09T18:13:16.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.290 7f4026dd2700 1 Processor -- start 2021-04-09T18:13:16.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.290 7f4026dd2700 1 -- start start 2021-04-09T18:13:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4026dd2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f40200cffa0 0x7f40200d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4026dd2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f40200d6c80 0x7f402010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4026dd2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40200d7280 con 0x7f40200d7800 2021-04-09T18:13:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4026dd2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40200d7080 con 0x7f40200cffa0 2021-04-09T18:13:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4026dd2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40200d91e0 con 0x7f40200d6c80 2021-04-09T18:13:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4024bee700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f40200cffa0 0x7f40200d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4024bee700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f40200cffa0 0x7f40200d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42268/0 (socket says 172.21.15.32:42268) 2021-04-09T18:13:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4024bee700 1 -- 172.21.15.32:0/3612086468 learned_addr learned my addr 172.21.15.32:0/3612086468 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f40253ef700 1 --2- 172.21.15.32:0/3612086468 >> v2:172.21.15.86:3300/0 conn(0x7f40200d6c80 0x7f402010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4024bee700 1 -- 172.21.15.32:0/3612086468 >> v2:172.21.15.86:3300/0 conn(0x7f40200d6c80 msgr2=0x7f402010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:16.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4024bee700 1 --2- 172.21.15.32:0/3612086468 >> v2:172.21.15.86:3300/0 conn(0x7f40200d6c80 0x7f402010cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:16.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4024bee700 1 -- 172.21.15.32:0/3612086468 >> v1:172.21.15.86:6789/0 conn(0x7f40200d7800 legacy=0x7f40200d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:16.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4024bee700 1 -- 172.21.15.32:0/3612086468 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40200d83f0 con 0x7f40200cffa0 2021-04-09T18:13:16.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f4024bee700 1 --2- 172.21.15.32:0/3612086468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40200cffa0 0x7f40200d2440 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4010006a80 tx=0x7f4010009f00).ready entity=mon.0 client_cookie=7cd97dd162425b5e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:16.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f401f7fe700 1 -- 172.21.15.32:0/3612086468 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f401000b5c0 con 0x7f40200cffa0 2021-04-09T18:13:16.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f401f7fe700 1 -- 172.21.15.32:0/3612086468 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401000b8c0 con 0x7f40200cffa0 2021-04-09T18:13:16.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.291 7f401f7fe700 1 -- 172.21.15.32:0/3612086468 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f401000bda0 con 0x7f40200cffa0 2021-04-09T18:13:16.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.292 7f4026dd2700 1 -- 172.21.15.32:0/3612086468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40200cffa0 msgr2=0x7f40200d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:16.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.292 7f4026dd2700 1 --2- 172.21.15.32:0/3612086468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40200cffa0 0x7f40200d2440 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4010006a80 tx=0x7f4010009f00).stop 2021-04-09T18:13:16.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.292 7f4026dd2700 1 -- 172.21.15.32:0/3612086468 shutdown_connections 2021-04-09T18:13:16.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.292 7f4026dd2700 1 --2- 172.21.15.32:0/3612086468 >> v2:172.21.15.86:3300/0 conn(0x7f40200d6c80 0x7f402010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.292 7f4026dd2700 1 --2- 172.21.15.32:0/3612086468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40200cffa0 0x7f40200d2440 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.292 7f4026dd2700 1 -- 172.21.15.32:0/3612086468 >> 172.21.15.32:0/3612086468 conn(0x7f40200ce5c0 msgr2=0x7f40200cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.292 7f4026dd2700 1 -- 172.21.15.32:0/3612086468 shutdown_connections 2021-04-09T18:13:16.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.292 7f4026dd2700 1 -- 172.21.15.32:0/3612086468 wait complete. 2021-04-09T18:13:16.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.312 7f4026dd2700 1 Processor -- start 2021-04-09T18:13:16.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.312 7f4026dd2700 1 -- start start 2021-04-09T18:13:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f4026dd2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40201842e0 0x7f402018dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f4026dd2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402019da40 0x7f402019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f4026dd2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f402019ec20 0x7f402019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f4026dd2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402019e6a0 con 0x7f40201842e0 2021-04-09T18:13:16.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f4026dd2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402019ca60 con 0x7f402019ec20 2021-04-09T18:13:16.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f4026dd2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402019f640 con 0x7f402019da40 2021-04-09T18:13:16.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f4024bee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40201842e0 0x7f402018dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:16.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f4024bee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40201842e0 0x7f402018dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42274/0 (socket says 172.21.15.32:42274) 2021-04-09T18:13:16.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f4024bee700 1 -- 172.21.15.32:0/3064541399 learned_addr learned my addr 172.21.15.32:0/3064541399 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:16.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f40253ef700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f402019ec20 0x7f402019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.313 7f401ffff700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402019da40 0x7f402019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f4024bee700 1 -- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402019da40 msgr2=0x7f402019de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:16.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f4024bee700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402019da40 0x7f402019de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f4024bee700 1 -- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f402019ec20 msgr2=0x7f402019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:16.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f4024bee700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f402019ec20 0x7f402019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f4024bee700 1 -- 172.21.15.32:0/3064541399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020184b10 con 0x7f40201842e0 2021-04-09T18:13:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f4024bee700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40201842e0 0x7f402018dce0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4010009450 tx=0x7f401000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f401d7fa700 1 -- 172.21.15.32:0/3064541399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4010006e40 con 0x7f40201842e0 2021-04-09T18:13:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f401d7fa700 1 -- 172.21.15.32:0/3064541399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401000be60 con 0x7f40201842e0 2021-04-09T18:13:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4020184b10 con 0x7f40201842e0 2021-04-09T18:13:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f401d7fa700 1 -- 172.21.15.32:0/3064541399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f401000c4e0 con 0x7f40201842e0 2021-04-09T18:13:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4020184b10 con 0x7f40201842e0 2021-04-09T18:13:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.314 7f401d7fa700 1 -- 172.21.15.32:0/3064541399 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4010005660 con 0x7f40201842e0 2021-04-09T18:13:16.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.315 7f401d7fa700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4004018930 0x7f400401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:16.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.315 7f401d7fa700 1 -- 172.21.15.32:0/3064541399 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4010032db0 con 0x7f40201842e0 2021-04-09T18:13:16.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.315 7f401ffff700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4004018930 0x7f400401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:16.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.315 7f401ffff700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4004018930 0x7f400401ae10 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:16.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.335 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4020182e10 con 0x7f40201842e0 2021-04-09T18:13:16.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.338 7f401d7fa700 1 -- 172.21.15.32:0/3064541399 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f401001faf0 con 0x7f40201842e0 2021-04-09T18:13:16.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:16.524 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7f40200d7280 con 0x7f40201842e0 2021-04-09T18:13:17.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.350 7f401d7fa700 1 -- 172.21.15.32:0/3064541399 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v49) v1 ==== 136+0+0 (secure 0 0 0) 0x7f4010005660 con 0x7f40201842e0 2021-04-09T18:13:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4004018930 msgr2=0x7f400401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4004018930 0x7f400401ae10 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40201842e0 msgr2=0x7f402018dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:17.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40201842e0 0x7f402018dce0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4010009450 tx=0x7f401000bd20).stop 2021-04-09T18:13:17.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 shutdown_connections 2021-04-09T18:13:17.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4004018930 0x7f400401ae10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f402019ec20 0x7f402019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402019da40 0x7f402019de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 --2- 172.21.15.32:0/3064541399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f40201842e0 0x7f402018dce0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 >> 172.21.15.32:0/3064541399 conn(0x7f4020097550 msgr2=0x7f402018b920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:17.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 shutdown_connections 2021-04-09T18:13:17.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.392 7f4026dd2700 1 -- 172.21.15.32:0/3064541399 wait complete. 2021-04-09T18:13:17.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:13:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.606 7f145eb6e700 1 Processor -- start 2021-04-09T18:13:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.606 7f145eb6e700 1 -- start start 2021-04-09T18:13:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.607 7f145eb6e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145810d010 0x7f14580d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.607 7f145eb6e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f14580d1da0 0x7f14580d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.607 7f145eb6e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145810d480 con 0x7f14580d82c0 2021-04-09T18:13:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.607 7f145eb6e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f145810e780 con 0x7f14580d1da0 2021-04-09T18:13:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.607 7f145eb6e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14580d8d30 con 0x7f145810d010 2021-04-09T18:13:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.607 7f145d18b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f14580d82c0 0x7f145810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51874/0 (socket says 172.21.15.32:51874) 2021-04-09T18:13:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.607 7f145d18b700 1 -- 172.21.15.32:0/2873212943 learned_addr learned my addr 172.21.15.32:0/2873212943 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.607 7f145c98a700 1 --2- 172.21.15.32:0/2873212943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145810d010 0x7f14580d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.607 7f1457fff700 1 --2- 172.21.15.32:0/2873212943 >> v2:172.21.15.86:3300/0 conn(0x7f14580d1da0 0x7f14580d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:17.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f14577fe700 1 -- 172.21.15.32:0/2873212943 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004563006 0 0) 0x7f1450000d30 con 0x7f14580d82c0 2021-04-09T18:13:17.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f14577fe700 1 -- 172.21.15.32:0/2873212943 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1440002fe0 con 0x7f14580d82c0 2021-04-09T18:13:17.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f145c98a700 1 -- 172.21.15.32:0/2873212943 >> v2:172.21.15.86:3300/0 conn(0x7f14580d1da0 msgr2=0x7f14580d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:17.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f145c98a700 1 --2- 172.21.15.32:0/2873212943 >> v2:172.21.15.86:3300/0 conn(0x7f14580d1da0 0x7f14580d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f145c98a700 1 -- 172.21.15.32:0/2873212943 >> v1:172.21.15.32:6789/0 conn(0x7f14580d82c0 legacy=0x7f145810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:17.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f145c98a700 1 -- 172.21.15.32:0/2873212943 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14580d9220 con 0x7f145810d010 2021-04-09T18:13:17.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f1457fff700 1 --2- 172.21.15.32:0/2873212943 >> v2:172.21.15.86:3300/0 conn(0x7f14580d1da0 0x7f14580d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:17.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f145c98a700 1 --2- 172.21.15.32:0/2873212943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145810d010 0x7f14580d1780 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1448006650 tx=0x7f1448009a80).ready entity=mon.2 client_cookie=a03b64e635cf8e08 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:17.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f14577fe700 1 -- 172.21.15.32:0/2873212943 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14480009b0 con 0x7f145810d010 2021-04-09T18:13:17.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f14577fe700 1 -- 172.21.15.32:0/2873212943 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144800c0c0 con 0x7f145810d010 2021-04-09T18:13:17.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f14577fe700 1 -- 172.21.15.32:0/2873212943 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f144800c510 con 0x7f145810d010 2021-04-09T18:13:17.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f145eb6e700 1 -- 172.21.15.32:0/2873212943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145810d010 msgr2=0x7f14580d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:17.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.608 7f145eb6e700 1 --2- 172.21.15.32:0/2873212943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145810d010 0x7f14580d1780 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1448006650 tx=0x7f1448009a80).stop 2021-04-09T18:13:17.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.609 7f145eb6e700 1 -- 172.21.15.32:0/2873212943 shutdown_connections 2021-04-09T18:13:17.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.609 7f145eb6e700 1 --2- 172.21.15.32:0/2873212943 >> v2:172.21.15.86:3300/0 conn(0x7f14580d1da0 0x7f14580d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.609 7f145eb6e700 1 --2- 172.21.15.32:0/2873212943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145810d010 0x7f14580d1780 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.609 7f145eb6e700 1 -- 172.21.15.32:0/2873212943 >> 172.21.15.32:0/2873212943 conn(0x7f14580ce5f0 msgr2=0x7f14580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:17.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.609 7f145eb6e700 1 -- 172.21.15.32:0/2873212943 shutdown_connections 2021-04-09T18:13:17.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.609 7f145eb6e700 1 -- 172.21.15.32:0/2873212943 wait complete. 2021-04-09T18:13:17.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.629 7f145eb6e700 1 Processor -- start 2021-04-09T18:13:17.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.629 7f145eb6e700 1 -- start start 2021-04-09T18:13:17.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f145eb6e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14580b2140 0x7f145809d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:17.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f145eb6e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145809e010 0x7f1458194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:17.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f145eb6e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14581950a0 0x7f14580a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:17.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f145eb6e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145809d470 con 0x7f145809e010 2021-04-09T18:13:17.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f145eb6e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145809e410 con 0x7f14581950a0 2021-04-09T18:13:17.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f145eb6e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14581954a0 con 0x7f14580b2140 2021-04-09T18:13:17.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f1457fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145809e010 0x7f1458194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f1457fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145809e010 0x7f1458194a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42288/0 (socket says 172.21.15.32:42288) 2021-04-09T18:13:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f1457fff700 1 -- 172.21.15.32:0/469861044 learned_addr learned my addr 172.21.15.32:0/469861044 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:17.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f145d18b700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14581950a0 0x7f14580a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:17.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f145c98a700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14580b2140 0x7f145809d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:17.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f1457fff700 1 -- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14580b2140 msgr2=0x7f145809d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:17.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f1457fff700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14580b2140 0x7f145809d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f1457fff700 1 -- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14581950a0 msgr2=0x7f14580a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f1457fff700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14581950a0 0x7f14580a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.630 7f1457fff700 1 -- 172.21.15.32:0/469861044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14581956c0 con 0x7f145809e010 2021-04-09T18:13:17.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.631 7f1457fff700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145809e010 0x7f1458194a80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f144c00b4b0 tx=0x7f144c00b8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.631 7f14557fa700 1 -- 172.21.15.32:0/469861044 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f144c000c90 con 0x7f145809e010 2021-04-09T18:13:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.631 7f14557fa700 1 -- 172.21.15.32:0/469861044 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c00bcb0 con 0x7f145809e010 2021-04-09T18:13:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.631 7f14557fa700 1 -- 172.21.15.32:0/469861044 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f144c002220 con 0x7f145809e010 2021-04-09T18:13:17.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.631 7f145eb6e700 1 -- 172.21.15.32:0/469861044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14581956c0 con 0x7f145809e010 2021-04-09T18:13:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.631 7f145eb6e700 1 -- 172.21.15.32:0/469861044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14581956c0 con 0x7f145809e010 2021-04-09T18:13:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.631 7f14557fa700 1 -- 172.21.15.32:0/469861044 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f144c000c90 con 0x7f145809e010 2021-04-09T18:13:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.631 7f14557fa700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1438018930 0x7f143801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:17.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.631 7f14557fa700 1 -- 172.21.15.32:0/469861044 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f144c034c30 con 0x7f145809e010 2021-04-09T18:13:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.632 7f145c98a700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1438018930 0x7f143801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.632 7f145c98a700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1438018930 0x7f143801adc0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:17.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.651 7f145eb6e700 1 -- 172.21.15.32:0/469861044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14581b7040 con 0x7f145809e010 2021-04-09T18:13:17.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.655 7f14557fa700 1 -- 172.21.15.32:0/469861044 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f144c006b00 con 0x7f145809e010 2021-04-09T18:13:17.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.867 7f145eb6e700 1 -- 172.21.15.32:0/469861044 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f14581b7040 con 0x7f145809e010 2021-04-09T18:13:17.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.870 7f14557fa700 1 -- 172.21.15.32:0/469861044 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f144c000c90 con 0x7f145809e010 2021-04-09T18:13:17.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.876 7f145eb6e700 1 -- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1438018930 msgr2=0x7f143801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:17.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.876 7f145eb6e700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1438018930 0x7f143801adc0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.876 7f145eb6e700 1 -- 172.21.15.32:0/469861044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145809e010 msgr2=0x7f1458194a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:17.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.876 7f145eb6e700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145809e010 0x7f1458194a80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f144c00b4b0 tx=0x7f144c00b8c0).stop 2021-04-09T18:13:17.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.877 7f145eb6e700 1 -- 172.21.15.32:0/469861044 shutdown_connections 2021-04-09T18:13:17.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.877 7f145eb6e700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1438018930 0x7f143801adc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.877 7f145eb6e700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f145809e010 0x7f1458194a80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.877 7f145eb6e700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14581950a0 0x7f14580a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.877 7f145eb6e700 1 --2- 172.21.15.32:0/469861044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14580b2140 0x7f145809d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:17.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.877 7f145eb6e700 1 -- 172.21.15.32:0/469861044 >> 172.21.15.32:0/469861044 conn(0x7f14580b27e0 msgr2=0x7f14581b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:17.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.877 7f145eb6e700 1 -- 172.21.15.32:0/469861044 shutdown_connections 2021-04-09T18:13:17.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:17.877 7f145eb6e700 1 -- 172.21.15.32:0/469861044 wait complete. 2021-04-09T18:13:17.914 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:13:17.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:13:18.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:13:18.103 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:13:18.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:18.104 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:13:18.105 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:13:18.153 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:13:18.154 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:13:18.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:18.158 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:13:18.159 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:13:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:13:18.205 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:13:18.205 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:13:18.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:13:18.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.357 7f36926ed700 1 Processor -- start 2021-04-09T18:13:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.357 7f36926ed700 1 -- start start 2021-04-09T18:13:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f36926ed700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f368c0d7ca0 0x7f368c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:18.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f36926ed700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f368c10da00 0x7f368c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:18.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f36926ed700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f368c10d480 con 0x7f368c10d010 2021-04-09T18:13:18.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f36926ed700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368c0d9220 con 0x7f368c0d7ca0 2021-04-09T18:13:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f36926ed700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368c112100 con 0x7f368c10da00 2021-04-09T18:13:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f3683fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f368c0d7ca0 0x7f368c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f3683fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f368c0d7ca0 0x7f368c0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42298/0 (socket says 172.21.15.32:42298) 2021-04-09T18:13:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f3690d0a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f368c10da00 0x7f368c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f3683fff700 1 -- 172.21.15.32:0/1463836113 learned_addr learned my addr 172.21.15.32:0/1463836113 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f368bfff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f368c10d010 0x7f368c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:49810/0 (socket says 172.21.15.32:49810) 2021-04-09T18:13:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f3683fff700 1 -- 172.21.15.32:0/1463836113 >> v2:172.21.15.86:3300/0 conn(0x7f368c10da00 msgr2=0x7f368c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:18.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f3683fff700 1 --2- 172.21.15.32:0/1463836113 >> v2:172.21.15.86:3300/0 conn(0x7f368c10da00 0x7f368c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:18.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f3683fff700 1 -- 172.21.15.32:0/1463836113 >> v1:172.21.15.86:6789/0 conn(0x7f368c10d010 legacy=0x7f368c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:18.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f3683fff700 1 -- 172.21.15.32:0/1463836113 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c0d86c0 con 0x7f368c0d7ca0 2021-04-09T18:13:18.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f3683fff700 1 --2- 172.21.15.32:0/1463836113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c0d7ca0 0x7f368c0d80a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f3674006700 tx=0x7f3674009b80).ready entity=mon.0 client_cookie=5cfa1db25d805311 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:18.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.358 7f368b7fe700 1 -- 172.21.15.32:0/1463836113 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f367400c0c0 con 0x7f368c0d7ca0 2021-04-09T18:13:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.359 7f368b7fe700 1 -- 172.21.15.32:0/1463836113 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367400c440 con 0x7f368c0d7ca0 2021-04-09T18:13:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.359 7f368b7fe700 1 -- 172.21.15.32:0/1463836113 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f367400c920 con 0x7f368c0d7ca0 2021-04-09T18:13:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.359 7f36926ed700 1 -- 172.21.15.32:0/1463836113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c0d7ca0 msgr2=0x7f368c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:18.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.359 7f36926ed700 1 --2- 172.21.15.32:0/1463836113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c0d7ca0 0x7f368c0d80a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f3674006700 tx=0x7f3674009b80).stop 2021-04-09T18:13:18.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.359 7f36926ed700 1 -- 172.21.15.32:0/1463836113 shutdown_connections 2021-04-09T18:13:18.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.359 7f36926ed700 1 --2- 172.21.15.32:0/1463836113 >> v2:172.21.15.86:3300/0 conn(0x7f368c10da00 0x7f368c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:18.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.359 7f36926ed700 1 --2- 172.21.15.32:0/1463836113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c0d7ca0 0x7f368c0d80a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.359 7f36926ed700 1 -- 172.21.15.32:0/1463836113 >> 172.21.15.32:0/1463836113 conn(0x7f368c0ce5f0 msgr2=0x7f368c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.359 7f36926ed700 1 -- 172.21.15.32:0/1463836113 shutdown_connections 2021-04-09T18:13:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.360 7f36926ed700 1 -- 172.21.15.32:0/1463836113 wait complete. 2021-04-09T18:13:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f36926ed700 1 Processor -- start 2021-04-09T18:13:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f36926ed700 1 -- start start 2021-04-09T18:13:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f36926ed700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f368c1ac380 0x7f368c18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f36926ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1ac780 0x7f368c18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f36926ed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f368c183520 0x7f368c183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f36926ed700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c1ace10 con 0x7f368c1ac780 2021-04-09T18:13:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f36926ed700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c189cf0 con 0x7f368c1ac380 2021-04-09T18:13:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f36926ed700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368c183f40 con 0x7f368c183520 2021-04-09T18:13:18.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f3683fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1ac780 0x7f368c18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f3683fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1ac780 0x7f368c18a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42300/0 (socket says 172.21.15.32:42300) 2021-04-09T18:13:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f3683fff700 1 -- 172.21.15.32:0/580804435 learned_addr learned my addr 172.21.15.32:0/580804435 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f368bfff700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f368c1ac380 0x7f368c18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.380 7f3690d0a700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f368c183520 0x7f368c183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f3683fff700 1 -- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f368c183520 msgr2=0x7f368c183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f3683fff700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f368c183520 0x7f368c183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:18.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f3683fff700 1 -- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f368c1ac380 msgr2=0x7f368c18a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f3683fff700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f368c1ac380 0x7f368c18a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f3683fff700 1 -- 172.21.15.32:0/580804435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c184160 con 0x7f368c1ac780 2021-04-09T18:13:18.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f3683fff700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1ac780 0x7f368c18a850 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f367400b650 tx=0x7f367400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f36897fa700 1 -- 172.21.15.32:0/580804435 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f367400bde0 con 0x7f368c1ac780 2021-04-09T18:13:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f36897fa700 1 -- 172.21.15.32:0/580804435 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3674000c90 con 0x7f368c1ac780 2021-04-09T18:13:18.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f36926ed700 1 -- 172.21.15.32:0/580804435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368c184160 con 0x7f368c1ac780 2021-04-09T18:13:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f36897fa700 1 -- 172.21.15.32:0/580804435 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3674002220 con 0x7f368c1ac780 2021-04-09T18:13:18.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f36926ed700 1 -- 172.21.15.32:0/580804435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368c184160 con 0x7f368c1ac780 2021-04-09T18:13:18.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.381 7f36897fa700 1 -- 172.21.15.32:0/580804435 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3674009f90 con 0x7f368c1ac780 2021-04-09T18:13:18.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.382 7f36897fa700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3670018930 0x7f367001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:18.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.382 7f36897fa700 1 -- 172.21.15.32:0/580804435 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3674032b50 con 0x7f368c1ac780 2021-04-09T18:13:18.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.382 7f368bfff700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3670018930 0x7f367001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:18.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.382 7f368bfff700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3670018930 0x7f367001ae10 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:18.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.401 7f36926ed700 1 -- 172.21.15.32:0/580804435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368c189cf0 con 0x7f368c1ac780 2021-04-09T18:13:18.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.405 7f36897fa700 1 -- 172.21.15.32:0/580804435 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f36740091e0 con 0x7f368c1ac780 2021-04-09T18:13:18.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.592 7f36926ed700 1 -- 172.21.15.32:0/580804435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f368c189cf0 con 0x7f368c1ac780 2021-04-09T18:13:18.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.594 7f36897fa700 1 -- 172.21.15.32:0/580804435 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v20) v1 ==== 81+0+30360 (secure 0 0 0) 0x7f3674000c90 con 0x7f368c1ac780 2021-04-09T18:13:18.619 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:18.620 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:13:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 20, 2021-04-09T18:13:18.620 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:13:18.621 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:13:18.621 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:12:34.773743", 2021-04-09T18:13:18.621 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:13:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:13:18.622 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:13:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:13:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:13:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:13:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:13:18.625 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:13:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:13:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:13:18.626 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:13:18.627 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:13:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:13:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:13:18.750 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 3, 2021-04-09T18:13:18.750 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:13:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:13:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:13:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:13:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:13:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:13:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:13:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:13:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:13:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:13:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:13:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:13:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:13:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:13:18.858 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:13:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:13:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:13:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:13:18.859 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:13:18.868 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:13:18.869 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:13:18.869 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:13:18.891 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:13:18.892 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:13:18.892 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:13:18.914 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:13:18.915 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:13:18.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:18.919 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:13:18.919 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:13:18.920 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:13:18.920 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:13:18.920 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:13:18.921 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:13:18.921 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:13:18.922 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:13:18.922 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:13:18.923 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:13:18.923 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:13:18.923 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:13:18.942 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:13:18.947 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:13:18.948 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:13:18.951 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:13:18.952 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:13:18.953 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:13:18.953 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:13:18.954 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:13:18.954 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:13:18.955 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:13:18.988 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:13:18.988 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:13:18.988 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:13:18.988 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:13:18.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:13:18.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:18.989 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:13:18.989 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:13:18.990 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:13:18.990 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:13:18.990 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:13:18.990 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:13:18.990 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:13:18.991 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:13:18.991 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:13:18.991 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:13:19.010 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:13:19.011 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:13:19.011 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:13:19.011 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:13:19.011 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.012 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 2, 2021-04-09T18:13:19.012 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:13:19.013 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:32.136283", 2021-04-09T18:13:19.013 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:19.013 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:13:19.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:13:19.013 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:13:19.024 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:13:19.024 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:13:19.025 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:13:19.025 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:13:19.025 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:13:19.025 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:13:19.026 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:13:19.036 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:13:19.037 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:13:19.038 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:13:19.040 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:13:19.040 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:13:19.043 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:13:19.044 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:13:19.044 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:13:19.045 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:13:19.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.045 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "19", 2021-04-09T18:13:19.050 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:13:19.051 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:13:19.073 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:13:19.074 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:13:19.075 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:13:19.075 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:13:19.075 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:13:19.075 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:13:19.076 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:13:19.076 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:13:19.080 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:13:19.080 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:13:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:13:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:13:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:13:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:13:19.083 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:13:19.084 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:13:19.084 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:13:19.084 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:13:19.085 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:13:19.085 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:13:19.085 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:13:19.086 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:13:19.086 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:13:19.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:13:19.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.086 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:13:19.087 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:13:19.087 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:13:19.088 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:13:19.089 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:13:19.089 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:13:19.089 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:13:19.089 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:13:19.089 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:13:19.090 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:13:19.090 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:13:19.090 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:13:19.090 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:13:19.091 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:13:19.091 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:13:19.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.091 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:13:19.091 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:13:19.092 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:13:19.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.093 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 3, 2021-04-09T18:13:19.093 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:13:19.093 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:32.831693", 2021-04-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:13:19.199 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:13:19.203 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:13:19.204 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:13:19.204 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:13:19.205 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:13:19.217 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:13:19.217 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:13:19.217 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:13:19.218 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:13:19.218 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:13:19.219 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:13:19.219 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:13:19.219 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:13:19.220 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:13:19.220 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:13:19.221 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:13:19.221 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:13:19.222 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:13:19.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.223 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "20", 2021-04-09T18:13:19.248 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:13:19.251 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:13:19.251 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:13:19.252 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:13:19.252 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:13:19.257 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:13:19.258 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:13:19.259 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:13:19.261 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:13:19.261 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:13:19.271 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:13:19.285 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:13:19.287 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:13:19.287 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:13:19.287 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:13:19.287 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:13:19.288 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:13:19.288 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:13:19.288 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:13:19.296 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:13:19.322 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:13:19.322 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:13:19.322 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:13:19.323 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:13:19.323 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:13:19.323 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:13:19.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.324 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:13:19.335 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:13:19.335 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:13:19.335 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:13:19.335 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:13:19.336 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:13:19.336 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:13:19.336 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:13:19.336 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:13:19.337 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:13:19.337 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:13:19.337 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:13:19.352 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:13:19.352 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:13:19.352 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:13:19.355 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.366 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:13:19.367 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:13:19.367 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.367 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:13:19.367 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:13:19.368 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:13:19.368 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:13:19.368 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:13:19.368 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:13:19.369 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:13:19.369 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:13:19.381 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:13:19.381 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 18, 2021-04-09T18:13:19.381 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:13:19.382 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:13:19.384 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:13:19.400 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.403 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.403 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:13:19.403 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:13:19.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.406 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:13:19.406 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:13:19.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.413 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.413 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:13:19.413 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.415 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.416 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.416 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:13:19.416 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:13:19.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.417 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.417 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:13:19.417 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:13:19.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.418 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.418 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:13:19.418 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.419 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.419 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:13:19.419 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:13:19.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.451 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:13:19.452 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:13:19.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.453 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.456 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:13:19.456 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.456 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.456 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:13:19.461 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:13:19.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.462 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.462 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:13:19.462 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:13:19.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.528 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.529 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:13:19.529 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:13:19.530 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:13:19.530 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:13:19.531 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:13:19.531 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:13:19.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:13:19.532 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.533 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:13:19.533 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:13:19.534 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:13:19.534 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:13:19.535 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:13:19.593 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:13:19.594 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:13:19.594 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:13:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:13:19.606 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 18, 2021-04-09T18:13:19.607 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:13:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:13:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:13:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:13:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:13:19.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.673 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:13:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:13:19.674 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.675 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:13:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.782 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:13:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:13:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.784 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.784 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:13:19.784 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:13:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.785 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.808 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:13:19.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.809 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:13:19.810 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:13:19.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.810 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.810 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:13:19.811 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:13:19.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.811 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.812 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:13:19.812 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.814 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.814 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:13:19.814 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:13:19.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:13:19.815 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:13:19.815 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.883 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.884 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:13:19.884 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:13:19.885 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:13:19.886 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:13:19.887 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:13:19.888 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:13:19.888 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:13:19.889 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.890 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:13:19.890 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:13:19.891 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:13:19.891 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:13:19.891 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:13:19.891 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:13:19.892 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:13:19.892 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:13:19.920 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:13:19.920 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 17, 2021-04-09T18:13:19.920 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:13:19.921 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:13:19.921 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:13:19.921 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.922 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.922 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:13:19.922 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:13:19.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.923 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:13:19.923 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:13:19.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.924 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.924 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:13:19.924 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.925 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.925 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:13:19.925 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:13:19.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:19.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:13:19.961 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:13:19.961 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:19.961 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:19.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:19.966 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:13:19.967 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:19.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:19.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:19.967 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:13:19.968 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:13:19.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.065 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:13:20.090 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:13:20.108 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.109 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.110 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:13:20.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.120 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:13:20.122 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:13:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.180 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:13:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:13:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.225 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.232 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:13:20.233 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:13:20.254 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:13:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:13:20.255 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:13:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:13:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:13:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:13:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:13:20.280 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:13:20.281 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:13:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:13:20.293 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:13:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:13:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:13:20.294 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:13:20.295 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 18, 2021-04-09T18:13:20.316 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:13:20.332 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:13:20.333 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:13:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:13:20.338 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:13:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.339 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.340 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:13:20.355 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:13:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:13:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.370 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:13:20.371 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:13:20.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.383 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:13:20.384 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:13:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.396 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.398 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:13:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.405 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:13:20.408 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:13:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:13:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:13:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.411 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:13:20.427 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.428 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:13:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:13:20.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:13:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:13:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:13:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:13:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:13:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:13:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:13:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:13:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:13:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:13:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:13:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:13:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:13:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:13:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:13:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:13:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:13:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:13:20.494 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 18, 2021-04-09T18:13:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:13:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:13:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:13:20.496 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.497 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:13:20.500 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:13:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:13:20.515 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:13:20.516 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.532 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:13:20.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.580 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:13:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:13:20.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:13:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:13:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.602 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.604 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:13:20.608 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:13:20.614 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:13:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.734 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:13:20.736 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:13:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.737 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.787 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:13:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.788 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:13:20.789 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:13:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:13:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:13:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:13:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:13:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:13:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:13:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:13:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:13:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:13:20.840 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:13:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:13:20.892 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:13:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:13:20.893 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:13:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:13:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:13:20.894 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:13:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:13:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 18, 2021-04-09T18:13:20.942 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:13:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:13:20.943 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:13:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.944 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:13:20.945 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:13:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:20.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:20.947 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:13:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:13:20.978 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:20.979 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:20.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:20.983 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:13:21.004 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.008 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:13:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:13:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.010 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:13:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:13:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.011 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.012 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:13:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:13:21.033 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:13:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.060 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:13:21.061 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:13:21.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.072 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:13:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:13:21.082 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:13:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:13:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:13:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:13:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:13:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:13:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:13:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:13:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:13:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:13:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:13:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:13:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:13:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:13:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:13:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:13:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:13:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:13:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:13:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 17, 2021-04-09T18:13:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:13:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:13:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:13:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:13:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:13:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:13:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:13:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:13:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:13:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:13:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:13:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:13:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:13:21.343 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:13:21.344 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:13:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.345 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.426 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:13:21.427 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:13:21.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.430 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:13:21.432 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.485 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.486 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:13:21.486 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:13:21.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.487 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:13:21.487 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:13:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:13:21.488 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:13:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:13:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:13:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:13:21.489 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:13:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:13:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:13:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:13:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:13:21.493 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:13:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:13:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:13:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:13:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:13:21.494 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:13:21.607 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 17, 2021-04-09T18:13:21.607 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:13:21.608 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:13:21.617 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:13:21.618 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.625 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.632 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:13:21.640 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:13:21.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.648 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:13:21.650 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:13:21.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.652 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.653 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:13:21.653 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.654 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:13:21.654 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:13:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.656 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:13:21.656 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:13:21.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.657 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.658 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:13:21.658 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.804 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:13:21.804 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:13:21.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.806 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:13:21.806 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:13:21.806 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.807 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.829 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:13:21.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:13:21.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:13:21.840 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:13:21.840 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:13:21.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:21.882 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:13:21.882 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:13:21.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.914 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:21.916 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:13:21.917 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:13:21.917 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:13:21.918 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:13:21.918 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:13:21.918 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:13:21.919 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:13:21.940 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:21.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:21.941 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:13:21.941 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:13:21.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:21.942 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:13:21.943 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:13:21.961 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:13:21.961 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:13:22.074 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:13:22.074 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:13:22.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:22.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:22.075 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:13:22.076 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:13:22.076 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:13:22.077 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:13:22.077 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:13:22.077 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:13:22.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:22.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:22.078 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:13:22.079 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:13:22.123 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:13:22.123 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:13:22.124 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:13:22.124 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:13:22.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:22.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:22.158 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:13:22.158 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:13:22.159 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:13:22.159 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:13:22.159 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:13:22.159 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:13:22.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:22.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:22.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:13:22.160 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:13:22.161 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:13:22.161 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:13:22.161 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:13:22.161 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:13:22.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:22.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:22.162 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:13:22.163 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:13:22.163 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:13:22.163 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:13:22.200 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:13:22.201 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:13:22.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:22.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:22.201 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:13:22.201 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:13:22.202 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:13:22.218 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:13:22.218 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:13:22.219 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:13:22.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:22.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:22.239 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:13:22.240 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:13:22.240 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:13:22.240 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:13:22.240 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:13:22.240 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:13:22.241 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:22.241 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:13:22.241 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:13:22.241 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:13:22.242 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:13:22.242 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:13:22.242 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:13:22.242 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:13:22.242 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:13:22.243 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:13:22.243 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:13:22.243 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:13:22.272 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:13:22.273 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:13:22.275 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:13:22.284 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:22.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:22.293 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:13:22.294 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:13:22.294 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:13:22.295 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:13:22.320 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:13:22.320 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:13:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 -- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3670018930 msgr2=0x7f367001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3670018930 0x7f367001ae10 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 -- 172.21.15.32:0/580804435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1ac780 msgr2=0x7f368c18a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1ac780 0x7f368c18a850 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f367400b650 tx=0x7f367400b9f0).stop 2021-04-09T18:13:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 -- 172.21.15.32:0/580804435 shutdown_connections 2021-04-09T18:13:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3670018930 0x7f367001ae10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1ac780 0x7f368c18a850 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f368c183520 0x7f368c183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 --2- 172.21.15.32:0/580804435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f368c1ac380 0x7f368c18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 -- 172.21.15.32:0/580804435 >> 172.21.15.32:0/580804435 conn(0x7f368c19b890 msgr2=0x7f368c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 -- 172.21.15.32:0/580804435 shutdown_connections 2021-04-09T18:13:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:18.598 7f36926ed700 1 -- 172.21.15.32:0/580804435 wait complete. 2021-04-09T18:13:22.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:22.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.516 7fa0a16fc700 1 Processor -- start 2021-04-09T18:13:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.516 7fa0a16fc700 1 -- start start 2021-04-09T18:13:22.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.516 7fa0a16fc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa09c0d7ca0 0x7fa09c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:22.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.516 7fa0a16fc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa09c10da00 0x7fa09c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:22.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.516 7fa0a16fc700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09c10d480 con 0x7fa09c10d010 2021-04-09T18:13:22.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.516 7fa0a16fc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa09c0d9220 con 0x7fa09c10da00 2021-04-09T18:13:22.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.516 7fa0a16fc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa09c0d86c0 con 0x7fa09c0d7ca0 2021-04-09T18:13:22.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09bac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa09c10da00 0x7fa09c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:22.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09bac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa09c10da00 0x7fa09c10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42310/0 (socket says 172.21.15.32:42310) 2021-04-09T18:13:22.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09bac0700 1 -- 172.21.15.32:0/4269161270 learned_addr learned my addr 172.21.15.32:0/4269161270 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09aabe700 1 --2- 172.21.15.32:0/4269161270 >> v2:172.21.15.86:3300/0 conn(0x7fa09c0d7ca0 0x7fa09c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09bac0700 1 -- 172.21.15.32:0/4269161270 >> v2:172.21.15.86:3300/0 conn(0x7fa09c0d7ca0 msgr2=0x7fa09c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:22.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09bac0700 1 --2- 172.21.15.32:0/4269161270 >> v2:172.21.15.86:3300/0 conn(0x7fa09c0d7ca0 0x7fa09c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09bac0700 1 -- 172.21.15.32:0/4269161270 >> v1:172.21.15.86:6789/0 conn(0x7fa09c10d010 legacy=0x7fa09c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09bac0700 1 -- 172.21.15.32:0/4269161270 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c0d1f70 con 0x7fa09c10da00 2021-04-09T18:13:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09aabe700 1 --2- 172.21.15.32:0/4269161270 >> v2:172.21.15.86:3300/0 conn(0x7fa09c0d7ca0 0x7fa09c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:22.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09bac0700 1 --2- 172.21.15.32:0/4269161270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c10da00 0x7fa09c10de00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa094006f00 tx=0x7fa094009f10).ready entity=mon.0 client_cookie=69d2082610e381c6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:22.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09a2bd700 1 -- 172.21.15.32:0/4269161270 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa09400b3b0 con 0x7fa09c10da00 2021-04-09T18:13:22.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09a2bd700 1 -- 172.21.15.32:0/4269161270 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09400b730 con 0x7fa09c10da00 2021-04-09T18:13:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa09a2bd700 1 -- 172.21.15.32:0/4269161270 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa09400bc10 con 0x7fa09c10da00 2021-04-09T18:13:22.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa0a16fc700 1 -- 172.21.15.32:0/4269161270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c10da00 msgr2=0x7fa09c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:22.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.517 7fa0a16fc700 1 --2- 172.21.15.32:0/4269161270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c10da00 0x7fa09c10de00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa094006f00 tx=0x7fa094009f10).stop 2021-04-09T18:13:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.518 7fa0a16fc700 1 -- 172.21.15.32:0/4269161270 shutdown_connections 2021-04-09T18:13:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.518 7fa0a16fc700 1 --2- 172.21.15.32:0/4269161270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c10da00 0x7fa09c10de00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.518 7fa0a16fc700 1 --2- 172.21.15.32:0/4269161270 >> v2:172.21.15.86:3300/0 conn(0x7fa09c0d7ca0 0x7fa09c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.518 7fa0a16fc700 1 -- 172.21.15.32:0/4269161270 >> 172.21.15.32:0/4269161270 conn(0x7fa09c0ce5f0 msgr2=0x7fa09c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.518 7fa0a16fc700 1 -- 172.21.15.32:0/4269161270 shutdown_connections 2021-04-09T18:13:22.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.518 7fa0a16fc700 1 -- 172.21.15.32:0/4269161270 wait complete. 2021-04-09T18:13:22.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.538 7fa0a16fc700 1 Processor -- start 2021-04-09T18:13:22.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.538 7fa0a16fc700 1 -- start start 2021-04-09T18:13:22.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa0a16fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c1a5210 0x7fa09c185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:22.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa0a16fc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c185a70 0x7fa09c0acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:22.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa0a16fc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c185e70 0x7fa09c0ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:22.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa0a16fc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c18f7b0 con 0x7fa09c1a5210 2021-04-09T18:13:22.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa0a16fc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0ad7f0 con 0x7fa09c185e70 2021-04-09T18:13:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa0a16fc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0ada10 con 0x7fa09c185a70 2021-04-09T18:13:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c1a5210 0x7fa09c185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:22.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c1a5210 0x7fa09c185450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42316/0 (socket says 172.21.15.32:42316) 2021-04-09T18:13:22.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09b2bf700 1 -- 172.21.15.32:0/3455835273 learned_addr learned my addr 172.21.15.32:0/3455835273 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:22.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09aabe700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c185a70 0x7fa09c0acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:22.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09bac0700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c185e70 0x7fa09c0ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:22.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09b2bf700 1 -- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c185a70 msgr2=0x7fa09c0acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:22.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09b2bf700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c185a70 0x7fa09c0acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09b2bf700 1 -- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c185e70 msgr2=0x7fa09c0ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:22.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09b2bf700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c185e70 0x7fa09c0ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09b2bf700 1 -- 172.21.15.32:0/3455835273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c179610 con 0x7fa09c1a5210 2021-04-09T18:13:22.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09bac0700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c185e70 0x7fa09c0ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:22.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa09b2bf700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c1a5210 0x7fa09c185450 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa08c00b5b0 tx=0x7fa08c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:22.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.539 7fa083fff700 1 -- 172.21.15.32:0/3455835273 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c00bd40 con 0x7fa09c1a5210 2021-04-09T18:13:22.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.540 7fa083fff700 1 -- 172.21.15.32:0/3455835273 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c006240 con 0x7fa09c1a5210 2021-04-09T18:13:22.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.540 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c179610 con 0x7fa09c1a5210 2021-04-09T18:13:22.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.540 7fa083fff700 1 -- 172.21.15.32:0/3455835273 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c005690 con 0x7fa09c1a5210 2021-04-09T18:13:22.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.540 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c179610 con 0x7fa09c1a5210 2021-04-09T18:13:22.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.540 7fa083fff700 1 -- 172.21.15.32:0/3455835273 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa08c0054a0 con 0x7fa09c1a5210 2021-04-09T18:13:22.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.540 7fa083fff700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa07c018930 0x7fa07c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:22.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.540 7fa083fff700 1 -- 172.21.15.32:0/3455835273 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa08c032a00 con 0x7fa09c1a5210 2021-04-09T18:13:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.540 7fa09aabe700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa07c018930 0x7fa07c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.541 7fa09aabe700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa07c018930 0x7fa07c01ae10 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.561 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09c18f7b0 con 0x7fa09c1a5210 2021-04-09T18:13:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.564 7fa083fff700 1 -- 172.21.15.32:0/3455835273 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa08c0009b0 con 0x7fa09c1a5210 2021-04-09T18:13:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.754 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa09c18f7b0 con 0x7fa09c1a5210 2021-04-09T18:13:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.755 7fa083fff700 1 -- 172.21.15.32:0/3455835273 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2769 (secure 0 0 0) 0x7fa08c0054a0 con 0x7fa09c1a5210 2021-04-09T18:13:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 6 2021-04-09T18:13:22.805 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:22.851 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4323,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3379802756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3379802756},{"type":"v1","addr":"172.21.15.32:6835","nonce":3379802756}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4},{"gid":4333,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3413992045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3413992045},{"type":"v1","addr":"172.21.15.86:6835","nonce":3413992045}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4},{"gid":4342,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2501526335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2501526335},{"type":"v1","addr":"172.21.15.86:6837","nonce":2501526335}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:12:33.772139","modified":"2021-04-09 18:12:42.800036","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4314},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4314":{"gid":4314,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6833/3268604338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3268604338},{"type":"v1","addr":"172.21.15.32:6833","nonce":3268604338}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-04-09T18:13:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa07c018930 msgr2=0x7fa07c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:22.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa07c018930 0x7fa07c01ae10 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c1a5210 msgr2=0x7fa09c185450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c1a5210 0x7fa09c185450 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa08c00b5b0 tx=0x7fa08c00b950).stop 2021-04-09T18:13:22.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 shutdown_connections 2021-04-09T18:13:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa07c018930 0x7fa07c01ae10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c185e70 0x7fa09c0ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c185a70 0x7fa09c0acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 --2- 172.21.15.32:0/3455835273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c1a5210 0x7fa09c185450 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:22.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 >> 172.21.15.32:0/3455835273 conn(0x7fa09c1a99c0 msgr2=0x7fa09c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:22.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 shutdown_connections 2021-04-09T18:13:22.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:22.757 7fa0a16fc700 1 -- 172.21.15.32:0/3455835273 wait complete. 2021-04-09T18:13:22.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:13:23.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.064 7faf1f1da700 1 Processor -- start 2021-04-09T18:13:23.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.064 7faf1f1da700 1 -- start start 2021-04-09T18:13:23.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.064 7faf1f1da700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf180cffb0 0x7faf180d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.064 7faf1f1da700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faf180d7810 0x7faf180d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.064 7faf1f1da700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf180d7290 con 0x7faf180d5c80 2021-04-09T18:13:23.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.064 7faf1f1da700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf180d8590 con 0x7faf180d7810 2021-04-09T18:13:23.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.064 7faf1f1da700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf180d91f0 con 0x7faf180cffb0 2021-04-09T18:13:23.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf17fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faf180d7810 0x7faf180d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf17fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faf180d7810 0x7faf180d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42322/0 (socket says 172.21.15.32:42322) 2021-04-09T18:13:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf17fff700 1 -- 172.21.15.32:0/2411055573 learned_addr learned my addr 172.21.15.32:0/2411055573 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:23.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf1cff6700 1 --2- 172.21.15.32:0/2411055573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf180cffb0 0x7faf180d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf17fff700 1 -- 172.21.15.32:0/2411055573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf180cffb0 msgr2=0x7faf180d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:23.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf17fff700 1 --2- 172.21.15.32:0/2411055573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf180cffb0 0x7faf180d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf177fe700 1 -- 172.21.15.32:0/2411055573 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636281869 0 0) 0x7faf10000d30 con 0x7faf180d5c80 2021-04-09T18:13:23.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf17fff700 1 -- 172.21.15.32:0/2411055573 >> v1:172.21.15.86:6789/0 conn(0x7faf180d5c80 legacy=0x7faf180d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf17fff700 1 -- 172.21.15.32:0/2411055573 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf180d8230 con 0x7faf180d7810 2021-04-09T18:13:23.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf17fff700 1 --2- 172.21.15.32:0/2411055573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf180d7810 0x7faf180d7c10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7faf08006700 tx=0x7faf08009b80).ready entity=mon.0 client_cookie=1f3917dd3db7a035 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:23.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf177fe700 1 -- 172.21.15.32:0/2411055573 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf0800c0c0 con 0x7faf180d7810 2021-04-09T18:13:23.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf177fe700 1 -- 172.21.15.32:0/2411055573 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0800c440 con 0x7faf180d7810 2021-04-09T18:13:23.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf177fe700 1 -- 172.21.15.32:0/2411055573 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf0800c920 con 0x7faf180d7810 2021-04-09T18:13:23.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf1f1da700 1 -- 172.21.15.32:0/2411055573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf180d7810 msgr2=0x7faf180d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.065 7faf1f1da700 1 --2- 172.21.15.32:0/2411055573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf180d7810 0x7faf180d7c10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7faf08006700 tx=0x7faf08009b80).stop 2021-04-09T18:13:23.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.066 7faf1f1da700 1 -- 172.21.15.32:0/2411055573 shutdown_connections 2021-04-09T18:13:23.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.066 7faf1f1da700 1 --2- 172.21.15.32:0/2411055573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf180d7810 0x7faf180d7c10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.066 7faf1f1da700 1 --2- 172.21.15.32:0/2411055573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf180cffb0 0x7faf180d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.066 7faf1f1da700 1 -- 172.21.15.32:0/2411055573 >> 172.21.15.32:0/2411055573 conn(0x7faf180ce5d0 msgr2=0x7faf180cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.066 7faf1f1da700 1 -- 172.21.15.32:0/2411055573 shutdown_connections 2021-04-09T18:13:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.066 7faf1f1da700 1 -- 172.21.15.32:0/2411055573 wait complete. 2021-04-09T18:13:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.086 7faf1f1da700 1 Processor -- start 2021-04-09T18:13:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.086 7faf1f1da700 1 -- start start 2021-04-09T18:13:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1f1da700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1808a620 0x7faf1808b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1f1da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf1808b6d0 0x7faf18180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1f1da700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf18180c50 0x7faf18181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1f1da700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1808e660 con 0x7faf1808b6d0 2021-04-09T18:13:23.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1f1da700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1808ab30 con 0x7faf1808a620 2021-04-09T18:13:23.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1f1da700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1808bad0 con 0x7faf18180c50 2021-04-09T18:13:23.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf1808b6d0 0x7faf18180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf1808b6d0 0x7faf18180630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42326/0 (socket says 172.21.15.32:42326) 2021-04-09T18:13:23.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf17fff700 1 -- 172.21.15.32:0/3865426531 learned_addr learned my addr 172.21.15.32:0/3865426531 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:23.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1cff6700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1808a620 0x7faf1808b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1d7f7700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf18180c50 0x7faf18181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf17fff700 1 -- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf18180c50 msgr2=0x7faf18181050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf17fff700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf18180c50 0x7faf18181050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf17fff700 1 -- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1808a620 msgr2=0x7faf1808b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf17fff700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1808a620 0x7faf1808b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf17fff700 1 -- 172.21.15.32:0/3865426531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1808e1f0 con 0x7faf1808b6d0 2021-04-09T18:13:23.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1cff6700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1808a620 0x7faf1808b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:23.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf1d7f7700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf18180c50 0x7faf18181050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:23.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf17fff700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf1808b6d0 0x7faf18180630 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7faf08009f90 tx=0x7faf0800b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:23.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.087 7faf157fa700 1 -- 172.21.15.32:0/3865426531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf0800bd00 con 0x7faf1808b6d0 2021-04-09T18:13:23.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.088 7faf157fa700 1 -- 172.21.15.32:0/3865426531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf08000c90 con 0x7faf1808b6d0 2021-04-09T18:13:23.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.088 7faf157fa700 1 -- 172.21.15.32:0/3865426531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf080022b0 con 0x7faf1808b6d0 2021-04-09T18:13:23.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.088 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1808e1f0 con 0x7faf1808b6d0 2021-04-09T18:13:23.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.088 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1808e1f0 con 0x7faf1808b6d0 2021-04-09T18:13:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.088 7faf157fa700 1 -- 172.21.15.32:0/3865426531 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faf08009fd0 con 0x7faf1808b6d0 2021-04-09T18:13:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.088 7faf157fa700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faefc018900 0x7faefc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.088 7faf157fa700 1 -- 172.21.15.32:0/3865426531 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5577+0+0 (secure 0 0 0) 0x7faf08034c20 con 0x7faf1808b6d0 2021-04-09T18:13:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.089 7faf1cff6700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faefc018900 0x7faefc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.089 7faf1cff6700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faefc018900 0x7faefc01adc0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:23.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.108 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf181b7180 con 0x7faf1808b6d0 2021-04-09T18:13:23.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.112 7faf157fa700 1 -- 172.21.15.32:0/3865426531 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faf080091e0 con 0x7faf1808b6d0 2021-04-09T18:13:23.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.292 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7faf180d7290 con 0x7faf1808b6d0 2021-04-09T18:13:23.485 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:13:23.315 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:13:23.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.317 7faf157fa700 1 -- 172.21.15.32:0/3865426531 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7faf08000c90 con 0x7faf1808b6d0 2021-04-09T18:13:23.490 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:13:23.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faefc018900 msgr2=0x7faefc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faefc018900 0x7faefc01adc0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf1808b6d0 msgr2=0x7faf18180630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf1808b6d0 0x7faf18180630 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7faf08009f90 tx=0x7faf0800b910).stop 2021-04-09T18:13:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 shutdown_connections 2021-04-09T18:13:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faefc018900 0x7faefc01adc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf1808b6d0 0x7faf18180630 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf18180c50 0x7faf18181050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 --2- 172.21.15.32:0/3865426531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1808a620 0x7faf1808b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 >> 172.21.15.32:0/3865426531 conn(0x7faf18189800 msgr2=0x7faf181b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:23.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 shutdown_connections 2021-04-09T18:13:23.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.325 7faf1f1da700 1 -- 172.21.15.32:0/3865426531 wait complete. 2021-04-09T18:13:23.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:23.575 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:13:23.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.726 7f6175957700 1 Processor -- start 2021-04-09T18:13:23.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.726 7f6175957700 1 -- start start 2021-04-09T18:13:23.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f6175957700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f617010da00 0x7f617010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f6175957700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617010d480 con 0x7f617010d010 2021-04-09T18:13:23.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f6175957700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61700d1f70 con 0x7f61700d7ca0 2021-04-09T18:13:23.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f6175957700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61700d9220 con 0x7f617010da00 2021-04-09T18:13:23.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616f574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f617010d010 0x7f61700d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51936/0 (socket says 172.21.15.32:51936) 2021-04-09T18:13:23.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616f574700 1 -- 172.21.15.32:0/1892944174 learned_addr learned my addr 172.21.15.32:0/1892944174 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:23.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616fd75700 1 --2- 172.21.15.32:0/1892944174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f617010da00 0x7f617010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616e572700 1 -- 172.21.15.32:0/1892944174 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581270692 0 0) 0x7f6160000ce0 con 0x7f617010d010 2021-04-09T18:13:23.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616e572700 1 -- 172.21.15.32:0/1892944174 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6158002fe0 con 0x7f617010d010 2021-04-09T18:13:23.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616e572700 1 -- 172.21.15.32:0/1892944174 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762362240 0 0) 0x7f6164000c60 con 0x7f61700d7ca0 2021-04-09T18:13:23.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616e572700 1 -- 172.21.15.32:0/1892944174 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6158003210 con 0x7f61700d7ca0 2021-04-09T18:13:23.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616fd75700 1 -- 172.21.15.32:0/1892944174 >> v1:172.21.15.86:6789/0 conn(0x7f61700d7ca0 legacy=0x7f61700d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616fd75700 1 -- 172.21.15.32:0/1892944174 >> v1:172.21.15.32:6789/0 conn(0x7f617010d010 legacy=0x7f61700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616e572700 1 -- 172.21.15.32:0/1892944174 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3902529479 0 0) 0x7f61600010e0 con 0x7f617010d010 2021-04-09T18:13:23.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616fd75700 1 -- 172.21.15.32:0/1892944174 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617010e420 con 0x7f617010da00 2021-04-09T18:13:23.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.727 7f616fd75700 1 --2- 172.21.15.32:0/1892944174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f617010da00 0x7f617010de00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6168006e50 tx=0x7f6168009df0).ready entity=mon.2 client_cookie=e927daf44a93e7d3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:23.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f616e572700 1 -- 172.21.15.32:0/1892944174 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f616800b510 con 0x7f617010da00 2021-04-09T18:13:23.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f616e572700 1 -- 172.21.15.32:0/1892944174 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168003030 con 0x7f617010da00 2021-04-09T18:13:23.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f616e572700 1 -- 172.21.15.32:0/1892944174 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f616800b7d0 con 0x7f617010da00 2021-04-09T18:13:23.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f6175957700 1 -- 172.21.15.32:0/1892944174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f617010da00 msgr2=0x7f617010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f6175957700 1 --2- 172.21.15.32:0/1892944174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f617010da00 0x7f617010de00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6168006e50 tx=0x7f6168009df0).stop 2021-04-09T18:13:23.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f6175957700 1 -- 172.21.15.32:0/1892944174 shutdown_connections 2021-04-09T18:13:23.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f6175957700 1 --2- 172.21.15.32:0/1892944174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f617010da00 0x7f617010de00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f6175957700 1 -- 172.21.15.32:0/1892944174 >> 172.21.15.32:0/1892944174 conn(0x7f61700ce5f0 msgr2=0x7f61700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:23.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f6175957700 1 -- 172.21.15.32:0/1892944174 shutdown_connections 2021-04-09T18:13:23.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.728 7f6175957700 1 -- 172.21.15.32:0/1892944174 wait complete. 2021-04-09T18:13:23.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.748 7f6175957700 1 Processor -- start 2021-04-09T18:13:23.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f6175957700 1 -- start start 2021-04-09T18:13:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f6175957700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61700b42d0 0x7f6170183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f6175957700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61701ac090 0x7f61701ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f6175957700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6170184d30 0x7f6170185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f6175957700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61700a1a90 con 0x7f6170184d30 2021-04-09T18:13:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f6175957700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61701acab0 con 0x7f61701ac090 2021-04-09T18:13:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f6175957700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6170185a00 con 0x7f61700b42d0 2021-04-09T18:13:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f616fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6170184d30 0x7f6170185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f616f574700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61700b42d0 0x7f6170183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f616ed73700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61701ac090 0x7f61701ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f616fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6170184d30 0x7f6170185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42354/0 (socket says 172.21.15.32:42354) 2021-04-09T18:13:23.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.749 7f616fd75700 1 -- 172.21.15.32:0/4254745789 learned_addr learned my addr 172.21.15.32:0/4254745789 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:23.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f616fd75700 1 -- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61700b42d0 msgr2=0x7f6170183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f616fd75700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61700b42d0 0x7f6170183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f616fd75700 1 -- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61701ac090 msgr2=0x7f61701ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:23.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f616fd75700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61701ac090 0x7f61701ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:23.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f616fd75700 1 -- 172.21.15.32:0/4254745789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6170185750 con 0x7f6170184d30 2021-04-09T18:13:23.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f616ed73700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61701ac090 0x7f61701ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:23.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f616f574700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61700b42d0 0x7f6170183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:23.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f616fd75700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6170184d30 0x7f6170185130 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6168009f80 tx=0x7f616800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:23.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f6157fff700 1 -- 172.21.15.32:0/4254745789 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61680093d0 con 0x7f6170184d30 2021-04-09T18:13:23.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f6157fff700 1 -- 172.21.15.32:0/4254745789 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168001240 con 0x7f6170184d30 2021-04-09T18:13:23.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f6175957700 1 -- 172.21.15.32:0/4254745789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6170185750 con 0x7f6170184d30 2021-04-09T18:13:23.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f6157fff700 1 -- 172.21.15.32:0/4254745789 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6168001690 con 0x7f6170184d30 2021-04-09T18:13:23.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.750 7f6175957700 1 -- 172.21.15.32:0/4254745789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6170185750 con 0x7f6170184d30 2021-04-09T18:13:23.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.751 7f6157fff700 1 -- 172.21.15.32:0/4254745789 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f61680097c0 con 0x7f6170184d30 2021-04-09T18:13:23.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.751 7f6157fff700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61500188d0 0x7f615001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:23.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.751 7f6157fff700 1 -- 172.21.15.32:0/4254745789 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f6168032df0 con 0x7f6170184d30 2021-04-09T18:13:23.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.751 7f616f574700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61500188d0 0x7f615001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:23.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.751 7f616f574700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61500188d0 0x7f615001add0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:23.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.771 7f6175957700 1 -- 172.21.15.32:0/4254745789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6170182d10 con 0x7f6170184d30 2021-04-09T18:13:23.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.774 7f6157fff700 1 -- 172.21.15.32:0/4254745789 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6168001f00 con 0x7f6170184d30 2021-04-09T18:13:24.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.966 7f6175957700 1 -- 172.21.15.32:0/4254745789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61700a1a90 con 0x7f6170184d30 2021-04-09T18:13:24.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.966 7f6157fff700 1 -- 172.21.15.32:0/4254745789 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2425 (secure 0 0 0) 0x7f61680097c0 con 0x7f6170184d30 2021-04-09T18:13:24.008 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 7 2021-04-09T18:13:24.016 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:24.016 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4323,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3379802756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3379802756},{"type":"v1","addr":"172.21.15.32:6835","nonce":3379802756}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4},{"gid":4333,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3413992045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3413992045},{"type":"v1","addr":"172.21.15.86:6835","nonce":3413992045}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4},{"gid":4342,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2501526335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2501526335},{"type":"v1","addr":"172.21.15.86:6837","nonce":2501526335}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":7,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:12:33.772139","modified":"2021-04-09 18:13:23.316842","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-04-09T18:13:24.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.968 7f6175957700 1 -- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61500188d0 msgr2=0x7f615001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:24.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.968 7f6175957700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61500188d0 0x7f615001add0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 -- 172.21.15.32:0/4254745789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6170184d30 msgr2=0x7f6170185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:24.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6170184d30 0x7f6170185130 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6168009f80 tx=0x7f616800bd10).stop 2021-04-09T18:13:24.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 -- 172.21.15.32:0/4254745789 shutdown_connections 2021-04-09T18:13:24.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61500188d0 0x7f615001add0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6170184d30 0x7f6170185130 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61701ac090 0x7f61701ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 --2- 172.21.15.32:0/4254745789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61700b42d0 0x7f6170183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 -- 172.21.15.32:0/4254745789 >> 172.21.15.32:0/4254745789 conn(0x7f617019b890 msgr2=0x7f61701a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:24.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 -- 172.21.15.32:0/4254745789 shutdown_connections 2021-04-09T18:13:24.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:23.969 7f6175957700 1 -- 172.21.15.32:0/4254745789 wait complete. 2021-04-09T18:13:24.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:13:24.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.191 7f7b67f64700 1 Processor -- start 2021-04-09T18:13:24.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.191 7f7b67f64700 1 -- start start 2021-04-09T18:13:24.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.191 7f7b67f64700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b600cffb0 0x7f7b600d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:24.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.191 7f7b67f64700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b600d7810 0x7f7b600d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:24.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.191 7f7b67f64700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b600d7290 con 0x7f7b600d8230 2021-04-09T18:13:24.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.191 7f7b67f64700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b600d91f0 con 0x7f7b600cffb0 2021-04-09T18:13:24.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.191 7f7b67f64700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6010dff0 con 0x7f7b600d7810 2021-04-09T18:13:24.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b65d80700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b600cffb0 0x7f7b600d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:24.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b65d80700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b600cffb0 0x7f7b600d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42360/0 (socket says 172.21.15.32:42360) 2021-04-09T18:13:24.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b65d80700 1 -- 172.21.15.32:0/1114239550 learned_addr learned my addr 172.21.15.32:0/1114239550 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:24.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b6557f700 1 --2- 172.21.15.32:0/1114239550 >> v2:172.21.15.86:3300/0 conn(0x7f7b600d7810 0x7f7b600d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:24.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b65d80700 1 -- 172.21.15.32:0/1114239550 >> v2:172.21.15.86:3300/0 conn(0x7f7b600d7810 msgr2=0x7f7b600d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:24.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b65d80700 1 --2- 172.21.15.32:0/1114239550 >> v2:172.21.15.86:3300/0 conn(0x7f7b600d7810 0x7f7b600d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b65d80700 1 -- 172.21.15.32:0/1114239550 >> v1:172.21.15.86:6789/0 conn(0x7f7b600d8230 legacy=0x7f7b600d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:24.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b65d80700 1 -- 172.21.15.32:0/1114239550 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6010e210 con 0x7f7b600cffb0 2021-04-09T18:13:24.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b65d80700 1 --2- 172.21.15.32:0/1114239550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b600cffb0 0x7f7b600d2450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f7b5c006700 tx=0x7f7b5c009b80).ready entity=mon.0 client_cookie=5448ab1b0c91a3db server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:24.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b64d7e700 1 -- 172.21.15.32:0/1114239550 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b5c00c0c0 con 0x7f7b600cffb0 2021-04-09T18:13:24.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b64d7e700 1 -- 172.21.15.32:0/1114239550 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5c00c440 con 0x7f7b600cffb0 2021-04-09T18:13:24.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b64d7e700 1 -- 172.21.15.32:0/1114239550 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b5c00c920 con 0x7f7b600cffb0 2021-04-09T18:13:24.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b67f64700 1 -- 172.21.15.32:0/1114239550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b600cffb0 msgr2=0x7f7b600d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:24.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.192 7f7b67f64700 1 --2- 172.21.15.32:0/1114239550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b600cffb0 0x7f7b600d2450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f7b5c006700 tx=0x7f7b5c009b80).stop 2021-04-09T18:13:24.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.193 7f7b67f64700 1 -- 172.21.15.32:0/1114239550 shutdown_connections 2021-04-09T18:13:24.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.193 7f7b67f64700 1 --2- 172.21.15.32:0/1114239550 >> v2:172.21.15.86:3300/0 conn(0x7f7b600d7810 0x7f7b600d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.193 7f7b67f64700 1 --2- 172.21.15.32:0/1114239550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b600cffb0 0x7f7b600d2450 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.193 7f7b67f64700 1 -- 172.21.15.32:0/1114239550 >> 172.21.15.32:0/1114239550 conn(0x7f7b600ce5d0 msgr2=0x7f7b600cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.193 7f7b67f64700 1 -- 172.21.15.32:0/1114239550 shutdown_connections 2021-04-09T18:13:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.193 7f7b67f64700 1 -- 172.21.15.32:0/1114239550 wait complete. 2021-04-09T18:13:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.213 7f7b67f64700 1 Processor -- start 2021-04-09T18:13:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.213 7f7b67f64700 1 -- start start 2021-04-09T18:13:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b67f64700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b601a7dc0 0x7f7b601a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:24.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b67f64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b60184080 0x7f7b60184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:24.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b67f64700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60184aa0 0x7f7b6008f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:24.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b67f64700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b601a8b40 con 0x7f7b60184080 2021-04-09T18:13:24.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b67f64700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b601a7870 con 0x7f7b601a7dc0 2021-04-09T18:13:24.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b67f64700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b601a87e0 con 0x7f7b60184aa0 2021-04-09T18:13:24.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b6557f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b60184080 0x7f7b60184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:24.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b6557f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b60184080 0x7f7b60184480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42366/0 (socket says 172.21.15.32:42366) 2021-04-09T18:13:24.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b6557f700 1 -- 172.21.15.32:0/1436641210 learned_addr learned my addr 172.21.15.32:0/1436641210 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:24.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b66581700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60184aa0 0x7f7b6008f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:24.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b65d80700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b601a7dc0 0x7f7b601a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:24.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b6557f700 1 -- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60184aa0 msgr2=0x7f7b6008f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:24.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b6557f700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60184aa0 0x7f7b6008f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b6557f700 1 -- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b601a7dc0 msgr2=0x7f7b601a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b6557f700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b601a7dc0 0x7f7b601a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b6557f700 1 -- 172.21.15.32:0/1436641210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6008f810 con 0x7f7b60184080 2021-04-09T18:13:24.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.214 7f7b6557f700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b60184080 0x7f7b60184480 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7b54006f60 tx=0x7f7b5400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:24.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.215 7f7b527fc700 1 -- 172.21.15.32:0/1436641210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b5400bd60 con 0x7f7b60184080 2021-04-09T18:13:24.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.215 7f7b527fc700 1 -- 172.21.15.32:0/1436641210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b54000c90 con 0x7f7b60184080 2021-04-09T18:13:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.215 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6008f810 con 0x7f7b60184080 2021-04-09T18:13:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.215 7f7b527fc700 1 -- 172.21.15.32:0/1436641210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b54002220 con 0x7f7b60184080 2021-04-09T18:13:24.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.215 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6008f810 con 0x7f7b60184080 2021-04-09T18:13:24.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.216 7f7b527fc700 1 -- 172.21.15.32:0/1436641210 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7b540052e0 con 0x7f7b60184080 2021-04-09T18:13:24.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.216 7f7b527fc700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b48018900 0x7f7b4801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:24.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.216 7f7b527fc700 1 -- 172.21.15.32:0/1436641210 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f7b54032a40 con 0x7f7b60184080 2021-04-09T18:13:24.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.216 7f7b65d80700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b48018900 0x7f7b4801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:24.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.216 7f7b65d80700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b48018900 0x7f7b4801adc0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:24.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.235 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b60182e80 con 0x7f7b60184080 2021-04-09T18:13:24.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.239 7f7b527fc700 1 -- 172.21.15.32:0/1436641210 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7b54000c90 con 0x7f7b60184080 2021-04-09T18:13:24.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:24.436 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7b60182e80 con 0x7f7b60184080 2021-04-09T18:13:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.328 7f7b527fc700 1 -- 172.21.15.32:0/1436641210 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7b540052e0 con 0x7f7b60184080 2021-04-09T18:13:25.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b48018900 msgr2=0x7f7b4801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:25.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b48018900 0x7f7b4801adc0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b60184080 msgr2=0x7f7b60184480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b60184080 0x7f7b60184480 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7b54006f60 tx=0x7f7b5400b970).stop 2021-04-09T18:13:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 shutdown_connections 2021-04-09T18:13:25.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b48018900 0x7f7b4801adc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b60184080 0x7f7b60184480 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60184aa0 0x7f7b6008f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 --2- 172.21.15.32:0/1436641210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b601a7dc0 0x7f7b601a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 >> 172.21.15.32:0/1436641210 conn(0x7f7b601a8f50 msgr2=0x7f7b60188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:25.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 shutdown_connections 2021-04-09T18:13:25.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.354 7f7b67f64700 1 -- 172.21.15.32:0/1436641210 wait complete. 2021-04-09T18:13:25.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:13:25.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.544 7f331b835700 1 Processor -- start 2021-04-09T18:13:25.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.544 7f331b835700 1 -- start start 2021-04-09T18:13:25.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f331b835700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33140d7cc0 0x7f33140d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:25.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f331b835700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f331410da80 0x7f331410de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f331b835700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331410d500 con 0x7f331410d050 2021-04-09T18:13:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f331b835700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33140d9260 con 0x7f331410da80 2021-04-09T18:13:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f331b835700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33140d0330 con 0x7f33140d7cc0 2021-04-09T18:13:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f3319e52700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f331410da80 0x7f331410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:25.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f3319651700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f331410d050 0x7f33140cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51964/0 (socket says 172.21.15.32:51964) 2021-04-09T18:13:25.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f3319651700 1 -- 172.21.15.32:0/4037965181 learned_addr learned my addr 172.21.15.32:0/4037965181 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:25.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f3318e50700 1 --2- 172.21.15.32:0/4037965181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33140d7cc0 0x7f33140d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:25.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f3319e52700 1 -- 172.21.15.32:0/4037965181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33140d7cc0 msgr2=0x7f33140d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:25.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f3319e52700 1 --2- 172.21.15.32:0/4037965181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33140d7cc0 0x7f33140d80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f3319e52700 1 -- 172.21.15.32:0/4037965181 >> v1:172.21.15.32:6789/0 conn(0x7f331410d050 legacy=0x7f33140cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:25.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f3319e52700 1 -- 172.21.15.32:0/4037965181 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33140d09f0 con 0x7f331410da80 2021-04-09T18:13:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.545 7f330bfff700 1 -- 172.21.15.32:0/4037965181 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277814863 0 0) 0x7f3310000ce0 con 0x7f331410d050 2021-04-09T18:13:25.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.546 7f3318e50700 1 --2- 172.21.15.32:0/4037965181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33140d7cc0 0x7f33140d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:25.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.546 7f3319e52700 1 --2- 172.21.15.32:0/4037965181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331410da80 0x7f331410de80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f330c006e30 tx=0x7f330c009e60).ready entity=mon.0 client_cookie=8486477405635703 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:25.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.546 7f330bfff700 1 -- 172.21.15.32:0/4037965181 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f330c006a50 con 0x7f331410da80 2021-04-09T18:13:25.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.546 7f330bfff700 1 -- 172.21.15.32:0/4037965181 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c00b300 con 0x7f331410da80 2021-04-09T18:13:25.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.546 7f330bfff700 1 -- 172.21.15.32:0/4037965181 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f330c00b750 con 0x7f331410da80 2021-04-09T18:13:25.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.546 7f331b835700 1 -- 172.21.15.32:0/4037965181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331410da80 msgr2=0x7f331410de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:25.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.546 7f331b835700 1 --2- 172.21.15.32:0/4037965181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331410da80 0x7f331410de80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f330c006e30 tx=0x7f330c009e60).stop 2021-04-09T18:13:25.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.547 7f331b835700 1 -- 172.21.15.32:0/4037965181 shutdown_connections 2021-04-09T18:13:25.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.547 7f331b835700 1 --2- 172.21.15.32:0/4037965181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331410da80 0x7f331410de80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.547 7f331b835700 1 --2- 172.21.15.32:0/4037965181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33140d7cc0 0x7f33140d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.547 7f331b835700 1 -- 172.21.15.32:0/4037965181 >> 172.21.15.32:0/4037965181 conn(0x7f33140ce620 msgr2=0x7f33140d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:25.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.547 7f331b835700 1 -- 172.21.15.32:0/4037965181 shutdown_connections 2021-04-09T18:13:25.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.547 7f331b835700 1 -- 172.21.15.32:0/4037965181 wait complete. 2021-04-09T18:13:25.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.567 7f331b835700 1 Processor -- start 2021-04-09T18:13:25.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.567 7f331b835700 1 -- start start 2021-04-09T18:13:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f331b835700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f331409b2f0 0x7f331409b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f331b835700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f331409a8b0 0x7f331409e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f331b835700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331409e9c0 0x7f331409edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f331b835700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33140bde80 con 0x7f331409e9c0 2021-04-09T18:13:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f331b835700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331409ad70 con 0x7f331409b2f0 2021-04-09T18:13:25.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f331b835700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331409bd10 con 0x7f331409a8b0 2021-04-09T18:13:25.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319e52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331409e9c0 0x7f331409edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:25.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319e52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331409e9c0 0x7f331409edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42382/0 (socket says 172.21.15.32:42382) 2021-04-09T18:13:25.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319e52700 1 -- 172.21.15.32:0/719438590 learned_addr learned my addr 172.21.15.32:0/719438590 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:25.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3318e50700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f331409a8b0 0x7f331409e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:25.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319651700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f331409b2f0 0x7f331409b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:25.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319e52700 1 -- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f331409a8b0 msgr2=0x7f331409e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:25.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319e52700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f331409a8b0 0x7f331409e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319e52700 1 -- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f331409b2f0 msgr2=0x7f331409b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319e52700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f331409b2f0 0x7f331409b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319e52700 1 -- 172.21.15.32:0/719438590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331409fbd0 con 0x7f331409e9c0 2021-04-09T18:13:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3319e52700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331409e9c0 0x7f331409edc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f330c001070 tx=0x7f330c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3309ffb700 1 -- 172.21.15.32:0/719438590 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f330c001270 con 0x7f331409e9c0 2021-04-09T18:13:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3309ffb700 1 -- 172.21.15.32:0/719438590 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c006df0 con 0x7f331409e9c0 2021-04-09T18:13:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f3309ffb700 1 -- 172.21.15.32:0/719438590 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f330c001720 con 0x7f331409e9c0 2021-04-09T18:13:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f331b835700 1 -- 172.21.15.32:0/719438590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331409fbd0 con 0x7f331409e9c0 2021-04-09T18:13:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.568 7f331b835700 1 -- 172.21.15.32:0/719438590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331409fbd0 con 0x7f331409e9c0 2021-04-09T18:13:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.569 7f3309ffb700 1 -- 172.21.15.32:0/719438590 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f330c0014f0 con 0x7f331409e9c0 2021-04-09T18:13:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.569 7f3309ffb700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32f4018930 0x7f32f401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.569 7f3309ffb700 1 -- 172.21.15.32:0/719438590 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f330c032df0 con 0x7f331409e9c0 2021-04-09T18:13:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.569 7f3319651700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32f4018930 0x7f32f401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.570 7f3319651700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32f4018930 0x7f32f401ae10 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.589 7f331b835700 1 -- 172.21.15.32:0/719438590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33140bde80 con 0x7f331409e9c0 2021-04-09T18:13:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.593 7f3309ffb700 1 -- 172.21.15.32:0/719438590 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f330c006dd0 con 0x7f331409e9c0 2021-04-09T18:13:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:25.776 7f331b835700 1 -- 172.21.15.32:0/719438590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f33140bde80 con 0x7f331409e9c0 2021-04-09T18:13:26.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.339 7f3309ffb700 1 -- 172.21.15.32:0/719438590 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v22) v1 ==== 185+0+0 (secure 0 0 0) 0x7f330c0014f0 con 0x7f331409e9c0 2021-04-09T18:13:26.362 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:13:26.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 -- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32f4018930 msgr2=0x7f32f401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:26.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32f4018930 0x7f32f401ae10 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 -- 172.21.15.32:0/719438590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331409e9c0 msgr2=0x7f331409edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:26.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331409e9c0 0x7f331409edc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f330c001070 tx=0x7f330c00bd40).stop 2021-04-09T18:13:26.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 -- 172.21.15.32:0/719438590 shutdown_connections 2021-04-09T18:13:26.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32f4018930 0x7f32f401ae10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f331409e9c0 0x7f331409edc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f331409a8b0 0x7f331409e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 --2- 172.21.15.32:0/719438590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f331409b2f0 0x7f331409b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 -- 172.21.15.32:0/719438590 >> 172.21.15.32:0/719438590 conn(0x7f33140ae6f0 msgr2=0x7f3314097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:26.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 -- 172.21.15.32:0/719438590 shutdown_connections 2021-04-09T18:13:26.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.344 7f331b835700 1 -- 172.21.15.32:0/719438590 wait complete. 2021-04-09T18:13:26.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:13:26.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.543 7f4dba0b5700 1 Processor -- start 2021-04-09T18:13:26.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.543 7f4dba0b5700 1 -- start start 2021-04-09T18:13:26.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4dba0b5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4db410d020 0x7f4db40d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:26.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4dba0b5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40d84a0 0x7f4db410da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4dba0b5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db410d490 con 0x7f4db40d1db0 2021-04-09T18:13:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4dba0b5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db40d8d40 con 0x7f4db410d020 2021-04-09T18:13:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4dba0b5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40d9230 con 0x7f4db40d84a0 2021-04-09T18:13:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4db3fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40d84a0 0x7f4db410da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4db37fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4db410d020 0x7f4db40d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4db2ffd700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4db40d1db0 0x7f4db40d7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:49908/0 (socket says 172.21.15.32:49908) 2021-04-09T18:13:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4db2ffd700 1 -- 172.21.15.32:0/3653701550 learned_addr learned my addr 172.21.15.32:0/3653701550 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4db27fc700 1 -- 172.21.15.32:0/3653701550 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669008354 0 0) 0x7f4d9c0010c0 con 0x7f4db40d1db0 2021-04-09T18:13:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4db27fc700 1 -- 172.21.15.32:0/3653701550 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da0002fe0 con 0x7f4db40d1db0 2021-04-09T18:13:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.544 7f4db3fff700 1 -- 172.21.15.32:0/3653701550 >> v2:172.21.15.86:3300/0 conn(0x7f4db410d020 msgr2=0x7f4db40d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4db3fff700 1 --2- 172.21.15.32:0/3653701550 >> v2:172.21.15.86:3300/0 conn(0x7f4db410d020 0x7f4db40d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4db3fff700 1 -- 172.21.15.32:0/3653701550 >> v1:172.21.15.86:6789/0 conn(0x7f4db40d1db0 legacy=0x7f4db40d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4db3fff700 1 -- 172.21.15.32:0/3653701550 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db410e030 con 0x7f4db40d84a0 2021-04-09T18:13:26.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4db37fe700 1 --2- 172.21.15.32:0/3653701550 >> v2:172.21.15.86:3300/0 conn(0x7f4db410d020 0x7f4db40d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:26.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4db3fff700 1 --2- 172.21.15.32:0/3653701550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40d84a0 0x7f4db410da10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f4dac006e50 tx=0x7f4dac009e70).ready entity=mon.2 client_cookie=1bd673944922c4a2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:26.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4db27fc700 1 -- 172.21.15.32:0/3653701550 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac006a20 con 0x7f4db40d84a0 2021-04-09T18:13:26.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4db27fc700 1 -- 172.21.15.32:0/3653701550 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac006790 con 0x7f4db40d84a0 2021-04-09T18:13:26.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4db27fc700 1 -- 172.21.15.32:0/3653701550 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac00b700 con 0x7f4db40d84a0 2021-04-09T18:13:26.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4dba0b5700 1 -- 172.21.15.32:0/3653701550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40d84a0 msgr2=0x7f4db410da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:26.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.545 7f4dba0b5700 1 --2- 172.21.15.32:0/3653701550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40d84a0 0x7f4db410da10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f4dac006e50 tx=0x7f4dac009e70).stop 2021-04-09T18:13:26.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.546 7f4dba0b5700 1 -- 172.21.15.32:0/3653701550 shutdown_connections 2021-04-09T18:13:26.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.546 7f4dba0b5700 1 --2- 172.21.15.32:0/3653701550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40d84a0 0x7f4db410da10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.546 7f4dba0b5700 1 --2- 172.21.15.32:0/3653701550 >> v2:172.21.15.86:3300/0 conn(0x7f4db410d020 0x7f4db40d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.546 7f4dba0b5700 1 -- 172.21.15.32:0/3653701550 >> 172.21.15.32:0/3653701550 conn(0x7f4db40ce600 msgr2=0x7f4db40d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:26.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.546 7f4dba0b5700 1 -- 172.21.15.32:0/3653701550 shutdown_connections 2021-04-09T18:13:26.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.546 7f4dba0b5700 1 -- 172.21.15.32:0/3653701550 wait complete. 2021-04-09T18:13:26.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.566 7f4dba0b5700 1 Processor -- start 2021-04-09T18:13:26.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.566 7f4dba0b5700 1 -- start start 2021-04-09T18:13:26.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dba0b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40c3360 0x7f4db40c3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:26.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dba0b5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40be760 0x7f4db40beb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dba0b5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db419fc20 0x7f4db41a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:26.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dba0b5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40be1e0 con 0x7f4db40c3360 2021-04-09T18:13:26.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dba0b5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40c4010 con 0x7f4db419fc20 2021-04-09T18:13:26.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dba0b5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41a0640 con 0x7f4db40be760 2021-04-09T18:13:26.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40c3360 0x7f4db40c3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:26.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40c3360 0x7f4db40c3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42398/0 (socket says 172.21.15.32:42398) 2021-04-09T18:13:26.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db37fe700 1 -- 172.21.15.32:0/1424467459 learned_addr learned my addr 172.21.15.32:0/1424467459 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:26.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db2ffd700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40be760 0x7f4db40beb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:26.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db3fff700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db419fc20 0x7f4db41a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:26.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db37fe700 1 -- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40be760 msgr2=0x7f4db40beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:26.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db37fe700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40be760 0x7f4db40beb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db37fe700 1 -- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db419fc20 msgr2=0x7f4db41a0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:26.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db37fe700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db419fc20 0x7f4db41a0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:26.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db37fe700 1 -- 172.21.15.32:0/1424467459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db41a0860 con 0x7f4db40c3360 2021-04-09T18:13:26.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4db37fe700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40c3360 0x7f4db40c3980 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4da4006f40 tx=0x7f4da400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:26.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dabfff700 1 -- 172.21.15.32:0/1424467459 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da400bd80 con 0x7f4db40c3360 2021-04-09T18:13:26.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dabfff700 1 -- 172.21.15.32:0/1424467459 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da4000c90 con 0x7f4db40c3360 2021-04-09T18:13:26.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dabfff700 1 -- 172.21.15.32:0/1424467459 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da4002220 con 0x7f4db40c3360 2021-04-09T18:13:26.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db41a0860 con 0x7f4db40c3360 2021-04-09T18:13:26.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.567 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db41a0860 con 0x7f4db40c3360 2021-04-09T18:13:26.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.568 7f4dabfff700 1 -- 172.21.15.32:0/1424467459 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4da400a000 con 0x7f4db40c3360 2021-04-09T18:13:26.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.568 7f4dabfff700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d980188c0 0x7f4d9801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.568 7f4dabfff700 1 -- 172.21.15.32:0/1424467459 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5225+0+0 (secure 0 0 0) 0x7f4da4032660 con 0x7f4db40c3360 2021-04-09T18:13:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.568 7f4db2ffd700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d980188c0 0x7f4d9801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.569 7f4db2ffd700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d980188c0 0x7f4d9801adc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:26.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.588 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db40b8fb0 con 0x7f4db40c3360 2021-04-09T18:13:26.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.591 7f4dabfff700 1 -- 172.21.15.32:0/1424467459 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4da4006b00 con 0x7f4db40c3360 2021-04-09T18:13:26.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:26.775 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4db410d490 con 0x7f4db40c3360 2021-04-09T18:13:27.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.343 7f4dabfff700 1 -- 172.21.15.32:0/1424467459 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v23) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4da400a000 con 0x7f4db40c3360 2021-04-09T18:13:27.417 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:13:27.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d980188c0 msgr2=0x7f4d9801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:27.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d980188c0 0x7f4d9801adc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:27.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40c3360 msgr2=0x7f4db40c3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:27.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40c3360 0x7f4db40c3980 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4da4006f40 tx=0x7f4da400b990).stop 2021-04-09T18:13:27.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 shutdown_connections 2021-04-09T18:13:27.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d980188c0 0x7f4d9801adc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:27.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4db419fc20 0x7f4db41a0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:27.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4db40be760 0x7f4db40beb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:27.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 --2- 172.21.15.32:0/1424467459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4db40c3360 0x7f4db40c3980 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:27.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 >> 172.21.15.32:0/1424467459 conn(0x7f4db4097d10 msgr2=0x7f4db4091330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 shutdown_connections 2021-04-09T18:13:27.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.392 7f4dba0b5700 1 -- 172.21.15.32:0/1424467459 wait complete. 2021-04-09T18:13:27.457 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:13:27.457 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:13:27.458 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:13:27.458 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:13:27.468 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:13:27.468 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:13:27.469 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:13:27.469 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:13:27.470 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:13:27.470 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:13:27.471 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:13:27.472 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:13:27.472 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:13:27.473 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:13:27.473 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:13:27.520 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:13:27.559 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:13:27.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:13:27.583 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:13:27.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:13:27.628 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:13:27.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:13:27.629 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:13:27.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:13:27.692 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:13:27.693 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:13:27.694 DEBUG:teuthology.parallel:result is None 2021-04-09T18:13:27.736 DEBUG:teuthology.parallel:result is None 2021-04-09T18:13:27.737 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:13:27.737 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:13:27.738 DEBUG:teuthology.parallel:result is None 2021-04-09T18:13:27.738 DEBUG:teuthology.parallel:result is None 2021-04-09T18:13:27.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:13:27.804 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:13:27.816 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:13:27.832 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:13:27.845 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:13:27.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.891 7f49d24aa700 1 Processor -- start 2021-04-09T18:13:27.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.891 7f49d24aa700 1 -- start start 2021-04-09T18:13:27.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49d24aa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc0cffb0 0x7f49cc0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:27.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49d24aa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49cc0d7810 0x7f49cc0d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49d24aa700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc0d7290 con 0x7f49cc0d5c80 2021-04-09T18:13:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49d24aa700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49cc0d8590 con 0x7f49cc0d7810 2021-04-09T18:13:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49d24aa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0d91f0 con 0x7f49cc0cffb0 2021-04-09T18:13:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49c3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49cc0d7810 0x7f49cc0d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49d0ac7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f49cc0d5c80 0x7f49cc0d6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52016/0 (socket says 172.21.15.32:52016) 2021-04-09T18:13:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49c3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49cc0d7810 0x7f49cc0d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42428/0 (socket says 172.21.15.32:42428) 2021-04-09T18:13:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49d0ac7700 1 -- 172.21.15.32:0/1063014020 learned_addr learned my addr 172.21.15.32:0/1063014020 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49cbfff700 1 --2- 172.21.15.32:0/1063014020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc0cffb0 0x7f49cc0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49c3fff700 1 -- 172.21.15.32:0/1063014020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc0cffb0 msgr2=0x7f49cc0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:27.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49c3fff700 1 --2- 172.21.15.32:0/1063014020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc0cffb0 0x7f49cc0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:27.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49c3fff700 1 -- 172.21.15.32:0/1063014020 >> v1:172.21.15.32:6789/0 conn(0x7f49cc0d5c80 legacy=0x7f49cc0d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49c3fff700 1 -- 172.21.15.32:0/1063014020 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc0d8230 con 0x7f49cc0d7810 2021-04-09T18:13:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49cb7fe700 1 -- 172.21.15.32:0/1063014020 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239212188 0 0) 0x7f49c4000d30 con 0x7f49cc0d5c80 2021-04-09T18:13:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.892 7f49c3fff700 1 --2- 172.21.15.32:0/1063014020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0d7810 0x7f49cc0d7c10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f49b4006650 tx=0x7f49b4009a00).ready entity=mon.0 client_cookie=2ce2e40f144a4aad server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:28.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49cb7fe700 1 -- 172.21.15.32:0/1063014020 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b4009df0 con 0x7f49cc0d7810 2021-04-09T18:13:28.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49cb7fe700 1 -- 172.21.15.32:0/1063014020 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b40009b0 con 0x7f49cc0d7810 2021-04-09T18:13:28.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49cb7fe700 1 -- 172.21.15.32:0/1063014020 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b400c2b0 con 0x7f49cc0d7810 2021-04-09T18:13:28.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49d24aa700 1 -- 172.21.15.32:0/1063014020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0d7810 msgr2=0x7f49cc0d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49d24aa700 1 --2- 172.21.15.32:0/1063014020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0d7810 0x7f49cc0d7c10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f49b4006650 tx=0x7f49b4009a00).stop 2021-04-09T18:13:28.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49d24aa700 1 -- 172.21.15.32:0/1063014020 shutdown_connections 2021-04-09T18:13:28.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49d24aa700 1 --2- 172.21.15.32:0/1063014020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0d7810 0x7f49cc0d7c10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49d24aa700 1 --2- 172.21.15.32:0/1063014020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc0cffb0 0x7f49cc0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49d24aa700 1 -- 172.21.15.32:0/1063014020 >> 172.21.15.32:0/1063014020 conn(0x7f49cc0ce5d0 msgr2=0x7f49cc0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:28.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49d24aa700 1 -- 172.21.15.32:0/1063014020 shutdown_connections 2021-04-09T18:13:28.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.893 7f49d24aa700 1 -- 172.21.15.32:0/1063014020 wait complete. 2021-04-09T18:13:28.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49d24aa700 1 Processor -- start 2021-04-09T18:13:28.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49d24aa700 1 -- start start 2021-04-09T18:13:28.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49d24aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0a54b0 0x7f49cc197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49d24aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49cc1a9ef0 0x7f49cc1aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49d24aa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc1aa2f0 0x7f49cc1ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49d24aa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc1aa900 con 0x7f49cc0a54b0 2021-04-09T18:13:28.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49d24aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0a58b0 con 0x7f49cc1a9ef0 2021-04-09T18:13:28.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49d24aa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc1984f0 con 0x7f49cc1aa2f0 2021-04-09T18:13:28.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0a54b0 0x7f49cc197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0a54b0 0x7f49cc197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42430/0 (socket says 172.21.15.32:42430) 2021-04-09T18:13:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49cbfff700 1 -- 172.21.15.32:0/3893648844 learned_addr learned my addr 172.21.15.32:0/3893648844 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.914 7f49c3fff700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49cc1a9ef0 0x7f49cc1aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49d0ac7700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc1aa2f0 0x7f49cc1ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49cbfff700 1 -- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc1aa2f0 msgr2=0x7f49cc1ab4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49cbfff700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc1aa2f0 0x7f49cc1ab4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49cbfff700 1 -- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49cc1a9ef0 msgr2=0x7f49cc1aae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49cbfff700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49cc1a9ef0 0x7f49cc1aae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49cbfff700 1 -- 172.21.15.32:0/3893648844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc0a5ce0 con 0x7f49cc0a54b0 2021-04-09T18:13:28.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49cbfff700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0a54b0 0x7f49cc197ed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f49bc00b560 tx=0x7f49bc00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:28.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49c97fa700 1 -- 172.21.15.32:0/3893648844 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49bc00bcf0 con 0x7f49cc0a54b0 2021-04-09T18:13:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49c97fa700 1 -- 172.21.15.32:0/3893648844 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc000c90 con 0x7f49cc0a54b0 2021-04-09T18:13:28.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc0a5ce0 con 0x7f49cc0a54b0 2021-04-09T18:13:28.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49c97fa700 1 -- 172.21.15.32:0/3893648844 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49bc002220 con 0x7f49cc0a54b0 2021-04-09T18:13:28.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.915 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc0a5ce0 con 0x7f49cc0a54b0 2021-04-09T18:13:28.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.916 7f49c97fa700 1 -- 172.21.15.32:0/3893648844 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f49bc0052e0 con 0x7f49cc0a54b0 2021-04-09T18:13:28.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.916 7f49c97fa700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b00188c0 0x7f49b001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.916 7f49c97fa700 1 -- 172.21.15.32:0/3893648844 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f49bc0328c0 con 0x7f49cc0a54b0 2021-04-09T18:13:28.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.916 7f49c3fff700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b00188c0 0x7f49b001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.916 7f49c3fff700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b00188c0 0x7f49b001adc0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:28.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.936 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc182e20 con 0x7f49cc0a54b0 2021-04-09T18:13:28.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:27.939 7f49c97fa700 1 -- 172.21.15.32:0/3893648844 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f49bc000c90 con 0x7f49cc0a54b0 2021-04-09T18:13:28.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.138 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f49cc182e20 con 0x7f49cc0a54b0 2021-04-09T18:13:28.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.389 7f49c97fa700 1 -- 172.21.15.32:0/3893648844 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v24) v1 ==== 97+0+0 (secure 0 0 0) 0x7f49bc0052e0 con 0x7f49cc0a54b0 2021-04-09T18:13:28.429 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:13:28.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b00188c0 msgr2=0x7f49b001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b00188c0 0x7f49b001adc0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0a54b0 msgr2=0x7f49cc197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0a54b0 0x7f49cc197ed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f49bc00b560 tx=0x7f49bc00b900).stop 2021-04-09T18:13:28.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 shutdown_connections 2021-04-09T18:13:28.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b00188c0 0x7f49b001adc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49cc1aa2f0 0x7f49cc1ab4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49cc1a9ef0 0x7f49cc1aae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 --2- 172.21.15.32:0/3893648844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49cc0a54b0 0x7f49cc197ed0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 >> 172.21.15.32:0/3893648844 conn(0x7f49cc1838a0 msgr2=0x7f49cc1a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:28.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 shutdown_connections 2021-04-09T18:13:28.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.404 7f49d24aa700 1 -- 172.21.15.32:0/3893648844 wait complete. 2021-04-09T18:13:28.449 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:13:28.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:13:28.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.596 7f2ca82d1700 1 Processor -- start 2021-04-09T18:13:28.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.596 7f2ca82d1700 1 -- start start 2021-04-09T18:13:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca82d1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca010d010 0x7f2ca00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca82d1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ca00d1da0 0x7f2ca00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca82d1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca010d480 con 0x7f2ca00d82c0 2021-04-09T18:13:28.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca82d1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca010e780 con 0x7f2ca00d1da0 2021-04-09T18:13:28.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca82d1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00d8d30 con 0x7f2ca010d010 2021-04-09T18:13:28.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca58ec700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ca00d1da0 0x7f2ca00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca58ec700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2ca00d1da0 0x7f2ca00d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42440/0 (socket says 172.21.15.32:42440) 2021-04-09T18:13:28.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca58ec700 1 -- 172.21.15.32:0/2752786886 learned_addr learned my addr 172.21.15.32:0/2752786886 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:28.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca60ed700 1 --2- 172.21.15.32:0/2752786886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca010d010 0x7f2ca00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca58ec700 1 -- 172.21.15.32:0/2752786886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca010d010 msgr2=0x7f2ca00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:28.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.597 7f2ca58ec700 1 --2- 172.21.15.32:0/2752786886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca010d010 0x7f2ca00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.598 7f2ca58ec700 1 -- 172.21.15.32:0/2752786886 >> v1:172.21.15.86:6789/0 conn(0x7f2ca00d82c0 legacy=0x7f2ca010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.598 7f2ca58ec700 1 -- 172.21.15.32:0/2752786886 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca00d9220 con 0x7f2ca00d1da0 2021-04-09T18:13:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.598 7f2ca50eb700 1 -- 172.21.15.32:0/2752786886 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257205235 0 0) 0x7f2c98000d30 con 0x7f2ca00d82c0 2021-04-09T18:13:28.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.598 7f2ca58ec700 1 --2- 172.21.15.32:0/2752786886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca00d1da0 0x7f2ca00d7ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f2c90006700 tx=0x7f2c90009b80).ready entity=mon.0 client_cookie=ed4df70efdf9dce3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.598 7f2ca50eb700 1 -- 172.21.15.32:0/2752786886 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9000c0c0 con 0x7f2ca00d1da0 2021-04-09T18:13:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.598 7f2ca50eb700 1 -- 172.21.15.32:0/2752786886 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9000c440 con 0x7f2ca00d1da0 2021-04-09T18:13:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.598 7f2ca50eb700 1 -- 172.21.15.32:0/2752786886 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9000c920 con 0x7f2ca00d1da0 2021-04-09T18:13:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.598 7f2ca82d1700 1 -- 172.21.15.32:0/2752786886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca00d1da0 msgr2=0x7f2ca00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.598 7f2ca82d1700 1 --2- 172.21.15.32:0/2752786886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca00d1da0 0x7f2ca00d7ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f2c90006700 tx=0x7f2c90009b80).stop 2021-04-09T18:13:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.599 7f2ca82d1700 1 -- 172.21.15.32:0/2752786886 shutdown_connections 2021-04-09T18:13:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.599 7f2ca82d1700 1 --2- 172.21.15.32:0/2752786886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca00d1da0 0x7f2ca00d7ca0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.599 7f2ca82d1700 1 --2- 172.21.15.32:0/2752786886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca010d010 0x7f2ca00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.599 7f2ca82d1700 1 -- 172.21.15.32:0/2752786886 >> 172.21.15.32:0/2752786886 conn(0x7f2ca00ce5f0 msgr2=0x7f2ca00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.599 7f2ca82d1700 1 -- 172.21.15.32:0/2752786886 shutdown_connections 2021-04-09T18:13:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.599 7f2ca82d1700 1 -- 172.21.15.32:0/2752786886 wait complete. 2021-04-09T18:13:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.619 7f2ca82d1700 1 Processor -- start 2021-04-09T18:13:28.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.619 7f2ca82d1700 1 -- start start 2021-04-09T18:13:28.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.619 7f2ca82d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca01a5210 0x7f2ca0185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.619 7f2ca82d1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ca0185a70 0x7f2ca00acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.619 7f2ca82d1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca0185e70 0x7f2ca00ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.619 7f2ca82d1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca018f7b0 con 0x7f2ca01a5210 2021-04-09T18:13:28.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.619 7f2ca82d1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00ad7f0 con 0x7f2ca0185a70 2021-04-09T18:13:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.619 7f2ca82d1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00ada10 con 0x7f2ca0185e70 2021-04-09T18:13:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca60ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca01a5210 0x7f2ca0185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca60ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca01a5210 0x7f2ca0185450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42444/0 (socket says 172.21.15.32:42444) 2021-04-09T18:13:28.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca60ed700 1 -- 172.21.15.32:0/1804672877 learned_addr learned my addr 172.21.15.32:0/1804672877 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:28.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca58ec700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ca0185a70 0x7f2ca00acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca68ee700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca0185e70 0x7f2ca00ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca60ed700 1 -- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca0185e70 msgr2=0x7f2ca00ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca60ed700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca0185e70 0x7f2ca00ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca60ed700 1 -- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ca0185a70 msgr2=0x7f2ca00acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:28.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca60ed700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ca0185a70 0x7f2ca00acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:28.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca60ed700 1 -- 172.21.15.32:0/1804672877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0179610 con 0x7f2ca01a5210 2021-04-09T18:13:28.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca60ed700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca01a5210 0x7f2ca0185450 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f2c9c006f60 tx=0x7f2c9c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:28.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2c96ffd700 1 -- 172.21.15.32:0/1804672877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9c00bd60 con 0x7f2ca01a5210 2021-04-09T18:13:28.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2c96ffd700 1 -- 172.21.15.32:0/1804672877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9c000c90 con 0x7f2ca01a5210 2021-04-09T18:13:28.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca0179610 con 0x7f2ca01a5210 2021-04-09T18:13:28.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2c96ffd700 1 -- 172.21.15.32:0/1804672877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9c002220 con 0x7f2ca01a5210 2021-04-09T18:13:28.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.620 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca0179610 con 0x7f2ca01a5210 2021-04-09T18:13:28.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.621 7f2c96ffd700 1 -- 172.21.15.32:0/1804672877 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2c9c0052e0 con 0x7f2ca01a5210 2021-04-09T18:13:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.621 7f2c96ffd700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2c880188d0 0x7f2c8801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.621 7f2c96ffd700 1 -- 172.21.15.32:0/1804672877 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f2c9c0326d0 con 0x7f2ca01a5210 2021-04-09T18:13:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.621 7f2ca58ec700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2c880188d0 0x7f2c8801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.621 7f2ca58ec700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2c880188d0 0x7f2c8801add0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:28.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.641 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca018f7b0 con 0x7f2ca01a5210 2021-04-09T18:13:28.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.644 7f2c96ffd700 1 -- 172.21.15.32:0/1804672877 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2c9c000c90 con 0x7f2ca01a5210 2021-04-09T18:13:28.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.829 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2ca018f7b0 con 0x7f2ca01a5210 2021-04-09T18:13:28.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.829 7f2c96ffd700 1 -- 172.21.15.32:0/1804672877 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v7) v1 ==== 80+0+5905 (secure 0 0 0) 0x7f2c9c006b00 con 0x7f2ca01a5210 2021-04-09T18:13:28.852 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:28.853 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:13:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:13:28.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:13:28.854 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:13:28.855 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:13:28.856 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:28.857 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:28.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:13:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:13:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:28.860 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:13:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:28.861 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:28.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:28.865 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:13:28.890 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:13:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:13:28.891 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:28.892 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:28.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:28.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:13:28.896 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:13:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:28.897 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:13:28.963 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:28.964 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:28.964 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:28.965 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:28.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:28.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:28.966 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:13:28.966 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:13:28.967 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.015 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:29.016 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:29.016 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.017 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.018 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:13:29.018 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:13:29.019 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.019 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:29.020 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:29.053 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.054 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.071 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:13:29.072 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:13:29.072 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.105 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:29.106 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:29.106 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.107 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.108 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:13:29.108 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:13:29.108 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.109 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:29.134 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:29.134 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.135 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.155 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:13:29.209 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:13:29.217 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.241 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:29.246 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:29.247 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.247 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.248 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:13:29.248 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:13:29.248 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.248 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:29.249 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:29.274 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.275 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:13:29.275 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:13:29.275 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.275 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:29.276 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:29.276 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.276 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.323 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:13:29.339 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:13:29.339 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.340 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:29.340 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:29.340 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.341 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.403 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:13:29.403 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:13:29.403 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.404 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:13:29.404 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2021-04-09T18:13:29.405 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2021-04-09T18:13:29.408 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2021-04-09T18:13:29.466 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.467 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:13:29.467 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:13:29.468 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.468 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:13:29.469 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow r", 2021-04-09T18:13:29.469 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow rw", 2021-04-09T18:13:29.469 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rwx" 2021-04-09T18:13:29.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.495 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:13:29.495 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:13:29.495 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.495 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:13:29.498 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:13:29.499 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:13:29.499 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.499 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.500 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-mds", 2021-04-09T18:13:29.500 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8mHBgdR6aHhAA31TGmKf/8VLqsLqqa8uZNA==", 2021-04-09T18:13:29.500 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.500 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-mds" 2021-04-09T18:13:29.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.530 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-mgr", 2021-04-09T18:13:29.531 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8mHBg8CqaHhAAE6TsKkR3NgF/OBz9f2x/gQ==", 2021-04-09T18:13:29.531 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.531 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-mgr" 2021-04-09T18:13:29.531 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.532 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-osd", 2021-04-09T18:13:29.546 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8mHBgPDSaHhAAWRSLb5+48wlN9o8Am58yeg==", 2021-04-09T18:13:29.568 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.568 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-osd" 2021-04-09T18:13:29.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.577 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-rbd", 2021-04-09T18:13:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8mHBghD2aHhAAUBlm6ajmIt4cXxFRchKZmw==", 2021-04-09T18:13:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-rbd" 2021-04-09T18:13:29.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.684 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-rbd-mirror", 2021-04-09T18:13:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8mHBg5UaaHhAAbSMUI9ChzZ3mtNZBWA3OtQ==", 2021-04-09T18:13:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2021-04-09T18:13:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.708 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.bootstrap-rgw", 2021-04-09T18:13:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQD8mHBg4U+aHhAArfnazln5aO1fpLAtAYzTuA==", 2021-04-09T18:13:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile bootstrap-rgw" 2021-04-09T18:13:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:13:29.710 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:13:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.711 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:13:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:13:29.732 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:29.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:13:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:13:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:13:29.734 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:13:29.738 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:13:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:13:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:29.742 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:13:29.742 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:13:29.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2c880188d0 msgr2=0x7f2c8801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2c880188d0 0x7f2c8801add0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:29.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca01a5210 msgr2=0x7f2ca0185450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:29.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca01a5210 0x7f2ca0185450 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f2c9c006f60 tx=0x7f2c9c00b970).stop 2021-04-09T18:13:29.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 shutdown_connections 2021-04-09T18:13:29.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2c880188d0 0x7f2c8801add0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:29.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ca0185e70 0x7f2ca00ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:29.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ca0185a70 0x7f2ca00acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:29.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 --2- 172.21.15.32:0/1804672877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca01a5210 0x7f2ca0185450 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:29.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 >> 172.21.15.32:0/1804672877 conn(0x7f2ca01a99c0 msgr2=0x7f2ca017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:29.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 shutdown_connections 2021-04-09T18:13:29.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:28.832 7f2ca82d1700 1 -- 172.21.15.32:0/1804672877 wait complete. 2021-04-09T18:13:29.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2021-04-09T18:13:29.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.936 7f91f5579700 1 Processor -- start 2021-04-09T18:13:29.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.937 7f91f5579700 1 -- start start 2021-04-09T18:13:29.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.937 7f91f5579700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f91f00cffb0 0x7f91f00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:29.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.937 7f91f5579700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f00d7290 con 0x7f91f00d7810 2021-04-09T18:13:29.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.937 7f91f5579700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f00d8400 con 0x7f91f00d6c90 2021-04-09T18:13:29.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.937 7f91f5579700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00d7090 con 0x7f91f00cffb0 2021-04-09T18:13:29.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.937 7f91eeabe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f91f00d7810 0x7f91f00d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52046/0 (socket says 172.21.15.32:52046) 2021-04-09T18:13:29.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.937 7f91eeabe700 1 -- 172.21.15.32:0/3559921937 learned_addr learned my addr 172.21.15.32:0/3559921937 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:29.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.937 7f91ef2bf700 1 --2- 172.21.15.32:0/3559921937 >> v2:172.21.15.86:3300/0 conn(0x7f91f00cffb0 0x7f91f00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:29.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891979320 0 0) 0x7f91e40009b0 con 0x7f91f00d7810 2021-04-09T18:13:29.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d8002fe0 con 0x7f91f00d7810 2021-04-09T18:13:29.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594098651 0 0) 0x7f91e8000d30 con 0x7f91f00d6c90 2021-04-09T18:13:29.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d8003210 con 0x7f91f00d6c90 2021-04-09T18:13:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2539766898 0 0) 0x7f91e4001010 con 0x7f91f00d7810 2021-04-09T18:13:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 >> v2:172.21.15.86:3300/0 conn(0x7f91f00cffb0 msgr2=0x7f91f00d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 --2- 172.21.15.32:0/3559921937 >> v2:172.21.15.86:3300/0 conn(0x7f91f00cffb0 0x7f91f00d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 >> v1:172.21.15.86:6789/0 conn(0x7f91f00d6c90 legacy=0x7f91f010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f00d91f0 con 0x7f91f00d7810 2021-04-09T18:13:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f91e4001530 con 0x7f91f00d7810 2021-04-09T18:13:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 >> v1:172.21.15.32:6789/0 conn(0x7f91f00d7810 legacy=0x7f91f00d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d8003b20 0x7f91d8005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d8006780 0x7f91d8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d80095a0 0x7f91d800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8050320 con 0x7f91d80095a0 2021-04-09T18:13:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80504d0 con 0x7f91d8006780 2021-04-09T18:13:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80506f0 con 0x7f91d8003b20 2021-04-09T18:13:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91eeabe700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d8006780 0x7f91d8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91efac0700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d80095a0 0x7f91d800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ef2bf700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d8003b20 0x7f91d8005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91efac0700 1 -- 172.21.15.32:0/3559921937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d8003b20 msgr2=0x7f91d8005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:30.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91efac0700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d8003b20 0x7f91d8005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91efac0700 1 -- 172.21.15.32:0/3559921937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d8006780 msgr2=0x7f91d8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:30.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91efac0700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d8006780 0x7f91d8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91efac0700 1 -- 172.21.15.32:0/3559921937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d8050990 con 0x7f91d80095a0 2021-04-09T18:13:30.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.938 7f91ef2bf700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d8003b20 0x7f91d8005f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:30.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91efac0700 1 -- 172.21.15.32:0/3559921937 reap_dead start 2021-04-09T18:13:30.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91efac0700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d80095a0 0x7f91d800bac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f91e80013c0 tx=0x7f91e8009e20).ready entity=mon.0 client_cookie=1f4b98a73af838ac server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:30.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e8003020 con 0x7f91d80095a0 2021-04-09T18:13:30.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e8001400 con 0x7f91d80095a0 2021-04-09T18:13:30.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91ee2bd700 1 -- 172.21.15.32:0/3559921937 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e8001860 con 0x7f91d80095a0 2021-04-09T18:13:30.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91f5579700 1 -- 172.21.15.32:0/3559921937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d80095a0 msgr2=0x7f91d800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91f5579700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d80095a0 0x7f91d800bac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f91e80013c0 tx=0x7f91e8009e20).stop 2021-04-09T18:13:30.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91f5579700 1 -- 172.21.15.32:0/3559921937 shutdown_connections 2021-04-09T18:13:30.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91f5579700 1 --2- 172.21.15.32:0/3559921937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d80095a0 0x7f91d800bac0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91f5579700 1 -- 172.21.15.32:0/3559921937 >> 172.21.15.32:0/3559921937 conn(0x7f91f00ce5d0 msgr2=0x7f91f00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:30.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91f5579700 1 -- 172.21.15.32:0/3559921937 shutdown_connections 2021-04-09T18:13:30.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.939 7f91f5579700 1 -- 172.21.15.32:0/3559921937 wait complete. 2021-04-09T18:13:30.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91f5579700 1 Processor -- start 2021-04-09T18:13:30.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91f5579700 1 -- start start 2021-04-09T18:13:30.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91f5579700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f01a7dc0 0x7f91f01a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91f5579700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0184080 0x7f91f0184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91f5579700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f0184aa0 0x7f91f008f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91f5579700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f01a8b40 con 0x7f91f0184080 2021-04-09T18:13:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91f5579700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f01a7870 con 0x7f91f01a7dc0 2021-04-09T18:13:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91f5579700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f01a87e0 con 0x7f91f0184aa0 2021-04-09T18:13:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91eeabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0184080 0x7f91f0184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.960 7f91eeabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0184080 0x7f91f0184480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42464/0 (socket says 172.21.15.32:42464) 2021-04-09T18:13:30.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91eeabe700 1 -- 172.21.15.32:0/324719295 learned_addr learned my addr 172.21.15.32:0/324719295 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91ef2bf700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f01a7dc0 0x7f91f01a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91efac0700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f0184aa0 0x7f91f008f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91eeabe700 1 -- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f0184aa0 msgr2=0x7f91f008f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91eeabe700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f0184aa0 0x7f91f008f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91eeabe700 1 -- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f01a7dc0 msgr2=0x7f91f01a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91eeabe700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f01a7dc0 0x7f91f01a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91eeabe700 1 -- 172.21.15.32:0/324719295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f008f810 con 0x7f91f0184080 2021-04-09T18:13:30.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91eeabe700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0184080 0x7f91f0184480 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f91e4001c60 tx=0x7f91e400a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:30.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91d7fff700 1 -- 172.21.15.32:0/324719295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e4001630 con 0x7f91f0184080 2021-04-09T18:13:30.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91d7fff700 1 -- 172.21.15.32:0/324719295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e400ad40 con 0x7f91f0184080 2021-04-09T18:13:30.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91f5579700 1 -- 172.21.15.32:0/324719295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f008f810 con 0x7f91f0184080 2021-04-09T18:13:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91d7fff700 1 -- 172.21.15.32:0/324719295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e40043c0 con 0x7f91f0184080 2021-04-09T18:13:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.961 7f91f5579700 1 -- 172.21.15.32:0/324719295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f008f810 con 0x7f91f0184080 2021-04-09T18:13:30.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.962 7f91d7fff700 1 -- 172.21.15.32:0/324719295 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f91e4000db0 con 0x7f91f0184080 2021-04-09T18:13:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.962 7f91d7fff700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91d0018930 0x7f91d001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.962 7f91d7fff700 1 -- 172.21.15.32:0/324719295 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f91e4032700 con 0x7f91f0184080 2021-04-09T18:13:30.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.962 7f91ef2bf700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91d0018930 0x7f91d001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.962 7f91ef2bf700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91d0018930 0x7f91d001ae10 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:30.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.981 7f91f5579700 1 -- 172.21.15.32:0/324719295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f0182e80 con 0x7f91f0184080 2021-04-09T18:13:30.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:29.985 7f91d7fff700 1 -- 172.21.15.32:0/324719295 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f91e40009a0 con 0x7f91f0184080 2021-04-09T18:13:30.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.173 7f91f5579700 1 -- 172.21.15.32:0/324719295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f91f00d7290 con 0x7f91f0184080 2021-04-09T18:13:30.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.176 7f91d7fff700 1 -- 172.21.15.32:0/324719295 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7f91e4000db0 con 0x7f91f0184080 2021-04-09T18:13:30.208 INFO:teuthology.orchestra.run.smithi032.stderr:updated 2021-04-09T18:13:30.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 -- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91d0018930 msgr2=0x7f91d001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91d0018930 0x7f91d001ae10 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 -- 172.21.15.32:0/324719295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0184080 msgr2=0x7f91f0184480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0184080 0x7f91f0184480 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f91e4001c60 tx=0x7f91e400a950).stop 2021-04-09T18:13:30.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 -- 172.21.15.32:0/324719295 shutdown_connections 2021-04-09T18:13:30.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91d0018930 0x7f91d001ae10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91f0184080 0x7f91f0184480 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f0184aa0 0x7f91f008f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 --2- 172.21.15.32:0/324719295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f01a7dc0 0x7f91f01a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 -- 172.21.15.32:0/324719295 >> 172.21.15.32:0/324719295 conn(0x7f91f01a8f50 msgr2=0x7f91f0188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:30.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 -- 172.21.15.32:0/324719295 shutdown_connections 2021-04-09T18:13:30.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.178 7f91f5579700 1 -- 172.21.15.32:0/324719295 wait complete. 2021-04-09T18:13:30.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2021-04-09T18:13:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.386 7f5763e05700 1 Processor -- start 2021-04-09T18:13:30.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.387 7f5763e05700 1 -- start start 2021-04-09T18:13:30.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.387 7f5763e05700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c0d6c90 0x7f575c0d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.387 7f5763e05700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f575c0d2450 0x7f575c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.387 7f5763e05700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c0d7290 con 0x7f575c0cffb0 2021-04-09T18:13:30.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.387 7f5763e05700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c0d7090 con 0x7f575c0d2450 2021-04-09T18:13:30.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.387 7f5763e05700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0d91f0 con 0x7f575c0d6c90 2021-04-09T18:13:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.387 7f5762422700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f575c0d2450 0x7f575c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.387 7f5761c21700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f575c0cffb0 0x7f575c0d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52062/0 (socket says 172.21.15.32:52062) 2021-04-09T18:13:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.387 7f5761c21700 1 -- 172.21.15.32:0/1830339651 learned_addr learned my addr 172.21.15.32:0/1830339651 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:30.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5761420700 1 --2- 172.21.15.32:0/1830339651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c0d6c90 0x7f575c0d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5760c1f700 1 -- 172.21.15.32:0/1830339651 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763451700 0 0) 0x7f5758000ce0 con 0x7f575c0cffb0 2021-04-09T18:13:30.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5760c1f700 1 -- 172.21.15.32:0/1830339651 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5748002fe0 con 0x7f575c0cffb0 2021-04-09T18:13:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5762422700 1 -- 172.21.15.32:0/1830339651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c0d6c90 msgr2=0x7f575c0d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5762422700 1 --2- 172.21.15.32:0/1830339651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c0d6c90 0x7f575c0d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5762422700 1 -- 172.21.15.32:0/1830339651 >> v1:172.21.15.32:6789/0 conn(0x7f575c0cffb0 legacy=0x7f575c0d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5762422700 1 -- 172.21.15.32:0/1830339651 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0d2850 con 0x7f575c0d2450 2021-04-09T18:13:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5762422700 1 --2- 172.21.15.32:0/1830339651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c0d2450 0x7f575c10cfe0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f5754006e30 tx=0x7f5754009e60).ready entity=mon.0 client_cookie=e590dee2fdf2543b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5760c1f700 1 -- 172.21.15.32:0/1830339651 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5754006a50 con 0x7f575c0d2450 2021-04-09T18:13:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5760c1f700 1 -- 172.21.15.32:0/1830339651 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575400b300 con 0x7f575c0d2450 2021-04-09T18:13:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5760c1f700 1 -- 172.21.15.32:0/1830339651 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f575400b750 con 0x7f575c0d2450 2021-04-09T18:13:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5763e05700 1 -- 172.21.15.32:0/1830339651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c0d2450 msgr2=0x7f575c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.388 7f5763e05700 1 --2- 172.21.15.32:0/1830339651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c0d2450 0x7f575c10cfe0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f5754006e30 tx=0x7f5754009e60).stop 2021-04-09T18:13:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.389 7f5763e05700 1 -- 172.21.15.32:0/1830339651 shutdown_connections 2021-04-09T18:13:30.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.389 7f5763e05700 1 --2- 172.21.15.32:0/1830339651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c0d2450 0x7f575c10cfe0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.389 7f5763e05700 1 --2- 172.21.15.32:0/1830339651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c0d6c90 0x7f575c0d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.389 7f5763e05700 1 -- 172.21.15.32:0/1830339651 >> 172.21.15.32:0/1830339651 conn(0x7f575c0ce5d0 msgr2=0x7f575c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.389 7f5763e05700 1 -- 172.21.15.32:0/1830339651 shutdown_connections 2021-04-09T18:13:30.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.389 7f5763e05700 1 -- 172.21.15.32:0/1830339651 wait complete. 2021-04-09T18:13:30.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.409 7f5763e05700 1 Processor -- start 2021-04-09T18:13:30.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.409 7f5763e05700 1 -- start start 2021-04-09T18:13:30.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5763e05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c184080 0x7f575c1955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5763e05700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c195c00 0x7f575c08ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5763e05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f575c08f550 0x7f575c08f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5763e05700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c090570 con 0x7f575c184080 2021-04-09T18:13:30.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5763e05700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c184480 con 0x7f575c08f550 2021-04-09T18:13:30.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5763e05700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c196000 con 0x7f575c195c00 2021-04-09T18:13:30.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761c21700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c184080 0x7f575c1955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761c21700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c184080 0x7f575c1955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42478/0 (socket says 172.21.15.32:42478) 2021-04-09T18:13:30.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761c21700 1 -- 172.21.15.32:0/2023301254 learned_addr learned my addr 172.21.15.32:0/2023301254 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:30.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761420700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c195c00 0x7f575c08ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5762422700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f575c08f550 0x7f575c08f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761c21700 1 -- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c195c00 msgr2=0x7f575c08ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761c21700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c195c00 0x7f575c08ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761c21700 1 -- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f575c08f550 msgr2=0x7f575c08f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761c21700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f575c08f550 0x7f575c08f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761c21700 1 -- 172.21.15.32:0/2023301254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c1847b0 con 0x7f575c184080 2021-04-09T18:13:30.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5762422700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f575c08f550 0x7f575c08f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:30.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f5761c21700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c184080 0x7f575c1955e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f575800a5a0 tx=0x7f575800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f57527fc700 1 -- 172.21.15.32:0/2023301254 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f575800ad30 con 0x7f575c184080 2021-04-09T18:13:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.410 7f57527fc700 1 -- 172.21.15.32:0/2023301254 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57580070d0 con 0x7f575c184080 2021-04-09T18:13:30.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.411 7f57527fc700 1 -- 172.21.15.32:0/2023301254 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5758004590 con 0x7f575c184080 2021-04-09T18:13:30.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.411 7f5763e05700 1 -- 172.21.15.32:0/2023301254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c1847b0 con 0x7f575c184080 2021-04-09T18:13:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.411 7f5763e05700 1 -- 172.21.15.32:0/2023301254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c1847b0 con 0x7f575c184080 2021-04-09T18:13:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.411 7f57527fc700 1 -- 172.21.15.32:0/2023301254 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5758008000 con 0x7f575c184080 2021-04-09T18:13:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.411 7f57527fc700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57440188c0 0x7f574401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.411 7f57527fc700 1 -- 172.21.15.32:0/2023301254 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f57580326c0 con 0x7f575c184080 2021-04-09T18:13:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.412 7f5761420700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57440188c0 0x7f574401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.412 7f5761420700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57440188c0 0x7f574401adc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:30.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.431 7f5763e05700 1 -- 172.21.15.32:0/2023301254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c182e80 con 0x7f575c184080 2021-04-09T18:13:30.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.435 7f57527fc700 1 -- 172.21.15.32:0/2023301254 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f57580009b0 con 0x7f575c184080 2021-04-09T18:13:30.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.621 7f5763e05700 1 -- 172.21.15.32:0/2023301254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f575c0d7290 con 0x7f575c184080 2021-04-09T18:13:30.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.624 7f57527fc700 1 -- 172.21.15.32:0/2023301254 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5758008000 con 0x7f575c184080 2021-04-09T18:13:30.945 INFO:teuthology.orchestra.run.smithi032.stderr:updated 2021-04-09T18:13:30.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 -- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57440188c0 msgr2=0x7f574401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57440188c0 0x7f574401adc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 -- 172.21.15.32:0/2023301254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c184080 msgr2=0x7f575c1955e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:30.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c184080 0x7f575c1955e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f575800a5a0 tx=0x7f575800a940).stop 2021-04-09T18:13:30.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 -- 172.21.15.32:0/2023301254 shutdown_connections 2021-04-09T18:13:30.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57440188c0 0x7f574401adc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f575c08f550 0x7f575c08f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f575c195c00 0x7f575c08ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 --2- 172.21.15.32:0/2023301254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f575c184080 0x7f575c1955e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 -- 172.21.15.32:0/2023301254 >> 172.21.15.32:0/2023301254 conn(0x7f575c1a8f50 msgr2=0x7f575c188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 -- 172.21.15.32:0/2023301254 shutdown_connections 2021-04-09T18:13:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:30.626 7f5763e05700 1 -- 172.21.15.32:0/2023301254 wait complete. 2021-04-09T18:13:30.987 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2021-04-09T18:13:31.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.210 7ff0e7b5c700 1 Processor -- start 2021-04-09T18:13:31.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.210 7ff0e7b5c700 1 -- start start 2021-04-09T18:13:31.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.210 7ff0e7b5c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e00cffb0 0x7ff0e00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:31.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.210 7ff0e7b5c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff0e00d7810 0x7ff0e00d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:31.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.210 7ff0e7b5c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e00d7290 con 0x7ff0e00d5c80 2021-04-09T18:13:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.210 7ff0e7b5c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e00d8590 con 0x7ff0e00d7810 2021-04-09T18:13:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.210 7ff0e7b5c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00d91f0 con 0x7ff0e00cffb0 2021-04-09T18:13:31.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e6179700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0e00d5c80 0x7ff0e00d6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52078/0 (socket says 172.21.15.32:52078) 2021-04-09T18:13:31.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e6179700 1 -- 172.21.15.32:0/3860860008 learned_addr learned my addr 172.21.15.32:0/3860860008 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:31.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e5978700 1 --2- 172.21.15.32:0/3860860008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e00cffb0 0x7ff0e00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:31.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e5177700 1 --2- 172.21.15.32:0/3860860008 >> v2:172.21.15.86:3300/0 conn(0x7ff0e00d7810 0x7ff0e00d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:31.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e4976700 1 -- 172.21.15.32:0/3860860008 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225338273 0 0) 0x7ff0d8000d30 con 0x7ff0e00d5c80 2021-04-09T18:13:31.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e4976700 1 -- 172.21.15.32:0/3860860008 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0cc002fe0 con 0x7ff0e00d5c80 2021-04-09T18:13:31.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e5978700 1 -- 172.21.15.32:0/3860860008 >> v2:172.21.15.86:3300/0 conn(0x7ff0e00d7810 msgr2=0x7ff0e00d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:31.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e5978700 1 --2- 172.21.15.32:0/3860860008 >> v2:172.21.15.86:3300/0 conn(0x7ff0e00d7810 0x7ff0e00d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e5978700 1 -- 172.21.15.32:0/3860860008 >> v1:172.21.15.32:6789/0 conn(0x7ff0e00d5c80 legacy=0x7ff0e00d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e5978700 1 -- 172.21.15.32:0/3860860008 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e00d8230 con 0x7ff0e00cffb0 2021-04-09T18:13:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e5177700 1 --2- 172.21.15.32:0/3860860008 >> v2:172.21.15.86:3300/0 conn(0x7ff0e00d7810 0x7ff0e00d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:31.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e5978700 1 --2- 172.21.15.32:0/3860860008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e00cffb0 0x7ff0e00d2450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff0dc006650 tx=0x7ff0dc009a80).ready entity=mon.2 client_cookie=48dadbc48dba65a1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:31.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e4976700 1 -- 172.21.15.32:0/3860860008 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0dc0009b0 con 0x7ff0e00cffb0 2021-04-09T18:13:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e4976700 1 -- 172.21.15.32:0/3860860008 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc00c0c0 con 0x7ff0e00cffb0 2021-04-09T18:13:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e4976700 1 -- 172.21.15.32:0/3860860008 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0dc00c510 con 0x7ff0e00cffb0 2021-04-09T18:13:31.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e7b5c700 1 -- 172.21.15.32:0/3860860008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e00cffb0 msgr2=0x7ff0e00d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:31.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.211 7ff0e7b5c700 1 --2- 172.21.15.32:0/3860860008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e00cffb0 0x7ff0e00d2450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff0dc006650 tx=0x7ff0dc009a80).stop 2021-04-09T18:13:31.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.212 7ff0e7b5c700 1 -- 172.21.15.32:0/3860860008 shutdown_connections 2021-04-09T18:13:31.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.212 7ff0e7b5c700 1 --2- 172.21.15.32:0/3860860008 >> v2:172.21.15.86:3300/0 conn(0x7ff0e00d7810 0x7ff0e00d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.212 7ff0e7b5c700 1 --2- 172.21.15.32:0/3860860008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e00cffb0 0x7ff0e00d2450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.212 7ff0e7b5c700 1 -- 172.21.15.32:0/3860860008 >> 172.21.15.32:0/3860860008 conn(0x7ff0e00ce5d0 msgr2=0x7ff0e00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:31.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.212 7ff0e7b5c700 1 -- 172.21.15.32:0/3860860008 shutdown_connections 2021-04-09T18:13:31.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.212 7ff0e7b5c700 1 -- 172.21.15.32:0/3860860008 wait complete. 2021-04-09T18:13:31.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.232 7ff0e7b5c700 1 Processor -- start 2021-04-09T18:13:31.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.232 7ff0e7b5c700 1 -- start start 2021-04-09T18:13:31.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e7b5c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e008a620 0x7ff0e008b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:31.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e7b5c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0e008b6d0 0x7ff0e0180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:31.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e7b5c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0e0180c50 0x7ff0e0181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:31.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e7b5c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e008e660 con 0x7ff0e0180c50 2021-04-09T18:13:31.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e7b5c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e008ab30 con 0x7ff0e008b6d0 2021-04-09T18:13:31.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e7b5c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e008bad0 con 0x7ff0e008a620 2021-04-09T18:13:31.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e6179700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0e0180c50 0x7ff0e0181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:31.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e6179700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0e0180c50 0x7ff0e0181050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42494/0 (socket says 172.21.15.32:42494) 2021-04-09T18:13:31.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e6179700 1 -- 172.21.15.32:0/786444180 learned_addr learned my addr 172.21.15.32:0/786444180 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:31.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e5177700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0e008b6d0 0x7ff0e0180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:31.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e5978700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e008a620 0x7ff0e008b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:31.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e6179700 1 -- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e008a620 msgr2=0x7ff0e008b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:31.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e6179700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e008a620 0x7ff0e008b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e6179700 1 -- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0e008b6d0 msgr2=0x7ff0e0180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:31.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e6179700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0e008b6d0 0x7ff0e0180630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e6179700 1 -- 172.21.15.32:0/786444180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e008e1f0 con 0x7ff0e0180c50 2021-04-09T18:13:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e5177700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0e008b6d0 0x7ff0e0180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:31.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.233 7ff0e6179700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0e0180c50 0x7ff0e0181050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7ff0d800b8c0 tx=0x7ff0d800bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.234 7ff0d67fc700 1 -- 172.21.15.32:0/786444180 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0d800bda0 con 0x7ff0e0180c50 2021-04-09T18:13:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.234 7ff0d67fc700 1 -- 172.21.15.32:0/786444180 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d8006030 con 0x7ff0e0180c50 2021-04-09T18:13:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.234 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e008e1f0 con 0x7ff0e0180c50 2021-04-09T18:13:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.234 7ff0d67fc700 1 -- 172.21.15.32:0/786444180 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0d80064c0 con 0x7ff0e0180c50 2021-04-09T18:13:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.234 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e008e1f0 con 0x7ff0e0180c50 2021-04-09T18:13:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.234 7ff0d67fc700 1 -- 172.21.15.32:0/786444180 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff0d8006210 con 0x7ff0e0180c50 2021-04-09T18:13:31.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.235 7ff0d67fc700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0c8018930 0x7ff0c801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:31.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.235 7ff0d67fc700 1 -- 172.21.15.32:0/786444180 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4757+0+0 (secure 0 0 0) 0x7ff0d8032a70 con 0x7ff0e0180c50 2021-04-09T18:13:31.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.235 7ff0e5978700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0c8018930 0x7ff0c801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:31.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.235 7ff0e5978700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0c8018930 0x7ff0c801ae10 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.254 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e01b7180 con 0x7ff0e0180c50 2021-04-09T18:13:31.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.258 7ff0d67fc700 1 -- 172.21.15.32:0/786444180 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff0d8000f90 con 0x7ff0e0180c50 2021-04-09T18:13:31.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.445 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7ff0e00d7290 con 0x7ff0e0180c50 2021-04-09T18:13:31.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.447 7ff0d67fc700 1 -- 172.21.15.32:0/786444180 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff0d8006030 con 0x7ff0e0180c50 2021-04-09T18:13:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:updated 2021-04-09T18:13:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0c8018930 msgr2=0x7ff0c801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0c8018930 0x7ff0c801ae10 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0e0180c50 msgr2=0x7ff0e0181050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0e0180c50 0x7ff0e0181050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7ff0d800b8c0 tx=0x7ff0d800bc60).stop 2021-04-09T18:13:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 shutdown_connections 2021-04-09T18:13:31.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0c8018930 0x7ff0c801ae10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0e0180c50 0x7ff0e0181050 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0e008b6d0 0x7ff0e0180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 --2- 172.21.15.32:0/786444180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0e008a620 0x7ff0e008b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 >> 172.21.15.32:0/786444180 conn(0x7ff0e0189800 msgr2=0x7ff0e01b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:31.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 shutdown_connections 2021-04-09T18:13:31.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.450 7ff0e7b5c700 1 -- 172.21.15.32:0/786444180 wait complete. 2021-04-09T18:13:31.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2021-04-09T18:13:32.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.990 7f6946e74700 1 Processor -- start 2021-04-09T18:13:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.990 7f6946e74700 1 -- start start 2021-04-09T18:13:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6946e74700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f69400cffa0 0x7f69400d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6946e74700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69400d7800 0x7f69400d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6946e74700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69400d8220 0x7f69400d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6946e74700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69400d91e0 con 0x7f69400cffa0 2021-04-09T18:13:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6946e74700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69400d7280 con 0x7f69400d8220 2021-04-09T18:13:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6946e74700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69400d6290 con 0x7f69400d7800 2021-04-09T18:13:32.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6944c90700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f69400cffa0 0x7f69400d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6944c90700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f69400cffa0 0x7f69400d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42500/0 (socket says 172.21.15.32:42500) 2021-04-09T18:13:32.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6944c90700 1 -- 172.21.15.32:0/3926589096 learned_addr learned my addr 172.21.15.32:0/3926589096 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:32.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6945491700 1 --2- 172.21.15.32:0/3926589096 >> v2:172.21.15.86:3300/0 conn(0x7f69400d8220 0x7f69400d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f693ffff700 1 --2- 172.21.15.32:0/3926589096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69400d7800 0x7f69400d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6944c90700 1 -- 172.21.15.32:0/3926589096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69400d7800 msgr2=0x7f69400d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:32.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6944c90700 1 --2- 172.21.15.32:0/3926589096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69400d7800 0x7f69400d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6944c90700 1 -- 172.21.15.32:0/3926589096 >> v2:172.21.15.86:3300/0 conn(0x7f69400d8220 msgr2=0x7f69400d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:32.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6944c90700 1 --2- 172.21.15.32:0/3926589096 >> v2:172.21.15.86:3300/0 conn(0x7f69400d8220 0x7f69400d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f6944c90700 1 -- 172.21.15.32:0/3926589096 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69400d64b0 con 0x7f69400cffa0 2021-04-09T18:13:32.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.991 7f693ffff700 1 --2- 172.21.15.32:0/3926589096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69400d7800 0x7f69400d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:32.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f6944c90700 1 --2- 172.21.15.32:0/3926589096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69400cffa0 0x7f69400d2440 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f6930006700 tx=0x7f6930009b80).ready entity=mon.0 client_cookie=a4eb0c76f35535c1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:32.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f693f7fe700 1 -- 172.21.15.32:0/3926589096 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f693000c0c0 con 0x7f69400cffa0 2021-04-09T18:13:32.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f693f7fe700 1 -- 172.21.15.32:0/3926589096 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693000c440 con 0x7f69400cffa0 2021-04-09T18:13:32.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f693f7fe700 1 -- 172.21.15.32:0/3926589096 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f693000c920 con 0x7f69400cffa0 2021-04-09T18:13:32.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f6946e74700 1 -- 172.21.15.32:0/3926589096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69400cffa0 msgr2=0x7f69400d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f6946e74700 1 --2- 172.21.15.32:0/3926589096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69400cffa0 0x7f69400d2440 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f6930006700 tx=0x7f6930009b80).stop 2021-04-09T18:13:32.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f6946e74700 1 -- 172.21.15.32:0/3926589096 shutdown_connections 2021-04-09T18:13:32.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f6946e74700 1 --2- 172.21.15.32:0/3926589096 >> v2:172.21.15.86:3300/0 conn(0x7f69400d8220 0x7f69400d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f6946e74700 1 --2- 172.21.15.32:0/3926589096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69400d7800 0x7f69400d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f6946e74700 1 --2- 172.21.15.32:0/3926589096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69400cffa0 0x7f69400d2440 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.992 7f6946e74700 1 -- 172.21.15.32:0/3926589096 >> 172.21.15.32:0/3926589096 conn(0x7f69400ce5c0 msgr2=0x7f69400cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:32.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.993 7f6946e74700 1 -- 172.21.15.32:0/3926589096 shutdown_connections 2021-04-09T18:13:32.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:31.993 7f6946e74700 1 -- 172.21.15.32:0/3926589096 wait complete. 2021-04-09T18:13:32.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f6946e74700 1 Processor -- start 2021-04-09T18:13:32.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f6946e74700 1 -- start start 2021-04-09T18:13:32.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f6946e74700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f694009cfd0 0x7f694009de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f6946e74700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f694009d3d0 0x7f694009e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f6946e74700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f694008ed20 0x7f694008f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f6946e74700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694009d910 con 0x7f694009d3d0 2021-04-09T18:13:32.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f6946e74700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694009ead0 con 0x7f694009cfd0 2021-04-09T18:13:32.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f6946e74700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69401a4100 con 0x7f694008ed20 2021-04-09T18:13:32.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f693ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f694009d3d0 0x7f694009e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.013 7f693ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f694009d3d0 0x7f694009e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42506/0 (socket says 172.21.15.32:42506) 2021-04-09T18:13:32.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693ffff700 1 -- 172.21.15.32:0/3744713900 learned_addr learned my addr 172.21.15.32:0/3744713900 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f6944c90700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f694009cfd0 0x7f694009de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f6945491700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f694008ed20 0x7f694008f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693ffff700 1 -- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f694008ed20 msgr2=0x7f694008f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693ffff700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f694008ed20 0x7f694008f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693ffff700 1 -- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f694009cfd0 msgr2=0x7f694009de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693ffff700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f694009cfd0 0x7f694009de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693ffff700 1 -- 172.21.15.32:0/3744713900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69401a3c90 con 0x7f694009d3d0 2021-04-09T18:13:32.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693ffff700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f694009d3d0 0x7f694009e4b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6934006f20 tx=0x7f693400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693d7fa700 1 -- 172.21.15.32:0/3744713900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f693400bd80 con 0x7f694009d3d0 2021-04-09T18:13:32.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693d7fa700 1 -- 172.21.15.32:0/3744713900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6934000c90 con 0x7f694009d3d0 2021-04-09T18:13:32.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f6946e74700 1 -- 172.21.15.32:0/3744713900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69401a3c90 con 0x7f694009d3d0 2021-04-09T18:13:32.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f693d7fa700 1 -- 172.21.15.32:0/3744713900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69340022b0 con 0x7f694009d3d0 2021-04-09T18:13:32.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.014 7f6946e74700 1 -- 172.21.15.32:0/3744713900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69401a3c90 con 0x7f694009d3d0 2021-04-09T18:13:32.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.015 7f693d7fa700 1 -- 172.21.15.32:0/3744713900 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f693400a000 con 0x7f694009d3d0 2021-04-09T18:13:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.015 7f693d7fa700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6920018930 0x7f692001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.015 7f693d7fa700 1 -- 172.21.15.32:0/3744713900 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f6934032700 con 0x7f694009d3d0 2021-04-09T18:13:32.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.015 7f6944c90700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6920018930 0x7f692001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.015 7f6944c90700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6920018930 0x7f692001adc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:32.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.035 7f6946e74700 1 -- 172.21.15.32:0/3744713900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69401b7170 con 0x7f694009d3d0 2021-04-09T18:13:32.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.039 7f693d7fa700 1 -- 172.21.15.32:0/3744713900 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6934006bb0 con 0x7f694009d3d0 2021-04-09T18:13:32.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.223 7f6946e74700 1 -- 172.21.15.32:0/3744713900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f69401b7170 con 0x7f694009d3d0 2021-04-09T18:13:32.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.226 7f693d7fa700 1 -- 172.21.15.32:0/3744713900 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7f693400a000 con 0x7f694009d3d0 2021-04-09T18:13:32.378 INFO:teuthology.orchestra.run.smithi032.stderr:updated 2021-04-09T18:13:32.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.228 7f6946e74700 1 -- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6920018930 msgr2=0x7f692001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.228 7f6946e74700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6920018930 0x7f692001adc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.228 7f6946e74700 1 -- 172.21.15.32:0/3744713900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f694009d3d0 msgr2=0x7f694009e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.228 7f6946e74700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f694009d3d0 0x7f694009e4b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6934006f20 tx=0x7f693400b990).stop 2021-04-09T18:13:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.228 7f6946e74700 1 -- 172.21.15.32:0/3744713900 shutdown_connections 2021-04-09T18:13:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.228 7f6946e74700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6920018930 0x7f692001adc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.229 7f6946e74700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f694009d3d0 0x7f694009e4b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.229 7f6946e74700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f694008ed20 0x7f694008f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.229 7f6946e74700 1 --2- 172.21.15.32:0/3744713900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f694009cfd0 0x7f694009de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.229 7f6946e74700 1 -- 172.21.15.32:0/3744713900 >> 172.21.15.32:0/3744713900 conn(0x7f694009cb30 msgr2=0x7f6940195de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:32.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.229 7f6946e74700 1 -- 172.21.15.32:0/3744713900 shutdown_connections 2021-04-09T18:13:32.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.229 7f6946e74700 1 -- 172.21.15.32:0/3744713900 wait complete. 2021-04-09T18:13:32.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2021-04-09T18:13:32.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.521 7fbe6cb2f700 1 Processor -- start 2021-04-09T18:13:32.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.521 7fbe6cb2f700 1 -- start start 2021-04-09T18:13:32.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.522 7fbe6cb2f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe6810e7e0 0x7fbe680d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.522 7fbe6cb2f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe680d7290 con 0x7fbe680d6c90 2021-04-09T18:13:32.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.522 7fbe6cb2f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe680d2450 con 0x7fbe680cffb0 2021-04-09T18:13:32.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.522 7fbe6cb2f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680d7090 con 0x7fbe6810e7e0 2021-04-09T18:13:32.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.522 7fbe6605d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbe680d6c90 0x7fbe6810cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52112/0 (socket says 172.21.15.32:52112) 2021-04-09T18:13:32.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.522 7fbe6605d700 1 -- 172.21.15.32:0/1646283338 learned_addr learned my addr 172.21.15.32:0/1646283338 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:32.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.522 7fbe6705f700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe6810e7e0 0x7fbe680d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.522 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674987178 0 0) 0x7fbe50000ce0 con 0x7fbe680d6c90 2021-04-09T18:13:32.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.522 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe54002fe0 con 0x7fbe680d6c90 2021-04-09T18:13:32.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358231827 0 0) 0x7fbe60001400 con 0x7fbe680cffb0 2021-04-09T18:13:32.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe54003210 con 0x7fbe680cffb0 2021-04-09T18:13:32.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4159303510 0 0) 0x7fbe500010e0 con 0x7fbe680d6c90 2021-04-09T18:13:32.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe6810e7e0 msgr2=0x7fbe680d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:32.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe6810e7e0 0x7fbe680d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 >> v1:172.21.15.86:6789/0 conn(0x7fbe680cffb0 legacy=0x7fbe680d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe681120d0 con 0x7fbe680d6c90 2021-04-09T18:13:32.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fbe50001600 con 0x7fbe680d6c90 2021-04-09T18:13:32.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 >> v1:172.21.15.32:6789/0 conn(0x7fbe680d6c90 legacy=0x7fbe6810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe54003b90 0x7fbe54005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe540067a0 0x7fbe54008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe54009510 0x7fbe5400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6685e700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe54003b90 0x7fbe54005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe540502f0 con 0x7fbe54003b90 2021-04-09T18:13:32.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe540504d0 con 0x7fbe54009510 2021-04-09T18:13:32.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe540506f0 con 0x7fbe540067a0 2021-04-09T18:13:32.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6605d700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe540067a0 0x7fbe54008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6705f700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe54009510 0x7fbe5400ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6685e700 1 -- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe540067a0 msgr2=0x7fbe54008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:32.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6685e700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe540067a0 0x7fbe54008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6685e700 1 -- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe54009510 msgr2=0x7fbe5400ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:32.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6685e700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe54009510 0x7fbe5400ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6685e700 1 -- 172.21.15.32:0/1646283338 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe54050990 con 0x7fbe54003b90 2021-04-09T18:13:32.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6685e700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe54003b90 0x7fbe54005fa0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbe60006f90 tx=0x7fbe60009de0).ready entity=mon.0 client_cookie=4cf33d4036a8ca07 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:32.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe600030c0 con 0x7fbe54003b90 2021-04-09T18:13:32.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6000b510 con 0x7fbe54003b90 2021-04-09T18:13:32.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6585c700 1 -- 172.21.15.32:0/1646283338 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe6000ba30 con 0x7fbe54003b90 2021-04-09T18:13:32.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6cb2f700 1 -- 172.21.15.32:0/1646283338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe54003b90 msgr2=0x7fbe54005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6cb2f700 1 --2- 172.21.15.32:0/1646283338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe54003b90 0x7fbe54005fa0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbe60006f90 tx=0x7fbe60009de0).stop 2021-04-09T18:13:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.523 7fbe6705f700 1 -- 172.21.15.32:0/1646283338 reap_dead start 2021-04-09T18:13:32.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.524 7fbe6cb2f700 1 -- 172.21.15.32:0/1646283338 shutdown_connections 2021-04-09T18:13:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.524 7fbe6cb2f700 1 -- 172.21.15.32:0/1646283338 >> 172.21.15.32:0/1646283338 conn(0x7fbe680ce5d0 msgr2=0x7fbe680cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.524 7fbe6cb2f700 1 -- 172.21.15.32:0/1646283338 shutdown_connections 2021-04-09T18:13:32.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.524 7fbe6cb2f700 1 -- 172.21.15.32:0/1646283338 wait complete. 2021-04-09T18:13:32.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.544 7fbe6cb2f700 1 Processor -- start 2021-04-09T18:13:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6cb2f700 1 -- start start 2021-04-09T18:13:32.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6cb2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe680a54b0 0x7fbe68197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6cb2f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe681a9ef0 0x7fbe681aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6cb2f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe681aa2f0 0x7fbe681ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6cb2f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681aa900 con 0x7fbe680a54b0 2021-04-09T18:13:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6cb2f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680a58b0 con 0x7fbe681a9ef0 2021-04-09T18:13:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6cb2f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681984f0 con 0x7fbe681aa2f0 2021-04-09T18:13:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe680a54b0 0x7fbe68197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe680a54b0 0x7fbe68197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42530/0 (socket says 172.21.15.32:42530) 2021-04-09T18:13:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6685e700 1 -- 172.21.15.32:0/977814332 learned_addr learned my addr 172.21.15.32:0/977814332 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6605d700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe681a9ef0 0x7fbe681aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.545 7fbe6705f700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe681aa2f0 0x7fbe681ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe6685e700 1 -- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe681aa2f0 msgr2=0x7fbe681ab4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe6685e700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe681aa2f0 0x7fbe681ab4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe6685e700 1 -- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe681a9ef0 msgr2=0x7fbe681aae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe6685e700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe681a9ef0 0x7fbe681aae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe6685e700 1 -- 172.21.15.32:0/977814332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe680a5ce0 con 0x7fbe680a54b0 2021-04-09T18:13:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe6685e700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe680a54b0 0x7fbe68197ed0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fbe60009990 tx=0x7fbe6000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe5f7fe700 1 -- 172.21.15.32:0/977814332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe60008030 con 0x7fbe680a54b0 2021-04-09T18:13:32.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe5f7fe700 1 -- 172.21.15.32:0/977814332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe60008330 con 0x7fbe680a54b0 2021-04-09T18:13:32.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe680a5ce0 con 0x7fbe680a54b0 2021-04-09T18:13:32.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe5f7fe700 1 -- 172.21.15.32:0/977814332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe60008790 con 0x7fbe680a54b0 2021-04-09T18:13:32.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe680a5ce0 con 0x7fbe680a54b0 2021-04-09T18:13:32.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.546 7fbe5f7fe700 1 -- 172.21.15.32:0/977814332 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbe60008510 con 0x7fbe680a54b0 2021-04-09T18:13:32.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.547 7fbe5f7fe700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe4c0188d0 0x7fbe4c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:32.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.547 7fbe5f7fe700 1 -- 172.21.15.32:0/977814332 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fbe60032a70 con 0x7fbe680a54b0 2021-04-09T18:13:32.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.547 7fbe6605d700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe4c0188d0 0x7fbe4c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:32.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.547 7fbe6605d700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe4c0188d0 0x7fbe4c01add0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:32.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.566 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe68182e20 con 0x7fbe680a54b0 2021-04-09T18:13:32.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.570 7fbe5f7fe700 1 -- 172.21.15.32:0/977814332 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbe6000be30 con 0x7fbe680a54b0 2021-04-09T18:13:32.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.756 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fbe68182e20 con 0x7fbe680a54b0 2021-04-09T18:13:32.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.759 7fbe5f7fe700 1 -- 172.21.15.32:0/977814332 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v12) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbe60008220 con 0x7fbe680a54b0 2021-04-09T18:13:32.809 INFO:teuthology.orchestra.run.smithi032.stderr:updated 2021-04-09T18:13:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe4c0188d0 msgr2=0x7fbe4c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe4c0188d0 0x7fbe4c01add0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe680a54b0 msgr2=0x7fbe68197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:32.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe680a54b0 0x7fbe68197ed0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fbe60009990 tx=0x7fbe6000bcf0).stop 2021-04-09T18:13:32.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 shutdown_connections 2021-04-09T18:13:32.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe4c0188d0 0x7fbe4c01add0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe681aa2f0 0x7fbe681ab4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe681a9ef0 0x7fbe681aae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 --2- 172.21.15.32:0/977814332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe680a54b0 0x7fbe68197ed0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:32.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 >> 172.21.15.32:0/977814332 conn(0x7fbe681838a0 msgr2=0x7fbe681a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:32.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 shutdown_connections 2021-04-09T18:13:32.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:32.765 7fbe6cb2f700 1 -- 172.21.15.32:0/977814332 wait complete. 2021-04-09T18:13:32.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2021-04-09T18:13:33.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.017 7f810b5e6700 1 Processor -- start 2021-04-09T18:13:33.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.017 7f810b5e6700 1 -- start start 2021-04-09T18:13:33.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.017 7f810b5e6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f810410e7e0 0x7f81040d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.017 7f810b5e6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81040d7290 con 0x7f81040d6c90 2021-04-09T18:13:33.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.017 7f810b5e6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81040d2450 con 0x7f81040cffb0 2021-04-09T18:13:33.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.017 7f810b5e6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81040d7090 con 0x7f810410e7e0 2021-04-09T18:13:33.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f8108c01700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f81040d6c90 0x7f810410cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52130/0 (socket says 172.21.15.32:52130) 2021-04-09T18:13:33.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f8108c01700 1 -- 172.21.15.32:0/608860695 learned_addr learned my addr 172.21.15.32:0/608860695 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:33.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f8109c03700 1 --2- 172.21.15.32:0/608860695 >> v2:172.21.15.86:3300/0 conn(0x7f810410e7e0 0x7f81040d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f80fbfff700 1 -- 172.21.15.32:0/608860695 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443468207 0 0) 0x7f80f4000ce0 con 0x7f81040d6c90 2021-04-09T18:13:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f80fbfff700 1 -- 172.21.15.32:0/608860695 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f0002fe0 con 0x7f81040d6c90 2021-04-09T18:13:33.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f80fbfff700 1 -- 172.21.15.32:0/608860695 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601746011 0 0) 0x7f8100000c60 con 0x7f81040cffb0 2021-04-09T18:13:33.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f80fbfff700 1 -- 172.21.15.32:0/608860695 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f0003210 con 0x7f81040cffb0 2021-04-09T18:13:33.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f80fbfff700 1 -- 172.21.15.32:0/608860695 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 330700627 0 0) 0x7f80f4000ed0 con 0x7f81040d6c90 2021-04-09T18:13:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f8109c03700 1 -- 172.21.15.32:0/608860695 >> v1:172.21.15.86:6789/0 conn(0x7f81040cffb0 legacy=0x7f81040d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f8109c03700 1 -- 172.21.15.32:0/608860695 >> v1:172.21.15.32:6789/0 conn(0x7f81040d6c90 legacy=0x7f810410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f8109c03700 1 -- 172.21.15.32:0/608860695 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81040d91f0 con 0x7f810410e7e0 2021-04-09T18:13:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.018 7f8109c03700 1 --2- 172.21.15.32:0/608860695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f810410e7e0 0x7f81040d8000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f80fc006e50 tx=0x7f80fc009e70).ready entity=mon.1 client_cookie=43bac582a7102b4b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:33.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.019 7f80fbfff700 1 -- 172.21.15.32:0/608860695 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80fc006a20 con 0x7f810410e7e0 2021-04-09T18:13:33.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.019 7f80fbfff700 1 -- 172.21.15.32:0/608860695 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc006790 con 0x7f810410e7e0 2021-04-09T18:13:33.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.019 7f80fbfff700 1 -- 172.21.15.32:0/608860695 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80fc006a20 con 0x7f810410e7e0 2021-04-09T18:13:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.019 7f810b5e6700 1 -- 172.21.15.32:0/608860695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f810410e7e0 msgr2=0x7f81040d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.019 7f810b5e6700 1 --2- 172.21.15.32:0/608860695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f810410e7e0 0x7f81040d8000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f80fc006e50 tx=0x7f80fc009e70).stop 2021-04-09T18:13:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.020 7f810b5e6700 1 -- 172.21.15.32:0/608860695 shutdown_connections 2021-04-09T18:13:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.020 7f810b5e6700 1 --2- 172.21.15.32:0/608860695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f810410e7e0 0x7f81040d8000 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.020 7f810b5e6700 1 -- 172.21.15.32:0/608860695 >> 172.21.15.32:0/608860695 conn(0x7f81040ce5d0 msgr2=0x7f81040cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.020 7f810b5e6700 1 -- 172.21.15.32:0/608860695 shutdown_connections 2021-04-09T18:13:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.020 7f810b5e6700 1 -- 172.21.15.32:0/608860695 wait complete. 2021-04-09T18:13:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.040 7f810b5e6700 1 Processor -- start 2021-04-09T18:13:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.040 7f810b5e6700 1 -- start start 2021-04-09T18:13:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f810b5e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8104184080 0x7f81041955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f810b5e6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8104195c00 0x7f810408ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f810b5e6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f810408f550 0x7f810408f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f810b5e6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8104090570 con 0x7f8104184080 2021-04-09T18:13:33.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f810b5e6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8104184480 con 0x7f8104195c00 2021-04-09T18:13:33.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f810b5e6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8104196000 con 0x7f810408f550 2021-04-09T18:13:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109402700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8104184080 0x7f81041955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109402700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8104184080 0x7f81041955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42544/0 (socket says 172.21.15.32:42544) 2021-04-09T18:13:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109402700 1 -- 172.21.15.32:0/131670029 learned_addr learned my addr 172.21.15.32:0/131670029 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8108c01700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8104195c00 0x7f810408ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109c03700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f810408f550 0x7f810408f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109402700 1 -- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f810408f550 msgr2=0x7f810408f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109402700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f810408f550 0x7f810408f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109402700 1 -- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8104195c00 msgr2=0x7f810408ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109402700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8104195c00 0x7f810408ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109402700 1 -- 172.21.15.32:0/131670029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81041847b0 con 0x7f8104184080 2021-04-09T18:13:33.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8108c01700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8104195c00 0x7f810408ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:33.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109c03700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f810408f550 0x7f810408f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:33.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f8109402700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8104184080 0x7f81041955e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f8100006f10 tx=0x7f810000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:33.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f80f9ffb700 1 -- 172.21.15.32:0/131670029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f810000bd80 con 0x7f8104184080 2021-04-09T18:13:33.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f80f9ffb700 1 -- 172.21.15.32:0/131670029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8100000e20 con 0x7f8104184080 2021-04-09T18:13:33.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.041 7f810b5e6700 1 -- 172.21.15.32:0/131670029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81041847b0 con 0x7f8104184080 2021-04-09T18:13:33.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.042 7f80f9ffb700 1 -- 172.21.15.32:0/131670029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81000054f0 con 0x7f8104184080 2021-04-09T18:13:33.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.042 7f810b5e6700 1 -- 172.21.15.32:0/131670029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81041847b0 con 0x7f8104184080 2021-04-09T18:13:33.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.042 7f80f9ffb700 1 -- 172.21.15.32:0/131670029 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8100009fe0 con 0x7f8104184080 2021-04-09T18:13:33.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.042 7f80f9ffb700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80e4018930 0x7f80e401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.042 7f80f9ffb700 1 -- 172.21.15.32:0/131670029 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f8100032710 con 0x7f8104184080 2021-04-09T18:13:33.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.042 7f8108c01700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80e4018930 0x7f80e401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.043 7f8108c01700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80e4018930 0x7f80e401adc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:33.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.062 7f810b5e6700 1 -- 172.21.15.32:0/131670029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8104182e80 con 0x7f8104184080 2021-04-09T18:13:33.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.065 7f80f9ffb700 1 -- 172.21.15.32:0/131670029 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8100000e20 con 0x7f8104184080 2021-04-09T18:13:33.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.251 7f810b5e6700 1 -- 172.21.15.32:0/131670029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f81040d7290 con 0x7f8104184080 2021-04-09T18:13:33.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.255 7f80f9ffb700 1 -- 172.21.15.32:0/131670029 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v13) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8100009fe0 con 0x7f8104184080 2021-04-09T18:13:33.321 INFO:teuthology.orchestra.run.smithi032.stderr:updated 2021-04-09T18:13:33.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.260 7f810b5e6700 1 -- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80e4018930 msgr2=0x7f80e401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.260 7f810b5e6700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80e4018930 0x7f80e401adc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.260 7f810b5e6700 1 -- 172.21.15.32:0/131670029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8104184080 msgr2=0x7f81041955e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.260 7f810b5e6700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8104184080 0x7f81041955e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f8100006f10 tx=0x7f810000b990).stop 2021-04-09T18:13:33.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.260 7f810b5e6700 1 -- 172.21.15.32:0/131670029 shutdown_connections 2021-04-09T18:13:33.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.261 7f810b5e6700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80e4018930 0x7f80e401adc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.261 7f810b5e6700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f810408f550 0x7f810408f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.261 7f810b5e6700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8104195c00 0x7f810408ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.261 7f810b5e6700 1 --2- 172.21.15.32:0/131670029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8104184080 0x7f81041955e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.261 7f810b5e6700 1 -- 172.21.15.32:0/131670029 >> 172.21.15.32:0/131670029 conn(0x7f81041a8f50 msgr2=0x7f8104188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:33.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.261 7f810b5e6700 1 -- 172.21.15.32:0/131670029 shutdown_connections 2021-04-09T18:13:33.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.261 7f810b5e6700 1 -- 172.21.15.32:0/131670029 wait complete. 2021-04-09T18:13:33.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:33.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.543 7f84532d9700 1 Processor -- start 2021-04-09T18:13:33.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.543 7f84532d9700 1 -- start start 2021-04-09T18:13:33.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.543 7f84532d9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0cffa0 0x7f844c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.543 7f84532d9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0d7280 con 0x7f844c0d6c80 2021-04-09T18:13:33.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.543 7f84532d9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c10e7d0 con 0x7f844c0d7800 2021-04-09T18:13:33.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.543 7f84532d9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0d7080 con 0x7f844c0cffa0 2021-04-09T18:13:33.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f84518f6700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f844c0d6c80 0x7f844c10cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52144/0 (socket says 172.21.15.32:52144) 2021-04-09T18:13:33.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f84518f6700 1 -- 172.21.15.32:0/1462276166 learned_addr learned my addr 172.21.15.32:0/1462276166 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:33.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f84510f5700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0cffa0 0x7f844c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047110625 0 0) 0x7f8444000d30 con 0x7f844c0d6c80 2021-04-09T18:13:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8438002fe0 con 0x7f844c0d6c80 2021-04-09T18:13:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729232182 0 0) 0x7f843c000c60 con 0x7f844c0d7800 2021-04-09T18:13:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8438003210 con 0x7f844c0d7800 2021-04-09T18:13:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 315029851 0 0) 0x7f8444001460 con 0x7f844c0d6c80 2021-04-09T18:13:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0cffa0 msgr2=0x7f844c0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:33.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0cffa0 0x7f844c0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 >> v1:172.21.15.86:6789/0 conn(0x7f844c0d7800 legacy=0x7f844c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c0d8580 con 0x7f844c0d6c80 2021-04-09T18:13:33.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f84440019e0 con 0x7f844c0d6c80 2021-04-09T18:13:33.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 >> v1:172.21.15.32:6789/0 conn(0x7f844c0d6c80 legacy=0x7f844c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8438003b90 0x7f8438005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8438006840 0x7f8438008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84380095b0 0x7f843800bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8438050390 con 0x7f84380095b0 2021-04-09T18:13:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8438050570 con 0x7f8438003b90 2021-04-09T18:13:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f8443fff700 1 -- 172.21.15.32:0/1462276166 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8438050790 con 0x7f8438006840 2021-04-09T18:13:33.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f84518f6700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84380095b0 0x7f843800bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f84510f5700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0cffa0 0x7f844c0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:33.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.544 7f84508f4700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8438006840 0x7f8438008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84510f5700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8438003b90 0x7f8438005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84518f6700 1 -- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8438006840 msgr2=0x7f8438008d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:33.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84518f6700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8438006840 0x7f8438008d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84518f6700 1 -- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8438003b90 msgr2=0x7f8438005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:33.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84518f6700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8438003b90 0x7f8438005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84518f6700 1 -- 172.21.15.32:0/1462276166 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8438050a30 con 0x7f84380095b0 2021-04-09T18:13:33.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84518f6700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84380095b0 0x7f843800bad0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f8444007cf0 tx=0x7f844400a310).ready entity=mon.0 client_cookie=5ee70358a57eeaac server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:33.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f8443fff700 1 -- 172.21.15.32:0/1462276166 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84440070c0 con 0x7f84380095b0 2021-04-09T18:13:33.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f8443fff700 1 -- 172.21.15.32:0/1462276166 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444007400 con 0x7f84380095b0 2021-04-09T18:13:33.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f8443fff700 1 -- 172.21.15.32:0/1462276166 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84440049a0 con 0x7f84380095b0 2021-04-09T18:13:33.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84532d9700 1 -- 172.21.15.32:0/1462276166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84380095b0 msgr2=0x7f843800bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84532d9700 1 --2- 172.21.15.32:0/1462276166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84380095b0 0x7f843800bad0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f8444007cf0 tx=0x7f844400a310).stop 2021-04-09T18:13:33.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.545 7f84518f6700 1 -- 172.21.15.32:0/1462276166 reap_dead start 2021-04-09T18:13:33.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.546 7f84532d9700 1 -- 172.21.15.32:0/1462276166 shutdown_connections 2021-04-09T18:13:33.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.546 7f84532d9700 1 -- 172.21.15.32:0/1462276166 >> 172.21.15.32:0/1462276166 conn(0x7f844c0ce5c0 msgr2=0x7f844c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:33.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.546 7f84532d9700 1 -- 172.21.15.32:0/1462276166 shutdown_connections 2021-04-09T18:13:33.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.546 7f84532d9700 1 -- 172.21.15.32:0/1462276166 wait complete. 2021-04-09T18:13:33.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.566 7f84532d9700 1 Processor -- start 2021-04-09T18:13:33.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.566 7f84532d9700 1 -- start start 2021-04-09T18:13:33.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84532d9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c08e030 0x7f844c08a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84532d9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c197ec0 0x7f844c08adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84532d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c183d00 0x7f844c184100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84532d9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c198410 con 0x7f844c183d00 2021-04-09T18:13:33.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84532d9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c08e430 con 0x7f844c08e030 2021-04-09T18:13:33.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84532d9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c08b410 con 0x7f844c197ec0 2021-04-09T18:13:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84518f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c183d00 0x7f844c184100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84518f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c183d00 0x7f844c184100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42566/0 (socket says 172.21.15.32:42566) 2021-04-09T18:13:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84518f6700 1 -- 172.21.15.32:0/605014716 learned_addr learned my addr 172.21.15.32:0/605014716 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84510f5700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c08e030 0x7f844c08a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84508f4700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c197ec0 0x7f844c08adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84518f6700 1 -- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c197ec0 msgr2=0x7f844c08adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84518f6700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c197ec0 0x7f844c08adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84518f6700 1 -- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c08e030 msgr2=0x7f844c08a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84518f6700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c08e030 0x7f844c08a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84518f6700 1 -- 172.21.15.32:0/605014716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c18e270 con 0x7f844c183d00 2021-04-09T18:13:33.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.567 7f84518f6700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c183d00 0x7f844c184100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f8444007620 tx=0x7f844400acd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:33.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.568 7f8441ffb700 1 -- 172.21.15.32:0/605014716 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8444007660 con 0x7f844c183d00 2021-04-09T18:13:33.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.568 7f8441ffb700 1 -- 172.21.15.32:0/605014716 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84440060a0 con 0x7f844c183d00 2021-04-09T18:13:33.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.568 7f84532d9700 1 -- 172.21.15.32:0/605014716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c18e270 con 0x7f844c183d00 2021-04-09T18:13:33.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.568 7f8441ffb700 1 -- 172.21.15.32:0/605014716 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84440064f0 con 0x7f844c183d00 2021-04-09T18:13:33.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.568 7f84532d9700 1 -- 172.21.15.32:0/605014716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c18e270 con 0x7f844c183d00 2021-04-09T18:13:33.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.568 7f8441ffb700 1 -- 172.21.15.32:0/605014716 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8444006280 con 0x7f844c183d00 2021-04-09T18:13:33.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.569 7f8441ffb700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:33.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.569 7f8441ffb700 1 -- 172.21.15.32:0/605014716 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f8444032a40 con 0x7f844c183d00 2021-04-09T18:13:33.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.569 7f84510f5700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:33.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.569 7f84510f5700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:33.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.587 7f84532d9700 1 -- 172.21.15.32:0/605014716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c182df0 con 0x7f844c183d00 2021-04-09T18:13:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.591 7f8441ffb700 1 -- 172.21.15.32:0/605014716 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8444000cc0 con 0x7f844c183d00 2021-04-09T18:13:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.778 7f84532d9700 1 -- 172.21.15.32:0/605014716 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f844c182df0 con 0x7f844c183d00 2021-04-09T18:13:33.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.779 7f8441ffb700 1 -- 172.21.15.32:0/605014716 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+11144 (secure 0 0 0) 0x7f8444005f90 con 0x7f844c183d00 2021-04-09T18:13:33.816 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:33.816 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:13:31.828231","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:33.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 -- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 msgr2=0x7f842c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 -- 172.21.15.32:0/605014716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c183d00 msgr2=0x7f844c184100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c183d00 0x7f844c184100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f8444007620 tx=0x7f844400acd0).stop 2021-04-09T18:13:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 -- 172.21.15.32:0/605014716 shutdown_connections 2021-04-09T18:13:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c183d00 0x7f844c184100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c197ec0 0x7f844c08adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 --2- 172.21.15.32:0/605014716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c08e030 0x7f844c08a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 -- 172.21.15.32:0/605014716 >> 172.21.15.32:0/605014716 conn(0x7f844c183870 msgr2=0x7f844c1a9650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 -- 172.21.15.32:0/605014716 shutdown_connections 2021-04-09T18:13:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:33.781 7f84532d9700 1 -- 172.21.15.32:0/605014716 wait complete. 2021-04-09T18:13:33.850 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:13:33.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:13:34.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.006 7f5f2aa30700 1 Processor -- start 2021-04-09T18:13:34.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.006 7f5f2aa30700 1 -- start start 2021-04-09T18:13:34.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.006 7f5f2aa30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2410d010 0x7f5f240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.006 7f5f2aa30700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2410d480 con 0x7f5f240d1da0 2021-04-09T18:13:34.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f2aa30700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2410e1f0 con 0x7f5f240d8490 2021-04-09T18:13:34.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f2aa30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2410e780 con 0x7f5f2410d010 2021-04-09T18:13:34.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f23fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5f240d1da0 0x7f5f240d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52162/0 (socket says 172.21.15.32:52162) 2021-04-09T18:13:34.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f23fff700 1 -- 172.21.15.32:0/246869287 learned_addr learned my addr 172.21.15.32:0/246869287 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:34.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f2884c700 1 --2- 172.21.15.32:0/246869287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2410d010 0x7f5f240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f237fe700 1 -- 172.21.15.32:0/246869287 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712666117 0 0) 0x7f5f14001060 con 0x7f5f240d1da0 2021-04-09T18:13:34.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f237fe700 1 -- 172.21.15.32:0/246869287 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f10002fe0 con 0x7f5f240d1da0 2021-04-09T18:13:34.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f237fe700 1 -- 172.21.15.32:0/246869287 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198140551 0 0) 0x7f5f1c0009b0 con 0x7f5f240d8490 2021-04-09T18:13:34.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f237fe700 1 -- 172.21.15.32:0/246869287 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f10003020 con 0x7f5f240d8490 2021-04-09T18:13:34.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f2884c700 1 -- 172.21.15.32:0/246869287 >> v1:172.21.15.86:6789/0 conn(0x7f5f240d8490 legacy=0x7f5f2410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f2884c700 1 -- 172.21.15.32:0/246869287 >> v1:172.21.15.32:6789/0 conn(0x7f5f240d1da0 legacy=0x7f5f240d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f2884c700 1 -- 172.21.15.32:0/246869287 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f240d8d30 con 0x7f5f2410d010 2021-04-09T18:13:34.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f2884c700 1 --2- 172.21.15.32:0/246869287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2410d010 0x7f5f240d1780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f5f0c006650 tx=0x7f5f0c009a80).ready entity=mon.2 client_cookie=29338f514b41a1d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:34.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f237fe700 1 -- 172.21.15.32:0/246869287 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f0c0009b0 con 0x7f5f2410d010 2021-04-09T18:13:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f237fe700 1 -- 172.21.15.32:0/246869287 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0c00c0c0 con 0x7f5f2410d010 2021-04-09T18:13:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.007 7f5f237fe700 1 -- 172.21.15.32:0/246869287 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f0c00c510 con 0x7f5f2410d010 2021-04-09T18:13:34.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.008 7f5f2aa30700 1 -- 172.21.15.32:0/246869287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2410d010 msgr2=0x7f5f240d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.008 7f5f2aa30700 1 --2- 172.21.15.32:0/246869287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2410d010 0x7f5f240d1780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f5f0c006650 tx=0x7f5f0c009a80).stop 2021-04-09T18:13:34.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.008 7f5f2aa30700 1 -- 172.21.15.32:0/246869287 shutdown_connections 2021-04-09T18:13:34.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.008 7f5f2aa30700 1 --2- 172.21.15.32:0/246869287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2410d010 0x7f5f240d1780 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.008 7f5f2aa30700 1 -- 172.21.15.32:0/246869287 >> 172.21.15.32:0/246869287 conn(0x7f5f240ce5f0 msgr2=0x7f5f240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:34.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.008 7f5f2aa30700 1 -- 172.21.15.32:0/246869287 shutdown_connections 2021-04-09T18:13:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.008 7f5f2aa30700 1 -- 172.21.15.32:0/246869287 wait complete. 2021-04-09T18:13:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.029 7f5f2aa30700 1 Processor -- start 2021-04-09T18:13:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.029 7f5f2aa30700 1 -- start start 2021-04-09T18:13:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.029 7f5f2aa30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24179610 0x7f5f24179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.029 7f5f2aa30700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2417a820 0x7f5f2417ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.029 7f5f2aa30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f241949f0 0x7f5f24194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.029 7f5f2aa30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f241b8d50 con 0x7f5f241949f0 2021-04-09T18:13:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.029 7f5f2aa30700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2417b240 con 0x7f5f2417a820 2021-04-09T18:13:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.029 7f5f2aa30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f24195410 con 0x7f5f24179610 2021-04-09T18:13:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2904d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f241949f0 0x7f5f24194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2904d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f241949f0 0x7f5f24194df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42580/0 (socket says 172.21.15.32:42580) 2021-04-09T18:13:34.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2904d700 1 -- 172.21.15.32:0/3409430747 learned_addr learned my addr 172.21.15.32:0/3409430747 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:34.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2884c700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24179610 0x7f5f24179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f23fff700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2417a820 0x7f5f2417ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2904d700 1 -- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24179610 msgr2=0x7f5f24179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2904d700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24179610 0x7f5f24179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2904d700 1 -- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2417a820 msgr2=0x7f5f2417ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2904d700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2417a820 0x7f5f2417ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2904d700 1 -- 172.21.15.32:0/3409430747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f24195630 con 0x7f5f241949f0 2021-04-09T18:13:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2904d700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f241949f0 0x7f5f24194df0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5f1c006f10 tx=0x7f5f1c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:34.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f217fa700 1 -- 172.21.15.32:0/3409430747 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f1c00bd80 con 0x7f5f241949f0 2021-04-09T18:13:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f217fa700 1 -- 172.21.15.32:0/3409430747 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c000e20 con 0x7f5f241949f0 2021-04-09T18:13:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f24195630 con 0x7f5f241949f0 2021-04-09T18:13:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f217fa700 1 -- 172.21.15.32:0/3409430747 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f1c0054f0 con 0x7f5f241949f0 2021-04-09T18:13:34.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.030 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f24195630 con 0x7f5f241949f0 2021-04-09T18:13:34.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.031 7f5f217fa700 1 -- 172.21.15.32:0/3409430747 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5f1c006240 con 0x7f5f241949f0 2021-04-09T18:13:34.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.031 7f5f217fa700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f08018930 0x7f5f0801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.031 7f5f217fa700 1 -- 172.21.15.32:0/3409430747 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f5f1c032750 con 0x7f5f241949f0 2021-04-09T18:13:34.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.031 7f5f2884c700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f08018930 0x7f5f0801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.032 7f5f2884c700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f08018930 0x7f5f0801ae10 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:34.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.051 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2417a2f0 con 0x7f5f241949f0 2021-04-09T18:13:34.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.054 7f5f217fa700 1 -- 172.21.15.32:0/3409430747 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5f1c006030 con 0x7f5f241949f0 2021-04-09T18:13:34.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.231 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f5f2417a2f0 con 0x7f5f241949f0 2021-04-09T18:13:34.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.231 7f5f217fa700 1 -- 172.21.15.32:0/3409430747 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5f1c006240 con 0x7f5f241949f0 2021-04-09T18:13:34.288 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:34.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f08018930 msgr2=0x7f5f0801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f08018930 0x7f5f0801ae10 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f241949f0 msgr2=0x7f5f24194df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f241949f0 0x7f5f24194df0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5f1c006f10 tx=0x7f5f1c00b990).stop 2021-04-09T18:13:34.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 shutdown_connections 2021-04-09T18:13:34.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f08018930 0x7f5f0801ae10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f241949f0 0x7f5f24194df0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2417a820 0x7f5f2417ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 --2- 172.21.15.32:0/3409430747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f24179610 0x7f5f24179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 >> 172.21.15.32:0/3409430747 conn(0x7f5f24092a00 msgr2=0x7f5f241b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:34.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 shutdown_connections 2021-04-09T18:13:34.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.234 7f5f2aa30700 1 -- 172.21.15.32:0/3409430747 wait complete. 2021-04-09T18:13:34.350 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:13:34.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:13:34.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:13:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.512 7f2e00981700 1 Processor -- start 2021-04-09T18:13:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.512 7f2e00981700 1 -- start start 2021-04-09T18:13:34.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2e00981700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2dfc10d010 0x7f2dfc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2e00981700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0d1da0 0x7f2dfc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2e00981700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0d82c0 0x7f2dfc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2e00981700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dfc0d8d30 con 0x7f2dfc10d010 2021-04-09T18:13:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2e00981700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dfc0d9220 con 0x7f2dfc0d1da0 2021-04-09T18:13:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2e00981700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc10d4b0 con 0x7f2dfc0d82c0 2021-04-09T18:13:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2dfa61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2dfc10d010 0x7f2dfc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2dfa61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2dfc10d010 0x7f2dfc0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42584/0 (socket says 172.21.15.32:42584) 2021-04-09T18:13:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2dfa61c700 1 -- 172.21.15.32:0/1069465801 learned_addr learned my addr 172.21.15.32:0/1069465801 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2df9e1b700 1 --2- 172.21.15.32:0/1069465801 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0d1da0 0x7f2dfc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2dfae1d700 1 --2- 172.21.15.32:0/1069465801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0d82c0 0x7f2dfc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2dfa61c700 1 -- 172.21.15.32:0/1069465801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0d82c0 msgr2=0x7f2dfc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2dfa61c700 1 --2- 172.21.15.32:0/1069465801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0d82c0 0x7f2dfc10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2dfa61c700 1 -- 172.21.15.32:0/1069465801 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0d1da0 msgr2=0x7f2dfc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.513 7f2dfa61c700 1 --2- 172.21.15.32:0/1069465801 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0d1da0 0x7f2dfc0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.514 7f2dfa61c700 1 -- 172.21.15.32:0/1069465801 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc0d86c0 con 0x7f2dfc10d010 2021-04-09T18:13:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.514 7f2df9e1b700 1 --2- 172.21.15.32:0/1069465801 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0d1da0 0x7f2dfc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.514 7f2dfa61c700 1 --2- 172.21.15.32:0/1069465801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc10d010 0x7f2dfc0d1780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f2dec006a80 tx=0x7f2dec009e80).ready entity=mon.0 client_cookie=8a69749679271086 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.514 7f2df961a700 1 -- 172.21.15.32:0/1069465801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dec00b510 con 0x7f2dfc10d010 2021-04-09T18:13:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.514 7f2df961a700 1 -- 172.21.15.32:0/1069465801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dec00b7d0 con 0x7f2dfc10d010 2021-04-09T18:13:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.514 7f2df961a700 1 -- 172.21.15.32:0/1069465801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dec00bcb0 con 0x7f2dfc10d010 2021-04-09T18:13:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.514 7f2e00981700 1 -- 172.21.15.32:0/1069465801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc10d010 msgr2=0x7f2dfc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.514 7f2e00981700 1 --2- 172.21.15.32:0/1069465801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc10d010 0x7f2dfc0d1780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f2dec006a80 tx=0x7f2dec009e80).stop 2021-04-09T18:13:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.515 7f2e00981700 1 -- 172.21.15.32:0/1069465801 shutdown_connections 2021-04-09T18:13:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.515 7f2e00981700 1 --2- 172.21.15.32:0/1069465801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0d82c0 0x7f2dfc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.515 7f2e00981700 1 --2- 172.21.15.32:0/1069465801 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0d1da0 0x7f2dfc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.515 7f2e00981700 1 --2- 172.21.15.32:0/1069465801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc10d010 0x7f2dfc0d1780 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.515 7f2e00981700 1 -- 172.21.15.32:0/1069465801 >> 172.21.15.32:0/1069465801 conn(0x7f2dfc0ce5f0 msgr2=0x7f2dfc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:34.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.515 7f2e00981700 1 -- 172.21.15.32:0/1069465801 shutdown_connections 2021-04-09T18:13:34.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.515 7f2e00981700 1 -- 172.21.15.32:0/1069465801 wait complete. 2021-04-09T18:13:34.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.535 7f2e00981700 1 Processor -- start 2021-04-09T18:13:34.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.535 7f2e00981700 1 -- start start 2021-04-09T18:13:34.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.535 7f2e00981700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc09af30 0x7f2dfc190ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.535 7f2e00981700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc09a410 0x7f2dfc091630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.535 7f2e00981700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc091c50 0x7f2dfc092770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.535 7f2e00981700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0921f0 con 0x7f2dfc091c50 2021-04-09T18:13:34.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2e00981700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc09b3b0 con 0x7f2dfc09af30 2021-04-09T18:13:34.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2e00981700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc09a810 con 0x7f2dfc09a410 2021-04-09T18:13:34.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc091c50 0x7f2dfc092770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc091c50 0x7f2dfc092770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42594/0 (socket says 172.21.15.32:42594) 2021-04-09T18:13:34.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfae1d700 1 -- 172.21.15.32:0/683554999 learned_addr learned my addr 172.21.15.32:0/683554999 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:34.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfa61c700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc09af30 0x7f2dfc190ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2df9e1b700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc09a410 0x7f2dfc091630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfae1d700 1 -- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc09a410 msgr2=0x7f2dfc091630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfae1d700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc09a410 0x7f2dfc091630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfae1d700 1 -- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc09af30 msgr2=0x7f2dfc190ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfae1d700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc09af30 0x7f2dfc190ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfae1d700 1 -- 172.21.15.32:0/683554999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc09aa30 con 0x7f2dfc091c50 2021-04-09T18:13:34.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2df9e1b700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc09a410 0x7f2dfc091630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:34.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfa61c700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc09af30 0x7f2dfc190ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2dfae1d700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc091c50 0x7f2dfc092770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2df4006b90 tx=0x7f2df400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2deaffd700 1 -- 172.21.15.32:0/683554999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df400bd80 con 0x7f2dfc091c50 2021-04-09T18:13:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2deaffd700 1 -- 172.21.15.32:0/683554999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df4002080 con 0x7f2dfc091c50 2021-04-09T18:13:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2e00981700 1 -- 172.21.15.32:0/683554999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc09aa30 con 0x7f2dfc091c50 2021-04-09T18:13:34.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2deaffd700 1 -- 172.21.15.32:0/683554999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df4002510 con 0x7f2dfc091c50 2021-04-09T18:13:34.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.536 7f2e00981700 1 -- 172.21.15.32:0/683554999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc09aa30 con 0x7f2dfc091c50 2021-04-09T18:13:34.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.537 7f2deaffd700 1 -- 172.21.15.32:0/683554999 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2df4002260 con 0x7f2dfc091c50 2021-04-09T18:13:34.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.537 7f2deaffd700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de00188c0 0x7f2de001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.537 7f2deaffd700 1 -- 172.21.15.32:0/683554999 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f2df4032710 con 0x7f2dfc091c50 2021-04-09T18:13:34.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.537 7f2dfa61c700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de00188c0 0x7f2de001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.538 7f2dfa61c700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de00188c0 0x7f2de001adc0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:34.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.557 7f2e00981700 1 -- 172.21.15.32:0/683554999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc182d10 con 0x7f2dfc091c50 2021-04-09T18:13:34.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.560 7f2deaffd700 1 -- 172.21.15.32:0/683554999 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2df401f780 con 0x7f2dfc091c50 2021-04-09T18:13:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.737 7f2e00981700 1 -- 172.21.15.32:0/683554999 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2dfc10d480 con 0x7f2dfc091c50 2021-04-09T18:13:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.738 7f2deaffd700 1 -- 172.21.15.32:0/683554999 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v28) v1 ==== 84+0+64 (secure 0 0 0) 0x7f2df401f780 con 0x7f2dfc091c50 2021-04-09T18:13:34.760 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:34.760 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:13:34.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:13:34.761 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:13:34.761 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:13:34.761 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:34.761 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:13:34.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 -- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de00188c0 msgr2=0x7f2de001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de00188c0 0x7f2de001adc0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 -- 172.21.15.32:0/683554999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc091c50 msgr2=0x7f2dfc092770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc091c50 0x7f2dfc092770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2df4006b90 tx=0x7f2df400b990).stop 2021-04-09T18:13:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 -- 172.21.15.32:0/683554999 shutdown_connections 2021-04-09T18:13:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de00188c0 0x7f2de001adc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc091c50 0x7f2dfc092770 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc09a410 0x7f2dfc091630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 --2- 172.21.15.32:0/683554999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc09af30 0x7f2dfc190ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 -- 172.21.15.32:0/683554999 >> 172.21.15.32:0/683554999 conn(0x7f2dfc19b890 msgr2=0x7f2dfc08c660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:34.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 -- 172.21.15.32:0/683554999 shutdown_connections 2021-04-09T18:13:34.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.740 7f2e00981700 1 -- 172.21.15.32:0/683554999 wait complete. 2021-04-09T18:13:34.771 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:13:34.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:13:34.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.907 7f56349ff700 1 Processor -- start 2021-04-09T18:13:34.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.907 7f56349ff700 1 -- start start 2021-04-09T18:13:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f56349ff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f563010d010 0x7f56300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f56349ff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56300d1da0 0x7f56300d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:34.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f56349ff700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563010d480 con 0x7f56300d82c0 2021-04-09T18:13:34.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f56349ff700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56300d8d30 con 0x7f56300d1da0 2021-04-09T18:13:34.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f56349ff700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56300d9220 con 0x7f563010d010 2021-04-09T18:13:34.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562de1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56300d1da0 0x7f56300d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562ee1d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f56300d82c0 0x7f563010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52190/0 (socket says 172.21.15.32:52190) 2021-04-09T18:13:34.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562ee1d700 1 -- 172.21.15.32:0/2643254004 learned_addr learned my addr 172.21.15.32:0/2643254004 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:34.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562e61c700 1 --2- 172.21.15.32:0/2643254004 >> v2:172.21.15.86:3300/0 conn(0x7f563010d010 0x7f56300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:34.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562d61a700 1 -- 172.21.15.32:0/2643254004 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960513332 0 0) 0x7f5628000d30 con 0x7f56300d82c0 2021-04-09T18:13:34.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562d61a700 1 -- 172.21.15.32:0/2643254004 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5618002fe0 con 0x7f56300d82c0 2021-04-09T18:13:34.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562de1b700 1 -- 172.21.15.32:0/2643254004 >> v2:172.21.15.86:3300/0 conn(0x7f563010d010 msgr2=0x7f56300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:34.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562de1b700 1 --2- 172.21.15.32:0/2643254004 >> v2:172.21.15.86:3300/0 conn(0x7f563010d010 0x7f56300d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:34.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562de1b700 1 -- 172.21.15.32:0/2643254004 >> v1:172.21.15.32:6789/0 conn(0x7f56300d82c0 legacy=0x7f563010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562de1b700 1 -- 172.21.15.32:0/2643254004 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56300d86f0 con 0x7f56300d1da0 2021-04-09T18:13:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.908 7f562de1b700 1 --2- 172.21.15.32:0/2643254004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56300d1da0 0x7f56300d7ca0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5624006650 tx=0x7f5624009a80).ready entity=mon.0 client_cookie=cd4ec4eaa78c4e5e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:35.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f562d61a700 1 -- 172.21.15.32:0/2643254004 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56240009b0 con 0x7f56300d1da0 2021-04-09T18:13:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f562d61a700 1 -- 172.21.15.32:0/2643254004 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562400c0c0 con 0x7f56300d1da0 2021-04-09T18:13:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f562d61a700 1 -- 172.21.15.32:0/2643254004 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f562400c510 con 0x7f56300d1da0 2021-04-09T18:13:35.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f56349ff700 1 -- 172.21.15.32:0/2643254004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56300d1da0 msgr2=0x7f56300d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f56349ff700 1 --2- 172.21.15.32:0/2643254004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56300d1da0 0x7f56300d7ca0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5624006650 tx=0x7f5624009a80).stop 2021-04-09T18:13:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f56349ff700 1 -- 172.21.15.32:0/2643254004 shutdown_connections 2021-04-09T18:13:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f56349ff700 1 --2- 172.21.15.32:0/2643254004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56300d1da0 0x7f56300d7ca0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f56349ff700 1 --2- 172.21.15.32:0/2643254004 >> v2:172.21.15.86:3300/0 conn(0x7f563010d010 0x7f56300d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f56349ff700 1 -- 172.21.15.32:0/2643254004 >> 172.21.15.32:0/2643254004 conn(0x7f56300ce5f0 msgr2=0x7f56300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f56349ff700 1 -- 172.21.15.32:0/2643254004 shutdown_connections 2021-04-09T18:13:35.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.909 7f56349ff700 1 -- 172.21.15.32:0/2643254004 wait complete. 2021-04-09T18:13:35.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.929 7f56349ff700 1 Processor -- start 2021-04-09T18:13:35.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f56349ff700 1 -- start start 2021-04-09T18:13:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f56349ff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56300c3c00 0x7f563009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f56349ff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5630095240 0x7f56301a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f56349ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56301a1270 0x7f56300b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f56349ff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563009cda0 con 0x7f56301a1270 2021-04-09T18:13:35.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f56349ff700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56300c3680 con 0x7f5630095240 2021-04-09T18:13:35.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f56349ff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5630095640 con 0x7f56300c3c00 2021-04-09T18:13:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f562ee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56301a1270 0x7f56300b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f562ee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56301a1270 0x7f56300b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42604/0 (socket says 172.21.15.32:42604) 2021-04-09T18:13:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f562ee1d700 1 -- 172.21.15.32:0/528380828 learned_addr learned my addr 172.21.15.32:0/528380828 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:35.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f562de1b700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5630095240 0x7f56301a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f562e61c700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56300c3c00 0x7f563009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.930 7f562ee1d700 1 -- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56300c3c00 msgr2=0x7f563009c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f562ee1d700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56300c3c00 0x7f563009c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f562ee1d700 1 -- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5630095240 msgr2=0x7f56301a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f562ee1d700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5630095240 0x7f56301a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f562ee1d700 1 -- 172.21.15.32:0/528380828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56301a1670 con 0x7f56301a1270 2021-04-09T18:13:35.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f562e61c700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56300c3c00 0x7f563009c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:35.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f562de1b700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5630095240 0x7f56301a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:35.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f562ee1d700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56301a1270 0x7f56300b0cf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f5628009980 tx=0x7f562800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:35.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f561effd700 1 -- 172.21.15.32:0/528380828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f562800be10 con 0x7f56301a1270 2021-04-09T18:13:35.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f561effd700 1 -- 172.21.15.32:0/528380828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5628006080 con 0x7f56301a1270 2021-04-09T18:13:35.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f56349ff700 1 -- 172.21.15.32:0/528380828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56301a1670 con 0x7f56301a1270 2021-04-09T18:13:35.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f561effd700 1 -- 172.21.15.32:0/528380828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5628004720 con 0x7f56301a1270 2021-04-09T18:13:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.931 7f56349ff700 1 -- 172.21.15.32:0/528380828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56301a1670 con 0x7f56301a1270 2021-04-09T18:13:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.932 7f561effd700 1 -- 172.21.15.32:0/528380828 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5628006260 con 0x7f56301a1270 2021-04-09T18:13:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.932 7f561effd700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5614018900 0x7f561401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.932 7f561effd700 1 -- 172.21.15.32:0/528380828 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f5628032a20 con 0x7f56301a1270 2021-04-09T18:13:35.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.932 7f562e61c700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5614018900 0x7f561401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.932 7f562e61c700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5614018900 0x7f561401adc0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:35.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.952 7f56349ff700 1 -- 172.21.15.32:0/528380828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5630093d50 con 0x7f56301a1270 2021-04-09T18:13:35.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:34.955 7f561effd700 1 -- 172.21.15.32:0/528380828 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5628000f90 con 0x7f56301a1270 2021-04-09T18:13:35.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.139 7f56349ff700 1 -- 172.21.15.32:0/528380828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f563010d480 con 0x7f56301a1270 2021-04-09T18:13:35.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.454 7f561effd700 1 -- 172.21.15.32:0/528380828 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v29) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5628006000 con 0x7f56301a1270 2021-04-09T18:13:35.524 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:13:35.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.505 7f56349ff700 1 -- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5614018900 msgr2=0x7f561401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.505 7f56349ff700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5614018900 0x7f561401adc0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.505 7f56349ff700 1 -- 172.21.15.32:0/528380828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56301a1270 msgr2=0x7f56300b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.505 7f56349ff700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56301a1270 0x7f56300b0cf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f5628009980 tx=0x7f562800bcd0).stop 2021-04-09T18:13:35.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.506 7f56349ff700 1 -- 172.21.15.32:0/528380828 shutdown_connections 2021-04-09T18:13:35.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.506 7f56349ff700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5614018900 0x7f561401adc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.506 7f56349ff700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56301a1270 0x7f56300b0cf0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.506 7f56349ff700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5630095240 0x7f56301a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.506 7f56349ff700 1 --2- 172.21.15.32:0/528380828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56300c3c00 0x7f563009c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.506 7f56349ff700 1 -- 172.21.15.32:0/528380828 >> 172.21.15.32:0/528380828 conn(0x7f56300975f0 msgr2=0x7f5630096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:35.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.506 7f56349ff700 1 -- 172.21.15.32:0/528380828 shutdown_connections 2021-04-09T18:13:35.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.506 7f56349ff700 1 -- 172.21.15.32:0/528380828 wait complete. 2021-04-09T18:13:35.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:13:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.673 7fe7559be700 1 Processor -- start 2021-04-09T18:13:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.673 7fe7559be700 1 -- start start 2021-04-09T18:13:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.673 7fe7559be700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7500d7ca0 0x7fe7500d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe7559be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe75010da00 0x7fe75010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe7559be700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75010d480 con 0x7fe75010d010 2021-04-09T18:13:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe7559be700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7500d9220 con 0x7fe7500d7ca0 2021-04-09T18:13:35.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe7559be700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500d86c0 con 0x7fe75010da00 2021-04-09T18:13:35.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74ed73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7500d7ca0 0x7fe7500d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74ed73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7500d7ca0 0x7fe7500d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42612/0 (socket says 172.21.15.32:42612) 2021-04-09T18:13:35.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74ed73700 1 -- 172.21.15.32:0/863745914 learned_addr learned my addr 172.21.15.32:0/863745914 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:35.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74fd75700 1 --2- 172.21.15.32:0/863745914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe75010da00 0x7fe75010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74ed73700 1 -- 172.21.15.32:0/863745914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe75010da00 msgr2=0x7fe75010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:35.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74ed73700 1 --2- 172.21.15.32:0/863745914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe75010da00 0x7fe75010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74ed73700 1 -- 172.21.15.32:0/863745914 >> v1:172.21.15.86:6789/0 conn(0x7fe75010d010 legacy=0x7fe7500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74ed73700 1 -- 172.21.15.32:0/863745914 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500d1f70 con 0x7fe7500d7ca0 2021-04-09T18:13:35.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74ed73700 1 --2- 172.21.15.32:0/863745914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7500d7ca0 0x7fe7500d80a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fe73c006700 tx=0x7fe73c009b00).ready entity=mon.0 client_cookie=8d23b9a514557503 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:35.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74e572700 1 -- 172.21.15.32:0/863745914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe73c00c0c0 con 0x7fe7500d7ca0 2021-04-09T18:13:35.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74e572700 1 -- 172.21.15.32:0/863745914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c00c400 con 0x7fe7500d7ca0 2021-04-09T18:13:35.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe74e572700 1 -- 172.21.15.32:0/863745914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe73c00c8e0 con 0x7fe7500d7ca0 2021-04-09T18:13:35.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe7559be700 1 -- 172.21.15.32:0/863745914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7500d7ca0 msgr2=0x7fe7500d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.674 7fe7559be700 1 --2- 172.21.15.32:0/863745914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7500d7ca0 0x7fe7500d80a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fe73c006700 tx=0x7fe73c009b00).stop 2021-04-09T18:13:35.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.675 7fe7559be700 1 -- 172.21.15.32:0/863745914 shutdown_connections 2021-04-09T18:13:35.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.675 7fe7559be700 1 --2- 172.21.15.32:0/863745914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe75010da00 0x7fe75010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.675 7fe7559be700 1 --2- 172.21.15.32:0/863745914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7500d7ca0 0x7fe7500d80a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.675 7fe7559be700 1 -- 172.21.15.32:0/863745914 >> 172.21.15.32:0/863745914 conn(0x7fe7500ce5f0 msgr2=0x7fe7500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:35.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.675 7fe7559be700 1 -- 172.21.15.32:0/863745914 shutdown_connections 2021-04-09T18:13:35.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.675 7fe7559be700 1 -- 172.21.15.32:0/863745914 wait complete. 2021-04-09T18:13:35.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.695 7fe7559be700 1 Processor -- start 2021-04-09T18:13:35.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.695 7fe7559be700 1 -- start start 2021-04-09T18:13:35.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe7559be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe75017bea0 0x7fe750185510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe7559be700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe750181970 0x7fe750181d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe7559be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe750182a00 0x7fe750182e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe7559be700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75017c2a0 con 0x7fe75017bea0 2021-04-09T18:13:35.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe7559be700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe750183420 con 0x7fe750181970 2021-04-09T18:13:35.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe7559be700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75008a620 con 0x7fe750182a00 2021-04-09T18:13:35.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74f574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe75017bea0 0x7fe750185510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74f574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe75017bea0 0x7fe750185510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42618/0 (socket says 172.21.15.32:42618) 2021-04-09T18:13:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74f574700 1 -- 172.21.15.32:0/4004169772 learned_addr learned my addr 172.21.15.32:0/4004169772 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74ed73700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe750181970 0x7fe750181d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74fd75700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe750182a00 0x7fe750182e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74f574700 1 -- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe750182a00 msgr2=0x7fe750182e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74f574700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe750182a00 0x7fe750182e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74f574700 1 -- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe750181970 msgr2=0x7fe750181d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74f574700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe750181970 0x7fe750181d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74f574700 1 -- 172.21.15.32:0/4004169772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75008a840 con 0x7fe75017bea0 2021-04-09T18:13:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74fd75700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe750182a00 0x7fe750182e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe74f574700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe75017bea0 0x7fe750185510 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fe74000b5b0 tx=0x7fe74000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.696 7fe747fff700 1 -- 172.21.15.32:0/4004169772 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe74000bd40 con 0x7fe75017bea0 2021-04-09T18:13:35.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.697 7fe747fff700 1 -- 172.21.15.32:0/4004169772 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe740006240 con 0x7fe75017bea0 2021-04-09T18:13:35.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.697 7fe7559be700 1 -- 172.21.15.32:0/4004169772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75008a840 con 0x7fe75017bea0 2021-04-09T18:13:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.697 7fe747fff700 1 -- 172.21.15.32:0/4004169772 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe740005690 con 0x7fe75017bea0 2021-04-09T18:13:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.697 7fe7559be700 1 -- 172.21.15.32:0/4004169772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75008a840 con 0x7fe75017bea0 2021-04-09T18:13:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.697 7fe747fff700 1 -- 172.21.15.32:0/4004169772 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe7400054a0 con 0x7fe75017bea0 2021-04-09T18:13:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.697 7fe747fff700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe72c0188c0 0x7fe72c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.697 7fe747fff700 1 -- 172.21.15.32:0/4004169772 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5111+0+0 (secure 0 0 0) 0x7fe740032860 con 0x7fe75017bea0 2021-04-09T18:13:35.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.698 7fe74ed73700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe72c0188c0 0x7fe72c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.698 7fe74ed73700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe72c0188c0 0x7fe72c01adc0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.718 7fe7559be700 1 -- 172.21.15.32:0/4004169772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75017c2a0 con 0x7fe75017bea0 2021-04-09T18:13:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.721 7fe747fff700 1 -- 172.21.15.32:0/4004169772 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe740000c60 con 0x7fe75017bea0 2021-04-09T18:13:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:35.903 7fe7559be700 1 -- 172.21.15.32:0/4004169772 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fe75017c2a0 con 0x7fe75017bea0 2021-04-09T18:13:36.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.456 7fe747fff700 1 -- 172.21.15.32:0/4004169772 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v30) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe7400054a0 con 0x7fe75017bea0 2021-04-09T18:13:36.489 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:13:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 -- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe72c0188c0 msgr2=0x7fe72c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe72c0188c0 0x7fe72c01adc0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 -- 172.21.15.32:0/4004169772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe75017bea0 msgr2=0x7fe750185510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:36.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe75017bea0 0x7fe750185510 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fe74000b5b0 tx=0x7fe74000b950).stop 2021-04-09T18:13:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 -- 172.21.15.32:0/4004169772 shutdown_connections 2021-04-09T18:13:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe72c0188c0 0x7fe72c01adc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe750182a00 0x7fe750182e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe750181970 0x7fe750181d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 --2- 172.21.15.32:0/4004169772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe75017bea0 0x7fe750185510 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 -- 172.21.15.32:0/4004169772 >> 172.21.15.32:0/4004169772 conn(0x7fe750181390 msgr2=0x7fe7500adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:36.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 -- 172.21.15.32:0/4004169772 shutdown_connections 2021-04-09T18:13:36.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.470 7fe7559be700 1 -- 172.21.15.32:0/4004169772 wait complete. 2021-04-09T18:13:36.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:13:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.656 7fee9bb7f700 1 Processor -- start 2021-04-09T18:13:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.656 7fee9bb7f700 1 -- start start 2021-04-09T18:13:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.656 7fee9bb7f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9410d020 0x7fee940d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.656 7fee9bb7f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee940d1db0 0x7fee940d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.656 7fee9bb7f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9410d490 con 0x7fee940d82d0 2021-04-09T18:13:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.656 7fee9bb7f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9410e790 con 0x7fee940d1db0 2021-04-09T18:13:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.656 7fee9bb7f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee940d8d40 con 0x7fee9410d020 2021-04-09T18:13:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee9a19c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fee940d82d0 0x7fee9410da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52218/0 (socket says 172.21.15.32:52218) 2021-04-09T18:13:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee9a19c700 1 -- 172.21.15.32:0/1157539863 learned_addr learned my addr 172.21.15.32:0/1157539863 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee9919a700 1 --2- 172.21.15.32:0/1157539863 >> v2:172.21.15.86:3300/0 conn(0x7fee940d1db0 0x7fee940d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee9999b700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9410d020 0x7fee940d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094937411 0 0) 0x7fee8c000d30 con 0x7fee940d82d0 2021-04-09T18:13:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee80002fe0 con 0x7fee940d82d0 2021-04-09T18:13:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1825421683 0 0) 0x7fee8c001270 con 0x7fee940d82d0 2021-04-09T18:13:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9410d020 msgr2=0x7fee940d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9410d020 0x7fee940d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 >> v2:172.21.15.86:3300/0 conn(0x7fee940d1db0 msgr2=0x7fee940d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 --2- 172.21.15.32:0/1157539863 >> v2:172.21.15.86:3300/0 conn(0x7fee940d1db0 0x7fee940d7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee940d9230 con 0x7fee940d82d0 2021-04-09T18:13:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fee8c001980 con 0x7fee940d82d0 2021-04-09T18:13:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 >> v1:172.21.15.32:6789/0 conn(0x7fee940d82d0 legacy=0x7fee9410da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee80003a40 0x7fee80005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:36.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee8004a7a0 0x7fee8004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:36.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee8004d510 0x7fee8004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:36.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee800502e0 con 0x7fee8004d510 2021-04-09T18:13:36.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee80050500 con 0x7fee80003a40 2021-04-09T18:13:36.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.657 7fee98999700 1 -- 172.21.15.32:0/1157539863 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee80050720 con 0x7fee8004a7a0 2021-04-09T18:13:36.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9a19c700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee8004d510 0x7fee8004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:36.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9999b700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee80003a40 0x7fee80005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:36.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9919a700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee8004a7a0 0x7fee8004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:36.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9a19c700 1 -- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee8004a7a0 msgr2=0x7fee8004cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:36.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9a19c700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee8004a7a0 0x7fee8004cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9a19c700 1 -- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee80003a40 msgr2=0x7fee80005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:36.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9a19c700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee80003a40 0x7fee80005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9a19c700 1 -- 172.21.15.32:0/1157539863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee800509c0 con 0x7fee8004d510 2021-04-09T18:13:36.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9a19c700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee8004d510 0x7fee8004fa30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fee8c007a10 tx=0x7fee8c00a310).ready entity=mon.0 client_cookie=84a5230958ac0f34 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee98999700 1 -- 172.21.15.32:0/1157539863 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee8c0070c0 con 0x7fee8004d510 2021-04-09T18:13:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee98999700 1 -- 172.21.15.32:0/1157539863 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8c001de0 con 0x7fee8004d510 2021-04-09T18:13:36.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee98999700 1 -- 172.21.15.32:0/1157539863 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee8c004720 con 0x7fee8004d510 2021-04-09T18:13:36.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9bb7f700 1 -- 172.21.15.32:0/1157539863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee8004d510 msgr2=0x7fee8004fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:36.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9bb7f700 1 --2- 172.21.15.32:0/1157539863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee8004d510 0x7fee8004fa30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fee8c007a10 tx=0x7fee8c00a310).stop 2021-04-09T18:13:36.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.658 7fee9a19c700 1 -- 172.21.15.32:0/1157539863 reap_dead start 2021-04-09T18:13:36.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.659 7fee9bb7f700 1 -- 172.21.15.32:0/1157539863 shutdown_connections 2021-04-09T18:13:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.659 7fee9bb7f700 1 -- 172.21.15.32:0/1157539863 >> 172.21.15.32:0/1157539863 conn(0x7fee940ce600 msgr2=0x7fee940d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:36.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.659 7fee9bb7f700 1 -- 172.21.15.32:0/1157539863 shutdown_connections 2021-04-09T18:13:36.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.659 7fee9bb7f700 1 -- 172.21.15.32:0/1157539863 wait complete. 2021-04-09T18:13:36.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.679 7fee9bb7f700 1 Processor -- start 2021-04-09T18:13:36.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.679 7fee9bb7f700 1 -- start start 2021-04-09T18:13:36.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9bb7f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee940ab560 0x7fee940ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:36.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9bb7f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee940a07d0 0x7fee9419ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:36.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9bb7f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee940a0bd0 0x7fee940b9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:36.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9bb7f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee940aafe0 con 0x7fee940a07d0 2021-04-09T18:13:36.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9bb7f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee940bce20 con 0x7fee940a0bd0 2021-04-09T18:13:36.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9bb7f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee941826a0 con 0x7fee940ab560 2021-04-09T18:13:36.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9919a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee940a07d0 0x7fee9419ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:36.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9919a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee940a07d0 0x7fee9419ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42640/0 (socket says 172.21.15.32:42640) 2021-04-09T18:13:36.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9919a700 1 -- 172.21.15.32:0/671819723 learned_addr learned my addr 172.21.15.32:0/671819723 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9a19c700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee940a0bd0 0x7fee940b9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9999b700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee940ab560 0x7fee940ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9919a700 1 -- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee940ab560 msgr2=0x7fee940ada20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9919a700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee940ab560 0x7fee940ada20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9919a700 1 -- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee940a0bd0 msgr2=0x7fee940b9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:36.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9919a700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee940a0bd0 0x7fee940b9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:36.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9919a700 1 -- 172.21.15.32:0/671819723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee941a0570 con 0x7fee940a07d0 2021-04-09T18:13:36.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9999b700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee940ab560 0x7fee940ada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:36.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9919a700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee940a07d0 0x7fee9419ff50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fee88006f60 tx=0x7fee8800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee867fc700 1 -- 172.21.15.32:0/671819723 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee8800bd60 con 0x7fee940a07d0 2021-04-09T18:13:36.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee867fc700 1 -- 172.21.15.32:0/671819723 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee88000c90 con 0x7fee940a07d0 2021-04-09T18:13:36.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee867fc700 1 -- 172.21.15.32:0/671819723 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee88002220 con 0x7fee940a07d0 2021-04-09T18:13:37.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.680 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee941a0570 con 0x7fee940a07d0 2021-04-09T18:13:37.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.681 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee941a0570 con 0x7fee940a07d0 2021-04-09T18:13:37.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.681 7fee867fc700 1 -- 172.21.15.32:0/671819723 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fee880052e0 con 0x7fee940a07d0 2021-04-09T18:13:37.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.681 7fee867fc700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee7c0188d0 0x7fee7c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.681 7fee867fc700 1 -- 172.21.15.32:0/671819723 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5461+0+0 (secure 0 0 0) 0x7fee880329b0 con 0x7fee940a07d0 2021-04-09T18:13:37.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.681 7fee9999b700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee7c0188d0 0x7fee7c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.682 7fee9999b700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee7c0188d0 0x7fee7c01add0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:37.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.701 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee940b9010 con 0x7fee940a07d0 2021-04-09T18:13:37.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.704 7fee867fc700 1 -- 172.21.15.32:0/671819723 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fee88000c90 con 0x7fee940a07d0 2021-04-09T18:13:37.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.887 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fee9410d490 con 0x7fee940a07d0 2021-04-09T18:13:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.892 7fee867fc700 1 -- 172.21.15.32:0/671819723 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 4 and data pool 5 v11) v1 ==== 187+0+0 (secure 0 0 0) 0x7fee880052e0 con 0x7fee940a07d0 2021-04-09T18:13:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 4 and data pool 5 2021-04-09T18:13:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.899 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee7c0188d0 msgr2=0x7fee7c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.899 7fee9bb7f700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee7c0188d0 0x7fee7c01add0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.899 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee940a07d0 msgr2=0x7fee9419ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:37.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.899 7fee9bb7f700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee940a07d0 0x7fee9419ff50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fee88006f60 tx=0x7fee8800b970).stop 2021-04-09T18:13:37.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.901 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 shutdown_connections 2021-04-09T18:13:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.901 7fee9bb7f700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee7c0188d0 0x7fee7c01add0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.901 7fee9bb7f700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee940a07d0 0x7fee9419ff50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.901 7fee9bb7f700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee940a0bd0 0x7fee940b9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.901 7fee9bb7f700 1 --2- 172.21.15.32:0/671819723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee940ab560 0x7fee940ada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.901 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 >> 172.21.15.32:0/671819723 conn(0x7fee940b9a90 msgr2=0x7fee940c1370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.901 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 shutdown_connections 2021-04-09T18:13:37.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:36.901 7fee9bb7f700 1 -- 172.21.15.32:0/671819723 wait complete. 2021-04-09T18:13:37.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:37.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.396 7f9a549e5700 1 Processor -- start 2021-04-09T18:13:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.396 7f9a549e5700 1 -- start start 2021-04-09T18:13:37.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a549e5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffa0 0x7f9a500d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a549e5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a500d7800 0x7f9a500d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a549e5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a500d7280 con 0x7f9a500d5c70 2021-04-09T18:13:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a549e5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a500d8580 con 0x7f9a500d7800 2021-04-09T18:13:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a549e5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a500d91e0 con 0x7f9a500cffa0 2021-04-09T18:13:37.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4ee1d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9a500d5c70 0x7f9a500d6070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52238/0 (socket says 172.21.15.32:52238) 2021-04-09T18:13:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4ee1d700 1 -- 172.21.15.32:0/1156887442 learned_addr learned my addr 172.21.15.32:0/1156887442 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:37.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4de1b700 1 --2- 172.21.15.32:0/1156887442 >> v2:172.21.15.86:3300/0 conn(0x7f9a500d7800 0x7f9a500d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4e61c700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffa0 0x7f9a500d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463216605 0 0) 0x7f9a480009b0 con 0x7f9a500d5c70 2021-04-09T18:13:37.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a38002fe0 con 0x7f9a500d5c70 2021-04-09T18:13:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 804813094 0 0) 0x7f9a480010e0 con 0x7f9a500d5c70 2021-04-09T18:13:37.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffa0 msgr2=0x7f9a500d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:37.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4d61a700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a500cffa0 0x7f9a500d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 >> v2:172.21.15.86:3300/0 conn(0x7f9a500d7800 msgr2=0x7f9a500d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:37.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4d61a700 1 --2- 172.21.15.32:0/1156887442 >> v2:172.21.15.86:3300/0 conn(0x7f9a500d7800 0x7f9a500d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a500d8220 con 0x7f9a500d5c70 2021-04-09T18:13:37.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.397 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f9a48001600 con 0x7f9a500d5c70 2021-04-09T18:13:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 >> v1:172.21.15.32:6789/0 conn(0x7f9a500d5c70 legacy=0x7f9a500d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a38003a60 0x7f9a38005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3804a760 0x7f9a3804cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a3804d500 0x7f9a3804fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a380502d0 con 0x7f9a3804a760 2021-04-09T18:13:37.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a380504f0 con 0x7f9a3804d500 2021-04-09T18:13:37.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a38050710 con 0x7f9a38003a60 2021-04-09T18:13:37.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4de1b700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3804a760 0x7f9a3804cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4ee1d700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a3804d500 0x7f9a3804fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4e61c700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a38003a60 0x7f9a38005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4de1b700 1 -- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a38003a60 msgr2=0x7f9a38005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:37.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4de1b700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a38003a60 0x7f9a38005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4de1b700 1 -- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a3804d500 msgr2=0x7f9a3804fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:37.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4de1b700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a3804d500 0x7f9a3804fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4de1b700 1 -- 172.21.15.32:0/1156887442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a380509b0 con 0x7f9a3804a760 2021-04-09T18:13:37.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4de1b700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3804a760 0x7f9a3804cc60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f9a44006fc0 tx=0x7f9a44009f30).ready entity=mon.0 client_cookie=99ffee997b15ee61 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:37.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a4400c420 con 0x7f9a3804a760 2021-04-09T18:13:37.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4400c720 con 0x7f9a3804a760 2021-04-09T18:13:37.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4d61a700 1 -- 172.21.15.32:0/1156887442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a4400cb70 con 0x7f9a3804a760 2021-04-09T18:13:37.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a549e5700 1 -- 172.21.15.32:0/1156887442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3804a760 msgr2=0x7f9a3804cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:37.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a549e5700 1 --2- 172.21.15.32:0/1156887442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a3804a760 0x7f9a3804cc60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f9a44006fc0 tx=0x7f9a44009f30).stop 2021-04-09T18:13:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.398 7f9a4ee1d700 1 -- 172.21.15.32:0/1156887442 reap_dead start 2021-04-09T18:13:37.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.399 7f9a549e5700 1 -- 172.21.15.32:0/1156887442 shutdown_connections 2021-04-09T18:13:37.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.399 7f9a549e5700 1 -- 172.21.15.32:0/1156887442 >> 172.21.15.32:0/1156887442 conn(0x7f9a500ce5c0 msgr2=0x7f9a500cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:37.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.399 7f9a549e5700 1 -- 172.21.15.32:0/1156887442 shutdown_connections 2021-04-09T18:13:37.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.399 7f9a549e5700 1 -- 172.21.15.32:0/1156887442 wait complete. 2021-04-09T18:13:37.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.419 7f9a549e5700 1 Processor -- start 2021-04-09T18:13:37.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.419 7f9a549e5700 1 -- start start 2021-04-09T18:13:37.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a549e5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a501acff0 0x7f9a501847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a549e5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a50183fc0 0x7f9a501a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a549e5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a50186ff0 0x7f9a501873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a549e5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a50188c30 con 0x7f9a50186ff0 2021-04-09T18:13:37.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a549e5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a501843c0 con 0x7f9a50183fc0 2021-04-09T18:13:37.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a549e5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a501a5c20 con 0x7f9a501acff0 2021-04-09T18:13:37.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4ee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a50186ff0 0x7f9a501873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4ee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a50186ff0 0x7f9a501873f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42662/0 (socket says 172.21.15.32:42662) 2021-04-09T18:13:37.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4ee1d700 1 -- 172.21.15.32:0/2338570714 learned_addr learned my addr 172.21.15.32:0/2338570714 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:37.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4de1b700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a50183fc0 0x7f9a501a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4ee1d700 1 -- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a501acff0 msgr2=0x7f9a501847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4e61c700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a501acff0 0x7f9a501847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4ee1d700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a501acff0 0x7f9a501847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4ee1d700 1 -- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a50183fc0 msgr2=0x7f9a501a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4ee1d700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a50183fc0 0x7f9a501a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4ee1d700 1 -- 172.21.15.32:0/2338570714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a50187a10 con 0x7f9a50186ff0 2021-04-09T18:13:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.420 7f9a4de1b700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a50183fc0 0x7f9a501a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.421 7f9a4ee1d700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a50186ff0 0x7f9a501873f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f9a48001060 tx=0x7f9a4800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.421 7f9a3effd700 1 -- 172.21.15.32:0/2338570714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a480070d0 con 0x7f9a50186ff0 2021-04-09T18:13:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.421 7f9a3effd700 1 -- 172.21.15.32:0/2338570714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a48001a20 con 0x7f9a50186ff0 2021-04-09T18:13:37.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.421 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a50187a10 con 0x7f9a50186ff0 2021-04-09T18:13:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.421 7f9a3effd700 1 -- 172.21.15.32:0/2338570714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a48004590 con 0x7f9a50186ff0 2021-04-09T18:13:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.421 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a50187a10 con 0x7f9a50186ff0 2021-04-09T18:13:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.422 7f9a3effd700 1 -- 172.21.15.32:0/2338570714 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9a480043a0 con 0x7f9a50186ff0 2021-04-09T18:13:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.422 7f9a3effd700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a34018930 0x7f9a3401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.422 7f9a3effd700 1 -- 172.21.15.32:0/2338570714 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9a480329d0 con 0x7f9a50186ff0 2021-04-09T18:13:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.422 7f9a4e61c700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a34018930 0x7f9a3401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:37.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.422 7f9a4e61c700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a34018930 0x7f9a3401adc0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:37.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.441 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a50182dc0 con 0x7f9a50186ff0 2021-04-09T18:13:37.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.444 7f9a3effd700 1 -- 172.21.15.32:0/2338570714 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9a480010a0 con 0x7f9a50186ff0 2021-04-09T18:13:37.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.634 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a500d7280 con 0x7f9a50186ff0 2021-04-09T18:13:37.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.635 7f9a3effd700 1 -- 172.21.15.32:0/2338570714 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f9a48001a20 con 0x7f9a50186ff0 2021-04-09T18:13:37.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:37.792 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":31,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:13:36.889458","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:13:35.174131","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-04-09 18:13:35.938524","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:37.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.637 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a34018930 msgr2=0x7f9a3401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:37.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.637 7f9a549e5700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a34018930 0x7f9a3401adc0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.637 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a50186ff0 msgr2=0x7f9a501873f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:37.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.637 7f9a549e5700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a50186ff0 0x7f9a501873f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f9a48001060 tx=0x7f9a4800a930).stop 2021-04-09T18:13:37.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.638 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 shutdown_connections 2021-04-09T18:13:37.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.638 7f9a549e5700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a34018930 0x7f9a3401adc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.638 7f9a549e5700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a50186ff0 0x7f9a501873f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.638 7f9a549e5700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a50183fc0 0x7f9a501a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.638 7f9a549e5700 1 --2- 172.21.15.32:0/2338570714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a501acff0 0x7f9a501847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.638 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 >> 172.21.15.32:0/2338570714 conn(0x7f9a5009abb0 msgr2=0x7f9a500a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:37.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.638 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 shutdown_connections 2021-04-09T18:13:37.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:37.638 7f9a549e5700 1 -- 172.21.15.32:0/2338570714 wait complete. 2021-04-09T18:13:37.921 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:38.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.079 7ff2f4761700 1 Processor -- start 2021-04-09T18:13:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.079 7ff2f4761700 1 -- start start 2021-04-09T18:13:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.080 7ff2f4761700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff2ec0d7ca0 0x7ff2ec0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.080 7ff2f4761700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec10d480 con 0x7ff2ec10da00 2021-04-09T18:13:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.080 7ff2f4761700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec0d86c0 con 0x7ff2ec10d010 2021-04-09T18:13:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.080 7ff2f4761700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2ec0d9220 con 0x7ff2ec0d7ca0 2021-04-09T18:13:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.080 7ff2f2d7e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff2ec10da00 0x7ff2ec10de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52258/0 (socket says 172.21.15.32:52258) 2021-04-09T18:13:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.080 7ff2f2d7e700 1 -- 172.21.15.32:0/1869806280 learned_addr learned my addr 172.21.15.32:0/1869806280 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.080 7ff2f1d7c700 1 --2- 172.21.15.32:0/1869806280 >> v2:172.21.15.86:3300/0 conn(0x7ff2ec0d7ca0 0x7ff2ec0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.080 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994832426 0 0) 0x7ff2e4000d30 con 0x7ff2ec10da00 2021-04-09T18:13:38.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.080 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d8002fe0 con 0x7ff2ec10da00 2021-04-09T18:13:38.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 679384044 0 0) 0x7ff2e4001270 con 0x7ff2ec10da00 2021-04-09T18:13:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 >> v2:172.21.15.86:3300/0 conn(0x7ff2ec0d7ca0 msgr2=0x7ff2ec0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 --2- 172.21.15.32:0/1869806280 >> v2:172.21.15.86:3300/0 conn(0x7ff2ec0d7ca0 0x7ff2ec0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 >> v1:172.21.15.86:6789/0 conn(0x7ff2ec10d010 legacy=0x7ff2ec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec112100 con 0x7ff2ec10da00 2021-04-09T18:13:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7ff2e4001980 con 0x7ff2ec10da00 2021-04-09T18:13:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 >> v1:172.21.15.32:6789/0 conn(0x7ff2ec10da00 legacy=0x7ff2ec10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2d8003a70 0x7ff2d8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2d80067a0 0x7ff2d8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d80095a0 0x7ff2d800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d800c310 con 0x7ff2d80095a0 2021-04-09T18:13:38.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d800c530 con 0x7ff2d8003a70 2021-04-09T18:13:38.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d8050760 con 0x7ff2d80067a0 2021-04-09T18:13:38.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f2d7e700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d80095a0 0x7ff2d800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f257d700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2d8003a70 0x7ff2d8005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f1d7c700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2d80067a0 0x7ff2d8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f2d7e700 1 -- 172.21.15.32:0/1869806280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2d80067a0 msgr2=0x7ff2d8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:38.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f2d7e700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2d80067a0 0x7ff2d8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f2d7e700 1 -- 172.21.15.32:0/1869806280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2d8003a70 msgr2=0x7ff2d8005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:38.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f2d7e700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2d8003a70 0x7ff2d8005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f2d7e700 1 -- 172.21.15.32:0/1869806280 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d8050990 con 0x7ff2d80095a0 2021-04-09T18:13:38.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f2d7e700 1 -- 172.21.15.32:0/1869806280 reap_dead start 2021-04-09T18:13:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f2d7e700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d80095a0 0x7ff2d800bac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7ff2e40021c0 tx=0x7ff2e400a060).ready entity=mon.0 client_cookie=8cbd978db35d934f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2e40070c0 con 0x7ff2d80095a0 2021-04-09T18:13:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e4007400 con 0x7ff2d80095a0 2021-04-09T18:13:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f157b700 1 -- 172.21.15.32:0/1869806280 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2e4004720 con 0x7ff2d80095a0 2021-04-09T18:13:38.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f4761700 1 -- 172.21.15.32:0/1869806280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d80095a0 msgr2=0x7ff2d800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.081 7ff2f4761700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d80095a0 0x7ff2d800bac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7ff2e40021c0 tx=0x7ff2e400a060).stop 2021-04-09T18:13:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.082 7ff2f4761700 1 -- 172.21.15.32:0/1869806280 shutdown_connections 2021-04-09T18:13:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.082 7ff2f4761700 1 --2- 172.21.15.32:0/1869806280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2d80095a0 0x7ff2d800bac0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.082 7ff2f4761700 1 -- 172.21.15.32:0/1869806280 >> 172.21.15.32:0/1869806280 conn(0x7ff2ec0ce5f0 msgr2=0x7ff2ec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:38.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.082 7ff2f4761700 1 -- 172.21.15.32:0/1869806280 shutdown_connections 2021-04-09T18:13:38.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.082 7ff2f4761700 1 -- 172.21.15.32:0/1869806280 wait complete. 2021-04-09T18:13:38.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.102 7ff2f4761700 1 Processor -- start 2021-04-09T18:13:38.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.102 7ff2f4761700 1 -- start start 2021-04-09T18:13:38.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f4761700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2ec1ac380 0x7ff2ec18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f4761700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2ec1ac780 0x7ff2ec18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f4761700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2ec183520 0x7ff2ec183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f4761700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec1ace10 con 0x7ff2ec1ac380 2021-04-09T18:13:38.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f4761700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec189cf0 con 0x7ff2ec1ac780 2021-04-09T18:13:38.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f4761700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec183f40 con 0x7ff2ec183520 2021-04-09T18:13:38.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f257d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2ec1ac380 0x7ff2ec18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f257d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2ec1ac380 0x7ff2ec18a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42682/0 (socket says 172.21.15.32:42682) 2021-04-09T18:13:38.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f257d700 1 -- 172.21.15.32:0/2331401225 learned_addr learned my addr 172.21.15.32:0/2331401225 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f2d7e700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2ec183520 0x7ff2ec183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f1d7c700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2ec1ac780 0x7ff2ec18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f257d700 1 -- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2ec183520 msgr2=0x7ff2ec183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f257d700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2ec183520 0x7ff2ec183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f257d700 1 -- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2ec1ac780 msgr2=0x7ff2ec18a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f257d700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2ec1ac780 0x7ff2ec18a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f257d700 1 -- 172.21.15.32:0/2331401225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec184160 con 0x7ff2ec1ac380 2021-04-09T18:13:38.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f1d7c700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2ec1ac780 0x7ff2ec18a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:38.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.103 7ff2f2d7e700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2ec183520 0x7ff2ec183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:38.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.104 7ff2f257d700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2ec1ac380 0x7ff2ec18a230 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff2e800b610 tx=0x7ff2e800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:38.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.104 7ff2e2ffd700 1 -- 172.21.15.32:0/2331401225 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2e800bda0 con 0x7ff2ec1ac380 2021-04-09T18:13:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.104 7ff2e2ffd700 1 -- 172.21.15.32:0/2331401225 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e8000ca0 con 0x7ff2ec1ac380 2021-04-09T18:13:38.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.104 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ec184160 con 0x7ff2ec1ac380 2021-04-09T18:13:38.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.104 7ff2e2ffd700 1 -- 172.21.15.32:0/2331401225 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2e8002220 con 0x7ff2ec1ac380 2021-04-09T18:13:38.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.104 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ec184160 con 0x7ff2ec1ac380 2021-04-09T18:13:38.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.104 7ff2e2ffd700 1 -- 172.21.15.32:0/2331401225 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff2e80052e0 con 0x7ff2ec1ac380 2021-04-09T18:13:38.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.104 7ff2e2ffd700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2d4018930 0x7ff2d401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.105 7ff2e2ffd700 1 -- 172.21.15.32:0/2331401225 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff2e80329d0 con 0x7ff2ec1ac380 2021-04-09T18:13:38.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.105 7ff2f1d7c700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2d4018930 0x7ff2d401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.105 7ff2f1d7c700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2d4018930 0x7ff2d401adc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:38.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.124 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ec182d10 con 0x7ff2ec1ac380 2021-04-09T18:13:38.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.128 7ff2e2ffd700 1 -- 172.21.15.32:0/2331401225 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff2e8006bb0 con 0x7ff2ec1ac380 2021-04-09T18:13:38.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.315 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2ec182d10 con 0x7ff2ec1ac380 2021-04-09T18:13:38.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.316 7ff2e2ffd700 1 -- 172.21.15.32:0/2331401225 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+13993 (secure 0 0 0) 0x7ff2e8000ca0 con 0x7ff2ec1ac380 2021-04-09T18:13:38.341 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:38.341 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":32,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:13:37.891009","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:13:35.174131","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-04-09 18:13:35.938524","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:38.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.318 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2d4018930 msgr2=0x7ff2d401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.318 7ff2f4761700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2d4018930 0x7ff2d401adc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.318 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2ec1ac380 msgr2=0x7ff2ec18a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.318 7ff2f4761700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2ec1ac380 0x7ff2ec18a230 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff2e800b610 tx=0x7ff2e800b9b0).stop 2021-04-09T18:13:38.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.319 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 shutdown_connections 2021-04-09T18:13:38.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.319 7ff2f4761700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2d4018930 0x7ff2d401adc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.319 7ff2f4761700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2ec183520 0x7ff2ec183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.319 7ff2f4761700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2ec1ac780 0x7ff2ec18a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.319 7ff2f4761700 1 --2- 172.21.15.32:0/2331401225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2ec1ac380 0x7ff2ec18a230 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.319 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 >> 172.21.15.32:0/2331401225 conn(0x7ff2ec19b890 msgr2=0x7ff2ec1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:38.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.319 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 shutdown_connections 2021-04-09T18:13:38.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.319 7ff2f4761700 1 -- 172.21.15.32:0/2331401225 wait complete. 2021-04-09T18:13:38.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:13:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.599 7f6f60415700 1 Processor -- start 2021-04-09T18:13:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.599 7f6f60415700 1 -- start start 2021-04-09T18:13:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f60415700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f5810d010 0x7f6f580d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f60415700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6f580d1da0 0x7f6f580d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f60415700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5810d480 con 0x7f6f580d82c0 2021-04-09T18:13:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f60415700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f580d8d30 con 0x7f6f5810d010 2021-04-09T18:13:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f60415700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f580d9220 con 0x7f6f580d1da0 2021-04-09T18:13:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f5e231700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f5810d010 0x7f6f580d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f5e231700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f5810d010 0x7f6f580d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42686/0 (socket says 172.21.15.32:42686) 2021-04-09T18:13:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f5e231700 1 -- 172.21.15.32:0/1817185373 learned_addr learned my addr 172.21.15.32:0/1817185373 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f5da30700 1 --2- 172.21.15.32:0/1817185373 >> v2:172.21.15.86:3300/0 conn(0x7f6f580d1da0 0x7f6f580d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f5e231700 1 -- 172.21.15.32:0/1817185373 >> v2:172.21.15.86:3300/0 conn(0x7f6f580d1da0 msgr2=0x7f6f580d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f5e231700 1 --2- 172.21.15.32:0/1817185373 >> v2:172.21.15.86:3300/0 conn(0x7f6f580d1da0 0x7f6f580d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f5e231700 1 -- 172.21.15.32:0/1817185373 >> v1:172.21.15.86:6789/0 conn(0x7f6f580d82c0 legacy=0x7f6f5810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.600 7f6f5e231700 1 -- 172.21.15.32:0/1817185373 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f580d86f0 con 0x7f6f5810d010 2021-04-09T18:13:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.601 7f6f5e231700 1 --2- 172.21.15.32:0/1817185373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5810d010 0x7f6f580d1780 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f6f54006700 tx=0x7f6f54009b80).ready entity=mon.0 client_cookie=b549e238362114fb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.601 7f6f5d22f700 1 -- 172.21.15.32:0/1817185373 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f5400c0c0 con 0x7f6f5810d010 2021-04-09T18:13:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.601 7f6f5d22f700 1 -- 172.21.15.32:0/1817185373 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5400c440 con 0x7f6f5810d010 2021-04-09T18:13:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.601 7f6f5d22f700 1 -- 172.21.15.32:0/1817185373 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f5400c920 con 0x7f6f5810d010 2021-04-09T18:13:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.601 7f6f60415700 1 -- 172.21.15.32:0/1817185373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5810d010 msgr2=0x7f6f580d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.601 7f6f60415700 1 --2- 172.21.15.32:0/1817185373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5810d010 0x7f6f580d1780 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f6f54006700 tx=0x7f6f54009b80).stop 2021-04-09T18:13:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.602 7f6f60415700 1 -- 172.21.15.32:0/1817185373 shutdown_connections 2021-04-09T18:13:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.602 7f6f60415700 1 --2- 172.21.15.32:0/1817185373 >> v2:172.21.15.86:3300/0 conn(0x7f6f580d1da0 0x7f6f580d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.602 7f6f60415700 1 --2- 172.21.15.32:0/1817185373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5810d010 0x7f6f580d1780 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.602 7f6f60415700 1 -- 172.21.15.32:0/1817185373 >> 172.21.15.32:0/1817185373 conn(0x7f6f580ce5f0 msgr2=0x7f6f580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.602 7f6f60415700 1 -- 172.21.15.32:0/1817185373 shutdown_connections 2021-04-09T18:13:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.602 7f6f60415700 1 -- 172.21.15.32:0/1817185373 wait complete. 2021-04-09T18:13:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.622 7f6f60415700 1 Processor -- start 2021-04-09T18:13:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.622 7f6f60415700 1 -- start start 2021-04-09T18:13:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.622 7f6f60415700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f580b1680 0x7f6f580b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.622 7f6f60415700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5809e330 0x7f6f580b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.622 7f6f60415700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f580b2190 0x7f6f5809f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.622 7f6f60415700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5809e730 con 0x7f6f580b2190 2021-04-09T18:13:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.622 7f6f60415700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5809f6c0 con 0x7f6f580b1680 2021-04-09T18:13:38.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.622 7f6f60415700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5809f8e0 con 0x7f6f5809e330 2021-04-09T18:13:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5ea32700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f580b2190 0x7f6f5809f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5ea32700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f580b2190 0x7f6f5809f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42696/0 (socket says 172.21.15.32:42696) 2021-04-09T18:13:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5ea32700 1 -- 172.21.15.32:0/4266944076 learned_addr learned my addr 172.21.15.32:0/4266944076 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5e231700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f580b1680 0x7f6f580b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5da30700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5809e330 0x7f6f580b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5ea32700 1 -- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5809e330 msgr2=0x7f6f580b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5ea32700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5809e330 0x7f6f580b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5ea32700 1 -- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f580b1680 msgr2=0x7f6f580b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:38.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5ea32700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f580b1680 0x7f6f580b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:38.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5ea32700 1 -- 172.21.15.32:0/4266944076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5809fb00 con 0x7f6f580b2190 2021-04-09T18:13:38.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5e231700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f580b1680 0x7f6f580b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:38.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f5ea32700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f580b2190 0x7f6f5809f0a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f6f5000b8e0 tx=0x7f6f5000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:38.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f4affd700 1 -- 172.21.15.32:0/4266944076 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f50000d30 con 0x7f6f580b2190 2021-04-09T18:13:38.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f4affd700 1 -- 172.21.15.32:0/4266944076 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5000c2f0 con 0x7f6f580b2190 2021-04-09T18:13:38.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f60415700 1 -- 172.21.15.32:0/4266944076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5809fb00 con 0x7f6f580b2190 2021-04-09T18:13:38.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f4affd700 1 -- 172.21.15.32:0/4266944076 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f5000c7d0 con 0x7f6f580b2190 2021-04-09T18:13:38.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.623 7f6f60415700 1 -- 172.21.15.32:0/4266944076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5809fb00 con 0x7f6f580b2190 2021-04-09T18:13:38.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.624 7f6f4affd700 1 -- 172.21.15.32:0/4266944076 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6f50001220 con 0x7f6f580b2190 2021-04-09T18:13:38.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.624 7f6f4affd700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:38.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.624 7f6f4affd700 1 -- 172.21.15.32:0/4266944076 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6f50032d90 con 0x7f6f580b2190 2021-04-09T18:13:38.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.624 7f6f5e231700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:38.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.625 7f6f5e231700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:38.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.644 7f6f60415700 1 -- 172.21.15.32:0/4266944076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5810d480 con 0x7f6f580b2190 2021-04-09T18:13:38.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.648 7f6f4affd700 1 -- 172.21.15.32:0/4266944076 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6f5000bdc0 con 0x7f6f580b2190 2021-04-09T18:13:38.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:38.846 7f6f60415700 1 -- 172.21.15.32:0/4266944076 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f6f5810d480 con 0x7f6f580b2190 2021-04-09T18:13:39.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.512 7f6f4affd700 1 -- 172.21.15.32:0/4266944076 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6f50001220 con 0x7f6f580b2190 2021-04-09T18:13:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 -- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 msgr2=0x7f6f4001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 -- 172.21.15.32:0/4266944076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f580b2190 msgr2=0x7f6f5809f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f580b2190 0x7f6f5809f0a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f6f5000b8e0 tx=0x7f6f5000bc80).stop 2021-04-09T18:13:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 -- 172.21.15.32:0/4266944076 shutdown_connections 2021-04-09T18:13:39.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:39.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f580b2190 0x7f6f5809f0a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5809e330 0x7f6f580b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 --2- 172.21.15.32:0/4266944076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f580b1680 0x7f6f580b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:39.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 -- 172.21.15.32:0/4266944076 >> 172.21.15.32:0/4266944076 conn(0x7f6f580c4290 msgr2=0x7f6f580c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:39.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 -- 172.21.15.32:0/4266944076 shutdown_connections 2021-04-09T18:13:39.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.564 7f6f60415700 1 -- 172.21.15.32:0/4266944076 wait complete. 2021-04-09T18:13:39.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:39.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.731 7f560aec5700 1 Processor -- start 2021-04-09T18:13:39.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.731 7f560aec5700 1 -- start start 2021-04-09T18:13:39.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.732 7f560aec5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56040d7ca0 0x7f56040d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:39.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.732 7f560aec5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560410d480 con 0x7f560410d010 2021-04-09T18:13:39.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.732 7f560aec5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56040d86c0 con 0x7f560410da00 2021-04-09T18:13:39.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.732 7f560aec5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56040d9220 con 0x7f56040d7ca0 2021-04-09T18:13:39.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.732 7f5603fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56040d7ca0 0x7f56040d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:39.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.732 7f5608ce1700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f560410d010 0x7f56040d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52290/0 (socket says 172.21.15.32:52290) 2021-04-09T18:13:39.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.732 7f5608ce1700 1 -- 172.21.15.32:0/2330677716 learned_addr learned my addr 172.21.15.32:0/2330677716 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:39.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.732 7f56037fe700 1 -- 172.21.15.32:0/2330677716 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813327796 0 0) 0x7f55ec000ce0 con 0x7f560410d010 2021-04-09T18:13:39.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f56037fe700 1 -- 172.21.15.32:0/2330677716 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f0002fe0 con 0x7f560410d010 2021-04-09T18:13:39.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f56037fe700 1 -- 172.21.15.32:0/2330677716 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710540448 0 0) 0x7f55fc000d30 con 0x7f560410da00 2021-04-09T18:13:39.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f56037fe700 1 -- 172.21.15.32:0/2330677716 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f00042c0 con 0x7f560410da00 2021-04-09T18:13:39.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f5603fff700 1 -- 172.21.15.32:0/2330677716 >> v1:172.21.15.86:6789/0 conn(0x7f560410da00 legacy=0x7f560410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f5603fff700 1 -- 172.21.15.32:0/2330677716 >> v1:172.21.15.32:6789/0 conn(0x7f560410d010 legacy=0x7f56040d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f5603fff700 1 -- 172.21.15.32:0/2330677716 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56040d1f70 con 0x7f56040d7ca0 2021-04-09T18:13:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f5603fff700 1 --2- 172.21.15.32:0/2330677716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56040d7ca0 0x7f56040d80a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f55f4006650 tx=0x7f55f4009a80).ready entity=mon.0 client_cookie=c936bedf2d82a858 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f56037fe700 1 -- 172.21.15.32:0/2330677716 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55f40009b0 con 0x7f56040d7ca0 2021-04-09T18:13:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f56037fe700 1 -- 172.21.15.32:0/2330677716 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f400c0c0 con 0x7f56040d7ca0 2021-04-09T18:13:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f56037fe700 1 -- 172.21.15.32:0/2330677716 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55f400c510 con 0x7f56040d7ca0 2021-04-09T18:13:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f560aec5700 1 -- 172.21.15.32:0/2330677716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56040d7ca0 msgr2=0x7f56040d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.733 7f560aec5700 1 --2- 172.21.15.32:0/2330677716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56040d7ca0 0x7f56040d80a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f55f4006650 tx=0x7f55f4009a80).stop 2021-04-09T18:13:39.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.734 7f560aec5700 1 -- 172.21.15.32:0/2330677716 shutdown_connections 2021-04-09T18:13:39.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.734 7f560aec5700 1 --2- 172.21.15.32:0/2330677716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56040d7ca0 0x7f56040d80a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.734 7f560aec5700 1 -- 172.21.15.32:0/2330677716 >> 172.21.15.32:0/2330677716 conn(0x7f56040ce5f0 msgr2=0x7f56040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.734 7f560aec5700 1 -- 172.21.15.32:0/2330677716 shutdown_connections 2021-04-09T18:13:39.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.734 7f560aec5700 1 -- 172.21.15.32:0/2330677716 wait complete. 2021-04-09T18:13:39.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.754 7f560aec5700 1 Processor -- start 2021-04-09T18:13:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.754 7f560aec5700 1 -- start start 2021-04-09T18:13:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.754 7f560aec5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5604099920 0x7f5604179f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.754 7f560aec5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f560417a560 0x7f56041b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:39.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.754 7f560aec5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56041b85a0 0x7f56041b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:39.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.754 7f560aec5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560417a960 con 0x7f56041b85a0 2021-04-09T18:13:39.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.754 7f560aec5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5604099d20 con 0x7f560417a560 2021-04-09T18:13:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f560aec5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5604179610 con 0x7f5604099920 2021-04-09T18:13:39.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56094e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56041b85a0 0x7f56041b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:39.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56094e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56041b85a0 0x7f56041b89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42708/0 (socket says 172.21.15.32:42708) 2021-04-09T18:13:39.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56094e2700 1 -- 172.21.15.32:0/567253907 learned_addr learned my addr 172.21.15.32:0/567253907 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:39.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f5603fff700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f560417a560 0x7f56041b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:39.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f5608ce1700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5604099920 0x7f5604179f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:39.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56094e2700 1 -- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5604099920 msgr2=0x7f5604179f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:39.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56094e2700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5604099920 0x7f5604179f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:39.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56094e2700 1 -- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f560417a560 msgr2=0x7f56041b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:39.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56094e2700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f560417a560 0x7f56041b7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56094e2700 1 -- 172.21.15.32:0/567253907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604180be0 con 0x7f56041b85a0 2021-04-09T18:13:40.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56094e2700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56041b85a0 0x7f56041b89a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f55fc009980 tx=0x7f55fc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:40.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56017fa700 1 -- 172.21.15.32:0/567253907 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55fc00be10 con 0x7f56041b85a0 2021-04-09T18:13:40.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56017fa700 1 -- 172.21.15.32:0/567253907 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc005870 con 0x7f56041b85a0 2021-04-09T18:13:40.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f560aec5700 1 -- 172.21.15.32:0/567253907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5604180be0 con 0x7f56041b85a0 2021-04-09T18:13:40.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f56017fa700 1 -- 172.21.15.32:0/567253907 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55fc005d50 con 0x7f56041b85a0 2021-04-09T18:13:40.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.755 7f560aec5700 1 -- 172.21.15.32:0/567253907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5604180be0 con 0x7f56041b85a0 2021-04-09T18:13:40.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.756 7f56017fa700 1 -- 172.21.15.32:0/567253907 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f55fc006030 con 0x7f56041b85a0 2021-04-09T18:13:40.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.756 7f56017fa700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55e8018930 0x7f55e801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.756 7f56017fa700 1 -- 172.21.15.32:0/567253907 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f55fc032dd0 con 0x7f56041b85a0 2021-04-09T18:13:40.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.756 7f5608ce1700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55e8018930 0x7f55e801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:40.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.757 7f5608ce1700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55e8018930 0x7f55e801ae10 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:40.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.776 7f560aec5700 1 -- 172.21.15.32:0/567253907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56041b7070 con 0x7f56041b85a0 2021-04-09T18:13:40.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.780 7f56017fa700 1 -- 172.21.15.32:0/567253907 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f55fc005870 con 0x7f56041b85a0 2021-04-09T18:13:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.970 7f560aec5700 1 -- 172.21.15.32:0/567253907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f560410d480 con 0x7f56041b85a0 2021-04-09T18:13:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.971 7f56017fa700 1 -- 172.21.15.32:0/567253907 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2776 (secure 0 0 0) 0x7f55fc006f20 con 0x7f56041b85a0 2021-04-09T18:13:40.053 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 13 2021-04-09T18:13:40.079 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:40.079 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:39.509220","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:13:40.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 -- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55e8018930 msgr2=0x7f55e801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55e8018930 0x7f55e801ae10 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 -- 172.21.15.32:0/567253907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56041b85a0 msgr2=0x7f56041b89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56041b85a0 0x7f56041b89a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f55fc009980 tx=0x7f55fc00bcd0).stop 2021-04-09T18:13:40.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 -- 172.21.15.32:0/567253907 shutdown_connections 2021-04-09T18:13:40.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55e8018930 0x7f55e801ae10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56041b85a0 0x7f56041b89a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f560417a560 0x7f56041b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 --2- 172.21.15.32:0/567253907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5604099920 0x7f5604179f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 -- 172.21.15.32:0/567253907 >> 172.21.15.32:0/567253907 conn(0x7f56041aa2b0 msgr2=0x7f56041a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:40.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 -- 172.21.15.32:0/567253907 shutdown_connections 2021-04-09T18:13:40.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:39.973 7f560aec5700 1 -- 172.21.15.32:0/567253907 wait complete. 2021-04-09T18:13:40.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:40.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.298 7f032314f700 1 Processor -- start 2021-04-09T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.298 7f032314f700 1 -- start start 2021-04-09T18:13:40.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f032314f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f031c10d010 0x7f031c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f032314f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f031c0d8490 0x7f031c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f032314f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c10d480 con 0x7f031c0d1da0 2021-04-09T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f032314f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031c0d8d30 con 0x7f031c10d010 2021-04-09T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f032314f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031c0d9220 con 0x7f031c0d8490 2021-04-09T18:13:40.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f0320f6b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f031c10d010 0x7f031c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:40.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f0320f6b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f031c10d010 0x7f031c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42712/0 (socket says 172.21.15.32:42712) 2021-04-09T18:13:40.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f0320f6b700 1 -- 172.21.15.32:0/4051853418 learned_addr learned my addr 172.21.15.32:0/4051853418 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f032176c700 1 --2- 172.21.15.32:0/4051853418 >> v2:172.21.15.86:3300/0 conn(0x7f031c0d8490 0x7f031c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f0320f6b700 1 -- 172.21.15.32:0/4051853418 >> v2:172.21.15.86:3300/0 conn(0x7f031c0d8490 msgr2=0x7f031c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:40.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f0320f6b700 1 --2- 172.21.15.32:0/4051853418 >> v2:172.21.15.86:3300/0 conn(0x7f031c0d8490 0x7f031c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f0320f6b700 1 -- 172.21.15.32:0/4051853418 >> v1:172.21.15.86:6789/0 conn(0x7f031c0d1da0 legacy=0x7f031c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.299 7f0320f6b700 1 -- 172.21.15.32:0/4051853418 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c10e020 con 0x7f031c10d010 2021-04-09T18:13:40.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.300 7f032176c700 1 --2- 172.21.15.32:0/4051853418 >> v2:172.21.15.86:3300/0 conn(0x7f031c0d8490 0x7f031c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:40.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.300 7f0320f6b700 1 --2- 172.21.15.32:0/4051853418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c10d010 0x7f031c0d1780 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f0304006700 tx=0x7f0304009b80).ready entity=mon.0 client_cookie=28e21f9787885587 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:40.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.300 7f031b7fe700 1 -- 172.21.15.32:0/4051853418 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f030400c0c0 con 0x7f031c10d010 2021-04-09T18:13:40.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.300 7f031b7fe700 1 -- 172.21.15.32:0/4051853418 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030400c440 con 0x7f031c10d010 2021-04-09T18:13:40.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.300 7f031b7fe700 1 -- 172.21.15.32:0/4051853418 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f030400c920 con 0x7f031c10d010 2021-04-09T18:13:40.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.300 7f032314f700 1 -- 172.21.15.32:0/4051853418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c10d010 msgr2=0x7f031c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.300 7f032314f700 1 --2- 172.21.15.32:0/4051853418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c10d010 0x7f031c0d1780 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f0304006700 tx=0x7f0304009b80).stop 2021-04-09T18:13:40.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.301 7f032314f700 1 -- 172.21.15.32:0/4051853418 shutdown_connections 2021-04-09T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.301 7f032314f700 1 --2- 172.21.15.32:0/4051853418 >> v2:172.21.15.86:3300/0 conn(0x7f031c0d8490 0x7f031c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.301 7f032314f700 1 --2- 172.21.15.32:0/4051853418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c10d010 0x7f031c0d1780 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.301 7f032314f700 1 -- 172.21.15.32:0/4051853418 >> 172.21.15.32:0/4051853418 conn(0x7f031c0ce5f0 msgr2=0x7f031c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.301 7f032314f700 1 -- 172.21.15.32:0/4051853418 shutdown_connections 2021-04-09T18:13:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.301 7f032314f700 1 -- 172.21.15.32:0/4051853418 wait complete. 2021-04-09T18:13:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.321 7f032314f700 1 Processor -- start 2021-04-09T18:13:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.321 7f032314f700 1 -- start start 2021-04-09T18:13:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.321 7f032314f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f031c179bf0 0x7f031c179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.321 7f032314f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f031c17c540 0x7f031c17c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.321 7f032314f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c0aca60 0x7f031c0ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.321 7f032314f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c17cf60 con 0x7f031c0aca60 2021-04-09T18:13:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.321 7f032314f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c0ad480 con 0x7f031c179bf0 2021-04-09T18:13:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.321 7f032314f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031c0ad6a0 con 0x7f031c17c540 2021-04-09T18:13:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.321 7f032176c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c0aca60 0x7f031c0ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032176c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c0aca60 0x7f031c0ace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42720/0 (socket says 172.21.15.32:42720) 2021-04-09T18:13:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032176c700 1 -- 172.21.15.32:0/2278172661 learned_addr learned my addr 172.21.15.32:0/2278172661 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:40.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f0320f6b700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f031c179bf0 0x7f031c179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:40.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f031bfff700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f031c17c540 0x7f031c17c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:40.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032176c700 1 -- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f031c17c540 msgr2=0x7f031c17c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032176c700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f031c17c540 0x7f031c17c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032176c700 1 -- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f031c179bf0 msgr2=0x7f031c179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032176c700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f031c179bf0 0x7f031c179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032176c700 1 -- 172.21.15.32:0/2278172661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c0ad8c0 con 0x7f031c0aca60 2021-04-09T18:13:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f031bfff700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f031c17c540 0x7f031c17c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:40.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032176c700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c0aca60 0x7f031c0ace60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f0314009980 tx=0x7f031400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:40.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f03197fa700 1 -- 172.21.15.32:0/2278172661 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f031400c2f0 con 0x7f031c0aca60 2021-04-09T18:13:40.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f03197fa700 1 -- 172.21.15.32:0/2278172661 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0314006790 con 0x7f031c0aca60 2021-04-09T18:13:40.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032314f700 1 -- 172.21.15.32:0/2278172661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031c0ad8c0 con 0x7f031c0aca60 2021-04-09T18:13:40.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f03197fa700 1 -- 172.21.15.32:0/2278172661 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f031400c7e0 con 0x7f031c0aca60 2021-04-09T18:13:40.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.322 7f032314f700 1 -- 172.21.15.32:0/2278172661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031c0ad8c0 con 0x7f031c0aca60 2021-04-09T18:13:40.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.323 7f03197fa700 1 -- 172.21.15.32:0/2278172661 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f03140011f0 con 0x7f031c0aca60 2021-04-09T18:13:40.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.323 7f03197fa700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0300018930 0x7f030001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.323 7f03197fa700 1 -- 172.21.15.32:0/2278172661 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0314032df0 con 0x7f031c0aca60 2021-04-09T18:13:40.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.323 7f0320f6b700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0300018930 0x7f030001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:40.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.324 7f0320f6b700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0300018930 0x7f030001adc0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:40.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.343 7f032314f700 1 -- 172.21.15.32:0/2278172661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031c17cf60 con 0x7f031c0aca60 2021-04-09T18:13:40.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.347 7f03197fa700 1 -- 172.21.15.32:0/2278172661 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f031400be10 con 0x7f031c0aca60 2021-04-09T18:13:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.537 7f032314f700 1 -- 172.21.15.32:0/2278172661 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f031c17cf60 con 0x7f031c0aca60 2021-04-09T18:13:40.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.538 7f03197fa700 1 -- 172.21.15.32:0/2278172661 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f0314006670 con 0x7f031c0aca60 2021-04-09T18:13:40.565 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:40.566 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":32,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:13:37.891009","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:13:35.174131","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-04-09 18:13:35.938524","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:40.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 -- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0300018930 msgr2=0x7f030001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0300018930 0x7f030001adc0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 -- 172.21.15.32:0/2278172661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c0aca60 msgr2=0x7f031c0ace60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c0aca60 0x7f031c0ace60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f0314009980 tx=0x7f031400bcd0).stop 2021-04-09T18:13:40.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 -- 172.21.15.32:0/2278172661 shutdown_connections 2021-04-09T18:13:40.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0300018930 0x7f030001adc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f031c0aca60 0x7f031c0ace60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f031c17c540 0x7f031c17c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 --2- 172.21.15.32:0/2278172661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f031c179bf0 0x7f031c179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 -- 172.21.15.32:0/2278172661 >> 172.21.15.32:0/2278172661 conn(0x7f031c1acae0 msgr2=0x7f031c17ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:40.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.540 7f032314f700 1 -- 172.21.15.32:0/2278172661 shutdown_connections 2021-04-09T18:13:40.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.541 7f032314f700 1 -- 172.21.15.32:0/2278172661 wait complete. 2021-04-09T18:13:40.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:13:40.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.820 7f4c0a2e2700 1 Processor -- start 2021-04-09T18:13:40.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.820 7f4c0a2e2700 1 -- start start 2021-04-09T18:13:40.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c0a2e2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c040cffb0 0x7f4c04087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c0a2e2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c04088460 con 0x7f4c040d8050 2021-04-09T18:13:40.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c0a2e2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c040d8620 con 0x7f4c040d7c50 2021-04-09T18:13:40.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c0a2e2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c041b0c80 con 0x7f4c040cffb0 2021-04-09T18:13:40.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c03fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c040cffb0 0x7f4c04087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:40.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c088ff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4c040d8050 0x7f4c0410d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52316/0 (socket says 172.21.15.32:52316) 2021-04-09T18:13:40.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c088ff700 1 -- 172.21.15.32:0/3677883760 learned_addr learned my addr 172.21.15.32:0/3677883760 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:40.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c037fe700 1 -- 172.21.15.32:0/3677883760 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163667955 0 0) 0x7f4bec000c60 con 0x7f4c040d7c50 2021-04-09T18:13:40.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c037fe700 1 -- 172.21.15.32:0/3677883760 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf0002fe0 con 0x7f4c040d7c50 2021-04-09T18:13:40.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c037fe700 1 -- 172.21.15.32:0/3677883760 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113297153 0 0) 0x7f4bfc000d30 con 0x7f4c040d8050 2021-04-09T18:13:40.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c03fff700 1 -- 172.21.15.32:0/3677883760 >> v1:172.21.15.86:6789/0 conn(0x7f4c040d7c50 legacy=0x7f4c041b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c03fff700 1 -- 172.21.15.32:0/3677883760 >> v1:172.21.15.32:6789/0 conn(0x7f4c040d8050 legacy=0x7f4c0410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c03fff700 1 -- 172.21.15.32:0/3677883760 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c040d91f0 con 0x7f4c040cffb0 2021-04-09T18:13:40.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c03fff700 1 --2- 172.21.15.32:0/3677883760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040cffb0 0x7f4c04087e40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f4bf4006650 tx=0x7f4bf4009a80).ready entity=mon.0 client_cookie=1b7572d489b64cbe server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:40.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.821 7f4c037fe700 1 -- 172.21.15.32:0/3677883760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bf40009b0 con 0x7f4c040cffb0 2021-04-09T18:13:40.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.822 7f4c037fe700 1 -- 172.21.15.32:0/3677883760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf400c0c0 con 0x7f4c040cffb0 2021-04-09T18:13:40.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.822 7f4c037fe700 1 -- 172.21.15.32:0/3677883760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bf400c510 con 0x7f4c040cffb0 2021-04-09T18:13:40.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.822 7f4c0a2e2700 1 -- 172.21.15.32:0/3677883760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040cffb0 msgr2=0x7f4c04087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.822 7f4c0a2e2700 1 --2- 172.21.15.32:0/3677883760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040cffb0 0x7f4c04087e40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f4bf4006650 tx=0x7f4bf4009a80).stop 2021-04-09T18:13:40.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.822 7f4c0a2e2700 1 -- 172.21.15.32:0/3677883760 shutdown_connections 2021-04-09T18:13:40.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.822 7f4c0a2e2700 1 --2- 172.21.15.32:0/3677883760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040cffb0 0x7f4c04087e40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:40.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.822 7f4c0a2e2700 1 -- 172.21.15.32:0/3677883760 >> 172.21.15.32:0/3677883760 conn(0x7f4c040ce5b0 msgr2=0x7f4c040cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:40.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.822 7f4c0a2e2700 1 -- 172.21.15.32:0/3677883760 shutdown_connections 2021-04-09T18:13:40.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.822 7f4c0a2e2700 1 -- 172.21.15.32:0/3677883760 wait complete. 2021-04-09T18:13:40.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.842 7f4c0a2e2700 1 Processor -- start 2021-04-09T18:13:40.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c0a2e2700 1 -- start start 2021-04-09T18:13:40.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c0a2e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040c0f30 0x7f4c040a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c0a2e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c040a08e0 0x7f4c040a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c0a2e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c040a5030 0x7f4c040a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:40.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c0a2e2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c040a54b0 con 0x7f4c040c0f30 2021-04-09T18:13:40.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c0a2e2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c040a0ee0 con 0x7f4c040a08e0 2021-04-09T18:13:41.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c0a2e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c040c1330 con 0x7f4c040a5030 2021-04-09T18:13:41.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c03fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040c0f30 0x7f4c040a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:41.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c03fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040c0f30 0x7f4c040a02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42730/0 (socket says 172.21.15.32:42730) 2021-04-09T18:13:41.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c03fff700 1 -- 172.21.15.32:0/3736578372 learned_addr learned my addr 172.21.15.32:0/3736578372 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:41.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4bfbfff700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c040a08e0 0x7f4c040a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:41.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.843 7f4c088ff700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c040a5030 0x7f4c040a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:41.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c03fff700 1 -- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c040a5030 msgr2=0x7f4c040a1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c03fff700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c040a5030 0x7f4c040a1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c03fff700 1 -- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c040a08e0 msgr2=0x7f4c040a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c03fff700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c040a08e0 0x7f4c040a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c03fff700 1 -- 172.21.15.32:0/3736578372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c040a3520 con 0x7f4c040c0f30 2021-04-09T18:13:41.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4bfbfff700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c040a08e0 0x7f4c040a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c03fff700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040c0f30 0x7f4c040a02c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f4bf4009650 tx=0x7f4bf400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:41.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c017fa700 1 -- 172.21.15.32:0/3736578372 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bf40093a0 con 0x7f4c040c0f30 2021-04-09T18:13:41.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c017fa700 1 -- 172.21.15.32:0/3736578372 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf4006070 con 0x7f4c040c0f30 2021-04-09T18:13:41.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c040a3520 con 0x7f4c040c0f30 2021-04-09T18:13:41.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c017fa700 1 -- 172.21.15.32:0/3736578372 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bf40052e0 con 0x7f4c040c0f30 2021-04-09T18:13:41.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.844 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c040a3520 con 0x7f4c040c0f30 2021-04-09T18:13:41.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.845 7f4c017fa700 1 -- 172.21.15.32:0/3736578372 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4bf4009300 con 0x7f4c040c0f30 2021-04-09T18:13:41.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.845 7f4c017fa700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4be80188d0 0x7f4be801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:41.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.845 7f4c017fa700 1 -- 172.21.15.32:0/3736578372 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4bf4032a00 con 0x7f4c040c0f30 2021-04-09T18:13:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.845 7f4bfbfff700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4be80188d0 0x7f4be801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:41.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.845 7f4bfbfff700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4be80188d0 0x7f4be801add0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:41.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.864 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c040a2060 con 0x7f4c040c0f30 2021-04-09T18:13:41.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:40.868 7f4c017fa700 1 -- 172.21.15.32:0/3736578372 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4bf4006030 con 0x7f4c040c0f30 2021-04-09T18:13:41.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.054 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4c040a2060 con 0x7f4c040c0f30 2021-04-09T18:13:41.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.057 7f4c017fa700 1 -- 172.21.15.32:0/3736578372 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4bf4000940 con 0x7f4c040c0f30 2021-04-09T18:13:41.164 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:13:41.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4be80188d0 msgr2=0x7f4be801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4be80188d0 0x7f4be801add0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040c0f30 msgr2=0x7f4c040a02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040c0f30 0x7f4c040a02c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f4bf4009650 tx=0x7f4bf400b8e0).stop 2021-04-09T18:13:41.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 shutdown_connections 2021-04-09T18:13:41.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4be80188d0 0x7f4be801add0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c040a5030 0x7f4c040a1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c040a08e0 0x7f4c040a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 --2- 172.21.15.32:0/3736578372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c040c0f30 0x7f4c040a02c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 >> 172.21.15.32:0/3736578372 conn(0x7f4c040adb50 msgr2=0x7f4c040ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 shutdown_connections 2021-04-09T18:13:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.063 7f4c0a2e2700 1 -- 172.21.15.32:0/3736578372 wait complete. 2021-04-09T18:13:41.254 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:13:41.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.402 7f6704e55700 1 Processor -- start 2021-04-09T18:13:41.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.402 7f6704e55700 1 -- start start 2021-04-09T18:13:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f6704e55700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000cffb0 0x7f6700087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f6704e55700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6700088460 con 0x7f67000d8050 2021-04-09T18:13:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f6704e55700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67000d8620 con 0x7f67000d7c50 2021-04-09T18:13:41.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f6704e55700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67001b0c80 con 0x7f67000cffb0 2021-04-09T18:13:41.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66ff314700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f67000d8050 0x7f670010d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52330/0 (socket says 172.21.15.32:52330) 2021-04-09T18:13:41.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66ff314700 1 -- 172.21.15.32:0/2122987106 learned_addr learned my addr 172.21.15.32:0/2122987106 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:41.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66feb13700 1 --2- 172.21.15.32:0/2122987106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000cffb0 0x7f6700087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:41.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66fdb11700 1 -- 172.21.15.32:0/2122987106 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357603812 0 0) 0x7f66f8000d30 con 0x7f67000d8050 2021-04-09T18:13:41.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66fdb11700 1 -- 172.21.15.32:0/2122987106 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66e8002fe0 con 0x7f67000d8050 2021-04-09T18:13:41.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66fdb11700 1 -- 172.21.15.32:0/2122987106 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820481877 0 0) 0x7f66f4000c60 con 0x7f67000d7c50 2021-04-09T18:13:41.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66fdb11700 1 -- 172.21.15.32:0/2122987106 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66e80042c0 con 0x7f67000d7c50 2021-04-09T18:13:41.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66feb13700 1 -- 172.21.15.32:0/2122987106 >> v1:172.21.15.86:6789/0 conn(0x7f67000d7c50 legacy=0x7f67001b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66feb13700 1 -- 172.21.15.32:0/2122987106 >> v1:172.21.15.32:6789/0 conn(0x7f67000d8050 legacy=0x7f670010d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.403 7f66feb13700 1 -- 172.21.15.32:0/2122987106 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67000d91f0 con 0x7f67000cffb0 2021-04-09T18:13:41.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.404 7f66feb13700 1 --2- 172.21.15.32:0/2122987106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000cffb0 0x7f6700087e40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f66f0006650 tx=0x7f66f0009a00).ready entity=mon.2 client_cookie=c797d594e432594b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:41.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.404 7f66fdb11700 1 -- 172.21.15.32:0/2122987106 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66f0009df0 con 0x7f67000cffb0 2021-04-09T18:13:41.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.404 7f66fdb11700 1 -- 172.21.15.32:0/2122987106 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f00009b0 con 0x7f67000cffb0 2021-04-09T18:13:41.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.404 7f66fdb11700 1 -- 172.21.15.32:0/2122987106 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66f000c2b0 con 0x7f67000cffb0 2021-04-09T18:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.404 7f6704e55700 1 -- 172.21.15.32:0/2122987106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000cffb0 msgr2=0x7f6700087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.404 7f6704e55700 1 --2- 172.21.15.32:0/2122987106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000cffb0 0x7f6700087e40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f66f0006650 tx=0x7f66f0009a00).stop 2021-04-09T18:13:41.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.405 7f6704e55700 1 -- 172.21.15.32:0/2122987106 shutdown_connections 2021-04-09T18:13:41.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.405 7f6704e55700 1 --2- 172.21.15.32:0/2122987106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000cffb0 0x7f6700087e40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.405 7f6704e55700 1 -- 172.21.15.32:0/2122987106 >> 172.21.15.32:0/2122987106 conn(0x7f67000ce5b0 msgr2=0x7f67000cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:41.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.405 7f6704e55700 1 -- 172.21.15.32:0/2122987106 shutdown_connections 2021-04-09T18:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.405 7f6704e55700 1 -- 172.21.15.32:0/2122987106 wait complete. 2021-04-09T18:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f6704e55700 1 Processor -- start 2021-04-09T18:13:41.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f6704e55700 1 -- start start 2021-04-09T18:13:41.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f6704e55700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000c0f30 0x7f67000a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:41.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f6704e55700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000a08e0 0x7f67000a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:41.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f6704e55700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67000a5030 0x7f67000a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:41.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f6704e55700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67000a54b0 con 0x7f67000a5030 2021-04-09T18:13:41.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f6704e55700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67000a0ee0 con 0x7f67000a08e0 2021-04-09T18:13:41.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f6704e55700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67000c1330 con 0x7f67000c0f30 2021-04-09T18:13:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f66ff314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67000a5030 0x7f67000a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:41.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f66ff314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67000a5030 0x7f67000a1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42746/0 (socket says 172.21.15.32:42746) 2021-04-09T18:13:41.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f66ff314700 1 -- 172.21.15.32:0/1956253082 learned_addr learned my addr 172.21.15.32:0/1956253082 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:41.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f66fe312700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000a08e0 0x7f67000a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:41.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.425 7f66feb13700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000c0f30 0x7f67000a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:41.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f66ff314700 1 -- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000c0f30 msgr2=0x7f67000a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f66ff314700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000c0f30 0x7f67000a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f66ff314700 1 -- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000a08e0 msgr2=0x7f67000a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f66ff314700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000a08e0 0x7f67000a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f66ff314700 1 -- 172.21.15.32:0/1956253082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67000a3520 con 0x7f67000a5030 2021-04-09T18:13:41.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f66ff314700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67000a5030 0x7f67000a1460 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f66f8009980 tx=0x7f66f800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:41.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f66ef7fe700 1 -- 172.21.15.32:0/1956253082 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66f800be10 con 0x7f67000a5030 2021-04-09T18:13:41.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f66ef7fe700 1 -- 172.21.15.32:0/1956253082 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f8006030 con 0x7f67000a5030 2021-04-09T18:13:41.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f66ef7fe700 1 -- 172.21.15.32:0/1956253082 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66f80064c0 con 0x7f67000a5030 2021-04-09T18:13:41.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f6704e55700 1 -- 172.21.15.32:0/1956253082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67000a3520 con 0x7f67000a5030 2021-04-09T18:13:41.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.426 7f6704e55700 1 -- 172.21.15.32:0/1956253082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67000a3520 con 0x7f67000a5030 2021-04-09T18:13:41.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.427 7f66ef7fe700 1 -- 172.21.15.32:0/1956253082 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f66f8006210 con 0x7f67000a5030 2021-04-09T18:13:41.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.427 7f66ef7fe700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66e40188c0 0x7f66e401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.427 7f66ef7fe700 1 -- 172.21.15.32:0/1956253082 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f66f8032cc0 con 0x7f67000a5030 2021-04-09T18:13:41.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.427 7f66feb13700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66e40188c0 0x7f66e401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:41.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.428 7f66feb13700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66e40188c0 0x7f66e401adc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:41.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.447 7f6704e55700 1 -- 172.21.15.32:0/1956253082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6700093ff0 con 0x7f67000a5030 2021-04-09T18:13:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.450 7f66ef7fe700 1 -- 172.21.15.32:0/1956253082 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f66f8000f90 con 0x7f67000a5030 2021-04-09T18:13:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.638 7f6704e55700 1 -- 172.21.15.32:0/1956253082 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f67000d8cd0 con 0x7f67000a5030 2021-04-09T18:13:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.640 7f66ef7fe700 1 -- 172.21.15.32:0/1956253082 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f66f8006210 con 0x7f67000a5030 2021-04-09T18:13:41.785 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:13:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 -- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66e40188c0 msgr2=0x7f66e401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66e40188c0 0x7f66e401adc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 -- 172.21.15.32:0/1956253082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67000a5030 msgr2=0x7f67000a1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67000a5030 0x7f67000a1460 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f66f8009980 tx=0x7f66f800bcd0).stop 2021-04-09T18:13:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 -- 172.21.15.32:0/1956253082 shutdown_connections 2021-04-09T18:13:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66e40188c0 0x7f66e401adc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67000a5030 0x7f67000a1460 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000a08e0 0x7f67000a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 --2- 172.21.15.32:0/1956253082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67000c0f30 0x7f67000a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 -- 172.21.15.32:0/1956253082 >> 172.21.15.32:0/1956253082 conn(0x7f67000adb50 msgr2=0x7f67000ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 -- 172.21.15.32:0/1956253082 shutdown_connections 2021-04-09T18:13:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.643 7f6704e55700 1 -- 172.21.15.32:0/1956253082 wait complete. 2021-04-09T18:13:41.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.959 7f7a3ec6e700 1 Processor -- start 2021-04-09T18:13:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.959 7f7a3ec6e700 1 -- start start 2021-04-09T18:13:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a3ec6e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7a380d7ca0 0x7f7a380d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a3ec6e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3810d480 con 0x7f7a3810d010 2021-04-09T18:13:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a3ec6e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a380d86c0 con 0x7f7a3810da00 2021-04-09T18:13:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a3ec6e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a380d9220 con 0x7f7a380d7ca0 2021-04-09T18:13:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a3ca8a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7a3810d010 0x7f7a380d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52342/0 (socket says 172.21.15.32:52342) 2021-04-09T18:13:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a3ca8a700 1 -- 172.21.15.32:0/3705079067 learned_addr learned my addr 172.21.15.32:0/3705079067 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a37fff700 1 --2- 172.21.15.32:0/3705079067 >> v2:172.21.15.86:3300/0 conn(0x7f7a380d7ca0 0x7f7a380d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a377fe700 1 -- 172.21.15.32:0/3705079067 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864878148 0 0) 0x7f7a28000ce0 con 0x7f7a3810d010 2021-04-09T18:13:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a377fe700 1 -- 172.21.15.32:0/3705079067 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a20002fe0 con 0x7f7a3810d010 2021-04-09T18:13:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a377fe700 1 -- 172.21.15.32:0/3705079067 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134229793 0 0) 0x7f7a30001400 con 0x7f7a3810da00 2021-04-09T18:13:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a377fe700 1 -- 172.21.15.32:0/3705079067 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a20003210 con 0x7f7a3810da00 2021-04-09T18:13:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a377fe700 1 -- 172.21.15.32:0/3705079067 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 229392212 0 0) 0x7f7a280010e0 con 0x7f7a3810d010 2021-04-09T18:13:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a37fff700 1 -- 172.21.15.32:0/3705079067 >> v1:172.21.15.86:6789/0 conn(0x7f7a3810da00 legacy=0x7f7a3810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a37fff700 1 -- 172.21.15.32:0/3705079067 >> v1:172.21.15.32:6789/0 conn(0x7f7a3810d010 legacy=0x7f7a380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a37fff700 1 -- 172.21.15.32:0/3705079067 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a380d09c0 con 0x7f7a380d7ca0 2021-04-09T18:13:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.960 7f7a37fff700 1 --2- 172.21.15.32:0/3705079067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a380d7ca0 0x7f7a380d80a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7a2c006650 tx=0x7f7a2c009a00).ready entity=mon.1 client_cookie=5a168e1c153ecb19 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a377fe700 1 -- 172.21.15.32:0/3705079067 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a2c009df0 con 0x7f7a380d7ca0 2021-04-09T18:13:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a377fe700 1 -- 172.21.15.32:0/3705079067 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c0009b0 con 0x7f7a380d7ca0 2021-04-09T18:13:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a377fe700 1 -- 172.21.15.32:0/3705079067 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a2c00c2b0 con 0x7f7a380d7ca0 2021-04-09T18:13:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a3ec6e700 1 -- 172.21.15.32:0/3705079067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a380d7ca0 msgr2=0x7f7a380d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:42.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a3ec6e700 1 --2- 172.21.15.32:0/3705079067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a380d7ca0 0x7f7a380d80a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7a2c006650 tx=0x7f7a2c009a00).stop 2021-04-09T18:13:42.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a3ec6e700 1 -- 172.21.15.32:0/3705079067 shutdown_connections 2021-04-09T18:13:42.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a3ec6e700 1 --2- 172.21.15.32:0/3705079067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a380d7ca0 0x7f7a380d80a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a3ec6e700 1 -- 172.21.15.32:0/3705079067 >> 172.21.15.32:0/3705079067 conn(0x7f7a380ce5f0 msgr2=0x7f7a380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a3ec6e700 1 -- 172.21.15.32:0/3705079067 shutdown_connections 2021-04-09T18:13:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.961 7f7a3ec6e700 1 -- 172.21.15.32:0/3705079067 wait complete. 2021-04-09T18:13:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.981 7f7a3ec6e700 1 Processor -- start 2021-04-09T18:13:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a3ec6e700 1 -- start start 2021-04-09T18:13:42.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a3ec6e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a381abcb0 0x7f7a381ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:42.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a3ec6e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a38191140 0x7f7a38191540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:42.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a3ec6e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a38191b60 0x7f7a381a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:42.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a3ec6e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a38191f60 con 0x7f7a38191140 2021-04-09T18:13:42.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a3ec6e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a381ac6d0 con 0x7f7a38191b60 2021-04-09T18:13:42.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a3ec6e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a381aa5c0 con 0x7f7a381abcb0 2021-04-09T18:13:42.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a38191140 0x7f7a38191540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:42.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a38191140 0x7f7a38191540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42760/0 (socket says 172.21.15.32:42760) 2021-04-09T18:13:42.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a37fff700 1 -- 172.21.15.32:0/2434742925 learned_addr learned my addr 172.21.15.32:0/2434742925 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:42.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a3ca8a700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a381abcb0 0x7f7a381ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:42.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.982 7f7a3d28b700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a38191b60 0x7f7a381a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a37fff700 1 -- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a381abcb0 msgr2=0x7f7a381ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a37fff700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a381abcb0 0x7f7a381ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a37fff700 1 -- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a38191b60 msgr2=0x7f7a381a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a37fff700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a38191b60 0x7f7a381a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a37fff700 1 -- 172.21.15.32:0/2434742925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a381aa7e0 con 0x7f7a38191140 2021-04-09T18:13:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a37fff700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a38191140 0x7f7a38191540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7a2c0096a0 tx=0x7f7a2c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a357fa700 1 -- 172.21.15.32:0/2434742925 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a2c000c90 con 0x7f7a38191140 2021-04-09T18:13:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a357fa700 1 -- 172.21.15.32:0/2434742925 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c002030 con 0x7f7a38191140 2021-04-09T18:13:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a381aa7e0 con 0x7f7a38191140 2021-04-09T18:13:42.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a357fa700 1 -- 172.21.15.32:0/2434742925 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a2c002550 con 0x7f7a38191140 2021-04-09T18:13:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.983 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a381aa7e0 con 0x7f7a38191140 2021-04-09T18:13:42.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.984 7f7a357fa700 1 -- 172.21.15.32:0/2434742925 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7a2c002210 con 0x7f7a38191140 2021-04-09T18:13:42.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.984 7f7a357fa700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7a18018930 0x7f7a1801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:42.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.984 7f7a357fa700 1 -- 172.21.15.32:0/2434742925 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7a2c032a40 con 0x7f7a38191140 2021-04-09T18:13:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.984 7f7a3ca8a700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7a18018930 0x7f7a1801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:41.984 7f7a3ca8a700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7a18018930 0x7f7a1801ae10 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.004 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a38191f60 con 0x7f7a38191140 2021-04-09T18:13:42.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.007 7f7a357fa700 1 -- 172.21.15.32:0/2434742925 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7a2c0009c0 con 0x7f7a38191140 2021-04-09T18:13:42.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.198 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a38191f60 con 0x7f7a38191140 2021-04-09T18:13:42.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.198 7f7a357fa700 1 -- 172.21.15.32:0/2434742925 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2776 (secure 0 0 0) 0x7f7a2c00bd80 con 0x7f7a38191140 2021-04-09T18:13:42.285 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 13 2021-04-09T18:13:42.289 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:42.299 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:39.509220","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:13:42.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7a18018930 msgr2=0x7f7a1801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:42.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7a18018930 0x7f7a1801ae10 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:42.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a38191140 msgr2=0x7f7a38191540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:42.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a38191140 0x7f7a38191540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7a2c0096a0 tx=0x7f7a2c00b990).stop 2021-04-09T18:13:42.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 shutdown_connections 2021-04-09T18:13:42.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7a18018930 0x7f7a1801ae10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a38191140 0x7f7a38191540 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:42.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a38191b60 0x7f7a381a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:42.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 --2- 172.21.15.32:0/2434742925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a381abcb0 0x7f7a381ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:42.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 >> 172.21.15.32:0/2434742925 conn(0x7f7a381837d0 msgr2=0x7f7a38195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:42.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 shutdown_connections 2021-04-09T18:13:42.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:42.201 7f7a3ec6e700 1 -- 172.21.15.32:0/2434742925 wait complete. 2021-04-09T18:13:42.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:39.509220', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:13:42.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:13:42.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:13:42.945 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:13:42.946 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:13:42.946 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:13:42.947 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 4693, 2021-04-09T18:13:42.947 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:13:42.948 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:13:42.948 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 6.0332842859999998, 2021-04-09T18:13:42.948 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 13, 2021-04-09T18:13:42.949 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 32, 2021-04-09T18:13:42.949 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 32, 2021-04-09T18:13:42.950 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 15.139791376 2021-04-09T18:13:42.950 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:13:42.955 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 4693, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.0332842859999998, "mdsmap_epoch": 13, "osdmap_epoch": 32, "osdmap_epoch_barrier": 32, "uptime": 15.139791376 } 2021-04-09T18:13:42.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:13:43.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.084 7f4370c54700 1 Processor -- start 2021-04-09T18:13:43.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.084 7f4370c54700 1 -- start start 2021-04-09T18:13:43.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.085 7f4370c54700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f436c0cffa0 0x7f436c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.085 7f4370c54700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f436c0d6c80 0x7f436c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.085 7f4370c54700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c0d7280 con 0x7f436c0d7800 2021-04-09T18:13:43.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.085 7f4370c54700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f436c0d7080 con 0x7f436c0cffa0 2021-04-09T18:13:43.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.085 7f4370c54700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f436c0d91e0 con 0x7f436c0d6c80 2021-04-09T18:13:43.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.085 7f436a85e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f436c0cffa0 0x7f436c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.085 7f436a05d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f436c0d7800 0x7f436c0d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52356/0 (socket says 172.21.15.32:52356) 2021-04-09T18:13:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.085 7f436a05d700 1 -- 172.21.15.32:0/2001949455 learned_addr learned my addr 172.21.15.32:0/2001949455 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:43.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.085 7f436b05f700 1 --2- 172.21.15.32:0/2001949455 >> v2:172.21.15.86:3300/0 conn(0x7f436c0d6c80 0x7f436c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436985c700 1 -- 172.21.15.32:0/2001949455 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604976001 0 0) 0x7f435c001060 con 0x7f436c0d7800 2021-04-09T18:13:43.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436985c700 1 -- 172.21.15.32:0/2001949455 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4358002fe0 con 0x7f436c0d7800 2021-04-09T18:13:43.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436a85e700 1 -- 172.21.15.32:0/2001949455 >> v2:172.21.15.86:3300/0 conn(0x7f436c0d6c80 msgr2=0x7f436c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:43.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436a85e700 1 --2- 172.21.15.32:0/2001949455 >> v2:172.21.15.86:3300/0 conn(0x7f436c0d6c80 0x7f436c10cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436a85e700 1 -- 172.21.15.32:0/2001949455 >> v1:172.21.15.32:6789/0 conn(0x7f436c0d7800 legacy=0x7f436c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436a85e700 1 -- 172.21.15.32:0/2001949455 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c0d83f0 con 0x7f436c0cffa0 2021-04-09T18:13:43.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436a85e700 1 --2- 172.21.15.32:0/2001949455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c0cffa0 0x7f436c0d2440 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f4354006650 tx=0x7f4354009a80).ready entity=mon.0 client_cookie=f1d5b93ec5969369 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:43.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436985c700 1 -- 172.21.15.32:0/2001949455 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43540009b0 con 0x7f436c0cffa0 2021-04-09T18:13:43.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436985c700 1 -- 172.21.15.32:0/2001949455 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435400c0c0 con 0x7f436c0cffa0 2021-04-09T18:13:43.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f436985c700 1 -- 172.21.15.32:0/2001949455 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f435400c510 con 0x7f436c0cffa0 2021-04-09T18:13:43.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f4370c54700 1 -- 172.21.15.32:0/2001949455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c0cffa0 msgr2=0x7f436c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.086 7f4370c54700 1 --2- 172.21.15.32:0/2001949455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c0cffa0 0x7f436c0d2440 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f4354006650 tx=0x7f4354009a80).stop 2021-04-09T18:13:43.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.087 7f4370c54700 1 -- 172.21.15.32:0/2001949455 shutdown_connections 2021-04-09T18:13:43.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.087 7f4370c54700 1 --2- 172.21.15.32:0/2001949455 >> v2:172.21.15.86:3300/0 conn(0x7f436c0d6c80 0x7f436c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.087 7f4370c54700 1 --2- 172.21.15.32:0/2001949455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c0cffa0 0x7f436c0d2440 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.087 7f4370c54700 1 -- 172.21.15.32:0/2001949455 >> 172.21.15.32:0/2001949455 conn(0x7f436c0ce5c0 msgr2=0x7f436c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:43.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.087 7f4370c54700 1 -- 172.21.15.32:0/2001949455 shutdown_connections 2021-04-09T18:13:43.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.087 7f4370c54700 1 -- 172.21.15.32:0/2001949455 wait complete. 2021-04-09T18:13:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.107 7f4370c54700 1 Processor -- start 2021-04-09T18:13:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.107 7f4370c54700 1 -- start start 2021-04-09T18:13:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f4370c54700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f436c1acff0 0x7f436c08fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f4370c54700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f436c090150 0x7f436c08c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f4370c54700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c08cca0 0x7f436c08e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f4370c54700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c08c100 con 0x7f436c08cca0 2021-04-09T18:13:43.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f4370c54700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c08d0a0 con 0x7f436c090150 2021-04-09T18:13:43.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f4370c54700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436c08e960 con 0x7f436c1acff0 2021-04-09T18:13:43.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436b05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c08cca0 0x7f436c08e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436b05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c08cca0 0x7f436c08e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42772/0 (socket says 172.21.15.32:42772) 2021-04-09T18:13:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436b05f700 1 -- 172.21.15.32:0/3972366119 learned_addr learned my addr 172.21.15.32:0/3972366119 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436a05d700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f436c090150 0x7f436c08c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436a85e700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f436c1acff0 0x7f436c08fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436b05f700 1 -- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f436c1acff0 msgr2=0x7f436c08fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436b05f700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f436c1acff0 0x7f436c08fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436b05f700 1 -- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f436c090150 msgr2=0x7f436c08c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436b05f700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f436c090150 0x7f436c08c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436b05f700 1 -- 172.21.15.32:0/3972366119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c08eb80 con 0x7f436c08cca0 2021-04-09T18:13:43.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f436b05f700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c08cca0 0x7f436c08e340 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f436400b580 tx=0x7f436400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:43.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f43637fe700 1 -- 172.21.15.32:0/3972366119 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f436400bd10 con 0x7f436c08cca0 2021-04-09T18:13:43.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f43637fe700 1 -- 172.21.15.32:0/3972366119 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364006720 con 0x7f436c08cca0 2021-04-09T18:13:43.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f4370c54700 1 -- 172.21.15.32:0/3972366119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c08eb80 con 0x7f436c08cca0 2021-04-09T18:13:43.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f43637fe700 1 -- 172.21.15.32:0/3972366119 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43640024d0 con 0x7f436c08cca0 2021-04-09T18:13:43.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.108 7f4370c54700 1 -- 172.21.15.32:0/3972366119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c08eb80 con 0x7f436c08cca0 2021-04-09T18:13:43.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.109 7f43637fe700 1 -- 172.21.15.32:0/3972366119 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4364005330 con 0x7f436c08cca0 2021-04-09T18:13:43.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.109 7f43637fe700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4350018900 0x7f435001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.109 7f43637fe700 1 -- 172.21.15.32:0/3972366119 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4364032b80 con 0x7f436c08cca0 2021-04-09T18:13:43.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.109 7f436a85e700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4350018900 0x7f435001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.110 7f436a85e700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4350018900 0x7f435001adc0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:43.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.129 7f4370c54700 1 -- 172.21.15.32:0/3972366119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c182dc0 con 0x7f436c08cca0 2021-04-09T18:13:43.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.133 7f43637fe700 1 -- 172.21.15.32:0/3972366119 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4364006370 con 0x7f436c08cca0 2021-04-09T18:13:43.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.321 7f4370c54700 1 -- 172.21.15.32:0/3972366119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f436c0d7280 con 0x7f436c08cca0 2021-04-09T18:13:43.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.322 7f43637fe700 1 -- 172.21.15.32:0/3972366119 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f4364000e70 con 0x7f436c08cca0 2021-04-09T18:13:43.362 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:43.371 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":32,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:13:37.891009","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:13:35.174131","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-04-09 18:13:35.938524","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:13:43.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 -- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4350018900 msgr2=0x7f435001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4350018900 0x7f435001adc0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 -- 172.21.15.32:0/3972366119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c08cca0 msgr2=0x7f436c08e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c08cca0 0x7f436c08e340 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f436400b580 tx=0x7f436400b920).stop 2021-04-09T18:13:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 -- 172.21.15.32:0/3972366119 shutdown_connections 2021-04-09T18:13:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4350018900 0x7f435001adc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c08cca0 0x7f436c08e340 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f436c090150 0x7f436c08c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 --2- 172.21.15.32:0/3972366119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f436c1acff0 0x7f436c08fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 -- 172.21.15.32:0/3972366119 >> 172.21.15.32:0/3972366119 conn(0x7f436c09abb0 msgr2=0x7f436c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 -- 172.21.15.32:0/3972366119 shutdown_connections 2021-04-09T18:13:43.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.324 7f4370c54700 1 -- 172.21.15.32:0/3972366119 wait complete. 2021-04-09T18:13:43.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:13:35.174131', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:13:35.938524', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:13:43.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:13:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.662 7f4ea2ac2700 1 Processor -- start 2021-04-09T18:13:43.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.662 7f4ea2ac2700 1 -- start start 2021-04-09T18:13:43.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4ea2ac2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c10d010 0x7f4e9c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4ea2ac2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4e9c0d1da0 0x7f4e9c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4ea2ac2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9c10d480 con 0x7f4e9c0d82c0 2021-04-09T18:13:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4ea2ac2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9c10e780 con 0x7f4e9c0d1da0 2021-04-09T18:13:43.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4ea2ac2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0d8d30 con 0x7f4e9c10d010 2021-04-09T18:13:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4ea10df700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4e9c0d82c0 0x7f4e9c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52368/0 (socket says 172.21.15.32:52368) 2021-04-09T18:13:43.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4ea10df700 1 -- 172.21.15.32:0/2879291175 learned_addr learned my addr 172.21.15.32:0/2879291175 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4e9bfff700 1 --2- 172.21.15.32:0/2879291175 >> v2:172.21.15.86:3300/0 conn(0x7f4e9c0d1da0 0x7f4e9c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4ea08de700 1 --2- 172.21.15.32:0/2879291175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c10d010 0x7f4e9c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4e9b7fe700 1 -- 172.21.15.32:0/2879291175 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582577286 0 0) 0x7f4e94000d30 con 0x7f4e9c0d82c0 2021-04-09T18:13:43.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4e9b7fe700 1 -- 172.21.15.32:0/2879291175 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e88002fe0 con 0x7f4e9c0d82c0 2021-04-09T18:13:43.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4e9bfff700 1 -- 172.21.15.32:0/2879291175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c10d010 msgr2=0x7f4e9c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:43.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4e9bfff700 1 --2- 172.21.15.32:0/2879291175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c10d010 0x7f4e9c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4e9bfff700 1 -- 172.21.15.32:0/2879291175 >> v1:172.21.15.32:6789/0 conn(0x7f4e9c0d82c0 legacy=0x7f4e9c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4e9bfff700 1 -- 172.21.15.32:0/2879291175 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c0d9220 con 0x7f4e9c0d1da0 2021-04-09T18:13:43.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4ea08de700 1 --2- 172.21.15.32:0/2879291175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c10d010 0x7f4e9c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:43.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4e9bfff700 1 --2- 172.21.15.32:0/2879291175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e9c0d1da0 0x7f4e9c0d7ca0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4e8c006650 tx=0x7f4e8c009a80).ready entity=mon.1 client_cookie=3c746e740e2e4c7d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:43.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.663 7f4e9b7fe700 1 -- 172.21.15.32:0/2879291175 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e8c0009b0 con 0x7f4e9c0d1da0 2021-04-09T18:13:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4e9b7fe700 1 -- 172.21.15.32:0/2879291175 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c00c0c0 con 0x7f4e9c0d1da0 2021-04-09T18:13:43.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4e9b7fe700 1 -- 172.21.15.32:0/2879291175 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e8c00c510 con 0x7f4e9c0d1da0 2021-04-09T18:13:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4ea2ac2700 1 -- 172.21.15.32:0/2879291175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e9c0d1da0 msgr2=0x7f4e9c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4ea2ac2700 1 --2- 172.21.15.32:0/2879291175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e9c0d1da0 0x7f4e9c0d7ca0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f4e8c006650 tx=0x7f4e8c009a80).stop 2021-04-09T18:13:43.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4ea2ac2700 1 -- 172.21.15.32:0/2879291175 shutdown_connections 2021-04-09T18:13:43.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4ea2ac2700 1 --2- 172.21.15.32:0/2879291175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e9c0d1da0 0x7f4e9c0d7ca0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4ea2ac2700 1 --2- 172.21.15.32:0/2879291175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c10d010 0x7f4e9c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4ea2ac2700 1 -- 172.21.15.32:0/2879291175 >> 172.21.15.32:0/2879291175 conn(0x7f4e9c0ce5f0 msgr2=0x7f4e9c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:43.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4ea2ac2700 1 -- 172.21.15.32:0/2879291175 shutdown_connections 2021-04-09T18:13:43.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.664 7f4ea2ac2700 1 -- 172.21.15.32:0/2879291175 wait complete. 2021-04-09T18:13:43.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.684 7f4ea2ac2700 1 Processor -- start 2021-04-09T18:13:43.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4ea2ac2700 1 -- start start 2021-04-09T18:13:43.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4ea2ac2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c0b1e50 0x7f4e9c0b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4ea2ac2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e9c0b9fd0 0x7f4e9c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4ea2ac2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e9c0b5510 0x7f4e9c0b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:43.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4ea2ac2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0b2250 con 0x7f4e9c0b9fd0 2021-04-09T18:13:43.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4ea2ac2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0a1220 con 0x7f4e9c0b5510 2021-04-09T18:13:43.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4ea2ac2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0b6b30 con 0x7f4e9c0b1e50 2021-04-09T18:13:43.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4e9bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e9c0b9fd0 0x7f4e9c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4e9bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e9c0b9fd0 0x7f4e9c0b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42782/0 (socket says 172.21.15.32:42782) 2021-04-09T18:13:43.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4e9bfff700 1 -- 172.21.15.32:0/1481663964 learned_addr learned my addr 172.21.15.32:0/1481663964 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:43.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4ea08de700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c0b1e50 0x7f4e9c0b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.685 7f4ea10df700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e9c0b5510 0x7f4e9c0b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4e9bfff700 1 -- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c0b1e50 msgr2=0x7f4e9c0b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4e9bfff700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c0b1e50 0x7f4e9c0b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4e9bfff700 1 -- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e9c0b5510 msgr2=0x7f4e9c0b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:43.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4e9bfff700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e9c0b5510 0x7f4e9c0b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:43.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4e9bfff700 1 -- 172.21.15.32:0/1481663964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c0b5f30 con 0x7f4e9c0b9fd0 2021-04-09T18:13:43.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4ea08de700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c0b1e50 0x7f4e9c0b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:43.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4e9bfff700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e9c0b9fd0 0x7f4e9c0b4ef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f4e8c009650 tx=0x7f4e8c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:43.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4e997fa700 1 -- 172.21.15.32:0/1481663964 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e8c0093a0 con 0x7f4e9c0b9fd0 2021-04-09T18:13:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4e997fa700 1 -- 172.21.15.32:0/1481663964 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c00bd50 con 0x7f4e9c0b9fd0 2021-04-09T18:13:43.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9c0b5f30 con 0x7f4e9c0b9fd0 2021-04-09T18:13:43.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4e997fa700 1 -- 172.21.15.32:0/1481663964 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e8c0052e0 con 0x7f4e9c0b9fd0 2021-04-09T18:13:43.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.686 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9c0b5f30 con 0x7f4e9c0b9fd0 2021-04-09T18:13:43.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.687 7f4e997fa700 1 -- 172.21.15.32:0/1481663964 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4e8c006030 con 0x7f4e9c0b9fd0 2021-04-09T18:13:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.687 7f4e997fa700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e80018930 0x7f4e8001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.687 7f4e997fa700 1 -- 172.21.15.32:0/1481663964 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4e8c032a00 con 0x7f4e9c0b9fd0 2021-04-09T18:13:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.687 7f4ea08de700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e80018930 0x7f4e8001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.687 7f4ea08de700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e80018930 0x7f4e8001adc0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.706 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9c1b7040 con 0x7f4e9c0b9fd0 2021-04-09T18:13:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.710 7f4e997fa700 1 -- 172.21.15.32:0/1481663964 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4e8c0009c0 con 0x7f4e9c0b9fd0 2021-04-09T18:13:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.887 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f4e9c1b7040 con 0x7f4e9c0b9fd0 2021-04-09T18:13:44.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.888 7f4e997fa700 1 -- 172.21.15.32:0/1481663964 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v32) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4e8c00bd50 con 0x7f4e9c0b9fd0 2021-04-09T18:13:44.051 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e80018930 msgr2=0x7f4e8001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:44.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e80018930 0x7f4e8001adc0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e9c0b9fd0 msgr2=0x7f4e9c0b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:44.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e9c0b9fd0 0x7f4e9c0b4ef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f4e8c009650 tx=0x7f4e8c00b960).stop 2021-04-09T18:13:44.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 shutdown_connections 2021-04-09T18:13:44.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e80018930 0x7f4e8001adc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e9c0b9fd0 0x7f4e9c0b4ef0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e9c0b5510 0x7f4e9c0b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 --2- 172.21.15.32:0/1481663964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e9c0b1e50 0x7f4e9c0b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 >> 172.21.15.32:0/1481663964 conn(0x7f4e9c0b27e0 msgr2=0x7f4e9c1b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:44.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 shutdown_connections 2021-04-09T18:13:44.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:43.890 7f4ea2ac2700 1 -- 172.21.15.32:0/1481663964 wait complete. 2021-04-09T18:13:44.087 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:13:44.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.243 7f6ad6ab3700 1 Processor -- start 2021-04-09T18:13:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.243 7f6ad6ab3700 1 -- start start 2021-04-09T18:13:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad6ab3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6ad00d7ca0 0x7f6ad00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:44.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad6ab3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6ad010da00 0x7f6ad010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:44.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad6ab3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad010d480 con 0x7f6ad010d010 2021-04-09T18:13:44.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad6ab3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad00d9220 con 0x7f6ad010da00 2021-04-09T18:13:44.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad6ab3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad00d86c0 con 0x7f6ad00d7ca0 2021-04-09T18:13:44.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad50d0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6ad010da00 0x7f6ad010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:44.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad50d0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6ad010da00 0x7f6ad010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42792/0 (socket says 172.21.15.32:42792) 2021-04-09T18:13:44.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad50d0700 1 -- 172.21.15.32:0/3602458629 learned_addr learned my addr 172.21.15.32:0/3602458629 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:44.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6acffff700 1 --2- 172.21.15.32:0/3602458629 >> v2:172.21.15.86:3300/0 conn(0x7f6ad00d7ca0 0x7f6ad00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:44.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad50d0700 1 -- 172.21.15.32:0/3602458629 >> v2:172.21.15.86:3300/0 conn(0x7f6ad00d7ca0 msgr2=0x7f6ad00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:44.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad50d0700 1 --2- 172.21.15.32:0/3602458629 >> v2:172.21.15.86:3300/0 conn(0x7f6ad00d7ca0 0x7f6ad00d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad50d0700 1 -- 172.21.15.32:0/3602458629 >> v1:172.21.15.86:6789/0 conn(0x7f6ad010d010 legacy=0x7f6ad00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:44.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad50d0700 1 -- 172.21.15.32:0/3602458629 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad00d1f70 con 0x7f6ad010da00 2021-04-09T18:13:44.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6acf7fe700 1 -- 172.21.15.32:0/3602458629 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879978998 0 0) 0x7f6ac0000c60 con 0x7f6ad010d010 2021-04-09T18:13:44.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.244 7f6ad50d0700 1 --2- 172.21.15.32:0/3602458629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad010da00 0x7f6ad010de00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f6ac8006f00 tx=0x7f6ac8009e90).ready entity=mon.0 client_cookie=66569d14676c96a5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:44.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6acf7fe700 1 -- 172.21.15.32:0/3602458629 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ac800b300 con 0x7f6ad010da00 2021-04-09T18:13:44.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6acf7fe700 1 -- 172.21.15.32:0/3602458629 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac800b640 con 0x7f6ad010da00 2021-04-09T18:13:44.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6acf7fe700 1 -- 172.21.15.32:0/3602458629 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ac8000f90 con 0x7f6ad010da00 2021-04-09T18:13:44.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6ad6ab3700 1 -- 172.21.15.32:0/3602458629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad010da00 msgr2=0x7f6ad010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:44.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6ad6ab3700 1 --2- 172.21.15.32:0/3602458629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad010da00 0x7f6ad010de00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f6ac8006f00 tx=0x7f6ac8009e90).stop 2021-04-09T18:13:44.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6ad6ab3700 1 -- 172.21.15.32:0/3602458629 shutdown_connections 2021-04-09T18:13:44.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6ad6ab3700 1 --2- 172.21.15.32:0/3602458629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad010da00 0x7f6ad010de00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6ad6ab3700 1 --2- 172.21.15.32:0/3602458629 >> v2:172.21.15.86:3300/0 conn(0x7f6ad00d7ca0 0x7f6ad00d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6ad6ab3700 1 -- 172.21.15.32:0/3602458629 >> 172.21.15.32:0/3602458629 conn(0x7f6ad00ce5f0 msgr2=0x7f6ad00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:44.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6ad6ab3700 1 -- 172.21.15.32:0/3602458629 shutdown_connections 2021-04-09T18:13:44.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.245 7f6ad6ab3700 1 -- 172.21.15.32:0/3602458629 wait complete. 2021-04-09T18:13:44.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.265 7f6ad6ab3700 1 Processor -- start 2021-04-09T18:13:44.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.265 7f6ad6ab3700 1 -- start start 2021-04-09T18:13:44.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad6ab3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad00b1e50 0x7f6ad00b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:44.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad6ab3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad00b9fd0 0x7f6ad00b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:44.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad6ab3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad00b5510 0x7f6ad00b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:44.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad6ab3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad00b2250 con 0x7f6ad00b1e50 2021-04-09T18:13:44.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad6ab3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad00a1220 con 0x7f6ad00b5510 2021-04-09T18:13:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad6ab3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad00b6b30 con 0x7f6ad00b9fd0 2021-04-09T18:13:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad48cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad00b1e50 0x7f6ad00b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad48cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad00b1e50 0x7f6ad00b99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42800/0 (socket says 172.21.15.32:42800) 2021-04-09T18:13:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad48cf700 1 -- 172.21.15.32:0/1558520877 learned_addr learned my addr 172.21.15.32:0/1558520877 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:44.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6ad50d0700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad00b5510 0x7f6ad00b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:44.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.266 7f6acffff700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad00b9fd0 0x7f6ad00b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6ad48cf700 1 -- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad00b9fd0 msgr2=0x7f6ad00b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:44.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6ad48cf700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad00b9fd0 0x7f6ad00b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6ad48cf700 1 -- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad00b5510 msgr2=0x7f6ad00b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:44.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6ad48cf700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad00b5510 0x7f6ad00b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6ad48cf700 1 -- 172.21.15.32:0/1558520877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad00b5f30 con 0x7f6ad00b1e50 2021-04-09T18:13:44.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6ad50d0700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad00b5510 0x7f6ad00b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:13:44.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6ad48cf700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad00b1e50 0x7f6ad00b99b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f6ac000b580 tx=0x7f6ac000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:44.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6acd7fa700 1 -- 172.21.15.32:0/1558520877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ac000bd10 con 0x7f6ad00b1e50 2021-04-09T18:13:44.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6acd7fa700 1 -- 172.21.15.32:0/1558520877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac0001080 con 0x7f6ad00b1e50 2021-04-09T18:13:44.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad00b5f30 con 0x7f6ad00b1e50 2021-04-09T18:13:44.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6acd7fa700 1 -- 172.21.15.32:0/1558520877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ac0001550 con 0x7f6ad00b1e50 2021-04-09T18:13:44.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.267 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad00b5f30 con 0x7f6ad00b1e50 2021-04-09T18:13:44.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.268 7f6acd7fa700 1 -- 172.21.15.32:0/1558520877 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6ac0001260 con 0x7f6ad00b1e50 2021-04-09T18:13:44.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.268 7f6acd7fa700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ab00188c0 0x7f6ab001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:44.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.268 7f6acd7fa700 1 -- 172.21.15.32:0/1558520877 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6ac00327f0 con 0x7f6ad00b1e50 2021-04-09T18:13:44.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.268 7f6acffff700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ab00188c0 0x7f6ab001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:44.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.268 7f6acffff700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ab00188c0 0x7f6ab001adc0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:44.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.288 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad01b7040 con 0x7f6ad00b1e50 2021-04-09T18:13:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.291 7f6acd7fa700 1 -- 172.21.15.32:0/1558520877 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6ac0028f40 con 0x7f6ad00b1e50 2021-04-09T18:13:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.472 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f6ad01b7040 con 0x7f6ad00b1e50 2021-04-09T18:13:44.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.472 7f6acd7fa700 1 -- 172.21.15.32:0/1558520877 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v32) v1 ==== 104+0+10 (secure 0 0 0) 0x7f6ac0028f40 con 0x7f6ad00b1e50 2021-04-09T18:13:44.614 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:44.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.474 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ab00188c0 msgr2=0x7f6ab001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.474 7f6ad6ab3700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ab00188c0 0x7f6ab001adc0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.474 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad00b1e50 msgr2=0x7f6ad00b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:44.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.474 7f6ad6ab3700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad00b1e50 0x7f6ad00b99b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f6ac000b580 tx=0x7f6ac000b920).stop 2021-04-09T18:13:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.475 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 shutdown_connections 2021-04-09T18:13:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.475 7f6ad6ab3700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ab00188c0 0x7f6ab001adc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.475 7f6ad6ab3700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ad00b5510 0x7f6ad00b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.475 7f6ad6ab3700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ad00b9fd0 0x7f6ad00b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.475 7f6ad6ab3700 1 --2- 172.21.15.32:0/1558520877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ad00b1e50 0x7f6ad00b99b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.475 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 >> 172.21.15.32:0/1558520877 conn(0x7f6ad00b27e0 msgr2=0x7f6ad01b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:44.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.475 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 shutdown_connections 2021-04-09T18:13:44.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.475 7f6ad6ab3700 1 -- 172.21.15.32:0/1558520877 wait complete. 2021-04-09T18:13:44.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:13:45.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.955 7f91a0c24700 1 Processor -- start 2021-04-09T18:13:45.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.955 7f91a0c24700 1 -- start start 2021-04-09T18:13:45.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f91a0c24700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c0d7ca0 0x7f919c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f91a0c24700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f919c10da00 0x7f919c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f91a0c24700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c10d480 con 0x7f919c10d010 2021-04-09T18:13:45.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f91a0c24700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f919c0d9220 con 0x7f919c10da00 2021-04-09T18:13:45.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f91a0c24700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919c112100 con 0x7f919c0d7ca0 2021-04-09T18:13:45.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f919a05d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c0d7ca0 0x7f919c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f919b05f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f919c10da00 0x7f919c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f919a85e700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f919c10d010 0x7f919c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:50320/0 (socket says 172.21.15.32:50320) 2021-04-09T18:13:45.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f919a85e700 1 -- 172.21.15.32:0/1416767842 learned_addr learned my addr 172.21.15.32:0/1416767842 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:45.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f919a05d700 1 -- 172.21.15.32:0/1416767842 >> v2:172.21.15.86:3300/0 conn(0x7f919c10da00 msgr2=0x7f919c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:45.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f919a05d700 1 --2- 172.21.15.32:0/1416767842 >> v2:172.21.15.86:3300/0 conn(0x7f919c10da00 0x7f919c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.956 7f919a05d700 1 -- 172.21.15.32:0/1416767842 >> v1:172.21.15.86:6789/0 conn(0x7f919c10d010 legacy=0x7f919c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.957 7f919a05d700 1 -- 172.21.15.32:0/1416767842 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c0d86c0 con 0x7f919c0d7ca0 2021-04-09T18:13:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.957 7f919a05d700 1 --2- 172.21.15.32:0/1416767842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c0d7ca0 0x7f919c0d80a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f91900069d0 tx=0x7f9190009e40).ready entity=mon.2 client_cookie=8c0048b31d4af7e4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.957 7f919985c700 1 -- 172.21.15.32:0/1416767842 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f919000b510 con 0x7f919c0d7ca0 2021-04-09T18:13:45.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.957 7f919985c700 1 -- 172.21.15.32:0/1416767842 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919000b890 con 0x7f919c0d7ca0 2021-04-09T18:13:45.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.957 7f919985c700 1 -- 172.21.15.32:0/1416767842 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f919000bd70 con 0x7f919c0d7ca0 2021-04-09T18:13:45.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.957 7f91a0c24700 1 -- 172.21.15.32:0/1416767842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c0d7ca0 msgr2=0x7f919c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.957 7f91a0c24700 1 --2- 172.21.15.32:0/1416767842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c0d7ca0 0x7f919c0d80a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f91900069d0 tx=0x7f9190009e40).stop 2021-04-09T18:13:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.958 7f91a0c24700 1 -- 172.21.15.32:0/1416767842 shutdown_connections 2021-04-09T18:13:45.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.958 7f91a0c24700 1 --2- 172.21.15.32:0/1416767842 >> v2:172.21.15.86:3300/0 conn(0x7f919c10da00 0x7f919c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.958 7f91a0c24700 1 --2- 172.21.15.32:0/1416767842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c0d7ca0 0x7f919c0d80a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.958 7f91a0c24700 1 -- 172.21.15.32:0/1416767842 >> 172.21.15.32:0/1416767842 conn(0x7f919c0ce5f0 msgr2=0x7f919c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.958 7f91a0c24700 1 -- 172.21.15.32:0/1416767842 shutdown_connections 2021-04-09T18:13:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.958 7f91a0c24700 1 -- 172.21.15.32:0/1416767842 wait complete. 2021-04-09T18:13:45.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.978 7f91a0c24700 1 Processor -- start 2021-04-09T18:13:45.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.978 7f91a0c24700 1 -- start start 2021-04-09T18:13:45.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f91a0c24700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c0b2140 0x7f919c09d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f91a0c24700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c09e010 0x7f919c194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f91a0c24700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f919c1950a0 0x7f919c0a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f91a0c24700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c09d470 con 0x7f919c0b2140 2021-04-09T18:13:45.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f91a0c24700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c09e410 con 0x7f919c1950a0 2021-04-09T18:13:45.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f91a0c24700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919c1954a0 con 0x7f919c09e010 2021-04-09T18:13:45.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c0b2140 0x7f919c09d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c0b2140 0x7f919c09d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42810/0 (socket says 172.21.15.32:42810) 2021-04-09T18:13:45.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a85e700 1 -- 172.21.15.32:0/373487579 learned_addr learned my addr 172.21.15.32:0/373487579 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a05d700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c09e010 0x7f919c194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919b05f700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f919c1950a0 0x7f919c0a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a85e700 1 -- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c09e010 msgr2=0x7f919c194a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a85e700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c09e010 0x7f919c194a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a85e700 1 -- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f919c1950a0 msgr2=0x7f919c0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a85e700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f919c1950a0 0x7f919c0a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a85e700 1 -- 172.21.15.32:0/373487579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c1956c0 con 0x7f919c0b2140 2021-04-09T18:13:45.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f919a85e700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c0b2140 0x7f919c09d9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f918c001610 tx=0x7f918c0019b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:45.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f918b7fe700 1 -- 172.21.15.32:0/373487579 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f918c007020 con 0x7f919c0b2140 2021-04-09T18:13:45.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f918b7fe700 1 -- 172.21.15.32:0/373487579 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918c004320 con 0x7f919c0b2140 2021-04-09T18:13:45.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f91a0c24700 1 -- 172.21.15.32:0/373487579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919c1956c0 con 0x7f919c0b2140 2021-04-09T18:13:45.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f918b7fe700 1 -- 172.21.15.32:0/373487579 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f918c004800 con 0x7f919c0b2140 2021-04-09T18:13:45.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.979 7f91a0c24700 1 -- 172.21.15.32:0/373487579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919c1956c0 con 0x7f919c0b2140 2021-04-09T18:13:45.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.980 7f918b7fe700 1 -- 172.21.15.32:0/373487579 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f918c00d250 con 0x7f919c0b2140 2021-04-09T18:13:45.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.980 7f918b7fe700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91800188d0 0x7f918001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.980 7f918b7fe700 1 -- 172.21.15.32:0/373487579 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f918c034c20 con 0x7f919c0b2140 2021-04-09T18:13:45.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.980 7f919a05d700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91800188d0 0x7f918001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:44.981 7f919a05d700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91800188d0 0x7f918001add0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.000 7f91a0c24700 1 -- 172.21.15.32:0/373487579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c1b7040 con 0x7f919c0b2140 2021-04-09T18:13:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.003 7f918b7fe700 1 -- 172.21.15.32:0/373487579 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f918c000c10 con 0x7f919c0b2140 2021-04-09T18:13:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.182 7f91a0c24700 1 -- 172.21.15.32:0/373487579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f919c1b7040 con 0x7f919c0b2140 2021-04-09T18:13:45.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.183 7f918b7fe700 1 -- 172.21.15.32:0/373487579 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v32) v1 ==== 100+0+10 (secure 0 0 0) 0x7f918c009ee0 con 0x7f919c0b2140 2021-04-09T18:13:45.344 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:13:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 -- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91800188d0 msgr2=0x7f918001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91800188d0 0x7f918001add0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 -- 172.21.15.32:0/373487579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c0b2140 msgr2=0x7f919c09d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c0b2140 0x7f919c09d9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f918c001610 tx=0x7f918c0019b0).stop 2021-04-09T18:13:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 -- 172.21.15.32:0/373487579 shutdown_connections 2021-04-09T18:13:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91800188d0 0x7f918001add0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f919c1950a0 0x7f919c0a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f919c09e010 0x7f919c194a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 --2- 172.21.15.32:0/373487579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f919c0b2140 0x7f919c09d9f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 -- 172.21.15.32:0/373487579 >> 172.21.15.32:0/373487579 conn(0x7f919c0b27e0 msgr2=0x7f919c1b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 -- 172.21.15.32:0/373487579 shutdown_connections 2021-04-09T18:13:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.185 7f91a0c24700 1 -- 172.21.15.32:0/373487579 wait complete. 2021-04-09T18:13:45.375 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:13:45.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:45.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.592 7f934d66b700 1 Processor -- start 2021-04-09T18:13:45.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.592 7f934d66b700 1 -- start start 2021-04-09T18:13:45.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.593 7f934d66b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93480d7ca0 0x7f93480d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.593 7f934d66b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f934810da00 0x7f934810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.593 7f934d66b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934810d480 con 0x7f934810d010 2021-04-09T18:13:45.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.593 7f934d66b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93480d9220 con 0x7f934810da00 2021-04-09T18:13:45.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.593 7f934d66b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9348112100 con 0x7f93480d7ca0 2021-04-09T18:13:45.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.593 7f9347ac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f934810da00 0x7f934810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.593 7f9347ac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f934810da00 0x7f934810de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42820/0 (socket says 172.21.15.32:42820) 2021-04-09T18:13:45.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.593 7f9347ac0700 1 -- 172.21.15.32:0/1366141951 learned_addr learned my addr 172.21.15.32:0/1366141951 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:45.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.593 7f9346abe700 1 --2- 172.21.15.32:0/1366141951 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93480d7ca0 0x7f93480d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f9347ac0700 1 -- 172.21.15.32:0/1366141951 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93480d7ca0 msgr2=0x7f93480d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:45.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f9347ac0700 1 --2- 172.21.15.32:0/1366141951 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93480d7ca0 0x7f93480d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f9347ac0700 1 -- 172.21.15.32:0/1366141951 >> v1:172.21.15.86:6789/0 conn(0x7f934810d010 legacy=0x7f93480d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f93462bd700 1 -- 172.21.15.32:0/1366141951 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201695117 0 0) 0x7f9338000c60 con 0x7f934810d010 2021-04-09T18:13:45.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f9347ac0700 1 -- 172.21.15.32:0/1366141951 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93480d86c0 con 0x7f934810da00 2021-04-09T18:13:45.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f9347ac0700 1 --2- 172.21.15.32:0/1366141951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934810da00 0x7f934810de00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f9340006f00 tx=0x7f9340009e90).ready entity=mon.0 client_cookie=623f417e44ce1571 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:45.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f93462bd700 1 -- 172.21.15.32:0/1366141951 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f934000b300 con 0x7f934810da00 2021-04-09T18:13:45.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f93462bd700 1 -- 172.21.15.32:0/1366141951 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934000b640 con 0x7f934810da00 2021-04-09T18:13:45.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f93462bd700 1 -- 172.21.15.32:0/1366141951 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f934000bb20 con 0x7f934810da00 2021-04-09T18:13:45.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f934d66b700 1 -- 172.21.15.32:0/1366141951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934810da00 msgr2=0x7f934810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.594 7f934d66b700 1 --2- 172.21.15.32:0/1366141951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934810da00 0x7f934810de00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f9340006f00 tx=0x7f9340009e90).stop 2021-04-09T18:13:45.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.595 7f934d66b700 1 -- 172.21.15.32:0/1366141951 shutdown_connections 2021-04-09T18:13:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.595 7f934d66b700 1 --2- 172.21.15.32:0/1366141951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f934810da00 0x7f934810de00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.595 7f934d66b700 1 --2- 172.21.15.32:0/1366141951 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93480d7ca0 0x7f93480d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.595 7f934d66b700 1 -- 172.21.15.32:0/1366141951 >> 172.21.15.32:0/1366141951 conn(0x7f93480ce5f0 msgr2=0x7f93480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:45.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.595 7f934d66b700 1 -- 172.21.15.32:0/1366141951 shutdown_connections 2021-04-09T18:13:45.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.595 7f934d66b700 1 -- 172.21.15.32:0/1366141951 wait complete. 2021-04-09T18:13:45.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.615 7f934d66b700 1 Processor -- start 2021-04-09T18:13:45.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.615 7f934d66b700 1 -- start start 2021-04-09T18:13:45.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f934d66b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93481acea0 0x7f93481ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f934d66b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934819b2c0 0x7f9348184da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f934d66b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93481853c0 0x7f9348185fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f934d66b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93481857c0 con 0x7f93481853c0 2021-04-09T18:13:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f934d66b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9348185a40 con 0x7f93481acea0 2021-04-09T18:13:45.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f934d66b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93481865e0 con 0x7f934819b2c0 2021-04-09T18:13:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9347ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93481853c0 0x7f9348185fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f93472bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93481acea0 0x7f93481ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9347ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93481853c0 0x7f9348185fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42826/0 (socket says 172.21.15.32:42826) 2021-04-09T18:13:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9346abe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934819b2c0 0x7f9348184da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f93472bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93481acea0 0x7f93481ac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:34582/0 (socket says 172.21.15.32:34582) 2021-04-09T18:13:45.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9347ac0700 1 -- 172.21.15.32:0/3780148990 learned_addr learned my addr 172.21.15.32:0/3780148990 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:45.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9347ac0700 1 -- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934819b2c0 msgr2=0x7f9348184da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9347ac0700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934819b2c0 0x7f9348184da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9347ac0700 1 -- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93481acea0 msgr2=0x7f93481ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9347ac0700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93481acea0 0x7f93481ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9347ac0700 1 -- 172.21.15.32:0/3780148990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9348186800 con 0x7f93481853c0 2021-04-09T18:13:45.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f9347ac0700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93481853c0 0x7f9348185fc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f934000b910 tx=0x7f934000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:45.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f932ffff700 1 -- 172.21.15.32:0/3780148990 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f934000bdf0 con 0x7f93481853c0 2021-04-09T18:13:45.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f932ffff700 1 -- 172.21.15.32:0/3780148990 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9340005820 con 0x7f93481853c0 2021-04-09T18:13:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f934d66b700 1 -- 172.21.15.32:0/3780148990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9348186800 con 0x7f93481853c0 2021-04-09T18:13:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f932ffff700 1 -- 172.21.15.32:0/3780148990 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9340005c70 con 0x7f93481853c0 2021-04-09T18:13:45.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.616 7f934d66b700 1 -- 172.21.15.32:0/3780148990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9348186800 con 0x7f93481853c0 2021-04-09T18:13:45.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.617 7f932ffff700 1 -- 172.21.15.32:0/3780148990 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9340005a00 con 0x7f93481853c0 2021-04-09T18:13:45.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.617 7f932ffff700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9328018930 0x7f932801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:45.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.617 7f932ffff700 1 -- 172.21.15.32:0/3780148990 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9340032da0 con 0x7f93481853c0 2021-04-09T18:13:45.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.617 7f93472bf700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9328018930 0x7f932801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:45.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.618 7f93472bf700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9328018930 0x7f932801ae10 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:45.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.637 7f934d66b700 1 -- 172.21.15.32:0/3780148990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9348182d80 con 0x7f93481853c0 2021-04-09T18:13:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.641 7f932ffff700 1 -- 172.21.15.32:0/3780148990 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f934000bdf0 con 0x7f93481853c0 2021-04-09T18:13:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.830 7f934d66b700 1 -- 172.21.15.32:0/3780148990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f934810d480 con 0x7f93481853c0 2021-04-09T18:13:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.830 7f932ffff700 1 -- 172.21.15.32:0/3780148990 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2776 (secure 0 0 0) 0x7f9340006ef0 con 0x7f93481853c0 2021-04-09T18:13:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 13 2021-04-09T18:13:45.921 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:45.933 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:39.509220","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:13:45.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.832 7f934d66b700 1 -- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9328018930 msgr2=0x7f932801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.832 7f934d66b700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9328018930 0x7f932801ae10 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.832 7f934d66b700 1 -- 172.21.15.32:0/3780148990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93481853c0 msgr2=0x7f9348185fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.832 7f934d66b700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93481853c0 0x7f9348185fc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f934000b910 tx=0x7f934000bcb0).stop 2021-04-09T18:13:45.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.833 7f934d66b700 1 -- 172.21.15.32:0/3780148990 shutdown_connections 2021-04-09T18:13:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.833 7f934d66b700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9328018930 0x7f932801ae10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.833 7f934d66b700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93481853c0 0x7f9348185fc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.833 7f934d66b700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934819b2c0 0x7f9348184da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:45.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.833 7f934d66b700 1 --2- 172.21.15.32:0/3780148990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93481acea0 0x7f93481ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:46.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.833 7f934d66b700 1 -- 172.21.15.32:0/3780148990 >> 172.21.15.32:0/3780148990 conn(0x7f934819b890 msgr2=0x7f93481a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:46.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.833 7f934d66b700 1 -- 172.21.15.32:0/3780148990 shutdown_connections 2021-04-09T18:13:46.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:45.833 7f934d66b700 1 -- 172.21.15.32:0/3780148990 wait complete. 2021-04-09T18:13:46.450 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:39.509220', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:13:46.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:13:46.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:13:46.594 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:13:46.594 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:13:46.594 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:13:46.595 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 4693, 2021-04-09T18:13:46.595 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:13:46.595 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:13:46.595 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 9.6815443440000006, 2021-04-09T18:13:46.595 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 13, 2021-04-09T18:13:46.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 32, 2021-04-09T18:13:46.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 32, 2021-04-09T18:13:46.596 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 18.788054247000002 2021-04-09T18:13:46.596 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:13:46.600 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 4693, "want_state": "up:active", "state": "up:active", "rank_uptime": 9.6815443440000006, "mdsmap_epoch": 13, "osdmap_epoch": 32, "osdmap_epoch_barrier": 32, "uptime": 18.788054247000002 } 2021-04-09T18:13:46.601 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:13:46.601 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:13:46.602 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:13:46.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:46.603 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:46.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:13:46.744 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:13:46.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:13:46.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:46.809 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:13:46.855 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:13:46.856 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:46.898 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:13:46.959 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:13:46.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:13:47.019 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:13:47.019 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:13:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:13:47.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:47.022 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:13:47.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:13:47.024 7fab12271f80 -1 init, newargv = 0x56490caa1350 newargc=9ceph-fuse[ 2021-04-09T18:13:47.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:31674]: starting ceph client 2021-04-09T18:13:47.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[31674]: starting fuse 2021-04-09T18:13:47.072 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:13:47.072 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:13:47.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:47.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:13:47.075 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:13:47.076 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:13:47.077 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:13:47.078 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:13:47.078 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:13:47.078 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:13:47.079 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:13:47.079 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:13:47.080 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:13:47.080 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:13:47.081 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:13:47.081 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:13:47.082 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:13:47.082 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:13:47.082 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:13:47.083 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:13:47.103 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:13:47.136 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:13:47.137 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:13:47.138 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:13:47.150 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:13:47.150 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:13:47.293 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.31674.asok 2021-04-09T18:13:47.297 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.31674.asok 2021-04-09T18:13:47.298 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:47.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.31674.asok status 2021-04-09T18:13:47.439 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:13:47.439 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:13:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:13:47.440 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:13:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:13:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:13:47.441 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:13:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "31674", 2021-04-09T18:13:47.442 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:13:47.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:13:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:13:47.446 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4917, 2021-04-09T18:13:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:13:47.447 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:13:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:13:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 4917 2021-04-09T18:13:47.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:13:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:47.449 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:13:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1190144033 2021-04-09T18:13:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:13:47.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:47.451 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:13:47.452 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:13:47.453 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:13:47.456 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1190144033 2021-04-09T18:13:47.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:13:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.4917 172.21.15.32:0/1190144033", 2021-04-09T18:13:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/1190144033", 2021-04-09T18:13:47.457 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:13:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 13, 2021-04-09T18:13:47.458 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 32, 2021-04-09T18:13:47.459 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:13:47.500 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:13:47.501 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:13:47.501 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:13:47.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:47.505 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:13:47.505 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:47.627 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:13:47.628 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:13:47.628 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:47.629 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:47.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:47.745 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:13:47.746 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:47.797 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:13:47.798 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:13:47.798 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:13:47.799 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:13:47.800 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:47.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[31674]: fuse finished with error 0 and tester_r 0 2021-04-09T18:13:47.913 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:13:47.914 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:13:47.915 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:47.968 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:13:47.968 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:13:47.969 INFO:teuthology.orchestra.run:waiting for 900 2021-04-09T18:13:47.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:13:47.999 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:13:48.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2021-04-09T18:13:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.225 7f18a0d74700 1 Processor -- start 2021-04-09T18:13:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.225 7f18a0d74700 1 -- start start 2021-04-09T18:13:48.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f18a0d74700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c10d010 0x7f189c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f18a0d74700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f189c0d8490 0x7f189c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f18a0d74700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f189c10d480 con 0x7f189c0d1da0 2021-04-09T18:13:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f18a0d74700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f189c10e780 con 0x7f189c0d8490 2021-04-09T18:13:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f18a0d74700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189c0d8d30 con 0x7f189c10d010 2021-04-09T18:13:48.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189b314700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f189c0d8490 0x7f189c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189a312700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f189c0d1da0 0x7f189c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52438/0 (socket says 172.21.15.32:52438) 2021-04-09T18:13:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189a312700 1 -- 172.21.15.32:0/2637549256 learned_addr learned my addr 172.21.15.32:0/2637549256 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189ab13700 1 --2- 172.21.15.32:0/2637549256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c10d010 0x7f189c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:48.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189b314700 1 -- 172.21.15.32:0/2637549256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c10d010 msgr2=0x7f189c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:48.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189b314700 1 --2- 172.21.15.32:0/2637549256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c10d010 0x7f189c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189b314700 1 -- 172.21.15.32:0/2637549256 >> v1:172.21.15.32:6789/0 conn(0x7f189c0d1da0 legacy=0x7f189c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189b314700 1 -- 172.21.15.32:0/2637549256 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189c0d9220 con 0x7f189c0d8490 2021-04-09T18:13:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f1899b11700 1 -- 172.21.15.32:0/2637549256 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471427831 0 0) 0x7f1890001060 con 0x7f189c0d1da0 2021-04-09T18:13:48.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189ab13700 1 --2- 172.21.15.32:0/2637549256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c10d010 0x7f189c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:13:48.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.226 7f189b314700 1 --2- 172.21.15.32:0/2637549256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0d8490 0x7f189c10da00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f1894006ab0 tx=0x7f1894009b40).ready entity=mon.0 client_cookie=ceebef498ba92c7c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:48.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f1899b11700 1 -- 172.21.15.32:0/2637549256 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18940022e0 con 0x7f189c0d8490 2021-04-09T18:13:48.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f1899b11700 1 -- 172.21.15.32:0/2637549256 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1894002660 con 0x7f189c0d8490 2021-04-09T18:13:48.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f1899b11700 1 -- 172.21.15.32:0/2637549256 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1894000c10 con 0x7f189c0d8490 2021-04-09T18:13:48.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f18a0d74700 1 -- 172.21.15.32:0/2637549256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0d8490 msgr2=0x7f189c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:48.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f18a0d74700 1 --2- 172.21.15.32:0/2637549256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0d8490 0x7f189c10da00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f1894006ab0 tx=0x7f1894009b40).stop 2021-04-09T18:13:48.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f18a0d74700 1 -- 172.21.15.32:0/2637549256 shutdown_connections 2021-04-09T18:13:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f18a0d74700 1 --2- 172.21.15.32:0/2637549256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0d8490 0x7f189c10da00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f18a0d74700 1 --2- 172.21.15.32:0/2637549256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c10d010 0x7f189c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f18a0d74700 1 -- 172.21.15.32:0/2637549256 >> 172.21.15.32:0/2637549256 conn(0x7f189c0ce5f0 msgr2=0x7f189c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:48.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f18a0d74700 1 -- 172.21.15.32:0/2637549256 shutdown_connections 2021-04-09T18:13:48.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.227 7f18a0d74700 1 -- 172.21.15.32:0/2637549256 wait complete. 2021-04-09T18:13:48.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.247 7f18a0d74700 1 Processor -- start 2021-04-09T18:13:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f18a0d74700 1 -- start start 2021-04-09T18:13:48.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f18a0d74700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c0b1e50 0x7f189c0b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f18a0d74700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0b9fd0 0x7f189c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:48.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f18a0d74700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189c0b5510 0x7f189c0b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f18a0d74700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c0b2250 con 0x7f189c0b9fd0 2021-04-09T18:13:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f18a0d74700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c0a1220 con 0x7f189c0b5510 2021-04-09T18:13:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f18a0d74700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189c0b6b30 con 0x7f189c0b1e50 2021-04-09T18:13:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f189a312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0b9fd0 0x7f189c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f189a312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0b9fd0 0x7f189c0b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42850/0 (socket says 172.21.15.32:42850) 2021-04-09T18:13:48.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f189a312700 1 -- 172.21.15.32:0/682655663 learned_addr learned my addr 172.21.15.32:0/682655663 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:48.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f189b314700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189c0b5510 0x7f189c0b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:48.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.248 7f189ab13700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c0b1e50 0x7f189c0b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:48.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f189a312700 1 -- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c0b1e50 msgr2=0x7f189c0b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:48.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f189a312700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c0b1e50 0x7f189c0b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:48.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f189a312700 1 -- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189c0b5510 msgr2=0x7f189c0b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:48.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f189a312700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189c0b5510 0x7f189c0b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:48.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f189a312700 1 -- 172.21.15.32:0/682655663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189c0b5f30 con 0x7f189c0b9fd0 2021-04-09T18:13:48.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f189a312700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0b9fd0 0x7f189c0b4ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f189000b8c0 tx=0x7f189000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f188b7fe700 1 -- 172.21.15.32:0/682655663 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1890006220 con 0x7f189c0b9fd0 2021-04-09T18:13:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f188b7fe700 1 -- 172.21.15.32:0/682655663 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18900065a0 con 0x7f189c0b9fd0 2021-04-09T18:13:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f18a0d74700 1 -- 172.21.15.32:0/682655663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f189c0b5f30 con 0x7f189c0b9fd0 2021-04-09T18:13:48.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f188b7fe700 1 -- 172.21.15.32:0/682655663 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1890004910 con 0x7f189c0b9fd0 2021-04-09T18:13:48.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.249 7f18a0d74700 1 -- 172.21.15.32:0/682655663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189c0b5f30 con 0x7f189c0b9fd0 2021-04-09T18:13:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.250 7f188b7fe700 1 -- 172.21.15.32:0/682655663 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f189000a000 con 0x7f189c0b9fd0 2021-04-09T18:13:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.250 7f188b7fe700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18800188d0 0x7f188001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.250 7f188b7fe700 1 -- 172.21.15.32:0/682655663 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1890032d60 con 0x7f189c0b9fd0 2021-04-09T18:13:48.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.250 7f189ab13700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18800188d0 0x7f188001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:48.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.250 7f189ab13700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18800188d0 0x7f188001add0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:48.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.270 7f18a0d74700 1 -- 172.21.15.32:0/682655663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189c1b7040 con 0x7f189c0b9fd0 2021-04-09T18:13:48.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.273 7f188b7fe700 1 -- 172.21.15.32:0/682655663 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f189000bda0 con 0x7f189c0b9fd0 2021-04-09T18:13:48.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:48.467 7f18a0d74700 1 -- 172.21.15.32:0/682655663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "true"} v 0) v1 -- 0x7f189c10d480 con 0x7f189c0b9fd0 2021-04-09T18:13:49.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.326 7f188b7fe700 1 -- 172.21.15.32:0/682655663 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "true"}]=0 cephfs marked down. v14) v1 ==== 125+0+0 (secure 0 0 0) 0x7f189000a000 con 0x7f189c0b9fd0 2021-04-09T18:13:49.353 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked down. 2021-04-09T18:13:49.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 -- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18800188d0 msgr2=0x7f188001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:49.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18800188d0 0x7f188001add0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 -- 172.21.15.32:0/682655663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0b9fd0 msgr2=0x7f189c0b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:49.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0b9fd0 0x7f189c0b4ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f189000b8c0 tx=0x7f189000bc60).stop 2021-04-09T18:13:49.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 -- 172.21.15.32:0/682655663 shutdown_connections 2021-04-09T18:13:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18800188d0 0x7f188001add0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f189c0b9fd0 0x7f189c0b4ef0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189c0b5510 0x7f189c0b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 --2- 172.21.15.32:0/682655663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f189c0b1e50 0x7f189c0b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 -- 172.21.15.32:0/682655663 >> 172.21.15.32:0/682655663 conn(0x7f189c0b27e0 msgr2=0x7f189c1b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:49.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 -- 172.21.15.32:0/682655663 shutdown_connections 2021-04-09T18:13:49.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.335 7f18a0d74700 1 -- 172.21.15.32:0/682655663 wait complete. 2021-04-09T18:13:49.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.516 7f3738e58700 1 Processor -- start 2021-04-09T18:13:49.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.516 7f3738e58700 1 -- start start 2021-04-09T18:13:49.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.516 7f3738e58700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340d6c80 0x7f37340d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:49.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.516 7f3738e58700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37340d7280 con 0x7f37340d2440 2021-04-09T18:13:49.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.516 7f3738e58700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37340d2840 con 0x7f37340cffa0 2021-04-09T18:13:49.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.516 7f3738e58700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37340d7080 con 0x7f37340d6c80 2021-04-09T18:13:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3733314700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f37340d2440 0x7f373410cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52450/0 (socket says 172.21.15.32:52450) 2021-04-09T18:13:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3733314700 1 -- 172.21.15.32:0/1084408467 learned_addr learned my addr 172.21.15.32:0/1084408467 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3732312700 1 --2- 172.21.15.32:0/1084408467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340d6c80 0x7f37340d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3731b11700 1 -- 172.21.15.32:0/1084408467 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759872559 0 0) 0x7f372c0009b0 con 0x7f37340d2440 2021-04-09T18:13:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3731b11700 1 -- 172.21.15.32:0/1084408467 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f371c002fe0 con 0x7f37340d2440 2021-04-09T18:13:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3731b11700 1 -- 172.21.15.32:0/1084408467 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641122587 0 0) 0x7f3724000fe0 con 0x7f37340cffa0 2021-04-09T18:13:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3731b11700 1 -- 172.21.15.32:0/1084408467 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f371c0042c0 con 0x7f37340cffa0 2021-04-09T18:13:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3732312700 1 -- 172.21.15.32:0/1084408467 >> v1:172.21.15.86:6789/0 conn(0x7f37340cffa0 legacy=0x7f37340d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:49.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3732312700 1 -- 172.21.15.32:0/1084408467 >> v1:172.21.15.32:6789/0 conn(0x7f37340d2440 legacy=0x7f373410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:49.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3731b11700 1 -- 172.21.15.32:0/1084408467 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3179336637 0 0) 0x7f372c0010e0 con 0x7f37340d2440 2021-04-09T18:13:49.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3732312700 1 -- 172.21.15.32:0/1084408467 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37340d91e0 con 0x7f37340d6c80 2021-04-09T18:13:49.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3732312700 1 --2- 172.21.15.32:0/1084408467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340d6c80 0x7f37340d7ff0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f3728006650 tx=0x7f3728009a00).ready entity=mon.2 client_cookie=79c0242a840df8ea server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:49.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3731b11700 1 -- 172.21.15.32:0/1084408467 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3728009df0 con 0x7f37340d6c80 2021-04-09T18:13:49.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3731b11700 1 -- 172.21.15.32:0/1084408467 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37280009b0 con 0x7f37340d6c80 2021-04-09T18:13:49.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.517 7f3731b11700 1 -- 172.21.15.32:0/1084408467 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372800c2b0 con 0x7f37340d6c80 2021-04-09T18:13:49.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.518 7f3738e58700 1 -- 172.21.15.32:0/1084408467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340d6c80 msgr2=0x7f37340d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:49.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.518 7f3738e58700 1 --2- 172.21.15.32:0/1084408467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340d6c80 0x7f37340d7ff0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f3728006650 tx=0x7f3728009a00).stop 2021-04-09T18:13:49.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.518 7f3738e58700 1 -- 172.21.15.32:0/1084408467 shutdown_connections 2021-04-09T18:13:49.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.518 7f3738e58700 1 --2- 172.21.15.32:0/1084408467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340d6c80 0x7f37340d7ff0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.518 7f3738e58700 1 -- 172.21.15.32:0/1084408467 >> 172.21.15.32:0/1084408467 conn(0x7f37340ce5c0 msgr2=0x7f37340cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:49.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.518 7f3738e58700 1 -- 172.21.15.32:0/1084408467 shutdown_connections 2021-04-09T18:13:49.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.518 7f3738e58700 1 -- 172.21.15.32:0/1084408467 wait complete. 2021-04-09T18:13:49.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.538 7f3738e58700 1 Processor -- start 2021-04-09T18:13:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.538 7f3738e58700 1 -- start start 2021-04-09T18:13:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3738e58700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340b72f0 0x7f37340b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3738e58700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373419c860 0x7f373419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3738e58700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373419fdd0 0x7f373419da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:49.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3738e58700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373419f230 con 0x7f373419c860 2021-04-09T18:13:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3738e58700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373408a6a0 con 0x7f373419fdd0 2021-04-09T18:13:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3738e58700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373418dce0 con 0x7f37340b72f0 2021-04-09T18:13:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3732312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373419c860 0x7f373419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3732312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373419c860 0x7f373419f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42866/0 (socket says 172.21.15.32:42866) 2021-04-09T18:13:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3732312700 1 -- 172.21.15.32:0/1165018125 learned_addr learned my addr 172.21.15.32:0/1165018125 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3733314700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373419fdd0 0x7f373419da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3732b13700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340b72f0 0x7f37340b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.539 7f3732312700 1 -- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340b72f0 msgr2=0x7f37340b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f3732312700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340b72f0 0x7f37340b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f3732312700 1 -- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373419fdd0 msgr2=0x7f373419da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f3732312700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373419fdd0 0x7f373419da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f3732312700 1 -- 172.21.15.32:0/1165018125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373418e150 con 0x7f373419c860 2021-04-09T18:13:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f3732312700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373419c860 0x7f373419f7b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f37280096a0 tx=0x7f372800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f37237fe700 1 -- 172.21.15.32:0/1165018125 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3728000c90 con 0x7f373419c860 2021-04-09T18:13:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f37237fe700 1 -- 172.21.15.32:0/1165018125 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3728002030 con 0x7f373419c860 2021-04-09T18:13:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f3738e58700 1 -- 172.21.15.32:0/1165018125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373418e150 con 0x7f373419c860 2021-04-09T18:13:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f37237fe700 1 -- 172.21.15.32:0/1165018125 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3728002550 con 0x7f373419c860 2021-04-09T18:13:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f3738e58700 1 -- 172.21.15.32:0/1165018125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373418e150 con 0x7f373419c860 2021-04-09T18:13:49.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.540 7f37237fe700 1 -- 172.21.15.32:0/1165018125 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3728002210 con 0x7f373419c860 2021-04-09T18:13:49.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.541 7f37237fe700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3718018930 0x7f371801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.541 7f37237fe700 1 -- 172.21.15.32:0/1165018125 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3728032a40 con 0x7f373419c860 2021-04-09T18:13:49.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.541 7f3732b13700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3718018930 0x7f371801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.541 7f3732b13700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3718018930 0x7f371801ae10 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:49.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.560 7f3738e58700 1 -- 172.21.15.32:0/1165018125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3734182e10 con 0x7f373419c860 2021-04-09T18:13:49.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.564 7f37237fe700 1 -- 172.21.15.32:0/1165018125 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f37280009c0 con 0x7f373419c860 2021-04-09T18:13:49.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.757 7f3738e58700 1 -- 172.21.15.32:0/1165018125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37340d7280 con 0x7f373419c860 2021-04-09T18:13:49.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.757 7f37237fe700 1 -- 172.21.15.32:0/1165018125 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7f372800bd80 con 0x7f373419c860 2021-04-09T18:13:49.785 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:13:49.786 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:49.786 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:13:49.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.758 7f3738e58700 1 -- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3718018930 msgr2=0x7f371801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:49.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.758 7f3738e58700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3718018930 0x7f371801ae10 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.758 7f3738e58700 1 -- 172.21.15.32:0/1165018125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373419c860 msgr2=0x7f373419f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.758 7f3738e58700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373419c860 0x7f373419f7b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f37280096a0 tx=0x7f372800b990).stop 2021-04-09T18:13:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.759 7f3738e58700 1 -- 172.21.15.32:0/1165018125 shutdown_connections 2021-04-09T18:13:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.759 7f3738e58700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3718018930 0x7f371801ae10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.759 7f3738e58700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373419c860 0x7f373419f7b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.759 7f3738e58700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373419fdd0 0x7f373419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.759 7f3738e58700 1 --2- 172.21.15.32:0/1165018125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37340b72f0 0x7f37340b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:49.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.759 7f3738e58700 1 -- 172.21.15.32:0/1165018125 >> 172.21.15.32:0/1165018125 conn(0x7f3734097550 msgr2=0x7f373418b920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:49.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.759 7f3738e58700 1 -- 172.21.15.32:0/1165018125 shutdown_connections 2021-04-09T18:13:49.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:49.759 7f3738e58700 1 -- 172.21.15.32:0/1165018125 wait complete. 2021-04-09T18:13:50.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:13:50.396 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:13:51.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.524 7fc3b5c99700 1 Processor -- start 2021-04-09T18:13:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.524 7fc3b5c99700 1 -- start start 2021-04-09T18:13:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3b5c99700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc3b00d7ca0 0x7fc3b00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:51.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3b5c99700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3b010da00 0x7fc3b010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3b5c99700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b010d480 con 0x7fc3b010d010 2021-04-09T18:13:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3b5c99700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b00d9220 con 0x7fc3b00d7ca0 2021-04-09T18:13:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3b5c99700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b00d86c0 con 0x7fc3b010da00 2021-04-09T18:13:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3aeffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc3b00d7ca0 0x7fc3b00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:51.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3aeffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc3b00d7ca0 0x7fc3b00d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42876/0 (socket says 172.21.15.32:42876) 2021-04-09T18:13:51.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3aeffd700 1 -- 172.21.15.32:0/3527461816 learned_addr learned my addr 172.21.15.32:0/3527461816 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:51.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3affff700 1 --2- 172.21.15.32:0/3527461816 >> v2:172.21.15.86:3300/0 conn(0x7fc3b010da00 0x7fc3b010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:51.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3aeffd700 1 -- 172.21.15.32:0/3527461816 >> v2:172.21.15.86:3300/0 conn(0x7fc3b010da00 msgr2=0x7fc3b010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:51.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3aeffd700 1 --2- 172.21.15.32:0/3527461816 >> v2:172.21.15.86:3300/0 conn(0x7fc3b010da00 0x7fc3b010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3aeffd700 1 -- 172.21.15.32:0/3527461816 >> v1:172.21.15.86:6789/0 conn(0x7fc3b010d010 legacy=0x7fc3b00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:51.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3aeffd700 1 -- 172.21.15.32:0/3527461816 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b00d1f70 con 0x7fc3b00d7ca0 2021-04-09T18:13:51.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3aeffd700 1 --2- 172.21.15.32:0/3527461816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b00d7ca0 0x7fc3b00d80a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fc39c006700 tx=0x7fc39c009b80).ready entity=mon.0 client_cookie=e95e83a252b1b457 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:51.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3ae7fc700 1 -- 172.21.15.32:0/3527461816 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc39c00c0c0 con 0x7fc3b00d7ca0 2021-04-09T18:13:51.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3ae7fc700 1 -- 172.21.15.32:0/3527461816 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c00c440 con 0x7fc3b00d7ca0 2021-04-09T18:13:51.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.525 7fc3ae7fc700 1 -- 172.21.15.32:0/3527461816 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc39c00c920 con 0x7fc3b00d7ca0 2021-04-09T18:13:51.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.526 7fc3b5c99700 1 -- 172.21.15.32:0/3527461816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b00d7ca0 msgr2=0x7fc3b00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:51.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.526 7fc3b5c99700 1 --2- 172.21.15.32:0/3527461816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b00d7ca0 0x7fc3b00d80a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fc39c006700 tx=0x7fc39c009b80).stop 2021-04-09T18:13:51.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.526 7fc3b5c99700 1 -- 172.21.15.32:0/3527461816 shutdown_connections 2021-04-09T18:13:51.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.526 7fc3b5c99700 1 --2- 172.21.15.32:0/3527461816 >> v2:172.21.15.86:3300/0 conn(0x7fc3b010da00 0x7fc3b010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.526 7fc3b5c99700 1 --2- 172.21.15.32:0/3527461816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b00d7ca0 0x7fc3b00d80a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.526 7fc3b5c99700 1 -- 172.21.15.32:0/3527461816 >> 172.21.15.32:0/3527461816 conn(0x7fc3b00ce5f0 msgr2=0x7fc3b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:51.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.526 7fc3b5c99700 1 -- 172.21.15.32:0/3527461816 shutdown_connections 2021-04-09T18:13:51.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.526 7fc3b5c99700 1 -- 172.21.15.32:0/3527461816 wait complete. 2021-04-09T18:13:51.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.546 7fc3b5c99700 1 Processor -- start 2021-04-09T18:13:51.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3b5c99700 1 -- start start 2021-04-09T18:13:51.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3b5c99700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b00b5360 0x7fc3b01a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:51.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3b5c99700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b018e140 0x7fc3b01aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:51.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3b5c99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b01ad040 0x7fc3b008fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:51.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3b5c99700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b01a5360 con 0x7fc3b01ad040 2021-04-09T18:13:51.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3b5c99700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00902c0 con 0x7fc3b00b5360 2021-04-09T18:13:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3b5c99700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00904e0 con 0x7fc3b018e140 2021-04-09T18:13:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b01ad040 0x7fc3b008fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b01ad040 0x7fc3b008fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42880/0 (socket says 172.21.15.32:42880) 2021-04-09T18:13:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3affff700 1 -- 172.21.15.32:0/2852915405 learned_addr learned my addr 172.21.15.32:0/2852915405 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:51.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3aeffd700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b018e140 0x7fc3b01aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:51.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.547 7fc3af7fe700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b00b5360 0x7fc3b01a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:51.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3affff700 1 -- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b018e140 msgr2=0x7fc3b01aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:51.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3affff700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b018e140 0x7fc3b01aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3affff700 1 -- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b00b5360 msgr2=0x7fc3b01a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:51.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3affff700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b00b5360 0x7fc3b01a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3affff700 1 -- 172.21.15.32:0/2852915405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b0183c80 con 0x7fc3b01ad040 2021-04-09T18:13:51.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3affff700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b01ad040 0x7fc3b008fca0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fc3a800b970 tx=0x7fc3a800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:51.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3a7fff700 1 -- 172.21.15.32:0/2852915405 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3a8000f90 con 0x7fc3b01ad040 2021-04-09T18:13:51.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3a7fff700 1 -- 172.21.15.32:0/2852915405 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a8006aa0 con 0x7fc3b01ad040 2021-04-09T18:13:51.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3a7fff700 1 -- 172.21.15.32:0/2852915405 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3a80015f0 con 0x7fc3b01ad040 2021-04-09T18:13:51.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b0183c80 con 0x7fc3b01ad040 2021-04-09T18:13:51.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.548 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b0183c80 con 0x7fc3b01ad040 2021-04-09T18:13:51.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.549 7fc3a7fff700 1 -- 172.21.15.32:0/2852915405 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc3a8000e70 con 0x7fc3b01ad040 2021-04-09T18:13:51.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.549 7fc3a7fff700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc38c018930 0x7fc38c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.549 7fc3a7fff700 1 -- 172.21.15.32:0/2852915405 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc3a8032d70 con 0x7fc3b01ad040 2021-04-09T18:13:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.549 7fc3af7fe700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc38c018930 0x7fc38c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:51.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.549 7fc3af7fe700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc38c018930 0x7fc38c01ae10 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.569 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b0182d70 con 0x7fc3b01ad040 2021-04-09T18:13:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.572 7fc3a7fff700 1 -- 172.21.15.32:0/2852915405 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc3a80066f0 con 0x7fc3b01ad040 2021-04-09T18:13:51.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.763 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3b0182d70 con 0x7fc3b01ad040 2021-04-09T18:13:51.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.764 7fc3a7fff700 1 -- 172.21.15.32:0/2852915405 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7fc3a8006aa0 con 0x7fc3b01ad040 2021-04-09T18:13:51.839 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:13:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:51.879 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:13:51.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc38c018930 msgr2=0x7fc38c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:51.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc38c018930 0x7fc38c01ae10 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b01ad040 msgr2=0x7fc3b008fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:51.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b01ad040 0x7fc3b008fca0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fc3a800b970 tx=0x7fc3a800bd10).stop 2021-04-09T18:13:51.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 shutdown_connections 2021-04-09T18:13:51.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc38c018930 0x7fc38c01ae10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3b01ad040 0x7fc3b008fca0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b018e140 0x7fc3b01aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 --2- 172.21.15.32:0/2852915405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b00b5360 0x7fc3b01a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:51.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 >> 172.21.15.32:0/2852915405 conn(0x7fc3b019b460 msgr2=0x7fc3b019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:51.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 shutdown_connections 2021-04-09T18:13:51.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:51.766 7fc3b5c99700 1 -- 172.21.15.32:0/2852915405 wait complete. 2021-04-09T18:13:52.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:13:52.342 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:13:53.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:53.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.487 7fb176890700 1 Processor -- start 2021-04-09T18:13:53.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.487 7fb176890700 1 -- start start 2021-04-09T18:13:53.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb176890700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb1700d7ca0 0x7fb1700d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:53.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb176890700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb17010da00 0x7fb17010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:53.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb176890700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17010d480 con 0x7fb17010d010 2021-04-09T18:13:53.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb176890700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1700d9220 con 0x7fb17010da00 2021-04-09T18:13:53.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb176890700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1700d86c0 con 0x7fb1700d7ca0 2021-04-09T18:13:53.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb174ead700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb17010da00 0x7fb17010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:53.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb174ead700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb17010da00 0x7fb17010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42890/0 (socket says 172.21.15.32:42890) 2021-04-09T18:13:53.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb174ead700 1 -- 172.21.15.32:0/182591910 learned_addr learned my addr 172.21.15.32:0/182591910 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:53.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb167fff700 1 --2- 172.21.15.32:0/182591910 >> v2:172.21.15.86:3300/0 conn(0x7fb1700d7ca0 0x7fb1700d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:53.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb174ead700 1 -- 172.21.15.32:0/182591910 >> v2:172.21.15.86:3300/0 conn(0x7fb1700d7ca0 msgr2=0x7fb1700d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:53.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb174ead700 1 --2- 172.21.15.32:0/182591910 >> v2:172.21.15.86:3300/0 conn(0x7fb1700d7ca0 0x7fb1700d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb174ead700 1 -- 172.21.15.32:0/182591910 >> v1:172.21.15.86:6789/0 conn(0x7fb17010d010 legacy=0x7fb1700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:53.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb174ead700 1 -- 172.21.15.32:0/182591910 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1700d1f70 con 0x7fb17010da00 2021-04-09T18:13:53.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb174ead700 1 --2- 172.21.15.32:0/182591910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17010da00 0x7fb17010de00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fb168006f00 tx=0x7fb168009e90).ready entity=mon.0 client_cookie=6b3da5600b73a3ff server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:53.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb16f7fe700 1 -- 172.21.15.32:0/182591910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb16800b300 con 0x7fb17010da00 2021-04-09T18:13:53.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb16f7fe700 1 -- 172.21.15.32:0/182591910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16800b640 con 0x7fb17010da00 2021-04-09T18:13:53.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.488 7fb16f7fe700 1 -- 172.21.15.32:0/182591910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb16800bb20 con 0x7fb17010da00 2021-04-09T18:13:53.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.489 7fb176890700 1 -- 172.21.15.32:0/182591910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17010da00 msgr2=0x7fb17010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:53.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.489 7fb176890700 1 --2- 172.21.15.32:0/182591910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17010da00 0x7fb17010de00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fb168006f00 tx=0x7fb168009e90).stop 2021-04-09T18:13:53.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.489 7fb176890700 1 -- 172.21.15.32:0/182591910 shutdown_connections 2021-04-09T18:13:53.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.489 7fb176890700 1 --2- 172.21.15.32:0/182591910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17010da00 0x7fb17010de00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.489 7fb176890700 1 --2- 172.21.15.32:0/182591910 >> v2:172.21.15.86:3300/0 conn(0x7fb1700d7ca0 0x7fb1700d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.489 7fb176890700 1 -- 172.21.15.32:0/182591910 >> 172.21.15.32:0/182591910 conn(0x7fb1700ce5f0 msgr2=0x7fb1700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:53.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.489 7fb176890700 1 -- 172.21.15.32:0/182591910 shutdown_connections 2021-04-09T18:13:53.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.489 7fb176890700 1 -- 172.21.15.32:0/182591910 wait complete. 2021-04-09T18:13:53.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb176890700 1 Processor -- start 2021-04-09T18:13:53.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb176890700 1 -- start start 2021-04-09T18:13:53.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb176890700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1701acea0 0x7fb17009e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:53.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb176890700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1701ac380 0x7fb17009d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:53.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb176890700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17009df40 0x7fb17009f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:53.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb176890700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17009edb0 con 0x7fb17009df40 2021-04-09T18:13:53.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb176890700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17009ea10 con 0x7fb1701acea0 2021-04-09T18:13:53.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb176890700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1701ac780 con 0x7fb1701ac380 2021-04-09T18:13:53.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb174ead700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17009df40 0x7fb17009f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:53.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb174ead700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17009df40 0x7fb17009f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42896/0 (socket says 172.21.15.32:42896) 2021-04-09T18:13:53.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb174ead700 1 -- 172.21.15.32:0/970930148 learned_addr learned my addr 172.21.15.32:0/970930148 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:53.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb16ffff700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1701acea0 0x7fb17009e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:53.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.510 7fb167fff700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1701ac380 0x7fb17009d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:53.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb174ead700 1 -- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1701ac380 msgr2=0x7fb17009d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:53.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb174ead700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1701ac380 0x7fb17009d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb174ead700 1 -- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1701acea0 msgr2=0x7fb17009e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:53.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb174ead700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1701acea0 0x7fb17009e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb174ead700 1 -- 172.21.15.32:0/970930148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1701ac9a0 con 0x7fb17009df40 2021-04-09T18:13:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb16ffff700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1701acea0 0x7fb17009e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb174ead700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17009df40 0x7fb17009f2b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fb1680099f0 tx=0x7fb16800bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb16d7fa700 1 -- 172.21.15.32:0/970930148 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb16800be20 con 0x7fb17009df40 2021-04-09T18:13:53.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb16d7fa700 1 -- 172.21.15.32:0/970930148 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb168003030 con 0x7fb17009df40 2021-04-09T18:13:53.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb16d7fa700 1 -- 172.21.15.32:0/970930148 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb168001460 con 0x7fb17009df40 2021-04-09T18:13:53.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb176890700 1 -- 172.21.15.32:0/970930148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1701ac9a0 con 0x7fb17009df40 2021-04-09T18:13:53.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb176890700 1 -- 172.21.15.32:0/970930148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1701ac9a0 con 0x7fb17009df40 2021-04-09T18:13:53.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.511 7fb16d7fa700 1 -- 172.21.15.32:0/970930148 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb168001270 con 0x7fb17009df40 2021-04-09T18:13:53.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.512 7fb16d7fa700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1540188c0 0x7fb15401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.512 7fb16d7fa700 1 -- 172.21.15.32:0/970930148 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb168032b80 con 0x7fb17009df40 2021-04-09T18:13:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.512 7fb16ffff700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1540188c0 0x7fb15401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.512 7fb16ffff700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1540188c0 0x7fb15401adc0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:53.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.531 7fb176890700 1 -- 172.21.15.32:0/970930148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb170182d80 con 0x7fb17009df40 2021-04-09T18:13:53.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.535 7fb16d7fa700 1 -- 172.21.15.32:0/970930148 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb168000cc0 con 0x7fb17009df40 2021-04-09T18:13:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.725 7fb176890700 1 -- 172.21.15.32:0/970930148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb17010d480 con 0x7fb17009df40 2021-04-09T18:13:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.726 7fb16d7fa700 1 -- 172.21.15.32:0/970930148 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7fb168001ee0 con 0x7fb17009df40 2021-04-09T18:13:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:13:53.785 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:53.785 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:13:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 -- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1540188c0 msgr2=0x7fb15401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1540188c0 0x7fb15401adc0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 -- 172.21.15.32:0/970930148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17009df40 msgr2=0x7fb17009f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:53.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17009df40 0x7fb17009f2b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fb1680099f0 tx=0x7fb16800bce0).stop 2021-04-09T18:13:53.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 -- 172.21.15.32:0/970930148 shutdown_connections 2021-04-09T18:13:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1540188c0 0x7fb15401adc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb17009df40 0x7fb17009f2b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1701ac380 0x7fb17009d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 --2- 172.21.15.32:0/970930148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1701acea0 0x7fb17009e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 -- 172.21.15.32:0/970930148 >> 172.21.15.32:0/970930148 conn(0x7fb17019b890 msgr2=0x7fb1701a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 -- 172.21.15.32:0/970930148 shutdown_connections 2021-04-09T18:13:53.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:53.728 7fb176890700 1 -- 172.21.15.32:0/970930148 wait complete. 2021-04-09T18:13:54.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:13:54.304 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:13:55.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:55.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.460 7fc1d5f0a700 1 Processor -- start 2021-04-09T18:13:55.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.460 7fc1d5f0a700 1 -- start start 2021-04-09T18:13:55.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1d5f0a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1d010da00 0x7fc1d010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:55.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1d5f0a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d010d480 con 0x7fc1d010d010 2021-04-09T18:13:55.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1d5f0a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d00d1f70 con 0x7fc1d00d7ca0 2021-04-09T18:13:55.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1d5f0a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d00d8d30 con 0x7fc1d010da00 2021-04-09T18:13:55.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1cffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1d010da00 0x7fc1d010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:55.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1cf7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc1d010d010 0x7fc1d00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52490/0 (socket says 172.21.15.32:52490) 2021-04-09T18:13:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1cf7fe700 1 -- 172.21.15.32:0/3775846300 learned_addr learned my addr 172.21.15.32:0/3775846300 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1cffff700 1 -- 172.21.15.32:0/3775846300 >> v1:172.21.15.86:6789/0 conn(0x7fc1d00d7ca0 legacy=0x7fc1d00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1cffff700 1 -- 172.21.15.32:0/3775846300 >> v1:172.21.15.32:6789/0 conn(0x7fc1d010d010 legacy=0x7fc1d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1ce7fc700 1 -- 172.21.15.32:0/3775846300 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948025999 0 0) 0x7fc1c0001060 con 0x7fc1d010d010 2021-04-09T18:13:55.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1cffff700 1 -- 172.21.15.32:0/3775846300 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d00d9220 con 0x7fc1d010da00 2021-04-09T18:13:55.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1cffff700 1 --2- 172.21.15.32:0/3775846300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d010da00 0x7fc1d010de00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fc1c8006ab0 tx=0x7fc1c8009b40).ready entity=mon.0 client_cookie=79ffebe8b9b431bd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:55.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.461 7fc1ce7fc700 1 -- 172.21.15.32:0/3775846300 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c80022e0 con 0x7fc1d010da00 2021-04-09T18:13:55.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.462 7fc1ce7fc700 1 -- 172.21.15.32:0/3775846300 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c8002660 con 0x7fc1d010da00 2021-04-09T18:13:55.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.462 7fc1ce7fc700 1 -- 172.21.15.32:0/3775846300 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c8002b40 con 0x7fc1d010da00 2021-04-09T18:13:55.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.462 7fc1d5f0a700 1 -- 172.21.15.32:0/3775846300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d010da00 msgr2=0x7fc1d010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:55.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.462 7fc1d5f0a700 1 --2- 172.21.15.32:0/3775846300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d010da00 0x7fc1d010de00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fc1c8006ab0 tx=0x7fc1c8009b40).stop 2021-04-09T18:13:55.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.462 7fc1d5f0a700 1 -- 172.21.15.32:0/3775846300 shutdown_connections 2021-04-09T18:13:55.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.462 7fc1d5f0a700 1 --2- 172.21.15.32:0/3775846300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d010da00 0x7fc1d010de00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:55.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.462 7fc1d5f0a700 1 -- 172.21.15.32:0/3775846300 >> 172.21.15.32:0/3775846300 conn(0x7fc1d00ce5f0 msgr2=0x7fc1d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:55.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.462 7fc1d5f0a700 1 -- 172.21.15.32:0/3775846300 shutdown_connections 2021-04-09T18:13:55.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.462 7fc1d5f0a700 1 -- 172.21.15.32:0/3775846300 wait complete. 2021-04-09T18:13:55.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1d5f0a700 1 Processor -- start 2021-04-09T18:13:55.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1d5f0a700 1 -- start start 2021-04-09T18:13:55.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1d5f0a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1d01a9a50 0x7fc1d008a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:55.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1d5f0a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1d008afb0 0x7fc1d009d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:55.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1d5f0a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d009ddf0 0x7fc1d0183b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:55.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1d5f0a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d01a9e50 con 0x7fc1d009ddf0 2021-04-09T18:13:55.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1d5f0a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d009e410 con 0x7fc1d01a9a50 2021-04-09T18:13:55.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1d5f0a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d01841a0 con 0x7fc1d008afb0 2021-04-09T18:13:55.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d009ddf0 0x7fc1d0183b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:55.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d009ddf0 0x7fc1d0183b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42910/0 (socket says 172.21.15.32:42910) 2021-04-09T18:13:55.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.483 7fc1cffff700 1 -- 172.21.15.32:0/3776920255 learned_addr learned my addr 172.21.15.32:0/3776920255 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:55.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1cf7fe700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1d01a9a50 0x7fc1d008a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:55.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1ceffd700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1d008afb0 0x7fc1d009d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:55.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1cffff700 1 -- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1d008afb0 msgr2=0x7fc1d009d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:55.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1cffff700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1d008afb0 0x7fc1d009d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:55.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1cffff700 1 -- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1d01a9a50 msgr2=0x7fc1d008a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:55.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1cffff700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1d01a9a50 0x7fc1d008a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:55.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1cffff700 1 -- 172.21.15.32:0/3776920255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d0184380 con 0x7fc1d009ddf0 2021-04-09T18:13:55.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1cffff700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d009ddf0 0x7fc1d0183b80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc1c800b5f0 tx=0x7fc1c800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:55.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1b7fff700 1 -- 172.21.15.32:0/3776920255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c800bd80 con 0x7fc1d009ddf0 2021-04-09T18:13:55.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1b7fff700 1 -- 172.21.15.32:0/3776920255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c8006020 con 0x7fc1d009ddf0 2021-04-09T18:13:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1b7fff700 1 -- 172.21.15.32:0/3776920255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c80054a0 con 0x7fc1d009ddf0 2021-04-09T18:13:55.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d0184380 con 0x7fc1d009ddf0 2021-04-09T18:13:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.484 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d0184380 con 0x7fc1d009ddf0 2021-04-09T18:13:55.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.485 7fc1b7fff700 1 -- 172.21.15.32:0/3776920255 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc1c80054a0 con 0x7fc1d009ddf0 2021-04-09T18:13:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.485 7fc1b7fff700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1b0018930 0x7fc1b001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.485 7fc1b7fff700 1 -- 172.21.15.32:0/3776920255 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc1c8032a10 con 0x7fc1d009ddf0 2021-04-09T18:13:55.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.485 7fc1cf7fe700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1b0018930 0x7fc1b001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:55.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.485 7fc1cf7fe700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1b0018930 0x7fc1b001ae10 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.505 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d010d480 con 0x7fc1d009ddf0 2021-04-09T18:13:55.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.508 7fc1b7fff700 1 -- 172.21.15.32:0/3776920255 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc1c801f750 con 0x7fc1d009ddf0 2021-04-09T18:13:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.698 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1d010d480 con 0x7fc1d009ddf0 2021-04-09T18:13:55.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.699 7fc1b7fff700 1 -- 172.21.15.32:0/3776920255 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7fc1c80054a0 con 0x7fc1d009ddf0 2021-04-09T18:13:55.790 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:13:55.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:55.793 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:13:55.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1b0018930 msgr2=0x7fc1b001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1b0018930 0x7fc1b001ae10 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d009ddf0 msgr2=0x7fc1d0183b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:55.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d009ddf0 0x7fc1d0183b80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc1c800b5f0 tx=0x7fc1c800b990).stop 2021-04-09T18:13:55.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 shutdown_connections 2021-04-09T18:13:55.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1b0018930 0x7fc1b001ae10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:55.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1d009ddf0 0x7fc1d0183b80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:55.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1d008afb0 0x7fc1d009d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 --2- 172.21.15.32:0/3776920255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1d01a9a50 0x7fc1d008a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:55.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 >> 172.21.15.32:0/3776920255 conn(0x7fc1d01965c0 msgr2=0x7fc1d01ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:55.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 shutdown_connections 2021-04-09T18:13:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:55.701 7fc1d5f0a700 1 -- 172.21.15.32:0/3776920255 wait complete. 2021-04-09T18:13:56.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:13:56.263 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:13:57.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.398 7feb28994700 1 Processor -- start 2021-04-09T18:13:57.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.398 7feb28994700 1 -- start start 2021-04-09T18:13:57.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.398 7feb28994700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb2410d010 0x7feb240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:57.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.398 7feb28994700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feb240d1da0 0x7feb240d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:57.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb28994700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2410d480 con 0x7feb240d82c0 2021-04-09T18:13:57.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb28994700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2410e780 con 0x7feb240d1da0 2021-04-09T18:13:57.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb28994700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb240d8d30 con 0x7feb2410d010 2021-04-09T18:13:57.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb21e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feb240d1da0 0x7feb240d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:57.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb21e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feb240d1da0 0x7feb240d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42918/0 (socket says 172.21.15.32:42918) 2021-04-09T18:13:57.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb21e1b700 1 -- 172.21.15.32:0/1808663961 learned_addr learned my addr 172.21.15.32:0/1808663961 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:57.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb2261c700 1 --2- 172.21.15.32:0/1808663961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb2410d010 0x7feb240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:57.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb21e1b700 1 -- 172.21.15.32:0/1808663961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb2410d010 msgr2=0x7feb240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:57.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb21e1b700 1 --2- 172.21.15.32:0/1808663961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb2410d010 0x7feb240d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb21e1b700 1 -- 172.21.15.32:0/1808663961 >> v1:172.21.15.86:6789/0 conn(0x7feb240d82c0 legacy=0x7feb2410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb21e1b700 1 -- 172.21.15.32:0/1808663961 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240d9220 con 0x7feb240d1da0 2021-04-09T18:13:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb2161a700 1 -- 172.21.15.32:0/1808663961 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614938094 0 0) 0x7feb1c000d30 con 0x7feb240d82c0 2021-04-09T18:13:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb21e1b700 1 --2- 172.21.15.32:0/1808663961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d1da0 0x7feb240d7ca0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7feb18006700 tx=0x7feb18009b00).ready entity=mon.0 client_cookie=77763f968aba27c9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb2161a700 1 -- 172.21.15.32:0/1808663961 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb1800c0c0 con 0x7feb240d1da0 2021-04-09T18:13:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb2161a700 1 -- 172.21.15.32:0/1808663961 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1800c400 con 0x7feb240d1da0 2021-04-09T18:13:57.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb2161a700 1 -- 172.21.15.32:0/1808663961 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb1800c8e0 con 0x7feb240d1da0 2021-04-09T18:13:57.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb28994700 1 -- 172.21.15.32:0/1808663961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d1da0 msgr2=0x7feb240d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:57.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.399 7feb28994700 1 --2- 172.21.15.32:0/1808663961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d1da0 0x7feb240d7ca0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7feb18006700 tx=0x7feb18009b00).stop 2021-04-09T18:13:57.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.400 7feb28994700 1 -- 172.21.15.32:0/1808663961 shutdown_connections 2021-04-09T18:13:57.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.400 7feb28994700 1 --2- 172.21.15.32:0/1808663961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb240d1da0 0x7feb240d7ca0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.400 7feb28994700 1 --2- 172.21.15.32:0/1808663961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb2410d010 0x7feb240d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.400 7feb28994700 1 -- 172.21.15.32:0/1808663961 >> 172.21.15.32:0/1808663961 conn(0x7feb240ce5f0 msgr2=0x7feb240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:57.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.400 7feb28994700 1 -- 172.21.15.32:0/1808663961 shutdown_connections 2021-04-09T18:13:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.400 7feb28994700 1 -- 172.21.15.32:0/1808663961 wait complete. 2021-04-09T18:13:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.420 7feb28994700 1 Processor -- start 2021-04-09T18:13:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.420 7feb28994700 1 -- start start 2021-04-09T18:13:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb28994700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb241a5210 0x7feb24185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:57.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb28994700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb24185a70 0x7feb240acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:57.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb28994700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb24185e70 0x7feb240ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:57.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb28994700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2418f7b0 con 0x7feb24185a70 2021-04-09T18:13:57.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb28994700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb240ad7f0 con 0x7feb241a5210 2021-04-09T18:13:57.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb28994700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb240ada10 con 0x7feb24185e70 2021-04-09T18:13:57.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb21e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb24185a70 0x7feb240acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:57.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb21e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb24185a70 0x7feb240acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42922/0 (socket says 172.21.15.32:42922) 2021-04-09T18:13:57.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb21e1b700 1 -- 172.21.15.32:0/1465209778 learned_addr learned my addr 172.21.15.32:0/1465209778 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:57.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb2261c700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb241a5210 0x7feb24185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:57.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb22e1d700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb24185e70 0x7feb240ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:57.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb21e1b700 1 -- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb24185e70 msgr2=0x7feb240ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:57.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb21e1b700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb24185e70 0x7feb240ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb21e1b700 1 -- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb241a5210 msgr2=0x7feb24185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:57.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb21e1b700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb241a5210 0x7feb24185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.421 7feb21e1b700 1 -- 172.21.15.32:0/1465209778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24179610 con 0x7feb24185a70 2021-04-09T18:13:57.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.422 7feb21e1b700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb24185a70 0x7feb240acbb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7feb1800b590 tx=0x7feb1800b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:57.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.422 7feb12ffd700 1 -- 172.21.15.32:0/1465209778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb18006390 con 0x7feb24185a70 2021-04-09T18:13:57.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.422 7feb12ffd700 1 -- 172.21.15.32:0/1465209778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1800be20 con 0x7feb24185a70 2021-04-09T18:13:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.422 7feb28994700 1 -- 172.21.15.32:0/1465209778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb24179610 con 0x7feb24185a70 2021-04-09T18:13:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.422 7feb12ffd700 1 -- 172.21.15.32:0/1465209778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb18002030 con 0x7feb24185a70 2021-04-09T18:13:57.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.422 7feb28994700 1 -- 172.21.15.32:0/1465209778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb24179610 con 0x7feb24185a70 2021-04-09T18:13:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.422 7feb12ffd700 1 -- 172.21.15.32:0/1465209778 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7feb180093e0 con 0x7feb24185a70 2021-04-09T18:13:57.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.423 7feb12ffd700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb080188d0 0x7feb0801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:57.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.423 7feb12ffd700 1 -- 172.21.15.32:0/1465209778 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7feb18032a10 con 0x7feb24185a70 2021-04-09T18:13:57.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.423 7feb2261c700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb080188d0 0x7feb0801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:57.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.423 7feb2261c700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb080188d0 0x7feb0801add0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:57.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.443 7feb28994700 1 -- 172.21.15.32:0/1465209778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2418f7b0 con 0x7feb24185a70 2021-04-09T18:13:57.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.446 7feb12ffd700 1 -- 172.21.15.32:0/1465209778 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7feb180090c0 con 0x7feb24185a70 2021-04-09T18:13:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.638 7feb28994700 1 -- 172.21.15.32:0/1465209778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb2418f7b0 con 0x7feb24185a70 2021-04-09T18:13:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.638 7feb12ffd700 1 -- 172.21.15.32:0/1465209778 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7feb180093e0 con 0x7feb24185a70 2021-04-09T18:13:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:13:57.664 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:57.665 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:13:57.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 -- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb080188d0 msgr2=0x7feb0801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:57.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb080188d0 0x7feb0801add0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 -- 172.21.15.32:0/1465209778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb24185a70 msgr2=0x7feb240acbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:57.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb24185a70 0x7feb240acbb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7feb1800b590 tx=0x7feb1800b930).stop 2021-04-09T18:13:57.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 -- 172.21.15.32:0/1465209778 shutdown_connections 2021-04-09T18:13:57.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb080188d0 0x7feb0801add0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb24185a70 0x7feb240acbb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb24185e70 0x7feb240ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 --2- 172.21.15.32:0/1465209778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb241a5210 0x7feb24185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:57.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 -- 172.21.15.32:0/1465209778 >> 172.21.15.32:0/1465209778 conn(0x7feb241a99c0 msgr2=0x7feb2417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:57.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 -- 172.21.15.32:0/1465209778 shutdown_connections 2021-04-09T18:13:57.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:57.641 7feb28994700 1 -- 172.21.15.32:0/1465209778 wait complete. 2021-04-09T18:13:58.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:13:58.170 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:13:59.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:13:59.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.311 7f7109c83700 1 Processor -- start 2021-04-09T18:13:59.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.311 7f7109c83700 1 -- start start 2021-04-09T18:13:59.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f7109c83700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f710410d010 0x7f71040d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:59.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f7109c83700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040d8490 0x7f710410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:59.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f7109c83700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710410d480 con 0x7f71040d1da0 2021-04-09T18:13:59.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f7109c83700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71040d8d30 con 0x7f710410d010 2021-04-09T18:13:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f7109c83700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71040d9220 con 0x7f71040d8490 2021-04-09T18:13:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f71037fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f710410d010 0x7f71040d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:59.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f7102ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f71040d1da0 0x7f71040d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52522/0 (socket says 172.21.15.32:52522) 2021-04-09T18:13:59.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f7102ffd700 1 -- 172.21.15.32:0/1863265433 learned_addr learned my addr 172.21.15.32:0/1863265433 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:13:59.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f7103fff700 1 --2- 172.21.15.32:0/1863265433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040d8490 0x7f710410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f71037fe700 1 -- 172.21.15.32:0/1863265433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040d8490 msgr2=0x7f710410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:13:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f71037fe700 1 --2- 172.21.15.32:0/1863265433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040d8490 0x7f710410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f71037fe700 1 -- 172.21.15.32:0/1863265433 >> v1:172.21.15.32:6789/0 conn(0x7f71040d1da0 legacy=0x7f71040d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f71027fc700 1 -- 172.21.15.32:0/1863265433 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343733514 0 0) 0x7f70f8000d30 con 0x7f71040d1da0 2021-04-09T18:13:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.312 7f71037fe700 1 -- 172.21.15.32:0/1863265433 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710410e020 con 0x7f710410d010 2021-04-09T18:13:59.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f71037fe700 1 --2- 172.21.15.32:0/1863265433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f710410d010 0x7f71040d1780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f70f4006650 tx=0x7f70f4009a00).ready entity=mon.0 client_cookie=b73cb5a88884855a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:59.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f71027fc700 1 -- 172.21.15.32:0/1863265433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70f4009df0 con 0x7f710410d010 2021-04-09T18:13:59.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f71027fc700 1 -- 172.21.15.32:0/1863265433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f40009b0 con 0x7f710410d010 2021-04-09T18:13:59.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f71027fc700 1 -- 172.21.15.32:0/1863265433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70f400c2b0 con 0x7f710410d010 2021-04-09T18:13:59.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f7109c83700 1 -- 172.21.15.32:0/1863265433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f710410d010 msgr2=0x7f71040d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:59.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f7109c83700 1 --2- 172.21.15.32:0/1863265433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f710410d010 0x7f71040d1780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f70f4006650 tx=0x7f70f4009a00).stop 2021-04-09T18:13:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f7109c83700 1 -- 172.21.15.32:0/1863265433 shutdown_connections 2021-04-09T18:13:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f7109c83700 1 --2- 172.21.15.32:0/1863265433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040d8490 0x7f710410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f7109c83700 1 --2- 172.21.15.32:0/1863265433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f710410d010 0x7f71040d1780 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.313 7f7109c83700 1 -- 172.21.15.32:0/1863265433 >> 172.21.15.32:0/1863265433 conn(0x7f71040ce5f0 msgr2=0x7f71040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:59.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.314 7f7109c83700 1 -- 172.21.15.32:0/1863265433 shutdown_connections 2021-04-09T18:13:59.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.314 7f7109c83700 1 -- 172.21.15.32:0/1863265433 wait complete. 2021-04-09T18:13:59.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.334 7f7109c83700 1 Processor -- start 2021-04-09T18:13:59.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.334 7f7109c83700 1 -- start start 2021-04-09T18:13:59.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.334 7f7109c83700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040b5360 0x7f71041a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:59.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.334 7f7109c83700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710418e140 0x7f71041aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:59.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.334 7f7109c83700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71041ad040 0x7f710408fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:59.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.334 7f7109c83700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71041a5360 con 0x7f71041ad040 2021-04-09T18:13:59.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.334 7f7109c83700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71040902c0 con 0x7f710418e140 2021-04-09T18:13:59.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.334 7f7109c83700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71040904e0 con 0x7f71040b5360 2021-04-09T18:13:59.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7103fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71041ad040 0x7f710408fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:59.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7103fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71041ad040 0x7f710408fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42938/0 (socket says 172.21.15.32:42938) 2021-04-09T18:13:59.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7103fff700 1 -- 172.21.15.32:0/1079654418 learned_addr learned my addr 172.21.15.32:0/1079654418 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:13:59.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f71037fe700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040b5360 0x7f71041a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:59.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7102ffd700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710418e140 0x7f71041aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7103fff700 1 -- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040b5360 msgr2=0x7f71041a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:59.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7103fff700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040b5360 0x7f71041a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7103fff700 1 -- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710418e140 msgr2=0x7f71041aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7103fff700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710418e140 0x7f71041aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7103fff700 1 -- 172.21.15.32:0/1079654418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7104183c80 con 0x7f71041ad040 2021-04-09T18:13:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7102ffd700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710418e140 0x7f71041aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:13:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7103fff700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71041ad040 0x7f710408fca0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f70fc006f20 tx=0x7f70fc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:59.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f70ebfff700 1 -- 172.21.15.32:0/1079654418 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70fc006bb0 con 0x7f71041ad040 2021-04-09T18:13:59.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f70ebfff700 1 -- 172.21.15.32:0/1079654418 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc00bd80 con 0x7f71041ad040 2021-04-09T18:13:59.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7109c83700 1 -- 172.21.15.32:0/1079654418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7104183c80 con 0x7f71041ad040 2021-04-09T18:13:59.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f70ebfff700 1 -- 172.21.15.32:0/1079654418 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70fc002220 con 0x7f71041ad040 2021-04-09T18:13:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.335 7f7109c83700 1 -- 172.21.15.32:0/1079654418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7104183c80 con 0x7f71041ad040 2021-04-09T18:13:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.336 7f70ebfff700 1 -- 172.21.15.32:0/1079654418 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f70fc005330 con 0x7f71041ad040 2021-04-09T18:13:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.336 7f70ebfff700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70e4018930 0x7f70e401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:13:59.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.336 7f70ebfff700 1 -- 172.21.15.32:0/1079654418 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f70fc032a70 con 0x7f71041ad040 2021-04-09T18:13:59.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.336 7f71037fe700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70e4018930 0x7f70e401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:13:59.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.337 7f71037fe700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70e4018930 0x7f70e401ae10 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:13:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.356 7f7109c83700 1 -- 172.21.15.32:0/1079654418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7104182d70 con 0x7f71041ad040 2021-04-09T18:13:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.360 7f70ebfff700 1 -- 172.21.15.32:0/1079654418 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f70fc006370 con 0x7f71041ad040 2021-04-09T18:13:59.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.553 7f7109c83700 1 -- 172.21.15.32:0/1079654418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7104182d70 con 0x7f71041ad040 2021-04-09T18:13:59.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.553 7f70ebfff700 1 -- 172.21.15.32:0/1079654418 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7f70fc00bd80 con 0x7f71041ad040 2021-04-09T18:13:59.577 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:13:59.578 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:13:59.579 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:13:59.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.555 7f7109c83700 1 -- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70e4018930 msgr2=0x7f70e401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:59.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.555 7f7109c83700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70e4018930 0x7f70e401ae10 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.555 7f7109c83700 1 -- 172.21.15.32:0/1079654418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71041ad040 msgr2=0x7f710408fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:13:59.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.555 7f7109c83700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71041ad040 0x7f710408fca0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f70fc006f20 tx=0x7f70fc00b990).stop 2021-04-09T18:13:59.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.556 7f7109c83700 1 -- 172.21.15.32:0/1079654418 shutdown_connections 2021-04-09T18:13:59.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.556 7f7109c83700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70e4018930 0x7f70e401ae10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.556 7f7109c83700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71041ad040 0x7f710408fca0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.556 7f7109c83700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710418e140 0x7f71041aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.556 7f7109c83700 1 --2- 172.21.15.32:0/1079654418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040b5360 0x7f71041a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:13:59.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.556 7f7109c83700 1 -- 172.21.15.32:0/1079654418 >> 172.21.15.32:0/1079654418 conn(0x7f710419b460 msgr2=0x7f710419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:13:59.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.556 7f7109c83700 1 -- 172.21.15.32:0/1079654418 shutdown_connections 2021-04-09T18:13:59.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:13:59.556 7f7109c83700 1 -- 172.21.15.32:0/1079654418 wait complete. 2021-04-09T18:13:59.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:13:59.995 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:14:00.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:01.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.131 7fc100040700 1 Processor -- start 2021-04-09T18:14:01.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.131 7fc100040700 1 -- start start 2021-04-09T18:14:01.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc100040700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f810d010 0x7fc0f80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:01.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc100040700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0f80d1da0 0x7fc0f80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:01.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc100040700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f810d480 con 0x7fc0f80d82c0 2021-04-09T18:14:01.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc100040700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f810e780 con 0x7fc0f80d1da0 2021-04-09T18:14:01.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc100040700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f80d8d30 con 0x7fc0f810d010 2021-04-09T18:14:01.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fd65b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0f80d1da0 0x7fc0f80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:01.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fd65b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0f80d1da0 0x7fc0f80d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42944/0 (socket says 172.21.15.32:42944) 2021-04-09T18:14:01.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fd65b700 1 -- 172.21.15.32:0/86151760 learned_addr learned my addr 172.21.15.32:0/86151760 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fde5c700 1 --2- 172.21.15.32:0/86151760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f810d010 0x7fc0f80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:01.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fd65b700 1 -- 172.21.15.32:0/86151760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f810d010 msgr2=0x7fc0f80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:01.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fd65b700 1 --2- 172.21.15.32:0/86151760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f810d010 0x7fc0f80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fd65b700 1 -- 172.21.15.32:0/86151760 >> v1:172.21.15.86:6789/0 conn(0x7fc0f80d82c0 legacy=0x7fc0f810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:01.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fd65b700 1 -- 172.21.15.32:0/86151760 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f80d9220 con 0x7fc0f80d1da0 2021-04-09T18:14:01.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fde5c700 1 --2- 172.21.15.32:0/86151760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f810d010 0x7fc0f80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:14:01.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.132 7fc0fd65b700 1 --2- 172.21.15.32:0/86151760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80d1da0 0x7fc0f80d7ca0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fc0e8006700 tx=0x7fc0e8009b80).ready entity=mon.0 client_cookie=9ab6b423b700e8a6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:01.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.133 7fc0fce5a700 1 -- 172.21.15.32:0/86151760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e800c0c0 con 0x7fc0f80d1da0 2021-04-09T18:14:01.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.133 7fc0fce5a700 1 -- 172.21.15.32:0/86151760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e800c440 con 0x7fc0f80d1da0 2021-04-09T18:14:01.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.133 7fc0fce5a700 1 -- 172.21.15.32:0/86151760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0e800c920 con 0x7fc0f80d1da0 2021-04-09T18:14:01.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.133 7fc100040700 1 -- 172.21.15.32:0/86151760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80d1da0 msgr2=0x7fc0f80d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:01.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.133 7fc100040700 1 --2- 172.21.15.32:0/86151760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80d1da0 0x7fc0f80d7ca0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fc0e8006700 tx=0x7fc0e8009b80).stop 2021-04-09T18:14:01.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.133 7fc100040700 1 -- 172.21.15.32:0/86151760 shutdown_connections 2021-04-09T18:14:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.133 7fc100040700 1 --2- 172.21.15.32:0/86151760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80d1da0 0x7fc0f80d7ca0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.133 7fc100040700 1 --2- 172.21.15.32:0/86151760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f810d010 0x7fc0f80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.133 7fc100040700 1 -- 172.21.15.32:0/86151760 >> 172.21.15.32:0/86151760 conn(0x7fc0f80ce5f0 msgr2=0x7fc0f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.134 7fc100040700 1 -- 172.21.15.32:0/86151760 shutdown_connections 2021-04-09T18:14:01.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.134 7fc100040700 1 -- 172.21.15.32:0/86151760 wait complete. 2021-04-09T18:14:01.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.154 7fc100040700 1 Processor -- start 2021-04-09T18:14:01.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.154 7fc100040700 1 -- start start 2021-04-09T18:14:01.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.154 7fc100040700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f81a5210 0x7fc0f81a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:01.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.154 7fc100040700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f81a6340 0x7fc0f81a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:01.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.154 7fc100040700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80abaa0 0x7fc0f80abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:01.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.154 7fc100040700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f81a6d60 con 0x7fc0f80abaa0 2021-04-09T18:14:01.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.154 7fc100040700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f80ac4c0 con 0x7fc0f81a5210 2021-04-09T18:14:01.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.154 7fc100040700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f80ac6e0 con 0x7fc0f81a6340 2021-04-09T18:14:01.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fe65d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80abaa0 0x7fc0f80abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:01.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fe65d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80abaa0 0x7fc0f80abea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42948/0 (socket says 172.21.15.32:42948) 2021-04-09T18:14:01.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fe65d700 1 -- 172.21.15.32:0/1849538615 learned_addr learned my addr 172.21.15.32:0/1849538615 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:01.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fd65b700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f81a6340 0x7fc0f81a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:01.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fde5c700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f81a5210 0x7fc0f81a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:01.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fe65d700 1 -- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f81a6340 msgr2=0x7fc0f81a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fe65d700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f81a6340 0x7fc0f81a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fe65d700 1 -- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f81a5210 msgr2=0x7fc0f81a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:01.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fe65d700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f81a5210 0x7fc0f81a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fe65d700 1 -- 172.21.15.32:0/1849538615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f817a720 con 0x7fc0f80abaa0 2021-04-09T18:14:01.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0fe65d700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80abaa0 0x7fc0f80abea0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fc0f0009940 tx=0x7fc0f000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:01.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0ee7fc700 1 -- 172.21.15.32:0/1849538615 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0f000c2f0 con 0x7fc0f80abaa0 2021-04-09T18:14:01.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0ee7fc700 1 -- 172.21.15.32:0/1849538615 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f000c570 con 0x7fc0f80abaa0 2021-04-09T18:14:01.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc100040700 1 -- 172.21.15.32:0/1849538615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f817a720 con 0x7fc0f80abaa0 2021-04-09T18:14:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc0ee7fc700 1 -- 172.21.15.32:0/1849538615 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0f000c9d0 con 0x7fc0f80abaa0 2021-04-09T18:14:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.155 7fc100040700 1 -- 172.21.15.32:0/1849538615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f817a720 con 0x7fc0f80abaa0 2021-04-09T18:14:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.156 7fc0ee7fc700 1 -- 172.21.15.32:0/1849538615 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc0f0003080 con 0x7fc0f80abaa0 2021-04-09T18:14:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.156 7fc0ee7fc700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0e0018930 0x7fc0e001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:01.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.156 7fc0ee7fc700 1 -- 172.21.15.32:0/1849538615 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc0f0032dd0 con 0x7fc0f80abaa0 2021-04-09T18:14:01.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.156 7fc0fde5c700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0e0018930 0x7fc0e001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:01.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.157 7fc0fde5c700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0e0018930 0x7fc0e001ae10 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:01.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.176 7fc100040700 1 -- 172.21.15.32:0/1849538615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f81b7070 con 0x7fc0f80abaa0 2021-04-09T18:14:01.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.179 7fc0ee7fc700 1 -- 172.21.15.32:0/1849538615 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc0f000be30 con 0x7fc0f80abaa0 2021-04-09T18:14:01.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.370 7fc100040700 1 -- 172.21.15.32:0/1849538615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0f810d480 con 0x7fc0f80abaa0 2021-04-09T18:14:01.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.370 7fc0ee7fc700 1 -- 172.21.15.32:0/1849538615 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7fc0f0003000 con 0x7fc0f80abaa0 2021-04-09T18:14:01.394 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:14:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:01.408 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:01.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 -- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0e0018930 msgr2=0x7fc0e001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:01.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0e0018930 0x7fc0e001ae10 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 -- 172.21.15.32:0/1849538615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80abaa0 msgr2=0x7fc0f80abea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:01.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80abaa0 0x7fc0f80abea0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fc0f0009940 tx=0x7fc0f000bcf0).stop 2021-04-09T18:14:01.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 -- 172.21.15.32:0/1849538615 shutdown_connections 2021-04-09T18:14:01.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0e0018930 0x7fc0e001ae10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0f80abaa0 0x7fc0f80abea0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0f81a6340 0x7fc0f81a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 --2- 172.21.15.32:0/1849538615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f81a5210 0x7fc0f81a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 -- 172.21.15.32:0/1849538615 >> 172.21.15.32:0/1849538615 conn(0x7fc0f81a99c0 msgr2=0x7fc0f817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 -- 172.21.15.32:0/1849538615 shutdown_connections 2021-04-09T18:14:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:01.373 7fc100040700 1 -- 172.21.15.32:0/1849538615 wait complete. 2021-04-09T18:14:01.853 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:01.854 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:14:02.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:03.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.997 7f0d8affe700 1 Processor -- start 2021-04-09T18:14:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.997 7f0d8affe700 1 -- start start 2021-04-09T18:14:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d8affe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0d8410d010 0x7f0d840d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:03.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d8affe700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8410d480 con 0x7f0d840d1da0 2021-04-09T18:14:03.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d8affe700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8410e1f0 con 0x7f0d840d8490 2021-04-09T18:14:03.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d8affe700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d840d8d30 con 0x7f0d8410d010 2021-04-09T18:14:03.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d83fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0d840d1da0 0x7f0d840d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52546/0 (socket says 172.21.15.32:52546) 2021-04-09T18:14:03.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d83fff700 1 -- 172.21.15.32:0/2626735481 learned_addr learned my addr 172.21.15.32:0/2626735481 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:03.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d88e1a700 1 --2- 172.21.15.32:0/2626735481 >> v2:172.21.15.86:3300/0 conn(0x7f0d8410d010 0x7f0d840d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:03.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d837fe700 1 -- 172.21.15.32:0/2626735481 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341220953 0 0) 0x7f0d74000ce0 con 0x7f0d840d1da0 2021-04-09T18:14:03.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d837fe700 1 -- 172.21.15.32:0/2626735481 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d70002fe0 con 0x7f0d840d1da0 2021-04-09T18:14:03.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d837fe700 1 -- 172.21.15.32:0/2626735481 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373485276 0 0) 0x7f0d7c000d30 con 0x7f0d840d8490 2021-04-09T18:14:03.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d837fe700 1 -- 172.21.15.32:0/2626735481 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d70003210 con 0x7f0d840d8490 2021-04-09T18:14:03.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d88e1a700 1 -- 172.21.15.32:0/2626735481 >> v1:172.21.15.86:6789/0 conn(0x7f0d840d8490 legacy=0x7f0d8410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:03.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d88e1a700 1 -- 172.21.15.32:0/2626735481 >> v1:172.21.15.32:6789/0 conn(0x7f0d840d1da0 legacy=0x7f0d840d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:03.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.998 7f0d88e1a700 1 -- 172.21.15.32:0/2626735481 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d840d9220 con 0x7f0d8410d010 2021-04-09T18:14:03.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.999 7f0d88e1a700 1 --2- 172.21.15.32:0/2626735481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8410d010 0x7f0d840d1780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0d6c006650 tx=0x7f0d6c009a80).ready entity=mon.1 client_cookie=f7bec6ae3cd48a34 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:03.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.999 7f0d837fe700 1 -- 172.21.15.32:0/2626735481 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d6c0009b0 con 0x7f0d8410d010 2021-04-09T18:14:03.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.999 7f0d837fe700 1 -- 172.21.15.32:0/2626735481 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c00c0c0 con 0x7f0d8410d010 2021-04-09T18:14:03.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.999 7f0d837fe700 1 -- 172.21.15.32:0/2626735481 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d6c00c510 con 0x7f0d8410d010 2021-04-09T18:14:03.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.999 7f0d8affe700 1 -- 172.21.15.32:0/2626735481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8410d010 msgr2=0x7f0d840d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:03.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:02.999 7f0d8affe700 1 --2- 172.21.15.32:0/2626735481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8410d010 0x7f0d840d1780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0d6c006650 tx=0x7f0d6c009a80).stop 2021-04-09T18:14:03.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.000 7f0d8affe700 1 -- 172.21.15.32:0/2626735481 shutdown_connections 2021-04-09T18:14:03.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.000 7f0d8affe700 1 --2- 172.21.15.32:0/2626735481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d8410d010 0x7f0d840d1780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:03.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.000 7f0d8affe700 1 -- 172.21.15.32:0/2626735481 >> 172.21.15.32:0/2626735481 conn(0x7f0d840ce5f0 msgr2=0x7f0d840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:03.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.000 7f0d8affe700 1 -- 172.21.15.32:0/2626735481 shutdown_connections 2021-04-09T18:14:03.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.000 7f0d8affe700 1 -- 172.21.15.32:0/2626735481 wait complete. 2021-04-09T18:14:03.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.020 7f0d8affe700 1 Processor -- start 2021-04-09T18:14:03.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.020 7f0d8affe700 1 -- start start 2021-04-09T18:14:03.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.020 7f0d8affe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d84185a00 0x7f0d84097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:03.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.020 7f0d8affe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d840983b0 0x7f0d8417abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:03.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.020 7f0d8affe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8417b1d0 0x7f0d84179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:03.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.020 7f0d8affe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d840987b0 con 0x7f0d8417b1d0 2021-04-09T18:14:03.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.020 7f0d8affe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8417b5d0 con 0x7f0d840983b0 2021-04-09T18:14:03.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.020 7f0d8affe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d84179c30 con 0x7f0d84185a00 2021-04-09T18:14:03.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.020 7f0d8961b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8417b1d0 0x7f0d84179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:03.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8961b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8417b1d0 0x7f0d84179610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42966/0 (socket says 172.21.15.32:42966) 2021-04-09T18:14:03.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8961b700 1 -- 172.21.15.32:0/4146619000 learned_addr learned my addr 172.21.15.32:0/4146619000 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:03.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d83fff700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d840983b0 0x7f0d8417abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:03.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d88e1a700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d84185a00 0x7f0d84097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:03.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8961b700 1 -- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d84185a00 msgr2=0x7f0d84097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:03.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8961b700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d84185a00 0x7f0d84097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:03.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8961b700 1 -- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d840983b0 msgr2=0x7f0d8417abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:03.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8961b700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d840983b0 0x7f0d8417abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8961b700 1 -- 172.21.15.32:0/4146619000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d84179e50 con 0x7f0d8417b1d0 2021-04-09T18:14:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8961b700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8417b1d0 0x7f0d84179610 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f0d7c009980 tx=0x7f0d7c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d817fa700 1 -- 172.21.15.32:0/4146619000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d7c00be10 con 0x7f0d8417b1d0 2021-04-09T18:14:03.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d817fa700 1 -- 172.21.15.32:0/4146619000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7c005870 con 0x7f0d8417b1d0 2021-04-09T18:14:03.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d84179e50 con 0x7f0d8417b1d0 2021-04-09T18:14:03.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d817fa700 1 -- 172.21.15.32:0/4146619000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d7c005d50 con 0x7f0d8417b1d0 2021-04-09T18:14:03.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.021 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d84179e50 con 0x7f0d8417b1d0 2021-04-09T18:14:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.022 7f0d817fa700 1 -- 172.21.15.32:0/4146619000 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0d7c005a50 con 0x7f0d8417b1d0 2021-04-09T18:14:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.022 7f0d817fa700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d68018930 0x7f0d6801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.022 7f0d817fa700 1 -- 172.21.15.32:0/4146619000 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0d7c032d80 con 0x7f0d8417b1d0 2021-04-09T18:14:03.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.022 7f0d88e1a700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d68018930 0x7f0d6801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:03.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.022 7f0d88e1a700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d68018930 0x7f0d6801ae10 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.042 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8417a3e0 con 0x7f0d8417b1d0 2021-04-09T18:14:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.045 7f0d817fa700 1 -- 172.21.15.32:0/4146619000 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0d7c00be10 con 0x7f0d8417b1d0 2021-04-09T18:14:03.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.238 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d8417a3e0 con 0x7f0d8417b1d0 2021-04-09T18:14:03.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.238 7f0d817fa700 1 -- 172.21.15.32:0/4146619000 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7f0d7c005870 con 0x7f0d8417b1d0 2021-04-09T18:14:03.309 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:14:03.309 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:03.310 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d68018930 msgr2=0x7f0d6801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:03.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d68018930 0x7f0d6801ae10 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8417b1d0 msgr2=0x7f0d84179610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8417b1d0 0x7f0d84179610 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f0d7c009980 tx=0x7f0d7c00bcd0).stop 2021-04-09T18:14:03.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 shutdown_connections 2021-04-09T18:14:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d68018930 0x7f0d6801ae10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d8417b1d0 0x7f0d84179610 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d840983b0 0x7f0d8417abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 --2- 172.21.15.32:0/4146619000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d84185a00 0x7f0d84097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:03.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 >> 172.21.15.32:0/4146619000 conn(0x7f0d84190550 msgr2=0x7f0d841b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:03.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 shutdown_connections 2021-04-09T18:14:03.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:03.241 7f0d8affe700 1 -- 172.21.15.32:0/4146619000 wait complete. 2021-04-09T18:14:03.797 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:03.797 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:14:04.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:14:04.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:14:04.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:04.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.937 7f71f4668700 1 Processor -- start 2021-04-09T18:14:04.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.937 7f71f4668700 1 -- start start 2021-04-09T18:14:04.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.938 7f71f4668700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0cffa0 0x7f71ec0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:04.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.938 7f71f4668700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f71ec0d7800 0x7f71ec0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:04.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.938 7f71f4668700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ec0d7280 con 0x7f71ec0d5c70 2021-04-09T18:14:04.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.938 7f71f4668700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ec0d8580 con 0x7f71ec0d7800 2021-04-09T18:14:04.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.938 7f71f4668700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ec0d91e0 con 0x7f71ec0cffa0 2021-04-09T18:14:04.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.938 7f71f2484700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0cffa0 0x7f71ec0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:04.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.938 7f71f1c83700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f71ec0d7800 0x7f71ec0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:04.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.938 7f71f2c85700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f71ec0d5c70 0x7f71ec0d6070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:50488/0 (socket says 172.21.15.32:50488) 2021-04-09T18:14:04.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.938 7f71f2c85700 1 -- 172.21.15.32:0/774065410 learned_addr learned my addr 172.21.15.32:0/774065410 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:04.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f1c83700 1 -- 172.21.15.32:0/774065410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0cffa0 msgr2=0x7f71ec0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:04.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f1c83700 1 --2- 172.21.15.32:0/774065410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0cffa0 0x7f71ec0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:04.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f1c83700 1 -- 172.21.15.32:0/774065410 >> v1:172.21.15.86:6789/0 conn(0x7f71ec0d5c70 legacy=0x7f71ec0d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:04.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f1c83700 1 -- 172.21.15.32:0/774065410 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec0d8220 con 0x7f71ec0d7800 2021-04-09T18:14:04.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f1482700 1 -- 172.21.15.32:0/774065410 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836984450 0 0) 0x7f71e4001440 con 0x7f71ec0d5c70 2021-04-09T18:14:04.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f2484700 1 --2- 172.21.15.32:0/774065410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0cffa0 0x7f71ec0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:04.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f1c83700 1 --2- 172.21.15.32:0/774065410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec0d7800 0x7f71ec0d7c00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f71e0006650 tx=0x7f71e0009a80).ready entity=mon.1 client_cookie=24222a856da301ce server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:04.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f1482700 1 -- 172.21.15.32:0/774065410 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71e00009b0 con 0x7f71ec0d7800 2021-04-09T18:14:04.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f1482700 1 -- 172.21.15.32:0/774065410 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e000c0c0 con 0x7f71ec0d7800 2021-04-09T18:14:04.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f1482700 1 -- 172.21.15.32:0/774065410 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71e000c510 con 0x7f71ec0d7800 2021-04-09T18:14:04.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f4668700 1 -- 172.21.15.32:0/774065410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec0d7800 msgr2=0x7f71ec0d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:05.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.939 7f71f4668700 1 --2- 172.21.15.32:0/774065410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec0d7800 0x7f71ec0d7c00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f71e0006650 tx=0x7f71e0009a80).stop 2021-04-09T18:14:05.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.940 7f71f4668700 1 -- 172.21.15.32:0/774065410 shutdown_connections 2021-04-09T18:14:05.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.940 7f71f4668700 1 --2- 172.21.15.32:0/774065410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec0d7800 0x7f71ec0d7c00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:05.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.940 7f71f4668700 1 --2- 172.21.15.32:0/774065410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0cffa0 0x7f71ec0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:05.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.940 7f71f4668700 1 -- 172.21.15.32:0/774065410 >> 172.21.15.32:0/774065410 conn(0x7f71ec0ce5c0 msgr2=0x7f71ec0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:05.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.940 7f71f4668700 1 -- 172.21.15.32:0/774065410 shutdown_connections 2021-04-09T18:14:05.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.940 7f71f4668700 1 -- 172.21.15.32:0/774065410 wait complete. 2021-04-09T18:14:05.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.960 7f71f4668700 1 Processor -- start 2021-04-09T18:14:05.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.960 7f71f4668700 1 -- start start 2021-04-09T18:14:05.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f4668700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ec1842b0 0x7f71ec1acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:05.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f4668700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec1847e0 0x7f71ec1857d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:05.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f4668700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0b6ff0 0x7f71ec0b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:05.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f4668700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ec184be0 con 0x7f71ec1842b0 2021-04-09T18:14:05.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f4668700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ec18d0f0 con 0x7f71ec1847e0 2021-04-09T18:14:05.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f4668700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ec1861e0 con 0x7f71ec0b6ff0 2021-04-09T18:14:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2484700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ec1842b0 0x7f71ec1acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:05.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2484700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ec1842b0 0x7f71ec1acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42976/0 (socket says 172.21.15.32:42976) 2021-04-09T18:14:05.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f1c83700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec1847e0 0x7f71ec1857d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2484700 1 -- 172.21.15.32:0/2124824613 learned_addr learned my addr 172.21.15.32:0/2124824613 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2c85700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0b6ff0 0x7f71ec0b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2484700 1 -- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0b6ff0 msgr2=0x7f71ec0b73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2484700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0b6ff0 0x7f71ec0b73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2484700 1 -- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec1847e0 msgr2=0x7f71ec1857d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2484700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec1847e0 0x7f71ec1857d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2484700 1 -- 172.21.15.32:0/2124824613 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec185df0 con 0x7f71ec1842b0 2021-04-09T18:14:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f1c83700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec1847e0 0x7f71ec1857d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f2484700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ec1842b0 0x7f71ec1acd90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f71e8006f40 tx=0x7f71e800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:05.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71deffd700 1 -- 172.21.15.32:0/2124824613 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71e800bd80 con 0x7f71ec1842b0 2021-04-09T18:14:05.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71deffd700 1 -- 172.21.15.32:0/2124824613 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e8000c90 con 0x7f71ec1842b0 2021-04-09T18:14:05.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71f4668700 1 -- 172.21.15.32:0/2124824613 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ec185df0 con 0x7f71ec1842b0 2021-04-09T18:14:05.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.961 7f71deffd700 1 -- 172.21.15.32:0/2124824613 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71e8002220 con 0x7f71ec1842b0 2021-04-09T18:14:05.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.962 7f71f4668700 1 -- 172.21.15.32:0/2124824613 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ec185df0 con 0x7f71ec1842b0 2021-04-09T18:14:05.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.962 7f71deffd700 1 -- 172.21.15.32:0/2124824613 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f71e800a000 con 0x7f71ec1842b0 2021-04-09T18:14:05.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.962 7f71deffd700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71d4018930 0x7f71d401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:05.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.962 7f71deffd700 1 -- 172.21.15.32:0/2124824613 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f71e8032b60 con 0x7f71ec1842b0 2021-04-09T18:14:05.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.962 7f71f1c83700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71d4018930 0x7f71d401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:05.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.963 7f71f1c83700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71d4018930 0x7f71d401adc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:05.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.982 7f71f4668700 1 -- 172.21.15.32:0/2124824613 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ec184be0 con 0x7f71ec1842b0 2021-04-09T18:14:05.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:04.985 7f71deffd700 1 -- 172.21.15.32:0/2124824613 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f71e8000c90 con 0x7f71ec1842b0 2021-04-09T18:14:05.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.177 7f71f4668700 1 -- 172.21.15.32:0/2124824613 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71ec184be0 con 0x7f71ec1842b0 2021-04-09T18:14:05.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.178 7f71deffd700 1 -- 172.21.15.32:0/2124824613 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7f71e8006b00 con 0x7f71ec1842b0 2021-04-09T18:14:05.236 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:14:05.236 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:05.236 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:05.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 -- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71d4018930 msgr2=0x7f71d401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:05.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71d4018930 0x7f71d401adc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:05.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 -- 172.21.15.32:0/2124824613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ec1842b0 msgr2=0x7f71ec1acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:05.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ec1842b0 0x7f71ec1acd90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f71e8006f40 tx=0x7f71e800b990).stop 2021-04-09T18:14:05.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 -- 172.21.15.32:0/2124824613 shutdown_connections 2021-04-09T18:14:05.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71d4018930 0x7f71d401adc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:05.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71ec0b6ff0 0x7f71ec0b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:05.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71ec1847e0 0x7f71ec1857d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:05.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 --2- 172.21.15.32:0/2124824613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71ec1842b0 0x7f71ec1acd90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:05.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 -- 172.21.15.32:0/2124824613 >> 172.21.15.32:0/2124824613 conn(0x7f71ec183840 msgr2=0x7f71ec19b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:05.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 -- 172.21.15.32:0/2124824613 shutdown_connections 2021-04-09T18:14:05.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:05.180 7f71f4668700 1 -- 172.21.15.32:0/2124824613 wait complete. 2021-04-09T18:14:05.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:05.666 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:14:06.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:06.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.809 7f681479c700 1 Processor -- start 2021-04-09T18:14:06.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.809 7f681479c700 1 -- start start 2021-04-09T18:14:06.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.809 7f681479c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f680c0cffa0 0x7f680c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:06.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.809 7f681479c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c0d7800 0x7f680c0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:06.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f681479c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680c0d7280 con 0x7f680c0d8220 2021-04-09T18:14:06.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f681479c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680c0d91e0 con 0x7f680c0cffa0 2021-04-09T18:14:06.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f681479c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c10dfe0 con 0x7f680c0d7800 2021-04-09T18:14:06.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f6811db7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c0d7800 0x7f680c0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:06.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f68125b8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f680c0cffa0 0x7f680c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:06.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f6812db9700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f680c0d8220 0x7f680c0d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:50500/0 (socket says 172.21.15.32:50500) 2021-04-09T18:14:06.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f6812db9700 1 -- 172.21.15.32:0/1895644248 learned_addr learned my addr 172.21.15.32:0/1895644248 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:06.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f68125b8700 1 -- 172.21.15.32:0/1895644248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c0d7800 msgr2=0x7f680c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:06.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f68125b8700 1 --2- 172.21.15.32:0/1895644248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c0d7800 0x7f680c0d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:06.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f68125b8700 1 -- 172.21.15.32:0/1895644248 >> v1:172.21.15.86:6789/0 conn(0x7f680c0d8220 legacy=0x7f680c0d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:06.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f68125b8700 1 -- 172.21.15.32:0/1895644248 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c10e200 con 0x7f680c0cffa0 2021-04-09T18:14:06.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f68115b6700 1 -- 172.21.15.32:0/1895644248 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289520618 0 0) 0x7f68040010c0 con 0x7f680c0d8220 2021-04-09T18:14:06.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f6811db7700 1 --2- 172.21.15.32:0/1895644248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c0d7800 0x7f680c0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:06.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.810 7f68125b8700 1 --2- 172.21.15.32:0/1895644248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f680c0cffa0 0x7f680c0d2440 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f68000069d0 tx=0x7f6800009e40).ready entity=mon.1 client_cookie=24500255f0535768 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:06.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.811 7f68115b6700 1 -- 172.21.15.32:0/1895644248 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f680000b510 con 0x7f680c0cffa0 2021-04-09T18:14:06.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.811 7f68115b6700 1 -- 172.21.15.32:0/1895644248 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680000b890 con 0x7f680c0cffa0 2021-04-09T18:14:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.811 7f68115b6700 1 -- 172.21.15.32:0/1895644248 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f680000bd70 con 0x7f680c0cffa0 2021-04-09T18:14:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.811 7f681479c700 1 -- 172.21.15.32:0/1895644248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f680c0cffa0 msgr2=0x7f680c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:06.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.811 7f681479c700 1 --2- 172.21.15.32:0/1895644248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f680c0cffa0 0x7f680c0d2440 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f68000069d0 tx=0x7f6800009e40).stop 2021-04-09T18:14:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.811 7f681479c700 1 -- 172.21.15.32:0/1895644248 shutdown_connections 2021-04-09T18:14:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.811 7f681479c700 1 --2- 172.21.15.32:0/1895644248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c0d7800 0x7f680c0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.811 7f681479c700 1 --2- 172.21.15.32:0/1895644248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f680c0cffa0 0x7f680c0d2440 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.811 7f681479c700 1 -- 172.21.15.32:0/1895644248 >> 172.21.15.32:0/1895644248 conn(0x7f680c0ce5c0 msgr2=0x7f680c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:06.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.812 7f681479c700 1 -- 172.21.15.32:0/1895644248 shutdown_connections 2021-04-09T18:14:06.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.812 7f681479c700 1 -- 172.21.15.32:0/1895644248 wait complete. 2021-04-09T18:14:06.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f681479c700 1 Processor -- start 2021-04-09T18:14:06.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f681479c700 1 -- start start 2021-04-09T18:14:06.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f681479c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f680c1acff0 0x7f680c1847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:06.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f681479c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f680c183fc0 0x7f680c1a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:06.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f681479c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c186ff0 0x7f680c1873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:06.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f681479c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c188c30 con 0x7f680c183fc0 2021-04-09T18:14:06.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f681479c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c1843c0 con 0x7f680c1acff0 2021-04-09T18:14:06.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f681479c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c1a5c20 con 0x7f680c186ff0 2021-04-09T18:14:06.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f6811db7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f680c183fc0 0x7f680c1a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:06.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f6811db7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f680c183fc0 0x7f680c1a5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42988/0 (socket says 172.21.15.32:42988) 2021-04-09T18:14:06.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f6811db7700 1 -- 172.21.15.32:0/3243528187 learned_addr learned my addr 172.21.15.32:0/3243528187 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:06.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f68125b8700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f680c1acff0 0x7f680c1847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:06.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.832 7f6812db9700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c186ff0 0x7f680c1873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:06.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f6811db7700 1 -- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c186ff0 msgr2=0x7f680c1873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:06.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f6811db7700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c186ff0 0x7f680c1873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:06.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f6811db7700 1 -- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f680c1acff0 msgr2=0x7f680c1847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:06.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f6811db7700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f680c1acff0 0x7f680c1847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:06.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f6811db7700 1 -- 172.21.15.32:0/3243528187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c187a10 con 0x7f680c183fc0 2021-04-09T18:14:06.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f6811db7700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f680c183fc0 0x7f680c1a5600 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f67fc006f40 tx=0x7f67fc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:06.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f680affd700 1 -- 172.21.15.32:0/3243528187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67fc00bd80 con 0x7f680c183fc0 2021-04-09T18:14:06.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f680affd700 1 -- 172.21.15.32:0/3243528187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc000c90 con 0x7f680c183fc0 2021-04-09T18:14:06.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f681479c700 1 -- 172.21.15.32:0/3243528187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c187a10 con 0x7f680c183fc0 2021-04-09T18:14:06.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f680affd700 1 -- 172.21.15.32:0/3243528187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67fc002220 con 0x7f680c183fc0 2021-04-09T18:14:06.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.833 7f681479c700 1 -- 172.21.15.32:0/3243528187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c187a10 con 0x7f680c183fc0 2021-04-09T18:14:06.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.834 7f680affd700 1 -- 172.21.15.32:0/3243528187 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f67fc000c90 con 0x7f680c183fc0 2021-04-09T18:14:06.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.834 7f680affd700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67ec0188c0 0x7f67ec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:06.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.834 7f680affd700 1 -- 172.21.15.32:0/3243528187 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f67fc032940 con 0x7f680c183fc0 2021-04-09T18:14:06.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.834 7f68125b8700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67ec0188c0 0x7f67ec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:06.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.835 7f68125b8700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67ec0188c0 0x7f67ec01adc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.854 7f681479c700 1 -- 172.21.15.32:0/3243528187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680c182dc0 con 0x7f680c183fc0 2021-04-09T18:14:06.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:06.857 7f680affd700 1 -- 172.21.15.32:0/3243528187 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f67fc006b00 con 0x7f680c183fc0 2021-04-09T18:14:07.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.049 7f681479c700 1 -- 172.21.15.32:0/3243528187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f680c0d7280 con 0x7f680c183fc0 2021-04-09T18:14:07.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.049 7f680affd700 1 -- 172.21.15.32:0/3243528187 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2778 (secure 0 0 0) 0x7f67fc00bd80 con 0x7f680c183fc0 2021-04-09T18:14:07.073 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 15 2021-04-09T18:14:07.074 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:07.074 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:13:49.326681","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":4693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4693":{"gid":4693,"name":"b","rank":0,"incarnation":11,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6835/2784917315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2784917315},{"type":"v1","addr":"172.21.15.86:6835","nonce":2784917315}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:07.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 -- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67ec0188c0 msgr2=0x7f67ec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:07.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67ec0188c0 0x7f67ec01adc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:07.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 -- 172.21.15.32:0/3243528187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f680c183fc0 msgr2=0x7f680c1a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f680c183fc0 0x7f680c1a5600 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f67fc006f40 tx=0x7f67fc00b990).stop 2021-04-09T18:14:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 -- 172.21.15.32:0/3243528187 shutdown_connections 2021-04-09T18:14:07.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67ec0188c0 0x7f67ec01adc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f680c183fc0 0x7f680c1a5600 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f680c186ff0 0x7f680c1873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 --2- 172.21.15.32:0/3243528187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f680c1acff0 0x7f680c1847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:07.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 -- 172.21.15.32:0/3243528187 >> 172.21.15.32:0/3243528187 conn(0x7f680c09abb0 msgr2=0x7f680c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 -- 172.21.15.32:0/3243528187 shutdown_connections 2021-04-09T18:14:07.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:07.052 7f681479c700 1 -- 172.21.15.32:0/3243528187 wait complete. 2021-04-09T18:14:07.495 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:13:49.326681', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 4693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4693': {'gid': 4693, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:stopping', 'state_seq': 4, 'addr': '172.21.15.86:6835/2784917315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2784917315}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2784917315}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:07.496 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4693:up:stopping 2021-04-09T18:14:07.922 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:14:08.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:08.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.633 7f6e611bb700 1 Processor -- start 2021-04-09T18:14:08.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.633 7f6e611bb700 1 -- start start 2021-04-09T18:14:08.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e611bb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6e5c0d7ca0 0x7f6e5c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e611bb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c10d480 con 0x7f6e5c10da00 2021-04-09T18:14:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e611bb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0d86c0 con 0x7f6e5c10d010 2021-04-09T18:14:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e611bb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5c0d9220 con 0x7f6e5c0d7ca0 2021-04-09T18:14:08.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e5b59e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6e5c10da00 0x7f6e5c10de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52588/0 (socket says 172.21.15.32:52588) 2021-04-09T18:14:08.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e5b59e700 1 -- 172.21.15.32:0/1365256369 learned_addr learned my addr 172.21.15.32:0/1365256369 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:08.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e5a59c700 1 --2- 172.21.15.32:0/1365256369 >> v2:172.21.15.86:3300/0 conn(0x7f6e5c0d7ca0 0x7f6e5c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:08.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e59d9b700 1 -- 172.21.15.32:0/1365256369 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885691862 0 0) 0x7f6e54000d30 con 0x7f6e5c10da00 2021-04-09T18:14:08.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e59d9b700 1 -- 172.21.15.32:0/1365256369 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48002fe0 con 0x7f6e5c10da00 2021-04-09T18:14:08.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e59d9b700 1 -- 172.21.15.32:0/1365256369 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099914291 0 0) 0x7f6e4c000c60 con 0x7f6e5c10d010 2021-04-09T18:14:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e59d9b700 1 -- 172.21.15.32:0/1365256369 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48003020 con 0x7f6e5c10d010 2021-04-09T18:14:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e5a59c700 1 -- 172.21.15.32:0/1365256369 >> v1:172.21.15.86:6789/0 conn(0x7f6e5c10d010 legacy=0x7f6e5c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e5a59c700 1 -- 172.21.15.32:0/1365256369 >> v1:172.21.15.32:6789/0 conn(0x7f6e5c10da00 legacy=0x7f6e5c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.634 7f6e5a59c700 1 -- 172.21.15.32:0/1365256369 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c112100 con 0x7f6e5c0d7ca0 2021-04-09T18:14:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.635 7f6e5a59c700 1 --2- 172.21.15.32:0/1365256369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0d7ca0 0x7f6e5c0d80a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f6e44006650 tx=0x7f6e44009a80).ready entity=mon.1 client_cookie=5d24b5d335a92fc7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:08.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.635 7f6e59d9b700 1 -- 172.21.15.32:0/1365256369 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e440009b0 con 0x7f6e5c0d7ca0 2021-04-09T18:14:08.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.635 7f6e59d9b700 1 -- 172.21.15.32:0/1365256369 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4400c0c0 con 0x7f6e5c0d7ca0 2021-04-09T18:14:08.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.635 7f6e59d9b700 1 -- 172.21.15.32:0/1365256369 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4400c510 con 0x7f6e5c0d7ca0 2021-04-09T18:14:08.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.635 7f6e611bb700 1 -- 172.21.15.32:0/1365256369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0d7ca0 msgr2=0x7f6e5c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:08.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.635 7f6e611bb700 1 --2- 172.21.15.32:0/1365256369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0d7ca0 0x7f6e5c0d80a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f6e44006650 tx=0x7f6e44009a80).stop 2021-04-09T18:14:08.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.636 7f6e611bb700 1 -- 172.21.15.32:0/1365256369 shutdown_connections 2021-04-09T18:14:08.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.636 7f6e611bb700 1 --2- 172.21.15.32:0/1365256369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0d7ca0 0x7f6e5c0d80a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:08.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.636 7f6e611bb700 1 -- 172.21.15.32:0/1365256369 >> 172.21.15.32:0/1365256369 conn(0x7f6e5c0ce5f0 msgr2=0x7f6e5c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:08.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.636 7f6e611bb700 1 -- 172.21.15.32:0/1365256369 shutdown_connections 2021-04-09T18:14:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.636 7f6e611bb700 1 -- 172.21.15.32:0/1365256369 wait complete. 2021-04-09T18:14:08.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.656 7f6e611bb700 1 Processor -- start 2021-04-09T18:14:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.656 7f6e611bb700 1 -- start start 2021-04-09T18:14:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.656 7f6e611bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c185a00 0x7f6e5c097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:08.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.656 7f6e611bb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0983b0 0x7f6e5c17abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:08.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.656 7f6e611bb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c17b1d0 0x7f6e5c179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:08.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.656 7f6e611bb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0987b0 con 0x7f6e5c185a00 2021-04-09T18:14:08.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.656 7f6e611bb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c17b5d0 con 0x7f6e5c0983b0 2021-04-09T18:14:08.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.656 7f6e611bb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c179c30 con 0x7f6e5c17b1d0 2021-04-09T18:14:08.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.656 7f6e5ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c185a00 0x7f6e5c097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:08.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c185a00 0x7f6e5c097d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43002/0 (socket says 172.21.15.32:43002) 2021-04-09T18:14:08.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5ad9d700 1 -- 172.21.15.32:0/569958551 learned_addr learned my addr 172.21.15.32:0/569958551 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:08.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5a59c700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0983b0 0x7f6e5c17abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:08.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5b59e700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c17b1d0 0x7f6e5c179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:08.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5ad9d700 1 -- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c17b1d0 msgr2=0x7f6e5c179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:08.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5ad9d700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c17b1d0 0x7f6e5c179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:08.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5ad9d700 1 -- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0983b0 msgr2=0x7f6e5c17abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:08.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5ad9d700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0983b0 0x7f6e5c17abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:08.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5ad9d700 1 -- 172.21.15.32:0/569958551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c179e50 con 0x7f6e5c185a00 2021-04-09T18:14:08.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5a59c700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0983b0 0x7f6e5c17abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:14:08.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5b59e700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c17b1d0 0x7f6e5c179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:14:08.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e5ad9d700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c185a00 0x7f6e5c097d90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f6e4c006f10 tx=0x7f6e4c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e537fe700 1 -- 172.21.15.32:0/569958551 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4c00bd80 con 0x7f6e5c185a00 2021-04-09T18:14:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e537fe700 1 -- 172.21.15.32:0/569958551 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4c000e20 con 0x7f6e5c185a00 2021-04-09T18:14:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e537fe700 1 -- 172.21.15.32:0/569958551 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4c0054f0 con 0x7f6e5c185a00 2021-04-09T18:14:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e611bb700 1 -- 172.21.15.32:0/569958551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c179e50 con 0x7f6e5c185a00 2021-04-09T18:14:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.657 7f6e611bb700 1 -- 172.21.15.32:0/569958551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c179e50 con 0x7f6e5c185a00 2021-04-09T18:14:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.658 7f6e537fe700 1 -- 172.21.15.32:0/569958551 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6e4c009fe0 con 0x7f6e5c185a00 2021-04-09T18:14:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.658 7f6e537fe700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e400188d0 0x7f6e4001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.658 7f6e537fe700 1 -- 172.21.15.32:0/569958551 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6e4c032a10 con 0x7f6e5c185a00 2021-04-09T18:14:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.658 7f6e5a59c700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e400188d0 0x7f6e4001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:08.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.659 7f6e5a59c700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e400188d0 0x7f6e4001add0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.678 7f6e611bb700 1 -- 172.21.15.32:0/569958551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c17a3e0 con 0x7f6e5c185a00 2021-04-09T18:14:08.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.681 7f6e537fe700 1 -- 172.21.15.32:0/569958551 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6e4c000e20 con 0x7f6e5c185a00 2021-04-09T18:14:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.873 7f6e611bb700 1 -- 172.21.15.32:0/569958551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e5c17a3e0 con 0x7f6e5c185a00 2021-04-09T18:14:08.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.873 7f6e537fe700 1 -- 172.21.15.32:0/569958551 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2429 (secure 0 0 0) 0x7f6e4c009fe0 con 0x7f6e5c185a00 2021-04-09T18:14:08.989 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 16 2021-04-09T18:14:08.989 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:09.032 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:14:07.836886","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:09.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 -- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e400188d0 msgr2=0x7f6e4001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:09.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e400188d0 0x7f6e4001add0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 -- 172.21.15.32:0/569958551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c185a00 msgr2=0x7f6e5c097d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c185a00 0x7f6e5c097d90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f6e4c006f10 tx=0x7f6e4c00b990).stop 2021-04-09T18:14:09.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 -- 172.21.15.32:0/569958551 shutdown_connections 2021-04-09T18:14:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e400188d0 0x7f6e4001add0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c17b1d0 0x7f6e5c179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0983b0 0x7f6e5c17abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 --2- 172.21.15.32:0/569958551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c185a00 0x7f6e5c097d90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 -- 172.21.15.32:0/569958551 >> 172.21.15.32:0/569958551 conn(0x7f6e5c190550 msgr2=0x7f6e5c1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 -- 172.21.15.32:0/569958551 shutdown_connections 2021-04-09T18:14:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:08.876 7f6e611bb700 1 -- 172.21.15.32:0/569958551 wait complete. 2021-04-09T18:14:09.491 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:14:07.836886', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:09.492 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2021-04-09T18:14:09.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:09.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.621 7f7dea635700 1 Processor -- start 2021-04-09T18:14:09.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.621 7f7dea635700 1 -- start start 2021-04-09T18:14:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.622 7f7dea635700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7de410da00 0x7f7de410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.622 7f7dea635700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de410d480 con 0x7f7de40d7ca0 2021-04-09T18:14:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.622 7f7dea635700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de40d1f70 con 0x7f7de410d010 2021-04-09T18:14:09.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.622 7f7dea635700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de40d8d30 con 0x7f7de410da00 2021-04-09T18:14:09.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.622 7f7ddbfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7de40d7ca0 0x7f7de40d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52602/0 (socket says 172.21.15.32:52602) 2021-04-09T18:14:09.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.622 7f7ddbfff700 1 -- 172.21.15.32:0/2942901782 learned_addr learned my addr 172.21.15.32:0/2942901782 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.622 7f7de8c52700 1 --2- 172.21.15.32:0/2942901782 >> v2:172.21.15.86:3300/0 conn(0x7f7de410da00 0x7f7de410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:09.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.622 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480347256 0 0) 0x7f7dcc000ce0 con 0x7f7de40d7ca0 2021-04-09T18:14:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.622 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd0002fe0 con 0x7f7de40d7ca0 2021-04-09T18:14:09.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2215013483 0 0) 0x7f7dcc0010e0 con 0x7f7de40d7ca0 2021-04-09T18:14:09.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 >> v2:172.21.15.86:3300/0 conn(0x7f7de410da00 msgr2=0x7f7de410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:09.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 --2- 172.21.15.32:0/2942901782 >> v2:172.21.15.86:3300/0 conn(0x7f7de410da00 0x7f7de410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 >> v1:172.21.15.86:6789/0 conn(0x7f7de410d010 legacy=0x7f7de40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de40d9220 con 0x7f7de40d7ca0 2021-04-09T18:14:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f7dcc001600 con 0x7f7de40d7ca0 2021-04-09T18:14:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 >> v1:172.21.15.32:6789/0 conn(0x7f7de40d7ca0 legacy=0x7f7de40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dd0003a40 0x7f7dd0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dd0006790 0x7f7dd0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dd0009590 0x7f7dd000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:09.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd000c300 con 0x7f7dd0009590 2021-04-09T18:14:09.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd000c520 con 0x7f7dd0006790 2021-04-09T18:14:09.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd0050750 con 0x7f7dd0003a40 2021-04-09T18:14:09.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de8c52700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dd0009590 0x7f7dd000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:09.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de3fff700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dd0003a40 0x7f7dd0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:09.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7ddbfff700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dd0006790 0x7f7dd0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:09.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de8c52700 1 -- 172.21.15.32:0/2942901782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dd0003a40 msgr2=0x7f7dd0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:09.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de8c52700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dd0003a40 0x7f7dd0005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de8c52700 1 -- 172.21.15.32:0/2942901782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dd0006790 msgr2=0x7f7dd0008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:09.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de8c52700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dd0006790 0x7f7dd0008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de8c52700 1 -- 172.21.15.32:0/2942901782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd00509b0 con 0x7f7dd0009590 2021-04-09T18:14:09.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de8c52700 1 -- 172.21.15.32:0/2942901782 reap_dead start 2021-04-09T18:14:09.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de8c52700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dd0009590 0x7f7dd000bab0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f7ddc006a40 tx=0x7f7ddc00c0c0).ready entity=mon.0 client_cookie=c9656ae09d052577 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:09.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ddc006ab0 con 0x7f7dd0009590 2021-04-09T18:14:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ddc007070 con 0x7f7dd0009590 2021-04-09T18:14:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7de37fe700 1 -- 172.21.15.32:0/2942901782 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ddc0074c0 con 0x7f7dd0009590 2021-04-09T18:14:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7dea635700 1 -- 172.21.15.32:0/2942901782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dd0009590 msgr2=0x7f7dd000bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.623 7f7dea635700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dd0009590 0x7f7dd000bab0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f7ddc006a40 tx=0x7f7ddc00c0c0).stop 2021-04-09T18:14:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.624 7f7dea635700 1 -- 172.21.15.32:0/2942901782 shutdown_connections 2021-04-09T18:14:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.624 7f7dea635700 1 --2- 172.21.15.32:0/2942901782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dd0009590 0x7f7dd000bab0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.624 7f7dea635700 1 -- 172.21.15.32:0/2942901782 >> 172.21.15.32:0/2942901782 conn(0x7f7de40ce5f0 msgr2=0x7f7de40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:09.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.624 7f7dea635700 1 -- 172.21.15.32:0/2942901782 shutdown_connections 2021-04-09T18:14:09.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.624 7f7dea635700 1 -- 172.21.15.32:0/2942901782 wait complete. 2021-04-09T18:14:09.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7dea635700 1 Processor -- start 2021-04-09T18:14:09.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7dea635700 1 -- start start 2021-04-09T18:14:09.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7dea635700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7de41ac380 0x7f7de418a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:09.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7dea635700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7de41ac780 0x7f7de418a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:09.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7dea635700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7de4183520 0x7f7de4183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:09.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7dea635700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de41ace10 con 0x7f7de41ac780 2021-04-09T18:14:09.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7dea635700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de4189cf0 con 0x7f7de4183520 2021-04-09T18:14:09.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7dea635700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de4183f40 con 0x7f7de41ac380 2021-04-09T18:14:09.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7ddbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7de41ac780 0x7f7de418a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:09.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7ddbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7de41ac780 0x7f7de418a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43024/0 (socket says 172.21.15.32:43024) 2021-04-09T18:14:09.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7ddbfff700 1 -- 172.21.15.32:0/3956454307 learned_addr learned my addr 172.21.15.32:0/3956454307 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:09.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7de8c52700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7de4183520 0x7f7de4183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:09.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.645 7f7de3fff700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7de41ac380 0x7f7de418a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:09.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7ddbfff700 1 -- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7de41ac380 msgr2=0x7f7de418a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:09.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7ddbfff700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7de41ac380 0x7f7de418a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7ddbfff700 1 -- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7de4183520 msgr2=0x7f7de4183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:09.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7ddbfff700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7de4183520 0x7f7de4183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7ddbfff700 1 -- 172.21.15.32:0/3956454307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de4184160 con 0x7f7de41ac780 2021-04-09T18:14:09.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7ddbfff700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7de41ac780 0x7f7de418a850 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f7dcc00a5d0 tx=0x7f7dcc00a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:09.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7de17fa700 1 -- 172.21.15.32:0/3956454307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7dcc000f40 con 0x7f7de41ac780 2021-04-09T18:14:09.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7de17fa700 1 -- 172.21.15.32:0/3956454307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc00ad60 con 0x7f7de41ac780 2021-04-09T18:14:09.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7de17fa700 1 -- 172.21.15.32:0/3956454307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7dcc0045b0 con 0x7f7de41ac780 2021-04-09T18:14:09.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7dea635700 1 -- 172.21.15.32:0/3956454307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de4184160 con 0x7f7de41ac780 2021-04-09T18:14:09.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.646 7f7dea635700 1 -- 172.21.15.32:0/3956454307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de4184160 con 0x7f7de41ac780 2021-04-09T18:14:09.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.647 7f7de17fa700 1 -- 172.21.15.32:0/3956454307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7dcc0043a0 con 0x7f7de41ac780 2021-04-09T18:14:09.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.647 7f7de17fa700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7dc8018930 0x7f7dc801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:09.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.647 7f7de17fa700 1 -- 172.21.15.32:0/3956454307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7dcc0329d0 con 0x7f7de41ac780 2021-04-09T18:14:09.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.647 7f7de3fff700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7dc8018930 0x7f7dc801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.647 7f7de3fff700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7dc8018930 0x7f7dc801adc0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:09.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.666 7f7dea635700 1 -- 172.21.15.32:0/3956454307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de4189cf0 con 0x7f7de41ac780 2021-04-09T18:14:09.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.669 7f7de17fa700 1 -- 172.21.15.32:0/3956454307 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7dcc000f40 con 0x7f7de41ac780 2021-04-09T18:14:09.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.861 7f7dea635700 1 -- 172.21.15.32:0/3956454307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7de4189cf0 con 0x7f7de41ac780 2021-04-09T18:14:09.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.861 7f7de17fa700 1 -- 172.21.15.32:0/3956454307 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2429 (secure 0 0 0) 0x7f7dcc00ad60 con 0x7f7de41ac780 2021-04-09T18:14:09.898 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 16 2021-04-09T18:14:09.899 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:09.899 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:14:07.836886","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:09.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 -- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7dc8018930 msgr2=0x7f7dc801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7dc8018930 0x7f7dc801adc0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 -- 172.21.15.32:0/3956454307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7de41ac780 msgr2=0x7f7de418a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7de41ac780 0x7f7de418a850 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f7dcc00a5d0 tx=0x7f7dcc00a970).stop 2021-04-09T18:14:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 -- 172.21.15.32:0/3956454307 shutdown_connections 2021-04-09T18:14:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7dc8018930 0x7f7dc801adc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7de41ac780 0x7f7de418a850 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7de4183520 0x7f7de4183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 --2- 172.21.15.32:0/3956454307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7de41ac380 0x7f7de418a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:09.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 -- 172.21.15.32:0/3956454307 >> 172.21.15.32:0/3956454307 conn(0x7f7de419b890 msgr2=0x7f7de41a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 -- 172.21.15.32:0/3956454307 shutdown_connections 2021-04-09T18:14:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:09.864 7f7dea635700 1 -- 172.21.15.32:0/3956454307 wait complete. 2021-04-09T18:14:09.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down false 2021-04-09T18:14:10.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.099 7feac45c8700 1 Processor -- start 2021-04-09T18:14:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.099 7feac45c8700 1 -- start start 2021-04-09T18:14:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac45c8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feabc10d010 0x7feabc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac45c8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc0d8490 0x7feabc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac45c8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc10d480 con 0x7feabc0d1da0 2021-04-09T18:14:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac45c8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feabc0d8d30 con 0x7feabc10d010 2021-04-09T18:14:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac45c8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0d9220 con 0x7feabc0d8490 2021-04-09T18:14:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac1be3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feabc0d1da0 0x7feabc0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52622/0 (socket says 172.21.15.32:52622) 2021-04-09T18:14:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac1be3700 1 -- 172.21.15.32:0/879678759 learned_addr learned my addr 172.21.15.32:0/879678759 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac2be5700 1 --2- 172.21.15.32:0/879678759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc0d8490 0x7feabc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac23e4700 1 --2- 172.21.15.32:0/879678759 >> v2:172.21.15.86:3300/0 conn(0x7feabc10d010 0x7feabc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac13e2700 1 -- 172.21.15.32:0/879678759 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117530867 0 0) 0x7feab0000ce0 con 0x7feabc0d1da0 2021-04-09T18:14:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac13e2700 1 -- 172.21.15.32:0/879678759 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa8002fe0 con 0x7feabc0d1da0 2021-04-09T18:14:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac23e4700 1 -- 172.21.15.32:0/879678759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc0d8490 msgr2=0x7feabc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac23e4700 1 --2- 172.21.15.32:0/879678759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc0d8490 0x7feabc10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac23e4700 1 -- 172.21.15.32:0/879678759 >> v1:172.21.15.32:6789/0 conn(0x7feabc0d1da0 legacy=0x7feabc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac23e4700 1 -- 172.21.15.32:0/879678759 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc10e020 con 0x7feabc10d010 2021-04-09T18:14:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.100 7feac2be5700 1 --2- 172.21.15.32:0/879678759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc0d8490 0x7feabc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.101 7feac23e4700 1 --2- 172.21.15.32:0/879678759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feabc10d010 0x7feabc0d1780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7feab8006650 tx=0x7feab8009a80).ready entity=mon.1 client_cookie=96a6328517e70f84 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.101 7feac13e2700 1 -- 172.21.15.32:0/879678759 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feab80009b0 con 0x7feabc10d010 2021-04-09T18:14:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.101 7feac13e2700 1 -- 172.21.15.32:0/879678759 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab800c0c0 con 0x7feabc10d010 2021-04-09T18:14:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.101 7feac13e2700 1 -- 172.21.15.32:0/879678759 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feab800c510 con 0x7feabc10d010 2021-04-09T18:14:10.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.101 7feac45c8700 1 -- 172.21.15.32:0/879678759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feabc10d010 msgr2=0x7feabc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.101 7feac45c8700 1 --2- 172.21.15.32:0/879678759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feabc10d010 0x7feabc0d1780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7feab8006650 tx=0x7feab8009a80).stop 2021-04-09T18:14:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.102 7feac45c8700 1 -- 172.21.15.32:0/879678759 shutdown_connections 2021-04-09T18:14:10.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.102 7feac45c8700 1 --2- 172.21.15.32:0/879678759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc0d8490 0x7feabc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.102 7feac45c8700 1 --2- 172.21.15.32:0/879678759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feabc10d010 0x7feabc0d1780 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.102 7feac45c8700 1 -- 172.21.15.32:0/879678759 >> 172.21.15.32:0/879678759 conn(0x7feabc0ce5f0 msgr2=0x7feabc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:10.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.102 7feac45c8700 1 -- 172.21.15.32:0/879678759 shutdown_connections 2021-04-09T18:14:10.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.102 7feac45c8700 1 -- 172.21.15.32:0/879678759 wait complete. 2021-04-09T18:14:10.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac45c8700 1 Processor -- start 2021-04-09T18:14:10.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac45c8700 1 -- start start 2021-04-09T18:14:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac45c8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc182db0 0x7feabc1837a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:10.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac45c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feabc183dc0 0x7feabc17db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:10.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac45c8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feabc1841c0 0x7feabc17e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:10.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac45c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc08e590 con 0x7feabc183dc0 2021-04-09T18:14:10.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac45c8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc183290 con 0x7feabc1841c0 2021-04-09T18:14:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac45c8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc17e7e0 con 0x7feabc182db0 2021-04-09T18:14:10.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac1be3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feabc183dc0 0x7feabc17db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:10.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac1be3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feabc183dc0 0x7feabc17db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43036/0 (socket says 172.21.15.32:43036) 2021-04-09T18:14:10.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac1be3700 1 -- 172.21.15.32:0/1692737134 learned_addr learned my addr 172.21.15.32:0/1692737134 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.122 7feac23e4700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc182db0 0x7feabc1837a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feac2be5700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feabc1841c0 0x7feabc17e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feac1be3700 1 -- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc182db0 msgr2=0x7feabc1837a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:10.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feac1be3700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc182db0 0x7feabc1837a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feac1be3700 1 -- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feabc1841c0 msgr2=0x7feabc17e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:10.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feac1be3700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feabc1841c0 0x7feabc17e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feac1be3700 1 -- 172.21.15.32:0/1692737134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc17ea00 con 0x7feabc183dc0 2021-04-09T18:14:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feac1be3700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feabc183dc0 0x7feabc17db70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7feab0006f10 tx=0x7feab000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:10.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feaaeffd700 1 -- 172.21.15.32:0/1692737134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feab0006020 con 0x7feabc183dc0 2021-04-09T18:14:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feaaeffd700 1 -- 172.21.15.32:0/1692737134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab0006320 con 0x7feabc183dc0 2021-04-09T18:14:10.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feac45c8700 1 -- 172.21.15.32:0/1692737134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc17ea00 con 0x7feabc183dc0 2021-04-09T18:14:10.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feaaeffd700 1 -- 172.21.15.32:0/1692737134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feab0004590 con 0x7feabc183dc0 2021-04-09T18:14:10.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.123 7feac45c8700 1 -- 172.21.15.32:0/1692737134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc17ea00 con 0x7feabc183dc0 2021-04-09T18:14:10.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.124 7feaaeffd700 1 -- 172.21.15.32:0/1692737134 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7feab0009fe0 con 0x7feabc183dc0 2021-04-09T18:14:10.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.124 7feaaeffd700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feaa4018930 0x7feaa401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:10.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.124 7feaaeffd700 1 -- 172.21.15.32:0/1692737134 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7feab0032a10 con 0x7feabc183dc0 2021-04-09T18:14:10.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.124 7feac23e4700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feaa4018930 0x7feaa401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.124 7feac23e4700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feaa4018930 0x7feaa401ae10 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:10.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.144 7feac45c8700 1 -- 172.21.15.32:0/1692737134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc1b7050 con 0x7feabc183dc0 2021-04-09T18:14:10.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.147 7feaaeffd700 1 -- 172.21.15.32:0/1692737134 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7feab0006ba0 con 0x7feabc183dc0 2021-04-09T18:14:10.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.340 7feac45c8700 1 -- 172.21.15.32:0/1692737134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7feabc1b7050 con 0x7feabc183dc0 2021-04-09T18:14:10.868 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:14:10.846 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:14:10.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.848 7feaaeffd700 1 -- 172.21.15.32:0/1692737134 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 cephfs marked up, max_mds = 1 v17) v1 ==== 135+0+0 (secure 0 0 0) 0x7feab0009fe0 con 0x7feabc183dc0 2021-04-09T18:14:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked up, max_mds = 1 2021-04-09T18:14:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.857 7feac45c8700 1 -- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feaa4018930 msgr2=0x7feaa401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.857 7feac45c8700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feaa4018930 0x7feaa401ae10 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.857 7feac45c8700 1 -- 172.21.15.32:0/1692737134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feabc183dc0 msgr2=0x7feabc17db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.857 7feac45c8700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feabc183dc0 0x7feabc17db70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7feab0006f10 tx=0x7feab000b990).stop 2021-04-09T18:14:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.858 7feac45c8700 1 -- 172.21.15.32:0/1692737134 shutdown_connections 2021-04-09T18:14:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.858 7feac45c8700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feaa4018930 0x7feaa401ae10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.858 7feac45c8700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feabc183dc0 0x7feabc17db70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.858 7feac45c8700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feabc1841c0 0x7feabc17e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.858 7feac45c8700 1 --2- 172.21.15.32:0/1692737134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feabc182db0 0x7feabc1837a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:10.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.858 7feac45c8700 1 -- 172.21.15.32:0/1692737134 >> 172.21.15.32:0/1692737134 conn(0x7feabc0c0430 msgr2=0x7feabc0bfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:10.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.858 7feac45c8700 1 -- 172.21.15.32:0/1692737134 shutdown_connections 2021-04-09T18:14:10.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:10.858 7feac45c8700 1 -- 172.21.15.32:0/1692737134 wait complete. 2021-04-09T18:14:10.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:11.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.027 7fe1beed4700 1 Processor -- start 2021-04-09T18:14:11.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.027 7fe1beed4700 1 -- start start 2021-04-09T18:14:11.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1beed4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1b80d6c80 0x7fe1b80d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1beed4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe1b80d2440 0x7fe1b810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1beed4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b80d7280 con 0x7fe1b80cffa0 2021-04-09T18:14:11.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1beed4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b80d7080 con 0x7fe1b80d6c80 2021-04-09T18:14:11.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1beed4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b80d91e0 con 0x7fe1b80d2440 2021-04-09T18:14:11.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1b7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1b80d6c80 0x7fe1b80d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1b7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1b80d6c80 0x7fe1b80d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43056/0 (socket says 172.21.15.32:43056) 2021-04-09T18:14:11.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1b7fff700 1 -- 172.21.15.32:0/3660347923 learned_addr learned my addr 172.21.15.32:0/3660347923 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1bd4f1700 1 --2- 172.21.15.32:0/3660347923 >> v2:172.21.15.86:3300/0 conn(0x7fe1b80d2440 0x7fe1b810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1b7fff700 1 -- 172.21.15.32:0/3660347923 >> v2:172.21.15.86:3300/0 conn(0x7fe1b80d2440 msgr2=0x7fe1b810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1b7fff700 1 --2- 172.21.15.32:0/3660347923 >> v2:172.21.15.86:3300/0 conn(0x7fe1b80d2440 0x7fe1b810cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1b7fff700 1 -- 172.21.15.32:0/3660347923 >> v1:172.21.15.86:6789/0 conn(0x7fe1b80cffa0 legacy=0x7fe1b80d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.028 7fe1b7fff700 1 -- 172.21.15.32:0/3660347923 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b80d2840 con 0x7fe1b80d6c80 2021-04-09T18:14:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.029 7fe1b7fff700 1 --2- 172.21.15.32:0/3660347923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b80d6c80 0x7fe1b80d7ff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fe1ac006700 tx=0x7fe1ac009b80).ready entity=mon.0 client_cookie=98fc188d3971e9df server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.029 7fe1b77fe700 1 -- 172.21.15.32:0/3660347923 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ac00c0c0 con 0x7fe1b80d6c80 2021-04-09T18:14:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.029 7fe1b77fe700 1 -- 172.21.15.32:0/3660347923 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac00c440 con 0x7fe1b80d6c80 2021-04-09T18:14:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.029 7fe1b77fe700 1 -- 172.21.15.32:0/3660347923 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ac00c920 con 0x7fe1b80d6c80 2021-04-09T18:14:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.029 7fe1beed4700 1 -- 172.21.15.32:0/3660347923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b80d6c80 msgr2=0x7fe1b80d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.029 7fe1beed4700 1 --2- 172.21.15.32:0/3660347923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b80d6c80 0x7fe1b80d7ff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fe1ac006700 tx=0x7fe1ac009b80).stop 2021-04-09T18:14:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.030 7fe1beed4700 1 -- 172.21.15.32:0/3660347923 shutdown_connections 2021-04-09T18:14:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.030 7fe1beed4700 1 --2- 172.21.15.32:0/3660347923 >> v2:172.21.15.86:3300/0 conn(0x7fe1b80d2440 0x7fe1b810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.030 7fe1beed4700 1 --2- 172.21.15.32:0/3660347923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b80d6c80 0x7fe1b80d7ff0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.030 7fe1beed4700 1 -- 172.21.15.32:0/3660347923 >> 172.21.15.32:0/3660347923 conn(0x7fe1b80ce5c0 msgr2=0x7fe1b80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.030 7fe1beed4700 1 -- 172.21.15.32:0/3660347923 shutdown_connections 2021-04-09T18:14:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.030 7fe1beed4700 1 -- 172.21.15.32:0/3660347923 wait complete. 2021-04-09T18:14:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.050 7fe1beed4700 1 Processor -- start 2021-04-09T18:14:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.050 7fe1beed4700 1 -- start start 2021-04-09T18:14:11.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.050 7fe1beed4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1b80a80c0 0x7fe1b817bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.050 7fe1beed4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1b817b1d0 0x7fe1b817b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.050 7fe1beed4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b808dc50 0x7fe1b808e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.050 7fe1beed4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b817bc70 con 0x7fe1b808dc50 2021-04-09T18:14:11.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.050 7fe1beed4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b808e670 con 0x7fe1b817b1d0 2021-04-09T18:14:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.050 7fe1beed4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b808e850 con 0x7fe1b80a80c0 2021-04-09T18:14:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bd4f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b808dc50 0x7fe1b808e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bd4f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b808dc50 0x7fe1b808e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43062/0 (socket says 172.21.15.32:43062) 2021-04-09T18:14:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bd4f1700 1 -- 172.21.15.32:0/3467435596 learned_addr learned my addr 172.21.15.32:0/3467435596 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1b7fff700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1b817b1d0 0x7fe1b817b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bccf0700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1b80a80c0 0x7fe1b817bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bd4f1700 1 -- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1b80a80c0 msgr2=0x7fe1b817bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bd4f1700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1b80a80c0 0x7fe1b817bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bd4f1700 1 -- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1b817b1d0 msgr2=0x7fe1b817b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bd4f1700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1b817b1d0 0x7fe1b817b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bd4f1700 1 -- 172.21.15.32:0/3467435596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b808c660 con 0x7fe1b808dc50 2021-04-09T18:14:11.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1bd4f1700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b808dc50 0x7fe1b808e050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fe1b0006f40 tx=0x7fe1b000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:11.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1b57fa700 1 -- 172.21.15.32:0/3467435596 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1b00011f0 con 0x7fe1b808dc50 2021-04-09T18:14:11.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1b57fa700 1 -- 172.21.15.32:0/3467435596 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b0003030 con 0x7fe1b808dc50 2021-04-09T18:14:11.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b808c660 con 0x7fe1b808dc50 2021-04-09T18:14:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1b57fa700 1 -- 172.21.15.32:0/3467435596 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1b00016a0 con 0x7fe1b808dc50 2021-04-09T18:14:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.051 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b808c660 con 0x7fe1b808dc50 2021-04-09T18:14:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.052 7fe1b57fa700 1 -- 172.21.15.32:0/3467435596 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe1b00011f0 con 0x7fe1b808dc50 2021-04-09T18:14:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.053 7fe1b57fa700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1980188c0 0x7fe19801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.053 7fe1b57fa700 1 -- 172.21.15.32:0/3467435596 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe1b0032cc0 con 0x7fe1b808dc50 2021-04-09T18:14:11.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.053 7fe1bccf0700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1980188c0 0x7fe19801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.053 7fe1bccf0700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1980188c0 0x7fe19801adc0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.072 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b808e850 con 0x7fe1b808dc50 2021-04-09T18:14:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.075 7fe1b57fa700 1 -- 172.21.15.32:0/3467435596 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe1b0000f40 con 0x7fe1b808dc50 2021-04-09T18:14:11.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.265 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1b808e850 con 0x7fe1b808dc50 2021-04-09T18:14:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.266 7fe1b57fa700 1 -- 172.21.15.32:0/3467435596 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2441 (secure 0 0 0) 0x7fe1b0001f00 con 0x7fe1b808dc50 2021-04-09T18:14:11.290 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2021-04-09T18:14:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:11.291 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:14:10.849274","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4722":{"gid":4722,"name":"a","rank":0,"incarnation":18,"state":"up:starting","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1980188c0 msgr2=0x7fe19801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1980188c0 0x7fe19801adc0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b808dc50 msgr2=0x7fe1b808e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b808dc50 0x7fe1b808e050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fe1b0006f40 tx=0x7fe1b000bd40).stop 2021-04-09T18:14:11.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 shutdown_connections 2021-04-09T18:14:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1980188c0 0x7fe19801adc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1b808dc50 0x7fe1b808e050 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1b817b1d0 0x7fe1b817b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 --2- 172.21.15.32:0/3467435596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1b80a80c0 0x7fe1b817bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.268 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 >> 172.21.15.32:0/3467435596 conn(0x7fe1b8189b00 msgr2=0x7fe1b81921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:11.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.269 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 shutdown_connections 2021-04-09T18:14:11.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.269 7fe1beed4700 1 -- 172.21.15.32:0/3467435596 wait complete. 2021-04-09T18:14:11.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:11.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.459 7f824af7b700 1 Processor -- start 2021-04-09T18:14:11.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.459 7f824af7b700 1 -- start start 2021-04-09T18:14:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.460 7f824af7b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f824410d010 0x7f82440d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.460 7f824af7b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82440d1da0 0x7f82440d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.460 7f824af7b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824410d480 con 0x7f82440d82c0 2021-04-09T18:14:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.460 7f824af7b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440d8d30 con 0x7f82440d1da0 2021-04-09T18:14:11.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.460 7f824af7b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440d9220 con 0x7f824410d010 2021-04-09T18:14:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.460 7f8243fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82440d1da0 0x7f82440d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.460 7f8249598700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f82440d82c0 0x7f824410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52656/0 (socket says 172.21.15.32:52656) 2021-04-09T18:14:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.460 7f8249598700 1 -- 172.21.15.32:0/2124714517 learned_addr learned my addr 172.21.15.32:0/2124714517 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:11.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.460 7f8248d97700 1 --2- 172.21.15.32:0/2124714517 >> v2:172.21.15.86:3300/0 conn(0x7f824410d010 0x7f82440d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f8243fff700 1 -- 172.21.15.32:0/2124714517 >> v2:172.21.15.86:3300/0 conn(0x7f824410d010 msgr2=0x7f82440d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f8243fff700 1 --2- 172.21.15.32:0/2124714517 >> v2:172.21.15.86:3300/0 conn(0x7f824410d010 0x7f82440d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f8243fff700 1 -- 172.21.15.32:0/2124714517 >> v1:172.21.15.32:6789/0 conn(0x7f82440d82c0 legacy=0x7f824410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f8243fff700 1 -- 172.21.15.32:0/2124714517 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440d86f0 con 0x7f82440d1da0 2021-04-09T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f82437fe700 1 -- 172.21.15.32:0/2124714517 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916311261 0 0) 0x7f823c0009b0 con 0x7f82440d82c0 2021-04-09T18:14:11.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f8248d97700 1 --2- 172.21.15.32:0/2124714517 >> v2:172.21.15.86:3300/0 conn(0x7f824410d010 0x7f82440d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:11.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f8243fff700 1 --2- 172.21.15.32:0/2124714517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82440d1da0 0x7f82440d7ca0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f82340069d0 tx=0x7f8234009e40).ready entity=mon.0 client_cookie=67a2b9a33a09c4aa server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:11.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f82437fe700 1 -- 172.21.15.32:0/2124714517 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f823400b510 con 0x7f82440d1da0 2021-04-09T18:14:11.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f82437fe700 1 -- 172.21.15.32:0/2124714517 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823400b890 con 0x7f82440d1da0 2021-04-09T18:14:11.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f82437fe700 1 -- 172.21.15.32:0/2124714517 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f823400bd70 con 0x7f82440d1da0 2021-04-09T18:14:11.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f824af7b700 1 -- 172.21.15.32:0/2124714517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82440d1da0 msgr2=0x7f82440d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.461 7f824af7b700 1 --2- 172.21.15.32:0/2124714517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82440d1da0 0x7f82440d7ca0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f82340069d0 tx=0x7f8234009e40).stop 2021-04-09T18:14:11.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.462 7f824af7b700 1 -- 172.21.15.32:0/2124714517 shutdown_connections 2021-04-09T18:14:11.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.462 7f824af7b700 1 --2- 172.21.15.32:0/2124714517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82440d1da0 0x7f82440d7ca0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.462 7f824af7b700 1 --2- 172.21.15.32:0/2124714517 >> v2:172.21.15.86:3300/0 conn(0x7f824410d010 0x7f82440d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.462 7f824af7b700 1 -- 172.21.15.32:0/2124714517 >> 172.21.15.32:0/2124714517 conn(0x7f82440ce5f0 msgr2=0x7f82440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:11.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.462 7f824af7b700 1 -- 172.21.15.32:0/2124714517 shutdown_connections 2021-04-09T18:14:11.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.462 7f824af7b700 1 -- 172.21.15.32:0/2124714517 wait complete. 2021-04-09T18:14:11.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.482 7f824af7b700 1 Processor -- start 2021-04-09T18:14:11.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.482 7f824af7b700 1 -- start start 2021-04-09T18:14:11.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f824af7b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82441a9360 0x7f82441918f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f824af7b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f824418c440 0x7f824418c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f824af7b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82441a80f0 0x7f82441a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f824af7b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8244191f10 con 0x7f82441a9360 2021-04-09T18:14:11.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f824af7b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82441a9760 con 0x7f824418c440 2021-04-09T18:14:11.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f824af7b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824418ce60 con 0x7f82441a80f0 2021-04-09T18:14:11.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8248d97700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82441a9360 0x7f82441918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8248d97700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82441a9360 0x7f82441918f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43070/0 (socket says 172.21.15.32:43070) 2021-04-09T18:14:11.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8248d97700 1 -- 172.21.15.32:0/1361925938 learned_addr learned my addr 172.21.15.32:0/1361925938 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:11.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8243fff700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f824418c440 0x7f824418c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8249598700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82441a80f0 0x7f82441a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8248d97700 1 -- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82441a80f0 msgr2=0x7f82441a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8248d97700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82441a80f0 0x7f82441a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8248d97700 1 -- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f824418c440 msgr2=0x7f824418c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8248d97700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f824418c440 0x7f824418c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8248d97700 1 -- 172.21.15.32:0/1361925938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82441a8b10 con 0x7f82441a9360 2021-04-09T18:14:11.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f8248d97700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82441a9360 0x7f82441918f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f822c006f20 tx=0x7f822c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:11.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f82417fa700 1 -- 172.21.15.32:0/1361925938 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f822c00bd80 con 0x7f82441a9360 2021-04-09T18:14:11.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f82417fa700 1 -- 172.21.15.32:0/1361925938 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c000c90 con 0x7f82441a9360 2021-04-09T18:14:11.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f82417fa700 1 -- 172.21.15.32:0/1361925938 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f822c0022b0 con 0x7f82441a9360 2021-04-09T18:14:11.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.483 7f824af7b700 1 -- 172.21.15.32:0/1361925938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82441a8b10 con 0x7f82441a9360 2021-04-09T18:14:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.484 7f824af7b700 1 -- 172.21.15.32:0/1361925938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82441a8b10 con 0x7f82441a9360 2021-04-09T18:14:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.484 7f82417fa700 1 -- 172.21.15.32:0/1361925938 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f822c00a000 con 0x7f82441a9360 2021-04-09T18:14:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.484 7f82417fa700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8228018930 0x7f822801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.484 7f82417fa700 1 -- 172.21.15.32:0/1361925938 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f822c032a30 con 0x7f82441a9360 2021-04-09T18:14:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.484 7f8243fff700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8228018930 0x7f822801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:11.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.485 7f8243fff700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8228018930 0x7f822801ae10 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.504 7f824af7b700 1 -- 172.21.15.32:0/1361925938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82441a9760 con 0x7f82441a9360 2021-04-09T18:14:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.508 7f82417fa700 1 -- 172.21.15.32:0/1361925938 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f822c006bb0 con 0x7f82441a9360 2021-04-09T18:14:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.697 7f824af7b700 1 -- 172.21.15.32:0/1361925938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82441a9760 con 0x7f82441a9360 2021-04-09T18:14:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.698 7f82417fa700 1 -- 172.21.15.32:0/1361925938 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+2441 (secure 0 0 0) 0x7f822c006bb0 con 0x7f82441a9360 2021-04-09T18:14:11.849 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 18 2021-04-09T18:14:11.879 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:11.879 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:14:10.849274","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4722":{"gid":4722,"name":"a","rank":0,"incarnation":18,"state":"up:starting","state_seq":2,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:11.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 -- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8228018930 msgr2=0x7f822801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8228018930 0x7f822801ae10 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 -- 172.21.15.32:0/1361925938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82441a9360 msgr2=0x7f82441918f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:11.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82441a9360 0x7f82441918f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f822c006f20 tx=0x7f822c00b990).stop 2021-04-09T18:14:11.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 -- 172.21.15.32:0/1361925938 shutdown_connections 2021-04-09T18:14:11.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8228018930 0x7f822801ae10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82441a80f0 0x7f82441a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f824418c440 0x7f824418c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 --2- 172.21.15.32:0/1361925938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82441a9360 0x7f82441918f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.700 7f824af7b700 1 -- 172.21.15.32:0/1361925938 >> 172.21.15.32:0/1361925938 conn(0x7f82441837d0 msgr2=0x7f8244195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.701 7f824af7b700 1 -- 172.21.15.32:0/1361925938 shutdown_connections 2021-04-09T18:14:11.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:11.701 7f824af7b700 1 -- 172.21.15.32:0/1361925938 wait complete. 2021-04-09T18:14:12.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:14:10.849274', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4722}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4722': {'gid': 4722, 'name': 'a', 'rank': 0, 'incarnation': 18, 'state': 'up:starting', 'state_seq': 2, 'addr': '172.21.15.32:6833/2659274405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2659274405}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2659274405}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:12.386 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4722:up:starting 2021-04-09T18:14:13.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:13.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f60999c3700 1 Processor -- start 2021-04-09T18:14:13.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f60999c3700 1 -- start start 2021-04-09T18:14:13.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f60999c3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60940d7ca0 0x7f60940d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:13.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f60999c3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f609410da00 0x7f609410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:13.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f60999c3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609410d480 con 0x7f609410d010 2021-04-09T18:14:13.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f60999c3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60940d9220 con 0x7f609410da00 2021-04-09T18:14:13.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f60999c3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60940d86c0 con 0x7f60940d7ca0 2021-04-09T18:14:13.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f6093574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f609410d010 0x7f60940d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52670/0 (socket says 172.21.15.32:52670) 2021-04-09T18:14:13.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f6093574700 1 -- 172.21.15.32:0/1240961615 learned_addr learned my addr 172.21.15.32:0/1240961615 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:13.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f6092d73700 1 --2- 172.21.15.32:0/1240961615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60940d7ca0 0x7f60940d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:13.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.517 7f6093d75700 1 --2- 172.21.15.32:0/1240961615 >> v2:172.21.15.86:3300/0 conn(0x7f609410da00 0x7f609410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:13.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6092572700 1 -- 172.21.15.32:0/1240961615 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265269018 0 0) 0x7f6084001060 con 0x7f609410d010 2021-04-09T18:14:13.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6092572700 1 -- 172.21.15.32:0/1240961615 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607c002fe0 con 0x7f609410d010 2021-04-09T18:14:13.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6093d75700 1 -- 172.21.15.32:0/1240961615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60940d7ca0 msgr2=0x7f60940d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:13.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6093d75700 1 --2- 172.21.15.32:0/1240961615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60940d7ca0 0x7f60940d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6093d75700 1 -- 172.21.15.32:0/1240961615 >> v1:172.21.15.32:6789/0 conn(0x7f609410d010 legacy=0x7f60940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:13.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6093d75700 1 -- 172.21.15.32:0/1240961615 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60940d1f70 con 0x7f609410da00 2021-04-09T18:14:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6092d73700 1 --2- 172.21.15.32:0/1240961615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60940d7ca0 0x7f60940d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:13.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6093d75700 1 --2- 172.21.15.32:0/1240961615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f609410da00 0x7f609410de00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f608c006ad0 tx=0x7f608c009ad0).ready entity=mon.1 client_cookie=7e5e71bedb836814 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:13.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6092572700 1 -- 172.21.15.32:0/1240961615 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f608c0022e0 con 0x7f609410da00 2021-04-09T18:14:13.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6092572700 1 -- 172.21.15.32:0/1240961615 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c006410 con 0x7f609410da00 2021-04-09T18:14:13.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f6092572700 1 -- 172.21.15.32:0/1240961615 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f608c002620 con 0x7f609410da00 2021-04-09T18:14:13.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f60999c3700 1 -- 172.21.15.32:0/1240961615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f609410da00 msgr2=0x7f609410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.518 7f60999c3700 1 --2- 172.21.15.32:0/1240961615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f609410da00 0x7f609410de00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f608c006ad0 tx=0x7f608c009ad0).stop 2021-04-09T18:14:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.519 7f60999c3700 1 -- 172.21.15.32:0/1240961615 shutdown_connections 2021-04-09T18:14:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.519 7f60999c3700 1 --2- 172.21.15.32:0/1240961615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f609410da00 0x7f609410de00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.519 7f60999c3700 1 --2- 172.21.15.32:0/1240961615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60940d7ca0 0x7f60940d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.519 7f60999c3700 1 -- 172.21.15.32:0/1240961615 >> 172.21.15.32:0/1240961615 conn(0x7f60940ce5f0 msgr2=0x7f60940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:13.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.519 7f60999c3700 1 -- 172.21.15.32:0/1240961615 shutdown_connections 2021-04-09T18:14:13.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.519 7f60999c3700 1 -- 172.21.15.32:0/1240961615 wait complete. 2021-04-09T18:14:13.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.539 7f60999c3700 1 Processor -- start 2021-04-09T18:14:13.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.539 7f60999c3700 1 -- start start 2021-04-09T18:14:13.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f60999c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60940a90c0 0x7f60941a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:13.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f60999c3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60941a76d0 0x7f60941a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:13.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f60999c3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f609417bce0 0x7f609417c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:13.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f60999c3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60941a7c60 con 0x7f60940a90c0 2021-04-09T18:14:13.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f60999c3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60941a8e20 con 0x7f60941a76d0 2021-04-09T18:14:13.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f60999c3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609417c700 con 0x7f609417bce0 2021-04-09T18:14:13.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60940a90c0 0x7f60941a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:13.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60940a90c0 0x7f60941a81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43086/0 (socket says 172.21.15.32:43086) 2021-04-09T18:14:13.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093574700 1 -- 172.21.15.32:0/1490907170 learned_addr learned my addr 172.21.15.32:0/1490907170 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6092d73700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60941a76d0 0x7f60941a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093d75700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f609417bce0 0x7f609417c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:13.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093574700 1 -- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f609417bce0 msgr2=0x7f609417c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:13.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093574700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f609417bce0 0x7f609417c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093574700 1 -- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60941a76d0 msgr2=0x7f60941a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:13.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093574700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60941a76d0 0x7f60941a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093574700 1 -- 172.21.15.32:0/1490907170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609417c920 con 0x7f60940a90c0 2021-04-09T18:14:13.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6092d73700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60941a76d0 0x7f60941a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:13.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093d75700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f609417bce0 0x7f609417c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:13.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f6093574700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60940a90c0 0x7f60941a81e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f6084009980 tx=0x7f608400bc50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:13.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f607bfff700 1 -- 172.21.15.32:0/1490907170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6084006030 con 0x7f60940a90c0 2021-04-09T18:14:13.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f607bfff700 1 -- 172.21.15.32:0/1490907170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6084006370 con 0x7f60940a90c0 2021-04-09T18:14:13.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f60999c3700 1 -- 172.21.15.32:0/1490907170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609417c920 con 0x7f60940a90c0 2021-04-09T18:14:13.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.540 7f607bfff700 1 -- 172.21.15.32:0/1490907170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6084004910 con 0x7f60940a90c0 2021-04-09T18:14:13.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.541 7f60999c3700 1 -- 172.21.15.32:0/1490907170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609417c920 con 0x7f60940a90c0 2021-04-09T18:14:13.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.541 7f607bfff700 1 -- 172.21.15.32:0/1490907170 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6084004720 con 0x7f60940a90c0 2021-04-09T18:14:13.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.541 7f607bfff700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6074018930 0x7f607401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.541 7f607bfff700 1 -- 172.21.15.32:0/1490907170 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6084034f80 con 0x7f60940a90c0 2021-04-09T18:14:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.542 7f6092d73700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6074018930 0x7f607401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:13.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.542 7f6092d73700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6074018930 0x7f607401adc0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:13.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.561 7f60999c3700 1 -- 172.21.15.32:0/1490907170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60941b7010 con 0x7f60940a90c0 2021-04-09T18:14:13.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.565 7f607bfff700 1 -- 172.21.15.32:0/1490907170 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6084004930 con 0x7f60940a90c0 2021-04-09T18:14:13.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.757 7f60999c3700 1 -- 172.21.15.32:0/1490907170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f609410d480 con 0x7f60940a90c0 2021-04-09T18:14:13.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.757 7f607bfff700 1 -- 172.21.15.32:0/1490907170 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2774 (secure 0 0 0) 0x7f6084004720 con 0x7f60940a90c0 2021-04-09T18:14:13.849 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 20 2021-04-09T18:14:13.849 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:13.850 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4960,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/425942817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":425942817},{"type":"v1","addr":"172.21.15.86:6835","nonce":425942817}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":20},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:14:11.854446","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4722":{"gid":4722,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:13.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.759 7f60999c3700 1 -- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6074018930 msgr2=0x7f607401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6074018930 0x7f607401adc0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 -- 172.21.15.32:0/1490907170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60940a90c0 msgr2=0x7f60941a81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:13.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60940a90c0 0x7f60941a81e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f6084009980 tx=0x7f608400bc50).stop 2021-04-09T18:14:13.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 -- 172.21.15.32:0/1490907170 shutdown_connections 2021-04-09T18:14:13.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6074018930 0x7f607401adc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f609417bce0 0x7f609417c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60941a76d0 0x7f60941a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 --2- 172.21.15.32:0/1490907170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60940a90c0 0x7f60941a81e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:13.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 -- 172.21.15.32:0/1490907170 >> 172.21.15.32:0/1490907170 conn(0x7f60941acae0 msgr2=0x7f609417ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:13.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 -- 172.21.15.32:0/1490907170 shutdown_connections 2021-04-09T18:14:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:13.760 7f60999c3700 1 -- 172.21.15.32:0/1490907170 wait complete. 2021-04-09T18:14:14.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:13:36.889598', 'modified': '2021-04-09 18:14:11.854446', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4722}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4722': {'gid': 4722, 'name': 'a', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.32:6833/2659274405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2659274405}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2659274405}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:14.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:14:14.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:14:14.515 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:14:14.516 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:14:14.516 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:14:14.516 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 4722, 2021-04-09T18:14:14.517 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:14:14.517 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:14:14.518 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 3.6385017890000002, 2021-04-09T18:14:14.518 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 19, 2021-04-09T18:14:14.518 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 32, 2021-04-09T18:14:14.519 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 32, 2021-04-09T18:14:14.519 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 46.694391193999998 2021-04-09T18:14:14.520 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:14:14.530 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 4722, "want_state": "up:active", "state": "up:active", "rank_uptime": 3.6385017890000002, "mdsmap_epoch": 19, "osdmap_epoch": 32, "osdmap_epoch_barrier": 32, "uptime": 46.694391193999998 } 2021-04-09T18:14:14.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:14.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.660 7f17effbd700 1 Processor -- start 2021-04-09T18:14:14.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.661 7f17effbd700 1 -- start start 2021-04-09T18:14:14.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.661 7f17effbd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f17e810d010 0x7f17e80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:14.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.661 7f17effbd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e810d480 con 0x7f17e80d8490 2021-04-09T18:14:14.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.661 7f17effbd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e810e1f0 con 0x7f17e80d1da0 2021-04-09T18:14:14.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.661 7f17effbd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e80d8d30 con 0x7f17e810d010 2021-04-09T18:14:14.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.661 7f17ee5da700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f17e80d8490 0x7f17e810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52686/0 (socket says 172.21.15.32:52686) 2021-04-09T18:14:14.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.661 7f17ee5da700 1 -- 172.21.15.32:0/2516831862 learned_addr learned my addr 172.21.15.32:0/2516831862 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:14.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17eddd9700 1 --2- 172.21.15.32:0/2516831862 >> v2:172.21.15.86:3300/0 conn(0x7f17e810d010 0x7f17e80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:14.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17ecdd7700 1 -- 172.21.15.32:0/2516831862 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034945104 0 0) 0x7f17e00009b0 con 0x7f17e80d8490 2021-04-09T18:14:14.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17ecdd7700 1 -- 172.21.15.32:0/2516831862 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d4002fe0 con 0x7f17e80d8490 2021-04-09T18:14:14.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17ecdd7700 1 -- 172.21.15.32:0/2516831862 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692472002 0 0) 0x7f17d8000c60 con 0x7f17e80d1da0 2021-04-09T18:14:14.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17ecdd7700 1 -- 172.21.15.32:0/2516831862 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d4003210 con 0x7f17e80d1da0 2021-04-09T18:14:14.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17ecdd7700 1 -- 172.21.15.32:0/2516831862 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2895646572 0 0) 0x7f17e00010e0 con 0x7f17e80d8490 2021-04-09T18:14:14.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17eddd9700 1 -- 172.21.15.32:0/2516831862 >> v1:172.21.15.86:6789/0 conn(0x7f17e80d1da0 legacy=0x7f17e80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:14.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17eddd9700 1 -- 172.21.15.32:0/2516831862 >> v1:172.21.15.32:6789/0 conn(0x7f17e80d8490 legacy=0x7f17e810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:14.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17eddd9700 1 -- 172.21.15.32:0/2516831862 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80d9220 con 0x7f17e810d010 2021-04-09T18:14:14.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17eddd9700 1 --2- 172.21.15.32:0/2516831862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17e810d010 0x7f17e80d1780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f17e40069d0 tx=0x7f17e4009e40).ready entity=mon.1 client_cookie=71833296f2d1a154 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:14.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17ecdd7700 1 -- 172.21.15.32:0/2516831862 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e400b510 con 0x7f17e810d010 2021-04-09T18:14:14.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17ecdd7700 1 -- 172.21.15.32:0/2516831862 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e400b890 con 0x7f17e810d010 2021-04-09T18:14:14.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17ecdd7700 1 -- 172.21.15.32:0/2516831862 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e400bd70 con 0x7f17e810d010 2021-04-09T18:14:14.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17effbd700 1 -- 172.21.15.32:0/2516831862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17e810d010 msgr2=0x7f17e80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:14.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.662 7f17effbd700 1 --2- 172.21.15.32:0/2516831862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17e810d010 0x7f17e80d1780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f17e40069d0 tx=0x7f17e4009e40).stop 2021-04-09T18:14:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.663 7f17effbd700 1 -- 172.21.15.32:0/2516831862 shutdown_connections 2021-04-09T18:14:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.663 7f17effbd700 1 --2- 172.21.15.32:0/2516831862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17e810d010 0x7f17e80d1780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.663 7f17effbd700 1 -- 172.21.15.32:0/2516831862 >> 172.21.15.32:0/2516831862 conn(0x7f17e80ce5f0 msgr2=0x7f17e80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.663 7f17effbd700 1 -- 172.21.15.32:0/2516831862 shutdown_connections 2021-04-09T18:14:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.663 7f17effbd700 1 -- 172.21.15.32:0/2516831862 wait complete. 2021-04-09T18:14:14.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.683 7f17effbd700 1 Processor -- start 2021-04-09T18:14:14.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.683 7f17effbd700 1 -- start start 2021-04-09T18:14:14.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17effbd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e81aca20 0x7f17e81ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:14.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17effbd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17e818bd40 0x7f17e818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17effbd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17e808fba0 0x7f17e808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17effbd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e818c7e0 con 0x7f17e81aca20 2021-04-09T18:14:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17effbd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e818ca30 con 0x7f17e808fba0 2021-04-09T18:14:14.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17effbd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80905f0 con 0x7f17e818bd40 2021-04-09T18:14:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17eddd9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e81aca20 0x7f17e81ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17eddd9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e81aca20 0x7f17e81ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43100/0 (socket says 172.21.15.32:43100) 2021-04-09T18:14:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17eddd9700 1 -- 172.21.15.32:0/1088107917 learned_addr learned my addr 172.21.15.32:0/1088107917 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17ee5da700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17e808fba0 0x7f17e808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:14.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17ed5d8700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17e818bd40 0x7f17e818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:14.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17eddd9700 1 -- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17e818bd40 msgr2=0x7f17e818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:14.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17eddd9700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17e818bd40 0x7f17e818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:14.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17eddd9700 1 -- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17e808fba0 msgr2=0x7f17e808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:14.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17eddd9700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17e808fba0 0x7f17e808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:14.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17eddd9700 1 -- 172.21.15.32:0/1088107917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80b4300 con 0x7f17e81aca20 2021-04-09T18:14:14.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.684 7f17eddd9700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e81aca20 0x7f17e81ace20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f17e4009990 tx=0x7f17e400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:14.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.685 7f17de7fc700 1 -- 172.21.15.32:0/1088107917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e400c2f0 con 0x7f17e81aca20 2021-04-09T18:14:14.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.685 7f17de7fc700 1 -- 172.21.15.32:0/1088107917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e4009020 con 0x7f17e81aca20 2021-04-09T18:14:14.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.685 7f17effbd700 1 -- 172.21.15.32:0/1088107917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e80b4300 con 0x7f17e81aca20 2021-04-09T18:14:14.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.685 7f17de7fc700 1 -- 172.21.15.32:0/1088107917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e400c870 con 0x7f17e81aca20 2021-04-09T18:14:14.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.685 7f17effbd700 1 -- 172.21.15.32:0/1088107917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e80b4300 con 0x7f17e81aca20 2021-04-09T18:14:14.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.685 7f17de7fc700 1 -- 172.21.15.32:0/1088107917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f17e4001010 con 0x7f17e81aca20 2021-04-09T18:14:14.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.685 7f17de7fc700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f17d0018930 0x7f17d001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:14.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.686 7f17de7fc700 1 -- 172.21.15.32:0/1088107917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f17e4032df0 con 0x7f17e81aca20 2021-04-09T18:14:14.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.686 7f17ed5d8700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f17d0018930 0x7f17d001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:14.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.686 7f17ed5d8700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f17d0018930 0x7f17d001ae10 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:14.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.705 7f17effbd700 1 -- 172.21.15.32:0/1088107917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e8182d70 con 0x7f17e81aca20 2021-04-09T18:14:14.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.709 7f17de7fc700 1 -- 172.21.15.32:0/1088107917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f17e4006ac0 con 0x7f17e81aca20 2021-04-09T18:14:14.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.901 7f17effbd700 1 -- 172.21.15.32:0/1088107917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17e818c760 con 0x7f17e81aca20 2021-04-09T18:14:14.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.902 7f17de7fc700 1 -- 172.21.15.32:0/1088107917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2774 (secure 0 0 0) 0x7f17e4001ef0 con 0x7f17e81aca20 2021-04-09T18:14:14.926 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 20 2021-04-09T18:14:14.927 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:14.928 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4960,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/425942817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":425942817},{"type":"v1","addr":"172.21.15.86:6835","nonce":425942817}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":20},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:14:11.854446","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4722":{"gid":4722,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:14.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 -- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f17d0018930 msgr2=0x7f17d001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:14.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f17d0018930 0x7f17d001ae10 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 -- 172.21.15.32:0/1088107917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e81aca20 msgr2=0x7f17e81ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e81aca20 0x7f17e81ace20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f17e4009990 tx=0x7f17e400bd20).stop 2021-04-09T18:14:14.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 -- 172.21.15.32:0/1088107917 shutdown_connections 2021-04-09T18:14:14.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f17d0018930 0x7f17d001ae10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:14.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17e808fba0 0x7f17e808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17e818bd40 0x7f17e818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 --2- 172.21.15.32:0/1088107917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17e81aca20 0x7f17e81ace20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 -- 172.21.15.32:0/1088107917 >> 172.21.15.32:0/1088107917 conn(0x7f17e819b460 msgr2=0x7f17e819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 -- 172.21.15.32:0/1088107917 shutdown_connections 2021-04-09T18:14:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:14.904 7f17effbd700 1 -- 172.21.15.32:0/1088107917 wait complete. 2021-04-09T18:14:14.940 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:14:14.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:14:15.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:14:15.090 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:14:15.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:15.092 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:14:15.092 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:14:15.144 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:14:15.145 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:14:15.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:14:15.147 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:15.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:15.245 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:14:15.246 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:14:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:14:15.291 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:14:15.292 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:14:15.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:14:15.293 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:14:15.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:14:15.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.514 7ffaa6c0a700 1 Processor -- start 2021-04-09T18:14:15.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.514 7ffaa6c0a700 1 -- start start 2021-04-09T18:14:15.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.514 7ffaa6c0a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffaa010d010 0x7ffaa00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.514 7ffaa6c0a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00d8490 0x7ffaa010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.514 7ffaa6c0a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa010d480 con 0x7ffaa00d1da0 2021-04-09T18:14:15.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.514 7ffaa6c0a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa00d8d30 con 0x7ffaa010d010 2021-04-09T18:14:15.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.514 7ffaa6c0a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00d9220 con 0x7ffaa00d8490 2021-04-09T18:14:15.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa4a26700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffaa010d010 0x7ffaa00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:15.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa4a26700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffaa010d010 0x7ffaa00d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43108/0 (socket says 172.21.15.32:43108) 2021-04-09T18:14:15.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa4a26700 1 -- 172.21.15.32:0/1373555475 learned_addr learned my addr 172.21.15.32:0/1373555475 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:15.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa5227700 1 --2- 172.21.15.32:0/1373555475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00d8490 0x7ffaa010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:15.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa4a26700 1 -- 172.21.15.32:0/1373555475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00d8490 msgr2=0x7ffaa010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:15.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa4a26700 1 --2- 172.21.15.32:0/1373555475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00d8490 0x7ffaa010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:15.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa4a26700 1 -- 172.21.15.32:0/1373555475 >> v1:172.21.15.86:6789/0 conn(0x7ffaa00d1da0 legacy=0x7ffaa00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:15.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffa9f7fe700 1 -- 172.21.15.32:0/1373555475 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704658323 0 0) 0x7ffa94000c60 con 0x7ffaa00d1da0 2021-04-09T18:14:15.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa4a26700 1 -- 172.21.15.32:0/1373555475 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa010e020 con 0x7ffaa010d010 2021-04-09T18:14:15.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa4a26700 1 --2- 172.21.15.32:0/1373555475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa010d010 0x7ffaa00d1780 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7ffa90006700 tx=0x7ffa90009b80).ready entity=mon.0 client_cookie=21cf50585c0ae76a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:15.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffa9f7fe700 1 -- 172.21.15.32:0/1373555475 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa9000c0c0 con 0x7ffaa010d010 2021-04-09T18:14:15.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffa9f7fe700 1 -- 172.21.15.32:0/1373555475 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9000c440 con 0x7ffaa010d010 2021-04-09T18:14:15.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffa9f7fe700 1 -- 172.21.15.32:0/1373555475 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa9000c920 con 0x7ffaa010d010 2021-04-09T18:14:15.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa6c0a700 1 -- 172.21.15.32:0/1373555475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa010d010 msgr2=0x7ffaa00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:15.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.515 7ffaa6c0a700 1 --2- 172.21.15.32:0/1373555475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa010d010 0x7ffaa00d1780 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7ffa90006700 tx=0x7ffa90009b80).stop 2021-04-09T18:14:15.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.516 7ffaa6c0a700 1 -- 172.21.15.32:0/1373555475 shutdown_connections 2021-04-09T18:14:15.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.516 7ffaa6c0a700 1 --2- 172.21.15.32:0/1373555475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00d8490 0x7ffaa010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:15.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.516 7ffaa6c0a700 1 --2- 172.21.15.32:0/1373555475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa010d010 0x7ffaa00d1780 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:15.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.516 7ffaa6c0a700 1 -- 172.21.15.32:0/1373555475 >> 172.21.15.32:0/1373555475 conn(0x7ffaa00ce5f0 msgr2=0x7ffaa00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:15.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.516 7ffaa6c0a700 1 -- 172.21.15.32:0/1373555475 shutdown_connections 2021-04-09T18:14:15.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.516 7ffaa6c0a700 1 -- 172.21.15.32:0/1373555475 wait complete. 2021-04-09T18:14:15.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.536 7ffaa6c0a700 1 Processor -- start 2021-04-09T18:14:15.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.536 7ffaa6c0a700 1 -- start start 2021-04-09T18:14:15.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa6c0a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00b42d0 0x7ffaa0183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:15.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa6c0a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaa01ac090 0x7ffaa01ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:15.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa6c0a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa0184d30 0x7ffaa0185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:15.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa6c0a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa00a1a90 con 0x7ffaa0184d30 2021-04-09T18:14:15.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa6c0a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa01acab0 con 0x7ffaa01ac090 2021-04-09T18:14:15.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa6c0a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa0185a00 con 0x7ffaa00b42d0 2021-04-09T18:14:15.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa5227700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa0184d30 0x7ffaa0185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa5227700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa0184d30 0x7ffaa0185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43118/0 (socket says 172.21.15.32:43118) 2021-04-09T18:14:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa5227700 1 -- 172.21.15.32:0/992352834 learned_addr learned my addr 172.21.15.32:0/992352834 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:15.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa4a26700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00b42d0 0x7ffaa0183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:15.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffa9ffff700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaa01ac090 0x7ffaa01ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:15.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa4a26700 1 -- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaa01ac090 msgr2=0x7ffaa01ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:15.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa4a26700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaa01ac090 0x7ffaa01ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:15.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa4a26700 1 -- 172.21.15.32:0/992352834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa0184d30 msgr2=0x7ffaa0185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:15.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa4a26700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa0184d30 0x7ffaa0185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:15.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa4a26700 1 -- 172.21.15.32:0/992352834 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa0185750 con 0x7ffaa00b42d0 2021-04-09T18:14:15.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.537 7ffaa5227700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa0184d30 0x7ffaa0185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:15.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.538 7ffaa4a26700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00b42d0 0x7ffaa0183520 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ffa900096d0 tx=0x7ffa9000b8c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:15.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.538 7ffa9d7fa700 1 -- 172.21.15.32:0/992352834 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa900009b0 con 0x7ffaa00b42d0 2021-04-09T18:14:15.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.538 7ffa9d7fa700 1 -- 172.21.15.32:0/992352834 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9000bcb0 con 0x7ffaa00b42d0 2021-04-09T18:14:15.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.538 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa0185750 con 0x7ffaa00b42d0 2021-04-09T18:14:15.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.538 7ffa9d7fa700 1 -- 172.21.15.32:0/992352834 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa900022d0 con 0x7ffaa00b42d0 2021-04-09T18:14:15.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.538 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa0185750 con 0x7ffaa00b42d0 2021-04-09T18:14:15.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.538 7ffa9d7fa700 1 -- 172.21.15.32:0/992352834 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ffa900064f0 con 0x7ffaa00b42d0 2021-04-09T18:14:15.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.538 7ffa9d7fa700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa800188d0 0x7ffa8001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:15.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.539 7ffa9d7fa700 1 -- 172.21.15.32:0/992352834 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ffa90032920 con 0x7ffaa00b42d0 2021-04-09T18:14:15.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.539 7ffa9ffff700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa800188d0 0x7ffa8001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:15.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.539 7ffa9ffff700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa800188d0 0x7ffa8001add0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:15.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.559 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa0182d10 con 0x7ffaa00b42d0 2021-04-09T18:14:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.562 7ffa9d7fa700 1 -- 172.21.15.32:0/992352834 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ffa90000c90 con 0x7ffaa00b42d0 2021-04-09T18:14:15.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.749 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ffaa010d480 con 0x7ffaa00b42d0 2021-04-09T18:14:15.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.750 7ffa9d7fa700 1 -- 172.21.15.32:0/992352834 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v32) v1 ==== 81+0+30360 (secure 0 0 0) 0x7ffa90000c90 con 0x7ffaa00b42d0 2021-04-09T18:14:15.911 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:15.912 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:14:15.912 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 32, 2021-04-09T18:14:15.914 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:14:15.915 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:14:15.915 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:13:37.891009", 2021-04-09T18:14:15.915 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:14:15.916 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:14:15.916 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:14:15.917 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:14:15.917 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:14:15.917 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:14:15.918 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:14:15.918 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:14:16.063 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:14:16.065 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:14:16.065 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:14:16.066 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:14:16.066 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:14:16.067 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:14:16.071 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:14:16.071 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 5, 2021-04-09T18:14:16.078 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:14:16.080 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:14:16.080 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:14:16.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:16.082 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:14:16.082 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:14:16.094 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:14:16.094 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:14:16.098 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:14:16.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:14:16.099 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:14:16.106 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:14:16.109 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:14:16.188 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:14:16.193 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:14:16.194 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:14:16.194 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:14:16.195 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:14:16.195 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:14:16.196 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:14:16.196 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:14:16.197 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:14:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:14:16.235 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:14:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:14:16.236 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:14:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:14:16.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:14:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:14:16.238 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:14:16.283 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:14:16.293 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:14:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:14:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:14:16.294 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:14:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:14:16.295 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:14:16.319 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:14:16.320 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:14:16.320 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:14:16.320 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:14:16.321 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:14:16.371 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:14:16.371 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:14:16.371 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:14:16.383 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:14:16.393 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:14:16.398 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:14:16.399 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:14:16.399 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:14:16.400 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:14:16.400 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:14:16.400 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:14:16.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:14:16.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:16.402 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:14:16.402 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:14:16.402 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:14:16.447 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:14:16.447 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:14:16.448 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:14:16.487 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:14:16.488 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:14:16.488 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:14:16.491 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:14:16.492 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:14:16.492 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:14:16.498 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:14:16.499 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:14:16.499 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:16.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:16.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:16.553 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 4, 2021-04-09T18:14:16.553 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:14:16.554 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:13:35.174131", 2021-04-09T18:14:16.554 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:14:16.580 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:14:16.581 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:14:16.590 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:14:16.591 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:14:16.591 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:14:16.591 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:14:16.603 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:14:16.604 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:14:16.604 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:14:16.604 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:14:16.605 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:14:16.605 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:14:16.606 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:14:16.606 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:14:16.628 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:14:16.628 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:14:16.639 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:14:16.640 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:14:16.640 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:14:16.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:16.648 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "31", 2021-04-09T18:14:16.649 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:14:16.658 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:14:16.661 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:14:16.662 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:14:16.678 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:14:16.678 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:14:16.687 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:14:16.689 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:14:16.690 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:14:16.690 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:14:16.691 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:14:16.691 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:14:16.691 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:14:16.692 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:14:16.692 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:14:16.724 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:14:16.725 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:14:16.725 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:14:16.725 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:14:16.745 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:14:16.746 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:14:16.746 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:14:16.748 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:14:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:14:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:14:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:14:16.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:16.789 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:14:16.789 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:14:16.790 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:14:16.790 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:14:16.790 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:14:16.791 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:14:16.791 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:14:16.792 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:14:16.792 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:14:16.792 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:14:16.877 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:14:16.877 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:14:16.878 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:14:16.885 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:14:16.885 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:14:16.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:16.886 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:14:16.886 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:14:16.886 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:14:16.887 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:16.887 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:16.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:16.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:16.893 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 5, 2021-04-09T18:14:16.893 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:14:16.893 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:13:35.938524", 2021-04-09T18:14:16.894 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:14:16.894 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:14:16.894 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:14:16.897 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:14:16.898 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:14:16.902 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:14:16.903 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:14:16.903 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:14:16.903 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:14:16.904 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:14:16.904 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:14:16.904 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:14:16.904 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:14:16.910 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:14:16.911 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:14:16.911 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:14:16.911 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:14:16.912 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:14:16.912 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:14:16.912 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:14:16.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:16.913 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "32", 2021-04-09T18:14:16.913 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:14:16.913 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:14:16.913 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:14:16.984 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:14:16.985 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:14:16.985 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:14:16.985 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:14:16.986 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:14:16.986 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:14:16.986 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:14:16.986 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:14:16.986 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:14:16.987 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:14:16.987 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:14:16.996 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:14:16.996 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:14:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:14:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:14:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:14:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:14:16.998 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:14:16.998 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:14:16.998 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:14:16.999 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:14:16.999 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:14:16.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:14:16.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.000 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:14:17.000 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:14:17.000 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:14:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:14:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:14:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:14:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:14:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:14:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:14:17.080 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:14:17.115 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:14:17.115 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:14:17.116 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:14:17.121 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:14:17.122 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:14:17.122 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.129 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.130 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:14:17.131 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:14:17.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.132 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:14:17.132 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:14:17.133 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:17.133 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:17.133 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:17.134 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:17.134 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:17.135 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:17.248 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:17.248 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:17.249 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:17.249 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:17.249 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:17.250 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:17.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.251 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:17.252 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:14:17.252 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:17.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:14:17.321 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:17.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.323 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:17.323 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:14:17.323 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:17.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.324 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:17.324 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:14:17.325 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:17.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.325 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:14:17.327 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:17.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.327 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:17.328 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:14:17.328 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:17.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.328 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.388 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:17.388 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:17.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.393 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:17.429 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:14:17.429 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:17.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:17.431 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:14:17.431 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:17.446 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.446 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.447 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:14:17.447 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:14:17.448 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:14:17.448 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:14:17.448 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:17.449 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:17.449 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:17.449 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.450 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:14:17.451 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:14:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:17.508 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:17.545 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:17.546 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 29, 2021-04-09T18:14:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:17.547 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:14:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:17.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:17.615 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:14:17.615 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:17.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.640 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.640 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:17.641 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:17.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:17.682 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:14:17.682 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:17.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.697 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:17.698 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:14:17.698 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:17.698 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.699 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.724 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:17.725 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:17.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:17.742 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:14:17.743 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:17.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.798 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:17.799 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:14:17.799 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:17.800 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.800 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.801 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:17.802 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:17.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.808 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:17.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:14:17.808 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:17.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.889 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:17.891 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:14:17.894 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:17.894 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:17.895 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.896 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:14:17.896 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:14:17.897 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:14:17.912 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:14:17.914 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:17.917 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:17.938 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:17.941 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:17.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.943 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:14:17.943 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:14:17.943 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:17.944 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:14:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:14:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:18.157 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:18.209 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.210 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:18.211 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:18.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.212 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:18.212 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:14:18.213 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:18.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.214 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.214 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:18.214 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:14:18.215 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:18.215 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:18.215 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.216 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:18.301 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:18.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.320 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:18.320 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:14:18.321 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:18.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.322 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:18.322 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:14:18.323 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:18.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:18.324 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.325 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:14:18.325 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:14:18.326 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:14:18.327 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:14:18.327 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:14:18.413 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:14:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:18.414 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:18.415 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:18.416 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:18.417 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:18.418 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:18.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:18.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:14:18.505 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:18.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:18.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:14:18.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:18.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:18.547 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.555 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:18.556 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:18.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:18.605 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:14:18.605 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:18.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:18.608 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:14:18.608 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:18.608 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.609 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:18.610 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:18.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:14:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:18.659 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:14:18.693 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:18.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:18.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.695 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:18.695 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:18.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.696 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:18.696 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:14:18.696 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:18.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.698 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:18.698 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:14:18.698 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:14:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:14:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:14:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:14:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:14:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:14:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:18.919 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:18.920 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:18.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:18.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:18.922 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:18.922 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:18.922 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:18.971 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:18.972 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:18.976 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:18.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:18.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:18.978 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:14:19.018 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:19.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.020 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.020 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:14:19.021 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:19.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.022 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.022 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:19.057 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.062 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:14:19.063 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:19.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.064 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.064 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:14:19.080 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.082 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:19.083 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:14:19.106 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:19.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.158 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:14:19.158 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:19.158 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.159 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.160 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:19.161 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.162 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.162 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:14:19.233 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:19.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.234 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.235 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:14:19.235 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:19.235 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.235 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.236 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:14:19.236 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:14:19.236 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:14:19.236 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:14:19.237 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:19.237 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:19.237 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:19.237 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.263 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:14:19.271 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:14:19.272 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:19.272 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:19.279 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:19.281 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:19.282 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:19.282 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:19.283 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:19.294 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:19.335 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:19.335 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:19.348 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:19.348 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.349 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.382 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:14:19.382 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:19.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.384 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:14:19.384 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:19.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.385 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.428 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:19.443 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.448 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:14:19.448 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.449 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.522 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:14:19.522 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:19.523 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.523 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.524 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:19.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:14:19.526 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:19.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.576 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.576 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:14:19.576 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:19.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.586 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.587 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:19.587 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.593 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.593 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:14:19.594 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:19.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.614 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:14:19.615 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:19.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:14:19.627 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:14:19.629 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:14:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:14:19.636 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:19.637 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:19.699 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.701 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:14:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:14:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:19.703 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:19.704 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:19.705 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:19.786 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:19.787 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:19.791 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.792 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.793 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:14:19.793 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:19.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.794 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.795 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:14:19.818 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:19.818 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.819 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.823 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:19.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:14:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.858 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:14:19.859 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:19.909 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:19.911 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.912 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:14:19.912 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:19.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.923 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:14:19.923 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:19.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.924 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.937 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:19.938 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.940 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:14:19.940 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:19.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:19.941 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:14:19.942 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:19.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:19.942 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.959 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:14:19.960 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:14:19.960 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:14:19.961 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:14:19.961 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:19.962 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:19.962 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:19.962 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:19.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:19.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.980 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:14:19.980 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:14:19.981 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:19.981 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:19.982 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:19.982 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:19.983 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:19.983 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:19.990 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:19.991 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:19.991 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:19.997 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:19.998 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:19.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:19.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:19.999 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:14:19.999 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:20.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.026 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:20.026 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:14:20.027 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:20.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:20.040 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:20.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.041 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:20.042 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:20.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.077 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:20.078 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:14:20.078 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:20.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.079 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:20.080 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:14:20.080 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:20.080 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:20.081 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:20.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.082 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:20.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:20.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:20.114 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:14:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:20.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:20.116 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:14:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:20.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:20.152 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.153 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:20.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:14:20.155 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:14:20.173 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:20.175 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:20.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.182 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:14:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:14:20.183 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:14:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:14:20.184 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:20.185 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:20.186 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:14:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:14:20.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:14:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:20.188 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:20.189 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:20.284 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:20.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.286 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:14:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:20.287 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:20.288 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:20.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:14:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:20.325 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:20.326 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:20.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:14:20.358 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:20.359 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:20.360 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:14:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:20.390 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:20.391 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:14:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:14:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:20.495 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:20.501 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:20.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:20.503 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:14:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:20.504 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:20.505 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:20.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:14:20.528 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:14:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:14:20.529 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:14:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:14:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:14:20.530 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:14:20.531 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:14:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:14:20.583 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:14:20.589 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:14:20.590 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:14:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:14:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:14:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:14:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:14:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:14:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:14:20.657 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:14:20.657 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:14:20.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa800188d0 msgr2=0x7ffa8001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:20.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa800188d0 0x7ffa8001add0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:20.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00b42d0 msgr2=0x7ffaa0183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:20.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00b42d0 0x7ffaa0183520 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ffa900096d0 tx=0x7ffa9000b8c0).stop 2021-04-09T18:14:20.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 shutdown_connections 2021-04-09T18:14:20.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa800188d0 0x7ffa8001add0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:20.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffaa0184d30 0x7ffaa0185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:20.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaa01ac090 0x7ffaa01ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:20.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 --2- 172.21.15.32:0/992352834 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaa00b42d0 0x7ffaa0183520 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:20.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 >> 172.21.15.32:0/992352834 conn(0x7ffaa019b890 msgr2=0x7ffaa01a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:20.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 shutdown_connections 2021-04-09T18:14:20.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:15.754 7ffaa6c0a700 1 -- 172.21.15.32:0/992352834 wait complete. 2021-04-09T18:14:20.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.872 7fb05eaa6700 1 Processor -- start 2021-04-09T18:14:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.872 7fb05eaa6700 1 -- start start 2021-04-09T18:14:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.873 7fb05eaa6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb05810da00 0x7fb05810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:20.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.873 7fb05eaa6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05810d480 con 0x7fb0580d7ca0 2021-04-09T18:14:20.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.873 7fb05eaa6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0580d1f70 con 0x7fb05810d010 2021-04-09T18:14:20.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.873 7fb05eaa6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0580d9220 con 0x7fb05810da00 2021-04-09T18:14:20.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.873 7fb057fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb0580d7ca0 0x7fb0580d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52714/0 (socket says 172.21.15.32:52714) 2021-04-09T18:14:20.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.873 7fb057fff700 1 -- 172.21.15.32:0/2711158094 learned_addr learned my addr 172.21.15.32:0/2711158094 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:20.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.873 7fb05d0c3700 1 --2- 172.21.15.32:0/2711158094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb05810da00 0x7fb05810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.873 7fb0577fe700 1 -- 172.21.15.32:0/2711158094 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084540412 0 0) 0x7fb04c000ce0 con 0x7fb0580d7ca0 2021-04-09T18:14:20.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb0577fe700 1 -- 172.21.15.32:0/2711158094 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb040002fe0 con 0x7fb0580d7ca0 2021-04-09T18:14:20.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb0577fe700 1 -- 172.21.15.32:0/2711158094 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813800570 0 0) 0x7fb048000c60 con 0x7fb05810d010 2021-04-09T18:14:20.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb0577fe700 1 -- 172.21.15.32:0/2711158094 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb040003210 con 0x7fb05810d010 2021-04-09T18:14:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb05d0c3700 1 -- 172.21.15.32:0/2711158094 >> v1:172.21.15.86:6789/0 conn(0x7fb05810d010 legacy=0x7fb0580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:20.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb05d0c3700 1 -- 172.21.15.32:0/2711158094 >> v1:172.21.15.32:6789/0 conn(0x7fb0580d7ca0 legacy=0x7fb0580d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb0577fe700 1 -- 172.21.15.32:0/2711158094 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4294660359 0 0) 0x7fb04c0010e0 con 0x7fb0580d7ca0 2021-04-09T18:14:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb05d0c3700 1 -- 172.21.15.32:0/2711158094 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058112100 con 0x7fb05810da00 2021-04-09T18:14:20.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb05d0c3700 1 --2- 172.21.15.32:0/2711158094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb05810da00 0x7fb05810de00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb050006e50 tx=0x7fb050009e70).ready entity=mon.2 client_cookie=3010064597d81530 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb0577fe700 1 -- 172.21.15.32:0/2711158094 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb050006a20 con 0x7fb05810da00 2021-04-09T18:14:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb0577fe700 1 -- 172.21.15.32:0/2711158094 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb050006790 con 0x7fb05810da00 2021-04-09T18:14:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb0577fe700 1 -- 172.21.15.32:0/2711158094 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05000b700 con 0x7fb05810da00 2021-04-09T18:14:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb05eaa6700 1 -- 172.21.15.32:0/2711158094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb05810da00 msgr2=0x7fb05810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.874 7fb05eaa6700 1 --2- 172.21.15.32:0/2711158094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb05810da00 0x7fb05810de00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb050006e50 tx=0x7fb050009e70).stop 2021-04-09T18:14:21.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.875 7fb05eaa6700 1 -- 172.21.15.32:0/2711158094 shutdown_connections 2021-04-09T18:14:21.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.875 7fb05eaa6700 1 --2- 172.21.15.32:0/2711158094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb05810da00 0x7fb05810de00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.875 7fb05eaa6700 1 -- 172.21.15.32:0/2711158094 >> 172.21.15.32:0/2711158094 conn(0x7fb0580ce5f0 msgr2=0x7fb0580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:21.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.875 7fb05eaa6700 1 -- 172.21.15.32:0/2711158094 shutdown_connections 2021-04-09T18:14:21.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.875 7fb05eaa6700 1 -- 172.21.15.32:0/2711158094 wait complete. 2021-04-09T18:14:21.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.895 7fb05eaa6700 1 Processor -- start 2021-04-09T18:14:21.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.895 7fb05eaa6700 1 -- start start 2021-04-09T18:14:21.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05eaa6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0581ac380 0x7fb05818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:21.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05eaa6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0581ac780 0x7fb05818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:21.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05eaa6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058183520 0x7fb058183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:21.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05eaa6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581ace10 con 0x7fb0581ac380 2021-04-09T18:14:21.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05eaa6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb058189cf0 con 0x7fb058183520 2021-04-09T18:14:21.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05eaa6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb058183f40 con 0x7fb0581ac780 2021-04-09T18:14:21.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05c8c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0581ac380 0x7fb05818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:21.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05c8c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0581ac380 0x7fb05818a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43128/0 (socket says 172.21.15.32:43128) 2021-04-09T18:14:21.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05c8c2700 1 -- 172.21.15.32:0/3875053398 learned_addr learned my addr 172.21.15.32:0/3875053398 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05d0c3700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058183520 0x7fb058183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:21.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb057fff700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0581ac780 0x7fb05818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:21.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05c8c2700 1 -- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0581ac780 msgr2=0x7fb05818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05c8c2700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0581ac780 0x7fb05818a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05c8c2700 1 -- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058183520 msgr2=0x7fb058183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05c8c2700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058183520 0x7fb058183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05c8c2700 1 -- 172.21.15.32:0/3875053398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058184160 con 0x7fb0581ac380 2021-04-09T18:14:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05d0c3700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058183520 0x7fb058183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:21.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.896 7fb05c8c2700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0581ac380 0x7fb05818a230 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb04800b580 tx=0x7fb04800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:21.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.897 7fb0557fa700 1 -- 172.21.15.32:0/3875053398 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb04800bd10 con 0x7fb0581ac380 2021-04-09T18:14:21.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.897 7fb0557fa700 1 -- 172.21.15.32:0/3875053398 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb048000e20 con 0x7fb0581ac380 2021-04-09T18:14:21.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.897 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb058184160 con 0x7fb0581ac380 2021-04-09T18:14:21.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.897 7fb0557fa700 1 -- 172.21.15.32:0/3875053398 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0480054f0 con 0x7fb0581ac380 2021-04-09T18:14:21.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.897 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb058184160 con 0x7fb0581ac380 2021-04-09T18:14:21.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.897 7fb0557fa700 1 -- 172.21.15.32:0/3875053398 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb0480059c0 con 0x7fb0581ac380 2021-04-09T18:14:21.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.897 7fb0557fa700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb038018930 0x7fb03801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:21.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.897 7fb0557fa700 1 -- 172.21.15.32:0/3875053398 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb048032a10 con 0x7fb0581ac380 2021-04-09T18:14:21.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.898 7fb057fff700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb038018930 0x7fb03801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:21.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.898 7fb057fff700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb038018930 0x7fb03801ae10 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:21.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.917 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb058182d10 con 0x7fb0581ac380 2021-04-09T18:14:21.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:20.921 7fb0557fa700 1 -- 172.21.15.32:0/3875053398 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb0480057b0 con 0x7fb0581ac380 2021-04-09T18:14:21.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.112 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb058182d10 con 0x7fb0581ac380 2021-04-09T18:14:21.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.113 7fb0557fa700 1 -- 172.21.15.32:0/3875053398 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2774 (secure 0 0 0) 0x7fb048005690 con 0x7fb0581ac380 2021-04-09T18:14:21.225 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 20 2021-04-09T18:14:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4960,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/425942817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":425942817},{"type":"v1","addr":"172.21.15.86:6835","nonce":425942817}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":20},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:14:11.854446","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4722":{"gid":4722,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6833/2659274405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2659274405},{"type":"v1","addr":"172.21.15.32:6833","nonce":2659274405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:21.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb038018930 msgr2=0x7fb03801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb038018930 0x7fb03801ae10 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0581ac380 msgr2=0x7fb05818a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0581ac380 0x7fb05818a230 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb04800b580 tx=0x7fb04800b920).stop 2021-04-09T18:14:21.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 shutdown_connections 2021-04-09T18:14:21.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb038018930 0x7fb03801ae10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058183520 0x7fb058183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0581ac780 0x7fb05818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 --2- 172.21.15.32:0/3875053398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0581ac380 0x7fb05818a230 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 >> 172.21.15.32:0/3875053398 conn(0x7fb05819b890 msgr2=0x7fb0581a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:21.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 shutdown_connections 2021-04-09T18:14:21.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.115 7fb05eaa6700 1 -- 172.21.15.32:0/3875053398 wait complete. 2021-04-09T18:14:21.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:14:21.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.482 7f5066b77700 1 Processor -- start 2021-04-09T18:14:21.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.482 7f5066b77700 1 -- start start 2021-04-09T18:14:21.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.482 7f5066b77700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50600d6c90 0x7f50600d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:21.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.482 7f5066b77700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50600d7290 con 0x7f50600d2450 2021-04-09T18:14:21.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.482 7f5066b77700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50600d2850 con 0x7f50600cffb0 2021-04-09T18:14:21.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.482 7f5066b77700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50600d7090 con 0x7f50600d6c90 2021-04-09T18:14:21.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.482 7f5065194700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f50600d2450 0x7f506010cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52724/0 (socket says 172.21.15.32:52724) 2021-04-09T18:14:21.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.482 7f5065194700 1 -- 172.21.15.32:0/679826287 learned_addr learned my addr 172.21.15.32:0/679826287 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:21.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505ffff700 1 --2- 172.21.15.32:0/679826287 >> v2:172.21.15.86:3300/0 conn(0x7f50600d6c90 0x7f50600d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:21.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505f7fe700 1 -- 172.21.15.32:0/679826287 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23293029 0 0) 0x7f50580009b0 con 0x7f50600d2450 2021-04-09T18:14:21.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505f7fe700 1 -- 172.21.15.32:0/679826287 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c002fe0 con 0x7f50600d2450 2021-04-09T18:14:21.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505f7fe700 1 -- 172.21.15.32:0/679826287 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 928908419 0 0) 0x7f50580010e0 con 0x7f50600d2450 2021-04-09T18:14:21.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505ffff700 1 -- 172.21.15.32:0/679826287 >> v1:172.21.15.86:6789/0 conn(0x7f50600cffb0 legacy=0x7f50600d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505ffff700 1 -- 172.21.15.32:0/679826287 >> v1:172.21.15.32:6789/0 conn(0x7f50600d2450 legacy=0x7f506010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505ffff700 1 -- 172.21.15.32:0/679826287 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600d91f0 con 0x7f50600d6c90 2021-04-09T18:14:21.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505ffff700 1 --2- 172.21.15.32:0/679826287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50600d6c90 0x7f50600d8000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f50500069d0 tx=0x7f5050009dc0).ready entity=mon.1 client_cookie=f964e036b5e8c6da server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:21.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505f7fe700 1 -- 172.21.15.32:0/679826287 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f505000b510 con 0x7f50600d6c90 2021-04-09T18:14:21.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505f7fe700 1 -- 172.21.15.32:0/679826287 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5050006ac0 con 0x7f50600d6c90 2021-04-09T18:14:21.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f505f7fe700 1 -- 172.21.15.32:0/679826287 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f505000ba80 con 0x7f50600d6c90 2021-04-09T18:14:21.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f5066b77700 1 -- 172.21.15.32:0/679826287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50600d6c90 msgr2=0x7f50600d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.483 7f5066b77700 1 --2- 172.21.15.32:0/679826287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50600d6c90 0x7f50600d8000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f50500069d0 tx=0x7f5050009dc0).stop 2021-04-09T18:14:21.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.484 7f5066b77700 1 -- 172.21.15.32:0/679826287 shutdown_connections 2021-04-09T18:14:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.484 7f5066b77700 1 --2- 172.21.15.32:0/679826287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50600d6c90 0x7f50600d8000 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.484 7f5066b77700 1 -- 172.21.15.32:0/679826287 >> 172.21.15.32:0/679826287 conn(0x7f50600ce5d0 msgr2=0x7f50600cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:21.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.484 7f5066b77700 1 -- 172.21.15.32:0/679826287 shutdown_connections 2021-04-09T18:14:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.484 7f5066b77700 1 -- 172.21.15.32:0/679826287 wait complete. 2021-04-09T18:14:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.504 7f5066b77700 1 Processor -- start 2021-04-09T18:14:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.504 7f5066b77700 1 -- start start 2021-04-09T18:14:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f5066b77700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f506018e880 0x7f50601812c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f5066b77700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5060180340 0x7f50601b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f5066b77700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50601b8c90 0x7f50600adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f5066b77700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060180d40 con 0x7f5060180340 2021-04-09T18:14:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f5066b77700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50601894f0 con 0x7f506018e880 2021-04-09T18:14:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f5066b77700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506018ec80 con 0x7f50601b8c90 2021-04-09T18:14:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f505ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5060180340 0x7f50601b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:21.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f505ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5060180340 0x7f50601b8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43138/0 (socket says 172.21.15.32:43138) 2021-04-09T18:14:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f505ffff700 1 -- 172.21.15.32:0/328251660 learned_addr learned my addr 172.21.15.32:0/328251660 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:21.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f5064993700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f506018e880 0x7f50601812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:21.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f5065194700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50601b8c90 0x7f50600adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:21.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f505ffff700 1 -- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50601b8c90 msgr2=0x7f50600adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f505ffff700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50601b8c90 0x7f50600adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f505ffff700 1 -- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f506018e880 msgr2=0x7f50601812c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:21.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f505ffff700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f506018e880 0x7f50601812c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:21.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.505 7f505ffff700 1 -- 172.21.15.32:0/328251660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50601818e0 con 0x7f5060180340 2021-04-09T18:14:21.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.506 7f505ffff700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5060180340 0x7f50601b8670 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f50500099e0 tx=0x7f505000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:21.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.506 7f505d7fa700 1 -- 172.21.15.32:0/328251660 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5050006030 con 0x7f5060180340 2021-04-09T18:14:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.506 7f505d7fa700 1 -- 172.21.15.32:0/328251660 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50500062f0 con 0x7f5060180340 2021-04-09T18:14:21.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.506 7f505d7fa700 1 -- 172.21.15.32:0/328251660 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5050005660 con 0x7f5060180340 2021-04-09T18:14:21.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.506 7f5066b77700 1 -- 172.21.15.32:0/328251660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50601818e0 con 0x7f5060180340 2021-04-09T18:14:21.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.506 7f5066b77700 1 -- 172.21.15.32:0/328251660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50601818e0 con 0x7f5060180340 2021-04-09T18:14:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.506 7f505d7fa700 1 -- 172.21.15.32:0/328251660 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f50500064d0 con 0x7f5060180340 2021-04-09T18:14:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.507 7f505d7fa700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5044018900 0x7f504401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.507 7f505d7fa700 1 -- 172.21.15.32:0/328251660 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5050032da0 con 0x7f5060180340 2021-04-09T18:14:21.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.507 7f5064993700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5044018900 0x7f504401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:21.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.507 7f5064993700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5044018900 0x7f504401adc0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:21.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.526 7f5066b77700 1 -- 172.21.15.32:0/328251660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506018ec80 con 0x7f5060180340 2021-04-09T18:14:21.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.530 7f505d7fa700 1 -- 172.21.15.32:0/328251660 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f505000be10 con 0x7f5060180340 2021-04-09T18:14:21.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:21.710 7f5066b77700 1 -- 172.21.15.32:0/328251660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f506018ec80 con 0x7f5060180340 2021-04-09T18:14:22.351 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:14:22.328 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:14:22.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.333 7f505d7fa700 1 -- 172.21.15.32:0/328251660 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) v1 ==== 161+0+0 (secure 0 0 0) 0x7f50500062f0 con 0x7f5060180340 2021-04-09T18:14:22.398 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:14:22.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.376 7f5066b77700 1 -- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5044018900 msgr2=0x7f504401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:22.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.376 7f5066b77700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5044018900 0x7f504401adc0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.376 7f5066b77700 1 -- 172.21.15.32:0/328251660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5060180340 msgr2=0x7f50601b8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:22.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.376 7f5066b77700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5060180340 0x7f50601b8670 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f50500099e0 tx=0x7f505000bcd0).stop 2021-04-09T18:14:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.377 7f5066b77700 1 -- 172.21.15.32:0/328251660 shutdown_connections 2021-04-09T18:14:22.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.377 7f5066b77700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5044018900 0x7f504401adc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.377 7f5066b77700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5060180340 0x7f50601b8670 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.377 7f5066b77700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50601b8c90 0x7f50600adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.377 7f5066b77700 1 --2- 172.21.15.32:0/328251660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f506018e880 0x7f50601812c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.377 7f5066b77700 1 -- 172.21.15.32:0/328251660 >> 172.21.15.32:0/328251660 conn(0x7f5060189800 msgr2=0x7f50601b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.377 7f5066b77700 1 -- 172.21.15.32:0/328251660 shutdown_connections 2021-04-09T18:14:22.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.377 7f5066b77700 1 -- 172.21.15.32:0/328251660 wait complete. 2021-04-09T18:14:22.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:22.424 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:14:22.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.548 7fe0c56b1700 1 Processor -- start 2021-04-09T18:14:22.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.548 7fe0c56b1700 1 -- start start 2021-04-09T18:14:22.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.548 7fe0c56b1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe0c010d010 0x7fe0c00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:22.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.548 7fe0c56b1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00d1da0 0x7fe0c00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:22.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.548 7fe0c56b1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0c00d82c0 0x7fe0c010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.548 7fe0c56b1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c00d8d30 con 0x7fe0c00d82c0 2021-04-09T18:14:22.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.548 7fe0c56b1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c00d9220 con 0x7fe0c010d010 2021-04-09T18:14:22.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.548 7fe0c56b1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c010d480 con 0x7fe0c00d1da0 2021-04-09T18:14:22.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.548 7fe0bfac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0c00d82c0 0x7fe0c010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:22.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0bfac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0c00d82c0 0x7fe0c010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43160/0 (socket says 172.21.15.32:43160) 2021-04-09T18:14:22.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0bfac0700 1 -- 172.21.15.32:0/3659535666 learned_addr learned my addr 172.21.15.32:0/3659535666 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:22.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0bf2bf700 1 --2- 172.21.15.32:0/3659535666 >> v2:172.21.15.86:3300/0 conn(0x7fe0c010d010 0x7fe0c00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:22.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0beabe700 1 --2- 172.21.15.32:0/3659535666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00d1da0 0x7fe0c00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:22.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0bfac0700 1 -- 172.21.15.32:0/3659535666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00d1da0 msgr2=0x7fe0c00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:22.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0bfac0700 1 --2- 172.21.15.32:0/3659535666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00d1da0 0x7fe0c00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0bfac0700 1 -- 172.21.15.32:0/3659535666 >> v2:172.21.15.86:3300/0 conn(0x7fe0c010d010 msgr2=0x7fe0c00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:22.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0bfac0700 1 --2- 172.21.15.32:0/3659535666 >> v2:172.21.15.86:3300/0 conn(0x7fe0c010d010 0x7fe0c00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0bfac0700 1 -- 172.21.15.32:0/3659535666 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c00d86c0 con 0x7fe0c00d82c0 2021-04-09T18:14:22.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0beabe700 1 --2- 172.21.15.32:0/3659535666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00d1da0 0x7fe0c00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:22.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0bfac0700 1 --2- 172.21.15.32:0/3659535666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00d82c0 0x7fe0c010da00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fe0b8006f00 tx=0x7fe0b8009f10).ready entity=mon.0 client_cookie=9f7c890654205e84 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:22.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0be2bd700 1 -- 172.21.15.32:0/3659535666 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0b800b3b0 con 0x7fe0c00d82c0 2021-04-09T18:14:22.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0be2bd700 1 -- 172.21.15.32:0/3659535666 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b800b730 con 0x7fe0c00d82c0 2021-04-09T18:14:22.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0be2bd700 1 -- 172.21.15.32:0/3659535666 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0b800bc10 con 0x7fe0c00d82c0 2021-04-09T18:14:22.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0c56b1700 1 -- 172.21.15.32:0/3659535666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00d82c0 msgr2=0x7fe0c010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:22.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.549 7fe0c56b1700 1 --2- 172.21.15.32:0/3659535666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00d82c0 0x7fe0c010da00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fe0b8006f00 tx=0x7fe0b8009f10).stop 2021-04-09T18:14:22.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.550 7fe0c56b1700 1 -- 172.21.15.32:0/3659535666 shutdown_connections 2021-04-09T18:14:22.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.550 7fe0c56b1700 1 --2- 172.21.15.32:0/3659535666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00d82c0 0x7fe0c010da00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.550 7fe0c56b1700 1 --2- 172.21.15.32:0/3659535666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00d1da0 0x7fe0c00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.550 7fe0c56b1700 1 --2- 172.21.15.32:0/3659535666 >> v2:172.21.15.86:3300/0 conn(0x7fe0c010d010 0x7fe0c00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.550 7fe0c56b1700 1 -- 172.21.15.32:0/3659535666 >> 172.21.15.32:0/3659535666 conn(0x7fe0c00ce5f0 msgr2=0x7fe0c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:22.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.550 7fe0c56b1700 1 -- 172.21.15.32:0/3659535666 shutdown_connections 2021-04-09T18:14:22.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.550 7fe0c56b1700 1 -- 172.21.15.32:0/3659535666 wait complete. 2021-04-09T18:14:22.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.570 7fe0c56b1700 1 Processor -- start 2021-04-09T18:14:22.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.570 7fe0c56b1700 1 -- start start 2021-04-09T18:14:22.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0c56b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00aa0f0 0x7fe0c01832b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0c56b1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c01838d0 0x7fe0c017ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0c56b1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c017f1d0 0x7fe0c017cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0c56b1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c017e630 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0c56b1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c0183cd0 con 0x7fe0c01838d0 2021-04-09T18:14:22.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0c56b1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c017f5d0 con 0x7fe0c017f1d0 2021-04-09T18:14:22.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bf2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00aa0f0 0x7fe0c01832b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:22.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bf2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00aa0f0 0x7fe0c01832b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43164/0 (socket says 172.21.15.32:43164) 2021-04-09T18:14:22.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bf2bf700 1 -- 172.21.15.32:0/4229950546 learned_addr learned my addr 172.21.15.32:0/4229950546 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:22.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bfac0700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c017f1d0 0x7fe0c017cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:22.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0beabe700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c01838d0 0x7fe0c017ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:22.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bf2bf700 1 -- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c017f1d0 msgr2=0x7fe0c017cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:22.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bf2bf700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c017f1d0 0x7fe0c017cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bf2bf700 1 -- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c01838d0 msgr2=0x7fe0c017ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:22.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bf2bf700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c01838d0 0x7fe0c017ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bf2bf700 1 -- 172.21.15.32:0/4229950546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c017d5e0 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.571 7fe0bf2bf700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00aa0f0 0x7fe0c01832b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe0b0006f60 tx=0x7fe0b000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:22.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.572 7fe0a7fff700 1 -- 172.21.15.32:0/4229950546 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0b000bd60 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.572 7fe0a7fff700 1 -- 172.21.15.32:0/4229950546 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b0000c90 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.572 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c017d5e0 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.572 7fe0a7fff700 1 -- 172.21.15.32:0/4229950546 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0b0002220 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.572 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c017d5e0 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.572 7fe0a7fff700 1 -- 172.21.15.32:0/4229950546 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe0b00052e0 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.572 7fe0a7fff700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0a0018930 0x7fe0a001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:22.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.572 7fe0a7fff700 1 -- 172.21.15.32:0/4229950546 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fe0b0032a20 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.573 7fe0beabe700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0a0018930 0x7fe0a001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:22.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.573 7fe0beabe700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0a0018930 0x7fe0a001adc0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.593 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c00f1850 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.596 7fe0a7fff700 1 -- 172.21.15.32:0/4229950546 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe0b0000c90 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.790 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0c00f1850 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.791 7fe0a7fff700 1 -- 172.21.15.32:0/4229950546 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2428 (secure 0 0 0) 0x7fe0b0006b00 con 0x7fe0c00aa0f0 2021-04-09T18:14:22.867 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 21 2021-04-09T18:14:22.877 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:22.878 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4960,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/425942817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":425942817},{"type":"v1","addr":"172.21.15.86:6835","nonce":425942817}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":20},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":21,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:13:36.889598","modified":"2021-04-09 18:14:22.329641","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-04-09T18:14:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0a0018930 msgr2=0x7fe0a001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0a0018930 0x7fe0a001adc0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00aa0f0 msgr2=0x7fe0c01832b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:22.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00aa0f0 0x7fe0c01832b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe0b0006f60 tx=0x7fe0b000b970).stop 2021-04-09T18:14:22.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 shutdown_connections 2021-04-09T18:14:22.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0a0018930 0x7fe0a001adc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c017f1d0 0x7fe0c017cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c01838d0 0x7fe0c017ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 --2- 172.21.15.32:0/4229950546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0c00aa0f0 0x7fe0c01832b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:22.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 >> 172.21.15.32:0/4229950546 conn(0x7fe0c01925d0 msgr2=0x7fe0c01a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:22.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 shutdown_connections 2021-04-09T18:14:22.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:22.793 7fe0c56b1700 1 -- 172.21.15.32:0/4229950546 wait complete. 2021-04-09T18:14:22.977 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:14:23.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.121 7fc8b0644700 1 Processor -- start 2021-04-09T18:14:23.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.121 7fc8b0644700 1 -- start start 2021-04-09T18:14:23.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.121 7fc8b0644700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc8a80cffb0 0x7fc8a80d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:23.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.121 7fc8b0644700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8a80d7810 0x7fc8a80d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:23.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.121 7fc8b0644700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a80d8230 0x7fc8a80d5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:23.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8b0644700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a80d91f0 con 0x7fc8a80d7810 2021-04-09T18:14:23.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8b0644700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a80d7290 con 0x7fc8a80cffb0 2021-04-09T18:14:23.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8b0644700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a80d62a0 con 0x7fc8a80d8230 2021-04-09T18:14:23.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8adc5f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8a80d7810 0x7fc8a80d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:23.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8adc5f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8a80d7810 0x7fc8a80d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43174/0 (socket says 172.21.15.32:43174) 2021-04-09T18:14:23.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8adc5f700 1 -- 172.21.15.32:0/3545625855 learned_addr learned my addr 172.21.15.32:0/3545625855 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:23.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8ae460700 1 --2- 172.21.15.32:0/3545625855 >> v2:172.21.15.86:3300/0 conn(0x7fc8a80cffb0 0x7fc8a80d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:23.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8aec61700 1 --2- 172.21.15.32:0/3545625855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a80d8230 0x7fc8a80d5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:23.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8adc5f700 1 -- 172.21.15.32:0/3545625855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a80d8230 msgr2=0x7fc8a80d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:23.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8adc5f700 1 --2- 172.21.15.32:0/3545625855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a80d8230 0x7fc8a80d5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:23.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8adc5f700 1 -- 172.21.15.32:0/3545625855 >> v2:172.21.15.86:3300/0 conn(0x7fc8a80cffb0 msgr2=0x7fc8a80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:23.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8adc5f700 1 --2- 172.21.15.32:0/3545625855 >> v2:172.21.15.86:3300/0 conn(0x7fc8a80cffb0 0x7fc8a80d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:23.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8adc5f700 1 -- 172.21.15.32:0/3545625855 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a80d6490 con 0x7fc8a80d7810 2021-04-09T18:14:23.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8ae460700 1 --2- 172.21.15.32:0/3545625855 >> v2:172.21.15.86:3300/0 conn(0x7fc8a80cffb0 0x7fc8a80d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:23.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8adc5f700 1 --2- 172.21.15.32:0/3545625855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a80d7810 0x7fc8a80d7c10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fc898006700 tx=0x7fc898009b00).ready entity=mon.0 client_cookie=2155a689f6fb25bc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:23.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8ad45e700 1 -- 172.21.15.32:0/3545625855 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc89800c0c0 con 0x7fc8a80d7810 2021-04-09T18:14:23.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8ad45e700 1 -- 172.21.15.32:0/3545625855 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89800c400 con 0x7fc8a80d7810 2021-04-09T18:14:23.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8ad45e700 1 -- 172.21.15.32:0/3545625855 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc89800c8e0 con 0x7fc8a80d7810 2021-04-09T18:14:23.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8b0644700 1 -- 172.21.15.32:0/3545625855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a80d7810 msgr2=0x7fc8a80d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:23.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.122 7fc8b0644700 1 --2- 172.21.15.32:0/3545625855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a80d7810 0x7fc8a80d7c10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fc898006700 tx=0x7fc898009b00).stop 2021-04-09T18:14:23.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.123 7fc8b0644700 1 -- 172.21.15.32:0/3545625855 shutdown_connections 2021-04-09T18:14:23.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.123 7fc8b0644700 1 --2- 172.21.15.32:0/3545625855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a80d8230 0x7fc8a80d5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:23.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.123 7fc8b0644700 1 --2- 172.21.15.32:0/3545625855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a80d7810 0x7fc8a80d7c10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:23.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.123 7fc8b0644700 1 --2- 172.21.15.32:0/3545625855 >> v2:172.21.15.86:3300/0 conn(0x7fc8a80cffb0 0x7fc8a80d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.123 7fc8b0644700 1 -- 172.21.15.32:0/3545625855 >> 172.21.15.32:0/3545625855 conn(0x7fc8a80ce5d0 msgr2=0x7fc8a80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.123 7fc8b0644700 1 -- 172.21.15.32:0/3545625855 shutdown_connections 2021-04-09T18:14:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.123 7fc8b0644700 1 -- 172.21.15.32:0/3545625855 wait complete. 2021-04-09T18:14:23.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.143 7fc8b0644700 1 Processor -- start 2021-04-09T18:14:23.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8b0644700 1 -- start start 2021-04-09T18:14:23.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8b0644700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a808a620 0x7fc8a808b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:23.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8b0644700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a808b6d0 0x7fc8a8180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8b0644700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a8180c50 0x7fc8a8181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:23.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8b0644700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a808e660 con 0x7fc8a808b6d0 2021-04-09T18:14:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8b0644700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a808ab30 con 0x7fc8a808a620 2021-04-09T18:14:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8b0644700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a808bad0 con 0x7fc8a8180c50 2021-04-09T18:14:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8adc5f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a808b6d0 0x7fc8a8180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8adc5f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a808b6d0 0x7fc8a8180630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43180/0 (socket says 172.21.15.32:43180) 2021-04-09T18:14:23.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8adc5f700 1 -- 172.21.15.32:0/3136679062 learned_addr learned my addr 172.21.15.32:0/3136679062 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:23.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8ae460700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a808a620 0x7fc8a808b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:23.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8aec61700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a8180c50 0x7fc8a8181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:23.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8adc5f700 1 -- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a8180c50 msgr2=0x7fc8a8181050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:23.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8adc5f700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a8180c50 0x7fc8a8181050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8adc5f700 1 -- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a808a620 msgr2=0x7fc8a808b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8adc5f700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a808a620 0x7fc8a808b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.144 7fc8adc5f700 1 -- 172.21.15.32:0/3136679062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a808e1f0 con 0x7fc8a808b6d0 2021-04-09T18:14:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.145 7fc8adc5f700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a808b6d0 0x7fc8a8180630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc89800b660 tx=0x7fc89800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:23.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.145 7fc89effd700 1 -- 172.21.15.32:0/3136679062 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc898009440 con 0x7fc8a808b6d0 2021-04-09T18:14:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.145 7fc89effd700 1 -- 172.21.15.32:0/3136679062 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc898006390 con 0x7fc8a808b6d0 2021-04-09T18:14:23.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.145 7fc89effd700 1 -- 172.21.15.32:0/3136679062 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc898002030 con 0x7fc8a808b6d0 2021-04-09T18:14:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.145 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a808e1f0 con 0x7fc8a808b6d0 2021-04-09T18:14:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.145 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a808e1f0 con 0x7fc8a808b6d0 2021-04-09T18:14:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.145 7fc89effd700 1 -- 172.21.15.32:0/3136679062 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc8980093e0 con 0x7fc8a808b6d0 2021-04-09T18:14:23.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.146 7fc89effd700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc890018930 0x7fc89001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:23.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.146 7fc89effd700 1 -- 172.21.15.32:0/3136679062 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fc898032a40 con 0x7fc8a808b6d0 2021-04-09T18:14:23.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.146 7fc8ae460700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc890018930 0x7fc89001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:23.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.146 7fc8ae460700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc890018930 0x7fc89001ae10 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:23.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.166 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a81b7180 con 0x7fc8a808b6d0 2021-04-09T18:14:23.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.169 7fc89effd700 1 -- 172.21.15.32:0/3136679062 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc8980090c0 con 0x7fc8a808b6d0 2021-04-09T18:14:23.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:23.364 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc8a80d7290 con 0x7fc8a808b6d0 2021-04-09T18:14:24.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.336 7fc89effd700 1 -- 172.21.15.32:0/3136679062 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc8980093e0 con 0x7fc8a808b6d0 2021-04-09T18:14:24.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc890018930 msgr2=0x7fc89001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:24.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc890018930 0x7fc89001ae10 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a808b6d0 msgr2=0x7fc8a8180630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:24.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a808b6d0 0x7fc8a8180630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc89800b660 tx=0x7fc89800ba00).stop 2021-04-09T18:14:24.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 shutdown_connections 2021-04-09T18:14:24.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc890018930 0x7fc89001ae10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8a808b6d0 0x7fc8a8180630 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a8180c50 0x7fc8a8181050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 --2- 172.21.15.32:0/3136679062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a808a620 0x7fc8a808b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 >> 172.21.15.32:0/3136679062 conn(0x7fc8a8189800 msgr2=0x7fc8a81b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:24.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 shutdown_connections 2021-04-09T18:14:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.384 7fc8b0644700 1 -- 172.21.15.32:0/3136679062 wait complete. 2021-04-09T18:14:24.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:14:24.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.572 7faf42f75700 1 Processor -- start 2021-04-09T18:14:24.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.572 7faf42f75700 1 -- start start 2021-04-09T18:14:24.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.572 7faf42f75700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faf3c10d050 0x7faf3c0cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:24.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.572 7faf42f75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faf3c0d0160 0x7faf3c0d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:24.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.572 7faf42f75700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0d82e0 0x7faf3c10da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:24.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.572 7faf42f75700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3c0d8d70 con 0x7faf3c0d0160 2021-04-09T18:14:24.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.572 7faf42f75700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3c0d9260 con 0x7faf3c10d050 2021-04-09T18:14:24.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.572 7faf42f75700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c10d530 con 0x7faf3c0d82e0 2021-04-09T18:14:24.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faf3c0d0160 0x7faf3c0d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:24.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faf3c0d0160 0x7faf3c0d7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43188/0 (socket says 172.21.15.32:43188) 2021-04-09T18:14:24.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3bfff700 1 -- 172.21.15.32:0/3990950693 learned_addr learned my addr 172.21.15.32:0/3990950693 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:24.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf40d91700 1 --2- 172.21.15.32:0/3990950693 >> v2:172.21.15.86:3300/0 conn(0x7faf3c10d050 0x7faf3c0cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:24.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf41592700 1 --2- 172.21.15.32:0/3990950693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0d82e0 0x7faf3c10da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:24.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3bfff700 1 -- 172.21.15.32:0/3990950693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0d82e0 msgr2=0x7faf3c10da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:24.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3bfff700 1 --2- 172.21.15.32:0/3990950693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0d82e0 0x7faf3c10da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3bfff700 1 -- 172.21.15.32:0/3990950693 >> v2:172.21.15.86:3300/0 conn(0x7faf3c10d050 msgr2=0x7faf3c0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:24.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3bfff700 1 --2- 172.21.15.32:0/3990950693 >> v2:172.21.15.86:3300/0 conn(0x7faf3c10d050 0x7faf3c0cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3bfff700 1 -- 172.21.15.32:0/3990950693 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c0d86e0 con 0x7faf3c0d0160 2021-04-09T18:14:24.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf40d91700 1 --2- 172.21.15.32:0/3990950693 >> v2:172.21.15.86:3300/0 conn(0x7faf3c10d050 0x7faf3c0cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:24.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3bfff700 1 --2- 172.21.15.32:0/3990950693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0d0160 0x7faf3c0d7cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7faf30006700 tx=0x7faf30009b00).ready entity=mon.0 client_cookie=5f9eb495da4a5e0a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:24.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3b7fe700 1 -- 172.21.15.32:0/3990950693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf3000c0c0 con 0x7faf3c0d0160 2021-04-09T18:14:24.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3b7fe700 1 -- 172.21.15.32:0/3990950693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3000c400 con 0x7faf3c0d0160 2021-04-09T18:14:24.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf3b7fe700 1 -- 172.21.15.32:0/3990950693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf3000c8e0 con 0x7faf3c0d0160 2021-04-09T18:14:24.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf42f75700 1 -- 172.21.15.32:0/3990950693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0d0160 msgr2=0x7faf3c0d7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:24.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.573 7faf42f75700 1 --2- 172.21.15.32:0/3990950693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0d0160 0x7faf3c0d7cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7faf30006700 tx=0x7faf30009b00).stop 2021-04-09T18:14:24.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.574 7faf42f75700 1 -- 172.21.15.32:0/3990950693 shutdown_connections 2021-04-09T18:14:24.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.574 7faf42f75700 1 --2- 172.21.15.32:0/3990950693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0d82e0 0x7faf3c10da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.574 7faf42f75700 1 --2- 172.21.15.32:0/3990950693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0d0160 0x7faf3c0d7cc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.574 7faf42f75700 1 --2- 172.21.15.32:0/3990950693 >> v2:172.21.15.86:3300/0 conn(0x7faf3c10d050 0x7faf3c0cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.574 7faf42f75700 1 -- 172.21.15.32:0/3990950693 >> 172.21.15.32:0/3990950693 conn(0x7faf3c0ce620 msgr2=0x7faf3c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:24.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.574 7faf42f75700 1 -- 172.21.15.32:0/3990950693 shutdown_connections 2021-04-09T18:14:24.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.574 7faf42f75700 1 -- 172.21.15.32:0/3990950693 wait complete. 2021-04-09T18:14:24.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.594 7faf42f75700 1 Processor -- start 2021-04-09T18:14:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.594 7faf42f75700 1 -- start start 2021-04-09T18:14:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf42f75700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf3c0ae5f0 0x7faf3c180040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf42f75700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0ab1a0 0x7faf3c1957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf42f75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0ab5a0 0x7faf3c195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf42f75700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0aac20 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf42f75700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0aedf0 con 0x7faf3c0ae5f0 2021-04-09T18:14:24.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf42f75700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0ae9f0 con 0x7faf3c0ab1a0 2021-04-09T18:14:24.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf41592700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0ab5a0 0x7faf3c195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:24.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf41592700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0ab5a0 0x7faf3c195df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43196/0 (socket says 172.21.15.32:43196) 2021-04-09T18:14:24.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf41592700 1 -- 172.21.15.32:0/2489086000 learned_addr learned my addr 172.21.15.32:0/2489086000 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:24.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf40d91700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf3c0ae5f0 0x7faf3c180040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:24.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf3bfff700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0ab1a0 0x7faf3c1957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:24.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf41592700 1 -- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0ab1a0 msgr2=0x7faf3c1957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:24.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf41592700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0ab1a0 0x7faf3c1957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf41592700 1 -- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf3c0ae5f0 msgr2=0x7faf3c180040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:24.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf41592700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf3c0ae5f0 0x7faf3c180040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:24.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.595 7faf41592700 1 -- 172.21.15.32:0/2489086000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c0bb370 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.596 7faf41592700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0ab5a0 0x7faf3c195df0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7faf340099a0 tx=0x7faf3400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.596 7faf397fa700 1 -- 172.21.15.32:0/2489086000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf340011f0 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.596 7faf397fa700 1 -- 172.21.15.32:0/2489086000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf34000fc0 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.596 7faf42f75700 1 -- 172.21.15.32:0/2489086000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3c0bb370 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.596 7faf397fa700 1 -- 172.21.15.32:0/2489086000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf340016a0 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.596 7faf42f75700 1 -- 172.21.15.32:0/2489086000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3c0bb370 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.596 7faf397fa700 1 -- 172.21.15.32:0/2489086000 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faf340068d0 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.597 7faf397fa700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf1c018930 0x7faf1c01adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:24.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.597 7faf397fa700 1 -- 172.21.15.32:0/2489086000 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5663+0+0 (secure 0 0 0) 0x7faf34032df0 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.597 7faf40d91700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf1c018930 0x7faf1c01adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:24.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.597 7faf40d91700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf1c018930 0x7faf1c01adf0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:24.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.616 7faf42f75700 1 -- 172.21.15.32:0/2489086000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3c0aeda0 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.620 7faf397fa700 1 -- 172.21.15.32:0/2489086000 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faf3400be10 con 0x7faf3c0ab5a0 2021-04-09T18:14:24.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:24.802 7faf42f75700 1 -- 172.21.15.32:0/2489086000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faf3c0aeda0 con 0x7faf3c0ab5a0 2021-04-09T18:14:25.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.338 7faf397fa700 1 -- 172.21.15.32:0/2489086000 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v34) v1 ==== 185+0+0 (secure 0 0 0) 0x7faf340068d0 con 0x7faf3c0ab5a0 2021-04-09T18:14:25.391 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:14:25.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 -- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf1c018930 msgr2=0x7faf1c01adf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:25.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf1c018930 0x7faf1c01adf0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 -- 172.21.15.32:0/2489086000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0ab5a0 msgr2=0x7faf3c195df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:25.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0ab5a0 0x7faf3c195df0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7faf340099a0 tx=0x7faf3400bcd0).stop 2021-04-09T18:14:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 -- 172.21.15.32:0/2489086000 shutdown_connections 2021-04-09T18:14:25.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf1c018930 0x7faf1c01adf0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c0ab5a0 0x7faf3c195df0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf3c0ab1a0 0x7faf3c1957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 --2- 172.21.15.32:0/2489086000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf3c0ae5f0 0x7faf3c180040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 -- 172.21.15.32:0/2489086000 >> 172.21.15.32:0/2489086000 conn(0x7faf3c0c3890 msgr2=0x7faf3c0af810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:25.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 -- 172.21.15.32:0/2489086000 shutdown_connections 2021-04-09T18:14:25.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.369 7faf42f75700 1 -- 172.21.15.32:0/2489086000 wait complete. 2021-04-09T18:14:25.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:14:25.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.598 7f4848397700 1 Processor -- start 2021-04-09T18:14:25.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.598 7f4848397700 1 -- start start 2021-04-09T18:14:25.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f4848397700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f484010d050 0x7f48400cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:25.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f4848397700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48400d0160 0x7f48400d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:25.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f4848397700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48400d82e0 0x7f484010da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:25.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f4848397700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48400d8d70 con 0x7f48400d0160 2021-04-09T18:14:25.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f4848397700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48400d9260 con 0x7f484010d050 2021-04-09T18:14:25.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f4848397700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484010d530 con 0x7f48400d82e0 2021-04-09T18:14:25.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48459b2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48400d0160 0x7f48400d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48459b2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48400d0160 0x7f48400d7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43204/0 (socket says 172.21.15.32:43204) 2021-04-09T18:14:25.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48459b2700 1 -- 172.21.15.32:0/3885286208 learned_addr learned my addr 172.21.15.32:0/3885286208 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:25.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48469b4700 1 --2- 172.21.15.32:0/3885286208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48400d82e0 0x7f484010da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:25.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48461b3700 1 --2- 172.21.15.32:0/3885286208 >> v2:172.21.15.86:3300/0 conn(0x7f484010d050 0x7f48400cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:25.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48459b2700 1 -- 172.21.15.32:0/3885286208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48400d82e0 msgr2=0x7f484010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:25.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48459b2700 1 --2- 172.21.15.32:0/3885286208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48400d82e0 0x7f484010da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48459b2700 1 -- 172.21.15.32:0/3885286208 >> v2:172.21.15.86:3300/0 conn(0x7f484010d050 msgr2=0x7f48400cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:25.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48459b2700 1 --2- 172.21.15.32:0/3885286208 >> v2:172.21.15.86:3300/0 conn(0x7f484010d050 0x7f48400cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.599 7f48459b2700 1 -- 172.21.15.32:0/3885286208 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48400d86e0 con 0x7f48400d0160 2021-04-09T18:14:25.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.600 7f48459b2700 1 --2- 172.21.15.32:0/3885286208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400d0160 0x7f48400d7cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f4830006700 tx=0x7f4830009b00).ready entity=mon.0 client_cookie=ab6b72c008ca2886 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:25.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.600 7f48451b1700 1 -- 172.21.15.32:0/3885286208 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483000c0c0 con 0x7f48400d0160 2021-04-09T18:14:25.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.600 7f48451b1700 1 -- 172.21.15.32:0/3885286208 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483000c400 con 0x7f48400d0160 2021-04-09T18:14:25.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.600 7f48451b1700 1 -- 172.21.15.32:0/3885286208 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483000c8e0 con 0x7f48400d0160 2021-04-09T18:14:25.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.600 7f4848397700 1 -- 172.21.15.32:0/3885286208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400d0160 msgr2=0x7f48400d7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:25.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.600 7f4848397700 1 --2- 172.21.15.32:0/3885286208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400d0160 0x7f48400d7cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f4830006700 tx=0x7f4830009b00).stop 2021-04-09T18:14:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.601 7f4848397700 1 -- 172.21.15.32:0/3885286208 shutdown_connections 2021-04-09T18:14:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.601 7f4848397700 1 --2- 172.21.15.32:0/3885286208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48400d82e0 0x7f484010da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.601 7f4848397700 1 --2- 172.21.15.32:0/3885286208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400d0160 0x7f48400d7cc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.601 7f4848397700 1 --2- 172.21.15.32:0/3885286208 >> v2:172.21.15.86:3300/0 conn(0x7f484010d050 0x7f48400cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.601 7f4848397700 1 -- 172.21.15.32:0/3885286208 >> 172.21.15.32:0/3885286208 conn(0x7f48400ce620 msgr2=0x7f48400d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.601 7f4848397700 1 -- 172.21.15.32:0/3885286208 shutdown_connections 2021-04-09T18:14:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.601 7f4848397700 1 -- 172.21.15.32:0/3885286208 wait complete. 2021-04-09T18:14:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.621 7f4848397700 1 Processor -- start 2021-04-09T18:14:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.621 7f4848397700 1 -- start start 2021-04-09T18:14:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.621 7f4848397700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48400ad5b0 0x7f48401926b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.621 7f4848397700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400990f0 0x7f48400994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.621 7f4848397700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48401a30b0 0x7f48401a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.621 7f4848397700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4840098b70 con 0x7f48400990f0 2021-04-09T18:14:25.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.621 7f4848397700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48400988a0 con 0x7f48400ad5b0 2021-04-09T18:14:25.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.621 7f4848397700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4840099b10 con 0x7f48401a30b0 2021-04-09T18:14:25.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.621 7f48459b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400990f0 0x7f48400994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48459b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400990f0 0x7f48400994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43208/0 (socket says 172.21.15.32:43208) 2021-04-09T18:14:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48459b2700 1 -- 172.21.15.32:0/3659527628 learned_addr learned my addr 172.21.15.32:0/3659527628 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48461b3700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48400ad5b0 0x7f48401926b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48469b4700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48401a30b0 0x7f48401a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48459b2700 1 -- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48401a30b0 msgr2=0x7f48401a34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:25.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48459b2700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48401a30b0 0x7f48401a34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48459b2700 1 -- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48400ad5b0 msgr2=0x7f48401926b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:25.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48459b2700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48400ad5b0 0x7f48401926b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:25.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48459b2700 1 -- 172.21.15.32:0/3659527628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48401a3ad0 con 0x7f48400990f0 2021-04-09T18:14:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f48459b2700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400990f0 0x7f48400994f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f4830006f20 tx=0x7f483000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:25.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f4836ffd700 1 -- 172.21.15.32:0/3659527628 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483000bdd0 con 0x7f48400990f0 2021-04-09T18:14:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f4836ffd700 1 -- 172.21.15.32:0/3659527628 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4830006440 con 0x7f48400990f0 2021-04-09T18:14:25.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f4848397700 1 -- 172.21.15.32:0/3659527628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48401a3ad0 con 0x7f48400990f0 2021-04-09T18:14:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f4836ffd700 1 -- 172.21.15.32:0/3659527628 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48300052e0 con 0x7f48400990f0 2021-04-09T18:14:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.622 7f4848397700 1 -- 172.21.15.32:0/3659527628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48401a3ad0 con 0x7f48400990f0 2021-04-09T18:14:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.623 7f4836ffd700 1 -- 172.21.15.32:0/3659527628 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f48300052e0 con 0x7f48400990f0 2021-04-09T18:14:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.623 7f4836ffd700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48280188c0 0x7f482801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:25.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.623 7f4836ffd700 1 -- 172.21.15.32:0/3659527628 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5225+0+0 (secure 0 0 0) 0x7f4830032650 con 0x7f48400990f0 2021-04-09T18:14:25.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.623 7f48461b3700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48280188c0 0x7f482801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:25.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.624 7f48461b3700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48280188c0 0x7f482801adc0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:25.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.643 7f4848397700 1 -- 172.21.15.32:0/3659527628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48400b9040 con 0x7f48400990f0 2021-04-09T18:14:25.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.646 7f4836ffd700 1 -- 172.21.15.32:0/3659527628 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4830006030 con 0x7f48400990f0 2021-04-09T18:14:25.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:25.829 7f4848397700 1 -- 172.21.15.32:0/3659527628 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f484010d500 con 0x7f48400990f0 2021-04-09T18:14:26.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.343 7f4836ffd700 1 -- 172.21.15.32:0/3659527628 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v35) v1 ==== 173+0+0 (secure 0 0 0) 0x7f48300052e0 con 0x7f48400990f0 2021-04-09T18:14:26.367 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:14:26.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 -- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48280188c0 msgr2=0x7f482801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48280188c0 0x7f482801adc0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 -- 172.21.15.32:0/3659527628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400990f0 msgr2=0x7f48400994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:26.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400990f0 0x7f48400994f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f4830006f20 tx=0x7f483000b9e0).stop 2021-04-09T18:14:26.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 -- 172.21.15.32:0/3659527628 shutdown_connections 2021-04-09T18:14:26.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48280188c0 0x7f482801adc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48400990f0 0x7f48400994f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48401a30b0 0x7f48401a34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 --2- 172.21.15.32:0/3659527628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48400ad5b0 0x7f48401926b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 -- 172.21.15.32:0/3659527628 >> 172.21.15.32:0/3659527628 conn(0x7f48400ae6f0 msgr2=0x7f4840097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:26.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 -- 172.21.15.32:0/3659527628 shutdown_connections 2021-04-09T18:14:26.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.347 7f4848397700 1 -- 172.21.15.32:0/3659527628 wait complete. 2021-04-09T18:14:26.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:14:26.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.523 7fe84227c700 1 Processor -- start 2021-04-09T18:14:26.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.523 7fe84227c700 1 -- start start 2021-04-09T18:14:26.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.523 7fe84227c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c0cffb0 0x7fe83c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:26.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.523 7fe84227c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe83c0d6c90 0x7fe83c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:26.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.523 7fe84227c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83c0d7290 con 0x7fe83c0d7810 2021-04-09T18:14:26.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.523 7fe84227c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe83c0d7090 con 0x7fe83c0d6c90 2021-04-09T18:14:26.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.523 7fe84227c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c0d8590 con 0x7fe83c0cffb0 2021-04-09T18:14:26.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe83bfff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c0cffb0 0x7fe83c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe840899700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe83c0d6c90 0x7fe83c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:26.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe833fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe83c0d7810 0x7fe83c0d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:50734/0 (socket says 172.21.15.32:50734) 2021-04-09T18:14:26.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe833fff700 1 -- 172.21.15.32:0/3972303602 learned_addr learned my addr 172.21.15.32:0/3972303602 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:26.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe83b7fe700 1 -- 172.21.15.32:0/3972303602 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968632604 0 0) 0x7fe8240009b0 con 0x7fe83c0d7810 2021-04-09T18:14:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe83b7fe700 1 -- 172.21.15.32:0/3972303602 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe828002fe0 con 0x7fe83c0d7810 2021-04-09T18:14:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe840899700 1 -- 172.21.15.32:0/3972303602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c0cffb0 msgr2=0x7fe83c0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe840899700 1 --2- 172.21.15.32:0/3972303602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c0cffb0 0x7fe83c0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe840899700 1 -- 172.21.15.32:0/3972303602 >> v1:172.21.15.86:6789/0 conn(0x7fe83c0d7810 legacy=0x7fe83c0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:26.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe840899700 1 -- 172.21.15.32:0/3972303602 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c0d91f0 con 0x7fe83c0d6c90 2021-04-09T18:14:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe83bfff700 1 --2- 172.21.15.32:0/3972303602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c0cffb0 0x7fe83c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe840899700 1 --2- 172.21.15.32:0/3972303602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe83c0d6c90 0x7fe83c10cfe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fe834006e50 tx=0x7fe834009e70).ready entity=mon.1 client_cookie=f59c0997d8075c02 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:26.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe83b7fe700 1 -- 172.21.15.32:0/3972303602 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe834006a20 con 0x7fe83c0d6c90 2021-04-09T18:14:26.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe83b7fe700 1 -- 172.21.15.32:0/3972303602 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe834006790 con 0x7fe83c0d6c90 2021-04-09T18:14:26.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.524 7fe83b7fe700 1 -- 172.21.15.32:0/3972303602 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe834000f90 con 0x7fe83c0d6c90 2021-04-09T18:14:26.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.525 7fe84227c700 1 -- 172.21.15.32:0/3972303602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe83c0d6c90 msgr2=0x7fe83c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:26.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.525 7fe84227c700 1 --2- 172.21.15.32:0/3972303602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe83c0d6c90 0x7fe83c10cfe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fe834006e50 tx=0x7fe834009e70).stop 2021-04-09T18:14:26.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.525 7fe84227c700 1 -- 172.21.15.32:0/3972303602 shutdown_connections 2021-04-09T18:14:26.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.525 7fe84227c700 1 --2- 172.21.15.32:0/3972303602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe83c0d6c90 0x7fe83c10cfe0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.525 7fe84227c700 1 --2- 172.21.15.32:0/3972303602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c0cffb0 0x7fe83c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.525 7fe84227c700 1 -- 172.21.15.32:0/3972303602 >> 172.21.15.32:0/3972303602 conn(0x7fe83c0ce5d0 msgr2=0x7fe83c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:26.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.525 7fe84227c700 1 -- 172.21.15.32:0/3972303602 shutdown_connections 2021-04-09T18:14:26.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.525 7fe84227c700 1 -- 172.21.15.32:0/3972303602 wait complete. 2021-04-09T18:14:26.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.545 7fe84227c700 1 Processor -- start 2021-04-09T18:14:26.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.545 7fe84227c700 1 -- start start 2021-04-09T18:14:26.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe84227c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c1b8340 0x7fe83c1b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:26.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe84227c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe83c192280 0x7fe83c192680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:26.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe84227c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe83c190ab0 0x7fe83c190eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:26.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe84227c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c1b8740 con 0x7fe83c192280 2021-04-09T18:14:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe84227c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c08fd40 con 0x7fe83c190ab0 2021-04-09T18:14:26.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe84227c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c08ff60 con 0x7fe83c1b8340 2021-04-09T18:14:26.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe833fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe83c192280 0x7fe83c192680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:26.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe833fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe83c192280 0x7fe83c192680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43220/0 (socket says 172.21.15.32:43220) 2021-04-09T18:14:26.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe833fff700 1 -- 172.21.15.32:0/2149327184 learned_addr learned my addr 172.21.15.32:0/2149327184 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:26.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe83bfff700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c1b8340 0x7fe83c1b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:26.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.546 7fe840899700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe83c190ab0 0x7fe83c190eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:26.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe833fff700 1 -- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c1b8340 msgr2=0x7fe83c1b8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:26.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe833fff700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c1b8340 0x7fe83c1b8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe833fff700 1 -- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe83c190ab0 msgr2=0x7fe83c190eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:26.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe833fff700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe83c190ab0 0x7fe83c190eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:26.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe833fff700 1 -- 172.21.15.32:0/2149327184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c08ea30 con 0x7fe83c192280 2021-04-09T18:14:26.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe833fff700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe83c192280 0x7fe83c192680 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fe824001c60 tx=0x7fe82400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:26.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe8397fa700 1 -- 172.21.15.32:0/2149327184 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe82400acc0 con 0x7fe83c192280 2021-04-09T18:14:26.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe8397fa700 1 -- 172.21.15.32:0/2149327184 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe824004140 con 0x7fe83c192280 2021-04-09T18:14:26.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe84227c700 1 -- 172.21.15.32:0/2149327184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83c08ea30 con 0x7fe83c192280 2021-04-09T18:14:26.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe8397fa700 1 -- 172.21.15.32:0/2149327184 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe824000e20 con 0x7fe83c192280 2021-04-09T18:14:26.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe84227c700 1 -- 172.21.15.32:0/2149327184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83c08ea30 con 0x7fe83c192280 2021-04-09T18:14:26.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.547 7fe8397fa700 1 -- 172.21.15.32:0/2149327184 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe824004320 con 0x7fe83c192280 2021-04-09T18:14:26.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.548 7fe8397fa700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe820018930 0x7fe82001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:26.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.548 7fe8397fa700 1 -- 172.21.15.32:0/2149327184 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fe824032710 con 0x7fe83c192280 2021-04-09T18:14:26.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.548 7fe83bfff700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe820018930 0x7fe82001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:26.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.548 7fe83bfff700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe820018930 0x7fe82001ae10 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.567 7fe84227c700 1 -- 172.21.15.32:0/2149327184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83c191d50 con 0x7fe83c192280 2021-04-09T18:14:27.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.571 7fe8397fa700 1 -- 172.21.15.32:0/2149327184 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe8240009b0 con 0x7fe83c192280 2021-04-09T18:14:27.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.781 7fe84227c700 1 -- 172.21.15.32:0/2149327184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe83c191d50 con 0x7fe83c192280 2021-04-09T18:14:27.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.784 7fe8397fa700 1 -- 172.21.15.32:0/2149327184 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe824004140 con 0x7fe83c192280 2021-04-09T18:14:27.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 -- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe820018930 msgr2=0x7fe82001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:27.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe820018930 0x7fe82001ae10 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 -- 172.21.15.32:0/2149327184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe83c192280 msgr2=0x7fe83c192680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:27.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe83c192280 0x7fe83c192680 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fe824001c60 tx=0x7fe82400a8d0).stop 2021-04-09T18:14:27.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 -- 172.21.15.32:0/2149327184 shutdown_connections 2021-04-09T18:14:27.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe820018930 0x7fe82001ae10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe83c192280 0x7fe83c192680 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe83c190ab0 0x7fe83c190eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 --2- 172.21.15.32:0/2149327184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe83c1b8340 0x7fe83c1b8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 -- 172.21.15.32:0/2149327184 >> 172.21.15.32:0/2149327184 conn(0x7fe83c1ad1e0 msgr2=0x7fe83c1b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:27.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 -- 172.21.15.32:0/2149327184 shutdown_connections 2021-04-09T18:14:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:26.790 7fe84227c700 1 -- 172.21.15.32:0/2149327184 wait complete. 2021-04-09T18:14:27.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down' 2021-04-09T18:14:27.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.224 7f9ad03fd700 1 Processor -- start 2021-04-09T18:14:27.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.224 7f9ad03fd700 1 -- start start 2021-04-09T18:14:27.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.224 7f9ad03fd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac810d010 0x7f9ac80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:27.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.224 7f9ad03fd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac810d480 con 0x7f9ac80d8490 2021-04-09T18:14:27.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.224 7f9ad03fd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac810e1f0 con 0x7f9ac80d1da0 2021-04-09T18:14:27.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.224 7f9ad03fd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac810e780 con 0x7f9ac810d010 2021-04-09T18:14:27.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acea1a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9ac80d8490 0x7f9ac810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52820/0 (socket says 172.21.15.32:52820) 2021-04-09T18:14:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acea1a700 1 -- 172.21.15.32:0/3423755113 learned_addr learned my addr 172.21.15.32:0/3423755113 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9ace219700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac810d010 0x7f9ac80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:27.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170564203 0 0) 0x7f9ab80009b0 con 0x7f9ac80d8490 2021-04-09T18:14:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab0002fe0 con 0x7f9ac80d8490 2021-04-09T18:14:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344026744 0 0) 0x7f9abc000c60 con 0x7f9ac80d1da0 2021-04-09T18:14:27.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab0003210 con 0x7f9ac80d1da0 2021-04-09T18:14:27.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4124662932 0 0) 0x7f9ab8000ef0 con 0x7f9ac80d8490 2021-04-09T18:14:27.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac810d010 msgr2=0x7f9ac80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:27.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac810d010 0x7f9ac80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 >> v1:172.21.15.86:6789/0 conn(0x7f9ac80d1da0 legacy=0x7f9ac80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:27.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac80d8d30 con 0x7f9ac80d8490 2021-04-09T18:14:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f9ab8001600 con 0x7f9ac80d8490 2021-04-09T18:14:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 >> v1:172.21.15.32:6789/0 conn(0x7f9ac80d8490 legacy=0x7f9ac810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0003bc0 0x7f9ab0005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:27.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab00067a0 0x7f9ab0008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ab0009520 0x7f9ab000ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0050300 con 0x7f9ab0009520 2021-04-09T18:14:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00504e0 con 0x7f9ab00067a0 2021-04-09T18:14:27.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acd217700 1 -- 172.21.15.32:0/3423755113 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0050700 con 0x7f9ab0003bc0 2021-04-09T18:14:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acea1a700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ab0009520 0x7f9ab000ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9ace219700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0003bc0 0x7f9ab0005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.225 7f9acda18700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab00067a0 0x7f9ab0008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acea1a700 1 -- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0003bc0 msgr2=0x7f9ab0005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:27.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acea1a700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0003bc0 0x7f9ab0005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acea1a700 1 -- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab00067a0 msgr2=0x7f9ab0008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:27.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acea1a700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab00067a0 0x7f9ab0008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acea1a700 1 -- 172.21.15.32:0/3423755113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00509a0 con 0x7f9ab0009520 2021-04-09T18:14:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acda18700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab00067a0 0x7f9ab0008c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acea1a700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ab0009520 0x7f9ab000ba40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f9ab8001410 tx=0x7f9ab8007c70).ready entity=mon.0 client_cookie=65e5588e509df3cf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acd217700 1 -- 172.21.15.32:0/3423755113 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ab8001480 con 0x7f9ab0009520 2021-04-09T18:14:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acd217700 1 -- 172.21.15.32:0/3423755113 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab80070c0 con 0x7f9ab0009520 2021-04-09T18:14:27.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acd217700 1 -- 172.21.15.32:0/3423755113 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ab8000ef0 con 0x7f9ab0009520 2021-04-09T18:14:27.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9ad03fd700 1 -- 172.21.15.32:0/3423755113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ab0009520 msgr2=0x7f9ab000ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:27.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9ad03fd700 1 --2- 172.21.15.32:0/3423755113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ab0009520 0x7f9ab000ba40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f9ab8001410 tx=0x7f9ab8007c70).stop 2021-04-09T18:14:27.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.226 7f9acea1a700 1 -- 172.21.15.32:0/3423755113 reap_dead start 2021-04-09T18:14:27.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.227 7f9ad03fd700 1 -- 172.21.15.32:0/3423755113 shutdown_connections 2021-04-09T18:14:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.227 7f9ad03fd700 1 -- 172.21.15.32:0/3423755113 >> 172.21.15.32:0/3423755113 conn(0x7f9ac80ce5f0 msgr2=0x7f9ac80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.227 7f9ad03fd700 1 -- 172.21.15.32:0/3423755113 shutdown_connections 2021-04-09T18:14:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.227 7f9ad03fd700 1 -- 172.21.15.32:0/3423755113 wait complete. 2021-04-09T18:14:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.247 7f9ad03fd700 1 Processor -- start 2021-04-09T18:14:27.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.247 7f9ad03fd700 1 -- start start 2021-04-09T18:14:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ad03fd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac80b5360 0x7f9ac81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ad03fd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ac818e140 0x7f9ac81aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ad03fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac81ad040 0x7f9ac808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ad03fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac81a5360 con 0x7f9ac81ad040 2021-04-09T18:14:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ad03fd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80902c0 con 0x7f9ac818e140 2021-04-09T18:14:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ad03fd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80904e0 con 0x7f9ac80b5360 2021-04-09T18:14:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acea1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac81ad040 0x7f9ac808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acea1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac81ad040 0x7f9ac808fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43244/0 (socket says 172.21.15.32:43244) 2021-04-09T18:14:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acea1a700 1 -- 172.21.15.32:0/449640876 learned_addr learned my addr 172.21.15.32:0/449640876 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acda18700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ac818e140 0x7f9ac81aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ace219700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac80b5360 0x7f9ac81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acea1a700 1 -- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac80b5360 msgr2=0x7f9ac81a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acea1a700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac80b5360 0x7f9ac81a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acea1a700 1 -- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ac818e140 msgr2=0x7f9ac81aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:27.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acea1a700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ac818e140 0x7f9ac81aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:27.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acea1a700 1 -- 172.21.15.32:0/449640876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac8183c80 con 0x7f9ac81ad040 2021-04-09T18:14:27.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ace219700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac80b5360 0x7f9ac81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:14:27.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9acea1a700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac81ad040 0x7f9ac808fca0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f9ab800b500 tx=0x7f9ab800b8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:27.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ac6ffd700 1 -- 172.21.15.32:0/449640876 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ab800bc90 con 0x7f9ac81ad040 2021-04-09T18:14:27.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ac6ffd700 1 -- 172.21.15.32:0/449640876 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab80043a0 con 0x7f9ac81ad040 2021-04-09T18:14:27.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ac6ffd700 1 -- 172.21.15.32:0/449640876 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ab8004880 con 0x7f9ac81ad040 2021-04-09T18:14:27.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac8183c80 con 0x7f9ac81ad040 2021-04-09T18:14:27.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.248 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac8183c80 con 0x7f9ac81ad040 2021-04-09T18:14:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.249 7f9ac6ffd700 1 -- 172.21.15.32:0/449640876 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9ab8001a10 con 0x7f9ac81ad040 2021-04-09T18:14:27.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.249 7f9ac6ffd700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9aa80188c0 0x7f9aa801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:27.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.249 7f9ac6ffd700 1 -- 172.21.15.32:0/449640876 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f9ab80324c0 con 0x7f9ac81ad040 2021-04-09T18:14:27.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.249 7f9ace219700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9aa80188c0 0x7f9aa801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:27.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.250 7f9ace219700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9aa80188c0 0x7f9aa801adc0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.269 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac8182d70 con 0x7f9ac81ad040 2021-04-09T18:14:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.272 7f9ac6ffd700 1 -- 172.21.15.32:0/449640876 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9ab800bc90 con 0x7f9ac81ad040 2021-04-09T18:14:27.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:27.458 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7f9ac8182d70 con 0x7f9ac81ad040 2021-04-09T18:14:28.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.350 7f9ac6ffd700 1 -- 172.21.15.32:0/449640876 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v101) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9ab8001a10 con 0x7f9ac81ad040 2021-04-09T18:14:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9aa80188c0 msgr2=0x7f9aa801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9aa80188c0 0x7f9aa801adc0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac81ad040 msgr2=0x7f9ac808fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:28.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac81ad040 0x7f9ac808fca0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f9ab800b500 tx=0x7f9ab800b8a0).stop 2021-04-09T18:14:28.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 shutdown_connections 2021-04-09T18:14:28.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9aa80188c0 0x7f9aa801adc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ac81ad040 0x7f9ac808fca0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ac818e140 0x7f9ac81aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 --2- 172.21.15.32:0/449640876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ac80b5360 0x7f9ac81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 >> 172.21.15.32:0/449640876 conn(0x7f9ac819b460 msgr2=0x7f9ac819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 shutdown_connections 2021-04-09T18:14:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.377 7f9ad03fd700 1 -- 172.21.15.32:0/449640876 wait complete. 2021-04-09T18:14:28.412 INFO:tasks.cephfs_test_runner:test_down (tasks.cephfs.test_failover.TestClusterResize) ... ok 2021-04-09T18:14:28.413 INFO:tasks.cephfs_test_runner:Starting test: test_down_grow (tasks.cephfs.test_failover.TestClusterResize) 2021-04-09T18:14:28.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2021-04-09T18:14:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.553 7fee18bcb700 1 Processor -- start 2021-04-09T18:14:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.553 7fee18bcb700 1 -- start start 2021-04-09T18:14:28.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee18bcb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee140d6c80 0x7fee140d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:28.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee18bcb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee140d2440 0x7fee1410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:28.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee18bcb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140d7280 con 0x7fee140cffa0 2021-04-09T18:14:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee18bcb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee140d7080 con 0x7fee140d2440 2021-04-09T18:14:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee18bcb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee140d91e0 con 0x7fee140d6c80 2021-04-09T18:14:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee1305f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee140d2440 0x7fee1410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:28.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee1305f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee140d2440 0x7fee1410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43252/0 (socket says 172.21.15.32:43252) 2021-04-09T18:14:28.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee1305f700 1 -- 172.21.15.32:0/3745477859 learned_addr learned my addr 172.21.15.32:0/3745477859 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:28.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee1205d700 1 --2- 172.21.15.32:0/3745477859 >> v2:172.21.15.86:3300/0 conn(0x7fee140d6c80 0x7fee140d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:28.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee1305f700 1 -- 172.21.15.32:0/3745477859 >> v2:172.21.15.86:3300/0 conn(0x7fee140d6c80 msgr2=0x7fee140d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:28.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee1305f700 1 --2- 172.21.15.32:0/3745477859 >> v2:172.21.15.86:3300/0 conn(0x7fee140d6c80 0x7fee140d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee1305f700 1 -- 172.21.15.32:0/3745477859 >> v1:172.21.15.86:6789/0 conn(0x7fee140cffa0 legacy=0x7fee140d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:28.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee1305f700 1 -- 172.21.15.32:0/3745477859 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee140d2840 con 0x7fee140d2440 2021-04-09T18:14:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.554 7fee1305f700 1 --2- 172.21.15.32:0/3745477859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee140d2440 0x7fee1410cfd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fee0c006f00 tx=0x7fee0c009e90).ready entity=mon.0 client_cookie=d9838d8531787ae server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.555 7fee1185c700 1 -- 172.21.15.32:0/3745477859 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee0c00b300 con 0x7fee140d2440 2021-04-09T18:14:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.555 7fee1185c700 1 -- 172.21.15.32:0/3745477859 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c00b640 con 0x7fee140d2440 2021-04-09T18:14:28.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.555 7fee1185c700 1 -- 172.21.15.32:0/3745477859 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee0c00bb20 con 0x7fee140d2440 2021-04-09T18:14:28.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.555 7fee18bcb700 1 -- 172.21.15.32:0/3745477859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee140d2440 msgr2=0x7fee1410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:28.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.555 7fee18bcb700 1 --2- 172.21.15.32:0/3745477859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee140d2440 0x7fee1410cfd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fee0c006f00 tx=0x7fee0c009e90).stop 2021-04-09T18:14:28.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.556 7fee18bcb700 1 -- 172.21.15.32:0/3745477859 shutdown_connections 2021-04-09T18:14:28.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.556 7fee18bcb700 1 --2- 172.21.15.32:0/3745477859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee140d2440 0x7fee1410cfd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.556 7fee18bcb700 1 --2- 172.21.15.32:0/3745477859 >> v2:172.21.15.86:3300/0 conn(0x7fee140d6c80 0x7fee140d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.556 7fee18bcb700 1 -- 172.21.15.32:0/3745477859 >> 172.21.15.32:0/3745477859 conn(0x7fee140ce5c0 msgr2=0x7fee140cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:28.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.556 7fee18bcb700 1 -- 172.21.15.32:0/3745477859 shutdown_connections 2021-04-09T18:14:28.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.556 7fee18bcb700 1 -- 172.21.15.32:0/3745477859 wait complete. 2021-04-09T18:14:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee18bcb700 1 Processor -- start 2021-04-09T18:14:28.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee18bcb700 1 -- start start 2021-04-09T18:14:28.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee18bcb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee14190320 0x7fee1418fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:28.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee18bcb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee1408ab30 0x7fee1408af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:28.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee18bcb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee140aa9b0 0x7fee140aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:28.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee18bcb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1408a650 con 0x7fee14190320 2021-04-09T18:14:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee18bcb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1408b550 con 0x7fee140aa9b0 2021-04-09T18:14:28.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee18bcb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee140ab3d0 con 0x7fee1408ab30 2021-04-09T18:14:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee1285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee14190320 0x7fee1418fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee1285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee14190320 0x7fee1418fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43258/0 (socket says 172.21.15.32:43258) 2021-04-09T18:14:28.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.576 7fee1285e700 1 -- 172.21.15.32:0/289525849 learned_addr learned my addr 172.21.15.32:0/289525849 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee1305f700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee140aa9b0 0x7fee140aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:28.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee1205d700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee1408ab30 0x7fee1408af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:28.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee1285e700 1 -- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee1408ab30 msgr2=0x7fee1408af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:28.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee1285e700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee1408ab30 0x7fee1408af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee1285e700 1 -- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee140aa9b0 msgr2=0x7fee140aadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:28.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee1285e700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee140aa9b0 0x7fee140aadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:28.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee1285e700 1 -- 172.21.15.32:0/289525849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee140ab5f0 con 0x7fee14190320 2021-04-09T18:14:28.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee1305f700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee140aa9b0 0x7fee140aadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:14:28.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee1285e700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee14190320 0x7fee1418fb20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fee0400b5b0 tx=0x7fee0400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:28.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee0b7fe700 1 -- 172.21.15.32:0/289525849 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee0400bd40 con 0x7fee14190320 2021-04-09T18:14:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee0b7fe700 1 -- 172.21.15.32:0/289525849 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee04006240 con 0x7fee14190320 2021-04-09T18:14:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee18bcb700 1 -- 172.21.15.32:0/289525849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee140ab5f0 con 0x7fee14190320 2021-04-09T18:14:28.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee0b7fe700 1 -- 172.21.15.32:0/289525849 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee04005690 con 0x7fee14190320 2021-04-09T18:14:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.577 7fee18bcb700 1 -- 172.21.15.32:0/289525849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee140ab5f0 con 0x7fee14190320 2021-04-09T18:14:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.578 7fee0b7fe700 1 -- 172.21.15.32:0/289525849 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fee040054a0 con 0x7fee14190320 2021-04-09T18:14:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.578 7fee0b7fe700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedf0018930 0x7fedf001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:28.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.578 7fee0b7fe700 1 -- 172.21.15.32:0/289525849 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fee04032750 con 0x7fee14190320 2021-04-09T18:14:28.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.578 7fee1205d700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedf0018930 0x7fedf001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:28.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.578 7fee1205d700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedf0018930 0x7fedf001adc0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:28.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.598 7fee18bcb700 1 -- 172.21.15.32:0/289525849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1408a650 con 0x7fee14190320 2021-04-09T18:14:28.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.602 7fee0b7fe700 1 -- 172.21.15.32:0/289525849 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fee04000c60 con 0x7fee14190320 2021-04-09T18:14:28.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:28.786 7fee18bcb700 1 -- 172.21.15.32:0/289525849 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7fee1408a650 con 0x7fee14190320 2021-04-09T18:14:29.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.352 7fee0b7fe700 1 -- 172.21.15.32:0/289525849 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v102) v1 ==== 141+0+0 (secure 0 0 0) 0x7fee040054a0 con 0x7fee14190320 2021-04-09T18:14:29.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.403 7fee18bcb700 1 -- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedf0018930 msgr2=0x7fedf001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:29.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.403 7fee18bcb700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedf0018930 0x7fedf001adc0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.403 7fee18bcb700 1 -- 172.21.15.32:0/289525849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee14190320 msgr2=0x7fee1418fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:29.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.403 7fee18bcb700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee14190320 0x7fee1418fb20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fee0400b5b0 tx=0x7fee0400b950).stop 2021-04-09T18:14:29.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.403 7fee18bcb700 1 -- 172.21.15.32:0/289525849 shutdown_connections 2021-04-09T18:14:29.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.404 7fee18bcb700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedf0018930 0x7fedf001adc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.404 7fee18bcb700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee140aa9b0 0x7fee140aadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.404 7fee18bcb700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee1408ab30 0x7fee1408af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.404 7fee18bcb700 1 --2- 172.21.15.32:0/289525849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee14190320 0x7fee1418fb20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.404 7fee18bcb700 1 -- 172.21.15.32:0/289525849 >> 172.21.15.32:0/289525849 conn(0x7fee141941b0 msgr2=0x7fee14183120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:29.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.404 7fee18bcb700 1 -- 172.21.15.32:0/289525849 shutdown_connections 2021-04-09T18:14:29.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.404 7fee18bcb700 1 -- 172.21.15.32:0/289525849 wait complete. 2021-04-09T18:14:29.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:14:29.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.649 7f9c3558f700 1 Processor -- start 2021-04-09T18:14:29.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.649 7f9c3558f700 1 -- start start 2021-04-09T18:14:29.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.650 7f9c3558f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c3010d010 0x7f9c300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:29.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.650 7f9c3558f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3010d480 con 0x7f9c300d8490 2021-04-09T18:14:29.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.650 7f9c3558f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3010e1f0 con 0x7f9c300d1da0 2021-04-09T18:14:29.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.650 7f9c3558f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3010e780 con 0x7f9c3010d010 2021-04-09T18:14:29.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.650 7f9c2fac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9c300d8490 0x7f9c3010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52856/0 (socket says 172.21.15.32:52856) 2021-04-09T18:14:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.650 7f9c2fac0700 1 -- 172.21.15.32:0/2379588844 learned_addr learned my addr 172.21.15.32:0/2379588844 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.650 7f9c2f2bf700 1 --2- 172.21.15.32:0/2379588844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c3010d010 0x7f9c300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.650 7f9c2e2bd700 1 -- 172.21.15.32:0/2379588844 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493026814 0 0) 0x7f9c28000d30 con 0x7f9c300d8490 2021-04-09T18:14:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.650 7f9c2e2bd700 1 -- 172.21.15.32:0/2379588844 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c18002fe0 con 0x7f9c300d8490 2021-04-09T18:14:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2e2bd700 1 -- 172.21.15.32:0/2379588844 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871260793 0 0) 0x7f9c24000c60 con 0x7f9c300d1da0 2021-04-09T18:14:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2e2bd700 1 -- 172.21.15.32:0/2379588844 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c18003210 con 0x7f9c300d1da0 2021-04-09T18:14:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2e2bd700 1 -- 172.21.15.32:0/2379588844 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3352480500 0 0) 0x7f9c28001460 con 0x7f9c300d8490 2021-04-09T18:14:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2f2bf700 1 -- 172.21.15.32:0/2379588844 >> v1:172.21.15.86:6789/0 conn(0x7f9c300d1da0 legacy=0x7f9c300d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2f2bf700 1 -- 172.21.15.32:0/2379588844 >> v1:172.21.15.32:6789/0 conn(0x7f9c300d8490 legacy=0x7f9c3010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2f2bf700 1 -- 172.21.15.32:0/2379588844 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c300d8d30 con 0x7f9c3010d010 2021-04-09T18:14:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2f2bf700 1 --2- 172.21.15.32:0/2379588844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c3010d010 0x7f9c300d1780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9c20006650 tx=0x7f9c20009a80).ready entity=mon.2 client_cookie=d522de4773d64248 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2e2bd700 1 -- 172.21.15.32:0/2379588844 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c200009b0 con 0x7f9c3010d010 2021-04-09T18:14:29.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2e2bd700 1 -- 172.21.15.32:0/2379588844 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2000c0c0 con 0x7f9c3010d010 2021-04-09T18:14:29.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c2e2bd700 1 -- 172.21.15.32:0/2379588844 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c2000c510 con 0x7f9c3010d010 2021-04-09T18:14:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c3558f700 1 -- 172.21.15.32:0/2379588844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c3010d010 msgr2=0x7f9c300d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.651 7f9c3558f700 1 --2- 172.21.15.32:0/2379588844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c3010d010 0x7f9c300d1780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9c20006650 tx=0x7f9c20009a80).stop 2021-04-09T18:14:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.652 7f9c3558f700 1 -- 172.21.15.32:0/2379588844 shutdown_connections 2021-04-09T18:14:29.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.652 7f9c3558f700 1 --2- 172.21.15.32:0/2379588844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c3010d010 0x7f9c300d1780 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.652 7f9c3558f700 1 -- 172.21.15.32:0/2379588844 >> 172.21.15.32:0/2379588844 conn(0x7f9c300ce5f0 msgr2=0x7f9c300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.652 7f9c3558f700 1 -- 172.21.15.32:0/2379588844 shutdown_connections 2021-04-09T18:14:29.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.652 7f9c3558f700 1 -- 172.21.15.32:0/2379588844 wait complete. 2021-04-09T18:14:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.672 7f9c3558f700 1 Processor -- start 2021-04-09T18:14:29.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.672 7f9c3558f700 1 -- start start 2021-04-09T18:14:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c3558f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c30184480 0x7f9c300befe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c3558f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c3019dd60 0x7f9c3019e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c3558f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c300c2ff0 0x7f9c300c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c3558f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c300ada20 con 0x7f9c30184480 2021-04-09T18:14:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c3558f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c300c2a70 con 0x7f9c3019dd60 2021-04-09T18:14:29.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c3558f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30184880 con 0x7f9c300c2ff0 2021-04-09T18:14:29.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c30184480 0x7f9c300befe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c30184480 0x7f9c300befe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43274/0 (socket says 172.21.15.32:43274) 2021-04-09T18:14:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2f2bf700 1 -- 172.21.15.32:0/252295158 learned_addr learned my addr 172.21.15.32:0/252295158 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2eabe700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c3019dd60 0x7f9c3019e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:29.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2fac0700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c300c2ff0 0x7f9c300c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2f2bf700 1 -- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c300c2ff0 msgr2=0x7f9c300c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2f2bf700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c300c2ff0 0x7f9c300c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2f2bf700 1 -- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c3019dd60 msgr2=0x7f9c3019e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2f2bf700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c3019dd60 0x7f9c3019e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2f2bf700 1 -- 172.21.15.32:0/252295158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c300c18c0 con 0x7f9c30184480 2021-04-09T18:14:29.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2fac0700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c300c2ff0 0x7f9c300c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:29.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c2f2bf700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c30184480 0x7f9c300befe0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9c20009650 tx=0x7f9c2000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:29.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c17fff700 1 -- 172.21.15.32:0/252295158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c2000bd50 con 0x7f9c30184480 2021-04-09T18:14:29.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c17fff700 1 -- 172.21.15.32:0/252295158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c20006030 con 0x7f9c30184480 2021-04-09T18:14:29.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c17fff700 1 -- 172.21.15.32:0/252295158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c20005560 con 0x7f9c30184480 2021-04-09T18:14:29.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c3558f700 1 -- 172.21.15.32:0/252295158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c300c18c0 con 0x7f9c30184480 2021-04-09T18:14:29.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.673 7f9c3558f700 1 -- 172.21.15.32:0/252295158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c300c18c0 con 0x7f9c30184480 2021-04-09T18:14:29.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.674 7f9c17fff700 1 -- 172.21.15.32:0/252295158 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9c20005700 con 0x7f9c30184480 2021-04-09T18:14:29.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.674 7f9c17fff700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c10018930 0x7f9c1001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:29.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.674 7f9c17fff700 1 -- 172.21.15.32:0/252295158 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f9c20032720 con 0x7f9c30184480 2021-04-09T18:14:29.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.674 7f9c2eabe700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c10018930 0x7f9c1001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:29.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.675 7f9c2eabe700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c10018930 0x7f9c1001ae10 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:29.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.694 7f9c3558f700 1 -- 172.21.15.32:0/252295158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c30093d50 con 0x7f9c30184480 2021-04-09T18:14:29.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.698 7f9c17fff700 1 -- 172.21.15.32:0/252295158 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9c200054f0 con 0x7f9c30184480 2021-04-09T18:14:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.909 7f9c3558f700 1 -- 172.21.15.32:0/252295158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9c3010d480 con 0x7f9c30184480 2021-04-09T18:14:29.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.912 7f9c17fff700 1 -- 172.21.15.32:0/252295158 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9c20005700 con 0x7f9c30184480 2021-04-09T18:14:29.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 -- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c10018930 msgr2=0x7f9c1001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c10018930 0x7f9c1001ae10 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 -- 172.21.15.32:0/252295158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c30184480 msgr2=0x7f9c300befe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c30184480 0x7f9c300befe0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9c20009650 tx=0x7f9c2000b960).stop 2021-04-09T18:14:29.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 -- 172.21.15.32:0/252295158 shutdown_connections 2021-04-09T18:14:29.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c10018930 0x7f9c1001ae10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c300c2ff0 0x7f9c300c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c3019dd60 0x7f9c3019e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 --2- 172.21.15.32:0/252295158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c30184480 0x7f9c300befe0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:29.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.918 7f9c3558f700 1 -- 172.21.15.32:0/252295158 >> 172.21.15.32:0/252295158 conn(0x7f9c300a0b50 msgr2=0x7f9c300ac820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:29.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.919 7f9c3558f700 1 -- 172.21.15.32:0/252295158 shutdown_connections 2021-04-09T18:14:29.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:29.919 7f9c3558f700 1 -- 172.21.15.32:0/252295158 wait complete. 2021-04-09T18:14:29.955 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:14:29.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:14:30.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:14:30.165 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:14:30.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:30.167 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:14:30.179 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:14:30.232 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:14:30.233 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:14:30.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:30.235 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:14:30.236 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:14:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:14:30.306 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:14:30.308 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:14:30.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:14:30.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.443 7f735a1cc700 1 Processor -- start 2021-04-09T18:14:30.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.443 7f735a1cc700 1 -- start start 2021-04-09T18:14:30.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f735a1cc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735410d010 0x7f73540d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:30.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f735a1cc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f73540d1da0 0x7f73540d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:30.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f735a1cc700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735410d480 con 0x7f73540d82c0 2021-04-09T18:14:30.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f735a1cc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735410e780 con 0x7f73540d1da0 2021-04-09T18:14:30.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f735a1cc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73540d8d30 con 0x7f735410d010 2021-04-09T18:14:30.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f7352ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f73540d1da0 0x7f73540d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:30.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f7352ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f73540d1da0 0x7f73540d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43282/0 (socket says 172.21.15.32:43282) 2021-04-09T18:14:30.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f7352ffd700 1 -- 172.21.15.32:0/361734026 learned_addr learned my addr 172.21.15.32:0/361734026 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:30.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f73537fe700 1 --2- 172.21.15.32:0/361734026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735410d010 0x7f73540d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:30.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f7352ffd700 1 -- 172.21.15.32:0/361734026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735410d010 msgr2=0x7f73540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:30.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f7352ffd700 1 --2- 172.21.15.32:0/361734026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735410d010 0x7f73540d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f7352ffd700 1 -- 172.21.15.32:0/361734026 >> v1:172.21.15.86:6789/0 conn(0x7f73540d82c0 legacy=0x7f735410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f73527fc700 1 -- 172.21.15.32:0/361734026 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321766490 0 0) 0x7f734c000d30 con 0x7f73540d82c0 2021-04-09T18:14:30.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f7352ffd700 1 -- 172.21.15.32:0/361734026 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73540d9220 con 0x7f73540d1da0 2021-04-09T18:14:30.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f7352ffd700 1 --2- 172.21.15.32:0/361734026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73540d1da0 0x7f73540d7ca0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f7348006700 tx=0x7f7348009b80).ready entity=mon.0 client_cookie=8cab8085ec5a1492 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:30.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.444 7f73527fc700 1 -- 172.21.15.32:0/361734026 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f734800c0c0 con 0x7f73540d1da0 2021-04-09T18:14:30.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f73527fc700 1 -- 172.21.15.32:0/361734026 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734800c440 con 0x7f73540d1da0 2021-04-09T18:14:30.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f73527fc700 1 -- 172.21.15.32:0/361734026 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f734800c920 con 0x7f73540d1da0 2021-04-09T18:14:30.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f735a1cc700 1 -- 172.21.15.32:0/361734026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73540d1da0 msgr2=0x7f73540d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:30.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f735a1cc700 1 --2- 172.21.15.32:0/361734026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73540d1da0 0x7f73540d7ca0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f7348006700 tx=0x7f7348009b80).stop 2021-04-09T18:14:30.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f735a1cc700 1 -- 172.21.15.32:0/361734026 shutdown_connections 2021-04-09T18:14:30.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f735a1cc700 1 --2- 172.21.15.32:0/361734026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73540d1da0 0x7f73540d7ca0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:30.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f735a1cc700 1 --2- 172.21.15.32:0/361734026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735410d010 0x7f73540d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:30.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f735a1cc700 1 -- 172.21.15.32:0/361734026 >> 172.21.15.32:0/361734026 conn(0x7f73540ce5f0 msgr2=0x7f73540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:30.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f735a1cc700 1 -- 172.21.15.32:0/361734026 shutdown_connections 2021-04-09T18:14:30.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.445 7f735a1cc700 1 -- 172.21.15.32:0/361734026 wait complete. 2021-04-09T18:14:30.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.466 7f735a1cc700 1 Processor -- start 2021-04-09T18:14:30.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.466 7f735a1cc700 1 -- start start 2021-04-09T18:14:30.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.466 7f735a1cc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541a5210 0x7f7354185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:30.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.466 7f735a1cc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7354185a70 0x7f73540acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:30.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.466 7f735a1cc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7354185e70 0x7f73540ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:30.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.466 7f735a1cc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735418f7b0 con 0x7f7354185e70 2021-04-09T18:14:30.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.466 7f735a1cc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73540ad7f0 con 0x7f7354185a70 2021-04-09T18:14:30.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.466 7f735a1cc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73540ada10 con 0x7f73541a5210 2021-04-09T18:14:30.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7353fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7354185e70 0x7f73540ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:30.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7353fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7354185e70 0x7f73540ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43284/0 (socket says 172.21.15.32:43284) 2021-04-09T18:14:30.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7353fff700 1 -- 172.21.15.32:0/1093136189 learned_addr learned my addr 172.21.15.32:0/1093136189 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:30.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f73537fe700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541a5210 0x7f7354185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:30.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7352ffd700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7354185a70 0x7f73540acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:30.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7353fff700 1 -- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541a5210 msgr2=0x7f7354185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:30.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7353fff700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541a5210 0x7f7354185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:30.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7353fff700 1 -- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7354185a70 msgr2=0x7f73540acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:30.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7353fff700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7354185a70 0x7f73540acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:30.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7353fff700 1 -- 172.21.15.32:0/1093136189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7354179610 con 0x7f7354185e70 2021-04-09T18:14:30.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f73537fe700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541a5210 0x7f7354185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:30.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f7353fff700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7354185e70 0x7f73540ad1d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f734c009980 tx=0x7f734c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:30.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f733bfff700 1 -- 172.21.15.32:0/1093136189 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f734c005870 con 0x7f7354185e70 2021-04-09T18:14:30.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f733bfff700 1 -- 172.21.15.32:0/1093136189 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c005b70 con 0x7f7354185e70 2021-04-09T18:14:30.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7354179610 con 0x7f7354185e70 2021-04-09T18:14:30.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f733bfff700 1 -- 172.21.15.32:0/1093136189 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f734c006030 con 0x7f7354185e70 2021-04-09T18:14:30.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.467 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7354179610 con 0x7f7354185e70 2021-04-09T18:14:30.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.468 7f733bfff700 1 -- 172.21.15.32:0/1093136189 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f734c006030 con 0x7f7354185e70 2021-04-09T18:14:30.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.468 7f733bfff700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7334018930 0x7f733401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:30.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.468 7f733bfff700 1 -- 172.21.15.32:0/1093136189 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f734c032b00 con 0x7f7354185e70 2021-04-09T18:14:30.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.468 7f73537fe700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7334018930 0x7f733401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:30.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.468 7f73537fe700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7334018930 0x7f733401ae10 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:30.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.488 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735418f7b0 con 0x7f7354185e70 2021-04-09T18:14:30.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.491 7f733bfff700 1 -- 172.21.15.32:0/1093136189 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f734c00be10 con 0x7f7354185e70 2021-04-09T18:14:30.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.679 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f735418f7b0 con 0x7f7354185e70 2021-04-09T18:14:30.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.680 7f733bfff700 1 -- 172.21.15.32:0/1093136189 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v35) v1 ==== 81+0+25540 (secure 0 0 0) 0x7f734c005b70 con 0x7f7354185e70 2021-04-09T18:14:30.737 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:30.737 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:14:30.738 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 35, 2021-04-09T18:14:30.738 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:14:30.738 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:14:30.739 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:14:26.338235", 2021-04-09T18:14:30.760 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:14:30.761 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:14:30.761 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:14:30.761 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:14:30.762 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:14:30.767 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:14:30.770 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:14:30.801 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:14:30.819 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:14:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:14:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:14:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:14:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:14:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:14:30.821 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:14:30.821 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 5, 2021-04-09T18:14:30.830 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:14:30.830 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:14:30.830 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:14:30.830 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:14:30.831 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:14:30.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:30.831 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:14:30.831 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:14:30.832 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:14:30.837 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:14:30.837 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:14:30.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:14:30.838 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:14:30.838 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:14:30.839 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:14:30.839 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:14:30.839 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:14:30.844 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:14:30.844 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:14:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:14:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:14:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:14:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:14:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:14:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:14:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:14:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:14:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:14:30.847 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:14:30.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:30.847 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:14:30.847 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:14:30.848 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:14:30.848 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:14:30.848 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:14:30.848 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:14:30.859 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:14:30.860 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:14:30.860 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:14:30.860 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:14:30.863 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:14:30.863 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:14:30.870 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:14:30.870 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:14:30.875 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:14:30.875 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:14:30.875 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:14:30.875 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:14:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:14:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:14:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:14:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:14:30.952 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:14:30.952 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:14:30.953 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:14:30.953 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:14:30.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:14:30.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:30.954 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:14:30.955 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:14:30.955 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:14:30.956 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:14:30.956 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:14:30.956 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:14:30.957 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:14:30.989 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:14:30.989 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:14:30.990 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:14:30.990 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:14:30.990 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:14:30.990 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:14:30.990 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:14:30.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:30.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:30.991 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:14:31.015 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:14:31.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.017 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:14:31.018 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:14:31.018 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:31.018 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:31.018 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:31.018 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:31.019 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:31.019 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:31.019 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:31.019 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:31.020 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:31.031 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:31.035 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:31.037 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.038 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:14:31.038 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:31.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.039 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.039 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:14:31.040 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:31.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.050 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.051 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:31.056 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.057 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.057 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:14:31.057 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:31.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.065 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:14:31.065 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:31.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.081 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.081 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:31.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.095 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:14:31.155 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:31.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.161 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.163 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:14:31.164 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:31.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.165 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.176 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:31.176 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.206 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:14:31.206 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:31.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.207 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.216 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:14:31.216 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:14:31.216 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.216 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.217 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:14:31.229 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:14:31.243 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:14:31.243 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:14:31.244 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:31.244 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:31.244 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:31.244 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.245 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:14:31.245 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:31.246 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:31.247 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:31.258 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:31.258 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:31.258 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 29, 2021-04-09T18:14:31.258 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:31.259 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:31.259 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:31.259 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.264 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:14:31.264 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:31.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.267 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.267 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:14:31.267 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:31.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.267 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.271 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:31.272 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.272 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.279 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:14:31.279 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:31.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.280 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.280 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.280 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:14:31.280 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:31.281 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.281 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.281 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:31.282 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.282 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:14:31.283 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:31.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.285 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.286 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:14:31.286 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:31.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.287 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.288 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:31.297 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.302 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:14:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.304 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:14:31.310 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:14:31.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.310 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:14:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:14:31.311 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:14:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:14:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:31.312 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:14:31.313 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:14:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:31.314 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:31.385 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:31.392 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.393 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.398 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:14:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.399 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:14:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.400 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:14:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.403 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:14:31.429 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:31.430 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.434 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.436 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:31.437 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.443 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.443 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:14:31.444 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:31.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:14:31.446 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:31.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.459 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.464 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:31.464 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.469 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.475 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:14:31.475 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:31.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.477 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.477 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:14:31.478 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:14:31.478 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.478 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.489 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:14:31.489 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:14:31.490 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:14:31.490 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:14:31.490 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:31.490 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:31.490 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:31.496 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.497 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:14:31.498 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:14:31.498 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:31.498 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:31.499 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:31.559 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:31.559 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:31.573 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:31.583 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:31.584 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:31.585 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:31.585 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:31.586 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:31.591 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.616 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:14:31.617 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:31.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.618 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:14:31.619 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:31.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.636 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.639 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:31.639 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:14:31.641 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:31.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.710 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:14:31.710 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:31.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.716 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.717 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:31.717 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.718 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:14:31.718 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:31.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.737 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:14:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:31.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.741 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.744 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.757 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:14:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:31.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:14:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:14:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.821 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:14:31.821 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:14:31.822 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:14:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:14:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:31.873 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:14:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:14:31.882 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:31.883 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:31.895 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:31.896 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:31.990 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:31.992 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:14:31.993 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:32.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.006 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:14:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.009 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:32.059 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:14:32.075 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:32.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.120 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:14:32.120 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:32.121 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.122 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.123 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:32.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:14:32.125 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:32.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.186 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.186 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:14:32.187 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:32.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.188 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.188 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:32.189 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.189 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:14:32.190 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:32.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.190 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.191 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:14:32.215 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:14:32.215 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.244 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:14:32.244 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:14:32.244 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:14:32.245 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:14:32.245 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:32.245 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:32.262 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:32.263 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.277 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:14:32.287 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:14:32.288 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:32.291 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:32.292 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:32.292 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:32.292 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:32.293 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:32.293 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:32.294 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:32.294 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:32.294 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:32.311 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:32.311 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.334 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.335 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:14:32.335 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:32.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.349 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.356 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:14:32.361 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:32.367 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.368 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:32.388 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.416 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.416 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:14:32.416 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:32.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.455 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:14:32.455 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:32.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.456 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.457 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:32.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.458 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.458 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:14:32.459 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:32.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.483 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:14:32.484 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:32.488 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:14:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:14:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.512 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.513 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:14:32.542 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:14:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:14:32.543 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:14:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:14:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:14:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:32.557 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:32.558 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:32.559 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:32.590 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:32.591 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:32.591 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.608 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:14:32.608 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:32.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.661 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.661 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:14:32.661 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:32.661 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.661 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.663 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:14:32.663 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:32.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.683 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:14:32.683 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:32.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.684 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.684 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:32.685 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.685 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.685 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:14:32.685 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:32.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.686 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.687 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:14:32.687 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:32.688 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.689 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.689 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:32.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.693 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:14:32.693 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:32.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.693 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.694 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:14:32.694 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:14:32.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:14:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:14:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:14:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:14:32.696 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:32.696 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:32.696 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:32.696 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.703 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:14:32.703 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:14:32.704 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:14:32.704 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:14:32.704 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:14:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:14:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:14:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:14:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:14:32.706 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:14:32.706 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:14:32.706 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:14:32.706 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:14:32.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:14:32.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:32.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.711 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:14:32.711 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:32.812 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.813 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.813 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:14:32.814 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:14:32.816 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:32.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:14:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:14:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:14:32.981 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:32.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:32.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:32.982 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:14:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:32.984 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:32.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:32.985 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:14:32.985 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:14:32.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:14:33.013 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:14:33.038 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:33.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.043 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:33.043 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:14:33.050 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:14:33.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:33.051 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:33.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:14:33.052 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:14:33.053 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:14:33.054 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:14:33.055 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:14:33.056 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:14:33.056 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:14:33.057 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:33.057 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:33.057 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:14:33.058 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:14:33.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.059 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:14:33.059 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:33.060 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:33.061 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:33.061 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:33.062 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:33.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.063 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:14:33.064 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:33.064 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:33.065 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:33.066 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:33.067 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:33.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.068 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:14:33.069 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:33.069 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:33.080 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:33.084 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:33.085 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:33.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.088 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:14:33.088 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:33.089 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:33.089 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:33.089 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:33.090 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:33.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.093 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:14:33.107 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:33.108 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:33.110 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:33.111 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:33.111 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:33.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.112 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:14:33.112 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:33.113 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:33.114 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:33.115 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:33.115 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:33.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.116 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:14:33.117 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:33.118 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:33.119 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:33.119 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:33.120 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:33.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:14:33.123 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:14:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:14:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:14:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:14:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:14:33.125 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:14:33.126 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:14:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:14:33.127 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:14:33.134 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:14:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:14:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/2659274405": "2021-04-10 18:14:21.711645", 2021-04-09T18:14:33.135 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/2659274405": "2021-04-10 18:14:21.711645" 2021-04-09T18:14:33.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.136 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:14:33.136 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:14:33.137 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:14:33.137 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:14:33.167 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:14:33.167 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:14:33.176 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:14:33.177 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:14:33.178 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:33.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:33.180 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:14:33.180 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:14:33.180 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:14:33.185 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:14:33.185 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:14:33.185 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:14:33.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7334018930 msgr2=0x7f733401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7334018930 0x7f733401ae10 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7354185e70 msgr2=0x7f73540ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7354185e70 0x7f73540ad1d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f734c009980 tx=0x7f734c00bcd0).stop 2021-04-09T18:14:33.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 shutdown_connections 2021-04-09T18:14:33.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7334018930 0x7f733401ae10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7354185e70 0x7f73540ad1d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7354185a70 0x7f73540acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 --2- 172.21.15.32:0/1093136189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73541a5210 0x7f7354185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 >> 172.21.15.32:0/1093136189 conn(0x7f73541a99c0 msgr2=0x7f735417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:33.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 shutdown_connections 2021-04-09T18:14:33.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:30.685 7f735a1cc700 1 -- 172.21.15.32:0/1093136189 wait complete. 2021-04-09T18:14:33.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:33.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.337 7f6627ae3700 1 Processor -- start 2021-04-09T18:14:33.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.338 7f6627ae3700 1 -- start start 2021-04-09T18:14:33.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.338 7f6627ae3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f662010d010 0x7f66200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.338 7f6627ae3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66200d1da0 0x7f66200d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.338 7f6627ae3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662010d480 con 0x7f66200d82c0 2021-04-09T18:14:33.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.338 7f6627ae3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66200d9220 con 0x7f662010d010 2021-04-09T18:14:33.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.338 7f6627ae3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66200d86c0 con 0x7f66200d1da0 2021-04-09T18:14:33.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.338 7f6626100700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f66200d82c0 0x7f662010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52882/0 (socket says 172.21.15.32:52882) 2021-04-09T18:14:33.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f6626100700 1 -- 172.21.15.32:0/1743339690 learned_addr learned my addr 172.21.15.32:0/1743339690 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:33.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66258ff700 1 --2- 172.21.15.32:0/1743339690 >> v2:172.21.15.86:3300/0 conn(0x7f662010d010 0x7f66200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66250fe700 1 --2- 172.21.15.32:0/1743339690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66200d1da0 0x7f66200d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66248fd700 1 -- 172.21.15.32:0/1743339690 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789591623 0 0) 0x7f6618000d30 con 0x7f66200d82c0 2021-04-09T18:14:33.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66248fd700 1 -- 172.21.15.32:0/1743339690 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f660c002fe0 con 0x7f66200d82c0 2021-04-09T18:14:33.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66258ff700 1 -- 172.21.15.32:0/1743339690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66200d1da0 msgr2=0x7f66200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:33.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66258ff700 1 --2- 172.21.15.32:0/1743339690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66200d1da0 0x7f66200d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66258ff700 1 -- 172.21.15.32:0/1743339690 >> v1:172.21.15.32:6789/0 conn(0x7f66200d82c0 legacy=0x7f662010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66258ff700 1 -- 172.21.15.32:0/1743339690 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662010e1f0 con 0x7f662010d010 2021-04-09T18:14:33.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66250fe700 1 --2- 172.21.15.32:0/1743339690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66200d1da0 0x7f66200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:33.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66258ff700 1 --2- 172.21.15.32:0/1743339690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f662010d010 0x7f66200d1780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f661c006650 tx=0x7f661c009a80).ready entity=mon.1 client_cookie=1cf3c4159bce4fb6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:33.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66248fd700 1 -- 172.21.15.32:0/1743339690 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661c0009b0 con 0x7f662010d010 2021-04-09T18:14:33.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66248fd700 1 -- 172.21.15.32:0/1743339690 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c00c0c0 con 0x7f662010d010 2021-04-09T18:14:33.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f66248fd700 1 -- 172.21.15.32:0/1743339690 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661c00c510 con 0x7f662010d010 2021-04-09T18:14:33.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f6627ae3700 1 -- 172.21.15.32:0/1743339690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f662010d010 msgr2=0x7f66200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.339 7f6627ae3700 1 --2- 172.21.15.32:0/1743339690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f662010d010 0x7f66200d1780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f661c006650 tx=0x7f661c009a80).stop 2021-04-09T18:14:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.340 7f6627ae3700 1 -- 172.21.15.32:0/1743339690 shutdown_connections 2021-04-09T18:14:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.340 7f6627ae3700 1 --2- 172.21.15.32:0/1743339690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66200d1da0 0x7f66200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.340 7f6627ae3700 1 --2- 172.21.15.32:0/1743339690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f662010d010 0x7f66200d1780 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.340 7f6627ae3700 1 -- 172.21.15.32:0/1743339690 >> 172.21.15.32:0/1743339690 conn(0x7f66200ce5f0 msgr2=0x7f66200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:33.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.340 7f6627ae3700 1 -- 172.21.15.32:0/1743339690 shutdown_connections 2021-04-09T18:14:33.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.340 7f6627ae3700 1 -- 172.21.15.32:0/1743339690 wait complete. 2021-04-09T18:14:33.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.360 7f6627ae3700 1 Processor -- start 2021-04-09T18:14:33.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f6627ae3700 1 -- start start 2021-04-09T18:14:33.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f6627ae3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620179bf0 0x7f6620179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f6627ae3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017c540 0x7f662017c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f6627ae3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66200aca60 0x7f66200ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f6627ae3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662017cf60 con 0x7f6620179bf0 2021-04-09T18:14:33.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f6627ae3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66200ad480 con 0x7f66200aca60 2021-04-09T18:14:33.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f6627ae3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66200ad6a0 con 0x7f662017c540 2021-04-09T18:14:33.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f66258ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620179bf0 0x7f6620179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f66258ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620179bf0 0x7f6620179ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43294/0 (socket says 172.21.15.32:43294) 2021-04-09T18:14:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f66258ff700 1 -- 172.21.15.32:0/3439809513 learned_addr learned my addr 172.21.15.32:0/3439809513 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f6626100700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66200aca60 0x7f66200ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.361 7f66250fe700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017c540 0x7f662017c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66258ff700 1 -- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017c540 msgr2=0x7f662017c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66258ff700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017c540 0x7f662017c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66258ff700 1 -- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66200aca60 msgr2=0x7f66200ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66258ff700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66200aca60 0x7f66200ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66258ff700 1 -- 172.21.15.32:0/3439809513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66200ad8c0 con 0x7f6620179bf0 2021-04-09T18:14:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66258ff700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620179bf0 0x7f6620179ff0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f661c009630 tx=0x7f661c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66127fc700 1 -- 172.21.15.32:0/3439809513 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661c0093a0 con 0x7f6620179bf0 2021-04-09T18:14:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66127fc700 1 -- 172.21.15.32:0/3439809513 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c00bcf0 con 0x7f6620179bf0 2021-04-09T18:14:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66127fc700 1 -- 172.21.15.32:0/3439809513 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661c0054d0 con 0x7f6620179bf0 2021-04-09T18:14:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66200ad8c0 con 0x7f6620179bf0 2021-04-09T18:14:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66200ad8c0 con 0x7f6620179bf0 2021-04-09T18:14:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.362 7f66127fc700 1 -- 172.21.15.32:0/3439809513 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f661c0054f0 con 0x7f6620179bf0 2021-04-09T18:14:33.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.363 7f66127fc700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66080188d0 0x7f660801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.363 7f66127fc700 1 -- 172.21.15.32:0/3439809513 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f661c032730 con 0x7f6620179bf0 2021-04-09T18:14:33.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.363 7f66250fe700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66080188d0 0x7f660801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.363 7f66250fe700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66080188d0 0x7f660801add0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.382 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66201b7010 con 0x7f6620179bf0 2021-04-09T18:14:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.386 7f66127fc700 1 -- 172.21.15.32:0/3439809513 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f661c0052e0 con 0x7f6620179bf0 2021-04-09T18:14:33.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.578 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f662010d480 con 0x7f6620179bf0 2021-04-09T18:14:33.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.578 7f66127fc700 1 -- 172.21.15.32:0/3439809513 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+1849 (secure 0 0 0) 0x7f661c0054f0 con 0x7f6620179bf0 2021-04-09T18:14:33.603 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2021-04-09T18:14:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:33.604 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4960,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/425942817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":425942817},{"type":"v1","addr":"172.21.15.86:6835","nonce":425942817}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":20},{"gid":5061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3129768501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3129768501},{"type":"v1","addr":"172.21.15.32:6833","nonce":3129768501}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":23},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[]} 2021-04-09T18:14:33.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66080188d0 msgr2=0x7f660801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66080188d0 0x7f660801add0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620179bf0 msgr2=0x7f6620179ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620179bf0 0x7f6620179ff0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f661c009630 tx=0x7f661c00b900).stop 2021-04-09T18:14:33.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 shutdown_connections 2021-04-09T18:14:33.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66080188d0 0x7f660801add0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66200aca60 0x7f66200ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017c540 0x7f662017c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 --2- 172.21.15.32:0/3439809513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620179bf0 0x7f6620179ff0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 >> 172.21.15.32:0/3439809513 conn(0x7f66201acae0 msgr2=0x7f662017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:33.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 shutdown_connections 2021-04-09T18:14:33.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.581 7f6627ae3700 1 -- 172.21.15.32:0/3439809513 wait complete. 2021-04-09T18:14:33.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.751 7f19907d8700 1 Processor -- start 2021-04-09T18:14:33.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.751 7f19907d8700 1 -- start start 2021-04-09T18:14:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.751 7f19907d8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19880d7ca0 0x7f19880d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.751 7f19907d8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f198810da00 0x7f198810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.751 7f19907d8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f198810d480 con 0x7f198810d010 2021-04-09T18:14:33.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.751 7f19907d8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19880d9220 con 0x7f19880d7ca0 2021-04-09T18:14:33.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.751 7f19907d8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19880d86c0 con 0x7f198810da00 2021-04-09T18:14:33.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198ddf3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19880d7ca0 0x7f19880d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198ddf3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19880d7ca0 0x7f19880d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43306/0 (socket says 172.21.15.32:43306) 2021-04-09T18:14:33.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198ddf3700 1 -- 172.21.15.32:0/4146392760 learned_addr learned my addr 172.21.15.32:0/4146392760 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:33.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198e5f4700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f198810d010 0x7f19880d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:50818/0 (socket says 172.21.15.32:50818) 2021-04-09T18:14:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198edf5700 1 --2- 172.21.15.32:0/4146392760 >> v2:172.21.15.86:3300/0 conn(0x7f198810da00 0x7f198810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198ddf3700 1 -- 172.21.15.32:0/4146392760 >> v2:172.21.15.86:3300/0 conn(0x7f198810da00 msgr2=0x7f198810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198ddf3700 1 --2- 172.21.15.32:0/4146392760 >> v2:172.21.15.86:3300/0 conn(0x7f198810da00 0x7f198810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198ddf3700 1 -- 172.21.15.32:0/4146392760 >> v1:172.21.15.86:6789/0 conn(0x7f198810d010 legacy=0x7f19880d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198ddf3700 1 -- 172.21.15.32:0/4146392760 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19880d1f70 con 0x7f19880d7ca0 2021-04-09T18:14:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198d5f2700 1 -- 172.21.15.32:0/4146392760 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487900100 0 0) 0x7f1984000c60 con 0x7f198810d010 2021-04-09T18:14:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198ddf3700 1 --2- 172.21.15.32:0/4146392760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19880d7ca0 0x7f19880d80a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f197c006700 tx=0x7f197c009b80).ready entity=mon.0 client_cookie=7de673d8d1e7511d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198d5f2700 1 -- 172.21.15.32:0/4146392760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f197c00c0c0 con 0x7f19880d7ca0 2021-04-09T18:14:33.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198d5f2700 1 -- 172.21.15.32:0/4146392760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197c00c440 con 0x7f19880d7ca0 2021-04-09T18:14:33.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f198d5f2700 1 -- 172.21.15.32:0/4146392760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f197c00c920 con 0x7f19880d7ca0 2021-04-09T18:14:33.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f19907d8700 1 -- 172.21.15.32:0/4146392760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19880d7ca0 msgr2=0x7f19880d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.752 7f19907d8700 1 --2- 172.21.15.32:0/4146392760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19880d7ca0 0x7f19880d80a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f197c006700 tx=0x7f197c009b80).stop 2021-04-09T18:14:33.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.753 7f19907d8700 1 -- 172.21.15.32:0/4146392760 shutdown_connections 2021-04-09T18:14:33.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.753 7f19907d8700 1 --2- 172.21.15.32:0/4146392760 >> v2:172.21.15.86:3300/0 conn(0x7f198810da00 0x7f198810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.753 7f19907d8700 1 --2- 172.21.15.32:0/4146392760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19880d7ca0 0x7f19880d80a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.753 7f19907d8700 1 -- 172.21.15.32:0/4146392760 >> 172.21.15.32:0/4146392760 conn(0x7f19880ce5f0 msgr2=0x7f19880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:33.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.753 7f19907d8700 1 -- 172.21.15.32:0/4146392760 shutdown_connections 2021-04-09T18:14:33.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.753 7f19907d8700 1 -- 172.21.15.32:0/4146392760 wait complete. 2021-04-09T18:14:33.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.773 7f19907d8700 1 Processor -- start 2021-04-09T18:14:33.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.773 7f19907d8700 1 -- start start 2021-04-09T18:14:33.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f19907d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19881aca20 0x7f19881ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f19907d8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198818bd40 0x7f198818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f19907d8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198808fba0 0x7f198808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f19907d8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198818c7e0 con 0x7f19881aca20 2021-04-09T18:14:33.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f19907d8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198818ca30 con 0x7f198808fba0 2021-04-09T18:14:33.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f19907d8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19880905f0 con 0x7f198818bd40 2021-04-09T18:14:33.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198e5f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19881aca20 0x7f19881ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198e5f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19881aca20 0x7f19881ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43308/0 (socket says 172.21.15.32:43308) 2021-04-09T18:14:33.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198e5f4700 1 -- 172.21.15.32:0/118744123 learned_addr learned my addr 172.21.15.32:0/118744123 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198edf5700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198808fba0 0x7f198808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198ddf3700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198818bd40 0x7f198818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198e5f4700 1 -- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198818bd40 msgr2=0x7f198818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198e5f4700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198818bd40 0x7f198818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198e5f4700 1 -- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198808fba0 msgr2=0x7f198808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198e5f4700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198808fba0 0x7f198808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:33.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198e5f4700 1 -- 172.21.15.32:0/118744123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19880b4300 con 0x7f19881aca20 2021-04-09T18:14:33.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.774 7f198edf5700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198808fba0 0x7f198808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:33.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.775 7f198e5f4700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19881aca20 0x7f19881ace20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f198400a4f0 tx=0x7f198400a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:33.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.775 7f197affd700 1 -- 172.21.15.32:0/118744123 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198400ac80 con 0x7f19881aca20 2021-04-09T18:14:33.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.775 7f197affd700 1 -- 172.21.15.32:0/118744123 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19840070d0 con 0x7f19881aca20 2021-04-09T18:14:33.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.775 7f197affd700 1 -- 172.21.15.32:0/118744123 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19840047b0 con 0x7f19881aca20 2021-04-09T18:14:33.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.775 7f19907d8700 1 -- 172.21.15.32:0/118744123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19880b4300 con 0x7f19881aca20 2021-04-09T18:14:33.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.775 7f19907d8700 1 -- 172.21.15.32:0/118744123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19880b4300 con 0x7f19881aca20 2021-04-09T18:14:33.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.775 7f197affd700 1 -- 172.21.15.32:0/118744123 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1984000c10 con 0x7f19881aca20 2021-04-09T18:14:33.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.776 7f197affd700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19700188d0 0x7f197001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:33.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.776 7f197affd700 1 -- 172.21.15.32:0/118744123 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f1984032720 con 0x7f19881aca20 2021-04-09T18:14:33.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.776 7f198ddf3700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19700188d0 0x7f197001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:33.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.776 7f198ddf3700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19700188d0 0x7f197001add0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:33.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.795 7f19907d8700 1 -- 172.21.15.32:0/118744123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1988182d70 con 0x7f19881aca20 2021-04-09T18:14:33.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.799 7f197affd700 1 -- 172.21.15.32:0/118744123 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f19840070d0 con 0x7f19881aca20 2021-04-09T18:14:34.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.991 7f19907d8700 1 -- 172.21.15.32:0/118744123 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f198818c760 con 0x7f19881aca20 2021-04-09T18:14:34.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.991 7f197affd700 1 -- 172.21.15.32:0/118744123 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+1849 (secure 0 0 0) 0x7f1984000c10 con 0x7f19881aca20 2021-04-09T18:14:34.021 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 23 2021-04-09T18:14:34.022 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:34.023 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":23,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4734,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1387466371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1387466371},{"type":"v1","addr":"172.21.15.32:6835","nonce":1387466371}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10},{"gid":4960,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/425942817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":425942817},{"type":"v1","addr":"172.21.15.86:6835","nonce":425942817}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":20},{"gid":5061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3129768501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3129768501},{"type":"v1","addr":"172.21.15.32:6833","nonce":3129768501}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":23},{"gid":54503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4021854443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4021854443},{"type":"v1","addr":"172.21.15.86:6837","nonce":4021854443}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[]} 2021-04-09T18:14:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 -- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19700188d0 msgr2=0x7f197001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:34.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19700188d0 0x7f197001add0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 -- 172.21.15.32:0/118744123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19881aca20 msgr2=0x7f19881ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:34.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19881aca20 0x7f19881ace20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f198400a4f0 tx=0x7f198400a890).stop 2021-04-09T18:14:34.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 -- 172.21.15.32:0/118744123 shutdown_connections 2021-04-09T18:14:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19700188d0 0x7f197001add0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198808fba0 0x7f198808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198818bd40 0x7f198818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 --2- 172.21.15.32:0/118744123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19881aca20 0x7f19881ace20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 -- 172.21.15.32:0/118744123 >> 172.21.15.32:0/118744123 conn(0x7f198819b460 msgr2=0x7f198819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:34.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 -- 172.21.15.32:0/118744123 shutdown_connections 2021-04-09T18:14:34.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:33.994 7f19907d8700 1 -- 172.21.15.32:0/118744123 wait complete. 2021-04-09T18:14:34.044 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:14:34.052 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:14:34.052 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:14:34.053 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:14:34.054 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:14:34.054 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:14:34.059 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:14:34.059 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:14:34.060 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:14:34.060 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:14:34.092 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:14:34.093 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:14:34.095 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:14:34.096 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:14:34.097 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:14:34.097 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:14:34.113 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:14:34.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:14:34.115 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:14:34.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:14:34.209 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:14:34.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:14:34.304 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:14:34.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:14:34.306 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:14:34.306 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:14:34.307 DEBUG:teuthology.parallel:result is None 2021-04-09T18:14:34.307 DEBUG:teuthology.parallel:result is None 2021-04-09T18:14:34.464 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:14:34.465 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:14:34.466 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:14:34.527 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:14:34.532 DEBUG:teuthology.parallel:result is None 2021-04-09T18:14:34.598 DEBUG:teuthology.parallel:result is None 2021-04-09T18:14:34.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:14:34.602 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:14:34.603 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:14:34.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:14:34.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:14:34.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.744 7f7ff9eaa700 1 Processor -- start 2021-04-09T18:14:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.744 7f7ff9eaa700 1 -- start start 2021-04-09T18:14:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff9eaa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40cffb0 0x7f7ff40d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff9eaa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ff40d7810 0x7f7ff40d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff9eaa700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7ff40d5c80 0x7f7ff40d6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff9eaa700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff40d8590 con 0x7f7ff40d7810 2021-04-09T18:14:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff9eaa700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff40d91f0 con 0x7f7ff40d5c80 2021-04-09T18:14:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff9eaa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff40d7290 con 0x7f7ff40cffb0 2021-04-09T18:14:34.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff2ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ff40d7810 0x7f7ff40d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:34.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff2ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ff40d7810 0x7f7ff40d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43344/0 (socket says 172.21.15.32:43344) 2021-04-09T18:14:34.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff2ffd700 1 -- 172.21.15.32:0/2335270071 learned_addr learned my addr 172.21.15.32:0/2335270071 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:34.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff3fff700 1 --2- 172.21.15.32:0/2335270071 >> v2:172.21.15.86:3300/0 conn(0x7f7ff40d5c80 0x7f7ff40d6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:34.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff37fe700 1 --2- 172.21.15.32:0/2335270071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40cffb0 0x7f7ff40d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:34.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff2ffd700 1 -- 172.21.15.32:0/2335270071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40cffb0 msgr2=0x7f7ff40d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:34.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff2ffd700 1 --2- 172.21.15.32:0/2335270071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40cffb0 0x7f7ff40d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff2ffd700 1 -- 172.21.15.32:0/2335270071 >> v2:172.21.15.86:3300/0 conn(0x7f7ff40d5c80 msgr2=0x7f7ff40d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:34.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff2ffd700 1 --2- 172.21.15.32:0/2335270071 >> v2:172.21.15.86:3300/0 conn(0x7f7ff40d5c80 0x7f7ff40d6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff2ffd700 1 -- 172.21.15.32:0/2335270071 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff40d8230 con 0x7f7ff40d7810 2021-04-09T18:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff2ffd700 1 --2- 172.21.15.32:0/2335270071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff40d7810 0x7f7ff40d7c10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f7fe8006700 tx=0x7f7fe8009b00).ready entity=mon.0 client_cookie=d4753ca8e7f16e45 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.745 7f7ff27fc700 1 -- 172.21.15.32:0/2335270071 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fe800c0c0 con 0x7f7ff40d7810 2021-04-09T18:14:34.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff27fc700 1 -- 172.21.15.32:0/2335270071 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe800c400 con 0x7f7ff40d7810 2021-04-09T18:14:34.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff27fc700 1 -- 172.21.15.32:0/2335270071 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fe800c8e0 con 0x7f7ff40d7810 2021-04-09T18:14:34.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff9eaa700 1 -- 172.21.15.32:0/2335270071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff40d7810 msgr2=0x7f7ff40d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:34.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff9eaa700 1 --2- 172.21.15.32:0/2335270071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff40d7810 0x7f7ff40d7c10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f7fe8006700 tx=0x7f7fe8009b00).stop 2021-04-09T18:14:34.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff9eaa700 1 -- 172.21.15.32:0/2335270071 shutdown_connections 2021-04-09T18:14:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff9eaa700 1 --2- 172.21.15.32:0/2335270071 >> v2:172.21.15.86:3300/0 conn(0x7f7ff40d5c80 0x7f7ff40d6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff9eaa700 1 --2- 172.21.15.32:0/2335270071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff40d7810 0x7f7ff40d7c10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff9eaa700 1 --2- 172.21.15.32:0/2335270071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40cffb0 0x7f7ff40d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff9eaa700 1 -- 172.21.15.32:0/2335270071 >> 172.21.15.32:0/2335270071 conn(0x7f7ff40ce5d0 msgr2=0x7f7ff40cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff9eaa700 1 -- 172.21.15.32:0/2335270071 shutdown_connections 2021-04-09T18:14:34.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.746 7f7ff9eaa700 1 -- 172.21.15.32:0/2335270071 wait complete. 2021-04-09T18:14:34.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.766 7f7ff9eaa700 1 Processor -- start 2021-04-09T18:14:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff9eaa700 1 -- start start 2021-04-09T18:14:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff9eaa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff41a7dc0 0x7f7ff41a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff9eaa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff4184080 0x7f7ff4184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:34.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff9eaa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff4184aa0 0x7f7ff408f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:34.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff9eaa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff41a8b40 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff9eaa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff41a7870 con 0x7f7ff4184aa0 2021-04-09T18:14:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff9eaa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff41a87e0 con 0x7f7ff4184080 2021-04-09T18:14:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff41a7dc0 0x7f7ff41a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff41a7dc0 0x7f7ff41a81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43346/0 (socket says 172.21.15.32:43346) 2021-04-09T18:14:34.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.767 7f7ff37fe700 1 -- 172.21.15.32:0/2455770297 learned_addr learned my addr 172.21.15.32:0/2455770297 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:34.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff2ffd700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff4184080 0x7f7ff4184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:34.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff3fff700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff4184aa0 0x7f7ff408f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:34.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff37fe700 1 -- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff4184080 msgr2=0x7f7ff4184480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:34.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff37fe700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff4184080 0x7f7ff4184480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff37fe700 1 -- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff4184aa0 msgr2=0x7f7ff408f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:34.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff37fe700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff4184aa0 0x7f7ff408f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:34.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff37fe700 1 -- 172.21.15.32:0/2455770297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff408f810 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff3fff700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff4184aa0 0x7f7ff408f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff37fe700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff41a7dc0 0x7f7ff41a81c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f7fe4006be0 tx=0x7f7fe400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7fdbfff700 1 -- 172.21.15.32:0/2455770297 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fe40064d0 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7fdbfff700 1 -- 172.21.15.32:0/2455770297 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe400bd80 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7fdbfff700 1 -- 172.21.15.32:0/2455770297 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fe4002220 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff408f810 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.768 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff408f810 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.769 7f7fdbfff700 1 -- 172.21.15.32:0/2455770297 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7fe4006c90 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.769 7f7fdbfff700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd4018930 0x7f7fd401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.769 7f7fdbfff700 1 -- 172.21.15.32:0/2455770297 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f7fe40326f0 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.769 7f7ff2ffd700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd4018930 0x7f7fd401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.769 7f7ff2ffd700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd4018930 0x7f7fd401adc0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.789 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff4182e80 con 0x7f7ff41a7dc0 2021-04-09T18:14:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.793 7f7fdbfff700 1 -- 172.21.15.32:0/2455770297 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7fe4000c90 con 0x7f7ff41a7dc0 2021-04-09T18:14:35.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:34.992 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f7ff40d7290 con 0x7f7ff41a7dc0 2021-04-09T18:14:35.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.642 7f7fdbfff700 1 -- 172.21.15.32:0/2455770297 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v36) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7fe4000ee0 con 0x7f7ff41a7dc0 2021-04-09T18:14:35.681 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:14:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.660 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd4018930 msgr2=0x7f7fd401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:35.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd4018930 0x7f7fd401adc0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff41a7dc0 msgr2=0x7f7ff41a81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:35.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff41a7dc0 0x7f7ff41a81c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f7fe4006be0 tx=0x7f7fe400b990).stop 2021-04-09T18:14:35.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 shutdown_connections 2021-04-09T18:14:35.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd4018930 0x7f7fd401adc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff4184aa0 0x7f7ff408f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff4184080 0x7f7ff4184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 --2- 172.21.15.32:0/2455770297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff41a7dc0 0x7f7ff41a81c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 >> 172.21.15.32:0/2455770297 conn(0x7f7ff41a8f50 msgr2=0x7f7ff4188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:35.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 shutdown_connections 2021-04-09T18:14:35.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.661 7f7ff9eaa700 1 -- 172.21.15.32:0/2455770297 wait complete. 2021-04-09T18:14:35.694 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:14:35.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:14:35.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.822 7efd72d54700 1 Processor -- start 2021-04-09T18:14:35.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.822 7efd72d54700 1 -- start start 2021-04-09T18:14:35.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.822 7efd72d54700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd6c10d010 0x7efd6c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:35.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.822 7efd72d54700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c0d1da0 0x7efd6c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:35.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.822 7efd72d54700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6c10d480 con 0x7efd6c0d82c0 2021-04-09T18:14:35.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.822 7efd72d54700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6c0d9220 con 0x7efd6c10d010 2021-04-09T18:14:35.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.822 7efd72d54700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c0d86c0 con 0x7efd6c0d1da0 2021-04-09T18:14:35.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd70b70700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd6c10d010 0x7efd6c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd71371700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efd6c0d82c0 0x7efd6c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:50872/0 (socket says 172.21.15.32:50872) 2021-04-09T18:14:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd70b70700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd6c10d010 0x7efd6c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43354/0 (socket says 172.21.15.32:43354) 2021-04-09T18:14:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd71371700 1 -- 172.21.15.32:0/3478436658 learned_addr learned my addr 172.21.15.32:0/3478436658 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd6bfff700 1 --2- 172.21.15.32:0/3478436658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c0d1da0 0x7efd6c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd6b7fe700 1 -- 172.21.15.32:0/3478436658 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164725610 0 0) 0x7efd64001440 con 0x7efd6c0d82c0 2021-04-09T18:14:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd70b70700 1 -- 172.21.15.32:0/3478436658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c0d1da0 msgr2=0x7efd6c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd70b70700 1 --2- 172.21.15.32:0/3478436658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c0d1da0 0x7efd6c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd70b70700 1 -- 172.21.15.32:0/3478436658 >> v1:172.21.15.86:6789/0 conn(0x7efd6c0d82c0 legacy=0x7efd6c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd70b70700 1 -- 172.21.15.32:0/3478436658 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c10e1f0 con 0x7efd6c10d010 2021-04-09T18:14:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd70b70700 1 --2- 172.21.15.32:0/3478436658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c10d010 0x7efd6c0d1780 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7efd5c006650 tx=0x7efd5c009a80).ready entity=mon.0 client_cookie=f6d535a7f41a49e3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd6b7fe700 1 -- 172.21.15.32:0/3478436658 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd5c0009b0 con 0x7efd6c10d010 2021-04-09T18:14:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd6b7fe700 1 -- 172.21.15.32:0/3478436658 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c00c0c0 con 0x7efd6c10d010 2021-04-09T18:14:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd6b7fe700 1 -- 172.21.15.32:0/3478436658 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd5c0009b0 con 0x7efd6c10d010 2021-04-09T18:14:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd72d54700 1 -- 172.21.15.32:0/3478436658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c10d010 msgr2=0x7efd6c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.823 7efd72d54700 1 --2- 172.21.15.32:0/3478436658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c10d010 0x7efd6c0d1780 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7efd5c006650 tx=0x7efd5c009a80).stop 2021-04-09T18:14:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.824 7efd72d54700 1 -- 172.21.15.32:0/3478436658 shutdown_connections 2021-04-09T18:14:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.824 7efd72d54700 1 --2- 172.21.15.32:0/3478436658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c0d1da0 0x7efd6c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.824 7efd72d54700 1 --2- 172.21.15.32:0/3478436658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c10d010 0x7efd6c0d1780 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.824 7efd72d54700 1 -- 172.21.15.32:0/3478436658 >> 172.21.15.32:0/3478436658 conn(0x7efd6c0ce5f0 msgr2=0x7efd6c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.824 7efd72d54700 1 -- 172.21.15.32:0/3478436658 shutdown_connections 2021-04-09T18:14:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.824 7efd72d54700 1 -- 172.21.15.32:0/3478436658 wait complete. 2021-04-09T18:14:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.844 7efd72d54700 1 Processor -- start 2021-04-09T18:14:35.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.844 7efd72d54700 1 -- start start 2021-04-09T18:14:35.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd72d54700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c1ac670 0x7efd6c1a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:35.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd72d54700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd6c1a5dc0 0x7efd6c0ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:35.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd72d54700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c1aca70 0x7efd6c0ba800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:35.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd72d54700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c1a5220 con 0x7efd6c1ac670 2021-04-09T18:14:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd72d54700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c18e100 con 0x7efd6c1a5dc0 2021-04-09T18:14:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd72d54700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c19ccc0 con 0x7efd6c1aca70 2021-04-09T18:14:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd70b70700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c1ac670 0x7efd6c1a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd70b70700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c1ac670 0x7efd6c1a57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43358/0 (socket says 172.21.15.32:43358) 2021-04-09T18:14:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd70b70700 1 -- 172.21.15.32:0/936922713 learned_addr learned my addr 172.21.15.32:0/936922713 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd6bfff700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd6c1a5dc0 0x7efd6c0ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd71371700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c1aca70 0x7efd6c0ba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd70b70700 1 -- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c1aca70 msgr2=0x7efd6c0ba800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd70b70700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c1aca70 0x7efd6c0ba800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd70b70700 1 -- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd6c1a5dc0 msgr2=0x7efd6c0ba1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd70b70700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd6c1a5dc0 0x7efd6c0ba1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd70b70700 1 -- 172.21.15.32:0/936922713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c0bae20 con 0x7efd6c1ac670 2021-04-09T18:14:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.845 7efd70b70700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c1ac670 0x7efd6c1a57a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7efd5c009630 tx=0x7efd5c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:35.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.846 7efd697fa700 1 -- 172.21.15.32:0/936922713 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd5c0093a0 con 0x7efd6c1ac670 2021-04-09T18:14:35.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.846 7efd697fa700 1 -- 172.21.15.32:0/936922713 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c00bd70 con 0x7efd6c1ac670 2021-04-09T18:14:35.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.846 7efd697fa700 1 -- 172.21.15.32:0/936922713 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd5c0054d0 con 0x7efd6c1ac670 2021-04-09T18:14:35.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.846 7efd72d54700 1 -- 172.21.15.32:0/936922713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6c0bae20 con 0x7efd6c1ac670 2021-04-09T18:14:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.846 7efd72d54700 1 -- 172.21.15.32:0/936922713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6c0bae20 con 0x7efd6c1ac670 2021-04-09T18:14:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.846 7efd697fa700 1 -- 172.21.15.32:0/936922713 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efd5c0054f0 con 0x7efd6c1ac670 2021-04-09T18:14:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.846 7efd697fa700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd4c018930 0x7efd4c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.846 7efd697fa700 1 -- 172.21.15.32:0/936922713 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4757+0+0 (secure 0 0 0) 0x7efd5c0326d0 con 0x7efd6c1ac670 2021-04-09T18:14:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.847 7efd6bfff700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd4c018930 0x7efd4c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.847 7efd6bfff700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd4c018930 0x7efd4c01ae10 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:35.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.867 7efd72d54700 1 -- 172.21.15.32:0/936922713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6c182d10 con 0x7efd6c1ac670 2021-04-09T18:14:35.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:35.870 7efd697fa700 1 -- 172.21.15.32:0/936922713 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efd5c0052e0 con 0x7efd6c1ac670 2021-04-09T18:14:36.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.055 7efd72d54700 1 -- 172.21.15.32:0/936922713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efd6c182d10 con 0x7efd6c1ac670 2021-04-09T18:14:36.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.055 7efd697fa700 1 -- 172.21.15.32:0/936922713 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+4539 (secure 0 0 0) 0x7efd5c00bd70 con 0x7efd6c1ac670 2021-04-09T18:14:36.080 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:36.080 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:14:36.081 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:14:36.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:14:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:14:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:14:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:14:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:14:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:14:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.095 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:14:36.096 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:14:36.116 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.116 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:14:36.117 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.121 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.121 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.122 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.151 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:14:36.178 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:14:36.178 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:14:36.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.221 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:14:36.221 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:14:36.221 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.221 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.222 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.222 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.228 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:14:36.228 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:14:36.228 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.229 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.229 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.258 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.258 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.266 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:14:36.266 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:14:36.266 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.267 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.267 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.275 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.275 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.279 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:14:36.280 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:14:36.284 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.284 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.284 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.285 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.285 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.285 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:14:36.286 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:14:36.286 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.286 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.286 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.287 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.287 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.288 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:14:36.288 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:14:36.288 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.288 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.289 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.289 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.307 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:14:36.310 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:14:36.314 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.314 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.318 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.320 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.321 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.321 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:14:36.322 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:14:36.324 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.324 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.324 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.325 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.325 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.326 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:14:36.326 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:14:36.326 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.326 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:14:36.326 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:14:36.327 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:14:36.327 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.336 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:14:36.336 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:14:36.336 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.342 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:14:36.342 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:14:36.348 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:14:36.348 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.349 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:14:36.349 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:14:36.349 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.349 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:14:36.350 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:14:36.350 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:14:36.350 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.356 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.364 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:14:36.364 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:14:36.365 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.367 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:14:36.367 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:14:36.368 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.369 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:36.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:36.386 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:14:36.387 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:14:36.387 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:14:36.388 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:14:36.388 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:14:36.388 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:14:36.389 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.389 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:36.390 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:14:36.390 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:14:36.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 -- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd4c018930 msgr2=0x7efd4c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:36.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd4c018930 0x7efd4c01ae10 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 -- 172.21.15.32:0/936922713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c1ac670 msgr2=0x7efd6c1a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:36.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c1ac670 0x7efd6c1a57a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7efd5c009630 tx=0x7efd5c00b980).stop 2021-04-09T18:14:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 -- 172.21.15.32:0/936922713 shutdown_connections 2021-04-09T18:14:36.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd4c018930 0x7efd4c01ae10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd6c1aca70 0x7efd6c0ba800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd6c1a5dc0 0x7efd6c0ba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 --2- 172.21.15.32:0/936922713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6c1ac670 0x7efd6c1a57a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 -- 172.21.15.32:0/936922713 >> 172.21.15.32:0/936922713 conn(0x7efd6c19b890 msgr2=0x7efd6c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 -- 172.21.15.32:0/936922713 shutdown_connections 2021-04-09T18:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.058 7efd72d54700 1 -- 172.21.15.32:0/936922713 wait complete. 2021-04-09T18:14:36.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:14:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.675 7fbb8626d700 1 Processor -- start 2021-04-09T18:14:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.675 7fbb8626d700 1 -- start start 2021-04-09T18:14:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb8626d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbb800d7ca0 0x7fbb800d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb8626d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8010d480 con 0x7fbb8010d010 2021-04-09T18:14:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb8626d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb800d86c0 con 0x7fbb8010da00 2021-04-09T18:14:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb8626d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb800d9220 con 0x7fbb800d7ca0 2021-04-09T18:14:36.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb7ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbb8010d010 0x7fbb800d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52956/0 (socket says 172.21.15.32:52956) 2021-04-09T18:14:36.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb7ffff700 1 -- 172.21.15.32:0/2667717584 learned_addr learned my addr 172.21.15.32:0/2667717584 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb77fff700 1 --2- 172.21.15.32:0/2667717584 >> v2:172.21.15.86:3300/0 conn(0x7fbb800d7ca0 0x7fbb800d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:36.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb7f7fe700 1 -- 172.21.15.32:0/2667717584 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520921432 0 0) 0x7fbb78000d30 con 0x7fbb8010da00 2021-04-09T18:14:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb7f7fe700 1 -- 172.21.15.32:0/2667717584 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6c002fe0 con 0x7fbb8010da00 2021-04-09T18:14:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb7f7fe700 1 -- 172.21.15.32:0/2667717584 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678285143 0 0) 0x7fbb70000ce0 con 0x7fbb8010d010 2021-04-09T18:14:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb7f7fe700 1 -- 172.21.15.32:0/2667717584 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6c003090 con 0x7fbb8010d010 2021-04-09T18:14:36.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb77fff700 1 -- 172.21.15.32:0/2667717584 >> v1:172.21.15.86:6789/0 conn(0x7fbb8010da00 legacy=0x7fbb8010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:36.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb77fff700 1 -- 172.21.15.32:0/2667717584 >> v1:172.21.15.32:6789/0 conn(0x7fbb8010d010 legacy=0x7fbb800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:36.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb77fff700 1 -- 172.21.15.32:0/2667717584 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb800d1f70 con 0x7fbb800d7ca0 2021-04-09T18:14:36.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.676 7fbb77fff700 1 --2- 172.21.15.32:0/2667717584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb800d7ca0 0x7fbb800d80a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbb68006650 tx=0x7fbb68009a80).ready entity=mon.1 client_cookie=5c27d25404252d68 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:36.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb7f7fe700 1 -- 172.21.15.32:0/2667717584 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb680009b0 con 0x7fbb800d7ca0 2021-04-09T18:14:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb7f7fe700 1 -- 172.21.15.32:0/2667717584 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6800c0c0 con 0x7fbb800d7ca0 2021-04-09T18:14:36.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb7f7fe700 1 -- 172.21.15.32:0/2667717584 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb6800c510 con 0x7fbb800d7ca0 2021-04-09T18:14:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb8626d700 1 -- 172.21.15.32:0/2667717584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb800d7ca0 msgr2=0x7fbb800d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb8626d700 1 --2- 172.21.15.32:0/2667717584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb800d7ca0 0x7fbb800d80a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbb68006650 tx=0x7fbb68009a80).stop 2021-04-09T18:14:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb8626d700 1 -- 172.21.15.32:0/2667717584 shutdown_connections 2021-04-09T18:14:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb8626d700 1 --2- 172.21.15.32:0/2667717584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb800d7ca0 0x7fbb800d80a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb8626d700 1 -- 172.21.15.32:0/2667717584 >> 172.21.15.32:0/2667717584 conn(0x7fbb800ce5f0 msgr2=0x7fbb800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb8626d700 1 -- 172.21.15.32:0/2667717584 shutdown_connections 2021-04-09T18:14:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.677 7fbb8626d700 1 -- 172.21.15.32:0/2667717584 wait complete. 2021-04-09T18:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.697 7fbb8626d700 1 Processor -- start 2021-04-09T18:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb8626d700 1 -- start start 2021-04-09T18:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb8626d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb80185a00 0x7fbb80097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:36.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb8626d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb800983b0 0x7fbb8017abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:36.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb8626d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb8017b1d0 0x7fbb80179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:36.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb8626d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800987b0 con 0x7fbb800983b0 2021-04-09T18:14:36.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb8626d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8017b5d0 con 0x7fbb80185a00 2021-04-09T18:14:36.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb8626d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb80179c30 con 0x7fbb8017b1d0 2021-04-09T18:14:36.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb77fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb800983b0 0x7fbb8017abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:36.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb77fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb800983b0 0x7fbb8017abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43376/0 (socket says 172.21.15.32:43376) 2021-04-09T18:14:36.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb77fff700 1 -- 172.21.15.32:0/3356746457 learned_addr learned my addr 172.21.15.32:0/3356746457 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb7ffff700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb80185a00 0x7fbb80097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.698 7fbb8488a700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb8017b1d0 0x7fbb80179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb77fff700 1 -- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb8017b1d0 msgr2=0x7fbb80179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb77fff700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb8017b1d0 0x7fbb80179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb77fff700 1 -- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb80185a00 msgr2=0x7fbb80097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb77fff700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb80185a00 0x7fbb80097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb77fff700 1 -- 172.21.15.32:0/3356746457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb80179e50 con 0x7fbb800983b0 2021-04-09T18:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb77fff700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb800983b0 0x7fbb8017abb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fbb68009650 tx=0x7fbb6800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb7d7fa700 1 -- 172.21.15.32:0/3356746457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb680093a0 con 0x7fbb800983b0 2021-04-09T18:14:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb7d7fa700 1 -- 172.21.15.32:0/3356746457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6800bd50 con 0x7fbb800983b0 2021-04-09T18:14:36.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb7d7fa700 1 -- 172.21.15.32:0/3356746457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb680052e0 con 0x7fbb800983b0 2021-04-09T18:14:36.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb80179e50 con 0x7fbb800983b0 2021-04-09T18:14:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb80179e50 con 0x7fbb800983b0 2021-04-09T18:14:36.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.699 7fbb7d7fa700 1 -- 172.21.15.32:0/3356746457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbb68006030 con 0x7fbb800983b0 2021-04-09T18:14:36.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.700 7fbb7d7fa700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbb640188c0 0x7fbb6401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:36.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.700 7fbb7d7fa700 1 -- 172.21.15.32:0/3356746457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fbb68032490 con 0x7fbb800983b0 2021-04-09T18:14:36.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.700 7fbb7ffff700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbb640188c0 0x7fbb6401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.700 7fbb7ffff700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbb640188c0 0x7fbb6401adc0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:36.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.720 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8017a3e0 con 0x7fbb800983b0 2021-04-09T18:14:36.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.723 7fbb7d7fa700 1 -- 172.21.15.32:0/3356746457 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbb680009c0 con 0x7fbb800983b0 2021-04-09T18:14:36.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.909 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb800987b0 con 0x7fbb800983b0 2021-04-09T18:14:36.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.910 7fbb7d7fa700 1 -- 172.21.15.32:0/3356746457 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+11144 (secure 0 0 0) 0x7fbb68006030 con 0x7fbb800983b0 2021-04-09T18:14:36.961 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:36.966 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":36,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:14:35.635939","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:14:36.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbb640188c0 msgr2=0x7fbb6401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:36.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbb640188c0 0x7fbb6401adc0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb800983b0 msgr2=0x7fbb8017abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:36.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb800983b0 0x7fbb8017abb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fbb68009650 tx=0x7fbb6800b960).stop 2021-04-09T18:14:36.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 shutdown_connections 2021-04-09T18:14:36.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbb640188c0 0x7fbb6401adc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb800983b0 0x7fbb8017abb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb8017b1d0 0x7fbb80179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 --2- 172.21.15.32:0/3356746457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb80185a00 0x7fbb80097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:36.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 >> 172.21.15.32:0/3356746457 conn(0x7fbb80190550 msgr2=0x7fbb801b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 shutdown_connections 2021-04-09T18:14:36.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:36.912 7fbb8626d700 1 -- 172.21.15.32:0/3356746457 wait complete. 2021-04-09T18:14:37.011 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:14:37.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:14:37.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.154 7f070e7c5700 1 Processor -- start 2021-04-09T18:14:37.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.154 7f070e7c5700 1 -- start start 2021-04-09T18:14:37.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.154 7f070e7c5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f070810da00 0x7f070810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.154 7f070e7c5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070810d480 con 0x7f070810d010 2021-04-09T18:14:37.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.154 7f070e7c5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07080d1f70 con 0x7f07080d7ca0 2021-04-09T18:14:37.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.154 7f070e7c5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07080d9220 con 0x7f070810da00 2021-04-09T18:14:37.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.154 7f0707fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f070810d010 0x7f07080d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52970/0 (socket says 172.21.15.32:52970) 2021-04-09T18:14:37.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.154 7f0707fff700 1 -- 172.21.15.32:0/3725290713 learned_addr learned my addr 172.21.15.32:0/3725290713 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:37.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.154 7f070cde2700 1 --2- 172.21.15.32:0/3725290713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f070810da00 0x7f070810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f07077fe700 1 -- 172.21.15.32:0/3725290713 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444703401 0 0) 0x7f06f8000ce0 con 0x7f070810d010 2021-04-09T18:14:37.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f07077fe700 1 -- 172.21.15.32:0/3725290713 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f4002fe0 con 0x7f070810d010 2021-04-09T18:14:37.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f07077fe700 1 -- 172.21.15.32:0/3725290713 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801775725 0 0) 0x7f06f0000c60 con 0x7f07080d7ca0 2021-04-09T18:14:37.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f07077fe700 1 -- 172.21.15.32:0/3725290713 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f40042c0 con 0x7f07080d7ca0 2021-04-09T18:14:37.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f070cde2700 1 -- 172.21.15.32:0/3725290713 >> v1:172.21.15.86:6789/0 conn(0x7f07080d7ca0 legacy=0x7f07080d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f070cde2700 1 -- 172.21.15.32:0/3725290713 >> v1:172.21.15.32:6789/0 conn(0x7f070810d010 legacy=0x7f07080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f070cde2700 1 -- 172.21.15.32:0/3725290713 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070810e420 con 0x7f070810da00 2021-04-09T18:14:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f070cde2700 1 --2- 172.21.15.32:0/3725290713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f070810da00 0x7f070810de00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0700006e50 tx=0x7f0700009e70).ready entity=mon.2 client_cookie=76de5d64a613705f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f07077fe700 1 -- 172.21.15.32:0/3725290713 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0700006a20 con 0x7f070810da00 2021-04-09T18:14:37.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f07077fe700 1 -- 172.21.15.32:0/3725290713 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0700006790 con 0x7f070810da00 2021-04-09T18:14:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f07077fe700 1 -- 172.21.15.32:0/3725290713 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f070000b700 con 0x7f070810da00 2021-04-09T18:14:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f070e7c5700 1 -- 172.21.15.32:0/3725290713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f070810da00 msgr2=0x7f070810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.155 7f070e7c5700 1 --2- 172.21.15.32:0/3725290713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f070810da00 0x7f070810de00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0700006e50 tx=0x7f0700009e70).stop 2021-04-09T18:14:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.156 7f070e7c5700 1 -- 172.21.15.32:0/3725290713 shutdown_connections 2021-04-09T18:14:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.156 7f070e7c5700 1 --2- 172.21.15.32:0/3725290713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f070810da00 0x7f070810de00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.156 7f070e7c5700 1 -- 172.21.15.32:0/3725290713 >> 172.21.15.32:0/3725290713 conn(0x7f07080ce5f0 msgr2=0x7f07080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:37.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.156 7f070e7c5700 1 -- 172.21.15.32:0/3725290713 shutdown_connections 2021-04-09T18:14:37.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.156 7f070e7c5700 1 -- 172.21.15.32:0/3725290713 wait complete. 2021-04-09T18:14:37.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.176 7f070e7c5700 1 Processor -- start 2021-04-09T18:14:37.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.176 7f070e7c5700 1 -- start start 2021-04-09T18:14:37.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070e7c5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0708191ee0 0x7f07080a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070e7c5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07080a6c30 0x7f07080b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070e7c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07080b4330 0x7f07081b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070e7c5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07080b3790 con 0x7f07080b4330 2021-04-09T18:14:37.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070e7c5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07080a7030 con 0x7f0708191ee0 2021-04-09T18:14:37.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070e7c5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07080b4730 con 0x7f07080a6c30 2021-04-09T18:14:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070cde2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07080b4330 0x7f07081b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070cde2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07080b4330 0x7f07081b7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43390/0 (socket says 172.21.15.32:43390) 2021-04-09T18:14:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070cde2700 1 -- 172.21.15.32:0/953716322 learned_addr learned my addr 172.21.15.32:0/953716322 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f0707fff700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0708191ee0 0x7f07080a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f06fffff700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07080a6c30 0x7f07080b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070cde2700 1 -- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07080a6c30 msgr2=0x7f07080b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070cde2700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07080a6c30 0x7f07080b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070cde2700 1 -- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0708191ee0 msgr2=0x7f07080a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070cde2700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0708191ee0 0x7f07080a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070cde2700 1 -- 172.21.15.32:0/953716322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07081b8580 con 0x7f07080b4330 2021-04-09T18:14:37.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070cde2700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07080b4330 0x7f07081b7f60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f07000099f0 tx=0x7f070000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:37.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f07057fa700 1 -- 172.21.15.32:0/953716322 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07000093d0 con 0x7f07080b4330 2021-04-09T18:14:37.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f07057fa700 1 -- 172.21.15.32:0/953716322 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07000090c0 con 0x7f07080b4330 2021-04-09T18:14:37.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f070e7c5700 1 -- 172.21.15.32:0/953716322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07081b8580 con 0x7f07080b4330 2021-04-09T18:14:37.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.177 7f07057fa700 1 -- 172.21.15.32:0/953716322 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0700001420 con 0x7f07080b4330 2021-04-09T18:14:37.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.178 7f070e7c5700 1 -- 172.21.15.32:0/953716322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07081b8580 con 0x7f07080b4330 2021-04-09T18:14:37.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.178 7f07057fa700 1 -- 172.21.15.32:0/953716322 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f07000056b0 con 0x7f07080b4330 2021-04-09T18:14:37.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.178 7f07057fa700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06ec0188c0 0x7f06ec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.178 7f07057fa700 1 -- 172.21.15.32:0/953716322 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f0700032a30 con 0x7f07080b4330 2021-04-09T18:14:37.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.178 7f0707fff700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06ec0188c0 0x7f06ec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.179 7f0707fff700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06ec0188c0 0x7f06ec01adc0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:37.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.198 7f070e7c5700 1 -- 172.21.15.32:0/953716322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070810d480 con 0x7f07080b4330 2021-04-09T18:14:37.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.201 7f07057fa700 1 -- 172.21.15.32:0/953716322 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f07000090a0 con 0x7f07080b4330 2021-04-09T18:14:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.390 7f070e7c5700 1 -- 172.21.15.32:0/953716322 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f070810d480 con 0x7f07080b4330 2021-04-09T18:14:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.390 7f07057fa700 1 -- 172.21.15.32:0/953716322 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v36) v1 ==== 92+0+10 (secure 0 0 0) 0x7f07000090a0 con 0x7f07080b4330 2021-04-09T18:14:37.415 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:14:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 -- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06ec0188c0 msgr2=0x7f06ec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06ec0188c0 0x7f06ec01adc0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 -- 172.21.15.32:0/953716322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07080b4330 msgr2=0x7f07081b7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07080b4330 0x7f07081b7f60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f07000099f0 tx=0x7f070000bd10).stop 2021-04-09T18:14:37.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 -- 172.21.15.32:0/953716322 shutdown_connections 2021-04-09T18:14:37.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06ec0188c0 0x7f06ec01adc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07080b4330 0x7f07081b7f60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07080a6c30 0x7f07080b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 --2- 172.21.15.32:0/953716322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0708191ee0 0x7f07080a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 -- 172.21.15.32:0/953716322 >> 172.21.15.32:0/953716322 conn(0x7f07080bbda0 msgr2=0x7f07081b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:37.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 -- 172.21.15.32:0/953716322 shutdown_connections 2021-04-09T18:14:37.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.393 7f070e7c5700 1 -- 172.21.15.32:0/953716322 wait complete. 2021-04-09T18:14:37.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:14:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.555 7fedfa6e7700 1 Processor -- start 2021-04-09T18:14:37.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.555 7fedfa6e7700 1 -- start start 2021-04-09T18:14:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedfa6e7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf40cffa0 0x7fedf40d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedfa6e7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fedf40d7800 0x7fedf40d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedfa6e7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedf40d5c70 0x7fedf40d6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedfa6e7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf40d8580 con 0x7fedf40d5c70 2021-04-09T18:14:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedfa6e7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf40d91e0 con 0x7fedf40d7800 2021-04-09T18:14:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedfa6e7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40d7280 con 0x7fedf40cffa0 2021-04-09T18:14:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedf8d04700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedf40d5c70 0x7fedf40d6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedf8d04700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedf40d5c70 0x7fedf40d6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43396/0 (socket says 172.21.15.32:43396) 2021-04-09T18:14:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedf8d04700 1 -- 172.21.15.32:0/753497182 learned_addr learned my addr 172.21.15.32:0/753497182 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedf3fff700 1 --2- 172.21.15.32:0/753497182 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf40cffa0 0x7fedf40d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedebfff700 1 --2- 172.21.15.32:0/753497182 >> v2:172.21.15.86:3300/0 conn(0x7fedf40d7800 0x7fedf40d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedf3fff700 1 -- 172.21.15.32:0/753497182 >> v2:172.21.15.86:3300/0 conn(0x7fedf40d7800 msgr2=0x7fedf40d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:37.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.556 7fedf3fff700 1 --2- 172.21.15.32:0/753497182 >> v2:172.21.15.86:3300/0 conn(0x7fedf40d7800 0x7fedf40d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedf3fff700 1 -- 172.21.15.32:0/753497182 >> v2:172.21.15.32:3300/0 conn(0x7fedf40d5c70 msgr2=0x7fedf40d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedf3fff700 1 --2- 172.21.15.32:0/753497182 >> v2:172.21.15.32:3300/0 conn(0x7fedf40d5c70 0x7fedf40d6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedf3fff700 1 -- 172.21.15.32:0/753497182 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf40d8220 con 0x7fedf40cffa0 2021-04-09T18:14:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedf8d04700 1 --2- 172.21.15.32:0/753497182 >> v2:172.21.15.32:3300/0 conn(0x7fedf40d5c70 0x7fedf40d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedf3fff700 1 --2- 172.21.15.32:0/753497182 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf40cffa0 0x7fedf40d2440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fede4006650 tx=0x7fede4009a00).ready entity=mon.2 client_cookie=85f8e0af94ed1632 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:37.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedf37fe700 1 -- 172.21.15.32:0/753497182 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fede4009df0 con 0x7fedf40cffa0 2021-04-09T18:14:37.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedf37fe700 1 -- 172.21.15.32:0/753497182 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede40009b0 con 0x7fedf40cffa0 2021-04-09T18:14:37.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedf37fe700 1 -- 172.21.15.32:0/753497182 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fede400c2b0 con 0x7fedf40cffa0 2021-04-09T18:14:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedfa6e7700 1 -- 172.21.15.32:0/753497182 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf40cffa0 msgr2=0x7fedf40d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.557 7fedfa6e7700 1 --2- 172.21.15.32:0/753497182 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf40cffa0 0x7fedf40d2440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fede4006650 tx=0x7fede4009a00).stop 2021-04-09T18:14:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.558 7fedfa6e7700 1 -- 172.21.15.32:0/753497182 shutdown_connections 2021-04-09T18:14:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.558 7fedfa6e7700 1 --2- 172.21.15.32:0/753497182 >> v2:172.21.15.32:3300/0 conn(0x7fedf40d5c70 0x7fedf40d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.558 7fedfa6e7700 1 --2- 172.21.15.32:0/753497182 >> v2:172.21.15.86:3300/0 conn(0x7fedf40d7800 0x7fedf40d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.558 7fedfa6e7700 1 --2- 172.21.15.32:0/753497182 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf40cffa0 0x7fedf40d2440 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.558 7fedfa6e7700 1 -- 172.21.15.32:0/753497182 >> 172.21.15.32:0/753497182 conn(0x7fedf40ce5c0 msgr2=0x7fedf40cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:37.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.558 7fedfa6e7700 1 -- 172.21.15.32:0/753497182 shutdown_connections 2021-04-09T18:14:37.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.558 7fedfa6e7700 1 -- 172.21.15.32:0/753497182 wait complete. 2021-04-09T18:14:37.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.578 7fedfa6e7700 1 Processor -- start 2021-04-09T18:14:37.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.578 7fedfa6e7700 1 -- start start 2021-04-09T18:14:37.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedfa6e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedf41acff0 0x7fedf408fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedfa6e7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf4090150 0x7fedf408c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedfa6e7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedf408cca0 0x7fedf408e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedfa6e7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf408c100 con 0x7fedf41acff0 2021-04-09T18:14:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedfa6e7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf408d0a0 con 0x7fedf408cca0 2021-04-09T18:14:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedfa6e7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf408e960 con 0x7fedf4090150 2021-04-09T18:14:37.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedf41acff0 0x7fedf408fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedf41acff0 0x7fedf408fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43400/0 (socket says 172.21.15.32:43400) 2021-04-09T18:14:37.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf3fff700 1 -- 172.21.15.32:0/3005800835 learned_addr learned my addr 172.21.15.32:0/3005800835 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedebfff700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf4090150 0x7fedf408c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf8d04700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedf408cca0 0x7fedf408e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf3fff700 1 -- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf4090150 msgr2=0x7fedf408c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf3fff700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf4090150 0x7fedf408c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf3fff700 1 -- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedf408cca0 msgr2=0x7fedf408e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf3fff700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedf408cca0 0x7fedf408e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf3fff700 1 -- 172.21.15.32:0/3005800835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf408eb80 con 0x7fedf41acff0 2021-04-09T18:14:37.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf3fff700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedf41acff0 0x7fedf408fb30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fede400b580 tx=0x7fede400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:37.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf17fa700 1 -- 172.21.15.32:0/3005800835 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fede4000c90 con 0x7fedf41acff0 2021-04-09T18:14:37.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf17fa700 1 -- 172.21.15.32:0/3005800835 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede4002030 con 0x7fedf41acff0 2021-04-09T18:14:37.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedf17fa700 1 -- 172.21.15.32:0/3005800835 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fede4002480 con 0x7fedf41acff0 2021-04-09T18:14:37.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf408eb80 con 0x7fedf41acff0 2021-04-09T18:14:37.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.579 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf408eb80 con 0x7fedf41acff0 2021-04-09T18:14:37.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.580 7fedf17fa700 1 -- 172.21.15.32:0/3005800835 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fede4002210 con 0x7fedf41acff0 2021-04-09T18:14:37.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.580 7fedf17fa700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedd8018930 0x7fedd801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:37.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.580 7fedf17fa700 1 -- 172.21.15.32:0/3005800835 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fede4032730 con 0x7fedf41acff0 2021-04-09T18:14:37.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.580 7fedebfff700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedd8018930 0x7fedd801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:37.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.581 7fedebfff700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedd8018930 0x7fedd801adc0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:37.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.600 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf4182dc0 con 0x7fedf41acff0 2021-04-09T18:14:37.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.604 7fedf17fa700 1 -- 172.21.15.32:0/3005800835 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fede4002240 con 0x7fedf41acff0 2021-04-09T18:14:37.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.780 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fedf40d7280 con 0x7fedf41acff0 2021-04-09T18:14:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.780 7fedf17fa700 1 -- 172.21.15.32:0/3005800835 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v36) v1 ==== 84+0+64 (secure 0 0 0) 0x7fede400bd10 con 0x7fedf41acff0 2021-04-09T18:14:37.900 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:37.901 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:14:37.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:14:37.901 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:14:37.904 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:14:37.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:37.908 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:14:37.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedd8018930 msgr2=0x7fedd801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedd8018930 0x7fedd801adc0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedf41acff0 msgr2=0x7fedf408fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:37.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedf41acff0 0x7fedf408fb30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fede400b580 tx=0x7fede400b920).stop 2021-04-09T18:14:37.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 shutdown_connections 2021-04-09T18:14:37.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedd8018930 0x7fedd801adc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedf408cca0 0x7fedf408e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedf4090150 0x7fedf408c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 --2- 172.21.15.32:0/3005800835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedf41acff0 0x7fedf408fb30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:37.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 >> 172.21.15.32:0/3005800835 conn(0x7fedf409abb0 msgr2=0x7fedf40a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 shutdown_connections 2021-04-09T18:14:37.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:37.782 7fedfa6e7700 1 -- 172.21.15.32:0/3005800835 wait complete. 2021-04-09T18:14:37.943 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:14:37.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:14:38.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.079 7f8c5f1b5700 1 Processor -- start 2021-04-09T18:14:38.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.079 7f8c5f1b5700 1 -- start start 2021-04-09T18:14:38.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c5f1b5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5810d010 0x7f8c580d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c5f1b5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8c580d1da0 0x7f8c580d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c5f1b5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5810d480 con 0x7f8c580d82c0 2021-04-09T18:14:38.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c5f1b5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c5810e780 con 0x7f8c580d1da0 2021-04-09T18:14:38.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c5f1b5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c580d8d30 con 0x7f8c5810d010 2021-04-09T18:14:38.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c57fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8c580d1da0 0x7f8c580d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c5d7d2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8c580d82c0 0x7f8c5810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52998/0 (socket says 172.21.15.32:52998) 2021-04-09T18:14:38.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c5d7d2700 1 -- 172.21.15.32:0/1904163589 learned_addr learned my addr 172.21.15.32:0/1904163589 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:38.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c5cfd1700 1 --2- 172.21.15.32:0/1904163589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5810d010 0x7f8c580d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c57fff700 1 -- 172.21.15.32:0/1904163589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5810d010 msgr2=0x7f8c580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:38.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c57fff700 1 --2- 172.21.15.32:0/1904163589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5810d010 0x7f8c580d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c57fff700 1 -- 172.21.15.32:0/1904163589 >> v1:172.21.15.32:6789/0 conn(0x7f8c580d82c0 legacy=0x7f8c5810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c57fff700 1 -- 172.21.15.32:0/1904163589 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c580d9220 con 0x7f8c580d1da0 2021-04-09T18:14:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c577fe700 1 -- 172.21.15.32:0/1904163589 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940044488 0 0) 0x7f8c50000d30 con 0x7f8c580d82c0 2021-04-09T18:14:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.080 7f8c5cfd1700 1 --2- 172.21.15.32:0/1904163589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5810d010 0x7f8c580d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:38.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c57fff700 1 --2- 172.21.15.32:0/1904163589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580d1da0 0x7f8c580d7ca0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f8c48006650 tx=0x7f8c48009a80).ready entity=mon.0 client_cookie=1cd4870716883de5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c577fe700 1 -- 172.21.15.32:0/1904163589 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c480009b0 con 0x7f8c580d1da0 2021-04-09T18:14:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c577fe700 1 -- 172.21.15.32:0/1904163589 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4800c0c0 con 0x7f8c580d1da0 2021-04-09T18:14:38.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c577fe700 1 -- 172.21.15.32:0/1904163589 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c4800c510 con 0x7f8c580d1da0 2021-04-09T18:14:38.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c5f1b5700 1 -- 172.21.15.32:0/1904163589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580d1da0 msgr2=0x7f8c580d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c5f1b5700 1 --2- 172.21.15.32:0/1904163589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580d1da0 0x7f8c580d7ca0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f8c48006650 tx=0x7f8c48009a80).stop 2021-04-09T18:14:38.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c5f1b5700 1 -- 172.21.15.32:0/1904163589 shutdown_connections 2021-04-09T18:14:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c5f1b5700 1 --2- 172.21.15.32:0/1904163589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c580d1da0 0x7f8c580d7ca0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c5f1b5700 1 --2- 172.21.15.32:0/1904163589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5810d010 0x7f8c580d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.081 7f8c5f1b5700 1 -- 172.21.15.32:0/1904163589 >> 172.21.15.32:0/1904163589 conn(0x7f8c580ce5f0 msgr2=0x7f8c580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.082 7f8c5f1b5700 1 -- 172.21.15.32:0/1904163589 shutdown_connections 2021-04-09T18:14:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.082 7f8c5f1b5700 1 -- 172.21.15.32:0/1904163589 wait complete. 2021-04-09T18:14:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5f1b5700 1 Processor -- start 2021-04-09T18:14:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5f1b5700 1 -- start start 2021-04-09T18:14:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5f1b5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580c3c00 0x7f8c5809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5f1b5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c58095240 0x7f8c581a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5f1b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c581a1270 0x7f8c580b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5f1b5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5809cda0 con 0x7f8c581a1270 2021-04-09T18:14:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5f1b5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c580c3680 con 0x7f8c580c3c00 2021-04-09T18:14:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5f1b5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c58095640 con 0x7f8c58095240 2021-04-09T18:14:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5d7d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c581a1270 0x7f8c580b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5d7d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c581a1270 0x7f8c580b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43412/0 (socket says 172.21.15.32:43412) 2021-04-09T18:14:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.102 7f8c5d7d2700 1 -- 172.21.15.32:0/203484328 learned_addr learned my addr 172.21.15.32:0/203484328 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c5cfd1700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580c3c00 0x7f8c5809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c57fff700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c58095240 0x7f8c581a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c5d7d2700 1 -- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c58095240 msgr2=0x7f8c581a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c5d7d2700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c58095240 0x7f8c581a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c5d7d2700 1 -- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580c3c00 msgr2=0x7f8c5809c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c5d7d2700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580c3c00 0x7f8c5809c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c5d7d2700 1 -- 172.21.15.32:0/203484328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c581a1670 con 0x7f8c581a1270 2021-04-09T18:14:38.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c5d7d2700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c581a1270 0x7f8c580b0cf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f8c50007f90 tx=0x7f8c5000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:38.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c557fa700 1 -- 172.21.15.32:0/203484328 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c50000cc0 con 0x7f8c581a1270 2021-04-09T18:14:38.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c557fa700 1 -- 172.21.15.32:0/203484328 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c50007020 con 0x7f8c581a1270 2021-04-09T18:14:38.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c557fa700 1 -- 172.21.15.32:0/203484328 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c50007500 con 0x7f8c581a1270 2021-04-09T18:14:38.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c581a1670 con 0x7f8c581a1270 2021-04-09T18:14:38.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.103 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c581a1670 con 0x7f8c581a1270 2021-04-09T18:14:38.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.104 7f8c557fa700 1 -- 172.21.15.32:0/203484328 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8c50007200 con 0x7f8c581a1270 2021-04-09T18:14:38.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.104 7f8c557fa700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c3c0188d0 0x7f8c3c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.104 7f8c557fa700 1 -- 172.21.15.32:0/203484328 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f8c50032bc0 con 0x7f8c581a1270 2021-04-09T18:14:38.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.104 7f8c5cfd1700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c3c0188d0 0x7f8c3c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.104 7f8c5cfd1700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c3c0188d0 0x7f8c3c01add0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:38.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.124 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c58093d50 con 0x7f8c581a1270 2021-04-09T18:14:38.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.127 7f8c557fa700 1 -- 172.21.15.32:0/203484328 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8c50007d00 con 0x7f8c581a1270 2021-04-09T18:14:38.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.309 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f8c5810d480 con 0x7f8c581a1270 2021-04-09T18:14:38.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.366 7f8c557fa700 1 -- 172.21.15.32:0/203484328 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v37) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8c50007020 con 0x7f8c581a1270 2021-04-09T18:14:38.396 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:14:38.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c3c0188d0 msgr2=0x7f8c3c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c3c0188d0 0x7f8c3c01add0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c581a1270 msgr2=0x7f8c580b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c581a1270 0x7f8c580b0cf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f8c50007f90 tx=0x7f8c5000ac50).stop 2021-04-09T18:14:38.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 shutdown_connections 2021-04-09T18:14:38.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c3c0188d0 0x7f8c3c01add0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c581a1270 0x7f8c580b0cf0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c58095240 0x7f8c581a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 --2- 172.21.15.32:0/203484328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580c3c00 0x7f8c5809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 >> 172.21.15.32:0/203484328 conn(0x7f8c580975f0 msgr2=0x7f8c58096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:38.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 shutdown_connections 2021-04-09T18:14:38.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.375 7f8c5f1b5700 1 -- 172.21.15.32:0/203484328 wait complete. 2021-04-09T18:14:38.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:14:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.538 7f19db486700 1 Processor -- start 2021-04-09T18:14:38.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.539 7f19db486700 1 -- start start 2021-04-09T18:14:38.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.539 7f19db486700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19d410da00 0x7f19d410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.539 7f19db486700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d410d480 con 0x7f19d410d010 2021-04-09T18:14:38.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.539 7f19db486700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d40d1f70 con 0x7f19d40d7ca0 2021-04-09T18:14:38.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.539 7f19db486700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d40d8d30 con 0x7f19d410da00 2021-04-09T18:14:38.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.539 7f19d9aa3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19d410da00 0x7f19d410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.539 7f19d92a2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f19d410d010 0x7f19d40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53010/0 (socket says 172.21.15.32:53010) 2021-04-09T18:14:38.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.539 7f19d92a2700 1 -- 172.21.15.32:0/3711536031 learned_addr learned my addr 172.21.15.32:0/3711536031 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19cbfff700 1 -- 172.21.15.32:0/3711536031 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167642256 0 0) 0x7f19d0000ce0 con 0x7f19d410d010 2021-04-09T18:14:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19cbfff700 1 -- 172.21.15.32:0/3711536031 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c0002fe0 con 0x7f19d410d010 2021-04-09T18:14:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19cbfff700 1 -- 172.21.15.32:0/3711536031 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461368188 0 0) 0x7f19c4000c60 con 0x7f19d40d7ca0 2021-04-09T18:14:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19d9aa3700 1 -- 172.21.15.32:0/3711536031 >> v1:172.21.15.86:6789/0 conn(0x7f19d40d7ca0 legacy=0x7f19d40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19d9aa3700 1 -- 172.21.15.32:0/3711536031 >> v1:172.21.15.32:6789/0 conn(0x7f19d410d010 legacy=0x7f19d40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19d9aa3700 1 -- 172.21.15.32:0/3711536031 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d40d9220 con 0x7f19d410da00 2021-04-09T18:14:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19d9aa3700 1 --2- 172.21.15.32:0/3711536031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d410da00 0x7f19d410de00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f19cc006e30 tx=0x7f19cc009e60).ready entity=mon.0 client_cookie=e34b3c9a33c60dcc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19cbfff700 1 -- 172.21.15.32:0/3711536031 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19cc006a50 con 0x7f19d410da00 2021-04-09T18:14:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19cbfff700 1 -- 172.21.15.32:0/3711536031 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc00b300 con 0x7f19d410da00 2021-04-09T18:14:38.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19cbfff700 1 -- 172.21.15.32:0/3711536031 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19cc00b750 con 0x7f19d410da00 2021-04-09T18:14:38.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19db486700 1 -- 172.21.15.32:0/3711536031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d410da00 msgr2=0x7f19d410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.540 7f19db486700 1 --2- 172.21.15.32:0/3711536031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d410da00 0x7f19d410de00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f19cc006e30 tx=0x7f19cc009e60).stop 2021-04-09T18:14:38.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.541 7f19db486700 1 -- 172.21.15.32:0/3711536031 shutdown_connections 2021-04-09T18:14:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.541 7f19db486700 1 --2- 172.21.15.32:0/3711536031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d410da00 0x7f19d410de00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.541 7f19db486700 1 -- 172.21.15.32:0/3711536031 >> 172.21.15.32:0/3711536031 conn(0x7f19d40ce5f0 msgr2=0x7f19d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.541 7f19db486700 1 -- 172.21.15.32:0/3711536031 shutdown_connections 2021-04-09T18:14:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.541 7f19db486700 1 -- 172.21.15.32:0/3711536031 wait complete. 2021-04-09T18:14:38.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.561 7f19db486700 1 Processor -- start 2021-04-09T18:14:38.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.561 7f19db486700 1 -- start start 2021-04-09T18:14:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19db486700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19d41b8550 0x7f19d40bff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19db486700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40c0540 0x7f19d4194ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19db486700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19d40c0940 0x7f19d41950f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19db486700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d41b8d80 con 0x7f19d40c0540 2021-04-09T18:14:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19db486700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d41b8950 con 0x7f19d40c0940 2021-04-09T18:14:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19db486700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d4195710 con 0x7f19d41b8550 2021-04-09T18:14:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d8aa1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40c0540 0x7f19d4194ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d8aa1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40c0540 0x7f19d4194ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43428/0 (socket says 172.21.15.32:43428) 2021-04-09T18:14:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d8aa1700 1 -- 172.21.15.32:0/971832294 learned_addr learned my addr 172.21.15.32:0/971832294 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d9aa3700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19d40c0940 0x7f19d41950f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d92a2700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19d41b8550 0x7f19d40bff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d8aa1700 1 -- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19d41b8550 msgr2=0x7f19d40bff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d8aa1700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19d41b8550 0x7f19d40bff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d8aa1700 1 -- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19d40c0940 msgr2=0x7f19d41950f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:38.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d8aa1700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19d40c0940 0x7f19d41950f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.562 7f19d8aa1700 1 -- 172.21.15.32:0/971832294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d40bb0f0 con 0x7f19d40c0540 2021-04-09T18:14:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.563 7f19d8aa1700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40c0540 0x7f19d4194ad0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f19c4006f20 tx=0x7f19c400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.563 7f19c9ffb700 1 -- 172.21.15.32:0/971832294 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19c400bd80 con 0x7f19d40c0540 2021-04-09T18:14:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.563 7f19c9ffb700 1 -- 172.21.15.32:0/971832294 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c4001080 con 0x7f19d40c0540 2021-04-09T18:14:38.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.563 7f19c9ffb700 1 -- 172.21.15.32:0/971832294 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19c4001550 con 0x7f19d40c0540 2021-04-09T18:14:38.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.563 7f19db486700 1 -- 172.21.15.32:0/971832294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d40bb0f0 con 0x7f19d40c0540 2021-04-09T18:14:38.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.563 7f19db486700 1 -- 172.21.15.32:0/971832294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d40bb0f0 con 0x7f19d40c0540 2021-04-09T18:14:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.563 7f19c9ffb700 1 -- 172.21.15.32:0/971832294 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f19c400a000 con 0x7f19d40c0540 2021-04-09T18:14:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.564 7f19c9ffb700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19b4018930 0x7f19b401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.564 7f19c9ffb700 1 -- 172.21.15.32:0/971832294 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5111+0+0 (secure 0 0 0) 0x7f19c4032810 con 0x7f19d40c0540 2021-04-09T18:14:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.564 7f19d92a2700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19b4018930 0x7f19b401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:38.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.564 7f19d92a2700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19b4018930 0x7f19b401adc0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:38.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.583 7f19db486700 1 -- 172.21.15.32:0/971832294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d410d480 con 0x7f19d40c0540 2021-04-09T18:14:38.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.587 7f19c9ffb700 1 -- 172.21.15.32:0/971832294 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f19c4006bf0 con 0x7f19d40c0540 2021-04-09T18:14:38.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:38.768 7f19db486700 1 -- 172.21.15.32:0/971832294 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f19d410d480 con 0x7f19d40c0540 2021-04-09T18:14:39.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.443 7f19c9ffb700 1 -- 172.21.15.32:0/971832294 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v41) v1 ==== 125+0+0 (secure 0 0 0) 0x7f19c400a000 con 0x7f19d40c0540 2021-04-09T18:14:39.508 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:14:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.485 7f19db486700 1 -- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19b4018930 msgr2=0x7f19b401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:39.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.485 7f19db486700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19b4018930 0x7f19b401adc0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.485 7f19db486700 1 -- 172.21.15.32:0/971832294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40c0540 msgr2=0x7f19d4194ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.485 7f19db486700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40c0540 0x7f19d4194ad0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f19c4006f20 tx=0x7f19c400b990).stop 2021-04-09T18:14:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.485 7f19db486700 1 -- 172.21.15.32:0/971832294 shutdown_connections 2021-04-09T18:14:39.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.485 7f19db486700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19b4018930 0x7f19b401adc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.485 7f19db486700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19d40c0540 0x7f19d4194ad0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.485 7f19db486700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19d40c0940 0x7f19d41950f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.485 7f19db486700 1 --2- 172.21.15.32:0/971832294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19d41b8550 0x7f19d40bff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.486 7f19db486700 1 -- 172.21.15.32:0/971832294 >> 172.21.15.32:0/971832294 conn(0x7f19d40bce00 msgr2=0x7f19d40b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:39.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.486 7f19db486700 1 -- 172.21.15.32:0/971832294 shutdown_connections 2021-04-09T18:14:39.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.486 7f19db486700 1 -- 172.21.15.32:0/971832294 wait complete. 2021-04-09T18:14:39.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:14:39.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.686 7f5b3ba9f700 1 Processor -- start 2021-04-09T18:14:39.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.686 7f5b3ba9f700 1 -- start start 2021-04-09T18:14:39.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.687 7f5b3ba9f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5b3410d050 0x7f5b340cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:39.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.687 7f5b3ba9f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340d84b0 0x7f5b3410da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:39.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.687 7f5b3ba9f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3410d500 con 0x7f5b340d0160 2021-04-09T18:14:39.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.687 7f5b3ba9f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b340d8d70 con 0x7f5b3410d050 2021-04-09T18:14:39.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.687 7f5b3ba9f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b340d9260 con 0x7f5b340d84b0 2021-04-09T18:14:39.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.687 7f5b3a0bc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340d84b0 0x7f5b3410da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:39.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.687 7f5b398bb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5b3410d050 0x7f5b340cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:39.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.687 7f5b390ba700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5b340d0160 0x7f5b340d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:50954/0 (socket says 172.21.15.32:50954) 2021-04-09T18:14:39.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.687 7f5b390ba700 1 -- 172.21.15.32:0/3586092600 learned_addr learned my addr 172.21.15.32:0/3586092600 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:39.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b388b9700 1 -- 172.21.15.32:0/3586092600 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049029286 0 0) 0x7f5b240010c0 con 0x7f5b340d0160 2021-04-09T18:14:39.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b398bb700 1 -- 172.21.15.32:0/3586092600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340d84b0 msgr2=0x7f5b3410da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:39.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b398bb700 1 --2- 172.21.15.32:0/3586092600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340d84b0 0x7f5b3410da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b398bb700 1 -- 172.21.15.32:0/3586092600 >> v1:172.21.15.86:6789/0 conn(0x7f5b340d0160 legacy=0x7f5b340d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:39.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b398bb700 1 -- 172.21.15.32:0/3586092600 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3410e0a0 con 0x7f5b3410d050 2021-04-09T18:14:39.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b3a0bc700 1 --2- 172.21.15.32:0/3586092600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340d84b0 0x7f5b3410da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:39.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b398bb700 1 --2- 172.21.15.32:0/3586092600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3410d050 0x7f5b340cfb40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5b280069d0 tx=0x7f5b28009e40).ready entity=mon.1 client_cookie=b4e33cd611d3de98 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:39.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b388b9700 1 -- 172.21.15.32:0/3586092600 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b2800b510 con 0x7f5b3410d050 2021-04-09T18:14:39.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b388b9700 1 -- 172.21.15.32:0/3586092600 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2800b890 con 0x7f5b3410d050 2021-04-09T18:14:39.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b388b9700 1 -- 172.21.15.32:0/3586092600 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b28000d30 con 0x7f5b3410d050 2021-04-09T18:14:39.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b3ba9f700 1 -- 172.21.15.32:0/3586092600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3410d050 msgr2=0x7f5b340cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.688 7f5b3ba9f700 1 --2- 172.21.15.32:0/3586092600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3410d050 0x7f5b340cfb40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5b280069d0 tx=0x7f5b28009e40).stop 2021-04-09T18:14:39.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.689 7f5b3ba9f700 1 -- 172.21.15.32:0/3586092600 shutdown_connections 2021-04-09T18:14:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.689 7f5b3ba9f700 1 --2- 172.21.15.32:0/3586092600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340d84b0 0x7f5b3410da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.689 7f5b3ba9f700 1 --2- 172.21.15.32:0/3586092600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3410d050 0x7f5b340cfb40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.689 7f5b3ba9f700 1 -- 172.21.15.32:0/3586092600 >> 172.21.15.32:0/3586092600 conn(0x7f5b340ce620 msgr2=0x7f5b340d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:39.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.689 7f5b3ba9f700 1 -- 172.21.15.32:0/3586092600 shutdown_connections 2021-04-09T18:14:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.689 7f5b3ba9f700 1 -- 172.21.15.32:0/3586092600 wait complete. 2021-04-09T18:14:39.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.709 7f5b3ba9f700 1 Processor -- start 2021-04-09T18:14:39.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.709 7f5b3ba9f700 1 -- start start 2021-04-09T18:14:39.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b3ba9f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340ad600 0x7f5b3409ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:39.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b3ba9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3409b1b0 0x7f5b340ba220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:39.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b3ba9f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b340ba840 0x7f5b340a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:39.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b3ba9f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3409fa80 con 0x7f5b3409b1b0 2021-04-09T18:14:39.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b3ba9f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3409b5b0 con 0x7f5b340ba840 2021-04-09T18:14:39.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b3ba9f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b340bac40 con 0x7f5b340ad600 2021-04-09T18:14:39.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b390ba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3409b1b0 0x7f5b340ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b390ba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3409b1b0 0x7f5b340ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43444/0 (socket says 172.21.15.32:43444) 2021-04-09T18:14:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b390ba700 1 -- 172.21.15.32:0/380981751 learned_addr learned my addr 172.21.15.32:0/380981751 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b3a0bc700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b340ba840 0x7f5b340a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:39.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b398bb700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340ad600 0x7f5b3409ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:39.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b390ba700 1 -- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340ad600 msgr2=0x7f5b3409ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:39.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b390ba700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340ad600 0x7f5b3409ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b390ba700 1 -- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b340ba840 msgr2=0x7f5b340a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:39.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b390ba700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b340ba840 0x7f5b340a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b390ba700 1 -- 172.21.15.32:0/380981751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b340a97c0 con 0x7f5b3409b1b0 2021-04-09T18:14:39.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b390ba700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3409b1b0 0x7f5b340ba220 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f5b2400a5d0 tx=0x7f5b2400a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:39.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b327fc700 1 -- 172.21.15.32:0/380981751 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b24000c10 con 0x7f5b3409b1b0 2021-04-09T18:14:39.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.710 7f5b327fc700 1 -- 172.21.15.32:0/380981751 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b240070d0 con 0x7f5b3409b1b0 2021-04-09T18:14:39.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.711 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b340a97c0 con 0x7f5b3409b1b0 2021-04-09T18:14:39.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.711 7f5b327fc700 1 -- 172.21.15.32:0/380981751 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b240047b0 con 0x7f5b3409b1b0 2021-04-09T18:14:39.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.711 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b340a97c0 con 0x7f5b3409b1b0 2021-04-09T18:14:39.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.711 7f5b327fc700 1 -- 172.21.15.32:0/380981751 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5b24004320 con 0x7f5b3409b1b0 2021-04-09T18:14:39.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.711 7f5b327fc700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b14018930 0x7f5b1401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:39.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.711 7f5b327fc700 1 -- 172.21.15.32:0/380981751 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f5b240329a0 con 0x7f5b3409b1b0 2021-04-09T18:14:39.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.712 7f5b398bb700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b14018930 0x7f5b1401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:39.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.712 7f5b398bb700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b14018930 0x7f5b1401ae10 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:39.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.731 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b340b9040 con 0x7f5b3409b1b0 2021-04-09T18:14:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.735 7f5b327fc700 1 -- 172.21.15.32:0/380981751 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5b240070d0 con 0x7f5b3409b1b0 2021-04-09T18:14:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.917 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f5b3410d500 con 0x7f5b3409b1b0 2021-04-09T18:14:39.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.921 7f5b327fc700 1 -- 172.21.15.32:0/380981751 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 6 and data pool 7 v25) v1 ==== 187+0+0 (secure 0 0 0) 0x7f5b24004780 con 0x7f5b3409b1b0 2021-04-09T18:14:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 6 and data pool 7 2021-04-09T18:14:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b14018930 msgr2=0x7f5b1401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b14018930 0x7f5b1401ae10 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3409b1b0 msgr2=0x7f5b340ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3409b1b0 0x7f5b340ba220 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f5b2400a5d0 tx=0x7f5b2400a970).stop 2021-04-09T18:14:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 shutdown_connections 2021-04-09T18:14:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b14018930 0x7f5b1401ae10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3409b1b0 0x7f5b340ba220 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b340ba840 0x7f5b340a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 --2- 172.21.15.32:0/380981751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b340ad600 0x7f5b3409ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.928 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 >> 172.21.15.32:0/380981751 conn(0x7f5b340ae6f0 msgr2=0x7f5b34097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:40.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.929 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 shutdown_connections 2021-04-09T18:14:40.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:39.929 7f5b3ba9f700 1 -- 172.21.15.32:0/380981751 wait complete. 2021-04-09T18:14:40.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:14:40.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.158 7fc142b9c700 1 Processor -- start 2021-04-09T18:14:40.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.158 7fc142b9c700 1 -- start start 2021-04-09T18:14:40.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc142b9c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c0d6c80 0x7fc13c0d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc142b9c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc13c0d2440 0x7fc13c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc142b9c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13c0d7280 con 0x7fc13c0cffa0 2021-04-09T18:14:40.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc142b9c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c0d7080 con 0x7fc13c0d2440 2021-04-09T18:14:40.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc142b9c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0d91e0 con 0x7fc13c0d6c80 2021-04-09T18:14:40.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc1411b9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc13c0d2440 0x7fc13c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc1411b9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc13c0d2440 0x7fc13c10cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43452/0 (socket says 172.21.15.32:43452) 2021-04-09T18:14:40.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc1411b9700 1 -- 172.21.15.32:0/3648734356 learned_addr learned my addr 172.21.15.32:0/3648734356 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:40.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc13bfff700 1 --2- 172.21.15.32:0/3648734356 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c0d6c80 0x7fc13c0d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc1411b9700 1 -- 172.21.15.32:0/3648734356 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c0d6c80 msgr2=0x7fc13c0d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:40.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc1411b9700 1 --2- 172.21.15.32:0/3648734356 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c0d6c80 0x7fc13c0d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc1411b9700 1 -- 172.21.15.32:0/3648734356 >> v1:172.21.15.86:6789/0 conn(0x7fc13c0cffa0 legacy=0x7fc13c0d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc1411b9700 1 -- 172.21.15.32:0/3648734356 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0d2840 con 0x7fc13c0d2440 2021-04-09T18:14:40.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc13b7fe700 1 -- 172.21.15.32:0/3648734356 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104839026 0 0) 0x7fc12c000c60 con 0x7fc13c0cffa0 2021-04-09T18:14:40.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.159 7fc1411b9700 1 --2- 172.21.15.32:0/3648734356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c0d2440 0x7fc13c10cfd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fc134006f00 tx=0x7fc134009f10).ready entity=mon.0 client_cookie=97f323cdd79d2af4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:40.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc13b7fe700 1 -- 172.21.15.32:0/3648734356 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc13400b3b0 con 0x7fc13c0d2440 2021-04-09T18:14:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc13b7fe700 1 -- 172.21.15.32:0/3648734356 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13400b730 con 0x7fc13c0d2440 2021-04-09T18:14:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc13b7fe700 1 -- 172.21.15.32:0/3648734356 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc13400bc10 con 0x7fc13c0d2440 2021-04-09T18:14:40.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc142b9c700 1 -- 172.21.15.32:0/3648734356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c0d2440 msgr2=0x7fc13c10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc142b9c700 1 --2- 172.21.15.32:0/3648734356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c0d2440 0x7fc13c10cfd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fc134006f00 tx=0x7fc134009f10).stop 2021-04-09T18:14:40.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc142b9c700 1 -- 172.21.15.32:0/3648734356 shutdown_connections 2021-04-09T18:14:40.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc142b9c700 1 --2- 172.21.15.32:0/3648734356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c0d2440 0x7fc13c10cfd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc142b9c700 1 --2- 172.21.15.32:0/3648734356 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c0d6c80 0x7fc13c0d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc142b9c700 1 -- 172.21.15.32:0/3648734356 >> 172.21.15.32:0/3648734356 conn(0x7fc13c0ce5c0 msgr2=0x7fc13c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:40.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc142b9c700 1 -- 172.21.15.32:0/3648734356 shutdown_connections 2021-04-09T18:14:40.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.160 7fc142b9c700 1 -- 172.21.15.32:0/3648734356 wait complete. 2021-04-09T18:14:40.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc142b9c700 1 Processor -- start 2021-04-09T18:14:40.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc142b9c700 1 -- start start 2021-04-09T18:14:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc142b9c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c18e990 0x7fc13c18ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc142b9c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c08bd20 0x7fc13c08c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc142b9c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc13c08a650 0x7fc13c08aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc142b9c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c08c740 con 0x7fc13c18e990 2021-04-09T18:14:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc142b9c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c08b070 con 0x7fc13c08a650 2021-04-09T18:14:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc142b9c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c08b290 con 0x7fc13c08bd20 2021-04-09T18:14:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc1409b8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c18e990 0x7fc13c18ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc1409b8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c18e990 0x7fc13c18ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43460/0 (socket says 172.21.15.32:43460) 2021-04-09T18:14:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc1409b8700 1 -- 172.21.15.32:0/3012170320 learned_addr learned my addr 172.21.15.32:0/3012170320 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:40.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc1411b9700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc13c08a650 0x7fc13c08aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.181 7fc13bfff700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c08bd20 0x7fc13c08c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc1409b8700 1 -- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c08bd20 msgr2=0x7fc13c08c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc1409b8700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c08bd20 0x7fc13c08c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc1409b8700 1 -- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc13c08a650 msgr2=0x7fc13c08aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc1409b8700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc13c08a650 0x7fc13c08aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc1409b8700 1 -- 172.21.15.32:0/3012170320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c08b4b0 con 0x7fc13c18e990 2021-04-09T18:14:40.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc1409b8700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c18e990 0x7fc13c18ed90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fc12c00b580 tx=0x7fc12c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:40.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc1397fa700 1 -- 172.21.15.32:0/3012170320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc12c00bd10 con 0x7fc13c18e990 2021-04-09T18:14:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc1397fa700 1 -- 172.21.15.32:0/3012170320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c001080 con 0x7fc13c18e990 2021-04-09T18:14:40.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c08b4b0 con 0x7fc13c18e990 2021-04-09T18:14:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc1397fa700 1 -- 172.21.15.32:0/3012170320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc12c001550 con 0x7fc13c18e990 2021-04-09T18:14:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.182 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c08b4b0 con 0x7fc13c18e990 2021-04-09T18:14:40.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.183 7fc1397fa700 1 -- 172.21.15.32:0/3012170320 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc12c00a000 con 0x7fc13c18e990 2021-04-09T18:14:40.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.183 7fc1397fa700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc11c018930 0x7fc11c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.183 7fc1397fa700 1 -- 172.21.15.32:0/3012170320 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc12c032a50 con 0x7fc13c18e990 2021-04-09T18:14:40.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.183 7fc13bfff700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc11c018930 0x7fc11c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.183 7fc13bfff700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc11c018930 0x7fc11c01ae10 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:40.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.203 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c1b70c0 con 0x7fc13c18e990 2021-04-09T18:14:40.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.206 7fc1397fa700 1 -- 172.21.15.32:0/3012170320 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc12c006bf0 con 0x7fc13c18e990 2021-04-09T18:14:40.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.394 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc13c1b70c0 con 0x7fc13c18e990 2021-04-09T18:14:40.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.395 7fc1397fa700 1 -- 172.21.15.32:0/3012170320 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+14006 (secure 0 0 0) 0x7fc12c00a000 con 0x7fc13c18e990 2021-04-09T18:14:40.424 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:40.425 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:14:39.918794","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:14:38.344198","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-04-09 18:14:38.803133","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:14:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc11c018930 msgr2=0x7fc11c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc11c018930 0x7fc11c01ae10 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c18e990 msgr2=0x7fc13c18ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c18e990 0x7fc13c18ed90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fc12c00b580 tx=0x7fc12c00b920).stop 2021-04-09T18:14:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 shutdown_connections 2021-04-09T18:14:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc11c018930 0x7fc11c01ae10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc13c08a650 0x7fc13c08aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc13c08bd20 0x7fc13c08c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 --2- 172.21.15.32:0/3012170320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc13c18e990 0x7fc13c18ed90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 >> 172.21.15.32:0/3012170320 conn(0x7fc13c189b00 msgr2=0x7fc13c1921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 shutdown_connections 2021-04-09T18:14:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.397 7fc142b9c700 1 -- 172.21.15.32:0/3012170320 wait complete. 2021-04-09T18:14:40.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:14:40.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f0706f97700 1 Processor -- start 2021-04-09T18:14:40.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f0706f97700 1 -- start start 2021-04-09T18:14:40.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f0706f97700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07000d7ca0 0x7f07000d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f0706f97700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f070010da00 0x7f070010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f0706f97700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070010d480 con 0x7f070010d010 2021-04-09T18:14:40.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f0706f97700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07000d9220 con 0x7f070010da00 2021-04-09T18:14:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f0706f97700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07000d86c0 con 0x7f07000d7ca0 2021-04-09T18:14:40.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f0704db3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f070010d010 0x7f07000d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53054/0 (socket says 172.21.15.32:53054) 2021-04-09T18:14:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f0704db3700 1 -- 172.21.15.32:0/397310795 learned_addr learned my addr 172.21.15.32:0/397310795 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f07055b4700 1 --2- 172.21.15.32:0/397310795 >> v2:172.21.15.86:3300/0 conn(0x7f070010da00 0x7f070010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.670 7f06f7fff700 1 --2- 172.21.15.32:0/397310795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07000d7ca0 0x7f07000d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f77fe700 1 -- 172.21.15.32:0/397310795 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980910214 0 0) 0x7f06fc000ce0 con 0x7f070010d010 2021-04-09T18:14:40.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f77fe700 1 -- 172.21.15.32:0/397310795 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e8002fe0 con 0x7f070010d010 2021-04-09T18:14:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f7fff700 1 -- 172.21.15.32:0/397310795 >> v2:172.21.15.86:3300/0 conn(0x7f070010da00 msgr2=0x7f070010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f7fff700 1 --2- 172.21.15.32:0/397310795 >> v2:172.21.15.86:3300/0 conn(0x7f070010da00 0x7f070010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f7fff700 1 -- 172.21.15.32:0/397310795 >> v1:172.21.15.32:6789/0 conn(0x7f070010d010 legacy=0x7f07000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f7fff700 1 -- 172.21.15.32:0/397310795 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07000d1f70 con 0x7f07000d7ca0 2021-04-09T18:14:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f07055b4700 1 --2- 172.21.15.32:0/397310795 >> v2:172.21.15.86:3300/0 conn(0x7f070010da00 0x7f070010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f7fff700 1 --2- 172.21.15.32:0/397310795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07000d7ca0 0x7f07000d80a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f06f0006650 tx=0x7f06f0009a80).ready entity=mon.2 client_cookie=1b3dfdb518bf9490 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:40.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f77fe700 1 -- 172.21.15.32:0/397310795 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f00009b0 con 0x7f07000d7ca0 2021-04-09T18:14:40.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f77fe700 1 -- 172.21.15.32:0/397310795 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f000c0c0 con 0x7f07000d7ca0 2021-04-09T18:14:40.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f06f77fe700 1 -- 172.21.15.32:0/397310795 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f000c510 con 0x7f07000d7ca0 2021-04-09T18:14:40.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f0706f97700 1 -- 172.21.15.32:0/397310795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07000d7ca0 msgr2=0x7f07000d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.671 7f0706f97700 1 --2- 172.21.15.32:0/397310795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07000d7ca0 0x7f07000d80a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f06f0006650 tx=0x7f06f0009a80).stop 2021-04-09T18:14:40.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.672 7f0706f97700 1 -- 172.21.15.32:0/397310795 shutdown_connections 2021-04-09T18:14:40.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.672 7f0706f97700 1 --2- 172.21.15.32:0/397310795 >> v2:172.21.15.86:3300/0 conn(0x7f070010da00 0x7f070010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.672 7f0706f97700 1 --2- 172.21.15.32:0/397310795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07000d7ca0 0x7f07000d80a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.672 7f0706f97700 1 -- 172.21.15.32:0/397310795 >> 172.21.15.32:0/397310795 conn(0x7f07000ce5f0 msgr2=0x7f07000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:40.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.672 7f0706f97700 1 -- 172.21.15.32:0/397310795 shutdown_connections 2021-04-09T18:14:40.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.672 7f0706f97700 1 -- 172.21.15.32:0/397310795 wait complete. 2021-04-09T18:14:40.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.692 7f0706f97700 1 Processor -- start 2021-04-09T18:14:40.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.692 7f0706f97700 1 -- start start 2021-04-09T18:14:40.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f0706f97700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07000a90c0 0x7f07001a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f0706f97700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07001a76d0 0x7f07001a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f0706f97700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f070017bce0 0x7f070017c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f0706f97700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07001a7c60 con 0x7f070017bce0 2021-04-09T18:14:40.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f0706f97700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07001a8e20 con 0x7f07000a90c0 2021-04-09T18:14:40.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f0706f97700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070017c700 con 0x7f07001a76d0 2021-04-09T18:14:40.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f07055b4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f070017bce0 0x7f070017c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f07055b4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f070017bce0 0x7f070017c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43472/0 (socket says 172.21.15.32:43472) 2021-04-09T18:14:40.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f07055b4700 1 -- 172.21.15.32:0/3102406509 learned_addr learned my addr 172.21.15.32:0/3102406509 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:40.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f0704db3700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07000a90c0 0x7f07001a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f06f7fff700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07001a76d0 0x7f07001a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f07055b4700 1 -- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07001a76d0 msgr2=0x7f07001a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f07055b4700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07001a76d0 0x7f07001a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f07055b4700 1 -- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07000a90c0 msgr2=0x7f07001a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f07055b4700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07000a90c0 0x7f07001a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f07055b4700 1 -- 172.21.15.32:0/3102406509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070017c920 con 0x7f070017bce0 2021-04-09T18:14:40.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.693 7f07055b4700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f070017bce0 0x7f070017c0e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f06f800b970 tx=0x7f06f800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:40.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.694 7f06f57fa700 1 -- 172.21.15.32:0/3102406509 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f8000f90 con 0x7f070017bce0 2021-04-09T18:14:40.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.694 7f06f57fa700 1 -- 172.21.15.32:0/3102406509 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f8006aa0 con 0x7f070017bce0 2021-04-09T18:14:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.694 7f06f57fa700 1 -- 172.21.15.32:0/3102406509 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f80015f0 con 0x7f070017bce0 2021-04-09T18:14:40.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.694 7f0706f97700 1 -- 172.21.15.32:0/3102406509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070017c920 con 0x7f070017bce0 2021-04-09T18:14:40.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.694 7f0706f97700 1 -- 172.21.15.32:0/3102406509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070017c920 con 0x7f070017bce0 2021-04-09T18:14:40.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.694 7f06f57fa700 1 -- 172.21.15.32:0/3102406509 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f06f8000e70 con 0x7f070017bce0 2021-04-09T18:14:40.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.694 7f06f57fa700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06e00188c0 0x7f06e001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:40.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.694 7f06f57fa700 1 -- 172.21.15.32:0/3102406509 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f06f8032b30 con 0x7f070017bce0 2021-04-09T18:14:40.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.695 7f0704db3700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06e00188c0 0x7f06e001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:40.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.695 7f0704db3700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06e00188c0 0x7f06e001adc0 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:40.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.714 7f0706f97700 1 -- 172.21.15.32:0/3102406509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07001b7010 con 0x7f070017bce0 2021-04-09T18:14:40.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.717 7f06f57fa700 1 -- 172.21.15.32:0/3102406509 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f06f80066f0 con 0x7f070017bce0 2021-04-09T18:14:40.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.905 7f0706f97700 1 -- 172.21.15.32:0/3102406509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f070010d480 con 0x7f070017bce0 2021-04-09T18:14:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.906 7f06f57fa700 1 -- 172.21.15.32:0/3102406509 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f06f8000e50 con 0x7f070017bce0 2021-04-09T18:14:40.932 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:40.932 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":42,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:14:39.918794","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:14:38.344198","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-04-09 18:14:38.803133","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:14:40.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.908 7f0706f97700 1 -- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06e00188c0 msgr2=0x7f06e001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.908 7f0706f97700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06e00188c0 0x7f06e001adc0 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.908 7f0706f97700 1 -- 172.21.15.32:0/3102406509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f070017bce0 msgr2=0x7f070017c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.908 7f0706f97700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f070017bce0 0x7f070017c0e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f06f800b970 tx=0x7f06f800bd10).stop 2021-04-09T18:14:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.909 7f0706f97700 1 -- 172.21.15.32:0/3102406509 shutdown_connections 2021-04-09T18:14:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.909 7f0706f97700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06e00188c0 0x7f06e001adc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.909 7f0706f97700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f070017bce0 0x7f070017c0e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.909 7f0706f97700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07001a76d0 0x7f07001a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.909 7f0706f97700 1 --2- 172.21.15.32:0/3102406509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07000a90c0 0x7f07001a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.909 7f0706f97700 1 -- 172.21.15.32:0/3102406509 >> 172.21.15.32:0/3102406509 conn(0x7f07001acae0 msgr2=0x7f070017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:40.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.909 7f0706f97700 1 -- 172.21.15.32:0/3102406509 shutdown_connections 2021-04-09T18:14:40.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:40.909 7f0706f97700 1 -- 172.21.15.32:0/3102406509 wait complete. 2021-04-09T18:14:40.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:14:41.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.093 7f35f757a700 1 Processor -- start 2021-04-09T18:14:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.093 7f35f757a700 1 -- start start 2021-04-09T18:14:41.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.094 7f35f757a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f010d010 0x7f35f00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:41.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.094 7f35f757a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f010d480 con 0x7f35f00d1da0 2021-04-09T18:14:41.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.094 7f35f757a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f010e1f0 con 0x7f35f00d8490 2021-04-09T18:14:41.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.094 7f35f757a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f010e780 con 0x7f35f010d010 2021-04-09T18:14:41.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.094 7f35f4b95700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f35f00d1da0 0x7f35f00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53068/0 (socket says 172.21.15.32:53068) 2021-04-09T18:14:41.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.094 7f35f4b95700 1 -- 172.21.15.32:0/1834519003 learned_addr learned my addr 172.21.15.32:0/1834519003 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:41.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.094 7f35f5396700 1 --2- 172.21.15.32:0/1834519003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f010d010 0x7f35f00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:41.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.094 7f35e7fff700 1 -- 172.21.15.32:0/1834519003 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454676270 0 0) 0x7f35ec000ce0 con 0x7f35f00d1da0 2021-04-09T18:14:41.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35e7fff700 1 -- 172.21.15.32:0/1834519003 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d8002fe0 con 0x7f35f00d1da0 2021-04-09T18:14:41.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35e7fff700 1 -- 172.21.15.32:0/1834519003 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922046398 0 0) 0x7f35e0001080 con 0x7f35f00d8490 2021-04-09T18:14:41.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35e7fff700 1 -- 172.21.15.32:0/1834519003 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d8003210 con 0x7f35f00d8490 2021-04-09T18:14:41.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35e7fff700 1 -- 172.21.15.32:0/1834519003 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3938788374 0 0) 0x7f35ec0010e0 con 0x7f35f00d1da0 2021-04-09T18:14:41.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35f5396700 1 -- 172.21.15.32:0/1834519003 >> v1:172.21.15.86:6789/0 conn(0x7f35f00d8490 legacy=0x7f35f010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:41.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35f5396700 1 -- 172.21.15.32:0/1834519003 >> v1:172.21.15.32:6789/0 conn(0x7f35f00d1da0 legacy=0x7f35f00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:41.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35f5396700 1 -- 172.21.15.32:0/1834519003 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00d8d30 con 0x7f35f010d010 2021-04-09T18:14:41.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35f5396700 1 --2- 172.21.15.32:0/1834519003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f010d010 0x7f35f00d1780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f35e80069d0 tx=0x7f35e8009dc0).ready entity=mon.2 client_cookie=b5274fa37b2076c5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:41.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35e7fff700 1 -- 172.21.15.32:0/1834519003 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e800b510 con 0x7f35f010d010 2021-04-09T18:14:41.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35e7fff700 1 -- 172.21.15.32:0/1834519003 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e8006ac0 con 0x7f35f010d010 2021-04-09T18:14:41.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35e7fff700 1 -- 172.21.15.32:0/1834519003 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e800ba80 con 0x7f35f010d010 2021-04-09T18:14:41.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35f757a700 1 -- 172.21.15.32:0/1834519003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f010d010 msgr2=0x7f35f00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:41.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.095 7f35f757a700 1 --2- 172.21.15.32:0/1834519003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f010d010 0x7f35f00d1780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f35e80069d0 tx=0x7f35e8009dc0).stop 2021-04-09T18:14:41.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.096 7f35f757a700 1 -- 172.21.15.32:0/1834519003 shutdown_connections 2021-04-09T18:14:41.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.096 7f35f757a700 1 --2- 172.21.15.32:0/1834519003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f010d010 0x7f35f00d1780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:41.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.096 7f35f757a700 1 -- 172.21.15.32:0/1834519003 >> 172.21.15.32:0/1834519003 conn(0x7f35f00ce5f0 msgr2=0x7f35f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:41.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.096 7f35f757a700 1 -- 172.21.15.32:0/1834519003 shutdown_connections 2021-04-09T18:14:41.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.096 7f35f757a700 1 -- 172.21.15.32:0/1834519003 wait complete. 2021-04-09T18:14:41.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.116 7f35f757a700 1 Processor -- start 2021-04-09T18:14:41.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.116 7f35f757a700 1 -- start start 2021-04-09T18:14:41.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f757a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f00a0a10 0x7f35f00bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:41.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f757a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f018ad00 0x7f35f019fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:41.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f757a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f01a03f0 0x7f35f0095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:41.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f757a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f018b100 con 0x7f35f018ad00 2021-04-09T18:14:41.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f757a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0095860 con 0x7f35f01a03f0 2021-04-09T18:14:41.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f757a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0095a80 con 0x7f35f00a0a10 2021-04-09T18:14:41.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f4b95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f018ad00 0x7f35f019fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:41.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f5b97700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f01a03f0 0x7f35f0095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:41.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f4b95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f018ad00 0x7f35f019fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43486/0 (socket says 172.21.15.32:43486) 2021-04-09T18:14:41.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f5396700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f00a0a10 0x7f35f00bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:41.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.117 7f35f4b95700 1 -- 172.21.15.32:0/1672187334 learned_addr learned my addr 172.21.15.32:0/1672187334 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:41.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35f4b95700 1 -- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f00a0a10 msgr2=0x7f35f00bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:41.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35f4b95700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f00a0a10 0x7f35f00bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35f4b95700 1 -- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f01a03f0 msgr2=0x7f35f0095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35f4b95700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f01a03f0 0x7f35f0095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:41.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35f4b95700 1 -- 172.21.15.32:0/1672187334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0189250 con 0x7f35f018ad00 2021-04-09T18:14:41.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35f5b97700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f01a03f0 0x7f35f0095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35f4b95700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f018ad00 0x7f35f019fdd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f35ec00a540 tx=0x7f35ec00a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:41.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35e5ffb700 1 -- 172.21.15.32:0/1672187334 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35ec00acd0 con 0x7f35f018ad00 2021-04-09T18:14:41.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35e5ffb700 1 -- 172.21.15.32:0/1672187334 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ec001af0 con 0x7f35f018ad00 2021-04-09T18:14:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35f757a700 1 -- 172.21.15.32:0/1672187334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f0189250 con 0x7f35f018ad00 2021-04-09T18:14:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35e5ffb700 1 -- 172.21.15.32:0/1672187334 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35ec0047a0 con 0x7f35f018ad00 2021-04-09T18:14:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.118 7f35f757a700 1 -- 172.21.15.32:0/1672187334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f0189250 con 0x7f35f018ad00 2021-04-09T18:14:41.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.119 7f35e5ffb700 1 -- 172.21.15.32:0/1672187334 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f35ec0043a0 con 0x7f35f018ad00 2021-04-09T18:14:41.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.119 7f35e5ffb700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35d0018930 0x7f35d001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:41.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.119 7f35e5ffb700 1 -- 172.21.15.32:0/1672187334 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f35ec032a10 con 0x7f35f018ad00 2021-04-09T18:14:41.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.119 7f35f5396700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35d0018930 0x7f35d001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:41.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.119 7f35f5396700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35d0018930 0x7f35d001ae10 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.138 7f35f757a700 1 -- 172.21.15.32:0/1672187334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f0093d50 con 0x7f35f018ad00 2021-04-09T18:14:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.142 7f35e5ffb700 1 -- 172.21.15.32:0/1672187334 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f35ec001af0 con 0x7f35f018ad00 2021-04-09T18:14:41.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.333 7f35f757a700 1 -- 172.21.15.32:0/1672187334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f35f010d480 con 0x7f35f018ad00 2021-04-09T18:14:41.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.929 7f35e5ffb700 1 -- 172.21.15.32:0/1672187334 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v28) v1 ==== 118+0+0 (secure 0 0 0) 0x7f35ec0043a0 con 0x7f35f018ad00 2021-04-09T18:14:41.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 -- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35d0018930 msgr2=0x7f35d001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:41.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35d0018930 0x7f35d001ae10 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:41.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 -- 172.21.15.32:0/1672187334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f018ad00 msgr2=0x7f35f019fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:41.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f018ad00 0x7f35f019fdd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f35ec00a540 tx=0x7f35ec00a8e0).stop 2021-04-09T18:14:41.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 -- 172.21.15.32:0/1672187334 shutdown_connections 2021-04-09T18:14:41.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35d0018930 0x7f35d001ae10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:41.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35f018ad00 0x7f35f019fdd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:41.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f01a03f0 0x7f35f0095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 --2- 172.21.15.32:0/1672187334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f00a0a10 0x7f35f00bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 -- 172.21.15.32:0/1672187334 >> 172.21.15.32:0/1672187334 conn(0x7f35f00975f0 msgr2=0x7f35f0096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 -- 172.21.15.32:0/1672187334 shutdown_connections 2021-04-09T18:14:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:41.950 7f35f757a700 1 -- 172.21.15.32:0/1672187334 wait complete. 2021-04-09T18:14:42.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:42.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.143 7f073083b700 1 Processor -- start 2021-04-09T18:14:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.143 7f073083b700 1 -- start start 2021-04-09T18:14:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f073083b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f072c0d7ca0 0x7f072c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f073083b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f072c10da00 0x7f072c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f073083b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072c10d480 con 0x7f072c10d010 2021-04-09T18:14:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f073083b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072c0d9220 con 0x7f072c0d7ca0 2021-04-09T18:14:42.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f073083b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072c0d86c0 con 0x7f072c10da00 2021-04-09T18:14:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f0729e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f072c0d7ca0 0x7f072c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f072a61c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f072c10d010 0x7f072c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53082/0 (socket says 172.21.15.32:53082) 2021-04-09T18:14:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f072a61c700 1 -- 172.21.15.32:0/3207336355 learned_addr learned my addr 172.21.15.32:0/3207336355 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f072ae1d700 1 --2- 172.21.15.32:0/3207336355 >> v2:172.21.15.86:3300/0 conn(0x7f072c10da00 0x7f072c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f072961a700 1 -- 172.21.15.32:0/3207336355 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325213052 0 0) 0x7f071c000ce0 con 0x7f072c10d010 2021-04-09T18:14:42.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.144 7f072961a700 1 -- 172.21.15.32:0/3207336355 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0714002fe0 con 0x7f072c10d010 2021-04-09T18:14:42.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f0729e1b700 1 -- 172.21.15.32:0/3207336355 >> v2:172.21.15.86:3300/0 conn(0x7f072c10da00 msgr2=0x7f072c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f0729e1b700 1 --2- 172.21.15.32:0/3207336355 >> v2:172.21.15.86:3300/0 conn(0x7f072c10da00 0x7f072c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f0729e1b700 1 -- 172.21.15.32:0/3207336355 >> v1:172.21.15.32:6789/0 conn(0x7f072c10d010 legacy=0x7f072c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f0729e1b700 1 -- 172.21.15.32:0/3207336355 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c0d1f70 con 0x7f072c0d7ca0 2021-04-09T18:14:42.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f0729e1b700 1 --2- 172.21.15.32:0/3207336355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c0d7ca0 0x7f072c0d80a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f0720006650 tx=0x7f0720009a80).ready entity=mon.0 client_cookie=4a0090d8ae49fa96 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f072961a700 1 -- 172.21.15.32:0/3207336355 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07200009b0 con 0x7f072c0d7ca0 2021-04-09T18:14:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f072961a700 1 -- 172.21.15.32:0/3207336355 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0720006740 con 0x7f072c0d7ca0 2021-04-09T18:14:42.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f072961a700 1 -- 172.21.15.32:0/3207336355 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f072000c0c0 con 0x7f072c0d7ca0 2021-04-09T18:14:42.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f073083b700 1 -- 172.21.15.32:0/3207336355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c0d7ca0 msgr2=0x7f072c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.145 7f073083b700 1 --2- 172.21.15.32:0/3207336355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c0d7ca0 0x7f072c0d80a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f0720006650 tx=0x7f0720009a80).stop 2021-04-09T18:14:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.146 7f073083b700 1 -- 172.21.15.32:0/3207336355 shutdown_connections 2021-04-09T18:14:42.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.146 7f073083b700 1 --2- 172.21.15.32:0/3207336355 >> v2:172.21.15.86:3300/0 conn(0x7f072c10da00 0x7f072c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.146 7f073083b700 1 --2- 172.21.15.32:0/3207336355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c0d7ca0 0x7f072c0d80a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.146 7f073083b700 1 -- 172.21.15.32:0/3207336355 >> 172.21.15.32:0/3207336355 conn(0x7f072c0ce5f0 msgr2=0x7f072c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:42.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.146 7f073083b700 1 -- 172.21.15.32:0/3207336355 shutdown_connections 2021-04-09T18:14:42.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.146 7f073083b700 1 -- 172.21.15.32:0/3207336355 wait complete. 2021-04-09T18:14:42.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.166 7f073083b700 1 Processor -- start 2021-04-09T18:14:42.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.166 7f073083b700 1 -- start start 2021-04-09T18:14:42.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f073083b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072c0b5360 0x7f072c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f073083b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c18e140 0x7f072c1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f073083b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f072c1ad040 0x7f072c08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f073083b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c1a5360 con 0x7f072c18e140 2021-04-09T18:14:42.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f073083b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c0902c0 con 0x7f072c0b5360 2021-04-09T18:14:42.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f073083b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c0904e0 con 0x7f072c1ad040 2021-04-09T18:14:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f0729e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c18e140 0x7f072c1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f0729e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c18e140 0x7f072c1aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43502/0 (socket says 172.21.15.32:43502) 2021-04-09T18:14:42.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f0729e1b700 1 -- 172.21.15.32:0/1517551005 learned_addr learned my addr 172.21.15.32:0/1517551005 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:42.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f072a61c700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072c0b5360 0x7f072c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f072ae1d700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f072c1ad040 0x7f072c08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f0729e1b700 1 -- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f072c1ad040 msgr2=0x7f072c08fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f0729e1b700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f072c1ad040 0x7f072c08fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f0729e1b700 1 -- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072c0b5360 msgr2=0x7f072c1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f0729e1b700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072c0b5360 0x7f072c1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f0729e1b700 1 -- 172.21.15.32:0/1517551005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c183c80 con 0x7f072c18e140 2021-04-09T18:14:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f0729e1b700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c18e140 0x7f072c1aca20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f0720009650 tx=0x7f072000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f071affd700 1 -- 172.21.15.32:0/1517551005 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07200093a0 con 0x7f072c18e140 2021-04-09T18:14:42.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f071affd700 1 -- 172.21.15.32:0/1517551005 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072000bd50 con 0x7f072c18e140 2021-04-09T18:14:42.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f073083b700 1 -- 172.21.15.32:0/1517551005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c183c80 con 0x7f072c18e140 2021-04-09T18:14:42.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f071affd700 1 -- 172.21.15.32:0/1517551005 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07200052e0 con 0x7f072c18e140 2021-04-09T18:14:42.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.167 7f073083b700 1 -- 172.21.15.32:0/1517551005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c183c80 con 0x7f072c18e140 2021-04-09T18:14:42.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.168 7f071affd700 1 -- 172.21.15.32:0/1517551005 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0720006030 con 0x7f072c18e140 2021-04-09T18:14:42.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.168 7f071affd700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0710018930 0x7f071001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.168 7f071affd700 1 -- 172.21.15.32:0/1517551005 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0720032b20 con 0x7f072c18e140 2021-04-09T18:14:42.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.168 7f072a61c700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0710018930 0x7f071001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.169 7f072a61c700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0710018930 0x7f071001adc0 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:42.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.188 7f073083b700 1 -- 172.21.15.32:0/1517551005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072c182d70 con 0x7f072c18e140 2021-04-09T18:14:42.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.192 7f071affd700 1 -- 172.21.15.32:0/1517551005 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f07200009c0 con 0x7f072c18e140 2021-04-09T18:14:42.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.381 7f073083b700 1 -- 172.21.15.32:0/1517551005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f072c10d480 con 0x7f072c18e140 2021-04-09T18:14:42.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.381 7f071affd700 1 -- 172.21.15.32:0/1517551005 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+2773 (secure 0 0 0) 0x7f072000bd50 con 0x7f072c18e140 2021-04-09T18:14:42.406 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2021-04-09T18:14:42.407 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:42.407 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:41.927797","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:14:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 -- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0710018930 msgr2=0x7f071001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0710018930 0x7f071001adc0 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 -- 172.21.15.32:0/1517551005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c18e140 msgr2=0x7f072c1aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c18e140 0x7f072c1aca20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f0720009650 tx=0x7f072000b960).stop 2021-04-09T18:14:42.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 -- 172.21.15.32:0/1517551005 shutdown_connections 2021-04-09T18:14:42.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0710018930 0x7f071001adc0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f072c18e140 0x7f072c1aca20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f072c1ad040 0x7f072c08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 --2- 172.21.15.32:0/1517551005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072c0b5360 0x7f072c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 -- 172.21.15.32:0/1517551005 >> 172.21.15.32:0/1517551005 conn(0x7f072c19b460 msgr2=0x7f072c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 -- 172.21.15.32:0/1517551005 shutdown_connections 2021-04-09T18:14:42.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.384 7f073083b700 1 -- 172.21.15.32:0/1517551005 wait complete. 2021-04-09T18:14:42.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:14:42.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.545 7efdd1aa0700 1 Processor -- start 2021-04-09T18:14:42.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.545 7efdd1aa0700 1 -- start start 2021-04-09T18:14:42.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.545 7efdd1aa0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc10d010 0x7efdcc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.545 7efdd1aa0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efdcc0d8490 0x7efdcc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.545 7efdd1aa0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdcc10d480 con 0x7efdcc0d1da0 2021-04-09T18:14:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.545 7efdd1aa0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdcc10e780 con 0x7efdcc0d8490 2021-04-09T18:14:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.545 7efdd1aa0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0d8d30 con 0x7efdcc10d010 2021-04-09T18:14:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.545 7efdcbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efdcc0d8490 0x7efdcc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdcaffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efdcc0d1da0 0x7efdcc0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53098/0 (socket says 172.21.15.32:53098) 2021-04-09T18:14:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdcaffd700 1 -- 172.21.15.32:0/1541967535 learned_addr learned my addr 172.21.15.32:0/1541967535 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:42.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdcb7fe700 1 --2- 172.21.15.32:0/1541967535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc10d010 0x7efdcc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdcbfff700 1 -- 172.21.15.32:0/1541967535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc10d010 msgr2=0x7efdcc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:42.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdcbfff700 1 --2- 172.21.15.32:0/1541967535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc10d010 0x7efdcc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdcbfff700 1 -- 172.21.15.32:0/1541967535 >> v1:172.21.15.32:6789/0 conn(0x7efdcc0d1da0 legacy=0x7efdcc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdca7fc700 1 -- 172.21.15.32:0/1541967535 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424510018 0 0) 0x7efdc0000ce0 con 0x7efdcc0d1da0 2021-04-09T18:14:42.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdcbfff700 1 -- 172.21.15.32:0/1541967535 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc0d9220 con 0x7efdcc0d8490 2021-04-09T18:14:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdcbfff700 1 --2- 172.21.15.32:0/1541967535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc0d8490 0x7efdcc10da00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7efdc4006e30 tx=0x7efdc4009de0).ready entity=mon.0 client_cookie=64c79907c502c249 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:42.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdca7fc700 1 -- 172.21.15.32:0/1541967535 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdc400b300 con 0x7efdcc0d8490 2021-04-09T18:14:42.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdca7fc700 1 -- 172.21.15.32:0/1541967535 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc400b680 con 0x7efdcc0d8490 2021-04-09T18:14:42.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdca7fc700 1 -- 172.21.15.32:0/1541967535 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdc400bb60 con 0x7efdcc0d8490 2021-04-09T18:14:42.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdd1aa0700 1 -- 172.21.15.32:0/1541967535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc0d8490 msgr2=0x7efdcc10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.546 7efdd1aa0700 1 --2- 172.21.15.32:0/1541967535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc0d8490 0x7efdcc10da00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7efdc4006e30 tx=0x7efdc4009de0).stop 2021-04-09T18:14:42.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.547 7efdd1aa0700 1 -- 172.21.15.32:0/1541967535 shutdown_connections 2021-04-09T18:14:42.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.547 7efdd1aa0700 1 --2- 172.21.15.32:0/1541967535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc0d8490 0x7efdcc10da00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.547 7efdd1aa0700 1 --2- 172.21.15.32:0/1541967535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc10d010 0x7efdcc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.547 7efdd1aa0700 1 -- 172.21.15.32:0/1541967535 >> 172.21.15.32:0/1541967535 conn(0x7efdcc0ce5f0 msgr2=0x7efdcc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:42.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.547 7efdd1aa0700 1 -- 172.21.15.32:0/1541967535 shutdown_connections 2021-04-09T18:14:42.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.547 7efdd1aa0700 1 -- 172.21.15.32:0/1541967535 wait complete. 2021-04-09T18:14:42.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.567 7efdd1aa0700 1 Processor -- start 2021-04-09T18:14:42.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.567 7efdd1aa0700 1 -- start start 2021-04-09T18:14:42.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdd1aa0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc1a5210 0x7efdcc1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdd1aa0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc1a6340 0x7efdcc1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdd1aa0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdcc0abaa0 0x7efdcc0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:42.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdd1aa0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc1a6d60 con 0x7efdcc1a5210 2021-04-09T18:14:42.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdd1aa0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0ac4c0 con 0x7efdcc0abaa0 2021-04-09T18:14:42.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdd1aa0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0ac6e0 con 0x7efdcc1a6340 2021-04-09T18:14:42.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc1a5210 0x7efdcc1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc1a5210 0x7efdcc1a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43512/0 (socket says 172.21.15.32:43512) 2021-04-09T18:14:42.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcb7fe700 1 -- 172.21.15.32:0/2882913112 learned_addr learned my addr 172.21.15.32:0/2882913112 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:42.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcbfff700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdcc0abaa0 0x7efdcc0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcaffd700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc1a6340 0x7efdcc1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:42.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcb7fe700 1 -- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc1a6340 msgr2=0x7efdcc1a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcb7fe700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc1a6340 0x7efdcc1a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcb7fe700 1 -- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdcc0abaa0 msgr2=0x7efdcc0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:42.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcb7fe700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdcc0abaa0 0x7efdcc0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:42.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcb7fe700 1 -- 172.21.15.32:0/2882913112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc17a720 con 0x7efdcc1a5210 2021-04-09T18:14:42.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.568 7efdcb7fe700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc1a5210 0x7efdcc1a56c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7efdbc006f60 tx=0x7efdbc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:43.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.569 7efdb3fff700 1 -- 172.21.15.32:0/2882913112 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdbc00bd60 con 0x7efdcc1a5210 2021-04-09T18:14:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.569 7efdb3fff700 1 -- 172.21.15.32:0/2882913112 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc000c90 con 0x7efdcc1a5210 2021-04-09T18:14:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.569 7efdb3fff700 1 -- 172.21.15.32:0/2882913112 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdbc002220 con 0x7efdcc1a5210 2021-04-09T18:14:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.569 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc17a720 con 0x7efdcc1a5210 2021-04-09T18:14:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.569 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc17a720 con 0x7efdcc1a5210 2021-04-09T18:14:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.569 7efdb3fff700 1 -- 172.21.15.32:0/2882913112 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efdbc0052e0 con 0x7efdcc1a5210 2021-04-09T18:14:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.569 7efdb3fff700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efdac0188d0 0x7efdac01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.569 7efdb3fff700 1 -- 172.21.15.32:0/2882913112 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efdbc0329f0 con 0x7efdcc1a5210 2021-04-09T18:14:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.570 7efdcaffd700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efdac0188d0 0x7efdac01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:43.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.570 7efdcaffd700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efdac0188d0 0x7efdac01add0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:43.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.589 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc1a6d60 con 0x7efdcc1a5210 2021-04-09T18:14:43.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.593 7efdb3fff700 1 -- 172.21.15.32:0/2882913112 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efdbc000c90 con 0x7efdcc1a5210 2021-04-09T18:14:43.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.782 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdcc1a6d60 con 0x7efdcc1a5210 2021-04-09T18:14:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.783 7efdb3fff700 1 -- 172.21.15.32:0/2882913112 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+13993 (secure 0 0 0) 0x7efdbc006b00 con 0x7efdcc1a5210 2021-04-09T18:14:43.091 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:43.092 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:14:40.920336","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:14:38.344198","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-04-09 18:14:38.803133","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:14:43.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efdac0188d0 msgr2=0x7efdac01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efdac0188d0 0x7efdac01add0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc1a5210 msgr2=0x7efdcc1a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:43.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc1a5210 0x7efdcc1a56c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7efdbc006f60 tx=0x7efdbc00b970).stop 2021-04-09T18:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 shutdown_connections 2021-04-09T18:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efdac0188d0 0x7efdac01add0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdcc0abaa0 0x7efdcc0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efdcc1a6340 0x7efdcc1a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 --2- 172.21.15.32:0/2882913112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efdcc1a5210 0x7efdcc1a56c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 >> 172.21.15.32:0/2882913112 conn(0x7efdcc1a99c0 msgr2=0x7efdcc17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 shutdown_connections 2021-04-09T18:14:43.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:42.785 7efdd1aa0700 1 -- 172.21.15.32:0/2882913112 wait complete. 2021-04-09T18:14:43.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:14:43.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.265 7f5a2a6d8700 1 Processor -- start 2021-04-09T18:14:43.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.265 7f5a2a6d8700 1 -- start start 2021-04-09T18:14:43.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.265 7f5a2a6d8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a240cffb0 0x7f5a24087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:43.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.265 7f5a2a6d8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a240d7c50 0x7f5a241b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:43.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.265 7f5a2a6d8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5a240d8050 0x7f5a2410d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:43.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.265 7f5a2a6d8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a240d91f0 con 0x7f5a240cffb0 2021-04-09T18:14:43.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.265 7f5a2a6d8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a24088460 con 0x7f5a240d8050 2021-04-09T18:14:43.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.265 7f5a2a6d8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240d8620 con 0x7f5a240d7c50 2021-04-09T18:14:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a23fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a240cffb0 0x7f5a24087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:43.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a23fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a240cffb0 0x7f5a24087e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43520/0 (socket says 172.21.15.32:43520) 2021-04-09T18:14:43.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a23fff700 1 -- 172.21.15.32:0/1901748946 learned_addr learned my addr 172.21.15.32:0/1901748946 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:43.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a28cf5700 1 --2- 172.21.15.32:0/1901748946 >> v2:172.21.15.86:3300/0 conn(0x7f5a240d8050 0x7f5a2410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:43.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a1bfff700 1 --2- 172.21.15.32:0/1901748946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a240d7c50 0x7f5a241b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:43.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a23fff700 1 -- 172.21.15.32:0/1901748946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a240d7c50 msgr2=0x7f5a241b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a23fff700 1 --2- 172.21.15.32:0/1901748946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a240d7c50 0x7f5a241b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a23fff700 1 -- 172.21.15.32:0/1901748946 >> v2:172.21.15.86:3300/0 conn(0x7f5a240d8050 msgr2=0x7f5a2410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:43.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a23fff700 1 --2- 172.21.15.32:0/1901748946 >> v2:172.21.15.86:3300/0 conn(0x7f5a240d8050 0x7f5a2410d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a23fff700 1 -- 172.21.15.32:0/1901748946 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240d2800 con 0x7f5a240cffb0 2021-04-09T18:14:43.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a23fff700 1 --2- 172.21.15.32:0/1901748946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a240cffb0 0x7f5a24087e40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5a14006700 tx=0x7f5a14009b00).ready entity=mon.0 client_cookie=21aa362254e7b1a8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:43.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a237fe700 1 -- 172.21.15.32:0/1901748946 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a1400c0c0 con 0x7f5a240cffb0 2021-04-09T18:14:43.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a237fe700 1 -- 172.21.15.32:0/1901748946 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1400c400 con 0x7f5a240cffb0 2021-04-09T18:14:43.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a237fe700 1 -- 172.21.15.32:0/1901748946 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a1400c8e0 con 0x7f5a240cffb0 2021-04-09T18:14:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a2a6d8700 1 -- 172.21.15.32:0/1901748946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a240cffb0 msgr2=0x7f5a24087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:43.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.266 7f5a2a6d8700 1 --2- 172.21.15.32:0/1901748946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a240cffb0 0x7f5a24087e40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5a14006700 tx=0x7f5a14009b00).stop 2021-04-09T18:14:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.267 7f5a2a6d8700 1 -- 172.21.15.32:0/1901748946 shutdown_connections 2021-04-09T18:14:43.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.267 7f5a2a6d8700 1 --2- 172.21.15.32:0/1901748946 >> v2:172.21.15.86:3300/0 conn(0x7f5a240d8050 0x7f5a2410d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.267 7f5a2a6d8700 1 --2- 172.21.15.32:0/1901748946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a240d7c50 0x7f5a241b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.267 7f5a2a6d8700 1 --2- 172.21.15.32:0/1901748946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a240cffb0 0x7f5a24087e40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.267 7f5a2a6d8700 1 -- 172.21.15.32:0/1901748946 >> 172.21.15.32:0/1901748946 conn(0x7f5a240ce5b0 msgr2=0x7f5a240cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:43.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.267 7f5a2a6d8700 1 -- 172.21.15.32:0/1901748946 shutdown_connections 2021-04-09T18:14:43.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.267 7f5a2a6d8700 1 -- 172.21.15.32:0/1901748946 wait complete. 2021-04-09T18:14:43.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.287 7f5a2a6d8700 1 Processor -- start 2021-04-09T18:14:43.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.287 7f5a2a6d8700 1 -- start start 2021-04-09T18:14:43.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a2a6d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a241a0840 0x7f5a24191ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:43.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a2a6d8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a24192100 0x7f5a240babe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:43.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a2a6d8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a240bb200 0x7f5a240c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:43.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a2a6d8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240ba660 con 0x7f5a241a0840 2021-04-09T18:14:43.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a2a6d8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a24192500 con 0x7f5a240bb200 2021-04-09T18:14:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a2a6d8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240bb600 con 0x7f5a24192100 2021-04-09T18:14:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a23fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a241a0840 0x7f5a24191ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a23fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a241a0840 0x7f5a24191ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43524/0 (socket says 172.21.15.32:43524) 2021-04-09T18:14:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a23fff700 1 -- 172.21.15.32:0/1165574481 learned_addr learned my addr 172.21.15.32:0/1165574481 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:43.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a1bfff700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a24192100 0x7f5a240babe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:43.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a28cf5700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a240bb200 0x7f5a240c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:43.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a23fff700 1 -- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a24192100 msgr2=0x7f5a240babe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:43.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a23fff700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a24192100 0x7f5a240babe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a23fff700 1 -- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a240bb200 msgr2=0x7f5a240c0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:43.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a23fff700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a240bb200 0x7f5a240c0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a23fff700 1 -- 172.21.15.32:0/1165574481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240c11a0 con 0x7f5a241a0840 2021-04-09T18:14:43.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.288 7f5a28cf5700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a240bb200 0x7f5a240c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:14:43.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.289 7f5a23fff700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a241a0840 0x7f5a24191ae0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f5a1400b590 tx=0x7f5a1400b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:43.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.289 7f5a217fa700 1 -- 172.21.15.32:0/1165574481 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a14006390 con 0x7f5a241a0840 2021-04-09T18:14:43.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.289 7f5a217fa700 1 -- 172.21.15.32:0/1165574481 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1400be20 con 0x7f5a241a0840 2021-04-09T18:14:43.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.289 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a240c11a0 con 0x7f5a241a0840 2021-04-09T18:14:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.289 7f5a217fa700 1 -- 172.21.15.32:0/1165574481 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a14002030 con 0x7f5a241a0840 2021-04-09T18:14:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.289 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a240c11a0 con 0x7f5a241a0840 2021-04-09T18:14:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.289 7f5a217fa700 1 -- 172.21.15.32:0/1165574481 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5a140093e0 con 0x7f5a241a0840 2021-04-09T18:14:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.289 7f5a217fa700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a08018930 0x7f5a0801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:43.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.289 7f5a217fa700 1 -- 172.21.15.32:0/1165574481 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5a14032a10 con 0x7f5a241a0840 2021-04-09T18:14:43.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.290 7f5a1bfff700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a08018930 0x7f5a0801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:43.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.290 7f5a1bfff700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a08018930 0x7f5a0801adc0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:43.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.309 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a240bb600 con 0x7f5a241a0840 2021-04-09T18:14:43.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.313 7f5a217fa700 1 -- 172.21.15.32:0/1165574481 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5a140090c0 con 0x7f5a241a0840 2021-04-09T18:14:43.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.500 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5a240bb600 con 0x7f5a241a0840 2021-04-09T18:14:43.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.503 7f5a217fa700 1 -- 172.21.15.32:0/1165574481 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5a1401f750 con 0x7f5a241a0840 2021-04-09T18:14:43.792 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:14:43.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a08018930 msgr2=0x7f5a0801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:43.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a08018930 0x7f5a0801adc0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a241a0840 msgr2=0x7f5a24191ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:43.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a241a0840 0x7f5a24191ae0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f5a1400b590 tx=0x7f5a1400b930).stop 2021-04-09T18:14:43.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 shutdown_connections 2021-04-09T18:14:43.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a08018930 0x7f5a0801adc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a240bb200 0x7f5a240c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a24192100 0x7f5a240babe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 --2- 172.21.15.32:0/1165574481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a241a0840 0x7f5a24191ae0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:43.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 >> 172.21.15.32:0/1165574481 conn(0x7f5a240adaf0 msgr2=0x7f5a240bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:43.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 shutdown_connections 2021-04-09T18:14:43.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.509 7f5a2a6d8700 1 -- 172.21.15.32:0/1165574481 wait complete. 2021-04-09T18:14:43.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:14:44.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.984 7f07cfde8700 1 Processor -- start 2021-04-09T18:14:44.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.984 7f07cfde8700 1 -- start start 2021-04-09T18:14:44.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07cfde8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f07c80cffb0 0x7f07c8087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07cfde8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07c80d8050 0x7f07c810d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07cfde8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c8088460 con 0x7f07c80d7c50 2021-04-09T18:14:44.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07cfde8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c81b0c80 con 0x7f07c80d8050 2021-04-09T18:14:44.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07cfde8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c80d91f0 con 0x7f07c80cffb0 2021-04-09T18:14:44.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07ce405700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07c80d8050 0x7f07c810d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07ce405700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07c80d8050 0x7f07c810d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43534/0 (socket says 172.21.15.32:43534) 2021-04-09T18:14:44.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07ce405700 1 -- 172.21.15.32:0/1034226591 learned_addr learned my addr 172.21.15.32:0/1034226591 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07cdc04700 1 --2- 172.21.15.32:0/1034226591 >> v2:172.21.15.86:3300/0 conn(0x7f07c80cffb0 0x7f07c8087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07ce405700 1 -- 172.21.15.32:0/1034226591 >> v2:172.21.15.86:3300/0 conn(0x7f07c80cffb0 msgr2=0x7f07c8087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:44.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07ce405700 1 --2- 172.21.15.32:0/1034226591 >> v2:172.21.15.86:3300/0 conn(0x7f07c80cffb0 0x7f07c8087e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07ce405700 1 -- 172.21.15.32:0/1034226591 >> v1:172.21.15.86:6789/0 conn(0x7f07c80d7c50 legacy=0x7f07c81b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.985 7f07ce405700 1 -- 172.21.15.32:0/1034226591 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c80d8620 con 0x7f07c80d8050 2021-04-09T18:14:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.986 7f07ce405700 1 --2- 172.21.15.32:0/1034226591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c80d8050 0x7f07c810d200 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f07c0006f00 tx=0x7f07c0009f10).ready entity=mon.0 client_cookie=f2d4b14288340ff1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:44.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.986 7f07ccc02700 1 -- 172.21.15.32:0/1034226591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07c000b3b0 con 0x7f07c80d8050 2021-04-09T18:14:44.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.986 7f07ccc02700 1 -- 172.21.15.32:0/1034226591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c000b730 con 0x7f07c80d8050 2021-04-09T18:14:44.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.986 7f07ccc02700 1 -- 172.21.15.32:0/1034226591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07c000bc10 con 0x7f07c80d8050 2021-04-09T18:14:44.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.986 7f07cfde8700 1 -- 172.21.15.32:0/1034226591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c80d8050 msgr2=0x7f07c810d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.986 7f07cfde8700 1 --2- 172.21.15.32:0/1034226591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c80d8050 0x7f07c810d200 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f07c0006f00 tx=0x7f07c0009f10).stop 2021-04-09T18:14:44.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.987 7f07cfde8700 1 -- 172.21.15.32:0/1034226591 shutdown_connections 2021-04-09T18:14:44.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.987 7f07cfde8700 1 --2- 172.21.15.32:0/1034226591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c80d8050 0x7f07c810d200 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.987 7f07cfde8700 1 --2- 172.21.15.32:0/1034226591 >> v2:172.21.15.86:3300/0 conn(0x7f07c80cffb0 0x7f07c8087e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.987 7f07cfde8700 1 -- 172.21.15.32:0/1034226591 >> 172.21.15.32:0/1034226591 conn(0x7f07c80ce5b0 msgr2=0x7f07c80cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.987 7f07cfde8700 1 -- 172.21.15.32:0/1034226591 shutdown_connections 2021-04-09T18:14:44.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:43.987 7f07cfde8700 1 -- 172.21.15.32:0/1034226591 wait complete. 2021-04-09T18:14:44.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.007 7f07cfde8700 1 Processor -- start 2021-04-09T18:14:44.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.007 7f07cfde8700 1 -- start start 2021-04-09T18:14:44.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07cfde8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07c8099d00 0x7f07c8182fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07cfde8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07c80a6c10 0x7f07c80a7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07cfde8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c8191ae0 0x7f07c8191ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07cfde8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80a6690 con 0x7f07c8191ae0 2021-04-09T18:14:44.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07cfde8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80bf0e0 con 0x7f07c80a6c10 2021-04-09T18:14:44.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07cfde8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80a7630 con 0x7f07c8099d00 2021-04-09T18:14:44.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07ce405700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c8191ae0 0x7f07c8191ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07ce405700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c8191ae0 0x7f07c8191ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43542/0 (socket says 172.21.15.32:43542) 2021-04-09T18:14:44.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07ce405700 1 -- 172.21.15.32:0/2011658196 learned_addr learned my addr 172.21.15.32:0/2011658196 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:44.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07cd403700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07c80a6c10 0x7f07c80a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07cdc04700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07c8099d00 0x7f07c8182fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07ce405700 1 -- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07c8099d00 msgr2=0x7f07c8182fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07ce405700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07c8099d00 0x7f07c8182fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07ce405700 1 -- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07c80a6c10 msgr2=0x7f07c80a7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07ce405700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07c80a6c10 0x7f07c80a7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07ce405700 1 -- 172.21.15.32:0/2011658196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c8192500 con 0x7f07c8191ae0 2021-04-09T18:14:44.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07cdc04700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07c8099d00 0x7f07c8182fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:14:44.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07ce405700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c8191ae0 0x7f07c8191ee0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f07c0009390 tx=0x7f07c000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:44.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.008 7f07be7fc700 1 -- 172.21.15.32:0/2011658196 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07c0003030 con 0x7f07c8191ae0 2021-04-09T18:14:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.009 7f07be7fc700 1 -- 172.21.15.32:0/2011658196 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c00068e0 con 0x7f07c8191ae0 2021-04-09T18:14:44.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.009 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c8192500 con 0x7f07c8191ae0 2021-04-09T18:14:44.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.009 7f07be7fc700 1 -- 172.21.15.32:0/2011658196 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07c000c570 con 0x7f07c8191ae0 2021-04-09T18:14:44.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.009 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c8192500 con 0x7f07c8191ae0 2021-04-09T18:14:44.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.009 7f07be7fc700 1 -- 172.21.15.32:0/2011658196 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f07c0005820 con 0x7f07c8191ae0 2021-04-09T18:14:44.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.009 7f07be7fc700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b0018930 0x7f07b001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.009 7f07be7fc700 1 -- 172.21.15.32:0/2011658196 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f07c0032d80 con 0x7f07c8191ae0 2021-04-09T18:14:44.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.010 7f07cdc04700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b0018930 0x7f07b001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.010 7f07cdc04700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b0018930 0x7f07b001ae10 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.029 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c81835d0 con 0x7f07c8191ae0 2021-04-09T18:14:44.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.032 7f07be7fc700 1 -- 172.21.15.32:0/2011658196 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f07c00068e0 con 0x7f07c8191ae0 2021-04-09T18:14:44.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.218 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f07c81835d0 con 0x7f07c8191ae0 2021-04-09T18:14:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.221 7f07be7fc700 1 -- 172.21.15.32:0/2011658196 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7f07c001faa0 con 0x7f07c8191ae0 2021-04-09T18:14:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:14:44.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b0018930 msgr2=0x7f07b001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b0018930 0x7f07b001ae10 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c8191ae0 msgr2=0x7f07c8191ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c8191ae0 0x7f07c8191ee0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f07c0009390 tx=0x7f07c000bd40).stop 2021-04-09T18:14:44.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 shutdown_connections 2021-04-09T18:14:44.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b0018930 0x7f07b001ae10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07c8191ae0 0x7f07c8191ee0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07c80a6c10 0x7f07c80a7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 --2- 172.21.15.32:0/2011658196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07c8099d00 0x7f07c8182fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.223 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 >> 172.21.15.32:0/2011658196 conn(0x7f07c80adaf0 msgr2=0x7f07c80bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:44.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.224 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 shutdown_connections 2021-04-09T18:14:44.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.224 7f07cfde8700 1 -- 172.21.15.32:0/2011658196 wait complete. 2021-04-09T18:14:44.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:44.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e930e2700 1 Processor -- start 2021-04-09T18:14:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e930e2700 1 -- start start 2021-04-09T18:14:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e930e2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e8c10d010 0x7f5e8c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e930e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c0d8490 0x7f5e8c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e930e2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8c10d480 con 0x7f5e8c0d1da0 2021-04-09T18:14:44.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e930e2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8c0d8d30 con 0x7f5e8c10d010 2021-04-09T18:14:44.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e930e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c0d9220 con 0x7f5e8c0d8490 2021-04-09T18:14:44.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e90efe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e8c10d010 0x7f5e8c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e90efe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e8c10d010 0x7f5e8c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43546/0 (socket says 172.21.15.32:43546) 2021-04-09T18:14:44.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e90efe700 1 -- 172.21.15.32:0/3879349890 learned_addr learned my addr 172.21.15.32:0/3879349890 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:44.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.648 7f5e916ff700 1 --2- 172.21.15.32:0/3879349890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c0d8490 0x7f5e8c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e90efe700 1 -- 172.21.15.32:0/3879349890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c0d8490 msgr2=0x7f5e8c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:44.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e90efe700 1 --2- 172.21.15.32:0/3879349890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c0d8490 0x7f5e8c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e8b7fe700 1 -- 172.21.15.32:0/3879349890 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693250108 0 0) 0x7f5e7c000c60 con 0x7f5e8c0d1da0 2021-04-09T18:14:44.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e90efe700 1 -- 172.21.15.32:0/3879349890 >> v1:172.21.15.86:6789/0 conn(0x7f5e8c0d1da0 legacy=0x7f5e8c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e90efe700 1 -- 172.21.15.32:0/3879349890 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c10e020 con 0x7f5e8c10d010 2021-04-09T18:14:44.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e916ff700 1 --2- 172.21.15.32:0/3879349890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c0d8490 0x7f5e8c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:44.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e90efe700 1 --2- 172.21.15.32:0/3879349890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c10d010 0x7f5e8c0d1780 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f5e74006700 tx=0x7f5e74009b80).ready entity=mon.0 client_cookie=18e75a78162be9e2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:44.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e8b7fe700 1 -- 172.21.15.32:0/3879349890 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7400c0c0 con 0x7f5e8c10d010 2021-04-09T18:14:44.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e8b7fe700 1 -- 172.21.15.32:0/3879349890 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7400c440 con 0x7f5e8c10d010 2021-04-09T18:14:44.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e8b7fe700 1 -- 172.21.15.32:0/3879349890 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7400c920 con 0x7f5e8c10d010 2021-04-09T18:14:44.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e930e2700 1 -- 172.21.15.32:0/3879349890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c10d010 msgr2=0x7f5e8c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.649 7f5e930e2700 1 --2- 172.21.15.32:0/3879349890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c10d010 0x7f5e8c0d1780 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f5e74006700 tx=0x7f5e74009b80).stop 2021-04-09T18:14:44.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.650 7f5e930e2700 1 -- 172.21.15.32:0/3879349890 shutdown_connections 2021-04-09T18:14:44.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.650 7f5e930e2700 1 --2- 172.21.15.32:0/3879349890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c0d8490 0x7f5e8c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.650 7f5e930e2700 1 --2- 172.21.15.32:0/3879349890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c10d010 0x7f5e8c0d1780 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.650 7f5e930e2700 1 -- 172.21.15.32:0/3879349890 >> 172.21.15.32:0/3879349890 conn(0x7f5e8c0ce5f0 msgr2=0x7f5e8c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.650 7f5e930e2700 1 -- 172.21.15.32:0/3879349890 shutdown_connections 2021-04-09T18:14:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.650 7f5e930e2700 1 -- 172.21.15.32:0/3879349890 wait complete. 2021-04-09T18:14:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.670 7f5e930e2700 1 Processor -- start 2021-04-09T18:14:44.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.670 7f5e930e2700 1 -- start start 2021-04-09T18:14:44.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e930e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e8c1889c0 0x7f5e8c09a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e930e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c18e2c0 0x7f5e8c1aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e930e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c1aaad0 0x7f5e8c0a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e930e2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c1aaed0 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e930e2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c1ab0f0 con 0x7f5e8c1889c0 2021-04-09T18:14:44.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e930e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c0a3ed0 con 0x7f5e8c18e2c0 2021-04-09T18:14:44.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e916ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c1aaad0 0x7f5e8c0a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e916ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c1aaad0 0x7f5e8c0a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43552/0 (socket says 172.21.15.32:43552) 2021-04-09T18:14:44.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e90efe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e8c1889c0 0x7f5e8c09a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e916ff700 1 -- 172.21.15.32:0/2687531672 learned_addr learned my addr 172.21.15.32:0/2687531672 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:44.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e8bfff700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c18e2c0 0x7f5e8c1aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e916ff700 1 -- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c18e2c0 msgr2=0x7f5e8c1aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e916ff700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c18e2c0 0x7f5e8c1aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e916ff700 1 -- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e8c1889c0 msgr2=0x7f5e8c09a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e916ff700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e8c1889c0 0x7f5e8c09a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e916ff700 1 -- 172.21.15.32:0/2687531672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c0a40f0 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e90efe700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e8c1889c0 0x7f5e8c09a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:44.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e916ff700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c1aaad0 0x7f5e8c0a38b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f5e8400b970 tx=0x7f5e8400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:44.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.671 7f5e897fa700 1 -- 172.21.15.32:0/2687531672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e84000f90 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.672 7f5e897fa700 1 -- 172.21.15.32:0/2687531672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e84006aa0 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.672 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8c0a40f0 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.672 7f5e897fa700 1 -- 172.21.15.32:0/2687531672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e840015f0 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.672 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8c0a40f0 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.672 7f5e897fa700 1 -- 172.21.15.32:0/2687531672 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5e84000e70 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.672 7f5e897fa700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e70018930 0x7f5e7001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:44.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.672 7f5e897fa700 1 -- 172.21.15.32:0/2687531672 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5e84032d70 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.672 7f5e90efe700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e70018930 0x7f5e7001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:44.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.673 7f5e90efe700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e70018930 0x7f5e7001adc0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:44.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.692 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8c10d480 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.696 7f5e897fa700 1 -- 172.21.15.32:0/2687531672 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5e840066f0 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.889 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e8c10d480 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.890 7f5e897fa700 1 -- 172.21.15.32:0/2687531672 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+2773 (secure 0 0 0) 0x7f5e84006aa0 con 0x7f5e8c1aaad0 2021-04-09T18:14:44.916 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2021-04-09T18:14:44.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:44.917 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:41.927797","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:14:44.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e70018930 msgr2=0x7f5e7001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e70018930 0x7f5e7001adc0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c1aaad0 msgr2=0x7f5e8c0a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:44.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c1aaad0 0x7f5e8c0a38b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f5e8400b970 tx=0x7f5e8400bd10).stop 2021-04-09T18:14:44.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 shutdown_connections 2021-04-09T18:14:44.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e70018930 0x7f5e7001adc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e8c1aaad0 0x7f5e8c0a38b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e8c18e2c0 0x7f5e8c1aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 --2- 172.21.15.32:0/2687531672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e8c1889c0 0x7f5e8c09a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:44.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 >> 172.21.15.32:0/2687531672 conn(0x7f5e8c1a0db0 msgr2=0x7f5e8c1a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:44.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 shutdown_connections 2021-04-09T18:14:44.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:44.892 7f5e930e2700 1 -- 172.21.15.32:0/2687531672 wait complete. 2021-04-09T18:14:45.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:41.927797', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:45.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:14:45.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:14:45.537 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:14:45.538 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:14:45.538 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:14:45.538 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 5119, 2021-04-09T18:14:45.538 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:14:45.539 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:14:45.539 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 5.5938384990000003, 2021-04-09T18:14:45.539 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 28, 2021-04-09T18:14:45.539 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 43, 2021-04-09T18:14:45.539 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 43, 2021-04-09T18:14:45.540 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 11.152866068 2021-04-09T18:14:45.540 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:14:45.543 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5119, "want_state": "up:active", "state": "up:active", "rank_uptime": 5.5938384990000003, "mdsmap_epoch": 28, "osdmap_epoch": 43, "osdmap_epoch_barrier": 43, "uptime": 11.152866068 } 2021-04-09T18:14:45.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:14:45.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.673 7f55bc271700 1 Processor -- start 2021-04-09T18:14:45.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.673 7f55bc271700 1 -- start start 2021-04-09T18:14:45.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.674 7f55bc271700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b40d7ca0 0x7f55b40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.674 7f55bc271700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55b410da00 0x7f55b410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.674 7f55bc271700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b410d480 con 0x7f55b410d010 2021-04-09T18:14:45.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.674 7f55bc271700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b40d9220 con 0x7f55b410da00 2021-04-09T18:14:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.674 7f55bc271700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40d86c0 con 0x7f55b40d7ca0 2021-04-09T18:14:45.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55ba88e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55b410da00 0x7f55b410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:45.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55b988c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b40d7ca0 0x7f55b40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:45.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55ba08d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f55b410d010 0x7f55b40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:51074/0 (socket says 172.21.15.32:51074) 2021-04-09T18:14:45.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55ba08d700 1 -- 172.21.15.32:0/1326779731 learned_addr learned my addr 172.21.15.32:0/1326779731 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:45.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55b988c700 1 --2- 172.21.15.32:0/1326779731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b40d7ca0 0x7f55b40d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:50788/0 (socket says 172.21.15.32:50788) 2021-04-09T18:14:45.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55ba88e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55b410da00 0x7f55b410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:35318/0 (socket says 172.21.15.32:35318) 2021-04-09T18:14:45.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55ba88e700 1 -- 172.21.15.32:0/1326779731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b40d7ca0 msgr2=0x7f55b40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:45.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55ba88e700 1 --2- 172.21.15.32:0/1326779731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b40d7ca0 0x7f55b40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55ba88e700 1 -- 172.21.15.32:0/1326779731 >> v1:172.21.15.86:6789/0 conn(0x7f55b410d010 legacy=0x7f55b40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55ba88e700 1 -- 172.21.15.32:0/1326779731 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b4112100 con 0x7f55b410da00 2021-04-09T18:14:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55b908b700 1 -- 172.21.15.32:0/1326779731 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413407149 0 0) 0x7f55b00010c0 con 0x7f55b410d010 2021-04-09T18:14:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55b988c700 1 --2- 172.21.15.32:0/1326779731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b40d7ca0 0x7f55b40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55ba88e700 1 --2- 172.21.15.32:0/1326779731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b410da00 0x7f55b410de00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f55ac006e50 tx=0x7f55ac009e70).ready entity=mon.1 client_cookie=96dacfbb938f64fb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55b908b700 1 -- 172.21.15.32:0/1326779731 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55ac006a20 con 0x7f55b410da00 2021-04-09T18:14:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55b908b700 1 -- 172.21.15.32:0/1326779731 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac006790 con 0x7f55b410da00 2021-04-09T18:14:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55b908b700 1 -- 172.21.15.32:0/1326779731 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55ac00b700 con 0x7f55b410da00 2021-04-09T18:14:45.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55bc271700 1 -- 172.21.15.32:0/1326779731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b410da00 msgr2=0x7f55b410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:45.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.675 7f55bc271700 1 --2- 172.21.15.32:0/1326779731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b410da00 0x7f55b410de00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f55ac006e50 tx=0x7f55ac009e70).stop 2021-04-09T18:14:45.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.676 7f55bc271700 1 -- 172.21.15.32:0/1326779731 shutdown_connections 2021-04-09T18:14:45.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.676 7f55bc271700 1 --2- 172.21.15.32:0/1326779731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b410da00 0x7f55b410de00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.676 7f55bc271700 1 --2- 172.21.15.32:0/1326779731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b40d7ca0 0x7f55b40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.676 7f55bc271700 1 -- 172.21.15.32:0/1326779731 >> 172.21.15.32:0/1326779731 conn(0x7f55b40ce5f0 msgr2=0x7f55b40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:45.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.676 7f55bc271700 1 -- 172.21.15.32:0/1326779731 shutdown_connections 2021-04-09T18:14:45.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.676 7f55bc271700 1 -- 172.21.15.32:0/1326779731 wait complete. 2021-04-09T18:14:45.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.696 7f55bc271700 1 Processor -- start 2021-04-09T18:14:45.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.696 7f55bc271700 1 -- start start 2021-04-09T18:14:45.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55bc271700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b41889c0 0x7f55b409a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:45.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55bc271700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b418e2c0 0x7f55b41aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:45.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55bc271700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b41aaad0 0x7f55b40a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:45.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55bc271700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b41aaed0 con 0x7f55b418e2c0 2021-04-09T18:14:45.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55bc271700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b41ab0f0 con 0x7f55b41889c0 2021-04-09T18:14:45.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55bc271700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40a3ed0 con 0x7f55b41aaad0 2021-04-09T18:14:45.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55b988c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b418e2c0 0x7f55b41aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:45.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55b988c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b418e2c0 0x7f55b41aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43564/0 (socket says 172.21.15.32:43564) 2021-04-09T18:14:45.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55b988c700 1 -- 172.21.15.32:0/1840991664 learned_addr learned my addr 172.21.15.32:0/1840991664 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:45.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55ba08d700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b41889c0 0x7f55b409a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:45.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55ba88e700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b41aaad0 0x7f55b40a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:45.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55b988c700 1 -- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b41aaad0 msgr2=0x7f55b40a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:45.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55b988c700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b41aaad0 0x7f55b40a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55b988c700 1 -- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b41889c0 msgr2=0x7f55b409a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:45.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55b988c700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b41889c0 0x7f55b409a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55b988c700 1 -- 172.21.15.32:0/1840991664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b40a40f0 con 0x7f55b418e2c0 2021-04-09T18:14:45.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.697 7f55b988c700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b418e2c0 0x7f55b41aa4b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f55a400b580 tx=0x7f55a400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:45.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.698 7f55aaffd700 1 -- 172.21.15.32:0/1840991664 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55a400bd10 con 0x7f55b418e2c0 2021-04-09T18:14:45.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.698 7f55aaffd700 1 -- 172.21.15.32:0/1840991664 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a4000c90 con 0x7f55b418e2c0 2021-04-09T18:14:45.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.698 7f55bc271700 1 -- 172.21.15.32:0/1840991664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b40a40f0 con 0x7f55b418e2c0 2021-04-09T18:14:45.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.698 7f55aaffd700 1 -- 172.21.15.32:0/1840991664 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55a4002220 con 0x7f55b418e2c0 2021-04-09T18:14:45.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.698 7f55bc271700 1 -- 172.21.15.32:0/1840991664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b40a40f0 con 0x7f55b418e2c0 2021-04-09T18:14:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.698 7f55aaffd700 1 -- 172.21.15.32:0/1840991664 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f55a4006b00 con 0x7f55b418e2c0 2021-04-09T18:14:45.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.698 7f55aaffd700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f559c018930 0x7f559c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:45.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.698 7f55aaffd700 1 -- 172.21.15.32:0/1840991664 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f55a40329f0 con 0x7f55b418e2c0 2021-04-09T18:14:45.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.699 7f55ba08d700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f559c018930 0x7f559c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:45.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.699 7f55ba08d700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f559c018930 0x7f559c01adc0 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:45.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.718 7f55bc271700 1 -- 172.21.15.32:0/1840991664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b410d480 con 0x7f55b418e2c0 2021-04-09T18:14:45.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.722 7f55aaffd700 1 -- 172.21.15.32:0/1840991664 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f55a400bd10 con 0x7f55b418e2c0 2021-04-09T18:14:45.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.912 7f55bc271700 1 -- 172.21.15.32:0/1840991664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55b410d480 con 0x7f55b418e2c0 2021-04-09T18:14:45.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.913 7f55aaffd700 1 -- 172.21.15.32:0/1840991664 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f55a4000c90 con 0x7f55b418e2c0 2021-04-09T18:14:45.946 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:45.947 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":43,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:14:40.920336","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:14:38.344198","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-04-09 18:14:38.803133","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:14:45.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 -- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f559c018930 msgr2=0x7f559c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:45.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f559c018930 0x7f559c01adc0 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 -- 172.21.15.32:0/1840991664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b418e2c0 msgr2=0x7f55b41aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:45.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b418e2c0 0x7f55b41aa4b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f55a400b580 tx=0x7f55a400b920).stop 2021-04-09T18:14:45.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 -- 172.21.15.32:0/1840991664 shutdown_connections 2021-04-09T18:14:45.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f559c018930 0x7f559c01adc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55b418e2c0 0x7f55b41aa4b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55b41aaad0 0x7f55b40a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 --2- 172.21.15.32:0/1840991664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b41889c0 0x7f55b409a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:45.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 -- 172.21.15.32:0/1840991664 >> 172.21.15.32:0/1840991664 conn(0x7f55b41a0db0 msgr2=0x7f55b41a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:45.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 -- 172.21.15.32:0/1840991664 shutdown_connections 2021-04-09T18:14:45.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:45.915 7f55bc271700 1 -- 172.21.15.32:0/1840991664 wait complete. 2021-04-09T18:14:45.998 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:14:38.344198', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:14:38.803133', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:14:46.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:14:46.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.142 7fe8e3e36700 1 Processor -- start 2021-04-09T18:14:46.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.142 7fe8e3e36700 1 -- start start 2021-04-09T18:14:46.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.143 7fe8e3e36700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc0d7ca0 0x7fe8dc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.143 7fe8e3e36700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe8dc10da00 0x7fe8dc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.143 7fe8e3e36700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8dc10d480 con 0x7fe8dc10d010 2021-04-09T18:14:46.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.143 7fe8e3e36700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8dc0d9220 con 0x7fe8dc10da00 2021-04-09T18:14:46.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.143 7fe8e3e36700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0d86c0 con 0x7fe8dc0d7ca0 2021-04-09T18:14:46.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.143 7fe8e1c52700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe8dc10d010 0x7fe8dc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53162/0 (socket says 172.21.15.32:53162) 2021-04-09T18:14:46.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.143 7fe8e1c52700 1 -- 172.21.15.32:0/1264133701 learned_addr learned my addr 172.21.15.32:0/1264133701 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:46.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.143 7fe8e1451700 1 --2- 172.21.15.32:0/1264133701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc0d7ca0 0x7fe8dc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.143 7fe8e2453700 1 --2- 172.21.15.32:0/1264133701 >> v2:172.21.15.86:3300/0 conn(0x7fe8dc10da00 0x7fe8dc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e0c50700 1 -- 172.21.15.32:0/1264133701 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740197585 0 0) 0x7fe8d8001060 con 0x7fe8dc10d010 2021-04-09T18:14:46.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e0c50700 1 -- 172.21.15.32:0/1264133701 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c8002fe0 con 0x7fe8dc10d010 2021-04-09T18:14:46.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e1451700 1 -- 172.21.15.32:0/1264133701 >> v2:172.21.15.86:3300/0 conn(0x7fe8dc10da00 msgr2=0x7fe8dc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:46.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e1451700 1 --2- 172.21.15.32:0/1264133701 >> v2:172.21.15.86:3300/0 conn(0x7fe8dc10da00 0x7fe8dc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e1451700 1 -- 172.21.15.32:0/1264133701 >> v1:172.21.15.32:6789/0 conn(0x7fe8dc10d010 legacy=0x7fe8dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e1451700 1 -- 172.21.15.32:0/1264133701 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc0d1f70 con 0x7fe8dc0d7ca0 2021-04-09T18:14:46.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e2453700 1 --2- 172.21.15.32:0/1264133701 >> v2:172.21.15.86:3300/0 conn(0x7fe8dc10da00 0x7fe8dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:14:46.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e1451700 1 --2- 172.21.15.32:0/1264133701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc0d7ca0 0x7fe8dc0d80a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe8cc006650 tx=0x7fe8cc009a80).ready entity=mon.2 client_cookie=7dea3a4341872b6a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:46.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e0c50700 1 -- 172.21.15.32:0/1264133701 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8cc0009b0 con 0x7fe8dc0d7ca0 2021-04-09T18:14:46.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e0c50700 1 -- 172.21.15.32:0/1264133701 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8cc00c0c0 con 0x7fe8dc0d7ca0 2021-04-09T18:14:46.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e0c50700 1 -- 172.21.15.32:0/1264133701 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8cc00c510 con 0x7fe8dc0d7ca0 2021-04-09T18:14:46.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e3e36700 1 -- 172.21.15.32:0/1264133701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc0d7ca0 msgr2=0x7fe8dc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.144 7fe8e3e36700 1 --2- 172.21.15.32:0/1264133701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc0d7ca0 0x7fe8dc0d80a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe8cc006650 tx=0x7fe8cc009a80).stop 2021-04-09T18:14:46.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.145 7fe8e3e36700 1 -- 172.21.15.32:0/1264133701 shutdown_connections 2021-04-09T18:14:46.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.145 7fe8e3e36700 1 --2- 172.21.15.32:0/1264133701 >> v2:172.21.15.86:3300/0 conn(0x7fe8dc10da00 0x7fe8dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.145 7fe8e3e36700 1 --2- 172.21.15.32:0/1264133701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc0d7ca0 0x7fe8dc0d80a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.145 7fe8e3e36700 1 -- 172.21.15.32:0/1264133701 >> 172.21.15.32:0/1264133701 conn(0x7fe8dc0ce5f0 msgr2=0x7fe8dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:46.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.145 7fe8e3e36700 1 -- 172.21.15.32:0/1264133701 shutdown_connections 2021-04-09T18:14:46.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.145 7fe8e3e36700 1 -- 172.21.15.32:0/1264133701 wait complete. 2021-04-09T18:14:46.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.165 7fe8e3e36700 1 Processor -- start 2021-04-09T18:14:46.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.165 7fe8e3e36700 1 -- start start 2021-04-09T18:14:46.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e3e36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8dc0b2140 0x7fe8dc09d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e3e36700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc09e010 0x7fe8dc194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e3e36700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8dc1950a0 0x7fe8dc0a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e3e36700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc09d470 con 0x7fe8dc0b2140 2021-04-09T18:14:46.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e3e36700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc09e410 con 0x7fe8dc1950a0 2021-04-09T18:14:46.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e3e36700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc1954a0 con 0x7fe8dc09e010 2021-04-09T18:14:46.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1c52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8dc0b2140 0x7fe8dc09d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1c52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8dc0b2140 0x7fe8dc09d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43578/0 (socket says 172.21.15.32:43578) 2021-04-09T18:14:46.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1c52700 1 -- 172.21.15.32:0/4097555276 learned_addr learned my addr 172.21.15.32:0/4097555276 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:46.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e2453700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8dc1950a0 0x7fe8dc0a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1451700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc09e010 0x7fe8dc194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1c52700 1 -- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc09e010 msgr2=0x7fe8dc194a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1c52700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc09e010 0x7fe8dc194a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1c52700 1 -- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8dc1950a0 msgr2=0x7fe8dc0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1c52700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8dc1950a0 0x7fe8dc0a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1c52700 1 -- 172.21.15.32:0/4097555276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc1956c0 con 0x7fe8dc0b2140 2021-04-09T18:14:46.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e1c52700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8dc0b2140 0x7fe8dc09d9f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fe8d8009980 tx=0x7fe8d800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:46.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8d27fc700 1 -- 172.21.15.32:0/4097555276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8d8006030 con 0x7fe8dc0b2140 2021-04-09T18:14:46.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8d27fc700 1 -- 172.21.15.32:0/4097555276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d8006370 con 0x7fe8dc0b2140 2021-04-09T18:14:46.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8d27fc700 1 -- 172.21.15.32:0/4097555276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8d8004910 con 0x7fe8dc0b2140 2021-04-09T18:14:46.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.166 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8dc1956c0 con 0x7fe8dc0b2140 2021-04-09T18:14:46.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.167 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8dc1956c0 con 0x7fe8dc0b2140 2021-04-09T18:14:46.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.167 7fe8d27fc700 1 -- 172.21.15.32:0/4097555276 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe8d8004720 con 0x7fe8dc0b2140 2021-04-09T18:14:46.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.167 7fe8d27fc700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8c4018900 0x7fe8c401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.167 7fe8d27fc700 1 -- 172.21.15.32:0/4097555276 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe8d8032db0 con 0x7fe8dc0b2140 2021-04-09T18:14:46.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.167 7fe8e1451700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8c4018900 0x7fe8c401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.168 7fe8e1451700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8c4018900 0x7fe8c401adc0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.187 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc1b7040 con 0x7fe8dc0b2140 2021-04-09T18:14:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.191 7fe8d27fc700 1 -- 172.21.15.32:0/4097555276 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe8d8000d30 con 0x7fe8dc0b2140 2021-04-09T18:14:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.373 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe8dc10d480 con 0x7fe8dc0b2140 2021-04-09T18:14:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.373 7fe8d27fc700 1 -- 172.21.15.32:0/4097555276 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v43) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe8d8006270 con 0x7fe8dc0b2140 2021-04-09T18:14:46.446 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:14:46.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.375 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8c4018900 msgr2=0x7fe8c401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.375 7fe8e3e36700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8c4018900 0x7fe8c401adc0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.375 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8dc0b2140 msgr2=0x7fe8dc09d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.375 7fe8e3e36700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8dc0b2140 0x7fe8dc09d9f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fe8d8009980 tx=0x7fe8d800bcd0).stop 2021-04-09T18:14:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.376 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 shutdown_connections 2021-04-09T18:14:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.376 7fe8e3e36700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8c4018900 0x7fe8c401adc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.376 7fe8e3e36700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8dc1950a0 0x7fe8dc0a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.376 7fe8e3e36700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8dc09e010 0x7fe8dc194a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.376 7fe8e3e36700 1 --2- 172.21.15.32:0/4097555276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8dc0b2140 0x7fe8dc09d9f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.376 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 >> 172.21.15.32:0/4097555276 conn(0x7fe8dc0b27e0 msgr2=0x7fe8dc1b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:46.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.376 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 shutdown_connections 2021-04-09T18:14:46.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.376 7fe8e3e36700 1 -- 172.21.15.32:0/4097555276 wait complete. 2021-04-09T18:14:46.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:14:46.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.675 7f82f5dc4700 1 Processor -- start 2021-04-09T18:14:46.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.676 7f82f5dc4700 1 -- start start 2021-04-09T18:14:46.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.676 7f82f5dc4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82f010d010 0x7f82f00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.676 7f82f5dc4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f010d480 con 0x7f82f00d1da0 2021-04-09T18:14:46.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.676 7f82f5dc4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f010e1f0 con 0x7f82f00d8490 2021-04-09T18:14:46.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.676 7f82f5dc4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f00d8d30 con 0x7f82f010d010 2021-04-09T18:14:46.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.676 7f82eeffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f82f00d1da0 0x7f82f00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53180/0 (socket says 172.21.15.32:53180) 2021-04-09T18:14:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.676 7f82eeffd700 1 -- 172.21.15.32:0/3717803523 learned_addr learned my addr 172.21.15.32:0/3717803523 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ef7fe700 1 --2- 172.21.15.32:0/3717803523 >> v2:172.21.15.86:3300/0 conn(0x7f82f010d010 0x7f82f00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993649151 0 0) 0x7f82dc000ce0 con 0x7f82f00d1da0 2021-04-09T18:14:46.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d4002fe0 con 0x7f82f00d1da0 2021-04-09T18:14:46.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167259566 0 0) 0x7f82e8000d30 con 0x7f82f00d8490 2021-04-09T18:14:46.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d4003210 con 0x7f82f00d8490 2021-04-09T18:14:46.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3401519624 0 0) 0x7f82dc0010e0 con 0x7f82f00d1da0 2021-04-09T18:14:46.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 >> v2:172.21.15.86:3300/0 conn(0x7f82f010d010 msgr2=0x7f82f00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:46.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 --2- 172.21.15.32:0/3717803523 >> v2:172.21.15.86:3300/0 conn(0x7f82f010d010 0x7f82f00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 >> v1:172.21.15.86:6789/0 conn(0x7f82f00d8490 legacy=0x7f82f010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00d9220 con 0x7f82f00d1da0 2021-04-09T18:14:46.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f82dc001600 con 0x7f82f00d1da0 2021-04-09T18:14:46.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 >> v1:172.21.15.32:6789/0 conn(0x7f82f00d1da0 legacy=0x7f82f00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82d4003b20 0x7f82d4005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82d40067b0 0x7f82d4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d4009590 0x7f82d400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ef7fe700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82d4003b20 0x7f82d4005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d4050310 con 0x7f82d4003b20 2021-04-09T18:14:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d40504f0 con 0x7f82d40067b0 2021-04-09T18:14:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d4050710 con 0x7f82d4009590 2021-04-09T18:14:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82eeffd700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82d40067b0 0x7f82d4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82effff700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d4009590 0x7f82d400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ef7fe700 1 -- 172.21.15.32:0/3717803523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d4009590 msgr2=0x7f82d400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:46.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ef7fe700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d4009590 0x7f82d400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ef7fe700 1 -- 172.21.15.32:0/3717803523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82d40067b0 msgr2=0x7f82d4008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:46.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ef7fe700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82d40067b0 0x7f82d4008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82ef7fe700 1 -- 172.21.15.32:0/3717803523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d40509b0 con 0x7f82d4003b20 2021-04-09T18:14:46.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82effff700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d4009590 0x7f82d400bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:46.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.677 7f82effff700 1 -- 172.21.15.32:0/3717803523 reap_dead start 2021-04-09T18:14:46.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82ef7fe700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82d4003b20 0x7f82d4005f30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f82e000c8f0 tx=0x7f82e000cc90).ready entity=mon.0 client_cookie=43d2ad1369780ca8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:46.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e0009310 con 0x7f82d4003b20 2021-04-09T18:14:46.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e0009610 con 0x7f82d4003b20 2021-04-09T18:14:46.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82ee7fc700 1 -- 172.21.15.32:0/3717803523 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e0009af0 con 0x7f82d4003b20 2021-04-09T18:14:46.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82f5dc4700 1 -- 172.21.15.32:0/3717803523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82d4003b20 msgr2=0x7f82d4005f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82f5dc4700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82d4003b20 0x7f82d4005f30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f82e000c8f0 tx=0x7f82e000cc90).stop 2021-04-09T18:14:46.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82f5dc4700 1 -- 172.21.15.32:0/3717803523 shutdown_connections 2021-04-09T18:14:46.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82f5dc4700 1 --2- 172.21.15.32:0/3717803523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82d4003b20 0x7f82d4005f30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82f5dc4700 1 -- 172.21.15.32:0/3717803523 >> 172.21.15.32:0/3717803523 conn(0x7f82f00ce5f0 msgr2=0x7f82f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82f5dc4700 1 -- 172.21.15.32:0/3717803523 shutdown_connections 2021-04-09T18:14:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.678 7f82f5dc4700 1 -- 172.21.15.32:0/3717803523 wait complete. 2021-04-09T18:14:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.699 7f82f5dc4700 1 Processor -- start 2021-04-09T18:14:46.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.699 7f82f5dc4700 1 -- start start 2021-04-09T18:14:46.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.699 7f82f5dc4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01a0050 0x7f82f01a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.699 7f82f5dc4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00bdcd0 0x7f82f00bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.699 7f82f5dc4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f00bf080 0x7f82f00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.699 7f82f5dc4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00bf500 con 0x7f82f00bdcd0 2021-04-09T18:14:46.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.699 7f82f5dc4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00bb7b0 con 0x7f82f01a0050 2021-04-09T18:14:46.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.699 7f82f5dc4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00be0d0 con 0x7f82f00bf080 2021-04-09T18:14:46.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82eeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00bdcd0 0x7f82f00bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82eeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00bdcd0 0x7f82f00bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43598/0 (socket says 172.21.15.32:43598) 2021-04-09T18:14:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82eeffd700 1 -- 172.21.15.32:0/1270533841 learned_addr learned my addr 172.21.15.32:0/1270533841 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82ef7fe700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01a0050 0x7f82f01a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82effff700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f00bf080 0x7f82f00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82eeffd700 1 -- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f00bf080 msgr2=0x7f82f00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82eeffd700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f00bf080 0x7f82f00b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82eeffd700 1 -- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01a0050 msgr2=0x7f82f01a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82eeffd700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01a0050 0x7f82f01a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82eeffd700 1 -- 172.21.15.32:0/1270533841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00b1310 con 0x7f82f00bdcd0 2021-04-09T18:14:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82eeffd700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00bdcd0 0x7f82f00bea60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f82dc007fc0 tx=0x7f82dc00a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82e7fff700 1 -- 172.21.15.32:0/1270533841 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82dc000f40 con 0x7f82f00bdcd0 2021-04-09T18:14:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82e7fff700 1 -- 172.21.15.32:0/1270533841 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc00ac70 con 0x7f82f00bdcd0 2021-04-09T18:14:46.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f00b1310 con 0x7f82f00bdcd0 2021-04-09T18:14:46.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82e7fff700 1 -- 172.21.15.32:0/1270533841 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82dc004620 con 0x7f82f00bdcd0 2021-04-09T18:14:46.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.700 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f00b1310 con 0x7f82f00bdcd0 2021-04-09T18:14:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.701 7f82e7fff700 1 -- 172.21.15.32:0/1270533841 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f82dc004830 con 0x7f82f00bdcd0 2021-04-09T18:14:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.701 7f82e7fff700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82cc018900 0x7f82cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:46.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.701 7f82e7fff700 1 -- 172.21.15.32:0/1270533841 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f82dc032b50 con 0x7f82f00bdcd0 2021-04-09T18:14:46.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.701 7f82ef7fe700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82cc018900 0x7f82cc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:46.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.701 7f82ef7fe700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82cc018900 0x7f82cc01adc0 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:46.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.720 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f0093d50 con 0x7f82f00bdcd0 2021-04-09T18:14:46.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.724 7f82e7fff700 1 -- 172.21.15.32:0/1270533841 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f82dc004500 con 0x7f82f00bdcd0 2021-04-09T18:14:46.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.902 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f82f010d480 con 0x7f82f00bdcd0 2021-04-09T18:14:46.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.903 7f82e7fff700 1 -- 172.21.15.32:0/1270533841 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v43) v1 ==== 104+0+10 (secure 0 0 0) 0x7f82dc004710 con 0x7f82f00bdcd0 2021-04-09T18:14:46.942 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:14:46.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82cc018900 msgr2=0x7f82cc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82cc018900 0x7f82cc01adc0 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00bdcd0 msgr2=0x7f82f00bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00bdcd0 0x7f82f00bea60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f82dc007fc0 tx=0x7f82dc00a880).stop 2021-04-09T18:14:46.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 shutdown_connections 2021-04-09T18:14:46.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82cc018900 0x7f82cc01adc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00bdcd0 0x7f82f00bea60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f00bf080 0x7f82f00b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 --2- 172.21.15.32:0/1270533841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01a0050 0x7f82f01a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:46.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 >> 172.21.15.32:0/1270533841 conn(0x7f82f00975f0 msgr2=0x7f82f0096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:46.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 shutdown_connections 2021-04-09T18:14:46.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:46.905 7f82f5dc4700 1 -- 172.21.15.32:0/1270533841 wait complete. 2021-04-09T18:14:46.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:14:47.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.167 7f9c1b965700 1 Processor -- start 2021-04-09T18:14:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.167 7f9c1b965700 1 -- start start 2021-04-09T18:14:47.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.167 7f9c1b965700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9c1410d010 0x7f9c140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.167 7f9c1b965700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c140d1da0 0x7f9c140d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.167 7f9c1b965700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c140d82c0 0x7f9c1410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.167 7f9c1b965700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c140d8d30 con 0x7f9c140d1da0 2021-04-09T18:14:47.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.167 7f9c1b965700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c140d9220 con 0x7f9c1410d010 2021-04-09T18:14:47.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.167 7f9c1b965700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1410d4b0 con 0x7f9c140d82c0 2021-04-09T18:14:47.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.167 7f9c18f80700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c140d1da0 0x7f9c140d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c18f80700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c140d1da0 0x7f9c140d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43606/0 (socket says 172.21.15.32:43606) 2021-04-09T18:14:47.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c18f80700 1 -- 172.21.15.32:0/3169549923 learned_addr learned my addr 172.21.15.32:0/3169549923 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:47.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c19781700 1 --2- 172.21.15.32:0/3169549923 >> v2:172.21.15.86:3300/0 conn(0x7f9c1410d010 0x7f9c140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c19f82700 1 --2- 172.21.15.32:0/3169549923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c140d82c0 0x7f9c1410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c18f80700 1 -- 172.21.15.32:0/3169549923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c140d82c0 msgr2=0x7f9c1410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:47.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c18f80700 1 --2- 172.21.15.32:0/3169549923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c140d82c0 0x7f9c1410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c18f80700 1 -- 172.21.15.32:0/3169549923 >> v2:172.21.15.86:3300/0 conn(0x7f9c1410d010 msgr2=0x7f9c140d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:47.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c18f80700 1 --2- 172.21.15.32:0/3169549923 >> v2:172.21.15.86:3300/0 conn(0x7f9c1410d010 0x7f9c140d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c18f80700 1 -- 172.21.15.32:0/3169549923 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140d86c0 con 0x7f9c140d1da0 2021-04-09T18:14:47.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c19781700 1 --2- 172.21.15.32:0/3169549923 >> v2:172.21.15.86:3300/0 conn(0x7f9c1410d010 0x7f9c140d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:47.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c18f80700 1 --2- 172.21.15.32:0/3169549923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c140d1da0 0x7f9c140d7ca0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f9c08006700 tx=0x7f9c08009b00).ready entity=mon.0 client_cookie=d32daa466958b4fc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:47.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c13fff700 1 -- 172.21.15.32:0/3169549923 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c0800c0c0 con 0x7f9c140d1da0 2021-04-09T18:14:47.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c13fff700 1 -- 172.21.15.32:0/3169549923 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0800c400 con 0x7f9c140d1da0 2021-04-09T18:14:47.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c13fff700 1 -- 172.21.15.32:0/3169549923 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c0800c8e0 con 0x7f9c140d1da0 2021-04-09T18:14:47.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c1b965700 1 -- 172.21.15.32:0/3169549923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c140d1da0 msgr2=0x7f9c140d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.168 7f9c1b965700 1 --2- 172.21.15.32:0/3169549923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c140d1da0 0x7f9c140d7ca0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f9c08006700 tx=0x7f9c08009b00).stop 2021-04-09T18:14:47.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.169 7f9c1b965700 1 -- 172.21.15.32:0/3169549923 shutdown_connections 2021-04-09T18:14:47.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.169 7f9c1b965700 1 --2- 172.21.15.32:0/3169549923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c140d82c0 0x7f9c1410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.169 7f9c1b965700 1 --2- 172.21.15.32:0/3169549923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c140d1da0 0x7f9c140d7ca0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.169 7f9c1b965700 1 --2- 172.21.15.32:0/3169549923 >> v2:172.21.15.86:3300/0 conn(0x7f9c1410d010 0x7f9c140d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.169 7f9c1b965700 1 -- 172.21.15.32:0/3169549923 >> 172.21.15.32:0/3169549923 conn(0x7f9c140ce5f0 msgr2=0x7f9c140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:47.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.169 7f9c1b965700 1 -- 172.21.15.32:0/3169549923 shutdown_connections 2021-04-09T18:14:47.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.169 7f9c1b965700 1 -- 172.21.15.32:0/3169549923 wait complete. 2021-04-09T18:14:47.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.189 7f9c1b965700 1 Processor -- start 2021-04-09T18:14:47.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.189 7f9c1b965700 1 -- start start 2021-04-09T18:14:47.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c1b965700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c14099380 0x7f9c140b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c1b965700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c140b6800 0x7f9c140b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c1b965700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c1408b1a0 0x7f9c1408b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c1b965700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140b7580 con 0x7f9c1408b1a0 2021-04-09T18:14:47.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c1b965700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140b6280 con 0x7f9c140b6800 2021-04-09T18:14:47.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c1b965700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14094640 con 0x7f9c14099380 2021-04-09T18:14:47.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c19f82700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c1408b1a0 0x7f9c1408b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c19f82700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c1408b1a0 0x7f9c1408b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43612/0 (socket says 172.21.15.32:43612) 2021-04-09T18:14:47.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c19f82700 1 -- 172.21.15.32:0/580780761 learned_addr learned my addr 172.21.15.32:0/580780761 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:47.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c18f80700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c140b6800 0x7f9c140b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c19781700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c14099380 0x7f9c140b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c19f82700 1 -- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c14099380 msgr2=0x7f9c140b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c19f82700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c14099380 0x7f9c140b5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c19f82700 1 -- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c140b6800 msgr2=0x7f9c140b6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c19f82700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c140b6800 0x7f9c140b6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.190 7f9c19f82700 1 -- 172.21.15.32:0/580780761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140b7220 con 0x7f9c1408b1a0 2021-04-09T18:14:47.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.191 7f9c19f82700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c1408b1a0 0x7f9c1408b5a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f9c04006f20 tx=0x7f9c0400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:47.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.191 7f9c11ffb700 1 -- 172.21.15.32:0/580780761 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c0400bd80 con 0x7f9c1408b1a0 2021-04-09T18:14:47.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.191 7f9c11ffb700 1 -- 172.21.15.32:0/580780761 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c040067e0 con 0x7f9c1408b1a0 2021-04-09T18:14:47.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.191 7f9c1b965700 1 -- 172.21.15.32:0/580780761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c140b7220 con 0x7f9c1408b1a0 2021-04-09T18:14:47.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.191 7f9c11ffb700 1 -- 172.21.15.32:0/580780761 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c04005330 con 0x7f9c1408b1a0 2021-04-09T18:14:47.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.191 7f9c1b965700 1 -- 172.21.15.32:0/580780761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c140b7220 con 0x7f9c1408b1a0 2021-04-09T18:14:47.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.191 7f9c11ffb700 1 -- 172.21.15.32:0/580780761 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9c04006030 con 0x7f9c1408b1a0 2021-04-09T18:14:47.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.191 7f9c11ffb700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9bf4018930 0x7f9bf401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.192 7f9c11ffb700 1 -- 172.21.15.32:0/580780761 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9c04032a70 con 0x7f9c1408b1a0 2021-04-09T18:14:47.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.192 7f9c19781700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9bf4018930 0x7f9bf401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.192 7f9c19781700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9bf4018930 0x7f9bf401ae10 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:47.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.211 7f9c1b965700 1 -- 172.21.15.32:0/580780761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c140f1810 con 0x7f9c1408b1a0 2021-04-09T18:14:47.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.215 7f9c11ffb700 1 -- 172.21.15.32:0/580780761 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9c04006030 con 0x7f9c1408b1a0 2021-04-09T18:14:47.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.394 7f9c1b965700 1 -- 172.21.15.32:0/580780761 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f9c1410d480 con 0x7f9c1408b1a0 2021-04-09T18:14:47.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.395 7f9c11ffb700 1 -- 172.21.15.32:0/580780761 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v43) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9c040067e0 con 0x7f9c1408b1a0 2021-04-09T18:14:47.420 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:14:47.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 -- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9bf4018930 msgr2=0x7f9bf401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9bf4018930 0x7f9bf401ae10 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 -- 172.21.15.32:0/580780761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c1408b1a0 msgr2=0x7f9c1408b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c1408b1a0 0x7f9c1408b5a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f9c04006f20 tx=0x7f9c0400b990).stop 2021-04-09T18:14:47.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 -- 172.21.15.32:0/580780761 shutdown_connections 2021-04-09T18:14:47.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9bf4018930 0x7f9bf401ae10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c1408b1a0 0x7f9c1408b5a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c140b6800 0x7f9c140b6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 --2- 172.21.15.32:0/580780761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c14099380 0x7f9c140b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 -- 172.21.15.32:0/580780761 >> 172.21.15.32:0/580780761 conn(0x7f9c140950b0 msgr2=0x7f9c140b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:47.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 -- 172.21.15.32:0/580780761 shutdown_connections 2021-04-09T18:14:47.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.397 7f9c1b965700 1 -- 172.21.15.32:0/580780761 wait complete. 2021-04-09T18:14:47.479 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:14:47.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:47.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.682 7f70810f0700 1 Processor -- start 2021-04-09T18:14:47.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.682 7f70810f0700 1 -- start start 2021-04-09T18:14:47.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f70810f0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f707c0d7ca0 0x7f707c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f70810f0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f707c10da00 0x7f707c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f70810f0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707c10d480 con 0x7f707c10d010 2021-04-09T18:14:47.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f70810f0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707c0d9220 con 0x7f707c0d7ca0 2021-04-09T18:14:47.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f70810f0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707c0d86c0 con 0x7f707c10da00 2021-04-09T18:14:47.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f707ad9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f707c10d010 0x7f707c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53208/0 (socket says 172.21.15.32:53208) 2021-04-09T18:14:47.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f707ad9d700 1 -- 172.21.15.32:0/2156088202 learned_addr learned my addr 172.21.15.32:0/2156088202 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:47.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f707b59e700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f707c10da00 0x7f707c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f707a59c700 1 --2- 172.21.15.32:0/2156088202 >> v2:172.21.15.86:3300/0 conn(0x7f707c0d7ca0 0x7f707c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042179997 0 0) 0x7f706c000ce0 con 0x7f707c10d010 2021-04-09T18:14:47.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7064002fe0 con 0x7f707c10d010 2021-04-09T18:14:47.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.683 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3227556294 0 0) 0x7f706c000ed0 con 0x7f707c10d010 2021-04-09T18:14:47.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f707c10da00 msgr2=0x7f707c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:47.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f707c10da00 0x7f707c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 >> v2:172.21.15.86:3300/0 conn(0x7f707c0d7ca0 msgr2=0x7f707c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:47.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 --2- 172.21.15.32:0/2156088202 >> v2:172.21.15.86:3300/0 conn(0x7f707c0d7ca0 0x7f707c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c0d1f70 con 0x7f707c10d010 2021-04-09T18:14:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f706c001600 con 0x7f707c10d010 2021-04-09T18:14:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 >> v1:172.21.15.32:6789/0 conn(0x7f707c10d010 legacy=0x7f707c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7064003a40 0x7f7064005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f706404a760 0x7f706404cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f706404d500 0x7f706404fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70640502d0 con 0x7f706404a760 2021-04-09T18:14:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70640504f0 con 0x7f706404d500 2021-04-09T18:14:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7064050710 con 0x7f7064003a40 2021-04-09T18:14:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707a59c700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f706404a760 0x7f706404cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707ad9d700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7064003a40 0x7f7064005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707b59e700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f706404d500 0x7f706404fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707a59c700 1 -- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7064003a40 msgr2=0x7f7064005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707a59c700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7064003a40 0x7f7064005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707a59c700 1 -- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f706404d500 msgr2=0x7f706404fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707a59c700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f706404d500 0x7f706404fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707a59c700 1 -- 172.21.15.32:0/2156088202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70640509b0 con 0x7f706404a760 2021-04-09T18:14:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707a59c700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f706404a760 0x7f706404cc60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f707000c870 tx=0x7f707000cc10).ready entity=mon.0 client_cookie=a897e03173a2dcef server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70700090d0 con 0x7f706404a760 2021-04-09T18:14:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7070009450 con 0x7f706404a760 2021-04-09T18:14:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f7079d9b700 1 -- 172.21.15.32:0/2156088202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7070009930 con 0x7f706404a760 2021-04-09T18:14:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f70810f0700 1 -- 172.21.15.32:0/2156088202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f706404a760 msgr2=0x7f706404cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f70810f0700 1 --2- 172.21.15.32:0/2156088202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f706404a760 0x7f706404cc60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f707000c870 tx=0x7f707000cc10).stop 2021-04-09T18:14:47.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.684 7f707b59e700 1 -- 172.21.15.32:0/2156088202 reap_dead start 2021-04-09T18:14:47.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.685 7f70810f0700 1 -- 172.21.15.32:0/2156088202 shutdown_connections 2021-04-09T18:14:47.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.685 7f70810f0700 1 -- 172.21.15.32:0/2156088202 >> 172.21.15.32:0/2156088202 conn(0x7f707c0ce5f0 msgr2=0x7f707c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:47.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.685 7f70810f0700 1 -- 172.21.15.32:0/2156088202 shutdown_connections 2021-04-09T18:14:47.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.685 7f70810f0700 1 -- 172.21.15.32:0/2156088202 wait complete. 2021-04-09T18:14:47.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.705 7f70810f0700 1 Processor -- start 2021-04-09T18:14:47.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.705 7f70810f0700 1 -- start start 2021-04-09T18:14:47.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f70810f0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f707c1889c0 0x7f707c09a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f70810f0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f707c18e2c0 0x7f707c1aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f70810f0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f707c1aaad0 0x7f707c0a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f70810f0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c1aaed0 con 0x7f707c1889c0 2021-04-09T18:14:47.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f70810f0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c1ab0f0 con 0x7f707c18e2c0 2021-04-09T18:14:47.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f70810f0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707c0a3ed0 con 0x7f707c1aaad0 2021-04-09T18:14:47.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f707c1889c0 0x7f707c09a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f707c1889c0 0x7f707c09a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43630/0 (socket says 172.21.15.32:43630) 2021-04-09T18:14:47.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707ad9d700 1 -- 172.21.15.32:0/855148090 learned_addr learned my addr 172.21.15.32:0/855148090 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:47.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707a59c700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f707c18e2c0 0x7f707c1aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707b59e700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f707c1aaad0 0x7f707c0a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707ad9d700 1 -- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f707c1aaad0 msgr2=0x7f707c0a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707ad9d700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f707c1aaad0 0x7f707c0a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707ad9d700 1 -- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f707c18e2c0 msgr2=0x7f707c1aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707ad9d700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f707c18e2c0 0x7f707c1aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.706 7f707ad9d700 1 -- 172.21.15.32:0/855148090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c0a40f0 con 0x7f707c1889c0 2021-04-09T18:14:47.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.707 7f707ad9d700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f707c1889c0 0x7f707c09a3f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f706c00a520 tx=0x7f706c00a8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:47.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.707 7f706b7fe700 1 -- 172.21.15.32:0/855148090 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f706c0014c0 con 0x7f707c1889c0 2021-04-09T18:14:47.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.707 7f706b7fe700 1 -- 172.21.15.32:0/855148090 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c0070d0 con 0x7f707c1889c0 2021-04-09T18:14:47.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.707 7f706b7fe700 1 -- 172.21.15.32:0/855148090 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f706c0043a0 con 0x7f707c1889c0 2021-04-09T18:14:47.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.707 7f70810f0700 1 -- 172.21.15.32:0/855148090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707c0a40f0 con 0x7f707c1889c0 2021-04-09T18:14:47.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.707 7f70810f0700 1 -- 172.21.15.32:0/855148090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707c0a40f0 con 0x7f707c1889c0 2021-04-09T18:14:47.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.707 7f706b7fe700 1 -- 172.21.15.32:0/855148090 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f706c0011a0 con 0x7f707c1889c0 2021-04-09T18:14:47.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.708 7f706b7fe700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70600188c0 0x7f706001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:47.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.708 7f706b7fe700 1 -- 172.21.15.32:0/855148090 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f706c0329c0 con 0x7f707c1889c0 2021-04-09T18:14:47.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.708 7f707a59c700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70600188c0 0x7f706001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:47.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.708 7f707a59c700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70600188c0 0x7f706001adc0 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:47.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.727 7f70810f0700 1 -- 172.21.15.32:0/855148090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707c10d480 con 0x7f707c1889c0 2021-04-09T18:14:47.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.730 7f706b7fe700 1 -- 172.21.15.32:0/855148090 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f706c0014c0 con 0x7f707c1889c0 2021-04-09T18:14:47.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.921 7f70810f0700 1 -- 172.21.15.32:0/855148090 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f707c10d480 con 0x7f707c1889c0 2021-04-09T18:14:47.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.922 7f706b7fe700 1 -- 172.21.15.32:0/855148090 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+2773 (secure 0 0 0) 0x7f706c0014c0 con 0x7f707c1889c0 2021-04-09T18:14:47.948 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2021-04-09T18:14:47.948 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:47.949 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:41.927797","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:14:47.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 -- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70600188c0 msgr2=0x7f706001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70600188c0 0x7f706001adc0 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 -- 172.21.15.32:0/855148090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f707c1889c0 msgr2=0x7f707c09a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:47.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f707c1889c0 0x7f707c09a3f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f706c00a520 tx=0x7f706c00a8c0).stop 2021-04-09T18:14:47.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 -- 172.21.15.32:0/855148090 shutdown_connections 2021-04-09T18:14:47.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70600188c0 0x7f706001adc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f707c1aaad0 0x7f707c0a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f707c18e2c0 0x7f707c1aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 --2- 172.21.15.32:0/855148090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f707c1889c0 0x7f707c09a3f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 -- 172.21.15.32:0/855148090 >> 172.21.15.32:0/855148090 conn(0x7f707c1a0db0 msgr2=0x7f707c1a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 -- 172.21.15.32:0/855148090 shutdown_connections 2021-04-09T18:14:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:47.924 7f70810f0700 1 -- 172.21.15.32:0/855148090 wait complete. 2021-04-09T18:14:48.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:41.927797', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:48.364 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:14:48.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:14:48.494 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:14:48.494 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:14:48.495 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:14:48.495 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 5119, 2021-04-09T18:14:48.496 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:14:48.496 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:14:48.497 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 8.5498003770000004, 2021-04-09T18:14:48.498 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 28, 2021-04-09T18:14:48.498 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 43, 2021-04-09T18:14:48.499 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 43, 2021-04-09T18:14:48.499 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 14.108828357 2021-04-09T18:14:48.500 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:14:48.507 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5119, "want_state": "up:active", "state": "up:active", "rank_uptime": 8.5498003770000004, "mdsmap_epoch": 28, "osdmap_epoch": 43, "osdmap_epoch_barrier": 43, "uptime": 14.108828357 } 2021-04-09T18:14:48.508 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:14:48.509 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:14:48.510 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:14:48.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:48.511 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:48.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:14:48.715 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:14:48.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:14:48.783 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:14:48.784 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:14:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:14:48.786 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:48.789 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:14:48.862 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:14:48.863 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:48.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:14:48.999 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:14:49.032 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:14:49.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:14:49.045 7f668c0e2f80 -1 init, newargv = 0x559cf5fdf3d0 newargc=9 2021-04-09T18:14:49.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[35486]: starting ceph client 2021-04-09T18:14:49.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[35486]: starting fuse 2021-04-09T18:14:49.122 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:14:49.127 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:14:49.128 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:14:49.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:49.129 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:14:49.216 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:14:49.217 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:14:49.217 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:49.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:14:49.218 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:14:49.218 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:14:49.223 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:14:49.224 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:14:49.224 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:14:49.230 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:14:49.231 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:14:49.231 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:14:49.232 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:14:49.232 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:14:49.233 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:14:49.233 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:14:49.234 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:14:49.234 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:14:49.235 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:14:49.243 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:14:49.243 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:14:49.244 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:14:49.244 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:14:49.272 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:14:49.273 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:14:49.273 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:14:49.365 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.35486.asok 2021-04-09T18:14:49.368 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.35486.asok 2021-04-09T18:14:49.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:49.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.35486.asok status 2021-04-09T18:14:49.496 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:14:49.496 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:14:49.497 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:14:49.497 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:14:49.497 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:14:49.497 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:14:49.498 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:14:49.498 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "35486", 2021-04-09T18:14:49.498 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:14:49.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:49.499 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:14:49.501 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:14:49.502 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5292, 2021-04-09T18:14:49.502 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:14:49.502 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:14:49.503 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:14:49.503 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 5292 2021-04-09T18:14:49.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:49.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:14:49.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:14:49.512 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3640378912 2021-04-09T18:14:49.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:14:49.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:49.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:14:49.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:14:49.521 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:14:49.528 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3640378912 2021-04-09T18:14:49.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:14:49.529 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.5292 172.21.15.32:0/3640378912", 2021-04-09T18:14:49.529 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/3640378912", 2021-04-09T18:14:49.530 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:14:49.530 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 28, 2021-04-09T18:14:49.530 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 43, 2021-04-09T18:14:49.530 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:14:49.531 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:14:49.534 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:14:49.534 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:14:49.544 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:49.545 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:14:49.551 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:49.607 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:14:49.657 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:14:49.727 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:49.729 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:49.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:49.799 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:14:49.800 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:49.916 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:14:49.921 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:14:49.922 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:14:49.922 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:14:49.928 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:50.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[35486]: fuse finished with error 0 and tester_r 0 2021-04-09T18:14:50.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:14:50.042 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:14:50.045 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:50.117 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:14:50.144 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:14:50.145 INFO:teuthology.orchestra.run:waiting for 900 2021-04-09T18:14:50.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:14:50.227 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:14:50.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2021-04-09T18:14:50.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.453 7f29022bc700 1 Processor -- start 2021-04-09T18:14:50.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.453 7f29022bc700 1 -- start start 2021-04-09T18:14:50.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f29022bc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28fc0cffc0 0x7f28fc0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f29022bc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc0d7820 0x7f28fc0d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f29022bc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0d72a0 con 0x7f28fc0d8240 2021-04-09T18:14:50.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f29022bc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc0d9200 con 0x7f28fc0cffc0 2021-04-09T18:14:50.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f29022bc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc10e000 con 0x7f28fc0d7820 2021-04-09T18:14:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28fbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28fc0cffc0 0x7f28fc0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f29008d9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f28fc0d8240 0x7f28fc0d5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53244/0 (socket says 172.21.15.32:53244) 2021-04-09T18:14:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f29008d9700 1 -- 172.21.15.32:0/3122444294 learned_addr learned my addr 172.21.15.32:0/3122444294 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28f3fff700 1 --2- 172.21.15.32:0/3122444294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc0d7820 0x7f28fc0d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28fbfff700 1 -- 172.21.15.32:0/3122444294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc0d7820 msgr2=0x7f28fc0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:50.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28fbfff700 1 --2- 172.21.15.32:0/3122444294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc0d7820 0x7f28fc0d7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:50.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28fbfff700 1 -- 172.21.15.32:0/3122444294 >> v1:172.21.15.32:6789/0 conn(0x7f28fc0d8240 legacy=0x7f28fc0d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28fbfff700 1 -- 172.21.15.32:0/3122444294 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc10e220 con 0x7f28fc0cffc0 2021-04-09T18:14:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28fb7fe700 1 -- 172.21.15.32:0/3122444294 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587159387 0 0) 0x7f28f4000d30 con 0x7f28fc0d8240 2021-04-09T18:14:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28fbfff700 1 --2- 172.21.15.32:0/3122444294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc0cffc0 0x7f28fc0d2460 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f28ec006650 tx=0x7f28ec009a00).ready entity=mon.0 client_cookie=2cea1a10c706012 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:50.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28fb7fe700 1 -- 172.21.15.32:0/3122444294 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ec009df0 con 0x7f28fc0cffc0 2021-04-09T18:14:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.454 7f28fb7fe700 1 -- 172.21.15.32:0/3122444294 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec0009b0 con 0x7f28fc0cffc0 2021-04-09T18:14:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.455 7f28fb7fe700 1 -- 172.21.15.32:0/3122444294 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ec00c2b0 con 0x7f28fc0cffc0 2021-04-09T18:14:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.455 7f29022bc700 1 -- 172.21.15.32:0/3122444294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc0cffc0 msgr2=0x7f28fc0d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.455 7f29022bc700 1 --2- 172.21.15.32:0/3122444294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc0cffc0 0x7f28fc0d2460 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f28ec006650 tx=0x7f28ec009a00).stop 2021-04-09T18:14:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.455 7f29022bc700 1 -- 172.21.15.32:0/3122444294 shutdown_connections 2021-04-09T18:14:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.455 7f29022bc700 1 --2- 172.21.15.32:0/3122444294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc0d7820 0x7f28fc0d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.455 7f29022bc700 1 --2- 172.21.15.32:0/3122444294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc0cffc0 0x7f28fc0d2460 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.455 7f29022bc700 1 -- 172.21.15.32:0/3122444294 >> 172.21.15.32:0/3122444294 conn(0x7f28fc0ce5e0 msgr2=0x7f28fc0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.455 7f29022bc700 1 -- 172.21.15.32:0/3122444294 shutdown_connections 2021-04-09T18:14:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.455 7f29022bc700 1 -- 172.21.15.32:0/3122444294 wait complete. 2021-04-09T18:14:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29022bc700 1 Processor -- start 2021-04-09T18:14:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29022bc700 1 -- start start 2021-04-09T18:14:50.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29022bc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28fc08ce70 0x7f28fc091d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:50.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29022bc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc092390 0x7f28fc1b7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29022bc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc1b7c40 0x7f28fc1b8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:50.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29022bc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0b5ff0 con 0x7f28fc1b7c40 2021-04-09T18:14:50.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29022bc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0917f0 con 0x7f28fc08ce70 2021-04-09T18:14:50.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29022bc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0a1340 con 0x7f28fc092390 2021-04-09T18:14:50.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29008d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc1b7c40 0x7f28fc1b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:50.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29008d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc1b7c40 0x7f28fc1b8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43660/0 (socket says 172.21.15.32:43660) 2021-04-09T18:14:50.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f29008d9700 1 -- 172.21.15.32:0/628573654 learned_addr learned my addr 172.21.15.32:0/628573654 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:50.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f28fbfff700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28fc08ce70 0x7f28fc091d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:50.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.476 7f28f3fff700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc092390 0x7f28fc1b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:50.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f29008d9700 1 -- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc092390 msgr2=0x7f28fc1b7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:50.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f29008d9700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc092390 0x7f28fc1b7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:50.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f29008d9700 1 -- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28fc08ce70 msgr2=0x7f28fc091d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:50.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f29008d9700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28fc08ce70 0x7f28fc091d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:50.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f29008d9700 1 -- 172.21.15.32:0/628573654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc08d270 con 0x7f28fc1b7c40 2021-04-09T18:14:50.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f29008d9700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc1b7c40 0x7f28fc1b8040 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f28f4007f90 tx=0x7f28f400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:50.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f28f97fa700 1 -- 172.21.15.32:0/628573654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f4000cc0 con 0x7f28fc1b7c40 2021-04-09T18:14:50.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f28f97fa700 1 -- 172.21.15.32:0/628573654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4007020 con 0x7f28fc1b7c40 2021-04-09T18:14:50.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f29022bc700 1 -- 172.21.15.32:0/628573654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc08d270 con 0x7f28fc1b7c40 2021-04-09T18:14:50.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f28f97fa700 1 -- 172.21.15.32:0/628573654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f4007500 con 0x7f28fc1b7c40 2021-04-09T18:14:50.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.477 7f29022bc700 1 -- 172.21.15.32:0/628573654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc08d270 con 0x7f28fc1b7c40 2021-04-09T18:14:50.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.478 7f28f97fa700 1 -- 172.21.15.32:0/628573654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f28f4007200 con 0x7f28fc1b7c40 2021-04-09T18:14:50.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.478 7f28f97fa700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f28e00188c0 0x7f28e001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:50.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.478 7f28f97fa700 1 -- 172.21.15.32:0/628573654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f28f4032ae0 con 0x7f28fc1b7c40 2021-04-09T18:14:50.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.478 7f28fbfff700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f28e00188c0 0x7f28e001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:50.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.478 7f28fbfff700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f28e00188c0 0x7f28e001adc0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:50.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.498 7f29022bc700 1 -- 172.21.15.32:0/628573654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc1b7100 con 0x7f28fc1b7c40 2021-04-09T18:14:50.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.501 7f28f97fa700 1 -- 172.21.15.32:0/628573654 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f28f402d340 con 0x7f28fc1b7c40 2021-04-09T18:14:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:50.695 7f29022bc700 1 -- 172.21.15.32:0/628573654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "true"} v 0) v1 -- 0x7f28fc0d72a0 con 0x7f28fc1b7c40 2021-04-09T18:14:51.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.338 7f28f97fa700 1 -- 172.21.15.32:0/628573654 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "true"}]=0 cephfs marked down. v29) v1 ==== 125+0+0 (secure 0 0 0) 0x7f28f402d340 con 0x7f28fc1b7c40 2021-04-09T18:14:51.383 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked down. 2021-04-09T18:14:51.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 -- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f28e00188c0 msgr2=0x7f28e001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:51.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f28e00188c0 0x7f28e001adc0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 -- 172.21.15.32:0/628573654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc1b7c40 msgr2=0x7f28fc1b8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc1b7c40 0x7f28fc1b8040 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f28f4007f90 tx=0x7f28f400ac50).stop 2021-04-09T18:14:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 -- 172.21.15.32:0/628573654 shutdown_connections 2021-04-09T18:14:51.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f28e00188c0 0x7f28e001adc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28fc1b7c40 0x7f28fc1b8040 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28fc092390 0x7f28fc1b7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 --2- 172.21.15.32:0/628573654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28fc08ce70 0x7f28fc091d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 -- 172.21.15.32:0/628573654 >> 172.21.15.32:0/628573654 conn(0x7f28fc1b8d00 msgr2=0x7f28fc0b62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:51.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 -- 172.21.15.32:0/628573654 shutdown_connections 2021-04-09T18:14:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.362 7f29022bc700 1 -- 172.21.15.32:0/628573654 wait complete. 2021-04-09T18:14:51.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:51.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.530 7f47e12d7700 1 Processor -- start 2021-04-09T18:14:51.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.531 7f47e12d7700 1 -- start start 2021-04-09T18:14:51.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.531 7f47e12d7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47dc0d7ca0 0x7f47dc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:51.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.531 7f47e12d7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47dc10da00 0x7f47dc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:51.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.531 7f47e12d7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc10d480 con 0x7f47dc10d010 2021-04-09T18:14:51.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.531 7f47e12d7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47dc0d9220 con 0x7f47dc0d7ca0 2021-04-09T18:14:51.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.531 7f47e12d7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47dc0d86c0 con 0x7f47dc10da00 2021-04-09T18:14:51.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.531 7f47da87c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47dc0d7ca0 0x7f47dc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:51.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.531 7f47da87c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47dc0d7ca0 0x7f47dc0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43670/0 (socket says 172.21.15.32:43670) 2021-04-09T18:14:51.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.531 7f47da87c700 1 -- 172.21.15.32:0/1698366694 learned_addr learned my addr 172.21.15.32:0/1698366694 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:51.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47db87e700 1 --2- 172.21.15.32:0/1698366694 >> v2:172.21.15.86:3300/0 conn(0x7f47dc10da00 0x7f47dc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:51.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47da87c700 1 -- 172.21.15.32:0/1698366694 >> v2:172.21.15.86:3300/0 conn(0x7f47dc10da00 msgr2=0x7f47dc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:51.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47da87c700 1 --2- 172.21.15.32:0/1698366694 >> v2:172.21.15.86:3300/0 conn(0x7f47dc10da00 0x7f47dc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47da87c700 1 -- 172.21.15.32:0/1698366694 >> v1:172.21.15.86:6789/0 conn(0x7f47dc10d010 legacy=0x7f47dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:51.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47da87c700 1 -- 172.21.15.32:0/1698366694 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc0d1f70 con 0x7f47dc0d7ca0 2021-04-09T18:14:51.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47db87e700 1 --2- 172.21.15.32:0/1698366694 >> v2:172.21.15.86:3300/0 conn(0x7f47dc10da00 0x7f47dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:51.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47da87c700 1 --2- 172.21.15.32:0/1698366694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc0d7ca0 0x7f47dc0d80a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f47c8006700 tx=0x7f47c8009b80).ready entity=mon.0 client_cookie=bacf173f78c503aa server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:51.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47da07b700 1 -- 172.21.15.32:0/1698366694 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c800c0c0 con 0x7f47dc0d7ca0 2021-04-09T18:14:51.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47da07b700 1 -- 172.21.15.32:0/1698366694 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c800c440 con 0x7f47dc0d7ca0 2021-04-09T18:14:51.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47da07b700 1 -- 172.21.15.32:0/1698366694 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c800c920 con 0x7f47dc0d7ca0 2021-04-09T18:14:51.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47e12d7700 1 -- 172.21.15.32:0/1698366694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc0d7ca0 msgr2=0x7f47dc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:51.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.532 7f47e12d7700 1 --2- 172.21.15.32:0/1698366694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc0d7ca0 0x7f47dc0d80a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f47c8006700 tx=0x7f47c8009b80).stop 2021-04-09T18:14:51.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.533 7f47e12d7700 1 -- 172.21.15.32:0/1698366694 shutdown_connections 2021-04-09T18:14:51.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.533 7f47e12d7700 1 --2- 172.21.15.32:0/1698366694 >> v2:172.21.15.86:3300/0 conn(0x7f47dc10da00 0x7f47dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.533 7f47e12d7700 1 --2- 172.21.15.32:0/1698366694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc0d7ca0 0x7f47dc0d80a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.533 7f47e12d7700 1 -- 172.21.15.32:0/1698366694 >> 172.21.15.32:0/1698366694 conn(0x7f47dc0ce5f0 msgr2=0x7f47dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:51.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.533 7f47e12d7700 1 -- 172.21.15.32:0/1698366694 shutdown_connections 2021-04-09T18:14:51.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.533 7f47e12d7700 1 -- 172.21.15.32:0/1698366694 wait complete. 2021-04-09T18:14:51.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.553 7f47e12d7700 1 Processor -- start 2021-04-09T18:14:51.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.553 7f47e12d7700 1 -- start start 2021-04-09T18:14:51.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47e12d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47dc1a5210 0x7f47dc185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47e12d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47dc185a70 0x7f47dc0acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:51.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47e12d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc185e70 0x7f47dc0ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47e12d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc18f7b0 con 0x7f47dc185e70 2021-04-09T18:14:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47e12d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc0ad7f0 con 0x7f47dc1a5210 2021-04-09T18:14:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47e12d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc0ada10 con 0x7f47dc185a70 2021-04-09T18:14:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc185e70 0x7f47dc0ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc185e70 0x7f47dc0ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43676/0 (socket says 172.21.15.32:43676) 2021-04-09T18:14:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db87e700 1 -- 172.21.15.32:0/1484133630 learned_addr learned my addr 172.21.15.32:0/1484133630 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47da87c700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47dc185a70 0x7f47dc0acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:51.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db07d700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47dc1a5210 0x7f47dc185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:51.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db87e700 1 -- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47dc185a70 msgr2=0x7f47dc0acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:51.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db87e700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47dc185a70 0x7f47dc0acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db87e700 1 -- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47dc1a5210 msgr2=0x7f47dc185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:51.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db87e700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47dc1a5210 0x7f47dc185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db87e700 1 -- 172.21.15.32:0/1484133630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc179610 con 0x7f47dc185e70 2021-04-09T18:14:51.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47db87e700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc185e70 0x7f47dc0ad1d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f47d4009980 tx=0x7f47d400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:51.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47d3fff700 1 -- 172.21.15.32:0/1484133630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47d400c2f0 con 0x7f47dc185e70 2021-04-09T18:14:51.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47d3fff700 1 -- 172.21.15.32:0/1484133630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d4006790 con 0x7f47dc185e70 2021-04-09T18:14:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc179610 con 0x7f47dc185e70 2021-04-09T18:14:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47d3fff700 1 -- 172.21.15.32:0/1484133630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47d400c7e0 con 0x7f47dc185e70 2021-04-09T18:14:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.554 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc179610 con 0x7f47dc185e70 2021-04-09T18:14:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.556 7f47d3fff700 1 -- 172.21.15.32:0/1484133630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f47d40011f0 con 0x7f47dc185e70 2021-04-09T18:14:51.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.556 7f47d3fff700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b80188c0 0x7f47b801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:51.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.556 7f47d3fff700 1 -- 172.21.15.32:0/1484133630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f47d4032b70 con 0x7f47dc185e70 2021-04-09T18:14:51.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.556 7f47db07d700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b80188c0 0x7f47b801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:51.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.556 7f47db07d700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b80188c0 0x7f47b801adc0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:51.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.575 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc18f7b0 con 0x7f47dc185e70 2021-04-09T18:14:51.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.579 7f47d3fff700 1 -- 172.21.15.32:0/1484133630 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f47d400be10 con 0x7f47dc185e70 2021-04-09T18:14:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.768 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47dc18f7b0 con 0x7f47dc185e70 2021-04-09T18:14:51.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.768 7f47d3fff700 1 -- 172.21.15.32:0/1484133630 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f47d40011f0 con 0x7f47dc185e70 2021-04-09T18:14:51.795 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:14:51.795 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:51.796 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:14:51.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b80188c0 msgr2=0x7f47b801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:51.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b80188c0 0x7f47b801adc0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc185e70 msgr2=0x7f47dc0ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:51.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc185e70 0x7f47dc0ad1d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f47d4009980 tx=0x7f47d400bcd0).stop 2021-04-09T18:14:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 shutdown_connections 2021-04-09T18:14:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b80188c0 0x7f47b801adc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47dc185e70 0x7f47dc0ad1d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47dc185a70 0x7f47dc0acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 --2- 172.21.15.32:0/1484133630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47dc1a5210 0x7f47dc185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 >> 172.21.15.32:0/1484133630 conn(0x7f47dc1a99c0 msgr2=0x7f47dc17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 shutdown_connections 2021-04-09T18:14:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:51.771 7f47e12d7700 1 -- 172.21.15.32:0/1484133630 wait complete. 2021-04-09T18:14:52.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:52.221 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:14:53.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:53.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.360 7f73d5e02700 1 Processor -- start 2021-04-09T18:14:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.360 7f73d5e02700 1 -- start start 2021-04-09T18:14:53.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73d5e02700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73d00d7ca0 0x7f73d00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73d5e02700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f73d010da00 0x7f73d010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73d5e02700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d010d480 con 0x7f73d010d010 2021-04-09T18:14:53.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73d5e02700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d00d9220 con 0x7f73d010da00 2021-04-09T18:14:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73d5e02700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00d86c0 con 0x7f73d00d7ca0 2021-04-09T18:14:53.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73cf7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f73d010d010 0x7f73d00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53270/0 (socket says 172.21.15.32:53270) 2021-04-09T18:14:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73cf7fe700 1 -- 172.21.15.32:0/3542247652 learned_addr learned my addr 172.21.15.32:0/3542247652 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:53.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73cffff700 1 --2- 172.21.15.32:0/3542247652 >> v2:172.21.15.86:3300/0 conn(0x7f73d010da00 0x7f73d010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73ceffd700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73d00d7ca0 0x7f73d00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:53.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014087840 0 0) 0x7f73c0000ce0 con 0x7f73d010d010 2021-04-09T18:14:53.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b8002fe0 con 0x7f73d010d010 2021-04-09T18:14:53.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4139314197 0 0) 0x7f73c0000ed0 con 0x7f73d010d010 2021-04-09T18:14:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73d00d7ca0 msgr2=0x7f73d00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73ce7fc700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73d00d7ca0 0x7f73d00d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 >> v2:172.21.15.86:3300/0 conn(0x7f73d010da00 msgr2=0x7f73d010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:53.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.361 7f73ce7fc700 1 --2- 172.21.15.32:0/3542247652 >> v2:172.21.15.86:3300/0 conn(0x7f73d010da00 0x7f73d010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d00d1f70 con 0x7f73d010d010 2021-04-09T18:14:53.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f73c0001600 con 0x7f73d010d010 2021-04-09T18:14:53.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 >> v1:172.21.15.32:6789/0 conn(0x7f73d010d010 legacy=0x7f73d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:53.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b8003a60 0x7f73b8005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:53.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73b804a760 0x7f73b804cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:53.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73b804d500 0x7f73b804fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:53.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80502d0 con 0x7f73b804d500 2021-04-09T18:14:53.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80504f0 con 0x7f73b804a760 2021-04-09T18:14:53.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b8050710 con 0x7f73b8003a60 2021-04-09T18:14:53.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73cffff700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73b804d500 0x7f73b804fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:53.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ceffd700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73b804a760 0x7f73b804cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:53.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73cf7fe700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b8003a60 0x7f73b8005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:53.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73cffff700 1 -- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b8003a60 msgr2=0x7f73b8005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:53.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73cffff700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b8003a60 0x7f73b8005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73cffff700 1 -- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73b804a760 msgr2=0x7f73b804cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:53.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73cffff700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73b804a760 0x7f73b804cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73cffff700 1 -- 172.21.15.32:0/3542247652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b80509b0 con 0x7f73b804d500 2021-04-09T18:14:53.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73cffff700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73b804d500 0x7f73b804fa20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f73c8006a40 tx=0x7f73c8009f40).ready entity=mon.0 client_cookie=564cc1a1b9473df3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:53.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c800c420 con 0x7f73b804d500 2021-04-09T18:14:53.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800c720 con 0x7f73b804d500 2021-04-09T18:14:53.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73ce7fc700 1 -- 172.21.15.32:0/3542247652 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c8000f90 con 0x7f73b804d500 2021-04-09T18:14:53.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73d5e02700 1 -- 172.21.15.32:0/3542247652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73b804d500 msgr2=0x7f73b804fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:53.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73d5e02700 1 --2- 172.21.15.32:0/3542247652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73b804d500 0x7f73b804fa20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f73c8006a40 tx=0x7f73c8009f40).stop 2021-04-09T18:14:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.362 7f73cffff700 1 -- 172.21.15.32:0/3542247652 reap_dead start 2021-04-09T18:14:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.363 7f73d5e02700 1 -- 172.21.15.32:0/3542247652 shutdown_connections 2021-04-09T18:14:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.363 7f73d5e02700 1 -- 172.21.15.32:0/3542247652 >> 172.21.15.32:0/3542247652 conn(0x7f73d00ce5f0 msgr2=0x7f73d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.363 7f73d5e02700 1 -- 172.21.15.32:0/3542247652 shutdown_connections 2021-04-09T18:14:53.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.363 7f73d5e02700 1 -- 172.21.15.32:0/3542247652 wait complete. 2021-04-09T18:14:53.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.383 7f73d5e02700 1 Processor -- start 2021-04-09T18:14:53.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.383 7f73d5e02700 1 -- start start 2021-04-09T18:14:53.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73d5e02700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d00b42d0 0x7f73d0183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:53.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73d5e02700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73d01ac090 0x7f73d01ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:53.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73d5e02700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73d0184d30 0x7f73d0185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:53.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73d5e02700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00a1a90 con 0x7f73d00b42d0 2021-04-09T18:14:53.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73d5e02700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d01acab0 con 0x7f73d01ac090 2021-04-09T18:14:53.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73d5e02700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0185a00 con 0x7f73d0184d30 2021-04-09T18:14:53.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73cf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d00b42d0 0x7f73d0183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:53.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73cf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d00b42d0 0x7f73d0183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43692/0 (socket says 172.21.15.32:43692) 2021-04-09T18:14:53.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73cf7fe700 1 -- 172.21.15.32:0/3925614531 learned_addr learned my addr 172.21.15.32:0/3925614531 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:53.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73ceffd700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73d01ac090 0x7f73d01ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73cffff700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73d0184d30 0x7f73d0185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:53.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73cf7fe700 1 -- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73d0184d30 msgr2=0x7f73d0185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:53.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73cf7fe700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73d0184d30 0x7f73d0185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73cf7fe700 1 -- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73d01ac090 msgr2=0x7f73d01ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:53.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73cf7fe700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73d01ac090 0x7f73d01ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.384 7f73cf7fe700 1 -- 172.21.15.32:0/3925614531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d0185750 con 0x7f73d00b42d0 2021-04-09T18:14:53.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.385 7f73cf7fe700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d00b42d0 0x7f73d0183520 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f73c000a580 tx=0x7f73c000a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:53.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.385 7f73b7fff700 1 -- 172.21.15.32:0/3925614531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c000ad10 con 0x7f73d00b42d0 2021-04-09T18:14:53.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.385 7f73b7fff700 1 -- 172.21.15.32:0/3925614531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c0001510 con 0x7f73d00b42d0 2021-04-09T18:14:53.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.385 7f73b7fff700 1 -- 172.21.15.32:0/3925614531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c00043a0 con 0x7f73d00b42d0 2021-04-09T18:14:53.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.385 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d0185750 con 0x7f73d00b42d0 2021-04-09T18:14:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.385 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d0185750 con 0x7f73d00b42d0 2021-04-09T18:14:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.386 7f73b7fff700 1 -- 172.21.15.32:0/3925614531 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f73c0009030 con 0x7f73d00b42d0 2021-04-09T18:14:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.386 7f73b7fff700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73b00188c0 0x7f73b001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.386 7f73b7fff700 1 -- 172.21.15.32:0/3925614531 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f73c0032990 con 0x7f73d00b42d0 2021-04-09T18:14:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.386 7f73ceffd700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73b00188c0 0x7f73b001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.386 7f73ceffd700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73b00188c0 0x7f73b001adc0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.405 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d0182d10 con 0x7f73d00b42d0 2021-04-09T18:14:53.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.408 7f73b7fff700 1 -- 172.21.15.32:0/3925614531 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f73c00011a0 con 0x7f73d00b42d0 2021-04-09T18:14:53.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.600 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73d010d480 con 0x7f73d00b42d0 2021-04-09T18:14:53.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.601 7f73b7fff700 1 -- 172.21.15.32:0/3925614531 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f73c0009030 con 0x7f73d00b42d0 2021-04-09T18:14:53.627 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:14:53.628 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:53.628 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:14:53.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73b00188c0 msgr2=0x7f73b001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:53.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73b00188c0 0x7f73b001adc0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d00b42d0 msgr2=0x7f73d0183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:53.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d00b42d0 0x7f73d0183520 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f73c000a580 tx=0x7f73c000a920).stop 2021-04-09T18:14:53.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 shutdown_connections 2021-04-09T18:14:53.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73b00188c0 0x7f73b001adc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73d0184d30 0x7f73d0185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73d01ac090 0x7f73d01ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 --2- 172.21.15.32:0/3925614531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73d00b42d0 0x7f73d0183520 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:53.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 >> 172.21.15.32:0/3925614531 conn(0x7f73d019b890 msgr2=0x7f73d01a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:53.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 shutdown_connections 2021-04-09T18:14:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:53.603 7f73d5e02700 1 -- 172.21.15.32:0/3925614531 wait complete. 2021-04-09T18:14:54.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:54.167 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:14:55.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:55.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.305 7efc0113c700 1 Processor -- start 2021-04-09T18:14:55.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.305 7efc0113c700 1 -- start start 2021-04-09T18:14:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efc0113c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc10d010 0x7efbfc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efc0113c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efbfc0d1da0 0x7efbfc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efc0113c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efbfc0d82c0 0x7efbfc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efc0113c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbfc10e780 con 0x7efbfc0d82c0 2021-04-09T18:14:55.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efc0113c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbfc0d8d30 con 0x7efbfc0d1da0 2021-04-09T18:14:55.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efc0113c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0d9220 con 0x7efbfc10d010 2021-04-09T18:14:55.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfb59e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efbfc0d82c0 0x7efbfc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:55.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfb59e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efbfc0d82c0 0x7efbfc10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43702/0 (socket says 172.21.15.32:43702) 2021-04-09T18:14:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfb59e700 1 -- 172.21.15.32:0/788824080 learned_addr learned my addr 172.21.15.32:0/788824080 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:55.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfa59c700 1 --2- 172.21.15.32:0/788824080 >> v2:172.21.15.86:3300/0 conn(0x7efbfc0d1da0 0x7efbfc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfad9d700 1 --2- 172.21.15.32:0/788824080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc10d010 0x7efbfc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:55.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfb59e700 1 -- 172.21.15.32:0/788824080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc10d010 msgr2=0x7efbfc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:55.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfb59e700 1 --2- 172.21.15.32:0/788824080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc10d010 0x7efbfc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfb59e700 1 -- 172.21.15.32:0/788824080 >> v2:172.21.15.86:3300/0 conn(0x7efbfc0d1da0 msgr2=0x7efbfc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfb59e700 1 --2- 172.21.15.32:0/788824080 >> v2:172.21.15.86:3300/0 conn(0x7efbfc0d1da0 0x7efbfc0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfb59e700 1 -- 172.21.15.32:0/788824080 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc10d480 con 0x7efbfc0d82c0 2021-04-09T18:14:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.306 7efbfb59e700 1 --2- 172.21.15.32:0/788824080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc0d82c0 0x7efbfc10da00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7efbf4006f00 tx=0x7efbf4009e90).ready entity=mon.0 client_cookie=fe77abfd4b9e46ef server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:55.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.307 7efbf9d9b700 1 -- 172.21.15.32:0/788824080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbf400b300 con 0x7efbfc0d82c0 2021-04-09T18:14:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.307 7efbf9d9b700 1 -- 172.21.15.32:0/788824080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf400b640 con 0x7efbfc0d82c0 2021-04-09T18:14:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.307 7efbf9d9b700 1 -- 172.21.15.32:0/788824080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbf400bb20 con 0x7efbfc0d82c0 2021-04-09T18:14:55.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.307 7efc0113c700 1 -- 172.21.15.32:0/788824080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc0d82c0 msgr2=0x7efbfc10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:55.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.307 7efc0113c700 1 --2- 172.21.15.32:0/788824080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc0d82c0 0x7efbfc10da00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7efbf4006f00 tx=0x7efbf4009e90).stop 2021-04-09T18:14:55.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.308 7efc0113c700 1 -- 172.21.15.32:0/788824080 shutdown_connections 2021-04-09T18:14:55.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.308 7efc0113c700 1 --2- 172.21.15.32:0/788824080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc0d82c0 0x7efbfc10da00 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.308 7efc0113c700 1 --2- 172.21.15.32:0/788824080 >> v2:172.21.15.86:3300/0 conn(0x7efbfc0d1da0 0x7efbfc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.308 7efc0113c700 1 --2- 172.21.15.32:0/788824080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc10d010 0x7efbfc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.308 7efc0113c700 1 -- 172.21.15.32:0/788824080 >> 172.21.15.32:0/788824080 conn(0x7efbfc0ce5f0 msgr2=0x7efbfc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:55.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.308 7efc0113c700 1 -- 172.21.15.32:0/788824080 shutdown_connections 2021-04-09T18:14:55.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.308 7efc0113c700 1 -- 172.21.15.32:0/788824080 wait complete. 2021-04-09T18:14:55.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.328 7efc0113c700 1 Processor -- start 2021-04-09T18:14:55.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.328 7efc0113c700 1 -- start start 2021-04-09T18:14:55.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.328 7efc0113c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc1aca20 0x7efbfc1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:55.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.328 7efc0113c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc18bd40 0x7efbfc18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:55.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.328 7efc0113c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbfc08fba0 0x7efbfc08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:55.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.328 7efc0113c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc18c7e0 con 0x7efbfc18bd40 2021-04-09T18:14:55.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.328 7efc0113c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc18ca30 con 0x7efbfc08fba0 2021-04-09T18:14:55.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.328 7efc0113c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0905f0 con 0x7efbfc1aca20 2021-04-09T18:14:55.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfa59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc18bd40 0x7efbfc18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:55.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfa59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc18bd40 0x7efbfc18c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43708/0 (socket says 172.21.15.32:43708) 2021-04-09T18:14:55.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfa59c700 1 -- 172.21.15.32:0/1963417261 learned_addr learned my addr 172.21.15.32:0/1963417261 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfb59e700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbfc08fba0 0x7efbfc08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfad9d700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc1aca20 0x7efbfc1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfa59c700 1 -- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc1aca20 msgr2=0x7efbfc1ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfa59c700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc1aca20 0x7efbfc1ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfa59c700 1 -- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbfc08fba0 msgr2=0x7efbfc08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfa59c700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbfc08fba0 0x7efbfc08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfa59c700 1 -- 172.21.15.32:0/1963417261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc0b4300 con 0x7efbfc18bd40 2021-04-09T18:14:55.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfb59e700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbfc08fba0 0x7efbfc08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:55.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbfa59c700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc18bd40 0x7efbfc18c140 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7efbf0006f60 tx=0x7efbf000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:55.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbeb7fe700 1 -- 172.21.15.32:0/1963417261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbf00009b0 con 0x7efbfc18bd40 2021-04-09T18:14:55.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbeb7fe700 1 -- 172.21.15.32:0/1963417261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf000bd60 con 0x7efbfc18bd40 2021-04-09T18:14:55.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efc0113c700 1 -- 172.21.15.32:0/1963417261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbfc0b4300 con 0x7efbfc18bd40 2021-04-09T18:14:55.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efbeb7fe700 1 -- 172.21.15.32:0/1963417261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbf0002030 con 0x7efbfc18bd40 2021-04-09T18:14:55.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.329 7efc0113c700 1 -- 172.21.15.32:0/1963417261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc0b4300 con 0x7efbfc18bd40 2021-04-09T18:14:55.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.330 7efbeb7fe700 1 -- 172.21.15.32:0/1963417261 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efbf00052e0 con 0x7efbfc18bd40 2021-04-09T18:14:55.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.330 7efbeb7fe700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbe0018900 0x7efbe001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:55.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.330 7efbeb7fe700 1 -- 172.21.15.32:0/1963417261 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efbf0032a00 con 0x7efbfc18bd40 2021-04-09T18:14:55.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.330 7efbfad9d700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbe0018900 0x7efbe001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:55.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.330 7efbfad9d700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbe0018900 0x7efbe001adc0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:55.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.350 7efc0113c700 1 -- 172.21.15.32:0/1963417261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbfc182d70 con 0x7efbfc18bd40 2021-04-09T18:14:55.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.353 7efbeb7fe700 1 -- 172.21.15.32:0/1963417261 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efbf00052e0 con 0x7efbfc18bd40 2021-04-09T18:14:55.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.544 7efc0113c700 1 -- 172.21.15.32:0/1963417261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efbfc18c760 con 0x7efbfc18bd40 2021-04-09T18:14:55.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.544 7efbeb7fe700 1 -- 172.21.15.32:0/1963417261 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7efbf000bd60 con 0x7efbfc18bd40 2021-04-09T18:14:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:14:55.619 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:55.620 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:14:55.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 -- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbe0018900 msgr2=0x7efbe001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:55.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbe0018900 0x7efbe001adc0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 -- 172.21.15.32:0/1963417261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc18bd40 msgr2=0x7efbfc18c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:55.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc18bd40 0x7efbfc18c140 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7efbf0006f60 tx=0x7efbf000b970).stop 2021-04-09T18:14:55.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 -- 172.21.15.32:0/1963417261 shutdown_connections 2021-04-09T18:14:55.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbe0018900 0x7efbe001adc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbfc18bd40 0x7efbfc18c140 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbfc08fba0 0x7efbfc08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 --2- 172.21.15.32:0/1963417261 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbfc1aca20 0x7efbfc1ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 -- 172.21.15.32:0/1963417261 >> 172.21.15.32:0/1963417261 conn(0x7efbfc19b460 msgr2=0x7efbfc19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:55.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 -- 172.21.15.32:0/1963417261 shutdown_connections 2021-04-09T18:14:55.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:55.547 7efc0113c700 1 -- 172.21.15.32:0/1963417261 wait complete. 2021-04-09T18:14:56.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:56.081 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:14:57.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:57.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.206 7f4b6d50c700 1 Processor -- start 2021-04-09T18:14:57.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.206 7f4b6d50c700 1 -- start start 2021-04-09T18:14:57.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.206 7f4b6d50c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6810da00 0x7f4b6810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:57.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.206 7f4b6d50c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6810d480 con 0x7f4b6810d010 2021-04-09T18:14:57.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.206 7f4b6d50c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b680d1f70 con 0x7f4b680d7ca0 2021-04-09T18:14:57.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.206 7f4b6d50c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b680d9220 con 0x7f4b6810da00 2021-04-09T18:14:57.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b672bf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4b6810d010 0x7f4b680d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53304/0 (socket says 172.21.15.32:53304) 2021-04-09T18:14:57.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b672bf700 1 -- 172.21.15.32:0/888888326 learned_addr learned my addr 172.21.15.32:0/888888326 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:14:57.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b67ac0700 1 --2- 172.21.15.32:0/888888326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6810da00 0x7f4b6810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:57.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b662bd700 1 -- 172.21.15.32:0/888888326 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703707256 0 0) 0x7f4b60000ce0 con 0x7f4b6810d010 2021-04-09T18:14:57.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b662bd700 1 -- 172.21.15.32:0/888888326 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b54002fe0 con 0x7f4b6810d010 2021-04-09T18:14:57.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b662bd700 1 -- 172.21.15.32:0/888888326 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686460088 0 0) 0x7f4b58000fe0 con 0x7f4b680d7ca0 2021-04-09T18:14:57.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b662bd700 1 -- 172.21.15.32:0/888888326 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b54003210 con 0x7f4b680d7ca0 2021-04-09T18:14:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b67ac0700 1 -- 172.21.15.32:0/888888326 >> v1:172.21.15.86:6789/0 conn(0x7f4b680d7ca0 legacy=0x7f4b680d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b662bd700 1 -- 172.21.15.32:0/888888326 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 730658335 0 0) 0x7f4b600010e0 con 0x7f4b6810d010 2021-04-09T18:14:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b67ac0700 1 -- 172.21.15.32:0/888888326 >> v1:172.21.15.32:6789/0 conn(0x7f4b6810d010 legacy=0x7f4b680d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b67ac0700 1 -- 172.21.15.32:0/888888326 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b680d09c0 con 0x7f4b6810da00 2021-04-09T18:14:57.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b67ac0700 1 --2- 172.21.15.32:0/888888326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6810da00 0x7f4b6810de00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4b50006ad0 tx=0x7f4b50009a50).ready entity=mon.2 client_cookie=6c571ddae57769ae server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:57.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b662bd700 1 -- 172.21.15.32:0/888888326 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b500066a0 con 0x7f4b6810da00 2021-04-09T18:14:57.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b662bd700 1 -- 172.21.15.32:0/888888326 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b50006410 con 0x7f4b6810da00 2021-04-09T18:14:57.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.207 7f4b662bd700 1 -- 172.21.15.32:0/888888326 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b500024d0 con 0x7f4b6810da00 2021-04-09T18:14:57.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.208 7f4b6d50c700 1 -- 172.21.15.32:0/888888326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6810da00 msgr2=0x7f4b6810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.208 7f4b6d50c700 1 --2- 172.21.15.32:0/888888326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6810da00 0x7f4b6810de00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4b50006ad0 tx=0x7f4b50009a50).stop 2021-04-09T18:14:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.208 7f4b6d50c700 1 -- 172.21.15.32:0/888888326 shutdown_connections 2021-04-09T18:14:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.208 7f4b6d50c700 1 --2- 172.21.15.32:0/888888326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6810da00 0x7f4b6810de00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.208 7f4b6d50c700 1 -- 172.21.15.32:0/888888326 >> 172.21.15.32:0/888888326 conn(0x7f4b680ce5f0 msgr2=0x7f4b680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.208 7f4b6d50c700 1 -- 172.21.15.32:0/888888326 shutdown_connections 2021-04-09T18:14:57.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.208 7f4b6d50c700 1 -- 172.21.15.32:0/888888326 wait complete. 2021-04-09T18:14:57.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.228 7f4b6d50c700 1 Processor -- start 2021-04-09T18:14:57.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.228 7f4b6d50c700 1 -- start start 2021-04-09T18:14:57.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b6d50c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b680b5360 0x7f4b681a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:57.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b6d50c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6818e140 0x7f4b681aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b6d50c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b681ad040 0x7f4b6808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b6d50c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b681a5360 con 0x7f4b681ad040 2021-04-09T18:14:57.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b6d50c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b680902c0 con 0x7f4b680b5360 2021-04-09T18:14:57.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b6d50c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b680904e0 con 0x7f4b6818e140 2021-04-09T18:14:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b67ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b681ad040 0x7f4b6808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:57.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b67ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b681ad040 0x7f4b6808fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43724/0 (socket says 172.21.15.32:43724) 2021-04-09T18:14:57.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b67ac0700 1 -- 172.21.15.32:0/2543924432 learned_addr learned my addr 172.21.15.32:0/2543924432 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b672bf700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b680b5360 0x7f4b681a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.229 7f4b66abe700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6818e140 0x7f4b681aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:57.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b67ac0700 1 -- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6818e140 msgr2=0x7f4b681aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:57.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b67ac0700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6818e140 0x7f4b681aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:57.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b67ac0700 1 -- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b680b5360 msgr2=0x7f4b681a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b67ac0700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b680b5360 0x7f4b681a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:57.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b67ac0700 1 -- 172.21.15.32:0/2543924432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68183c80 con 0x7f4b681ad040 2021-04-09T18:14:57.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b67ac0700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b681ad040 0x7f4b6808fca0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f4b5000b690 tx=0x7f4b5000ba30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:57.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b5ffff700 1 -- 172.21.15.32:0/2543924432 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b50006cd0 con 0x7f4b681ad040 2021-04-09T18:14:57.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b5ffff700 1 -- 172.21.15.32:0/2543924432 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b50002030 con 0x7f4b681ad040 2021-04-09T18:14:57.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b5ffff700 1 -- 172.21.15.32:0/2543924432 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b50002480 con 0x7f4b681ad040 2021-04-09T18:14:57.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b68183c80 con 0x7f4b681ad040 2021-04-09T18:14:57.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b68183c80 con 0x7f4b681ad040 2021-04-09T18:14:57.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.230 7f4b5ffff700 1 -- 172.21.15.32:0/2543924432 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4b50002210 con 0x7f4b681ad040 2021-04-09T18:14:57.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.231 7f4b5ffff700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b4c0188d0 0x7f4b4c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:57.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.231 7f4b5ffff700 1 -- 172.21.15.32:0/2543924432 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4b50032a30 con 0x7f4b681ad040 2021-04-09T18:14:57.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.231 7f4b672bf700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b4c0188d0 0x7f4b4c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:57.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.231 7f4b672bf700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b4c0188d0 0x7f4b4c01add0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:57.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.250 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b68182d70 con 0x7f4b681ad040 2021-04-09T18:14:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.254 7f4b5ffff700 1 -- 172.21.15.32:0/2543924432 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4b50006560 con 0x7f4b681ad040 2021-04-09T18:14:57.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.445 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b68182d70 con 0x7f4b681ad040 2021-04-09T18:14:57.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.446 7f4b5ffff700 1 -- 172.21.15.32:0/2543924432 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f4b50002010 con 0x7f4b681ad040 2021-04-09T18:14:57.473 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:14:57.474 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:57.474 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:14:57.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b4c0188d0 msgr2=0x7f4b4c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:57.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b4c0188d0 0x7f4b4c01add0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:57.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b681ad040 msgr2=0x7f4b6808fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:57.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b681ad040 0x7f4b6808fca0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f4b5000b690 tx=0x7f4b5000ba30).stop 2021-04-09T18:14:57.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 shutdown_connections 2021-04-09T18:14:57.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b4c0188d0 0x7f4b4c01add0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:57.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b681ad040 0x7f4b6808fca0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b6818e140 0x7f4b681aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 --2- 172.21.15.32:0/2543924432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b680b5360 0x7f4b681a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 >> 172.21.15.32:0/2543924432 conn(0x7f4b6819b460 msgr2=0x7f4b6819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 shutdown_connections 2021-04-09T18:14:57.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:57.448 7f4b6d50c700 1 -- 172.21.15.32:0/2543924432 wait complete. 2021-04-09T18:14:57.892 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:57.893 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:14:58.894 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:14:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.019 7f1c16fff700 1 Processor -- start 2021-04-09T18:14:59.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.019 7f1c16fff700 1 -- start start 2021-04-09T18:14:59.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c16fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c100d7ca0 0x7f1c100d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:59.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c16fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1c1010da00 0x7f1c1010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c16fff700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1010d480 con 0x7f1c1010d010 2021-04-09T18:14:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c16fff700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c100d9220 con 0x7f1c1010da00 2021-04-09T18:14:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c16fff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100d86c0 con 0x7f1c100d7ca0 2021-04-09T18:14:59.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c1561c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1c1010da00 0x7f1c1010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c1561c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1c1010da00 0x7f1c1010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43732/0 (socket says 172.21.15.32:43732) 2021-04-09T18:14:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c1561c700 1 -- 172.21.15.32:0/2339568337 learned_addr learned my addr 172.21.15.32:0/2339568337 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:59.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c0ffff700 1 --2- 172.21.15.32:0/2339568337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c100d7ca0 0x7f1c100d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:59.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c1561c700 1 -- 172.21.15.32:0/2339568337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c100d7ca0 msgr2=0x7f1c100d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:14:59.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c1561c700 1 --2- 172.21.15.32:0/2339568337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c100d7ca0 0x7f1c100d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c1561c700 1 -- 172.21.15.32:0/2339568337 >> v1:172.21.15.86:6789/0 conn(0x7f1c1010d010 legacy=0x7f1c100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c0f7fe700 1 -- 172.21.15.32:0/2339568337 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958818935 0 0) 0x7f1bf8000c60 con 0x7f1c1010d010 2021-04-09T18:14:59.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c1561c700 1 -- 172.21.15.32:0/2339568337 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c10112100 con 0x7f1c1010da00 2021-04-09T18:14:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c1561c700 1 --2- 172.21.15.32:0/2339568337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c1010da00 0x7f1c1010de00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f1c08006f00 tx=0x7f1c08009f10).ready entity=mon.0 client_cookie=c851b511b749a986 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c0f7fe700 1 -- 172.21.15.32:0/2339568337 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c0800b3b0 con 0x7f1c1010da00 2021-04-09T18:14:59.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c0f7fe700 1 -- 172.21.15.32:0/2339568337 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0800b730 con 0x7f1c1010da00 2021-04-09T18:14:59.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.020 7f1c0f7fe700 1 -- 172.21.15.32:0/2339568337 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c0800bc10 con 0x7f1c1010da00 2021-04-09T18:14:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.021 7f1c16fff700 1 -- 172.21.15.32:0/2339568337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c1010da00 msgr2=0x7f1c1010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.021 7f1c16fff700 1 --2- 172.21.15.32:0/2339568337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c1010da00 0x7f1c1010de00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f1c08006f00 tx=0x7f1c08009f10).stop 2021-04-09T18:14:59.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.021 7f1c16fff700 1 -- 172.21.15.32:0/2339568337 shutdown_connections 2021-04-09T18:14:59.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.021 7f1c16fff700 1 --2- 172.21.15.32:0/2339568337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c1010da00 0x7f1c1010de00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.021 7f1c16fff700 1 --2- 172.21.15.32:0/2339568337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c100d7ca0 0x7f1c100d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.021 7f1c16fff700 1 -- 172.21.15.32:0/2339568337 >> 172.21.15.32:0/2339568337 conn(0x7f1c100ce5f0 msgr2=0x7f1c100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.021 7f1c16fff700 1 -- 172.21.15.32:0/2339568337 shutdown_connections 2021-04-09T18:14:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.021 7f1c16fff700 1 -- 172.21.15.32:0/2339568337 wait complete. 2021-04-09T18:14:59.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.041 7f1c16fff700 1 Processor -- start 2021-04-09T18:14:59.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.041 7f1c16fff700 1 -- start start 2021-04-09T18:14:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c16fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c100b42d0 0x7f1c10183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c16fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c101ac090 0x7f1c101ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c16fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c10184d30 0x7f1c10185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:59.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c16fff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100a1a90 con 0x7f1c100b42d0 2021-04-09T18:14:59.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c16fff700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c101acab0 con 0x7f1c10184d30 2021-04-09T18:14:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c16fff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c10185a00 con 0x7f1c101ac090 2021-04-09T18:14:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c14e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c100b42d0 0x7f1c10183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:59.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c14e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c100b42d0 0x7f1c10183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43738/0 (socket says 172.21.15.32:43738) 2021-04-09T18:14:59.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c14e1b700 1 -- 172.21.15.32:0/3789287583 learned_addr learned my addr 172.21.15.32:0/3789287583 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:14:59.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c1561c700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c10184d30 0x7f1c10185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:59.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c0ffff700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c101ac090 0x7f1c101ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:59.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c14e1b700 1 -- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c101ac090 msgr2=0x7f1c101ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:59.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c14e1b700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c101ac090 0x7f1c101ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c14e1b700 1 -- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c10184d30 msgr2=0x7f1c10185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:59.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c14e1b700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c10184d30 0x7f1c10185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c14e1b700 1 -- 172.21.15.32:0/3789287583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c10185750 con 0x7f1c100b42d0 2021-04-09T18:14:59.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c1561c700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c10184d30 0x7f1c10185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.042 7f1c14e1b700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c100b42d0 0x7f1c10183520 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f1bf800b580 tx=0x7f1bf800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:59.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.043 7f1c0d7fa700 1 -- 172.21.15.32:0/3789287583 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bf800bd10 con 0x7f1c100b42d0 2021-04-09T18:14:59.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.043 7f1c0d7fa700 1 -- 172.21.15.32:0/3789287583 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf8001080 con 0x7f1c100b42d0 2021-04-09T18:14:59.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.043 7f1c0d7fa700 1 -- 172.21.15.32:0/3789287583 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bf8001550 con 0x7f1c100b42d0 2021-04-09T18:14:59.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.043 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c10185750 con 0x7f1c100b42d0 2021-04-09T18:14:59.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.043 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c10185750 con 0x7f1c100b42d0 2021-04-09T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.043 7f1c0d7fa700 1 -- 172.21.15.32:0/3789287583 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1bf800a000 con 0x7f1c100b42d0 2021-04-09T18:14:59.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.044 7f1c0d7fa700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bf40188c0 0x7f1bf401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.044 7f1c0d7fa700 1 -- 172.21.15.32:0/3789287583 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1bf8032830 con 0x7f1c100b42d0 2021-04-09T18:14:59.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.044 7f1c0ffff700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bf40188c0 0x7f1bf401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:14:59.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.044 7f1c0ffff700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bf40188c0 0x7f1bf401adc0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:14:59.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.064 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c10182d10 con 0x7f1c100b42d0 2021-04-09T18:14:59.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.067 7f1c0d7fa700 1 -- 172.21.15.32:0/3789287583 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1bf8006bf0 con 0x7f1c100b42d0 2021-04-09T18:14:59.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.259 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c100a1a90 con 0x7f1c100b42d0 2021-04-09T18:14:59.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.259 7f1c0d7fa700 1 -- 172.21.15.32:0/3789287583 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f1bf800a000 con 0x7f1c100b42d0 2021-04-09T18:14:59.402 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:14:59.403 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:14:59.406 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:14:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bf40188c0 msgr2=0x7f1bf401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:59.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bf40188c0 0x7f1bf401adc0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c100b42d0 msgr2=0x7f1c10183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:14:59.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c100b42d0 0x7f1c10183520 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f1bf800b580 tx=0x7f1bf800b920).stop 2021-04-09T18:14:59.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 shutdown_connections 2021-04-09T18:14:59.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bf40188c0 0x7f1bf401adc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c10184d30 0x7f1c10185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c101ac090 0x7f1c101ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 --2- 172.21.15.32:0/3789287583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c100b42d0 0x7f1c10183520 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:14:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 >> 172.21.15.32:0/3789287583 conn(0x7f1c1019b890 msgr2=0x7f1c101a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:14:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 shutdown_connections 2021-04-09T18:14:59.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:14:59.262 7f1c16fff700 1 -- 172.21.15.32:0/3789287583 wait complete. 2021-04-09T18:14:59.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:14:59.961 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:15:00.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:01.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.093 7faf58750700 1 Processor -- start 2021-04-09T18:15:01.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.093 7faf58750700 1 -- start start 2021-04-09T18:15:01.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf58750700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500d7ca0 0x7faf500d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:01.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf58750700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faf5010da00 0x7faf5010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:01.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf58750700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5010d480 con 0x7faf5010d010 2021-04-09T18:15:01.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf58750700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf500d9220 con 0x7faf5010da00 2021-04-09T18:15:01.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf58750700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf500d86c0 con 0x7faf500d7ca0 2021-04-09T18:15:01.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf5656c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faf5010d010 0x7faf500d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53332/0 (socket says 172.21.15.32:53332) 2021-04-09T18:15:01.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf5656c700 1 -- 172.21.15.32:0/3103276116 learned_addr learned my addr 172.21.15.32:0/3103276116 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:01.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf56d6d700 1 --2- 172.21.15.32:0/3103276116 >> v2:172.21.15.86:3300/0 conn(0x7faf5010da00 0x7faf5010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:01.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf55d6b700 1 --2- 172.21.15.32:0/3103276116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500d7ca0 0x7faf500d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:01.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.094 7faf5556a700 1 -- 172.21.15.32:0/3103276116 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334725060 0 0) 0x7faf4c000ce0 con 0x7faf5010d010 2021-04-09T18:15:01.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf5556a700 1 -- 172.21.15.32:0/3103276116 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf3c002fe0 con 0x7faf5010d010 2021-04-09T18:15:01.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf5556a700 1 -- 172.21.15.32:0/3103276116 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2313482418 0 0) 0x7faf4c000ed0 con 0x7faf5010d010 2021-04-09T18:15:01.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf55d6b700 1 -- 172.21.15.32:0/3103276116 >> v2:172.21.15.86:3300/0 conn(0x7faf5010da00 msgr2=0x7faf5010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:01.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf55d6b700 1 --2- 172.21.15.32:0/3103276116 >> v2:172.21.15.86:3300/0 conn(0x7faf5010da00 0x7faf5010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf55d6b700 1 -- 172.21.15.32:0/3103276116 >> v1:172.21.15.32:6789/0 conn(0x7faf5010d010 legacy=0x7faf500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:01.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf55d6b700 1 -- 172.21.15.32:0/3103276116 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf500d1f70 con 0x7faf500d7ca0 2021-04-09T18:15:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf56d6d700 1 --2- 172.21.15.32:0/3103276116 >> v2:172.21.15.86:3300/0 conn(0x7faf5010da00 0x7faf5010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf55d6b700 1 --2- 172.21.15.32:0/3103276116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500d7ca0 0x7faf500d80a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7faf40006650 tx=0x7faf40009a80).ready entity=mon.2 client_cookie=ba000ff24c307f94 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf5556a700 1 -- 172.21.15.32:0/3103276116 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf400009b0 con 0x7faf500d7ca0 2021-04-09T18:15:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf5556a700 1 -- 172.21.15.32:0/3103276116 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4000c0c0 con 0x7faf500d7ca0 2021-04-09T18:15:01.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf5556a700 1 -- 172.21.15.32:0/3103276116 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf4000c510 con 0x7faf500d7ca0 2021-04-09T18:15:01.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf58750700 1 -- 172.21.15.32:0/3103276116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500d7ca0 msgr2=0x7faf500d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.095 7faf58750700 1 --2- 172.21.15.32:0/3103276116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500d7ca0 0x7faf500d80a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7faf40006650 tx=0x7faf40009a80).stop 2021-04-09T18:15:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.096 7faf58750700 1 -- 172.21.15.32:0/3103276116 shutdown_connections 2021-04-09T18:15:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.096 7faf58750700 1 --2- 172.21.15.32:0/3103276116 >> v2:172.21.15.86:3300/0 conn(0x7faf5010da00 0x7faf5010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.096 7faf58750700 1 --2- 172.21.15.32:0/3103276116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500d7ca0 0x7faf500d80a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.096 7faf58750700 1 -- 172.21.15.32:0/3103276116 >> 172.21.15.32:0/3103276116 conn(0x7faf500ce5f0 msgr2=0x7faf500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.096 7faf58750700 1 -- 172.21.15.32:0/3103276116 shutdown_connections 2021-04-09T18:15:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.096 7faf58750700 1 -- 172.21.15.32:0/3103276116 wait complete. 2021-04-09T18:15:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.116 7faf58750700 1 Processor -- start 2021-04-09T18:15:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.116 7faf58750700 1 -- start start 2021-04-09T18:15:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf58750700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf50179bf0 0x7faf50179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf58750700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf5017c540 0x7faf5017c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf58750700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500aca60 0x7faf500ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:01.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf58750700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5017cf60 con 0x7faf50179bf0 2021-04-09T18:15:01.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf58750700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf500ad480 con 0x7faf5017c540 2021-04-09T18:15:01.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf58750700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf500ad6a0 con 0x7faf500aca60 2021-04-09T18:15:01.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf5656c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf50179bf0 0x7faf50179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:01.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf5656c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf50179bf0 0x7faf50179ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43748/0 (socket says 172.21.15.32:43748) 2021-04-09T18:15:01.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf5656c700 1 -- 172.21.15.32:0/3104163413 learned_addr learned my addr 172.21.15.32:0/3104163413 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:01.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf55d6b700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf5017c540 0x7faf5017c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:01.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf56d6d700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500aca60 0x7faf500ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:01.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf5656c700 1 -- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500aca60 msgr2=0x7faf500ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:01.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf5656c700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500aca60 0x7faf500ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf5656c700 1 -- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf5017c540 msgr2=0x7faf5017c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:01.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf5656c700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf5017c540 0x7faf5017c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf5656c700 1 -- 172.21.15.32:0/3104163413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf500ad8c0 con 0x7faf50179bf0 2021-04-09T18:15:01.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf55d6b700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf5017c540 0x7faf5017c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:01.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf5656c700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf50179bf0 0x7faf50179ff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7faf4c00a4d0 tx=0x7faf4c00a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:01.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.117 7faf46ffd700 1 -- 172.21.15.32:0/3104163413 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf4c0070d0 con 0x7faf50179bf0 2021-04-09T18:15:01.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.118 7faf46ffd700 1 -- 172.21.15.32:0/3104163413 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4c00ac60 con 0x7faf50179bf0 2021-04-09T18:15:01.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.118 7faf46ffd700 1 -- 172.21.15.32:0/3104163413 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf4c004830 con 0x7faf50179bf0 2021-04-09T18:15:01.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.118 7faf58750700 1 -- 172.21.15.32:0/3104163413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf500ad8c0 con 0x7faf50179bf0 2021-04-09T18:15:01.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.118 7faf58750700 1 -- 172.21.15.32:0/3104163413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf500ad8c0 con 0x7faf50179bf0 2021-04-09T18:15:01.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.119 7faf46ffd700 1 -- 172.21.15.32:0/3104163413 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faf4c0043a0 con 0x7faf50179bf0 2021-04-09T18:15:01.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.119 7faf46ffd700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf380188c0 0x7faf3801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:01.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.119 7faf46ffd700 1 -- 172.21.15.32:0/3104163413 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7faf4c032760 con 0x7faf50179bf0 2021-04-09T18:15:01.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.119 7faf55d6b700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf380188c0 0x7faf3801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:01.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.119 7faf55d6b700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf380188c0 0x7faf3801adc0 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:01.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.138 7faf58750700 1 -- 172.21.15.32:0/3104163413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5017cf60 con 0x7faf50179bf0 2021-04-09T18:15:01.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.142 7faf46ffd700 1 -- 172.21.15.32:0/3104163413 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faf4c0011c0 con 0x7faf50179bf0 2021-04-09T18:15:01.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.334 7faf58750700 1 -- 172.21.15.32:0/3104163413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf5017cf60 con 0x7faf50179bf0 2021-04-09T18:15:01.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.334 7faf46ffd700 1 -- 172.21.15.32:0/3104163413 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7faf4c0043a0 con 0x7faf50179bf0 2021-04-09T18:15:01.474 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:15:01.475 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:01.476 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:01.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 -- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf380188c0 msgr2=0x7faf3801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf380188c0 0x7faf3801adc0 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 -- 172.21.15.32:0/3104163413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf50179bf0 msgr2=0x7faf50179ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:01.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf50179bf0 0x7faf50179ff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7faf4c00a4d0 tx=0x7faf4c00a870).stop 2021-04-09T18:15:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 -- 172.21.15.32:0/3104163413 shutdown_connections 2021-04-09T18:15:01.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faf380188c0 0x7faf3801adc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf500aca60 0x7faf500ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf5017c540 0x7faf5017c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 --2- 172.21.15.32:0/3104163413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf50179bf0 0x7faf50179ff0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:01.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 -- 172.21.15.32:0/3104163413 >> 172.21.15.32:0/3104163413 conn(0x7faf501acae0 msgr2=0x7faf5017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:01.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 -- 172.21.15.32:0/3104163413 shutdown_connections 2021-04-09T18:15:01.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:01.337 7faf58750700 1 -- 172.21.15.32:0/3104163413 wait complete. 2021-04-09T18:15:01.978 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:01.979 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:15:02.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:03.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.117 7fcd6e547700 1 Processor -- start 2021-04-09T18:15:03.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.117 7fcd6e547700 1 -- start start 2021-04-09T18:15:03.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.118 7fcd6e547700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcd680d6c80 0x7fcd680d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:03.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.118 7fcd6e547700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd680d7280 con 0x7fcd680cffa0 2021-04-09T18:15:03.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.118 7fcd6e547700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd680d2840 con 0x7fcd680d2440 2021-04-09T18:15:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.118 7fcd6e547700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd680d7080 con 0x7fcd680d6c80 2021-04-09T18:15:03.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.118 7fcd67fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcd680cffa0 0x7fcd680d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53346/0 (socket says 172.21.15.32:53346) 2021-04-09T18:15:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.118 7fcd67fff700 1 -- 172.21.15.32:0/3871824748 learned_addr learned my addr 172.21.15.32:0/3871824748 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.118 7fcd5ffff700 1 --2- 172.21.15.32:0/3871824748 >> v2:172.21.15.86:3300/0 conn(0x7fcd680d6c80 0x7fcd680d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.118 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45509772 0 0) 0x7fcd58000ce0 con 0x7fcd680cffa0 2021-04-09T18:15:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.118 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd54002fe0 con 0x7fcd680cffa0 2021-04-09T18:15:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1317839209 0 0) 0x7fcd580010e0 con 0x7fcd680cffa0 2021-04-09T18:15:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 >> v2:172.21.15.86:3300/0 conn(0x7fcd680d6c80 msgr2=0x7fcd680d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 --2- 172.21.15.32:0/3871824748 >> v2:172.21.15.86:3300/0 conn(0x7fcd680d6c80 0x7fcd680d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 >> v1:172.21.15.86:6789/0 conn(0x7fcd680d2440 legacy=0x7fcd6810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:03.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd680d91e0 con 0x7fcd680cffa0 2021-04-09T18:15:03.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fcd58001600 con 0x7fcd680cffa0 2021-04-09T18:15:03.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 >> v1:172.21.15.32:6789/0 conn(0x7fcd680cffa0 legacy=0x7fcd680d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:03.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd54003a60 0x7fcd54005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd54006790 0x7fcd54008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd54009590 0x7fcd5400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5400c300 con 0x7fcd54006790 2021-04-09T18:15:03.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5400c520 con 0x7fcd54009590 2021-04-09T18:15:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd54050750 con 0x7fcd54003a60 2021-04-09T18:15:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd5ffff700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd54006790 0x7fcd54008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd67fff700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd54003a60 0x7fcd54005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:03.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd6cb64700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd54009590 0x7fcd5400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:03.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd5ffff700 1 -- 172.21.15.32:0/3871824748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd54003a60 msgr2=0x7fcd54005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:03.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd5ffff700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd54003a60 0x7fcd54005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd5ffff700 1 -- 172.21.15.32:0/3871824748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd54009590 msgr2=0x7fcd5400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:03.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd5ffff700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd54009590 0x7fcd5400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd5ffff700 1 -- 172.21.15.32:0/3871824748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd540509b0 con 0x7fcd54006790 2021-04-09T18:15:03.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd6cb64700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd54009590 0x7fcd5400bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:03.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd6cb64700 1 -- 172.21.15.32:0/3871824748 reap_dead start 2021-04-09T18:15:03.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd5ffff700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd54006790 0x7fcd54008c90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fcd5000c870 tx=0x7fcd5000cc10).ready entity=mon.0 client_cookie=3dc325503681bd6c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:03.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd500090d0 con 0x7fcd54006790 2021-04-09T18:15:03.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd50009450 con 0x7fcd54006790 2021-04-09T18:15:03.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd677fe700 1 -- 172.21.15.32:0/3871824748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd500098b0 con 0x7fcd54006790 2021-04-09T18:15:03.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd6e547700 1 -- 172.21.15.32:0/3871824748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd54006790 msgr2=0x7fcd54008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:03.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.119 7fcd6e547700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd54006790 0x7fcd54008c90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fcd5000c870 tx=0x7fcd5000cc10).stop 2021-04-09T18:15:03.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.120 7fcd6e547700 1 -- 172.21.15.32:0/3871824748 shutdown_connections 2021-04-09T18:15:03.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.120 7fcd6e547700 1 --2- 172.21.15.32:0/3871824748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd54006790 0x7fcd54008c90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.120 7fcd6e547700 1 -- 172.21.15.32:0/3871824748 >> 172.21.15.32:0/3871824748 conn(0x7fcd680ce5c0 msgr2=0x7fcd680cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:03.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.120 7fcd6e547700 1 -- 172.21.15.32:0/3871824748 shutdown_connections 2021-04-09T18:15:03.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.120 7fcd6e547700 1 -- 172.21.15.32:0/3871824748 wait complete. 2021-04-09T18:15:03.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.140 7fcd6e547700 1 Processor -- start 2021-04-09T18:15:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6e547700 1 -- start start 2021-04-09T18:15:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6e547700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd680aa220 0x7fcd680a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:03.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6e547700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd68193440 0x7fcd680b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6e547700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd680b5530 0x7fcd68197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6e547700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd68198b40 con 0x7fcd680b5530 2021-04-09T18:15:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6e547700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680aa6a0 con 0x7fcd68193440 2021-04-09T18:15:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6e547700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd68198510 con 0x7fcd680aa220 2021-04-09T18:15:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6cb64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd680b5530 0x7fcd68197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6cb64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd680b5530 0x7fcd68197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43772/0 (socket says 172.21.15.32:43772) 2021-04-09T18:15:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd6cb64700 1 -- 172.21.15.32:0/2470466815 learned_addr learned my addr 172.21.15.32:0/2470466815 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd5ffff700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd68193440 0x7fcd680b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:03.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.141 7fcd67fff700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd680aa220 0x7fcd680a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:03.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd6cb64700 1 -- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd680aa220 msgr2=0x7fcd680a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:03.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd6cb64700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd680aa220 0x7fcd680a99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd6cb64700 1 -- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd68193440 msgr2=0x7fcd680b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:03.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd6cb64700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd68193440 0x7fcd680b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd6cb64700 1 -- 172.21.15.32:0/2470466815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68198730 con 0x7fcd680b5530 2021-04-09T18:15:03.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd5ffff700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd68193440 0x7fcd680b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:03.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd6cb64700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd680b5530 0x7fcd68197ef0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fcd600099a0 tx=0x7fcd6000bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:03.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd657fa700 1 -- 172.21.15.32:0/2470466815 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd60000f40 con 0x7fcd680b5530 2021-04-09T18:15:03.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd657fa700 1 -- 172.21.15.32:0/2470466815 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd600011a0 con 0x7fcd680b5530 2021-04-09T18:15:03.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd68198730 con 0x7fcd680b5530 2021-04-09T18:15:03.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd657fa700 1 -- 172.21.15.32:0/2470466815 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd600016c0 con 0x7fcd680b5530 2021-04-09T18:15:03.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.142 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd68198730 con 0x7fcd680b5530 2021-04-09T18:15:03.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.143 7fcd657fa700 1 -- 172.21.15.32:0/2470466815 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcd60001380 con 0x7fcd680b5530 2021-04-09T18:15:03.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.143 7fcd657fa700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd4c0188c0 0x7fcd4c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:03.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.143 7fcd657fa700 1 -- 172.21.15.32:0/2470466815 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcd60032b30 con 0x7fcd680b5530 2021-04-09T18:15:03.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.143 7fcd67fff700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd4c0188c0 0x7fcd4c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:03.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.143 7fcd67fff700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd4c0188c0 0x7fcd4c01adc0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:03.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.162 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd68182e10 con 0x7fcd680b5530 2021-04-09T18:15:03.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.165 7fcd657fa700 1 -- 172.21.15.32:0/2470466815 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcd6000bdb0 con 0x7fcd680b5530 2021-04-09T18:15:03.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.356 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd680d7280 con 0x7fcd680b5530 2021-04-09T18:15:03.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.356 7fcd657fa700 1 -- 172.21.15.32:0/2470466815 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7fcd60026f20 con 0x7fcd680b5530 2021-04-09T18:15:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:15:03.384 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:03.385 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:03.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.358 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd4c0188c0 msgr2=0x7fcd4c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:03.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.358 7fcd6e547700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd4c0188c0 0x7fcd4c01adc0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.358 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd680b5530 msgr2=0x7fcd68197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:03.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.358 7fcd6e547700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd680b5530 0x7fcd68197ef0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fcd600099a0 tx=0x7fcd6000bc70).stop 2021-04-09T18:15:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.358 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 shutdown_connections 2021-04-09T18:15:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.358 7fcd6e547700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd4c0188c0 0x7fcd4c01adc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.358 7fcd6e547700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd680b5530 0x7fcd68197ef0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.358 7fcd6e547700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd68193440 0x7fcd680b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.358 7fcd6e547700 1 --2- 172.21.15.32:0/2470466815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd680aa220 0x7fcd680a99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:03.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.359 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 >> 172.21.15.32:0/2470466815 conn(0x7fcd681a0320 msgr2=0x7fcd6819ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:03.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.359 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 shutdown_connections 2021-04-09T18:15:03.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:03.359 7fcd6e547700 1 -- 172.21.15.32:0/2470466815 wait complete. 2021-04-09T18:15:03.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:03.828 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:15:04.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:15:04.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:15:04.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:05.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.954 7f30b221b700 1 Processor -- start 2021-04-09T18:15:05.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.954 7f30b221b700 1 -- start start 2021-04-09T18:15:05.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30b221b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f30ac10d010 0x7f30ac0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:05.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30b221b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac0d1da0 0x7f30ac0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:05.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30b221b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac10d480 con 0x7f30ac0d82c0 2021-04-09T18:15:05.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30b221b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ac0d9220 con 0x7f30ac10d010 2021-04-09T18:15:05.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30b221b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0d86c0 con 0x7f30ac0d1da0 2021-04-09T18:15:05.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30a3fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac0d1da0 0x7f30ac0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:05.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30abfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f30ac10d010 0x7f30ac0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:05.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30b0838700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f30ac0d82c0 0x7f30ac10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:51294/0 (socket says 172.21.15.32:51294) 2021-04-09T18:15:05.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30a3fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac0d1da0 0x7f30ac0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:51002/0 (socket says 172.21.15.32:51002) 2021-04-09T18:15:05.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30abfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f30ac10d010 0x7f30ac0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:35534/0 (socket says 172.21.15.32:35534) 2021-04-09T18:15:05.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30b0838700 1 -- 172.21.15.32:0/1749908974 learned_addr learned my addr 172.21.15.32:0/1749908974 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:05.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30abfff700 1 -- 172.21.15.32:0/1749908974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac0d1da0 msgr2=0x7f30ac0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:05.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.955 7f30abfff700 1 --2- 172.21.15.32:0/1749908974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac0d1da0 0x7f30ac0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30abfff700 1 -- 172.21.15.32:0/1749908974 >> v1:172.21.15.86:6789/0 conn(0x7f30ac0d82c0 legacy=0x7f30ac10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:05.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30ab7fe700 1 -- 172.21.15.32:0/1749908974 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099740477 0 0) 0x7f30a4001440 con 0x7f30ac0d82c0 2021-04-09T18:15:05.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30abfff700 1 -- 172.21.15.32:0/1749908974 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac10e1f0 con 0x7f30ac10d010 2021-04-09T18:15:05.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30a3fff700 1 --2- 172.21.15.32:0/1749908974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac0d1da0 0x7f30ac0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:05.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30abfff700 1 --2- 172.21.15.32:0/1749908974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30ac10d010 0x7f30ac0d1780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f309c006650 tx=0x7f309c009a80).ready entity=mon.1 client_cookie=8e784146b82c4f8f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:05.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30ab7fe700 1 -- 172.21.15.32:0/1749908974 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309c0009b0 con 0x7f30ac10d010 2021-04-09T18:15:05.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30ab7fe700 1 -- 172.21.15.32:0/1749908974 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00c0c0 con 0x7f30ac10d010 2021-04-09T18:15:05.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30ab7fe700 1 -- 172.21.15.32:0/1749908974 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309c0009b0 con 0x7f30ac10d010 2021-04-09T18:15:05.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30b221b700 1 -- 172.21.15.32:0/1749908974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30ac10d010 msgr2=0x7f30ac0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:05.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.956 7f30b221b700 1 --2- 172.21.15.32:0/1749908974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30ac10d010 0x7f30ac0d1780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f309c006650 tx=0x7f309c009a80).stop 2021-04-09T18:15:05.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.957 7f30b221b700 1 -- 172.21.15.32:0/1749908974 shutdown_connections 2021-04-09T18:15:05.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.957 7f30b221b700 1 --2- 172.21.15.32:0/1749908974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac0d1da0 0x7f30ac0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.957 7f30b221b700 1 --2- 172.21.15.32:0/1749908974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30ac10d010 0x7f30ac0d1780 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.957 7f30b221b700 1 -- 172.21.15.32:0/1749908974 >> 172.21.15.32:0/1749908974 conn(0x7f30ac0ce5f0 msgr2=0x7f30ac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.957 7f30b221b700 1 -- 172.21.15.32:0/1749908974 shutdown_connections 2021-04-09T18:15:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.957 7f30b221b700 1 -- 172.21.15.32:0/1749908974 wait complete. 2021-04-09T18:15:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.977 7f30b221b700 1 Processor -- start 2021-04-09T18:15:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.977 7f30b221b700 1 -- start start 2021-04-09T18:15:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30b221b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30ac186770 0x7f30ac181690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30b221b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac181cb0 0x7f30ac08eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:05.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30b221b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30ac08f0d0 0x7f30ac1a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30b221b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac1820b0 con 0x7f30ac186770 2021-04-09T18:15:05.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30b221b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac186b70 con 0x7f30ac08f0d0 2021-04-09T18:15:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30b221b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac08f4d0 con 0x7f30ac181cb0 2021-04-09T18:15:05.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30abfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30ac186770 0x7f30ac181690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:05.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30abfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30ac186770 0x7f30ac181690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43782/0 (socket says 172.21.15.32:43782) 2021-04-09T18:15:05.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30abfff700 1 -- 172.21.15.32:0/3960891987 learned_addr learned my addr 172.21.15.32:0/3960891987 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:05.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30a3fff700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac181cb0 0x7f30ac08eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:05.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30b0838700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30ac08f0d0 0x7f30ac1a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:05.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30abfff700 1 -- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac181cb0 msgr2=0x7f30ac08eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:05.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30abfff700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac181cb0 0x7f30ac08eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30abfff700 1 -- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30ac08f0d0 msgr2=0x7f30ac1a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:05.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30abfff700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30ac08f0d0 0x7f30ac1a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30abfff700 1 -- 172.21.15.32:0/3960891987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac1a5d00 con 0x7f30ac186770 2021-04-09T18:15:05.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30abfff700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30ac186770 0x7f30ac181690 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f309c009650 tx=0x7f309c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:05.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30a97fa700 1 -- 172.21.15.32:0/3960891987 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309c0093a0 con 0x7f30ac186770 2021-04-09T18:15:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30a97fa700 1 -- 172.21.15.32:0/3960891987 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00bd50 con 0x7f30ac186770 2021-04-09T18:15:05.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30a97fa700 1 -- 172.21.15.32:0/3960891987 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309c0052e0 con 0x7f30ac186770 2021-04-09T18:15:05.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.978 7f30b221b700 1 -- 172.21.15.32:0/3960891987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac1a5d00 con 0x7f30ac186770 2021-04-09T18:15:05.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.979 7f30b221b700 1 -- 172.21.15.32:0/3960891987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac1a5d00 con 0x7f30ac186770 2021-04-09T18:15:05.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.979 7f30a97fa700 1 -- 172.21.15.32:0/3960891987 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f309c006030 con 0x7f30ac186770 2021-04-09T18:15:05.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.979 7f30a97fa700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3090018930 0x7f309001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:05.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.979 7f30a97fa700 1 -- 172.21.15.32:0/3960891987 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f309c0329e0 con 0x7f30ac186770 2021-04-09T18:15:05.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.979 7f30a3fff700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3090018930 0x7f309001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:05.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.980 7f30a3fff700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3090018930 0x7f309001ae10 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:05.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:04.999 7f30b221b700 1 -- 172.21.15.32:0/3960891987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac181160 con 0x7f30ac186770 2021-04-09T18:15:05.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.002 7f30a97fa700 1 -- 172.21.15.32:0/3960891987 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f309c000940 con 0x7f30ac186770 2021-04-09T18:15:05.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.194 7f30b221b700 1 -- 172.21.15.32:0/3960891987 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30ac181160 con 0x7f30ac186770 2021-04-09T18:15:05.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.194 7f30a97fa700 1 -- 172.21.15.32:0/3960891987 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f309c00bd50 con 0x7f30ac186770 2021-04-09T18:15:05.224 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:15:05.225 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:05.225 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:05.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 -- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3090018930 msgr2=0x7f309001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:05.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3090018930 0x7f309001ae10 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 -- 172.21.15.32:0/3960891987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30ac186770 msgr2=0x7f30ac181690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:05.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30ac186770 0x7f30ac181690 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f309c009650 tx=0x7f309c00b960).stop 2021-04-09T18:15:05.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 -- 172.21.15.32:0/3960891987 shutdown_connections 2021-04-09T18:15:05.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3090018930 0x7f309001ae10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30ac08f0d0 0x7f30ac1a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30ac181cb0 0x7f30ac08eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 --2- 172.21.15.32:0/3960891987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30ac186770 0x7f30ac181690 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:05.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 -- 172.21.15.32:0/3960891987 >> 172.21.15.32:0/3960891987 conn(0x7f30ac195e70 msgr2=0x7f30ac1a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:05.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 -- 172.21.15.32:0/3960891987 shutdown_connections 2021-04-09T18:15:05.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:05.197 7f30b221b700 1 -- 172.21.15.32:0/3960891987 wait complete. 2021-04-09T18:15:05.683 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:05.684 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:15:06.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:06.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.820 7fb4048b9700 1 Processor -- start 2021-04-09T18:15:06.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.820 7fb4048b9700 1 -- start start 2021-04-09T18:15:06.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.820 7fb4048b9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb4000d7ca0 0x7fb4000d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:06.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.820 7fb4048b9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40010d480 con 0x7fb40010d010 2021-04-09T18:15:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.820 7fb4048b9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4000d86c0 con 0x7fb40010da00 2021-04-09T18:15:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.820 7fb4048b9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4000d9220 con 0x7fb4000d7ca0 2021-04-09T18:15:06.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fe61c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb40010d010 0x7fb4000d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53378/0 (socket says 172.21.15.32:53378) 2021-04-09T18:15:06.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fde1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb4000d7ca0 0x7fb4000d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:06.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fe61c700 1 -- 172.21.15.32:0/2417959495 learned_addr learned my addr 172.21.15.32:0/2417959495 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:06.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fd61a700 1 -- 172.21.15.32:0/2417959495 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192613805 0 0) 0x7fb3f0000ce0 con 0x7fb40010d010 2021-04-09T18:15:06.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fd61a700 1 -- 172.21.15.32:0/2417959495 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e8002fe0 con 0x7fb40010d010 2021-04-09T18:15:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fd61a700 1 -- 172.21.15.32:0/2417959495 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231596283 0 0) 0x7fb3f8000d30 con 0x7fb40010da00 2021-04-09T18:15:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fd61a700 1 -- 172.21.15.32:0/2417959495 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e8004370 con 0x7fb40010da00 2021-04-09T18:15:06.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fde1b700 1 -- 172.21.15.32:0/2417959495 >> v1:172.21.15.86:6789/0 conn(0x7fb40010da00 legacy=0x7fb40010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:06.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fde1b700 1 -- 172.21.15.32:0/2417959495 >> v1:172.21.15.32:6789/0 conn(0x7fb40010d010 legacy=0x7fb4000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fde1b700 1 -- 172.21.15.32:0/2417959495 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4000d1f70 con 0x7fb4000d7ca0 2021-04-09T18:15:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fde1b700 1 --2- 172.21.15.32:0/2417959495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000d7ca0 0x7fb4000d80a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fb3f4006650 tx=0x7fb3f4009a80).ready entity=mon.0 client_cookie=2c46c52503579c69 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:06.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fd61a700 1 -- 172.21.15.32:0/2417959495 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f40009b0 con 0x7fb4000d7ca0 2021-04-09T18:15:06.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fd61a700 1 -- 172.21.15.32:0/2417959495 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f400c0c0 con 0x7fb4000d7ca0 2021-04-09T18:15:06.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb3fd61a700 1 -- 172.21.15.32:0/2417959495 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f400c510 con 0x7fb4000d7ca0 2021-04-09T18:15:06.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb4048b9700 1 -- 172.21.15.32:0/2417959495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000d7ca0 msgr2=0x7fb4000d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:06.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.821 7fb4048b9700 1 --2- 172.21.15.32:0/2417959495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000d7ca0 0x7fb4000d80a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fb3f4006650 tx=0x7fb3f4009a80).stop 2021-04-09T18:15:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.822 7fb4048b9700 1 -- 172.21.15.32:0/2417959495 shutdown_connections 2021-04-09T18:15:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.822 7fb4048b9700 1 --2- 172.21.15.32:0/2417959495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000d7ca0 0x7fb4000d80a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:06.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.822 7fb4048b9700 1 -- 172.21.15.32:0/2417959495 >> 172.21.15.32:0/2417959495 conn(0x7fb4000ce5f0 msgr2=0x7fb4000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:06.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.822 7fb4048b9700 1 -- 172.21.15.32:0/2417959495 shutdown_connections 2021-04-09T18:15:06.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.822 7fb4048b9700 1 -- 172.21.15.32:0/2417959495 wait complete. 2021-04-09T18:15:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.842 7fb4048b9700 1 Processor -- start 2021-04-09T18:15:06.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.842 7fb4048b9700 1 -- start start 2021-04-09T18:15:06.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb4048b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000ab300 0x7fb4001a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:06.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb4048b9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4001a8ad0 0x7fb4001a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:06.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb4048b9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb400180db0 0x7fb4001811b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:06.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb4048b9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4001817d0 con 0x7fb4000ab300 2021-04-09T18:15:06.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb4048b9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4001819f0 con 0x7fb4001a8ad0 2021-04-09T18:15:06.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb4048b9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4001a1f80 con 0x7fb400180db0 2021-04-09T18:15:06.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fe61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000ab300 0x7fb4001a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:06.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fe61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000ab300 0x7fb4001a12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43798/0 (socket says 172.21.15.32:43798) 2021-04-09T18:15:06.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fe61c700 1 -- 172.21.15.32:0/1638109537 learned_addr learned my addr 172.21.15.32:0/1638109537 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:06.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fde1b700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4001a8ad0 0x7fb4001a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:06.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fee1d700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb400180db0 0x7fb4001811b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:06.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fe61c700 1 -- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb400180db0 msgr2=0x7fb4001811b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:06.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fe61c700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb400180db0 0x7fb4001811b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:06.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fe61c700 1 -- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4001a8ad0 msgr2=0x7fb4001a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:06.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fe61c700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4001a8ad0 0x7fb4001a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:06.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fe61c700 1 -- 172.21.15.32:0/1638109537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4001a21a0 con 0x7fb4000ab300 2021-04-09T18:15:06.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.843 7fb3fde1b700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4001a8ad0 0x7fb4001a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:06.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.844 7fb3fe61c700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000ab300 0x7fb4001a12d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb3f0006f00 tx=0x7fb3f000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:06.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.844 7fb3eeffd700 1 -- 172.21.15.32:0/1638109537 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f000bd30 con 0x7fb4000ab300 2021-04-09T18:15:06.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.844 7fb3eeffd700 1 -- 172.21.15.32:0/1638109537 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f0006030 con 0x7fb4000ab300 2021-04-09T18:15:06.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.844 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4001a21a0 con 0x7fb4000ab300 2021-04-09T18:15:07.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.844 7fb3eeffd700 1 -- 172.21.15.32:0/1638109537 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3f0004620 con 0x7fb4000ab300 2021-04-09T18:15:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.844 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4001a21a0 con 0x7fb4000ab300 2021-04-09T18:15:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.845 7fb3eeffd700 1 -- 172.21.15.32:0/1638109537 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb3f00043a0 con 0x7fb4000ab300 2021-04-09T18:15:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.845 7fb3eeffd700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3e40188c0 0x7fb3e401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.845 7fb3eeffd700 1 -- 172.21.15.32:0/1638109537 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb3f0032730 con 0x7fb4000ab300 2021-04-09T18:15:07.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.845 7fb3fde1b700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3e40188c0 0x7fb3e401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:07.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.846 7fb3fde1b700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3e40188c0 0x7fb3e401adc0 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.865 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4001b7070 con 0x7fb4000ab300 2021-04-09T18:15:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:06.868 7fb3eeffd700 1 -- 172.21.15.32:0/1638109537 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb3f00009b0 con 0x7fb4000ab300 2021-04-09T18:15:07.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.061 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb40010d480 con 0x7fb4000ab300 2021-04-09T18:15:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.062 7fb3eeffd700 1 -- 172.21.15.32:0/1638109537 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7fb3f000bd30 con 0x7fb4000ab300 2021-04-09T18:15:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:15:07.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:07.093 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:07.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3e40188c0 msgr2=0x7fb3e401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:07.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3e40188c0 0x7fb3e401adc0 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:07.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000ab300 msgr2=0x7fb4001a12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:07.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000ab300 0x7fb4001a12d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb3f0006f00 tx=0x7fb3f000b940).stop 2021-04-09T18:15:07.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 shutdown_connections 2021-04-09T18:15:07.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3e40188c0 0x7fb3e401adc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:07.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb400180db0 0x7fb4001811b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:07.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4001a8ad0 0x7fb4001a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:07.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 --2- 172.21.15.32:0/1638109537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4000ab300 0x7fb4001a12d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:07.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.064 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 >> 172.21.15.32:0/1638109537 conn(0x7fb4001aa2b0 msgr2=0x7fb4001a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:07.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.065 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 shutdown_connections 2021-04-09T18:15:07.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:07.065 7fb4048b9700 1 -- 172.21.15.32:0/1638109537 wait complete. 2021-04-09T18:15:07.631 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:07.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:15:08.633 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:08.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.769 7f613f6ea700 1 Processor -- start 2021-04-09T18:15:08.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.769 7f613f6ea700 1 -- start start 2021-04-09T18:15:08.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613f6ea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613810da00 0x7f613810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:08.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613f6ea700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613810d480 con 0x7f613810d010 2021-04-09T18:15:08.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613f6ea700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61380d1f70 con 0x7f61380d7ca0 2021-04-09T18:15:08.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613f6ea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61380d9220 con 0x7f613810da00 2021-04-09T18:15:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613d506700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f613810d010 0x7f61380d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53392/0 (socket says 172.21.15.32:53392) 2021-04-09T18:15:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613d506700 1 -- 172.21.15.32:0/989659046 learned_addr learned my addr 172.21.15.32:0/989659046 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:08.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613dd07700 1 --2- 172.21.15.32:0/989659046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613810da00 0x7f613810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f612bfff700 1 -- 172.21.15.32:0/989659046 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419476271 0 0) 0x7f6134000ce0 con 0x7f613810d010 2021-04-09T18:15:08.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f612bfff700 1 -- 172.21.15.32:0/989659046 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6124002fe0 con 0x7f613810d010 2021-04-09T18:15:08.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f612bfff700 1 -- 172.21.15.32:0/989659046 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592172221 0 0) 0x7f612c000c60 con 0x7f61380d7ca0 2021-04-09T18:15:08.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f612bfff700 1 -- 172.21.15.32:0/989659046 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6124003210 con 0x7f61380d7ca0 2021-04-09T18:15:08.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613dd07700 1 -- 172.21.15.32:0/989659046 >> v1:172.21.15.86:6789/0 conn(0x7f61380d7ca0 legacy=0x7f61380d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:08.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613dd07700 1 -- 172.21.15.32:0/989659046 >> v1:172.21.15.32:6789/0 conn(0x7f613810d010 legacy=0x7f61380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:08.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613dd07700 1 -- 172.21.15.32:0/989659046 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613810e420 con 0x7f613810da00 2021-04-09T18:15:08.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.770 7f613dd07700 1 --2- 172.21.15.32:0/989659046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613810da00 0x7f613810de00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6130006e50 tx=0x7f6130009e70).ready entity=mon.2 client_cookie=54cd065eed87d20b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:08.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.771 7f612bfff700 1 -- 172.21.15.32:0/989659046 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6130006a20 con 0x7f613810da00 2021-04-09T18:15:08.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.771 7f612bfff700 1 -- 172.21.15.32:0/989659046 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6130006790 con 0x7f613810da00 2021-04-09T18:15:08.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.771 7f612bfff700 1 -- 172.21.15.32:0/989659046 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f613000b700 con 0x7f613810da00 2021-04-09T18:15:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.771 7f613f6ea700 1 -- 172.21.15.32:0/989659046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613810da00 msgr2=0x7f613810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.771 7f613f6ea700 1 --2- 172.21.15.32:0/989659046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613810da00 0x7f613810de00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6130006e50 tx=0x7f6130009e70).stop 2021-04-09T18:15:08.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.771 7f613f6ea700 1 -- 172.21.15.32:0/989659046 shutdown_connections 2021-04-09T18:15:08.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.771 7f613f6ea700 1 --2- 172.21.15.32:0/989659046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613810da00 0x7f613810de00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:08.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.771 7f613f6ea700 1 -- 172.21.15.32:0/989659046 >> 172.21.15.32:0/989659046 conn(0x7f61380ce5f0 msgr2=0x7f61380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:08.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.772 7f613f6ea700 1 -- 172.21.15.32:0/989659046 shutdown_connections 2021-04-09T18:15:08.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.772 7f613f6ea700 1 -- 172.21.15.32:0/989659046 wait complete. 2021-04-09T18:15:08.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.792 7f613f6ea700 1 Processor -- start 2021-04-09T18:15:08.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.792 7f613f6ea700 1 -- start start 2021-04-09T18:15:08.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.792 7f613f6ea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61381ac380 0x7f613818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:08.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.792 7f613f6ea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381ac780 0x7f613818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.792 7f613f6ea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6138183520 0x7f6138183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.792 7f613f6ea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61381ace10 con 0x7f61381ac780 2021-04-09T18:15:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.792 7f613f6ea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6138189cf0 con 0x7f6138183520 2021-04-09T18:15:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.792 7f613f6ea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6138183f40 con 0x7f61381ac380 2021-04-09T18:15:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613cd05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381ac780 0x7f613818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613cd05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381ac780 0x7f613818a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43810/0 (socket says 172.21.15.32:43810) 2021-04-09T18:15:08.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613cd05700 1 -- 172.21.15.32:0/57599714 learned_addr learned my addr 172.21.15.32:0/57599714 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:08.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613dd07700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6138183520 0x7f6138183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:08.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613d506700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61381ac380 0x7f613818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:08.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613cd05700 1 -- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61381ac380 msgr2=0x7f613818a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:08.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613cd05700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61381ac380 0x7f613818a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:08.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613cd05700 1 -- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6138183520 msgr2=0x7f6138183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:08.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613cd05700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6138183520 0x7f6138183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:08.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613cd05700 1 -- 172.21.15.32:0/57599714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138184160 con 0x7f61381ac780 2021-04-09T18:15:08.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613dd07700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6138183520 0x7f6138183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:08.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613cd05700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381ac780 0x7f613818a850 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f612c006f10 tx=0x7f612c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:08.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f6129ffb700 1 -- 172.21.15.32:0/57599714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f612c00bd80 con 0x7f61381ac780 2021-04-09T18:15:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f6129ffb700 1 -- 172.21.15.32:0/57599714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612c000e20 con 0x7f61381ac780 2021-04-09T18:15:08.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613f6ea700 1 -- 172.21.15.32:0/57599714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6138184160 con 0x7f61381ac780 2021-04-09T18:15:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f6129ffb700 1 -- 172.21.15.32:0/57599714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f612c0054f0 con 0x7f61381ac780 2021-04-09T18:15:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.793 7f613f6ea700 1 -- 172.21.15.32:0/57599714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6138184160 con 0x7f61381ac780 2021-04-09T18:15:08.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.794 7f6129ffb700 1 -- 172.21.15.32:0/57599714 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f612c005700 con 0x7f61381ac780 2021-04-09T18:15:08.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.794 7f6129ffb700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6118018930 0x7f611801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:09.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.794 7f6129ffb700 1 -- 172.21.15.32:0/57599714 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f612c0329e0 con 0x7f61381ac780 2021-04-09T18:15:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.794 7f613d506700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6118018930 0x7f611801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:09.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.795 7f613d506700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6118018930 0x7f611801ae10 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:09.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.813 7f613f6ea700 1 -- 172.21.15.32:0/57599714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6138189cf0 con 0x7f61381ac780 2021-04-09T18:15:09.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:08.817 7f6129ffb700 1 -- 172.21.15.32:0/57599714 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f612c0054f0 con 0x7f61381ac780 2021-04-09T18:15:09.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.009 7f613f6ea700 1 -- 172.21.15.32:0/57599714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6138189cf0 con 0x7f61381ac780 2021-04-09T18:15:09.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.009 7f6129ffb700 1 -- 172.21.15.32:0/57599714 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f612c005700 con 0x7f61381ac780 2021-04-09T18:15:09.076 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 30 2021-04-09T18:15:09.076 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:09.080 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:14:51.338207","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5119":{"gid":5119,"name":"b","rank":0,"incarnation":25,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6835/3858997347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3858997347},{"type":"v1","addr":"172.21.15.86:6835","nonce":3858997347}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:09.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 -- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6118018930 msgr2=0x7f611801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:09.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6118018930 0x7f611801ae10 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:09.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 -- 172.21.15.32:0/57599714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381ac780 msgr2=0x7f613818a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:09.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381ac780 0x7f613818a850 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f612c006f10 tx=0x7f612c00b990).stop 2021-04-09T18:15:09.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 -- 172.21.15.32:0/57599714 shutdown_connections 2021-04-09T18:15:09.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6118018930 0x7f611801ae10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:09.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381ac780 0x7f613818a850 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:09.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6138183520 0x7f6138183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:09.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 --2- 172.21.15.32:0/57599714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61381ac380 0x7f613818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:09.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 -- 172.21.15.32:0/57599714 >> 172.21.15.32:0/57599714 conn(0x7f613819b890 msgr2=0x7f61381a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 -- 172.21.15.32:0/57599714 shutdown_connections 2021-04-09T18:15:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:09.012 7f613f6ea700 1 -- 172.21.15.32:0/57599714 wait complete. 2021-04-09T18:15:09.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:14:51.338207', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5119}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5119': {'gid': 5119, 'name': 'b', 'rank': 0, 'incarnation': 25, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6835/3858997347', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3858997347}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3858997347}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:09.517 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5119:up:stopping 2021-04-09T18:15:09.759 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:15:10.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:10.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.646 7f93c1398700 1 Processor -- start 2021-04-09T18:15:10.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.646 7f93c1398700 1 -- start start 2021-04-09T18:15:10.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93c1398700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f93bc0d7ca0 0x7f93bc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:10.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93c1398700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93bc10d480 con 0x7f93bc10da00 2021-04-09T18:15:10.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93c1398700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93bc0d86c0 con 0x7f93bc10d010 2021-04-09T18:15:10.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93c1398700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93bc0d9220 con 0x7f93bc0d7ca0 2021-04-09T18:15:10.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93bb87e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f93bc10da00 0x7f93bc10de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53408/0 (socket says 172.21.15.32:53408) 2021-04-09T18:15:10.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93bb87e700 1 -- 172.21.15.32:0/2621381794 learned_addr learned my addr 172.21.15.32:0/2621381794 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:10.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93ba87c700 1 --2- 172.21.15.32:0/2621381794 >> v2:172.21.15.86:3300/0 conn(0x7f93bc0d7ca0 0x7f93bc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:10.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93ba07b700 1 -- 172.21.15.32:0/2621381794 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778611699 0 0) 0x7f93b4000d30 con 0x7f93bc10da00 2021-04-09T18:15:10.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93ba07b700 1 -- 172.21.15.32:0/2621381794 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93a4002fe0 con 0x7f93bc10da00 2021-04-09T18:15:10.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93ba87c700 1 -- 172.21.15.32:0/2621381794 >> v1:172.21.15.86:6789/0 conn(0x7f93bc10d010 legacy=0x7f93bc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93ba87c700 1 -- 172.21.15.32:0/2621381794 >> v1:172.21.15.32:6789/0 conn(0x7f93bc10da00 legacy=0x7f93bc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:10.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93ba07b700 1 -- 172.21.15.32:0/2621381794 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171018461 0 0) 0x7f93ac001080 con 0x7f93bc10d010 2021-04-09T18:15:10.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93ba87c700 1 -- 172.21.15.32:0/2621381794 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc112100 con 0x7f93bc0d7ca0 2021-04-09T18:15:10.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.647 7f93ba87c700 1 --2- 172.21.15.32:0/2621381794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93bc0d7ca0 0x7f93bc0d80a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f93b0006650 tx=0x7f93b0009a80).ready entity=mon.1 client_cookie=3a8a0f1e11a57804 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:10.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93ba07b700 1 -- 172.21.15.32:0/2621381794 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93b00009b0 con 0x7f93bc0d7ca0 2021-04-09T18:15:10.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93ba07b700 1 -- 172.21.15.32:0/2621381794 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b000c0c0 con 0x7f93bc0d7ca0 2021-04-09T18:15:10.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93ba07b700 1 -- 172.21.15.32:0/2621381794 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93b000c510 con 0x7f93bc0d7ca0 2021-04-09T18:15:10.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93c1398700 1 -- 172.21.15.32:0/2621381794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93bc0d7ca0 msgr2=0x7f93bc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:10.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93c1398700 1 --2- 172.21.15.32:0/2621381794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93bc0d7ca0 0x7f93bc0d80a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f93b0006650 tx=0x7f93b0009a80).stop 2021-04-09T18:15:10.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93c1398700 1 -- 172.21.15.32:0/2621381794 shutdown_connections 2021-04-09T18:15:10.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93c1398700 1 --2- 172.21.15.32:0/2621381794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93bc0d7ca0 0x7f93bc0d80a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:10.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93c1398700 1 -- 172.21.15.32:0/2621381794 >> 172.21.15.32:0/2621381794 conn(0x7f93bc0ce5f0 msgr2=0x7f93bc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:10.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93c1398700 1 -- 172.21.15.32:0/2621381794 shutdown_connections 2021-04-09T18:15:10.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.648 7f93c1398700 1 -- 172.21.15.32:0/2621381794 wait complete. 2021-04-09T18:15:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.668 7f93c1398700 1 Processor -- start 2021-04-09T18:15:10.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93c1398700 1 -- start start 2021-04-09T18:15:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93c1398700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93bc181e10 0x7f93bc19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93c1398700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93bc1b8500 0x7f93bc1b8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93c1398700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93bc08ced0 0x7f93bc08d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93c1398700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc08c950 con 0x7f93bc181e10 2021-04-09T18:15:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93c1398700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc185ba0 con 0x7f93bc08ced0 2021-04-09T18:15:10.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93c1398700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc08d8f0 con 0x7f93bc1b8500 2021-04-09T18:15:10.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93bb07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93bc181e10 0x7f93bc19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:10.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93bb07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93bc181e10 0x7f93bc19d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43822/0 (socket says 172.21.15.32:43822) 2021-04-09T18:15:10.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93bb07d700 1 -- 172.21.15.32:0/3718360341 learned_addr learned my addr 172.21.15.32:0/3718360341 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:10.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93bb87e700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93bc08ced0 0x7f93bc08d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:10.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.669 7f93ba87c700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93bc1b8500 0x7f93bc1b8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93bb07d700 1 -- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93bc1b8500 msgr2=0x7f93bc1b8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93bb07d700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93bc1b8500 0x7f93bc1b8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93bb07d700 1 -- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93bc08ced0 msgr2=0x7f93bc08d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:10.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93bb07d700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93bc08ced0 0x7f93bc08d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:10.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93bb07d700 1 -- 172.21.15.32:0/3718360341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc08db10 con 0x7f93bc181e10 2021-04-09T18:15:10.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93bb07d700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93bc181e10 0x7f93bc19d330 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f93ac006f30 tx=0x7f93ac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93a3fff700 1 -- 172.21.15.32:0/3718360341 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93ac00bd80 con 0x7f93bc181e10 2021-04-09T18:15:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93a3fff700 1 -- 172.21.15.32:0/3718360341 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac006030 con 0x7f93bc181e10 2021-04-09T18:15:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93c1398700 1 -- 172.21.15.32:0/3718360341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93bc08db10 con 0x7f93bc181e10 2021-04-09T18:15:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93a3fff700 1 -- 172.21.15.32:0/3718360341 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93ac005540 con 0x7f93bc181e10 2021-04-09T18:15:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.670 7f93c1398700 1 -- 172.21.15.32:0/3718360341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93bc08db10 con 0x7f93bc181e10 2021-04-09T18:15:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.671 7f93a3fff700 1 -- 172.21.15.32:0/3718360341 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f93ac006210 con 0x7f93bc181e10 2021-04-09T18:15:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.671 7f93a3fff700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f939c0188d0 0x7f939c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.671 7f93a3fff700 1 -- 172.21.15.32:0/3718360341 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f93ac0329f0 con 0x7f93bc181e10 2021-04-09T18:15:11.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.671 7f93ba87c700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f939c0188d0 0x7f939c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:11.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.671 7f93ba87c700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f939c0188d0 0x7f939c01add0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:11.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.691 7f93c1398700 1 -- 172.21.15.32:0/3718360341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93bc1b7010 con 0x7f93bc181e10 2021-04-09T18:15:11.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.694 7f93a3fff700 1 -- 172.21.15.32:0/3718360341 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f93ac000a60 con 0x7f93bc181e10 2021-04-09T18:15:11.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.886 7f93c1398700 1 -- 172.21.15.32:0/3718360341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93bc10d480 con 0x7f93bc181e10 2021-04-09T18:15:11.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.886 7f93a3fff700 1 -- 172.21.15.32:0/3718360341 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2426 (secure 0 0 0) 0x7f93ac006c00 con 0x7f93bc181e10 2021-04-09T18:15:11.091 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 31 2021-04-09T18:15:11.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:11.093 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":31,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:15:09.674958","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:11.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.888 7f93c1398700 1 -- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f939c0188d0 msgr2=0x7f939c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:11.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.888 7f93c1398700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f939c0188d0 0x7f939c01add0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.888 7f93c1398700 1 -- 172.21.15.32:0/3718360341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93bc181e10 msgr2=0x7f93bc19d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:11.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.888 7f93c1398700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93bc181e10 0x7f93bc19d330 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f93ac006f30 tx=0x7f93ac00b990).stop 2021-04-09T18:15:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.889 7f93c1398700 1 -- 172.21.15.32:0/3718360341 shutdown_connections 2021-04-09T18:15:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.889 7f93c1398700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f939c0188d0 0x7f939c01add0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:11.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.889 7f93c1398700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93bc08ced0 0x7f93bc08d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:11.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.889 7f93c1398700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93bc1b8500 0x7f93bc1b8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.889 7f93c1398700 1 --2- 172.21.15.32:0/3718360341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93bc181e10 0x7f93bc19d330 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:11.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.889 7f93c1398700 1 -- 172.21.15.32:0/3718360341 >> 172.21.15.32:0/3718360341 conn(0x7f93bc190550 msgr2=0x7f93bc1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:11.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.889 7f93c1398700 1 -- 172.21.15.32:0/3718360341 shutdown_connections 2021-04-09T18:15:11.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:10.889 7f93c1398700 1 -- 172.21.15.32:0/3718360341 wait complete. 2021-04-09T18:15:11.611 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 31, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:15:09.674958', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:11.612 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2021-04-09T18:15:11.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:15:11.748 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:15:11.748 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:15:11.749 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:15:11.753 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:15:11.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:11.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.878 7fa70050d700 1 Processor -- start 2021-04-09T18:15:11.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.878 7fa70050d700 1 -- start start 2021-04-09T18:15:11.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa70050d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f810da00 0x7fa6f810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:11.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa70050d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f810d480 con 0x7fa6f80d7ca0 2021-04-09T18:15:11.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa70050d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f80d1f70 con 0x7fa6f810d010 2021-04-09T18:15:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa70050d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f80d9220 con 0x7fa6f810da00 2021-04-09T18:15:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6fdb28700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa6f80d7ca0 0x7fa6f80d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53422/0 (socket says 172.21.15.32:53422) 2021-04-09T18:15:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6fdb28700 1 -- 172.21.15.32:0/1374690635 learned_addr learned my addr 172.21.15.32:0/1374690635 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:11.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6feb2a700 1 --2- 172.21.15.32:0/1374690635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f810da00 0x7fa6f810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:11.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6fd327700 1 -- 172.21.15.32:0/1374690635 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842633931 0 0) 0x7fa6e8000ce0 con 0x7fa6f80d7ca0 2021-04-09T18:15:11.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6fd327700 1 -- 172.21.15.32:0/1374690635 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e4002fe0 con 0x7fa6f80d7ca0 2021-04-09T18:15:11.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6fd327700 1 -- 172.21.15.32:0/1374690635 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903155308 0 0) 0x7fa6f4000c60 con 0x7fa6f810d010 2021-04-09T18:15:11.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6fd327700 1 -- 172.21.15.32:0/1374690635 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e40042c0 con 0x7fa6f810d010 2021-04-09T18:15:11.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6feb2a700 1 -- 172.21.15.32:0/1374690635 >> v1:172.21.15.86:6789/0 conn(0x7fa6f810d010 legacy=0x7fa6f80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:11.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6feb2a700 1 -- 172.21.15.32:0/1374690635 >> v1:172.21.15.32:6789/0 conn(0x7fa6f80d7ca0 legacy=0x7fa6f80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:11.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6feb2a700 1 -- 172.21.15.32:0/1374690635 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f810e420 con 0x7fa6f810da00 2021-04-09T18:15:11.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.879 7fa6feb2a700 1 --2- 172.21.15.32:0/1374690635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f810da00 0x7fa6f810de00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fa6f0006e50 tx=0x7fa6f0009e70).ready entity=mon.2 client_cookie=f0a3a8634e7d603c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:11.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.880 7fa6fd327700 1 -- 172.21.15.32:0/1374690635 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6f0006a20 con 0x7fa6f810da00 2021-04-09T18:15:11.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.880 7fa6fd327700 1 -- 172.21.15.32:0/1374690635 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f0006790 con 0x7fa6f810da00 2021-04-09T18:15:11.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.880 7fa6fd327700 1 -- 172.21.15.32:0/1374690635 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6f000b700 con 0x7fa6f810da00 2021-04-09T18:15:11.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.880 7fa70050d700 1 -- 172.21.15.32:0/1374690635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f810da00 msgr2=0x7fa6f810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.880 7fa70050d700 1 --2- 172.21.15.32:0/1374690635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f810da00 0x7fa6f810de00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fa6f0006e50 tx=0x7fa6f0009e70).stop 2021-04-09T18:15:11.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.881 7fa70050d700 1 -- 172.21.15.32:0/1374690635 shutdown_connections 2021-04-09T18:15:11.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.881 7fa70050d700 1 --2- 172.21.15.32:0/1374690635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f810da00 0x7fa6f810de00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:11.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.881 7fa70050d700 1 -- 172.21.15.32:0/1374690635 >> 172.21.15.32:0/1374690635 conn(0x7fa6f80ce5f0 msgr2=0x7fa6f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:11.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.881 7fa70050d700 1 -- 172.21.15.32:0/1374690635 shutdown_connections 2021-04-09T18:15:11.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.881 7fa70050d700 1 -- 172.21.15.32:0/1374690635 wait complete. 2021-04-09T18:15:11.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa70050d700 1 Processor -- start 2021-04-09T18:15:11.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa70050d700 1 -- start start 2021-04-09T18:15:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa70050d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f8099920 0x7fa6f8179f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa70050d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f817a560 0x7fa6f81b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa70050d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f81b85a0 0x7fa6f81b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa70050d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f817a960 con 0x7fa6f817a560 2021-04-09T18:15:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa70050d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8099d20 con 0x7fa6f81b85a0 2021-04-09T18:15:12.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa70050d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8179610 con 0x7fa6f8099920 2021-04-09T18:15:12.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa6fdb28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f817a560 0x7fa6f81b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa6fdb28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f817a560 0x7fa6f81b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43834/0 (socket says 172.21.15.32:43834) 2021-04-09T18:15:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.901 7fa6fdb28700 1 -- 172.21.15.32:0/3578693841 learned_addr learned my addr 172.21.15.32:0/3578693841 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6feb2a700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f81b85a0 0x7fa6f81b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6fe329700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f8099920 0x7fa6f8179f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6fdb28700 1 -- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f8099920 msgr2=0x7fa6f8179f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6fdb28700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f8099920 0x7fa6f8179f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6fdb28700 1 -- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f81b85a0 msgr2=0x7fa6f81b89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6fdb28700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f81b85a0 0x7fa6f81b89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6fdb28700 1 -- 172.21.15.32:0/3578693841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f8180be0 con 0x7fa6f817a560 2021-04-09T18:15:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6feb2a700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f81b85a0 0x7fa6f81b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6fdb28700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f817a560 0x7fa6f81b7f80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fa6e800b530 tx=0x7fa6e800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6eeffd700 1 -- 172.21.15.32:0/3578693841 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6e800bcc0 con 0x7fa6f817a560 2021-04-09T18:15:12.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6eeffd700 1 -- 172.21.15.32:0/3578693841 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e800bcc0 con 0x7fa6f817a560 2021-04-09T18:15:12.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa70050d700 1 -- 172.21.15.32:0/3578693841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f8180be0 con 0x7fa6f817a560 2021-04-09T18:15:12.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa6eeffd700 1 -- 172.21.15.32:0/3578693841 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6e8006030 con 0x7fa6f817a560 2021-04-09T18:15:12.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.902 7fa70050d700 1 -- 172.21.15.32:0/3578693841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f8180be0 con 0x7fa6f817a560 2021-04-09T18:15:12.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.903 7fa6eeffd700 1 -- 172.21.15.32:0/3578693841 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa6e80043a0 con 0x7fa6f817a560 2021-04-09T18:15:12.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.903 7fa6eeffd700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6e0018930 0x7fa6e001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.903 7fa6eeffd700 1 -- 172.21.15.32:0/3578693841 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa6e80329e0 con 0x7fa6f817a560 2021-04-09T18:15:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.903 7fa6fe329700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6e0018930 0x7fa6e001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.903 7fa6fe329700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6e0018930 0x7fa6e001adc0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.923 7fa70050d700 1 -- 172.21.15.32:0/3578693841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f81b7070 con 0x7fa6f817a560 2021-04-09T18:15:12.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:11.926 7fa6eeffd700 1 -- 172.21.15.32:0/3578693841 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa6e8006b90 con 0x7fa6f817a560 2021-04-09T18:15:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.119 7fa70050d700 1 -- 172.21.15.32:0/3578693841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6f810d480 con 0x7fa6f817a560 2021-04-09T18:15:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.120 7fa6eeffd700 1 -- 172.21.15.32:0/3578693841 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2426 (secure 0 0 0) 0x7fa6e80043a0 con 0x7fa6f817a560 2021-04-09T18:15:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 31 2021-04-09T18:15:12.188 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:12.189 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":31,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:15:09.674958","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:12.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 -- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6e0018930 msgr2=0x7fa6e001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6e0018930 0x7fa6e001adc0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 -- 172.21.15.32:0/3578693841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f817a560 msgr2=0x7fa6f81b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f817a560 0x7fa6f81b7f80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fa6e800b530 tx=0x7fa6e800b8d0).stop 2021-04-09T18:15:12.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 -- 172.21.15.32:0/3578693841 shutdown_connections 2021-04-09T18:15:12.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6e0018930 0x7fa6e001adc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6f817a560 0x7fa6f81b7f80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f81b85a0 0x7fa6f81b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 --2- 172.21.15.32:0/3578693841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f8099920 0x7fa6f8179f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 -- 172.21.15.32:0/3578693841 >> 172.21.15.32:0/3578693841 conn(0x7fa6f81aa2b0 msgr2=0x7fa6f81a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 -- 172.21.15.32:0/3578693841 shutdown_connections 2021-04-09T18:15:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.122 7fa70050d700 1 -- 172.21.15.32:0/3578693841 wait complete. 2021-04-09T18:15:12.314 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 3, "epoch": 31, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 3, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:14:39.918931", "damaged": [], "data_pools": [ 7 ], "enabled": true, "epoch": 31, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [], "info": {}, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 0, "metadata_pool": 6, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:15:09.674958", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [ 0 ], "tableserver": 0, "up": {} } } ], "standbys": [ { "addr": "172.21.15.32:6833/3664667451", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 3664667451, "type": "v2" }, { "addr": "172.21.15.32:6833", "nonce": 3664667451, "type": "v1" } ] }, "epoch": 24, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 5136, "incarnation": 0, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.32:6835/1051691429", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6834", "nonce": 1051691429, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 1051691429, "type": "v1" } ] }, "epoch": 24, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 5151, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6837/631157840", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6836", "nonce": 631157840, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 631157840, "type": "v1" } ] }, "epoch": 24, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 54800, "incarnation": 0, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:15:12.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:12.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.438 7f63af5ee700 1 Processor -- start 2021-04-09T18:15:12.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.439 7f63af5ee700 1 -- start start 2021-04-09T18:15:12.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.439 7f63af5ee700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63a810d010 0x7f63a80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.439 7f63af5ee700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f63a80d1da0 0x7f63a80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.439 7f63af5ee700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a810d480 con 0x7f63a80d82c0 2021-04-09T18:15:12.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.439 7f63af5ee700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a80d8d30 con 0x7f63a80d1da0 2021-04-09T18:15:12.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.439 7f63af5ee700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a80d9220 con 0x7f63a810d010 2021-04-09T18:15:12.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63acc09700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f63a80d1da0 0x7f63a80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63acc09700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f63a80d1da0 0x7f63a80d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43844/0 (socket says 172.21.15.32:43844) 2021-04-09T18:15:12.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63acc09700 1 -- 172.21.15.32:0/1279345233 learned_addr learned my addr 172.21.15.32:0/1279345233 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:12.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63ad40a700 1 --2- 172.21.15.32:0/1279345233 >> v2:172.21.15.86:3300/0 conn(0x7f63a810d010 0x7f63a80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63acc09700 1 -- 172.21.15.32:0/1279345233 >> v2:172.21.15.86:3300/0 conn(0x7f63a810d010 msgr2=0x7f63a80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:12.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63acc09700 1 --2- 172.21.15.32:0/1279345233 >> v2:172.21.15.86:3300/0 conn(0x7f63a810d010 0x7f63a80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63acc09700 1 -- 172.21.15.32:0/1279345233 >> v1:172.21.15.86:6789/0 conn(0x7f63a80d82c0 legacy=0x7f63a810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63acc09700 1 -- 172.21.15.32:0/1279345233 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a80d86f0 con 0x7f63a80d1da0 2021-04-09T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63acc09700 1 --2- 172.21.15.32:0/1279345233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a80d1da0 0x7f63a80d7ca0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f639c006700 tx=0x7f639c009b80).ready entity=mon.0 client_cookie=260d99faf5ba9de3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f639bfff700 1 -- 172.21.15.32:0/1279345233 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f639c00c0c0 con 0x7f63a80d1da0 2021-04-09T18:15:12.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f639bfff700 1 -- 172.21.15.32:0/1279345233 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c00c440 con 0x7f63a80d1da0 2021-04-09T18:15:12.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f639bfff700 1 -- 172.21.15.32:0/1279345233 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f639c00c920 con 0x7f63a80d1da0 2021-04-09T18:15:12.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63af5ee700 1 -- 172.21.15.32:0/1279345233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a80d1da0 msgr2=0x7f63a80d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.440 7f63af5ee700 1 --2- 172.21.15.32:0/1279345233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a80d1da0 0x7f63a80d7ca0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f639c006700 tx=0x7f639c009b80).stop 2021-04-09T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.441 7f63af5ee700 1 -- 172.21.15.32:0/1279345233 shutdown_connections 2021-04-09T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.441 7f63af5ee700 1 --2- 172.21.15.32:0/1279345233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a80d1da0 0x7f63a80d7ca0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.441 7f63af5ee700 1 --2- 172.21.15.32:0/1279345233 >> v2:172.21.15.86:3300/0 conn(0x7f63a810d010 0x7f63a80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.441 7f63af5ee700 1 -- 172.21.15.32:0/1279345233 >> 172.21.15.32:0/1279345233 conn(0x7f63a80ce5f0 msgr2=0x7f63a80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:12.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.441 7f63af5ee700 1 -- 172.21.15.32:0/1279345233 shutdown_connections 2021-04-09T18:15:12.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.441 7f63af5ee700 1 -- 172.21.15.32:0/1279345233 wait complete. 2021-04-09T18:15:12.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.461 7f63af5ee700 1 Processor -- start 2021-04-09T18:15:12.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.461 7f63af5ee700 1 -- start start 2021-04-09T18:15:12.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63af5ee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a80b5360 0x7f63a81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63af5ee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63a818e140 0x7f63a81aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63af5ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81ad040 0x7f63a808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63af5ee700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a81a5360 con 0x7f63a81ad040 2021-04-09T18:15:12.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63af5ee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a80902c0 con 0x7f63a80b5360 2021-04-09T18:15:12.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63af5ee700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a80904e0 con 0x7f63a818e140 2021-04-09T18:15:12.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63adc0b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81ad040 0x7f63a808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63adc0b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81ad040 0x7f63a808fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43850/0 (socket says 172.21.15.32:43850) 2021-04-09T18:15:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63adc0b700 1 -- 172.21.15.32:0/3153438462 learned_addr learned my addr 172.21.15.32:0/3153438462 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63ad40a700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a80b5360 0x7f63a81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.462 7f63acc09700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63a818e140 0x7f63a81aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f63adc0b700 1 -- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63a818e140 msgr2=0x7f63a81aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f63adc0b700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63a818e140 0x7f63a81aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f63adc0b700 1 -- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a80b5360 msgr2=0x7f63a81a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f63adc0b700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a80b5360 0x7f63a81a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f63adc0b700 1 -- 172.21.15.32:0/3153438462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a8183c80 con 0x7f63a81ad040 2021-04-09T18:15:12.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f63ad40a700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a80b5360 0x7f63a81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:12.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f63adc0b700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81ad040 0x7f63a808fca0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f63a0009940 tx=0x7f63a000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:12.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f6399ffb700 1 -- 172.21.15.32:0/3153438462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63a0000f40 con 0x7f63a81ad040 2021-04-09T18:15:12.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f6399ffb700 1 -- 172.21.15.32:0/3153438462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a000c2f0 con 0x7f63a81ad040 2021-04-09T18:15:12.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a8183c80 con 0x7f63a81ad040 2021-04-09T18:15:12.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f6399ffb700 1 -- 172.21.15.32:0/3153438462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63a000c7d0 con 0x7f63a81ad040 2021-04-09T18:15:12.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a8183c80 con 0x7f63a81ad040 2021-04-09T18:15:12.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.463 7f6399ffb700 1 -- 172.21.15.32:0/3153438462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f63a0001220 con 0x7f63a81ad040 2021-04-09T18:15:12.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.464 7f6399ffb700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6388018930 0x7f638801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.464 7f6399ffb700 1 -- 172.21.15.32:0/3153438462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f63a0032d70 con 0x7f63a81ad040 2021-04-09T18:15:12.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.464 7f63ad40a700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6388018930 0x7f638801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.464 7f63ad40a700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6388018930 0x7f638801ae10 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.483 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a8182d70 con 0x7f63a81ad040 2021-04-09T18:15:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.487 7f6399ffb700 1 -- 172.21.15.32:0/3153438462 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f63a000be30 con 0x7f63a81ad040 2021-04-09T18:15:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.679 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63a8182d70 con 0x7f63a81ad040 2021-04-09T18:15:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.679 7f6399ffb700 1 -- 172.21.15.32:0/3153438462 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2426 (secure 0 0 0) 0x7f63a0001220 con 0x7f63a81ad040 2021-04-09T18:15:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 31 2021-04-09T18:15:12.710 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:12.710 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":31,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5136,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:15:09.674958","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6388018930 msgr2=0x7f638801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6388018930 0x7f638801ae10 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81ad040 msgr2=0x7f63a808fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:12.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81ad040 0x7f63a808fca0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f63a0009940 tx=0x7f63a000bcf0).stop 2021-04-09T18:15:12.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 shutdown_connections 2021-04-09T18:15:12.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6388018930 0x7f638801ae10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a81ad040 0x7f63a808fca0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63a818e140 0x7f63a81aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 --2- 172.21.15.32:0/3153438462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63a80b5360 0x7f63a81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 >> 172.21.15.32:0/3153438462 conn(0x7f63a819b460 msgr2=0x7f63a819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 shutdown_connections 2021-04-09T18:15:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.682 7f63af5ee700 1 -- 172.21.15.32:0/3153438462 wait complete. 2021-04-09T18:15:12.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:15:12.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.902 7fbae9e52700 1 Processor -- start 2021-04-09T18:15:12.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.902 7fbae9e52700 1 -- start start 2021-04-09T18:15:12.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae9e52700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbae40d7ca0 0x7fbae40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae9e52700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbae410da00 0x7fbae410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:12.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae9e52700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae410d480 con 0x7fbae410d010 2021-04-09T18:15:12.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae9e52700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae40d9220 con 0x7fbae40d7ca0 2021-04-09T18:15:12.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae9e52700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae40d86c0 con 0x7fbae410da00 2021-04-09T18:15:12.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae2ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbae40d7ca0 0x7fbae40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:12.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae2ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbae40d7ca0 0x7fbae40d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43860/0 (socket says 172.21.15.32:43860) 2021-04-09T18:15:12.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae2ffd700 1 -- 172.21.15.32:0/1475168898 learned_addr learned my addr 172.21.15.32:0/1475168898 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:12.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae3fff700 1 --2- 172.21.15.32:0/1475168898 >> v2:172.21.15.86:3300/0 conn(0x7fbae410da00 0x7fbae410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:13.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae2ffd700 1 -- 172.21.15.32:0/1475168898 >> v2:172.21.15.86:3300/0 conn(0x7fbae410da00 msgr2=0x7fbae410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:13.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae2ffd700 1 --2- 172.21.15.32:0/1475168898 >> v2:172.21.15.86:3300/0 conn(0x7fbae410da00 0x7fbae410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae2ffd700 1 -- 172.21.15.32:0/1475168898 >> v1:172.21.15.86:6789/0 conn(0x7fbae410d010 legacy=0x7fbae40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:13.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae2ffd700 1 -- 172.21.15.32:0/1475168898 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae40d1f70 con 0x7fbae40d7ca0 2021-04-09T18:15:13.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.903 7fbae2ffd700 1 --2- 172.21.15.32:0/1475168898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae40d7ca0 0x7fbae40d80a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fbad8006700 tx=0x7fbad8009b80).ready entity=mon.0 client_cookie=53fc2b1450f14e2c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.904 7fbae27fc700 1 -- 172.21.15.32:0/1475168898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad800c0c0 con 0x7fbae40d7ca0 2021-04-09T18:15:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.904 7fbae27fc700 1 -- 172.21.15.32:0/1475168898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad800c440 con 0x7fbae40d7ca0 2021-04-09T18:15:13.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.904 7fbae27fc700 1 -- 172.21.15.32:0/1475168898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad800c920 con 0x7fbae40d7ca0 2021-04-09T18:15:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.904 7fbae9e52700 1 -- 172.21.15.32:0/1475168898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae40d7ca0 msgr2=0x7fbae40d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.904 7fbae9e52700 1 --2- 172.21.15.32:0/1475168898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae40d7ca0 0x7fbae40d80a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fbad8006700 tx=0x7fbad8009b80).stop 2021-04-09T18:15:13.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.904 7fbae9e52700 1 -- 172.21.15.32:0/1475168898 shutdown_connections 2021-04-09T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.904 7fbae9e52700 1 --2- 172.21.15.32:0/1475168898 >> v2:172.21.15.86:3300/0 conn(0x7fbae410da00 0x7fbae410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.904 7fbae9e52700 1 --2- 172.21.15.32:0/1475168898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae40d7ca0 0x7fbae40d80a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.904 7fbae9e52700 1 -- 172.21.15.32:0/1475168898 >> 172.21.15.32:0/1475168898 conn(0x7fbae40ce5f0 msgr2=0x7fbae40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.905 7fbae9e52700 1 -- 172.21.15.32:0/1475168898 shutdown_connections 2021-04-09T18:15:13.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.905 7fbae9e52700 1 -- 172.21.15.32:0/1475168898 wait complete. 2021-04-09T18:15:13.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.925 7fbae9e52700 1 Processor -- start 2021-04-09T18:15:13.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.925 7fbae9e52700 1 -- start start 2021-04-09T18:15:13.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.925 7fbae9e52700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae40a0110 0x7fbae418cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:13.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.925 7fbae9e52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae418d320 0x7fbae40a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:13.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.925 7fbae9e52700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae40a1230 0x7fbae41a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:13.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.925 7fbae9e52700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40a0690 con 0x7fbae418d320 2021-04-09T18:15:13.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.925 7fbae9e52700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40a1630 con 0x7fbae40a1230 2021-04-09T18:15:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.925 7fbae9e52700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae41a1370 con 0x7fbae40a0110 2021-04-09T18:15:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae418d320 0x7fbae40a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae418d320 0x7fbae40a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43862/0 (socket says 172.21.15.32:43862) 2021-04-09T18:15:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae2ffd700 1 -- 172.21.15.32:0/330118851 learned_addr learned my addr 172.21.15.32:0/330118851 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae3fff700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae40a1230 0x7fbae41a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae37fe700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae40a0110 0x7fbae418cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae2ffd700 1 -- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae40a0110 msgr2=0x7fbae418cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae2ffd700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae40a0110 0x7fbae418cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae2ffd700 1 -- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae40a1230 msgr2=0x7fbae41a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae2ffd700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae40a1230 0x7fbae41a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae2ffd700 1 -- 172.21.15.32:0/330118851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae41a1590 con 0x7fbae418d320 2021-04-09T18:15:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae2ffd700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae418d320 0x7fbae40a0c10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fbad800b650 tx=0x7fbad800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbacbfff700 1 -- 172.21.15.32:0/330118851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad800bde0 con 0x7fbae418d320 2021-04-09T18:15:13.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbacbfff700 1 -- 172.21.15.32:0/330118851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad8000c90 con 0x7fbae418d320 2021-04-09T18:15:13.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbacbfff700 1 -- 172.21.15.32:0/330118851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad8002220 con 0x7fbae418d320 2021-04-09T18:15:13.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae9e52700 1 -- 172.21.15.32:0/330118851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae41a1590 con 0x7fbae418d320 2021-04-09T18:15:13.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.926 7fbae9e52700 1 -- 172.21.15.32:0/330118851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae41a1590 con 0x7fbae418d320 2021-04-09T18:15:13.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.927 7fbacbfff700 1 -- 172.21.15.32:0/330118851 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbad8009f90 con 0x7fbae418d320 2021-04-09T18:15:13.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.927 7fbacbfff700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac4018930 0x7fbac401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:13.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.927 7fbacbfff700 1 -- 172.21.15.32:0/330118851 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbad80329f0 con 0x7fbae418d320 2021-04-09T18:15:13.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.927 7fbae37fe700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac4018930 0x7fbac401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:13.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.927 7fbae37fe700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac4018930 0x7fbac401ae10 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:13.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.947 7fbae9e52700 1 -- 172.21.15.32:0/330118851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae410d480 con 0x7fbae418d320 2021-04-09T18:15:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:12.950 7fbacbfff700 1 -- 172.21.15.32:0/330118851 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbad80091e0 con 0x7fbae418d320 2021-04-09T18:15:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.144 7fbae9e52700 1 -- 172.21.15.32:0/330118851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fbae410d480 con 0x7fbae418d320 2021-04-09T18:15:13.714 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:15:13.688 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:15:13.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.696 7fbacbfff700 1 -- 172.21.15.32:0/330118851 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v32) v1 ==== 105+0+0 (secure 0 0 0) 0x7fbad8009f90 con 0x7fbae418d320 2021-04-09T18:15:13.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.711 7fbae9e52700 1 -- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac4018930 msgr2=0x7fbac401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:13.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac4018930 0x7fbac401ae10 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 -- 172.21.15.32:0/330118851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae418d320 msgr2=0x7fbae40a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:13.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae418d320 0x7fbae40a0c10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fbad800b650 tx=0x7fbad800b9f0).stop 2021-04-09T18:15:13.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 -- 172.21.15.32:0/330118851 shutdown_connections 2021-04-09T18:15:13.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac4018930 0x7fbac401ae10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae418d320 0x7fbae40a0c10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae40a1230 0x7fbae41a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 --2- 172.21.15.32:0/330118851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae40a0110 0x7fbae418cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 -- 172.21.15.32:0/330118851 >> 172.21.15.32:0/330118851 conn(0x7fbae40b93a0 msgr2=0x7fbae40b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:13.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 -- 172.21.15.32:0/330118851 shutdown_connections 2021-04-09T18:15:13.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.712 7fbae9e52700 1 -- 172.21.15.32:0/330118851 wait complete. 2021-04-09T18:15:13.755 INFO:tasks.cephfs.test_failover:Waiting for cluster to grow. 2021-04-09T18:15:13.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:13.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.920 7fecb1db8700 1 Processor -- start 2021-04-09T18:15:13.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.920 7fecb1db8700 1 -- start start 2021-04-09T18:15:13.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecb1db8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fecac10d010 0x7fecac0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:13.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecb1db8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fecac0d8490 0x7fecac10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecb1db8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecac10d480 con 0x7fecac0d1da0 2021-04-09T18:15:13.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecb1db8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecac0d8d30 con 0x7fecac0d8490 2021-04-09T18:15:13.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecb1db8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecac0d9220 con 0x7fecac10d010 2021-04-09T18:15:13.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecabfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fecac0d8490 0x7fecac10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:13.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecaaffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fecac0d1da0 0x7fecac0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53470/0 (socket says 172.21.15.32:53470) 2021-04-09T18:15:13.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecaaffd700 1 -- 172.21.15.32:0/1030832312 learned_addr learned my addr 172.21.15.32:0/1030832312 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:13.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecab7fe700 1 --2- 172.21.15.32:0/1030832312 >> v2:172.21.15.86:3300/0 conn(0x7fecac10d010 0x7fecac0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:13.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecaa7fc700 1 -- 172.21.15.32:0/1030832312 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262882101 0 0) 0x7feca00009b0 con 0x7fecac0d1da0 2021-04-09T18:15:13.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecaa7fc700 1 -- 172.21.15.32:0/1030832312 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec94002fe0 con 0x7fecac0d1da0 2021-04-09T18:15:13.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecabfff700 1 -- 172.21.15.32:0/1030832312 >> v2:172.21.15.86:3300/0 conn(0x7fecac10d010 msgr2=0x7fecac0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecabfff700 1 --2- 172.21.15.32:0/1030832312 >> v2:172.21.15.86:3300/0 conn(0x7fecac10d010 0x7fecac0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:13.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecabfff700 1 -- 172.21.15.32:0/1030832312 >> v1:172.21.15.32:6789/0 conn(0x7fecac0d1da0 legacy=0x7fecac0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:13.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.921 7fecabfff700 1 -- 172.21.15.32:0/1030832312 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac10e020 con 0x7fecac0d8490 2021-04-09T18:15:13.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecabfff700 1 --2- 172.21.15.32:0/1030832312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac0d8490 0x7fecac10da00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7feca4006ab0 tx=0x7feca4009ac0).ready entity=mon.0 client_cookie=5be0fb45af9ed914 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:13.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecaa7fc700 1 -- 172.21.15.32:0/1030832312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feca40066d0 con 0x7fecac0d8490 2021-04-09T18:15:13.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecaa7fc700 1 -- 172.21.15.32:0/1030832312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca40022e0 con 0x7fecac0d8490 2021-04-09T18:15:13.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecaa7fc700 1 -- 172.21.15.32:0/1030832312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feca40027c0 con 0x7fecac0d8490 2021-04-09T18:15:13.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecb1db8700 1 -- 172.21.15.32:0/1030832312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac0d8490 msgr2=0x7fecac10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:13.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecb1db8700 1 --2- 172.21.15.32:0/1030832312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac0d8490 0x7fecac10da00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7feca4006ab0 tx=0x7feca4009ac0).stop 2021-04-09T18:15:13.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecb1db8700 1 -- 172.21.15.32:0/1030832312 shutdown_connections 2021-04-09T18:15:13.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecb1db8700 1 --2- 172.21.15.32:0/1030832312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac0d8490 0x7fecac10da00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:14.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecb1db8700 1 --2- 172.21.15.32:0/1030832312 >> v2:172.21.15.86:3300/0 conn(0x7fecac10d010 0x7fecac0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:14.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.922 7fecb1db8700 1 -- 172.21.15.32:0/1030832312 >> 172.21.15.32:0/1030832312 conn(0x7fecac0ce5f0 msgr2=0x7fecac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:14.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.923 7fecb1db8700 1 -- 172.21.15.32:0/1030832312 shutdown_connections 2021-04-09T18:15:14.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.923 7fecb1db8700 1 -- 172.21.15.32:0/1030832312 wait complete. 2021-04-09T18:15:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecb1db8700 1 Processor -- start 2021-04-09T18:15:14.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecb1db8700 1 -- start start 2021-04-09T18:15:14.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecb1db8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac1a5210 0x7fecac1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecb1db8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecac1a6340 0x7fecac1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecb1db8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecac0abaa0 0x7fecac0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:14.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecb1db8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac1a6d60 con 0x7fecac1a5210 2021-04-09T18:15:14.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecb1db8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0ac4c0 con 0x7fecac1a6340 2021-04-09T18:15:14.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecb1db8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0ac6e0 con 0x7fecac0abaa0 2021-04-09T18:15:14.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecab7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac1a5210 0x7fecac1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:14.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.943 7fecab7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac1a5210 0x7fecac1a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43882/0 (socket says 172.21.15.32:43882) 2021-04-09T18:15:14.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecab7fe700 1 -- 172.21.15.32:0/4184530295 learned_addr learned my addr 172.21.15.32:0/4184530295 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:14.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecaaffd700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecac1a6340 0x7fecac1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:14.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecabfff700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecac0abaa0 0x7fecac0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:14.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecab7fe700 1 -- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecac0abaa0 msgr2=0x7fecac0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:14.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecab7fe700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecac0abaa0 0x7fecac0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:14.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecab7fe700 1 -- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecac1a6340 msgr2=0x7fecac1a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:14.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecab7fe700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecac1a6340 0x7fecac1a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:14.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecab7fe700 1 -- 172.21.15.32:0/4184530295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac17a720 con 0x7fecac1a5210 2021-04-09T18:15:14.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecab7fe700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac1a5210 0x7fecac1a56c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fec9c00b8e0 tx=0x7fec9c00bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:14.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fec93fff700 1 -- 172.21.15.32:0/4184530295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec9c009140 con 0x7fecac1a5210 2021-04-09T18:15:14.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fec93fff700 1 -- 172.21.15.32:0/4184530295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c00c2f0 con 0x7fecac1a5210 2021-04-09T18:15:14.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecac17a720 con 0x7fecac1a5210 2021-04-09T18:15:14.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fec93fff700 1 -- 172.21.15.32:0/4184530295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec9c00c740 con 0x7fecac1a5210 2021-04-09T18:15:14.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.944 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac17a720 con 0x7fecac1a5210 2021-04-09T18:15:14.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.945 7fec93fff700 1 -- 172.21.15.32:0/4184530295 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fec9c001010 con 0x7fecac1a5210 2021-04-09T18:15:14.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.945 7fec93fff700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec8c018930 0x7fec8c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:14.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.945 7fec93fff700 1 -- 172.21.15.32:0/4184530295 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fec9c032d50 con 0x7fecac1a5210 2021-04-09T18:15:14.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.945 7fecaaffd700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec8c018930 0x7fec8c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:14.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.946 7fecaaffd700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec8c018930 0x7fec8c01adc0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:14.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.965 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecac1a6d60 con 0x7fecac1a5210 2021-04-09T18:15:14.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:13.969 7fec93fff700 1 -- 172.21.15.32:0/4184530295 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fec9c00bdc0 con 0x7fecac1a5210 2021-04-09T18:15:14.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.161 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecac1a6d60 con 0x7fecac1a5210 2021-04-09T18:15:14.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.162 7fec93fff700 1 -- 172.21.15.32:0/4184530295 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+2438 (secure 0 0 0) 0x7fec9c01fa90 con 0x7fecac1a5210 2021-04-09T18:15:14.217 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 33 2021-04-09T18:15:14.241 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:14.242 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":33,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5151,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:15:13.697844","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":5136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5136":{"gid":5136,"name":"a","rank":0,"incarnation":33,"state":"up:starting","state_seq":2,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:14.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec8c018930 msgr2=0x7fec8c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:14.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec8c018930 0x7fec8c01adc0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:14.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac1a5210 msgr2=0x7fecac1a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:14.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac1a5210 0x7fecac1a56c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fec9c00b8e0 tx=0x7fec9c00bc80).stop 2021-04-09T18:15:14.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 shutdown_connections 2021-04-09T18:15:14.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec8c018930 0x7fec8c01adc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:14.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecac0abaa0 0x7fecac0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:14.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecac1a6340 0x7fecac1a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:14.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 --2- 172.21.15.32:0/4184530295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecac1a5210 0x7fecac1a56c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:14.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 >> 172.21.15.32:0/4184530295 conn(0x7fecac1a99c0 msgr2=0x7fecac17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:14.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 shutdown_connections 2021-04-09T18:15:14.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:14.164 7fecb1db8700 1 -- 172.21.15.32:0/4184530295 wait complete. 2021-04-09T18:15:14.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:15:13.697844', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 5136}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5136': {'gid': 5136, 'name': 'a', 'rank': 0, 'incarnation': 33, 'state': 'up:starting', 'state_seq': 2, 'addr': '172.21.15.32:6833/3664667451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3664667451}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3664667451}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:14.708 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5136:up:starting 2021-04-09T18:15:15.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:15.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.856 7fdd8e86e700 1 Processor -- start 2021-04-09T18:15:15.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.856 7fdd8e86e700 1 -- start start 2021-04-09T18:15:15.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd8e86e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdd8810d010 0x7fdd880d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd8e86e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880d1da0 0x7fdd880d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd8e86e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8810d480 con 0x7fdd880d82c0 2021-04-09T18:15:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd8e86e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd880d9220 con 0x7fdd8810d010 2021-04-09T18:15:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd8e86e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd880d86c0 con 0x7fdd880d1da0 2021-04-09T18:15:15.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd7ffff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880d1da0 0x7fdd880d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:15.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd87fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdd8810d010 0x7fdd880d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:15.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd8ce8b700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdd880d82c0 0x7fdd8810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:51432/0 (socket says 172.21.15.32:51432) 2021-04-09T18:15:15.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd87fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdd8810d010 0x7fdd880d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:35670/0 (socket says 172.21.15.32:35670) 2021-04-09T18:15:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd8ce8b700 1 -- 172.21.15.32:0/2433322713 learned_addr learned my addr 172.21.15.32:0/2433322713 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:15.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd7ffff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880d1da0 0x7fdd880d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:51138/0 (socket says 172.21.15.32:51138) 2021-04-09T18:15:15.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd87fff700 1 -- 172.21.15.32:0/2433322713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880d1da0 msgr2=0x7fdd880d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:15.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd87fff700 1 --2- 172.21.15.32:0/2433322713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880d1da0 0x7fdd880d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.857 7fdd87fff700 1 -- 172.21.15.32:0/2433322713 >> v1:172.21.15.86:6789/0 conn(0x7fdd880d82c0 legacy=0x7fdd8810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.858 7fdd87fff700 1 -- 172.21.15.32:0/2433322713 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8810e1f0 con 0x7fdd8810d010 2021-04-09T18:15:15.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.858 7fdd877fe700 1 -- 172.21.15.32:0/2433322713 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182248209 0 0) 0x7fdd80001510 con 0x7fdd880d82c0 2021-04-09T18:15:15.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.858 7fdd7ffff700 1 --2- 172.21.15.32:0/2433322713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880d1da0 0x7fdd880d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:15.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.858 7fdd87fff700 1 --2- 172.21.15.32:0/2433322713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8810d010 0x7fdd880d1780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fdd78006650 tx=0x7fdd78009a80).ready entity=mon.1 client_cookie=1c10901a9687d73f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:15.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.858 7fdd877fe700 1 -- 172.21.15.32:0/2433322713 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd780009b0 con 0x7fdd8810d010 2021-04-09T18:15:15.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.858 7fdd877fe700 1 -- 172.21.15.32:0/2433322713 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7800c0c0 con 0x7fdd8810d010 2021-04-09T18:15:15.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.858 7fdd877fe700 1 -- 172.21.15.32:0/2433322713 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd7800c510 con 0x7fdd8810d010 2021-04-09T18:15:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.858 7fdd8e86e700 1 -- 172.21.15.32:0/2433322713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8810d010 msgr2=0x7fdd880d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.858 7fdd8e86e700 1 --2- 172.21.15.32:0/2433322713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8810d010 0x7fdd880d1780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fdd78006650 tx=0x7fdd78009a80).stop 2021-04-09T18:15:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.859 7fdd8e86e700 1 -- 172.21.15.32:0/2433322713 shutdown_connections 2021-04-09T18:15:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.859 7fdd8e86e700 1 --2- 172.21.15.32:0/2433322713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880d1da0 0x7fdd880d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.859 7fdd8e86e700 1 --2- 172.21.15.32:0/2433322713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8810d010 0x7fdd880d1780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:15.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.859 7fdd8e86e700 1 -- 172.21.15.32:0/2433322713 >> 172.21.15.32:0/2433322713 conn(0x7fdd880ce5f0 msgr2=0x7fdd880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:15.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.859 7fdd8e86e700 1 -- 172.21.15.32:0/2433322713 shutdown_connections 2021-04-09T18:15:15.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.859 7fdd8e86e700 1 -- 172.21.15.32:0/2433322713 wait complete. 2021-04-09T18:15:15.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.879 7fdd8e86e700 1 Processor -- start 2021-04-09T18:15:15.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.879 7fdd8e86e700 1 -- start start 2021-04-09T18:15:15.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.879 7fdd8e86e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880a90c0 0x7fdd881a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:15.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.879 7fdd8e86e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd881a76d0 0x7fdd881a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:15.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.879 7fdd8e86e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8817bce0 0x7fdd8817c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:15.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.879 7fdd8e86e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd881a7c60 con 0x7fdd881a76d0 2021-04-09T18:15:15.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.879 7fdd8e86e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd881a8e20 con 0x7fdd8817bce0 2021-04-09T18:15:15.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.879 7fdd8e86e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8817c700 con 0x7fdd880a90c0 2021-04-09T18:15:15.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd7ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd881a76d0 0x7fdd881a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:15.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd7ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd881a76d0 0x7fdd881a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43918/0 (socket says 172.21.15.32:43918) 2021-04-09T18:15:15.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd7ffff700 1 -- 172.21.15.32:0/2254099188 learned_addr learned my addr 172.21.15.32:0/2254099188 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd8ce8b700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8817bce0 0x7fdd8817c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd87fff700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880a90c0 0x7fdd881a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:15.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd7ffff700 1 -- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880a90c0 msgr2=0x7fdd881a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:15.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd7ffff700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880a90c0 0x7fdd881a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:15.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd7ffff700 1 -- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8817bce0 msgr2=0x7fdd8817c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:15.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd7ffff700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8817bce0 0x7fdd8817c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:15.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd7ffff700 1 -- 172.21.15.32:0/2254099188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8817c920 con 0x7fdd881a76d0 2021-04-09T18:15:15.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd8ce8b700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8817bce0 0x7fdd8817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:15.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd7ffff700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd881a76d0 0x7fdd881a8800 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fdd70006f40 tx=0x7fdd7000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:15.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd857fa700 1 -- 172.21.15.32:0/2254099188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd7000bd80 con 0x7fdd881a76d0 2021-04-09T18:15:16.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd857fa700 1 -- 172.21.15.32:0/2254099188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd70000c90 con 0x7fdd881a76d0 2021-04-09T18:15:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd857fa700 1 -- 172.21.15.32:0/2254099188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd70002220 con 0x7fdd881a76d0 2021-04-09T18:15:16.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8817c920 con 0x7fdd881a76d0 2021-04-09T18:15:16.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.880 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8817c920 con 0x7fdd881a76d0 2021-04-09T18:15:16.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.881 7fdd857fa700 1 -- 172.21.15.32:0/2254099188 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdd7000a000 con 0x7fdd881a76d0 2021-04-09T18:15:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.881 7fdd857fa700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdd6c0188c0 0x7fdd6c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.881 7fdd857fa700 1 -- 172.21.15.32:0/2254099188 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdd700327b0 con 0x7fdd881a76d0 2021-04-09T18:15:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.881 7fdd87fff700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdd6c0188c0 0x7fdd6c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:16.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.881 7fdd87fff700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdd6c0188c0 0x7fdd6c01adc0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.900 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd881b7010 con 0x7fdd881a76d0 2021-04-09T18:15:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:15.904 7fdd857fa700 1 -- 172.21.15.32:0/2254099188 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdd7002cf80 con 0x7fdd881a76d0 2021-04-09T18:15:16.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.095 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd8810d480 con 0x7fdd881a76d0 2021-04-09T18:15:16.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.096 7fdd857fa700 1 -- 172.21.15.32:0/2254099188 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2789 (secure 0 0 0) 0x7fdd7002cf80 con 0x7fdd881a76d0 2021-04-09T18:15:16.134 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 36 2021-04-09T18:15:16.135 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:16.135 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/1797590590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1797590590},{"type":"v1","addr":"172.21.15.86:6835","nonce":1797590590}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":34},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:15:15.703898","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5136,"mds_1":5151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5136":{"gid":5136,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":11,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_5151":{"gid":5151,"name":"c","rank":1,"incarnation":35,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:16.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdd6c0188c0 msgr2=0x7fdd6c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:16.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdd6c0188c0 0x7fdd6c01adc0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:16.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd881a76d0 msgr2=0x7fdd881a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:16.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd881a76d0 0x7fdd881a8800 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fdd70006f40 tx=0x7fdd7000b990).stop 2021-04-09T18:15:16.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 shutdown_connections 2021-04-09T18:15:16.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdd6c0188c0 0x7fdd6c01adc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:16.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd881a76d0 0x7fdd881a8800 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:16.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd8817bce0 0x7fdd8817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:16.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 --2- 172.21.15.32:0/2254099188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd880a90c0 0x7fdd881a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:16.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 >> 172.21.15.32:0/2254099188 conn(0x7fdd881acae0 msgr2=0x7fdd8817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:16.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 shutdown_connections 2021-04-09T18:15:16.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:16.098 7fdd8e86e700 1 -- 172.21.15.32:0/2254099188 wait complete. 2021-04-09T18:15:16.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:15:15.703898', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 5136, 'mds_1': 5151}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5136': {'gid': 5136, 'name': 'a', 'rank': 0, 'incarnation': 33, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.32:6833/3664667451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3664667451}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3664667451}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_5151': {'gid': 5151, 'name': 'c', 'rank': 1, 'incarnation': 35, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.32:6835/1051691429', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1051691429}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1051691429}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:16.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:15:16.602 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:15:16.751 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:15:16.752 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:15:16.752 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:15:16.753 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5136, 2021-04-09T18:15:16.753 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:15:16.754 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:15:16.754 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 3.0270676989999998, 2021-04-09T18:15:16.755 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 36, 2021-04-09T18:15:16.759 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 43, 2021-04-09T18:15:16.760 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 43, 2021-04-09T18:15:16.760 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 42.351333472999997 2021-04-09T18:15:16.761 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:15:16.763 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5136, "want_state": "up:active", "state": "up:active", "rank_uptime": 3.0270676989999998, "mdsmap_epoch": 36, "osdmap_epoch": 43, "osdmap_epoch_barrier": 43, "uptime": 42.351333472999997 } 2021-04-09T18:15:16.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:15:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:15:16.906 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:15:16.907 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:15:16.907 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5151, 2021-04-09T18:15:16.908 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:15:16.908 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:15:16.909 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 2.178628094, 2021-04-09T18:15:16.909 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 36, 2021-04-09T18:15:16.910 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 43, 2021-04-09T18:15:16.924 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 43, 2021-04-09T18:15:16.925 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 42.44831156 2021-04-09T18:15:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:15:16.927 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 5151, "want_state": "up:active", "state": "up:active", "rank_uptime": 2.178628094, "mdsmap_epoch": 36, "osdmap_epoch": 43, "osdmap_epoch_barrier": 43, "uptime": 42.44831156 } 2021-04-09T18:15:16.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:17.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.065 7f6374934700 1 Processor -- start 2021-04-09T18:15:17.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.065 7f6374934700 1 -- start start 2021-04-09T18:15:17.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f6374934700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63700d7ca0 0x7f63700d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:17.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f6374934700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f637010da00 0x7f637010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:17.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f6374934700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637010d480 con 0x7f637010d010 2021-04-09T18:15:17.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f6374934700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63700d9220 con 0x7f637010da00 2021-04-09T18:15:17.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f6374934700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63700d86c0 con 0x7f63700d7ca0 2021-04-09T18:15:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636de1b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63700d7ca0 0x7f63700d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636ee1d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f637010da00 0x7f637010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:17.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636de1b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63700d7ca0 0x7f63700d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:51156/0 (socket says 172.21.15.32:51156) 2021-04-09T18:15:17.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636ee1d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f637010da00 0x7f637010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:35686/0 (socket says 172.21.15.32:35686) 2021-04-09T18:15:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636de1b700 1 -- 172.21.15.32:0/902878874 learned_addr learned my addr 172.21.15.32:0/902878874 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636ee1d700 1 -- 172.21.15.32:0/902878874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63700d7ca0 msgr2=0x7f63700d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636ee1d700 1 --2- 172.21.15.32:0/902878874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63700d7ca0 0x7f63700d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636ee1d700 1 -- 172.21.15.32:0/902878874 >> v1:172.21.15.86:6789/0 conn(0x7f637010d010 legacy=0x7f63700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:17.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636d61a700 1 -- 172.21.15.32:0/902878874 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216714834 0 0) 0x7f6360000fe0 con 0x7f637010d010 2021-04-09T18:15:17.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636ee1d700 1 -- 172.21.15.32:0/902878874 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63700d1f70 con 0x7f637010da00 2021-04-09T18:15:17.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.066 7f636de1b700 1 --2- 172.21.15.32:0/902878874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63700d7ca0 0x7f63700d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.067 7f636ee1d700 1 --2- 172.21.15.32:0/902878874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637010da00 0x7f637010de00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f6368006ad0 tx=0x7f6368009ad0).ready entity=mon.1 client_cookie=fcbd8b860d33fdc4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.067 7f636d61a700 1 -- 172.21.15.32:0/902878874 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63680022e0 con 0x7f637010da00 2021-04-09T18:15:17.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.067 7f636d61a700 1 -- 172.21.15.32:0/902878874 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6368006410 con 0x7f637010da00 2021-04-09T18:15:17.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.067 7f636d61a700 1 -- 172.21.15.32:0/902878874 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6368002620 con 0x7f637010da00 2021-04-09T18:15:17.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.067 7f6374934700 1 -- 172.21.15.32:0/902878874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637010da00 msgr2=0x7f637010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:17.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.067 7f6374934700 1 --2- 172.21.15.32:0/902878874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637010da00 0x7f637010de00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f6368006ad0 tx=0x7f6368009ad0).stop 2021-04-09T18:15:17.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.068 7f6374934700 1 -- 172.21.15.32:0/902878874 shutdown_connections 2021-04-09T18:15:17.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.068 7f6374934700 1 --2- 172.21.15.32:0/902878874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637010da00 0x7f637010de00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.068 7f6374934700 1 --2- 172.21.15.32:0/902878874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63700d7ca0 0x7f63700d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.068 7f6374934700 1 -- 172.21.15.32:0/902878874 >> 172.21.15.32:0/902878874 conn(0x7f63700ce5f0 msgr2=0x7f63700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:17.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.068 7f6374934700 1 -- 172.21.15.32:0/902878874 shutdown_connections 2021-04-09T18:15:17.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.068 7f6374934700 1 -- 172.21.15.32:0/902878874 wait complete. 2021-04-09T18:15:17.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.088 7f6374934700 1 Processor -- start 2021-04-09T18:15:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.088 7f6374934700 1 -- start start 2021-04-09T18:15:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.088 7f6374934700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63701a91e0 0x7f63700a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.088 7f6374934700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63701a95e0 0x7f63700a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.088 7f6374934700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63701ac850 0x7f63701acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.088 7f6374934700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63701ad270 con 0x7f63701a91e0 2021-04-09T18:15:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.088 7f6374934700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63700b7640 con 0x7f63701a95e0 2021-04-09T18:15:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.088 7f6374934700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6370183c80 con 0x7f63701ac850 2021-04-09T18:15:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636e61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63701a91e0 0x7f63700a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:17.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636e61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63701a91e0 0x7f63700a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43934/0 (socket says 172.21.15.32:43934) 2021-04-09T18:15:17.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636e61c700 1 -- 172.21.15.32:0/3728575976 learned_addr learned my addr 172.21.15.32:0/3728575976 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:17.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636ee1d700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63701ac850 0x7f63701acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:17.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636de1b700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63701a95e0 0x7f63700a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:17.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636e61c700 1 -- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63701ac850 msgr2=0x7f63701acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:17.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636e61c700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63701ac850 0x7f63701acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636e61c700 1 -- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63701a95e0 msgr2=0x7f63700a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:17.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636e61c700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63701a95e0 0x7f63700a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636e61c700 1 -- 172.21.15.32:0/3728575976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6370183ea0 con 0x7f63701a91e0 2021-04-09T18:15:17.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636de1b700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63701a95e0 0x7f63700a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636ee1d700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63701ac850 0x7f63701acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f636e61c700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63701a91e0 0x7f63700a4a60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f6360009980 tx=0x7f636000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:17.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f635effd700 1 -- 172.21.15.32:0/3728575976 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6360003030 con 0x7f63701a91e0 2021-04-09T18:15:17.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f635effd700 1 -- 172.21.15.32:0/3728575976 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6360001400 con 0x7f63701a91e0 2021-04-09T18:15:17.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f6374934700 1 -- 172.21.15.32:0/3728575976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6370183ea0 con 0x7f63701a91e0 2021-04-09T18:15:17.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f635effd700 1 -- 172.21.15.32:0/3728575976 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6360000f40 con 0x7f63701a91e0 2021-04-09T18:15:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.089 7f6374934700 1 -- 172.21.15.32:0/3728575976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6370183ea0 con 0x7f63701a91e0 2021-04-09T18:15:17.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.090 7f635effd700 1 -- 172.21.15.32:0/3728575976 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f63600015e0 con 0x7f63701a91e0 2021-04-09T18:15:17.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.090 7f635effd700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63540188d0 0x7f635401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:17.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.090 7f635effd700 1 -- 172.21.15.32:0/3728575976 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6360032da0 con 0x7f63701a91e0 2021-04-09T18:15:17.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.090 7f636de1b700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63540188d0 0x7f635401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:17.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.091 7f636de1b700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63540188d0 0x7f635401add0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:17.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.110 7f6374934700 1 -- 172.21.15.32:0/3728575976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6370182d70 con 0x7f63701a91e0 2021-04-09T18:15:17.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.113 7f635effd700 1 -- 172.21.15.32:0/3728575976 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f636000be10 con 0x7f63701a91e0 2021-04-09T18:15:17.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.304 7f6374934700 1 -- 172.21.15.32:0/3728575976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f637010d480 con 0x7f63701a91e0 2021-04-09T18:15:17.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.305 7f635effd700 1 -- 172.21.15.32:0/3728575976 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2789 (secure 0 0 0) 0x7f6360001400 con 0x7f63701a91e0 2021-04-09T18:15:17.474 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 36 2021-04-09T18:15:17.515 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:17.518 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/1797590590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1797590590},{"type":"v1","addr":"172.21.15.86:6835","nonce":1797590590}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":34},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:15:15.703898","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5136,"mds_1":5151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5136":{"gid":5136,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":11,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_5151":{"gid":5151,"name":"c","rank":1,"incarnation":35,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:17.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 -- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63540188d0 msgr2=0x7f635401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:17.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63540188d0 0x7f635401add0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 -- 172.21.15.32:0/3728575976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63701a91e0 msgr2=0x7f63700a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:17.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63701a91e0 0x7f63700a4a60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f6360009980 tx=0x7f636000bcd0).stop 2021-04-09T18:15:17.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 -- 172.21.15.32:0/3728575976 shutdown_connections 2021-04-09T18:15:17.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63540188d0 0x7f635401add0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63701ac850 0x7f63701acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63701a95e0 0x7f63700a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 --2- 172.21.15.32:0/3728575976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63701a91e0 0x7f63700a4a60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:17.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 -- 172.21.15.32:0/3728575976 >> 172.21.15.32:0/3728575976 conn(0x7f637019b460 msgr2=0x7f637019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:17.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 -- 172.21.15.32:0/3728575976 shutdown_connections 2021-04-09T18:15:17.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:17.307 7f6374934700 1 -- 172.21.15.32:0/3728575976 wait complete. 2021-04-09T18:15:18.004 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:14:39.918931', 'modified': '2021-04-09 18:15:15.703898', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 5136, 'mds_1': 5151}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5136': {'gid': 5136, 'name': 'a', 'rank': 0, 'incarnation': 33, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.32:6833/3664667451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3664667451}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3664667451}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_5151': {'gid': 5151, 'name': 'c', 'rank': 1, 'incarnation': 35, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.32:6835/1051691429', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 1051691429}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1051691429}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:18.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:15:18.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:15:18.145 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:15:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:15:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:15:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5136, 2021-04-09T18:15:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:15:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:15:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 4.4194564610000002, 2021-04-09T18:15:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 36, 2021-04-09T18:15:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 43, 2021-04-09T18:15:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 43, 2021-04-09T18:15:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 43.743724411999999 2021-04-09T18:15:18.154 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:15:18.156 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5136, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.4194564610000002, "mdsmap_epoch": 36, "osdmap_epoch": 43, "osdmap_epoch_barrier": 43, "uptime": 43.743724411999999 } 2021-04-09T18:15:18.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:15:18.320 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:15:18.321 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:15:18.321 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:15:18.322 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5151, 2021-04-09T18:15:18.322 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:15:18.322 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:15:18.329 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 3.5931523890000001, 2021-04-09T18:15:18.330 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 36, 2021-04-09T18:15:18.331 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 43, 2021-04-09T18:15:18.331 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 43, 2021-04-09T18:15:18.332 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 43.862834089000003 2021-04-09T18:15:18.333 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:15:18.335 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 5151, "want_state": "up:active", "state": "up:active", "rank_uptime": 3.5931523890000001, "mdsmap_epoch": 36, "osdmap_epoch": 43, "osdmap_epoch_barrier": 43, "uptime": 43.862834089000003 } 2021-04-09T18:15:18.368 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:15:18.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:15:18.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:15:18.540 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:15:18.541 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:18.542 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:15:18.542 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:15:18.641 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:15:18.642 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:15:18.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:15:18.643 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:18.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:18.732 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:15:18.733 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:15:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:15:18.816 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:15:18.816 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:15:18.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:15:18.820 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:15:18.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:15:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.020 7fcaaf7bd700 1 Processor -- start 2021-04-09T18:15:19.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.021 7fcaaf7bd700 1 -- start start 2021-04-09T18:15:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.021 7fcaaf7bd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcaa80d7ca0 0x7fcaa80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:19.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.021 7fcaaf7bd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa810d480 con 0x7fcaa810d010 2021-04-09T18:15:19.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.021 7fcaaf7bd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa80d86c0 con 0x7fcaa810da00 2021-04-09T18:15:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.021 7fcaaf7bd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa80d9220 con 0x7fcaa80d7ca0 2021-04-09T18:15:19.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.021 7fcaad5d9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcaa810d010 0x7fcaa80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53530/0 (socket says 172.21.15.32:53530) 2021-04-09T18:15:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.021 7fcaad5d9700 1 -- 172.21.15.32:0/3194469031 learned_addr learned my addr 172.21.15.32:0/3194469031 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:19.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.021 7fcaacdd8700 1 --2- 172.21.15.32:0/3194469031 >> v2:172.21.15.86:3300/0 conn(0x7fcaa80d7ca0 0x7fcaa80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fca9ffff700 1 -- 172.21.15.32:0/3194469031 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688149557 0 0) 0x7fcaa4000ce0 con 0x7fcaa810d010 2021-04-09T18:15:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fca9ffff700 1 -- 172.21.15.32:0/3194469031 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca94002fe0 con 0x7fcaa810d010 2021-04-09T18:15:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fca9ffff700 1 -- 172.21.15.32:0/3194469031 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819000976 0 0) 0x7fcaa0000d30 con 0x7fcaa810da00 2021-04-09T18:15:19.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fca9ffff700 1 -- 172.21.15.32:0/3194469031 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca94003210 con 0x7fcaa810da00 2021-04-09T18:15:19.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fcaacdd8700 1 -- 172.21.15.32:0/3194469031 >> v1:172.21.15.86:6789/0 conn(0x7fcaa810da00 legacy=0x7fcaa810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fcaacdd8700 1 -- 172.21.15.32:0/3194469031 >> v1:172.21.15.32:6789/0 conn(0x7fcaa810d010 legacy=0x7fcaa80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fcaacdd8700 1 -- 172.21.15.32:0/3194469031 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa80d1f70 con 0x7fcaa80d7ca0 2021-04-09T18:15:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fcaacdd8700 1 --2- 172.21.15.32:0/3194469031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa80d7ca0 0x7fcaa80d80a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fca98006650 tx=0x7fca98009a80).ready entity=mon.1 client_cookie=ab8208dc9e22225a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fca9ffff700 1 -- 172.21.15.32:0/3194469031 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca980009b0 con 0x7fcaa80d7ca0 2021-04-09T18:15:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fca9ffff700 1 -- 172.21.15.32:0/3194469031 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9800c0c0 con 0x7fcaa80d7ca0 2021-04-09T18:15:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fca9ffff700 1 -- 172.21.15.32:0/3194469031 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca9800c510 con 0x7fcaa80d7ca0 2021-04-09T18:15:19.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fcaaf7bd700 1 -- 172.21.15.32:0/3194469031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa80d7ca0 msgr2=0x7fcaa80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:19.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.022 7fcaaf7bd700 1 --2- 172.21.15.32:0/3194469031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa80d7ca0 0x7fcaa80d80a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fca98006650 tx=0x7fca98009a80).stop 2021-04-09T18:15:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.023 7fcaaf7bd700 1 -- 172.21.15.32:0/3194469031 shutdown_connections 2021-04-09T18:15:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.023 7fcaaf7bd700 1 --2- 172.21.15.32:0/3194469031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa80d7ca0 0x7fcaa80d80a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.023 7fcaaf7bd700 1 -- 172.21.15.32:0/3194469031 >> 172.21.15.32:0/3194469031 conn(0x7fcaa80ce5f0 msgr2=0x7fcaa80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:19.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.023 7fcaaf7bd700 1 -- 172.21.15.32:0/3194469031 shutdown_connections 2021-04-09T18:15:19.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.023 7fcaaf7bd700 1 -- 172.21.15.32:0/3194469031 wait complete. 2021-04-09T18:15:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.043 7fcaaf7bd700 1 Processor -- start 2021-04-09T18:15:19.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.043 7fcaaf7bd700 1 -- start start 2021-04-09T18:15:19.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaaf7bd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcaa8185a00 0x7fcaa8097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaaf7bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaa80983b0 0x7fcaa817abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaaf7bd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa817b1d0 0x7fcaa8179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaaf7bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa80987b0 con 0x7fcaa80983b0 2021-04-09T18:15:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaaf7bd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa817b5d0 con 0x7fcaa817b1d0 2021-04-09T18:15:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaaf7bd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa8179c30 con 0x7fcaa8185a00 2021-04-09T18:15:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaacdd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaa80983b0 0x7fcaa817abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaacdd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaa80983b0 0x7fcaa817abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43944/0 (socket says 172.21.15.32:43944) 2021-04-09T18:15:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaacdd8700 1 -- 172.21.15.32:0/3983061120 learned_addr learned my addr 172.21.15.32:0/3983061120 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaad5d9700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcaa8185a00 0x7fcaa8097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:19.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaaddda700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa817b1d0 0x7fcaa8179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:19.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaacdd8700 1 -- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcaa8185a00 msgr2=0x7fcaa8097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:19.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaacdd8700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcaa8185a00 0x7fcaa8097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:19.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaacdd8700 1 -- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa817b1d0 msgr2=0x7fcaa8179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaacdd8700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa817b1d0 0x7fcaa8179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaacdd8700 1 -- 172.21.15.32:0/3983061120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa8179e50 con 0x7fcaa80983b0 2021-04-09T18:15:19.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.044 7fcaacdd8700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaa80983b0 0x7fcaa817abb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fca98009650 tx=0x7fca9800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.045 7fca9dffb700 1 -- 172.21.15.32:0/3983061120 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca980093a0 con 0x7fcaa80983b0 2021-04-09T18:15:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.045 7fca9dffb700 1 -- 172.21.15.32:0/3983061120 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9800bd50 con 0x7fcaa80983b0 2021-04-09T18:15:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.045 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa8179e50 con 0x7fcaa80983b0 2021-04-09T18:15:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.045 7fca9dffb700 1 -- 172.21.15.32:0/3983061120 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca980052e0 con 0x7fcaa80983b0 2021-04-09T18:15:19.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.045 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa8179e50 con 0x7fcaa80983b0 2021-04-09T18:15:19.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.045 7fca9dffb700 1 -- 172.21.15.32:0/3983061120 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fca98006030 con 0x7fcaa80983b0 2021-04-09T18:15:19.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.046 7fca9dffb700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca880188d0 0x7fca8801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:19.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.046 7fca9dffb700 1 -- 172.21.15.32:0/3983061120 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fca980329e0 con 0x7fcaa80983b0 2021-04-09T18:15:19.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.046 7fcaad5d9700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca880188d0 0x7fca8801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:19.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.046 7fcaad5d9700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca880188d0 0x7fca8801add0 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:19.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.065 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa817a3e0 con 0x7fcaa80983b0 2021-04-09T18:15:19.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.069 7fca9dffb700 1 -- 172.21.15.32:0/3983061120 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fca980009c0 con 0x7fcaa80983b0 2021-04-09T18:15:19.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.258 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fcaa80987b0 con 0x7fcaa80983b0 2021-04-09T18:15:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.259 7fca9dffb700 1 -- 172.21.15.32:0/3983061120 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v43) v1 ==== 81+0+30360 (secure 0 0 0) 0x7fca98006030 con 0x7fcaa80983b0 2021-04-09T18:15:19.485 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:19.485 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:15:19.486 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 43, 2021-04-09T18:15:19.486 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:15:19.487 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:15:19.504 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:14:40.920336", 2021-04-09T18:15:19.505 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:15:19.505 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:15:19.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:15:19.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:15:19.507 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:15:19.507 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:15:19.507 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:15:19.508 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:15:19.508 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:15:19.509 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:15:19.509 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:15:19.509 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:15:19.510 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:15:19.510 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:15:19.511 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:15:19.592 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 7, 2021-04-09T18:15:19.596 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:15:19.596 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:15:19.597 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:15:19.597 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:15:19.598 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:15:19.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:19.611 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:15:19.612 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:15:19.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:15:19.621 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:15:19.622 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:15:19.649 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:15:19.650 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:15:19.659 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:15:19.660 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:15:19.661 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:15:19.689 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:19.690 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:15:19.693 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:15:19.698 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:15:19.707 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:15:19.708 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:15:19.708 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:15:19.708 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:15:19.709 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:15:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:15:19.727 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:15:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:15:19.729 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:15:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:15:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:15:19.743 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:15:19.744 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:15:19.744 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:15:19.744 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:15:19.744 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:15:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:15:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:15:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:15:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:15:19.745 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:15:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:15:19.746 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:15:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:15:19.756 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:15:19.757 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:15:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:15:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:15:19.760 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:15:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:15:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:15:19.761 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:15:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:15:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:15:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:15:19.788 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:15:19.816 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:15:19.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:19.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:19.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:19.835 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 6, 2021-04-09T18:15:19.840 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:15:19.840 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:14:38.344198", 2021-04-09T18:15:19.843 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:15:19.844 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:15:19.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:15:19.863 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:15:19.863 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:15:19.868 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:15:19.870 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:15:19.871 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:15:19.871 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:15:19.871 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:15:19.875 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:15:19.875 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:15:19.875 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:15:19.876 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:15:19.901 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:15:19.901 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:15:19.901 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:15:19.901 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:15:19.901 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:15:19.902 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:15:19.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:19.902 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "42", 2021-04-09T18:15:19.916 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:15:19.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:15:19.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:15:19.917 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:15:19.917 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:15:19.918 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:15:19.918 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:15:19.918 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:15:19.918 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:15:19.919 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:15:19.927 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:15:19.927 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:15:19.927 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:15:20.047 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:15:20.047 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:15:20.048 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:15:20.048 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:15:20.049 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:15:20.049 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:15:20.049 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:15:20.050 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:15:20.050 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:15:20.051 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:15:20.065 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:15:20.065 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:15:20.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:15:20.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.079 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:15:20.079 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:15:20.080 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:15:20.080 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:15:20.081 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:15:20.081 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:15:20.081 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:15:20.082 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:15:20.082 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:15:20.083 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:15:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:15:20.241 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:15:20.242 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:15:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:15:20.247 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:15:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.248 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:15:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:15:20.249 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:15:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.250 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.312 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 7, 2021-04-09T18:15:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:15:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:14:38.803133", 2021-04-09T18:15:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:15:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:15:20.314 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:15:20.315 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:15:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:15:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:15:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:15:20.364 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:15:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:15:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:15:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:15:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:15:20.365 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:15:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:15:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:15:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:15:20.366 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:15:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:15:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:15:20.367 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:15:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "43", 2021-04-09T18:15:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:15:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:15:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:15:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:15:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:15:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:15:20.369 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:15:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:15:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:15:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:15:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:15:20.508 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:15:20.535 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:15:20.540 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:15:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:15:20.541 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:15:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:15:20.543 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:15:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:15:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:15:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:15:20.544 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:15:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:15:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:15:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:15:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:15:20.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:15:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:15:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:15:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:15:20.546 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:15:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:15:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:15:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:15:20.547 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:15:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:15:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:15:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:15:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:15:20.548 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:15:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:15:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.549 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:15:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:15:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.550 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:15:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:15:20.551 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:20.615 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:20.616 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:20.620 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:20.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:15:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:15:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:20.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:20.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.625 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:20.678 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:15:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:20.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:15:20.681 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.682 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:20.685 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:15:20.686 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:15:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:15:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:20.691 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:15:20.722 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.723 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:15:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:15:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:15:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:15:20.724 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:15:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:15:20.726 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:20.729 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 40, 2021-04-09T18:15:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:20.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.743 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:20.746 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:15:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:20.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:15:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:20.819 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:20.820 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:15:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:20.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:15:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:20.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.836 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:15:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:15:20.841 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.842 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:20.843 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:15:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:15:20.845 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:20.885 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.886 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:15:20.887 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:15:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:15:20.889 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:15:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:20.890 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:20.891 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:20.896 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:20.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:20.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:15:20.901 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:15:20.904 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:20.905 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:20.999 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:21.000 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:15:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:21.014 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:21.015 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.016 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:15:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:21.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.018 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:15:21.062 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.063 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:15:21.064 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:15:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:15:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:15:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:15:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:15:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:15:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:15:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:15:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:15:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:15:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:15:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:15:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:15:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:15:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:15:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:15:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:15:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:15:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:15:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:15:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:15:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:15:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:15:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:15:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:15:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:21.317 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:21.319 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:21.322 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:21.327 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:21.328 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.330 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:15:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:15:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.332 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:15:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:15:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.335 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:15:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:15:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.338 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.364 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:21.389 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.390 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.391 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:15:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:21.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:15:21.458 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:21.459 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.466 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.504 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:15:21.508 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:15:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:15:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:15:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:15:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:21.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:15:21.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:21.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.552 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:15:21.552 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:21.553 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.553 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.558 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:21.558 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:15:21.563 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:21.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.573 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.574 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:15:21.576 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:21.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.587 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.593 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:21.593 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.632 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:15:21.632 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:21.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:15:21.648 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:21.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.649 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.711 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:21.712 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.713 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.731 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:15:21.732 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:21.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.750 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:15:21.750 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:21.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.751 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.752 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:15:21.753 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:15:21.753 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:15:21.753 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:15:21.754 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:21.754 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:21.793 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:21.794 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.795 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:15:21.803 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:15:21.804 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:21.805 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:21.805 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:21.806 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:21.806 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:21.806 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:21.807 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:21.825 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:21.826 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:21.826 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:21.831 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:21.832 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.833 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:15:21.833 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:21.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.835 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:15:21.876 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:21.879 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.880 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.903 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:21.904 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.905 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:15:21.905 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:21.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.943 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:15:21.943 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:21.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.944 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.945 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:21.958 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.959 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:15:21.959 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:21.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:21.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:15:21.965 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:21.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:21.966 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:21.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:21.967 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:21.994 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:21.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:21.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:21.999 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:15:22.003 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:22.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.024 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:22.031 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:15:22.031 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:22.032 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:22.032 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:22.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.033 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:15:22.034 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:15:22.034 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:15:22.042 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:15:22.043 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:22.044 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:22.044 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:22.044 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:22.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:15:22.048 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:15:22.049 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:22.049 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:22.084 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:22.087 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:22.088 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:22.088 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:22.089 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:22.089 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:22.089 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:22.090 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:22.090 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:22.091 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:22.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.109 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:22.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:15:22.110 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:22.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.183 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:22.184 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:15:22.184 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:22.184 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:22.185 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:22.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.186 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:22.186 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:22.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.191 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:22.192 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:15:22.192 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:22.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.195 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:22.195 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:15:22.195 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:22.196 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:22.196 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:22.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.197 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:22.197 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:22.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:22.282 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:15:22.282 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:22.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.283 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:22.283 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:15:22.283 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:22.284 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:22.284 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:22.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.284 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:22.285 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.332 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:15:22.333 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.334 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:22.335 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:15:22.351 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:22.358 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.359 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:15:22.360 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:15:22.361 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:15:22.460 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:22.460 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:22.461 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:22.461 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:22.462 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:22.462 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:15:22.463 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:15:22.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.464 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:15:22.464 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:22.464 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:22.465 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:22.465 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:22.466 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:22.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.492 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:15:22.492 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:22.492 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:22.493 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:22.493 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:22.515 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:22.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.516 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:15:22.516 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:22.530 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:22.530 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:22.531 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:22.531 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:22.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.532 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:15:22.533 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:22.533 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:22.534 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:22.534 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:22.535 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:22.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.536 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:15:22.562 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:22.562 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:22.562 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:22.563 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:22.572 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:22.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.573 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:15:22.574 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:22.574 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:22.574 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:22.575 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:22.605 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:22.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.612 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:15:22.612 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:22.612 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:22.613 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:22.613 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:22.613 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:22.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:22.614 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:15:22.615 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:22.635 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:22.636 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:22.647 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:22.648 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:15:22.655 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:15:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:15:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:15:22.656 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:15:22.657 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:15:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:15:22.660 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:15:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:15:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:15:22.661 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:15:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:15:22.662 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:15:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:15:22.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:22.664 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:15:22.691 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:15:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:15:22.692 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:15:22.693 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:15:22.693 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:15:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca880188d0 msgr2=0x7fca8801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:22.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca880188d0 0x7fca8801add0 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:22.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaa80983b0 msgr2=0x7fcaa817abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:22.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaa80983b0 0x7fcaa817abb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fca98009650 tx=0x7fca9800b960).stop 2021-04-09T18:15:22.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 shutdown_connections 2021-04-09T18:15:22.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca880188d0 0x7fca8801add0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:22.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcaa80983b0 0x7fcaa817abb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:22.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa817b1d0 0x7fcaa8179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:22.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 --2- 172.21.15.32:0/3983061120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcaa8185a00 0x7fcaa8097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:22.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.263 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 >> 172.21.15.32:0/3983061120 conn(0x7fcaa8190550 msgr2=0x7fcaa81b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:22.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.264 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 shutdown_connections 2021-04-09T18:15:22.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:19.264 7fcaaf7bd700 1 -- 172.21.15.32:0/3983061120 wait complete. 2021-04-09T18:15:22.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.013 7fa2ebabd700 1 Processor -- start 2021-04-09T18:15:23.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.013 7fa2ebabd700 1 -- start start 2021-04-09T18:15:23.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ebabd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e410d010 0x7fa2e40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ebabd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa2e40d8490 0x7fa2e410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ebabd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e410d480 con 0x7fa2e40d1da0 2021-04-09T18:15:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ebabd700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e410e780 con 0x7fa2e40d8490 2021-04-09T18:15:23.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ebabd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40d8d30 con 0x7fa2e410d010 2021-04-09T18:15:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ea0da700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa2e40d8490 0x7fa2e410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2e90d8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa2e40d1da0 0x7fa2e40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53546/0 (socket says 172.21.15.32:53546) 2021-04-09T18:15:23.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ea0da700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa2e40d8490 0x7fa2e410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43954/0 (socket says 172.21.15.32:43954) 2021-04-09T18:15:23.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2e90d8700 1 -- 172.21.15.32:0/3562551624 learned_addr learned my addr 172.21.15.32:0/3562551624 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:23.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2e98d9700 1 --2- 172.21.15.32:0/3562551624 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e410d010 0x7fa2e40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ea0da700 1 -- 172.21.15.32:0/3562551624 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e410d010 msgr2=0x7fa2e40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:23.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ea0da700 1 --2- 172.21.15.32:0/3562551624 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e410d010 0x7fa2e40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ea0da700 1 -- 172.21.15.32:0/3562551624 >> v1:172.21.15.32:6789/0 conn(0x7fa2e40d1da0 legacy=0x7fa2e40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2e88d7700 1 -- 172.21.15.32:0/3562551624 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587209431 0 0) 0x7fa2d4000ce0 con 0x7fa2e40d1da0 2021-04-09T18:15:23.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ea0da700 1 -- 172.21.15.32:0/3562551624 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40d9220 con 0x7fa2e40d8490 2021-04-09T18:15:23.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.014 7fa2ea0da700 1 --2- 172.21.15.32:0/3562551624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e40d8490 0x7fa2e410da00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa2dc006e30 tx=0x7fa2dc009de0).ready entity=mon.0 client_cookie=25b4f5cc00171af2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:23.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.015 7fa2e88d7700 1 -- 172.21.15.32:0/3562551624 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2dc00b300 con 0x7fa2e40d8490 2021-04-09T18:15:23.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.015 7fa2e88d7700 1 -- 172.21.15.32:0/3562551624 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc00b680 con 0x7fa2e40d8490 2021-04-09T18:15:23.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.015 7fa2e88d7700 1 -- 172.21.15.32:0/3562551624 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2dc00bb60 con 0x7fa2e40d8490 2021-04-09T18:15:23.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.015 7fa2ebabd700 1 -- 172.21.15.32:0/3562551624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e40d8490 msgr2=0x7fa2e410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.015 7fa2ebabd700 1 --2- 172.21.15.32:0/3562551624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e40d8490 0x7fa2e410da00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa2dc006e30 tx=0x7fa2dc009de0).stop 2021-04-09T18:15:23.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.015 7fa2ebabd700 1 -- 172.21.15.32:0/3562551624 shutdown_connections 2021-04-09T18:15:23.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.015 7fa2ebabd700 1 --2- 172.21.15.32:0/3562551624 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e40d8490 0x7fa2e410da00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.015 7fa2ebabd700 1 --2- 172.21.15.32:0/3562551624 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e410d010 0x7fa2e40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.015 7fa2ebabd700 1 -- 172.21.15.32:0/3562551624 >> 172.21.15.32:0/3562551624 conn(0x7fa2e40ce5f0 msgr2=0x7fa2e40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:23.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.016 7fa2ebabd700 1 -- 172.21.15.32:0/3562551624 shutdown_connections 2021-04-09T18:15:23.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.016 7fa2ebabd700 1 -- 172.21.15.32:0/3562551624 wait complete. 2021-04-09T18:15:23.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2ebabd700 1 Processor -- start 2021-04-09T18:15:23.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2ebabd700 1 -- start start 2021-04-09T18:15:23.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2ebabd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e40b5360 0x7fa2e41a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2ebabd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e418e140 0x7fa2e41aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2ebabd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2e41ad040 0x7fa2e408fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2ebabd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e41a5360 con 0x7fa2e418e140 2021-04-09T18:15:23.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2ebabd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40902c0 con 0x7fa2e41ad040 2021-04-09T18:15:23.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2ebabd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40904e0 con 0x7fa2e40b5360 2021-04-09T18:15:23.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2e90d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e418e140 0x7fa2e41aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2e90d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e418e140 0x7fa2e41aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43958/0 (socket says 172.21.15.32:43958) 2021-04-09T18:15:23.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.036 7fa2e90d8700 1 -- 172.21.15.32:0/906544985 learned_addr learned my addr 172.21.15.32:0/906544985 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:23.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2ea0da700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2e41ad040 0x7fa2e408fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2e98d9700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e40b5360 0x7fa2e41a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2e90d8700 1 -- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e40b5360 msgr2=0x7fa2e41a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2e90d8700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e40b5360 0x7fa2e41a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2e90d8700 1 -- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2e41ad040 msgr2=0x7fa2e408fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2e90d8700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2e41ad040 0x7fa2e408fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2e90d8700 1 -- 172.21.15.32:0/906544985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e4183c80 con 0x7fa2e418e140 2021-04-09T18:15:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2ea0da700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2e41ad040 0x7fa2e408fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2e90d8700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e418e140 0x7fa2e41aca20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa2d4006f00 tx=0x7fa2d400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2da7fc700 1 -- 172.21.15.32:0/906544985 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2d4006020 con 0x7fa2e418e140 2021-04-09T18:15:23.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2da7fc700 1 -- 172.21.15.32:0/906544985 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d40063a0 con 0x7fa2e418e140 2021-04-09T18:15:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2da7fc700 1 -- 172.21.15.32:0/906544985 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2d4004620 con 0x7fa2e418e140 2021-04-09T18:15:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e4183c80 con 0x7fa2e418e140 2021-04-09T18:15:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.037 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e4183c80 con 0x7fa2e418e140 2021-04-09T18:15:23.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.038 7fa2da7fc700 1 -- 172.21.15.32:0/906544985 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa2d400a000 con 0x7fa2e418e140 2021-04-09T18:15:23.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.038 7fa2da7fc700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2cc0188c0 0x7fa2cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.038 7fa2da7fc700 1 -- 172.21.15.32:0/906544985 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa2d40327c0 con 0x7fa2e418e140 2021-04-09T18:15:23.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.038 7fa2e98d9700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2cc0188c0 0x7fa2cc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.038 7fa2e98d9700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2cc0188c0 0x7fa2cc01adc0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:23.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.058 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4182d70 con 0x7fa2e418e140 2021-04-09T18:15:23.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.061 7fa2da7fc700 1 -- 172.21.15.32:0/906544985 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa2d4006b90 con 0x7fa2e418e140 2021-04-09T18:15:23.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.253 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2e410d480 con 0x7fa2e418e140 2021-04-09T18:15:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.254 7fa2da7fc700 1 -- 172.21.15.32:0/906544985 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2789 (secure 0 0 0) 0x7fa2d400a000 con 0x7fa2e418e140 2021-04-09T18:15:23.281 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 36 2021-04-09T18:15:23.282 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:23.282 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/1797590590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1797590590},{"type":"v1","addr":"172.21.15.86:6835","nonce":1797590590}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":34},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:15:15.703898","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5136,"mds_1":5151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5136":{"gid":5136,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":11,"addr":"172.21.15.32:6833/3664667451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3664667451},{"type":"v1","addr":"172.21.15.32:6833","nonce":3664667451}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_5151":{"gid":5151,"name":"c","rank":1,"incarnation":35,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6835/1051691429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":1051691429},{"type":"v1","addr":"172.21.15.32:6835","nonce":1051691429}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2cc0188c0 msgr2=0x7fa2cc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2cc0188c0 0x7fa2cc01adc0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e418e140 msgr2=0x7fa2e41aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e418e140 0x7fa2e41aca20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa2d4006f00 tx=0x7fa2d400b940).stop 2021-04-09T18:15:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 shutdown_connections 2021-04-09T18:15:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2cc0188c0 0x7fa2cc01adc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2e418e140 0x7fa2e41aca20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2e41ad040 0x7fa2e408fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 --2- 172.21.15.32:0/906544985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2e40b5360 0x7fa2e41a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 >> 172.21.15.32:0/906544985 conn(0x7fa2e419b460 msgr2=0x7fa2e419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:23.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 shutdown_connections 2021-04-09T18:15:23.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.256 7fa2ebabd700 1 -- 172.21.15.32:0/906544985 wait complete. 2021-04-09T18:15:23.323 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:15:23.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.466 7fca55b96700 1 Processor -- start 2021-04-09T18:15:23.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.466 7fca55b96700 1 -- start start 2021-04-09T18:15:23.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca55b96700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca500cffb0 0x7fca500d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca55b96700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca500d7290 con 0x7fca500d6c90 2021-04-09T18:15:23.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca55b96700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5010e7e0 con 0x7fca500d7810 2021-04-09T18:15:23.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca55b96700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca500d7090 con 0x7fca500cffb0 2021-04-09T18:15:23.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fca500d6c90 0x7fca5010cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53558/0 (socket says 172.21.15.32:53558) 2021-04-09T18:15:23.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4ffff700 1 -- 172.21.15.32:0/678878868 learned_addr learned my addr 172.21.15.32:0/678878868 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:23.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4f7fe700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca500cffb0 0x7fca500d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861468606 0 0) 0x7fca480009b0 con 0x7fca500d6c90 2021-04-09T18:15:23.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca38002fe0 con 0x7fca500d6c90 2021-04-09T18:15:23.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661032285 0 0) 0x7fca44000c60 con 0x7fca500d7810 2021-04-09T18:15:23.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca38003210 con 0x7fca500d7810 2021-04-09T18:15:23.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 846317167 0 0) 0x7fca480010e0 con 0x7fca500d6c90 2021-04-09T18:15:23.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca500cffb0 msgr2=0x7fca500d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:23.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca500cffb0 0x7fca500d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 >> v1:172.21.15.86:6789/0 conn(0x7fca500d7810 legacy=0x7fca500d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca500d8590 con 0x7fca500d6c90 2021-04-09T18:15:23.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fca48001600 con 0x7fca500d6c90 2021-04-09T18:15:23.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4f7fe700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca500cffb0 0x7fca500d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:23.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 >> v1:172.21.15.32:6789/0 conn(0x7fca500d6c90 legacy=0x7fca5010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca38003b90 0x7fca38005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca380067a0 0x7fca38008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.467 7fca4e7fc700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca38009510 0x7fca3800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca380502f0 con 0x7fca38003b90 2021-04-09T18:15:23.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca38050510 con 0x7fca38009510 2021-04-09T18:15:23.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca38050730 con 0x7fca380067a0 2021-04-09T18:15:23.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4f7fe700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca38003b90 0x7fca38005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4effd700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca380067a0 0x7fca38008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4ffff700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca38009510 0x7fca3800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4f7fe700 1 -- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca380067a0 msgr2=0x7fca38008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:23.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4f7fe700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca380067a0 0x7fca38008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4f7fe700 1 -- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca38009510 msgr2=0x7fca3800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:23.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4f7fe700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca38009510 0x7fca3800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4f7fe700 1 -- 172.21.15.32:0/678878868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca380509d0 con 0x7fca38003b90 2021-04-09T18:15:23.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4ffff700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca38009510 0x7fca3800ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4f7fe700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca38003b90 0x7fca38005fa0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fca40006f80 tx=0x7fca40009e20).ready entity=mon.0 client_cookie=53bd7416a72c6e7f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca4000b510 con 0x7fca38003b90 2021-04-09T18:15:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4000b850 con 0x7fca38003b90 2021-04-09T18:15:23.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4e7fc700 1 -- 172.21.15.32:0/678878868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca4000bcb0 con 0x7fca38003b90 2021-04-09T18:15:23.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca55b96700 1 -- 172.21.15.32:0/678878868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca38003b90 msgr2=0x7fca38005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca55b96700 1 --2- 172.21.15.32:0/678878868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca38003b90 0x7fca38005fa0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fca40006f80 tx=0x7fca40009e20).stop 2021-04-09T18:15:23.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.468 7fca4ffff700 1 -- 172.21.15.32:0/678878868 reap_dead start 2021-04-09T18:15:23.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.469 7fca55b96700 1 -- 172.21.15.32:0/678878868 shutdown_connections 2021-04-09T18:15:23.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.469 7fca55b96700 1 -- 172.21.15.32:0/678878868 >> 172.21.15.32:0/678878868 conn(0x7fca500ce5d0 msgr2=0x7fca500cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:23.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.469 7fca55b96700 1 -- 172.21.15.32:0/678878868 shutdown_connections 2021-04-09T18:15:23.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.469 7fca55b96700 1 -- 172.21.15.32:0/678878868 wait complete. 2021-04-09T18:15:23.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.489 7fca55b96700 1 Processor -- start 2021-04-09T18:15:23.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.489 7fca55b96700 1 -- start start 2021-04-09T18:15:23.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca55b96700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca50188cf0 0x7fca501a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca55b96700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca501a73d0 0x7fca501a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca55b96700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca501890f0 0x7fca501a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca55b96700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca500b10d0 con 0x7fca501890f0 2021-04-09T18:15:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca55b96700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca501a7890 con 0x7fca501a73d0 2021-04-09T18:15:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca55b96700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca501a4020 con 0x7fca50188cf0 2021-04-09T18:15:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca501890f0 0x7fca501a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca501890f0 0x7fca501a8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43982/0 (socket says 172.21.15.32:43982) 2021-04-09T18:15:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4ffff700 1 -- 172.21.15.32:0/4216949752 learned_addr learned my addr 172.21.15.32:0/4216949752 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:23.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4effd700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca501a73d0 0x7fca501a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4f7fe700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca50188cf0 0x7fca501a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4ffff700 1 -- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca50188cf0 msgr2=0x7fca501a6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4ffff700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca50188cf0 0x7fca501a6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4ffff700 1 -- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca501a73d0 msgr2=0x7fca501a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:23.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4ffff700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca501a73d0 0x7fca501a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:23.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4ffff700 1 -- 172.21.15.32:0/4216949752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca501a4200 con 0x7fca501890f0 2021-04-09T18:15:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4f7fe700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca50188cf0 0x7fca501a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:23.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.490 7fca4ffff700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca501890f0 0x7fca501a8430 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fca48007fa0 tx=0x7fca4800a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.491 7fca37fff700 1 -- 172.21.15.32:0/4216949752 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca48001870 con 0x7fca501890f0 2021-04-09T18:15:23.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.491 7fca37fff700 1 -- 172.21.15.32:0/4216949752 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca480070d0 con 0x7fca501890f0 2021-04-09T18:15:23.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.491 7fca55b96700 1 -- 172.21.15.32:0/4216949752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca501a4200 con 0x7fca501890f0 2021-04-09T18:15:23.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.491 7fca37fff700 1 -- 172.21.15.32:0/4216949752 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca480045b0 con 0x7fca501890f0 2021-04-09T18:15:23.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.491 7fca55b96700 1 -- 172.21.15.32:0/4216949752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca501a4200 con 0x7fca501890f0 2021-04-09T18:15:23.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.491 7fca37fff700 1 -- 172.21.15.32:0/4216949752 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fca480043a0 con 0x7fca501890f0 2021-04-09T18:15:23.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.491 7fca37fff700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca300188c0 0x7fca3001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:23.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.491 7fca37fff700 1 -- 172.21.15.32:0/4216949752 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fca48032990 con 0x7fca501890f0 2021-04-09T18:15:23.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.492 7fca4f7fe700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca300188c0 0x7fca3001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:23.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.492 7fca4f7fe700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca300188c0 0x7fca3001adc0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:23.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.511 7fca55b96700 1 -- 172.21.15.32:0/4216949752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca50182e20 con 0x7fca501890f0 2021-04-09T18:15:23.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.514 7fca37fff700 1 -- 172.21.15.32:0/4216949752 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fca480070d0 con 0x7fca501890f0 2021-04-09T18:15:23.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:23.696 7fca55b96700 1 -- 172.21.15.32:0/4216949752 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fca50182e20 con 0x7fca501890f0 2021-04-09T18:15:24.367 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:15:24.341 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:15:24.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.348 7fca37fff700 1 -- 172.21.15.32:0/4216949752 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) v1 ==== 161+0+0 (secure 0 0 0) 0x7fca480043a0 con 0x7fca501890f0 2021-04-09T18:15:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:15:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 -- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca300188c0 msgr2=0x7fca3001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca300188c0 0x7fca3001adc0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 -- 172.21.15.32:0/4216949752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca501890f0 msgr2=0x7fca501a8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:24.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca501890f0 0x7fca501a8430 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fca48007fa0 tx=0x7fca4800a8a0).stop 2021-04-09T18:15:24.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 -- 172.21.15.32:0/4216949752 shutdown_connections 2021-04-09T18:15:24.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca300188c0 0x7fca3001adc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca501890f0 0x7fca501a8430 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca501a73d0 0x7fca501a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 --2- 172.21.15.32:0/4216949752 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca50188cf0 0x7fca501a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 -- 172.21.15.32:0/4216949752 >> 172.21.15.32:0/4216949752 conn(0x7fca501a36a0 msgr2=0x7fca50185120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 -- 172.21.15.32:0/4216949752 shutdown_connections 2021-04-09T18:15:24.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.364 7fca55b96700 1 -- 172.21.15.32:0/4216949752 wait complete. 2021-04-09T18:15:24.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:24.436 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:15:24.438 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:15:24.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.538 7fe89abda700 1 Processor -- start 2021-04-09T18:15:24.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.538 7fe89abda700 1 -- start start 2021-04-09T18:15:24.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe89abda700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe8940d6c80 0x7fe8940d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:24.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe89abda700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8940d2440 0x7fe89410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:24.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe89abda700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8940d7280 con 0x7fe8940cffa0 2021-04-09T18:15:24.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe89abda700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8940d7080 con 0x7fe8940d2440 2021-04-09T18:15:24.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe89abda700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8940d91e0 con 0x7fe8940d6c80 2021-04-09T18:15:24.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe8991f7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8940d2440 0x7fe89410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:24.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe8991f7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8940d2440 0x7fe89410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44012/0 (socket says 172.21.15.32:44012) 2021-04-09T18:15:24.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe8991f7700 1 -- 172.21.15.32:0/3294527120 learned_addr learned my addr 172.21.15.32:0/3294527120 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:24.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe893fff700 1 --2- 172.21.15.32:0/3294527120 >> v2:172.21.15.86:3300/0 conn(0x7fe8940d6c80 0x7fe8940d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:24.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe8991f7700 1 -- 172.21.15.32:0/3294527120 >> v2:172.21.15.86:3300/0 conn(0x7fe8940d6c80 msgr2=0x7fe8940d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:24.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe8991f7700 1 --2- 172.21.15.32:0/3294527120 >> v2:172.21.15.86:3300/0 conn(0x7fe8940d6c80 0x7fe8940d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe8991f7700 1 -- 172.21.15.32:0/3294527120 >> v1:172.21.15.86:6789/0 conn(0x7fe8940cffa0 legacy=0x7fe8940d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:24.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe8991f7700 1 -- 172.21.15.32:0/3294527120 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8940d2840 con 0x7fe8940d2440 2021-04-09T18:15:24.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe8937fe700 1 -- 172.21.15.32:0/3294527120 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091948234 0 0) 0x7fe884000c60 con 0x7fe8940cffa0 2021-04-09T18:15:24.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe893fff700 1 --2- 172.21.15.32:0/3294527120 >> v2:172.21.15.86:3300/0 conn(0x7fe8940d6c80 0x7fe8940d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:24.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.539 7fe8991f7700 1 --2- 172.21.15.32:0/3294527120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940d2440 0x7fe89410cfd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fe88c006f00 tx=0x7fe88c009f10).ready entity=mon.0 client_cookie=813711fa0820705a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:24.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe8937fe700 1 -- 172.21.15.32:0/3294527120 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe88c00b3b0 con 0x7fe8940d2440 2021-04-09T18:15:24.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe8937fe700 1 -- 172.21.15.32:0/3294527120 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c00b730 con 0x7fe8940d2440 2021-04-09T18:15:24.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe8937fe700 1 -- 172.21.15.32:0/3294527120 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe88c00bc10 con 0x7fe8940d2440 2021-04-09T18:15:24.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe89abda700 1 -- 172.21.15.32:0/3294527120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940d2440 msgr2=0x7fe89410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:24.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe89abda700 1 --2- 172.21.15.32:0/3294527120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940d2440 0x7fe89410cfd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fe88c006f00 tx=0x7fe88c009f10).stop 2021-04-09T18:15:24.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe89abda700 1 -- 172.21.15.32:0/3294527120 shutdown_connections 2021-04-09T18:15:24.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe89abda700 1 --2- 172.21.15.32:0/3294527120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8940d2440 0x7fe89410cfd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe89abda700 1 --2- 172.21.15.32:0/3294527120 >> v2:172.21.15.86:3300/0 conn(0x7fe8940d6c80 0x7fe8940d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe89abda700 1 -- 172.21.15.32:0/3294527120 >> 172.21.15.32:0/3294527120 conn(0x7fe8940ce5c0 msgr2=0x7fe8940cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe89abda700 1 -- 172.21.15.32:0/3294527120 shutdown_connections 2021-04-09T18:15:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.540 7fe89abda700 1 -- 172.21.15.32:0/3294527120 wait complete. 2021-04-09T18:15:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.560 7fe89abda700 1 Processor -- start 2021-04-09T18:15:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe89abda700 1 -- start start 2021-04-09T18:15:24.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe89abda700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8941b85b0 0x7fe8941b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe89abda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe89408c210 0x7fe89408c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe89abda700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8940ab9a0 0x7fe8940abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:24.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe89abda700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89408cc30 con 0x7fe89408c210 2021-04-09T18:15:24.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe89abda700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89419a630 con 0x7fe8941b85b0 2021-04-09T18:15:24.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe89abda700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940ac3c0 con 0x7fe8940ab9a0 2021-04-09T18:15:24.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe893fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe89408c210 0x7fe89408c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:24.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe893fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe89408c210 0x7fe89408c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44018/0 (socket says 172.21.15.32:44018) 2021-04-09T18:15:24.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe893fff700 1 -- 172.21.15.32:0/3647796564 learned_addr learned my addr 172.21.15.32:0/3647796564 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:24.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe8989f6700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8941b85b0 0x7fe8941b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:24.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.561 7fe8991f7700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8940ab9a0 0x7fe8940abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:24.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe893fff700 1 -- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8940ab9a0 msgr2=0x7fe8940abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:24.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe893fff700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8940ab9a0 0x7fe8940abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe893fff700 1 -- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8941b85b0 msgr2=0x7fe8941b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:24.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe893fff700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8941b85b0 0x7fe8941b89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe893fff700 1 -- 172.21.15.32:0/3647796564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8940ac5e0 con 0x7fe89408c210 2021-04-09T18:15:24.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe893fff700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe89408c210 0x7fe89408c610 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fe888006f40 tx=0x7fe88800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:24.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe8917fa700 1 -- 172.21.15.32:0/3647796564 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe88800bd80 con 0x7fe89408c210 2021-04-09T18:15:24.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe8917fa700 1 -- 172.21.15.32:0/3647796564 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe888000c90 con 0x7fe89408c210 2021-04-09T18:15:24.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe89abda700 1 -- 172.21.15.32:0/3647796564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8940ac5e0 con 0x7fe89408c210 2021-04-09T18:15:24.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe8917fa700 1 -- 172.21.15.32:0/3647796564 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8880009b0 con 0x7fe89408c210 2021-04-09T18:15:24.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.562 7fe89abda700 1 -- 172.21.15.32:0/3647796564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8940ac5e0 con 0x7fe89408c210 2021-04-09T18:15:24.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.563 7fe8917fa700 1 -- 172.21.15.32:0/3647796564 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe888006b00 con 0x7fe89408c210 2021-04-09T18:15:24.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.563 7fe8917fa700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe874018930 0x7fe87401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:24.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.563 7fe8917fa700 1 -- 172.21.15.32:0/3647796564 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fe888032a90 con 0x7fe89408c210 2021-04-09T18:15:24.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.563 7fe8989f6700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe874018930 0x7fe87401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:24.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.563 7fe8989f6700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe874018930 0x7fe87401adc0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:24.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.583 7fe89abda700 1 -- 172.21.15.32:0/3647796564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89408cc30 con 0x7fe89408c210 2021-04-09T18:15:24.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.587 7fe8917fa700 1 -- 172.21.15.32:0/3647796564 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe8880009b0 con 0x7fe89408c210 2021-04-09T18:15:24.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.779 7fe89abda700 1 -- 172.21.15.32:0/3647796564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe89408cc30 con 0x7fe89408c210 2021-04-09T18:15:24.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.779 7fe8917fa700 1 -- 172.21.15.32:0/3647796564 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2095 (secure 0 0 0) 0x7fe888006b00 con 0x7fe89408c210 2021-04-09T18:15:24.807 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 37 2021-04-09T18:15:24.808 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:24.808 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/1797590590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1797590590},{"type":"v1","addr":"172.21.15.86:6835","nonce":1797590590}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":34},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":37,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:14:39.918931","modified":"2021-04-09 18:15:24.342216","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-04-09T18:15:24.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 -- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe874018930 msgr2=0x7fe87401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:24.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe874018930 0x7fe87401adc0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 -- 172.21.15.32:0/3647796564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe89408c210 msgr2=0x7fe89408c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:24.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe89408c210 0x7fe89408c610 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fe888006f40 tx=0x7fe88800b990).stop 2021-04-09T18:15:24.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 -- 172.21.15.32:0/3647796564 shutdown_connections 2021-04-09T18:15:24.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe874018930 0x7fe87401adc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe89408c210 0x7fe89408c610 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8940ab9a0 0x7fe8940abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 --2- 172.21.15.32:0/3647796564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8941b85b0 0x7fe8941b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 -- 172.21.15.32:0/3647796564 >> 172.21.15.32:0/3647796564 conn(0x7fe8941864e0 msgr2=0x7fe8941b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 -- 172.21.15.32:0/3647796564 shutdown_connections 2021-04-09T18:15:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.782 7fe89abda700 1 -- 172.21.15.32:0/3647796564 wait complete. 2021-04-09T18:15:24.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:15:25.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.985 7f799b28b700 1 Processor -- start 2021-04-09T18:15:25.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.985 7f799b28b700 1 -- start start 2021-04-09T18:15:25.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f799b28b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940d6c90 0x7f79940d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:25.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f799b28b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79940d2450 0x7f799410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:25.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f799b28b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79940d7290 con 0x7f79940cffb0 2021-04-09T18:15:25.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f799b28b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79940d7090 con 0x7f79940d2450 2021-04-09T18:15:25.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f799b28b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79940d91f0 con 0x7f79940d6c90 2021-04-09T18:15:25.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79998a8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79940d2450 0x7f799410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:25.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79998a8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79940d2450 0x7f799410cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44026/0 (socket says 172.21.15.32:44026) 2021-04-09T18:15:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79988a6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940d6c90 0x7f79940d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:25.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79998a8700 1 -- 172.21.15.32:0/1494279066 learned_addr learned my addr 172.21.15.32:0/1494279066 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:25.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79990a7700 1 --1- 172.21.15.32:0/1494279066 >> v1:172.21.15.86:6789/0 conn(0x7f79940cffb0 0x7f79940d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:51540/0 (socket says 172.21.15.32:51540) 2021-04-09T18:15:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79998a8700 1 -- 172.21.15.32:0/1494279066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940d6c90 msgr2=0x7f79940d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79998a8700 1 --2- 172.21.15.32:0/1494279066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940d6c90 0x7f79940d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79998a8700 1 -- 172.21.15.32:0/1494279066 >> v1:172.21.15.86:6789/0 conn(0x7f79940cffb0 legacy=0x7f79940d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f798bfff700 1 -- 172.21.15.32:0/1494279066 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097235516 0 0) 0x7f7990000c60 con 0x7f79940cffb0 2021-04-09T18:15:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79998a8700 1 -- 172.21.15.32:0/1494279066 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79941120d0 con 0x7f79940d2450 2021-04-09T18:15:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79988a6700 1 --2- 172.21.15.32:0/1494279066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940d6c90 0x7f79940d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f79998a8700 1 --2- 172.21.15.32:0/1494279066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79940d2450 0x7f799410cfe0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f798c006f00 tx=0x7f798c009f10).ready entity=mon.0 client_cookie=a05c480d138e2c70 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f798bfff700 1 -- 172.21.15.32:0/1494279066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f798c00b3b0 con 0x7f79940d2450 2021-04-09T18:15:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f798bfff700 1 -- 172.21.15.32:0/1494279066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c00b730 con 0x7f79940d2450 2021-04-09T18:15:25.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.986 7f798bfff700 1 -- 172.21.15.32:0/1494279066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f798c00bc10 con 0x7f79940d2450 2021-04-09T18:15:25.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.987 7f799b28b700 1 -- 172.21.15.32:0/1494279066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79940d2450 msgr2=0x7f799410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:25.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.987 7f799b28b700 1 --2- 172.21.15.32:0/1494279066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79940d2450 0x7f799410cfe0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f798c006f00 tx=0x7f798c009f10).stop 2021-04-09T18:15:25.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.987 7f799b28b700 1 -- 172.21.15.32:0/1494279066 shutdown_connections 2021-04-09T18:15:25.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.987 7f799b28b700 1 --2- 172.21.15.32:0/1494279066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79940d2450 0x7f799410cfe0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.987 7f799b28b700 1 --2- 172.21.15.32:0/1494279066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940d6c90 0x7f79940d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.987 7f799b28b700 1 -- 172.21.15.32:0/1494279066 >> 172.21.15.32:0/1494279066 conn(0x7f79940ce5d0 msgr2=0x7f79940cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:25.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.987 7f799b28b700 1 -- 172.21.15.32:0/1494279066 shutdown_connections 2021-04-09T18:15:25.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:24.987 7f799b28b700 1 -- 172.21.15.32:0/1494279066 wait complete. 2021-04-09T18:15:25.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.007 7f799b28b700 1 Processor -- start 2021-04-09T18:15:25.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f799b28b700 1 -- start start 2021-04-09T18:15:25.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f799b28b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799408d210 0x7f799408fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:25.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f799b28b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940903f0 0x7f79941a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:25.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f799b28b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79941a6080 0x7f79941a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f799b28b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799408d690 con 0x7f799408d210 2021-04-09T18:15:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f799b28b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79940b0fd0 con 0x7f79941a6080 2021-04-09T18:15:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f799b28b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79941a6480 con 0x7f79940903f0 2021-04-09T18:15:25.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f79990a7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799408d210 0x7f799408fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f79990a7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799408d210 0x7f799408fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44030/0 (socket says 172.21.15.32:44030) 2021-04-09T18:15:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f79990a7700 1 -- 172.21.15.32:0/3346144818 learned_addr learned my addr 172.21.15.32:0/3346144818 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:25.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f79998a8700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79941a6080 0x7f79941a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:25.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.008 7f79988a6700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940903f0 0x7f79941a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:25.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f79990a7700 1 -- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940903f0 msgr2=0x7f79941a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:25.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f79990a7700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940903f0 0x7f79941a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f79990a7700 1 -- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79941a6080 msgr2=0x7f79941a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:25.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f79990a7700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79941a6080 0x7f79941a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f79990a7700 1 -- 172.21.15.32:0/3346144818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799408d8b0 con 0x7f799408d210 2021-04-09T18:15:25.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f79990a7700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799408d210 0x7f799408fdd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f799000a4f0 tx=0x7f799000a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:25.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f7989ffb700 1 -- 172.21.15.32:0/3346144818 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f799000ac80 con 0x7f799408d210 2021-04-09T18:15:25.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f7989ffb700 1 -- 172.21.15.32:0/3346144818 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79900070d0 con 0x7f799408d210 2021-04-09T18:15:25.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f799b28b700 1 -- 172.21.15.32:0/3346144818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799408d8b0 con 0x7f799408d210 2021-04-09T18:15:25.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f7989ffb700 1 -- 172.21.15.32:0/3346144818 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79900047b0 con 0x7f799408d210 2021-04-09T18:15:25.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.009 7f799b28b700 1 -- 172.21.15.32:0/3346144818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799408d8b0 con 0x7f799408d210 2021-04-09T18:15:25.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.010 7f7989ffb700 1 -- 172.21.15.32:0/3346144818 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7990000c10 con 0x7f799408d210 2021-04-09T18:15:25.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.010 7f7989ffb700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7974018930 0x7f797401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:25.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.010 7f7989ffb700 1 -- 172.21.15.32:0/3346144818 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f7990032ab0 con 0x7f799408d210 2021-04-09T18:15:25.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.010 7f79988a6700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7974018930 0x7f797401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:25.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.010 7f79988a6700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7974018930 0x7f797401ae10 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:25.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.030 7f799b28b700 1 -- 172.21.15.32:0/3346144818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79941a52b0 con 0x7f799408d210 2021-04-09T18:15:25.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.033 7f7989ffb700 1 -- 172.21.15.32:0/3346144818 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f79900070d0 con 0x7f799408d210 2021-04-09T18:15:25.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.227 7f799b28b700 1 -- 172.21.15.32:0/3346144818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f79941a52b0 con 0x7f799408d210 2021-04-09T18:15:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.701 7f7989ffb700 1 -- 172.21.15.32:0/3346144818 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7990000c10 con 0x7f799408d210 2021-04-09T18:15:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 -- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7974018930 msgr2=0x7f797401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7974018930 0x7f797401ae10 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 -- 172.21.15.32:0/3346144818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799408d210 msgr2=0x7f799408fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:25.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799408d210 0x7f799408fdd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f799000a4f0 tx=0x7f799000a890).stop 2021-04-09T18:15:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 -- 172.21.15.32:0/3346144818 shutdown_connections 2021-04-09T18:15:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7974018930 0x7f797401ae10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79941a6080 0x7f79941a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79940903f0 0x7f79941a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 --2- 172.21.15.32:0/3346144818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799408d210 0x7f799408fdd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 -- 172.21.15.32:0/3346144818 >> 172.21.15.32:0/3346144818 conn(0x7f79941a8f50 msgr2=0x7f7994188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:25.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 -- 172.21.15.32:0/3346144818 shutdown_connections 2021-04-09T18:15:25.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.744 7f799b28b700 1 -- 172.21.15.32:0/3346144818 wait complete. 2021-04-09T18:15:25.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:15:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.915 7fe0de4e2700 1 Processor -- start 2021-04-09T18:15:25.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.915 7fe0de4e2700 1 -- start start 2021-04-09T18:15:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0de4e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d810d050 0x7fe0d80cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:25.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0de4e2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe0d80d0160 0x7fe0d80d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0de4e2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0d80d82e0 0x7fe0d810da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:25.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0de4e2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d80d8d70 con 0x7fe0d80d82e0 2021-04-09T18:15:25.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0de4e2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d80d9260 con 0x7fe0d80d0160 2021-04-09T18:15:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0de4e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d810d500 con 0x7fe0d810d050 2021-04-09T18:15:25.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0dcaff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0d80d82e0 0x7fe0d810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0dcaff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0d80d82e0 0x7fe0d810da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44040/0 (socket says 172.21.15.32:44040) 2021-04-09T18:15:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0dcaff700 1 -- 172.21.15.32:0/3598299905 learned_addr learned my addr 172.21.15.32:0/3598299905 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:25.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0cffff700 1 --2- 172.21.15.32:0/3598299905 >> v2:172.21.15.86:3300/0 conn(0x7fe0d80d0160 0x7fe0d80d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0d7fff700 1 --2- 172.21.15.32:0/3598299905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d810d050 0x7fe0d80cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:25.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0dcaff700 1 -- 172.21.15.32:0/3598299905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d810d050 msgr2=0x7fe0d80cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:25.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0dcaff700 1 --2- 172.21.15.32:0/3598299905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d810d050 0x7fe0d80cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0dcaff700 1 -- 172.21.15.32:0/3598299905 >> v2:172.21.15.86:3300/0 conn(0x7fe0d80d0160 msgr2=0x7fe0d80d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:25.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0dcaff700 1 --2- 172.21.15.32:0/3598299905 >> v2:172.21.15.86:3300/0 conn(0x7fe0d80d0160 0x7fe0d80d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0dcaff700 1 -- 172.21.15.32:0/3598299905 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d80d86e0 con 0x7fe0d80d82e0 2021-04-09T18:15:25.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0dcaff700 1 --2- 172.21.15.32:0/3598299905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d80d82e0 0x7fe0d810da80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fe0d0006f00 tx=0x7fe0d0009e90).ready entity=mon.0 client_cookie=cc8aa6f713880ab2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:25.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.916 7fe0d77fe700 1 -- 172.21.15.32:0/3598299905 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0d000b300 con 0x7fe0d80d82e0 2021-04-09T18:15:25.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0d77fe700 1 -- 172.21.15.32:0/3598299905 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d000b640 con 0x7fe0d80d82e0 2021-04-09T18:15:25.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0d77fe700 1 -- 172.21.15.32:0/3598299905 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0d000bb20 con 0x7fe0d80d82e0 2021-04-09T18:15:26.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0de4e2700 1 -- 172.21.15.32:0/3598299905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d80d82e0 msgr2=0x7fe0d810da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:26.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0de4e2700 1 --2- 172.21.15.32:0/3598299905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d80d82e0 0x7fe0d810da80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fe0d0006f00 tx=0x7fe0d0009e90).stop 2021-04-09T18:15:26.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0de4e2700 1 -- 172.21.15.32:0/3598299905 shutdown_connections 2021-04-09T18:15:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0de4e2700 1 --2- 172.21.15.32:0/3598299905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d80d82e0 0x7fe0d810da80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0de4e2700 1 --2- 172.21.15.32:0/3598299905 >> v2:172.21.15.86:3300/0 conn(0x7fe0d80d0160 0x7fe0d80d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0de4e2700 1 --2- 172.21.15.32:0/3598299905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d810d050 0x7fe0d80cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0de4e2700 1 -- 172.21.15.32:0/3598299905 >> 172.21.15.32:0/3598299905 conn(0x7fe0d80ce620 msgr2=0x7fe0d80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:26.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0de4e2700 1 -- 172.21.15.32:0/3598299905 shutdown_connections 2021-04-09T18:15:26.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.917 7fe0de4e2700 1 -- 172.21.15.32:0/3598299905 wait complete. 2021-04-09T18:15:26.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0de4e2700 1 Processor -- start 2021-04-09T18:15:26.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0de4e2700 1 -- start start 2021-04-09T18:15:26.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0de4e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d80ad5b0 0x7fe0d81926b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:26.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0de4e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d80990f0 0x7fe0d80994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:26.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0de4e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d81a30b0 0x7fe0d81a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0de4e2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8098b70 con 0x7fe0d81a30b0 2021-04-09T18:15:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0de4e2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d80988a0 con 0x7fe0d80990f0 2021-04-09T18:15:26.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0de4e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8099b10 con 0x7fe0d80ad5b0 2021-04-09T18:15:26.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0dcaff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d81a30b0 0x7fe0d81a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:26.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0dcaff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d81a30b0 0x7fe0d81a34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44046/0 (socket says 172.21.15.32:44046) 2021-04-09T18:15:26.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0dcaff700 1 -- 172.21.15.32:0/1666044430 learned_addr learned my addr 172.21.15.32:0/1666044430 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:26.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0cffff700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d80990f0 0x7fe0d80994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.938 7fe0d7fff700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d80ad5b0 0x7fe0d81926b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0dcaff700 1 -- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d80ad5b0 msgr2=0x7fe0d81926b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0dcaff700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d80ad5b0 0x7fe0d81926b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0dcaff700 1 -- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d80990f0 msgr2=0x7fe0d80994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0dcaff700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d80990f0 0x7fe0d80994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0dcaff700 1 -- 172.21.15.32:0/1666044430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d81a3ad0 con 0x7fe0d81a30b0 2021-04-09T18:15:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0cffff700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d80990f0 0x7fe0d80994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0dcaff700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d81a30b0 0x7fe0d81a34b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fe0d000b9c0 tx=0x7fe0d000bd60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0d57fa700 1 -- 172.21.15.32:0/1666044430 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0d0005820 con 0x7fe0d81a30b0 2021-04-09T18:15:26.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0d57fa700 1 -- 172.21.15.32:0/1666044430 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d0005b20 con 0x7fe0d81a30b0 2021-04-09T18:15:26.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d81a3ad0 con 0x7fe0d81a30b0 2021-04-09T18:15:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0d57fa700 1 -- 172.21.15.32:0/1666044430 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0d0006030 con 0x7fe0d81a30b0 2021-04-09T18:15:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.939 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d81a3ad0 con 0x7fe0d81a30b0 2021-04-09T18:15:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.940 7fe0d57fa700 1 -- 172.21.15.32:0/1666044430 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe0d0005d00 con 0x7fe0d81a30b0 2021-04-09T18:15:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.940 7fe0d57fa700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0bc018930 0x7fe0bc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.940 7fe0d57fa700 1 -- 172.21.15.32:0/1666044430 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fe0d0032e50 con 0x7fe0d81a30b0 2021-04-09T18:15:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.940 7fe0d7fff700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0bc018930 0x7fe0bc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.940 7fe0d7fff700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0bc018930 0x7fe0bc01ae10 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:26.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.960 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d80b9040 con 0x7fe0d81a30b0 2021-04-09T18:15:26.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:25.963 7fe0d57fa700 1 -- 172.21.15.32:0/1666044430 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe0d0000f90 con 0x7fe0d81a30b0 2021-04-09T18:15:26.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.146 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe0d810d500 con 0x7fe0d81a30b0 2021-04-09T18:15:26.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.707 7fe0d57fa700 1 -- 172.21.15.32:0/1666044430 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v45) v1 ==== 185+0+0 (secure 0 0 0) 0x7fe0d0005ce0 con 0x7fe0d81a30b0 2021-04-09T18:15:26.737 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:15:26.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0bc018930 msgr2=0x7fe0bc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:26.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0bc018930 0x7fe0bc01ae10 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d81a30b0 msgr2=0x7fe0d81a34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:26.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d81a30b0 0x7fe0d81a34b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fe0d000b9c0 tx=0x7fe0d000bd60).stop 2021-04-09T18:15:26.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 shutdown_connections 2021-04-09T18:15:26.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0bc018930 0x7fe0bc01ae10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d81a30b0 0x7fe0d81a34b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d80990f0 0x7fe0d80994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 --2- 172.21.15.32:0/1666044430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d80ad5b0 0x7fe0d81926b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 >> 172.21.15.32:0/1666044430 conn(0x7fe0d80ae6f0 msgr2=0x7fe0d8097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:26.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 shutdown_connections 2021-04-09T18:15:26.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.713 7fe0de4e2700 1 -- 172.21.15.32:0/1666044430 wait complete. 2021-04-09T18:15:26.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:15:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.894 7fbbe7376700 1 Processor -- start 2021-04-09T18:15:26.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.895 7fbbe7376700 1 -- start start 2021-04-09T18:15:26.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.895 7fbbe7376700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbbe010d050 0x7fbbe00cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:26.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.895 7fbbe7376700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00d0160 0x7fbbe00d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:26.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.895 7fbbe7376700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe010d500 con 0x7fbbe00d82e0 2021-04-09T18:15:26.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.895 7fbbe7376700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe00d9260 con 0x7fbbe010d050 2021-04-09T18:15:26.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.895 7fbbe7376700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00d86e0 con 0x7fbbe00d0160 2021-04-09T18:15:26.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe4991700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00d0160 0x7fbbe00d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:26.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe5192700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbbe010d050 0x7fbbe00cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:26.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe5192700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbbe010d050 0x7fbbe00cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:35808/0 (socket says 172.21.15.32:35808) 2021-04-09T18:15:26.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe4991700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00d0160 0x7fbbe00d7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:51278/0 (socket says 172.21.15.32:51278) 2021-04-09T18:15:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe5993700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbbe00d82e0 0x7fbbe010da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:51568/0 (socket says 172.21.15.32:51568) 2021-04-09T18:15:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe5192700 1 -- 172.21.15.32:0/3967159569 learned_addr learned my addr 172.21.15.32:0/3967159569 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:26.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe4991700 1 -- 172.21.15.32:0/3967159569 >> v2:172.21.15.86:3300/0 conn(0x7fbbe010d050 msgr2=0x7fbbe00cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:26.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe4991700 1 --2- 172.21.15.32:0/3967159569 >> v2:172.21.15.86:3300/0 conn(0x7fbbe010d050 0x7fbbe00cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:26.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe4991700 1 -- 172.21.15.32:0/3967159569 >> v1:172.21.15.86:6789/0 conn(0x7fbbe00d82e0 legacy=0x7fbbe010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:26.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe4991700 1 -- 172.21.15.32:0/3967159569 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe010e270 con 0x7fbbe00d0160 2021-04-09T18:15:26.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbd7fff700 1 -- 172.21.15.32:0/3967159569 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522443068 0 0) 0x7fbbd8000d30 con 0x7fbbe00d82e0 2021-04-09T18:15:26.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe5192700 1 --2- 172.21.15.32:0/3967159569 >> v2:172.21.15.86:3300/0 conn(0x7fbbe010d050 0x7fbbe00cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:26.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbe4991700 1 --2- 172.21.15.32:0/3967159569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00d0160 0x7fbbe00d7cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbbd0006650 tx=0x7fbbd0009a80).ready entity=mon.2 client_cookie=3f2d6c12816f78f3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:26.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbd7fff700 1 -- 172.21.15.32:0/3967159569 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbd00009b0 con 0x7fbbe00d0160 2021-04-09T18:15:26.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbd7fff700 1 -- 172.21.15.32:0/3967159569 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd000c0c0 con 0x7fbbe00d0160 2021-04-09T18:15:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.896 7fbbd7fff700 1 -- 172.21.15.32:0/3967159569 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbd000c510 con 0x7fbbe00d0160 2021-04-09T18:15:26.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.897 7fbbe7376700 1 -- 172.21.15.32:0/3967159569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00d0160 msgr2=0x7fbbe00d7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:27.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.897 7fbbe7376700 1 --2- 172.21.15.32:0/3967159569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00d0160 0x7fbbe00d7cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbbd0006650 tx=0x7fbbd0009a80).stop 2021-04-09T18:15:27.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.897 7fbbe7376700 1 -- 172.21.15.32:0/3967159569 shutdown_connections 2021-04-09T18:15:27.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.897 7fbbe7376700 1 --2- 172.21.15.32:0/3967159569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00d0160 0x7fbbe00d7cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.897 7fbbe7376700 1 --2- 172.21.15.32:0/3967159569 >> v2:172.21.15.86:3300/0 conn(0x7fbbe010d050 0x7fbbe00cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.897 7fbbe7376700 1 -- 172.21.15.32:0/3967159569 >> 172.21.15.32:0/3967159569 conn(0x7fbbe00ce620 msgr2=0x7fbbe00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:27.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.897 7fbbe7376700 1 -- 172.21.15.32:0/3967159569 shutdown_connections 2021-04-09T18:15:27.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.897 7fbbe7376700 1 -- 172.21.15.32:0/3967159569 wait complete. 2021-04-09T18:15:27.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.917 7fbbe7376700 1 Processor -- start 2021-04-09T18:15:27.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.917 7fbbe7376700 1 -- start start 2021-04-09T18:15:27.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe7376700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00b84d0 0x7fbbe00c2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:27.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe7376700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbe00c2e20 0x7fbbe00a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:27.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe7376700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbe00a1b60 0x7fbbe00a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:27.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe7376700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe018e9f0 con 0x7fbbe00c2e20 2021-04-09T18:15:27.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe7376700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00a8420 con 0x7fbbe00a1b60 2021-04-09T18:15:27.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe7376700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00b3d90 con 0x7fbbe00b84d0 2021-04-09T18:15:27.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe4991700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbe00c2e20 0x7fbbe00a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:27.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe4991700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbe00c2e20 0x7fbbe00a1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44056/0 (socket says 172.21.15.32:44056) 2021-04-09T18:15:27.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe5993700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbe00a1b60 0x7fbbe00a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:27.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe4991700 1 -- 172.21.15.32:0/2194074619 learned_addr learned my addr 172.21.15.32:0/2194074619 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:27.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.918 7fbbe5192700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00b84d0 0x7fbbe00c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:27.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe5993700 1 -- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00b84d0 msgr2=0x7fbbe00c2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe5993700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00b84d0 0x7fbbe00c2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe5993700 1 -- 172.21.15.32:0/2194074619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbe00c2e20 msgr2=0x7fbbe00a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:27.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe5993700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbe00c2e20 0x7fbbe00a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe5993700 1 -- 172.21.15.32:0/2194074619 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe00a7cb0 con 0x7fbbe00a1b60 2021-04-09T18:15:27.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe4991700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbe00c2e20 0x7fbbe00a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe5192700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00b84d0 0x7fbbe00c2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe5993700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbe00a1b60 0x7fbbe00a7690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fbbd8001b10 tx=0x7fbbd800aba0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:27.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbd5ffb700 1 -- 172.21.15.32:0/2194074619 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbd80070d0 con 0x7fbbe00a1b60 2021-04-09T18:15:27.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbd5ffb700 1 -- 172.21.15.32:0/2194074619 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd80073d0 con 0x7fbbe00a1b60 2021-04-09T18:15:27.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbd5ffb700 1 -- 172.21.15.32:0/2194074619 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbd8004920 con 0x7fbbe00a1b60 2021-04-09T18:15:27.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe00a7cb0 con 0x7fbbe00a1b60 2021-04-09T18:15:27.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.919 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe00a7cb0 con 0x7fbbe00a1b60 2021-04-09T18:15:27.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.920 7fbbd5ffb700 1 -- 172.21.15.32:0/2194074619 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbbd8008000 con 0x7fbbe00a1b60 2021-04-09T18:15:27.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.920 7fbbd5ffb700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbbc00188c0 0x7fbbc001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:27.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.920 7fbbd5ffb700 1 -- 172.21.15.32:0/2194074619 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5311+0+0 (secure 0 0 0) 0x7fbbd8032930 con 0x7fbbe00a1b60 2021-04-09T18:15:27.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.920 7fbbe5192700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbbc00188c0 0x7fbbc001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:27.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.920 7fbbe5192700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbbc00188c0 0x7fbbc001adc0 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:27.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.939 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe01b7180 con 0x7fbbe00a1b60 2021-04-09T18:15:27.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:26.943 7fbbd5ffb700 1 -- 172.21.15.32:0/2194074619 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbbd8000cc0 con 0x7fbbe00a1b60 2021-04-09T18:15:27.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.124 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbbe010d500 con 0x7fbbe00a1b60 2021-04-09T18:15:27.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.716 7fbbd5ffb700 1 -- 172.21.15.32:0/2194074619 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v46) v1 ==== 173+0+0 (secure 0 0 0) 0x7fbbd8008000 con 0x7fbbe00a1b60 2021-04-09T18:15:27.766 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:15:27.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.741 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbbc00188c0 msgr2=0x7fbbc001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:27.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.741 7fbbe7376700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbbc00188c0 0x7fbbc001adc0 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.741 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbe00a1b60 msgr2=0x7fbbe00a7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:27.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.741 7fbbe7376700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbe00a1b60 0x7fbbe00a7690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fbbd8001b10 tx=0x7fbbd800aba0).stop 2021-04-09T18:15:27.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.742 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 shutdown_connections 2021-04-09T18:15:27.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.742 7fbbe7376700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbbc00188c0 0x7fbbc001adc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.742 7fbbe7376700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbe00c2e20 0x7fbbe00a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.742 7fbbe7376700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbe00a1b60 0x7fbbe00a7690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.742 7fbbe7376700 1 --2- 172.21.15.32:0/2194074619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbe00b84d0 0x7fbbe00c2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.742 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 >> 172.21.15.32:0/2194074619 conn(0x7fbbe00b6b70 msgr2=0x7fbbe00b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.742 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 shutdown_connections 2021-04-09T18:15:27.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.742 7fbbe7376700 1 -- 172.21.15.32:0/2194074619 wait complete. 2021-04-09T18:15:27.780 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:15:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.917 7f86de018700 1 Processor -- start 2021-04-09T18:15:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.917 7f86de018700 1 -- start start 2021-04-09T18:15:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.917 7f86de018700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86d80d6c90 0x7f86d80d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.917 7f86de018700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f86d80d2450 0x7f86d810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.917 7f86de018700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86d80d7290 con 0x7f86d80cffb0 2021-04-09T18:15:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.917 7f86de018700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d80d7090 con 0x7f86d80d2450 2021-04-09T18:15:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.917 7f86de018700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86d80d91f0 con 0x7f86d80d6c90 2021-04-09T18:15:27.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.917 7f86d7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f86d80d2450 0x7f86d810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:27.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.917 7f86d77fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f86d80cffb0 0x7f86d80d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53654/0 (socket says 172.21.15.32:53654) 2021-04-09T18:15:27.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d77fe700 1 -- 172.21.15.32:0/931569831 learned_addr learned my addr 172.21.15.32:0/931569831 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86ceffd700 1 --2- 172.21.15.32:0/931569831 >> v2:172.21.15.86:3300/0 conn(0x7f86d80d6c90 0x7f86d80d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d6ffd700 1 -- 172.21.15.32:0/931569831 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810530652 0 0) 0x7f86c8000ce0 con 0x7f86d80cffb0 2021-04-09T18:15:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d6ffd700 1 -- 172.21.15.32:0/931569831 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c4002fe0 con 0x7f86d80cffb0 2021-04-09T18:15:27.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d7fff700 1 -- 172.21.15.32:0/931569831 >> v2:172.21.15.86:3300/0 conn(0x7f86d80d6c90 msgr2=0x7f86d80d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:27.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d7fff700 1 --2- 172.21.15.32:0/931569831 >> v2:172.21.15.86:3300/0 conn(0x7f86d80d6c90 0x7f86d80d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d7fff700 1 -- 172.21.15.32:0/931569831 >> v1:172.21.15.32:6789/0 conn(0x7f86d80cffb0 legacy=0x7f86d80d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:27.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d7fff700 1 -- 172.21.15.32:0/931569831 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d80d2850 con 0x7f86d80d2450 2021-04-09T18:15:27.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d7fff700 1 --2- 172.21.15.32:0/931569831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80d2450 0x7f86d810cfe0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f86d0006e30 tx=0x7f86d0009e60).ready entity=mon.0 client_cookie=3abd44c8d4e97ca7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:27.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d6ffd700 1 -- 172.21.15.32:0/931569831 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86d0006a50 con 0x7f86d80d2450 2021-04-09T18:15:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d6ffd700 1 -- 172.21.15.32:0/931569831 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d000b300 con 0x7f86d80d2450 2021-04-09T18:15:27.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86d6ffd700 1 -- 172.21.15.32:0/931569831 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86d000b750 con 0x7f86d80d2450 2021-04-09T18:15:27.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86de018700 1 -- 172.21.15.32:0/931569831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80d2450 msgr2=0x7f86d810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:27.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.918 7f86de018700 1 --2- 172.21.15.32:0/931569831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80d2450 0x7f86d810cfe0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f86d0006e30 tx=0x7f86d0009e60).stop 2021-04-09T18:15:27.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.919 7f86de018700 1 -- 172.21.15.32:0/931569831 shutdown_connections 2021-04-09T18:15:27.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.919 7f86de018700 1 --2- 172.21.15.32:0/931569831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80d2450 0x7f86d810cfe0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.919 7f86de018700 1 --2- 172.21.15.32:0/931569831 >> v2:172.21.15.86:3300/0 conn(0x7f86d80d6c90 0x7f86d80d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:27.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.919 7f86de018700 1 -- 172.21.15.32:0/931569831 >> 172.21.15.32:0/931569831 conn(0x7f86d80ce5d0 msgr2=0x7f86d80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:27.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.919 7f86de018700 1 -- 172.21.15.32:0/931569831 shutdown_connections 2021-04-09T18:15:27.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.919 7f86de018700 1 -- 172.21.15.32:0/931569831 wait complete. 2021-04-09T18:15:27.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.939 7f86de018700 1 Processor -- start 2021-04-09T18:15:27.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.939 7f86de018700 1 -- start start 2021-04-09T18:15:28.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86de018700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d808d210 0x7f86d808fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86de018700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80903f0 0x7f86d81a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86de018700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86d81a6080 0x7f86d81a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86de018700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d808d690 con 0x7f86d80903f0 2021-04-09T18:15:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86de018700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d80b0fd0 con 0x7f86d808d210 2021-04-09T18:15:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86de018700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86d81a6480 con 0x7f86d81a6080 2021-04-09T18:15:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86ceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80903f0 0x7f86d81a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86ceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80903f0 0x7f86d81a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44070/0 (socket says 172.21.15.32:44070) 2021-04-09T18:15:28.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86ceffd700 1 -- 172.21.15.32:0/71106809 learned_addr learned my addr 172.21.15.32:0/71106809 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:28.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86d77fe700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d808d210 0x7f86d808fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86d7fff700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86d81a6080 0x7f86d81a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86ceffd700 1 -- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86d81a6080 msgr2=0x7f86d81a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:28.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86ceffd700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86d81a6080 0x7f86d81a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86ceffd700 1 -- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d808d210 msgr2=0x7f86d808fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:28.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86ceffd700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d808d210 0x7f86d808fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86ceffd700 1 -- 172.21.15.32:0/71106809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d808d8b0 con 0x7f86d80903f0 2021-04-09T18:15:28.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.940 7f86ceffd700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80903f0 0x7f86d81a5a60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f86c0006f60 tx=0x7f86c000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:28.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.941 7f86d4ff9700 1 -- 172.21.15.32:0/71106809 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86c000bd60 con 0x7f86d80903f0 2021-04-09T18:15:28.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.941 7f86d4ff9700 1 -- 172.21.15.32:0/71106809 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c0000c90 con 0x7f86d80903f0 2021-04-09T18:15:28.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.941 7f86de018700 1 -- 172.21.15.32:0/71106809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d808d8b0 con 0x7f86d80903f0 2021-04-09T18:15:28.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.941 7f86d4ff9700 1 -- 172.21.15.32:0/71106809 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86c0002220 con 0x7f86d80903f0 2021-04-09T18:15:28.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.941 7f86de018700 1 -- 172.21.15.32:0/71106809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d808d8b0 con 0x7f86d80903f0 2021-04-09T18:15:28.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.941 7f86d4ff9700 1 -- 172.21.15.32:0/71106809 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f86c00052e0 con 0x7f86d80903f0 2021-04-09T18:15:28.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.941 7f86d4ff9700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86bc018930 0x7f86bc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.941 7f86d4ff9700 1 -- 172.21.15.32:0/71106809 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f86c0032750 con 0x7f86d80903f0 2021-04-09T18:15:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.942 7f86d77fe700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86bc018930 0x7f86bc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.942 7f86d77fe700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86bc018930 0x7f86bc01adc0 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.961 7f86de018700 1 -- 172.21.15.32:0/71106809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d81a52b0 con 0x7f86d80903f0 2021-04-09T18:15:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:27.965 7f86d4ff9700 1 -- 172.21.15.32:0/71106809 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f86c0000c90 con 0x7f86d80903f0 2021-04-09T18:15:28.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.174 7f86de018700 1 -- 172.21.15.32:0/71106809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f86d81a52b0 con 0x7f86d80903f0 2021-04-09T18:15:28.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.177 7f86d4ff9700 1 -- 172.21.15.32:0/71106809 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7f86c00052e0 con 0x7f86d80903f0 2021-04-09T18:15:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 -- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86bc018930 msgr2=0x7f86bc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86bc018930 0x7f86bc01adc0 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 -- 172.21.15.32:0/71106809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80903f0 msgr2=0x7f86d81a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80903f0 0x7f86d81a5a60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f86c0006f60 tx=0x7f86c000b970).stop 2021-04-09T18:15:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 -- 172.21.15.32:0/71106809 shutdown_connections 2021-04-09T18:15:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86bc018930 0x7f86bc01adc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86d80903f0 0x7f86d81a5a60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86d81a6080 0x7f86d81a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 --2- 172.21.15.32:0/71106809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86d808d210 0x7f86d808fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.183 7f86de018700 1 -- 172.21.15.32:0/71106809 >> 172.21.15.32:0/71106809 conn(0x7f86d81a8f50 msgr2=0x7f86d8188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.184 7f86de018700 1 -- 172.21.15.32:0/71106809 shutdown_connections 2021-04-09T18:15:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.184 7f86de018700 1 -- 172.21.15.32:0/71106809 wait complete. 2021-04-09T18:15:28.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2021-04-09T18:15:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.492 7faab953a700 1 Processor -- start 2021-04-09T18:15:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.492 7faab953a700 1 -- start start 2021-04-09T18:15:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.493 7faab953a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faab410d010 0x7faab40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.493 7faab953a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40d8490 0x7faab410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.493 7faab953a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab410d480 con 0x7faab40d1da0 2021-04-09T18:15:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.493 7faab953a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab40d8d30 con 0x7faab410d010 2021-04-09T18:15:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.493 7faab953a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab40d9220 con 0x7faab40d8490 2021-04-09T18:15:28.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.493 7faab32bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faab410d010 0x7faab40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.493 7faab32bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faab410d010 0x7faab40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44078/0 (socket says 172.21.15.32:44078) 2021-04-09T18:15:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.493 7faab32bf700 1 -- 172.21.15.32:0/3994716193 learned_addr learned my addr 172.21.15.32:0/3994716193 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:28.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.493 7faab2abe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faab40d1da0 0x7faab40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53672/0 (socket says 172.21.15.32:53672) 2021-04-09T18:15:28.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab3ac0700 1 --2- 172.21.15.32:0/3994716193 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40d8490 0x7faab410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab32bf700 1 -- 172.21.15.32:0/3994716193 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40d8490 msgr2=0x7faab410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:28.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab32bf700 1 --2- 172.21.15.32:0/3994716193 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40d8490 0x7faab410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab32bf700 1 -- 172.21.15.32:0/3994716193 >> v1:172.21.15.32:6789/0 conn(0x7faab40d1da0 legacy=0x7faab40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:28.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab22bd700 1 -- 172.21.15.32:0/3994716193 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567442025 0 0) 0x7faaa8000ce0 con 0x7faab40d1da0 2021-04-09T18:15:28.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab32bf700 1 -- 172.21.15.32:0/3994716193 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab410e020 con 0x7faab410d010 2021-04-09T18:15:28.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab32bf700 1 --2- 172.21.15.32:0/3994716193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab410d010 0x7faab40d1780 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7faaa4006700 tx=0x7faaa4009b00).ready entity=mon.0 client_cookie=efa49ab380c25a56 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:28.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab22bd700 1 -- 172.21.15.32:0/3994716193 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa400c0c0 con 0x7faab410d010 2021-04-09T18:15:28.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab22bd700 1 -- 172.21.15.32:0/3994716193 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa400c400 con 0x7faab410d010 2021-04-09T18:15:28.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab22bd700 1 -- 172.21.15.32:0/3994716193 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa400c8e0 con 0x7faab410d010 2021-04-09T18:15:28.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab953a700 1 -- 172.21.15.32:0/3994716193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab410d010 msgr2=0x7faab40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:28.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.494 7faab953a700 1 --2- 172.21.15.32:0/3994716193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab410d010 0x7faab40d1780 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7faaa4006700 tx=0x7faaa4009b00).stop 2021-04-09T18:15:28.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.495 7faab953a700 1 -- 172.21.15.32:0/3994716193 shutdown_connections 2021-04-09T18:15:28.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.495 7faab953a700 1 --2- 172.21.15.32:0/3994716193 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40d8490 0x7faab410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.495 7faab953a700 1 --2- 172.21.15.32:0/3994716193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab410d010 0x7faab40d1780 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.495 7faab953a700 1 -- 172.21.15.32:0/3994716193 >> 172.21.15.32:0/3994716193 conn(0x7faab40ce5f0 msgr2=0x7faab40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.495 7faab953a700 1 -- 172.21.15.32:0/3994716193 shutdown_connections 2021-04-09T18:15:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.495 7faab953a700 1 -- 172.21.15.32:0/3994716193 wait complete. 2021-04-09T18:15:28.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.515 7faab953a700 1 Processor -- start 2021-04-09T18:15:28.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.515 7faab953a700 1 -- start start 2021-04-09T18:15:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab953a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40b5360 0x7faab41a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab953a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab418e140 0x7faab41aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab953a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab41ad040 0x7faab408fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab953a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab41a5360 con 0x7faab418e140 2021-04-09T18:15:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab953a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab40902c0 con 0x7faab41ad040 2021-04-09T18:15:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab953a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab40904e0 con 0x7faab40b5360 2021-04-09T18:15:28.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab2abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab418e140 0x7faab41aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab2abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab418e140 0x7faab41aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44086/0 (socket says 172.21.15.32:44086) 2021-04-09T18:15:28.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab2abe700 1 -- 172.21.15.32:0/1622212808 learned_addr learned my addr 172.21.15.32:0/1622212808 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:28.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab32bf700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40b5360 0x7faab41a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab3ac0700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab41ad040 0x7faab408fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab2abe700 1 -- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40b5360 msgr2=0x7faab41a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:28.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab2abe700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40b5360 0x7faab41a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab2abe700 1 -- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab41ad040 msgr2=0x7faab408fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:28.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab2abe700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab41ad040 0x7faab408fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:28.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab2abe700 1 -- 172.21.15.32:0/1622212808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab4183c80 con 0x7faab418e140 2021-04-09T18:15:28.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab2abe700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab418e140 0x7faab41aca20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7faaa800b560 tx=0x7faaa800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:28.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faa9bfff700 1 -- 172.21.15.32:0/1622212808 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa8006030 con 0x7faab418e140 2021-04-09T18:15:28.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faa9bfff700 1 -- 172.21.15.32:0/1622212808 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa80043a0 con 0x7faab418e140 2021-04-09T18:15:28.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab953a700 1 -- 172.21.15.32:0/1622212808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab4183c80 con 0x7faab418e140 2021-04-09T18:15:28.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faa9bfff700 1 -- 172.21.15.32:0/1622212808 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa8004880 con 0x7faab418e140 2021-04-09T18:15:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.516 7faab953a700 1 -- 172.21.15.32:0/1622212808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab4183c80 con 0x7faab418e140 2021-04-09T18:15:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.517 7faa9bfff700 1 -- 172.21.15.32:0/1622212808 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faaa8004580 con 0x7faab418e140 2021-04-09T18:15:28.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.517 7faa9bfff700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faa94018930 0x7faa9401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:28.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.517 7faa9bfff700 1 -- 172.21.15.32:0/1622212808 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4929+0+0 (secure 0 0 0) 0x7faaa8032730 con 0x7faab418e140 2021-04-09T18:15:28.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.517 7faab32bf700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faa94018930 0x7faa9401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:28.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.518 7faab32bf700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faa94018930 0x7faa9401adc0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:28.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.537 7faab953a700 1 -- 172.21.15.32:0/1622212808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab4182d70 con 0x7faab418e140 2021-04-09T18:15:28.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.541 7faa9bfff700 1 -- 172.21.15.32:0/1622212808 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faaa8006b80 con 0x7faab418e140 2021-04-09T18:15:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:28.729 7faab953a700 1 -- 172.21.15.32:0/1622212808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7faab410d480 con 0x7faab418e140 2021-04-09T18:15:29.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.716 7faa9bfff700 1 -- 172.21.15.32:0/1622212808 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v146) v1 ==== 138+0+0 (secure 0 0 0) 0x7faaa8006150 con 0x7faab418e140 2021-04-09T18:15:29.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 -- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faa94018930 msgr2=0x7faa9401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:29.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faa94018930 0x7faa9401adc0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:29.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 -- 172.21.15.32:0/1622212808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab418e140 msgr2=0x7faab41aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:29.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab418e140 0x7faab41aca20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7faaa800b560 tx=0x7faaa800b900).stop 2021-04-09T18:15:29.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 -- 172.21.15.32:0/1622212808 shutdown_connections 2021-04-09T18:15:29.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faa94018930 0x7faa9401adc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:29.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faab418e140 0x7faab41aca20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:29.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab41ad040 0x7faab408fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:29.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 --2- 172.21.15.32:0/1622212808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faab40b5360 0x7faab41a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:29.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 -- 172.21.15.32:0/1622212808 >> 172.21.15.32:0/1622212808 conn(0x7faab419b460 msgr2=0x7faab419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:29.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 -- 172.21.15.32:0/1622212808 shutdown_connections 2021-04-09T18:15:29.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.748 7faab953a700 1 -- 172.21.15.32:0/1622212808 wait complete. 2021-04-09T18:15:29.812 INFO:tasks.cephfs_test_runner:test_down_grow (tasks.cephfs.test_failover.TestClusterResize) ... ok 2021-04-09T18:15:29.813 INFO:tasks.cephfs_test_runner:Starting test: test_down_health (tasks.cephfs.test_failover.TestClusterResize) 2021-04-09T18:15:29.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2021-04-09T18:15:30.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.955 7fb640a39700 1 Processor -- start 2021-04-09T18:15:30.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.955 7fb640a39700 1 -- start start 2021-04-09T18:15:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb640a39700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb63c0cffa0 0x7fb63c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb640a39700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb63c0d7800 0x7fb63c0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb640a39700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63c0d7280 con 0x7fb63c0d8220 2021-04-09T18:15:30.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb640a39700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb63c0d91e0 con 0x7fb63c0cffa0 2021-04-09T18:15:30.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb640a39700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb63c10dfe0 con 0x7fb63c0d7800 2021-04-09T18:15:30.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb63a61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb63c0cffa0 0x7fb63c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:30.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb63a61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb63c0cffa0 0x7fb63c0d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44096/0 (socket says 172.21.15.32:44096) 2021-04-09T18:15:30.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb63a61c700 1 -- 172.21.15.32:0/2132095715 learned_addr learned my addr 172.21.15.32:0/2132095715 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:30.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb639e1b700 1 --2- 172.21.15.32:0/2132095715 >> v2:172.21.15.86:3300/0 conn(0x7fb63c0d7800 0x7fb63c0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:30.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb63a61c700 1 -- 172.21.15.32:0/2132095715 >> v2:172.21.15.86:3300/0 conn(0x7fb63c0d7800 msgr2=0x7fb63c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:30.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb63a61c700 1 --2- 172.21.15.32:0/2132095715 >> v2:172.21.15.86:3300/0 conn(0x7fb63c0d7800 0x7fb63c0d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb63a61c700 1 -- 172.21.15.32:0/2132095715 >> v1:172.21.15.86:6789/0 conn(0x7fb63c0d8220 legacy=0x7fb63c0d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.956 7fb63a61c700 1 -- 172.21.15.32:0/2132095715 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c10e200 con 0x7fb63c0cffa0 2021-04-09T18:15:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.957 7fb63a61c700 1 --2- 172.21.15.32:0/2132095715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c0cffa0 0x7fb63c0d2440 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fb62c006700 tx=0x7fb62c009b00).ready entity=mon.0 client_cookie=9e8992f239f8ffc3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.957 7fb63961a700 1 -- 172.21.15.32:0/2132095715 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb62c00c0c0 con 0x7fb63c0cffa0 2021-04-09T18:15:30.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.957 7fb63961a700 1 -- 172.21.15.32:0/2132095715 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c00c400 con 0x7fb63c0cffa0 2021-04-09T18:15:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.957 7fb63961a700 1 -- 172.21.15.32:0/2132095715 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb62c00c8e0 con 0x7fb63c0cffa0 2021-04-09T18:15:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.957 7fb640a39700 1 -- 172.21.15.32:0/2132095715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c0cffa0 msgr2=0x7fb63c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:30.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.957 7fb640a39700 1 --2- 172.21.15.32:0/2132095715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c0cffa0 0x7fb63c0d2440 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fb62c006700 tx=0x7fb62c009b00).stop 2021-04-09T18:15:30.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.958 7fb640a39700 1 -- 172.21.15.32:0/2132095715 shutdown_connections 2021-04-09T18:15:30.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.958 7fb640a39700 1 --2- 172.21.15.32:0/2132095715 >> v2:172.21.15.86:3300/0 conn(0x7fb63c0d7800 0x7fb63c0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.958 7fb640a39700 1 --2- 172.21.15.32:0/2132095715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c0cffa0 0x7fb63c0d2440 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.958 7fb640a39700 1 -- 172.21.15.32:0/2132095715 >> 172.21.15.32:0/2132095715 conn(0x7fb63c0ce5c0 msgr2=0x7fb63c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.958 7fb640a39700 1 -- 172.21.15.32:0/2132095715 shutdown_connections 2021-04-09T18:15:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.958 7fb640a39700 1 -- 172.21.15.32:0/2132095715 wait complete. 2021-04-09T18:15:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.978 7fb640a39700 1 Processor -- start 2021-04-09T18:15:30.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.978 7fb640a39700 1 -- start start 2021-04-09T18:15:30.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.978 7fb640a39700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63c1929f0 0x7fb63c192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.978 7fb640a39700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63c08ddc0 0x7fb63c08e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:30.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.978 7fb640a39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c08e7e0 0x7fb63c08c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.978 7fb640a39700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c193410 con 0x7fb63c08e7e0 2021-04-09T18:15:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.978 7fb640a39700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c0a9140 con 0x7fb63c1929f0 2021-04-09T18:15:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.978 7fb640a39700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c08ce80 con 0x7fb63c08ddc0 2021-04-09T18:15:30.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.978 7fb63ae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c08e7e0 0x7fb63c08c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:30.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb63ae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c08e7e0 0x7fb63c08c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44106/0 (socket says 172.21.15.32:44106) 2021-04-09T18:15:30.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb63ae1d700 1 -- 172.21.15.32:0/2372433063 learned_addr learned my addr 172.21.15.32:0/2372433063 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:30.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb63a61c700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63c1929f0 0x7fb63c192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb639e1b700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63c08ddc0 0x7fb63c08e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb63ae1d700 1 -- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63c08ddc0 msgr2=0x7fb63c08e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb63ae1d700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63c08ddc0 0x7fb63c08e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb63ae1d700 1 -- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63c1929f0 msgr2=0x7fb63c192df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb63ae1d700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63c1929f0 0x7fb63c192df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb63ae1d700 1 -- 172.21.15.32:0/2372433063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c08d0a0 con 0x7fb63c08e7e0 2021-04-09T18:15:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb63ae1d700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c08e7e0 0x7fb63c08c860 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fb63400b8e0 tx=0x7fb63400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:30.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb62affd700 1 -- 172.21.15.32:0/2372433063 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb634000d30 con 0x7fb63c08e7e0 2021-04-09T18:15:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb640a39700 1 -- 172.21.15.32:0/2372433063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63c08d0a0 con 0x7fb63c08e7e0 2021-04-09T18:15:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb640a39700 1 -- 172.21.15.32:0/2372433063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63c08d0a0 con 0x7fb63c08e7e0 2021-04-09T18:15:30.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb62affd700 1 -- 172.21.15.32:0/2372433063 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63400bdc0 con 0x7fb63c08e7e0 2021-04-09T18:15:30.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.979 7fb62affd700 1 -- 172.21.15.32:0/2372433063 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb63400c2f0 con 0x7fb63c08e7e0 2021-04-09T18:15:30.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.980 7fb62affd700 1 -- 172.21.15.32:0/2372433063 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb634001220 con 0x7fb63c08e7e0 2021-04-09T18:15:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.980 7fb62affd700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb620018930 0x7fb62001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:30.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.980 7fb62affd700 1 -- 172.21.15.32:0/2372433063 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fb634032b80 con 0x7fb63c08e7e0 2021-04-09T18:15:30.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.980 7fb63a61c700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb620018930 0x7fb62001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:30.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:29.980 7fb63a61c700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb620018930 0x7fb62001ae10 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:30.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.000 7fb640a39700 1 -- 172.21.15.32:0/2372433063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63c1b70c0 con 0x7fb63c08e7e0 2021-04-09T18:15:30.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.003 7fb62affd700 1 -- 172.21.15.32:0/2372433063 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb634001220 con 0x7fb63c08e7e0 2021-04-09T18:15:30.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.190 7fb640a39700 1 -- 172.21.15.32:0/2372433063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7fb63c0d7280 con 0x7fb63c08e7e0 2021-04-09T18:15:30.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.719 7fb62affd700 1 -- 172.21.15.32:0/2372433063 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v147) v1 ==== 143+0+0 (secure 0 0 0) 0x7fb63400bdc0 con 0x7fb63c08e7e0 2021-04-09T18:15:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 -- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb620018930 msgr2=0x7fb62001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb620018930 0x7fb62001ae10 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 -- 172.21.15.32:0/2372433063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c08e7e0 msgr2=0x7fb63c08c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:30.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c08e7e0 0x7fb63c08c860 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fb63400b8e0 tx=0x7fb63400bc80).stop 2021-04-09T18:15:30.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 -- 172.21.15.32:0/2372433063 shutdown_connections 2021-04-09T18:15:30.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb620018930 0x7fb62001ae10 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb63c08e7e0 0x7fb63c08c860 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63c08ddc0 0x7fb63c08e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 --2- 172.21.15.32:0/2372433063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63c1929f0 0x7fb63c192df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:30.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 -- 172.21.15.32:0/2372433063 >> 172.21.15.32:0/2372433063 conn(0x7fb63c189b00 msgr2=0x7fb63c1921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:30.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 -- 172.21.15.32:0/2372433063 shutdown_connections 2021-04-09T18:15:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.758 7fb640a39700 1 -- 172.21.15.32:0/2372433063 wait complete. 2021-04-09T18:15:30.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:15:31.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.962 7fe019540700 1 Processor -- start 2021-04-09T18:15:31.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.962 7fe019540700 1 -- start start 2021-04-09T18:15:31.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe019540700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe01410d010 0x7fe0140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe019540700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0140d1da0 0x7fe0140d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe019540700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01410d480 con 0x7fe0140d82c0 2021-04-09T18:15:31.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe019540700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01410e780 con 0x7fe0140d1da0 2021-04-09T18:15:31.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe019540700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140d8d30 con 0x7fe01410d010 2021-04-09T18:15:31.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe012abe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0140d1da0 0x7fe0140d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe013ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe0140d82c0 0x7fe01410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53702/0 (socket says 172.21.15.32:53702) 2021-04-09T18:15:31.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe013ac0700 1 -- 172.21.15.32:0/1420569565 learned_addr learned my addr 172.21.15.32:0/1420569565 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:31.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe0132bf700 1 --2- 172.21.15.32:0/1420569565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe01410d010 0x7fe0140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe012abe700 1 -- 172.21.15.32:0/1420569565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe01410d010 msgr2=0x7fe0140d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:31.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.963 7fe012abe700 1 --2- 172.21.15.32:0/1420569565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe01410d010 0x7fe0140d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.964 7fe012abe700 1 -- 172.21.15.32:0/1420569565 >> v1:172.21.15.32:6789/0 conn(0x7fe0140d82c0 legacy=0x7fe01410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.964 7fe012abe700 1 -- 172.21.15.32:0/1420569565 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0140d9220 con 0x7fe0140d1da0 2021-04-09T18:15:31.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.964 7fe0122bd700 1 -- 172.21.15.32:0/1420569565 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238282539 0 0) 0x7fe00c000d30 con 0x7fe0140d82c0 2021-04-09T18:15:31.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.964 7fe012abe700 1 --2- 172.21.15.32:0/1420569565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0140d1da0 0x7fe0140d7ca0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fe008006650 tx=0x7fe008009a00).ready entity=mon.0 client_cookie=5a0085b74894f076 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:31.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.964 7fe0122bd700 1 -- 172.21.15.32:0/1420569565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe008009df0 con 0x7fe0140d1da0 2021-04-09T18:15:31.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.964 7fe0122bd700 1 -- 172.21.15.32:0/1420569565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0080009b0 con 0x7fe0140d1da0 2021-04-09T18:15:31.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.964 7fe0122bd700 1 -- 172.21.15.32:0/1420569565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe008009df0 con 0x7fe0140d1da0 2021-04-09T18:15:31.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.964 7fe019540700 1 -- 172.21.15.32:0/1420569565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0140d1da0 msgr2=0x7fe0140d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.964 7fe019540700 1 --2- 172.21.15.32:0/1420569565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0140d1da0 0x7fe0140d7ca0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fe008006650 tx=0x7fe008009a00).stop 2021-04-09T18:15:31.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.965 7fe019540700 1 -- 172.21.15.32:0/1420569565 shutdown_connections 2021-04-09T18:15:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.965 7fe019540700 1 --2- 172.21.15.32:0/1420569565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0140d1da0 0x7fe0140d7ca0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.965 7fe019540700 1 --2- 172.21.15.32:0/1420569565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe01410d010 0x7fe0140d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.965 7fe019540700 1 -- 172.21.15.32:0/1420569565 >> 172.21.15.32:0/1420569565 conn(0x7fe0140ce5f0 msgr2=0x7fe0140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:31.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.965 7fe019540700 1 -- 172.21.15.32:0/1420569565 shutdown_connections 2021-04-09T18:15:31.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.965 7fe019540700 1 -- 172.21.15.32:0/1420569565 wait complete. 2021-04-09T18:15:31.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.985 7fe019540700 1 Processor -- start 2021-04-09T18:15:31.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.985 7fe019540700 1 -- start start 2021-04-09T18:15:31.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.985 7fe019540700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01417bb20 0x7fe0140c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.985 7fe019540700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0140b7620 0x7fe0140c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.985 7fe019540700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe01417bf20 0x7fe01408a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.985 7fe019540700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c1e00 con 0x7fe01417bb20 2021-04-09T18:15:31.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.985 7fe019540700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c0b00 con 0x7fe01417bf20 2021-04-09T18:15:31.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.985 7fe019540700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01408ac40 con 0x7fe0140b7620 2021-04-09T18:15:31.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe0132bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01417bb20 0x7fe0140c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe0132bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01417bb20 0x7fe0140c1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44116/0 (socket says 172.21.15.32:44116) 2021-04-09T18:15:31.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe0132bf700 1 -- 172.21.15.32:0/3378090447 learned_addr learned my addr 172.21.15.32:0/3378090447 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:31.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe013ac0700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe01417bf20 0x7fe01408a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe012abe700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0140b7620 0x7fe0140c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe0132bf700 1 -- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0140b7620 msgr2=0x7fe0140c16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe0132bf700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0140b7620 0x7fe0140c16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe0132bf700 1 -- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe01417bf20 msgr2=0x7fe01408a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe0132bf700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe01417bf20 0x7fe01408a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe0132bf700 1 -- 172.21.15.32:0/3378090447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01408ae60 con 0x7fe01417bb20 2021-04-09T18:15:31.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe013ac0700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe01417bf20 0x7fe01408a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:31.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe012abe700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0140b7620 0x7fe0140c16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:31.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe0132bf700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01417bb20 0x7fe0140c1080 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fe004006f60 tx=0x7fe00400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:31.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fdffbfff700 1 -- 172.21.15.32:0/3378090447 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe00400bd60 con 0x7fe01417bb20 2021-04-09T18:15:31.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fdffbfff700 1 -- 172.21.15.32:0/3378090447 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe004000c90 con 0x7fe01417bb20 2021-04-09T18:15:31.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fdffbfff700 1 -- 172.21.15.32:0/3378090447 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe004002220 con 0x7fe01417bb20 2021-04-09T18:15:31.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe019540700 1 -- 172.21.15.32:0/3378090447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01408ae60 con 0x7fe01417bb20 2021-04-09T18:15:31.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.986 7fe019540700 1 -- 172.21.15.32:0/3378090447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01408ae60 con 0x7fe01417bb20 2021-04-09T18:15:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.987 7fdffbfff700 1 -- 172.21.15.32:0/3378090447 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe004006b00 con 0x7fe01417bb20 2021-04-09T18:15:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.987 7fdffbfff700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdff40188d0 0x7fdff401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.987 7fdffbfff700 1 -- 172.21.15.32:0/3378090447 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fe004032900 con 0x7fe01417bb20 2021-04-09T18:15:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.987 7fe012abe700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdff40188d0 0x7fdff401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:30.988 7fe012abe700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdff40188d0 0x7fdff401add0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:31.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.007 7fe019540700 1 -- 172.21.15.32:0/3378090447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01410d480 con 0x7fe01417bb20 2021-04-09T18:15:31.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.011 7fdffbfff700 1 -- 172.21.15.32:0/3378090447 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe004006340 con 0x7fe01417bb20 2021-04-09T18:15:31.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.224 7fe019540700 1 -- 172.21.15.32:0/3378090447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe01410d480 con 0x7fe01417bb20 2021-04-09T18:15:31.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.227 7fdffbfff700 1 -- 172.21.15.32:0/3378090447 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe004006b00 con 0x7fe01417bb20 2021-04-09T18:15:31.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 -- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdff40188d0 msgr2=0x7fdff401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdff40188d0 0x7fdff401add0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 -- 172.21.15.32:0/3378090447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01417bb20 msgr2=0x7fe0140c1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01417bb20 0x7fe0140c1080 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fe004006f60 tx=0x7fe00400b970).stop 2021-04-09T18:15:31.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 -- 172.21.15.32:0/3378090447 shutdown_connections 2021-04-09T18:15:31.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdff40188d0 0x7fdff401add0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe01417bf20 0x7fe01408a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0140b7620 0x7fe0140c16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 --2- 172.21.15.32:0/3378090447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe01417bb20 0x7fe0140c1080 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 -- 172.21.15.32:0/3378090447 >> 172.21.15.32:0/3378090447 conn(0x7fe014181390 msgr2=0x7fe0140adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 -- 172.21.15.32:0/3378090447 shutdown_connections 2021-04-09T18:15:31.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.233 7fe019540700 1 -- 172.21.15.32:0/3378090447 wait complete. 2021-04-09T18:15:31.315 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:15:31.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:15:31.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:15:31.448 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:15:31.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:31.450 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:15:31.453 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:15:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:15:31.509 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:15:31.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:31.510 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:15:31.511 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:15:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:15:31.565 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:15:31.566 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:15:31.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:15:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.693 7f609ec85700 1 Processor -- start 2021-04-09T18:15:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.693 7f609ec85700 1 -- start start 2021-04-09T18:15:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.694 7f609ec85700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f609810da00 0x7f609810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.694 7f609ec85700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609810d480 con 0x7f609810d010 2021-04-09T18:15:31.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.694 7f609ec85700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60980d1f70 con 0x7f60980d7ca0 2021-04-09T18:15:31.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.694 7f609ec85700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60980d8d30 con 0x7f609810da00 2021-04-09T18:15:31.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.694 7f609d2a2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f609810da00 0x7f609810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.694 7f609caa1700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f609810d010 0x7f60980d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53714/0 (socket says 172.21.15.32:53714) 2021-04-09T18:15:31.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.694 7f609d2a2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f609810da00 0x7f609810de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44124/0 (socket says 172.21.15.32:44124) 2021-04-09T18:15:31.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.694 7f609caa1700 1 -- 172.21.15.32:0/1024389796 learned_addr learned my addr 172.21.15.32:0/1024389796 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:31.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.694 7f609d2a2700 1 -- 172.21.15.32:0/1024389796 >> v1:172.21.15.86:6789/0 conn(0x7f60980d7ca0 legacy=0x7f60980d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.695 7f60977fe700 1 -- 172.21.15.32:0/1024389796 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359030385 0 0) 0x7f6080000ce0 con 0x7f609810d010 2021-04-09T18:15:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.695 7f609d2a2700 1 -- 172.21.15.32:0/1024389796 >> v1:172.21.15.32:6789/0 conn(0x7f609810d010 legacy=0x7f60980d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.695 7f609d2a2700 1 -- 172.21.15.32:0/1024389796 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60980d9220 con 0x7f609810da00 2021-04-09T18:15:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.695 7f609d2a2700 1 --2- 172.21.15.32:0/1024389796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f609810da00 0x7f609810de00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f6090006e30 tx=0x7f6090009de0).ready entity=mon.0 client_cookie=297906096b134916 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:31.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.695 7f60977fe700 1 -- 172.21.15.32:0/1024389796 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f609000b300 con 0x7f609810da00 2021-04-09T18:15:31.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.695 7f60977fe700 1 -- 172.21.15.32:0/1024389796 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609000b680 con 0x7f609810da00 2021-04-09T18:15:31.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.695 7f60977fe700 1 -- 172.21.15.32:0/1024389796 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f609000bb60 con 0x7f609810da00 2021-04-09T18:15:31.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.695 7f609ec85700 1 -- 172.21.15.32:0/1024389796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f609810da00 msgr2=0x7f609810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.695 7f609ec85700 1 --2- 172.21.15.32:0/1024389796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f609810da00 0x7f609810de00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f6090006e30 tx=0x7f6090009de0).stop 2021-04-09T18:15:31.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.696 7f609ec85700 1 -- 172.21.15.32:0/1024389796 shutdown_connections 2021-04-09T18:15:31.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.696 7f609ec85700 1 --2- 172.21.15.32:0/1024389796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f609810da00 0x7f609810de00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.696 7f609ec85700 1 -- 172.21.15.32:0/1024389796 >> 172.21.15.32:0/1024389796 conn(0x7f60980ce5f0 msgr2=0x7f60980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.696 7f609ec85700 1 -- 172.21.15.32:0/1024389796 shutdown_connections 2021-04-09T18:15:31.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.696 7f609ec85700 1 -- 172.21.15.32:0/1024389796 wait complete. 2021-04-09T18:15:31.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.716 7f609ec85700 1 Processor -- start 2021-04-09T18:15:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.716 7f609ec85700 1 -- start start 2021-04-09T18:15:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.716 7f609ec85700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60980a75d0 0x7f60981bef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609ec85700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60980bc870 0x7f60980b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609ec85700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60981bf520 0x7f60980b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609ec85700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60981abd30 con 0x7f60980a75d0 2021-04-09T18:15:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609ec85700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60980b4290 con 0x7f60981bf520 2021-04-09T18:15:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609ec85700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60980a7b00 con 0x7f60980bc870 2021-04-09T18:15:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609caa1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60980a75d0 0x7f60981bef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609caa1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60980a75d0 0x7f60981bef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44128/0 (socket says 172.21.15.32:44128) 2021-04-09T18:15:31.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609caa1700 1 -- 172.21.15.32:0/2275122479 learned_addr learned my addr 172.21.15.32:0/2275122479 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:31.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609d2a2700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60981bf520 0x7f60980b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f6097fff700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60980bc870 0x7f60980b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609caa1700 1 -- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60980bc870 msgr2=0x7f60980b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609caa1700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60980bc870 0x7f60980b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609caa1700 1 -- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60981bf520 msgr2=0x7f60980b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:31.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609caa1700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60981bf520 0x7f60980b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:31.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609caa1700 1 -- 172.21.15.32:0/2275122479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60980a7d20 con 0x7f60980a75d0 2021-04-09T18:15:31.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609d2a2700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60981bf520 0x7f60980b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:31.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f6097fff700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60980bc870 0x7f60980b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:31.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609caa1700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60980a75d0 0x7f60981bef00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f6080006f00 tx=0x7f608000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f60957fa700 1 -- 172.21.15.32:0/2275122479 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f608000bd30 con 0x7f60980a75d0 2021-04-09T18:15:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f60957fa700 1 -- 172.21.15.32:0/2275122479 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60800060d0 con 0x7f60980a75d0 2021-04-09T18:15:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f60957fa700 1 -- 172.21.15.32:0/2275122479 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6080004620 con 0x7f60980a75d0 2021-04-09T18:15:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609ec85700 1 -- 172.21.15.32:0/2275122479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60980a7d20 con 0x7f60980a75d0 2021-04-09T18:15:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.717 7f609ec85700 1 -- 172.21.15.32:0/2275122479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60980a7d20 con 0x7f60980a75d0 2021-04-09T18:15:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.718 7f60957fa700 1 -- 172.21.15.32:0/2275122479 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f60800062b0 con 0x7f60980a75d0 2021-04-09T18:15:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.718 7f60957fa700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f607c018930 0x7f607c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.718 7f60957fa700 1 -- 172.21.15.32:0/2275122479 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f6080032760 con 0x7f60980a75d0 2021-04-09T18:15:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.718 7f6097fff700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f607c018930 0x7f607c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.719 7f6097fff700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f607c018930 0x7f607c01ae10 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.738 7f609ec85700 1 -- 172.21.15.32:0/2275122479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60981abd30 con 0x7f60980a75d0 2021-04-09T18:15:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.741 7f60957fa700 1 -- 172.21.15.32:0/2275122479 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6080006b90 con 0x7f60980a75d0 2021-04-09T18:15:31.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.930 7f609ec85700 1 -- 172.21.15.32:0/2275122479 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f60981abd30 con 0x7f60980a75d0 2021-04-09T18:15:31.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.931 7f60957fa700 1 -- 172.21.15.32:0/2275122479 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v46) v1 ==== 81+0+25680 (secure 0 0 0) 0x7f60800060d0 con 0x7f60980a75d0 2021-04-09T18:15:31.963 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:31.964 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:15:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 46, 2021-04-09T18:15:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:15:31.964 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:15:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:15:27.707288", 2021-04-09T18:15:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:15:31.965 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:15:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:15:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:15:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:15:31.966 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:15:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:15:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:15:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:15:31.967 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:15:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:15:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:15:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:15:31.968 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:15:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:15:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 7, 2021-04-09T18:15:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:15:31.984 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:15:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:15:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:15:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:15:31.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:15:31.986 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:15:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:15:32.023 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:15:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:15:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:15:32.028 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:15:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:15:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:15:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:15:32.029 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:15:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:15:32.030 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:15:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:15:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:15:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:15:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:15:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:15:32.048 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:15:32.056 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:15:32.069 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:15:32.069 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:15:32.073 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:15:32.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.074 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:15:32.074 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:15:32.077 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:15:32.078 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:15:32.078 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:15:32.078 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:15:32.078 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:15:32.079 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:15:32.079 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:15:32.079 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:15:32.079 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:15:32.080 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:15:32.080 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:15:32.080 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:15:32.080 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:15:32.081 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:15:32.081 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:15:32.121 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:15:32.122 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:15:32.122 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:15:32.132 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:15:32.132 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:15:32.132 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:15:32.133 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:15:32.133 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:15:32.134 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:15:32.134 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:15:32.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.135 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:15:32.145 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:15:32.145 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:15:32.146 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:15:32.146 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:15:32.155 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:15:32.156 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:15:32.157 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:15:32.158 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:15:32.160 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:15:32.160 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:15:32.161 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:15:32.161 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:15:32.162 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:15:32.162 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.162 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.163 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:15:32.163 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:15:32.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:15:32.219 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:15:32.219 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:32.219 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:32.220 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:32.220 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:32.220 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:32.231 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:32.232 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:32.232 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:32.232 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:32.233 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:32.233 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:32.233 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.234 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:15:32.234 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:32.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.235 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.235 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:15:32.235 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:32.235 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.277 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:32.278 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.279 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:15:32.284 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:32.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.288 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.288 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:15:32.289 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:32.289 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.290 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.323 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:32.324 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.330 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.330 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:15:32.331 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:32.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.370 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.370 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:15:32.373 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:32.373 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.378 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.379 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:32.388 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.389 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.389 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:15:32.390 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:32.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.405 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:15:32.405 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:15:32.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.410 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.413 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:15:32.413 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:15:32.440 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:15:32.441 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:15:32.441 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:32.448 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:32.448 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:32.448 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.450 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:15:32.450 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:15:32.451 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:32.470 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:32.481 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:32.489 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:32.490 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 40, 2021-04-09T18:15:32.491 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:32.491 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:15:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:15:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.521 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:32.522 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:15:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:15:32.562 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.564 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:32.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.574 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:15:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:15:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.602 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:15:32.602 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:32.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.603 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:15:32.603 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:15:32.603 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.604 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.604 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:15:32.604 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:15:32.635 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:15:32.635 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:15:32.637 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:32.637 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:32.637 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:32.637 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.638 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:15:32.638 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:15:32.638 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:32.641 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:32.641 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:32.642 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:32.642 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:32.647 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:32.647 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:32.648 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:15:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:32.663 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.663 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:15:32.664 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:32.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.665 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:15:32.665 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:32.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.665 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.666 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:32.666 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.667 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:15:32.694 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:32.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:15:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.706 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:32.706 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.708 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:15:32.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:32.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.731 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.731 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:15:32.749 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:32.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.751 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.755 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:32.755 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.756 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.756 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:15:32.757 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:32.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.758 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.758 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:15:32.759 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:15:32.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.789 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.789 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:15:32.790 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:15:32.790 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:15:32.790 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:15:32.791 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:32.791 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:32.792 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:32.792 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.847 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:15:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:15:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:32.849 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:32.849 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:32.849 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:32.850 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:32.850 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:32.853 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:32.853 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:32.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:15:32.855 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:32.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:15:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:15:32.978 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:32.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.980 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:15:32.981 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:32.981 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:32.982 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:32.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:32.983 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:32.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.986 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:32.986 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:15:32.986 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:32.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:32.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:32.996 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:32.996 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:15:32.997 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:32.997 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.009 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.010 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:33.010 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.011 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.018 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:15:33.022 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:33.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.023 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.023 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:15:33.024 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:15:33.024 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.026 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.027 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:15:33.027 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:15:33.030 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:15:33.030 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:15:33.031 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:33.031 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:33.032 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:33.032 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.045 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:15:33.045 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:15:33.046 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:33.046 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:33.047 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:33.047 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:33.048 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:33.057 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:33.057 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:33.058 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:33.058 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:33.058 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:33.059 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:33.059 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.060 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.060 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.060 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:15:33.061 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:33.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.073 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.073 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:15:33.074 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:33.074 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.074 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.075 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:33.075 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.076 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:15:33.113 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:33.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.115 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.115 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:15:33.118 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:33.119 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.119 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:33.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:15:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:33.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.132 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:15:33.133 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:33.170 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.170 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.171 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:33.172 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.187 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.187 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:15:33.187 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:33.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.209 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:15:33.209 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:15:33.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.210 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.211 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:15:33.211 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:15:33.211 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:15:33.212 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:15:33.212 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:33.212 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:33.270 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:33.271 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.272 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:15:33.285 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:15:33.286 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:33.298 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:33.301 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:33.302 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:33.383 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:33.390 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:33.392 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:33.393 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:33.400 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:33.401 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:33.401 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:33.408 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.426 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:15:33.426 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:33.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.445 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:15:33.446 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:33.446 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.446 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.447 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:33.448 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.448 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:15:33.505 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:33.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:15:33.526 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.527 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:33.528 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.529 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:15:33.530 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:33.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:15:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:33.595 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.595 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:15:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:33.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.598 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.598 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:15:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:15:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:15:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:15:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:15:33.604 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:15:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:33.605 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.615 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:15:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:15:33.616 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:33.624 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:15:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:33.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.656 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:15:33.656 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:33.660 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.661 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.661 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:33.661 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.662 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.662 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:15:33.667 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:33.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.680 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.680 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:15:33.688 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:33.688 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.688 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.689 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:33.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.690 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:15:33.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:33.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:15:33.733 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:33.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.733 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.734 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:33.734 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.783 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:15:33.783 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:33.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.784 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.785 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:15:33.785 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:15:33.785 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.786 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.787 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:15:33.814 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:15:33.814 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:15:33.815 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:15:33.815 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:33.815 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:33.816 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:33.816 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.843 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:15:33.844 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:15:33.844 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:15:33.844 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:15:33.844 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:15:33.845 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:15:33.845 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:15:33.845 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:15:33.845 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:15:33.847 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 41, 2021-04-09T18:15:33.847 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:15:33.848 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:15:33.848 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:15:33.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:15:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.850 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.850 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:15:33.850 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:33.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:15:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:15:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.853 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:15:33.853 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:33.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:15:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:15:33.880 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:33.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.881 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:15:33.881 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:33.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.881 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.893 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:15:33.893 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:15:33.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.894 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:15:33.899 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:15:33.899 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:33.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:33.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:15:33.900 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:15:33.900 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.900 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:15:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:15:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:15:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:15:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:15:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:15:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:15:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:15:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:15:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:15:33.904 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:33.953 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:33.953 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:33.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.954 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:15:33.955 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:33.955 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:33.955 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:33.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:33.990 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:15:33.991 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:33.991 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:33.992 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:33.992 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:33.992 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:33.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:34.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:34.036 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:15:34.037 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:34.037 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:34.037 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:34.038 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:34.039 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:34.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:34.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:34.041 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:15:34.095 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:34.100 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:34.101 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:34.101 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:34.102 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:34.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:34.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:34.103 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:15:34.103 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:34.103 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:34.104 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:34.104 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:34.105 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:34.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:34.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:34.106 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:15:34.106 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:34.106 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:34.107 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:34.164 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:34.164 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:34.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:34.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:34.165 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:15:34.166 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:15:34.166 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:15:34.166 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:15:34.167 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:15:34.204 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:15:34.204 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:34.204 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:15:34.205 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:15:34.205 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:15:34.205 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:15:34.206 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:15:34.206 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:15:34.206 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/1051691429": "2021-04-10 18:15:23.697687", 2021-04-09T18:15:34.206 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/1051691429": "2021-04-10 18:15:23.697687", 2021-04-09T18:15:34.207 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/3664667451": "2021-04-10 18:15:23.697671", 2021-04-09T18:15:34.207 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/3664667451": "2021-04-10 18:15:23.697671" 2021-04-09T18:15:34.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:34.207 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:15:34.208 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:15:34.208 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:15:34.208 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:15:34.208 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:15:34.209 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:15:34.262 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:15:34.262 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:15:34.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:34.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:34.263 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:15:34.264 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:15:34.264 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:15:34.287 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:15:34.288 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:15:34.288 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:15:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 -- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f607c018930 msgr2=0x7f607c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f607c018930 0x7f607c01ae10 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 -- 172.21.15.32:0/2275122479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60980a75d0 msgr2=0x7f60981bef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:34.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60980a75d0 0x7f60981bef00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f6080006f00 tx=0x7f608000b940).stop 2021-04-09T18:15:34.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 -- 172.21.15.32:0/2275122479 shutdown_connections 2021-04-09T18:15:34.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f607c018930 0x7f607c01ae10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60981bf520 0x7f60980b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60980bc870 0x7f60980b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 --2- 172.21.15.32:0/2275122479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60980a75d0 0x7f60981bef00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 -- 172.21.15.32:0/2275122479 >> 172.21.15.32:0/2275122479 conn(0x7f60981acbe0 msgr2=0x7f609808dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:34.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 -- 172.21.15.32:0/2275122479 shutdown_connections 2021-04-09T18:15:34.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:31.935 7f609ec85700 1 -- 172.21.15.32:0/2275122479 wait complete. 2021-04-09T18:15:34.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.505 7f49ded8f700 1 Processor -- start 2021-04-09T18:15:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.505 7f49ded8f700 1 -- start start 2021-04-09T18:15:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.505 7f49ded8f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49d80d7ca0 0x7f49d80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.505 7f49ded8f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f49d810da00 0x7f49d810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.505 7f49ded8f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d810d480 con 0x7f49d810d010 2021-04-09T18:15:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.505 7f49ded8f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d80d9220 con 0x7f49d80d7ca0 2021-04-09T18:15:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.505 7f49ded8f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d80d86c0 con 0x7f49d810da00 2021-04-09T18:15:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49d80d7ca0 0x7f49d80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49d80d7ca0 0x7f49d80d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44136/0 (socket says 172.21.15.32:44136) 2021-04-09T18:15:34.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d7fff700 1 -- 172.21.15.32:0/3875530313 learned_addr learned my addr 172.21.15.32:0/3875530313 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:34.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49dd3ac700 1 --2- 172.21.15.32:0/3875530313 >> v2:172.21.15.86:3300/0 conn(0x7f49d810da00 0x7f49d810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:34.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d7fff700 1 -- 172.21.15.32:0/3875530313 >> v2:172.21.15.86:3300/0 conn(0x7f49d810da00 msgr2=0x7f49d810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:34.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d7fff700 1 --2- 172.21.15.32:0/3875530313 >> v2:172.21.15.86:3300/0 conn(0x7f49d810da00 0x7f49d810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d7fff700 1 -- 172.21.15.32:0/3875530313 >> v1:172.21.15.86:6789/0 conn(0x7f49d810d010 legacy=0x7f49d80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:34.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d7fff700 1 -- 172.21.15.32:0/3875530313 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d80d1f70 con 0x7f49d80d7ca0 2021-04-09T18:15:34.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49dd3ac700 1 --2- 172.21.15.32:0/3875530313 >> v2:172.21.15.86:3300/0 conn(0x7f49d810da00 0x7f49d810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:34.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d7fff700 1 --2- 172.21.15.32:0/3875530313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80d7ca0 0x7f49d80d80a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f49c0006700 tx=0x7f49c0009b80).ready entity=mon.0 client_cookie=c11709281a5b741d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:34.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d77fe700 1 -- 172.21.15.32:0/3875530313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c000c0c0 con 0x7f49d80d7ca0 2021-04-09T18:15:34.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d77fe700 1 -- 172.21.15.32:0/3875530313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c000c440 con 0x7f49d80d7ca0 2021-04-09T18:15:34.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49d77fe700 1 -- 172.21.15.32:0/3875530313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c000c920 con 0x7f49d80d7ca0 2021-04-09T18:15:34.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49ded8f700 1 -- 172.21.15.32:0/3875530313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80d7ca0 msgr2=0x7f49d80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:34.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.506 7f49ded8f700 1 --2- 172.21.15.32:0/3875530313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80d7ca0 0x7f49d80d80a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f49c0006700 tx=0x7f49c0009b80).stop 2021-04-09T18:15:34.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.507 7f49ded8f700 1 -- 172.21.15.32:0/3875530313 shutdown_connections 2021-04-09T18:15:34.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.507 7f49ded8f700 1 --2- 172.21.15.32:0/3875530313 >> v2:172.21.15.86:3300/0 conn(0x7f49d810da00 0x7f49d810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.507 7f49ded8f700 1 --2- 172.21.15.32:0/3875530313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d80d7ca0 0x7f49d80d80a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.507 7f49ded8f700 1 -- 172.21.15.32:0/3875530313 >> 172.21.15.32:0/3875530313 conn(0x7f49d80ce5f0 msgr2=0x7f49d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:34.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.507 7f49ded8f700 1 -- 172.21.15.32:0/3875530313 shutdown_connections 2021-04-09T18:15:34.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.507 7f49ded8f700 1 -- 172.21.15.32:0/3875530313 wait complete. 2021-04-09T18:15:34.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.527 7f49ded8f700 1 Processor -- start 2021-04-09T18:15:34.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.527 7f49ded8f700 1 -- start start 2021-04-09T18:15:34.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49ded8f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d81a5210 0x7f49d8185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:34.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49ded8f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8185a70 0x7f49d80acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:34.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49ded8f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d8185e70 0x7f49d80ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:34.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49ded8f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d818f7b0 con 0x7f49d8185a70 2021-04-09T18:15:34.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49ded8f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80ad7f0 con 0x7f49d8185e70 2021-04-09T18:15:34.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49ded8f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80ada10 con 0x7f49d81a5210 2021-04-09T18:15:34.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49d7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8185a70 0x7f49d80acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:34.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49d7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8185a70 0x7f49d80acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44146/0 (socket says 172.21.15.32:44146) 2021-04-09T18:15:34.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49d7fff700 1 -- 172.21.15.32:0/542257789 learned_addr learned my addr 172.21.15.32:0/542257789 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:34.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49dd3ac700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d8185e70 0x7f49d80ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:34.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49dcbab700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d81a5210 0x7f49d8185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:34.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49d7fff700 1 -- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d81a5210 msgr2=0x7f49d8185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:34.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49d7fff700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d81a5210 0x7f49d8185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49d7fff700 1 -- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d8185e70 msgr2=0x7f49d80ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49d7fff700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d8185e70 0x7f49d80ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49d7fff700 1 -- 172.21.15.32:0/542257789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d8179610 con 0x7f49d8185a70 2021-04-09T18:15:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.528 7f49dcbab700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d81a5210 0x7f49d8185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.529 7f49d7fff700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8185a70 0x7f49d80acbb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f49c000b5e0 tx=0x7f49c000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.529 7f49d57fa700 1 -- 172.21.15.32:0/542257789 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c000bd70 con 0x7f49d8185a70 2021-04-09T18:15:34.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.529 7f49d57fa700 1 -- 172.21.15.32:0/542257789 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0002030 con 0x7f49d8185a70 2021-04-09T18:15:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.529 7f49d57fa700 1 -- 172.21.15.32:0/542257789 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c0002550 con 0x7f49d8185a70 2021-04-09T18:15:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.529 7f49ded8f700 1 -- 172.21.15.32:0/542257789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d8179610 con 0x7f49d8185a70 2021-04-09T18:15:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.529 7f49ded8f700 1 -- 172.21.15.32:0/542257789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d8179610 con 0x7f49d8185a70 2021-04-09T18:15:34.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.529 7f49d57fa700 1 -- 172.21.15.32:0/542257789 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f49c0000c90 con 0x7f49d8185a70 2021-04-09T18:15:34.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.529 7f49d57fa700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49bc018930 0x7f49bc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:34.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.530 7f49d57fa700 1 -- 172.21.15.32:0/542257789 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f49c00327b0 con 0x7f49d8185a70 2021-04-09T18:15:34.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.530 7f49dcbab700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49bc018930 0x7f49bc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:34.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.530 7f49dcbab700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49bc018930 0x7f49bc01adc0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:34.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.549 7f49ded8f700 1 -- 172.21.15.32:0/542257789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d81b7070 con 0x7f49d8185a70 2021-04-09T18:15:34.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.553 7f49d57fa700 1 -- 172.21.15.32:0/542257789 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f49c001f770 con 0x7f49d8185a70 2021-04-09T18:15:34.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.745 7f49ded8f700 1 -- 172.21.15.32:0/542257789 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49d810d480 con 0x7f49d8185a70 2021-04-09T18:15:34.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.745 7f49d57fa700 1 -- 172.21.15.32:0/542257789 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+1850 (secure 0 0 0) 0x7f49c000bd70 con 0x7f49d8185a70 2021-04-09T18:15:34.773 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 39 2021-04-09T18:15:34.773 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:34.773 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/1797590590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1797590590},{"type":"v1","addr":"172.21.15.86:6835","nonce":1797590590}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":34},{"gid":5445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2789516883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2789516883},{"type":"v1","addr":"172.21.15.32:6835","nonce":2789516883}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":39},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":55013,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3273459469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3273459469},{"type":"v1","addr":"172.21.15.32:6833","nonce":3273459469}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":39}],"filesystems":[]} 2021-04-09T18:15:34.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.747 7f49ded8f700 1 -- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49bc018930 msgr2=0x7f49bc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:34.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.747 7f49ded8f700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49bc018930 0x7f49bc01adc0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.747 7f49ded8f700 1 -- 172.21.15.32:0/542257789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8185a70 msgr2=0x7f49d80acbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:34.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.747 7f49ded8f700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8185a70 0x7f49d80acbb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f49c000b5e0 tx=0x7f49c000b980).stop 2021-04-09T18:15:34.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.748 7f49ded8f700 1 -- 172.21.15.32:0/542257789 shutdown_connections 2021-04-09T18:15:34.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.748 7f49ded8f700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49bc018930 0x7f49bc01adc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.748 7f49ded8f700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d8185a70 0x7f49d80acbb0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.748 7f49ded8f700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d8185e70 0x7f49d80ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.748 7f49ded8f700 1 --2- 172.21.15.32:0/542257789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d81a5210 0x7f49d8185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.748 7f49ded8f700 1 -- 172.21.15.32:0/542257789 >> 172.21.15.32:0/542257789 conn(0x7f49d81a99c0 msgr2=0x7f49d817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:34.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.748 7f49ded8f700 1 -- 172.21.15.32:0/542257789 shutdown_connections 2021-04-09T18:15:34.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.748 7f49ded8f700 1 -- 172.21.15.32:0/542257789 wait complete. 2021-04-09T18:15:34.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:34.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:15:34.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:15:34.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.908 7efe75381700 1 Processor -- start 2021-04-09T18:15:34.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.908 7efe75381700 1 -- start start 2021-04-09T18:15:34.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe75381700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe700d7ca0 0x7efe700d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:34.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe75381700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe7010da00 0x7efe7010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:34.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe75381700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7010d480 con 0x7efe7010d010 2021-04-09T18:15:34.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe75381700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe700d9220 con 0x7efe7010da00 2021-04-09T18:15:34.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe75381700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe700d86c0 con 0x7efe700d7ca0 2021-04-09T18:15:34.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6f07d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efe7010d010 0x7efe700d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53740/0 (socket says 172.21.15.32:53740) 2021-04-09T18:15:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6f07d700 1 -- 172.21.15.32:0/2205057338 learned_addr learned my addr 172.21.15.32:0/2205057338 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6f87e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe7010da00 0x7efe7010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:34.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6e87c700 1 --2- 172.21.15.32:0/2205057338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe700d7ca0 0x7efe700d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:34.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6e07b700 1 -- 172.21.15.32:0/2205057338 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700605387 0 0) 0x7efe60000ce0 con 0x7efe7010d010 2021-04-09T18:15:34.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6e07b700 1 -- 172.21.15.32:0/2205057338 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe58002fe0 con 0x7efe7010d010 2021-04-09T18:15:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6f87e700 1 -- 172.21.15.32:0/2205057338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe700d7ca0 msgr2=0x7efe700d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:34.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6f87e700 1 --2- 172.21.15.32:0/2205057338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe700d7ca0 0x7efe700d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:34.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6f87e700 1 -- 172.21.15.32:0/2205057338 >> v1:172.21.15.32:6789/0 conn(0x7efe7010d010 legacy=0x7efe700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:34.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6f87e700 1 -- 172.21.15.32:0/2205057338 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe700d1f70 con 0x7efe7010da00 2021-04-09T18:15:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6f87e700 1 --2- 172.21.15.32:0/2205057338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7010da00 0x7efe7010de00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7efe68006e30 tx=0x7efe68009de0).ready entity=mon.0 client_cookie=77416892f16d3610 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6e07b700 1 -- 172.21.15.32:0/2205057338 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe6800b300 con 0x7efe7010da00 2021-04-09T18:15:34.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6e07b700 1 -- 172.21.15.32:0/2205057338 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6800b680 con 0x7efe7010da00 2021-04-09T18:15:34.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe6e07b700 1 -- 172.21.15.32:0/2205057338 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe6800bb60 con 0x7efe7010da00 2021-04-09T18:15:34.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe75381700 1 -- 172.21.15.32:0/2205057338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7010da00 msgr2=0x7efe7010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:34.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.909 7efe75381700 1 --2- 172.21.15.32:0/2205057338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7010da00 0x7efe7010de00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7efe68006e30 tx=0x7efe68009de0).stop 2021-04-09T18:15:35.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.910 7efe75381700 1 -- 172.21.15.32:0/2205057338 shutdown_connections 2021-04-09T18:15:35.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.910 7efe75381700 1 --2- 172.21.15.32:0/2205057338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe7010da00 0x7efe7010de00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:35.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.910 7efe75381700 1 --2- 172.21.15.32:0/2205057338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe700d7ca0 0x7efe700d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:35.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.910 7efe75381700 1 -- 172.21.15.32:0/2205057338 >> 172.21.15.32:0/2205057338 conn(0x7efe700ce5f0 msgr2=0x7efe700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:35.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.910 7efe75381700 1 -- 172.21.15.32:0/2205057338 shutdown_connections 2021-04-09T18:15:35.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.910 7efe75381700 1 -- 172.21.15.32:0/2205057338 wait complete. 2021-04-09T18:15:35.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.930 7efe75381700 1 Processor -- start 2021-04-09T18:15:35.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.930 7efe75381700 1 -- start start 2021-04-09T18:15:35.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe75381700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe700a90c0 0x7efe701a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:35.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe75381700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe701a76d0 0x7efe701a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:35.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe75381700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe7017bce0 0x7efe7017c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:35.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe75381700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe701a7c60 con 0x7efe700a90c0 2021-04-09T18:15:35.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe75381700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe701a8e20 con 0x7efe701a76d0 2021-04-09T18:15:35.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe75381700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7017c700 con 0x7efe7017bce0 2021-04-09T18:15:35.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe700a90c0 0x7efe701a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:35.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe700a90c0 0x7efe701a81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44156/0 (socket says 172.21.15.32:44156) 2021-04-09T18:15:35.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f07d700 1 -- 172.21.15.32:0/8444889 learned_addr learned my addr 172.21.15.32:0/8444889 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:35.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6e87c700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe701a76d0 0x7efe701a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f87e700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe7017bce0 0x7efe7017c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f07d700 1 -- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe7017bce0 msgr2=0x7efe7017c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:35.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f07d700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe7017bce0 0x7efe7017c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f07d700 1 -- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe701a76d0 msgr2=0x7efe701a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f07d700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe701a76d0 0x7efe701a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f07d700 1 -- 172.21.15.32:0/8444889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7017c920 con 0x7efe700a90c0 2021-04-09T18:15:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.931 7efe6f07d700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe700a90c0 0x7efe701a81e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7efe60001400 tx=0x7efe6000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.932 7efe57fff700 1 -- 172.21.15.32:0/8444889 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe60000ef0 con 0x7efe700a90c0 2021-04-09T18:15:35.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.932 7efe57fff700 1 -- 172.21.15.32:0/8444889 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6000acf0 con 0x7efe700a90c0 2021-04-09T18:15:35.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.932 7efe75381700 1 -- 172.21.15.32:0/8444889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7017c920 con 0x7efe700a90c0 2021-04-09T18:15:35.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.932 7efe57fff700 1 -- 172.21.15.32:0/8444889 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe60000ef0 con 0x7efe700a90c0 2021-04-09T18:15:35.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.932 7efe75381700 1 -- 172.21.15.32:0/8444889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7017c920 con 0x7efe700a90c0 2021-04-09T18:15:35.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.932 7efe57fff700 1 -- 172.21.15.32:0/8444889 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efe600043a0 con 0x7efe700a90c0 2021-04-09T18:15:35.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.933 7efe57fff700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe500188d0 0x7efe5001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:35.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.933 7efe57fff700 1 -- 172.21.15.32:0/8444889 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4929+0+0 (secure 0 0 0) 0x7efe60032750 con 0x7efe700a90c0 2021-04-09T18:15:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.933 7efe6e87c700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe500188d0 0x7efe5001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:35.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.933 7efe6e87c700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe500188d0 0x7efe5001add0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.953 7efe75381700 1 -- 172.21.15.32:0/8444889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe701b7010 con 0x7efe700a90c0 2021-04-09T18:15:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:34.956 7efe57fff700 1 -- 172.21.15.32:0/8444889 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efe60000ed0 con 0x7efe700a90c0 2021-04-09T18:15:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.147 7efe75381700 1 -- 172.21.15.32:0/8444889 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe7010d480 con 0x7efe700a90c0 2021-04-09T18:15:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.147 7efe57fff700 1 -- 172.21.15.32:0/8444889 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+1850 (secure 0 0 0) 0x7efe6000acf0 con 0x7efe700a90c0 2021-04-09T18:15:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 39 2021-04-09T18:15:35.200 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:35.200 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/1797590590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1797590590},{"type":"v1","addr":"172.21.15.86:6835","nonce":1797590590}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":34},{"gid":5445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2789516883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2789516883},{"type":"v1","addr":"172.21.15.32:6835","nonce":2789516883}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":39},{"gid":54800,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/631157840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":631157840},{"type":"v1","addr":"172.21.15.86:6837","nonce":631157840}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24},{"gid":55013,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3273459469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3273459469},{"type":"v1","addr":"172.21.15.32:6833","nonce":3273459469}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":39}],"filesystems":[]} 2021-04-09T18:15:35.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 -- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe500188d0 msgr2=0x7efe5001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:35.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe500188d0 0x7efe5001add0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:35.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 -- 172.21.15.32:0/8444889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe700a90c0 msgr2=0x7efe701a81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:35.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe700a90c0 0x7efe701a81e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7efe60001400 tx=0x7efe6000a900).stop 2021-04-09T18:15:35.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 -- 172.21.15.32:0/8444889 shutdown_connections 2021-04-09T18:15:35.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe500188d0 0x7efe5001add0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:35.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe7017bce0 0x7efe7017c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:35.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe701a76d0 0x7efe701a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:35.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 --2- 172.21.15.32:0/8444889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe700a90c0 0x7efe701a81e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:35.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 -- 172.21.15.32:0/8444889 >> 172.21.15.32:0/8444889 conn(0x7efe701acae0 msgr2=0x7efe7017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:35.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 -- 172.21.15.32:0/8444889 shutdown_connections 2021-04-09T18:15:35.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:35.150 7efe75381700 1 -- 172.21.15.32:0/8444889 wait complete. 2021-04-09T18:15:35.342 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:15:35.343 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:15:35.390 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:15:35.396 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:15:35.397 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:15:35.409 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:15:35.410 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:15:35.446 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:15:35.447 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:15:35.471 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:15:35.472 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:15:35.472 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:15:35.473 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:15:35.473 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:15:35.474 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:15:35.502 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:15:35.526 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:15:35.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:15:35.555 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:15:35.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:15:35.575 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:15:35.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:15:35.613 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:15:35.615 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:15:35.615 DEBUG:teuthology.parallel:result is None 2021-04-09T18:15:35.659 DEBUG:teuthology.parallel:result is None 2021-04-09T18:15:35.663 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:15:35.663 DEBUG:teuthology.parallel:result is None 2021-04-09T18:15:35.738 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:15:35.749 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:15:35.761 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:15:41.514 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:15:41.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:15:41.519 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:15:41.520 DEBUG:teuthology.parallel:result is None 2021-04-09T18:15:41.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:15:41.679 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:15:41.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.659 7f851ce80700 1 Processor -- start 2021-04-09T18:15:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.659 7f851ce80700 1 -- start start 2021-04-09T18:15:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.660 7f851ce80700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85180d6c90 0x7f85180d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.660 7f851ce80700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85180d7290 con 0x7f85180cffb0 2021-04-09T18:15:41.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.660 7f851ce80700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85180d2850 con 0x7f85180d2450 2021-04-09T18:15:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.660 7f851ce80700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85180d7090 con 0x7f85180d6c90 2021-04-09T18:15:41.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.660 7f8516b13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f85180cffb0 0x7f85180d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53778/0 (socket says 172.21.15.32:53778) 2021-04-09T18:15:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.660 7f850e312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85180d6c90 0x7f85180d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:41.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.660 7f8516b13700 1 -- 172.21.15.32:0/1218059711 learned_addr learned my addr 172.21.15.32:0/1218059711 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f850e312700 1 -- 172.21.15.32:0/1218059711 >> v1:172.21.15.86:6789/0 conn(0x7f85180d2450 legacy=0x7f851810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:41.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f850e312700 1 -- 172.21.15.32:0/1218059711 >> v1:172.21.15.32:6789/0 conn(0x7f85180cffb0 legacy=0x7f85180d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f8516312700 1 -- 172.21.15.32:0/1218059711 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245204545 0 0) 0x7f8510000d30 con 0x7f85180d2450 2021-04-09T18:15:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f850e312700 1 -- 172.21.15.32:0/1218059711 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85180d91f0 con 0x7f85180d6c90 2021-04-09T18:15:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f850e312700 1 --2- 172.21.15.32:0/1218059711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180d6c90 0x7f85180d8000 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f8500006650 tx=0x7f8500009a00).ready entity=mon.0 client_cookie=6cba64efcf89b526 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f8516312700 1 -- 172.21.15.32:0/1218059711 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8500009df0 con 0x7f85180d6c90 2021-04-09T18:15:41.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f8516312700 1 -- 172.21.15.32:0/1218059711 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85000009b0 con 0x7f85180d6c90 2021-04-09T18:15:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f8516312700 1 -- 172.21.15.32:0/1218059711 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f850000c2b0 con 0x7f85180d6c90 2021-04-09T18:15:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f851ce80700 1 -- 172.21.15.32:0/1218059711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180d6c90 msgr2=0x7f85180d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:41.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.661 7f851ce80700 1 --2- 172.21.15.32:0/1218059711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180d6c90 0x7f85180d8000 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f8500006650 tx=0x7f8500009a00).stop 2021-04-09T18:15:41.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.662 7f851ce80700 1 -- 172.21.15.32:0/1218059711 shutdown_connections 2021-04-09T18:15:41.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.662 7f851ce80700 1 --2- 172.21.15.32:0/1218059711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85180d6c90 0x7f85180d8000 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:41.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.662 7f851ce80700 1 -- 172.21.15.32:0/1218059711 >> 172.21.15.32:0/1218059711 conn(0x7f85180ce5d0 msgr2=0x7f85180cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:41.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.662 7f851ce80700 1 -- 172.21.15.32:0/1218059711 shutdown_connections 2021-04-09T18:15:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.662 7f851ce80700 1 -- 172.21.15.32:0/1218059711 wait complete. 2021-04-09T18:15:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.682 7f851ce80700 1 Processor -- start 2021-04-09T18:15:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.682 7f851ce80700 1 -- start start 2021-04-09T18:15:41.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.682 7f851ce80700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85180a7500 0x7f85181c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.682 7f851ce80700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85181c2a00 0x7f85180bb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.682 7f851ce80700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85180a7900 0x7f85180bb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.682 7f851ce80700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85180bab70 con 0x7f85181c2a00 2021-04-09T18:15:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.682 7f851ce80700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85181a7dd0 con 0x7f85180a7900 2021-04-09T18:15:41.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.682 7f851ce80700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85180b9290 con 0x7f85180a7500 2021-04-09T18:15:41.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850e312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85181c2a00 0x7f85180bb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:41.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850e312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85181c2a00 0x7f85180bb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44194/0 (socket says 172.21.15.32:44194) 2021-04-09T18:15:41.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850e312700 1 -- 172.21.15.32:0/3875068861 learned_addr learned my addr 172.21.15.32:0/3875068861 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:41.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f8517314700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85180a7900 0x7f85180bb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f8516b13700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85180a7500 0x7f85181c23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:41.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850e312700 1 -- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85180a7500 msgr2=0x7f85181c23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850e312700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85180a7500 0x7f85181c23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850e312700 1 -- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85180a7900 msgr2=0x7f85180bb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850e312700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85180a7900 0x7f85180bb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:41.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850e312700 1 -- 172.21.15.32:0/3875068861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85180b9810 con 0x7f85181c2a00 2021-04-09T18:15:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850e312700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85181c2a00 0x7f85180bb0f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f85000096a0 tx=0x7f850000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850ffff700 1 -- 172.21.15.32:0/3875068861 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8500000c90 con 0x7f85181c2a00 2021-04-09T18:15:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850ffff700 1 -- 172.21.15.32:0/3875068861 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8500002030 con 0x7f85181c2a00 2021-04-09T18:15:41.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f851ce80700 1 -- 172.21.15.32:0/3875068861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85180b9810 con 0x7f85181c2a00 2021-04-09T18:15:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f850ffff700 1 -- 172.21.15.32:0/3875068861 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8500002550 con 0x7f85181c2a00 2021-04-09T18:15:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.683 7f851ce80700 1 -- 172.21.15.32:0/3875068861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85180b9810 con 0x7f85181c2a00 2021-04-09T18:15:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.684 7f850ffff700 1 -- 172.21.15.32:0/3875068861 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8500002210 con 0x7f85181c2a00 2021-04-09T18:15:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.684 7f850ffff700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84fc018930 0x7f84fc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:41.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.684 7f850ffff700 1 -- 172.21.15.32:0/3875068861 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f8500032940 con 0x7f85181c2a00 2021-04-09T18:15:41.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.684 7f8516b13700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84fc018930 0x7f84fc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.685 7f8516b13700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84fc018930 0x7f84fc01ae10 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:41.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.704 7f851ce80700 1 -- 172.21.15.32:0/3875068861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85181a7dd0 con 0x7f85181c2a00 2021-04-09T18:15:41.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.708 7f850ffff700 1 -- 172.21.15.32:0/3875068861 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f85000009c0 con 0x7f85181c2a00 2021-04-09T18:15:41.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.909 7f851ce80700 1 -- 172.21.15.32:0/3875068861 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f85181a7dd0 con 0x7f85181c2a00 2021-04-09T18:15:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.962 7f850ffff700 1 -- 172.21.15.32:0/3875068861 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v50) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8500002030 con 0x7f85181c2a00 2021-04-09T18:15:42.001 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:15:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.974 7f851ce80700 1 -- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84fc018930 msgr2=0x7f84fc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:42.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.974 7f851ce80700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84fc018930 0x7f84fc01ae10 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.974 7f851ce80700 1 -- 172.21.15.32:0/3875068861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85181c2a00 msgr2=0x7f85180bb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.974 7f851ce80700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85181c2a00 0x7f85180bb0f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f85000096a0 tx=0x7f850000b990).stop 2021-04-09T18:15:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.975 7f851ce80700 1 -- 172.21.15.32:0/3875068861 shutdown_connections 2021-04-09T18:15:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.975 7f851ce80700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84fc018930 0x7f84fc01ae10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.975 7f851ce80700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85181c2a00 0x7f85180bb0f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.975 7f851ce80700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85180a7900 0x7f85180bb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.975 7f851ce80700 1 --2- 172.21.15.32:0/3875068861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85180a7500 0x7f85181c23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.975 7f851ce80700 1 -- 172.21.15.32:0/3875068861 >> 172.21.15.32:0/3875068861 conn(0x7f85181a80a0 msgr2=0x7f85181ab840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.975 7f851ce80700 1 -- 172.21.15.32:0/3875068861 shutdown_connections 2021-04-09T18:15:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:41.975 7f851ce80700 1 -- 172.21.15.32:0/3875068861 wait complete. 2021-04-09T18:15:42.030 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:15:42.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:15:42.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.174 7f6c5f0a5700 1 Processor -- start 2021-04-09T18:15:42.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.174 7f6c5f0a5700 1 -- start start 2021-04-09T18:15:42.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.174 7f6c5f0a5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c5810d010 0x7f6c580d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.174 7f6c5f0a5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5810d480 con 0x7f6c580d8490 2021-04-09T18:15:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.174 7f6c5f0a5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5810e1f0 con 0x7f6c580d1da0 2021-04-09T18:15:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.174 7f6c5f0a5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c580d8d30 con 0x7f6c5810d010 2021-04-09T18:15:42.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5cec1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c5810d010 0x7f6c580d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:42.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5d6c2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6c580d8490 0x7f6c5810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53790/0 (socket says 172.21.15.32:53790) 2021-04-09T18:15:42.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5cec1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c5810d010 0x7f6c580d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44200/0 (socket says 172.21.15.32:44200) 2021-04-09T18:15:42.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5d6c2700 1 -- 172.21.15.32:0/2357182537 learned_addr learned my addr 172.21.15.32:0/2357182537 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:42.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5cec1700 1 -- 172.21.15.32:0/2357182537 >> v1:172.21.15.86:6789/0 conn(0x7f6c580d1da0 legacy=0x7f6c580d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:42.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5cec1700 1 -- 172.21.15.32:0/2357182537 >> v1:172.21.15.32:6789/0 conn(0x7f6c580d8490 legacy=0x7f6c5810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:42.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c577fe700 1 -- 172.21.15.32:0/2357182537 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814295292 0 0) 0x7f6c50000d30 con 0x7f6c580d8490 2021-04-09T18:15:42.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5cec1700 1 -- 172.21.15.32:0/2357182537 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c580d9220 con 0x7f6c5810d010 2021-04-09T18:15:42.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5cec1700 1 --2- 172.21.15.32:0/2357182537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c5810d010 0x7f6c580d1780 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f6c40006650 tx=0x7f6c40009a80).ready entity=mon.0 client_cookie=f70e0e1c9b4c794a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:42.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c577fe700 1 -- 172.21.15.32:0/2357182537 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c400009b0 con 0x7f6c5810d010 2021-04-09T18:15:42.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c577fe700 1 -- 172.21.15.32:0/2357182537 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4000c0c0 con 0x7f6c5810d010 2021-04-09T18:15:42.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c577fe700 1 -- 172.21.15.32:0/2357182537 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c4000c510 con 0x7f6c5810d010 2021-04-09T18:15:42.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5f0a5700 1 -- 172.21.15.32:0/2357182537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c5810d010 msgr2=0x7f6c580d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:42.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.175 7f6c5f0a5700 1 --2- 172.21.15.32:0/2357182537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c5810d010 0x7f6c580d1780 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f6c40006650 tx=0x7f6c40009a80).stop 2021-04-09T18:15:42.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.176 7f6c5f0a5700 1 -- 172.21.15.32:0/2357182537 shutdown_connections 2021-04-09T18:15:42.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.176 7f6c5f0a5700 1 --2- 172.21.15.32:0/2357182537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c5810d010 0x7f6c580d1780 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.176 7f6c5f0a5700 1 -- 172.21.15.32:0/2357182537 >> 172.21.15.32:0/2357182537 conn(0x7f6c580ce5f0 msgr2=0x7f6c580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:42.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.176 7f6c5f0a5700 1 -- 172.21.15.32:0/2357182537 shutdown_connections 2021-04-09T18:15:42.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.176 7f6c5f0a5700 1 -- 172.21.15.32:0/2357182537 wait complete. 2021-04-09T18:15:42.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.196 7f6c5f0a5700 1 Processor -- start 2021-04-09T18:15:42.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.196 7f6c5f0a5700 1 -- start start 2021-04-09T18:15:42.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5f0a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c581a9a50 0x7f6c5808a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:42.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5f0a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c5808afb0 0x7f6c5809d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:42.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5f0a5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c5809ddf0 0x7f6c58183b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:42.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5f0a5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c581a9e50 con 0x7f6c581a9a50 2021-04-09T18:15:42.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5f0a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5809e410 con 0x7f6c5808afb0 2021-04-09T18:15:42.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5f0a5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c581841a0 con 0x7f6c5809ddf0 2021-04-09T18:15:42.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5cec1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c581a9a50 0x7f6c5808a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:42.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5cec1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c581a9a50 0x7f6c5808a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44204/0 (socket says 172.21.15.32:44204) 2021-04-09T18:15:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5cec1700 1 -- 172.21.15.32:0/2213480737 learned_addr learned my addr 172.21.15.32:0/2213480737 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c57fff700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c5808afb0 0x7f6c5809d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:42.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5d6c2700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c5809ddf0 0x7f6c58183b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:42.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5cec1700 1 -- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c5809ddf0 msgr2=0x7f6c58183b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5cec1700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c5809ddf0 0x7f6c58183b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5cec1700 1 -- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c5808afb0 msgr2=0x7f6c5809d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5cec1700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c5808afb0 0x7f6c5809d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5cec1700 1 -- 172.21.15.32:0/2213480737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c58184380 con 0x7f6c581a9a50 2021-04-09T18:15:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c57fff700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c5808afb0 0x7f6c5809d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:42.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5d6c2700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c5809ddf0 0x7f6c58183b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.197 7f6c5cec1700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c581a9a50 0x7f6c5808a990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f6c40009650 tx=0x7f6c4000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.198 7f6c557fa700 1 -- 172.21.15.32:0/2213480737 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c400093a0 con 0x7f6c581a9a50 2021-04-09T18:15:42.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.198 7f6c557fa700 1 -- 172.21.15.32:0/2213480737 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4000bd50 con 0x7f6c581a9a50 2021-04-09T18:15:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.198 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c58184380 con 0x7f6c581a9a50 2021-04-09T18:15:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.198 7f6c557fa700 1 -- 172.21.15.32:0/2213480737 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c400052e0 con 0x7f6c581a9a50 2021-04-09T18:15:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.198 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c58184380 con 0x7f6c581a9a50 2021-04-09T18:15:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.198 7f6c557fa700 1 -- 172.21.15.32:0/2213480737 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6c40006030 con 0x7f6c581a9a50 2021-04-09T18:15:42.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.198 7f6c557fa700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c3c018930 0x7f6c3c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:42.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.198 7f6c557fa700 1 -- 172.21.15.32:0/2213480737 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f6c400326b0 con 0x7f6c581a9a50 2021-04-09T18:15:42.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.199 7f6c57fff700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c3c018930 0x7f6c3c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:42.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.199 7f6c57fff700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c3c018930 0x7f6c3c01ae10 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:42.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.218 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5810d480 con 0x7f6c581a9a50 2021-04-09T18:15:42.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.222 7f6c557fa700 1 -- 172.21.15.32:0/2213480737 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6c400009c0 con 0x7f6c581a9a50 2021-04-09T18:15:42.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.408 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6c5810d480 con 0x7f6c581a9a50 2021-04-09T18:15:42.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.409 7f6c557fa700 1 -- 172.21.15.32:0/2213480737 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f6c4000bd50 con 0x7f6c581a9a50 2021-04-09T18:15:42.438 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:42.439 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:15:42.439 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:15:42.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.439 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:15:42.440 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:15:42.440 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.440 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:15:42.441 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.441 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.441 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.441 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.442 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:15:42.442 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:15:42.443 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.443 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:15:42.443 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.443 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.444 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.445 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:15:42.445 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:15:42.445 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.446 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:15:42.446 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.446 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.446 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.446 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.447 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:15:42.447 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:15:42.448 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.448 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:15:42.448 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.448 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.448 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.449 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.449 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:15:42.449 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:15:42.450 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.507 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.512 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.514 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:15:42.515 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:15:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.520 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.529 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.530 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:15:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:15:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:15:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:15:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:15:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:15:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.581 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.582 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.606 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:15:42.621 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:15:42.624 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.624 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.626 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.634 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.655 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:15:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:15:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.656 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:15:42.657 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:15:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:15:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:15:42.673 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.705 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:15:42.706 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:15:42.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:15:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:15:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:15:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:15:42.750 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:15:42.750 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:15:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.810 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:15:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:15:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.820 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:15:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:15:42.821 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:15:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.822 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.847 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:15:42.850 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:15:42.851 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:15:42.852 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:15:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:15:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:15:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:15:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:15:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:15:42.856 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:15:42.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:42.878 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:15:42.884 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:15:42.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c3c018930 msgr2=0x7f6c3c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c3c018930 0x7f6c3c01ae10 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c581a9a50 msgr2=0x7f6c5808a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:42.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c581a9a50 0x7f6c5808a990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f6c40009650 tx=0x7f6c4000b960).stop 2021-04-09T18:15:42.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 shutdown_connections 2021-04-09T18:15:42.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c3c018930 0x7f6c3c01ae10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c5809ddf0 0x7f6c58183b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c5808afb0 0x7f6c5809d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 --2- 172.21.15.32:0/2213480737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c581a9a50 0x7f6c5808a990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:42.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 >> 172.21.15.32:0/2213480737 conn(0x7f6c581965c0 msgr2=0x7f6c581ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:42.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 shutdown_connections 2021-04-09T18:15:42.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:42.411 7f6c5f0a5700 1 -- 172.21.15.32:0/2213480737 wait complete. 2021-04-09T18:15:42.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:15:43.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.084 7f095e02b700 1 Processor -- start 2021-04-09T18:15:43.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.084 7f095e02b700 1 -- start start 2021-04-09T18:15:43.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f095e02b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f095810d010 0x7f09580d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f095e02b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f09580d8490 0x7f095810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f095e02b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095810d480 con 0x7f09580d1da0 2021-04-09T18:15:43.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f095e02b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09580d8d30 con 0x7f09580d8490 2021-04-09T18:15:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f095e02b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09580d9220 con 0x7f095810d010 2021-04-09T18:15:43.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f0957fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f09580d8490 0x7f095810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f0956ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f09580d1da0 0x7f09580d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53806/0 (socket says 172.21.15.32:53806) 2021-04-09T18:15:43.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f0956ffd700 1 -- 172.21.15.32:0/2368735748 learned_addr learned my addr 172.21.15.32:0/2368735748 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:43.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f09577fe700 1 --2- 172.21.15.32:0/2368735748 >> v2:172.21.15.86:3300/0 conn(0x7f095810d010 0x7f09580d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f09567fc700 1 -- 172.21.15.32:0/2368735748 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152247638 0 0) 0x7f09400009b0 con 0x7f09580d1da0 2021-04-09T18:15:43.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f0957fff700 1 -- 172.21.15.32:0/2368735748 >> v2:172.21.15.86:3300/0 conn(0x7f095810d010 msgr2=0x7f09580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:43.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f0957fff700 1 --2- 172.21.15.32:0/2368735748 >> v2:172.21.15.86:3300/0 conn(0x7f095810d010 0x7f09580d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f0957fff700 1 -- 172.21.15.32:0/2368735748 >> v1:172.21.15.32:6789/0 conn(0x7f09580d1da0 legacy=0x7f09580d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f0957fff700 1 -- 172.21.15.32:0/2368735748 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095810e020 con 0x7f09580d8490 2021-04-09T18:15:43.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.085 7f09577fe700 1 --2- 172.21.15.32:0/2368735748 >> v2:172.21.15.86:3300/0 conn(0x7f095810d010 0x7f09580d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:43.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f0957fff700 1 --2- 172.21.15.32:0/2368735748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09580d8490 0x7f095810da00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f0950006e30 tx=0x7f0950009e60).ready entity=mon.0 client_cookie=892b12e7b12245a9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:43.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f09567fc700 1 -- 172.21.15.32:0/2368735748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0950006a50 con 0x7f09580d8490 2021-04-09T18:15:43.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f09567fc700 1 -- 172.21.15.32:0/2368735748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095000b300 con 0x7f09580d8490 2021-04-09T18:15:43.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f09567fc700 1 -- 172.21.15.32:0/2368735748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f095000b750 con 0x7f09580d8490 2021-04-09T18:15:43.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f095e02b700 1 -- 172.21.15.32:0/2368735748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09580d8490 msgr2=0x7f095810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f095e02b700 1 --2- 172.21.15.32:0/2368735748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09580d8490 0x7f095810da00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f0950006e30 tx=0x7f0950009e60).stop 2021-04-09T18:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f095e02b700 1 -- 172.21.15.32:0/2368735748 shutdown_connections 2021-04-09T18:15:43.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f095e02b700 1 --2- 172.21.15.32:0/2368735748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09580d8490 0x7f095810da00 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f095e02b700 1 --2- 172.21.15.32:0/2368735748 >> v2:172.21.15.86:3300/0 conn(0x7f095810d010 0x7f09580d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f095e02b700 1 -- 172.21.15.32:0/2368735748 >> 172.21.15.32:0/2368735748 conn(0x7f09580ce5f0 msgr2=0x7f09580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:43.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f095e02b700 1 -- 172.21.15.32:0/2368735748 shutdown_connections 2021-04-09T18:15:43.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.086 7f095e02b700 1 -- 172.21.15.32:0/2368735748 wait complete. 2021-04-09T18:15:43.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.107 7f095e02b700 1 Processor -- start 2021-04-09T18:15:43.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.107 7f095e02b700 1 -- start start 2021-04-09T18:15:43.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.107 7f095e02b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09581a9360 0x7f09581918f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.107 7f095e02b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f095818c440 0x7f095818c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.107 7f095e02b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09581a80f0 0x7f09581a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.107 7f095e02b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0958191f10 con 0x7f095818c440 2021-04-09T18:15:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.107 7f095e02b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09581a9760 con 0x7f09581a9360 2021-04-09T18:15:43.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.107 7f095e02b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095818ce60 con 0x7f09581a80f0 2021-04-09T18:15:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0956ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f095818c440 0x7f095818c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0956ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f095818c440 0x7f095818c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44218/0 (socket says 172.21.15.32:44218) 2021-04-09T18:15:43.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0956ffd700 1 -- 172.21.15.32:0/1708574646 learned_addr learned my addr 172.21.15.32:0/1708574646 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:43.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0957fff700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09581a80f0 0x7f09581a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f09577fe700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09581a9360 0x7f09581918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0956ffd700 1 -- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09581a80f0 msgr2=0x7f09581a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0956ffd700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09581a80f0 0x7f09581a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0956ffd700 1 -- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09581a9360 msgr2=0x7f09581918f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0956ffd700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09581a9360 0x7f09581918f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0956ffd700 1 -- 172.21.15.32:0/1708574646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09581a8b10 con 0x7f095818c440 2021-04-09T18:15:43.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f09577fe700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09581a9360 0x7f09581918f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:43.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0957fff700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09581a80f0 0x7f09581a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:43.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f0956ffd700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f095818c440 0x7f095818c840 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f0940001c60 tx=0x7f094000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:43.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f094ffff700 1 -- 172.21.15.32:0/1708574646 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f094000acc0 con 0x7f095818c440 2021-04-09T18:15:43.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f094ffff700 1 -- 172.21.15.32:0/1708574646 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0940004140 con 0x7f095818c440 2021-04-09T18:15:43.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f094ffff700 1 -- 172.21.15.32:0/1708574646 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0940004620 con 0x7f095818c440 2021-04-09T18:15:43.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f095e02b700 1 -- 172.21.15.32:0/1708574646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09581a8b10 con 0x7f095818c440 2021-04-09T18:15:43.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.108 7f095e02b700 1 -- 172.21.15.32:0/1708574646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09581a8b10 con 0x7f095818c440 2021-04-09T18:15:43.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.109 7f094ffff700 1 -- 172.21.15.32:0/1708574646 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0940000dd0 con 0x7f095818c440 2021-04-09T18:15:43.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.109 7f094ffff700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f093c018930 0x7f093c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.109 7f094ffff700 1 -- 172.21.15.32:0/1708574646 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f09400326f0 con 0x7f095818c440 2021-04-09T18:15:43.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.109 7f09577fe700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f093c018930 0x7f093c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.110 7f09577fe700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f093c018930 0x7f093c01ae10 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:43.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.129 7f095e02b700 1 -- 172.21.15.32:0/1708574646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09581a9760 con 0x7f095818c440 2021-04-09T18:15:43.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.132 7f094ffff700 1 -- 172.21.15.32:0/1708574646 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f09400009b0 con 0x7f095818c440 2021-04-09T18:15:43.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.318 7f095e02b700 1 -- 172.21.15.32:0/1708574646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f095818bf60 con 0x7f095818c440 2021-04-09T18:15:43.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.319 7f094ffff700 1 -- 172.21.15.32:0/1708574646 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+11144 (secure 0 0 0) 0x7f0940000dd0 con 0x7f095818c440 2021-04-09T18:15:43.358 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:43.358 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":50,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:15:41.960883","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:15:43.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 -- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f093c018930 msgr2=0x7f093c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f093c018930 0x7f093c01ae10 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 -- 172.21.15.32:0/1708574646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f095818c440 msgr2=0x7f095818c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f095818c440 0x7f095818c840 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f0940001c60 tx=0x7f094000a8d0).stop 2021-04-09T18:15:43.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 -- 172.21.15.32:0/1708574646 shutdown_connections 2021-04-09T18:15:43.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f093c018930 0x7f093c01ae10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f095818c440 0x7f095818c840 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09581a80f0 0x7f09581a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 --2- 172.21.15.32:0/1708574646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09581a9360 0x7f09581918f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 -- 172.21.15.32:0/1708574646 >> 172.21.15.32:0/1708574646 conn(0x7f09581837d0 msgr2=0x7f0958195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:43.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 -- 172.21.15.32:0/1708574646 shutdown_connections 2021-04-09T18:15:43.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.321 7f095e02b700 1 -- 172.21.15.32:0/1708574646 wait complete. 2021-04-09T18:15:43.431 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:15:43.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:15:43.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65d2e30700 1 Processor -- start 2021-04-09T18:15:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65d2e30700 1 -- start start 2021-04-09T18:15:43.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65d2e30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc10d010 0x7f65cc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65d2e30700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65cc0d1da0 0x7f65cc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65d2e30700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f65cc0d82c0 0x7f65cc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65d2e30700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc10e780 con 0x7f65cc0d1da0 2021-04-09T18:15:43.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65d2e30700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc0d8d30 con 0x7f65cc0d82c0 2021-04-09T18:15:43.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65d2e30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0d9220 con 0x7f65cc10d010 2021-04-09T18:15:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65cbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65cc0d1da0 0x7f65cc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65cbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65cc0d1da0 0x7f65cc0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44226/0 (socket says 172.21.15.32:44226) 2021-04-09T18:15:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.585 7f65cbfff700 1 -- 172.21.15.32:0/2644342277 learned_addr learned my addr 172.21.15.32:0/2644342277 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65d0c4c700 1 --2- 172.21.15.32:0/2644342277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc10d010 0x7f65cc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65d144d700 1 --2- 172.21.15.32:0/2644342277 >> v2:172.21.15.86:3300/0 conn(0x7f65cc0d82c0 0x7f65cc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65cbfff700 1 -- 172.21.15.32:0/2644342277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc10d010 msgr2=0x7f65cc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65cbfff700 1 --2- 172.21.15.32:0/2644342277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc10d010 0x7f65cc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65cbfff700 1 -- 172.21.15.32:0/2644342277 >> v2:172.21.15.86:3300/0 conn(0x7f65cc0d82c0 msgr2=0x7f65cc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65cbfff700 1 --2- 172.21.15.32:0/2644342277 >> v2:172.21.15.86:3300/0 conn(0x7f65cc0d82c0 0x7f65cc10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65cbfff700 1 -- 172.21.15.32:0/2644342277 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc10d480 con 0x7f65cc0d1da0 2021-04-09T18:15:43.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65cbfff700 1 --2- 172.21.15.32:0/2644342277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0d1da0 0x7f65cc0d7ca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f65bc006700 tx=0x7f65bc009b00).ready entity=mon.0 client_cookie=70affd92a1a489a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65cb7fe700 1 -- 172.21.15.32:0/2644342277 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65bc00c0c0 con 0x7f65cc0d1da0 2021-04-09T18:15:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65cb7fe700 1 -- 172.21.15.32:0/2644342277 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65bc00c400 con 0x7f65cc0d1da0 2021-04-09T18:15:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65cb7fe700 1 -- 172.21.15.32:0/2644342277 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65bc00c8e0 con 0x7f65cc0d1da0 2021-04-09T18:15:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65d2e30700 1 -- 172.21.15.32:0/2644342277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0d1da0 msgr2=0x7f65cc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.586 7f65d2e30700 1 --2- 172.21.15.32:0/2644342277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0d1da0 0x7f65cc0d7ca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f65bc006700 tx=0x7f65bc009b00).stop 2021-04-09T18:15:43.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.587 7f65d2e30700 1 -- 172.21.15.32:0/2644342277 shutdown_connections 2021-04-09T18:15:43.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.587 7f65d2e30700 1 --2- 172.21.15.32:0/2644342277 >> v2:172.21.15.86:3300/0 conn(0x7f65cc0d82c0 0x7f65cc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.587 7f65d2e30700 1 --2- 172.21.15.32:0/2644342277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0d1da0 0x7f65cc0d7ca0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.587 7f65d2e30700 1 --2- 172.21.15.32:0/2644342277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc10d010 0x7f65cc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.587 7f65d2e30700 1 -- 172.21.15.32:0/2644342277 >> 172.21.15.32:0/2644342277 conn(0x7f65cc0ce5f0 msgr2=0x7f65cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.587 7f65d2e30700 1 -- 172.21.15.32:0/2644342277 shutdown_connections 2021-04-09T18:15:43.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.587 7f65d2e30700 1 -- 172.21.15.32:0/2644342277 wait complete. 2021-04-09T18:15:43.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.607 7f65d2e30700 1 Processor -- start 2021-04-09T18:15:43.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.607 7f65d2e30700 1 -- start start 2021-04-09T18:15:43.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65d2e30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc1a0050 0x7f65cc1a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65d2e30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0bdcd0 0x7f65cc0bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65d2e30700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc0bf080 0x7f65cc0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65d2e30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0bf500 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65d2e30700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0bb7b0 con 0x7f65cc0bf080 2021-04-09T18:15:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65d2e30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0be0d0 con 0x7f65cc1a0050 2021-04-09T18:15:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0bdcd0 0x7f65cc0bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0bdcd0 0x7f65cc0bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44234/0 (socket says 172.21.15.32:44234) 2021-04-09T18:15:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65cbfff700 1 -- 172.21.15.32:0/4069211486 learned_addr learned my addr 172.21.15.32:0/4069211486 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65d0c4c700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc1a0050 0x7f65cc1a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65d144d700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc0bf080 0x7f65cc0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65cbfff700 1 -- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc1a0050 msgr2=0x7f65cc1a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65cbfff700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc1a0050 0x7f65cc1a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65cbfff700 1 -- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc0bf080 msgr2=0x7f65cc0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65cbfff700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc0bf080 0x7f65cc0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65cbfff700 1 -- 172.21.15.32:0/4069211486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc0b1310 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.608 7f65cbfff700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0bdcd0 0x7f65cc0bea60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f65bc006f20 tx=0x7f65bc00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:43.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.609 7f65c97fa700 1 -- 172.21.15.32:0/4069211486 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65bc00bdd0 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.609 7f65c97fa700 1 -- 172.21.15.32:0/4069211486 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65bc006440 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.609 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65cc0b1310 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.609 7f65c97fa700 1 -- 172.21.15.32:0/4069211486 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65bc0052e0 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.609 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc0b1310 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.609 7f65c97fa700 1 -- 172.21.15.32:0/4069211486 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f65bc0052e0 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.609 7f65c97fa700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b0018930 0x7f65b001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.609 7f65c97fa700 1 -- 172.21.15.32:0/4069211486 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f65bc0326d0 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.610 7f65d0c4c700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b0018930 0x7f65b001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:43.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.610 7f65d0c4c700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b0018930 0x7f65b001ae10 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.630 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc093d50 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.633 7f65c97fa700 1 -- 172.21.15.32:0/4069211486 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f65bc006440 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.815 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f65cc10d480 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.815 7f65c97fa700 1 -- 172.21.15.32:0/4069211486 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v50) v1 ==== 92+0+10 (secure 0 0 0) 0x7f65bc00bdd0 con 0x7f65cc0bdcd0 2021-04-09T18:15:43.844 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:15:43.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.817 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b0018930 msgr2=0x7f65b001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.817 7f65d2e30700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b0018930 0x7f65b001ae10 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.817 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0bdcd0 msgr2=0x7f65cc0bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.817 7f65d2e30700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0bdcd0 0x7f65cc0bea60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f65bc006f20 tx=0x7f65bc00b9e0).stop 2021-04-09T18:15:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.818 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 shutdown_connections 2021-04-09T18:15:43.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.818 7f65d2e30700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b0018930 0x7f65b001ae10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.818 7f65d2e30700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc0bdcd0 0x7f65cc0bea60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.818 7f65d2e30700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc0bf080 0x7f65cc0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.818 7f65d2e30700 1 --2- 172.21.15.32:0/4069211486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc1a0050 0x7f65cc1a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:43.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.818 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 >> 172.21.15.32:0/4069211486 conn(0x7f65cc0975f0 msgr2=0x7f65cc096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:43.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.818 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 shutdown_connections 2021-04-09T18:15:43.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:43.818 7f65d2e30700 1 -- 172.21.15.32:0/4069211486 wait complete. 2021-04-09T18:15:43.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:15:44.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.016 7f7635ba2700 1 Processor -- start 2021-04-09T18:15:44.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.016 7f7635ba2700 1 -- start start 2021-04-09T18:15:44.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f7635ba2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f763010d010 0x7f76300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f7635ba2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f76300d1da0 0x7f76300d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f7635ba2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76300d82c0 0x7f763010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f7635ba2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763010e780 con 0x7f76300d82c0 2021-04-09T18:15:44.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f7635ba2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76300d8d30 con 0x7f76300d1da0 2021-04-09T18:15:44.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f7635ba2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76300d9220 con 0x7f763010d010 2021-04-09T18:15:44.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76300d82c0 0x7f763010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76300d82c0 0x7f763010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44242/0 (socket says 172.21.15.32:44242) 2021-04-09T18:15:44.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762ffff700 1 -- 172.21.15.32:0/3768106705 learned_addr learned my addr 172.21.15.32:0/3768106705 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:44.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762effd700 1 --2- 172.21.15.32:0/3768106705 >> v2:172.21.15.86:3300/0 conn(0x7f76300d1da0 0x7f76300d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762f7fe700 1 --2- 172.21.15.32:0/3768106705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f763010d010 0x7f76300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762ffff700 1 -- 172.21.15.32:0/3768106705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f763010d010 msgr2=0x7f76300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:44.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762ffff700 1 --2- 172.21.15.32:0/3768106705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f763010d010 0x7f76300d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762ffff700 1 -- 172.21.15.32:0/3768106705 >> v2:172.21.15.86:3300/0 conn(0x7f76300d1da0 msgr2=0x7f76300d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:44.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762ffff700 1 --2- 172.21.15.32:0/3768106705 >> v2:172.21.15.86:3300/0 conn(0x7f76300d1da0 0x7f76300d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762ffff700 1 -- 172.21.15.32:0/3768106705 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763010d480 con 0x7f76300d82c0 2021-04-09T18:15:44.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.017 7f762effd700 1 --2- 172.21.15.32:0/3768106705 >> v2:172.21.15.86:3300/0 conn(0x7f76300d1da0 0x7f76300d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:44.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.018 7f762ffff700 1 --2- 172.21.15.32:0/3768106705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76300d82c0 0x7f763010da00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f7628006f00 tx=0x7f7628009e90).ready entity=mon.0 client_cookie=137617b70e45d583 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:44.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.018 7f762e7fc700 1 -- 172.21.15.32:0/3768106705 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f762800b300 con 0x7f76300d82c0 2021-04-09T18:15:44.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.018 7f762e7fc700 1 -- 172.21.15.32:0/3768106705 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762800b640 con 0x7f76300d82c0 2021-04-09T18:15:44.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.018 7f762e7fc700 1 -- 172.21.15.32:0/3768106705 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f762800bb20 con 0x7f76300d82c0 2021-04-09T18:15:44.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.018 7f7635ba2700 1 -- 172.21.15.32:0/3768106705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76300d82c0 msgr2=0x7f763010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.018 7f7635ba2700 1 --2- 172.21.15.32:0/3768106705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76300d82c0 0x7f763010da00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f7628006f00 tx=0x7f7628009e90).stop 2021-04-09T18:15:44.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.019 7f7635ba2700 1 -- 172.21.15.32:0/3768106705 shutdown_connections 2021-04-09T18:15:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.019 7f7635ba2700 1 --2- 172.21.15.32:0/3768106705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76300d82c0 0x7f763010da00 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.019 7f7635ba2700 1 --2- 172.21.15.32:0/3768106705 >> v2:172.21.15.86:3300/0 conn(0x7f76300d1da0 0x7f76300d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.019 7f7635ba2700 1 --2- 172.21.15.32:0/3768106705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f763010d010 0x7f76300d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.019 7f7635ba2700 1 -- 172.21.15.32:0/3768106705 >> 172.21.15.32:0/3768106705 conn(0x7f76300ce5f0 msgr2=0x7f76300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:44.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.019 7f7635ba2700 1 -- 172.21.15.32:0/3768106705 shutdown_connections 2021-04-09T18:15:44.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.019 7f7635ba2700 1 -- 172.21.15.32:0/3768106705 wait complete. 2021-04-09T18:15:44.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.039 7f7635ba2700 1 Processor -- start 2021-04-09T18:15:44.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.039 7f7635ba2700 1 -- start start 2021-04-09T18:15:44.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.039 7f7635ba2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76301a2e80 0x7f763008d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.039 7f7635ba2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f763008dfc0 0x7f7630090920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.039 7f7635ba2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7630090f40 0x7f763008c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.039 7f7635ba2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7630096ce0 con 0x7f7630090f40 2021-04-09T18:15:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.039 7f7635ba2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7630091340 con 0x7f763008dfc0 2021-04-09T18:15:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.039 7f7635ba2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7630091560 con 0x7f76301a2e80 2021-04-09T18:15:44.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7630090f40 0x7f763008c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7630090f40 0x7f763008c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44250/0 (socket says 172.21.15.32:44250) 2021-04-09T18:15:44.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762ffff700 1 -- 172.21.15.32:0/3500342630 learned_addr learned my addr 172.21.15.32:0/3500342630 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:44.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762f7fe700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76301a2e80 0x7f763008d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762effd700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f763008dfc0 0x7f7630090920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762ffff700 1 -- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76301a2e80 msgr2=0x7f763008d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762ffff700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76301a2e80 0x7f763008d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762ffff700 1 -- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f763008dfc0 msgr2=0x7f7630090920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762ffff700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f763008dfc0 0x7f7630090920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762ffff700 1 -- 172.21.15.32:0/3500342630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763008ce80 con 0x7f7630090f40 2021-04-09T18:15:44.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f762ffff700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7630090f40 0x7f763008c860 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f762800b8e0 tx=0x7f762800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:44.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f7627fff700 1 -- 172.21.15.32:0/3500342630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f762800bdc0 con 0x7f7630090f40 2021-04-09T18:15:44.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f7627fff700 1 -- 172.21.15.32:0/3500342630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7628005820 con 0x7f7630090f40 2021-04-09T18:15:44.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f7627fff700 1 -- 172.21.15.32:0/3500342630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7628005d00 con 0x7f7630090f40 2021-04-09T18:15:44.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763008ce80 con 0x7f7630090f40 2021-04-09T18:15:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.040 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763008ce80 con 0x7f7630090f40 2021-04-09T18:15:44.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.041 7f7627fff700 1 -- 172.21.15.32:0/3500342630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7628005a00 con 0x7f7630090f40 2021-04-09T18:15:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.041 7f7627fff700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f760c018930 0x7f760c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.041 7f7627fff700 1 -- 172.21.15.32:0/3500342630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f7628032a70 con 0x7f7630090f40 2021-04-09T18:15:44.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.041 7f762f7fe700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f760c018930 0x7f760c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.042 7f762f7fe700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f760c018930 0x7f760c01adc0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:44.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.061 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76300f1820 con 0x7f7630090f40 2021-04-09T18:15:44.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.064 7f7627fff700 1 -- 172.21.15.32:0/3500342630 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7628005d00 con 0x7f7630090f40 2021-04-09T18:15:44.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.242 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f763010d480 con 0x7f7630090f40 2021-04-09T18:15:44.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.242 7f7627fff700 1 -- 172.21.15.32:0/3500342630 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v50) v1 ==== 84+0+64 (secure 0 0 0) 0x7f7628000f90 con 0x7f7630090f40 2021-04-09T18:15:44.310 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:44.311 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:15:44.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:15:44.312 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:15:44.312 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:15:44.312 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:44.313 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:15:44.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f760c018930 msgr2=0x7f760c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f760c018930 0x7f760c01adc0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7630090f40 msgr2=0x7f763008c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7630090f40 0x7f763008c860 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f762800b8e0 tx=0x7f762800bc80).stop 2021-04-09T18:15:44.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 shutdown_connections 2021-04-09T18:15:44.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f760c018930 0x7f760c01adc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7630090f40 0x7f763008c860 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f763008dfc0 0x7f7630090920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 --2- 172.21.15.32:0/3500342630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76301a2e80 0x7f763008d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 >> 172.21.15.32:0/3500342630 conn(0x7f7630192580 msgr2=0x7f76300f1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:44.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 shutdown_connections 2021-04-09T18:15:44.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.245 7f7635ba2700 1 -- 172.21.15.32:0/3500342630 wait complete. 2021-04-09T18:15:44.369 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:15:44.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:15:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.510 7f9c30c0b700 1 Processor -- start 2021-04-09T18:15:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.510 7f9c30c0b700 1 -- start start 2021-04-09T18:15:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.511 7f9c30c0b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c2c10d010 0x7f9c2c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.511 7f9c30c0b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c2c10d480 con 0x7f9c2c0d8490 2021-04-09T18:15:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.511 7f9c30c0b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c2c10e1f0 con 0x7f9c2c0d1da0 2021-04-09T18:15:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.511 7f9c30c0b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c0d8d30 con 0x7f9c2c10d010 2021-04-09T18:15:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.511 7f9c2a85e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c2c10d010 0x7f9c2c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.511 7f9c2b05f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9c2c0d8490 0x7f9c2c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53846/0 (socket says 172.21.15.32:53846) 2021-04-09T18:15:44.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.511 7f9c2b05f700 1 -- 172.21.15.32:0/656334159 learned_addr learned my addr 172.21.15.32:0/656334159 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c2a85e700 1 -- 172.21.15.32:0/656334159 >> v1:172.21.15.86:6789/0 conn(0x7f9c2c0d1da0 legacy=0x7f9c2c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c2a85e700 1 -- 172.21.15.32:0/656334159 >> v1:172.21.15.32:6789/0 conn(0x7f9c2c0d8490 legacy=0x7f9c2c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c2a85e700 1 -- 172.21.15.32:0/656334159 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c0d9220 con 0x7f9c2c10d010 2021-04-09T18:15:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c2985c700 1 -- 172.21.15.32:0/656334159 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006977817 0 0) 0x7f9c24000d30 con 0x7f9c2c0d8490 2021-04-09T18:15:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c2a85e700 1 --2- 172.21.15.32:0/656334159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c10d010 0x7f9c2c0d1780 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f9c1c006650 tx=0x7f9c1c009a80).ready entity=mon.0 client_cookie=4d871b080be43e6f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c2985c700 1 -- 172.21.15.32:0/656334159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c1c0009b0 con 0x7f9c2c10d010 2021-04-09T18:15:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c2985c700 1 -- 172.21.15.32:0/656334159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c00c0c0 con 0x7f9c2c10d010 2021-04-09T18:15:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c2985c700 1 -- 172.21.15.32:0/656334159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c1c00c510 con 0x7f9c2c10d010 2021-04-09T18:15:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c30c0b700 1 -- 172.21.15.32:0/656334159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c10d010 msgr2=0x7f9c2c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.512 7f9c30c0b700 1 --2- 172.21.15.32:0/656334159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c10d010 0x7f9c2c0d1780 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f9c1c006650 tx=0x7f9c1c009a80).stop 2021-04-09T18:15:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.513 7f9c30c0b700 1 -- 172.21.15.32:0/656334159 shutdown_connections 2021-04-09T18:15:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.513 7f9c30c0b700 1 --2- 172.21.15.32:0/656334159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c10d010 0x7f9c2c0d1780 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.513 7f9c30c0b700 1 -- 172.21.15.32:0/656334159 >> 172.21.15.32:0/656334159 conn(0x7f9c2c0ce5f0 msgr2=0x7f9c2c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.513 7f9c30c0b700 1 -- 172.21.15.32:0/656334159 shutdown_connections 2021-04-09T18:15:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.513 7f9c30c0b700 1 -- 172.21.15.32:0/656334159 wait complete. 2021-04-09T18:15:44.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.533 7f9c30c0b700 1 Processor -- start 2021-04-09T18:15:44.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.533 7f9c30c0b700 1 -- start start 2021-04-09T18:15:44.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c30c0b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c2c17ff00 0x7f9c2c090a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c30c0b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c2c0acef0 0x7f9c2c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c30c0b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c19b0f0 0x7f9c2c0ab890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c30c0b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c17f980 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c30c0b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c0aee20 con 0x7f9c2c17ff00 2021-04-09T18:15:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c30c0b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c091080 con 0x7f9c2c0acef0 2021-04-09T18:15:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2b05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c19b0f0 0x7f9c2c0ab890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2b05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c19b0f0 0x7f9c2c0ab890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44264/0 (socket says 172.21.15.32:44264) 2021-04-09T18:15:44.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2b05f700 1 -- 172.21.15.32:0/1886249293 learned_addr learned my addr 172.21.15.32:0/1886249293 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:44.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2a85e700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c2c17ff00 0x7f9c2c090a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2a05d700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c2c0acef0 0x7f9c2c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2b05f700 1 -- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c2c0acef0 msgr2=0x7f9c2c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2b05f700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c2c0acef0 0x7f9c2c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2b05f700 1 -- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c2c17ff00 msgr2=0x7f9c2c090a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:44.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2b05f700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c2c17ff00 0x7f9c2c090a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:44.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2b05f700 1 -- 172.21.15.32:0/1886249293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c19b4f0 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.534 7f9c2b05f700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c19b0f0 0x7f9c2c0ab890 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f9c24007f90 tx=0x7f9c2400abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:44.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.535 7f9c1b7fe700 1 -- 172.21.15.32:0/1886249293 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c24000cc0 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.535 7f9c1b7fe700 1 -- 172.21.15.32:0/1886249293 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c240070d0 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.535 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c19b4f0 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.535 7f9c1b7fe700 1 -- 172.21.15.32:0/1886249293 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c24007520 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.535 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c19b4f0 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.535 7f9c1b7fe700 1 -- 172.21.15.32:0/1886249293 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9c240072b0 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.535 7f9c1b7fe700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c100188d0 0x7f9c1001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:44.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.535 7f9c1b7fe700 1 -- 172.21.15.32:0/1886249293 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f9c24032a00 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.536 7f9c2a85e700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c100188d0 0x7f9c1001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:44.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.536 7f9c2a85e700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c100188d0 0x7f9c1001add0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:44.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.556 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c091080 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.559 7f9c1b7fe700 1 -- 172.21.15.32:0/1886249293 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9c24007d00 con 0x7f9c2c19b0f0 2021-04-09T18:15:44.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.738 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f9c2c091080 con 0x7f9c2c19b0f0 2021-04-09T18:15:45.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:44.983 7f9c1b7fe700 1 -- 172.21.15.32:0/1886249293 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v51) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9c240070d0 con 0x7f9c2c19b0f0 2021-04-09T18:15:45.030 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:15:45.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c100188d0 msgr2=0x7f9c1001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c100188d0 0x7f9c1001add0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c19b0f0 msgr2=0x7f9c2c0ab890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c19b0f0 0x7f9c2c0ab890 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f9c24007f90 tx=0x7f9c2400abd0).stop 2021-04-09T18:15:45.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 shutdown_connections 2021-04-09T18:15:45.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c100188d0 0x7f9c1001add0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c2c19b0f0 0x7f9c2c0ab890 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c2c0acef0 0x7f9c2c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 --2- 172.21.15.32:0/1886249293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c2c17ff00 0x7f9c2c090a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 >> 172.21.15.32:0/1886249293 conn(0x7f9c2c0bb410 msgr2=0x7f9c2c08dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:45.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 shutdown_connections 2021-04-09T18:15:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.005 7f9c30c0b700 1 -- 172.21.15.32:0/1886249293 wait complete. 2021-04-09T18:15:45.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:15:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.168 7fcb30baf700 1 Processor -- start 2021-04-09T18:15:45.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.168 7fcb30baf700 1 -- start start 2021-04-09T18:15:45.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.169 7fcb30baf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb2c0d7ca0 0x7fcb2c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.169 7fcb30baf700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcb2c10da00 0x7fcb2c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.169 7fcb30baf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2c10d480 con 0x7fcb2c10d010 2021-04-09T18:15:45.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.169 7fcb30baf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2c0d9220 con 0x7fcb2c0d7ca0 2021-04-09T18:15:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.169 7fcb30baf700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2c0d86c0 con 0x7fcb2c10da00 2021-04-09T18:15:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.169 7fcb2a05d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb2c0d7ca0 0x7fcb2c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:45.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.169 7fcb2a85e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcb2c10d010 0x7fcb2c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53858/0 (socket says 172.21.15.32:53858) 2021-04-09T18:15:45.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.169 7fcb2a85e700 1 -- 172.21.15.32:0/3674752489 learned_addr learned my addr 172.21.15.32:0/3674752489 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:45.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.169 7fcb2b05f700 1 --2- 172.21.15.32:0/3674752489 >> v2:172.21.15.86:3300/0 conn(0x7fcb2c10da00 0x7fcb2c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:45.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb2a05d700 1 -- 172.21.15.32:0/3674752489 >> v2:172.21.15.86:3300/0 conn(0x7fcb2c10da00 msgr2=0x7fcb2c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:45.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb2a05d700 1 --2- 172.21.15.32:0/3674752489 >> v2:172.21.15.86:3300/0 conn(0x7fcb2c10da00 0x7fcb2c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb2a05d700 1 -- 172.21.15.32:0/3674752489 >> v1:172.21.15.32:6789/0 conn(0x7fcb2c10d010 legacy=0x7fcb2c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb2a05d700 1 -- 172.21.15.32:0/3674752489 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c0d1f70 con 0x7fcb2c0d7ca0 2021-04-09T18:15:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb2985c700 1 -- 172.21.15.32:0/3674752489 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894485076 0 0) 0x7fcb1c000ce0 con 0x7fcb2c10d010 2021-04-09T18:15:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb2a05d700 1 --2- 172.21.15.32:0/3674752489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0d7ca0 0x7fcb2c0d80a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fcb20006650 tx=0x7fcb20009a00).ready entity=mon.0 client_cookie=f47e4fe9cf6ba491 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb2985c700 1 -- 172.21.15.32:0/3674752489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb20009df0 con 0x7fcb2c0d7ca0 2021-04-09T18:15:45.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb2985c700 1 -- 172.21.15.32:0/3674752489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb200009b0 con 0x7fcb2c0d7ca0 2021-04-09T18:15:45.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb2985c700 1 -- 172.21.15.32:0/3674752489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb2000c2b0 con 0x7fcb2c0d7ca0 2021-04-09T18:15:45.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb30baf700 1 -- 172.21.15.32:0/3674752489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0d7ca0 msgr2=0x7fcb2c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.170 7fcb30baf700 1 --2- 172.21.15.32:0/3674752489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0d7ca0 0x7fcb2c0d80a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fcb20006650 tx=0x7fcb20009a00).stop 2021-04-09T18:15:45.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.171 7fcb30baf700 1 -- 172.21.15.32:0/3674752489 shutdown_connections 2021-04-09T18:15:45.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.171 7fcb30baf700 1 --2- 172.21.15.32:0/3674752489 >> v2:172.21.15.86:3300/0 conn(0x7fcb2c10da00 0x7fcb2c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.171 7fcb30baf700 1 --2- 172.21.15.32:0/3674752489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0d7ca0 0x7fcb2c0d80a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.171 7fcb30baf700 1 -- 172.21.15.32:0/3674752489 >> 172.21.15.32:0/3674752489 conn(0x7fcb2c0ce5f0 msgr2=0x7fcb2c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.171 7fcb30baf700 1 -- 172.21.15.32:0/3674752489 shutdown_connections 2021-04-09T18:15:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.171 7fcb30baf700 1 -- 172.21.15.32:0/3674752489 wait complete. 2021-04-09T18:15:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.191 7fcb30baf700 1 Processor -- start 2021-04-09T18:15:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.191 7fcb30baf700 1 -- start start 2021-04-09T18:15:45.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb30baf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb2c0a0110 0x7fcb2c18c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb30baf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0a0510 0x7fcb2c0c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb30baf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb2c0c36e0 0x7fcb2c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb30baf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c18cf60 con 0x7fcb2c0a0510 2021-04-09T18:15:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb30baf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c199ea0 con 0x7fcb2c0a0110 2021-04-09T18:15:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb30baf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c187d70 con 0x7fcb2c0c36e0 2021-04-09T18:15:45.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0a0510 0x7fcb2c0c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a85e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb2c0a0110 0x7fcb2c18c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0a0510 0x7fcb2c0c30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44278/0 (socket says 172.21.15.32:44278) 2021-04-09T18:15:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a05d700 1 -- 172.21.15.32:0/2514766335 learned_addr learned my addr 172.21.15.32:0/2514766335 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:45.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2b05f700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb2c0c36e0 0x7fcb2c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a05d700 1 -- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb2c0c36e0 msgr2=0x7fcb2c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a05d700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb2c0c36e0 0x7fcb2c0c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a05d700 1 -- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb2c0a0110 msgr2=0x7fcb2c18c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a05d700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb2c0a0110 0x7fcb2c18c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a05d700 1 -- 172.21.15.32:0/2514766335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c18d4e0 con 0x7fcb2c0a0510 2021-04-09T18:15:45.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb2a05d700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0a0510 0x7fcb2c0c30c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fcb2000b560 tx=0x7fcb2000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:45.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb1b7fe700 1 -- 172.21.15.32:0/2514766335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb2000bcf0 con 0x7fcb2c0a0510 2021-04-09T18:15:45.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb1b7fe700 1 -- 172.21.15.32:0/2514766335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb20000c90 con 0x7fcb2c0a0510 2021-04-09T18:15:45.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb1b7fe700 1 -- 172.21.15.32:0/2514766335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb20002220 con 0x7fcb2c0a0510 2021-04-09T18:15:45.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2c18d4e0 con 0x7fcb2c0a0510 2021-04-09T18:15:45.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.192 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb2c18d4e0 con 0x7fcb2c0a0510 2021-04-09T18:15:45.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.193 7fcb1b7fe700 1 -- 172.21.15.32:0/2514766335 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcb2000a000 con 0x7fcb2c0a0510 2021-04-09T18:15:45.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.193 7fcb1b7fe700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb100188c0 0x7fcb1001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.193 7fcb1b7fe700 1 -- 172.21.15.32:0/2514766335 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5111+0+0 (secure 0 0 0) 0x7fcb20034980 con 0x7fcb2c0a0510 2021-04-09T18:15:45.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.193 7fcb2a85e700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb100188c0 0x7fcb1001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:45.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.194 7fcb2a85e700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb100188c0 0x7fcb1001adc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:45.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.213 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb2c10d480 con 0x7fcb2c0a0510 2021-04-09T18:15:45.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.217 7fcb1b7fe700 1 -- 172.21.15.32:0/2514766335 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcb200009c0 con 0x7fcb2c0a0510 2021-04-09T18:15:45.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.401 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fcb2c10d480 con 0x7fcb2c0a0510 2021-04-09T18:15:45.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.660 7fcb1b7fe700 1 -- 172.21.15.32:0/2514766335 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v52) v1 ==== 125+0+0 (secure 0 0 0) 0x7fcb2000a000 con 0x7fcb2c0a0510 2021-04-09T18:15:45.695 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:15:45.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb100188c0 msgr2=0x7fcb1001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb100188c0 0x7fcb1001adc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0a0510 msgr2=0x7fcb2c0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0a0510 0x7fcb2c0c30c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fcb2000b560 tx=0x7fcb2000b900).stop 2021-04-09T18:15:45.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 shutdown_connections 2021-04-09T18:15:45.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb100188c0 0x7fcb1001adc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2c0a0510 0x7fcb2c0c30c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb2c0c36e0 0x7fcb2c0c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 --2- 172.21.15.32:0/2514766335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb2c0a0110 0x7fcb2c18c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 >> 172.21.15.32:0/2514766335 conn(0x7fcb2c0b93a0 msgr2=0x7fcb2c0b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:45.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 shutdown_connections 2021-04-09T18:15:45.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.667 7fcb30baf700 1 -- 172.21.15.32:0/2514766335 wait complete. 2021-04-09T18:15:45.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:15:45.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.839 7f7aed8c0700 1 Processor -- start 2021-04-09T18:15:45.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.839 7f7aed8c0700 1 -- start start 2021-04-09T18:15:45.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.839 7f7aed8c0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ae810d050 0x7f7ae80cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.839 7f7aed8c0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae810d500 con 0x7f7ae80d84b0 2021-04-09T18:15:45.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.839 7f7aed8c0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae810e270 con 0x7f7ae80d0160 2021-04-09T18:15:45.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.839 7f7aed8c0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae80d8d70 con 0x7f7ae810d050 2021-04-09T18:15:45.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae7574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ae810d050 0x7f7ae80cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:45.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae7d75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7ae80d84b0 0x7f7ae810da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53874/0 (socket says 172.21.15.32:53874) 2021-04-09T18:15:45.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae7d75700 1 -- 172.21.15.32:0/774778740 learned_addr learned my addr 172.21.15.32:0/774778740 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:45.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae7574700 1 -- 172.21.15.32:0/774778740 >> v1:172.21.15.86:6789/0 conn(0x7f7ae80d0160 legacy=0x7f7ae80d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae7574700 1 -- 172.21.15.32:0/774778740 >> v1:172.21.15.32:6789/0 conn(0x7f7ae80d84b0 legacy=0x7f7ae810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae6572700 1 -- 172.21.15.32:0/774778740 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921620812 0 0) 0x7f7ae0000d30 con 0x7f7ae80d84b0 2021-04-09T18:15:45.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae7574700 1 -- 172.21.15.32:0/774778740 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae80d9260 con 0x7f7ae810d050 2021-04-09T18:15:45.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae7574700 1 --2- 172.21.15.32:0/774778740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae810d050 0x7f7ae80cfb40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f7ad8006650 tx=0x7f7ad8009a80).ready entity=mon.0 client_cookie=2d9fde887f16df0e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:45.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae6572700 1 -- 172.21.15.32:0/774778740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ad80009b0 con 0x7f7ae810d050 2021-04-09T18:15:45.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae6572700 1 -- 172.21.15.32:0/774778740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad800c0c0 con 0x7f7ae810d050 2021-04-09T18:15:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7ae6572700 1 -- 172.21.15.32:0/774778740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ad800c510 con 0x7f7ae810d050 2021-04-09T18:15:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7aed8c0700 1 -- 172.21.15.32:0/774778740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae810d050 msgr2=0x7f7ae80cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.840 7f7aed8c0700 1 --2- 172.21.15.32:0/774778740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae810d050 0x7f7ae80cfb40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f7ad8006650 tx=0x7f7ad8009a80).stop 2021-04-09T18:15:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.841 7f7aed8c0700 1 -- 172.21.15.32:0/774778740 shutdown_connections 2021-04-09T18:15:45.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.841 7f7aed8c0700 1 --2- 172.21.15.32:0/774778740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae810d050 0x7f7ae80cfb40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:45.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.841 7f7aed8c0700 1 -- 172.21.15.32:0/774778740 >> 172.21.15.32:0/774778740 conn(0x7f7ae80ce620 msgr2=0x7f7ae80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:45.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.841 7f7aed8c0700 1 -- 172.21.15.32:0/774778740 shutdown_connections 2021-04-09T18:15:45.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.841 7f7aed8c0700 1 -- 172.21.15.32:0/774778740 wait complete. 2021-04-09T18:15:45.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.861 7f7aed8c0700 1 Processor -- start 2021-04-09T18:15:45.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.861 7f7aed8c0700 1 -- start start 2021-04-09T18:15:45.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7aed8c0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae80a1360 0x7f7ae80a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7aed8c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae808a060 0x7f7ae808a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7aed8c0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ae808b1f0 0x7f7ae808b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:45.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7aed8c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae808ac70 con 0x7f7ae808a060 2021-04-09T18:15:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7aed8c0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae808bc10 con 0x7f7ae80a1360 2021-04-09T18:15:45.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7aed8c0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80a1ed0 con 0x7f7ae808b1f0 2021-04-09T18:15:45.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae6d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae808a060 0x7f7ae808a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:45.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae6d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae808a060 0x7f7ae808a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44292/0 (socket says 172.21.15.32:44292) 2021-04-09T18:15:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae6d73700 1 -- 172.21.15.32:0/4056784244 learned_addr learned my addr 172.21.15.32:0/4056784244 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:45.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae7d75700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ae808b1f0 0x7f7ae808b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae7574700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae80a1360 0x7f7ae80a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae6d73700 1 -- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ae808b1f0 msgr2=0x7f7ae808b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae6d73700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ae808b1f0 0x7f7ae808b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae6d73700 1 -- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae80a1360 msgr2=0x7f7ae80a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae6d73700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae80a1360 0x7f7ae80a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.862 7f7ae6d73700 1 -- 172.21.15.32:0/4056784244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae80a20f0 con 0x7f7ae808a060 2021-04-09T18:15:46.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.863 7f7ae6d73700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae808a060 0x7f7ae808a460 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f7ad400b5b0 tx=0x7f7ad400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:46.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.863 7f7adffff700 1 -- 172.21.15.32:0/4056784244 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ad400bd40 con 0x7f7ae808a060 2021-04-09T18:15:46.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.863 7f7adffff700 1 -- 172.21.15.32:0/4056784244 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad4006240 con 0x7f7ae808a060 2021-04-09T18:15:46.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.863 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae80a20f0 con 0x7f7ae808a060 2021-04-09T18:15:46.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.863 7f7adffff700 1 -- 172.21.15.32:0/4056784244 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ad4005690 con 0x7f7ae808a060 2021-04-09T18:15:46.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.863 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae80a20f0 con 0x7f7ae808a060 2021-04-09T18:15:46.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.863 7f7adffff700 1 -- 172.21.15.32:0/4056784244 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7ad40054a0 con 0x7f7ae808a060 2021-04-09T18:15:46.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.865 7f7adffff700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ac40188c0 0x7f7ac401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.865 7f7adffff700 1 -- 172.21.15.32:0/4056784244 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f7ad40329c0 con 0x7f7ae808a060 2021-04-09T18:15:46.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.865 7f7ae7574700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ac40188c0 0x7f7ac401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.865 7f7ae7574700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ac40188c0 0x7f7ac401adc0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:46.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.884 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae808bc10 con 0x7f7ae808a060 2021-04-09T18:15:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:45.888 7f7adffff700 1 -- 172.21.15.32:0/4056784244 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7ad4000c60 con 0x7f7ae808a060 2021-04-09T18:15:46.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.072 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f7ae808bc10 con 0x7f7ae808a060 2021-04-09T18:15:46.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.076 7f7adffff700 1 -- 172.21.15.32:0/4056784244 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 8 and data pool 9 v42) v1 ==== 187+0+0 (secure 0 0 0) 0x7f7ad4006240 con 0x7f7ae808a060 2021-04-09T18:15:46.116 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 8 and data pool 9 2021-04-09T18:15:46.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ac40188c0 msgr2=0x7f7ac401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ac40188c0 0x7f7ac401adc0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae808a060 msgr2=0x7f7ae808a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae808a060 0x7f7ae808a460 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f7ad400b5b0 tx=0x7f7ad400b950).stop 2021-04-09T18:15:46.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 shutdown_connections 2021-04-09T18:15:46.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ac40188c0 0x7f7ac401adc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ae808a060 0x7f7ae808a460 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ae808b1f0 0x7f7ae808b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 --2- 172.21.15.32:0/4056784244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae80a1360 0x7f7ae80a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 >> 172.21.15.32:0/4056784244 conn(0x7f7ae80920d0 msgr2=0x7f7ae80a56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:46.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 shutdown_connections 2021-04-09T18:15:46.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.085 7f7aed8c0700 1 -- 172.21.15.32:0/4056784244 wait complete. 2021-04-09T18:15:46.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:15:46.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.305 7f3a73de3700 1 Processor -- start 2021-04-09T18:15:46.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a73de3700 1 -- start start 2021-04-09T18:15:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a73de3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a6c10d010 0x7f3a6c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a73de3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c0d1da0 0x7f3a6c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a73de3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3a6c0d82c0 0x7f3a6c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a73de3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6c0d8d30 con 0x7f3a6c10d010 2021-04-09T18:15:46.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a73de3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6c0d9220 con 0x7f3a6c0d82c0 2021-04-09T18:15:46.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a73de3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c10d480 con 0x7f3a6c0d1da0 2021-04-09T18:15:46.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a71bff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a6c10d010 0x7f3a6c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a71bff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a6c10d010 0x7f3a6c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44298/0 (socket says 172.21.15.32:44298) 2021-04-09T18:15:46.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.306 7f3a71bff700 1 -- 172.21.15.32:0/2557463518 learned_addr learned my addr 172.21.15.32:0/2557463518 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:46.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a72400700 1 --2- 172.21.15.32:0/2557463518 >> v2:172.21.15.86:3300/0 conn(0x7f3a6c0d82c0 0x7f3a6c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a713fe700 1 --2- 172.21.15.32:0/2557463518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c0d1da0 0x7f3a6c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a71bff700 1 -- 172.21.15.32:0/2557463518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c0d1da0 msgr2=0x7f3a6c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a71bff700 1 --2- 172.21.15.32:0/2557463518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c0d1da0 0x7f3a6c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a71bff700 1 -- 172.21.15.32:0/2557463518 >> v2:172.21.15.86:3300/0 conn(0x7f3a6c0d82c0 msgr2=0x7f3a6c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a71bff700 1 --2- 172.21.15.32:0/2557463518 >> v2:172.21.15.86:3300/0 conn(0x7f3a6c0d82c0 0x7f3a6c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a71bff700 1 -- 172.21.15.32:0/2557463518 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c0d86c0 con 0x7f3a6c10d010 2021-04-09T18:15:46.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a713fe700 1 --2- 172.21.15.32:0/2557463518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c0d1da0 0x7f3a6c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:46.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a71bff700 1 --2- 172.21.15.32:0/2557463518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c10d010 0x7f3a6c0d1780 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f3a68006700 tx=0x7f3a68009b80).ready entity=mon.0 client_cookie=a7b55044427eb8df server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:46.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a70bfd700 1 -- 172.21.15.32:0/2557463518 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a6800c0c0 con 0x7f3a6c10d010 2021-04-09T18:15:46.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a70bfd700 1 -- 172.21.15.32:0/2557463518 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6800c440 con 0x7f3a6c10d010 2021-04-09T18:15:46.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a70bfd700 1 -- 172.21.15.32:0/2557463518 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a6800c920 con 0x7f3a6c10d010 2021-04-09T18:15:46.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a73de3700 1 -- 172.21.15.32:0/2557463518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c10d010 msgr2=0x7f3a6c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.307 7f3a73de3700 1 --2- 172.21.15.32:0/2557463518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c10d010 0x7f3a6c0d1780 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f3a68006700 tx=0x7f3a68009b80).stop 2021-04-09T18:15:46.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.308 7f3a73de3700 1 -- 172.21.15.32:0/2557463518 shutdown_connections 2021-04-09T18:15:46.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.308 7f3a73de3700 1 --2- 172.21.15.32:0/2557463518 >> v2:172.21.15.86:3300/0 conn(0x7f3a6c0d82c0 0x7f3a6c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.308 7f3a73de3700 1 --2- 172.21.15.32:0/2557463518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c0d1da0 0x7f3a6c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.308 7f3a73de3700 1 --2- 172.21.15.32:0/2557463518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c10d010 0x7f3a6c0d1780 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.308 7f3a73de3700 1 -- 172.21.15.32:0/2557463518 >> 172.21.15.32:0/2557463518 conn(0x7f3a6c0ce5f0 msgr2=0x7f3a6c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:46.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.308 7f3a73de3700 1 -- 172.21.15.32:0/2557463518 shutdown_connections 2021-04-09T18:15:46.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.308 7f3a73de3700 1 -- 172.21.15.32:0/2557463518 wait complete. 2021-04-09T18:15:46.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.328 7f3a73de3700 1 Processor -- start 2021-04-09T18:15:46.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.328 7f3a73de3700 1 -- start start 2021-04-09T18:15:46.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a73de3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c1ac380 0x7f3a6c18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a73de3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c1ac780 0x7f3a6c18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a73de3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a6c183520 0x7f3a6c183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a73de3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c1ace10 con 0x7f3a6c1ac380 2021-04-09T18:15:46.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a73de3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c189cf0 con 0x7f3a6c183520 2021-04-09T18:15:46.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a73de3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c183f40 con 0x7f3a6c1ac780 2021-04-09T18:15:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a71bff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c1ac380 0x7f3a6c18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a71bff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c1ac380 0x7f3a6c18a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44306/0 (socket says 172.21.15.32:44306) 2021-04-09T18:15:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a71bff700 1 -- 172.21.15.32:0/4218473795 learned_addr learned my addr 172.21.15.32:0/4218473795 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:46.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a72400700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a6c183520 0x7f3a6c183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a713fe700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c1ac780 0x7f3a6c18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a71bff700 1 -- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c1ac780 msgr2=0x7f3a6c18a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a71bff700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c1ac780 0x7f3a6c18a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a71bff700 1 -- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a6c183520 msgr2=0x7f3a6c183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.329 7f3a71bff700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a6c183520 0x7f3a6c183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.330 7f3a71bff700 1 -- 172.21.15.32:0/4218473795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c184160 con 0x7f3a6c1ac380 2021-04-09T18:15:46.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.330 7f3a71bff700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c1ac380 0x7f3a6c18a230 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f3a6800b5e0 tx=0x7f3a6800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:46.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.330 7f3a627fc700 1 -- 172.21.15.32:0/4218473795 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a6800bd70 con 0x7f3a6c1ac380 2021-04-09T18:15:46.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.330 7f3a627fc700 1 -- 172.21.15.32:0/4218473795 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a68002030 con 0x7f3a6c1ac380 2021-04-09T18:15:46.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.330 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c184160 con 0x7f3a6c1ac380 2021-04-09T18:15:46.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.330 7f3a627fc700 1 -- 172.21.15.32:0/4218473795 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a68002550 con 0x7f3a6c1ac380 2021-04-09T18:15:46.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.330 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c184160 con 0x7f3a6c1ac380 2021-04-09T18:15:46.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.330 7f3a627fc700 1 -- 172.21.15.32:0/4218473795 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3a68002210 con 0x7f3a6c1ac380 2021-04-09T18:15:46.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.331 7f3a627fc700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a54018930 0x7f3a5401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.331 7f3a627fc700 1 -- 172.21.15.32:0/4218473795 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3a680329d0 con 0x7f3a6c1ac380 2021-04-09T18:15:46.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.331 7f3a713fe700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a54018930 0x7f3a5401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.331 7f3a713fe700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a54018930 0x7f3a5401ae10 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:46.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.350 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c182d10 con 0x7f3a6c1ac380 2021-04-09T18:15:46.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.353 7f3a627fc700 1 -- 172.21.15.32:0/4218473795 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3a68002240 con 0x7f3a6c1ac380 2021-04-09T18:15:46.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.542 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a6c182d10 con 0x7f3a6c1ac380 2021-04-09T18:15:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.543 7f3a627fc700 1 -- 172.21.15.32:0/4218473795 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f3a68002030 con 0x7f3a6c1ac380 2021-04-09T18:15:46.633 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:46.634 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":53,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:15:46.073468","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:15:44.773626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-04-09 18:15:45.435836","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:15:46.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a54018930 msgr2=0x7f3a5401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a54018930 0x7f3a5401ae10 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c1ac380 msgr2=0x7f3a6c18a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c1ac380 0x7f3a6c18a230 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f3a6800b5e0 tx=0x7f3a6800b980).stop 2021-04-09T18:15:46.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 shutdown_connections 2021-04-09T18:15:46.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a54018930 0x7f3a5401ae10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a6c183520 0x7f3a6c183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a6c1ac780 0x7f3a6c18a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 --2- 172.21.15.32:0/4218473795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a6c1ac380 0x7f3a6c18a230 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 >> 172.21.15.32:0/4218473795 conn(0x7f3a6c19b890 msgr2=0x7f3a6c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:46.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 shutdown_connections 2021-04-09T18:15:46.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.545 7f3a73de3700 1 -- 172.21.15.32:0/4218473795 wait complete. 2021-04-09T18:15:46.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:15:46.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.858 7fa6538c0700 1 Processor -- start 2021-04-09T18:15:46.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.858 7fa6538c0700 1 -- start start 2021-04-09T18:15:46.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa6538c0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa64c10da00 0x7fa64c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa6538c0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c10d480 con 0x7fa64c10d010 2021-04-09T18:15:46.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa6538c0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0d1f70 con 0x7fa64c0d7ca0 2021-04-09T18:15:46.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa6538c0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c0d8d30 con 0x7fa64c10da00 2021-04-09T18:15:46.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa6516dc700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa64c10d010 0x7fa64c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53902/0 (socket says 172.21.15.32:53902) 2021-04-09T18:15:46.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa6516dc700 1 -- 172.21.15.32:0/4229172118 learned_addr learned my addr 172.21.15.32:0/4229172118 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:46.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa651edd700 1 --2- 172.21.15.32:0/4229172118 >> v2:172.21.15.86:3300/0 conn(0x7fa64c10da00 0x7fa64c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa643fff700 1 -- 172.21.15.32:0/4229172118 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816355069 0 0) 0x7fa648001060 con 0x7fa64c10d010 2021-04-09T18:15:46.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa643fff700 1 -- 172.21.15.32:0/4229172118 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa638002fe0 con 0x7fa64c10d010 2021-04-09T18:15:46.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa643fff700 1 -- 172.21.15.32:0/4229172118 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2999963107 0 0) 0x7fa648001250 con 0x7fa64c10d010 2021-04-09T18:15:46.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa643fff700 1 -- 172.21.15.32:0/4229172118 >> v2:172.21.15.86:3300/0 conn(0x7fa64c10da00 msgr2=0x7fa64c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:46.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa643fff700 1 --2- 172.21.15.32:0/4229172118 >> v2:172.21.15.86:3300/0 conn(0x7fa64c10da00 0x7fa64c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.859 7fa643fff700 1 -- 172.21.15.32:0/4229172118 >> v1:172.21.15.86:6789/0 conn(0x7fa64c0d7ca0 legacy=0x7fa64c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 -- 172.21.15.32:0/4229172118 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0d9220 con 0x7fa64c10d010 2021-04-09T18:15:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 -- 172.21.15.32:0/4229172118 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fa648001980 con 0x7fa64c10d010 2021-04-09T18:15:46.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 -- 172.21.15.32:0/4229172118 >> v1:172.21.15.32:6789/0 conn(0x7fa64c10d010 legacy=0x7fa64c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa638003a70 0x7fa638005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6380067a0 0x7fa638008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6380095a0 0x7fa63800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:46.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 -- 172.21.15.32:0/4229172118 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63800c310 con 0x7fa638003a70 2021-04-09T18:15:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 -- 172.21.15.32:0/4229172118 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63800c530 con 0x7fa6380095a0 2021-04-09T18:15:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa6516dc700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa638003a70 0x7fa638005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 -- 172.21.15.32:0/4229172118 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa638050760 con 0x7fa6380067a0 2021-04-09T18:15:46.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa651edd700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6380095a0 0x7fa63800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa650edb700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6380067a0 0x7fa638008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:46.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa6516dc700 1 -- 172.21.15.32:0/4229172118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6380067a0 msgr2=0x7fa638008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:46.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa6516dc700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6380067a0 0x7fa638008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa6516dc700 1 -- 172.21.15.32:0/4229172118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6380095a0 msgr2=0x7fa63800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:46.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa6516dc700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6380095a0 0x7fa63800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:46.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa6516dc700 1 -- 172.21.15.32:0/4229172118 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6380509c0 con 0x7fa638003a70 2021-04-09T18:15:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa651edd700 1 -- 172.21.15.32:0/4229172118 reap_dead start 2021-04-09T18:15:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa6516dc700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa638003a70 0x7fa638005f00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fa648007cf0 tx=0x7fa64800a310).ready entity=mon.0 client_cookie=6894896b3d432d72 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 -- 172.21.15.32:0/4229172118 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6480070c0 con 0x7fa638003a70 2021-04-09T18:15:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 -- 172.21.15.32:0/4229172118 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa648007440 con 0x7fa638003a70 2021-04-09T18:15:47.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa643fff700 1 -- 172.21.15.32:0/4229172118 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6480049a0 con 0x7fa638003a70 2021-04-09T18:15:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa6538c0700 1 -- 172.21.15.32:0/4229172118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa638003a70 msgr2=0x7fa638005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.860 7fa6538c0700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa638003a70 0x7fa638005f00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fa648007cf0 tx=0x7fa64800a310).stop 2021-04-09T18:15:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.861 7fa6538c0700 1 -- 172.21.15.32:0/4229172118 shutdown_connections 2021-04-09T18:15:47.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.861 7fa6538c0700 1 --2- 172.21.15.32:0/4229172118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa638003a70 0x7fa638005f00 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.861 7fa6538c0700 1 -- 172.21.15.32:0/4229172118 >> 172.21.15.32:0/4229172118 conn(0x7fa64c0ce5f0 msgr2=0x7fa64c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:47.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.861 7fa6538c0700 1 -- 172.21.15.32:0/4229172118 shutdown_connections 2021-04-09T18:15:47.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.861 7fa6538c0700 1 -- 172.21.15.32:0/4229172118 wait complete. 2021-04-09T18:15:47.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.881 7fa6538c0700 1 Processor -- start 2021-04-09T18:15:47.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.881 7fa6538c0700 1 -- start start 2021-04-09T18:15:47.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa6538c0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa64c09a670 0x7fa64c093610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa6538c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa64c1a9fb0 0x7fa64c1aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa6538c0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa64c0a3ab0 0x7fa64c0a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa6538c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c094060 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa6538c0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c094710 con 0x7fa64c09a670 2021-04-09T18:15:47.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa6538c0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c093c30 con 0x7fa64c0a3ab0 2021-04-09T18:15:47.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa650edb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa64c1a9fb0 0x7fa64c1aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa650edb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa64c1a9fb0 0x7fa64c1aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44324/0 (socket says 172.21.15.32:44324) 2021-04-09T18:15:47.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa650edb700 1 -- 172.21.15.32:0/1181983220 learned_addr learned my addr 172.21.15.32:0/1181983220 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:47.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa6516dc700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa64c09a670 0x7fa64c093610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa651edd700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa64c0a3ab0 0x7fa64c0a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa650edb700 1 -- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa64c0a3ab0 msgr2=0x7fa64c0a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:47.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa650edb700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa64c0a3ab0 0x7fa64c0a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa650edb700 1 -- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa64c09a670 msgr2=0x7fa64c093610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:47.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa650edb700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa64c09a670 0x7fa64c093610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.882 7fa650edb700 1 -- 172.21.15.32:0/1181983220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c1aa9d0 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.883 7fa650edb700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa64c1a9fb0 0x7fa64c1aa3b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fa63c006610 tx=0x7fa63c00a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:47.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.883 7fa641ffb700 1 -- 172.21.15.32:0/1181983220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa63c00ad40 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.883 7fa641ffb700 1 -- 172.21.15.32:0/1181983220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c00c0d0 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.883 7fa641ffb700 1 -- 172.21.15.32:0/1181983220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa63c00c560 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.883 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c1aa9d0 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.883 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c1aa9d0 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.883 7fa641ffb700 1 -- 172.21.15.32:0/1181983220 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa63c00afc0 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.884 7fa641ffb700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa62c0188d0 0x7fa62c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.884 7fa641ffb700 1 -- 172.21.15.32:0/1181983220 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa63c032a30 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.884 7fa6516dc700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa62c0188d0 0x7fa62c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.884 7fa6516dc700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa62c0188d0 0x7fa62c01add0 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:47.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.903 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c10d480 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:46.906 7fa641ffb700 1 -- 172.21.15.32:0/1181983220 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa63c000c10 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.098 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa64c10d480 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.099 7fa641ffb700 1 -- 172.21.15.32:0/1181983220 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fa63c00af40 con 0x7fa64c1a9fb0 2021-04-09T18:15:47.166 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:47.167 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":54,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:15:47.075022","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:15:44.773626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-04-09 18:15:45.435836","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:15:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.101 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa62c0188d0 msgr2=0x7fa62c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:47.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.101 7fa6538c0700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa62c0188d0 0x7fa62c01add0 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.101 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa64c1a9fb0 msgr2=0x7fa64c1aa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:47.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.101 7fa6538c0700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa64c1a9fb0 0x7fa64c1aa3b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fa63c006610 tx=0x7fa63c00a950).stop 2021-04-09T18:15:47.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.101 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 shutdown_connections 2021-04-09T18:15:47.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.101 7fa6538c0700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa62c0188d0 0x7fa62c01add0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.101 7fa6538c0700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa64c1a9fb0 0x7fa64c1aa3b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.102 7fa6538c0700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa64c0a3ab0 0x7fa64c0a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.102 7fa6538c0700 1 --2- 172.21.15.32:0/1181983220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa64c09a670 0x7fa64c093610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.102 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 >> 172.21.15.32:0/1181983220 conn(0x7fa64c1a0db0 msgr2=0x7fa64c1a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:47.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.102 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 shutdown_connections 2021-04-09T18:15:47.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.102 7fa6538c0700 1 -- 172.21.15.32:0/1181983220 wait complete. 2021-04-09T18:15:47.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:15:47.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.329 7f05e13fe700 1 Processor -- start 2021-04-09T18:15:47.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.329 7f05e13fe700 1 -- start start 2021-04-09T18:15:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.329 7f05e13fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05dc0d7ca0 0x7f05dc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.329 7f05e13fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc10da00 0x7f05dc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.329 7f05e13fe700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05dc10d480 con 0x7f05dc10d010 2021-04-09T18:15:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.329 7f05e13fe700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05dc0d9220 con 0x7f05dc0d7ca0 2021-04-09T18:15:47.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.329 7f05e13fe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc0d86c0 con 0x7f05dc10da00 2021-04-09T18:15:47.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.329 7f05da87c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05dc0d7ca0 0x7f05dc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da87c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f05dc0d7ca0 0x7f05dc0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44336/0 (socket says 172.21.15.32:44336) 2021-04-09T18:15:47.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da87c700 1 -- 172.21.15.32:0/1318435603 learned_addr learned my addr 172.21.15.32:0/1318435603 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:47.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05db87e700 1 --2- 172.21.15.32:0/1318435603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc10da00 0x7f05dc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da87c700 1 -- 172.21.15.32:0/1318435603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc10da00 msgr2=0x7f05dc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:47.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da87c700 1 --2- 172.21.15.32:0/1318435603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc10da00 0x7f05dc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da87c700 1 -- 172.21.15.32:0/1318435603 >> v1:172.21.15.86:6789/0 conn(0x7f05dc10d010 legacy=0x7f05dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:47.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da87c700 1 -- 172.21.15.32:0/1318435603 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05dc0d1f70 con 0x7f05dc0d7ca0 2021-04-09T18:15:47.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da87c700 1 --2- 172.21.15.32:0/1318435603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc0d7ca0 0x7f05dc0d80a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f05d0006700 tx=0x7f05d0009b00).ready entity=mon.0 client_cookie=f8f91ac670b92266 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:47.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da07b700 1 -- 172.21.15.32:0/1318435603 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05d000c0c0 con 0x7f05dc0d7ca0 2021-04-09T18:15:47.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da07b700 1 -- 172.21.15.32:0/1318435603 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d000c400 con 0x7f05dc0d7ca0 2021-04-09T18:15:47.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05da07b700 1 -- 172.21.15.32:0/1318435603 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05d000c8e0 con 0x7f05dc0d7ca0 2021-04-09T18:15:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05e13fe700 1 -- 172.21.15.32:0/1318435603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc0d7ca0 msgr2=0x7f05dc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.330 7f05e13fe700 1 --2- 172.21.15.32:0/1318435603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc0d7ca0 0x7f05dc0d80a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f05d0006700 tx=0x7f05d0009b00).stop 2021-04-09T18:15:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.331 7f05e13fe700 1 -- 172.21.15.32:0/1318435603 shutdown_connections 2021-04-09T18:15:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.331 7f05e13fe700 1 --2- 172.21.15.32:0/1318435603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc10da00 0x7f05dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.331 7f05e13fe700 1 --2- 172.21.15.32:0/1318435603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc0d7ca0 0x7f05dc0d80a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.331 7f05e13fe700 1 -- 172.21.15.32:0/1318435603 >> 172.21.15.32:0/1318435603 conn(0x7f05dc0ce5f0 msgr2=0x7f05dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.331 7f05e13fe700 1 -- 172.21.15.32:0/1318435603 shutdown_connections 2021-04-09T18:15:47.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.331 7f05e13fe700 1 -- 172.21.15.32:0/1318435603 wait complete. 2021-04-09T18:15:47.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.351 7f05e13fe700 1 Processor -- start 2021-04-09T18:15:47.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.351 7f05e13fe700 1 -- start start 2021-04-09T18:15:47.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05e13fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0a10 0x7f05dc0bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05e13fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc18ad00 0x7f05dc19fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05e13fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc1a03f0 0x7f05dc095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05e13fe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc18b100 con 0x7f05dc18ad00 2021-04-09T18:15:47.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05e13fe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc095860 con 0x7f05dc1a03f0 2021-04-09T18:15:47.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05e13fe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05dc095a80 con 0x7f05dc0a0a10 2021-04-09T18:15:47.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05da87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc18ad00 0x7f05dc19fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05da87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc18ad00 0x7f05dc19fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44340/0 (socket says 172.21.15.32:44340) 2021-04-09T18:15:47.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05da87c700 1 -- 172.21.15.32:0/2238031650 learned_addr learned my addr 172.21.15.32:0/2238031650 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:47.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05db07d700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0a10 0x7f05dc0bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05db87e700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc1a03f0 0x7f05dc095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05da87c700 1 -- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0a10 msgr2=0x7f05dc0bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:47.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05da87c700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0a10 0x7f05dc0bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05da87c700 1 -- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc1a03f0 msgr2=0x7f05dc095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:47.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05da87c700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc1a03f0 0x7f05dc095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:47.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05da87c700 1 -- 172.21.15.32:0/2238031650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05dc189250 con 0x7f05dc18ad00 2021-04-09T18:15:47.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05da87c700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc18ad00 0x7f05dc19fdd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f05d0006f20 tx=0x7f05d000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:47.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.352 7f05c3fff700 1 -- 172.21.15.32:0/2238031650 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05d000bdd0 con 0x7f05dc18ad00 2021-04-09T18:15:47.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.353 7f05c3fff700 1 -- 172.21.15.32:0/2238031650 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d0006440 con 0x7f05dc18ad00 2021-04-09T18:15:47.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.353 7f05c3fff700 1 -- 172.21.15.32:0/2238031650 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05d00052e0 con 0x7f05dc18ad00 2021-04-09T18:15:47.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.353 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05dc189250 con 0x7f05dc18ad00 2021-04-09T18:15:47.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.353 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05dc189250 con 0x7f05dc18ad00 2021-04-09T18:15:47.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.353 7f05c3fff700 1 -- 172.21.15.32:0/2238031650 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f05d00052e0 con 0x7f05dc18ad00 2021-04-09T18:15:47.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.353 7f05c3fff700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05bc0188d0 0x7f05bc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:47.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.353 7f05c3fff700 1 -- 172.21.15.32:0/2238031650 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f05d0032a00 con 0x7f05dc18ad00 2021-04-09T18:15:47.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.353 7f05db07d700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05bc0188d0 0x7f05bc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:47.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.354 7f05db07d700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05bc0188d0 0x7f05bc01add0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:47.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.374 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05dc093d50 con 0x7f05dc18ad00 2021-04-09T18:15:47.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.377 7f05c3fff700 1 -- 172.21.15.32:0/2238031650 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f05d0006440 con 0x7f05dc18ad00 2021-04-09T18:15:47.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:47.570 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f05dc10d480 con 0x7f05dc18ad00 2021-04-09T18:15:48.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.084 7f05c3fff700 1 -- 172.21.15.32:0/2238031650 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v45) v1 ==== 118+0+0 (secure 0 0 0) 0x7f05d00052e0 con 0x7f05dc18ad00 2021-04-09T18:15:48.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.086 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05bc0188d0 msgr2=0x7f05bc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.086 7f05e13fe700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05bc0188d0 0x7f05bc01add0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.086 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc18ad00 msgr2=0x7f05dc19fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.086 7f05e13fe700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc18ad00 0x7f05dc19fdd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f05d0006f20 tx=0x7f05d000b9e0).stop 2021-04-09T18:15:48.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.086 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 shutdown_connections 2021-04-09T18:15:48.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.086 7f05e13fe700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05bc0188d0 0x7f05bc01add0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.087 7f05e13fe700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05dc18ad00 0x7f05dc19fdd0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.087 7f05e13fe700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc1a03f0 0x7f05dc095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.087 7f05e13fe700 1 --2- 172.21.15.32:0/2238031650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0a10 0x7f05dc0bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.087 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 >> 172.21.15.32:0/2238031650 conn(0x7f05dc0975f0 msgr2=0x7f05dc096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:48.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.087 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 shutdown_connections 2021-04-09T18:15:48.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.087 7f05e13fe700 1 -- 172.21.15.32:0/2238031650 wait complete. 2021-04-09T18:15:48.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:48.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.257 7f2b96e52700 1 Processor -- start 2021-04-09T18:15:48.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.257 7f2b96e52700 1 -- start start 2021-04-09T18:15:48.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.257 7f2b96e52700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2b9010d010 0x7f2b900d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.257 7f2b96e52700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9010d480 con 0x7f2b900d8490 2021-04-09T18:15:48.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.257 7f2b96e52700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9010e1f0 con 0x7f2b900d1da0 2021-04-09T18:15:48.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.257 7f2b96e52700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b900d8d30 con 0x7f2b9010d010 2021-04-09T18:15:48.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b9546f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2b900d8490 0x7f2b9010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53938/0 (socket says 172.21.15.32:53938) 2021-04-09T18:15:48.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b9546f700 1 -- 172.21.15.32:0/4080723502 learned_addr learned my addr 172.21.15.32:0/4080723502 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:48.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b94c6e700 1 --2- 172.21.15.32:0/4080723502 >> v2:172.21.15.86:3300/0 conn(0x7f2b9010d010 0x7f2b900d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b8f7fe700 1 -- 172.21.15.32:0/4080723502 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780340983 0 0) 0x7f2b88000d30 con 0x7f2b900d8490 2021-04-09T18:15:48.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b8f7fe700 1 -- 172.21.15.32:0/4080723502 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b78002fe0 con 0x7f2b900d8490 2021-04-09T18:15:48.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b8f7fe700 1 -- 172.21.15.32:0/4080723502 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290025283 0 0) 0x7f2b84000c60 con 0x7f2b900d1da0 2021-04-09T18:15:48.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b8f7fe700 1 -- 172.21.15.32:0/4080723502 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b780042c0 con 0x7f2b900d1da0 2021-04-09T18:15:48.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b94c6e700 1 -- 172.21.15.32:0/4080723502 >> v1:172.21.15.86:6789/0 conn(0x7f2b900d1da0 legacy=0x7f2b900d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b94c6e700 1 -- 172.21.15.32:0/4080723502 >> v1:172.21.15.32:6789/0 conn(0x7f2b900d8490 legacy=0x7f2b9010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b94c6e700 1 -- 172.21.15.32:0/4080723502 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b900d9220 con 0x7f2b9010d010 2021-04-09T18:15:48.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b94c6e700 1 --2- 172.21.15.32:0/4080723502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9010d010 0x7f2b900d1780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f2b80006650 tx=0x7f2b80009a80).ready entity=mon.1 client_cookie=f4a2788c5aaa8b2f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b8f7fe700 1 -- 172.21.15.32:0/4080723502 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b800009b0 con 0x7f2b9010d010 2021-04-09T18:15:48.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b8f7fe700 1 -- 172.21.15.32:0/4080723502 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8000c0c0 con 0x7f2b9010d010 2021-04-09T18:15:48.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b8f7fe700 1 -- 172.21.15.32:0/4080723502 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b8000c510 con 0x7f2b9010d010 2021-04-09T18:15:48.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.258 7f2b96e52700 1 -- 172.21.15.32:0/4080723502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9010d010 msgr2=0x7f2b900d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.259 7f2b96e52700 1 --2- 172.21.15.32:0/4080723502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9010d010 0x7f2b900d1780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f2b80006650 tx=0x7f2b80009a80).stop 2021-04-09T18:15:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.259 7f2b96e52700 1 -- 172.21.15.32:0/4080723502 shutdown_connections 2021-04-09T18:15:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.259 7f2b96e52700 1 --2- 172.21.15.32:0/4080723502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9010d010 0x7f2b900d1780 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.259 7f2b96e52700 1 -- 172.21.15.32:0/4080723502 >> 172.21.15.32:0/4080723502 conn(0x7f2b900ce5f0 msgr2=0x7f2b900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.259 7f2b96e52700 1 -- 172.21.15.32:0/4080723502 shutdown_connections 2021-04-09T18:15:48.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.259 7f2b96e52700 1 -- 172.21.15.32:0/4080723502 wait complete. 2021-04-09T18:15:48.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.279 7f2b96e52700 1 Processor -- start 2021-04-09T18:15:48.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.279 7f2b96e52700 1 -- start start 2021-04-09T18:15:48.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b96e52700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b90179610 0x7f2b90179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b96e52700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9017a6e0 0x7f2b9017aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b96e52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b9017b100 0x7f2b9008db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b96e52700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9017d720 con 0x7f2b9017b100 2021-04-09T18:15:48.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b96e52700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90185ba0 con 0x7f2b9017a6e0 2021-04-09T18:15:48.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b96e52700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9008e1b0 con 0x7f2b90179610 2021-04-09T18:15:48.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b9546f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b9017b100 0x7f2b9008db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b9546f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b9017b100 0x7f2b9008db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44356/0 (socket says 172.21.15.32:44356) 2021-04-09T18:15:48.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b9546f700 1 -- 172.21.15.32:0/2589641461 learned_addr learned my addr 172.21.15.32:0/2589641461 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:48.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b8ffff700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9017a6e0 0x7f2b9017aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.280 7f2b94c6e700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b90179610 0x7f2b90179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b9546f700 1 -- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b90179610 msgr2=0x7f2b90179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b9546f700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b90179610 0x7f2b90179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b9546f700 1 -- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9017a6e0 msgr2=0x7f2b9017aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b9546f700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9017a6e0 0x7f2b9017aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b9546f700 1 -- 172.21.15.32:0/2589641461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9008e3d0 con 0x7f2b9017b100 2021-04-09T18:15:48.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b9546f700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b9017b100 0x7f2b9008db90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f2b8800bb90 tx=0x7f2b8800def0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:48.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b8d7fa700 1 -- 172.21.15.32:0/2589641461 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b8800b020 con 0x7f2b9017b100 2021-04-09T18:15:48.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b8d7fa700 1 -- 172.21.15.32:0/2589641461 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b88006030 con 0x7f2b9017b100 2021-04-09T18:15:48.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b8d7fa700 1 -- 172.21.15.32:0/2589641461 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b88006480 con 0x7f2b9017b100 2021-04-09T18:15:48.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9008e3d0 con 0x7f2b9017b100 2021-04-09T18:15:48.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.281 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9008e3d0 con 0x7f2b9017b100 2021-04-09T18:15:48.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.282 7f2b8d7fa700 1 -- 172.21.15.32:0/2589641461 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2b88006210 con 0x7f2b9017b100 2021-04-09T18:15:48.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.282 7f2b8d7fa700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b70018930 0x7f2b7001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.282 7f2b8d7fa700 1 -- 172.21.15.32:0/2589641461 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2b88034f80 con 0x7f2b9017b100 2021-04-09T18:15:48.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.282 7f2b94c6e700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b70018930 0x7f2b7001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.282 7f2b94c6e700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b70018930 0x7f2b7001ae10 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:48.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.301 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9017a1b0 con 0x7f2b9017b100 2021-04-09T18:15:48.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.305 7f2b8d7fa700 1 -- 172.21.15.32:0/2589641461 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2b88000f90 con 0x7f2b9017b100 2021-04-09T18:15:48.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.496 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b9017a1b0 con 0x7f2b9017b100 2021-04-09T18:15:48.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.496 7f2b8d7fa700 1 -- 172.21.15.32:0/2589641461 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2776 (secure 0 0 0) 0x7f2b88006030 con 0x7f2b9017b100 2021-04-09T18:15:48.540 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 45 2021-04-09T18:15:48.540 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:48.541 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":45,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3388525530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3388525530},{"type":"v1","addr":"172.21.15.32:6833","nonce":3388525530}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2165727103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2165727103},{"type":"v1","addr":"172.21.15.32:6835","nonce":2165727103}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":41},{"gid":55067,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3036306475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3036306475},{"type":"v1","addr":"172.21.15.86:6835","nonce":3036306475}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:15:46.073623","modified":"2021-04-09 18:15:48.082998","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5500":{"gid":5500,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6837/4251594905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4251594905},{"type":"v1","addr":"172.21.15.86:6837","nonce":4251594905}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-04-09T18:15:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b70018930 msgr2=0x7f2b7001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b70018930 0x7f2b7001ae10 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b9017b100 msgr2=0x7f2b9008db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b9017b100 0x7f2b9008db90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f2b8800bb90 tx=0x7f2b8800def0).stop 2021-04-09T18:15:48.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 shutdown_connections 2021-04-09T18:15:48.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b70018930 0x7f2b7001ae10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b9017b100 0x7f2b9008db90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9017a6e0 0x7f2b9017aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 --2- 172.21.15.32:0/2589641461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b90179610 0x7f2b90179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 >> 172.21.15.32:0/2589641461 conn(0x7f2b90190550 msgr2=0x7f2b901b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:48.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 shutdown_connections 2021-04-09T18:15:48.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.499 7f2b96e52700 1 -- 172.21.15.32:0/2589641461 wait complete. 2021-04-09T18:15:48.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:15:48.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4533159700 1 Processor -- start 2021-04-09T18:15:48.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4533159700 1 -- start start 2021-04-09T18:15:48.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4533159700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f452c0cffa0 0x7f452c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4533159700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c0d7800 0x7f452c0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4533159700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c0d7280 con 0x7f452c0d8220 2021-04-09T18:15:48.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4533159700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c0d91e0 con 0x7f452c0cffa0 2021-04-09T18:15:48.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4533159700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c10dfe0 con 0x7f452c0d7800 2021-04-09T18:15:48.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4530f75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f452c0cffa0 0x7f452c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4530f75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f452c0cffa0 0x7f452c0d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44360/0 (socket says 172.21.15.32:44360) 2021-04-09T18:15:48.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.786 7f4530f75700 1 -- 172.21.15.32:0/950998669 learned_addr learned my addr 172.21.15.32:0/950998669 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:48.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f452bfff700 1 --2- 172.21.15.32:0/950998669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c0d7800 0x7f452c0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f4530f75700 1 -- 172.21.15.32:0/950998669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c0d7800 msgr2=0x7f452c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:48.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f4530f75700 1 --2- 172.21.15.32:0/950998669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c0d7800 0x7f452c0d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f4530f75700 1 -- 172.21.15.32:0/950998669 >> v1:172.21.15.86:6789/0 conn(0x7f452c0d8220 legacy=0x7f452c0d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:48.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f4530f75700 1 -- 172.21.15.32:0/950998669 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c10e200 con 0x7f452c0cffa0 2021-04-09T18:15:48.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f452bfff700 1 --2- 172.21.15.32:0/950998669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c0d7800 0x7f452c0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:48.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f4530f75700 1 --2- 172.21.15.32:0/950998669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0cffa0 0x7f452c0d2440 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f4514006700 tx=0x7f4514009b80).ready entity=mon.0 client_cookie=ecf210005c2ca6d4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:48.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f452b7fe700 1 -- 172.21.15.32:0/950998669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f451400c0c0 con 0x7f452c0cffa0 2021-04-09T18:15:48.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f452b7fe700 1 -- 172.21.15.32:0/950998669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451400c440 con 0x7f452c0cffa0 2021-04-09T18:15:48.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f452b7fe700 1 -- 172.21.15.32:0/950998669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f451400c920 con 0x7f452c0cffa0 2021-04-09T18:15:48.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f4533159700 1 -- 172.21.15.32:0/950998669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0cffa0 msgr2=0x7f452c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.787 7f4533159700 1 --2- 172.21.15.32:0/950998669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0cffa0 0x7f452c0d2440 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f4514006700 tx=0x7f4514009b80).stop 2021-04-09T18:15:48.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.788 7f4533159700 1 -- 172.21.15.32:0/950998669 shutdown_connections 2021-04-09T18:15:48.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.788 7f4533159700 1 --2- 172.21.15.32:0/950998669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c0d7800 0x7f452c0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.788 7f4533159700 1 --2- 172.21.15.32:0/950998669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0cffa0 0x7f452c0d2440 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.788 7f4533159700 1 -- 172.21.15.32:0/950998669 >> 172.21.15.32:0/950998669 conn(0x7f452c0ce5c0 msgr2=0x7f452c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.788 7f4533159700 1 -- 172.21.15.32:0/950998669 shutdown_connections 2021-04-09T18:15:48.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.788 7f4533159700 1 -- 172.21.15.32:0/950998669 wait complete. 2021-04-09T18:15:48.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.808 7f4533159700 1 Processor -- start 2021-04-09T18:15:48.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.808 7f4533159700 1 -- start start 2021-04-09T18:15:48.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4533159700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0a80c0 0x7f452c17bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4533159700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c17b1d0 0x7f452c17b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4533159700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c08dc50 0x7f452c08e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:48.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4533159700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c17bc70 con 0x7f452c0a80c0 2021-04-09T18:15:48.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4533159700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c08e670 con 0x7f452c17b1d0 2021-04-09T18:15:48.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4533159700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c08e850 con 0x7f452c08dc50 2021-04-09T18:15:48.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4530f75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0a80c0 0x7f452c17bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4530f75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0a80c0 0x7f452c17bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44366/0 (socket says 172.21.15.32:44366) 2021-04-09T18:15:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4530f75700 1 -- 172.21.15.32:0/2449719505 learned_addr learned my addr 172.21.15.32:0/2449719505 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f452bfff700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c17b1d0 0x7f452c17b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4531776700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c08dc50 0x7f452c08e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4530f75700 1 -- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c08dc50 msgr2=0x7f452c08e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4530f75700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c08dc50 0x7f452c08e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:48.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4530f75700 1 -- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c17b1d0 msgr2=0x7f452c17b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4530f75700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c17b1d0 0x7f452c17b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4530f75700 1 -- 172.21.15.32:0/2449719505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c08c660 con 0x7f452c0a80c0 2021-04-09T18:15:49.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f452bfff700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c17b1d0 0x7f452c17b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:49.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4531776700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c08dc50 0x7f452c08e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.809 7f4530f75700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0a80c0 0x7f452c17bfe0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f451400b650 tx=0x7f451400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.810 7f45297fa700 1 -- 172.21.15.32:0/2449719505 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f451400bde0 con 0x7f452c0a80c0 2021-04-09T18:15:49.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.810 7f45297fa700 1 -- 172.21.15.32:0/2449719505 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45140091e0 con 0x7f452c0a80c0 2021-04-09T18:15:49.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.810 7f4533159700 1 -- 172.21.15.32:0/2449719505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c08c660 con 0x7f452c0a80c0 2021-04-09T18:15:49.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.810 7f45297fa700 1 -- 172.21.15.32:0/2449719505 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45140022b0 con 0x7f452c0a80c0 2021-04-09T18:15:49.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.810 7f4533159700 1 -- 172.21.15.32:0/2449719505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c08c660 con 0x7f452c0a80c0 2021-04-09T18:15:49.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.810 7f45297fa700 1 -- 172.21.15.32:0/2449719505 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4514000c90 con 0x7f452c0a80c0 2021-04-09T18:15:49.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.810 7f45297fa700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45100188c0 0x7f451001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:49.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.810 7f45297fa700 1 -- 172.21.15.32:0/2449719505 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4514032a20 con 0x7f452c0a80c0 2021-04-09T18:15:49.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.811 7f452bfff700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45100188c0 0x7f451001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:49.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.811 7f452bfff700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45100188c0 0x7f451001adc0 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:49.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.830 7f4533159700 1 -- 172.21.15.32:0/2449719505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452c1b70c0 con 0x7f452c0a80c0 2021-04-09T18:15:49.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:48.834 7f45297fa700 1 -- 172.21.15.32:0/2449719505 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f45140091e0 con 0x7f452c0a80c0 2021-04-09T18:15:49.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.025 7f4533159700 1 -- 172.21.15.32:0/2449719505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f452c1b70c0 con 0x7f452c0a80c0 2021-04-09T18:15:49.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.026 7f45297fa700 1 -- 172.21.15.32:0/2449719505 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f4514000c90 con 0x7f452c0a80c0 2021-04-09T18:15:49.089 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:49.090 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":54,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:15:47.075022","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:15:44.773626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-04-09 18:15:45.435836","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:15:49.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 -- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45100188c0 msgr2=0x7f451001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45100188c0 0x7f451001adc0 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 -- 172.21.15.32:0/2449719505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0a80c0 msgr2=0x7f452c17bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0a80c0 0x7f452c17bfe0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f451400b650 tx=0x7f451400b9f0).stop 2021-04-09T18:15:49.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 -- 172.21.15.32:0/2449719505 shutdown_connections 2021-04-09T18:15:49.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45100188c0 0x7f451001adc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c08dc50 0x7f452c08e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c17b1d0 0x7f452c17b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 --2- 172.21.15.32:0/2449719505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0a80c0 0x7f452c17bfe0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 -- 172.21.15.32:0/2449719505 >> 172.21.15.32:0/2449719505 conn(0x7f452c189b00 msgr2=0x7f452c1921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:49.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 -- 172.21.15.32:0/2449719505 shutdown_connections 2021-04-09T18:15:49.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.028 7f4533159700 1 -- 172.21.15.32:0/2449719505 wait complete. 2021-04-09T18:15:49.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:15:49.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.379 7fb877e7c700 1 Processor -- start 2021-04-09T18:15:49.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.379 7fb877e7c700 1 -- start start 2021-04-09T18:15:49.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.380 7fb877e7c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb8700cffb0 0x7fb870087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.380 7fb877e7c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb870088460 con 0x7fb8700d7c50 2021-04-09T18:15:49.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.380 7fb877e7c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8700d8620 con 0x7fb8700d8050 2021-04-09T18:15:49.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.380 7fb877e7c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8701b0c80 con 0x7fb8700cffb0 2021-04-09T18:15:49.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.380 7fb875497700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb8700d7c50 0x7fb8701b0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53968/0 (socket says 172.21.15.32:53968) 2021-04-09T18:15:49.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.380 7fb875497700 1 -- 172.21.15.32:0/2903505054 learned_addr learned my addr 172.21.15.32:0/2903505054 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:49.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.380 7fb875c98700 1 --2- 172.21.15.32:0/2903505054 >> v2:172.21.15.86:3300/0 conn(0x7fb8700cffb0 0x7fb870087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:49.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.380 7fb874c96700 1 -- 172.21.15.32:0/2903505054 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111821129 0 0) 0x7fb8600009b0 con 0x7fb8700d7c50 2021-04-09T18:15:49.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb874c96700 1 -- 172.21.15.32:0/2903505054 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c002fe0 con 0x7fb8700d7c50 2021-04-09T18:15:49.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb874c96700 1 -- 172.21.15.32:0/2903505054 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688022168 0 0) 0x7fb868000d30 con 0x7fb8700d8050 2021-04-09T18:15:49.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb874c96700 1 -- 172.21.15.32:0/2903505054 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c003020 con 0x7fb8700d8050 2021-04-09T18:15:49.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb875c98700 1 -- 172.21.15.32:0/2903505054 >> v1:172.21.15.86:6789/0 conn(0x7fb8700d8050 legacy=0x7fb87010d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb875c98700 1 -- 172.21.15.32:0/2903505054 >> v1:172.21.15.32:6789/0 conn(0x7fb8700d7c50 legacy=0x7fb8701b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb875c98700 1 -- 172.21.15.32:0/2903505054 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8700d91f0 con 0x7fb8700cffb0 2021-04-09T18:15:49.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb875c98700 1 --2- 172.21.15.32:0/2903505054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8700cffb0 0x7fb870087e40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fb86c006650 tx=0x7fb86c009a80).ready entity=mon.1 client_cookie=4c57cbfefec960c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:49.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb874c96700 1 -- 172.21.15.32:0/2903505054 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb86c0009b0 con 0x7fb8700cffb0 2021-04-09T18:15:49.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb874c96700 1 -- 172.21.15.32:0/2903505054 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c00c0c0 con 0x7fb8700cffb0 2021-04-09T18:15:49.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb874c96700 1 -- 172.21.15.32:0/2903505054 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb86c00c510 con 0x7fb8700cffb0 2021-04-09T18:15:49.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb877e7c700 1 -- 172.21.15.32:0/2903505054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8700cffb0 msgr2=0x7fb870087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.381 7fb877e7c700 1 --2- 172.21.15.32:0/2903505054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8700cffb0 0x7fb870087e40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fb86c006650 tx=0x7fb86c009a80).stop 2021-04-09T18:15:49.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.382 7fb877e7c700 1 -- 172.21.15.32:0/2903505054 shutdown_connections 2021-04-09T18:15:49.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.382 7fb877e7c700 1 --2- 172.21.15.32:0/2903505054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8700cffb0 0x7fb870087e40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.382 7fb877e7c700 1 -- 172.21.15.32:0/2903505054 >> 172.21.15.32:0/2903505054 conn(0x7fb8700ce5b0 msgr2=0x7fb8700cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:49.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.382 7fb877e7c700 1 -- 172.21.15.32:0/2903505054 shutdown_connections 2021-04-09T18:15:49.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.382 7fb877e7c700 1 -- 172.21.15.32:0/2903505054 wait complete. 2021-04-09T18:15:49.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.402 7fb877e7c700 1 Processor -- start 2021-04-09T18:15:49.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.402 7fb877e7c700 1 -- start start 2021-04-09T18:15:49.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb877e7c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8700c4080 0x7fb8700958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:49.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb877e7c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb870094f20 0x7fb870182540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:49.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb877e7c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb870095320 0x7fb870182b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:49.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb877e7c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8701832c0 con 0x7fb870095320 2021-04-09T18:15:49.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb877e7c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb870181fc0 con 0x7fb8700c4080 2021-04-09T18:15:49.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb877e7c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700c4480 con 0x7fb870094f20 2021-04-09T18:15:49.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb876499700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb870095320 0x7fb870182b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:49.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb875497700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb870094f20 0x7fb870182540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb876499700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb870095320 0x7fb870182b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44384/0 (socket says 172.21.15.32:44384) 2021-04-09T18:15:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb876499700 1 -- 172.21.15.32:0/3244673436 learned_addr learned my addr 172.21.15.32:0/3244673436 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb875497700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb870094f20 0x7fb870182540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:51608/0 (socket says 172.21.15.32:51608) 2021-04-09T18:15:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb875c98700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8700c4080 0x7fb8700958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb876499700 1 -- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb870094f20 msgr2=0x7fb870182540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb876499700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb870094f20 0x7fb870182540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb876499700 1 -- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8700c4080 msgr2=0x7fb8700958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb876499700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8700c4080 0x7fb8700958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb876499700 1 -- 172.21.15.32:0/3244673436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8700bc4a0 con 0x7fb870095320 2021-04-09T18:15:49.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb875497700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb870094f20 0x7fb870182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb876499700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb870095320 0x7fb870182b60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fb868009980 tx=0x7fb86800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb8667fc700 1 -- 172.21.15.32:0/3244673436 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb86800be10 con 0x7fb870095320 2021-04-09T18:15:49.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb8667fc700 1 -- 172.21.15.32:0/3244673436 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868005870 con 0x7fb870095320 2021-04-09T18:15:49.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.403 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8700bc4a0 con 0x7fb870095320 2021-04-09T18:15:49.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.404 7fb8667fc700 1 -- 172.21.15.32:0/3244673436 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb868005d50 con 0x7fb870095320 2021-04-09T18:15:49.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.404 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8700bc4a0 con 0x7fb870095320 2021-04-09T18:15:49.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.404 7fb8667fc700 1 -- 172.21.15.32:0/3244673436 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb868005a50 con 0x7fb870095320 2021-04-09T18:15:49.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.404 7fb8667fc700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb858018930 0x7fb85801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:49.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.404 7fb8667fc700 1 -- 172.21.15.32:0/3244673436 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb868032d80 con 0x7fb870095320 2021-04-09T18:15:49.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.405 7fb875c98700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb858018930 0x7fb85801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:49.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.405 7fb875c98700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb858018930 0x7fb85801ae10 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:49.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.424 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb870093ff0 con 0x7fb870095320 2021-04-09T18:15:49.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.428 7fb8667fc700 1 -- 172.21.15.32:0/3244673436 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb86800be10 con 0x7fb870095320 2021-04-09T18:15:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.612 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb8700d8620 con 0x7fb870095320 2021-04-09T18:15:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.616 7fb8667fc700 1 -- 172.21.15.32:0/3244673436 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb868005870 con 0x7fb870095320 2021-04-09T18:15:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:15:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb858018930 msgr2=0x7fb85801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb858018930 0x7fb85801ae10 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb870095320 msgr2=0x7fb870182b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb870095320 0x7fb870182b60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fb868009980 tx=0x7fb86800bcd0).stop 2021-04-09T18:15:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 shutdown_connections 2021-04-09T18:15:49.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb858018930 0x7fb85801ae10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb870095320 0x7fb870182b60 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb870094f20 0x7fb870182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 --2- 172.21.15.32:0/3244673436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8700c4080 0x7fb8700958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.621 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 >> 172.21.15.32:0/3244673436 conn(0x7fb870099610 msgr2=0x7fb8700acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.622 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 shutdown_connections 2021-04-09T18:15:49.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.622 7fb877e7c700 1 -- 172.21.15.32:0/3244673436 wait complete. 2021-04-09T18:15:49.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:15:49.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.843 7f08b497b700 1 Processor -- start 2021-04-09T18:15:49.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.843 7f08b497b700 1 -- start start 2021-04-09T18:15:49.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08b497b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08b00d7c50 0x7f08b01b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:49.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08b497b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8050 0x7f08b010d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:49.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08b497b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00d8620 con 0x7f08b00cffb0 2021-04-09T18:15:49.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08b497b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b00d91f0 con 0x7f08b00d7c50 2021-04-09T18:15:49.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08b497b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00d2800 con 0x7f08b00d8050 2021-04-09T18:15:49.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08ade1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08b00d7c50 0x7f08b01b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:49.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08ae61c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f08b00cffb0 0x7f08b0087e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53978/0 (socket says 172.21.15.32:53978) 2021-04-09T18:15:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08ade1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08b00d7c50 0x7f08b01b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44392/0 (socket says 172.21.15.32:44392) 2021-04-09T18:15:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08ae61c700 1 -- 172.21.15.32:0/3906726371 learned_addr learned my addr 172.21.15.32:0/3906726371 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.844 7f08aee1d700 1 --2- 172.21.15.32:0/3906726371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8050 0x7f08b010d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:49.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08ade1b700 1 -- 172.21.15.32:0/3906726371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8050 msgr2=0x7f08b010d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:49.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08ade1b700 1 --2- 172.21.15.32:0/3906726371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8050 0x7f08b010d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08ade1b700 1 -- 172.21.15.32:0/3906726371 >> v1:172.21.15.32:6789/0 conn(0x7f08b00cffb0 legacy=0x7f08b0087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08ad61a700 1 -- 172.21.15.32:0/3906726371 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157781385 0 0) 0x7f08a0000ce0 con 0x7f08b00cffb0 2021-04-09T18:15:49.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08ade1b700 1 -- 172.21.15.32:0/3906726371 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b01b0ab0 con 0x7f08b00d7c50 2021-04-09T18:15:49.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08ade1b700 1 --2- 172.21.15.32:0/3906726371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00d7c50 0x7f08b01b0490 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f089c006650 tx=0x7f089c009a00).ready entity=mon.0 client_cookie=5f0356f8ac79f709 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:49.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08ad61a700 1 -- 172.21.15.32:0/3906726371 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f089c009df0 con 0x7f08b00d7c50 2021-04-09T18:15:49.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08ad61a700 1 -- 172.21.15.32:0/3906726371 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c0009b0 con 0x7f08b00d7c50 2021-04-09T18:15:49.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08ad61a700 1 -- 172.21.15.32:0/3906726371 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f089c00c2b0 con 0x7f08b00d7c50 2021-04-09T18:15:49.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08b497b700 1 -- 172.21.15.32:0/3906726371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00d7c50 msgr2=0x7f08b01b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:49.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.845 7f08b497b700 1 --2- 172.21.15.32:0/3906726371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00d7c50 0x7f08b01b0490 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f089c006650 tx=0x7f089c009a00).stop 2021-04-09T18:15:49.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.846 7f08b497b700 1 -- 172.21.15.32:0/3906726371 shutdown_connections 2021-04-09T18:15:49.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.846 7f08b497b700 1 --2- 172.21.15.32:0/3906726371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8050 0x7f08b010d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.846 7f08b497b700 1 --2- 172.21.15.32:0/3906726371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00d7c50 0x7f08b01b0490 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:49.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.846 7f08b497b700 1 -- 172.21.15.32:0/3906726371 >> 172.21.15.32:0/3906726371 conn(0x7f08b00ce5b0 msgr2=0x7f08b00cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:49.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.846 7f08b497b700 1 -- 172.21.15.32:0/3906726371 shutdown_connections 2021-04-09T18:15:49.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.846 7f08b497b700 1 -- 172.21.15.32:0/3906726371 wait complete. 2021-04-09T18:15:49.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.866 7f08b497b700 1 Processor -- start 2021-04-09T18:15:50.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.866 7f08b497b700 1 -- start start 2021-04-09T18:15:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.866 7f08b497b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00b1950 0x7f08b0183e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08b497b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b01795e0 0x7f08b0094eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08b497b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b01799e0 0x7f08b00954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08b497b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00b13d0 con 0x7f08b01799e0 2021-04-09T18:15:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08b497b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00b1d50 con 0x7f08b01795e0 2021-04-09T18:15:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08b497b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b019e4a0 con 0x7f08b00b1950 2021-04-09T18:15:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08aee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b01799e0 0x7f08b00954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08aee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b01799e0 0x7f08b00954d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44398/0 (socket says 172.21.15.32:44398) 2021-04-09T18:15:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08aee1d700 1 -- 172.21.15.32:0/520647405 learned_addr learned my addr 172.21.15.32:0/520647405 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:50.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08ade1b700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b01795e0 0x7f08b0094eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08ae61c700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00b1950 0x7f08b0183e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08aee1d700 1 -- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00b1950 msgr2=0x7f08b0183e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08aee1d700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00b1950 0x7f08b0183e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08aee1d700 1 -- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b01795e0 msgr2=0x7f08b0094eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08aee1d700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b01795e0 0x7f08b0094eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08aee1d700 1 -- 172.21.15.32:0/520647405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b00b0180 con 0x7f08b01799e0 2021-04-09T18:15:50.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08ade1b700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b01795e0 0x7f08b0094eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:50.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08ae61c700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00b1950 0x7f08b0183e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:50.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08aee1d700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b01799e0 0x7f08b00954d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f08a8006f40 tx=0x7f08a800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:50.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08a6ffd700 1 -- 172.21.15.32:0/520647405 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a80011f0 con 0x7f08b01799e0 2021-04-09T18:15:50.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08a6ffd700 1 -- 172.21.15.32:0/520647405 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a8003030 con 0x7f08b01799e0 2021-04-09T18:15:50.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08b497b700 1 -- 172.21.15.32:0/520647405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b00b0180 con 0x7f08b01799e0 2021-04-09T18:15:50.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08a6ffd700 1 -- 172.21.15.32:0/520647405 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a80016a0 con 0x7f08b01799e0 2021-04-09T18:15:50.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.867 7f08b497b700 1 -- 172.21.15.32:0/520647405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b00b0180 con 0x7f08b01799e0 2021-04-09T18:15:50.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.868 7f08a6ffd700 1 -- 172.21.15.32:0/520647405 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f08a80056b0 con 0x7f08b01799e0 2021-04-09T18:15:50.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.868 7f08a6ffd700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f088c0188c0 0x7f088c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.868 7f08a6ffd700 1 -- 172.21.15.32:0/520647405 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f08a8032d70 con 0x7f08b01799e0 2021-04-09T18:15:50.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.869 7f08ae61c700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f088c0188c0 0x7f088c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.869 7f08ae61c700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f088c0188c0 0x7f088c01adc0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:50.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.888 7f08b497b700 1 -- 172.21.15.32:0/520647405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b0093fc0 con 0x7f08b01799e0 2021-04-09T18:15:50.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:49.892 7f08a6ffd700 1 -- 172.21.15.32:0/520647405 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f08a8000f40 con 0x7f08b01799e0 2021-04-09T18:15:50.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.078 7f08b497b700 1 -- 172.21.15.32:0/520647405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f08b00d8620 con 0x7f08b01799e0 2021-04-09T18:15:50.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.081 7f08a6ffd700 1 -- 172.21.15.32:0/520647405 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v19) v1 ==== 184+0+0 (secure 0 0 0) 0x7f08a80056b0 con 0x7f08b01799e0 2021-04-09T18:15:50.115 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:15:50.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 -- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f088c0188c0 msgr2=0x7f088c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f088c0188c0 0x7f088c01adc0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 -- 172.21.15.32:0/520647405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b01799e0 msgr2=0x7f08b00954d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b01799e0 0x7f08b00954d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f08a8006f40 tx=0x7f08a800bd40).stop 2021-04-09T18:15:50.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 -- 172.21.15.32:0/520647405 shutdown_connections 2021-04-09T18:15:50.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f088c0188c0 0x7f088c01adc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b01799e0 0x7f08b00954d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b01795e0 0x7f08b0094eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 --2- 172.21.15.32:0/520647405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00b1950 0x7f08b0183e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 -- 172.21.15.32:0/520647405 >> 172.21.15.32:0/520647405 conn(0x7f08b01889b0 msgr2=0x7f08b0189210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:50.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 -- 172.21.15.32:0/520647405 shutdown_connections 2021-04-09T18:15:50.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.087 7f08b497b700 1 -- 172.21.15.32:0/520647405 wait complete. 2021-04-09T18:15:50.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:50.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.280 7f390c037700 1 Processor -- start 2021-04-09T18:15:50.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.280 7f390c037700 1 -- start start 2021-04-09T18:15:50.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f390c037700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39040d7ca0 0x7f39040d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f390c037700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390410da00 0x7f390410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f390c037700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390410d480 con 0x7f390410d010 2021-04-09T18:15:50.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f390c037700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39040d9220 con 0x7f39040d7ca0 2021-04-09T18:15:50.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f390c037700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040d86c0 con 0x7f390410da00 2021-04-09T18:15:50.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3909652700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39040d7ca0 0x7f39040d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3909e53700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f390410d010 0x7f39040d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53992/0 (socket says 172.21.15.32:53992) 2021-04-09T18:15:50.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3909e53700 1 -- 172.21.15.32:0/2516132662 learned_addr learned my addr 172.21.15.32:0/2516132662 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:50.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f390a654700 1 --2- 172.21.15.32:0/2516132662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390410da00 0x7f390410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3909652700 1 -- 172.21.15.32:0/2516132662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390410da00 msgr2=0x7f390410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:50.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3909652700 1 --2- 172.21.15.32:0/2516132662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390410da00 0x7f390410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3909652700 1 -- 172.21.15.32:0/2516132662 >> v1:172.21.15.32:6789/0 conn(0x7f390410d010 legacy=0x7f39040d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3909652700 1 -- 172.21.15.32:0/2516132662 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39040d1f70 con 0x7f39040d7ca0 2021-04-09T18:15:50.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3908e51700 1 -- 172.21.15.32:0/2516132662 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824564370 0 0) 0x7f3900000ce0 con 0x7f390410d010 2021-04-09T18:15:50.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f390a654700 1 --2- 172.21.15.32:0/2516132662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390410da00 0x7f390410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:50.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3909652700 1 --2- 172.21.15.32:0/2516132662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39040d7ca0 0x7f39040d80a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f38f4006650 tx=0x7f38f4009a80).ready entity=mon.0 client_cookie=bc3fd457141ef615 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:50.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.281 7f3908e51700 1 -- 172.21.15.32:0/2516132662 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38f40009b0 con 0x7f39040d7ca0 2021-04-09T18:15:50.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f3908e51700 1 -- 172.21.15.32:0/2516132662 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f400c0c0 con 0x7f39040d7ca0 2021-04-09T18:15:50.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f3908e51700 1 -- 172.21.15.32:0/2516132662 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38f40009b0 con 0x7f39040d7ca0 2021-04-09T18:15:50.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f390c037700 1 -- 172.21.15.32:0/2516132662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39040d7ca0 msgr2=0x7f39040d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f390c037700 1 --2- 172.21.15.32:0/2516132662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39040d7ca0 0x7f39040d80a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f38f4006650 tx=0x7f38f4009a80).stop 2021-04-09T18:15:50.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f390c037700 1 -- 172.21.15.32:0/2516132662 shutdown_connections 2021-04-09T18:15:50.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f390c037700 1 --2- 172.21.15.32:0/2516132662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390410da00 0x7f390410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f390c037700 1 --2- 172.21.15.32:0/2516132662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39040d7ca0 0x7f39040d80a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f390c037700 1 -- 172.21.15.32:0/2516132662 >> 172.21.15.32:0/2516132662 conn(0x7f39040ce5f0 msgr2=0x7f39040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:50.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f390c037700 1 -- 172.21.15.32:0/2516132662 shutdown_connections 2021-04-09T18:15:50.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.282 7f390c037700 1 -- 172.21.15.32:0/2516132662 wait complete. 2021-04-09T18:15:50.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f390c037700 1 Processor -- start 2021-04-09T18:15:50.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f390c037700 1 -- start start 2021-04-09T18:15:50.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f390c037700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390417ea50 0x7f390418c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f390c037700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f390418ce30 0x7f390417f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f390c037700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f390417fc50 0x7f390418f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f390c037700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390417f0b0 con 0x7f390418ce30 2021-04-09T18:15:50.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f390c037700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904180050 con 0x7f390417fc50 2021-04-09T18:15:50.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f390c037700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390418f970 con 0x7f390417ea50 2021-04-09T18:15:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f3909652700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f390418ce30 0x7f390417f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f3909652700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f390418ce30 0x7f390417f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44410/0 (socket says 172.21.15.32:44410) 2021-04-09T18:15:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f3909652700 1 -- 172.21.15.32:0/477438675 learned_addr learned my addr 172.21.15.32:0/477438675 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:50.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f3909e53700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390417ea50 0x7f390418c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.303 7f390a654700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f390417fc50 0x7f390418f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f3909652700 1 -- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390417ea50 msgr2=0x7f390418c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f3909652700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390417ea50 0x7f390418c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f3909652700 1 -- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f390417fc50 msgr2=0x7f390418f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f3909652700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f390417fc50 0x7f390418f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f3909652700 1 -- 172.21.15.32:0/477438675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390418fb90 con 0x7f390418ce30 2021-04-09T18:15:50.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f3909652700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f390418ce30 0x7f390417f630 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f38f4009650 tx=0x7f38f400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:50.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f38fa7fc700 1 -- 172.21.15.32:0/477438675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38f40093a0 con 0x7f390418ce30 2021-04-09T18:15:50.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f38fa7fc700 1 -- 172.21.15.32:0/477438675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f400bd50 con 0x7f390418ce30 2021-04-09T18:15:50.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f390c037700 1 -- 172.21.15.32:0/477438675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390418fb90 con 0x7f390418ce30 2021-04-09T18:15:50.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f38fa7fc700 1 -- 172.21.15.32:0/477438675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38f40052e0 con 0x7f390418ce30 2021-04-09T18:15:50.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.304 7f390c037700 1 -- 172.21.15.32:0/477438675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390418fb90 con 0x7f390418ce30 2021-04-09T18:15:50.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.305 7f38fa7fc700 1 -- 172.21.15.32:0/477438675 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f38f4006030 con 0x7f390418ce30 2021-04-09T18:15:50.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.305 7f38fa7fc700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38ec018930 0x7f38ec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:50.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.305 7f38fa7fc700 1 -- 172.21.15.32:0/477438675 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f38f40329e0 con 0x7f390418ce30 2021-04-09T18:15:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.305 7f3909e53700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38ec018930 0x7f38ec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:50.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.305 7f3909e53700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38ec018930 0x7f38ec01ae10 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.325 7f390c037700 1 -- 172.21.15.32:0/477438675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39041b7050 con 0x7f390418ce30 2021-04-09T18:15:50.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.328 7f38fa7fc700 1 -- 172.21.15.32:0/477438675 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f38f40009c0 con 0x7f390418ce30 2021-04-09T18:15:50.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.520 7f390c037700 1 -- 172.21.15.32:0/477438675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39041b7050 con 0x7f390418ce30 2021-04-09T18:15:50.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.520 7f38fa7fc700 1 -- 172.21.15.32:0/477438675 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2776 (secure 0 0 0) 0x7f38f400bd50 con 0x7f390418ce30 2021-04-09T18:15:50.583 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 45 2021-04-09T18:15:50.585 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:50.585 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":45,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3388525530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3388525530},{"type":"v1","addr":"172.21.15.32:6833","nonce":3388525530}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2165727103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2165727103},{"type":"v1","addr":"172.21.15.32:6835","nonce":2165727103}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":41},{"gid":55067,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3036306475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3036306475},{"type":"v1","addr":"172.21.15.86:6835","nonce":3036306475}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:15:46.073623","modified":"2021-04-09 18:15:48.082998","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5500":{"gid":5500,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6837/4251594905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4251594905},{"type":"v1","addr":"172.21.15.86:6837","nonce":4251594905}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-04-09T18:15:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 -- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38ec018930 msgr2=0x7f38ec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38ec018930 0x7f38ec01ae10 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 -- 172.21.15.32:0/477438675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f390418ce30 msgr2=0x7f390417f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:50.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f390418ce30 0x7f390417f630 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f38f4009650 tx=0x7f38f400b960).stop 2021-04-09T18:15:50.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 -- 172.21.15.32:0/477438675 shutdown_connections 2021-04-09T18:15:50.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38ec018930 0x7f38ec01ae10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f390418ce30 0x7f390417f630 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f390417fc50 0x7f390418f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 --2- 172.21.15.32:0/477438675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f390417ea50 0x7f390418c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 -- 172.21.15.32:0/477438675 >> 172.21.15.32:0/477438675 conn(0x7f390417db00 msgr2=0x7f39041a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 -- 172.21.15.32:0/477438675 shutdown_connections 2021-04-09T18:15:50.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:50.523 7f390c037700 1 -- 172.21.15.32:0/477438675 wait complete. 2021-04-09T18:15:51.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:15:46.073623', 'modified': '2021-04-09 18:15:48.082998', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5500': {'gid': 5500, 'name': 'd', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.86:6837/4251594905', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 4251594905}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 4251594905}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:51.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:15:51.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:15:51.276 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:15:51.276 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:15:51.277 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:15:51.277 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 5500, 2021-04-09T18:15:51.277 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:15:51.277 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:15:51.278 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 5.1728552409999997, 2021-04-09T18:15:51.278 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 45, 2021-04-09T18:15:51.278 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 54, 2021-04-09T18:15:51.278 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 54, 2021-04-09T18:15:51.278 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 15.524932700000001 2021-04-09T18:15:51.279 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:15:51.285 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5500, "want_state": "up:active", "state": "up:active", "rank_uptime": 5.1728552409999997, "mdsmap_epoch": 45, "osdmap_epoch": 54, "osdmap_epoch_barrier": 54, "uptime": 15.524932700000001 } 2021-04-09T18:15:51.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:15:51.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.421 7fee0db0f700 1 Processor -- start 2021-04-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.421 7fee0db0f700 1 -- start start 2021-04-09T18:15:51.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee0db0f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee0810d010 0x7fee080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:51.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee0db0f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0810d480 con 0x7fee080d1da0 2021-04-09T18:15:51.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee0db0f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0810e1f0 con 0x7fee080d8490 2021-04-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee0db0f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080d8d30 con 0x7fee0810d010 2021-04-09T18:15:51.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee077fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee0810d010 0x7fee080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:51.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee06ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fee080d1da0 0x7fee080d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54010/0 (socket says 172.21.15.32:54010) 2021-04-09T18:15:51.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee06ffd700 1 -- 172.21.15.32:0/3074042134 learned_addr learned my addr 172.21.15.32:0/3074042134 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:51.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee077fe700 1 -- 172.21.15.32:0/3074042134 >> v1:172.21.15.86:6789/0 conn(0x7fee080d8490 legacy=0x7fee0810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:51.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee077fe700 1 -- 172.21.15.32:0/3074042134 >> v1:172.21.15.32:6789/0 conn(0x7fee080d1da0 legacy=0x7fee080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee077fe700 1 -- 172.21.15.32:0/3074042134 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080d9220 con 0x7fee0810d010 2021-04-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.422 7fee067fc700 1 -- 172.21.15.32:0/3074042134 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625854390 0 0) 0x7fedfc000ce0 con 0x7fee080d1da0 2021-04-09T18:15:51.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee077fe700 1 --2- 172.21.15.32:0/3074042134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0810d010 0x7fee080d1780 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7fedf8006650 tx=0x7fedf8009a80).ready entity=mon.0 client_cookie=3ed86e3aeafadf1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:51.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee067fc700 1 -- 172.21.15.32:0/3074042134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf80009b0 con 0x7fee0810d010 2021-04-09T18:15:51.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee067fc700 1 -- 172.21.15.32:0/3074042134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf800c0c0 con 0x7fee0810d010 2021-04-09T18:15:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee067fc700 1 -- 172.21.15.32:0/3074042134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf800c510 con 0x7fee0810d010 2021-04-09T18:15:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee0db0f700 1 -- 172.21.15.32:0/3074042134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0810d010 msgr2=0x7fee080d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee0db0f700 1 --2- 172.21.15.32:0/3074042134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0810d010 0x7fee080d1780 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7fedf8006650 tx=0x7fedf8009a80).stop 2021-04-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee0db0f700 1 -- 172.21.15.32:0/3074042134 shutdown_connections 2021-04-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee0db0f700 1 --2- 172.21.15.32:0/3074042134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0810d010 0x7fee080d1780 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee0db0f700 1 -- 172.21.15.32:0/3074042134 >> 172.21.15.32:0/3074042134 conn(0x7fee080ce5f0 msgr2=0x7fee080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:51.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee0db0f700 1 -- 172.21.15.32:0/3074042134 shutdown_connections 2021-04-09T18:15:51.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.423 7fee0db0f700 1 -- 172.21.15.32:0/3074042134 wait complete. 2021-04-09T18:15:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.444 7fee0db0f700 1 Processor -- start 2021-04-09T18:15:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.444 7fee0db0f700 1 -- start start 2021-04-09T18:15:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.444 7fee0db0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0817e010 0x7fee0817e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.444 7fee0db0f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee0809f280 0x7fee0809f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.444 7fee0db0f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee080b1b00 0x7fee080b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.444 7fee0db0f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0809ed00 con 0x7fee0817e010 2021-04-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.444 7fee0db0f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0809fca0 con 0x7fee0809f280 2021-04-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.444 7fee0db0f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee080b2520 con 0x7fee080b1b00 2021-04-09T18:15:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.444 7fee077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0817e010 0x7fee0817e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0817e010 0x7fee0817e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44422/0 (socket says 172.21.15.32:44422) 2021-04-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee077fe700 1 -- 172.21.15.32:0/3796960854 learned_addr learned my addr 172.21.15.32:0/3796960854 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee06ffd700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee0809f280 0x7fee0809f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee07fff700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee080b1b00 0x7fee080b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee077fe700 1 -- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee080b1b00 msgr2=0x7fee080b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee077fe700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee080b1b00 0x7fee080b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee077fe700 1 -- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee0809f280 msgr2=0x7fee0809f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee077fe700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee0809f280 0x7fee0809f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee077fe700 1 -- 172.21.15.32:0/3796960854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080b2740 con 0x7fee0817e010 2021-04-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee077fe700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0817e010 0x7fee0817e410 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7fedf8009650 tx=0x7fedf800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fedeffff700 1 -- 172.21.15.32:0/3796960854 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf80093a0 con 0x7fee0817e010 2021-04-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fedeffff700 1 -- 172.21.15.32:0/3796960854 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8006070 con 0x7fee0817e010 2021-04-09T18:15:51.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fedeffff700 1 -- 172.21.15.32:0/3796960854 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf80052e0 con 0x7fee0817e010 2021-04-09T18:15:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee080b2740 con 0x7fee0817e010 2021-04-09T18:15:51.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.445 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee080b2740 con 0x7fee0817e010 2021-04-09T18:15:51.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.446 7fedeffff700 1 -- 172.21.15.32:0/3796960854 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fedf8009300 con 0x7fee0817e010 2021-04-09T18:15:51.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.446 7fedeffff700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fede8018930 0x7fede801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:51.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.446 7fedeffff700 1 -- 172.21.15.32:0/3796960854 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fedf8032a10 con 0x7fee0817e010 2021-04-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.446 7fee06ffd700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fede8018930 0x7fede801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.446 7fee06ffd700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fede8018930 0x7fede801adc0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:51.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.466 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0817cb20 con 0x7fee0817e010 2021-04-09T18:15:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.469 7fedeffff700 1 -- 172.21.15.32:0/3796960854 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fedf8006030 con 0x7fee0817e010 2021-04-09T18:15:51.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.657 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee0810d480 con 0x7fee0817e010 2021-04-09T18:15:51.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.658 7fedeffff700 1 -- 172.21.15.32:0/3796960854 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fedf8000940 con 0x7fee0817e010 2021-04-09T18:15:51.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:51.692 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":54,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:15:47.075022","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:15:44.773626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-04-09 18:15:45.435836","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:15:51.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fede8018930 msgr2=0x7fede801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:51.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fede8018930 0x7fede801adc0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0817e010 msgr2=0x7fee0817e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0817e010 0x7fee0817e410 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7fedf8009650 tx=0x7fedf800b8e0).stop 2021-04-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 shutdown_connections 2021-04-09T18:15:51.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fede8018930 0x7fede801adc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee080b1b00 0x7fee080b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee0809f280 0x7fee0809f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 --2- 172.21.15.32:0/3796960854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee0817e010 0x7fee0817e410 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.660 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 >> 172.21.15.32:0/3796960854 conn(0x7fee081965a0 msgr2=0x7fee0819dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.661 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 shutdown_connections 2021-04-09T18:15:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.661 7fee0db0f700 1 -- 172.21.15.32:0/3796960854 wait complete. 2021-04-09T18:15:51.754 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:15:44.773626', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:15:45.435836', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:15:51.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:15:51.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.905 7fc936fc6700 1 Processor -- start 2021-04-09T18:15:51.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.905 7fc936fc6700 1 -- start start 2021-04-09T18:15:51.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc936fc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9300d7ca0 0x7fc9300d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:51.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc936fc6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc93010da00 0x7fc93010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:51.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc936fc6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93010d480 con 0x7fc93010d010 2021-04-09T18:15:51.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc936fc6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9300d9220 con 0x7fc93010da00 2021-04-09T18:15:51.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc936fc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300d86c0 con 0x7fc9300d7ca0 2021-04-09T18:15:51.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc934de2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc93010d010 0x7fc9300d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54020/0 (socket says 172.21.15.32:54020) 2021-04-09T18:15:51.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc934de2700 1 -- 172.21.15.32:0/1054827759 learned_addr learned my addr 172.21.15.32:0/1054827759 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:51.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc92ffff700 1 --2- 172.21.15.32:0/1054827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9300d7ca0 0x7fc9300d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:51.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc9355e3700 1 --2- 172.21.15.32:0/1054827759 >> v2:172.21.15.86:3300/0 conn(0x7fc93010da00 0x7fc93010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:51.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc92f7fe700 1 -- 172.21.15.32:0/1054827759 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520438162 0 0) 0x7fc920000ce0 con 0x7fc93010d010 2021-04-09T18:15:51.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc92f7fe700 1 -- 172.21.15.32:0/1054827759 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc918002fe0 con 0x7fc93010d010 2021-04-09T18:15:51.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.906 7fc92f7fe700 1 -- 172.21.15.32:0/1054827759 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 779067585 0 0) 0x7fc9200010e0 con 0x7fc93010d010 2021-04-09T18:15:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc92ffff700 1 -- 172.21.15.32:0/1054827759 >> v2:172.21.15.86:3300/0 conn(0x7fc93010da00 msgr2=0x7fc93010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc92ffff700 1 --2- 172.21.15.32:0/1054827759 >> v2:172.21.15.86:3300/0 conn(0x7fc93010da00 0x7fc93010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc92ffff700 1 -- 172.21.15.32:0/1054827759 >> v1:172.21.15.32:6789/0 conn(0x7fc93010d010 legacy=0x7fc9300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc92ffff700 1 -- 172.21.15.32:0/1054827759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9300d1f70 con 0x7fc9300d7ca0 2021-04-09T18:15:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc92ffff700 1 --2- 172.21.15.32:0/1054827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9300d7ca0 0x7fc9300d80a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fc924006650 tx=0x7fc924009a80).ready entity=mon.2 client_cookie=9fc07e79ae5f6cad server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc92f7fe700 1 -- 172.21.15.32:0/1054827759 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9240009b0 con 0x7fc9300d7ca0 2021-04-09T18:15:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc92f7fe700 1 -- 172.21.15.32:0/1054827759 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92400c0c0 con 0x7fc9300d7ca0 2021-04-09T18:15:51.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc92f7fe700 1 -- 172.21.15.32:0/1054827759 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc92400c510 con 0x7fc9300d7ca0 2021-04-09T18:15:51.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc936fc6700 1 -- 172.21.15.32:0/1054827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9300d7ca0 msgr2=0x7fc9300d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:51.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.907 7fc936fc6700 1 --2- 172.21.15.32:0/1054827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9300d7ca0 0x7fc9300d80a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fc924006650 tx=0x7fc924009a80).stop 2021-04-09T18:15:51.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.908 7fc936fc6700 1 -- 172.21.15.32:0/1054827759 shutdown_connections 2021-04-09T18:15:51.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.908 7fc936fc6700 1 --2- 172.21.15.32:0/1054827759 >> v2:172.21.15.86:3300/0 conn(0x7fc93010da00 0x7fc93010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.908 7fc936fc6700 1 --2- 172.21.15.32:0/1054827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9300d7ca0 0x7fc9300d80a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:51.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.908 7fc936fc6700 1 -- 172.21.15.32:0/1054827759 >> 172.21.15.32:0/1054827759 conn(0x7fc9300ce5f0 msgr2=0x7fc9300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.908 7fc936fc6700 1 -- 172.21.15.32:0/1054827759 shutdown_connections 2021-04-09T18:15:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.908 7fc936fc6700 1 -- 172.21.15.32:0/1054827759 wait complete. 2021-04-09T18:15:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.928 7fc936fc6700 1 Processor -- start 2021-04-09T18:15:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.928 7fc936fc6700 1 -- start start 2021-04-09T18:15:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.928 7fc936fc6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc930179bf0 0x7fc930179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.928 7fc936fc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc930094f60 0x7fc930095360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.928 7fc936fc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc930095980 0x7fc93009ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.928 7fc936fc6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9301796f0 con 0x7fc930095980 2021-04-09T18:15:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.928 7fc936fc6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93017a610 con 0x7fc930179bf0 2021-04-09T18:15:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.928 7fc936fc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93009b230 con 0x7fc930094f60 2021-04-09T18:15:52.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc9355e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc930095980 0x7fc93009ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc9355e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc930095980 0x7fc93009ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44440/0 (socket says 172.21.15.32:44440) 2021-04-09T18:15:52.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc9355e3700 1 -- 172.21.15.32:0/2238891170 learned_addr learned my addr 172.21.15.32:0/2238891170 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:52.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc934de2700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc930179bf0 0x7fc930179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc92ffff700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc930094f60 0x7fc930095360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc9355e3700 1 -- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc930094f60 msgr2=0x7fc930095360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:52.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc9355e3700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc930094f60 0x7fc930095360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc9355e3700 1 -- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc930179bf0 msgr2=0x7fc930179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:52.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc9355e3700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc930179bf0 0x7fc930179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc9355e3700 1 -- 172.21.15.32:0/2238891170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93009b450 con 0x7fc930095980 2021-04-09T18:15:52.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc92ffff700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc930094f60 0x7fc930095360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:15:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc9355e3700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc930095980 0x7fc93009ac10 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fc92800b970 tx=0x7fc92800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc92d7fa700 1 -- 172.21.15.32:0/2238891170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc928000f90 con 0x7fc930095980 2021-04-09T18:15:52.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc92d7fa700 1 -- 172.21.15.32:0/2238891170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc928003030 con 0x7fc930095980 2021-04-09T18:15:52.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93009b450 con 0x7fc930095980 2021-04-09T18:15:52.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc92d7fa700 1 -- 172.21.15.32:0/2238891170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc928001400 con 0x7fc930095980 2021-04-09T18:15:52.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.929 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93009b450 con 0x7fc930095980 2021-04-09T18:15:52.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.930 7fc92d7fa700 1 -- 172.21.15.32:0/2238891170 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc928000e70 con 0x7fc930095980 2021-04-09T18:15:52.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.930 7fc92d7fa700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc910018930 0x7fc91001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.930 7fc92d7fa700 1 -- 172.21.15.32:0/2238891170 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc928032db0 con 0x7fc930095980 2021-04-09T18:15:52.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.930 7fc934de2700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc910018930 0x7fc91001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.931 7fc934de2700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc910018930 0x7fc91001adc0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:52.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.950 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93017a610 con 0x7fc930095980 2021-04-09T18:15:52.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:51.953 7fc92d7fa700 1 -- 172.21.15.32:0/2238891170 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc92801faf0 con 0x7fc930095980 2021-04-09T18:15:52.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.135 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc93017a610 con 0x7fc930095980 2021-04-09T18:15:52.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.135 7fc92d7fa700 1 -- 172.21.15.32:0/2238891170 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v54) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc928000df0 con 0x7fc930095980 2021-04-09T18:15:52.165 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:15:52.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc910018930 msgr2=0x7fc91001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:52.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc910018930 0x7fc91001adc0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc930095980 msgr2=0x7fc93009ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc930095980 0x7fc93009ac10 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fc92800b970 tx=0x7fc92800bd10).stop 2021-04-09T18:15:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 shutdown_connections 2021-04-09T18:15:52.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc910018930 0x7fc91001adc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc930095980 0x7fc93009ac10 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc930094f60 0x7fc930095360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 --2- 172.21.15.32:0/2238891170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc930179bf0 0x7fc930179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 >> 172.21.15.32:0/2238891170 conn(0x7fc9300c4290 msgr2=0x7fc9300c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:52.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 shutdown_connections 2021-04-09T18:15:52.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.138 7fc936fc6700 1 -- 172.21.15.32:0/2238891170 wait complete. 2021-04-09T18:15:52.177 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:15:52.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f69795dc700 1 Processor -- start 2021-04-09T18:15:52.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f69795dc700 1 -- start start 2021-04-09T18:15:52.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f69795dc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f697410d010 0x7f69740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f69795dc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69740d1da0 0x7f69740d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f69795dc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f69740d82c0 0x7f697410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f69795dc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697410e780 con 0x7f69740d82c0 2021-04-09T18:15:52.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f69795dc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69740d8d30 con 0x7f69740d1da0 2021-04-09T18:15:52.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f69795dc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69740d9220 con 0x7f697410d010 2021-04-09T18:15:52.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f6973ac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f69740d82c0 0x7f697410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f6973ac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f69740d82c0 0x7f697410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44446/0 (socket says 172.21.15.32:44446) 2021-04-09T18:15:52.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.340 7f6973ac0700 1 -- 172.21.15.32:0/3838057052 learned_addr learned my addr 172.21.15.32:0/3838057052 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f6972abe700 1 --2- 172.21.15.32:0/3838057052 >> v2:172.21.15.86:3300/0 conn(0x7f69740d1da0 0x7f69740d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f69732bf700 1 --2- 172.21.15.32:0/3838057052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f697410d010 0x7f69740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f6973ac0700 1 -- 172.21.15.32:0/3838057052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f697410d010 msgr2=0x7f69740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f6973ac0700 1 --2- 172.21.15.32:0/3838057052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f697410d010 0x7f69740d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f6973ac0700 1 -- 172.21.15.32:0/3838057052 >> v2:172.21.15.86:3300/0 conn(0x7f69740d1da0 msgr2=0x7f69740d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f6973ac0700 1 --2- 172.21.15.32:0/3838057052 >> v2:172.21.15.86:3300/0 conn(0x7f69740d1da0 0x7f69740d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f6973ac0700 1 -- 172.21.15.32:0/3838057052 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697410d480 con 0x7f69740d82c0 2021-04-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f69732bf700 1 --2- 172.21.15.32:0/3838057052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f697410d010 0x7f69740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f6973ac0700 1 --2- 172.21.15.32:0/3838057052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69740d82c0 0x7f697410da00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f696c006f00 tx=0x7f696c009f10).ready entity=mon.0 client_cookie=61957ab032ff3337 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:52.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f69722bd700 1 -- 172.21.15.32:0/3838057052 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f696c00b3b0 con 0x7f69740d82c0 2021-04-09T18:15:52.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f69722bd700 1 -- 172.21.15.32:0/3838057052 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696c00b3b0 con 0x7f69740d82c0 2021-04-09T18:15:52.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f69722bd700 1 -- 172.21.15.32:0/3838057052 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f696c00b730 con 0x7f69740d82c0 2021-04-09T18:15:52.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f69795dc700 1 -- 172.21.15.32:0/3838057052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69740d82c0 msgr2=0x7f697410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.341 7f69795dc700 1 --2- 172.21.15.32:0/3838057052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69740d82c0 0x7f697410da00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f696c006f00 tx=0x7f696c009f10).stop 2021-04-09T18:15:52.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.342 7f69795dc700 1 -- 172.21.15.32:0/3838057052 shutdown_connections 2021-04-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.342 7f69795dc700 1 --2- 172.21.15.32:0/3838057052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f69740d82c0 0x7f697410da00 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.342 7f69795dc700 1 --2- 172.21.15.32:0/3838057052 >> v2:172.21.15.86:3300/0 conn(0x7f69740d1da0 0x7f69740d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.342 7f69795dc700 1 --2- 172.21.15.32:0/3838057052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f697410d010 0x7f69740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.342 7f69795dc700 1 -- 172.21.15.32:0/3838057052 >> 172.21.15.32:0/3838057052 conn(0x7f69740ce5f0 msgr2=0x7f69740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:52.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.342 7f69795dc700 1 -- 172.21.15.32:0/3838057052 shutdown_connections 2021-04-09T18:15:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.342 7f69795dc700 1 -- 172.21.15.32:0/3838057052 wait complete. 2021-04-09T18:15:52.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.362 7f69795dc700 1 Processor -- start 2021-04-09T18:15:52.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.362 7f69795dc700 1 -- start start 2021-04-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f69795dc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69740a0a10 0x7f69740bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f69795dc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f697418ad00 0x7f697419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f69795dc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69741a03f0 0x7f6974095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f69795dc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697418b100 con 0x7f697418ad00 2021-04-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f69795dc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6974095860 con 0x7f69741a03f0 2021-04-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f69795dc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6974095a80 con 0x7f69740a0a10 2021-04-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6972abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f697418ad00 0x7f697419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6972abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f697418ad00 0x7f697419fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44450/0 (socket says 172.21.15.32:44450) 2021-04-09T18:15:52.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6972abe700 1 -- 172.21.15.32:0/145429425 learned_addr learned my addr 172.21.15.32:0/145429425 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6973ac0700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69741a03f0 0x7f6974095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f69732bf700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69740a0a10 0x7f69740bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6972abe700 1 -- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69740a0a10 msgr2=0x7f69740bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6972abe700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69740a0a10 0x7f69740bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6972abe700 1 -- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69741a03f0 msgr2=0x7f6974095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:52.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6972abe700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69741a03f0 0x7f6974095240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6972abe700 1 -- 172.21.15.32:0/145429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6974189250 con 0x7f697418ad00 2021-04-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f6972abe700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f697418ad00 0x7f697419fdd0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f695c00b530 tx=0x7f695c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f696bfff700 1 -- 172.21.15.32:0/145429425 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f695c00bcc0 con 0x7f697418ad00 2021-04-09T18:15:52.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f696bfff700 1 -- 172.21.15.32:0/145429425 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c002030 con 0x7f697418ad00 2021-04-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f69795dc700 1 -- 172.21.15.32:0/145429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6974189250 con 0x7f697418ad00 2021-04-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f696bfff700 1 -- 172.21.15.32:0/145429425 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f695c002480 con 0x7f697418ad00 2021-04-09T18:15:52.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.363 7f69795dc700 1 -- 172.21.15.32:0/145429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6974189250 con 0x7f697418ad00 2021-04-09T18:15:52.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.364 7f696bfff700 1 -- 172.21.15.32:0/145429425 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f695c002210 con 0x7f697418ad00 2021-04-09T18:15:52.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.364 7f696bfff700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6958018930 0x7f695801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:52.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.364 7f696bfff700 1 -- 172.21.15.32:0/145429425 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f695c0329f0 con 0x7f697418ad00 2021-04-09T18:15:52.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.364 7f69732bf700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6958018930 0x7f695801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:52.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.365 7f69732bf700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6958018930 0x7f695801ae10 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:52.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.384 7f69795dc700 1 -- 172.21.15.32:0/145429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6974093d50 con 0x7f697418ad00 2021-04-09T18:15:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.388 7f696bfff700 1 -- 172.21.15.32:0/145429425 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f695c002240 con 0x7f697418ad00 2021-04-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.568 7f69795dc700 1 -- 172.21.15.32:0/145429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f697410d480 con 0x7f697418ad00 2021-04-09T18:15:52.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.568 7f696bfff700 1 -- 172.21.15.32:0/145429425 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v54) v1 ==== 104+0+10 (secure 0 0 0) 0x7f695c002030 con 0x7f697418ad00 2021-04-09T18:15:52.739 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:15:52.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 -- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6958018930 msgr2=0x7f695801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:52.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6958018930 0x7f695801ae10 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 -- 172.21.15.32:0/145429425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f697418ad00 msgr2=0x7f697419fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:52.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f697418ad00 0x7f697419fdd0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f695c00b530 tx=0x7f695c00b8d0).stop 2021-04-09T18:15:52.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 -- 172.21.15.32:0/145429425 shutdown_connections 2021-04-09T18:15:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6958018930 0x7f695801ae10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f697418ad00 0x7f697419fdd0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69741a03f0 0x7f6974095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 --2- 172.21.15.32:0/145429425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69740a0a10 0x7f69740bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:52.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 -- 172.21.15.32:0/145429425 >> 172.21.15.32:0/145429425 conn(0x7f69740975f0 msgr2=0x7f6974096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:52.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 -- 172.21.15.32:0/145429425 shutdown_connections 2021-04-09T18:15:52.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:52.571 7f69795dc700 1 -- 172.21.15.32:0/145429425 wait complete. 2021-04-09T18:15:52.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:15:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.009 7f568cced700 1 Processor -- start 2021-04-09T18:15:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.009 7f568cced700 1 -- start start 2021-04-09T18:15:53.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f568cced700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f56880d6ca0 0x7f56880d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f568cced700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56880d2460 0x7f568810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f568cced700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56880d72a0 con 0x7f56880cffc0 2021-04-09T18:15:53.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f568cced700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56880d70a0 con 0x7f56880d2460 2021-04-09T18:15:53.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f568cced700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56880d9200 con 0x7f56880d6ca0 2021-04-09T18:15:53.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f56872e9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f56880d2460 0x7f568810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f5686ae8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f56880cffc0 0x7f56880d7820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54048/0 (socket says 172.21.15.32:54048) 2021-04-09T18:15:53.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f5686ae8700 1 -- 172.21.15.32:0/2081520262 learned_addr learned my addr 172.21.15.32:0/2081520262 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:53.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f56862e7700 1 --2- 172.21.15.32:0/2081520262 >> v2:172.21.15.86:3300/0 conn(0x7f56880d6ca0 0x7f56880d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f56872e9700 1 -- 172.21.15.32:0/2081520262 >> v2:172.21.15.86:3300/0 conn(0x7f56880d6ca0 msgr2=0x7f56880d8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f56872e9700 1 --2- 172.21.15.32:0/2081520262 >> v2:172.21.15.86:3300/0 conn(0x7f56880d6ca0 0x7f56880d8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f56872e9700 1 -- 172.21.15.32:0/2081520262 >> v1:172.21.15.32:6789/0 conn(0x7f56880cffc0 legacy=0x7f56880d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f5685ae6700 1 -- 172.21.15.32:0/2081520262 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632722535 0 0) 0x7f5678000ce0 con 0x7f56880cffc0 2021-04-09T18:15:53.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f56872e9700 1 -- 172.21.15.32:0/2081520262 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56880d2860 con 0x7f56880d2460 2021-04-09T18:15:53.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.010 7f56872e9700 1 --2- 172.21.15.32:0/2081520262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56880d2460 0x7f568810cff0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f5680006e30 tx=0x7f5680009e60).ready entity=mon.0 client_cookie=e7cc88f82b3a3f97 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.011 7f5685ae6700 1 -- 172.21.15.32:0/2081520262 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5680006a50 con 0x7f56880d2460 2021-04-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.011 7f5685ae6700 1 -- 172.21.15.32:0/2081520262 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568000b300 con 0x7f56880d2460 2021-04-09T18:15:53.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.011 7f5685ae6700 1 -- 172.21.15.32:0/2081520262 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f568000b750 con 0x7f56880d2460 2021-04-09T18:15:53.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.011 7f568cced700 1 -- 172.21.15.32:0/2081520262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56880d2460 msgr2=0x7f568810cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.011 7f568cced700 1 --2- 172.21.15.32:0/2081520262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56880d2460 0x7f568810cff0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f5680006e30 tx=0x7f5680009e60).stop 2021-04-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.011 7f568cced700 1 -- 172.21.15.32:0/2081520262 shutdown_connections 2021-04-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.011 7f568cced700 1 --2- 172.21.15.32:0/2081520262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56880d2460 0x7f568810cff0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.011 7f568cced700 1 --2- 172.21.15.32:0/2081520262 >> v2:172.21.15.86:3300/0 conn(0x7f56880d6ca0 0x7f56880d8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.012 7f568cced700 1 -- 172.21.15.32:0/2081520262 >> 172.21.15.32:0/2081520262 conn(0x7f56880ce5e0 msgr2=0x7f56880cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:53.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.012 7f568cced700 1 -- 172.21.15.32:0/2081520262 shutdown_connections 2021-04-09T18:15:53.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.012 7f568cced700 1 -- 172.21.15.32:0/2081520262 wait complete. 2021-04-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f568cced700 1 Processor -- start 2021-04-09T18:15:53.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f568cced700 1 -- start start 2021-04-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f568cced700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5688190280 0x7f5688187b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f568cced700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5688190680 0x7f5688188130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f568cced700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5688099800 0x7f5688099c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f568cced700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5688187590 con 0x7f5688190280 2021-04-09T18:15:53.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f568cced700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568809a220 con 0x7f5688099800 2021-04-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f568cced700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5688098680 con 0x7f5688190680 2021-04-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f5686ae8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5688190280 0x7f5688187b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f5686ae8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5688190280 0x7f5688187b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44464/0 (socket says 172.21.15.32:44464) 2021-04-09T18:15:53.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.032 7f5686ae8700 1 -- 172.21.15.32:0/4106721961 learned_addr learned my addr 172.21.15.32:0/4106721961 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f56872e9700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5688099800 0x7f5688099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f5686ae8700 1 -- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5688190680 msgr2=0x7f5688188130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f56862e7700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5688190680 0x7f5688188130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f5686ae8700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5688190680 0x7f5688188130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f5686ae8700 1 -- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5688099800 msgr2=0x7f5688099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f5686ae8700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5688099800 0x7f5688099c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f5686ae8700 1 -- 172.21.15.32:0/4106721961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56880988a0 con 0x7f5688190280 2021-04-09T18:15:53.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f5686ae8700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5688190280 0x7f5688187b10 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f567800b530 tx=0x7f567800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:53.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f56777fe700 1 -- 172.21.15.32:0/4106721961 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5678006020 con 0x7f5688190280 2021-04-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f56777fe700 1 -- 172.21.15.32:0/4106721961 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56780063a0 con 0x7f5688190280 2021-04-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f568cced700 1 -- 172.21.15.32:0/4106721961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56880988a0 con 0x7f5688190280 2021-04-09T18:15:53.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f56777fe700 1 -- 172.21.15.32:0/4106721961 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5678004590 con 0x7f5688190280 2021-04-09T18:15:53.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.033 7f568cced700 1 -- 172.21.15.32:0/4106721961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56880988a0 con 0x7f5688190280 2021-04-09T18:15:53.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.034 7f56777fe700 1 -- 172.21.15.32:0/4106721961 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f567800a000 con 0x7f5688190280 2021-04-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.034 7f56777fe700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f566c0188c0 0x7f566c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.034 7f56777fe700 1 -- 172.21.15.32:0/4106721961 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f56780329d0 con 0x7f5688190280 2021-04-09T18:15:53.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.034 7f56862e7700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f566c0188c0 0x7f566c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.034 7f56862e7700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f566c0188c0 0x7f566c01adc0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.054 7f568cced700 1 -- 172.21.15.32:0/4106721961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56880d72a0 con 0x7f5688190280 2021-04-09T18:15:53.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.057 7f56777fe700 1 -- 172.21.15.32:0/4106721961 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5678006b90 con 0x7f5688190280 2021-04-09T18:15:53.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.237 7f568cced700 1 -- 172.21.15.32:0/4106721961 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f56880d72a0 con 0x7f5688190280 2021-04-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.237 7f56777fe700 1 -- 172.21.15.32:0/4106721961 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v54) v1 ==== 100+0+10 (secure 0 0 0) 0x7f567800a000 con 0x7f5688190280 2021-04-09T18:15:53.269 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:15:53.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 -- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f566c0188c0 msgr2=0x7f566c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f566c0188c0 0x7f566c01adc0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 -- 172.21.15.32:0/4106721961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5688190280 msgr2=0x7f5688187b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5688190280 0x7f5688187b10 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f567800b530 tx=0x7f567800b8d0).stop 2021-04-09T18:15:53.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 -- 172.21.15.32:0/4106721961 shutdown_connections 2021-04-09T18:15:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f566c0188c0 0x7f566c01adc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5688099800 0x7f5688099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5688190680 0x7f5688188130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 --2- 172.21.15.32:0/4106721961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5688190280 0x7f5688187b10 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 -- 172.21.15.32:0/4106721961 >> 172.21.15.32:0/4106721961 conn(0x7f568809bbc0 msgr2=0x7f56880ad9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 -- 172.21.15.32:0/4106721961 shutdown_connections 2021-04-09T18:15:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.240 7f568cced700 1 -- 172.21.15.32:0/4106721961 wait complete. 2021-04-09T18:15:53.408 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:15:53.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.586 7fd55fd4f700 1 Processor -- start 2021-04-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.586 7fd55fd4f700 1 -- start start 2021-04-09T18:15:53.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55fd4f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580d7ca0 0x7fd5580d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55fd4f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd55810da00 0x7fd55810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55fd4f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55810d480 con 0x7fd55810d010 2021-04-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55fd4f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5580d9220 con 0x7fd55810da00 2021-04-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55fd4f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5580d86c0 con 0x7fd5580d7ca0 2021-04-09T18:15:53.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55e36c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd55810da00 0x7fd55810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55db6b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd55810d010 0x7fd5580d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54062/0 (socket says 172.21.15.32:54062) 2021-04-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55db6b700 1 -- 172.21.15.32:0/2610215166 learned_addr learned my addr 172.21.15.32:0/2610215166 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55d36a700 1 --2- 172.21.15.32:0/2610215166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580d7ca0 0x7fd5580d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55cb69700 1 -- 172.21.15.32:0/2610215166 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339186316 0 0) 0x7fd554000ce0 con 0x7fd55810d010 2021-04-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55cb69700 1 -- 172.21.15.32:0/2610215166 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd544002fe0 con 0x7fd55810d010 2021-04-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55e36c700 1 -- 172.21.15.32:0/2610215166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580d7ca0 msgr2=0x7fd5580d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55e36c700 1 --2- 172.21.15.32:0/2610215166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580d7ca0 0x7fd5580d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55e36c700 1 -- 172.21.15.32:0/2610215166 >> v1:172.21.15.32:6789/0 conn(0x7fd55810d010 legacy=0x7fd5580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55e36c700 1 -- 172.21.15.32:0/2610215166 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5580d1f70 con 0x7fd55810da00 2021-04-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55d36a700 1 --2- 172.21.15.32:0/2610215166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580d7ca0 0x7fd5580d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55e36c700 1 --2- 172.21.15.32:0/2610215166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd55810da00 0x7fd55810de00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fd550006e30 tx=0x7fd550009e60).ready entity=mon.0 client_cookie=3f01a6538e97d436 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:53.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55cb69700 1 -- 172.21.15.32:0/2610215166 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd550006a50 con 0x7fd55810da00 2021-04-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55cb69700 1 -- 172.21.15.32:0/2610215166 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55000b300 con 0x7fd55810da00 2021-04-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55cb69700 1 -- 172.21.15.32:0/2610215166 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd55000b750 con 0x7fd55810da00 2021-04-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55fd4f700 1 -- 172.21.15.32:0/2610215166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd55810da00 msgr2=0x7fd55810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.587 7fd55fd4f700 1 --2- 172.21.15.32:0/2610215166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd55810da00 0x7fd55810de00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fd550006e30 tx=0x7fd550009e60).stop 2021-04-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.588 7fd55fd4f700 1 -- 172.21.15.32:0/2610215166 shutdown_connections 2021-04-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.588 7fd55fd4f700 1 --2- 172.21.15.32:0/2610215166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd55810da00 0x7fd55810de00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.588 7fd55fd4f700 1 --2- 172.21.15.32:0/2610215166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580d7ca0 0x7fd5580d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.588 7fd55fd4f700 1 -- 172.21.15.32:0/2610215166 >> 172.21.15.32:0/2610215166 conn(0x7fd5580ce5f0 msgr2=0x7fd5580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:53.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.588 7fd55fd4f700 1 -- 172.21.15.32:0/2610215166 shutdown_connections 2021-04-09T18:15:53.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.588 7fd55fd4f700 1 -- 172.21.15.32:0/2610215166 wait complete. 2021-04-09T18:15:53.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.608 7fd55fd4f700 1 Processor -- start 2021-04-09T18:15:53.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.608 7fd55fd4f700 1 -- start start 2021-04-09T18:15:53.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55fd4f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd55809a670 0x7fd558093610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55fd4f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5581a9fb0 0x7fd5581aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55fd4f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580a3ab0 0x7fd5580a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55fd4f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558094060 con 0x7fd5581a9fb0 2021-04-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55fd4f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558094710 con 0x7fd55809a670 2021-04-09T18:15:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55fd4f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd558093c30 con 0x7fd5580a3ab0 2021-04-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55d36a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5581a9fb0 0x7fd5581aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55d36a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5581a9fb0 0x7fd5581aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44480/0 (socket says 172.21.15.32:44480) 2021-04-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55d36a700 1 -- 172.21.15.32:0/508902462 learned_addr learned my addr 172.21.15.32:0/508902462 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55db6b700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd55809a670 0x7fd558093610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55e36c700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580a3ab0 0x7fd5580a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55d36a700 1 -- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580a3ab0 msgr2=0x7fd5580a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55d36a700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580a3ab0 0x7fd5580a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55d36a700 1 -- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd55809a670 msgr2=0x7fd558093610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55d36a700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd55809a670 0x7fd558093610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.609 7fd55d36a700 1 -- 172.21.15.32:0/508902462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5581aa9d0 con 0x7fd5581a9fb0 2021-04-09T18:15:53.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.610 7fd55d36a700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5581a9fb0 0x7fd5581aa3b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7fd54800b580 tx=0x7fd54800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.610 7fd54e7fc700 1 -- 172.21.15.32:0/508902462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd54800bd10 con 0x7fd5581a9fb0 2021-04-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.610 7fd54e7fc700 1 -- 172.21.15.32:0/508902462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548000c90 con 0x7fd5581a9fb0 2021-04-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.610 7fd54e7fc700 1 -- 172.21.15.32:0/508902462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd548002220 con 0x7fd5581a9fb0 2021-04-09T18:15:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.610 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5581aa9d0 con 0x7fd5581a9fb0 2021-04-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.610 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5581aa9d0 con 0x7fd5581a9fb0 2021-04-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.610 7fd54e7fc700 1 -- 172.21.15.32:0/508902462 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd54800a000 con 0x7fd5581a9fb0 2021-04-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.610 7fd54e7fc700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5400188c0 0x7fd54001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.610 7fd54e7fc700 1 -- 172.21.15.32:0/508902462 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd5480327a0 con 0x7fd5581a9fb0 2021-04-09T18:15:53.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.611 7fd55db6b700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5400188c0 0x7fd54001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:53.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.611 7fd55db6b700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5400188c0 0x7fd54001adc0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:53.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.631 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55810d480 con 0x7fd5581a9fb0 2021-04-09T18:15:53.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.634 7fd54e7fc700 1 -- 172.21.15.32:0/508902462 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd548006bb0 con 0x7fd5581a9fb0 2021-04-09T18:15:53.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.824 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd55810d480 con 0x7fd5581a9fb0 2021-04-09T18:15:53.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.824 7fd54e7fc700 1 -- 172.21.15.32:0/508902462 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2776 (secure 0 0 0) 0x7fd54800a000 con 0x7fd5581a9fb0 2021-04-09T18:15:53.859 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 45 2021-04-09T18:15:53.860 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:53.861 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":45,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3388525530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3388525530},{"type":"v1","addr":"172.21.15.32:6833","nonce":3388525530}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2165727103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2165727103},{"type":"v1","addr":"172.21.15.32:6835","nonce":2165727103}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":41},{"gid":55067,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3036306475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3036306475},{"type":"v1","addr":"172.21.15.86:6835","nonce":3036306475}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:15:46.073623","modified":"2021-04-09 18:15:48.082998","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5500":{"gid":5500,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6837/4251594905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":4251594905},{"type":"v1","addr":"172.21.15.86:6837","nonce":4251594905}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-04-09T18:15:53.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5400188c0 msgr2=0x7fd54001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5400188c0 0x7fd54001adc0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5581a9fb0 msgr2=0x7fd5581aa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:53.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5581a9fb0 0x7fd5581aa3b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7fd54800b580 tx=0x7fd54800b920).stop 2021-04-09T18:15:53.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 shutdown_connections 2021-04-09T18:15:53.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5400188c0 0x7fd54001adc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd5581a9fb0 0x7fd5581aa3b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5580a3ab0 0x7fd5580a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 --2- 172.21.15.32:0/508902462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd55809a670 0x7fd558093610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:53.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 >> 172.21.15.32:0/508902462 conn(0x7fd5581a0db0 msgr2=0x7fd5581a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:53.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 shutdown_connections 2021-04-09T18:15:53.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:53.827 7fd55fd4f700 1 -- 172.21.15.32:0/508902462 wait complete. 2021-04-09T18:15:54.299 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:15:46.073623', 'modified': '2021-04-09 18:15:48.082998', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5500': {'gid': 5500, 'name': 'd', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.86:6837/4251594905', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 4251594905}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 4251594905}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:15:54.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:15:54.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:15:54.440 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:15:54.440 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:15:54.441 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:15:54.441 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 5500, 2021-04-09T18:15:54.441 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:15:54.442 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:15:54.442 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 8.3380280839999994, 2021-04-09T18:15:54.443 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 45, 2021-04-09T18:15:54.443 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 54, 2021-04-09T18:15:54.444 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 54, 2021-04-09T18:15:54.444 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 18.690109134 2021-04-09T18:15:54.444 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:15:54.448 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5500, "want_state": "up:active", "state": "up:active", "rank_uptime": 8.3380280839999994, "mdsmap_epoch": 45, "osdmap_epoch": 54, "osdmap_epoch_barrier": 54, "uptime": 18.690109134 } 2021-04-09T18:15:54.449 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:15:54.450 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:15:54.450 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:15:54.451 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:54.452 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:54.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:15:54.606 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:15:54.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:15:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:15:54.673 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:15:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:15:54.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:54.676 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:15:54.732 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:15:54.733 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:54.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:15:54.835 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:15:54.836 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:15:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:15:54.897 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:15:54.898 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:15:54.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:54.898 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:15:54.924 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:15:54.896 7fd0b5183f80 -1 init, newargv = 0x55de2c20d380 newargc=9 2021-04-09T18:15:54.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[39495]: starting ceph client 2021-04-09T18:15:54.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[39495]: starting fuse 2021-04-09T18:15:54.955 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:15:54.956 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:15:54.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:54.957 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:15:54.958 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:15:54.959 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:15:54.959 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:15:54.960 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:15:54.961 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:15:54.962 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:15:54.963 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:15:54.967 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:15:54.968 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:15:55.018 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:15:55.019 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:15:55.020 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:15:55.022 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:15:55.028 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:15:55.029 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:15:55.030 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:15:55.044 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:15:55.044 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:15:55.045 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:15:55.046 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:15:55.046 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:15:55.047 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:15:55.244 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.39495.asok 2021-04-09T18:15:55.247 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.39495.asok 2021-04-09T18:15:55.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:55.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.39495.asok status 2021-04-09T18:15:55.379 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:15:55.379 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:15:55.380 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:15:55.380 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:15:55.380 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:15:55.381 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:15:55.381 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:15:55.381 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "39495", 2021-04-09T18:15:55.382 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:15:55.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:55.383 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:15:55.383 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:15:55.383 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5712, 2021-04-09T18:15:55.384 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:15:55.384 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:15:55.386 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:15:55.387 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 5712 2021-04-09T18:15:55.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:55.387 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:15:55.388 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:55.388 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:15:55.389 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3375803547 2021-04-09T18:15:55.389 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:15:55.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:55.390 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:15:55.390 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:15:55.391 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:15:55.391 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3375803547 2021-04-09T18:15:55.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:15:55.392 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.5712 172.21.15.32:0/3375803547", 2021-04-09T18:15:55.392 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/3375803547", 2021-04-09T18:15:55.392 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:15:55.393 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 45, 2021-04-09T18:15:55.393 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 54, 2021-04-09T18:15:55.393 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:15:55.394 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:15:55.394 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:15:55.394 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:15:55.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:55.413 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:15:55.414 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:55.478 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:15:55.478 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:15:55.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:55.480 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:55.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:55.547 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:15:55.548 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:55.630 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:15:55.631 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:15:55.632 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:15:55.632 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:15:55.633 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:55.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[39495]: fuse finished with error 0 and tester_r 0 2021-04-09T18:15:55.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:15:55.746 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:15:55.747 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:55.792 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:15:55.792 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:15:55.793 INFO:teuthology.orchestra.run:waiting for 900 2021-04-09T18:15:55.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:15:55.878 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:15:55.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2021-04-09T18:15:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.088 7f52fe457700 1 Processor -- start 2021-04-09T18:15:56.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.088 7f52fe457700 1 -- start start 2021-04-09T18:15:56.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52fe457700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f52f810d010 0x7f52f80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:56.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52fe457700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52f80d1da0 0x7f52f80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:56.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52fe457700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f810d480 con 0x7f52f80d82c0 2021-04-09T18:15:56.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52fe457700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f80d8d30 con 0x7f52f80d1da0 2021-04-09T18:15:56.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52fe457700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f80d9220 con 0x7f52f810d010 2021-04-09T18:15:56.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52effff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52f80d1da0 0x7f52f80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:56.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52effff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52f80d1da0 0x7f52f80d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44506/0 (socket says 172.21.15.32:44506) 2021-04-09T18:15:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52effff700 1 -- 172.21.15.32:0/2014336226 learned_addr learned my addr 172.21.15.32:0/2014336226 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52f7fff700 1 --2- 172.21.15.32:0/2014336226 >> v2:172.21.15.86:3300/0 conn(0x7f52f810d010 0x7f52f80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:56.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52effff700 1 -- 172.21.15.32:0/2014336226 >> v2:172.21.15.86:3300/0 conn(0x7f52f810d010 msgr2=0x7f52f80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:56.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52effff700 1 --2- 172.21.15.32:0/2014336226 >> v2:172.21.15.86:3300/0 conn(0x7f52f810d010 0x7f52f80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:56.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52effff700 1 -- 172.21.15.32:0/2014336226 >> v1:172.21.15.86:6789/0 conn(0x7f52f80d82c0 legacy=0x7f52f810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:56.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52effff700 1 -- 172.21.15.32:0/2014336226 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f80d86f0 con 0x7f52f80d1da0 2021-04-09T18:15:56.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.089 7f52effff700 1 --2- 172.21.15.32:0/2014336226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f80d1da0 0x7f52f80d7ca0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f52e0006700 tx=0x7f52e0009b80).ready entity=mon.0 client_cookie=eebd2f5fd1f198c6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:56.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52f77fe700 1 -- 172.21.15.32:0/2014336226 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52e000c0c0 con 0x7f52f80d1da0 2021-04-09T18:15:56.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52f77fe700 1 -- 172.21.15.32:0/2014336226 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e000c440 con 0x7f52f80d1da0 2021-04-09T18:15:56.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52f77fe700 1 -- 172.21.15.32:0/2014336226 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52e000c920 con 0x7f52f80d1da0 2021-04-09T18:15:56.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52fe457700 1 -- 172.21.15.32:0/2014336226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f80d1da0 msgr2=0x7f52f80d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:56.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52fe457700 1 --2- 172.21.15.32:0/2014336226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f80d1da0 0x7f52f80d7ca0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f52e0006700 tx=0x7f52e0009b80).stop 2021-04-09T18:15:56.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52fe457700 1 -- 172.21.15.32:0/2014336226 shutdown_connections 2021-04-09T18:15:56.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52fe457700 1 --2- 172.21.15.32:0/2014336226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f80d1da0 0x7f52f80d7ca0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:56.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52fe457700 1 --2- 172.21.15.32:0/2014336226 >> v2:172.21.15.86:3300/0 conn(0x7f52f810d010 0x7f52f80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:56.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52fe457700 1 -- 172.21.15.32:0/2014336226 >> 172.21.15.32:0/2014336226 conn(0x7f52f80ce5f0 msgr2=0x7f52f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:56.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52fe457700 1 -- 172.21.15.32:0/2014336226 shutdown_connections 2021-04-09T18:15:56.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.090 7f52fe457700 1 -- 172.21.15.32:0/2014336226 wait complete. 2021-04-09T18:15:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.111 7f52fe457700 1 Processor -- start 2021-04-09T18:15:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.111 7f52fe457700 1 -- start start 2021-04-09T18:15:56.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52fe457700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52f8097710 0x7f52f8192a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:56.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52fe457700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f8193090 0x7f52f817cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:56.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52fe457700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52f817d1f0 0x7f52f817b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:56.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52fe457700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f817d5f0 con 0x7f52f8193090 2021-04-09T18:15:56.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52fe457700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f8097330 con 0x7f52f817d1f0 2021-04-09T18:15:56.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52fe457700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f8097b10 con 0x7f52f8097710 2021-04-09T18:15:56.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f8193090 0x7f52f817cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:56.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f8193090 0x7f52f817cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44514/0 (socket says 172.21.15.32:44514) 2021-04-09T18:15:56.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52effff700 1 -- 172.21.15.32:0/3697488890 learned_addr learned my addr 172.21.15.32:0/3697488890 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:56.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52f7fff700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52f8097710 0x7f52f8192a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:56.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52fca74700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52f817d1f0 0x7f52f817b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:56.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52effff700 1 -- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52f8097710 msgr2=0x7f52f8192a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:56.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52effff700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52f8097710 0x7f52f8192a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:56.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52effff700 1 -- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52f817d1f0 msgr2=0x7f52f817b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:56.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52effff700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52f817d1f0 0x7f52f817b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:56.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52effff700 1 -- 172.21.15.32:0/3697488890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f80a3330 con 0x7f52f8193090 2021-04-09T18:15:56.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52effff700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f8193090 0x7f52f817cbd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f52e000b5e0 tx=0x7f52e000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:56.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.112 7f52f57fa700 1 -- 172.21.15.32:0/3697488890 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52e000bd70 con 0x7f52f8193090 2021-04-09T18:15:56.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.113 7f52f57fa700 1 -- 172.21.15.32:0/3697488890 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e0002030 con 0x7f52f8193090 2021-04-09T18:15:56.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.113 7f52f57fa700 1 -- 172.21.15.32:0/3697488890 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52e0002550 con 0x7f52f8193090 2021-04-09T18:15:56.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.113 7f52fe457700 1 -- 172.21.15.32:0/3697488890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f80a3330 con 0x7f52f8193090 2021-04-09T18:15:56.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.113 7f52fe457700 1 -- 172.21.15.32:0/3697488890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f80a3330 con 0x7f52f8193090 2021-04-09T18:15:56.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.113 7f52f57fa700 1 -- 172.21.15.32:0/3697488890 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f52e0000c90 con 0x7f52f8193090 2021-04-09T18:15:56.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.113 7f52f57fa700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52dc018930 0x7f52dc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:56.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.113 7f52f57fa700 1 -- 172.21.15.32:0/3697488890 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f52e0032a60 con 0x7f52f8193090 2021-04-09T18:15:56.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.113 7f52f7fff700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52dc018930 0x7f52dc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:56.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.114 7f52f7fff700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52dc018930 0x7f52dc01ae10 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.133 7f52fe457700 1 -- 172.21.15.32:0/3697488890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f817c6a0 con 0x7f52f8193090 2021-04-09T18:15:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.136 7f52f57fa700 1 -- 172.21.15.32:0/3697488890 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f52e001f7a0 con 0x7f52f8193090 2021-04-09T18:15:56.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:56.329 7f52fe457700 1 -- 172.21.15.32:0/3697488890 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "true"} v 0) v1 -- 0x7f52f817c6a0 con 0x7f52f8193090 2021-04-09T18:15:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.106 7f52f57fa700 1 -- 172.21.15.32:0/3697488890 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "true"}]=0 cephfs marked down. v46) v1 ==== 125+0+0 (secure 0 0 0) 0x7f52e001f7a0 con 0x7f52f8193090 2021-04-09T18:15:57.172 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked down. 2021-04-09T18:15:57.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 -- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52dc018930 msgr2=0x7f52dc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:57.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52dc018930 0x7f52dc01ae10 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 -- 172.21.15.32:0/3697488890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f8193090 msgr2=0x7f52f817cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f8193090 0x7f52f817cbd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f52e000b5e0 tx=0x7f52e000b980).stop 2021-04-09T18:15:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 -- 172.21.15.32:0/3697488890 shutdown_connections 2021-04-09T18:15:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52dc018930 0x7f52dc01ae10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52f8193090 0x7f52f817cbd0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52f817d1f0 0x7f52f817b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 --2- 172.21.15.32:0/3697488890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52f8097710 0x7f52f8192a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 -- 172.21.15.32:0/3697488890 >> 172.21.15.32:0/3697488890 conn(0x7f52f8093a60 msgr2=0x7f52f80a21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:57.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 -- 172.21.15.32:0/3697488890 shutdown_connections 2021-04-09T18:15:57.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.147 7f52fe457700 1 -- 172.21.15.32:0/3697488890 wait complete. 2021-04-09T18:15:57.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:15:57.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.312 7f31903fb700 1 Processor -- start 2021-04-09T18:15:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.312 7f31903fb700 1 -- start start 2021-04-09T18:15:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f31903fb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31880cffa0 0x7f31880d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:57.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f31903fb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31880d6c80 0x7f318810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:57.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f31903fb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31880d7280 con 0x7f31880d7800 2021-04-09T18:15:57.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f31903fb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31880d7080 con 0x7f31880d6c80 2021-04-09T18:15:57.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f31903fb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31880d8580 con 0x7f31880cffa0 2021-04-09T18:15:57.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318e217700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31880cffa0 0x7f31880d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:57.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318ea18700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31880d6c80 0x7f318810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:57.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318da16700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f31880d7800 0x7f31880d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52038/0 (socket says 172.21.15.32:52038) 2021-04-09T18:15:57.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318da16700 1 -- 172.21.15.32:0/3964957627 learned_addr learned my addr 172.21.15.32:0/3964957627 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:15:57.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318ea18700 1 -- 172.21.15.32:0/3964957627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31880cffa0 msgr2=0x7f31880d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:15:57.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318ea18700 1 --2- 172.21.15.32:0/3964957627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31880cffa0 0x7f31880d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318ea18700 1 -- 172.21.15.32:0/3964957627 >> v1:172.21.15.86:6789/0 conn(0x7f31880d7800 legacy=0x7f31880d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:57.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318d215700 1 -- 172.21.15.32:0/3964957627 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606333460 0 0) 0x7f31780009b0 con 0x7f31880d7800 2021-04-09T18:15:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318ea18700 1 -- 172.21.15.32:0/3964957627 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880d91e0 con 0x7f31880d6c80 2021-04-09T18:15:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.313 7f318e217700 1 --2- 172.21.15.32:0/3964957627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31880cffa0 0x7f31880d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:15:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.314 7f318ea18700 1 --2- 172.21.15.32:0/3964957627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31880d6c80 0x7f318810cfd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f3180006e50 tx=0x7f3180009e70).ready entity=mon.1 client_cookie=55ca3b8c48ef7ab6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:57.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.314 7f318d215700 1 -- 172.21.15.32:0/3964957627 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3180006a20 con 0x7f31880d6c80 2021-04-09T18:15:57.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.314 7f318d215700 1 -- 172.21.15.32:0/3964957627 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3180006790 con 0x7f31880d6c80 2021-04-09T18:15:57.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.314 7f318d215700 1 -- 172.21.15.32:0/3964957627 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f318000b700 con 0x7f31880d6c80 2021-04-09T18:15:57.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.314 7f31903fb700 1 -- 172.21.15.32:0/3964957627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31880d6c80 msgr2=0x7f318810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:57.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.314 7f31903fb700 1 --2- 172.21.15.32:0/3964957627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31880d6c80 0x7f318810cfd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f3180006e50 tx=0x7f3180009e70).stop 2021-04-09T18:15:57.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.315 7f31903fb700 1 -- 172.21.15.32:0/3964957627 shutdown_connections 2021-04-09T18:15:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.315 7f31903fb700 1 --2- 172.21.15.32:0/3964957627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31880d6c80 0x7f318810cfd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.315 7f31903fb700 1 --2- 172.21.15.32:0/3964957627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31880cffa0 0x7f31880d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.315 7f31903fb700 1 -- 172.21.15.32:0/3964957627 >> 172.21.15.32:0/3964957627 conn(0x7f31880ce5c0 msgr2=0x7f31880cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.315 7f31903fb700 1 -- 172.21.15.32:0/3964957627 shutdown_connections 2021-04-09T18:15:57.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.315 7f31903fb700 1 -- 172.21.15.32:0/3964957627 wait complete. 2021-04-09T18:15:57.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f31903fb700 1 Processor -- start 2021-04-09T18:15:57.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f31903fb700 1 -- start start 2021-04-09T18:15:57.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f31903fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3188187f20 0x7f3188195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:57.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f31903fb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3188194f90 0x7f31880b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:57.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f31903fb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3188184280 0x7f3188184680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:57.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f31903fb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3188196450 con 0x7f3188187f20 2021-04-09T18:15:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f31903fb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31881966d0 con 0x7f3188194f90 2021-04-09T18:15:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f31903fb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31880b54e0 con 0x7f3188184280 2021-04-09T18:15:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f318e217700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3188187f20 0x7f3188195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f318e217700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3188187f20 0x7f3188195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44524/0 (socket says 172.21.15.32:44524) 2021-04-09T18:15:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f318e217700 1 -- 172.21.15.32:0/2468112026 learned_addr learned my addr 172.21.15.32:0/2468112026 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:15:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f318da16700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3188194f90 0x7f31880b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.335 7f318ea18700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3188184280 0x7f3188184680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:57.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f318e217700 1 -- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3188184280 msgr2=0x7f3188184680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:57.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f318e217700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3188184280 0x7f3188184680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f318e217700 1 -- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3188194f90 msgr2=0x7f31880b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:57.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f318e217700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3188194f90 0x7f31880b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f318e217700 1 -- 172.21.15.32:0/2468112026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880b56c0 con 0x7f3188187f20 2021-04-09T18:15:57.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f318e217700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3188187f20 0x7f3188195e30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f3184006f40 tx=0x7f318400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:57.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f317effd700 1 -- 172.21.15.32:0/2468112026 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f318400bd80 con 0x7f3188187f20 2021-04-09T18:15:57.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f317effd700 1 -- 172.21.15.32:0/2468112026 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3184000c90 con 0x7f3188187f20 2021-04-09T18:15:57.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f31903fb700 1 -- 172.21.15.32:0/2468112026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31880b56c0 con 0x7f3188187f20 2021-04-09T18:15:57.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f317effd700 1 -- 172.21.15.32:0/2468112026 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3184002220 con 0x7f3188187f20 2021-04-09T18:15:57.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.336 7f31903fb700 1 -- 172.21.15.32:0/2468112026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31880b56c0 con 0x7f3188187f20 2021-04-09T18:15:57.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.337 7f317effd700 1 -- 172.21.15.32:0/2468112026 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f318400a000 con 0x7f3188187f20 2021-04-09T18:15:57.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.337 7f317effd700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3170018930 0x7f317001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:15:57.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.337 7f317effd700 1 -- 172.21.15.32:0/2468112026 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f31840329e0 con 0x7f3188187f20 2021-04-09T18:15:57.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.337 7f318da16700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3170018930 0x7f317001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:15:57.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.337 7f318da16700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3170018930 0x7f317001adc0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:15:57.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.357 7f31903fb700 1 -- 172.21.15.32:0/2468112026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3188182d90 con 0x7f3188187f20 2021-04-09T18:15:57.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.360 7f317effd700 1 -- 172.21.15.32:0/2468112026 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3184000c90 con 0x7f3188187f20 2021-04-09T18:15:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.546 7f31903fb700 1 -- 172.21.15.32:0/2468112026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3188182d90 con 0x7f3188187f20 2021-04-09T18:15:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.547 7f317effd700 1 -- 172.21.15.32:0/2468112026 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7f3184006b00 con 0x7f3188187f20 2021-04-09T18:15:57.597 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:15:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 -- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3170018930 msgr2=0x7f317001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3170018930 0x7f317001adc0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 -- 172.21.15.32:0/2468112026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3188187f20 msgr2=0x7f3188195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:15:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3188187f20 0x7f3188195e30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f3184006f40 tx=0x7f318400b990).stop 2021-04-09T18:15:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 -- 172.21.15.32:0/2468112026 shutdown_connections 2021-04-09T18:15:57.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3170018930 0x7f317001adc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3188184280 0x7f3188184680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3188194f90 0x7f31880b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 --2- 172.21.15.32:0/2468112026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3188187f20 0x7f3188195e30 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:15:57.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 -- 172.21.15.32:0/2468112026 >> 172.21.15.32:0/2468112026 conn(0x7f31881a8740 msgr2=0x7f318809c770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:15:57.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 -- 172.21.15.32:0/2468112026 shutdown_connections 2021-04-09T18:15:57.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:15:57.549 7f31903fb700 1 -- 172.21.15.32:0/2468112026 wait complete. 2021-04-09T18:15:57.633 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:15:57.633 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:15:57.634 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:16:02.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:16:02.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.757 7f43dd1f5700 1 Processor -- start 2021-04-09T18:16:02.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.757 7f43dd1f5700 1 -- start start 2021-04-09T18:16:02.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43dd1f5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d80cffa0 0x7f43d80d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:02.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43dd1f5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d80d7280 con 0x7f43d80d7800 2021-04-09T18:16:02.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43dd1f5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d810e7d0 con 0x7f43d80d6c80 2021-04-09T18:16:02.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43dd1f5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80d7080 con 0x7f43d80cffa0 2021-04-09T18:16:02.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d659c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f43d80d7800 0x7f43d80d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54126/0 (socket says 172.21.15.32:54126) 2021-04-09T18:16:02.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d659c700 1 -- 172.21.15.32:0/4236967778 learned_addr learned my addr 172.21.15.32:0/4236967778 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:02.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d6d9d700 1 --2- 172.21.15.32:0/4236967778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d80cffa0 0x7f43d80d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:02.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d5d9b700 1 -- 172.21.15.32:0/4236967778 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945671392 0 0) 0x7f43cc0009b0 con 0x7f43d80d7800 2021-04-09T18:16:02.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d5d9b700 1 -- 172.21.15.32:0/4236967778 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c0002fe0 con 0x7f43d80d7800 2021-04-09T18:16:02.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d5d9b700 1 -- 172.21.15.32:0/4236967778 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872484551 0 0) 0x7f43d0000d30 con 0x7f43d80d6c80 2021-04-09T18:16:02.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d5d9b700 1 -- 172.21.15.32:0/4236967778 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c00042c0 con 0x7f43d80d6c80 2021-04-09T18:16:02.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d6d9d700 1 -- 172.21.15.32:0/4236967778 >> v1:172.21.15.86:6789/0 conn(0x7f43d80d6c80 legacy=0x7f43d810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:02.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d6d9d700 1 -- 172.21.15.32:0/4236967778 >> v1:172.21.15.32:6789/0 conn(0x7f43d80d7800 legacy=0x7f43d80d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:02.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d6d9d700 1 -- 172.21.15.32:0/4236967778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d80d8580 con 0x7f43d80cffa0 2021-04-09T18:16:02.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.758 7f43d6d9d700 1 --2- 172.21.15.32:0/4236967778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d80cffa0 0x7f43d80d2440 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f43c8006650 tx=0x7f43c8009a80).ready entity=mon.2 client_cookie=ea406eea2f65900f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:02.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.759 7f43d5d9b700 1 -- 172.21.15.32:0/4236967778 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43c80009b0 con 0x7f43d80cffa0 2021-04-09T18:16:02.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.759 7f43d5d9b700 1 -- 172.21.15.32:0/4236967778 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c800c0c0 con 0x7f43d80cffa0 2021-04-09T18:16:02.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.759 7f43d5d9b700 1 -- 172.21.15.32:0/4236967778 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43c800c510 con 0x7f43d80cffa0 2021-04-09T18:16:02.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.759 7f43dd1f5700 1 -- 172.21.15.32:0/4236967778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d80cffa0 msgr2=0x7f43d80d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:02.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.759 7f43dd1f5700 1 --2- 172.21.15.32:0/4236967778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d80cffa0 0x7f43d80d2440 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f43c8006650 tx=0x7f43c8009a80).stop 2021-04-09T18:16:02.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.759 7f43dd1f5700 1 -- 172.21.15.32:0/4236967778 shutdown_connections 2021-04-09T18:16:02.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.759 7f43dd1f5700 1 --2- 172.21.15.32:0/4236967778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d80cffa0 0x7f43d80d2440 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:02.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.759 7f43dd1f5700 1 -- 172.21.15.32:0/4236967778 >> 172.21.15.32:0/4236967778 conn(0x7f43d80ce5c0 msgr2=0x7f43d80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.760 7f43dd1f5700 1 -- 172.21.15.32:0/4236967778 shutdown_connections 2021-04-09T18:16:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.760 7f43dd1f5700 1 -- 172.21.15.32:0/4236967778 wait complete. 2021-04-09T18:16:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43dd1f5700 1 Processor -- start 2021-04-09T18:16:02.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43dd1f5700 1 -- start start 2021-04-09T18:16:02.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43dd1f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43d81b8580 0x7f43d817da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:02.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43dd1f5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d81b8b10 0x7f43d817e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:02.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43dd1f5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43d817ece0 0x7f43d817f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:02.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43dd1f5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d817f700 con 0x7f43d81b8580 2021-04-09T18:16:02.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43dd1f5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80aba80 con 0x7f43d817ece0 2021-04-09T18:16:02.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43dd1f5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80abca0 con 0x7f43d81b8b10 2021-04-09T18:16:02.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43d6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43d81b8580 0x7f43d817da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:02.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43d6d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43d81b8580 0x7f43d817da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44542/0 (socket says 172.21.15.32:44542) 2021-04-09T18:16:02.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43d6d9d700 1 -- 172.21.15.32:0/1485165736 learned_addr learned my addr 172.21.15.32:0/1485165736 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:02.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43d659c700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d81b8b10 0x7f43d817e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:02.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.780 7f43d759e700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43d817ece0 0x7f43d817f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:02.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43d6d9d700 1 -- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d81b8b10 msgr2=0x7f43d817e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:02.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43d6d9d700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d81b8b10 0x7f43d817e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:02.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43d6d9d700 1 -- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43d817ece0 msgr2=0x7f43d817f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:02.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43d6d9d700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43d817ece0 0x7f43d817f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43d6d9d700 1 -- 172.21.15.32:0/1485165736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d80abec0 con 0x7f43d81b8580 2021-04-09T18:16:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43d759e700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43d817ece0 0x7f43d817f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:02.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43d6d9d700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43d81b8580 0x7f43d817da90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f43c8009650 tx=0x7f43c800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:02.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43c77fe700 1 -- 172.21.15.32:0/1485165736 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43c80093a0 con 0x7f43d81b8580 2021-04-09T18:16:02.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43c77fe700 1 -- 172.21.15.32:0/1485165736 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c8006070 con 0x7f43d81b8580 2021-04-09T18:16:02.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43c77fe700 1 -- 172.21.15.32:0/1485165736 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43c80052e0 con 0x7f43d81b8580 2021-04-09T18:16:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d80abec0 con 0x7f43d81b8580 2021-04-09T18:16:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.781 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d80abec0 con 0x7f43d81b8580 2021-04-09T18:16:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.782 7f43c77fe700 1 -- 172.21.15.32:0/1485165736 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f43c8009300 con 0x7f43d81b8580 2021-04-09T18:16:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.782 7f43c77fe700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43bc018930 0x7f43bc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:02.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.782 7f43c77fe700 1 -- 172.21.15.32:0/1485165736 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f43c8032a00 con 0x7f43d81b8580 2021-04-09T18:16:02.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.782 7f43d659c700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43bc018930 0x7f43bc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:02.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.782 7f43d659c700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43bc018930 0x7f43bc01ae10 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:02.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.802 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d817f700 con 0x7f43d81b8580 2021-04-09T18:16:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.805 7f43c77fe700 1 -- 172.21.15.32:0/1485165736 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f43c8006030 con 0x7f43d81b8580 2021-04-09T18:16:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.991 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f43d817f700 con 0x7f43d81b8580 2021-04-09T18:16:03.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.992 7f43c77fe700 1 -- 172.21.15.32:0/1485165736 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7f43c8000940 con 0x7f43d81b8580 2021-04-09T18:16:03.040 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:03.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43bc018930 msgr2=0x7f43bc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:03.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43bc018930 0x7f43bc01ae10 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:03.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43d81b8580 msgr2=0x7f43d817da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:03.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43d81b8580 0x7f43d817da90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f43c8009650 tx=0x7f43c800b8e0).stop 2021-04-09T18:16:03.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 shutdown_connections 2021-04-09T18:16:03.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43bc018930 0x7f43bc01ae10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:03.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43d817ece0 0x7f43d817f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:03.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43d81b8b10 0x7f43d817e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:03.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 --2- 172.21.15.32:0/1485165736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43d81b8580 0x7f43d817da90 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:03.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.994 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 >> 172.21.15.32:0/1485165736 conn(0x7f43d808b020 msgr2=0x7f43d8191ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:03.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.995 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 shutdown_connections 2021-04-09T18:16:03.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:02.995 7f43dd1f5700 1 -- 172.21.15.32:0/1485165736 wait complete. 2021-04-09T18:16:03.105 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:16:03.105 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:16:03.105 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:16:04.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:16:04.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:16:08.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:16:08.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10eae7700 1 Processor -- start 2021-04-09T18:16:08.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10eae7700 1 -- start start 2021-04-09T18:16:08.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10eae7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff1080cffb0 0x7ff108087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:08.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10eae7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff1080d7c50 0x7ff1081b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10eae7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff108088460 con 0x7ff1080d8050 2021-04-09T18:16:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10eae7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1080d91f0 con 0x7ff1080cffb0 2021-04-09T18:16:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10eae7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1080d8620 con 0x7ff1080d7c50 2021-04-09T18:16:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10c903700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff1080cffb0 0x7ff108087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:08.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10d104700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff1080d8050 0x7ff10810d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54140/0 (socket says 172.21.15.32:54140) 2021-04-09T18:16:08.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.250 7ff10d104700 1 -- 172.21.15.32:0/1038629265 learned_addr learned my addr 172.21.15.32:0/1038629265 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:08.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff107fff700 1 --2- 172.21.15.32:0/1038629265 >> v2:172.21.15.86:3300/0 conn(0x7ff1080d7c50 0x7ff1081b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:08.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff10c903700 1 -- 172.21.15.32:0/1038629265 >> v2:172.21.15.86:3300/0 conn(0x7ff1080d7c50 msgr2=0x7ff1081b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:08.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff10c903700 1 --2- 172.21.15.32:0/1038629265 >> v2:172.21.15.86:3300/0 conn(0x7ff1080d7c50 0x7ff1081b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff10c903700 1 -- 172.21.15.32:0/1038629265 >> v1:172.21.15.32:6789/0 conn(0x7ff1080d8050 legacy=0x7ff10810d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:08.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff10c903700 1 -- 172.21.15.32:0/1038629265 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1080d2800 con 0x7ff1080cffb0 2021-04-09T18:16:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff1077fe700 1 -- 172.21.15.32:0/1038629265 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266792088 0 0) 0x7ff1000009b0 con 0x7ff1080d8050 2021-04-09T18:16:08.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff10c903700 1 --2- 172.21.15.32:0/1038629265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080cffb0 0x7ff108087e40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7ff0f80069d0 tx=0x7ff0f8009dc0).ready entity=mon.0 client_cookie=e28cb07efaffbc3b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff1077fe700 1 -- 172.21.15.32:0/1038629265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0f800b510 con 0x7ff1080cffb0 2021-04-09T18:16:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff1077fe700 1 -- 172.21.15.32:0/1038629265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f8006ac0 con 0x7ff1080cffb0 2021-04-09T18:16:08.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff1077fe700 1 -- 172.21.15.32:0/1038629265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0f800ba80 con 0x7ff1080cffb0 2021-04-09T18:16:08.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff10eae7700 1 -- 172.21.15.32:0/1038629265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080cffb0 msgr2=0x7ff108087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:08.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.251 7ff10eae7700 1 --2- 172.21.15.32:0/1038629265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080cffb0 0x7ff108087e40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7ff0f80069d0 tx=0x7ff0f8009dc0).stop 2021-04-09T18:16:08.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.252 7ff10eae7700 1 -- 172.21.15.32:0/1038629265 shutdown_connections 2021-04-09T18:16:08.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.252 7ff10eae7700 1 --2- 172.21.15.32:0/1038629265 >> v2:172.21.15.86:3300/0 conn(0x7ff1080d7c50 0x7ff1081b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.252 7ff10eae7700 1 --2- 172.21.15.32:0/1038629265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1080cffb0 0x7ff108087e40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.252 7ff10eae7700 1 -- 172.21.15.32:0/1038629265 >> 172.21.15.32:0/1038629265 conn(0x7ff1080ce5b0 msgr2=0x7ff1080cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:08.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.252 7ff10eae7700 1 -- 172.21.15.32:0/1038629265 shutdown_connections 2021-04-09T18:16:08.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.252 7ff10eae7700 1 -- 172.21.15.32:0/1038629265 wait complete. 2021-04-09T18:16:08.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.272 7ff10eae7700 1 Processor -- start 2021-04-09T18:16:08.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.272 7ff10eae7700 1 -- start start 2021-04-09T18:16:08.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff10eae7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10819dfe0 0x7ff10819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:08.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff10eae7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff108095530 0x7ff108095930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:08.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff10eae7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1080a8a70 0x7ff1080a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:08.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff10eae7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1080a9490 con 0x7ff108095530 2021-04-09T18:16:08.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff10eae7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff108098ba0 con 0x7ff10819dfe0 2021-04-09T18:16:08.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff10eae7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff108098dc0 con 0x7ff1080a8a70 2021-04-09T18:16:08.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff107fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff108095530 0x7ff108095930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff107fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff108095530 0x7ff108095930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44556/0 (socket says 172.21.15.32:44556) 2021-04-09T18:16:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff107fff700 1 -- 172.21.15.32:0/284065470 learned_addr learned my addr 172.21.15.32:0/284065470 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff10c903700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10819dfe0 0x7ff10819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:08.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff10d104700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1080a8a70 0x7ff1080a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:08.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff107fff700 1 -- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1080a8a70 msgr2=0x7ff1080a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:08.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff107fff700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1080a8a70 0x7ff1080a8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff107fff700 1 -- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10819dfe0 msgr2=0x7ff10819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff107fff700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10819dfe0 0x7ff10819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff107fff700 1 -- 172.21.15.32:0/284065470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff108098fe0 con 0x7ff108095530 2021-04-09T18:16:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff10d104700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1080a8a70 0x7ff1080a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:16:08.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.273 7ff107fff700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff108095530 0x7ff108095930 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7ff0f00067e0 tx=0x7ff0f000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.274 7ff1057fa700 1 -- 172.21.15.32:0/284065470 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0f000bd80 con 0x7ff108095530 2021-04-09T18:16:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.274 7ff1057fa700 1 -- 172.21.15.32:0/284065470 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f0006900 con 0x7ff108095530 2021-04-09T18:16:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.274 7ff10eae7700 1 -- 172.21.15.32:0/284065470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff108098fe0 con 0x7ff108095530 2021-04-09T18:16:08.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.274 7ff1057fa700 1 -- 172.21.15.32:0/284065470 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0f0002080 con 0x7ff108095530 2021-04-09T18:16:08.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.274 7ff10eae7700 1 -- 172.21.15.32:0/284065470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff108098fe0 con 0x7ff108095530 2021-04-09T18:16:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.274 7ff1057fa700 1 -- 172.21.15.32:0/284065470 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff0f0000c90 con 0x7ff108095530 2021-04-09T18:16:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.274 7ff1057fa700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0ec018930 0x7ff0ec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.274 7ff1057fa700 1 -- 172.21.15.32:0/284065470 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff0f0032a10 con 0x7ff108095530 2021-04-09T18:16:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.275 7ff10c903700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0ec018930 0x7ff0ec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.275 7ff10c903700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0ec018930 0x7ff0ec01ae10 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.294 7ff10eae7700 1 -- 172.21.15.32:0/284065470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1080a9490 con 0x7ff108095530 2021-04-09T18:16:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.298 7ff1057fa700 1 -- 172.21.15.32:0/284065470 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff0f0000940 con 0x7ff108095530 2021-04-09T18:16:08.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.485 7ff10eae7700 1 -- 172.21.15.32:0/284065470 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff1080a9490 con 0x7ff108095530 2021-04-09T18:16:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.485 7ff1057fa700 1 -- 172.21.15.32:0/284065470 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7ff0f00068b0 con 0x7ff108095530 2021-04-09T18:16:08.529 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:08.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.487 7ff10eae7700 1 -- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0ec018930 msgr2=0x7ff0ec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.487 7ff10eae7700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0ec018930 0x7ff0ec01ae10 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.487 7ff10eae7700 1 -- 172.21.15.32:0/284065470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff108095530 msgr2=0x7ff108095930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:08.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.487 7ff10eae7700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff108095530 0x7ff108095930 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7ff0f00067e0 tx=0x7ff0f000b990).stop 2021-04-09T18:16:08.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.488 7ff10eae7700 1 -- 172.21.15.32:0/284065470 shutdown_connections 2021-04-09T18:16:08.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.488 7ff10eae7700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0ec018930 0x7ff0ec01ae10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.488 7ff10eae7700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff108095530 0x7ff108095930 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.488 7ff10eae7700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1080a8a70 0x7ff1080a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.488 7ff10eae7700 1 --2- 172.21.15.32:0/284065470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10819dfe0 0x7ff10819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:08.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.488 7ff10eae7700 1 -- 172.21.15.32:0/284065470 >> 172.21.15.32:0/284065470 conn(0x7ff1080b9c80 msgr2=0x7ff10817ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:08.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.488 7ff10eae7700 1 -- 172.21.15.32:0/284065470 shutdown_connections 2021-04-09T18:16:08.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:08.488 7ff10eae7700 1 -- 172.21.15.32:0/284065470 wait complete. 2021-04-09T18:16:08.614 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:16:08.614 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:16:08.615 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:16:13.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:16:13.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.739 7f862f9cb700 1 Processor -- start 2021-04-09T18:16:13.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.739 7f862f9cb700 1 -- start start 2021-04-09T18:16:13.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862f9cb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86280d6c80 0x7f86280d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:13.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862f9cb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f86280d2440 0x7f862810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:13.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862f9cb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280d7280 con 0x7f86280cffa0 2021-04-09T18:16:13.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862f9cb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86280d7080 con 0x7f86280d2440 2021-04-09T18:16:13.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862f9cb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86280d91e0 con 0x7f86280d6c80 2021-04-09T18:16:13.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862dfe8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f86280d2440 0x7f862810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:13.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862d7e7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f86280cffa0 0x7f86280d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54150/0 (socket says 172.21.15.32:54150) 2021-04-09T18:16:13.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862d7e7700 1 -- 172.21.15.32:0/1812595367 learned_addr learned my addr 172.21.15.32:0/1812595367 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:13.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862cfe6700 1 --2- 172.21.15.32:0/1812595367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86280d6c80 0x7f86280d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:13.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862dfe8700 1 -- 172.21.15.32:0/1812595367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86280d6c80 msgr2=0x7f86280d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:13.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862dfe8700 1 --2- 172.21.15.32:0/1812595367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86280d6c80 0x7f86280d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:13.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862dfe8700 1 -- 172.21.15.32:0/1812595367 >> v1:172.21.15.32:6789/0 conn(0x7f86280cffa0 legacy=0x7f86280d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:13.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f861bfff700 1 -- 172.21.15.32:0/1812595367 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207878165 0 0) 0x7f8624000ce0 con 0x7f86280cffa0 2021-04-09T18:16:13.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862dfe8700 1 -- 172.21.15.32:0/1812595367 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86280d2840 con 0x7f86280d2440 2021-04-09T18:16:13.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.740 7f862dfe8700 1 --2- 172.21.15.32:0/1812595367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86280d2440 0x7f862810cfd0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f8620006e30 tx=0x7f8620009de0).ready entity=mon.0 client_cookie=57f1d6bfaccdb44f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:13.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.741 7f861bfff700 1 -- 172.21.15.32:0/1812595367 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f862000b300 con 0x7f86280d2440 2021-04-09T18:16:13.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.741 7f861bfff700 1 -- 172.21.15.32:0/1812595367 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862000b680 con 0x7f86280d2440 2021-04-09T18:16:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.741 7f861bfff700 1 -- 172.21.15.32:0/1812595367 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f862000bb60 con 0x7f86280d2440 2021-04-09T18:16:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.741 7f862f9cb700 1 -- 172.21.15.32:0/1812595367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86280d2440 msgr2=0x7f862810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:13.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.741 7f862f9cb700 1 --2- 172.21.15.32:0/1812595367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86280d2440 0x7f862810cfd0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f8620006e30 tx=0x7f8620009de0).stop 2021-04-09T18:16:13.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.741 7f862f9cb700 1 -- 172.21.15.32:0/1812595367 shutdown_connections 2021-04-09T18:16:13.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.741 7f862f9cb700 1 --2- 172.21.15.32:0/1812595367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86280d2440 0x7f862810cfd0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:13.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.741 7f862f9cb700 1 --2- 172.21.15.32:0/1812595367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86280d6c80 0x7f86280d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:13.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.741 7f862f9cb700 1 -- 172.21.15.32:0/1812595367 >> 172.21.15.32:0/1812595367 conn(0x7f86280ce5c0 msgr2=0x7f86280cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:13.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.742 7f862f9cb700 1 -- 172.21.15.32:0/1812595367 shutdown_connections 2021-04-09T18:16:13.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.742 7f862f9cb700 1 -- 172.21.15.32:0/1812595367 wait complete. 2021-04-09T18:16:13.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.762 7f862f9cb700 1 Processor -- start 2021-04-09T18:16:13.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.762 7f862f9cb700 1 -- start start 2021-04-09T18:16:13.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.762 7f862f9cb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86281aa070 0x7f862819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:13.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.762 7f862f9cb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f862819e480 0x7f862819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:13.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.762 7f862f9cb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f862819d100 0x7f8628192d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.762 7f862f9cb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862819d8e0 con 0x7f862819e480 2021-04-09T18:16:13.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.762 7f862f9cb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862819e880 con 0x7f86281aa070 2021-04-09T18:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.762 7f862f9cb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862819d500 con 0x7f862819d100 2021-04-09T18:16:13.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862cfe6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f862819e480 0x7f862819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:13.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862cfe6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f862819e480 0x7f862819cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44570/0 (socket says 172.21.15.32:44570) 2021-04-09T18:16:13.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862cfe6700 1 -- 172.21.15.32:0/3822625816 learned_addr learned my addr 172.21.15.32:0/3822625816 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:13.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862d7e7700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86281aa070 0x7f862819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:13.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862dfe8700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f862819d100 0x7f8628192d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862cfe6700 1 -- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f862819d100 msgr2=0x7f8628192d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862cfe6700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f862819d100 0x7f8628192d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862cfe6700 1 -- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86281aa070 msgr2=0x7f862819de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:13.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862cfe6700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86281aa070 0x7f862819de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:13.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862cfe6700 1 -- 172.21.15.32:0/3822625816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8628193340 con 0x7f862819e480 2021-04-09T18:16:13.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862dfe8700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f862819d100 0x7f8628192d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:13.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862d7e7700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86281aa070 0x7f862819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:13.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862cfe6700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f862819e480 0x7f862819cae0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f861c006f60 tx=0x7f861c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:13.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f8619ffb700 1 -- 172.21.15.32:0/3822625816 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f861c00bd60 con 0x7f862819e480 2021-04-09T18:16:14.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f8619ffb700 1 -- 172.21.15.32:0/3822625816 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c000c90 con 0x7f862819e480 2021-04-09T18:16:14.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8628193340 con 0x7f862819e480 2021-04-09T18:16:14.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f8619ffb700 1 -- 172.21.15.32:0/3822625816 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f861c002220 con 0x7f862819e480 2021-04-09T18:16:14.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.763 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8628193340 con 0x7f862819e480 2021-04-09T18:16:14.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.764 7f8619ffb700 1 -- 172.21.15.32:0/3822625816 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f861c0052e0 con 0x7f862819e480 2021-04-09T18:16:14.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.764 7f8619ffb700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8608018930 0x7f860801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:14.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.764 7f8619ffb700 1 -- 172.21.15.32:0/3822625816 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f861c0329f0 con 0x7f862819e480 2021-04-09T18:16:14.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.764 7f862d7e7700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8608018930 0x7f860801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:14.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.764 7f862d7e7700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8608018930 0x7f860801ae10 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:14.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.784 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86281b7060 con 0x7f862819e480 2021-04-09T18:16:14.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.787 7f8619ffb700 1 -- 172.21.15.32:0/3822625816 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f861c000c90 con 0x7f862819e480 2021-04-09T18:16:14.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.975 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f86280d7280 con 0x7f862819e480 2021-04-09T18:16:14.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.975 7f8619ffb700 1 -- 172.21.15.32:0/3822625816 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7f861c006b00 con 0x7f862819e480 2021-04-09T18:16:14.070 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:14.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8608018930 msgr2=0x7f860801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8608018930 0x7f860801ae10 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f862819e480 msgr2=0x7f862819cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f862819e480 0x7f862819cae0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f861c006f60 tx=0x7f861c00b970).stop 2021-04-09T18:16:14.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 shutdown_connections 2021-04-09T18:16:14.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8608018930 0x7f860801ae10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:14.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f862819e480 0x7f862819cae0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f862819d100 0x7f8628192d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:14.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 --2- 172.21.15.32:0/3822625816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86281aa070 0x7f862819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:14.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 >> 172.21.15.32:0/3822625816 conn(0x7f86281b7290 msgr2=0x7f86281b7750 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:14.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 shutdown_connections 2021-04-09T18:16:14.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:13.978 7f862f9cb700 1 -- 172.21.15.32:0/3822625816 wait complete. 2021-04-09T18:16:14.175 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:16:14.175 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:16:14.176 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:16:16.115 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:16:19.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:16:19.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.310 7f59fc0bf700 1 Processor -- start 2021-04-09T18:16:19.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.310 7f59fc0bf700 1 -- start start 2021-04-09T18:16:19.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.310 7f59fc0bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f59f40d7c50 0x7f59f41b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:19.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.310 7f59fc0bf700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f59f40d8050 0x7f59f410d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:19.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.310 7f59fc0bf700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40d8620 con 0x7f59f40cffb0 2021-04-09T18:16:19.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.310 7f59fc0bf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f40d91f0 con 0x7f59f40d7c50 2021-04-09T18:16:19.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59fc0bf700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f40d2800 con 0x7f59f40d8050 2021-04-09T18:16:19.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f96da700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f59f40d7c50 0x7f59f41b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:19.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f96da700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f59f40d7c50 0x7f59f41b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44578/0 (socket says 172.21.15.32:44578) 2021-04-09T18:16:19.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f96da700 1 -- 172.21.15.32:0/783556918 learned_addr learned my addr 172.21.15.32:0/783556918 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:19.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59fa6dc700 1 --2- 172.21.15.32:0/783556918 >> v2:172.21.15.86:3300/0 conn(0x7f59f40d8050 0x7f59f410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:19.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f96da700 1 -- 172.21.15.32:0/783556918 >> v2:172.21.15.86:3300/0 conn(0x7f59f40d8050 msgr2=0x7f59f410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f96da700 1 --2- 172.21.15.32:0/783556918 >> v2:172.21.15.86:3300/0 conn(0x7f59f40d8050 0x7f59f410d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f96da700 1 -- 172.21.15.32:0/783556918 >> v1:172.21.15.86:6789/0 conn(0x7f59f40cffb0 legacy=0x7f59f4087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:19.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f96da700 1 -- 172.21.15.32:0/783556918 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f41b0ab0 con 0x7f59f40d7c50 2021-04-09T18:16:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59fa6dc700 1 --2- 172.21.15.32:0/783556918 >> v2:172.21.15.86:3300/0 conn(0x7f59f40d8050 0x7f59f410d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:19.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f96da700 1 --2- 172.21.15.32:0/783556918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f40d7c50 0x7f59f41b0490 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f59e4006700 tx=0x7f59e4009b80).ready entity=mon.0 client_cookie=523b066a7959b53b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:19.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f8ed9700 1 -- 172.21.15.32:0/783556918 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59e400c0c0 con 0x7f59f40d7c50 2021-04-09T18:16:19.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f8ed9700 1 -- 172.21.15.32:0/783556918 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e400c440 con 0x7f59f40d7c50 2021-04-09T18:16:19.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59f8ed9700 1 -- 172.21.15.32:0/783556918 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59e400c920 con 0x7f59f40d7c50 2021-04-09T18:16:19.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59fc0bf700 1 -- 172.21.15.32:0/783556918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f40d7c50 msgr2=0x7f59f41b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:19.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.311 7f59fc0bf700 1 --2- 172.21.15.32:0/783556918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f40d7c50 0x7f59f41b0490 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f59e4006700 tx=0x7f59e4009b80).stop 2021-04-09T18:16:19.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.312 7f59fc0bf700 1 -- 172.21.15.32:0/783556918 shutdown_connections 2021-04-09T18:16:19.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.312 7f59fc0bf700 1 --2- 172.21.15.32:0/783556918 >> v2:172.21.15.86:3300/0 conn(0x7f59f40d8050 0x7f59f410d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.312 7f59fc0bf700 1 --2- 172.21.15.32:0/783556918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f40d7c50 0x7f59f41b0490 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.312 7f59fc0bf700 1 -- 172.21.15.32:0/783556918 >> 172.21.15.32:0/783556918 conn(0x7f59f40ce5b0 msgr2=0x7f59f40cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:19.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.312 7f59fc0bf700 1 -- 172.21.15.32:0/783556918 shutdown_connections 2021-04-09T18:16:19.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.312 7f59fc0bf700 1 -- 172.21.15.32:0/783556918 wait complete. 2021-04-09T18:16:19.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.332 7f59fc0bf700 1 Processor -- start 2021-04-09T18:16:19.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.332 7f59fc0bf700 1 -- start start 2021-04-09T18:16:19.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59fc0bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59f419dfe0 0x7f59f419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:19.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59fc0bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f4095530 0x7f59f4095930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:19.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59fc0bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59f40a8a70 0x7f59f40a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:19.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59fc0bf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f40a9490 con 0x7f59f4095530 2021-04-09T18:16:19.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59fc0bf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4098ba0 con 0x7f59f419dfe0 2021-04-09T18:16:19.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59fc0bf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4098dc0 con 0x7f59f40a8a70 2021-04-09T18:16:19.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59f96da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f4095530 0x7f59f4095930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:19.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59f96da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f4095530 0x7f59f4095930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44582/0 (socket says 172.21.15.32:44582) 2021-04-09T18:16:19.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59f96da700 1 -- 172.21.15.32:0/369106678 learned_addr learned my addr 172.21.15.32:0/369106678 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59fa6dc700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59f40a8a70 0x7f59f40a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:19.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59f9edb700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59f419dfe0 0x7f59f419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:19.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59f96da700 1 -- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59f40a8a70 msgr2=0x7f59f40a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:19.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59f96da700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59f40a8a70 0x7f59f40a8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59f96da700 1 -- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59f419dfe0 msgr2=0x7f59f419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:19.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59f96da700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59f419dfe0 0x7f59f419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.333 7f59f96da700 1 -- 172.21.15.32:0/369106678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4098fe0 con 0x7f59f4095530 2021-04-09T18:16:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.334 7f59f96da700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f4095530 0x7f59f4095930 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f59e400b580 tx=0x7f59e400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.334 7f59ea7fc700 1 -- 172.21.15.32:0/369106678 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59e400bd10 con 0x7f59f4095530 2021-04-09T18:16:19.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.334 7f59ea7fc700 1 -- 172.21.15.32:0/369106678 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e4000c90 con 0x7f59f4095530 2021-04-09T18:16:19.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.334 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f4098fe0 con 0x7f59f4095530 2021-04-09T18:16:19.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.334 7f59ea7fc700 1 -- 172.21.15.32:0/369106678 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59e4002220 con 0x7f59f4095530 2021-04-09T18:16:19.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.334 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f4098fe0 con 0x7f59f4095530 2021-04-09T18:16:19.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.334 7f59ea7fc700 1 -- 172.21.15.32:0/369106678 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f59e4009f90 con 0x7f59f4095530 2021-04-09T18:16:19.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.335 7f59ea7fc700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59dc018930 0x7f59dc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:19.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.335 7f59ea7fc700 1 -- 172.21.15.32:0/369106678 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f59e40329f0 con 0x7f59f4095530 2021-04-09T18:16:19.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.335 7f59f9edb700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59dc018930 0x7f59dc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:19.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.335 7f59f9edb700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59dc018930 0x7f59dc01ae10 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:19.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.355 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f40d8620 con 0x7f59f4095530 2021-04-09T18:16:19.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.358 7f59ea7fc700 1 -- 172.21.15.32:0/369106678 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f59e40091e0 con 0x7f59f4095530 2021-04-09T18:16:19.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.545 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f59f40d8620 con 0x7f59f4095530 2021-04-09T18:16:19.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.546 7f59ea7fc700 1 -- 172.21.15.32:0/369106678 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7f59e4000c90 con 0x7f59f4095530 2021-04-09T18:16:19.618 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:19.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59dc018930 msgr2=0x7f59dc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:19.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59dc018930 0x7f59dc01ae10 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f4095530 msgr2=0x7f59f4095930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:19.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f4095530 0x7f59f4095930 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f59e400b580 tx=0x7f59e400b920).stop 2021-04-09T18:16:19.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 shutdown_connections 2021-04-09T18:16:19.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59dc018930 0x7f59dc01ae10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59f4095530 0x7f59f4095930 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59f40a8a70 0x7f59f40a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 --2- 172.21.15.32:0/369106678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59f419dfe0 0x7f59f419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:19.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 >> 172.21.15.32:0/369106678 conn(0x7f59f40b9c80 msgr2=0x7f59f417ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:19.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 shutdown_connections 2021-04-09T18:16:19.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:19.548 7f59fc0bf700 1 -- 172.21.15.32:0/369106678 wait complete. 2021-04-09T18:16:19.649 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:16:19.665 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:16:19.665 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:16:24.666 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:16:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.786 7f38b1161700 1 Processor -- start 2021-04-09T18:16:24.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.786 7f38b1161700 1 -- start start 2021-04-09T18:16:24.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.787 7f38b1161700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f38ac0cffa0 0x7f38ac0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:24.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.787 7f38b1161700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ac0d7280 con 0x7f38ac0d6c80 2021-04-09T18:16:24.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.787 7f38b1161700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ac0d83f0 con 0x7f38ac0d7800 2021-04-09T18:16:24.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.787 7f38b1161700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ac0d7080 con 0x7f38ac0cffa0 2021-04-09T18:16:24.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.787 7f38ab59e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f38ac0d6c80 0x7f38ac10cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54180/0 (socket says 172.21.15.32:54180) 2021-04-09T18:16:24.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.787 7f38ab59e700 1 -- 172.21.15.32:0/1152840078 learned_addr learned my addr 172.21.15.32:0/1152840078 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.787 7f38aad9d700 1 --2- 172.21.15.32:0/1152840078 >> v2:172.21.15.86:3300/0 conn(0x7f38ac0cffa0 0x7f38ac0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.787 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761003288 0 0) 0x7f38a4000d30 con 0x7f38ac0d6c80 2021-04-09T18:16:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3894002fe0 con 0x7f38ac0d6c80 2021-04-09T18:16:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436014730 0 0) 0x7f38a0000c60 con 0x7f38ac0d7800 2021-04-09T18:16:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3894003210 con 0x7f38ac0d7800 2021-04-09T18:16:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3833419159 0 0) 0x7f38a4001460 con 0x7f38ac0d6c80 2021-04-09T18:16:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 >> v2:172.21.15.86:3300/0 conn(0x7f38ac0cffa0 msgr2=0x7f38ac0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 --2- 172.21.15.32:0/1152840078 >> v2:172.21.15.86:3300/0 conn(0x7f38ac0cffa0 0x7f38ac0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 >> v1:172.21.15.86:6789/0 conn(0x7f38ac0d7800 legacy=0x7f38ac0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac0d91e0 con 0x7f38ac0d6c80 2021-04-09T18:16:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f38a4001980 con 0x7f38ac0d6c80 2021-04-09T18:16:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 >> v1:172.21.15.32:6789/0 conn(0x7f38ac0d6c80 legacy=0x7f38ac10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3894003b10 0x7f3894005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38940067a0 0x7f3894008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38940095a0 0x7f389400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3894050320 con 0x7f38940095a0 2021-04-09T18:16:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3894050500 con 0x7f3894003b10 2021-04-09T18:16:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3894050720 con 0x7f38940067a0 2021-04-09T18:16:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38ab59e700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38940095a0 0x7f389400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:24.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38aad9d700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3894003b10 0x7f3894005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:24.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38aa59c700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38940067a0 0x7f3894008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:24.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38ab59e700 1 -- 172.21.15.32:0/1152840078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38940067a0 msgr2=0x7f3894008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38ab59e700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38940067a0 0x7f3894008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:24.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38ab59e700 1 -- 172.21.15.32:0/1152840078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3894003b10 msgr2=0x7f3894005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:24.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38ab59e700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3894003b10 0x7f3894005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:24.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38ab59e700 1 -- 172.21.15.32:0/1152840078 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38940509c0 con 0x7f38940095a0 2021-04-09T18:16:24.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38aad9d700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3894003b10 0x7f3894005f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:24.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38ab59e700 1 -- 172.21.15.32:0/1152840078 reap_dead start 2021-04-09T18:16:24.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.788 7f38ab59e700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38940095a0 0x7f389400bac0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f38a4007c70 tx=0x7f38a400a310).ready entity=mon.0 client_cookie=6fee65fb591ae842 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:24.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.789 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a4007340 con 0x7f38940095a0 2021-04-09T18:16:24.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.789 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a4004720 con 0x7f38940095a0 2021-04-09T18:16:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.789 7f38a9d9b700 1 -- 172.21.15.32:0/1152840078 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a4004b70 con 0x7f38940095a0 2021-04-09T18:16:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.789 7f38b1161700 1 -- 172.21.15.32:0/1152840078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38940095a0 msgr2=0x7f389400bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:24.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.789 7f38b1161700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38940095a0 0x7f389400bac0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f38a4007c70 tx=0x7f38a400a310).stop 2021-04-09T18:16:24.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.790 7f38b1161700 1 -- 172.21.15.32:0/1152840078 shutdown_connections 2021-04-09T18:16:25.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.790 7f38b1161700 1 --2- 172.21.15.32:0/1152840078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38940095a0 0x7f389400bac0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:25.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.790 7f38b1161700 1 -- 172.21.15.32:0/1152840078 >> 172.21.15.32:0/1152840078 conn(0x7f38ac0ce5c0 msgr2=0x7f38ac0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:25.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.790 7f38b1161700 1 -- 172.21.15.32:0/1152840078 shutdown_connections 2021-04-09T18:16:25.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.790 7f38b1161700 1 -- 172.21.15.32:0/1152840078 wait complete. 2021-04-09T18:16:25.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38b1161700 1 Processor -- start 2021-04-09T18:16:25.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38b1161700 1 -- start start 2021-04-09T18:16:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38b1161700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38ac1a9090 0x7f38ac0a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38b1161700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ac1a9c00 0x7f38ac1aa000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:25.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38b1161700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38ac183c70 0x7f38ac184070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:25.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38b1161700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac1a9680 con 0x7f38ac183c70 2021-04-09T18:16:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38b1161700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac1aa620 con 0x7f38ac1a9c00 2021-04-09T18:16:25.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38b1161700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac1a5970 con 0x7f38ac1a9090 2021-04-09T18:16:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38ab59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38ac183c70 0x7f38ac184070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:25.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38ab59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38ac183c70 0x7f38ac184070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44604/0 (socket says 172.21.15.32:44604) 2021-04-09T18:16:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.810 7f38ab59e700 1 -- 172.21.15.32:0/2191472748 learned_addr learned my addr 172.21.15.32:0/2191472748 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38aad9d700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38ac1a9090 0x7f38ac0a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38aa59c700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ac1a9c00 0x7f38ac1aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:25.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38ab59e700 1 -- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38ac1a9090 msgr2=0x7f38ac0a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:25.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38ab59e700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38ac1a9090 0x7f38ac0a27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:25.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38ab59e700 1 -- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ac1a9c00 msgr2=0x7f38ac1aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:25.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38ab59e700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ac1a9c00 0x7f38ac1aa000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:25.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38ab59e700 1 -- 172.21.15.32:0/2191472748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac184690 con 0x7f38ac183c70 2021-04-09T18:16:25.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38aa59c700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ac1a9c00 0x7f38ac1aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:16:25.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38ab59e700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38ac183c70 0x7f38ac184070 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f38a4007a00 tx=0x7f38a400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:25.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f389b7fe700 1 -- 172.21.15.32:0/2191472748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a40016d0 con 0x7f38ac183c70 2021-04-09T18:16:25.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f389b7fe700 1 -- 172.21.15.32:0/2191472748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a4007020 con 0x7f38ac183c70 2021-04-09T18:16:25.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f389b7fe700 1 -- 172.21.15.32:0/2191472748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a40074b0 con 0x7f38ac183c70 2021-04-09T18:16:25.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38b1161700 1 -- 172.21.15.32:0/2191472748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ac184690 con 0x7f38ac183c70 2021-04-09T18:16:25.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.811 7f38b1161700 1 -- 172.21.15.32:0/2191472748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ac184690 con 0x7f38ac183c70 2021-04-09T18:16:25.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.812 7f389b7fe700 1 -- 172.21.15.32:0/2191472748 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f38a4007470 con 0x7f38ac183c70 2021-04-09T18:16:25.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.812 7f389b7fe700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38900188d0 0x7f389001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:25.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.812 7f389b7fe700 1 -- 172.21.15.32:0/2191472748 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f38a4032d80 con 0x7f38ac183c70 2021-04-09T18:16:25.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.812 7f38aad9d700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38900188d0 0x7f389001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:25.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.813 7f38aad9d700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38900188d0 0x7f389001add0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:25.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.831 7f38b1161700 1 -- 172.21.15.32:0/2191472748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ac182d60 con 0x7f38ac183c70 2021-04-09T18:16:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:24.835 7f389b7fe700 1 -- 172.21.15.32:0/2191472748 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f38a4006f10 con 0x7f38ac183c70 2021-04-09T18:16:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.021 7f38b1161700 1 -- 172.21.15.32:0/2191472748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f38ac0d7280 con 0x7f38ac183c70 2021-04-09T18:16:25.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.021 7f389b7fe700 1 -- 172.21.15.32:0/2191472748 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7f38a4007120 con 0x7f38ac183c70 2021-04-09T18:16:25.173 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:25.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 -- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38900188d0 msgr2=0x7f389001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:25.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38900188d0 0x7f389001add0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:25.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 -- 172.21.15.32:0/2191472748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38ac183c70 msgr2=0x7f38ac184070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:25.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38ac183c70 0x7f38ac184070 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f38a4007a00 tx=0x7f38a400ac50).stop 2021-04-09T18:16:25.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 -- 172.21.15.32:0/2191472748 shutdown_connections 2021-04-09T18:16:25.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38900188d0 0x7f389001add0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:25.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38ac183c70 0x7f38ac184070 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:25.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38ac1a9c00 0x7f38ac1aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:25.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 --2- 172.21.15.32:0/2191472748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38ac1a9090 0x7f38ac0a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:25.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 -- 172.21.15.32:0/2191472748 >> 172.21.15.32:0/2191472748 conn(0x7f38ac192920 msgr2=0x7f38ac187f60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:25.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 -- 172.21.15.32:0/2191472748 shutdown_connections 2021-04-09T18:16:25.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:25.024 7f38b1161700 1 -- 172.21.15.32:0/2191472748 wait complete. 2021-04-09T18:16:25.238 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:16:25.239 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:16:25.239 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:16:30.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:16:30.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.452 7f5bf1d84700 1 Processor -- start 2021-04-09T18:16:30.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.452 7f5bf1d84700 1 -- start start 2021-04-09T18:16:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5bf1d84700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5bec0cffa0 0x7f5bec0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5bf1d84700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5bec0d6c80 0x7f5bec10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5bf1d84700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bec0d7280 con 0x7f5bec0d7800 2021-04-09T18:16:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5bf1d84700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bec0d7080 con 0x7f5bec0cffa0 2021-04-09T18:16:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5bf1d84700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bec0d91e0 con 0x7f5bec0d6c80 2021-04-09T18:16:30.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5beb7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5bec0cffa0 0x7f5bec0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:30.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5beb7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5bec0cffa0 0x7f5bec0d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44608/0 (socket says 172.21.15.32:44608) 2021-04-09T18:16:30.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5beb7fe700 1 -- 172.21.15.32:0/1915817179 learned_addr learned my addr 172.21.15.32:0/1915817179 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:30.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5bebfff700 1 --2- 172.21.15.32:0/1915817179 >> v2:172.21.15.86:3300/0 conn(0x7f5bec0d6c80 0x7f5bec10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5beb7fe700 1 -- 172.21.15.32:0/1915817179 >> v2:172.21.15.86:3300/0 conn(0x7f5bec0d6c80 msgr2=0x7f5bec10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5beb7fe700 1 --2- 172.21.15.32:0/1915817179 >> v2:172.21.15.86:3300/0 conn(0x7f5bec0d6c80 0x7f5bec10cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5beb7fe700 1 -- 172.21.15.32:0/1915817179 >> v1:172.21.15.86:6789/0 conn(0x7f5bec0d7800 legacy=0x7f5bec0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5beb7fe700 1 -- 172.21.15.32:0/1915817179 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec0d83f0 con 0x7f5bec0cffa0 2021-04-09T18:16:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5beb7fe700 1 --2- 172.21.15.32:0/1915817179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec0cffa0 0x7f5bec0d2440 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f5bdc006700 tx=0x7f5bdc009b00).ready entity=mon.0 client_cookie=29b3a3cafabee7c1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:30.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5bea7fc700 1 -- 172.21.15.32:0/1915817179 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bdc00c0c0 con 0x7f5bec0cffa0 2021-04-09T18:16:30.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5bea7fc700 1 -- 172.21.15.32:0/1915817179 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc00c400 con 0x7f5bec0cffa0 2021-04-09T18:16:30.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.453 7f5bea7fc700 1 -- 172.21.15.32:0/1915817179 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bdc00c8e0 con 0x7f5bec0cffa0 2021-04-09T18:16:30.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.454 7f5bf1d84700 1 -- 172.21.15.32:0/1915817179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec0cffa0 msgr2=0x7f5bec0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:30.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.454 7f5bf1d84700 1 --2- 172.21.15.32:0/1915817179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec0cffa0 0x7f5bec0d2440 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f5bdc006700 tx=0x7f5bdc009b00).stop 2021-04-09T18:16:30.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.454 7f5bf1d84700 1 -- 172.21.15.32:0/1915817179 shutdown_connections 2021-04-09T18:16:30.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.454 7f5bf1d84700 1 --2- 172.21.15.32:0/1915817179 >> v2:172.21.15.86:3300/0 conn(0x7f5bec0d6c80 0x7f5bec10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.454 7f5bf1d84700 1 --2- 172.21.15.32:0/1915817179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec0cffa0 0x7f5bec0d2440 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.454 7f5bf1d84700 1 -- 172.21.15.32:0/1915817179 >> 172.21.15.32:0/1915817179 conn(0x7f5bec0ce5c0 msgr2=0x7f5bec0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:30.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.454 7f5bf1d84700 1 -- 172.21.15.32:0/1915817179 shutdown_connections 2021-04-09T18:16:30.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.454 7f5bf1d84700 1 -- 172.21.15.32:0/1915817179 wait complete. 2021-04-09T18:16:30.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.474 7f5bf1d84700 1 Processor -- start 2021-04-09T18:16:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5bf1d84700 1 -- start start 2021-04-09T18:16:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5bf1d84700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bec0b4250 0x7f5bec09a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5bf1d84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec18e280 0x7f5bec192370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:30.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5bf1d84700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bec184d80 0x7f5bec191250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:30.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5bf1d84700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec192a10 con 0x7f5bec18e280 2021-04-09T18:16:30.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5bf1d84700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec191df0 con 0x7f5bec184d80 2021-04-09T18:16:30.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5bf1d84700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec191870 con 0x7f5bec0b4250 2021-04-09T18:16:30.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5beaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec18e280 0x7f5bec192370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:30.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5beaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec18e280 0x7f5bec192370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44614/0 (socket says 172.21.15.32:44614) 2021-04-09T18:16:30.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5beaffd700 1 -- 172.21.15.32:0/2390653184 learned_addr learned my addr 172.21.15.32:0/2390653184 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:30.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5bebfff700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bec184d80 0x7f5bec191250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:30.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.475 7f5beb7fe700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bec0b4250 0x7f5bec09a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:30.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5beaffd700 1 -- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bec0b4250 msgr2=0x7f5bec09a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:30.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5beaffd700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bec0b4250 0x7f5bec09a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5beaffd700 1 -- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bec184d80 msgr2=0x7f5bec191250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:30.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5beaffd700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bec184d80 0x7f5bec191250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5beaffd700 1 -- 172.21.15.32:0/2390653184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec191a90 con 0x7f5bec18e280 2021-04-09T18:16:30.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5beaffd700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec18e280 0x7f5bec192370 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f5be0006ed0 tx=0x7f5be000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:30.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5bd3fff700 1 -- 172.21.15.32:0/2390653184 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5be0006020 con 0x7f5bec18e280 2021-04-09T18:16:30.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5bd3fff700 1 -- 172.21.15.32:0/2390653184 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be00062a0 con 0x7f5bec18e280 2021-04-09T18:16:30.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5bd3fff700 1 -- 172.21.15.32:0/2390653184 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5be0005730 con 0x7f5bec18e280 2021-04-09T18:16:30.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bec191a90 con 0x7f5bec18e280 2021-04-09T18:16:30.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.476 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bec191a90 con 0x7f5bec18e280 2021-04-09T18:16:30.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.477 7f5bd3fff700 1 -- 172.21.15.32:0/2390653184 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5be00054b0 con 0x7f5bec18e280 2021-04-09T18:16:30.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.477 7f5bd3fff700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5bcc0188c0 0x7f5bcc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:30.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.477 7f5bd3fff700 1 -- 172.21.15.32:0/2390653184 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5be0032790 con 0x7f5bec18e280 2021-04-09T18:16:30.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.477 7f5beb7fe700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5bcc0188c0 0x7f5bcc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:30.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.477 7f5beb7fe700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5bcc0188c0 0x7f5bcc01adc0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:30.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.497 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bec182d60 con 0x7f5bec18e280 2021-04-09T18:16:30.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.500 7f5bd3fff700 1 -- 172.21.15.32:0/2390653184 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5be0006380 con 0x7f5bec18e280 2021-04-09T18:16:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.685 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5bec182d60 con 0x7f5bec18e280 2021-04-09T18:16:30.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.686 7f5bd3fff700 1 -- 172.21.15.32:0/2390653184 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7f5be0006380 con 0x7f5bec18e280 2021-04-09T18:16:30.763 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:30.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5bcc0188c0 msgr2=0x7f5bcc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:30.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5bcc0188c0 0x7f5bcc01adc0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec18e280 msgr2=0x7f5bec192370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:30.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec18e280 0x7f5bec192370 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f5be0006ed0 tx=0x7f5be000b960).stop 2021-04-09T18:16:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 shutdown_connections 2021-04-09T18:16:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5bcc0188c0 0x7f5bcc01adc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bec18e280 0x7f5bec192370 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bec184d80 0x7f5bec191250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 --2- 172.21.15.32:0/2390653184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bec0b4250 0x7f5bec09a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 >> 172.21.15.32:0/2390653184 conn(0x7f5bec090a90 msgr2=0x7f5bec1832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 shutdown_connections 2021-04-09T18:16:30.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:30.688 7f5bf1d84700 1 -- 172.21.15.32:0/2390653184 wait complete. 2021-04-09T18:16:30.862 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:16:30.863 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:16:30.864 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:16:30.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:16:30.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:16:31.016 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:16:31.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:31.017 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:16:31.018 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:16:31.066 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:16:31.067 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:16:31.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:16:31.068 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:31.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:31.152 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:16:31.153 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:16:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:16:31.197 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:16:31.197 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:16:31.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:16:31.198 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:16:31.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:16:31.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.394 7f9634f0e700 1 Processor -- start 2021-04-09T18:16:31.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.394 7f9634f0e700 1 -- start start 2021-04-09T18:16:31.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.395 7f9634f0e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f963010da00 0x7f963010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.395 7f9634f0e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963010d480 con 0x7f96300d7ca0 2021-04-09T18:16:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.395 7f9634f0e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96300d1f70 con 0x7f963010d010 2021-04-09T18:16:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.395 7f9634f0e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96300d8d30 con 0x7f963010da00 2021-04-09T18:16:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.395 7f962f314700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f963010da00 0x7f963010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.395 7f962e312700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f96300d7ca0 0x7f96300d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54214/0 (socket says 172.21.15.32:54214) 2021-04-09T18:16:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.395 7f962e312700 1 -- 172.21.15.32:0/3734344702 learned_addr learned my addr 172.21.15.32:0/3734344702 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.395 7f962eb13700 1 --1- 172.21.15.32:0/3734344702 >> v1:172.21.15.86:6789/0 conn(0x7f963010d010 0x7f96300d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52138/0 (socket says 172.21.15.32:52138) 2021-04-09T18:16:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962db11700 1 -- 172.21.15.32:0/3734344702 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316442111 0 0) 0x7f9624000ce0 con 0x7f96300d7ca0 2021-04-09T18:16:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962db11700 1 -- 172.21.15.32:0/3734344702 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9618002fe0 con 0x7f96300d7ca0 2021-04-09T18:16:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962db11700 1 -- 172.21.15.32:0/3734344702 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154548620 0 0) 0x7f9620000c60 con 0x7f963010d010 2021-04-09T18:16:31.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962db11700 1 -- 172.21.15.32:0/3734344702 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96180042c0 con 0x7f963010d010 2021-04-09T18:16:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962f314700 1 -- 172.21.15.32:0/3734344702 >> v1:172.21.15.86:6789/0 conn(0x7f963010d010 legacy=0x7f96300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:31.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962f314700 1 -- 172.21.15.32:0/3734344702 >> v1:172.21.15.32:6789/0 conn(0x7f96300d7ca0 legacy=0x7f96300d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:31.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962f314700 1 -- 172.21.15.32:0/3734344702 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300d9220 con 0x7f963010da00 2021-04-09T18:16:31.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962f314700 1 --2- 172.21.15.32:0/3734344702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f963010da00 0x7f963010de00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f9628006e30 tx=0x7f9628009e60).ready entity=mon.0 client_cookie=b353cc1fb7b233a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:31.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962db11700 1 -- 172.21.15.32:0/3734344702 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9628006a50 con 0x7f963010da00 2021-04-09T18:16:31.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962db11700 1 -- 172.21.15.32:0/3734344702 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962800b300 con 0x7f963010da00 2021-04-09T18:16:31.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f962db11700 1 -- 172.21.15.32:0/3734344702 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f962800b750 con 0x7f963010da00 2021-04-09T18:16:31.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f9634f0e700 1 -- 172.21.15.32:0/3734344702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f963010da00 msgr2=0x7f963010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:31.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.396 7f9634f0e700 1 --2- 172.21.15.32:0/3734344702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f963010da00 0x7f963010de00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f9628006e30 tx=0x7f9628009e60).stop 2021-04-09T18:16:31.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.397 7f9634f0e700 1 -- 172.21.15.32:0/3734344702 shutdown_connections 2021-04-09T18:16:31.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.397 7f9634f0e700 1 --2- 172.21.15.32:0/3734344702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f963010da00 0x7f963010de00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:31.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.397 7f9634f0e700 1 -- 172.21.15.32:0/3734344702 >> 172.21.15.32:0/3734344702 conn(0x7f96300ce5f0 msgr2=0x7f96300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:31.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.397 7f9634f0e700 1 -- 172.21.15.32:0/3734344702 shutdown_connections 2021-04-09T18:16:31.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.397 7f9634f0e700 1 -- 172.21.15.32:0/3734344702 wait complete. 2021-04-09T18:16:31.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.417 7f9634f0e700 1 Processor -- start 2021-04-09T18:16:31.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.417 7f9634f0e700 1 -- start start 2021-04-09T18:16:31.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f9634f0e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300b42d0 0x7f9630183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:31.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f9634f0e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96301ac090 0x7f96301ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:31.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f9634f0e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9630184d30 0x7f9630185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:31.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f9634f0e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300a1a90 con 0x7f96300b42d0 2021-04-09T18:16:31.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f9634f0e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96301acab0 con 0x7f9630184d30 2021-04-09T18:16:31.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f9634f0e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9630185a00 con 0x7f96301ac090 2021-04-09T18:16:31.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962eb13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300b42d0 0x7f9630183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:31.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962eb13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300b42d0 0x7f9630183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44628/0 (socket says 172.21.15.32:44628) 2021-04-09T18:16:31.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962eb13700 1 -- 172.21.15.32:0/3692821600 learned_addr learned my addr 172.21.15.32:0/3692821600 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:31.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962f314700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9630184d30 0x7f9630185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:31.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962e312700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96301ac090 0x7f96301ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:31.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962eb13700 1 -- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96301ac090 msgr2=0x7f96301ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:31.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962eb13700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96301ac090 0x7f96301ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:31.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962eb13700 1 -- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9630184d30 msgr2=0x7f9630185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:31.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962eb13700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9630184d30 0x7f9630185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:31.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962eb13700 1 -- 172.21.15.32:0/3692821600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9630185750 con 0x7f96300b42d0 2021-04-09T18:16:31.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f962eb13700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300b42d0 0x7f9630183520 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f9620007e80 tx=0x7f962000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:31.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f961f7fe700 1 -- 172.21.15.32:0/3692821600 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f962000acf0 con 0x7f96300b42d0 2021-04-09T18:16:31.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f961f7fe700 1 -- 172.21.15.32:0/3692821600 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9620004320 con 0x7f96300b42d0 2021-04-09T18:16:31.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9630185750 con 0x7f96300b42d0 2021-04-09T18:16:31.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f961f7fe700 1 -- 172.21.15.32:0/3692821600 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9620004840 con 0x7f96300b42d0 2021-04-09T18:16:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.418 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9630185750 con 0x7f96300b42d0 2021-04-09T18:16:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.419 7f961f7fe700 1 -- 172.21.15.32:0/3692821600 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9620000c10 con 0x7f96300b42d0 2021-04-09T18:16:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.419 7f961f7fe700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9614018930 0x7f961401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.419 7f961f7fe700 1 -- 172.21.15.32:0/3692821600 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9620032a30 con 0x7f96300b42d0 2021-04-09T18:16:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.419 7f962e312700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9614018930 0x7f961401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:31.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.420 7f962e312700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9614018930 0x7f961401adc0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:31.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.439 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9630182d10 con 0x7f96300b42d0 2021-04-09T18:16:31.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.443 7f961f7fe700 1 -- 172.21.15.32:0/3692821600 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f962001f770 con 0x7f96300b42d0 2021-04-09T18:16:31.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.631 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f963010d480 con 0x7f96300b42d0 2021-04-09T18:16:31.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.632 7f961f7fe700 1 -- 172.21.15.32:0/3692821600 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v54) v1 ==== 81+0+30360 (secure 0 0 0) 0x7f9620000c10 con 0x7f96300b42d0 2021-04-09T18:16:31.669 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:31.669 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:16:31.669 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 54, 2021-04-09T18:16:31.670 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:16:31.671 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:16:31.671 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:15:47.075022", 2021-04-09T18:16:31.672 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:16:31.672 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:16:31.673 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:16:31.673 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:16:31.673 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:16:31.674 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:16:31.675 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:16:31.675 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:16:31.690 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:16:31.691 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:16:31.691 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:16:31.692 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:16:31.707 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:16:31.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:16:31.708 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:16:31.714 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 9, 2021-04-09T18:16:31.715 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:16:31.715 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:16:31.715 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:16:31.716 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:16:31.716 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:16:31.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:31.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:16:31.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:16:31.718 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:16:31.718 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:16:31.718 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:16:31.719 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:16:31.719 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:16:31.720 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:16:31.720 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:16:31.720 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:16:31.721 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:16:31.721 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:16:31.776 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:16:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:16:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:16:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:16:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:16:31.778 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:16:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:16:31.789 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:16:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:16:31.790 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:16:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:16:31.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:16:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:16:31.792 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:16:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:16:31.793 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:16:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:16:31.794 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:16:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:16:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:16:31.795 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:16:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:16:31.796 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:16:31.844 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:16:31.844 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:16:31.845 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:16:31.845 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:16:31.846 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:16:31.846 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:16:31.846 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:16:31.847 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:16:31.847 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:16:31.847 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:16:31.848 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:16:31.848 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:16:31.849 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:16:31.849 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:16:31.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:16:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:16:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:16:31.899 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:16:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:16:31.900 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:16:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:16:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:16:31.901 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:16:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:16:31.902 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:16:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:16:31.903 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:16:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:16:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:16:31.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:31.905 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 8, 2021-04-09T18:16:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:16:31.906 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:15:44.773626", 2021-04-09T18:16:31.950 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:16:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:16:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:16:31.951 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:16:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:16:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:16:31.952 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:16:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:16:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:16:31.953 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:16:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:16:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:16:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:16:31.954 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:16:31.975 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:16:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:16:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:16:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:16:31.976 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:16:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:16:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "53", 2021-04-09T18:16:31.977 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:16:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:16:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:16:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:16:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:16:31.978 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:16:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:16:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:16:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:16:31.979 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:16:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:16:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:16:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:16:31.980 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:16:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:16:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:16:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:16:31.981 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:16:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:16:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:16:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:16:31.982 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:16:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:16:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:16:31.983 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:16:32.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:16:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.008 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:16:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:16:32.010 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:16:32.015 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:16:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:16:32.016 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:16:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:16:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:16:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:16:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:16:32.018 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:16:32.019 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:16:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:16:32.025 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:16:32.026 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:16:32.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.084 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:16:32.087 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:16:32.087 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:16:32.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.110 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 9, 2021-04-09T18:16:32.111 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:16:32.111 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:15:45.435836", 2021-04-09T18:16:32.117 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:16:32.117 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:16:32.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:16:32.124 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:16:32.124 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:16:32.124 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:16:32.125 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:16:32.125 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:16:32.126 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:16:32.126 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:16:32.126 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:16:32.127 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:16:32.147 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:16:32.148 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:16:32.148 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:16:32.148 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:16:32.149 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:16:32.163 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:16:32.164 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:16:32.164 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:16:32.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.165 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "54", 2021-04-09T18:16:32.165 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:16:32.166 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:16:32.166 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:16:32.166 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:16:32.167 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:16:32.189 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:16:32.189 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:16:32.189 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:16:32.189 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:16:32.191 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:16:32.193 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:16:32.193 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:16:32.193 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:16:32.193 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:16:32.194 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:16:32.194 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:16:32.194 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:16:32.194 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:16:32.195 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:16:32.206 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:16:32.208 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:16:32.208 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:16:32.209 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:16:32.209 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:16:32.209 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:16:32.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:16:32.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.210 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:16:32.217 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:16:32.217 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:16:32.217 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:16:32.217 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:16:32.218 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:16:32.218 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:16:32.219 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:16:32.219 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:16:32.219 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:16:32.219 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:16:32.220 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:16:32.220 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:16:32.220 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:16:32.220 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:16:32.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.241 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:16:32.254 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:16:32.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.254 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:16:32.254 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:16:32.255 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:32.272 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:32.273 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:32.273 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:32.273 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:32.274 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:32.274 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:32.274 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:32.274 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:32.274 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:32.321 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:32.322 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.322 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.329 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.329 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:16:32.330 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:32.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.402 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:16:32.402 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:32.403 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.403 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.404 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:32.404 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.405 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.406 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:16:32.406 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:32.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.437 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:16:32.441 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:32.441 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.445 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.446 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:32.446 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.452 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:16:32.453 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:32.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.462 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:16:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.463 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.505 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:16:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:16:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.508 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.509 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:16:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:16:32.510 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:16:32.511 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:16:32.545 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:32.546 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:16:32.547 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:16:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:32.548 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:32.549 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:32.594 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:32.594 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:32.594 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:32.594 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:32.595 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:32.595 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:32.595 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.595 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.596 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:16:32.600 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:32.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.603 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:16:32.603 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:32.603 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.603 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.604 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:32.604 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.605 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:16:32.605 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:32.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.606 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:16:32.606 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:32.606 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.607 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.609 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:32.609 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.611 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.612 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:16:32.612 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:32.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.624 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:16:32.624 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:32.625 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.625 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.626 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:32.626 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.627 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.627 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:16:32.627 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:32.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.628 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:16:32.628 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:32.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.629 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.629 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:16:32.630 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:16:32.630 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:16:32.630 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:16:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:32.631 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:32.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.633 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:16:32.633 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:16:32.633 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:32.634 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:32.634 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:32.634 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:32.635 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:32.635 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:32.636 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:32.636 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:16:32.637 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:32.637 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:32.637 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:32.638 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.638 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:16:32.639 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:32.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:16:32.640 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:32.640 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.641 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.642 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:32.642 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.643 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:16:32.643 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:32.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.644 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:16:32.645 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:32.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.645 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.646 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:32.646 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:16:32.647 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:32.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.648 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:16:32.648 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:32.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.648 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.649 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:32.649 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.649 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.650 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:16:32.650 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:32.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.657 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:16:32.657 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:32.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.658 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:16:32.658 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:16:32.658 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:16:32.659 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:16:32.659 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:32.659 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:32.660 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:32.660 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.661 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:16:32.661 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:16:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:32.662 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:32.663 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:32.665 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:32.666 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:32.666 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:32.667 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:32.667 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:32.674 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:32.674 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.676 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:16:32.677 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:32.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.688 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:16:32.689 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:32.689 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.695 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:32.696 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.702 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.702 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:16:32.703 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:32.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:16:32.704 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:32.704 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.704 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.705 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.706 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:16:32.714 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:32.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.714 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.715 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:16:32.715 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:32.715 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.715 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.716 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:32.716 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.726 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:16:32.726 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:32.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:16:32.730 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:32.730 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.730 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.731 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:16:32.731 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:16:32.736 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:16:32.736 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:16:32.738 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:32.738 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:32.745 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:32.745 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:16:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:16:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:32.747 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:32.747 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:32.747 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:32.747 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:32.826 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:32.833 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:32.834 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 51, 2021-04-09T18:16:32.834 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:32.836 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:32.837 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:32.840 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:16:32.845 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:32.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:16:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.858 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:32.858 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.859 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.859 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:16:32.860 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:32.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.872 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:16:32.873 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:32.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.875 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:16:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:16:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:16:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:16:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:16:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:16:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:16:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:16:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:16:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:16:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:16:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:16:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:16:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:16:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:32.977 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:32.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:32.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:32.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.003 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:16:33.056 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:33.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.057 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.057 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:16:33.058 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:33.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.067 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.074 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:33.074 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:33.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.075 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.075 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:16:33.076 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:33.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.174 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:16:33.184 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:33.184 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.207 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.208 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:16:33.208 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:16:33.213 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:16:33.217 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:16:33.217 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:33.217 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:33.217 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:33.218 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.218 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:16:33.219 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:16:33.219 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:33.219 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:33.239 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:33.240 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:33.240 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:33.240 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:33.255 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:33.255 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 51, 2021-04-09T18:16:33.255 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:33.256 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:33.258 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:33.258 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:33.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.259 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.259 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:16:33.259 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:33.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.266 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.266 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:16:33.267 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:33.269 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.269 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.270 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:33.285 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:33.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.286 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:16:33.287 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:33.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.287 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.287 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:16:33.288 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:33.288 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.288 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.289 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:33.289 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:33.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.289 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.289 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:16:33.290 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:33.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.303 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:16:33.304 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:33.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.305 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.305 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:33.306 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:33.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.328 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.328 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:16:33.329 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:33.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.329 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.329 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:16:33.330 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:33.330 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.330 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.333 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:16:33.334 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:16:33.334 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:16:33.334 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:16:33.334 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:33.334 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:33.335 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:33.337 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.338 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:16:33.338 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:16:33.343 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:33.344 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:33.344 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:33.344 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:33.345 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:33.345 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:33.345 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:33.345 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:33.346 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.347 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.347 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:16:33.347 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:33.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.348 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:16:33.348 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:33.348 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.349 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.349 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:33.349 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:33.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.350 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.350 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:16:33.350 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:33.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.425 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:16:33.425 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:33.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.426 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.438 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:33.439 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:33.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.450 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:16:33.457 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:33.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.473 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.479 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:16:33.501 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:33.502 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.502 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.502 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:33.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:33.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:16:33.533 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:16:33.552 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.554 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:16:33.595 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:16:33.596 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:16:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:16:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:33.599 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:33.600 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:33.601 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.603 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:16:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:16:33.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:16:33.627 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:33.628 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:33.645 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:33.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.647 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:16:33.648 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:33.728 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:33.729 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:33.729 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:33.729 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:33.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.731 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:16:33.731 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:33.732 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:33.732 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:33.732 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:33.734 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:33.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.790 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:16:33.791 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:33.794 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:33.795 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:33.795 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:33.796 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:33.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.797 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:16:33.798 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:33.798 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:33.859 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:33.860 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:33.860 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:33.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.861 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:16:33.861 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:33.862 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:33.862 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:33.862 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.904 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:16:33.904 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:33.904 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:33.905 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:33.905 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:33.929 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:33.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:33.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:33.940 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:16:33.941 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:33.941 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:33.942 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:33.942 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:33.942 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:33.943 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.943 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:16:33.944 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:16:33.944 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:16:33.944 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:16:33.945 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:16:33.945 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:16:33.946 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:16:33.982 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:16:33.982 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:16:33.983 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:16:33.983 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:16:33.990 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:16:33.991 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:16:33.992 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:16:33.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:33.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:34.002 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:16:34.003 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:16:34.003 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:16:34.003 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:16:34.004 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:16:34.004 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:16:34.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9614018930 msgr2=0x7f961401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9614018930 0x7f961401adc0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300b42d0 msgr2=0x7f9630183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300b42d0 0x7f9630183520 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f9620007e80 tx=0x7f962000a900).stop 2021-04-09T18:16:34.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 shutdown_connections 2021-04-09T18:16:34.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9614018930 0x7f961401adc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9630184d30 0x7f9630185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96301ac090 0x7f96301ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 --2- 172.21.15.32:0/3692821600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96300b42d0 0x7f9630183520 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 >> 172.21.15.32:0/3692821600 conn(0x7f963019b890 msgr2=0x7f96301a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:34.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 shutdown_connections 2021-04-09T18:16:34.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:31.636 7f9634f0e700 1 -- 172.21.15.32:0/3692821600 wait complete. 2021-04-09T18:16:34.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:16:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.220 7f5a65ae3700 1 Processor -- start 2021-04-09T18:16:34.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.220 7f5a65ae3700 1 -- start start 2021-04-09T18:16:34.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a65ae3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6010d010 0x7f5a600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a65ae3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a600d1da0 0x7f5a600d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a65ae3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6010d480 con 0x7f5a600d82c0 2021-04-09T18:16:34.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a65ae3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6010e780 con 0x7f5a600d1da0 2021-04-09T18:16:34.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a65ae3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a600d8d30 con 0x7f5a6010d010 2021-04-09T18:16:34.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5effd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a600d1da0 0x7f5a600d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5effd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a600d1da0 0x7f5a600d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44636/0 (socket says 172.21.15.32:44636) 2021-04-09T18:16:34.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5effd700 1 -- 172.21.15.32:0/4182058767 learned_addr learned my addr 172.21.15.32:0/4182058767 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:34.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5f7fe700 1 --2- 172.21.15.32:0/4182058767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6010d010 0x7f5a600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5effd700 1 -- 172.21.15.32:0/4182058767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6010d010 msgr2=0x7f5a600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:34.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5effd700 1 --2- 172.21.15.32:0/4182058767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6010d010 0x7f5a600d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5effd700 1 -- 172.21.15.32:0/4182058767 >> v1:172.21.15.86:6789/0 conn(0x7f5a600d82c0 legacy=0x7f5a6010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:34.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5effd700 1 -- 172.21.15.32:0/4182058767 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a600d9220 con 0x7f5a600d1da0 2021-04-09T18:16:34.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5f7fe700 1 --2- 172.21.15.32:0/4182058767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6010d010 0x7f5a600d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:16:34.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5effd700 1 --2- 172.21.15.32:0/4182058767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a600d1da0 0x7f5a600d7ca0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f5a4c006700 tx=0x7f5a4c009b80).ready entity=mon.0 client_cookie=5681184310bcabb2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:34.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5e7fc700 1 -- 172.21.15.32:0/4182058767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a4c00c0c0 con 0x7f5a600d1da0 2021-04-09T18:16:34.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5e7fc700 1 -- 172.21.15.32:0/4182058767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4c00c440 con 0x7f5a600d1da0 2021-04-09T18:16:34.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.221 7f5a5e7fc700 1 -- 172.21.15.32:0/4182058767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a4c00c920 con 0x7f5a600d1da0 2021-04-09T18:16:34.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.222 7f5a65ae3700 1 -- 172.21.15.32:0/4182058767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a600d1da0 msgr2=0x7f5a600d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.222 7f5a65ae3700 1 --2- 172.21.15.32:0/4182058767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a600d1da0 0x7f5a600d7ca0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f5a4c006700 tx=0x7f5a4c009b80).stop 2021-04-09T18:16:34.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.222 7f5a65ae3700 1 -- 172.21.15.32:0/4182058767 shutdown_connections 2021-04-09T18:16:34.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.222 7f5a65ae3700 1 --2- 172.21.15.32:0/4182058767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a600d1da0 0x7f5a600d7ca0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.222 7f5a65ae3700 1 --2- 172.21.15.32:0/4182058767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6010d010 0x7f5a600d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.222 7f5a65ae3700 1 -- 172.21.15.32:0/4182058767 >> 172.21.15.32:0/4182058767 conn(0x7f5a600ce5f0 msgr2=0x7f5a600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:34.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.222 7f5a65ae3700 1 -- 172.21.15.32:0/4182058767 shutdown_connections 2021-04-09T18:16:34.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.222 7f5a65ae3700 1 -- 172.21.15.32:0/4182058767 wait complete. 2021-04-09T18:16:34.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.242 7f5a65ae3700 1 Processor -- start 2021-04-09T18:16:34.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a65ae3700 1 -- start start 2021-04-09T18:16:34.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a65ae3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a601a8d20 0x7f5a601a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a65ae3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a601a4180 0x7f5a601a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a65ae3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a601a4580 0x7f5a601a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a65ae3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601a35e0 con 0x7f5a601a8d20 2021-04-09T18:16:34.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a65ae3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601a9120 con 0x7f5a601a4580 2021-04-09T18:16:34.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a65ae3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6008f8f0 con 0x7f5a601a4180 2021-04-09T18:16:34.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a5f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a601a8d20 0x7f5a601a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a5f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a601a8d20 0x7f5a601a3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44640/0 (socket says 172.21.15.32:44640) 2021-04-09T18:16:34.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a5f7fe700 1 -- 172.21.15.32:0/118491928 learned_addr learned my addr 172.21.15.32:0/118491928 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:34.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a5ffff700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a601a4580 0x7f5a601a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.243 7f5a5effd700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a601a4180 0x7f5a601a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a5f7fe700 1 -- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a601a4180 msgr2=0x7f5a601a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a5f7fe700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a601a4180 0x7f5a601a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a5f7fe700 1 -- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a601a4580 msgr2=0x7f5a601a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a5f7fe700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a601a4580 0x7f5a601a2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a5f7fe700 1 -- 172.21.15.32:0/118491928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6008f400 con 0x7f5a601a8d20 2021-04-09T18:16:34.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a5f7fe700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a601a8d20 0x7f5a601a3b60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f5a50006f20 tx=0x7f5a5000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:34.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a57fff700 1 -- 172.21.15.32:0/118491928 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a5000bd80 con 0x7f5a601a8d20 2021-04-09T18:16:34.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a57fff700 1 -- 172.21.15.32:0/118491928 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a50000c90 con 0x7f5a601a8d20 2021-04-09T18:16:34.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6008f400 con 0x7f5a601a8d20 2021-04-09T18:16:34.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a57fff700 1 -- 172.21.15.32:0/118491928 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a500022b0 con 0x7f5a601a8d20 2021-04-09T18:16:34.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.244 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6008f400 con 0x7f5a601a8d20 2021-04-09T18:16:34.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.245 7f5a57fff700 1 -- 172.21.15.32:0/118491928 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5a5000a000 con 0x7f5a601a8d20 2021-04-09T18:16:34.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.245 7f5a57fff700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a3c018930 0x7f5a3c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.245 7f5a57fff700 1 -- 172.21.15.32:0/118491928 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5a50032a30 con 0x7f5a601a8d20 2021-04-09T18:16:34.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.245 7f5a5effd700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a3c018930 0x7f5a3c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.245 7f5a5effd700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a3c018930 0x7f5a3c01ae10 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:34.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.265 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6010d480 con 0x7f5a601a8d20 2021-04-09T18:16:34.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.268 7f5a57fff700 1 -- 172.21.15.32:0/118491928 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5a50006bb0 con 0x7f5a601a8d20 2021-04-09T18:16:34.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.462 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a6010d480 con 0x7f5a601a8d20 2021-04-09T18:16:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.462 7f5a57fff700 1 -- 172.21.15.32:0/118491928 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2767 (secure 0 0 0) 0x7f5a50006bb0 con 0x7f5a601a8d20 2021-04-09T18:16:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 49 2021-04-09T18:16:34.497 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:34.497 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":49,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3388525530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3388525530},{"type":"v1","addr":"172.21.15.32:6833","nonce":3388525530}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2165727103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2165727103},{"type":"v1","addr":"172.21.15.32:6835","nonce":2165727103}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":41},{"gid":55067,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3036306475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3036306475},{"type":"v1","addr":"172.21.15.86:6835","nonce":3036306475}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":55178,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1716252024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1716252024},{"type":"v1","addr":"172.21.15.86:6837","nonce":1716252024}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:15:46.073623","modified":"2021-04-09 18:16:16.028692","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-04-09T18:16:34.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.464 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a3c018930 msgr2=0x7f5a3c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.464 7f5a65ae3700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a3c018930 0x7f5a3c01ae10 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.464 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a601a8d20 msgr2=0x7f5a601a3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.464 7f5a65ae3700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a601a8d20 0x7f5a601a3b60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f5a50006f20 tx=0x7f5a5000b990).stop 2021-04-09T18:16:34.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.464 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 shutdown_connections 2021-04-09T18:16:34.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.464 7f5a65ae3700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a3c018930 0x7f5a3c01ae10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.465 7f5a65ae3700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a601a4580 0x7f5a601a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.465 7f5a65ae3700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a601a4180 0x7f5a601a22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.465 7f5a65ae3700 1 --2- 172.21.15.32:0/118491928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a601a8d20 0x7f5a601a3b60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.465 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 >> 172.21.15.32:0/118491928 conn(0x7f5a60179fd0 msgr2=0x7f5a6017f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.465 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 shutdown_connections 2021-04-09T18:16:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.465 7f5a65ae3700 1 -- 172.21.15.32:0/118491928 wait complete. 2021-04-09T18:16:34.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:16:34.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d21d9f700 1 Processor -- start 2021-04-09T18:16:34.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d21d9f700 1 -- start start 2021-04-09T18:16:34.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d21d9f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0d1c0cffb0 0x7f0d1c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d21d9f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0d1c0d6c90 0x7f0d1c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d21d9f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c0d7290 con 0x7f0d1c0d7810 2021-04-09T18:16:34.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d21d9f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c0d7090 con 0x7f0d1c0d6c90 2021-04-09T18:16:34.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d21d9f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c0d91f0 con 0x7f0d1c0cffb0 2021-04-09T18:16:34.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d1bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0d1c0d6c90 0x7f0d1c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d1affd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0d1c0d7810 0x7f0d1c0d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54242/0 (socket says 172.21.15.32:54242) 2021-04-09T18:16:34.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d1affd700 1 -- 172.21.15.32:0/301477079 learned_addr learned my addr 172.21.15.32:0/301477079 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:34.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.700 7f0d1b7fe700 1 --2- 172.21.15.32:0/301477079 >> v2:172.21.15.86:3300/0 conn(0x7f0d1c0cffb0 0x7f0d1c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1a7fc700 1 -- 172.21.15.32:0/301477079 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442013913 0 0) 0x7f0d100009b0 con 0x7f0d1c0d7810 2021-04-09T18:16:34.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1a7fc700 1 -- 172.21.15.32:0/301477079 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d04002fe0 con 0x7f0d1c0d7810 2021-04-09T18:16:34.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1bfff700 1 -- 172.21.15.32:0/301477079 >> v2:172.21.15.86:3300/0 conn(0x7f0d1c0cffb0 msgr2=0x7f0d1c0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:34.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1bfff700 1 --2- 172.21.15.32:0/301477079 >> v2:172.21.15.86:3300/0 conn(0x7f0d1c0cffb0 0x7f0d1c0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1bfff700 1 -- 172.21.15.32:0/301477079 >> v1:172.21.15.32:6789/0 conn(0x7f0d1c0d7810 legacy=0x7f0d1c0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1bfff700 1 -- 172.21.15.32:0/301477079 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c0d8400 con 0x7f0d1c0d6c90 2021-04-09T18:16:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1bfff700 1 --2- 172.21.15.32:0/301477079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0d6c90 0x7f0d1c10cfe0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f0d14006e30 tx=0x7f0d14009e60).ready entity=mon.0 client_cookie=e788bc4ee5e69b78 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1a7fc700 1 -- 172.21.15.32:0/301477079 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d14006a50 con 0x7f0d1c0d6c90 2021-04-09T18:16:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1a7fc700 1 -- 172.21.15.32:0/301477079 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1400b300 con 0x7f0d1c0d6c90 2021-04-09T18:16:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d1a7fc700 1 -- 172.21.15.32:0/301477079 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d1400b750 con 0x7f0d1c0d6c90 2021-04-09T18:16:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d21d9f700 1 -- 172.21.15.32:0/301477079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0d6c90 msgr2=0x7f0d1c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.701 7f0d21d9f700 1 --2- 172.21.15.32:0/301477079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0d6c90 0x7f0d1c10cfe0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f0d14006e30 tx=0x7f0d14009e60).stop 2021-04-09T18:16:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.702 7f0d21d9f700 1 -- 172.21.15.32:0/301477079 shutdown_connections 2021-04-09T18:16:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.702 7f0d21d9f700 1 --2- 172.21.15.32:0/301477079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0d6c90 0x7f0d1c10cfe0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.702 7f0d21d9f700 1 --2- 172.21.15.32:0/301477079 >> v2:172.21.15.86:3300/0 conn(0x7f0d1c0cffb0 0x7f0d1c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.702 7f0d21d9f700 1 -- 172.21.15.32:0/301477079 >> 172.21.15.32:0/301477079 conn(0x7f0d1c0ce5d0 msgr2=0x7f0d1c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:34.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.702 7f0d21d9f700 1 -- 172.21.15.32:0/301477079 shutdown_connections 2021-04-09T18:16:34.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.702 7f0d21d9f700 1 -- 172.21.15.32:0/301477079 wait complete. 2021-04-09T18:16:34.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.722 7f0d21d9f700 1 Processor -- start 2021-04-09T18:16:34.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.722 7f0d21d9f700 1 -- start start 2021-04-09T18:16:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d21d9f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d1c08d210 0x7f0d1c08fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d21d9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0903f0 0x7f0d1c1a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d21d9f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d1c1a6080 0x7f0d1c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d21d9f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c08d690 con 0x7f0d1c0903f0 2021-04-09T18:16:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d21d9f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0b0fd0 con 0x7f0d1c1a6080 2021-04-09T18:16:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d21d9f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c1a6480 con 0x7f0d1c08d210 2021-04-09T18:16:34.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0903f0 0x7f0d1c1a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0903f0 0x7f0d1c1a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44656/0 (socket says 172.21.15.32:44656) 2021-04-09T18:16:34.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1affd700 1 -- 172.21.15.32:0/3557536808 learned_addr learned my addr 172.21.15.32:0/3557536808 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:34.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1b7fe700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d1c08d210 0x7f0d1c08fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1bfff700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d1c1a6080 0x7f0d1c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1affd700 1 -- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d1c08d210 msgr2=0x7f0d1c08fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1affd700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d1c08d210 0x7f0d1c08fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1affd700 1 -- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d1c1a6080 msgr2=0x7f0d1c1a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:34.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1affd700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d1c1a6080 0x7f0d1c1a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:34.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1affd700 1 -- 172.21.15.32:0/3557536808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c08d8b0 con 0x7f0d1c0903f0 2021-04-09T18:16:34.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d1affd700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0903f0 0x7f0d1c1a5a60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f0d1000a4c0 tx=0x7f0d1000a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:34.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.723 7f0d03fff700 1 -- 172.21.15.32:0/3557536808 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d1000ac50 con 0x7f0d1c0903f0 2021-04-09T18:16:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.724 7f0d03fff700 1 -- 172.21.15.32:0/3557536808 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d10004140 con 0x7f0d1c0903f0 2021-04-09T18:16:34.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.724 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c08d8b0 con 0x7f0d1c0903f0 2021-04-09T18:16:34.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.724 7f0d03fff700 1 -- 172.21.15.32:0/3557536808 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d10004620 con 0x7f0d1c0903f0 2021-04-09T18:16:34.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.724 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c08d8b0 con 0x7f0d1c0903f0 2021-04-09T18:16:34.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.724 7f0d03fff700 1 -- 172.21.15.32:0/3557536808 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0d10004570 con 0x7f0d1c0903f0 2021-04-09T18:16:34.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.724 7f0d03fff700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0cfc018930 0x7f0cfc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:34.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.724 7f0d03fff700 1 -- 172.21.15.32:0/3557536808 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0d10032b80 con 0x7f0d1c0903f0 2021-04-09T18:16:34.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.724 7f0d1b7fe700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0cfc018930 0x7f0cfc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:34.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.725 7f0d1b7fe700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0cfc018930 0x7f0cfc01ae10 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.744 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c1a52b0 con 0x7f0d1c0903f0 2021-04-09T18:16:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.748 7f0d03fff700 1 -- 172.21.15.32:0/3557536808 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0d10004360 con 0x7f0d1c0903f0 2021-04-09T18:16:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:34.928 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0d1c1a52b0 con 0x7f0d1c0903f0 2021-04-09T18:16:35.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:16:35.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:16:35.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.359 7f0d03fff700 1 -- 172.21.15.32:0/3557536808 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0d10004570 con 0x7f0d1c0903f0 2021-04-09T18:16:35.423 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.395 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0cfc018930 msgr2=0x7f0cfc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:35.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0cfc018930 0x7f0cfc01ae10 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0903f0 msgr2=0x7f0d1c1a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:35.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0903f0 0x7f0d1c1a5a60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f0d1000a4c0 tx=0x7f0d1000a860).stop 2021-04-09T18:16:35.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 shutdown_connections 2021-04-09T18:16:35.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0cfc018930 0x7f0cfc01ae10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d1c0903f0 0x7f0d1c1a5a60 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d1c1a6080 0x7f0d1c1a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 --2- 172.21.15.32:0/3557536808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d1c08d210 0x7f0d1c08fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 >> 172.21.15.32:0/3557536808 conn(0x7f0d1c1a8f50 msgr2=0x7f0d1c188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:35.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 shutdown_connections 2021-04-09T18:16:35.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.396 7f0d21d9f700 1 -- 172.21.15.32:0/3557536808 wait complete. 2021-04-09T18:16:35.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:16:35.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.559 7f74c5793700 1 Processor -- start 2021-04-09T18:16:35.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.559 7f74c5793700 1 -- start start 2021-04-09T18:16:35.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.559 7f74c5793700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c010d010 0x7f74c00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.559 7f74c5793700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f74c00d1da0 0x7f74c00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.559 7f74c5793700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c010d480 con 0x7f74c00d82c0 2021-04-09T18:16:35.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.559 7f74c5793700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c010e780 con 0x7f74c00d1da0 2021-04-09T18:16:35.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.559 7f74c5793700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c00d8d30 con 0x7f74c010d010 2021-04-09T18:16:35.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bed73700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f74c00d1da0 0x7f74c00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:35.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bf574700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c010d010 0x7f74c00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:35.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bfd75700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f74c00d82c0 0x7f74c010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52180/0 (socket says 172.21.15.32:52180) 2021-04-09T18:16:35.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bfd75700 1 -- 172.21.15.32:0/1227547542 learned_addr learned my addr 172.21.15.32:0/1227547542 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:35.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bed73700 1 -- 172.21.15.32:0/1227547542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c010d010 msgr2=0x7f74c00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:35.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bed73700 1 --2- 172.21.15.32:0/1227547542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c010d010 0x7f74c00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bed73700 1 -- 172.21.15.32:0/1227547542 >> v1:172.21.15.86:6789/0 conn(0x7f74c00d82c0 legacy=0x7f74c010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:35.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bed73700 1 -- 172.21.15.32:0/1227547542 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c00d9220 con 0x7f74c00d1da0 2021-04-09T18:16:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74be572700 1 -- 172.21.15.32:0/1227547542 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932937294 0 0) 0x7f74b8001440 con 0x7f74c00d82c0 2021-04-09T18:16:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bf574700 1 --2- 172.21.15.32:0/1227547542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c010d010 0x7f74c00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:35.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74bed73700 1 --2- 172.21.15.32:0/1227547542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74c00d1da0 0x7f74c00d7ca0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f74b4006650 tx=0x7f74b4009a80).ready entity=mon.1 client_cookie=72fbd3d39a3233d3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:35.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74be572700 1 -- 172.21.15.32:0/1227547542 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74b40009b0 con 0x7f74c00d1da0 2021-04-09T18:16:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74be572700 1 -- 172.21.15.32:0/1227547542 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b400c0c0 con 0x7f74c00d1da0 2021-04-09T18:16:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74be572700 1 -- 172.21.15.32:0/1227547542 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74b400c510 con 0x7f74c00d1da0 2021-04-09T18:16:35.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.560 7f74c5793700 1 -- 172.21.15.32:0/1227547542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74c00d1da0 msgr2=0x7f74c00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:35.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.561 7f74c5793700 1 --2- 172.21.15.32:0/1227547542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74c00d1da0 0x7f74c00d7ca0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f74b4006650 tx=0x7f74b4009a80).stop 2021-04-09T18:16:35.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.561 7f74c5793700 1 -- 172.21.15.32:0/1227547542 shutdown_connections 2021-04-09T18:16:35.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.561 7f74c5793700 1 --2- 172.21.15.32:0/1227547542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74c00d1da0 0x7f74c00d7ca0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.561 7f74c5793700 1 --2- 172.21.15.32:0/1227547542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c010d010 0x7f74c00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.561 7f74c5793700 1 -- 172.21.15.32:0/1227547542 >> 172.21.15.32:0/1227547542 conn(0x7f74c00ce5f0 msgr2=0x7f74c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:35.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.561 7f74c5793700 1 -- 172.21.15.32:0/1227547542 shutdown_connections 2021-04-09T18:16:35.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.561 7f74c5793700 1 -- 172.21.15.32:0/1227547542 wait complete. 2021-04-09T18:16:35.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.581 7f74c5793700 1 Processor -- start 2021-04-09T18:16:35.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.581 7f74c5793700 1 -- start start 2021-04-09T18:16:35.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74c5793700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c0185060 0x7f74c0185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:35.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74c5793700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c0186c00 0x7f74c0187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:35.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74c5793700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74c0191360 0x7f74c0191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:35.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74c5793700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c01863a0 con 0x7f74c0186c00 2021-04-09T18:16:35.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74c5793700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c0185a60 con 0x7f74c0191360 2021-04-09T18:16:35.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74c5793700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c0187620 con 0x7f74c0185060 2021-04-09T18:16:35.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c0186c00 0x7f74c0187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:35.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c0186c00 0x7f74c0187000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44672/0 (socket says 172.21.15.32:44672) 2021-04-09T18:16:35.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bed73700 1 -- 172.21.15.32:0/964218341 learned_addr learned my addr 172.21.15.32:0/964218341 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:35.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bf574700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c0185060 0x7f74c0185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:35.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bfd75700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74c0191360 0x7f74c0191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bed73700 1 -- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c0185060 msgr2=0x7f74c0185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bed73700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c0185060 0x7f74c0185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bed73700 1 -- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74c0191360 msgr2=0x7f74c0191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:35.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bed73700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74c0191360 0x7f74c0191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.582 7f74bed73700 1 -- 172.21.15.32:0/964218341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c0185460 con 0x7f74c0186c00 2021-04-09T18:16:35.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.583 7f74bed73700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c0186c00 0x7f74c0187000 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f74b4009650 tx=0x7f74b400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:35.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.583 7f74a7fff700 1 -- 172.21.15.32:0/964218341 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74b40093a0 con 0x7f74c0186c00 2021-04-09T18:16:35.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.583 7f74a7fff700 1 -- 172.21.15.32:0/964218341 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b400bd50 con 0x7f74c0186c00 2021-04-09T18:16:35.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.583 7f74a7fff700 1 -- 172.21.15.32:0/964218341 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74b40052e0 con 0x7f74c0186c00 2021-04-09T18:16:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.583 7f74c5793700 1 -- 172.21.15.32:0/964218341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c0185460 con 0x7f74c0186c00 2021-04-09T18:16:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.583 7f74c5793700 1 -- 172.21.15.32:0/964218341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c0185460 con 0x7f74c0186c00 2021-04-09T18:16:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.583 7f74a7fff700 1 -- 172.21.15.32:0/964218341 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f74b4006030 con 0x7f74c0186c00 2021-04-09T18:16:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.584 7f74a7fff700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74a00188c0 0x7f74a001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.584 7f74a7fff700 1 -- 172.21.15.32:0/964218341 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f74b40329e0 con 0x7f74c0186c00 2021-04-09T18:16:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.584 7f74bf574700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74a00188c0 0x7f74a001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.584 7f74bf574700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74a00188c0 0x7f74a001adc0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.604 7f74c5793700 1 -- 172.21.15.32:0/964218341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c01866d0 con 0x7f74c0186c00 2021-04-09T18:16:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.607 7f74a7fff700 1 -- 172.21.15.32:0/964218341 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f74b40009c0 con 0x7f74c0186c00 2021-04-09T18:16:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.798 7f74c5793700 1 -- 172.21.15.32:0/964218341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74c01866d0 con 0x7f74c0186c00 2021-04-09T18:16:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.799 7f74a7fff700 1 -- 172.21.15.32:0/964218341 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+2767 (secure 0 0 0) 0x7f74b400bd50 con 0x7f74c0186c00 2021-04-09T18:16:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 50 2021-04-09T18:16:35.882 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:35.883 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":50,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3388525530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3388525530},{"type":"v1","addr":"172.21.15.32:6833","nonce":3388525530}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2165727103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2165727103},{"type":"v1","addr":"172.21.15.32:6835","nonce":2165727103}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":41},{"gid":55067,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3036306475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3036306475},{"type":"v1","addr":"172.21.15.86:6835","nonce":3036306475}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":55178,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1716252024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1716252024},{"type":"v1","addr":"172.21.15.86:6837","nonce":1716252024}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:15:46.073623","modified":"2021-04-09 18:16:35.355249","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-04-09T18:16:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 -- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74a00188c0 msgr2=0x7f74a001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74a00188c0 0x7f74a001adc0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 -- 172.21.15.32:0/964218341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c0186c00 msgr2=0x7f74c0187000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c0186c00 0x7f74c0187000 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f74b4009650 tx=0x7f74b400b960).stop 2021-04-09T18:16:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 -- 172.21.15.32:0/964218341 shutdown_connections 2021-04-09T18:16:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74a00188c0 0x7f74a001adc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74c0186c00 0x7f74c0187000 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74c0191360 0x7f74c0191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 --2- 172.21.15.32:0/964218341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74c0185060 0x7f74c0185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:35.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.801 7f74c5793700 1 -- 172.21.15.32:0/964218341 >> 172.21.15.32:0/964218341 conn(0x7f74c01837d0 msgr2=0x7f74c0195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:35.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.802 7f74c5793700 1 -- 172.21.15.32:0/964218341 shutdown_connections 2021-04-09T18:16:35.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:35.802 7f74c5793700 1 -- 172.21.15.32:0/964218341 wait complete. 2021-04-09T18:16:35.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:16:36.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.071 7fb41f435700 1 Processor -- start 2021-04-09T18:16:36.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.071 7fb41f435700 1 -- start start 2021-04-09T18:16:36.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41f435700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4180cffb0 0x7fb4180d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41f435700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb4180d6c90 0x7fb41810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41f435700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4180d7290 con 0x7fb4180d7810 2021-04-09T18:16:36.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41f435700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4180d7090 con 0x7fb4180d6c90 2021-04-09T18:16:36.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41f435700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180d8590 con 0x7fb4180cffb0 2021-04-09T18:16:36.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41ca50700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb4180d7810 0x7fb4180d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54270/0 (socket says 172.21.15.32:54270) 2021-04-09T18:16:36.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41ca50700 1 -- 172.21.15.32:0/158246926 learned_addr learned my addr 172.21.15.32:0/158246926 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:36.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41da52700 1 --2- 172.21.15.32:0/158246926 >> v2:172.21.15.86:3300/0 conn(0x7fb4180d6c90 0x7fb41810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41d251700 1 --2- 172.21.15.32:0/158246926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4180cffb0 0x7fb4180d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb40bfff700 1 -- 172.21.15.32:0/158246926 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834649863 0 0) 0x7fb40c0009b0 con 0x7fb4180d7810 2021-04-09T18:16:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb40bfff700 1 -- 172.21.15.32:0/158246926 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb404002fe0 con 0x7fb4180d7810 2021-04-09T18:16:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41da52700 1 -- 172.21.15.32:0/158246926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4180cffb0 msgr2=0x7fb4180d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41da52700 1 --2- 172.21.15.32:0/158246926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4180cffb0 0x7fb4180d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41da52700 1 -- 172.21.15.32:0/158246926 >> v1:172.21.15.32:6789/0 conn(0x7fb4180d7810 legacy=0x7fb4180d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41da52700 1 -- 172.21.15.32:0/158246926 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4180d91f0 con 0x7fb4180d6c90 2021-04-09T18:16:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.072 7fb41d251700 1 --2- 172.21.15.32:0/158246926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4180cffb0 0x7fb4180d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.073 7fb41da52700 1 --2- 172.21.15.32:0/158246926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4180d6c90 0x7fb41810cfe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fb410006e50 tx=0x7fb410009e70).ready entity=mon.1 client_cookie=6085e8626ca0fd94 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:36.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.073 7fb40bfff700 1 -- 172.21.15.32:0/158246926 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb410006a20 con 0x7fb4180d6c90 2021-04-09T18:16:36.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.073 7fb40bfff700 1 -- 172.21.15.32:0/158246926 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb410006790 con 0x7fb4180d6c90 2021-04-09T18:16:36.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.073 7fb40bfff700 1 -- 172.21.15.32:0/158246926 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb41000b700 con 0x7fb4180d6c90 2021-04-09T18:16:36.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.073 7fb41f435700 1 -- 172.21.15.32:0/158246926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4180d6c90 msgr2=0x7fb41810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.073 7fb41f435700 1 --2- 172.21.15.32:0/158246926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4180d6c90 0x7fb41810cfe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fb410006e50 tx=0x7fb410009e70).stop 2021-04-09T18:16:36.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.074 7fb41f435700 1 -- 172.21.15.32:0/158246926 shutdown_connections 2021-04-09T18:16:36.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.074 7fb41f435700 1 --2- 172.21.15.32:0/158246926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4180d6c90 0x7fb41810cfe0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.074 7fb41f435700 1 --2- 172.21.15.32:0/158246926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4180cffb0 0x7fb4180d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.074 7fb41f435700 1 -- 172.21.15.32:0/158246926 >> 172.21.15.32:0/158246926 conn(0x7fb4180ce5d0 msgr2=0x7fb4180cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:36.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.074 7fb41f435700 1 -- 172.21.15.32:0/158246926 shutdown_connections 2021-04-09T18:16:36.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.074 7fb41f435700 1 -- 172.21.15.32:0/158246926 wait complete. 2021-04-09T18:16:36.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.094 7fb41f435700 1 Processor -- start 2021-04-09T18:16:36.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.094 7fb41f435700 1 -- start start 2021-04-09T18:16:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41f435700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4181ac250 0x7fb4181ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41f435700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb418095550 0x7fb41817ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41f435700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb418095950 0x7fb41817b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41f435700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4181abae0 con 0x7fb418095950 2021-04-09T18:16:36.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41f435700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4181abd10 con 0x7fb4181ac250 2021-04-09T18:16:36.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41f435700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41817ba80 con 0x7fb418095550 2021-04-09T18:16:36.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41da52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb418095950 0x7fb41817b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41da52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb418095950 0x7fb41817b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44686/0 (socket says 172.21.15.32:44686) 2021-04-09T18:16:36.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41da52700 1 -- 172.21.15.32:0/370491941 learned_addr learned my addr 172.21.15.32:0/370491941 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:36.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41d251700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4181ac250 0x7fb4181ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41ca50700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb418095550 0x7fb41817ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41da52700 1 -- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb418095550 msgr2=0x7fb41817ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41da52700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb418095550 0x7fb41817ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41da52700 1 -- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4181ac250 msgr2=0x7fb4181ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41da52700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4181ac250 0x7fb4181ab4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41da52700 1 -- 172.21.15.32:0/370491941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41817bca0 con 0x7fb418095950 2021-04-09T18:16:36.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41da52700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb418095950 0x7fb41817b460 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fb4100099f0 tx=0x7fb41000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:36.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb409ffb700 1 -- 172.21.15.32:0/370491941 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4100093d0 con 0x7fb418095950 2021-04-09T18:16:36.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb409ffb700 1 -- 172.21.15.32:0/370491941 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4100090c0 con 0x7fb418095950 2021-04-09T18:16:36.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb41f435700 1 -- 172.21.15.32:0/370491941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41817bca0 con 0x7fb418095950 2021-04-09T18:16:36.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.095 7fb409ffb700 1 -- 172.21.15.32:0/370491941 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb410001420 con 0x7fb418095950 2021-04-09T18:16:36.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.096 7fb41f435700 1 -- 172.21.15.32:0/370491941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41817bca0 con 0x7fb418095950 2021-04-09T18:16:36.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.096 7fb409ffb700 1 -- 172.21.15.32:0/370491941 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb4100056b0 con 0x7fb418095950 2021-04-09T18:16:36.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.096 7fb409ffb700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3f8018930 0x7fb3f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.096 7fb409ffb700 1 -- 172.21.15.32:0/370491941 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb410032d60 con 0x7fb418095950 2021-04-09T18:16:36.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.096 7fb41d251700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3f8018930 0x7fb3f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.097 7fb41d251700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3f8018930 0x7fb3f801ae10 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.116 7fb41f435700 1 -- 172.21.15.32:0/370491941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4181b7140 con 0x7fb418095950 2021-04-09T18:16:36.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.119 7fb409ffb700 1 -- 172.21.15.32:0/370491941 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb4100090a0 con 0x7fb418095950 2021-04-09T18:16:36.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.313 7fb41f435700 1 -- 172.21.15.32:0/370491941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb4181b7140 con 0x7fb418095950 2021-04-09T18:16:36.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.363 7fb409ffb700 1 -- 172.21.15.32:0/370491941 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb4100090a0 con 0x7fb418095950 2021-04-09T18:16:36.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 -- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3f8018930 msgr2=0x7fb3f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3f8018930 0x7fb3f801ae10 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 -- 172.21.15.32:0/370491941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb418095950 msgr2=0x7fb41817b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb418095950 0x7fb41817b460 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fb4100099f0 tx=0x7fb41000bd10).stop 2021-04-09T18:16:36.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 -- 172.21.15.32:0/370491941 shutdown_connections 2021-04-09T18:16:36.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3f8018930 0x7fb3f801ae10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb418095950 0x7fb41817b460 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb418095550 0x7fb41817ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 --2- 172.21.15.32:0/370491941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4181ac250 0x7fb4181ab4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 -- 172.21.15.32:0/370491941 >> 172.21.15.32:0/370491941 conn(0x7fb4181b7370 msgr2=0x7fb41808ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:36.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 -- 172.21.15.32:0/370491941 shutdown_connections 2021-04-09T18:16:36.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.379 7fb41f435700 1 -- 172.21.15.32:0/370491941 wait complete. 2021-04-09T18:16:36.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:16:36.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.551 7f433c346700 1 Processor -- start 2021-04-09T18:16:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.551 7f433c346700 1 -- start start 2021-04-09T18:16:36.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f433c346700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f43340d7cc0 0x7f43340d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f433c346700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433410da80 0x7f433410de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f433c346700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433410d500 con 0x7f433410d050 2021-04-09T18:16:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f433c346700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43340d9260 con 0x7f43340d7cc0 2021-04-09T18:16:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f433c346700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43340d0330 con 0x7f433410da80 2021-04-09T18:16:36.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f433a162700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f433410d050 0x7f43340cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54280/0 (socket says 172.21.15.32:54280) 2021-04-09T18:16:36.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f433a162700 1 -- 172.21.15.32:0/2525149819 learned_addr learned my addr 172.21.15.32:0/2525149819 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:36.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f4339961700 1 --2- 172.21.15.32:0/2525149819 >> v2:172.21.15.86:3300/0 conn(0x7f43340d7cc0 0x7f43340d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f433a963700 1 --2- 172.21.15.32:0/2525149819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433410da80 0x7f433410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.552 7f4339160700 1 -- 172.21.15.32:0/2525149819 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211880823 0 0) 0x7f4330000ce0 con 0x7f433410d050 2021-04-09T18:16:36.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f4339160700 1 -- 172.21.15.32:0/2525149819 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4320002fe0 con 0x7f433410d050 2021-04-09T18:16:36.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f4339961700 1 -- 172.21.15.32:0/2525149819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433410da80 msgr2=0x7f433410de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:36.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f4339961700 1 --2- 172.21.15.32:0/2525149819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433410da80 0x7f433410de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f4339961700 1 -- 172.21.15.32:0/2525149819 >> v1:172.21.15.32:6789/0 conn(0x7f433410d050 legacy=0x7f43340cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f4339961700 1 -- 172.21.15.32:0/2525149819 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340d86e0 con 0x7f43340d7cc0 2021-04-09T18:16:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f433a963700 1 --2- 172.21.15.32:0/2525149819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433410da80 0x7f433410de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:36.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f4339961700 1 --2- 172.21.15.32:0/2525149819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43340d7cc0 0x7f43340d80c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f4324006650 tx=0x7f4324009a80).ready entity=mon.1 client_cookie=93cbf25b15fb0273 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:36.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f4339160700 1 -- 172.21.15.32:0/2525149819 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43240009b0 con 0x7f43340d7cc0 2021-04-09T18:16:36.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f4339160700 1 -- 172.21.15.32:0/2525149819 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432400c0c0 con 0x7f43340d7cc0 2021-04-09T18:16:36.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f4339160700 1 -- 172.21.15.32:0/2525149819 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432400c510 con 0x7f43340d7cc0 2021-04-09T18:16:36.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f433c346700 1 -- 172.21.15.32:0/2525149819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43340d7cc0 msgr2=0x7f43340d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.553 7f433c346700 1 --2- 172.21.15.32:0/2525149819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43340d7cc0 0x7f43340d80c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f4324006650 tx=0x7f4324009a80).stop 2021-04-09T18:16:36.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.554 7f433c346700 1 -- 172.21.15.32:0/2525149819 shutdown_connections 2021-04-09T18:16:36.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.554 7f433c346700 1 --2- 172.21.15.32:0/2525149819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433410da80 0x7f433410de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.554 7f433c346700 1 --2- 172.21.15.32:0/2525149819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43340d7cc0 0x7f43340d80c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.554 7f433c346700 1 -- 172.21.15.32:0/2525149819 >> 172.21.15.32:0/2525149819 conn(0x7f43340ce620 msgr2=0x7f43340d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.554 7f433c346700 1 -- 172.21.15.32:0/2525149819 shutdown_connections 2021-04-09T18:16:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.554 7f433c346700 1 -- 172.21.15.32:0/2525149819 wait complete. 2021-04-09T18:16:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.574 7f433c346700 1 Processor -- start 2021-04-09T18:16:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.574 7f433c346700 1 -- start start 2021-04-09T18:16:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.574 7f433c346700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43340b84d0 0x7f43340c2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.574 7f433c346700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43340c2e20 0x7f43340a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.574 7f433c346700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43340a1b60 0x7f43340a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.574 7f433c346700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433418e9f0 con 0x7f43340a1b60 2021-04-09T18:16:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.574 7f433c346700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43340a8420 con 0x7f43340c2e20 2021-04-09T18:16:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.574 7f433c346700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43340b3d90 con 0x7f43340b84d0 2021-04-09T18:16:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a963700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43340a1b60 0x7f43340a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a963700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43340a1b60 0x7f43340a7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44700/0 (socket says 172.21.15.32:44700) 2021-04-09T18:16:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a963700 1 -- 172.21.15.32:0/988507094 learned_addr learned my addr 172.21.15.32:0/988507094 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f4339961700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43340c2e20 0x7f43340a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a162700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43340b84d0 0x7f43340c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a963700 1 -- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43340b84d0 msgr2=0x7f43340c2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a963700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43340b84d0 0x7f43340c2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a963700 1 -- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43340c2e20 msgr2=0x7f43340a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a963700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43340c2e20 0x7f43340a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a963700 1 -- 172.21.15.32:0/988507094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340a7cb0 con 0x7f43340a1b60 2021-04-09T18:16:36.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433a963700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43340a1b60 0x7f43340a7690 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f432c00b970 tx=0x7f432c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:36.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f432affd700 1 -- 172.21.15.32:0/988507094 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432c000f90 con 0x7f43340a1b60 2021-04-09T18:16:36.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433c346700 1 -- 172.21.15.32:0/988507094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43340a7cb0 con 0x7f43340a1b60 2021-04-09T18:16:36.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f432affd700 1 -- 172.21.15.32:0/988507094 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432c006aa0 con 0x7f43340a1b60 2021-04-09T18:16:36.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f432affd700 1 -- 172.21.15.32:0/988507094 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432c0015f0 con 0x7f43340a1b60 2021-04-09T18:16:36.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.575 7f433c346700 1 -- 172.21.15.32:0/988507094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43340a7cb0 con 0x7f43340a1b60 2021-04-09T18:16:36.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.576 7f432affd700 1 -- 172.21.15.32:0/988507094 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f432c000ed0 con 0x7f43340a1b60 2021-04-09T18:16:36.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.576 7f432affd700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f431c018930 0x7f431c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:36.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.576 7f432affd700 1 -- 172.21.15.32:0/988507094 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f432c032da0 con 0x7f43340a1b60 2021-04-09T18:16:36.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.576 7f433a162700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f431c018930 0x7f431c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:36.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.576 7f433a162700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f431c018930 0x7f431c01adc0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:36.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.596 7f433c346700 1 -- 172.21.15.32:0/988507094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43341b7180 con 0x7f43340a1b60 2021-04-09T18:16:36.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.600 7f432affd700 1 -- 172.21.15.32:0/988507094 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f432c0066f0 con 0x7f43340a1b60 2021-04-09T18:16:36.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:36.783 7f433c346700 1 -- 172.21.15.32:0/988507094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f433410d500 con 0x7f43340a1b60 2021-04-09T18:16:37.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.365 7f432affd700 1 -- 172.21.15.32:0/988507094 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v55) v1 ==== 185+0+0 (secure 0 0 0) 0x7f432c000ed0 con 0x7f43340a1b60 2021-04-09T18:16:37.428 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:16:37.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 -- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f431c018930 msgr2=0x7f431c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:37.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f431c018930 0x7f431c01adc0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 -- 172.21.15.32:0/988507094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43340a1b60 msgr2=0x7f43340a7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43340a1b60 0x7f43340a7690 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f432c00b970 tx=0x7f432c00bd10).stop 2021-04-09T18:16:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 -- 172.21.15.32:0/988507094 shutdown_connections 2021-04-09T18:16:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f431c018930 0x7f431c01adc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43340a1b60 0x7f43340a7690 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43340c2e20 0x7f43340a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 --2- 172.21.15.32:0/988507094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43340b84d0 0x7f43340c2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 -- 172.21.15.32:0/988507094 >> 172.21.15.32:0/988507094 conn(0x7f43340b6b70 msgr2=0x7f43340b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 -- 172.21.15.32:0/988507094 shutdown_connections 2021-04-09T18:16:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.400 7f433c346700 1 -- 172.21.15.32:0/988507094 wait complete. 2021-04-09T18:16:37.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:16:37.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.575 7fc99b21c700 1 Processor -- start 2021-04-09T18:16:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.575 7fc99b21c700 1 -- start start 2021-04-09T18:16:37.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc99b21c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc99410d050 0x7fc9940cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:37.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc99b21c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc9940d84b0 0x7fc99410da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:37.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc99b21c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99410d500 con 0x7fc9940d0160 2021-04-09T18:16:37.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc99b21c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99410e800 con 0x7fc9940d84b0 2021-04-09T18:16:37.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc99b21c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9940d8d70 con 0x7fc99410d050 2021-04-09T18:16:37.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc999038700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc99410d050 0x7fc9940cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:37.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc999839700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc9940d84b0 0x7fc99410da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:37.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc998837700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc9940d0160 0x7fc9940d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52222/0 (socket says 172.21.15.32:52222) 2021-04-09T18:16:37.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc998837700 1 -- 172.21.15.32:0/1875657468 learned_addr learned my addr 172.21.15.32:0/1875657468 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:37.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc98bfff700 1 -- 172.21.15.32:0/1875657468 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251057448 0 0) 0x7fc9840010c0 con 0x7fc9940d0160 2021-04-09T18:16:37.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc98bfff700 1 -- 172.21.15.32:0/1875657468 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc980002fe0 con 0x7fc9940d0160 2021-04-09T18:16:37.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc999839700 1 -- 172.21.15.32:0/1875657468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc99410d050 msgr2=0x7fc9940cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:37.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc999839700 1 --2- 172.21.15.32:0/1875657468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc99410d050 0x7fc9940cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc999839700 1 -- 172.21.15.32:0/1875657468 >> v1:172.21.15.86:6789/0 conn(0x7fc9940d0160 legacy=0x7fc9940d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:37.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc999839700 1 -- 172.21.15.32:0/1875657468 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9940d9260 con 0x7fc9940d84b0 2021-04-09T18:16:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.576 7fc999038700 1 --2- 172.21.15.32:0/1875657468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc99410d050 0x7fc9940cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:37.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.577 7fc999839700 1 --2- 172.21.15.32:0/1875657468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9940d84b0 0x7fc99410da80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc98c006e50 tx=0x7fc98c009e70).ready entity=mon.1 client_cookie=5b9399b78054ba13 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:37.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.577 7fc98bfff700 1 -- 172.21.15.32:0/1875657468 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc98c006a20 con 0x7fc9940d84b0 2021-04-09T18:16:37.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.577 7fc98bfff700 1 -- 172.21.15.32:0/1875657468 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98c006790 con 0x7fc9940d84b0 2021-04-09T18:16:37.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.577 7fc98bfff700 1 -- 172.21.15.32:0/1875657468 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc98c00b700 con 0x7fc9940d84b0 2021-04-09T18:16:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.577 7fc99b21c700 1 -- 172.21.15.32:0/1875657468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9940d84b0 msgr2=0x7fc99410da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.577 7fc99b21c700 1 --2- 172.21.15.32:0/1875657468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9940d84b0 0x7fc99410da80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc98c006e50 tx=0x7fc98c009e70).stop 2021-04-09T18:16:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.578 7fc99b21c700 1 -- 172.21.15.32:0/1875657468 shutdown_connections 2021-04-09T18:16:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.578 7fc99b21c700 1 --2- 172.21.15.32:0/1875657468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9940d84b0 0x7fc99410da80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.578 7fc99b21c700 1 --2- 172.21.15.32:0/1875657468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc99410d050 0x7fc9940cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.578 7fc99b21c700 1 -- 172.21.15.32:0/1875657468 >> 172.21.15.32:0/1875657468 conn(0x7fc9940ce620 msgr2=0x7fc9940d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:37.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.578 7fc99b21c700 1 -- 172.21.15.32:0/1875657468 shutdown_connections 2021-04-09T18:16:37.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.578 7fc99b21c700 1 -- 172.21.15.32:0/1875657468 wait complete. 2021-04-09T18:16:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.598 7fc99b21c700 1 Processor -- start 2021-04-09T18:16:37.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.598 7fc99b21c700 1 -- start start 2021-04-09T18:16:37.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc99b21c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9940b0920 0x7fc9940b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:37.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc99b21c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9940a7690 0x7fc9940a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:37.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc99b21c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9940c2510 0x7fc9940c2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:37.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc99b21c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9940b1340 con 0x7fc9940b0920 2021-04-09T18:16:37.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc99b21c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9940a82e0 con 0x7fc9940c2510 2021-04-09T18:16:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc99b21c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9940a8820 con 0x7fc9940a7690 2021-04-09T18:16:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999038700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9940b0920 0x7fc9940b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999038700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9940b0920 0x7fc9940b0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44712/0 (socket says 172.21.15.32:44712) 2021-04-09T18:16:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999038700 1 -- 172.21.15.32:0/4132724015 learned_addr learned my addr 172.21.15.32:0/4132724015 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc998837700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9940a7690 0x7fc9940a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999839700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9940c2510 0x7fc9940c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999038700 1 -- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9940a7690 msgr2=0x7fc9940a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999038700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9940a7690 0x7fc9940a7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999038700 1 -- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9940c2510 msgr2=0x7fc9940c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999038700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9940c2510 0x7fc9940c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999038700 1 -- 172.21.15.32:0/4132724015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99408fba0 con 0x7fc9940b0920 2021-04-09T18:16:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc999038700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9940b0920 0x7fc9940b0d20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fc990006f40 tx=0x7fc99000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc989ffb700 1 -- 172.21.15.32:0/4132724015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc99000bd80 con 0x7fc9940b0920 2021-04-09T18:16:37.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc989ffb700 1 -- 172.21.15.32:0/4132724015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc990000c90 con 0x7fc9940b0920 2021-04-09T18:16:37.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc989ffb700 1 -- 172.21.15.32:0/4132724015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc990002220 con 0x7fc9940b0920 2021-04-09T18:16:37.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.599 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99408fba0 con 0x7fc9940b0920 2021-04-09T18:16:37.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.600 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99408fba0 con 0x7fc9940b0920 2021-04-09T18:16:37.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.600 7fc989ffb700 1 -- 172.21.15.32:0/4132724015 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc99000a000 con 0x7fc9940b0920 2021-04-09T18:16:37.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.600 7fc989ffb700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc974018930 0x7fc97401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.600 7fc989ffb700 1 -- 172.21.15.32:0/4132724015 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5139+0+0 (secure 0 0 0) 0x7fc990032830 con 0x7fc9940b0920 2021-04-09T18:16:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.600 7fc998837700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc974018930 0x7fc97401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:37.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.601 7fc998837700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc974018930 0x7fc97401ae10 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.620 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9941b7180 con 0x7fc9940b0920 2021-04-09T18:16:37.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.623 7fc989ffb700 1 -- 172.21.15.32:0/4132724015 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc990000c90 con 0x7fc9940b0920 2021-04-09T18:16:37.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:37.805 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc99410d500 con 0x7fc9940b0920 2021-04-09T18:16:38.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.376 7fc989ffb700 1 -- 172.21.15.32:0/4132724015 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v56) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc99000a000 con 0x7fc9940b0920 2021-04-09T18:16:38.450 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:16:38.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc974018930 msgr2=0x7fc97401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:38.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc974018930 0x7fc97401ae10 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9940b0920 msgr2=0x7fc9940b0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:38.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9940b0920 0x7fc9940b0d20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fc990006f40 tx=0x7fc99000b990).stop 2021-04-09T18:16:38.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 shutdown_connections 2021-04-09T18:16:38.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc974018930 0x7fc97401ae10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9940c2510 0x7fc9940c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9940a7690 0x7fc9940a7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 --2- 172.21.15.32:0/4132724015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9940b0920 0x7fc9940b0d20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 >> 172.21.15.32:0/4132724015 conn(0x7fc9940b6b70 msgr2=0x7fc9940b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:38.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 shutdown_connections 2021-04-09T18:16:38.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.422 7fc99b21c700 1 -- 172.21.15.32:0/4132724015 wait complete. 2021-04-09T18:16:38.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:16:38.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.588 7f97ae6c3700 1 Processor -- start 2021-04-09T18:16:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.588 7f97ae6c3700 1 -- start start 2021-04-09T18:16:38.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f97ae6c3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a80cffb0 0x7f97a80d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:38.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f97ae6c3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80d7290 con 0x7f97a80d7810 2021-04-09T18:16:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f97ae6c3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a810e7e0 con 0x7f97a80d6c90 2021-04-09T18:16:38.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f97ae6c3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80d7090 con 0x7f97a80cffb0 2021-04-09T18:16:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f979ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f97a80d7810 0x7f97a80d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54308/0 (socket says 172.21.15.32:54308) 2021-04-09T18:16:38.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f979ffff700 1 -- 172.21.15.32:0/3036052372 learned_addr learned my addr 172.21.15.32:0/3036052372 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f97a7fff700 1 --2- 172.21.15.32:0/3036052372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a80cffb0 0x7f97a80d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f97a77fe700 1 -- 172.21.15.32:0/3036052372 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458557430 0 0) 0x7f9790000ce0 con 0x7f97a80d7810 2021-04-09T18:16:38.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f97a77fe700 1 -- 172.21.15.32:0/3036052372 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9794002fe0 con 0x7f97a80d7810 2021-04-09T18:16:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.589 7f97a77fe700 1 -- 172.21.15.32:0/3036052372 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615161131 0 0) 0x7f97a0000f90 con 0x7f97a80d6c90 2021-04-09T18:16:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97a77fe700 1 -- 172.21.15.32:0/3036052372 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9794003210 con 0x7f97a80d6c90 2021-04-09T18:16:38.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97a7fff700 1 -- 172.21.15.32:0/3036052372 >> v1:172.21.15.86:6789/0 conn(0x7f97a80d6c90 legacy=0x7f97a810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97a7fff700 1 -- 172.21.15.32:0/3036052372 >> v1:172.21.15.32:6789/0 conn(0x7f97a80d7810 legacy=0x7f97a80d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:38.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97a77fe700 1 -- 172.21.15.32:0/3036052372 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1357115851 0 0) 0x7f97900010e0 con 0x7f97a80d7810 2021-04-09T18:16:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97a7fff700 1 -- 172.21.15.32:0/3036052372 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80d8590 con 0x7f97a80cffb0 2021-04-09T18:16:38.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97a7fff700 1 --2- 172.21.15.32:0/3036052372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a80cffb0 0x7f97a80d2450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9798006650 tx=0x7f9798009a00).ready entity=mon.2 client_cookie=53367a6cee668676 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97a77fe700 1 -- 172.21.15.32:0/3036052372 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9798009df0 con 0x7f97a80cffb0 2021-04-09T18:16:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97a77fe700 1 -- 172.21.15.32:0/3036052372 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97980009b0 con 0x7f97a80cffb0 2021-04-09T18:16:38.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97a77fe700 1 -- 172.21.15.32:0/3036052372 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f979800c2b0 con 0x7f97a80cffb0 2021-04-09T18:16:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97ae6c3700 1 -- 172.21.15.32:0/3036052372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a80cffb0 msgr2=0x7f97a80d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.590 7f97ae6c3700 1 --2- 172.21.15.32:0/3036052372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a80cffb0 0x7f97a80d2450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9798006650 tx=0x7f9798009a00).stop 2021-04-09T18:16:38.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.591 7f97ae6c3700 1 -- 172.21.15.32:0/3036052372 shutdown_connections 2021-04-09T18:16:38.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.591 7f97ae6c3700 1 --2- 172.21.15.32:0/3036052372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a80cffb0 0x7f97a80d2450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.591 7f97ae6c3700 1 -- 172.21.15.32:0/3036052372 >> 172.21.15.32:0/3036052372 conn(0x7f97a80ce5d0 msgr2=0x7f97a80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.591 7f97ae6c3700 1 -- 172.21.15.32:0/3036052372 shutdown_connections 2021-04-09T18:16:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.591 7f97ae6c3700 1 -- 172.21.15.32:0/3036052372 wait complete. 2021-04-09T18:16:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.611 7f97ae6c3700 1 Processor -- start 2021-04-09T18:16:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.611 7f97ae6c3700 1 -- start start 2021-04-09T18:16:38.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97ae6c3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97a8188cf0 0x7f97a818d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97ae6c3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a81890f0 0x7f97a818dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97ae6c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a818c180 0x7f97a818c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:38.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97ae6c3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a818d0b0 con 0x7f97a818c180 2021-04-09T18:16:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97ae6c3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a818cba0 con 0x7f97a8188cf0 2021-04-09T18:16:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97ae6c3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a818cd80 con 0x7f97a81890f0 2021-04-09T18:16:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97acce0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a818c180 0x7f97a818c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:38.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97acce0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a818c180 0x7f97a818c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44726/0 (socket says 172.21.15.32:44726) 2021-04-09T18:16:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97acce0700 1 -- 172.21.15.32:0/1029157723 learned_addr learned my addr 172.21.15.32:0/1029157723 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:38.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97a7fff700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97a8188cf0 0x7f97a818d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:38.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f979ffff700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a81890f0 0x7f97a818dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:38.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97acce0700 1 -- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a81890f0 msgr2=0x7f97a818dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:38.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97acce0700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a81890f0 0x7f97a818dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97acce0700 1 -- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97a8188cf0 msgr2=0x7f97a818d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:38.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97acce0700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97a8188cf0 0x7f97a818d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97acce0700 1 -- 172.21.15.32:0/1029157723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a8183d10 con 0x7f97a818c180 2021-04-09T18:16:38.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f979ffff700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a81890f0 0x7f97a818dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:38.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97acce0700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a818c180 0x7f97a818c580 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f97a00099a0 tx=0x7f97a000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:38.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97a57fa700 1 -- 172.21.15.32:0/1029157723 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97a0003020 con 0x7f97a818c180 2021-04-09T18:16:38.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97a57fa700 1 -- 172.21.15.32:0/1029157723 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a00058a0 con 0x7f97a818c180 2021-04-09T18:16:38.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a8183d10 con 0x7f97a818c180 2021-04-09T18:16:38.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97a57fa700 1 -- 172.21.15.32:0/1029157723 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97a0005d80 con 0x7f97a818c180 2021-04-09T18:16:38.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.612 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a8183d10 con 0x7f97a818c180 2021-04-09T18:16:38.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.613 7f97a57fa700 1 -- 172.21.15.32:0/1029157723 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f97a0005a80 con 0x7f97a818c180 2021-04-09T18:16:38.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.613 7f97a57fa700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f978c018930 0x7f978c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:38.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.613 7f97a57fa700 1 -- 172.21.15.32:0/1029157723 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f97a0032a70 con 0x7f97a818c180 2021-04-09T18:16:38.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.613 7f97a7fff700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f978c018930 0x7f978c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:38.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.614 7f97a7fff700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f978c018930 0x7f978c01ae10 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.633 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a8182e20 con 0x7f97a818c180 2021-04-09T18:16:38.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.636 7f97a57fa700 1 -- 172.21.15.32:0/1029157723 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f97a000be30 con 0x7f97a818c180 2021-04-09T18:16:38.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.847 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f97a8182e20 con 0x7f97a818c180 2021-04-09T18:16:38.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.849 7f97a57fa700 1 -- 172.21.15.32:0/1029157723 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7f97a00058a0 con 0x7f97a818c180 2021-04-09T18:16:38.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f978c018930 msgr2=0x7f978c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f978c018930 0x7f978c01ae10 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a818c180 msgr2=0x7f97a818c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:38.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a818c180 0x7f97a818c580 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f97a00099a0 tx=0x7f97a000bcf0).stop 2021-04-09T18:16:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 shutdown_connections 2021-04-09T18:16:38.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f978c018930 0x7f978c01ae10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97a818c180 0x7f97a818c580 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97a81890f0 0x7f97a818dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 --2- 172.21.15.32:0/1029157723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97a8188cf0 0x7f97a818d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:38.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 >> 172.21.15.32:0/1029157723 conn(0x7f97a81a36a0 msgr2=0x7f97a8185120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 shutdown_connections 2021-04-09T18:16:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:38.852 7f97ae6c3700 1 -- 172.21.15.32:0/1029157723 wait complete. 2021-04-09T18:16:38.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2021-04-09T18:16:39.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.033 7fc0604e5700 1 Processor -- start 2021-04-09T18:16:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.033 7fc0604e5700 1 -- start start 2021-04-09T18:16:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.033 7fc0604e5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc05810d010 0x7fc0580d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.033 7fc0604e5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05810d480 con 0x7fc0580d8490 2021-04-09T18:16:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.033 7fc0604e5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05810e1f0 con 0x7fc0580d1da0 2021-04-09T18:16:39.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc0604e5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0580d8d30 con 0x7fc05810d010 2021-04-09T18:16:39.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05eb02700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc0580d8490 0x7fc05810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54322/0 (socket says 172.21.15.32:54322) 2021-04-09T18:16:39.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05eb02700 1 -- 172.21.15.32:0/3730789020 learned_addr learned my addr 172.21.15.32:0/3730789020 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:39.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05e301700 1 --2- 172.21.15.32:0/3730789020 >> v2:172.21.15.86:3300/0 conn(0x7fc05810d010 0x7fc0580d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140454961 0 0) 0x7fc050000d30 con 0x7fc0580d8490 2021-04-09T18:16:39.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc044002fe0 con 0x7fc0580d8490 2021-04-09T18:16:39.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3763655671 0 0) 0x7fc050001270 con 0x7fc0580d8490 2021-04-09T18:16:39.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 >> v2:172.21.15.86:3300/0 conn(0x7fc05810d010 msgr2=0x7fc0580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:39.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 --2- 172.21.15.32:0/3730789020 >> v2:172.21.15.86:3300/0 conn(0x7fc05810d010 0x7fc0580d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 >> v1:172.21.15.86:6789/0 conn(0x7fc0580d1da0 legacy=0x7fc0580d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0580d9220 con 0x7fc0580d8490 2021-04-09T18:16:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fc050001980 con 0x7fc0580d8490 2021-04-09T18:16:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 >> v1:172.21.15.32:6789/0 conn(0x7fc0580d8490 legacy=0x7fc05810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc044003a40 0x7fc044005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc044006790 0x7fc044008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc044009590 0x7fc04400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04400c300 con 0x7fc044003a40 2021-04-09T18:16:39.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04400c520 con 0x7fc044009590 2021-04-09T18:16:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc044050750 con 0x7fc044006790 2021-04-09T18:16:39.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05e301700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc044003a40 0x7fc044005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.034 7fc05eb02700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc044009590 0x7fc04400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05db00700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc044006790 0x7fc044008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05e301700 1 -- 172.21.15.32:0/3730789020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc044006790 msgr2=0x7fc044008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:39.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05e301700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc044006790 0x7fc044008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05e301700 1 -- 172.21.15.32:0/3730789020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc044009590 msgr2=0x7fc04400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:39.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05e301700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc044009590 0x7fc04400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05e301700 1 -- 172.21.15.32:0/3730789020 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440509b0 con 0x7fc044003a40 2021-04-09T18:16:39.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05eb02700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc044009590 0x7fc04400bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:39.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05eb02700 1 -- 172.21.15.32:0/3730789020 reap_dead start 2021-04-09T18:16:39.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05e301700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc044003a40 0x7fc044005ef0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fc05400c870 tx=0x7fc05400cc10).ready entity=mon.0 client_cookie=b4f95e87eb77b223 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:39.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0540090d0 con 0x7fc044003a40 2021-04-09T18:16:39.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc054009450 con 0x7fc044003a40 2021-04-09T18:16:39.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc05d2ff700 1 -- 172.21.15.32:0/3730789020 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0540009b0 con 0x7fc044003a40 2021-04-09T18:16:39.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc0604e5700 1 -- 172.21.15.32:0/3730789020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc044003a40 msgr2=0x7fc044005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.035 7fc0604e5700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc044003a40 0x7fc044005ef0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fc05400c870 tx=0x7fc05400cc10).stop 2021-04-09T18:16:39.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.036 7fc0604e5700 1 -- 172.21.15.32:0/3730789020 shutdown_connections 2021-04-09T18:16:39.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.036 7fc0604e5700 1 --2- 172.21.15.32:0/3730789020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc044003a40 0x7fc044005ef0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.036 7fc0604e5700 1 -- 172.21.15.32:0/3730789020 >> 172.21.15.32:0/3730789020 conn(0x7fc0580ce5f0 msgr2=0x7fc0580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:39.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.036 7fc0604e5700 1 -- 172.21.15.32:0/3730789020 shutdown_connections 2021-04-09T18:16:39.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.036 7fc0604e5700 1 -- 172.21.15.32:0/3730789020 wait complete. 2021-04-09T18:16:39.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.056 7fc0604e5700 1 Processor -- start 2021-04-09T18:16:39.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.056 7fc0604e5700 1 -- start start 2021-04-09T18:16:39.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc0604e5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580b7620 0x7fc058184c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc0604e5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581852b0 0x7fc0581a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc0604e5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0581a6840 0x7fc0581a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc0604e5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc058184710 con 0x7fc0580b7620 2021-04-09T18:16:39.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc0604e5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0581856b0 con 0x7fc0581a6840 2021-04-09T18:16:39.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc0604e5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc058182f80 con 0x7fc0581852b0 2021-04-09T18:16:39.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05e301700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580b7620 0x7fc058184c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05e301700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580b7620 0x7fc058184c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44742/0 (socket says 172.21.15.32:44742) 2021-04-09T18:16:39.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05e301700 1 -- 172.21.15.32:0/4220149674 learned_addr learned my addr 172.21.15.32:0/4220149674 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05eb02700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0581a6840 0x7fc0581a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05db00700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581852b0 0x7fc0581a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05e301700 1 -- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581852b0 msgr2=0x7fc0581a6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05e301700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581852b0 0x7fc0581a6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05e301700 1 -- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0581a6840 msgr2=0x7fc0581a6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05e301700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0581a6840 0x7fc0581a6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05e301700 1 -- 172.21.15.32:0/4220149674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0581831a0 con 0x7fc0580b7620 2021-04-09T18:16:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05db00700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581852b0 0x7fc0581a6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:16:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc05e301700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580b7620 0x7fc058184c90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fc0540064b0 tx=0x7fc05400b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:39.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc04affd700 1 -- 172.21.15.32:0/4220149674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0540064f0 con 0x7fc0580b7620 2021-04-09T18:16:39.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc04affd700 1 -- 172.21.15.32:0/4220149674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc054000940 con 0x7fc0580b7620 2021-04-09T18:16:39.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0581831a0 con 0x7fc0580b7620 2021-04-09T18:16:39.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc04affd700 1 -- 172.21.15.32:0/4220149674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc05400c770 con 0x7fc0580b7620 2021-04-09T18:16:39.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.057 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0581831a0 con 0x7fc0580b7620 2021-04-09T18:16:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.058 7fc04affd700 1 -- 172.21.15.32:0/4220149674 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc054000b20 con 0x7fc0580b7620 2021-04-09T18:16:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.058 7fc04affd700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc040018930 0x7fc04001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.058 7fc04affd700 1 -- 172.21.15.32:0/4220149674 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fc0540326b0 con 0x7fc0580b7620 2021-04-09T18:16:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.058 7fc05db00700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc040018930 0x7fc04001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.059 7fc05db00700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc040018930 0x7fc04001adc0 crc :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:39.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.078 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05810d480 con 0x7fc0580b7620 2021-04-09T18:16:39.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.081 7fc04affd700 1 -- 172.21.15.32:0/4220149674 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc054006ab0 con 0x7fc0580b7620 2021-04-09T18:16:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.268 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7fc05810d480 con 0x7fc0580b7620 2021-04-09T18:16:39.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.378 7fc04affd700 1 -- 172.21.15.32:0/4220149674 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v186) v1 ==== 140+0+0 (secure 0 0 0) 0x7fc054000940 con 0x7fc0580b7620 2021-04-09T18:16:39.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.383 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc040018930 msgr2=0x7fc04001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.383 7fc0604e5700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc040018930 0x7fc04001adc0 crc :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.383 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580b7620 msgr2=0x7fc058184c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.383 7fc0604e5700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580b7620 0x7fc058184c90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fc0540064b0 tx=0x7fc05400b300).stop 2021-04-09T18:16:39.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.384 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 shutdown_connections 2021-04-09T18:16:39.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.384 7fc0604e5700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc040018930 0x7fc04001adc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.384 7fc0604e5700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0581a6840 0x7fc0581a6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.384 7fc0604e5700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581852b0 0x7fc0581a6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.384 7fc0604e5700 1 --2- 172.21.15.32:0/4220149674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580b7620 0x7fc058184c90 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.384 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 >> 172.21.15.32:0/4220149674 conn(0x7fc05819f830 msgr2=0x7fc05819eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:39.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.384 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 shutdown_connections 2021-04-09T18:16:39.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.384 7fc0604e5700 1 -- 172.21.15.32:0/4220149674 wait complete. 2021-04-09T18:16:39.435 INFO:tasks.cephfs_test_runner:test_down_health (tasks.cephfs.test_failover.TestClusterResize) ... ok 2021-04-09T18:16:39.436 INFO:tasks.cephfs_test_runner:Starting test: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2021-04-09T18:16:39.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2021-04-09T18:16:39.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.563 7f250de4c700 1 Processor -- start 2021-04-09T18:16:39.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.563 7f250de4c700 1 -- start start 2021-04-09T18:16:39.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f250de4c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f25080d7ca0 0x7f25080d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f250de4c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250810d480 con 0x7f250810da00 2021-04-09T18:16:39.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f250de4c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25080d86c0 con 0x7f250810d010 2021-04-09T18:16:39.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f250de4c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25080d9220 con 0x7f25080d7ca0 2021-04-09T18:16:39.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f2507fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f250810da00 0x7f250810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54340/0 (socket says 172.21.15.32:54340) 2021-04-09T18:16:39.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f2507fff700 1 -- 172.21.15.32:0/1547446912 learned_addr learned my addr 172.21.15.32:0/1547446912 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:39.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f2506ffd700 1 --2- 172.21.15.32:0/1547446912 >> v2:172.21.15.86:3300/0 conn(0x7f25080d7ca0 0x7f25080d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f25067fc700 1 -- 172.21.15.32:0/1547446912 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540584213 0 0) 0x7f2500000d30 con 0x7f250810da00 2021-04-09T18:16:39.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f25067fc700 1 -- 172.21.15.32:0/1547446912 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f0002fe0 con 0x7f250810da00 2021-04-09T18:16:39.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f2506ffd700 1 -- 172.21.15.32:0/1547446912 >> v1:172.21.15.86:6789/0 conn(0x7f250810d010 legacy=0x7f25080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f2506ffd700 1 -- 172.21.15.32:0/1547446912 >> v1:172.21.15.32:6789/0 conn(0x7f250810da00 legacy=0x7f250810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f25067fc700 1 -- 172.21.15.32:0/1547446912 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746581695 0 0) 0x7f24f8000c60 con 0x7f250810d010 2021-04-09T18:16:39.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f2506ffd700 1 -- 172.21.15.32:0/1547446912 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080d1f70 con 0x7f25080d7ca0 2021-04-09T18:16:39.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f2506ffd700 1 --2- 172.21.15.32:0/1547446912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25080d7ca0 0x7f25080d80a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f24fc006650 tx=0x7f24fc009a80).ready entity=mon.1 client_cookie=46b7557741bbe888 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:39.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f25067fc700 1 -- 172.21.15.32:0/1547446912 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24fc0009b0 con 0x7f25080d7ca0 2021-04-09T18:16:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f25067fc700 1 -- 172.21.15.32:0/1547446912 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc00c0c0 con 0x7f25080d7ca0 2021-04-09T18:16:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f25067fc700 1 -- 172.21.15.32:0/1547446912 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24fc00c510 con 0x7f25080d7ca0 2021-04-09T18:16:39.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f250de4c700 1 -- 172.21.15.32:0/1547446912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25080d7ca0 msgr2=0x7f25080d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.564 7f250de4c700 1 --2- 172.21.15.32:0/1547446912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25080d7ca0 0x7f25080d80a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f24fc006650 tx=0x7f24fc009a80).stop 2021-04-09T18:16:39.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.565 7f250de4c700 1 -- 172.21.15.32:0/1547446912 shutdown_connections 2021-04-09T18:16:39.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.565 7f250de4c700 1 --2- 172.21.15.32:0/1547446912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25080d7ca0 0x7f25080d80a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.565 7f250de4c700 1 -- 172.21.15.32:0/1547446912 >> 172.21.15.32:0/1547446912 conn(0x7f25080ce5f0 msgr2=0x7f25080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.565 7f250de4c700 1 -- 172.21.15.32:0/1547446912 shutdown_connections 2021-04-09T18:16:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.565 7f250de4c700 1 -- 172.21.15.32:0/1547446912 wait complete. 2021-04-09T18:16:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.585 7f250de4c700 1 Processor -- start 2021-04-09T18:16:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f250de4c700 1 -- start start 2021-04-09T18:16:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f250de4c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25080a7f50 0x7f2508180be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f250de4c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2508181200 0x7f250808ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f250de4c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f250808b3b0 0x7f25081a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f250de4c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250808a890 con 0x7f250808b3b0 2021-04-09T18:16:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f250de4c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2508181640 con 0x7f2508181200 2021-04-09T18:16:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f250de4c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25081819f0 con 0x7f25080a7f50 2021-04-09T18:16:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f2507fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f250808b3b0 0x7f25081a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f2507fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f250808b3b0 0x7f25081a32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44760/0 (socket says 172.21.15.32:44760) 2021-04-09T18:16:39.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.586 7f2507fff700 1 -- 172.21.15.32:0/498714375 learned_addr learned my addr 172.21.15.32:0/498714375 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:39.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f2506ffd700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2508181200 0x7f250808ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f25077fe700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25080a7f50 0x7f2508180be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f2507fff700 1 -- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25080a7f50 msgr2=0x7f2508180be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f2507fff700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25080a7f50 0x7f2508180be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f2507fff700 1 -- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2508181200 msgr2=0x7f250808ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:39.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f2507fff700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2508181200 0x7f250808ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:39.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f2507fff700 1 -- 172.21.15.32:0/498714375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250808b7b0 con 0x7f250808b3b0 2021-04-09T18:16:39.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f2506ffd700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2508181200 0x7f250808ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:39.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f25077fe700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25080a7f50 0x7f2508180be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:39.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f2507fff700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f250808b3b0 0x7f25081a32e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f2500009980 tx=0x7f250000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:39.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f24effff700 1 -- 172.21.15.32:0/498714375 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250000be10 con 0x7f250808b3b0 2021-04-09T18:16:39.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f24effff700 1 -- 172.21.15.32:0/498714375 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2500006030 con 0x7f250808b3b0 2021-04-09T18:16:39.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f250de4c700 1 -- 172.21.15.32:0/498714375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f250808b7b0 con 0x7f250808b3b0 2021-04-09T18:16:39.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f24effff700 1 -- 172.21.15.32:0/498714375 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25000064c0 con 0x7f250808b3b0 2021-04-09T18:16:39.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.587 7f250de4c700 1 -- 172.21.15.32:0/498714375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250808b7b0 con 0x7f250808b3b0 2021-04-09T18:16:39.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.588 7f24effff700 1 -- 172.21.15.32:0/498714375 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2500006210 con 0x7f250808b3b0 2021-04-09T18:16:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.588 7f24effff700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24e8018930 0x7f24e801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:39.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.588 7f24effff700 1 -- 172.21.15.32:0/498714375 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f2500032a40 con 0x7f250808b3b0 2021-04-09T18:16:39.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.588 7f25077fe700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24e8018930 0x7f24e801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:39.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.588 7f25077fe700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24e8018930 0x7f24e801ae10 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:39.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.608 7f250de4c700 1 -- 172.21.15.32:0/498714375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25081b7070 con 0x7f250808b3b0 2021-04-09T18:16:39.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.611 7f24effff700 1 -- 172.21.15.32:0/498714375 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2500000f90 con 0x7f250808b3b0 2021-04-09T18:16:39.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:39.797 7f250de4c700 1 -- 172.21.15.32:0/498714375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7f25081b7070 con 0x7f250808b3b0 2021-04-09T18:16:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.380 7f24effff700 1 -- 172.21.15.32:0/498714375 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v187) v1 ==== 142+0+0 (secure 0 0 0) 0x7f2500006030 con 0x7f250808b3b0 2021-04-09T18:16:40.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.412 7f250de4c700 1 -- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24e8018930 msgr2=0x7f24e801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:40.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.412 7f250de4c700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24e8018930 0x7f24e801ae10 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.412 7f250de4c700 1 -- 172.21.15.32:0/498714375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f250808b3b0 msgr2=0x7f25081a32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:40.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.412 7f250de4c700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f250808b3b0 0x7f25081a32e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f2500009980 tx=0x7f250000bcd0).stop 2021-04-09T18:16:40.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.412 7f250de4c700 1 -- 172.21.15.32:0/498714375 shutdown_connections 2021-04-09T18:16:40.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.412 7f250de4c700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24e8018930 0x7f24e801ae10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.412 7f250de4c700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f250808b3b0 0x7f25081a32e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.412 7f250de4c700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2508181200 0x7f250808ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.412 7f250de4c700 1 --2- 172.21.15.32:0/498714375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25080a7f50 0x7f2508180be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.413 7f250de4c700 1 -- 172.21.15.32:0/498714375 >> 172.21.15.32:0/498714375 conn(0x7f25081aa2b0 msgr2=0x7f25081a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:40.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.413 7f250de4c700 1 -- 172.21.15.32:0/498714375 shutdown_connections 2021-04-09T18:16:40.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.413 7f250de4c700 1 -- 172.21.15.32:0/498714375 wait complete. 2021-04-09T18:16:40.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:16:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.595 7fe9ce09c700 1 Processor -- start 2021-04-09T18:16:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.595 7fe9ce09c700 1 -- start start 2021-04-09T18:16:40.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9ce09c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe9c80cffc0 0x7fe9c80d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:40.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9ce09c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c80d72a0 con 0x7fe9c80d7820 2021-04-09T18:16:40.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9ce09c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c80d8410 con 0x7fe9c80d6ca0 2021-04-09T18:16:40.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9ce09c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c80d70a0 con 0x7fe9c80cffc0 2021-04-09T18:16:40.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c6ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe9c80d7820 0x7fe9c80d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54356/0 (socket says 172.21.15.32:54356) 2021-04-09T18:16:40.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c6ffd700 1 -- 172.21.15.32:0/2325552446 learned_addr learned my addr 172.21.15.32:0/2325552446 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:40.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c77fe700 1 --2- 172.21.15.32:0/2325552446 >> v2:172.21.15.86:3300/0 conn(0x7fe9c80cffc0 0x7fe9c80d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:40.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051432814 0 0) 0x7fe9bc0009b0 con 0x7fe9c80d7820 2021-04-09T18:16:40.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0002fe0 con 0x7fe9c80d7820 2021-04-09T18:16:40.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006851691 0 0) 0x7fe9c0001400 con 0x7fe9c80d6ca0 2021-04-09T18:16:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0003210 con 0x7fe9c80d6ca0 2021-04-09T18:16:40.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3439301950 0 0) 0x7fe9bc000e20 con 0x7fe9c80d7820 2021-04-09T18:16:40.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 >> v2:172.21.15.86:3300/0 conn(0x7fe9c80cffc0 msgr2=0x7fe9c80d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 --2- 172.21.15.32:0/2325552446 >> v2:172.21.15.86:3300/0 conn(0x7fe9c80cffc0 0x7fe9c80d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 >> v1:172.21.15.86:6789/0 conn(0x7fe9c80d6ca0 legacy=0x7fe9c810cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:40.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80d9200 con 0x7fe9c80d7820 2021-04-09T18:16:40.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fe9bc001530 con 0x7fe9c80d7820 2021-04-09T18:16:40.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.596 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 >> v1:172.21.15.32:6789/0 conn(0x7fe9c80d7820 legacy=0x7fe9c80d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:40.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c67fc700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b0003b90 0x7fe9b0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:40.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c67fc700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9b0006820 0x7fe9b0008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:40.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c67fc700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9b0009640 0x7fe9b000bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:40.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00503c0 con 0x7fe9b0009640 2021-04-09T18:16:40.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00505a0 con 0x7fe9b0003b90 2021-04-09T18:16:40.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00507c0 con 0x7fe9b0006820 2021-04-09T18:16:40.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c77fe700 1 --2- 172.21.15.32:0/2325552446 >> v2:172.21.15.86:3300/0 conn(0x7fe9c80cffc0 0x7fe9c80d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:40.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c7fff700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9b0009640 0x7fe9b000bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c6ffd700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9b0006820 0x7fe9b0008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:40.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c77fe700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b0003b90 0x7fe9b0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:40.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c7fff700 1 -- 172.21.15.32:0/2325552446 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9b0006820 msgr2=0x7fe9b0008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:40.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c7fff700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9b0006820 0x7fe9b0008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c7fff700 1 -- 172.21.15.32:0/2325552446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b0003b90 msgr2=0x7fe9b0005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:40.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c7fff700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b0003b90 0x7fe9b0005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c7fff700 1 -- 172.21.15.32:0/2325552446 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b0050a60 con 0x7fe9b0009640 2021-04-09T18:16:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c6ffd700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9b0006820 0x7fe9b0008d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c7fff700 1 -- 172.21.15.32:0/2325552446 reap_dead start 2021-04-09T18:16:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c7fff700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9b0009640 0x7fe9b000bb60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fe9c00013c0 tx=0x7fe9c0009e60).ready entity=mon.0 client_cookie=9f2e940e593ca684 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9c0001400 con 0x7fe9b0009640 2021-04-09T18:16:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c0003210 con 0x7fe9b0009640 2021-04-09T18:16:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9c67fc700 1 -- 172.21.15.32:0/2325552446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9c0001780 con 0x7fe9b0009640 2021-04-09T18:16:40.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9ce09c700 1 -- 172.21.15.32:0/2325552446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9b0009640 msgr2=0x7fe9b000bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:40.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.597 7fe9ce09c700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9b0009640 0x7fe9b000bb60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fe9c00013c0 tx=0x7fe9c0009e60).stop 2021-04-09T18:16:40.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.598 7fe9ce09c700 1 -- 172.21.15.32:0/2325552446 shutdown_connections 2021-04-09T18:16:40.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.598 7fe9ce09c700 1 --2- 172.21.15.32:0/2325552446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9b0009640 0x7fe9b000bb60 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.598 7fe9ce09c700 1 -- 172.21.15.32:0/2325552446 >> 172.21.15.32:0/2325552446 conn(0x7fe9c80ce5e0 msgr2=0x7fe9c80cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:40.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.598 7fe9ce09c700 1 -- 172.21.15.32:0/2325552446 shutdown_connections 2021-04-09T18:16:40.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.598 7fe9ce09c700 1 -- 172.21.15.32:0/2325552446 wait complete. 2021-04-09T18:16:40.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.618 7fe9ce09c700 1 Processor -- start 2021-04-09T18:16:40.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.618 7fe9ce09c700 1 -- start start 2021-04-09T18:16:40.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9ce09c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9c819d160 0x7fe9c809adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:40.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9ce09c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c80a3870 0x7fe9c80989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:40.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9ce09c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9c8098ff0 0x7fe9c80a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:40.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9ce09c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80a32f0 con 0x7fe9c80a3870 2021-04-09T18:16:40.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9ce09c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80993f0 con 0x7fe9c819d160 2021-04-09T18:16:40.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9ce09c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80a3c70 con 0x7fe9c8098ff0 2021-04-09T18:16:40.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c80a3870 0x7fe9c80989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:40.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c80a3870 0x7fe9c80989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44776/0 (socket says 172.21.15.32:44776) 2021-04-09T18:16:40.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c6ffd700 1 -- 172.21.15.32:0/2092640451 learned_addr learned my addr 172.21.15.32:0/2092640451 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:40.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c77fe700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9c819d160 0x7fe9c809adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:40.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c7fff700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9c8098ff0 0x7fe9c80a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:40.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c6ffd700 1 -- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9c8098ff0 msgr2=0x7fe9c80a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:40.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c6ffd700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9c8098ff0 0x7fe9c80a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c6ffd700 1 -- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9c819d160 msgr2=0x7fe9c809adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:40.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c6ffd700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9c819d160 0x7fe9c809adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c6ffd700 1 -- 172.21.15.32:0/2092640451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80a2450 con 0x7fe9c80a3870 2021-04-09T18:16:40.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.619 7fe9c6ffd700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c80a3870 0x7fe9c80989d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fe9bc0077b0 tx=0x7fe9bc007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:40.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.620 7fe9affff700 1 -- 172.21.15.32:0/2092640451 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9bc004350 con 0x7fe9c80a3870 2021-04-09T18:16:40.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.620 7fe9affff700 1 -- 172.21.15.32:0/2092640451 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc004690 con 0x7fe9c80a3870 2021-04-09T18:16:40.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.620 7fe9affff700 1 -- 172.21.15.32:0/2092640451 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9bc004b70 con 0x7fe9c80a3870 2021-04-09T18:16:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.620 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c80a2450 con 0x7fe9c80a3870 2021-04-09T18:16:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.620 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c80a2450 con 0x7fe9c80a3870 2021-04-09T18:16:40.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.620 7fe9affff700 1 -- 172.21.15.32:0/2092640451 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe9bc001750 con 0x7fe9c80a3870 2021-04-09T18:16:40.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.620 7fe9affff700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe9a80188c0 0x7fe9a801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:40.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.621 7fe9affff700 1 -- 172.21.15.32:0/2092640451 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fe9bc0346e0 con 0x7fe9c80a3870 2021-04-09T18:16:40.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.621 7fe9c77fe700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe9a80188c0 0x7fe9a801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:40.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.621 7fe9c77fe700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe9a80188c0 0x7fe9a801adc0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:40.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.640 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c8093e30 con 0x7fe9c80a3870 2021-04-09T18:16:40.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.643 7fe9affff700 1 -- 172.21.15.32:0/2092640451 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe9bc004140 con 0x7fe9c80a3870 2021-04-09T18:16:40.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.852 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe9c80d72a0 con 0x7fe9c80a3870 2021-04-09T18:16:40.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.857 7fe9affff700 1 -- 172.21.15.32:0/2092640451 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe9bc001750 con 0x7fe9c80a3870 2021-04-09T18:16:40.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe9a80188c0 msgr2=0x7fe9a801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:40.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe9a80188c0 0x7fe9a801adc0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c80a3870 msgr2=0x7fe9c80989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c80a3870 0x7fe9c80989d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fe9bc0077b0 tx=0x7fe9bc007b50).stop 2021-04-09T18:16:40.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 shutdown_connections 2021-04-09T18:16:40.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe9a80188c0 0x7fe9a801adc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9c80a3870 0x7fe9c80989d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9c8098ff0 0x7fe9c80a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 --2- 172.21.15.32:0/2092640451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9c819d160 0x7fe9c809adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:40.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 >> 172.21.15.32:0/2092640451 conn(0x7fe9c809fa00 msgr2=0x7fe9c80c3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 shutdown_connections 2021-04-09T18:16:40.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:40.862 7fe9ce09c700 1 -- 172.21.15.32:0/2092640451 wait complete. 2021-04-09T18:16:40.928 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:16:40.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:16:41.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:16:41.064 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:16:41.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:41.066 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:16:41.067 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:16:41.116 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:16:41.116 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:16:41.117 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:41.118 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:16:41.122 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:16:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:16:41.166 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:16:41.167 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:16:41.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:16:41.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.290 7fcde139f700 1 Processor -- start 2021-04-09T18:16:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.290 7fcde139f700 1 -- start start 2021-04-09T18:16:41.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.291 7fcde139f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc0d7ca0 0x7fcddc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:41.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.291 7fcde139f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcddc10da00 0x7fcddc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:41.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.291 7fcde139f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcddc10d480 con 0x7fcddc10d010 2021-04-09T18:16:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.291 7fcde139f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0d9220 con 0x7fcddc10da00 2021-04-09T18:16:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.291 7fcde139f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0d86c0 con 0x7fcddc0d7ca0 2021-04-09T18:16:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.291 7fcdda87c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc0d7ca0 0x7fcddc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:41.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.291 7fcddb87e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcddc10da00 0x7fcddc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:41.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.291 7fcddb07d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcddc10d010 0x7fcddc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52298/0 (socket says 172.21.15.32:52298) 2021-04-09T18:16:41.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.291 7fcddb07d700 1 -- 172.21.15.32:0/27619966 learned_addr learned my addr 172.21.15.32:0/27619966 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:41.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcdda07b700 1 -- 172.21.15.32:0/27619966 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186647285 0 0) 0x7fcdc4001510 con 0x7fcddc10d010 2021-04-09T18:16:41.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcdda87c700 1 -- 172.21.15.32:0/27619966 >> v2:172.21.15.86:3300/0 conn(0x7fcddc10da00 msgr2=0x7fcddc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:41.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcdda87c700 1 --2- 172.21.15.32:0/27619966 >> v2:172.21.15.86:3300/0 conn(0x7fcddc10da00 0x7fcddc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:41.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcdda87c700 1 -- 172.21.15.32:0/27619966 >> v1:172.21.15.86:6789/0 conn(0x7fcddc10d010 legacy=0x7fcddc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:41.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcdda87c700 1 -- 172.21.15.32:0/27619966 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc0d1f70 con 0x7fcddc0d7ca0 2021-04-09T18:16:41.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcddb87e700 1 --2- 172.21.15.32:0/27619966 >> v2:172.21.15.86:3300/0 conn(0x7fcddc10da00 0x7fcddc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:41.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcdda87c700 1 --2- 172.21.15.32:0/27619966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc0d7ca0 0x7fcddc0d80a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcdcc006650 tx=0x7fcdcc009a80).ready entity=mon.2 client_cookie=8155d44d4e751577 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:41.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcdda07b700 1 -- 172.21.15.32:0/27619966 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdcc0009b0 con 0x7fcddc0d7ca0 2021-04-09T18:16:41.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcdda07b700 1 -- 172.21.15.32:0/27619966 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc00c0c0 con 0x7fcddc0d7ca0 2021-04-09T18:16:41.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcdda07b700 1 -- 172.21.15.32:0/27619966 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdcc0009b0 con 0x7fcddc0d7ca0 2021-04-09T18:16:41.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcde139f700 1 -- 172.21.15.32:0/27619966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc0d7ca0 msgr2=0x7fcddc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:41.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.292 7fcde139f700 1 --2- 172.21.15.32:0/27619966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc0d7ca0 0x7fcddc0d80a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcdcc006650 tx=0x7fcdcc009a80).stop 2021-04-09T18:16:41.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.293 7fcde139f700 1 -- 172.21.15.32:0/27619966 shutdown_connections 2021-04-09T18:16:41.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.293 7fcde139f700 1 --2- 172.21.15.32:0/27619966 >> v2:172.21.15.86:3300/0 conn(0x7fcddc10da00 0x7fcddc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:41.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.293 7fcde139f700 1 --2- 172.21.15.32:0/27619966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc0d7ca0 0x7fcddc0d80a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.293 7fcde139f700 1 -- 172.21.15.32:0/27619966 >> 172.21.15.32:0/27619966 conn(0x7fcddc0ce5f0 msgr2=0x7fcddc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:41.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.293 7fcde139f700 1 -- 172.21.15.32:0/27619966 shutdown_connections 2021-04-09T18:16:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.293 7fcde139f700 1 -- 172.21.15.32:0/27619966 wait complete. 2021-04-09T18:16:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.313 7fcde139f700 1 Processor -- start 2021-04-09T18:16:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.313 7fcde139f700 1 -- start start 2021-04-09T18:16:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcde139f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcddc185060 0x7fcddc185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcde139f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcddc186c00 0x7fcddc187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:41.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcde139f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc191360 0x7fcddc191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:41.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcde139f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc1863a0 con 0x7fcddc186c00 2021-04-09T18:16:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcde139f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc185a60 con 0x7fcddc185060 2021-04-09T18:16:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcde139f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc187620 con 0x7fcddc191360 2021-04-09T18:16:41.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcdda87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcddc186c00 0x7fcddc187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:41.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcddb07d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcddc185060 0x7fcddc185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:41.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcddb87e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc191360 0x7fcddc191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:41.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcdda87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcddc186c00 0x7fcddc187000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44792/0 (socket says 172.21.15.32:44792) 2021-04-09T18:16:41.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcdda87c700 1 -- 172.21.15.32:0/3715085192 learned_addr learned my addr 172.21.15.32:0/3715085192 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:41.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcddb87e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc191360 0x7fcddc191760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:52016/0 (socket says 172.21.15.32:52016) 2021-04-09T18:16:41.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcddb07d700 1 -- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc191360 msgr2=0x7fcddc191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:41.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcddb07d700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc191360 0x7fcddc191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:41.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcddb07d700 1 -- 172.21.15.32:0/3715085192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcddc186c00 msgr2=0x7fcddc187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:41.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcddb07d700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcddc186c00 0x7fcddc187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:41.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcddb07d700 1 -- 172.21.15.32:0/3715085192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc185460 con 0x7fcddc185060 2021-04-09T18:16:41.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcdda87c700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcddc186c00 0x7fcddc187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:41.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcddb07d700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcddc185060 0x7fcddc185d80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fcdc4001af0 tx=0x7fcdc400ab60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:41.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcdd3fff700 1 -- 172.21.15.32:0/3715085192 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdc4007030 con 0x7fcddc185060 2021-04-09T18:16:41.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcdd3fff700 1 -- 172.21.15.32:0/3715085192 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc40073b0 con 0x7fcddc185060 2021-04-09T18:16:41.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcde139f700 1 -- 172.21.15.32:0/3715085192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc185460 con 0x7fcddc185060 2021-04-09T18:16:41.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcdd3fff700 1 -- 172.21.15.32:0/3715085192 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdc4004920 con 0x7fcddc185060 2021-04-09T18:16:41.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.314 7fcde139f700 1 -- 172.21.15.32:0/3715085192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc185460 con 0x7fcddc185060 2021-04-09T18:16:41.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.315 7fcdd3fff700 1 -- 172.21.15.32:0/3715085192 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcdc40046a0 con 0x7fcddc185060 2021-04-09T18:16:41.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.315 7fcdd3fff700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdc00188c0 0x7fcdc001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:41.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.315 7fcdd3fff700 1 -- 172.21.15.32:0/3715085192 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fcdc4032950 con 0x7fcddc185060 2021-04-09T18:16:41.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.315 7fcdda87c700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdc00188c0 0x7fcdc001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:41.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.316 7fcdda87c700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdc00188c0 0x7fcdc001adc0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:41.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.336 7fcde139f700 1 -- 172.21.15.32:0/3715085192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcddc1866d0 con 0x7fcddc185060 2021-04-09T18:16:41.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.340 7fcdd3fff700 1 -- 172.21.15.32:0/3715085192 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcdc40078c0 con 0x7fcddc185060 2021-04-09T18:16:41.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.528 7fcde139f700 1 -- 172.21.15.32:0/3715085192 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fcddc1866d0 con 0x7fcddc185060 2021-04-09T18:16:41.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.530 7fcdd3fff700 1 -- 172.21.15.32:0/3715085192 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v56) v1 ==== 81+0+25396 (secure 0 0 0) 0x7fcdc40073b0 con 0x7fcddc185060 2021-04-09T18:16:41.582 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:41.582 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:16:41.583 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 56, 2021-04-09T18:16:41.583 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:16:41.584 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:16:41.584 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:16:38.368481", 2021-04-09T18:16:41.584 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:16:41.585 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:16:41.585 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:16:41.586 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:16:41.586 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:16:41.587 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:16:41.595 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:16:41.595 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:16:41.596 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:16:41.597 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:16:41.597 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:16:41.598 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:16:41.598 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:16:41.598 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:16:41.599 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:16:41.599 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 9, 2021-04-09T18:16:41.600 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:16:41.600 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:16:41.601 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:16:41.601 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:16:41.625 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:16:41.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.626 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:16:41.627 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:16:41.627 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:16:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:16:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:16:41.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:16:41.629 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:16:41.636 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:16:41.637 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:16:41.663 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:16:41.664 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:16:41.664 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:16:41.664 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:16:41.665 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:16:41.671 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:16:41.671 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:16:41.672 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:16:41.672 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:16:41.673 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:16:41.673 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:16:41.674 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:16:41.674 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:16:41.674 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:16:41.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.675 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:16:41.676 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:16:41.676 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:16:41.677 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:16:41.677 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:16:41.677 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:16:41.678 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:16:41.678 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:16:41.679 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:16:41.745 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:16:41.746 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:16:41.746 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:16:41.746 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:16:41.746 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:16:41.747 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:16:41.747 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:16:41.747 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:16:41.747 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:16:41.748 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:16:41.748 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:16:41.748 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:16:41.748 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:16:41.748 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:16:41.749 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:16:41.749 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:16:41.749 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:16:41.749 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:16:41.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.750 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:16:41.750 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:16:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:16:41.758 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:16:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:16:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:16:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:16:41.759 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:16:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:16:41.760 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:16:41.823 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:16:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:16:41.824 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:16:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:16:41.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:41.826 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:16:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:16:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.828 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:16:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:16:41.829 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:41.830 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:41.860 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:41.861 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:41.862 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:41.864 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:41.866 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:16:41.905 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:41.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:41.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:16:41.906 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:41.906 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:41.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:41.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.907 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:41.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:41.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:41.908 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:16:41.908 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:41.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:41.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:16:41.909 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:41.934 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:41.934 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:41.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.935 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:41.935 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:41.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:41.947 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:16:41.948 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:41.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:41.950 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:16:41.950 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:41.951 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:41.951 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:41.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.952 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:41.953 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:41.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:41.955 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:16:41.955 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:41.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:41.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:16:41.962 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:16:41.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:41.962 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:41.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.963 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:16:41.964 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:16:41.964 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:16:41.965 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:16:41.965 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:41.965 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:41.966 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:41.966 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:41.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:41.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:41.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:16:41.968 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:16:41.968 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:41.969 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:41.969 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:41.970 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:41.972 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:41.973 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:42.006 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:42.007 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:42.007 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:42.008 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:42.008 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:42.009 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.028 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:16:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.030 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:16:42.031 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.035 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.037 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:16:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.038 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:16:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.039 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.047 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.048 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:16:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:42.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.050 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.093 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:16:42.094 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:42.094 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.095 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.095 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:42.096 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.097 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:16:42.098 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:42.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.124 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:16:42.124 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:16:42.125 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.146 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.179 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:16:42.179 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:16:42.179 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:16:42.180 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:16:42.180 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:42.180 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:42.180 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:42.181 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:16:42.190 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:16:42.191 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:42.191 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:42.191 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:42.192 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:42.192 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:42.193 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:42.193 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:42.193 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 30, 2021-04-09T18:16:42.194 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:42.217 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:16:42.249 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.256 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:16:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.257 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.261 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:16:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:42.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.298 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:16:42.299 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.301 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:16:42.302 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.303 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:16:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.304 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.305 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:16:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.306 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:16:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:16:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.307 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:16:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:16:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:16:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:16:42.308 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.310 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:16:42.331 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:16:42.332 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:42.332 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:42.332 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:42.335 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:42.335 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:42.335 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:42.363 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:42.428 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:42.429 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:42.429 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:42.429 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:42.430 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.466 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:16:42.466 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:16:42.468 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:42.477 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.478 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.489 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:16:42.491 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:42.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:16:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:42.508 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.526 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.527 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.528 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.532 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:16:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:16:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:16:42.536 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.537 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:16:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:16:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:16:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:16:42.539 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:16:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:16:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:42.540 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:16:42.541 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:16:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:42.542 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 51, 2021-04-09T18:16:42.543 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.544 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:16:42.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:42.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:16:42.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:42.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.546 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.547 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:42.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.548 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.548 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:16:42.548 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:42.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.549 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.549 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:16:42.549 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:42.549 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.549 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.551 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:42.551 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.552 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:16:42.552 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:42.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:16:42.553 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:42.553 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.554 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.554 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:42.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.555 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.555 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:16:42.555 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:42.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.556 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.556 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:16:42.556 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:16:42.556 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.556 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.557 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:16:42.557 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:16:42.557 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:16:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:16:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:42.566 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:42.570 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:16:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:16:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:42.573 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:42.574 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:42.578 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.579 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:16:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.636 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:16:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:42.637 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.640 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:42.641 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.643 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:16:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:42.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:16:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:42.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:42.651 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.659 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:16:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.694 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.694 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:16:42.694 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:42.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.748 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.750 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:16:42.750 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.751 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:16:42.752 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:16:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.753 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:16:42.795 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:16:42.796 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:16:42.805 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:16:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:42.813 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:16:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:16:42.827 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:42.828 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:42.829 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:42.830 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 51, 2021-04-09T18:16:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:42.831 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:42.832 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.846 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.846 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:16:42.847 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:42.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:16:42.861 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:42.862 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.862 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.863 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:42.864 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.865 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:16:42.865 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:42.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.866 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.867 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:16:42.867 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:42.868 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.891 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:42.892 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.893 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:16:42.894 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:16:42.896 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:42.934 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.938 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:42.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:42.950 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:16:42.953 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:42.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:42.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:16:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:42.986 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:16:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:16:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:16:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:16:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:42.988 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:42.989 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:16:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:16:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:16:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:16:42.990 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:16:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:16:42.991 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:16:43.024 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:16:43.055 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:16:43.055 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:16:43.056 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:16:43.056 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:16:43.056 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:16:43.056 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:43.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.057 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:43.057 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:16:43.057 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:43.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:43.058 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:16:43.058 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:43.059 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:43.059 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:43.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.078 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:16:43.078 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:43.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.079 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:43.079 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:16:43.079 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:43.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.080 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:43.080 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:16:43.080 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:43.081 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:43.081 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:43.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.081 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:16:43.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:43.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.082 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:43.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:16:43.082 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:43.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.087 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:16:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:43.088 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:16:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:16:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.089 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:16:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:16:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:43.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:43.113 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:16:43.113 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:16:43.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:43.141 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:43.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.156 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:16:43.163 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:16:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:16:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:16:43.164 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:16:43.181 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:16:43.182 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:16:43.182 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:43.182 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:43.183 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:16:43.183 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:16:43.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:16:43.185 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:43.185 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:43.185 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:43.186 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:43.186 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:43.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:16:43.247 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:43.256 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:43.257 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:43.257 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:43.258 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:43.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.260 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:16:43.261 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:43.261 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:43.262 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:43.262 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:43.263 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:43.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.264 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:16:43.264 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:43.287 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:43.287 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:43.288 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:43.288 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:43.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.290 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:16:43.290 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:43.290 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:43.291 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:43.291 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:43.292 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:43.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.293 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:16:43.293 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:43.294 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:43.294 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:43.295 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:16:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:43.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:43.311 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:16:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:16:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:16:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:16:43.313 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:16:43.313 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:16:43.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:43.314 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:16:43.314 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:16:43.315 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:16:43.315 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:16:43.315 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:16:43.316 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:16:43.316 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:16:43.316 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:16:43.317 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:16:43.317 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:16:43.318 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:16:43.331 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:16:43.332 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:16:43.332 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:16:43.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:43.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:43.333 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:16:43.334 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:16:43.334 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:16:43.334 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:16:43.335 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:16:43.335 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:16:43.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 -- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdc00188c0 msgr2=0x7fcdc001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:43.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdc00188c0 0x7fcdc001adc0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 -- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcddc185060 msgr2=0x7fcddc185d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:43.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcddc185060 0x7fcddc185d80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fcdc4001af0 tx=0x7fcdc400ab60).stop 2021-04-09T18:16:43.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 -- 172.21.15.32:0/3715085192 shutdown_connections 2021-04-09T18:16:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdc00188c0 0x7fcdc001adc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcddc186c00 0x7fcddc187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcddc191360 0x7fcddc191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 --2- 172.21.15.32:0/3715085192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcddc185060 0x7fcddc185d80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 -- 172.21.15.32:0/3715085192 >> 172.21.15.32:0/3715085192 conn(0x7fcddc1837d0 msgr2=0x7fcddc195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:43.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 -- 172.21.15.32:0/3715085192 shutdown_connections 2021-04-09T18:16:43.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:41.534 7fcde139f700 1 -- 172.21.15.32:0/3715085192 wait complete. 2021-04-09T18:16:43.423 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:16:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.552 7f644cd5a700 1 Processor -- start 2021-04-09T18:16:43.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.552 7f644cd5a700 1 -- start start 2021-04-09T18:16:43.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.553 7f644cd5a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f644810da00 0x7f644810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:43.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.553 7f644cd5a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644810d480 con 0x7f64480d7ca0 2021-04-09T18:16:43.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.553 7f644cd5a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64480d1f70 con 0x7f644810d010 2021-04-09T18:16:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.553 7f644cd5a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64480d8d30 con 0x7f644810da00 2021-04-09T18:16:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.553 7f6446312700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f64480d7ca0 0x7f64480d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54388/0 (socket says 172.21.15.32:54388) 2021-04-09T18:16:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.553 7f6446312700 1 -- 172.21.15.32:0/1770737160 learned_addr learned my addr 172.21.15.32:0/1770737160 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.553 7f6447314700 1 --2- 172.21.15.32:0/1770737160 >> v2:172.21.15.86:3300/0 conn(0x7f644810da00 0x7f644810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6445b11700 1 -- 172.21.15.32:0/1770737160 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047691632 0 0) 0x7f643c000ce0 con 0x7f64480d7ca0 2021-04-09T18:16:43.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6445b11700 1 -- 172.21.15.32:0/1770737160 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6430002fe0 con 0x7f64480d7ca0 2021-04-09T18:16:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6445b11700 1 -- 172.21.15.32:0/1770737160 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513759012 0 0) 0x7f6438000c60 con 0x7f644810d010 2021-04-09T18:16:43.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6445b11700 1 -- 172.21.15.32:0/1770737160 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64300042c0 con 0x7f644810d010 2021-04-09T18:16:43.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6447314700 1 -- 172.21.15.32:0/1770737160 >> v1:172.21.15.86:6789/0 conn(0x7f644810d010 legacy=0x7f64480d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:43.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6445b11700 1 -- 172.21.15.32:0/1770737160 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 251015346 0 0) 0x7f643c0010e0 con 0x7f64480d7ca0 2021-04-09T18:16:43.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6447314700 1 -- 172.21.15.32:0/1770737160 >> v1:172.21.15.32:6789/0 conn(0x7f64480d7ca0 legacy=0x7f64480d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:43.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6447314700 1 -- 172.21.15.32:0/1770737160 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64480d9220 con 0x7f644810da00 2021-04-09T18:16:43.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6447314700 1 --2- 172.21.15.32:0/1770737160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f644810da00 0x7f644810de00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f6440006e50 tx=0x7f6440009df0).ready entity=mon.1 client_cookie=edb58fab1f03ce65 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:43.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6445b11700 1 -- 172.21.15.32:0/1770737160 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f644000b510 con 0x7f644810da00 2021-04-09T18:16:43.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6445b11700 1 -- 172.21.15.32:0/1770737160 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6440003030 con 0x7f644810da00 2021-04-09T18:16:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f6445b11700 1 -- 172.21.15.32:0/1770737160 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f644000b7d0 con 0x7f644810da00 2021-04-09T18:16:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f644cd5a700 1 -- 172.21.15.32:0/1770737160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f644810da00 msgr2=0x7f644810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:43.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.554 7f644cd5a700 1 --2- 172.21.15.32:0/1770737160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f644810da00 0x7f644810de00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f6440006e50 tx=0x7f6440009df0).stop 2021-04-09T18:16:43.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.555 7f644cd5a700 1 -- 172.21.15.32:0/1770737160 shutdown_connections 2021-04-09T18:16:43.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.555 7f644cd5a700 1 --2- 172.21.15.32:0/1770737160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f644810da00 0x7f644810de00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.555 7f644cd5a700 1 -- 172.21.15.32:0/1770737160 >> 172.21.15.32:0/1770737160 conn(0x7f64480ce5f0 msgr2=0x7f64480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:43.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.555 7f644cd5a700 1 -- 172.21.15.32:0/1770737160 shutdown_connections 2021-04-09T18:16:43.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.555 7f644cd5a700 1 -- 172.21.15.32:0/1770737160 wait complete. 2021-04-09T18:16:43.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.575 7f644cd5a700 1 Processor -- start 2021-04-09T18:16:43.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.575 7f644cd5a700 1 -- start start 2021-04-09T18:16:43.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f644cd5a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64480b5360 0x7f64481a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:43.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f644cd5a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f644818e140 0x7f64481aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:43.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f644cd5a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64481ad040 0x7f644808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:43.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f644cd5a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64481a5360 con 0x7f64481ad040 2021-04-09T18:16:43.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f644cd5a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64480902c0 con 0x7f64480b5360 2021-04-09T18:16:43.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f644cd5a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64480904e0 con 0x7f644818e140 2021-04-09T18:16:43.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6447314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64481ad040 0x7f644808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:43.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6447314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64481ad040 0x7f644808fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44802/0 (socket says 172.21.15.32:44802) 2021-04-09T18:16:43.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6447314700 1 -- 172.21.15.32:0/1985910669 learned_addr learned my addr 172.21.15.32:0/1985910669 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:43.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6446b13700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64480b5360 0x7f64481a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:43.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6446312700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f644818e140 0x7f64481aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6447314700 1 -- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f644818e140 msgr2=0x7f64481aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6447314700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f644818e140 0x7f64481aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6447314700 1 -- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64480b5360 msgr2=0x7f64481a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:43.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6447314700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64480b5360 0x7f64481a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6447314700 1 -- 172.21.15.32:0/1985910669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6448183c80 con 0x7f64481ad040 2021-04-09T18:16:43.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6446312700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f644818e140 0x7f64481aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:43.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.576 7f6446b13700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64480b5360 0x7f64481a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.577 7f6447314700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64481ad040 0x7f644808fca0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f6440009f80 tx=0x7f644000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.577 7f64377fe700 1 -- 172.21.15.32:0/1985910669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64400097c0 con 0x7f64481ad040 2021-04-09T18:16:43.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.577 7f64377fe700 1 -- 172.21.15.32:0/1985910669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64400093d0 con 0x7f64481ad040 2021-04-09T18:16:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.577 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6448183c80 con 0x7f64481ad040 2021-04-09T18:16:43.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.577 7f64377fe700 1 -- 172.21.15.32:0/1985910669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6440001430 con 0x7f64481ad040 2021-04-09T18:16:43.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.577 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6448183c80 con 0x7f64481ad040 2021-04-09T18:16:43.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.578 7f64377fe700 1 -- 172.21.15.32:0/1985910669 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6440001030 con 0x7f64481ad040 2021-04-09T18:16:43.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.578 7f64377fe700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f642c0188c0 0x7f642c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:43.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.578 7f64377fe700 1 -- 172.21.15.32:0/1985910669 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f64400329a0 con 0x7f64481ad040 2021-04-09T18:16:43.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.578 7f6446b13700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f642c0188c0 0x7f642c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:43.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.578 7f6446b13700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f642c0188c0 0x7f642c01adc0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:43.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.597 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6448182d70 con 0x7f64481ad040 2021-04-09T18:16:43.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.601 7f64377fe700 1 -- 172.21.15.32:0/1985910669 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f64400097c0 con 0x7f64481ad040 2021-04-09T18:16:43.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.792 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6448182d70 con 0x7f64481ad040 2021-04-09T18:16:43.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.792 7f64377fe700 1 -- 172.21.15.32:0/1985910669 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+1853 (secure 0 0 0) 0x7f6440001030 con 0x7f64481ad040 2021-04-09T18:16:43.824 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 51 2021-04-09T18:16:43.825 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:43.826 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3388525530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3388525530},{"type":"v1","addr":"172.21.15.32:6833","nonce":3388525530}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2165727103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2165727103},{"type":"v1","addr":"172.21.15.32:6835","nonce":2165727103}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":41},{"gid":55067,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3036306475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3036306475},{"type":"v1","addr":"172.21.15.86:6835","nonce":3036306475}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":55178,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1716252024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1716252024},{"type":"v1","addr":"172.21.15.86:6837","nonce":1716252024}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[]} 2021-04-09T18:16:43.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f642c0188c0 msgr2=0x7f642c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:43.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f642c0188c0 0x7f642c01adc0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64481ad040 msgr2=0x7f644808fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:43.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64481ad040 0x7f644808fca0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f6440009f80 tx=0x7f644000bd10).stop 2021-04-09T18:16:43.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 shutdown_connections 2021-04-09T18:16:43.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f642c0188c0 0x7f642c01adc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64481ad040 0x7f644808fca0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f644818e140 0x7f64481aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 --2- 172.21.15.32:0/1985910669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64480b5360 0x7f64481a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:43.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 >> 172.21.15.32:0/1985910669 conn(0x7f644819b460 msgr2=0x7f644819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:43.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 shutdown_connections 2021-04-09T18:16:43.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.795 7f644cd5a700 1 -- 172.21.15.32:0/1985910669 wait complete. 2021-04-09T18:16:43.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:16:44.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.974 7f50dd0bf700 1 Processor -- start 2021-04-09T18:16:44.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.974 7f50dd0bf700 1 -- start start 2021-04-09T18:16:44.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50dd0bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d810d010 0x7f50d80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50dd0bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f50d80d8490 0x7f50d810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50dd0bf700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d810d480 con 0x7f50d80d1da0 2021-04-09T18:16:44.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50dd0bf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d810e780 con 0x7f50d80d8490 2021-04-09T18:16:44.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50dd0bf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d80d8d30 con 0x7f50d810d010 2021-04-09T18:16:44.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d759e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f50d80d8490 0x7f50d810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d759e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f50d80d8490 0x7f50d810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44810/0 (socket says 172.21.15.32:44810) 2021-04-09T18:16:44.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d759e700 1 -- 172.21.15.32:0/764570681 learned_addr learned my addr 172.21.15.32:0/764570681 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:44.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d6d9d700 1 --2- 172.21.15.32:0/764570681 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d810d010 0x7f50d80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d759e700 1 -- 172.21.15.32:0/764570681 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d810d010 msgr2=0x7f50d80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:44.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d759e700 1 --2- 172.21.15.32:0/764570681 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d810d010 0x7f50d80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d759e700 1 -- 172.21.15.32:0/764570681 >> v1:172.21.15.86:6789/0 conn(0x7f50d80d1da0 legacy=0x7f50d80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d659c700 1 -- 172.21.15.32:0/764570681 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097768751 0 0) 0x7f50c0000c60 con 0x7f50d80d1da0 2021-04-09T18:16:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d759e700 1 -- 172.21.15.32:0/764570681 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d80d9220 con 0x7f50d80d8490 2021-04-09T18:16:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.975 7f50d759e700 1 --2- 172.21.15.32:0/764570681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d80d8490 0x7f50d810da00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f50d0006f00 tx=0x7f50d0009e90).ready entity=mon.0 client_cookie=15f532a351caa5f0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:44.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50d659c700 1 -- 172.21.15.32:0/764570681 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50d000b300 con 0x7f50d80d8490 2021-04-09T18:16:44.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50d659c700 1 -- 172.21.15.32:0/764570681 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d000b640 con 0x7f50d80d8490 2021-04-09T18:16:44.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50d659c700 1 -- 172.21.15.32:0/764570681 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50d000bb20 con 0x7f50d80d8490 2021-04-09T18:16:44.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50dd0bf700 1 -- 172.21.15.32:0/764570681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d80d8490 msgr2=0x7f50d810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50dd0bf700 1 --2- 172.21.15.32:0/764570681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d80d8490 0x7f50d810da00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f50d0006f00 tx=0x7f50d0009e90).stop 2021-04-09T18:16:44.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50dd0bf700 1 -- 172.21.15.32:0/764570681 shutdown_connections 2021-04-09T18:16:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50dd0bf700 1 --2- 172.21.15.32:0/764570681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d80d8490 0x7f50d810da00 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50dd0bf700 1 --2- 172.21.15.32:0/764570681 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d810d010 0x7f50d80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50dd0bf700 1 -- 172.21.15.32:0/764570681 >> 172.21.15.32:0/764570681 conn(0x7f50d80ce5f0 msgr2=0x7f50d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:44.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50dd0bf700 1 -- 172.21.15.32:0/764570681 shutdown_connections 2021-04-09T18:16:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.976 7f50dd0bf700 1 -- 172.21.15.32:0/764570681 wait complete. 2021-04-09T18:16:44.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.996 7f50dd0bf700 1 Processor -- start 2021-04-09T18:16:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50dd0bf700 1 -- start start 2021-04-09T18:16:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50dd0bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d808f040 0x7f50d81a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50dd0bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d808f440 0x7f50d81a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50dd0bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d81a22e0 0x7f50d81a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50dd0bf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d81a3100 con 0x7f50d808f440 2021-04-09T18:16:44.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50dd0bf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d81a2d00 con 0x7f50d808f040 2021-04-09T18:16:44.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50dd0bf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d817c7e0 con 0x7f50d81a22e0 2021-04-09T18:16:44.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50ce59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d808f440 0x7f50d81a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50d759e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d81a22e0 0x7f50d81a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50d6d9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d808f040 0x7f50d81a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50ce59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d808f440 0x7f50d81a3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44816/0 (socket says 172.21.15.32:44816) 2021-04-09T18:16:44.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.997 7f50ce59c700 1 -- 172.21.15.32:0/2007922524 learned_addr learned my addr 172.21.15.32:0/2007922524 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50d759e700 1 -- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d808f040 msgr2=0x7f50d81a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50d759e700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d808f040 0x7f50d81a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50d759e700 1 -- 172.21.15.32:0/2007922524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d808f440 msgr2=0x7f50d81a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50d759e700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d808f440 0x7f50d81a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50d759e700 1 -- 172.21.15.32:0/2007922524 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d80aee10 con 0x7f50d81a22e0 2021-04-09T18:16:44.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50ce59c700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d808f440 0x7f50d81a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:44.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50d759e700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d81a22e0 0x7f50d81a26e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f50d000b8c0 tx=0x7f50d000bc60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:44.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50cffff700 1 -- 172.21.15.32:0/2007922524 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50d0000cc0 con 0x7f50d81a22e0 2021-04-09T18:16:44.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50cffff700 1 -- 172.21.15.32:0/2007922524 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d0003030 con 0x7f50d81a22e0 2021-04-09T18:16:44.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d80aee10 con 0x7f50d81a22e0 2021-04-09T18:16:44.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50cffff700 1 -- 172.21.15.32:0/2007922524 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50d000c4e0 con 0x7f50d81a22e0 2021-04-09T18:16:44.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.998 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d80aee10 con 0x7f50d81a22e0 2021-04-09T18:16:44.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.999 7f50cffff700 1 -- 172.21.15.32:0/2007922524 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f50d0001270 con 0x7f50d81a22e0 2021-04-09T18:16:44.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.999 7f50cffff700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50bc018930 0x7f50bc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.999 7f50cffff700 1 -- 172.21.15.32:0/2007922524 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f50d00329a0 con 0x7f50d81a22e0 2021-04-09T18:16:44.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.999 7f50d6d9d700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50bc018930 0x7f50bc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:43.999 7f50d6d9d700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50bc018930 0x7f50bc01ae10 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:44.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.019 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d81b7050 con 0x7f50d81a22e0 2021-04-09T18:16:44.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.023 7f50cffff700 1 -- 172.21.15.32:0/2007922524 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f50d0006d90 con 0x7f50d81a22e0 2021-04-09T18:16:44.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.214 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50d81b7050 con 0x7f50d81a22e0 2021-04-09T18:16:44.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.215 7f50cffff700 1 -- 172.21.15.32:0/2007922524 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+1853 (secure 0 0 0) 0x7f50d0006d90 con 0x7f50d81a22e0 2021-04-09T18:16:44.247 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 51 2021-04-09T18:16:44.248 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:44.248 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3388525530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3388525530},{"type":"v1","addr":"172.21.15.32:6833","nonce":3388525530}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2165727103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2165727103},{"type":"v1","addr":"172.21.15.32:6835","nonce":2165727103}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":41},{"gid":55067,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/3036306475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3036306475},{"type":"v1","addr":"172.21.15.86:6835","nonce":3036306475}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40},{"gid":55178,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1716252024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1716252024},{"type":"v1","addr":"172.21.15.86:6837","nonce":1716252024}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[]} 2021-04-09T18:16:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50bc018930 msgr2=0x7f50bc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50bc018930 0x7f50bc01ae10 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d81a22e0 msgr2=0x7f50d81a26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d81a22e0 0x7f50d81a26e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f50d000b8c0 tx=0x7f50d000bc60).stop 2021-04-09T18:16:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 shutdown_connections 2021-04-09T18:16:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50bc018930 0x7f50bc01ae10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50d808f440 0x7f50d81a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d81a22e0 0x7f50d81a26e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 --2- 172.21.15.32:0/2007922524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d808f040 0x7f50d81a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.217 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 >> 172.21.15.32:0/2007922524 conn(0x7f50d8199680 msgr2=0x7f50d81a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:44.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.218 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 shutdown_connections 2021-04-09T18:16:44.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.218 7f50dd0bf700 1 -- 172.21.15.32:0/2007922524 wait complete. 2021-04-09T18:16:44.259 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:16:44.259 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:16:44.259 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:16:44.260 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:16:44.260 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:16:44.260 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:16:44.261 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:16:44.261 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:16:44.262 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:16:44.262 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:16:44.263 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:16:44.263 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:16:44.264 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:16:44.264 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:16:44.265 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:16:44.265 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:16:44.269 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:16:44.270 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:16:44.271 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:16:44.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:16:44.302 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:16:44.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:16:44.339 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:16:44.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:16:44.359 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:16:44.360 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:16:44.361 DEBUG:teuthology.parallel:result is None 2021-04-09T18:16:44.362 DEBUG:teuthology.parallel:result is None 2021-04-09T18:16:44.385 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:16:44.386 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:16:44.387 DEBUG:teuthology.parallel:result is None 2021-04-09T18:16:44.388 DEBUG:teuthology.parallel:result is None 2021-04-09T18:16:44.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:16:44.517 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:16:44.517 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:16:44.537 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:16:44.539 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:16:44.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.549 7eff1cc0b700 1 Processor -- start 2021-04-09T18:16:44.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.549 7eff1cc0b700 1 -- start start 2021-04-09T18:16:44.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1cc0b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7eff180d6c90 0x7eff180d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1cc0b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7eff180d2450 0x7eff1810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1cc0b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff180d7290 con 0x7eff180cffb0 2021-04-09T18:16:44.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1cc0b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff180d7090 con 0x7eff180d6c90 2021-04-09T18:16:44.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1cc0b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff180d91f0 con 0x7eff180d2450 2021-04-09T18:16:44.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1685e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7eff180cffb0 0x7eff180d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54436/0 (socket says 172.21.15.32:54436) 2021-04-09T18:16:44.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1685e700 1 -- 172.21.15.32:0/2038324691 learned_addr learned my addr 172.21.15.32:0/2038324691 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:44.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1605d700 1 --2- 172.21.15.32:0/2038324691 >> v2:172.21.15.32:3300/0 conn(0x7eff180d6c90 0x7eff180d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1705f700 1 --2- 172.21.15.32:0/2038324691 >> v2:172.21.15.86:3300/0 conn(0x7eff180d2450 0x7eff1810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1585c700 1 -- 172.21.15.32:0/2038324691 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020756212 0 0) 0x7eff08000ce0 con 0x7eff180cffb0 2021-04-09T18:16:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1585c700 1 -- 172.21.15.32:0/2038324691 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff00002fe0 con 0x7eff180cffb0 2021-04-09T18:16:44.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1605d700 1 -- 172.21.15.32:0/2038324691 >> v2:172.21.15.86:3300/0 conn(0x7eff180d2450 msgr2=0x7eff1810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:44.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1605d700 1 --2- 172.21.15.32:0/2038324691 >> v2:172.21.15.86:3300/0 conn(0x7eff180d2450 0x7eff1810cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1605d700 1 -- 172.21.15.32:0/2038324691 >> v1:172.21.15.32:6789/0 conn(0x7eff180cffb0 legacy=0x7eff180d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1605d700 1 -- 172.21.15.32:0/2038324691 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff180d2850 con 0x7eff180d6c90 2021-04-09T18:16:44.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1605d700 1 --2- 172.21.15.32:0/2038324691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff180d6c90 0x7eff180d8000 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7eff0c006650 tx=0x7eff0c009a80).ready entity=mon.0 client_cookie=9d045b9e9e274591 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:44.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1585c700 1 -- 172.21.15.32:0/2038324691 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff0c0009b0 con 0x7eff180d6c90 2021-04-09T18:16:44.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1585c700 1 -- 172.21.15.32:0/2038324691 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0c00c0c0 con 0x7eff180d6c90 2021-04-09T18:16:44.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1585c700 1 -- 172.21.15.32:0/2038324691 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff0c00c510 con 0x7eff180d6c90 2021-04-09T18:16:44.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1cc0b700 1 -- 172.21.15.32:0/2038324691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff180d6c90 msgr2=0x7eff180d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.550 7eff1cc0b700 1 --2- 172.21.15.32:0/2038324691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff180d6c90 0x7eff180d8000 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7eff0c006650 tx=0x7eff0c009a80).stop 2021-04-09T18:16:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.551 7eff1cc0b700 1 -- 172.21.15.32:0/2038324691 shutdown_connections 2021-04-09T18:16:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.551 7eff1cc0b700 1 --2- 172.21.15.32:0/2038324691 >> v2:172.21.15.86:3300/0 conn(0x7eff180d2450 0x7eff1810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.551 7eff1cc0b700 1 --2- 172.21.15.32:0/2038324691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff180d6c90 0x7eff180d8000 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.551 7eff1cc0b700 1 -- 172.21.15.32:0/2038324691 >> 172.21.15.32:0/2038324691 conn(0x7eff180ce5d0 msgr2=0x7eff180cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:44.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.551 7eff1cc0b700 1 -- 172.21.15.32:0/2038324691 shutdown_connections 2021-04-09T18:16:44.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.551 7eff1cc0b700 1 -- 172.21.15.32:0/2038324691 wait complete. 2021-04-09T18:16:44.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.571 7eff1cc0b700 1 Processor -- start 2021-04-09T18:16:44.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.571 7eff1cc0b700 1 -- start start 2021-04-09T18:16:44.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1cc0b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff1808d210 0x7eff1808fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1cc0b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff180903f0 0x7eff181a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1cc0b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff181a6080 0x7eff181a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1cc0b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1808d690 con 0x7eff181a6080 2021-04-09T18:16:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1cc0b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180b0fd0 con 0x7eff180903f0 2021-04-09T18:16:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1cc0b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff181a6480 con 0x7eff1808d210 2021-04-09T18:16:44.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff181a6080 0x7eff181a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff181a6080 0x7eff181a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44854/0 (socket says 172.21.15.32:44854) 2021-04-09T18:16:44.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1705f700 1 -- 172.21.15.32:0/2893765598 learned_addr learned my addr 172.21.15.32:0/2893765598 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:44.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1605d700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff180903f0 0x7eff181a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1685e700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff1808d210 0x7eff1808fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1705f700 1 -- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff1808d210 msgr2=0x7eff1808fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1705f700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff1808d210 0x7eff1808fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1705f700 1 -- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff180903f0 msgr2=0x7eff181a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1705f700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff180903f0 0x7eff181a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.572 7eff1705f700 1 -- 172.21.15.32:0/2893765598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1808d870 con 0x7eff181a6080 2021-04-09T18:16:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.573 7eff1705f700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff181a6080 0x7eff181a46b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7eff1000b910 tx=0x7eff1000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.573 7eff077fe700 1 -- 172.21.15.32:0/2893765598 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff10006aa0 con 0x7eff181a6080 2021-04-09T18:16:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.573 7eff077fe700 1 -- 172.21.15.32:0/2893765598 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff100011f0 con 0x7eff181a6080 2021-04-09T18:16:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.573 7eff077fe700 1 -- 172.21.15.32:0/2893765598 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff10001640 con 0x7eff181a6080 2021-04-09T18:16:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.573 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1808d870 con 0x7eff181a6080 2021-04-09T18:16:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.573 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1808d870 con 0x7eff181a6080 2021-04-09T18:16:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.573 7eff077fe700 1 -- 172.21.15.32:0/2893765598 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7eff100013d0 con 0x7eff181a6080 2021-04-09T18:16:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.574 7eff077fe700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efefc0188d0 0x7efefc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:44.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.574 7eff077fe700 1 -- 172.21.15.32:0/2893765598 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4757+0+0 (secure 0 0 0) 0x7eff10032c20 con 0x7eff181a6080 2021-04-09T18:16:44.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.574 7eff1685e700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efefc0188d0 0x7efefc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:44.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.574 7eff1685e700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efefc0188d0 0x7efefc01add0 crc :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:44.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.594 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff181a52b0 con 0x7eff181a6080 2021-04-09T18:16:44.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.597 7eff077fe700 1 -- 172.21.15.32:0/2893765598 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7eff1000bdf0 con 0x7eff181a6080 2021-04-09T18:16:44.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.797 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7eff181a52b0 con 0x7eff181a6080 2021-04-09T18:16:44.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.849 7eff077fe700 1 -- 172.21.15.32:0/2893765598 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v57) v1 ==== 97+0+0 (secure 0 0 0) 0x7eff100011f0 con 0x7eff181a6080 2021-04-09T18:16:44.891 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:16:44.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efefc0188d0 msgr2=0x7efefc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efefc0188d0 0x7efefc01add0 crc :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff181a6080 msgr2=0x7eff181a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:44.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff181a6080 0x7eff181a46b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7eff1000b910 tx=0x7eff1000bcb0).stop 2021-04-09T18:16:44.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 shutdown_connections 2021-04-09T18:16:44.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efefc0188d0 0x7efefc01add0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff181a6080 0x7eff181a46b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff180903f0 0x7eff181a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 --2- 172.21.15.32:0/2893765598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff1808d210 0x7eff1808fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:44.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 >> 172.21.15.32:0/2893765598 conn(0x7eff181a8f50 msgr2=0x7eff18188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:44.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 shutdown_connections 2021-04-09T18:16:44.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:44.863 7eff1cc0b700 1 -- 172.21.15.32:0/2893765598 wait complete. 2021-04-09T18:16:44.904 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:16:44.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:16:45.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.031 7fb550270700 1 Processor -- start 2021-04-09T18:16:45.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.031 7fb550270700 1 -- start start 2021-04-09T18:16:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb550270700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480cffa0 0x7fb5480d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb550270700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5480d7800 0x7fb5480d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb550270700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5480d7280 con 0x7fb5480d5c70 2021-04-09T18:16:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb550270700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5480d8580 con 0x7fb5480d7800 2021-04-09T18:16:45.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb550270700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5480d91e0 con 0x7fb5480cffa0 2021-04-09T18:16:45.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb54d88b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5480d7800 0x7fb5480d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb54e88d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb5480d5c70 0x7fb5480d6070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54452/0 (socket says 172.21.15.32:54452) 2021-04-09T18:16:45.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb54e88d700 1 -- 172.21.15.32:0/104635061 learned_addr learned my addr 172.21.15.32:0/104635061 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:45.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb54e08c700 1 --2- 172.21.15.32:0/104635061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480cffa0 0x7fb5480d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.032 7fb54d08a700 1 -- 172.21.15.32:0/104635061 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029887774 0 0) 0x7fb540000d30 con 0x7fb5480d5c70 2021-04-09T18:16:45.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54d08a700 1 -- 172.21.15.32:0/104635061 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb534002fe0 con 0x7fb5480d5c70 2021-04-09T18:16:45.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54d88b700 1 -- 172.21.15.32:0/104635061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480cffa0 msgr2=0x7fb5480d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:45.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54d88b700 1 --2- 172.21.15.32:0/104635061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480cffa0 0x7fb5480d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54d88b700 1 -- 172.21.15.32:0/104635061 >> v1:172.21.15.32:6789/0 conn(0x7fb5480d5c70 legacy=0x7fb5480d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54d88b700 1 -- 172.21.15.32:0/104635061 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5480d8220 con 0x7fb5480d7800 2021-04-09T18:16:45.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54e08c700 1 --2- 172.21.15.32:0/104635061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480cffa0 0x7fb5480d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:45.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54d88b700 1 --2- 172.21.15.32:0/104635061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5480d7800 0x7fb5480d7c00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fb53c006650 tx=0x7fb53c009a80).ready entity=mon.0 client_cookie=c8a52e6e8687a4a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:45.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54d08a700 1 -- 172.21.15.32:0/104635061 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb53c0009b0 con 0x7fb5480d7800 2021-04-09T18:16:45.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54d08a700 1 -- 172.21.15.32:0/104635061 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c00c0c0 con 0x7fb5480d7800 2021-04-09T18:16:45.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb54d08a700 1 -- 172.21.15.32:0/104635061 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb53c00c510 con 0x7fb5480d7800 2021-04-09T18:16:45.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb550270700 1 -- 172.21.15.32:0/104635061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5480d7800 msgr2=0x7fb5480d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.033 7fb550270700 1 --2- 172.21.15.32:0/104635061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5480d7800 0x7fb5480d7c00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fb53c006650 tx=0x7fb53c009a80).stop 2021-04-09T18:16:45.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.034 7fb550270700 1 -- 172.21.15.32:0/104635061 shutdown_connections 2021-04-09T18:16:45.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.034 7fb550270700 1 --2- 172.21.15.32:0/104635061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5480d7800 0x7fb5480d7c00 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.034 7fb550270700 1 --2- 172.21.15.32:0/104635061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480cffa0 0x7fb5480d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.034 7fb550270700 1 -- 172.21.15.32:0/104635061 >> 172.21.15.32:0/104635061 conn(0x7fb5480ce5c0 msgr2=0x7fb5480cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:45.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.034 7fb550270700 1 -- 172.21.15.32:0/104635061 shutdown_connections 2021-04-09T18:16:45.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.034 7fb550270700 1 -- 172.21.15.32:0/104635061 wait complete. 2021-04-09T18:16:45.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.054 7fb550270700 1 Processor -- start 2021-04-09T18:16:45.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.054 7fb550270700 1 -- start start 2021-04-09T18:16:45.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.054 7fb550270700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480a80c0 0x7fb54817bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.054 7fb550270700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54817b1d0 0x7fb54817b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb550270700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54808dc50 0x7fb54808e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb550270700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54817bc70 con 0x7fb54808dc50 2021-04-09T18:16:45.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb550270700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54808e670 con 0x7fb54817b1d0 2021-04-09T18:16:45.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb550270700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54808e850 con 0x7fb5480a80c0 2021-04-09T18:16:45.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e88d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54808dc50 0x7fb54808e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e88d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54808dc50 0x7fb54808e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44868/0 (socket says 172.21.15.32:44868) 2021-04-09T18:16:45.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e88d700 1 -- 172.21.15.32:0/3105626485 learned_addr learned my addr 172.21.15.32:0/3105626485 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:45.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54d88b700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54817b1d0 0x7fb54817b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e08c700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480a80c0 0x7fb54817bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e88d700 1 -- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480a80c0 msgr2=0x7fb54817bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e88d700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480a80c0 0x7fb54817bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e88d700 1 -- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54817b1d0 msgr2=0x7fb54817b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e88d700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54817b1d0 0x7fb54817b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e88d700 1 -- 172.21.15.32:0/3105626485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54808c660 con 0x7fb54808dc50 2021-04-09T18:16:45.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb54e88d700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54808dc50 0x7fb54808e050 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fb540009980 tx=0x7fb54000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:45.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb53affd700 1 -- 172.21.15.32:0/3105626485 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb54000be10 con 0x7fb54808dc50 2021-04-09T18:16:45.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb53affd700 1 -- 172.21.15.32:0/3105626485 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540006030 con 0x7fb54808dc50 2021-04-09T18:16:45.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb550270700 1 -- 172.21.15.32:0/3105626485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54808c660 con 0x7fb54808dc50 2021-04-09T18:16:45.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb53affd700 1 -- 172.21.15.32:0/3105626485 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5400064c0 con 0x7fb54808dc50 2021-04-09T18:16:45.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.055 7fb550270700 1 -- 172.21.15.32:0/3105626485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54808c660 con 0x7fb54808dc50 2021-04-09T18:16:45.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.056 7fb53affd700 1 -- 172.21.15.32:0/3105626485 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb540006210 con 0x7fb54808dc50 2021-04-09T18:16:45.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.056 7fb53affd700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb530018930 0x7fb53001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.056 7fb53affd700 1 -- 172.21.15.32:0/3105626485 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fb540032a30 con 0x7fb54808dc50 2021-04-09T18:16:45.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.056 7fb54e08c700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb530018930 0x7fb53001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.056 7fb54e08c700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb530018930 0x7fb53001ae10 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:45.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.076 7fb550270700 1 -- 172.21.15.32:0/3105626485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5481b70c0 con 0x7fb54808dc50 2021-04-09T18:16:45.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.080 7fb53affd700 1 -- 172.21.15.32:0/3105626485 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb540000f90 con 0x7fb54808dc50 2021-04-09T18:16:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.266 7fb550270700 1 -- 172.21.15.32:0/3105626485 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb5481b70c0 con 0x7fb54808dc50 2021-04-09T18:16:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.266 7fb53affd700 1 -- 172.21.15.32:0/3105626485 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+4539 (secure 0 0 0) 0x7fb540006030 con 0x7fb54808dc50 2021-04-09T18:16:45.298 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:45.299 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:16:45.299 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:16:45.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.300 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:16:45.300 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:16:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:16:45.301 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.302 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.302 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.302 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.303 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:16:45.304 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:16:45.305 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.305 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:16:45.305 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.306 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.306 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.307 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.308 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:16:45.308 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:16:45.308 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.309 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:16:45.309 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.310 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.310 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.320 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:16:45.320 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:16:45.321 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.321 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:16:45.321 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.322 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.322 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.324 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.337 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:16:45.338 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:16:45.338 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.339 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.339 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.339 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.340 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.341 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.341 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:16:45.341 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:16:45.342 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.342 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.343 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.360 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.360 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.363 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:16:45.364 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:16:45.364 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.364 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.364 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.373 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.373 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.379 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:16:45.380 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:16:45.384 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.385 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.386 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.387 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.387 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.388 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:16:45.389 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:16:45.400 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.401 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.401 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.401 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.402 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.403 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:16:45.404 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:16:45.404 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.405 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.405 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.405 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.442 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:16:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:16:45.443 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.446 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:16:45.459 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:16:45.460 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.486 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.488 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:16:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:16:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.489 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:16:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:16:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:16:45.490 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:16:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:16:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.525 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:16:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:16:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:16:45.526 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:16:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:16:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.544 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:16:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:16:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:16:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:16:45.546 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:16:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:16:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:16:45.563 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.567 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:45.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:45.567 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:16:45.568 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:16:45.568 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:16:45.568 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:16:45.607 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:16:45.607 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:16:45.608 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.608 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:45.608 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:16:45.608 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:16:45.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 -- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb530018930 msgr2=0x7fb53001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb530018930 0x7fb53001ae10 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 -- 172.21.15.32:0/3105626485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54808dc50 msgr2=0x7fb54808e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54808dc50 0x7fb54808e050 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fb540009980 tx=0x7fb54000bcd0).stop 2021-04-09T18:16:45.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 -- 172.21.15.32:0/3105626485 shutdown_connections 2021-04-09T18:16:45.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb530018930 0x7fb53001ae10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54808dc50 0x7fb54808e050 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54817b1d0 0x7fb54817b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 --2- 172.21.15.32:0/3105626485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5480a80c0 0x7fb54817bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 -- 172.21.15.32:0/3105626485 >> 172.21.15.32:0/3105626485 conn(0x7fb548189b00 msgr2=0x7fb5481921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:45.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 -- 172.21.15.32:0/3105626485 shutdown_connections 2021-04-09T18:16:45.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.269 7fb550270700 1 -- 172.21.15.32:0/3105626485 wait complete. 2021-04-09T18:16:45.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:16:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.823 7f459aca1700 1 Processor -- start 2021-04-09T18:16:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.823 7f459aca1700 1 -- start start 2021-04-09T18:16:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.823 7f459aca1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f459410d010 0x7f45940d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.823 7f459aca1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f45940d1da0 0x7f45940d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.823 7f459aca1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459410d480 con 0x7f45940d82c0 2021-04-09T18:16:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.823 7f459aca1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45940d8d30 con 0x7f459410d010 2021-04-09T18:16:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.823 7f459aca1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45940d9220 con 0x7f45940d1da0 2021-04-09T18:16:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f4598abd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f459410d010 0x7f45940d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f4598abd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f459410d010 0x7f45940d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44874/0 (socket says 172.21.15.32:44874) 2021-04-09T18:16:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f4598abd700 1 -- 172.21.15.32:0/1994413631 learned_addr learned my addr 172.21.15.32:0/1994413631 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f4593fff700 1 --2- 172.21.15.32:0/1994413631 >> v2:172.21.15.86:3300/0 conn(0x7f45940d1da0 0x7f45940d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f4598abd700 1 -- 172.21.15.32:0/1994413631 >> v2:172.21.15.86:3300/0 conn(0x7f45940d1da0 msgr2=0x7f45940d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f4598abd700 1 --2- 172.21.15.32:0/1994413631 >> v2:172.21.15.86:3300/0 conn(0x7f45940d1da0 0x7f45940d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f4598abd700 1 -- 172.21.15.32:0/1994413631 >> v1:172.21.15.86:6789/0 conn(0x7f45940d82c0 legacy=0x7f459410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f4598abd700 1 -- 172.21.15.32:0/1994413631 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45940d86f0 con 0x7f459410d010 2021-04-09T18:16:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f4598abd700 1 --2- 172.21.15.32:0/1994413631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f459410d010 0x7f45940d1780 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f457c006700 tx=0x7f457c009b00).ready entity=mon.0 client_cookie=fca9e75c7a850b8b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f45937fe700 1 -- 172.21.15.32:0/1994413631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f457c00c0c0 con 0x7f459410d010 2021-04-09T18:16:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f45937fe700 1 -- 172.21.15.32:0/1994413631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c00c400 con 0x7f459410d010 2021-04-09T18:16:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f45937fe700 1 -- 172.21.15.32:0/1994413631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f457c00c8e0 con 0x7f459410d010 2021-04-09T18:16:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f459aca1700 1 -- 172.21.15.32:0/1994413631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f459410d010 msgr2=0x7f45940d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.824 7f459aca1700 1 --2- 172.21.15.32:0/1994413631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f459410d010 0x7f45940d1780 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f457c006700 tx=0x7f457c009b00).stop 2021-04-09T18:16:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.825 7f459aca1700 1 -- 172.21.15.32:0/1994413631 shutdown_connections 2021-04-09T18:16:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.825 7f459aca1700 1 --2- 172.21.15.32:0/1994413631 >> v2:172.21.15.86:3300/0 conn(0x7f45940d1da0 0x7f45940d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.825 7f459aca1700 1 --2- 172.21.15.32:0/1994413631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f459410d010 0x7f45940d1780 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.825 7f459aca1700 1 -- 172.21.15.32:0/1994413631 >> 172.21.15.32:0/1994413631 conn(0x7f45940ce5f0 msgr2=0x7f45940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.825 7f459aca1700 1 -- 172.21.15.32:0/1994413631 shutdown_connections 2021-04-09T18:16:45.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.825 7f459aca1700 1 -- 172.21.15.32:0/1994413631 wait complete. 2021-04-09T18:16:45.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.845 7f459aca1700 1 Processor -- start 2021-04-09T18:16:45.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.845 7f459aca1700 1 -- start start 2021-04-09T18:16:45.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f459aca1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45941ac380 0x7f459418a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f459aca1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45941ac780 0x7f459418a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f459aca1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4594183520 0x7f4594183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f459aca1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45941ace10 con 0x7f45941ac780 2021-04-09T18:16:45.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f459aca1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4594189cf0 con 0x7f4594183520 2021-04-09T18:16:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f459aca1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4594183f40 con 0x7f45941ac380 2021-04-09T18:16:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4593fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45941ac780 0x7f459418a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4593fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45941ac780 0x7f459418a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44880/0 (socket says 172.21.15.32:44880) 2021-04-09T18:16:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4593fff700 1 -- 172.21.15.32:0/4193356647 learned_addr learned my addr 172.21.15.32:0/4193356647 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:45.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f45992be700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4594183520 0x7f4594183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4598abd700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45941ac380 0x7f459418a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4593fff700 1 -- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45941ac380 msgr2=0x7f459418a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4593fff700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45941ac380 0x7f459418a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4593fff700 1 -- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4594183520 msgr2=0x7f4594183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4593fff700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4594183520 0x7f4594183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4593fff700 1 -- 172.21.15.32:0/4193356647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594184160 con 0x7f45941ac780 2021-04-09T18:16:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4598abd700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45941ac380 0x7f459418a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.846 7f4593fff700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45941ac780 0x7f459418a850 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f45840099e0 tx=0x7f458400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:45.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.847 7f45917fa700 1 -- 172.21.15.32:0/4193356647 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f458400c2f0 con 0x7f45941ac780 2021-04-09T18:16:45.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.847 7f45917fa700 1 -- 172.21.15.32:0/4193356647 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458400c5f0 con 0x7f45941ac780 2021-04-09T18:16:45.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.847 7f45917fa700 1 -- 172.21.15.32:0/4193356647 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f458400cad0 con 0x7f45941ac780 2021-04-09T18:16:45.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.847 7f459aca1700 1 -- 172.21.15.32:0/4193356647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4594184160 con 0x7f45941ac780 2021-04-09T18:16:45.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.847 7f459aca1700 1 -- 172.21.15.32:0/4193356647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4594184160 con 0x7f45941ac780 2021-04-09T18:16:45.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.847 7f45917fa700 1 -- 172.21.15.32:0/4193356647 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4584005660 con 0x7f45941ac780 2021-04-09T18:16:45.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.847 7f45917fa700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4578018930 0x7f457801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:45.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.847 7f45917fa700 1 -- 172.21.15.32:0/4193356647 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f4584032a40 con 0x7f45941ac780 2021-04-09T18:16:45.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.848 7f4598abd700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4578018930 0x7f457801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:45.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.848 7f4598abd700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4578018930 0x7f457801ae10 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:45.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.868 7f459aca1700 1 -- 172.21.15.32:0/4193356647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4594189cf0 con 0x7f45941ac780 2021-04-09T18:16:45.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:45.871 7f45917fa700 1 -- 172.21.15.32:0/4193356647 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f458401fab0 con 0x7f45941ac780 2021-04-09T18:16:46.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.058 7f459aca1700 1 -- 172.21.15.32:0/4193356647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4594189cf0 con 0x7f45941ac780 2021-04-09T18:16:46.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.059 7f45917fa700 1 -- 172.21.15.32:0/4193356647 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+11144 (secure 0 0 0) 0x7f4584005660 con 0x7f45941ac780 2021-04-09T18:16:46.094 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:46.094 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":57,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:16:44.845481","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:16:46.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 -- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4578018930 msgr2=0x7f457801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4578018930 0x7f457801ae10 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 -- 172.21.15.32:0/4193356647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45941ac780 msgr2=0x7f459418a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45941ac780 0x7f459418a850 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f45840099e0 tx=0x7f458400bd40).stop 2021-04-09T18:16:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 -- 172.21.15.32:0/4193356647 shutdown_connections 2021-04-09T18:16:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4578018930 0x7f457801ae10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45941ac780 0x7f459418a850 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4594183520 0x7f4594183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 --2- 172.21.15.32:0/4193356647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45941ac380 0x7f459418a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 -- 172.21.15.32:0/4193356647 >> 172.21.15.32:0/4193356647 conn(0x7f459419b890 msgr2=0x7f45941a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:46.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 -- 172.21.15.32:0/4193356647 shutdown_connections 2021-04-09T18:16:46.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.061 7f459aca1700 1 -- 172.21.15.32:0/4193356647 wait complete. 2021-04-09T18:16:46.110 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:16:46.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:16:46.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.242 7fc6dbb1b700 1 Processor -- start 2021-04-09T18:16:46.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.242 7fc6dbb1b700 1 -- start start 2021-04-09T18:16:46.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.243 7fc6dbb1b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc6d410d010 0x7fc6d40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.243 7fc6dbb1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc6d40d1da0 0x7fc6d40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.243 7fc6dbb1b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d410d480 con 0x7fc6d40d82c0 2021-04-09T18:16:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.243 7fc6dbb1b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d40d8d30 con 0x7fc6d40d1da0 2021-04-09T18:16:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.243 7fc6dbb1b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d40d9220 con 0x7fc6d410d010 2021-04-09T18:16:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.243 7fc6d9136700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc6d40d1da0 0x7fc6d40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.243 7fc6da138700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc6d40d82c0 0x7fc6d410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54478/0 (socket says 172.21.15.32:54478) 2021-04-09T18:16:46.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.243 7fc6da138700 1 -- 172.21.15.32:0/3484385165 learned_addr learned my addr 172.21.15.32:0/3484385165 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:46.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.243 7fc6d9937700 1 --2- 172.21.15.32:0/3484385165 >> v2:172.21.15.86:3300/0 conn(0x7fc6d410d010 0x7fc6d40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d9136700 1 -- 172.21.15.32:0/3484385165 >> v2:172.21.15.86:3300/0 conn(0x7fc6d410d010 msgr2=0x7fc6d40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:46.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d9136700 1 --2- 172.21.15.32:0/3484385165 >> v2:172.21.15.86:3300/0 conn(0x7fc6d410d010 0x7fc6d40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d9136700 1 -- 172.21.15.32:0/3484385165 >> v1:172.21.15.32:6789/0 conn(0x7fc6d40d82c0 legacy=0x7fc6d410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d8935700 1 -- 172.21.15.32:0/3484385165 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455800924 0 0) 0x7fc6cc000d30 con 0x7fc6d40d82c0 2021-04-09T18:16:46.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d9136700 1 -- 172.21.15.32:0/3484385165 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d40d86f0 con 0x7fc6d40d1da0 2021-04-09T18:16:46.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d9937700 1 --2- 172.21.15.32:0/3484385165 >> v2:172.21.15.86:3300/0 conn(0x7fc6d410d010 0x7fc6d40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:46.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d9136700 1 --2- 172.21.15.32:0/3484385165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d40d1da0 0x7fc6d40d7ca0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7fc6c4006650 tx=0x7fc6c4009a80).ready entity=mon.0 client_cookie=214f6b79c4f4e316 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:46.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d8935700 1 -- 172.21.15.32:0/3484385165 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6c40009b0 con 0x7fc6d40d1da0 2021-04-09T18:16:46.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d8935700 1 -- 172.21.15.32:0/3484385165 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c400c0c0 con 0x7fc6d40d1da0 2021-04-09T18:16:46.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6d8935700 1 -- 172.21.15.32:0/3484385165 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6c400c510 con 0x7fc6d40d1da0 2021-04-09T18:16:46.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6dbb1b700 1 -- 172.21.15.32:0/3484385165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d40d1da0 msgr2=0x7fc6d40d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.244 7fc6dbb1b700 1 --2- 172.21.15.32:0/3484385165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d40d1da0 0x7fc6d40d7ca0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7fc6c4006650 tx=0x7fc6c4009a80).stop 2021-04-09T18:16:46.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.245 7fc6dbb1b700 1 -- 172.21.15.32:0/3484385165 shutdown_connections 2021-04-09T18:16:46.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.245 7fc6dbb1b700 1 --2- 172.21.15.32:0/3484385165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d40d1da0 0x7fc6d40d7ca0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.245 7fc6dbb1b700 1 --2- 172.21.15.32:0/3484385165 >> v2:172.21.15.86:3300/0 conn(0x7fc6d410d010 0x7fc6d40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.245 7fc6dbb1b700 1 -- 172.21.15.32:0/3484385165 >> 172.21.15.32:0/3484385165 conn(0x7fc6d40ce5f0 msgr2=0x7fc6d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:46.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.245 7fc6dbb1b700 1 -- 172.21.15.32:0/3484385165 shutdown_connections 2021-04-09T18:16:46.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.245 7fc6dbb1b700 1 -- 172.21.15.32:0/3484385165 wait complete. 2021-04-09T18:16:46.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.265 7fc6dbb1b700 1 Processor -- start 2021-04-09T18:16:46.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.265 7fc6dbb1b700 1 -- start start 2021-04-09T18:16:46.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6dbb1b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d40a0a10 0x7fc6d40bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6dbb1b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d418ad00 0x7fc6d419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6dbb1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d41a03f0 0x7fc6d4095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6dbb1b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d418b100 con 0x7fc6d41a03f0 2021-04-09T18:16:46.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6dbb1b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d4095860 con 0x7fc6d40a0a10 2021-04-09T18:16:46.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6dbb1b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d4095a80 con 0x7fc6d418ad00 2021-04-09T18:16:46.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6da138700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d41a03f0 0x7fc6d4095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6da138700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d41a03f0 0x7fc6d4095240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44896/0 (socket says 172.21.15.32:44896) 2021-04-09T18:16:46.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6da138700 1 -- 172.21.15.32:0/2542953828 learned_addr learned my addr 172.21.15.32:0/2542953828 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6d9937700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d40a0a10 0x7fc6d40bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6d9136700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d418ad00 0x7fc6d419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6da138700 1 -- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d418ad00 msgr2=0x7fc6d419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6da138700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d418ad00 0x7fc6d419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6da138700 1 -- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d40a0a10 msgr2=0x7fc6d40bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6da138700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d40a0a10 0x7fc6d40bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6da138700 1 -- 172.21.15.32:0/2542953828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d4189250 con 0x7fc6d41a03f0 2021-04-09T18:16:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6d9937700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d40a0a10 0x7fc6d40bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6da138700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d41a03f0 0x7fc6d4095240 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7fc6cc007f90 tx=0x7fc6cc00ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:46.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6ca7fc700 1 -- 172.21.15.32:0/2542953828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6cc000cc0 con 0x7fc6d41a03f0 2021-04-09T18:16:46.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6ca7fc700 1 -- 172.21.15.32:0/2542953828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc007020 con 0x7fc6d41a03f0 2021-04-09T18:16:46.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6ca7fc700 1 -- 172.21.15.32:0/2542953828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6cc007500 con 0x7fc6d41a03f0 2021-04-09T18:16:46.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.266 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d4189250 con 0x7fc6d41a03f0 2021-04-09T18:16:46.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.267 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d4189250 con 0x7fc6d41a03f0 2021-04-09T18:16:46.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.267 7fc6ca7fc700 1 -- 172.21.15.32:0/2542953828 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc6cc007200 con 0x7fc6d41a03f0 2021-04-09T18:16:46.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.267 7fc6ca7fc700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6bc018900 0x7fc6bc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.267 7fc6ca7fc700 1 -- 172.21.15.32:0/2542953828 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fc6cc032a40 con 0x7fc6d41a03f0 2021-04-09T18:16:46.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.267 7fc6d9937700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6bc018900 0x7fc6bc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.268 7fc6d9937700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6bc018900 0x7fc6bc01adc0 crc :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:46.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.287 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d4093d50 con 0x7fc6d41a03f0 2021-04-09T18:16:46.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.291 7fc6ca7fc700 1 -- 172.21.15.32:0/2542953828 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc6cc007d00 con 0x7fc6d41a03f0 2021-04-09T18:16:46.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.469 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc6d410d480 con 0x7fc6d41a03f0 2021-04-09T18:16:46.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.470 7fc6ca7fc700 1 -- 172.21.15.32:0/2542953828 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v57) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc6cc007000 con 0x7fc6d41a03f0 2021-04-09T18:16:46.502 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:16:46.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6bc018900 msgr2=0x7fc6bc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6bc018900 0x7fc6bc01adc0 crc :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d41a03f0 msgr2=0x7fc6d4095240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d41a03f0 0x7fc6d4095240 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7fc6cc007f90 tx=0x7fc6cc00ac50).stop 2021-04-09T18:16:46.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 shutdown_connections 2021-04-09T18:16:46.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6bc018900 0x7fc6bc01adc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6d41a03f0 0x7fc6d4095240 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d418ad00 0x7fc6d419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 --2- 172.21.15.32:0/2542953828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d40a0a10 0x7fc6d40bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 >> 172.21.15.32:0/2542953828 conn(0x7fc6d40975f0 msgr2=0x7fc6d4096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 shutdown_connections 2021-04-09T18:16:46.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.472 7fc6dbb1b700 1 -- 172.21.15.32:0/2542953828 wait complete. 2021-04-09T18:16:46.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:16:46.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.640 7f2f860a3700 1 Processor -- start 2021-04-09T18:16:46.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.640 7f2f860a3700 1 -- start start 2021-04-09T18:16:46.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f860a3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8010d010 0x7f2f800d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f860a3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2f800d1da0 0x7f2f800d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f860a3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8010d480 con 0x7f2f800d82c0 2021-04-09T18:16:46.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f860a3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8010e780 con 0x7f2f800d1da0 2021-04-09T18:16:46.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f860a3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f800d8d30 con 0x7f2f8010d010 2021-04-09T18:16:46.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f7f7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8010d010 0x7f2f800d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f7effd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2f800d1da0 0x7f2f800d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f7ffff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2f800d82c0 0x7f2f8010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52418/0 (socket says 172.21.15.32:52418) 2021-04-09T18:16:46.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f7ffff700 1 -- 172.21.15.32:0/1973929646 learned_addr learned my addr 172.21.15.32:0/1973929646 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:46.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f7e7fc700 1 -- 172.21.15.32:0/1973929646 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198511609 0 0) 0x7f2f78001510 con 0x7f2f800d82c0 2021-04-09T18:16:46.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f7e7fc700 1 -- 172.21.15.32:0/1973929646 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f6c002fe0 con 0x7f2f800d82c0 2021-04-09T18:16:46.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.641 7f2f7effd700 1 -- 172.21.15.32:0/1973929646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8010d010 msgr2=0x7f2f800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:46.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f7effd700 1 --2- 172.21.15.32:0/1973929646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8010d010 0x7f2f800d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f7effd700 1 -- 172.21.15.32:0/1973929646 >> v1:172.21.15.86:6789/0 conn(0x7f2f800d82c0 legacy=0x7f2f8010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f7effd700 1 -- 172.21.15.32:0/1973929646 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f800d9220 con 0x7f2f800d1da0 2021-04-09T18:16:46.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f7f7fe700 1 --2- 172.21.15.32:0/1973929646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8010d010 0x7f2f800d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:46.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f7effd700 1 --2- 172.21.15.32:0/1973929646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f800d1da0 0x7f2f800d7ca0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f2f68006650 tx=0x7f2f68009a80).ready entity=mon.1 client_cookie=acbe4fff51f59cbd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:46.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f7e7fc700 1 -- 172.21.15.32:0/1973929646 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f680009b0 con 0x7f2f800d1da0 2021-04-09T18:16:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f7e7fc700 1 -- 172.21.15.32:0/1973929646 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6800c0c0 con 0x7f2f800d1da0 2021-04-09T18:16:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f7e7fc700 1 -- 172.21.15.32:0/1973929646 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f6800c510 con 0x7f2f800d1da0 2021-04-09T18:16:46.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f860a3700 1 -- 172.21.15.32:0/1973929646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f800d1da0 msgr2=0x7f2f800d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.642 7f2f860a3700 1 --2- 172.21.15.32:0/1973929646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f800d1da0 0x7f2f800d7ca0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f2f68006650 tx=0x7f2f68009a80).stop 2021-04-09T18:16:46.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.643 7f2f860a3700 1 -- 172.21.15.32:0/1973929646 shutdown_connections 2021-04-09T18:16:46.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.643 7f2f860a3700 1 --2- 172.21.15.32:0/1973929646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f800d1da0 0x7f2f800d7ca0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.643 7f2f860a3700 1 --2- 172.21.15.32:0/1973929646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8010d010 0x7f2f800d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.643 7f2f860a3700 1 -- 172.21.15.32:0/1973929646 >> 172.21.15.32:0/1973929646 conn(0x7f2f800ce5f0 msgr2=0x7f2f800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.643 7f2f860a3700 1 -- 172.21.15.32:0/1973929646 shutdown_connections 2021-04-09T18:16:46.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.643 7f2f860a3700 1 -- 172.21.15.32:0/1973929646 wait complete. 2021-04-09T18:16:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.663 7f2f860a3700 1 Processor -- start 2021-04-09T18:16:46.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.663 7f2f860a3700 1 -- start start 2021-04-09T18:16:46.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.663 7f2f860a3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f80193b40 0x7f2f800b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.663 7f2f860a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f800b75b0 0x7f2f80194520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.663 7f2f860a3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8019aee0 0x7f2f80194b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.663 7f2f860a3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f80195160 con 0x7f2f800b75b0 2021-04-09T18:16:46.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.663 7f2f860a3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f801936a0 con 0x7f2f80193b40 2021-04-09T18:16:46.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.663 7f2f860a3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8018d570 con 0x7f2f8019aee0 2021-04-09T18:16:46.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f800b75b0 0x7f2f80194520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f800b75b0 0x7f2f80194520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44908/0 (socket says 172.21.15.32:44908) 2021-04-09T18:16:46.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7effd700 1 -- 172.21.15.32:0/1909847220 learned_addr learned my addr 172.21.15.32:0/1909847220 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:46.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7f7fe700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f80193b40 0x7f2f800b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7ffff700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8019aee0 0x7f2f80194b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7effd700 1 -- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8019aee0 msgr2=0x7f2f80194b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7effd700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8019aee0 0x7f2f80194b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7effd700 1 -- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f80193b40 msgr2=0x7f2f800b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7effd700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f80193b40 0x7f2f800b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7effd700 1 -- 172.21.15.32:0/1909847220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8018d790 con 0x7f2f800b75b0 2021-04-09T18:16:46.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7f7fe700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f80193b40 0x7f2f800b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f7effd700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f800b75b0 0x7f2f80194520 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f2f68009650 tx=0x7f2f6800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:46.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f77fff700 1 -- 172.21.15.32:0/1909847220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f680093a0 con 0x7f2f800b75b0 2021-04-09T18:16:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f77fff700 1 -- 172.21.15.32:0/1909847220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f68006070 con 0x7f2f800b75b0 2021-04-09T18:16:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8018d790 con 0x7f2f800b75b0 2021-04-09T18:16:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f77fff700 1 -- 172.21.15.32:0/1909847220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f680052e0 con 0x7f2f800b75b0 2021-04-09T18:16:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.664 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8018d790 con 0x7f2f800b75b0 2021-04-09T18:16:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.665 7f2f77fff700 1 -- 172.21.15.32:0/1909847220 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2f68009300 con 0x7f2f800b75b0 2021-04-09T18:16:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.665 7f2f77fff700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f64018930 0x7f2f6401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.665 7f2f77fff700 1 -- 172.21.15.32:0/1909847220 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f2f680326e0 con 0x7f2f800b75b0 2021-04-09T18:16:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.665 7f2f7f7fe700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f64018930 0x7f2f6401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:46.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.665 7f2f7f7fe700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f64018930 0x7f2f6401ae10 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:46.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.685 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f80195160 con 0x7f2f800b75b0 2021-04-09T18:16:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.688 7f2f77fff700 1 -- 172.21.15.32:0/1909847220 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2f68006030 con 0x7f2f800b75b0 2021-04-09T18:16:46.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.864 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2f80195160 con 0x7f2f800b75b0 2021-04-09T18:16:46.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.864 7f2f77fff700 1 -- 172.21.15.32:0/1909847220 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v57) v1 ==== 84+0+64 (secure 0 0 0) 0x7f2f68000940 con 0x7f2f800b75b0 2021-04-09T18:16:46.896 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:46.897 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:16:46.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:16:46.897 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:16:46.897 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:16:46.898 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:46.898 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:16:46.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f64018930 msgr2=0x7f2f6401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f64018930 0x7f2f6401ae10 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f800b75b0 msgr2=0x7f2f80194520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:46.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f800b75b0 0x7f2f80194520 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f2f68009650 tx=0x7f2f6800b8e0).stop 2021-04-09T18:16:46.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 shutdown_connections 2021-04-09T18:16:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2f64018930 0x7f2f6401ae10 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f800b75b0 0x7f2f80194520 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f8019aee0 0x7f2f80194b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 --2- 172.21.15.32:0/1909847220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f80193b40 0x7f2f800b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:46.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 >> 172.21.15.32:0/1909847220 conn(0x7f2f801a23e0 msgr2=0x7f2f8019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:46.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 shutdown_connections 2021-04-09T18:16:46.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:46.867 7f2f860a3700 1 -- 172.21.15.32:0/1909847220 wait complete. 2021-04-09T18:16:46.914 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:16:46.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:16:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.056 7ffa03435700 1 Processor -- start 2021-04-09T18:16:47.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.056 7ffa03435700 1 -- start start 2021-04-09T18:16:47.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa03435700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0d6ca0 0x7ff9fc0d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa03435700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9fc0d2460 0x7ff9fc10cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa03435700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9fc0d72a0 con 0x7ff9fc0cffc0 2021-04-09T18:16:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa03435700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9fc0d70a0 con 0x7ff9fc0d2460 2021-04-09T18:16:47.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa03435700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc0d9200 con 0x7ff9fc0d6ca0 2021-04-09T18:16:47.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa01a52700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9fc0d2460 0x7ff9fc10cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:47.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa01a52700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9fc0d2460 0x7ff9fc10cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44916/0 (socket says 172.21.15.32:44916) 2021-04-09T18:16:47.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa01a52700 1 -- 172.21.15.32:0/4196090884 learned_addr learned my addr 172.21.15.32:0/4196090884 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa00a50700 1 --2- 172.21.15.32:0/4196090884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0d6ca0 0x7ff9fc0d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa01a52700 1 -- 172.21.15.32:0/4196090884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0d6ca0 msgr2=0x7ff9fc0d8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:47.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa01a52700 1 --2- 172.21.15.32:0/4196090884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0d6ca0 0x7ff9fc0d8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa01a52700 1 -- 172.21.15.32:0/4196090884 >> v1:172.21.15.86:6789/0 conn(0x7ff9fc0cffc0 legacy=0x7ff9fc0d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:47.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa01a52700 1 -- 172.21.15.32:0/4196090884 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9fc0d2860 con 0x7ff9fc0d2460 2021-04-09T18:16:47.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ff9f3fff700 1 -- 172.21.15.32:0/4196090884 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931911381 0 0) 0x7ff9f8000fe0 con 0x7ff9fc0cffc0 2021-04-09T18:16:47.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa01a52700 1 --2- 172.21.15.32:0/4196090884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0d2460 0x7ff9fc10cff0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7ff9f4006b80 tx=0x7ff9f4009b10).ready entity=mon.0 client_cookie=c5b6d64c36863143 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:47.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ff9f3fff700 1 -- 172.21.15.32:0/4196090884 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9f40022e0 con 0x7ff9fc0d2460 2021-04-09T18:16:47.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ff9f3fff700 1 -- 172.21.15.32:0/4196090884 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f4002620 con 0x7ff9fc0d2460 2021-04-09T18:16:47.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ff9f3fff700 1 -- 172.21.15.32:0/4196090884 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9f4002b00 con 0x7ff9fc0d2460 2021-04-09T18:16:47.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa03435700 1 -- 172.21.15.32:0/4196090884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0d2460 msgr2=0x7ff9fc10cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:47.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.057 7ffa03435700 1 --2- 172.21.15.32:0/4196090884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0d2460 0x7ff9fc10cff0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7ff9f4006b80 tx=0x7ff9f4009b10).stop 2021-04-09T18:16:47.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.058 7ffa03435700 1 -- 172.21.15.32:0/4196090884 shutdown_connections 2021-04-09T18:16:47.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.058 7ffa03435700 1 --2- 172.21.15.32:0/4196090884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0d2460 0x7ff9fc10cff0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.058 7ffa03435700 1 --2- 172.21.15.32:0/4196090884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0d6ca0 0x7ff9fc0d8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.058 7ffa03435700 1 -- 172.21.15.32:0/4196090884 >> 172.21.15.32:0/4196090884 conn(0x7ff9fc0ce5e0 msgr2=0x7ff9fc0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:47.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.058 7ffa03435700 1 -- 172.21.15.32:0/4196090884 shutdown_connections 2021-04-09T18:16:47.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.058 7ffa03435700 1 -- 172.21.15.32:0/4196090884 wait complete. 2021-04-09T18:16:47.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.078 7ffa03435700 1 Processor -- start 2021-04-09T18:16:47.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.078 7ffa03435700 1 -- start start 2021-04-09T18:16:47.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa03435700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9fc0b89a0 0x7ff9fc0c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:47.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa03435700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0b8da0 0x7ff9fc0be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:47.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa03435700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0b91a0 0x7ff9fc0bead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:47.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa03435700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc0c39b0 con 0x7ff9fc0b8da0 2021-04-09T18:16:47.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa03435700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc0c0d60 con 0x7ff9fc0b89a0 2021-04-09T18:16:47.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa03435700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc0bf0f0 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa00a50700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0b8da0 0x7ff9fc0be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:47.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa00a50700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0b8da0 0x7ff9fc0be4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44920/0 (socket says 172.21.15.32:44920) 2021-04-09T18:16:47.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa00a50700 1 -- 172.21.15.32:0/65244768 learned_addr learned my addr 172.21.15.32:0/65244768 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:47.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa01a52700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0b91a0 0x7ff9fc0bead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:47.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.079 7ffa01251700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9fc0b89a0 0x7ff9fc0c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:47.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ffa01a52700 1 -- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9fc0b89a0 msgr2=0x7ff9fc0c31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:47.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ffa01a52700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9fc0b89a0 0x7ff9fc0c31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ffa01a52700 1 -- 172.21.15.32:0/65244768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0b8da0 msgr2=0x7ff9fc0be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:47.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ffa01a52700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0b8da0 0x7ff9fc0be4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ffa01a52700 1 -- 172.21.15.32:0/65244768 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9fc0bf310 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ffa00a50700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0b8da0 0x7ff9fc0be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:47.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ffa01a52700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0b91a0 0x7ff9fc0bead0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff9f4006f30 tx=0x7ff9f400b840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:47.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ff9f1ffb700 1 -- 172.21.15.32:0/65244768 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9f4009400 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ff9f1ffb700 1 -- 172.21.15.32:0/65244768 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f4006240 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ffa03435700 1 -- 172.21.15.32:0/65244768 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9fc0bf310 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ff9f1ffb700 1 -- 172.21.15.32:0/65244768 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9f40054a0 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.080 7ffa03435700 1 -- 172.21.15.32:0/65244768 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9fc0bf310 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.081 7ff9f1ffb700 1 -- 172.21.15.32:0/65244768 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff9f40054a0 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.081 7ff9f1ffb700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9dc018930 0x7ff9dc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:47.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.081 7ff9f1ffb700 1 -- 172.21.15.32:0/65244768 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4757+0+0 (secure 0 0 0) 0x7ff9f4032630 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.081 7ffa01251700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9dc018930 0x7ff9dc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:47.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.081 7ffa01251700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9dc018930 0x7ff9dc01ae10 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:47.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.100 7ffa03435700 1 -- 172.21.15.32:0/65244768 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9fc1b70c0 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.105 7ff9f1ffb700 1 -- 172.21.15.32:0/65244768 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff9f4006030 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.288 7ffa03435700 1 -- 172.21.15.32:0/65244768 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff9fc1b70c0 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.859 7ff9f1ffb700 1 -- 172.21.15.32:0/65244768 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v58) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff9f40054a0 con 0x7ff9fc0b91a0 2021-04-09T18:16:47.934 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:16:47.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 -- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9dc018930 msgr2=0x7ff9dc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9dc018930 0x7ff9dc01ae10 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 -- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0b91a0 msgr2=0x7ff9fc0bead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0b91a0 0x7ff9fc0bead0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff9f4006f30 tx=0x7ff9f400b840).stop 2021-04-09T18:16:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 -- 172.21.15.32:0/65244768 shutdown_connections 2021-04-09T18:16:47.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9dc018930 0x7ff9dc01ae10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9fc0b8da0 0x7ff9fc0be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9fc0b91a0 0x7ff9fc0bead0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 --2- 172.21.15.32:0/65244768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9fc0b89a0 0x7ff9fc0c31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:47.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 -- 172.21.15.32:0/65244768 >> 172.21.15.32:0/65244768 conn(0x7ff9fc0a06a0 msgr2=0x7ff9fc08fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:47.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 -- 172.21.15.32:0/65244768 shutdown_connections 2021-04-09T18:16:47.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:47.906 7ffa03435700 1 -- 172.21.15.32:0/65244768 wait complete. 2021-04-09T18:16:47.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:16:48.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.076 7fe372665700 1 Processor -- start 2021-04-09T18:16:48.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.076 7fe372665700 1 -- start start 2021-04-09T18:16:48.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe372665700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe36c10d010 0x7fe36c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe372665700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe36c0d1da0 0x7fe36c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe372665700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0d82c0 0x7fe36c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe372665700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c0d8d30 con 0x7fe36c10d010 2021-04-09T18:16:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe372665700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c0d9220 con 0x7fe36c0d1da0 2021-04-09T18:16:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe372665700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c10d4b0 con 0x7fe36c0d82c0 2021-04-09T18:16:48.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe36c10d010 0x7fe36c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe36c10d010 0x7fe36c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44928/0 (socket says 172.21.15.32:44928) 2021-04-09T18:16:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36bfff700 1 -- 172.21.15.32:0/2662514337 learned_addr learned my addr 172.21.15.32:0/2662514337 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe363fff700 1 --2- 172.21.15.32:0/2662514337 >> v2:172.21.15.86:3300/0 conn(0x7fe36c0d1da0 0x7fe36c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe370c82700 1 --2- 172.21.15.32:0/2662514337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0d82c0 0x7fe36c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36bfff700 1 -- 172.21.15.32:0/2662514337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0d82c0 msgr2=0x7fe36c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:48.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36bfff700 1 --2- 172.21.15.32:0/2662514337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0d82c0 0x7fe36c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36bfff700 1 -- 172.21.15.32:0/2662514337 >> v2:172.21.15.86:3300/0 conn(0x7fe36c0d1da0 msgr2=0x7fe36c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:48.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36bfff700 1 --2- 172.21.15.32:0/2662514337 >> v2:172.21.15.86:3300/0 conn(0x7fe36c0d1da0 0x7fe36c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36bfff700 1 -- 172.21.15.32:0/2662514337 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0d86c0 con 0x7fe36c10d010 2021-04-09T18:16:48.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36bfff700 1 --2- 172.21.15.32:0/2662514337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c10d010 0x7fe36c0d1780 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7fe35c006700 tx=0x7fe35c009b00).ready entity=mon.0 client_cookie=1780a98da1f5cbd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:48.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.077 7fe36b7fe700 1 -- 172.21.15.32:0/2662514337 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35c00c0c0 con 0x7fe36c10d010 2021-04-09T18:16:48.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe36b7fe700 1 -- 172.21.15.32:0/2662514337 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c00c400 con 0x7fe36c10d010 2021-04-09T18:16:48.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe36b7fe700 1 -- 172.21.15.32:0/2662514337 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35c00c8e0 con 0x7fe36c10d010 2021-04-09T18:16:48.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe372665700 1 -- 172.21.15.32:0/2662514337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c10d010 msgr2=0x7fe36c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:48.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe372665700 1 --2- 172.21.15.32:0/2662514337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c10d010 0x7fe36c0d1780 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7fe35c006700 tx=0x7fe35c009b00).stop 2021-04-09T18:16:48.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe372665700 1 -- 172.21.15.32:0/2662514337 shutdown_connections 2021-04-09T18:16:48.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe372665700 1 --2- 172.21.15.32:0/2662514337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0d82c0 0x7fe36c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe372665700 1 --2- 172.21.15.32:0/2662514337 >> v2:172.21.15.86:3300/0 conn(0x7fe36c0d1da0 0x7fe36c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe372665700 1 --2- 172.21.15.32:0/2662514337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c10d010 0x7fe36c0d1780 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe372665700 1 -- 172.21.15.32:0/2662514337 >> 172.21.15.32:0/2662514337 conn(0x7fe36c0ce5f0 msgr2=0x7fe36c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:48.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe372665700 1 -- 172.21.15.32:0/2662514337 shutdown_connections 2021-04-09T18:16:48.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.078 7fe372665700 1 -- 172.21.15.32:0/2662514337 wait complete. 2021-04-09T18:16:48.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe372665700 1 Processor -- start 2021-04-09T18:16:48.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe372665700 1 -- start start 2021-04-09T18:16:48.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe372665700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c1a0050 0x7fe36c1a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe372665700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0bdcd0 0x7fe36c0bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe372665700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe36c0bf080 0x7fe36c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe372665700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0bf500 con 0x7fe36c1a0050 2021-04-09T18:16:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe372665700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0bb7b0 con 0x7fe36c0bf080 2021-04-09T18:16:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe372665700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0be0d0 con 0x7fe36c0bdcd0 2021-04-09T18:16:48.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe36bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c1a0050 0x7fe36c1a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe36bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c1a0050 0x7fe36c1a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44934/0 (socket says 172.21.15.32:44934) 2021-04-09T18:16:48.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe36bfff700 1 -- 172.21.15.32:0/3503931884 learned_addr learned my addr 172.21.15.32:0/3503931884 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:48.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.099 7fe370c82700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe36c0bf080 0x7fe36c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe363fff700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0bdcd0 0x7fe36c0bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe36bfff700 1 -- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0bdcd0 msgr2=0x7fe36c0bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:48.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe36bfff700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0bdcd0 0x7fe36c0bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe36bfff700 1 -- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe36c0bf080 msgr2=0x7fe36c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:48.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe36bfff700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe36c0bf080 0x7fe36c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe36bfff700 1 -- 172.21.15.32:0/3503931884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0b1310 con 0x7fe36c1a0050 2021-04-09T18:16:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe36bfff700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c1a0050 0x7fe36c1a0450 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7fe35c00b590 tx=0x7fe35c00b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe3697fa700 1 -- 172.21.15.32:0/3503931884 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35c006390 con 0x7fe36c1a0050 2021-04-09T18:16:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe3697fa700 1 -- 172.21.15.32:0/3503931884 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c00be20 con 0x7fe36c1a0050 2021-04-09T18:16:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe372665700 1 -- 172.21.15.32:0/3503931884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c0b1310 con 0x7fe36c1a0050 2021-04-09T18:16:48.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe3697fa700 1 -- 172.21.15.32:0/3503931884 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35c002030 con 0x7fe36c1a0050 2021-04-09T18:16:48.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.100 7fe372665700 1 -- 172.21.15.32:0/3503931884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c0b1310 con 0x7fe36c1a0050 2021-04-09T18:16:48.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.101 7fe3697fa700 1 -- 172.21.15.32:0/3503931884 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe35c0093e0 con 0x7fe36c1a0050 2021-04-09T18:16:48.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.101 7fe3697fa700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3500188c0 0x7fe35001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.101 7fe3697fa700 1 -- 172.21.15.32:0/3503931884 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5111+0+0 (secure 0 0 0) 0x7fe35c032610 con 0x7fe36c1a0050 2021-04-09T18:16:48.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.101 7fe363fff700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3500188c0 0x7fe35001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.101 7fe363fff700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3500188c0 0x7fe35001adc0 crc :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:48.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.120 7fe372665700 1 -- 172.21.15.32:0/3503931884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c0bb7b0 con 0x7fe36c1a0050 2021-04-09T18:16:48.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.124 7fe3697fa700 1 -- 172.21.15.32:0/3503931884 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe35c0090c0 con 0x7fe36c1a0050 2021-04-09T18:16:48.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.311 7fe372665700 1 -- 172.21.15.32:0/3503931884 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fe36c0bb7b0 con 0x7fe36c1a0050 2021-04-09T18:16:48.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.494 7fe3697fa700 1 -- 172.21.15.32:0/3503931884 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v59) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe35c0093e0 con 0x7fe36c1a0050 2021-04-09T18:16:48.562 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:16:48.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 -- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3500188c0 msgr2=0x7fe35001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:48.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3500188c0 0x7fe35001adc0 crc :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 -- 172.21.15.32:0/3503931884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c1a0050 msgr2=0x7fe36c1a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c1a0050 0x7fe36c1a0450 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7fe35c00b590 tx=0x7fe35c00b930).stop 2021-04-09T18:16:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 -- 172.21.15.32:0/3503931884 shutdown_connections 2021-04-09T18:16:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3500188c0 0x7fe35001adc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe36c0bf080 0x7fe36c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe36c0bdcd0 0x7fe36c0bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 --2- 172.21.15.32:0/3503931884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe36c1a0050 0x7fe36c1a0450 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 -- 172.21.15.32:0/3503931884 >> 172.21.15.32:0/3503931884 conn(0x7fe36c0975f0 msgr2=0x7fe36c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 -- 172.21.15.32:0/3503931884 shutdown_connections 2021-04-09T18:16:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.527 7fe372665700 1 -- 172.21.15.32:0/3503931884 wait complete. 2021-04-09T18:16:48.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:16:48.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.708 7fbc5dc34700 1 Processor -- start 2021-04-09T18:16:48.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.708 7fbc5dc34700 1 -- start start 2021-04-09T18:16:48.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc5dc34700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc5810d050 0x7fbc580cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc5dc34700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580d0160 0x7fbc580d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc5dc34700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5810d500 con 0x7fbc580d82e0 2021-04-09T18:16:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc5dc34700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc580d9260 con 0x7fbc5810d050 2021-04-09T18:16:48.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc5dc34700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc580d86e0 con 0x7fbc580d0160 2021-04-09T18:16:48.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc577fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc5810d050 0x7fbc580cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc577fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc5810d050 0x7fbc580cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44942/0 (socket says 172.21.15.32:44942) 2021-04-09T18:16:48.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc577fe700 1 -- 172.21.15.32:0/3395676656 learned_addr learned my addr 172.21.15.32:0/3395676656 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:48.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc56ffd700 1 --2- 172.21.15.32:0/3395676656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580d0160 0x7fbc580d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc577fe700 1 -- 172.21.15.32:0/3395676656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580d0160 msgr2=0x7fbc580d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc577fe700 1 --2- 172.21.15.32:0/3395676656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580d0160 0x7fbc580d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc577fe700 1 -- 172.21.15.32:0/3395676656 >> v1:172.21.15.86:6789/0 conn(0x7fbc580d82e0 legacy=0x7fbc5810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:48.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc577fe700 1 -- 172.21.15.32:0/3395676656 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5810e270 con 0x7fbc5810d050 2021-04-09T18:16:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.709 7fbc577fe700 1 --2- 172.21.15.32:0/3395676656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc5810d050 0x7fbc580cfb40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fbc48006700 tx=0x7fbc48009b80).ready entity=mon.0 client_cookie=d3b2f52745317d8f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc567fc700 1 -- 172.21.15.32:0/3395676656 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc4800c0c0 con 0x7fbc5810d050 2021-04-09T18:16:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc567fc700 1 -- 172.21.15.32:0/3395676656 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4800c440 con 0x7fbc5810d050 2021-04-09T18:16:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc567fc700 1 -- 172.21.15.32:0/3395676656 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc4800c920 con 0x7fbc5810d050 2021-04-09T18:16:48.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc5dc34700 1 -- 172.21.15.32:0/3395676656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc5810d050 msgr2=0x7fbc580cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc5dc34700 1 --2- 172.21.15.32:0/3395676656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc5810d050 0x7fbc580cfb40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fbc48006700 tx=0x7fbc48009b80).stop 2021-04-09T18:16:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc5dc34700 1 -- 172.21.15.32:0/3395676656 shutdown_connections 2021-04-09T18:16:48.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc5dc34700 1 --2- 172.21.15.32:0/3395676656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580d0160 0x7fbc580d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc5dc34700 1 --2- 172.21.15.32:0/3395676656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc5810d050 0x7fbc580cfb40 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc5dc34700 1 -- 172.21.15.32:0/3395676656 >> 172.21.15.32:0/3395676656 conn(0x7fbc580ce620 msgr2=0x7fbc580d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc5dc34700 1 -- 172.21.15.32:0/3395676656 shutdown_connections 2021-04-09T18:16:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.710 7fbc5dc34700 1 -- 172.21.15.32:0/3395676656 wait complete. 2021-04-09T18:16:48.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.730 7fbc5dc34700 1 Processor -- start 2021-04-09T18:16:48.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc5dc34700 1 -- start start 2021-04-09T18:16:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc5dc34700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580bc610 0x7fbc580bca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc5dc34700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580a5100 0x7fbc580a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc5dc34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc580a5b20 0x7fbc580b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc5dc34700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc580bd030 con 0x7fbc580a5b20 2021-04-09T18:16:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc5dc34700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc580b79a0 con 0x7fbc580bc610 2021-04-09T18:16:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc5dc34700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc580b7bc0 con 0x7fbc580a5100 2021-04-09T18:16:48.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc57fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc580a5b20 0x7fbc580b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc577fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580bc610 0x7fbc580bca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc577fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580bc610 0x7fbc580bca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:36706/0 (socket says 172.21.15.32:36706) 2021-04-09T18:16:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc57fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc580a5b20 0x7fbc580b7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44952/0 (socket says 172.21.15.32:44952) 2021-04-09T18:16:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc577fe700 1 -- 172.21.15.32:0/367176297 learned_addr learned my addr 172.21.15.32:0/367176297 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.731 7fbc56ffd700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580a5100 0x7fbc580a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:48.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc577fe700 1 -- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580a5100 msgr2=0x7fbc580a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:48.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc577fe700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580a5100 0x7fbc580a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc577fe700 1 -- 172.21.15.32:0/367176297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc580a5b20 msgr2=0x7fbc580b7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:48.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc577fe700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc580a5b20 0x7fbc580b7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:48.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc577fe700 1 -- 172.21.15.32:0/367176297 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc580b7de0 con 0x7fbc580bc610 2021-04-09T18:16:48.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc57fff700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc580a5b20 0x7fbc580b7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:48.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc577fe700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580bc610 0x7fbc580bca10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fbc4800b650 tx=0x7fbc4800b9f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:48.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc4ffff700 1 -- 172.21.15.32:0/367176297 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc4800bde0 con 0x7fbc580bc610 2021-04-09T18:16:48.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc4ffff700 1 -- 172.21.15.32:0/367176297 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc48006570 con 0x7fbc580bc610 2021-04-09T18:16:48.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc580b7de0 con 0x7fbc580bc610 2021-04-09T18:16:48.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc4ffff700 1 -- 172.21.15.32:0/367176297 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc48002220 con 0x7fbc580bc610 2021-04-09T18:16:48.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.732 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc580b7de0 con 0x7fbc580bc610 2021-04-09T18:16:48.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.733 7fbc4ffff700 1 -- 172.21.15.32:0/367176297 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbc48009f90 con 0x7fbc580bc610 2021-04-09T18:16:48.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.733 7fbc4ffff700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc3c018930 0x7fbc3c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.733 7fbc4ffff700 1 -- 172.21.15.32:0/367176297 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5461+0+0 (secure 0 0 0) 0x7fbc480329c0 con 0x7fbc580bc610 2021-04-09T18:16:49.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.733 7fbc56ffd700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc3c018930 0x7fbc3c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.733 7fbc56ffd700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc3c018930 0x7fbc3c01adc0 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.753 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc580bc0e0 con 0x7fbc580bc610 2021-04-09T18:16:49.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.757 7fbc4ffff700 1 -- 172.21.15.32:0/367176297 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbc48009230 con 0x7fbc580bc610 2021-04-09T18:16:49.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.942 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fbc580bc0e0 con 0x7fbc580bc610 2021-04-09T18:16:49.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.947 7fbc4ffff700 1 -- 172.21.15.32:0/367176297 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 10 and data pool 11 v53) v1 ==== 189+0+0 (secure 0 0 0) 0x7fbc48006570 con 0x7fbc580bc610 2021-04-09T18:16:49.042 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 10 and data pool 11 2021-04-09T18:16:49.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.953 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc3c018930 msgr2=0x7fbc3c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.953 7fbc5dc34700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc3c018930 0x7fbc3c01adc0 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.953 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580bc610 msgr2=0x7fbc580bca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.953 7fbc5dc34700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580bc610 0x7fbc580bca10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fbc4800b650 tx=0x7fbc4800b9f0).stop 2021-04-09T18:16:49.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.954 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 shutdown_connections 2021-04-09T18:16:49.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.954 7fbc5dc34700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc3c018930 0x7fbc3c01adc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.954 7fbc5dc34700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc580a5b20 0x7fbc580b7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.954 7fbc5dc34700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc580a5100 0x7fbc580a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.954 7fbc5dc34700 1 --2- 172.21.15.32:0/367176297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc580bc610 0x7fbc580bca10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.954 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 >> 172.21.15.32:0/367176297 conn(0x7fbc580c3870 msgr2=0x7fbc58099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:49.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.954 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 shutdown_connections 2021-04-09T18:16:49.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:48.954 7fbc5dc34700 1 -- 172.21.15.32:0/367176297 wait complete. 2021-04-09T18:16:49.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:16:49.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.194 7f01a4603700 1 Processor -- start 2021-04-09T18:16:49.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.194 7f01a4603700 1 -- start start 2021-04-09T18:16:49.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.195 7f01a4603700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f019c10d010 0x7f019c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.195 7f01a4603700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c0d1da0 0x7f019c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.195 7f01a4603700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019c10d480 con 0x7f019c0d82c0 2021-04-09T18:16:49.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.195 7f01a4603700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019c0d9220 con 0x7f019c10d010 2021-04-09T18:16:49.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.195 7f01a4603700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0d86c0 con 0x7f019c0d1da0 2021-04-09T18:16:49.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.195 7f01a241f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f019c10d010 0x7f019c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.195 7f01a241f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f019c10d010 0x7f019c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44960/0 (socket says 172.21.15.32:44960) 2021-04-09T18:16:49.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.195 7f01a241f700 1 -- 172.21.15.32:0/1741386949 learned_addr learned my addr 172.21.15.32:0/1741386949 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:49.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.195 7f01a1c1e700 1 --2- 172.21.15.32:0/1741386949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c0d1da0 0x7f019c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a241f700 1 -- 172.21.15.32:0/1741386949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c0d1da0 msgr2=0x7f019c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:49.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a241f700 1 --2- 172.21.15.32:0/1741386949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c0d1da0 0x7f019c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a241f700 1 -- 172.21.15.32:0/1741386949 >> v1:172.21.15.86:6789/0 conn(0x7f019c0d82c0 legacy=0x7f019c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a241f700 1 -- 172.21.15.32:0/1741386949 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c10e1f0 con 0x7f019c10d010 2021-04-09T18:16:49.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a141d700 1 -- 172.21.15.32:0/1741386949 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311850077 0 0) 0x7f01940009b0 con 0x7f019c0d82c0 2021-04-09T18:16:49.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a241f700 1 --2- 172.21.15.32:0/1741386949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c10d010 0x7f019c0d1780 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f0198006a80 tx=0x7f0198009e80).ready entity=mon.0 client_cookie=5fafe48e2ace6571 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:49.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a141d700 1 -- 172.21.15.32:0/1741386949 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f019800b510 con 0x7f019c10d010 2021-04-09T18:16:49.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a141d700 1 -- 172.21.15.32:0/1741386949 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019800b7d0 con 0x7f019c10d010 2021-04-09T18:16:49.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a141d700 1 -- 172.21.15.32:0/1741386949 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f019800bcb0 con 0x7f019c10d010 2021-04-09T18:16:49.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a4603700 1 -- 172.21.15.32:0/1741386949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c10d010 msgr2=0x7f019c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.196 7f01a4603700 1 --2- 172.21.15.32:0/1741386949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c10d010 0x7f019c0d1780 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f0198006a80 tx=0x7f0198009e80).stop 2021-04-09T18:16:49.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.197 7f01a4603700 1 -- 172.21.15.32:0/1741386949 shutdown_connections 2021-04-09T18:16:49.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.197 7f01a4603700 1 --2- 172.21.15.32:0/1741386949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c0d1da0 0x7f019c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.197 7f01a4603700 1 --2- 172.21.15.32:0/1741386949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c10d010 0x7f019c0d1780 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.197 7f01a4603700 1 -- 172.21.15.32:0/1741386949 >> 172.21.15.32:0/1741386949 conn(0x7f019c0ce5f0 msgr2=0x7f019c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:49.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.197 7f01a4603700 1 -- 172.21.15.32:0/1741386949 shutdown_connections 2021-04-09T18:16:49.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.197 7f01a4603700 1 -- 172.21.15.32:0/1741386949 wait complete. 2021-04-09T18:16:49.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.217 7f01a4603700 1 Processor -- start 2021-04-09T18:16:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.217 7f01a4603700 1 -- start start 2021-04-09T18:16:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.217 7f01a4603700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c179bf0 0x7f019c179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.217 7f01a4603700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c17c540 0x7f019c17c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a4603700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f019c0aca60 0x7f019c0ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a4603700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c17cf60 con 0x7f019c17c540 2021-04-09T18:16:49.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a4603700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0ad480 con 0x7f019c0aca60 2021-04-09T18:16:49.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a4603700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0ad6a0 con 0x7f019c179bf0 2021-04-09T18:16:49.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a1c1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c17c540 0x7f019c17c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a1c1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c17c540 0x7f019c17c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44966/0 (socket says 172.21.15.32:44966) 2021-04-09T18:16:49.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a1c1e700 1 -- 172.21.15.32:0/2858925790 learned_addr learned my addr 172.21.15.32:0/2858925790 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:49.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a241f700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c179bf0 0x7f019c179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a2c20700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f019c0aca60 0x7f019c0ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a1c1e700 1 -- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c179bf0 msgr2=0x7f019c179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a1c1e700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c179bf0 0x7f019c179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a1c1e700 1 -- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f019c0aca60 msgr2=0x7f019c0ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a1c1e700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f019c0aca60 0x7f019c0ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a1c1e700 1 -- 172.21.15.32:0/2858925790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c0ad8c0 con 0x7f019c17c540 2021-04-09T18:16:49.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a1c1e700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c17c540 0x7f019c17c940 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f018c006f60 tx=0x7f018c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:49.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f0192ffd700 1 -- 172.21.15.32:0/2858925790 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f018c00bd60 con 0x7f019c17c540 2021-04-09T18:16:49.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f0192ffd700 1 -- 172.21.15.32:0/2858925790 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018c000c90 con 0x7f019c17c540 2021-04-09T18:16:49.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f0192ffd700 1 -- 172.21.15.32:0/2858925790 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f018c002220 con 0x7f019c17c540 2021-04-09T18:16:49.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a4603700 1 -- 172.21.15.32:0/2858925790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019c0ad8c0 con 0x7f019c17c540 2021-04-09T18:16:49.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.218 7f01a4603700 1 -- 172.21.15.32:0/2858925790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019c0ad8c0 con 0x7f019c17c540 2021-04-09T18:16:49.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.219 7f0192ffd700 1 -- 172.21.15.32:0/2858925790 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f018c0052e0 con 0x7f019c17c540 2021-04-09T18:16:49.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.219 7f0192ffd700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01840188d0 0x7f018401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.219 7f0192ffd700 1 -- 172.21.15.32:0/2858925790 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f018c0329f0 con 0x7f019c17c540 2021-04-09T18:16:49.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.219 7f01a241f700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01840188d0 0x7f018401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.220 7f01a241f700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01840188d0 0x7f018401add0 crc :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:49.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.239 7f01a4603700 1 -- 172.21.15.32:0/2858925790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f019c17cf60 con 0x7f019c17c540 2021-04-09T18:16:49.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.243 7f0192ffd700 1 -- 172.21.15.32:0/2858925790 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f018c000c90 con 0x7f019c17c540 2021-04-09T18:16:49.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.431 7f01a4603700 1 -- 172.21.15.32:0/2858925790 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f019c17cf60 con 0x7f019c17c540 2021-04-09T18:16:49.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.432 7f0192ffd700 1 -- 172.21.15.32:0/2858925790 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f018c006b00 con 0x7f019c17c540 2021-04-09T18:16:49.464 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:49.464 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":63,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:16:48.944337","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:16:47.324034","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-04-09 18:16:48.346213","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:16:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.434 7f01a4603700 1 -- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01840188d0 msgr2=0x7f018401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.434 7f01a4603700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01840188d0 0x7f018401add0 crc :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.434 7f01a4603700 1 -- 172.21.15.32:0/2858925790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c17c540 msgr2=0x7f019c17c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.434 7f01a4603700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c17c540 0x7f019c17c940 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f018c006f60 tx=0x7f018c00b970).stop 2021-04-09T18:16:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.435 7f01a4603700 1 -- 172.21.15.32:0/2858925790 shutdown_connections 2021-04-09T18:16:49.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.435 7f01a4603700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01840188d0 0x7f018401add0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.435 7f01a4603700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019c17c540 0x7f019c17c940 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.435 7f01a4603700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f019c0aca60 0x7f019c0ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.435 7f01a4603700 1 --2- 172.21.15.32:0/2858925790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019c179bf0 0x7f019c179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.435 7f01a4603700 1 -- 172.21.15.32:0/2858925790 >> 172.21.15.32:0/2858925790 conn(0x7f019c1acae0 msgr2=0x7f019c17ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:49.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.435 7f01a4603700 1 -- 172.21.15.32:0/2858925790 shutdown_connections 2021-04-09T18:16:49.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.435 7f01a4603700 1 -- 172.21.15.32:0/2858925790 wait complete. 2021-04-09T18:16:49.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:16:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.595 7ff994051700 1 Processor -- start 2021-04-09T18:16:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.595 7ff994051700 1 -- start start 2021-04-09T18:16:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff994051700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff98c0d7ca0 0x7ff98c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff994051700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c10da00 0x7ff98c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff994051700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98c10d480 con 0x7ff98c10d010 2021-04-09T18:16:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff994051700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98c0d9220 con 0x7ff98c0d7ca0 2021-04-09T18:16:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff994051700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c0d86c0 con 0x7ff98c10da00 2021-04-09T18:16:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99166c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff98c0d7ca0 0x7ff98c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99166c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff98c0d7ca0 0x7ff98c0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44978/0 (socket says 172.21.15.32:44978) 2021-04-09T18:16:49.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99166c700 1 -- 172.21.15.32:0/1742167490 learned_addr learned my addr 172.21.15.32:0/1742167490 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:49.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99266e700 1 --2- 172.21.15.32:0/1742167490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c10da00 0x7ff98c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99166c700 1 -- 172.21.15.32:0/1742167490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c10da00 msgr2=0x7ff98c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99166c700 1 --2- 172.21.15.32:0/1742167490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c10da00 0x7ff98c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99166c700 1 -- 172.21.15.32:0/1742167490 >> v1:172.21.15.86:6789/0 conn(0x7ff98c10d010 legacy=0x7ff98c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99166c700 1 -- 172.21.15.32:0/1742167490 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c0d1f70 con 0x7ff98c0d7ca0 2021-04-09T18:16:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99266e700 1 --2- 172.21.15.32:0/1742167490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c10da00 0x7ff98c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff99166c700 1 --2- 172.21.15.32:0/1742167490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c0d7ca0 0x7ff98c0d80a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7ff97c006700 tx=0x7ff97c009b80).ready entity=mon.0 client_cookie=48ebdbc545a4a757 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.596 7ff990e6b700 1 -- 172.21.15.32:0/1742167490 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff97c00c0c0 con 0x7ff98c0d7ca0 2021-04-09T18:16:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff990e6b700 1 -- 172.21.15.32:0/1742167490 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c00c440 con 0x7ff98c0d7ca0 2021-04-09T18:16:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff990e6b700 1 -- 172.21.15.32:0/1742167490 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff97c00c920 con 0x7ff98c0d7ca0 2021-04-09T18:16:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff994051700 1 -- 172.21.15.32:0/1742167490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c0d7ca0 msgr2=0x7ff98c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff994051700 1 --2- 172.21.15.32:0/1742167490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c0d7ca0 0x7ff98c0d80a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7ff97c006700 tx=0x7ff97c009b80).stop 2021-04-09T18:16:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff994051700 1 -- 172.21.15.32:0/1742167490 shutdown_connections 2021-04-09T18:16:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff994051700 1 --2- 172.21.15.32:0/1742167490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c10da00 0x7ff98c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff994051700 1 --2- 172.21.15.32:0/1742167490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c0d7ca0 0x7ff98c0d80a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff994051700 1 -- 172.21.15.32:0/1742167490 >> 172.21.15.32:0/1742167490 conn(0x7ff98c0ce5f0 msgr2=0x7ff98c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:49.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff994051700 1 -- 172.21.15.32:0/1742167490 shutdown_connections 2021-04-09T18:16:49.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.597 7ff994051700 1 -- 172.21.15.32:0/1742167490 wait complete. 2021-04-09T18:16:49.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff994051700 1 Processor -- start 2021-04-09T18:16:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff994051700 1 -- start start 2021-04-09T18:16:49.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff994051700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c1a5210 0x7ff98c1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff994051700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c1a6340 0x7ff98c1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff994051700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff98c0abaa0 0x7ff98c0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff994051700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c1a6d60 con 0x7ff98c1a6340 2021-04-09T18:16:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff994051700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c0ac4c0 con 0x7ff98c0abaa0 2021-04-09T18:16:49.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff994051700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c0ac6e0 con 0x7ff98c1a5210 2021-04-09T18:16:49.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff99166c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c1a6340 0x7ff98c1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff99166c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c1a6340 0x7ff98c1a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44982/0 (socket says 172.21.15.32:44982) 2021-04-09T18:16:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.618 7ff99166c700 1 -- 172.21.15.32:0/2288209101 learned_addr learned my addr 172.21.15.32:0/2288209101 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff99266e700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff98c0abaa0 0x7ff98c0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff991e6d700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c1a5210 0x7ff98c1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff99166c700 1 -- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c1a5210 msgr2=0x7ff98c1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff99166c700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c1a5210 0x7ff98c1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff99166c700 1 -- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff98c0abaa0 msgr2=0x7ff98c0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff99166c700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff98c0abaa0 0x7ff98c0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff99166c700 1 -- 172.21.15.32:0/2288209101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c17a720 con 0x7ff98c1a6340 2021-04-09T18:16:49.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff99266e700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff98c0abaa0 0x7ff98c0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:49.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff99166c700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c1a6340 0x7ff98c1a6740 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7ff97c009f90 tx=0x7ff97c00b860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff9827fc700 1 -- 172.21.15.32:0/2288209101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff97c000c90 con 0x7ff98c1a6340 2021-04-09T18:16:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff9827fc700 1 -- 172.21.15.32:0/2288209101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c00bc50 con 0x7ff98c1a6340 2021-04-09T18:16:49.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff994051700 1 -- 172.21.15.32:0/2288209101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98c17a9c0 con 0x7ff98c1a6340 2021-04-09T18:16:49.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff9827fc700 1 -- 172.21.15.32:0/2288209101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff97c002220 con 0x7ff98c1a6340 2021-04-09T18:16:49.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.619 7ff994051700 1 -- 172.21.15.32:0/2288209101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98c17acb0 con 0x7ff98c1a6340 2021-04-09T18:16:49.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.620 7ff9827fc700 1 -- 172.21.15.32:0/2288209101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff97c002020 con 0x7ff98c1a6340 2021-04-09T18:16:49.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.620 7ff9827fc700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff974018930 0x7ff97401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:49.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.620 7ff9827fc700 1 -- 172.21.15.32:0/2288209101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff97c034c30 con 0x7ff98c1a6340 2021-04-09T18:16:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.620 7ff991e6d700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff974018930 0x7ff97401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.620 7ff991e6d700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff974018930 0x7ff97401ae10 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.640 7ff994051700 1 -- 172.21.15.32:0/2288209101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98c1b7070 con 0x7ff98c1a6340 2021-04-09T18:16:49.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.643 7ff9827fc700 1 -- 172.21.15.32:0/2288209101 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff97c0091e0 con 0x7ff98c1a6340 2021-04-09T18:16:49.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.834 7ff994051700 1 -- 172.21.15.32:0/2288209101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98c10d480 con 0x7ff98c1a6340 2021-04-09T18:16:49.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.835 7ff9827fc700 1 -- 172.21.15.32:0/2288209101 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+13996 (secure 0 0 0) 0x7ff97c0091e0 con 0x7ff98c1a6340 2021-04-09T18:16:49.892 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:49.893 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":63,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:16:48.944337","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:16:47.324034","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-04-09 18:16:48.346213","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:16:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 -- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff974018930 msgr2=0x7ff97401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff974018930 0x7ff97401ae10 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 -- 172.21.15.32:0/2288209101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c1a6340 msgr2=0x7ff98c1a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:49.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c1a6340 0x7ff98c1a6740 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7ff97c009f90 tx=0x7ff97c00b860).stop 2021-04-09T18:16:49.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 -- 172.21.15.32:0/2288209101 shutdown_connections 2021-04-09T18:16:49.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff974018930 0x7ff97401ae10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff98c1a6340 0x7ff98c1a6740 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff98c0abaa0 0x7ff98c0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 --2- 172.21.15.32:0/2288209101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff98c1a5210 0x7ff98c1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:49.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 -- 172.21.15.32:0/2288209101 >> 172.21.15.32:0/2288209101 conn(0x7ff98c1a99c0 msgr2=0x7ff98c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:49.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 -- 172.21.15.32:0/2288209101 shutdown_connections 2021-04-09T18:16:49.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:49.837 7ff994051700 1 -- 172.21.15.32:0/2288209101 wait complete. 2021-04-09T18:16:49.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:16:50.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.093 7f6f9d1d4700 1 Processor -- start 2021-04-09T18:16:50.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.093 7f6f9d1d4700 1 -- start start 2021-04-09T18:16:50.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.094 7f6f9d1d4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980d7ca0 0x7f6f980d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:50.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.094 7f6f9d1d4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9810d480 con 0x7f6f9810da00 2021-04-09T18:16:50.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.094 7f6f9d1d4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f980d86c0 con 0x7f6f9810d010 2021-04-09T18:16:50.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.094 7f6f9d1d4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980d9220 con 0x7f6f980d7ca0 2021-04-09T18:16:50.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.094 7f6f9759e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6f9810da00 0x7f6f9810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54580/0 (socket says 172.21.15.32:54580) 2021-04-09T18:16:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.094 7f6f9759e700 1 -- 172.21.15.32:0/2439323812 learned_addr learned my addr 172.21.15.32:0/2439323812 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:50.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.094 7f6f9659c700 1 --2- 172.21.15.32:0/2439323812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980d7ca0 0x7f6f980d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:50.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f95d9b700 1 -- 172.21.15.32:0/2439323812 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717646713 0 0) 0x7f6f90000d30 con 0x7f6f9810da00 2021-04-09T18:16:50.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f95d9b700 1 -- 172.21.15.32:0/2439323812 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f80002fe0 con 0x7f6f9810da00 2021-04-09T18:16:50.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f95d9b700 1 -- 172.21.15.32:0/2439323812 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429546938 0 0) 0x7f6f88000c60 con 0x7f6f9810d010 2021-04-09T18:16:50.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f95d9b700 1 -- 172.21.15.32:0/2439323812 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f800042c0 con 0x7f6f9810d010 2021-04-09T18:16:50.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f9659c700 1 -- 172.21.15.32:0/2439323812 >> v1:172.21.15.86:6789/0 conn(0x7f6f9810d010 legacy=0x7f6f980d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:50.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f9659c700 1 -- 172.21.15.32:0/2439323812 >> v1:172.21.15.32:6789/0 conn(0x7f6f9810da00 legacy=0x7f6f9810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:50.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f9659c700 1 -- 172.21.15.32:0/2439323812 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980d1f70 con 0x7f6f980d7ca0 2021-04-09T18:16:50.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f9659c700 1 --2- 172.21.15.32:0/2439323812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980d7ca0 0x7f6f980d80a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6f8c006650 tx=0x7f6f8c009a80).ready entity=mon.2 client_cookie=d16e7a311279cc7a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:50.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f95d9b700 1 -- 172.21.15.32:0/2439323812 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f8c0009b0 con 0x7f6f980d7ca0 2021-04-09T18:16:50.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f95d9b700 1 -- 172.21.15.32:0/2439323812 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c00c0c0 con 0x7f6f980d7ca0 2021-04-09T18:16:50.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f95d9b700 1 -- 172.21.15.32:0/2439323812 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f8c00c510 con 0x7f6f980d7ca0 2021-04-09T18:16:50.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f9d1d4700 1 -- 172.21.15.32:0/2439323812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980d7ca0 msgr2=0x7f6f980d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:50.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.095 7f6f9d1d4700 1 --2- 172.21.15.32:0/2439323812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980d7ca0 0x7f6f980d80a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6f8c006650 tx=0x7f6f8c009a80).stop 2021-04-09T18:16:50.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.096 7f6f9d1d4700 1 -- 172.21.15.32:0/2439323812 shutdown_connections 2021-04-09T18:16:50.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.096 7f6f9d1d4700 1 --2- 172.21.15.32:0/2439323812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980d7ca0 0x7f6f980d80a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:50.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.096 7f6f9d1d4700 1 -- 172.21.15.32:0/2439323812 >> 172.21.15.32:0/2439323812 conn(0x7f6f980ce5f0 msgr2=0x7f6f980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:50.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.096 7f6f9d1d4700 1 -- 172.21.15.32:0/2439323812 shutdown_connections 2021-04-09T18:16:50.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.096 7f6f9d1d4700 1 -- 172.21.15.32:0/2439323812 wait complete. 2021-04-09T18:16:50.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.116 7f6f9d1d4700 1 Processor -- start 2021-04-09T18:16:50.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.116 7f6f9d1d4700 1 -- start start 2021-04-09T18:16:50.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f9d1d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f98092ad0 0x7f6f98091d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:50.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f9d1d4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f980a2300 0x7f6f980aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:50.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f9d1d4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980befd0 0x7f6f980bf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:50.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f9d1d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980a2700 con 0x7f6f98092ad0 2021-04-09T18:16:50.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f9d1d4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f98092550 con 0x7f6f980a2300 2021-04-09T18:16:50.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f9d1d4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980aacb0 con 0x7f6f980befd0 2021-04-09T18:16:50.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f96d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f98092ad0 0x7f6f98091d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:50.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f96d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f98092ad0 0x7f6f98091d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44998/0 (socket says 172.21.15.32:44998) 2021-04-09T18:16:50.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f96d9d700 1 -- 172.21.15.32:0/3371972019 learned_addr learned my addr 172.21.15.32:0/3371972019 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:50.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f9759e700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980befd0 0x7f6f980bf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:50.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f9659c700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f980a2300 0x7f6f980aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:50.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f96d9d700 1 -- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980befd0 msgr2=0x7f6f980bf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:50.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f96d9d700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980befd0 0x7f6f980bf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:50.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f96d9d700 1 -- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f980a2300 msgr2=0x7f6f980aa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:50.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f96d9d700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f980a2300 0x7f6f980aa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:50.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f96d9d700 1 -- 172.21.15.32:0/3371972019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980bf9f0 con 0x7f6f98092ad0 2021-04-09T18:16:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.117 7f6f96d9d700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f98092ad0 0x7f6f98091d50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f6f88006f10 tx=0x7f6f8800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.118 7f6f877fe700 1 -- 172.21.15.32:0/3371972019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f8800bd80 con 0x7f6f98092ad0 2021-04-09T18:16:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.118 7f6f877fe700 1 -- 172.21.15.32:0/3371972019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f88000e20 con 0x7f6f98092ad0 2021-04-09T18:16:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.118 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f980bf9f0 con 0x7f6f98092ad0 2021-04-09T18:16:50.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.118 7f6f877fe700 1 -- 172.21.15.32:0/3371972019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f880054f0 con 0x7f6f98092ad0 2021-04-09T18:16:50.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.118 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f980bf9f0 con 0x7f6f98092ad0 2021-04-09T18:16:50.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.118 7f6f877fe700 1 -- 172.21.15.32:0/3371972019 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6f88009fe0 con 0x7f6f98092ad0 2021-04-09T18:16:50.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.118 7f6f877fe700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f7c018930 0x7f6f7c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:50.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.118 7f6f877fe700 1 -- 172.21.15.32:0/3371972019 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6f880329e0 con 0x7f6f98092ad0 2021-04-09T18:16:50.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.119 7f6f9659c700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f7c018930 0x7f6f7c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:50.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.119 7f6f9659c700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f7c018930 0x7f6f7c01adc0 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:50.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.138 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f981b7050 con 0x7f6f98092ad0 2021-04-09T18:16:50.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.142 7f6f877fe700 1 -- 172.21.15.32:0/3371972019 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6f88000e20 con 0x7f6f98092ad0 2021-04-09T18:16:50.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.335 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f6f9810d480 con 0x7f6f98092ad0 2021-04-09T18:16:50.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:50.953 7f6f877fe700 1 -- 172.21.15.32:0/3371972019 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6f88009fe0 con 0x7f6f98092ad0 2021-04-09T18:16:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f7c018930 msgr2=0x7f6f7c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f7c018930 0x7f6f7c01adc0 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f98092ad0 msgr2=0x7f6f98091d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f98092ad0 0x7f6f98091d50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f6f88006f10 tx=0x7f6f8800b990).stop 2021-04-09T18:16:51.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 shutdown_connections 2021-04-09T18:16:51.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f7c018930 0x7f6f7c01adc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f980befd0 0x7f6f980bf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f980a2300 0x7f6f980aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 --2- 172.21.15.32:0/3371972019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f98092ad0 0x7f6f98091d50 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 >> 172.21.15.32:0/3371972019 conn(0x7f6f98093070 msgr2=0x7f6f980a3150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:51.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 shutdown_connections 2021-04-09T18:16:51.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.003 7f6f9d1d4700 1 -- 172.21.15.32:0/3371972019 wait complete. 2021-04-09T18:16:51.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:16:51.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.196 7fb6407ad700 1 Processor -- start 2021-04-09T18:16:51.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.196 7fb6407ad700 1 -- start start 2021-04-09T18:16:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.196 7fb6407ad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63810d010 0x7fb6380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.196 7fb6407ad700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb6380d1da0 0x7fb6380d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.196 7fb6407ad700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb6380d82c0 0x7fb63810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.196 7fb6407ad700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb63810e780 con 0x7fb6380d82c0 2021-04-09T18:16:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.196 7fb6407ad700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6380d8d30 con 0x7fb6380d1da0 2021-04-09T18:16:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.196 7fb6407ad700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6380d9220 con 0x7fb63810d010 2021-04-09T18:16:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63edca700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb6380d82c0 0x7fb63810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63edca700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb6380d82c0 0x7fb63810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45006/0 (socket says 172.21.15.32:45006) 2021-04-09T18:16:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63edca700 1 -- 172.21.15.32:0/598055303 learned_addr learned my addr 172.21.15.32:0/598055303 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63e5c9700 1 --2- 172.21.15.32:0/598055303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63810d010 0x7fb6380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63ddc8700 1 --2- 172.21.15.32:0/598055303 >> v2:172.21.15.86:3300/0 conn(0x7fb6380d1da0 0x7fb6380d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63edca700 1 -- 172.21.15.32:0/598055303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63810d010 msgr2=0x7fb6380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63edca700 1 --2- 172.21.15.32:0/598055303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63810d010 0x7fb6380d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63edca700 1 -- 172.21.15.32:0/598055303 >> v2:172.21.15.86:3300/0 conn(0x7fb6380d1da0 msgr2=0x7fb6380d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:51.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63edca700 1 --2- 172.21.15.32:0/598055303 >> v2:172.21.15.86:3300/0 conn(0x7fb6380d1da0 0x7fb6380d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63edca700 1 -- 172.21.15.32:0/598055303 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63810d480 con 0x7fb6380d82c0 2021-04-09T18:16:51.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63edca700 1 --2- 172.21.15.32:0/598055303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6380d82c0 0x7fb63810da00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7fb630006f00 tx=0x7fb630009e90).ready entity=mon.0 client_cookie=ebfed94520dce127 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:51.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63d5c7700 1 -- 172.21.15.32:0/598055303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb63000b300 con 0x7fb6380d82c0 2021-04-09T18:16:51.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63d5c7700 1 -- 172.21.15.32:0/598055303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63000b640 con 0x7fb6380d82c0 2021-04-09T18:16:51.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb63d5c7700 1 -- 172.21.15.32:0/598055303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb63000bb20 con 0x7fb6380d82c0 2021-04-09T18:16:51.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb6407ad700 1 -- 172.21.15.32:0/598055303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6380d82c0 msgr2=0x7fb63810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.197 7fb6407ad700 1 --2- 172.21.15.32:0/598055303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6380d82c0 0x7fb63810da00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7fb630006f00 tx=0x7fb630009e90).stop 2021-04-09T18:16:51.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.198 7fb6407ad700 1 -- 172.21.15.32:0/598055303 shutdown_connections 2021-04-09T18:16:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.198 7fb6407ad700 1 --2- 172.21.15.32:0/598055303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6380d82c0 0x7fb63810da00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.198 7fb6407ad700 1 --2- 172.21.15.32:0/598055303 >> v2:172.21.15.86:3300/0 conn(0x7fb6380d1da0 0x7fb6380d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.198 7fb6407ad700 1 --2- 172.21.15.32:0/598055303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63810d010 0x7fb6380d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.198 7fb6407ad700 1 -- 172.21.15.32:0/598055303 >> 172.21.15.32:0/598055303 conn(0x7fb6380ce5f0 msgr2=0x7fb6380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.198 7fb6407ad700 1 -- 172.21.15.32:0/598055303 shutdown_connections 2021-04-09T18:16:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.198 7fb6407ad700 1 -- 172.21.15.32:0/598055303 wait complete. 2021-04-09T18:16:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.218 7fb6407ad700 1 Processor -- start 2021-04-09T18:16:51.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.218 7fb6407ad700 1 -- start start 2021-04-09T18:16:51.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb6407ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381aca20 0x7fb6381ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb6407ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63818bd40 0x7fb63818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb6407ad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63808fba0 0x7fb63808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb6407ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63818c7e0 con 0x7fb6381aca20 2021-04-09T18:16:51.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb6407ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63818ca30 con 0x7fb63818bd40 2021-04-09T18:16:51.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb6407ad700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6380905f0 con 0x7fb63808fba0 2021-04-09T18:16:51.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63e5c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381aca20 0x7fb6381ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63e5c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381aca20 0x7fb6381ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45008/0 (socket says 172.21.15.32:45008) 2021-04-09T18:16:51.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63e5c9700 1 -- 172.21.15.32:0/243342803 learned_addr learned my addr 172.21.15.32:0/243342803 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:51.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63ddc8700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63818bd40 0x7fb63818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63edca700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63808fba0 0x7fb63808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63e5c9700 1 -- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63808fba0 msgr2=0x7fb63808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63e5c9700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63808fba0 0x7fb63808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63e5c9700 1 -- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63818bd40 msgr2=0x7fb63818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63e5c9700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63818bd40 0x7fb63818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63e5c9700 1 -- 172.21.15.32:0/243342803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6380b4300 con 0x7fb6381aca20 2021-04-09T18:16:51.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.219 7fb63e5c9700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381aca20 0x7fb6381ace20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7fb634006be0 tx=0x7fb63400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:51.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.220 7fb62effd700 1 -- 172.21.15.32:0/243342803 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6340064d0 con 0x7fb6381aca20 2021-04-09T18:16:51.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.220 7fb62effd700 1 -- 172.21.15.32:0/243342803 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63400bd80 con 0x7fb6381aca20 2021-04-09T18:16:51.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.220 7fb6407ad700 1 -- 172.21.15.32:0/243342803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6380b4300 con 0x7fb6381aca20 2021-04-09T18:16:51.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.220 7fb62effd700 1 -- 172.21.15.32:0/243342803 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb634002030 con 0x7fb6381aca20 2021-04-09T18:16:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.220 7fb6407ad700 1 -- 172.21.15.32:0/243342803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6380b4300 con 0x7fb6381aca20 2021-04-09T18:16:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.220 7fb62effd700 1 -- 172.21.15.32:0/243342803 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb634006c90 con 0x7fb6381aca20 2021-04-09T18:16:51.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.220 7fb62effd700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6200188d0 0x7fb62001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.220 7fb62effd700 1 -- 172.21.15.32:0/243342803 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb634032a10 con 0x7fb6381aca20 2021-04-09T18:16:51.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.221 7fb63ddc8700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6200188d0 0x7fb62001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.221 7fb63ddc8700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6200188d0 0x7fb62001add0 crc :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:51.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.240 7fb6407ad700 1 -- 172.21.15.32:0/243342803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb638182d70 con 0x7fb6381aca20 2021-04-09T18:16:51.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.244 7fb62effd700 1 -- 172.21.15.32:0/243342803 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb634006820 con 0x7fb6381aca20 2021-04-09T18:16:51.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.435 7fb6407ad700 1 -- 172.21.15.32:0/243342803 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb63818c760 con 0x7fb6381aca20 2021-04-09T18:16:51.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.435 7fb62effd700 1 -- 172.21.15.32:0/243342803 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2778 (secure 0 0 0) 0x7fb63400bd80 con 0x7fb6381aca20 2021-04-09T18:16:51.471 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 56 2021-04-09T18:16:51.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:51.473 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5880,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:16:50.949963","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:16:51.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 -- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6200188d0 msgr2=0x7fb62001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6200188d0 0x7fb62001add0 crc :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 -- 172.21.15.32:0/243342803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381aca20 msgr2=0x7fb6381ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381aca20 0x7fb6381ace20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7fb634006be0 tx=0x7fb63400b990).stop 2021-04-09T18:16:51.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 -- 172.21.15.32:0/243342803 shutdown_connections 2021-04-09T18:16:51.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb6200188d0 0x7fb62001add0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63808fba0 0x7fb63808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb63818bd40 0x7fb63818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 --2- 172.21.15.32:0/243342803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381aca20 0x7fb6381ace20 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 -- 172.21.15.32:0/243342803 >> 172.21.15.32:0/243342803 conn(0x7fb63819b460 msgr2=0x7fb63819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:51.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 -- 172.21.15.32:0/243342803 shutdown_connections 2021-04-09T18:16:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.438 7fb6407ad700 1 -- 172.21.15.32:0/243342803 wait complete. 2021-04-09T18:16:51.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:16:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.655 7f05a17e6700 1 Processor -- start 2021-04-09T18:16:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.655 7f05a17e6700 1 -- start start 2021-04-09T18:16:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.655 7f05a17e6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c0d7ca0 0x7f059c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.655 7f05a17e6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f059c10da00 0x7f059c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.655 7f05a17e6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c10d480 con 0x7f059c10d010 2021-04-09T18:16:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.655 7f05a17e6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059c0d9220 con 0x7f059c10da00 2021-04-09T18:16:51.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.655 7f05a17e6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c0d86c0 con 0x7f059c0d7ca0 2021-04-09T18:16:51.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.655 7f059bd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f059c10da00 0x7f059c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059bd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f059c10da00 0x7f059c10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45016/0 (socket says 172.21.15.32:45016) 2021-04-09T18:16:51.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059ad73700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c0d7ca0 0x7f059c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059bd75700 1 -- 172.21.15.32:0/2414511860 learned_addr learned my addr 172.21.15.32:0/2414511860 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:51.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059bd75700 1 -- 172.21.15.32:0/2414511860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c0d7ca0 msgr2=0x7f059c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:51.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059bd75700 1 --2- 172.21.15.32:0/2414511860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c0d7ca0 0x7f059c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059bd75700 1 -- 172.21.15.32:0/2414511860 >> v1:172.21.15.86:6789/0 conn(0x7f059c10d010 legacy=0x7f059c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059bd75700 1 -- 172.21.15.32:0/2414511860 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c0d1f70 con 0x7f059c10da00 2021-04-09T18:16:51.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059a572700 1 -- 172.21.15.32:0/2414511860 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592944019 0 0) 0x7f058c001080 con 0x7f059c10d010 2021-04-09T18:16:51.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059ad73700 1 --2- 172.21.15.32:0/2414511860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c0d7ca0 0x7f059c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:51.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059bd75700 1 --2- 172.21.15.32:0/2414511860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c10da00 0x7f059c10de00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f0594006b80 tx=0x7f0594009b90).ready entity=mon.0 client_cookie=545ef65ffb87b4e4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:51.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059a572700 1 -- 172.21.15.32:0/2414511860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05940022e0 con 0x7f059c10da00 2021-04-09T18:16:51.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059a572700 1 -- 172.21.15.32:0/2414511860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0594002620 con 0x7f059c10da00 2021-04-09T18:16:51.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f059a572700 1 -- 172.21.15.32:0/2414511860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0594002b00 con 0x7f059c10da00 2021-04-09T18:16:51.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f05a17e6700 1 -- 172.21.15.32:0/2414511860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c10da00 msgr2=0x7f059c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.656 7f05a17e6700 1 --2- 172.21.15.32:0/2414511860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c10da00 0x7f059c10de00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f0594006b80 tx=0x7f0594009b90).stop 2021-04-09T18:16:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.657 7f05a17e6700 1 -- 172.21.15.32:0/2414511860 shutdown_connections 2021-04-09T18:16:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.657 7f05a17e6700 1 --2- 172.21.15.32:0/2414511860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c10da00 0x7f059c10de00 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.657 7f05a17e6700 1 --2- 172.21.15.32:0/2414511860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c0d7ca0 0x7f059c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.657 7f05a17e6700 1 -- 172.21.15.32:0/2414511860 >> 172.21.15.32:0/2414511860 conn(0x7f059c0ce5f0 msgr2=0x7f059c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:51.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.657 7f05a17e6700 1 -- 172.21.15.32:0/2414511860 shutdown_connections 2021-04-09T18:16:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.657 7f05a17e6700 1 -- 172.21.15.32:0/2414511860 wait complete. 2021-04-09T18:16:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.677 7f05a17e6700 1 Processor -- start 2021-04-09T18:16:51.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.677 7f05a17e6700 1 -- start start 2021-04-09T18:16:51.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f05a17e6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059c0b5360 0x7f059c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f05a17e6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c18e140 0x7f059c1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f05a17e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c1ad040 0x7f059c08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f05a17e6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c1a5360 con 0x7f059c1ad040 2021-04-09T18:16:51.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f05a17e6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c0902c0 con 0x7f059c0b5360 2021-04-09T18:16:51.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f05a17e6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c0904e0 con 0x7f059c18e140 2021-04-09T18:16:51.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059bd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c1ad040 0x7f059c08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059bd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c1ad040 0x7f059c08fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45024/0 (socket says 172.21.15.32:45024) 2021-04-09T18:16:51.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059bd75700 1 -- 172.21.15.32:0/3403652336 learned_addr learned my addr 172.21.15.32:0/3403652336 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059ad73700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c18e140 0x7f059c1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059b574700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059c0b5360 0x7f059c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059bd75700 1 -- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c18e140 msgr2=0x7f059c1aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059bd75700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c18e140 0x7f059c1aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059bd75700 1 -- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059c0b5360 msgr2=0x7f059c1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059bd75700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059c0b5360 0x7f059c1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059bd75700 1 -- 172.21.15.32:0/3403652336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c183c80 con 0x7f059c1ad040 2021-04-09T18:16:51.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059ad73700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c18e140 0x7f059c1aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:16:51.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059b574700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059c0b5360 0x7f059c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:16:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.678 7f059bd75700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c1ad040 0x7f059c08fca0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f0594006f10 tx=0x7f059400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.679 7f0583fff700 1 -- 172.21.15.32:0/3403652336 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0594000b90 con 0x7f059c1ad040 2021-04-09T18:16:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.679 7f0583fff700 1 -- 172.21.15.32:0/3403652336 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059400bd50 con 0x7f059c1ad040 2021-04-09T18:16:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.679 7f0583fff700 1 -- 172.21.15.32:0/3403652336 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0594002220 con 0x7f059c1ad040 2021-04-09T18:16:51.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.679 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059c183c80 con 0x7f059c1ad040 2021-04-09T18:16:51.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.679 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c183c80 con 0x7f059c1ad040 2021-04-09T18:16:51.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.679 7f0583fff700 1 -- 172.21.15.32:0/3403652336 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0594009100 con 0x7f059c1ad040 2021-04-09T18:16:51.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.679 7f0583fff700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f057c0188d0 0x7f057c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.679 7f0583fff700 1 -- 172.21.15.32:0/3403652336 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f05940329e0 con 0x7f059c1ad040 2021-04-09T18:16:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.680 7f059b574700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f057c0188d0 0x7f057c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.680 7f059b574700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f057c0188d0 0x7f057c01add0 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:51.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.700 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059c182d70 con 0x7f059c1ad040 2021-04-09T18:16:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.703 7f0583fff700 1 -- 172.21.15.32:0/3403652336 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0594000b90 con 0x7f059c1ad040 2021-04-09T18:16:51.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.890 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f059c182d70 con 0x7f059c1ad040 2021-04-09T18:16:51.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.890 7f0583fff700 1 -- 172.21.15.32:0/3403652336 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f0594000ef0 con 0x7f059c1ad040 2021-04-09T18:16:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:51.923 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":63,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:16:48.944337","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:16:47.324034","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-04-09 18:16:48.346213","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:16:51.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.892 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f057c0188d0 msgr2=0x7f057c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.892 7f05a17e6700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f057c0188d0 0x7f057c01add0 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.892 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c1ad040 msgr2=0x7f059c08fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:51.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.892 7f05a17e6700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c1ad040 0x7f059c08fca0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f0594006f10 tx=0x7f059400b960).stop 2021-04-09T18:16:51.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.892 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 shutdown_connections 2021-04-09T18:16:51.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.892 7f05a17e6700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f057c0188d0 0x7f057c01add0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.892 7f05a17e6700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f059c1ad040 0x7f059c08fca0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.893 7f05a17e6700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f059c18e140 0x7f059c1aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.893 7f05a17e6700 1 --2- 172.21.15.32:0/3403652336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059c0b5360 0x7f059c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:51.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.893 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 >> 172.21.15.32:0/3403652336 conn(0x7f059c19b460 msgr2=0x7f059c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:51.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.893 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 shutdown_connections 2021-04-09T18:16:51.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:51.893 7f05a17e6700 1 -- 172.21.15.32:0/3403652336 wait complete. 2021-04-09T18:16:51.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:16:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.091 7fccf997a700 1 Processor -- start 2021-04-09T18:16:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf997a700 1 -- start start 2021-04-09T18:16:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf997a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fccf40cffb0 0x7fccf4087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf997a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40d8050 0x7fccf410d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf997a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf4088460 con 0x7fccf40d7c50 2021-04-09T18:16:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf997a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf41b0c80 con 0x7fccf40cffb0 2021-04-09T18:16:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf997a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf40d91f0 con 0x7fccf40d8050 2021-04-09T18:16:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf3574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fccf40cffb0 0x7fccf4087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf3574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fccf40cffb0 0x7fccf4087e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45028/0 (socket says 172.21.15.32:45028) 2021-04-09T18:16:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf3574700 1 -- 172.21.15.32:0/1470155500 learned_addr learned my addr 172.21.15.32:0/1470155500 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.092 7fccf3d75700 1 --2- 172.21.15.32:0/1470155500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40d8050 0x7fccf410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf3574700 1 -- 172.21.15.32:0/1470155500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40d8050 msgr2=0x7fccf410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf3574700 1 --2- 172.21.15.32:0/1470155500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40d8050 0x7fccf410d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf3574700 1 -- 172.21.15.32:0/1470155500 >> v1:172.21.15.86:6789/0 conn(0x7fccf40d7c50 legacy=0x7fccf41b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf3574700 1 -- 172.21.15.32:0/1470155500 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf40d8620 con 0x7fccf40cffb0 2021-04-09T18:16:52.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf2572700 1 -- 172.21.15.32:0/1470155500 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297602416 0 0) 0x7fcce8000c60 con 0x7fccf40d7c50 2021-04-09T18:16:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf3574700 1 --2- 172.21.15.32:0/1470155500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf40cffb0 0x7fccf4087e40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fcce4006700 tx=0x7fcce4009b00).ready entity=mon.0 client_cookie=b8c38cbb798d6b69 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf2572700 1 -- 172.21.15.32:0/1470155500 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcce400c0c0 con 0x7fccf40cffb0 2021-04-09T18:16:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf2572700 1 -- 172.21.15.32:0/1470155500 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce400c400 con 0x7fccf40cffb0 2021-04-09T18:16:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf2572700 1 -- 172.21.15.32:0/1470155500 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcce40009b0 con 0x7fccf40cffb0 2021-04-09T18:16:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf997a700 1 -- 172.21.15.32:0/1470155500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf40cffb0 msgr2=0x7fccf4087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.093 7fccf997a700 1 --2- 172.21.15.32:0/1470155500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf40cffb0 0x7fccf4087e40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fcce4006700 tx=0x7fcce4009b00).stop 2021-04-09T18:16:52.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.094 7fccf997a700 1 -- 172.21.15.32:0/1470155500 shutdown_connections 2021-04-09T18:16:52.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.094 7fccf997a700 1 --2- 172.21.15.32:0/1470155500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40d8050 0x7fccf410d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.094 7fccf997a700 1 --2- 172.21.15.32:0/1470155500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf40cffb0 0x7fccf4087e40 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.094 7fccf997a700 1 -- 172.21.15.32:0/1470155500 >> 172.21.15.32:0/1470155500 conn(0x7fccf40ce5b0 msgr2=0x7fccf40cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.094 7fccf997a700 1 -- 172.21.15.32:0/1470155500 shutdown_connections 2021-04-09T18:16:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.094 7fccf997a700 1 -- 172.21.15.32:0/1470155500 wait complete. 2021-04-09T18:16:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.114 7fccf997a700 1 Processor -- start 2021-04-09T18:16:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.114 7fccf997a700 1 -- start start 2021-04-09T18:16:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf997a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf41920c0 0x7fccf40b8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf997a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40c0b80 0x7fccf40c0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf997a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf40c15a0 0x7fccf41a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf997a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf40bf0e0 con 0x7fccf41920c0 2021-04-09T18:16:52.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf997a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf41924c0 con 0x7fccf40c15a0 2021-04-09T18:16:52.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf997a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf40a64a0 con 0x7fccf40c0b80 2021-04-09T18:16:52.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf41920c0 0x7fccf40b8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf41920c0 0x7fccf40b8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45034/0 (socket says 172.21.15.32:45034) 2021-04-09T18:16:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3574700 1 -- 172.21.15.32:0/1976158135 learned_addr learned my addr 172.21.15.32:0/1976158135 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3d75700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf40c15a0 0x7fccf41a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf2d73700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40c0b80 0x7fccf40c0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3574700 1 -- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40c0b80 msgr2=0x7fccf40c0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3574700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40c0b80 0x7fccf40c0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3574700 1 -- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf40c15a0 msgr2=0x7fccf41a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3574700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf40c15a0 0x7fccf41a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3574700 1 -- 172.21.15.32:0/1976158135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf40b8e40 con 0x7fccf41920c0 2021-04-09T18:16:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf2d73700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40c0b80 0x7fccf40c0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:52.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf3574700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf41920c0 0x7fccf40b8820 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fcce4006f20 tx=0x7fcce400b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:52.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccdbfff700 1 -- 172.21.15.32:0/1976158135 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcce400bdd0 con 0x7fccf41920c0 2021-04-09T18:16:52.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccdbfff700 1 -- 172.21.15.32:0/1976158135 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce4006440 con 0x7fccf41920c0 2021-04-09T18:16:52.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccdbfff700 1 -- 172.21.15.32:0/1976158135 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcce4005560 con 0x7fccf41920c0 2021-04-09T18:16:52.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf997a700 1 -- 172.21.15.32:0/1976158135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf40b8e40 con 0x7fccf41920c0 2021-04-09T18:16:52.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.115 7fccf997a700 1 -- 172.21.15.32:0/1976158135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf40b8e40 con 0x7fccf41920c0 2021-04-09T18:16:52.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.116 7fccdbfff700 1 -- 172.21.15.32:0/1976158135 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcce40052e0 con 0x7fccf41920c0 2021-04-09T18:16:52.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.116 7fccdbfff700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccd40188c0 0x7fccd401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.116 7fccdbfff700 1 -- 172.21.15.32:0/1976158135 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcce40327d0 con 0x7fccf41920c0 2021-04-09T18:16:52.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.116 7fccf2d73700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccd40188c0 0x7fccd401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.117 7fccf2d73700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccd40188c0 0x7fccd401adc0 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:52.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.137 7fccf997a700 1 -- 172.21.15.32:0/1976158135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf4093fc0 con 0x7fccf41920c0 2021-04-09T18:16:52.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.140 7fccdbfff700 1 -- 172.21.15.32:0/1976158135 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcce4025f20 con 0x7fccf41920c0 2021-04-09T18:16:52.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.326 7fccf997a700 1 -- 172.21.15.32:0/1976158135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fccf40d8620 con 0x7fccf41920c0 2021-04-09T18:16:52.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.329 7fccdbfff700 1 -- 172.21.15.32:0/1976158135 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v20) v1 ==== 184+0+0 (secure 0 0 0) 0x7fcce40052e0 con 0x7fccf41920c0 2021-04-09T18:16:52.359 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:16:52.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 -- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccd40188c0 msgr2=0x7fccd401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccd40188c0 0x7fccd401adc0 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 -- 172.21.15.32:0/1976158135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf41920c0 msgr2=0x7fccf40b8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf41920c0 0x7fccf40b8820 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fcce4006f20 tx=0x7fcce400b9e0).stop 2021-04-09T18:16:52.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 -- 172.21.15.32:0/1976158135 shutdown_connections 2021-04-09T18:16:52.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccd40188c0 0x7fccd401adc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf40c15a0 0x7fccf41a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf40c0b80 0x7fccf40c0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 --2- 172.21.15.32:0/1976158135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf41920c0 0x7fccf40b8820 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 -- 172.21.15.32:0/1976158135 >> 172.21.15.32:0/1976158135 conn(0x7fccf40adaf0 msgr2=0x7fccf40bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:52.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 -- 172.21.15.32:0/1976158135 shutdown_connections 2021-04-09T18:16:52.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.331 7fccf997a700 1 -- 172.21.15.32:0/1976158135 wait complete. 2021-04-09T18:16:52.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:16:52.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.503 7f68bf52b700 1 Processor -- start 2021-04-09T18:16:52.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.503 7f68bf52b700 1 -- start start 2021-04-09T18:16:52.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bf52b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b80d7c50 0x7f68b81b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bf52b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f68b80d8050 0x7f68b810d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bf52b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b80d8620 con 0x7f68b80cffb0 2021-04-09T18:16:52.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bf52b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b80d91f0 con 0x7f68b80d8050 2021-04-09T18:16:52.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bf52b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80d2800 con 0x7f68b80d7c50 2021-04-09T18:16:52.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bdb48700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f68b80d8050 0x7f68b810d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bd347700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f68b80cffb0 0x7f68b8087e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52558/0 (socket says 172.21.15.32:52558) 2021-04-09T18:16:52.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bd347700 1 -- 172.21.15.32:0/2318017274 learned_addr learned my addr 172.21.15.32:0/2318017274 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:52.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bcb46700 1 --2- 172.21.15.32:0/2318017274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b80d7c50 0x7f68b81b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bdb48700 1 -- 172.21.15.32:0/2318017274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b80d7c50 msgr2=0x7f68b81b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:52.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bdb48700 1 --2- 172.21.15.32:0/2318017274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b80d7c50 0x7f68b81b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bdb48700 1 -- 172.21.15.32:0/2318017274 >> v1:172.21.15.86:6789/0 conn(0x7f68b80cffb0 legacy=0x7f68b8087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bdb48700 1 -- 172.21.15.32:0/2318017274 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b81b0ab0 con 0x7f68b80d8050 2021-04-09T18:16:52.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68affff700 1 -- 172.21.15.32:0/2318017274 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517524312 0 0) 0x7f68b40010c0 con 0x7f68b80cffb0 2021-04-09T18:16:52.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.504 7f68bcb46700 1 --2- 172.21.15.32:0/2318017274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b80d7c50 0x7f68b81b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:52.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.505 7f68bdb48700 1 --2- 172.21.15.32:0/2318017274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68b80d8050 0x7f68b810d200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f68b0006e50 tx=0x7f68b0009e70).ready entity=mon.1 client_cookie=8698480ed4e6954b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:52.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.505 7f68affff700 1 -- 172.21.15.32:0/2318017274 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68b0006a20 con 0x7f68b80d8050 2021-04-09T18:16:52.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.505 7f68affff700 1 -- 172.21.15.32:0/2318017274 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b0006790 con 0x7f68b80d8050 2021-04-09T18:16:52.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.505 7f68affff700 1 -- 172.21.15.32:0/2318017274 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68b000b700 con 0x7f68b80d8050 2021-04-09T18:16:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.505 7f68bf52b700 1 -- 172.21.15.32:0/2318017274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68b80d8050 msgr2=0x7f68b810d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.505 7f68bf52b700 1 --2- 172.21.15.32:0/2318017274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68b80d8050 0x7f68b810d200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f68b0006e50 tx=0x7f68b0009e70).stop 2021-04-09T18:16:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.506 7f68bf52b700 1 -- 172.21.15.32:0/2318017274 shutdown_connections 2021-04-09T18:16:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.506 7f68bf52b700 1 --2- 172.21.15.32:0/2318017274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68b80d8050 0x7f68b810d200 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.506 7f68bf52b700 1 --2- 172.21.15.32:0/2318017274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b80d7c50 0x7f68b81b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.506 7f68bf52b700 1 -- 172.21.15.32:0/2318017274 >> 172.21.15.32:0/2318017274 conn(0x7f68b80ce5b0 msgr2=0x7f68b80cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.506 7f68bf52b700 1 -- 172.21.15.32:0/2318017274 shutdown_connections 2021-04-09T18:16:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.506 7f68bf52b700 1 -- 172.21.15.32:0/2318017274 wait complete. 2021-04-09T18:16:52.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.526 7f68bf52b700 1 Processor -- start 2021-04-09T18:16:52.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.526 7f68bf52b700 1 -- start start 2021-04-09T18:16:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bf52b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68b809d680 0x7f68b809afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bf52b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b809da80 0x7f68b8191dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bf52b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b81923f0 0x7f68b809be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bf52b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b809cbe0 con 0x7f68b809da80 2021-04-09T18:16:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bf52b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b809b8e0 con 0x7f68b809d680 2021-04-09T18:16:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bf52b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b809b5c0 con 0x7f68b81923f0 2021-04-09T18:16:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bcb46700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b809da80 0x7f68b8191dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bcb46700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b809da80 0x7f68b8191dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45048/0 (socket says 172.21.15.32:45048) 2021-04-09T18:16:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bcb46700 1 -- 172.21.15.32:0/3571720528 learned_addr learned my addr 172.21.15.32:0/3571720528 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bdb48700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b81923f0 0x7f68b809be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bd347700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68b809d680 0x7f68b809afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bcb46700 1 -- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b81923f0 msgr2=0x7f68b809be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bcb46700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b81923f0 0x7f68b809be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bcb46700 1 -- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68b809d680 msgr2=0x7f68b809afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bcb46700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68b809d680 0x7f68b809afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bcb46700 1 -- 172.21.15.32:0/3571720528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b809c480 con 0x7f68b809da80 2021-04-09T18:16:52.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bcb46700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b809da80 0x7f68b8191dd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f68a8006f40 tx=0x7f68a800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:52.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68adffb700 1 -- 172.21.15.32:0/3571720528 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68a800bd80 con 0x7f68b809da80 2021-04-09T18:16:52.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68adffb700 1 -- 172.21.15.32:0/3571720528 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a8000c90 con 0x7f68b809da80 2021-04-09T18:16:52.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b809c480 con 0x7f68b809da80 2021-04-09T18:16:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68adffb700 1 -- 172.21.15.32:0/3571720528 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68a8002220 con 0x7f68b809da80 2021-04-09T18:16:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.527 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b809c480 con 0x7f68b809da80 2021-04-09T18:16:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.528 7f68adffb700 1 -- 172.21.15.32:0/3571720528 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f68a800a000 con 0x7f68b809da80 2021-04-09T18:16:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.528 7f68adffb700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6898018930 0x7f689801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.528 7f68adffb700 1 -- 172.21.15.32:0/3571720528 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f68a80329e0 con 0x7f68b809da80 2021-04-09T18:16:52.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.528 7f68bd347700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6898018930 0x7f689801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.529 7f68bd347700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6898018930 0x7f689801ae10 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:52.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.548 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b8093fc0 con 0x7f68b809da80 2021-04-09T18:16:52.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.551 7f68adffb700 1 -- 172.21.15.32:0/3571720528 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f68a8000c90 con 0x7f68b809da80 2021-04-09T18:16:52.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.736 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f68b8093fc0 con 0x7f68b809da80 2021-04-09T18:16:52.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.739 7f68adffb700 1 -- 172.21.15.32:0/3571720528 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v21) v1 ==== 184+0+0 (secure 0 0 0) 0x7f68a8006b00 con 0x7f68b809da80 2021-04-09T18:16:52.774 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:16:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6898018930 msgr2=0x7f689801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6898018930 0x7f689801ae10 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b809da80 msgr2=0x7f68b8191dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b809da80 0x7f68b8191dd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f68a8006f40 tx=0x7f68a800b990).stop 2021-04-09T18:16:52.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 shutdown_connections 2021-04-09T18:16:52.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6898018930 0x7f689801ae10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68b809da80 0x7f68b8191dd0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68b81923f0 0x7f68b809be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 --2- 172.21.15.32:0/3571720528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68b809d680 0x7f68b809afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 >> 172.21.15.32:0/3571720528 conn(0x7f68b80adaf0 msgr2=0x7f68b80bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 shutdown_connections 2021-04-09T18:16:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.745 7f68bf52b700 1 -- 172.21.15.32:0/3571720528 wait complete. 2021-04-09T18:16:52.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:16:52.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.909 7f123aaa2700 1 Processor -- start 2021-04-09T18:16:52.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.909 7f123aaa2700 1 -- start start 2021-04-09T18:16:52.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f123aaa2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f123410d010 0x7f12340d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f123aaa2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f12340d1da0 0x7f12340d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:52.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f123aaa2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123410d480 con 0x7f12340d82c0 2021-04-09T18:16:52.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f123aaa2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f123410e780 con 0x7f12340d1da0 2021-04-09T18:16:52.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f123aaa2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12340d8d30 con 0x7f123410d010 2021-04-09T18:16:52.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f1233fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f12340d1da0 0x7f12340d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f12388be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f123410d010 0x7f12340d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:52.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f12390bf700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f12340d82c0 0x7f123410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52574/0 (socket says 172.21.15.32:52574) 2021-04-09T18:16:52.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f12390bf700 1 -- 172.21.15.32:0/395050554 learned_addr learned my addr 172.21.15.32:0/395050554 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f12388be700 1 -- 172.21.15.32:0/395050554 >> v2:172.21.15.86:3300/0 conn(0x7f12340d1da0 msgr2=0x7f12340d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f12388be700 1 --2- 172.21.15.32:0/395050554 >> v2:172.21.15.86:3300/0 conn(0x7f12340d1da0 0x7f12340d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.911 7f12388be700 1 -- 172.21.15.32:0/395050554 >> v1:172.21.15.86:6789/0 conn(0x7f12340d82c0 legacy=0x7f123410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.911 7f12388be700 1 -- 172.21.15.32:0/395050554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12340d9220 con 0x7f123410d010 2021-04-09T18:16:52.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.910 7f12337fe700 1 -- 172.21.15.32:0/395050554 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292467742 0 0) 0x7f122c001440 con 0x7f12340d82c0 2021-04-09T18:16:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.911 7f1233fff700 1 --2- 172.21.15.32:0/395050554 >> v2:172.21.15.86:3300/0 conn(0x7f12340d1da0 0x7f12340d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.911 7f12388be700 1 --2- 172.21.15.32:0/395050554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f123410d010 0x7f12340d1780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1224006650 tx=0x7f1224009a80).ready entity=mon.2 client_cookie=315d0ba7e1dd0c7f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.911 7f12337fe700 1 -- 172.21.15.32:0/395050554 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12240009b0 con 0x7f123410d010 2021-04-09T18:16:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.911 7f12337fe700 1 -- 172.21.15.32:0/395050554 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122400c0c0 con 0x7f123410d010 2021-04-09T18:16:52.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.911 7f12337fe700 1 -- 172.21.15.32:0/395050554 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f122400c510 con 0x7f123410d010 2021-04-09T18:16:52.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.911 7f123aaa2700 1 -- 172.21.15.32:0/395050554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f123410d010 msgr2=0x7f12340d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:52.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.911 7f123aaa2700 1 --2- 172.21.15.32:0/395050554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f123410d010 0x7f12340d1780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1224006650 tx=0x7f1224009a80).stop 2021-04-09T18:16:52.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.912 7f123aaa2700 1 -- 172.21.15.32:0/395050554 shutdown_connections 2021-04-09T18:16:52.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.912 7f123aaa2700 1 --2- 172.21.15.32:0/395050554 >> v2:172.21.15.86:3300/0 conn(0x7f12340d1da0 0x7f12340d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.912 7f123aaa2700 1 --2- 172.21.15.32:0/395050554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f123410d010 0x7f12340d1780 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.912 7f123aaa2700 1 -- 172.21.15.32:0/395050554 >> 172.21.15.32:0/395050554 conn(0x7f12340ce5f0 msgr2=0x7f12340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.912 7f123aaa2700 1 -- 172.21.15.32:0/395050554 shutdown_connections 2021-04-09T18:16:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.912 7f123aaa2700 1 -- 172.21.15.32:0/395050554 wait complete. 2021-04-09T18:16:52.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.932 7f123aaa2700 1 Processor -- start 2021-04-09T18:16:52.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.932 7f123aaa2700 1 -- start start 2021-04-09T18:16:53.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.932 7f123aaa2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12341aca20 0x7f12341ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:53.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.932 7f123aaa2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f123418bd40 0x7f123418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:53.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.932 7f123aaa2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f123408fba0 0x7f123408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:53.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.932 7f123aaa2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123418c7e0 con 0x7f123418bd40 2021-04-09T18:16:53.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.932 7f123aaa2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123418ca30 con 0x7f123408fba0 2021-04-09T18:16:53.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.932 7f123aaa2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12340905f0 con 0x7f12341aca20 2021-04-09T18:16:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f1233fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f123418bd40 0x7f123418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f1233fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f123418bd40 0x7f123418c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45064/0 (socket says 172.21.15.32:45064) 2021-04-09T18:16:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f1233fff700 1 -- 172.21.15.32:0/1250787155 learned_addr learned my addr 172.21.15.32:0/1250787155 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f12390bf700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f123408fba0 0x7f123408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f12388be700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12341aca20 0x7f12341ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f1233fff700 1 -- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12341aca20 msgr2=0x7f12341ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:53.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f1233fff700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12341aca20 0x7f12341ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:53.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f1233fff700 1 -- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f123408fba0 msgr2=0x7f123408ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f1233fff700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f123408fba0 0x7f123408ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f1233fff700 1 -- 172.21.15.32:0/1250787155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12340b4300 con 0x7f123418bd40 2021-04-09T18:16:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f1233fff700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f123418bd40 0x7f123418c140 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f1228006f40 tx=0x7f122800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f12317fa700 1 -- 172.21.15.32:0/1250787155 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f122800bd80 con 0x7f123418bd40 2021-04-09T18:16:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f12317fa700 1 -- 172.21.15.32:0/1250787155 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1228000c90 con 0x7f123418bd40 2021-04-09T18:16:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12340b4300 con 0x7f123418bd40 2021-04-09T18:16:53.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f12317fa700 1 -- 172.21.15.32:0/1250787155 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1228002220 con 0x7f123418bd40 2021-04-09T18:16:53.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.933 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12340b4300 con 0x7f123418bd40 2021-04-09T18:16:53.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.934 7f12317fa700 1 -- 172.21.15.32:0/1250787155 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f122800a000 con 0x7f123418bd40 2021-04-09T18:16:53.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.934 7f12317fa700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1214018930 0x7f121401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:53.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.934 7f12317fa700 1 -- 172.21.15.32:0/1250787155 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1228032a10 con 0x7f123418bd40 2021-04-09T18:16:53.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.934 7f12388be700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1214018930 0x7f121401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:53.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.935 7f12388be700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1214018930 0x7f121401ae10 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:53.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.954 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1234182d70 con 0x7f123418bd40 2021-04-09T18:16:53.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:52.959 7f12317fa700 1 -- 172.21.15.32:0/1250787155 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1228006b00 con 0x7f123418bd40 2021-04-09T18:16:53.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.149 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f123418c760 con 0x7f123418bd40 2021-04-09T18:16:53.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.150 7f12317fa700 1 -- 172.21.15.32:0/1250787155 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2778 (secure 0 0 0) 0x7f122800a000 con 0x7f123418bd40 2021-04-09T18:16:53.186 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 56 2021-04-09T18:16:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:53.188 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5880,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:16:50.949963","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:16:53.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.152 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1214018930 msgr2=0x7f121401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:53.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.152 7f123aaa2700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1214018930 0x7f121401ae10 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:53.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.152 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f123418bd40 msgr2=0x7f123418c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:53.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.152 7f123aaa2700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f123418bd40 0x7f123418c140 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f1228006f40 tx=0x7f122800b990).stop 2021-04-09T18:16:53.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.152 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 shutdown_connections 2021-04-09T18:16:53.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.153 7f123aaa2700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1214018930 0x7f121401ae10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:53.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.153 7f123aaa2700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f123418bd40 0x7f123418c140 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:53.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.153 7f123aaa2700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f123408fba0 0x7f123408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:53.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.153 7f123aaa2700 1 --2- 172.21.15.32:0/1250787155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12341aca20 0x7f12341ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:53.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.153 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 >> 172.21.15.32:0/1250787155 conn(0x7f123419b460 msgr2=0x7f123419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:53.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.153 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 shutdown_connections 2021-04-09T18:16:53.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.153 7f123aaa2700 1 -- 172.21.15.32:0/1250787155 wait complete. 2021-04-09T18:16:53.606 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:16:50.949963', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:16:53.606 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:16:53.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:16:53.737 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:16:53.738 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:16:53.738 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:16:53.739 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 5803, 2021-04-09T18:16:53.739 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:16:53.739 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:16:53.740 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.7607310600000003, 2021-04-09T18:16:53.740 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 56, 2021-04-09T18:16:53.741 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 63, 2021-04-09T18:16:53.741 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 63, 2021-04-09T18:16:53.741 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 9.2176195189999994 2021-04-09T18:16:53.742 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:16:53.746 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5803, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.7607310600000003, "mdsmap_epoch": 56, "osdmap_epoch": 63, "osdmap_epoch_barrier": 63, "uptime": 9.2176195189999994 } 2021-04-09T18:16:53.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:16:53.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.878 7fa1875c3700 1 Processor -- start 2021-04-09T18:16:53.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.878 7fa1875c3700 1 -- start start 2021-04-09T18:16:53.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.878 7fa1875c3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa1800cffa0 0x7fa1800d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.878 7fa1875c3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1800d7280 con 0x7fa1800d6c80 2021-04-09T18:16:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.878 7fa1875c3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1800d83f0 con 0x7fa1800d7800 2021-04-09T18:16:53.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.878 7fa1875c3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1800d7080 con 0x7fa1800cffa0 2021-04-09T18:16:53.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa185be0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa1800d6c80 0x7fa18010cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54662/0 (socket says 172.21.15.32:54662) 2021-04-09T18:16:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa185be0700 1 -- 172.21.15.32:0/3895828677 learned_addr learned my addr 172.21.15.32:0/3895828677 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:53.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa1853df700 1 --2- 172.21.15.32:0/3895828677 >> v2:172.21.15.86:3300/0 conn(0x7fa1800cffa0 0x7fa1800d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:53.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa177fff700 1 -- 172.21.15.32:0/3895828677 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412287063 0 0) 0x7fa178000d30 con 0x7fa1800d6c80 2021-04-09T18:16:53.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa177fff700 1 -- 172.21.15.32:0/3895828677 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16c002fe0 con 0x7fa1800d6c80 2021-04-09T18:16:53.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa177fff700 1 -- 172.21.15.32:0/3895828677 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610049508 0 0) 0x7fa170000c60 con 0x7fa1800d7800 2021-04-09T18:16:53.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa177fff700 1 -- 172.21.15.32:0/3895828677 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16c0042c0 con 0x7fa1800d7800 2021-04-09T18:16:53.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa1853df700 1 -- 172.21.15.32:0/3895828677 >> v1:172.21.15.86:6789/0 conn(0x7fa1800d7800 legacy=0x7fa1800d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:53.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa1853df700 1 -- 172.21.15.32:0/3895828677 >> v1:172.21.15.32:6789/0 conn(0x7fa1800d6c80 legacy=0x7fa18010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:53.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa1853df700 1 -- 172.21.15.32:0/3895828677 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1800d91e0 con 0x7fa1800cffa0 2021-04-09T18:16:53.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa1853df700 1 --2- 172.21.15.32:0/3895828677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800cffa0 0x7fa1800d2440 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fa17c006650 tx=0x7fa17c009a80).ready entity=mon.1 client_cookie=e552954b0c48460b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:53.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa177fff700 1 -- 172.21.15.32:0/3895828677 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa17c0009b0 con 0x7fa1800cffa0 2021-04-09T18:16:53.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa177fff700 1 -- 172.21.15.32:0/3895828677 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c00c0c0 con 0x7fa1800cffa0 2021-04-09T18:16:53.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa177fff700 1 -- 172.21.15.32:0/3895828677 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa17c00c510 con 0x7fa1800cffa0 2021-04-09T18:16:53.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.879 7fa1875c3700 1 -- 172.21.15.32:0/3895828677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800cffa0 msgr2=0x7fa1800d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:53.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.880 7fa1875c3700 1 --2- 172.21.15.32:0/3895828677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800cffa0 0x7fa1800d2440 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fa17c006650 tx=0x7fa17c009a80).stop 2021-04-09T18:16:53.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.880 7fa1875c3700 1 -- 172.21.15.32:0/3895828677 shutdown_connections 2021-04-09T18:16:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.880 7fa1875c3700 1 --2- 172.21.15.32:0/3895828677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800cffa0 0x7fa1800d2440 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:53.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.880 7fa1875c3700 1 -- 172.21.15.32:0/3895828677 >> 172.21.15.32:0/3895828677 conn(0x7fa1800ce5c0 msgr2=0x7fa1800cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:53.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.880 7fa1875c3700 1 -- 172.21.15.32:0/3895828677 shutdown_connections 2021-04-09T18:16:53.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.880 7fa1875c3700 1 -- 172.21.15.32:0/3895828677 wait complete. 2021-04-09T18:16:53.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.900 7fa1875c3700 1 Processor -- start 2021-04-09T18:16:53.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.900 7fa1875c3700 1 -- start start 2021-04-09T18:16:53.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa1875c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1801ac530 0x7fa1800ac280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:53.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa1875c3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800ac8a0 0x7fa1801a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa1875c3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1801a12e0 0x7fa1801a2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa1875c3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1801a2a90 con 0x7fa1801ac530 2021-04-09T18:16:54.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa1875c3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18017ad40 con 0x7fa1800ac8a0 2021-04-09T18:16:54.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa1875c3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18017af60 con 0x7fa1801a12e0 2021-04-09T18:16:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa1853df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1801ac530 0x7fa1800ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa1853df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1801ac530 0x7fa1800ac280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45076/0 (socket says 172.21.15.32:45076) 2021-04-09T18:16:54.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa1853df700 1 -- 172.21.15.32:0/884758500 learned_addr learned my addr 172.21.15.32:0/884758500 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:54.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa184bde700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800ac8a0 0x7fa1801a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.901 7fa185be0700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1801a12e0 0x7fa1801a2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa1853df700 1 -- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1801a12e0 msgr2=0x7fa1801a2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa1853df700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1801a12e0 0x7fa1801a2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa1853df700 1 -- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800ac8a0 msgr2=0x7fa1801a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa1853df700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800ac8a0 0x7fa1801a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa1853df700 1 -- 172.21.15.32:0/884758500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18017b180 con 0x7fa1801ac530 2021-04-09T18:16:54.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa1853df700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1801ac530 0x7fa1800ac280 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fa17c009650 tx=0x7fa17c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:54.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa175ffb700 1 -- 172.21.15.32:0/884758500 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa17c0093a0 con 0x7fa1801ac530 2021-04-09T18:16:54.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa175ffb700 1 -- 172.21.15.32:0/884758500 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c006070 con 0x7fa1801ac530 2021-04-09T18:16:54.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa175ffb700 1 -- 172.21.15.32:0/884758500 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa17c0052e0 con 0x7fa1801ac530 2021-04-09T18:16:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa1875c3700 1 -- 172.21.15.32:0/884758500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18017b180 con 0x7fa1801ac530 2021-04-09T18:16:54.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.902 7fa1875c3700 1 -- 172.21.15.32:0/884758500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18017b180 con 0x7fa1801ac530 2021-04-09T18:16:54.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.903 7fa175ffb700 1 -- 172.21.15.32:0/884758500 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa17c009300 con 0x7fa1801ac530 2021-04-09T18:16:54.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.903 7fa175ffb700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1600188c0 0x7fa16001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.903 7fa175ffb700 1 -- 172.21.15.32:0/884758500 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa17c0327e0 con 0x7fa1801ac530 2021-04-09T18:16:54.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.903 7fa184bde700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1600188c0 0x7fa16001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.903 7fa184bde700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1600188c0 0x7fa16001adc0 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:54.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.922 7fa1875c3700 1 -- 172.21.15.32:0/884758500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1801a1920 con 0x7fa1801ac530 2021-04-09T18:16:54.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:53.926 7fa175ffb700 1 -- 172.21.15.32:0/884758500 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa17c000940 con 0x7fa1801ac530 2021-04-09T18:16:54.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.116 7fa1875c3700 1 -- 172.21.15.32:0/884758500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1801a1920 con 0x7fa1801ac530 2021-04-09T18:16:54.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.117 7fa175ffb700 1 -- 172.21.15.32:0/884758500 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fa17c009300 con 0x7fa1801ac530 2021-04-09T18:16:54.149 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:54.149 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":63,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:16:48.944337","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:16:47.324034","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-04-09 18:16:48.346213","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:16:54.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.118 7fa1875c3700 1 -- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1600188c0 msgr2=0x7fa16001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.118 7fa1875c3700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1600188c0 0x7fa16001adc0 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.118 7fa1875c3700 1 -- 172.21.15.32:0/884758500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1801ac530 msgr2=0x7fa1800ac280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.118 7fa1875c3700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1801ac530 0x7fa1800ac280 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fa17c009650 tx=0x7fa17c00b8e0).stop 2021-04-09T18:16:54.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.119 7fa1875c3700 1 -- 172.21.15.32:0/884758500 shutdown_connections 2021-04-09T18:16:54.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.119 7fa1875c3700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1600188c0 0x7fa16001adc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.119 7fa1875c3700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1801a12e0 0x7fa1801a2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.119 7fa1875c3700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800ac8a0 0x7fa1801a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.119 7fa1875c3700 1 --2- 172.21.15.32:0/884758500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1801ac530 0x7fa1800ac280 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.119 7fa1875c3700 1 -- 172.21.15.32:0/884758500 >> 172.21.15.32:0/884758500 conn(0x7fa1801941b0 msgr2=0x7fa18018d640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:54.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.119 7fa1875c3700 1 -- 172.21.15.32:0/884758500 shutdown_connections 2021-04-09T18:16:54.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.119 7fa1875c3700 1 -- 172.21.15.32:0/884758500 wait complete. 2021-04-09T18:16:54.189 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:16:47.324034', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:16:48.346213', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:16:54.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:16:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.360 7fd6dc901700 1 Processor -- start 2021-04-09T18:16:54.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.360 7fd6dc901700 1 -- start start 2021-04-09T18:16:54.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.361 7fd6dc901700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d810d010 0x7fd6d80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.361 7fd6dc901700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d810d480 con 0x7fd6d80d8490 2021-04-09T18:16:54.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.361 7fd6dc901700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d810e1f0 con 0x7fd6d80d1da0 2021-04-09T18:16:54.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.361 7fd6dc901700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d810e780 con 0x7fd6d810d010 2021-04-09T18:16:54.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.361 7fd6d6e1d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd6d80d8490 0x7fd6d810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54676/0 (socket says 172.21.15.32:54676) 2021-04-09T18:16:54.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.361 7fd6d6e1d700 1 -- 172.21.15.32:0/4226370517 learned_addr learned my addr 172.21.15.32:0/4226370517 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:54.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.361 7fd6d661c700 1 --2- 172.21.15.32:0/4226370517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d810d010 0x7fd6d80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.361 7fd6d561a700 1 -- 172.21.15.32:0/4226370517 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169286185 0 0) 0x7fd6d0000d30 con 0x7fd6d80d8490 2021-04-09T18:16:54.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d561a700 1 -- 172.21.15.32:0/4226370517 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c0002fe0 con 0x7fd6d80d8490 2021-04-09T18:16:54.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d561a700 1 -- 172.21.15.32:0/4226370517 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656833564 0 0) 0x7fd6cc000c60 con 0x7fd6d80d1da0 2021-04-09T18:16:54.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d561a700 1 -- 172.21.15.32:0/4226370517 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c0003210 con 0x7fd6d80d1da0 2021-04-09T18:16:54.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d661c700 1 -- 172.21.15.32:0/4226370517 >> v1:172.21.15.86:6789/0 conn(0x7fd6d80d1da0 legacy=0x7fd6d80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d661c700 1 -- 172.21.15.32:0/4226370517 >> v1:172.21.15.32:6789/0 conn(0x7fd6d80d8490 legacy=0x7fd6d810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d661c700 1 -- 172.21.15.32:0/4226370517 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d80d8d30 con 0x7fd6d810d010 2021-04-09T18:16:54.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d661c700 1 --2- 172.21.15.32:0/4226370517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d810d010 0x7fd6d80d1780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fd6c8006650 tx=0x7fd6c8009a80).ready entity=mon.2 client_cookie=af0ba1b30d1e177c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:54.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d561a700 1 -- 172.21.15.32:0/4226370517 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c8006390 con 0x7fd6d810d010 2021-04-09T18:16:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d561a700 1 -- 172.21.15.32:0/4226370517 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c80009b0 con 0x7fd6d810d010 2021-04-09T18:16:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6d561a700 1 -- 172.21.15.32:0/4226370517 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c800c2b0 con 0x7fd6d810d010 2021-04-09T18:16:54.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6dc901700 1 -- 172.21.15.32:0/4226370517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d810d010 msgr2=0x7fd6d80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.362 7fd6dc901700 1 --2- 172.21.15.32:0/4226370517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d810d010 0x7fd6d80d1780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fd6c8006650 tx=0x7fd6c8009a80).stop 2021-04-09T18:16:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.363 7fd6dc901700 1 -- 172.21.15.32:0/4226370517 shutdown_connections 2021-04-09T18:16:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.363 7fd6dc901700 1 --2- 172.21.15.32:0/4226370517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d810d010 0x7fd6d80d1780 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.363 7fd6dc901700 1 -- 172.21.15.32:0/4226370517 >> 172.21.15.32:0/4226370517 conn(0x7fd6d80ce5f0 msgr2=0x7fd6d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:54.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.363 7fd6dc901700 1 -- 172.21.15.32:0/4226370517 shutdown_connections 2021-04-09T18:16:54.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.363 7fd6dc901700 1 -- 172.21.15.32:0/4226370517 wait complete. 2021-04-09T18:16:54.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.383 7fd6dc901700 1 Processor -- start 2021-04-09T18:16:54.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.383 7fd6dc901700 1 -- start start 2021-04-09T18:16:54.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6dc901700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d8179610 0x7fd6d8179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6dc901700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6d817a820 0x7fd6d817ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6dc901700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6d81949f0 0x7fd6d8194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6dc901700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d81b8d50 con 0x7fd6d81949f0 2021-04-09T18:16:54.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6dc901700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d817b240 con 0x7fd6d817a820 2021-04-09T18:16:54.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6dc901700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8195410 con 0x7fd6d8179610 2021-04-09T18:16:54.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d6e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6d81949f0 0x7fd6d8194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d6e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6d81949f0 0x7fd6d8194df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45090/0 (socket says 172.21.15.32:45090) 2021-04-09T18:16:54.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d6e1d700 1 -- 172.21.15.32:0/1888270181 learned_addr learned my addr 172.21.15.32:0/1888270181 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:54.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d5e1b700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6d817a820 0x7fd6d817ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d661c700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d8179610 0x7fd6d8179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d6e1d700 1 -- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d8179610 msgr2=0x7fd6d8179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d6e1d700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d8179610 0x7fd6d8179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d6e1d700 1 -- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6d817a820 msgr2=0x7fd6d817ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d6e1d700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6d817a820 0x7fd6d817ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d6e1d700 1 -- 172.21.15.32:0/1888270181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d8195630 con 0x7fd6d81949f0 2021-04-09T18:16:54.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d5e1b700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6d817a820 0x7fd6d817ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:54.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6d6e1d700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6d81949f0 0x7fd6d8194df0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fd6d000b8c0 tx=0x7fd6d000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:54.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6c6ffd700 1 -- 172.21.15.32:0/1888270181 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6d000bda0 con 0x7fd6d81949f0 2021-04-09T18:16:54.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6c6ffd700 1 -- 172.21.15.32:0/1888270181 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d0006030 con 0x7fd6d81949f0 2021-04-09T18:16:54.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d8195630 con 0x7fd6d81949f0 2021-04-09T18:16:54.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6c6ffd700 1 -- 172.21.15.32:0/1888270181 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6d00064c0 con 0x7fd6d81949f0 2021-04-09T18:16:54.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.384 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d8195630 con 0x7fd6d81949f0 2021-04-09T18:16:54.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.385 7fd6c6ffd700 1 -- 172.21.15.32:0/1888270181 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd6d0006210 con 0x7fd6d81949f0 2021-04-09T18:16:54.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.385 7fd6c6ffd700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6bc018930 0x7fd6bc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.385 7fd6c6ffd700 1 -- 172.21.15.32:0/1888270181 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd6d0032da0 con 0x7fd6d81949f0 2021-04-09T18:16:54.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.385 7fd6d661c700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6bc018930 0x7fd6bc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.386 7fd6d661c700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6bc018930 0x7fd6bc01ae10 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:54.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.405 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d817a2f0 con 0x7fd6d81949f0 2021-04-09T18:16:54.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.408 7fd6c6ffd700 1 -- 172.21.15.32:0/1888270181 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd6d0000f90 con 0x7fd6d81949f0 2021-04-09T18:16:54.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.589 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd6d817a2f0 con 0x7fd6d81949f0 2021-04-09T18:16:54.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.589 7fd6c6ffd700 1 -- 172.21.15.32:0/1888270181 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v63) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd6d0006030 con 0x7fd6d81949f0 2021-04-09T18:16:54.621 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:16:54.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.591 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6bc018930 msgr2=0x7fd6bc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.591 7fd6dc901700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6bc018930 0x7fd6bc01ae10 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.591 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6d81949f0 msgr2=0x7fd6d8194df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.591 7fd6dc901700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6d81949f0 0x7fd6d8194df0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fd6d000b8c0 tx=0x7fd6d000bc60).stop 2021-04-09T18:16:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.592 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 shutdown_connections 2021-04-09T18:16:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.592 7fd6dc901700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6bc018930 0x7fd6bc01ae10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.592 7fd6dc901700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6d81949f0 0x7fd6d8194df0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.592 7fd6dc901700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6d817a820 0x7fd6d817ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.592 7fd6dc901700 1 --2- 172.21.15.32:0/1888270181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6d8179610 0x7fd6d8179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.592 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 >> 172.21.15.32:0/1888270181 conn(0x7fd6d8092a00 msgr2=0x7fd6d81b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.592 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 shutdown_connections 2021-04-09T18:16:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.592 7fd6dc901700 1 -- 172.21.15.32:0/1888270181 wait complete. 2021-04-09T18:16:54.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:16:54.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.789 7f21ad4e4700 1 Processor -- start 2021-04-09T18:16:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.789 7f21ad4e4700 1 -- start start 2021-04-09T18:16:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21ad4e4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a810d010 0x7f21a80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21ad4e4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f21a80d8490 0x7f21a810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21ad4e4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a810d480 con 0x7f21a80d1da0 2021-04-09T18:16:54.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21ad4e4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a810e780 con 0x7f21a80d8490 2021-04-09T18:16:54.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21ad4e4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80d8d30 con 0x7f21a810d010 2021-04-09T18:16:54.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a7ac0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f21a80d8490 0x7f21a810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a72bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a810d010 0x7f21a80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a6abe700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f21a80d1da0 0x7f21a80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52614/0 (socket says 172.21.15.32:52614) 2021-04-09T18:16:54.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a6abe700 1 -- 172.21.15.32:0/4144532552 learned_addr learned my addr 172.21.15.32:0/4144532552 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:54.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a62bd700 1 -- 172.21.15.32:0/4144532552 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847585056 0 0) 0x7f21940010c0 con 0x7f21a80d1da0 2021-04-09T18:16:54.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a62bd700 1 -- 172.21.15.32:0/4144532552 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f218c002fe0 con 0x7f21a80d1da0 2021-04-09T18:16:54.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a72bf700 1 -- 172.21.15.32:0/4144532552 >> v2:172.21.15.86:3300/0 conn(0x7f21a80d8490 msgr2=0x7f21a810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:54.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a72bf700 1 --2- 172.21.15.32:0/4144532552 >> v2:172.21.15.86:3300/0 conn(0x7f21a80d8490 0x7f21a810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a72bf700 1 -- 172.21.15.32:0/4144532552 >> v1:172.21.15.86:6789/0 conn(0x7f21a80d1da0 legacy=0x7f21a80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a72bf700 1 -- 172.21.15.32:0/4144532552 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80d9220 con 0x7f21a810d010 2021-04-09T18:16:54.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.790 7f21a7ac0700 1 --2- 172.21.15.32:0/4144532552 >> v2:172.21.15.86:3300/0 conn(0x7f21a80d8490 0x7f21a810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:54.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.791 7f21a72bf700 1 --2- 172.21.15.32:0/4144532552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a810d010 0x7f21a80d1780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2198006650 tx=0x7f2198009a80).ready entity=mon.2 client_cookie=83df2cd2d274ff28 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:54.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.791 7f21a62bd700 1 -- 172.21.15.32:0/4144532552 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21980009b0 con 0x7f21a810d010 2021-04-09T18:16:54.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.791 7f21a62bd700 1 -- 172.21.15.32:0/4144532552 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219800c0c0 con 0x7f21a810d010 2021-04-09T18:16:54.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.791 7f21a62bd700 1 -- 172.21.15.32:0/4144532552 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f219800c510 con 0x7f21a810d010 2021-04-09T18:16:54.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.791 7f21ad4e4700 1 -- 172.21.15.32:0/4144532552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a810d010 msgr2=0x7f21a80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.791 7f21ad4e4700 1 --2- 172.21.15.32:0/4144532552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a810d010 0x7f21a80d1780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2198006650 tx=0x7f2198009a80).stop 2021-04-09T18:16:54.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.791 7f21ad4e4700 1 -- 172.21.15.32:0/4144532552 shutdown_connections 2021-04-09T18:16:54.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.791 7f21ad4e4700 1 --2- 172.21.15.32:0/4144532552 >> v2:172.21.15.86:3300/0 conn(0x7f21a80d8490 0x7f21a810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.791 7f21ad4e4700 1 --2- 172.21.15.32:0/4144532552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a810d010 0x7f21a80d1780 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.792 7f21ad4e4700 1 -- 172.21.15.32:0/4144532552 >> 172.21.15.32:0/4144532552 conn(0x7f21a80ce5f0 msgr2=0x7f21a80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:54.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.792 7f21ad4e4700 1 -- 172.21.15.32:0/4144532552 shutdown_connections 2021-04-09T18:16:54.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.792 7f21ad4e4700 1 -- 172.21.15.32:0/4144532552 wait complete. 2021-04-09T18:16:54.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.811 7f21ad4e4700 1 Processor -- start 2021-04-09T18:16:54.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21ad4e4700 1 -- start start 2021-04-09T18:16:54.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21ad4e4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21a80b2140 0x7f21a808b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21ad4e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a809f550 0x7f21a809f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21ad4e4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a8197c50 0x7f21a8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21ad4e4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a809f010 con 0x7f21a809f550 2021-04-09T18:16:54.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21ad4e4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a809ff70 con 0x7f21a80b2140 2021-04-09T18:16:54.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21ad4e4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a8198670 con 0x7f21a8197c50 2021-04-09T18:16:54.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21a6abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a809f550 0x7f21a809f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21a6abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a809f550 0x7f21a809f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45102/0 (socket says 172.21.15.32:45102) 2021-04-09T18:16:54.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21a6abe700 1 -- 172.21.15.32:0/1223146079 learned_addr learned my addr 172.21.15.32:0/1223146079 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:54.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21a7ac0700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a8197c50 0x7f21a8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.812 7f21a72bf700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21a80b2140 0x7f21a808b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f21a6abe700 1 -- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a8197c50 msgr2=0x7f21a8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f21a6abe700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a8197c50 0x7f21a8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f21a6abe700 1 -- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21a80b2140 msgr2=0x7f21a808b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f21a6abe700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21a80b2140 0x7f21a808b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f21a6abe700 1 -- 172.21.15.32:0/1223146079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a8198890 con 0x7f21a809f550 2021-04-09T18:16:54.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f21a7ac0700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a8197c50 0x7f21a8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:54.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f21a6abe700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a809f550 0x7f21a809f950 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f219400a550 tx=0x7f219400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:54.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f219ffff700 1 -- 172.21.15.32:0/1223146079 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2194000e70 con 0x7f21a809f550 2021-04-09T18:16:54.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f219ffff700 1 -- 172.21.15.32:0/1223146079 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219400ace0 con 0x7f21a809f550 2021-04-09T18:16:54.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a8198890 con 0x7f21a809f550 2021-04-09T18:16:54.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f219ffff700 1 -- 172.21.15.32:0/1223146079 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2194004510 con 0x7f21a809f550 2021-04-09T18:16:54.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.813 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a8198890 con 0x7f21a809f550 2021-04-09T18:16:54.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.814 7f219ffff700 1 -- 172.21.15.32:0/1223146079 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2194004320 con 0x7f21a809f550 2021-04-09T18:16:54.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.814 7f219ffff700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21840188d0 0x7f218401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:54.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.814 7f219ffff700 1 -- 172.21.15.32:0/1223146079 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2194032a30 con 0x7f21a809f550 2021-04-09T18:16:54.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.814 7f21a72bf700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21840188d0 0x7f218401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.814 7f21a72bf700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21840188d0 0x7f218401add0 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.834 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a809f010 con 0x7f21a809f550 2021-04-09T18:16:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:54.838 7f219ffff700 1 -- 172.21.15.32:0/1223146079 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2194000c10 con 0x7f21a809f550 2021-04-09T18:16:55.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.019 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f21a809f010 con 0x7f21a809f550 2021-04-09T18:16:55.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.019 7f219ffff700 1 -- 172.21.15.32:0/1223146079 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v63) v1 ==== 104+0+10 (secure 0 0 0) 0x7f219400ace0 con 0x7f21a809f550 2021-04-09T18:16:55.052 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:16:55.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21840188d0 msgr2=0x7f218401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21840188d0 0x7f218401add0 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a809f550 msgr2=0x7f21a809f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a809f550 0x7f21a809f950 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f219400a550 tx=0x7f219400a8f0).stop 2021-04-09T18:16:55.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 shutdown_connections 2021-04-09T18:16:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21840188d0 0x7f218401add0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21a809f550 0x7f21a809f950 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21a8197c50 0x7f21a8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 --2- 172.21.15.32:0/1223146079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21a80b2140 0x7f21a808b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 >> 172.21.15.32:0/1223146079 conn(0x7f21a80b27e0 msgr2=0x7f21a81b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 shutdown_connections 2021-04-09T18:16:55.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.022 7f21ad4e4700 1 -- 172.21.15.32:0/1223146079 wait complete. 2021-04-09T18:16:55.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:16:55.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.210 7f41dbffe700 1 Processor -- start 2021-04-09T18:16:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.210 7f41dbffe700 1 -- start start 2021-04-09T18:16:55.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41dbffe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41d410d010 0x7f41d40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41dbffe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d40d8490 0x7f41d410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41dbffe700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d410d480 con 0x7f41d40d1da0 2021-04-09T18:16:55.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41dbffe700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d40d8d30 con 0x7f41d410d010 2021-04-09T18:16:55.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41dbffe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40d9220 con 0x7f41d40d8490 2021-04-09T18:16:55.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41d9e1a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41d410d010 0x7f41d40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41d9e1a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41d410d010 0x7f41d40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45110/0 (socket says 172.21.15.32:45110) 2021-04-09T18:16:55.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41d9e1a700 1 -- 172.21.15.32:0/3780915320 learned_addr learned my addr 172.21.15.32:0/3780915320 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:55.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41da61b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d40d8490 0x7f41d410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.211 7f41d9619700 1 --1- 172.21.15.32:0/3780915320 >> v1:172.21.15.86:6789/0 conn(0x7f41d40d1da0 0x7f41d40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52630/0 (socket says 172.21.15.32:52630) 2021-04-09T18:16:55.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41d9e1a700 1 -- 172.21.15.32:0/3780915320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d40d8490 msgr2=0x7f41d410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:55.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41d9e1a700 1 --2- 172.21.15.32:0/3780915320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d40d8490 0x7f41d410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41d9e1a700 1 -- 172.21.15.32:0/3780915320 >> v1:172.21.15.86:6789/0 conn(0x7f41d40d1da0 legacy=0x7f41d40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41d8e18700 1 -- 172.21.15.32:0/3780915320 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305330169 0 0) 0x7f41c8000c60 con 0x7f41d40d1da0 2021-04-09T18:16:55.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41d9e1a700 1 -- 172.21.15.32:0/3780915320 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d410e020 con 0x7f41d410d010 2021-04-09T18:16:55.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41da61b700 1 --2- 172.21.15.32:0/3780915320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d40d8490 0x7f41d410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:55.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41d9e1a700 1 --2- 172.21.15.32:0/3780915320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d410d010 0x7f41d40d1780 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f41d0006700 tx=0x7f41d0009b80).ready entity=mon.0 client_cookie=d4188254d9ca959e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:55.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41d8e18700 1 -- 172.21.15.32:0/3780915320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41d000c0c0 con 0x7f41d410d010 2021-04-09T18:16:55.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41d8e18700 1 -- 172.21.15.32:0/3780915320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d000c440 con 0x7f41d410d010 2021-04-09T18:16:55.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41d8e18700 1 -- 172.21.15.32:0/3780915320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41d00009b0 con 0x7f41d410d010 2021-04-09T18:16:55.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41dbffe700 1 -- 172.21.15.32:0/3780915320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d410d010 msgr2=0x7f41d40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.212 7f41dbffe700 1 --2- 172.21.15.32:0/3780915320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d410d010 0x7f41d40d1780 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f41d0006700 tx=0x7f41d0009b80).stop 2021-04-09T18:16:55.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.213 7f41dbffe700 1 -- 172.21.15.32:0/3780915320 shutdown_connections 2021-04-09T18:16:55.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.213 7f41dbffe700 1 --2- 172.21.15.32:0/3780915320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d40d8490 0x7f41d410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.213 7f41dbffe700 1 --2- 172.21.15.32:0/3780915320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d410d010 0x7f41d40d1780 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.213 7f41dbffe700 1 -- 172.21.15.32:0/3780915320 >> 172.21.15.32:0/3780915320 conn(0x7f41d40ce5f0 msgr2=0x7f41d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.213 7f41dbffe700 1 -- 172.21.15.32:0/3780915320 shutdown_connections 2021-04-09T18:16:55.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.213 7f41dbffe700 1 -- 172.21.15.32:0/3780915320 wait complete. 2021-04-09T18:16:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.233 7f41dbffe700 1 Processor -- start 2021-04-09T18:16:55.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.233 7f41dbffe700 1 -- start start 2021-04-09T18:16:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41dbffe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d409e530 0x7f41d409b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41dbffe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41d409e930 0x7f41d409f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41dbffe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d409f990 0x7f41d4094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41dbffe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d409edf0 con 0x7f41d409f990 2021-04-09T18:16:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41dbffe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d409fd90 con 0x7f41d409e930 2021-04-09T18:16:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41dbffe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d409bd40 con 0x7f41d409e530 2021-04-09T18:16:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41da61b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d409f990 0x7f41d4094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41d9e1a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d409e530 0x7f41d409b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41d9619700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41d409e930 0x7f41d409f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41da61b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d409f990 0x7f41d4094c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45120/0 (socket says 172.21.15.32:45120) 2021-04-09T18:16:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41da61b700 1 -- 172.21.15.32:0/1156024536 learned_addr learned my addr 172.21.15.32:0/1156024536 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41da61b700 1 -- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d409e530 msgr2=0x7f41d409b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41da61b700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d409e530 0x7f41d409b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41da61b700 1 -- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41d409e930 msgr2=0x7f41d409f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41da61b700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41d409e930 0x7f41d409f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41da61b700 1 -- 172.21.15.32:0/1156024536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d4095290 con 0x7f41d409f990 2021-04-09T18:16:55.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41da61b700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d409f990 0x7f41d4094c70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f41cc00b970 tx=0x7f41cc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:55.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41c67fc700 1 -- 172.21.15.32:0/1156024536 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41cc000f90 con 0x7f41d409f990 2021-04-09T18:16:55.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41c67fc700 1 -- 172.21.15.32:0/1156024536 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc006aa0 con 0x7f41d409f990 2021-04-09T18:16:55.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d4095290 con 0x7f41d409f990 2021-04-09T18:16:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.234 7f41c67fc700 1 -- 172.21.15.32:0/1156024536 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41cc0015f0 con 0x7f41d409f990 2021-04-09T18:16:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.235 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d4095290 con 0x7f41d409f990 2021-04-09T18:16:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.235 7f41c67fc700 1 -- 172.21.15.32:0/1156024536 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f41cc000e70 con 0x7f41d409f990 2021-04-09T18:16:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.235 7f41c67fc700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41bc018930 0x7f41bc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.235 7f41c67fc700 1 -- 172.21.15.32:0/1156024536 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f41cc032ef0 con 0x7f41d409f990 2021-04-09T18:16:55.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.235 7f41d9e1a700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41bc018930 0x7f41bc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.236 7f41d9e1a700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41bc018930 0x7f41bc01ae10 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:55.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.255 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d410d480 con 0x7f41d409f990 2021-04-09T18:16:55.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.259 7f41c67fc700 1 -- 172.21.15.32:0/1156024536 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f41cc0066f0 con 0x7f41d409f990 2021-04-09T18:16:55.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.439 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f41d410d480 con 0x7f41d409f990 2021-04-09T18:16:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.439 7f41c67fc700 1 -- 172.21.15.32:0/1156024536 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v63) v1 ==== 100+0+10 (secure 0 0 0) 0x7f41cc006aa0 con 0x7f41d409f990 2021-04-09T18:16:55.472 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:16:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.441 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41bc018930 msgr2=0x7f41bc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.441 7f41dbffe700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41bc018930 0x7f41bc01ae10 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.441 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d409f990 msgr2=0x7f41d4094c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.441 7f41dbffe700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d409f990 0x7f41d4094c70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f41cc00b970 tx=0x7f41cc00bd10).stop 2021-04-09T18:16:55.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.442 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 shutdown_connections 2021-04-09T18:16:55.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.442 7f41dbffe700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41bc018930 0x7f41bc01ae10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.442 7f41dbffe700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d409f990 0x7f41d4094c70 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.442 7f41dbffe700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41d409e930 0x7f41d409f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.442 7f41dbffe700 1 --2- 172.21.15.32:0/1156024536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41d409e530 0x7f41d409b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.442 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 >> 172.21.15.32:0/1156024536 conn(0x7f41d40c4290 msgr2=0x7f41d40c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:55.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.442 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 shutdown_connections 2021-04-09T18:16:55.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.442 7f41dbffe700 1 -- 172.21.15.32:0/1156024536 wait complete. 2021-04-09T18:16:55.553 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:16:55.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:16:55.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.672 7ff11818a700 1 Processor -- start 2021-04-09T18:16:55.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.672 7ff11818a700 1 -- start start 2021-04-09T18:16:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff11818a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff11010da00 0x7ff11010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff11818a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11010d480 con 0x7ff11010d010 2021-04-09T18:16:55.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff11818a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100d1f70 con 0x7ff1100d7ca0 2021-04-09T18:16:55.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff11818a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100d9220 con 0x7ff11010da00 2021-04-09T18:16:55.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff115fa6700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff11010d010 0x7ff1100d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54714/0 (socket says 172.21.15.32:54714) 2021-04-09T18:16:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff115fa6700 1 -- 172.21.15.32:0/2114548222 learned_addr learned my addr 172.21.15.32:0/2114548222 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:55.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff1167a7700 1 --2- 172.21.15.32:0/2114548222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff11010da00 0x7ff11010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff114fa4700 1 -- 172.21.15.32:0/2114548222 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614439971 0 0) 0x7ff10c000ce0 con 0x7ff11010d010 2021-04-09T18:16:55.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff114fa4700 1 -- 172.21.15.32:0/2114548222 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0fc002fe0 con 0x7ff11010d010 2021-04-09T18:16:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff114fa4700 1 -- 172.21.15.32:0/2114548222 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770144226 0 0) 0x7ff100000c60 con 0x7ff1100d7ca0 2021-04-09T18:16:55.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff114fa4700 1 -- 172.21.15.32:0/2114548222 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0fc003020 con 0x7ff1100d7ca0 2021-04-09T18:16:55.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff1167a7700 1 -- 172.21.15.32:0/2114548222 >> v1:172.21.15.86:6789/0 conn(0x7ff1100d7ca0 legacy=0x7ff1100d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff1167a7700 1 -- 172.21.15.32:0/2114548222 >> v1:172.21.15.32:6789/0 conn(0x7ff11010d010 legacy=0x7ff1100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff114fa4700 1 -- 172.21.15.32:0/2114548222 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2005507876 0 0) 0x7ff10c000ed0 con 0x7ff11010d010 2021-04-09T18:16:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff1167a7700 1 -- 172.21.15.32:0/2114548222 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11010e420 con 0x7ff11010da00 2021-04-09T18:16:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.673 7ff1167a7700 1 --2- 172.21.15.32:0/2114548222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff11010da00 0x7ff11010de00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ff108006e50 tx=0x7ff108009df0).ready entity=mon.2 client_cookie=14a3b7a83e4ea708 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.674 7ff114fa4700 1 -- 172.21.15.32:0/2114548222 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff10800b510 con 0x7ff11010da00 2021-04-09T18:16:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.674 7ff114fa4700 1 -- 172.21.15.32:0/2114548222 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff108003030 con 0x7ff11010da00 2021-04-09T18:16:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.674 7ff114fa4700 1 -- 172.21.15.32:0/2114548222 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff10800b7d0 con 0x7ff11010da00 2021-04-09T18:16:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.674 7ff11818a700 1 -- 172.21.15.32:0/2114548222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff11010da00 msgr2=0x7ff11010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.674 7ff11818a700 1 --2- 172.21.15.32:0/2114548222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff11010da00 0x7ff11010de00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ff108006e50 tx=0x7ff108009df0).stop 2021-04-09T18:16:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.674 7ff11818a700 1 -- 172.21.15.32:0/2114548222 shutdown_connections 2021-04-09T18:16:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.674 7ff11818a700 1 --2- 172.21.15.32:0/2114548222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff11010da00 0x7ff11010de00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.674 7ff11818a700 1 -- 172.21.15.32:0/2114548222 >> 172.21.15.32:0/2114548222 conn(0x7ff1100ce5f0 msgr2=0x7ff1100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:55.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.675 7ff11818a700 1 -- 172.21.15.32:0/2114548222 shutdown_connections 2021-04-09T18:16:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.675 7ff11818a700 1 -- 172.21.15.32:0/2114548222 wait complete. 2021-04-09T18:16:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff11818a700 1 Processor -- start 2021-04-09T18:16:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff11818a700 1 -- start start 2021-04-09T18:16:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff11818a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1101ac380 0x7ff11018a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff11818a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1101ac780 0x7ff11018a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff11818a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110183520 0x7ff110183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff11818a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1101ace10 con 0x7ff1101ac380 2021-04-09T18:16:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff11818a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff110189cf0 con 0x7ff110183520 2021-04-09T18:16:55.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff11818a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff110183f40 con 0x7ff1101ac780 2021-04-09T18:16:55.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff115fa6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1101ac380 0x7ff11018a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff115fa6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1101ac380 0x7ff11018a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45128/0 (socket says 172.21.15.32:45128) 2021-04-09T18:16:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff115fa6700 1 -- 172.21.15.32:0/530782412 learned_addr learned my addr 172.21.15.32:0/530782412 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.695 7ff1167a7700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110183520 0x7ff110183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff1157a5700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1101ac780 0x7ff11018a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff115fa6700 1 -- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1101ac780 msgr2=0x7ff11018a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff115fa6700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1101ac780 0x7ff11018a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff115fa6700 1 -- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110183520 msgr2=0x7ff110183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff115fa6700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110183520 0x7ff110183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff115fa6700 1 -- 172.21.15.32:0/530782412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110184160 con 0x7ff1101ac380 2021-04-09T18:16:55.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff1167a7700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110183520 0x7ff110183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:16:55.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff115fa6700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1101ac380 0x7ff11018a230 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7ff10c00a4d0 tx=0x7ff10c00a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:55.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff1067fc700 1 -- 172.21.15.32:0/530782412 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff10c0070d0 con 0x7ff1101ac380 2021-04-09T18:16:55.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff1067fc700 1 -- 172.21.15.32:0/530782412 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c001200 con 0x7ff1101ac380 2021-04-09T18:16:55.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff11818a700 1 -- 172.21.15.32:0/530782412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff110184160 con 0x7ff1101ac380 2021-04-09T18:16:55.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff1067fc700 1 -- 172.21.15.32:0/530782412 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff10c0045b0 con 0x7ff1101ac380 2021-04-09T18:16:55.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.696 7ff11818a700 1 -- 172.21.15.32:0/530782412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff110184160 con 0x7ff1101ac380 2021-04-09T18:16:55.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.697 7ff1067fc700 1 -- 172.21.15.32:0/530782412 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff10c0043a0 con 0x7ff1101ac380 2021-04-09T18:16:55.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.697 7ff1067fc700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0f8018930 0x7ff0f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:55.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.697 7ff1067fc700 1 -- 172.21.15.32:0/530782412 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff10c032a20 con 0x7ff1101ac380 2021-04-09T18:16:55.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.697 7ff1157a5700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0f8018930 0x7ff0f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:55.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.697 7ff1157a5700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0f8018930 0x7ff0f801ae10 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:55.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.717 7ff11818a700 1 -- 172.21.15.32:0/530782412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff110182d10 con 0x7ff1101ac380 2021-04-09T18:16:55.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.720 7ff1067fc700 1 -- 172.21.15.32:0/530782412 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff10c000940 con 0x7ff1101ac380 2021-04-09T18:16:55.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.910 7ff11818a700 1 -- 172.21.15.32:0/530782412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff110182d10 con 0x7ff1101ac380 2021-04-09T18:16:55.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.911 7ff1067fc700 1 -- 172.21.15.32:0/530782412 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2778 (secure 0 0 0) 0x7ff10c0043a0 con 0x7ff1101ac380 2021-04-09T18:16:55.959 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 56 2021-04-09T18:16:55.960 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:55.961 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5880,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:16:50.949963","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:16:55.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 -- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0f8018930 msgr2=0x7ff0f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0f8018930 0x7ff0f801ae10 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 -- 172.21.15.32:0/530782412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1101ac380 msgr2=0x7ff11018a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1101ac380 0x7ff11018a230 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7ff10c00a4d0 tx=0x7ff10c00a870).stop 2021-04-09T18:16:55.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 -- 172.21.15.32:0/530782412 shutdown_connections 2021-04-09T18:16:55.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0f8018930 0x7ff0f801ae10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff110183520 0x7ff110183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1101ac780 0x7ff11018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 --2- 172.21.15.32:0/530782412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1101ac380 0x7ff11018a230 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:55.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 -- 172.21.15.32:0/530782412 >> 172.21.15.32:0/530782412 conn(0x7ff11019b890 msgr2=0x7ff1101a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:55.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 -- 172.21.15.32:0/530782412 shutdown_connections 2021-04-09T18:16:55.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:55.913 7ff11818a700 1 -- 172.21.15.32:0/530782412 wait complete. 2021-04-09T18:16:56.391 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:16:50.949963', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:16:56.391 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:16:56.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:16:56.526 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:16:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:16:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:16:56.527 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 5803, 2021-04-09T18:16:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:16:56.528 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:16:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 7.5498928359999997, 2021-04-09T18:16:56.529 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 56, 2021-04-09T18:16:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 63, 2021-04-09T18:16:56.530 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 63, 2021-04-09T18:16:56.531 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12.006779276 2021-04-09T18:16:56.531 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:16:56.533 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5803, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.5498928359999997, "mdsmap_epoch": 56, "osdmap_epoch": 63, "osdmap_epoch_barrier": 63, "uptime": 12.006779276 } 2021-04-09T18:16:56.534 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:16:56.534 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:16:56.539 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:16:56.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:56.541 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:56.600 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:16:56.663 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:16:56.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:16:56.742 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:16:56.742 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:16:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:16:56.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:56.744 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:16:56.787 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:16:56.788 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:56.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:16:56.893 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:16:56.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:16:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:16:56.958 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:16:56.959 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:16:56.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:56.969 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:16:56.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:16:56.948 7f009573ff80 -1 init, newargv = 0x558be6fdf260 newargc=9 2021-04-09T18:16:56.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[42698]: starting ceph client 2021-04-09T18:16:56.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[42698]: starting fuse 2021-04-09T18:16:57.021 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:16:57.021 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:16:57.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:57.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:16:57.025 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:16:57.026 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:16:57.027 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:16:57.028 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:16:57.028 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:16:57.029 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:16:57.030 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:16:57.030 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:16:57.089 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:16:57.095 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:16:57.095 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:16:57.096 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:16:57.097 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:16:57.098 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:16:57.098 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:16:57.121 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:16:57.130 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:16:57.130 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:16:57.136 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:16:57.136 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:16:57.139 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:16:57.151 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:16:57.238 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.42698.asok 2021-04-09T18:16:57.243 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42698.asok 2021-04-09T18:16:57.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:57.244 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42698.asok status 2021-04-09T18:16:57.367 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:16:57.368 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:16:57.368 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:16:57.369 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:16:57.369 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:16:57.370 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:16:57.370 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:16:57.371 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "42698", 2021-04-09T18:16:57.371 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:16:57.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:57.372 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:16:57.373 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:16:57.373 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6012, 2021-04-09T18:16:57.374 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:16:57.375 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:16:57.376 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:16:57.376 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 6012 2021-04-09T18:16:57.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:57.377 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:16:57.377 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:57.377 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:16:57.378 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3236358464 2021-04-09T18:16:57.378 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:16:57.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:57.378 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:16:57.378 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:16:57.379 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:16:57.379 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3236358464 2021-04-09T18:16:57.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:16:57.380 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.6012 172.21.15.32:0/3236358464", 2021-04-09T18:16:57.380 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/3236358464", 2021-04-09T18:16:57.380 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:16:57.380 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 56, 2021-04-09T18:16:57.380 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 63, 2021-04-09T18:16:57.381 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:16:57.381 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:16:57.381 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:16:57.381 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:16:57.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:57.429 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:16:57.429 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:57.530 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:16:57.531 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:16:57.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:57.532 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:57.628 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:57.629 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:16:57.629 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:57.675 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:16:57.675 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:16:57.677 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:16:57.678 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:16:57.678 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:57.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[42698]: fuse finished with error 0 and tester_r 0 2021-04-09T18:16:57.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:16:57.791 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:16:57.793 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:57.839 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:16:57.840 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:16:57.840 INFO:teuthology.orchestra.run:waiting for 900 2021-04-09T18:16:57.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:16:57.934 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:16:57.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:16:58.145 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:16:58.146 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:16:58.146 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:16:58.152 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:16:58.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:16:58.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.273 7f6e51d32700 1 Processor -- start 2021-04-09T18:16:58.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.273 7f6e51d32700 1 -- start start 2021-04-09T18:16:58.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e51d32700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6e4c10d010 0x7f6e4c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e51d32700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6e4c0d8490 0x7f6e4c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e51d32700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c10d480 con 0x7f6e4c0d1da0 2021-04-09T18:16:58.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e51d32700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c0d8d30 con 0x7f6e4c10d010 2021-04-09T18:16:58.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e51d32700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c0d9220 con 0x7f6e4c0d8490 2021-04-09T18:16:58.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4b7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6e4c10d010 0x7f6e4c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4affd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6e4c0d1da0 0x7f6e4c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54740/0 (socket says 172.21.15.32:54740) 2021-04-09T18:16:58.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4affd700 1 -- 172.21.15.32:0/893885709 learned_addr learned my addr 172.21.15.32:0/893885709 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:16:58.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4bfff700 1 --2- 172.21.15.32:0/893885709 >> v2:172.21.15.86:3300/0 conn(0x7f6e4c0d8490 0x7f6e4c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4a7fc700 1 -- 172.21.15.32:0/893885709 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761146893 0 0) 0x7f6e34000ce0 con 0x7f6e4c0d1da0 2021-04-09T18:16:58.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4a7fc700 1 -- 172.21.15.32:0/893885709 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e38002fe0 con 0x7f6e4c0d1da0 2021-04-09T18:16:58.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4b7fe700 1 -- 172.21.15.32:0/893885709 >> v2:172.21.15.86:3300/0 conn(0x7f6e4c0d8490 msgr2=0x7f6e4c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:58.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4b7fe700 1 --2- 172.21.15.32:0/893885709 >> v2:172.21.15.86:3300/0 conn(0x7f6e4c0d8490 0x7f6e4c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4b7fe700 1 -- 172.21.15.32:0/893885709 >> v1:172.21.15.32:6789/0 conn(0x7f6e4c0d1da0 legacy=0x7f6e4c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4b7fe700 1 -- 172.21.15.32:0/893885709 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c10e020 con 0x7f6e4c10d010 2021-04-09T18:16:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.274 7f6e4b7fe700 1 --2- 172.21.15.32:0/893885709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c10d010 0x7f6e4c0d1780 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f6e440069d0 tx=0x7f6e44009e40).ready entity=mon.0 client_cookie=1d6b847e85e1eb4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:58.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e4a7fc700 1 -- 172.21.15.32:0/893885709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4400b510 con 0x7f6e4c10d010 2021-04-09T18:16:58.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e4a7fc700 1 -- 172.21.15.32:0/893885709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4400b890 con 0x7f6e4c10d010 2021-04-09T18:16:58.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e4a7fc700 1 -- 172.21.15.32:0/893885709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4400bd70 con 0x7f6e4c10d010 2021-04-09T18:16:58.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e51d32700 1 -- 172.21.15.32:0/893885709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c10d010 msgr2=0x7f6e4c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e51d32700 1 --2- 172.21.15.32:0/893885709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c10d010 0x7f6e4c0d1780 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f6e440069d0 tx=0x7f6e44009e40).stop 2021-04-09T18:16:58.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e51d32700 1 -- 172.21.15.32:0/893885709 shutdown_connections 2021-04-09T18:16:58.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e51d32700 1 --2- 172.21.15.32:0/893885709 >> v2:172.21.15.86:3300/0 conn(0x7f6e4c0d8490 0x7f6e4c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e51d32700 1 --2- 172.21.15.32:0/893885709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c10d010 0x7f6e4c0d1780 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e51d32700 1 -- 172.21.15.32:0/893885709 >> 172.21.15.32:0/893885709 conn(0x7f6e4c0ce5f0 msgr2=0x7f6e4c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:58.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e51d32700 1 -- 172.21.15.32:0/893885709 shutdown_connections 2021-04-09T18:16:58.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.275 7f6e51d32700 1 -- 172.21.15.32:0/893885709 wait complete. 2021-04-09T18:16:58.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.295 7f6e51d32700 1 Processor -- start 2021-04-09T18:16:58.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e51d32700 1 -- start start 2021-04-09T18:16:58.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e51d32700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c1a5210 0x7f6e4c1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e51d32700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4c1a6340 0x7f6e4c1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e51d32700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e4c0abaa0 0x7f6e4c0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e51d32700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c1a6d60 con 0x7f6e4c1a5210 2021-04-09T18:16:58.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e51d32700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0ac4c0 con 0x7f6e4c1a6340 2021-04-09T18:16:58.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e51d32700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0ac6e0 con 0x7f6e4c0abaa0 2021-04-09T18:16:58.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e4b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c1a5210 0x7f6e4c1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e4b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c1a5210 0x7f6e4c1a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45156/0 (socket says 172.21.15.32:45156) 2021-04-09T18:16:58.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e4b7fe700 1 -- 172.21.15.32:0/1858096181 learned_addr learned my addr 172.21.15.32:0/1858096181 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:58.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e4bfff700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e4c0abaa0 0x7f6e4c0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.296 7f6e4affd700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4c1a6340 0x7f6e4c1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e4b7fe700 1 -- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e4c0abaa0 msgr2=0x7f6e4c0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e4b7fe700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e4c0abaa0 0x7f6e4c0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e4b7fe700 1 -- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4c1a6340 msgr2=0x7f6e4c1a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e4b7fe700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4c1a6340 0x7f6e4c1a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e4b7fe700 1 -- 172.21.15.32:0/1858096181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c17a720 con 0x7f6e4c1a5210 2021-04-09T18:16:58.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e4b7fe700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c1a5210 0x7f6e4c1a56c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f6e44009f40 tx=0x7f6e4400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:58.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e43fff700 1 -- 172.21.15.32:0/1858096181 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e44006240 con 0x7f6e4c1a5210 2021-04-09T18:16:58.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e43fff700 1 -- 172.21.15.32:0/1858096181 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e440065c0 con 0x7f6e4c1a5210 2021-04-09T18:16:58.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e43fff700 1 -- 172.21.15.32:0/1858096181 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e440058e0 con 0x7f6e4c1a5210 2021-04-09T18:16:58.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c17a9c0 con 0x7f6e4c1a5210 2021-04-09T18:16:58.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.297 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c17a9c0 con 0x7f6e4c1a5210 2021-04-09T18:16:58.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.298 7f6e43fff700 1 -- 172.21.15.32:0/1858096181 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6e44005660 con 0x7f6e4c1a5210 2021-04-09T18:16:58.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.298 7f6e43fff700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e30018930 0x7f6e3001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.298 7f6e43fff700 1 -- 172.21.15.32:0/1858096181 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6e44032da0 con 0x7f6e4c1a5210 2021-04-09T18:16:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.298 7f6e4affd700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e30018930 0x7f6e3001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.298 7f6e4affd700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e30018930 0x7f6e3001adc0 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:58.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.318 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c1a6d60 con 0x7f6e4c1a5210 2021-04-09T18:16:58.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.321 7f6e43fff700 1 -- 172.21.15.32:0/1858096181 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6e44000cc0 con 0x7f6e4c1a5210 2021-04-09T18:16:58.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.513 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e4c1a6d60 con 0x7f6e4c1a5210 2021-04-09T18:16:58.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.514 7f6e43fff700 1 -- 172.21.15.32:0/1858096181 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2778 (secure 0 0 0) 0x7f6e440065a0 con 0x7f6e4c1a5210 2021-04-09T18:16:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 56 2021-04-09T18:16:58.548 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:58.548 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5880,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:16:50.949963","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:16:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e30018930 msgr2=0x7f6e3001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e30018930 0x7f6e3001adc0 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c1a5210 msgr2=0x7f6e4c1a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c1a5210 0x7f6e4c1a56c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f6e44009f40 tx=0x7f6e4400bd50).stop 2021-04-09T18:16:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 shutdown_connections 2021-04-09T18:16:58.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e30018930 0x7f6e3001adc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e4c0abaa0 0x7f6e4c0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4c1a6340 0x7f6e4c1a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 --2- 172.21.15.32:0/1858096181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e4c1a5210 0x7f6e4c1a56c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 >> 172.21.15.32:0/1858096181 conn(0x7f6e4c1a99c0 msgr2=0x7f6e4c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:58.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.516 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 shutdown_connections 2021-04-09T18:16:58.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.517 7f6e51d32700 1 -- 172.21.15.32:0/1858096181 wait complete. 2021-04-09T18:16:58.559 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 5, "epoch": 56, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 5, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:16:48.944470", "damaged": [], "data_pools": [ 11 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5803": { "addr": "172.21.15.86:6837/3241800545", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6835", "nonce": 3241800545, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 3241800545, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 5803, "incarnation": 53, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 10, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:16:50.949963", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5803 } } } ], "standbys": [ { "addr": "172.21.15.32:6833/1942724067", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 1942724067, "type": "v2" }, { "addr": "172.21.15.32:6833", "nonce": 1942724067, "type": "v1" } ] }, "epoch": 52, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 5880, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.32:6835/3552289149", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6834", "nonce": 3552289149, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 3552289149, "type": "v1" } ] }, "epoch": 52, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 5883, "incarnation": 0, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6836/1208639138", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 1208639138, "type": "v2" }, { "addr": "172.21.15.86:6836", "nonce": 1208639138, "type": "v1" } ] }, "epoch": 52, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55253, "incarnation": 0, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:16:58.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:16:58.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.702 7fbe5fb11700 1 Processor -- start 2021-04-09T18:16:58.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.703 7fbe5fb11700 1 -- start start 2021-04-09T18:16:58.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.703 7fbe5fb11700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbe580d6c80 0x7fbe580d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.703 7fbe5fb11700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe580d2440 0x7fbe5810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.703 7fbe5fb11700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe580d7280 con 0x7fbe580cffa0 2021-04-09T18:16:58.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.703 7fbe5fb11700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe580d7080 con 0x7fbe580d6c80 2021-04-09T18:16:58.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.703 7fbe5fb11700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe580d91e0 con 0x7fbe580d2440 2021-04-09T18:16:58.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.703 7fbe5d12c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbe580d6c80 0x7fbe580d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5d12c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbe580d6c80 0x7fbe580d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45168/0 (socket says 172.21.15.32:45168) 2021-04-09T18:16:58.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5d12c700 1 -- 172.21.15.32:0/2032723730 learned_addr learned my addr 172.21.15.32:0/2032723730 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:58.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5e12e700 1 --2- 172.21.15.32:0/2032723730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe580d2440 0x7fbe5810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5d12c700 1 -- 172.21.15.32:0/2032723730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe580d2440 msgr2=0x7fbe5810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:58.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5d12c700 1 --2- 172.21.15.32:0/2032723730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe580d2440 0x7fbe5810cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5d12c700 1 -- 172.21.15.32:0/2032723730 >> v1:172.21.15.86:6789/0 conn(0x7fbe580cffa0 legacy=0x7fbe580d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5d12c700 1 -- 172.21.15.32:0/2032723730 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe580d2840 con 0x7fbe580d6c80 2021-04-09T18:16:58.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5d12c700 1 --2- 172.21.15.32:0/2032723730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe580d6c80 0x7fbe580d7ff0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7fbe48006700 tx=0x7fbe48009b80).ready entity=mon.0 client_cookie=43603798364425a5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:58.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5c92b700 1 -- 172.21.15.32:0/2032723730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe4800c0c0 con 0x7fbe580d6c80 2021-04-09T18:16:58.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5c92b700 1 -- 172.21.15.32:0/2032723730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4800c440 con 0x7fbe580d6c80 2021-04-09T18:16:58.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5c92b700 1 -- 172.21.15.32:0/2032723730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe4800c920 con 0x7fbe580d6c80 2021-04-09T18:16:58.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5fb11700 1 -- 172.21.15.32:0/2032723730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe580d6c80 msgr2=0x7fbe580d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.704 7fbe5fb11700 1 --2- 172.21.15.32:0/2032723730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe580d6c80 0x7fbe580d7ff0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7fbe48006700 tx=0x7fbe48009b80).stop 2021-04-09T18:16:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.705 7fbe5fb11700 1 -- 172.21.15.32:0/2032723730 shutdown_connections 2021-04-09T18:16:58.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.705 7fbe5fb11700 1 --2- 172.21.15.32:0/2032723730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe580d2440 0x7fbe5810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.705 7fbe5fb11700 1 --2- 172.21.15.32:0/2032723730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe580d6c80 0x7fbe580d7ff0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.705 7fbe5fb11700 1 -- 172.21.15.32:0/2032723730 >> 172.21.15.32:0/2032723730 conn(0x7fbe580ce5c0 msgr2=0x7fbe580cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:58.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.705 7fbe5fb11700 1 -- 172.21.15.32:0/2032723730 shutdown_connections 2021-04-09T18:16:58.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.705 7fbe5fb11700 1 -- 172.21.15.32:0/2032723730 wait complete. 2021-04-09T18:16:58.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.725 7fbe5fb11700 1 Processor -- start 2021-04-09T18:16:58.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.725 7fbe5fb11700 1 -- start start 2021-04-09T18:16:58.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5fb11700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe581929f0 0x7fbe58192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5fb11700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe5808ddc0 0x7fbe5808e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5fb11700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe5808e7e0 0x7fbe5808c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5fb11700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe58193410 con 0x7fbe5808e7e0 2021-04-09T18:16:58.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5fb11700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe580a9140 con 0x7fbe5808ddc0 2021-04-09T18:16:58.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5fb11700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5808ce80 con 0x7fbe581929f0 2021-04-09T18:16:58.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5e12e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe5808e7e0 0x7fbe5808c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5e12e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe5808e7e0 0x7fbe5808c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45172/0 (socket says 172.21.15.32:45172) 2021-04-09T18:16:58.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5e12e700 1 -- 172.21.15.32:0/776670152 learned_addr learned my addr 172.21.15.32:0/776670152 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:58.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5d12c700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe5808ddc0 0x7fbe5808e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5d92d700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe581929f0 0x7fbe58192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5e12e700 1 -- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe581929f0 msgr2=0x7fbe58192df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5e12e700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe581929f0 0x7fbe58192df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5e12e700 1 -- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe5808ddc0 msgr2=0x7fbe5808e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5e12e700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe5808ddc0 0x7fbe5808e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5e12e700 1 -- 172.21.15.32:0/776670152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5808d0a0 con 0x7fbe5808e7e0 2021-04-09T18:16:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5d12c700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe5808ddc0 0x7fbe5808e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.726 7fbe5e12e700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe5808e7e0 0x7fbe5808c860 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fbe50006f20 tx=0x7fbe5000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.727 7fbe4e7fc700 1 -- 172.21.15.32:0/776670152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe50006bb0 con 0x7fbe5808e7e0 2021-04-09T18:16:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.727 7fbe4e7fc700 1 -- 172.21.15.32:0/776670152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5000bd80 con 0x7fbe5808e7e0 2021-04-09T18:16:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.727 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5808d0a0 con 0x7fbe5808e7e0 2021-04-09T18:16:58.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.727 7fbe4e7fc700 1 -- 172.21.15.32:0/776670152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe50002220 con 0x7fbe5808e7e0 2021-04-09T18:16:58.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.727 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5808d0a0 con 0x7fbe5808e7e0 2021-04-09T18:16:58.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.727 7fbe4e7fc700 1 -- 172.21.15.32:0/776670152 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbe50002020 con 0x7fbe5808e7e0 2021-04-09T18:16:58.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.728 7fbe4e7fc700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe40018930 0x7fbe4001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:58.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.728 7fbe4e7fc700 1 -- 172.21.15.32:0/776670152 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbe50032a40 con 0x7fbe5808e7e0 2021-04-09T18:16:58.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.728 7fbe5d92d700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe40018930 0x7fbe4001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:58.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.728 7fbe5d92d700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe40018930 0x7fbe4001ae10 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.747 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe581b70c0 con 0x7fbe5808e7e0 2021-04-09T18:16:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.751 7fbe4e7fc700 1 -- 172.21.15.32:0/776670152 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbe50006370 con 0x7fbe5808e7e0 2021-04-09T18:16:58.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.942 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe580d7280 con 0x7fbe5808e7e0 2021-04-09T18:16:58.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.943 7fbe4e7fc700 1 -- 172.21.15.32:0/776670152 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2778 (secure 0 0 0) 0x7fbe5000bd80 con 0x7fbe5808e7e0 2021-04-09T18:16:58.976 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 56 2021-04-09T18:16:58.977 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:16:58.977 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5880,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:16:50.949963","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:16:58.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe40018930 msgr2=0x7fbe4001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe40018930 0x7fbe4001ae10 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe5808e7e0 msgr2=0x7fbe5808c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:58.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe5808e7e0 0x7fbe5808c860 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fbe50006f20 tx=0x7fbe5000b990).stop 2021-04-09T18:16:58.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 shutdown_connections 2021-04-09T18:16:58.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe40018930 0x7fbe4001ae10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe5808e7e0 0x7fbe5808c860 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe5808ddc0 0x7fbe5808e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 --2- 172.21.15.32:0/776670152 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe581929f0 0x7fbe58192df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:58.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 >> 172.21.15.32:0/776670152 conn(0x7fbe58189b00 msgr2=0x7fbe581921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:58.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 shutdown_connections 2021-04-09T18:16:58.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:58.945 7fbe5fb11700 1 -- 172.21.15.32:0/776670152 wait complete. 2021-04-09T18:16:59.012 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:16:59.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.170 7f7cb82b2700 1 Processor -- start 2021-04-09T18:16:59.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.170 7f7cb82b2700 1 -- start start 2021-04-09T18:16:59.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.170 7f7cb82b2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb010d010 0x7f7cb00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:59.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.170 7f7cb82b2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7cb00d1da0 0x7f7cb00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:59.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.170 7f7cb82b2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7cb00d82c0 0x7f7cb010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:59.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb82b2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb010e780 con 0x7f7cb00d1da0 2021-04-09T18:16:59.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb82b2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb00d8d30 con 0x7f7cb00d82c0 2021-04-09T18:16:59.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb82b2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb00d9220 con 0x7f7cb010d010 2021-04-09T18:16:59.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb58cd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7cb00d1da0 0x7f7cb00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:59.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb58cd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7cb00d1da0 0x7f7cb00d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45182/0 (socket says 172.21.15.32:45182) 2021-04-09T18:16:59.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb58cd700 1 -- 172.21.15.32:0/9145064 learned_addr learned my addr 172.21.15.32:0/9145064 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:59.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb68cf700 1 --2- 172.21.15.32:0/9145064 >> v2:172.21.15.86:3300/0 conn(0x7f7cb00d82c0 0x7f7cb010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:59.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb60ce700 1 --2- 172.21.15.32:0/9145064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb010d010 0x7f7cb00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:59.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb58cd700 1 -- 172.21.15.32:0/9145064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb010d010 msgr2=0x7f7cb00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:59.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb58cd700 1 --2- 172.21.15.32:0/9145064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb010d010 0x7f7cb00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:59.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb58cd700 1 -- 172.21.15.32:0/9145064 >> v2:172.21.15.86:3300/0 conn(0x7f7cb00d82c0 msgr2=0x7f7cb010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:16:59.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb58cd700 1 --2- 172.21.15.32:0/9145064 >> v2:172.21.15.86:3300/0 conn(0x7f7cb00d82c0 0x7f7cb010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:59.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb58cd700 1 -- 172.21.15.32:0/9145064 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb010d480 con 0x7f7cb00d1da0 2021-04-09T18:16:59.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb68cf700 1 --2- 172.21.15.32:0/9145064 >> v2:172.21.15.86:3300/0 conn(0x7f7cb00d82c0 0x7f7cb010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:59.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb58cd700 1 --2- 172.21.15.32:0/9145064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb00d1da0 0x7f7cb00d7ca0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f7ca0006700 tx=0x7f7ca0009b00).ready entity=mon.0 client_cookie=af3bd0b243186c09 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:59.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb50cc700 1 -- 172.21.15.32:0/9145064 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ca000c0c0 con 0x7f7cb00d1da0 2021-04-09T18:16:59.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb50cc700 1 -- 172.21.15.32:0/9145064 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca000c400 con 0x7f7cb00d1da0 2021-04-09T18:16:59.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.171 7f7cb50cc700 1 -- 172.21.15.32:0/9145064 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ca000c8e0 con 0x7f7cb00d1da0 2021-04-09T18:16:59.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.172 7f7cb82b2700 1 -- 172.21.15.32:0/9145064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb00d1da0 msgr2=0x7f7cb00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:59.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.172 7f7cb82b2700 1 --2- 172.21.15.32:0/9145064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb00d1da0 0x7f7cb00d7ca0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f7ca0006700 tx=0x7f7ca0009b00).stop 2021-04-09T18:16:59.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.172 7f7cb82b2700 1 -- 172.21.15.32:0/9145064 shutdown_connections 2021-04-09T18:16:59.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.172 7f7cb82b2700 1 --2- 172.21.15.32:0/9145064 >> v2:172.21.15.86:3300/0 conn(0x7f7cb00d82c0 0x7f7cb010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.172 7f7cb82b2700 1 --2- 172.21.15.32:0/9145064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb00d1da0 0x7f7cb00d7ca0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.172 7f7cb82b2700 1 --2- 172.21.15.32:0/9145064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb010d010 0x7f7cb00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.172 7f7cb82b2700 1 -- 172.21.15.32:0/9145064 >> 172.21.15.32:0/9145064 conn(0x7f7cb00ce5f0 msgr2=0x7f7cb00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:16:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.172 7f7cb82b2700 1 -- 172.21.15.32:0/9145064 shutdown_connections 2021-04-09T18:16:59.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.172 7f7cb82b2700 1 -- 172.21.15.32:0/9145064 wait complete. 2021-04-09T18:16:59.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.192 7f7cb82b2700 1 Processor -- start 2021-04-09T18:16:59.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.192 7f7cb82b2700 1 -- start start 2021-04-09T18:16:59.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb82b2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cb0093dc0 0x7f7cb00941c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:59.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb82b2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb00b5710 0x7f7cb00b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:59.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb82b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb0092670 0x7f7cb0092a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:59.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb82b2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb00b6130 con 0x7f7cb0092670 2021-04-09T18:16:59.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb82b2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb0099430 con 0x7f7cb0093dc0 2021-04-09T18:16:59.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb82b2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb0093090 con 0x7f7cb00b5710 2021-04-09T18:16:59.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb68cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb0092670 0x7f7cb0092a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:59.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb68cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb0092670 0x7f7cb0092a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45184/0 (socket says 172.21.15.32:45184) 2021-04-09T18:16:59.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb68cf700 1 -- 172.21.15.32:0/589718075 learned_addr learned my addr 172.21.15.32:0/589718075 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:16:59.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb60ce700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cb0093dc0 0x7f7cb00941c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:59.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb58cd700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb00b5710 0x7f7cb00b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:59.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb68cf700 1 -- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb00b5710 msgr2=0x7f7cb00b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:59.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb68cf700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb00b5710 0x7f7cb00b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:59.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb68cf700 1 -- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cb0093dc0 msgr2=0x7f7cb00941c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:16:59.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb68cf700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cb0093dc0 0x7f7cb00941c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:16:59.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb68cf700 1 -- 172.21.15.32:0/589718075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb00932b0 con 0x7f7cb0092670 2021-04-09T18:16:59.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb60ce700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cb0093dc0 0x7f7cb00941c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:16:59.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.193 7f7cb68cf700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb0092670 0x7f7cb0092a70 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f7ca8009920 tx=0x7f7ca800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:59.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.194 7f7ca6ffd700 1 -- 172.21.15.32:0/589718075 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ca8006240 con 0x7f7cb0092670 2021-04-09T18:16:59.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.194 7f7ca6ffd700 1 -- 172.21.15.32:0/589718075 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca8006540 con 0x7f7cb0092670 2021-04-09T18:16:59.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.194 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb00932b0 con 0x7f7cb0092670 2021-04-09T18:16:59.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.194 7f7ca6ffd700 1 -- 172.21.15.32:0/589718075 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ca8005930 con 0x7f7cb0092670 2021-04-09T18:16:59.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.194 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb00932b0 con 0x7f7cb0092670 2021-04-09T18:16:59.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.194 7f7ca6ffd700 1 -- 172.21.15.32:0/589718075 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7ca8006030 con 0x7f7cb0092670 2021-04-09T18:16:59.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.194 7f7ca6ffd700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c980188d0 0x7f7c9801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:16:59.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.194 7f7ca6ffd700 1 -- 172.21.15.32:0/589718075 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7ca8032da0 con 0x7f7cb0092670 2021-04-09T18:16:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.195 7f7cb60ce700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c980188d0 0x7f7c9801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:16:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.195 7f7cb60ce700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c980188d0 0x7f7c9801add0 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:16:59.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.215 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb0099410 con 0x7f7cb0092670 2021-04-09T18:16:59.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.218 7f7ca6ffd700 1 -- 172.21.15.32:0/589718075 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7ca8006540 con 0x7f7cb0092670 2021-04-09T18:16:59.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:16:59.411 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f7cb00b51e0 con 0x7f7cb0092670 2021-04-09T18:17:00.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.365 7f7ca6ffd700 1 -- 172.21.15.32:0/589718075 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v57) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7ca8006030 con 0x7f7cb0092670 2021-04-09T18:17:00.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c980188d0 msgr2=0x7f7c9801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:00.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c980188d0 0x7f7c9801add0 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb0092670 msgr2=0x7f7cb0092a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb0092670 0x7f7cb0092a70 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f7ca8009920 tx=0x7f7ca800bd40).stop 2021-04-09T18:17:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 shutdown_connections 2021-04-09T18:17:00.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c980188d0 0x7f7c9801add0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cb0092670 0x7f7cb0092a70 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cb00b5710 0x7f7cb00b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 --2- 172.21.15.32:0/589718075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cb0093dc0 0x7f7cb00941c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 >> 172.21.15.32:0/589718075 conn(0x7f7cb00950b0 msgr2=0x7f7cb00b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:00.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 shutdown_connections 2021-04-09T18:17:00.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.381 7f7cb82b2700 1 -- 172.21.15.32:0/589718075 wait complete. 2021-04-09T18:17:00.459 INFO:tasks.cephfs.test_failover:Waiting for cluster to grow. 2021-04-09T18:17:00.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:00.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.613 7f4861387700 1 Processor -- start 2021-04-09T18:17:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.613 7f4861387700 1 -- start start 2021-04-09T18:17:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f4861387700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f485c0cffa0 0x7f485c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f4861387700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0d7280 con 0x7f485c0d6c80 2021-04-09T18:17:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f4861387700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0d83f0 con 0x7f485c0d7800 2021-04-09T18:17:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f4861387700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c0d7080 con 0x7f485c0cffa0 2021-04-09T18:17:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485b87e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f485c0d6c80 0x7f485c10cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54794/0 (socket says 172.21.15.32:54794) 2021-04-09T18:17:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485b87e700 1 -- 172.21.15.32:0/2505875101 learned_addr learned my addr 172.21.15.32:0/2505875101 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485b07d700 1 --2- 172.21.15.32:0/2505875101 >> v2:172.21.15.86:3300/0 conn(0x7f485c0cffa0 0x7f485c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:00.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485a07b700 1 -- 172.21.15.32:0/2505875101 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710056536 0 0) 0x7f4854000d30 con 0x7f485c0d6c80 2021-04-09T18:17:00.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485a07b700 1 -- 172.21.15.32:0/2505875101 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840002fe0 con 0x7f485c0d6c80 2021-04-09T18:17:00.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485a07b700 1 -- 172.21.15.32:0/2505875101 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363943707 0 0) 0x7f4848001080 con 0x7f485c0d7800 2021-04-09T18:17:00.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485a07b700 1 -- 172.21.15.32:0/2505875101 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840003210 con 0x7f485c0d7800 2021-04-09T18:17:00.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485a07b700 1 -- 172.21.15.32:0/2505875101 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3021241249 0 0) 0x7f4854001270 con 0x7f485c0d6c80 2021-04-09T18:17:00.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485b07d700 1 -- 172.21.15.32:0/2505875101 >> v1:172.21.15.86:6789/0 conn(0x7f485c0d7800 legacy=0x7f485c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:00.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485b07d700 1 -- 172.21.15.32:0/2505875101 >> v1:172.21.15.32:6789/0 conn(0x7f485c0d6c80 legacy=0x7f485c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:00.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.614 7f485b07d700 1 -- 172.21.15.32:0/2505875101 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c0d91e0 con 0x7f485c0cffa0 2021-04-09T18:17:00.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.615 7f485b07d700 1 --2- 172.21.15.32:0/2505875101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c0cffa0 0x7f485c0d2440 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f484c006650 tx=0x7f484c009a80).ready entity=mon.1 client_cookie=ba3334e406e0c217 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:00.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.615 7f485a07b700 1 -- 172.21.15.32:0/2505875101 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f484c0009b0 con 0x7f485c0cffa0 2021-04-09T18:17:00.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.615 7f485a07b700 1 -- 172.21.15.32:0/2505875101 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c00c0c0 con 0x7f485c0cffa0 2021-04-09T18:17:00.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.615 7f485a07b700 1 -- 172.21.15.32:0/2505875101 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f484c00c510 con 0x7f485c0cffa0 2021-04-09T18:17:00.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.615 7f4861387700 1 -- 172.21.15.32:0/2505875101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c0cffa0 msgr2=0x7f485c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.615 7f4861387700 1 --2- 172.21.15.32:0/2505875101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c0cffa0 0x7f485c0d2440 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f484c006650 tx=0x7f484c009a80).stop 2021-04-09T18:17:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.616 7f4861387700 1 -- 172.21.15.32:0/2505875101 shutdown_connections 2021-04-09T18:17:00.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.616 7f4861387700 1 --2- 172.21.15.32:0/2505875101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c0cffa0 0x7f485c0d2440 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.616 7f4861387700 1 -- 172.21.15.32:0/2505875101 >> 172.21.15.32:0/2505875101 conn(0x7f485c0ce5c0 msgr2=0x7f485c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:00.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.616 7f4861387700 1 -- 172.21.15.32:0/2505875101 shutdown_connections 2021-04-09T18:17:00.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.616 7f4861387700 1 -- 172.21.15.32:0/2505875101 wait complete. 2021-04-09T18:17:00.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f4861387700 1 Processor -- start 2021-04-09T18:17:00.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f4861387700 1 -- start start 2021-04-09T18:17:00.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f4861387700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c1acff0 0x7f485c1847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:00.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f4861387700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f485c183fc0 0x7f485c1a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:00.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f4861387700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c186ff0 0x7f485c1873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:00.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f4861387700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c188c30 con 0x7f485c186ff0 2021-04-09T18:17:00.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f4861387700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c1843c0 con 0x7f485c1acff0 2021-04-09T18:17:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f4861387700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c1a5c20 con 0x7f485c183fc0 2021-04-09T18:17:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f485b87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c186ff0 0x7f485c1873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f485b87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c186ff0 0x7f485c1873f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45208/0 (socket says 172.21.15.32:45208) 2021-04-09T18:17:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.636 7f485b87e700 1 -- 172.21.15.32:0/1203462271 learned_addr learned my addr 172.21.15.32:0/1203462271 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485b07d700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c1acff0 0x7f485c1847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:00.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485a87c700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f485c183fc0 0x7f485c1a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485b87e700 1 -- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f485c183fc0 msgr2=0x7f485c1a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485b87e700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f485c183fc0 0x7f485c1a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485b87e700 1 -- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c1acff0 msgr2=0x7f485c1847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:00.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485b87e700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c1acff0 0x7f485c1847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485b87e700 1 -- 172.21.15.32:0/1203462271 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c187a10 con 0x7f485c186ff0 2021-04-09T18:17:00.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485a87c700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f485c183fc0 0x7f485c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:00.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485b07d700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c1acff0 0x7f485c1847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:00.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f485b87e700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c186ff0 0x7f485c1873f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f485400a900 tx=0x7f485400aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:00.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f4853fff700 1 -- 172.21.15.32:0/1203462271 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4854004930 con 0x7f485c186ff0 2021-04-09T18:17:00.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f4853fff700 1 -- 172.21.15.32:0/1203462271 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4854004bf0 con 0x7f485c186ff0 2021-04-09T18:17:00.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f4861387700 1 -- 172.21.15.32:0/1203462271 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c187a10 con 0x7f485c186ff0 2021-04-09T18:17:00.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f4853fff700 1 -- 172.21.15.32:0/1203462271 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48540072a0 con 0x7f485c186ff0 2021-04-09T18:17:00.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.637 7f4861387700 1 -- 172.21.15.32:0/1203462271 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c187a10 con 0x7f485c186ff0 2021-04-09T18:17:00.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.638 7f4853fff700 1 -- 172.21.15.32:0/1203462271 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4854004720 con 0x7f485c186ff0 2021-04-09T18:17:00.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.638 7f4853fff700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48380188d0 0x7f483801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:00.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.638 7f4853fff700 1 -- 172.21.15.32:0/1203462271 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4854032d70 con 0x7f485c186ff0 2021-04-09T18:17:00.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.638 7f485b07d700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48380188d0 0x7f483801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:00.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.638 7f485b07d700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48380188d0 0x7f483801add0 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.658 7f4861387700 1 -- 172.21.15.32:0/1203462271 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c182dc0 con 0x7f485c186ff0 2021-04-09T18:17:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.661 7f4853fff700 1 -- 172.21.15.32:0/1203462271 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4854007560 con 0x7f485c186ff0 2021-04-09T18:17:00.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.853 7f4861387700 1 -- 172.21.15.32:0/1203462271 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f485c0d7280 con 0x7f485c186ff0 2021-04-09T18:17:00.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.853 7f4853fff700 1 -- 172.21.15.32:0/1203462271 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2794 (secure 0 0 0) 0x7f4854004bf0 con 0x7f485c186ff0 2021-04-09T18:17:00.889 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 58 2021-04-09T18:17:00.890 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:00.890 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":58,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:00.366240","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5803,"mds_1":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_5880":{"gid":5880,"name":"c","rank":1,"incarnation":58,"state":"up:creating","state_seq":2,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:00.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 -- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48380188d0 msgr2=0x7f483801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:00.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48380188d0 0x7f483801add0 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 -- 172.21.15.32:0/1203462271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c186ff0 msgr2=0x7f485c1873f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:00.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c186ff0 0x7f485c1873f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f485400a900 tx=0x7f485400aca0).stop 2021-04-09T18:17:00.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 -- 172.21.15.32:0/1203462271 shutdown_connections 2021-04-09T18:17:00.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48380188d0 0x7f483801add0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c186ff0 0x7f485c1873f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f485c183fc0 0x7f485c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 --2- 172.21.15.32:0/1203462271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f485c1acff0 0x7f485c1847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:00.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 -- 172.21.15.32:0/1203462271 >> 172.21.15.32:0/1203462271 conn(0x7f485c09abb0 msgr2=0x7f485c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:00.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 -- 172.21.15.32:0/1203462271 shutdown_connections 2021-04-09T18:17:00.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:00.856 7f4861387700 1 -- 172.21.15.32:0/1203462271 wait complete. 2021-04-09T18:17:01.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:00.366240', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 5803, 'mds_1': 5880}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_5880': {'gid': 5880, 'name': 'c', 'rank': 1, 'incarnation': 58, 'state': 'up:creating', 'state_seq': 2, 'addr': '172.21.15.32:6833/1942724067', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1942724067}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 1942724067}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:01.329 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5880:up:creating 2021-04-09T18:17:02.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:02.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.456 7f8b3aeae700 1 Processor -- start 2021-04-09T18:17:02.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.456 7f8b3aeae700 1 -- start start 2021-04-09T18:17:02.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.456 7f8b3aeae700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8b3410d010 0x7f8b340d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:02.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b3aeae700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b340d8490 0x7f8b3410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:02.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b3aeae700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3410d480 con 0x7f8b340d1da0 2021-04-09T18:17:02.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b3aeae700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b340d8d30 con 0x7f8b340d8490 2021-04-09T18:17:02.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b3aeae700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b340d9220 con 0x7f8b3410d010 2021-04-09T18:17:02.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b394cb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b340d8490 0x7f8b3410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b33fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8b340d1da0 0x7f8b340d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54810/0 (socket says 172.21.15.32:54810) 2021-04-09T18:17:02.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b33fff700 1 -- 172.21.15.32:0/3847973032 learned_addr learned my addr 172.21.15.32:0/3847973032 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b38cca700 1 --2- 172.21.15.32:0/3847973032 >> v2:172.21.15.86:3300/0 conn(0x7f8b3410d010 0x7f8b340d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b394cb700 1 -- 172.21.15.32:0/3847973032 >> v2:172.21.15.86:3300/0 conn(0x7f8b3410d010 msgr2=0x7f8b340d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b394cb700 1 --2- 172.21.15.32:0/3847973032 >> v2:172.21.15.86:3300/0 conn(0x7f8b3410d010 0x7f8b340d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b394cb700 1 -- 172.21.15.32:0/3847973032 >> v1:172.21.15.32:6789/0 conn(0x7f8b340d1da0 legacy=0x7f8b340d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:02.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b394cb700 1 -- 172.21.15.32:0/3847973032 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3410e020 con 0x7f8b340d8490 2021-04-09T18:17:02.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b337fe700 1 -- 172.21.15.32:0/3847973032 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329598368 0 0) 0x7f8b20000ce0 con 0x7f8b340d1da0 2021-04-09T18:17:02.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b38cca700 1 --2- 172.21.15.32:0/3847973032 >> v2:172.21.15.86:3300/0 conn(0x7f8b3410d010 0x7f8b340d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:02.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b394cb700 1 --2- 172.21.15.32:0/3847973032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340d8490 0x7f8b3410da00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f8b2c006e30 tx=0x7f8b2c009e60).ready entity=mon.0 client_cookie=89d6e0444ce1f896 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:02.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b337fe700 1 -- 172.21.15.32:0/3847973032 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b2c006a50 con 0x7f8b340d8490 2021-04-09T18:17:02.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b337fe700 1 -- 172.21.15.32:0/3847973032 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c00b300 con 0x7f8b340d8490 2021-04-09T18:17:02.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b337fe700 1 -- 172.21.15.32:0/3847973032 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b2c00b750 con 0x7f8b340d8490 2021-04-09T18:17:02.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b3aeae700 1 -- 172.21.15.32:0/3847973032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340d8490 msgr2=0x7f8b3410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:02.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.457 7f8b3aeae700 1 --2- 172.21.15.32:0/3847973032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340d8490 0x7f8b3410da00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f8b2c006e30 tx=0x7f8b2c009e60).stop 2021-04-09T18:17:02.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.458 7f8b3aeae700 1 -- 172.21.15.32:0/3847973032 shutdown_connections 2021-04-09T18:17:02.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.458 7f8b3aeae700 1 --2- 172.21.15.32:0/3847973032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b340d8490 0x7f8b3410da00 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.458 7f8b3aeae700 1 --2- 172.21.15.32:0/3847973032 >> v2:172.21.15.86:3300/0 conn(0x7f8b3410d010 0x7f8b340d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.458 7f8b3aeae700 1 -- 172.21.15.32:0/3847973032 >> 172.21.15.32:0/3847973032 conn(0x7f8b340ce5f0 msgr2=0x7f8b340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:02.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.458 7f8b3aeae700 1 -- 172.21.15.32:0/3847973032 shutdown_connections 2021-04-09T18:17:02.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.458 7f8b3aeae700 1 -- 172.21.15.32:0/3847973032 wait complete. 2021-04-09T18:17:02.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.478 7f8b3aeae700 1 Processor -- start 2021-04-09T18:17:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.478 7f8b3aeae700 1 -- start start 2021-04-09T18:17:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b3aeae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b341abcb0 0x7f8b341ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b3aeae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b34191140 0x7f8b34191540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:02.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b3aeae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b34191b60 0x7f8b341a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:02.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b3aeae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b34191f60 con 0x7f8b341abcb0 2021-04-09T18:17:02.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b3aeae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b341ac6d0 con 0x7f8b34191b60 2021-04-09T18:17:02.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b3aeae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b341aa5c0 con 0x7f8b34191140 2021-04-09T18:17:02.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b38cca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b341abcb0 0x7f8b341ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:02.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b38cca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b341abcb0 0x7f8b341ac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45222/0 (socket says 172.21.15.32:45222) 2021-04-09T18:17:02.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b38cca700 1 -- 172.21.15.32:0/3382061351 learned_addr learned my addr 172.21.15.32:0/3382061351 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:02.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b394cb700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b34191b60 0x7f8b341a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:02.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b33fff700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b34191140 0x7f8b34191540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b38cca700 1 -- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b34191140 msgr2=0x7f8b34191540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:02.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b38cca700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b34191140 0x7f8b34191540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b38cca700 1 -- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b34191b60 msgr2=0x7f8b341a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b38cca700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b34191b60 0x7f8b341a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b38cca700 1 -- 172.21.15.32:0/3382061351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b341aa7e0 con 0x7f8b341abcb0 2021-04-09T18:17:02.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.479 7f8b33fff700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b34191140 0x7f8b34191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:02.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.480 7f8b38cca700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b341abcb0 0x7f8b341ac0b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f8b28006f20 tx=0x7f8b2800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:02.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.480 7f8b317fa700 1 -- 172.21.15.32:0/3382061351 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b2800bd80 con 0x7f8b341abcb0 2021-04-09T18:17:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.480 7f8b317fa700 1 -- 172.21.15.32:0/3382061351 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b28000c90 con 0x7f8b341abcb0 2021-04-09T18:17:02.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.480 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b341aa7e0 con 0x7f8b341abcb0 2021-04-09T18:17:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.480 7f8b317fa700 1 -- 172.21.15.32:0/3382061351 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b280022b0 con 0x7f8b341abcb0 2021-04-09T18:17:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.480 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b341aa7e0 con 0x7f8b341abcb0 2021-04-09T18:17:02.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.480 7f8b317fa700 1 -- 172.21.15.32:0/3382061351 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8b2800a000 con 0x7f8b341abcb0 2021-04-09T18:17:02.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.480 7f8b317fa700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b14018930 0x7f8b1401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:02.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.481 7f8b317fa700 1 -- 172.21.15.32:0/3382061351 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8b28032a30 con 0x7f8b341abcb0 2021-04-09T18:17:02.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.481 7f8b33fff700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b14018930 0x7f8b1401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:02.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.481 7f8b33fff700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b14018930 0x7f8b1401ae10 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:02.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.501 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b34191f60 con 0x7f8b341abcb0 2021-04-09T18:17:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.504 7f8b317fa700 1 -- 172.21.15.32:0/3382061351 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8b28006bb0 con 0x7f8b341abcb0 2021-04-09T18:17:02.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.697 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b34191f60 con 0x7f8b341abcb0 2021-04-09T18:17:02.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.698 7f8b317fa700 1 -- 172.21.15.32:0/3382061351 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2792 (secure 0 0 0) 0x7f8b28006bb0 con 0x7f8b341abcb0 2021-04-09T18:17:02.731 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 59 2021-04-09T18:17:02.732 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:02.733 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":59,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:01.371432","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5803,"mds_1":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_5880":{"gid":5880,"name":"c","rank":1,"incarnation":58,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:02.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b14018930 msgr2=0x7f8b1401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b14018930 0x7f8b1401ae10 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b341abcb0 msgr2=0x7f8b341ac0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b341abcb0 0x7f8b341ac0b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f8b28006f20 tx=0x7f8b2800b990).stop 2021-04-09T18:17:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 shutdown_connections 2021-04-09T18:17:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b14018930 0x7f8b1401ae10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b34191b60 0x7f8b341a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b34191140 0x7f8b34191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 --2- 172.21.15.32:0/3382061351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b341abcb0 0x7f8b341ac0b0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:02.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 >> 172.21.15.32:0/3382061351 conn(0x7f8b341837d0 msgr2=0x7f8b34195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:02.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 shutdown_connections 2021-04-09T18:17:02.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:02.700 7f8b3aeae700 1 -- 172.21.15.32:0/3382061351 wait complete. 2021-04-09T18:17:03.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:01.371432', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 5803, 'mds_1': 5880}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_5880': {'gid': 5880, 'name': 'c', 'rank': 1, 'incarnation': 58, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6833/1942724067', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1942724067}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 1942724067}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:03.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:17:03.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:17:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:17:03.281 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:17:03.282 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:17:03.282 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 5803, 2021-04-09T18:17:03.283 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:17:03.283 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:17:03.283 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 14.30284254, 2021-04-09T18:17:03.284 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 59, 2021-04-09T18:17:03.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 63, 2021-04-09T18:17:03.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 63, 2021-04-09T18:17:03.288 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 18.759731438999999 2021-04-09T18:17:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:17:03.292 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5803, "want_state": "up:active", "state": "up:active", "rank_uptime": 14.30284254, "mdsmap_epoch": 59, "osdmap_epoch": 63, "osdmap_epoch_barrier": 63, "uptime": 18.759731438999999 } 2021-04-09T18:17:03.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:17:03.442 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:17:03.442 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:17:03.443 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:17:03.443 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5880, 2021-04-09T18:17:03.443 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:17:03.444 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:17:03.444 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 3.0396103370000001, 2021-04-09T18:17:03.444 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 59, 2021-04-09T18:17:03.445 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 63, 2021-04-09T18:17:03.445 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 63, 2021-04-09T18:17:03.446 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 18.901766264999999 2021-04-09T18:17:03.446 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:17:03.450 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 5880, "want_state": "up:active", "state": "up:active", "rank_uptime": 3.0396103370000001, "mdsmap_epoch": 59, "osdmap_epoch": 63, "osdmap_epoch_barrier": 63, "uptime": 18.901766264999999 } 2021-04-09T18:17:03.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2021-04-09T18:17:03.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.579 7f798c16e700 1 Processor -- start 2021-04-09T18:17:03.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.579 7f798c16e700 1 -- start start 2021-04-09T18:17:03.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f798c16e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f798410d010 0x7f79840d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:03.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f798c16e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840d8490 0x7f798410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:03.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f798c16e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798410d480 con 0x7f79840d1da0 2021-04-09T18:17:03.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f798c16e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79840d8d30 con 0x7f798410d010 2021-04-09T18:17:03.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f798c16e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79840d9220 con 0x7f79840d8490 2021-04-09T18:17:03.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f798a78b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840d8490 0x7f798410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:03.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f7989f8a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f798410d010 0x7f79840d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:03.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f7989789700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f79840d1da0 0x7f79840d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52748/0 (socket says 172.21.15.32:52748) 2021-04-09T18:17:03.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f7989789700 1 -- 172.21.15.32:0/2546981890 learned_addr learned my addr 172.21.15.32:0/2546981890 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:03.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.580 7f7988f88700 1 -- 172.21.15.32:0/2546981890 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836593993 0 0) 0x7f79740010c0 con 0x7f79840d1da0 2021-04-09T18:17:03.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f7988f88700 1 -- 172.21.15.32:0/2546981890 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7970002fe0 con 0x7f79840d1da0 2021-04-09T18:17:03.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f798a78b700 1 -- 172.21.15.32:0/2546981890 >> v2:172.21.15.86:3300/0 conn(0x7f798410d010 msgr2=0x7f79840d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:03.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f798a78b700 1 --2- 172.21.15.32:0/2546981890 >> v2:172.21.15.86:3300/0 conn(0x7f798410d010 0x7f79840d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:03.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f798a78b700 1 -- 172.21.15.32:0/2546981890 >> v1:172.21.15.86:6789/0 conn(0x7f79840d1da0 legacy=0x7f79840d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:03.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f798a78b700 1 -- 172.21.15.32:0/2546981890 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798410e020 con 0x7f79840d8490 2021-04-09T18:17:03.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f7989f8a700 1 --2- 172.21.15.32:0/2546981890 >> v2:172.21.15.86:3300/0 conn(0x7f798410d010 0x7f79840d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:03.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f798a78b700 1 --2- 172.21.15.32:0/2546981890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840d8490 0x7f798410da00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f797c006e50 tx=0x7f797c009e70).ready entity=mon.2 client_cookie=8b6721c5921a2e20 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:03.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f7988f88700 1 -- 172.21.15.32:0/2546981890 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f797c006a20 con 0x7f79840d8490 2021-04-09T18:17:03.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f7988f88700 1 -- 172.21.15.32:0/2546981890 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c006790 con 0x7f79840d8490 2021-04-09T18:17:03.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f7988f88700 1 -- 172.21.15.32:0/2546981890 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f797c00b700 con 0x7f79840d8490 2021-04-09T18:17:03.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f798c16e700 1 -- 172.21.15.32:0/2546981890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840d8490 msgr2=0x7f798410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:03.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.581 7f798c16e700 1 --2- 172.21.15.32:0/2546981890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840d8490 0x7f798410da00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f797c006e50 tx=0x7f797c009e70).stop 2021-04-09T18:17:03.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.582 7f798c16e700 1 -- 172.21.15.32:0/2546981890 shutdown_connections 2021-04-09T18:17:03.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.582 7f798c16e700 1 --2- 172.21.15.32:0/2546981890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840d8490 0x7f798410da00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:03.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.582 7f798c16e700 1 --2- 172.21.15.32:0/2546981890 >> v2:172.21.15.86:3300/0 conn(0x7f798410d010 0x7f79840d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:03.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.582 7f798c16e700 1 -- 172.21.15.32:0/2546981890 >> 172.21.15.32:0/2546981890 conn(0x7f79840ce5f0 msgr2=0x7f79840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:03.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.582 7f798c16e700 1 -- 172.21.15.32:0/2546981890 shutdown_connections 2021-04-09T18:17:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.582 7f798c16e700 1 -- 172.21.15.32:0/2546981890 wait complete. 2021-04-09T18:17:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.602 7f798c16e700 1 Processor -- start 2021-04-09T18:17:03.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.602 7f798c16e700 1 -- start start 2021-04-09T18:17:03.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798c16e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840b2140 0x7f798408b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:03.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798c16e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f798409f550 0x7f798409f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798c16e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7984197c50 0x7f7984198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798c16e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798409f010 con 0x7f7984197c50 2021-04-09T18:17:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798c16e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798409ff70 con 0x7f798409f550 2021-04-09T18:17:03.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798c16e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7984198670 con 0x7f79840b2140 2021-04-09T18:17:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798a78b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7984197c50 0x7f7984198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798a78b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7984197c50 0x7f7984198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45240/0 (socket says 172.21.15.32:45240) 2021-04-09T18:17:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798a78b700 1 -- 172.21.15.32:0/85663487 learned_addr learned my addr 172.21.15.32:0/85663487 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:03.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f7989f8a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840b2140 0x7f798408b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:03.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f7989789700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f798409f550 0x7f798409f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:03.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798a78b700 1 -- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840b2140 msgr2=0x7f798408b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:03.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798a78b700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840b2140 0x7f798408b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798a78b700 1 -- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f798409f550 msgr2=0x7f798409f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:03.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798a78b700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f798409f550 0x7f798409f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:03.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798a78b700 1 -- 172.21.15.32:0/85663487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7984198890 con 0x7f7984197c50 2021-04-09T18:17:03.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798a78b700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7984197c50 0x7f7984198050 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f797c006e60 tx=0x7f797c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:03.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f797a7fc700 1 -- 172.21.15.32:0/85663487 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f797c009130 con 0x7f7984197c50 2021-04-09T18:17:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f797a7fc700 1 -- 172.21.15.32:0/85663487 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c009430 con 0x7f7984197c50 2021-04-09T18:17:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f798c16e700 1 -- 172.21.15.32:0/85663487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7984198890 con 0x7f7984197c50 2021-04-09T18:17:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.603 7f797a7fc700 1 -- 172.21.15.32:0/85663487 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f797c001390 con 0x7f7984197c50 2021-04-09T18:17:03.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.604 7f798c16e700 1 -- 172.21.15.32:0/85663487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7984198890 con 0x7f7984197c50 2021-04-09T18:17:03.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.604 7f797a7fc700 1 -- 172.21.15.32:0/85663487 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f797c009130 con 0x7f7984197c50 2021-04-09T18:17:03.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.604 7f797a7fc700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f796c018930 0x7f796c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:03.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.604 7f797a7fc700 1 -- 172.21.15.32:0/85663487 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f797c032d50 con 0x7f7984197c50 2021-04-09T18:17:03.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.604 7f7989f8a700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f796c018930 0x7f796c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:03.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.605 7f7989f8a700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f796c018930 0x7f796c01adc0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:03.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.624 7f798c16e700 1 -- 172.21.15.32:0/85663487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798409f010 con 0x7f7984197c50 2021-04-09T18:17:03.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.628 7f797a7fc700 1 -- 172.21.15.32:0/85663487 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f797c009430 con 0x7f7984197c50 2021-04-09T18:17:03.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:03.823 7f798c16e700 1 -- 172.21.15.32:0/85663487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "true"} v 0) v1 -- 0x7f798409f010 con 0x7f7984197c50 2021-04-09T18:17:04.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.384 7f797a7fc700 1 -- 172.21.15.32:0/85663487 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "true"}]=0 cephfs marked down. v60) v1 ==== 125+0+0 (secure 0 0 0) 0x7f797c009430 con 0x7f7984197c50 2021-04-09T18:17:04.418 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked down. 2021-04-09T18:17:04.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 -- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f796c018930 msgr2=0x7f796c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:04.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f796c018930 0x7f796c01adc0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 -- 172.21.15.32:0/85663487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7984197c50 msgr2=0x7f7984198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:04.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7984197c50 0x7f7984198050 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f797c006e60 tx=0x7f797c00bcf0).stop 2021-04-09T18:17:04.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 -- 172.21.15.32:0/85663487 shutdown_connections 2021-04-09T18:17:04.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f796c018930 0x7f796c01adc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7984197c50 0x7f7984198050 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f798409f550 0x7f798409f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 --2- 172.21.15.32:0/85663487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79840b2140 0x7f798408b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 -- 172.21.15.32:0/85663487 >> 172.21.15.32:0/85663487 conn(0x7f79840b27e0 msgr2=0x7f79841b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 -- 172.21.15.32:0/85663487 shutdown_connections 2021-04-09T18:17:04.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.390 7f798c16e700 1 -- 172.21.15.32:0/85663487 wait complete. 2021-04-09T18:17:04.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:04.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.555 7f6a9f3dd700 1 Processor -- start 2021-04-09T18:17:04.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.555 7f6a9f3dd700 1 -- start start 2021-04-09T18:17:04.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9f3dd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a980cffa0 0x7f6a980d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:04.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9f3dd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a980d7800 0x7f6a980d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:04.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9f3dd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a980d7280 con 0x7f6a980d5c70 2021-04-09T18:17:04.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9f3dd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a980d8580 con 0x7f6a980d7800 2021-04-09T18:17:04.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9f3dd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a980d91e0 con 0x7f6a980cffa0 2021-04-09T18:17:04.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9d9fa700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6a980d5c70 0x7f6a980d6070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54836/0 (socket says 172.21.15.32:54836) 2021-04-09T18:17:04.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9d9fa700 1 -- 172.21.15.32:0/772746175 learned_addr learned my addr 172.21.15.32:0/772746175 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:04.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9c9f8700 1 --2- 172.21.15.32:0/772746175 >> v2:172.21.15.86:3300/0 conn(0x7f6a980d7800 0x7f6a980d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:04.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9d1f9700 1 --2- 172.21.15.32:0/772746175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a980cffa0 0x7f6a980d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:04.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a8ffff700 1 -- 172.21.15.32:0/772746175 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759014958 0 0) 0x7f6a90000d30 con 0x7f6a980d5c70 2021-04-09T18:17:04.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a8ffff700 1 -- 172.21.15.32:0/772746175 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a84002fe0 con 0x7f6a980d5c70 2021-04-09T18:17:04.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9c9f8700 1 -- 172.21.15.32:0/772746175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a980cffa0 msgr2=0x7f6a980d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:04.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9c9f8700 1 --2- 172.21.15.32:0/772746175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a980cffa0 0x7f6a980d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9c9f8700 1 -- 172.21.15.32:0/772746175 >> v1:172.21.15.32:6789/0 conn(0x7f6a980d5c70 legacy=0x7f6a980d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:04.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9c9f8700 1 -- 172.21.15.32:0/772746175 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a980d8220 con 0x7f6a980d7800 2021-04-09T18:17:04.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9d1f9700 1 --2- 172.21.15.32:0/772746175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a980cffa0 0x7f6a980d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:04.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.556 7f6a9c9f8700 1 --2- 172.21.15.32:0/772746175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a980d7800 0x7f6a980d7c00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6a88006650 tx=0x7f6a88009a80).ready entity=mon.1 client_cookie=509608de11e9836e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:04.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.557 7f6a8ffff700 1 -- 172.21.15.32:0/772746175 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a880009b0 con 0x7f6a980d7800 2021-04-09T18:17:04.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.557 7f6a8ffff700 1 -- 172.21.15.32:0/772746175 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8800c0c0 con 0x7f6a980d7800 2021-04-09T18:17:04.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.557 7f6a8ffff700 1 -- 172.21.15.32:0/772746175 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a8800c510 con 0x7f6a980d7800 2021-04-09T18:17:04.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.557 7f6a9f3dd700 1 -- 172.21.15.32:0/772746175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a980d7800 msgr2=0x7f6a980d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:04.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.557 7f6a9f3dd700 1 --2- 172.21.15.32:0/772746175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a980d7800 0x7f6a980d7c00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6a88006650 tx=0x7f6a88009a80).stop 2021-04-09T18:17:04.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.557 7f6a9f3dd700 1 -- 172.21.15.32:0/772746175 shutdown_connections 2021-04-09T18:17:04.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.557 7f6a9f3dd700 1 --2- 172.21.15.32:0/772746175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a980d7800 0x7f6a980d7c00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.557 7f6a9f3dd700 1 --2- 172.21.15.32:0/772746175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a980cffa0 0x7f6a980d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.557 7f6a9f3dd700 1 -- 172.21.15.32:0/772746175 >> 172.21.15.32:0/772746175 conn(0x7f6a980ce5c0 msgr2=0x7f6a980cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:04.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.558 7f6a9f3dd700 1 -- 172.21.15.32:0/772746175 shutdown_connections 2021-04-09T18:17:04.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.558 7f6a9f3dd700 1 -- 172.21.15.32:0/772746175 wait complete. 2021-04-09T18:17:04.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.578 7f6a9f3dd700 1 Processor -- start 2021-04-09T18:17:04.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.578 7f6a9f3dd700 1 -- start start 2021-04-09T18:17:04.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.578 7f6a9f3dd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a981b85b0 0x7f6a9817dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:04.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.578 7f6a9f3dd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a9817e0e0 0x7f6a980adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:04.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.578 7f6a9f3dd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a980ae220 0x7f6a980ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:04.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.578 7f6a9f3dd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a980964d0 con 0x7f6a981b85b0 2021-04-09T18:17:04.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.578 7f6a9f3dd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a981b8e00 con 0x7f6a980ae220 2021-04-09T18:17:04.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.578 7f6a9f3dd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a981b89b0 con 0x7f6a9817e0e0 2021-04-09T18:17:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9d1f9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a981b85b0 0x7f6a9817dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:04.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9d1f9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a981b85b0 0x7f6a9817dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45250/0 (socket says 172.21.15.32:45250) 2021-04-09T18:17:04.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9d1f9700 1 -- 172.21.15.32:0/2580791521 learned_addr learned my addr 172.21.15.32:0/2580791521 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9c9f8700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a9817e0e0 0x7f6a980adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9d9fa700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a980ae220 0x7f6a980ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9c9f8700 1 -- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a980ae220 msgr2=0x7f6a980ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:04.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9c9f8700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a980ae220 0x7f6a980ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9c9f8700 1 -- 172.21.15.32:0/2580791521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a981b85b0 msgr2=0x7f6a9817dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:04.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9c9f8700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a981b85b0 0x7f6a9817dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9c9f8700 1 -- 172.21.15.32:0/2580791521 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a98095040 con 0x7f6a9817e0e0 2021-04-09T18:17:04.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9d1f9700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a981b85b0 0x7f6a9817dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:04.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9c9f8700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a9817e0e0 0x7f6a980adc00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6a8800b530 tx=0x7f6a8800b8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:04.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a8dffb700 1 -- 172.21.15.32:0/2580791521 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a8800bcc0 con 0x7f6a9817e0e0 2021-04-09T18:17:04.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a8dffb700 1 -- 172.21.15.32:0/2580791521 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a880091c0 con 0x7f6a9817e0e0 2021-04-09T18:17:04.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a98095040 con 0x7f6a9817e0e0 2021-04-09T18:17:04.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a8dffb700 1 -- 172.21.15.32:0/2580791521 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a880054d0 con 0x7f6a9817e0e0 2021-04-09T18:17:04.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.579 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a98095040 con 0x7f6a9817e0e0 2021-04-09T18:17:04.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.580 7f6a8dffb700 1 -- 172.21.15.32:0/2580791521 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6a880052e0 con 0x7f6a9817e0e0 2021-04-09T18:17:04.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.580 7f6a8dffb700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a78018930 0x7f6a7801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:04.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.580 7f6a8dffb700 1 -- 172.21.15.32:0/2580791521 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6a88032900 con 0x7f6a9817e0e0 2021-04-09T18:17:04.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.580 7f6a9d1f9700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a78018930 0x7f6a7801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:04.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.581 7f6a9d1f9700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a78018930 0x7f6a7801adc0 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:04.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.600 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a981b89b0 con 0x7f6a9817e0e0 2021-04-09T18:17:04.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.604 7f6a8dffb700 1 -- 172.21.15.32:0/2580791521 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6a880091c0 con 0x7f6a9817e0e0 2021-04-09T18:17:04.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.795 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a981b89b0 con 0x7f6a9817e0e0 2021-04-09T18:17:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.796 7f6a8dffb700 1 -- 172.21.15.32:0/2580791521 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2794 (secure 0 0 0) 0x7f6a880093d0 con 0x7f6a9817e0e0 2021-04-09T18:17:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 61 2021-04-09T18:17:04.828 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:04.828 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:04.385004","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0,1],"up":{"mds_0":5803,"mds_1":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_5880":{"gid":5880,"name":"c","rank":1,"incarnation":58,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a78018930 msgr2=0x7f6a7801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a78018930 0x7f6a7801adc0 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a9817e0e0 msgr2=0x7f6a980adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a9817e0e0 0x7f6a980adc00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6a8800b530 tx=0x7f6a8800b8d0).stop 2021-04-09T18:17:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 shutdown_connections 2021-04-09T18:17:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a78018930 0x7f6a7801adc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a980ae220 0x7f6a980ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a9817e0e0 0x7f6a980adc00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 --2- 172.21.15.32:0/2580791521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a981b85b0 0x7f6a9817dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 >> 172.21.15.32:0/2580791521 conn(0x7f6a981864e0 msgr2=0x7f6a981b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:04.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 shutdown_connections 2021-04-09T18:17:04.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:04.798 7f6a9f3dd700 1 -- 172.21.15.32:0/2580791521 wait complete. 2021-04-09T18:17:05.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:17:05.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:17:05.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:04.385004', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 5803, 'mds_1': 5880}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_5880': {'gid': 5880, 'name': 'c', 'rank': 1, 'incarnation': 58, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6833/1942724067', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1942724067}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 1942724067}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:05.269 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5880:up:stopping 2021-04-09T18:17:06.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:06.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.402 7f023fa68700 1 Processor -- start 2021-04-09T18:17:06.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.402 7f023fa68700 1 -- start start 2021-04-09T18:17:06.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023fa68700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f023810d010 0x7f02380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:06.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023fa68700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02380d8490 0x7f023810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:06.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023fa68700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023810d480 con 0x7f02380d1da0 2021-04-09T18:17:06.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023fa68700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02380d8d30 con 0x7f023810d010 2021-04-09T18:17:06.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023fa68700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02380d9220 con 0x7f02380d8490 2021-04-09T18:17:06.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023d083700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f02380d1da0 0x7f02380d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54852/0 (socket says 172.21.15.32:54852) 2021-04-09T18:17:06.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023d083700 1 -- 172.21.15.32:0/4085486627 learned_addr learned my addr 172.21.15.32:0/4085486627 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:06.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023d884700 1 --2- 172.21.15.32:0/4085486627 >> v2:172.21.15.86:3300/0 conn(0x7f023810d010 0x7f02380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:06.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023e085700 1 --2- 172.21.15.32:0/4085486627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02380d8490 0x7f023810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:06.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023c882700 1 -- 172.21.15.32:0/4085486627 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074654215 0 0) 0x7f022c000ce0 con 0x7f02380d1da0 2021-04-09T18:17:06.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023c882700 1 -- 172.21.15.32:0/4085486627 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0220002fe0 con 0x7f02380d1da0 2021-04-09T18:17:06.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023e085700 1 -- 172.21.15.32:0/4085486627 >> v2:172.21.15.86:3300/0 conn(0x7f023810d010 msgr2=0x7f02380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:06.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023e085700 1 --2- 172.21.15.32:0/4085486627 >> v2:172.21.15.86:3300/0 conn(0x7f023810d010 0x7f02380d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023e085700 1 -- 172.21.15.32:0/4085486627 >> v1:172.21.15.32:6789/0 conn(0x7f02380d1da0 legacy=0x7f02380d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:06.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023c882700 1 -- 172.21.15.32:0/4085486627 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2741048232 0 0) 0x7f022c0010e0 con 0x7f02380d1da0 2021-04-09T18:17:06.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023e085700 1 -- 172.21.15.32:0/4085486627 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023810e020 con 0x7f02380d8490 2021-04-09T18:17:06.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.403 7f023d884700 1 --2- 172.21.15.32:0/4085486627 >> v2:172.21.15.86:3300/0 conn(0x7f023810d010 0x7f02380d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:06.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.404 7f023e085700 1 --2- 172.21.15.32:0/4085486627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02380d8490 0x7f023810da00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f0228006ad0 tx=0x7f0228009ad0).ready entity=mon.2 client_cookie=a69b6c09b138cf55 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:06.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.404 7f023c882700 1 -- 172.21.15.32:0/4085486627 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02280022e0 con 0x7f02380d8490 2021-04-09T18:17:06.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.404 7f023c882700 1 -- 172.21.15.32:0/4085486627 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0228006410 con 0x7f02380d8490 2021-04-09T18:17:06.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.404 7f023c882700 1 -- 172.21.15.32:0/4085486627 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0228002620 con 0x7f02380d8490 2021-04-09T18:17:06.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.404 7f023fa68700 1 -- 172.21.15.32:0/4085486627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02380d8490 msgr2=0x7f023810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:06.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.404 7f023fa68700 1 --2- 172.21.15.32:0/4085486627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02380d8490 0x7f023810da00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f0228006ad0 tx=0x7f0228009ad0).stop 2021-04-09T18:17:06.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.405 7f023fa68700 1 -- 172.21.15.32:0/4085486627 shutdown_connections 2021-04-09T18:17:06.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.405 7f023fa68700 1 --2- 172.21.15.32:0/4085486627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02380d8490 0x7f023810da00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.405 7f023fa68700 1 --2- 172.21.15.32:0/4085486627 >> v2:172.21.15.86:3300/0 conn(0x7f023810d010 0x7f02380d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.405 7f023fa68700 1 -- 172.21.15.32:0/4085486627 >> 172.21.15.32:0/4085486627 conn(0x7f02380ce5f0 msgr2=0x7f02380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:06.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.405 7f023fa68700 1 -- 172.21.15.32:0/4085486627 shutdown_connections 2021-04-09T18:17:06.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.405 7f023fa68700 1 -- 172.21.15.32:0/4085486627 wait complete. 2021-04-09T18:17:06.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.425 7f023fa68700 1 Processor -- start 2021-04-09T18:17:06.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.425 7f023fa68700 1 -- start start 2021-04-09T18:17:06.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023fa68700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0238185060 0x7f0238185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:06.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023fa68700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0238186c00 0x7f0238187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:06.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023fa68700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0238191360 0x7f0238191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:06.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023fa68700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02381863a0 con 0x7f0238185060 2021-04-09T18:17:06.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023fa68700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0238185a60 con 0x7f0238186c00 2021-04-09T18:17:06.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023fa68700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0238187620 con 0x7f0238191360 2021-04-09T18:17:06.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d884700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0238185060 0x7f0238185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:06.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d884700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0238185060 0x7f0238185d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45264/0 (socket says 172.21.15.32:45264) 2021-04-09T18:17:06.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d884700 1 -- 172.21.15.32:0/3402049189 learned_addr learned my addr 172.21.15.32:0/3402049189 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:06.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023e085700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0238191360 0x7f0238191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:06.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d083700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0238186c00 0x7f0238187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:06.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d884700 1 -- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0238191360 msgr2=0x7f0238191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:06.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d884700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0238191360 0x7f0238191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d884700 1 -- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0238186c00 msgr2=0x7f0238187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:06.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d884700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0238186c00 0x7f0238187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d884700 1 -- 172.21.15.32:0/3402049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0238185460 con 0x7f0238185060 2021-04-09T18:17:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023d884700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0238185060 0x7f0238185d80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f023000b970 tx=0x7f023000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f02367fc700 1 -- 172.21.15.32:0/3402049189 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0230000d30 con 0x7f0238185060 2021-04-09T18:17:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f02367fc700 1 -- 172.21.15.32:0/3402049189 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0230006240 con 0x7f0238185060 2021-04-09T18:17:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f023fa68700 1 -- 172.21.15.32:0/3402049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0238185460 con 0x7f0238185060 2021-04-09T18:17:06.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.426 7f02367fc700 1 -- 172.21.15.32:0/3402049189 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0230005660 con 0x7f0238185060 2021-04-09T18:17:06.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.427 7f023fa68700 1 -- 172.21.15.32:0/3402049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0238185460 con 0x7f0238185060 2021-04-09T18:17:06.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.427 7f02367fc700 1 -- 172.21.15.32:0/3402049189 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0230005660 con 0x7f0238185060 2021-04-09T18:17:06.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.427 7f02367fc700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0218018930 0x7f021801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:06.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.427 7f02367fc700 1 -- 172.21.15.32:0/3402049189 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0230032e00 con 0x7f0238185060 2021-04-09T18:17:06.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.427 7f023d083700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0218018930 0x7f021801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:06.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.428 7f023d083700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0218018930 0x7f021801ae10 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:06.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.447 7f023fa68700 1 -- 172.21.15.32:0/3402049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02381866d0 con 0x7f0238185060 2021-04-09T18:17:06.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.450 7f02367fc700 1 -- 172.21.15.32:0/3402049189 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0230006240 con 0x7f0238185060 2021-04-09T18:17:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.642 7f023fa68700 1 -- 172.21.15.32:0/3402049189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02381866d0 con 0x7f0238185060 2021-04-09T18:17:06.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.643 7f02367fc700 1 -- 172.21.15.32:0/3402049189 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2794 (secure 0 0 0) 0x7f0230005660 con 0x7f0238185060 2021-04-09T18:17:06.676 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 61 2021-04-09T18:17:06.677 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:06.677 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:04.385004","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0,1],"up":{"mds_0":5803,"mds_1":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_5880":{"gid":5880,"name":"c","rank":1,"incarnation":58,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:06.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 -- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0218018930 msgr2=0x7f021801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:06.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0218018930 0x7f021801ae10 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 -- 172.21.15.32:0/3402049189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0238185060 msgr2=0x7f0238185d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:06.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0238185060 0x7f0238185d80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f023000b970 tx=0x7f023000bd10).stop 2021-04-09T18:17:06.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 -- 172.21.15.32:0/3402049189 shutdown_connections 2021-04-09T18:17:06.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0218018930 0x7f021801ae10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0238191360 0x7f0238191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0238186c00 0x7f0238187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 --2- 172.21.15.32:0/3402049189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0238185060 0x7f0238185d80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:06.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 -- 172.21.15.32:0/3402049189 >> 172.21.15.32:0/3402049189 conn(0x7f02381837d0 msgr2=0x7f0238195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 -- 172.21.15.32:0/3402049189 shutdown_connections 2021-04-09T18:17:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:06.645 7f023fa68700 1 -- 172.21.15.32:0/3402049189 wait complete. 2021-04-09T18:17:07.110 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:04.385004', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 5803, 'mds_1': 5880}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_5880': {'gid': 5880, 'name': 'c', 'rank': 1, 'incarnation': 58, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6833/1942724067', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1942724067}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 1942724067}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:07.110 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5880:up:stopping 2021-04-09T18:17:08.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:08.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.247 7f350e556700 1 Processor -- start 2021-04-09T18:17:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.247 7f350e556700 1 -- start start 2021-04-09T18:17:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.248 7f350e556700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350810da00 0x7f350810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.248 7f350e556700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350810d480 con 0x7f35080d7ca0 2021-04-09T18:17:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.248 7f350e556700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35080d1f70 con 0x7f350810d010 2021-04-09T18:17:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.248 7f350e556700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35080d9220 con 0x7f350810da00 2021-04-09T18:17:08.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.248 7f34fffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f35080d7ca0 0x7f35080d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54864/0 (socket says 172.21.15.32:54864) 2021-04-09T18:17:08.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.248 7f34fffff700 1 -- 172.21.15.32:0/3503699488 learned_addr learned my addr 172.21.15.32:0/3503699488 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:08.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.248 7f350cb73700 1 --2- 172.21.15.32:0/3503699488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350810da00 0x7f350810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:08.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f35077fe700 1 -- 172.21.15.32:0/3503699488 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189325292 0 0) 0x7f34f0000ce0 con 0x7f35080d7ca0 2021-04-09T18:17:08.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f35077fe700 1 -- 172.21.15.32:0/3503699488 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f4002fe0 con 0x7f35080d7ca0 2021-04-09T18:17:08.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f35077fe700 1 -- 172.21.15.32:0/3503699488 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265074443 0 0) 0x7f34f8000c60 con 0x7f350810d010 2021-04-09T18:17:08.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f35077fe700 1 -- 172.21.15.32:0/3503699488 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f40042c0 con 0x7f350810d010 2021-04-09T18:17:08.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f350cb73700 1 -- 172.21.15.32:0/3503699488 >> v1:172.21.15.86:6789/0 conn(0x7f350810d010 legacy=0x7f35080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:08.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f350cb73700 1 -- 172.21.15.32:0/3503699488 >> v1:172.21.15.32:6789/0 conn(0x7f35080d7ca0 legacy=0x7f35080d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:08.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f350cb73700 1 -- 172.21.15.32:0/3503699488 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350810e420 con 0x7f350810da00 2021-04-09T18:17:08.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f350cb73700 1 --2- 172.21.15.32:0/3503699488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350810da00 0x7f350810de00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3500006e50 tx=0x7f3500009df0).ready entity=mon.2 client_cookie=af6258380594e947 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f35077fe700 1 -- 172.21.15.32:0/3503699488 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f350000b510 con 0x7f350810da00 2021-04-09T18:17:08.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f35077fe700 1 -- 172.21.15.32:0/3503699488 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500003030 con 0x7f350810da00 2021-04-09T18:17:08.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f35077fe700 1 -- 172.21.15.32:0/3503699488 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f350000b7d0 con 0x7f350810da00 2021-04-09T18:17:08.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f350e556700 1 -- 172.21.15.32:0/3503699488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350810da00 msgr2=0x7f350810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:08.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.249 7f350e556700 1 --2- 172.21.15.32:0/3503699488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350810da00 0x7f350810de00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3500006e50 tx=0x7f3500009df0).stop 2021-04-09T18:17:08.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.250 7f350e556700 1 -- 172.21.15.32:0/3503699488 shutdown_connections 2021-04-09T18:17:08.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.250 7f350e556700 1 --2- 172.21.15.32:0/3503699488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350810da00 0x7f350810de00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:08.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.250 7f350e556700 1 -- 172.21.15.32:0/3503699488 >> 172.21.15.32:0/3503699488 conn(0x7f35080ce5f0 msgr2=0x7f35080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:08.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.250 7f350e556700 1 -- 172.21.15.32:0/3503699488 shutdown_connections 2021-04-09T18:17:08.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.250 7f350e556700 1 -- 172.21.15.32:0/3503699488 wait complete. 2021-04-09T18:17:08.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.271 7f350e556700 1 Processor -- start 2021-04-09T18:17:08.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.271 7f350e556700 1 -- start start 2021-04-09T18:17:08.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.271 7f350e556700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35080ac1e0 0x7f35080ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:08.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.271 7f350e556700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f350817d020 0x7f350808add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:08.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.271 7f350e556700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350808b3f0 0x7f3508179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:08.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.271 7f350e556700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350808a850 con 0x7f35080ac1e0 2021-04-09T18:17:08.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.271 7f350e556700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350808b7f0 con 0x7f350817d020 2021-04-09T18:17:08.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.271 7f350e556700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3508090d60 con 0x7f350808b3f0 2021-04-09T18:17:08.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f3507fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35080ac1e0 0x7f35080ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:08.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f3507fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35080ac1e0 0x7f35080ac5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45278/0 (socket says 172.21.15.32:45278) 2021-04-09T18:17:08.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f3507fff700 1 -- 172.21.15.32:0/987778868 learned_addr learned my addr 172.21.15.32:0/987778868 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:08.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f34fffff700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f350817d020 0x7f350808add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:08.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f350cb73700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350808b3f0 0x7f3508179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:08.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f3507fff700 1 -- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350808b3f0 msgr2=0x7f3508179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:08.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f3507fff700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350808b3f0 0x7f3508179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:08.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f3507fff700 1 -- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f350817d020 msgr2=0x7f350808add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:08.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f3507fff700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f350817d020 0x7f350808add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:08.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f3507fff700 1 -- 172.21.15.32:0/987778868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3508179c30 con 0x7f35080ac1e0 2021-04-09T18:17:08.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f34fffff700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f350817d020 0x7f350808add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:08.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f3507fff700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35080ac1e0 0x7f35080ac5e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f34f800b580 tx=0x7f34f800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:08.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f35057fa700 1 -- 172.21.15.32:0/987778868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34f800bd10 con 0x7f35080ac1e0 2021-04-09T18:17:08.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f35057fa700 1 -- 172.21.15.32:0/987778868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f8000e20 con 0x7f35080ac1e0 2021-04-09T18:17:08.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f350e556700 1 -- 172.21.15.32:0/987778868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3508179c30 con 0x7f35080ac1e0 2021-04-09T18:17:08.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f35057fa700 1 -- 172.21.15.32:0/987778868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34f80054f0 con 0x7f35080ac1e0 2021-04-09T18:17:08.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.272 7f350e556700 1 -- 172.21.15.32:0/987778868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3508179c30 con 0x7f35080ac1e0 2021-04-09T18:17:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.273 7f35057fa700 1 -- 172.21.15.32:0/987778868 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f34f8009fe0 con 0x7f35080ac1e0 2021-04-09T18:17:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.273 7f35057fa700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34ec0188d0 0x7f34ec01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.273 7f35057fa700 1 -- 172.21.15.32:0/987778868 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f34f8032a50 con 0x7f35080ac1e0 2021-04-09T18:17:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.273 7f34fffff700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34ec0188d0 0x7f34ec01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.274 7f34fffff700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34ec0188d0 0x7f34ec01add0 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.292 7f350e556700 1 -- 172.21.15.32:0/987778868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350817d420 con 0x7f35080ac1e0 2021-04-09T18:17:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.296 7f35057fa700 1 -- 172.21.15.32:0/987778868 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f34f8000e20 con 0x7f35080ac1e0 2021-04-09T18:17:08.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.487 7f350e556700 1 -- 172.21.15.32:0/987778868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f350817d420 con 0x7f35080ac1e0 2021-04-09T18:17:08.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.487 7f35057fa700 1 -- 172.21.15.32:0/987778868 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2794 (secure 0 0 0) 0x7f34f8009fe0 con 0x7f35080ac1e0 2021-04-09T18:17:08.550 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 61 2021-04-09T18:17:08.551 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:08.551 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:04.385004","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0,1],"up":{"mds_0":5803,"mds_1":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_5880":{"gid":5880,"name":"c","rank":1,"incarnation":58,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:08.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 -- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34ec0188d0 msgr2=0x7f34ec01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:08.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34ec0188d0 0x7f34ec01add0 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:08.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 -- 172.21.15.32:0/987778868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35080ac1e0 msgr2=0x7f35080ac5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:08.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35080ac1e0 0x7f35080ac5e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f34f800b580 tx=0x7f34f800b920).stop 2021-04-09T18:17:08.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 -- 172.21.15.32:0/987778868 shutdown_connections 2021-04-09T18:17:08.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34ec0188d0 0x7f34ec01add0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f350808b3f0 0x7f3508179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:08.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f350817d020 0x7f350808add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 --2- 172.21.15.32:0/987778868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35080ac1e0 0x7f35080ac5e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:08.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 -- 172.21.15.32:0/987778868 >> 172.21.15.32:0/987778868 conn(0x7f3508190570 msgr2=0x7f35081acb80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 -- 172.21.15.32:0/987778868 shutdown_connections 2021-04-09T18:17:08.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:08.490 7f350e556700 1 -- 172.21.15.32:0/987778868 wait complete. 2021-04-09T18:17:08.999 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:04.385004', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 5803, 'mds_1': 5880}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_5880': {'gid': 5880, 'name': 'c', 'rank': 1, 'incarnation': 58, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6833/1942724067', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1942724067}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 1942724067}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:08.999 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5880:up:stopping 2021-04-09T18:17:10.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:10.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.136 7f5a6dc95700 1 Processor -- start 2021-04-09T18:17:10.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.136 7f5a6dc95700 1 -- start start 2021-04-09T18:17:10.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a6dc95700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6810da00 0x7f5a6810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:10.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a6dc95700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6810d480 con 0x7f5a680d7ca0 2021-04-09T18:17:10.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a6dc95700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a680d1f70 con 0x7f5a6810d010 2021-04-09T18:17:10.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a6dc95700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680d9220 con 0x7f5a6810da00 2021-04-09T18:17:10.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a66ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5a680d7ca0 0x7f5a680d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54876/0 (socket says 172.21.15.32:54876) 2021-04-09T18:17:10.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a66ffd700 1 -- 172.21.15.32:0/2174944521 learned_addr learned my addr 172.21.15.32:0/2174944521 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:10.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a67fff700 1 --2- 172.21.15.32:0/2174944521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6810da00 0x7f5a6810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:10.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a667fc700 1 -- 172.21.15.32:0/2174944521 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280187370 0 0) 0x7f5a50000ce0 con 0x7f5a680d7ca0 2021-04-09T18:17:10.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a667fc700 1 -- 172.21.15.32:0/2174944521 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a54002fe0 con 0x7f5a680d7ca0 2021-04-09T18:17:10.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a667fc700 1 -- 172.21.15.32:0/2174944521 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032297185 0 0) 0x7f5a58000fe0 con 0x7f5a6810d010 2021-04-09T18:17:10.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a667fc700 1 -- 172.21.15.32:0/2174944521 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a54003210 con 0x7f5a6810d010 2021-04-09T18:17:10.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a667fc700 1 -- 172.21.15.32:0/2174944521 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2102194673 0 0) 0x7f5a500010e0 con 0x7f5a680d7ca0 2021-04-09T18:17:10.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a67fff700 1 -- 172.21.15.32:0/2174944521 >> v1:172.21.15.86:6789/0 conn(0x7f5a6810d010 legacy=0x7f5a680d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:10.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a67fff700 1 -- 172.21.15.32:0/2174944521 >> v1:172.21.15.32:6789/0 conn(0x7f5a680d7ca0 legacy=0x7f5a680d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:10.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a67fff700 1 -- 172.21.15.32:0/2174944521 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6810e420 con 0x7f5a6810da00 2021-04-09T18:17:10.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.137 7f5a67fff700 1 --2- 172.21.15.32:0/2174944521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6810da00 0x7f5a6810de00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5a60006ad0 tx=0x7f5a60009ad0).ready entity=mon.2 client_cookie=fc935178654b123 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:10.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.138 7f5a667fc700 1 -- 172.21.15.32:0/2174944521 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a600022e0 con 0x7f5a6810da00 2021-04-09T18:17:10.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.138 7f5a667fc700 1 -- 172.21.15.32:0/2174944521 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a60006410 con 0x7f5a6810da00 2021-04-09T18:17:10.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.138 7f5a667fc700 1 -- 172.21.15.32:0/2174944521 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a60002620 con 0x7f5a6810da00 2021-04-09T18:17:10.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.138 7f5a6dc95700 1 -- 172.21.15.32:0/2174944521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6810da00 msgr2=0x7f5a6810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:10.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.138 7f5a6dc95700 1 --2- 172.21.15.32:0/2174944521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6810da00 0x7f5a6810de00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5a60006ad0 tx=0x7f5a60009ad0).stop 2021-04-09T18:17:10.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.138 7f5a6dc95700 1 -- 172.21.15.32:0/2174944521 shutdown_connections 2021-04-09T18:17:10.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.138 7f5a6dc95700 1 --2- 172.21.15.32:0/2174944521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a6810da00 0x7f5a6810de00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.138 7f5a6dc95700 1 -- 172.21.15.32:0/2174944521 >> 172.21.15.32:0/2174944521 conn(0x7f5a680ce5f0 msgr2=0x7f5a680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.138 7f5a6dc95700 1 -- 172.21.15.32:0/2174944521 shutdown_connections 2021-04-09T18:17:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.139 7f5a6dc95700 1 -- 172.21.15.32:0/2174944521 wait complete. 2021-04-09T18:17:10.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.159 7f5a6dc95700 1 Processor -- start 2021-04-09T18:17:10.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.159 7f5a6dc95700 1 -- start start 2021-04-09T18:17:10.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.159 7f5a6dc95700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a68185060 0x7f5a68185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:10.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.159 7f5a6dc95700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a68186c00 0x7f5a68187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:10.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.159 7f5a6dc95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a68191360 0x7f5a68191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:10.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.159 7f5a6dc95700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a681863a0 con 0x7f5a68191360 2021-04-09T18:17:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.159 7f5a6dc95700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a68185a60 con 0x7f5a68186c00 2021-04-09T18:17:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.159 7f5a6dc95700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a68187620 con 0x7f5a68185060 2021-04-09T18:17:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.159 7f5a67fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a68191360 0x7f5a68191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a67fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a68191360 0x7f5a68191760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45294/0 (socket says 172.21.15.32:45294) 2021-04-09T18:17:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a67fff700 1 -- 172.21.15.32:0/2127026701 learned_addr learned my addr 172.21.15.32:0/2127026701 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:10.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a677fe700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a68185060 0x7f5a68185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:10.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a66ffd700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a68186c00 0x7f5a68187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:10.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a67fff700 1 -- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a68185060 msgr2=0x7f5a68185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:10.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a67fff700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a68185060 0x7f5a68185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:10.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a67fff700 1 -- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a68186c00 msgr2=0x7f5a68187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:10.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a67fff700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a68186c00 0x7f5a68187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:10.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a67fff700 1 -- 172.21.15.32:0/2127026701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a68185460 con 0x7f5a68191360 2021-04-09T18:17:10.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a67fff700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a68191360 0x7f5a68191760 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f5a60006850 tx=0x7f5a6000dba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:10.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a5ffff700 1 -- 172.21.15.32:0/2127026701 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a6000b020 con 0x7f5a68191360 2021-04-09T18:17:10.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a5ffff700 1 -- 172.21.15.32:0/2127026701 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6000b220 con 0x7f5a68191360 2021-04-09T18:17:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a5ffff700 1 -- 172.21.15.32:0/2127026701 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a600093e0 con 0x7f5a68191360 2021-04-09T18:17:10.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a68185460 con 0x7f5a68191360 2021-04-09T18:17:10.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.160 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a68185460 con 0x7f5a68191360 2021-04-09T18:17:10.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.161 7f5a5ffff700 1 -- 172.21.15.32:0/2127026701 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5a60005330 con 0x7f5a68191360 2021-04-09T18:17:10.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.161 7f5a5ffff700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a4c0188d0 0x7f5a4c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:10.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.161 7f5a5ffff700 1 -- 172.21.15.32:0/2127026701 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5a60034be0 con 0x7f5a68191360 2021-04-09T18:17:10.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.161 7f5a677fe700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a4c0188d0 0x7f5a4c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:10.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.161 7f5a677fe700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a4c0188d0 0x7f5a4c01add0 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:10.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.180 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a681866d0 con 0x7f5a68191360 2021-04-09T18:17:10.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.184 7f5a5ffff700 1 -- 172.21.15.32:0/2127026701 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5a6000b220 con 0x7f5a68191360 2021-04-09T18:17:10.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.375 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a681866d0 con 0x7f5a68191360 2021-04-09T18:17:10.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.376 7f5a5ffff700 1 -- 172.21.15.32:0/2127026701 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2794 (secure 0 0 0) 0x7f5a60000da0 con 0x7f5a68191360 2021-04-09T18:17:10.409 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 61 2021-04-09T18:17:10.409 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:10.410 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:04.385004","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0,1],"up":{"mds_0":5803,"mds_1":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_5880":{"gid":5880,"name":"c","rank":1,"incarnation":58,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6833/1942724067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1942724067},{"type":"v1","addr":"172.21.15.32:6833","nonce":1942724067}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:10.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a4c0188d0 msgr2=0x7f5a4c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:10.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a4c0188d0 0x7f5a4c01add0 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:10.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a68191360 msgr2=0x7f5a68191760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:10.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a68191360 0x7f5a68191760 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f5a60006850 tx=0x7f5a6000dba0).stop 2021-04-09T18:17:10.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 shutdown_connections 2021-04-09T18:17:10.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a4c0188d0 0x7f5a4c01add0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:10.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a68191360 0x7f5a68191760 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:10.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a68186c00 0x7f5a68187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:10.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 --2- 172.21.15.32:0/2127026701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a68185060 0x7f5a68185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:10.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.378 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 >> 172.21.15.32:0/2127026701 conn(0x7f5a681837d0 msgr2=0x7f5a68195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:10.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.379 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 shutdown_connections 2021-04-09T18:17:10.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:10.379 7f5a6dc95700 1 -- 172.21.15.32:0/2127026701 wait complete. 2021-04-09T18:17:10.487 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:17:10.824 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:04.385004', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 5803, 'mds_1': 5880}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_5880': {'gid': 5880, 'name': 'c', 'rank': 1, 'incarnation': 58, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6833/1942724067', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1942724067}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 1942724067}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:10.825 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5880:up:stopping 2021-04-09T18:17:11.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:12.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.958 7f8748c45700 1 Processor -- start 2021-04-09T18:17:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.958 7f8748c45700 1 -- start start 2021-04-09T18:17:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.958 7f8748c45700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f874410d010 0x7f87440d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.958 7f8748c45700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87440d8490 0x7f874410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.958 7f8748c45700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874410d480 con 0x7f87440d1da0 2021-04-09T18:17:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.958 7f8748c45700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87440d8d30 con 0x7f874410d010 2021-04-09T18:17:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.958 7f8748c45700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87440d9220 con 0x7f87440d8490 2021-04-09T18:17:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874205d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f87440d1da0 0x7f87440d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54900/0 (socket says 172.21.15.32:54900) 2021-04-09T18:17:12.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874205d700 1 -- 172.21.15.32:0/329996592 learned_addr learned my addr 172.21.15.32:0/329996592 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:12.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874285e700 1 --2- 172.21.15.32:0/329996592 >> v2:172.21.15.86:3300/0 conn(0x7f874410d010 0x7f87440d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874305f700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87440d8490 0x7f874410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626263823 0 0) 0x7f8738001060 con 0x7f87440d1da0 2021-04-09T18:17:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f872c002fe0 con 0x7f87440d1da0 2021-04-09T18:17:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4219964288 0 0) 0x7f8738001250 con 0x7f87440d1da0 2021-04-09T18:17:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87440d8490 msgr2=0x7f874410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:12.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87440d8490 0x7f874410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 >> v2:172.21.15.86:3300/0 conn(0x7f874410d010 msgr2=0x7f87440d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:12.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 --2- 172.21.15.32:0/329996592 >> v2:172.21.15.86:3300/0 conn(0x7f874410d010 0x7f87440d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874410e020 con 0x7f87440d1da0 2021-04-09T18:17:12.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f8738001980 con 0x7f87440d1da0 2021-04-09T18:17:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874285e700 1 --2- 172.21.15.32:0/329996592 >> v2:172.21.15.86:3300/0 conn(0x7f874410d010 0x7f87440d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 >> v1:172.21.15.32:6789/0 conn(0x7f87440d1da0 legacy=0x7f87440d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:12.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f872c047aa0 0x7f872c049f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:12.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f872c04a790 0x7f872c04ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:12.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872c04d5b0 0x7f872c04faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:12.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c0502f0 con 0x7f872c04a790 2021-04-09T18:17:12.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c050510 con 0x7f872c047aa0 2021-04-09T18:17:12.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.959 7f874185c700 1 -- 172.21.15.32:0/329996592 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872c050730 con 0x7f872c04d5b0 2021-04-09T18:17:12.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874205d700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f872c04a790 0x7f872c04ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:12.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874285e700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f872c047aa0 0x7f872c049f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:12.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874305f700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872c04d5b0 0x7f872c04faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:12.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874205d700 1 -- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872c04d5b0 msgr2=0x7f872c04faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:12.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874205d700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872c04d5b0 0x7f872c04faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874205d700 1 -- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f872c047aa0 msgr2=0x7f872c049f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:12.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874205d700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f872c047aa0 0x7f872c049f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874205d700 1 -- 172.21.15.32:0/329996592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c0509d0 con 0x7f872c04a790 2021-04-09T18:17:12.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874205d700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f872c04a790 0x7f872c04ccb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f8738007a30 tx=0x7f873800a310).ready entity=mon.0 client_cookie=fea3b82d62965cfc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:12.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874185c700 1 -- 172.21.15.32:0/329996592 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8738001900 con 0x7f872c04a790 2021-04-09T18:17:12.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874185c700 1 -- 172.21.15.32:0/329996592 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87380070c0 con 0x7f872c04a790 2021-04-09T18:17:12.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874185c700 1 -- 172.21.15.32:0/329996592 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8738007520 con 0x7f872c04a790 2021-04-09T18:17:12.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f8748c45700 1 -- 172.21.15.32:0/329996592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f872c04a790 msgr2=0x7f872c04ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:12.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f8748c45700 1 --2- 172.21.15.32:0/329996592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f872c04a790 0x7f872c04ccb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f8738007a30 tx=0x7f873800a310).stop 2021-04-09T18:17:12.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.960 7f874305f700 1 -- 172.21.15.32:0/329996592 reap_dead start 2021-04-09T18:17:12.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.961 7f8748c45700 1 -- 172.21.15.32:0/329996592 shutdown_connections 2021-04-09T18:17:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.961 7f8748c45700 1 -- 172.21.15.32:0/329996592 >> 172.21.15.32:0/329996592 conn(0x7f87440ce5f0 msgr2=0x7f87440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:12.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.961 7f8748c45700 1 -- 172.21.15.32:0/329996592 shutdown_connections 2021-04-09T18:17:12.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.961 7f8748c45700 1 -- 172.21.15.32:0/329996592 wait complete. 2021-04-09T18:17:12.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.981 7f8748c45700 1 Processor -- start 2021-04-09T18:17:12.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.981 7f8748c45700 1 -- start start 2021-04-09T18:17:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f8748c45700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87441ad040 0x7f8744192100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:12.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f8748c45700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f874418f380 0x7f8744192720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:12.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f8748c45700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8744190330 0x7f8744190730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:12.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f8748c45700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8744191b80 con 0x7f87441ad040 2021-04-09T18:17:12.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f8748c45700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8744190d50 con 0x7f874418f380 2021-04-09T18:17:12.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f8748c45700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8744190f30 con 0x7f8744190330 2021-04-09T18:17:12.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87441ad040 0x7f8744192100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:12.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87441ad040 0x7f8744192100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45322/0 (socket says 172.21.15.32:45322) 2021-04-09T18:17:12.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874285e700 1 -- 172.21.15.32:0/3147819334 learned_addr learned my addr 172.21.15.32:0/3147819334 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:12.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874305f700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8744190330 0x7f8744190730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:12.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874205d700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f874418f380 0x7f8744192720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:12.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874285e700 1 -- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8744190330 msgr2=0x7f8744190730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874285e700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8744190330 0x7f8744190730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874285e700 1 -- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f874418f380 msgr2=0x7f8744192720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874285e700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f874418f380 0x7f8744192720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874285e700 1 -- 172.21.15.32:0/3147819334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8744191150 con 0x7f87441ad040 2021-04-09T18:17:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874205d700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f874418f380 0x7f8744192720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874305f700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8744190330 0x7f8744190730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f874285e700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87441ad040 0x7f8744192100 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f873400b5e0 tx=0x7f873400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:12.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.982 7f87337fe700 1 -- 172.21.15.32:0/3147819334 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f873400bd70 con 0x7f87441ad040 2021-04-09T18:17:12.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.983 7f87337fe700 1 -- 172.21.15.32:0/3147819334 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734006740 con 0x7f87441ad040 2021-04-09T18:17:12.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.983 7f8748c45700 1 -- 172.21.15.32:0/3147819334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8744191150 con 0x7f87441ad040 2021-04-09T18:17:12.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.983 7f87337fe700 1 -- 172.21.15.32:0/3147819334 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87340022b0 con 0x7f87441ad040 2021-04-09T18:17:12.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.983 7f8748c45700 1 -- 172.21.15.32:0/3147819334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8744191150 con 0x7f87441ad040 2021-04-09T18:17:12.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.983 7f87337fe700 1 -- 172.21.15.32:0/3147819334 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f873400a000 con 0x7f87441ad040 2021-04-09T18:17:12.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.983 7f87337fe700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8728018930 0x7f872801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:12.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.983 7f87337fe700 1 -- 172.21.15.32:0/3147819334 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f87340329f0 con 0x7f87441ad040 2021-04-09T18:17:12.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.984 7f874205d700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8728018930 0x7f872801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:12.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:11.984 7f874205d700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8728018930 0x7f872801ae10 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:12.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.002 7f8748c45700 1 -- 172.21.15.32:0/3147819334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8744182d50 con 0x7f87441ad040 2021-04-09T18:17:12.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.006 7f87337fe700 1 -- 172.21.15.32:0/3147819334 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8734000c90 con 0x7f87441ad040 2021-04-09T18:17:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.196 7f8748c45700 1 -- 172.21.15.32:0/3147819334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f874410d480 con 0x7f87441ad040 2021-04-09T18:17:12.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.196 7f87337fe700 1 -- 172.21.15.32:0/3147819334 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2444 (secure 0 0 0) 0x7f8734000ea0 con 0x7f87441ad040 2021-04-09T18:17:12.390 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 63 2021-04-09T18:17:12.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:12.398 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":63,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:12.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 -- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8728018930 msgr2=0x7f872801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:12.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8728018930 0x7f872801ae10 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 -- 172.21.15.32:0/3147819334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87441ad040 msgr2=0x7f8744192100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87441ad040 0x7f8744192100 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f873400b5e0 tx=0x7f873400b980).stop 2021-04-09T18:17:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 -- 172.21.15.32:0/3147819334 shutdown_connections 2021-04-09T18:17:12.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8728018930 0x7f872801ae10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8744190330 0x7f8744190730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f874418f380 0x7f8744192720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 --2- 172.21.15.32:0/3147819334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87441ad040 0x7f8744192100 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:12.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 -- 172.21.15.32:0/3147819334 >> 172.21.15.32:0/3147819334 conn(0x7f874418cc60 msgr2=0x7f874419ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:12.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 -- 172.21.15.32:0/3147819334 shutdown_connections 2021-04-09T18:17:12.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:12.199 7f8748c45700 1 -- 172.21.15.32:0/3147819334 wait complete. 2021-04-09T18:17:12.884 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:12.885 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:13.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:14.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.006 7f228883d700 1 Processor -- start 2021-04-09T18:17:14.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.006 7f228883d700 1 -- start start 2021-04-09T18:17:14.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.006 7f228883d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f228410d010 0x7f22840d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:14.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.006 7f228883d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22840d8490 0x7f228410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:14.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228883d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228410d480 con 0x7f22840d1da0 2021-04-09T18:17:14.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228883d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228410e780 con 0x7f22840d8490 2021-04-09T18:17:14.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228883d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22840d8d30 con 0x7f228410d010 2021-04-09T18:17:14.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f2282e1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22840d8490 0x7f228410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:14.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f2282e1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22840d8490 0x7f228410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45332/0 (socket says 172.21.15.32:45332) 2021-04-09T18:17:14.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f2282e1d700 1 -- 172.21.15.32:0/3195772252 learned_addr learned my addr 172.21.15.32:0/3195772252 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:14.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228261c700 1 --2- 172.21.15.32:0/3195772252 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f228410d010 0x7f22840d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:14.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f2282e1d700 1 -- 172.21.15.32:0/3195772252 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f228410d010 msgr2=0x7f22840d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:14.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f2282e1d700 1 --2- 172.21.15.32:0/3195772252 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f228410d010 0x7f22840d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228161a700 1 -- 172.21.15.32:0/3195772252 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622126167 0 0) 0x7f2278000c60 con 0x7f22840d1da0 2021-04-09T18:17:14.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f2282e1d700 1 -- 172.21.15.32:0/3195772252 >> v1:172.21.15.86:6789/0 conn(0x7f22840d1da0 legacy=0x7f22840d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:14.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f2282e1d700 1 -- 172.21.15.32:0/3195772252 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22840d9220 con 0x7f22840d8490 2021-04-09T18:17:14.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f2282e1d700 1 --2- 172.21.15.32:0/3195772252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22840d8490 0x7f228410da00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f227c006f00 tx=0x7f227c009f10).ready entity=mon.0 client_cookie=4cde93739dcc0f55 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:14.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228161a700 1 -- 172.21.15.32:0/3195772252 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f227c00b3b0 con 0x7f22840d8490 2021-04-09T18:17:14.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228161a700 1 -- 172.21.15.32:0/3195772252 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c00b730 con 0x7f22840d8490 2021-04-09T18:17:14.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228161a700 1 -- 172.21.15.32:0/3195772252 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f227c00bc10 con 0x7f22840d8490 2021-04-09T18:17:14.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228883d700 1 -- 172.21.15.32:0/3195772252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22840d8490 msgr2=0x7f228410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:14.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.007 7f228883d700 1 --2- 172.21.15.32:0/3195772252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22840d8490 0x7f228410da00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f227c006f00 tx=0x7f227c009f10).stop 2021-04-09T18:17:14.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.008 7f228883d700 1 -- 172.21.15.32:0/3195772252 shutdown_connections 2021-04-09T18:17:14.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.008 7f228883d700 1 --2- 172.21.15.32:0/3195772252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22840d8490 0x7f228410da00 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.008 7f228883d700 1 --2- 172.21.15.32:0/3195772252 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f228410d010 0x7f22840d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.008 7f228883d700 1 -- 172.21.15.32:0/3195772252 >> 172.21.15.32:0/3195772252 conn(0x7f22840ce5f0 msgr2=0x7f22840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:14.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.008 7f228883d700 1 -- 172.21.15.32:0/3195772252 shutdown_connections 2021-04-09T18:17:14.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.008 7f228883d700 1 -- 172.21.15.32:0/3195772252 wait complete. 2021-04-09T18:17:14.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.028 7f228883d700 1 Processor -- start 2021-04-09T18:17:14.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.028 7f228883d700 1 -- start start 2021-04-09T18:17:14.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f228883d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22841a5210 0x7f2284185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:14.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f228883d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2284185a70 0x7f22840acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:14.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f228883d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284185e70 0x7f22840ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:14.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f228883d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228418f7b0 con 0x7f2284185e70 2021-04-09T18:17:14.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f228883d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22840ad7f0 con 0x7f22841a5210 2021-04-09T18:17:14.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f228883d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22840ada10 con 0x7f2284185a70 2021-04-09T18:17:14.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f2282e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284185e70 0x7f22840ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:14.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f2282e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284185e70 0x7f22840ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45340/0 (socket says 172.21.15.32:45340) 2021-04-09T18:17:14.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f2282e1d700 1 -- 172.21.15.32:0/1718689346 learned_addr learned my addr 172.21.15.32:0/1718689346 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:14.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f2281e1b700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2284185a70 0x7f22840acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:14.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f228261c700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22841a5210 0x7f2284185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:14.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f2282e1d700 1 -- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2284185a70 msgr2=0x7f22840acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:14.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f2282e1d700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2284185a70 0x7f22840acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f2282e1d700 1 -- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22841a5210 msgr2=0x7f2284185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:14.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f2282e1d700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22841a5210 0x7f2284185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.029 7f2282e1d700 1 -- 172.21.15.32:0/1718689346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2284179610 con 0x7f2284185e70 2021-04-09T18:17:14.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.030 7f2282e1d700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284185e70 0x7f22840ad1d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f227c009390 tx=0x7f227c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:14.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.030 7f2272ffd700 1 -- 172.21.15.32:0/1718689346 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f227c003030 con 0x7f2284185e70 2021-04-09T18:17:14.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.030 7f2272ffd700 1 -- 172.21.15.32:0/1718689346 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c0068e0 con 0x7f2284185e70 2021-04-09T18:17:14.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.030 7f228883d700 1 -- 172.21.15.32:0/1718689346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2284179610 con 0x7f2284185e70 2021-04-09T18:17:14.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.030 7f2272ffd700 1 -- 172.21.15.32:0/1718689346 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f227c00c570 con 0x7f2284185e70 2021-04-09T18:17:14.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.030 7f228883d700 1 -- 172.21.15.32:0/1718689346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2284179610 con 0x7f2284185e70 2021-04-09T18:17:14.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.030 7f2272ffd700 1 -- 172.21.15.32:0/1718689346 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f227c005820 con 0x7f2284185e70 2021-04-09T18:17:14.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.030 7f2272ffd700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2268018930 0x7f226801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:14.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.030 7f2272ffd700 1 -- 172.21.15.32:0/1718689346 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f227c032d80 con 0x7f2284185e70 2021-04-09T18:17:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.031 7f228261c700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2268018930 0x7f226801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:14.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.031 7f228261c700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2268018930 0x7f226801ae10 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.051 7f228883d700 1 -- 172.21.15.32:0/1718689346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228418f7b0 con 0x7f2284185e70 2021-04-09T18:17:14.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.054 7f2272ffd700 1 -- 172.21.15.32:0/1718689346 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f227c006c80 con 0x7f2284185e70 2021-04-09T18:17:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.245 7f228883d700 1 -- 172.21.15.32:0/1718689346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f228418f7b0 con 0x7f2284185e70 2021-04-09T18:17:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.246 7f2272ffd700 1 -- 172.21.15.32:0/1718689346 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2444 (secure 0 0 0) 0x7f227c0068e0 con 0x7f2284185e70 2021-04-09T18:17:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 63 2021-04-09T18:17:14.281 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:14.284 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":63,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 -- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2268018930 msgr2=0x7f226801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2268018930 0x7f226801ae10 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 -- 172.21.15.32:0/1718689346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284185e70 msgr2=0x7f22840ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284185e70 0x7f22840ad1d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f227c009390 tx=0x7f227c00bd40).stop 2021-04-09T18:17:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 -- 172.21.15.32:0/1718689346 shutdown_connections 2021-04-09T18:17:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2268018930 0x7f226801ae10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2284185e70 0x7f22840ad1d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2284185a70 0x7f22840acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 --2- 172.21.15.32:0/1718689346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22841a5210 0x7f2284185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:14.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 -- 172.21.15.32:0/1718689346 >> 172.21.15.32:0/1718689346 conn(0x7f22841a99c0 msgr2=0x7f228417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:14.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 -- 172.21.15.32:0/1718689346 shutdown_connections 2021-04-09T18:17:14.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:14.248 7f228883d700 1 -- 172.21.15.32:0/1718689346 wait complete. 2021-04-09T18:17:14.787 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:14.788 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:15.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:15.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.908 7ff7812f0700 1 Processor -- start 2021-04-09T18:17:15.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.908 7ff7812f0700 1 -- start start 2021-04-09T18:17:15.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff7812f0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff77c10d010 0x7ff77c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:15.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff7812f0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c10d480 con 0x7ff77c0d8490 2021-04-09T18:17:15.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff7812f0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c10e1f0 con 0x7ff77c0d1da0 2021-04-09T18:17:15.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff7812f0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c0d8d30 con 0x7ff77c10d010 2021-04-09T18:17:15.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77b07d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff77c10d010 0x7ff77c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:15.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77b87e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff77c0d8490 0x7ff77c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54938/0 (socket says 172.21.15.32:54938) 2021-04-09T18:17:15.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77b87e700 1 -- 172.21.15.32:0/3784750844 learned_addr learned my addr 172.21.15.32:0/3784750844 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:15.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77b07d700 1 -- 172.21.15.32:0/3784750844 >> v1:172.21.15.86:6789/0 conn(0x7ff77c0d1da0 legacy=0x7ff77c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:15.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77b07d700 1 -- 172.21.15.32:0/3784750844 >> v1:172.21.15.32:6789/0 conn(0x7ff77c0d8490 legacy=0x7ff77c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:15.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77b07d700 1 -- 172.21.15.32:0/3784750844 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c0d9220 con 0x7ff77c10d010 2021-04-09T18:17:15.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77a07b700 1 -- 172.21.15.32:0/3784750844 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389500558 0 0) 0x7ff774000d30 con 0x7ff77c0d8490 2021-04-09T18:17:15.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77b07d700 1 --2- 172.21.15.32:0/3784750844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c10d010 0x7ff77c0d1780 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7ff76c006650 tx=0x7ff76c009a00).ready entity=mon.0 client_cookie=a08063ae18218c1f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:15.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77a07b700 1 -- 172.21.15.32:0/3784750844 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff76c009df0 con 0x7ff77c10d010 2021-04-09T18:17:15.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77a07b700 1 -- 172.21.15.32:0/3784750844 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76c0009b0 con 0x7ff77c10d010 2021-04-09T18:17:15.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff77a07b700 1 -- 172.21.15.32:0/3784750844 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff76c00c2b0 con 0x7ff77c10d010 2021-04-09T18:17:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff7812f0700 1 -- 172.21.15.32:0/3784750844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c10d010 msgr2=0x7ff77c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.909 7ff7812f0700 1 --2- 172.21.15.32:0/3784750844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c10d010 0x7ff77c0d1780 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7ff76c006650 tx=0x7ff76c009a00).stop 2021-04-09T18:17:15.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.910 7ff7812f0700 1 -- 172.21.15.32:0/3784750844 shutdown_connections 2021-04-09T18:17:15.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.910 7ff7812f0700 1 --2- 172.21.15.32:0/3784750844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c10d010 0x7ff77c0d1780 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:15.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.910 7ff7812f0700 1 -- 172.21.15.32:0/3784750844 >> 172.21.15.32:0/3784750844 conn(0x7ff77c0ce5f0 msgr2=0x7ff77c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:15.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.910 7ff7812f0700 1 -- 172.21.15.32:0/3784750844 shutdown_connections 2021-04-09T18:17:15.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.910 7ff7812f0700 1 -- 172.21.15.32:0/3784750844 wait complete. 2021-04-09T18:17:15.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.930 7ff7812f0700 1 Processor -- start 2021-04-09T18:17:15.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.930 7ff7812f0700 1 -- start start 2021-04-09T18:17:15.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff7812f0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff77c1a5cc0 0x7ff77c1a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:15.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff7812f0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff77c1c0000 0x7ff77c1c0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:15.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff7812f0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c0b6b60 0x7ff77c0b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:15.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff7812f0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0b65e0 con 0x7ff77c0b6b60 2021-04-09T18:17:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff7812f0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0bca70 con 0x7ff77c1c0000 2021-04-09T18:17:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff7812f0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0b7580 con 0x7ff77c1a5cc0 2021-04-09T18:17:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c0b6b60 0x7ff77c0b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c0b6b60 0x7ff77c0b6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45356/0 (socket says 172.21.15.32:45356) 2021-04-09T18:17:15.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b87e700 1 -- 172.21.15.32:0/708818714 learned_addr learned my addr 172.21.15.32:0/708818714 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:15.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77a87c700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff77c1c0000 0x7ff77c1c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:15.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b07d700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff77c1a5cc0 0x7ff77c1a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:15.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b87e700 1 -- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff77c1a5cc0 msgr2=0x7ff77c1a61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b87e700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff77c1a5cc0 0x7ff77c1a61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b87e700 1 -- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff77c1c0000 msgr2=0x7ff77c1c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b87e700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff77c1c0000 0x7ff77c1c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b87e700 1 -- 172.21.15.32:0/708818714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c0b5570 con 0x7ff77c0b6b60 2021-04-09T18:17:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b07d700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff77c1a5cc0 0x7ff77c1a61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.931 7ff77b87e700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c0b6b60 0x7ff77c0b6f60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7ff77400a8f0 tx=0x7ff77400ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.932 7ff773fff700 1 -- 172.21.15.32:0/708818714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff774001270 con 0x7ff77c0b6b60 2021-04-09T18:17:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.932 7ff773fff700 1 -- 172.21.15.32:0/708818714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff774007020 con 0x7ff77c0b6b60 2021-04-09T18:17:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.932 7ff7812f0700 1 -- 172.21.15.32:0/708818714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c0b5570 con 0x7ff77c0b6b60 2021-04-09T18:17:16.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.932 7ff773fff700 1 -- 172.21.15.32:0/708818714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff774001270 con 0x7ff77c0b6b60 2021-04-09T18:17:16.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.932 7ff7812f0700 1 -- 172.21.15.32:0/708818714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c0b5570 con 0x7ff77c0b6b60 2021-04-09T18:17:16.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.932 7ff773fff700 1 -- 172.21.15.32:0/708818714 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff774007200 con 0x7ff77c0b6b60 2021-04-09T18:17:16.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.932 7ff773fff700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7600188d0 0x7ff76001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:16.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.932 7ff773fff700 1 -- 172.21.15.32:0/708818714 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff774032d10 con 0x7ff77c0b6b60 2021-04-09T18:17:16.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.933 7ff77b07d700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7600188d0 0x7ff76001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:16.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.933 7ff77b07d700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7600188d0 0x7ff76001add0 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:16.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.954 7ff7812f0700 1 -- 172.21.15.32:0/708818714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c1a47d0 con 0x7ff77c0b6b60 2021-04-09T18:17:16.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:15.957 7ff773fff700 1 -- 172.21.15.32:0/708818714 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff774007d10 con 0x7ff77c0b6b60 2021-04-09T18:17:16.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.150 7ff7812f0700 1 -- 172.21.15.32:0/708818714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff77c112070 con 0x7ff77c0b6b60 2021-04-09T18:17:16.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.150 7ff773fff700 1 -- 172.21.15.32:0/708818714 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2781 (secure 0 0 0) 0x7ff774007020 con 0x7ff77c0b6b60 2021-04-09T18:17:16.183 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2021-04-09T18:17:16.184 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:16.185 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:16.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.152 7ff7812f0700 1 -- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7600188d0 msgr2=0x7ff76001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:16.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.152 7ff7812f0700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7600188d0 0x7ff76001add0 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:16.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.152 7ff7812f0700 1 -- 172.21.15.32:0/708818714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c0b6b60 msgr2=0x7ff77c0b6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:16.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.152 7ff7812f0700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c0b6b60 0x7ff77c0b6f60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7ff77400a8f0 tx=0x7ff77400ac90).stop 2021-04-09T18:17:16.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.153 7ff7812f0700 1 -- 172.21.15.32:0/708818714 shutdown_connections 2021-04-09T18:17:16.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.153 7ff7812f0700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7600188d0 0x7ff76001add0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:16.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.153 7ff7812f0700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff77c0b6b60 0x7ff77c0b6f60 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:16.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.153 7ff7812f0700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff77c1c0000 0x7ff77c1c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:16.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.153 7ff7812f0700 1 --2- 172.21.15.32:0/708818714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff77c1a5cc0 0x7ff77c1a61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:16.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.153 7ff7812f0700 1 -- 172.21.15.32:0/708818714 >> 172.21.15.32:0/708818714 conn(0x7ff77c0bcd80 msgr2=0x7ff77c08dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:16.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.153 7ff7812f0700 1 -- 172.21.15.32:0/708818714 shutdown_connections 2021-04-09T18:17:16.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:16.153 7ff7812f0700 1 -- 172.21.15.32:0/708818714 wait complete. 2021-04-09T18:17:16.611 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:16.611 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:17.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:17.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.740 7f8edb83a700 1 Processor -- start 2021-04-09T18:17:17.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.740 7f8edb83a700 1 -- start start 2021-04-09T18:17:17.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8edb83a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed410d010 0x7f8ed40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:17.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8edb83a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8ed40d8490 0x7f8ed410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:17.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8edb83a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed410d480 con 0x7f8ed40d1da0 2021-04-09T18:17:17.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8edb83a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed410e780 con 0x7f8ed40d8490 2021-04-09T18:17:17.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8edb83a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40d8d30 con 0x7f8ed410d010 2021-04-09T18:17:17.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ed9e57700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8ed40d8490 0x7f8ed410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:17.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ed8e55700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8ed40d1da0 0x7f8ed40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:52880/0 (socket says 172.21.15.32:52880) 2021-04-09T18:17:17.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ed9656700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed410d010 0x7f8ed40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:17.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ed8e55700 1 -- 172.21.15.32:0/713912271 learned_addr learned my addr 172.21.15.32:0/713912271 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:17.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ecbfff700 1 -- 172.21.15.32:0/713912271 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946635013 0 0) 0x7f8ec40009b0 con 0x7f8ed40d1da0 2021-04-09T18:17:17.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ecbfff700 1 -- 172.21.15.32:0/713912271 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec0002fe0 con 0x7f8ed40d1da0 2021-04-09T18:17:17.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ed9656700 1 -- 172.21.15.32:0/713912271 >> v2:172.21.15.86:3300/0 conn(0x7f8ed40d8490 msgr2=0x7f8ed410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:17.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ed9656700 1 --2- 172.21.15.32:0/713912271 >> v2:172.21.15.86:3300/0 conn(0x7f8ed40d8490 0x7f8ed410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:17.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ed9656700 1 -- 172.21.15.32:0/713912271 >> v1:172.21.15.86:6789/0 conn(0x7f8ed40d1da0 legacy=0x7f8ed40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:17.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ed9656700 1 -- 172.21.15.32:0/713912271 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed40d9220 con 0x7f8ed410d010 2021-04-09T18:17:17.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.741 7f8ed9e57700 1 --2- 172.21.15.32:0/713912271 >> v2:172.21.15.86:3300/0 conn(0x7f8ed40d8490 0x7f8ed410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:17.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.742 7f8ed9656700 1 --2- 172.21.15.32:0/713912271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed410d010 0x7f8ed40d1780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f8ed00069d0 tx=0x7f8ed0009e40).ready entity=mon.2 client_cookie=7730228995981abf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:17.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.742 7f8ecbfff700 1 -- 172.21.15.32:0/713912271 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ed000b510 con 0x7f8ed410d010 2021-04-09T18:17:17.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.742 7f8ecbfff700 1 -- 172.21.15.32:0/713912271 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed000b890 con 0x7f8ed410d010 2021-04-09T18:17:17.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.742 7f8ecbfff700 1 -- 172.21.15.32:0/713912271 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ed000b510 con 0x7f8ed410d010 2021-04-09T18:17:17.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.742 7f8edb83a700 1 -- 172.21.15.32:0/713912271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed410d010 msgr2=0x7f8ed40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.742 7f8edb83a700 1 --2- 172.21.15.32:0/713912271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed410d010 0x7f8ed40d1780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f8ed00069d0 tx=0x7f8ed0009e40).stop 2021-04-09T18:17:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.743 7f8edb83a700 1 -- 172.21.15.32:0/713912271 shutdown_connections 2021-04-09T18:17:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.743 7f8edb83a700 1 --2- 172.21.15.32:0/713912271 >> v2:172.21.15.86:3300/0 conn(0x7f8ed40d8490 0x7f8ed410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.743 7f8edb83a700 1 --2- 172.21.15.32:0/713912271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed410d010 0x7f8ed40d1780 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:17.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.743 7f8edb83a700 1 -- 172.21.15.32:0/713912271 >> 172.21.15.32:0/713912271 conn(0x7f8ed40ce5f0 msgr2=0x7f8ed40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:17.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.743 7f8edb83a700 1 -- 172.21.15.32:0/713912271 shutdown_connections 2021-04-09T18:17:17.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.743 7f8edb83a700 1 -- 172.21.15.32:0/713912271 wait complete. 2021-04-09T18:17:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.763 7f8edb83a700 1 Processor -- start 2021-04-09T18:17:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.763 7f8edb83a700 1 -- start start 2021-04-09T18:17:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8edb83a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed40b7620 0x7f8ed4184c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8edb83a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed41852b0 0x7f8ed41a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:17.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8edb83a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed41a6840 0x7f8ed41a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:17.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8edb83a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed4184710 con 0x7f8ed41a6840 2021-04-09T18:17:17.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8edb83a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41856b0 con 0x7f8ed40b7620 2021-04-09T18:17:17.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8edb83a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed4182f80 con 0x7f8ed41852b0 2021-04-09T18:17:17.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9e57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed41a6840 0x7f8ed41a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:17.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9e57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed41a6840 0x7f8ed41a6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45366/0 (socket says 172.21.15.32:45366) 2021-04-09T18:17:17.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9e57700 1 -- 172.21.15.32:0/1605233240 learned_addr learned my addr 172.21.15.32:0/1605233240 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9656700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed40b7620 0x7f8ed4184c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:17.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed8e55700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed41852b0 0x7f8ed41a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:17.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9e57700 1 -- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed41852b0 msgr2=0x7f8ed41a6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:17.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9e57700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed41852b0 0x7f8ed41a6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:17.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9e57700 1 -- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed40b7620 msgr2=0x7f8ed4184c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:17.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9e57700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed40b7620 0x7f8ed4184c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:17.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9e57700 1 -- 172.21.15.32:0/1605233240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed41831a0 con 0x7f8ed41a6840 2021-04-09T18:17:17.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9656700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed40b7620 0x7f8ed4184c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:17.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ed9e57700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed41a6840 0x7f8ed41a6c40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f8ecc00b5e0 tx=0x7f8ecc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:17.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ec9ffb700 1 -- 172.21.15.32:0/1605233240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ecc00bd70 con 0x7f8ed41a6840 2021-04-09T18:17:17.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ec9ffb700 1 -- 172.21.15.32:0/1605233240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc006410 con 0x7f8ed41a6840 2021-04-09T18:17:17.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed41831a0 con 0x7f8ed41a6840 2021-04-09T18:17:17.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8ec9ffb700 1 -- 172.21.15.32:0/1605233240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ecc002560 con 0x7f8ed41a6840 2021-04-09T18:17:17.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.764 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed41831a0 con 0x7f8ed41a6840 2021-04-09T18:17:17.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.765 7f8ec9ffb700 1 -- 172.21.15.32:0/1605233240 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8ecc005330 con 0x7f8ed41a6840 2021-04-09T18:17:17.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.765 7f8ec9ffb700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8eb4018930 0x7f8eb401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:17.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.765 7f8ec9ffb700 1 -- 172.21.15.32:0/1605233240 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8ecc032a20 con 0x7f8ed41a6840 2021-04-09T18:17:17.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.765 7f8ed9656700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8eb4018930 0x7f8eb401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:17.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.766 7f8ed9656700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8eb4018930 0x7f8eb401ae10 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:17.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.785 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed410d480 con 0x7f8ed41a6840 2021-04-09T18:17:17.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.788 7f8ec9ffb700 1 -- 172.21.15.32:0/1605233240 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8ecc0067f0 con 0x7f8ed41a6840 2021-04-09T18:17:18.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.978 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ed410d480 con 0x7f8ed41a6840 2021-04-09T18:17:18.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.978 7f8ec9ffb700 1 -- 172.21.15.32:0/1605233240 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2781 (secure 0 0 0) 0x7f8ecc00bd70 con 0x7f8ed41a6840 2021-04-09T18:17:18.012 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2021-04-09T18:17:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:18.014 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:18.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.980 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8eb4018930 msgr2=0x7f8eb401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:18.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.980 7f8edb83a700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8eb4018930 0x7f8eb401ae10 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:18.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.980 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed41a6840 msgr2=0x7f8ed41a6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:18.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.980 7f8edb83a700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed41a6840 0x7f8ed41a6c40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f8ecc00b5e0 tx=0x7f8ecc00b980).stop 2021-04-09T18:17:18.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.981 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 shutdown_connections 2021-04-09T18:17:18.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.981 7f8edb83a700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8eb4018930 0x7f8eb401ae10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:18.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.981 7f8edb83a700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ed41a6840 0x7f8ed41a6c40 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:18.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.981 7f8edb83a700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed41852b0 0x7f8ed41a6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:18.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.981 7f8edb83a700 1 --2- 172.21.15.32:0/1605233240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed40b7620 0x7f8ed4184c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:18.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.981 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 >> 172.21.15.32:0/1605233240 conn(0x7f8ed419f830 msgr2=0x7f8ed419eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:18.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.981 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 shutdown_connections 2021-04-09T18:17:18.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:17.981 7f8edb83a700 1 -- 172.21.15.32:0/1605233240 wait complete. 2021-04-09T18:17:18.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:18.437 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:19.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:19.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.559 7f58ab9d0700 1 Processor -- start 2021-04-09T18:17:19.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.559 7f58ab9d0700 1 -- start start 2021-04-09T18:17:19.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f58ab9d0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a410da00 0x7f58a410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:19.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f58ab9d0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a410d480 con 0x7f58a410d010 2021-04-09T18:17:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f58ab9d0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a40d1f70 con 0x7f58a40d7ca0 2021-04-09T18:17:19.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f58ab9d0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a40d9220 con 0x7f58a410da00 2021-04-09T18:17:19.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f58a97ec700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f58a410d010 0x7f58a40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54964/0 (socket says 172.21.15.32:54964) 2021-04-09T18:17:19.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f58a97ec700 1 -- 172.21.15.32:0/2227465043 learned_addr learned my addr 172.21.15.32:0/2227465043 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:19.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f58a9fed700 1 --2- 172.21.15.32:0/2227465043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a410da00 0x7f58a410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:19.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f589bfff700 1 -- 172.21.15.32:0/2227465043 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263369517 0 0) 0x7f58a0000ce0 con 0x7f58a410d010 2021-04-09T18:17:19.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f589bfff700 1 -- 172.21.15.32:0/2227465043 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5890002fe0 con 0x7f58a410d010 2021-04-09T18:17:19.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f589bfff700 1 -- 172.21.15.32:0/2227465043 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691506110 0 0) 0x7f5894000c60 con 0x7f58a40d7ca0 2021-04-09T18:17:19.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.560 7f589bfff700 1 -- 172.21.15.32:0/2227465043 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5890003020 con 0x7f58a40d7ca0 2021-04-09T18:17:19.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f58a9fed700 1 -- 172.21.15.32:0/2227465043 >> v1:172.21.15.86:6789/0 conn(0x7f58a40d7ca0 legacy=0x7f58a40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:19.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f58a9fed700 1 -- 172.21.15.32:0/2227465043 >> v1:172.21.15.32:6789/0 conn(0x7f58a410d010 legacy=0x7f58a40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:19.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f589bfff700 1 -- 172.21.15.32:0/2227465043 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3307763056 0 0) 0x7f58a0000ed0 con 0x7f58a410d010 2021-04-09T18:17:19.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f58a9fed700 1 -- 172.21.15.32:0/2227465043 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a410e420 con 0x7f58a410da00 2021-04-09T18:17:19.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f58a9fed700 1 --2- 172.21.15.32:0/2227465043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a410da00 0x7f58a410de00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f589c006e50 tx=0x7f589c009df0).ready entity=mon.2 client_cookie=cc135e92e701f150 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:19.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f589bfff700 1 -- 172.21.15.32:0/2227465043 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c00b510 con 0x7f58a410da00 2021-04-09T18:17:19.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f589bfff700 1 -- 172.21.15.32:0/2227465043 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c003030 con 0x7f58a410da00 2021-04-09T18:17:19.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f589bfff700 1 -- 172.21.15.32:0/2227465043 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c00b7d0 con 0x7f58a410da00 2021-04-09T18:17:19.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f58ab9d0700 1 -- 172.21.15.32:0/2227465043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a410da00 msgr2=0x7f58a410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:19.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.561 7f58ab9d0700 1 --2- 172.21.15.32:0/2227465043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a410da00 0x7f58a410de00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f589c006e50 tx=0x7f589c009df0).stop 2021-04-09T18:17:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.562 7f58ab9d0700 1 -- 172.21.15.32:0/2227465043 shutdown_connections 2021-04-09T18:17:19.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.562 7f58ab9d0700 1 --2- 172.21.15.32:0/2227465043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a410da00 0x7f58a410de00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:19.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.562 7f58ab9d0700 1 -- 172.21.15.32:0/2227465043 >> 172.21.15.32:0/2227465043 conn(0x7f58a40ce5f0 msgr2=0x7f58a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:19.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.562 7f58ab9d0700 1 -- 172.21.15.32:0/2227465043 shutdown_connections 2021-04-09T18:17:19.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.562 7f58ab9d0700 1 -- 172.21.15.32:0/2227465043 wait complete. 2021-04-09T18:17:19.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.582 7f58ab9d0700 1 Processor -- start 2021-04-09T18:17:19.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.582 7f58ab9d0700 1 -- start start 2021-04-09T18:17:19.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58ab9d0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a40b42d0 0x7f58a4183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:19.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58ab9d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41ac090 0x7f58a41ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:19.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58ab9d0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a4184d30 0x7f58a4185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:19.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58ab9d0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a40a1a90 con 0x7f58a41ac090 2021-04-09T18:17:19.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58ab9d0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a41acab0 con 0x7f58a40b42d0 2021-04-09T18:17:19.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58ab9d0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4185a00 con 0x7f58a4184d30 2021-04-09T18:17:19.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a8feb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41ac090 0x7f58a41ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:19.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a8feb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41ac090 0x7f58a41ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45378/0 (socket says 172.21.15.32:45378) 2021-04-09T18:17:19.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a8feb700 1 -- 172.21.15.32:0/440129542 learned_addr learned my addr 172.21.15.32:0/440129542 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:19.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a97ec700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a40b42d0 0x7f58a4183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:19.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a9fed700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a4184d30 0x7f58a4185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:19.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a8feb700 1 -- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a4184d30 msgr2=0x7f58a4185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:19.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a8feb700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a4184d30 0x7f58a4185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:19.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a8feb700 1 -- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a40b42d0 msgr2=0x7f58a4183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:19.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a8feb700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a40b42d0 0x7f58a4183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:19.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a8feb700 1 -- 172.21.15.32:0/440129542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a4185750 con 0x7f58a41ac090 2021-04-09T18:17:19.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f58a8feb700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41ac090 0x7f58a41ac490 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f5894006f10 tx=0x7f589400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:19.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f5899ffb700 1 -- 172.21.15.32:0/440129542 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589400bd80 con 0x7f58a41ac090 2021-04-09T18:17:19.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.583 7f5899ffb700 1 -- 172.21.15.32:0/440129542 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5894000e20 con 0x7f58a41ac090 2021-04-09T18:17:19.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.584 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a4185750 con 0x7f58a41ac090 2021-04-09T18:17:19.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.584 7f5899ffb700 1 -- 172.21.15.32:0/440129542 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58940054f0 con 0x7f58a41ac090 2021-04-09T18:17:19.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.584 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a4185750 con 0x7f58a41ac090 2021-04-09T18:17:19.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.584 7f5899ffb700 1 -- 172.21.15.32:0/440129542 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5894009fe0 con 0x7f58a41ac090 2021-04-09T18:17:19.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.584 7f5899ffb700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5884018930 0x7f588401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:19.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.584 7f5899ffb700 1 -- 172.21.15.32:0/440129542 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5894032a10 con 0x7f58a41ac090 2021-04-09T18:17:19.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.584 7f58a97ec700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5884018930 0x7f588401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:19.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.585 7f58a97ec700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5884018930 0x7f588401ae10 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:19.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.604 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a40a1a90 con 0x7f58a41ac090 2021-04-09T18:17:19.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.608 7f5899ffb700 1 -- 172.21.15.32:0/440129542 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5894000e20 con 0x7f58a41ac090 2021-04-09T18:17:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.800 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58a4185750 con 0x7f58a41ac090 2021-04-09T18:17:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.801 7f5899ffb700 1 -- 172.21.15.32:0/440129542 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2781 (secure 0 0 0) 0x7f5894009fe0 con 0x7f58a41ac090 2021-04-09T18:17:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2021-04-09T18:17:19.835 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:19.835 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5884018930 msgr2=0x7f588401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5884018930 0x7f588401ae10 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41ac090 msgr2=0x7f58a41ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41ac090 0x7f58a41ac490 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f5894006f10 tx=0x7f589400b990).stop 2021-04-09T18:17:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 shutdown_connections 2021-04-09T18:17:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5884018930 0x7f588401ae10 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41ac090 0x7f58a41ac490 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a4184d30 0x7f58a4185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 --2- 172.21.15.32:0/440129542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a40b42d0 0x7f58a4183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 >> 172.21.15.32:0/440129542 conn(0x7f58a419b890 msgr2=0x7f58a41a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 shutdown_connections 2021-04-09T18:17:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:19.803 7f58ab9d0700 1 -- 172.21.15.32:0/440129542 wait complete. 2021-04-09T18:17:20.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:20.253 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:21.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.384 7f588c77b700 1 Processor -- start 2021-04-09T18:17:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.385 7f588c77b700 1 -- start start 2021-04-09T18:17:21.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.385 7f588c77b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f58840d7ca0 0x7f58840d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:21.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.385 7f588c77b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f588410da00 0x7f588410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:21.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.385 7f588c77b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588410d480 con 0x7f588410d010 2021-04-09T18:17:21.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.385 7f588c77b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58840d9220 con 0x7f58840d7ca0 2021-04-09T18:17:21.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.385 7f588c77b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58840d86c0 con 0x7f588410da00 2021-04-09T18:17:21.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f588a597700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f588410d010 0x7f58840d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:54976/0 (socket says 172.21.15.32:54976) 2021-04-09T18:17:21.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f588a597700 1 -- 172.21.15.32:0/1330571762 learned_addr learned my addr 172.21.15.32:0/1330571762 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:21.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889d96700 1 --2- 172.21.15.32:0/1330571762 >> v2:172.21.15.86:3300/0 conn(0x7f58840d7ca0 0x7f58840d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:21.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f588ad98700 1 --2- 172.21.15.32:0/1330571762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f588410da00 0x7f588410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:21.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889595700 1 -- 172.21.15.32:0/1330571762 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586168396 0 0) 0x7f5880000ce0 con 0x7f588410d010 2021-04-09T18:17:21.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889595700 1 -- 172.21.15.32:0/1330571762 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5870002fe0 con 0x7f588410d010 2021-04-09T18:17:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889d96700 1 -- 172.21.15.32:0/1330571762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f588410da00 msgr2=0x7f588410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:21.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889d96700 1 --2- 172.21.15.32:0/1330571762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f588410da00 0x7f588410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889d96700 1 -- 172.21.15.32:0/1330571762 >> v1:172.21.15.32:6789/0 conn(0x7f588410d010 legacy=0x7f58840d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889d96700 1 -- 172.21.15.32:0/1330571762 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58840d1f70 con 0x7f58840d7ca0 2021-04-09T18:17:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f588ad98700 1 --2- 172.21.15.32:0/1330571762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f588410da00 0x7f588410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889d96700 1 --2- 172.21.15.32:0/1330571762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58840d7ca0 0x7f58840d80a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f5874006650 tx=0x7f5874009a80).ready entity=mon.1 client_cookie=683e47e780cdf297 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889595700 1 -- 172.21.15.32:0/1330571762 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58740009b0 con 0x7f58840d7ca0 2021-04-09T18:17:21.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889595700 1 -- 172.21.15.32:0/1330571762 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587400c0c0 con 0x7f58840d7ca0 2021-04-09T18:17:21.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f5889595700 1 -- 172.21.15.32:0/1330571762 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f587400c510 con 0x7f58840d7ca0 2021-04-09T18:17:21.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f588c77b700 1 -- 172.21.15.32:0/1330571762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58840d7ca0 msgr2=0x7f58840d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:21.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.386 7f588c77b700 1 --2- 172.21.15.32:0/1330571762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58840d7ca0 0x7f58840d80a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f5874006650 tx=0x7f5874009a80).stop 2021-04-09T18:17:21.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.387 7f588c77b700 1 -- 172.21.15.32:0/1330571762 shutdown_connections 2021-04-09T18:17:21.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.387 7f588c77b700 1 --2- 172.21.15.32:0/1330571762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f588410da00 0x7f588410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.387 7f588c77b700 1 --2- 172.21.15.32:0/1330571762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58840d7ca0 0x7f58840d80a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.387 7f588c77b700 1 -- 172.21.15.32:0/1330571762 >> 172.21.15.32:0/1330571762 conn(0x7f58840ce5f0 msgr2=0x7f58840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:21.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.387 7f588c77b700 1 -- 172.21.15.32:0/1330571762 shutdown_connections 2021-04-09T18:17:21.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.387 7f588c77b700 1 -- 172.21.15.32:0/1330571762 wait complete. 2021-04-09T18:17:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.407 7f588c77b700 1 Processor -- start 2021-04-09T18:17:21.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.407 7f588c77b700 1 -- start start 2021-04-09T18:17:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588c77b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840a90c0 0x7f58841a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:21.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588c77b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58841a76d0 0x7f58841a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:21.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588c77b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f588417bce0 0x7f588417c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:21.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588c77b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58841a7c60 con 0x7f58840a90c0 2021-04-09T18:17:21.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588c77b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58841a8e20 con 0x7f588417bce0 2021-04-09T18:17:21.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588c77b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588417c700 con 0x7f58841a76d0 2021-04-09T18:17:21.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588a597700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840a90c0 0x7f58841a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:21.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588a597700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840a90c0 0x7f58841a81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45392/0 (socket says 172.21.15.32:45392) 2021-04-09T18:17:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588a597700 1 -- 172.21.15.32:0/1278919366 learned_addr learned my addr 172.21.15.32:0/1278919366 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:21.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588ad98700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f588417bce0 0x7f588417c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f5889d96700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58841a76d0 0x7f58841a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:21.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588a597700 1 -- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58841a76d0 msgr2=0x7f58841a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:21.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588a597700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58841a76d0 0x7f58841a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588a597700 1 -- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f588417bce0 msgr2=0x7f588417c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588a597700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f588417bce0 0x7f588417c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588a597700 1 -- 172.21.15.32:0/1278919366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588417c920 con 0x7f58840a90c0 2021-04-09T18:17:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.408 7f588ad98700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f588417bce0 0x7f588417c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.409 7f588a597700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840a90c0 0x7f58841a81e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f5880006f00 tx=0x7f588000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:21.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.409 7f587affd700 1 -- 172.21.15.32:0/1278919366 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f588000bd30 con 0x7f58840a90c0 2021-04-09T18:17:21.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.409 7f587affd700 1 -- 172.21.15.32:0/1278919366 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5880006030 con 0x7f58840a90c0 2021-04-09T18:17:21.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.409 7f588c77b700 1 -- 172.21.15.32:0/1278919366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588417c920 con 0x7f58840a90c0 2021-04-09T18:17:21.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.409 7f587affd700 1 -- 172.21.15.32:0/1278919366 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5880004620 con 0x7f58840a90c0 2021-04-09T18:17:21.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.409 7f588c77b700 1 -- 172.21.15.32:0/1278919366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588417c920 con 0x7f58840a90c0 2021-04-09T18:17:21.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.409 7f587affd700 1 -- 172.21.15.32:0/1278919366 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f58800043a0 con 0x7f58840a90c0 2021-04-09T18:17:21.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.410 7f587affd700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f586c018930 0x7f586c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:21.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.410 7f587affd700 1 -- 172.21.15.32:0/1278919366 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f58800329e0 con 0x7f58840a90c0 2021-04-09T18:17:21.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.410 7f5889d96700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f586c018930 0x7f586c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:21.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.410 7f5889d96700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f586c018930 0x7f586c01adc0 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:21.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.429 7f588c77b700 1 -- 172.21.15.32:0/1278919366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58841b7010 con 0x7f58840a90c0 2021-04-09T18:17:21.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.433 7f587affd700 1 -- 172.21.15.32:0/1278919366 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f58800009b0 con 0x7f58840a90c0 2021-04-09T18:17:21.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.623 7f588c77b700 1 -- 172.21.15.32:0/1278919366 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f588410d480 con 0x7f58840a90c0 2021-04-09T18:17:21.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.623 7f587affd700 1 -- 172.21.15.32:0/1278919366 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2781 (secure 0 0 0) 0x7f588000bd30 con 0x7f58840a90c0 2021-04-09T18:17:21.665 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2021-04-09T18:17:21.679 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:21.680 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:21.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 -- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f586c018930 msgr2=0x7f586c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:21.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f586c018930 0x7f586c01adc0 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 -- 172.21.15.32:0/1278919366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840a90c0 msgr2=0x7f58841a81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:21.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840a90c0 0x7f58841a81e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f5880006f00 tx=0x7f588000b940).stop 2021-04-09T18:17:21.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 -- 172.21.15.32:0/1278919366 shutdown_connections 2021-04-09T18:17:21.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f586c018930 0x7f586c01adc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f588417bce0 0x7f588417c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58841a76d0 0x7f58841a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 --2- 172.21.15.32:0/1278919366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58840a90c0 0x7f58841a81e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:21.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 -- 172.21.15.32:0/1278919366 >> 172.21.15.32:0/1278919366 conn(0x7f58841acae0 msgr2=0x7f588417ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:21.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 -- 172.21.15.32:0/1278919366 shutdown_connections 2021-04-09T18:17:21.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:21.626 7f588c77b700 1 -- 172.21.15.32:0/1278919366 wait complete. 2021-04-09T18:17:22.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:22.117 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:23.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:23.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.251 7f6c1f244700 1 Processor -- start 2021-04-09T18:17:23.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.251 7f6c1f244700 1 -- start start 2021-04-09T18:17:23.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1f244700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c1810d010 0x7f6c180d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:23.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1f244700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6c180d1da0 0x7f6c180d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:23.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1f244700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c180d82c0 0x7f6c1810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:23.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1f244700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c180d8d30 con 0x7f6c1810d010 2021-04-09T18:17:23.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1f244700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c180d9220 con 0x7f6c180d1da0 2021-04-09T18:17:23.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1f244700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1810d4b0 con 0x7f6c180d82c0 2021-04-09T18:17:23.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d060700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c1810d010 0x7f6c180d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:23.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d060700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6c1810d010 0x7f6c180d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45400/0 (socket says 172.21.15.32:45400) 2021-04-09T18:17:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d060700 1 -- 172.21.15.32:0/3954442898 learned_addr learned my addr 172.21.15.32:0/3954442898 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:23.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d861700 1 --2- 172.21.15.32:0/3954442898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c180d82c0 0x7f6c1810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:23.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1c85f700 1 --2- 172.21.15.32:0/3954442898 >> v2:172.21.15.86:3300/0 conn(0x7f6c180d1da0 0x7f6c180d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:23.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d060700 1 -- 172.21.15.32:0/3954442898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c180d82c0 msgr2=0x7f6c1810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:23.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d060700 1 --2- 172.21.15.32:0/3954442898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c180d82c0 0x7f6c1810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d060700 1 -- 172.21.15.32:0/3954442898 >> v2:172.21.15.86:3300/0 conn(0x7f6c180d1da0 msgr2=0x7f6c180d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:23.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d060700 1 --2- 172.21.15.32:0/3954442898 >> v2:172.21.15.86:3300/0 conn(0x7f6c180d1da0 0x7f6c180d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d060700 1 -- 172.21.15.32:0/3954442898 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c180d86c0 con 0x7f6c1810d010 2021-04-09T18:17:23.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1c85f700 1 --2- 172.21.15.32:0/3954442898 >> v2:172.21.15.86:3300/0 conn(0x7f6c180d1da0 0x7f6c180d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:23.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.252 7f6c1d060700 1 --2- 172.21.15.32:0/3954442898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1810d010 0x7f6c180d1780 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f6c14006700 tx=0x7f6c14009b80).ready entity=mon.0 client_cookie=995db5e5c92daf58 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:23.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c0bfff700 1 -- 172.21.15.32:0/3954442898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c1400c0c0 con 0x7f6c1810d010 2021-04-09T18:17:23.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c0bfff700 1 -- 172.21.15.32:0/3954442898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1400c440 con 0x7f6c1810d010 2021-04-09T18:17:23.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c0bfff700 1 -- 172.21.15.32:0/3954442898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c1400c920 con 0x7f6c1810d010 2021-04-09T18:17:23.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c1f244700 1 -- 172.21.15.32:0/3954442898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1810d010 msgr2=0x7f6c180d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:23.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c1f244700 1 --2- 172.21.15.32:0/3954442898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1810d010 0x7f6c180d1780 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f6c14006700 tx=0x7f6c14009b80).stop 2021-04-09T18:17:23.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c1f244700 1 -- 172.21.15.32:0/3954442898 shutdown_connections 2021-04-09T18:17:23.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c1f244700 1 --2- 172.21.15.32:0/3954442898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c180d82c0 0x7f6c1810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c1f244700 1 --2- 172.21.15.32:0/3954442898 >> v2:172.21.15.86:3300/0 conn(0x7f6c180d1da0 0x7f6c180d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c1f244700 1 --2- 172.21.15.32:0/3954442898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1810d010 0x7f6c180d1780 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.253 7f6c1f244700 1 -- 172.21.15.32:0/3954442898 >> 172.21.15.32:0/3954442898 conn(0x7f6c180ce5f0 msgr2=0x7f6c180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:23.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.254 7f6c1f244700 1 -- 172.21.15.32:0/3954442898 shutdown_connections 2021-04-09T18:17:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.254 7f6c1f244700 1 -- 172.21.15.32:0/3954442898 wait complete. 2021-04-09T18:17:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.274 7f6c1f244700 1 Processor -- start 2021-04-09T18:17:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.274 7f6c1f244700 1 -- start start 2021-04-09T18:17:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1f244700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c181acea0 0x7f6c181ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:23.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1f244700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1819b2c0 0x7f6c18184da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:23.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1f244700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c181853c0 0x7f6c18185fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:23.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1f244700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c181857c0 con 0x7f6c1819b2c0 2021-04-09T18:17:23.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1f244700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c18185a40 con 0x7f6c181acea0 2021-04-09T18:17:23.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1f244700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c181865e0 con 0x7f6c181853c0 2021-04-09T18:17:23.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1c85f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1819b2c0 0x7f6c18184da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:23.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1c85f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1819b2c0 0x7f6c18184da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45410/0 (socket says 172.21.15.32:45410) 2021-04-09T18:17:23.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1d060700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c181acea0 0x7f6c181ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:23.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1c85f700 1 -- 172.21.15.32:0/662153 learned_addr learned my addr 172.21.15.32:0/662153 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:23.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1d861700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c181853c0 0x7f6c18185fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:23.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1c85f700 1 -- 172.21.15.32:0/662153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c181853c0 msgr2=0x7f6c18185fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:23.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1c85f700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c181853c0 0x7f6c18185fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1c85f700 1 -- 172.21.15.32:0/662153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c181acea0 msgr2=0x7f6c181ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:23.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1c85f700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c181acea0 0x7f6c181ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1c85f700 1 -- 172.21.15.32:0/662153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c18186800 con 0x7f6c1819b2c0 2021-04-09T18:17:23.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.275 7f6c1c85f700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1819b2c0 0x7f6c18184da0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f6c0c006f20 tx=0x7f6c0c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:23.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.276 7f6c09ffb700 1 -- 172.21.15.32:0/662153 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c0c00bd80 con 0x7f6c1819b2c0 2021-04-09T18:17:23.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.276 7f6c09ffb700 1 -- 172.21.15.32:0/662153 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c000c90 con 0x7f6c1819b2c0 2021-04-09T18:17:23.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.276 7f6c1f244700 1 -- 172.21.15.32:0/662153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c18186800 con 0x7f6c1819b2c0 2021-04-09T18:17:23.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.276 7f6c09ffb700 1 -- 172.21.15.32:0/662153 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c0c0022b0 con 0x7f6c1819b2c0 2021-04-09T18:17:23.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.276 7f6c1f244700 1 -- 172.21.15.32:0/662153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c18186800 con 0x7f6c1819b2c0 2021-04-09T18:17:23.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.276 7f6c09ffb700 1 -- 172.21.15.32:0/662153 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6c0c00a000 con 0x7f6c1819b2c0 2021-04-09T18:17:23.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.276 7f6c09ffb700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bf8018930 0x7f6bf801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:23.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.276 7f6c09ffb700 1 -- 172.21.15.32:0/662153 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6c0c032a30 con 0x7f6c1819b2c0 2021-04-09T18:17:23.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.277 7f6c1d060700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bf8018930 0x7f6bf801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:23.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.277 7f6c1d060700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bf8018930 0x7f6bf801ae10 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:23.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.296 7f6c1f244700 1 -- 172.21.15.32:0/662153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c18182d80 con 0x7f6c1819b2c0 2021-04-09T18:17:23.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.299 7f6c09ffb700 1 -- 172.21.15.32:0/662153 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6c0c006bb0 con 0x7f6c1819b2c0 2021-04-09T18:17:23.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.492 7f6c1f244700 1 -- 172.21.15.32:0/662153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c1810d480 con 0x7f6c1819b2c0 2021-04-09T18:17:23.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.492 7f6c09ffb700 1 -- 172.21.15.32:0/662153 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2781 (secure 0 0 0) 0x7f6c0c006bb0 con 0x7f6c1819b2c0 2021-04-09T18:17:23.527 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2021-04-09T18:17:23.528 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:23.528 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:23.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 -- 172.21.15.32:0/662153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bf8018930 msgr2=0x7f6bf801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:23.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bf8018930 0x7f6bf801ae10 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 -- 172.21.15.32:0/662153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1819b2c0 msgr2=0x7f6c18184da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:23.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1819b2c0 0x7f6c18184da0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f6c0c006f20 tx=0x7f6c0c00b990).stop 2021-04-09T18:17:23.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 -- 172.21.15.32:0/662153 shutdown_connections 2021-04-09T18:17:23.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bf8018930 0x7f6bf801ae10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c1819b2c0 0x7f6c18184da0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c181853c0 0x7f6c18185fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 --2- 172.21.15.32:0/662153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c181acea0 0x7f6c181ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:23.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 -- 172.21.15.32:0/662153 >> 172.21.15.32:0/662153 conn(0x7f6c1819b890 msgr2=0x7f6c181a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:23.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 -- 172.21.15.32:0/662153 shutdown_connections 2021-04-09T18:17:23.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:23.495 7f6c1f244700 1 -- 172.21.15.32:0/662153 wait complete. 2021-04-09T18:17:23.972 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:23.973 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:24.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:25.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.094 7fef34379700 1 Processor -- start 2021-04-09T18:17:25.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.094 7fef34379700 1 -- start start 2021-04-09T18:17:25.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.094 7fef34379700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fef2c0d7ca0 0x7fef2c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:25.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.094 7fef34379700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c10da00 0x7fef2c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:25.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.094 7fef34379700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c10d480 con 0x7fef2c10d010 2021-04-09T18:17:25.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.094 7fef34379700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2c0d9220 con 0x7fef2c0d7ca0 2021-04-09T18:17:25.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.094 7fef34379700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0d86c0 con 0x7fef2c10da00 2021-04-09T18:17:25.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef32195700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fef2c10d010 0x7fef2c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55004/0 (socket says 172.21.15.32:55004) 2021-04-09T18:17:25.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31994700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fef2c0d7ca0 0x7fef2c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:25.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef32195700 1 -- 172.21.15.32:0/152418862 learned_addr learned my addr 172.21.15.32:0/152418862 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:25.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef32996700 1 --2- 172.21.15.32:0/152418862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c10da00 0x7fef2c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31193700 1 -- 172.21.15.32:0/152418862 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137673776 0 0) 0x7fef28000ce0 con 0x7fef2c10d010 2021-04-09T18:17:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31193700 1 -- 172.21.15.32:0/152418862 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef18002fe0 con 0x7fef2c10d010 2021-04-09T18:17:25.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31994700 1 -- 172.21.15.32:0/152418862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c10da00 msgr2=0x7fef2c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:25.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31994700 1 --2- 172.21.15.32:0/152418862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c10da00 0x7fef2c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31994700 1 -- 172.21.15.32:0/152418862 >> v1:172.21.15.32:6789/0 conn(0x7fef2c10d010 legacy=0x7fef2c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:25.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31994700 1 -- 172.21.15.32:0/152418862 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0d1f70 con 0x7fef2c0d7ca0 2021-04-09T18:17:25.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31994700 1 --2- 172.21.15.32:0/152418862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0d7ca0 0x7fef2c0d80a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fef20006650 tx=0x7fef20009a80).ready entity=mon.0 client_cookie=17cc2aaddadbba5c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31193700 1 -- 172.21.15.32:0/152418862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef200009b0 con 0x7fef2c0d7ca0 2021-04-09T18:17:25.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31193700 1 -- 172.21.15.32:0/152418862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2000c0c0 con 0x7fef2c0d7ca0 2021-04-09T18:17:25.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef31193700 1 -- 172.21.15.32:0/152418862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef2000c510 con 0x7fef2c0d7ca0 2021-04-09T18:17:25.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef34379700 1 -- 172.21.15.32:0/152418862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0d7ca0 msgr2=0x7fef2c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:25.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.095 7fef34379700 1 --2- 172.21.15.32:0/152418862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0d7ca0 0x7fef2c0d80a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fef20006650 tx=0x7fef20009a80).stop 2021-04-09T18:17:25.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.096 7fef34379700 1 -- 172.21.15.32:0/152418862 shutdown_connections 2021-04-09T18:17:25.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.096 7fef34379700 1 --2- 172.21.15.32:0/152418862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c10da00 0x7fef2c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.096 7fef34379700 1 --2- 172.21.15.32:0/152418862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0d7ca0 0x7fef2c0d80a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.096 7fef34379700 1 -- 172.21.15.32:0/152418862 >> 172.21.15.32:0/152418862 conn(0x7fef2c0ce5f0 msgr2=0x7fef2c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.096 7fef34379700 1 -- 172.21.15.32:0/152418862 shutdown_connections 2021-04-09T18:17:25.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.096 7fef34379700 1 -- 172.21.15.32:0/152418862 wait complete. 2021-04-09T18:17:25.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.116 7fef34379700 1 Processor -- start 2021-04-09T18:17:25.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.116 7fef34379700 1 -- start start 2021-04-09T18:17:25.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef34379700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef2c18db50 0x7fef2c19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:25.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef34379700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c18df50 0x7fef2c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:25.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef34379700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0b7df0 0x7fef2c0b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:25.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef34379700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c199b00 con 0x7fef2c0b7df0 2021-04-09T18:17:25.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef34379700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0b8810 con 0x7fef2c18db50 2021-04-09T18:17:25.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef34379700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0b8a30 con 0x7fef2c18df50 2021-04-09T18:17:25.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef32996700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0b7df0 0x7fef2c0b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:25.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef32996700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0b7df0 0x7fef2c0b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45424/0 (socket says 172.21.15.32:45424) 2021-04-09T18:17:25.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef32996700 1 -- 172.21.15.32:0/1242330759 learned_addr learned my addr 172.21.15.32:0/1242330759 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:25.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef31994700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c18df50 0x7fef2c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:25.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef32195700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef2c18db50 0x7fef2c19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:25.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef32996700 1 -- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c18df50 msgr2=0x7fef2c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:25.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef32996700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c18df50 0x7fef2c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef32996700 1 -- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef2c18db50 msgr2=0x7fef2c19a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:25.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef32996700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef2c18db50 0x7fef2c19a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.117 7fef32996700 1 -- 172.21.15.32:0/1242330759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c198500 con 0x7fef2c0b7df0 2021-04-09T18:17:25.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.118 7fef32996700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0b7df0 0x7fef2c0b81f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7fef2400b970 tx=0x7fef2400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:25.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.118 7fef1effd700 1 -- 172.21.15.32:0/1242330759 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef24000f90 con 0x7fef2c0b7df0 2021-04-09T18:17:25.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.118 7fef1effd700 1 -- 172.21.15.32:0/1242330759 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef24006aa0 con 0x7fef2c0b7df0 2021-04-09T18:17:25.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.118 7fef1effd700 1 -- 172.21.15.32:0/1242330759 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef240015f0 con 0x7fef2c0b7df0 2021-04-09T18:17:25.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.118 7fef34379700 1 -- 172.21.15.32:0/1242330759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c198500 con 0x7fef2c0b7df0 2021-04-09T18:17:25.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.118 7fef34379700 1 -- 172.21.15.32:0/1242330759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c198500 con 0x7fef2c0b7df0 2021-04-09T18:17:25.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.118 7fef1effd700 1 -- 172.21.15.32:0/1242330759 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fef24000ed0 con 0x7fef2c0b7df0 2021-04-09T18:17:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.118 7fef1effd700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef14018930 0x7fef1401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.118 7fef1effd700 1 -- 172.21.15.32:0/1242330759 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fef24032da0 con 0x7fef2c0b7df0 2021-04-09T18:17:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.119 7fef32195700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef14018930 0x7fef1401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:25.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.119 7fef32195700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef14018930 0x7fef1401ae10 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:25.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.138 7fef34379700 1 -- 172.21.15.32:0/1242330759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c182d70 con 0x7fef2c0b7df0 2021-04-09T18:17:25.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.142 7fef1effd700 1 -- 172.21.15.32:0/1242330759 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fef240066f0 con 0x7fef2c0b7df0 2021-04-09T18:17:25.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.334 7fef34379700 1 -- 172.21.15.32:0/1242330759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef2c10d480 con 0x7fef2c0b7df0 2021-04-09T18:17:25.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.335 7fef1effd700 1 -- 172.21.15.32:0/1242330759 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2781 (secure 0 0 0) 0x7fef24006aa0 con 0x7fef2c0b7df0 2021-04-09T18:17:25.370 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2021-04-09T18:17:25.371 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:25.371 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:25.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 -- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef14018930 msgr2=0x7fef1401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:25.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef14018930 0x7fef1401ae10 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 -- 172.21.15.32:0/1242330759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0b7df0 msgr2=0x7fef2c0b81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:25.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0b7df0 0x7fef2c0b81f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7fef2400b970 tx=0x7fef2400bd10).stop 2021-04-09T18:17:25.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 -- 172.21.15.32:0/1242330759 shutdown_connections 2021-04-09T18:17:25.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef14018930 0x7fef1401ae10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef2c0b7df0 0x7fef2c0b81f0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef2c18df50 0x7fef2c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 --2- 172.21.15.32:0/1242330759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef2c18db50 0x7fef2c19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:25.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 -- 172.21.15.32:0/1242330759 >> 172.21.15.32:0/1242330759 conn(0x7fef2c1a23e0 msgr2=0x7fef2c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:25.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.337 7fef34379700 1 -- 172.21.15.32:0/1242330759 shutdown_connections 2021-04-09T18:17:25.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:25.338 7fef34379700 1 -- 172.21.15.32:0/1242330759 wait complete. 2021-04-09T18:17:25.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:25.839 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:26.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.970 7f019d822700 1 Processor -- start 2021-04-09T18:17:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.970 7f019d822700 1 -- start start 2021-04-09T18:17:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.971 7f019d822700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f019810d010 0x7f01980d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:27.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.971 7f019d822700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f01980d1da0 0x7f01980d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.971 7f019d822700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019810d480 con 0x7f01980d82c0 2021-04-09T18:17:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.971 7f019d822700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01980d8d30 con 0x7f019810d010 2021-04-09T18:17:27.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.971 7f019d822700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01980d9220 con 0x7f01980d1da0 2021-04-09T18:17:27.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.971 7f0197574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f019810d010 0x7f01980d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.971 7f0197d75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f01980d82c0 0x7f019810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55020/0 (socket says 172.21.15.32:55020) 2021-04-09T18:17:27.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.971 7f0197d75700 1 -- 172.21.15.32:0/236749001 learned_addr learned my addr 172.21.15.32:0/236749001 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0196d73700 1 --2- 172.21.15.32:0/236749001 >> v2:172.21.15.86:3300/0 conn(0x7f01980d1da0 0x7f01980d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0196572700 1 -- 172.21.15.32:0/236749001 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698501954 0 0) 0x7f0190000d30 con 0x7f01980d82c0 2021-04-09T18:17:27.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0196572700 1 -- 172.21.15.32:0/236749001 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f017c002fe0 con 0x7f01980d82c0 2021-04-09T18:17:27.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0197574700 1 -- 172.21.15.32:0/236749001 >> v2:172.21.15.86:3300/0 conn(0x7f01980d1da0 msgr2=0x7f01980d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:27.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0197574700 1 --2- 172.21.15.32:0/236749001 >> v2:172.21.15.86:3300/0 conn(0x7f01980d1da0 0x7f01980d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0197574700 1 -- 172.21.15.32:0/236749001 >> v1:172.21.15.32:6789/0 conn(0x7f01980d82c0 legacy=0x7f019810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:27.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0197574700 1 -- 172.21.15.32:0/236749001 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01980d86f0 con 0x7f019810d010 2021-04-09T18:17:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0196d73700 1 --2- 172.21.15.32:0/236749001 >> v2:172.21.15.86:3300/0 conn(0x7f01980d1da0 0x7f01980d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:27.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0197574700 1 --2- 172.21.15.32:0/236749001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019810d010 0x7f01980d1780 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f0188006650 tx=0x7f0188009a80).ready entity=mon.0 client_cookie=16a9bc51686e7cf5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0196572700 1 -- 172.21.15.32:0/236749001 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01880009b0 con 0x7f019810d010 2021-04-09T18:17:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0196572700 1 -- 172.21.15.32:0/236749001 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018800c0c0 con 0x7f019810d010 2021-04-09T18:17:27.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f0196572700 1 -- 172.21.15.32:0/236749001 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01880009b0 con 0x7f019810d010 2021-04-09T18:17:27.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f019d822700 1 -- 172.21.15.32:0/236749001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019810d010 msgr2=0x7f01980d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:27.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.972 7f019d822700 1 --2- 172.21.15.32:0/236749001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019810d010 0x7f01980d1780 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f0188006650 tx=0x7f0188009a80).stop 2021-04-09T18:17:27.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.973 7f019d822700 1 -- 172.21.15.32:0/236749001 shutdown_connections 2021-04-09T18:17:27.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.973 7f019d822700 1 --2- 172.21.15.32:0/236749001 >> v2:172.21.15.86:3300/0 conn(0x7f01980d1da0 0x7f01980d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.973 7f019d822700 1 --2- 172.21.15.32:0/236749001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f019810d010 0x7f01980d1780 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.973 7f019d822700 1 -- 172.21.15.32:0/236749001 >> 172.21.15.32:0/236749001 conn(0x7f01980ce5f0 msgr2=0x7f01980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:27.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.973 7f019d822700 1 -- 172.21.15.32:0/236749001 shutdown_connections 2021-04-09T18:17:27.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.973 7f019d822700 1 -- 172.21.15.32:0/236749001 wait complete. 2021-04-09T18:17:27.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.993 7f019d822700 1 Processor -- start 2021-04-09T18:17:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.993 7f019d822700 1 -- start start 2021-04-09T18:17:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.993 7f019d822700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01980a90c0 0x7f01981a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.993 7f019d822700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01981a76d0 0x7f01981a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.993 7f019d822700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019817bce0 0x7f019817c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.993 7f019d822700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01981a7c60 con 0x7f01981a76d0 2021-04-09T18:17:27.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.993 7f019d822700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01981a8e20 con 0x7f01980a90c0 2021-04-09T18:17:27.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.993 7f019d822700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019817c700 con 0x7f019817bce0 2021-04-09T18:17:27.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0196d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01981a76d0 0x7f01981a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:27.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0196d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01981a76d0 0x7f01981a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45438/0 (socket says 172.21.15.32:45438) 2021-04-09T18:17:27.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0196d73700 1 -- 172.21.15.32:0/814990111 learned_addr learned my addr 172.21.15.32:0/814990111 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:27.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0197574700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01980a90c0 0x7f01981a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:27.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0197d75700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019817bce0 0x7f019817c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:27.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0196d73700 1 -- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019817bce0 msgr2=0x7f019817c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:27.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0196d73700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019817bce0 0x7f019817c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0196d73700 1 -- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01980a90c0 msgr2=0x7f01981a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:27.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0196d73700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01980a90c0 0x7f01981a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0196d73700 1 -- 172.21.15.32:0/814990111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019817c920 con 0x7f01981a76d0 2021-04-09T18:17:27.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0197d75700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019817bce0 0x7f019817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:27.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f0196d73700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01981a76d0 0x7f01981a8800 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f0184006f20 tx=0x7f018400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:27.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f018ffff700 1 -- 172.21.15.32:0/814990111 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f018400bd80 con 0x7f01981a76d0 2021-04-09T18:17:27.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f018ffff700 1 -- 172.21.15.32:0/814990111 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0184000c90 con 0x7f01981a76d0 2021-04-09T18:17:27.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f018ffff700 1 -- 172.21.15.32:0/814990111 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01840022b0 con 0x7f01981a76d0 2021-04-09T18:17:27.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f019d822700 1 -- 172.21.15.32:0/814990111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019817c920 con 0x7f01981a76d0 2021-04-09T18:17:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.994 7f019d822700 1 -- 172.21.15.32:0/814990111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019817c920 con 0x7f01981a76d0 2021-04-09T18:17:27.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.995 7f018ffff700 1 -- 172.21.15.32:0/814990111 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f018400a000 con 0x7f01981a76d0 2021-04-09T18:17:27.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.995 7f018ffff700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0174018930 0x7f017401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:27.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.995 7f018ffff700 1 -- 172.21.15.32:0/814990111 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0184032a30 con 0x7f01981a76d0 2021-04-09T18:17:27.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.995 7f0197574700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0174018930 0x7f017401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:27.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:26.996 7f0197574700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0174018930 0x7f017401ae10 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.015 7f019d822700 1 -- 172.21.15.32:0/814990111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01981b7010 con 0x7f01981a76d0 2021-04-09T18:17:27.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.019 7f018ffff700 1 -- 172.21.15.32:0/814990111 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0184006bb0 con 0x7f01981a76d0 2021-04-09T18:17:27.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.209 7f019d822700 1 -- 172.21.15.32:0/814990111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f019810d480 con 0x7f01981a76d0 2021-04-09T18:17:27.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.210 7f018ffff700 1 -- 172.21.15.32:0/814990111 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2781 (secure 0 0 0) 0x7f0184006bb0 con 0x7f01981a76d0 2021-04-09T18:17:27.245 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2021-04-09T18:17:27.246 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:27.246 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:27.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 -- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0174018930 msgr2=0x7f017401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:27.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0174018930 0x7f017401ae10 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 -- 172.21.15.32:0/814990111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01981a76d0 msgr2=0x7f01981a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:27.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01981a76d0 0x7f01981a8800 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f0184006f20 tx=0x7f018400b990).stop 2021-04-09T18:17:27.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 -- 172.21.15.32:0/814990111 shutdown_connections 2021-04-09T18:17:27.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0174018930 0x7f017401ae10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01981a76d0 0x7f01981a8800 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f019817bce0 0x7f019817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 --2- 172.21.15.32:0/814990111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01980a90c0 0x7f01981a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:27.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 -- 172.21.15.32:0/814990111 >> 172.21.15.32:0/814990111 conn(0x7f01981acae0 msgr2=0x7f019817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:27.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 -- 172.21.15.32:0/814990111 shutdown_connections 2021-04-09T18:17:27.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:27.212 7f019d822700 1 -- 172.21.15.32:0/814990111 wait complete. 2021-04-09T18:17:27.673 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:27.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:28.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.803 7fc327426700 1 Processor -- start 2021-04-09T18:17:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.803 7fc327426700 1 -- start start 2021-04-09T18:17:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.804 7fc327426700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc32010d010 0x7fc3200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:28.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.804 7fc327426700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32010d480 con 0x7fc3200d1da0 2021-04-09T18:17:28.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.804 7fc327426700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32010e1f0 con 0x7fc3200d8490 2021-04-09T18:17:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.804 7fc327426700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3200d8d30 con 0x7fc32010d010 2021-04-09T18:17:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.804 7fc324a41700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc3200d1da0 0x7fc3200d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55036/0 (socket says 172.21.15.32:55036) 2021-04-09T18:17:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.804 7fc324a41700 1 -- 172.21.15.32:0/1463046684 learned_addr learned my addr 172.21.15.32:0/1463046684 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:28.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.804 7fc325242700 1 --2- 172.21.15.32:0/1463046684 >> v2:172.21.15.86:3300/0 conn(0x7fc32010d010 0x7fc3200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:28.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc317fff700 1 -- 172.21.15.32:0/1463046684 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019628486 0 0) 0x7fc3100009b0 con 0x7fc3200d1da0 2021-04-09T18:17:28.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc317fff700 1 -- 172.21.15.32:0/1463046684 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc30c002fe0 con 0x7fc3200d1da0 2021-04-09T18:17:28.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc317fff700 1 -- 172.21.15.32:0/1463046684 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918839773 0 0) 0x7fc318000d30 con 0x7fc3200d8490 2021-04-09T18:17:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc317fff700 1 -- 172.21.15.32:0/1463046684 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc30c0042c0 con 0x7fc3200d8490 2021-04-09T18:17:28.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc325242700 1 -- 172.21.15.32:0/1463046684 >> v1:172.21.15.86:6789/0 conn(0x7fc3200d8490 legacy=0x7fc32010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:28.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc325242700 1 -- 172.21.15.32:0/1463046684 >> v1:172.21.15.32:6789/0 conn(0x7fc3200d1da0 legacy=0x7fc3200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:28.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc325242700 1 -- 172.21.15.32:0/1463046684 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3200d9220 con 0x7fc32010d010 2021-04-09T18:17:28.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc325242700 1 --2- 172.21.15.32:0/1463046684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32010d010 0x7fc3200d1780 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fc31c006650 tx=0x7fc31c009a80).ready entity=mon.1 client_cookie=e226802412b73084 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:28.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc317fff700 1 -- 172.21.15.32:0/1463046684 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc31c0009b0 con 0x7fc32010d010 2021-04-09T18:17:28.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc317fff700 1 -- 172.21.15.32:0/1463046684 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31c00c0c0 con 0x7fc32010d010 2021-04-09T18:17:28.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc317fff700 1 -- 172.21.15.32:0/1463046684 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc31c00c510 con 0x7fc32010d010 2021-04-09T18:17:28.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc327426700 1 -- 172.21.15.32:0/1463046684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32010d010 msgr2=0x7fc3200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:28.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.805 7fc327426700 1 --2- 172.21.15.32:0/1463046684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32010d010 0x7fc3200d1780 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fc31c006650 tx=0x7fc31c009a80).stop 2021-04-09T18:17:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.806 7fc327426700 1 -- 172.21.15.32:0/1463046684 shutdown_connections 2021-04-09T18:17:28.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.806 7fc327426700 1 --2- 172.21.15.32:0/1463046684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32010d010 0x7fc3200d1780 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:28.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.806 7fc327426700 1 -- 172.21.15.32:0/1463046684 >> 172.21.15.32:0/1463046684 conn(0x7fc3200ce5f0 msgr2=0x7fc3200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:28.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.806 7fc327426700 1 -- 172.21.15.32:0/1463046684 shutdown_connections 2021-04-09T18:17:28.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.806 7fc327426700 1 -- 172.21.15.32:0/1463046684 wait complete. 2021-04-09T18:17:28.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.826 7fc327426700 1 Processor -- start 2021-04-09T18:17:28.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.826 7fc327426700 1 -- start start 2021-04-09T18:17:28.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc327426700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3201814c0 0x7fc3200aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:28.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc327426700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3201b8540 0x7fc3201b8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:28.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc327426700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32019b2f0 0x7fc32019b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:28.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc327426700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32008dcd0 con 0x7fc3201814c0 2021-04-09T18:17:28.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc327426700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3200ab310 con 0x7fc3201b8540 2021-04-09T18:17:28.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc327426700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3200ab4f0 con 0x7fc32019b2f0 2021-04-09T18:17:28.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325242700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3201814c0 0x7fc3200aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:28.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325242700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3201814c0 0x7fc3200aacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45448/0 (socket says 172.21.15.32:45448) 2021-04-09T18:17:28.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325242700 1 -- 172.21.15.32:0/1537845540 learned_addr learned my addr 172.21.15.32:0/1537845540 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:28.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc324a41700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3201b8540 0x7fc3201b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:28.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325a43700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32019b2f0 0x7fc32019b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:28.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325242700 1 -- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32019b2f0 msgr2=0x7fc32019b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:28.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325242700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32019b2f0 0x7fc32019b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:28.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325242700 1 -- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3201b8540 msgr2=0x7fc3201b8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:28.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325242700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3201b8540 0x7fc3201b8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:28.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325242700 1 -- 172.21.15.32:0/1537845540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32008cca0 con 0x7fc3201814c0 2021-04-09T18:17:28.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.827 7fc325242700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3201814c0 0x7fc3200aacf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fc31c009650 tx=0x7fc31c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:28.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.828 7fc315ffb700 1 -- 172.21.15.32:0/1537845540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc31c0093a0 con 0x7fc3201814c0 2021-04-09T18:17:28.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.828 7fc315ffb700 1 -- 172.21.15.32:0/1537845540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31c00bd50 con 0x7fc3201814c0 2021-04-09T18:17:28.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.828 7fc327426700 1 -- 172.21.15.32:0/1537845540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32008cca0 con 0x7fc3201814c0 2021-04-09T18:17:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.828 7fc315ffb700 1 -- 172.21.15.32:0/1537845540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc31c0093a0 con 0x7fc3201814c0 2021-04-09T18:17:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.828 7fc327426700 1 -- 172.21.15.32:0/1537845540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32008cca0 con 0x7fc3201814c0 2021-04-09T18:17:28.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.828 7fc315ffb700 1 -- 172.21.15.32:0/1537845540 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc31c006030 con 0x7fc3201814c0 2021-04-09T18:17:28.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.828 7fc315ffb700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc300018930 0x7fc30001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:28.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.828 7fc315ffb700 1 -- 172.21.15.32:0/1537845540 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc31c032a10 con 0x7fc3201814c0 2021-04-09T18:17:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.829 7fc324a41700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc300018930 0x7fc30001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.829 7fc324a41700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc300018930 0x7fc30001ae10 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.848 7fc327426700 1 -- 172.21.15.32:0/1537845540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3201b7050 con 0x7fc3201814c0 2021-04-09T18:17:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:28.852 7fc315ffb700 1 -- 172.21.15.32:0/1537845540 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc31c009360 con 0x7fc3201814c0 2021-04-09T18:17:29.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.043 7fc327426700 1 -- 172.21.15.32:0/1537845540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3201b7050 con 0x7fc3201814c0 2021-04-09T18:17:29.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.044 7fc315ffb700 1 -- 172.21.15.32:0/1537845540 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2781 (secure 0 0 0) 0x7fc31c00bd50 con 0x7fc3201814c0 2021-04-09T18:17:29.093 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 64 2021-04-09T18:17:29.110 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:29.111 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:10.398416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[0],"up":{"mds_0":5803},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5803":{"gid":5803,"name":"d","rank":0,"incarnation":53,"state":"up:stopping","state_seq":3,"addr":"172.21.15.86:6837/3241800545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3241800545},{"type":"v1","addr":"172.21.15.86:6837","nonce":3241800545}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:29.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 -- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc300018930 msgr2=0x7fc30001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc300018930 0x7fc30001ae10 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 -- 172.21.15.32:0/1537845540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3201814c0 msgr2=0x7fc3200aacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3201814c0 0x7fc3200aacf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fc31c009650 tx=0x7fc31c00b960).stop 2021-04-09T18:17:29.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 -- 172.21.15.32:0/1537845540 shutdown_connections 2021-04-09T18:17:29.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc300018930 0x7fc30001ae10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:29.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32019b2f0 0x7fc32019b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:29.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3201b8540 0x7fc3201b8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:29.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 --2- 172.21.15.32:0/1537845540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3201814c0 0x7fc3200aacf0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:29.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.046 7fc327426700 1 -- 172.21.15.32:0/1537845540 >> 172.21.15.32:0/1537845540 conn(0x7fc320190570 msgr2=0x7fc32018ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:29.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.047 7fc327426700 1 -- 172.21.15.32:0/1537845540 shutdown_connections 2021-04-09T18:17:29.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:29.047 7fc327426700 1 -- 172.21.15.32:0/1537845540 wait complete. 2021-04-09T18:17:29.578 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:10.398416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 5803}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5803': {'gid': 5803, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.86:6837/3241800545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 3241800545}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3241800545}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:29.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5803:up:stopping 2021-04-09T18:17:30.491 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:17:30.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:30.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.707 7fd4d4121700 1 Processor -- start 2021-04-09T18:17:30.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.708 7fd4d4121700 1 -- start start 2021-04-09T18:17:30.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.708 7fd4d4121700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc0cffa0 0x7fd4cc0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.708 7fd4d4121700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd4cc0d7800 0x7fd4cc0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.708 7fd4d4121700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd4cc0d5c70 0x7fd4cc0d6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.708 7fd4d4121700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4cc0d8580 con 0x7fd4cc0d5c70 2021-04-09T18:17:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.708 7fd4d4121700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4cc0d91e0 con 0x7fd4cc0d7800 2021-04-09T18:17:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.708 7fd4d4121700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc0d7280 con 0x7fd4cc0cffa0 2021-04-09T18:17:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.708 7fd4d273e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd4cc0d5c70 0x7fd4cc0d6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d273e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd4cc0d5c70 0x7fd4cc0d6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45460/0 (socket says 172.21.15.32:45460) 2021-04-09T18:17:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d273e700 1 -- 172.21.15.32:0/3435625384 learned_addr learned my addr 172.21.15.32:0/3435625384 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:30.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d1f3d700 1 --2- 172.21.15.32:0/3435625384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc0cffa0 0x7fd4cc0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d173c700 1 --2- 172.21.15.32:0/3435625384 >> v2:172.21.15.86:3300/0 conn(0x7fd4cc0d7800 0x7fd4cc0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d273e700 1 -- 172.21.15.32:0/3435625384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc0cffa0 msgr2=0x7fd4cc0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d273e700 1 --2- 172.21.15.32:0/3435625384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc0cffa0 0x7fd4cc0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d273e700 1 -- 172.21.15.32:0/3435625384 >> v2:172.21.15.86:3300/0 conn(0x7fd4cc0d7800 msgr2=0x7fd4cc0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d273e700 1 --2- 172.21.15.32:0/3435625384 >> v2:172.21.15.86:3300/0 conn(0x7fd4cc0d7800 0x7fd4cc0d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d273e700 1 -- 172.21.15.32:0/3435625384 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc0d8220 con 0x7fd4cc0d5c70 2021-04-09T18:17:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d1f3d700 1 --2- 172.21.15.32:0/3435625384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc0cffa0 0x7fd4cc0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d273e700 1 --2- 172.21.15.32:0/3435625384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc0d5c70 0x7fd4cc0d6070 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fd4c4006f00 tx=0x7fd4c4009f10).ready entity=mon.0 client_cookie=d2942a32306f30f1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:30.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d0f3b700 1 -- 172.21.15.32:0/3435625384 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4c400b3b0 con 0x7fd4cc0d5c70 2021-04-09T18:17:30.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d0f3b700 1 -- 172.21.15.32:0/3435625384 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c400b730 con 0x7fd4cc0d5c70 2021-04-09T18:17:30.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d0f3b700 1 -- 172.21.15.32:0/3435625384 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4c400bc10 con 0x7fd4cc0d5c70 2021-04-09T18:17:30.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d4121700 1 -- 172.21.15.32:0/3435625384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc0d5c70 msgr2=0x7fd4cc0d6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:30.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.709 7fd4d4121700 1 --2- 172.21.15.32:0/3435625384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc0d5c70 0x7fd4cc0d6070 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fd4c4006f00 tx=0x7fd4c4009f10).stop 2021-04-09T18:17:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.710 7fd4d4121700 1 -- 172.21.15.32:0/3435625384 shutdown_connections 2021-04-09T18:17:30.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.710 7fd4d4121700 1 --2- 172.21.15.32:0/3435625384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc0d5c70 0x7fd4cc0d6070 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.710 7fd4d4121700 1 --2- 172.21.15.32:0/3435625384 >> v2:172.21.15.86:3300/0 conn(0x7fd4cc0d7800 0x7fd4cc0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.710 7fd4d4121700 1 --2- 172.21.15.32:0/3435625384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc0cffa0 0x7fd4cc0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.710 7fd4d4121700 1 -- 172.21.15.32:0/3435625384 >> 172.21.15.32:0/3435625384 conn(0x7fd4cc0ce5c0 msgr2=0x7fd4cc0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.710 7fd4d4121700 1 -- 172.21.15.32:0/3435625384 shutdown_connections 2021-04-09T18:17:30.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.710 7fd4d4121700 1 -- 172.21.15.32:0/3435625384 wait complete. 2021-04-09T18:17:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.730 7fd4d4121700 1 Processor -- start 2021-04-09T18:17:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.730 7fd4d4121700 1 -- start start 2021-04-09T18:17:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d4121700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc08ec80 0x7fd4cc08f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:30.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d4121700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4cc08c950 0x7fd4cc08cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d4121700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc08d560 0x7fd4cc08d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d4121700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc08e700 con 0x7fd4cc08d560 2021-04-09T18:17:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d4121700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc08f6a0 con 0x7fd4cc08c950 2021-04-09T18:17:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d4121700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc1acff0 con 0x7fd4cc08ec80 2021-04-09T18:17:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d273e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc08d560 0x7fd4cc08d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d273e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc08d560 0x7fd4cc08d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45464/0 (socket says 172.21.15.32:45464) 2021-04-09T18:17:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d273e700 1 -- 172.21.15.32:0/4095457930 learned_addr learned my addr 172.21.15.32:0/4095457930 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d1f3d700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc08ec80 0x7fd4cc08f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d173c700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4cc08c950 0x7fd4cc08cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d273e700 1 -- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc08ec80 msgr2=0x7fd4cc08f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d273e700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc08ec80 0x7fd4cc08f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d273e700 1 -- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4cc08c950 msgr2=0x7fd4cc08cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d273e700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4cc08c950 0x7fd4cc08cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d273e700 1 -- 172.21.15.32:0/4095457930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc08df80 con 0x7fd4cc08d560 2021-04-09T18:17:30.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.731 7fd4d273e700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc08d560 0x7fd4cc08d960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7fd4c40099f0 tx=0x7fd4c400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:30.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.732 7fd4c27fc700 1 -- 172.21.15.32:0/4095457930 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4c4000f90 con 0x7fd4cc08d560 2021-04-09T18:17:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.732 7fd4c27fc700 1 -- 172.21.15.32:0/4095457930 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c400be10 con 0x7fd4cc08d560 2021-04-09T18:17:30.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.732 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4cc08df80 con 0x7fd4cc08d560 2021-04-09T18:17:30.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.732 7fd4c27fc700 1 -- 172.21.15.32:0/4095457930 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4c400c2f0 con 0x7fd4cc08d560 2021-04-09T18:17:30.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.732 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4cc08df80 con 0x7fd4cc08d560 2021-04-09T18:17:30.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.732 7fd4c27fc700 1 -- 172.21.15.32:0/4095457930 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd4c4001270 con 0x7fd4cc08d560 2021-04-09T18:17:30.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.732 7fd4c27fc700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4b4018930 0x7fd4b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.732 7fd4c27fc700 1 -- 172.21.15.32:0/4095457930 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd4c4032d60 con 0x7fd4cc08d560 2021-04-09T18:17:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.733 7fd4d1f3d700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4b4018930 0x7fd4b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.733 7fd4d1f3d700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4b4018930 0x7fd4b401ae10 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.753 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4cc182dc0 con 0x7fd4cc08d560 2021-04-09T18:17:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.756 7fd4c27fc700 1 -- 172.21.15.32:0/4095457930 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd4c4000f90 con 0x7fd4cc08d560 2021-04-09T18:17:30.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.947 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4cc0d7280 con 0x7fd4cc08d560 2021-04-09T18:17:30.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.947 7fd4c27fc700 1 -- 172.21.15.32:0/4095457930 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2433 (secure 0 0 0) 0x7fd4c400be10 con 0x7fd4cc08d560 2021-04-09T18:17:30.981 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 65 2021-04-09T18:17:30.982 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:30.982 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":65,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:30.396349","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0,1],"info":{},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:30.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4b4018930 msgr2=0x7fd4b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:30.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4b4018930 0x7fd4b401ae10 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc08d560 msgr2=0x7fd4cc08d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:30.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc08d560 0x7fd4cc08d960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7fd4c40099f0 tx=0x7fd4c400bcd0).stop 2021-04-09T18:17:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 shutdown_connections 2021-04-09T18:17:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4b4018930 0x7fd4b401ae10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc08d560 0x7fd4cc08d960 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4cc08c950 0x7fd4cc08cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 --2- 172.21.15.32:0/4095457930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4cc08ec80 0x7fd4cc08f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 >> 172.21.15.32:0/4095457930 conn(0x7fd4cc09abb0 msgr2=0x7fd4cc0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 shutdown_connections 2021-04-09T18:17:30.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:30.950 7fd4d4121700 1 -- 172.21.15.32:0/4095457930 wait complete. 2021-04-09T18:17:31.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:30.396349', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0, 1], 'info': {}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:31.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2021-04-09T18:17:31.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down false 2021-04-09T18:17:31.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.523 7f6b78d84700 1 Processor -- start 2021-04-09T18:17:31.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.523 7f6b78d84700 1 -- start start 2021-04-09T18:17:31.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b78d84700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b7410d010 0x7f6b740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b78d84700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6b740d1da0 0x7f6b740d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:31.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b78d84700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740d82c0 0x7f6b7410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b78d84700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b740d8d30 con 0x7f6b7410d010 2021-04-09T18:17:31.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b78d84700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b740d9220 con 0x7f6b740d1da0 2021-04-09T18:17:31.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b78d84700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7410d4b0 con 0x7f6b740d82c0 2021-04-09T18:17:31.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b72b13700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b7410d010 0x7f6b740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b72b13700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6b7410d010 0x7f6b740d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45470/0 (socket says 172.21.15.32:45470) 2021-04-09T18:17:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b72b13700 1 -- 172.21.15.32:0/597914638 learned_addr learned my addr 172.21.15.32:0/597914638 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:31.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b72312700 1 --2- 172.21.15.32:0/597914638 >> v2:172.21.15.86:3300/0 conn(0x7f6b740d1da0 0x7f6b740d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:31.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.524 7f6b73314700 1 --2- 172.21.15.32:0/597914638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740d82c0 0x7f6b7410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:31.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b72b13700 1 -- 172.21.15.32:0/597914638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740d82c0 msgr2=0x7f6b7410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:31.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b72b13700 1 --2- 172.21.15.32:0/597914638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740d82c0 0x7f6b7410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:31.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b72b13700 1 -- 172.21.15.32:0/597914638 >> v2:172.21.15.86:3300/0 conn(0x7f6b740d1da0 msgr2=0x7f6b740d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:31.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b72b13700 1 --2- 172.21.15.32:0/597914638 >> v2:172.21.15.86:3300/0 conn(0x7f6b740d1da0 0x7f6b740d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:31.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b72b13700 1 -- 172.21.15.32:0/597914638 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b740d86c0 con 0x7f6b7410d010 2021-04-09T18:17:31.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b72b13700 1 --2- 172.21.15.32:0/597914638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7410d010 0x7f6b740d1780 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f6b64006a80 tx=0x7f6b64009f00).ready entity=mon.0 client_cookie=2fd6a3a8a9cb3c93 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:31.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b71b11700 1 -- 172.21.15.32:0/597914638 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b6400b5c0 con 0x7f6b7410d010 2021-04-09T18:17:31.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b71b11700 1 -- 172.21.15.32:0/597914638 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6400b8c0 con 0x7f6b7410d010 2021-04-09T18:17:31.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b71b11700 1 -- 172.21.15.32:0/597914638 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b6400bde0 con 0x7f6b7410d010 2021-04-09T18:17:31.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b78d84700 1 -- 172.21.15.32:0/597914638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7410d010 msgr2=0x7f6b740d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:31.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.525 7f6b78d84700 1 --2- 172.21.15.32:0/597914638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7410d010 0x7f6b740d1780 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f6b64006a80 tx=0x7f6b64009f00).stop 2021-04-09T18:17:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.526 7f6b78d84700 1 -- 172.21.15.32:0/597914638 shutdown_connections 2021-04-09T18:17:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.526 7f6b78d84700 1 --2- 172.21.15.32:0/597914638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740d82c0 0x7f6b7410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:31.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.526 7f6b78d84700 1 --2- 172.21.15.32:0/597914638 >> v2:172.21.15.86:3300/0 conn(0x7f6b740d1da0 0x7f6b740d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:31.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.526 7f6b78d84700 1 --2- 172.21.15.32:0/597914638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b7410d010 0x7f6b740d1780 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:31.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.526 7f6b78d84700 1 -- 172.21.15.32:0/597914638 >> 172.21.15.32:0/597914638 conn(0x7f6b740ce5f0 msgr2=0x7f6b740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:31.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.526 7f6b78d84700 1 -- 172.21.15.32:0/597914638 shutdown_connections 2021-04-09T18:17:31.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.526 7f6b78d84700 1 -- 172.21.15.32:0/597914638 wait complete. 2021-04-09T18:17:31.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.546 7f6b78d84700 1 Processor -- start 2021-04-09T18:17:31.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.546 7f6b78d84700 1 -- start start 2021-04-09T18:17:31.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b78d84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b740b1680 0x7f6b740b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:31.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b78d84700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b7409e330 0x7f6b740b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:31.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b78d84700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740b2190 0x7f6b7409f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:31.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b78d84700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7409e730 con 0x7f6b740b1680 2021-04-09T18:17:31.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b78d84700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7409f6c0 con 0x7f6b7409e330 2021-04-09T18:17:31.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b78d84700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7409f8e0 con 0x7f6b740b2190 2021-04-09T18:17:31.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b740b1680 0x7f6b740b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:31.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b740b1680 0x7f6b740b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45476/0 (socket says 172.21.15.32:45476) 2021-04-09T18:17:31.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72b13700 1 -- 172.21.15.32:0/2363632352 learned_addr learned my addr 172.21.15.32:0/2363632352 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72312700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b7409e330 0x7f6b740b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:31.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b73314700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740b2190 0x7f6b7409f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:31.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72312700 1 -- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740b2190 msgr2=0x7f6b7409f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72312700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740b2190 0x7f6b7409f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:31.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72312700 1 -- 172.21.15.32:0/2363632352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b740b1680 msgr2=0x7f6b740b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:31.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72312700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b740b1680 0x7f6b740b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72312700 1 -- 172.21.15.32:0/2363632352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7409fb00 con 0x7f6b7409e330 2021-04-09T18:17:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72b13700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b740b1680 0x7f6b740b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:31.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.547 7f6b72312700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b7409e330 0x7f6b740b1b70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f6b5c009540 tx=0x7f6b5c00b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:31.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.548 7f6b6b7fe700 1 -- 172.21.15.32:0/2363632352 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b5c00bcd0 con 0x7f6b7409e330 2021-04-09T18:17:31.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.548 7f6b6b7fe700 1 -- 172.21.15.32:0/2363632352 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c006740 con 0x7f6b7409e330 2021-04-09T18:17:31.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.548 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7409fb00 con 0x7f6b7409e330 2021-04-09T18:17:31.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.548 7f6b6b7fe700 1 -- 172.21.15.32:0/2363632352 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b5c002030 con 0x7f6b7409e330 2021-04-09T18:17:31.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.548 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7409fb00 con 0x7f6b7409e330 2021-04-09T18:17:31.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.548 7f6b6b7fe700 1 -- 172.21.15.32:0/2363632352 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6b5c0052e0 con 0x7f6b7409e330 2021-04-09T18:17:31.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.548 7f6b6b7fe700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6b58018930 0x7f6b5801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:31.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.548 7f6b6b7fe700 1 -- 172.21.15.32:0/2363632352 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6b5c032930 con 0x7f6b7409e330 2021-04-09T18:17:31.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.548 7f6b72b13700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6b58018930 0x7f6b5801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:31.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.549 7f6b72b13700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6b58018930 0x7f6b5801adc0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.568 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7410d480 con 0x7f6b7409e330 2021-04-09T18:17:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.572 7f6b6b7fe700 1 -- 172.21.15.32:0/2363632352 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6b5c006ae0 con 0x7f6b7409e330 2021-04-09T18:17:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:31.768 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7f6b7410d480 con 0x7f6b7409e330 2021-04-09T18:17:32.436 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:17:32.402 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:17:32.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.407 7f6b6b7fe700 1 -- 172.21.15.32:0/2363632352 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"var": "down", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 cephfs marked up, max_mds = 2 v66) v1 ==== 135+0+0 (secure 0 0 0) 0x7f6b5c0052e0 con 0x7f6b7409e330 2021-04-09T18:17:32.467 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked up, max_mds = 2 2021-04-09T18:17:32.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6b58018930 msgr2=0x7f6b5801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:32.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6b58018930 0x7f6b5801adc0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b7409e330 msgr2=0x7f6b740b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:32.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b7409e330 0x7f6b740b1b70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f6b5c009540 tx=0x7f6b5c00b8e0).stop 2021-04-09T18:17:32.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 shutdown_connections 2021-04-09T18:17:32.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6b58018930 0x7f6b5801adc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b740b2190 0x7f6b7409f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b7409e330 0x7f6b740b1b70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 --2- 172.21.15.32:0/2363632352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b740b1680 0x7f6b740b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 >> 172.21.15.32:0/2363632352 conn(0x7f6b740c4290 msgr2=0x7f6b740c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:32.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.436 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 shutdown_connections 2021-04-09T18:17:32.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.437 7f6b78d84700 1 -- 172.21.15.32:0/2363632352 wait complete. 2021-04-09T18:17:32.497 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:32.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.645 7fec19fcc700 1 Processor -- start 2021-04-09T18:17:32.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.645 7fec19fcc700 1 -- start start 2021-04-09T18:17:32.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.645 7fec19fcc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec1410d010 0x7fec140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:32.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.645 7fec19fcc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec140d8490 0x7fec1410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.645 7fec19fcc700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1410d480 con 0x7fec140d1da0 2021-04-09T18:17:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.645 7fec19fcc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec140d8d30 con 0x7fec140d8490 2021-04-09T18:17:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.645 7fec19fcc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec140d9220 con 0x7fec1410d010 2021-04-09T18:17:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.645 7fec13fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec140d8490 0x7fec1410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:32.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec13fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fec140d8490 0x7fec1410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45494/0 (socket says 172.21.15.32:45494) 2021-04-09T18:17:32.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec13fff700 1 -- 172.21.15.32:0/97197397 learned_addr learned my addr 172.21.15.32:0/97197397 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec137fe700 1 --2- 172.21.15.32:0/97197397 >> v2:172.21.15.86:3300/0 conn(0x7fec1410d010 0x7fec140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec13fff700 1 -- 172.21.15.32:0/97197397 >> v2:172.21.15.86:3300/0 conn(0x7fec1410d010 msgr2=0x7fec140d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec13fff700 1 --2- 172.21.15.32:0/97197397 >> v2:172.21.15.86:3300/0 conn(0x7fec1410d010 0x7fec140d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec13fff700 1 -- 172.21.15.32:0/97197397 >> v1:172.21.15.86:6789/0 conn(0x7fec140d1da0 legacy=0x7fec140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec13fff700 1 -- 172.21.15.32:0/97197397 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1410e020 con 0x7fec140d8490 2021-04-09T18:17:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec137fe700 1 --2- 172.21.15.32:0/97197397 >> v2:172.21.15.86:3300/0 conn(0x7fec1410d010 0x7fec140d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:32.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec13fff700 1 --2- 172.21.15.32:0/97197397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec140d8490 0x7fec1410da00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fec0c006f00 tx=0x7fec0c009f10).ready entity=mon.0 client_cookie=f591ecbfa8583be2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec127fc700 1 -- 172.21.15.32:0/97197397 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec0c00b3b0 con 0x7fec140d8490 2021-04-09T18:17:32.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec127fc700 1 -- 172.21.15.32:0/97197397 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c00b730 con 0x7fec140d8490 2021-04-09T18:17:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec127fc700 1 -- 172.21.15.32:0/97197397 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec0c00bc10 con 0x7fec140d8490 2021-04-09T18:17:32.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec19fcc700 1 -- 172.21.15.32:0/97197397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec140d8490 msgr2=0x7fec1410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:32.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.646 7fec19fcc700 1 --2- 172.21.15.32:0/97197397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec140d8490 0x7fec1410da00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fec0c006f00 tx=0x7fec0c009f10).stop 2021-04-09T18:17:32.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.647 7fec19fcc700 1 -- 172.21.15.32:0/97197397 shutdown_connections 2021-04-09T18:17:32.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.647 7fec19fcc700 1 --2- 172.21.15.32:0/97197397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec140d8490 0x7fec1410da00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.647 7fec19fcc700 1 --2- 172.21.15.32:0/97197397 >> v2:172.21.15.86:3300/0 conn(0x7fec1410d010 0x7fec140d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.647 7fec19fcc700 1 -- 172.21.15.32:0/97197397 >> 172.21.15.32:0/97197397 conn(0x7fec140ce5f0 msgr2=0x7fec140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:32.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.647 7fec19fcc700 1 -- 172.21.15.32:0/97197397 shutdown_connections 2021-04-09T18:17:32.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.647 7fec19fcc700 1 -- 172.21.15.32:0/97197397 wait complete. 2021-04-09T18:17:32.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.667 7fec19fcc700 1 Processor -- start 2021-04-09T18:17:32.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.667 7fec19fcc700 1 -- start start 2021-04-09T18:17:32.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec19fcc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec1408f040 0x7fec141a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:32.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec19fcc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec1408f440 0x7fec141a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:32.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec19fcc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec141a22e0 0x7fec141a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:32.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec19fcc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141a3100 con 0x7fec141a22e0 2021-04-09T18:17:32.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec19fcc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141a2d00 con 0x7fec1408f440 2021-04-09T18:17:32.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec19fcc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1417c7e0 con 0x7fec1408f040 2021-04-09T18:17:32.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec13fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec141a22e0 0x7fec141a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:32.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec13fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec141a22e0 0x7fec141a26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45498/0 (socket says 172.21.15.32:45498) 2021-04-09T18:17:32.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec13fff700 1 -- 172.21.15.32:0/1928932192 learned_addr learned my addr 172.21.15.32:0/1928932192 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:32.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec12ffd700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec1408f440 0x7fec141a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:32.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec137fe700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec1408f040 0x7fec141a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:32.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec13fff700 1 -- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec1408f040 msgr2=0x7fec141a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:32.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec13fff700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec1408f040 0x7fec141a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec13fff700 1 -- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec1408f440 msgr2=0x7fec141a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:32.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec13fff700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec1408f440 0x7fec141a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec13fff700 1 -- 172.21.15.32:0/1928932192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140aee10 con 0x7fec141a22e0 2021-04-09T18:17:32.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.668 7fec13fff700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec141a22e0 0x7fec141a26e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fec0c009390 tx=0x7fec0c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:32.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.669 7fec0bfff700 1 -- 172.21.15.32:0/1928932192 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec0c003030 con 0x7fec141a22e0 2021-04-09T18:17:32.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.669 7fec0bfff700 1 -- 172.21.15.32:0/1928932192 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c0068e0 con 0x7fec141a22e0 2021-04-09T18:17:32.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.669 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec140aee10 con 0x7fec141a22e0 2021-04-09T18:17:32.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.669 7fec0bfff700 1 -- 172.21.15.32:0/1928932192 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec0c00c570 con 0x7fec141a22e0 2021-04-09T18:17:32.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.669 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec140aee10 con 0x7fec141a22e0 2021-04-09T18:17:32.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.669 7fec0bfff700 1 -- 172.21.15.32:0/1928932192 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fec0c005820 con 0x7fec141a22e0 2021-04-09T18:17:32.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.669 7fec0bfff700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febf0018930 0x7febf001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:32.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.669 7fec0bfff700 1 -- 172.21.15.32:0/1928932192 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fec0c032ee0 con 0x7fec141a22e0 2021-04-09T18:17:32.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.670 7fec137fe700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febf0018930 0x7febf001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:32.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.670 7fec137fe700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febf0018930 0x7febf001ae10 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:32.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.689 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec141b7050 con 0x7fec141a22e0 2021-04-09T18:17:32.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.693 7fec0bfff700 1 -- 172.21.15.32:0/1928932192 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fec0c006c80 con 0x7fec141a22e0 2021-04-09T18:17:32.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.883 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec141b7050 con 0x7fec141a22e0 2021-04-09T18:17:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.883 7fec0bfff700 1 -- 172.21.15.32:0/1928932192 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2444 (secure 0 0 0) 0x7fec0c0068e0 con 0x7fec141a22e0 2021-04-09T18:17:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2021-04-09T18:17:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:32.924 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:32.408652","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":5883},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5883":{"gid":5883,"name":"a","rank":0,"incarnation":67,"state":"up:starting","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.885 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febf0018930 msgr2=0x7febf001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:32.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.885 7fec19fcc700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febf0018930 0x7febf001ae10 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.885 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec141a22e0 msgr2=0x7fec141a26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.885 7fec19fcc700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec141a22e0 0x7fec141a26e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fec0c009390 tx=0x7fec0c00bd40).stop 2021-04-09T18:17:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.886 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 shutdown_connections 2021-04-09T18:17:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.886 7fec19fcc700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febf0018930 0x7febf001ae10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.886 7fec19fcc700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec141a22e0 0x7fec141a26e0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.886 7fec19fcc700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec1408f440 0x7fec141a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.886 7fec19fcc700 1 --2- 172.21.15.32:0/1928932192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec1408f040 0x7fec141a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.886 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 >> 172.21.15.32:0/1928932192 conn(0x7fec14199680 msgr2=0x7fec141a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.886 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 shutdown_connections 2021-04-09T18:17:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:32.886 7fec19fcc700 1 -- 172.21.15.32:0/1928932192 wait complete. 2021-04-09T18:17:32.948 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.088 7f5686918700 1 Processor -- start 2021-04-09T18:17:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.088 7f5686918700 1 -- start start 2021-04-09T18:17:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.088 7f5686918700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56800d7ca0 0x7f56800d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5686918700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568010d480 con 0x7f568010da00 2021-04-09T18:17:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5686918700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56800d86c0 con 0x7f568010d010 2021-04-09T18:17:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5686918700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56800d9220 con 0x7f56800d7ca0 2021-04-09T18:17:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5684f35700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f568010da00 0x7f568010de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55096/0 (socket says 172.21.15.32:55096) 2021-04-09T18:17:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5684f35700 1 -- 172.21.15.32:0/2360538781 learned_addr learned my addr 172.21.15.32:0/2360538781 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5677fff700 1 --2- 172.21.15.32:0/2360538781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56800d7ca0 0x7f56800d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f567f7fe700 1 -- 172.21.15.32:0/2360538781 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546515438 0 0) 0x7f5678000d30 con 0x7f568010da00 2021-04-09T18:17:33.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f567f7fe700 1 -- 172.21.15.32:0/2360538781 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566c002fe0 con 0x7f568010da00 2021-04-09T18:17:33.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f567f7fe700 1 -- 172.21.15.32:0/2360538781 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514386471 0 0) 0x7f5670000c60 con 0x7f568010d010 2021-04-09T18:17:33.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f567f7fe700 1 -- 172.21.15.32:0/2360538781 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566c0042c0 con 0x7f568010d010 2021-04-09T18:17:33.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5677fff700 1 -- 172.21.15.32:0/2360538781 >> v1:172.21.15.86:6789/0 conn(0x7f568010d010 legacy=0x7f56800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:33.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5677fff700 1 -- 172.21.15.32:0/2360538781 >> v1:172.21.15.32:6789/0 conn(0x7f568010da00 legacy=0x7f568010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:33.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5677fff700 1 -- 172.21.15.32:0/2360538781 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5680112100 con 0x7f56800d7ca0 2021-04-09T18:17:33.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5677fff700 1 --2- 172.21.15.32:0/2360538781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56800d7ca0 0x7f56800d80a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f5668006650 tx=0x7f5668009a80).ready entity=mon.2 client_cookie=5dd2db428527cf8f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:33.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f567f7fe700 1 -- 172.21.15.32:0/2360538781 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56680009b0 con 0x7f56800d7ca0 2021-04-09T18:17:33.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f567f7fe700 1 -- 172.21.15.32:0/2360538781 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566800c0c0 con 0x7f56800d7ca0 2021-04-09T18:17:33.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f567f7fe700 1 -- 172.21.15.32:0/2360538781 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f566800c510 con 0x7f56800d7ca0 2021-04-09T18:17:33.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5686918700 1 -- 172.21.15.32:0/2360538781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56800d7ca0 msgr2=0x7f56800d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:33.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.089 7f5686918700 1 --2- 172.21.15.32:0/2360538781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56800d7ca0 0x7f56800d80a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f5668006650 tx=0x7f5668009a80).stop 2021-04-09T18:17:33.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.090 7f5686918700 1 -- 172.21.15.32:0/2360538781 shutdown_connections 2021-04-09T18:17:33.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.090 7f5686918700 1 --2- 172.21.15.32:0/2360538781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56800d7ca0 0x7f56800d80a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.090 7f5686918700 1 -- 172.21.15.32:0/2360538781 >> 172.21.15.32:0/2360538781 conn(0x7f56800ce5f0 msgr2=0x7f56800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:33.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.090 7f5686918700 1 -- 172.21.15.32:0/2360538781 shutdown_connections 2021-04-09T18:17:33.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.090 7f5686918700 1 -- 172.21.15.32:0/2360538781 wait complete. 2021-04-09T18:17:33.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.110 7f5686918700 1 Processor -- start 2021-04-09T18:17:33.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.110 7f5686918700 1 -- start start 2021-04-09T18:17:33.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5686918700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680179610 0x7f5680179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:33.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5686918700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f568017a6e0 0x7f568017aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:33.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5686918700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f568017b100 0x7f568008db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:33.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5686918700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568017d720 con 0x7f568017b100 2021-04-09T18:17:33.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5686918700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5680185ba0 con 0x7f5680179610 2021-04-09T18:17:33.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5686918700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5680185d50 con 0x7f568017a6e0 2021-04-09T18:17:33.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5684f35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f568017b100 0x7f568008db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:33.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5684f35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f568017b100 0x7f568008db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45510/0 (socket says 172.21.15.32:45510) 2021-04-09T18:17:33.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5684f35700 1 -- 172.21.15.32:0/2322913365 learned_addr learned my addr 172.21.15.32:0/2322913365 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:33.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f567ffff700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680179610 0x7f5680179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:33.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5677fff700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f568017a6e0 0x7f568017aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5684f35700 1 -- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f568017a6e0 msgr2=0x7f568017aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5684f35700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f568017a6e0 0x7f568017aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5684f35700 1 -- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680179610 msgr2=0x7f5680179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5684f35700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680179610 0x7f5680179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5684f35700 1 -- 172.21.15.32:0/2322913365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568008e1b0 con 0x7f568017b100 2021-04-09T18:17:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f567ffff700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680179610 0x7f5680179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.111 7f5684f35700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f568017b100 0x7f568008db90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f567800b8c0 tx=0x7f567800bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.112 7f567d7fa700 1 -- 172.21.15.32:0/2322913365 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f567800bda0 con 0x7f568017b100 2021-04-09T18:17:33.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.112 7f567d7fa700 1 -- 172.21.15.32:0/2322913365 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5678006030 con 0x7f568017b100 2021-04-09T18:17:33.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.112 7f5686918700 1 -- 172.21.15.32:0/2322913365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568008e1b0 con 0x7f568017b100 2021-04-09T18:17:33.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.112 7f567d7fa700 1 -- 172.21.15.32:0/2322913365 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56780064c0 con 0x7f568017b100 2021-04-09T18:17:33.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.112 7f5686918700 1 -- 172.21.15.32:0/2322913365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568008e1b0 con 0x7f568017b100 2021-04-09T18:17:33.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.112 7f567d7fa700 1 -- 172.21.15.32:0/2322913365 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5678006210 con 0x7f568017b100 2021-04-09T18:17:33.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.113 7f567d7fa700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f56640188c0 0x7f566401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:33.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.113 7f567d7fa700 1 -- 172.21.15.32:0/2322913365 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5678032b30 con 0x7f568017b100 2021-04-09T18:17:33.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.113 7f567ffff700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f56640188c0 0x7f566401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:33.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.113 7f567ffff700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f56640188c0 0x7f566401adc0 crc :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:33.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.132 7f5686918700 1 -- 172.21.15.32:0/2322913365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568017a1b0 con 0x7f568017b100 2021-04-09T18:17:33.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.136 7f567d7fa700 1 -- 172.21.15.32:0/2322913365 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f567802fc90 con 0x7f568017b100 2021-04-09T18:17:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.328 7f5686918700 1 -- 172.21.15.32:0/2322913365 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f568017a1b0 con 0x7f568017b100 2021-04-09T18:17:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.328 7f567d7fa700 1 -- 172.21.15.32:0/2322913365 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2444 (secure 0 0 0) 0x7f567802fc90 con 0x7f568017b100 2021-04-09T18:17:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 67 2021-04-09T18:17:33.364 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:33.364 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":64},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:32.408652","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":5883},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_5883":{"gid":5883,"name":"a","rank":0,"incarnation":67,"state":"up:starting","state_seq":2,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.330 7f5686918700 1 -- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f56640188c0 msgr2=0x7f566401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.330 7f5686918700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f56640188c0 0x7f566401adc0 crc :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.330 7f5686918700 1 -- 172.21.15.32:0/2322913365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f568017b100 msgr2=0x7f568008db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.330 7f5686918700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f568017b100 0x7f568008db90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f567800b8c0 tx=0x7f567800bc60).stop 2021-04-09T18:17:33.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.331 7f5686918700 1 -- 172.21.15.32:0/2322913365 shutdown_connections 2021-04-09T18:17:33.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.331 7f5686918700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f56640188c0 0x7f566401adc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:33.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.331 7f5686918700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f568017b100 0x7f568008db90 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.331 7f5686918700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f568017a6e0 0x7f568017aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.331 7f5686918700 1 --2- 172.21.15.32:0/2322913365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680179610 0x7f5680179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.331 7f5686918700 1 -- 172.21.15.32:0/2322913365 >> 172.21.15.32:0/2322913365 conn(0x7f5680190550 msgr2=0x7f56801b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:33.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.331 7f5686918700 1 -- 172.21.15.32:0/2322913365 shutdown_connections 2021-04-09T18:17:33.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:33.331 7f5686918700 1 -- 172.21.15.32:0/2322913365 wait complete. 2021-04-09T18:17:33.787 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:32.408652', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 5883}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_5883': {'gid': 5883, 'name': 'a', 'rank': 0, 'incarnation': 67, 'state': 'up:starting', 'state_seq': 2, 'addr': '172.21.15.32:6835/3552289149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3552289149}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3552289149}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:33.788 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5883:up:starting 2021-04-09T18:17:34.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:34.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.908 7f71d5643700 1 Processor -- start 2021-04-09T18:17:34.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.908 7f71d5643700 1 -- start start 2021-04-09T18:17:34.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71d5643700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d00d7ca0 0x7f71d00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:34.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71d5643700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71d010da00 0x7f71d010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:34.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71d5643700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d010d480 con 0x7f71d010d010 2021-04-09T18:17:34.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71d5643700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d00d9220 con 0x7f71d010da00 2021-04-09T18:17:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71d5643700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00d86c0 con 0x7f71d00d7ca0 2021-04-09T18:17:34.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71cfac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71d010da00 0x7f71d010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:34.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71cfac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71d010da00 0x7f71d010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45540/0 (socket says 172.21.15.32:45540) 2021-04-09T18:17:34.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71cfac0700 1 -- 172.21.15.32:0/2106800076 learned_addr learned my addr 172.21.15.32:0/2106800076 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71ceabe700 1 --2- 172.21.15.32:0/2106800076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d00d7ca0 0x7f71d00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:34.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71cfac0700 1 -- 172.21.15.32:0/2106800076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d00d7ca0 msgr2=0x7f71d00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:34.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71cfac0700 1 --2- 172.21.15.32:0/2106800076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d00d7ca0 0x7f71d00d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:34.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71cfac0700 1 -- 172.21.15.32:0/2106800076 >> v1:172.21.15.86:6789/0 conn(0x7f71d010d010 legacy=0x7f71d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:34.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71cfac0700 1 -- 172.21.15.32:0/2106800076 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d0112100 con 0x7f71d010da00 2021-04-09T18:17:34.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71cfac0700 1 --2- 172.21.15.32:0/2106800076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d010da00 0x7f71d010de00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f71c8006f00 tx=0x7f71c8009f10).ready entity=mon.0 client_cookie=4c42bd08b20752c7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:35.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71ce2bd700 1 -- 172.21.15.32:0/2106800076 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71c800b3b0 con 0x7f71d010da00 2021-04-09T18:17:35.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71ce2bd700 1 -- 172.21.15.32:0/2106800076 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c800b730 con 0x7f71d010da00 2021-04-09T18:17:35.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71ce2bd700 1 -- 172.21.15.32:0/2106800076 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71c800bc10 con 0x7f71d010da00 2021-04-09T18:17:35.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71d5643700 1 -- 172.21.15.32:0/2106800076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d010da00 msgr2=0x7f71d010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:35.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.909 7f71d5643700 1 --2- 172.21.15.32:0/2106800076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d010da00 0x7f71d010de00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f71c8006f00 tx=0x7f71c8009f10).stop 2021-04-09T18:17:35.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.910 7f71d5643700 1 -- 172.21.15.32:0/2106800076 shutdown_connections 2021-04-09T18:17:35.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.910 7f71d5643700 1 --2- 172.21.15.32:0/2106800076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d010da00 0x7f71d010de00 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:35.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.910 7f71d5643700 1 --2- 172.21.15.32:0/2106800076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d00d7ca0 0x7f71d00d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:35.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.910 7f71d5643700 1 -- 172.21.15.32:0/2106800076 >> 172.21.15.32:0/2106800076 conn(0x7f71d00ce5f0 msgr2=0x7f71d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:35.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.910 7f71d5643700 1 -- 172.21.15.32:0/2106800076 shutdown_connections 2021-04-09T18:17:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.910 7f71d5643700 1 -- 172.21.15.32:0/2106800076 wait complete. 2021-04-09T18:17:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.930 7f71d5643700 1 Processor -- start 2021-04-09T18:17:35.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71d5643700 1 -- start start 2021-04-09T18:17:35.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71d5643700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d01ac380 0x7f71d018a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71d5643700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d01ac780 0x7f71d018a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71d5643700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71d0183520 0x7f71d0183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:35.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71d5643700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d01ace10 con 0x7f71d01ac380 2021-04-09T18:17:35.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71d5643700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d0189cf0 con 0x7f71d0183520 2021-04-09T18:17:35.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71d5643700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d0183f40 con 0x7f71d01ac780 2021-04-09T18:17:35.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71cf2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d01ac380 0x7f71d018a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71cf2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d01ac380 0x7f71d018a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45544/0 (socket says 172.21.15.32:45544) 2021-04-09T18:17:35.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71cf2bf700 1 -- 172.21.15.32:0/4208390471 learned_addr learned my addr 172.21.15.32:0/4208390471 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:35.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71cfac0700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71d0183520 0x7f71d0183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:35.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.931 7f71ceabe700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d01ac780 0x7f71d018a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:35.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71cf2bf700 1 -- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d01ac780 msgr2=0x7f71d018a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:35.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71cf2bf700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d01ac780 0x7f71d018a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:35.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71cf2bf700 1 -- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71d0183520 msgr2=0x7f71d0183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:35.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71cf2bf700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71d0183520 0x7f71d0183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:35.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71cf2bf700 1 -- 172.21.15.32:0/4208390471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d0184160 con 0x7f71d01ac380 2021-04-09T18:17:35.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71ceabe700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d01ac780 0x7f71d018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:35.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71cfac0700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71d0183520 0x7f71d0183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:35.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71cf2bf700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d01ac380 0x7f71d018a230 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f71c000b5b0 tx=0x7f71c000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:35.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71c7fff700 1 -- 172.21.15.32:0/4208390471 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71c000bd40 con 0x7f71d01ac380 2021-04-09T18:17:35.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71c7fff700 1 -- 172.21.15.32:0/4208390471 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c0006020 con 0x7f71d01ac380 2021-04-09T18:17:35.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71d5643700 1 -- 172.21.15.32:0/4208390471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d0184160 con 0x7f71d01ac380 2021-04-09T18:17:35.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71c7fff700 1 -- 172.21.15.32:0/4208390471 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71c00054a0 con 0x7f71d01ac380 2021-04-09T18:17:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.932 7f71d5643700 1 -- 172.21.15.32:0/4208390471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d0184160 con 0x7f71d01ac380 2021-04-09T18:17:35.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.933 7f71c7fff700 1 -- 172.21.15.32:0/4208390471 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f71c0006200 con 0x7f71d01ac380 2021-04-09T18:17:35.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.933 7f71c7fff700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71ac018930 0x7f71ac01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:35.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.933 7f71c7fff700 1 -- 172.21.15.32:0/4208390471 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f71c0032a00 con 0x7f71d01ac380 2021-04-09T18:17:35.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.933 7f71ceabe700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71ac018930 0x7f71ac01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:35.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.933 7f71ceabe700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71ac018930 0x7f71ac01ae10 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:35.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.953 7f71d5643700 1 -- 172.21.15.32:0/4208390471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d0182d10 con 0x7f71d01ac380 2021-04-09T18:17:35.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:34.956 7f71c7fff700 1 -- 172.21.15.32:0/4208390471 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f71c000bd40 con 0x7f71d01ac380 2021-04-09T18:17:35.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.146 7f71d5643700 1 -- 172.21.15.32:0/4208390471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71d0182d10 con 0x7f71d01ac380 2021-04-09T18:17:35.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.146 7f71c7fff700 1 -- 172.21.15.32:0/4208390471 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2456 (secure 0 0 0) 0x7f71c0006000 con 0x7f71d01ac380 2021-04-09T18:17:35.188 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 70 2021-04-09T18:17:35.189 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:35.189 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:34.416201","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5883,"mds_1":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5883":{"gid":5883,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":13,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6084":{"gid":6084,"name":"c","rank":1,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:35.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 -- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71ac018930 msgr2=0x7f71ac01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:35.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71ac018930 0x7f71ac01ae10 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:35.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 -- 172.21.15.32:0/4208390471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d01ac380 msgr2=0x7f71d018a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d01ac380 0x7f71d018a230 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f71c000b5b0 tx=0x7f71c000b950).stop 2021-04-09T18:17:35.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 -- 172.21.15.32:0/4208390471 shutdown_connections 2021-04-09T18:17:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71ac018930 0x7f71ac01ae10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:35.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71d0183520 0x7f71d0183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71d01ac780 0x7f71d018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 --2- 172.21.15.32:0/4208390471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71d01ac380 0x7f71d018a230 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:35.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 -- 172.21.15.32:0/4208390471 >> 172.21.15.32:0/4208390471 conn(0x7f71d019b890 msgr2=0x7f71d01a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:35.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 -- 172.21.15.32:0/4208390471 shutdown_connections 2021-04-09T18:17:35.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:35.149 7f71d5643700 1 -- 172.21.15.32:0/4208390471 wait complete. 2021-04-09T18:17:35.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:17:35.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:17:35.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:16:48.944470', 'modified': '2021-04-09 18:17:34.416201', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 5883, 'mds_1': 6084}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5883': {'gid': 5883, 'name': 'a', 'rank': 0, 'incarnation': 67, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.32:6835/3552289149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3552289149}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3552289149}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6084': {'gid': 6084, 'name': 'c', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.32:6833/3850456471', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3850456471}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 3850456471}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:17:35.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:17:35.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:17:35.762 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:17:35.763 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:17:35.763 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:17:35.763 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 5883, 2021-04-09T18:17:35.763 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:17:35.764 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:17:35.764 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 3.315814928, 2021-04-09T18:17:35.764 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 70, 2021-04-09T18:17:35.764 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 63, 2021-04-09T18:17:35.765 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 63, 2021-04-09T18:17:35.765 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 51.213550671 2021-04-09T18:17:35.765 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:17:35.767 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 5883, "want_state": "up:active", "state": "up:active", "rank_uptime": 3.315814928, "mdsmap_epoch": 70, "osdmap_epoch": 63, "osdmap_epoch_barrier": 63, "uptime": 51.213550671 } 2021-04-09T18:17:35.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:17:35.921 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:17:35.922 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:17:35.922 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:17:35.923 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6084, 2021-04-09T18:17:35.923 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:17:35.924 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:17:35.924 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 2.4725094150000002, 2021-04-09T18:17:35.925 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 70, 2021-04-09T18:17:35.925 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 63, 2021-04-09T18:17:35.925 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 63, 2021-04-09T18:17:35.926 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 25.431670562000001 2021-04-09T18:17:35.926 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:17:35.934 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 6084, "want_state": "up:active", "state": "up:active", "rank_uptime": 2.4725094150000002, "mdsmap_epoch": 70, "osdmap_epoch": 63, "osdmap_epoch_barrier": 63, "uptime": 25.431670562000001 } 2021-04-09T18:17:35.936 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:17:35.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:17:36.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:17:36.063 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:17:36.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:17:36.072 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:17:36.072 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:17:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:17:36.131 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:17:36.132 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:17:36.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:17:36.133 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:17:36.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:17:36.199 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:17:36.200 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:17:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:17:36.253 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:17:36.254 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:17:36.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:17:36.256 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:17:36.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:17:36.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.458 7ff73e1e2700 1 Processor -- start 2021-04-09T18:17:36.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.458 7ff73e1e2700 1 -- start start 2021-04-09T18:17:36.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff73e1e2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff7380d7ca0 0x7ff7380d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff73e1e2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff73810da00 0x7ff73810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff73e1e2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73810d480 con 0x7ff73810d010 2021-04-09T18:17:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff73e1e2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7380d9220 con 0x7ff7380d7ca0 2021-04-09T18:17:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff73e1e2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7380d86c0 con 0x7ff73810da00 2021-04-09T18:17:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff736ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff7380d7ca0 0x7ff7380d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff7377fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff73810d010 0x7ff7380d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55140/0 (socket says 172.21.15.32:55140) 2021-04-09T18:17:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff7377fe700 1 -- 172.21.15.32:0/4052144989 learned_addr learned my addr 172.21.15.32:0/4052144989 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:36.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff737fff700 1 --2- 172.21.15.32:0/4052144989 >> v2:172.21.15.86:3300/0 conn(0x7ff73810da00 0x7ff73810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:36.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff736ffd700 1 -- 172.21.15.32:0/4052144989 >> v2:172.21.15.86:3300/0 conn(0x7ff73810da00 msgr2=0x7ff73810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff736ffd700 1 --2- 172.21.15.32:0/4052144989 >> v2:172.21.15.86:3300/0 conn(0x7ff73810da00 0x7ff73810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff736ffd700 1 -- 172.21.15.32:0/4052144989 >> v1:172.21.15.32:6789/0 conn(0x7ff73810d010 legacy=0x7ff7380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff736ffd700 1 -- 172.21.15.32:0/4052144989 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7380d1f70 con 0x7ff7380d7ca0 2021-04-09T18:17:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff7367fc700 1 -- 172.21.15.32:0/4052144989 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395372814 0 0) 0x7ff728000ce0 con 0x7ff73810d010 2021-04-09T18:17:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff736ffd700 1 --2- 172.21.15.32:0/4052144989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7380d7ca0 0x7ff7380d80a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7ff72c006650 tx=0x7ff72c009a00).ready entity=mon.0 client_cookie=902c299bc6113a86 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.459 7ff7367fc700 1 -- 172.21.15.32:0/4052144989 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff72c009df0 con 0x7ff7380d7ca0 2021-04-09T18:17:36.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff7367fc700 1 -- 172.21.15.32:0/4052144989 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c0009b0 con 0x7ff7380d7ca0 2021-04-09T18:17:36.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff7367fc700 1 -- 172.21.15.32:0/4052144989 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff72c00c2b0 con 0x7ff7380d7ca0 2021-04-09T18:17:36.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff73e1e2700 1 -- 172.21.15.32:0/4052144989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7380d7ca0 msgr2=0x7ff7380d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:36.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff73e1e2700 1 --2- 172.21.15.32:0/4052144989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7380d7ca0 0x7ff7380d80a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7ff72c006650 tx=0x7ff72c009a00).stop 2021-04-09T18:17:36.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff73e1e2700 1 -- 172.21.15.32:0/4052144989 shutdown_connections 2021-04-09T18:17:36.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff73e1e2700 1 --2- 172.21.15.32:0/4052144989 >> v2:172.21.15.86:3300/0 conn(0x7ff73810da00 0x7ff73810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:36.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff73e1e2700 1 --2- 172.21.15.32:0/4052144989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7380d7ca0 0x7ff7380d80a0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:36.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff73e1e2700 1 -- 172.21.15.32:0/4052144989 >> 172.21.15.32:0/4052144989 conn(0x7ff7380ce5f0 msgr2=0x7ff7380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:36.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff73e1e2700 1 -- 172.21.15.32:0/4052144989 shutdown_connections 2021-04-09T18:17:36.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.460 7ff73e1e2700 1 -- 172.21.15.32:0/4052144989 wait complete. 2021-04-09T18:17:36.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.480 7ff73e1e2700 1 Processor -- start 2021-04-09T18:17:36.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff73e1e2700 1 -- start start 2021-04-09T18:17:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff73e1e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7381a5210 0x7ff7381a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff73e1e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7381a6340 0x7ff7381a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:36.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff73e1e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7380abaa0 0x7ff7380abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:36.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff73e1e2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381a6d60 con 0x7ff7381a5210 2021-04-09T18:17:36.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff73e1e2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380ac4c0 con 0x7ff7380abaa0 2021-04-09T18:17:36.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff73e1e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380ac6e0 con 0x7ff7381a6340 2021-04-09T18:17:36.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff7377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7381a5210 0x7ff7381a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff7377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7381a5210 0x7ff7381a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45556/0 (socket says 172.21.15.32:45556) 2021-04-09T18:17:36.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff7377fe700 1 -- 172.21.15.32:0/1562333893 learned_addr learned my addr 172.21.15.32:0/1562333893 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:36.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff736ffd700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7381a6340 0x7ff7381a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:36.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.481 7ff737fff700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7380abaa0 0x7ff7380abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:36.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff7377fe700 1 -- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7381a6340 msgr2=0x7ff7381a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff7377fe700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7381a6340 0x7ff7381a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:36.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff7377fe700 1 -- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7380abaa0 msgr2=0x7ff7380abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:36.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff7377fe700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7380abaa0 0x7ff7380abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:36.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff7377fe700 1 -- 172.21.15.32:0/1562333893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff738179610 con 0x7ff7381a5210 2021-04-09T18:17:36.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff7377fe700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7381a5210 0x7ff7381a56c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7ff728006f00 tx=0x7ff72800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:36.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff71ffff700 1 -- 172.21.15.32:0/1562333893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff728006020 con 0x7ff7381a5210 2021-04-09T18:17:36.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff71ffff700 1 -- 172.21.15.32:0/1562333893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7280063a0 con 0x7ff7381a5210 2021-04-09T18:17:36.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff738179610 con 0x7ff7381a5210 2021-04-09T18:17:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff71ffff700 1 -- 172.21.15.32:0/1562333893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff728004620 con 0x7ff7381a5210 2021-04-09T18:17:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.482 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff738179610 con 0x7ff7381a5210 2021-04-09T18:17:36.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.483 7ff71ffff700 1 -- 172.21.15.32:0/1562333893 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff72800a000 con 0x7ff7381a5210 2021-04-09T18:17:36.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.483 7ff71ffff700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff718018930 0x7ff71801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:36.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.483 7ff71ffff700 1 -- 172.21.15.32:0/1562333893 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff7280329e0 con 0x7ff7381a5210 2021-04-09T18:17:36.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.483 7ff736ffd700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff718018930 0x7ff71801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:36.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.483 7ff736ffd700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff718018930 0x7ff71801adc0 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:36.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.503 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7381a6d60 con 0x7ff7381a5210 2021-04-09T18:17:36.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.506 7ff71ffff700 1 -- 172.21.15.32:0/1562333893 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff728006b90 con 0x7ff7381a5210 2021-04-09T18:17:36.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.694 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff7381a6d60 con 0x7ff7381a5210 2021-04-09T18:17:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.695 7ff71ffff700 1 -- 172.21.15.32:0/1562333893 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v63) v1 ==== 81+0+30363 (secure 0 0 0) 0x7ff72800a000 con 0x7ff7381a5210 2021-04-09T18:17:36.731 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:36.731 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:17:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 63, 2021-04-09T18:17:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:17:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:17:36.732 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:16:48.944337", 2021-04-09T18:17:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:17:36.733 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:17:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:17:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:17:36.735 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:17:36.736 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:17:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:17:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:17:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:17:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:17:36.738 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:17:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:17:36.739 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:17:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:17:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:17:36.761 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 11, 2021-04-09T18:17:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:17:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:17:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:17:36.762 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:17:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:17:36.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:17:36.776 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:17:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:17:36.777 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:17:36.782 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:17:36.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:17:36.782 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:17:36.782 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:17:36.783 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:17:36.783 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:17:36.783 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:17:36.783 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:17:36.804 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:17:36.812 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:17:36.812 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:17:36.818 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:17:36.819 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:17:36.826 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:17:36.835 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:17:36.836 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:17:36.836 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:17:36.840 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:17:36.841 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:17:36.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:36.841 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:17:36.845 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:17:36.846 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:17:36.846 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:17:36.846 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:17:36.855 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:17:36.855 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:17:36.855 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:17:36.892 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:17:36.893 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:17:36.893 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:17:36.899 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:17:36.910 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:17:36.927 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:17:36.928 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:17:36.928 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:17:36.928 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:17:36.929 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:17:36.929 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:17:36.929 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:17:36.943 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:17:36.944 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:17:36.944 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:17:36.957 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:17:36.957 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:17:36.975 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:17:36.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:17:36.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:36.983 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:17:37.022 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:17:37.023 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:17:37.023 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:17:37.024 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:17:37.024 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:17:37.025 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:17:37.025 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:17:37.025 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:17:37.026 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:17:37.027 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:17:37.050 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:17:37.053 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:17:37.053 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:17:37.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.067 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 10, 2021-04-09T18:17:37.067 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:17:37.068 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:16:47.324034", 2021-04-09T18:17:37.068 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:17:37.069 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:17:37.069 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:17:37.069 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:17:37.070 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:17:37.070 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:17:37.085 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:17:37.085 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:17:37.085 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:17:37.095 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:17:37.096 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:17:37.100 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:17:37.101 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:17:37.101 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:17:37.102 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:17:37.104 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:17:37.104 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:17:37.112 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:17:37.112 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:17:37.113 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:17:37.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.141 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "63", 2021-04-09T18:17:37.142 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:17:37.142 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:17:37.143 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:17:37.177 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:17:37.181 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:17:37.182 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:17:37.210 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:17:37.215 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:17:37.215 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:17:37.215 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:17:37.216 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:17:37.216 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:17:37.216 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:17:37.217 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:17:37.217 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:17:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:17:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:17:37.218 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:17:37.219 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:17:37.238 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:17:37.239 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:17:37.239 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:17:37.240 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:17:37.240 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:17:37.253 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:17:37.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:17:37.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.254 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:17:37.255 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:17:37.255 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:17:37.256 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:17:37.256 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:17:37.256 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:17:37.257 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:17:37.257 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:17:37.258 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:17:37.258 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:17:37.258 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:17:37.317 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:17:37.325 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:17:37.332 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:17:37.332 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:17:37.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.332 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:17:37.333 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:17:37.333 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:17:37.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.334 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 11, 2021-04-09T18:17:37.334 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:17:37.334 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:16:48.346213", 2021-04-09T18:17:37.335 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:17:37.335 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:17:37.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:17:37.364 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:17:37.365 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:17:37.365 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:17:37.365 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:17:37.366 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:17:37.366 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:17:37.367 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:17:37.395 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:17:37.396 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:17:37.396 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:17:37.397 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:17:37.397 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:17:37.397 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:17:37.398 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:17:37.398 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:17:37.398 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:17:37.400 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:17:37.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.400 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "63", 2021-04-09T18:17:37.401 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:17:37.401 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:17:37.401 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:17:37.402 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:17:37.449 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:17:37.449 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:17:37.450 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:17:37.450 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:17:37.450 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:17:37.451 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:17:37.452 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:17:37.452 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:17:37.453 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:17:37.453 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:17:37.485 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:17:37.485 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:17:37.485 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:17:37.485 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:17:37.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:17:37.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:17:37.493 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:17:37.493 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:17:37.494 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:17:37.494 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:17:37.498 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:17:37.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:17:37.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.499 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:17:37.499 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:17:37.501 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:17:37.501 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:17:37.501 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:17:37.501 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:17:37.501 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:17:37.502 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:17:37.516 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:17:37.516 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:17:37.517 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:17:37.517 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:17:37.517 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:17:37.517 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:17:37.517 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:17:37.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.524 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:17:37.525 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:17:37.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.525 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:17:37.525 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:17:37.526 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:37.526 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:37.526 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:37.526 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:37.527 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:37.534 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:37.534 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:37.534 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:37.535 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:37.535 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:37.535 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:37.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:37.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:37.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:17:37.536 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:37.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.537 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:37.537 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:17:37.537 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:37.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.538 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.553 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:37.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:37.553 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:37.553 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:17:37.554 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:37.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:37.555 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:17:37.555 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:37.555 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.555 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.556 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:37.556 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:37.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:37.634 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:17:37.635 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:37.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:37.643 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:17:37.644 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:37.644 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.644 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.644 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:37.645 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:37.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.645 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:37.645 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:17:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.670 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:17:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:17:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:17:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:17:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:17:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:37.688 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:17:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:17:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:37.690 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:37.691 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:37.706 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:17:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:37.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.719 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:37.722 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:17:37.725 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:37.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.738 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:37.754 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:37.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.754 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:37.756 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:17:37.756 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:37.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.758 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:37.758 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:17:37.759 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:37.759 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.767 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.768 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:37.768 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:37.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.769 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:37.769 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:17:37.769 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:37.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.788 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:37.788 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:17:37.788 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:37.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.797 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.798 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:37.798 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:37.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:37.807 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:17:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:37.829 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:17:37.829 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:37.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.830 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.831 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:17:37.831 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:17:37.864 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:17:37.899 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:17:37.899 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:37.900 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:37.900 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:37.900 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:37.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.901 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:17:37.901 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:17:37.902 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:37.902 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:37.902 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:37.902 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:37.902 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:37.903 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:37.903 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:37.951 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:37.951 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:37.952 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:37.953 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:37.993 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:37.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.993 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:37.994 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:17:37.995 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:37.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:37.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:37.997 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:37.997 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:17:37.998 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:37.998 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:37.998 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.045 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:38.046 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.047 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.052 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:17:38.053 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:38.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.054 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.100 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:17:38.101 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:38.101 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.101 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.102 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:38.103 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:17:38.104 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:38.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.139 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:17:38.140 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:38.140 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.141 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.141 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:38.188 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.189 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:17:38.212 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:38.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.214 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.217 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:17:38.218 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:38.218 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.218 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.219 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:17:38.220 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:17:38.234 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:17:38.235 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:17:38.240 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:38.240 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:38.241 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:38.241 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:17:38.243 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:17:38.243 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:38.275 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:38.281 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:38.282 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:38.307 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:38.308 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:17:38.331 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:38.332 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.334 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:17:38.345 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.376 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:17:38.376 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:38.377 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.377 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.386 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:38.386 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.387 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.391 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.391 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:17:38.393 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:38.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.394 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.422 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:17:38.425 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:38.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.426 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.427 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:38.429 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.431 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:17:38.431 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:38.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.472 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:17:38.473 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:38.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.473 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.474 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:38.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.514 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:17:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:38.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.516 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:17:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:38.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:17:38.518 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:17:38.519 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:17:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:17:38.569 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:38.577 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:38.578 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:17:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:17:38.580 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:38.581 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:38.582 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:38.583 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:38.632 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 58, 2021-04-09T18:17:38.632 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:38.632 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:38.632 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:38.633 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.633 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.633 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:17:38.634 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:38.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.666 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:17:38.667 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:38.667 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.667 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.668 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:38.668 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.677 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:17:38.677 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:38.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:17:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.734 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:17:38.734 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:38.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:17:38.736 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:38.748 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.749 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.773 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:38.777 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.778 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.778 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:17:38.778 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:38.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.791 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:17:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:17:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:17:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:17:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:17:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:17:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:17:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:17:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:17:38.938 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:38.938 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.939 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.940 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:38.940 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.941 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:17:38.941 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:38.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:17:38.984 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:38.985 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.986 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.986 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:38.986 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:17:38.987 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:38.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:17:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:38.989 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:38.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.989 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:38.989 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:38.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:38.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:38.990 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:17:38.990 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:38.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:38.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.015 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:39.017 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:17:39.017 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:39.018 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.018 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.019 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:17:39.019 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:17:39.019 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:17:39.022 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:17:39.022 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:39.026 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:39.028 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:39.028 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.029 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:17:39.029 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:17:39.030 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:39.030 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:39.030 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:39.031 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:39.063 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:39.063 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:39.064 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:39.064 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:39.065 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:39.065 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:39.066 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:39.066 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:39.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:39.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:17:39.068 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:39.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:39.133 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:17:39.134 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:39.141 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.141 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.145 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:39.146 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:39.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.204 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:39.204 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:17:39.205 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:39.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:39.213 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:17:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.214 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.215 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:39.249 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.250 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:17:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:39.251 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.271 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:17:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:39.272 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.273 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.293 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:39.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:17:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:39.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.334 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:39.335 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:17:39.339 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:39.344 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.346 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.347 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:17:39.379 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:17:39.380 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:17:39.392 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:17:39.395 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:39.396 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:39.396 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:39.396 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.398 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:17:39.398 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:17:39.398 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:39.407 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:39.407 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:39.408 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:39.408 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:39.408 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:39.409 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:39.409 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:39.410 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:39.410 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:39.433 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:39.436 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:39.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.437 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:39.437 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:17:39.438 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:39.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.439 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:39.439 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:17:39.440 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:39.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.440 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.441 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:39.442 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:39.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:39.443 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:17:39.480 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:39.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.506 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:39.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:17:39.508 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:39.514 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.514 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.515 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:39.515 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:39.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:39.525 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:17:39.525 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:39.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.526 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:39.527 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:17:39.527 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:39.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.528 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.528 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:39.529 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:39.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:39.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:17:39.569 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:39.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:39.571 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:17:39.571 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:39.600 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.625 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.628 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:17:39.629 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:17:39.629 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:17:39.631 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:17:39.631 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:39.631 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:39.632 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:39.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:39.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.673 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:17:39.674 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:17:39.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.675 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:17:39.675 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:39.716 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:39.716 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:39.717 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:39.717 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:39.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.720 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:17:39.720 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:39.721 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:39.721 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:39.721 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:39.722 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:39.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.755 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:17:39.756 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:39.756 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:39.756 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:39.757 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:39.757 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:39.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.784 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:17:39.785 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:39.785 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:39.786 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:39.786 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:39.798 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:39.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.799 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:17:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:39.800 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:39.801 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:17:39.819 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:39.820 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:39.820 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:39.821 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:39.833 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:39.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:17:39.842 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:39.843 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:39.843 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:39.853 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:39.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:39.865 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:17:39.866 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:39.866 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:39.867 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:39.872 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:39.872 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:39.873 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.873 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:17:39.879 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:17:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:17:39.895 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:17:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:17:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:17:39.896 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:17:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:17:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:17:39.897 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:17:39.931 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:17:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:17:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:17:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:17:39.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:39.933 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:17:39.938 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:17:39.939 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:17:39.939 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:17:39.939 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:17:39.939 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:17:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff718018930 msgr2=0x7ff71801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff718018930 0x7ff71801adc0 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:39.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7381a5210 msgr2=0x7ff7381a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7381a5210 0x7ff7381a56c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7ff728006f00 tx=0x7ff72800b940).stop 2021-04-09T18:17:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 shutdown_connections 2021-04-09T18:17:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff718018930 0x7ff71801adc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:39.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7380abaa0 0x7ff7380abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7381a6340 0x7ff7381a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 --2- 172.21.15.32:0/1562333893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7381a5210 0x7ff7381a56c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 >> 172.21.15.32:0/1562333893 conn(0x7ff7381a99c0 msgr2=0x7ff73817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 shutdown_connections 2021-04-09T18:17:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:36.699 7ff73e1e2700 1 -- 172.21.15.32:0/1562333893 wait complete. 2021-04-09T18:17:40.003 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:40.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.133 7f6369b5b700 1 Processor -- start 2021-04-09T18:17:40.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.133 7f6369b5b700 1 -- start start 2021-04-09T18:17:40.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f6369b5b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63640d7ca0 0x7f63640d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f6369b5b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636410d480 con 0x7f636410da00 2021-04-09T18:17:40.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f6369b5b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63640d86c0 con 0x7f636410d010 2021-04-09T18:17:40.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f6369b5b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63640d9220 con 0x7f63640d7ca0 2021-04-09T18:17:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f6363fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f636410da00 0x7f636410de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55156/0 (socket says 172.21.15.32:55156) 2021-04-09T18:17:40.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f6363fff700 1 -- 172.21.15.32:0/1207600682 learned_addr learned my addr 172.21.15.32:0/1207600682 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:40.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f6362ffd700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63640d7ca0 0x7f63640d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f63627fc700 1 -- 172.21.15.32:0/1207600682 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308534897 0 0) 0x7f635c000d30 con 0x7f636410da00 2021-04-09T18:17:40.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f63627fc700 1 -- 172.21.15.32:0/1207600682 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c002fe0 con 0x7f636410da00 2021-04-09T18:17:40.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f63627fc700 1 -- 172.21.15.32:0/1207600682 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398219782 0 0) 0x7f6354000c60 con 0x7f636410d010 2021-04-09T18:17:40.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.134 7f63627fc700 1 -- 172.21.15.32:0/1207600682 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c003210 con 0x7f636410d010 2021-04-09T18:17:40.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2819606175 0 0) 0x7f635c001460 con 0x7f636410da00 2021-04-09T18:17:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63640d7ca0 msgr2=0x7f63640d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63640d7ca0 0x7f63640d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 >> v1:172.21.15.86:6789/0 conn(0x7f636410d010 legacy=0x7f63640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364112100 con 0x7f636410da00 2021-04-09T18:17:40.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f635c001980 con 0x7f636410da00 2021-04-09T18:17:40.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 >> v1:172.21.15.32:6789/0 conn(0x7f636410da00 legacy=0x7f636410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c003b90 0x7f634c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f634c0067a0 0x7f634c008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f634c009510 0x7f634c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0502f0 con 0x7f634c003b90 2021-04-09T18:17:40.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0504d0 con 0x7f634c009510 2021-04-09T18:17:40.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0506f0 con 0x7f634c0067a0 2021-04-09T18:17:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63637fe700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c003b90 0x7f634c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f6363fff700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f634c009510 0x7f634c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f6362ffd700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f634c0067a0 0x7f634c008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63637fe700 1 -- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f634c0067a0 msgr2=0x7f634c008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63637fe700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f634c0067a0 0x7f634c008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63637fe700 1 -- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f634c009510 msgr2=0x7f634c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:40.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63637fe700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f634c009510 0x7f634c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63637fe700 1 -- 172.21.15.32:0/1207600682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c050990 con 0x7f634c003b90 2021-04-09T18:17:40.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63637fe700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c003b90 0x7f634c005fa0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f6354001040 tx=0x7f6354009aa0).ready entity=mon.0 client_cookie=474bbc855d5ddd2c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:40.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63540066e0 con 0x7f634c003b90 2021-04-09T18:17:40.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635400c0c0 con 0x7f634c003b90 2021-04-09T18:17:40.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f63627fc700 1 -- 172.21.15.32:0/1207600682 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f635400c620 con 0x7f634c003b90 2021-04-09T18:17:40.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f6369b5b700 1 -- 172.21.15.32:0/1207600682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c003b90 msgr2=0x7f634c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f6369b5b700 1 --2- 172.21.15.32:0/1207600682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c003b90 0x7f634c005fa0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f6354001040 tx=0x7f6354009aa0).stop 2021-04-09T18:17:40.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.135 7f6363fff700 1 -- 172.21.15.32:0/1207600682 reap_dead start 2021-04-09T18:17:40.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.136 7f6369b5b700 1 -- 172.21.15.32:0/1207600682 shutdown_connections 2021-04-09T18:17:40.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.136 7f6369b5b700 1 -- 172.21.15.32:0/1207600682 >> 172.21.15.32:0/1207600682 conn(0x7f63640ce5f0 msgr2=0x7f63640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.136 7f6369b5b700 1 -- 172.21.15.32:0/1207600682 shutdown_connections 2021-04-09T18:17:40.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.136 7f6369b5b700 1 -- 172.21.15.32:0/1207600682 wait complete. 2021-04-09T18:17:40.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.156 7f6369b5b700 1 Processor -- start 2021-04-09T18:17:40.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f6369b5b700 1 -- start start 2021-04-09T18:17:40.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f6369b5b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63641acea0 0x7f63641ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f6369b5b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f636419b2c0 0x7f6364184da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f6369b5b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63641853c0 0x7f6364185fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f6369b5b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63641857c0 con 0x7f63641acea0 2021-04-09T18:17:40.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f6369b5b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6364185a40 con 0x7f636419b2c0 2021-04-09T18:17:40.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f6369b5b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63641865e0 con 0x7f63641853c0 2021-04-09T18:17:40.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f63637fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63641acea0 0x7f63641ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f63637fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63641acea0 0x7f63641ac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45576/0 (socket says 172.21.15.32:45576) 2021-04-09T18:17:40.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f63637fe700 1 -- 172.21.15.32:0/2535441591 learned_addr learned my addr 172.21.15.32:0/2535441591 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:40.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f6362ffd700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f636419b2c0 0x7f6364184da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.157 7f6363fff700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63641853c0 0x7f6364185fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f63637fe700 1 -- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63641853c0 msgr2=0x7f6364185fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f63637fe700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63641853c0 0x7f6364185fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f63637fe700 1 -- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f636419b2c0 msgr2=0x7f6364184da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f63637fe700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f636419b2c0 0x7f6364184da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f63637fe700 1 -- 172.21.15.32:0/2535441591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364186800 con 0x7f63641acea0 2021-04-09T18:17:40.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f6362ffd700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f636419b2c0 0x7f6364184da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:40.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f63637fe700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63641acea0 0x7f63641ac670 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f6354006e90 tx=0x7f635400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:40.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f634bfff700 1 -- 172.21.15.32:0/2535441591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6354006240 con 0x7f63641acea0 2021-04-09T18:17:40.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f634bfff700 1 -- 172.21.15.32:0/2535441591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6354009020 con 0x7f63641acea0 2021-04-09T18:17:40.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6364186800 con 0x7f63641acea0 2021-04-09T18:17:40.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f634bfff700 1 -- 172.21.15.32:0/2535441591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6354005540 con 0x7f63641acea0 2021-04-09T18:17:40.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.158 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6364186800 con 0x7f63641acea0 2021-04-09T18:17:40.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.159 7f634bfff700 1 -- 172.21.15.32:0/2535441591 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6354006030 con 0x7f63641acea0 2021-04-09T18:17:40.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.159 7f634bfff700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63440188d0 0x7f634401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.159 7f634bfff700 1 -- 172.21.15.32:0/2535441591 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6354032a00 con 0x7f63641acea0 2021-04-09T18:17:40.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.159 7f6362ffd700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63440188d0 0x7f634401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.159 7f6362ffd700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63440188d0 0x7f634401add0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:40.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.178 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6364182d80 con 0x7f63641acea0 2021-04-09T18:17:40.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.181 7f634bfff700 1 -- 172.21.15.32:0/2535441591 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6354009020 con 0x7f63641acea0 2021-04-09T18:17:40.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.372 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f636410d480 con 0x7f63641acea0 2021-04-09T18:17:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.372 7f634bfff700 1 -- 172.21.15.32:0/2535441591 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2794 (secure 0 0 0) 0x7f635400bd50 con 0x7f63641acea0 2021-04-09T18:17:40.408 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 71 2021-04-09T18:17:40.409 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:40.410 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":71,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2614834008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2614834008},{"type":"v1","addr":"172.21.15.86:6837","nonce":2614834008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:34.416201","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5883,"mds_1":6084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5883":{"gid":5883,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":13,"addr":"172.21.15.32:6835/3552289149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3552289149},{"type":"v1","addr":"172.21.15.32:6835","nonce":3552289149}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6084":{"gid":6084,"name":"c","rank":1,"incarnation":69,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6833/3850456471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3850456471},{"type":"v1","addr":"172.21.15.32:6833","nonce":3850456471}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:40.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63440188d0 msgr2=0x7f634401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63440188d0 0x7f634401add0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63641acea0 msgr2=0x7f63641ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63641acea0 0x7f63641ac670 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f6354006e90 tx=0x7f635400b960).stop 2021-04-09T18:17:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 shutdown_connections 2021-04-09T18:17:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63440188d0 0x7f634401add0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63641853c0 0x7f6364185fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f636419b2c0 0x7f6364184da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 --2- 172.21.15.32:0/2535441591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63641acea0 0x7f63641ac670 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 >> 172.21.15.32:0/2535441591 conn(0x7f636419b890 msgr2=0x7f63641a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 shutdown_connections 2021-04-09T18:17:40.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.375 7f6369b5b700 1 -- 172.21.15.32:0/2535441591 wait complete. 2021-04-09T18:17:40.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:17:40.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.558 7fba2f87e700 1 Processor -- start 2021-04-09T18:17:40.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.558 7fba2f87e700 1 -- start start 2021-04-09T18:17:40.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2f87e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fba280d6c90 0x7fba280d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2f87e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280d7290 con 0x7fba280cffb0 2021-04-09T18:17:40.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2f87e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280d2850 con 0x7fba280d2450 2021-04-09T18:17:40.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2f87e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba280d7090 con 0x7fba280d6c90 2021-04-09T18:17:40.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2d69a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fba280cffb0 0x7fba280d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55174/0 (socket says 172.21.15.32:55174) 2021-04-09T18:17:40.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2d69a700 1 -- 172.21.15.32:0/3215782495 learned_addr learned my addr 172.21.15.32:0/3215782495 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:40.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2ce99700 1 --2- 172.21.15.32:0/3215782495 >> v2:172.21.15.86:3300/0 conn(0x7fba280d6c90 0x7fba280d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba1ffff700 1 -- 172.21.15.32:0/3215782495 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645910469 0 0) 0x7fba24000ce0 con 0x7fba280cffb0 2021-04-09T18:17:40.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba1ffff700 1 -- 172.21.15.32:0/3215782495 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba14002fe0 con 0x7fba280cffb0 2021-04-09T18:17:40.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba1ffff700 1 -- 172.21.15.32:0/3215782495 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222100803 0 0) 0x7fba20000d30 con 0x7fba280d2450 2021-04-09T18:17:40.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba1ffff700 1 -- 172.21.15.32:0/3215782495 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba14003210 con 0x7fba280d2450 2021-04-09T18:17:40.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba1ffff700 1 -- 172.21.15.32:0/3215782495 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 574933340 0 0) 0x7fba24000ed0 con 0x7fba280cffb0 2021-04-09T18:17:40.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2ce99700 1 -- 172.21.15.32:0/3215782495 >> v1:172.21.15.86:6789/0 conn(0x7fba280d2450 legacy=0x7fba2810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2ce99700 1 -- 172.21.15.32:0/3215782495 >> v1:172.21.15.32:6789/0 conn(0x7fba280cffb0 legacy=0x7fba280d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.559 7fba2ce99700 1 -- 172.21.15.32:0/3215782495 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba280d91f0 con 0x7fba280d6c90 2021-04-09T18:17:40.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.560 7fba2ce99700 1 --2- 172.21.15.32:0/3215782495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba280d6c90 0x7fba280d8000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fba18006650 tx=0x7fba18009a00).ready entity=mon.1 client_cookie=728517e135b2c7c0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:40.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.560 7fba1ffff700 1 -- 172.21.15.32:0/3215782495 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba18009df0 con 0x7fba280d6c90 2021-04-09T18:17:40.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.560 7fba1ffff700 1 -- 172.21.15.32:0/3215782495 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba180009b0 con 0x7fba280d6c90 2021-04-09T18:17:40.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.560 7fba1ffff700 1 -- 172.21.15.32:0/3215782495 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba1800c2b0 con 0x7fba280d6c90 2021-04-09T18:17:40.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.560 7fba2f87e700 1 -- 172.21.15.32:0/3215782495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba280d6c90 msgr2=0x7fba280d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.560 7fba2f87e700 1 --2- 172.21.15.32:0/3215782495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba280d6c90 0x7fba280d8000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fba18006650 tx=0x7fba18009a00).stop 2021-04-09T18:17:40.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.561 7fba2f87e700 1 -- 172.21.15.32:0/3215782495 shutdown_connections 2021-04-09T18:17:40.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.561 7fba2f87e700 1 --2- 172.21.15.32:0/3215782495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba280d6c90 0x7fba280d8000 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.561 7fba2f87e700 1 -- 172.21.15.32:0/3215782495 >> 172.21.15.32:0/3215782495 conn(0x7fba280ce5d0 msgr2=0x7fba280cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:40.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.561 7fba2f87e700 1 -- 172.21.15.32:0/3215782495 shutdown_connections 2021-04-09T18:17:40.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.561 7fba2f87e700 1 -- 172.21.15.32:0/3215782495 wait complete. 2021-04-09T18:17:40.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.581 7fba2f87e700 1 Processor -- start 2021-04-09T18:17:40.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.581 7fba2f87e700 1 -- start start 2021-04-09T18:17:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.581 7fba2f87e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba281a7dc0 0x7fba281a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.581 7fba2f87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba28184080 0x7fba28184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.581 7fba2f87e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba28184aa0 0x7fba2808f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.581 7fba2f87e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba281a8b40 con 0x7fba28184080 2021-04-09T18:17:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2f87e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba281a7870 con 0x7fba281a7dc0 2021-04-09T18:17:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2f87e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba281a87e0 con 0x7fba28184aa0 2021-04-09T18:17:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2ce99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba28184080 0x7fba28184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2ce99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba28184080 0x7fba28184480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45592/0 (socket says 172.21.15.32:45592) 2021-04-09T18:17:40.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2ce99700 1 -- 172.21.15.32:0/4100712359 learned_addr learned my addr 172.21.15.32:0/4100712359 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2d69a700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba281a7dc0 0x7fba281a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2de9b700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba28184aa0 0x7fba2808f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2ce99700 1 -- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba28184aa0 msgr2=0x7fba2808f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2ce99700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba28184aa0 0x7fba2808f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2ce99700 1 -- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba281a7dc0 msgr2=0x7fba281a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2ce99700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba281a7dc0 0x7fba281a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:40.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2ce99700 1 -- 172.21.15.32:0/4100712359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2808f810 con 0x7fba28184080 2021-04-09T18:17:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2ce99700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba28184080 0x7fba28184480 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fba1800b580 tx=0x7fba1800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba1dffb700 1 -- 172.21.15.32:0/4100712359 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba18000c90 con 0x7fba28184080 2021-04-09T18:17:40.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba1dffb700 1 -- 172.21.15.32:0/4100712359 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba18002030 con 0x7fba28184080 2021-04-09T18:17:40.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2808f810 con 0x7fba28184080 2021-04-09T18:17:40.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba1dffb700 1 -- 172.21.15.32:0/4100712359 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba18002480 con 0x7fba28184080 2021-04-09T18:17:40.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.582 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2808f810 con 0x7fba28184080 2021-04-09T18:17:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.583 7fba1dffb700 1 -- 172.21.15.32:0/4100712359 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fba18002210 con 0x7fba28184080 2021-04-09T18:17:40.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.583 7fba1dffb700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba08018930 0x7fba0801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.583 7fba1dffb700 1 -- 172.21.15.32:0/4100712359 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fba18032be0 con 0x7fba28184080 2021-04-09T18:17:40.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.583 7fba2d69a700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba08018930 0x7fba0801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:40.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.584 7fba2d69a700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba08018930 0x7fba0801adc0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:40.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.603 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba28182e80 con 0x7fba28184080 2021-04-09T18:17:40.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.607 7fba1dffb700 1 -- 172.21.15.32:0/4100712359 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fba18002240 con 0x7fba28184080 2021-04-09T18:17:40.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:40.789 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fba280d7290 con 0x7fba28184080 2021-04-09T18:17:41.401 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:17:41.368 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:17:41.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.373 7fba1dffb700 1 -- 172.21.15.32:0/4100712359 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v72) v1 ==== 161+0+0 (secure 0 0 0) 0x7fba18002030 con 0x7fba28184080 2021-04-09T18:17:41.437 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:17:41.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.406 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba08018930 msgr2=0x7fba0801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:41.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.406 7fba2f87e700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba08018930 0x7fba0801adc0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba28184080 msgr2=0x7fba28184480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:41.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba28184080 0x7fba28184480 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fba1800b580 tx=0x7fba1800b920).stop 2021-04-09T18:17:41.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 shutdown_connections 2021-04-09T18:17:41.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba08018930 0x7fba0801adc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba28184080 0x7fba28184480 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba28184aa0 0x7fba2808f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 --2- 172.21.15.32:0/4100712359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba281a7dc0 0x7fba281a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 >> 172.21.15.32:0/4100712359 conn(0x7fba281a8f50 msgr2=0x7fba28188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:41.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 shutdown_connections 2021-04-09T18:17:41.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.407 7fba2f87e700 1 -- 172.21.15.32:0/4100712359 wait complete. 2021-04-09T18:17:41.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:41.467 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:17:41.467 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:17:41.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.570 7f372751a700 1 Processor -- start 2021-04-09T18:17:41.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.570 7f372751a700 1 -- start start 2021-04-09T18:17:41.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f372751a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37200cffa0 0x7f37200d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f372751a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37200d7800 0x7f37200d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f372751a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37200d7280 con 0x7f37200d8220 2021-04-09T18:17:41.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f372751a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37200d91e0 con 0x7f37200cffa0 2021-04-09T18:17:41.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f372751a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372010dfe0 con 0x7f37200d7800 2021-04-09T18:17:41.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3725336700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37200cffa0 0x7f37200d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:41.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3725b37700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f37200d8220 0x7f37200d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55208/0 (socket says 172.21.15.32:55208) 2021-04-09T18:17:41.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3725b37700 1 -- 172.21.15.32:0/3197734855 learned_addr learned my addr 172.21.15.32:0/3197734855 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3724b35700 1 --2- 172.21.15.32:0/3197734855 >> v2:172.21.15.86:3300/0 conn(0x7f37200d7800 0x7f37200d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:41.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3725336700 1 -- 172.21.15.32:0/3197734855 >> v2:172.21.15.86:3300/0 conn(0x7f37200d7800 msgr2=0x7f37200d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:41.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3725336700 1 --2- 172.21.15.32:0/3197734855 >> v2:172.21.15.86:3300/0 conn(0x7f37200d7800 0x7f37200d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3725336700 1 -- 172.21.15.32:0/3197734855 >> v1:172.21.15.32:6789/0 conn(0x7f37200d8220 legacy=0x7f37200d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:41.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3717fff700 1 -- 172.21.15.32:0/3197734855 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158045982 0 0) 0x7f3718000d30 con 0x7f37200d8220 2021-04-09T18:17:41.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3725336700 1 -- 172.21.15.32:0/3197734855 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372010e200 con 0x7f37200cffa0 2021-04-09T18:17:41.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.571 7f3724b35700 1 --2- 172.21.15.32:0/3197734855 >> v2:172.21.15.86:3300/0 conn(0x7f37200d7800 0x7f37200d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:41.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.572 7f3725336700 1 --2- 172.21.15.32:0/3197734855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37200cffa0 0x7f37200d2440 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f371c006650 tx=0x7f371c009a80).ready entity=mon.0 client_cookie=6b0b987ae062ba9a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:41.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.572 7f3717fff700 1 -- 172.21.15.32:0/3197734855 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f371c0009b0 con 0x7f37200cffa0 2021-04-09T18:17:41.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.572 7f3717fff700 1 -- 172.21.15.32:0/3197734855 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c006740 con 0x7f37200cffa0 2021-04-09T18:17:41.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.572 7f3717fff700 1 -- 172.21.15.32:0/3197734855 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f371c00c0c0 con 0x7f37200cffa0 2021-04-09T18:17:41.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.572 7f372751a700 1 -- 172.21.15.32:0/3197734855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37200cffa0 msgr2=0x7f37200d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:41.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.572 7f372751a700 1 --2- 172.21.15.32:0/3197734855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37200cffa0 0x7f37200d2440 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f371c006650 tx=0x7f371c009a80).stop 2021-04-09T18:17:41.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.573 7f372751a700 1 -- 172.21.15.32:0/3197734855 shutdown_connections 2021-04-09T18:17:41.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.573 7f372751a700 1 --2- 172.21.15.32:0/3197734855 >> v2:172.21.15.86:3300/0 conn(0x7f37200d7800 0x7f37200d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.573 7f372751a700 1 --2- 172.21.15.32:0/3197734855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37200cffa0 0x7f37200d2440 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.573 7f372751a700 1 -- 172.21.15.32:0/3197734855 >> 172.21.15.32:0/3197734855 conn(0x7f37200ce5c0 msgr2=0x7f37200cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:41.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.573 7f372751a700 1 -- 172.21.15.32:0/3197734855 shutdown_connections 2021-04-09T18:17:41.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.573 7f372751a700 1 -- 172.21.15.32:0/3197734855 wait complete. 2021-04-09T18:17:41.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.593 7f372751a700 1 Processor -- start 2021-04-09T18:17:41.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.593 7f372751a700 1 -- start start 2021-04-09T18:17:41.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f372751a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37201acff0 0x7f372008fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:41.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f372751a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720090150 0x7f372008c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:41.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f372751a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f372008cca0 0x7f372008e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:41.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f372751a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372008c100 con 0x7f37201acff0 2021-04-09T18:17:41.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f372751a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372008d0a0 con 0x7f372008cca0 2021-04-09T18:17:41.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f372751a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372008e960 con 0x7f3720090150 2021-04-09T18:17:41.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725336700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37201acff0 0x7f372008fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:41.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725336700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37201acff0 0x7f372008fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45622/0 (socket says 172.21.15.32:45622) 2021-04-09T18:17:41.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725336700 1 -- 172.21.15.32:0/776788394 learned_addr learned my addr 172.21.15.32:0/776788394 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:41.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725b37700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f372008cca0 0x7f372008e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:41.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3724b35700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720090150 0x7f372008c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:41.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725336700 1 -- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720090150 msgr2=0x7f372008c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:41.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725336700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720090150 0x7f372008c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725336700 1 -- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f372008cca0 msgr2=0x7f372008e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:41.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725336700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f372008cca0 0x7f372008e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725336700 1 -- 172.21.15.32:0/776788394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372008eb80 con 0x7f37201acff0 2021-04-09T18:17:41.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725b37700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f372008cca0 0x7f372008e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:41.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3724b35700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720090150 0x7f372008c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:41.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3725336700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37201acff0 0x7f372008fb30 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f371c009650 tx=0x7f371c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:41.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3715ffb700 1 -- 172.21.15.32:0/776788394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f371c0093a0 con 0x7f37201acff0 2021-04-09T18:17:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3715ffb700 1 -- 172.21.15.32:0/776788394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c00bd50 con 0x7f37201acff0 2021-04-09T18:17:41.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f3715ffb700 1 -- 172.21.15.32:0/776788394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f371c0052e0 con 0x7f37201acff0 2021-04-09T18:17:41.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f372751a700 1 -- 172.21.15.32:0/776788394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372008eb80 con 0x7f37201acff0 2021-04-09T18:17:41.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.594 7f372751a700 1 -- 172.21.15.32:0/776788394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372008eb80 con 0x7f37201acff0 2021-04-09T18:17:41.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.595 7f3715ffb700 1 -- 172.21.15.32:0/776788394 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f371c006030 con 0x7f37201acff0 2021-04-09T18:17:41.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.595 7f3715ffb700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3700018930 0x7f370001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:41.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.595 7f3715ffb700 1 -- 172.21.15.32:0/776788394 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f371c032a90 con 0x7f37201acff0 2021-04-09T18:17:41.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.596 7f3724b35700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3700018930 0x7f370001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:41.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.596 7f3724b35700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3700018930 0x7f370001adc0 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:41.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.615 7f372751a700 1 -- 172.21.15.32:0/776788394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3720182dc0 con 0x7f37201acff0 2021-04-09T18:17:41.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.619 7f3715ffb700 1 -- 172.21.15.32:0/776788394 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f371c0009c0 con 0x7f37201acff0 2021-04-09T18:17:41.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.809 7f372751a700 1 -- 172.21.15.32:0/776788394 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37200d7280 con 0x7f37201acff0 2021-04-09T18:17:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.810 7f3715ffb700 1 -- 172.21.15.32:0/776788394 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f371c00bd50 con 0x7f37201acff0 2021-04-09T18:17:41.845 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 72 2021-04-09T18:17:41.846 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:41.846 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":72,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2614834008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2614834008},{"type":"v1","addr":"172.21.15.86:6837","nonce":2614834008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":72,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:16:48.944470","modified":"2021-04-09 18:17:41.368152","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-04-09T18:17:41.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.812 7f372751a700 1 -- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3700018930 msgr2=0x7f370001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.812 7f372751a700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3700018930 0x7f370001adc0 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.812 7f372751a700 1 -- 172.21.15.32:0/776788394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37201acff0 msgr2=0x7f372008fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.812 7f372751a700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37201acff0 0x7f372008fb30 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f371c009650 tx=0x7f371c00b960).stop 2021-04-09T18:17:41.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.813 7f372751a700 1 -- 172.21.15.32:0/776788394 shutdown_connections 2021-04-09T18:17:41.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.813 7f372751a700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3700018930 0x7f370001adc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.813 7f372751a700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f372008cca0 0x7f372008e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.813 7f372751a700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3720090150 0x7f372008c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.813 7f372751a700 1 --2- 172.21.15.32:0/776788394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37201acff0 0x7f372008fb30 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.813 7f372751a700 1 -- 172.21.15.32:0/776788394 >> 172.21.15.32:0/776788394 conn(0x7f372009abb0 msgr2=0x7f37200a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.813 7f372751a700 1 -- 172.21.15.32:0/776788394 shutdown_connections 2021-04-09T18:17:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:41.813 7f372751a700 1 -- 172.21.15.32:0/776788394 wait complete. 2021-04-09T18:17:41.874 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:17:42.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.063 7fd012fe7700 1 Processor -- start 2021-04-09T18:17:42.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.063 7fd012fe7700 1 -- start start 2021-04-09T18:17:42.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.063 7fd012fe7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd00c10e7e0 0x7fd00c0d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.063 7fd012fe7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd00c0d7290 con 0x7fd00c0d6c90 2021-04-09T18:17:42.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.063 7fd012fe7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd00c0d2450 con 0x7fd00c0cffb0 2021-04-09T18:17:42.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.063 7fd012fe7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00c0d7090 con 0x7fd00c10e7e0 2021-04-09T18:17:42.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.063 7fd00bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd00c0d6c90 0x7fd00c10cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55222/0 (socket says 172.21.15.32:55222) 2021-04-09T18:17:42.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.063 7fd00bfff700 1 -- 172.21.15.32:0/3373001842 learned_addr learned my addr 172.21.15.32:0/3373001842 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:42.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd011604700 1 --2- 172.21.15.32:0/3373001842 >> v2:172.21.15.86:3300/0 conn(0x7fd00c10e7e0 0x7fd00c0d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd00b7fe700 1 -- 172.21.15.32:0/3373001842 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786490080 0 0) 0x7fd000000ce0 con 0x7fd00c0d6c90 2021-04-09T18:17:42.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd00b7fe700 1 -- 172.21.15.32:0/3373001842 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff4002fe0 con 0x7fd00c0d6c90 2021-04-09T18:17:42.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd00b7fe700 1 -- 172.21.15.32:0/3373001842 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378629236 0 0) 0x7fcffc000c60 con 0x7fd00c0cffb0 2021-04-09T18:17:42.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd00b7fe700 1 -- 172.21.15.32:0/3373001842 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff4003210 con 0x7fd00c0cffb0 2021-04-09T18:17:42.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd00b7fe700 1 -- 172.21.15.32:0/3373001842 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1034716515 0 0) 0x7fd0000010e0 con 0x7fd00c0d6c90 2021-04-09T18:17:42.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd011604700 1 -- 172.21.15.32:0/3373001842 >> v1:172.21.15.86:6789/0 conn(0x7fd00c0cffb0 legacy=0x7fd00c0d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:42.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd011604700 1 -- 172.21.15.32:0/3373001842 >> v1:172.21.15.32:6789/0 conn(0x7fd00c0d6c90 legacy=0x7fd00c10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:42.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd011604700 1 -- 172.21.15.32:0/3373001842 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c0d91f0 con 0x7fd00c10e7e0 2021-04-09T18:17:42.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd011604700 1 --2- 172.21.15.32:0/3373001842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd00c10e7e0 0x7fd00c0d8000 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fd004006e50 tx=0x7fd004009e70).ready entity=mon.1 client_cookie=47c42daf221711cc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:42.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd00b7fe700 1 -- 172.21.15.32:0/3373001842 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd004006a20 con 0x7fd00c10e7e0 2021-04-09T18:17:42.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd00b7fe700 1 -- 172.21.15.32:0/3373001842 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd004006790 con 0x7fd00c10e7e0 2021-04-09T18:17:42.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd00b7fe700 1 -- 172.21.15.32:0/3373001842 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd00400b700 con 0x7fd00c10e7e0 2021-04-09T18:17:42.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd012fe7700 1 -- 172.21.15.32:0/3373001842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd00c10e7e0 msgr2=0x7fd00c0d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:42.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.064 7fd012fe7700 1 --2- 172.21.15.32:0/3373001842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd00c10e7e0 0x7fd00c0d8000 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fd004006e50 tx=0x7fd004009e70).stop 2021-04-09T18:17:42.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.065 7fd012fe7700 1 -- 172.21.15.32:0/3373001842 shutdown_connections 2021-04-09T18:17:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.065 7fd012fe7700 1 --2- 172.21.15.32:0/3373001842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd00c10e7e0 0x7fd00c0d8000 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.065 7fd012fe7700 1 -- 172.21.15.32:0/3373001842 >> 172.21.15.32:0/3373001842 conn(0x7fd00c0ce5d0 msgr2=0x7fd00c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.065 7fd012fe7700 1 -- 172.21.15.32:0/3373001842 shutdown_connections 2021-04-09T18:17:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.065 7fd012fe7700 1 -- 172.21.15.32:0/3373001842 wait complete. 2021-04-09T18:17:42.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.086 7fd012fe7700 1 Processor -- start 2021-04-09T18:17:42.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.086 7fd012fe7700 1 -- start start 2021-04-09T18:17:42.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.086 7fd012fe7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd00c190100 0x7fd00c190d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.086 7fd012fe7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd00c1913a0 0x7fd00c0b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.086 7fd012fe7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00c0b8330 0x7fd00c0b8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.086 7fd012fe7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c190500 con 0x7fd00c0b8330 2021-04-09T18:17:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.086 7fd012fe7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c1917a0 con 0x7fd00c1913a0 2021-04-09T18:17:42.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.086 7fd012fe7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c193ab0 con 0x7fd00c190100 2021-04-09T18:17:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.086 7fd011604700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00c0b8330 0x7fd00c0b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd011604700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00c0b8330 0x7fd00c0b8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45640/0 (socket says 172.21.15.32:45640) 2021-04-09T18:17:42.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd011604700 1 -- 172.21.15.32:0/2314214096 learned_addr learned my addr 172.21.15.32:0/2314214096 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:42.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd00bfff700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd00c1913a0 0x7fd00c0b7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd010e03700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd00c190100 0x7fd00c190d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd011604700 1 -- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd00c190100 msgr2=0x7fd00c190d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:42.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd011604700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd00c190100 0x7fd00c190d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd011604700 1 -- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd00c1913a0 msgr2=0x7fd00c0b7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:42.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd011604700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd00c1913a0 0x7fd00c0b7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd011604700 1 -- 172.21.15.32:0/2314214096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c1ac2e0 con 0x7fd00c0b8330 2021-04-09T18:17:42.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd011604700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00c0b8330 0x7fd00c0b8730 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7fd0040099f0 tx=0x7fd00400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:42.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd0097fa700 1 -- 172.21.15.32:0/2314214096 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0040093d0 con 0x7fd00c0b8330 2021-04-09T18:17:42.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd0097fa700 1 -- 172.21.15.32:0/2314214096 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0040090c0 con 0x7fd00c0b8330 2021-04-09T18:17:42.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00c1ac2e0 con 0x7fd00c0b8330 2021-04-09T18:17:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd0097fa700 1 -- 172.21.15.32:0/2314214096 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd004001420 con 0x7fd00c0b8330 2021-04-09T18:17:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.087 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00c1ac2e0 con 0x7fd00c0b8330 2021-04-09T18:17:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.088 7fd0097fa700 1 -- 172.21.15.32:0/2314214096 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd0040056b0 con 0x7fd00c0b8330 2021-04-09T18:17:42.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.088 7fd0097fa700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfec018930 0x7fcfec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.088 7fd0097fa700 1 -- 172.21.15.32:0/2314214096 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fd004032f90 con 0x7fd00c0b8330 2021-04-09T18:17:42.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.088 7fd010e03700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfec018930 0x7fcfec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.088 7fd010e03700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfec018930 0x7fcfec01ae10 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:42.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.107 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00c182e80 con 0x7fd00c0b8330 2021-04-09T18:17:42.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.111 7fd0097fa700 1 -- 172.21.15.32:0/2314214096 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd0040090a0 con 0x7fd00c0b8330 2021-04-09T18:17:42.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.309 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd00c0d7290 con 0x7fd00c0b8330 2021-04-09T18:17:42.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.375 7fd0097fa700 1 -- 172.21.15.32:0/2314214096 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v73) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd0040090a0 con 0x7fd00c0b8330 2021-04-09T18:17:42.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.424 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfec018930 msgr2=0x7fcfec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:42.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.424 7fd012fe7700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfec018930 0x7fcfec01ae10 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.424 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00c0b8330 msgr2=0x7fd00c0b8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:42.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.424 7fd012fe7700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00c0b8330 0x7fd00c0b8730 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7fd0040099f0 tx=0x7fd00400bd10).stop 2021-04-09T18:17:42.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.425 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 shutdown_connections 2021-04-09T18:17:42.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.425 7fd012fe7700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfec018930 0x7fcfec01ae10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.425 7fd012fe7700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00c0b8330 0x7fd00c0b8730 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.425 7fd012fe7700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd00c1913a0 0x7fd00c0b7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.425 7fd012fe7700 1 --2- 172.21.15.32:0/2314214096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd00c190100 0x7fd00c190d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.425 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 >> 172.21.15.32:0/2314214096 conn(0x7fd00c184a80 msgr2=0x7fd00c1a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:42.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.425 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 shutdown_connections 2021-04-09T18:17:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.425 7fd012fe7700 1 -- 172.21.15.32:0/2314214096 wait complete. 2021-04-09T18:17:42.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:17:42.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.613 7f04db46c700 1 Processor -- start 2021-04-09T18:17:42.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.613 7f04db46c700 1 -- start start 2021-04-09T18:17:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04db46c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f04d410d050 0x7f04d40cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04db46c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d40d84b0 0x7f04d410da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04db46c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d410d500 con 0x7f04d40d0160 2021-04-09T18:17:42.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04db46c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d40d8d70 con 0x7f04d410d050 2021-04-09T18:17:42.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04db46c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40d9260 con 0x7f04d40d84b0 2021-04-09T18:17:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04d8a87700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f04d40d0160 0x7f04d40d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55238/0 (socket says 172.21.15.32:55238) 2021-04-09T18:17:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04d8a87700 1 -- 172.21.15.32:0/2047132087 learned_addr learned my addr 172.21.15.32:0/2047132087 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04d9288700 1 --2- 172.21.15.32:0/2047132087 >> v2:172.21.15.86:3300/0 conn(0x7f04d410d050 0x7f04d40cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04d9a89700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d40d84b0 0x7f04d410da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.614 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676518749 0 0) 0x7f04c8000ce0 con 0x7f04d40d0160 2021-04-09T18:17:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c0002fe0 con 0x7f04d40d0160 2021-04-09T18:17:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1281565984 0 0) 0x7f04c80010e0 con 0x7f04d40d0160 2021-04-09T18:17:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d40d84b0 msgr2=0x7f04d410da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:42.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d40d84b0 0x7f04d410da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 >> v2:172.21.15.86:3300/0 conn(0x7f04d410d050 msgr2=0x7f04d40cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:42.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 --2- 172.21.15.32:0/2047132087 >> v2:172.21.15.86:3300/0 conn(0x7f04d410d050 0x7f04d40cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d410e0a0 con 0x7f04d40d0160 2021-04-09T18:17:42.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f04c8001600 con 0x7f04d40d0160 2021-04-09T18:17:42.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 >> v1:172.21.15.32:6789/0 conn(0x7f04d40d0160 legacy=0x7f04d40d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:42.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04c0003af0 0x7f04c0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04c004a850 0x7f04c004cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c004d5c0 0x7f04c004fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:42.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c0050390 con 0x7f04c004d5c0 2021-04-09T18:17:42.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00505b0 con 0x7f04c0003af0 2021-04-09T18:17:42.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00507d0 con 0x7f04c004a850 2021-04-09T18:17:42.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d9288700 1 --2- 172.21.15.32:0/2047132087 >> v2:172.21.15.86:3300/0 conn(0x7f04d410d050 0x7f04d40cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:42.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d9a89700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c004d5c0 0x7f04c004fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d8a87700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04c004a850 0x7f04c004cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d9288700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04c0003af0 0x7f04c0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:42.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d9a89700 1 -- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04c004a850 msgr2=0x7f04c004cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:42.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d9a89700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04c004a850 0x7f04c004cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d9a89700 1 -- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04c0003af0 msgr2=0x7f04c0005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:42.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d9a89700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04c0003af0 0x7f04c0005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:42.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d9a89700 1 -- 172.21.15.32:0/2047132087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c0050a70 con 0x7f04c004d5c0 2021-04-09T18:17:42.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.615 7f04d9a89700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c004d5c0 0x7f04c004fae0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f04cc009a40 tx=0x7f04cc009de0).ready entity=mon.0 client_cookie=4af6d3467977795f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:42.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.616 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04cc006a90 con 0x7f04c004d5c0 2021-04-09T18:17:42.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.616 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc003030 con 0x7f04c004d5c0 2021-04-09T18:17:42.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.616 7f04c7fff700 1 -- 172.21.15.32:0/2047132087 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04cc00b790 con 0x7f04c004d5c0 2021-04-09T18:17:42.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.616 7f04db46c700 1 -- 172.21.15.32:0/2047132087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c004d5c0 msgr2=0x7f04c004fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:42.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.616 7f04db46c700 1 --2- 172.21.15.32:0/2047132087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04c004d5c0 0x7f04c004fae0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f04cc009a40 tx=0x7f04cc009de0).stop 2021-04-09T18:17:42.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.616 7f04d9a89700 1 -- 172.21.15.32:0/2047132087 reap_dead start 2021-04-09T18:17:42.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.616 7f04db46c700 1 -- 172.21.15.32:0/2047132087 shutdown_connections 2021-04-09T18:17:42.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.616 7f04db46c700 1 -- 172.21.15.32:0/2047132087 >> 172.21.15.32:0/2047132087 conn(0x7f04d40ce620 msgr2=0x7f04d40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:42.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.617 7f04db46c700 1 -- 172.21.15.32:0/2047132087 shutdown_connections 2021-04-09T18:17:42.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.617 7f04db46c700 1 -- 172.21.15.32:0/2047132087 wait complete. 2021-04-09T18:17:42.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.637 7f04db46c700 1 Processor -- start 2021-04-09T18:17:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.637 7f04db46c700 1 -- start start 2021-04-09T18:17:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.637 7f04db46c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d409b2f0 0x7f04d409b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.637 7f04db46c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04d409a8b0 0x7f04d409e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.637 7f04db46c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04d409e9c0 0x7f04d409edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.637 7f04db46c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40bde80 con 0x7f04d409a8b0 2021-04-09T18:17:43.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.637 7f04db46c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d409ad70 con 0x7f04d409e9c0 2021-04-09T18:17:43.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.637 7f04db46c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d409bd10 con 0x7f04d409b2f0 2021-04-09T18:17:43.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d8a87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04d409a8b0 0x7f04d409e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d8a87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04d409a8b0 0x7f04d409e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45658/0 (socket says 172.21.15.32:45658) 2021-04-09T18:17:43.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d9288700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d409b2f0 0x7f04d409b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d8a87700 1 -- 172.21.15.32:0/1422096209 learned_addr learned my addr 172.21.15.32:0/1422096209 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d9a89700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04d409e9c0 0x7f04d409edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d8a87700 1 -- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d409b2f0 msgr2=0x7f04d409b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d8a87700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d409b2f0 0x7f04d409b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d8a87700 1 -- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04d409e9c0 msgr2=0x7f04d409edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d8a87700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04d409e9c0 0x7f04d409edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d8a87700 1 -- 172.21.15.32:0/1422096209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d409fbd0 con 0x7f04d409a8b0 2021-04-09T18:17:43.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d9288700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d409b2f0 0x7f04d409b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04d8a87700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04d409a8b0 0x7f04d409e3a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f04c8000e80 tx=0x7f04c800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04c5ffb700 1 -- 172.21.15.32:0/1422096209 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c80070d0 con 0x7f04d409a8b0 2021-04-09T18:17:43.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04c5ffb700 1 -- 172.21.15.32:0/1422096209 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c8001a10 con 0x7f04d409a8b0 2021-04-09T18:17:43.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04db46c700 1 -- 172.21.15.32:0/1422096209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d409fbd0 con 0x7f04d409a8b0 2021-04-09T18:17:43.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04c5ffb700 1 -- 172.21.15.32:0/1422096209 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c8004830 con 0x7f04d409a8b0 2021-04-09T18:17:43.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.638 7f04db46c700 1 -- 172.21.15.32:0/1422096209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d409fbd0 con 0x7f04d409a8b0 2021-04-09T18:17:43.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.639 7f04c5ffb700 1 -- 172.21.15.32:0/1422096209 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f04c80043a0 con 0x7f04d409a8b0 2021-04-09T18:17:43.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.639 7f04c5ffb700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04b4018930 0x7f04b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.639 7f04c5ffb700 1 -- 172.21.15.32:0/1422096209 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f04c8032a80 con 0x7f04d409a8b0 2021-04-09T18:17:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.639 7f04d9288700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04b4018930 0x7f04b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.639 7f04d9288700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04b4018930 0x7f04b401ae10 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.658 7f04db46c700 1 -- 172.21.15.32:0/1422096209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d40b9040 con 0x7f04d409a8b0 2021-04-09T18:17:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.662 7f04c5ffb700 1 -- 172.21.15.32:0/1422096209 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f04c8004700 con 0x7f04d409a8b0 2021-04-09T18:17:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:42.844 7f04db46c700 1 -- 172.21.15.32:0/1422096209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f04d410d500 con 0x7f04d409a8b0 2021-04-09T18:17:43.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.376 7f04c5ffb700 1 -- 172.21.15.32:0/1422096209 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v65) v1 ==== 185+0+0 (secure 0 0 0) 0x7f04c8004700 con 0x7f04d409a8b0 2021-04-09T18:17:43.443 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:17:43.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 -- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04b4018930 msgr2=0x7f04b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:43.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04b4018930 0x7f04b401ae10 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 -- 172.21.15.32:0/1422096209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04d409a8b0 msgr2=0x7f04d409e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04d409a8b0 0x7f04d409e3a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f04c8000e80 tx=0x7f04c800a930).stop 2021-04-09T18:17:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 -- 172.21.15.32:0/1422096209 shutdown_connections 2021-04-09T18:17:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04b4018930 0x7f04b401ae10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04d409a8b0 0x7f04d409e3a0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04d409e9c0 0x7f04d409edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 --2- 172.21.15.32:0/1422096209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04d409b2f0 0x7f04d409b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 -- 172.21.15.32:0/1422096209 >> 172.21.15.32:0/1422096209 conn(0x7f04d40ae6f0 msgr2=0x7f04d4097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 -- 172.21.15.32:0/1422096209 shutdown_connections 2021-04-09T18:17:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.411 7f04db46c700 1 -- 172.21.15.32:0/1422096209 wait complete. 2021-04-09T18:17:43.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:17:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.588 7ff301cf3700 1 Processor -- start 2021-04-09T18:17:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.588 7ff301cf3700 1 -- start start 2021-04-09T18:17:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.588 7ff301cf3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff2fc10d050 0x7ff2fc0cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.588 7ff301cf3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff2fc0d0160 0x7ff2fc0d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff301cf3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc10d500 con 0x7ff2fc0d82e0 2021-04-09T18:17:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff301cf3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2fc0d9260 con 0x7ff2fc10d050 2021-04-09T18:17:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff301cf3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2fc0d86e0 con 0x7ff2fc0d0160 2021-04-09T18:17:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fb7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff2fc10d050 0x7ff2fc0cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fbfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff2fc0d82e0 0x7ff2fc10da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55256/0 (socket says 172.21.15.32:55256) 2021-04-09T18:17:43.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fbfff700 1 -- 172.21.15.32:0/4133069952 learned_addr learned my addr 172.21.15.32:0/4133069952 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:43.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2faffd700 1 --2- 172.21.15.32:0/4133069952 >> v2:172.21.15.86:3300/0 conn(0x7ff2fc0d0160 0x7ff2fc0d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fb7fe700 1 -- 172.21.15.32:0/4133069952 >> v2:172.21.15.86:3300/0 conn(0x7ff2fc0d0160 msgr2=0x7ff2fc0d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fb7fe700 1 --2- 172.21.15.32:0/4133069952 >> v2:172.21.15.86:3300/0 conn(0x7ff2fc0d0160 0x7ff2fc0d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fb7fe700 1 -- 172.21.15.32:0/4133069952 >> v1:172.21.15.32:6789/0 conn(0x7ff2fc0d82e0 legacy=0x7ff2fc10da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fb7fe700 1 -- 172.21.15.32:0/4133069952 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc10e270 con 0x7ff2fc10d050 2021-04-09T18:17:43.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fa7fc700 1 -- 172.21.15.32:0/4133069952 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414243830 0 0) 0x7ff2f4000d30 con 0x7ff2fc0d82e0 2021-04-09T18:17:43.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fb7fe700 1 --2- 172.21.15.32:0/4133069952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc10d050 0x7ff2fc0cfb40 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7ff2ec006650 tx=0x7ff2ec009a00).ready entity=mon.0 client_cookie=d7c94963e0c0d0bd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:43.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fa7fc700 1 -- 172.21.15.32:0/4133069952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2ec009df0 con 0x7ff2fc10d050 2021-04-09T18:17:43.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fa7fc700 1 -- 172.21.15.32:0/4133069952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec0009b0 con 0x7ff2fc10d050 2021-04-09T18:17:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff2fa7fc700 1 -- 172.21.15.32:0/4133069952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2ec00c2b0 con 0x7ff2fc10d050 2021-04-09T18:17:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff301cf3700 1 -- 172.21.15.32:0/4133069952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc10d050 msgr2=0x7ff2fc0cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.589 7ff301cf3700 1 --2- 172.21.15.32:0/4133069952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc10d050 0x7ff2fc0cfb40 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7ff2ec006650 tx=0x7ff2ec009a00).stop 2021-04-09T18:17:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.590 7ff301cf3700 1 -- 172.21.15.32:0/4133069952 shutdown_connections 2021-04-09T18:17:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.590 7ff301cf3700 1 --2- 172.21.15.32:0/4133069952 >> v2:172.21.15.86:3300/0 conn(0x7ff2fc0d0160 0x7ff2fc0d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.590 7ff301cf3700 1 --2- 172.21.15.32:0/4133069952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc10d050 0x7ff2fc0cfb40 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.590 7ff301cf3700 1 -- 172.21.15.32:0/4133069952 >> 172.21.15.32:0/4133069952 conn(0x7ff2fc0ce620 msgr2=0x7ff2fc0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.590 7ff301cf3700 1 -- 172.21.15.32:0/4133069952 shutdown_connections 2021-04-09T18:17:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.590 7ff301cf3700 1 -- 172.21.15.32:0/4133069952 wait complete. 2021-04-09T18:17:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.610 7ff301cf3700 1 Processor -- start 2021-04-09T18:17:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.610 7ff301cf3700 1 -- start start 2021-04-09T18:17:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff301cf3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2fc0ad600 0x7ff2fc09ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff301cf3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc09b1b0 0x7ff2fc0ba220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff301cf3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2fc0ba840 0x7ff2fc0a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff301cf3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc09fa80 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff301cf3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc09b5b0 con 0x7ff2fc0ad600 2021-04-09T18:17:43.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff301cf3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc0bac40 con 0x7ff2fc0ba840 2021-04-09T18:17:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc09b1b0 0x7ff2fc0ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc09b1b0 0x7ff2fc0ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45672/0 (socket says 172.21.15.32:45672) 2021-04-09T18:17:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2faffd700 1 -- 172.21.15.32:0/910344039 learned_addr learned my addr 172.21.15.32:0/910344039 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2fb7fe700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2fc0ad600 0x7ff2fc09ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2fbfff700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2fc0ba840 0x7ff2fc0a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2faffd700 1 -- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2fc0ba840 msgr2=0x7ff2fc0a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:43.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2faffd700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2fc0ba840 0x7ff2fc0a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2faffd700 1 -- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2fc0ad600 msgr2=0x7ff2fc09ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:43.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2faffd700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2fc0ad600 0x7ff2fc09ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2faffd700 1 -- 172.21.15.32:0/910344039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc0a97c0 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2fb7fe700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2fc0ad600 0x7ff2fc09ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:43.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.611 7ff2fbfff700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2fc0ba840 0x7ff2fc0a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:43.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.612 7ff2faffd700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc09b1b0 0x7ff2fc0ba220 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7ff2f00067e0 tx=0x7ff2f000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:43.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.612 7ff2e3fff700 1 -- 172.21.15.32:0/910344039 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2f000bd80 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.612 7ff2e3fff700 1 -- 172.21.15.32:0/910344039 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f0000c90 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.612 7ff301cf3700 1 -- 172.21.15.32:0/910344039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc0a97c0 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.612 7ff2e3fff700 1 -- 172.21.15.32:0/910344039 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2f0002220 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.612 7ff301cf3700 1 -- 172.21.15.32:0/910344039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2fc0a97c0 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.612 7ff2e3fff700 1 -- 172.21.15.32:0/910344039 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff2f00068b0 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.613 7ff2e3fff700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2dc018930 0x7ff2dc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.613 7ff2e3fff700 1 -- 172.21.15.32:0/910344039 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5311+0+0 (secure 0 0 0) 0x7ff2f0032900 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.613 7ff2fb7fe700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2dc018930 0x7ff2dc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:43.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.613 7ff2fb7fe700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2dc018930 0x7ff2dc01ae10 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:43.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.633 7ff301cf3700 1 -- 172.21.15.32:0/910344039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2fc0aa330 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.636 7ff2e3fff700 1 -- 172.21.15.32:0/910344039 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff2f0000940 con 0x7ff2fc09b1b0 2021-04-09T18:17:43.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:43.821 7ff301cf3700 1 -- 172.21.15.32:0/910344039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff2fc0aa330 con 0x7ff2fc09b1b0 2021-04-09T18:17:44.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.380 7ff2e3fff700 1 -- 172.21.15.32:0/910344039 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v66) v1 ==== 173+0+0 (secure 0 0 0) 0x7ff2f00068b0 con 0x7ff2fc09b1b0 2021-04-09T18:17:44.420 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:17:44.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 -- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2dc018930 msgr2=0x7ff2dc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:44.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2dc018930 0x7ff2dc01ae10 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 -- 172.21.15.32:0/910344039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc09b1b0 msgr2=0x7ff2fc0ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc09b1b0 0x7ff2fc0ba220 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7ff2f00067e0 tx=0x7ff2f000b990).stop 2021-04-09T18:17:44.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 -- 172.21.15.32:0/910344039 shutdown_connections 2021-04-09T18:17:44.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2dc018930 0x7ff2dc01ae10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2fc09b1b0 0x7ff2fc0ba220 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2fc0ba840 0x7ff2fc0a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 --2- 172.21.15.32:0/910344039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2fc0ad600 0x7ff2fc09ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.388 7ff301cf3700 1 -- 172.21.15.32:0/910344039 >> 172.21.15.32:0/910344039 conn(0x7ff2fc0ae6f0 msgr2=0x7ff2fc097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:44.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.389 7ff301cf3700 1 -- 172.21.15.32:0/910344039 shutdown_connections 2021-04-09T18:17:44.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.389 7ff301cf3700 1 -- 172.21.15.32:0/910344039 wait complete. 2021-04-09T18:17:44.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:17:44.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.558 7fc88efde700 1 Processor -- start 2021-04-09T18:17:44.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.558 7fc88efde700 1 -- start start 2021-04-09T18:17:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.558 7fc88efde700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8880d6c90 0x7fc8880d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:44.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc88efde700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc8880d2450 0x7fc88810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:44.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc88efde700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8880d7290 con 0x7fc8880cffb0 2021-04-09T18:17:44.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc88efde700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8880d7090 con 0x7fc8880d6c90 2021-04-09T18:17:44.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc88efde700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8880d91f0 con 0x7fc8880d2450 2021-04-09T18:17:44.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc887fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8880d6c90 0x7fc8880d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:44.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc887fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc8880d6c90 0x7fc8880d8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45680/0 (socket says 172.21.15.32:45680) 2021-04-09T18:17:44.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc887fff700 1 -- 172.21.15.32:0/626626538 learned_addr learned my addr 172.21.15.32:0/626626538 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:44.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc88d5fb700 1 --2- 172.21.15.32:0/626626538 >> v2:172.21.15.86:3300/0 conn(0x7fc8880d2450 0x7fc88810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:44.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc887fff700 1 -- 172.21.15.32:0/626626538 >> v2:172.21.15.86:3300/0 conn(0x7fc8880d2450 msgr2=0x7fc88810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:44.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc887fff700 1 --2- 172.21.15.32:0/626626538 >> v2:172.21.15.86:3300/0 conn(0x7fc8880d2450 0x7fc88810cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc887fff700 1 -- 172.21.15.32:0/626626538 >> v1:172.21.15.86:6789/0 conn(0x7fc8880cffb0 legacy=0x7fc8880d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:44.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc887fff700 1 -- 172.21.15.32:0/626626538 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8880d2850 con 0x7fc8880d6c90 2021-04-09T18:17:44.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc887fff700 1 --2- 172.21.15.32:0/626626538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8880d6c90 0x7fc8880d8000 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7fc874006700 tx=0x7fc874009b80).ready entity=mon.0 client_cookie=3066bac48f453f3f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:44.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc8877fe700 1 -- 172.21.15.32:0/626626538 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc87400c0c0 con 0x7fc8880d6c90 2021-04-09T18:17:44.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc8877fe700 1 -- 172.21.15.32:0/626626538 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87400c440 con 0x7fc8880d6c90 2021-04-09T18:17:44.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc8877fe700 1 -- 172.21.15.32:0/626626538 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc87400c920 con 0x7fc8880d6c90 2021-04-09T18:17:44.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc88efde700 1 -- 172.21.15.32:0/626626538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8880d6c90 msgr2=0x7fc8880d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:44.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.559 7fc88efde700 1 --2- 172.21.15.32:0/626626538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8880d6c90 0x7fc8880d8000 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7fc874006700 tx=0x7fc874009b80).stop 2021-04-09T18:17:44.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.560 7fc88efde700 1 -- 172.21.15.32:0/626626538 shutdown_connections 2021-04-09T18:17:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.560 7fc88efde700 1 --2- 172.21.15.32:0/626626538 >> v2:172.21.15.86:3300/0 conn(0x7fc8880d2450 0x7fc88810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.560 7fc88efde700 1 --2- 172.21.15.32:0/626626538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8880d6c90 0x7fc8880d8000 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.560 7fc88efde700 1 -- 172.21.15.32:0/626626538 >> 172.21.15.32:0/626626538 conn(0x7fc8880ce5d0 msgr2=0x7fc8880cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.560 7fc88efde700 1 -- 172.21.15.32:0/626626538 shutdown_connections 2021-04-09T18:17:44.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.560 7fc88efde700 1 -- 172.21.15.32:0/626626538 wait complete. 2021-04-09T18:17:44.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.580 7fc88efde700 1 Processor -- start 2021-04-09T18:17:44.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.580 7fc88efde700 1 -- start start 2021-04-09T18:17:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88efde700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc88808a620 0x7fc88808b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88efde700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc88808b6d0 0x7fc888180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88efde700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc888180c50 0x7fc888181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:44.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88efde700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88808e660 con 0x7fc88808a620 2021-04-09T18:17:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88efde700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88808ab30 con 0x7fc88808b6d0 2021-04-09T18:17:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88efde700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88808bad0 con 0x7fc888180c50 2021-04-09T18:17:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88cdfa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc88808a620 0x7fc88808b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:44.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88cdfa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc88808a620 0x7fc88808b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45686/0 (socket says 172.21.15.32:45686) 2021-04-09T18:17:44.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88cdfa700 1 -- 172.21.15.32:0/1208976871 learned_addr learned my addr 172.21.15.32:0/1208976871 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:44.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc887fff700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc88808b6d0 0x7fc888180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:44.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88d5fb700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc888180c50 0x7fc888181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:44.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88cdfa700 1 -- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc888180c50 msgr2=0x7fc888181050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:44.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88cdfa700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc888180c50 0x7fc888181050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88cdfa700 1 -- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc88808b6d0 msgr2=0x7fc888180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:44.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88cdfa700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc88808b6d0 0x7fc888180630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.581 7fc88cdfa700 1 -- 172.21.15.32:0/1208976871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88808e1f0 con 0x7fc88808a620 2021-04-09T18:17:44.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.582 7fc88cdfa700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc88808a620 0x7fc88808b0b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fc87c00b530 tx=0x7fc87c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:44.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.582 7fc8857fa700 1 -- 172.21.15.32:0/1208976871 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc87c00bcc0 con 0x7fc88808a620 2021-04-09T18:17:44.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.582 7fc8857fa700 1 -- 172.21.15.32:0/1208976871 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c006020 con 0x7fc88808a620 2021-04-09T18:17:44.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.582 7fc8857fa700 1 -- 172.21.15.32:0/1208976871 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc87c0054a0 con 0x7fc88808a620 2021-04-09T18:17:44.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.582 7fc88efde700 1 -- 172.21.15.32:0/1208976871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88808e1f0 con 0x7fc88808a620 2021-04-09T18:17:44.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.582 7fc88efde700 1 -- 172.21.15.32:0/1208976871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88808e1f0 con 0x7fc88808a620 2021-04-09T18:17:44.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.583 7fc8857fa700 1 -- 172.21.15.32:0/1208976871 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc87c00a000 con 0x7fc88808a620 2021-04-09T18:17:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.583 7fc8857fa700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8680188c0 0x7fc86801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.583 7fc8857fa700 1 -- 172.21.15.32:0/1208976871 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fc87c032530 con 0x7fc88808a620 2021-04-09T18:17:44.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.583 7fc887fff700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8680188c0 0x7fc86801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:44.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.584 7fc887fff700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8680188c0 0x7fc86801adc0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:44.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.603 7fc88efde700 1 -- 172.21.15.32:0/1208976871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8881b7180 con 0x7fc88808a620 2021-04-09T18:17:44.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.606 7fc8857fa700 1 -- 172.21.15.32:0/1208976871 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc87c01f4f0 con 0x7fc88808a620 2021-04-09T18:17:44.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.820 7fc88efde700 1 -- 172.21.15.32:0/1208976871 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc8880d7290 con 0x7fc88808a620 2021-04-09T18:17:44.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.823 7fc8857fa700 1 -- 172.21.15.32:0/1208976871 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc87c00a000 con 0x7fc88808a620 2021-04-09T18:17:44.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 -- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8680188c0 msgr2=0x7fc86801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:44.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8680188c0 0x7fc86801adc0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 -- 172.21.15.32:0/1208976871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc88808a620 msgr2=0x7fc88808b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:44.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc88808a620 0x7fc88808b0b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fc87c00b530 tx=0x7fc87c00b8d0).stop 2021-04-09T18:17:44.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 -- 172.21.15.32:0/1208976871 shutdown_connections 2021-04-09T18:17:44.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8680188c0 0x7fc86801adc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc888180c50 0x7fc888181050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc88808b6d0 0x7fc888180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 --2- 172.21.15.32:0/1208976871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc88808a620 0x7fc88808b0b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:44.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 -- 172.21.15.32:0/1208976871 >> 172.21.15.32:0/1208976871 conn(0x7fc888189800 msgr2=0x7fc8881b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:44.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 -- 172.21.15.32:0/1208976871 shutdown_connections 2021-04-09T18:17:44.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.830 7fc88efde700 1 -- 172.21.15.32:0/1208976871 wait complete. 2021-04-09T18:17:44.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2021-04-09T18:17:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.988 7f5f40955700 1 Processor -- start 2021-04-09T18:17:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.988 7f5f40955700 1 -- start start 2021-04-09T18:17:45.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.989 7f5f40955700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c10d010 0x7f5f3c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.989 7f5f40955700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f3c0d1da0 0x7f5f3c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.989 7f5f40955700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3c10d480 con 0x7f5f3c0d82c0 2021-04-09T18:17:45.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.989 7f5f40955700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3c10e780 con 0x7f5f3c0d1da0 2021-04-09T18:17:45.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.989 7f5f40955700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c0d8d30 con 0x7f5f3c10d010 2021-04-09T18:17:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.989 7f5f39e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f3c0d1da0 0x7f5f3c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.989 7f5f3ae1d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5f3c0d82c0 0x7f5f3c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55284/0 (socket says 172.21.15.32:55284) 2021-04-09T18:17:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.989 7f5f3ae1d700 1 -- 172.21.15.32:0/461277310 learned_addr learned my addr 172.21.15.32:0/461277310 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f3a61c700 1 --2- 172.21.15.32:0/461277310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c10d010 0x7f5f3c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f39e1b700 1 -- 172.21.15.32:0/461277310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c10d010 msgr2=0x7f5f3c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:45.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f39e1b700 1 --2- 172.21.15.32:0/461277310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c10d010 0x7f5f3c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f39e1b700 1 -- 172.21.15.32:0/461277310 >> v1:172.21.15.32:6789/0 conn(0x7f5f3c0d82c0 legacy=0x7f5f3c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f39e1b700 1 -- 172.21.15.32:0/461277310 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c0d9220 con 0x7f5f3c0d1da0 2021-04-09T18:17:45.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f3961a700 1 -- 172.21.15.32:0/461277310 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830011656 0 0) 0x7f5f34000d30 con 0x7f5f3c0d82c0 2021-04-09T18:17:45.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f39e1b700 1 --2- 172.21.15.32:0/461277310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c0d1da0 0x7f5f3c0d7ca0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f5f28006650 tx=0x7f5f28009a00).ready entity=mon.0 client_cookie=c868caab8e03b622 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:45.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f3961a700 1 -- 172.21.15.32:0/461277310 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f28009df0 con 0x7f5f3c0d1da0 2021-04-09T18:17:45.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f3961a700 1 -- 172.21.15.32:0/461277310 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f280009b0 con 0x7f5f3c0d1da0 2021-04-09T18:17:45.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f3961a700 1 -- 172.21.15.32:0/461277310 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f2800c2b0 con 0x7f5f3c0d1da0 2021-04-09T18:17:45.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f40955700 1 -- 172.21.15.32:0/461277310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c0d1da0 msgr2=0x7f5f3c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.990 7f5f40955700 1 --2- 172.21.15.32:0/461277310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c0d1da0 0x7f5f3c0d7ca0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f5f28006650 tx=0x7f5f28009a00).stop 2021-04-09T18:17:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.991 7f5f40955700 1 -- 172.21.15.32:0/461277310 shutdown_connections 2021-04-09T18:17:45.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.991 7f5f40955700 1 --2- 172.21.15.32:0/461277310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c0d1da0 0x7f5f3c0d7ca0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.991 7f5f40955700 1 --2- 172.21.15.32:0/461277310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c10d010 0x7f5f3c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.991 7f5f40955700 1 -- 172.21.15.32:0/461277310 >> 172.21.15.32:0/461277310 conn(0x7f5f3c0ce5f0 msgr2=0x7f5f3c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:45.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.991 7f5f40955700 1 -- 172.21.15.32:0/461277310 shutdown_connections 2021-04-09T18:17:45.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:44.991 7f5f40955700 1 -- 172.21.15.32:0/461277310 wait complete. 2021-04-09T18:17:45.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.011 7f5f40955700 1 Processor -- start 2021-04-09T18:17:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.011 7f5f40955700 1 -- start start 2021-04-09T18:17:45.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f40955700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c1a5210 0x7f5f3c08eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f40955700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f3c1a8ab0 0x7f5f3c1a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f40955700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c1a76d0 0x7f5f3c1a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f40955700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c17ea50 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f40955700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c08f4f0 con 0x7f5f3c1a8ab0 2021-04-09T18:17:45.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f40955700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c08ea60 con 0x7f5f3c1a5210 2021-04-09T18:17:45.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3ae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c1a76d0 0x7f5f3c1a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3ae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c1a76d0 0x7f5f3c1a7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45700/0 (socket says 172.21.15.32:45700) 2021-04-09T18:17:45.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3ae1d700 1 -- 172.21.15.32:0/1858290009 learned_addr learned my addr 172.21.15.32:0/1858290009 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:45.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f39e1b700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f3c1a8ab0 0x7f5f3c1a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3a61c700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c1a5210 0x7f5f3c08eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3ae1d700 1 -- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c1a5210 msgr2=0x7f5f3c08eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3ae1d700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c1a5210 0x7f5f3c08eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3ae1d700 1 -- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f3c1a8ab0 msgr2=0x7f5f3c1a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3ae1d700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f3c1a8ab0 0x7f5f3c1a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3ae1d700 1 -- 172.21.15.32:0/1858290009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c1a80f0 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f3ae1d700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c1a76d0 0x7f5f3c1a7ad0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f5f34007f90 tx=0x7f5f3400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:45.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.012 7f5f32ffd700 1 -- 172.21.15.32:0/1858290009 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f34000cc0 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.013 7f5f32ffd700 1 -- 172.21.15.32:0/1858290009 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f34007020 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.013 7f5f32ffd700 1 -- 172.21.15.32:0/1858290009 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f34007500 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.013 7f5f40955700 1 -- 172.21.15.32:0/1858290009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3c1a80f0 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.013 7f5f40955700 1 -- 172.21.15.32:0/1858290009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3c1a80f0 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.013 7f5f32ffd700 1 -- 172.21.15.32:0/1858290009 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5f34007200 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.013 7f5f32ffd700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f180188c0 0x7f5f1801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.013 7f5f32ffd700 1 -- 172.21.15.32:0/1858290009 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f5f34032af0 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.013 7f5f3a61c700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f180188c0 0x7f5f1801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.014 7f5f3a61c700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f180188c0 0x7f5f1801adc0 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:45.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.033 7f5f40955700 1 -- 172.21.15.32:0/1858290009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3c1b7070 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.037 7f5f32ffd700 1 -- 172.21.15.32:0/1858290009 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5f34007d00 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.227 7f5f40955700 1 -- 172.21.15.32:0/1858290009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7f5f3c10d480 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.382 7f5f32ffd700 1 -- 172.21.15.32:0/1858290009 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v238) v1 ==== 139+0+0 (secure 0 0 0) 0x7f5f34007020 con 0x7f5f3c1a76d0 2021-04-09T18:17:45.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 -- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f180188c0 msgr2=0x7f5f1801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f180188c0 0x7f5f1801adc0 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 -- 172.21.15.32:0/1858290009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c1a76d0 msgr2=0x7f5f3c1a7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c1a76d0 0x7f5f3c1a7ad0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f5f34007f90 tx=0x7f5f3400ac50).stop 2021-04-09T18:17:45.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 -- 172.21.15.32:0/1858290009 shutdown_connections 2021-04-09T18:17:45.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f180188c0 0x7f5f1801adc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3c1a76d0 0x7f5f3c1a7ad0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f3c1a8ab0 0x7f5f3c1a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 --2- 172.21.15.32:0/1858290009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f3c1a5210 0x7f5f3c08eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 -- 172.21.15.32:0/1858290009 >> 172.21.15.32:0/1858290009 conn(0x7f5f3c1a99c0 msgr2=0x7f5f3c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:45.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 -- 172.21.15.32:0/1858290009 shutdown_connections 2021-04-09T18:17:45.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.393 7f5f40955700 1 -- 172.21.15.32:0/1858290009 wait complete. 2021-04-09T18:17:45.438 INFO:tasks.cephfs_test_runner:test_down_twice (tasks.cephfs.test_failover.TestClusterResize) ... ok 2021-04-09T18:17:45.439 INFO:tasks.cephfs_test_runner:Starting test: test_grow (tasks.cephfs.test_failover.TestClusterResize) 2021-04-09T18:17:45.440 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_grow' 2021-04-09T18:17:45.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.562 7fb4f9f7a700 1 Processor -- start 2021-04-09T18:17:45.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.562 7fb4f9f7a700 1 -- start start 2021-04-09T18:17:45.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f9f7a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb4f410d010 0x7fb4f40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f9f7a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f40d8490 0x7fb4f410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f9f7a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f410d480 con 0x7fb4f40d1da0 2021-04-09T18:17:45.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f9f7a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f40d8d30 con 0x7fb4f410d010 2021-04-09T18:17:45.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f9f7a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40d9220 con 0x7fb4f40d8490 2021-04-09T18:17:45.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f2ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb4f40d1da0 0x7fb4f40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55300/0 (socket says 172.21.15.32:55300) 2021-04-09T18:17:45.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f2ffd700 1 -- 172.21.15.32:0/3810120171 learned_addr learned my addr 172.21.15.32:0/3810120171 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f3fff700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f40d8490 0x7fb4f410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f37fe700 1 --2- 172.21.15.32:0/3810120171 >> v2:172.21.15.86:3300/0 conn(0x7fb4f410d010 0x7fb4f40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608886553 0 0) 0x7fb4dc0009b0 con 0x7fb4f40d1da0 2021-04-09T18:17:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d4002fe0 con 0x7fb4f40d1da0 2021-04-09T18:17:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 337623062 0 0) 0x7fb4dc001010 con 0x7fb4f40d1da0 2021-04-09T18:17:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f40d8490 msgr2=0x7fb4f410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f40d8490 0x7fb4f410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 >> v2:172.21.15.86:3300/0 conn(0x7fb4f410d010 msgr2=0x7fb4f40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:45.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 --2- 172.21.15.32:0/3810120171 >> v2:172.21.15.86:3300/0 conn(0x7fb4f410d010 0x7fb4f40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f410e020 con 0x7fb4f40d1da0 2021-04-09T18:17:45.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fb4dc001530 con 0x7fb4f40d1da0 2021-04-09T18:17:45.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 >> v1:172.21.15.32:6789/0 conn(0x7fb4f40d1da0 legacy=0x7fb4f40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.563 7fb4f27fc700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4d4003b90 0x7fb4d4006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f27fc700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4d404a8b0 0x7fb4d404cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f27fc700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d404d650 0x7fb4d404fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d4050420 con 0x7fb4d404a8b0 2021-04-09T18:17:45.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d4050640 con 0x7fb4d404d650 2021-04-09T18:17:45.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d4050860 con 0x7fb4d4003b90 2021-04-09T18:17:45.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f37fe700 1 --2- 172.21.15.32:0/3810120171 >> v2:172.21.15.86:3300/0 conn(0x7fb4f410d010 0x7fb4f40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:45.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f3fff700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f40d8490 0x7fb4f410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:45.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f2ffd700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4d404a8b0 0x7fb4d404cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f3fff700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d404d650 0x7fb4d404fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f37fe700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4d4003b90 0x7fb4d4006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f2ffd700 1 -- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4d4003b90 msgr2=0x7fb4d4006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:45.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f2ffd700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4d4003b90 0x7fb4d4006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f2ffd700 1 -- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d404d650 msgr2=0x7fb4d404fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:45.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f2ffd700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d404d650 0x7fb4d404fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f2ffd700 1 -- 172.21.15.32:0/3810120171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d4050b00 con 0x7fb4d404a8b0 2021-04-09T18:17:45.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f2ffd700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4d404a8b0 0x7fb4d404cdb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7fb4dc0017d0 tx=0x7fb4dc007d30).ready entity=mon.0 client_cookie=4c036f1f3adc5087 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:45.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4dc0013a0 con 0x7fb4d404a8b0 2021-04-09T18:17:45.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc00a310 con 0x7fb4d404a8b0 2021-04-09T18:17:45.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f27fc700 1 -- 172.21.15.32:0/3810120171 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4dc00a760 con 0x7fb4d404a8b0 2021-04-09T18:17:45.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f9f7a700 1 -- 172.21.15.32:0/3810120171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4d404a8b0 msgr2=0x7fb4d404cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f9f7a700 1 --2- 172.21.15.32:0/3810120171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4d404a8b0 0x7fb4d404cdb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7fb4dc0017d0 tx=0x7fb4dc007d30).stop 2021-04-09T18:17:45.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.564 7fb4f3fff700 1 -- 172.21.15.32:0/3810120171 reap_dead start 2021-04-09T18:17:45.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.565 7fb4f9f7a700 1 -- 172.21.15.32:0/3810120171 shutdown_connections 2021-04-09T18:17:45.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.565 7fb4f9f7a700 1 -- 172.21.15.32:0/3810120171 >> 172.21.15.32:0/3810120171 conn(0x7fb4f40ce5f0 msgr2=0x7fb4f40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:45.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.565 7fb4f9f7a700 1 -- 172.21.15.32:0/3810120171 shutdown_connections 2021-04-09T18:17:45.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.565 7fb4f9f7a700 1 -- 172.21.15.32:0/3810120171 wait complete. 2021-04-09T18:17:45.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.585 7fb4f9f7a700 1 Processor -- start 2021-04-09T18:17:45.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.585 7fb4f9f7a700 1 -- start start 2021-04-09T18:17:45.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f9f7a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f41aca20 0x7fb4f41ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f9f7a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4f418bd40 0x7fb4f418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f9f7a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f408fba0 0x7fb4f408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f9f7a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f418c7e0 con 0x7fb4f418bd40 2021-04-09T18:17:45.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f9f7a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f418ca30 con 0x7fb4f41aca20 2021-04-09T18:17:45.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f9f7a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40905f0 con 0x7fb4f408fba0 2021-04-09T18:17:45.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4f418bd40 0x7fb4f418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4f418bd40 0x7fb4f418c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45720/0 (socket says 172.21.15.32:45720) 2021-04-09T18:17:45.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f2ffd700 1 -- 172.21.15.32:0/259873105 learned_addr learned my addr 172.21.15.32:0/259873105 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f37fe700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f41aca20 0x7fb4f41ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f3fff700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f408fba0 0x7fb4f408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f2ffd700 1 -- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f408fba0 msgr2=0x7fb4f408ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f2ffd700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f408fba0 0x7fb4f408ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f2ffd700 1 -- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f41aca20 msgr2=0x7fb4f41ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f2ffd700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f41aca20 0x7fb4f41ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.586 7fb4f2ffd700 1 -- 172.21.15.32:0/259873105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40b4300 con 0x7fb4f418bd40 2021-04-09T18:17:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.587 7fb4f2ffd700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4f418bd40 0x7fb4f418c140 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fb4dc007fc0 tx=0x7fb4dc00a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.587 7fb4ebfff700 1 -- 172.21.15.32:0/259873105 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4dc0009b0 con 0x7fb4f418bd40 2021-04-09T18:17:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.587 7fb4ebfff700 1 -- 172.21.15.32:0/259873105 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc0075d0 con 0x7fb4f418bd40 2021-04-09T18:17:45.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.587 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f40b4300 con 0x7fb4f418bd40 2021-04-09T18:17:45.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.587 7fb4ebfff700 1 -- 172.21.15.32:0/259873105 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4dc004350 con 0x7fb4f418bd40 2021-04-09T18:17:45.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.587 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f40b4300 con 0x7fb4f418bd40 2021-04-09T18:17:45.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.587 7fb4ebfff700 1 -- 172.21.15.32:0/259873105 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb4dc004140 con 0x7fb4f418bd40 2021-04-09T18:17:45.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.588 7fb4ebfff700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb4cc018930 0x7fb4cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:45.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.588 7fb4ebfff700 1 -- 172.21.15.32:0/259873105 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fb4dc0327b0 con 0x7fb4f418bd40 2021-04-09T18:17:45.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.588 7fb4f37fe700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb4cc018930 0x7fb4cc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:45.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.588 7fb4f37fe700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb4cc018930 0x7fb4cc01adc0 crc :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.607 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f4182d70 con 0x7fb4f418bd40 2021-04-09T18:17:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.610 7fb4ebfff700 1 -- 172.21.15.32:0/259873105 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb4dc01f770 con 0x7fb4f418bd40 2021-04-09T18:17:45.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:45.796 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_grow"]} v 0) v1 -- 0x7fb4f418c760 con 0x7fb4f418bd40 2021-04-09T18:17:46.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.383 7fb4ebfff700 1 -- 172.21.15.32:0/259873105 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_grow"]}]=0 v239) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb4dc004140 con 0x7fb4f418bd40 2021-04-09T18:17:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb4cc018930 msgr2=0x7fb4cc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:46.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb4cc018930 0x7fb4cc01adc0 crc :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4f418bd40 msgr2=0x7fb4f418c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:46.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4f418bd40 0x7fb4f418c140 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fb4dc007fc0 tx=0x7fb4dc00a8a0).stop 2021-04-09T18:17:46.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 shutdown_connections 2021-04-09T18:17:46.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb4cc018930 0x7fb4cc01adc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4f418bd40 0x7fb4f418c140 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f408fba0 0x7fb4f408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 --2- 172.21.15.32:0/259873105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f41aca20 0x7fb4f41ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 >> 172.21.15.32:0/259873105 conn(0x7fb4f419b460 msgr2=0x7fb4f419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:46.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 shutdown_connections 2021-04-09T18:17:46.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.414 7fb4f9f7a700 1 -- 172.21.15.32:0/259873105 wait complete. 2021-04-09T18:17:46.460 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:17:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.587 7fbe4ce3d700 1 Processor -- start 2021-04-09T18:17:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.587 7fbe4ce3d700 1 -- start start 2021-04-09T18:17:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.588 7fbe4ce3d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbe4810d010 0x7fbe480d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.588 7fbe4ce3d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbe480d8490 0x7fbe4810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.588 7fbe4ce3d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4810d480 con 0x7fbe480d1da0 2021-04-09T18:17:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.588 7fbe4ce3d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe480d8d30 con 0x7fbe480d8490 2021-04-09T18:17:46.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.588 7fbe4ce3d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe480d9220 con 0x7fbe4810d010 2021-04-09T18:17:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.588 7fbe47314700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbe480d8490 0x7fbe4810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.588 7fbe47314700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbe480d8490 0x7fbe4810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45732/0 (socket says 172.21.15.32:45732) 2021-04-09T18:17:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.588 7fbe47314700 1 -- 172.21.15.32:0/1226098348 learned_addr learned my addr 172.21.15.32:0/1226098348 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:46.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe46b13700 1 --2- 172.21.15.32:0/1226098348 >> v2:172.21.15.86:3300/0 conn(0x7fbe4810d010 0x7fbe480d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe47314700 1 -- 172.21.15.32:0/1226098348 >> v2:172.21.15.86:3300/0 conn(0x7fbe4810d010 msgr2=0x7fbe480d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe47314700 1 --2- 172.21.15.32:0/1226098348 >> v2:172.21.15.86:3300/0 conn(0x7fbe4810d010 0x7fbe480d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe47314700 1 -- 172.21.15.32:0/1226098348 >> v1:172.21.15.86:6789/0 conn(0x7fbe480d1da0 legacy=0x7fbe480d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe47314700 1 -- 172.21.15.32:0/1226098348 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4810e020 con 0x7fbe480d8490 2021-04-09T18:17:46.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe47314700 1 --2- 172.21.15.32:0/1226098348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480d8490 0x7fbe4810da00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7fbe40006f00 tx=0x7fbe40009f10).ready entity=mon.0 client_cookie=f59ea517ae5acf05 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe45b11700 1 -- 172.21.15.32:0/1226098348 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe4000b3b0 con 0x7fbe480d8490 2021-04-09T18:17:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe45b11700 1 -- 172.21.15.32:0/1226098348 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4000b730 con 0x7fbe480d8490 2021-04-09T18:17:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe45b11700 1 -- 172.21.15.32:0/1226098348 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe4000bc10 con 0x7fbe480d8490 2021-04-09T18:17:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe4ce3d700 1 -- 172.21.15.32:0/1226098348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480d8490 msgr2=0x7fbe4810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.589 7fbe4ce3d700 1 --2- 172.21.15.32:0/1226098348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480d8490 0x7fbe4810da00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7fbe40006f00 tx=0x7fbe40009f10).stop 2021-04-09T18:17:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.590 7fbe4ce3d700 1 -- 172.21.15.32:0/1226098348 shutdown_connections 2021-04-09T18:17:46.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.590 7fbe4ce3d700 1 --2- 172.21.15.32:0/1226098348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480d8490 0x7fbe4810da00 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.590 7fbe4ce3d700 1 --2- 172.21.15.32:0/1226098348 >> v2:172.21.15.86:3300/0 conn(0x7fbe4810d010 0x7fbe480d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.590 7fbe4ce3d700 1 -- 172.21.15.32:0/1226098348 >> 172.21.15.32:0/1226098348 conn(0x7fbe480ce5f0 msgr2=0x7fbe480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:46.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.590 7fbe4ce3d700 1 -- 172.21.15.32:0/1226098348 shutdown_connections 2021-04-09T18:17:46.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.590 7fbe4ce3d700 1 -- 172.21.15.32:0/1226098348 wait complete. 2021-04-09T18:17:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.610 7fbe4ce3d700 1 Processor -- start 2021-04-09T18:17:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.610 7fbe4ce3d700 1 -- start start 2021-04-09T18:17:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe4ce3d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480c3c00 0x7fbe4809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe4ce3d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe48095240 0x7fbe481a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:46.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe4ce3d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481a1270 0x7fbe480b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:46.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe4ce3d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4809cda0 con 0x7fbe480c3c00 2021-04-09T18:17:46.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe4ce3d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe480c3680 con 0x7fbe481a1270 2021-04-09T18:17:46.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe4ce3d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48095640 con 0x7fbe48095240 2021-04-09T18:17:46.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480c3c00 0x7fbe4809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:46.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480c3c00 0x7fbe4809c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45736/0 (socket says 172.21.15.32:45736) 2021-04-09T18:17:46.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46b13700 1 -- 172.21.15.32:0/2038516340 learned_addr learned my addr 172.21.15.32:0/2038516340 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:46.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46312700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe48095240 0x7fbe481a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:46.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe47314700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481a1270 0x7fbe480b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:46.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46b13700 1 -- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe48095240 msgr2=0x7fbe481a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:46.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46b13700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe48095240 0x7fbe481a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46b13700 1 -- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481a1270 msgr2=0x7fbe480b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:46.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46b13700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481a1270 0x7fbe480b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46b13700 1 -- 172.21.15.32:0/2038516340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe481a1670 con 0x7fbe480c3c00 2021-04-09T18:17:46.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46312700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe48095240 0x7fbe481a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:46.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe47314700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481a1270 0x7fbe480b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:46.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe46b13700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480c3c00 0x7fbe4809c600 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7fbe38006f60 tx=0x7fbe3800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:46.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe377fe700 1 -- 172.21.15.32:0/2038516340 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe380009b0 con 0x7fbe480c3c00 2021-04-09T18:17:46.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe377fe700 1 -- 172.21.15.32:0/2038516340 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3800bd60 con 0x7fbe480c3c00 2021-04-09T18:17:46.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe481a1670 con 0x7fbe480c3c00 2021-04-09T18:17:46.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.611 7fbe377fe700 1 -- 172.21.15.32:0/2038516340 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe38002030 con 0x7fbe480c3c00 2021-04-09T18:17:46.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.612 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe481a1670 con 0x7fbe480c3c00 2021-04-09T18:17:46.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.612 7fbe377fe700 1 -- 172.21.15.32:0/2038516340 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbe38006b00 con 0x7fbe480c3c00 2021-04-09T18:17:46.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.612 7fbe377fe700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe2c018930 0x7fbe2c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:46.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.612 7fbe377fe700 1 -- 172.21.15.32:0/2038516340 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fbe380327a0 con 0x7fbe480c3c00 2021-04-09T18:17:46.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.612 7fbe46312700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe2c018930 0x7fbe2c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:46.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.613 7fbe46312700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe2c018930 0x7fbe2c01ae10 crc :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:46.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.632 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe48093d50 con 0x7fbe480c3c00 2021-04-09T18:17:46.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.636 7fbe377fe700 1 -- 172.21.15.32:0/2038516340 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbe38000c90 con 0x7fbe480c3c00 2021-04-09T18:17:46.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.849 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbe4810d480 con 0x7fbe480c3c00 2021-04-09T18:17:46.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.853 7fbe377fe700 1 -- 172.21.15.32:0/2038516340 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbe38006b00 con 0x7fbe480c3c00 2021-04-09T18:17:46.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe2c018930 msgr2=0x7fbe2c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:46.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe2c018930 0x7fbe2c01ae10 crc :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480c3c00 msgr2=0x7fbe4809c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:46.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480c3c00 0x7fbe4809c600 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7fbe38006f60 tx=0x7fbe3800b970).stop 2021-04-09T18:17:46.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 shutdown_connections 2021-04-09T18:17:46.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe2c018930 0x7fbe2c01ae10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481a1270 0x7fbe480b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe48095240 0x7fbe481a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 --2- 172.21.15.32:0/2038516340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe480c3c00 0x7fbe4809c600 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:46.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 >> 172.21.15.32:0/2038516340 conn(0x7fbe480975f0 msgr2=0x7fbe48096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:46.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 shutdown_connections 2021-04-09T18:17:46.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:46.859 7fbe4ce3d700 1 -- 172.21.15.32:0/2038516340 wait complete. 2021-04-09T18:17:46.912 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:17:46.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:17:47.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:17:47.064 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:17:47.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:17:47.066 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:17:47.067 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:17:47.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:17:47.120 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:17:47.121 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:17:47.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:17:47.123 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:17:47.124 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:17:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:17:47.169 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:17:47.170 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:17:47.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:17:47.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.286 7f2d9e42d700 1 Processor -- start 2021-04-09T18:17:47.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.286 7f2d9e42d700 1 -- start start 2021-04-09T18:17:47.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.286 7f2d9e42d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d9810d010 0x7f2d980d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:47.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.286 7f2d9e42d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2d980d1da0 0x7f2d980d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:47.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d9e42d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9810d480 con 0x7f2d980d82c0 2021-04-09T18:17:47.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d9e42d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d9810e780 con 0x7f2d980d1da0 2021-04-09T18:17:47.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d9e42d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d980d8d30 con 0x7f2d9810d010 2021-04-09T18:17:47.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d8f7fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2d980d1da0 0x7f2d980d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:47.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d97fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d9810d010 0x7f2d980d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:47.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d9ca4a700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2d980d82c0 0x7f2d9810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:53262/0 (socket says 172.21.15.32:53262) 2021-04-09T18:17:47.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d9ca4a700 1 -- 172.21.15.32:0/34626240 learned_addr learned my addr 172.21.15.32:0/34626240 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:47.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d977fe700 1 -- 172.21.15.32:0/34626240 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508847401 0 0) 0x7f2d90001440 con 0x7f2d980d82c0 2021-04-09T18:17:47.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d977fe700 1 -- 172.21.15.32:0/34626240 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d84002fe0 con 0x7f2d980d82c0 2021-04-09T18:17:47.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d8f7fe700 1 -- 172.21.15.32:0/34626240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d9810d010 msgr2=0x7f2d980d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:47.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d8f7fe700 1 --2- 172.21.15.32:0/34626240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d9810d010 0x7f2d980d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:47.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d8f7fe700 1 -- 172.21.15.32:0/34626240 >> v1:172.21.15.86:6789/0 conn(0x7f2d980d82c0 legacy=0x7f2d9810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:47.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d8f7fe700 1 -- 172.21.15.32:0/34626240 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980d9220 con 0x7f2d980d1da0 2021-04-09T18:17:47.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.287 7f2d97fff700 1 --2- 172.21.15.32:0/34626240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d9810d010 0x7f2d980d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:47.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d8f7fe700 1 --2- 172.21.15.32:0/34626240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d980d1da0 0x7f2d980d7ca0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f2d80006650 tx=0x7f2d80009a80).ready entity=mon.1 client_cookie=35d39a6b57306016 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:47.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d977fe700 1 -- 172.21.15.32:0/34626240 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d800009b0 con 0x7f2d980d1da0 2021-04-09T18:17:47.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d977fe700 1 -- 172.21.15.32:0/34626240 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8000c0c0 con 0x7f2d980d1da0 2021-04-09T18:17:47.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d977fe700 1 -- 172.21.15.32:0/34626240 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d8000c510 con 0x7f2d980d1da0 2021-04-09T18:17:47.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d9e42d700 1 -- 172.21.15.32:0/34626240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d980d1da0 msgr2=0x7f2d980d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:47.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d9e42d700 1 --2- 172.21.15.32:0/34626240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d980d1da0 0x7f2d980d7ca0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f2d80006650 tx=0x7f2d80009a80).stop 2021-04-09T18:17:47.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d9e42d700 1 -- 172.21.15.32:0/34626240 shutdown_connections 2021-04-09T18:17:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d9e42d700 1 --2- 172.21.15.32:0/34626240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d980d1da0 0x7f2d980d7ca0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d9e42d700 1 --2- 172.21.15.32:0/34626240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d9810d010 0x7f2d980d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d9e42d700 1 -- 172.21.15.32:0/34626240 >> 172.21.15.32:0/34626240 conn(0x7f2d980ce5f0 msgr2=0x7f2d980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:47.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d9e42d700 1 -- 172.21.15.32:0/34626240 shutdown_connections 2021-04-09T18:17:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.288 7f2d9e42d700 1 -- 172.21.15.32:0/34626240 wait complete. 2021-04-09T18:17:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.309 7f2d9e42d700 1 Processor -- start 2021-04-09T18:17:47.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.309 7f2d9e42d700 1 -- start start 2021-04-09T18:17:47.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.309 7f2d9e42d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d98179fd0 0x7f2d980976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:47.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.309 7f2d9e42d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d98097cc0 0x7f2d981a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:47.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.309 7f2d9e42d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d981a4b10 0x7f2d981a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.309 7f2d9e42d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98098140 con 0x7f2d98179fd0 2021-04-09T18:17:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.309 7f2d9e42d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9817a3d0 con 0x7f2d981a4b10 2021-04-09T18:17:47.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.309 7f2d9e42d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9817c870 con 0x7f2d98097cc0 2021-04-09T18:17:47.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d97fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d98179fd0 0x7f2d980976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:47.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d97fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d98179fd0 0x7f2d980976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45754/0 (socket says 172.21.15.32:45754) 2021-04-09T18:17:47.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d97fff700 1 -- 172.21.15.32:0/3073381206 learned_addr learned my addr 172.21.15.32:0/3073381206 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:47.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d9ca4a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d981a4b10 0x7f2d981a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:47.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d8f7fe700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d98097cc0 0x7f2d981a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:47.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d97fff700 1 -- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d98097cc0 msgr2=0x7f2d981a44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:47.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d97fff700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d98097cc0 0x7f2d981a44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:47.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d97fff700 1 -- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d981a4b10 msgr2=0x7f2d981a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:47.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d97fff700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d981a4b10 0x7f2d981a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:47.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d97fff700 1 -- 172.21.15.32:0/3073381206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d98179bf0 con 0x7f2d98179fd0 2021-04-09T18:17:47.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d97fff700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d98179fd0 0x7f2d980976a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f2d88006f40 tx=0x7f2d8800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:47.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d957fa700 1 -- 172.21.15.32:0/3073381206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d8800bd80 con 0x7f2d98179fd0 2021-04-09T18:17:47.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d957fa700 1 -- 172.21.15.32:0/3073381206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d88000c90 con 0x7f2d98179fd0 2021-04-09T18:17:47.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d957fa700 1 -- 172.21.15.32:0/3073381206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d88002220 con 0x7f2d98179fd0 2021-04-09T18:17:47.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d98179bf0 con 0x7f2d98179fd0 2021-04-09T18:17:47.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.310 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d98179bf0 con 0x7f2d98179fd0 2021-04-09T18:17:47.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.311 7f2d957fa700 1 -- 172.21.15.32:0/3073381206 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2d8800a000 con 0x7f2d98179fd0 2021-04-09T18:17:47.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.311 7f2d957fa700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d7c018930 0x7f2d7c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:47.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.311 7f2d957fa700 1 -- 172.21.15.32:0/3073381206 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f2d880328e0 con 0x7f2d98179fd0 2021-04-09T18:17:47.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.311 7f2d8f7fe700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d7c018930 0x7f2d7c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:47.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.312 7f2d8f7fe700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d7c018930 0x7f2d7c01adc0 crc :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:47.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.331 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d981b7050 con 0x7f2d98179fd0 2021-04-09T18:17:47.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.334 7f2d957fa700 1 -- 172.21.15.32:0/3073381206 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2d88000c90 con 0x7f2d98179fd0 2021-04-09T18:17:47.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.525 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2d9810d480 con 0x7f2d98179fd0 2021-04-09T18:17:47.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.526 7f2d957fa700 1 -- 172.21.15.32:0/3073381206 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v66) v1 ==== 81+0+25681 (secure 0 0 0) 0x7f2d88006b00 con 0x7f2d98179fd0 2021-04-09T18:17:47.564 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:47.564 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:17:47.565 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 66, 2021-04-09T18:17:47.565 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:17:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:17:47.566 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:17:44.376834", 2021-04-09T18:17:47.567 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:17:47.576 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:17:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:17:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:17:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:17:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:17:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:17:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:17:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:17:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:17:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:17:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:17:47.583 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:17:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:17:47.584 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:17:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 11, 2021-04-09T18:17:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:17:47.585 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:17:47.586 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:17:47.604 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:17:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:17:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:47.605 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:17:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:17:47.606 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:17:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:17:47.607 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:17:47.625 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:17:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:17:47.630 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:17:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:17:47.640 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:17:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:17:47.641 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:17:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:17:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:17:47.642 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:17:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:17:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:17:47.643 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:17:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:17:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:17:47.644 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:17:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:17:47.645 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:17:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:17:47.676 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:17:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:17:47.682 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:17:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:17:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:17:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:17:47.683 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:17:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:17:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:17:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:17:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:17:47.684 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:17:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:17:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:17:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:17:47.685 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:17:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:17:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:17:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:17:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:17:47.686 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:17:47.687 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:17:47.740 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:17:47.745 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:17:47.747 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:17:47.747 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:17:47.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:47.748 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:17:47.749 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:17:47.749 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:17:47.750 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:17:47.750 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:17:47.750 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:17:47.751 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:17:47.776 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:17:47.788 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:17:47.788 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:17:47.789 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:17:47.789 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:17:47.799 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:17:47.799 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:17:47.800 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:47.800 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:47.800 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:17:47.801 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:17:47.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:47.801 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:17:47.802 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:17:47.802 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:47.820 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:47.820 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:47.821 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:47.821 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:47.849 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:47.849 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:47.850 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:47.850 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:47.851 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:47.851 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:47.869 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:47.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:47.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:47.870 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:17:47.898 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:47.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:47.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:47.920 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:47.927 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:17:47.928 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:47.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:47.928 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:47.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:47.929 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:47.930 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:47.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:47.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:17:48.022 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.024 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:17:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.026 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:48.027 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.075 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:17:48.076 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:48.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:17:48.090 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:48.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.092 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.092 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:48.093 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.102 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:17:48.103 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:48.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:17:48.109 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:17:48.109 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.109 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.110 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:17:48.115 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:17:48.115 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:17:48.119 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:17:48.119 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:48.119 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:48.120 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:48.120 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:17:48.228 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:17:48.232 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:48.246 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:48.246 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:48.246 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:48.246 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:48.247 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:48.247 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:48.247 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:48.247 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:48.248 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:48.248 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:48.248 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.249 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:17:48.249 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:48.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.250 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.250 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:17:48.250 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:48.251 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.251 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.290 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:48.290 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.291 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:17:48.291 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:48.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.297 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.297 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:17:48.298 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:48.302 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.302 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.303 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:48.303 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.304 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.308 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:17:48.308 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:48.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.309 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.310 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:17:48.310 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:48.311 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.311 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:48.344 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.345 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.346 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:17:48.346 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.349 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:17:48.353 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:17:48.354 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.357 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:17:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:17:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:17:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:17:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:17:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:17:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:17:48.440 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:48.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.442 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:17:48.451 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:48.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.452 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.453 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:48.453 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.480 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.484 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:17:48.484 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:48.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:17:48.520 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:48.520 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.520 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.521 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:48.522 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.522 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:17:48.523 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.570 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:17:48.570 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:48.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.571 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.602 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:48.603 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.607 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:17:48.607 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:48.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.608 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:17:48.619 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:17:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:17:48.624 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:17:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:17:48.625 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:17:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:48.626 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:48.627 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:17:48.643 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:17:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:48.651 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:48.652 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:48.653 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:17:48.654 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:48.667 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:48.668 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:17:48.673 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:48.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.700 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.700 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:17:48.701 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:48.701 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.704 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:48.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:17:48.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:48.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:17:48.710 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:48.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.711 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.732 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:48.733 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:17:48.750 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:48.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.773 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:17:48.774 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:48.774 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.777 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.778 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:48.779 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.780 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:17:48.780 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:48.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:17:48.808 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:17:48.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.817 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.818 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:17:48.826 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:17:48.827 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:17:48.827 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:17:48.828 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:48.828 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:48.834 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:48.835 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.846 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:17:48.846 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:17:48.846 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:48.847 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:48.847 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:48.854 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:48.855 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:48.855 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:48.856 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:48.856 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 58, 2021-04-09T18:17:48.857 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:48.857 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:48.858 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:48.858 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.869 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:17:48.870 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:48.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.881 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:17:48.882 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:48.882 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.883 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.886 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:48.886 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.893 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.893 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:17:48.894 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:48.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:17:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:48.909 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.910 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.911 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:48.911 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.912 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:17:48.913 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:48.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.934 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.936 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:17:48.937 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:48.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.937 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.946 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:48.946 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:48.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:48.949 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:17:48.949 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:48.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:48.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:48.963 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:17:48.963 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:17:48.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:48.964 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:48.965 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:17:48.966 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:17:48.966 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:17:48.966 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:17:48.967 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:48.988 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:48.989 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:48.989 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:48.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.006 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:17:49.011 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:17:49.011 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:49.020 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:49.025 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:49.026 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:49.026 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:49.027 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:49.027 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:49.027 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:49.028 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:49.028 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:49.029 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:49.029 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.051 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.052 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:17:49.052 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:49.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.053 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.054 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:17:49.054 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:49.055 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.055 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.056 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:49.056 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.063 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:17:49.063 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:49.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.064 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.065 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:17:49.065 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:49.065 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.066 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.066 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:49.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.134 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:17:49.134 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:49.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.144 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.144 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:17:49.145 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:49.145 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.146 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.147 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:49.147 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.148 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.148 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:17:49.149 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:49.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.150 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:17:49.151 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:17:49.182 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.182 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.183 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:17:49.183 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:17:49.184 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:17:49.184 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:17:49.185 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:49.207 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:49.216 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:49.217 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.251 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:17:49.251 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:17:49.252 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:49.253 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:49.254 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:49.254 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:49.254 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:49.255 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:49.255 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:49.256 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:49.256 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:49.256 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:49.257 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:49.257 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.259 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:17:49.295 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:49.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.297 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.297 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:17:49.297 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:49.298 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.298 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.299 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:49.299 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.300 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.301 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:17:49.301 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:49.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.302 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.303 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:17:49.359 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:49.360 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.360 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.360 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:49.360 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.361 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.361 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:17:49.361 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:49.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.372 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.372 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:17:49.372 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:49.372 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.372 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.373 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:49.373 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.374 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.374 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:17:49.374 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:49.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.375 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:17:49.375 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:17:49.375 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.376 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.376 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:17:49.376 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:17:49.376 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:17:49.377 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:17:49.377 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:49.377 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:49.377 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:49.377 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.378 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:17:49.378 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:17:49.379 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:17:49.379 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:17:49.408 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:17:49.409 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:17:49.409 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:17:49.414 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:17:49.415 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:17:49.415 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:17:49.415 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:17:49.415 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:17:49.416 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:17:49.416 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.416 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.417 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:17:49.417 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:49.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.418 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:17:49.419 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:49.419 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.419 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.445 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:17:49.445 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.447 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:17:49.447 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:49.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.454 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:17:49.458 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:49.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.459 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.463 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:17:49.464 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.469 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.469 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:17:49.474 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:49.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.475 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:17:49.476 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:49.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.476 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.477 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:17:49.477 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:17:49.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.477 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:17:49.477 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:17:49.478 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:49.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.478 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:17:49.478 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:17:49.503 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:17:49.503 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.510 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.515 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:17:49.515 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:17:49.516 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:17:49.516 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:17:49.516 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:17:49.517 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:17:49.517 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:17:49.517 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:49.533 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.534 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:17:49.535 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:17:49.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.536 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:17:49.538 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:49.539 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:49.542 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:49.543 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:49.544 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:49.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.552 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:17:49.552 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:49.555 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:49.555 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:49.556 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:49.558 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:49.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.560 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:17:49.560 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:49.560 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.563 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:17:49.585 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:49.586 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:49.586 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:49.587 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:49.587 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:49.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.588 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:17:49.588 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:49.589 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:49.589 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:49.589 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:49.595 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:49.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.601 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:17:49.601 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:49.602 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:49.602 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:49.602 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:49.603 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:49.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.604 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:17:49.604 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:49.615 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:49.616 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:49.616 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:49.621 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:49.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:49.622 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:17:49.622 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:17:49.623 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:17:49.623 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:17:49.624 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:17:49.624 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:17:49.624 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:49.625 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:17:49.625 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:17:49.640 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:17:49.648 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:17:49.649 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:17:49.649 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:17:49.666 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/3850456471": "2021-04-10 18:17:40.790342", 2021-04-09T18:17:49.667 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/3552289149": "2021-04-10 18:17:40.790326", 2021-04-09T18:17:49.667 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/3850456471": "2021-04-10 18:17:40.790342", 2021-04-09T18:17:49.668 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/3552289149": "2021-04-10 18:17:40.790326" 2021-04-09T18:17:49.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:49.668 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:17:49.669 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:17:49.677 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:17:49.678 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:17:49.678 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:17:49.678 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:17:50.029 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:17:50.030 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:17:50.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:50.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:50.031 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:17:50.031 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:17:50.032 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:17:50.032 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:17:50.033 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:17:50.033 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:17:50.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d7c018930 msgr2=0x7f2d7c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d7c018930 0x7f2d7c01adc0 crc :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d98179fd0 msgr2=0x7f2d980976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d98179fd0 0x7f2d980976a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f2d88006f40 tx=0x7f2d8800b990).stop 2021-04-09T18:17:50.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 shutdown_connections 2021-04-09T18:17:50.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d7c018930 0x7f2d7c01adc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d981a4b10 0x7f2d981a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d98097cc0 0x7f2d981a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 --2- 172.21.15.32:0/3073381206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d98179fd0 0x7f2d980976a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 >> 172.21.15.32:0/3073381206 conn(0x7f2d98199680 msgr2=0x7f2d981a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:50.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 shutdown_connections 2021-04-09T18:17:50.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:47.530 7f2d9e42d700 1 -- 172.21.15.32:0/3073381206 wait complete. 2021-04-09T18:17:50.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:50.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.172 7f127c13f700 1 Processor -- start 2021-04-09T18:17:50.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.172 7f127c13f700 1 -- start start 2021-04-09T18:17:50.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.173 7f127c13f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f127410d010 0x7f12740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.173 7f127c13f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12740d1da0 0x7f12740d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.173 7f127c13f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127410d480 con 0x7f12740d82c0 2021-04-09T18:17:50.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.173 7f127c13f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12740d9220 con 0x7f127410d010 2021-04-09T18:17:50.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.173 7f127c13f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12740d86c0 con 0x7f12740d1da0 2021-04-09T18:17:50.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.173 7f1279f5b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f127410d010 0x7f12740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.173 7f127a75c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f12740d82c0 0x7f127410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55352/0 (socket says 172.21.15.32:55352) 2021-04-09T18:17:50.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.173 7f1279f5b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f127410d010 0x7f12740d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45760/0 (socket says 172.21.15.32:45760) 2021-04-09T18:17:50.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.173 7f127a75c700 1 -- 172.21.15.32:0/1833197781 learned_addr learned my addr 172.21.15.32:0/1833197781 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:50.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f127975a700 1 --2- 172.21.15.32:0/1833197781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12740d1da0 0x7f12740d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1278f59700 1 -- 172.21.15.32:0/1833197781 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553694675 0 0) 0x7f126c000d30 con 0x7f12740d82c0 2021-04-09T18:17:50.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1278f59700 1 -- 172.21.15.32:0/1833197781 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1260002fe0 con 0x7f12740d82c0 2021-04-09T18:17:50.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1279f5b700 1 -- 172.21.15.32:0/1833197781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12740d1da0 msgr2=0x7f12740d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:50.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1279f5b700 1 --2- 172.21.15.32:0/1833197781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12740d1da0 0x7f12740d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1279f5b700 1 -- 172.21.15.32:0/1833197781 >> v1:172.21.15.32:6789/0 conn(0x7f12740d82c0 legacy=0x7f127410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1279f5b700 1 -- 172.21.15.32:0/1833197781 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127410e1f0 con 0x7f127410d010 2021-04-09T18:17:50.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1279f5b700 1 --2- 172.21.15.32:0/1833197781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127410d010 0x7f12740d1780 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f1270006650 tx=0x7f1270009a80).ready entity=mon.0 client_cookie=cf7cd4485ab9dbc7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:50.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1278f59700 1 -- 172.21.15.32:0/1833197781 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12700009b0 con 0x7f127410d010 2021-04-09T18:17:50.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1278f59700 1 -- 172.21.15.32:0/1833197781 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127000c0c0 con 0x7f127410d010 2021-04-09T18:17:50.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f1278f59700 1 -- 172.21.15.32:0/1833197781 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f127000c510 con 0x7f127410d010 2021-04-09T18:17:50.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f127c13f700 1 -- 172.21.15.32:0/1833197781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127410d010 msgr2=0x7f12740d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.174 7f127c13f700 1 --2- 172.21.15.32:0/1833197781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127410d010 0x7f12740d1780 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f1270006650 tx=0x7f1270009a80).stop 2021-04-09T18:17:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.175 7f127c13f700 1 -- 172.21.15.32:0/1833197781 shutdown_connections 2021-04-09T18:17:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.175 7f127c13f700 1 --2- 172.21.15.32:0/1833197781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12740d1da0 0x7f12740d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.175 7f127c13f700 1 --2- 172.21.15.32:0/1833197781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127410d010 0x7f12740d1780 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.175 7f127c13f700 1 -- 172.21.15.32:0/1833197781 >> 172.21.15.32:0/1833197781 conn(0x7f12740ce5f0 msgr2=0x7f12740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:50.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.175 7f127c13f700 1 -- 172.21.15.32:0/1833197781 shutdown_connections 2021-04-09T18:17:50.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.175 7f127c13f700 1 -- 172.21.15.32:0/1833197781 wait complete. 2021-04-09T18:17:50.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.195 7f127c13f700 1 Processor -- start 2021-04-09T18:17:50.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.195 7f127c13f700 1 -- start start 2021-04-09T18:17:50.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127c13f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1274179bf0 0x7f1274179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127c13f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127417c540 0x7f127417c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127c13f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12740aca60 0x7f12740ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127c13f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127417cf60 con 0x7f127417c540 2021-04-09T18:17:50.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127c13f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740ad480 con 0x7f1274179bf0 2021-04-09T18:17:50.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127c13f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12740ad6a0 con 0x7f12740aca60 2021-04-09T18:17:50.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127975a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127417c540 0x7f127417c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127975a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127417c540 0x7f127417c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45766/0 (socket says 172.21.15.32:45766) 2021-04-09T18:17:50.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127975a700 1 -- 172.21.15.32:0/2613241442 learned_addr learned my addr 172.21.15.32:0/2613241442 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:50.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f1279f5b700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1274179bf0 0x7f1274179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127975a700 1 -- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12740aca60 msgr2=0x7f12740ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127975a700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12740aca60 0x7f12740ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:17:50.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127975a700 1 -- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1274179bf0 msgr2=0x7f1274179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127975a700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1274179bf0 0x7f1274179ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127975a700 1 -- 172.21.15.32:0/2613241442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12740ad8c0 con 0x7f127417c540 2021-04-09T18:17:50.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127975a700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127417c540 0x7f127417c940 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f126400b530 tx=0x7f126400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:50.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f126a7fc700 1 -- 172.21.15.32:0/2613241442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f126400bcc0 con 0x7f127417c540 2021-04-09T18:17:50.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f126a7fc700 1 -- 172.21.15.32:0/2613241442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1264002030 con 0x7f127417c540 2021-04-09T18:17:50.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f127c13f700 1 -- 172.21.15.32:0/2613241442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12740ad8c0 con 0x7f127417c540 2021-04-09T18:17:50.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.196 7f126a7fc700 1 -- 172.21.15.32:0/2613241442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1264002480 con 0x7f127417c540 2021-04-09T18:17:50.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.197 7f127c13f700 1 -- 172.21.15.32:0/2613241442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12740ad8c0 con 0x7f127417c540 2021-04-09T18:17:50.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.197 7f126a7fc700 1 -- 172.21.15.32:0/2613241442 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1264002210 con 0x7f127417c540 2021-04-09T18:17:50.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.197 7f126a7fc700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f125c018930 0x7f125c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.197 7f126a7fc700 1 -- 172.21.15.32:0/2613241442 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f1264032770 con 0x7f127417c540 2021-04-09T18:17:50.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.198 7f1279f5b700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f125c018930 0x7f125c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.198 7f1279f5b700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f125c018930 0x7f125c01ae10 crc :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.217 7f127c13f700 1 -- 172.21.15.32:0/2613241442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12741b7010 con 0x7f127417c540 2021-04-09T18:17:50.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.221 7f126a7fc700 1 -- 172.21.15.32:0/2613241442 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1264002240 con 0x7f127417c540 2021-04-09T18:17:50.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.412 7f127c13f700 1 -- 172.21.15.32:0/2613241442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f127410d480 con 0x7f127417c540 2021-04-09T18:17:50.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.413 7f126a7fc700 1 -- 172.21.15.32:0/2613241442 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1853 (secure 0 0 0) 0x7f1264002030 con 0x7f127417c540 2021-04-09T18:17:50.486 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 74 2021-04-09T18:17:50.487 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:50.487 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6142,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/1922876620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1922876620},{"type":"v1","addr":"172.21.15.32:6834","nonce":1922876620}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":74},{"gid":6216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4113967866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4113967866},{"type":"v1","addr":"172.21.15.32:6835","nonce":4113967866}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":74},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2614834008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2614834008},{"type":"v1","addr":"172.21.15.86:6837","nonce":2614834008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":71}],"filesystems":[]} 2021-04-09T18:17:50.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.415 7f127c13f700 1 -- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f125c018930 msgr2=0x7f125c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.415 7f127c13f700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f125c018930 0x7f125c01ae10 crc :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.415 7f127c13f700 1 -- 172.21.15.32:0/2613241442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127417c540 msgr2=0x7f127417c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.415 7f127c13f700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127417c540 0x7f127417c940 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f126400b530 tx=0x7f126400b8d0).stop 2021-04-09T18:17:50.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.416 7f127c13f700 1 -- 172.21.15.32:0/2613241442 shutdown_connections 2021-04-09T18:17:50.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.416 7f127c13f700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f125c018930 0x7f125c01ae10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.416 7f127c13f700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f127417c540 0x7f127417c940 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.416 7f127c13f700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12740aca60 0x7f12740ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:17:50.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.416 7f127c13f700 1 --2- 172.21.15.32:0/2613241442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1274179bf0 0x7f1274179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.416 7f127c13f700 1 -- 172.21.15.32:0/2613241442 >> 172.21.15.32:0/2613241442 conn(0x7f12741acae0 msgr2=0x7f127417ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:50.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.416 7f127c13f700 1 -- 172.21.15.32:0/2613241442 shutdown_connections 2021-04-09T18:17:50.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.416 7f127c13f700 1 -- 172.21.15.32:0/2613241442 wait complete. 2021-04-09T18:17:50.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:50.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.646 7fe8f2a43700 1 Processor -- start 2021-04-09T18:17:50.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.646 7fe8f2a43700 1 -- start start 2021-04-09T18:17:50.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f2a43700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec10d010 0x7fe8ec0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f2a43700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8ec0d8490 0x7fe8ec10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f2a43700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ec10d480 con 0x7fe8ec0d1da0 2021-04-09T18:17:50.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f2a43700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8ec10e780 con 0x7fe8ec0d8490 2021-04-09T18:17:50.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f2a43700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec0d8d30 con 0x7fe8ec10d010 2021-04-09T18:17:50.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f1060700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8ec0d8490 0x7fe8ec10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f1060700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8ec0d8490 0x7fe8ec10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45776/0 (socket says 172.21.15.32:45776) 2021-04-09T18:17:50.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f1060700 1 -- 172.21.15.32:0/2102044439 learned_addr learned my addr 172.21.15.32:0/2102044439 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:50.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f085f700 1 --2- 172.21.15.32:0/2102044439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec10d010 0x7fe8ec0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.647 7fe8f1060700 1 -- 172.21.15.32:0/2102044439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec10d010 msgr2=0x7fe8ec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:50.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8f1060700 1 --2- 172.21.15.32:0/2102044439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec10d010 0x7fe8ec0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8f1060700 1 -- 172.21.15.32:0/2102044439 >> v1:172.21.15.86:6789/0 conn(0x7fe8ec0d1da0 legacy=0x7fe8ec0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8f1060700 1 -- 172.21.15.32:0/2102044439 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec0d9220 con 0x7fe8ec0d8490 2021-04-09T18:17:50.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8f085f700 1 --2- 172.21.15.32:0/2102044439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec10d010 0x7fe8ec0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:50.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8f1060700 1 --2- 172.21.15.32:0/2102044439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec0d8490 0x7fe8ec10da00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fe8e4006f00 tx=0x7fe8e4009f10).ready entity=mon.0 client_cookie=7ab3d7fd7f84ede2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:50.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8eb7fe700 1 -- 172.21.15.32:0/2102044439 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8e400b3b0 con 0x7fe8ec0d8490 2021-04-09T18:17:50.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8eb7fe700 1 -- 172.21.15.32:0/2102044439 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e400b730 con 0x7fe8ec0d8490 2021-04-09T18:17:50.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8eb7fe700 1 -- 172.21.15.32:0/2102044439 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8e400bc10 con 0x7fe8ec0d8490 2021-04-09T18:17:50.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8f2a43700 1 -- 172.21.15.32:0/2102044439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec0d8490 msgr2=0x7fe8ec10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.648 7fe8f2a43700 1 --2- 172.21.15.32:0/2102044439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec0d8490 0x7fe8ec10da00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fe8e4006f00 tx=0x7fe8e4009f10).stop 2021-04-09T18:17:50.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.649 7fe8f2a43700 1 -- 172.21.15.32:0/2102044439 shutdown_connections 2021-04-09T18:17:50.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.649 7fe8f2a43700 1 --2- 172.21.15.32:0/2102044439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec0d8490 0x7fe8ec10da00 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.649 7fe8f2a43700 1 --2- 172.21.15.32:0/2102044439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec10d010 0x7fe8ec0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.649 7fe8f2a43700 1 -- 172.21.15.32:0/2102044439 >> 172.21.15.32:0/2102044439 conn(0x7fe8ec0ce5f0 msgr2=0x7fe8ec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.649 7fe8f2a43700 1 -- 172.21.15.32:0/2102044439 shutdown_connections 2021-04-09T18:17:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.649 7fe8f2a43700 1 -- 172.21.15.32:0/2102044439 wait complete. 2021-04-09T18:17:50.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.669 7fe8f2a43700 1 Processor -- start 2021-04-09T18:17:50.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.669 7fe8f2a43700 1 -- start start 2021-04-09T18:17:50.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f2a43700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec1a5210 0x7fe8ec08eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f2a43700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec1a8ab0 0x7fe8ec1a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f2a43700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec1a76d0 0x7fe8ec1a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f2a43700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec17ea50 con 0x7fe8ec1a5210 2021-04-09T18:17:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f2a43700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec08f4f0 con 0x7fe8ec1a76d0 2021-04-09T18:17:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f2a43700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec08ea60 con 0x7fe8ec1a8ab0 2021-04-09T18:17:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f085f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec1a5210 0x7fe8ec08eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f085f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec1a5210 0x7fe8ec08eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45780/0 (socket says 172.21.15.32:45780) 2021-04-09T18:17:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f085f700 1 -- 172.21.15.32:0/2716122369 learned_addr learned my addr 172.21.15.32:0/2716122369 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f1060700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec1a76d0 0x7fe8ec1a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8ebfff700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec1a8ab0 0x7fe8ec1a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f085f700 1 -- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec1a8ab0 msgr2=0x7fe8ec1a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f085f700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec1a8ab0 0x7fe8ec1a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f085f700 1 -- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec1a76d0 msgr2=0x7fe8ec1a7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f085f700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec1a76d0 0x7fe8ec1a7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f085f700 1 -- 172.21.15.32:0/2716122369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec1a80f0 con 0x7fe8ec1a5210 2021-04-09T18:17:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f085f700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec1a5210 0x7fe8ec08eed0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fe8dc00b580 tx=0x7fe8dc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:50.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8e97fa700 1 -- 172.21.15.32:0/2716122369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc00bd10 con 0x7fe8ec1a5210 2021-04-09T18:17:50.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8e97fa700 1 -- 172.21.15.32:0/2716122369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc000c90 con 0x7fe8ec1a5210 2021-04-09T18:17:50.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.670 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ec1a80f0 con 0x7fe8ec1a5210 2021-04-09T18:17:50.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.671 7fe8e97fa700 1 -- 172.21.15.32:0/2716122369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc002220 con 0x7fe8ec1a5210 2021-04-09T18:17:50.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.671 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ec1a80f0 con 0x7fe8ec1a5210 2021-04-09T18:17:50.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.671 7fe8e97fa700 1 -- 172.21.15.32:0/2716122369 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe8dc00a000 con 0x7fe8ec1a5210 2021-04-09T18:17:50.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.671 7fe8e97fa700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8cc018930 0x7fe8cc01adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:50.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.671 7fe8e97fa700 1 -- 172.21.15.32:0/2716122369 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fe8dc032760 con 0x7fe8ec1a5210 2021-04-09T18:17:50.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.671 7fe8ebfff700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8cc018930 0x7fe8cc01adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:50.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.672 7fe8ebfff700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8cc018930 0x7fe8cc01adf0 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:50.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.691 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ec1b7070 con 0x7fe8ec1a5210 2021-04-09T18:17:50.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.694 7fe8e97fa700 1 -- 172.21.15.32:0/2716122369 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe8dc006bb0 con 0x7fe8ec1a5210 2021-04-09T18:17:50.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.886 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8ec1b7070 con 0x7fe8ec1a5210 2021-04-09T18:17:50.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.887 7fe8e97fa700 1 -- 172.21.15.32:0/2716122369 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1853 (secure 0 0 0) 0x7fe8dc000c90 con 0x7fe8ec1a5210 2021-04-09T18:17:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 74 2021-04-09T18:17:50.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:50.936 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6142,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/1922876620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1922876620},{"type":"v1","addr":"172.21.15.32:6834","nonce":1922876620}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":74},{"gid":6216,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4113967866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4113967866},{"type":"v1","addr":"172.21.15.32:6835","nonce":4113967866}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":74},{"gid":55253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1208639138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1208639138},{"type":"v1","addr":"172.21.15.86:6836","nonce":1208639138}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":52},{"gid":55469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2614834008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2614834008},{"type":"v1","addr":"172.21.15.86:6837","nonce":2614834008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":71}],"filesystems":[]} 2021-04-09T18:17:50.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8cc018930 msgr2=0x7fe8cc01adf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8cc018930 0x7fe8cc01adf0 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec1a5210 msgr2=0x7fe8ec08eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:50.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec1a5210 0x7fe8ec08eed0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fe8dc00b580 tx=0x7fe8dc00b920).stop 2021-04-09T18:17:50.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 shutdown_connections 2021-04-09T18:17:50.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8cc018930 0x7fe8cc01adf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec1a76d0 0x7fe8ec1a7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec1a8ab0 0x7fe8ec1a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 --2- 172.21.15.32:0/2716122369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec1a5210 0x7fe8ec08eed0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:50.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 >> 172.21.15.32:0/2716122369 conn(0x7fe8ec1a99c0 msgr2=0x7fe8ec17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:50.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 shutdown_connections 2021-04-09T18:17:50.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:50.889 7fe8f2a43700 1 -- 172.21.15.32:0/2716122369 wait complete. 2021-04-09T18:17:50.996 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:17:50.999 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:17:51.000 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:17:51.000 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:17:51.001 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:17:51.001 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:17:51.002 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:17:51.002 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:17:51.014 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:17:51.015 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:17:51.022 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:17:51.022 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:17:51.023 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:17:51.024 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:17:51.050 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:17:51.050 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:17:51.139 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:17:51.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:17:51.198 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:17:51.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:17:51.201 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:17:51.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:17:51.280 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:17:51.282 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:17:51.284 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:17:51.285 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:17:51.286 DEBUG:teuthology.parallel:result is None 2021-04-09T18:17:51.287 DEBUG:teuthology.parallel:result is None 2021-04-09T18:17:51.319 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:17:51.320 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:17:51.321 DEBUG:teuthology.parallel:result is None 2021-04-09T18:17:51.322 DEBUG:teuthology.parallel:result is None 2021-04-09T18:17:51.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:17:51.382 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:17:51.390 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:17:51.425 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:17:51.436 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:17:51.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.458 7ff815a66700 1 Processor -- start 2021-04-09T18:17:51.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.458 7ff815a66700 1 -- start start 2021-04-09T18:17:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.458 7ff815a66700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8100d6c80 0x7ff8100d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.458 7ff815a66700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff8100d2440 0x7ff81010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.458 7ff815a66700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8100d7280 con 0x7ff8100cffa0 2021-04-09T18:17:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.458 7ff815a66700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8100d7080 con 0x7ff8100d2440 2021-04-09T18:17:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.458 7ff815a66700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8100d91e0 con 0x7ff8100d6c80 2021-04-09T18:17:51.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80effd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8100d6c80 0x7ff8100d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:51.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80ffff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff8100d2440 0x7ff81010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:51.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80f7fe700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff8100cffa0 0x7ff8100d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:53328/0 (socket says 172.21.15.32:53328) 2021-04-09T18:17:51.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80f7fe700 1 -- 172.21.15.32:0/2336470110 learned_addr learned my addr 172.21.15.32:0/2336470110 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:51.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80e7fc700 1 -- 172.21.15.32:0/2336470110 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578830697 0 0) 0x7ff8000010c0 con 0x7ff8100cffa0 2021-04-09T18:17:51.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80effd700 1 -- 172.21.15.32:0/2336470110 >> v2:172.21.15.86:3300/0 conn(0x7ff8100d2440 msgr2=0x7ff81010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:51.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80effd700 1 --2- 172.21.15.32:0/2336470110 >> v2:172.21.15.86:3300/0 conn(0x7ff8100d2440 0x7ff81010cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:51.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80effd700 1 -- 172.21.15.32:0/2336470110 >> v1:172.21.15.86:6789/0 conn(0x7ff8100cffa0 legacy=0x7ff8100d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:51.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80effd700 1 -- 172.21.15.32:0/2336470110 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8100d2840 con 0x7ff8100d6c80 2021-04-09T18:17:51.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80effd700 1 --2- 172.21.15.32:0/2336470110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8100d6c80 0x7ff8100d7ff0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff804006650 tx=0x7ff804009a80).ready entity=mon.2 client_cookie=8a9ab9b3bdc2e21a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80e7fc700 1 -- 172.21.15.32:0/2336470110 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8040009b0 con 0x7ff8100d6c80 2021-04-09T18:17:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80e7fc700 1 -- 172.21.15.32:0/2336470110 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80400c0c0 con 0x7ff8100d6c80 2021-04-09T18:17:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff80e7fc700 1 -- 172.21.15.32:0/2336470110 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff80400c510 con 0x7ff8100d6c80 2021-04-09T18:17:51.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff815a66700 1 -- 172.21.15.32:0/2336470110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8100d6c80 msgr2=0x7ff8100d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:51.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.459 7ff815a66700 1 --2- 172.21.15.32:0/2336470110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8100d6c80 0x7ff8100d7ff0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff804006650 tx=0x7ff804009a80).stop 2021-04-09T18:17:51.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.460 7ff815a66700 1 -- 172.21.15.32:0/2336470110 shutdown_connections 2021-04-09T18:17:51.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.460 7ff815a66700 1 --2- 172.21.15.32:0/2336470110 >> v2:172.21.15.86:3300/0 conn(0x7ff8100d2440 0x7ff81010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:51.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.460 7ff815a66700 1 --2- 172.21.15.32:0/2336470110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8100d6c80 0x7ff8100d7ff0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.460 7ff815a66700 1 -- 172.21.15.32:0/2336470110 >> 172.21.15.32:0/2336470110 conn(0x7ff8100ce5c0 msgr2=0x7ff8100cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.460 7ff815a66700 1 -- 172.21.15.32:0/2336470110 shutdown_connections 2021-04-09T18:17:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.460 7ff815a66700 1 -- 172.21.15.32:0/2336470110 wait complete. 2021-04-09T18:17:51.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.480 7ff815a66700 1 Processor -- start 2021-04-09T18:17:51.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.480 7ff815a66700 1 -- start start 2021-04-09T18:17:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff815a66700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8100a80a0 0x7ff81017cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff815a66700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff81017d0d0 0x7ff81008c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:51.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff815a66700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff81008ce00 0x7ff81017afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff815a66700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81008c2e0 con 0x7ff8100a80a0 2021-04-09T18:17:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff815a66700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81008d200 con 0x7ff81008ce00 2021-04-09T18:17:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff815a66700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81018f270 con 0x7ff81017d0d0 2021-04-09T18:17:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff80f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8100a80a0 0x7ff81017cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:51.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff80f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8100a80a0 0x7ff81017cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45818/0 (socket says 172.21.15.32:45818) 2021-04-09T18:17:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff80f7fe700 1 -- 172.21.15.32:0/1016049163 learned_addr learned my addr 172.21.15.32:0/1016049163 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff80effd700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff81017d0d0 0x7ff81008c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.481 7ff80ffff700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff81008ce00 0x7ff81017afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:51.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff80f7fe700 1 -- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff81017d0d0 msgr2=0x7ff81008c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff80f7fe700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff81017d0d0 0x7ff81008c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff80f7fe700 1 -- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff81008ce00 msgr2=0x7ff81017afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff80f7fe700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff81008ce00 0x7ff81017afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:51.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff80f7fe700 1 -- 172.21.15.32:0/1016049163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8101b8020 con 0x7ff8100a80a0 2021-04-09T18:17:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff80f7fe700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8100a80a0 0x7ff81017cab0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7ff80000a5d0 tx=0x7ff80000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff7f7fff700 1 -- 172.21.15.32:0/1016049163 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff800000c10 con 0x7ff8100a80a0 2021-04-09T18:17:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff7f7fff700 1 -- 172.21.15.32:0/1016049163 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8000070d0 con 0x7ff8100a80a0 2021-04-09T18:17:51.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff815a66700 1 -- 172.21.15.32:0/1016049163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8101b8020 con 0x7ff8100a80a0 2021-04-09T18:17:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff7f7fff700 1 -- 172.21.15.32:0/1016049163 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff80000ad60 con 0x7ff8100a80a0 2021-04-09T18:17:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff815a66700 1 -- 172.21.15.32:0/1016049163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8101b8020 con 0x7ff8100a80a0 2021-04-09T18:17:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.482 7ff7f7fff700 1 -- 172.21.15.32:0/1016049163 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff800004530 con 0x7ff8100a80a0 2021-04-09T18:17:51.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.483 7ff7f7fff700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7f0018930 0x7ff7f001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:51.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.483 7ff7f7fff700 1 -- 172.21.15.32:0/1016049163 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4929+0+0 (secure 0 0 0) 0x7ff800032760 con 0x7ff8100a80a0 2021-04-09T18:17:51.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.483 7ff80effd700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7f0018930 0x7ff7f001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:51.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.483 7ff80effd700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7f0018930 0x7ff7f001adc0 crc :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:51.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.503 7ff815a66700 1 -- 172.21.15.32:0/1016049163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8101b7110 con 0x7ff8100a80a0 2021-04-09T18:17:51.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.506 7ff7f7fff700 1 -- 172.21.15.32:0/1016049163 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff800004320 con 0x7ff8100a80a0 2021-04-09T18:17:51.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:51.708 7ff815a66700 1 -- 172.21.15.32:0/1016049163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7ff8101b7110 con 0x7ff8100a80a0 2021-04-09T18:17:52.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.375 7ff7f7fff700 1 -- 172.21.15.32:0/1016049163 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v67) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff800004530 con 0x7ff8100a80a0 2021-04-09T18:17:52.458 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:17:52.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 -- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7f0018930 msgr2=0x7ff7f001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:52.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7f0018930 0x7ff7f001adc0 crc :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 -- 172.21.15.32:0/1016049163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8100a80a0 msgr2=0x7ff81017cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:52.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8100a80a0 0x7ff81017cab0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7ff80000a5d0 tx=0x7ff80000a970).stop 2021-04-09T18:17:52.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 -- 172.21.15.32:0/1016049163 shutdown_connections 2021-04-09T18:17:52.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7f0018930 0x7ff7f001adc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff81008ce00 0x7ff81017afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff81017d0d0 0x7ff81008c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 --2- 172.21.15.32:0/1016049163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8100a80a0 0x7ff81017cab0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.425 7ff815a66700 1 -- 172.21.15.32:0/1016049163 >> 172.21.15.32:0/1016049163 conn(0x7ff8101905d0 msgr2=0x7ff8101b7340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:52.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.426 7ff815a66700 1 -- 172.21.15.32:0/1016049163 shutdown_connections 2021-04-09T18:17:52.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.426 7ff815a66700 1 -- 172.21.15.32:0/1016049163 wait complete. 2021-04-09T18:17:52.472 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:17:52.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:17:52.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.591 7fa6ed679700 1 Processor -- start 2021-04-09T18:17:52.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.591 7fa6ed679700 1 -- start start 2021-04-09T18:17:52.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6ed679700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa6e810d010 0x7fa6e80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:52.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6ed679700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e80d8490 0x7fa6e810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:52.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6ed679700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e810d480 con 0x7fa6e80d1da0 2021-04-09T18:17:52.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6ed679700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e80d8d30 con 0x7fa6e810d010 2021-04-09T18:17:52.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6ed679700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80d9220 con 0x7fa6e80d8490 2021-04-09T18:17:52.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e72bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa6e810d010 0x7fa6e80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:52.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e6abe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa6e80d1da0 0x7fa6e80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55420/0 (socket says 172.21.15.32:55420) 2021-04-09T18:17:52.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e72bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa6e810d010 0x7fa6e80d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45826/0 (socket says 172.21.15.32:45826) 2021-04-09T18:17:52.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e6abe700 1 -- 172.21.15.32:0/3214608855 learned_addr learned my addr 172.21.15.32:0/3214608855 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:52.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e7ac0700 1 --2- 172.21.15.32:0/3214608855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e80d8490 0x7fa6e810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:52.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e72bf700 1 -- 172.21.15.32:0/3214608855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e80d8490 msgr2=0x7fa6e810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:52.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e72bf700 1 --2- 172.21.15.32:0/3214608855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e80d8490 0x7fa6e810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e72bf700 1 -- 172.21.15.32:0/3214608855 >> v1:172.21.15.32:6789/0 conn(0x7fa6e80d1da0 legacy=0x7fa6e80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:52.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e72bf700 1 -- 172.21.15.32:0/3214608855 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e810e020 con 0x7fa6e810d010 2021-04-09T18:17:52.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e62bd700 1 -- 172.21.15.32:0/3214608855 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414996800 0 0) 0x7fa6dc000ce0 con 0x7fa6e80d1da0 2021-04-09T18:17:52.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e72bf700 1 --2- 172.21.15.32:0/3214608855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e810d010 0x7fa6e80d1780 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fa6d8006650 tx=0x7fa6d8009a00).ready entity=mon.0 client_cookie=9dfdf9a3222b415e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:52.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e62bd700 1 -- 172.21.15.32:0/3214608855 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6d8009df0 con 0x7fa6e810d010 2021-04-09T18:17:52.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.592 7fa6e62bd700 1 -- 172.21.15.32:0/3214608855 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d80009b0 con 0x7fa6e810d010 2021-04-09T18:17:52.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.593 7fa6e62bd700 1 -- 172.21.15.32:0/3214608855 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6d800c2b0 con 0x7fa6e810d010 2021-04-09T18:17:52.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.593 7fa6ed679700 1 -- 172.21.15.32:0/3214608855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e810d010 msgr2=0x7fa6e80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:52.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.593 7fa6ed679700 1 --2- 172.21.15.32:0/3214608855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e810d010 0x7fa6e80d1780 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fa6d8006650 tx=0x7fa6d8009a00).stop 2021-04-09T18:17:52.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.593 7fa6ed679700 1 -- 172.21.15.32:0/3214608855 shutdown_connections 2021-04-09T18:17:52.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.593 7fa6ed679700 1 --2- 172.21.15.32:0/3214608855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e80d8490 0x7fa6e810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.593 7fa6ed679700 1 --2- 172.21.15.32:0/3214608855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e810d010 0x7fa6e80d1780 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.593 7fa6ed679700 1 -- 172.21.15.32:0/3214608855 >> 172.21.15.32:0/3214608855 conn(0x7fa6e80ce5f0 msgr2=0x7fa6e80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:52.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.593 7fa6ed679700 1 -- 172.21.15.32:0/3214608855 shutdown_connections 2021-04-09T18:17:52.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.593 7fa6ed679700 1 -- 172.21.15.32:0/3214608855 wait complete. 2021-04-09T18:17:52.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.614 7fa6ed679700 1 Processor -- start 2021-04-09T18:17:52.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.614 7fa6ed679700 1 -- start start 2021-04-09T18:17:52.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.614 7fa6ed679700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e80b5360 0x7fa6e81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:52.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.614 7fa6ed679700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e818e140 0x7fa6e81aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:52.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.614 7fa6ed679700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e81ad040 0x7fa6e808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.614 7fa6ed679700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e81a5360 con 0x7fa6e818e140 2021-04-09T18:17:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.614 7fa6ed679700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80902c0 con 0x7fa6e80b5360 2021-04-09T18:17:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.614 7fa6ed679700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80904e0 con 0x7fa6e81ad040 2021-04-09T18:17:52.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.614 7fa6e6abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e818e140 0x7fa6e81aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:52.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e6abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e818e140 0x7fa6e81aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45834/0 (socket says 172.21.15.32:45834) 2021-04-09T18:17:52.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e6abe700 1 -- 172.21.15.32:0/1124415348 learned_addr learned my addr 172.21.15.32:0/1124415348 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:52.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e72bf700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e80b5360 0x7fa6e81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e7ac0700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e81ad040 0x7fa6e808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:52.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e6abe700 1 -- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e81ad040 msgr2=0x7fa6e808fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:52.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e6abe700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e81ad040 0x7fa6e808fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e6abe700 1 -- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e80b5360 msgr2=0x7fa6e81a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:52.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e6abe700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e80b5360 0x7fa6e81a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:52.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e6abe700 1 -- 172.21.15.32:0/1124415348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e8183c80 con 0x7fa6e818e140 2021-04-09T18:17:52.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6e6abe700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e818e140 0x7fa6e81aca20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fa6dc006f00 tx=0x7fa6dc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:52.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6cffff700 1 -- 172.21.15.32:0/1124415348 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6dc006020 con 0x7fa6e818e140 2021-04-09T18:17:52.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6cffff700 1 -- 172.21.15.32:0/1124415348 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc0063a0 con 0x7fa6e818e140 2021-04-09T18:17:52.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e8183c80 con 0x7fa6e818e140 2021-04-09T18:17:52.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6cffff700 1 -- 172.21.15.32:0/1124415348 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6dc004620 con 0x7fa6e818e140 2021-04-09T18:17:52.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.615 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e8183c80 con 0x7fa6e818e140 2021-04-09T18:17:52.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.616 7fa6cffff700 1 -- 172.21.15.32:0/1124415348 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa6dc00a000 con 0x7fa6e818e140 2021-04-09T18:17:52.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.616 7fa6cffff700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6c8018930 0x7fa6c801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:52.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.616 7fa6cffff700 1 -- 172.21.15.32:0/1124415348 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fa6dc0326e0 con 0x7fa6e818e140 2021-04-09T18:17:52.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.616 7fa6e72bf700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6c8018930 0x7fa6c801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:52.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.617 7fa6e72bf700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6c8018930 0x7fa6c801ae10 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.636 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e8182d70 con 0x7fa6e818e140 2021-04-09T18:17:52.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.640 7fa6cffff700 1 -- 172.21.15.32:0/1124415348 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa6dc006b90 con 0x7fa6e818e140 2021-04-09T18:17:52.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.826 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa6e810d480 con 0x7fa6e818e140 2021-04-09T18:17:52.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.826 7fa6cffff700 1 -- 172.21.15.32:0/1124415348 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+4539 (secure 0 0 0) 0x7fa6dc00a000 con 0x7fa6e818e140 2021-04-09T18:17:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:52.862 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:17:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:17:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:17:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:17:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:17:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:17:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:17:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:17:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:17:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:17:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:17:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:17:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:17:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:17:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:17:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:17:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:17:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:17:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:17:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:17:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:17:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:17:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:17:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:17:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:17:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:17:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:17:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:17:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:17:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:17:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:17:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:17:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:17:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:17:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:17:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:17:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:17:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:17:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:17:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:17:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:17:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:17:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:17:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:17:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:17:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:17:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:17:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:17:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:17:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:17:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:17:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:17:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:17:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:17:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:17:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:17:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:17:53.203 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:17:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6c8018930 msgr2=0x7fa6c801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:53.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6c8018930 0x7fa6c801ae10 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e818e140 msgr2=0x7fa6e81aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:53.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e818e140 0x7fa6e81aca20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fa6dc006f00 tx=0x7fa6dc00b940).stop 2021-04-09T18:17:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 shutdown_connections 2021-04-09T18:17:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6c8018930 0x7fa6c801ae10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e818e140 0x7fa6e81aca20 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6e81ad040 0x7fa6e808fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 --2- 172.21.15.32:0/1124415348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6e80b5360 0x7fa6e81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 >> 172.21.15.32:0/1124415348 conn(0x7fa6e819b460 msgr2=0x7fa6e819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:53.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 shutdown_connections 2021-04-09T18:17:53.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:52.829 7fa6ed679700 1 -- 172.21.15.32:0/1124415348 wait complete. 2021-04-09T18:17:53.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:17:53.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.460 7fc3c7943700 1 Processor -- start 2021-04-09T18:17:53.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.460 7fc3c7943700 1 -- start start 2021-04-09T18:17:53.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c7943700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3c010d010 0x7fc3c00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c7943700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc3c00d8490 0x7fc3c010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c7943700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c010d480 con 0x7fc3c00d1da0 2021-04-09T18:17:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c7943700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3c00d8d30 con 0x7fc3c00d8490 2021-04-09T18:17:53.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c7943700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3c00d9220 con 0x7fc3c010d010 2021-04-09T18:17:53.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c5f60700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc3c00d8490 0x7fc3c010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:53.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c5f60700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc3c00d8490 0x7fc3c010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45842/0 (socket says 172.21.15.32:45842) 2021-04-09T18:17:53.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c4f5e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc3c00d1da0 0x7fc3c00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55434/0 (socket says 172.21.15.32:55434) 2021-04-09T18:17:53.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c4f5e700 1 -- 172.21.15.32:0/3473543531 learned_addr learned my addr 172.21.15.32:0/3473543531 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:53.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c575f700 1 --2- 172.21.15.32:0/3473543531 >> v2:172.21.15.86:3300/0 conn(0x7fc3c010d010 0x7fc3c00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:53.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c5f60700 1 -- 172.21.15.32:0/3473543531 >> v2:172.21.15.86:3300/0 conn(0x7fc3c010d010 msgr2=0x7fc3c00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:53.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c5f60700 1 --2- 172.21.15.32:0/3473543531 >> v2:172.21.15.86:3300/0 conn(0x7fc3c010d010 0x7fc3c00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c5f60700 1 -- 172.21.15.32:0/3473543531 >> v1:172.21.15.32:6789/0 conn(0x7fc3c00d1da0 legacy=0x7fc3c00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:53.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c5f60700 1 -- 172.21.15.32:0/3473543531 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c010e020 con 0x7fc3c00d8490 2021-04-09T18:17:53.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3b3fff700 1 -- 172.21.15.32:0/3473543531 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278534982 0 0) 0x7fc3b40009b0 con 0x7fc3c00d1da0 2021-04-09T18:17:53.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3c5f60700 1 --2- 172.21.15.32:0/3473543531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c00d8490 0x7fc3c010da00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fc3b8006e30 tx=0x7fc3b8009de0).ready entity=mon.0 client_cookie=7e9305b0526ac099 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:53.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3b3fff700 1 -- 172.21.15.32:0/3473543531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3b800b300 con 0x7fc3c00d8490 2021-04-09T18:17:53.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.461 7fc3b3fff700 1 -- 172.21.15.32:0/3473543531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b800b680 con 0x7fc3c00d8490 2021-04-09T18:17:53.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.462 7fc3b3fff700 1 -- 172.21.15.32:0/3473543531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3b8000f90 con 0x7fc3c00d8490 2021-04-09T18:17:53.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.462 7fc3c7943700 1 -- 172.21.15.32:0/3473543531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c00d8490 msgr2=0x7fc3c010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:53.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.462 7fc3c7943700 1 --2- 172.21.15.32:0/3473543531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c00d8490 0x7fc3c010da00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fc3b8006e30 tx=0x7fc3b8009de0).stop 2021-04-09T18:17:53.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.462 7fc3c7943700 1 -- 172.21.15.32:0/3473543531 shutdown_connections 2021-04-09T18:17:53.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.462 7fc3c7943700 1 --2- 172.21.15.32:0/3473543531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c00d8490 0x7fc3c010da00 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.462 7fc3c7943700 1 --2- 172.21.15.32:0/3473543531 >> v2:172.21.15.86:3300/0 conn(0x7fc3c010d010 0x7fc3c00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.462 7fc3c7943700 1 -- 172.21.15.32:0/3473543531 >> 172.21.15.32:0/3473543531 conn(0x7fc3c00ce5f0 msgr2=0x7fc3c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:53.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.462 7fc3c7943700 1 -- 172.21.15.32:0/3473543531 shutdown_connections 2021-04-09T18:17:53.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.462 7fc3c7943700 1 -- 172.21.15.32:0/3473543531 wait complete. 2021-04-09T18:17:53.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.482 7fc3c7943700 1 Processor -- start 2021-04-09T18:17:53.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c7943700 1 -- start start 2021-04-09T18:17:53.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c7943700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c01a5210 0x7fc3c0185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:53.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c7943700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3c0185a70 0x7fc3c00acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:53.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c7943700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3c0185e70 0x7fc3c00ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:53.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c7943700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c018f7b0 con 0x7fc3c01a5210 2021-04-09T18:17:53.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c7943700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c00ad7f0 con 0x7fc3c0185a70 2021-04-09T18:17:53.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c7943700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c00ada10 con 0x7fc3c0185e70 2021-04-09T18:17:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c575f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c01a5210 0x7fc3c0185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c575f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c01a5210 0x7fc3c0185450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45846/0 (socket says 172.21.15.32:45846) 2021-04-09T18:17:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c575f700 1 -- 172.21.15.32:0/2981819160 learned_addr learned my addr 172.21.15.32:0/2981819160 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.483 7fc3c4f5e700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3c0185a70 0x7fc3c00acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c5f60700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3c0185e70 0x7fc3c00ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:53.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c575f700 1 -- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3c0185e70 msgr2=0x7fc3c00ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:53.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c575f700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3c0185e70 0x7fc3c00ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c575f700 1 -- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3c0185a70 msgr2=0x7fc3c00acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c575f700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3c0185a70 0x7fc3c00acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c575f700 1 -- 172.21.15.32:0/2981819160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c0179610 con 0x7fc3c01a5210 2021-04-09T18:17:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c5f60700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3c0185e70 0x7fc3c00ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:53.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c4f5e700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3c0185a70 0x7fc3c00acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:53.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c575f700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c01a5210 0x7fc3c0185450 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fc3bc006f60 tx=0x7fc3bc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:53.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3b1ffb700 1 -- 172.21.15.32:0/2981819160 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3bc0009b0 con 0x7fc3c01a5210 2021-04-09T18:17:53.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3b1ffb700 1 -- 172.21.15.32:0/2981819160 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3bc00bd60 con 0x7fc3c01a5210 2021-04-09T18:17:53.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3b1ffb700 1 -- 172.21.15.32:0/2981819160 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3bc002030 con 0x7fc3c01a5210 2021-04-09T18:17:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c0179610 con 0x7fc3c01a5210 2021-04-09T18:17:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.484 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c0179610 con 0x7fc3c01a5210 2021-04-09T18:17:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.485 7fc3b1ffb700 1 -- 172.21.15.32:0/2981819160 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc3bc000c90 con 0x7fc3c01a5210 2021-04-09T18:17:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.485 7fc3b1ffb700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3a0018930 0x7fc3a001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:53.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.485 7fc3b1ffb700 1 -- 172.21.15.32:0/2981819160 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fc3bc0326c0 con 0x7fc3c01a5210 2021-04-09T18:17:53.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.485 7fc3c4f5e700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3a0018930 0x7fc3a001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:53.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.485 7fc3c4f5e700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3a0018930 0x7fc3a001ae10 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:53.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.505 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c018f7b0 con 0x7fc3c01a5210 2021-04-09T18:17:53.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.508 7fc3b1ffb700 1 -- 172.21.15.32:0/2981819160 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc3bc006b90 con 0x7fc3c01a5210 2021-04-09T18:17:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.695 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3c018f7b0 con 0x7fc3c01a5210 2021-04-09T18:17:53.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.696 7fc3b1ffb700 1 -- 172.21.15.32:0/2981819160 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+11145 (secure 0 0 0) 0x7fc3bc000c90 con 0x7fc3c01a5210 2021-04-09T18:17:53.733 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:53.733 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":67,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:17:52.370753","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:17:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.698 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3a0018930 msgr2=0x7fc3a001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.698 7fc3c7943700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3a0018930 0x7fc3a001ae10 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.698 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c01a5210 msgr2=0x7fc3c0185450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.698 7fc3c7943700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c01a5210 0x7fc3c0185450 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fc3bc006f60 tx=0x7fc3bc00b970).stop 2021-04-09T18:17:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.699 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 shutdown_connections 2021-04-09T18:17:53.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.699 7fc3c7943700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3a0018930 0x7fc3a001ae10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.699 7fc3c7943700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3c0185e70 0x7fc3c00ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.699 7fc3c7943700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3c0185a70 0x7fc3c00acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.699 7fc3c7943700 1 --2- 172.21.15.32:0/2981819160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3c01a5210 0x7fc3c0185450 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.699 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 >> 172.21.15.32:0/2981819160 conn(0x7fc3c01a99c0 msgr2=0x7fc3c017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.699 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 shutdown_connections 2021-04-09T18:17:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.699 7fc3c7943700 1 -- 172.21.15.32:0/2981819160 wait complete. 2021-04-09T18:17:53.758 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:17:53.758 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:17:53.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.887 7f7806456700 1 Processor -- start 2021-04-09T18:17:53.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.887 7f7806456700 1 -- start start 2021-04-09T18:17:53.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.888 7f7806456700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f780010d010 0x7f78000d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:53.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.888 7f7806456700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000d1da0 0x7f78000d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:53.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.888 7f7806456700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f780010d480 con 0x7f78000d82c0 2021-04-09T18:17:53.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.888 7f7806456700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78000d9220 con 0x7f780010d010 2021-04-09T18:17:53.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.888 7f7806456700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78000d86c0 con 0x7f78000d1da0 2021-04-09T18:17:53.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.888 7f77f7fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000d1da0 0x7f78000d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:53.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.888 7f77fffff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f780010d010 0x7f78000d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:53.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.888 7f7804a73700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f78000d82c0 0x7f780010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:53372/0 (socket says 172.21.15.32:53372) 2021-04-09T18:17:53.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.888 7f7804a73700 1 -- 172.21.15.32:0/32578337 learned_addr learned my addr 172.21.15.32:0/32578337 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77ff7fe700 1 -- 172.21.15.32:0/32578337 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160182713 0 0) 0x7f77f8001440 con 0x7f78000d82c0 2021-04-09T18:17:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77ff7fe700 1 -- 172.21.15.32:0/32578337 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77ec002fe0 con 0x7f78000d82c0 2021-04-09T18:17:53.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77f7fff700 1 -- 172.21.15.32:0/32578337 >> v2:172.21.15.86:3300/0 conn(0x7f780010d010 msgr2=0x7f78000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:53.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77f7fff700 1 --2- 172.21.15.32:0/32578337 >> v2:172.21.15.86:3300/0 conn(0x7f780010d010 0x7f78000d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:53.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77f7fff700 1 -- 172.21.15.32:0/32578337 >> v1:172.21.15.86:6789/0 conn(0x7f78000d82c0 legacy=0x7f780010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:53.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77f7fff700 1 -- 172.21.15.32:0/32578337 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780010e1f0 con 0x7f78000d1da0 2021-04-09T18:17:53.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77fffff700 1 --2- 172.21.15.32:0/32578337 >> v2:172.21.15.86:3300/0 conn(0x7f780010d010 0x7f78000d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:54.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77f7fff700 1 --2- 172.21.15.32:0/32578337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000d1da0 0x7f78000d7ca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f77e8006650 tx=0x7f77e8009a80).ready entity=mon.2 client_cookie=531a086857d5438c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:54.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77ff7fe700 1 -- 172.21.15.32:0/32578337 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77e80009b0 con 0x7f78000d1da0 2021-04-09T18:17:54.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77ff7fe700 1 -- 172.21.15.32:0/32578337 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e800c0c0 con 0x7f78000d1da0 2021-04-09T18:17:54.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f77ff7fe700 1 -- 172.21.15.32:0/32578337 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77e80009b0 con 0x7f78000d1da0 2021-04-09T18:17:54.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f7806456700 1 -- 172.21.15.32:0/32578337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000d1da0 msgr2=0x7f78000d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.889 7f7806456700 1 --2- 172.21.15.32:0/32578337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000d1da0 0x7f78000d7ca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f77e8006650 tx=0x7f77e8009a80).stop 2021-04-09T18:17:54.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.890 7f7806456700 1 -- 172.21.15.32:0/32578337 shutdown_connections 2021-04-09T18:17:54.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.890 7f7806456700 1 --2- 172.21.15.32:0/32578337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000d1da0 0x7f78000d7ca0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.890 7f7806456700 1 --2- 172.21.15.32:0/32578337 >> v2:172.21.15.86:3300/0 conn(0x7f780010d010 0x7f78000d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.890 7f7806456700 1 -- 172.21.15.32:0/32578337 >> 172.21.15.32:0/32578337 conn(0x7f78000ce5f0 msgr2=0x7f78000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:54.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.890 7f7806456700 1 -- 172.21.15.32:0/32578337 shutdown_connections 2021-04-09T18:17:54.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.890 7f7806456700 1 -- 172.21.15.32:0/32578337 wait complete. 2021-04-09T18:17:54.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.910 7f7806456700 1 Processor -- start 2021-04-09T18:17:54.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.910 7f7806456700 1 -- start start 2021-04-09T18:17:54.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7806456700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000c3c00 0x7f780009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7806456700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7800095240 0x7f78001a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7806456700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78001a1270 0x7f78000b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7806456700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780009cda0 con 0x7f78001a1270 2021-04-09T18:17:54.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7806456700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78000c3680 con 0x7f7800095240 2021-04-09T18:17:54.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7806456700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7800095640 con 0x7f78000c3c00 2021-04-09T18:17:54.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7804a73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78001a1270 0x7f78000b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7804a73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78001a1270 0x7f78000b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45862/0 (socket says 172.21.15.32:45862) 2021-04-09T18:17:54.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7804a73700 1 -- 172.21.15.32:0/101869679 learned_addr learned my addr 172.21.15.32:0/101869679 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:54.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f77f7fff700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7800095240 0x7f78001a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f77fffff700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000c3c00 0x7f780009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7804a73700 1 -- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000c3c00 msgr2=0x7f780009c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7804a73700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000c3c00 0x7f780009c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7804a73700 1 -- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7800095240 msgr2=0x7f78001a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7804a73700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7800095240 0x7f78001a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7804a73700 1 -- 172.21.15.32:0/101869679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78001a1670 con 0x7f78001a1270 2021-04-09T18:17:54.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.911 7f7804a73700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78001a1270 0x7f78000b0cf0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f77f8001b90 tx=0x7f77f800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:54.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.912 7f77fd7fa700 1 -- 172.21.15.32:0/101869679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77f80070d0 con 0x7f78001a1270 2021-04-09T18:17:54.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.912 7f77fd7fa700 1 -- 172.21.15.32:0/101869679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f8007450 con 0x7f78001a1270 2021-04-09T18:17:54.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.912 7f7806456700 1 -- 172.21.15.32:0/101869679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78001a1670 con 0x7f78001a1270 2021-04-09T18:17:54.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.912 7f77fd7fa700 1 -- 172.21.15.32:0/101869679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77f8004aa0 con 0x7f78001a1270 2021-04-09T18:17:54.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.912 7f7806456700 1 -- 172.21.15.32:0/101869679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78001a1670 con 0x7f78001a1270 2021-04-09T18:17:54.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.913 7f77fd7fa700 1 -- 172.21.15.32:0/101869679 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f77f8007240 con 0x7f78001a1270 2021-04-09T18:17:54.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.913 7f77fd7fa700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f77e40188c0 0x7f77e401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.913 7f77fd7fa700 1 -- 172.21.15.32:0/101869679 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f77f80327f0 con 0x7f78001a1270 2021-04-09T18:17:54.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.913 7f77fffff700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f77e40188c0 0x7f77e401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.913 7f77fffff700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f77e40188c0 0x7f77e401adc0 crc :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:54.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.932 7f7806456700 1 -- 172.21.15.32:0/101869679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7800093d50 con 0x7f78001a1270 2021-04-09T18:17:54.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:53.935 7f77fd7fa700 1 -- 172.21.15.32:0/101869679 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f77f8007030 con 0x7f78001a1270 2021-04-09T18:17:54.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.112 7f7806456700 1 -- 172.21.15.32:0/101869679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f780010d480 con 0x7f78001a1270 2021-04-09T18:17:54.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.113 7f77fd7fa700 1 -- 172.21.15.32:0/101869679 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v67) v1 ==== 92+0+10 (secure 0 0 0) 0x7f77f8007240 con 0x7f78001a1270 2021-04-09T18:17:54.165 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:17:54.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 -- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f77e40188c0 msgr2=0x7f77e401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f77e40188c0 0x7f77e401adc0 crc :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 -- 172.21.15.32:0/101869679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78001a1270 msgr2=0x7f78000b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78001a1270 0x7f78000b0cf0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f77f8001b90 tx=0x7f77f800ac50).stop 2021-04-09T18:17:54.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 -- 172.21.15.32:0/101869679 shutdown_connections 2021-04-09T18:17:54.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f77e40188c0 0x7f77e401adc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78001a1270 0x7f78000b0cf0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7800095240 0x7f78001a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 --2- 172.21.15.32:0/101869679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78000c3c00 0x7f780009c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 -- 172.21.15.32:0/101869679 >> 172.21.15.32:0/101869679 conn(0x7f78000975f0 msgr2=0x7f7800096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:54.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 -- 172.21.15.32:0/101869679 shutdown_connections 2021-04-09T18:17:54.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.115 7f7806456700 1 -- 172.21.15.32:0/101869679 wait complete. 2021-04-09T18:17:54.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:17:54.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.349 7f5e51845700 1 Processor -- start 2021-04-09T18:17:54.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.349 7f5e51845700 1 -- start start 2021-04-09T18:17:54.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e51845700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c10d010 0x7f5e4c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e51845700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5e4c0d1da0 0x7f5e4c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e51845700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4c10d480 con 0x7f5e4c0d82c0 2021-04-09T18:17:54.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e51845700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4c10e780 con 0x7f5e4c0d1da0 2021-04-09T18:17:54.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e51845700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0d8d30 con 0x7f5e4c10d010 2021-04-09T18:17:54.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e4b574700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c10d010 0x7f5e4c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e4ad73700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5e4c0d1da0 0x7f5e4c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e4bd75700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5e4c0d82c0 0x7f5e4c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:53386/0 (socket says 172.21.15.32:53386) 2021-04-09T18:17:54.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e4ad73700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5e4c0d1da0 0x7f5e4c0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:37630/0 (socket says 172.21.15.32:37630) 2021-04-09T18:17:54.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e4b574700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c10d010 0x7f5e4c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:53094/0 (socket says 172.21.15.32:53094) 2021-04-09T18:17:54.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.350 7f5e4bd75700 1 -- 172.21.15.32:0/431954532 learned_addr learned my addr 172.21.15.32:0/431954532 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:54.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4ad73700 1 -- 172.21.15.32:0/431954532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c10d010 msgr2=0x7f5e4c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:54.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4ad73700 1 --2- 172.21.15.32:0/431954532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c10d010 0x7f5e4c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4ad73700 1 -- 172.21.15.32:0/431954532 >> v1:172.21.15.86:6789/0 conn(0x7f5e4c0d82c0 legacy=0x7f5e4c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4ad73700 1 -- 172.21.15.32:0/431954532 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c0d9220 con 0x7f5e4c0d1da0 2021-04-09T18:17:54.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4a572700 1 -- 172.21.15.32:0/431954532 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897772147 0 0) 0x7f5e3c0010c0 con 0x7f5e4c0d82c0 2021-04-09T18:17:54.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4b574700 1 --2- 172.21.15.32:0/431954532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c10d010 0x7f5e4c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:54.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4ad73700 1 --2- 172.21.15.32:0/431954532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c0d1da0 0x7f5e4c0d7ca0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f5e34006650 tx=0x7f5e34009a80).ready entity=mon.1 client_cookie=fb978cb746d467c0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:54.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4a572700 1 -- 172.21.15.32:0/431954532 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e340009b0 con 0x7f5e4c0d1da0 2021-04-09T18:17:54.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4a572700 1 -- 172.21.15.32:0/431954532 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3400c0c0 con 0x7f5e4c0d1da0 2021-04-09T18:17:54.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e4a572700 1 -- 172.21.15.32:0/431954532 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e3400c510 con 0x7f5e4c0d1da0 2021-04-09T18:17:54.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e51845700 1 -- 172.21.15.32:0/431954532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c0d1da0 msgr2=0x7f5e4c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.351 7f5e51845700 1 --2- 172.21.15.32:0/431954532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c0d1da0 0x7f5e4c0d7ca0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f5e34006650 tx=0x7f5e34009a80).stop 2021-04-09T18:17:54.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.352 7f5e51845700 1 -- 172.21.15.32:0/431954532 shutdown_connections 2021-04-09T18:17:54.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.352 7f5e51845700 1 --2- 172.21.15.32:0/431954532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c0d1da0 0x7f5e4c0d7ca0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.352 7f5e51845700 1 --2- 172.21.15.32:0/431954532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c10d010 0x7f5e4c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.352 7f5e51845700 1 -- 172.21.15.32:0/431954532 >> 172.21.15.32:0/431954532 conn(0x7f5e4c0ce5f0 msgr2=0x7f5e4c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:54.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.352 7f5e51845700 1 -- 172.21.15.32:0/431954532 shutdown_connections 2021-04-09T18:17:54.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.352 7f5e51845700 1 -- 172.21.15.32:0/431954532 wait complete. 2021-04-09T18:17:54.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.372 7f5e51845700 1 Processor -- start 2021-04-09T18:17:54.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.372 7f5e51845700 1 -- start start 2021-04-09T18:17:54.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e51845700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c0b5360 0x7f5e4c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e51845700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e4c18e140 0x7f5e4c1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e51845700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c1ad040 0x7f5e4c08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e51845700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c1a5360 con 0x7f5e4c18e140 2021-04-09T18:17:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e51845700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0902c0 con 0x7f5e4c1ad040 2021-04-09T18:17:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e51845700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0904e0 con 0x7f5e4c0b5360 2021-04-09T18:17:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4ad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e4c18e140 0x7f5e4c1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4ad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e4c18e140 0x7f5e4c1aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45874/0 (socket says 172.21.15.32:45874) 2021-04-09T18:17:54.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4ad73700 1 -- 172.21.15.32:0/3371247540 learned_addr learned my addr 172.21.15.32:0/3371247540 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:54.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4bd75700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c1ad040 0x7f5e4c08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4b574700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c0b5360 0x7f5e4c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4ad73700 1 -- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c0b5360 msgr2=0x7f5e4c1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4ad73700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c0b5360 0x7f5e4c1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4ad73700 1 -- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c1ad040 msgr2=0x7f5e4c08fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4ad73700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c1ad040 0x7f5e4c08fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4ad73700 1 -- 172.21.15.32:0/3371247540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c183c80 con 0x7f5e4c18e140 2021-04-09T18:17:54.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4b574700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c0b5360 0x7f5e4c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4bd75700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c1ad040 0x7f5e4c08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:54.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.373 7f5e4ad73700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e4c18e140 0x7f5e4c1aca20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f5e34009650 tx=0x7f5e3400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:54.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.374 7f5e43fff700 1 -- 172.21.15.32:0/3371247540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e34009360 con 0x7f5e4c18e140 2021-04-09T18:17:54.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.374 7f5e43fff700 1 -- 172.21.15.32:0/3371247540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3400bd50 con 0x7f5e4c18e140 2021-04-09T18:17:54.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.374 7f5e51845700 1 -- 172.21.15.32:0/3371247540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4c183c80 con 0x7f5e4c18e140 2021-04-09T18:17:54.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.374 7f5e43fff700 1 -- 172.21.15.32:0/3371247540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e340052e0 con 0x7f5e4c18e140 2021-04-09T18:17:54.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.374 7f5e51845700 1 -- 172.21.15.32:0/3371247540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4c183c80 con 0x7f5e4c18e140 2021-04-09T18:17:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.374 7f5e43fff700 1 -- 172.21.15.32:0/3371247540 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5e34006030 con 0x7f5e4c18e140 2021-04-09T18:17:54.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.374 7f5e43fff700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e30018930 0x7f5e3001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.374 7f5e43fff700 1 -- 172.21.15.32:0/3371247540 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f5e340326b0 con 0x7f5e4c18e140 2021-04-09T18:17:54.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.375 7f5e4b574700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e30018930 0x7f5e3001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.375 7f5e4b574700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e30018930 0x7f5e3001ae10 crc :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:54.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.394 7f5e51845700 1 -- 172.21.15.32:0/3371247540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4c182d70 con 0x7f5e4c18e140 2021-04-09T18:17:54.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.398 7f5e43fff700 1 -- 172.21.15.32:0/3371247540 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5e340009c0 con 0x7f5e4c18e140 2021-04-09T18:17:54.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.576 7f5e51845700 1 -- 172.21.15.32:0/3371247540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5e4c10d480 con 0x7f5e4c18e140 2021-04-09T18:17:54.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.577 7f5e43fff700 1 -- 172.21.15.32:0/3371247540 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v67) v1 ==== 84+0+64 (secure 0 0 0) 0x7f5e3400bd50 con 0x7f5e4c18e140 2021-04-09T18:17:54.656 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:54.657 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:17:54.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:17:54.668 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:17:54.668 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:17:54.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:17:54.669 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:17:54.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.579 7f5e51845700 1 -- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e30018930 msgr2=0x7f5e3001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.579 7f5e51845700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e30018930 0x7f5e3001ae10 crc :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.579 7f5e51845700 1 -- 172.21.15.32:0/3371247540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e4c18e140 msgr2=0x7f5e4c1aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.579 7f5e51845700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e4c18e140 0x7f5e4c1aca20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f5e34009650 tx=0x7f5e3400b960).stop 2021-04-09T18:17:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.580 7f5e51845700 1 -- 172.21.15.32:0/3371247540 shutdown_connections 2021-04-09T18:17:54.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.580 7f5e51845700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e30018930 0x7f5e3001ae10 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.580 7f5e51845700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e4c18e140 0x7f5e4c1aca20 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.580 7f5e51845700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c1ad040 0x7f5e4c08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.580 7f5e51845700 1 --2- 172.21.15.32:0/3371247540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e4c0b5360 0x7f5e4c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.580 7f5e51845700 1 -- 172.21.15.32:0/3371247540 >> 172.21.15.32:0/3371247540 conn(0x7f5e4c19b460 msgr2=0x7f5e4c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:54.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.580 7f5e51845700 1 -- 172.21.15.32:0/3371247540 shutdown_connections 2021-04-09T18:17:54.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.580 7f5e51845700 1 -- 172.21.15.32:0/3371247540 wait complete. 2021-04-09T18:17:54.787 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:17:54.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:17:54.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.905 7ff2ac7c7700 1 Processor -- start 2021-04-09T18:17:54.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.905 7ff2ac7c7700 1 -- start start 2021-04-09T18:17:54.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2ac7c7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff2a410d010 0x7ff2a40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2ac7c7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff2a40d1da0 0x7ff2a40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:54.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2ac7c7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a410d480 con 0x7ff2a40d82c0 2021-04-09T18:17:54.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2ac7c7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a40d8d30 con 0x7ff2a410d010 2021-04-09T18:17:54.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2ac7c7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a40d9220 con 0x7ff2a40d1da0 2021-04-09T18:17:54.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2aa5e3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff2a410d010 0x7ff2a40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2aa5e3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff2a410d010 0x7ff2a40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45880/0 (socket says 172.21.15.32:45880) 2021-04-09T18:17:54.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2aa5e3700 1 -- 172.21.15.32:0/844144254 learned_addr learned my addr 172.21.15.32:0/844144254 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:54.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2a9de2700 1 --2- 172.21.15.32:0/844144254 >> v2:172.21.15.86:3300/0 conn(0x7ff2a40d1da0 0x7ff2a40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:54.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2aa5e3700 1 -- 172.21.15.32:0/844144254 >> v2:172.21.15.86:3300/0 conn(0x7ff2a40d1da0 msgr2=0x7ff2a40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:54.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2aa5e3700 1 --2- 172.21.15.32:0/844144254 >> v2:172.21.15.86:3300/0 conn(0x7ff2a40d1da0 0x7ff2a40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:54.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2aa5e3700 1 -- 172.21.15.32:0/844144254 >> v1:172.21.15.86:6789/0 conn(0x7ff2a40d82c0 legacy=0x7ff2a410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.906 7ff2aa5e3700 1 -- 172.21.15.32:0/844144254 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40d86f0 con 0x7ff2a410d010 2021-04-09T18:17:54.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.907 7ff2aa5e3700 1 --2- 172.21.15.32:0/844144254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a410d010 0x7ff2a40d1780 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7ff29c006700 tx=0x7ff29c009b80).ready entity=mon.0 client_cookie=37d9e577256937a2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:54.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.907 7ff2a95e1700 1 -- 172.21.15.32:0/844144254 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29c00c0c0 con 0x7ff2a410d010 2021-04-09T18:17:54.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.907 7ff2a95e1700 1 -- 172.21.15.32:0/844144254 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29c00c440 con 0x7ff2a410d010 2021-04-09T18:17:54.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.907 7ff2a95e1700 1 -- 172.21.15.32:0/844144254 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29c00c920 con 0x7ff2a410d010 2021-04-09T18:17:54.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.907 7ff2ac7c7700 1 -- 172.21.15.32:0/844144254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a410d010 msgr2=0x7ff2a40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:54.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.907 7ff2ac7c7700 1 --2- 172.21.15.32:0/844144254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a410d010 0x7ff2a40d1780 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7ff29c006700 tx=0x7ff29c009b80).stop 2021-04-09T18:17:54.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.908 7ff2ac7c7700 1 -- 172.21.15.32:0/844144254 shutdown_connections 2021-04-09T18:17:55.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.908 7ff2ac7c7700 1 --2- 172.21.15.32:0/844144254 >> v2:172.21.15.86:3300/0 conn(0x7ff2a40d1da0 0x7ff2a40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.908 7ff2ac7c7700 1 --2- 172.21.15.32:0/844144254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a410d010 0x7ff2a40d1780 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.908 7ff2ac7c7700 1 -- 172.21.15.32:0/844144254 >> 172.21.15.32:0/844144254 conn(0x7ff2a40ce5f0 msgr2=0x7ff2a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.908 7ff2ac7c7700 1 -- 172.21.15.32:0/844144254 shutdown_connections 2021-04-09T18:17:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.908 7ff2ac7c7700 1 -- 172.21.15.32:0/844144254 wait complete. 2021-04-09T18:17:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.928 7ff2ac7c7700 1 Processor -- start 2021-04-09T18:17:55.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.928 7ff2ac7c7700 1 -- start start 2021-04-09T18:17:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.928 7ff2ac7c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a40b1680 0x7ff2a40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2ac7c7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2a409e330 0x7ff2a40b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2ac7c7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2a40b2190 0x7ff2a409f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2ac7c7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a409e730 con 0x7ff2a40b1680 2021-04-09T18:17:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2ac7c7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a409f6c0 con 0x7ff2a40b2190 2021-04-09T18:17:55.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2ac7c7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a409f8e0 con 0x7ff2a409e330 2021-04-09T18:17:55.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aa5e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a40b1680 0x7ff2a40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aa5e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a40b1680 0x7ff2a40b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45886/0 (socket says 172.21.15.32:45886) 2021-04-09T18:17:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aa5e3700 1 -- 172.21.15.32:0/477402943 learned_addr learned my addr 172.21.15.32:0/477402943 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aade4700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2a40b2190 0x7ff2a409f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2a9de2700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2a409e330 0x7ff2a40b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aa5e3700 1 -- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2a409e330 msgr2=0x7ff2a40b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aa5e3700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2a409e330 0x7ff2a40b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aa5e3700 1 -- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2a40b2190 msgr2=0x7ff2a409f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:55.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aa5e3700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2a40b2190 0x7ff2a409f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aa5e3700 1 -- 172.21.15.32:0/477402943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a409fb00 con 0x7ff2a40b1680 2021-04-09T18:17:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2aa5e3700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a40b1680 0x7ff2a40b0e30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7ff29c00b650 tx=0x7ff29c00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2a2ffd700 1 -- 172.21.15.32:0/477402943 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29c00bde0 con 0x7ff2a40b1680 2021-04-09T18:17:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2a2ffd700 1 -- 172.21.15.32:0/477402943 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29c000c90 con 0x7ff2a40b1680 2021-04-09T18:17:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a409fb00 con 0x7ff2a40b1680 2021-04-09T18:17:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2a2ffd700 1 -- 172.21.15.32:0/477402943 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29c002220 con 0x7ff2a40b1680 2021-04-09T18:17:55.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.929 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a409fb00 con 0x7ff2a40b1680 2021-04-09T18:17:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.930 7ff2a2ffd700 1 -- 172.21.15.32:0/477402943 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff29c009f90 con 0x7ff2a40b1680 2021-04-09T18:17:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.930 7ff2a2ffd700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff284018930 0x7ff28401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.930 7ff2a2ffd700 1 -- 172.21.15.32:0/477402943 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4757+0+0 (secure 0 0 0) 0x7ff29c0326e0 con 0x7ff2a40b1680 2021-04-09T18:17:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.930 7ff2a9de2700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff284018930 0x7ff28401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.931 7ff2a9de2700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff284018930 0x7ff28401ae10 crc :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.950 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a410d480 con 0x7ff2a40b1680 2021-04-09T18:17:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:54.953 7ff2a2ffd700 1 -- 172.21.15.32:0/477402943 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff29c0091e0 con 0x7ff2a40b1680 2021-04-09T18:17:55.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.135 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff2a410d480 con 0x7ff2a40b1680 2021-04-09T18:17:55.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.355 7ff2a2ffd700 1 -- 172.21.15.32:0/477402943 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v68) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff29c009f90 con 0x7ff2a40b1680 2021-04-09T18:17:55.434 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:17:55.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff284018930 msgr2=0x7ff28401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:55.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff284018930 0x7ff28401ae10 crc :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a40b1680 msgr2=0x7ff2a40b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:55.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a40b1680 0x7ff2a40b0e30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7ff29c00b650 tx=0x7ff29c00b9f0).stop 2021-04-09T18:17:55.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 shutdown_connections 2021-04-09T18:17:55.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff284018930 0x7ff28401ae10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2a40b2190 0x7ff2a409f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2a409e330 0x7ff2a40b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 --2- 172.21.15.32:0/477402943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2a40b1680 0x7ff2a40b0e30 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 >> 172.21.15.32:0/477402943 conn(0x7ff2a40c4290 msgr2=0x7ff2a40c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 shutdown_connections 2021-04-09T18:17:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.402 7ff2ac7c7700 1 -- 172.21.15.32:0/477402943 wait complete. 2021-04-09T18:17:55.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:17:55.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.568 7f9dd4dca700 1 Processor -- start 2021-04-09T18:17:55.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.569 7f9dd4dca700 1 -- start start 2021-04-09T18:17:55.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.569 7f9dd4dca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd00d7ca0 0x7f9dd00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.569 7f9dd4dca700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9dd010da00 0x7f9dd010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.569 7f9dd4dca700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd010d480 con 0x7f9dd010d010 2021-04-09T18:17:55.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.569 7f9dd4dca700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd00d9220 con 0x7f9dd010da00 2021-04-09T18:17:55.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.569 7f9dd4dca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd00d86c0 con 0x7f9dd00d7ca0 2021-04-09T18:17:55.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dcf314700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9dd010da00 0x7f9dd010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dce312700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd00d7ca0 0x7f9dd00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dceb13700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9dd010d010 0x7f9dd00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:53412/0 (socket says 172.21.15.32:53412) 2021-04-09T18:17:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dceb13700 1 -- 172.21.15.32:0/2248072515 learned_addr learned my addr 172.21.15.32:0/2248072515 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dcdb11700 1 -- 172.21.15.32:0/2248072515 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032678905 0 0) 0x7f9dc00010c0 con 0x7f9dd010d010 2021-04-09T18:17:55.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dce312700 1 -- 172.21.15.32:0/2248072515 >> v2:172.21.15.86:3300/0 conn(0x7f9dd010da00 msgr2=0x7f9dd010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dce312700 1 --2- 172.21.15.32:0/2248072515 >> v2:172.21.15.86:3300/0 conn(0x7f9dd010da00 0x7f9dd010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dce312700 1 -- 172.21.15.32:0/2248072515 >> v1:172.21.15.86:6789/0 conn(0x7f9dd010d010 legacy=0x7f9dd00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dce312700 1 -- 172.21.15.32:0/2248072515 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd0112100 con 0x7f9dd00d7ca0 2021-04-09T18:17:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.570 7f9dce312700 1 --2- 172.21.15.32:0/2248072515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd00d7ca0 0x7f9dd00d80a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f9dc40069d0 tx=0x7f9dc4009e40).ready entity=mon.2 client_cookie=6e38f2100055e1ee server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:55.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.571 7f9dcdb11700 1 -- 172.21.15.32:0/2248072515 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dc400b510 con 0x7f9dd00d7ca0 2021-04-09T18:17:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.571 7f9dcdb11700 1 -- 172.21.15.32:0/2248072515 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc400b890 con 0x7f9dd00d7ca0 2021-04-09T18:17:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.571 7f9dcdb11700 1 -- 172.21.15.32:0/2248072515 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dc400bd70 con 0x7f9dd00d7ca0 2021-04-09T18:17:55.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.571 7f9dd4dca700 1 -- 172.21.15.32:0/2248072515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd00d7ca0 msgr2=0x7f9dd00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:55.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.571 7f9dd4dca700 1 --2- 172.21.15.32:0/2248072515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd00d7ca0 0x7f9dd00d80a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f9dc40069d0 tx=0x7f9dc4009e40).stop 2021-04-09T18:17:55.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.572 7f9dd4dca700 1 -- 172.21.15.32:0/2248072515 shutdown_connections 2021-04-09T18:17:55.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.572 7f9dd4dca700 1 --2- 172.21.15.32:0/2248072515 >> v2:172.21.15.86:3300/0 conn(0x7f9dd010da00 0x7f9dd010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.572 7f9dd4dca700 1 --2- 172.21.15.32:0/2248072515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd00d7ca0 0x7f9dd00d80a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.572 7f9dd4dca700 1 -- 172.21.15.32:0/2248072515 >> 172.21.15.32:0/2248072515 conn(0x7f9dd00ce5f0 msgr2=0x7f9dd00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:55.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.572 7f9dd4dca700 1 -- 172.21.15.32:0/2248072515 shutdown_connections 2021-04-09T18:17:55.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.572 7f9dd4dca700 1 -- 172.21.15.32:0/2248072515 wait complete. 2021-04-09T18:17:55.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dd4dca700 1 Processor -- start 2021-04-09T18:17:55.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dd4dca700 1 -- start start 2021-04-09T18:17:55.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dd4dca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009c5b0 0x7f9dd009d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dd4dca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009c9b0 0x7f9dd009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dd4dca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd0090390 0x7f9dd0090790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dd4dca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd009e290 con 0x7f9dd009c5b0 2021-04-09T18:17:55.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dd4dca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd009cf90 con 0x7f9dd0090390 2021-04-09T18:17:55.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dd4dca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd0090db0 con 0x7f9dd009c9b0 2021-04-09T18:17:55.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dceb13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009c5b0 0x7f9dd009d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dceb13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009c5b0 0x7f9dd009d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45902/0 (socket says 172.21.15.32:45902) 2021-04-09T18:17:55.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dceb13700 1 -- 172.21.15.32:0/3704702237 learned_addr learned my addr 172.21.15.32:0/3704702237 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:55.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.592 7f9dce312700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009c9b0 0x7f9dd009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dcf314700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd0090390 0x7f9dd0090790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dceb13700 1 -- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009c9b0 msgr2=0x7f9dd009db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dceb13700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009c9b0 0x7f9dd009db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dceb13700 1 -- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd0090390 msgr2=0x7f9dd0090790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:55.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dceb13700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd0090390 0x7f9dd0090790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:55.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dceb13700 1 -- 172.21.15.32:0/3704702237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd0090f90 con 0x7f9dd009c5b0 2021-04-09T18:17:55.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dce312700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009c9b0 0x7f9dd009db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:55.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dceb13700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009c5b0 0x7f9dd009d510 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f9dc000a5d0 tx=0x7f9dc000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dbf7fe700 1 -- 172.21.15.32:0/3704702237 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dc0000c10 con 0x7f9dd009c5b0 2021-04-09T18:17:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dbf7fe700 1 -- 172.21.15.32:0/3704702237 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc00070d0 con 0x7f9dd009c5b0 2021-04-09T18:17:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd0090f90 con 0x7f9dd009c5b0 2021-04-09T18:17:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dbf7fe700 1 -- 172.21.15.32:0/3704702237 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dc00047b0 con 0x7f9dd009c5b0 2021-04-09T18:17:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.593 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd0090f90 con 0x7f9dd009c5b0 2021-04-09T18:17:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.594 7f9dbf7fe700 1 -- 172.21.15.32:0/3704702237 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9dc0004320 con 0x7f9dd009c5b0 2021-04-09T18:17:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.594 7f9dbf7fe700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db40188c0 0x7f9db401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.594 7f9dbf7fe700 1 -- 172.21.15.32:0/3704702237 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5111+0+0 (secure 0 0 0) 0x7f9dc0032750 con 0x7f9dd009c5b0 2021-04-09T18:17:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.594 7f9dce312700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db40188c0 0x7f9db401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.594 7f9dce312700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db40188c0 0x7f9db401adc0 crc :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:55.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.613 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd010d480 con 0x7f9dd009c5b0 2021-04-09T18:17:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.617 7f9dbf7fe700 1 -- 172.21.15.32:0/3704702237 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9dc0004780 con 0x7f9dd009c5b0 2021-04-09T18:17:55.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:55.799 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f9dd010d480 con 0x7f9dd009c5b0 2021-04-09T18:17:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.415 7f9dbf7fe700 1 -- 172.21.15.32:0/3704702237 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v72) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9dc0004320 con 0x7f9dd009c5b0 2021-04-09T18:17:56.509 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:17:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db40188c0 msgr2=0x7f9db401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:56.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db40188c0 0x7f9db401adc0 crc :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009c5b0 msgr2=0x7f9dd009d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:56.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009c5b0 0x7f9dd009d510 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f9dc000a5d0 tx=0x7f9dc000a970).stop 2021-04-09T18:17:56.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 shutdown_connections 2021-04-09T18:17:56.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db40188c0 0x7f9db401adc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd0090390 0x7f9dd0090790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009c9b0 0x7f9dd009db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 --2- 172.21.15.32:0/3704702237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009c5b0 0x7f9dd009d510 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 >> 172.21.15.32:0/3704702237 conn(0x7f9dd00b2fa0 msgr2=0x7f9dd00a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 shutdown_connections 2021-04-09T18:17:56.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.467 7f9dd4dca700 1 -- 172.21.15.32:0/3704702237 wait complete. 2021-04-09T18:17:56.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:17:56.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.695 7f7d65305700 1 Processor -- start 2021-04-09T18:17:56.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.695 7f7d65305700 1 -- start start 2021-04-09T18:17:56.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d65305700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d6010d050 0x7f7d600cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:56.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d65305700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7d600d84b0 0x7f7d6010da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:56.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d65305700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6010d500 con 0x7f7d600d0160 2021-04-09T18:17:56.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d65305700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d600d8d70 con 0x7f7d6010d050 2021-04-09T18:17:56.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d65305700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d600d9260 con 0x7f7d600d84b0 2021-04-09T18:17:56.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5f07d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d6010d050 0x7f7d600cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:56.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5e87c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7d600d0160 0x7f7d600d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55506/0 (socket says 172.21.15.32:55506) 2021-04-09T18:17:56.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5f07d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d6010d050 0x7f7d600cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45912/0 (socket says 172.21.15.32:45912) 2021-04-09T18:17:56.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5e87c700 1 -- 172.21.15.32:0/377520216 learned_addr learned my addr 172.21.15.32:0/377520216 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:56.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5f87e700 1 --2- 172.21.15.32:0/377520216 >> v2:172.21.15.86:3300/0 conn(0x7f7d600d84b0 0x7f7d6010da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:56.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5f07d700 1 -- 172.21.15.32:0/377520216 >> v2:172.21.15.86:3300/0 conn(0x7f7d600d84b0 msgr2=0x7f7d6010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:56.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5f07d700 1 --2- 172.21.15.32:0/377520216 >> v2:172.21.15.86:3300/0 conn(0x7f7d600d84b0 0x7f7d6010da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5f07d700 1 -- 172.21.15.32:0/377520216 >> v1:172.21.15.32:6789/0 conn(0x7f7d600d0160 legacy=0x7f7d600d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:56.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5f07d700 1 -- 172.21.15.32:0/377520216 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6010e0a0 con 0x7f7d6010d050 2021-04-09T18:17:56.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5e07b700 1 -- 172.21.15.32:0/377520216 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142750246 0 0) 0x7f7d54000ce0 con 0x7f7d600d0160 2021-04-09T18:17:56.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5f07d700 1 --2- 172.21.15.32:0/377520216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d6010d050 0x7f7d600cfb40 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f7d50006650 tx=0x7f7d50009a00).ready entity=mon.0 client_cookie=99b1f32dfce33e7e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:56.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5e07b700 1 -- 172.21.15.32:0/377520216 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d50009df0 con 0x7f7d6010d050 2021-04-09T18:17:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5e07b700 1 -- 172.21.15.32:0/377520216 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d500009b0 con 0x7f7d6010d050 2021-04-09T18:17:56.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.696 7f7d5e07b700 1 -- 172.21.15.32:0/377520216 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d50009df0 con 0x7f7d6010d050 2021-04-09T18:17:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.697 7f7d65305700 1 -- 172.21.15.32:0/377520216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d6010d050 msgr2=0x7f7d600cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.697 7f7d65305700 1 --2- 172.21.15.32:0/377520216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d6010d050 0x7f7d600cfb40 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f7d50006650 tx=0x7f7d50009a00).stop 2021-04-09T18:17:56.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.697 7f7d65305700 1 -- 172.21.15.32:0/377520216 shutdown_connections 2021-04-09T18:17:56.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.697 7f7d65305700 1 --2- 172.21.15.32:0/377520216 >> v2:172.21.15.86:3300/0 conn(0x7f7d600d84b0 0x7f7d6010da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.697 7f7d65305700 1 --2- 172.21.15.32:0/377520216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d6010d050 0x7f7d600cfb40 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.697 7f7d65305700 1 -- 172.21.15.32:0/377520216 >> 172.21.15.32:0/377520216 conn(0x7f7d600ce620 msgr2=0x7f7d600d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:56.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.697 7f7d65305700 1 -- 172.21.15.32:0/377520216 shutdown_connections 2021-04-09T18:17:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.697 7f7d65305700 1 -- 172.21.15.32:0/377520216 wait complete. 2021-04-09T18:17:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.717 7f7d65305700 1 Processor -- start 2021-04-09T18:17:56.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d65305700 1 -- start start 2021-04-09T18:17:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d65305700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d600ad600 0x7f7d6009ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:56.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d65305700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d6009b1b0 0x7f7d600ba220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d65305700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d600ba840 0x7f7d600a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d65305700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6009fa80 con 0x7f7d600ba840 2021-04-09T18:17:56.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d65305700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6009b5b0 con 0x7f7d6009b1b0 2021-04-09T18:17:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d65305700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d600bac40 con 0x7f7d600ad600 2021-04-09T18:17:56.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d5f87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d600ba840 0x7f7d600a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:56.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d5f87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d600ba840 0x7f7d600a91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45920/0 (socket says 172.21.15.32:45920) 2021-04-09T18:17:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d5f87e700 1 -- 172.21.15.32:0/1958905827 learned_addr learned my addr 172.21.15.32:0/1958905827 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:56.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d5e87c700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d6009b1b0 0x7f7d600ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:56.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.718 7f7d5f07d700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d600ad600 0x7f7d6009ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:56.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d5f87e700 1 -- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d600ad600 msgr2=0x7f7d6009ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:56.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d5f87e700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d600ad600 0x7f7d6009ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d5f87e700 1 -- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d6009b1b0 msgr2=0x7f7d600ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:56.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d5f87e700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d6009b1b0 0x7f7d600ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d5f87e700 1 -- 172.21.15.32:0/1958905827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d600a97c0 con 0x7f7d600ba840 2021-04-09T18:17:56.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d5f87e700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d600ba840 0x7f7d600a91a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f7d580099a0 tx=0x7f7d5800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:56.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d47fff700 1 -- 172.21.15.32:0/1958905827 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d580066f0 con 0x7f7d600ba840 2021-04-09T18:17:56.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d47fff700 1 -- 172.21.15.32:0/1958905827 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d58003030 con 0x7f7d600ba840 2021-04-09T18:17:56.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d65305700 1 -- 172.21.15.32:0/1958905827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d600a97c0 con 0x7f7d600ba840 2021-04-09T18:17:56.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d47fff700 1 -- 172.21.15.32:0/1958905827 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d58001470 con 0x7f7d600ba840 2021-04-09T18:17:56.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.719 7f7d65305700 1 -- 172.21.15.32:0/1958905827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d600a97c0 con 0x7f7d600ba840 2021-04-09T18:17:56.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.720 7f7d47fff700 1 -- 172.21.15.32:0/1958905827 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7d580011f0 con 0x7f7d600ba840 2021-04-09T18:17:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.720 7f7d47fff700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d400188c0 0x7f7d4001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:56.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.720 7f7d47fff700 1 -- 172.21.15.32:0/1958905827 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f7d58032ac0 con 0x7f7d600ba840 2021-04-09T18:17:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.720 7f7d5f07d700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d400188c0 0x7f7d4001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:56.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.721 7f7d5f07d700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d400188c0 0x7f7d4001adc0 crc :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:56.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.740 7f7d65305700 1 -- 172.21.15.32:0/1958905827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d600aa330 con 0x7f7d600ba840 2021-04-09T18:17:56.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.743 7f7d47fff700 1 -- 172.21.15.32:0/1958905827 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7d5800be30 con 0x7f7d600ba840 2021-04-09T18:17:56.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.926 7f7d65305700 1 -- 172.21.15.32:0/1958905827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f7d600aa330 con 0x7f7d600ba840 2021-04-09T18:17:56.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.930 7f7d47fff700 1 -- 172.21.15.32:0/1958905827 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 12 and data pool 13 v77) v1 ==== 189+0+0 (secure 0 0 0) 0x7f7d5801f870 con 0x7f7d600ba840 2021-04-09T18:17:56.972 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 12 and data pool 13 2021-04-09T18:17:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.936 7f7d65305700 1 -- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d400188c0 msgr2=0x7f7d4001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.936 7f7d65305700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d400188c0 0x7f7d4001adc0 crc :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.936 7f7d65305700 1 -- 172.21.15.32:0/1958905827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d600ba840 msgr2=0x7f7d600a91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:56.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.936 7f7d65305700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d600ba840 0x7f7d600a91a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f7d580099a0 tx=0x7f7d5800bcf0).stop 2021-04-09T18:17:56.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.938 7f7d65305700 1 -- 172.21.15.32:0/1958905827 shutdown_connections 2021-04-09T18:17:56.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.938 7f7d65305700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d400188c0 0x7f7d4001adc0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.938 7f7d65305700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d600ba840 0x7f7d600a91a0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.938 7f7d65305700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d6009b1b0 0x7f7d600ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.938 7f7d65305700 1 --2- 172.21.15.32:0/1958905827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d600ad600 0x7f7d6009ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:56.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.938 7f7d65305700 1 -- 172.21.15.32:0/1958905827 >> 172.21.15.32:0/1958905827 conn(0x7f7d600ae6f0 msgr2=0x7f7d60097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:56.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.939 7f7d65305700 1 -- 172.21.15.32:0/1958905827 shutdown_connections 2021-04-09T18:17:56.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:56.939 7f7d65305700 1 -- 172.21.15.32:0/1958905827 wait complete. 2021-04-09T18:17:56.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:17:57.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.114 7f655f594700 1 Processor -- start 2021-04-09T18:17:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.114 7f655f594700 1 -- start start 2021-04-09T18:17:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655f594700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f655810d010 0x7f65580d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655f594700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65580d8490 0x7f655810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655f594700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655810d480 con 0x7f65580d1da0 2021-04-09T18:17:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655f594700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65580d8d30 con 0x7f65580d8490 2021-04-09T18:17:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655f594700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65580d9220 con 0x7f655810d010 2021-04-09T18:17:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655dbb1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65580d8490 0x7f655810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655dbb1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65580d8490 0x7f655810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45926/0 (socket says 172.21.15.32:45926) 2021-04-09T18:17:57.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655dbb1700 1 -- 172.21.15.32:0/4294824853 learned_addr learned my addr 172.21.15.32:0/4294824853 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:57.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655d3b0700 1 --2- 172.21.15.32:0/4294824853 >> v2:172.21.15.86:3300/0 conn(0x7f655810d010 0x7f65580d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655dbb1700 1 -- 172.21.15.32:0/4294824853 >> v2:172.21.15.86:3300/0 conn(0x7f655810d010 msgr2=0x7f65580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:57.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655dbb1700 1 --2- 172.21.15.32:0/4294824853 >> v2:172.21.15.86:3300/0 conn(0x7f655810d010 0x7f65580d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655dbb1700 1 -- 172.21.15.32:0/4294824853 >> v1:172.21.15.86:6789/0 conn(0x7f65580d1da0 legacy=0x7f65580d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655dbb1700 1 -- 172.21.15.32:0/4294824853 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655810e020 con 0x7f65580d8490 2021-04-09T18:17:57.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655d3b0700 1 --2- 172.21.15.32:0/4294824853 >> v2:172.21.15.86:3300/0 conn(0x7f655810d010 0x7f65580d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:17:57.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.115 7f655dbb1700 1 --2- 172.21.15.32:0/4294824853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65580d8490 0x7f655810da00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7f6550006f00 tx=0x7f6550009f10).ready entity=mon.0 client_cookie=c1bfa3ac1229aaf2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:57.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.116 7f654ffff700 1 -- 172.21.15.32:0/4294824853 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f655000b3b0 con 0x7f65580d8490 2021-04-09T18:17:57.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.116 7f654ffff700 1 -- 172.21.15.32:0/4294824853 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655000b730 con 0x7f65580d8490 2021-04-09T18:17:57.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.116 7f654ffff700 1 -- 172.21.15.32:0/4294824853 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f655000bc10 con 0x7f65580d8490 2021-04-09T18:17:57.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.116 7f655f594700 1 -- 172.21.15.32:0/4294824853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65580d8490 msgr2=0x7f655810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.116 7f655f594700 1 --2- 172.21.15.32:0/4294824853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65580d8490 0x7f655810da00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7f6550006f00 tx=0x7f6550009f10).stop 2021-04-09T18:17:57.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.116 7f655f594700 1 -- 172.21.15.32:0/4294824853 shutdown_connections 2021-04-09T18:17:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.116 7f655f594700 1 --2- 172.21.15.32:0/4294824853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65580d8490 0x7f655810da00 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.116 7f655f594700 1 --2- 172.21.15.32:0/4294824853 >> v2:172.21.15.86:3300/0 conn(0x7f655810d010 0x7f65580d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.116 7f655f594700 1 -- 172.21.15.32:0/4294824853 >> 172.21.15.32:0/4294824853 conn(0x7f65580ce5f0 msgr2=0x7f65580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.117 7f655f594700 1 -- 172.21.15.32:0/4294824853 shutdown_connections 2021-04-09T18:17:57.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.117 7f655f594700 1 -- 172.21.15.32:0/4294824853 wait complete. 2021-04-09T18:17:57.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.137 7f655f594700 1 Processor -- start 2021-04-09T18:17:57.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.137 7f655f594700 1 -- start start 2021-04-09T18:17:57.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.137 7f655f594700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65581a5210 0x7f65581a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.137 7f655f594700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65581a6340 0x7f65581a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.137 7f655f594700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65580abaa0 0x7f65580abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.137 7f655f594700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65581a6d60 con 0x7f65581a5210 2021-04-09T18:17:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.137 7f655f594700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65580ac4c0 con 0x7f65581a6340 2021-04-09T18:17:57.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.137 7f655f594700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65580ac6e0 con 0x7f65580abaa0 2021-04-09T18:17:57.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.137 7f655d3b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65581a5210 0x7f65581a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655d3b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65581a5210 0x7f65581a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45936/0 (socket says 172.21.15.32:45936) 2021-04-09T18:17:57.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655d3b0700 1 -- 172.21.15.32:0/3387999004 learned_addr learned my addr 172.21.15.32:0/3387999004 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:57.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655cbaf700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65581a6340 0x7f65581a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655dbb1700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65580abaa0 0x7f65580abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655d3b0700 1 -- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65580abaa0 msgr2=0x7f65580abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655d3b0700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65580abaa0 0x7f65580abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655d3b0700 1 -- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65581a6340 msgr2=0x7f65581a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655d3b0700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65581a6340 0x7f65581a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655d3b0700 1 -- 172.21.15.32:0/3387999004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655817a720 con 0x7f65581a5210 2021-04-09T18:17:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655d3b0700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65581a5210 0x7f65581a56c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f6554006f20 tx=0x7f655400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:57.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f654dffb700 1 -- 172.21.15.32:0/3387999004 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f655400bd80 con 0x7f65581a5210 2021-04-09T18:17:57.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f654dffb700 1 -- 172.21.15.32:0/3387999004 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6554000c90 con 0x7f65581a5210 2021-04-09T18:17:57.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655f594700 1 -- 172.21.15.32:0/3387999004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655817a720 con 0x7f65581a5210 2021-04-09T18:17:57.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f654dffb700 1 -- 172.21.15.32:0/3387999004 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65540022b0 con 0x7f65581a5210 2021-04-09T18:17:57.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.138 7f655f594700 1 -- 172.21.15.32:0/3387999004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655817a720 con 0x7f65581a5210 2021-04-09T18:17:57.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.139 7f654dffb700 1 -- 172.21.15.32:0/3387999004 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f655400a000 con 0x7f65581a5210 2021-04-09T18:17:57.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.139 7f654dffb700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65380188c0 0x7f653801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.139 7f654dffb700 1 -- 172.21.15.32:0/3387999004 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f65540329a0 con 0x7f65581a5210 2021-04-09T18:17:57.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.139 7f655cbaf700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65380188c0 0x7f653801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.139 7f655cbaf700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65380188c0 0x7f653801adc0 crc :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:57.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.159 7f655f594700 1 -- 172.21.15.32:0/3387999004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65581a6d60 con 0x7f65581a5210 2021-04-09T18:17:57.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.163 7f654dffb700 1 -- 172.21.15.32:0/3387999004 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6554006bb0 con 0x7f65581a5210 2021-04-09T18:17:57.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.352 7f655f594700 1 -- 172.21.15.32:0/3387999004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65581a6d60 con 0x7f65581a5210 2021-04-09T18:17:57.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.353 7f654dffb700 1 -- 172.21.15.32:0/3387999004 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f6554006bb0 con 0x7f65581a5210 2021-04-09T18:17:57.411 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:57.414 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":73,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:17:56.928194","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:17:55.170353","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-04-09 18:17:55.834924","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:17:57.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 -- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65380188c0 msgr2=0x7f653801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65380188c0 0x7f653801adc0 crc :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 -- 172.21.15.32:0/3387999004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65581a5210 msgr2=0x7f65581a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65581a5210 0x7f65581a56c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f6554006f20 tx=0x7f655400b990).stop 2021-04-09T18:17:57.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 -- 172.21.15.32:0/3387999004 shutdown_connections 2021-04-09T18:17:57.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65380188c0 0x7f653801adc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65580abaa0 0x7f65580abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65581a6340 0x7f65581a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 --2- 172.21.15.32:0/3387999004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65581a5210 0x7f65581a56c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.355 7f655f594700 1 -- 172.21.15.32:0/3387999004 >> 172.21.15.32:0/3387999004 conn(0x7f65581a99c0 msgr2=0x7f655817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:57.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.356 7f655f594700 1 -- 172.21.15.32:0/3387999004 shutdown_connections 2021-04-09T18:17:57.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.356 7f655f594700 1 -- 172.21.15.32:0/3387999004 wait complete. 2021-04-09T18:17:57.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:17:57.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.564 7f38abea0700 1 Processor -- start 2021-04-09T18:17:57.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.564 7f38abea0700 1 -- start start 2021-04-09T18:17:57.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38abea0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38a410d010 0x7f38a40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38abea0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f38a40d1da0 0x7f38a40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38abea0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40d82c0 0x7f38a410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38abea0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a40d8d30 con 0x7f38a410d010 2021-04-09T18:17:57.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38abea0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a40d9220 con 0x7f38a40d1da0 2021-04-09T18:17:57.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38abea0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a410d4b0 con 0x7f38a40d82c0 2021-04-09T18:17:57.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a9cbc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38a410d010 0x7f38a40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a9cbc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38a410d010 0x7f38a40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45940/0 (socket says 172.21.15.32:45940) 2021-04-09T18:17:57.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a9cbc700 1 -- 172.21.15.32:0/2006414348 learned_addr learned my addr 172.21.15.32:0/2006414348 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:57.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a94bb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f38a40d1da0 0x7f38a40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38aa4bd700 1 --2- 172.21.15.32:0/2006414348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40d82c0 0x7f38a410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a9cbc700 1 -- 172.21.15.32:0/2006414348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40d82c0 msgr2=0x7f38a410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:57.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a9cbc700 1 --2- 172.21.15.32:0/2006414348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40d82c0 0x7f38a410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a9cbc700 1 -- 172.21.15.32:0/2006414348 >> v2:172.21.15.86:3300/0 conn(0x7f38a40d1da0 msgr2=0x7f38a40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:57.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a9cbc700 1 --2- 172.21.15.32:0/2006414348 >> v2:172.21.15.86:3300/0 conn(0x7f38a40d1da0 0x7f38a40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a9cbc700 1 -- 172.21.15.32:0/2006414348 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a40d86c0 con 0x7f38a410d010 2021-04-09T18:17:57.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a94bb700 1 --2- 172.21.15.32:0/2006414348 >> v2:172.21.15.86:3300/0 conn(0x7f38a40d1da0 0x7f38a40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:57.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.565 7f38a9cbc700 1 --2- 172.21.15.32:0/2006414348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a410d010 0x7f38a40d1780 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f38a0006700 tx=0x7f38a0009b80).ready entity=mon.0 client_cookie=96bed131b83774a0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:57.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38a8cba700 1 -- 172.21.15.32:0/2006414348 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a000c0c0 con 0x7f38a410d010 2021-04-09T18:17:57.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38a8cba700 1 -- 172.21.15.32:0/2006414348 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a000c440 con 0x7f38a410d010 2021-04-09T18:17:57.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38a8cba700 1 -- 172.21.15.32:0/2006414348 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38a000c920 con 0x7f38a410d010 2021-04-09T18:17:57.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38abea0700 1 -- 172.21.15.32:0/2006414348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a410d010 msgr2=0x7f38a40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38abea0700 1 --2- 172.21.15.32:0/2006414348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a410d010 0x7f38a40d1780 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f38a0006700 tx=0x7f38a0009b80).stop 2021-04-09T18:17:57.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38abea0700 1 -- 172.21.15.32:0/2006414348 shutdown_connections 2021-04-09T18:17:57.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38abea0700 1 --2- 172.21.15.32:0/2006414348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40d82c0 0x7f38a410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38abea0700 1 --2- 172.21.15.32:0/2006414348 >> v2:172.21.15.86:3300/0 conn(0x7f38a40d1da0 0x7f38a40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38abea0700 1 --2- 172.21.15.32:0/2006414348 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a410d010 0x7f38a40d1780 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.566 7f38abea0700 1 -- 172.21.15.32:0/2006414348 >> 172.21.15.32:0/2006414348 conn(0x7f38a40ce5f0 msgr2=0x7f38a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:57.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.567 7f38abea0700 1 -- 172.21.15.32:0/2006414348 shutdown_connections 2021-04-09T18:17:57.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.567 7f38abea0700 1 -- 172.21.15.32:0/2006414348 wait complete. 2021-04-09T18:17:57.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.587 7f38abea0700 1 Processor -- start 2021-04-09T18:17:57.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.587 7f38abea0700 1 -- start start 2021-04-09T18:17:57.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.587 7f38abea0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40a2570 0x7f38a40ba6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.587 7f38abea0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a40bad00 0x7f38a41a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.587 7f38abea0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38a41a0720 0x7f38a40a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.587 7f38abea0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a41a0ba0 con 0x7f38a40bad00 2021-04-09T18:17:57.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.587 7f38abea0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a4185890 con 0x7f38a41a0720 2021-04-09T18:17:57.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.587 7f38abea0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a40a0dc0 con 0x7f38a40a2570 2021-04-09T18:17:57.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a94bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a40bad00 0x7f38a41a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a94bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a40bad00 0x7f38a41a0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45948/0 (socket says 172.21.15.32:45948) 2021-04-09T18:17:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a94bb700 1 -- 172.21.15.32:0/1262806868 learned_addr learned my addr 172.21.15.32:0/1262806868 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38aa4bd700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38a41a0720 0x7f38a40a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a9cbc700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40a2570 0x7f38a40ba6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a94bb700 1 -- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40a2570 msgr2=0x7f38a40ba6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a94bb700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40a2570 0x7f38a40ba6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a94bb700 1 -- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38a41a0720 msgr2=0x7f38a40a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a94bb700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38a41a0720 0x7f38a40a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a94bb700 1 -- 172.21.15.32:0/1262806868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a40a0fe0 con 0x7f38a40bad00 2021-04-09T18:17:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38aa4bd700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38a41a0720 0x7f38a40a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38a94bb700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a40bad00 0x7f38a41a0100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f3894006f40 tx=0x7f389400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f389a7fc700 1 -- 172.21.15.32:0/1262806868 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f389400bd80 con 0x7f38a40bad00 2021-04-09T18:17:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f389a7fc700 1 -- 172.21.15.32:0/1262806868 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3894000c90 con 0x7f38a40bad00 2021-04-09T18:17:57.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38abea0700 1 -- 172.21.15.32:0/1262806868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a40a0fe0 con 0x7f38a40bad00 2021-04-09T18:17:57.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f389a7fc700 1 -- 172.21.15.32:0/1262806868 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3894002220 con 0x7f38a40bad00 2021-04-09T18:17:57.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.588 7f38abea0700 1 -- 172.21.15.32:0/1262806868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a40a0fe0 con 0x7f38a40bad00 2021-04-09T18:17:57.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.589 7f389a7fc700 1 -- 172.21.15.32:0/1262806868 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f389400a000 con 0x7f38a40bad00 2021-04-09T18:17:57.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.589 7f389a7fc700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f388c018930 0x7f388c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.589 7f389a7fc700 1 -- 172.21.15.32:0/1262806868 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f38940329e0 con 0x7f38a40bad00 2021-04-09T18:17:57.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.589 7f38a9cbc700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f388c018930 0x7f388c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:57.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.589 7f38a9cbc700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f388c018930 0x7f388c01ae10 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:57.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.609 7f38abea0700 1 -- 172.21.15.32:0/1262806868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a4182d10 con 0x7f38a40bad00 2021-04-09T18:17:57.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.612 7f389a7fc700 1 -- 172.21.15.32:0/1262806868 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3894000c90 con 0x7f38a40bad00 2021-04-09T18:17:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.802 7f38abea0700 1 -- 172.21.15.32:0/1262806868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38a4182d10 con 0x7f38a40bad00 2021-04-09T18:17:57.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.802 7f389a7fc700 1 -- 172.21.15.32:0/1262806868 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f3894006b00 con 0x7f38a40bad00 2021-04-09T18:17:57.841 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:57.841 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":73,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:17:56.928194","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:17:55.170353","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-04-09 18:17:55.834924","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:17:57.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.804 7f38abea0700 1 -- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f388c018930 msgr2=0x7f388c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.804 7f38abea0700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f388c018930 0x7f388c01ae10 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.804 7f38abea0700 1 -- 172.21.15.32:0/1262806868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a40bad00 msgr2=0x7f38a41a0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:57.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.804 7f38abea0700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a40bad00 0x7f38a41a0100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f3894006f40 tx=0x7f389400b990).stop 2021-04-09T18:17:57.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.805 7f38abea0700 1 -- 172.21.15.32:0/1262806868 shutdown_connections 2021-04-09T18:17:57.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.805 7f38abea0700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f388c018930 0x7f388c01ae10 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.805 7f38abea0700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a40bad00 0x7f38a41a0100 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.805 7f38abea0700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38a41a0720 0x7f38a40a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.805 7f38abea0700 1 --2- 172.21.15.32:0/1262806868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38a40a2570 0x7f38a40ba6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:57.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.805 7f38abea0700 1 -- 172.21.15.32:0/1262806868 >> 172.21.15.32:0/1262806868 conn(0x7f38a409de50 msgr2=0x7f38a419f010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:57.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.805 7f38abea0700 1 -- 172.21.15.32:0/1262806868 shutdown_connections 2021-04-09T18:17:57.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:57.805 7f38abea0700 1 -- 172.21.15.32:0/1262806868 wait complete. 2021-04-09T18:17:57.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:17:58.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.018 7fc67debc700 1 Processor -- start 2021-04-09T18:17:58.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.018 7fc67debc700 1 -- start start 2021-04-09T18:17:58.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc67debc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc67810d010 0x7fc6780d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:58.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc67debc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc6780d1da0 0x7fc6780d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:58.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc67debc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67810d480 con 0x7fc6780d82c0 2021-04-09T18:17:58.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc67debc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6780d8d30 con 0x7fc67810d010 2021-04-09T18:17:58.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc67debc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6780d9220 con 0x7fc6780d1da0 2021-04-09T18:17:58.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc6777fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc67810d010 0x7fc6780d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc677fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc6780d82c0 0x7fc67810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55546/0 (socket says 172.21.15.32:55546) 2021-04-09T18:17:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc677fff700 1 -- 172.21.15.32:0/1671415189 learned_addr learned my addr 172.21.15.32:0/1671415189 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc676ffd700 1 --2- 172.21.15.32:0/1671415189 >> v2:172.21.15.86:3300/0 conn(0x7fc6780d1da0 0x7fc6780d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc6777fe700 1 -- 172.21.15.32:0/1671415189 >> v2:172.21.15.86:3300/0 conn(0x7fc6780d1da0 msgr2=0x7fc6780d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc6777fe700 1 --2- 172.21.15.32:0/1671415189 >> v2:172.21.15.86:3300/0 conn(0x7fc6780d1da0 0x7fc6780d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:58.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc6777fe700 1 -- 172.21.15.32:0/1671415189 >> v1:172.21.15.32:6789/0 conn(0x7fc6780d82c0 legacy=0x7fc67810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:58.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc6767fc700 1 -- 172.21.15.32:0/1671415189 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573988718 0 0) 0x7fc670000d30 con 0x7fc6780d82c0 2021-04-09T18:17:58.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.019 7fc6777fe700 1 -- 172.21.15.32:0/1671415189 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6780d86f0 con 0x7fc67810d010 2021-04-09T18:17:58.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.020 7fc6777fe700 1 --2- 172.21.15.32:0/1671415189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67810d010 0x7fc6780d1780 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fc668006650 tx=0x7fc668009a00).ready entity=mon.0 client_cookie=ccc2844939eefa9f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:58.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.020 7fc6767fc700 1 -- 172.21.15.32:0/1671415189 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc668009df0 con 0x7fc67810d010 2021-04-09T18:17:58.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.020 7fc6767fc700 1 -- 172.21.15.32:0/1671415189 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6680009b0 con 0x7fc67810d010 2021-04-09T18:17:58.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.020 7fc6767fc700 1 -- 172.21.15.32:0/1671415189 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc66800c2b0 con 0x7fc67810d010 2021-04-09T18:17:58.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.020 7fc67debc700 1 -- 172.21.15.32:0/1671415189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67810d010 msgr2=0x7fc6780d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:58.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.020 7fc67debc700 1 --2- 172.21.15.32:0/1671415189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67810d010 0x7fc6780d1780 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fc668006650 tx=0x7fc668009a00).stop 2021-04-09T18:17:58.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.021 7fc67debc700 1 -- 172.21.15.32:0/1671415189 shutdown_connections 2021-04-09T18:17:58.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.021 7fc67debc700 1 --2- 172.21.15.32:0/1671415189 >> v2:172.21.15.86:3300/0 conn(0x7fc6780d1da0 0x7fc6780d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:58.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.021 7fc67debc700 1 --2- 172.21.15.32:0/1671415189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67810d010 0x7fc6780d1780 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:58.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.021 7fc67debc700 1 -- 172.21.15.32:0/1671415189 >> 172.21.15.32:0/1671415189 conn(0x7fc6780ce5f0 msgr2=0x7fc6780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:58.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.021 7fc67debc700 1 -- 172.21.15.32:0/1671415189 shutdown_connections 2021-04-09T18:17:58.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.021 7fc67debc700 1 -- 172.21.15.32:0/1671415189 wait complete. 2021-04-09T18:17:58.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.041 7fc67debc700 1 Processor -- start 2021-04-09T18:17:58.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.041 7fc67debc700 1 -- start start 2021-04-09T18:17:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.041 7fc67debc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc67809e530 0x7fc67809b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.041 7fc67debc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67809e930 0x7fc67809f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:58.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.041 7fc67debc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc67809f990 0x7fc678094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:58.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.041 7fc67debc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67809edf0 con 0x7fc67809e930 2021-04-09T18:17:58.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.041 7fc67debc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67809fd90 con 0x7fc67809f990 2021-04-09T18:17:58.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.041 7fc67debc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc67809bd40 con 0x7fc67809e530 2021-04-09T18:17:58.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc676ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67809e930 0x7fc67809f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:58.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc676ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67809e930 0x7fc67809f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45960/0 (socket says 172.21.15.32:45960) 2021-04-09T18:17:58.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc676ffd700 1 -- 172.21.15.32:0/2301968309 learned_addr learned my addr 172.21.15.32:0/2301968309 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:58.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc6777fe700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc67809e530 0x7fc67809b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:58.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc677fff700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc67809f990 0x7fc678094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:58.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc676ffd700 1 -- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc67809e530 msgr2=0x7fc67809b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:58.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc676ffd700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc67809e530 0x7fc67809b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:58.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc676ffd700 1 -- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc67809f990 msgr2=0x7fc678094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:58.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc676ffd700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc67809f990 0x7fc678094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:58.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc676ffd700 1 -- 172.21.15.32:0/2301968309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc678095290 con 0x7fc67809e930 2021-04-09T18:17:58.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc676ffd700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67809e930 0x7fc67809f370 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7fc66400b580 tx=0x7fc66400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:58.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc66ffff700 1 -- 172.21.15.32:0/2301968309 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6640064d0 con 0x7fc67809e930 2021-04-09T18:17:58.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc66ffff700 1 -- 172.21.15.32:0/2301968309 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66400bd10 con 0x7fc67809e930 2021-04-09T18:17:58.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc67debc700 1 -- 172.21.15.32:0/2301968309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc678095290 con 0x7fc67809e930 2021-04-09T18:17:58.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc66ffff700 1 -- 172.21.15.32:0/2301968309 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc664002030 con 0x7fc67809e930 2021-04-09T18:17:58.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.042 7fc67debc700 1 -- 172.21.15.32:0/2301968309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc678095290 con 0x7fc67809e930 2021-04-09T18:17:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.043 7fc66ffff700 1 -- 172.21.15.32:0/2301968309 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc6640052e0 con 0x7fc67809e930 2021-04-09T18:17:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.043 7fc66ffff700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc654018930 0x7fc65401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.043 7fc66ffff700 1 -- 172.21.15.32:0/2301968309 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc6640329d0 con 0x7fc67809e930 2021-04-09T18:17:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.043 7fc6777fe700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc654018930 0x7fc65401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.044 7fc6777fe700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc654018930 0x7fc65401ae10 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.063 7fc67debc700 1 -- 172.21.15.32:0/2301968309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc678095b10 con 0x7fc67809e930 2021-04-09T18:17:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.066 7fc66ffff700 1 -- 172.21.15.32:0/2301968309 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc664006820 con 0x7fc67809e930 2021-04-09T18:17:58.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.260 7fc67debc700 1 -- 172.21.15.32:0/2301968309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fc678095b10 con 0x7fc67809e930 2021-04-09T18:17:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.933 7fc66ffff700 1 -- 172.21.15.32:0/2301968309 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v79) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc6640052e0 con 0x7fc67809e930 2021-04-09T18:17:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.977 7fc67debc700 1 -- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc654018930 msgr2=0x7fc65401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.977 7fc67debc700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc654018930 0x7fc65401ae10 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 -- 172.21.15.32:0/2301968309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67809e930 msgr2=0x7fc67809f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67809e930 0x7fc67809f370 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7fc66400b580 tx=0x7fc66400b920).stop 2021-04-09T18:17:59.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 -- 172.21.15.32:0/2301968309 shutdown_connections 2021-04-09T18:17:59.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc654018930 0x7fc65401ae10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc67809e930 0x7fc67809f370 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc67809f990 0x7fc678094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 --2- 172.21.15.32:0/2301968309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc67809e530 0x7fc67809b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 -- 172.21.15.32:0/2301968309 >> 172.21.15.32:0/2301968309 conn(0x7fc6780c4290 msgr2=0x7fc6780c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 -- 172.21.15.32:0/2301968309 shutdown_connections 2021-04-09T18:17:59.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:58.978 7fc67debc700 1 -- 172.21.15.32:0/2301968309 wait complete. 2021-04-09T18:17:59.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:17:59.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.142 7f41ff7bf700 1 Processor -- start 2021-04-09T18:17:59.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.142 7f41ff7bf700 1 -- start start 2021-04-09T18:17:59.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41ff7bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41f810d010 0x7f41f80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41ff7bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f80d1da0 0x7f41f80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41ff7bf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f810d480 con 0x7f41f80d82c0 2021-04-09T18:17:59.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41ff7bf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f80d9220 con 0x7f41f810d010 2021-04-09T18:17:59.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41ff7bf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f80d86c0 con 0x7f41f80d1da0 2021-04-09T18:17:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fd5db700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f41f810d010 0x7f41f80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fdddc700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f41f80d82c0 0x7f41f810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55560/0 (socket says 172.21.15.32:55560) 2021-04-09T18:17:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fdddc700 1 -- 172.21.15.32:0/2974950020 learned_addr learned my addr 172.21.15.32:0/2974950020 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:17:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fcdda700 1 --2- 172.21.15.32:0/2974950020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f80d1da0 0x7f41f80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fd5db700 1 -- 172.21.15.32:0/2974950020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f80d1da0 msgr2=0x7f41f80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fd5db700 1 --2- 172.21.15.32:0/2974950020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f80d1da0 0x7f41f80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fd5db700 1 -- 172.21.15.32:0/2974950020 >> v1:172.21.15.32:6789/0 conn(0x7f41f80d82c0 legacy=0x7f41f810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fd5db700 1 -- 172.21.15.32:0/2974950020 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f810e1f0 con 0x7f41f810d010 2021-04-09T18:17:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41effff700 1 -- 172.21.15.32:0/2974950020 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604856427 0 0) 0x7f41f0000d30 con 0x7f41f80d82c0 2021-04-09T18:17:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fcdda700 1 --2- 172.21.15.32:0/2974950020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f80d1da0 0x7f41f80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41fd5db700 1 --2- 172.21.15.32:0/2974950020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f810d010 0x7f41f80d1780 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f41f4006650 tx=0x7f41f4009a80).ready entity=mon.0 client_cookie=c206a5e5fcd3fdff server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.143 7f41effff700 1 -- 172.21.15.32:0/2974950020 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41f40009b0 con 0x7f41f810d010 2021-04-09T18:17:59.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41effff700 1 -- 172.21.15.32:0/2974950020 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f400c0c0 con 0x7f41f810d010 2021-04-09T18:17:59.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41effff700 1 -- 172.21.15.32:0/2974950020 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41f400c510 con 0x7f41f810d010 2021-04-09T18:17:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41ff7bf700 1 -- 172.21.15.32:0/2974950020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f810d010 msgr2=0x7f41f80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41ff7bf700 1 --2- 172.21.15.32:0/2974950020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f810d010 0x7f41f80d1780 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f41f4006650 tx=0x7f41f4009a80).stop 2021-04-09T18:17:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41ff7bf700 1 -- 172.21.15.32:0/2974950020 shutdown_connections 2021-04-09T18:17:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41ff7bf700 1 --2- 172.21.15.32:0/2974950020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f80d1da0 0x7f41f80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41ff7bf700 1 --2- 172.21.15.32:0/2974950020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f810d010 0x7f41f80d1780 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41ff7bf700 1 -- 172.21.15.32:0/2974950020 >> 172.21.15.32:0/2974950020 conn(0x7f41f80ce5f0 msgr2=0x7f41f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:59.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41ff7bf700 1 -- 172.21.15.32:0/2974950020 shutdown_connections 2021-04-09T18:17:59.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.144 7f41ff7bf700 1 -- 172.21.15.32:0/2974950020 wait complete. 2021-04-09T18:17:59.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.165 7f41ff7bf700 1 Processor -- start 2021-04-09T18:17:59.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.165 7f41ff7bf700 1 -- start start 2021-04-09T18:17:59.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.165 7f41ff7bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f81acea0 0x7f41f81ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.165 7f41ff7bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f819b2c0 0x7f41f8184da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.165 7f41ff7bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f81853c0 0x7f41f8185fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.165 7f41ff7bf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f81857c0 con 0x7f41f819b2c0 2021-04-09T18:17:59.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.165 7f41ff7bf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f8185a40 con 0x7f41f81acea0 2021-04-09T18:17:59.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.165 7f41ff7bf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f81865e0 con 0x7f41f81853c0 2021-04-09T18:17:59.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fcdda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f819b2c0 0x7f41f8184da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fcdda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f819b2c0 0x7f41f8184da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45978/0 (socket says 172.21.15.32:45978) 2021-04-09T18:17:59.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fcdda700 1 -- 172.21.15.32:0/2382428188 learned_addr learned my addr 172.21.15.32:0/2382428188 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:59.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fd5db700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f81acea0 0x7f41f81ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fdddc700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f81853c0 0x7f41f8185fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fcdda700 1 -- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f81853c0 msgr2=0x7f41f8185fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fcdda700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f81853c0 0x7f41f8185fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fcdda700 1 -- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f81acea0 msgr2=0x7f41f81ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fcdda700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f81acea0 0x7f41f81ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fcdda700 1 -- 172.21.15.32:0/2382428188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f8186800 con 0x7f41f819b2c0 2021-04-09T18:17:59.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fdddc700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f81853c0 0x7f41f8185fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:59.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fd5db700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f81acea0 0x7f41f81ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:17:59.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41fcdda700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f819b2c0 0x7f41f8184da0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f41e8006f40 tx=0x7f41e800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:59.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41edffb700 1 -- 172.21.15.32:0/2382428188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41e800bd80 con 0x7f41f819b2c0 2021-04-09T18:17:59.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41edffb700 1 -- 172.21.15.32:0/2382428188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e8000c90 con 0x7f41f819b2c0 2021-04-09T18:17:59.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f8186800 con 0x7f41f819b2c0 2021-04-09T18:17:59.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41edffb700 1 -- 172.21.15.32:0/2382428188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41e8002220 con 0x7f41f819b2c0 2021-04-09T18:17:59.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.166 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f8186800 con 0x7f41f819b2c0 2021-04-09T18:17:59.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.167 7f41edffb700 1 -- 172.21.15.32:0/2382428188 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f41e800a000 con 0x7f41f819b2c0 2021-04-09T18:17:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.167 7f41edffb700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41d8018930 0x7f41d801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.167 7f41edffb700 1 -- 172.21.15.32:0/2382428188 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f41e80329e0 con 0x7f41f819b2c0 2021-04-09T18:17:59.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.167 7f41fd5db700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41d8018930 0x7f41d801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.167 7f41fd5db700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41d8018930 0x7f41d801ae10 crc :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:59.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.187 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f81857c0 con 0x7f41f819b2c0 2021-04-09T18:17:59.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.190 7f41edffb700 1 -- 172.21.15.32:0/2382428188 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f41e8000c90 con 0x7f41f819b2c0 2021-04-09T18:17:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.382 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41f81857c0 con 0x7f41f819b2c0 2021-04-09T18:17:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.382 7f41edffb700 1 -- 172.21.15.32:0/2382428188 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f41e8006b00 con 0x7f41f819b2c0 2021-04-09T18:17:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 79 2021-04-09T18:17:59.422 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:59.423 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":79,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":75},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:17:58.930994","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:17:59.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41d8018930 msgr2=0x7f41d801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41d8018930 0x7f41d801ae10 crc :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f819b2c0 msgr2=0x7f41f8184da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f819b2c0 0x7f41f8184da0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f41e8006f40 tx=0x7f41e800b990).stop 2021-04-09T18:17:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 shutdown_connections 2021-04-09T18:17:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f41d8018930 0x7f41d801ae10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41f819b2c0 0x7f41f8184da0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41f81853c0 0x7f41f8185fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 --2- 172.21.15.32:0/2382428188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f81acea0 0x7f41f81ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 >> 172.21.15.32:0/2382428188 conn(0x7f41f819b890 msgr2=0x7f41f81a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 shutdown_connections 2021-04-09T18:17:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.385 7f41ff7bf700 1 -- 172.21.15.32:0/2382428188 wait complete. 2021-04-09T18:17:59.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:17:59.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.546 7fe095793700 1 Processor -- start 2021-04-09T18:17:59.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.546 7fe095793700 1 -- start start 2021-04-09T18:17:59.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe095793700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe0900d7ca0 0x7fe0900d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe095793700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe09010da00 0x7fe09010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe095793700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09010d480 con 0x7fe09010d010 2021-04-09T18:17:59.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe095793700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0900d9220 con 0x7fe09010da00 2021-04-09T18:17:59.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe095793700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0900d86c0 con 0x7fe0900d7ca0 2021-04-09T18:17:59.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08fd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe09010da00 0x7fe09010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08fd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe09010da00 0x7fe09010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45984/0 (socket says 172.21.15.32:45984) 2021-04-09T18:17:59.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08fd75700 1 -- 172.21.15.32:0/378120978 learned_addr learned my addr 172.21.15.32:0/378120978 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:59.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08ed73700 1 --2- 172.21.15.32:0/378120978 >> v2:172.21.15.86:3300/0 conn(0x7fe0900d7ca0 0x7fe0900d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08fd75700 1 -- 172.21.15.32:0/378120978 >> v2:172.21.15.86:3300/0 conn(0x7fe0900d7ca0 msgr2=0x7fe0900d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:17:59.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08fd75700 1 --2- 172.21.15.32:0/378120978 >> v2:172.21.15.86:3300/0 conn(0x7fe0900d7ca0 0x7fe0900d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08fd75700 1 -- 172.21.15.32:0/378120978 >> v1:172.21.15.86:6789/0 conn(0x7fe09010d010 legacy=0x7fe0900d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08fd75700 1 -- 172.21.15.32:0/378120978 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0900d1f70 con 0x7fe09010da00 2021-04-09T18:17:59.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08fd75700 1 --2- 172.21.15.32:0/378120978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe09010da00 0x7fe09010de00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7fe088006f00 tx=0x7fe088009f10).ready entity=mon.0 client_cookie=7957c8287ece59eb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:59.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08e572700 1 -- 172.21.15.32:0/378120978 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe08800b3b0 con 0x7fe09010da00 2021-04-09T18:17:59.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08e572700 1 -- 172.21.15.32:0/378120978 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08800b730 con 0x7fe09010da00 2021-04-09T18:17:59.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe08e572700 1 -- 172.21.15.32:0/378120978 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe08800bc10 con 0x7fe09010da00 2021-04-09T18:17:59.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe095793700 1 -- 172.21.15.32:0/378120978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe09010da00 msgr2=0x7fe09010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.547 7fe095793700 1 --2- 172.21.15.32:0/378120978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe09010da00 0x7fe09010de00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7fe088006f00 tx=0x7fe088009f10).stop 2021-04-09T18:17:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.548 7fe095793700 1 -- 172.21.15.32:0/378120978 shutdown_connections 2021-04-09T18:17:59.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.548 7fe095793700 1 --2- 172.21.15.32:0/378120978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe09010da00 0x7fe09010de00 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.548 7fe095793700 1 --2- 172.21.15.32:0/378120978 >> v2:172.21.15.86:3300/0 conn(0x7fe0900d7ca0 0x7fe0900d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.548 7fe095793700 1 -- 172.21.15.32:0/378120978 >> 172.21.15.32:0/378120978 conn(0x7fe0900ce5f0 msgr2=0x7fe0900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:59.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.548 7fe095793700 1 -- 172.21.15.32:0/378120978 shutdown_connections 2021-04-09T18:17:59.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.548 7fe095793700 1 -- 172.21.15.32:0/378120978 wait complete. 2021-04-09T18:17:59.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.568 7fe095793700 1 Processor -- start 2021-04-09T18:17:59.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.568 7fe095793700 1 -- start start 2021-04-09T18:17:59.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe095793700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0901aca20 0x7fe0901ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe095793700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe09018bd40 0x7fe09018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe095793700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe09008fba0 0x7fe09008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe095793700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09018c7e0 con 0x7fe0901aca20 2021-04-09T18:17:59.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe095793700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09018ca30 con 0x7fe09008fba0 2021-04-09T18:17:59.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe095793700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900905f0 con 0x7fe09018bd40 2021-04-09T18:17:59.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08f574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0901aca20 0x7fe0901ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08f574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0901aca20 0x7fe0901ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45990/0 (socket says 172.21.15.32:45990) 2021-04-09T18:17:59.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08f574700 1 -- 172.21.15.32:0/4030309572 learned_addr learned my addr 172.21.15.32:0/4030309572 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:17:59.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08ed73700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe09018bd40 0x7fe09018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08fd75700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe09008fba0 0x7fe09008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08f574700 1 -- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe09018bd40 msgr2=0x7fe09018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08f574700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe09018bd40 0x7fe09018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08f574700 1 -- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe09008fba0 msgr2=0x7fe09008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08f574700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe09008fba0 0x7fe09008ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08f574700 1 -- 172.21.15.32:0/4030309572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0900b4300 con 0x7fe0901aca20 2021-04-09T18:17:59.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.569 7fe08ed73700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe09018bd40 0x7fe09018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:17:59.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.570 7fe08f574700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0901aca20 0x7fe0901ace20 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7fe08000b5b0 tx=0x7fe08000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:59.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.570 7fe087fff700 1 -- 172.21.15.32:0/4030309572 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe08000bd40 con 0x7fe0901aca20 2021-04-09T18:17:59.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.570 7fe087fff700 1 -- 172.21.15.32:0/4030309572 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe080006240 con 0x7fe0901aca20 2021-04-09T18:17:59.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.570 7fe095793700 1 -- 172.21.15.32:0/4030309572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0900b4300 con 0x7fe0901aca20 2021-04-09T18:17:59.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.570 7fe087fff700 1 -- 172.21.15.32:0/4030309572 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe080005690 con 0x7fe0901aca20 2021-04-09T18:17:59.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.570 7fe095793700 1 -- 172.21.15.32:0/4030309572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0900b4300 con 0x7fe0901aca20 2021-04-09T18:17:59.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.570 7fe087fff700 1 -- 172.21.15.32:0/4030309572 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe0800054a0 con 0x7fe0901aca20 2021-04-09T18:17:59.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.571 7fe087fff700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe074018930 0x7fe07401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:17:59.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.571 7fe087fff700 1 -- 172.21.15.32:0/4030309572 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe080032a30 con 0x7fe0901aca20 2021-04-09T18:17:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.571 7fe08ed73700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe074018930 0x7fe07401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:17:59.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.571 7fe08ed73700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe074018930 0x7fe07401adc0 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:17:59.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.591 7fe095793700 1 -- 172.21.15.32:0/4030309572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe090182d70 con 0x7fe0901aca20 2021-04-09T18:17:59.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.594 7fe087fff700 1 -- 172.21.15.32:0/4030309572 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe080000c60 con 0x7fe0901aca20 2021-04-09T18:17:59.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.782 7fe095793700 1 -- 172.21.15.32:0/4030309572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe09018c760 con 0x7fe0901aca20 2021-04-09T18:17:59.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.783 7fe087fff700 1 -- 172.21.15.32:0/4030309572 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fe0800054a0 con 0x7fe0901aca20 2021-04-09T18:17:59.819 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:17:59.819 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":74,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:17:57.929427","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:17:55.170353","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-04-09 18:17:55.834924","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:17:59.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 -- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe074018930 msgr2=0x7fe07401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe074018930 0x7fe07401adc0 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 -- 172.21.15.32:0/4030309572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0901aca20 msgr2=0x7fe0901ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:17:59.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0901aca20 0x7fe0901ace20 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7fe08000b5b0 tx=0x7fe08000b950).stop 2021-04-09T18:17:59.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 -- 172.21.15.32:0/4030309572 shutdown_connections 2021-04-09T18:17:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe074018930 0x7fe07401adc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe09008fba0 0x7fe09008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe09018bd40 0x7fe09018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 --2- 172.21.15.32:0/4030309572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0901aca20 0x7fe0901ace20 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:17:59.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 -- 172.21.15.32:0/4030309572 >> 172.21.15.32:0/4030309572 conn(0x7fe09019b460 msgr2=0x7fe09019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:17:59.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 -- 172.21.15.32:0/4030309572 shutdown_connections 2021-04-09T18:17:59.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.785 7fe095793700 1 -- 172.21.15.32:0/4030309572 wait complete. 2021-04-09T18:17:59.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:18:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.990 7fe051505700 1 Processor -- start 2021-04-09T18:18:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.990 7fe051505700 1 -- start start 2021-04-09T18:18:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe051505700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe04c0cffb0 0x7fe04c087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe051505700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0d7c50 0x7fe04c1b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe051505700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c088460 con 0x7fe04c0d8050 2021-04-09T18:18:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe051505700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe04c0d91f0 con 0x7fe04c0cffb0 2021-04-09T18:18:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe051505700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0d8620 con 0x7fe04c0d7c50 2021-04-09T18:18:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04b2bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe04c0cffb0 0x7fe04c087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04bac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe04c0d8050 0x7fe04c10d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55588/0 (socket says 172.21.15.32:55588) 2021-04-09T18:18:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04b2bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe04c0cffb0 0x7fe04c087e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45996/0 (socket says 172.21.15.32:45996) 2021-04-09T18:18:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04bac0700 1 -- 172.21.15.32:0/2529936617 learned_addr learned my addr 172.21.15.32:0/2529936617 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04aabe700 1 --2- 172.21.15.32:0/2529936617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0d7c50 0x7fe04c1b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04b2bf700 1 -- 172.21.15.32:0/2529936617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0d7c50 msgr2=0x7fe04c1b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04b2bf700 1 --2- 172.21.15.32:0/2529936617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0d7c50 0x7fe04c1b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04b2bf700 1 -- 172.21.15.32:0/2529936617 >> v1:172.21.15.32:6789/0 conn(0x7fe04c0d8050 legacy=0x7fe04c10d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04b2bf700 1 -- 172.21.15.32:0/2529936617 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c0d2800 con 0x7fe04c0cffb0 2021-04-09T18:18:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04a2bd700 1 -- 172.21.15.32:0/2529936617 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874423145 0 0) 0x7fe044000d30 con 0x7fe04c0d8050 2021-04-09T18:18:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04b2bf700 1 --2- 172.21.15.32:0/2529936617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c0cffb0 0x7fe04c087e40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7fe03c006650 tx=0x7fe03c009a80).ready entity=mon.0 client_cookie=2b6a5f9dfbae2d81 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04a2bd700 1 -- 172.21.15.32:0/2529936617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe03c0009b0 con 0x7fe04c0cffb0 2021-04-09T18:18:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04a2bd700 1 -- 172.21.15.32:0/2529936617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c00c0c0 con 0x7fe04c0cffb0 2021-04-09T18:18:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.991 7fe04a2bd700 1 -- 172.21.15.32:0/2529936617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe03c00c510 con 0x7fe04c0cffb0 2021-04-09T18:18:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.992 7fe051505700 1 -- 172.21.15.32:0/2529936617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c0cffb0 msgr2=0x7fe04c087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.992 7fe051505700 1 --2- 172.21.15.32:0/2529936617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c0cffb0 0x7fe04c087e40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7fe03c006650 tx=0x7fe03c009a80).stop 2021-04-09T18:18:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.992 7fe051505700 1 -- 172.21.15.32:0/2529936617 shutdown_connections 2021-04-09T18:18:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.992 7fe051505700 1 --2- 172.21.15.32:0/2529936617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0d7c50 0x7fe04c1b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.992 7fe051505700 1 --2- 172.21.15.32:0/2529936617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c0cffb0 0x7fe04c087e40 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.992 7fe051505700 1 -- 172.21.15.32:0/2529936617 >> 172.21.15.32:0/2529936617 conn(0x7fe04c0ce5b0 msgr2=0x7fe04c0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.992 7fe051505700 1 -- 172.21.15.32:0/2529936617 shutdown_connections 2021-04-09T18:18:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:17:59.992 7fe051505700 1 -- 172.21.15.32:0/2529936617 wait complete. 2021-04-09T18:18:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.012 7fe051505700 1 Processor -- start 2021-04-09T18:18:00.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.012 7fe051505700 1 -- start start 2021-04-09T18:18:00.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe051505700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c1920c0 0x7fe04c0b8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe051505700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0c0b80 0x7fe04c0c0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe051505700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe04c0c15a0 0x7fe04c1a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe051505700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0bf0e0 con 0x7fe04c1920c0 2021-04-09T18:18:00.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe051505700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c1924c0 con 0x7fe04c0c15a0 2021-04-09T18:18:00.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe051505700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0a64a0 con 0x7fe04c0c0b80 2021-04-09T18:18:00.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c1920c0 0x7fe04c0b8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c1920c0 0x7fe04c0b8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46002/0 (socket says 172.21.15.32:46002) 2021-04-09T18:18:00.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04b2bf700 1 -- 172.21.15.32:0/1237927260 learned_addr learned my addr 172.21.15.32:0/1237927260 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:00.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04bac0700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe04c0c15a0 0x7fe04c1a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04aabe700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0c0b80 0x7fe04c0c0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04b2bf700 1 -- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0c0b80 msgr2=0x7fe04c0c0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04b2bf700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0c0b80 0x7fe04c0c0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04b2bf700 1 -- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe04c0c15a0 msgr2=0x7fe04c1a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04b2bf700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe04c0c15a0 0x7fe04c1a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04b2bf700 1 -- 172.21.15.32:0/1237927260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c0b8e40 con 0x7fe04c1920c0 2021-04-09T18:18:00.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.013 7fe04b2bf700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c1920c0 0x7fe04c0b8820 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7fe03c009650 tx=0x7fe03c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:00.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.014 7fe033fff700 1 -- 172.21.15.32:0/1237927260 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe03c0093a0 con 0x7fe04c1920c0 2021-04-09T18:18:00.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.014 7fe033fff700 1 -- 172.21.15.32:0/1237927260 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c00bd50 con 0x7fe04c1920c0 2021-04-09T18:18:00.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.014 7fe051505700 1 -- 172.21.15.32:0/1237927260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04c0b8e40 con 0x7fe04c1920c0 2021-04-09T18:18:00.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.014 7fe033fff700 1 -- 172.21.15.32:0/1237927260 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe03c0052e0 con 0x7fe04c1920c0 2021-04-09T18:18:00.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.014 7fe051505700 1 -- 172.21.15.32:0/1237927260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04c0b8e40 con 0x7fe04c1920c0 2021-04-09T18:18:00.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.014 7fe033fff700 1 -- 172.21.15.32:0/1237927260 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe03c006030 con 0x7fe04c1920c0 2021-04-09T18:18:00.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.014 7fe033fff700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe02c018930 0x7fe02c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.014 7fe033fff700 1 -- 172.21.15.32:0/1237927260 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe03c0329e0 con 0x7fe04c1920c0 2021-04-09T18:18:00.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.015 7fe04aabe700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe02c018930 0x7fe02c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.015 7fe04aabe700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe02c018930 0x7fe02c01ae10 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:00.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.035 7fe051505700 1 -- 172.21.15.32:0/1237927260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04c093fc0 con 0x7fe04c1920c0 2021-04-09T18:18:00.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.038 7fe033fff700 1 -- 172.21.15.32:0/1237927260 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe03c0009c0 con 0x7fe04c1920c0 2021-04-09T18:18:00.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.223 7fe051505700 1 -- 172.21.15.32:0/1237927260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe04c0d8620 con 0x7fe04c1920c0 2021-04-09T18:18:00.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.226 7fe033fff700 1 -- 172.21.15.32:0/1237927260 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v22) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe03c006030 con 0x7fe04c1920c0 2021-04-09T18:18:00.301 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:18:00.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 -- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe02c018930 msgr2=0x7fe02c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe02c018930 0x7fe02c01ae10 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 -- 172.21.15.32:0/1237927260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c1920c0 msgr2=0x7fe04c0b8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c1920c0 0x7fe04c0b8820 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7fe03c009650 tx=0x7fe03c00b960).stop 2021-04-09T18:18:00.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 -- 172.21.15.32:0/1237927260 shutdown_connections 2021-04-09T18:18:00.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe02c018930 0x7fe02c01ae10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe04c0c15a0 0x7fe04c1a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe04c0c0b80 0x7fe04c0c0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 --2- 172.21.15.32:0/1237927260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe04c1920c0 0x7fe04c0b8820 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.232 7fe051505700 1 -- 172.21.15.32:0/1237927260 >> 172.21.15.32:0/1237927260 conn(0x7fe04c0adaf0 msgr2=0x7fe04c0bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.233 7fe051505700 1 -- 172.21.15.32:0/1237927260 shutdown_connections 2021-04-09T18:18:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.233 7fe051505700 1 -- 172.21.15.32:0/1237927260 wait complete. 2021-04-09T18:18:00.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:18:00.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.469 7fdae97bc700 1 Processor -- start 2021-04-09T18:18:00.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.469 7fdae97bc700 1 -- start start 2021-04-09T18:18:00.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae97bc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdae40cffb0 0x7fdae4087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae97bc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae40d8050 0x7fdae410d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae97bc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae4088460 con 0x7fdae40d7c50 2021-04-09T18:18:00.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae97bc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae41b0c80 con 0x7fdae40cffb0 2021-04-09T18:18:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae97bc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae40d91f0 con 0x7fdae40d8050 2021-04-09T18:18:00.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae3574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdae40cffb0 0x7fdae4087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae2d73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdae40d7c50 0x7fdae41b0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55604/0 (socket says 172.21.15.32:55604) 2021-04-09T18:18:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae2d73700 1 -- 172.21.15.32:0/874271289 learned_addr learned my addr 172.21.15.32:0/874271289 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:00.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae3d75700 1 --2- 172.21.15.32:0/874271289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae40d8050 0x7fdae410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae3574700 1 -- 172.21.15.32:0/874271289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae40d8050 msgr2=0x7fdae410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae3574700 1 --2- 172.21.15.32:0/874271289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae40d8050 0x7fdae410d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae3574700 1 -- 172.21.15.32:0/874271289 >> v1:172.21.15.32:6789/0 conn(0x7fdae40d7c50 legacy=0x7fdae41b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae2572700 1 -- 172.21.15.32:0/874271289 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901442479 0 0) 0x7fdad80009b0 con 0x7fdae40d7c50 2021-04-09T18:18:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.470 7fdae3574700 1 -- 172.21.15.32:0/874271289 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae40d8620 con 0x7fdae40cffb0 2021-04-09T18:18:00.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.471 7fdae3574700 1 --2- 172.21.15.32:0/874271289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae40cffb0 0x7fdae4087e40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7fdad40069d0 tx=0x7fdad4009e40).ready entity=mon.0 client_cookie=270552de00f2397f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:00.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.471 7fdae2572700 1 -- 172.21.15.32:0/874271289 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdad400b510 con 0x7fdae40cffb0 2021-04-09T18:18:00.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.471 7fdae2572700 1 -- 172.21.15.32:0/874271289 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad400b890 con 0x7fdae40cffb0 2021-04-09T18:18:00.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.471 7fdae2572700 1 -- 172.21.15.32:0/874271289 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdad400bd70 con 0x7fdae40cffb0 2021-04-09T18:18:00.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.471 7fdae97bc700 1 -- 172.21.15.32:0/874271289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae40cffb0 msgr2=0x7fdae4087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.471 7fdae97bc700 1 --2- 172.21.15.32:0/874271289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae40cffb0 0x7fdae4087e40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7fdad40069d0 tx=0x7fdad4009e40).stop 2021-04-09T18:18:00.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.472 7fdae97bc700 1 -- 172.21.15.32:0/874271289 shutdown_connections 2021-04-09T18:18:00.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.472 7fdae97bc700 1 --2- 172.21.15.32:0/874271289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae40d8050 0x7fdae410d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.472 7fdae97bc700 1 --2- 172.21.15.32:0/874271289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae40cffb0 0x7fdae4087e40 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.472 7fdae97bc700 1 -- 172.21.15.32:0/874271289 >> 172.21.15.32:0/874271289 conn(0x7fdae40ce5b0 msgr2=0x7fdae40cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:00.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.472 7fdae97bc700 1 -- 172.21.15.32:0/874271289 shutdown_connections 2021-04-09T18:18:00.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.472 7fdae97bc700 1 -- 172.21.15.32:0/874271289 wait complete. 2021-04-09T18:18:00.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.492 7fdae97bc700 1 Processor -- start 2021-04-09T18:18:00.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.492 7fdae97bc700 1 -- start start 2021-04-09T18:18:00.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.492 7fdae97bc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdae40b1950 0x7fdae4183e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.492 7fdae97bc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae41795e0 0x7fdae4094eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.492 7fdae97bc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae41799e0 0x7fdae40954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.492 7fdae97bc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae40b13d0 con 0x7fdae41795e0 2021-04-09T18:18:00.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.492 7fdae97bc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae40b1d50 con 0x7fdae40b1950 2021-04-09T18:18:00.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.492 7fdae97bc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae419e4a0 con 0x7fdae41799e0 2021-04-09T18:18:00.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae2d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae41795e0 0x7fdae4094eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae2d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae41795e0 0x7fdae4094eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46016/0 (socket says 172.21.15.32:46016) 2021-04-09T18:18:00.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae2d73700 1 -- 172.21.15.32:0/3075706610 learned_addr learned my addr 172.21.15.32:0/3075706610 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:00.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae3574700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdae40b1950 0x7fdae4183e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae3d75700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae41799e0 0x7fdae40954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae2d73700 1 -- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae41799e0 msgr2=0x7fdae40954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae2d73700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae41799e0 0x7fdae40954d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae2d73700 1 -- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdae40b1950 msgr2=0x7fdae4183e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae2d73700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdae40b1950 0x7fdae4183e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae2d73700 1 -- 172.21.15.32:0/3075706610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae40b0180 con 0x7fdae41795e0 2021-04-09T18:18:00.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae2d73700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae41795e0 0x7fdae4094eb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7fdad800a4c0 tx=0x7fdad800a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:00.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdacbfff700 1 -- 172.21.15.32:0/3075706610 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdad800ac50 con 0x7fdae41795e0 2021-04-09T18:18:00.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdacbfff700 1 -- 172.21.15.32:0/3075706610 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad8004140 con 0x7fdae41795e0 2021-04-09T18:18:00.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae40b0180 con 0x7fdae41795e0 2021-04-09T18:18:00.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdacbfff700 1 -- 172.21.15.32:0/3075706610 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdad8004620 con 0x7fdae41795e0 2021-04-09T18:18:00.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.493 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae40b0180 con 0x7fdae41795e0 2021-04-09T18:18:00.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.494 7fdacbfff700 1 -- 172.21.15.32:0/3075706610 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdad8004320 con 0x7fdae41795e0 2021-04-09T18:18:00.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.494 7fdacbfff700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdac40188d0 0x7fdac401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.494 7fdacbfff700 1 -- 172.21.15.32:0/3075706610 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdad80329a0 con 0x7fdae41795e0 2021-04-09T18:18:00.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.494 7fdae3574700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdac40188d0 0x7fdac401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.495 7fdae3574700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdac40188d0 0x7fdac401add0 crc :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:00.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.514 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae4093fc0 con 0x7fdae41795e0 2021-04-09T18:18:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.518 7fdacbfff700 1 -- 172.21.15.32:0/3075706610 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdad80009b0 con 0x7fdae41795e0 2021-04-09T18:18:00.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.703 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fdae40d8620 con 0x7fdae41795e0 2021-04-09T18:18:00.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.706 7fdacbfff700 1 -- 172.21.15.32:0/3075706610 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v23) v1 ==== 184+0+0 (secure 0 0 0) 0x7fdad8004340 con 0x7fdae41795e0 2021-04-09T18:18:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:18:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdac40188d0 msgr2=0x7fdac401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdac40188d0 0x7fdac401add0 crc :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae41795e0 msgr2=0x7fdae4094eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae41795e0 0x7fdae4094eb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7fdad800a4c0 tx=0x7fdad800a860).stop 2021-04-09T18:18:00.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 shutdown_connections 2021-04-09T18:18:00.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdac40188d0 0x7fdac401add0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdae41795e0 0x7fdae4094eb0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdae41799e0 0x7fdae40954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 --2- 172.21.15.32:0/3075706610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdae40b1950 0x7fdae4183e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 >> 172.21.15.32:0/3075706610 conn(0x7fdae41889b0 msgr2=0x7fdae4189210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 shutdown_connections 2021-04-09T18:18:00.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.712 7fdae97bc700 1 -- 172.21.15.32:0/3075706610 wait complete. 2021-04-09T18:18:00.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:00.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.909 7f5d95a35700 1 Processor -- start 2021-04-09T18:18:00.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.909 7f5d95a35700 1 -- start start 2021-04-09T18:18:00.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.909 7f5d95a35700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d9010d010 0x7f5d900d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.909 7f5d95a35700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5d900d8490 0x7f5d9010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:00.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.909 7f5d95a35700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d9010d480 con 0x7f5d900d1da0 2021-04-09T18:18:00.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.909 7f5d95a35700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d9010e780 con 0x7f5d900d8490 2021-04-09T18:18:00.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.909 7f5d95a35700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d900d8d30 con 0x7f5d9010d010 2021-04-09T18:18:00.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5d900d8490 0x7f5d9010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8effd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5d900d1da0 0x7f5d900d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55616/0 (socket says 172.21.15.32:55616) 2021-04-09T18:18:00.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8effd700 1 -- 172.21.15.32:0/4275573828 learned_addr learned my addr 172.21.15.32:0/4275573828 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:00.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8f7fe700 1 --2- 172.21.15.32:0/4275573828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d9010d010 0x7f5d900d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:00.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8ffff700 1 -- 172.21.15.32:0/4275573828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d9010d010 msgr2=0x7f5d900d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:00.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8ffff700 1 --2- 172.21.15.32:0/4275573828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d9010d010 0x7f5d900d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:00.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8ffff700 1 -- 172.21.15.32:0/4275573828 >> v1:172.21.15.32:6789/0 conn(0x7f5d900d1da0 legacy=0x7f5d900d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8ffff700 1 -- 172.21.15.32:0/4275573828 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d900d9220 con 0x7f5d900d8490 2021-04-09T18:18:00.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8e7fc700 1 -- 172.21.15.32:0/4275573828 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556462289 0 0) 0x7f5d78000ce0 con 0x7f5d900d1da0 2021-04-09T18:18:00.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8ffff700 1 --2- 172.21.15.32:0/4275573828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d900d8490 0x7f5d9010da00 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7f5d88006e30 tx=0x7f5d88009de0).ready entity=mon.0 client_cookie=41205772e01a27ff server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:00.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8e7fc700 1 -- 172.21.15.32:0/4275573828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d8800b300 con 0x7f5d900d8490 2021-04-09T18:18:00.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8e7fc700 1 -- 172.21.15.32:0/4275573828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8800b680 con 0x7f5d900d8490 2021-04-09T18:18:00.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d8e7fc700 1 -- 172.21.15.32:0/4275573828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d8800bb60 con 0x7f5d900d8490 2021-04-09T18:18:00.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d95a35700 1 -- 172.21.15.32:0/4275573828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d900d8490 msgr2=0x7f5d9010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:00.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.910 7f5d95a35700 1 --2- 172.21.15.32:0/4275573828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d900d8490 0x7f5d9010da00 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7f5d88006e30 tx=0x7f5d88009de0).stop 2021-04-09T18:18:01.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.911 7f5d95a35700 1 -- 172.21.15.32:0/4275573828 shutdown_connections 2021-04-09T18:18:01.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.911 7f5d95a35700 1 --2- 172.21.15.32:0/4275573828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d900d8490 0x7f5d9010da00 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:01.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.911 7f5d95a35700 1 --2- 172.21.15.32:0/4275573828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d9010d010 0x7f5d900d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:01.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.911 7f5d95a35700 1 -- 172.21.15.32:0/4275573828 >> 172.21.15.32:0/4275573828 conn(0x7f5d900ce5f0 msgr2=0x7f5d900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:01.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.911 7f5d95a35700 1 -- 172.21.15.32:0/4275573828 shutdown_connections 2021-04-09T18:18:01.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.911 7f5d95a35700 1 -- 172.21.15.32:0/4275573828 wait complete. 2021-04-09T18:18:01.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.931 7f5d95a35700 1 Processor -- start 2021-04-09T18:18:01.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.931 7f5d95a35700 1 -- start start 2021-04-09T18:18:01.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d95a35700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d900ac0f0 0x7f5d900ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:01.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d95a35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d901a6c20 0x7f5d901a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:01.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d95a35700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d901a56c0 0x7f5d901a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:01.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d95a35700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d901a76c0 con 0x7f5d901a6c20 2021-04-09T18:18:01.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d95a35700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9017c7e0 con 0x7f5d901a56c0 2021-04-09T18:18:01.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d95a35700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9017ca00 con 0x7f5d900ac0f0 2021-04-09T18:18:01.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d901a6c20 0x7f5d901a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:01.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d901a6c20 0x7f5d901a7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46030/0 (socket says 172.21.15.32:46030) 2021-04-09T18:18:01.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8effd700 1 -- 172.21.15.32:0/3833057362 learned_addr learned my addr 172.21.15.32:0/3833057362 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:01.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8ffff700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d901a56c0 0x7f5d901a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:01.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8f7fe700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d900ac0f0 0x7f5d900ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:01.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8effd700 1 -- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d900ac0f0 msgr2=0x7f5d900ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:01.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8effd700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d900ac0f0 0x7f5d900ac4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:01.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8effd700 1 -- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d901a56c0 msgr2=0x7f5d901a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:01.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8effd700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d901a56c0 0x7f5d901a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:01.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8effd700 1 -- 172.21.15.32:0/3833057362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d901a60e0 con 0x7f5d901a6c20 2021-04-09T18:18:01.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8ffff700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d901a56c0 0x7f5d901a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:01.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8f7fe700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d900ac0f0 0x7f5d900ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:01.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.932 7f5d8effd700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d901a6c20 0x7f5d901a7020 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f5d78006f00 tx=0x7f5d7800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:01.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.933 7f5d87fff700 1 -- 172.21.15.32:0/3833057362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d78006020 con 0x7f5d901a6c20 2021-04-09T18:18:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.933 7f5d87fff700 1 -- 172.21.15.32:0/3833057362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d780063a0 con 0x7f5d901a6c20 2021-04-09T18:18:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.933 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d901a60e0 con 0x7f5d901a6c20 2021-04-09T18:18:01.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.933 7f5d87fff700 1 -- 172.21.15.32:0/3833057362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d78004620 con 0x7f5d901a6c20 2021-04-09T18:18:01.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.933 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d901a60e0 con 0x7f5d901a6c20 2021-04-09T18:18:01.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.933 7f5d87fff700 1 -- 172.21.15.32:0/3833057362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5d7800a000 con 0x7f5d901a6c20 2021-04-09T18:18:01.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.933 7f5d87fff700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d74018930 0x7f5d7401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.933 7f5d87fff700 1 -- 172.21.15.32:0/3833057362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5d78032a10 con 0x7f5d901a6c20 2021-04-09T18:18:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.934 7f5d8f7fe700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d74018930 0x7f5d7401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.934 7f5d8f7fe700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d74018930 0x7f5d7401ae10 crc :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.953 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d901b7050 con 0x7f5d901a6c20 2021-04-09T18:18:01.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:00.957 7f5d87fff700 1 -- 172.21.15.32:0/3833057362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5d78006b90 con 0x7f5d901a6c20 2021-04-09T18:18:01.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.147 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d901b7050 con 0x7f5d901a6c20 2021-04-09T18:18:01.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.147 7f5d87fff700 1 -- 172.21.15.32:0/3833057362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f5d7800a000 con 0x7f5d901a6c20 2021-04-09T18:18:01.203 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 79 2021-04-09T18:18:01.204 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:01.204 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":79,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":75},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:17:58.930994","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:01.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d74018930 msgr2=0x7f5d7401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:01.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d74018930 0x7f5d7401ae10 crc :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:01.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d901a6c20 msgr2=0x7f5d901a7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d901a6c20 0x7f5d901a7020 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f5d78006f00 tx=0x7f5d7800b940).stop 2021-04-09T18:18:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 shutdown_connections 2021-04-09T18:18:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d74018930 0x7f5d7401ae10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d901a6c20 0x7f5d901a7020 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d901a56c0 0x7f5d901a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:01.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 --2- 172.21.15.32:0/3833057362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d900ac0f0 0x7f5d900ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 >> 172.21.15.32:0/3833057362 conn(0x7f5d90199680 msgr2=0x7f5d901a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 shutdown_connections 2021-04-09T18:18:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.150 7f5d95a35700 1 -- 172.21.15.32:0/3833057362 wait complete. 2021-04-09T18:18:01.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:17:56.928328', 'modified': '2021-04-09 18:17:58.930994', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6211}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6211': {'gid': 6211, 'name': 'b', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/1080414756', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 1080414756}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 1080414756}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:01.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:18:01.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:18:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:18:01.771 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:01.772 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:18:01.772 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 6211, 2021-04-09T18:18:01.773 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:18:01.773 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:18:01.774 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.8063066829999999, 2021-04-09T18:18:01.774 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 79, 2021-04-09T18:18:01.774 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 74, 2021-04-09T18:18:01.775 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 74, 2021-04-09T18:18:01.776 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10.386293223999999 2021-04-09T18:18:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:18:01.780 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 6211, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.8063066829999999, "mdsmap_epoch": 79, "osdmap_epoch": 74, "osdmap_epoch_barrier": 74, "uptime": 10.386293223999999 } 2021-04-09T18:18:01.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:18:01.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.904 7f6caa4f9700 1 Processor -- start 2021-04-09T18:18:01.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.904 7f6caa4f9700 1 -- start start 2021-04-09T18:18:01.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6caa4f9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6ca40d7ca0 0x7f6ca40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:01.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6caa4f9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca410da00 0x7f6ca410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:01.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6caa4f9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca410d480 con 0x7f6ca410d010 2021-04-09T18:18:01.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6caa4f9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca40d9220 con 0x7f6ca40d7ca0 2021-04-09T18:18:01.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6caa4f9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca4112100 con 0x7f6ca410da00 2021-04-09T18:18:01.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6ca3fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6ca410d010 0x7f6ca40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:53554/0 (socket says 172.21.15.32:53554) 2021-04-09T18:18:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6ca3fff700 1 -- 172.21.15.32:0/2371038702 learned_addr learned my addr 172.21.15.32:0/2371038702 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:01.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6ca8b16700 1 --2- 172.21.15.32:0/2371038702 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca410da00 0x7f6ca410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:01.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6c9bfff700 1 --2- 172.21.15.32:0/2371038702 >> v2:172.21.15.86:3300/0 conn(0x7f6ca40d7ca0 0x7f6ca40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6ca37fe700 1 -- 172.21.15.32:0/2371038702 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754568167 0 0) 0x7f6c940010c0 con 0x7f6ca410d010 2021-04-09T18:18:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6ca37fe700 1 -- 172.21.15.32:0/2371038702 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c90002fe0 con 0x7f6ca410d010 2021-04-09T18:18:01.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6c9bfff700 1 -- 172.21.15.32:0/2371038702 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca410da00 msgr2=0x7f6ca410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.905 7f6c9bfff700 1 --2- 172.21.15.32:0/2371038702 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca410da00 0x7f6ca410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.906 7f6c9bfff700 1 -- 172.21.15.32:0/2371038702 >> v1:172.21.15.86:6789/0 conn(0x7f6ca410d010 legacy=0x7f6ca40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.906 7f6c9bfff700 1 -- 172.21.15.32:0/2371038702 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca40d86c0 con 0x7f6ca40d7ca0 2021-04-09T18:18:02.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.906 7f6ca8b16700 1 --2- 172.21.15.32:0/2371038702 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca410da00 0x7f6ca410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:02.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.906 7f6c9bfff700 1 --2- 172.21.15.32:0/2371038702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca40d7ca0 0x7f6ca40d80a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f6c8c006650 tx=0x7f6c8c009a80).ready entity=mon.1 client_cookie=56d49119dfb39f2e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:02.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.906 7f6ca37fe700 1 -- 172.21.15.32:0/2371038702 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c8c0009b0 con 0x7f6ca40d7ca0 2021-04-09T18:18:02.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.906 7f6ca37fe700 1 -- 172.21.15.32:0/2371038702 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8c00c0c0 con 0x7f6ca40d7ca0 2021-04-09T18:18:02.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.906 7f6ca37fe700 1 -- 172.21.15.32:0/2371038702 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c8c00c510 con 0x7f6ca40d7ca0 2021-04-09T18:18:02.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.906 7f6caa4f9700 1 -- 172.21.15.32:0/2371038702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca40d7ca0 msgr2=0x7f6ca40d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.906 7f6caa4f9700 1 --2- 172.21.15.32:0/2371038702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca40d7ca0 0x7f6ca40d80a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f6c8c006650 tx=0x7f6c8c009a80).stop 2021-04-09T18:18:02.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.907 7f6caa4f9700 1 -- 172.21.15.32:0/2371038702 shutdown_connections 2021-04-09T18:18:02.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.907 7f6caa4f9700 1 --2- 172.21.15.32:0/2371038702 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca410da00 0x7f6ca410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.907 7f6caa4f9700 1 --2- 172.21.15.32:0/2371038702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca40d7ca0 0x7f6ca40d80a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.907 7f6caa4f9700 1 -- 172.21.15.32:0/2371038702 >> 172.21.15.32:0/2371038702 conn(0x7f6ca40ce5f0 msgr2=0x7f6ca40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:02.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.907 7f6caa4f9700 1 -- 172.21.15.32:0/2371038702 shutdown_connections 2021-04-09T18:18:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.907 7f6caa4f9700 1 -- 172.21.15.32:0/2371038702 wait complete. 2021-04-09T18:18:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.927 7f6caa4f9700 1 Processor -- start 2021-04-09T18:18:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.927 7f6caa4f9700 1 -- start start 2021-04-09T18:18:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.927 7f6caa4f9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca4179bf0 0x7f6ca4179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.927 7f6caa4f9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca417c540 0x7f6ca417c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.927 7f6caa4f9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca40aca60 0x7f6ca40ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.927 7f6caa4f9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca417cf60 con 0x7f6ca40aca60 2021-04-09T18:18:02.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.927 7f6caa4f9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca40ad480 con 0x7f6ca4179bf0 2021-04-09T18:18:02.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.927 7f6caa4f9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca40ad6a0 con 0x7f6ca417c540 2021-04-09T18:18:02.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca8b16700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca40aca60 0x7f6ca40ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca8b16700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca40aca60 0x7f6ca40ace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46048/0 (socket says 172.21.15.32:46048) 2021-04-09T18:18:02.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca3fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca4179bf0 0x7f6ca4179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca8b16700 1 -- 172.21.15.32:0/3925538791 learned_addr learned my addr 172.21.15.32:0/3925538791 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6c9bfff700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca417c540 0x7f6ca417c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca8b16700 1 -- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca417c540 msgr2=0x7f6ca417c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca8b16700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca417c540 0x7f6ca417c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca8b16700 1 -- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca4179bf0 msgr2=0x7f6ca4179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca8b16700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca4179bf0 0x7f6ca4179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca8b16700 1 -- 172.21.15.32:0/3925538791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca40ad8c0 con 0x7f6ca40aca60 2021-04-09T18:18:02.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca3fff700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca4179bf0 0x7f6ca4179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca8b16700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca40aca60 0x7f6ca40ace60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f6c9c00b970 tx=0x7f6c9c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:02.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca17fa700 1 -- 172.21.15.32:0/3925538791 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c9c000f90 con 0x7f6ca40aca60 2021-04-09T18:18:02.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca17fa700 1 -- 172.21.15.32:0/3925538791 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c006aa0 con 0x7f6ca40aca60 2021-04-09T18:18:02.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca40ad8c0 con 0x7f6ca40aca60 2021-04-09T18:18:02.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6ca17fa700 1 -- 172.21.15.32:0/3925538791 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c9c0015f0 con 0x7f6ca40aca60 2021-04-09T18:18:02.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.928 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca40ad8c0 con 0x7f6ca40aca60 2021-04-09T18:18:02.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.929 7f6ca17fa700 1 -- 172.21.15.32:0/3925538791 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6c9c000e70 con 0x7f6ca40aca60 2021-04-09T18:18:02.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.929 7f6ca17fa700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c880188d0 0x7f6c8801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.929 7f6ca17fa700 1 -- 172.21.15.32:0/3925538791 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6c9c032d70 con 0x7f6ca40aca60 2021-04-09T18:18:02.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.929 7f6ca3fff700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c880188d0 0x7f6c8801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.930 7f6ca3fff700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c880188d0 0x7f6c8801add0 crc :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:02.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.949 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca417cf60 con 0x7f6ca40aca60 2021-04-09T18:18:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:01.952 7f6ca17fa700 1 -- 172.21.15.32:0/3925538791 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6c9c0066f0 con 0x7f6ca40aca60 2021-04-09T18:18:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.139 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ca417cf60 con 0x7f6ca40aca60 2021-04-09T18:18:02.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.140 7f6ca17fa700 1 -- 172.21.15.32:0/3925538791 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f6c9c006aa0 con 0x7f6ca40aca60 2021-04-09T18:18:02.210 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:02.211 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":74,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:17:57.929427","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:17:55.170353","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-04-09 18:17:55.834924","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:18:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c880188d0 msgr2=0x7f6c8801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c880188d0 0x7f6c8801add0 crc :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca40aca60 msgr2=0x7f6ca40ace60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca40aca60 0x7f6ca40ace60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f6c9c00b970 tx=0x7f6c9c00bd10).stop 2021-04-09T18:18:02.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 shutdown_connections 2021-04-09T18:18:02.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6c880188d0 0x7f6c8801add0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ca40aca60 0x7f6ca40ace60 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ca417c540 0x7f6ca417c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 --2- 172.21.15.32:0/3925538791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ca4179bf0 0x7f6ca4179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 >> 172.21.15.32:0/3925538791 conn(0x7f6ca41acae0 msgr2=0x7f6ca417ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:02.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 shutdown_connections 2021-04-09T18:18:02.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.142 7f6caa4f9700 1 -- 172.21.15.32:0/3925538791 wait complete. 2021-04-09T18:18:02.325 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:17:55.170353', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:17:55.834924', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:18:02.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:18:02.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.471 7f68b4746700 1 Processor -- start 2021-04-09T18:18:02.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.471 7f68b4746700 1 -- start start 2021-04-09T18:18:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b4746700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac10d010 0x7f68ac0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b4746700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f68ac0d1da0 0x7f68ac0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b4746700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f68ac0d82c0 0x7f68ac10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b4746700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ac10e780 con 0x7f68ac0d1da0 2021-04-09T18:18:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b4746700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ac0d8d30 con 0x7f68ac0d82c0 2021-04-09T18:18:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b4746700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0d9220 con 0x7f68ac10d010 2021-04-09T18:18:02.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1d61700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f68ac0d1da0 0x7f68ac0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1d61700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f68ac0d1da0 0x7f68ac0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46056/0 (socket says 172.21.15.32:46056) 2021-04-09T18:18:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1d61700 1 -- 172.21.15.32:0/160889394 learned_addr learned my addr 172.21.15.32:0/160889394 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:02.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b2d63700 1 --2- 172.21.15.32:0/160889394 >> v2:172.21.15.86:3300/0 conn(0x7f68ac0d82c0 0x7f68ac10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b2562700 1 --2- 172.21.15.32:0/160889394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac10d010 0x7f68ac0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1d61700 1 -- 172.21.15.32:0/160889394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac10d010 msgr2=0x7f68ac0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1d61700 1 --2- 172.21.15.32:0/160889394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac10d010 0x7f68ac0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1d61700 1 -- 172.21.15.32:0/160889394 >> v2:172.21.15.86:3300/0 conn(0x7f68ac0d82c0 msgr2=0x7f68ac10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1d61700 1 --2- 172.21.15.32:0/160889394 >> v2:172.21.15.86:3300/0 conn(0x7f68ac0d82c0 0x7f68ac10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1d61700 1 -- 172.21.15.32:0/160889394 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac10d480 con 0x7f68ac0d1da0 2021-04-09T18:18:02.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1d61700 1 --2- 172.21.15.32:0/160889394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0d1da0 0x7f68ac0d7ca0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f689c006700 tx=0x7f689c009b00).ready entity=mon.0 client_cookie=d3926e7716de3ce8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:02.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.472 7f68b1560700 1 -- 172.21.15.32:0/160889394 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f689c00c0c0 con 0x7f68ac0d1da0 2021-04-09T18:18:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b1560700 1 -- 172.21.15.32:0/160889394 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c00c400 con 0x7f68ac0d1da0 2021-04-09T18:18:02.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b1560700 1 -- 172.21.15.32:0/160889394 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f689c00c8e0 con 0x7f68ac0d1da0 2021-04-09T18:18:02.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b4746700 1 -- 172.21.15.32:0/160889394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0d1da0 msgr2=0x7f68ac0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b4746700 1 --2- 172.21.15.32:0/160889394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0d1da0 0x7f68ac0d7ca0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f689c006700 tx=0x7f689c009b00).stop 2021-04-09T18:18:02.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b4746700 1 -- 172.21.15.32:0/160889394 shutdown_connections 2021-04-09T18:18:02.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b4746700 1 --2- 172.21.15.32:0/160889394 >> v2:172.21.15.86:3300/0 conn(0x7f68ac0d82c0 0x7f68ac10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b4746700 1 --2- 172.21.15.32:0/160889394 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0d1da0 0x7f68ac0d7ca0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b4746700 1 --2- 172.21.15.32:0/160889394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac10d010 0x7f68ac0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b4746700 1 -- 172.21.15.32:0/160889394 >> 172.21.15.32:0/160889394 conn(0x7f68ac0ce5f0 msgr2=0x7f68ac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:02.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b4746700 1 -- 172.21.15.32:0/160889394 shutdown_connections 2021-04-09T18:18:02.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.473 7f68b4746700 1 -- 172.21.15.32:0/160889394 wait complete. 2021-04-09T18:18:02.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.493 7f68b4746700 1 Processor -- start 2021-04-09T18:18:02.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b4746700 1 -- start start 2021-04-09T18:18:02.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b4746700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac1a0050 0x7f68ac1a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b4746700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0bdcd0 0x7f68ac0bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b4746700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ac0bf080 0x7f68ac0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b4746700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0bf500 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b4746700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0bb7b0 con 0x7f68ac0bf080 2021-04-09T18:18:02.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b4746700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0be0d0 con 0x7f68ac1a0050 2021-04-09T18:18:02.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b1d61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0bdcd0 0x7f68ac0bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b1d61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0bdcd0 0x7f68ac0bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46060/0 (socket says 172.21.15.32:46060) 2021-04-09T18:18:02.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b1d61700 1 -- 172.21.15.32:0/1889620257 learned_addr learned my addr 172.21.15.32:0/1889620257 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:02.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b2562700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac1a0050 0x7f68ac1a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b2d63700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ac0bf080 0x7f68ac0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b1d61700 1 -- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac1a0050 msgr2=0x7f68ac1a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.494 7f68b1d61700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac1a0050 0x7f68ac1a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68b1d61700 1 -- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ac0bf080 msgr2=0x7f68ac0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68b1d61700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ac0bf080 0x7f68ac0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68b1d61700 1 -- 172.21.15.32:0/1889620257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac0b1310 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68b2d63700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ac0bf080 0x7f68ac0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:02.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68b1d61700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0bdcd0 0x7f68ac0bea60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f689c006f20 tx=0x7f689c00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:02.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68a2ffd700 1 -- 172.21.15.32:0/1889620257 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f689c00bdd0 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68a2ffd700 1 -- 172.21.15.32:0/1889620257 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c0090d0 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68b4746700 1 -- 172.21.15.32:0/1889620257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac0b1310 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68a2ffd700 1 -- 172.21.15.32:0/1889620257 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f689c0054d0 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68b4746700 1 -- 172.21.15.32:0/1889620257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac0b1310 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.495 7f68a2ffd700 1 -- 172.21.15.32:0/1889620257 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f689c006030 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.496 7f68a2ffd700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6894018930 0x7f689401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.496 7f68a2ffd700 1 -- 172.21.15.32:0/1889620257 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f689c032a20 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.496 7f68b2562700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6894018930 0x7f689401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.496 7f68b2562700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6894018930 0x7f689401ae10 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.516 7f68b4746700 1 -- 172.21.15.32:0/1889620257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac093d50 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.519 7f68a2ffd700 1 -- 172.21.15.32:0/1889620257 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f689c006030 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.698 7f68b4746700 1 -- 172.21.15.32:0/1889620257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f68ac10d480 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.699 7f68a2ffd700 1 -- 172.21.15.32:0/1889620257 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v74) v1 ==== 92+0+10 (secure 0 0 0) 0x7f689c00bdd0 con 0x7f68ac0bdcd0 2021-04-09T18:18:02.737 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:18:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 -- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6894018930 msgr2=0x7f689401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6894018930 0x7f689401ae10 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 -- 172.21.15.32:0/1889620257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0bdcd0 msgr2=0x7f68ac0bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0bdcd0 0x7f68ac0bea60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f689c006f20 tx=0x7f689c00b9e0).stop 2021-04-09T18:18:02.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 -- 172.21.15.32:0/1889620257 shutdown_connections 2021-04-09T18:18:02.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6894018930 0x7f689401ae10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ac0bdcd0 0x7f68ac0bea60 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ac0bf080 0x7f68ac0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 --2- 172.21.15.32:0/1889620257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ac1a0050 0x7f68ac1a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 -- 172.21.15.32:0/1889620257 >> 172.21.15.32:0/1889620257 conn(0x7f68ac0975f0 msgr2=0x7f68ac096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:02.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 -- 172.21.15.32:0/1889620257 shutdown_connections 2021-04-09T18:18:02.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.701 7f68b4746700 1 -- 172.21.15.32:0/1889620257 wait complete. 2021-04-09T18:18:02.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:18:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.901 7f7b31287700 1 Processor -- start 2021-04-09T18:18:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.901 7f7b31287700 1 -- start start 2021-04-09T18:18:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.901 7f7b31287700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b2c10d010 0x7f7b2c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.901 7f7b31287700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b2c0d8490 0x7f7b2c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:02.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.901 7f7b31287700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c10d480 con 0x7f7b2c0d1da0 2021-04-09T18:18:02.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.901 7f7b31287700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2c0d8d30 con 0x7f7b2c0d8490 2021-04-09T18:18:02.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.901 7f7b31287700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2c0d9220 con 0x7f7b2c10d010 2021-04-09T18:18:02.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.901 7f7b2b87e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b2c0d8490 0x7f7b2c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.901 7f7b2b87e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b2c0d8490 0x7f7b2c10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46068/0 (socket says 172.21.15.32:46068) 2021-04-09T18:18:02.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2b87e700 1 -- 172.21.15.32:0/1863642258 learned_addr learned my addr 172.21.15.32:0/1863642258 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:02.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2b07d700 1 --2- 172.21.15.32:0/1863642258 >> v2:172.21.15.86:3300/0 conn(0x7f7b2c10d010 0x7f7b2c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:02.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2b87e700 1 -- 172.21.15.32:0/1863642258 >> v2:172.21.15.86:3300/0 conn(0x7f7b2c10d010 msgr2=0x7f7b2c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2b87e700 1 --2- 172.21.15.32:0/1863642258 >> v2:172.21.15.86:3300/0 conn(0x7f7b2c10d010 0x7f7b2c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2b87e700 1 -- 172.21.15.32:0/1863642258 >> v1:172.21.15.86:6789/0 conn(0x7f7b2c0d1da0 legacy=0x7f7b2c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:02.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2b87e700 1 -- 172.21.15.32:0/1863642258 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c10e020 con 0x7f7b2c0d8490 2021-04-09T18:18:03.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2b87e700 1 --2- 172.21.15.32:0/1863642258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0d8490 0x7f7b2c10da00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7f7b24006f00 tx=0x7f7b24009f10).ready entity=mon.0 client_cookie=b1c0fa59a1e6ede4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:03.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2a07b700 1 -- 172.21.15.32:0/1863642258 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b2400b3b0 con 0x7f7b2c0d8490 2021-04-09T18:18:03.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2a07b700 1 -- 172.21.15.32:0/1863642258 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2400b730 con 0x7f7b2c0d8490 2021-04-09T18:18:03.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b2a07b700 1 -- 172.21.15.32:0/1863642258 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b2400bc10 con 0x7f7b2c0d8490 2021-04-09T18:18:03.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b31287700 1 -- 172.21.15.32:0/1863642258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0d8490 msgr2=0x7f7b2c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.902 7f7b31287700 1 --2- 172.21.15.32:0/1863642258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0d8490 0x7f7b2c10da00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7f7b24006f00 tx=0x7f7b24009f10).stop 2021-04-09T18:18:03.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.903 7f7b31287700 1 -- 172.21.15.32:0/1863642258 shutdown_connections 2021-04-09T18:18:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.903 7f7b31287700 1 --2- 172.21.15.32:0/1863642258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0d8490 0x7f7b2c10da00 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.903 7f7b31287700 1 --2- 172.21.15.32:0/1863642258 >> v2:172.21.15.86:3300/0 conn(0x7f7b2c10d010 0x7f7b2c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.903 7f7b31287700 1 -- 172.21.15.32:0/1863642258 >> 172.21.15.32:0/1863642258 conn(0x7f7b2c0ce5f0 msgr2=0x7f7b2c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:03.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.903 7f7b31287700 1 -- 172.21.15.32:0/1863642258 shutdown_connections 2021-04-09T18:18:03.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.903 7f7b31287700 1 -- 172.21.15.32:0/1863642258 wait complete. 2021-04-09T18:18:03.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.923 7f7b31287700 1 Processor -- start 2021-04-09T18:18:03.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.923 7f7b31287700 1 -- start start 2021-04-09T18:18:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b31287700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0a0a10 0x7f7b2c0bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b31287700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2c18ad00 0x7f7b2c19fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b31287700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b2c1a03f0 0x7f7b2c095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b31287700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c18b100 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b31287700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c095860 con 0x7f7b2c18ad00 2021-04-09T18:18:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b31287700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c095a80 con 0x7f7b2c1a03f0 2021-04-09T18:18:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0a0a10 0x7f7b2c0bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0a0a10 0x7f7b2c0bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46072/0 (socket says 172.21.15.32:46072) 2021-04-09T18:18:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b07d700 1 -- 172.21.15.32:0/3319050383 learned_addr learned my addr 172.21.15.32:0/3319050383 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:03.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b87e700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b2c1a03f0 0x7f7b2c095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2a87c700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2c18ad00 0x7f7b2c19fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b07d700 1 -- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b2c1a03f0 msgr2=0x7f7b2c095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b07d700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b2c1a03f0 0x7f7b2c095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b07d700 1 -- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2c18ad00 msgr2=0x7f7b2c19fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b07d700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2c18ad00 0x7f7b2c19fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b07d700 1 -- 172.21.15.32:0/3319050383 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c189250 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2a87c700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2c18ad00 0x7f7b2c19fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:03.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b2b07d700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0a0a10 0x7f7b2c0bf870 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f7b1c006f60 tx=0x7f7b1c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:03.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b23fff700 1 -- 172.21.15.32:0/3319050383 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b1c00bd60 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b23fff700 1 -- 172.21.15.32:0/3319050383 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c000c90 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b23fff700 1 -- 172.21.15.32:0/3319050383 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b1c002220 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.924 7f7b31287700 1 -- 172.21.15.32:0/3319050383 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c189250 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.925 7f7b31287700 1 -- 172.21.15.32:0/3319050383 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c189250 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.925 7f7b23fff700 1 -- 172.21.15.32:0/3319050383 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7b1c0052e0 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.925 7f7b23fff700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b100188c0 0x7f7b1001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.925 7f7b23fff700 1 -- 172.21.15.32:0/3319050383 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7b1c0327f0 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.925 7f7b2a87c700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b100188c0 0x7f7b1001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.926 7f7b2a87c700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b100188c0 0x7f7b1001adc0 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:03.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.945 7f7b31287700 1 -- 172.21.15.32:0/3319050383 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2c093d50 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:02.949 7f7b23fff700 1 -- 172.21.15.32:0/3319050383 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7b1c006b00 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.129 7f7b31287700 1 -- 172.21.15.32:0/3319050383 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f7b2c10d480 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.130 7f7b23fff700 1 -- 172.21.15.32:0/3319050383 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v74) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7b1c0052e0 con 0x7f7b2c0a0a10 2021-04-09T18:18:03.182 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:18:03.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 -- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b100188c0 msgr2=0x7f7b1001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b100188c0 0x7f7b1001adc0 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 -- 172.21.15.32:0/3319050383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0a0a10 msgr2=0x7f7b2c0bf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0a0a10 0x7f7b2c0bf870 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f7b1c006f60 tx=0x7f7b1c00b970).stop 2021-04-09T18:18:03.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 -- 172.21.15.32:0/3319050383 shutdown_connections 2021-04-09T18:18:03.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b100188c0 0x7f7b1001adc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b2c1a03f0 0x7f7b2c095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2c18ad00 0x7f7b2c19fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 --2- 172.21.15.32:0/3319050383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2c0a0a10 0x7f7b2c0bf870 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 -- 172.21.15.32:0/3319050383 >> 172.21.15.32:0/3319050383 conn(0x7f7b2c0975f0 msgr2=0x7f7b2c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 -- 172.21.15.32:0/3319050383 shutdown_connections 2021-04-09T18:18:03.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.132 7f7b31287700 1 -- 172.21.15.32:0/3319050383 wait complete. 2021-04-09T18:18:03.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:18:03.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.312 7f938e3e7700 1 Processor -- start 2021-04-09T18:18:03.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.312 7f938e3e7700 1 -- start start 2021-04-09T18:18:03.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.313 7f938e3e7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f93880cffc0 0x7f93880d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.313 7f938e3e7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93880d7820 0x7f93880d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.313 7f938e3e7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93880d72a0 con 0x7f93880d8240 2021-04-09T18:18:03.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.313 7f938e3e7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93880d9200 con 0x7f93880d7820 2021-04-09T18:18:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.313 7f938e3e7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f938810e000 con 0x7f93880cffc0 2021-04-09T18:18:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.313 7f937ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93880d7820 0x7f93880d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.313 7f937ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f93880d7820 0x7f93880d7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46084/0 (socket says 172.21.15.32:46084) 2021-04-09T18:18:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.313 7f937ffff700 1 -- 172.21.15.32:0/2023505002 learned_addr learned my addr 172.21.15.32:0/2023505002 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:03.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.313 7f9387fff700 1 --2- 172.21.15.32:0/2023505002 >> v2:172.21.15.86:3300/0 conn(0x7f93880cffc0 0x7f93880d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f937ffff700 1 -- 172.21.15.32:0/2023505002 >> v2:172.21.15.86:3300/0 conn(0x7f93880cffc0 msgr2=0x7f93880d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f937ffff700 1 --2- 172.21.15.32:0/2023505002 >> v2:172.21.15.86:3300/0 conn(0x7f93880cffc0 0x7f93880d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f937ffff700 1 -- 172.21.15.32:0/2023505002 >> v1:172.21.15.32:6789/0 conn(0x7f93880d8240 legacy=0x7f93880d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f93877fe700 1 -- 172.21.15.32:0/2023505002 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384151793 0 0) 0x7f9380000d30 con 0x7f93880d8240 2021-04-09T18:18:03.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f937ffff700 1 -- 172.21.15.32:0/2023505002 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938810e220 con 0x7f93880d7820 2021-04-09T18:18:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f937ffff700 1 --2- 172.21.15.32:0/2023505002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880d7820 0x7f93880d7c20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f9370006700 tx=0x7f9370009b00).ready entity=mon.0 client_cookie=4d6df55da6591b8c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:03.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f93877fe700 1 -- 172.21.15.32:0/2023505002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f937000c0c0 con 0x7f93880d7820 2021-04-09T18:18:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f93877fe700 1 -- 172.21.15.32:0/2023505002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937000c400 con 0x7f93880d7820 2021-04-09T18:18:03.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f93877fe700 1 -- 172.21.15.32:0/2023505002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f937000c8e0 con 0x7f93880d7820 2021-04-09T18:18:03.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f938e3e7700 1 -- 172.21.15.32:0/2023505002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880d7820 msgr2=0x7f93880d7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.314 7f938e3e7700 1 --2- 172.21.15.32:0/2023505002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880d7820 0x7f93880d7c20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f9370006700 tx=0x7f9370009b00).stop 2021-04-09T18:18:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.315 7f938e3e7700 1 -- 172.21.15.32:0/2023505002 shutdown_connections 2021-04-09T18:18:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.315 7f938e3e7700 1 --2- 172.21.15.32:0/2023505002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880d7820 0x7f93880d7c20 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.315 7f938e3e7700 1 --2- 172.21.15.32:0/2023505002 >> v2:172.21.15.86:3300/0 conn(0x7f93880cffc0 0x7f93880d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.315 7f938e3e7700 1 -- 172.21.15.32:0/2023505002 >> 172.21.15.32:0/2023505002 conn(0x7f93880ce5e0 msgr2=0x7f93880cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:03.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.315 7f938e3e7700 1 -- 172.21.15.32:0/2023505002 shutdown_connections 2021-04-09T18:18:03.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.315 7f938e3e7700 1 -- 172.21.15.32:0/2023505002 wait complete. 2021-04-09T18:18:03.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.335 7f938e3e7700 1 Processor -- start 2021-04-09T18:18:03.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.335 7f938e3e7700 1 -- start start 2021-04-09T18:18:03.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f938e3e7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93880b5800 0x7f93880b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f938e3e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880bfab0 0x7f93880c09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f938e3e7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93880bfeb0 0x7f93880c0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f938e3e7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880c04c0 con 0x7f93880bfab0 2021-04-09T18:18:03.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f938e3e7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93880b6260 con 0x7f93880b5800 2021-04-09T18:18:03.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f938e3e7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93880b64b0 con 0x7f93880bfeb0 2021-04-09T18:18:03.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f937ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880bfab0 0x7f93880c09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f937ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880bfab0 0x7f93880c09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46086/0 (socket says 172.21.15.32:46086) 2021-04-09T18:18:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f937ffff700 1 -- 172.21.15.32:0/1048531897 learned_addr learned my addr 172.21.15.32:0/1048531897 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:03.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f9387fff700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93880b5800 0x7f93880b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f938ca04700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93880bfeb0 0x7f93880c0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f937ffff700 1 -- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93880bfeb0 msgr2=0x7f93880c0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f937ffff700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93880bfeb0 0x7f93880c0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f937ffff700 1 -- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93880b5800 msgr2=0x7f93880b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f937ffff700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93880b5800 0x7f93880b5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f937ffff700 1 -- 172.21.15.32:0/1048531897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938817da00 con 0x7f93880bfab0 2021-04-09T18:18:03.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f937ffff700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880bfab0 0x7f93880c09c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f9370006f20 tx=0x7f937000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:03.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f93857fa700 1 -- 172.21.15.32:0/1048531897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f937000bdd0 con 0x7f93880bfab0 2021-04-09T18:18:03.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.336 7f93857fa700 1 -- 172.21.15.32:0/1048531897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9370006440 con 0x7f93880bfab0 2021-04-09T18:18:03.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.337 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938817da00 con 0x7f93880bfab0 2021-04-09T18:18:03.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.337 7f93857fa700 1 -- 172.21.15.32:0/1048531897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93700052e0 con 0x7f93880bfab0 2021-04-09T18:18:03.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.337 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938817da00 con 0x7f93880bfab0 2021-04-09T18:18:03.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.337 7f93857fa700 1 -- 172.21.15.32:0/1048531897 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f93700052e0 con 0x7f93880bfab0 2021-04-09T18:18:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.337 7f93857fa700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f936c0188d0 0x7f936c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.337 7f93857fa700 1 -- 172.21.15.32:0/1048531897 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9370032a00 con 0x7f93880bfab0 2021-04-09T18:18:03.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.338 7f9387fff700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f936c0188d0 0x7f936c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.338 7f9387fff700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f936c0188d0 0x7f936c01add0 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:03.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.357 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93880d72a0 con 0x7f93880bfab0 2021-04-09T18:18:03.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.360 7f93857fa700 1 -- 172.21.15.32:0/1048531897 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9370006440 con 0x7f93880bfab0 2021-04-09T18:18:03.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.539 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f93880d72a0 con 0x7f93880bfab0 2021-04-09T18:18:03.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.539 7f93857fa700 1 -- 172.21.15.32:0/1048531897 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v74) v1 ==== 100+0+10 (secure 0 0 0) 0x7f937000bdd0 con 0x7f93880bfab0 2021-04-09T18:18:03.606 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:18:03.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f936c0188d0 msgr2=0x7f936c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f936c0188d0 0x7f936c01add0 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880bfab0 msgr2=0x7f93880c09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880bfab0 0x7f93880c09c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f9370006f20 tx=0x7f937000b9e0).stop 2021-04-09T18:18:03.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 shutdown_connections 2021-04-09T18:18:03.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f936c0188d0 0x7f936c01add0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f93880bfab0 0x7f93880c09c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93880bfeb0 0x7f93880c0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 --2- 172.21.15.32:0/1048531897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93880b5800 0x7f93880b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 >> 172.21.15.32:0/1048531897 conn(0x7f93880c38e0 msgr2=0x7f93881b9230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:03.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 shutdown_connections 2021-04-09T18:18:03.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.542 7f938e3e7700 1 -- 172.21.15.32:0/1048531897 wait complete. 2021-04-09T18:18:03.643 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:18:03.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:03.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.778 7f3736693700 1 Processor -- start 2021-04-09T18:18:03.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.778 7f3736693700 1 -- start start 2021-04-09T18:18:03.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3736693700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37300cffa0 0x7f37300d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3736693700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37300d6c80 0x7f373010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3736693700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37300d7280 con 0x7f37300d7800 2021-04-09T18:18:03.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3736693700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37300d7080 con 0x7f37300d6c80 2021-04-09T18:18:03.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3736693700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37300d8580 con 0x7f37300cffa0 2021-04-09T18:18:03.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3734cb0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37300d6c80 0x7f373010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3734cb0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37300d6c80 0x7f373010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46096/0 (socket says 172.21.15.32:46096) 2021-04-09T18:18:03.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3734cb0700 1 -- 172.21.15.32:0/1567679141 learned_addr learned my addr 172.21.15.32:0/1567679141 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:03.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f372ffff700 1 --2- 172.21.15.32:0/1567679141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37300cffa0 0x7f37300d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3734cb0700 1 -- 172.21.15.32:0/1567679141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37300cffa0 msgr2=0x7f37300d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:03.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3734cb0700 1 --2- 172.21.15.32:0/1567679141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37300cffa0 0x7f37300d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3734cb0700 1 -- 172.21.15.32:0/1567679141 >> v1:172.21.15.86:6789/0 conn(0x7f37300d7800 legacy=0x7f37300d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3734cb0700 1 -- 172.21.15.32:0/1567679141 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37300d91e0 con 0x7f37300d6c80 2021-04-09T18:18:03.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f372ffff700 1 --2- 172.21.15.32:0/1567679141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37300cffa0 0x7f37300d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:03.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f3734cb0700 1 --2- 172.21.15.32:0/1567679141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37300d6c80 0x7f373010cfd0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7f3728006f00 tx=0x7f3728009f10).ready entity=mon.0 client_cookie=dc1fd68aec009776 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:03.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f372f7fe700 1 -- 172.21.15.32:0/1567679141 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372800b3b0 con 0x7f37300d6c80 2021-04-09T18:18:03.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.779 7f372f7fe700 1 -- 172.21.15.32:0/1567679141 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372800b730 con 0x7f37300d6c80 2021-04-09T18:18:03.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.780 7f372f7fe700 1 -- 172.21.15.32:0/1567679141 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372800bc10 con 0x7f37300d6c80 2021-04-09T18:18:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.780 7f3736693700 1 -- 172.21.15.32:0/1567679141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37300d6c80 msgr2=0x7f373010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.780 7f3736693700 1 --2- 172.21.15.32:0/1567679141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37300d6c80 0x7f373010cfd0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7f3728006f00 tx=0x7f3728009f10).stop 2021-04-09T18:18:03.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.780 7f3736693700 1 -- 172.21.15.32:0/1567679141 shutdown_connections 2021-04-09T18:18:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.780 7f3736693700 1 --2- 172.21.15.32:0/1567679141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37300d6c80 0x7f373010cfd0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.780 7f3736693700 1 --2- 172.21.15.32:0/1567679141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37300cffa0 0x7f37300d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.780 7f3736693700 1 -- 172.21.15.32:0/1567679141 >> 172.21.15.32:0/1567679141 conn(0x7f37300ce5c0 msgr2=0x7f37300cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:03.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.780 7f3736693700 1 -- 172.21.15.32:0/1567679141 shutdown_connections 2021-04-09T18:18:03.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.780 7f3736693700 1 -- 172.21.15.32:0/1567679141 wait complete. 2021-04-09T18:18:03.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.800 7f3736693700 1 Processor -- start 2021-04-09T18:18:03.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f3736693700 1 -- start start 2021-04-09T18:18:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f3736693700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37301929f0 0x7f3730192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f3736693700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373008ddc0 0x7f373008e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f3736693700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f373008e7e0 0x7f373008c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f3736693700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3730193410 con 0x7f37301929f0 2021-04-09T18:18:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f3736693700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37300a9140 con 0x7f373008ddc0 2021-04-09T18:18:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f3736693700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373008ce80 con 0x7f373008e7e0 2021-04-09T18:18:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f372ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37301929f0 0x7f3730192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f372ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37301929f0 0x7f3730192df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46102/0 (socket says 172.21.15.32:46102) 2021-04-09T18:18:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f372ffff700 1 -- 172.21.15.32:0/1932962903 learned_addr learned my addr 172.21.15.32:0/1932962903 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f3727fff700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373008ddc0 0x7f373008e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.801 7f3734cb0700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f373008e7e0 0x7f373008c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f372ffff700 1 -- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f373008e7e0 msgr2=0x7f373008c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f372ffff700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f373008e7e0 0x7f373008c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f372ffff700 1 -- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373008ddc0 msgr2=0x7f373008e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:03.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f372ffff700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373008ddc0 0x7f373008e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f372ffff700 1 -- 172.21.15.32:0/1932962903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373008d0a0 con 0x7f37301929f0 2021-04-09T18:18:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f3727fff700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373008ddc0 0x7f373008e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:03.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f372ffff700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37301929f0 0x7f3730192df0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f372000b580 tx=0x7f372000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f372d7fa700 1 -- 172.21.15.32:0/1932962903 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372000bd10 con 0x7f37301929f0 2021-04-09T18:18:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f372d7fa700 1 -- 172.21.15.32:0/1932962903 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3720000c90 con 0x7f37301929f0 2021-04-09T18:18:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f372d7fa700 1 -- 172.21.15.32:0/1932962903 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3720002220 con 0x7f37301929f0 2021-04-09T18:18:03.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f3736693700 1 -- 172.21.15.32:0/1932962903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373008d0a0 con 0x7f37301929f0 2021-04-09T18:18:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.802 7f3736693700 1 -- 172.21.15.32:0/1932962903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373008d0a0 con 0x7f37301929f0 2021-04-09T18:18:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.803 7f372d7fa700 1 -- 172.21.15.32:0/1932962903 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f372000a000 con 0x7f37301929f0 2021-04-09T18:18:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.803 7f372d7fa700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37140188c0 0x7f371401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:03.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.803 7f372d7fa700 1 -- 172.21.15.32:0/1932962903 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f37200327c0 con 0x7f37301929f0 2021-04-09T18:18:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.803 7f3727fff700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37140188c0 0x7f371401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.803 7f3727fff700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37140188c0 0x7f371401adc0 crc :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.823 7f3736693700 1 -- 172.21.15.32:0/1932962903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37301b70c0 con 0x7f37301929f0 2021-04-09T18:18:03.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:03.826 7f372d7fa700 1 -- 172.21.15.32:0/1932962903 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3720006bb0 con 0x7f37301929f0 2021-04-09T18:18:04.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.017 7f3736693700 1 -- 172.21.15.32:0/1932962903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37300d7280 con 0x7f37301929f0 2021-04-09T18:18:04.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.018 7f372d7fa700 1 -- 172.21.15.32:0/1932962903 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f372000a000 con 0x7f37301929f0 2021-04-09T18:18:04.055 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 79 2021-04-09T18:18:04.056 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:04.056 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":79,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":75},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:17:58.930994","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 -- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37140188c0 msgr2=0x7f371401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37140188c0 0x7f371401adc0 crc :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:04.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 -- 172.21.15.32:0/1932962903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37301929f0 msgr2=0x7f3730192df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:04.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37301929f0 0x7f3730192df0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f372000b580 tx=0x7f372000b920).stop 2021-04-09T18:18:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 -- 172.21.15.32:0/1932962903 shutdown_connections 2021-04-09T18:18:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37140188c0 0x7f371401adc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:04.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f373008e7e0 0x7f373008c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:04.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f373008ddc0 0x7f373008e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:04.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 --2- 172.21.15.32:0/1932962903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37301929f0 0x7f3730192df0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:04.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.020 7f3736693700 1 -- 172.21.15.32:0/1932962903 >> 172.21.15.32:0/1932962903 conn(0x7f3730189b00 msgr2=0x7f37301921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:04.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.021 7f3736693700 1 -- 172.21.15.32:0/1932962903 shutdown_connections 2021-04-09T18:18:04.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:04.021 7f3736693700 1 -- 172.21.15.32:0/1932962903 wait complete. 2021-04-09T18:18:04.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:17:56.928328', 'modified': '2021-04-09 18:17:58.930994', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6211}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6211': {'gid': 6211, 'name': 'b', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/1080414756', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 1080414756}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 1080414756}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:04.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:18:04.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:18:04.608 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:18:04.608 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:04.609 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:18:04.609 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 6211, 2021-04-09T18:18:04.609 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:18:04.610 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:18:04.610 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 7.6431919380000002, 2021-04-09T18:18:04.611 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 79, 2021-04-09T18:18:04.612 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 74, 2021-04-09T18:18:04.613 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 74, 2021-04-09T18:18:04.614 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 13.223178097 2021-04-09T18:18:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:18:04.616 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 6211, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.6431919380000002, "mdsmap_epoch": 79, "osdmap_epoch": 74, "osdmap_epoch_barrier": 74, "uptime": 13.223178097 } 2021-04-09T18:18:04.617 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:18:04.618 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:18:04.618 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:18:04.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:04.620 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:04.691 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:18:04.793 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:18:04.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:18:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:18:04.868 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:18:04.868 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:18:04.869 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:04.869 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:18:04.913 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:18:04.914 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:04.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:18:05.020 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:18:05.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:18:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:18:05.085 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:18:05.086 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:18:05.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:05.088 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:18:05.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:18:05.070 7fbbae309f80 -1 init, newargv = 0x55d2da8d9260 newargc=9 2021-04-09T18:18:05.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[47083]: starting ceph client 2021-04-09T18:18:05.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[47083]: starting fuse 2021-04-09T18:18:05.149 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:18:05.150 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:18:05.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:05.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:18:05.152 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:18:05.153 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:18:05.154 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:18:05.154 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:18:05.155 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:18:05.161 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:18:05.161 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:18:05.162 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:18:05.172 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:18:05.188 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:18:05.189 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:18:05.221 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:18:05.221 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:18:05.222 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:18:05.222 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:18:05.222 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:18:05.223 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:18:05.223 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:18:05.224 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:18:05.237 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:18:05.238 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:18:05.238 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:18:05.329 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.47083.asok 2021-04-09T18:18:05.334 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47083.asok 2021-04-09T18:18:05.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:05.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47083.asok status 2021-04-09T18:18:05.449 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:18:05.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:18:05.469 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:05.470 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:18:05.481 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:18:05.486 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:18:05.486 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:18:05.487 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:18:05.487 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:18:05.488 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "47083", 2021-04-09T18:18:05.488 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:18:05.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:05.499 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:18:05.501 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:18:05.501 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6462, 2021-04-09T18:18:05.501 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:18:05.502 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:18:05.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:18:05.513 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 6462 2021-04-09T18:18:05.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:05.514 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:18:05.515 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:05.515 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:18:05.515 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1461532854 2021-04-09T18:18:05.516 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:05.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:05.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:18:05.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:05.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:18:05.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1461532854 2021-04-09T18:18:05.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:05.519 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.6462 172.21.15.32:0/1461532854", 2021-04-09T18:18:05.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/1461532854", 2021-04-09T18:18:05.519 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:18:05.557 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 79, 2021-04-09T18:18:05.580 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 74, 2021-04-09T18:18:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:18:05.581 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:18:05.582 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:18:05.582 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:05.591 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:05.592 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:18:05.593 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:05.684 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:18:05.684 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:18:05.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:05.686 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:05.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:18:05.912 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:05.913 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:18:05.913 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:05.919 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:18:05.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.035 7f0949fc9700 1 Processor -- start 2021-04-09T18:18:06.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.035 7f0949fc9700 1 -- start start 2021-04-09T18:18:06.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0949fc9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f094410d010 0x7f09440d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0949fc9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f09440d1da0 0x7f09440d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0949fc9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09440d82c0 0x7f094410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0949fc9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09440d8d30 con 0x7f094410d010 2021-04-09T18:18:06.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0949fc9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09440d9220 con 0x7f09440d1da0 2021-04-09T18:18:06.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0949fc9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094410d4b0 con 0x7f09440d82c0 2021-04-09T18:18:06.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f09437fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f094410d010 0x7f09440d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f09437fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f094410d010 0x7f09440d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46122/0 (socket says 172.21.15.32:46122) 2021-04-09T18:18:06.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0942ffd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f09440d1da0 0x7f09440d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0943fff700 1 --2- 172.21.15.32:0/2310154632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09440d82c0 0x7f094410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f09437fe700 1 -- 172.21.15.32:0/2310154632 learned_addr learned my addr 172.21.15.32:0/2310154632 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f09437fe700 1 -- 172.21.15.32:0/2310154632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09440d82c0 msgr2=0x7f094410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f09437fe700 1 --2- 172.21.15.32:0/2310154632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09440d82c0 0x7f094410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f09437fe700 1 -- 172.21.15.32:0/2310154632 >> v2:172.21.15.86:3300/0 conn(0x7f09440d1da0 msgr2=0x7f09440d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:06.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f09437fe700 1 --2- 172.21.15.32:0/2310154632 >> v2:172.21.15.86:3300/0 conn(0x7f09440d1da0 0x7f09440d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f09437fe700 1 -- 172.21.15.32:0/2310154632 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09440d86c0 con 0x7f094410d010 2021-04-09T18:18:06.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0942ffd700 1 --2- 172.21.15.32:0/2310154632 >> v2:172.21.15.86:3300/0 conn(0x7f09440d1da0 0x7f09440d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f0943fff700 1 --2- 172.21.15.32:0/2310154632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09440d82c0 0x7f094410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.036 7f09437fe700 1 --2- 172.21.15.32:0/2310154632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f094410d010 0x7f09440d1780 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f0934006700 tx=0x7f0934009b80).ready entity=mon.0 client_cookie=8aa1dc9c66b0347c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:06.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f09427fc700 1 -- 172.21.15.32:0/2310154632 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f093400c0c0 con 0x7f094410d010 2021-04-09T18:18:06.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f09427fc700 1 -- 172.21.15.32:0/2310154632 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093400c440 con 0x7f094410d010 2021-04-09T18:18:06.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f09427fc700 1 -- 172.21.15.32:0/2310154632 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f093400c920 con 0x7f094410d010 2021-04-09T18:18:06.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f0949fc9700 1 -- 172.21.15.32:0/2310154632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f094410d010 msgr2=0x7f09440d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f0949fc9700 1 --2- 172.21.15.32:0/2310154632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f094410d010 0x7f09440d1780 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f0934006700 tx=0x7f0934009b80).stop 2021-04-09T18:18:06.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f0949fc9700 1 -- 172.21.15.32:0/2310154632 shutdown_connections 2021-04-09T18:18:06.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f0949fc9700 1 --2- 172.21.15.32:0/2310154632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09440d82c0 0x7f094410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f0949fc9700 1 --2- 172.21.15.32:0/2310154632 >> v2:172.21.15.86:3300/0 conn(0x7f09440d1da0 0x7f09440d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f0949fc9700 1 --2- 172.21.15.32:0/2310154632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f094410d010 0x7f09440d1780 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f0949fc9700 1 -- 172.21.15.32:0/2310154632 >> 172.21.15.32:0/2310154632 conn(0x7f09440ce5f0 msgr2=0x7f09440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:06.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f0949fc9700 1 -- 172.21.15.32:0/2310154632 shutdown_connections 2021-04-09T18:18:06.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.037 7f0949fc9700 1 -- 172.21.15.32:0/2310154632 wait complete. 2021-04-09T18:18:06.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.057 7f0949fc9700 1 Processor -- start 2021-04-09T18:18:06.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0949fc9700 1 -- start start 2021-04-09T18:18:06.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0949fc9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09441ac670 0x7f09441a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0949fc9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09441a5dc0 0x7f09440ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0949fc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09441aca70 0x7f09440ba800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0949fc9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09441a5220 con 0x7f09441aca70 2021-04-09T18:18:06.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0949fc9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094418e100 con 0x7f09441ac670 2021-04-09T18:18:06.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0949fc9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094419ccc0 con 0x7f09441a5dc0 2021-04-09T18:18:06.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0943fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09441aca70 0x7f09440ba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0943fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09441aca70 0x7f09440ba800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46132/0 (socket says 172.21.15.32:46132) 2021-04-09T18:18:06.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0943fff700 1 -- 172.21.15.32:0/1775593371 learned_addr learned my addr 172.21.15.32:0/1775593371 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:06.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f09437fe700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09441ac670 0x7f09441a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.058 7f0942ffd700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09441a5dc0 0x7f09440ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f0943fff700 1 -- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09441a5dc0 msgr2=0x7f09440ba1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f0943fff700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09441a5dc0 0x7f09440ba1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f0943fff700 1 -- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09441ac670 msgr2=0x7f09441a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f0943fff700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09441ac670 0x7f09441a57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f0943fff700 1 -- 172.21.15.32:0/1775593371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09440bae20 con 0x7f09441aca70 2021-04-09T18:18:06.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f0943fff700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09441aca70 0x7f09440ba800 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f093c00a940 tx=0x7f093c00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:06.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f092bfff700 1 -- 172.21.15.32:0/1775593371 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f093c001400 con 0x7f09441aca70 2021-04-09T18:18:06.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f092bfff700 1 -- 172.21.15.32:0/1775593371 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c00ae20 con 0x7f09441aca70 2021-04-09T18:18:06.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09440bae20 con 0x7f09441aca70 2021-04-09T18:18:06.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f092bfff700 1 -- 172.21.15.32:0/1775593371 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f093c001930 con 0x7f09441aca70 2021-04-09T18:18:06.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.059 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09440bae20 con 0x7f09441aca70 2021-04-09T18:18:06.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.060 7f092bfff700 1 -- 172.21.15.32:0/1775593371 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f093c0011f0 con 0x7f09441aca70 2021-04-09T18:18:06.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.060 7f092bfff700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09240188c0 0x7f092401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.060 7f092bfff700 1 -- 172.21.15.32:0/1775593371 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f093c032b00 con 0x7f09441aca70 2021-04-09T18:18:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.060 7f09437fe700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09240188c0 0x7f092401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.060 7f09437fe700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09240188c0 0x7f092401adc0 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:06.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.080 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0944182d10 con 0x7f09441aca70 2021-04-09T18:18:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.082 7f092bfff700 1 -- 172.21.15.32:0/1775593371 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f093c0066f0 con 0x7f09441aca70 2021-04-09T18:18:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.274 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0944182d10 con 0x7f09441aca70 2021-04-09T18:18:06.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.274 7f092bfff700 1 -- 172.21.15.32:0/1775593371 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2775 (secure 0 0 0) 0x7f093c0011f0 con 0x7f09441aca70 2021-04-09T18:18:06.329 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 79 2021-04-09T18:18:06.329 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:06.330 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":79,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":75},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:17:58.930994","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09240188c0 msgr2=0x7f092401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09240188c0 0x7f092401adc0 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09441aca70 msgr2=0x7f09440ba800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09441aca70 0x7f09440ba800 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f093c00a940 tx=0x7f093c00ace0).stop 2021-04-09T18:18:06.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 shutdown_connections 2021-04-09T18:18:06.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09240188c0 0x7f092401adc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09441aca70 0x7f09440ba800 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09441a5dc0 0x7f09440ba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 --2- 172.21.15.32:0/1775593371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09441ac670 0x7f09441a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 >> 172.21.15.32:0/1775593371 conn(0x7f094419b890 msgr2=0x7f09441a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:06.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 shutdown_connections 2021-04-09T18:18:06.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.277 7f0949fc9700 1 -- 172.21.15.32:0/1775593371 wait complete. 2021-04-09T18:18:06.415 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 6, "epoch": 79, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 6, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:17:56.928328", "damaged": [], "data_pools": [ 13 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6211": { "addr": "172.21.15.86:6835/1080414756", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 1080414756, "type": "v2" }, { "addr": "172.21.15.86:6835", "nonce": 1080414756, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6211, "incarnation": 77, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 12, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:17:58.930994", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6211 } } } ], "standbys": [ { "addr": "172.21.15.32:6833/2067133784", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 2067133784, "type": "v2" }, { "addr": "172.21.15.32:6833", "nonce": 2067133784, "type": "v1" } ] }, "epoch": 75, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6306, "incarnation": 0, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.32:6835/3097053176", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6834", "nonce": 3097053176, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 3097053176, "type": "v1" } ] }, "epoch": 76, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6315, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6837/713422835", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6836", "nonce": 713422835, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 713422835, "type": "v1" } ] }, "epoch": 76, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55577, "incarnation": 0, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:18:06.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.529 7fe1fdd90700 1 Processor -- start 2021-04-09T18:18:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.529 7fe1fdd90700 1 -- start start 2021-04-09T18:18:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.530 7fe1fdd90700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f80d6c80 0x7fe1f80d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.530 7fe1fdd90700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1f80d2440 0x7fe1f810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.530 7fe1fdd90700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80d7280 con 0x7fe1f80cffa0 2021-04-09T18:18:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.530 7fe1fdd90700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f80d7080 con 0x7fe1f80d2440 2021-04-09T18:18:06.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.530 7fe1fdd90700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80d91e0 con 0x7fe1f80d6c80 2021-04-09T18:18:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.530 7fe1f7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1f80d2440 0x7fe1f810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.530 7fe1f7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1f80d2440 0x7fe1f810cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46138/0 (socket says 172.21.15.32:46138) 2021-04-09T18:18:06.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.530 7fe1f7fff700 1 -- 172.21.15.32:0/1267313840 learned_addr learned my addr 172.21.15.32:0/1267313840 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f6ffd700 1 --2- 172.21.15.32:0/1267313840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f80d6c80 0x7fe1f80d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f7fff700 1 -- 172.21.15.32:0/1267313840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f80d6c80 msgr2=0x7fe1f80d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f7fff700 1 --2- 172.21.15.32:0/1267313840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f80d6c80 0x7fe1f80d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f7fff700 1 -- 172.21.15.32:0/1267313840 >> v1:172.21.15.86:6789/0 conn(0x7fe1f80cffa0 legacy=0x7fe1f80d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f7fff700 1 -- 172.21.15.32:0/1267313840 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f80d2840 con 0x7fe1f80d2440 2021-04-09T18:18:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f6ffd700 1 --2- 172.21.15.32:0/1267313840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f80d6c80 0x7fe1f80d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f7fff700 1 --2- 172.21.15.32:0/1267313840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f80d2440 0x7fe1f810cfd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7fe1f0006f00 tx=0x7fe1f0009f10).ready entity=mon.0 client_cookie=c08ad266b1ce9b1e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f67fc700 1 -- 172.21.15.32:0/1267313840 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f000b3b0 con 0x7fe1f80d2440 2021-04-09T18:18:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f67fc700 1 -- 172.21.15.32:0/1267313840 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f000b730 con 0x7fe1f80d2440 2021-04-09T18:18:06.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1f67fc700 1 -- 172.21.15.32:0/1267313840 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1f000bc10 con 0x7fe1f80d2440 2021-04-09T18:18:06.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1fdd90700 1 -- 172.21.15.32:0/1267313840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f80d2440 msgr2=0x7fe1f810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.531 7fe1fdd90700 1 --2- 172.21.15.32:0/1267313840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f80d2440 0x7fe1f810cfd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7fe1f0006f00 tx=0x7fe1f0009f10).stop 2021-04-09T18:18:06.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.532 7fe1fdd90700 1 -- 172.21.15.32:0/1267313840 shutdown_connections 2021-04-09T18:18:06.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.532 7fe1fdd90700 1 --2- 172.21.15.32:0/1267313840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f80d2440 0x7fe1f810cfd0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.532 7fe1fdd90700 1 --2- 172.21.15.32:0/1267313840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f80d6c80 0x7fe1f80d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.532 7fe1fdd90700 1 -- 172.21.15.32:0/1267313840 >> 172.21.15.32:0/1267313840 conn(0x7fe1f80ce5c0 msgr2=0x7fe1f80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:06.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.532 7fe1fdd90700 1 -- 172.21.15.32:0/1267313840 shutdown_connections 2021-04-09T18:18:06.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.532 7fe1fdd90700 1 -- 172.21.15.32:0/1267313840 wait complete. 2021-04-09T18:18:06.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.552 7fe1fdd90700 1 Processor -- start 2021-04-09T18:18:06.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.552 7fe1fdd90700 1 -- start start 2021-04-09T18:18:06.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1fdd90700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1f80a80c0 0x7fe1f817bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1fdd90700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f817b1d0 0x7fe1f817b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1fdd90700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f808dc50 0x7fe1f808e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1fdd90700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f817bc70 con 0x7fe1f817b1d0 2021-04-09T18:18:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1fdd90700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f808e670 con 0x7fe1f80a80c0 2021-04-09T18:18:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1fdd90700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f808e850 con 0x7fe1f808dc50 2021-04-09T18:18:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f817b1d0 0x7fe1f817b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f817b1d0 0x7fe1f817b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46142/0 (socket says 172.21.15.32:46142) 2021-04-09T18:18:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f6ffd700 1 -- 172.21.15.32:0/870747430 learned_addr learned my addr 172.21.15.32:0/870747430 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f77fe700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1f80a80c0 0x7fe1f817bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f7fff700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f808dc50 0x7fe1f808e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f6ffd700 1 -- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f808dc50 msgr2=0x7fe1f808e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f6ffd700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f808dc50 0x7fe1f808e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f6ffd700 1 -- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1f80a80c0 msgr2=0x7fe1f817bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f6ffd700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1f80a80c0 0x7fe1f817bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f6ffd700 1 -- 172.21.15.32:0/870747430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f808c660 con 0x7fe1f817b1d0 2021-04-09T18:18:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1f6ffd700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f817b1d0 0x7fe1f817b5d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7fe1ec00b580 tx=0x7fe1ec00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:06.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1dffff700 1 -- 172.21.15.32:0/870747430 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ec00bd10 con 0x7fe1f817b1d0 2021-04-09T18:18:06.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1dffff700 1 -- 172.21.15.32:0/870747430 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec000c90 con 0x7fe1f817b1d0 2021-04-09T18:18:06.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f808c660 con 0x7fe1f817b1d0 2021-04-09T18:18:06.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1dffff700 1 -- 172.21.15.32:0/870747430 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ec002220 con 0x7fe1f817b1d0 2021-04-09T18:18:06.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.553 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f808c660 con 0x7fe1f817b1d0 2021-04-09T18:18:06.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.554 7fe1dffff700 1 -- 172.21.15.32:0/870747430 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe1ec00a000 con 0x7fe1f817b1d0 2021-04-09T18:18:06.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.554 7fe1dffff700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1d8018930 0x7fe1d801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:06.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.554 7fe1dffff700 1 -- 172.21.15.32:0/870747430 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe1ec0329f0 con 0x7fe1f817b1d0 2021-04-09T18:18:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.554 7fe1f77fe700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1d8018930 0x7fe1d801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.555 7fe1f77fe700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1d8018930 0x7fe1d801adc0 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.574 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f81b70c0 con 0x7fe1f817b1d0 2021-04-09T18:18:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.578 7fe1dffff700 1 -- 172.21.15.32:0/870747430 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe1ec006bb0 con 0x7fe1f817b1d0 2021-04-09T18:18:06.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.769 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1f81b70c0 con 0x7fe1f817b1d0 2021-04-09T18:18:06.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.769 7fe1dffff700 1 -- 172.21.15.32:0/870747430 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2775 (secure 0 0 0) 0x7fe1ec000c90 con 0x7fe1f817b1d0 2021-04-09T18:18:06.806 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 79 2021-04-09T18:18:06.806 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:06.806 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":79,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":75},{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:17:58.930994","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:06.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1d8018930 msgr2=0x7fe1d801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1d8018930 0x7fe1d801adc0 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f817b1d0 msgr2=0x7fe1f817b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:06.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f817b1d0 0x7fe1f817b5d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7fe1ec00b580 tx=0x7fe1ec00b920).stop 2021-04-09T18:18:06.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 shutdown_connections 2021-04-09T18:18:06.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1d8018930 0x7fe1d801adc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1f817b1d0 0x7fe1f817b5d0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1f808dc50 0x7fe1f808e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 --2- 172.21.15.32:0/870747430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1f80a80c0 0x7fe1f817bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:06.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 >> 172.21.15.32:0/870747430 conn(0x7fe1f8189b00 msgr2=0x7fe1f81921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:06.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 shutdown_connections 2021-04-09T18:18:06.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.772 7fe1fdd90700 1 -- 172.21.15.32:0/870747430 wait complete. 2021-04-09T18:18:06.818 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:18:07.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.938 7fe7522f6700 1 Processor -- start 2021-04-09T18:18:07.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.938 7fe7522f6700 1 -- start start 2021-04-09T18:18:07.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe7522f6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c10d010 0x7fe74c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe7522f6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe74c0d8490 0x7fe74c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe7522f6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c10d480 con 0x7fe74c0d1da0 2021-04-09T18:18:07.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe7522f6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74c10e780 con 0x7fe74c0d8490 2021-04-09T18:18:07.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe7522f6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c0d8d30 con 0x7fe74c10d010 2021-04-09T18:18:07.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe74bfff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c10d010 0x7fe74c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe750913700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe74c0d8490 0x7fe74c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe74bfff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c10d010 0x7fe74c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:53376/0 (socket says 172.21.15.32:53376) 2021-04-09T18:18:07.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe74bfff700 1 -- 172.21.15.32:0/866798128 learned_addr learned my addr 172.21.15.32:0/866798128 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:07.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.939 7fe743fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe74c0d1da0 0x7fe74c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:53666/0 (socket says 172.21.15.32:53666) 2021-04-09T18:18:07.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe74bfff700 1 -- 172.21.15.32:0/866798128 >> v2:172.21.15.86:3300/0 conn(0x7fe74c0d8490 msgr2=0x7fe74c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:07.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe74bfff700 1 --2- 172.21.15.32:0/866798128 >> v2:172.21.15.86:3300/0 conn(0x7fe74c0d8490 0x7fe74c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe74bfff700 1 -- 172.21.15.32:0/866798128 >> v1:172.21.15.86:6789/0 conn(0x7fe74c0d1da0 legacy=0x7fe74c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe74bfff700 1 -- 172.21.15.32:0/866798128 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c0d9220 con 0x7fe74c10d010 2021-04-09T18:18:07.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe74b7fe700 1 -- 172.21.15.32:0/866798128 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48117174 0 0) 0x7fe734000c60 con 0x7fe74c0d1da0 2021-04-09T18:18:07.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe750913700 1 --2- 172.21.15.32:0/866798128 >> v2:172.21.15.86:3300/0 conn(0x7fe74c0d8490 0x7fe74c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:07.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe74bfff700 1 --2- 172.21.15.32:0/866798128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c10d010 0x7fe74c0d1780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe73c006700 tx=0x7fe73c009b80).ready entity=mon.2 client_cookie=69c27def18ab68d0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:07.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe74b7fe700 1 -- 172.21.15.32:0/866798128 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe73c00c0c0 con 0x7fe74c10d010 2021-04-09T18:18:07.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe74b7fe700 1 -- 172.21.15.32:0/866798128 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c00c440 con 0x7fe74c10d010 2021-04-09T18:18:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe74b7fe700 1 -- 172.21.15.32:0/866798128 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe73c00c920 con 0x7fe74c10d010 2021-04-09T18:18:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe7522f6700 1 -- 172.21.15.32:0/866798128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c10d010 msgr2=0x7fe74c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.940 7fe7522f6700 1 --2- 172.21.15.32:0/866798128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c10d010 0x7fe74c0d1780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe73c006700 tx=0x7fe73c009b80).stop 2021-04-09T18:18:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.941 7fe7522f6700 1 -- 172.21.15.32:0/866798128 shutdown_connections 2021-04-09T18:18:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.941 7fe7522f6700 1 --2- 172.21.15.32:0/866798128 >> v2:172.21.15.86:3300/0 conn(0x7fe74c0d8490 0x7fe74c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.941 7fe7522f6700 1 --2- 172.21.15.32:0/866798128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c10d010 0x7fe74c0d1780 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.941 7fe7522f6700 1 -- 172.21.15.32:0/866798128 >> 172.21.15.32:0/866798128 conn(0x7fe74c0ce5f0 msgr2=0x7fe74c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:07.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.941 7fe7522f6700 1 -- 172.21.15.32:0/866798128 shutdown_connections 2021-04-09T18:18:07.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.941 7fe7522f6700 1 -- 172.21.15.32:0/866798128 wait complete. 2021-04-09T18:18:07.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe7522f6700 1 Processor -- start 2021-04-09T18:18:07.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe7522f6700 1 -- start start 2021-04-09T18:18:07.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe7522f6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c1a0050 0x7fe74c1a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe7522f6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe74c0bdcd0 0x7fe74c0bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe7522f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe74c0bf080 0x7fe74c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe7522f6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c0bf500 con 0x7fe74c0bf080 2021-04-09T18:18:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe7522f6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c0bb7b0 con 0x7fe74c0bdcd0 2021-04-09T18:18:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe7522f6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c0be0d0 con 0x7fe74c1a0050 2021-04-09T18:18:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe750913700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe74c0bf080 0x7fe74c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe750913700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe74c0bf080 0x7fe74c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46160/0 (socket says 172.21.15.32:46160) 2021-04-09T18:18:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe743fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe74c0bdcd0 0x7fe74c0bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe750913700 1 -- 172.21.15.32:0/1017548853 learned_addr learned my addr 172.21.15.32:0/1017548853 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.962 7fe74bfff700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c1a0050 0x7fe74c1a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe750913700 1 -- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c1a0050 msgr2=0x7fe74c1a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe750913700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c1a0050 0x7fe74c1a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe750913700 1 -- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe74c0bdcd0 msgr2=0x7fe74c0bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe750913700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe74c0bdcd0 0x7fe74c0bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe750913700 1 -- 172.21.15.32:0/1017548853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c0b1310 con 0x7fe74c0bf080 2021-04-09T18:18:07.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe743fff700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe74c0bdcd0 0x7fe74c0bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:07.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe750913700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe74c0bf080 0x7fe74c0b0cf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7fe74400b970 tx=0x7fe74400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:07.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe7497fa700 1 -- 172.21.15.32:0/1017548853 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe744000f90 con 0x7fe74c0bf080 2021-04-09T18:18:07.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe7497fa700 1 -- 172.21.15.32:0/1017548853 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe744006aa0 con 0x7fe74c0bf080 2021-04-09T18:18:07.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74c0b1310 con 0x7fe74c0bf080 2021-04-09T18:18:07.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe7497fa700 1 -- 172.21.15.32:0/1017548853 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7440015f0 con 0x7fe74c0bf080 2021-04-09T18:18:07.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.963 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74c0b1310 con 0x7fe74c0bf080 2021-04-09T18:18:07.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.964 7fe7497fa700 1 -- 172.21.15.32:0/1017548853 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe744000ed0 con 0x7fe74c0bf080 2021-04-09T18:18:07.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.964 7fe7497fa700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe730018930 0x7fe73001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.964 7fe7497fa700 1 -- 172.21.15.32:0/1017548853 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe744032da0 con 0x7fe74c0bf080 2021-04-09T18:18:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.964 7fe74bfff700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe730018930 0x7fe73001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.964 7fe74bfff700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe730018930 0x7fe73001ae10 crc :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:07.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.983 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74c0bb7b0 con 0x7fe74c0bf080 2021-04-09T18:18:07.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:06.986 7fe7497fa700 1 -- 172.21.15.32:0/1017548853 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe7440066f0 con 0x7fe74c0bf080 2021-04-09T18:18:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.179 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fe74c0bb7b0 con 0x7fe74c0bf080 2021-04-09T18:18:07.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.530 7fe7497fa700 1 -- 172.21.15.32:0/1017548853 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v80) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe744000ed0 con 0x7fe74c0bf080 2021-04-09T18:18:07.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe730018930 msgr2=0x7fe73001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe730018930 0x7fe73001ae10 crc :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe74c0bf080 msgr2=0x7fe74c0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe74c0bf080 0x7fe74c0b0cf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7fe74400b970 tx=0x7fe74400bd10).stop 2021-04-09T18:18:07.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 shutdown_connections 2021-04-09T18:18:07.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe730018930 0x7fe73001ae10 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe74c0bf080 0x7fe74c0b0cf0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe74c0bdcd0 0x7fe74c0bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 --2- 172.21.15.32:0/1017548853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe74c1a0050 0x7fe74c1a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.545 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 >> 172.21.15.32:0/1017548853 conn(0x7fe74c0975f0 msgr2=0x7fe74c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:07.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.546 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 shutdown_connections 2021-04-09T18:18:07.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.546 7fe7522f6700 1 -- 172.21.15.32:0/1017548853 wait complete. 2021-04-09T18:18:07.592 INFO:tasks.cephfs.test_failover:Waiting for cluster to grow. 2021-04-09T18:18:07.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:07.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.709 7fe32ec24700 1 Processor -- start 2021-04-09T18:18:07.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.709 7fe32ec24700 1 -- start start 2021-04-09T18:18:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.710 7fe32ec24700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32810da00 0x7fe32810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.710 7fe32ec24700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32810d480 con 0x7fe3280d7ca0 2021-04-09T18:18:07.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.710 7fe32ec24700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3280d1f70 con 0x7fe32810d010 2021-04-09T18:18:07.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.710 7fe32ec24700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3280d9220 con 0x7fe32810da00 2021-04-09T18:18:07.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.710 7fe327fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe3280d7ca0 0x7fe3280d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55762/0 (socket says 172.21.15.32:55762) 2021-04-09T18:18:07.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.710 7fe327fff700 1 -- 172.21.15.32:0/858063400 learned_addr learned my addr 172.21.15.32:0/858063400 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:07.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.710 7fe32d241700 1 --2- 172.21.15.32:0/858063400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32810da00 0x7fe32810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.710 7fe3277fe700 1 -- 172.21.15.32:0/858063400 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085054843 0 0) 0x7fe318001060 con 0x7fe3280d7ca0 2021-04-09T18:18:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe3277fe700 1 -- 172.21.15.32:0/858063400 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe314002fe0 con 0x7fe3280d7ca0 2021-04-09T18:18:07.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe3277fe700 1 -- 172.21.15.32:0/858063400 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850479219 0 0) 0x7fe310000c60 con 0x7fe32810d010 2021-04-09T18:18:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe3277fe700 1 -- 172.21.15.32:0/858063400 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe314003210 con 0x7fe32810d010 2021-04-09T18:18:07.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe3277fe700 1 -- 172.21.15.32:0/858063400 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3912723501 0 0) 0x7fe318001460 con 0x7fe3280d7ca0 2021-04-09T18:18:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe32d241700 1 -- 172.21.15.32:0/858063400 >> v1:172.21.15.86:6789/0 conn(0x7fe32810d010 legacy=0x7fe3280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe32d241700 1 -- 172.21.15.32:0/858063400 >> v1:172.21.15.32:6789/0 conn(0x7fe3280d7ca0 legacy=0x7fe3280d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe32d241700 1 -- 172.21.15.32:0/858063400 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32810e420 con 0x7fe32810da00 2021-04-09T18:18:07.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe32d241700 1 --2- 172.21.15.32:0/858063400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32810da00 0x7fe32810de00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe320006ad0 tx=0x7fe320009a50).ready entity=mon.2 client_cookie=ad9ab97cdba75466 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:07.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe3277fe700 1 -- 172.21.15.32:0/858063400 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3200066a0 con 0x7fe32810da00 2021-04-09T18:18:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe3277fe700 1 -- 172.21.15.32:0/858063400 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe320006410 con 0x7fe32810da00 2021-04-09T18:18:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe3277fe700 1 -- 172.21.15.32:0/858063400 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3200024d0 con 0x7fe32810da00 2021-04-09T18:18:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe32ec24700 1 -- 172.21.15.32:0/858063400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32810da00 msgr2=0x7fe32810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.711 7fe32ec24700 1 --2- 172.21.15.32:0/858063400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32810da00 0x7fe32810de00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe320006ad0 tx=0x7fe320009a50).stop 2021-04-09T18:18:07.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.712 7fe32ec24700 1 -- 172.21.15.32:0/858063400 shutdown_connections 2021-04-09T18:18:07.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.712 7fe32ec24700 1 --2- 172.21.15.32:0/858063400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32810da00 0x7fe32810de00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.712 7fe32ec24700 1 -- 172.21.15.32:0/858063400 >> 172.21.15.32:0/858063400 conn(0x7fe3280ce5f0 msgr2=0x7fe3280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:07.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.712 7fe32ec24700 1 -- 172.21.15.32:0/858063400 shutdown_connections 2021-04-09T18:18:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.712 7fe32ec24700 1 -- 172.21.15.32:0/858063400 wait complete. 2021-04-09T18:18:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.732 7fe32ec24700 1 Processor -- start 2021-04-09T18:18:07.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.732 7fe32ec24700 1 -- start start 2021-04-09T18:18:07.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ec24700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3281a19b0 0x7fe32808ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ec24700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32817dee0 0x7fe32817e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ec24700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328179e60 0x7fe32817a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ec24700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32817e900 con 0x7fe3281a19b0 2021-04-09T18:18:07.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ec24700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3281798e0 con 0x7fe328179e60 2021-04-09T18:18:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ec24700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe328179610 con 0x7fe32817dee0 2021-04-09T18:18:07.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ca40700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3281a19b0 0x7fe32808ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ca40700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3281a19b0 0x7fe32808ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46176/0 (socket says 172.21.15.32:46176) 2021-04-09T18:18:07.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ca40700 1 -- 172.21.15.32:0/43632098 learned_addr learned my addr 172.21.15.32:0/43632098 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:07.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32d241700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328179e60 0x7fe32817a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe327fff700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32817dee0 0x7fe32817e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ca40700 1 -- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32817dee0 msgr2=0x7fe32817e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ca40700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32817dee0 0x7fe32817e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ca40700 1 -- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328179e60 msgr2=0x7fe32817a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ca40700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328179e60 0x7fe32817a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ca40700 1 -- 172.21.15.32:0/43632098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3281a1db0 con 0x7fe3281a19b0 2021-04-09T18:18:07.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe327fff700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32817dee0 0x7fe32817e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:07.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ca40700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3281a19b0 0x7fe32808ef30 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7fe310006f30 tx=0x7fe31000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:07.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe3257fa700 1 -- 172.21.15.32:0/43632098 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe310006030 con 0x7fe3281a19b0 2021-04-09T18:18:07.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe3257fa700 1 -- 172.21.15.32:0/43632098 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe310006370 con 0x7fe3281a19b0 2021-04-09T18:18:07.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe32ec24700 1 -- 172.21.15.32:0/43632098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3281a1db0 con 0x7fe3281a19b0 2021-04-09T18:18:07.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.733 7fe3257fa700 1 -- 172.21.15.32:0/43632098 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3100057c0 con 0x7fe3281a19b0 2021-04-09T18:18:07.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.734 7fe32ec24700 1 -- 172.21.15.32:0/43632098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3281a1db0 con 0x7fe3281a19b0 2021-04-09T18:18:07.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.734 7fe3257fa700 1 -- 172.21.15.32:0/43632098 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe310005540 con 0x7fe3281a19b0 2021-04-09T18:18:07.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.734 7fe3257fa700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe30c018930 0x7fe30c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:07.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.734 7fe3257fa700 1 -- 172.21.15.32:0/43632098 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe310032a10 con 0x7fe3281a19b0 2021-04-09T18:18:07.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.734 7fe327fff700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe30c018930 0x7fe30c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:07.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.735 7fe327fff700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe30c018930 0x7fe30c01ae10 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:07.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.754 7fe32ec24700 1 -- 172.21.15.32:0/43632098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32817e900 con 0x7fe3281a19b0 2021-04-09T18:18:07.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.758 7fe3257fa700 1 -- 172.21.15.32:0/43632098 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe310006c20 con 0x7fe3281a19b0 2021-04-09T18:18:07.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.949 7fe32ec24700 1 -- 172.21.15.32:0/43632098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe32817e900 con 0x7fe3281a19b0 2021-04-09T18:18:07.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.950 7fe3257fa700 1 -- 172.21.15.32:0/43632098 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2791 (secure 0 0 0) 0x7fe310006370 con 0x7fe3281a19b0 2021-04-09T18:18:07.987 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 80 2021-04-09T18:18:07.988 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:07.988 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":80,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:18:07.528973","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6211,"mds_1":6306},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6306":{"gid":6306,"name":"a","rank":1,"incarnation":80,"state":"up:creating","state_seq":2,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:07.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 -- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe30c018930 msgr2=0x7fe30c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe30c018930 0x7fe30c01ae10 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 -- 172.21.15.32:0/43632098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3281a19b0 msgr2=0x7fe32808ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:07.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3281a19b0 0x7fe32808ef30 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7fe310006f30 tx=0x7fe31000b960).stop 2021-04-09T18:18:07.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 -- 172.21.15.32:0/43632098 shutdown_connections 2021-04-09T18:18:07.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe30c018930 0x7fe30c01ae10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328179e60 0x7fe32817a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe32817dee0 0x7fe32817e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 --2- 172.21.15.32:0/43632098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3281a19b0 0x7fe32808ef30 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:07.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 -- 172.21.15.32:0/43632098 >> 172.21.15.32:0/43632098 conn(0x7fe328191660 msgr2=0x7fe328192d20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:07.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 -- 172.21.15.32:0/43632098 shutdown_connections 2021-04-09T18:18:07.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:07.952 7fe32ec24700 1 -- 172.21.15.32:0/43632098 wait complete. 2021-04-09T18:18:08.408 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:17:56.928328', 'modified': '2021-04-09 18:18:07.528973', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6211, 'mds_1': 6306}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6211': {'gid': 6211, 'name': 'b', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/1080414756', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 1080414756}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 1080414756}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6306': {'gid': 6306, 'name': 'a', 'rank': 1, 'incarnation': 80, 'state': 'up:creating', 'state_seq': 2, 'addr': '172.21.15.32:6833/2067133784', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2067133784}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2067133784}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:08.409 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6306:up:creating 2021-04-09T18:18:09.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.534 7f32007d1700 1 Processor -- start 2021-04-09T18:18:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.534 7f32007d1700 1 -- start start 2021-04-09T18:18:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f32007d1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31f810d010 0x7f31f80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:09.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f32007d1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80d1da0 0x7f31f80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:09.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f32007d1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31f80d82c0 0x7f31f810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:09.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f32007d1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f80d8d30 con 0x7f31f80d82c0 2021-04-09T18:18:09.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f32007d1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f80d9220 con 0x7f31f810d010 2021-04-09T18:18:09.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f32007d1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f810d480 con 0x7f31f80d1da0 2021-04-09T18:18:09.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fedee700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31f80d82c0 0x7f31f810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:09.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fedee700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31f80d82c0 0x7f31f810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46188/0 (socket says 172.21.15.32:46188) 2021-04-09T18:18:09.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fedee700 1 -- 172.21.15.32:0/988256714 learned_addr learned my addr 172.21.15.32:0/988256714 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:09.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fe5ed700 1 --2- 172.21.15.32:0/988256714 >> v2:172.21.15.86:3300/0 conn(0x7f31f810d010 0x7f31f80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:09.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fddec700 1 --2- 172.21.15.32:0/988256714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80d1da0 0x7f31f80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:09.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fedee700 1 -- 172.21.15.32:0/988256714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80d1da0 msgr2=0x7f31f80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:09.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fedee700 1 --2- 172.21.15.32:0/988256714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80d1da0 0x7f31f80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fedee700 1 -- 172.21.15.32:0/988256714 >> v2:172.21.15.86:3300/0 conn(0x7f31f810d010 msgr2=0x7f31f80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:09.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fedee700 1 --2- 172.21.15.32:0/988256714 >> v2:172.21.15.86:3300/0 conn(0x7f31f810d010 0x7f31f80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.535 7f31fedee700 1 -- 172.21.15.32:0/988256714 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f80d86c0 con 0x7f31f80d82c0 2021-04-09T18:18:09.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.536 7f31fedee700 1 --2- 172.21.15.32:0/988256714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80d82c0 0x7f31f810da00 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f31f0006f00 tx=0x7f31f0009e90).ready entity=mon.0 client_cookie=f854ac8d1cc9d89e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:09.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.536 7f31fd5eb700 1 -- 172.21.15.32:0/988256714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31f000b300 con 0x7f31f80d82c0 2021-04-09T18:18:09.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.536 7f31fd5eb700 1 -- 172.21.15.32:0/988256714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f000b640 con 0x7f31f80d82c0 2021-04-09T18:18:09.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.536 7f31fd5eb700 1 -- 172.21.15.32:0/988256714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31f000baa0 con 0x7f31f80d82c0 2021-04-09T18:18:09.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.536 7f32007d1700 1 -- 172.21.15.32:0/988256714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80d82c0 msgr2=0x7f31f810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:09.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.536 7f32007d1700 1 --2- 172.21.15.32:0/988256714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80d82c0 0x7f31f810da00 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f31f0006f00 tx=0x7f31f0009e90).stop 2021-04-09T18:18:09.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.537 7f32007d1700 1 -- 172.21.15.32:0/988256714 shutdown_connections 2021-04-09T18:18:09.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.537 7f32007d1700 1 --2- 172.21.15.32:0/988256714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80d82c0 0x7f31f810da00 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.537 7f32007d1700 1 --2- 172.21.15.32:0/988256714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80d1da0 0x7f31f80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.537 7f32007d1700 1 --2- 172.21.15.32:0/988256714 >> v2:172.21.15.86:3300/0 conn(0x7f31f810d010 0x7f31f80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.537 7f32007d1700 1 -- 172.21.15.32:0/988256714 >> 172.21.15.32:0/988256714 conn(0x7f31f80ce5f0 msgr2=0x7f31f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:09.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.537 7f32007d1700 1 -- 172.21.15.32:0/988256714 shutdown_connections 2021-04-09T18:18:09.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.537 7f32007d1700 1 -- 172.21.15.32:0/988256714 wait complete. 2021-04-09T18:18:09.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.557 7f32007d1700 1 Processor -- start 2021-04-09T18:18:09.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.557 7f32007d1700 1 -- start start 2021-04-09T18:18:09.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f32007d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80b5360 0x7f31f81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:09.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f32007d1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f818e140 0x7f31f81aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:09.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f32007d1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f81ad040 0x7f31f808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:09.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f32007d1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f81a5360 con 0x7f31f80b5360 2021-04-09T18:18:09.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f32007d1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f80902c0 con 0x7f31f818e140 2021-04-09T18:18:09.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f32007d1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f80904e0 con 0x7f31f81ad040 2021-04-09T18:18:09.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fe5ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80b5360 0x7f31f81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:09.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fe5ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80b5360 0x7f31f81a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46192/0 (socket says 172.21.15.32:46192) 2021-04-09T18:18:09.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fe5ed700 1 -- 172.21.15.32:0/509644228 learned_addr learned my addr 172.21.15.32:0/509644228 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:09.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fddec700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f818e140 0x7f31f81aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:09.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fedee700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f81ad040 0x7f31f808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:09.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fe5ed700 1 -- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f81ad040 msgr2=0x7f31f808fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:09.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fe5ed700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f81ad040 0x7f31f808fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fe5ed700 1 -- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f818e140 msgr2=0x7f31f81aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:09.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fe5ed700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f818e140 0x7f31f81aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fe5ed700 1 -- 172.21.15.32:0/509644228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f8183c80 con 0x7f31f80b5360 2021-04-09T18:18:09.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fddec700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f818e140 0x7f31f81aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:09.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fedee700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f81ad040 0x7f31f808fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:09.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31fe5ed700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80b5360 0x7f31f81a4d40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7f31f4006f60 tx=0x7f31f400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:09.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31eeffd700 1 -- 172.21.15.32:0/509644228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31f400bd60 con 0x7f31f80b5360 2021-04-09T18:18:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31eeffd700 1 -- 172.21.15.32:0/509644228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f4000c90 con 0x7f31f80b5360 2021-04-09T18:18:09.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f32007d1700 1 -- 172.21.15.32:0/509644228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f8183c80 con 0x7f31f80b5360 2021-04-09T18:18:09.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f31eeffd700 1 -- 172.21.15.32:0/509644228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31f4002220 con 0x7f31f80b5360 2021-04-09T18:18:09.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.558 7f32007d1700 1 -- 172.21.15.32:0/509644228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f8183c80 con 0x7f31f80b5360 2021-04-09T18:18:09.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.559 7f31eeffd700 1 -- 172.21.15.32:0/509644228 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f31f40052e0 con 0x7f31f80b5360 2021-04-09T18:18:09.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.559 7f31eeffd700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f31e0018930 0x7f31e001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:09.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.559 7f31eeffd700 1 -- 172.21.15.32:0/509644228 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f31f4032a00 con 0x7f31f80b5360 2021-04-09T18:18:09.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.559 7f31fddec700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f31e0018930 0x7f31e001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:09.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.560 7f31fddec700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f31e0018930 0x7f31e001ae10 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.579 7f32007d1700 1 -- 172.21.15.32:0/509644228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f8182d70 con 0x7f31f80b5360 2021-04-09T18:18:09.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.582 7f31eeffd700 1 -- 172.21.15.32:0/509644228 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f31f4000c90 con 0x7f31f80b5360 2021-04-09T18:18:09.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.774 7f32007d1700 1 -- 172.21.15.32:0/509644228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31f8182d70 con 0x7f31f80b5360 2021-04-09T18:18:09.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.775 7f31eeffd700 1 -- 172.21.15.32:0/509644228 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2789 (secure 0 0 0) 0x7f31f4006b00 con 0x7f31f80b5360 2021-04-09T18:18:09.812 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 81 2021-04-09T18:18:09.812 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:09.813 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:18:08.530171","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6211,"mds_1":6306},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6306":{"gid":6306,"name":"a","rank":1,"incarnation":80,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:09.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 -- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f31e0018930 msgr2=0x7f31e001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:09.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f31e0018930 0x7f31e001ae10 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 -- 172.21.15.32:0/509644228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80b5360 msgr2=0x7f31f81a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:09.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80b5360 0x7f31f81a4d40 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7f31f4006f60 tx=0x7f31f400b970).stop 2021-04-09T18:18:09.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 -- 172.21.15.32:0/509644228 shutdown_connections 2021-04-09T18:18:09.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f31e0018930 0x7f31e001ae10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f81ad040 0x7f31f808fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f818e140 0x7f31f81aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 --2- 172.21.15.32:0/509644228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31f80b5360 0x7f31f81a4d40 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:09.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 -- 172.21.15.32:0/509644228 >> 172.21.15.32:0/509644228 conn(0x7f31f819b460 msgr2=0x7f31f819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:09.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 -- 172.21.15.32:0/509644228 shutdown_connections 2021-04-09T18:18:09.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:09.777 7f32007d1700 1 -- 172.21.15.32:0/509644228 wait complete. 2021-04-09T18:18:10.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:17:56.928328', 'modified': '2021-04-09 18:18:08.530171', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6211, 'mds_1': 6306}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6211': {'gid': 6211, 'name': 'b', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/1080414756', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 1080414756}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 1080414756}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6306': {'gid': 6306, 'name': 'a', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6833/2067133784', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2067133784}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2067133784}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:10.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:18:10.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:18:10.369 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:18:10.369 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:10.369 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:18:10.370 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 6211, 2021-04-09T18:18:10.370 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:18:10.370 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:18:10.370 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 13.403592702999999, 2021-04-09T18:18:10.370 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 81, 2021-04-09T18:18:10.371 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 74, 2021-04-09T18:18:10.371 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 74, 2021-04-09T18:18:10.371 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 18.983578212000001 2021-04-09T18:18:10.371 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:18:10.375 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 6211, "want_state": "up:active", "state": "up:active", "rank_uptime": 13.403592702999999, "mdsmap_epoch": 81, "osdmap_epoch": 74, "osdmap_epoch_barrier": 74, "uptime": 18.983578212000001 } 2021-04-09T18:18:10.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:18:10.527 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:10.528 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:10.528 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:18:10.529 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6306, 2021-04-09T18:18:10.530 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:18:10.530 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:18:10.530 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 2.9543981609999999, 2021-04-09T18:18:10.531 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 81, 2021-04-09T18:18:10.531 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 74, 2021-04-09T18:18:10.531 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 74, 2021-04-09T18:18:10.532 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 19.126706714000001 2021-04-09T18:18:10.532 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:10.533 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 6306, "want_state": "up:active", "state": "up:active", "rank_uptime": 2.9543981609999999, "mdsmap_epoch": 81, "osdmap_epoch": 74, "osdmap_epoch_barrier": 74, "uptime": 19.126706714000001 } 2021-04-09T18:18:10.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:18:10.665 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:10.665 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:18:10.666 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:10.669 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:18:10.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:10.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.783 7fe2c272c700 1 Processor -- start 2021-04-09T18:18:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.783 7fe2c272c700 1 -- start start 2021-04-09T18:18:10.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2c272c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe2bc10d010 0x7fe2bc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:10.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2c272c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc0d1da0 0x7fe2bc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:10.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2c272c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc10d480 con 0x7fe2bc0d82c0 2021-04-09T18:18:10.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2c272c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2bc0d9220 con 0x7fe2bc10d010 2021-04-09T18:18:10.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2c272c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc0d86c0 con 0x7fe2bc0d1da0 2021-04-09T18:18:10.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2c0d49700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe2bc0d82c0 0x7fe2bc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55792/0 (socket says 172.21.15.32:55792) 2021-04-09T18:18:10.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2c0d49700 1 -- 172.21.15.32:0/1014467618 learned_addr learned my addr 172.21.15.32:0/1014467618 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:10.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2bbfff700 1 --2- 172.21.15.32:0/1014467618 >> v2:172.21.15.86:3300/0 conn(0x7fe2bc10d010 0x7fe2bc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:10.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2b3fff700 1 --2- 172.21.15.32:0/1014467618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc0d1da0 0x7fe2bc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:10.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.784 7fe2bb7fe700 1 -- 172.21.15.32:0/1014467618 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190807796 0 0) 0x7fe2b4000f90 con 0x7fe2bc0d82c0 2021-04-09T18:18:10.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bb7fe700 1 -- 172.21.15.32:0/1014467618 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a8002fe0 con 0x7fe2bc0d82c0 2021-04-09T18:18:10.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bbfff700 1 -- 172.21.15.32:0/1014467618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc0d1da0 msgr2=0x7fe2bc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:10.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bbfff700 1 --2- 172.21.15.32:0/1014467618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc0d1da0 0x7fe2bc0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:10.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bbfff700 1 -- 172.21.15.32:0/1014467618 >> v1:172.21.15.32:6789/0 conn(0x7fe2bc0d82c0 legacy=0x7fe2bc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:10.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bb7fe700 1 -- 172.21.15.32:0/1014467618 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1786820500 0 0) 0x7fe2b4001460 con 0x7fe2bc0d82c0 2021-04-09T18:18:10.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bbfff700 1 -- 172.21.15.32:0/1014467618 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc10e1f0 con 0x7fe2bc10d010 2021-04-09T18:18:10.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bbfff700 1 --2- 172.21.15.32:0/1014467618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc10d010 0x7fe2bc0d1780 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe2ac006650 tx=0x7fe2ac009a00).ready entity=mon.1 client_cookie=41d7acc8e5c3d597 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:10.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bb7fe700 1 -- 172.21.15.32:0/1014467618 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2ac009df0 con 0x7fe2bc10d010 2021-04-09T18:18:10.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bb7fe700 1 -- 172.21.15.32:0/1014467618 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac0009b0 con 0x7fe2bc10d010 2021-04-09T18:18:10.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2bb7fe700 1 -- 172.21.15.32:0/1014467618 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2ac00c2b0 con 0x7fe2bc10d010 2021-04-09T18:18:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2c272c700 1 -- 172.21.15.32:0/1014467618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc10d010 msgr2=0x7fe2bc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.785 7fe2c272c700 1 --2- 172.21.15.32:0/1014467618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc10d010 0x7fe2bc0d1780 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe2ac006650 tx=0x7fe2ac009a00).stop 2021-04-09T18:18:10.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.786 7fe2c272c700 1 -- 172.21.15.32:0/1014467618 shutdown_connections 2021-04-09T18:18:10.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.786 7fe2c272c700 1 --2- 172.21.15.32:0/1014467618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc0d1da0 0x7fe2bc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:10.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.786 7fe2c272c700 1 --2- 172.21.15.32:0/1014467618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc10d010 0x7fe2bc0d1780 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.786 7fe2c272c700 1 -- 172.21.15.32:0/1014467618 >> 172.21.15.32:0/1014467618 conn(0x7fe2bc0ce5f0 msgr2=0x7fe2bc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:10.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.786 7fe2c272c700 1 -- 172.21.15.32:0/1014467618 shutdown_connections 2021-04-09T18:18:10.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.786 7fe2c272c700 1 -- 172.21.15.32:0/1014467618 wait complete. 2021-04-09T18:18:10.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.806 7fe2c272c700 1 Processor -- start 2021-04-09T18:18:10.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.806 7fe2c272c700 1 -- start start 2021-04-09T18:18:10.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2c272c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc1ac380 0x7fe2bc18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:10.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2c272c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2bc1ac780 0x7fe2bc18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2c272c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc183520 0x7fe2bc183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:10.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2c272c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc1ace10 con 0x7fe2bc1ac780 2021-04-09T18:18:10.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2c272c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc189cf0 con 0x7fe2bc1ac380 2021-04-09T18:18:10.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2c272c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc183f40 con 0x7fe2bc183520 2021-04-09T18:18:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2bc1ac780 0x7fe2bc18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2bc1ac780 0x7fe2bc18a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46204/0 (socket says 172.21.15.32:46204) 2021-04-09T18:18:10.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b3fff700 1 -- 172.21.15.32:0/3764960942 learned_addr learned my addr 172.21.15.32:0/3764960942 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2bbfff700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc1ac380 0x7fe2bc18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2c0d49700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc183520 0x7fe2bc183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:10.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b3fff700 1 -- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc183520 msgr2=0x7fe2bc183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:10.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b3fff700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc183520 0x7fe2bc183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:10.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b3fff700 1 -- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc1ac380 msgr2=0x7fe2bc18a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:10.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b3fff700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc1ac380 0x7fe2bc18a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:10.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b3fff700 1 -- 172.21.15.32:0/3764960942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc184160 con 0x7fe2bc1ac780 2021-04-09T18:18:10.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2bbfff700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc1ac380 0x7fe2bc18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:10.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2c0d49700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc183520 0x7fe2bc183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:10.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b3fff700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2bc1ac780 0x7fe2bc18a850 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7fe2a4006f60 tx=0x7fe2a400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:10.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b97fa700 1 -- 172.21.15.32:0/3764960942 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2a400bd60 con 0x7fe2bc1ac780 2021-04-09T18:18:10.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.807 7fe2b97fa700 1 -- 172.21.15.32:0/3764960942 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a4000c90 con 0x7fe2bc1ac780 2021-04-09T18:18:10.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.808 7fe2b97fa700 1 -- 172.21.15.32:0/3764960942 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2a4002220 con 0x7fe2bc1ac780 2021-04-09T18:18:10.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.808 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2bc184160 con 0x7fe2bc1ac780 2021-04-09T18:18:10.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.808 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2bc184160 con 0x7fe2bc1ac780 2021-04-09T18:18:10.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.808 7fe2b97fa700 1 -- 172.21.15.32:0/3764960942 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe2a40052e0 con 0x7fe2bc1ac780 2021-04-09T18:18:10.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.808 7fe2b97fa700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2a00188c0 0x7fe2a001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:10.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.808 7fe2b97fa700 1 -- 172.21.15.32:0/3764960942 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe2a4032a00 con 0x7fe2bc1ac780 2021-04-09T18:18:11.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.808 7fe2bbfff700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2a00188c0 0x7fe2a001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.809 7fe2bbfff700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2a00188c0 0x7fe2a001adc0 crc :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:11.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.828 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2bc189cf0 con 0x7fe2bc1ac780 2021-04-09T18:18:11.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:10.832 7fe2b97fa700 1 -- 172.21.15.32:0/3764960942 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe2a4000c90 con 0x7fe2bc1ac780 2021-04-09T18:18:11.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.022 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2bc189cf0 con 0x7fe2bc1ac780 2021-04-09T18:18:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.023 7fe2b97fa700 1 -- 172.21.15.32:0/3764960942 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2789 (secure 0 0 0) 0x7fe2a4006b00 con 0x7fe2bc1ac780 2021-04-09T18:18:11.060 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 81 2021-04-09T18:18:11.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:11.062 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:18:08.530171","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6211,"mds_1":6306},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6306":{"gid":6306,"name":"a","rank":1,"incarnation":80,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:11.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2a00188c0 msgr2=0x7fe2a001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2a00188c0 0x7fe2a001adc0 crc :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2bc1ac780 msgr2=0x7fe2bc18a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2bc1ac780 0x7fe2bc18a850 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7fe2a4006f60 tx=0x7fe2a400b970).stop 2021-04-09T18:18:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 shutdown_connections 2021-04-09T18:18:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2a00188c0 0x7fe2a001adc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2bc1ac780 0x7fe2bc18a850 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2bc183520 0x7fe2bc183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 --2- 172.21.15.32:0/3764960942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2bc1ac380 0x7fe2bc18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 >> 172.21.15.32:0/3764960942 conn(0x7fe2bc19b890 msgr2=0x7fe2bc1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:11.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 shutdown_connections 2021-04-09T18:18:11.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.025 7fe2c272c700 1 -- 172.21.15.32:0/3764960942 wait complete. 2021-04-09T18:18:11.101 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 6, "epoch": 81, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 6, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:17:56.928328", "damaged": [], "data_pools": [ 13 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6211": { "addr": "172.21.15.86:6835/1080414756", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 1080414756, "type": "v2" }, { "addr": "172.21.15.86:6835", "nonce": 1080414756, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6211, "incarnation": 77, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 }, "gid_6306": { "addr": "172.21.15.32:6833/2067133784", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 2067133784, "type": "v2" }, { "addr": "172.21.15.32:6833", "nonce": 2067133784, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6306, "incarnation": 80, "name": "a", "rank": 1, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 2, "metadata_pool": 12, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:18:08.530171", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6211, "mds_1": 6306 } } } ], "standbys": [ { "addr": "172.21.15.32:6835/3097053176", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6834", "nonce": 3097053176, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 3097053176, "type": "v1" } ] }, "epoch": 76, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6315, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6837/713422835", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6836", "nonce": 713422835, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 713422835, "type": "v1" } ] }, "epoch": 76, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55577, "incarnation": 0, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:18:11.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.226 7f7e97205700 1 Processor -- start 2021-04-09T18:18:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.226 7f7e97205700 1 -- start start 2021-04-09T18:18:11.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e97205700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e9010d010 0x7f7e900d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e97205700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e900d8490 0x7f7e9010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e97205700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9010d480 con 0x7f7e900d1da0 2021-04-09T18:18:11.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e97205700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9010e780 con 0x7f7e900d8490 2021-04-09T18:18:11.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e97205700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900d8d30 con 0x7f7e9010d010 2021-04-09T18:18:11.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e95822700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7e900d8490 0x7f7e9010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e94820700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7e900d1da0 0x7f7e900d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55806/0 (socket says 172.21.15.32:55806) 2021-04-09T18:18:11.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e94820700 1 -- 172.21.15.32:0/3974634551 learned_addr learned my addr 172.21.15.32:0/3974634551 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:11.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e95021700 1 --2- 172.21.15.32:0/3974634551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e9010d010 0x7f7e900d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e95822700 1 -- 172.21.15.32:0/3974634551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e9010d010 msgr2=0x7f7e900d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e95822700 1 --2- 172.21.15.32:0/3974634551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e9010d010 0x7f7e900d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e95822700 1 -- 172.21.15.32:0/3974634551 >> v1:172.21.15.32:6789/0 conn(0x7f7e900d1da0 legacy=0x7f7e900d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e87fff700 1 -- 172.21.15.32:0/3974634551 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663339103 0 0) 0x7f7e80000ce0 con 0x7f7e900d1da0 2021-04-09T18:18:11.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.227 7f7e95822700 1 -- 172.21.15.32:0/3974634551 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e900d9220 con 0x7f7e900d8490 2021-04-09T18:18:11.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.228 7f7e95822700 1 --2- 172.21.15.32:0/3974634551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e900d8490 0x7f7e9010da00 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f7e88006e30 tx=0x7f7e88009de0).ready entity=mon.0 client_cookie=f6f372df762b628d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:11.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.228 7f7e87fff700 1 -- 172.21.15.32:0/3974634551 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e8800b300 con 0x7f7e900d8490 2021-04-09T18:18:11.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.228 7f7e87fff700 1 -- 172.21.15.32:0/3974634551 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8800b680 con 0x7f7e900d8490 2021-04-09T18:18:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.228 7f7e87fff700 1 -- 172.21.15.32:0/3974634551 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e8800bb60 con 0x7f7e900d8490 2021-04-09T18:18:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.228 7f7e97205700 1 -- 172.21.15.32:0/3974634551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e900d8490 msgr2=0x7f7e9010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.228 7f7e97205700 1 --2- 172.21.15.32:0/3974634551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e900d8490 0x7f7e9010da00 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f7e88006e30 tx=0x7f7e88009de0).stop 2021-04-09T18:18:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.229 7f7e97205700 1 -- 172.21.15.32:0/3974634551 shutdown_connections 2021-04-09T18:18:11.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.229 7f7e97205700 1 --2- 172.21.15.32:0/3974634551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e900d8490 0x7f7e9010da00 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.229 7f7e97205700 1 --2- 172.21.15.32:0/3974634551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e9010d010 0x7f7e900d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.229 7f7e97205700 1 -- 172.21.15.32:0/3974634551 >> 172.21.15.32:0/3974634551 conn(0x7f7e900ce5f0 msgr2=0x7f7e900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:11.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.229 7f7e97205700 1 -- 172.21.15.32:0/3974634551 shutdown_connections 2021-04-09T18:18:11.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.229 7f7e97205700 1 -- 172.21.15.32:0/3974634551 wait complete. 2021-04-09T18:18:11.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.249 7f7e97205700 1 Processor -- start 2021-04-09T18:18:11.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.249 7f7e97205700 1 -- start start 2021-04-09T18:18:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e97205700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e901a9360 0x7f7e901918f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e97205700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9018c440 0x7f7e9018c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e97205700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e901a80f0 0x7f7e901a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e97205700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e90191f10 con 0x7f7e9018c440 2021-04-09T18:18:11.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e97205700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e901a9760 con 0x7f7e901a80f0 2021-04-09T18:18:11.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e97205700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9018ce60 con 0x7f7e901a9360 2021-04-09T18:18:11.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e94820700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9018c440 0x7f7e9018c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e94820700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9018c440 0x7f7e9018c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46218/0 (socket says 172.21.15.32:46218) 2021-04-09T18:18:11.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e94820700 1 -- 172.21.15.32:0/3178481945 learned_addr learned my addr 172.21.15.32:0/3178481945 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e95822700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e901a80f0 0x7f7e901a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e95021700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e901a9360 0x7f7e901918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e94820700 1 -- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e901a9360 msgr2=0x7f7e901918f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e94820700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e901a9360 0x7f7e901918f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e94820700 1 -- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e901a80f0 msgr2=0x7f7e901a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e94820700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e901a80f0 0x7f7e901a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e94820700 1 -- 172.21.15.32:0/3178481945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e901a8b10 con 0x7f7e9018c440 2021-04-09T18:18:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e94820700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9018c440 0x7f7e9018c840 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f7e80006f00 tx=0x7f7e8000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e85ffb700 1 -- 172.21.15.32:0/3178481945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e80006020 con 0x7f7e9018c440 2021-04-09T18:18:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e85ffb700 1 -- 172.21.15.32:0/3178481945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e800063a0 con 0x7f7e9018c440 2021-04-09T18:18:11.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e97205700 1 -- 172.21.15.32:0/3178481945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e901a8b10 con 0x7f7e9018c440 2021-04-09T18:18:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e85ffb700 1 -- 172.21.15.32:0/3178481945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e80004620 con 0x7f7e9018c440 2021-04-09T18:18:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.250 7f7e97205700 1 -- 172.21.15.32:0/3178481945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e901a8b10 con 0x7f7e9018c440 2021-04-09T18:18:11.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.251 7f7e85ffb700 1 -- 172.21.15.32:0/3178481945 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7e8000a000 con 0x7f7e9018c440 2021-04-09T18:18:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.251 7f7e85ffb700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e70018930 0x7f7e7001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.251 7f7e85ffb700 1 -- 172.21.15.32:0/3178481945 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7e800329e0 con 0x7f7e9018c440 2021-04-09T18:18:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.251 7f7e95021700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e70018930 0x7f7e7001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.252 7f7e95021700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e70018930 0x7f7e7001ae10 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:11.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.271 7f7e97205700 1 -- 172.21.15.32:0/3178481945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e901a9760 con 0x7f7e9018c440 2021-04-09T18:18:11.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.274 7f7e85ffb700 1 -- 172.21.15.32:0/3178481945 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7e80006b90 con 0x7f7e9018c440 2021-04-09T18:18:11.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.467 7f7e97205700 1 -- 172.21.15.32:0/3178481945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e9018bf60 con 0x7f7e9018c440 2021-04-09T18:18:11.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.467 7f7e85ffb700 1 -- 172.21.15.32:0/3178481945 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2789 (secure 0 0 0) 0x7f7e8000a000 con 0x7f7e9018c440 2021-04-09T18:18:11.505 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 81 2021-04-09T18:18:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:11.507 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:18:08.530171","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6211,"mds_1":6306},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6306":{"gid":6306,"name":"a","rank":1,"incarnation":80,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:11.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 -- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e70018930 msgr2=0x7f7e7001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e70018930 0x7f7e7001ae10 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 -- 172.21.15.32:0/3178481945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9018c440 msgr2=0x7f7e9018c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9018c440 0x7f7e9018c840 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f7e80006f00 tx=0x7f7e8000b940).stop 2021-04-09T18:18:11.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 -- 172.21.15.32:0/3178481945 shutdown_connections 2021-04-09T18:18:11.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e70018930 0x7f7e7001ae10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e9018c440 0x7f7e9018c840 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e901a80f0 0x7f7e901a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 --2- 172.21.15.32:0/3178481945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e901a9360 0x7f7e901918f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 -- 172.21.15.32:0/3178481945 >> 172.21.15.32:0/3178481945 conn(0x7f7e901837d0 msgr2=0x7f7e90195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:11.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 -- 172.21.15.32:0/3178481945 shutdown_connections 2021-04-09T18:18:11.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.470 7f7e97205700 1 -- 172.21.15.32:0/3178481945 wait complete. 2021-04-09T18:18:11.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:18:11.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.665 7ff898d81700 1 Processor -- start 2021-04-09T18:18:11.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.665 7ff898d81700 1 -- start start 2021-04-09T18:18:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff898d81700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff8940d7ca0 0x7ff8940d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff898d81700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89410d480 con 0x7ff89410d010 2021-04-09T18:18:11.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff898d81700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8940d86c0 con 0x7ff89410da00 2021-04-09T18:18:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff898d81700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8940d9220 con 0x7ff8940d7ca0 2021-04-09T18:18:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff892b13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff89410d010 0x7ff8940d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55816/0 (socket says 172.21.15.32:55816) 2021-04-09T18:18:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff892b13700 1 -- 172.21.15.32:0/3837976130 learned_addr learned my addr 172.21.15.32:0/3837976130 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff892312700 1 --2- 172.21.15.32:0/3837976130 >> v2:172.21.15.86:3300/0 conn(0x7ff8940d7ca0 0x7ff8940d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff891b11700 1 -- 172.21.15.32:0/3837976130 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373625457 0 0) 0x7ff884000ce0 con 0x7ff89410d010 2021-04-09T18:18:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff891b11700 1 -- 172.21.15.32:0/3837976130 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff87c002fe0 con 0x7ff89410d010 2021-04-09T18:18:11.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff891b11700 1 -- 172.21.15.32:0/3837976130 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571935066 0 0) 0x7ff88c000d30 con 0x7ff89410da00 2021-04-09T18:18:11.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff891b11700 1 -- 172.21.15.32:0/3837976130 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff87c003210 con 0x7ff89410da00 2021-04-09T18:18:11.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff892312700 1 -- 172.21.15.32:0/3837976130 >> v1:172.21.15.86:6789/0 conn(0x7ff89410da00 legacy=0x7ff89410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff892312700 1 -- 172.21.15.32:0/3837976130 >> v1:172.21.15.32:6789/0 conn(0x7ff89410d010 legacy=0x7ff8940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff892312700 1 -- 172.21.15.32:0/3837976130 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8940d1f70 con 0x7ff8940d7ca0 2021-04-09T18:18:11.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff892312700 1 --2- 172.21.15.32:0/3837976130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8940d7ca0 0x7ff8940d80a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff888006650 tx=0x7ff888009a80).ready entity=mon.1 client_cookie=b603f303a281d167 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff891b11700 1 -- 172.21.15.32:0/3837976130 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8880009b0 con 0x7ff8940d7ca0 2021-04-09T18:18:11.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff891b11700 1 -- 172.21.15.32:0/3837976130 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88800c0c0 con 0x7ff8940d7ca0 2021-04-09T18:18:11.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff891b11700 1 -- 172.21.15.32:0/3837976130 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff88800c510 con 0x7ff8940d7ca0 2021-04-09T18:18:11.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.666 7ff898d81700 1 -- 172.21.15.32:0/3837976130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8940d7ca0 msgr2=0x7ff8940d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.667 7ff898d81700 1 --2- 172.21.15.32:0/3837976130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8940d7ca0 0x7ff8940d80a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff888006650 tx=0x7ff888009a80).stop 2021-04-09T18:18:11.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.667 7ff898d81700 1 -- 172.21.15.32:0/3837976130 shutdown_connections 2021-04-09T18:18:11.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.667 7ff898d81700 1 --2- 172.21.15.32:0/3837976130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8940d7ca0 0x7ff8940d80a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.667 7ff898d81700 1 -- 172.21.15.32:0/3837976130 >> 172.21.15.32:0/3837976130 conn(0x7ff8940ce5f0 msgr2=0x7ff8940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:11.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.667 7ff898d81700 1 -- 172.21.15.32:0/3837976130 shutdown_connections 2021-04-09T18:18:11.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.667 7ff898d81700 1 -- 172.21.15.32:0/3837976130 wait complete. 2021-04-09T18:18:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff898d81700 1 Processor -- start 2021-04-09T18:18:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff898d81700 1 -- start start 2021-04-09T18:18:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff898d81700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff894191ee0 0x7ff8940a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff898d81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8940a6c30 0x7ff8940b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff898d81700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8940b4330 0x7ff8941b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff898d81700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940b3790 con 0x7ff8940a6c30 2021-04-09T18:18:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff898d81700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940a7030 con 0x7ff8940b4330 2021-04-09T18:18:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff898d81700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940b4730 con 0x7ff894191ee0 2021-04-09T18:18:11.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff892312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8940a6c30 0x7ff8940b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff892312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8940a6c30 0x7ff8940b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46234/0 (socket says 172.21.15.32:46234) 2021-04-09T18:18:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff892312700 1 -- 172.21.15.32:0/3810250362 learned_addr learned my addr 172.21.15.32:0/3810250362 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff892b13700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff894191ee0 0x7ff8940a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.688 7ff893314700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8940b4330 0x7ff8941b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff892312700 1 -- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff894191ee0 msgr2=0x7ff8940a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff892312700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff894191ee0 0x7ff8940a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff892312700 1 -- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8940b4330 msgr2=0x7ff8941b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:11.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff892312700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8940b4330 0x7ff8941b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:11.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff892312700 1 -- 172.21.15.32:0/3810250362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8941b8580 con 0x7ff8940a6c30 2021-04-09T18:18:11.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff892312700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8940a6c30 0x7ff8940b3d10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7ff888009650 tx=0x7ff88800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:11.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff8837fe700 1 -- 172.21.15.32:0/3810250362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8880093a0 con 0x7ff8940a6c30 2021-04-09T18:18:11.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff8837fe700 1 -- 172.21.15.32:0/3810250362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88800bd50 con 0x7ff8940a6c30 2021-04-09T18:18:11.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff898d81700 1 -- 172.21.15.32:0/3810250362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8941b8580 con 0x7ff8940a6c30 2021-04-09T18:18:11.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff8837fe700 1 -- 172.21.15.32:0/3810250362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8880052e0 con 0x7ff8940a6c30 2021-04-09T18:18:11.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.689 7ff898d81700 1 -- 172.21.15.32:0/3810250362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8941b8580 con 0x7ff8940a6c30 2021-04-09T18:18:11.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.690 7ff8837fe700 1 -- 172.21.15.32:0/3810250362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff888006030 con 0x7ff8940a6c30 2021-04-09T18:18:11.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.690 7ff8837fe700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff878018930 0x7ff87801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:11.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.690 7ff8837fe700 1 -- 172.21.15.32:0/3810250362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff8880329e0 con 0x7ff8940a6c30 2021-04-09T18:18:11.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.690 7ff892b13700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff878018930 0x7ff87801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:11.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.690 7ff892b13700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff878018930 0x7ff87801ae10 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:11.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.710 7ff898d81700 1 -- 172.21.15.32:0/3810250362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89410d480 con 0x7ff8940a6c30 2021-04-09T18:18:11.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.713 7ff8837fe700 1 -- 172.21.15.32:0/3810250362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff8880009c0 con 0x7ff8940a6c30 2021-04-09T18:18:11.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:11.909 7ff898d81700 1 -- 172.21.15.32:0/3810250362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7ff89410d480 con 0x7ff8940a6c30 2021-04-09T18:18:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.531 7ff8837fe700 1 -- 172.21.15.32:0/3810250362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v82) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff888006030 con 0x7ff8940a6c30 2021-04-09T18:18:12.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 -- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff878018930 msgr2=0x7ff87801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:12.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff878018930 0x7ff87801ae10 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 -- 172.21.15.32:0/3810250362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8940a6c30 msgr2=0x7ff8940b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:12.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8940a6c30 0x7ff8940b3d10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7ff888009650 tx=0x7ff88800b960).stop 2021-04-09T18:18:12.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 -- 172.21.15.32:0/3810250362 shutdown_connections 2021-04-09T18:18:12.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff878018930 0x7ff87801ae10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8940a6c30 0x7ff8940b3d10 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8940b4330 0x7ff8941b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 --2- 172.21.15.32:0/3810250362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff894191ee0 0x7ff8940a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 -- 172.21.15.32:0/3810250362 >> 172.21.15.32:0/3810250362 conn(0x7ff8940bbda0 msgr2=0x7ff8941b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:12.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 -- 172.21.15.32:0/3810250362 shutdown_connections 2021-04-09T18:18:12.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.578 7ff898d81700 1 -- 172.21.15.32:0/3810250362 wait complete. 2021-04-09T18:18:12.625 INFO:tasks.cephfs.test_failover:Waiting for cluster to grow. 2021-04-09T18:18:12.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:12.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.743 7f2b1b0da700 1 Processor -- start 2021-04-09T18:18:12.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.743 7f2b1b0da700 1 -- start start 2021-04-09T18:18:12.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b1b0da700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2b1410d010 0x7f2b140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:12.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b1b0da700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b140d1da0 0x7f2b140d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:12.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b1b0da700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1410d480 con 0x7f2b140d82c0 2021-04-09T18:18:12.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b1b0da700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b140d8d30 con 0x7f2b140d1da0 2021-04-09T18:18:12.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b1b0da700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b140d9220 con 0x7f2b1410d010 2021-04-09T18:18:12.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b13fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b140d1da0 0x7f2b140d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:12.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b196f7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2b140d82c0 0x7f2b1410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55840/0 (socket says 172.21.15.32:55840) 2021-04-09T18:18:12.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b196f7700 1 -- 172.21.15.32:0/2855081019 learned_addr learned my addr 172.21.15.32:0/2855081019 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:12.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b18ef6700 1 --2- 172.21.15.32:0/2855081019 >> v2:172.21.15.86:3300/0 conn(0x7f2b1410d010 0x7f2b140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:12.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b13fff700 1 -- 172.21.15.32:0/2855081019 >> v2:172.21.15.86:3300/0 conn(0x7f2b1410d010 msgr2=0x7f2b140d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:12.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b13fff700 1 --2- 172.21.15.32:0/2855081019 >> v2:172.21.15.86:3300/0 conn(0x7f2b1410d010 0x7f2b140d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b13fff700 1 -- 172.21.15.32:0/2855081019 >> v1:172.21.15.32:6789/0 conn(0x7f2b140d82c0 legacy=0x7f2b1410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:12.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.745 7f2b13fff700 1 -- 172.21.15.32:0/2855081019 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140d86f0 con 0x7f2b140d1da0 2021-04-09T18:18:12.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.744 7f2b137fe700 1 -- 172.21.15.32:0/2855081019 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788674644 0 0) 0x7f2b0c000d30 con 0x7f2b140d82c0 2021-04-09T18:18:12.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.745 7f2b13fff700 1 --2- 172.21.15.32:0/2855081019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b140d1da0 0x7f2b140d7ca0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f2b08006650 tx=0x7f2b08009a80).ready entity=mon.0 client_cookie=2d56577f49bbf983 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:12.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.745 7f2b137fe700 1 -- 172.21.15.32:0/2855081019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b080009b0 con 0x7f2b140d1da0 2021-04-09T18:18:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.745 7f2b137fe700 1 -- 172.21.15.32:0/2855081019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0800c0c0 con 0x7f2b140d1da0 2021-04-09T18:18:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.745 7f2b137fe700 1 -- 172.21.15.32:0/2855081019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b0800c510 con 0x7f2b140d1da0 2021-04-09T18:18:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.745 7f2b1b0da700 1 -- 172.21.15.32:0/2855081019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b140d1da0 msgr2=0x7f2b140d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.745 7f2b1b0da700 1 --2- 172.21.15.32:0/2855081019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b140d1da0 0x7f2b140d7ca0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7f2b08006650 tx=0x7f2b08009a80).stop 2021-04-09T18:18:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.746 7f2b1b0da700 1 -- 172.21.15.32:0/2855081019 shutdown_connections 2021-04-09T18:18:12.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.746 7f2b1b0da700 1 --2- 172.21.15.32:0/2855081019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b140d1da0 0x7f2b140d7ca0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.746 7f2b1b0da700 1 --2- 172.21.15.32:0/2855081019 >> v2:172.21.15.86:3300/0 conn(0x7f2b1410d010 0x7f2b140d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.746 7f2b1b0da700 1 -- 172.21.15.32:0/2855081019 >> 172.21.15.32:0/2855081019 conn(0x7f2b140ce5f0 msgr2=0x7f2b140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.746 7f2b1b0da700 1 -- 172.21.15.32:0/2855081019 shutdown_connections 2021-04-09T18:18:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.746 7f2b1b0da700 1 -- 172.21.15.32:0/2855081019 wait complete. 2021-04-09T18:18:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.766 7f2b1b0da700 1 Processor -- start 2021-04-09T18:18:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.766 7f2b1b0da700 1 -- start start 2021-04-09T18:18:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.766 7f2b1b0da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b141aca20 0x7f2b141ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.766 7f2b1b0da700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b1418bd40 0x7f2b1418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.766 7f2b1b0da700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b1408fba0 0x7f2b1408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.766 7f2b1b0da700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1418c7e0 con 0x7f2b141aca20 2021-04-09T18:18:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.766 7f2b1b0da700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1418ca30 con 0x7f2b1418bd40 2021-04-09T18:18:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.766 7f2b1b0da700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140905f0 con 0x7f2b1408fba0 2021-04-09T18:18:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b18ef6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b141aca20 0x7f2b141ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b18ef6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b141aca20 0x7f2b141ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46254/0 (socket says 172.21.15.32:46254) 2021-04-09T18:18:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b18ef6700 1 -- 172.21.15.32:0/1596027335 learned_addr learned my addr 172.21.15.32:0/1596027335 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b13fff700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b1418bd40 0x7f2b1418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b196f7700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b1408fba0 0x7f2b1408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b18ef6700 1 -- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b1408fba0 msgr2=0x7f2b1408ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b18ef6700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b1408fba0 0x7f2b1408ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b18ef6700 1 -- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b1418bd40 msgr2=0x7f2b1418c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b18ef6700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b1418bd40 0x7f2b1418c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b18ef6700 1 -- 172.21.15.32:0/1596027335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140b4300 con 0x7f2b141aca20 2021-04-09T18:18:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b196f7700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b1408fba0 0x7f2b1408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b18ef6700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b141aca20 0x7f2b141ace20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f2b04006f00 tx=0x7f2b0400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b117fa700 1 -- 172.21.15.32:0/1596027335 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b0400bd30 con 0x7f2b141aca20 2021-04-09T18:18:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b117fa700 1 -- 172.21.15.32:0/1596027335 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b04002030 con 0x7f2b141aca20 2021-04-09T18:18:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b140b4300 con 0x7f2b141aca20 2021-04-09T18:18:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b117fa700 1 -- 172.21.15.32:0/1596027335 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b04002590 con 0x7f2b141aca20 2021-04-09T18:18:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.767 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b140b4300 con 0x7f2b141aca20 2021-04-09T18:18:12.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.768 7f2b117fa700 1 -- 172.21.15.32:0/1596027335 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2b04000c90 con 0x7f2b141aca20 2021-04-09T18:18:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.768 7f2b117fa700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2af4018930 0x7f2af401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.768 7f2b117fa700 1 -- 172.21.15.32:0/1596027335 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2b04032a50 con 0x7f2b141aca20 2021-04-09T18:18:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.768 7f2b13fff700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2af4018930 0x7f2af401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.769 7f2b13fff700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2af4018930 0x7f2af401ae10 crc :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:12.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.788 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b14182d70 con 0x7f2b141aca20 2021-04-09T18:18:12.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.792 7f2b117fa700 1 -- 172.21.15.32:0/1596027335 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2b040022d0 con 0x7f2b141aca20 2021-04-09T18:18:13.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.980 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b1418c760 con 0x7f2b141aca20 2021-04-09T18:18:13.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.981 7f2b117fa700 1 -- 172.21.15.32:0/1596027335 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2805 (secure 0 0 0) 0x7f2b04000c90 con 0x7f2b141aca20 2021-04-09T18:18:13.018 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 82 2021-04-09T18:18:13.019 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:13.021 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":82,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:18:12.529726","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6211,"mds_1":6306,"mds_2":6315},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6306":{"gid":6306,"name":"a","rank":1,"incarnation":80,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6315":{"gid":6315,"name":"c","rank":2,"incarnation":82,"state":"up:creating","state_seq":2,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:13.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2af4018930 msgr2=0x7f2af401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:13.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2af4018930 0x7f2af401ae10 crc :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:13.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b141aca20 msgr2=0x7f2b141ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:13.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b141aca20 0x7f2b141ace20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f2b04006f00 tx=0x7f2b0400b940).stop 2021-04-09T18:18:13.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 shutdown_connections 2021-04-09T18:18:13.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2af4018930 0x7f2af401ae10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:13.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b1408fba0 0x7f2b1408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:13.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b1418bd40 0x7f2b1418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:13.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 --2- 172.21.15.32:0/1596027335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b141aca20 0x7f2b141ace20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:13.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 >> 172.21.15.32:0/1596027335 conn(0x7f2b1419b460 msgr2=0x7f2b1419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:13.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.983 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 shutdown_connections 2021-04-09T18:18:13.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:12.984 7f2b1b0da700 1 -- 172.21.15.32:0/1596027335 wait complete. 2021-04-09T18:18:13.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:17:56.928328', 'modified': '2021-04-09 18:18:12.529726', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 6211, 'mds_1': 6306, 'mds_2': 6315}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6211': {'gid': 6211, 'name': 'b', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/1080414756', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 1080414756}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 1080414756}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6306': {'gid': 6306, 'name': 'a', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6833/2067133784', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2067133784}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2067133784}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6315': {'gid': 6315, 'name': 'c', 'rank': 2, 'incarnation': 82, 'state': 'up:creating', 'state_seq': 2, 'addr': '172.21.15.32:6835/3097053176', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3097053176}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3097053176}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:13.435 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6315:up:creating 2021-04-09T18:18:14.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.560 7fac6a3c4700 1 Processor -- start 2021-04-09T18:18:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.560 7fac6a3c4700 1 -- start start 2021-04-09T18:18:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac6a3c4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac6410d010 0x7fac640d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:14.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac6a3c4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6410d480 con 0x7fac640d1da0 2021-04-09T18:18:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac6a3c4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6410e1f0 con 0x7fac640d8490 2021-04-09T18:18:14.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac6a3c4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac640d8d30 con 0x7fac6410d010 2021-04-09T18:18:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac5bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fac640d1da0 0x7fac640d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55854/0 (socket says 172.21.15.32:55854) 2021-04-09T18:18:14.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac5bfff700 1 -- 172.21.15.32:0/4279212313 learned_addr learned my addr 172.21.15.32:0/4279212313 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:14.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac63fff700 1 --2- 172.21.15.32:0/4279212313 >> v2:172.21.15.86:3300/0 conn(0x7fac6410d010 0x7fac640d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:14.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac637fe700 1 -- 172.21.15.32:0/4279212313 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535577027 0 0) 0x7fac4c000ce0 con 0x7fac640d1da0 2021-04-09T18:18:14.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac637fe700 1 -- 172.21.15.32:0/4279212313 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac50002fe0 con 0x7fac640d1da0 2021-04-09T18:18:14.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac63fff700 1 -- 172.21.15.32:0/4279212313 >> v1:172.21.15.86:6789/0 conn(0x7fac640d8490 legacy=0x7fac6410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:14.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac63fff700 1 -- 172.21.15.32:0/4279212313 >> v1:172.21.15.32:6789/0 conn(0x7fac640d1da0 legacy=0x7fac640d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:14.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac637fe700 1 -- 172.21.15.32:0/4279212313 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876422433 0 0) 0x7fac5c000f90 con 0x7fac640d8490 2021-04-09T18:18:14.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac63fff700 1 -- 172.21.15.32:0/4279212313 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac640d9220 con 0x7fac6410d010 2021-04-09T18:18:14.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.561 7fac63fff700 1 --2- 172.21.15.32:0/4279212313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6410d010 0x7fac640d1780 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fac54006650 tx=0x7fac54009a80).ready entity=mon.1 client_cookie=60eac84ba85e33fb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:14.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.562 7fac637fe700 1 -- 172.21.15.32:0/4279212313 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac540009b0 con 0x7fac6410d010 2021-04-09T18:18:14.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.562 7fac637fe700 1 -- 172.21.15.32:0/4279212313 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5400c0c0 con 0x7fac6410d010 2021-04-09T18:18:14.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.562 7fac637fe700 1 -- 172.21.15.32:0/4279212313 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac5400c510 con 0x7fac6410d010 2021-04-09T18:18:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.562 7fac6a3c4700 1 -- 172.21.15.32:0/4279212313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6410d010 msgr2=0x7fac640d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.562 7fac6a3c4700 1 --2- 172.21.15.32:0/4279212313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6410d010 0x7fac640d1780 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fac54006650 tx=0x7fac54009a80).stop 2021-04-09T18:18:14.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.562 7fac6a3c4700 1 -- 172.21.15.32:0/4279212313 shutdown_connections 2021-04-09T18:18:14.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.563 7fac6a3c4700 1 --2- 172.21.15.32:0/4279212313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6410d010 0x7fac640d1780 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.563 7fac6a3c4700 1 -- 172.21.15.32:0/4279212313 >> 172.21.15.32:0/4279212313 conn(0x7fac640ce5f0 msgr2=0x7fac640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.563 7fac6a3c4700 1 -- 172.21.15.32:0/4279212313 shutdown_connections 2021-04-09T18:18:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.563 7fac6a3c4700 1 -- 172.21.15.32:0/4279212313 wait complete. 2021-04-09T18:18:14.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.583 7fac6a3c4700 1 Processor -- start 2021-04-09T18:18:14.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.583 7fac6a3c4700 1 -- start start 2021-04-09T18:18:14.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.583 7fac6a3c4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac640a91b0 0x7fac64197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:14.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.583 7fac6a3c4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6408e2d0 0x7fac641989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:14.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.583 7fac6a3c4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac64198ff0 0x7fac6418e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:14.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.583 7fac6a3c4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac64198450 con 0x7fac64198ff0 2021-04-09T18:18:14.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.583 7fac6a3c4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac641993f0 con 0x7fac640a91b0 2021-04-09T18:18:14.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.583 7fac6a3c4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6408de30 con 0x7fac6408e2d0 2021-04-09T18:18:14.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac689e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac64198ff0 0x7fac6418e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:14.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac689e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac64198ff0 0x7fac6418e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46274/0 (socket says 172.21.15.32:46274) 2021-04-09T18:18:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac63fff700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac640a91b0 0x7fac64197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:14.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac689e1700 1 -- 172.21.15.32:0/3184497242 learned_addr learned my addr 172.21.15.32:0/3184497242 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:14.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac5bfff700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6408e2d0 0x7fac641989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:14.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac689e1700 1 -- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6408e2d0 msgr2=0x7fac641989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:14.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac689e1700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6408e2d0 0x7fac641989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:14.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac689e1700 1 -- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac640a91b0 msgr2=0x7fac64197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:14.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac689e1700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac640a91b0 0x7fac64197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:14.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac689e1700 1 -- 172.21.15.32:0/3184497242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6418e790 con 0x7fac64198ff0 2021-04-09T18:18:14.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac63fff700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac640a91b0 0x7fac64197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:14.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac689e1700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac64198ff0 0x7fac6418e170 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7fac5c00b8c0 tx=0x7fac5c00bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:14.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac617fa700 1 -- 172.21.15.32:0/3184497242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac5c0058a0 con 0x7fac64198ff0 2021-04-09T18:18:14.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac617fa700 1 -- 172.21.15.32:0/3184497242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c003210 con 0x7fac64198ff0 2021-04-09T18:18:14.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac617fa700 1 -- 172.21.15.32:0/3184497242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac5c006030 con 0x7fac64198ff0 2021-04-09T18:18:14.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6418e790 con 0x7fac64198ff0 2021-04-09T18:18:14.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.584 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6418e790 con 0x7fac64198ff0 2021-04-09T18:18:14.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.585 7fac617fa700 1 -- 172.21.15.32:0/3184497242 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fac5c005b20 con 0x7fac64198ff0 2021-04-09T18:18:14.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.585 7fac617fa700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac480188d0 0x7fac4801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:14.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.585 7fac617fa700 1 -- 172.21.15.32:0/3184497242 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fac5c032dd0 con 0x7fac64198ff0 2021-04-09T18:18:14.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.585 7fac63fff700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac480188d0 0x7fac4801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:14.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.586 7fac63fff700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac480188d0 0x7fac4801add0 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:14.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.604 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6410d480 con 0x7fac64198ff0 2021-04-09T18:18:14.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.607 7fac617fa700 1 -- 172.21.15.32:0/3184497242 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fac5c00bda0 con 0x7fac64198ff0 2021-04-09T18:18:14.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.801 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac6410d480 con 0x7fac64198ff0 2021-04-09T18:18:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.802 7fac617fa700 1 -- 172.21.15.32:0/3184497242 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2803 (secure 0 0 0) 0x7fac5c005aa0 con 0x7fac64198ff0 2021-04-09T18:18:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 83 2021-04-09T18:18:14.844 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:14.845 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":83,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:18:13.542732","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6211,"mds_1":6306,"mds_2":6315},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6306":{"gid":6306,"name":"a","rank":1,"incarnation":80,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6315":{"gid":6315,"name":"c","rank":2,"incarnation":82,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac480188d0 msgr2=0x7fac4801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac480188d0 0x7fac4801add0 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac64198ff0 msgr2=0x7fac6418e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac64198ff0 0x7fac6418e170 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7fac5c00b8c0 tx=0x7fac5c00bc60).stop 2021-04-09T18:18:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 shutdown_connections 2021-04-09T18:18:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac480188d0 0x7fac4801add0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:14.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac64198ff0 0x7fac6418e170 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:14.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6408e2d0 0x7fac641989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:14.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 --2- 172.21.15.32:0/3184497242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac640a91b0 0x7fac64197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:14.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 >> 172.21.15.32:0/3184497242 conn(0x7fac641905b0 msgr2=0x7fac6418b220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:14.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 shutdown_connections 2021-04-09T18:18:14.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:14.804 7fac6a3c4700 1 -- 172.21.15.32:0/3184497242 wait complete. 2021-04-09T18:18:15.271 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:17:56.928328', 'modified': '2021-04-09 18:18:13.542732', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 6211, 'mds_1': 6306, 'mds_2': 6315}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6211': {'gid': 6211, 'name': 'b', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/1080414756', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 1080414756}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 1080414756}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6306': {'gid': 6306, 'name': 'a', 'rank': 1, 'incarnation': 80, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.32:6833/2067133784', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2067133784}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2067133784}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6315': {'gid': 6315, 'name': 'c', 'rank': 2, 'incarnation': 82, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.32:6835/3097053176', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3097053176}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3097053176}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:15.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2021-04-09T18:18:15.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:18:15.402 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:18:15.403 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:15.403 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:18:15.403 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 6211, 2021-04-09T18:18:15.403 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:18:15.404 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:18:15.404 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 18.437407745000002, 2021-04-09T18:18:15.404 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 83, 2021-04-09T18:18:15.404 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 74, 2021-04-09T18:18:15.405 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 74, 2021-04-09T18:18:15.405 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 24.017395033 2021-04-09T18:18:15.405 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:18:15.408 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 6211, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.437407745000002, "mdsmap_epoch": 83, "osdmap_epoch": 74, "osdmap_epoch_barrier": 74, "uptime": 24.017395033 } 2021-04-09T18:18:15.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:18:15.557 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:15.558 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:15.558 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:18:15.558 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6306, 2021-04-09T18:18:15.559 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:18:15.559 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:18:15.560 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.9889509470000002, 2021-04-09T18:18:15.560 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 83, 2021-04-09T18:18:15.560 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 74, 2021-04-09T18:18:15.561 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 74, 2021-04-09T18:18:15.561 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 24.161259597000001 2021-04-09T18:18:15.561 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:15.564 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 6306, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.9889509470000002, "mdsmap_epoch": 83, "osdmap_epoch": 74, "osdmap_epoch_barrier": 74, "uptime": 24.161259597000001 } 2021-04-09T18:18:15.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:18:15.756 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:15.757 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:15.758 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 2, 2021-04-09T18:18:15.758 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6315, 2021-04-09T18:18:15.758 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:18:15.759 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:18:15.759 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 3.1903840059999999, 2021-04-09T18:18:15.760 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 83, 2021-04-09T18:18:15.760 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 74, 2021-04-09T18:18:15.761 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 74, 2021-04-09T18:18:15.763 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 24.317513668 2021-04-09T18:18:15.763 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:15.766 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 2, "id": 6315, "want_state": "up:active", "state": "up:active", "rank_uptime": 3.1903840059999999, "mdsmap_epoch": 83, "osdmap_epoch": 74, "osdmap_epoch_barrier": 74, "uptime": 24.317513668 } 2021-04-09T18:18:15.767 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:18:15.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:18:15.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:18:15.897 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:18:15.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:15.898 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:18:15.903 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:15.953 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:18:15.954 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:18:15.954 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:18:15.955 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:18:15.955 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:16.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[47083]: fuse finished with error 0 and tester_r 0 2021-04-09T18:18:16.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:16.063 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:18:16.065 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:16.110 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:18:16.111 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:18:16.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:18:16.222 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:16.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:16.312 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:18:16.316 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:18:16.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:18:16.364 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:18:16.364 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:18:16.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:18:16.365 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:18:16.450 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:18:16.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.565 7fa3b2fdb700 1 Processor -- start 2021-04-09T18:18:16.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.565 7fa3b2fdb700 1 -- start start 2021-04-09T18:18:16.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b2fdb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa3ac10da00 0x7fa3ac10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:16.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b2fdb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac10d480 con 0x7fa3ac0d7ca0 2021-04-09T18:18:16.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b2fdb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0d1f70 con 0x7fa3ac10d010 2021-04-09T18:18:16.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b2fdb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ac0d8d30 con 0x7fa3ac10da00 2021-04-09T18:18:16.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b15f8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa3ac10da00 0x7fa3ac10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:16.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3abfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa3ac0d7ca0 0x7fa3ac0d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55870/0 (socket says 172.21.15.32:55870) 2021-04-09T18:18:16.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3abfff700 1 -- 172.21.15.32:0/384472566 learned_addr learned my addr 172.21.15.32:0/384472566 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:16.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3ab7fe700 1 -- 172.21.15.32:0/384472566 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174088950 0 0) 0x7fa39c000ce0 con 0x7fa3ac0d7ca0 2021-04-09T18:18:16.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3ab7fe700 1 -- 172.21.15.32:0/384472566 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa398002fe0 con 0x7fa3ac0d7ca0 2021-04-09T18:18:16.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3ab7fe700 1 -- 172.21.15.32:0/384472566 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684087260 0 0) 0x7fa3a4000fe0 con 0x7fa3ac10d010 2021-04-09T18:18:16.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b15f8700 1 -- 172.21.15.32:0/384472566 >> v1:172.21.15.86:6789/0 conn(0x7fa3ac10d010 legacy=0x7fa3ac0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:16.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b15f8700 1 -- 172.21.15.32:0/384472566 >> v1:172.21.15.32:6789/0 conn(0x7fa3ac0d7ca0 legacy=0x7fa3ac0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:16.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b15f8700 1 -- 172.21.15.32:0/384472566 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac0d9220 con 0x7fa3ac10da00 2021-04-09T18:18:16.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b15f8700 1 --2- 172.21.15.32:0/384472566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac10da00 0x7fa3ac10de00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7fa394006ab0 tx=0x7fa394009b40).ready entity=mon.0 client_cookie=6089188c0f3267bc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:16.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3ab7fe700 1 -- 172.21.15.32:0/384472566 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3940022e0 con 0x7fa3ac10da00 2021-04-09T18:18:16.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3ab7fe700 1 -- 172.21.15.32:0/384472566 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa394002660 con 0x7fa3ac10da00 2021-04-09T18:18:16.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3ab7fe700 1 -- 172.21.15.32:0/384472566 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa394002b40 con 0x7fa3ac10da00 2021-04-09T18:18:16.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b2fdb700 1 -- 172.21.15.32:0/384472566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac10da00 msgr2=0x7fa3ac10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:16.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.566 7fa3b2fdb700 1 --2- 172.21.15.32:0/384472566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac10da00 0x7fa3ac10de00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7fa394006ab0 tx=0x7fa394009b40).stop 2021-04-09T18:18:16.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.567 7fa3b2fdb700 1 -- 172.21.15.32:0/384472566 shutdown_connections 2021-04-09T18:18:16.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.567 7fa3b2fdb700 1 --2- 172.21.15.32:0/384472566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac10da00 0x7fa3ac10de00 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:16.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.567 7fa3b2fdb700 1 -- 172.21.15.32:0/384472566 >> 172.21.15.32:0/384472566 conn(0x7fa3ac0ce5f0 msgr2=0x7fa3ac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:16.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.567 7fa3b2fdb700 1 -- 172.21.15.32:0/384472566 shutdown_connections 2021-04-09T18:18:16.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.567 7fa3b2fdb700 1 -- 172.21.15.32:0/384472566 wait complete. 2021-04-09T18:18:16.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.587 7fa3b2fdb700 1 Processor -- start 2021-04-09T18:18:16.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.587 7fa3b2fdb700 1 -- start start 2021-04-09T18:18:16.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b2fdb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3ac179610 0x7fa3ac179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:16.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b2fdb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3ac17a6e0 0x7fa3ac17aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b2fdb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac17b100 0x7fa3ac08db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:16.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b2fdb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac17d720 con 0x7fa3ac17b100 2021-04-09T18:18:16.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b2fdb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac185ba0 con 0x7fa3ac179610 2021-04-09T18:18:16.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b2fdb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac08e1b0 con 0x7fa3ac17a6e0 2021-04-09T18:18:16.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b15f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac17b100 0x7fa3ac08db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:16.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b15f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac17b100 0x7fa3ac08db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46286/0 (socket says 172.21.15.32:46286) 2021-04-09T18:18:16.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b15f8700 1 -- 172.21.15.32:0/2855196695 learned_addr learned my addr 172.21.15.32:0/2855196695 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:16.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3b0df7700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3ac179610 0x7fa3ac179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:16.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.588 7fa3abfff700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3ac17a6e0 0x7fa3ac17aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:16.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3b15f8700 1 -- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3ac17a6e0 msgr2=0x7fa3ac17aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:16.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3b15f8700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3ac17a6e0 0x7fa3ac17aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:16.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3b15f8700 1 -- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3ac179610 msgr2=0x7fa3ac179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:16.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3b15f8700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3ac179610 0x7fa3ac179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:16.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3b15f8700 1 -- 172.21.15.32:0/2855196695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac08e3d0 con 0x7fa3ac17b100 2021-04-09T18:18:16.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3b15f8700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac17b100 0x7fa3ac08db90 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7fa39400b580 tx=0x7fa39400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:16.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3a97fa700 1 -- 172.21.15.32:0/2855196695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa394009070 con 0x7fa3ac17b100 2021-04-09T18:18:16.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3a97fa700 1 -- 172.21.15.32:0/2855196695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa394002280 con 0x7fa3ac17b100 2021-04-09T18:18:16.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3a97fa700 1 -- 172.21.15.32:0/2855196695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3940027a0 con 0x7fa3ac17b100 2021-04-09T18:18:16.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac08e3d0 con 0x7fa3ac17b100 2021-04-09T18:18:16.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.589 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac08e3d0 con 0x7fa3ac17b100 2021-04-09T18:18:16.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.590 7fa3a97fa700 1 -- 172.21.15.32:0/2855196695 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa394006a20 con 0x7fa3ac17b100 2021-04-09T18:18:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.590 7fa3a97fa700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa390018930 0x7fa39001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.590 7fa3a97fa700 1 -- 172.21.15.32:0/2855196695 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa394032a00 con 0x7fa3ac17b100 2021-04-09T18:18:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.590 7fa3b0df7700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa390018930 0x7fa39001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:16.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.590 7fa3b0df7700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa390018930 0x7fa39001ae10 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:16.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.609 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac17a1b0 con 0x7fa3ac17b100 2021-04-09T18:18:16.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.613 7fa3a97fa700 1 -- 172.21.15.32:0/2855196695 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa394000ef0 con 0x7fa3ac17b100 2021-04-09T18:18:16.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.802 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa3ac17a1b0 con 0x7fa3ac17b100 2021-04-09T18:18:16.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.803 7fa3a97fa700 1 -- 172.21.15.32:0/2855196695 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v74) v1 ==== 81+0+30363 (secure 0 0 0) 0x7fa394006a20 con 0x7fa3ac17b100 2021-04-09T18:18:16.841 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:16.841 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:16.842 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 74, 2021-04-09T18:18:16.842 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:16.843 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:18:16.843 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:17:57.929427", 2021-04-09T18:18:16.843 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:18:16.844 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:18:16.844 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:18:16.845 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:18:16.845 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:18:16.845 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:18:16.846 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:18:16.847 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:18:16.847 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:18:16.848 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:18:16.848 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:18:16.848 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:18:16.849 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:18:16.849 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:18:16.850 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:18:16.851 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 13, 2021-04-09T18:18:16.851 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:18:16.852 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:18:16.852 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:18:16.853 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:18:16.853 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:18:16.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:16.854 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:18:16.854 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:18:16.855 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:18:16.855 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:18:16.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:18:16.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:18:16.860 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:18:16.860 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:18:16.861 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:18:16.861 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:18:16.861 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:18:16.861 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:18:16.861 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:18:16.862 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:18:16.862 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:18:16.923 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:18:16.924 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:18:16.926 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:18:16.926 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:18:16.926 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:18:16.927 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:18:16.927 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:18:16.928 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:18:16.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:16.929 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:18:16.936 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:18:16.936 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:18:16.937 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:18:16.937 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:18:16.937 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:18:16.937 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:18:16.938 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:18:16.938 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:18:16.938 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:18:16.938 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:18:16.939 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:18:16.939 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:18:16.939 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:18:16.939 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:18:16.939 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:18:16.940 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:18:16.940 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:18:16.940 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:18:16.948 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:18:16.948 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:18:16.949 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:18:16.949 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:18:16.949 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:18:16.949 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:18:16.950 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:18:16.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:18:16.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:16.950 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:18:16.953 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:18:16.953 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:18:16.954 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:18:16.954 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:18:16.954 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:18:16.954 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:18:16.955 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:18:16.955 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:18:16.955 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:18:16.955 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:18:16.956 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:18:16.956 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:18:16.956 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:18:16.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:16.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:16.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:16.957 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 12, 2021-04-09T18:18:16.957 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:18:16.979 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:17:55.170353", 2021-04-09T18:18:16.979 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:18:16.980 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:18:16.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:18:16.980 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:18:16.980 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:18:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:18:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:18:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:18:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:18:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:18:16.982 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:18:16.982 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:18:16.983 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:18:16.983 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:18:17.012 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:18:17.012 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:18:17.013 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:18:17.023 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:18:17.023 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:18:17.024 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:18:17.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.024 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "73", 2021-04-09T18:18:17.025 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:18:17.036 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:18:17.041 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:18:17.041 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:18:17.042 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:18:17.042 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:18:17.042 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:18:17.060 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:18:17.061 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:18:17.061 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:18:17.061 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:18:17.062 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:18:17.062 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:18:17.062 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:18:17.063 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:18:17.063 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:18:17.064 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:18:17.081 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:18:17.082 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:18:17.082 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:18:17.083 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:18:17.083 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:18:17.083 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:18:17.084 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:18:17.084 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:18:17.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:18:17.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.103 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:18:17.103 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:18:17.104 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:18:17.104 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:18:17.104 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:18:17.105 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:18:17.105 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:18:17.106 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:18:17.122 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:18:17.126 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:18:17.126 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:18:17.126 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:18:17.127 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:18:17.136 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:18:17.137 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:18:17.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.138 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:18:17.185 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:18:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:18:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.234 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 13, 2021-04-09T18:18:17.234 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:18:17.234 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:17:55.834924", 2021-04-09T18:18:17.235 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:18:17.235 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:18:17.235 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:18:17.235 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:18:17.236 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:18:17.237 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:18:17.238 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:18:17.238 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:18:17.238 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:18:17.238 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:18:17.238 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:18:17.239 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:18:17.239 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:18:17.239 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:18:17.250 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:18:17.250 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:18:17.250 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:18:17.250 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:18:17.251 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:18:17.251 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:18:17.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.253 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "74", 2021-04-09T18:18:17.254 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:18:17.254 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:18:17.254 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:18:17.255 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:18:17.256 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:18:17.256 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:18:17.257 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:18:17.257 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:18:17.257 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:18:17.257 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:18:17.257 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:18:17.269 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:18:17.269 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:18:17.275 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:18:17.276 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:18:17.276 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:18:17.276 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:18:17.276 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:18:17.277 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:18:17.287 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:18:17.287 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:18:17.288 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:18:17.288 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:18:17.288 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:18:17.288 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:18:17.289 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:18:17.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.289 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:18:17.289 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:18:17.289 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:18:17.293 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:18:17.293 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:18:17.293 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:18:17.294 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:18:17.294 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:18:17.294 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:18:17.300 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:18:17.301 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:18:17.302 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:18:17.302 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:18:17.302 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:18:17.303 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:18:17.305 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.306 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.307 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.307 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:18:17.307 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:18:17.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.308 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:18:17.308 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:18:17.309 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:17.309 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:17.309 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:17.309 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:17.319 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:17.319 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:17.319 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:17.319 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:17.325 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:17.332 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.334 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:18:17.338 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:17.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.363 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:18:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.367 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.370 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:17.370 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.370 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.380 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:18:17.380 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:17.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.380 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.381 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:18:17.381 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:17.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.381 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.391 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:17.391 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.397 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:18:17.401 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:17.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.402 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:18:17.402 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:17.412 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.412 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.423 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:17.423 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.425 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:18:17.432 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:17.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.432 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.433 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:18:17.433 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:17.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.433 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:18:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:18:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:18:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:18:17.435 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:17.451 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:17.451 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:17.452 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.457 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:18:17.458 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:18:17.458 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:17.458 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:17.458 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:17.462 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:17.462 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:17.466 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 71, 2021-04-09T18:18:17.466 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:17.467 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:17.468 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:17.468 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.469 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.469 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:18:17.469 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:17.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.470 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.470 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:18:17.471 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:17.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:18:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:18:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:18:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:18:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.481 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:18:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:18:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:18:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:18:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:18:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:18:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:18:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:18:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:18:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:18:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:18:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:17.503 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:17.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.505 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:18:17.507 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:17.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:18:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:17.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:17.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:18:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:17.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:18:17.541 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:17.548 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.549 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:18:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:17.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.563 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:18:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:17.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.578 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.579 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:18:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:18:17.580 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:18:17.581 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:18:17.591 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.595 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:18:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:18:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:17.596 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:17.597 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:17.598 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:18:17.599 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:17.600 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:17.601 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.616 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:18:17.617 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:17.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.619 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:18:17.619 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:17.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.620 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.621 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:17.622 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.634 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:18:17.637 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:17.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.639 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:18:17.639 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:17.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.640 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.648 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:17.654 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.674 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.674 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:18:17.674 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:17.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.675 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.675 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:18:17.675 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:17.675 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.676 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.676 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:17.681 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.682 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:18:17.682 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:17.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.683 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:18:17.683 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:17.684 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.684 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.684 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:18:17.685 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:18:17.685 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:18:17.685 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:18:17.685 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:17.686 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:17.686 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:17.707 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.708 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:18:17.709 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:18:17.717 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:17.718 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:17.725 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:17.727 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:17.727 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:17.727 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:17.728 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:17.728 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:17.728 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:17.728 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:17.729 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:17.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.729 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:18:17.730 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:17.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.737 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:18:17.737 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:17.738 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.738 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.738 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:17.739 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.739 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.739 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:18:17.740 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:17.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.778 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.778 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:18:17.811 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:17.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.836 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.837 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:17.837 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:18:17.839 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:17.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.840 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.840 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:18:17.840 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:17.870 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.871 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.871 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:17.871 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.879 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.880 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:18:17.880 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:17.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:17.882 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:18:17.895 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:17.896 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:17.896 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.897 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:18:17.898 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:18:17.898 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:18:17.898 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:18:17.947 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:17.948 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:17.948 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:17.948 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:17.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.950 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:18:17.950 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:18:17.951 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:17.951 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:17.951 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:17.952 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:17.952 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:17.953 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:17.953 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:17.954 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:18:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:18:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:18:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.095 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.095 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:18:18.096 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:18.096 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.097 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.098 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:18.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.099 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:18:18.099 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:18.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.100 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.101 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:18:18.101 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:18.102 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.102 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.103 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:18:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:18:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:18:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:18:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:18:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:18:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:18:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:18:18.203 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:18.204 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:18.217 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:18.217 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:18.217 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:18.217 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:18.218 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:18.218 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:18.218 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:18.218 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:18.219 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:18.219 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:18:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.221 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:18:18.221 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:18.221 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.221 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.222 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:18.222 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.231 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.231 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:18:18.232 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:18.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.233 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:18:18.233 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:18.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.266 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.267 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:18.267 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.279 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:18:18.280 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:18.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.292 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.292 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:18:18.293 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:18.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.294 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.307 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:18.307 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.308 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.309 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:18:18.309 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:18.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.311 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:18:18.327 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:18.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.328 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.329 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:18:18.329 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:18:18.329 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:18:18.330 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:18:18.330 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:18.330 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:18.330 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:18.331 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.331 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.331 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:18:18.332 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:18:18.332 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:18.332 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:18.332 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:18.332 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:18.333 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:18.355 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:18.355 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:18.355 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:18.356 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:18.356 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:18.356 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:18.356 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.374 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.374 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:18:18.380 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:18.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.393 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.393 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:18:18.394 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:18.394 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.394 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.395 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:18.395 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.407 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:18:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:18.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.409 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:18:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.410 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:18.411 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.453 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.453 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:18:18.453 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:18.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.454 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:18:18.455 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:18.455 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.455 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.456 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:18.456 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:18.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.472 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:18.472 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:18:18.472 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:18.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.473 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.473 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:18.473 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:18:18.482 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:18.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.492 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.493 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:18:18.493 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:18:18.494 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:18:18.494 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:18:18.495 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:18.495 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:18.495 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:18.496 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:18.507 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.508 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:18:18.508 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:18:18.508 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.508 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:18:18.587 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:18.590 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:18.591 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:18.591 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:18.592 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:18.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.593 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:18:18.593 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:18.624 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:18:18.635 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:18.636 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:18:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:18.663 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:18.663 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:18.663 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:18.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.668 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:18:18.669 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:18.669 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:18.670 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:18.671 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:18.671 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:18.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.671 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:18:18.672 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:18.672 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:18.672 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:18.672 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:18:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:18.675 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:18:18.675 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:18.675 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:18:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:18:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:18:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:18:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:18:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:18:18.678 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:18:18.678 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:18:18.678 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:18:18.678 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:18:18.679 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:18:18.679 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:18:18.679 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:18:18.704 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:18:18.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:18.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:18.708 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:18:18.709 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:18:18.711 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:18:18.712 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:18:18.712 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:18:18.725 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa390018930 msgr2=0x7fa39001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:18.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa390018930 0x7fa39001ae10 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac17b100 msgr2=0x7fa3ac08db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac17b100 0x7fa3ac08db90 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7fa39400b580 tx=0x7fa39400b920).stop 2021-04-09T18:18:18.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 shutdown_connections 2021-04-09T18:18:18.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa390018930 0x7fa39001ae10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:18.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3ac17b100 0x7fa3ac08db90 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:18.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3ac17a6e0 0x7fa3ac17aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:18.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 --2- 172.21.15.32:0/2855196695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3ac179610 0x7fa3ac179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:18.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 >> 172.21.15.32:0/2855196695 conn(0x7fa3ac190550 msgr2=0x7fa3ac1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:18.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 shutdown_connections 2021-04-09T18:18:18.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:16.807 7fa3b2fdb700 1 -- 172.21.15.32:0/2855196695 wait complete. 2021-04-09T18:18:18.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:18.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.868 7ff34663b700 1 Processor -- start 2021-04-09T18:18:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.868 7ff34663b700 1 -- start start 2021-04-09T18:18:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.869 7ff34663b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff34010d010 0x7ff3400d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.869 7ff34663b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff3400d8490 0x7ff34010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:18.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.869 7ff34663b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34010d480 con 0x7ff3400d1da0 2021-04-09T18:18:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.869 7ff34663b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3400d8d30 con 0x7ff34010d010 2021-04-09T18:18:18.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.869 7ff34663b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3400d9220 con 0x7ff3400d8490 2021-04-09T18:18:18.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.869 7ff33ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff34010d010 0x7ff3400d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:18.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.869 7ff33ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff34010d010 0x7ff3400d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46292/0 (socket says 172.21.15.32:46292) 2021-04-09T18:18:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.869 7ff33ffff700 1 -- 172.21.15.32:0/2527957175 learned_addr learned my addr 172.21.15.32:0/2527957175 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:18.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.869 7ff344c58700 1 --2- 172.21.15.32:0/2527957175 >> v2:172.21.15.86:3300/0 conn(0x7ff3400d8490 0x7ff34010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:18.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff33ffff700 1 -- 172.21.15.32:0/2527957175 >> v2:172.21.15.86:3300/0 conn(0x7ff3400d8490 msgr2=0x7ff34010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:18.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff33ffff700 1 --2- 172.21.15.32:0/2527957175 >> v2:172.21.15.86:3300/0 conn(0x7ff3400d8490 0x7ff34010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:18.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff33ffff700 1 -- 172.21.15.32:0/2527957175 >> v1:172.21.15.86:6789/0 conn(0x7ff3400d1da0 legacy=0x7ff3400d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:18.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff33ffff700 1 -- 172.21.15.32:0/2527957175 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34010e020 con 0x7ff34010d010 2021-04-09T18:18:18.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff33ffff700 1 --2- 172.21.15.32:0/2527957175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff34010d010 0x7ff3400d1780 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7ff330006700 tx=0x7ff330009b80).ready entity=mon.0 client_cookie=cafc9399ecbf9a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:18.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff33f7fe700 1 -- 172.21.15.32:0/2527957175 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff33000c0c0 con 0x7ff34010d010 2021-04-09T18:18:18.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff33f7fe700 1 -- 172.21.15.32:0/2527957175 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33000c440 con 0x7ff34010d010 2021-04-09T18:18:18.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff33f7fe700 1 -- 172.21.15.32:0/2527957175 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff33000c920 con 0x7ff34010d010 2021-04-09T18:18:18.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff34663b700 1 -- 172.21.15.32:0/2527957175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff34010d010 msgr2=0x7ff3400d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:19.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.870 7ff34663b700 1 --2- 172.21.15.32:0/2527957175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff34010d010 0x7ff3400d1780 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7ff330006700 tx=0x7ff330009b80).stop 2021-04-09T18:18:19.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.871 7ff34663b700 1 -- 172.21.15.32:0/2527957175 shutdown_connections 2021-04-09T18:18:19.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.871 7ff34663b700 1 --2- 172.21.15.32:0/2527957175 >> v2:172.21.15.86:3300/0 conn(0x7ff3400d8490 0x7ff34010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.871 7ff34663b700 1 --2- 172.21.15.32:0/2527957175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff34010d010 0x7ff3400d1780 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.871 7ff34663b700 1 -- 172.21.15.32:0/2527957175 >> 172.21.15.32:0/2527957175 conn(0x7ff3400ce5f0 msgr2=0x7ff3400d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.871 7ff34663b700 1 -- 172.21.15.32:0/2527957175 shutdown_connections 2021-04-09T18:18:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.871 7ff34663b700 1 -- 172.21.15.32:0/2527957175 wait complete. 2021-04-09T18:18:19.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.891 7ff34663b700 1 Processor -- start 2021-04-09T18:18:19.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.891 7ff34663b700 1 -- start start 2021-04-09T18:18:19.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff34663b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3400b42d0 0x7ff340183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff34663b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3401ac090 0x7ff3401ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff34663b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff340184d30 0x7ff340185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff34663b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3400a1a90 con 0x7ff340184d30 2021-04-09T18:18:19.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff34663b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3401acab0 con 0x7ff3400b42d0 2021-04-09T18:18:19.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff34663b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff340185a00 con 0x7ff3401ac090 2021-04-09T18:18:19.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff344c58700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff340184d30 0x7ff340185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff344c58700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff340184d30 0x7ff340185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46302/0 (socket says 172.21.15.32:46302) 2021-04-09T18:18:19.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff344c58700 1 -- 172.21.15.32:0/3053472089 learned_addr learned my addr 172.21.15.32:0/3053472089 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:19.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff33ffff700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3400b42d0 0x7ff340183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff337fff700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3401ac090 0x7ff3401ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff344c58700 1 -- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3401ac090 msgr2=0x7ff3401ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:19.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff344c58700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3401ac090 0x7ff3401ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff344c58700 1 -- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3400b42d0 msgr2=0x7ff340183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff344c58700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3400b42d0 0x7ff340183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff344c58700 1 -- 172.21.15.32:0/3053472089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff340185750 con 0x7ff340184d30 2021-04-09T18:18:19.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff344c58700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff340184d30 0x7ff340185130 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7ff33800b910 tx=0x7ff33800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:19.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff33d7fa700 1 -- 172.21.15.32:0/3053472089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff338006aa0 con 0x7ff340184d30 2021-04-09T18:18:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff33d7fa700 1 -- 172.21.15.32:0/3053472089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3380011f0 con 0x7ff340184d30 2021-04-09T18:18:19.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff34663b700 1 -- 172.21.15.32:0/3053472089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff340185750 con 0x7ff340184d30 2021-04-09T18:18:19.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff33d7fa700 1 -- 172.21.15.32:0/3053472089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff338001640 con 0x7ff340184d30 2021-04-09T18:18:19.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.892 7ff34663b700 1 -- 172.21.15.32:0/3053472089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff340185750 con 0x7ff340184d30 2021-04-09T18:18:19.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.893 7ff33d7fa700 1 -- 172.21.15.32:0/3053472089 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff3380013d0 con 0x7ff340184d30 2021-04-09T18:18:19.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.893 7ff33d7fa700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3240188d0 0x7ff32401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.893 7ff33d7fa700 1 -- 172.21.15.32:0/3053472089 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff338032dd0 con 0x7ff340184d30 2021-04-09T18:18:19.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.893 7ff33ffff700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3240188d0 0x7ff32401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.894 7ff33ffff700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3240188d0 0x7ff32401add0 crc :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:19.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.913 7ff34663b700 1 -- 172.21.15.32:0/3053472089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff340182d10 con 0x7ff340184d30 2021-04-09T18:18:19.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:18.916 7ff33d7fa700 1 -- 172.21.15.32:0/3053472089 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff33800bdf0 con 0x7ff340184d30 2021-04-09T18:18:19.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.109 7ff34663b700 1 -- 172.21.15.32:0/3053472089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff34010d480 con 0x7ff340184d30 2021-04-09T18:18:19.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.110 7ff33d7fa700 1 -- 172.21.15.32:0/3053472089 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2803 (secure 0 0 0) 0x7ff3380011f0 con 0x7ff340184d30 2021-04-09T18:18:19.160 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 83 2021-04-09T18:18:19.161 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:19.161 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":83,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:18:13.542732","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6211,"mds_1":6306,"mds_2":6315},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6211":{"gid":6211,"name":"b","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/1080414756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1080414756},{"type":"v1","addr":"172.21.15.86:6835","nonce":1080414756}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6306":{"gid":6306,"name":"a","rank":1,"incarnation":80,"state":"up:active","state_seq":6,"addr":"172.21.15.32:6833/2067133784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2067133784},{"type":"v1","addr":"172.21.15.32:6833","nonce":2067133784}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6315":{"gid":6315,"name":"c","rank":2,"incarnation":82,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/3097053176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3097053176},{"type":"v1","addr":"172.21.15.32:6835","nonce":3097053176}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 -- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3240188d0 msgr2=0x7ff32401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3240188d0 0x7ff32401add0 crc :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 -- 172.21.15.32:0/3053472089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff340184d30 msgr2=0x7ff340185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff340184d30 0x7ff340185130 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7ff33800b910 tx=0x7ff33800bcb0).stop 2021-04-09T18:18:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 -- 172.21.15.32:0/3053472089 shutdown_connections 2021-04-09T18:18:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3240188d0 0x7ff32401add0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff340184d30 0x7ff340185130 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3401ac090 0x7ff3401ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 --2- 172.21.15.32:0/3053472089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3400b42d0 0x7ff340183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 -- 172.21.15.32:0/3053472089 >> 172.21.15.32:0/3053472089 conn(0x7ff34019b890 msgr2=0x7ff3401a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:19.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 -- 172.21.15.32:0/3053472089 shutdown_connections 2021-04-09T18:18:19.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.112 7ff34663b700 1 -- 172.21.15.32:0/3053472089 wait complete. 2021-04-09T18:18:19.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:18:19.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.368 7fa716ae1700 1 Processor -- start 2021-04-09T18:18:19.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.368 7fa716ae1700 1 -- start start 2021-04-09T18:18:19.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa716ae1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa7100cffb0 0x7fa7100d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa716ae1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7100d7810 0x7fa7100d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa716ae1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7100d8230 0x7fa7100d5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa716ae1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7100d91f0 con 0x7fa7100d7810 2021-04-09T18:18:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa716ae1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7100d7290 con 0x7fa7100cffb0 2021-04-09T18:18:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa716ae1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100d62a0 con 0x7fa7100d8230 2021-04-09T18:18:19.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7100d7810 0x7fa7100d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7100d7810 0x7fa7100d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46306/0 (socket says 172.21.15.32:46306) 2021-04-09T18:18:19.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70ffff700 1 -- 172.21.15.32:0/263934510 learned_addr learned my addr 172.21.15.32:0/263934510 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:19.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa7148fd700 1 --2- 172.21.15.32:0/263934510 >> v2:172.21.15.86:3300/0 conn(0x7fa7100cffb0 0x7fa7100d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa7150fe700 1 --2- 172.21.15.32:0/263934510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7100d8230 0x7fa7100d5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70ffff700 1 -- 172.21.15.32:0/263934510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7100d8230 msgr2=0x7fa7100d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70ffff700 1 --2- 172.21.15.32:0/263934510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7100d8230 0x7fa7100d5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70ffff700 1 -- 172.21.15.32:0/263934510 >> v2:172.21.15.86:3300/0 conn(0x7fa7100cffb0 msgr2=0x7fa7100d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70ffff700 1 --2- 172.21.15.32:0/263934510 >> v2:172.21.15.86:3300/0 conn(0x7fa7100cffb0 0x7fa7100d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70ffff700 1 -- 172.21.15.32:0/263934510 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7100d6490 con 0x7fa7100d7810 2021-04-09T18:18:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70ffff700 1 --2- 172.21.15.32:0/263934510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7100d7810 0x7fa7100d7c10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7fa700006700 tx=0x7fa700009b00).ready entity=mon.0 client_cookie=6a30583bef21ea83 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.369 7fa70f7fe700 1 -- 172.21.15.32:0/263934510 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa70000c0c0 con 0x7fa7100d7810 2021-04-09T18:18:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa70f7fe700 1 -- 172.21.15.32:0/263934510 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70000c400 con 0x7fa7100d7810 2021-04-09T18:18:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa70f7fe700 1 -- 172.21.15.32:0/263934510 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa70000c8e0 con 0x7fa7100d7810 2021-04-09T18:18:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa716ae1700 1 -- 172.21.15.32:0/263934510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7100d7810 msgr2=0x7fa7100d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa716ae1700 1 --2- 172.21.15.32:0/263934510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7100d7810 0x7fa7100d7c10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7fa700006700 tx=0x7fa700009b00).stop 2021-04-09T18:18:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa716ae1700 1 -- 172.21.15.32:0/263934510 shutdown_connections 2021-04-09T18:18:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa716ae1700 1 --2- 172.21.15.32:0/263934510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7100d8230 0x7fa7100d5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa716ae1700 1 --2- 172.21.15.32:0/263934510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7100d7810 0x7fa7100d7c10 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa716ae1700 1 --2- 172.21.15.32:0/263934510 >> v2:172.21.15.86:3300/0 conn(0x7fa7100cffb0 0x7fa7100d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa716ae1700 1 -- 172.21.15.32:0/263934510 >> 172.21.15.32:0/263934510 conn(0x7fa7100ce5d0 msgr2=0x7fa7100cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa716ae1700 1 -- 172.21.15.32:0/263934510 shutdown_connections 2021-04-09T18:18:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.370 7fa716ae1700 1 -- 172.21.15.32:0/263934510 wait complete. 2021-04-09T18:18:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.390 7fa716ae1700 1 Processor -- start 2021-04-09T18:18:19.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa716ae1700 1 -- start start 2021-04-09T18:18:19.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa716ae1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa71008d210 0x7fa71008fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa716ae1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7100903f0 0x7fa7101a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa716ae1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7101a6080 0x7fa7101a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa716ae1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71008d690 con 0x7fa7101a6080 2021-04-09T18:18:19.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa716ae1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100b0fd0 con 0x7fa7100903f0 2021-04-09T18:18:19.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa716ae1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7101a6480 con 0x7fa71008d210 2021-04-09T18:18:19.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa7150fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7101a6080 0x7fa7101a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa7150fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7101a6080 0x7fa7101a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46314/0 (socket says 172.21.15.32:46314) 2021-04-09T18:18:19.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.391 7fa7150fe700 1 -- 172.21.15.32:0/1479426419 learned_addr learned my addr 172.21.15.32:0/1479426419 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:19.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa70ffff700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7100903f0 0x7fa7101a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa7148fd700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa71008d210 0x7fa71008fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa7150fe700 1 -- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa71008d210 msgr2=0x7fa71008fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:19.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa7150fe700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa71008d210 0x7fa71008fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa7150fe700 1 -- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7100903f0 msgr2=0x7fa7101a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:19.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa7150fe700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7100903f0 0x7fa7101a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:19.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa7150fe700 1 -- 172.21.15.32:0/1479426419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71008d8b0 con 0x7fa7101a6080 2021-04-09T18:18:19.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa7148fd700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa71008d210 0x7fa71008fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:19.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa7150fe700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7101a6080 0x7fa7101a46b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7fa70800b970 tx=0x7fa70800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:19.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa70d7fa700 1 -- 172.21.15.32:0/1479426419 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa708000f90 con 0x7fa7101a6080 2021-04-09T18:18:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa70d7fa700 1 -- 172.21.15.32:0/1479426419 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa708003030 con 0x7fa7101a6080 2021-04-09T18:18:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa71008d8b0 con 0x7fa7101a6080 2021-04-09T18:18:19.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa70d7fa700 1 -- 172.21.15.32:0/1479426419 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7080013e0 con 0x7fa7101a6080 2021-04-09T18:18:19.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.392 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71008d8b0 con 0x7fa7101a6080 2021-04-09T18:18:19.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.393 7fa70d7fa700 1 -- 172.21.15.32:0/1479426419 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa7080011f0 con 0x7fa7101a6080 2021-04-09T18:18:19.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.393 7fa70d7fa700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:19.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.393 7fa70d7fa700 1 -- 172.21.15.32:0/1479426419 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa708032d70 con 0x7fa7101a6080 2021-04-09T18:18:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.393 7fa7148fd700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.393 7fa7148fd700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.413 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7101a52b0 con 0x7fa7101a6080 2021-04-09T18:18:19.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.416 7fa70d7fa700 1 -- 172.21.15.32:0/1479426419 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa7080066f0 con 0x7fa7101a6080 2021-04-09T18:18:19.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:19.598 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa7101a52b0 con 0x7fa7101a6080 2021-04-09T18:18:20.595 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:18:20.559 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:18:20.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.564 7fa70d7fa700 1 -- 172.21.15.32:0/1479426419 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa7080011f0 con 0x7fa7101a6080 2021-04-09T18:18:20.650 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:18:20.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.616 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 msgr2=0x7fa6f401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:20.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.616 7fa716ae1700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.616 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7101a6080 msgr2=0x7fa7101a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:20.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.616 7fa716ae1700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7101a6080 0x7fa7101a46b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7fa70800b970 tx=0x7fa70800bd10).stop 2021-04-09T18:18:20.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.617 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 shutdown_connections 2021-04-09T18:18:20.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.617 7fa716ae1700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.617 7fa716ae1700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7101a6080 0x7fa7101a46b0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.617 7fa716ae1700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7100903f0 0x7fa7101a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.617 7fa716ae1700 1 --2- 172.21.15.32:0/1479426419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa71008d210 0x7fa71008fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.617 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 >> 172.21.15.32:0/1479426419 conn(0x7fa7101a8f50 msgr2=0x7fa710188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:20.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.617 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 shutdown_connections 2021-04-09T18:18:20.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.617 7fa716ae1700 1 -- 172.21.15.32:0/1479426419 wait complete. 2021-04-09T18:18:20.658 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:18:20.671 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:18:20.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:20.687 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:18:20.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.802 7f555a31b700 1 Processor -- start 2021-04-09T18:18:20.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.802 7f555a31b700 1 -- start start 2021-04-09T18:18:20.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f555a31b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f555410d010 0x7f55540d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:20.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f555a31b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55540d8490 0x7f555410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:20.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f555a31b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555410d480 con 0x7f55540d1da0 2021-04-09T18:18:20.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f555a31b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555410e780 con 0x7f55540d8490 2021-04-09T18:18:20.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f555a31b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55540d8d30 con 0x7f555410d010 2021-04-09T18:18:20.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f554bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f55540d1da0 0x7f55540d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55936/0 (socket says 172.21.15.32:55936) 2021-04-09T18:18:20.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f554bfff700 1 -- 172.21.15.32:0/3320660594 learned_addr learned my addr 172.21.15.32:0/3320660594 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:20.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f5558938700 1 --2- 172.21.15.32:0/3320660594 >> v2:172.21.15.86:3300/0 conn(0x7f55540d8490 0x7f555410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:20.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f5553fff700 1 --2- 172.21.15.32:0/3320660594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f555410d010 0x7f55540d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:20.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f55537fe700 1 -- 172.21.15.32:0/3320660594 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996134551 0 0) 0x7f553c000ce0 con 0x7f55540d1da0 2021-04-09T18:18:20.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f55537fe700 1 -- 172.21.15.32:0/3320660594 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5540002fe0 con 0x7f55540d1da0 2021-04-09T18:18:20.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f5558938700 1 -- 172.21.15.32:0/3320660594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f555410d010 msgr2=0x7f55540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:20.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f5558938700 1 --2- 172.21.15.32:0/3320660594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f555410d010 0x7f55540d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f5558938700 1 -- 172.21.15.32:0/3320660594 >> v1:172.21.15.32:6789/0 conn(0x7f55540d1da0 legacy=0x7f55540d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f5558938700 1 -- 172.21.15.32:0/3320660594 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55540d9220 con 0x7f55540d8490 2021-04-09T18:18:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.803 7f5553fff700 1 --2- 172.21.15.32:0/3320660594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f555410d010 0x7f55540d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.804 7f5558938700 1 --2- 172.21.15.32:0/3320660594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55540d8490 0x7f555410da00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f554c006e50 tx=0x7f554c009e70).ready entity=mon.1 client_cookie=a73a245d176cd29b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.804 7f55537fe700 1 -- 172.21.15.32:0/3320660594 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f554c006a20 con 0x7f55540d8490 2021-04-09T18:18:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.804 7f55537fe700 1 -- 172.21.15.32:0/3320660594 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c006790 con 0x7f55540d8490 2021-04-09T18:18:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.804 7f55537fe700 1 -- 172.21.15.32:0/3320660594 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f554c00b700 con 0x7f55540d8490 2021-04-09T18:18:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.804 7f555a31b700 1 -- 172.21.15.32:0/3320660594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55540d8490 msgr2=0x7f555410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.804 7f555a31b700 1 --2- 172.21.15.32:0/3320660594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55540d8490 0x7f555410da00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f554c006e50 tx=0x7f554c009e70).stop 2021-04-09T18:18:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.805 7f555a31b700 1 -- 172.21.15.32:0/3320660594 shutdown_connections 2021-04-09T18:18:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.805 7f555a31b700 1 --2- 172.21.15.32:0/3320660594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55540d8490 0x7f555410da00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.805 7f555a31b700 1 --2- 172.21.15.32:0/3320660594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f555410d010 0x7f55540d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.805 7f555a31b700 1 -- 172.21.15.32:0/3320660594 >> 172.21.15.32:0/3320660594 conn(0x7f55540ce5f0 msgr2=0x7f55540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:20.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.805 7f555a31b700 1 -- 172.21.15.32:0/3320660594 shutdown_connections 2021-04-09T18:18:20.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.805 7f555a31b700 1 -- 172.21.15.32:0/3320660594 wait complete. 2021-04-09T18:18:20.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f555a31b700 1 Processor -- start 2021-04-09T18:18:20.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f555a31b700 1 -- start start 2021-04-09T18:18:20.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f555a31b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55541a5210 0x7f5554185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:20.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f555a31b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554185a70 0x7f55540acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:20.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f555a31b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5554185e70 0x7f55540ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:20.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f555a31b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555418f7b0 con 0x7f5554185a70 2021-04-09T18:18:20.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f555a31b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55540ad7f0 con 0x7f5554185e70 2021-04-09T18:18:20.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f555a31b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55540ada10 con 0x7f55541a5210 2021-04-09T18:18:20.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f554bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554185a70 0x7f55540acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:20.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f554bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554185a70 0x7f55540acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46348/0 (socket says 172.21.15.32:46348) 2021-04-09T18:18:20.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.825 7f554bfff700 1 -- 172.21.15.32:0/1590175490 learned_addr learned my addr 172.21.15.32:0/1590175490 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:20.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f5558938700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5554185e70 0x7f55540ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:20.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f5553fff700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55541a5210 0x7f5554185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:20.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f554bfff700 1 -- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55541a5210 msgr2=0x7f5554185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:20.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f554bfff700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55541a5210 0x7f5554185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f554bfff700 1 -- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5554185e70 msgr2=0x7f55540ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f554bfff700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5554185e70 0x7f55540ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f554bfff700 1 -- 172.21.15.32:0/1590175490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5554179610 con 0x7f5554185a70 2021-04-09T18:18:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f554bfff700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554185a70 0x7f55540acbb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f553c006f00 tx=0x7f553c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:20.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f55517fa700 1 -- 172.21.15.32:0/1590175490 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f553c00bd30 con 0x7f5554185a70 2021-04-09T18:18:20.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f55517fa700 1 -- 172.21.15.32:0/1590175490 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c006030 con 0x7f5554185a70 2021-04-09T18:18:20.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f55517fa700 1 -- 172.21.15.32:0/1590175490 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f553c004590 con 0x7f5554185a70 2021-04-09T18:18:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f555a31b700 1 -- 172.21.15.32:0/1590175490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5554179610 con 0x7f5554185a70 2021-04-09T18:18:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.826 7f555a31b700 1 -- 172.21.15.32:0/1590175490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5554179610 con 0x7f5554185a70 2021-04-09T18:18:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.827 7f55517fa700 1 -- 172.21.15.32:0/1590175490 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f553c00a000 con 0x7f5554185a70 2021-04-09T18:18:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.827 7f55517fa700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55380188c0 0x7f553801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:20.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.827 7f55517fa700 1 -- 172.21.15.32:0/1590175490 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f553c0328c0 con 0x7f5554185a70 2021-04-09T18:18:20.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.827 7f5553fff700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55380188c0 0x7f553801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:20.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.827 7f5553fff700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55380188c0 0x7f553801adc0 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:20.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.847 7f555a31b700 1 -- 172.21.15.32:0/1590175490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555418f7b0 con 0x7f5554185a70 2021-04-09T18:18:20.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:20.850 7f55517fa700 1 -- 172.21.15.32:0/1590175490 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f553c0009b0 con 0x7f5554185a70 2021-04-09T18:18:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.043 7f555a31b700 1 -- 172.21.15.32:0/1590175490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f555418f7b0 con 0x7f5554185a70 2021-04-09T18:18:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.043 7f55517fa700 1 -- 172.21.15.32:0/1590175490 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+1764 (secure 0 0 0) 0x7f553c00bd30 con 0x7f5554185a70 2021-04-09T18:18:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 84 2021-04-09T18:18:21.081 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:21.081 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":84,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[{"mdsmap":{"epoch":84,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:17:56.928328","modified":"2021-04-09 18:18:20.560014","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{},"failed":[0,1,2],"damaged":[],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-04-09T18:18:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 -- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55380188c0 msgr2=0x7f553801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55380188c0 0x7f553801adc0 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 -- 172.21.15.32:0/1590175490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554185a70 msgr2=0x7f55540acbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554185a70 0x7f55540acbb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f553c006f00 tx=0x7f553c00b940).stop 2021-04-09T18:18:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 -- 172.21.15.32:0/1590175490 shutdown_connections 2021-04-09T18:18:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f55380188c0 0x7f553801adc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5554185a70 0x7f55540acbb0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5554185e70 0x7f55540ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 --2- 172.21.15.32:0/1590175490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55541a5210 0x7f5554185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 -- 172.21.15.32:0/1590175490 >> 172.21.15.32:0/1590175490 conn(0x7f55541a99c0 msgr2=0x7f555417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 -- 172.21.15.32:0/1590175490 shutdown_connections 2021-04-09T18:18:21.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.046 7f555a31b700 1 -- 172.21.15.32:0/1590175490 wait complete. 2021-04-09T18:18:21.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:18:21.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.220 7f77a48bb700 1 Processor -- start 2021-04-09T18:18:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.220 7f77a48bb700 1 -- start start 2021-04-09T18:18:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.220 7f77a48bb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f77a00cffb0 0x7f77a00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.220 7f77a48bb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a00d7290 con 0x7f77a00d7810 2021-04-09T18:18:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f77a48bb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a00d8400 con 0x7f77a00d6c90 2021-04-09T18:18:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f77a48bb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a00d7090 con 0x7f77a00cffb0 2021-04-09T18:18:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779de1b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f77a00d7810 0x7f77a00d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55950/0 (socket says 172.21.15.32:55950) 2021-04-09T18:18:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779de1b700 1 -- 172.21.15.32:0/3126155342 learned_addr learned my addr 172.21.15.32:0/3126155342 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779e61c700 1 --2- 172.21.15.32:0/3126155342 >> v2:172.21.15.86:3300/0 conn(0x7f77a00cffb0 0x7f77a00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779d61a700 1 -- 172.21.15.32:0/3126155342 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653677100 0 0) 0x7f77880009b0 con 0x7f77a00d7810 2021-04-09T18:18:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779d61a700 1 -- 172.21.15.32:0/3126155342 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f778c002fe0 con 0x7f77a00d7810 2021-04-09T18:18:21.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779d61a700 1 -- 172.21.15.32:0/3126155342 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1909275498 0 0) 0x7f7788001010 con 0x7f77a00d7810 2021-04-09T18:18:21.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779e61c700 1 -- 172.21.15.32:0/3126155342 >> v1:172.21.15.86:6789/0 conn(0x7f77a00d6c90 legacy=0x7f77a010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779e61c700 1 -- 172.21.15.32:0/3126155342 >> v1:172.21.15.32:6789/0 conn(0x7f77a00d7810 legacy=0x7f77a00d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779e61c700 1 -- 172.21.15.32:0/3126155342 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a00d91f0 con 0x7f77a00cffb0 2021-04-09T18:18:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.221 7f779e61c700 1 --2- 172.21.15.32:0/3126155342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a00cffb0 0x7f77a00d2450 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f7790006650 tx=0x7f7790009a00).ready entity=mon.1 client_cookie=6d8cfa7ecef473d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.222 7f779d61a700 1 -- 172.21.15.32:0/3126155342 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7790009df0 con 0x7f77a00cffb0 2021-04-09T18:18:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.222 7f779d61a700 1 -- 172.21.15.32:0/3126155342 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77900009b0 con 0x7f77a00cffb0 2021-04-09T18:18:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.222 7f779d61a700 1 -- 172.21.15.32:0/3126155342 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f779000c2b0 con 0x7f77a00cffb0 2021-04-09T18:18:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.222 7f77a48bb700 1 -- 172.21.15.32:0/3126155342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a00cffb0 msgr2=0x7f77a00d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.222 7f77a48bb700 1 --2- 172.21.15.32:0/3126155342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a00cffb0 0x7f77a00d2450 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f7790006650 tx=0x7f7790009a00).stop 2021-04-09T18:18:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.222 7f77a48bb700 1 -- 172.21.15.32:0/3126155342 shutdown_connections 2021-04-09T18:18:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.223 7f77a48bb700 1 --2- 172.21.15.32:0/3126155342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a00cffb0 0x7f77a00d2450 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.223 7f77a48bb700 1 -- 172.21.15.32:0/3126155342 >> 172.21.15.32:0/3126155342 conn(0x7f77a00ce5d0 msgr2=0x7f77a00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.223 7f77a48bb700 1 -- 172.21.15.32:0/3126155342 shutdown_connections 2021-04-09T18:18:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.223 7f77a48bb700 1 -- 172.21.15.32:0/3126155342 wait complete. 2021-04-09T18:18:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f77a48bb700 1 Processor -- start 2021-04-09T18:18:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f77a48bb700 1 -- start start 2021-04-09T18:18:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f77a48bb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a018e880 0x7f77a01812c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f77a48bb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0180340 0x7f77a01b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f77a48bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77a01b8c90 0x7f77a00adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f77a48bb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a0180d40 con 0x7f77a01b8c90 2021-04-09T18:18:21.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f77a48bb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a01894f0 con 0x7f77a0180340 2021-04-09T18:18:21.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f77a48bb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a018ec80 con 0x7f77a018e880 2021-04-09T18:18:21.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f779ee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77a01b8c90 0x7f77a00adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f779ee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77a01b8c90 0x7f77a00adf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46362/0 (socket says 172.21.15.32:46362) 2021-04-09T18:18:21.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.243 7f779ee1d700 1 -- 172.21.15.32:0/3082207406 learned_addr learned my addr 172.21.15.32:0/3082207406 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:21.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f779e61c700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a018e880 0x7f77a01812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f779de1b700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0180340 0x7f77a01b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f779ee1d700 1 -- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a018e880 msgr2=0x7f77a01812c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f779ee1d700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a018e880 0x7f77a01812c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f779ee1d700 1 -- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0180340 msgr2=0x7f77a01b8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f779ee1d700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0180340 0x7f77a01b8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f779ee1d700 1 -- 172.21.15.32:0/3082207406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a01818e0 con 0x7f77a01b8c90 2021-04-09T18:18:21.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f779e61c700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a018e880 0x7f77a01812c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f779ee1d700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77a01b8c90 0x7f77a00adf80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f7798009980 tx=0x7f779800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:21.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f7796ffd700 1 -- 172.21.15.32:0/3082207406 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7798003220 con 0x7f77a01b8c90 2021-04-09T18:18:21.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f7796ffd700 1 -- 172.21.15.32:0/3082207406 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77980011f0 con 0x7f77a01b8c90 2021-04-09T18:18:21.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a01818e0 con 0x7f77a01b8c90 2021-04-09T18:18:21.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f7796ffd700 1 -- 172.21.15.32:0/3082207406 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7798001680 con 0x7f77a01b8c90 2021-04-09T18:18:21.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.244 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a01818e0 con 0x7f77a01b8c90 2021-04-09T18:18:21.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.245 7f7796ffd700 1 -- 172.21.15.32:0/3082207406 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f77980013d0 con 0x7f77a01b8c90 2021-04-09T18:18:21.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.245 7f7796ffd700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7784018930 0x7f778401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.245 7f7796ffd700 1 -- 172.21.15.32:0/3082207406 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f7798032e30 con 0x7f77a01b8c90 2021-04-09T18:18:21.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.245 7f779e61c700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7784018930 0x7f778401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.245 7f779e61c700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7784018930 0x7f778401ae10 crc :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:21.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.265 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a018ec80 con 0x7f77a01b8c90 2021-04-09T18:18:21.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.268 7f7796ffd700 1 -- 172.21.15.32:0/3082207406 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f779800be10 con 0x7f77a01b8c90 2021-04-09T18:18:21.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.463 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f77a018ec80 con 0x7f77a01b8c90 2021-04-09T18:18:21.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.602 7f7796ffd700 1 -- 172.21.15.32:0/3082207406 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7f779801fa70 con 0x7f77a01b8c90 2021-04-09T18:18:21.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.629 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7784018930 msgr2=0x7f778401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.629 7f77a48bb700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7784018930 0x7f778401ae10 crc :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.629 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77a01b8c90 msgr2=0x7f77a00adf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.629 7f77a48bb700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77a01b8c90 0x7f77a00adf80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f7798009980 tx=0x7f779800bcd0).stop 2021-04-09T18:18:21.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.630 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 shutdown_connections 2021-04-09T18:18:21.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.630 7f77a48bb700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7784018930 0x7f778401ae10 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.630 7f77a48bb700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77a01b8c90 0x7f77a00adf80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.630 7f77a48bb700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0180340 0x7f77a01b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.630 7f77a48bb700 1 --2- 172.21.15.32:0/3082207406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a018e880 0x7f77a01812c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.630 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 >> 172.21.15.32:0/3082207406 conn(0x7f77a0189800 msgr2=0x7f77a01b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:21.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.630 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 shutdown_connections 2021-04-09T18:18:21.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.630 7f77a48bb700 1 -- 172.21.15.32:0/3082207406 wait complete. 2021-04-09T18:18:21.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:18:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.830 7f80b1877700 1 Processor -- start 2021-04-09T18:18:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.830 7f80b1877700 1 -- start start 2021-04-09T18:18:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.831 7f80b1877700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80ac10da80 0x7f80ac10de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.831 7f80b1877700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ac10d500 con 0x7f80ac0d7cc0 2021-04-09T18:18:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.831 7f80b1877700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ac0d0330 con 0x7f80ac10d050 2021-04-09T18:18:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.831 7f80b1877700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ac0d8d70 con 0x7f80ac10da80 2021-04-09T18:18:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.831 7f80abd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80ac10da80 0x7f80ac10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.831 7f80abd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80ac10da80 0x7f80ac10de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46370/0 (socket says 172.21.15.32:46370) 2021-04-09T18:18:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.831 7f80aad73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f80ac0d7cc0 0x7f80ac0d80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55960/0 (socket says 172.21.15.32:55960) 2021-04-09T18:18:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.831 7f80abd75700 1 -- 172.21.15.32:0/992262692 learned_addr learned my addr 172.21.15.32:0/992262692 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80abd75700 1 -- 172.21.15.32:0/992262692 >> v1:172.21.15.86:6789/0 conn(0x7f80ac10d050 legacy=0x7f80ac0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80abd75700 1 -- 172.21.15.32:0/992262692 >> v1:172.21.15.32:6789/0 conn(0x7f80ac0d7cc0 legacy=0x7f80ac0d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80abd75700 1 -- 172.21.15.32:0/992262692 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac0d9260 con 0x7f80ac10da80 2021-04-09T18:18:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80aa572700 1 -- 172.21.15.32:0/992262692 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643292697 0 0) 0x7f80a0000ce0 con 0x7f80ac0d7cc0 2021-04-09T18:18:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80abd75700 1 --2- 172.21.15.32:0/992262692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac10da80 0x7f80ac10de80 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f80a4006f00 tx=0x7f80a4009e90).ready entity=mon.0 client_cookie=c883e7b9f5b887ef server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80aa572700 1 -- 172.21.15.32:0/992262692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a400b300 con 0x7f80ac10da80 2021-04-09T18:18:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80aa572700 1 -- 172.21.15.32:0/992262692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a400b640 con 0x7f80ac10da80 2021-04-09T18:18:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80aa572700 1 -- 172.21.15.32:0/992262692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a400bb20 con 0x7f80ac10da80 2021-04-09T18:18:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80b1877700 1 -- 172.21.15.32:0/992262692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac10da80 msgr2=0x7f80ac10de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.832 7f80b1877700 1 --2- 172.21.15.32:0/992262692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac10da80 0x7f80ac10de80 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f80a4006f00 tx=0x7f80a4009e90).stop 2021-04-09T18:18:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.833 7f80b1877700 1 -- 172.21.15.32:0/992262692 shutdown_connections 2021-04-09T18:18:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.833 7f80b1877700 1 --2- 172.21.15.32:0/992262692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac10da80 0x7f80ac10de80 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.833 7f80b1877700 1 -- 172.21.15.32:0/992262692 >> 172.21.15.32:0/992262692 conn(0x7f80ac0ce620 msgr2=0x7f80ac0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.833 7f80b1877700 1 -- 172.21.15.32:0/992262692 shutdown_connections 2021-04-09T18:18:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.833 7f80b1877700 1 -- 172.21.15.32:0/992262692 wait complete. 2021-04-09T18:18:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.853 7f80b1877700 1 Processor -- start 2021-04-09T18:18:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.853 7f80b1877700 1 -- start start 2021-04-09T18:18:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80b1877700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80ac197ff0 0x7f80ac0b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80b1877700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac0b4290 0x7f80ac09d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80b1877700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac09d6b0 0x7f80ac0b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80b1877700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac0b4aa0 con 0x7f80ac09d6b0 2021-04-09T18:18:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80b1877700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac09dab0 con 0x7f80ac0b4290 2021-04-09T18:18:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80b1877700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac09dcd0 con 0x7f80ac197ff0 2021-04-09T18:18:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80abd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac09d6b0 0x7f80ac0b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80abd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac09d6b0 0x7f80ac0b6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46380/0 (socket says 172.21.15.32:46380) 2021-04-09T18:18:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80abd75700 1 -- 172.21.15.32:0/1244596437 learned_addr learned my addr 172.21.15.32:0/1244596437 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80ab574700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80ac197ff0 0x7f80ac0b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80aad73700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac0b4290 0x7f80ac09d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80abd75700 1 -- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80ac197ff0 msgr2=0x7f80ac0b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80abd75700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80ac197ff0 0x7f80ac0b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80abd75700 1 -- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac0b4290 msgr2=0x7f80ac09d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80abd75700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac0b4290 0x7f80ac09d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80abd75700 1 -- 172.21.15.32:0/1244596437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac0b75b0 con 0x7f80ac09d6b0 2021-04-09T18:18:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80abd75700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac09d6b0 0x7f80ac0b6f90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f80a40099f0 tx=0x7f80a400bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f8093fff700 1 -- 172.21.15.32:0/1244596437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a4003030 con 0x7f80ac09d6b0 2021-04-09T18:18:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f8093fff700 1 -- 172.21.15.32:0/1244596437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a4000cc0 con 0x7f80ac09d6b0 2021-04-09T18:18:21.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80b1877700 1 -- 172.21.15.32:0/1244596437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ac0b75b0 con 0x7f80ac09d6b0 2021-04-09T18:18:21.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f8093fff700 1 -- 172.21.15.32:0/1244596437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80a4001530 con 0x7f80ac09d6b0 2021-04-09T18:18:21.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.854 7f80b1877700 1 -- 172.21.15.32:0/1244596437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ac0b75b0 con 0x7f80ac09d6b0 2021-04-09T18:18:21.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.855 7f8093fff700 1 -- 172.21.15.32:0/1244596437 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f80a4001270 con 0x7f80ac09d6b0 2021-04-09T18:18:21.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.855 7f8093fff700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f808c0188d0 0x7f808c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:21.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.855 7f8093fff700 1 -- 172.21.15.32:0/1244596437 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f80a4032ed0 con 0x7f80ac09d6b0 2021-04-09T18:18:21.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.855 7f80ab574700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f808c0188d0 0x7f808c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:21.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.856 7f80ab574700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f808c0188d0 0x7f808c01add0 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:21.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.875 7f80b1877700 1 -- 172.21.15.32:0/1244596437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ac0b2d60 con 0x7f80ac09d6b0 2021-04-09T18:18:21.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:21.879 7f8093fff700 1 -- 172.21.15.32:0/1244596437 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f80a400be20 con 0x7f80ac09d6b0 2021-04-09T18:18:22.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.061 7f80b1877700 1 -- 172.21.15.32:0/1244596437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f80ac10d500 con 0x7f80ac09d6b0 2021-04-09T18:18:22.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.606 7f8093fff700 1 -- 172.21.15.32:0/1244596437 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v76) v1 ==== 185+0+0 (secure 0 0 0) 0x7f80a401fb10 con 0x7f80ac09d6b0 2021-04-09T18:18:22.663 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:18:22.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 -- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f808c0188d0 msgr2=0x7f808c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:22.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f808c0188d0 0x7f808c01add0 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 -- 172.21.15.32:0/1244596437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac09d6b0 msgr2=0x7f80ac0b6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:22.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac09d6b0 0x7f80ac0b6f90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f80a40099f0 tx=0x7f80a400bce0).stop 2021-04-09T18:18:22.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 -- 172.21.15.32:0/1244596437 shutdown_connections 2021-04-09T18:18:22.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f808c0188d0 0x7f808c01add0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80ac09d6b0 0x7f80ac0b6f90 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80ac0b4290 0x7f80ac09d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 --2- 172.21.15.32:0/1244596437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80ac197ff0 0x7f80ac0b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 -- 172.21.15.32:0/1244596437 >> 172.21.15.32:0/1244596437 conn(0x7f80ac0911b0 msgr2=0x7f80ac0b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:22.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 -- 172.21.15.32:0/1244596437 shutdown_connections 2021-04-09T18:18:22.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.628 7f80b1877700 1 -- 172.21.15.32:0/1244596437 wait complete. 2021-04-09T18:18:22.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:18:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.830 7f06bb036700 1 Processor -- start 2021-04-09T18:18:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.830 7f06bb036700 1 -- start start 2021-04-09T18:18:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06bb036700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f06b40d7cc0 0x7f06b40d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06bb036700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b410da80 0x7f06b410de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06bb036700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b410d500 con 0x7f06b410d050 2021-04-09T18:18:22.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06bb036700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b40d9260 con 0x7f06b40d7cc0 2021-04-09T18:18:22.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06bb036700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40d0330 con 0x7f06b410da80 2021-04-09T18:18:22.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06b8e52700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f06b410d050 0x7f06b40cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55974/0 (socket says 172.21.15.32:55974) 2021-04-09T18:18:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06b8e52700 1 -- 172.21.15.32:0/958826825 learned_addr learned my addr 172.21.15.32:0/958826825 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06b9653700 1 --2- 172.21.15.32:0/958826825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b410da80 0x7f06b410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06b3fff700 1 --2- 172.21.15.32:0/958826825 >> v2:172.21.15.86:3300/0 conn(0x7f06b40d7cc0 0x7f06b40d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06b37fe700 1 -- 172.21.15.32:0/958826825 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836252142 0 0) 0x7f069c000ce0 con 0x7f06b410d050 2021-04-09T18:18:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.831 7f06b37fe700 1 -- 172.21.15.32:0/958826825 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a0002fe0 con 0x7f06b410d050 2021-04-09T18:18:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b9653700 1 -- 172.21.15.32:0/958826825 >> v2:172.21.15.86:3300/0 conn(0x7f06b40d7cc0 msgr2=0x7f06b40d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b9653700 1 --2- 172.21.15.32:0/958826825 >> v2:172.21.15.86:3300/0 conn(0x7f06b40d7cc0 0x7f06b40d80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b37fe700 1 -- 172.21.15.32:0/958826825 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3975703471 0 0) 0x7f069c000ed0 con 0x7f06b410d050 2021-04-09T18:18:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b9653700 1 -- 172.21.15.32:0/958826825 >> v1:172.21.15.32:6789/0 conn(0x7f06b410d050 legacy=0x7f06b40cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:22.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b9653700 1 -- 172.21.15.32:0/958826825 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40d86e0 con 0x7f06b410da80 2021-04-09T18:18:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b3fff700 1 --2- 172.21.15.32:0/958826825 >> v2:172.21.15.86:3300/0 conn(0x7f06b40d7cc0 0x7f06b40d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b9653700 1 --2- 172.21.15.32:0/958826825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b410da80 0x7f06b410de80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f06ac006ad0 tx=0x7f06ac009ad0).ready entity=mon.2 client_cookie=c4d4b624beb4ae70 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b37fe700 1 -- 172.21.15.32:0/958826825 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06ac0022e0 con 0x7f06b410da80 2021-04-09T18:18:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b37fe700 1 -- 172.21.15.32:0/958826825 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ac006410 con 0x7f06b410da80 2021-04-09T18:18:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06b37fe700 1 -- 172.21.15.32:0/958826825 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06ac002620 con 0x7f06b410da80 2021-04-09T18:18:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06bb036700 1 -- 172.21.15.32:0/958826825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b410da80 msgr2=0x7f06b410de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.832 7f06bb036700 1 --2- 172.21.15.32:0/958826825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b410da80 0x7f06b410de80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f06ac006ad0 tx=0x7f06ac009ad0).stop 2021-04-09T18:18:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.833 7f06bb036700 1 -- 172.21.15.32:0/958826825 shutdown_connections 2021-04-09T18:18:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.833 7f06bb036700 1 --2- 172.21.15.32:0/958826825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b410da80 0x7f06b410de80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.833 7f06bb036700 1 --2- 172.21.15.32:0/958826825 >> v2:172.21.15.86:3300/0 conn(0x7f06b40d7cc0 0x7f06b40d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.833 7f06bb036700 1 -- 172.21.15.32:0/958826825 >> 172.21.15.32:0/958826825 conn(0x7f06b40ce620 msgr2=0x7f06b40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:22.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.833 7f06bb036700 1 -- 172.21.15.32:0/958826825 shutdown_connections 2021-04-09T18:18:22.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.833 7f06bb036700 1 -- 172.21.15.32:0/958826825 wait complete. 2021-04-09T18:18:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.853 7f06bb036700 1 Processor -- start 2021-04-09T18:18:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.853 7f06bb036700 1 -- start start 2021-04-09T18:18:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06bb036700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b40c2cb0 0x7f06b40c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06bb036700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06b40b7b70 0x7f06b40b7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06bb036700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b4184bf0 0x7f06b4184ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:22.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06bb036700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40b8590 con 0x7f06b4184bf0 2021-04-09T18:18:22.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06bb036700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b4185610 con 0x7f06b40b7b70 2021-04-09T18:18:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06bb036700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40b6ae0 con 0x7f06b40c2cb0 2021-04-09T18:18:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b9653700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b4184bf0 0x7f06b4184ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b9653700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b4184bf0 0x7f06b4184ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46392/0 (socket says 172.21.15.32:46392) 2021-04-09T18:18:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b9653700 1 -- 172.21.15.32:0/4257516840 learned_addr learned my addr 172.21.15.32:0/4257516840 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b8e52700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b40c2cb0 0x7f06b40c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b3fff700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06b40b7b70 0x7f06b40b7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b9653700 1 -- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b40c2cb0 msgr2=0x7f06b40c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b9653700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b40c2cb0 0x7f06b40c30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b9653700 1 -- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06b40b7b70 msgr2=0x7f06b40b7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:22.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b9653700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06b40b7b70 0x7f06b40b7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:22.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b9653700 1 -- 172.21.15.32:0/4257516840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40b6d00 con 0x7f06b4184bf0 2021-04-09T18:18:22.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b3fff700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06b40b7b70 0x7f06b40b7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:22.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.854 7f06b9653700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b4184bf0 0x7f06b4184ff0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f06ac009670 tx=0x7f06ac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:22.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.855 7f06b17fa700 1 -- 172.21.15.32:0/4257516840 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06ac009020 con 0x7f06b4184bf0 2021-04-09T18:18:22.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.855 7f06b17fa700 1 -- 172.21.15.32:0/4257516840 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ac00bd80 con 0x7f06b4184bf0 2021-04-09T18:18:22.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.855 7f06bb036700 1 -- 172.21.15.32:0/4257516840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b40b6d00 con 0x7f06b4184bf0 2021-04-09T18:18:22.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.855 7f06b17fa700 1 -- 172.21.15.32:0/4257516840 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06ac002270 con 0x7f06b4184bf0 2021-04-09T18:18:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.855 7f06bb036700 1 -- 172.21.15.32:0/4257516840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b40b6d00 con 0x7f06b4184bf0 2021-04-09T18:18:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.856 7f06b17fa700 1 -- 172.21.15.32:0/4257516840 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f06ac002020 con 0x7f06b4184bf0 2021-04-09T18:18:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.856 7f06b17fa700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06980188c0 0x7f069801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.856 7f06b17fa700 1 -- 172.21.15.32:0/4257516840 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5397+0+0 (secure 0 0 0) 0x7f06ac0326e0 con 0x7f06b4184bf0 2021-04-09T18:18:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.856 7f06b8e52700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06980188c0 0x7f069801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.857 7f06b8e52700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06980188c0 0x7f069801adc0 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.875 7f06bb036700 1 -- 172.21.15.32:0/4257516840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b41b7180 con 0x7f06b4184bf0 2021-04-09T18:18:22.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:22.879 7f06b17fa700 1 -- 172.21.15.32:0/4257516840 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f06ac000dc0 con 0x7f06b4184bf0 2021-04-09T18:18:23.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.061 7f06bb036700 1 -- 172.21.15.32:0/4257516840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f06b410d500 con 0x7f06b4184bf0 2021-04-09T18:18:23.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.610 7f06b17fa700 1 -- 172.21.15.32:0/4257516840 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v77) v1 ==== 173+0+0 (secure 0 0 0) 0x7f06ac01f4d0 con 0x7f06b4184bf0 2021-04-09T18:18:23.663 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:18:23.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 -- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06980188c0 msgr2=0x7f069801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06980188c0 0x7f069801adc0 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:23.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 -- 172.21.15.32:0/4257516840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b4184bf0 msgr2=0x7f06b4184ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:23.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b4184bf0 0x7f06b4184ff0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f06ac009670 tx=0x7f06ac00b990).stop 2021-04-09T18:18:23.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 -- 172.21.15.32:0/4257516840 shutdown_connections 2021-04-09T18:18:23.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f06980188c0 0x7f069801adc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:23.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06b4184bf0 0x7f06b4184ff0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:23.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06b40b7b70 0x7f06b40b7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:23.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 --2- 172.21.15.32:0/4257516840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06b40c2cb0 0x7f06b40c30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.628 7f06bb036700 1 -- 172.21.15.32:0/4257516840 >> 172.21.15.32:0/4257516840 conn(0x7f06b409fc60 msgr2=0x7f06b4182170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.629 7f06bb036700 1 -- 172.21.15.32:0/4257516840 shutdown_connections 2021-04-09T18:18:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.629 7f06bb036700 1 -- 172.21.15.32:0/4257516840 wait complete. 2021-04-09T18:18:23.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:18:23.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.800 7eff062aa700 1 Processor -- start 2021-04-09T18:18:23.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.800 7eff062aa700 1 -- start start 2021-04-09T18:18:23.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.801 7eff062aa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff000d6c90 0x7eff000d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:23.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.801 7eff062aa700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff000d7290 con 0x7eff000d2450 2021-04-09T18:18:23.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.801 7eff062aa700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff000d2850 con 0x7eff000cffb0 2021-04-09T18:18:23.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.801 7eff062aa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff000d7090 con 0x7eff000d6c90 2021-04-09T18:18:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.801 7eff048c7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7eff000d2450 0x7eff0010cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55988/0 (socket says 172.21.15.32:55988) 2021-04-09T18:18:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.801 7eff048c7700 1 -- 172.21.15.32:0/2024970149 learned_addr learned my addr 172.21.15.32:0/2024970149 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.801 7efef7fff700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff000d6c90 0x7eff000d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.801 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857036608 0 0) 0x7efef8000d30 con 0x7eff000d2450 2021-04-09T18:18:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.801 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeec002fe0 con 0x7eff000d2450 2021-04-09T18:18:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728114665 0 0) 0x7efef0000c60 con 0x7eff000cffb0 2021-04-09T18:18:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeec003210 con 0x7eff000cffb0 2021-04-09T18:18:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2521302273 0 0) 0x7efef8001270 con 0x7eff000d2450 2021-04-09T18:18:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff000d6c90 msgr2=0x7eff000d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff000d6c90 0x7eff000d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 >> v1:172.21.15.86:6789/0 conn(0x7eff000cffb0 legacy=0x7eff000d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff000d91f0 con 0x7eff000d2450 2021-04-09T18:18:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7efef8001980 con 0x7eff000d2450 2021-04-09T18:18:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 >> v1:172.21.15.32:6789/0 conn(0x7eff000d2450 legacy=0x7eff0010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeec003b90 0x7efeec005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efeec0067a0 0x7efeec008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efeec009510 0x7efeec00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec0502f0 con 0x7efeec009510 2021-04-09T18:18:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec0504d0 con 0x7efeec003b90 2021-04-09T18:18:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec0506f0 con 0x7efeec0067a0 2021-04-09T18:18:23.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff048c7700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efeec009510 0x7efeec00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:23.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efefffff700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeec003b90 0x7efeec005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:23.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efef7fff700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efeec0067a0 0x7efeec008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:23.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff048c7700 1 -- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efeec0067a0 msgr2=0x7efeec008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:23.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff048c7700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efeec0067a0 0x7efeec008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:23.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff048c7700 1 -- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeec003b90 msgr2=0x7efeec005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:23.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff048c7700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeec003b90 0x7efeec005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:23.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff048c7700 1 -- 172.21.15.32:0/2024970149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec050990 con 0x7efeec009510 2021-04-09T18:18:23.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff048c7700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efeec009510 0x7efeec00ba30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7efef80020d0 tx=0x7efef800a310).ready entity=mon.0 client_cookie=ee902bc1bc147146 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:23.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef8001930 con 0x7efeec009510 2021-04-09T18:18:23.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef80070c0 con 0x7efeec009510 2021-04-09T18:18:23.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7efeff7fe700 1 -- 172.21.15.32:0/2024970149 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef8007510 con 0x7efeec009510 2021-04-09T18:18:23.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff062aa700 1 -- 172.21.15.32:0/2024970149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efeec009510 msgr2=0x7efeec00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:23.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff062aa700 1 --2- 172.21.15.32:0/2024970149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efeec009510 0x7efeec00ba30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7efef80020d0 tx=0x7efef800a310).stop 2021-04-09T18:18:23.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.802 7eff048c7700 1 -- 172.21.15.32:0/2024970149 reap_dead start 2021-04-09T18:18:23.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.803 7eff062aa700 1 -- 172.21.15.32:0/2024970149 shutdown_connections 2021-04-09T18:18:23.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.803 7eff062aa700 1 -- 172.21.15.32:0/2024970149 >> 172.21.15.32:0/2024970149 conn(0x7eff000ce5d0 msgr2=0x7eff000cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:23.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.803 7eff062aa700 1 -- 172.21.15.32:0/2024970149 shutdown_connections 2021-04-09T18:18:23.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.803 7eff062aa700 1 -- 172.21.15.32:0/2024970149 wait complete. 2021-04-09T18:18:23.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.823 7eff062aa700 1 Processor -- start 2021-04-09T18:18:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.823 7eff062aa700 1 -- start start 2021-04-09T18:18:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7eff062aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00184080 0x7eff001955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:23.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7eff062aa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff00195c00 0x7eff0008ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:23.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7eff062aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff0008f550 0x7eff0008f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:23.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7eff062aa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff00090570 con 0x7eff00184080 2021-04-09T18:18:23.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7eff062aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff00184480 con 0x7eff0008f550 2021-04-09T18:18:23.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7eff062aa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff00196000 con 0x7eff00195c00 2021-04-09T18:18:23.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7efefffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00184080 0x7eff001955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:23.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7efefffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00184080 0x7eff001955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46408/0 (socket says 172.21.15.32:46408) 2021-04-09T18:18:23.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7efefffff700 1 -- 172.21.15.32:0/3299371570 learned_addr learned my addr 172.21.15.32:0/3299371570 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7efefffff700 1 -- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff00195c00 msgr2=0x7eff0008ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:23.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7eff048c7700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff0008f550 0x7eff0008f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:23.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7efefffff700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff00195c00 0x7eff0008ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:18:23.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7efefffff700 1 -- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff0008f550 msgr2=0x7eff0008f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:23.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7efefffff700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff0008f550 0x7eff0008f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:23.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.824 7efefffff700 1 -- 172.21.15.32:0/3299371570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff001847b0 con 0x7eff00184080 2021-04-09T18:18:23.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.825 7efefffff700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00184080 0x7eff001955e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7efef000a520 tx=0x7efef000a8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:23.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.825 7efefd7fa700 1 -- 172.21.15.32:0/3299371570 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef000acb0 con 0x7eff00184080 2021-04-09T18:18:23.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.825 7efefd7fa700 1 -- 172.21.15.32:0/3299371570 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef0000e20 con 0x7eff00184080 2021-04-09T18:18:23.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.825 7eff062aa700 1 -- 172.21.15.32:0/3299371570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff001847b0 con 0x7eff00184080 2021-04-09T18:18:23.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.825 7efefd7fa700 1 -- 172.21.15.32:0/3299371570 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef000b260 con 0x7eff00184080 2021-04-09T18:18:23.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.825 7eff062aa700 1 -- 172.21.15.32:0/3299371570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff001847b0 con 0x7eff00184080 2021-04-09T18:18:23.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.825 7efefd7fa700 1 -- 172.21.15.32:0/3299371570 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efef0006740 con 0x7eff00184080 2021-04-09T18:18:23.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.826 7efefd7fa700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018900 0x7efee401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:23.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.826 7efefd7fa700 1 -- 172.21.15.32:0/3299371570 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5015+0+0 (secure 0 0 0) 0x7efef00327c0 con 0x7eff00184080 2021-04-09T18:18:23.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.826 7efef7fff700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018900 0x7efee401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:23.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.826 7efef7fff700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018900 0x7efee401adc0 crc :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:23.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.845 7eff062aa700 1 -- 172.21.15.32:0/3299371570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff00182e80 con 0x7eff00184080 2021-04-09T18:18:23.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:23.848 7efefd7fa700 1 -- 172.21.15.32:0/3299371570 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efef0000c10 con 0x7eff00184080 2021-04-09T18:18:24.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.059 7eff062aa700 1 -- 172.21.15.32:0/3299371570 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7eff000d7290 con 0x7eff00184080 2021-04-09T18:18:24.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.063 7efefd7fa700 1 -- 172.21.15.32:0/3299371570 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7efef0006740 con 0x7eff00184080 2021-04-09T18:18:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 -- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018900 msgr2=0x7efee401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018900 0x7efee401adc0 crc :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 -- 172.21.15.32:0/3299371570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00184080 msgr2=0x7eff001955e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00184080 0x7eff001955e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7efef000a520 tx=0x7efef000a8c0).stop 2021-04-09T18:18:24.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 -- 172.21.15.32:0/3299371570 shutdown_connections 2021-04-09T18:18:24.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efee4018900 0x7efee401adc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff0008f550 0x7eff0008f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff00195c00 0x7eff0008ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:18:24.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 --2- 172.21.15.32:0/3299371570 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff00184080 0x7eff001955e0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 -- 172.21.15.32:0/3299371570 >> 172.21.15.32:0/3299371570 conn(0x7eff001a8f50 msgr2=0x7eff00188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:24.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 -- 172.21.15.32:0/3299371570 shutdown_connections 2021-04-09T18:18:24.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.069 7eff062aa700 1 -- 172.21.15.32:0/3299371570 wait complete. 2021-04-09T18:18:24.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_grow' 2021-04-09T18:18:24.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.233 7f3abb085700 1 Processor -- start 2021-04-09T18:18:24.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.233 7f3abb085700 1 -- start start 2021-04-09T18:18:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.233 7f3abb085700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3ab410d010 0x7f3ab40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.233 7f3abb085700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ab40d1da0 0x7f3ab40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.233 7f3abb085700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab410d480 con 0x7f3ab40d82c0 2021-04-09T18:18:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3abb085700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab40d8d30 con 0x7f3ab410d010 2021-04-09T18:18:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3abb085700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab40d9220 con 0x7f3ab40d1da0 2021-04-09T18:18:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab8ea1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3ab410d010 0x7f3ab40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab96a2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3ab40d82c0 0x7f3ab410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56008/0 (socket says 172.21.15.32:56008) 2021-04-09T18:18:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab96a2700 1 -- 172.21.15.32:0/2973335998 learned_addr learned my addr 172.21.15.32:0/2973335998 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab3fff700 1 --2- 172.21.15.32:0/2973335998 >> v2:172.21.15.86:3300/0 conn(0x7f3ab40d1da0 0x7f3ab40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab8ea1700 1 -- 172.21.15.32:0/2973335998 >> v2:172.21.15.86:3300/0 conn(0x7f3ab40d1da0 msgr2=0x7f3ab40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab8ea1700 1 --2- 172.21.15.32:0/2973335998 >> v2:172.21.15.86:3300/0 conn(0x7f3ab40d1da0 0x7f3ab40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab8ea1700 1 -- 172.21.15.32:0/2973335998 >> v1:172.21.15.32:6789/0 conn(0x7f3ab40d82c0 legacy=0x7f3ab410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab8ea1700 1 -- 172.21.15.32:0/2973335998 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab40d86f0 con 0x7f3ab410d010 2021-04-09T18:18:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab37fe700 1 -- 172.21.15.32:0/2973335998 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550478838 0 0) 0x7f3aac000d30 con 0x7f3ab40d82c0 2021-04-09T18:18:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab3fff700 1 --2- 172.21.15.32:0/2973335998 >> v2:172.21.15.86:3300/0 conn(0x7f3ab40d1da0 0x7f3ab40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab8ea1700 1 --2- 172.21.15.32:0/2973335998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab410d010 0x7f3ab40d1780 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f3aa4006650 tx=0x7f3aa4009a80).ready entity=mon.0 client_cookie=af9ac275744117c8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab37fe700 1 -- 172.21.15.32:0/2973335998 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3aa40009b0 con 0x7f3ab410d010 2021-04-09T18:18:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab37fe700 1 -- 172.21.15.32:0/2973335998 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa400c0c0 con 0x7f3ab410d010 2021-04-09T18:18:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3ab37fe700 1 -- 172.21.15.32:0/2973335998 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3aa400c510 con 0x7f3ab410d010 2021-04-09T18:18:24.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3abb085700 1 -- 172.21.15.32:0/2973335998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab410d010 msgr2=0x7f3ab40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.234 7f3abb085700 1 --2- 172.21.15.32:0/2973335998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab410d010 0x7f3ab40d1780 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f3aa4006650 tx=0x7f3aa4009a80).stop 2021-04-09T18:18:24.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.235 7f3abb085700 1 -- 172.21.15.32:0/2973335998 shutdown_connections 2021-04-09T18:18:24.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.235 7f3abb085700 1 --2- 172.21.15.32:0/2973335998 >> v2:172.21.15.86:3300/0 conn(0x7f3ab40d1da0 0x7f3ab40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.235 7f3abb085700 1 --2- 172.21.15.32:0/2973335998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab410d010 0x7f3ab40d1780 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.235 7f3abb085700 1 -- 172.21.15.32:0/2973335998 >> 172.21.15.32:0/2973335998 conn(0x7f3ab40ce5f0 msgr2=0x7f3ab40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.235 7f3abb085700 1 -- 172.21.15.32:0/2973335998 shutdown_connections 2021-04-09T18:18:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.235 7f3abb085700 1 -- 172.21.15.32:0/2973335998 wait complete. 2021-04-09T18:18:24.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.255 7f3abb085700 1 Processor -- start 2021-04-09T18:18:24.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.255 7f3abb085700 1 -- start start 2021-04-09T18:18:24.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3abb085700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ab4179bf0 0x7f3ab4179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3abb085700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab417c540 0x7f3ab417c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3abb085700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ab40aca60 0x7f3ab40ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3abb085700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab417cf60 con 0x7f3ab417c540 2021-04-09T18:18:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3abb085700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab40ad480 con 0x7f3ab4179bf0 2021-04-09T18:18:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3abb085700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab40ad6a0 con 0x7f3ab40aca60 2021-04-09T18:18:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab417c540 0x7f3ab417c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab417c540 0x7f3ab417c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46426/0 (socket says 172.21.15.32:46426) 2021-04-09T18:18:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab3fff700 1 -- 172.21.15.32:0/3164625148 learned_addr learned my addr 172.21.15.32:0/3164625148 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab8ea1700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ab4179bf0 0x7f3ab4179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab96a2700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ab40aca60 0x7f3ab40ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab3fff700 1 -- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ab40aca60 msgr2=0x7f3ab40ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab3fff700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ab40aca60 0x7f3ab40ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab3fff700 1 -- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ab4179bf0 msgr2=0x7f3ab4179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab3fff700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ab4179bf0 0x7f3ab4179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab3fff700 1 -- 172.21.15.32:0/3164625148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab40ad8c0 con 0x7f3ab417c540 2021-04-09T18:18:24.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.256 7f3ab96a2700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ab40aca60 0x7f3ab40ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:24.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.257 7f3ab3fff700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab417c540 0x7f3ab417c940 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f3aa8006f20 tx=0x7f3aa800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.257 7f3ab17fa700 1 -- 172.21.15.32:0/3164625148 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3aa800bd80 con 0x7f3ab417c540 2021-04-09T18:18:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.257 7f3ab17fa700 1 -- 172.21.15.32:0/3164625148 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa8002030 con 0x7f3ab417c540 2021-04-09T18:18:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.257 7f3abb085700 1 -- 172.21.15.32:0/3164625148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab40ad8c0 con 0x7f3ab417c540 2021-04-09T18:18:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.257 7f3ab17fa700 1 -- 172.21.15.32:0/3164625148 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3aa80024c0 con 0x7f3ab417c540 2021-04-09T18:18:24.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.257 7f3abb085700 1 -- 172.21.15.32:0/3164625148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab40ad8c0 con 0x7f3ab417c540 2021-04-09T18:18:24.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.257 7f3ab17fa700 1 -- 172.21.15.32:0/3164625148 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3aa8002210 con 0x7f3ab417c540 2021-04-09T18:18:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.258 7f3ab17fa700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a940188d0 0x7f3a9401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.258 7f3ab17fa700 1 -- 172.21.15.32:0/3164625148 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5015+0+0 (secure 0 0 0) 0x7f3aa8032810 con 0x7f3ab417c540 2021-04-09T18:18:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.258 7f3ab8ea1700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a940188d0 0x7f3a9401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.258 7f3ab8ea1700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a940188d0 0x7f3a9401add0 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:24.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.278 7f3abb085700 1 -- 172.21.15.32:0/3164625148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab417cf60 con 0x7f3ab417c540 2021-04-09T18:18:24.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.281 7f3ab17fa700 1 -- 172.21.15.32:0/3164625148 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3aa8002240 con 0x7f3ab417c540 2021-04-09T18:18:24.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.469 7f3abb085700 1 -- 172.21.15.32:0/3164625148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_grow"]} v 0) v1 -- 0x7f3ab417cf60 con 0x7f3ab417c540 2021-04-09T18:18:24.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.612 7f3ab17fa700 1 -- 172.21.15.32:0/3164625148 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_grow"]}]=0 v273) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3aa8002030 con 0x7f3ab417c540 2021-04-09T18:18:24.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.634 7f3abb085700 1 -- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a940188d0 msgr2=0x7f3a9401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.634 7f3abb085700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a940188d0 0x7f3a9401add0 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.634 7f3abb085700 1 -- 172.21.15.32:0/3164625148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab417c540 msgr2=0x7f3ab417c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.634 7f3abb085700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab417c540 0x7f3ab417c940 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f3aa8006f20 tx=0x7f3aa800b990).stop 2021-04-09T18:18:24.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.635 7f3abb085700 1 -- 172.21.15.32:0/3164625148 shutdown_connections 2021-04-09T18:18:24.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.635 7f3abb085700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a940188d0 0x7f3a9401add0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.635 7f3abb085700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab417c540 0x7f3ab417c940 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.635 7f3abb085700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ab40aca60 0x7f3ab40ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.635 7f3abb085700 1 --2- 172.21.15.32:0/3164625148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ab4179bf0 0x7f3ab4179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.635 7f3abb085700 1 -- 172.21.15.32:0/3164625148 >> 172.21.15.32:0/3164625148 conn(0x7f3ab41acae0 msgr2=0x7f3ab417ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:24.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.635 7f3abb085700 1 -- 172.21.15.32:0/3164625148 shutdown_connections 2021-04-09T18:18:24.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.635 7f3abb085700 1 -- 172.21.15.32:0/3164625148 wait complete. 2021-04-09T18:18:24.684 INFO:tasks.cephfs_test_runner:test_grow (tasks.cephfs.test_failover.TestClusterResize) ... ok 2021-04-09T18:18:24.684 INFO:tasks.cephfs_test_runner:Starting test: test_hole (tasks.cephfs.test_failover.TestClusterResize) 2021-04-09T18:18:24.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_hole' 2021-04-09T18:18:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.799 7fb5dff72700 1 Processor -- start 2021-04-09T18:18:24.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.799 7fb5dff72700 1 -- start start 2021-04-09T18:18:24.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5dff72700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb5d80d7ca0 0x7fb5d80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5dff72700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5d810da00 0x7fb5d810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5dff72700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d810d480 con 0x7fb5d810d010 2021-04-09T18:18:24.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5dff72700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d80d9220 con 0x7fb5d810da00 2021-04-09T18:18:24.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5dff72700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d80d86c0 con 0x7fb5d80d7ca0 2021-04-09T18:18:24.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5de58f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5d810da00 0x7fb5d810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5ddd8e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb5d810d010 0x7fb5d80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56020/0 (socket says 172.21.15.32:56020) 2021-04-09T18:18:24.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5de58f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb5d810da00 0x7fb5d810de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46434/0 (socket says 172.21.15.32:46434) 2021-04-09T18:18:24.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5ddd8e700 1 -- 172.21.15.32:0/2595412277 learned_addr learned my addr 172.21.15.32:0/2595412277 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:24.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5dd58d700 1 --2- 172.21.15.32:0/2595412277 >> v2:172.21.15.86:3300/0 conn(0x7fb5d80d7ca0 0x7fb5d80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5de58f700 1 -- 172.21.15.32:0/2595412277 >> v2:172.21.15.86:3300/0 conn(0x7fb5d80d7ca0 msgr2=0x7fb5d80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:24.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5de58f700 1 --2- 172.21.15.32:0/2595412277 >> v2:172.21.15.86:3300/0 conn(0x7fb5d80d7ca0 0x7fb5d80d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5de58f700 1 -- 172.21.15.32:0/2595412277 >> v1:172.21.15.32:6789/0 conn(0x7fb5d810d010 legacy=0x7fb5d80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5dcd8c700 1 -- 172.21.15.32:0/2595412277 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846139615 0 0) 0x7fb5d4000ce0 con 0x7fb5d810d010 2021-04-09T18:18:24.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5de58f700 1 -- 172.21.15.32:0/2595412277 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d80d1f70 con 0x7fb5d810da00 2021-04-09T18:18:24.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5dd58d700 1 --2- 172.21.15.32:0/2595412277 >> v2:172.21.15.86:3300/0 conn(0x7fb5d80d7ca0 0x7fb5d80d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:24.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5de58f700 1 --2- 172.21.15.32:0/2595412277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d810da00 0x7fb5d810de00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7fb5d0006e30 tx=0x7fb5d0009e60).ready entity=mon.0 client_cookie=5f6fa6e338ae10c1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:24.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.800 7fb5dcd8c700 1 -- 172.21.15.32:0/2595412277 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d0006a50 con 0x7fb5d810da00 2021-04-09T18:18:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.801 7fb5dcd8c700 1 -- 172.21.15.32:0/2595412277 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d000b300 con 0x7fb5d810da00 2021-04-09T18:18:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.801 7fb5dcd8c700 1 -- 172.21.15.32:0/2595412277 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d000b750 con 0x7fb5d810da00 2021-04-09T18:18:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.801 7fb5dff72700 1 -- 172.21.15.32:0/2595412277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d810da00 msgr2=0x7fb5d810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.801 7fb5dff72700 1 --2- 172.21.15.32:0/2595412277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d810da00 0x7fb5d810de00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7fb5d0006e30 tx=0x7fb5d0009e60).stop 2021-04-09T18:18:24.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.801 7fb5dff72700 1 -- 172.21.15.32:0/2595412277 shutdown_connections 2021-04-09T18:18:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.801 7fb5dff72700 1 --2- 172.21.15.32:0/2595412277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d810da00 0x7fb5d810de00 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.801 7fb5dff72700 1 --2- 172.21.15.32:0/2595412277 >> v2:172.21.15.86:3300/0 conn(0x7fb5d80d7ca0 0x7fb5d80d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.801 7fb5dff72700 1 -- 172.21.15.32:0/2595412277 >> 172.21.15.32:0/2595412277 conn(0x7fb5d80ce5f0 msgr2=0x7fb5d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.802 7fb5dff72700 1 -- 172.21.15.32:0/2595412277 shutdown_connections 2021-04-09T18:18:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.802 7fb5dff72700 1 -- 172.21.15.32:0/2595412277 wait complete. 2021-04-09T18:18:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5dff72700 1 Processor -- start 2021-04-09T18:18:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5dff72700 1 -- start start 2021-04-09T18:18:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5dff72700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d80b42d0 0x7fb5d8183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5dff72700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d81ac090 0x7fb5d81ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5dff72700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d8184d30 0x7fb5d8185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5dff72700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d80a1a90 con 0x7fb5d80b42d0 2021-04-09T18:18:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5dff72700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d81acab0 con 0x7fb5d8184d30 2021-04-09T18:18:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5dff72700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d8185a00 con 0x7fb5d81ac090 2021-04-09T18:18:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5ddd8e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d80b42d0 0x7fb5d8183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5ddd8e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d80b42d0 0x7fb5d8183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46436/0 (socket says 172.21.15.32:46436) 2021-04-09T18:18:24.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5ddd8e700 1 -- 172.21.15.32:0/601980870 learned_addr learned my addr 172.21.15.32:0/601980870 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:24.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5de58f700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d8184d30 0x7fb5d8185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.822 7fb5dd58d700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d81ac090 0x7fb5d81ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:24.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5ddd8e700 1 -- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d81ac090 msgr2=0x7fb5d81ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5ddd8e700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d81ac090 0x7fb5d81ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5ddd8e700 1 -- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d8184d30 msgr2=0x7fb5d8185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:24.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5ddd8e700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d8184d30 0x7fb5d8185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:24.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5ddd8e700 1 -- 172.21.15.32:0/601980870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d8185750 con 0x7fb5d80b42d0 2021-04-09T18:18:24.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5de58f700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d8184d30 0x7fb5d8185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:24.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5ddd8e700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d80b42d0 0x7fb5d8183520 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fb5d400b530 tx=0x7fb5d400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:24.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5ce7fc700 1 -- 172.21.15.32:0/601980870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d4006020 con 0x7fb5d80b42d0 2021-04-09T18:18:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5ce7fc700 1 -- 172.21.15.32:0/601980870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d40063a0 con 0x7fb5d80b42d0 2021-04-09T18:18:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5dff72700 1 -- 172.21.15.32:0/601980870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d8185750 con 0x7fb5d80b42d0 2021-04-09T18:18:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5ce7fc700 1 -- 172.21.15.32:0/601980870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d4004590 con 0x7fb5d80b42d0 2021-04-09T18:18:25.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.823 7fb5dff72700 1 -- 172.21.15.32:0/601980870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d8185750 con 0x7fb5d80b42d0 2021-04-09T18:18:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.824 7fb5ce7fc700 1 -- 172.21.15.32:0/601980870 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb5d400a000 con 0x7fb5d80b42d0 2021-04-09T18:18:25.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.824 7fb5ce7fc700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5c0018930 0x7fb5c001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:25.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.824 7fb5ce7fc700 1 -- 172.21.15.32:0/601980870 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5015+0+0 (secure 0 0 0) 0x7fb5d40327a0 con 0x7fb5d80b42d0 2021-04-09T18:18:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.824 7fb5dd58d700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5c0018930 0x7fb5c001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.824 7fb5dd58d700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5c0018930 0x7fb5c001ae10 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:25.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.844 7fb5dff72700 1 -- 172.21.15.32:0/601980870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d8182d10 con 0x7fb5d80b42d0 2021-04-09T18:18:25.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:24.847 7fb5ce7fc700 1 -- 172.21.15.32:0/601980870 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb5d4006b90 con 0x7fb5d80b42d0 2021-04-09T18:18:25.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.033 7fb5dff72700 1 -- 172.21.15.32:0/601980870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_hole"]} v 0) v1 -- 0x7fb5d810d480 con 0x7fb5d80b42d0 2021-04-09T18:18:25.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.615 7fb5ce7fc700 1 -- 172.21.15.32:0/601980870 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_hole"]}]=0 v274) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb5d400a000 con 0x7fb5d80b42d0 2021-04-09T18:18:25.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 -- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5c0018930 msgr2=0x7fb5c001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:25.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5c0018930 0x7fb5c001ae10 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 -- 172.21.15.32:0/601980870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d80b42d0 msgr2=0x7fb5d8183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d80b42d0 0x7fb5d8183520 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fb5d400b530 tx=0x7fb5d400b8d0).stop 2021-04-09T18:18:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 -- 172.21.15.32:0/601980870 shutdown_connections 2021-04-09T18:18:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5c0018930 0x7fb5c001ae10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d8184d30 0x7fb5d8185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d81ac090 0x7fb5d81ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:25.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 --2- 172.21.15.32:0/601980870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5d80b42d0 0x7fb5d8183520 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:25.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 -- 172.21.15.32:0/601980870 >> 172.21.15.32:0/601980870 conn(0x7fb5d819b890 msgr2=0x7fb5d81a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:25.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 -- 172.21.15.32:0/601980870 shutdown_connections 2021-04-09T18:18:25.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.651 7fb5dff72700 1 -- 172.21.15.32:0/601980870 wait complete. 2021-04-09T18:18:25.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:18:25.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.820 7fe308860700 1 Processor -- start 2021-04-09T18:18:25.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.820 7fe308860700 1 -- start start 2021-04-09T18:18:25.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe308860700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe30410d010 0x7fe3040d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe308860700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3040d1da0 0x7fe3040d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe308860700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30410d480 con 0x7fe3040d82c0 2021-04-09T18:18:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe308860700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3040d9220 con 0x7fe30410d010 2021-04-09T18:18:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe308860700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040d86c0 con 0x7fe3040d1da0 2021-04-09T18:18:25.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe30261c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe30410d010 0x7fe3040d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:25.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe30261c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe30410d010 0x7fe3040d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46448/0 (socket says 172.21.15.32:46448) 2021-04-09T18:18:25.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe30261c700 1 -- 172.21.15.32:0/1030937709 learned_addr learned my addr 172.21.15.32:0/1030937709 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:25.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe301e1b700 1 --2- 172.21.15.32:0/1030937709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3040d1da0 0x7fe3040d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe30261c700 1 -- 172.21.15.32:0/1030937709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3040d1da0 msgr2=0x7fe3040d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe30261c700 1 --2- 172.21.15.32:0/1030937709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3040d1da0 0x7fe3040d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:25.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe30261c700 1 -- 172.21.15.32:0/1030937709 >> v1:172.21.15.86:6789/0 conn(0x7fe3040d82c0 legacy=0x7fe30410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:25.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe30261c700 1 -- 172.21.15.32:0/1030937709 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30410e1f0 con 0x7fe30410d010 2021-04-09T18:18:25.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.821 7fe301e1b700 1 --2- 172.21.15.32:0/1030937709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3040d1da0 0x7fe3040d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:25.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.822 7fe30261c700 1 --2- 172.21.15.32:0/1030937709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30410d010 0x7fe3040d1780 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fe2f4006700 tx=0x7fe2f4009b80).ready entity=mon.0 client_cookie=c01ab55e328adcb6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:25.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.822 7fe30161a700 1 -- 172.21.15.32:0/1030937709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2f400c0c0 con 0x7fe30410d010 2021-04-09T18:18:25.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.822 7fe30161a700 1 -- 172.21.15.32:0/1030937709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f400c440 con 0x7fe30410d010 2021-04-09T18:18:25.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.822 7fe30161a700 1 -- 172.21.15.32:0/1030937709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2f400c920 con 0x7fe30410d010 2021-04-09T18:18:25.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.822 7fe308860700 1 -- 172.21.15.32:0/1030937709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30410d010 msgr2=0x7fe3040d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:25.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.822 7fe308860700 1 --2- 172.21.15.32:0/1030937709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30410d010 0x7fe3040d1780 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fe2f4006700 tx=0x7fe2f4009b80).stop 2021-04-09T18:18:25.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.823 7fe308860700 1 -- 172.21.15.32:0/1030937709 shutdown_connections 2021-04-09T18:18:25.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.823 7fe308860700 1 --2- 172.21.15.32:0/1030937709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3040d1da0 0x7fe3040d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:25.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.823 7fe308860700 1 --2- 172.21.15.32:0/1030937709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30410d010 0x7fe3040d1780 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:25.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.823 7fe308860700 1 -- 172.21.15.32:0/1030937709 >> 172.21.15.32:0/1030937709 conn(0x7fe3040ce5f0 msgr2=0x7fe3040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:25.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.823 7fe308860700 1 -- 172.21.15.32:0/1030937709 shutdown_connections 2021-04-09T18:18:25.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.823 7fe308860700 1 -- 172.21.15.32:0/1030937709 wait complete. 2021-04-09T18:18:25.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.843 7fe308860700 1 Processor -- start 2021-04-09T18:18:25.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.843 7fe308860700 1 -- start start 2021-04-09T18:18:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.843 7fe308860700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30408a910 0x7fe30409e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.843 7fe308860700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe30408ad10 0x7fe30409ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.843 7fe308860700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe304092380 0x7fe304092780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.843 7fe308860700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30408b110 con 0x7fe30408a910 2021-04-09T18:18:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.843 7fe308860700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30409e0d0 con 0x7fe30408ad10 2021-04-09T18:18:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.843 7fe308860700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30408b3f0 con 0x7fe304092380 2021-04-09T18:18:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe30261c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30408a910 0x7fe30409e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe30261c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30408a910 0x7fe30409e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46454/0 (socket says 172.21.15.32:46454) 2021-04-09T18:18:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe30261c700 1 -- 172.21.15.32:0/2659277263 learned_addr learned my addr 172.21.15.32:0/2659277263 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe30261c700 1 -- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe304092380 msgr2=0x7fe304092780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe30261c700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe304092380 0x7fe304092780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:18:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe30261c700 1 -- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe30408ad10 msgr2=0x7fe30409ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe30261c700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe30408ad10 0x7fe30409ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:18:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe30261c700 1 -- 172.21.15.32:0/2659277263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304092da0 con 0x7fe30408a910 2021-04-09T18:18:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe30261c700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30408a910 0x7fe30409e650 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fe2f400b650 tx=0x7fe2f400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe2faffd700 1 -- 172.21.15.32:0/2659277263 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2f400bde0 con 0x7fe30408a910 2021-04-09T18:18:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe2faffd700 1 -- 172.21.15.32:0/2659277263 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4000c90 con 0x7fe30408a910 2021-04-09T18:18:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe308860700 1 -- 172.21.15.32:0/2659277263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe304092da0 con 0x7fe30408a910 2021-04-09T18:18:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe2faffd700 1 -- 172.21.15.32:0/2659277263 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2f4002220 con 0x7fe30408a910 2021-04-09T18:18:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.844 7fe308860700 1 -- 172.21.15.32:0/2659277263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe304092da0 con 0x7fe30408a910 2021-04-09T18:18:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.845 7fe2faffd700 1 -- 172.21.15.32:0/2659277263 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe2f4009f90 con 0x7fe30408a910 2021-04-09T18:18:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.845 7fe2faffd700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2e80188d0 0x7fe2e801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.845 7fe2faffd700 1 -- 172.21.15.32:0/2659277263 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5015+0+0 (secure 0 0 0) 0x7fe2f40327a0 con 0x7fe30408a910 2021-04-09T18:18:25.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.845 7fe301e1b700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2e80188d0 0x7fe2e801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:25.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.845 7fe301e1b700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2e80188d0 0x7fe2e801add0 crc :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:25.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.865 7fe308860700 1 -- 172.21.15.32:0/2659277263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30418fc90 con 0x7fe30408a910 2021-04-09T18:18:25.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:25.869 7fe2faffd700 1 -- 172.21.15.32:0/2659277263 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe2f40091e0 con 0x7fe30408a910 2021-04-09T18:18:26.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.080 7fe308860700 1 -- 172.21.15.32:0/2659277263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe30418fc90 con 0x7fe30408a910 2021-04-09T18:18:26.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.083 7fe2faffd700 1 -- 172.21.15.32:0/2659277263 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe2f4009f90 con 0x7fe30408a910 2021-04-09T18:18:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 -- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2e80188d0 msgr2=0x7fe2e801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:26.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2e80188d0 0x7fe2e801add0 crc :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 -- 172.21.15.32:0/2659277263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30408a910 msgr2=0x7fe30409e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30408a910 0x7fe30409e650 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fe2f400b650 tx=0x7fe2f400b9f0).stop 2021-04-09T18:18:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 -- 172.21.15.32:0/2659277263 shutdown_connections 2021-04-09T18:18:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe2e80188d0 0x7fe2e801add0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:26.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe304092380 0x7fe304092780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:18:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe30408ad10 0x7fe30409ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:18:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 --2- 172.21.15.32:0/2659277263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe30408a910 0x7fe30409e650 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 -- 172.21.15.32:0/2659277263 >> 172.21.15.32:0/2659277263 conn(0x7fe3040b2fa0 msgr2=0x7fe3040a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 -- 172.21.15.32:0/2659277263 shutdown_connections 2021-04-09T18:18:26.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.090 7fe308860700 1 -- 172.21.15.32:0/2659277263 wait complete. 2021-04-09T18:18:26.138 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:18:26.139 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:18:26.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:18:26.257 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:18:26.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:26.259 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:18:26.259 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:18:26.312 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:18:26.313 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:18:26.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:26.314 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:18:26.315 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:18:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:18:26.366 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:18:26.367 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:18:26.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:18:26.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.479 7fc0e1e40700 1 Processor -- start 2021-04-09T18:18:26.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.479 7fc0e1e40700 1 -- start start 2021-04-09T18:18:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.479 7fc0e1e40700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc0d7ca0 0x7fc0dc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.479 7fc0e1e40700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0dc10da00 0x7fc0dc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.479 7fc0e1e40700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0dc10d480 con 0x7fc0dc10d010 2021-04-09T18:18:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.479 7fc0e1e40700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0dc0d9220 con 0x7fc0dc10da00 2021-04-09T18:18:26.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.479 7fc0e1e40700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc0d86c0 con 0x7fc0dc0d7ca0 2021-04-09T18:18:26.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.479 7fc0dbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0dc10da00 0x7fc0dc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0dbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0dc10da00 0x7fc0dc10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46462/0 (socket says 172.21.15.32:46462) 2021-04-09T18:18:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0db7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc0dc10d010 0x7fc0dc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56052/0 (socket says 172.21.15.32:56052) 2021-04-09T18:18:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0dbfff700 1 -- 172.21.15.32:0/1080905083 learned_addr learned my addr 172.21.15.32:0/1080905083 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:26.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0daffd700 1 --2- 172.21.15.32:0/1080905083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc0d7ca0 0x7fc0dc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:26.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0dbfff700 1 -- 172.21.15.32:0/1080905083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc0d7ca0 msgr2=0x7fc0dc0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:26.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0dbfff700 1 --2- 172.21.15.32:0/1080905083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc0d7ca0 0x7fc0dc0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:26.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0dbfff700 1 -- 172.21.15.32:0/1080905083 >> v1:172.21.15.32:6789/0 conn(0x7fc0dc10d010 legacy=0x7fc0dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0dbfff700 1 -- 172.21.15.32:0/1080905083 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc0d1f70 con 0x7fc0dc10da00 2021-04-09T18:18:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0da7fc700 1 -- 172.21.15.32:0/1080905083 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803846543 0 0) 0x7fc0cc001060 con 0x7fc0dc10d010 2021-04-09T18:18:26.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0dbfff700 1 --2- 172.21.15.32:0/1080905083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc10da00 0x7fc0dc10de00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7fc0d4006b80 tx=0x7fc0d4009b90).ready entity=mon.0 client_cookie=49c92a4ec4cdb879 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:26.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0da7fc700 1 -- 172.21.15.32:0/1080905083 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0d40022e0 con 0x7fc0dc10da00 2021-04-09T18:18:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0da7fc700 1 -- 172.21.15.32:0/1080905083 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d4002620 con 0x7fc0dc10da00 2021-04-09T18:18:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0da7fc700 1 -- 172.21.15.32:0/1080905083 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0d4002b00 con 0x7fc0dc10da00 2021-04-09T18:18:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0e1e40700 1 -- 172.21.15.32:0/1080905083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc10da00 msgr2=0x7fc0dc10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:26.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.480 7fc0e1e40700 1 --2- 172.21.15.32:0/1080905083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc10da00 0x7fc0dc10de00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7fc0d4006b80 tx=0x7fc0d4009b90).stop 2021-04-09T18:18:26.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.481 7fc0e1e40700 1 -- 172.21.15.32:0/1080905083 shutdown_connections 2021-04-09T18:18:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.481 7fc0e1e40700 1 --2- 172.21.15.32:0/1080905083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc10da00 0x7fc0dc10de00 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.481 7fc0e1e40700 1 --2- 172.21.15.32:0/1080905083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc0d7ca0 0x7fc0dc0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:26.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.481 7fc0e1e40700 1 -- 172.21.15.32:0/1080905083 >> 172.21.15.32:0/1080905083 conn(0x7fc0dc0ce5f0 msgr2=0x7fc0dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:26.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.481 7fc0e1e40700 1 -- 172.21.15.32:0/1080905083 shutdown_connections 2021-04-09T18:18:26.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.481 7fc0e1e40700 1 -- 172.21.15.32:0/1080905083 wait complete. 2021-04-09T18:18:26.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.501 7fc0e1e40700 1 Processor -- start 2021-04-09T18:18:26.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.501 7fc0e1e40700 1 -- start start 2021-04-09T18:18:26.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0e1e40700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc0b42d0 0x7fc0dc183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:26.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0e1e40700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0dc1ac090 0x7fc0dc1ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:26.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0e1e40700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc184d30 0x7fc0dc185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:26.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0e1e40700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc0a1a90 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0e1e40700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc1acab0 con 0x7fc0dc1ac090 2021-04-09T18:18:26.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0e1e40700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc185a00 con 0x7fc0dc184d30 2021-04-09T18:18:26.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0db7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc0b42d0 0x7fc0dc183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:26.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0db7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc0b42d0 0x7fc0dc183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46468/0 (socket says 172.21.15.32:46468) 2021-04-09T18:18:26.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0db7fe700 1 -- 172.21.15.32:0/3818393551 learned_addr learned my addr 172.21.15.32:0/3818393551 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:26.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0daffd700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0dc1ac090 0x7fc0dc1ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:26.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0dbfff700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc184d30 0x7fc0dc185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:26.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0db7fe700 1 -- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc184d30 msgr2=0x7fc0dc185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:26.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.502 7fc0db7fe700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc184d30 0x7fc0dc185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:26.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0db7fe700 1 -- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0dc1ac090 msgr2=0x7fc0dc1ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:26.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0db7fe700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0dc1ac090 0x7fc0dc1ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:26.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0db7fe700 1 -- 172.21.15.32:0/3818393551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc185750 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0db7fe700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc0b42d0 0x7fc0dc183520 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7fc0cc009980 tx=0x7fc0cc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:26.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0c3fff700 1 -- 172.21.15.32:0/3818393551 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0cc006300 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0c3fff700 1 -- 172.21.15.32:0/3818393551 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0cc006580 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0dc185750 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0c3fff700 1 -- 172.21.15.32:0/3818393551 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0cc004910 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0dc185750 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.503 7fc0c3fff700 1 -- 172.21.15.32:0/3818393551 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc0cc006020 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.504 7fc0c3fff700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0bc018930 0x7fc0bc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:26.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.504 7fc0c3fff700 1 -- 172.21.15.32:0/3818393551 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5015+0+0 (secure 0 0 0) 0x7fc0cc032b70 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.504 7fc0daffd700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0bc018930 0x7fc0bc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:26.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.504 7fc0daffd700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0bc018930 0x7fc0bc01ae10 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:26.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.524 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0dc182d10 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.527 7fc0c3fff700 1 -- 172.21.15.32:0/3818393551 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc0cc00be10 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.715 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fc0dc10d480 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.716 7fc0c3fff700 1 -- 172.21.15.32:0/3818393551 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v77) v1 ==== 81+0+25821 (secure 0 0 0) 0x7fc0cc006580 con 0x7fc0dc0b42d0 2021-04-09T18:18:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:26.756 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 77, 2021-04-09T18:18:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:18:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:18:23.606810", 2021-04-09T18:18:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:18:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:18:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:18:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:18:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:18:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:18:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:18:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:18:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:18:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:18:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:18:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:18:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:18:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:18:26.769 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:18:26.769 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 13, 2021-04-09T18:18:26.769 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:18:26.770 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:18:26.770 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:18:26.771 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:18:26.771 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:18:26.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:26.772 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:18:26.781 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:18:26.782 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:18:26.784 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:18:26.785 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:18:26.786 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:18:26.786 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:18:26.787 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:18:26.787 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:18:26.787 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:18:26.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:18:26.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:18:26.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:18:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:18:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:18:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:18:26.858 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:18:26.918 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:18:26.918 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:18:26.919 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:18:26.919 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:18:26.919 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:18:26.920 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:18:26.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:26.922 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:18:26.922 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:18:26.923 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:18:26.923 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:18:26.923 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:18:26.924 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:18:26.924 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:18:26.925 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:18:26.925 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:18:26.925 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:18:26.926 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:18:26.926 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:18:26.927 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:18:26.927 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:18:26.935 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:18:26.935 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:18:26.936 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:18:26.937 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:18:26.938 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:18:26.938 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:18:26.939 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:18:26.939 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:18:26.940 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:18:26.940 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:18:26.940 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:18:26.941 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:18:26.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:18:26.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:26.945 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:18:26.945 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:18:26.946 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:18:26.946 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:18:26.947 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:18:26.947 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:18:26.948 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:18:26.948 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:18:26.949 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:18:26.949 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:18:26.949 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:18:26.950 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:18:26.950 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:18:26.951 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:18:26.951 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:26.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:26.963 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:18:26.963 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:18:26.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:26.964 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:18:26.965 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:18:26.965 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:26.965 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:26.966 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:27.047 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:27.048 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:27.048 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:27.049 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:27.049 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:27.049 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:27.050 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:27.050 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:27.051 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.052 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:18:27.053 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:27.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.084 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.084 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:18:27.084 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:27.085 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.085 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.086 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:27.108 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:18:27.110 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:27.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.114 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:18:27.114 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:27.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.114 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.115 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:27.115 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.116 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:18:27.116 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:27.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:18:27.117 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:27.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.118 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.139 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:27.139 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.140 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:18:27.140 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:27.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.141 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.141 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:18:27.141 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:18:27.141 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.142 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.142 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:18:27.142 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:18:27.143 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:18:27.151 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:18:27.152 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:27.153 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:27.154 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:27.154 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.176 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:18:27.176 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:18:27.177 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:27.177 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:27.177 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:27.177 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:27.178 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:27.179 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:27.179 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:27.179 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 71, 2021-04-09T18:18:27.179 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:27.180 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:27.180 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:27.180 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.181 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.181 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:18:27.181 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:27.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.184 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.184 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:18:27.184 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:27.184 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.185 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.185 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:27.185 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.186 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.186 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:18:27.186 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:27.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.194 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:18:27.194 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:27.194 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.195 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.195 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:27.195 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.231 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.231 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:18:27.232 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:27.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.233 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.233 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:18:27.234 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:27.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.235 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.236 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:27.236 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.238 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:18:27.238 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:27.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.240 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.240 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:18:27.241 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:18:27.241 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.242 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.243 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:18:27.243 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:18:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:18:27.289 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:18:27.290 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:27.290 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:27.291 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.333 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:18:27.333 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:18:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:27.334 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:27.335 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:27.336 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 59, 2021-04-09T18:18:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:27.337 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:27.338 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.374 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.375 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:18:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:18:27.377 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:27.396 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:27.418 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.419 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:18:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:27.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.421 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:18:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:27.422 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.423 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:27.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.458 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:18:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:27.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.460 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:18:27.461 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:27.472 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.475 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:18:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:27.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.481 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:18:27.494 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:18:27.495 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.502 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.502 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:18:27.503 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:18:27.503 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:18:27.504 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:18:27.507 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:27.510 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:27.510 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:27.511 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.518 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:18:27.528 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:18:27.529 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:27.529 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:27.530 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:27.530 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:27.530 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:27.531 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:27.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:27.532 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 52, 2021-04-09T18:18:27.532 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:27.532 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:27.533 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:27.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.543 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:18:27.544 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:27.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:18:27.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:27.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.547 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.555 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:27.580 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.581 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.581 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:18:27.582 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:27.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.583 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.584 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:18:27.584 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:27.584 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.585 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.585 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:27.586 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.587 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.607 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:18:27.608 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:27.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.609 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.609 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:18:27.610 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:27.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.638 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:27.638 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.642 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:18:27.642 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:27.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.644 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:18:27.644 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:18:27.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.645 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.646 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:18:27.646 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:18:27.647 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:18:27.672 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:18:27.672 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:27.673 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:27.673 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:27.674 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.675 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:18:27.676 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:27.677 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:27.678 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:27.679 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:27.683 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:27.684 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:27.684 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:27.684 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:27.685 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.685 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.686 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:18:27.686 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:27.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.687 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.688 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:18:27.688 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:27.688 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.689 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.689 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:27.690 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.691 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:18:27.691 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:27.691 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.700 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:18:27.700 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:27.700 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.701 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.702 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:27.702 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:18:27.705 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:27.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.706 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:18:27.707 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:27.707 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.709 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:27.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.711 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:18:27.711 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:27.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.712 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.713 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:18:27.713 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:18:27.714 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:18:27.716 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:18:27.717 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:18:27.717 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:27.717 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:27.718 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:27.718 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.720 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:18:27.722 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:18:27.722 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:27.722 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:27.723 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:27.723 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:27.723 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:27.724 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:27.724 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:27.725 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:27.726 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:27.726 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:27.726 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:27.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:18:27.729 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:27.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:18:27.731 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:27.731 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.732 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.732 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:27.733 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.734 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.734 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:18:27.734 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:27.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.738 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:18:27.739 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:27.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.740 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.741 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:27.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.743 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:18:27.743 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:27.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.744 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:18:27.745 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:27.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.746 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.756 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:27.756 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:18:27.760 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:27.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.762 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:18:27.762 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:18:27.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.763 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.763 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:18:27.764 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:18:27.764 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:18:27.764 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:18:27.765 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:27.765 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:27.766 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:27.766 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.767 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:18:27.776 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:18:27.777 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:27.777 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:27.778 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:27.782 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:27.782 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:27.783 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:27.783 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:27.783 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:27.784 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:27.784 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:27.785 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:27.785 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.786 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.786 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:18:27.787 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:27.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.800 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:18:27.801 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:27.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.801 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.802 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:27.803 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.804 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.804 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:18:27.804 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:27.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.806 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.806 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:18:27.806 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:27.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.807 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.808 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:27.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.810 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:18:27.810 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:27.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.835 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.835 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:18:27.835 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:27.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.836 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.855 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:27.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.857 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.857 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:18:27.857 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:27.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.861 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.861 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:18:27.861 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:18:27.862 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.862 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.863 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:18:27.864 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:18:27.864 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:18:27.864 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:18:27.877 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:27.877 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:27.877 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:27.878 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.879 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:18:27.879 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:18:27.880 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:18:27.880 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:18:27.880 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:18:27.881 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:18:27.881 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:18:27.882 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:18:27.904 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:18:27.904 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:18:27.904 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:18:27.905 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:18:27.905 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:18:27.905 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:18:27.906 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:27.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.907 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:18:27.907 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:27.908 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.908 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.908 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:18:27.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:18:27.909 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:27.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.910 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.910 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:18:27.911 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:27.911 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.911 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.912 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:18:27.912 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.913 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:18:27.913 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:27.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.955 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:18:27.956 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:27.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.956 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.957 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:18:27.957 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:18:27.957 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:18:27.958 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:18:27.958 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:27.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.958 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:27.959 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:18:27.959 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:18:27.961 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.961 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.962 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:18:27.962 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:18:27.962 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:18:27.962 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:18:27.963 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:18:27.963 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:18:27.963 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:18:27.963 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:27.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:27.964 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:18:27.964 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:18:27.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.964 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:18:27.965 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:27.965 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:27.965 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:27.965 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:27.965 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:27.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.966 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:18:27.966 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:27.967 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:27.967 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:27.967 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:27.967 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:27.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:18:27.968 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:27.968 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:27.969 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:27.969 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:27.969 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:27.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.970 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:18:27.970 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:27.970 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:27.970 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:27.982 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:27.983 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:27.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.984 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:18:27.993 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:27.993 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:27.993 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:27.994 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:27.994 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:27.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:27.997 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:18:27.997 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:27.997 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:27.998 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:27.998 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:27.998 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:27.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:27.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:28.003 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:18:28.004 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:28.004 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:28.004 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:28.005 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:28.005 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:28.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:28.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:28.007 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:18:28.007 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:18:28.008 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:18:28.008 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:18:28.008 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:18:28.009 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:18:28.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:28.036 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:18:28.037 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:18:28.037 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:18:28.037 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:18:28.038 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:18:28.057 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:18:28.057 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/3097053176": "2021-04-10 18:18:19.599506", 2021-04-09T18:18:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/3097053176": "2021-04-10 18:18:19.599506", 2021-04-09T18:18:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/2067133784": "2021-04-10 18:18:19.599501", 2021-04-09T18:18:28.058 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/1080414756": "2021-04-10 18:18:19.599487", 2021-04-09T18:18:28.059 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/2067133784": "2021-04-10 18:18:19.599501", 2021-04-09T18:18:28.059 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/1080414756": "2021-04-10 18:18:19.599487" 2021-04-09T18:18:28.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:28.075 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:18:28.076 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:18:28.076 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:18:28.077 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:18:28.077 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:18:28.078 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:18:28.079 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:18:28.079 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:18:28.079 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:28.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:28.080 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:18:28.081 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:18:28.081 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:18:28.081 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:18:28.082 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:18:28.082 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0bc018930 msgr2=0x7fc0bc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0bc018930 0x7fc0bc01ae10 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc0b42d0 msgr2=0x7fc0dc183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc0b42d0 0x7fc0dc183520 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7fc0cc009980 tx=0x7fc0cc00bcd0).stop 2021-04-09T18:18:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 shutdown_connections 2021-04-09T18:18:28.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0bc018930 0x7fc0bc01ae10 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0dc184d30 0x7fc0dc185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0dc1ac090 0x7fc0dc1ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 --2- 172.21.15.32:0/3818393551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0dc0b42d0 0x7fc0dc183520 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.718 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 >> 172.21.15.32:0/3818393551 conn(0x7fc0dc19b890 msgr2=0x7fc0dc1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.719 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 shutdown_connections 2021-04-09T18:18:28.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:26.719 7fc0e1e40700 1 -- 172.21.15.32:0/3818393551 wait complete. 2021-04-09T18:18:28.149 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:28.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.267 7faadb90b700 1 Processor -- start 2021-04-09T18:18:28.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.267 7faadb90b700 1 -- start start 2021-04-09T18:18:28.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faadb90b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faad410d010 0x7faad40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faadb90b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faad40d1da0 0x7faad40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faadb90b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad40d82c0 0x7faad410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faadb90b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad40d8d30 con 0x7faad410d010 2021-04-09T18:18:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faadb90b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad40d9220 con 0x7faad40d1da0 2021-04-09T18:18:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faadb90b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad410d4b0 con 0x7faad40d82c0 2021-04-09T18:18:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faad9727700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faad410d010 0x7faad40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faad9727700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faad410d010 0x7faad40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46478/0 (socket says 172.21.15.32:46478) 2021-04-09T18:18:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faad9727700 1 -- 172.21.15.32:0/36758808 learned_addr learned my addr 172.21.15.32:0/36758808 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:28.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faad9f28700 1 --2- 172.21.15.32:0/36758808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad40d82c0 0x7faad410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faad8f26700 1 --2- 172.21.15.32:0/36758808 >> v2:172.21.15.86:3300/0 conn(0x7faad40d1da0 0x7faad40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.268 7faad9727700 1 -- 172.21.15.32:0/36758808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad40d82c0 msgr2=0x7faad410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:28.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faad9727700 1 --2- 172.21.15.32:0/36758808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad40d82c0 0x7faad410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faad9727700 1 -- 172.21.15.32:0/36758808 >> v2:172.21.15.86:3300/0 conn(0x7faad40d1da0 msgr2=0x7faad40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faad9727700 1 --2- 172.21.15.32:0/36758808 >> v2:172.21.15.86:3300/0 conn(0x7faad40d1da0 0x7faad40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faad9727700 1 -- 172.21.15.32:0/36758808 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad40d86c0 con 0x7faad410d010 2021-04-09T18:18:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faad9727700 1 --2- 172.21.15.32:0/36758808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad410d010 0x7faad40d1780 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7faad0006700 tx=0x7faad0009b00).ready entity=mon.0 client_cookie=97cfc242327895fd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faacbfff700 1 -- 172.21.15.32:0/36758808 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad000c0c0 con 0x7faad410d010 2021-04-09T18:18:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faacbfff700 1 -- 172.21.15.32:0/36758808 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad000c400 con 0x7faad410d010 2021-04-09T18:18:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faacbfff700 1 -- 172.21.15.32:0/36758808 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad000c8e0 con 0x7faad410d010 2021-04-09T18:18:28.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faadb90b700 1 -- 172.21.15.32:0/36758808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad410d010 msgr2=0x7faad40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.269 7faadb90b700 1 --2- 172.21.15.32:0/36758808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad410d010 0x7faad40d1780 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7faad0006700 tx=0x7faad0009b00).stop 2021-04-09T18:18:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.270 7faadb90b700 1 -- 172.21.15.32:0/36758808 shutdown_connections 2021-04-09T18:18:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.270 7faadb90b700 1 --2- 172.21.15.32:0/36758808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad40d82c0 0x7faad410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.270 7faadb90b700 1 --2- 172.21.15.32:0/36758808 >> v2:172.21.15.86:3300/0 conn(0x7faad40d1da0 0x7faad40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.270 7faadb90b700 1 --2- 172.21.15.32:0/36758808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad410d010 0x7faad40d1780 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.270 7faadb90b700 1 -- 172.21.15.32:0/36758808 >> 172.21.15.32:0/36758808 conn(0x7faad40ce5f0 msgr2=0x7faad40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:28.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.270 7faadb90b700 1 -- 172.21.15.32:0/36758808 shutdown_connections 2021-04-09T18:18:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.270 7faadb90b700 1 -- 172.21.15.32:0/36758808 wait complete. 2021-04-09T18:18:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.290 7faadb90b700 1 Processor -- start 2021-04-09T18:18:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.290 7faadb90b700 1 -- start start 2021-04-09T18:18:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faadb90b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad41ac380 0x7faad41ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faadb90b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad4186c90 0x7faad4187090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faadb90b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad4185530 0x7faad4185930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faadb90b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad4186790 con 0x7faad4185530 2021-04-09T18:18:28.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faadb90b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad41acda0 con 0x7faad41ac380 2021-04-09T18:18:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faadb90b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad418e100 con 0x7faad4186c90 2021-04-09T18:18:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9f28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad4185530 0x7faad4185930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9f28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad4185530 0x7faad4185930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46484/0 (socket says 172.21.15.32:46484) 2021-04-09T18:18:28.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9f28700 1 -- 172.21.15.32:0/2989556918 learned_addr learned my addr 172.21.15.32:0/2989556918 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:28.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9727700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad41ac380 0x7faad41ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad8f26700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad4186c90 0x7faad4187090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9f28700 1 -- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad4186c90 msgr2=0x7faad4187090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9f28700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad4186c90 0x7faad4187090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9f28700 1 -- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad41ac380 msgr2=0x7faad41ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9f28700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad41ac380 0x7faad41ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9f28700 1 -- 172.21.15.32:0/2989556918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad41876b0 con 0x7faad4185530 2021-04-09T18:18:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faad9f28700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad4185530 0x7faad4185930 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7faacc00b970 tx=0x7faacc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.291 7faac9ffb700 1 -- 172.21.15.32:0/2989556918 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faacc000f90 con 0x7faad4185530 2021-04-09T18:18:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.292 7faac9ffb700 1 -- 172.21.15.32:0/2989556918 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faacc006aa0 con 0x7faad4185530 2021-04-09T18:18:28.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.292 7faadb90b700 1 -- 172.21.15.32:0/2989556918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad41876b0 con 0x7faad4185530 2021-04-09T18:18:28.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.292 7faac9ffb700 1 -- 172.21.15.32:0/2989556918 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faacc001470 con 0x7faad4185530 2021-04-09T18:18:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.292 7faadb90b700 1 -- 172.21.15.32:0/2989556918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad41876b0 con 0x7faad4185530 2021-04-09T18:18:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.292 7faac9ffb700 1 -- 172.21.15.32:0/2989556918 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faacc0011f0 con 0x7faad4185530 2021-04-09T18:18:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.292 7faac9ffb700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab4018930 0x7faab401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.292 7faac9ffb700 1 -- 172.21.15.32:0/2989556918 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5015+0+0 (secure 0 0 0) 0x7faacc032b40 con 0x7faad4185530 2021-04-09T18:18:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.292 7faad9727700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab4018930 0x7faab401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.293 7faad9727700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab4018930 0x7faab401ae10 crc :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:28.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.312 7faadb90b700 1 -- 172.21.15.32:0/2989556918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad41acda0 con 0x7faad4185530 2021-04-09T18:18:28.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.315 7faac9ffb700 1 -- 172.21.15.32:0/2989556918 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faacc0066f0 con 0x7faad4185530 2021-04-09T18:18:28.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.507 7faadb90b700 1 -- 172.21.15.32:0/2989556918 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faad41acda0 con 0x7faad4185530 2021-04-09T18:18:28.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.507 7faac9ffb700 1 -- 172.21.15.32:0/2989556918 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+1843 (secure 0 0 0) 0x7faacc002f20 con 0x7faad4185530 2021-04-09T18:18:28.546 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 86 2021-04-09T18:18:28.547 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:28.547 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":86,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/531351568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":531351568},{"type":"v1","addr":"172.21.15.86:6835","nonce":531351568}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":86},{"gid":6457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3677210434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3677210434},{"type":"v1","addr":"172.21.15.32:6833","nonce":3677210434}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":86},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/185947558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":185947558},{"type":"v1","addr":"172.21.15.32:6835","nonce":185947558}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":86},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[]} 2021-04-09T18:18:28.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 -- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab4018930 msgr2=0x7faab401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab4018930 0x7faab401ae10 crc :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 -- 172.21.15.32:0/2989556918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad4185530 msgr2=0x7faad4185930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad4185530 0x7faad4185930 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7faacc00b970 tx=0x7faacc00bd10).stop 2021-04-09T18:18:28.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 -- 172.21.15.32:0/2989556918 shutdown_connections 2021-04-09T18:18:28.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab4018930 0x7faab401ae10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad4185530 0x7faad4185930 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad4186c90 0x7faad4187090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 --2- 172.21.15.32:0/2989556918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad41ac380 0x7faad41ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 -- 172.21.15.32:0/2989556918 >> 172.21.15.32:0/2989556918 conn(0x7faad419b890 msgr2=0x7faad41a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:28.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 -- 172.21.15.32:0/2989556918 shutdown_connections 2021-04-09T18:18:28.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.510 7faadb90b700 1 -- 172.21.15.32:0/2989556918 wait complete. 2021-04-09T18:18:28.561 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:28.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.679 7faecffb7700 1 Processor -- start 2021-04-09T18:18:28.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.679 7faecffb7700 1 -- start start 2021-04-09T18:18:28.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.679 7faecffb7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faec80cffa0 0x7faec80d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.679 7faecffb7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faec80d7800 0x7faec80d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.679 7faecffb7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80d8220 0x7faec80d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.679 7faecffb7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec80d91e0 con 0x7faec80d7800 2021-04-09T18:18:28.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.679 7faecffb7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec80d7280 con 0x7faec80cffa0 2021-04-09T18:18:28.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.679 7faecffb7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec80d6290 con 0x7faec80d8220 2021-04-09T18:18:28.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecd5d2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faec80d7800 0x7faec80d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecd5d2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faec80d7800 0x7faec80d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46494/0 (socket says 172.21.15.32:46494) 2021-04-09T18:18:28.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecd5d2700 1 -- 172.21.15.32:0/3613343038 learned_addr learned my addr 172.21.15.32:0/3613343038 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:28.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecddd3700 1 --2- 172.21.15.32:0/3613343038 >> v2:172.21.15.86:3300/0 conn(0x7faec80cffa0 0x7faec80d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faece5d4700 1 --2- 172.21.15.32:0/3613343038 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80d8220 0x7faec80d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecd5d2700 1 -- 172.21.15.32:0/3613343038 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80d8220 msgr2=0x7faec80d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecd5d2700 1 --2- 172.21.15.32:0/3613343038 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80d8220 0x7faec80d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecd5d2700 1 -- 172.21.15.32:0/3613343038 >> v2:172.21.15.86:3300/0 conn(0x7faec80cffa0 msgr2=0x7faec80d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:28.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecd5d2700 1 --2- 172.21.15.32:0/3613343038 >> v2:172.21.15.86:3300/0 conn(0x7faec80cffa0 0x7faec80d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecd5d2700 1 -- 172.21.15.32:0/3613343038 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec80d6480 con 0x7faec80d7800 2021-04-09T18:18:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecddd3700 1 --2- 172.21.15.32:0/3613343038 >> v2:172.21.15.86:3300/0 conn(0x7faec80cffa0 0x7faec80d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecd5d2700 1 --2- 172.21.15.32:0/3613343038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec80d7800 0x7faec80d7c00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7faebc006700 tx=0x7faebc009b80).ready entity=mon.0 client_cookie=68208fc342fc0a6b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faeccdd1700 1 -- 172.21.15.32:0/3613343038 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faebc00c0c0 con 0x7faec80d7800 2021-04-09T18:18:28.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faeccdd1700 1 -- 172.21.15.32:0/3613343038 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc00c440 con 0x7faec80d7800 2021-04-09T18:18:28.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faeccdd1700 1 -- 172.21.15.32:0/3613343038 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faebc00c920 con 0x7faec80d7800 2021-04-09T18:18:28.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecffb7700 1 -- 172.21.15.32:0/3613343038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec80d7800 msgr2=0x7faec80d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.680 7faecffb7700 1 --2- 172.21.15.32:0/3613343038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec80d7800 0x7faec80d7c00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7faebc006700 tx=0x7faebc009b80).stop 2021-04-09T18:18:28.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.681 7faecffb7700 1 -- 172.21.15.32:0/3613343038 shutdown_connections 2021-04-09T18:18:28.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.681 7faecffb7700 1 --2- 172.21.15.32:0/3613343038 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80d8220 0x7faec80d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.681 7faecffb7700 1 --2- 172.21.15.32:0/3613343038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec80d7800 0x7faec80d7c00 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.681 7faecffb7700 1 --2- 172.21.15.32:0/3613343038 >> v2:172.21.15.86:3300/0 conn(0x7faec80cffa0 0x7faec80d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.681 7faecffb7700 1 -- 172.21.15.32:0/3613343038 >> 172.21.15.32:0/3613343038 conn(0x7faec80ce5c0 msgr2=0x7faec80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:28.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.681 7faecffb7700 1 -- 172.21.15.32:0/3613343038 shutdown_connections 2021-04-09T18:18:28.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.681 7faecffb7700 1 -- 172.21.15.32:0/3613343038 wait complete. 2021-04-09T18:18:28.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.701 7faecffb7700 1 Processor -- start 2021-04-09T18:18:28.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.701 7faecffb7700 1 -- start start 2021-04-09T18:18:28.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecffb7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec81acff0 0x7faec81847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecffb7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec8183fc0 0x7faec81a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecffb7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec8186ff0 0x7faec81873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecffb7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec8188c30 con 0x7faec8183fc0 2021-04-09T18:18:28.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecffb7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec81843c0 con 0x7faec8186ff0 2021-04-09T18:18:28.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecffb7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec81a5c20 con 0x7faec81acff0 2021-04-09T18:18:28.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecd5d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec8183fc0 0x7faec81a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecd5d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec8183fc0 0x7faec81a5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46498/0 (socket says 172.21.15.32:46498) 2021-04-09T18:18:28.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecd5d2700 1 -- 172.21.15.32:0/729038571 learned_addr learned my addr 172.21.15.32:0/729038571 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:28.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faece5d4700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec8186ff0 0x7faec81873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecddd3700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec81acff0 0x7faec81847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecd5d2700 1 -- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec81acff0 msgr2=0x7faec81847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecd5d2700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec81acff0 0x7faec81847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecd5d2700 1 -- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec8186ff0 msgr2=0x7faec81873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.702 7faecd5d2700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec8186ff0 0x7faec81873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faecd5d2700 1 -- 172.21.15.32:0/729038571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec8187a10 con 0x7faec8183fc0 2021-04-09T18:18:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faecddd3700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec81acff0 0x7faec81847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:28.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faece5d4700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec8186ff0 0x7faec81873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:28.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faecd5d2700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec8183fc0 0x7faec81a5600 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7faebc00b650 tx=0x7faebc00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:28.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faeba7fc700 1 -- 172.21.15.32:0/729038571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faebc00bde0 con 0x7faec8183fc0 2021-04-09T18:18:28.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faeba7fc700 1 -- 172.21.15.32:0/729038571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc000c90 con 0x7faec8183fc0 2021-04-09T18:18:28.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faecffb7700 1 -- 172.21.15.32:0/729038571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec8187a10 con 0x7faec8183fc0 2021-04-09T18:18:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faeba7fc700 1 -- 172.21.15.32:0/729038571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faebc002220 con 0x7faec8183fc0 2021-04-09T18:18:28.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faecffb7700 1 -- 172.21.15.32:0/729038571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec8187a10 con 0x7faec8183fc0 2021-04-09T18:18:28.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.703 7faeba7fc700 1 -- 172.21.15.32:0/729038571 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faebc002020 con 0x7faec8183fc0 2021-04-09T18:18:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.704 7faeba7fc700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faeb0018930 0x7faeb001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:28.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.704 7faeba7fc700 1 -- 172.21.15.32:0/729038571 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5015+0+0 (secure 0 0 0) 0x7faebc0327b0 con 0x7faec8183fc0 2021-04-09T18:18:28.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.704 7faecddd3700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faeb0018930 0x7faeb001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:28.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.704 7faecddd3700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faeb0018930 0x7faeb001ae10 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:28.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.724 7faecffb7700 1 -- 172.21.15.32:0/729038571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec8182dc0 con 0x7faec8183fc0 2021-04-09T18:18:28.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.727 7faeba7fc700 1 -- 172.21.15.32:0/729038571 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faebc0091e0 con 0x7faec8183fc0 2021-04-09T18:18:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.918 7faecffb7700 1 -- 172.21.15.32:0/729038571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faec80d7280 con 0x7faec8183fc0 2021-04-09T18:18:28.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.919 7faeba7fc700 1 -- 172.21.15.32:0/729038571 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+1843 (secure 0 0 0) 0x7faebc00bde0 con 0x7faec8183fc0 2021-04-09T18:18:28.956 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 86 2021-04-09T18:18:28.957 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:28.958 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":86,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6454,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/531351568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":531351568},{"type":"v1","addr":"172.21.15.86:6835","nonce":531351568}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":86},{"gid":6457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/3677210434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3677210434},{"type":"v1","addr":"172.21.15.32:6833","nonce":3677210434}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":86},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/185947558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":185947558},{"type":"v1","addr":"172.21.15.32:6835","nonce":185947558}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":86},{"gid":55577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/713422835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":713422835},{"type":"v1","addr":"172.21.15.86:6837","nonce":713422835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":76}],"filesystems":[]} 2021-04-09T18:18:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 -- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faeb0018930 msgr2=0x7faeb001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faeb0018930 0x7faeb001ae10 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 -- 172.21.15.32:0/729038571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec8183fc0 msgr2=0x7faec81a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:28.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec8183fc0 0x7faec81a5600 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7faebc00b650 tx=0x7faebc00b9f0).stop 2021-04-09T18:18:28.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 -- 172.21.15.32:0/729038571 shutdown_connections 2021-04-09T18:18:28.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faeb0018930 0x7faeb001ae10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faec8183fc0 0x7faec81a5600 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec8186ff0 0x7faec81873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 --2- 172.21.15.32:0/729038571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec81acff0 0x7faec81847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:28.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 -- 172.21.15.32:0/729038571 >> 172.21.15.32:0/729038571 conn(0x7faec809abb0 msgr2=0x7faec80a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:28.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 -- 172.21.15.32:0/729038571 shutdown_connections 2021-04-09T18:18:28.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:28.921 7faecffb7700 1 -- 172.21.15.32:0/729038571 wait complete. 2021-04-09T18:18:28.969 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:18:28.969 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:18:28.970 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:18:28.970 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:18:28.996 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:18:28.997 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:18:29.001 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:18:29.002 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:18:29.002 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:18:29.002 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:18:29.003 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:18:29.024 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:18:29.027 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:18:29.027 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:18:29.035 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:18:29.035 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:18:29.039 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:18:29.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:18:29.040 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:18:29.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:18:29.043 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:18:29.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:18:29.088 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:18:29.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:18:29.099 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:18:29.103 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:18:29.103 DEBUG:teuthology.parallel:result is None 2021-04-09T18:18:29.104 DEBUG:teuthology.parallel:result is None 2021-04-09T18:18:29.129 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:18:29.129 DEBUG:teuthology.parallel:result is None 2021-04-09T18:18:29.132 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:18:29.133 DEBUG:teuthology.parallel:result is None 2021-04-09T18:18:29.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:18:29.243 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:18:29.243 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:18:29.244 INFO:tasks.ceph.mds.a.smithi032.stderr:2021-04-09 18:18:29.207 7fa99b826700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2021-04-09T18:18:29.273 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:18:29.274 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:18:29.274 INFO:tasks.ceph.mds.d.smithi086.stderr:2021-04-09 18:18:29.236 7f2b8ab32700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2021-04-09T18:18:29.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.260 7f5fae3c4700 1 Processor -- start 2021-04-09T18:18:29.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.261 7f5fae3c4700 1 -- start start 2021-04-09T18:18:29.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.261 7f5fae3c4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5fa80cffb0 0x7f5fa80d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.261 7f5fae3c4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5fa80d7810 0x7f5fa80d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.261 7f5fae3c4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa80d7290 con 0x7f5fa80d8230 2021-04-09T18:18:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.261 7f5fae3c4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa80d91f0 con 0x7f5fa80d7810 2021-04-09T18:18:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.261 7f5fae3c4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa810dff0 con 0x7f5fa80cffb0 2021-04-09T18:18:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.261 7f5f9ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5fa80d7810 0x7f5fa80d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.261 7f5fac9e1700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5fa80d8230 0x7f5fa80d5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56122/0 (socket says 172.21.15.32:56122) 2021-04-09T18:18:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.261 7f5fac9e1700 1 -- 172.21.15.32:0/2876028589 learned_addr learned my addr 172.21.15.32:0/2876028589 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5fa7fff700 1 --2- 172.21.15.32:0/2876028589 >> v2:172.21.15.86:3300/0 conn(0x7f5fa80cffb0 0x7f5fa80d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5fa77fe700 1 -- 172.21.15.32:0/2876028589 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983152206 0 0) 0x7f5fa0000d30 con 0x7f5fa80d8230 2021-04-09T18:18:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5fa77fe700 1 -- 172.21.15.32:0/2876028589 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f94002fe0 con 0x7f5fa80d8230 2021-04-09T18:18:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5f9ffff700 1 -- 172.21.15.32:0/2876028589 >> v2:172.21.15.86:3300/0 conn(0x7f5fa80cffb0 msgr2=0x7f5fa80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5f9ffff700 1 --2- 172.21.15.32:0/2876028589 >> v2:172.21.15.86:3300/0 conn(0x7f5fa80cffb0 0x7f5fa80d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5f9ffff700 1 -- 172.21.15.32:0/2876028589 >> v1:172.21.15.32:6789/0 conn(0x7f5fa80d8230 legacy=0x7f5fa80d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5f9ffff700 1 -- 172.21.15.32:0/2876028589 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa810e210 con 0x7f5fa80d7810 2021-04-09T18:18:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5fa7fff700 1 --2- 172.21.15.32:0/2876028589 >> v2:172.21.15.86:3300/0 conn(0x7f5fa80cffb0 0x7f5fa80d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5f9ffff700 1 --2- 172.21.15.32:0/2876028589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa80d7810 0x7f5fa80d7c10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f5f90006650 tx=0x7f5f90009a80).ready entity=mon.0 client_cookie=33fa8e5cbd477680 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:29.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5fa77fe700 1 -- 172.21.15.32:0/2876028589 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f900009b0 con 0x7f5fa80d7810 2021-04-09T18:18:29.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5fa77fe700 1 -- 172.21.15.32:0/2876028589 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9000c0c0 con 0x7f5fa80d7810 2021-04-09T18:18:29.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5fa77fe700 1 -- 172.21.15.32:0/2876028589 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f9000c510 con 0x7f5fa80d7810 2021-04-09T18:18:29.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5fae3c4700 1 -- 172.21.15.32:0/2876028589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa80d7810 msgr2=0x7f5fa80d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.262 7f5fae3c4700 1 --2- 172.21.15.32:0/2876028589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa80d7810 0x7f5fa80d7c10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f5f90006650 tx=0x7f5f90009a80).stop 2021-04-09T18:18:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.263 7f5fae3c4700 1 -- 172.21.15.32:0/2876028589 shutdown_connections 2021-04-09T18:18:29.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.263 7f5fae3c4700 1 --2- 172.21.15.32:0/2876028589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa80d7810 0x7f5fa80d7c10 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.263 7f5fae3c4700 1 --2- 172.21.15.32:0/2876028589 >> v2:172.21.15.86:3300/0 conn(0x7f5fa80cffb0 0x7f5fa80d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.263 7f5fae3c4700 1 -- 172.21.15.32:0/2876028589 >> 172.21.15.32:0/2876028589 conn(0x7f5fa80ce5d0 msgr2=0x7f5fa80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.263 7f5fae3c4700 1 -- 172.21.15.32:0/2876028589 shutdown_connections 2021-04-09T18:18:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.263 7f5fae3c4700 1 -- 172.21.15.32:0/2876028589 wait complete. 2021-04-09T18:18:29.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.283 7f5fae3c4700 1 Processor -- start 2021-04-09T18:18:29.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.283 7f5fae3c4700 1 -- start start 2021-04-09T18:18:29.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fae3c4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fa81a7dc0 0x7f5fa81a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fae3c4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5fa8184080 0x7f5fa8184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fae3c4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa8184aa0 0x7f5fa808f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fae3c4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa81a8b40 con 0x7f5fa8184aa0 2021-04-09T18:18:29.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fae3c4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa81a7870 con 0x7f5fa81a7dc0 2021-04-09T18:18:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fae3c4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa81a87e0 con 0x7f5fa8184080 2021-04-09T18:18:29.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fac9e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa8184aa0 0x7f5fa808f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fac9e1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa8184aa0 0x7f5fa808f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46538/0 (socket says 172.21.15.32:46538) 2021-04-09T18:18:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fac9e1700 1 -- 172.21.15.32:0/2265589418 learned_addr learned my addr 172.21.15.32:0/2265589418 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fa7fff700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fa81a7dc0 0x7f5fa81a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5f9ffff700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5fa8184080 0x7f5fa8184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fac9e1700 1 -- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5fa8184080 msgr2=0x7f5fa8184480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fac9e1700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5fa8184080 0x7f5fa8184480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fac9e1700 1 -- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fa81a7dc0 msgr2=0x7f5fa81a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fac9e1700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fa81a7dc0 0x7f5fa81a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fac9e1700 1 -- 172.21.15.32:0/2265589418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa808f810 con 0x7f5fa8184aa0 2021-04-09T18:18:29.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5f9ffff700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5fa8184080 0x7f5fa8184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:29.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fac9e1700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa8184aa0 0x7f5fa808f1f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f5fa000a900 tx=0x7f5fa000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:29.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fa57fa700 1 -- 172.21.15.32:0/2265589418 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fa0001270 con 0x7f5fa8184aa0 2021-04-09T18:18:29.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fa57fa700 1 -- 172.21.15.32:0/2265589418 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa00070d0 con 0x7f5fa8184aa0 2021-04-09T18:18:29.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa808f810 con 0x7f5fa8184aa0 2021-04-09T18:18:29.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fa57fa700 1 -- 172.21.15.32:0/2265589418 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fa0004720 con 0x7f5fa8184aa0 2021-04-09T18:18:29.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.284 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa808f810 con 0x7f5fa8184aa0 2021-04-09T18:18:29.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.285 7f5fa57fa700 1 -- 172.21.15.32:0/2265589418 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5fa00072b0 con 0x7f5fa8184aa0 2021-04-09T18:18:29.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.285 7f5fa57fa700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f8c018930 0x7f5f8c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.285 7f5fa57fa700 1 -- 172.21.15.32:0/2265589418 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5015+0+0 (secure 0 0 0) 0x7f5fa0032b20 con 0x7f5fa8184aa0 2021-04-09T18:18:29.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.285 7f5fa7fff700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f8c018930 0x7f5f8c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.286 7f5fa7fff700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f8c018930 0x7f5f8c01ae10 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:29.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.305 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa81a7840 con 0x7f5fa8184aa0 2021-04-09T18:18:29.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.308 7f5fa57fa700 1 -- 172.21.15.32:0/2265589418 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5fa0007cb0 con 0x7f5fa8184aa0 2021-04-09T18:18:29.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.511 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f5fa81a7840 con 0x7f5fa8184aa0 2021-04-09T18:18:29.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.626 7f5fa57fa700 1 -- 172.21.15.32:0/2265589418 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v78) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5fa00070d0 con 0x7f5fa8184aa0 2021-04-09T18:18:29.711 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:18:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f8c018930 msgr2=0x7f5f8c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f8c018930 0x7f5f8c01ae10 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa8184aa0 msgr2=0x7f5fa808f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa8184aa0 0x7f5fa808f1f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f5fa000a900 tx=0x7f5fa000aca0).stop 2021-04-09T18:18:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 shutdown_connections 2021-04-09T18:18:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f8c018930 0x7f5f8c01ae10 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5fa8184aa0 0x7f5fa808f1f0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5fa8184080 0x7f5fa8184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 --2- 172.21.15.32:0/2265589418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fa81a7dc0 0x7f5fa81a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 >> 172.21.15.32:0/2265589418 conn(0x7f5fa81a8f50 msgr2=0x7f5fa8188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 shutdown_connections 2021-04-09T18:18:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.677 7f5fae3c4700 1 -- 172.21.15.32:0/2265589418 wait complete. 2021-04-09T18:18:29.724 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:18:29.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:18:29.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.838 7f3e37d15700 1 Processor -- start 2021-04-09T18:18:29.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.838 7f3e37d15700 1 -- start start 2021-04-09T18:18:29.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.838 7f3e37d15700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3010d010 0x7f3e300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.838 7f3e37d15700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3e300d8490 0x7f3e3010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.838 7f3e37d15700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3010d480 con 0x7f3e300d1da0 2021-04-09T18:18:29.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.838 7f3e37d15700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3010e780 con 0x7f3e300d8490 2021-04-09T18:18:29.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.838 7f3e37d15700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300d8d30 con 0x7f3e3010d010 2021-04-09T18:18:29.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e36332700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3e300d8490 0x7f3e3010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e36332700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3e300d8490 0x7f3e3010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46546/0 (socket says 172.21.15.32:46546) 2021-04-09T18:18:29.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e36332700 1 -- 172.21.15.32:0/2553119278 learned_addr learned my addr 172.21.15.32:0/2553119278 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:29.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e35b31700 1 --2- 172.21.15.32:0/2553119278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3010d010 0x7f3e300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e36332700 1 -- 172.21.15.32:0/2553119278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3010d010 msgr2=0x7f3e300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:29.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e36332700 1 --2- 172.21.15.32:0/2553119278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3010d010 0x7f3e300d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e36332700 1 -- 172.21.15.32:0/2553119278 >> v1:172.21.15.86:6789/0 conn(0x7f3e300d1da0 legacy=0x7f3e300d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e34b2f700 1 -- 172.21.15.32:0/2553119278 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502362229 0 0) 0x7f3e20000c60 con 0x7f3e300d1da0 2021-04-09T18:18:29.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e36332700 1 -- 172.21.15.32:0/2553119278 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e300d9220 con 0x7f3e300d8490 2021-04-09T18:18:29.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e36332700 1 --2- 172.21.15.32:0/2553119278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300d8490 0x7f3e3010da00 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f3e28006f00 tx=0x7f3e28009f10).ready entity=mon.0 client_cookie=9656268c0aa32e38 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:29.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e34b2f700 1 -- 172.21.15.32:0/2553119278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e2800b3b0 con 0x7f3e300d8490 2021-04-09T18:18:29.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e34b2f700 1 -- 172.21.15.32:0/2553119278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2800b730 con 0x7f3e300d8490 2021-04-09T18:18:29.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e34b2f700 1 -- 172.21.15.32:0/2553119278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e2800bc10 con 0x7f3e300d8490 2021-04-09T18:18:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e37d15700 1 -- 172.21.15.32:0/2553119278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300d8490 msgr2=0x7f3e3010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.839 7f3e37d15700 1 --2- 172.21.15.32:0/2553119278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300d8490 0x7f3e3010da00 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f3e28006f00 tx=0x7f3e28009f10).stop 2021-04-09T18:18:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.840 7f3e37d15700 1 -- 172.21.15.32:0/2553119278 shutdown_connections 2021-04-09T18:18:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.840 7f3e37d15700 1 --2- 172.21.15.32:0/2553119278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300d8490 0x7f3e3010da00 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.840 7f3e37d15700 1 --2- 172.21.15.32:0/2553119278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3010d010 0x7f3e300d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.840 7f3e37d15700 1 -- 172.21.15.32:0/2553119278 >> 172.21.15.32:0/2553119278 conn(0x7f3e300ce5f0 msgr2=0x7f3e300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.840 7f3e37d15700 1 -- 172.21.15.32:0/2553119278 shutdown_connections 2021-04-09T18:18:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.840 7f3e37d15700 1 -- 172.21.15.32:0/2553119278 wait complete. 2021-04-09T18:18:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.860 7f3e37d15700 1 Processor -- start 2021-04-09T18:18:29.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.860 7f3e37d15700 1 -- start start 2021-04-09T18:18:29.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e37d15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e301a5210 0x7f3e301a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e37d15700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301a6340 0x7f3e301a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e37d15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300abaa0 0x7f3e300abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e37d15700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e301a6d60 con 0x7f3e300abaa0 2021-04-09T18:18:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e37d15700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300ac4c0 con 0x7f3e301a5210 2021-04-09T18:18:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e37d15700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e300ac6e0 con 0x7f3e301a6340 2021-04-09T18:18:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e36332700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300abaa0 0x7f3e300abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e36332700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300abaa0 0x7f3e300abea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46550/0 (socket says 172.21.15.32:46550) 2021-04-09T18:18:29.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e36332700 1 -- 172.21.15.32:0/2361131185 learned_addr learned my addr 172.21.15.32:0/2361131185 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:29.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e35b31700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e301a5210 0x7f3e301a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e35330700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301a6340 0x7f3e301a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:29.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e36332700 1 -- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301a6340 msgr2=0x7f3e301a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e36332700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301a6340 0x7f3e301a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e36332700 1 -- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e301a5210 msgr2=0x7f3e301a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:29.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e36332700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e301a5210 0x7f3e301a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:29.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e36332700 1 -- 172.21.15.32:0/2361131185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3017a720 con 0x7f3e300abaa0 2021-04-09T18:18:29.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e35330700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301a6340 0x7f3e301a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e35b31700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e301a5210 0x7f3e301a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.861 7f3e36332700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300abaa0 0x7f3e300abea0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f3e2800b9b0 tx=0x7f3e2800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.862 7f3e267fc700 1 -- 172.21.15.32:0/2361131185 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e28003030 con 0x7f3e300abaa0 2021-04-09T18:18:30.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.862 7f3e267fc700 1 -- 172.21.15.32:0/2361131185 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2800c2f0 con 0x7f3e300abaa0 2021-04-09T18:18:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.862 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3017a720 con 0x7f3e300abaa0 2021-04-09T18:18:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.862 7f3e267fc700 1 -- 172.21.15.32:0/2361131185 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e2800c740 con 0x7f3e300abaa0 2021-04-09T18:18:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.862 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3017a720 con 0x7f3e300abaa0 2021-04-09T18:18:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.862 7f3e267fc700 1 -- 172.21.15.32:0/2361131185 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3e28001270 con 0x7f3e300abaa0 2021-04-09T18:18:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.862 7f3e267fc700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e18018930 0x7f3e1801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:30.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.862 7f3e267fc700 1 -- 172.21.15.32:0/2361131185 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f3e28032a70 con 0x7f3e300abaa0 2021-04-09T18:18:30.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.863 7f3e35b31700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e18018930 0x7f3e1801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:30.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.863 7f3e35b31700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e18018930 0x7f3e1801ae10 crc :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:30.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.883 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e301b7070 con 0x7f3e300abaa0 2021-04-09T18:18:30.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:29.886 7f3e267fc700 1 -- 172.21.15.32:0/2361131185 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3e28000f90 con 0x7f3e300abaa0 2021-04-09T18:18:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.072 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3e3010d480 con 0x7f3e300abaa0 2021-04-09T18:18:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.072 7f3e267fc700 1 -- 172.21.15.32:0/2361131185 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f3e28001270 con 0x7f3e300abaa0 2021-04-09T18:18:30.114 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:30.115 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:30.115 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:18:30.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.115 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:18:30.116 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:18:30.116 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.116 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:18:30.116 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.116 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.117 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.118 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:18:30.118 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:18:30.118 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.118 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:18:30.119 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.119 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.119 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.119 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.120 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:18:30.121 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:18:30.121 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.121 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:18:30.122 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.122 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.122 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.122 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.123 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:18:30.123 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:18:30.123 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.124 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:18:30.124 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.124 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.124 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.124 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.125 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:18:30.125 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:18:30.126 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.126 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.142 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.143 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.143 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.144 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:18:30.144 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:18:30.144 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.144 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.144 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.145 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.145 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.145 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:18:30.146 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:18:30.146 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.146 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.172 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.172 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.173 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:18:30.173 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:18:30.173 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.173 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.174 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.174 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.175 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:18:30.175 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:18:30.176 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.177 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.177 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.179 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:18:30.181 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:18:30.182 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.182 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.183 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.183 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.185 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:18:30.191 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:18:30.192 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.193 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.193 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.193 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.195 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:18:30.195 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:18:30.195 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.196 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.197 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.197 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.198 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:18:30.199 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:18:30.205 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.205 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:18:30.206 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:18:30.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:18:30.225 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.226 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:18:30.235 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:18:30.235 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.236 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:18:30.236 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:18:30.237 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:18:30.237 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.256 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:18:30.256 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:18:30.256 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.257 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:18:30.257 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:18:30.262 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:18:30.263 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.264 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:18:30.264 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:18:30.265 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.265 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:18:30.266 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:18:30.266 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.267 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:30.267 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:30.268 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:18:30.268 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:18:30.269 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:18:30.269 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:18:30.269 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:18:30.300 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:18:30.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:30.312 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:18:30.312 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:30.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e18018930 msgr2=0x7f3e1801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:30.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e18018930 0x7f3e1801ae10 crc :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300abaa0 msgr2=0x7f3e300abea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:30.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300abaa0 0x7f3e300abea0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f3e2800b9b0 tx=0x7f3e2800bd50).stop 2021-04-09T18:18:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 shutdown_connections 2021-04-09T18:18:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e18018930 0x7f3e1801ae10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e300abaa0 0x7f3e300abea0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301a6340 0x7f3e301a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 --2- 172.21.15.32:0/2361131185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e301a5210 0x7f3e301a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 >> 172.21.15.32:0/2361131185 conn(0x7f3e301a99c0 msgr2=0x7f3e3017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:30.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 shutdown_connections 2021-04-09T18:18:30.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.075 7f3e37d15700 1 -- 172.21.15.32:0/2361131185 wait complete. 2021-04-09T18:18:30.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:18:30.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.452 7f00d7c38700 1 Processor -- start 2021-04-09T18:18:30.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.453 7f00d7c38700 1 -- start start 2021-04-09T18:18:30.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.453 7f00d7c38700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f00d010d010 0x7f00d00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:30.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.453 7f00d7c38700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00d00d1da0 0x7f00d00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.453 7f00d7c38700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d00d82c0 0x7f00d010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.453 7f00d7c38700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d00d8d30 con 0x7f00d00d1da0 2021-04-09T18:18:30.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.453 7f00d7c38700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d00d9220 con 0x7f00d010d010 2021-04-09T18:18:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.453 7f00d7c38700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d010d4b0 con 0x7f00d00d82c0 2021-04-09T18:18:30.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.453 7f00d5253700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00d00d1da0 0x7f00d00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:30.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5253700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00d00d1da0 0x7f00d00d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46560/0 (socket says 172.21.15.32:46560) 2021-04-09T18:18:30.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5253700 1 -- 172.21.15.32:0/635037876 learned_addr learned my addr 172.21.15.32:0/635037876 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:30.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5a54700 1 --2- 172.21.15.32:0/635037876 >> v2:172.21.15.86:3300/0 conn(0x7f00d010d010 0x7f00d00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:30.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d6255700 1 --2- 172.21.15.32:0/635037876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d00d82c0 0x7f00d010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:30.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5253700 1 -- 172.21.15.32:0/635037876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d00d82c0 msgr2=0x7f00d010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5253700 1 --2- 172.21.15.32:0/635037876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d00d82c0 0x7f00d010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5253700 1 -- 172.21.15.32:0/635037876 >> v2:172.21.15.86:3300/0 conn(0x7f00d010d010 msgr2=0x7f00d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5253700 1 --2- 172.21.15.32:0/635037876 >> v2:172.21.15.86:3300/0 conn(0x7f00d010d010 0x7f00d00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5253700 1 -- 172.21.15.32:0/635037876 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d00d86c0 con 0x7f00d00d1da0 2021-04-09T18:18:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d6255700 1 --2- 172.21.15.32:0/635037876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d00d82c0 0x7f00d010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5a54700 1 --2- 172.21.15.32:0/635037876 >> v2:172.21.15.86:3300/0 conn(0x7f00d010d010 0x7f00d00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d5253700 1 --2- 172.21.15.32:0/635037876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00d1da0 0x7f00d00d7ca0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f00c0006700 tx=0x7f00c0009b80).ready entity=mon.0 client_cookie=c664bd22e5eaaf76 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:30.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d4a52700 1 -- 172.21.15.32:0/635037876 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c000c0c0 con 0x7f00d00d1da0 2021-04-09T18:18:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d4a52700 1 -- 172.21.15.32:0/635037876 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c000c440 con 0x7f00d00d1da0 2021-04-09T18:18:30.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d4a52700 1 -- 172.21.15.32:0/635037876 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c000c920 con 0x7f00d00d1da0 2021-04-09T18:18:30.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d7c38700 1 -- 172.21.15.32:0/635037876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00d1da0 msgr2=0x7f00d00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:30.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.454 7f00d7c38700 1 --2- 172.21.15.32:0/635037876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00d1da0 0x7f00d00d7ca0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f00c0006700 tx=0x7f00c0009b80).stop 2021-04-09T18:18:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.455 7f00d7c38700 1 -- 172.21.15.32:0/635037876 shutdown_connections 2021-04-09T18:18:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.455 7f00d7c38700 1 --2- 172.21.15.32:0/635037876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d00d82c0 0x7f00d010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.455 7f00d7c38700 1 --2- 172.21.15.32:0/635037876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00d1da0 0x7f00d00d7ca0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.455 7f00d7c38700 1 --2- 172.21.15.32:0/635037876 >> v2:172.21.15.86:3300/0 conn(0x7f00d010d010 0x7f00d00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.455 7f00d7c38700 1 -- 172.21.15.32:0/635037876 >> 172.21.15.32:0/635037876 conn(0x7f00d00ce5f0 msgr2=0x7f00d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.455 7f00d7c38700 1 -- 172.21.15.32:0/635037876 shutdown_connections 2021-04-09T18:18:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.455 7f00d7c38700 1 -- 172.21.15.32:0/635037876 wait complete. 2021-04-09T18:18:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.475 7f00d7c38700 1 Processor -- start 2021-04-09T18:18:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.475 7f00d7c38700 1 -- start start 2021-04-09T18:18:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d7c38700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00b75a0 0x7f00d0191040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d7c38700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00d00a21f0 0x7f00d00a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:30.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d7c38700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d01a8930 0x7f00d01a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:30.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d7c38700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d01a9350 con 0x7f00d00b75a0 2021-04-09T18:18:30.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d7c38700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d0190c00 con 0x7f00d00a21f0 2021-04-09T18:18:30.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d7c38700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d01a72c0 con 0x7f00d01a8930 2021-04-09T18:18:30.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d5a54700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00b75a0 0x7f00d0191040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:30.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d5a54700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00b75a0 0x7f00d0191040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46564/0 (socket says 172.21.15.32:46564) 2021-04-09T18:18:30.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d5a54700 1 -- 172.21.15.32:0/740831050 learned_addr learned my addr 172.21.15.32:0/740831050 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:30.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d6255700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d01a8930 0x7f00d01a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:30.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d5a54700 1 -- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d01a8930 msgr2=0x7f00d01a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:30.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d5a54700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d01a8930 0x7f00d01a8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d5a54700 1 -- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00d00a21f0 msgr2=0x7f00d00a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:30.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d5253700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00d00a21f0 0x7f00d00a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:30.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d5a54700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00d00a21f0 0x7f00d00a25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.476 7f00d5a54700 1 -- 172.21.15.32:0/740831050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d01a74e0 con 0x7f00d00b75a0 2021-04-09T18:18:30.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.477 7f00d5a54700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00b75a0 0x7f00d0191040 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7f00cc00b580 tx=0x7f00cc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:30.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.477 7f00c67fc700 1 -- 172.21.15.32:0/740831050 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc00bd10 con 0x7f00d00b75a0 2021-04-09T18:18:30.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.477 7f00c67fc700 1 -- 172.21.15.32:0/740831050 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc000c90 con 0x7f00d00b75a0 2021-04-09T18:18:30.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.477 7f00c67fc700 1 -- 172.21.15.32:0/740831050 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc002220 con 0x7f00d00b75a0 2021-04-09T18:18:30.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.477 7f00d7c38700 1 -- 172.21.15.32:0/740831050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d01a74e0 con 0x7f00d00b75a0 2021-04-09T18:18:30.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.477 7f00d7c38700 1 -- 172.21.15.32:0/740831050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d01a74e0 con 0x7f00d00b75a0 2021-04-09T18:18:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.477 7f00c67fc700 1 -- 172.21.15.32:0/740831050 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f00cc006b00 con 0x7f00d00b75a0 2021-04-09T18:18:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.478 7f00c67fc700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00b8018930 0x7f00b801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.478 7f00c67fc700 1 -- 172.21.15.32:0/740831050 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f00cc0326c0 con 0x7f00d00b75a0 2021-04-09T18:18:30.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.478 7f00d5253700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00b8018930 0x7f00b801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:30.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.478 7f00d5253700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00b8018930 0x7f00b801ae10 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.498 7f00d7c38700 1 -- 172.21.15.32:0/740831050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d01a8400 con 0x7f00d00b75a0 2021-04-09T18:18:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.501 7f00c67fc700 1 -- 172.21.15.32:0/740831050 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f00cc00bd10 con 0x7f00d00b75a0 2021-04-09T18:18:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.689 7f00d7c38700 1 -- 172.21.15.32:0/740831050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00d01a8400 con 0x7f00d00b75a0 2021-04-09T18:18:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.690 7f00c67fc700 1 -- 172.21.15.32:0/740831050 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f00cc000c90 con 0x7f00d00b75a0 2021-04-09T18:18:30.762 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:30.762 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":78,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:18:29.621190","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:18:30.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 -- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00b8018930 msgr2=0x7f00b801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:30.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00b8018930 0x7f00b801ae10 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 -- 172.21.15.32:0/740831050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00b75a0 msgr2=0x7f00d0191040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:30.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00b75a0 0x7f00d0191040 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7f00cc00b580 tx=0x7f00cc00b920).stop 2021-04-09T18:18:30.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 -- 172.21.15.32:0/740831050 shutdown_connections 2021-04-09T18:18:30.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00b8018930 0x7f00b801ae10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00d01a8930 0x7f00d01a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00d00a21f0 0x7f00d00a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 --2- 172.21.15.32:0/740831050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00d00b75a0 0x7f00d0191040 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:30.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 -- 172.21.15.32:0/740831050 >> 172.21.15.32:0/740831050 conn(0x7f00d01ac4f0 msgr2=0x7f00d0190410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:30.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 -- 172.21.15.32:0/740831050 shutdown_connections 2021-04-09T18:18:30.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.693 7f00d7c38700 1 -- 172.21.15.32:0/740831050 wait complete. 2021-04-09T18:18:30.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:18:30.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:18:31.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.955 7f72a012a700 1 Processor -- start 2021-04-09T18:18:31.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.955 7f72a012a700 1 -- start start 2021-04-09T18:18:31.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f72a012a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f729810d010 0x7f72980d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f72a012a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729810d480 con 0x7f72980d1da0 2021-04-09T18:18:31.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f72a012a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729810e1f0 con 0x7f72980d8490 2021-04-09T18:18:31.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f72a012a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729810e780 con 0x7f729810d010 2021-04-09T18:18:31.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729d745700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f72980d1da0 0x7f72980d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56166/0 (socket says 172.21.15.32:56166) 2021-04-09T18:18:31.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729d745700 1 -- 172.21.15.32:0/2112040032 learned_addr learned my addr 172.21.15.32:0/2112040032 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:31.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729df46700 1 --2- 172.21.15.32:0/2112040032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f729810d010 0x7f72980d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729cf44700 1 -- 172.21.15.32:0/2112040032 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771713289 0 0) 0x7f7288000ce0 con 0x7f72980d1da0 2021-04-09T18:18:31.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729cf44700 1 -- 172.21.15.32:0/2112040032 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7284002fe0 con 0x7f72980d1da0 2021-04-09T18:18:31.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729cf44700 1 -- 172.21.15.32:0/2112040032 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438607621 0 0) 0x7f7290000d30 con 0x7f72980d8490 2021-04-09T18:18:31.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729cf44700 1 -- 172.21.15.32:0/2112040032 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7284004370 con 0x7f72980d8490 2021-04-09T18:18:31.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729df46700 1 -- 172.21.15.32:0/2112040032 >> v1:172.21.15.86:6789/0 conn(0x7f72980d8490 legacy=0x7f729810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729df46700 1 -- 172.21.15.32:0/2112040032 >> v1:172.21.15.32:6789/0 conn(0x7f72980d1da0 legacy=0x7f72980d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.956 7f729df46700 1 -- 172.21.15.32:0/2112040032 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72980d8d30 con 0x7f729810d010 2021-04-09T18:18:31.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.957 7f729df46700 1 --2- 172.21.15.32:0/2112040032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f729810d010 0x7f72980d1780 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f7294006650 tx=0x7f7294009a80).ready entity=mon.2 client_cookie=2d4e485c27aa3a5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:31.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.957 7f729cf44700 1 -- 172.21.15.32:0/2112040032 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72940009b0 con 0x7f729810d010 2021-04-09T18:18:31.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.957 7f729cf44700 1 -- 172.21.15.32:0/2112040032 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729400c0c0 con 0x7f729810d010 2021-04-09T18:18:31.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.957 7f729cf44700 1 -- 172.21.15.32:0/2112040032 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f729400c510 con 0x7f729810d010 2021-04-09T18:18:31.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.957 7f72a012a700 1 -- 172.21.15.32:0/2112040032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f729810d010 msgr2=0x7f72980d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.957 7f72a012a700 1 --2- 172.21.15.32:0/2112040032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f729810d010 0x7f72980d1780 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f7294006650 tx=0x7f7294009a80).stop 2021-04-09T18:18:31.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.958 7f72a012a700 1 -- 172.21.15.32:0/2112040032 shutdown_connections 2021-04-09T18:18:31.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.958 7f72a012a700 1 --2- 172.21.15.32:0/2112040032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f729810d010 0x7f72980d1780 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.958 7f72a012a700 1 -- 172.21.15.32:0/2112040032 >> 172.21.15.32:0/2112040032 conn(0x7f72980ce5f0 msgr2=0x7f72980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:31.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.958 7f72a012a700 1 -- 172.21.15.32:0/2112040032 shutdown_connections 2021-04-09T18:18:31.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.958 7f72a012a700 1 -- 172.21.15.32:0/2112040032 wait complete. 2021-04-09T18:18:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f72a012a700 1 Processor -- start 2021-04-09T18:18:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f72a012a700 1 -- start start 2021-04-09T18:18:31.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f72a012a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72980b8de0 0x7f72980c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f72a012a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72980c0640 0x7f72981936d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f72a012a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298193cf0 0x7f72981940f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f72a012a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72980c0a40 con 0x7f7298193cf0 2021-04-09T18:18:31.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f72a012a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72980b89a0 con 0x7f72980b8de0 2021-04-09T18:18:31.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f72a012a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72980bb2d0 con 0x7f72980c0640 2021-04-09T18:18:31.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f729e747700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298193cf0 0x7f72981940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f729e747700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298193cf0 0x7f72981940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46580/0 (socket says 172.21.15.32:46580) 2021-04-09T18:18:31.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.978 7f729e747700 1 -- 172.21.15.32:0/622112650 learned_addr learned my addr 172.21.15.32:0/622112650 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:31.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729d745700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72980c0640 0x7f72981936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729df46700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72980b8de0 0x7f72980c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729e747700 1 -- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72980c0640 msgr2=0x7f72981936d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729e747700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72980c0640 0x7f72981936d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729e747700 1 -- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72980b8de0 msgr2=0x7f72980c0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729e747700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72980b8de0 0x7f72980c0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729e747700 1 -- 172.21.15.32:0/622112650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72980bb4f0 con 0x7f7298193cf0 2021-04-09T18:18:31.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729d745700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72980c0640 0x7f72981936d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:31.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729df46700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72980b8de0 0x7f72980c0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:31.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f729e747700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298193cf0 0x7f72981940f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7f7290009980 tx=0x7f729000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f728e7fc700 1 -- 172.21.15.32:0/622112650 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7290003020 con 0x7f7298193cf0 2021-04-09T18:18:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f728e7fc700 1 -- 172.21.15.32:0/622112650 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7290005870 con 0x7f7298193cf0 2021-04-09T18:18:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f72a012a700 1 -- 172.21.15.32:0/622112650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72980bb4f0 con 0x7f7298193cf0 2021-04-09T18:18:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f728e7fc700 1 -- 172.21.15.32:0/622112650 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7290005d50 con 0x7f7298193cf0 2021-04-09T18:18:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.979 7f72a012a700 1 -- 172.21.15.32:0/622112650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72980bb4f0 con 0x7f7298193cf0 2021-04-09T18:18:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.980 7f728e7fc700 1 -- 172.21.15.32:0/622112650 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7290005a50 con 0x7f7298193cf0 2021-04-09T18:18:31.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.980 7f728e7fc700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7280018930 0x7f728001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.980 7f728e7fc700 1 -- 172.21.15.32:0/622112650 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f7290032bc0 con 0x7f7298193cf0 2021-04-09T18:18:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.980 7f729df46700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7280018930 0x7f728001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:30.981 7f729df46700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7280018930 0x7f728001ae10 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.000 7f72a012a700 1 -- 172.21.15.32:0/622112650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72981b7040 con 0x7f7298193cf0 2021-04-09T18:18:31.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.003 7f728e7fc700 1 -- 172.21.15.32:0/622112650 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f729000be10 con 0x7f7298193cf0 2021-04-09T18:18:31.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.184 7f72a012a700 1 -- 172.21.15.32:0/622112650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f72981b7040 con 0x7f7298193cf0 2021-04-09T18:18:31.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.185 7f728e7fc700 1 -- 172.21.15.32:0/622112650 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v78) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7290003100 con 0x7f7298193cf0 2021-04-09T18:18:31.224 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:18:31.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 -- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7280018930 msgr2=0x7f728001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7280018930 0x7f728001ae10 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 -- 172.21.15.32:0/622112650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298193cf0 msgr2=0x7f72981940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298193cf0 0x7f72981940f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7f7290009980 tx=0x7f729000bcd0).stop 2021-04-09T18:18:31.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 -- 172.21.15.32:0/622112650 shutdown_connections 2021-04-09T18:18:31.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7280018930 0x7f728001ae10 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7298193cf0 0x7f72981940f0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72980c0640 0x7f72981936d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 --2- 172.21.15.32:0/622112650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72980b8de0 0x7f72980c0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.187 7f72a012a700 1 -- 172.21.15.32:0/622112650 >> 172.21.15.32:0/622112650 conn(0x7f7298092a00 msgr2=0x7f72981b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:31.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.188 7f72a012a700 1 -- 172.21.15.32:0/622112650 shutdown_connections 2021-04-09T18:18:31.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.188 7f72a012a700 1 -- 172.21.15.32:0/622112650 wait complete. 2021-04-09T18:18:31.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:18:31.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.361 7f70ca870700 1 Processor -- start 2021-04-09T18:18:31.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.361 7f70ca870700 1 -- start start 2021-04-09T18:18:31.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.361 7f70ca870700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40d7ca0 0x7f70c40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.361 7f70ca870700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c410d480 con 0x7f70c410da00 2021-04-09T18:18:31.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.361 7f70ca870700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c40d86c0 con 0x7f70c410d010 2021-04-09T18:18:31.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.361 7f70ca870700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40d9220 con 0x7f70c40d7ca0 2021-04-09T18:18:31.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.361 7f70c8e8d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f70c410da00 0x7f70c410de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56178/0 (socket says 172.21.15.32:56178) 2021-04-09T18:18:31.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.361 7f70c8e8d700 1 -- 172.21.15.32:0/3532481370 learned_addr learned my addr 172.21.15.32:0/3532481370 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:31.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c37fe700 1 --2- 172.21.15.32:0/3532481370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40d7ca0 0x7f70c40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c2ffd700 1 -- 172.21.15.32:0/3532481370 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181970439 0 0) 0x7f70bc000d30 con 0x7f70c410da00 2021-04-09T18:18:31.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c2ffd700 1 -- 172.21.15.32:0/3532481370 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ac002fe0 con 0x7f70c410da00 2021-04-09T18:18:31.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c2ffd700 1 -- 172.21.15.32:0/3532481370 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663314607 0 0) 0x7f70b4000c60 con 0x7f70c410d010 2021-04-09T18:18:31.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c2ffd700 1 -- 172.21.15.32:0/3532481370 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ac003210 con 0x7f70c410d010 2021-04-09T18:18:31.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c37fe700 1 -- 172.21.15.32:0/3532481370 >> v1:172.21.15.86:6789/0 conn(0x7f70c410d010 legacy=0x7f70c40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c2ffd700 1 -- 172.21.15.32:0/3532481370 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2554180724 0 0) 0x7f70bc001270 con 0x7f70c410da00 2021-04-09T18:18:31.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c37fe700 1 -- 172.21.15.32:0/3532481370 >> v1:172.21.15.32:6789/0 conn(0x7f70c410da00 legacy=0x7f70c410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c37fe700 1 -- 172.21.15.32:0/3532481370 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4112100 con 0x7f70c40d7ca0 2021-04-09T18:18:31.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c37fe700 1 --2- 172.21.15.32:0/3532481370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40d7ca0 0x7f70c40d80a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f70b8006650 tx=0x7f70b8009a80).ready entity=mon.2 client_cookie=a0b5e0a6868ad5ab server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:31.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c2ffd700 1 -- 172.21.15.32:0/3532481370 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70b80009b0 con 0x7f70c40d7ca0 2021-04-09T18:18:31.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c2ffd700 1 -- 172.21.15.32:0/3532481370 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b800c0c0 con 0x7f70c40d7ca0 2021-04-09T18:18:31.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70c2ffd700 1 -- 172.21.15.32:0/3532481370 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70b800c510 con 0x7f70c40d7ca0 2021-04-09T18:18:31.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.362 7f70ca870700 1 -- 172.21.15.32:0/3532481370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40d7ca0 msgr2=0x7f70c40d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.363 7f70ca870700 1 --2- 172.21.15.32:0/3532481370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40d7ca0 0x7f70c40d80a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f70b8006650 tx=0x7f70b8009a80).stop 2021-04-09T18:18:31.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.363 7f70ca870700 1 -- 172.21.15.32:0/3532481370 shutdown_connections 2021-04-09T18:18:31.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.363 7f70ca870700 1 --2- 172.21.15.32:0/3532481370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40d7ca0 0x7f70c40d80a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.363 7f70ca870700 1 -- 172.21.15.32:0/3532481370 >> 172.21.15.32:0/3532481370 conn(0x7f70c40ce5f0 msgr2=0x7f70c40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:31.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.363 7f70ca870700 1 -- 172.21.15.32:0/3532481370 shutdown_connections 2021-04-09T18:18:31.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.363 7f70ca870700 1 -- 172.21.15.32:0/3532481370 wait complete. 2021-04-09T18:18:31.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.383 7f70ca870700 1 Processor -- start 2021-04-09T18:18:31.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.383 7f70ca870700 1 -- start start 2021-04-09T18:18:31.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70ca870700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c409a670 0x7f70c4093610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70ca870700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c41a9fb0 0x7f70c41aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70ca870700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40a3ab0 0x7f70c40a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70ca870700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4094060 con 0x7f70c40a3ab0 2021-04-09T18:18:31.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70ca870700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4094710 con 0x7f70c41a9fb0 2021-04-09T18:18:31.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70ca870700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4093c30 con 0x7f70c409a670 2021-04-09T18:18:31.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c8e8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40a3ab0 0x7f70c40a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c8e8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40a3ab0 0x7f70c40a3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46594/0 (socket says 172.21.15.32:46594) 2021-04-09T18:18:31.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c8e8d700 1 -- 172.21.15.32:0/1232195709 learned_addr learned my addr 172.21.15.32:0/1232195709 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:31.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c37fe700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c41a9fb0 0x7f70c41aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c3fff700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c409a670 0x7f70c4093610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c8e8d700 1 -- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c409a670 msgr2=0x7f70c4093610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c8e8d700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c409a670 0x7f70c4093610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c8e8d700 1 -- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c41a9fb0 msgr2=0x7f70c41aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c8e8d700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c41a9fb0 0x7f70c41aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.384 7f70c8e8d700 1 -- 172.21.15.32:0/1232195709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c41aa9d0 con 0x7f70c40a3ab0 2021-04-09T18:18:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.385 7f70c8e8d700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40a3ab0 0x7f70c40a3eb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f70bc001840 tx=0x7f70bc00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:31.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.385 7f70c0ff9700 1 -- 172.21.15.32:0/1232195709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70bc004930 con 0x7f70c40a3ab0 2021-04-09T18:18:31.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.385 7f70c0ff9700 1 -- 172.21.15.32:0/1232195709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc000d30 con 0x7f70c40a3ab0 2021-04-09T18:18:31.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.385 7f70ca870700 1 -- 172.21.15.32:0/1232195709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c41aa9d0 con 0x7f70c40a3ab0 2021-04-09T18:18:31.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.385 7f70c0ff9700 1 -- 172.21.15.32:0/1232195709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70bc004de0 con 0x7f70c40a3ab0 2021-04-09T18:18:31.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.385 7f70ca870700 1 -- 172.21.15.32:0/1232195709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c41aa9d0 con 0x7f70c40a3ab0 2021-04-09T18:18:31.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.385 7f70c0ff9700 1 -- 172.21.15.32:0/1232195709 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f70bc007230 con 0x7f70c40a3ab0 2021-04-09T18:18:31.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.386 7f70c0ff9700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70a4018930 0x7f70a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.386 7f70c0ff9700 1 -- 172.21.15.32:0/1232195709 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f70bc032a40 con 0x7f70c40a3ab0 2021-04-09T18:18:31.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.386 7f70c3fff700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70a4018930 0x7f70a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.386 7f70c3fff700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70a4018930 0x7f70a401ae10 crc :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:31.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.405 7f70ca870700 1 -- 172.21.15.32:0/1232195709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c410d480 con 0x7f70c40a3ab0 2021-04-09T18:18:31.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.409 7f70c0ff9700 1 -- 172.21.15.32:0/1232195709 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f70bc007020 con 0x7f70c40a3ab0 2021-04-09T18:18:31.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.587 7f70ca870700 1 -- 172.21.15.32:0/1232195709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f70c40946d0 con 0x7f70c40a3ab0 2021-04-09T18:18:31.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.588 7f70c0ff9700 1 -- 172.21.15.32:0/1232195709 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+64 (secure 0 0 0) 0x7f70bc007210 con 0x7f70c40a3ab0 2021-04-09T18:18:31.626 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:31.627 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:18:31.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:18:31.628 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:18:31.628 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:18:31.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:31.629 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:18:31.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 -- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70a4018930 msgr2=0x7f70a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70a4018930 0x7f70a401ae10 crc :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 -- 172.21.15.32:0/1232195709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40a3ab0 msgr2=0x7f70c40a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40a3ab0 0x7f70c40a3eb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f70bc001840 tx=0x7f70bc00aca0).stop 2021-04-09T18:18:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 -- 172.21.15.32:0/1232195709 shutdown_connections 2021-04-09T18:18:31.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f70a4018930 0x7f70a401ae10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70c40a3ab0 0x7f70c40a3eb0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c41a9fb0 0x7f70c41aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 --2- 172.21.15.32:0/1232195709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c409a670 0x7f70c4093610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 -- 172.21.15.32:0/1232195709 >> 172.21.15.32:0/1232195709 conn(0x7f70c41a0db0 msgr2=0x7f70c41a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:31.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 -- 172.21.15.32:0/1232195709 shutdown_connections 2021-04-09T18:18:31.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.590 7f70ca870700 1 -- 172.21.15.32:0/1232195709 wait complete. 2021-04-09T18:18:31.640 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:18:31.648 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:18:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.772 7fe692ee3700 1 Processor -- start 2021-04-09T18:18:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.772 7fe692ee3700 1 -- start start 2021-04-09T18:18:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe692ee3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c10da00 0x7fe68c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe692ee3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c10d480 con 0x7fe68c10d010 2021-04-09T18:18:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe692ee3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c0d1f70 con 0x7fe68c0d7ca0 2021-04-09T18:18:31.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe692ee3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c0d9220 con 0x7fe68c10da00 2021-04-09T18:18:31.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe690cff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe68c10d010 0x7fe68c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56190/0 (socket says 172.21.15.32:56190) 2021-04-09T18:18:31.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe690cff700 1 -- 172.21.15.32:0/1889360358 learned_addr learned my addr 172.21.15.32:0/1889360358 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe691500700 1 --2- 172.21.15.32:0/1889360358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c10da00 0x7fe68c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe68b7fe700 1 -- 172.21.15.32:0/1889360358 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102597257 0 0) 0x7fe67c000ce0 con 0x7fe68c10d010 2021-04-09T18:18:31.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe68b7fe700 1 -- 172.21.15.32:0/1889360358 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe674002fe0 con 0x7fe68c10d010 2021-04-09T18:18:31.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe68b7fe700 1 -- 172.21.15.32:0/1889360358 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255822225 0 0) 0x7fe680000c60 con 0x7fe68c0d7ca0 2021-04-09T18:18:31.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.773 7fe68b7fe700 1 -- 172.21.15.32:0/1889360358 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe674003210 con 0x7fe68c0d7ca0 2021-04-09T18:18:31.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe68b7fe700 1 -- 172.21.15.32:0/1889360358 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3942546303 0 0) 0x7fe67c0010e0 con 0x7fe68c10d010 2021-04-09T18:18:31.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe691500700 1 -- 172.21.15.32:0/1889360358 >> v1:172.21.15.86:6789/0 conn(0x7fe68c0d7ca0 legacy=0x7fe68c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe691500700 1 -- 172.21.15.32:0/1889360358 >> v1:172.21.15.32:6789/0 conn(0x7fe68c10d010 legacy=0x7fe68c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe691500700 1 -- 172.21.15.32:0/1889360358 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c10e420 con 0x7fe68c10da00 2021-04-09T18:18:31.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe691500700 1 --2- 172.21.15.32:0/1889360358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c10da00 0x7fe68c10de00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe684006e50 tx=0x7fe684009e70).ready entity=mon.2 client_cookie=7ebfe5ce5612e373 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:31.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe68b7fe700 1 -- 172.21.15.32:0/1889360358 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe684006a20 con 0x7fe68c10da00 2021-04-09T18:18:31.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe68b7fe700 1 -- 172.21.15.32:0/1889360358 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe684006790 con 0x7fe68c10da00 2021-04-09T18:18:31.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe68b7fe700 1 -- 172.21.15.32:0/1889360358 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe68400b700 con 0x7fe68c10da00 2021-04-09T18:18:31.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe692ee3700 1 -- 172.21.15.32:0/1889360358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c10da00 msgr2=0x7fe68c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.774 7fe692ee3700 1 --2- 172.21.15.32:0/1889360358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c10da00 0x7fe68c10de00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe684006e50 tx=0x7fe684009e70).stop 2021-04-09T18:18:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.775 7fe692ee3700 1 -- 172.21.15.32:0/1889360358 shutdown_connections 2021-04-09T18:18:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.775 7fe692ee3700 1 --2- 172.21.15.32:0/1889360358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c10da00 0x7fe68c10de00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.775 7fe692ee3700 1 -- 172.21.15.32:0/1889360358 >> 172.21.15.32:0/1889360358 conn(0x7fe68c0ce5f0 msgr2=0x7fe68c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:31.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.775 7fe692ee3700 1 -- 172.21.15.32:0/1889360358 shutdown_connections 2021-04-09T18:18:31.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.775 7fe692ee3700 1 -- 172.21.15.32:0/1889360358 wait complete. 2021-04-09T18:18:31.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.795 7fe692ee3700 1 Processor -- start 2021-04-09T18:18:31.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.795 7fe692ee3700 1 -- start start 2021-04-09T18:18:31.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.795 7fe692ee3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0b1680 0x7fe68c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.795 7fe692ee3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c09e330 0x7fe68c0b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe692ee3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c0b2190 0x7fe68c09f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe692ee3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c09e730 con 0x7fe68c09e330 2021-04-09T18:18:31.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe692ee3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c09f6c0 con 0x7fe68c0b1680 2021-04-09T18:18:31.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe692ee3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c09f8e0 con 0x7fe68c0b2190 2021-04-09T18:18:31.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe68bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c09e330 0x7fe68c0b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe68bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c09e330 0x7fe68c0b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46608/0 (socket says 172.21.15.32:46608) 2021-04-09T18:18:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe68bfff700 1 -- 172.21.15.32:0/3302620866 learned_addr learned my addr 172.21.15.32:0/3302620866 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe690cff700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0b1680 0x7fe68c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe691500700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c0b2190 0x7fe68c09f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:31.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe68bfff700 1 -- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c0b2190 msgr2=0x7fe68c09f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe68bfff700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c0b2190 0x7fe68c09f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe68bfff700 1 -- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0b1680 msgr2=0x7fe68c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe68bfff700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0b1680 0x7fe68c0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe68bfff700 1 -- 172.21.15.32:0/3302620866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c09fb00 con 0x7fe68c09e330 2021-04-09T18:18:31.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe690cff700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0b1680 0x7fe68c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:31.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe68bfff700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c09e330 0x7fe68c0b1b70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fe68000b580 tx=0x7fe68000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:31.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe6897fa700 1 -- 172.21.15.32:0/3302620866 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe68000bd10 con 0x7fe68c09e330 2021-04-09T18:18:31.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe6897fa700 1 -- 172.21.15.32:0/3302620866 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe680000e20 con 0x7fe68c09e330 2021-04-09T18:18:31.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68c09fb00 con 0x7fe68c09e330 2021-04-09T18:18:31.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe6897fa700 1 -- 172.21.15.32:0/3302620866 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6800054f0 con 0x7fe68c09e330 2021-04-09T18:18:31.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.796 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68c09fb00 con 0x7fe68c09e330 2021-04-09T18:18:31.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.798 7fe6897fa700 1 -- 172.21.15.32:0/3302620866 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe68000bd10 con 0x7fe68c09e330 2021-04-09T18:18:31.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.798 7fe6897fa700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe66c0188c0 0x7fe66c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:31.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.798 7fe6897fa700 1 -- 172.21.15.32:0/3302620866 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fe6800325d0 con 0x7fe68c09e330 2021-04-09T18:18:31.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.798 7fe690cff700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe66c0188c0 0x7fe66c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:32.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.798 7fe690cff700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe66c0188c0 0x7fe66c01adc0 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:32.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.817 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68c10d480 con 0x7fe68c09e330 2021-04-09T18:18:32.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:31.820 7fe6897fa700 1 -- 172.21.15.32:0/3302620866 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe68001f4c0 con 0x7fe68c09e330 2021-04-09T18:18:32.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.005 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe68c10d480 con 0x7fe68c09e330 2021-04-09T18:18:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.637 7fe6897fa700 1 -- 172.21.15.32:0/3302620866 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v79) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe68000bd10 con 0x7fe68c09e330 2021-04-09T18:18:32.708 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:18:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.672 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe66c0188c0 msgr2=0x7fe66c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:32.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.672 7fe692ee3700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe66c0188c0 0x7fe66c01adc0 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c09e330 msgr2=0x7fe68c0b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:32.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c09e330 0x7fe68c0b1b70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fe68000b580 tx=0x7fe68000b920).stop 2021-04-09T18:18:32.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 shutdown_connections 2021-04-09T18:18:32.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe66c0188c0 0x7fe66c01adc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:32.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe68c09e330 0x7fe68c0b1b70 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe68c0b2190 0x7fe68c09f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 --2- 172.21.15.32:0/3302620866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0b1680 0x7fe68c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:32.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 >> 172.21.15.32:0/3302620866 conn(0x7fe68c0c4290 msgr2=0x7fe68c0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 shutdown_connections 2021-04-09T18:18:32.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.673 7fe692ee3700 1 -- 172.21.15.32:0/3302620866 wait complete. 2021-04-09T18:18:32.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:18:32.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.859 7fd327ae5700 1 Processor -- start 2021-04-09T18:18:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.859 7fd327ae5700 1 -- start start 2021-04-09T18:18:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd327ae5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd3200d7ca0 0x7fd3200d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd327ae5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd32010da00 0x7fd32010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:32.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd327ae5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32010d480 con 0x7fd32010d010 2021-04-09T18:18:32.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd327ae5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3200d9220 con 0x7fd32010da00 2021-04-09T18:18:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd327ae5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3200d86c0 con 0x7fd3200d7ca0 2021-04-09T18:18:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd326102700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd32010da00 0x7fd32010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:32.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd325901700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd32010d010 0x7fd3200d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56204/0 (socket says 172.21.15.32:56204) 2021-04-09T18:18:32.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd326102700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd32010da00 0x7fd32010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46616/0 (socket says 172.21.15.32:46616) 2021-04-09T18:18:32.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd325901700 1 -- 172.21.15.32:0/3010596393 learned_addr learned my addr 172.21.15.32:0/3010596393 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:32.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd325100700 1 --2- 172.21.15.32:0/3010596393 >> v2:172.21.15.86:3300/0 conn(0x7fd3200d7ca0 0x7fd3200d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd326102700 1 -- 172.21.15.32:0/3010596393 >> v2:172.21.15.86:3300/0 conn(0x7fd3200d7ca0 msgr2=0x7fd3200d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd326102700 1 --2- 172.21.15.32:0/3010596393 >> v2:172.21.15.86:3300/0 conn(0x7fd3200d7ca0 0x7fd3200d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd326102700 1 -- 172.21.15.32:0/3010596393 >> v1:172.21.15.32:6789/0 conn(0x7fd32010d010 legacy=0x7fd3200d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd3248ff700 1 -- 172.21.15.32:0/3010596393 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596651852 0 0) 0x7fd31c000ce0 con 0x7fd32010d010 2021-04-09T18:18:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd326102700 1 -- 172.21.15.32:0/3010596393 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200d1f70 con 0x7fd32010da00 2021-04-09T18:18:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.860 7fd325100700 1 --2- 172.21.15.32:0/3010596393 >> v2:172.21.15.86:3300/0 conn(0x7fd3200d7ca0 0x7fd3200d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd326102700 1 --2- 172.21.15.32:0/3010596393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32010da00 0x7fd32010de00 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7fd318006e30 tx=0x7fd318009e60).ready entity=mon.0 client_cookie=462d77ea8d82f7de server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd3248ff700 1 -- 172.21.15.32:0/3010596393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd318006a50 con 0x7fd32010da00 2021-04-09T18:18:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd3248ff700 1 -- 172.21.15.32:0/3010596393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31800b300 con 0x7fd32010da00 2021-04-09T18:18:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd3248ff700 1 -- 172.21.15.32:0/3010596393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd318000f90 con 0x7fd32010da00 2021-04-09T18:18:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd327ae5700 1 -- 172.21.15.32:0/3010596393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32010da00 msgr2=0x7fd32010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd327ae5700 1 --2- 172.21.15.32:0/3010596393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32010da00 0x7fd32010de00 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7fd318006e30 tx=0x7fd318009e60).stop 2021-04-09T18:18:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd327ae5700 1 -- 172.21.15.32:0/3010596393 shutdown_connections 2021-04-09T18:18:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd327ae5700 1 --2- 172.21.15.32:0/3010596393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32010da00 0x7fd32010de00 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd327ae5700 1 --2- 172.21.15.32:0/3010596393 >> v2:172.21.15.86:3300/0 conn(0x7fd3200d7ca0 0x7fd3200d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.861 7fd327ae5700 1 -- 172.21.15.32:0/3010596393 >> 172.21.15.32:0/3010596393 conn(0x7fd3200ce5f0 msgr2=0x7fd3200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.862 7fd327ae5700 1 -- 172.21.15.32:0/3010596393 shutdown_connections 2021-04-09T18:18:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.862 7fd327ae5700 1 -- 172.21.15.32:0/3010596393 wait complete. 2021-04-09T18:18:32.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.882 7fd327ae5700 1 Processor -- start 2021-04-09T18:18:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.882 7fd327ae5700 1 -- start start 2021-04-09T18:18:32.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.882 7fd327ae5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd320191df0 0x7fd320090ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.882 7fd327ae5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3200914f0 0x7fd32008d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.882 7fd327ae5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32008da90 0x7fd32008b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:32.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.882 7fd327ae5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32008de90 con 0x7fd32008da90 2021-04-09T18:18:32.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.882 7fd327ae5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32008e080 con 0x7fd3200914f0 2021-04-09T18:18:32.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.882 7fd327ae5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32008bd50 con 0x7fd320191df0 2021-04-09T18:18:32.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd326102700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32008da90 0x7fd32008b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:32.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd326102700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32008da90 0x7fd32008b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46622/0 (socket says 172.21.15.32:46622) 2021-04-09T18:18:32.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd326102700 1 -- 172.21.15.32:0/1143064000 learned_addr learned my addr 172.21.15.32:0/1143064000 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:33.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd325901700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd320191df0 0x7fd320090ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd325100700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3200914f0 0x7fd32008d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd326102700 1 -- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd320191df0 msgr2=0x7fd320090ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd326102700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd320191df0 0x7fd320090ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd326102700 1 -- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3200914f0 msgr2=0x7fd32008d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd326102700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3200914f0 0x7fd32008d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd326102700 1 -- 172.21.15.32:0/1143064000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32008bf70 con 0x7fd32008da90 2021-04-09T18:18:33.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd326102700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32008da90 0x7fd32008b730 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7fd318006d20 tx=0x7fd31800bd90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:33.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd3127fc700 1 -- 172.21.15.32:0/1143064000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd318009070 con 0x7fd32008da90 2021-04-09T18:18:33.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd3127fc700 1 -- 172.21.15.32:0/1143064000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd318001270 con 0x7fd32008da90 2021-04-09T18:18:33.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32008bf70 con 0x7fd32008da90 2021-04-09T18:18:33.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd3127fc700 1 -- 172.21.15.32:0/1143064000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd318001740 con 0x7fd32008da90 2021-04-09T18:18:33.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.883 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32008bf70 con 0x7fd32008da90 2021-04-09T18:18:33.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.884 7fd3127fc700 1 -- 172.21.15.32:0/1143064000 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd318001450 con 0x7fd32008da90 2021-04-09T18:18:33.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.884 7fd3127fc700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd308018930 0x7fd30801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.884 7fd3127fc700 1 -- 172.21.15.32:0/1143064000 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5111+0+0 (secure 0 0 0) 0x7fd318032ba0 con 0x7fd32008da90 2021-04-09T18:18:33.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.884 7fd325901700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd308018930 0x7fd30801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.885 7fd325901700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd308018930 0x7fd30801ae10 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:33.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.904 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32010d480 con 0x7fd32008da90 2021-04-09T18:18:33.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:32.908 7fd3127fc700 1 -- 172.21.15.32:0/1143064000 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd318003070 con 0x7fd32008da90 2021-04-09T18:18:33.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.090 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fd32010d480 con 0x7fd32008da90 2021-04-09T18:18:33.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.213 7fd3127fc700 1 -- 172.21.15.32:0/1143064000 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v80) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd318001270 con 0x7fd32008da90 2021-04-09T18:18:33.291 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:18:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd308018930 msgr2=0x7fd30801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd308018930 0x7fd30801ae10 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32008da90 msgr2=0x7fd32008b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32008da90 0x7fd32008b730 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7fd318006d20 tx=0x7fd31800bd90).stop 2021-04-09T18:18:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 shutdown_connections 2021-04-09T18:18:33.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd308018930 0x7fd30801ae10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd32008da90 0x7fd32008b730 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3200914f0 0x7fd32008d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 --2- 172.21.15.32:0/1143064000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd320191df0 0x7fd320090ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 >> 172.21.15.32:0/1143064000 conn(0x7fd3200b2fa0 msgr2=0x7fd3200a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:33.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 shutdown_connections 2021-04-09T18:18:33.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.256 7fd327ae5700 1 -- 172.21.15.32:0/1143064000 wait complete. 2021-04-09T18:18:33.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:18:33.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.429 7f9dd6af8700 1 Processor -- start 2021-04-09T18:18:33.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.429 7f9dd6af8700 1 -- start start 2021-04-09T18:18:33.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.429 7f9dd6af8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd010da80 0x7f9dd010de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.429 7f9dd6af8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd010d500 con 0x7f9dd010d050 2021-04-09T18:18:33.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.429 7f9dd6af8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd00d0330 con 0x7f9dd00d7cc0 2021-04-09T18:18:33.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.429 7f9dd6af8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd00d8d70 con 0x7f9dd010da80 2021-04-09T18:18:33.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.429 7f9dd4914700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9dd010d050 0x7f9dd00cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56218/0 (socket says 172.21.15.32:56218) 2021-04-09T18:18:33.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.429 7f9dd4914700 1 -- 172.21.15.32:0/3396429827 learned_addr learned my addr 172.21.15.32:0/3396429827 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:33.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dd5115700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd010da80 0x7f9dd010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246886887 0 0) 0x7f9dc0000ce0 con 0x7f9dd010d050 2021-04-09T18:18:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9db8002fe0 con 0x7f9dd010d050 2021-04-09T18:18:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104041194 0 0) 0x7f9dc4000c60 con 0x7f9dd00d7cc0 2021-04-09T18:18:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9db8003210 con 0x7f9dd00d7cc0 2021-04-09T18:18:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1477544575 0 0) 0x7f9dc00010e0 con 0x7f9dd010d050 2021-04-09T18:18:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd010da80 msgr2=0x7f9dd010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd010da80 0x7f9dd010de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 >> v1:172.21.15.86:6789/0 conn(0x7f9dd00d7cc0 legacy=0x7f9dd00d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd010e4a0 con 0x7f9dd010d050 2021-04-09T18:18:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f9dc0001600 con 0x7f9dd010d050 2021-04-09T18:18:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 >> v1:172.21.15.32:6789/0 conn(0x7f9dd010d050 legacy=0x7f9dd00cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9db8003b90 0x7f9db8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9db80067a0 0x7f9db8008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9db8009510 0x7f9db800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80502f0 con 0x7f9db8009510 2021-04-09T18:18:33.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80504d0 con 0x7f9db8003b90 2021-04-09T18:18:33.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dd5115700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9db8009510 0x7f9db800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80506f0 con 0x7f9db80067a0 2021-04-09T18:18:33.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dd4914700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9db8003b90 0x7f9db8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dcffff700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9db80067a0 0x7f9db8008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.430 7f9dd5115700 1 -- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9db80067a0 msgr2=0x7f9db8008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:33.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dd5115700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9db80067a0 0x7f9db8008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dd5115700 1 -- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9db8003b90 msgr2=0x7f9db8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:33.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dd5115700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9db8003b90 0x7f9db8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dd5115700 1 -- 172.21.15.32:0/3396429827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db8050990 con 0x7f9db8009510 2021-04-09T18:18:33.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dd4914700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9db8003b90 0x7f9db8005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:33.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dd5115700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9db8009510 0x7f9db800ba30 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f9dc8009b10 tx=0x7f9dc8009eb0).ready entity=mon.0 client_cookie=6117974aa33e86df server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dc800b510 con 0x7f9db8009510 2021-04-09T18:18:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc8003030 con 0x7f9db8009510 2021-04-09T18:18:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dcf7fe700 1 -- 172.21.15.32:0/3396429827 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dc800b790 con 0x7f9db8009510 2021-04-09T18:18:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dd6af8700 1 -- 172.21.15.32:0/3396429827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9db8009510 msgr2=0x7f9db800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dd6af8700 1 --2- 172.21.15.32:0/3396429827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9db8009510 0x7f9db800ba30 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f9dc8009b10 tx=0x7f9dc8009eb0).stop 2021-04-09T18:18:33.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.431 7f9dd5115700 1 -- 172.21.15.32:0/3396429827 reap_dead start 2021-04-09T18:18:33.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.432 7f9dd6af8700 1 -- 172.21.15.32:0/3396429827 shutdown_connections 2021-04-09T18:18:33.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.432 7f9dd6af8700 1 -- 172.21.15.32:0/3396429827 >> 172.21.15.32:0/3396429827 conn(0x7f9dd00ce620 msgr2=0x7f9dd00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:33.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.432 7f9dd6af8700 1 -- 172.21.15.32:0/3396429827 shutdown_connections 2021-04-09T18:18:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.432 7f9dd6af8700 1 -- 172.21.15.32:0/3396429827 wait complete. 2021-04-09T18:18:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.452 7f9dd6af8700 1 Processor -- start 2021-04-09T18:18:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.452 7f9dd6af8700 1 -- start start 2021-04-09T18:18:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.452 7f9dd6af8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009b2f0 0x7f9dd009b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.452 7f9dd6af8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009a8b0 0x7f9dd009e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.452 7f9dd6af8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd009e9c0 0x7f9dd009edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.452 7f9dd6af8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd00bde80 con 0x7f9dd009b2f0 2021-04-09T18:18:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.452 7f9dd6af8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd009ad70 con 0x7f9dd009e9c0 2021-04-09T18:18:33.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.452 7f9dd6af8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd009bd10 con 0x7f9dd009a8b0 2021-04-09T18:18:33.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd4914700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009b2f0 0x7f9dd009b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd4914700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009b2f0 0x7f9dd009b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46640/0 (socket says 172.21.15.32:46640) 2021-04-09T18:18:33.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd4914700 1 -- 172.21.15.32:0/2718094050 learned_addr learned my addr 172.21.15.32:0/2718094050 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:33.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dcffff700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009a8b0 0x7f9dd009e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd5115700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd009e9c0 0x7f9dd009edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd4914700 1 -- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009a8b0 msgr2=0x7f9dd009e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd4914700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009a8b0 0x7f9dd009e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd4914700 1 -- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd009e9c0 msgr2=0x7f9dd009edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd4914700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd009e9c0 0x7f9dd009edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd4914700 1 -- 172.21.15.32:0/2718094050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd009fb80 con 0x7f9dd009b2f0 2021-04-09T18:18:33.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd4914700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009b2f0 0x7f9dd009b6f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7f9dc0001400 tx=0x7f9dc000a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:33.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dcd7fa700 1 -- 172.21.15.32:0/2718094050 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dc00070d0 con 0x7f9dd009b2f0 2021-04-09T18:18:33.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dcd7fa700 1 -- 172.21.15.32:0/2718094050 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc0001a10 con 0x7f9dd009b2f0 2021-04-09T18:18:33.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dcd7fa700 1 -- 172.21.15.32:0/2718094050 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dc00045b0 con 0x7f9dd009b2f0 2021-04-09T18:18:33.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd009fb80 con 0x7f9dd009b2f0 2021-04-09T18:18:33.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.453 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd009fb80 con 0x7f9dd009b2f0 2021-04-09T18:18:33.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.454 7f9dcd7fa700 1 -- 172.21.15.32:0/2718094050 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9dc0007f60 con 0x7f9dd009b2f0 2021-04-09T18:18:33.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.454 7f9dcd7fa700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db00188c0 0x7f9db001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.454 7f9dcd7fa700 1 -- 172.21.15.32:0/2718094050 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f9dc00326d0 con 0x7f9dd009b2f0 2021-04-09T18:18:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.454 7f9dcffff700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db00188c0 0x7f9db001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.455 7f9dcffff700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db00188c0 0x7f9db001adc0 crc :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:33.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.473 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd00bde80 con 0x7f9dd009b2f0 2021-04-09T18:18:33.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.477 7f9dcd7fa700 1 -- 172.21.15.32:0/2718094050 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9dc002df30 con 0x7f9dd009b2f0 2021-04-09T18:18:33.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.662 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f9dd00bde80 con 0x7f9dd009b2f0 2021-04-09T18:18:33.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.665 7f9dcd7fa700 1 -- 172.21.15.32:0/2718094050 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 14 and data pool 15 v88) v1 ==== 189+0+0 (secure 0 0 0) 0x7f9dc0007f60 con 0x7f9dd009b2f0 2021-04-09T18:18:33.754 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 14 and data pool 15 2021-04-09T18:18:33.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.673 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db00188c0 msgr2=0x7f9db001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.673 7f9dd6af8700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db00188c0 0x7f9db001adc0 crc :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.673 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009b2f0 msgr2=0x7f9dd009b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.673 7f9dd6af8700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009b2f0 0x7f9dd009b6f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7f9dc0001400 tx=0x7f9dc000a880).stop 2021-04-09T18:18:33.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.674 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 shutdown_connections 2021-04-09T18:18:33.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.674 7f9dd6af8700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9db00188c0 0x7f9db001adc0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.674 7f9dd6af8700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dd009e9c0 0x7f9dd009edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.674 7f9dd6af8700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dd009a8b0 0x7f9dd009e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.674 7f9dd6af8700 1 --2- 172.21.15.32:0/2718094050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dd009b2f0 0x7f9dd009b6f0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.674 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 >> 172.21.15.32:0/2718094050 conn(0x7f9dd00ae6f0 msgr2=0x7f9dd0097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:33.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.674 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 shutdown_connections 2021-04-09T18:18:33.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.674 7f9dd6af8700 1 -- 172.21.15.32:0/2718094050 wait complete. 2021-04-09T18:18:33.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:18:33.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.879 7f2136868700 1 Processor -- start 2021-04-09T18:18:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.879 7f2136868700 1 -- start start 2021-04-09T18:18:33.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f2136868700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f213010d010 0x7f21300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f2136868700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f21300d1da0 0x7f21300d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f2136868700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213010d480 con 0x7f21300d82c0 2021-04-09T18:18:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f2136868700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213010e780 con 0x7f21300d1da0 2021-04-09T18:18:33.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f2136868700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21300d8d30 con 0x7f213010d010 2021-04-09T18:18:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f2127fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f21300d1da0 0x7f21300d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f212ffff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f213010d010 0x7f21300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f2134e85700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f21300d82c0 0x7f213010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54178/0 (socket says 172.21.15.32:54178) 2021-04-09T18:18:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f2134e85700 1 -- 172.21.15.32:0/2495457300 learned_addr learned my addr 172.21.15.32:0/2495457300 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:33.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f212f7fe700 1 -- 172.21.15.32:0/2495457300 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455874582 0 0) 0x7f2128001510 con 0x7f21300d82c0 2021-04-09T18:18:33.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.880 7f212f7fe700 1 -- 172.21.15.32:0/2495457300 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f211c002fe0 con 0x7f21300d82c0 2021-04-09T18:18:33.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f2127fff700 1 -- 172.21.15.32:0/2495457300 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f213010d010 msgr2=0x7f21300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:33.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f2127fff700 1 --2- 172.21.15.32:0/2495457300 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f213010d010 0x7f21300d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f2127fff700 1 -- 172.21.15.32:0/2495457300 >> v1:172.21.15.86:6789/0 conn(0x7f21300d82c0 legacy=0x7f213010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f2127fff700 1 -- 172.21.15.32:0/2495457300 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21300d9220 con 0x7f21300d1da0 2021-04-09T18:18:33.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f212ffff700 1 --2- 172.21.15.32:0/2495457300 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f213010d010 0x7f21300d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f2127fff700 1 --2- 172.21.15.32:0/2495457300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21300d1da0 0x7f21300d7ca0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f2118006650 tx=0x7f2118009a80).ready entity=mon.1 client_cookie=ab09d9397b03f3c1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:33.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f212f7fe700 1 -- 172.21.15.32:0/2495457300 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21180009b0 con 0x7f21300d1da0 2021-04-09T18:18:33.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f212f7fe700 1 -- 172.21.15.32:0/2495457300 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211800c0c0 con 0x7f21300d1da0 2021-04-09T18:18:33.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f212f7fe700 1 -- 172.21.15.32:0/2495457300 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f211800c510 con 0x7f21300d1da0 2021-04-09T18:18:33.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f2136868700 1 -- 172.21.15.32:0/2495457300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21300d1da0 msgr2=0x7f21300d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.881 7f2136868700 1 --2- 172.21.15.32:0/2495457300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21300d1da0 0x7f21300d7ca0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f2118006650 tx=0x7f2118009a80).stop 2021-04-09T18:18:33.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.882 7f2136868700 1 -- 172.21.15.32:0/2495457300 shutdown_connections 2021-04-09T18:18:33.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.882 7f2136868700 1 --2- 172.21.15.32:0/2495457300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21300d1da0 0x7f21300d7ca0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.882 7f2136868700 1 --2- 172.21.15.32:0/2495457300 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f213010d010 0x7f21300d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:33.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.882 7f2136868700 1 -- 172.21.15.32:0/2495457300 >> 172.21.15.32:0/2495457300 conn(0x7f21300ce5f0 msgr2=0x7f21300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:33.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.882 7f2136868700 1 -- 172.21.15.32:0/2495457300 shutdown_connections 2021-04-09T18:18:33.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.882 7f2136868700 1 -- 172.21.15.32:0/2495457300 wait complete. 2021-04-09T18:18:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.902 7f2136868700 1 Processor -- start 2021-04-09T18:18:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.902 7f2136868700 1 -- start start 2021-04-09T18:18:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.902 7f2136868700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21300ac0f0 0x7f21300ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.902 7f2136868700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21301a6c20 0x7f21301a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.902 7f2136868700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21301a56c0 0x7f21301a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.902 7f2136868700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21301a76c0 con 0x7f21301a56c0 2021-04-09T18:18:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.902 7f2136868700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213017c7e0 con 0x7f21301a6c20 2021-04-09T18:18:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.902 7f2136868700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213017ca00 con 0x7f21300ac0f0 2021-04-09T18:18:33.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2134e85700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21301a56c0 0x7f21301a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2134e85700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21301a56c0 0x7f21301a5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46666/0 (socket says 172.21.15.32:46666) 2021-04-09T18:18:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2134e85700 1 -- 172.21.15.32:0/3461686058 learned_addr learned my addr 172.21.15.32:0/3461686058 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2127fff700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21301a6c20 0x7f21301a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f212ffff700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21300ac0f0 0x7f21300ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:33.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2134e85700 1 -- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21300ac0f0 msgr2=0x7f21300ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:33.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2134e85700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21300ac0f0 0x7f21300ac4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2134e85700 1 -- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21301a6c20 msgr2=0x7f21301a7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2134e85700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21301a6c20 0x7f21301a7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2134e85700 1 -- 172.21.15.32:0/3461686058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21301a60e0 con 0x7f21301a56c0 2021-04-09T18:18:34.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2127fff700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21301a6c20 0x7f21301a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:34.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2134e85700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21301a56c0 0x7f21301a5ac0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f2128001ba0 tx=0x7f212800ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:34.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f212d7fa700 1 -- 172.21.15.32:0/3461686058 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21280070d0 con 0x7f21301a56c0 2021-04-09T18:18:34.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f212d7fa700 1 -- 172.21.15.32:0/3461686058 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2128007390 con 0x7f21301a56c0 2021-04-09T18:18:34.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2136868700 1 -- 172.21.15.32:0/3461686058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21301a60e0 con 0x7f21301a56c0 2021-04-09T18:18:34.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f212d7fa700 1 -- 172.21.15.32:0/3461686058 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2128004b30 con 0x7f21301a56c0 2021-04-09T18:18:34.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.903 7f2136868700 1 -- 172.21.15.32:0/3461686058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21301a60e0 con 0x7f21301a56c0 2021-04-09T18:18:34.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.904 7f212d7fa700 1 -- 172.21.15.32:0/3461686058 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f21280046a0 con 0x7f21301a56c0 2021-04-09T18:18:34.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.904 7f212d7fa700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21140188d0 0x7f211401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.904 7f212d7fa700 1 -- 172.21.15.32:0/3461686058 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2128032d50 con 0x7f21301a56c0 2021-04-09T18:18:34.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.904 7f212ffff700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21140188d0 0x7f211401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.905 7f212ffff700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21140188d0 0x7f211401add0 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:34.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.924 7f2136868700 1 -- 172.21.15.32:0/3461686058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21301b7050 con 0x7f21301a56c0 2021-04-09T18:18:34.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:33.927 7f212d7fa700 1 -- 172.21.15.32:0/3461686058 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2128001280 con 0x7f21301a56c0 2021-04-09T18:18:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.116 7f2136868700 1 -- 172.21.15.32:0/3461686058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21301b7050 con 0x7f21301a56c0 2021-04-09T18:18:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.117 7f212d7fa700 1 -- 172.21.15.32:0/3461686058 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f2128007390 con 0x7f21301a56c0 2021-04-09T18:18:34.156 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:34.157 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":84,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:18:33.664214","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:18:32.041217","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-04-09 18:18:33.125737","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:18:34.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 -- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21140188d0 msgr2=0x7f211401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21140188d0 0x7f211401add0 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 -- 172.21.15.32:0/3461686058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21301a56c0 msgr2=0x7f21301a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21301a56c0 0x7f21301a5ac0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f2128001ba0 tx=0x7f212800ac80).stop 2021-04-09T18:18:34.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 -- 172.21.15.32:0/3461686058 shutdown_connections 2021-04-09T18:18:34.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f21140188d0 0x7f211401add0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21301a56c0 0x7f21301a5ac0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21301a6c20 0x7f21301a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 --2- 172.21.15.32:0/3461686058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21300ac0f0 0x7f21300ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 -- 172.21.15.32:0/3461686058 >> 172.21.15.32:0/3461686058 conn(0x7f2130199680 msgr2=0x7f21301a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:34.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 -- 172.21.15.32:0/3461686058 shutdown_connections 2021-04-09T18:18:34.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.119 7f2136868700 1 -- 172.21.15.32:0/3461686058 wait complete. 2021-04-09T18:18:34.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:18:34.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.284 7f2a64176700 1 Processor -- start 2021-04-09T18:18:34.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.284 7f2a64176700 1 -- start start 2021-04-09T18:18:34.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a64176700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c10d010 0x7f2a5c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a64176700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a5c0d1da0 0x7f2a5c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a64176700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c10d480 con 0x7f2a5c0d82c0 2021-04-09T18:18:34.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a64176700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a5c10e780 con 0x7f2a5c0d1da0 2021-04-09T18:18:34.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a64176700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c0d8d30 con 0x7f2a5c10d010 2021-04-09T18:18:34.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a61791700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a5c0d1da0 0x7f2a5c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a61791700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a5c0d1da0 0x7f2a5c0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46678/0 (socket says 172.21.15.32:46678) 2021-04-09T18:18:34.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a61791700 1 -- 172.21.15.32:0/3326557248 learned_addr learned my addr 172.21.15.32:0/3326557248 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:34.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a61f92700 1 --2- 172.21.15.32:0/3326557248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c10d010 0x7f2a5c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a61791700 1 -- 172.21.15.32:0/3326557248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c10d010 msgr2=0x7f2a5c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:34.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a61791700 1 --2- 172.21.15.32:0/3326557248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c10d010 0x7f2a5c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a61791700 1 -- 172.21.15.32:0/3326557248 >> v1:172.21.15.86:6789/0 conn(0x7f2a5c0d82c0 legacy=0x7f2a5c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a61791700 1 -- 172.21.15.32:0/3326557248 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c0d9220 con 0x7f2a5c0d1da0 2021-04-09T18:18:34.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.285 7f2a61791700 1 --2- 172.21.15.32:0/3326557248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c0d1da0 0x7f2a5c0d7ca0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f2a50006700 tx=0x7f2a50009b80).ready entity=mon.0 client_cookie=78bd99f0ec9696c2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:34.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.286 7f2a60f90700 1 -- 172.21.15.32:0/3326557248 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a5000c0c0 con 0x7f2a5c0d1da0 2021-04-09T18:18:34.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.286 7f2a60f90700 1 -- 172.21.15.32:0/3326557248 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5000c440 con 0x7f2a5c0d1da0 2021-04-09T18:18:34.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.286 7f2a60f90700 1 -- 172.21.15.32:0/3326557248 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a5000c920 con 0x7f2a5c0d1da0 2021-04-09T18:18:34.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.286 7f2a64176700 1 -- 172.21.15.32:0/3326557248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c0d1da0 msgr2=0x7f2a5c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.286 7f2a64176700 1 --2- 172.21.15.32:0/3326557248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c0d1da0 0x7f2a5c0d7ca0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f2a50006700 tx=0x7f2a50009b80).stop 2021-04-09T18:18:34.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.286 7f2a64176700 1 -- 172.21.15.32:0/3326557248 shutdown_connections 2021-04-09T18:18:34.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.286 7f2a64176700 1 --2- 172.21.15.32:0/3326557248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c0d1da0 0x7f2a5c0d7ca0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.286 7f2a64176700 1 --2- 172.21.15.32:0/3326557248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c10d010 0x7f2a5c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.286 7f2a64176700 1 -- 172.21.15.32:0/3326557248 >> 172.21.15.32:0/3326557248 conn(0x7f2a5c0ce5f0 msgr2=0x7f2a5c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:34.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.287 7f2a64176700 1 -- 172.21.15.32:0/3326557248 shutdown_connections 2021-04-09T18:18:34.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.287 7f2a64176700 1 -- 172.21.15.32:0/3326557248 wait complete. 2021-04-09T18:18:34.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.307 7f2a64176700 1 Processor -- start 2021-04-09T18:18:34.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.307 7f2a64176700 1 -- start start 2021-04-09T18:18:34.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.307 7f2a64176700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1ac380 0x7f2a5c1ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.307 7f2a64176700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c186c90 0x7f2a5c187090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.307 7f2a64176700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c185530 0x7f2a5c185930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.307 7f2a64176700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c186790 con 0x7f2a5c185530 2021-04-09T18:18:34.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.307 7f2a64176700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c1acda0 con 0x7f2a5c1ac380 2021-04-09T18:18:34.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.307 7f2a64176700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c18e100 con 0x7f2a5c186c90 2021-04-09T18:18:34.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a62793700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c185530 0x7f2a5c185930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a62793700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c185530 0x7f2a5c185930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46684/0 (socket says 172.21.15.32:46684) 2021-04-09T18:18:34.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a62793700 1 -- 172.21.15.32:0/1806427219 learned_addr learned my addr 172.21.15.32:0/1806427219 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:34.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a61f92700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1ac380 0x7f2a5c1ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a61791700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c186c90 0x7f2a5c187090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a62793700 1 -- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c186c90 msgr2=0x7f2a5c187090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a62793700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c186c90 0x7f2a5c187090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a62793700 1 -- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1ac380 msgr2=0x7f2a5c1ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a62793700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1ac380 0x7f2a5c1ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a62793700 1 -- 172.21.15.32:0/1806427219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c1876b0 con 0x7f2a5c185530 2021-04-09T18:18:34.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a61791700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c186c90 0x7f2a5c187090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:34.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a61f92700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1ac380 0x7f2a5c1ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:34.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a62793700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c185530 0x7f2a5c185930 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f2a54009940 tx=0x7f2a5400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:34.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a4e7fc700 1 -- 172.21.15.32:0/1806427219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a54000f40 con 0x7f2a5c185530 2021-04-09T18:18:34.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a4e7fc700 1 -- 172.21.15.32:0/1806427219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5400c2f0 con 0x7f2a5c185530 2021-04-09T18:18:34.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a64176700 1 -- 172.21.15.32:0/1806427219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a5c1876b0 con 0x7f2a5c185530 2021-04-09T18:18:34.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a4e7fc700 1 -- 172.21.15.32:0/1806427219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a5400c7d0 con 0x7f2a5c185530 2021-04-09T18:18:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.308 7f2a64176700 1 -- 172.21.15.32:0/1806427219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a5c1876b0 con 0x7f2a5c185530 2021-04-09T18:18:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.309 7f2a4e7fc700 1 -- 172.21.15.32:0/1806427219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2a54001220 con 0x7f2a5c185530 2021-04-09T18:18:34.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.309 7f2a4e7fc700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a44018930 0x7f2a4401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.309 7f2a4e7fc700 1 -- 172.21.15.32:0/1806427219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2a54032d80 con 0x7f2a5c185530 2021-04-09T18:18:34.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.309 7f2a61f92700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a44018930 0x7f2a4401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.310 7f2a61f92700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a44018930 0x7f2a4401ae10 crc :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.329 7f2a64176700 1 -- 172.21.15.32:0/1806427219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a5c182d10 con 0x7f2a5c185530 2021-04-09T18:18:34.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.332 7f2a4e7fc700 1 -- 172.21.15.32:0/1806427219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2a5400be30 con 0x7f2a5c185530 2021-04-09T18:18:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.521 7f2a64176700 1 -- 172.21.15.32:0/1806427219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a5c182d10 con 0x7f2a5c185530 2021-04-09T18:18:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.522 7f2a4e7fc700 1 -- 172.21.15.32:0/1806427219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f2a54001220 con 0x7f2a5c185530 2021-04-09T18:18:34.560 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:34.560 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":84,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:18:33.664214","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:18:32.041217","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-04-09 18:18:33.125737","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:18:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 -- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a44018930 msgr2=0x7f2a4401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a44018930 0x7f2a4401ae10 crc :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 -- 172.21.15.32:0/1806427219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c185530 msgr2=0x7f2a5c185930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c185530 0x7f2a5c185930 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f2a54009940 tx=0x7f2a5400bcf0).stop 2021-04-09T18:18:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 -- 172.21.15.32:0/1806427219 shutdown_connections 2021-04-09T18:18:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a44018930 0x7f2a4401ae10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a5c185530 0x7f2a5c185930 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a5c186c90 0x7f2a5c187090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 --2- 172.21.15.32:0/1806427219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1ac380 0x7f2a5c1ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 -- 172.21.15.32:0/1806427219 >> 172.21.15.32:0/1806427219 conn(0x7f2a5c19b890 msgr2=0x7f2a5c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 -- 172.21.15.32:0/1806427219 shutdown_connections 2021-04-09T18:18:34.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.524 7f2a64176700 1 -- 172.21.15.32:0/1806427219 wait complete. 2021-04-09T18:18:34.575 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:18:34.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.689 7fc020a48700 1 Processor -- start 2021-04-09T18:18:34.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.689 7fc020a48700 1 -- start start 2021-04-09T18:18:34.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc020a48700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc01c10d010 0x7fc01c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc020a48700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0d1da0 0x7fc01c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc020a48700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01c10d480 con 0x7fc01c0d82c0 2021-04-09T18:18:34.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc020a48700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01c0d9220 con 0x7fc01c10d010 2021-04-09T18:18:34.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc020a48700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0d86c0 con 0x7fc01c0d1da0 2021-04-09T18:18:34.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc01a85e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc01c10d010 0x7fc01c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc01205d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0d1da0 0x7fc01c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc01b05f700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc01c0d82c0 0x7fc01c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54206/0 (socket says 172.21.15.32:54206) 2021-04-09T18:18:34.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc01b05f700 1 -- 172.21.15.32:0/2251040692 learned_addr learned my addr 172.21.15.32:0/2251040692 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:34.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc01a05d700 1 -- 172.21.15.32:0/2251040692 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925488300 0 0) 0x7fc014001440 con 0x7fc01c0d82c0 2021-04-09T18:18:34.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.690 7fc01a05d700 1 -- 172.21.15.32:0/2251040692 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbffc002fe0 con 0x7fc01c0d82c0 2021-04-09T18:18:34.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc01a85e700 1 -- 172.21.15.32:0/2251040692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0d1da0 msgr2=0x7fc01c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:34.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc01a85e700 1 --2- 172.21.15.32:0/2251040692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0d1da0 0x7fc01c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc01a85e700 1 -- 172.21.15.32:0/2251040692 >> v1:172.21.15.86:6789/0 conn(0x7fc01c0d82c0 legacy=0x7fc01c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc01a85e700 1 -- 172.21.15.32:0/2251040692 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c10e1f0 con 0x7fc01c10d010 2021-04-09T18:18:34.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc01205d700 1 --2- 172.21.15.32:0/2251040692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0d1da0 0x7fc01c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:34.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc01a85e700 1 --2- 172.21.15.32:0/2251040692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c10d010 0x7fc01c0d1780 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fc00c006650 tx=0x7fc00c009a80).ready entity=mon.1 client_cookie=bf8c6f923bb30985 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc01a05d700 1 -- 172.21.15.32:0/2251040692 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc00c0009b0 con 0x7fc01c10d010 2021-04-09T18:18:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc01a05d700 1 -- 172.21.15.32:0/2251040692 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c00c0c0 con 0x7fc01c10d010 2021-04-09T18:18:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc01a05d700 1 -- 172.21.15.32:0/2251040692 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc00c00c510 con 0x7fc01c10d010 2021-04-09T18:18:34.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc020a48700 1 -- 172.21.15.32:0/2251040692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c10d010 msgr2=0x7fc01c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.691 7fc020a48700 1 --2- 172.21.15.32:0/2251040692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c10d010 0x7fc01c0d1780 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fc00c006650 tx=0x7fc00c009a80).stop 2021-04-09T18:18:34.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.692 7fc020a48700 1 -- 172.21.15.32:0/2251040692 shutdown_connections 2021-04-09T18:18:34.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.692 7fc020a48700 1 --2- 172.21.15.32:0/2251040692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0d1da0 0x7fc01c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.692 7fc020a48700 1 --2- 172.21.15.32:0/2251040692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c10d010 0x7fc01c0d1780 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.692 7fc020a48700 1 -- 172.21.15.32:0/2251040692 >> 172.21.15.32:0/2251040692 conn(0x7fc01c0ce5f0 msgr2=0x7fc01c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:34.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.692 7fc020a48700 1 -- 172.21.15.32:0/2251040692 shutdown_connections 2021-04-09T18:18:34.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.692 7fc020a48700 1 -- 172.21.15.32:0/2251040692 wait complete. 2021-04-09T18:18:34.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.712 7fc020a48700 1 Processor -- start 2021-04-09T18:18:34.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.712 7fc020a48700 1 -- start start 2021-04-09T18:18:34.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.712 7fc020a48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c17bb20 0x7fc01c0c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.712 7fc020a48700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0b7620 0x7fc01c0c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.712 7fc020a48700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c17bf20 0x7fc01c08a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.712 7fc020a48700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0c1e00 con 0x7fc01c17bb20 2021-04-09T18:18:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.712 7fc020a48700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0c0b00 con 0x7fc01c17bf20 2021-04-09T18:18:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.712 7fc020a48700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c08ac40 con 0x7fc01c0b7620 2021-04-09T18:18:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c17bb20 0x7fc01c0c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c17bb20 0x7fc01c0c1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46694/0 (socket says 172.21.15.32:46694) 2021-04-09T18:18:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01a85e700 1 -- 172.21.15.32:0/3858464051 learned_addr learned my addr 172.21.15.32:0/3858464051 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01205d700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0b7620 0x7fc01c0c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01b05f700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c17bf20 0x7fc01c08a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01205d700 1 -- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c17bf20 msgr2=0x7fc01c08a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01205d700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c17bf20 0x7fc01c08a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01205d700 1 -- 172.21.15.32:0/3858464051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c17bb20 msgr2=0x7fc01c0c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:34.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01205d700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c17bb20 0x7fc01c0c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:34.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01205d700 1 -- 172.21.15.32:0/3858464051 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c08ae60 con 0x7fc01c0b7620 2021-04-09T18:18:34.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01a85e700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c17bb20 0x7fc01c0c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:34.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc01205d700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0b7620 0x7fc01c0c16a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc004009540 tx=0x7fc00400b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:34.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc013fff700 1 -- 172.21.15.32:0/3858464051 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc00400bcd0 con 0x7fc01c0b7620 2021-04-09T18:18:34.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc013fff700 1 -- 172.21.15.32:0/3858464051 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0040063e0 con 0x7fc01c0b7620 2021-04-09T18:18:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc013fff700 1 -- 172.21.15.32:0/3858464051 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc004002030 con 0x7fc01c0b7620 2021-04-09T18:18:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc020a48700 1 -- 172.21.15.32:0/3858464051 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01c08ae60 con 0x7fc01c0b7620 2021-04-09T18:18:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.713 7fc020a48700 1 -- 172.21.15.32:0/3858464051 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01c08ae60 con 0x7fc01c0b7620 2021-04-09T18:18:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.714 7fc013fff700 1 -- 172.21.15.32:0/3858464051 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc004000c90 con 0x7fc01c0b7620 2021-04-09T18:18:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.714 7fc013fff700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbff4018930 0x7fbff401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.714 7fc013fff700 1 -- 172.21.15.32:0/3858464051 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc004032910 con 0x7fc01c0b7620 2021-04-09T18:18:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.714 7fc01a85e700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbff4018930 0x7fbff401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.715 7fc01a85e700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbff4018930 0x7fbff401ae10 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:34.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.734 7fc020a48700 1 -- 172.21.15.32:0/3858464051 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01c10d480 con 0x7fc01c0b7620 2021-04-09T18:18:34.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.738 7fc013fff700 1 -- 172.21.15.32:0/3858464051 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc00400bcd0 con 0x7fc01c0b7620 2021-04-09T18:18:34.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:34.930 7fc020a48700 1 -- 172.21.15.32:0/3858464051 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fc01c10d480 con 0x7fc01c0b7620 2021-04-09T18:18:35.616 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:18:35.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:18:35.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.678 7fc013fff700 1 -- 172.21.15.32:0/3858464051 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v91) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc004000c90 con 0x7fc01c0b7620 2021-04-09T18:18:35.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 -- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbff4018930 msgr2=0x7fbff401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:35.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbff4018930 0x7fbff401ae10 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:35.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 -- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0b7620 msgr2=0x7fc01c0c16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:35.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0b7620 0x7fc01c0c16a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc004009540 tx=0x7fc00400b8e0).stop 2021-04-09T18:18:35.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 -- 172.21.15.32:0/3858464051 shutdown_connections 2021-04-09T18:18:35.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbff4018930 0x7fbff401ae10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:35.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c17bf20 0x7fc01c08a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:35.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0b7620 0x7fc01c0c16a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 --2- 172.21.15.32:0/3858464051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c17bb20 0x7fc01c0c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 -- 172.21.15.32:0/3858464051 >> 172.21.15.32:0/3858464051 conn(0x7fc01c181390 msgr2=0x7fc01c0adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 -- 172.21.15.32:0/3858464051 shutdown_connections 2021-04-09T18:18:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.698 7fc020a48700 1 -- 172.21.15.32:0/3858464051 wait complete. 2021-04-09T18:18:35.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:35.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.901 7f84c955b700 1 Processor -- start 2021-04-09T18:18:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.901 7f84c955b700 1 -- start start 2021-04-09T18:18:35.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c955b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84c410d010 0x7f84c40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:35.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c955b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f84c40d1da0 0x7f84c40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:35.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c955b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c410d480 con 0x7f84c40d82c0 2021-04-09T18:18:35.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c955b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c410e780 con 0x7f84c40d1da0 2021-04-09T18:18:35.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c955b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40d8d30 con 0x7f84c410d010 2021-04-09T18:18:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c3ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f84c40d82c0 0x7f84c410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56294/0 (socket says 172.21.15.32:56294) 2021-04-09T18:18:35.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c3ac0700 1 -- 172.21.15.32:0/3102534945 learned_addr learned my addr 172.21.15.32:0/3102534945 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c2abe700 1 --2- 172.21.15.32:0/3102534945 >> v2:172.21.15.86:3300/0 conn(0x7f84c40d1da0 0x7f84c40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c32bf700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84c410d010 0x7f84c40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:35.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079586913 0 0) 0x7f84bc000d30 con 0x7f84c40d82c0 2021-04-09T18:18:36.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.902 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84ac002fe0 con 0x7f84c40d82c0 2021-04-09T18:18:36.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 794213615 0 0) 0x7f84bc001270 con 0x7f84c40d82c0 2021-04-09T18:18:36.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84c410d010 msgr2=0x7f84c40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:36.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84c410d010 0x7f84c40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 >> v2:172.21.15.86:3300/0 conn(0x7f84c40d1da0 msgr2=0x7f84c40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:36.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 --2- 172.21.15.32:0/3102534945 >> v2:172.21.15.86:3300/0 conn(0x7f84c40d1da0 0x7f84c40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c40d9220 con 0x7f84c40d82c0 2021-04-09T18:18:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f84bc001980 con 0x7f84c40d82c0 2021-04-09T18:18:36.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c2abe700 1 --2- 172.21.15.32:0/3102534945 >> v2:172.21.15.86:3300/0 conn(0x7f84c40d1da0 0x7f84c40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:36.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 >> v1:172.21.15.32:6789/0 conn(0x7f84c40d82c0 legacy=0x7f84c410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84ac047aa0 0x7f84ac049ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84ac04a770 0x7f84ac04cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84ac04d590 0x7f84ac04fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ac0502d0 con 0x7f84ac04a770 2021-04-09T18:18:36.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ac0504f0 con 0x7f84ac047aa0 2021-04-09T18:18:36.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ac050710 con 0x7f84ac04d590 2021-04-09T18:18:36.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c2abe700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84ac04a770 0x7f84ac04cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c32bf700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84ac047aa0 0x7f84ac049ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c3ac0700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84ac04d590 0x7f84ac04fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c2abe700 1 -- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84ac04d590 msgr2=0x7f84ac04fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c2abe700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84ac04d590 0x7f84ac04fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c2abe700 1 -- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84ac047aa0 msgr2=0x7f84ac049ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c2abe700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84ac047aa0 0x7f84ac049ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c2abe700 1 -- 172.21.15.32:0/3102534945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ac0509b0 con 0x7f84ac04a770 2021-04-09T18:18:36.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c2abe700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84ac04a770 0x7f84ac04cc90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f84b8006b00 tx=0x7f84b800ce20).ready entity=mon.0 client_cookie=8cdb5c8df7c7c713 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:36.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b8009310 con 0x7f84ac04a770 2021-04-09T18:18:36.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b80068d0 con 0x7f84ac04a770 2021-04-09T18:18:36.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c22bd700 1 -- 172.21.15.32:0/3102534945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b8009890 con 0x7f84ac04a770 2021-04-09T18:18:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c955b700 1 -- 172.21.15.32:0/3102534945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84ac04a770 msgr2=0x7f84ac04cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c955b700 1 --2- 172.21.15.32:0/3102534945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84ac04a770 0x7f84ac04cc90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f84b8006b00 tx=0x7f84b800ce20).stop 2021-04-09T18:18:36.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.903 7f84c3ac0700 1 -- 172.21.15.32:0/3102534945 reap_dead start 2021-04-09T18:18:36.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.904 7f84c955b700 1 -- 172.21.15.32:0/3102534945 shutdown_connections 2021-04-09T18:18:36.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.904 7f84c955b700 1 -- 172.21.15.32:0/3102534945 >> 172.21.15.32:0/3102534945 conn(0x7f84c40ce5f0 msgr2=0x7f84c40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:36.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.904 7f84c955b700 1 -- 172.21.15.32:0/3102534945 shutdown_connections 2021-04-09T18:18:36.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.904 7f84c955b700 1 -- 172.21.15.32:0/3102534945 wait complete. 2021-04-09T18:18:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.924 7f84c955b700 1 Processor -- start 2021-04-09T18:18:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.924 7f84c955b700 1 -- start start 2021-04-09T18:18:36.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c955b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84c41a91e0 0x7f84c40a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c955b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c41a95e0 0x7f84c40a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c955b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84c41ac850 0x7f84c41acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c955b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c41ad270 con 0x7f84c41a95e0 2021-04-09T18:18:36.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c955b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40b7640 con 0x7f84c41a91e0 2021-04-09T18:18:36.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c955b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4183c80 con 0x7f84c41ac850 2021-04-09T18:18:36.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c2abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c41a95e0 0x7f84c40a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c2abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c41a95e0 0x7f84c40a5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46718/0 (socket says 172.21.15.32:46718) 2021-04-09T18:18:36.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c2abe700 1 -- 172.21.15.32:0/3159504067 learned_addr learned my addr 172.21.15.32:0/3159504067 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:36.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c32bf700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84c41a91e0 0x7f84c40a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c3ac0700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84c41ac850 0x7f84c41acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c2abe700 1 -- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84c41ac850 msgr2=0x7f84c41acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c2abe700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84c41ac850 0x7f84c41acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c2abe700 1 -- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84c41a91e0 msgr2=0x7f84c40a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c2abe700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84c41a91e0 0x7f84c40a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.925 7f84c2abe700 1 -- 172.21.15.32:0/3159504067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c4183ea0 con 0x7f84c41a95e0 2021-04-09T18:18:36.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.926 7f84c2abe700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c41a95e0 0x7f84c40a5080 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f84b8009b20 tx=0x7f84b8009ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:36.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.926 7f84abfff700 1 -- 172.21.15.32:0/3159504067 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b800c410 con 0x7f84c41a95e0 2021-04-09T18:18:36.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.926 7f84abfff700 1 -- 172.21.15.32:0/3159504067 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b800ca90 con 0x7f84c41a95e0 2021-04-09T18:18:36.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.926 7f84c955b700 1 -- 172.21.15.32:0/3159504067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c4183ea0 con 0x7f84c41a95e0 2021-04-09T18:18:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.926 7f84abfff700 1 -- 172.21.15.32:0/3159504067 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b8002030 con 0x7f84c41a95e0 2021-04-09T18:18:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.926 7f84c955b700 1 -- 172.21.15.32:0/3159504067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c4183ea0 con 0x7f84c41a95e0 2021-04-09T18:18:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.926 7f84abfff700 1 -- 172.21.15.32:0/3159504067 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f84b8000c90 con 0x7f84c41a95e0 2021-04-09T18:18:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.926 7f84abfff700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84a4018930 0x7f84a401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.926 7f84abfff700 1 -- 172.21.15.32:0/3159504067 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f84b8032a10 con 0x7f84c41a95e0 2021-04-09T18:18:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.927 7f84c32bf700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84a4018930 0x7f84a401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.927 7f84c32bf700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84a4018930 0x7f84a401adc0 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:36.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.946 7f84c955b700 1 -- 172.21.15.32:0/3159504067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c4182d70 con 0x7f84c41a95e0 2021-04-09T18:18:36.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:35.950 7f84abfff700 1 -- 172.21.15.32:0/3159504067 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f84b800c410 con 0x7f84c41a95e0 2021-04-09T18:18:36.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.141 7f84c955b700 1 -- 172.21.15.32:0/3159504067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84c410d480 con 0x7f84c41a95e0 2021-04-09T18:18:36.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.142 7f84abfff700 1 -- 172.21.15.32:0/3159504067 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2779 (secure 0 0 0) 0x7f84b8000c90 con 0x7f84c41a95e0 2021-04-09T18:18:36.180 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2021-04-09T18:18:36.181 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:36.181 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6654,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:35.676838","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 -- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84a4018930 msgr2=0x7f84a401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84a4018930 0x7f84a401adc0 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 -- 172.21.15.32:0/3159504067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c41a95e0 msgr2=0x7f84c40a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c41a95e0 0x7f84c40a5080 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f84b8009b20 tx=0x7f84b8009ec0).stop 2021-04-09T18:18:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 -- 172.21.15.32:0/3159504067 shutdown_connections 2021-04-09T18:18:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84a4018930 0x7f84a401adc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84c41a95e0 0x7f84c40a5080 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84c41ac850 0x7f84c41acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 --2- 172.21.15.32:0/3159504067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84c41a91e0 0x7f84c40a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 -- 172.21.15.32:0/3159504067 >> 172.21.15.32:0/3159504067 conn(0x7f84c419b460 msgr2=0x7f84c419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.144 7f84c955b700 1 -- 172.21.15.32:0/3159504067 shutdown_connections 2021-04-09T18:18:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.145 7f84c955b700 1 -- 172.21.15.32:0/3159504067 wait complete. 2021-04-09T18:18:36.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:18:36.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.307 7fb84f54c700 1 Processor -- start 2021-04-09T18:18:36.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.307 7fb84f54c700 1 -- start start 2021-04-09T18:18:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb84f54c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8480d7ca0 0x7fb8480d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb84f54c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84810d480 con 0x7fb84810da00 2021-04-09T18:18:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb84f54c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480d86c0 con 0x7fb84810d010 2021-04-09T18:18:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb84f54c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480d9220 con 0x7fb8480d7ca0 2021-04-09T18:18:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb84db69700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb84810da00 0x7fb84810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56312/0 (socket says 172.21.15.32:56312) 2021-04-09T18:18:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb84db69700 1 -- 172.21.15.32:0/3675770042 learned_addr learned my addr 172.21.15.32:0/3675770042 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb84cb67700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8480d7ca0 0x7fb8480d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790162602 0 0) 0x7fb840000d30 con 0x7fb84810da00 2021-04-09T18:18:36.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb834002fe0 con 0x7fb84810da00 2021-04-09T18:18:36.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3676958889 0 0) 0x7fb840001270 con 0x7fb84810da00 2021-04-09T18:18:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8480d7ca0 msgr2=0x7fb8480d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb83ffff700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8480d7ca0 0x7fb8480d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 >> v1:172.21.15.86:6789/0 conn(0x7fb84810d010 legacy=0x7fb8480d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.308 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8480d1f70 con 0x7fb84810da00 2021-04-09T18:18:36.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fb840001980 con 0x7fb84810da00 2021-04-09T18:18:36.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 >> v1:172.21.15.32:6789/0 conn(0x7fb84810da00 legacy=0x7fb84810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb834003a70 0x7fb834005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb834006790 0x7fb834008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb834009590 0x7fb83400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83400c2d0 con 0x7fb834006790 2021-04-09T18:18:36.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83400c4f0 con 0x7fb834003a70 2021-04-09T18:18:36.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84cb67700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb834006790 0x7fb834008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb834050720 con 0x7fb834009590 2021-04-09T18:18:36.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84d368700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb834003a70 0x7fb834005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84db69700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb834009590 0x7fb83400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84cb67700 1 -- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb834009590 msgr2=0x7fb83400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:36.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84cb67700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb834009590 0x7fb83400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84cb67700 1 -- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb834003a70 msgr2=0x7fb834005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:36.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84cb67700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb834003a70 0x7fb834005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84cb67700 1 -- 172.21.15.32:0/3675770042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834050980 con 0x7fb834006790 2021-04-09T18:18:36.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84cb67700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb834006790 0x7fb834008c90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fb838006ac0 tx=0x7fb838009ab0).ready entity=mon.0 client_cookie=54bf747177ef1ad6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:36.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb83800c0c0 con 0x7fb834006790 2021-04-09T18:18:36.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83800c3c0 con 0x7fb834006790 2021-04-09T18:18:36.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb83ffff700 1 -- 172.21.15.32:0/3675770042 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb83800c8a0 con 0x7fb834006790 2021-04-09T18:18:36.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84f54c700 1 -- 172.21.15.32:0/3675770042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb834006790 msgr2=0x7fb834008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84f54c700 1 --2- 172.21.15.32:0/3675770042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb834006790 0x7fb834008c90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fb838006ac0 tx=0x7fb838009ab0).stop 2021-04-09T18:18:36.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.309 7fb84db69700 1 -- 172.21.15.32:0/3675770042 reap_dead start 2021-04-09T18:18:36.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.310 7fb84f54c700 1 -- 172.21.15.32:0/3675770042 shutdown_connections 2021-04-09T18:18:36.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.310 7fb84f54c700 1 -- 172.21.15.32:0/3675770042 >> 172.21.15.32:0/3675770042 conn(0x7fb8480ce5f0 msgr2=0x7fb8480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:36.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.310 7fb84f54c700 1 -- 172.21.15.32:0/3675770042 shutdown_connections 2021-04-09T18:18:36.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.310 7fb84f54c700 1 -- 172.21.15.32:0/3675770042 wait complete. 2021-04-09T18:18:36.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.330 7fb84f54c700 1 Processor -- start 2021-04-09T18:18:36.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.330 7fb84f54c700 1 -- start start 2021-04-09T18:18:36.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84f54c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8480b5360 0x7fb8481a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84f54c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb84818e140 0x7fb8481aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84f54c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8481ad040 0x7fb84808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84f54c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8481a5360 con 0x7fb8480b5360 2021-04-09T18:18:36.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84f54c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480902c0 con 0x7fb84818e140 2021-04-09T18:18:36.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84f54c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480904e0 con 0x7fb8481ad040 2021-04-09T18:18:36.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84d368700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8480b5360 0x7fb8481a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84d368700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8480b5360 0x7fb8481a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46734/0 (socket says 172.21.15.32:46734) 2021-04-09T18:18:36.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84d368700 1 -- 172.21.15.32:0/3459646765 learned_addr learned my addr 172.21.15.32:0/3459646765 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:36.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84db69700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8481ad040 0x7fb84808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.331 7fb84cb67700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb84818e140 0x7fb8481aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb84d368700 1 -- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8481ad040 msgr2=0x7fb84808fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb84d368700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8481ad040 0x7fb84808fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb84d368700 1 -- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb84818e140 msgr2=0x7fb8481aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb84d368700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb84818e140 0x7fb8481aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb84d368700 1 -- 172.21.15.32:0/3459646765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb848183c80 con 0x7fb8480b5360 2021-04-09T18:18:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb84d368700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8480b5360 0x7fb8481a4d40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fb844005c30 tx=0x7fb84400a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb83dffb700 1 -- 172.21.15.32:0/3459646765 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb84400ac90 con 0x7fb8480b5360 2021-04-09T18:18:36.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb83dffb700 1 -- 172.21.15.32:0/3459646765 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84400c0d0 con 0x7fb8480b5360 2021-04-09T18:18:36.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb848183c80 con 0x7fb8480b5360 2021-04-09T18:18:36.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb83dffb700 1 -- 172.21.15.32:0/3459646765 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb84400c5b0 con 0x7fb8480b5360 2021-04-09T18:18:36.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.332 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb848183c80 con 0x7fb8480b5360 2021-04-09T18:18:36.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.333 7fb83dffb700 1 -- 172.21.15.32:0/3459646765 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb84400af10 con 0x7fb8480b5360 2021-04-09T18:18:36.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.333 7fb83dffb700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb828018930 0x7fb82801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.333 7fb83dffb700 1 -- 172.21.15.32:0/3459646765 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb8440329e0 con 0x7fb8480b5360 2021-04-09T18:18:36.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.333 7fb84cb67700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb828018930 0x7fb82801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.333 7fb84cb67700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb828018930 0x7fb82801ae10 crc :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:36.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.352 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb848182d70 con 0x7fb8480b5360 2021-04-09T18:18:36.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.355 7fb83dffb700 1 -- 172.21.15.32:0/3459646765 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb844000c10 con 0x7fb8480b5360 2021-04-09T18:18:36.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.544 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb848182d70 con 0x7fb8480b5360 2021-04-09T18:18:36.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.545 7fb83dffb700 1 -- 172.21.15.32:0/3459646765 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fb84400ae90 con 0x7fb8480b5360 2021-04-09T18:18:36.598 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:36.598 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":84,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:18:33.664214","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:18:32.041217","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-04-09 18:18:33.125737","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:18:36.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb828018930 msgr2=0x7fb82801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb828018930 0x7fb82801ae10 crc :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8480b5360 msgr2=0x7fb8481a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8480b5360 0x7fb8481a4d40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fb844005c30 tx=0x7fb84400a8a0).stop 2021-04-09T18:18:36.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 shutdown_connections 2021-04-09T18:18:36.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb828018930 0x7fb82801ae10 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8481ad040 0x7fb84808fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb84818e140 0x7fb8481aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 --2- 172.21.15.32:0/3459646765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8480b5360 0x7fb8481a4d40 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 >> 172.21.15.32:0/3459646765 conn(0x7fb84819b460 msgr2=0x7fb84819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 shutdown_connections 2021-04-09T18:18:36.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.547 7fb84f54c700 1 -- 172.21.15.32:0/3459646765 wait complete. 2021-04-09T18:18:36.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:18:36.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.793 7fb7350c5700 1 Processor -- start 2021-04-09T18:18:36.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.793 7fb7350c5700 1 -- start start 2021-04-09T18:18:36.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb7350c5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7300d2700 0x7fb7300cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb7350c5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb7301b04f0 0x7fb7301b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb7350c5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7300d8700 con 0x7fb7300f28c0 2021-04-09T18:18:36.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb7350c5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7300f2e80 con 0x7fb7300d2700 2021-04-09T18:18:36.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb7350c5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7300d91e0 con 0x7fb7301b04f0 2021-04-09T18:18:36.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb72ed9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7300d2700 0x7fb7300cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb72ed9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb7300d2700 0x7fb7300cfeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46742/0 (socket says 172.21.15.32:46742) 2021-04-09T18:18:36.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb72ed9d700 1 -- 172.21.15.32:0/3844232877 learned_addr learned my addr 172.21.15.32:0/3844232877 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:36.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb72f59e700 1 --2- 172.21.15.32:0/3844232877 >> v2:172.21.15.86:3300/0 conn(0x7fb7301b04f0 0x7fb7301b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb72ed9d700 1 -- 172.21.15.32:0/3844232877 >> v2:172.21.15.86:3300/0 conn(0x7fb7301b04f0 msgr2=0x7fb7301b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:36.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb72ed9d700 1 --2- 172.21.15.32:0/3844232877 >> v2:172.21.15.86:3300/0 conn(0x7fb7301b04f0 0x7fb7301b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb72ed9d700 1 -- 172.21.15.32:0/3844232877 >> v1:172.21.15.86:6789/0 conn(0x7fb7300f28c0 legacy=0x7fb7300d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb72ed9d700 1 -- 172.21.15.32:0/3844232877 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7300d8480 con 0x7fb7300d2700 2021-04-09T18:18:36.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.794 7fb72dd9b700 1 -- 172.21.15.32:0/3844232877 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013456057 0 0) 0x7fb724001370 con 0x7fb7300f28c0 2021-04-09T18:18:36.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.795 7fb72f59e700 1 --2- 172.21.15.32:0/3844232877 >> v2:172.21.15.86:3300/0 conn(0x7fb7301b04f0 0x7fb7301b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:36.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.795 7fb72ed9d700 1 --2- 172.21.15.32:0/3844232877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7300d2700 0x7fb7300cfeb0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7fb720006700 tx=0x7fb720009b80).ready entity=mon.0 client_cookie=4fc9d8d3da4e81b9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:36.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.795 7fb72dd9b700 1 -- 172.21.15.32:0/3844232877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb72000c0c0 con 0x7fb7300d2700 2021-04-09T18:18:36.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.795 7fb72dd9b700 1 -- 172.21.15.32:0/3844232877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72000c440 con 0x7fb7300d2700 2021-04-09T18:18:36.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.795 7fb72dd9b700 1 -- 172.21.15.32:0/3844232877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb72000c920 con 0x7fb7300d2700 2021-04-09T18:18:36.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.795 7fb7350c5700 1 -- 172.21.15.32:0/3844232877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7300d2700 msgr2=0x7fb7300cfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.795 7fb7350c5700 1 --2- 172.21.15.32:0/3844232877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7300d2700 0x7fb7300cfeb0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7fb720006700 tx=0x7fb720009b80).stop 2021-04-09T18:18:36.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.796 7fb7350c5700 1 -- 172.21.15.32:0/3844232877 shutdown_connections 2021-04-09T18:18:36.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.796 7fb7350c5700 1 --2- 172.21.15.32:0/3844232877 >> v2:172.21.15.86:3300/0 conn(0x7fb7301b04f0 0x7fb7301b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.796 7fb7350c5700 1 --2- 172.21.15.32:0/3844232877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7300d2700 0x7fb7300cfeb0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.796 7fb7350c5700 1 -- 172.21.15.32:0/3844232877 >> 172.21.15.32:0/3844232877 conn(0x7fb7300ce590 msgr2=0x7fb7300cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.796 7fb7350c5700 1 -- 172.21.15.32:0/3844232877 shutdown_connections 2021-04-09T18:18:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.796 7fb7350c5700 1 -- 172.21.15.32:0/3844232877 wait complete. 2021-04-09T18:18:36.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.816 7fb7350c5700 1 Processor -- start 2021-04-09T18:18:36.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.816 7fb7350c5700 1 -- start start 2021-04-09T18:18:36.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.816 7fb7350c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb73009a540 0x7fb7300c3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.816 7fb7350c5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7300c3e40 0x7fb73008ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.816 7fb7350c5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7300c0550 0x7fb73008e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.816 7fb7350c5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73009ef20 con 0x7fb73009a540 2021-04-09T18:18:36.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.816 7fb7350c5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7300c3060 con 0x7fb7300c0550 2021-04-09T18:18:36.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.816 7fb7350c5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7300c32a0 con 0x7fb7300c3e40 2021-04-09T18:18:36.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb73009a540 0x7fb7300c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72ed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb73009a540 0x7fb7300c3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46748/0 (socket says 172.21.15.32:46748) 2021-04-09T18:18:36.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72ed9d700 1 -- 172.21.15.32:0/2854193216 learned_addr learned my addr 172.21.15.32:0/2854193216 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:36.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72e59c700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7300c3e40 0x7fb73008ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72ed9d700 1 -- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7300c3e40 msgr2=0x7fb73008ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72f59e700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7300c0550 0x7fb73008e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72ed9d700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7300c3e40 0x7fb73008ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72ed9d700 1 -- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7300c0550 msgr2=0x7fb73008e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:36.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72ed9d700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7300c0550 0x7fb73008e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:36.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72ed9d700 1 -- 172.21.15.32:0/2854193216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73008ea30 con 0x7fb73009a540 2021-04-09T18:18:36.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72e59c700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7300c3e40 0x7fb73008ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb72ed9d700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb73009a540 0x7fb7300c3820 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7fb72000b5e0 tx=0x7fb72000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb71f7fe700 1 -- 172.21.15.32:0/2854193216 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb72000bd70 con 0x7fb73009a540 2021-04-09T18:18:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb71f7fe700 1 -- 172.21.15.32:0/2854193216 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb720002030 con 0x7fb73009a540 2021-04-09T18:18:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73008ea30 con 0x7fb73009a540 2021-04-09T18:18:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb71f7fe700 1 -- 172.21.15.32:0/2854193216 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb720002590 con 0x7fb73009a540 2021-04-09T18:18:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.817 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73008ea30 con 0x7fb73009a540 2021-04-09T18:18:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.818 7fb71f7fe700 1 -- 172.21.15.32:0/2854193216 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb720000c90 con 0x7fb73009a540 2021-04-09T18:18:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.818 7fb71f7fe700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb714018930 0x7fb71401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.818 7fb71f7fe700 1 -- 172.21.15.32:0/2854193216 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb720032a60 con 0x7fb73009a540 2021-04-09T18:18:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.818 7fb72e59c700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb714018930 0x7fb71401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.819 7fb72e59c700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb714018930 0x7fb71401adc0 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:36.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.838 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7300c3060 con 0x7fb73009a540 2021-04-09T18:18:36.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:36.842 7fb71f7fe700 1 -- 172.21.15.32:0/2854193216 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb72001f7a0 con 0x7fb73009a540 2021-04-09T18:18:37.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.026 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb7300c3060 con 0x7fb73009a540 2021-04-09T18:18:37.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.030 7fb71f7fe700 1 -- 172.21.15.32:0/2854193216 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v24) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb720000c90 con 0x7fb73009a540 2021-04-09T18:18:37.075 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:18:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb714018930 msgr2=0x7fb71401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb714018930 0x7fb71401adc0 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb73009a540 msgr2=0x7fb7300c3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb73009a540 0x7fb7300c3820 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7fb72000b5e0 tx=0x7fb72000b980).stop 2021-04-09T18:18:37.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 shutdown_connections 2021-04-09T18:18:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb714018930 0x7fb71401adc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7300c0550 0x7fb73008e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7300c3e40 0x7fb73008ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 --2- 172.21.15.32:0/2854193216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb73009a540 0x7fb7300c3820 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 >> 172.21.15.32:0/2854193216 conn(0x7fb7300c42e0 msgr2=0x7fb73008fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:37.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 shutdown_connections 2021-04-09T18:18:37.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.036 7fb7350c5700 1 -- 172.21.15.32:0/2854193216 wait complete. 2021-04-09T18:18:37.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:18:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.220 7f97cb0dc700 1 Processor -- start 2021-04-09T18:18:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.220 7f97cb0dc700 1 -- start start 2021-04-09T18:18:37.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97cb0dc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f97c40cffb0 0x7f97c4087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97cb0dc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40d8050 0x7f97c410d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97cb0dc700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c4088460 con 0x7f97c40d7c50 2021-04-09T18:18:37.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97cb0dc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c41b0c80 con 0x7f97c40cffb0 2021-04-09T18:18:37.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97cb0dc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40d91f0 con 0x7f97c40d8050 2021-04-09T18:18:37.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97c96f9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40d8050 0x7f97c410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97c8ef8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f97c40cffb0 0x7f97c4087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97c3fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f97c40d7c50 0x7f97c41b0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54274/0 (socket says 172.21.15.32:54274) 2021-04-09T18:18:37.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97c96f9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40d8050 0x7f97c410d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:53984/0 (socket says 172.21.15.32:53984) 2021-04-09T18:18:37.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97c8ef8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f97c40cffb0 0x7f97c4087e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:38514/0 (socket says 172.21.15.32:38514) 2021-04-09T18:18:37.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97c3fff700 1 -- 172.21.15.32:0/74217875 learned_addr learned my addr 172.21.15.32:0/74217875 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.221 7f97c37fe700 1 -- 172.21.15.32:0/74217875 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368004557 0 0) 0x7f97b80010c0 con 0x7f97c40d7c50 2021-04-09T18:18:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c37fe700 1 -- 172.21.15.32:0/74217875 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ac002fe0 con 0x7f97c40d7c50 2021-04-09T18:18:37.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c8ef8700 1 -- 172.21.15.32:0/74217875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40d8050 msgr2=0x7f97c410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c8ef8700 1 --2- 172.21.15.32:0/74217875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40d8050 0x7f97c410d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c8ef8700 1 -- 172.21.15.32:0/74217875 >> v1:172.21.15.86:6789/0 conn(0x7f97c40d7c50 legacy=0x7f97c41b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c8ef8700 1 -- 172.21.15.32:0/74217875 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40d8620 con 0x7f97c40cffb0 2021-04-09T18:18:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c96f9700 1 --2- 172.21.15.32:0/74217875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40d8050 0x7f97c410d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:37.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c8ef8700 1 --2- 172.21.15.32:0/74217875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40cffb0 0x7f97c4087e40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f97b4006650 tx=0x7f97b4009a80).ready entity=mon.1 client_cookie=135e93abfee8093d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:37.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c37fe700 1 -- 172.21.15.32:0/74217875 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b40009b0 con 0x7f97c40cffb0 2021-04-09T18:18:37.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c37fe700 1 -- 172.21.15.32:0/74217875 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b400c0c0 con 0x7f97c40cffb0 2021-04-09T18:18:37.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97c37fe700 1 -- 172.21.15.32:0/74217875 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97b400c510 con 0x7f97c40cffb0 2021-04-09T18:18:37.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97cb0dc700 1 -- 172.21.15.32:0/74217875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40cffb0 msgr2=0x7f97c4087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.222 7f97cb0dc700 1 --2- 172.21.15.32:0/74217875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40cffb0 0x7f97c4087e40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f97b4006650 tx=0x7f97b4009a80).stop 2021-04-09T18:18:37.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.223 7f97cb0dc700 1 -- 172.21.15.32:0/74217875 shutdown_connections 2021-04-09T18:18:37.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.223 7f97cb0dc700 1 --2- 172.21.15.32:0/74217875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40d8050 0x7f97c410d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.223 7f97cb0dc700 1 --2- 172.21.15.32:0/74217875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40cffb0 0x7f97c4087e40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.223 7f97cb0dc700 1 -- 172.21.15.32:0/74217875 >> 172.21.15.32:0/74217875 conn(0x7f97c40ce5b0 msgr2=0x7f97c40cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:37.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.223 7f97cb0dc700 1 -- 172.21.15.32:0/74217875 shutdown_connections 2021-04-09T18:18:37.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.223 7f97cb0dc700 1 -- 172.21.15.32:0/74217875 wait complete. 2021-04-09T18:18:37.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.243 7f97cb0dc700 1 Processor -- start 2021-04-09T18:18:37.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.243 7f97cb0dc700 1 -- start start 2021-04-09T18:18:37.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97cb0dc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40bc9b0 0x7f97c40b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97cb0dc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40b0ec0 0x7f97c40b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97cb0dc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c40b4260 0x7f97c40b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97cb0dc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40b0320 con 0x7f97c40b4260 2021-04-09T18:18:37.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97cb0dc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40b12c0 con 0x7f97c40b0ec0 2021-04-09T18:18:37.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97cb0dc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40aff80 con 0x7f97c40bc9b0 2021-04-09T18:18:37.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c96f9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c40b4260 0x7f97c40b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c96f9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c40b4260 0x7f97c40b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46766/0 (socket says 172.21.15.32:46766) 2021-04-09T18:18:37.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c96f9700 1 -- 172.21.15.32:0/3380094369 learned_addr learned my addr 172.21.15.32:0/3380094369 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:37.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c8ef8700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40bc9b0 0x7f97c40b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c3fff700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40b0ec0 0x7f97c40b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c96f9700 1 -- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40bc9b0 msgr2=0x7f97c40b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c96f9700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40bc9b0 0x7f97c40b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c96f9700 1 -- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40b0ec0 msgr2=0x7f97c40b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c96f9700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40b0ec0 0x7f97c40b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c96f9700 1 -- 172.21.15.32:0/3380094369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40b4c80 con 0x7f97c40b4260 2021-04-09T18:18:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c96f9700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c40b4260 0x7f97c40b4660 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7f97bc00b970 tx=0x7f97bc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c17fa700 1 -- 172.21.15.32:0/3380094369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97bc000f90 con 0x7f97c40b4260 2021-04-09T18:18:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c17fa700 1 -- 172.21.15.32:0/3380094369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc006aa0 con 0x7f97c40b4260 2021-04-09T18:18:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c40b4c80 con 0x7f97c40b4260 2021-04-09T18:18:37.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97c17fa700 1 -- 172.21.15.32:0/3380094369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97bc0015f0 con 0x7f97c40b4260 2021-04-09T18:18:37.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.244 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c40b4c80 con 0x7f97c40b4260 2021-04-09T18:18:37.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.245 7f97c17fa700 1 -- 172.21.15.32:0/3380094369 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f97bc000ed0 con 0x7f97c40b4260 2021-04-09T18:18:37.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.245 7f97c17fa700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97a4018930 0x7f97a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.245 7f97c17fa700 1 -- 172.21.15.32:0/3380094369 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f97bc032da0 con 0x7f97c40b4260 2021-04-09T18:18:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.245 7f97c8ef8700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97a4018930 0x7f97a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.246 7f97c8ef8700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97a4018930 0x7f97a401ae10 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:37.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.265 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c4093fc0 con 0x7f97c40b4260 2021-04-09T18:18:37.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.269 7f97c17fa700 1 -- 172.21.15.32:0/3380094369 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f97bc0066f0 con 0x7f97c40b4260 2021-04-09T18:18:37.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.454 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f97c4093fc0 con 0x7f97c40b4260 2021-04-09T18:18:37.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.458 7f97c17fa700 1 -- 172.21.15.32:0/3380094369 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v25) v1 ==== 184+0+0 (secure 0 0 0) 0x7f97bc000ed0 con 0x7f97c40b4260 2021-04-09T18:18:37.499 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:18:37.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97a4018930 msgr2=0x7f97a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97a4018930 0x7f97a401ae10 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c40b4260 msgr2=0x7f97c40b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c40b4260 0x7f97c40b4660 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7f97bc00b970 tx=0x7f97bc00bd10).stop 2021-04-09T18:18:37.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 shutdown_connections 2021-04-09T18:18:37.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97a4018930 0x7f97a401ae10 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97c40b4260 0x7f97c40b4660 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40b0ec0 0x7f97c40b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 --2- 172.21.15.32:0/3380094369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c40bc9b0 0x7f97c40b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 >> 172.21.15.32:0/3380094369 conn(0x7f97c41889b0 msgr2=0x7f97c4189210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:37.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 shutdown_connections 2021-04-09T18:18:37.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.464 7f97cb0dc700 1 -- 172.21.15.32:0/3380094369 wait complete. 2021-04-09T18:18:37.513 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.633 7fb130061700 1 Processor -- start 2021-04-09T18:18:37.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.633 7fb130061700 1 -- start start 2021-04-09T18:18:37.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.634 7fb130061700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb12810da00 0x7fb12810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.634 7fb130061700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb12810d480 con 0x7fb1280d7ca0 2021-04-09T18:18:37.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.634 7fb130061700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1280d1f70 con 0x7fb12810d010 2021-04-09T18:18:37.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.634 7fb130061700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1280d8d30 con 0x7fb12810da00 2021-04-09T18:18:37.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.634 7fb12e67e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb12810da00 0x7fb12810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.634 7fb12d67c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb1280d7ca0 0x7fb1280d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56360/0 (socket says 172.21.15.32:56360) 2021-04-09T18:18:37.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.634 7fb12d67c700 1 -- 172.21.15.32:0/3311424265 learned_addr learned my addr 172.21.15.32:0/3311424265 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb12e67e700 1 -- 172.21.15.32:0/3311424265 >> v1:172.21.15.86:6789/0 conn(0x7fb12810d010 legacy=0x7fb1280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb12e67e700 1 -- 172.21.15.32:0/3311424265 >> v1:172.21.15.32:6789/0 conn(0x7fb1280d7ca0 legacy=0x7fb1280d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb12ce7b700 1 -- 172.21.15.32:0/3311424265 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467783812 0 0) 0x7fb118000ce0 con 0x7fb1280d7ca0 2021-04-09T18:18:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb12e67e700 1 -- 172.21.15.32:0/3311424265 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1280d9220 con 0x7fb12810da00 2021-04-09T18:18:37.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb12e67e700 1 --2- 172.21.15.32:0/3311424265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb12810da00 0x7fb12810de00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fb120006e30 tx=0x7fb120009de0).ready entity=mon.0 client_cookie=33de7e92d91687ff server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb12ce7b700 1 -- 172.21.15.32:0/3311424265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb12000b300 con 0x7fb12810da00 2021-04-09T18:18:37.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb12ce7b700 1 -- 172.21.15.32:0/3311424265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12000b680 con 0x7fb12810da00 2021-04-09T18:18:37.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb12ce7b700 1 -- 172.21.15.32:0/3311424265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb12000bb60 con 0x7fb12810da00 2021-04-09T18:18:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb130061700 1 -- 172.21.15.32:0/3311424265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb12810da00 msgr2=0x7fb12810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.635 7fb130061700 1 --2- 172.21.15.32:0/3311424265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb12810da00 0x7fb12810de00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fb120006e30 tx=0x7fb120009de0).stop 2021-04-09T18:18:37.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.636 7fb130061700 1 -- 172.21.15.32:0/3311424265 shutdown_connections 2021-04-09T18:18:37.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.636 7fb130061700 1 --2- 172.21.15.32:0/3311424265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb12810da00 0x7fb12810de00 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.636 7fb130061700 1 -- 172.21.15.32:0/3311424265 >> 172.21.15.32:0/3311424265 conn(0x7fb1280ce5f0 msgr2=0x7fb1280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:37.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.636 7fb130061700 1 -- 172.21.15.32:0/3311424265 shutdown_connections 2021-04-09T18:18:37.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.636 7fb130061700 1 -- 172.21.15.32:0/3311424265 wait complete. 2021-04-09T18:18:37.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.656 7fb130061700 1 Processor -- start 2021-04-09T18:18:37.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.656 7fb130061700 1 -- start start 2021-04-09T18:18:37.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb130061700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb12817e010 0x7fb12817e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb130061700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb12809f290 0x7fb12809f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb130061700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1280b1b70 0x7fb1280b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb130061700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb128182a50 con 0x7fb1280b1b70 2021-04-09T18:18:37.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb130061700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12809fcb0 con 0x7fb12809f290 2021-04-09T18:18:37.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb130061700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280b2590 con 0x7fb12817e010 2021-04-09T18:18:37.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12e67e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1280b1b70 0x7fb1280b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12e67e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1280b1b70 0x7fb1280b1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46778/0 (socket says 172.21.15.32:46778) 2021-04-09T18:18:37.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12e67e700 1 -- 172.21.15.32:0/3510992990 learned_addr learned my addr 172.21.15.32:0/3510992990 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:37.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12de7d700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb12817e010 0x7fb12817e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12d67c700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb12809f290 0x7fb12809f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12e67e700 1 -- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb12817e010 msgr2=0x7fb12817e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12e67e700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb12817e010 0x7fb12817e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12e67e700 1 -- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb12809f290 msgr2=0x7fb12809f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12e67e700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb12809f290 0x7fb12809f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12e67e700 1 -- 172.21.15.32:0/3510992990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1280b27b0 con 0x7fb1280b1b70 2021-04-09T18:18:37.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb12e67e700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1280b1b70 0x7fb1280b1f70 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fb120009fc0 tx=0x7fb12000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:37.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb11e7fc700 1 -- 172.21.15.32:0/3510992990 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1200093d0 con 0x7fb1280b1b70 2021-04-09T18:18:37.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb11e7fc700 1 -- 172.21.15.32:0/3510992990 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb120009690 con 0x7fb1280b1b70 2021-04-09T18:18:37.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb130061700 1 -- 172.21.15.32:0/3510992990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1280b27b0 con 0x7fb1280b1b70 2021-04-09T18:18:37.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb11e7fc700 1 -- 172.21.15.32:0/3510992990 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb120005a10 con 0x7fb1280b1b70 2021-04-09T18:18:37.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.657 7fb130061700 1 -- 172.21.15.32:0/3510992990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1280b27b0 con 0x7fb1280b1b70 2021-04-09T18:18:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.658 7fb11e7fc700 1 -- 172.21.15.32:0/3510992990 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb120006bd0 con 0x7fb1280b1b70 2021-04-09T18:18:37.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.658 7fb11e7fc700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb110018930 0x7fb11001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.658 7fb11e7fc700 1 -- 172.21.15.32:0/3510992990 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb120032dd0 con 0x7fb1280b1b70 2021-04-09T18:18:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.658 7fb12de7d700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb110018930 0x7fb11001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.658 7fb12de7d700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb110018930 0x7fb11001ae10 crc :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.678 7fb130061700 1 -- 172.21.15.32:0/3510992990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12809ed60 con 0x7fb1280b1b70 2021-04-09T18:18:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.682 7fb11e7fc700 1 -- 172.21.15.32:0/3510992990 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb120005cd0 con 0x7fb1280b1b70 2021-04-09T18:18:37.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.876 7fb130061700 1 -- 172.21.15.32:0/3510992990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb12809ed60 con 0x7fb1280b1b70 2021-04-09T18:18:37.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.876 7fb11e7fc700 1 -- 172.21.15.32:0/3510992990 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2779 (secure 0 0 0) 0x7fb120006bd0 con 0x7fb1280b1b70 2021-04-09T18:18:37.915 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2021-04-09T18:18:37.917 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:37.917 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6654,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:35.676838","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 -- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb110018930 msgr2=0x7fb11001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb110018930 0x7fb11001ae10 crc :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 -- 172.21.15.32:0/3510992990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1280b1b70 msgr2=0x7fb1280b1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:37.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1280b1b70 0x7fb1280b1f70 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fb120009fc0 tx=0x7fb12000bd40).stop 2021-04-09T18:18:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 -- 172.21.15.32:0/3510992990 shutdown_connections 2021-04-09T18:18:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb110018930 0x7fb11001ae10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1280b1b70 0x7fb1280b1f70 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb12809f290 0x7fb12809f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 --2- 172.21.15.32:0/3510992990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb12817e010 0x7fb12817e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:37.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 -- 172.21.15.32:0/3510992990 >> 172.21.15.32:0/3510992990 conn(0x7fb1281965a0 msgr2=0x7fb12819dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:37.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 -- 172.21.15.32:0/3510992990 shutdown_connections 2021-04-09T18:18:37.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:37.879 7fb130061700 1 -- 172.21.15.32:0/3510992990 wait complete. 2021-04-09T18:18:38.349 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:35.676838', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:38.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:18:38.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:18:38.488 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6526, 2021-04-09T18:18:38.489 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:18:38.490 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 4.7862475609999997, 2021-04-09T18:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 91, 2021-04-09T18:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 84, 2021-04-09T18:18:38.491 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 84, 2021-04-09T18:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 9.2432503589999992 2021-04-09T18:18:38.492 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:38.496 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 6526, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.7862475609999997, "mdsmap_epoch": 91, "osdmap_epoch": 84, "osdmap_epoch_barrier": 84, "uptime": 9.2432503589999992 } 2021-04-09T18:18:38.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:18:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.614 7f2e35627700 1 Processor -- start 2021-04-09T18:18:38.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.614 7f2e35627700 1 -- start start 2021-04-09T18:18:38.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.615 7f2e35627700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e3010d010 0x7f2e300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:38.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.615 7f2e35627700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e300d1da0 0x7f2e300d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:38.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.615 7f2e35627700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3010d480 con 0x7f2e300d82c0 2021-04-09T18:18:38.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.615 7f2e35627700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300d8d30 con 0x7f2e300d1da0 2021-04-09T18:18:38.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.615 7f2e35627700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300d9220 con 0x7f2e3010d010 2021-04-09T18:18:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.615 7f2e2eabe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e300d1da0 0x7f2e300d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:38.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.615 7f2e2fac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2e300d82c0 0x7f2e3010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56374/0 (socket says 172.21.15.32:56374) 2021-04-09T18:18:38.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.615 7f2e2fac0700 1 -- 172.21.15.32:0/3955815401 learned_addr learned my addr 172.21.15.32:0/3955815401 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:38.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.615 7f2e2f2bf700 1 --2- 172.21.15.32:0/3955815401 >> v2:172.21.15.86:3300/0 conn(0x7f2e3010d010 0x7f2e300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e2eabe700 1 -- 172.21.15.32:0/3955815401 >> v2:172.21.15.86:3300/0 conn(0x7f2e3010d010 msgr2=0x7f2e300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:38.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e2eabe700 1 --2- 172.21.15.32:0/3955815401 >> v2:172.21.15.86:3300/0 conn(0x7f2e3010d010 0x7f2e300d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e2eabe700 1 -- 172.21.15.32:0/3955815401 >> v1:172.21.15.32:6789/0 conn(0x7f2e300d82c0 legacy=0x7f2e3010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:38.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e2e2bd700 1 -- 172.21.15.32:0/3955815401 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656742141 0 0) 0x7f2e28000d30 con 0x7f2e300d82c0 2021-04-09T18:18:38.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e2eabe700 1 -- 172.21.15.32:0/3955815401 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e300d86f0 con 0x7f2e300d1da0 2021-04-09T18:18:38.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e2eabe700 1 --2- 172.21.15.32:0/3955815401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e300d1da0 0x7f2e300d7ca0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7f2e24006650 tx=0x7f2e24009a00).ready entity=mon.0 client_cookie=2a120b09e4ddfd58 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:38.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e2e2bd700 1 -- 172.21.15.32:0/3955815401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e24009df0 con 0x7f2e300d1da0 2021-04-09T18:18:38.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e2e2bd700 1 -- 172.21.15.32:0/3955815401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e240009b0 con 0x7f2e300d1da0 2021-04-09T18:18:38.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e2e2bd700 1 -- 172.21.15.32:0/3955815401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e2400c2b0 con 0x7f2e300d1da0 2021-04-09T18:18:38.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e35627700 1 -- 172.21.15.32:0/3955815401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e300d1da0 msgr2=0x7f2e300d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:38.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.616 7f2e35627700 1 --2- 172.21.15.32:0/3955815401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e300d1da0 0x7f2e300d7ca0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7f2e24006650 tx=0x7f2e24009a00).stop 2021-04-09T18:18:38.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.617 7f2e35627700 1 -- 172.21.15.32:0/3955815401 shutdown_connections 2021-04-09T18:18:38.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.617 7f2e35627700 1 --2- 172.21.15.32:0/3955815401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e300d1da0 0x7f2e300d7ca0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.617 7f2e35627700 1 --2- 172.21.15.32:0/3955815401 >> v2:172.21.15.86:3300/0 conn(0x7f2e3010d010 0x7f2e300d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.617 7f2e35627700 1 -- 172.21.15.32:0/3955815401 >> 172.21.15.32:0/3955815401 conn(0x7f2e300ce5f0 msgr2=0x7f2e300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.617 7f2e35627700 1 -- 172.21.15.32:0/3955815401 shutdown_connections 2021-04-09T18:18:38.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.617 7f2e35627700 1 -- 172.21.15.32:0/3955815401 wait complete. 2021-04-09T18:18:38.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.637 7f2e35627700 1 Processor -- start 2021-04-09T18:18:38.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.637 7f2e35627700 1 -- start start 2021-04-09T18:18:38.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e35627700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e301aca20 0x7f2e301ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:38.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e35627700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e3018bd40 0x7f2e3018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:38.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e35627700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3008fba0 0x7f2e3008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:38.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e35627700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3018c7e0 con 0x7f2e3018bd40 2021-04-09T18:18:38.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e35627700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3018ca30 con 0x7f2e301aca20 2021-04-09T18:18:38.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e35627700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300905c0 con 0x7f2e3008fba0 2021-04-09T18:18:38.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2eabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e3018bd40 0x7f2e3018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:38.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2eabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e3018bd40 0x7f2e3018c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46788/0 (socket says 172.21.15.32:46788) 2021-04-09T18:18:38.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2eabe700 1 -- 172.21.15.32:0/606878311 learned_addr learned my addr 172.21.15.32:0/606878311 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:38.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2fac0700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3008fba0 0x7f2e3008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:38.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2f2bf700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e301aca20 0x7f2e301ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:38.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2eabe700 1 -- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3008fba0 msgr2=0x7f2e3008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2eabe700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3008fba0 0x7f2e3008ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2eabe700 1 -- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e301aca20 msgr2=0x7f2e301ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:38.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2eabe700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e301aca20 0x7f2e301ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2eabe700 1 -- 172.21.15.32:0/606878311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e300b4300 con 0x7f2e3018bd40 2021-04-09T18:18:38.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e2eabe700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e3018bd40 0x7f2e3018c140 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f2e2400b580 tx=0x7f2e2400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:38.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e17fff700 1 -- 172.21.15.32:0/606878311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e24000c90 con 0x7f2e3018bd40 2021-04-09T18:18:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e17fff700 1 -- 172.21.15.32:0/606878311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e24002030 con 0x7f2e3018bd40 2021-04-09T18:18:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e35627700 1 -- 172.21.15.32:0/606878311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e300b4300 con 0x7f2e3018bd40 2021-04-09T18:18:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e17fff700 1 -- 172.21.15.32:0/606878311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e24002480 con 0x7f2e3018bd40 2021-04-09T18:18:38.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.638 7f2e35627700 1 -- 172.21.15.32:0/606878311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300b4300 con 0x7f2e3018bd40 2021-04-09T18:18:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.639 7f2e17fff700 1 -- 172.21.15.32:0/606878311 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2e24002210 con 0x7f2e3018bd40 2021-04-09T18:18:38.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.639 7f2e17fff700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2e10018930 0x7f2e1001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:38.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.639 7f2e17fff700 1 -- 172.21.15.32:0/606878311 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2e24032a60 con 0x7f2e3018bd40 2021-04-09T18:18:38.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.640 7f2e2f2bf700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2e10018930 0x7f2e1001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:38.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.640 7f2e2f2bf700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2e10018930 0x7f2e1001ae10 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:38.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.659 7f2e35627700 1 -- 172.21.15.32:0/606878311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e300905c0 con 0x7f2e3018bd40 2021-04-09T18:18:38.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.663 7f2e17fff700 1 -- 172.21.15.32:0/606878311 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2e24002240 con 0x7f2e3018bd40 2021-04-09T18:18:38.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.851 7f2e35627700 1 -- 172.21.15.32:0/606878311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e300905c0 con 0x7f2e3018bd40 2021-04-09T18:18:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.852 7f2e17fff700 1 -- 172.21.15.32:0/606878311 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f2e24002030 con 0x7f2e3018bd40 2021-04-09T18:18:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:38.891 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":84,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:18:33.664214","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:18:32.041217","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-04-09 18:18:33.125737","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:18:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 -- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2e10018930 msgr2=0x7f2e1001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:38.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2e10018930 0x7f2e1001ae10 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 -- 172.21.15.32:0/606878311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e3018bd40 msgr2=0x7f2e3018c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:38.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e3018bd40 0x7f2e3018c140 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f2e2400b580 tx=0x7f2e2400b920).stop 2021-04-09T18:18:38.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 -- 172.21.15.32:0/606878311 shutdown_connections 2021-04-09T18:18:38.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2e10018930 0x7f2e1001ae10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e3018bd40 0x7f2e3018c140 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3008fba0 0x7f2e3008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 --2- 172.21.15.32:0/606878311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e301aca20 0x7f2e301ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 -- 172.21.15.32:0/606878311 >> 172.21.15.32:0/606878311 conn(0x7f2e3019b460 msgr2=0x7f2e3019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 -- 172.21.15.32:0/606878311 shutdown_connections 2021-04-09T18:18:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:38.854 7f2e35627700 1 -- 172.21.15.32:0/606878311 wait complete. 2021-04-09T18:18:38.902 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:18:32.041217', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:18:33.125737', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:18:38.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:18:39.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.019 7fea28811700 1 Processor -- start 2021-04-09T18:18:39.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.019 7fea28811700 1 -- start start 2021-04-09T18:18:39.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea28811700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea240d7ca0 0x7fea240d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea28811700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fea2410da00 0x7fea2410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea28811700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2410d480 con 0x7fea2410d010 2021-04-09T18:18:39.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea28811700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea240d9220 con 0x7fea240d7ca0 2021-04-09T18:18:39.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea28811700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea240d86c0 con 0x7fea2410da00 2021-04-09T18:18:39.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea21e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fea240d7ca0 0x7fea240d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea2261c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fea2410d010 0x7fea240d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56384/0 (socket says 172.21.15.32:56384) 2021-04-09T18:18:39.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea2261c700 1 -- 172.21.15.32:0/2179997205 learned_addr learned my addr 172.21.15.32:0/2179997205 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:39.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea22e1d700 1 --2- 172.21.15.32:0/2179997205 >> v2:172.21.15.86:3300/0 conn(0x7fea2410da00 0x7fea2410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea21e1b700 1 -- 172.21.15.32:0/2179997205 >> v2:172.21.15.86:3300/0 conn(0x7fea2410da00 msgr2=0x7fea2410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:39.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea21e1b700 1 --2- 172.21.15.32:0/2179997205 >> v2:172.21.15.86:3300/0 conn(0x7fea2410da00 0x7fea2410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea21e1b700 1 -- 172.21.15.32:0/2179997205 >> v1:172.21.15.32:6789/0 conn(0x7fea2410d010 legacy=0x7fea240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea2161a700 1 -- 172.21.15.32:0/2179997205 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158382795 0 0) 0x7fea14001060 con 0x7fea2410d010 2021-04-09T18:18:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea21e1b700 1 -- 172.21.15.32:0/2179997205 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea240d1f70 con 0x7fea240d7ca0 2021-04-09T18:18:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea22e1d700 1 --2- 172.21.15.32:0/2179997205 >> v2:172.21.15.86:3300/0 conn(0x7fea2410da00 0x7fea2410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:39.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea21e1b700 1 --2- 172.21.15.32:0/2179997205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea240d7ca0 0x7fea240d80a0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7fea18006650 tx=0x7fea18009a80).ready entity=mon.0 client_cookie=4522a5c4b6d98ad server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:39.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea2161a700 1 -- 172.21.15.32:0/2179997205 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea180009b0 con 0x7fea240d7ca0 2021-04-09T18:18:39.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea2161a700 1 -- 172.21.15.32:0/2179997205 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1800c0c0 con 0x7fea240d7ca0 2021-04-09T18:18:39.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea2161a700 1 -- 172.21.15.32:0/2179997205 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea180009b0 con 0x7fea240d7ca0 2021-04-09T18:18:39.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea28811700 1 -- 172.21.15.32:0/2179997205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea240d7ca0 msgr2=0x7fea240d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.020 7fea28811700 1 --2- 172.21.15.32:0/2179997205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea240d7ca0 0x7fea240d80a0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7fea18006650 tx=0x7fea18009a80).stop 2021-04-09T18:18:39.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.021 7fea28811700 1 -- 172.21.15.32:0/2179997205 shutdown_connections 2021-04-09T18:18:39.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.021 7fea28811700 1 --2- 172.21.15.32:0/2179997205 >> v2:172.21.15.86:3300/0 conn(0x7fea2410da00 0x7fea2410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.021 7fea28811700 1 --2- 172.21.15.32:0/2179997205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea240d7ca0 0x7fea240d80a0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.021 7fea28811700 1 -- 172.21.15.32:0/2179997205 >> 172.21.15.32:0/2179997205 conn(0x7fea240ce5f0 msgr2=0x7fea240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:39.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.021 7fea28811700 1 -- 172.21.15.32:0/2179997205 shutdown_connections 2021-04-09T18:18:39.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.021 7fea28811700 1 -- 172.21.15.32:0/2179997205 wait complete. 2021-04-09T18:18:39.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.041 7fea28811700 1 Processor -- start 2021-04-09T18:18:39.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.041 7fea28811700 1 -- start start 2021-04-09T18:18:39.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea28811700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea240c3c00 0x7fea2409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea28811700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea24095240 0x7fea241a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea28811700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241a1270 0x7fea240b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea28811700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2409cda0 con 0x7fea241a1270 2021-04-09T18:18:39.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea28811700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea240c3680 con 0x7fea24095240 2021-04-09T18:18:39.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea28811700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea24095640 con 0x7fea240c3c00 2021-04-09T18:18:39.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea22e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241a1270 0x7fea240b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea22e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241a1270 0x7fea240b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46802/0 (socket says 172.21.15.32:46802) 2021-04-09T18:18:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea22e1d700 1 -- 172.21.15.32:0/4202810733 learned_addr learned my addr 172.21.15.32:0/4202810733 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea21e1b700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea24095240 0x7fea241a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea2261c700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea240c3c00 0x7fea2409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea22e1d700 1 -- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea240c3c00 msgr2=0x7fea2409c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea22e1d700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea240c3c00 0x7fea2409c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.042 7fea22e1d700 1 -- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea24095240 msgr2=0x7fea241a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.043 7fea22e1d700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea24095240 0x7fea241a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.043 7fea22e1d700 1 -- 172.21.15.32:0/4202810733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea241a1670 con 0x7fea241a1270 2021-04-09T18:18:39.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.043 7fea22e1d700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241a1270 0x7fea240b0cf0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7fea1c00b580 tx=0x7fea1c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:39.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.043 7fea12ffd700 1 -- 172.21.15.32:0/4202810733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea1c00bd10 con 0x7fea241a1270 2021-04-09T18:18:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.043 7fea12ffd700 1 -- 172.21.15.32:0/4202810733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1c006720 con 0x7fea241a1270 2021-04-09T18:18:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.043 7fea28811700 1 -- 172.21.15.32:0/4202810733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea241a1670 con 0x7fea241a1270 2021-04-09T18:18:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.043 7fea12ffd700 1 -- 172.21.15.32:0/4202810733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea1c0024d0 con 0x7fea241a1270 2021-04-09T18:18:39.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.043 7fea28811700 1 -- 172.21.15.32:0/4202810733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea241a1670 con 0x7fea241a1270 2021-04-09T18:18:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.044 7fea12ffd700 1 -- 172.21.15.32:0/4202810733 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fea1c005330 con 0x7fea241a1270 2021-04-09T18:18:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.044 7fea12ffd700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea08018930 0x7fea0801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.044 7fea12ffd700 1 -- 172.21.15.32:0/4202810733 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fea1c0329e0 con 0x7fea241a1270 2021-04-09T18:18:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.044 7fea2261c700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea08018930 0x7fea0801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.044 7fea2261c700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea08018930 0x7fea0801adc0 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:39.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.064 7fea28811700 1 -- 172.21.15.32:0/4202810733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea24093d50 con 0x7fea241a1270 2021-04-09T18:18:39.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.067 7fea12ffd700 1 -- 172.21.15.32:0/4202810733 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fea1c00bd10 con 0x7fea241a1270 2021-04-09T18:18:39.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.246 7fea28811700 1 -- 172.21.15.32:0/4202810733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fea2410d480 con 0x7fea241a1270 2021-04-09T18:18:39.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.246 7fea12ffd700 1 -- 172.21.15.32:0/4202810733 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v84) v1 ==== 92+0+10 (secure 0 0 0) 0x7fea1c000e70 con 0x7fea241a1270 2021-04-09T18:18:39.285 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:18:39.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 -- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea08018930 msgr2=0x7fea0801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea08018930 0x7fea0801adc0 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 -- 172.21.15.32:0/4202810733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241a1270 msgr2=0x7fea240b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241a1270 0x7fea240b0cf0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7fea1c00b580 tx=0x7fea1c00b920).stop 2021-04-09T18:18:39.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 -- 172.21.15.32:0/4202810733 shutdown_connections 2021-04-09T18:18:39.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea08018930 0x7fea0801adc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea241a1270 0x7fea240b0cf0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea24095240 0x7fea241a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 --2- 172.21.15.32:0/4202810733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea240c3c00 0x7fea2409c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 -- 172.21.15.32:0/4202810733 >> 172.21.15.32:0/4202810733 conn(0x7fea240975f0 msgr2=0x7fea24096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:39.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 -- 172.21.15.32:0/4202810733 shutdown_connections 2021-04-09T18:18:39.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.249 7fea28811700 1 -- 172.21.15.32:0/4202810733 wait complete. 2021-04-09T18:18:39.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:18:39.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.412 7f909f11e700 1 Processor -- start 2021-04-09T18:18:39.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.412 7f909f11e700 1 -- start start 2021-04-09T18:18:39.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.412 7f909f11e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f909810d010 0x7f90980d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.412 7f909f11e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90980d1da0 0x7f90980d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.412 7f909f11e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f90980d82c0 0x7f909810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.412 7f909f11e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90980d8d30 con 0x7f909810d010 2021-04-09T18:18:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.412 7f909f11e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90980d9220 con 0x7f90980d82c0 2021-04-09T18:18:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.412 7f909f11e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909810d480 con 0x7f90980d1da0 2021-04-09T18:18:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909cf3a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f909810d010 0x7f90980d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909cf3a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f909810d010 0x7f90980d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46808/0 (socket says 172.21.15.32:46808) 2021-04-09T18:18:39.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909cf3a700 1 -- 172.21.15.32:0/3426087730 learned_addr learned my addr 172.21.15.32:0/3426087730 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:39.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909d73b700 1 --2- 172.21.15.32:0/3426087730 >> v2:172.21.15.86:3300/0 conn(0x7f90980d82c0 0x7f909810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f9097fff700 1 --2- 172.21.15.32:0/3426087730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90980d1da0 0x7f90980d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909cf3a700 1 -- 172.21.15.32:0/3426087730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90980d1da0 msgr2=0x7f90980d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:39.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909cf3a700 1 --2- 172.21.15.32:0/3426087730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90980d1da0 0x7f90980d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909cf3a700 1 -- 172.21.15.32:0/3426087730 >> v2:172.21.15.86:3300/0 conn(0x7f90980d82c0 msgr2=0x7f909810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:39.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909cf3a700 1 --2- 172.21.15.32:0/3426087730 >> v2:172.21.15.86:3300/0 conn(0x7f90980d82c0 0x7f909810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909cf3a700 1 -- 172.21.15.32:0/3426087730 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90980d86c0 con 0x7f909810d010 2021-04-09T18:18:39.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f9097fff700 1 --2- 172.21.15.32:0/3426087730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90980d1da0 0x7f90980d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:39.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909d73b700 1 --2- 172.21.15.32:0/3426087730 >> v2:172.21.15.86:3300/0 conn(0x7f90980d82c0 0x7f909810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:39.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909cf3a700 1 --2- 172.21.15.32:0/3426087730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f909810d010 0x7f90980d1780 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7f9080006700 tx=0x7f9080009b80).ready entity=mon.0 client_cookie=5cc0d401fc3bb451 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:39.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f90977fe700 1 -- 172.21.15.32:0/3426087730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f908000c0c0 con 0x7f909810d010 2021-04-09T18:18:39.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f90977fe700 1 -- 172.21.15.32:0/3426087730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908000c440 con 0x7f909810d010 2021-04-09T18:18:39.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f90977fe700 1 -- 172.21.15.32:0/3426087730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f908000c920 con 0x7f909810d010 2021-04-09T18:18:39.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909f11e700 1 -- 172.21.15.32:0/3426087730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f909810d010 msgr2=0x7f90980d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.413 7f909f11e700 1 --2- 172.21.15.32:0/3426087730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f909810d010 0x7f90980d1780 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7f9080006700 tx=0x7f9080009b80).stop 2021-04-09T18:18:39.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.414 7f909f11e700 1 -- 172.21.15.32:0/3426087730 shutdown_connections 2021-04-09T18:18:39.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.414 7f909f11e700 1 --2- 172.21.15.32:0/3426087730 >> v2:172.21.15.86:3300/0 conn(0x7f90980d82c0 0x7f909810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.414 7f909f11e700 1 --2- 172.21.15.32:0/3426087730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90980d1da0 0x7f90980d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.414 7f909f11e700 1 --2- 172.21.15.32:0/3426087730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f909810d010 0x7f90980d1780 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.414 7f909f11e700 1 -- 172.21.15.32:0/3426087730 >> 172.21.15.32:0/3426087730 conn(0x7f90980ce5f0 msgr2=0x7f90980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:39.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.414 7f909f11e700 1 -- 172.21.15.32:0/3426087730 shutdown_connections 2021-04-09T18:18:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.414 7f909f11e700 1 -- 172.21.15.32:0/3426087730 wait complete. 2021-04-09T18:18:39.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.434 7f909f11e700 1 Processor -- start 2021-04-09T18:18:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.434 7f909f11e700 1 -- start start 2021-04-09T18:18:39.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909f11e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90980ae2b0 0x7f90980c03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909f11e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9098179610 0x7f9098196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909f11e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098179a10 0x7f9098196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909f11e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90980c0be0 con 0x7f9098179a10 2021-04-09T18:18:39.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909f11e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098197480 con 0x7f90980ae2b0 2021-04-09T18:18:39.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909f11e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90981976a0 con 0x7f9098179610 2021-04-09T18:18:39.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909d73b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098179a10 0x7f9098196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909d73b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098179a10 0x7f9098196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46814/0 (socket says 172.21.15.32:46814) 2021-04-09T18:18:39.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909d73b700 1 -- 172.21.15.32:0/3926210089 learned_addr learned my addr 172.21.15.32:0/3926210089 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:39.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f9097fff700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9098179610 0x7f9098196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909cf3a700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90980ae2b0 0x7f90980c03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909d73b700 1 -- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9098179610 msgr2=0x7f9098196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909d73b700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9098179610 0x7f9098196840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909d73b700 1 -- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90980ae2b0 msgr2=0x7f90980c03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909d73b700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90980ae2b0 0x7f90980c03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909d73b700 1 -- 172.21.15.32:0/3926210089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90980ac4e0 con 0x7f9098179a10 2021-04-09T18:18:39.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f9097fff700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9098179610 0x7f9098196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:39.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909cf3a700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90980ae2b0 0x7f90980c03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:39.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f909d73b700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098179a10 0x7f9098196e60 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f90900099a0 tx=0x7f909000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:39.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.435 7f90957fa700 1 -- 172.21.15.32:0/3926210089 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90900011f0 con 0x7f9098179a10 2021-04-09T18:18:39.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.436 7f90957fa700 1 -- 172.21.15.32:0/3926210089 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9090003030 con 0x7f9098179a10 2021-04-09T18:18:39.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.436 7f90957fa700 1 -- 172.21.15.32:0/3926210089 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9090001570 con 0x7f9098179a10 2021-04-09T18:18:39.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.436 7f909f11e700 1 -- 172.21.15.32:0/3926210089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90980ac4e0 con 0x7f9098179a10 2021-04-09T18:18:39.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.436 7f909f11e700 1 -- 172.21.15.32:0/3926210089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90980ac4e0 con 0x7f9098179a10 2021-04-09T18:18:39.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.436 7f90957fa700 1 -- 172.21.15.32:0/3926210089 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9090006950 con 0x7f9098179a10 2021-04-09T18:18:39.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.436 7f90957fa700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f907c018930 0x7f907c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.436 7f90957fa700 1 -- 172.21.15.32:0/3926210089 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9090032d60 con 0x7f9098179a10 2021-04-09T18:18:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.436 7f909cf3a700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f907c018930 0x7f907c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.437 7f909cf3a700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f907c018930 0x7f907c01adc0 crc :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.456 7f909f11e700 1 -- 172.21.15.32:0/3926210089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909810d480 con 0x7f9098179a10 2021-04-09T18:18:39.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.460 7f90957fa700 1 -- 172.21.15.32:0/3926210089 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9090003010 con 0x7f9098179a10 2021-04-09T18:18:39.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.641 7f909f11e700 1 -- 172.21.15.32:0/3926210089 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f909810d480 con 0x7f9098179a10 2021-04-09T18:18:39.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.642 7f90957fa700 1 -- 172.21.15.32:0/3926210089 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v84) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9090003010 con 0x7f9098179a10 2021-04-09T18:18:39.681 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:18:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 -- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f907c018930 msgr2=0x7f907c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f907c018930 0x7f907c01adc0 crc :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 -- 172.21.15.32:0/3926210089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098179a10 msgr2=0x7f9098196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098179a10 0x7f9098196e60 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f90900099a0 tx=0x7f909000bcf0).stop 2021-04-09T18:18:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 -- 172.21.15.32:0/3926210089 shutdown_connections 2021-04-09T18:18:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f907c018930 0x7f907c01adc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9098179a10 0x7f9098196e60 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9098179610 0x7f9098196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 --2- 172.21.15.32:0/3926210089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90980ae2b0 0x7f90980c03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 -- 172.21.15.32:0/3926210089 >> 172.21.15.32:0/3926210089 conn(0x7f9098095040 msgr2=0x7f90980bd220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 -- 172.21.15.32:0/3926210089 shutdown_connections 2021-04-09T18:18:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.644 7f909f11e700 1 -- 172.21.15.32:0/3926210089 wait complete. 2021-04-09T18:18:39.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:18:39.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.808 7efe2ec0d700 1 Processor -- start 2021-04-09T18:18:39.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.808 7efe2ec0d700 1 -- start start 2021-04-09T18:18:39.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.809 7efe2ec0d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2810d010 0x7efe280d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.809 7efe2ec0d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efe280d1da0 0x7efe280d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.809 7efe2ec0d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2810d480 con 0x7efe280d82c0 2021-04-09T18:18:39.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.809 7efe2ec0d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2810e780 con 0x7efe280d1da0 2021-04-09T18:18:39.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.809 7efe2ec0d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe280d8d30 con 0x7efe2810d010 2021-04-09T18:18:39.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.809 7efe27fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efe280d1da0 0x7efe280d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.809 7efe2ca29700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2810d010 0x7efe280d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.809 7efe2d22a700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efe280d82c0 0x7efe2810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54340/0 (socket says 172.21.15.32:54340) 2021-04-09T18:18:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.809 7efe2d22a700 1 -- 172.21.15.32:0/2280420865 learned_addr learned my addr 172.21.15.32:0/2280420865 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:39.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe2ca29700 1 -- 172.21.15.32:0/2280420865 >> v2:172.21.15.86:3300/0 conn(0x7efe280d1da0 msgr2=0x7efe280d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:39.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe2ca29700 1 --2- 172.21.15.32:0/2280420865 >> v2:172.21.15.86:3300/0 conn(0x7efe280d1da0 0x7efe280d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe2ca29700 1 -- 172.21.15.32:0/2280420865 >> v1:172.21.15.86:6789/0 conn(0x7efe280d82c0 legacy=0x7efe2810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe2ca29700 1 -- 172.21.15.32:0/2280420865 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe280d9220 con 0x7efe2810d010 2021-04-09T18:18:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe277fe700 1 -- 172.21.15.32:0/2280420865 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504848222 0 0) 0x7efe200010c0 con 0x7efe280d82c0 2021-04-09T18:18:39.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe27fff700 1 --2- 172.21.15.32:0/2280420865 >> v2:172.21.15.86:3300/0 conn(0x7efe280d1da0 0x7efe280d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:39.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe2ca29700 1 --2- 172.21.15.32:0/2280420865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2810d010 0x7efe280d1780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7efe10006650 tx=0x7efe10009a80).ready entity=mon.2 client_cookie=98850400a7821348 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:39.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe277fe700 1 -- 172.21.15.32:0/2280420865 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe100009b0 con 0x7efe2810d010 2021-04-09T18:18:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe277fe700 1 -- 172.21.15.32:0/2280420865 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1000c0c0 con 0x7efe2810d010 2021-04-09T18:18:39.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe277fe700 1 -- 172.21.15.32:0/2280420865 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe1000c510 con 0x7efe2810d010 2021-04-09T18:18:39.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe2ec0d700 1 -- 172.21.15.32:0/2280420865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2810d010 msgr2=0x7efe280d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:39.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.810 7efe2ec0d700 1 --2- 172.21.15.32:0/2280420865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2810d010 0x7efe280d1780 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7efe10006650 tx=0x7efe10009a80).stop 2021-04-09T18:18:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.811 7efe2ec0d700 1 -- 172.21.15.32:0/2280420865 shutdown_connections 2021-04-09T18:18:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.811 7efe2ec0d700 1 --2- 172.21.15.32:0/2280420865 >> v2:172.21.15.86:3300/0 conn(0x7efe280d1da0 0x7efe280d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.811 7efe2ec0d700 1 --2- 172.21.15.32:0/2280420865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2810d010 0x7efe280d1780 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:39.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.811 7efe2ec0d700 1 -- 172.21.15.32:0/2280420865 >> 172.21.15.32:0/2280420865 conn(0x7efe280ce5f0 msgr2=0x7efe280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:39.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.811 7efe2ec0d700 1 -- 172.21.15.32:0/2280420865 shutdown_connections 2021-04-09T18:18:39.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.811 7efe2ec0d700 1 -- 172.21.15.32:0/2280420865 wait complete. 2021-04-09T18:18:39.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.831 7efe2ec0d700 1 Processor -- start 2021-04-09T18:18:39.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.831 7efe2ec0d700 1 -- start start 2021-04-09T18:18:39.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.831 7efe2ec0d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe280a0110 0x7efe2818c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.831 7efe2ec0d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a0510 0x7efe280c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.831 7efe2ec0d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280c36e0 0x7efe280c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:39.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.831 7efe2ec0d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2818cf60 con 0x7efe280a0110 2021-04-09T18:18:39.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.831 7efe2ec0d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28199ea0 con 0x7efe280a0510 2021-04-09T18:18:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.831 7efe2ec0d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe28187d70 con 0x7efe280c36e0 2021-04-09T18:18:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ca29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe280a0110 0x7efe2818c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ca29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe280a0110 0x7efe2818c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46826/0 (socket says 172.21.15.32:46826) 2021-04-09T18:18:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ca29700 1 -- 172.21.15.32:0/320757005 learned_addr learned my addr 172.21.15.32:0/320757005 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:40.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2d22a700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280c36e0 0x7efe280c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:40.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe27fff700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a0510 0x7efe280c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:40.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ca29700 1 -- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280c36e0 msgr2=0x7efe280c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ca29700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280c36e0 0x7efe280c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ca29700 1 -- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a0510 msgr2=0x7efe280c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ca29700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a0510 0x7efe280c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ca29700 1 -- 172.21.15.32:0/320757005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2818d4e0 con 0x7efe280a0110 2021-04-09T18:18:40.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ca29700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe280a0110 0x7efe2818c750 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7efe10009630 tx=0x7efe1000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:40.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe257fa700 1 -- 172.21.15.32:0/320757005 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe100093a0 con 0x7efe280a0110 2021-04-09T18:18:40.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe257fa700 1 -- 172.21.15.32:0/320757005 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1000bd70 con 0x7efe280a0110 2021-04-09T18:18:40.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2818d4e0 con 0x7efe280a0110 2021-04-09T18:18:40.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe257fa700 1 -- 172.21.15.32:0/320757005 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe100054d0 con 0x7efe280a0110 2021-04-09T18:18:40.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.832 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2818d4e0 con 0x7efe280a0110 2021-04-09T18:18:40.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.833 7efe257fa700 1 -- 172.21.15.32:0/320757005 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efe100054f0 con 0x7efe280a0110 2021-04-09T18:18:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.833 7efe257fa700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe0c018930 0x7efe0c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:40.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.833 7efe257fa700 1 -- 172.21.15.32:0/320757005 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efe10032a00 con 0x7efe280a0110 2021-04-09T18:18:40.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.833 7efe27fff700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe0c018930 0x7efe0c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:40.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.834 7efe27fff700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe0c018930 0x7efe0c01ae10 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:40.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.853 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2810d480 con 0x7efe280a0110 2021-04-09T18:18:40.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:39.856 7efe257fa700 1 -- 172.21.15.32:0/320757005 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efe100052e0 con 0x7efe280a0110 2021-04-09T18:18:40.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.035 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7efe2810d480 con 0x7efe280a0110 2021-04-09T18:18:40.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.036 7efe257fa700 1 -- 172.21.15.32:0/320757005 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v84) v1 ==== 100+0+10 (secure 0 0 0) 0x7efe1000bd70 con 0x7efe280a0110 2021-04-09T18:18:40.077 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:18:40.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe0c018930 msgr2=0x7efe0c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe0c018930 0x7efe0c01ae10 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe280a0110 msgr2=0x7efe2818c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe280a0110 0x7efe2818c750 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7efe10009630 tx=0x7efe1000b980).stop 2021-04-09T18:18:40.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 shutdown_connections 2021-04-09T18:18:40.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe0c018930 0x7efe0c01ae10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280c36e0 0x7efe280c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a0510 0x7efe280c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 --2- 172.21.15.32:0/320757005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe280a0110 0x7efe2818c750 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 >> 172.21.15.32:0/320757005 conn(0x7efe280b93a0 msgr2=0x7efe280b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:40.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 shutdown_connections 2021-04-09T18:18:40.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.039 7efe2ec0d700 1 -- 172.21.15.32:0/320757005 wait complete. 2021-04-09T18:18:40.113 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:18:40.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:40.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.254 7fa194341700 1 Processor -- start 2021-04-09T18:18:40.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.254 7fa194341700 1 -- start start 2021-04-09T18:18:40.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa194341700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa18c0d7ca0 0x7fa18c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:40.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa194341700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa18c10da00 0x7fa18c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:40.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa194341700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa18c10d480 con 0x7fa18c10d010 2021-04-09T18:18:40.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa194341700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18c0d9220 con 0x7fa18c10da00 2021-04-09T18:18:40.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa194341700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18c0d86c0 con 0x7fa18c0d7ca0 2021-04-09T18:18:40.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19295e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa18c10da00 0x7fa18c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:40.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19295e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa18c10da00 0x7fa18c10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46836/0 (socket says 172.21.15.32:46836) 2021-04-09T18:18:40.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19295e700 1 -- 172.21.15.32:0/2741274999 learned_addr learned my addr 172.21.15.32:0/2741274999 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:40.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19195c700 1 --2- 172.21.15.32:0/2741274999 >> v2:172.21.15.86:3300/0 conn(0x7fa18c0d7ca0 0x7fa18c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:40.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19295e700 1 -- 172.21.15.32:0/2741274999 >> v2:172.21.15.86:3300/0 conn(0x7fa18c0d7ca0 msgr2=0x7fa18c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:40.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19295e700 1 --2- 172.21.15.32:0/2741274999 >> v2:172.21.15.86:3300/0 conn(0x7fa18c0d7ca0 0x7fa18c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19295e700 1 -- 172.21.15.32:0/2741274999 >> v1:172.21.15.86:6789/0 conn(0x7fa18c10d010 legacy=0x7fa18c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19295e700 1 -- 172.21.15.32:0/2741274999 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c0d1f70 con 0x7fa18c10da00 2021-04-09T18:18:40.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19295e700 1 --2- 172.21.15.32:0/2741274999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c10da00 0x7fa18c10de00 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fa184006f00 tx=0x7fa184009e90).ready entity=mon.0 client_cookie=b63b0003de12d545 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:40.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19115b700 1 -- 172.21.15.32:0/2741274999 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa18400b300 con 0x7fa18c10da00 2021-04-09T18:18:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19115b700 1 -- 172.21.15.32:0/2741274999 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18400b640 con 0x7fa18c10da00 2021-04-09T18:18:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.255 7fa19115b700 1 -- 172.21.15.32:0/2741274999 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa18400bb20 con 0x7fa18c10da00 2021-04-09T18:18:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.256 7fa194341700 1 -- 172.21.15.32:0/2741274999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c10da00 msgr2=0x7fa18c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.256 7fa194341700 1 --2- 172.21.15.32:0/2741274999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c10da00 0x7fa18c10de00 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fa184006f00 tx=0x7fa184009e90).stop 2021-04-09T18:18:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.256 7fa194341700 1 -- 172.21.15.32:0/2741274999 shutdown_connections 2021-04-09T18:18:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.256 7fa194341700 1 --2- 172.21.15.32:0/2741274999 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c10da00 0x7fa18c10de00 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.256 7fa194341700 1 --2- 172.21.15.32:0/2741274999 >> v2:172.21.15.86:3300/0 conn(0x7fa18c0d7ca0 0x7fa18c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.256 7fa194341700 1 -- 172.21.15.32:0/2741274999 >> 172.21.15.32:0/2741274999 conn(0x7fa18c0ce5f0 msgr2=0x7fa18c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.256 7fa194341700 1 -- 172.21.15.32:0/2741274999 shutdown_connections 2021-04-09T18:18:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.256 7fa194341700 1 -- 172.21.15.32:0/2741274999 wait complete. 2021-04-09T18:18:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.276 7fa194341700 1 Processor -- start 2021-04-09T18:18:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa194341700 1 -- start start 2021-04-09T18:18:40.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa194341700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa18c185060 0x7fa18c185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:40.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa194341700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18c186c00 0x7fa18c187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:40.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa194341700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c191360 0x7fa18c191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:40.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa194341700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c1863a0 con 0x7fa18c191360 2021-04-09T18:18:40.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa194341700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c185a60 con 0x7fa18c185060 2021-04-09T18:18:40.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa194341700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c187620 con 0x7fa18c186c00 2021-04-09T18:18:40.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19295e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c191360 0x7fa18c191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:40.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19295e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c191360 0x7fa18c191760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46844/0 (socket says 172.21.15.32:46844) 2021-04-09T18:18:40.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19295e700 1 -- 172.21.15.32:0/1240748239 learned_addr learned my addr 172.21.15.32:0/1240748239 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:40.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19195c700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18c186c00 0x7fa18c187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:40.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19215d700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa18c185060 0x7fa18c185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:40.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19295e700 1 -- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18c186c00 msgr2=0x7fa18c187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19295e700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18c186c00 0x7fa18c187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19295e700 1 -- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa18c185060 msgr2=0x7fa18c185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19295e700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa18c185060 0x7fa18c185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.277 7fa19295e700 1 -- 172.21.15.32:0/1240748239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c185460 con 0x7fa18c191360 2021-04-09T18:18:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.278 7fa19295e700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c191360 0x7fa18c191760 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7fa184009970 tx=0x7fa18400bbe0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.278 7fa182ffd700 1 -- 172.21.15.32:0/1240748239 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa184003030 con 0x7fa18c191360 2021-04-09T18:18:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.278 7fa182ffd700 1 -- 172.21.15.32:0/1240748239 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa184001270 con 0x7fa18c191360 2021-04-09T18:18:40.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.278 7fa194341700 1 -- 172.21.15.32:0/1240748239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18c185460 con 0x7fa18c191360 2021-04-09T18:18:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.278 7fa182ffd700 1 -- 172.21.15.32:0/1240748239 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa184001790 con 0x7fa18c191360 2021-04-09T18:18:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.278 7fa194341700 1 -- 172.21.15.32:0/1240748239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c185460 con 0x7fa18c191360 2021-04-09T18:18:40.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.278 7fa182ffd700 1 -- 172.21.15.32:0/1240748239 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa184005820 con 0x7fa18c191360 2021-04-09T18:18:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.279 7fa182ffd700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa174018930 0x7fa17401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:40.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.279 7fa182ffd700 1 -- 172.21.15.32:0/1240748239 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa184034f60 con 0x7fa18c191360 2021-04-09T18:18:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.279 7fa19215d700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa174018930 0x7fa17401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:40.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.279 7fa19215d700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa174018930 0x7fa17401adc0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:40.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.299 7fa194341700 1 -- 172.21.15.32:0/1240748239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c1866d0 con 0x7fa18c191360 2021-04-09T18:18:40.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.302 7fa182ffd700 1 -- 172.21.15.32:0/1240748239 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa184000cc0 con 0x7fa18c191360 2021-04-09T18:18:40.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.495 7fa194341700 1 -- 172.21.15.32:0/1240748239 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa18c1866d0 con 0x7fa18c191360 2021-04-09T18:18:40.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.495 7fa182ffd700 1 -- 172.21.15.32:0/1240748239 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2779 (secure 0 0 0) 0x7fa184021c80 con 0x7fa18c191360 2021-04-09T18:18:40.545 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2021-04-09T18:18:40.546 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:40.546 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6654,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:35.676838","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:40.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.497 7fa194341700 1 -- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa174018930 msgr2=0x7fa17401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.497 7fa194341700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa174018930 0x7fa17401adc0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.497 7fa194341700 1 -- 172.21.15.32:0/1240748239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c191360 msgr2=0x7fa18c191760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:40.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.497 7fa194341700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c191360 0x7fa18c191760 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7fa184009970 tx=0x7fa18400bbe0).stop 2021-04-09T18:18:40.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.498 7fa194341700 1 -- 172.21.15.32:0/1240748239 shutdown_connections 2021-04-09T18:18:40.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.498 7fa194341700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa174018930 0x7fa17401adc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.498 7fa194341700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa18c191360 0x7fa18c191760 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.498 7fa194341700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18c186c00 0x7fa18c187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.498 7fa194341700 1 --2- 172.21.15.32:0/1240748239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa18c185060 0x7fa18c185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.498 7fa194341700 1 -- 172.21.15.32:0/1240748239 >> 172.21.15.32:0/1240748239 conn(0x7fa18c1837d0 msgr2=0x7fa18c195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.498 7fa194341700 1 -- 172.21.15.32:0/1240748239 shutdown_connections 2021-04-09T18:18:40.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:40.498 7fa194341700 1 -- 172.21.15.32:0/1240748239 wait complete. 2021-04-09T18:18:40.974 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:35.676838', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:40.975 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:18:40.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:18:41.114 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:41.115 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:41.115 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:18:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6526, 2021-04-09T18:18:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:18:41.116 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:18:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.4117283220000001, 2021-04-09T18:18:41.117 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 91, 2021-04-09T18:18:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 84, 2021-04-09T18:18:41.118 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 84, 2021-04-09T18:18:41.119 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 11.868731465 2021-04-09T18:18:41.121 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:41.123 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 6526, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.4117283220000001, "mdsmap_epoch": 91, "osdmap_epoch": 84, "osdmap_epoch_barrier": 84, "uptime": 11.868731465 } 2021-04-09T18:18:41.124 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:18:41.125 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:18:41.125 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:18:41.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:41.127 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:41.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:18:41.286 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:18:41.287 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:18:41.347 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:18:41.348 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:18:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:18:41.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:41.352 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:18:41.397 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:18:41.398 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:41.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:18:41.504 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:18:41.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:18:41.567 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:18:41.567 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:18:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:18:41.570 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:41.572 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:18:41.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:18:41.549 7f34ea7aff80 -1 init, newargv = 0x55f996aed3a0 newargc=9ceph-fuse[50636]: starting ceph client 2021-04-09T18:18:41.588 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr: 2021-04-09T18:18:41.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[50636]: starting fuse 2021-04-09T18:18:41.618 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:18:41.618 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:18:41.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:41.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:18:41.620 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:18:41.620 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:18:41.620 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:18:41.621 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:18:41.621 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:18:41.621 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:18:41.622 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:18:41.622 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:18:41.623 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:18:41.623 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:18:41.623 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:18:41.624 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:18:41.624 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:18:41.625 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:18:41.625 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:18:41.628 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:18:41.629 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:18:41.638 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:18:41.646 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:18:41.663 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:18:41.664 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:18:41.664 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:18:41.763 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.50636.asok 2021-04-09T18:18:41.765 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.50636.asok 2021-04-09T18:18:41.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:41.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.50636.asok status 2021-04-09T18:18:41.893 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:41.894 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:18:41.894 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:18:41.894 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:18:41.895 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:18:41.895 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:18:41.895 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:18:41.895 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "50636", 2021-04-09T18:18:41.895 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:18:41.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:41.896 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:18:41.896 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:18:41.896 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6795, 2021-04-09T18:18:41.897 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:18:41.897 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:18:41.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:18:41.897 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 6795 2021-04-09T18:18:41.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:41.898 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:18:41.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:41.899 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:18:41.899 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 133857261 2021-04-09T18:18:41.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:18:41.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:41.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:18:41.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:18:41.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:18:41.900 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 133857261 2021-04-09T18:18:41.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:18:41.901 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.6795 172.21.15.32:0/133857261", 2021-04-09T18:18:41.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/133857261", 2021-04-09T18:18:41.901 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:18:41.901 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 91, 2021-04-09T18:18:41.902 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 84, 2021-04-09T18:18:41.902 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:18:41.902 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:18:41.902 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:18:41.902 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:41.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:41.918 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:18:41.919 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:41.979 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:18:41.979 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:18:41.980 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:18:41.981 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:18:42.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:18:42.172 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:42.173 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:18:42.173 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:42.179 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:18:42.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:42.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.289 7f94fc453700 1 Processor -- start 2021-04-09T18:18:42.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.289 7f94fc453700 1 -- start start 2021-04-09T18:18:42.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.290 7f94fc453700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94f40cffa0 0x7f94f40d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.290 7f94fc453700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f94f40d6c80 0x7f94f410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.290 7f94fc453700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f40d7280 con 0x7f94f40d7800 2021-04-09T18:18:42.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.290 7f94fc453700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f40d7080 con 0x7f94f40d6c80 2021-04-09T18:18:42.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.290 7f94fc453700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f40d91e0 con 0x7f94f40cffa0 2021-04-09T18:18:42.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.290 7f94faa70700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f94f40d6c80 0x7f94f410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.290 7f94f9a6e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f94f40d7800 0x7f94f40d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56452/0 (socket says 172.21.15.32:56452) 2021-04-09T18:18:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.290 7f94f9a6e700 1 -- 172.21.15.32:0/68787516 learned_addr learned my addr 172.21.15.32:0/68787516 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.290 7f94fa26f700 1 --2- 172.21.15.32:0/68787516 >> v2:172.21.15.86:3300/0 conn(0x7f94f40cffa0 0x7f94f40d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94faa70700 1 -- 172.21.15.32:0/68787516 >> v2:172.21.15.86:3300/0 conn(0x7f94f40cffa0 msgr2=0x7f94f40d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:42.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94faa70700 1 --2- 172.21.15.32:0/68787516 >> v2:172.21.15.86:3300/0 conn(0x7f94f40cffa0 0x7f94f40d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94faa70700 1 -- 172.21.15.32:0/68787516 >> v1:172.21.15.32:6789/0 conn(0x7f94f40d7800 legacy=0x7f94f40d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94f926d700 1 -- 172.21.15.32:0/68787516 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196363206 0 0) 0x7f94e8000ce0 con 0x7f94f40d7800 2021-04-09T18:18:42.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94faa70700 1 -- 172.21.15.32:0/68787516 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40d83f0 con 0x7f94f40d6c80 2021-04-09T18:18:42.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94faa70700 1 --2- 172.21.15.32:0/68787516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40d6c80 0x7f94f410cfd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f94e4006ab0 tx=0x7f94e4009ac0).ready entity=mon.0 client_cookie=186503853e00afc4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:42.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94f926d700 1 -- 172.21.15.32:0/68787516 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94e40066d0 con 0x7f94f40d6c80 2021-04-09T18:18:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94f926d700 1 -- 172.21.15.32:0/68787516 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e40022e0 con 0x7f94f40d6c80 2021-04-09T18:18:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94f926d700 1 -- 172.21.15.32:0/68787516 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94e40027c0 con 0x7f94f40d6c80 2021-04-09T18:18:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94fc453700 1 -- 172.21.15.32:0/68787516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40d6c80 msgr2=0x7f94f410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.291 7f94fc453700 1 --2- 172.21.15.32:0/68787516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40d6c80 0x7f94f410cfd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f94e4006ab0 tx=0x7f94e4009ac0).stop 2021-04-09T18:18:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.292 7f94fc453700 1 -- 172.21.15.32:0/68787516 shutdown_connections 2021-04-09T18:18:42.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.292 7f94fc453700 1 --2- 172.21.15.32:0/68787516 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40d6c80 0x7f94f410cfd0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.292 7f94fc453700 1 --2- 172.21.15.32:0/68787516 >> v2:172.21.15.86:3300/0 conn(0x7f94f40cffa0 0x7f94f40d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.292 7f94fc453700 1 -- 172.21.15.32:0/68787516 >> 172.21.15.32:0/68787516 conn(0x7f94f40ce5c0 msgr2=0x7f94f40cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:42.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.292 7f94fc453700 1 -- 172.21.15.32:0/68787516 shutdown_connections 2021-04-09T18:18:42.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.292 7f94fc453700 1 -- 172.21.15.32:0/68787516 wait complete. 2021-04-09T18:18:42.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.312 7f94fc453700 1 Processor -- start 2021-04-09T18:18:42.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.312 7f94fc453700 1 -- start start 2021-04-09T18:18:42.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fc453700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40a80c0 0x7f94f417bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fc453700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f417b1d0 0x7f94f417b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fc453700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f408dc50 0x7f94f408e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fc453700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f417bc70 con 0x7f94f40a80c0 2021-04-09T18:18:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fc453700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f408e670 con 0x7f94f408dc50 2021-04-09T18:18:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fc453700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f408e850 con 0x7f94f417b1d0 2021-04-09T18:18:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fa26f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40a80c0 0x7f94f417bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fa26f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40a80c0 0x7f94f417bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46868/0 (socket says 172.21.15.32:46868) 2021-04-09T18:18:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fa26f700 1 -- 172.21.15.32:0/3032567735 learned_addr learned my addr 172.21.15.32:0/3032567735 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94f9a6e700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f417b1d0 0x7f94f417b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94faa70700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f408dc50 0x7f94f408e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fa26f700 1 -- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f417b1d0 msgr2=0x7f94f417b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fa26f700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f417b1d0 0x7f94f417b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fa26f700 1 -- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f408dc50 msgr2=0x7f94f408e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fa26f700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f408dc50 0x7f94f408e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fa26f700 1 -- 172.21.15.32:0/3032567735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f408c660 con 0x7f94f40a80c0 2021-04-09T18:18:42.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94fa26f700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40a80c0 0x7f94f417bfe0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f94ec0099e0 tx=0x7f94ec00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.313 7f94f2ffd700 1 -- 172.21.15.32:0/3032567735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94ec00c2f0 con 0x7f94f40a80c0 2021-04-09T18:18:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.314 7f94f2ffd700 1 -- 172.21.15.32:0/3032567735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec00c5f0 con 0x7f94f40a80c0 2021-04-09T18:18:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.314 7f94fc453700 1 -- 172.21.15.32:0/3032567735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f408c660 con 0x7f94f40a80c0 2021-04-09T18:18:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.314 7f94f2ffd700 1 -- 172.21.15.32:0/3032567735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94ec00cad0 con 0x7f94f40a80c0 2021-04-09T18:18:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.314 7f94fc453700 1 -- 172.21.15.32:0/3032567735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f408c660 con 0x7f94f40a80c0 2021-04-09T18:18:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.314 7f94f2ffd700 1 -- 172.21.15.32:0/3032567735 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f94ec00a020 con 0x7f94f40a80c0 2021-04-09T18:18:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.314 7f94f2ffd700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94d4018930 0x7f94d401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.314 7f94f2ffd700 1 -- 172.21.15.32:0/3032567735 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f94ec032dd0 con 0x7f94f40a80c0 2021-04-09T18:18:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.315 7f94f9a6e700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94d4018930 0x7f94d401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.315 7f94f9a6e700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94d4018930 0x7f94d401ae10 crc :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.334 7f94fc453700 1 -- 172.21.15.32:0/3032567735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f41b70c0 con 0x7f94f40a80c0 2021-04-09T18:18:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.338 7f94f2ffd700 1 -- 172.21.15.32:0/3032567735 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f94ec01fb10 con 0x7f94f40a80c0 2021-04-09T18:18:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.532 7f94fc453700 1 -- 172.21.15.32:0/3032567735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94f41b70c0 con 0x7f94f40a80c0 2021-04-09T18:18:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.532 7f94f2ffd700 1 -- 172.21.15.32:0/3032567735 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2779 (secure 0 0 0) 0x7f94ec009fe0 con 0x7f94f40a80c0 2021-04-09T18:18:42.572 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2021-04-09T18:18:42.572 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:42.572 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6654,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:35.676838","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 -- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94d4018930 msgr2=0x7f94d401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94d4018930 0x7f94d401ae10 crc :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 -- 172.21.15.32:0/3032567735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40a80c0 msgr2=0x7f94f417bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40a80c0 0x7f94f417bfe0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f94ec0099e0 tx=0x7f94ec00bd40).stop 2021-04-09T18:18:42.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 -- 172.21.15.32:0/3032567735 shutdown_connections 2021-04-09T18:18:42.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94d4018930 0x7f94d401ae10 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f408dc50 0x7f94f408e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f417b1d0 0x7f94f417b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 --2- 172.21.15.32:0/3032567735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94f40a80c0 0x7f94f417bfe0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 -- 172.21.15.32:0/3032567735 >> 172.21.15.32:0/3032567735 conn(0x7f94f4189b00 msgr2=0x7f94f41921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 -- 172.21.15.32:0/3032567735 shutdown_connections 2021-04-09T18:18:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.535 7f94fc453700 1 -- 172.21.15.32:0/3032567735 wait complete. 2021-04-09T18:18:42.584 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 91, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 7, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:18:33.664336", "damaged": [], "data_pools": [ 15 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6526": { "addr": "172.21.15.32:6835/2067624954", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6833", "nonce": 2067624954, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 2067624954, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6526, "incarnation": 88, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 14, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:18:35.676838", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6526 } } } ], "standbys": [ { "addr": "172.21.15.32:6834/1291090515", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 1291090515, "type": "v2" }, { "addr": "172.21.15.32:6834", "nonce": 1291090515, "type": "v1" } ] }, "epoch": 87, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6654, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6836/2928911362", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 2928911362, "type": "v2" }, { "addr": "172.21.15.86:6836", "nonce": 2928911362, "type": "v1" } ] }, "epoch": 87, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55799, "incarnation": 0, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6837/3460852521", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6835", "nonce": 3460852521, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 3460852521, "type": "v1" } ] }, "epoch": 87, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55802, "incarnation": 0, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:18:42.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:42.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.702 7fa20cd22700 1 Processor -- start 2021-04-09T18:18:42.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.702 7fa20cd22700 1 -- start start 2021-04-09T18:18:42.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.703 7fa20cd22700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa20810da00 0x7fa20810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.703 7fa20cd22700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20810d480 con 0x7fa20810d010 2021-04-09T18:18:42.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.703 7fa20cd22700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2080d1f70 con 0x7fa2080d7ca0 2021-04-09T18:18:42.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.703 7fa20cd22700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2080d8d30 con 0x7fa20810da00 2021-04-09T18:18:42.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.703 7fa207314700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa20810da00 0x7fa20810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.703 7fa206b13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa20810d010 0x7fa2080d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56466/0 (socket says 172.21.15.32:56466) 2021-04-09T18:18:42.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.703 7fa206b13700 1 -- 172.21.15.32:0/1492636824 learned_addr learned my addr 172.21.15.32:0/1492636824 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:42.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa205b11700 1 -- 172.21.15.32:0/1492636824 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269615912 0 0) 0x7fa1f8000ce0 con 0x7fa20810d010 2021-04-09T18:18:42.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa205b11700 1 -- 172.21.15.32:0/1492636824 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1ec002fe0 con 0x7fa20810d010 2021-04-09T18:18:42.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa207314700 1 -- 172.21.15.32:0/1492636824 >> v1:172.21.15.86:6789/0 conn(0x7fa2080d7ca0 legacy=0x7fa2080d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa207314700 1 -- 172.21.15.32:0/1492636824 >> v1:172.21.15.32:6789/0 conn(0x7fa20810d010 legacy=0x7fa2080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa207314700 1 -- 172.21.15.32:0/1492636824 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2080d9220 con 0x7fa20810da00 2021-04-09T18:18:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa207314700 1 --2- 172.21.15.32:0/1492636824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20810da00 0x7fa20810de00 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7fa200006e30 tx=0x7fa200009e60).ready entity=mon.0 client_cookie=5b6ce84ea85cf837 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa205b11700 1 -- 172.21.15.32:0/1492636824 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa200006a50 con 0x7fa20810da00 2021-04-09T18:18:42.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa205b11700 1 -- 172.21.15.32:0/1492636824 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20000b300 con 0x7fa20810da00 2021-04-09T18:18:42.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa205b11700 1 -- 172.21.15.32:0/1492636824 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa20000b750 con 0x7fa20810da00 2021-04-09T18:18:42.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa20cd22700 1 -- 172.21.15.32:0/1492636824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20810da00 msgr2=0x7fa20810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.704 7fa20cd22700 1 --2- 172.21.15.32:0/1492636824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20810da00 0x7fa20810de00 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7fa200006e30 tx=0x7fa200009e60).stop 2021-04-09T18:18:42.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.705 7fa20cd22700 1 -- 172.21.15.32:0/1492636824 shutdown_connections 2021-04-09T18:18:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.705 7fa20cd22700 1 --2- 172.21.15.32:0/1492636824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20810da00 0x7fa20810de00 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.705 7fa20cd22700 1 -- 172.21.15.32:0/1492636824 >> 172.21.15.32:0/1492636824 conn(0x7fa2080ce5f0 msgr2=0x7fa2080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.705 7fa20cd22700 1 -- 172.21.15.32:0/1492636824 shutdown_connections 2021-04-09T18:18:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.705 7fa20cd22700 1 -- 172.21.15.32:0/1492636824 wait complete. 2021-04-09T18:18:42.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.725 7fa20cd22700 1 Processor -- start 2021-04-09T18:18:42.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.725 7fa20cd22700 1 -- start start 2021-04-09T18:18:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa20cd22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20817e010 0x7fa20817e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa20cd22700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa20809f290 0x7fa20809f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa20cd22700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2080b1b70 0x7fa2080b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa20cd22700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa208182a50 con 0x7fa20817e010 2021-04-09T18:18:42.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa20cd22700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20809fcb0 con 0x7fa2080b1b70 2021-04-09T18:18:42.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa20cd22700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2080b2590 con 0x7fa20809f290 2021-04-09T18:18:42.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20817e010 0x7fa20817e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20817e010 0x7fa20817e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46882/0 (socket says 172.21.15.32:46882) 2021-04-09T18:18:42.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206b13700 1 -- 172.21.15.32:0/1685901736 learned_addr learned my addr 172.21.15.32:0/1685901736 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:42.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa207314700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2080b1b70 0x7fa2080b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206312700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa20809f290 0x7fa20809f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206b13700 1 -- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa20809f290 msgr2=0x7fa20809f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206b13700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa20809f290 0x7fa20809f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206b13700 1 -- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2080b1b70 msgr2=0x7fa2080b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:42.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206b13700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2080b1b70 0x7fa2080b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:42.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206b13700 1 -- 172.21.15.32:0/1685901736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2080b27b0 con 0x7fa20817e010 2021-04-09T18:18:42.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206312700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa20809f290 0x7fa20809f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:42.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa206b13700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20817e010 0x7fa20817e410 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7fa1f8006f00 tx=0x7fa1f800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:42.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa1ff7fe700 1 -- 172.21.15.32:0/1685901736 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1f8006020 con 0x7fa20817e010 2021-04-09T18:18:42.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa1ff7fe700 1 -- 172.21.15.32:0/1685901736 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f8006360 con 0x7fa20817e010 2021-04-09T18:18:42.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2080b27b0 con 0x7fa20817e010 2021-04-09T18:18:42.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa1ff7fe700 1 -- 172.21.15.32:0/1685901736 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1f8004620 con 0x7fa20817e010 2021-04-09T18:18:42.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.726 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2080b27b0 con 0x7fa20817e010 2021-04-09T18:18:42.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.727 7fa1ff7fe700 1 -- 172.21.15.32:0/1685901736 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa1f800a000 con 0x7fa20817e010 2021-04-09T18:18:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.727 7fa1ff7fe700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1e4018930 0x7fa1e401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.727 7fa1ff7fe700 1 -- 172.21.15.32:0/1685901736 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa1f80329e0 con 0x7fa20817e010 2021-04-09T18:18:42.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.727 7fa206312700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1e4018930 0x7fa1e401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:42.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.728 7fa206312700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1e4018930 0x7fa1e401adc0 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:42.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.748 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20809ed60 con 0x7fa20817e010 2021-04-09T18:18:42.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.751 7fa1ff7fe700 1 -- 172.21.15.32:0/1685901736 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa1f80009b0 con 0x7fa20817e010 2021-04-09T18:18:42.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.942 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa20809ed60 con 0x7fa20817e010 2021-04-09T18:18:42.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.943 7fa1ff7fe700 1 -- 172.21.15.32:0/1685901736 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2779 (secure 0 0 0) 0x7fa1f800a000 con 0x7fa20817e010 2021-04-09T18:18:42.986 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 91 2021-04-09T18:18:42.987 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:42.987 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6654,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:35.676838","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1e4018930 msgr2=0x7fa1e401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1e4018930 0x7fa1e401adc0 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20817e010 msgr2=0x7fa20817e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:43.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20817e010 0x7fa20817e410 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7fa1f8006f00 tx=0x7fa1f800b940).stop 2021-04-09T18:18:43.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 shutdown_connections 2021-04-09T18:18:43.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1e4018930 0x7fa1e401adc0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:43.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2080b1b70 0x7fa2080b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa20809f290 0x7fa20809f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 --2- 172.21.15.32:0/1685901736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa20817e010 0x7fa20817e410 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 >> 172.21.15.32:0/1685901736 conn(0x7fa2081965a0 msgr2=0x7fa20819dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 shutdown_connections 2021-04-09T18:18:43.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:42.945 7fa20cd22700 1 -- 172.21.15.32:0/1685901736 wait complete. 2021-04-09T18:18:43.011 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:18:43.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.141 7f2535856700 1 Processor -- start 2021-04-09T18:18:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.141 7f2535856700 1 -- start start 2021-04-09T18:18:43.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.141 7f2535856700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f253010d010 0x7f25300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.141 7f2535856700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253010d480 con 0x7f25300d1da0 2021-04-09T18:18:43.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.141 7f2535856700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253010e1f0 con 0x7f25300d8490 2021-04-09T18:18:43.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.141 7f2535856700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253010e780 con 0x7f253010d010 2021-04-09T18:18:43.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.141 7f252ed73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f25300d1da0 0x7f25300d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56484/0 (socket says 172.21.15.32:56484) 2021-04-09T18:18:43.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.141 7f252ed73700 1 -- 172.21.15.32:0/2283477415 learned_addr learned my addr 172.21.15.32:0/2283477415 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252f574700 1 --2- 172.21.15.32:0/2283477415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f253010d010 0x7f25300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252e572700 1 -- 172.21.15.32:0/2283477415 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684404325 0 0) 0x7f25240009b0 con 0x7f25300d1da0 2021-04-09T18:18:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252e572700 1 -- 172.21.15.32:0/2283477415 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2518002fe0 con 0x7f25300d1da0 2021-04-09T18:18:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252e572700 1 -- 172.21.15.32:0/2283477415 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721743931 0 0) 0x7f2528000d30 con 0x7f25300d8490 2021-04-09T18:18:43.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252e572700 1 -- 172.21.15.32:0/2283477415 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2518003210 con 0x7f25300d8490 2021-04-09T18:18:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252e572700 1 -- 172.21.15.32:0/2283477415 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1272363451 0 0) 0x7f2524001010 con 0x7f25300d1da0 2021-04-09T18:18:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252f574700 1 -- 172.21.15.32:0/2283477415 >> v1:172.21.15.86:6789/0 conn(0x7f25300d8490 legacy=0x7f253010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252f574700 1 -- 172.21.15.32:0/2283477415 >> v1:172.21.15.32:6789/0 conn(0x7f25300d1da0 legacy=0x7f25300d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252f574700 1 -- 172.21.15.32:0/2283477415 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25300d8d30 con 0x7f253010d010 2021-04-09T18:18:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252f574700 1 --2- 172.21.15.32:0/2283477415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f253010d010 0x7f25300d1780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f2520006650 tx=0x7f2520009a00).ready entity=mon.2 client_cookie=da0841393fc77c4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252e572700 1 -- 172.21.15.32:0/2283477415 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2520009df0 con 0x7f253010d010 2021-04-09T18:18:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252e572700 1 -- 172.21.15.32:0/2283477415 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25200009b0 con 0x7f253010d010 2021-04-09T18:18:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f252e572700 1 -- 172.21.15.32:0/2283477415 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f252000c2b0 con 0x7f253010d010 2021-04-09T18:18:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f2535856700 1 -- 172.21.15.32:0/2283477415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f253010d010 msgr2=0x7f25300d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.142 7f2535856700 1 --2- 172.21.15.32:0/2283477415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f253010d010 0x7f25300d1780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f2520006650 tx=0x7f2520009a00).stop 2021-04-09T18:18:43.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.143 7f2535856700 1 -- 172.21.15.32:0/2283477415 shutdown_connections 2021-04-09T18:18:43.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.143 7f2535856700 1 --2- 172.21.15.32:0/2283477415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f253010d010 0x7f25300d1780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:43.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.143 7f2535856700 1 -- 172.21.15.32:0/2283477415 >> 172.21.15.32:0/2283477415 conn(0x7f25300ce5f0 msgr2=0x7f25300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:43.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.143 7f2535856700 1 -- 172.21.15.32:0/2283477415 shutdown_connections 2021-04-09T18:18:43.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.143 7f2535856700 1 -- 172.21.15.32:0/2283477415 wait complete. 2021-04-09T18:18:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.163 7f2535856700 1 Processor -- start 2021-04-09T18:18:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.163 7f2535856700 1 -- start start 2021-04-09T18:18:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f2535856700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25300a0110 0x7f253018cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f2535856700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253018d320 0x7f25300a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f2535856700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25300a1230 0x7f25301a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:43.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f2535856700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25300a0690 con 0x7f25300a0110 2021-04-09T18:18:43.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f2535856700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25300a1630 con 0x7f253018d320 2021-04-09T18:18:43.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f2535856700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25301a1370 con 0x7f25300a1230 2021-04-09T18:18:43.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252f574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25300a0110 0x7f253018cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:43.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252f574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25300a0110 0x7f253018cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46896/0 (socket says 172.21.15.32:46896) 2021-04-09T18:18:43.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252f574700 1 -- 172.21.15.32:0/2808225917 learned_addr learned my addr 172.21.15.32:0/2808225917 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:43.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252fd75700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25300a1230 0x7f25301a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:43.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252ed73700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253018d320 0x7f25300a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:43.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252f574700 1 -- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25300a1230 msgr2=0x7f25301a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:43.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252f574700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25300a1230 0x7f25301a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:43.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252f574700 1 -- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253018d320 msgr2=0x7f25300a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252f574700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253018d320 0x7f25300a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.164 7f252f574700 1 -- 172.21.15.32:0/2808225917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25301a1590 con 0x7f25300a0110 2021-04-09T18:18:43.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.165 7f252f574700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25300a0110 0x7f253018cd00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f25200096a0 tx=0x7f252000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:43.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.165 7f2517fff700 1 -- 172.21.15.32:0/2808225917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2520000c90 con 0x7f25300a0110 2021-04-09T18:18:43.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.165 7f2517fff700 1 -- 172.21.15.32:0/2808225917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2520002030 con 0x7f25300a0110 2021-04-09T18:18:43.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.165 7f2535856700 1 -- 172.21.15.32:0/2808225917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25301a1590 con 0x7f25300a0110 2021-04-09T18:18:43.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.165 7f2517fff700 1 -- 172.21.15.32:0/2808225917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2520002550 con 0x7f25300a0110 2021-04-09T18:18:43.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.165 7f2535856700 1 -- 172.21.15.32:0/2808225917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25301a1590 con 0x7f25300a0110 2021-04-09T18:18:43.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.165 7f2517fff700 1 -- 172.21.15.32:0/2808225917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2520002210 con 0x7f25300a0110 2021-04-09T18:18:43.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.166 7f2517fff700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25100188c0 0x7f251001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:43.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.166 7f2517fff700 1 -- 172.21.15.32:0/2808225917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2520032a40 con 0x7f25300a0110 2021-04-09T18:18:43.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.166 7f252ed73700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25100188c0 0x7f251001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:43.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.166 7f252ed73700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25100188c0 0x7f251001adc0 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:43.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.185 7f2535856700 1 -- 172.21.15.32:0/2808225917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253010d480 con 0x7f25300a0110 2021-04-09T18:18:43.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.189 7f2517fff700 1 -- 172.21.15.32:0/2808225917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f25200009c0 con 0x7f25300a0110 2021-04-09T18:18:43.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:43.381 7f2535856700 1 -- 172.21.15.32:0/2808225917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f253010d480 con 0x7f25300a0110 2021-04-09T18:18:44.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.384 7f2517fff700 1 -- 172.21.15.32:0/2808225917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v92) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2520002030 con 0x7f25300a0110 2021-04-09T18:18:44.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 -- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25100188c0 msgr2=0x7f251001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:44.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25100188c0 0x7f251001adc0 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 -- 172.21.15.32:0/2808225917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25300a0110 msgr2=0x7f253018cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25300a0110 0x7f253018cd00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f25200096a0 tx=0x7f252000b990).stop 2021-04-09T18:18:44.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 -- 172.21.15.32:0/2808225917 shutdown_connections 2021-04-09T18:18:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25100188c0 0x7f251001adc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25300a1230 0x7f25301a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253018d320 0x7f25300a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 --2- 172.21.15.32:0/2808225917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25300a0110 0x7f253018cd00 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 -- 172.21.15.32:0/2808225917 >> 172.21.15.32:0/2808225917 conn(0x7f25300b93a0 msgr2=0x7f25300b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:44.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 -- 172.21.15.32:0/2808225917 shutdown_connections 2021-04-09T18:18:44.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.400 7f2535856700 1 -- 172.21.15.32:0/2808225917 wait complete. 2021-04-09T18:18:44.455 INFO:tasks.cephfs.test_failover:Waiting for cluster to grow. 2021-04-09T18:18:44.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.572 7f9a239ed700 1 Processor -- start 2021-04-09T18:18:44.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.572 7f9a239ed700 1 -- start start 2021-04-09T18:18:44.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.572 7f9a239ed700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a1c10d010 0x7f9a1c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:44.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.572 7f9a239ed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c0d1da0 0x7f9a1c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:44.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a239ed700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1c10d480 con 0x7f9a1c0d82c0 2021-04-09T18:18:44.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a239ed700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a1c0d9220 con 0x7f9a1c10d010 2021-04-09T18:18:44.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a239ed700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0d86c0 con 0x7f9a1c0d1da0 2021-04-09T18:18:44.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a21809700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a1c10d010 0x7f9a1c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:44.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a21008700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c0d1da0 0x7f9a1c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:44.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a21809700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a1c10d010 0x7f9a1c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:38670/0 (socket says 172.21.15.32:38670) 2021-04-09T18:18:44.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a2200a700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9a1c0d82c0 0x7f9a1c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54430/0 (socket says 172.21.15.32:54430) 2021-04-09T18:18:44.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a21809700 1 -- 172.21.15.32:0/3662058514 learned_addr learned my addr 172.21.15.32:0/3662058514 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:44.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a21809700 1 -- 172.21.15.32:0/3662058514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c0d1da0 msgr2=0x7f9a1c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:44.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a21809700 1 --2- 172.21.15.32:0/3662058514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c0d1da0 0x7f9a1c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a21809700 1 -- 172.21.15.32:0/3662058514 >> v1:172.21.15.86:6789/0 conn(0x7f9a1c0d82c0 legacy=0x7f9a1c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:44.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a21809700 1 -- 172.21.15.32:0/3662058514 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c10e1f0 con 0x7f9a1c10d010 2021-04-09T18:18:44.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a21008700 1 --2- 172.21.15.32:0/3662058514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c0d1da0 0x7f9a1c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.573 7f9a20807700 1 -- 172.21.15.32:0/3662058514 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365016875 0 0) 0x7f9a0c0009b0 con 0x7f9a1c0d82c0 2021-04-09T18:18:44.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.574 7f9a21809700 1 --2- 172.21.15.32:0/3662058514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a1c10d010 0x7f9a1c0d1780 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f9a10006a80 tx=0x7f9a10009f00).ready entity=mon.1 client_cookie=43c6f42de67729c5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:44.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.574 7f9a20807700 1 -- 172.21.15.32:0/3662058514 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a1000b5c0 con 0x7f9a1c10d010 2021-04-09T18:18:44.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.574 7f9a20807700 1 -- 172.21.15.32:0/3662058514 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1000b8c0 con 0x7f9a1c10d010 2021-04-09T18:18:44.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.574 7f9a20807700 1 -- 172.21.15.32:0/3662058514 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a10000d30 con 0x7f9a1c10d010 2021-04-09T18:18:44.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.574 7f9a239ed700 1 -- 172.21.15.32:0/3662058514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a1c10d010 msgr2=0x7f9a1c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:44.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.574 7f9a239ed700 1 --2- 172.21.15.32:0/3662058514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a1c10d010 0x7f9a1c0d1780 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f9a10006a80 tx=0x7f9a10009f00).stop 2021-04-09T18:18:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.574 7f9a239ed700 1 -- 172.21.15.32:0/3662058514 shutdown_connections 2021-04-09T18:18:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.574 7f9a239ed700 1 --2- 172.21.15.32:0/3662058514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c0d1da0 0x7f9a1c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.574 7f9a239ed700 1 --2- 172.21.15.32:0/3662058514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a1c10d010 0x7f9a1c0d1780 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.575 7f9a239ed700 1 -- 172.21.15.32:0/3662058514 >> 172.21.15.32:0/3662058514 conn(0x7f9a1c0ce5f0 msgr2=0x7f9a1c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:44.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.575 7f9a239ed700 1 -- 172.21.15.32:0/3662058514 shutdown_connections 2021-04-09T18:18:44.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.575 7f9a239ed700 1 -- 172.21.15.32:0/3662058514 wait complete. 2021-04-09T18:18:44.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a239ed700 1 Processor -- start 2021-04-09T18:18:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a239ed700 1 -- start start 2021-04-09T18:18:44.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a239ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a1c1aca20 0x7f9a1c1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a239ed700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a1c18bd40 0x7f9a1c18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a239ed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c08fba0 0x7f9a1c08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a239ed700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c18c7e0 con 0x7f9a1c1aca20 2021-04-09T18:18:44.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a239ed700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c18ca30 con 0x7f9a1c18bd40 2021-04-09T18:18:44.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a239ed700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0905c0 con 0x7f9a1c08fba0 2021-04-09T18:18:44.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a21809700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a1c1aca20 0x7f9a1c1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:44.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a21809700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a1c1aca20 0x7f9a1c1ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46918/0 (socket says 172.21.15.32:46918) 2021-04-09T18:18:44.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.595 7f9a21809700 1 -- 172.21.15.32:0/3843051026 learned_addr learned my addr 172.21.15.32:0/3843051026 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:44.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a21008700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a1c18bd40 0x7f9a1c18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:44.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a2200a700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c08fba0 0x7f9a1c08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:44.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a21809700 1 -- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c08fba0 msgr2=0x7f9a1c08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:44.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a21809700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c08fba0 0x7f9a1c08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a21809700 1 -- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a1c18bd40 msgr2=0x7f9a1c18c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:44.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a21809700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a1c18bd40 0x7f9a1c18c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a21809700 1 -- 172.21.15.32:0/3843051026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c0b4300 con 0x7f9a1c1aca20 2021-04-09T18:18:44.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a21809700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a1c1aca20 0x7f9a1c1ace20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f9a10009450 tx=0x7f9a1000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:44.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a1a7fc700 1 -- 172.21.15.32:0/3843051026 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a10006e40 con 0x7f9a1c1aca20 2021-04-09T18:18:44.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a1a7fc700 1 -- 172.21.15.32:0/3843051026 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1000be60 con 0x7f9a1c1aca20 2021-04-09T18:18:44.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a1a7fc700 1 -- 172.21.15.32:0/3843051026 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a1000c4e0 con 0x7f9a1c1aca20 2021-04-09T18:18:44.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1c0b4300 con 0x7f9a1c1aca20 2021-04-09T18:18:44.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.596 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1c0b4300 con 0x7f9a1c1aca20 2021-04-09T18:18:44.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.597 7f9a1a7fc700 1 -- 172.21.15.32:0/3843051026 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9a10008020 con 0x7f9a1c1aca20 2021-04-09T18:18:44.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.597 7f9a1a7fc700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99fc018930 0x7f99fc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:44.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.597 7f9a1a7fc700 1 -- 172.21.15.32:0/3843051026 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9a10032db0 con 0x7f9a1c1aca20 2021-04-09T18:18:44.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.597 7f9a21008700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99fc018930 0x7f99fc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:44.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.597 7f9a21008700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99fc018930 0x7f99fc01ae10 crc :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:44.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.617 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1c182d70 con 0x7f9a1c1aca20 2021-04-09T18:18:44.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.620 7f9a1a7fc700 1 -- 172.21.15.32:0/3843051026 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9a1001faf0 con 0x7f9a1c1aca20 2021-04-09T18:18:44.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.809 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a1c18c760 con 0x7f9a1c1aca20 2021-04-09T18:18:44.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.809 7f9a1a7fc700 1 -- 172.21.15.32:0/3843051026 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2795 (secure 0 0 0) 0x7f9a10007fe0 con 0x7f9a1c1aca20 2021-04-09T18:18:44.848 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 93 2021-04-09T18:18:44.849 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:44.849 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":93,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:44.386032","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6526,"mds_1":6654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6654":{"gid":6654,"name":"c","rank":1,"incarnation":93,"state":"up:creating","state_seq":2,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:44.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99fc018930 msgr2=0x7f99fc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:44.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99fc018930 0x7f99fc01ae10 crc :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a1c1aca20 msgr2=0x7f9a1c1ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:44.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a1c1aca20 0x7f9a1c1ace20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f9a10009450 tx=0x7f9a1000bd20).stop 2021-04-09T18:18:44.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 shutdown_connections 2021-04-09T18:18:44.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99fc018930 0x7f99fc01ae10 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a1c08fba0 0x7f9a1c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a1c18bd40 0x7f9a1c18c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 --2- 172.21.15.32:0/3843051026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a1c1aca20 0x7f9a1c1ace20 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:44.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 >> 172.21.15.32:0/3843051026 conn(0x7f9a1c19b460 msgr2=0x7f9a1c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:44.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 shutdown_connections 2021-04-09T18:18:44.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:44.812 7f9a239ed700 1 -- 172.21.15.32:0/3843051026 wait complete. 2021-04-09T18:18:45.275 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 93, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:44.386032', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6526, 'mds_1': 6654}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6654': {'gid': 6654, 'name': 'c', 'rank': 1, 'incarnation': 93, 'state': 'up:creating', 'state_seq': 2, 'addr': '172.21.15.32:6834/1291090515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1291090515}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 1291090515}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:45.276 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6654:up:creating 2021-04-09T18:18:46.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.392 7f23b5e2c700 1 Processor -- start 2021-04-09T18:18:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.392 7f23b5e2c700 1 -- start start 2021-04-09T18:18:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23b5e2c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f23b00cffa0 0x7f23b00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23b5e2c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b00d6c80 0x7f23b010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23b5e2c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b00d7280 con 0x7f23b00d7800 2021-04-09T18:18:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23b5e2c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b00d7080 con 0x7f23b00cffa0 2021-04-09T18:18:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23b5e2c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00d91e0 con 0x7f23b00d6c80 2021-04-09T18:18:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23affff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b00d6c80 0x7f23b010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23af7fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f23b00cffa0 0x7f23b00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:46.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23affff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b00d6c80 0x7f23b010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:54158/0 (socket says 172.21.15.32:54158) 2021-04-09T18:18:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23af7fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f23b00cffa0 0x7f23b00d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:38688/0 (socket says 172.21.15.32:38688) 2021-04-09T18:18:46.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23affff700 1 -- 172.21.15.32:0/2192254227 learned_addr learned my addr 172.21.15.32:0/2192254227 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:46.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.393 7f23aeffd700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f23b00d7800 0x7f23b00d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54448/0 (socket says 172.21.15.32:54448) 2021-04-09T18:18:46.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23affff700 1 -- 172.21.15.32:0/2192254227 >> v2:172.21.15.86:3300/0 conn(0x7f23b00cffa0 msgr2=0x7f23b00d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:46.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23affff700 1 --2- 172.21.15.32:0/2192254227 >> v2:172.21.15.86:3300/0 conn(0x7f23b00cffa0 0x7f23b00d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23affff700 1 -- 172.21.15.32:0/2192254227 >> v1:172.21.15.86:6789/0 conn(0x7f23b00d7800 legacy=0x7f23b00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:46.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23affff700 1 -- 172.21.15.32:0/2192254227 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00d83f0 con 0x7f23b00d6c80 2021-04-09T18:18:46.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23affff700 1 --2- 172.21.15.32:0/2192254227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b00d6c80 0x7f23b010cfd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f23a8006aa0 tx=0x7f23a8009f20).ready entity=mon.2 client_cookie=2c583e07f940b719 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:46.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23ae7fc700 1 -- 172.21.15.32:0/2192254227 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23a800b5c0 con 0x7f23b00d6c80 2021-04-09T18:18:46.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23ae7fc700 1 -- 172.21.15.32:0/2192254227 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a8003030 con 0x7f23b00d6c80 2021-04-09T18:18:46.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23ae7fc700 1 -- 172.21.15.32:0/2192254227 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23a800bb40 con 0x7f23b00d6c80 2021-04-09T18:18:46.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23b5e2c700 1 -- 172.21.15.32:0/2192254227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b00d6c80 msgr2=0x7f23b010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:46.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.394 7f23b5e2c700 1 --2- 172.21.15.32:0/2192254227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b00d6c80 0x7f23b010cfd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f23a8006aa0 tx=0x7f23a8009f20).stop 2021-04-09T18:18:46.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.395 7f23b5e2c700 1 -- 172.21.15.32:0/2192254227 shutdown_connections 2021-04-09T18:18:46.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.395 7f23b5e2c700 1 --2- 172.21.15.32:0/2192254227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b00d6c80 0x7f23b010cfd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.395 7f23b5e2c700 1 --2- 172.21.15.32:0/2192254227 >> v2:172.21.15.86:3300/0 conn(0x7f23b00cffa0 0x7f23b00d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.395 7f23b5e2c700 1 -- 172.21.15.32:0/2192254227 >> 172.21.15.32:0/2192254227 conn(0x7f23b00ce5c0 msgr2=0x7f23b00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:46.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.395 7f23b5e2c700 1 -- 172.21.15.32:0/2192254227 shutdown_connections 2021-04-09T18:18:46.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.395 7f23b5e2c700 1 -- 172.21.15.32:0/2192254227 wait complete. 2021-04-09T18:18:46.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.415 7f23b5e2c700 1 Processor -- start 2021-04-09T18:18:46.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.415 7f23b5e2c700 1 -- start start 2021-04-09T18:18:46.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23b5e2c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23b00b72f0 0x7f23b00b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:46.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23b5e2c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b019c860 0x7f23b019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:46.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23b5e2c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b019fdd0 0x7f23b019da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23b5e2c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b019f230 con 0x7f23b00b72f0 2021-04-09T18:18:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23b5e2c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b008a6a0 con 0x7f23b019c860 2021-04-09T18:18:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23b5e2c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b018dce0 con 0x7f23b019fdd0 2021-04-09T18:18:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23b00b72f0 0x7f23b00b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:46.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23af7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23b00b72f0 0x7f23b00b76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46940/0 (socket says 172.21.15.32:46940) 2021-04-09T18:18:46.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23af7fe700 1 -- 172.21.15.32:0/2703558529 learned_addr learned my addr 172.21.15.32:0/2703558529 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:46.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23affff700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b019fdd0 0x7f23b019da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:46.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23aeffd700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b019c860 0x7f23b019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:46.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23af7fe700 1 -- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b019fdd0 msgr2=0x7f23b019da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23af7fe700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b019fdd0 0x7f23b019da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23af7fe700 1 -- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b019c860 msgr2=0x7f23b019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23af7fe700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b019c860 0x7f23b019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23af7fe700 1 -- 172.21.15.32:0/2703558529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b018e150 con 0x7f23b00b72f0 2021-04-09T18:18:46.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.416 7f23af7fe700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23b00b72f0 0x7f23b00b76f0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f23a0006f60 tx=0x7f23a000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:46.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.417 7f2397fff700 1 -- 172.21.15.32:0/2703558529 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23a000bd60 con 0x7f23b00b72f0 2021-04-09T18:18:46.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.417 7f2397fff700 1 -- 172.21.15.32:0/2703558529 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a0000c90 con 0x7f23b00b72f0 2021-04-09T18:18:46.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.417 7f2397fff700 1 -- 172.21.15.32:0/2703558529 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23a0002220 con 0x7f23b00b72f0 2021-04-09T18:18:46.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.417 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b018e150 con 0x7f23b00b72f0 2021-04-09T18:18:46.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.417 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b018e150 con 0x7f23b00b72f0 2021-04-09T18:18:46.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.417 7f2397fff700 1 -- 172.21.15.32:0/2703558529 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f23a00052e0 con 0x7f23b00b72f0 2021-04-09T18:18:46.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.417 7f2397fff700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2390018930 0x7f239001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:46.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.417 7f2397fff700 1 -- 172.21.15.32:0/2703558529 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f23a00329f0 con 0x7f23b00b72f0 2021-04-09T18:18:46.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.418 7f23aeffd700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2390018930 0x7f239001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:46.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.418 7f23aeffd700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2390018930 0x7f239001ae10 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:46.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.437 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b0182e10 con 0x7f23b00b72f0 2021-04-09T18:18:46.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.440 7f2397fff700 1 -- 172.21.15.32:0/2703558529 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f23a0000c90 con 0x7f23b00b72f0 2021-04-09T18:18:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.630 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23b0182e10 con 0x7f23b00b72f0 2021-04-09T18:18:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.631 7f2397fff700 1 -- 172.21.15.32:0/2703558529 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+2793 (secure 0 0 0) 0x7f23a0006b00 con 0x7f23b00b72f0 2021-04-09T18:18:46.670 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 94 2021-04-09T18:18:46.670 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:46.670 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":94,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:45.387034","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6526,"mds_1":6654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6654":{"gid":6654,"name":"c","rank":1,"incarnation":93,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:46.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2390018930 msgr2=0x7f239001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:46.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2390018930 0x7f239001ae10 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23b00b72f0 msgr2=0x7f23b00b76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:46.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23b00b72f0 0x7f23b00b76f0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f23a0006f60 tx=0x7f23a000b970).stop 2021-04-09T18:18:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 shutdown_connections 2021-04-09T18:18:46.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2390018930 0x7f239001ae10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23b019fdd0 0x7f23b019da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b019c860 0x7f23b019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 --2- 172.21.15.32:0/2703558529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23b00b72f0 0x7f23b00b76f0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:46.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.633 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 >> 172.21.15.32:0/2703558529 conn(0x7f23b0097550 msgr2=0x7f23b018b920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:46.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.634 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 shutdown_connections 2021-04-09T18:18:46.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:46.634 7f23b5e2c700 1 -- 172.21.15.32:0/2703558529 wait complete. 2021-04-09T18:18:47.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:45.387034', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6526, 'mds_1': 6654}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6654': {'gid': 6654, 'name': 'c', 'rank': 1, 'incarnation': 93, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6834/1291090515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1291090515}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 1291090515}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:47.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:18:47.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:18:47.236 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:47.237 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:47.237 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:18:47.238 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6526, 2021-04-09T18:18:47.238 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:18:47.238 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:18:47.239 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 13.533233128999999, 2021-04-09T18:18:47.240 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 94, 2021-04-09T18:18:47.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 84, 2021-04-09T18:18:47.240 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 84, 2021-04-09T18:18:47.240 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 17.990238098999999 2021-04-09T18:18:47.242 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:47.243 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 6526, "want_state": "up:active", "state": "up:active", "rank_uptime": 13.533233128999999, "mdsmap_epoch": 94, "osdmap_epoch": 84, "osdmap_epoch_barrier": 84, "uptime": 17.990238098999999 } 2021-04-09T18:18:47.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:18:47.392 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:18:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:18:47.393 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:18:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 6654, 2021-04-09T18:18:47.394 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:18:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:18:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 2.967826842, 2021-04-09T18:18:47.395 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 94, 2021-04-09T18:18:47.396 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 84, 2021-04-09T18:18:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 84, 2021-04-09T18:18:47.398 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 18.146551032000001 2021-04-09T18:18:47.399 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:18:47.401 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 6654, "want_state": "up:active", "state": "up:active", "rank_uptime": 2.967826842, "mdsmap_epoch": 94, "osdmap_epoch": 84, "osdmap_epoch_barrier": 84, "uptime": 18.146551032000001 } 2021-04-09T18:18:47.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:18:47.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.519 7f1f35794700 1 Processor -- start 2021-04-09T18:18:47.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.519 7f1f35794700 1 -- start start 2021-04-09T18:18:47.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f35794700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f300d7ca0 0x7f1f300d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:47.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f35794700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3010d480 con 0x7f1f3010da00 2021-04-09T18:18:47.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f35794700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f300d86c0 con 0x7f1f3010d010 2021-04-09T18:18:47.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f35794700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300d9220 con 0x7f1f300d7ca0 2021-04-09T18:18:47.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2fd75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1f3010da00 0x7f1f3010de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56534/0 (socket says 172.21.15.32:56534) 2021-04-09T18:18:47.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2fd75700 1 -- 172.21.15.32:0/1100523986 learned_addr learned my addr 172.21.15.32:0/1100523986 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:47.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2ed73700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f300d7ca0 0x7f1f300d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:47.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181023748 0 0) 0x7f1f28000d30 con 0x7f1f3010da00 2021-04-09T18:18:47.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f18002fe0 con 0x7f1f3010da00 2021-04-09T18:18:47.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2916345708 0 0) 0x7f1f28001270 con 0x7f1f3010da00 2021-04-09T18:18:47.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f300d7ca0 msgr2=0x7f1f300d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:47.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f300d7ca0 0x7f1f300d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:47.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 >> v1:172.21.15.86:6789/0 conn(0x7f1f3010d010 legacy=0x7f1f300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:47.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300d1f70 con 0x7f1f3010da00 2021-04-09T18:18:47.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f1f28001980 con 0x7f1f3010da00 2021-04-09T18:18:47.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 >> v1:172.21.15.32:6789/0 conn(0x7f1f3010da00 legacy=0x7f1f3010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:47.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f18003a70 0x7f1f18005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:47.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180067a0 0x7f1f18008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:47.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f180095a0 0x7f1f1800ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:47.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1800c2e0 con 0x7f1f180067a0 2021-04-09T18:18:47.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.520 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1800c500 con 0x7f1f18003a70 2021-04-09T18:18:47.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18050730 con 0x7f1f180095a0 2021-04-09T18:18:47.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2ed73700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180067a0 0x7f1f18008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:47.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2f574700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f18003a70 0x7f1f18005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:47.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2fd75700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f180095a0 0x7f1f1800ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2ed73700 1 -- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f180095a0 msgr2=0x7f1f1800ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2ed73700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f180095a0 0x7f1f1800ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:47.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2ed73700 1 -- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f18003a70 msgr2=0x7f1f18005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:47.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2ed73700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f18003a70 0x7f1f18005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:47.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2ed73700 1 -- 172.21.15.32:0/1100523986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f18050960 con 0x7f1f180067a0 2021-04-09T18:18:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2ed73700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180067a0 0x7f1f18008ca0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f1f24006ac0 tx=0x7f1f24009ab0).ready entity=mon.0 client_cookie=f84a29ffb8f1f4d2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:47.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f2400c0c0 con 0x7f1f180067a0 2021-04-09T18:18:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2400c3c0 con 0x7f1f180067a0 2021-04-09T18:18:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2e572700 1 -- 172.21.15.32:0/1100523986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f2400c820 con 0x7f1f180067a0 2021-04-09T18:18:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f35794700 1 -- 172.21.15.32:0/1100523986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180067a0 msgr2=0x7f1f18008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f35794700 1 --2- 172.21.15.32:0/1100523986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f180067a0 0x7f1f18008ca0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f1f24006ac0 tx=0x7f1f24009ab0).stop 2021-04-09T18:18:47.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.521 7f1f2fd75700 1 -- 172.21.15.32:0/1100523986 reap_dead start 2021-04-09T18:18:47.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.522 7f1f35794700 1 -- 172.21.15.32:0/1100523986 shutdown_connections 2021-04-09T18:18:47.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.522 7f1f35794700 1 -- 172.21.15.32:0/1100523986 >> 172.21.15.32:0/1100523986 conn(0x7f1f300ce5f0 msgr2=0x7f1f300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:47.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.522 7f1f35794700 1 -- 172.21.15.32:0/1100523986 shutdown_connections 2021-04-09T18:18:47.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.522 7f1f35794700 1 -- 172.21.15.32:0/1100523986 wait complete. 2021-04-09T18:18:47.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.542 7f1f35794700 1 Processor -- start 2021-04-09T18:18:47.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.542 7f1f35794700 1 -- start start 2021-04-09T18:18:47.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f35794700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f300b1680 0x7f1f300b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:47.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f35794700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3009e330 0x7f1f300b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:47.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f35794700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300b2190 0x7f1f3009f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:47.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f35794700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3009e730 con 0x7f1f300b2190 2021-04-09T18:18:47.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f35794700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3009f6c0 con 0x7f1f300b1680 2021-04-09T18:18:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f35794700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3009f8e0 con 0x7f1f3009e330 2021-04-09T18:18:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300b2190 0x7f1f3009f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:47.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300b2190 0x7f1f3009f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46958/0 (socket says 172.21.15.32:46958) 2021-04-09T18:18:47.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2fd75700 1 -- 172.21.15.32:0/3154307776 learned_addr learned my addr 172.21.15.32:0/3154307776 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:47.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2f574700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f300b1680 0x7f1f300b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:47.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2ed73700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3009e330 0x7f1f300b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:47.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2fd75700 1 -- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3009e330 msgr2=0x7f1f300b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:47.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2fd75700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3009e330 0x7f1f300b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2fd75700 1 -- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f300b1680 msgr2=0x7f1f300b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:47.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2fd75700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f300b1680 0x7f1f300b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:47.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2fd75700 1 -- 172.21.15.32:0/3154307776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3009fb00 con 0x7f1f300b2190 2021-04-09T18:18:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f2fd75700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300b2190 0x7f1f3009f0a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f1f280015a0 tx=0x7f1f2800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f17fff700 1 -- 172.21.15.32:0/3154307776 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f28007020 con 0x7f1f300b2190 2021-04-09T18:18:47.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.543 7f1f17fff700 1 -- 172.21.15.32:0/3154307776 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f28007360 con 0x7f1f300b2190 2021-04-09T18:18:47.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.544 7f1f35794700 1 -- 172.21.15.32:0/3154307776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3009fb00 con 0x7f1f300b2190 2021-04-09T18:18:47.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.544 7f1f17fff700 1 -- 172.21.15.32:0/3154307776 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f28004910 con 0x7f1f300b2190 2021-04-09T18:18:47.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.544 7f1f35794700 1 -- 172.21.15.32:0/3154307776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3009fb00 con 0x7f1f300b2190 2021-04-09T18:18:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.544 7f1f17fff700 1 -- 172.21.15.32:0/3154307776 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1f28001bf0 con 0x7f1f300b2190 2021-04-09T18:18:47.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.544 7f1f17fff700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f10018930 0x7f1f1001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.544 7f1f17fff700 1 -- 172.21.15.32:0/3154307776 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1f28032d50 con 0x7f1f300b2190 2021-04-09T18:18:47.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.544 7f1f2f574700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f10018930 0x7f1f1001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:47.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.545 7f1f2f574700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f10018930 0x7f1f1001ae10 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:47.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.564 7f1f35794700 1 -- 172.21.15.32:0/3154307776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3010d480 con 0x7f1f300b2190 2021-04-09T18:18:47.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.567 7f1f17fff700 1 -- 172.21.15.32:0/3154307776 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1f28000ec0 con 0x7f1f300b2190 2021-04-09T18:18:47.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:47.761 7f1f35794700 1 -- 172.21.15.32:0/3154307776 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f1f3010d480 con 0x7f1f300b2190 2021-04-09T18:18:48.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.398 7f1f17fff700 1 -- 172.21.15.32:0/3154307776 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v95) v1 ==== 105+0+0 (secure 0 0 0) 0x7f1f28007360 con 0x7f1f300b2190 2021-04-09T18:18:48.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 -- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f10018930 msgr2=0x7f1f1001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:48.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f10018930 0x7f1f1001ae10 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 -- 172.21.15.32:0/3154307776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300b2190 msgr2=0x7f1f3009f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300b2190 0x7f1f3009f0a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f1f280015a0 tx=0x7f1f2800ac90).stop 2021-04-09T18:18:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 -- 172.21.15.32:0/3154307776 shutdown_connections 2021-04-09T18:18:48.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f10018930 0x7f1f1001ae10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f300b2190 0x7f1f3009f0a0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3009e330 0x7f1f300b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 --2- 172.21.15.32:0/3154307776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f300b1680 0x7f1f300b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 -- 172.21.15.32:0/3154307776 >> 172.21.15.32:0/3154307776 conn(0x7f1f300c4290 msgr2=0x7f1f300c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:48.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 -- 172.21.15.32:0/3154307776 shutdown_connections 2021-04-09T18:18:48.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.429 7f1f35794700 1 -- 172.21.15.32:0/3154307776 wait complete. 2021-04-09T18:18:48.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:48.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.603 7fabd632e700 1 Processor -- start 2021-04-09T18:18:48.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.603 7fabd632e700 1 -- start start 2021-04-09T18:18:48.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd632e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd010d010 0x7fabd00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:48.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd632e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabd00d1da0 0x7fabd00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:48.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd632e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fabd00d82c0 0x7fabd010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:48.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd632e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd010e780 con 0x7fabd00d82c0 2021-04-09T18:18:48.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd632e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd00d8d30 con 0x7fabd00d1da0 2021-04-09T18:18:48.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd632e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00d9220 con 0x7fabd010d010 2021-04-09T18:18:48.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd494b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fabd00d82c0 0x7fabd010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:48.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd494b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fabd00d82c0 0x7fabd010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46966/0 (socket says 172.21.15.32:46966) 2021-04-09T18:18:48.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd494b700 1 -- 172.21.15.32:0/2146858408 learned_addr learned my addr 172.21.15.32:0/2146858408 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:48.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabcffff700 1 --2- 172.21.15.32:0/2146858408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd010d010 0x7fabd00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:48.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabc7fff700 1 --2- 172.21.15.32:0/2146858408 >> v2:172.21.15.86:3300/0 conn(0x7fabd00d1da0 0x7fabd00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd494b700 1 -- 172.21.15.32:0/2146858408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd010d010 msgr2=0x7fabd00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd494b700 1 --2- 172.21.15.32:0/2146858408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd010d010 0x7fabd00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd494b700 1 -- 172.21.15.32:0/2146858408 >> v2:172.21.15.86:3300/0 conn(0x7fabd00d1da0 msgr2=0x7fabd00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd494b700 1 --2- 172.21.15.32:0/2146858408 >> v2:172.21.15.86:3300/0 conn(0x7fabd00d1da0 0x7fabd00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd494b700 1 -- 172.21.15.32:0/2146858408 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd010d480 con 0x7fabd00d82c0 2021-04-09T18:18:48.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabd494b700 1 --2- 172.21.15.32:0/2146858408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd00d82c0 0x7fabd010da00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7fabc8006f00 tx=0x7fabc8009e90).ready entity=mon.0 client_cookie=581a95ca92574d01 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:48.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.604 7fabcf7fe700 1 -- 172.21.15.32:0/2146858408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc800b300 con 0x7fabd00d82c0 2021-04-09T18:18:48.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabcf7fe700 1 -- 172.21.15.32:0/2146858408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc800b640 con 0x7fabd00d82c0 2021-04-09T18:18:48.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabcf7fe700 1 -- 172.21.15.32:0/2146858408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc800bb20 con 0x7fabd00d82c0 2021-04-09T18:18:48.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabd632e700 1 -- 172.21.15.32:0/2146858408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd00d82c0 msgr2=0x7fabd010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:48.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabd632e700 1 --2- 172.21.15.32:0/2146858408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd00d82c0 0x7fabd010da00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7fabc8006f00 tx=0x7fabc8009e90).stop 2021-04-09T18:18:48.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabd632e700 1 -- 172.21.15.32:0/2146858408 shutdown_connections 2021-04-09T18:18:48.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabd632e700 1 --2- 172.21.15.32:0/2146858408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd00d82c0 0x7fabd010da00 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabd632e700 1 --2- 172.21.15.32:0/2146858408 >> v2:172.21.15.86:3300/0 conn(0x7fabd00d1da0 0x7fabd00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabd632e700 1 --2- 172.21.15.32:0/2146858408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd010d010 0x7fabd00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabd632e700 1 -- 172.21.15.32:0/2146858408 >> 172.21.15.32:0/2146858408 conn(0x7fabd00ce5f0 msgr2=0x7fabd00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:48.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabd632e700 1 -- 172.21.15.32:0/2146858408 shutdown_connections 2021-04-09T18:18:48.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.605 7fabd632e700 1 -- 172.21.15.32:0/2146858408 wait complete. 2021-04-09T18:18:48.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.625 7fabd632e700 1 Processor -- start 2021-04-09T18:18:48.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd632e700 1 -- start start 2021-04-09T18:18:48.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd632e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00b42d0 0x7fabd0183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:48.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd632e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd01ac090 0x7fabd01ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:48.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd632e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd0184d30 0x7fabd0185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd632e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00a1a90 con 0x7fabd0184d30 2021-04-09T18:18:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd632e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01acab0 con 0x7fabd01ac090 2021-04-09T18:18:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd632e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0185a00 con 0x7fabd00b42d0 2021-04-09T18:18:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd494b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd0184d30 0x7fabd0185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd494b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd0184d30 0x7fabd0185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46972/0 (socket says 172.21.15.32:46972) 2021-04-09T18:18:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabd494b700 1 -- 172.21.15.32:0/3854129373 learned_addr learned my addr 172.21.15.32:0/3854129373 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:48.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabcffff700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00b42d0 0x7fabd0183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:48.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.626 7fabc7fff700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd01ac090 0x7fabd01ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:48.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabd494b700 1 -- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00b42d0 msgr2=0x7fabd0183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabd494b700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00b42d0 0x7fabd0183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabd494b700 1 -- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd01ac090 msgr2=0x7fabd01ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:48.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabd494b700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd01ac090 0x7fabd01ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabd494b700 1 -- 172.21.15.32:0/3854129373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd0185750 con 0x7fabd0184d30 2021-04-09T18:18:48.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabd494b700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd0184d30 0x7fabd0185130 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7fabc800b8e0 tx=0x7fabc800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:48.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabcd7fa700 1 -- 172.21.15.32:0/3854129373 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc800bdc0 con 0x7fabd0184d30 2021-04-09T18:18:48.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabcd7fa700 1 -- 172.21.15.32:0/3854129373 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc8005820 con 0x7fabd0184d30 2021-04-09T18:18:48.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabd632e700 1 -- 172.21.15.32:0/3854129373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd0185750 con 0x7fabd0184d30 2021-04-09T18:18:48.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabcd7fa700 1 -- 172.21.15.32:0/3854129373 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc8005d00 con 0x7fabd0184d30 2021-04-09T18:18:48.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.627 7fabd632e700 1 -- 172.21.15.32:0/3854129373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd0185750 con 0x7fabd0184d30 2021-04-09T18:18:48.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.628 7fabcd7fa700 1 -- 172.21.15.32:0/3854129373 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fabc8005a00 con 0x7fabd0184d30 2021-04-09T18:18:48.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.628 7fabcd7fa700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabb40188d0 0x7fabb401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.628 7fabcd7fa700 1 -- 172.21.15.32:0/3854129373 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fabc8032d30 con 0x7fabd0184d30 2021-04-09T18:18:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.628 7fabcffff700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabb40188d0 0x7fabb401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.628 7fabcffff700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabb40188d0 0x7fabb401add0 crc :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.648 7fabd632e700 1 -- 172.21.15.32:0/3854129373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd0182d10 con 0x7fabd0184d30 2021-04-09T18:18:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.651 7fabcd7fa700 1 -- 172.21.15.32:0/3854129373 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fabc8005d00 con 0x7fabd0184d30 2021-04-09T18:18:48.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.842 7fabd632e700 1 -- 172.21.15.32:0/3854129373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabd010d480 con 0x7fabd0184d30 2021-04-09T18:18:48.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.843 7fabcd7fa700 1 -- 172.21.15.32:0/3854129373 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2795 (secure 0 0 0) 0x7fabc8005820 con 0x7fabd0184d30 2021-04-09T18:18:48.899 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 96 2021-04-09T18:18:48.900 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:48.900 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":96,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:48.399430","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6526,"mds_1":6654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6654":{"gid":6654,"name":"c","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 -- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabb40188d0 msgr2=0x7fabb401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:48.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabb40188d0 0x7fabb401add0 crc :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 -- 172.21.15.32:0/3854129373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd0184d30 msgr2=0x7fabd0185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:48.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd0184d30 0x7fabd0185130 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7fabc800b8e0 tx=0x7fabc800bc80).stop 2021-04-09T18:18:48.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 -- 172.21.15.32:0/3854129373 shutdown_connections 2021-04-09T18:18:48.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabb40188d0 0x7fabb401add0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabd0184d30 0x7fabd0185130 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd01ac090 0x7fabd01ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 --2- 172.21.15.32:0/3854129373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd00b42d0 0x7fabd0183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:48.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 -- 172.21.15.32:0/3854129373 >> 172.21.15.32:0/3854129373 conn(0x7fabd019b890 msgr2=0x7fabd01a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:48.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 -- 172.21.15.32:0/3854129373 shutdown_connections 2021-04-09T18:18:48.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:48.845 7fabd632e700 1 -- 172.21.15.32:0/3854129373 wait complete. 2021-04-09T18:18:48.969 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 96, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 7, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:18:33.664336", "damaged": [], "data_pools": [ 15 ], "enabled": true, "epoch": 96, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6526": { "addr": "172.21.15.32:6835/2067624954", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6833", "nonce": 2067624954, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 2067624954, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6526, "incarnation": 88, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 }, "gid_6654": { "addr": "172.21.15.32:6834/1291090515", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 1291090515, "type": "v2" }, { "addr": "172.21.15.32:6834", "nonce": 1291090515, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6654, "incarnation": 93, "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 14, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:18:48.399430", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6526, "mds_1": 6654 } } } ], "standbys": [ { "addr": "172.21.15.86:6836/2928911362", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 2928911362, "type": "v2" }, { "addr": "172.21.15.86:6836", "nonce": 2928911362, "type": "v1" } ] }, "epoch": 87, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55799, "incarnation": 0, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6837/3460852521", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6835", "nonce": 3460852521, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 3460852521, "type": "v1" } ] }, "epoch": 87, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55802, "incarnation": 0, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:18:48.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:18:49.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.110 7f4285f88700 1 Processor -- start 2021-04-09T18:18:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.110 7f4285f88700 1 -- start start 2021-04-09T18:18:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f4285f88700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f42800d7ca0 0x7f42800d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f4285f88700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f428010da00 0x7f428010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f4285f88700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428010d480 con 0x7f428010d010 2021-04-09T18:18:49.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f4285f88700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42800d9220 con 0x7f42800d7ca0 2021-04-09T18:18:49.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f4285f88700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42800d86c0 con 0x7f428010da00 2021-04-09T18:18:49.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427effd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f42800d7ca0 0x7f42800d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427f7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f428010d010 0x7f42800d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56566/0 (socket says 172.21.15.32:56566) 2021-04-09T18:18:49.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427f7fe700 1 -- 172.21.15.32:0/4118979510 learned_addr learned my addr 172.21.15.32:0/4118979510 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:49.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427ffff700 1 --2- 172.21.15.32:0/4118979510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f428010da00 0x7f428010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427e7fc700 1 -- 172.21.15.32:0/4118979510 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698974541 0 0) 0x7f4270000ce0 con 0x7f428010d010 2021-04-09T18:18:49.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427effd700 1 -- 172.21.15.32:0/4118979510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f428010da00 msgr2=0x7f428010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:49.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427effd700 1 --2- 172.21.15.32:0/4118979510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f428010da00 0x7f428010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427effd700 1 -- 172.21.15.32:0/4118979510 >> v1:172.21.15.32:6789/0 conn(0x7f428010d010 legacy=0x7f42800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427effd700 1 -- 172.21.15.32:0/4118979510 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800d1f70 con 0x7f42800d7ca0 2021-04-09T18:18:49.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427ffff700 1 --2- 172.21.15.32:0/4118979510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f428010da00 0x7f428010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:49.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.111 7f427effd700 1 --2- 172.21.15.32:0/4118979510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42800d7ca0 0x7f42800d80a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f42740069d0 tx=0x7f4274009e40).ready entity=mon.0 client_cookie=443bb71e47f89542 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:49.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f427e7fc700 1 -- 172.21.15.32:0/4118979510 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f427400b510 con 0x7f42800d7ca0 2021-04-09T18:18:49.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f427e7fc700 1 -- 172.21.15.32:0/4118979510 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427400b890 con 0x7f42800d7ca0 2021-04-09T18:18:49.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f427e7fc700 1 -- 172.21.15.32:0/4118979510 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f427400bd70 con 0x7f42800d7ca0 2021-04-09T18:18:49.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f4285f88700 1 -- 172.21.15.32:0/4118979510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42800d7ca0 msgr2=0x7f42800d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f4285f88700 1 --2- 172.21.15.32:0/4118979510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42800d7ca0 0x7f42800d80a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f42740069d0 tx=0x7f4274009e40).stop 2021-04-09T18:18:49.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f4285f88700 1 -- 172.21.15.32:0/4118979510 shutdown_connections 2021-04-09T18:18:49.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f4285f88700 1 --2- 172.21.15.32:0/4118979510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f428010da00 0x7f428010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f4285f88700 1 --2- 172.21.15.32:0/4118979510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42800d7ca0 0x7f42800d80a0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f4285f88700 1 -- 172.21.15.32:0/4118979510 >> 172.21.15.32:0/4118979510 conn(0x7f42800ce5f0 msgr2=0x7f42800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:49.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f4285f88700 1 -- 172.21.15.32:0/4118979510 shutdown_connections 2021-04-09T18:18:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.112 7f4285f88700 1 -- 172.21.15.32:0/4118979510 wait complete. 2021-04-09T18:18:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.132 7f4285f88700 1 Processor -- start 2021-04-09T18:18:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f4285f88700 1 -- start start 2021-04-09T18:18:49.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f4285f88700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f428017bb20 0x7f42800c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f4285f88700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42800b7620 0x7f42800c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f4285f88700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428017bf20 0x7f428008a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f4285f88700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42800c1e00 con 0x7f428017bf20 2021-04-09T18:18:49.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f4285f88700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42800c0b00 con 0x7f428017bb20 2021-04-09T18:18:49.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f4285f88700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428008ac40 con 0x7f42800b7620 2021-04-09T18:18:49.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f427ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428017bf20 0x7f428008a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f427ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428017bf20 0x7f428008a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46986/0 (socket says 172.21.15.32:46986) 2021-04-09T18:18:49.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f427ffff700 1 -- 172.21.15.32:0/1601741052 learned_addr learned my addr 172.21.15.32:0/1601741052 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:49.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f427f7fe700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f428017bb20 0x7f42800c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.133 7f427effd700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42800b7620 0x7f42800c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f427ffff700 1 -- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42800b7620 msgr2=0x7f42800c16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f427ffff700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42800b7620 0x7f42800c16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f427ffff700 1 -- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f428017bb20 msgr2=0x7f42800c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f427ffff700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f428017bb20 0x7f42800c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f427ffff700 1 -- 172.21.15.32:0/1601741052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428008ae60 con 0x7f428017bf20 2021-04-09T18:18:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f427ffff700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428017bf20 0x7f428008a620 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f4278006f10 tx=0x7f427800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f4267fff700 1 -- 172.21.15.32:0/1601741052 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f427800bd80 con 0x7f428017bf20 2021-04-09T18:18:49.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f4267fff700 1 -- 172.21.15.32:0/1601741052 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4278006410 con 0x7f428017bf20 2021-04-09T18:18:49.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f4285f88700 1 -- 172.21.15.32:0/1601741052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428008ae60 con 0x7f428017bf20 2021-04-09T18:18:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f4267fff700 1 -- 172.21.15.32:0/1601741052 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42780022b0 con 0x7f428017bf20 2021-04-09T18:18:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.134 7f4285f88700 1 -- 172.21.15.32:0/1601741052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428008ae60 con 0x7f428017bf20 2021-04-09T18:18:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.135 7f4267fff700 1 -- 172.21.15.32:0/1601741052 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f427800a030 con 0x7f428017bf20 2021-04-09T18:18:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.135 7f4267fff700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42600188c0 0x7f426001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.135 7f4267fff700 1 -- 172.21.15.32:0/1601741052 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4278032a10 con 0x7f428017bf20 2021-04-09T18:18:49.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.135 7f427f7fe700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42600188c0 0x7f426001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.135 7f427f7fe700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42600188c0 0x7f426001adc0 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:49.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.155 7f4285f88700 1 -- 172.21.15.32:0/1601741052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428010d480 con 0x7f428017bf20 2021-04-09T18:18:49.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.159 7f4267fff700 1 -- 172.21.15.32:0/1601741052 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4278006ba0 con 0x7f428017bf20 2021-04-09T18:18:49.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.352 7f4285f88700 1 -- 172.21.15.32:0/1601741052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f428010d480 con 0x7f428017bf20 2021-04-09T18:18:49.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.402 7f4267fff700 1 -- 172.21.15.32:0/1601741052 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v97) v1 ==== 105+0+0 (secure 0 0 0) 0x7f427800a030 con 0x7f428017bf20 2021-04-09T18:18:49.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 -- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42600188c0 msgr2=0x7f426001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42600188c0 0x7f426001adc0 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 -- 172.21.15.32:0/1601741052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428017bf20 msgr2=0x7f428008a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428017bf20 0x7f428008a620 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f4278006f10 tx=0x7f427800b990).stop 2021-04-09T18:18:49.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 -- 172.21.15.32:0/1601741052 shutdown_connections 2021-04-09T18:18:49.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42600188c0 0x7f426001adc0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f428017bf20 0x7f428008a620 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42800b7620 0x7f42800c16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 --2- 172.21.15.32:0/1601741052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f428017bb20 0x7f42800c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 -- 172.21.15.32:0/1601741052 >> 172.21.15.32:0/1601741052 conn(0x7f4280181390 msgr2=0x7f42800adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 -- 172.21.15.32:0/1601741052 shutdown_connections 2021-04-09T18:18:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.418 7f4285f88700 1 -- 172.21.15.32:0/1601741052 wait complete. 2021-04-09T18:18:49.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:18:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.581 7f8b9941e700 1 Processor -- start 2021-04-09T18:18:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.581 7f8b9941e700 1 -- start start 2021-04-09T18:18:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9941e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8b9410da00 0x7f8b9410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9941e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9410d480 con 0x7f8b940d7ca0 2021-04-09T18:18:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9941e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b940d1f70 con 0x7f8b9410d010 2021-04-09T18:18:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9941e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b940d8d30 con 0x7f8b9410da00 2021-04-09T18:18:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9287c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8b940d7ca0 0x7f8b940d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56582/0 (socket says 172.21.15.32:56582) 2021-04-09T18:18:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9287c700 1 -- 172.21.15.32:0/3452823300 learned_addr learned my addr 172.21.15.32:0/3452823300 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9387e700 1 --2- 172.21.15.32:0/3452823300 >> v2:172.21.15.86:3300/0 conn(0x7f8b9410da00 0x7f8b9410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9207b700 1 -- 172.21.15.32:0/3452823300 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376782169 0 0) 0x7f8b88000ce0 con 0x7f8b940d7ca0 2021-04-09T18:18:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9207b700 1 -- 172.21.15.32:0/3452823300 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b7c002fe0 con 0x7f8b940d7ca0 2021-04-09T18:18:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9207b700 1 -- 172.21.15.32:0/3452823300 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487873397 0 0) 0x7f8b84000c60 con 0x7f8b9410d010 2021-04-09T18:18:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.582 7f8b9207b700 1 -- 172.21.15.32:0/3452823300 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b7c003210 con 0x7f8b9410d010 2021-04-09T18:18:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9387e700 1 -- 172.21.15.32:0/3452823300 >> v1:172.21.15.86:6789/0 conn(0x7f8b9410d010 legacy=0x7f8b940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9207b700 1 -- 172.21.15.32:0/3452823300 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3175163407 0 0) 0x7f8b880010e0 con 0x7f8b940d7ca0 2021-04-09T18:18:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9387e700 1 -- 172.21.15.32:0/3452823300 >> v1:172.21.15.32:6789/0 conn(0x7f8b940d7ca0 legacy=0x7f8b940d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9387e700 1 -- 172.21.15.32:0/3452823300 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b940d9220 con 0x7f8b9410da00 2021-04-09T18:18:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9387e700 1 --2- 172.21.15.32:0/3452823300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9410da00 0x7f8b9410de00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f8b8c006e50 tx=0x7f8b8c009e70).ready entity=mon.1 client_cookie=befbee911fea3ba7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9207b700 1 -- 172.21.15.32:0/3452823300 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8c006a20 con 0x7f8b9410da00 2021-04-09T18:18:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9207b700 1 -- 172.21.15.32:0/3452823300 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c006790 con 0x7f8b9410da00 2021-04-09T18:18:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9207b700 1 -- 172.21.15.32:0/3452823300 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8c00b700 con 0x7f8b9410da00 2021-04-09T18:18:49.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9941e700 1 -- 172.21.15.32:0/3452823300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9410da00 msgr2=0x7f8b9410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.583 7f8b9941e700 1 --2- 172.21.15.32:0/3452823300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9410da00 0x7f8b9410de00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f8b8c006e50 tx=0x7f8b8c009e70).stop 2021-04-09T18:18:49.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.584 7f8b9941e700 1 -- 172.21.15.32:0/3452823300 shutdown_connections 2021-04-09T18:18:49.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.584 7f8b9941e700 1 --2- 172.21.15.32:0/3452823300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9410da00 0x7f8b9410de00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.584 7f8b9941e700 1 -- 172.21.15.32:0/3452823300 >> 172.21.15.32:0/3452823300 conn(0x7f8b940ce5f0 msgr2=0x7f8b940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:49.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.584 7f8b9941e700 1 -- 172.21.15.32:0/3452823300 shutdown_connections 2021-04-09T18:18:49.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.584 7f8b9941e700 1 -- 172.21.15.32:0/3452823300 wait complete. 2021-04-09T18:18:49.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.604 7f8b9941e700 1 Processor -- start 2021-04-09T18:18:49.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.604 7f8b9941e700 1 -- start start 2021-04-09T18:18:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.604 7f8b9941e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b940b1680 0x7f8b940b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.604 7f8b9941e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9409e330 0x7f8b940b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.604 7f8b9941e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b940b2190 0x7f8b9409f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.604 7f8b9941e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9409e730 con 0x7f8b9409e330 2021-04-09T18:18:49.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.604 7f8b9941e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9409f6c0 con 0x7f8b940b1680 2021-04-09T18:18:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.604 7f8b9941e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9409f8e0 con 0x7f8b940b2190 2021-04-09T18:18:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9287c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9409e330 0x7f8b940b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9287c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9409e330 0x7f8b940b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46996/0 (socket says 172.21.15.32:46996) 2021-04-09T18:18:49.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9287c700 1 -- 172.21.15.32:0/2046401552 learned_addr learned my addr 172.21.15.32:0/2046401552 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9387e700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b940b2190 0x7f8b9409f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9307d700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b940b1680 0x7f8b940b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9287c700 1 -- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b940b2190 msgr2=0x7f8b9409f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9287c700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b940b2190 0x7f8b9409f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9287c700 1 -- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b940b1680 msgr2=0x7f8b940b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9287c700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b940b1680 0x7f8b940b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9287c700 1 -- 172.21.15.32:0/2046401552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9409fb00 con 0x7f8b9409e330 2021-04-09T18:18:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9287c700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9409e330 0x7f8b940b1b70 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f8b88000f90 tx=0x7f8b8800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:49.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b7bfff700 1 -- 172.21.15.32:0/2046401552 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b88001a30 con 0x7f8b9409e330 2021-04-09T18:18:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9409eb70 con 0x7f8b9409e330 2021-04-09T18:18:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b7bfff700 1 -- 172.21.15.32:0/2046401552 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b88007070 con 0x7f8b9409e330 2021-04-09T18:18:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b7bfff700 1 -- 172.21.15.32:0/2046401552 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b880043a0 con 0x7f8b9409e330 2021-04-09T18:18:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.605 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9409eb70 con 0x7f8b9409e330 2021-04-09T18:18:49.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.606 7f8b7bfff700 1 -- 172.21.15.32:0/2046401552 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8b880043a0 con 0x7f8b9409e330 2021-04-09T18:18:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.606 7f8b7bfff700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b7401cd30 0x7f8b7401f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.606 7f8b7bfff700 1 -- 172.21.15.32:0/2046401552 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8b880329b0 con 0x7f8b9409e330 2021-04-09T18:18:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.606 7f8b9307d700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b7401cd30 0x7f8b7401f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:49.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.607 7f8b9307d700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b7401cd30 0x7f8b7401f230 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:49.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.626 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9409eb70 con 0x7f8b9409e330 2021-04-09T18:18:49.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.630 7f8b7bfff700 1 -- 172.21.15.32:0/2046401552 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8b88006f70 con 0x7f8b9409e330 2021-04-09T18:18:49.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:49.825 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f8b9409eb70 con 0x7f8b9409e330 2021-04-09T18:18:50.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.408 7f8b7bfff700 1 -- 172.21.15.32:0/2046401552 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v98) v1 ==== 105+0+0 (secure 0 0 0) 0x7f8b88001a10 con 0x7f8b9409e330 2021-04-09T18:18:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b7401cd30 msgr2=0x7f8b7401f230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b7401cd30 0x7f8b7401f230 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9409e330 msgr2=0x7f8b940b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9409e330 0x7f8b940b1b70 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f8b88000f90 tx=0x7f8b8800a940).stop 2021-04-09T18:18:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 shutdown_connections 2021-04-09T18:18:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b7401cd30 0x7f8b7401f230 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9409e330 0x7f8b940b1b70 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b940b2190 0x7f8b9409f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 --2- 172.21.15.32:0/2046401552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b940b1680 0x7f8b940b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 >> 172.21.15.32:0/2046401552 conn(0x7f8b940c4290 msgr2=0x7f8b940c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 shutdown_connections 2021-04-09T18:18:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.443 7f8b9941e700 1 -- 172.21.15.32:0/2046401552 wait complete. 2021-04-09T18:18:50.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable false 2021-04-09T18:18:50.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.649 7f85a05fc700 1 Processor -- start 2021-04-09T18:18:50.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.649 7f85a05fc700 1 -- start start 2021-04-09T18:18:50.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.650 7f85a05fc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f859810d010 0x7f85980d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:50.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.650 7f85a05fc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980d8490 0x7f859810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:50.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.650 7f85a05fc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859810d480 con 0x7f85980d1da0 2021-04-09T18:18:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.650 7f85a05fc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85980d8d30 con 0x7f859810d010 2021-04-09T18:18:50.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.650 7f85a05fc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980d9220 con 0x7f85980d8490 2021-04-09T18:18:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.650 7f859e418700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f859810d010 0x7f85980d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:50.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.650 7f859dc17700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f85980d1da0 0x7f85980d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56596/0 (socket says 172.21.15.32:56596) 2021-04-09T18:18:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.650 7f859dc17700 1 -- 172.21.15.32:0/2288256214 learned_addr learned my addr 172.21.15.32:0/2288256214 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:50.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.650 7f859ec19700 1 --2- 172.21.15.32:0/2288256214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980d8490 0x7f859810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:50.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f859e418700 1 -- 172.21.15.32:0/2288256214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980d8490 msgr2=0x7f859810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:50.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f859e418700 1 --2- 172.21.15.32:0/2288256214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980d8490 0x7f859810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f859e418700 1 -- 172.21.15.32:0/2288256214 >> v1:172.21.15.32:6789/0 conn(0x7f85980d1da0 legacy=0x7f85980d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:50.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f859e418700 1 -- 172.21.15.32:0/2288256214 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859810e020 con 0x7f859810d010 2021-04-09T18:18:50.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f859d416700 1 -- 172.21.15.32:0/2288256214 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377542705 0 0) 0x7f8588000ce0 con 0x7f85980d1da0 2021-04-09T18:18:50.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f859e418700 1 --2- 172.21.15.32:0/2288256214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859810d010 0x7f85980d1780 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f8594006650 tx=0x7f8594009a00).ready entity=mon.0 client_cookie=5468364c39ea34cb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:50.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f859d416700 1 -- 172.21.15.32:0/2288256214 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8594009df0 con 0x7f859810d010 2021-04-09T18:18:50.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f859d416700 1 -- 172.21.15.32:0/2288256214 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85940009b0 con 0x7f859810d010 2021-04-09T18:18:50.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f859d416700 1 -- 172.21.15.32:0/2288256214 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f859400c2b0 con 0x7f859810d010 2021-04-09T18:18:50.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f85a05fc700 1 -- 172.21.15.32:0/2288256214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859810d010 msgr2=0x7f85980d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:50.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.651 7f85a05fc700 1 --2- 172.21.15.32:0/2288256214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859810d010 0x7f85980d1780 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f8594006650 tx=0x7f8594009a00).stop 2021-04-09T18:18:50.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.652 7f85a05fc700 1 -- 172.21.15.32:0/2288256214 shutdown_connections 2021-04-09T18:18:50.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.652 7f85a05fc700 1 --2- 172.21.15.32:0/2288256214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980d8490 0x7f859810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.652 7f85a05fc700 1 --2- 172.21.15.32:0/2288256214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f859810d010 0x7f85980d1780 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.652 7f85a05fc700 1 -- 172.21.15.32:0/2288256214 >> 172.21.15.32:0/2288256214 conn(0x7f85980ce5f0 msgr2=0x7f85980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:50.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.652 7f85a05fc700 1 -- 172.21.15.32:0/2288256214 shutdown_connections 2021-04-09T18:18:50.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.652 7f85a05fc700 1 -- 172.21.15.32:0/2288256214 wait complete. 2021-04-09T18:18:50.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.672 7f85a05fc700 1 Processor -- start 2021-04-09T18:18:50.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.672 7f85a05fc700 1 -- start start 2021-04-09T18:18:50.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f85a05fc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980b1e50 0x7f85980b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:50.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f85a05fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85980b9fd0 0x7f85980b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:50.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f85a05fc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85980b5510 0x7f85980b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:50.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f85a05fc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85980b2250 con 0x7f85980b9fd0 2021-04-09T18:18:50.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f85a05fc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85980a1220 con 0x7f85980b5510 2021-04-09T18:18:50.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f85a05fc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980b6b30 con 0x7f85980b1e50 2021-04-09T18:18:50.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859dc17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85980b9fd0 0x7f85980b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:50.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859dc17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85980b9fd0 0x7f85980b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47012/0 (socket says 172.21.15.32:47012) 2021-04-09T18:18:50.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859dc17700 1 -- 172.21.15.32:0/3243080575 learned_addr learned my addr 172.21.15.32:0/3243080575 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:50.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859ec19700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85980b5510 0x7f85980b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:50.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859e418700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980b1e50 0x7f85980b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:50.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859dc17700 1 -- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980b1e50 msgr2=0x7f85980b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:50.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859dc17700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980b1e50 0x7f85980b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859dc17700 1 -- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85980b5510 msgr2=0x7f85980b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:50.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859dc17700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85980b5510 0x7f85980b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:50.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859dc17700 1 -- 172.21.15.32:0/3243080575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980b5f30 con 0x7f85980b9fd0 2021-04-09T18:18:50.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f859dc17700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85980b9fd0 0x7f85980b4ef0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7f8588006f00 tx=0x7f858800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f858effd700 1 -- 172.21.15.32:0/3243080575 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8588006020 con 0x7f85980b9fd0 2021-04-09T18:18:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f858effd700 1 -- 172.21.15.32:0/3243080575 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85880063a0 con 0x7f85980b9fd0 2021-04-09T18:18:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85980b5f30 con 0x7f85980b9fd0 2021-04-09T18:18:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f858effd700 1 -- 172.21.15.32:0/3243080575 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8588004620 con 0x7f85980b9fd0 2021-04-09T18:18:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.673 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85980b5f30 con 0x7f85980b9fd0 2021-04-09T18:18:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.674 7f858effd700 1 -- 172.21.15.32:0/3243080575 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f858800a000 con 0x7f85980b9fd0 2021-04-09T18:18:50.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.674 7f858effd700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8580018900 0x7f858001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:50.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.674 7f858effd700 1 -- 172.21.15.32:0/3243080575 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8588032a10 con 0x7f85980b9fd0 2021-04-09T18:18:50.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.674 7f859e418700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8580018900 0x7f858001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:50.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.675 7f859e418700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8580018900 0x7f858001adc0 crc :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.694 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85981b7040 con 0x7f85980b9fd0 2021-04-09T18:18:50.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.698 7f858effd700 1 -- 172.21.15.32:0/3243080575 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8588006b90 con 0x7f85980b9fd0 2021-04-09T18:18:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:50.890 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "joinable", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7f85981b7040 con 0x7f85980b9fd0 2021-04-09T18:18:51.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.409 7f858effd700 1 -- 172.21.15.32:0/3243080575 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "joinable", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 cephfs marked not joinable; MDS cannot join as newly active. v99) v1 ==== 170+0+0 (secure 0 0 0) 0x7f858800a000 con 0x7f85980b9fd0 2021-04-09T18:18:51.519 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join as newly active. 2021-04-09T18:18:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8580018900 msgr2=0x7f858001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:51.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8580018900 0x7f858001adc0 crc :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85980b9fd0 msgr2=0x7f85980b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:51.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85980b9fd0 0x7f85980b4ef0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7f8588006f00 tx=0x7f858800b940).stop 2021-04-09T18:18:51.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 shutdown_connections 2021-04-09T18:18:51.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8580018900 0x7f858001adc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85980b9fd0 0x7f85980b4ef0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85980b5510 0x7f85980b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 --2- 172.21.15.32:0/3243080575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85980b1e50 0x7f85980b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 >> 172.21.15.32:0/3243080575 conn(0x7f85980b27e0 msgr2=0x7f85981b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:51.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 shutdown_connections 2021-04-09T18:18:51.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.457 7f85a05fc700 1 -- 172.21.15.32:0/3243080575 wait complete. 2021-04-09T18:18:51.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.644 7f0c5c0e7700 1 Processor -- start 2021-04-09T18:18:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.644 7f0c5c0e7700 1 -- start start 2021-04-09T18:18:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c5c0e7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c540d7ca0 0x7f0c540d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:51.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c5c0e7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0c5410da00 0x7f0c5410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:51.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c5c0e7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5410d480 con 0x7f0c5410d010 2021-04-09T18:18:51.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c5c0e7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c540d9220 con 0x7f0c540d7ca0 2021-04-09T18:18:51.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c5c0e7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c540d86c0 con 0x7f0c5410da00 2021-04-09T18:18:51.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c59702700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c540d7ca0 0x7f0c540d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:51.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c59702700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c540d7ca0 0x7f0c540d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47022/0 (socket says 172.21.15.32:47022) 2021-04-09T18:18:51.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c59702700 1 -- 172.21.15.32:0/2025846678 learned_addr learned my addr 172.21.15.32:0/2025846678 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:51.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c5a704700 1 --2- 172.21.15.32:0/2025846678 >> v2:172.21.15.86:3300/0 conn(0x7f0c5410da00 0x7f0c5410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:51.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c59702700 1 -- 172.21.15.32:0/2025846678 >> v2:172.21.15.86:3300/0 conn(0x7f0c5410da00 msgr2=0x7f0c5410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:51.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c59702700 1 --2- 172.21.15.32:0/2025846678 >> v2:172.21.15.86:3300/0 conn(0x7f0c5410da00 0x7f0c5410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c59702700 1 -- 172.21.15.32:0/2025846678 >> v1:172.21.15.86:6789/0 conn(0x7f0c5410d010 legacy=0x7f0c540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:51.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c59702700 1 -- 172.21.15.32:0/2025846678 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c54112100 con 0x7f0c540d7ca0 2021-04-09T18:18:51.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c5a704700 1 --2- 172.21.15.32:0/2025846678 >> v2:172.21.15.86:3300/0 conn(0x7f0c5410da00 0x7f0c5410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:51.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.645 7f0c59702700 1 --2- 172.21.15.32:0/2025846678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c540d7ca0 0x7f0c540d80a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f0c44006700 tx=0x7f0c44009b80).ready entity=mon.0 client_cookie=3dd86d58a084e977 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:51.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.646 7f0c58f01700 1 -- 172.21.15.32:0/2025846678 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c4400c0c0 con 0x7f0c540d7ca0 2021-04-09T18:18:51.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.646 7f0c58f01700 1 -- 172.21.15.32:0/2025846678 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4400c440 con 0x7f0c540d7ca0 2021-04-09T18:18:51.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.646 7f0c58f01700 1 -- 172.21.15.32:0/2025846678 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c4400c920 con 0x7f0c540d7ca0 2021-04-09T18:18:51.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.646 7f0c5c0e7700 1 -- 172.21.15.32:0/2025846678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c540d7ca0 msgr2=0x7f0c540d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:51.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.646 7f0c5c0e7700 1 --2- 172.21.15.32:0/2025846678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c540d7ca0 0x7f0c540d80a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f0c44006700 tx=0x7f0c44009b80).stop 2021-04-09T18:18:51.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.646 7f0c5c0e7700 1 -- 172.21.15.32:0/2025846678 shutdown_connections 2021-04-09T18:18:51.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.646 7f0c5c0e7700 1 --2- 172.21.15.32:0/2025846678 >> v2:172.21.15.86:3300/0 conn(0x7f0c5410da00 0x7f0c5410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.646 7f0c5c0e7700 1 --2- 172.21.15.32:0/2025846678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c540d7ca0 0x7f0c540d80a0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.646 7f0c5c0e7700 1 -- 172.21.15.32:0/2025846678 >> 172.21.15.32:0/2025846678 conn(0x7f0c540ce5f0 msgr2=0x7f0c540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:51.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.647 7f0c5c0e7700 1 -- 172.21.15.32:0/2025846678 shutdown_connections 2021-04-09T18:18:51.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.647 7f0c5c0e7700 1 -- 172.21.15.32:0/2025846678 wait complete. 2021-04-09T18:18:51.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c5c0e7700 1 Processor -- start 2021-04-09T18:18:51.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c5c0e7700 1 -- start start 2021-04-09T18:18:51.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c5c0e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c54184a30 0x7f0c541acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:51.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c5c0e7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c541a6b90 0x7f0c541a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:51.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c5c0e7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c541a5150 0x7f0c541a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:51.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c5c0e7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541a75b0 con 0x7f0c54184a30 2021-04-09T18:18:51.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c5c0e7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c54184590 con 0x7f0c541a6b90 2021-04-09T18:18:51.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c5c0e7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541a5b70 con 0x7f0c541a5150 2021-04-09T18:18:51.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c59f03700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c54184a30 0x7f0c541acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:51.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c59f03700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c54184a30 0x7f0c541acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47024/0 (socket says 172.21.15.32:47024) 2021-04-09T18:18:51.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c59f03700 1 -- 172.21.15.32:0/2822428742 learned_addr learned my addr 172.21.15.32:0/2822428742 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:51.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.667 7f0c59702700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c541a6b90 0x7f0c541a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:51.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c5a704700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c541a5150 0x7f0c541a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:51.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c59f03700 1 -- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c541a5150 msgr2=0x7f0c541a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:51.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c59f03700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c541a5150 0x7f0c541a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c59f03700 1 -- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c541a6b90 msgr2=0x7f0c541a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c59f03700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c541a6b90 0x7f0c541a6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c59f03700 1 -- 172.21.15.32:0/2822428742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c541a5d90 con 0x7f0c54184a30 2021-04-09T18:18:51.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c5a704700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c541a5150 0x7f0c541a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:51.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c59f03700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c54184a30 0x7f0c541acb80 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7f0c5000b5b0 tx=0x7f0c5000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:51.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c4a7fc700 1 -- 172.21.15.32:0/2822428742 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c5000bd40 con 0x7f0c54184a30 2021-04-09T18:18:51.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c4a7fc700 1 -- 172.21.15.32:0/2822428742 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c50006240 con 0x7f0c54184a30 2021-04-09T18:18:51.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c4a7fc700 1 -- 172.21.15.32:0/2822428742 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c50005690 con 0x7f0c54184a30 2021-04-09T18:18:51.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c541a5d90 con 0x7f0c54184a30 2021-04-09T18:18:51.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.668 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c541a5d90 con 0x7f0c54184a30 2021-04-09T18:18:51.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.669 7f0c4a7fc700 1 -- 172.21.15.32:0/2822428742 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0c500054a0 con 0x7f0c54184a30 2021-04-09T18:18:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.669 7f0c4a7fc700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0c3c0188d0 0x7f0c3c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.669 7f0c4a7fc700 1 -- 172.21.15.32:0/2822428742 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0c50032a30 con 0x7f0c54184a30 2021-04-09T18:18:51.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.669 7f0c59702700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0c3c0188d0 0x7f0c3c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:51.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.669 7f0c59702700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0c3c0188d0 0x7f0c3c01add0 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:51.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.689 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c541b7080 con 0x7f0c54184a30 2021-04-09T18:18:51.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.693 7f0c4a7fc700 1 -- 172.21.15.32:0/2822428742 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0c50000c60 con 0x7f0c54184a30 2021-04-09T18:18:51.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.884 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c5410d480 con 0x7f0c54184a30 2021-04-09T18:18:51.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.885 7f0c4a7fc700 1 -- 172.21.15.32:0/2822428742 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+2795 (secure 0 0 0) 0x7f0c500054a0 con 0x7f0c54184a30 2021-04-09T18:18:51.926 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2021-04-09T18:18:51.929 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:51.930 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":99,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:51.408068","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6526,"mds_1":6654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6654":{"gid":6654,"name":"c","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:51.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0c3c0188d0 msgr2=0x7f0c3c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0c3c0188d0 0x7f0c3c01add0 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c54184a30 msgr2=0x7f0c541acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:51.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c54184a30 0x7f0c541acb80 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7f0c5000b5b0 tx=0x7f0c5000b950).stop 2021-04-09T18:18:51.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 shutdown_connections 2021-04-09T18:18:51.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0c3c0188d0 0x7f0c3c01add0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c541a5150 0x7f0c541a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c541a6b90 0x7f0c541a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 --2- 172.21.15.32:0/2822428742 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c54184a30 0x7f0c541acb80 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:51.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 >> 172.21.15.32:0/2822428742 conn(0x7f0c54191560 msgr2=0x7f0c541b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:51.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 shutdown_connections 2021-04-09T18:18:51.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:51.887 7f0c5c0e7700 1 -- 172.21.15.32:0/2822428742 wait complete. 2021-04-09T18:18:52.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:51.408068', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6526, 'mds_1': 6654}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6654': {'gid': 6654, 'name': 'c', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6834/1291090515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1291090515}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 1291090515}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:52.366 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6654:up:stopping 2021-04-09T18:18:53.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.489 7f627b61b700 1 Processor -- start 2021-04-09T18:18:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.489 7f627b61b700 1 -- start start 2021-04-09T18:18:53.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f627b61b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62740cffa0 0x7f62740d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:53.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f627b61b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62740d7280 con 0x7f62740d6c80 2021-04-09T18:18:53.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f627b61b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62740d83f0 con 0x7f62740d7800 2021-04-09T18:18:53.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f627b61b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62740d7080 con 0x7f62740cffa0 2021-04-09T18:18:53.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f6279437700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62740cffa0 0x7f62740d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:53.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f6279c38700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f62740d6c80 0x7f627410cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56624/0 (socket says 172.21.15.32:56624) 2021-04-09T18:18:53.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f6279c38700 1 -- 172.21.15.32:0/2491230247 learned_addr learned my addr 172.21.15.32:0/2491230247 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:53.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f6267fff700 1 -- 172.21.15.32:0/2491230247 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173369806 0 0) 0x7f626c000d30 con 0x7f62740d6c80 2021-04-09T18:18:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f6267fff700 1 -- 172.21.15.32:0/2491230247 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6260002fe0 con 0x7f62740d6c80 2021-04-09T18:18:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f6279437700 1 -- 172.21.15.32:0/2491230247 >> v1:172.21.15.86:6789/0 conn(0x7f62740d7800 legacy=0x7f62740d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f6279437700 1 -- 172.21.15.32:0/2491230247 >> v1:172.21.15.32:6789/0 conn(0x7f62740d6c80 legacy=0x7f627410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f6279437700 1 -- 172.21.15.32:0/2491230247 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62740d91e0 con 0x7f62740cffa0 2021-04-09T18:18:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.490 7f6279437700 1 --2- 172.21.15.32:0/2491230247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62740cffa0 0x7f62740d2440 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f6270006650 tx=0x7f6270009a80).ready entity=mon.0 client_cookie=3551d693ca767cc6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.491 7f6267fff700 1 -- 172.21.15.32:0/2491230247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62700009b0 con 0x7f62740cffa0 2021-04-09T18:18:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.491 7f6267fff700 1 -- 172.21.15.32:0/2491230247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627000c0c0 con 0x7f62740cffa0 2021-04-09T18:18:53.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.491 7f6267fff700 1 -- 172.21.15.32:0/2491230247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f627000c510 con 0x7f62740cffa0 2021-04-09T18:18:53.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.491 7f627b61b700 1 -- 172.21.15.32:0/2491230247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62740cffa0 msgr2=0x7f62740d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:53.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.491 7f627b61b700 1 --2- 172.21.15.32:0/2491230247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62740cffa0 0x7f62740d2440 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f6270006650 tx=0x7f6270009a80).stop 2021-04-09T18:18:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.491 7f627b61b700 1 -- 172.21.15.32:0/2491230247 shutdown_connections 2021-04-09T18:18:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.491 7f627b61b700 1 --2- 172.21.15.32:0/2491230247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62740cffa0 0x7f62740d2440 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.491 7f627b61b700 1 -- 172.21.15.32:0/2491230247 >> 172.21.15.32:0/2491230247 conn(0x7f62740ce5c0 msgr2=0x7f62740cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.492 7f627b61b700 1 -- 172.21.15.32:0/2491230247 shutdown_connections 2021-04-09T18:18:53.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.492 7f627b61b700 1 -- 172.21.15.32:0/2491230247 wait complete. 2021-04-09T18:18:53.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.512 7f627b61b700 1 Processor -- start 2021-04-09T18:18:53.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.512 7f627b61b700 1 -- start start 2021-04-09T18:18:53.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.512 7f627b61b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62741a2770 0x7f62741a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:53.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.512 7f627b61b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62741a2b70 0x7f62741a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:53.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.512 7f627b61b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62741a3bb0 0x7f62741ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.512 7f627b61b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62741a3fb0 con 0x7f62741a2770 2021-04-09T18:18:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.512 7f627b61b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62741a1a10 con 0x7f62741a2b70 2021-04-09T18:18:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.512 7f627b61b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62741acea0 con 0x7f62741a3bb0 2021-04-09T18:18:53.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279437700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62741a2770 0x7f62741a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:53.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279437700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62741a2770 0x7f62741a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47038/0 (socket says 172.21.15.32:47038) 2021-04-09T18:18:53.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279437700 1 -- 172.21.15.32:0/2533238804 learned_addr learned my addr 172.21.15.32:0/2533238804 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279c38700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62741a3bb0 0x7f62741ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6278c36700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62741a2b70 0x7f62741a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279437700 1 -- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62741a3bb0 msgr2=0x7f62741ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:53.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279437700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62741a3bb0 0x7f62741ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:53.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279437700 1 -- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62741a2b70 msgr2=0x7f62741a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:53.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279437700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62741a2b70 0x7f62741a3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:53.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279437700 1 -- 172.21.15.32:0/2533238804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62741ad0c0 con 0x7f62741a2770 2021-04-09T18:18:53.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279c38700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62741a3bb0 0x7f62741ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:18:53.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6279437700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62741a2770 0x7f62741a13f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7f6270009650 tx=0x7f627000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:53.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6265ffb700 1 -- 172.21.15.32:0/2533238804 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f627000bd50 con 0x7f62741a2770 2021-04-09T18:18:53.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6265ffb700 1 -- 172.21.15.32:0/2533238804 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6270006030 con 0x7f62741a2770 2021-04-09T18:18:53.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f627b61b700 1 -- 172.21.15.32:0/2533238804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62741ad0c0 con 0x7f62741a2770 2021-04-09T18:18:53.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f6265ffb700 1 -- 172.21.15.32:0/2533238804 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62700054d0 con 0x7f62741a2770 2021-04-09T18:18:53.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.513 7f627b61b700 1 -- 172.21.15.32:0/2533238804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62741ad0c0 con 0x7f62741a2770 2021-04-09T18:18:53.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.514 7f6265ffb700 1 -- 172.21.15.32:0/2533238804 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f62700052e0 con 0x7f62741a2770 2021-04-09T18:18:53.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.514 7f6265ffb700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6254018930 0x7f625401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:53.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.514 7f6265ffb700 1 -- 172.21.15.32:0/2533238804 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6270032b70 con 0x7f62741a2770 2021-04-09T18:18:53.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.514 7f6278c36700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6254018930 0x7f625401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:53.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.515 7f6278c36700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6254018930 0x7f625401adc0 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:53.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.534 7f627b61b700 1 -- 172.21.15.32:0/2533238804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62741a3fb0 con 0x7f62741a2770 2021-04-09T18:18:53.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.537 7f6265ffb700 1 -- 172.21.15.32:0/2533238804 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6270000940 con 0x7f62741a2770 2021-04-09T18:18:53.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.730 7f627b61b700 1 -- 172.21.15.32:0/2533238804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62741a3fb0 con 0x7f62741a2770 2021-04-09T18:18:53.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.731 7f6265ffb700 1 -- 172.21.15.32:0/2533238804 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+2795 (secure 0 0 0) 0x7f6270006030 con 0x7f62741a2770 2021-04-09T18:18:53.771 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 99 2021-04-09T18:18:53.772 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:53.773 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":99,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:51.408068","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6526,"mds_1":6654},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_6654":{"gid":6654,"name":"c","rank":1,"incarnation":93,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6834/1291090515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1291090515},{"type":"v1","addr":"172.21.15.32:6834","nonce":1291090515}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:53.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 -- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6254018930 msgr2=0x7f625401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:53.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6254018930 0x7f625401adc0 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:53.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 -- 172.21.15.32:0/2533238804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62741a2770 msgr2=0x7f62741a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:53.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62741a2770 0x7f62741a13f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7f6270009650 tx=0x7f627000b960).stop 2021-04-09T18:18:53.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 -- 172.21.15.32:0/2533238804 shutdown_connections 2021-04-09T18:18:53.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6254018930 0x7f625401adc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:53.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62741a3bb0 0x7f62741ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:53.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62741a2b70 0x7f62741a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:53.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 --2- 172.21.15.32:0/2533238804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62741a2770 0x7f62741a13f0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:53.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 -- 172.21.15.32:0/2533238804 >> 172.21.15.32:0/2533238804 conn(0x7f6274099f20 msgr2=0x7f6274094920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:53.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 -- 172.21.15.32:0/2533238804 shutdown_connections 2021-04-09T18:18:53.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:53.733 7f627b61b700 1 -- 172.21.15.32:0/2533238804 wait complete. 2021-04-09T18:18:54.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:51.408068', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 6526, 'mds_1': 6654}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_6654': {'gid': 6654, 'name': 'c', 'rank': 1, 'incarnation': 93, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6834/1291090515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 1291090515}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 1291090515}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:54.191 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6654:up:stopping 2021-04-09T18:18:54.887 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:18:55.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:55.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.305 7ff442f2b700 1 Processor -- start 2021-04-09T18:18:55.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.305 7ff442f2b700 1 -- start start 2021-04-09T18:18:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.305 7ff442f2b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff43c0cffa0 0x7ff43c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.305 7ff442f2b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0d6c80 0x7ff43c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.305 7ff442f2b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43c0d7280 con 0x7ff43c0d7800 2021-04-09T18:18:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.305 7ff442f2b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c0d7080 con 0x7ff43c0cffa0 2021-04-09T18:18:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.305 7ff442f2b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0d91e0 con 0x7ff43c0d6c80 2021-04-09T18:18:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff43bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff43c0d7800 0x7ff43c0d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56646/0 (socket says 172.21.15.32:56646) 2021-04-09T18:18:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff43bfff700 1 -- 172.21.15.32:0/1805704330 learned_addr learned my addr 172.21.15.32:0/1805704330 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff441548700 1 --2- 172.21.15.32:0/1805704330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0d6c80 0x7ff43c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:55.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff440d47700 1 --2- 172.21.15.32:0/1805704330 >> v2:172.21.15.86:3300/0 conn(0x7ff43c0cffa0 0x7ff43c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:55.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff43b7fe700 1 -- 172.21.15.32:0/1805704330 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538538464 0 0) 0x7ff42c001060 con 0x7ff43c0d7800 2021-04-09T18:18:55.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff43b7fe700 1 -- 172.21.15.32:0/1805704330 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff428002fe0 con 0x7ff43c0d7800 2021-04-09T18:18:55.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff440d47700 1 -- 172.21.15.32:0/1805704330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0d6c80 msgr2=0x7ff43c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff440d47700 1 --2- 172.21.15.32:0/1805704330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0d6c80 0x7ff43c10cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff440d47700 1 -- 172.21.15.32:0/1805704330 >> v1:172.21.15.32:6789/0 conn(0x7ff43c0d7800 legacy=0x7ff43c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff440d47700 1 -- 172.21.15.32:0/1805704330 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c0d83f0 con 0x7ff43c0cffa0 2021-04-09T18:18:55.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff441548700 1 --2- 172.21.15.32:0/1805704330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0d6c80 0x7ff43c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff440d47700 1 --2- 172.21.15.32:0/1805704330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c0cffa0 0x7ff43c0d2440 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7ff424006650 tx=0x7ff424009a80).ready entity=mon.1 client_cookie=3631b26b18ab0ed1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff43b7fe700 1 -- 172.21.15.32:0/1805704330 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4240009b0 con 0x7ff43c0cffa0 2021-04-09T18:18:55.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff43b7fe700 1 -- 172.21.15.32:0/1805704330 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42400c0c0 con 0x7ff43c0cffa0 2021-04-09T18:18:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff43b7fe700 1 -- 172.21.15.32:0/1805704330 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff42400c510 con 0x7ff43c0cffa0 2021-04-09T18:18:55.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff442f2b700 1 -- 172.21.15.32:0/1805704330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c0cffa0 msgr2=0x7ff43c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:55.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.306 7ff442f2b700 1 --2- 172.21.15.32:0/1805704330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c0cffa0 0x7ff43c0d2440 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7ff424006650 tx=0x7ff424009a80).stop 2021-04-09T18:18:55.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.307 7ff442f2b700 1 -- 172.21.15.32:0/1805704330 shutdown_connections 2021-04-09T18:18:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.307 7ff442f2b700 1 --2- 172.21.15.32:0/1805704330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0d6c80 0x7ff43c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.307 7ff442f2b700 1 --2- 172.21.15.32:0/1805704330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c0cffa0 0x7ff43c0d2440 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.307 7ff442f2b700 1 -- 172.21.15.32:0/1805704330 >> 172.21.15.32:0/1805704330 conn(0x7ff43c0ce5c0 msgr2=0x7ff43c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:55.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.307 7ff442f2b700 1 -- 172.21.15.32:0/1805704330 shutdown_connections 2021-04-09T18:18:55.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.307 7ff442f2b700 1 -- 172.21.15.32:0/1805704330 wait complete. 2021-04-09T18:18:55.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.327 7ff442f2b700 1 Processor -- start 2021-04-09T18:18:55.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.327 7ff442f2b700 1 -- start start 2021-04-09T18:18:55.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff442f2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0a80c0 0x7ff43c17bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:55.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff442f2b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c17b1d0 0x7ff43c17b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:55.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff442f2b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c08dc50 0x7ff43c08e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:55.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff442f2b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c17bc70 con 0x7ff43c0a80c0 2021-04-09T18:18:55.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff442f2b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c08e670 con 0x7ff43c17b1d0 2021-04-09T18:18:55.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff442f2b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c08e850 con 0x7ff43c08dc50 2021-04-09T18:18:55.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff440d47700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0a80c0 0x7ff43c17bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:55.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff440d47700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0a80c0 0x7ff43c17bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47066/0 (socket says 172.21.15.32:47066) 2021-04-09T18:18:55.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff440d47700 1 -- 172.21.15.32:0/3847862576 learned_addr learned my addr 172.21.15.32:0/3847862576 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:55.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff43bfff700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c17b1d0 0x7ff43c17b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:55.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff441548700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c08dc50 0x7ff43c08e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:55.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff440d47700 1 -- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c08dc50 msgr2=0x7ff43c08e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:55.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff440d47700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c08dc50 0x7ff43c08e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff440d47700 1 -- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c17b1d0 msgr2=0x7ff43c17b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.328 7ff440d47700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c17b1d0 0x7ff43c17b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.329 7ff440d47700 1 -- 172.21.15.32:0/3847862576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c08c660 con 0x7ff43c0a80c0 2021-04-09T18:18:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.329 7ff440d47700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0a80c0 0x7ff43c17bfe0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7ff424009650 tx=0x7ff42400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.329 7ff4397fa700 1 -- 172.21.15.32:0/3847862576 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4240093a0 con 0x7ff43c0a80c0 2021-04-09T18:18:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.329 7ff4397fa700 1 -- 172.21.15.32:0/3847862576 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42400bd50 con 0x7ff43c0a80c0 2021-04-09T18:18:55.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.329 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43c08c660 con 0x7ff43c0a80c0 2021-04-09T18:18:55.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.329 7ff4397fa700 1 -- 172.21.15.32:0/3847862576 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4240052e0 con 0x7ff43c0a80c0 2021-04-09T18:18:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.329 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c08c660 con 0x7ff43c0a80c0 2021-04-09T18:18:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.330 7ff4397fa700 1 -- 172.21.15.32:0/3847862576 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff424006030 con 0x7ff43c0a80c0 2021-04-09T18:18:55.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.330 7ff4397fa700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff420018930 0x7ff42001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:55.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.330 7ff4397fa700 1 -- 172.21.15.32:0/3847862576 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff4240329e0 con 0x7ff43c0a80c0 2021-04-09T18:18:55.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.330 7ff43bfff700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff420018930 0x7ff42001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.330 7ff43bfff700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff420018930 0x7ff42001ae10 crc :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.349 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43c1b70c0 con 0x7ff43c0a80c0 2021-04-09T18:18:55.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.353 7ff4397fa700 1 -- 172.21.15.32:0/3847862576 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff4240009c0 con 0x7ff43c0a80c0 2021-04-09T18:18:55.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.544 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff43c1b70c0 con 0x7ff43c0a80c0 2021-04-09T18:18:55.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.545 7ff4397fa700 1 -- 172.21.15.32:0/3847862576 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2445 (secure 0 0 0) 0x7ff42400bd50 con 0x7ff43c0a80c0 2021-04-09T18:18:55.584 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2021-04-09T18:18:55.585 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:55.585 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:55.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff420018930 msgr2=0x7ff42001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:55.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff420018930 0x7ff42001ae10 crc :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0a80c0 msgr2=0x7ff43c17bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:55.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0a80c0 0x7ff43c17bfe0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7ff424009650 tx=0x7ff42400b960).stop 2021-04-09T18:18:55.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 shutdown_connections 2021-04-09T18:18:55.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff420018930 0x7ff42001ae10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c08dc50 0x7ff43c08e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c17b1d0 0x7ff43c17b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 --2- 172.21.15.32:0/3847862576 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff43c0a80c0 0x7ff43c17bfe0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:55.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 >> 172.21.15.32:0/3847862576 conn(0x7ff43c189b00 msgr2=0x7ff43c1921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:55.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 shutdown_connections 2021-04-09T18:18:55.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:55.547 7ff442f2b700 1 -- 172.21.15.32:0/3847862576 wait complete. 2021-04-09T18:18:56.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:56.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:18:57.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:57.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.181 7fd56773e700 1 Processor -- start 2021-04-09T18:18:57.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.181 7fd56773e700 1 -- start start 2021-04-09T18:18:57.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.181 7fd56773e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5600cffa0 0x7fd5600d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:57.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.181 7fd56773e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5600d6c80 0x7fd56010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:57.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.181 7fd56773e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600d7280 con 0x7fd5600d7800 2021-04-09T18:18:57.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.181 7fd56773e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5600d7080 con 0x7fd5600cffa0 2021-04-09T18:18:57.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.181 7fd56773e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5600d91e0 con 0x7fd5600d6c80 2021-04-09T18:18:57.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd565d5b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5600d6c80 0x7fd56010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:57.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd56555a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5600cffa0 0x7fd5600d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:57.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd565d5b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5600d6c80 0x7fd56010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:54298/0 (socket says 172.21.15.32:54298) 2021-04-09T18:18:57.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd56555a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5600cffa0 0x7fd5600d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:38828/0 (socket says 172.21.15.32:38828) 2021-04-09T18:18:57.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd565d5b700 1 -- 172.21.15.32:0/3811542499 learned_addr learned my addr 172.21.15.32:0/3811542499 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:57.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd557fff700 1 -- 172.21.15.32:0/3811542499 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024190510 0 0) 0x7fd550000ff0 con 0x7fd5600d7800 2021-04-09T18:18:57.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd557fff700 1 -- 172.21.15.32:0/3811542499 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd54c002fe0 con 0x7fd5600d7800 2021-04-09T18:18:57.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd56555a700 1 -- 172.21.15.32:0/3811542499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5600d6c80 msgr2=0x7fd56010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:57.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd56555a700 1 --2- 172.21.15.32:0/3811542499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5600d6c80 0x7fd56010cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd56555a700 1 -- 172.21.15.32:0/3811542499 >> v1:172.21.15.86:6789/0 conn(0x7fd5600d7800 legacy=0x7fd5600d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:57.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd56555a700 1 -- 172.21.15.32:0/3811542499 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5600d83f0 con 0x7fd5600cffa0 2021-04-09T18:18:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd565d5b700 1 --2- 172.21.15.32:0/3811542499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5600d6c80 0x7fd56010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.182 7fd56555a700 1 --2- 172.21.15.32:0/3811542499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5600cffa0 0x7fd5600d2440 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fd55c006650 tx=0x7fd55c009a80).ready entity=mon.1 client_cookie=9e754070cde0e1bf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.183 7fd557fff700 1 -- 172.21.15.32:0/3811542499 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd55c0009b0 con 0x7fd5600cffa0 2021-04-09T18:18:57.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.183 7fd557fff700 1 -- 172.21.15.32:0/3811542499 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c00c0c0 con 0x7fd5600cffa0 2021-04-09T18:18:57.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.183 7fd557fff700 1 -- 172.21.15.32:0/3811542499 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd55c00c510 con 0x7fd5600cffa0 2021-04-09T18:18:57.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.183 7fd56773e700 1 -- 172.21.15.32:0/3811542499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5600cffa0 msgr2=0x7fd5600d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.183 7fd56773e700 1 --2- 172.21.15.32:0/3811542499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5600cffa0 0x7fd5600d2440 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fd55c006650 tx=0x7fd55c009a80).stop 2021-04-09T18:18:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.183 7fd56773e700 1 -- 172.21.15.32:0/3811542499 shutdown_connections 2021-04-09T18:18:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.183 7fd56773e700 1 --2- 172.21.15.32:0/3811542499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5600d6c80 0x7fd56010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.183 7fd56773e700 1 --2- 172.21.15.32:0/3811542499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5600cffa0 0x7fd5600d2440 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.183 7fd56773e700 1 -- 172.21.15.32:0/3811542499 >> 172.21.15.32:0/3811542499 conn(0x7fd5600ce5c0 msgr2=0x7fd5600cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.184 7fd56773e700 1 -- 172.21.15.32:0/3811542499 shutdown_connections 2021-04-09T18:18:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.184 7fd56773e700 1 -- 172.21.15.32:0/3811542499 wait complete. 2021-04-09T18:18:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.204 7fd56773e700 1 Processor -- start 2021-04-09T18:18:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.204 7fd56773e700 1 -- start start 2021-04-09T18:18:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.204 7fd56773e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5601acff0 0x7fd56008fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.204 7fd56773e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd560090150 0x7fd56008c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.204 7fd56773e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56008cca0 0x7fd56008e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:57.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd56773e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56008c100 con 0x7fd56008cca0 2021-04-09T18:18:57.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd56773e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56008d0a0 con 0x7fd560090150 2021-04-09T18:18:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd56773e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd560183cd0 con 0x7fd5601acff0 2021-04-09T18:18:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd565d5b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56008cca0 0x7fd56008e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:57.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd565d5b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56008cca0 0x7fd56008e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47076/0 (socket says 172.21.15.32:47076) 2021-04-09T18:18:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd565d5b700 1 -- 172.21.15.32:0/1991301379 learned_addr learned my addr 172.21.15.32:0/1991301379 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd56555a700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5601acff0 0x7fd56008fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:57.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd564d59700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd560090150 0x7fd56008c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:57.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd565d5b700 1 -- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5601acff0 msgr2=0x7fd56008fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:57.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd565d5b700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5601acff0 0x7fd56008fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd565d5b700 1 -- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd560090150 msgr2=0x7fd56008c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:57.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd565d5b700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd560090150 0x7fd56008c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd565d5b700 1 -- 172.21.15.32:0/1991301379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd560183ef0 con 0x7fd56008cca0 2021-04-09T18:18:57.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd564d59700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd560090150 0x7fd56008c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:18:57.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd565d5b700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56008cca0 0x7fd56008e340 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7fd55800b8e0 tx=0x7fd55800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:57.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd555ffb700 1 -- 172.21.15.32:0/1991301379 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd558000f90 con 0x7fd56008cca0 2021-04-09T18:18:57.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd555ffb700 1 -- 172.21.15.32:0/1991301379 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd558006790 con 0x7fd56008cca0 2021-04-09T18:18:57.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd56773e700 1 -- 172.21.15.32:0/1991301379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd560183ef0 con 0x7fd56008cca0 2021-04-09T18:18:57.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd555ffb700 1 -- 172.21.15.32:0/1991301379 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd55800c4e0 con 0x7fd56008cca0 2021-04-09T18:18:57.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.205 7fd56773e700 1 -- 172.21.15.32:0/1991301379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd560183ef0 con 0x7fd56008cca0 2021-04-09T18:18:57.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.206 7fd555ffb700 1 -- 172.21.15.32:0/1991301379 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd5580011f0 con 0x7fd56008cca0 2021-04-09T18:18:57.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.206 7fd555ffb700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd540018930 0x7fd54001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:57.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.206 7fd555ffb700 1 -- 172.21.15.32:0/1991301379 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd558032dd0 con 0x7fd56008cca0 2021-04-09T18:18:57.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.206 7fd56555a700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd540018930 0x7fd54001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.206 7fd56555a700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd540018930 0x7fd54001ae10 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.225 7fd56773e700 1 -- 172.21.15.32:0/1991301379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd560182dc0 con 0x7fd56008cca0 2021-04-09T18:18:57.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.229 7fd555ffb700 1 -- 172.21.15.32:0/1991301379 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd55800bdc0 con 0x7fd56008cca0 2021-04-09T18:18:57.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.421 7fd56773e700 1 -- 172.21.15.32:0/1991301379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd56008c100 con 0x7fd56008cca0 2021-04-09T18:18:57.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.421 7fd555ffb700 1 -- 172.21.15.32:0/1991301379 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2445 (secure 0 0 0) 0x7fd558006770 con 0x7fd56008cca0 2021-04-09T18:18:57.461 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2021-04-09T18:18:57.461 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:57.461 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.423 7fd56773e700 1 -- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd540018930 msgr2=0x7fd54001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.423 7fd56773e700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd540018930 0x7fd54001ae10 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.423 7fd56773e700 1 -- 172.21.15.32:0/1991301379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56008cca0 msgr2=0x7fd56008e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.423 7fd56773e700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56008cca0 0x7fd56008e340 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7fd55800b8e0 tx=0x7fd55800bc80).stop 2021-04-09T18:18:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.424 7fd56773e700 1 -- 172.21.15.32:0/1991301379 shutdown_connections 2021-04-09T18:18:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.424 7fd56773e700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd540018930 0x7fd54001ae10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.424 7fd56773e700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56008cca0 0x7fd56008e340 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.424 7fd56773e700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd560090150 0x7fd56008c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.424 7fd56773e700 1 --2- 172.21.15.32:0/1991301379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5601acff0 0x7fd56008fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:57.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.424 7fd56773e700 1 -- 172.21.15.32:0/1991301379 >> 172.21.15.32:0/1991301379 conn(0x7fd56009abb0 msgr2=0x7fd5600a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:57.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.424 7fd56773e700 1 -- 172.21.15.32:0/1991301379 shutdown_connections 2021-04-09T18:18:57.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:57.424 7fd56773e700 1 -- 172.21.15.32:0/1991301379 wait complete. 2021-04-09T18:18:57.876 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:57.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:18:58.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:18:59.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.995 7fc7b1417700 1 Processor -- start 2021-04-09T18:18:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.995 7fc7b1417700 1 -- start start 2021-04-09T18:18:59.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.995 7fc7b1417700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc7ac10d010 0x7fc7ac0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:59.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.995 7fc7b1417700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0d8490 0x7fc7ac10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.995 7fc7b1417700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac10d480 con 0x7fc7ac0d1da0 2021-04-09T18:18:59.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.995 7fc7b1417700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac0d8d30 con 0x7fc7ac10d010 2021-04-09T18:18:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.995 7fc7b1417700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0d9220 con 0x7fc7ac0d8490 2021-04-09T18:18:59.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7ab07d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc7ac10d010 0x7fc7ac0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:59.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7ab87e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0d8490 0x7fc7ac10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:59.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7aa87c700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc7ac0d1da0 0x7fc7ac0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54604/0 (socket says 172.21.15.32:54604) 2021-04-09T18:18:59.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7aa87c700 1 -- 172.21.15.32:0/4190302847 learned_addr learned my addr 172.21.15.32:0/4190302847 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:18:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7aa07b700 1 -- 172.21.15.32:0/4190302847 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409078505 0 0) 0x7fc7980010c0 con 0x7fc7ac0d1da0 2021-04-09T18:18:59.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7aa07b700 1 -- 172.21.15.32:0/4190302847 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc790002fe0 con 0x7fc7ac0d1da0 2021-04-09T18:18:59.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7ab87e700 1 -- 172.21.15.32:0/4190302847 >> v2:172.21.15.86:3300/0 conn(0x7fc7ac10d010 msgr2=0x7fc7ac0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:18:59.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7ab87e700 1 --2- 172.21.15.32:0/4190302847 >> v2:172.21.15.86:3300/0 conn(0x7fc7ac10d010 0x7fc7ac0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7ab87e700 1 -- 172.21.15.32:0/4190302847 >> v1:172.21.15.86:6789/0 conn(0x7fc7ac0d1da0 legacy=0x7fc7ac0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:59.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7ab87e700 1 -- 172.21.15.32:0/4190302847 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac10e020 con 0x7fc7ac0d8490 2021-04-09T18:18:59.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7ab07d700 1 --2- 172.21.15.32:0/4190302847 >> v2:172.21.15.86:3300/0 conn(0x7fc7ac10d010 0x7fc7ac0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:59.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7ab87e700 1 --2- 172.21.15.32:0/4190302847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0d8490 0x7fc7ac10da00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc7a4006e50 tx=0x7fc7a4009e70).ready entity=mon.2 client_cookie=90456af5b9ceeafe server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:59.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7aa07b700 1 -- 172.21.15.32:0/4190302847 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7a4006a20 con 0x7fc7ac0d8490 2021-04-09T18:18:59.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7aa07b700 1 -- 172.21.15.32:0/4190302847 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a4006790 con 0x7fc7ac0d8490 2021-04-09T18:18:59.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7aa07b700 1 -- 172.21.15.32:0/4190302847 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7a400b700 con 0x7fc7ac0d8490 2021-04-09T18:18:59.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7b1417700 1 -- 172.21.15.32:0/4190302847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0d8490 msgr2=0x7fc7ac10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:59.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.996 7fc7b1417700 1 --2- 172.21.15.32:0/4190302847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0d8490 0x7fc7ac10da00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc7a4006e50 tx=0x7fc7a4009e70).stop 2021-04-09T18:18:59.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.997 7fc7b1417700 1 -- 172.21.15.32:0/4190302847 shutdown_connections 2021-04-09T18:18:59.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.997 7fc7b1417700 1 --2- 172.21.15.32:0/4190302847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0d8490 0x7fc7ac10da00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.997 7fc7b1417700 1 --2- 172.21.15.32:0/4190302847 >> v2:172.21.15.86:3300/0 conn(0x7fc7ac10d010 0x7fc7ac0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.997 7fc7b1417700 1 -- 172.21.15.32:0/4190302847 >> 172.21.15.32:0/4190302847 conn(0x7fc7ac0ce5f0 msgr2=0x7fc7ac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:59.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.997 7fc7b1417700 1 -- 172.21.15.32:0/4190302847 shutdown_connections 2021-04-09T18:18:59.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:58.997 7fc7b1417700 1 -- 172.21.15.32:0/4190302847 wait complete. 2021-04-09T18:18:59.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.017 7fc7b1417700 1 Processor -- start 2021-04-09T18:18:59.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.017 7fc7b1417700 1 -- start start 2021-04-09T18:18:59.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7b1417700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7ac1aca20 0x7fc7ac1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:59.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7b1417700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac18bd40 0x7fc7ac18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:59.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7b1417700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac08fba0 0x7fc7ac08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:59.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7b1417700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac18c7e0 con 0x7fc7ac1aca20 2021-04-09T18:18:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7b1417700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac18ca30 con 0x7fc7ac08fba0 2021-04-09T18:18:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7b1417700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0905f0 con 0x7fc7ac18bd40 2021-04-09T18:18:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7ac1aca20 0x7fc7ac1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7ac1aca20 0x7fc7ac1ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47090/0 (socket says 172.21.15.32:47090) 2021-04-09T18:18:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab07d700 1 -- 172.21.15.32:0/1053429801 learned_addr learned my addr 172.21.15.32:0/1053429801 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:18:59.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7aa87c700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac18bd40 0x7fc7ac18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:59.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab87e700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac08fba0 0x7fc7ac08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:59.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab07d700 1 -- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac18bd40 msgr2=0x7fc7ac18c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:59.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab07d700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac18bd40 0x7fc7ac18c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab07d700 1 -- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac08fba0 msgr2=0x7fc7ac08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:59.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab07d700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac08fba0 0x7fc7ac08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab07d700 1 -- 172.21.15.32:0/1053429801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0b4300 con 0x7fc7ac1aca20 2021-04-09T18:18:59.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.018 7fc7ab87e700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac08fba0 0x7fc7ac08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:18:59.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.019 7fc7ab07d700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7ac1aca20 0x7fc7ac1ace20 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7fc79c006f40 tx=0x7fc79c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:59.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.019 7fc7a3fff700 1 -- 172.21.15.32:0/1053429801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc79c00bd80 con 0x7fc7ac1aca20 2021-04-09T18:18:59.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.019 7fc7a3fff700 1 -- 172.21.15.32:0/1053429801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c000c90 con 0x7fc7ac1aca20 2021-04-09T18:18:59.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.019 7fc7a3fff700 1 -- 172.21.15.32:0/1053429801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc79c002220 con 0x7fc7ac1aca20 2021-04-09T18:18:59.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.019 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac0b4300 con 0x7fc7ac1aca20 2021-04-09T18:18:59.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.019 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac0b4300 con 0x7fc7ac1aca20 2021-04-09T18:18:59.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.020 7fc7a3fff700 1 -- 172.21.15.32:0/1053429801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc79c00a000 con 0x7fc7ac1aca20 2021-04-09T18:18:59.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.020 7fc7a3fff700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc7880188c0 0x7fc78801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:18:59.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.020 7fc7a3fff700 1 -- 172.21.15.32:0/1053429801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc79c032a40 con 0x7fc7ac1aca20 2021-04-09T18:18:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.020 7fc7aa87c700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc7880188c0 0x7fc78801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:18:59.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.020 7fc7aa87c700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc7880188c0 0x7fc78801adc0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:18:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.040 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac182d70 con 0x7fc7ac1aca20 2021-04-09T18:18:59.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.043 7fc7a3fff700 1 -- 172.21.15.32:0/1053429801 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc79c006b00 con 0x7fc7ac1aca20 2021-04-09T18:18:59.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.233 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7ac18c760 con 0x7fc7ac1aca20 2021-04-09T18:18:59.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.233 7fc7a3fff700 1 -- 172.21.15.32:0/1053429801 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2445 (secure 0 0 0) 0x7fc79c00a000 con 0x7fc7ac1aca20 2021-04-09T18:18:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 100 2021-04-09T18:18:59.275 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:18:59.275 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":100,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:18:59.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc7880188c0 msgr2=0x7fc78801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:59.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc7880188c0 0x7fc78801adc0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7ac1aca20 msgr2=0x7fc7ac1ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:18:59.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7ac1aca20 0x7fc7ac1ace20 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7fc79c006f40 tx=0x7fc79c00b990).stop 2021-04-09T18:18:59.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 shutdown_connections 2021-04-09T18:18:59.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc7880188c0 0x7fc78801adc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac08fba0 0x7fc7ac08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac18bd40 0x7fc7ac18c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 --2- 172.21.15.32:0/1053429801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7ac1aca20 0x7fc7ac1ace20 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:18:59.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 >> 172.21.15.32:0/1053429801 conn(0x7fc7ac19b460 msgr2=0x7fc7ac19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:18:59.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 shutdown_connections 2021-04-09T18:18:59.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:18:59.236 7fc7b1417700 1 -- 172.21.15.32:0/1053429801 wait complete. 2021-04-09T18:18:59.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:18:59.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:00.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:00.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.831 7f75dc1fe700 1 Processor -- start 2021-04-09T18:19:00.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.831 7f75dc1fe700 1 -- start start 2021-04-09T18:19:00.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.832 7f75dc1fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75d410d010 0x7f75d40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:00.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.832 7f75dc1fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f75d40d8490 0x7f75d410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:00.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.832 7f75dc1fe700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d410d480 con 0x7f75d40d1da0 2021-04-09T18:19:00.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.832 7f75dc1fe700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d40d8d30 con 0x7f75d410d010 2021-04-09T18:19:00.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.832 7f75dc1fe700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d40d9220 con 0x7f75d40d8490 2021-04-09T18:19:00.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.832 7f75da01a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75d410d010 0x7f75d40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:00.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.832 7f75da01a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75d410d010 0x7f75d40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47100/0 (socket says 172.21.15.32:47100) 2021-04-09T18:19:00.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.832 7f75da01a700 1 -- 172.21.15.32:0/3649649240 learned_addr learned my addr 172.21.15.32:0/3649649240 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75da81b700 1 --2- 172.21.15.32:0/3649649240 >> v2:172.21.15.86:3300/0 conn(0x7f75d40d8490 0x7f75d410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75da01a700 1 -- 172.21.15.32:0/3649649240 >> v2:172.21.15.86:3300/0 conn(0x7f75d40d8490 msgr2=0x7f75d410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75da01a700 1 --2- 172.21.15.32:0/3649649240 >> v2:172.21.15.86:3300/0 conn(0x7f75d40d8490 0x7f75d410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75da01a700 1 -- 172.21.15.32:0/3649649240 >> v1:172.21.15.86:6789/0 conn(0x7f75d40d1da0 legacy=0x7f75d40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:00.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75da01a700 1 -- 172.21.15.32:0/3649649240 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d410e020 con 0x7f75d410d010 2021-04-09T18:19:00.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75da81b700 1 --2- 172.21.15.32:0/3649649240 >> v2:172.21.15.86:3300/0 conn(0x7f75d40d8490 0x7f75d410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:00.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75da01a700 1 --2- 172.21.15.32:0/3649649240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d410d010 0x7f75d40d1780 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7f75d0006700 tx=0x7f75d0009b80).ready entity=mon.0 client_cookie=79fb8bd9fb48c062 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:00.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75d9018700 1 -- 172.21.15.32:0/3649649240 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75d000c0c0 con 0x7f75d410d010 2021-04-09T18:19:00.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75d9018700 1 -- 172.21.15.32:0/3649649240 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d000c440 con 0x7f75d410d010 2021-04-09T18:19:00.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75d9018700 1 -- 172.21.15.32:0/3649649240 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75d000c920 con 0x7f75d410d010 2021-04-09T18:19:00.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75dc1fe700 1 -- 172.21.15.32:0/3649649240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d410d010 msgr2=0x7f75d40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:00.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.833 7f75dc1fe700 1 --2- 172.21.15.32:0/3649649240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d410d010 0x7f75d40d1780 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7f75d0006700 tx=0x7f75d0009b80).stop 2021-04-09T18:19:00.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.834 7f75dc1fe700 1 -- 172.21.15.32:0/3649649240 shutdown_connections 2021-04-09T18:19:00.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.834 7f75dc1fe700 1 --2- 172.21.15.32:0/3649649240 >> v2:172.21.15.86:3300/0 conn(0x7f75d40d8490 0x7f75d410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:00.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.834 7f75dc1fe700 1 --2- 172.21.15.32:0/3649649240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d410d010 0x7f75d40d1780 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.834 7f75dc1fe700 1 -- 172.21.15.32:0/3649649240 >> 172.21.15.32:0/3649649240 conn(0x7f75d40ce5f0 msgr2=0x7f75d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.834 7f75dc1fe700 1 -- 172.21.15.32:0/3649649240 shutdown_connections 2021-04-09T18:19:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.834 7f75dc1fe700 1 -- 172.21.15.32:0/3649649240 wait complete. 2021-04-09T18:19:00.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.854 7f75dc1fe700 1 Processor -- start 2021-04-09T18:19:00.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.854 7f75dc1fe700 1 -- start start 2021-04-09T18:19:00.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75dc1fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d4179fd0 0x7f75d40976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:00.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75dc1fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d4097cc0 0x7f75d41a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75dc1fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d41a4b10 0x7f75d41a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75dc1fe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d4098140 con 0x7f75d4097cc0 2021-04-09T18:19:00.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75dc1fe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d417a3d0 con 0x7f75d4179fd0 2021-04-09T18:19:00.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75dc1fe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d417c870 con 0x7f75d41a4b10 2021-04-09T18:19:00.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75d9819700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d4097cc0 0x7f75d41a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:00.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75d9819700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d4097cc0 0x7f75d41a44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47108/0 (socket says 172.21.15.32:47108) 2021-04-09T18:19:00.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75da01a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d4179fd0 0x7f75d40976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:00.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75d9819700 1 -- 172.21.15.32:0/272117114 learned_addr learned my addr 172.21.15.32:0/272117114 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:00.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75da81b700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d41a4b10 0x7f75d41a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:00.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75d9819700 1 -- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d41a4b10 msgr2=0x7f75d41a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:00.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75d9819700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d41a4b10 0x7f75d41a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:00.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75d9819700 1 -- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d4179fd0 msgr2=0x7f75d40976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:00.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75d9819700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d4179fd0 0x7f75d40976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:00.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75d9819700 1 -- 172.21.15.32:0/272117114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d4179bf0 con 0x7f75d4097cc0 2021-04-09T18:19:00.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75d9819700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d4097cc0 0x7f75d41a44f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f75c800b530 tx=0x7f75c800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:00.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75c6ffd700 1 -- 172.21.15.32:0/272117114 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75c800bcc0 con 0x7f75d4097cc0 2021-04-09T18:19:00.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75c6ffd700 1 -- 172.21.15.32:0/272117114 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c80009b0 con 0x7f75d4097cc0 2021-04-09T18:19:00.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d4179bf0 con 0x7f75d4097cc0 2021-04-09T18:19:00.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75c6ffd700 1 -- 172.21.15.32:0/272117114 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75c80062a0 con 0x7f75d4097cc0 2021-04-09T18:19:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.855 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d4179bf0 con 0x7f75d4097cc0 2021-04-09T18:19:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.856 7f75c6ffd700 1 -- 172.21.15.32:0/272117114 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f75c800a000 con 0x7f75d4097cc0 2021-04-09T18:19:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.856 7f75c6ffd700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75bc018930 0x7f75bc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.856 7f75c6ffd700 1 -- 172.21.15.32:0/272117114 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f75c80329f0 con 0x7f75d4097cc0 2021-04-09T18:19:00.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.856 7f75da01a700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75bc018930 0x7f75bc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:00.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.857 7f75da01a700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75bc018930 0x7f75bc01adc0 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:00.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.876 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d41b7050 con 0x7f75d4097cc0 2021-04-09T18:19:00.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:00.880 7f75c6ffd700 1 -- 172.21.15.32:0/272117114 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f75c800bcc0 con 0x7f75d4097cc0 2021-04-09T18:19:01.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.070 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75d410d480 con 0x7f75d4097cc0 2021-04-09T18:19:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.070 7f75c6ffd700 1 -- 172.21.15.32:0/272117114 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f75c801f730 con 0x7f75d4097cc0 2021-04-09T18:19:01.113 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:01.114 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:01.114 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:01.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75bc018930 msgr2=0x7f75bc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75bc018930 0x7f75bc01adc0 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:01.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d4097cc0 msgr2=0x7f75d41a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:01.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d4097cc0 0x7f75d41a44f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f75c800b530 tx=0x7f75c800b8d0).stop 2021-04-09T18:19:01.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 shutdown_connections 2021-04-09T18:19:01.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75bc018930 0x7f75bc01adc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:01.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d4097cc0 0x7f75d41a44f0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:01.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d41a4b10 0x7f75d41a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:01.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 --2- 172.21.15.32:0/272117114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d4179fd0 0x7f75d40976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:01.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 >> 172.21.15.32:0/272117114 conn(0x7f75d4199680 msgr2=0x7f75d41a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:01.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 shutdown_connections 2021-04-09T18:19:01.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:01.073 7f75dc1fe700 1 -- 172.21.15.32:0/272117114 wait complete. 2021-04-09T18:19:01.527 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:01.527 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:02.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:02.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.654 7ff016a69700 1 Processor -- start 2021-04-09T18:19:02.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.654 7ff016a69700 1 -- start start 2021-04-09T18:19:02.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.654 7ff016a69700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff01010d010 0x7ff0100d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:02.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.654 7ff016a69700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01010d480 con 0x7ff0100d1da0 2021-04-09T18:19:02.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.654 7ff016a69700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01010e1f0 con 0x7ff0100d8490 2021-04-09T18:19:02.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.654 7ff016a69700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0100d8d30 con 0x7ff01010d010 2021-04-09T18:19:02.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0100d1da0 0x7ff0100d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56708/0 (socket says 172.21.15.32:56708) 2021-04-09T18:19:02.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00ffff700 1 -- 172.21.15.32:0/1249557603 learned_addr learned my addr 172.21.15.32:0/1249557603 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:02.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff014885700 1 --2- 172.21.15.32:0/1249557603 >> v2:172.21.15.86:3300/0 conn(0x7ff01010d010 0x7ff0100d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:02.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775641156 0 0) 0x7ff0040009b0 con 0x7ff0100d1da0 2021-04-09T18:19:02.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff8002fe0 con 0x7ff0100d1da0 2021-04-09T18:19:02.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2919308982 0 0) 0x7ff004001010 con 0x7ff0100d1da0 2021-04-09T18:19:02.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 >> v2:172.21.15.86:3300/0 conn(0x7ff01010d010 msgr2=0x7ff0100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:02.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 --2- 172.21.15.32:0/1249557603 >> v2:172.21.15.86:3300/0 conn(0x7ff01010d010 0x7ff0100d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 >> v1:172.21.15.86:6789/0 conn(0x7ff0100d8490 legacy=0x7ff01010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:02.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0100d9220 con 0x7ff0100d1da0 2021-04-09T18:19:02.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7ff004001530 con 0x7ff0100d1da0 2021-04-09T18:19:02.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 >> v1:172.21.15.32:6789/0 conn(0x7ff0100d1da0 legacy=0x7ff0100d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:02.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feff8003a70 0x7feff8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:02.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feff80067a0 0x7feff8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:02.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feff80095a0 0x7feff800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:02.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff800c310 con 0x7feff80067a0 2021-04-09T18:19:02.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff800c530 con 0x7feff80095a0 2021-04-09T18:19:02.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff8050760 con 0x7feff8003a70 2021-04-09T18:19:02.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff00ffff700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feff80067a0 0x7feff8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:02.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff015086700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feff80095a0 0x7feff800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:02.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.655 7ff014885700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feff8003a70 0x7feff8005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:02.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff00ffff700 1 -- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feff8003a70 msgr2=0x7feff8005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff00ffff700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feff8003a70 0x7feff8005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff00ffff700 1 -- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feff80095a0 msgr2=0x7feff800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff00ffff700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feff80095a0 0x7feff800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff00ffff700 1 -- 172.21.15.32:0/1249557603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff80509c0 con 0x7feff80067a0 2021-04-09T18:19:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff014885700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feff8003a70 0x7feff8005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff015086700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feff80095a0 0x7feff800bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff015086700 1 -- 172.21.15.32:0/1249557603 reap_dead start 2021-04-09T18:19:02.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff00ffff700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feff80067a0 0x7feff8008ca0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7ff0040017d0 tx=0x7ff004007d30).ready entity=mon.0 client_cookie=fb24a8ccdafd8601 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0040070c0 con 0x7feff80067a0 2021-04-09T18:19:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff004001010 con 0x7feff80067a0 2021-04-09T18:19:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff00f7fe700 1 -- 172.21.15.32:0/1249557603 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff00400a5c0 con 0x7feff80067a0 2021-04-09T18:19:02.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff016a69700 1 -- 172.21.15.32:0/1249557603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feff80067a0 msgr2=0x7feff8008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:02.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.656 7ff016a69700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feff80067a0 0x7feff8008ca0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7ff0040017d0 tx=0x7ff004007d30).stop 2021-04-09T18:19:02.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.657 7ff016a69700 1 -- 172.21.15.32:0/1249557603 shutdown_connections 2021-04-09T18:19:02.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.657 7ff016a69700 1 --2- 172.21.15.32:0/1249557603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feff80067a0 0x7feff8008ca0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.657 7ff016a69700 1 -- 172.21.15.32:0/1249557603 >> 172.21.15.32:0/1249557603 conn(0x7ff0100ce5f0 msgr2=0x7ff0100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:02.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.657 7ff016a69700 1 -- 172.21.15.32:0/1249557603 shutdown_connections 2021-04-09T18:19:02.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.657 7ff016a69700 1 -- 172.21.15.32:0/1249557603 wait complete. 2021-04-09T18:19:02.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.677 7ff016a69700 1 Processor -- start 2021-04-09T18:19:02.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.677 7ff016a69700 1 -- start start 2021-04-09T18:19:02.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff016a69700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0101a91e0 0x7ff0100a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:02.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff016a69700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0101a95e0 0x7ff0100a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:02.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff016a69700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0101ac850 0x7ff0101acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:02.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff016a69700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0101ad270 con 0x7ff0101a95e0 2021-04-09T18:19:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff016a69700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0100b7640 con 0x7ff0101a91e0 2021-04-09T18:19:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff016a69700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff010183c80 con 0x7ff0101ac850 2021-04-09T18:19:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0101a95e0 0x7ff0100a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:02.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0101a95e0 0x7ff0100a5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47128/0 (socket says 172.21.15.32:47128) 2021-04-09T18:19:02.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00ffff700 1 -- 172.21.15.32:0/3661154191 learned_addr learned my addr 172.21.15.32:0/3661154191 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:02.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff015086700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0101ac850 0x7ff0101acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:02.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff014885700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0101a91e0 0x7ff0100a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:02.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00ffff700 1 -- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0101ac850 msgr2=0x7ff0101acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:02.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00ffff700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0101ac850 0x7ff0101acc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00ffff700 1 -- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0101a91e0 msgr2=0x7ff0100a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:02.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00ffff700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0101a91e0 0x7ff0100a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00ffff700 1 -- 172.21.15.32:0/3661154191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff010183ea0 con 0x7ff0101a95e0 2021-04-09T18:19:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff014885700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0101a91e0 0x7ff0100a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00ffff700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0101a95e0 0x7ff0100a5080 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7ff004001c60 tx=0x7ff00400a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:02.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00d7fa700 1 -- 172.21.15.32:0/3661154191 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0040072a0 con 0x7ff0101a95e0 2021-04-09T18:19:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00d7fa700 1 -- 172.21.15.32:0/3661154191 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff004000a00 con 0x7ff0101a95e0 2021-04-09T18:19:02.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff016a69700 1 -- 172.21.15.32:0/3661154191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff010183ea0 con 0x7ff0101a95e0 2021-04-09T18:19:02.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff00d7fa700 1 -- 172.21.15.32:0/3661154191 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff004004350 con 0x7ff0101a95e0 2021-04-09T18:19:02.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.678 7ff016a69700 1 -- 172.21.15.32:0/3661154191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff010183ea0 con 0x7ff0101a95e0 2021-04-09T18:19:02.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.679 7ff00d7fa700 1 -- 172.21.15.32:0/3661154191 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff004004140 con 0x7ff0101a95e0 2021-04-09T18:19:02.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.679 7ff00d7fa700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feff0018930 0x7feff001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:02.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.679 7ff00d7fa700 1 -- 172.21.15.32:0/3661154191 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff0040329d0 con 0x7ff0101a95e0 2021-04-09T18:19:02.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.679 7ff014885700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feff0018930 0x7feff001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:02.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.680 7ff014885700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feff0018930 0x7feff001ae10 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:02.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.698 7ff016a69700 1 -- 172.21.15.32:0/3661154191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff010182d70 con 0x7ff0101a95e0 2021-04-09T18:19:02.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.702 7ff00d7fa700 1 -- 172.21.15.32:0/3661154191 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff004004140 con 0x7ff0101a95e0 2021-04-09T18:19:02.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.893 7ff016a69700 1 -- 172.21.15.32:0/3661154191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff01010d480 con 0x7ff0101a95e0 2021-04-09T18:19:02.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.894 7ff00d7fa700 1 -- 172.21.15.32:0/3661154191 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7ff00400ac70 con 0x7ff0101a95e0 2021-04-09T18:19:02.934 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:02.934 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:02.935 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:02.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 -- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feff0018930 msgr2=0x7feff001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:02.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feff0018930 0x7feff001ae10 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 -- 172.21.15.32:0/3661154191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0101a95e0 msgr2=0x7ff0100a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:02.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0101a95e0 0x7ff0100a5080 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7ff004001c60 tx=0x7ff00400a880).stop 2021-04-09T18:19:02.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 -- 172.21.15.32:0/3661154191 shutdown_connections 2021-04-09T18:19:02.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feff0018930 0x7feff001ae10 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0101a95e0 0x7ff0100a5080 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0101ac850 0x7ff0101acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 --2- 172.21.15.32:0/3661154191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0101a91e0 0x7ff0100a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.896 7ff016a69700 1 -- 172.21.15.32:0/3661154191 >> 172.21.15.32:0/3661154191 conn(0x7ff01019b460 msgr2=0x7ff01019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.897 7ff016a69700 1 -- 172.21.15.32:0/3661154191 shutdown_connections 2021-04-09T18:19:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:02.897 7ff016a69700 1 -- 172.21.15.32:0/3661154191 wait complete. 2021-04-09T18:19:03.353 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:03.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:04.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:04.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.469 7f3e42a9e700 1 Processor -- start 2021-04-09T18:19:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.469 7f3e42a9e700 1 -- start start 2021-04-09T18:19:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.469 7f3e42a9e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3e3c0cffa0 0x7f3e3c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:04.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.469 7f3e42a9e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3e3c0d7800 0x7f3e3c0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.469 7f3e42a9e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0d8220 0x7f3e3c0d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.469 7f3e42a9e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3c0d91e0 con 0x7f3e3c0cffa0 2021-04-09T18:19:04.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.469 7f3e42a9e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3c0d7280 con 0x7f3e3c0d7800 2021-04-09T18:19:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.469 7f3e42a9e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c0d6290 con 0x7f3e3c0d8220 2021-04-09T18:19:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e408ba700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3e3c0cffa0 0x7f3e3c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e408ba700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3e3c0cffa0 0x7f3e3c0d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47134/0 (socket says 172.21.15.32:47134) 2021-04-09T18:19:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e408ba700 1 -- 172.21.15.32:0/2195639004 learned_addr learned my addr 172.21.15.32:0/2195639004 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:04.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e410bb700 1 --2- 172.21.15.32:0/2195639004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0d8220 0x7f3e3c0d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:04.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e3bfff700 1 --2- 172.21.15.32:0/2195639004 >> v2:172.21.15.86:3300/0 conn(0x7f3e3c0d7800 0x7f3e3c0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e408ba700 1 -- 172.21.15.32:0/2195639004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0d8220 msgr2=0x7f3e3c0d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e408ba700 1 --2- 172.21.15.32:0/2195639004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0d8220 0x7f3e3c0d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e408ba700 1 -- 172.21.15.32:0/2195639004 >> v2:172.21.15.86:3300/0 conn(0x7f3e3c0d7800 msgr2=0x7f3e3c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e408ba700 1 --2- 172.21.15.32:0/2195639004 >> v2:172.21.15.86:3300/0 conn(0x7f3e3c0d7800 0x7f3e3c0d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e408ba700 1 -- 172.21.15.32:0/2195639004 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c0d6480 con 0x7f3e3c0cffa0 2021-04-09T18:19:04.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e410bb700 1 --2- 172.21.15.32:0/2195639004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0d8220 0x7f3e3c0d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e3bfff700 1 --2- 172.21.15.32:0/2195639004 >> v2:172.21.15.86:3300/0 conn(0x7f3e3c0d7800 0x7f3e3c0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e408ba700 1 --2- 172.21.15.32:0/2195639004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c0cffa0 0x7f3e3c0d2440 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7f3e24006700 tx=0x7f3e24009b80).ready entity=mon.0 client_cookie=ed60ce47278dbb26 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e3b7fe700 1 -- 172.21.15.32:0/2195639004 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e2400c0c0 con 0x7f3e3c0cffa0 2021-04-09T18:19:04.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e3b7fe700 1 -- 172.21.15.32:0/2195639004 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2400c440 con 0x7f3e3c0cffa0 2021-04-09T18:19:04.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e3b7fe700 1 -- 172.21.15.32:0/2195639004 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e2400c920 con 0x7f3e3c0cffa0 2021-04-09T18:19:04.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e42a9e700 1 -- 172.21.15.32:0/2195639004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c0cffa0 msgr2=0x7f3e3c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:04.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.470 7f3e42a9e700 1 --2- 172.21.15.32:0/2195639004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c0cffa0 0x7f3e3c0d2440 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7f3e24006700 tx=0x7f3e24009b80).stop 2021-04-09T18:19:04.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.471 7f3e42a9e700 1 -- 172.21.15.32:0/2195639004 shutdown_connections 2021-04-09T18:19:04.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.471 7f3e42a9e700 1 --2- 172.21.15.32:0/2195639004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0d8220 0x7f3e3c0d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.471 7f3e42a9e700 1 --2- 172.21.15.32:0/2195639004 >> v2:172.21.15.86:3300/0 conn(0x7f3e3c0d7800 0x7f3e3c0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.471 7f3e42a9e700 1 --2- 172.21.15.32:0/2195639004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c0cffa0 0x7f3e3c0d2440 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.471 7f3e42a9e700 1 -- 172.21.15.32:0/2195639004 >> 172.21.15.32:0/2195639004 conn(0x7f3e3c0ce5c0 msgr2=0x7f3e3c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:04.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.471 7f3e42a9e700 1 -- 172.21.15.32:0/2195639004 shutdown_connections 2021-04-09T18:19:04.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.471 7f3e42a9e700 1 -- 172.21.15.32:0/2195639004 wait complete. 2021-04-09T18:19:04.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.491 7f3e42a9e700 1 Processor -- start 2021-04-09T18:19:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.491 7f3e42a9e700 1 -- start start 2021-04-09T18:19:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e42a9e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0b72f0 0x7f3e3c0b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:04.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e42a9e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c19c860 0x7f3e3c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e42a9e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c19fdd0 0x7f3e3c19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e42a9e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c19f230 con 0x7f3e3c19c860 2021-04-09T18:19:04.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e42a9e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c08a6a0 con 0x7f3e3c19fdd0 2021-04-09T18:19:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e42a9e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c18dce0 con 0x7f3e3c0b72f0 2021-04-09T18:19:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e3bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c19c860 0x7f3e3c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e3bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c19c860 0x7f3e3c19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47142/0 (socket says 172.21.15.32:47142) 2021-04-09T18:19:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e3bfff700 1 -- 172.21.15.32:0/2832280408 learned_addr learned my addr 172.21.15.32:0/2832280408 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e410bb700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c19fdd0 0x7f3e3c19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e408ba700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0b72f0 0x7f3e3c0b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e3bfff700 1 -- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0b72f0 msgr2=0x7f3e3c0b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:04.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e3bfff700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0b72f0 0x7f3e3c0b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e3bfff700 1 -- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c19fdd0 msgr2=0x7f3e3c19da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:04.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.492 7f3e3bfff700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c19fdd0 0x7f3e3c19da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.493 7f3e3bfff700 1 -- 172.21.15.32:0/2832280408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c18e150 con 0x7f3e3c19c860 2021-04-09T18:19:04.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.493 7f3e3bfff700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c19c860 0x7f3e3c19f7b0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f3e2c0099a0 tx=0x7f3e2c00bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:04.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.493 7f3e397fa700 1 -- 172.21.15.32:0/2832280408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e2c00c2f0 con 0x7f3e3c19c860 2021-04-09T18:19:04.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.493 7f3e397fa700 1 -- 172.21.15.32:0/2832280408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2c00c670 con 0x7f3e3c19c860 2021-04-09T18:19:04.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.493 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c18e150 con 0x7f3e3c19c860 2021-04-09T18:19:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.493 7f3e397fa700 1 -- 172.21.15.32:0/2832280408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e2c00cb50 con 0x7f3e3c19c860 2021-04-09T18:19:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.493 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c18e150 con 0x7f3e3c19c860 2021-04-09T18:19:04.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.493 7f3e397fa700 1 -- 172.21.15.32:0/2832280408 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3e2c00a000 con 0x7f3e3c19c860 2021-04-09T18:19:04.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.494 7f3e397fa700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e20018930 0x7f3e2001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.494 7f3e397fa700 1 -- 172.21.15.32:0/2832280408 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3e2c032d90 con 0x7f3e3c19c860 2021-04-09T18:19:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.494 7f3e408ba700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e20018930 0x7f3e2001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.494 7f3e408ba700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e20018930 0x7f3e2001ae10 crc :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:04.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.513 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c182e10 con 0x7f3e3c19c860 2021-04-09T18:19:04.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.517 7f3e397fa700 1 -- 172.21.15.32:0/2832280408 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3e2c00bdb0 con 0x7f3e3c19c860 2021-04-09T18:19:04.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.708 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e3c0d7280 con 0x7f3e3c19c860 2021-04-09T18:19:04.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.708 7f3e397fa700 1 -- 172.21.15.32:0/2832280408 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f3e2c00a000 con 0x7f3e3c19c860 2021-04-09T18:19:04.752 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:04.752 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:04.754 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:04.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.710 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e20018930 msgr2=0x7f3e2001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.710 7f3e42a9e700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e20018930 0x7f3e2001ae10 crc :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.710 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c19c860 msgr2=0x7f3e3c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:04.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.710 7f3e42a9e700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c19c860 0x7f3e3c19f7b0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f3e2c0099a0 tx=0x7f3e2c00bc70).stop 2021-04-09T18:19:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.711 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 shutdown_connections 2021-04-09T18:19:04.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.711 7f3e42a9e700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e20018930 0x7f3e2001ae10 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.711 7f3e42a9e700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e3c19c860 0x7f3e3c19f7b0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.711 7f3e42a9e700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c19fdd0 0x7f3e3c19da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.711 7f3e42a9e700 1 --2- 172.21.15.32:0/2832280408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c0b72f0 0x7f3e3c0b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:04.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.711 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 >> 172.21.15.32:0/2832280408 conn(0x7f3e3c097550 msgr2=0x7f3e3c18b920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:04.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.711 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 shutdown_connections 2021-04-09T18:19:04.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:04.711 7f3e42a9e700 1 -- 172.21.15.32:0/2832280408 wait complete. 2021-04-09T18:19:05.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:05.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:05.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:19:05.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:19:06.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:06.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.290 7f54d1b30700 1 Processor -- start 2021-04-09T18:19:06.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.290 7f54d1b30700 1 -- start start 2021-04-09T18:19:06.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54d1b30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc10d010 0x7f54cc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:06.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54d1b30700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f54cc0d1da0 0x7f54cc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:06.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54d1b30700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54cc0d82c0 0x7f54cc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:06.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54d1b30700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54cc10e780 con 0x7f54cc0d82c0 2021-04-09T18:19:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54d1b30700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54cc0d8d30 con 0x7f54cc0d1da0 2021-04-09T18:19:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54d1b30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc0d9220 con 0x7f54cc10d010 2021-04-09T18:19:06.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54cc0d82c0 0x7f54cc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54cc0d82c0 0x7f54cc10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47152/0 (socket says 172.21.15.32:47152) 2021-04-09T18:19:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cbfff700 1 -- 172.21.15.32:0/715545987 learned_addr learned my addr 172.21.15.32:0/715545987 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:06.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cb7fe700 1 --2- 172.21.15.32:0/715545987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc10d010 0x7f54cc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54caffd700 1 --2- 172.21.15.32:0/715545987 >> v2:172.21.15.86:3300/0 conn(0x7f54cc0d1da0 0x7f54cc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:06.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cbfff700 1 -- 172.21.15.32:0/715545987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc10d010 msgr2=0x7f54cc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cbfff700 1 --2- 172.21.15.32:0/715545987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc10d010 0x7f54cc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cbfff700 1 -- 172.21.15.32:0/715545987 >> v2:172.21.15.86:3300/0 conn(0x7f54cc0d1da0 msgr2=0x7f54cc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cbfff700 1 --2- 172.21.15.32:0/715545987 >> v2:172.21.15.86:3300/0 conn(0x7f54cc0d1da0 0x7f54cc0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cbfff700 1 -- 172.21.15.32:0/715545987 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc10d480 con 0x7f54cc0d82c0 2021-04-09T18:19:06.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54caffd700 1 --2- 172.21.15.32:0/715545987 >> v2:172.21.15.86:3300/0 conn(0x7f54cc0d1da0 0x7f54cc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:06.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.291 7f54cbfff700 1 --2- 172.21.15.32:0/715545987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0d82c0 0x7f54cc10da00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f54c4006f00 tx=0x7f54c4009f10).ready entity=mon.0 client_cookie=bba82407490d0e59 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:06.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54ca7fc700 1 -- 172.21.15.32:0/715545987 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54c400b3b0 con 0x7f54cc0d82c0 2021-04-09T18:19:06.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54ca7fc700 1 -- 172.21.15.32:0/715545987 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c400b730 con 0x7f54cc0d82c0 2021-04-09T18:19:06.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54ca7fc700 1 -- 172.21.15.32:0/715545987 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54c400bc10 con 0x7f54cc0d82c0 2021-04-09T18:19:06.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54d1b30700 1 -- 172.21.15.32:0/715545987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0d82c0 msgr2=0x7f54cc10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54d1b30700 1 --2- 172.21.15.32:0/715545987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0d82c0 0x7f54cc10da00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f54c4006f00 tx=0x7f54c4009f10).stop 2021-04-09T18:19:06.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54d1b30700 1 -- 172.21.15.32:0/715545987 shutdown_connections 2021-04-09T18:19:06.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54d1b30700 1 --2- 172.21.15.32:0/715545987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc0d82c0 0x7f54cc10da00 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54d1b30700 1 --2- 172.21.15.32:0/715545987 >> v2:172.21.15.86:3300/0 conn(0x7f54cc0d1da0 0x7f54cc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54d1b30700 1 --2- 172.21.15.32:0/715545987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc10d010 0x7f54cc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54d1b30700 1 -- 172.21.15.32:0/715545987 >> 172.21.15.32:0/715545987 conn(0x7f54cc0ce5f0 msgr2=0x7f54cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:06.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54d1b30700 1 -- 172.21.15.32:0/715545987 shutdown_connections 2021-04-09T18:19:06.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.292 7f54d1b30700 1 -- 172.21.15.32:0/715545987 wait complete. 2021-04-09T18:19:06.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54d1b30700 1 Processor -- start 2021-04-09T18:19:06.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54d1b30700 1 -- start start 2021-04-09T18:19:06.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54d1b30700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc180f10 0x7f54cc181310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:06.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54d1b30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc17a190 0x7f54cc17a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54d1b30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc17abb0 0x7f54cc17fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:06.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54d1b30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc08fb80 con 0x7f54cc17a190 2021-04-09T18:19:06.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54d1b30700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc181930 con 0x7f54cc180f10 2021-04-09T18:19:06.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54d1b30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc184020 con 0x7f54cc17abb0 2021-04-09T18:19:06.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54caffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc17a190 0x7f54cc17a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54caffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc17a190 0x7f54cc17a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47154/0 (socket says 172.21.15.32:47154) 2021-04-09T18:19:06.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54caffd700 1 -- 172.21.15.32:0/1066107167 learned_addr learned my addr 172.21.15.32:0/1066107167 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54cb7fe700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc180f10 0x7f54cc181310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.313 7f54cbfff700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc17abb0 0x7f54cc17fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54caffd700 1 -- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc17abb0 msgr2=0x7f54cc17fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54caffd700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc17abb0 0x7f54cc17fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54caffd700 1 -- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc180f10 msgr2=0x7f54cc181310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54caffd700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc180f10 0x7f54cc181310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54caffd700 1 -- 172.21.15.32:0/1066107167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc180070 con 0x7f54cc17a190 2021-04-09T18:19:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54caffd700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc17a190 0x7f54cc17a590 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f54c0006f20 tx=0x7f54c000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54b3fff700 1 -- 172.21.15.32:0/1066107167 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54c000bd80 con 0x7f54cc17a190 2021-04-09T18:19:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54b3fff700 1 -- 172.21.15.32:0/1066107167 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0000c90 con 0x7f54cc17a190 2021-04-09T18:19:06.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54cc180070 con 0x7f54cc17a190 2021-04-09T18:19:06.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54b3fff700 1 -- 172.21.15.32:0/1066107167 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54c00022b0 con 0x7f54cc17a190 2021-04-09T18:19:06.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.314 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54cc180070 con 0x7f54cc17a190 2021-04-09T18:19:06.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.315 7f54b3fff700 1 -- 172.21.15.32:0/1066107167 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f54c000a000 con 0x7f54cc17a190 2021-04-09T18:19:06.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.315 7f54b3fff700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54ac018930 0x7f54ac01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:06.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.315 7f54b3fff700 1 -- 172.21.15.32:0/1066107167 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f54c0032a30 con 0x7f54cc17a190 2021-04-09T18:19:06.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.315 7f54cb7fe700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54ac018930 0x7f54ac01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:06.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.315 7f54cb7fe700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54ac018930 0x7f54ac01adc0 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.335 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54cc181930 con 0x7f54cc17a190 2021-04-09T18:19:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.338 7f54b3fff700 1 -- 172.21.15.32:0/1066107167 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f54c0006bb0 con 0x7f54cc17a190 2021-04-09T18:19:06.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.529 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54cc181930 con 0x7f54cc17a190 2021-04-09T18:19:06.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.530 7f54b3fff700 1 -- 172.21.15.32:0/1066107167 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f54c0006bb0 con 0x7f54cc17a190 2021-04-09T18:19:06.571 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:06.572 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:06.572 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:06.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54ac018930 msgr2=0x7f54ac01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:06.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54ac018930 0x7f54ac01adc0 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc17a190 msgr2=0x7f54cc17a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:06.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc17a190 0x7f54cc17a590 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f54c0006f20 tx=0x7f54c000b990).stop 2021-04-09T18:19:06.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 shutdown_connections 2021-04-09T18:19:06.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54ac018930 0x7f54ac01adc0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54cc17a190 0x7f54cc17a590 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54cc17abb0 0x7f54cc17fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 --2- 172.21.15.32:0/1066107167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54cc180f10 0x7f54cc181310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:06.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.532 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 >> 172.21.15.32:0/1066107167 conn(0x7f54cc192570 msgr2=0x7f54cc1917c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:06.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.533 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 shutdown_connections 2021-04-09T18:19:06.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:06.533 7f54d1b30700 1 -- 172.21.15.32:0/1066107167 wait complete. 2021-04-09T18:19:06.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:06.993 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:07.994 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.115 7effaccfd700 1 Processor -- start 2021-04-09T18:19:08.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.115 7effaccfd700 1 -- start start 2021-04-09T18:19:08.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effaccfd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7effa80d7ca0 0x7effa80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:08.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effaccfd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa810d480 con 0x7effa810d010 2021-04-09T18:19:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effaccfd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa80d86c0 con 0x7effa810da00 2021-04-09T18:19:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effaccfd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa80d9220 con 0x7effa80d7ca0 2021-04-09T18:19:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effa6b13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7effa810d010 0x7effa80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56752/0 (socket says 172.21.15.32:56752) 2021-04-09T18:19:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effa6b13700 1 -- 172.21.15.32:0/2048646419 learned_addr learned my addr 172.21.15.32:0/2048646419 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:08.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effa6312700 1 --2- 172.21.15.32:0/2048646419 >> v2:172.21.15.86:3300/0 conn(0x7effa80d7ca0 0x7effa80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:08.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effa5b11700 1 -- 172.21.15.32:0/2048646419 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496847675 0 0) 0x7eff98000ce0 con 0x7effa810d010 2021-04-09T18:19:08.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effa5b11700 1 -- 172.21.15.32:0/2048646419 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff90002fe0 con 0x7effa810d010 2021-04-09T18:19:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effa5b11700 1 -- 172.21.15.32:0/2048646419 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261221750 0 0) 0x7effa0000d30 con 0x7effa810da00 2021-04-09T18:19:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.116 7effa5b11700 1 -- 172.21.15.32:0/2048646419 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff90003210 con 0x7effa810da00 2021-04-09T18:19:08.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effa5b11700 1 -- 172.21.15.32:0/2048646419 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3998368803 0 0) 0x7eff980010e0 con 0x7effa810d010 2021-04-09T18:19:08.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effa6312700 1 -- 172.21.15.32:0/2048646419 >> v1:172.21.15.86:6789/0 conn(0x7effa810da00 legacy=0x7effa810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:08.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effa6312700 1 -- 172.21.15.32:0/2048646419 >> v1:172.21.15.32:6789/0 conn(0x7effa810d010 legacy=0x7effa80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:08.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effa6312700 1 -- 172.21.15.32:0/2048646419 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa80d1f70 con 0x7effa80d7ca0 2021-04-09T18:19:08.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effa6312700 1 --2- 172.21.15.32:0/2048646419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa80d7ca0 0x7effa80d80a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7eff9c006650 tx=0x7eff9c009a80).ready entity=mon.1 client_cookie=2e4a721879e8375d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:08.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effa5b11700 1 -- 172.21.15.32:0/2048646419 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff9c0009b0 con 0x7effa80d7ca0 2021-04-09T18:19:08.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effa5b11700 1 -- 172.21.15.32:0/2048646419 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c00c0c0 con 0x7effa80d7ca0 2021-04-09T18:19:08.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effa5b11700 1 -- 172.21.15.32:0/2048646419 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff9c00c510 con 0x7effa80d7ca0 2021-04-09T18:19:08.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effaccfd700 1 -- 172.21.15.32:0/2048646419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa80d7ca0 msgr2=0x7effa80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:08.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.117 7effaccfd700 1 --2- 172.21.15.32:0/2048646419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa80d7ca0 0x7effa80d80a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7eff9c006650 tx=0x7eff9c009a80).stop 2021-04-09T18:19:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.118 7effaccfd700 1 -- 172.21.15.32:0/2048646419 shutdown_connections 2021-04-09T18:19:08.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.118 7effaccfd700 1 --2- 172.21.15.32:0/2048646419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa80d7ca0 0x7effa80d80a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:08.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.118 7effaccfd700 1 -- 172.21.15.32:0/2048646419 >> 172.21.15.32:0/2048646419 conn(0x7effa80ce5f0 msgr2=0x7effa80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:08.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.118 7effaccfd700 1 -- 172.21.15.32:0/2048646419 shutdown_connections 2021-04-09T18:19:08.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.118 7effaccfd700 1 -- 172.21.15.32:0/2048646419 wait complete. 2021-04-09T18:19:08.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.138 7effaccfd700 1 Processor -- start 2021-04-09T18:19:08.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.138 7effaccfd700 1 -- start start 2021-04-09T18:19:08.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effaccfd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa81ac380 0x7effa818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:08.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effaccfd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7effa81ac780 0x7effa818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:08.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effaccfd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa8183520 0x7effa8183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:08.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effaccfd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa81ace10 con 0x7effa81ac380 2021-04-09T18:19:08.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effaccfd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa8189cf0 con 0x7effa8183520 2021-04-09T18:19:08.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effaccfd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa8183f40 con 0x7effa81ac780 2021-04-09T18:19:08.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa81ac380 0x7effa818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:08.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa81ac380 0x7effa818a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47168/0 (socket says 172.21.15.32:47168) 2021-04-09T18:19:08.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6b13700 1 -- 172.21.15.32:0/2631483672 learned_addr learned my addr 172.21.15.32:0/2631483672 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6312700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7effa81ac780 0x7effa818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa7314700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa8183520 0x7effa8183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6b13700 1 -- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7effa81ac780 msgr2=0x7effa818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:08.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6b13700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7effa81ac780 0x7effa818a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:08.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6b13700 1 -- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa8183520 msgr2=0x7effa8183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:08.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6b13700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa8183520 0x7effa8183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:08.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6b13700 1 -- 172.21.15.32:0/2631483672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa8184160 con 0x7effa81ac380 2021-04-09T18:19:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa7314700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa8183520 0x7effa8183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effa6b13700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa81ac380 0x7effa818a230 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7eff98000f90 tx=0x7eff9800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:08.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7eff977fe700 1 -- 172.21.15.32:0/2631483672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff98001a30 con 0x7effa81ac380 2021-04-09T18:19:08.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7eff977fe700 1 -- 172.21.15.32:0/2631483672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff980043a0 con 0x7effa81ac380 2021-04-09T18:19:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effaccfd700 1 -- 172.21.15.32:0/2631483672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa8184160 con 0x7effa81ac380 2021-04-09T18:19:08.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7eff977fe700 1 -- 172.21.15.32:0/2631483672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff98001a30 con 0x7effa81ac380 2021-04-09T18:19:08.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.139 7effaccfd700 1 -- 172.21.15.32:0/2631483672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa8184160 con 0x7effa81ac380 2021-04-09T18:19:08.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.140 7eff977fe700 1 -- 172.21.15.32:0/2631483672 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7eff98008000 con 0x7effa81ac380 2021-04-09T18:19:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.140 7eff977fe700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7eff8c018930 0x7eff8c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:08.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.140 7eff977fe700 1 -- 172.21.15.32:0/2631483672 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7eff98032a20 con 0x7effa81ac380 2021-04-09T18:19:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.140 7effa6312700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7eff8c018930 0x7eff8c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:08.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.141 7effa6312700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7eff8c018930 0x7eff8c01adc0 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:08.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.160 7effaccfd700 1 -- 172.21.15.32:0/2631483672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa8182d10 con 0x7effa81ac380 2021-04-09T18:19:08.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.163 7eff977fe700 1 -- 172.21.15.32:0/2631483672 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7eff98007070 con 0x7effa81ac380 2021-04-09T18:19:08.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.354 7effaccfd700 1 -- 172.21.15.32:0/2631483672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effa8182d10 con 0x7effa81ac380 2021-04-09T18:19:08.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.354 7eff977fe700 1 -- 172.21.15.32:0/2631483672 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7eff9800ad30 con 0x7effa81ac380 2021-04-09T18:19:08.396 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:08.396 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:08.397 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:08.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 -- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7eff8c018930 msgr2=0x7eff8c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:08.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7eff8c018930 0x7eff8c01adc0 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 -- 172.21.15.32:0/2631483672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa81ac380 msgr2=0x7effa818a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:08.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa81ac380 0x7effa818a230 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7eff98000f90 tx=0x7eff9800a940).stop 2021-04-09T18:19:08.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 -- 172.21.15.32:0/2631483672 shutdown_connections 2021-04-09T18:19:08.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7eff8c018930 0x7eff8c01adc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:08.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7effa8183520 0x7effa8183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:08.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7effa81ac780 0x7effa818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:08.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 --2- 172.21.15.32:0/2631483672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effa81ac380 0x7effa818a230 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:08.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 -- 172.21.15.32:0/2631483672 >> 172.21.15.32:0/2631483672 conn(0x7effa819b890 msgr2=0x7effa81a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:08.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 -- 172.21.15.32:0/2631483672 shutdown_connections 2021-04-09T18:19:08.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:08.357 7effaccfd700 1 -- 172.21.15.32:0/2631483672 wait complete. 2021-04-09T18:19:08.810 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:08.811 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:09.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:09.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.924 7f94b6e59700 1 Processor -- start 2021-04-09T18:19:09.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.924 7f94b6e59700 1 -- start start 2021-04-09T18:19:09.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.924 7f94b6e59700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b00cffa0 0x7f94b00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:09.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.924 7f94b6e59700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f94b00d7800 0x7f94b00d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:09.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.924 7f94b6e59700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b00d7280 con 0x7f94b00d5c70 2021-04-09T18:19:09.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.924 7f94b6e59700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b00d8580 con 0x7f94b00d7800 2021-04-09T18:19:09.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.924 7f94b6e59700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b00d91e0 con 0x7f94b00cffa0 2021-04-09T18:19:09.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94affff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f94b00d7800 0x7f94b00d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:09.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94b5476700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f94b00d5c70 0x7f94b00d6070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56766/0 (socket says 172.21.15.32:56766) 2021-04-09T18:19:09.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94b5476700 1 -- 172.21.15.32:0/4169070158 learned_addr learned my addr 172.21.15.32:0/4169070158 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:09.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94b4c75700 1 --2- 172.21.15.32:0/4169070158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b00cffa0 0x7f94b00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94affff700 1 -- 172.21.15.32:0/4169070158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b00cffa0 msgr2=0x7f94b00d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:10.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94affff700 1 --2- 172.21.15.32:0/4169070158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b00cffa0 0x7f94b00d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94affff700 1 -- 172.21.15.32:0/4169070158 >> v1:172.21.15.32:6789/0 conn(0x7f94b00d5c70 legacy=0x7f94b00d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:10.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94af7fe700 1 -- 172.21.15.32:0/4169070158 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017719790 0 0) 0x7f94a8000d30 con 0x7f94b00d5c70 2021-04-09T18:19:10.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94affff700 1 -- 172.21.15.32:0/4169070158 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b00d8220 con 0x7f94b00d7800 2021-04-09T18:19:10.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94b4c75700 1 --2- 172.21.15.32:0/4169070158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b00cffa0 0x7f94b00d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:10.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94affff700 1 --2- 172.21.15.32:0/4169070158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b00d7800 0x7f94b00d7c00 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f94a0006650 tx=0x7f94a0009a80).ready entity=mon.0 client_cookie=200af5795ec88424 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:10.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94af7fe700 1 -- 172.21.15.32:0/4169070158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a00009b0 con 0x7f94b00d7800 2021-04-09T18:19:10.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94af7fe700 1 -- 172.21.15.32:0/4169070158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a000c0c0 con 0x7f94b00d7800 2021-04-09T18:19:10.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94af7fe700 1 -- 172.21.15.32:0/4169070158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a000c510 con 0x7f94b00d7800 2021-04-09T18:19:10.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94b6e59700 1 -- 172.21.15.32:0/4169070158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b00d7800 msgr2=0x7f94b00d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:10.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.925 7f94b6e59700 1 --2- 172.21.15.32:0/4169070158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b00d7800 0x7f94b00d7c00 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f94a0006650 tx=0x7f94a0009a80).stop 2021-04-09T18:19:10.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.926 7f94b6e59700 1 -- 172.21.15.32:0/4169070158 shutdown_connections 2021-04-09T18:19:10.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.926 7f94b6e59700 1 --2- 172.21.15.32:0/4169070158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b00d7800 0x7f94b00d7c00 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.926 7f94b6e59700 1 --2- 172.21.15.32:0/4169070158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b00cffa0 0x7f94b00d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.926 7f94b6e59700 1 -- 172.21.15.32:0/4169070158 >> 172.21.15.32:0/4169070158 conn(0x7f94b00ce5c0 msgr2=0x7f94b00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:10.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.926 7f94b6e59700 1 -- 172.21.15.32:0/4169070158 shutdown_connections 2021-04-09T18:19:10.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.926 7f94b6e59700 1 -- 172.21.15.32:0/4169070158 wait complete. 2021-04-09T18:19:10.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.946 7f94b6e59700 1 Processor -- start 2021-04-09T18:19:10.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.946 7f94b6e59700 1 -- start start 2021-04-09T18:19:10.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94b6e59700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b01acff0 0x7f94b008fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:10.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94b6e59700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0090150 0x7f94b008c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:10.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94b6e59700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b008cca0 0x7f94b008e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:10.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94b6e59700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b008c100 con 0x7f94b0090150 2021-04-09T18:19:10.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94b6e59700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b008d0a0 con 0x7f94b01acff0 2021-04-09T18:19:10.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94b6e59700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b008e960 con 0x7f94b008cca0 2021-04-09T18:19:10.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0090150 0x7f94b008c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:10.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0090150 0x7f94b008c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47180/0 (socket says 172.21.15.32:47180) 2021-04-09T18:19:10.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94affff700 1 -- 172.21.15.32:0/1503347715 learned_addr learned my addr 172.21.15.32:0/1503347715 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:10.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94b4c75700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b01acff0 0x7f94b008fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.947 7f94b5476700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b008cca0 0x7f94b008e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94affff700 1 -- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b008cca0 msgr2=0x7f94b008e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94affff700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b008cca0 0x7f94b008e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94affff700 1 -- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b01acff0 msgr2=0x7f94b008fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:10.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94affff700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b01acff0 0x7f94b008fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94affff700 1 -- 172.21.15.32:0/1503347715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b008eb80 con 0x7f94b0090150 2021-04-09T18:19:10.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94b4c75700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b01acff0 0x7f94b008fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:10.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94affff700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0090150 0x7f94b008c680 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7f94a0009650 tx=0x7f94a000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:10.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94ad7fa700 1 -- 172.21.15.32:0/1503347715 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a0006030 con 0x7f94b0090150 2021-04-09T18:19:10.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94ad7fa700 1 -- 172.21.15.32:0/1503347715 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a000bcd0 con 0x7f94b0090150 2021-04-09T18:19:10.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b008eb80 con 0x7f94b0090150 2021-04-09T18:19:10.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94ad7fa700 1 -- 172.21.15.32:0/1503347715 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a00054d0 con 0x7f94b0090150 2021-04-09T18:19:10.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.948 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b008eb80 con 0x7f94b0090150 2021-04-09T18:19:10.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.949 7f94ad7fa700 1 -- 172.21.15.32:0/1503347715 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f94a0006030 con 0x7f94b0090150 2021-04-09T18:19:10.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.949 7f94ad7fa700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94940188c0 0x7f949401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.949 7f94ad7fa700 1 -- 172.21.15.32:0/1503347715 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f94a0032780 con 0x7f94b0090150 2021-04-09T18:19:10.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.949 7f94b4c75700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94940188c0 0x7f949401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:10.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.950 7f94b4c75700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94940188c0 0x7f949401adc0 crc :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:10.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.968 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b0182dc0 con 0x7f94b0090150 2021-04-09T18:19:10.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:09.972 7f94ad7fa700 1 -- 172.21.15.32:0/1503347715 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f94a0000940 con 0x7f94b0090150 2021-04-09T18:19:10.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.164 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94b00d7280 con 0x7f94b0090150 2021-04-09T18:19:10.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.164 7f94ad7fa700 1 -- 172.21.15.32:0/1503347715 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f94a0006030 con 0x7f94b0090150 2021-04-09T18:19:10.205 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:10.206 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:10.207 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:10.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94940188c0 msgr2=0x7f949401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:10.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94940188c0 0x7f949401adc0 crc :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0090150 msgr2=0x7f94b008c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:10.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0090150 0x7f94b008c680 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7f94a0009650 tx=0x7f94a000b8e0).stop 2021-04-09T18:19:10.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 shutdown_connections 2021-04-09T18:19:10.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94940188c0 0x7f949401adc0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94b0090150 0x7f94b008c680 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b008cca0 0x7f94b008e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 --2- 172.21.15.32:0/1503347715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b01acff0 0x7f94b008fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:10.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 >> 172.21.15.32:0/1503347715 conn(0x7f94b009abb0 msgr2=0x7f94b00a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:10.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 shutdown_connections 2021-04-09T18:19:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:10.167 7f94b6e59700 1 -- 172.21.15.32:0/1503347715 wait complete. 2021-04-09T18:19:10.621 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:10.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:11.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.744 7f0551a21700 1 Processor -- start 2021-04-09T18:19:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.744 7f0551a21700 1 -- start start 2021-04-09T18:19:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.745 7f0551a21700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f054c10da00 0x7f054c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.745 7f0551a21700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c10d480 con 0x7f054c0d7ca0 2021-04-09T18:19:11.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.745 7f0551a21700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c0d1f70 con 0x7f054c10d010 2021-04-09T18:19:11.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.745 7f0551a21700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054c0d8d30 con 0x7f054c10da00 2021-04-09T18:19:11.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.745 7f054bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f054c10da00 0x7f054c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:11.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.745 7f054affd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f054c0d7ca0 0x7f054c0d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56778/0 (socket says 172.21.15.32:56778) 2021-04-09T18:19:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.745 7f054affd700 1 -- 172.21.15.32:0/4147400519 learned_addr learned my addr 172.21.15.32:0/4147400519 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f054a7fc700 1 -- 172.21.15.32:0/4147400519 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010870667 0 0) 0x7f0534000ce0 con 0x7f054c0d7ca0 2021-04-09T18:19:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f054a7fc700 1 -- 172.21.15.32:0/4147400519 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0538002fe0 con 0x7f054c0d7ca0 2021-04-09T18:19:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f054bfff700 1 -- 172.21.15.32:0/4147400519 >> v1:172.21.15.86:6789/0 conn(0x7f054c10d010 legacy=0x7f054c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f054bfff700 1 -- 172.21.15.32:0/4147400519 >> v1:172.21.15.32:6789/0 conn(0x7f054c0d7ca0 legacy=0x7f054c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:11.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f054bfff700 1 -- 172.21.15.32:0/4147400519 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0d9220 con 0x7f054c10da00 2021-04-09T18:19:11.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f054bfff700 1 --2- 172.21.15.32:0/4147400519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c10da00 0x7f054c10de00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f0544006e30 tx=0x7f0544009e60).ready entity=mon.0 client_cookie=1e3bed3167b34579 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:11.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f054a7fc700 1 -- 172.21.15.32:0/4147400519 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0544006a50 con 0x7f054c10da00 2021-04-09T18:19:11.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f054a7fc700 1 -- 172.21.15.32:0/4147400519 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054400b300 con 0x7f054c10da00 2021-04-09T18:19:11.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f054a7fc700 1 -- 172.21.15.32:0/4147400519 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f054400b750 con 0x7f054c10da00 2021-04-09T18:19:11.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f0551a21700 1 -- 172.21.15.32:0/4147400519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c10da00 msgr2=0x7f054c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:11.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.746 7f0551a21700 1 --2- 172.21.15.32:0/4147400519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c10da00 0x7f054c10de00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f0544006e30 tx=0x7f0544009e60).stop 2021-04-09T18:19:11.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.747 7f0551a21700 1 -- 172.21.15.32:0/4147400519 shutdown_connections 2021-04-09T18:19:11.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.747 7f0551a21700 1 --2- 172.21.15.32:0/4147400519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c10da00 0x7f054c10de00 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.747 7f0551a21700 1 -- 172.21.15.32:0/4147400519 >> 172.21.15.32:0/4147400519 conn(0x7f054c0ce5f0 msgr2=0x7f054c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.747 7f0551a21700 1 -- 172.21.15.32:0/4147400519 shutdown_connections 2021-04-09T18:19:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.747 7f0551a21700 1 -- 172.21.15.32:0/4147400519 wait complete. 2021-04-09T18:19:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.767 7f0551a21700 1 Processor -- start 2021-04-09T18:19:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.767 7f0551a21700 1 -- start start 2021-04-09T18:19:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0551a21700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c17e010 0x7f054c17e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0551a21700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f054c09f280 0x7f054c09f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0551a21700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c0b1b00 0x7f054c0b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0551a21700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c09ed00 con 0x7f054c0b1b00 2021-04-09T18:19:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0551a21700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c09fca0 con 0x7f054c17e010 2021-04-09T18:19:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0551a21700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0b2520 con 0x7f054c09f280 2021-04-09T18:19:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c0b1b00 0x7f054c0b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:11.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c0b1b00 0x7f054c0b1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47196/0 (socket says 172.21.15.32:47196) 2021-04-09T18:19:11.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054bfff700 1 -- 172.21.15.32:0/2355281032 learned_addr learned my addr 172.21.15.32:0/2355281032 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:11.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054affd700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f054c09f280 0x7f054c09f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:11.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054b7fe700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c17e010 0x7f054c17e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:11.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054bfff700 1 -- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f054c09f280 msgr2=0x7f054c09f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:11.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054bfff700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f054c09f280 0x7f054c09f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:11.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054bfff700 1 -- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c17e010 msgr2=0x7f054c17e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:11.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054bfff700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c17e010 0x7f054c17e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:11.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054bfff700 1 -- 172.21.15.32:0/2355281032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0b2740 con 0x7f054c0b1b00 2021-04-09T18:19:11.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f054bfff700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c0b1b00 0x7f054c0b1f00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f0544001070 tx=0x7f054400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:11.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0543fff700 1 -- 172.21.15.32:0/2355281032 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0544001270 con 0x7f054c0b1b00 2021-04-09T18:19:11.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0543fff700 1 -- 172.21.15.32:0/2355281032 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0544006df0 con 0x7f054c0b1b00 2021-04-09T18:19:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0543fff700 1 -- 172.21.15.32:0/2355281032 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0544001720 con 0x7f054c0b1b00 2021-04-09T18:19:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.768 7f0551a21700 1 -- 172.21.15.32:0/2355281032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c0b2740 con 0x7f054c0b1b00 2021-04-09T18:19:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.769 7f0551a21700 1 -- 172.21.15.32:0/2355281032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c0b2740 con 0x7f054c0b1b00 2021-04-09T18:19:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.769 7f0543fff700 1 -- 172.21.15.32:0/2355281032 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f05440014f0 con 0x7f054c0b1b00 2021-04-09T18:19:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.769 7f0543fff700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0530018930 0x7f053001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.769 7f0543fff700 1 -- 172.21.15.32:0/2355281032 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0544032f20 con 0x7f054c0b1b00 2021-04-09T18:19:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.770 7f054b7fe700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0530018930 0x7f053001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:11.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.770 7f054b7fe700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0530018930 0x7f053001ae10 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.789 7f0551a21700 1 -- 172.21.15.32:0/2355281032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c17cb20 con 0x7f054c0b1b00 2021-04-09T18:19:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.793 7f0543fff700 1 -- 172.21.15.32:0/2355281032 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0544006dd0 con 0x7f054c0b1b00 2021-04-09T18:19:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.983 7f0551a21700 1 -- 172.21.15.32:0/2355281032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f054c10d480 con 0x7f054c0b1b00 2021-04-09T18:19:12.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.983 7f0543fff700 1 -- 172.21.15.32:0/2355281032 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f0544006dd0 con 0x7f054c0b1b00 2021-04-09T18:19:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:12.025 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:12.026 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:12.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.985 7f0551a21700 1 -- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0530018930 msgr2=0x7f053001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.985 7f0551a21700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0530018930 0x7f053001ae10 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.985 7f0551a21700 1 -- 172.21.15.32:0/2355281032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c0b1b00 msgr2=0x7f054c0b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:12.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.985 7f0551a21700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c0b1b00 0x7f054c0b1f00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f0544001070 tx=0x7f054400bd40).stop 2021-04-09T18:19:12.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.986 7f0551a21700 1 -- 172.21.15.32:0/2355281032 shutdown_connections 2021-04-09T18:19:12.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.986 7f0551a21700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0530018930 0x7f053001ae10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:12.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.986 7f0551a21700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f054c0b1b00 0x7f054c0b1f00 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:12.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.986 7f0551a21700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f054c09f280 0x7f054c09f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:12.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.986 7f0551a21700 1 --2- 172.21.15.32:0/2355281032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f054c17e010 0x7f054c17e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.986 7f0551a21700 1 -- 172.21.15.32:0/2355281032 >> 172.21.15.32:0/2355281032 conn(0x7f054c1965a0 msgr2=0x7f054c19dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.986 7f0551a21700 1 -- 172.21.15.32:0/2355281032 shutdown_connections 2021-04-09T18:19:12.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:11.986 7f0551a21700 1 -- 172.21.15.32:0/2355281032 wait complete. 2021-04-09T18:19:12.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:12.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:13.445 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:13.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.565 7f077c57c700 1 Processor -- start 2021-04-09T18:19:13.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.565 7f077c57c700 1 -- start start 2021-04-09T18:19:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077c57c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07740d7ca0 0x7f07740d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077c57c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f077410da00 0x7f077410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077c57c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077410d480 con 0x7f077410d010 2021-04-09T18:19:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077c57c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07740d9220 con 0x7f077410da00 2021-04-09T18:19:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077c57c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07740d86c0 con 0x7f07740d7ca0 2021-04-09T18:19:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077ab99700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f077410da00 0x7f077410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077ab99700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f077410da00 0x7f077410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47202/0 (socket says 172.21.15.32:47202) 2021-04-09T18:19:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077ab99700 1 -- 172.21.15.32:0/2196844019 learned_addr learned my addr 172.21.15.32:0/2196844019 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f0779b97700 1 --2- 172.21.15.32:0/2196844019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07740d7ca0 0x7f07740d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:13.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077a398700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f077410d010 0x7f07740d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54716/0 (socket says 172.21.15.32:54716) 2021-04-09T18:19:13.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077ab99700 1 -- 172.21.15.32:0/2196844019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07740d7ca0 msgr2=0x7f07740d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:13.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077ab99700 1 --2- 172.21.15.32:0/2196844019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07740d7ca0 0x7f07740d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077ab99700 1 -- 172.21.15.32:0/2196844019 >> v1:172.21.15.86:6789/0 conn(0x7f077410d010 legacy=0x7f07740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077ab99700 1 -- 172.21.15.32:0/2196844019 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07740d1f70 con 0x7f077410da00 2021-04-09T18:19:13.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f0779396700 1 -- 172.21.15.32:0/2196844019 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255422687 0 0) 0x7f0770000c60 con 0x7f077410d010 2021-04-09T18:19:13.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f077ab99700 1 --2- 172.21.15.32:0/2196844019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077410da00 0x7f077410de00 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f076c006f00 tx=0x7f076c009f10).ready entity=mon.0 client_cookie=f83caccb4df9e617 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:13.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f0779396700 1 -- 172.21.15.32:0/2196844019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f076c00b3b0 con 0x7f077410da00 2021-04-09T18:19:13.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f0779396700 1 -- 172.21.15.32:0/2196844019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c00b730 con 0x7f077410da00 2021-04-09T18:19:13.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.566 7f0779396700 1 -- 172.21.15.32:0/2196844019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f076c00bc10 con 0x7f077410da00 2021-04-09T18:19:13.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.567 7f077c57c700 1 -- 172.21.15.32:0/2196844019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077410da00 msgr2=0x7f077410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:13.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.567 7f077c57c700 1 --2- 172.21.15.32:0/2196844019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077410da00 0x7f077410de00 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f076c006f00 tx=0x7f076c009f10).stop 2021-04-09T18:19:13.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.567 7f077c57c700 1 -- 172.21.15.32:0/2196844019 shutdown_connections 2021-04-09T18:19:13.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.567 7f077c57c700 1 --2- 172.21.15.32:0/2196844019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f077410da00 0x7f077410de00 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.567 7f077c57c700 1 --2- 172.21.15.32:0/2196844019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07740d7ca0 0x7f07740d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.567 7f077c57c700 1 -- 172.21.15.32:0/2196844019 >> 172.21.15.32:0/2196844019 conn(0x7f07740ce5f0 msgr2=0x7f07740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:13.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.567 7f077c57c700 1 -- 172.21.15.32:0/2196844019 shutdown_connections 2021-04-09T18:19:13.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.567 7f077c57c700 1 -- 172.21.15.32:0/2196844019 wait complete. 2021-04-09T18:19:13.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.587 7f077c57c700 1 Processor -- start 2021-04-09T18:19:13.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f077c57c700 1 -- start start 2021-04-09T18:19:13.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f077c57c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f077418c5c0 0x7f07741a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:13.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f077c57c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0774191880 0x7f077418cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:13.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f077c57c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0774191c80 0x7f077418d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:13.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f077c57c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077418c040 con 0x7f0774191880 2021-04-09T18:19:13.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f077c57c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741aa5c0 con 0x7f077418c5c0 2021-04-09T18:19:13.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f077c57c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077418bd20 con 0x7f0774191c80 2021-04-09T18:19:13.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f0779b97700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0774191880 0x7f077418cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:13.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f0779b97700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0774191880 0x7f077418cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47208/0 (socket says 172.21.15.32:47208) 2021-04-09T18:19:13.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f0779b97700 1 -- 172.21.15.32:0/3268278412 learned_addr learned my addr 172.21.15.32:0/3268278412 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:13.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f077ab99700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0774191c80 0x7f077418d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:13.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.588 7f077a398700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f077418c5c0 0x7f07741a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:13.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f0779b97700 1 -- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0774191c80 msgr2=0x7f077418d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:13.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f0779b97700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0774191c80 0x7f077418d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f0779b97700 1 -- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f077418c5c0 msgr2=0x7f07741a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:13.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f0779b97700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f077418c5c0 0x7f07741a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f0779b97700 1 -- 172.21.15.32:0/3268278412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077418da30 con 0x7f0774191880 2021-04-09T18:19:13.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f0779b97700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0774191880 0x7f077418cdf0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f076400b580 tx=0x7f076400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:13.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f076affd700 1 -- 172.21.15.32:0/3268278412 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07640064d0 con 0x7f0774191880 2021-04-09T18:19:13.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f076affd700 1 -- 172.21.15.32:0/3268278412 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076400bd10 con 0x7f0774191880 2021-04-09T18:19:13.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f077c57c700 1 -- 172.21.15.32:0/3268278412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077418da30 con 0x7f0774191880 2021-04-09T18:19:13.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f076affd700 1 -- 172.21.15.32:0/3268278412 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0764002030 con 0x7f0774191880 2021-04-09T18:19:13.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.589 7f077c57c700 1 -- 172.21.15.32:0/3268278412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077418da30 con 0x7f0774191880 2021-04-09T18:19:13.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.590 7f076affd700 1 -- 172.21.15.32:0/3268278412 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0764006c90 con 0x7f0774191880 2021-04-09T18:19:13.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.590 7f076affd700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f075c0188c0 0x7f075c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:13.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.590 7f076affd700 1 -- 172.21.15.32:0/3268278412 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0764032780 con 0x7f0774191880 2021-04-09T18:19:13.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.590 7f077a398700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f075c0188c0 0x7f075c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:13.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.591 7f077a398700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f075c0188c0 0x7f075c01adc0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:13.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.610 7f077c57c700 1 -- 172.21.15.32:0/3268278412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077418dfc0 con 0x7f0774191880 2021-04-09T18:19:13.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.613 7f076affd700 1 -- 172.21.15.32:0/3268278412 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0764006820 con 0x7f0774191880 2021-04-09T18:19:13.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.807 7f077c57c700 1 -- 172.21.15.32:0/3268278412 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f077418dfc0 con 0x7f0774191880 2021-04-09T18:19:13.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.807 7f076affd700 1 -- 172.21.15.32:0/3268278412 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f076400bd10 con 0x7f0774191880 2021-04-09T18:19:13.848 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:13.848 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:13.848 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:13.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 -- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f075c0188c0 msgr2=0x7f075c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:13.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f075c0188c0 0x7f075c01adc0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 -- 172.21.15.32:0/3268278412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0774191880 msgr2=0x7f077418cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:13.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0774191880 0x7f077418cdf0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f076400b580 tx=0x7f076400b920).stop 2021-04-09T18:19:13.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 -- 172.21.15.32:0/3268278412 shutdown_connections 2021-04-09T18:19:13.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f075c0188c0 0x7f075c01adc0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0774191880 0x7f077418cdf0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0774191c80 0x7f077418d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 --2- 172.21.15.32:0/3268278412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f077418c5c0 0x7f07741a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:13.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 -- 172.21.15.32:0/3268278412 >> 172.21.15.32:0/3268278412 conn(0x7f07741837d0 msgr2=0x7f0774195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:13.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 -- 172.21.15.32:0/3268278412 shutdown_connections 2021-04-09T18:19:13.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:13.810 7f077c57c700 1 -- 172.21.15.32:0/3268278412 wait complete. 2021-04-09T18:19:14.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:14.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:15.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:15.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.385 7fcbd31d7700 1 Processor -- start 2021-04-09T18:19:15.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.385 7fcbd31d7700 1 -- start start 2021-04-09T18:19:15.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbd31d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc10d010 0x7fcbcc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:15.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbd31d7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcbcc0d1da0 0x7fcbcc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:15.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbd31d7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc10d480 con 0x7fcbcc0d82c0 2021-04-09T18:19:15.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbd31d7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbcc10e780 con 0x7fcbcc0d1da0 2021-04-09T18:19:15.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbd31d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc0d8d30 con 0x7fcbcc10d010 2021-04-09T18:19:15.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbd17f4700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcbcc0d82c0 0x7fcbcc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56804/0 (socket says 172.21.15.32:56804) 2021-04-09T18:19:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbd17f4700 1 -- 172.21.15.32:0/3146710155 learned_addr learned my addr 172.21.15.32:0/3146710155 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbcbfff700 1 --2- 172.21.15.32:0/3146710155 >> v2:172.21.15.86:3300/0 conn(0x7fcbcc0d1da0 0x7fcbcc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:15.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbd0ff3700 1 --2- 172.21.15.32:0/3146710155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc10d010 0x7fcbcc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbcb7fe700 1 -- 172.21.15.32:0/3146710155 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485853989 0 0) 0x7fcbc40009b0 con 0x7fcbcc0d82c0 2021-04-09T18:19:15.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.386 7fcbcb7fe700 1 -- 172.21.15.32:0/3146710155 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb8002fe0 con 0x7fcbcc0d82c0 2021-04-09T18:19:15.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbd0ff3700 1 -- 172.21.15.32:0/3146710155 >> v2:172.21.15.86:3300/0 conn(0x7fcbcc0d1da0 msgr2=0x7fcbcc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:15.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbd0ff3700 1 --2- 172.21.15.32:0/3146710155 >> v2:172.21.15.86:3300/0 conn(0x7fcbcc0d1da0 0x7fcbcc0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbd0ff3700 1 -- 172.21.15.32:0/3146710155 >> v1:172.21.15.32:6789/0 conn(0x7fcbcc0d82c0 legacy=0x7fcbcc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:15.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbd0ff3700 1 -- 172.21.15.32:0/3146710155 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc0d9220 con 0x7fcbcc10d010 2021-04-09T18:19:15.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbcbfff700 1 --2- 172.21.15.32:0/3146710155 >> v2:172.21.15.86:3300/0 conn(0x7fcbcc0d1da0 0x7fcbcc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:19:15.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbd0ff3700 1 --2- 172.21.15.32:0/3146710155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc10d010 0x7fcbcc0d1780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fcbb4006650 tx=0x7fcbb4009a80).ready entity=mon.2 client_cookie=41c5f91c51b4a028 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:15.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbcb7fe700 1 -- 172.21.15.32:0/3146710155 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb40009b0 con 0x7fcbcc10d010 2021-04-09T18:19:15.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbcb7fe700 1 -- 172.21.15.32:0/3146710155 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb400c0c0 con 0x7fcbcc10d010 2021-04-09T18:19:15.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbcb7fe700 1 -- 172.21.15.32:0/3146710155 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb400c510 con 0x7fcbcc10d010 2021-04-09T18:19:15.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbd31d7700 1 -- 172.21.15.32:0/3146710155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc10d010 msgr2=0x7fcbcc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.387 7fcbd31d7700 1 --2- 172.21.15.32:0/3146710155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc10d010 0x7fcbcc0d1780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fcbb4006650 tx=0x7fcbb4009a80).stop 2021-04-09T18:19:15.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.388 7fcbd31d7700 1 -- 172.21.15.32:0/3146710155 shutdown_connections 2021-04-09T18:19:15.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.388 7fcbd31d7700 1 --2- 172.21.15.32:0/3146710155 >> v2:172.21.15.86:3300/0 conn(0x7fcbcc0d1da0 0x7fcbcc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.388 7fcbd31d7700 1 --2- 172.21.15.32:0/3146710155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc10d010 0x7fcbcc0d1780 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.388 7fcbd31d7700 1 -- 172.21.15.32:0/3146710155 >> 172.21.15.32:0/3146710155 conn(0x7fcbcc0ce5f0 msgr2=0x7fcbcc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:15.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.388 7fcbd31d7700 1 -- 172.21.15.32:0/3146710155 shutdown_connections 2021-04-09T18:19:15.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.388 7fcbd31d7700 1 -- 172.21.15.32:0/3146710155 wait complete. 2021-04-09T18:19:15.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.408 7fcbd31d7700 1 Processor -- start 2021-04-09T18:19:15.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.408 7fcbd31d7700 1 -- start start 2021-04-09T18:19:15.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd31d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbcc1a5210 0x7fcbcc1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:15.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd31d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbcc1a6340 0x7fcbcc1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:15.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd31d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc0abaa0 0x7fcbcc0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:15.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd31d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc1a6d60 con 0x7fcbcc1a6340 2021-04-09T18:19:15.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd31d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc0ac4c0 con 0x7fcbcc1a5210 2021-04-09T18:19:15.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd31d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc0ac6e0 con 0x7fcbcc0abaa0 2021-04-09T18:19:15.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbcbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbcc1a6340 0x7fcbcc1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:15.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbcbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbcc1a6340 0x7fcbcc1a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47216/0 (socket says 172.21.15.32:47216) 2021-04-09T18:19:15.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbcbfff700 1 -- 172.21.15.32:0/502340860 learned_addr learned my addr 172.21.15.32:0/502340860 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:15.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd17f4700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc0abaa0 0x7fcbcc0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:15.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd0ff3700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbcc1a5210 0x7fcbcc1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:15.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbcbfff700 1 -- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc0abaa0 msgr2=0x7fcbcc0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbcbfff700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc0abaa0 0x7fcbcc0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbcbfff700 1 -- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbcc1a5210 msgr2=0x7fcbcc1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbcbfff700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbcc1a5210 0x7fcbcc1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbcbfff700 1 -- 172.21.15.32:0/502340860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc17a720 con 0x7fcbcc1a6340 2021-04-09T18:19:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbcbfff700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbcc1a6340 0x7fcbcc1a6740 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7fcbbc00b870 tx=0x7fcbbc00bc10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbc97fa700 1 -- 172.21.15.32:0/502340860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbbc006240 con 0x7fcbcc1a6340 2021-04-09T18:19:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbc97fa700 1 -- 172.21.15.32:0/502340860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbbc006540 con 0x7fcbcc1a6340 2021-04-09T18:19:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbcc17a9c0 con 0x7fcbcc1a6340 2021-04-09T18:19:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbc97fa700 1 -- 172.21.15.32:0/502340860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbbc0058e0 con 0x7fcbcc1a6340 2021-04-09T18:19:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.409 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbcc17a9c0 con 0x7fcbcc1a6340 2021-04-09T18:19:15.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.410 7fcbc97fa700 1 -- 172.21.15.32:0/502340860 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcbbc005af0 con 0x7fcbcc1a6340 2021-04-09T18:19:15.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.410 7fcbc97fa700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbb00188d0 0x7fcbb001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.410 7fcbc97fa700 1 -- 172.21.15.32:0/502340860 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcbbc034fa0 con 0x7fcbcc1a6340 2021-04-09T18:19:15.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.410 7fcbd0ff3700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbb00188d0 0x7fcbb001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:15.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.411 7fcbd0ff3700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbb00188d0 0x7fcbb001add0 crc :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.430 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbcc1a6d60 con 0x7fcbcc1a6340 2021-04-09T18:19:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.433 7fcbc97fa700 1 -- 172.21.15.32:0/502340860 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcbbc0058c0 con 0x7fcbcc1a6340 2021-04-09T18:19:15.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.624 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbcc1a6d60 con 0x7fcbcc1a6340 2021-04-09T18:19:15.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.624 7fcbc97fa700 1 -- 172.21.15.32:0/502340860 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fcbbc005ad0 con 0x7fcbcc1a6340 2021-04-09T18:19:15.665 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:15.666 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:15.666 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.626 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbb00188d0 msgr2=0x7fcbb001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbb00188d0 0x7fcbb001add0 crc :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbcc1a6340 msgr2=0x7fcbcc1a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:15.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbcc1a6340 0x7fcbcc1a6740 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7fcbbc00b870 tx=0x7fcbbc00bc10).stop 2021-04-09T18:19:15.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 shutdown_connections 2021-04-09T18:19:15.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbb00188d0 0x7fcbb001add0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbcc1a6340 0x7fcbcc1a6740 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbcc0abaa0 0x7fcbcc0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 --2- 172.21.15.32:0/502340860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbcc1a5210 0x7fcbcc1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:15.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 >> 172.21.15.32:0/502340860 conn(0x7fcbcc1a99c0 msgr2=0x7fcbcc17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:15.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 shutdown_connections 2021-04-09T18:19:15.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:15.627 7fcbd31d7700 1 -- 172.21.15.32:0/502340860 wait complete. 2021-04-09T18:19:16.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:16.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:17.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.202 7f9a834ca700 1 Processor -- start 2021-04-09T18:19:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.203 7f9a834ca700 1 -- start start 2021-04-09T18:19:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.203 7f9a834ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c0cffa0 0x7f9a7c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.203 7f9a834ca700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9a7c0d7800 0x7f9a7c0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.203 7f9a834ca700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a7c0d5c70 0x7f9a7c0d6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.203 7f9a834ca700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7c0d8580 con 0x7f9a7c0d7800 2021-04-09T18:19:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.203 7f9a834ca700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7c0d91e0 con 0x7f9a7c0d5c70 2021-04-09T18:19:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.203 7f9a834ca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c0d7280 con 0x7f9a7c0cffa0 2021-04-09T18:19:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.203 7f9a80ae5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9a7c0d7800 0x7f9a7c0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.203 7f9a80ae5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9a7c0d7800 0x7f9a7c0d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47222/0 (socket says 172.21.15.32:47222) 2021-04-09T18:19:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a80ae5700 1 -- 172.21.15.32:0/2439236844 learned_addr learned my addr 172.21.15.32:0/2439236844 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a81ae7700 1 --2- 172.21.15.32:0/2439236844 >> v2:172.21.15.86:3300/0 conn(0x7f9a7c0d5c70 0x7f9a7c0d6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:17.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a812e6700 1 --2- 172.21.15.32:0/2439236844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c0cffa0 0x7f9a7c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:17.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a80ae5700 1 -- 172.21.15.32:0/2439236844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c0cffa0 msgr2=0x7f9a7c0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:17.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a80ae5700 1 --2- 172.21.15.32:0/2439236844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c0cffa0 0x7f9a7c0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a80ae5700 1 -- 172.21.15.32:0/2439236844 >> v2:172.21.15.86:3300/0 conn(0x7f9a7c0d5c70 msgr2=0x7f9a7c0d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a80ae5700 1 --2- 172.21.15.32:0/2439236844 >> v2:172.21.15.86:3300/0 conn(0x7f9a7c0d5c70 0x7f9a7c0d6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a80ae5700 1 -- 172.21.15.32:0/2439236844 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c0d8220 con 0x7f9a7c0d7800 2021-04-09T18:19:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a80ae5700 1 --2- 172.21.15.32:0/2439236844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0d7800 0x7f9a7c0d7c00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f9a6c006700 tx=0x7f9a6c009b00).ready entity=mon.0 client_cookie=b4a6bdb40b48b886 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a73fff700 1 -- 172.21.15.32:0/2439236844 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a6c00c0c0 con 0x7f9a7c0d7800 2021-04-09T18:19:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a73fff700 1 -- 172.21.15.32:0/2439236844 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c00c400 con 0x7f9a7c0d7800 2021-04-09T18:19:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a73fff700 1 -- 172.21.15.32:0/2439236844 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a6c00c8e0 con 0x7f9a7c0d7800 2021-04-09T18:19:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a834ca700 1 -- 172.21.15.32:0/2439236844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0d7800 msgr2=0x7f9a7c0d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.204 7f9a834ca700 1 --2- 172.21.15.32:0/2439236844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0d7800 0x7f9a7c0d7c00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f9a6c006700 tx=0x7f9a6c009b00).stop 2021-04-09T18:19:17.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.205 7f9a834ca700 1 -- 172.21.15.32:0/2439236844 shutdown_connections 2021-04-09T18:19:17.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.205 7f9a834ca700 1 --2- 172.21.15.32:0/2439236844 >> v2:172.21.15.86:3300/0 conn(0x7f9a7c0d5c70 0x7f9a7c0d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.205 7f9a834ca700 1 --2- 172.21.15.32:0/2439236844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c0d7800 0x7f9a7c0d7c00 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.205 7f9a834ca700 1 --2- 172.21.15.32:0/2439236844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c0cffa0 0x7f9a7c0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.205 7f9a834ca700 1 -- 172.21.15.32:0/2439236844 >> 172.21.15.32:0/2439236844 conn(0x7f9a7c0ce5c0 msgr2=0x7f9a7c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:17.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.205 7f9a834ca700 1 -- 172.21.15.32:0/2439236844 shutdown_connections 2021-04-09T18:19:17.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.205 7f9a834ca700 1 -- 172.21.15.32:0/2439236844 wait complete. 2021-04-09T18:19:17.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.225 7f9a834ca700 1 Processor -- start 2021-04-09T18:19:17.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.225 7f9a834ca700 1 -- start start 2021-04-09T18:19:17.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a834ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c08ec80 0x7f9a7c08f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:17.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a834ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c08c950 0x7f9a7c08cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:17.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a834ca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c08d560 0x7f9a7c08d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:17.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a834ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c08e700 con 0x7f9a7c08c950 2021-04-09T18:19:17.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a834ca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c08f6a0 con 0x7f9a7c08d560 2021-04-09T18:19:17.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a834ca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c1acff0 con 0x7f9a7c08ec80 2021-04-09T18:19:17.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a80ae5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c08c950 0x7f9a7c08cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:17.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a80ae5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c08c950 0x7f9a7c08cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47228/0 (socket says 172.21.15.32:47228) 2021-04-09T18:19:17.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a80ae5700 1 -- 172.21.15.32:0/233984359 learned_addr learned my addr 172.21.15.32:0/233984359 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.226 7f9a812e6700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c08ec80 0x7f9a7c08f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.227 7f9a81ae7700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c08d560 0x7f9a7c08d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.227 7f9a80ae5700 1 -- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c08ec80 msgr2=0x7f9a7c08f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:17.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.227 7f9a80ae5700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c08ec80 0x7f9a7c08f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.227 7f9a80ae5700 1 -- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c08d560 msgr2=0x7f9a7c08d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.227 7f9a80ae5700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c08d560 0x7f9a7c08d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.227 7f9a80ae5700 1 -- 172.21.15.32:0/233984359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c08df80 con 0x7f9a7c08c950 2021-04-09T18:19:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.227 7f9a81ae7700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c08d560 0x7f9a7c08d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:17.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.227 7f9a80ae5700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c08c950 0x7f9a7c08cf40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f9a6c00b590 tx=0x7f9a6c00b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:17.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.228 7f9a71ffb700 1 -- 172.21.15.32:0/233984359 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a6c00bd20 con 0x7f9a7c08c950 2021-04-09T18:19:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.228 7f9a71ffb700 1 -- 172.21.15.32:0/233984359 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c006030 con 0x7f9a7c08c950 2021-04-09T18:19:17.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.228 7f9a834ca700 1 -- 172.21.15.32:0/233984359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c08df80 con 0x7f9a7c08c950 2021-04-09T18:19:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.228 7f9a71ffb700 1 -- 172.21.15.32:0/233984359 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a6c005560 con 0x7f9a7c08c950 2021-04-09T18:19:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.228 7f9a834ca700 1 -- 172.21.15.32:0/233984359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c08df80 con 0x7f9a7c08c950 2021-04-09T18:19:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.228 7f9a71ffb700 1 -- 172.21.15.32:0/233984359 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9a6c0093e0 con 0x7f9a7c08c950 2021-04-09T18:19:17.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.228 7f9a71ffb700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a5c018930 0x7f9a5c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.228 7f9a71ffb700 1 -- 172.21.15.32:0/233984359 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9a6c032a50 con 0x7f9a7c08c950 2021-04-09T18:19:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.229 7f9a812e6700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a5c018930 0x7f9a5c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.229 7f9a812e6700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a5c018930 0x7f9a5c01ae10 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:17.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.247 7f9a834ca700 1 -- 172.21.15.32:0/233984359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c182dc0 con 0x7f9a7c08c950 2021-04-09T18:19:17.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.251 7f9a71ffb700 1 -- 172.21.15.32:0/233984359 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9a6c0090c0 con 0x7f9a7c08c950 2021-04-09T18:19:17.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.442 7f9a834ca700 1 -- 172.21.15.32:0/233984359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a7c0d7280 con 0x7f9a7c08c950 2021-04-09T18:19:17.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.442 7f9a71ffb700 1 -- 172.21.15.32:0/233984359 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f9a6c006030 con 0x7f9a7c08c950 2021-04-09T18:19:17.484 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 -- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a5c018930 msgr2=0x7f9a5c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a5c018930 0x7f9a5c01ae10 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 -- 172.21.15.32:0/233984359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c08c950 msgr2=0x7f9a7c08cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c08c950 0x7f9a7c08cf40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f9a6c00b590 tx=0x7f9a6c00b930).stop 2021-04-09T18:19:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 -- 172.21.15.32:0/233984359 shutdown_connections 2021-04-09T18:19:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9a5c018930 0x7f9a5c01ae10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a7c08c950 0x7f9a7c08cf40 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a7c08d560 0x7f9a7c08d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 --2- 172.21.15.32:0/233984359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a7c08ec80 0x7f9a7c08f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 -- 172.21.15.32:0/233984359 >> 172.21.15.32:0/233984359 conn(0x7f9a7c09abb0 msgr2=0x7f9a7c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 -- 172.21.15.32:0/233984359 shutdown_connections 2021-04-09T18:19:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:17.445 7f9a834ca700 1 -- 172.21.15.32:0/233984359 wait complete. 2021-04-09T18:19:17.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:17.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:18.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:19.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.024 7f044a1ca700 1 Processor -- start 2021-04-09T18:19:19.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.024 7f044a1ca700 1 -- start start 2021-04-09T18:19:19.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.024 7f044a1ca700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f044410d010 0x7f04440d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:19.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.024 7f044a1ca700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044410d480 con 0x7f04440d8490 2021-04-09T18:19:19.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.024 7f044a1ca700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044410e1f0 con 0x7f04440d1da0 2021-04-09T18:19:19.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.024 7f044a1ca700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04440d8d30 con 0x7f044410d010 2021-04-09T18:19:19.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f04437fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f044410d010 0x7f04440d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:19.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f0443fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f04440d8490 0x7f044410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56826/0 (socket says 172.21.15.32:56826) 2021-04-09T18:19:19.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f0443fff700 1 -- 172.21.15.32:0/2009870401 learned_addr learned my addr 172.21.15.32:0/2009870401 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:19.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f04437fe700 1 -- 172.21.15.32:0/2009870401 >> v1:172.21.15.86:6789/0 conn(0x7f04440d1da0 legacy=0x7f04440d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:19.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f04437fe700 1 -- 172.21.15.32:0/2009870401 >> v1:172.21.15.32:6789/0 conn(0x7f04440d8490 legacy=0x7f044410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:19.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f04437fe700 1 -- 172.21.15.32:0/2009870401 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04440d9220 con 0x7f044410d010 2021-04-09T18:19:19.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f04427fc700 1 -- 172.21.15.32:0/2009870401 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385434686 0 0) 0x7f043c000d30 con 0x7f04440d8490 2021-04-09T18:19:19.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f04437fe700 1 --2- 172.21.15.32:0/2009870401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f044410d010 0x7f04440d1780 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f0434006650 tx=0x7f0434009a00).ready entity=mon.0 client_cookie=c3a36fc22f71ed3e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:19.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f04427fc700 1 -- 172.21.15.32:0/2009870401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0434009df0 con 0x7f044410d010 2021-04-09T18:19:19.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f04427fc700 1 -- 172.21.15.32:0/2009870401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04340009b0 con 0x7f044410d010 2021-04-09T18:19:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f04427fc700 1 -- 172.21.15.32:0/2009870401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f043400c2b0 con 0x7f044410d010 2021-04-09T18:19:19.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f044a1ca700 1 -- 172.21.15.32:0/2009870401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f044410d010 msgr2=0x7f04440d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.025 7f044a1ca700 1 --2- 172.21.15.32:0/2009870401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f044410d010 0x7f04440d1780 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f0434006650 tx=0x7f0434009a00).stop 2021-04-09T18:19:19.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.026 7f044a1ca700 1 -- 172.21.15.32:0/2009870401 shutdown_connections 2021-04-09T18:19:19.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.026 7f044a1ca700 1 --2- 172.21.15.32:0/2009870401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f044410d010 0x7f04440d1780 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:19.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.026 7f044a1ca700 1 -- 172.21.15.32:0/2009870401 >> 172.21.15.32:0/2009870401 conn(0x7f04440ce5f0 msgr2=0x7f04440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:19.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.026 7f044a1ca700 1 -- 172.21.15.32:0/2009870401 shutdown_connections 2021-04-09T18:19:19.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.026 7f044a1ca700 1 -- 172.21.15.32:0/2009870401 wait complete. 2021-04-09T18:19:19.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.046 7f044a1ca700 1 Processor -- start 2021-04-09T18:19:19.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.046 7f044a1ca700 1 -- start start 2021-04-09T18:19:19.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f044a1ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04441a9a50 0x7f044408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:19.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f044a1ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f044409d7d0 0x7f044409dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:19.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f044a1ca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0444183c20 0x7f0444184020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:19.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f044a1ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044409e1f0 con 0x7f04441a9a50 2021-04-09T18:19:19.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f044a1ca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044408ea60 con 0x7f0444183c20 2021-04-09T18:19:19.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f044a1ca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044408f540 con 0x7f044409d7d0 2021-04-09T18:19:19.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f04437fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04441a9a50 0x7f044408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:19.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f04437fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04441a9a50 0x7f044408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47240/0 (socket says 172.21.15.32:47240) 2021-04-09T18:19:19.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f04437fe700 1 -- 172.21.15.32:0/873284943 learned_addr learned my addr 172.21.15.32:0/873284943 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:19.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f0443fff700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0444183c20 0x7f0444184020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:19.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.047 7f0442ffd700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f044409d7d0 0x7f044409dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:19.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f04437fe700 1 -- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f044409d7d0 msgr2=0x7f044409dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:19.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f04437fe700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f044409d7d0 0x7f044409dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:19.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f04437fe700 1 -- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0444183c20 msgr2=0x7f0444184020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:19.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f04437fe700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0444183c20 0x7f0444184020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:19.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f04437fe700 1 -- 172.21.15.32:0/873284943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0444184640 con 0x7f04441a9a50 2021-04-09T18:19:19.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f0443fff700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0444183c20 0x7f0444184020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:19.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f0442ffd700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f044409d7d0 0x7f044409dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:19.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f04437fe700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04441a9a50 0x7f044408ef20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f04340096a0 tx=0x7f043400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:19.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f042bfff700 1 -- 172.21.15.32:0/873284943 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f043400bd80 con 0x7f04441a9a50 2021-04-09T18:19:19.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f042bfff700 1 -- 172.21.15.32:0/873284943 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0434006030 con 0x7f04441a9a50 2021-04-09T18:19:19.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f044a1ca700 1 -- 172.21.15.32:0/873284943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0444184640 con 0x7f04441a9a50 2021-04-09T18:19:19.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f042bfff700 1 -- 172.21.15.32:0/873284943 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0434005560 con 0x7f04441a9a50 2021-04-09T18:19:19.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.048 7f044a1ca700 1 -- 172.21.15.32:0/873284943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0444184640 con 0x7f04441a9a50 2021-04-09T18:19:19.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.049 7f042bfff700 1 -- 172.21.15.32:0/873284943 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f04340052e0 con 0x7f04441a9a50 2021-04-09T18:19:19.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.049 7f042bfff700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04240188c0 0x7f042401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:19.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.049 7f042bfff700 1 -- 172.21.15.32:0/873284943 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f04340093a0 con 0x7f04441a9a50 2021-04-09T18:19:19.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.049 7f0442ffd700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04240188c0 0x7f042401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:19.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.050 7f0442ffd700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04240188c0 0x7f042401adc0 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:19.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.069 7f044a1ca700 1 -- 172.21.15.32:0/873284943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044410d480 con 0x7f04441a9a50 2021-04-09T18:19:19.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.072 7f042bfff700 1 -- 172.21.15.32:0/873284943 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f04340052e0 con 0x7f04441a9a50 2021-04-09T18:19:19.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.263 7f044a1ca700 1 -- 172.21.15.32:0/873284943 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f044410d480 con 0x7f04441a9a50 2021-04-09T18:19:19.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.264 7f042bfff700 1 -- 172.21.15.32:0/873284943 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f04340093a0 con 0x7f04441a9a50 2021-04-09T18:19:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:19.306 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:19.307 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:19.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 -- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04240188c0 msgr2=0x7f042401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:19.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04240188c0 0x7f042401adc0 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:19.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 -- 172.21.15.32:0/873284943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04441a9a50 msgr2=0x7f044408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:19.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04441a9a50 0x7f044408ef20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f04340096a0 tx=0x7f043400b990).stop 2021-04-09T18:19:19.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 -- 172.21.15.32:0/873284943 shutdown_connections 2021-04-09T18:19:19.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04240188c0 0x7f042401adc0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0444183c20 0x7f0444184020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f044409d7d0 0x7f044409dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 --2- 172.21.15.32:0/873284943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04441a9a50 0x7f044408ef20 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 -- 172.21.15.32:0/873284943 >> 172.21.15.32:0/873284943 conn(0x7f04441965c0 msgr2=0x7f04441ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.266 7f044a1ca700 1 -- 172.21.15.32:0/873284943 shutdown_connections 2021-04-09T18:19:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:19.267 7f044a1ca700 1 -- 172.21.15.32:0/873284943 wait complete. 2021-04-09T18:19:19.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:19.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:20.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.842 7f062188b700 1 Processor -- start 2021-04-09T18:19:20.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.842 7f062188b700 1 -- start start 2021-04-09T18:19:20.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f062188b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f061c10d010 0x7f061c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:20.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f062188b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c0d1da0 0x7f061c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:20.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f062188b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061c10d480 con 0x7f061c0d82c0 2021-04-09T18:19:20.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f062188b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f061c0d9220 con 0x7f061c10d010 2021-04-09T18:19:20.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f062188b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061c0d86c0 con 0x7f061c0d1da0 2021-04-09T18:19:20.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f061bd75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f061c0d82c0 0x7f061c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56840/0 (socket says 172.21.15.32:56840) 2021-04-09T18:19:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f061bd75700 1 -- 172.21.15.32:0/2012365310 learned_addr learned my addr 172.21.15.32:0/2012365310 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f061b574700 1 --2- 172.21.15.32:0/2012365310 >> v2:172.21.15.86:3300/0 conn(0x7f061c10d010 0x7f061c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f061ad73700 1 --2- 172.21.15.32:0/2012365310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c0d1da0 0x7f061c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:20.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.843 7f061a572700 1 -- 172.21.15.32:0/2012365310 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731521801 0 0) 0x7f06140009b0 con 0x7f061c0d82c0 2021-04-09T18:19:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061a572700 1 -- 172.21.15.32:0/2012365310 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0604002fe0 con 0x7f061c0d82c0 2021-04-09T18:19:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061ad73700 1 -- 172.21.15.32:0/2012365310 >> v2:172.21.15.86:3300/0 conn(0x7f061c10d010 msgr2=0x7f061c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061ad73700 1 --2- 172.21.15.32:0/2012365310 >> v2:172.21.15.86:3300/0 conn(0x7f061c10d010 0x7f061c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:20.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061ad73700 1 -- 172.21.15.32:0/2012365310 >> v1:172.21.15.32:6789/0 conn(0x7f061c0d82c0 legacy=0x7f061c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:20.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061ad73700 1 -- 172.21.15.32:0/2012365310 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c10e1f0 con 0x7f061c0d1da0 2021-04-09T18:19:20.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061b574700 1 --2- 172.21.15.32:0/2012365310 >> v2:172.21.15.86:3300/0 conn(0x7f061c10d010 0x7f061c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:19:20.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061ad73700 1 --2- 172.21.15.32:0/2012365310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c0d1da0 0x7f061c0d7ca0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f06100069d0 tx=0x7f0610009e40).ready entity=mon.2 client_cookie=969b1af6e7a4966d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:20.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061a572700 1 -- 172.21.15.32:0/2012365310 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f061000b510 con 0x7f061c0d1da0 2021-04-09T18:19:20.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061a572700 1 -- 172.21.15.32:0/2012365310 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061000b890 con 0x7f061c0d1da0 2021-04-09T18:19:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f061a572700 1 -- 172.21.15.32:0/2012365310 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f061000bd70 con 0x7f061c0d1da0 2021-04-09T18:19:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f062188b700 1 -- 172.21.15.32:0/2012365310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c0d1da0 msgr2=0x7f061c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.844 7f062188b700 1 --2- 172.21.15.32:0/2012365310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c0d1da0 0x7f061c0d7ca0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f06100069d0 tx=0x7f0610009e40).stop 2021-04-09T18:19:20.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.845 7f062188b700 1 -- 172.21.15.32:0/2012365310 shutdown_connections 2021-04-09T18:19:20.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.845 7f062188b700 1 --2- 172.21.15.32:0/2012365310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c0d1da0 0x7f061c0d7ca0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:20.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.845 7f062188b700 1 --2- 172.21.15.32:0/2012365310 >> v2:172.21.15.86:3300/0 conn(0x7f061c10d010 0x7f061c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:20.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.845 7f062188b700 1 -- 172.21.15.32:0/2012365310 >> 172.21.15.32:0/2012365310 conn(0x7f061c0ce5f0 msgr2=0x7f061c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:20.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.845 7f062188b700 1 -- 172.21.15.32:0/2012365310 shutdown_connections 2021-04-09T18:19:20.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.845 7f062188b700 1 -- 172.21.15.32:0/2012365310 wait complete. 2021-04-09T18:19:20.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.865 7f062188b700 1 Processor -- start 2021-04-09T18:19:20.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.865 7f062188b700 1 -- start start 2021-04-09T18:19:20.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f062188b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f061c0b5360 0x7f061c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f062188b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c18e140 0x7f061c1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:20.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f062188b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061c1ad040 0x7f061c08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f062188b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c1a5360 con 0x7f061c0b5360 2021-04-09T18:19:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f062188b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c0902c0 con 0x7f061c1ad040 2021-04-09T18:19:20.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f062188b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061c0904e0 con 0x7f061c18e140 2021-04-09T18:19:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f061c0b5360 0x7f061c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f061c0b5360 0x7f061c1a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47254/0 (socket says 172.21.15.32:47254) 2021-04-09T18:19:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061b574700 1 -- 172.21.15.32:0/364732821 learned_addr learned my addr 172.21.15.32:0/364732821 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:20.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061ad73700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c18e140 0x7f061c1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:20.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061bd75700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061c1ad040 0x7f061c08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:20.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061b574700 1 -- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c18e140 msgr2=0x7f061c1aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:20.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061b574700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c18e140 0x7f061c1aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:20.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061b574700 1 -- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061c1ad040 msgr2=0x7f061c08fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061b574700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061c1ad040 0x7f061c08fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061b574700 1 -- 172.21.15.32:0/364732821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c183c80 con 0x7f061c0b5360 2021-04-09T18:19:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061bd75700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061c1ad040 0x7f061c08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:20.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f061b574700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f061c0b5360 0x7f061c1a4d40 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f060c00b580 tx=0x7f060c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f0603fff700 1 -- 172.21.15.32:0/364732821 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f060c00bd10 con 0x7f061c0b5360 2021-04-09T18:19:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f0603fff700 1 -- 172.21.15.32:0/364732821 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060c000c90 con 0x7f061c0b5360 2021-04-09T18:19:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f062188b700 1 -- 172.21.15.32:0/364732821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061c183c80 con 0x7f061c0b5360 2021-04-09T18:19:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f0603fff700 1 -- 172.21.15.32:0/364732821 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f060c002220 con 0x7f061c0b5360 2021-04-09T18:19:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.866 7f062188b700 1 -- 172.21.15.32:0/364732821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061c183c80 con 0x7f061c0b5360 2021-04-09T18:19:20.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.867 7f0603fff700 1 -- 172.21.15.32:0/364732821 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f060c006b00 con 0x7f061c0b5360 2021-04-09T18:19:20.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.867 7f0603fff700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc018930 0x7f05fc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:20.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.867 7f0603fff700 1 -- 172.21.15.32:0/364732821 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f060c0329f0 con 0x7f061c0b5360 2021-04-09T18:19:20.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.867 7f061ad73700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc018930 0x7f05fc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:20.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.868 7f061ad73700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc018930 0x7f05fc01ae10 crc :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:20.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.887 7f062188b700 1 -- 172.21.15.32:0/364732821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061c182d70 con 0x7f061c0b5360 2021-04-09T18:19:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:20.891 7f0603fff700 1 -- 172.21.15.32:0/364732821 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f060c00bd10 con 0x7f061c0b5360 2021-04-09T18:19:21.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.081 7f062188b700 1 -- 172.21.15.32:0/364732821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f061c182d70 con 0x7f061c0b5360 2021-04-09T18:19:21.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.082 7f0603fff700 1 -- 172.21.15.32:0/364732821 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f060c000c90 con 0x7f061c0b5360 2021-04-09T18:19:21.128 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:21.129 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:21.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 -- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc018930 msgr2=0x7f05fc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:21.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc018930 0x7f05fc01ae10 crc :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:21.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 -- 172.21.15.32:0/364732821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f061c0b5360 msgr2=0x7f061c1a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:21.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f061c0b5360 0x7f061c1a4d40 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f060c00b580 tx=0x7f060c00b920).stop 2021-04-09T18:19:21.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 -- 172.21.15.32:0/364732821 shutdown_connections 2021-04-09T18:19:21.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc018930 0x7f05fc01ae10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:21.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061c1ad040 0x7f061c08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:21.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f061c18e140 0x7f061c1aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:21.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 --2- 172.21.15.32:0/364732821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f061c0b5360 0x7f061c1a4d40 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:21.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 -- 172.21.15.32:0/364732821 >> 172.21.15.32:0/364732821 conn(0x7f061c19b460 msgr2=0x7f061c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:21.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 -- 172.21.15.32:0/364732821 shutdown_connections 2021-04-09T18:19:21.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:21.084 7f062188b700 1 -- 172.21.15.32:0/364732821 wait complete. 2021-04-09T18:19:21.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:21.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:22.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:22.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.661 7f7165f7e700 1 Processor -- start 2021-04-09T18:19:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.661 7f7165f7e700 1 -- start start 2021-04-09T18:19:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.661 7f7165f7e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f716010da00 0x7f716010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.661 7f7165f7e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716010d480 con 0x7f716010d010 2021-04-09T18:19:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.661 7f7165f7e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71600d1f70 con 0x7f71600d7ca0 2021-04-09T18:19:22.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.661 7f7165f7e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71600d8d30 con 0x7f716010da00 2021-04-09T18:19:22.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.661 7f715f7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f716010d010 0x7f71600d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56852/0 (socket says 172.21.15.32:56852) 2021-04-09T18:19:22.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.661 7f715f7fe700 1 -- 172.21.15.32:0/144513000 learned_addr learned my addr 172.21.15.32:0/144513000 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:22.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715ffff700 1 --2- 172.21.15.32:0/144513000 >> v2:172.21.15.86:3300/0 conn(0x7f716010da00 0x7f716010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:22.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715e7fc700 1 -- 172.21.15.32:0/144513000 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344689662 0 0) 0x7f7150000ce0 con 0x7f716010d010 2021-04-09T18:19:22.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715e7fc700 1 -- 172.21.15.32:0/144513000 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f714c002fe0 con 0x7f716010d010 2021-04-09T18:19:22.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715ffff700 1 -- 172.21.15.32:0/144513000 >> v1:172.21.15.86:6789/0 conn(0x7f71600d7ca0 legacy=0x7f71600d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:22.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715e7fc700 1 -- 172.21.15.32:0/144513000 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012874905 0 0) 0x7f7148000fe0 con 0x7f71600d7ca0 2021-04-09T18:19:22.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715ffff700 1 -- 172.21.15.32:0/144513000 >> v1:172.21.15.32:6789/0 conn(0x7f716010d010 legacy=0x7f71600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:22.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715ffff700 1 -- 172.21.15.32:0/144513000 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71600d9220 con 0x7f716010da00 2021-04-09T18:19:22.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715ffff700 1 --2- 172.21.15.32:0/144513000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f716010da00 0x7f716010de00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f7158006ad0 tx=0x7f7158009ad0).ready entity=mon.1 client_cookie=81b331f4b05edec2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:22.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715e7fc700 1 -- 172.21.15.32:0/144513000 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71580022e0 con 0x7f716010da00 2021-04-09T18:19:22.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715e7fc700 1 -- 172.21.15.32:0/144513000 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7158006410 con 0x7f716010da00 2021-04-09T18:19:22.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f715e7fc700 1 -- 172.21.15.32:0/144513000 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7158002620 con 0x7f716010da00 2021-04-09T18:19:22.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f7165f7e700 1 -- 172.21.15.32:0/144513000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f716010da00 msgr2=0x7f716010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:22.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.662 7f7165f7e700 1 --2- 172.21.15.32:0/144513000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f716010da00 0x7f716010de00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f7158006ad0 tx=0x7f7158009ad0).stop 2021-04-09T18:19:22.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.663 7f7165f7e700 1 -- 172.21.15.32:0/144513000 shutdown_connections 2021-04-09T18:19:22.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.663 7f7165f7e700 1 --2- 172.21.15.32:0/144513000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f716010da00 0x7f716010de00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:22.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.663 7f7165f7e700 1 -- 172.21.15.32:0/144513000 >> 172.21.15.32:0/144513000 conn(0x7f71600ce5f0 msgr2=0x7f71600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:22.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.663 7f7165f7e700 1 -- 172.21.15.32:0/144513000 shutdown_connections 2021-04-09T18:19:22.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.663 7f7165f7e700 1 -- 172.21.15.32:0/144513000 wait complete. 2021-04-09T18:19:22.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.683 7f7165f7e700 1 Processor -- start 2021-04-09T18:19:22.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.683 7f7165f7e700 1 -- start start 2021-04-09T18:19:22.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f7165f7e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f716017a200 0x7f71600a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:22.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f7165f7e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f716017ca90 0x7f716017ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:22.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f7165f7e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71601a46c0 0x7f71601a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:22.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f7165f7e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71601a50e0 con 0x7f716017ca90 2021-04-09T18:19:22.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f7165f7e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716017a600 con 0x7f716017a200 2021-04-09T18:19:22.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f7165f7e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716017a7b0 con 0x7f71601a46c0 2021-04-09T18:19:22.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f715effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f716017ca90 0x7f716017ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:22.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f715effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f716017ca90 0x7f716017ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47268/0 (socket says 172.21.15.32:47268) 2021-04-09T18:19:22.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f715effd700 1 -- 172.21.15.32:0/222268107 learned_addr learned my addr 172.21.15.32:0/222268107 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:22.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f715ffff700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71601a46c0 0x7f71601a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:22.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.684 7f715f7fe700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f716017a200 0x7f71600a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:22.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f715effd700 1 -- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71601a46c0 msgr2=0x7f71601a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:22.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f715effd700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71601a46c0 0x7f71601a4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:22.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f715effd700 1 -- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f716017a200 msgr2=0x7f71600a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:22.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f715effd700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f716017a200 0x7f71600a7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:22.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f715effd700 1 -- 172.21.15.32:0/222268107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71600a8080 con 0x7f716017ca90 2021-04-09T18:19:22.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f715effd700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f716017ca90 0x7f716017ce90 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f7148009980 tx=0x7f714800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:22.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f7157fff700 1 -- 172.21.15.32:0/222268107 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7148003030 con 0x7f716017ca90 2021-04-09T18:19:22.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f7157fff700 1 -- 172.21.15.32:0/222268107 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7148001400 con 0x7f716017ca90 2021-04-09T18:19:22.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f7165f7e700 1 -- 172.21.15.32:0/222268107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71600a8080 con 0x7f716017ca90 2021-04-09T18:19:22.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f7157fff700 1 -- 172.21.15.32:0/222268107 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7148001850 con 0x7f716017ca90 2021-04-09T18:19:22.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f7165f7e700 1 -- 172.21.15.32:0/222268107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71600a8080 con 0x7f716017ca90 2021-04-09T18:19:22.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.685 7f7157fff700 1 -- 172.21.15.32:0/222268107 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f71480015e0 con 0x7f716017ca90 2021-04-09T18:19:22.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.686 7f7157fff700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7144018930 0x7f714401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:22.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.686 7f7157fff700 1 -- 172.21.15.32:0/222268107 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7148032da0 con 0x7f716017ca90 2021-04-09T18:19:22.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.686 7f715f7fe700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7144018930 0x7f714401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:22.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.686 7f715f7fe700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7144018930 0x7f714401ae10 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:22.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.705 7f7165f7e700 1 -- 172.21.15.32:0/222268107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716010d480 con 0x7f716017ca90 2021-04-09T18:19:22.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.709 7f7157fff700 1 -- 172.21.15.32:0/222268107 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f714800be10 con 0x7f716017ca90 2021-04-09T18:19:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.901 7f7165f7e700 1 -- 172.21.15.32:0/222268107 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f716010d480 con 0x7f716017ca90 2021-04-09T18:19:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.901 7f7157fff700 1 -- 172.21.15.32:0/222268107 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f7148001400 con 0x7f716017ca90 2021-04-09T18:19:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:22.943 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:22.943 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:22.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.903 7f7165f7e700 1 -- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7144018930 msgr2=0x7f714401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:22.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.903 7f7165f7e700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7144018930 0x7f714401ae10 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.903 7f7165f7e700 1 -- 172.21.15.32:0/222268107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f716017ca90 msgr2=0x7f716017ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.903 7f7165f7e700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f716017ca90 0x7f716017ce90 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f7148009980 tx=0x7f714800bcd0).stop 2021-04-09T18:19:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.904 7f7165f7e700 1 -- 172.21.15.32:0/222268107 shutdown_connections 2021-04-09T18:19:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.904 7f7165f7e700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7144018930 0x7f714401ae10 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.904 7f7165f7e700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f716017ca90 0x7f716017ce90 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.904 7f7165f7e700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71601a46c0 0x7f71601a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.904 7f7165f7e700 1 --2- 172.21.15.32:0/222268107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f716017a200 0x7f71600a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.904 7f7165f7e700 1 -- 172.21.15.32:0/222268107 >> 172.21.15.32:0/222268107 conn(0x7f7160190570 msgr2=0x7f716018ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.904 7f7165f7e700 1 -- 172.21.15.32:0/222268107 shutdown_connections 2021-04-09T18:19:22.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:22.904 7f7165f7e700 1 -- 172.21.15.32:0/222268107 wait complete. 2021-04-09T18:19:23.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:23.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:24.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:24.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.470 7f70051cc700 1 Processor -- start 2021-04-09T18:19:24.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.470 7f70051cc700 1 -- start start 2021-04-09T18:19:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.471 7f70051cc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f700010da00 0x7f700010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.471 7f70051cc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700010d480 con 0x7f70000d7ca0 2021-04-09T18:19:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.471 7f70051cc700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70000d1f70 con 0x7f700010d010 2021-04-09T18:19:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.471 7f70051cc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70000d9220 con 0x7f700010da00 2021-04-09T18:19:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.471 7f6ffe59c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f70000d7ca0 0x7f70000d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56868/0 (socket says 172.21.15.32:56868) 2021-04-09T18:19:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.471 7f6ffe59c700 1 -- 172.21.15.32:0/3464743073 learned_addr learned my addr 172.21.15.32:0/3464743073 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.471 7f6fff59e700 1 --2- 172.21.15.32:0/3464743073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f700010da00 0x7f700010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:24.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6ffdd9b700 1 -- 172.21.15.32:0/3464743073 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848279877 0 0) 0x7f6fec000ce0 con 0x7f70000d7ca0 2021-04-09T18:19:24.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6ffdd9b700 1 -- 172.21.15.32:0/3464743073 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe4002fe0 con 0x7f70000d7ca0 2021-04-09T18:19:24.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6ffdd9b700 1 -- 172.21.15.32:0/3464743073 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282519209 0 0) 0x7f6ff0000c60 con 0x7f700010d010 2021-04-09T18:19:24.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6ffdd9b700 1 -- 172.21.15.32:0/3464743073 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe40042c0 con 0x7f700010d010 2021-04-09T18:19:24.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6fff59e700 1 -- 172.21.15.32:0/3464743073 >> v1:172.21.15.86:6789/0 conn(0x7f700010d010 legacy=0x7f70000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:24.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6fff59e700 1 -- 172.21.15.32:0/3464743073 >> v1:172.21.15.32:6789/0 conn(0x7f70000d7ca0 legacy=0x7f70000d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:24.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6ffdd9b700 1 -- 172.21.15.32:0/3464743073 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3349373124 0 0) 0x7f6fec0010e0 con 0x7f70000d7ca0 2021-04-09T18:19:24.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6fff59e700 1 -- 172.21.15.32:0/3464743073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000112100 con 0x7f700010da00 2021-04-09T18:19:24.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6fff59e700 1 --2- 172.21.15.32:0/3464743073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f700010da00 0x7f700010de00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6ff8006e50 tx=0x7f6ff8009df0).ready entity=mon.2 client_cookie=6e4adbc2b019c719 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:24.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6ffdd9b700 1 -- 172.21.15.32:0/3464743073 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff800b510 con 0x7f700010da00 2021-04-09T18:19:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6ffdd9b700 1 -- 172.21.15.32:0/3464743073 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff8003030 con 0x7f700010da00 2021-04-09T18:19:24.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f6ffdd9b700 1 -- 172.21.15.32:0/3464743073 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff800b7d0 con 0x7f700010da00 2021-04-09T18:19:24.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f70051cc700 1 -- 172.21.15.32:0/3464743073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f700010da00 msgr2=0x7f700010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:24.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.472 7f70051cc700 1 --2- 172.21.15.32:0/3464743073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f700010da00 0x7f700010de00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6ff8006e50 tx=0x7f6ff8009df0).stop 2021-04-09T18:19:24.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.473 7f70051cc700 1 -- 172.21.15.32:0/3464743073 shutdown_connections 2021-04-09T18:19:24.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.473 7f70051cc700 1 --2- 172.21.15.32:0/3464743073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f700010da00 0x7f700010de00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:24.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.473 7f70051cc700 1 -- 172.21.15.32:0/3464743073 >> 172.21.15.32:0/3464743073 conn(0x7f70000ce5f0 msgr2=0x7f70000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:24.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.473 7f70051cc700 1 -- 172.21.15.32:0/3464743073 shutdown_connections 2021-04-09T18:19:24.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.473 7f70051cc700 1 -- 172.21.15.32:0/3464743073 wait complete. 2021-04-09T18:19:24.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.493 7f70051cc700 1 Processor -- start 2021-04-09T18:19:24.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.493 7f70051cc700 1 -- start start 2021-04-09T18:19:24.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f70051cc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70001ac380 0x7f700018a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:24.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f70051cc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70001ac780 0x7f700018a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f70051cc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7000183520 0x7f7000183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f70051cc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70001ace10 con 0x7f7000183520 2021-04-09T18:19:24.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f70051cc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7000189cf0 con 0x7f70001ac380 2021-04-09T18:19:24.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f70051cc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7000183f40 con 0x7f70001ac780 2021-04-09T18:19:24.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6fff59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7000183520 0x7f7000183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:24.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6fff59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7000183520 0x7f7000183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47282/0 (socket says 172.21.15.32:47282) 2021-04-09T18:19:24.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6fff59e700 1 -- 172.21.15.32:0/2607240152 learned_addr learned my addr 172.21.15.32:0/2607240152 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:24.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6ffe59c700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70001ac780 0x7f700018a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:24.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6ffed9d700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70001ac380 0x7f700018a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:24.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6fff59e700 1 -- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70001ac780 msgr2=0x7f700018a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:24.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6fff59e700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70001ac780 0x7f700018a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:24.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6fff59e700 1 -- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70001ac380 msgr2=0x7f700018a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:24.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6fff59e700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70001ac380 0x7f700018a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:24.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6fff59e700 1 -- 172.21.15.32:0/2607240152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000184160 con 0x7f7000183520 2021-04-09T18:19:24.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6ffe59c700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70001ac780 0x7f700018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:24.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.494 7f6fff59e700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7000183520 0x7f7000183920 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f6ff8009f80 tx=0x7f6ff800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:24.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.495 7f6ff77fe700 1 -- 172.21.15.32:0/2607240152 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff80097c0 con 0x7f7000183520 2021-04-09T18:19:24.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.495 7f6ff77fe700 1 -- 172.21.15.32:0/2607240152 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff80093d0 con 0x7f7000183520 2021-04-09T18:19:24.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.495 7f70051cc700 1 -- 172.21.15.32:0/2607240152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7000184160 con 0x7f7000183520 2021-04-09T18:19:24.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.495 7f6ff77fe700 1 -- 172.21.15.32:0/2607240152 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff8007430 con 0x7f7000183520 2021-04-09T18:19:24.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.495 7f70051cc700 1 -- 172.21.15.32:0/2607240152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7000184160 con 0x7f7000183520 2021-04-09T18:19:24.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.495 7f6ff77fe700 1 -- 172.21.15.32:0/2607240152 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6ff8007030 con 0x7f7000183520 2021-04-09T18:19:24.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.495 7f6ff77fe700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fdc018930 0x7f6fdc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:24.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.495 7f6ff77fe700 1 -- 172.21.15.32:0/2607240152 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6ff8032da0 con 0x7f7000183520 2021-04-09T18:19:24.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.495 7f6ffed9d700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fdc018930 0x7f6fdc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:24.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.496 7f6ffed9d700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fdc018930 0x7f6fdc01ae10 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:24.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.515 7f70051cc700 1 -- 172.21.15.32:0/2607240152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7000189cf0 con 0x7f7000183520 2021-04-09T18:19:24.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.519 7f6ff77fe700 1 -- 172.21.15.32:0/2607240152 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6ff80093b0 con 0x7f7000183520 2021-04-09T18:19:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.711 7f70051cc700 1 -- 172.21.15.32:0/2607240152 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7000189cf0 con 0x7f7000183520 2021-04-09T18:19:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.712 7f6ff77fe700 1 -- 172.21.15.32:0/2607240152 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f6ff80093b0 con 0x7f7000183520 2021-04-09T18:19:24.754 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:24.755 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:24.755 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 -- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fdc018930 msgr2=0x7f6fdc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:24.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fdc018930 0x7f6fdc01ae10 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:24.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 -- 172.21.15.32:0/2607240152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7000183520 msgr2=0x7f7000183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:24.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7000183520 0x7f7000183920 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f6ff8009f80 tx=0x7f6ff800bd10).stop 2021-04-09T18:19:24.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 -- 172.21.15.32:0/2607240152 shutdown_connections 2021-04-09T18:19:24.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fdc018930 0x7f6fdc01ae10 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:24.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7000183520 0x7f7000183920 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:24.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70001ac780 0x7f700018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:24.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 --2- 172.21.15.32:0/2607240152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70001ac380 0x7f700018a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:24.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 -- 172.21.15.32:0/2607240152 >> 172.21.15.32:0/2607240152 conn(0x7f700019b890 msgr2=0x7f70001a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:24.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 -- 172.21.15.32:0/2607240152 shutdown_connections 2021-04-09T18:19:24.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:24.714 7f70051cc700 1 -- 172.21.15.32:0/2607240152 wait complete. 2021-04-09T18:19:25.183 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:25.184 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:26.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:26.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.308 7fc6960a4700 1 Processor -- start 2021-04-09T18:19:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.308 7fc6960a4700 1 -- start start 2021-04-09T18:19:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc6960a4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc69010d010 0x7fc6900d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:26.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc6960a4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900d8490 0x7fc69010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:26.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc6960a4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69010d480 con 0x7fc6900d1da0 2021-04-09T18:19:26.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc6960a4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6900d8d30 con 0x7fc69010d010 2021-04-09T18:19:26.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc6960a4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6900d9220 con 0x7fc6900d8490 2021-04-09T18:19:26.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc69010d010 0x7fc6900d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:26.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc69010d010 0x7fc6900d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47290/0 (socket says 172.21.15.32:47290) 2021-04-09T18:19:26.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68f7fe700 1 -- 172.21.15.32:0/2841246980 learned_addr learned my addr 172.21.15.32:0/2841246980 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:26.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68ffff700 1 --2- 172.21.15.32:0/2841246980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900d8490 0x7fc69010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:26.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68f7fe700 1 -- 172.21.15.32:0/2841246980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900d8490 msgr2=0x7fc69010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:26.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68f7fe700 1 --2- 172.21.15.32:0/2841246980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900d8490 0x7fc69010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68f7fe700 1 -- 172.21.15.32:0/2841246980 >> v1:172.21.15.86:6789/0 conn(0x7fc6900d1da0 legacy=0x7fc6900d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:26.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68f7fe700 1 -- 172.21.15.32:0/2841246980 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69010e020 con 0x7fc69010d010 2021-04-09T18:19:26.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68f7fe700 1 --2- 172.21.15.32:0/2841246980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc69010d010 0x7fc6900d1780 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7fc678006700 tx=0x7fc678009b80).ready entity=mon.0 client_cookie=3d9ba8527621a1d7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:26.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68e7fc700 1 -- 172.21.15.32:0/2841246980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc67800c0c0 con 0x7fc69010d010 2021-04-09T18:19:26.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68e7fc700 1 -- 172.21.15.32:0/2841246980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67800c440 con 0x7fc69010d010 2021-04-09T18:19:26.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.309 7fc68e7fc700 1 -- 172.21.15.32:0/2841246980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc67800c920 con 0x7fc69010d010 2021-04-09T18:19:26.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.310 7fc6960a4700 1 -- 172.21.15.32:0/2841246980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc69010d010 msgr2=0x7fc6900d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:26.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.310 7fc6960a4700 1 --2- 172.21.15.32:0/2841246980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc69010d010 0x7fc6900d1780 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7fc678006700 tx=0x7fc678009b80).stop 2021-04-09T18:19:26.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.310 7fc6960a4700 1 -- 172.21.15.32:0/2841246980 shutdown_connections 2021-04-09T18:19:26.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.310 7fc6960a4700 1 --2- 172.21.15.32:0/2841246980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900d8490 0x7fc69010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.310 7fc6960a4700 1 --2- 172.21.15.32:0/2841246980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc69010d010 0x7fc6900d1780 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.310 7fc6960a4700 1 -- 172.21.15.32:0/2841246980 >> 172.21.15.32:0/2841246980 conn(0x7fc6900ce5f0 msgr2=0x7fc6900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:26.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.310 7fc6960a4700 1 -- 172.21.15.32:0/2841246980 shutdown_connections 2021-04-09T18:19:26.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.310 7fc6960a4700 1 -- 172.21.15.32:0/2841246980 wait complete. 2021-04-09T18:19:26.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.330 7fc6960a4700 1 Processor -- start 2021-04-09T18:19:26.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc6960a4700 1 -- start start 2021-04-09T18:19:26.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc6960a4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6900aa2b0 0x7fc690187620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:26.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc6960a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc690196f70 0x7fc690187c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:26.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc6960a4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900b7e90 0x7fc6900b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:26.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc6960a4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6900aa6f0 con 0x7fc690196f70 2021-04-09T18:19:26.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc6960a4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6901870a0 con 0x7fc6900aa2b0 2021-04-09T18:19:26.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc6960a4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6900b88b0 con 0x7fc6900b7e90 2021-04-09T18:19:26.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc68effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc690196f70 0x7fc690187c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:26.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc68effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc690196f70 0x7fc690187c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47298/0 (socket says 172.21.15.32:47298) 2021-04-09T18:19:26.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc68effd700 1 -- 172.21.15.32:0/1056023291 learned_addr learned my addr 172.21.15.32:0/1056023291 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc68f7fe700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6900aa2b0 0x7fc690187620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.331 7fc68ffff700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900b7e90 0x7fc6900b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc68effd700 1 -- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900b7e90 msgr2=0x7fc6900b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc68effd700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900b7e90 0x7fc6900b8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc68effd700 1 -- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6900aa2b0 msgr2=0x7fc690187620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc68effd700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6900aa2b0 0x7fc690187620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc68effd700 1 -- 172.21.15.32:0/1056023291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6900b8a90 con 0x7fc690196f70 2021-04-09T18:19:26.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc68effd700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc690196f70 0x7fc690187c40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fc68000b530 tx=0x7fc68000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:26.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc687fff700 1 -- 172.21.15.32:0/1056023291 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc68000bcc0 con 0x7fc690196f70 2021-04-09T18:19:26.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc687fff700 1 -- 172.21.15.32:0/1056023291 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc680006020 con 0x7fc690196f70 2021-04-09T18:19:26.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc687fff700 1 -- 172.21.15.32:0/1056023291 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6800054a0 con 0x7fc690196f70 2021-04-09T18:19:26.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6900b8a90 con 0x7fc690196f70 2021-04-09T18:19:26.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.332 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6900b8a90 con 0x7fc690196f70 2021-04-09T18:19:26.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.333 7fc687fff700 1 -- 172.21.15.32:0/1056023291 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc680006200 con 0x7fc690196f70 2021-04-09T18:19:26.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.333 7fc687fff700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc674018930 0x7fc67401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:26.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.333 7fc687fff700 1 -- 172.21.15.32:0/1056023291 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc680032a30 con 0x7fc690196f70 2021-04-09T18:19:26.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.333 7fc68f7fe700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc674018930 0x7fc67401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:26.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.333 7fc68f7fe700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc674018930 0x7fc67401adc0 crc :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:26.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.353 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc690182d10 con 0x7fc690196f70 2021-04-09T18:19:26.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.356 7fc687fff700 1 -- 172.21.15.32:0/1056023291 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc68000bcc0 con 0x7fc690196f70 2021-04-09T18:19:26.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.550 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc69010d480 con 0x7fc690196f70 2021-04-09T18:19:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.550 7fc687fff700 1 -- 172.21.15.32:0/1056023291 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fc680006020 con 0x7fc690196f70 2021-04-09T18:19:26.592 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:26.593 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:26.593 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc674018930 msgr2=0x7fc67401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:26.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc674018930 0x7fc67401adc0 crc :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc690196f70 msgr2=0x7fc690187c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc690196f70 0x7fc690187c40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fc68000b530 tx=0x7fc68000b8d0).stop 2021-04-09T18:19:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 shutdown_connections 2021-04-09T18:19:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc674018930 0x7fc67401adc0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc690196f70 0x7fc690187c40 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6900b7e90 0x7fc6900b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 --2- 172.21.15.32:0/1056023291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6900aa2b0 0x7fc690187620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 >> 172.21.15.32:0/1056023291 conn(0x7fc6901a13a0 msgr2=0x7fc690094bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 shutdown_connections 2021-04-09T18:19:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:26.553 7fc6960a4700 1 -- 172.21.15.32:0/1056023291 wait complete. 2021-04-09T18:19:27.011 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:27.011 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:28.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:28.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.133 7f651092c700 1 Processor -- start 2021-04-09T18:19:28.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.133 7f651092c700 1 -- start start 2021-04-09T18:19:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.133 7f651092c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f650c10d010 0x7f650c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.133 7f651092c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f650c0d1da0 0x7f650c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.133 7f651092c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650c10d480 con 0x7f650c0d82c0 2021-04-09T18:19:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.133 7f651092c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f650c0d8d30 con 0x7f650c0d1da0 2021-04-09T18:19:28.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f651092c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f650c0d9220 con 0x7f650c10d010 2021-04-09T18:19:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f650ae1d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f650c0d82c0 0x7f650c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56896/0 (socket says 172.21.15.32:56896) 2021-04-09T18:19:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f6509e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f650c0d1da0 0x7f650c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f650ae1d700 1 -- 172.21.15.32:0/333804277 learned_addr learned my addr 172.21.15.32:0/333804277 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:28.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f650a61c700 1 --2- 172.21.15.32:0/333804277 >> v2:172.21.15.86:3300/0 conn(0x7f650c10d010 0x7f650c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:28.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f650961a700 1 -- 172.21.15.32:0/333804277 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094855913 0 0) 0x7f6504000d30 con 0x7f650c0d82c0 2021-04-09T18:19:28.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f650961a700 1 -- 172.21.15.32:0/333804277 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f8002fe0 con 0x7f650c0d82c0 2021-04-09T18:19:28.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f6509e1b700 1 -- 172.21.15.32:0/333804277 >> v2:172.21.15.86:3300/0 conn(0x7f650c10d010 msgr2=0x7f650c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:28.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f6509e1b700 1 --2- 172.21.15.32:0/333804277 >> v2:172.21.15.86:3300/0 conn(0x7f650c10d010 0x7f650c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f6509e1b700 1 -- 172.21.15.32:0/333804277 >> v1:172.21.15.32:6789/0 conn(0x7f650c0d82c0 legacy=0x7f650c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f6509e1b700 1 -- 172.21.15.32:0/333804277 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c0d86f0 con 0x7f650c0d1da0 2021-04-09T18:19:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f6509e1b700 1 --2- 172.21.15.32:0/333804277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c0d1da0 0x7f650c0d7ca0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f64f4006650 tx=0x7f64f4009a80).ready entity=mon.0 client_cookie=485004d6599c38a1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f650961a700 1 -- 172.21.15.32:0/333804277 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64f40009b0 con 0x7f650c0d1da0 2021-04-09T18:19:28.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f650961a700 1 -- 172.21.15.32:0/333804277 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f400c0c0 con 0x7f650c0d1da0 2021-04-09T18:19:28.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f650961a700 1 -- 172.21.15.32:0/333804277 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64f400c510 con 0x7f650c0d1da0 2021-04-09T18:19:28.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f651092c700 1 -- 172.21.15.32:0/333804277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c0d1da0 msgr2=0x7f650c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:28.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.134 7f651092c700 1 --2- 172.21.15.32:0/333804277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c0d1da0 0x7f650c0d7ca0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f64f4006650 tx=0x7f64f4009a80).stop 2021-04-09T18:19:28.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.135 7f651092c700 1 -- 172.21.15.32:0/333804277 shutdown_connections 2021-04-09T18:19:28.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.135 7f651092c700 1 --2- 172.21.15.32:0/333804277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c0d1da0 0x7f650c0d7ca0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.135 7f651092c700 1 --2- 172.21.15.32:0/333804277 >> v2:172.21.15.86:3300/0 conn(0x7f650c10d010 0x7f650c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.135 7f651092c700 1 -- 172.21.15.32:0/333804277 >> 172.21.15.32:0/333804277 conn(0x7f650c0ce5f0 msgr2=0x7f650c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:28.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.135 7f651092c700 1 -- 172.21.15.32:0/333804277 shutdown_connections 2021-04-09T18:19:28.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.135 7f651092c700 1 -- 172.21.15.32:0/333804277 wait complete. 2021-04-09T18:19:28.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.155 7f651092c700 1 Processor -- start 2021-04-09T18:19:28.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f651092c700 1 -- start start 2021-04-09T18:19:28.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f651092c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f650c1abcb0 0x7f650c1ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:28.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f651092c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f650c191140 0x7f650c191540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:28.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f651092c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c191b60 0x7f650c1a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:28.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f651092c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c191f60 con 0x7f650c191b60 2021-04-09T18:19:28.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f651092c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c1ac6d0 con 0x7f650c191140 2021-04-09T18:19:28.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f651092c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f650c1aa5c0 con 0x7f650c1abcb0 2021-04-09T18:19:28.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f650ae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c191b60 0x7f650c1a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:28.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f650ae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c191b60 0x7f650c1a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47314/0 (socket says 172.21.15.32:47314) 2021-04-09T18:19:28.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f650ae1d700 1 -- 172.21.15.32:0/1538912340 learned_addr learned my addr 172.21.15.32:0/1538912340 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:28.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f650a61c700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f650c1abcb0 0x7f650c1ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:28.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.156 7f6509e1b700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f650c191140 0x7f650c191540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:28.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f650ae1d700 1 -- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f650c1abcb0 msgr2=0x7f650c1ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:28.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f650ae1d700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f650c1abcb0 0x7f650c1ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f650ae1d700 1 -- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f650c191140 msgr2=0x7f650c191540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:28.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f650ae1d700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f650c191140 0x7f650c191540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f650ae1d700 1 -- 172.21.15.32:0/1538912340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c1aa7e0 con 0x7f650c191b60 2021-04-09T18:19:28.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f650ae1d700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c191b60 0x7f650c1a9fa0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f6504009980 tx=0x7f650400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:28.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f6502ffd700 1 -- 172.21.15.32:0/1538912340 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f650400be10 con 0x7f650c191b60 2021-04-09T18:19:28.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f6502ffd700 1 -- 172.21.15.32:0/1538912340 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6504006030 con 0x7f650c191b60 2021-04-09T18:19:28.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f651092c700 1 -- 172.21.15.32:0/1538912340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650c1aa7e0 con 0x7f650c191b60 2021-04-09T18:19:28.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f6502ffd700 1 -- 172.21.15.32:0/1538912340 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65040064c0 con 0x7f650c191b60 2021-04-09T18:19:28.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.157 7f651092c700 1 -- 172.21.15.32:0/1538912340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650c1aa7e0 con 0x7f650c191b60 2021-04-09T18:19:28.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.158 7f6502ffd700 1 -- 172.21.15.32:0/1538912340 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6504006210 con 0x7f650c191b60 2021-04-09T18:19:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.158 7f6502ffd700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64f00188c0 0x7f64f001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.158 7f6502ffd700 1 -- 172.21.15.32:0/1538912340 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6504032b10 con 0x7f650c191b60 2021-04-09T18:19:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.158 7f650a61c700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64f00188c0 0x7f64f001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:28.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.158 7f650a61c700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64f00188c0 0x7f64f001adc0 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.178 7f651092c700 1 -- 172.21.15.32:0/1538912340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650c191f60 con 0x7f650c191b60 2021-04-09T18:19:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.181 7f6502ffd700 1 -- 172.21.15.32:0/1538912340 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6504000f90 con 0x7f650c191b60 2021-04-09T18:19:28.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.371 7f651092c700 1 -- 172.21.15.32:0/1538912340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f650c191f60 con 0x7f650c191b60 2021-04-09T18:19:28.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.372 7f6502ffd700 1 -- 172.21.15.32:0/1538912340 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f6504006210 con 0x7f650c191b60 2021-04-09T18:19:28.414 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:28.415 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:28.416 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 -- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64f00188c0 msgr2=0x7f64f001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64f00188c0 0x7f64f001adc0 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 -- 172.21.15.32:0/1538912340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c191b60 msgr2=0x7f650c1a9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c191b60 0x7f650c1a9fa0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f6504009980 tx=0x7f650400bcd0).stop 2021-04-09T18:19:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 -- 172.21.15.32:0/1538912340 shutdown_connections 2021-04-09T18:19:28.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64f00188c0 0x7f64f001adc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f650c191b60 0x7f650c1a9fa0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f650c191140 0x7f650c191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 --2- 172.21.15.32:0/1538912340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f650c1abcb0 0x7f650c1ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:28.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 -- 172.21.15.32:0/1538912340 >> 172.21.15.32:0/1538912340 conn(0x7f650c1837d0 msgr2=0x7f650c195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:28.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 -- 172.21.15.32:0/1538912340 shutdown_connections 2021-04-09T18:19:28.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:28.375 7f651092c700 1 -- 172.21.15.32:0/1538912340 wait complete. 2021-04-09T18:19:28.829 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:28.830 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:29.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:30.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47d2831700 1 Processor -- start 2021-04-09T18:19:30.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47d2831700 1 -- start start 2021-04-09T18:19:30.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47d2831700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47cc10d010 0x7f47cc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47d2831700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47cc10d480 con 0x7f47cc0d1da0 2021-04-09T18:19:30.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47d2831700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47cc10e1f0 con 0x7f47cc0d8490 2021-04-09T18:19:30.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47d2831700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47cc0d8d30 con 0x7f47cc10d010 2021-04-09T18:19:30.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47cbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47cc10d010 0x7f47cc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:30.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47c3fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f47cc0d1da0 0x7f47cc0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56910/0 (socket says 172.21.15.32:56910) 2021-04-09T18:19:30.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47cbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47cc10d010 0x7f47cc0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47318/0 (socket says 172.21.15.32:47318) 2021-04-09T18:19:30.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.950 7f47c3fff700 1 -- 172.21.15.32:0/616614594 learned_addr learned my addr 172.21.15.32:0/616614594 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:30.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47cbfff700 1 -- 172.21.15.32:0/616614594 >> v1:172.21.15.86:6789/0 conn(0x7f47cc0d8490 legacy=0x7f47cc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47cbfff700 1 -- 172.21.15.32:0/616614594 >> v1:172.21.15.32:6789/0 conn(0x7f47cc0d1da0 legacy=0x7f47cc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47cbfff700 1 -- 172.21.15.32:0/616614594 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc0d9220 con 0x7f47cc10d010 2021-04-09T18:19:30.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47cb7fe700 1 -- 172.21.15.32:0/616614594 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903477234 0 0) 0x7f47b4000ce0 con 0x7f47cc0d1da0 2021-04-09T18:19:30.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47cbfff700 1 --2- 172.21.15.32:0/616614594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc10d010 0x7f47cc0d1780 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f47bc006650 tx=0x7f47bc009a00).ready entity=mon.0 client_cookie=7fbca644759ee892 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:30.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47cb7fe700 1 -- 172.21.15.32:0/616614594 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47bc009df0 con 0x7f47cc10d010 2021-04-09T18:19:30.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47cb7fe700 1 -- 172.21.15.32:0/616614594 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47bc0009b0 con 0x7f47cc10d010 2021-04-09T18:19:30.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47cb7fe700 1 -- 172.21.15.32:0/616614594 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47bc00c2b0 con 0x7f47cc10d010 2021-04-09T18:19:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47d2831700 1 -- 172.21.15.32:0/616614594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc10d010 msgr2=0x7f47cc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:30.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.951 7f47d2831700 1 --2- 172.21.15.32:0/616614594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc10d010 0x7f47cc0d1780 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f47bc006650 tx=0x7f47bc009a00).stop 2021-04-09T18:19:30.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.952 7f47d2831700 1 -- 172.21.15.32:0/616614594 shutdown_connections 2021-04-09T18:19:30.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.952 7f47d2831700 1 --2- 172.21.15.32:0/616614594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc10d010 0x7f47cc0d1780 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:30.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.952 7f47d2831700 1 -- 172.21.15.32:0/616614594 >> 172.21.15.32:0/616614594 conn(0x7f47cc0ce5f0 msgr2=0x7f47cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:30.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.952 7f47d2831700 1 -- 172.21.15.32:0/616614594 shutdown_connections 2021-04-09T18:19:30.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.952 7f47d2831700 1 -- 172.21.15.32:0/616614594 wait complete. 2021-04-09T18:19:30.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.972 7f47d2831700 1 Processor -- start 2021-04-09T18:19:30.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.972 7f47d2831700 1 -- start start 2021-04-09T18:19:30.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d2831700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47cc0b1280 0x7f47cc08b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:30.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d2831700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47cc08b930 0x7f47cc091b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:30.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d2831700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc092130 0x7f47cc0981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:30.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d2831700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc092530 con 0x7f47cc092130 2021-04-09T18:19:30.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d2831700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc180940 con 0x7f47cc0b1280 2021-04-09T18:19:30.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d2831700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc180b60 con 0x7f47cc08b930 2021-04-09T18:19:30.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d0e4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc092130 0x7f47cc0981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:30.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d0e4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc092130 0x7f47cc0981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47328/0 (socket says 172.21.15.32:47328) 2021-04-09T18:19:30.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d0e4e700 1 -- 172.21.15.32:0/3512101642 learned_addr learned my addr 172.21.15.32:0/3512101642 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:30.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47cbfff700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47cc0b1280 0x7f47cc08b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:30.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47c3fff700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47cc08b930 0x7f47cc091b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:30.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d0e4e700 1 -- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47cc08b930 msgr2=0x7f47cc091b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:30.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d0e4e700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47cc08b930 0x7f47cc091b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:30.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d0e4e700 1 -- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47cc0b1280 msgr2=0x7f47cc08b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:30.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d0e4e700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47cc0b1280 0x7f47cc08b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:30.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d0e4e700 1 -- 172.21.15.32:0/3512101642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc0987e0 con 0x7f47cc092130 2021-04-09T18:19:30.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d0e4e700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc092130 0x7f47cc0981c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f47c4009940 tx=0x7f47c400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:30.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47c97fa700 1 -- 172.21.15.32:0/3512101642 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c4000f40 con 0x7f47cc092130 2021-04-09T18:19:30.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47c97fa700 1 -- 172.21.15.32:0/3512101642 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c400c2f0 con 0x7f47cc092130 2021-04-09T18:19:30.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.973 7f47d2831700 1 -- 172.21.15.32:0/3512101642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47cc0987e0 con 0x7f47cc092130 2021-04-09T18:19:30.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.974 7f47c97fa700 1 -- 172.21.15.32:0/3512101642 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c400c7d0 con 0x7f47cc092130 2021-04-09T18:19:30.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.974 7f47d2831700 1 -- 172.21.15.32:0/3512101642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47cc0987e0 con 0x7f47cc092130 2021-04-09T18:19:30.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.974 7f47c97fa700 1 -- 172.21.15.32:0/3512101642 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f47c4001220 con 0x7f47cc092130 2021-04-09T18:19:30.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.974 7f47c97fa700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b0018930 0x7f47b001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:30.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.974 7f47c97fa700 1 -- 172.21.15.32:0/3512101642 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f47c4032d80 con 0x7f47cc092130 2021-04-09T18:19:30.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.974 7f47cbfff700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b0018930 0x7f47b001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.975 7f47cbfff700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b0018930 0x7f47b001ae10 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:30.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.994 7f47d2831700 1 -- 172.21.15.32:0/3512101642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47cc17cae0 con 0x7f47cc092130 2021-04-09T18:19:30.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:29.998 7f47c97fa700 1 -- 172.21.15.32:0/3512101642 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f47c400be30 con 0x7f47cc092130 2021-04-09T18:19:30.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.189 7f47d2831700 1 -- 172.21.15.32:0/3512101642 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47cc10d480 con 0x7f47cc092130 2021-04-09T18:19:30.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.189 7f47c97fa700 1 -- 172.21.15.32:0/3512101642 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f47c4001220 con 0x7f47cc092130 2021-04-09T18:19:30.232 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:30.233 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:30.233 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:30.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 -- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b0018930 msgr2=0x7f47b001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:30.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b0018930 0x7f47b001ae10 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:30.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 -- 172.21.15.32:0/3512101642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc092130 msgr2=0x7f47cc0981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:30.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc092130 0x7f47cc0981c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f47c4009940 tx=0x7f47c400bcf0).stop 2021-04-09T18:19:30.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 -- 172.21.15.32:0/3512101642 shutdown_connections 2021-04-09T18:19:30.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47b0018930 0x7f47b001ae10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:30.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47cc092130 0x7f47cc0981c0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:30.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47cc08b930 0x7f47cc091b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:30.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 --2- 172.21.15.32:0/3512101642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47cc0b1280 0x7f47cc08b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:30.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.191 7f47d2831700 1 -- 172.21.15.32:0/3512101642 >> 172.21.15.32:0/3512101642 conn(0x7f47cc17e6e0 msgr2=0x7f47cc094dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:30.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.192 7f47d2831700 1 -- 172.21.15.32:0/3512101642 shutdown_connections 2021-04-09T18:19:30.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:30.192 7f47d2831700 1 -- 172.21.15.32:0/3512101642 wait complete. 2021-04-09T18:19:30.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:30.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:31.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:31.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.763 7fbd8a810700 1 Processor -- start 2021-04-09T18:19:31.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.763 7fbd8a810700 1 -- start start 2021-04-09T18:19:31.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.763 7fbd8a810700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd8410d010 0x7fbd840d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:31.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.763 7fbd8a810700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd840d1da0 0x7fbd840d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:31.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.763 7fbd8a810700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8410d480 con 0x7fbd840d82c0 2021-04-09T18:19:31.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.763 7fbd8a810700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd840d9220 con 0x7fbd8410d010 2021-04-09T18:19:31.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.763 7fbd8a810700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840d86c0 con 0x7fbd840d1da0 2021-04-09T18:19:31.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd88e2d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbd840d82c0 0x7fbd8410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56926/0 (socket says 172.21.15.32:56926) 2021-04-09T18:19:31.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd88e2d700 1 -- 172.21.15.32:0/1628116631 learned_addr learned my addr 172.21.15.32:0/1628116631 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:31.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd83fff700 1 --2- 172.21.15.32:0/1628116631 >> v2:172.21.15.86:3300/0 conn(0x7fbd8410d010 0x7fbd840d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:31.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd7bfff700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd840d1da0 0x7fbd840d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763937501 0 0) 0x7fbd7c000f90 con 0x7fbd840d82c0 2021-04-09T18:19:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd70002fe0 con 0x7fbd840d82c0 2021-04-09T18:19:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3301366847 0 0) 0x7fbd7c001460 con 0x7fbd840d82c0 2021-04-09T18:19:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd840d1da0 msgr2=0x7fbd840d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd840d1da0 0x7fbd840d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 >> v2:172.21.15.86:3300/0 conn(0x7fbd8410d010 msgr2=0x7fbd840d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 --2- 172.21.15.32:0/1628116631 >> v2:172.21.15.86:3300/0 conn(0x7fbd8410d010 0x7fbd840d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8410e1f0 con 0x7fbd840d82c0 2021-04-09T18:19:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fbd7c001980 con 0x7fbd840d82c0 2021-04-09T18:19:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 >> v1:172.21.15.32:6789/0 conn(0x7fbd840d82c0 legacy=0x7fbd8410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd70003a70 0x7fbd70005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd7004a7b0 0x7fbd7004ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd7004d5b0 0x7fbd7004faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:31.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd700502f0 con 0x7fbd7004a7b0 2021-04-09T18:19:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd70050510 con 0x7fbd70003a70 2021-04-09T18:19:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd70050730 con 0x7fbd7004d5b0 2021-04-09T18:19:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd7bfff700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd7004a7b0 0x7fbd7004ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd83fff700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd70003a70 0x7fbd70005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:31.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.764 7fbd88e2d700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd7004d5b0 0x7fbd7004faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:31.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd7bfff700 1 -- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd7004d5b0 msgr2=0x7fbd7004faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:31.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd7bfff700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd7004d5b0 0x7fbd7004faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:31.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd7bfff700 1 -- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd70003a70 msgr2=0x7fbd70005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:31.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd7bfff700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd70003a70 0x7fbd70005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd7bfff700 1 -- 172.21.15.32:0/1628116631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd700509d0 con 0x7fbd7004a7b0 2021-04-09T18:19:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd88e2d700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd7004d5b0 0x7fbd7004faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd7bfff700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd7004a7b0 0x7fbd7004ccb0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7fbd6c006ac0 tx=0x7fbd6c009ab0).ready entity=mon.0 client_cookie=ed9dfba68774b780 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:31.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd6c00c0c0 con 0x7fbd7004a7b0 2021-04-09T18:19:31.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c00c3c0 con 0x7fbd7004a7b0 2021-04-09T18:19:31.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd837fe700 1 -- 172.21.15.32:0/1628116631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd6c00c8a0 con 0x7fbd7004a7b0 2021-04-09T18:19:31.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd8a810700 1 -- 172.21.15.32:0/1628116631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd7004a7b0 msgr2=0x7fbd7004ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:31.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd8a810700 1 --2- 172.21.15.32:0/1628116631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd7004a7b0 0x7fbd7004ccb0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7fbd6c006ac0 tx=0x7fbd6c009ab0).stop 2021-04-09T18:19:31.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.765 7fbd88e2d700 1 -- 172.21.15.32:0/1628116631 reap_dead start 2021-04-09T18:19:31.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.766 7fbd8a810700 1 -- 172.21.15.32:0/1628116631 shutdown_connections 2021-04-09T18:19:31.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.766 7fbd8a810700 1 -- 172.21.15.32:0/1628116631 >> 172.21.15.32:0/1628116631 conn(0x7fbd840ce5f0 msgr2=0x7fbd840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:31.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.766 7fbd8a810700 1 -- 172.21.15.32:0/1628116631 shutdown_connections 2021-04-09T18:19:31.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.766 7fbd8a810700 1 -- 172.21.15.32:0/1628116631 wait complete. 2021-04-09T18:19:31.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.786 7fbd8a810700 1 Processor -- start 2021-04-09T18:19:31.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.786 7fbd8a810700 1 -- start start 2021-04-09T18:19:31.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd8a810700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd841ac380 0x7fbd8418a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:31.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd8a810700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd841ac780 0x7fbd8418a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:31.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd8a810700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd84183520 0x7fbd84183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:31.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd8a810700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd841ace10 con 0x7fbd841ac380 2021-04-09T18:19:31.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd8a810700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd84189cf0 con 0x7fbd841ac780 2021-04-09T18:19:31.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd8a810700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd84183f40 con 0x7fbd84183520 2021-04-09T18:19:31.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd83fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd841ac380 0x7fbd8418a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:31.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd83fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd841ac380 0x7fbd8418a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47344/0 (socket says 172.21.15.32:47344) 2021-04-09T18:19:31.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd83fff700 1 -- 172.21.15.32:0/2108386147 learned_addr learned my addr 172.21.15.32:0/2108386147 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:31.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd88e2d700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd84183520 0x7fbd84183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:31.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd7bfff700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd841ac780 0x7fbd8418a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:31.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd83fff700 1 -- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd84183520 msgr2=0x7fbd84183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:31.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd83fff700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd84183520 0x7fbd84183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:31.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd83fff700 1 -- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd841ac780 msgr2=0x7fbd8418a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:31.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd83fff700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd841ac780 0x7fbd8418a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:31.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd83fff700 1 -- 172.21.15.32:0/2108386147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd84184160 con 0x7fbd841ac380 2021-04-09T18:19:31.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd83fff700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd841ac380 0x7fbd8418a230 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7fbd7400b560 tx=0x7fbd7400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd817fa700 1 -- 172.21.15.32:0/2108386147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd7400bcf0 con 0x7fbd841ac380 2021-04-09T18:19:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd817fa700 1 -- 172.21.15.32:0/2108386147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd74000c90 con 0x7fbd841ac380 2021-04-09T18:19:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd84184160 con 0x7fbd841ac380 2021-04-09T18:19:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd817fa700 1 -- 172.21.15.32:0/2108386147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd74002220 con 0x7fbd841ac380 2021-04-09T18:19:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.787 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd84184160 con 0x7fbd841ac380 2021-04-09T18:19:31.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.788 7fbd817fa700 1 -- 172.21.15.32:0/2108386147 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbd740052e0 con 0x7fbd841ac380 2021-04-09T18:19:31.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.788 7fbd817fa700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd68018930 0x7fbd6801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:31.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.788 7fbd817fa700 1 -- 172.21.15.32:0/2108386147 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbd74032a20 con 0x7fbd841ac380 2021-04-09T18:19:31.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.788 7fbd7bfff700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd68018930 0x7fbd6801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:31.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.789 7fbd7bfff700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd68018930 0x7fbd6801ae10 crc :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:31.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.807 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd84182d10 con 0x7fbd841ac380 2021-04-09T18:19:31.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:31.811 7fbd817fa700 1 -- 172.21.15.32:0/2108386147 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbd74000c90 con 0x7fbd841ac380 2021-04-09T18:19:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.003 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd84182d10 con 0x7fbd841ac380 2021-04-09T18:19:32.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.004 7fbd817fa700 1 -- 172.21.15.32:0/2108386147 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fbd74006b00 con 0x7fbd841ac380 2021-04-09T18:19:32.046 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:32.047 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:32.048 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd68018930 msgr2=0x7fbd6801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd68018930 0x7fbd6801ae10 crc :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd841ac380 msgr2=0x7fbd8418a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd841ac380 0x7fbd8418a230 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7fbd7400b560 tx=0x7fbd7400b900).stop 2021-04-09T18:19:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 shutdown_connections 2021-04-09T18:19:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd68018930 0x7fbd6801ae10 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd84183520 0x7fbd84183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd841ac780 0x7fbd8418a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 --2- 172.21.15.32:0/2108386147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd841ac380 0x7fbd8418a230 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 >> 172.21.15.32:0/2108386147 conn(0x7fbd8419b890 msgr2=0x7fbd841a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 shutdown_connections 2021-04-09T18:19:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:32.006 7fbd8a810700 1 -- 172.21.15.32:0/2108386147 wait complete. 2021-04-09T18:19:32.468 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:32.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:33.476 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:33.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.596 7f58c7035700 1 Processor -- start 2021-04-09T18:19:33.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.596 7f58c7035700 1 -- start start 2021-04-09T18:19:33.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.596 7f58c7035700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c010d010 0x7f58c00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:33.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.596 7f58c7035700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f58c00d1da0 0x7f58c00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:33.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.596 7f58c7035700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58c00d82c0 0x7f58c010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:33.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.596 7f58c7035700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c010e780 con 0x7f58c00d82c0 2021-04-09T18:19:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.596 7f58c7035700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c00d8d30 con 0x7f58c00d1da0 2021-04-09T18:19:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.596 7f58c7035700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c00d9220 con 0x7f58c010d010 2021-04-09T18:19:33.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c5652700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58c00d82c0 0x7f58c010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:33.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c5652700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58c00d82c0 0x7f58c010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47352/0 (socket says 172.21.15.32:47352) 2021-04-09T18:19:33.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c5652700 1 -- 172.21.15.32:0/4073969150 learned_addr learned my addr 172.21.15.32:0/4073969150 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:33.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c4e51700 1 --2- 172.21.15.32:0/4073969150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c010d010 0x7f58c00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58bffff700 1 --2- 172.21.15.32:0/4073969150 >> v2:172.21.15.86:3300/0 conn(0x7f58c00d1da0 0x7f58c00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c5652700 1 -- 172.21.15.32:0/4073969150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c010d010 msgr2=0x7f58c00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:33.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c5652700 1 --2- 172.21.15.32:0/4073969150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c010d010 0x7f58c00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c5652700 1 -- 172.21.15.32:0/4073969150 >> v2:172.21.15.86:3300/0 conn(0x7f58c00d1da0 msgr2=0x7f58c00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:33.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c5652700 1 --2- 172.21.15.32:0/4073969150 >> v2:172.21.15.86:3300/0 conn(0x7f58c00d1da0 0x7f58c00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c5652700 1 -- 172.21.15.32:0/4073969150 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c010d480 con 0x7f58c00d82c0 2021-04-09T18:19:33.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c4e51700 1 --2- 172.21.15.32:0/4073969150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c010d010 0x7f58c00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:33.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58bffff700 1 --2- 172.21.15.32:0/4073969150 >> v2:172.21.15.86:3300/0 conn(0x7f58c00d1da0 0x7f58c00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:33.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c5652700 1 --2- 172.21.15.32:0/4073969150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c00d82c0 0x7f58c010da00 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f58b8006f00 tx=0x7f58b8009f10).ready entity=mon.0 client_cookie=a85a9f8189d419d0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:33.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58bf7fe700 1 -- 172.21.15.32:0/4073969150 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58b800b3b0 con 0x7f58c00d82c0 2021-04-09T18:19:33.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58bf7fe700 1 -- 172.21.15.32:0/4073969150 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b800b730 con 0x7f58c00d82c0 2021-04-09T18:19:33.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58bf7fe700 1 -- 172.21.15.32:0/4073969150 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58b800bc10 con 0x7f58c00d82c0 2021-04-09T18:19:33.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c7035700 1 -- 172.21.15.32:0/4073969150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c00d82c0 msgr2=0x7f58c010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:33.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.597 7f58c7035700 1 --2- 172.21.15.32:0/4073969150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c00d82c0 0x7f58c010da00 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f58b8006f00 tx=0x7f58b8009f10).stop 2021-04-09T18:19:33.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.598 7f58c7035700 1 -- 172.21.15.32:0/4073969150 shutdown_connections 2021-04-09T18:19:33.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.598 7f58c7035700 1 --2- 172.21.15.32:0/4073969150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c00d82c0 0x7f58c010da00 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.598 7f58c7035700 1 --2- 172.21.15.32:0/4073969150 >> v2:172.21.15.86:3300/0 conn(0x7f58c00d1da0 0x7f58c00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.598 7f58c7035700 1 --2- 172.21.15.32:0/4073969150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c010d010 0x7f58c00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.598 7f58c7035700 1 -- 172.21.15.32:0/4073969150 >> 172.21.15.32:0/4073969150 conn(0x7f58c00ce5f0 msgr2=0x7f58c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:33.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.598 7f58c7035700 1 -- 172.21.15.32:0/4073969150 shutdown_connections 2021-04-09T18:19:33.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.598 7f58c7035700 1 -- 172.21.15.32:0/4073969150 wait complete. 2021-04-09T18:19:33.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.618 7f58c7035700 1 Processor -- start 2021-04-09T18:19:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.618 7f58c7035700 1 -- start start 2021-04-09T18:19:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c7035700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c00be8b0 0x7f58c00becb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c7035700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58c008ff60 0x7f58c0090360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c7035700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c018b820 0x7f58c018bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c7035700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c018c240 con 0x7f58c018b820 2021-04-09T18:19:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c7035700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c00b0860 con 0x7f58c008ff60 2021-04-09T18:19:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c7035700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c00b0a80 con 0x7f58c00be8b0 2021-04-09T18:19:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c5652700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c018b820 0x7f58c018bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c5652700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c018b820 0x7f58c018bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47362/0 (socket says 172.21.15.32:47362) 2021-04-09T18:19:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c5652700 1 -- 172.21.15.32:0/4051694137 learned_addr learned my addr 172.21.15.32:0/4051694137 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c4e51700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c00be8b0 0x7f58c00becb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58bffff700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58c008ff60 0x7f58c0090360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.619 7f58c5652700 1 -- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c00be8b0 msgr2=0x7f58c00becb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58c5652700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c00be8b0 0x7f58c00becb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58c5652700 1 -- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58c008ff60 msgr2=0x7f58c0090360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:33.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58c5652700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58c008ff60 0x7f58c0090360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58c5652700 1 -- 172.21.15.32:0/4051694137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c00c3660 con 0x7f58c018b820 2021-04-09T18:19:33.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58bffff700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58c008ff60 0x7f58c0090360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:33.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58c5652700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c018b820 0x7f58c018bc20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f58b800b8c0 tx=0x7f58b800bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:33.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58bd7fa700 1 -- 172.21.15.32:0/4051694137 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58b8000f90 con 0x7f58c018b820 2021-04-09T18:19:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58bd7fa700 1 -- 172.21.15.32:0/4051694137 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b800bda0 con 0x7f58c018b820 2021-04-09T18:19:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58bd7fa700 1 -- 172.21.15.32:0/4051694137 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58b8001460 con 0x7f58c018b820 2021-04-09T18:19:33.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58c7035700 1 -- 172.21.15.32:0/4051694137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c00c3660 con 0x7f58c018b820 2021-04-09T18:19:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.620 7f58c7035700 1 -- 172.21.15.32:0/4051694137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c00c3660 con 0x7f58c018b820 2021-04-09T18:19:33.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.621 7f58bd7fa700 1 -- 172.21.15.32:0/4051694137 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f58b8001270 con 0x7f58c018b820 2021-04-09T18:19:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.621 7f58bd7fa700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58a4018930 0x7f58a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.621 7f58bd7fa700 1 -- 172.21.15.32:0/4051694137 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f58b8032dd0 con 0x7f58c018b820 2021-04-09T18:19:33.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.621 7f58c4e51700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58a4018930 0x7f58a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:33.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.621 7f58c4e51700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58a4018930 0x7f58a401ae10 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.640 7f58c7035700 1 -- 172.21.15.32:0/4051694137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c0182d70 con 0x7f58c018b820 2021-04-09T18:19:33.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.644 7f58bd7fa700 1 -- 172.21.15.32:0/4051694137 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f58b8000cc0 con 0x7f58c018b820 2021-04-09T18:19:33.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.835 7f58c7035700 1 -- 172.21.15.32:0/4051694137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58c010d480 con 0x7f58c018b820 2021-04-09T18:19:33.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.836 7f58bd7fa700 1 -- 172.21.15.32:0/4051694137 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f58b800bda0 con 0x7f58c018b820 2021-04-09T18:19:33.883 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:33.884 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:33.885 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 -- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58a4018930 msgr2=0x7f58a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58a4018930 0x7f58a401ae10 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 -- 172.21.15.32:0/4051694137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c018b820 msgr2=0x7f58c018bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c018b820 0x7f58c018bc20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f58b800b8c0 tx=0x7f58b800bc60).stop 2021-04-09T18:19:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 -- 172.21.15.32:0/4051694137 shutdown_connections 2021-04-09T18:19:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58a4018930 0x7f58a401ae10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58c018b820 0x7f58c018bc20 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58c008ff60 0x7f58c0090360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 --2- 172.21.15.32:0/4051694137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58c00be8b0 0x7f58c00becb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 -- 172.21.15.32:0/4051694137 >> 172.21.15.32:0/4051694137 conn(0x7f58c019b430 msgr2=0x7f58c019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.838 7f58c7035700 1 -- 172.21.15.32:0/4051694137 shutdown_connections 2021-04-09T18:19:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:33.839 7f58c7035700 1 -- 172.21.15.32:0/4051694137 wait complete. 2021-04-09T18:19:34.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:34.295 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:35.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:35.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.421 7f71464ff700 1 Processor -- start 2021-04-09T18:19:35.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.421 7f71464ff700 1 -- start start 2021-04-09T18:19:35.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f71464ff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f714010d010 0x7f71400d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:35.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f71464ff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71400d8490 0x7f714010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:35.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f71464ff700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714010d480 con 0x7f71400d1da0 2021-04-09T18:19:35.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f71464ff700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71400d8d30 con 0x7f714010d010 2021-04-09T18:19:35.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f71464ff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71400d9220 con 0x7f71400d8490 2021-04-09T18:19:35.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f7137fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f71400d1da0 0x7f71400d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56956/0 (socket says 172.21.15.32:56956) 2021-04-09T18:19:35.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f7137fff700 1 -- 172.21.15.32:0/508962819 learned_addr learned my addr 172.21.15.32:0/508962819 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:35.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f7144b1c700 1 --2- 172.21.15.32:0/508962819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71400d8490 0x7f714010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:35.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f713ffff700 1 --2- 172.21.15.32:0/508962819 >> v2:172.21.15.86:3300/0 conn(0x7f714010d010 0x7f71400d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:35.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f713f7fe700 1 -- 172.21.15.32:0/508962819 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127608059 0 0) 0x7f7128000ce0 con 0x7f71400d1da0 2021-04-09T18:19:35.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f713f7fe700 1 -- 172.21.15.32:0/508962819 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f712c002fe0 con 0x7f71400d1da0 2021-04-09T18:19:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f7144b1c700 1 -- 172.21.15.32:0/508962819 >> v2:172.21.15.86:3300/0 conn(0x7f714010d010 msgr2=0x7f71400d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f7144b1c700 1 --2- 172.21.15.32:0/508962819 >> v2:172.21.15.86:3300/0 conn(0x7f714010d010 0x7f71400d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f7144b1c700 1 -- 172.21.15.32:0/508962819 >> v1:172.21.15.32:6789/0 conn(0x7f71400d1da0 legacy=0x7f71400d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f713f7fe700 1 -- 172.21.15.32:0/508962819 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2015276100 0 0) 0x7f7128000ed0 con 0x7f71400d1da0 2021-04-09T18:19:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f7144b1c700 1 -- 172.21.15.32:0/508962819 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714010e020 con 0x7f71400d8490 2021-04-09T18:19:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f713ffff700 1 --2- 172.21.15.32:0/508962819 >> v2:172.21.15.86:3300/0 conn(0x7f714010d010 0x7f71400d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:19:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.422 7f7144b1c700 1 --2- 172.21.15.32:0/508962819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71400d8490 0x7f714010da00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7138006e50 tx=0x7f7138009e70).ready entity=mon.2 client_cookie=510c2776a5b6657c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:35.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.423 7f713f7fe700 1 -- 172.21.15.32:0/508962819 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7138006a20 con 0x7f71400d8490 2021-04-09T18:19:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.423 7f713f7fe700 1 -- 172.21.15.32:0/508962819 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7138006790 con 0x7f71400d8490 2021-04-09T18:19:35.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.423 7f713f7fe700 1 -- 172.21.15.32:0/508962819 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f713800b700 con 0x7f71400d8490 2021-04-09T18:19:35.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.423 7f71464ff700 1 -- 172.21.15.32:0/508962819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71400d8490 msgr2=0x7f714010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:35.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.423 7f71464ff700 1 --2- 172.21.15.32:0/508962819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71400d8490 0x7f714010da00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7138006e50 tx=0x7f7138009e70).stop 2021-04-09T18:19:35.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.423 7f71464ff700 1 -- 172.21.15.32:0/508962819 shutdown_connections 2021-04-09T18:19:35.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.423 7f71464ff700 1 --2- 172.21.15.32:0/508962819 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71400d8490 0x7f714010da00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.423 7f71464ff700 1 --2- 172.21.15.32:0/508962819 >> v2:172.21.15.86:3300/0 conn(0x7f714010d010 0x7f71400d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.423 7f71464ff700 1 -- 172.21.15.32:0/508962819 >> 172.21.15.32:0/508962819 conn(0x7f71400ce5f0 msgr2=0x7f71400d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:35.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.424 7f71464ff700 1 -- 172.21.15.32:0/508962819 shutdown_connections 2021-04-09T18:19:35.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.424 7f71464ff700 1 -- 172.21.15.32:0/508962819 wait complete. 2021-04-09T18:19:35.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f71464ff700 1 Processor -- start 2021-04-09T18:19:35.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f71464ff700 1 -- start start 2021-04-09T18:19:35.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f71464ff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71401aca20 0x7f71401ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:35.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f71464ff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f714018bd40 0x7f714018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:35.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f71464ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f714008fba0 0x7f714008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:35.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f71464ff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714018c7e0 con 0x7f714008fba0 2021-04-09T18:19:35.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f71464ff700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714018ca30 con 0x7f71401aca20 2021-04-09T18:19:35.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f71464ff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71400905f0 con 0x7f714018bd40 2021-04-09T18:19:35.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f7144b1c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f714008fba0 0x7f714008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:35.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f7144b1c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f714008fba0 0x7f714008ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47374/0 (socket says 172.21.15.32:47374) 2021-04-09T18:19:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f7144b1c700 1 -- 172.21.15.32:0/648873882 learned_addr learned my addr 172.21.15.32:0/648873882 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.444 7f713ffff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71401aca20 0x7f71401ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f7137fff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f714018bd40 0x7f714018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f7144b1c700 1 -- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f714018bd40 msgr2=0x7f714018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:35.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f7144b1c700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f714018bd40 0x7f714018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f7144b1c700 1 -- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71401aca20 msgr2=0x7f71401ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:35.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f7144b1c700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71401aca20 0x7f71401ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f7144b1c700 1 -- 172.21.15.32:0/648873882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71400b4300 con 0x7f714008fba0 2021-04-09T18:19:35.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f7144b1c700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f714008fba0 0x7f714008ffa0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f7138006e60 tx=0x7f713800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:35.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f713d7fa700 1 -- 172.21.15.32:0/648873882 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7138009130 con 0x7f714008fba0 2021-04-09T18:19:35.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f713d7fa700 1 -- 172.21.15.32:0/648873882 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71380093b0 con 0x7f714008fba0 2021-04-09T18:19:35.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f71464ff700 1 -- 172.21.15.32:0/648873882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71400b4300 con 0x7f714008fba0 2021-04-09T18:19:35.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f713d7fa700 1 -- 172.21.15.32:0/648873882 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71380013b0 con 0x7f714008fba0 2021-04-09T18:19:35.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.445 7f71464ff700 1 -- 172.21.15.32:0/648873882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71400b4300 con 0x7f714008fba0 2021-04-09T18:19:35.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.446 7f713d7fa700 1 -- 172.21.15.32:0/648873882 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7138009130 con 0x7f714008fba0 2021-04-09T18:19:35.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.446 7f713d7fa700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71240188c0 0x7f712401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:35.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.446 7f713d7fa700 1 -- 172.21.15.32:0/648873882 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7138032ae0 con 0x7f714008fba0 2021-04-09T18:19:35.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.446 7f713ffff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71240188c0 0x7f712401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:35.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.447 7f713ffff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71240188c0 0x7f712401adc0 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:35.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.465 7f71464ff700 1 -- 172.21.15.32:0/648873882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7140182d70 con 0x7f714008fba0 2021-04-09T18:19:35.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.469 7f713d7fa700 1 -- 172.21.15.32:0/648873882 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f713800be30 con 0x7f714008fba0 2021-04-09T18:19:35.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.660 7f71464ff700 1 -- 172.21.15.32:0/648873882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f714018c760 con 0x7f714008fba0 2021-04-09T18:19:35.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.661 7f713d7fa700 1 -- 172.21.15.32:0/648873882 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f7138009130 con 0x7f714008fba0 2021-04-09T18:19:35.704 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:35.705 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:35.705 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:35.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 -- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71240188c0 msgr2=0x7f712401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:35.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71240188c0 0x7f712401adc0 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 -- 172.21.15.32:0/648873882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f714008fba0 msgr2=0x7f714008ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:35.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f714008fba0 0x7f714008ffa0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f7138006e60 tx=0x7f713800bcf0).stop 2021-04-09T18:19:35.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 -- 172.21.15.32:0/648873882 shutdown_connections 2021-04-09T18:19:35.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71240188c0 0x7f712401adc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f714008fba0 0x7f714008ffa0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f714018bd40 0x7f714018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 --2- 172.21.15.32:0/648873882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71401aca20 0x7f71401ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:35.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 -- 172.21.15.32:0/648873882 >> 172.21.15.32:0/648873882 conn(0x7f714019b460 msgr2=0x7f714019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:35.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 -- 172.21.15.32:0/648873882 shutdown_connections 2021-04-09T18:19:35.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:35.663 7f71464ff700 1 -- 172.21.15.32:0/648873882 wait complete. 2021-04-09T18:19:35.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:19:35.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:19:36.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:36.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:37.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:37.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.255 7f14dba81700 1 Processor -- start 2021-04-09T18:19:37.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.255 7f14dba81700 1 -- start start 2021-04-09T18:19:37.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.256 7f14dba81700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f14d40d7ca0 0x7f14d40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:37.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.256 7f14dba81700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d410da00 0x7f14d410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.256 7f14dba81700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d410d480 con 0x7f14d410d010 2021-04-09T18:19:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.256 7f14dba81700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d40d9220 con 0x7f14d40d7ca0 2021-04-09T18:19:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.256 7f14dba81700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d4112100 con 0x7f14d410da00 2021-04-09T18:19:37.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.256 7f14d989d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f14d410d010 0x7f14d40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54894/0 (socket says 172.21.15.32:54894) 2021-04-09T18:19:37.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.256 7f14d989d700 1 -- 172.21.15.32:0/3079149273 learned_addr learned my addr 172.21.15.32:0/3079149273 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:37.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.256 7f14d909c700 1 --2- 172.21.15.32:0/3079149273 >> v2:172.21.15.86:3300/0 conn(0x7f14d40d7ca0 0x7f14d40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:37.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.256 7f14da09e700 1 --2- 172.21.15.32:0/3079149273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d410da00 0x7f14d410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:37.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14d889b700 1 -- 172.21.15.32:0/3079149273 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103283992 0 0) 0x7f14c40010c0 con 0x7f14d410d010 2021-04-09T18:19:37.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14d889b700 1 -- 172.21.15.32:0/3079149273 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14bc002fe0 con 0x7f14d410d010 2021-04-09T18:19:37.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14da09e700 1 -- 172.21.15.32:0/3079149273 >> v2:172.21.15.86:3300/0 conn(0x7f14d40d7ca0 msgr2=0x7f14d40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:37.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14da09e700 1 --2- 172.21.15.32:0/3079149273 >> v2:172.21.15.86:3300/0 conn(0x7f14d40d7ca0 0x7f14d40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14da09e700 1 -- 172.21.15.32:0/3079149273 >> v1:172.21.15.86:6789/0 conn(0x7f14d410d010 legacy=0x7f14d40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:37.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14da09e700 1 -- 172.21.15.32:0/3079149273 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d40d86c0 con 0x7f14d410da00 2021-04-09T18:19:37.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14d909c700 1 --2- 172.21.15.32:0/3079149273 >> v2:172.21.15.86:3300/0 conn(0x7f14d40d7ca0 0x7f14d40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:19:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14da09e700 1 --2- 172.21.15.32:0/3079149273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d410da00 0x7f14d410de00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f14cc006ad0 tx=0x7f14cc009ad0).ready entity=mon.2 client_cookie=b9c2355f937617d6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14d889b700 1 -- 172.21.15.32:0/3079149273 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14cc0022e0 con 0x7f14d410da00 2021-04-09T18:19:37.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14d889b700 1 -- 172.21.15.32:0/3079149273 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc006410 con 0x7f14d410da00 2021-04-09T18:19:37.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14d889b700 1 -- 172.21.15.32:0/3079149273 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14cc002620 con 0x7f14d410da00 2021-04-09T18:19:37.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14dba81700 1 -- 172.21.15.32:0/3079149273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d410da00 msgr2=0x7f14d410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:37.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.257 7f14dba81700 1 --2- 172.21.15.32:0/3079149273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d410da00 0x7f14d410de00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f14cc006ad0 tx=0x7f14cc009ad0).stop 2021-04-09T18:19:37.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.258 7f14dba81700 1 -- 172.21.15.32:0/3079149273 shutdown_connections 2021-04-09T18:19:37.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.258 7f14dba81700 1 --2- 172.21.15.32:0/3079149273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d410da00 0x7f14d410de00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.258 7f14dba81700 1 --2- 172.21.15.32:0/3079149273 >> v2:172.21.15.86:3300/0 conn(0x7f14d40d7ca0 0x7f14d40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.258 7f14dba81700 1 -- 172.21.15.32:0/3079149273 >> 172.21.15.32:0/3079149273 conn(0x7f14d40ce5f0 msgr2=0x7f14d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:37.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.258 7f14dba81700 1 -- 172.21.15.32:0/3079149273 shutdown_connections 2021-04-09T18:19:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.258 7f14dba81700 1 -- 172.21.15.32:0/3079149273 wait complete. 2021-04-09T18:19:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.278 7f14dba81700 1 Processor -- start 2021-04-09T18:19:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.278 7f14dba81700 1 -- start start 2021-04-09T18:19:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14dba81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14d40b42d0 0x7f14d4183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14dba81700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14d41ac090 0x7f14d41ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14dba81700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d4184d30 0x7f14d4185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14dba81700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40a1a90 con 0x7f14d40b42d0 2021-04-09T18:19:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14dba81700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d41acab0 con 0x7f14d41ac090 2021-04-09T18:19:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14dba81700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d4185a00 con 0x7f14d4184d30 2021-04-09T18:19:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14d989d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14d40b42d0 0x7f14d4183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14d989d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14d40b42d0 0x7f14d4183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47386/0 (socket says 172.21.15.32:47386) 2021-04-09T18:19:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14d989d700 1 -- 172.21.15.32:0/526029040 learned_addr learned my addr 172.21.15.32:0/526029040 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14d909c700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14d41ac090 0x7f14d41ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14da09e700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d4184d30 0x7f14d4185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14d989d700 1 -- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d4184d30 msgr2=0x7f14d4185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14d989d700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d4184d30 0x7f14d4185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14d989d700 1 -- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14d41ac090 msgr2=0x7f14d41ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:37.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14d989d700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14d41ac090 0x7f14d41ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.279 7f14d989d700 1 -- 172.21.15.32:0/526029040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d4185750 con 0x7f14d40b42d0 2021-04-09T18:19:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.280 7f14d989d700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14d40b42d0 0x7f14d4183520 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f14c400a550 tx=0x7f14c400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.280 7f14ca7fc700 1 -- 172.21.15.32:0/526029040 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14c4000e70 con 0x7f14d40b42d0 2021-04-09T18:19:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.280 7f14ca7fc700 1 -- 172.21.15.32:0/526029040 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c400ace0 con 0x7f14d40b42d0 2021-04-09T18:19:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.280 7f14ca7fc700 1 -- 172.21.15.32:0/526029040 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14c4004510 con 0x7f14d40b42d0 2021-04-09T18:19:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.280 7f14dba81700 1 -- 172.21.15.32:0/526029040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d4185750 con 0x7f14d40b42d0 2021-04-09T18:19:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.280 7f14dba81700 1 -- 172.21.15.32:0/526029040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d4185750 con 0x7f14d40b42d0 2021-04-09T18:19:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.280 7f14ca7fc700 1 -- 172.21.15.32:0/526029040 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f14c4004320 con 0x7f14d40b42d0 2021-04-09T18:19:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.281 7f14ca7fc700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14b4018930 0x7f14b401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.281 7f14ca7fc700 1 -- 172.21.15.32:0/526029040 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f14c4032a80 con 0x7f14d40b42d0 2021-04-09T18:19:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.281 7f14d909c700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14b4018930 0x7f14b401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.281 7f14d909c700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14b4018930 0x7f14b401adc0 crc :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:37.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.300 7f14dba81700 1 -- 172.21.15.32:0/526029040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d4182d10 con 0x7f14d40b42d0 2021-04-09T18:19:37.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.303 7f14ca7fc700 1 -- 172.21.15.32:0/526029040 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f14c4000c10 con 0x7f14d40b42d0 2021-04-09T18:19:37.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.495 7f14dba81700 1 -- 172.21.15.32:0/526029040 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14d410d480 con 0x7f14d40b42d0 2021-04-09T18:19:37.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.496 7f14ca7fc700 1 -- 172.21.15.32:0/526029040 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f14c4004320 con 0x7f14d40b42d0 2021-04-09T18:19:37.538 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:37.538 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:37.539 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 -- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14b4018930 msgr2=0x7f14b401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14b4018930 0x7f14b401adc0 crc :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 -- 172.21.15.32:0/526029040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14d40b42d0 msgr2=0x7f14d4183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14d40b42d0 0x7f14d4183520 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f14c400a550 tx=0x7f14c400a8f0).stop 2021-04-09T18:19:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 -- 172.21.15.32:0/526029040 shutdown_connections 2021-04-09T18:19:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14b4018930 0x7f14b401adc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14d4184d30 0x7f14d4185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14d41ac090 0x7f14d41ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 --2- 172.21.15.32:0/526029040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14d40b42d0 0x7f14d4183520 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 -- 172.21.15.32:0/526029040 >> 172.21.15.32:0/526029040 conn(0x7f14d419b890 msgr2=0x7f14d41a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 -- 172.21.15.32:0/526029040 shutdown_connections 2021-04-09T18:19:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:37.498 7f14dba81700 1 -- 172.21.15.32:0/526029040 wait complete. 2021-04-09T18:19:37.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:37.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:38.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:39.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.064 7f22243ca700 1 Processor -- start 2021-04-09T18:19:39.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.064 7f22243ca700 1 -- start start 2021-04-09T18:19:39.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22243ca700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f221c10da00 0x7f221c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:39.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22243ca700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221c10d480 con 0x7f221c10d010 2021-04-09T18:19:39.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22243ca700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221c0d1f70 con 0x7f221c0d7ca0 2021-04-09T18:19:39.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22243ca700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f221c0d8d30 con 0x7f221c10da00 2021-04-09T18:19:39.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22221e6700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f221c10d010 0x7f221c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56982/0 (socket says 172.21.15.32:56982) 2021-04-09T18:19:39.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22221e6700 1 -- 172.21.15.32:0/2513432421 learned_addr learned my addr 172.21.15.32:0/2513432421 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:39.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 -- 172.21.15.32:0/2513432421 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131542776 0 0) 0x7f2218001060 con 0x7f221c10d010 2021-04-09T18:19:39.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22229e7700 1 --2- 172.21.15.32:0/2513432421 >> v2:172.21.15.86:3300/0 conn(0x7f221c10da00 0x7f221c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:39.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 -- 172.21.15.32:0/2513432421 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2208002fe0 con 0x7f221c10d010 2021-04-09T18:19:39.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 -- 172.21.15.32:0/2513432421 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1500354810 0 0) 0x7f2218001460 con 0x7f221c10d010 2021-04-09T18:19:39.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 -- 172.21.15.32:0/2513432421 >> v2:172.21.15.86:3300/0 conn(0x7f221c10da00 msgr2=0x7f221c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:39.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 --2- 172.21.15.32:0/2513432421 >> v2:172.21.15.86:3300/0 conn(0x7f221c10da00 0x7f221c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 -- 172.21.15.32:0/2513432421 >> v1:172.21.15.86:6789/0 conn(0x7f221c0d7ca0 legacy=0x7f221c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:39.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 -- 172.21.15.32:0/2513432421 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c0d9220 con 0x7f221c10d010 2021-04-09T18:19:39.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 -- 172.21.15.32:0/2513432421 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f2218001980 con 0x7f221c10d010 2021-04-09T18:19:39.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 -- 172.21.15.32:0/2513432421 >> v1:172.21.15.32:6789/0 conn(0x7f221c10d010 legacy=0x7f221c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:39.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2208003a40 0x7f2208005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:39.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.065 7f22211e4700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2208006770 0x7f2208008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:39.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22211e4700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2208009590 0x7f220800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:39.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22211e4700 1 -- 172.21.15.32:0/2513432421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220800c300 con 0x7f2208003a40 2021-04-09T18:19:39.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22211e4700 1 -- 172.21.15.32:0/2513432421 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220800c520 con 0x7f2208006770 2021-04-09T18:19:39.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22221e6700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2208003a40 0x7f2208005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:39.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22211e4700 1 -- 172.21.15.32:0/2513432421 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2208050750 con 0x7f2208009590 2021-04-09T18:19:39.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22219e5700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2208006770 0x7f2208008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:39.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22229e7700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2208009590 0x7f220800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:39.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22221e6700 1 -- 172.21.15.32:0/2513432421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2208009590 msgr2=0x7f220800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22221e6700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2208009590 0x7f220800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22221e6700 1 -- 172.21.15.32:0/2513432421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2208006770 msgr2=0x7f2208008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22221e6700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2208006770 0x7f2208008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22229e7700 1 -- 172.21.15.32:0/2513432421 reap_dead start 2021-04-09T18:19:39.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22221e6700 1 -- 172.21.15.32:0/2513432421 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22080509b0 con 0x7f2208003a40 2021-04-09T18:19:39.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22221e6700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2208003a40 0x7f2208005ef0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f22180079f0 tx=0x7f221800a060).ready entity=mon.0 client_cookie=a7777b198c8a5ad7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:39.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22211e4700 1 -- 172.21.15.32:0/2513432421 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22180070c0 con 0x7f2208003a40 2021-04-09T18:19:39.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22211e4700 1 -- 172.21.15.32:0/2513432421 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22180073c0 con 0x7f2208003a40 2021-04-09T18:19:39.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22211e4700 1 -- 172.21.15.32:0/2513432421 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2218004720 con 0x7f2208003a40 2021-04-09T18:19:39.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22243ca700 1 -- 172.21.15.32:0/2513432421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2208003a40 msgr2=0x7f2208005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:39.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.066 7f22243ca700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2208003a40 0x7f2208005ef0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f22180079f0 tx=0x7f221800a060).stop 2021-04-09T18:19:39.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.067 7f22243ca700 1 -- 172.21.15.32:0/2513432421 shutdown_connections 2021-04-09T18:19:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.067 7f22243ca700 1 --2- 172.21.15.32:0/2513432421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2208003a40 0x7f2208005ef0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.067 7f22243ca700 1 -- 172.21.15.32:0/2513432421 >> 172.21.15.32:0/2513432421 conn(0x7f221c0ce5f0 msgr2=0x7f221c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:39.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.067 7f22243ca700 1 -- 172.21.15.32:0/2513432421 shutdown_connections 2021-04-09T18:19:39.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.067 7f22243ca700 1 -- 172.21.15.32:0/2513432421 wait complete. 2021-04-09T18:19:39.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.087 7f22243ca700 1 Processor -- start 2021-04-09T18:19:39.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.087 7f22243ca700 1 -- start start 2021-04-09T18:19:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22243ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f221c1a9360 0x7f221c1918f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22243ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f221c18c440 0x7f221c18c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22243ca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f221c1a80f0 0x7f221c1a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:39.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22243ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c191f10 con 0x7f221c18c440 2021-04-09T18:19:39.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22243ca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c1a9760 con 0x7f221c1a80f0 2021-04-09T18:19:39.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22243ca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f221c18ce60 con 0x7f221c1a9360 2021-04-09T18:19:39.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22219e5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f221c18c440 0x7f221c18c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:39.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22219e5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f221c18c440 0x7f221c18c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47402/0 (socket says 172.21.15.32:47402) 2021-04-09T18:19:39.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22219e5700 1 -- 172.21.15.32:0/1565717832 learned_addr learned my addr 172.21.15.32:0/1565717832 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:39.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22229e7700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f221c1a80f0 0x7f221c1a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:39.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22221e6700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f221c1a9360 0x7f221c1918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:39.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22219e5700 1 -- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f221c1a9360 msgr2=0x7f221c1918f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:39.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22219e5700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f221c1a9360 0x7f221c1918f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22219e5700 1 -- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f221c1a80f0 msgr2=0x7f221c1a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:39.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22219e5700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f221c1a80f0 0x7f221c1a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22219e5700 1 -- 172.21.15.32:0/1565717832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c1a8b10 con 0x7f221c18c440 2021-04-09T18:19:39.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22229e7700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f221c1a80f0 0x7f221c1a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.088 7f22219e5700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f221c18c440 0x7f221c18c840 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f220c005c30 tx=0x7f220c00a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.089 7f2212ffd700 1 -- 172.21.15.32:0/1565717832 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f220c00ac90 con 0x7f221c18c440 2021-04-09T18:19:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.089 7f2212ffd700 1 -- 172.21.15.32:0/1565717832 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220c00b070 con 0x7f221c18c440 2021-04-09T18:19:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.089 7f2212ffd700 1 -- 172.21.15.32:0/1565717832 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f220c00b4c0 con 0x7f221c18c440 2021-04-09T18:19:39.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.089 7f22243ca700 1 -- 172.21.15.32:0/1565717832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221c1a8b10 con 0x7f221c18c440 2021-04-09T18:19:39.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.089 7f22243ca700 1 -- 172.21.15.32:0/1565717832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221c1a8b10 con 0x7f221c18c440 2021-04-09T18:19:39.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.089 7f2212ffd700 1 -- 172.21.15.32:0/1565717832 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f220c000c10 con 0x7f221c18c440 2021-04-09T18:19:39.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.089 7f2212ffd700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2204018930 0x7f220401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:39.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.089 7f2212ffd700 1 -- 172.21.15.32:0/1565717832 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f220c032a00 con 0x7f221c18c440 2021-04-09T18:19:39.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.090 7f22221e6700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2204018930 0x7f220401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:39.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.090 7f22221e6700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2204018930 0x7f220401ae10 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:39.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.109 7f22243ca700 1 -- 172.21.15.32:0/1565717832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221c1a9760 con 0x7f221c18c440 2021-04-09T18:19:39.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.112 7f2212ffd700 1 -- 172.21.15.32:0/1565717832 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f220c00ac90 con 0x7f221c18c440 2021-04-09T18:19:39.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.303 7f22243ca700 1 -- 172.21.15.32:0/1565717832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f221c18bf60 con 0x7f221c18c440 2021-04-09T18:19:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.303 7f2212ffd700 1 -- 172.21.15.32:0/1565717832 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f220c000c10 con 0x7f221c18c440 2021-04-09T18:19:39.346 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:39.346 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:39.347 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:39.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 -- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2204018930 msgr2=0x7f220401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:39.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2204018930 0x7f220401ae10 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 -- 172.21.15.32:0/1565717832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f221c18c440 msgr2=0x7f221c18c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:39.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f221c18c440 0x7f221c18c840 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f220c005c30 tx=0x7f220c00a8a0).stop 2021-04-09T18:19:39.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 -- 172.21.15.32:0/1565717832 shutdown_connections 2021-04-09T18:19:39.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2204018930 0x7f220401ae10 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f221c18c440 0x7f221c18c840 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f221c1a80f0 0x7f221c1a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 --2- 172.21.15.32:0/1565717832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f221c1a9360 0x7f221c1918f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 -- 172.21.15.32:0/1565717832 >> 172.21.15.32:0/1565717832 conn(0x7f221c1837d0 msgr2=0x7f221c195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:39.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 -- 172.21.15.32:0/1565717832 shutdown_connections 2021-04-09T18:19:39.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:39.306 7f22243ca700 1 -- 172.21.15.32:0/1565717832 wait complete. 2021-04-09T18:19:39.761 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:39.761 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:40.763 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:40.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.882 7ffa3ce64700 1 Processor -- start 2021-04-09T18:19:40.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.882 7ffa3ce64700 1 -- start start 2021-04-09T18:19:40.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa3ce64700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffa380d6c80 0x7ffa380d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:40.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa3ce64700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa380d2440 0x7ffa3810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:40.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa3ce64700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa380d7280 con 0x7ffa380cffa0 2021-04-09T18:19:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa3ce64700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa380d7080 con 0x7ffa380d6c80 2021-04-09T18:19:40.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa3ce64700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa380d91e0 con 0x7ffa380d2440 2021-04-09T18:19:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa36312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffa380d6c80 0x7ffa380d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:40.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa36312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffa380d6c80 0x7ffa380d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47414/0 (socket says 172.21.15.32:47414) 2021-04-09T18:19:40.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa36312700 1 -- 172.21.15.32:0/3641575994 learned_addr learned my addr 172.21.15.32:0/3641575994 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:40.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa37314700 1 --2- 172.21.15.32:0/3641575994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa380d2440 0x7ffa3810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:40.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa36312700 1 -- 172.21.15.32:0/3641575994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa380d2440 msgr2=0x7ffa3810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:40.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa36312700 1 --2- 172.21.15.32:0/3641575994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa380d2440 0x7ffa3810cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:40.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa36312700 1 -- 172.21.15.32:0/3641575994 >> v1:172.21.15.86:6789/0 conn(0x7ffa380cffa0 legacy=0x7ffa380d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:40.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa36312700 1 -- 172.21.15.32:0/3641575994 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa380d2840 con 0x7ffa380d6c80 2021-04-09T18:19:40.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.883 7ffa36312700 1 --2- 172.21.15.32:0/3641575994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa380d6c80 0x7ffa380d7ff0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7ffa2c006700 tx=0x7ffa2c009b00).ready entity=mon.0 client_cookie=c54246a06f4fe6b9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:40.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.884 7ffa35b11700 1 -- 172.21.15.32:0/3641575994 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa2c00c0c0 con 0x7ffa380d6c80 2021-04-09T18:19:40.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.884 7ffa35b11700 1 -- 172.21.15.32:0/3641575994 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2c00c400 con 0x7ffa380d6c80 2021-04-09T18:19:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.884 7ffa35b11700 1 -- 172.21.15.32:0/3641575994 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa2c00c8e0 con 0x7ffa380d6c80 2021-04-09T18:19:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.884 7ffa3ce64700 1 -- 172.21.15.32:0/3641575994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa380d6c80 msgr2=0x7ffa380d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.884 7ffa3ce64700 1 --2- 172.21.15.32:0/3641575994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa380d6c80 0x7ffa380d7ff0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7ffa2c006700 tx=0x7ffa2c009b00).stop 2021-04-09T18:19:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.884 7ffa3ce64700 1 -- 172.21.15.32:0/3641575994 shutdown_connections 2021-04-09T18:19:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.885 7ffa3ce64700 1 --2- 172.21.15.32:0/3641575994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa380d2440 0x7ffa3810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.885 7ffa3ce64700 1 --2- 172.21.15.32:0/3641575994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa380d6c80 0x7ffa380d7ff0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.885 7ffa3ce64700 1 -- 172.21.15.32:0/3641575994 >> 172.21.15.32:0/3641575994 conn(0x7ffa380ce5c0 msgr2=0x7ffa380cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.885 7ffa3ce64700 1 -- 172.21.15.32:0/3641575994 shutdown_connections 2021-04-09T18:19:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.885 7ffa3ce64700 1 -- 172.21.15.32:0/3641575994 wait complete. 2021-04-09T18:19:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa3ce64700 1 Processor -- start 2021-04-09T18:19:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa3ce64700 1 -- start start 2021-04-09T18:19:40.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa3ce64700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa381acff0 0x7ffa381847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:40.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa3ce64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa38183fc0 0x7ffa381a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:40.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa3ce64700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa38186ff0 0x7ffa381873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:40.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa3ce64700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa38188c30 con 0x7ffa38183fc0 2021-04-09T18:19:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa3ce64700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa381843c0 con 0x7ffa38186ff0 2021-04-09T18:19:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa3ce64700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa381a5c20 con 0x7ffa381acff0 2021-04-09T18:19:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa36312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa38183fc0 0x7ffa381a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa36312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa38183fc0 0x7ffa381a5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47420/0 (socket says 172.21.15.32:47420) 2021-04-09T18:19:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa36312700 1 -- 172.21.15.32:0/2527168464 learned_addr learned my addr 172.21.15.32:0/2527168464 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa36b13700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa381acff0 0x7ffa381847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.905 7ffa37314700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa38186ff0 0x7ffa381873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa36312700 1 -- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa381acff0 msgr2=0x7ffa381847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa36312700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa381acff0 0x7ffa381847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa36312700 1 -- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa38186ff0 msgr2=0x7ffa381873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa36312700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa38186ff0 0x7ffa381873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa36312700 1 -- 172.21.15.32:0/2527168464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa38187a10 con 0x7ffa38183fc0 2021-04-09T18:19:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa36312700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa38183fc0 0x7ffa381a5600 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7ffa2c006f20 tx=0x7ffa2c00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:40.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa277fe700 1 -- 172.21.15.32:0/2527168464 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa2c00bdd0 con 0x7ffa38183fc0 2021-04-09T18:19:40.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa277fe700 1 -- 172.21.15.32:0/2527168464 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2c006440 con 0x7ffa38183fc0 2021-04-09T18:19:40.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa277fe700 1 -- 172.21.15.32:0/2527168464 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa2c0052e0 con 0x7ffa38183fc0 2021-04-09T18:19:40.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa38187a10 con 0x7ffa38183fc0 2021-04-09T18:19:40.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.906 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa38187a10 con 0x7ffa38183fc0 2021-04-09T18:19:40.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.907 7ffa277fe700 1 -- 172.21.15.32:0/2527168464 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ffa2c0052e0 con 0x7ffa38183fc0 2021-04-09T18:19:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.907 7ffa277fe700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa1c018930 0x7ffa1c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.907 7ffa277fe700 1 -- 172.21.15.32:0/2527168464 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ffa2c032a00 con 0x7ffa38183fc0 2021-04-09T18:19:40.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.907 7ffa36b13700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa1c018930 0x7ffa1c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:40.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.907 7ffa36b13700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa1c018930 0x7ffa1c01adc0 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:40.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.927 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa38182dc0 con 0x7ffa38183fc0 2021-04-09T18:19:40.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:40.930 7ffa277fe700 1 -- 172.21.15.32:0/2527168464 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ffa2c006440 con 0x7ffa38183fc0 2021-04-09T18:19:41.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.121 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa380d7280 con 0x7ffa38183fc0 2021-04-09T18:19:41.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.121 7ffa277fe700 1 -- 172.21.15.32:0/2527168464 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7ffa2c00bdd0 con 0x7ffa38183fc0 2021-04-09T18:19:41.165 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:41.166 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:41.166 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:41.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa1c018930 msgr2=0x7ffa1c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:41.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa1c018930 0x7ffa1c01adc0 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:41.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa38183fc0 msgr2=0x7ffa381a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:41.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa38183fc0 0x7ffa381a5600 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7ffa2c006f20 tx=0x7ffa2c00b9e0).stop 2021-04-09T18:19:41.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 shutdown_connections 2021-04-09T18:19:41.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa1c018930 0x7ffa1c01adc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:41.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa38183fc0 0x7ffa381a5600 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:41.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa38186ff0 0x7ffa381873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:41.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 --2- 172.21.15.32:0/2527168464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa381acff0 0x7ffa381847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:41.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 >> 172.21.15.32:0/2527168464 conn(0x7ffa3809abb0 msgr2=0x7ffa380a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:41.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 shutdown_connections 2021-04-09T18:19:41.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:41.124 7ffa3ce64700 1 -- 172.21.15.32:0/2527168464 wait complete. 2021-04-09T18:19:41.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:41.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:42.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:42.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.694 7f52e1b8a700 1 Processor -- start 2021-04-09T18:19:42.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.694 7f52e1b8a700 1 -- start start 2021-04-09T18:19:42.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52e1b8a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52dc0cffa0 0x7f52dc0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:42.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52e1b8a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f52dc0d6c80 0x7f52dc10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:42.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52e1b8a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52dc0d7280 con 0x7f52dc0d7800 2021-04-09T18:19:42.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52e1b8a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52dc0d7080 con 0x7f52dc0cffa0 2021-04-09T18:19:42.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52e1b8a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52dc0d91e0 con 0x7f52dc0d6c80 2021-04-09T18:19:42.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52db7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52dc0cffa0 0x7f52dc0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:42.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52db7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52dc0cffa0 0x7f52dc0d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47424/0 (socket says 172.21.15.32:47424) 2021-04-09T18:19:42.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52db7fe700 1 -- 172.21.15.32:0/33807870 learned_addr learned my addr 172.21.15.32:0/33807870 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52dbfff700 1 --2- 172.21.15.32:0/33807870 >> v2:172.21.15.86:3300/0 conn(0x7f52dc0d6c80 0x7f52dc10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:42.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52db7fe700 1 -- 172.21.15.32:0/33807870 >> v2:172.21.15.86:3300/0 conn(0x7f52dc0d6c80 msgr2=0x7f52dc10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52db7fe700 1 --2- 172.21.15.32:0/33807870 >> v2:172.21.15.86:3300/0 conn(0x7f52dc0d6c80 0x7f52dc10cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52db7fe700 1 -- 172.21.15.32:0/33807870 >> v1:172.21.15.86:6789/0 conn(0x7f52dc0d7800 legacy=0x7f52dc0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:42.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52db7fe700 1 -- 172.21.15.32:0/33807870 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc0d83f0 con 0x7f52dc0cffa0 2021-04-09T18:19:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52db7fe700 1 --2- 172.21.15.32:0/33807870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc0cffa0 0x7f52dc0d2440 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f52cc006700 tx=0x7f52cc009b80).ready entity=mon.0 client_cookie=96354cec31c7a77c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:42.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52da7fc700 1 -- 172.21.15.32:0/33807870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52cc00c0c0 con 0x7f52dc0cffa0 2021-04-09T18:19:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52da7fc700 1 -- 172.21.15.32:0/33807870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc00c440 con 0x7f52dc0cffa0 2021-04-09T18:19:42.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.695 7f52da7fc700 1 -- 172.21.15.32:0/33807870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52cc00c920 con 0x7f52dc0cffa0 2021-04-09T18:19:42.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.696 7f52e1b8a700 1 -- 172.21.15.32:0/33807870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc0cffa0 msgr2=0x7f52dc0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:42.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.696 7f52e1b8a700 1 --2- 172.21.15.32:0/33807870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc0cffa0 0x7f52dc0d2440 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f52cc006700 tx=0x7f52cc009b80).stop 2021-04-09T18:19:42.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.696 7f52e1b8a700 1 -- 172.21.15.32:0/33807870 shutdown_connections 2021-04-09T18:19:42.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.696 7f52e1b8a700 1 --2- 172.21.15.32:0/33807870 >> v2:172.21.15.86:3300/0 conn(0x7f52dc0d6c80 0x7f52dc10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.696 7f52e1b8a700 1 --2- 172.21.15.32:0/33807870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc0cffa0 0x7f52dc0d2440 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.696 7f52e1b8a700 1 -- 172.21.15.32:0/33807870 >> 172.21.15.32:0/33807870 conn(0x7f52dc0ce5c0 msgr2=0x7f52dc0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:42.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.696 7f52e1b8a700 1 -- 172.21.15.32:0/33807870 shutdown_connections 2021-04-09T18:19:42.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.696 7f52e1b8a700 1 -- 172.21.15.32:0/33807870 wait complete. 2021-04-09T18:19:42.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52e1b8a700 1 Processor -- start 2021-04-09T18:19:42.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52e1b8a700 1 -- start start 2021-04-09T18:19:42.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52e1b8a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc1acff0 0x7f52dc1847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:42.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52e1b8a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52dc183fc0 0x7f52dc1a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:42.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52e1b8a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc186ff0 0x7f52dc1873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:42.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52e1b8a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc188c30 con 0x7f52dc186ff0 2021-04-09T18:19:42.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52e1b8a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1843c0 con 0x7f52dc1acff0 2021-04-09T18:19:42.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52e1b8a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1a5c20 con 0x7f52dc183fc0 2021-04-09T18:19:42.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52dbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc186ff0 0x7f52dc1873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:42.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52dbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc186ff0 0x7f52dc1873f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47432/0 (socket says 172.21.15.32:47432) 2021-04-09T18:19:42.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52dbfff700 1 -- 172.21.15.32:0/2804799036 learned_addr learned my addr 172.21.15.32:0/2804799036 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:42.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52daffd700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52dc183fc0 0x7f52dc1a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:42.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.717 7f52db7fe700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc1acff0 0x7f52dc1847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:42.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52dbfff700 1 -- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52dc183fc0 msgr2=0x7f52dc1a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:42.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52dbfff700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52dc183fc0 0x7f52dc1a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52dbfff700 1 -- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc1acff0 msgr2=0x7f52dc1847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52dbfff700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc1acff0 0x7f52dc1847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52dbfff700 1 -- 172.21.15.32:0/2804799036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc187a10 con 0x7f52dc186ff0 2021-04-09T18:19:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52dbfff700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc186ff0 0x7f52dc1873f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7f52d400b910 tx=0x7f52d400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52c3fff700 1 -- 172.21.15.32:0/2804799036 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d4006aa0 con 0x7f52dc186ff0 2021-04-09T18:19:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52c3fff700 1 -- 172.21.15.32:0/2804799036 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d40011f0 con 0x7f52dc186ff0 2021-04-09T18:19:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52dc187a10 con 0x7f52dc186ff0 2021-04-09T18:19:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52c3fff700 1 -- 172.21.15.32:0/2804799036 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d4001640 con 0x7f52dc186ff0 2021-04-09T18:19:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.718 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52dc187a10 con 0x7f52dc186ff0 2021-04-09T18:19:42.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.719 7f52c3fff700 1 -- 172.21.15.32:0/2804799036 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f52d40056b0 con 0x7f52dc186ff0 2021-04-09T18:19:42.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.719 7f52c3fff700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52bc018930 0x7f52bc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:42.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.719 7f52c3fff700 1 -- 172.21.15.32:0/2804799036 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f52d4032da0 con 0x7f52dc186ff0 2021-04-09T18:19:42.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.719 7f52db7fe700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52bc018930 0x7f52bc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:42.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.719 7f52db7fe700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52bc018930 0x7f52bc01adc0 crc :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:42.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.738 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52dc182dc0 con 0x7f52dc186ff0 2021-04-09T18:19:42.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.742 7f52c3fff700 1 -- 172.21.15.32:0/2804799036 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f52d400bdf0 con 0x7f52dc186ff0 2021-04-09T18:19:42.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.932 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52dc0d7280 con 0x7f52dc186ff0 2021-04-09T18:19:42.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.933 7f52c3fff700 1 -- 172.21.15.32:0/2804799036 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f52d40011f0 con 0x7f52dc186ff0 2021-04-09T18:19:42.976 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:42.977 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:42.978 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:42.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52bc018930 msgr2=0x7f52bc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:42.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52bc018930 0x7f52bc01adc0 crc :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc186ff0 msgr2=0x7f52dc1873f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:42.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc186ff0 0x7f52dc1873f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7f52d400b910 tx=0x7f52d400bcb0).stop 2021-04-09T18:19:42.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 shutdown_connections 2021-04-09T18:19:42.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52bc018930 0x7f52bc01adc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52dc186ff0 0x7f52dc1873f0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52dc183fc0 0x7f52dc1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 --2- 172.21.15.32:0/2804799036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc1acff0 0x7f52dc1847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:42.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 >> 172.21.15.32:0/2804799036 conn(0x7f52dc09abb0 msgr2=0x7f52dc0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:42.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 shutdown_connections 2021-04-09T18:19:42.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:42.935 7f52e1b8a700 1 -- 172.21.15.32:0/2804799036 wait complete. 2021-04-09T18:19:43.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:43.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:44.405 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:44.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.526 7fa30ac87700 1 Processor -- start 2021-04-09T18:19:44.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.526 7fa30ac87700 1 -- start start 2021-04-09T18:19:44.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa30ac87700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa3040cffa0 0x7fa3040d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa30ac87700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040d6c80 0x7fa30410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa30ac87700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040d7280 con 0x7fa3040d7800 2021-04-09T18:19:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa30ac87700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3040d7080 con 0x7fa3040cffa0 2021-04-09T18:19:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa30ac87700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040d91e0 con 0x7fa3040d6c80 2021-04-09T18:19:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa3092a4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040d6c80 0x7fa30410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa308aa3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa3040cffa0 0x7fa3040d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa303fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa3040d7800 0x7fa3040d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:54952/0 (socket says 172.21.15.32:54952) 2021-04-09T18:19:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa303fff700 1 -- 172.21.15.32:0/928069686 learned_addr learned my addr 172.21.15.32:0/928069686 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa3037fe700 1 -- 172.21.15.32:0/928069686 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893026844 0 0) 0x7fa2f4001440 con 0x7fa3040d7800 2021-04-09T18:19:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa3037fe700 1 -- 172.21.15.32:0/928069686 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0002fe0 con 0x7fa3040d7800 2021-04-09T18:19:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa308aa3700 1 -- 172.21.15.32:0/928069686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040d6c80 msgr2=0x7fa30410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa308aa3700 1 --2- 172.21.15.32:0/928069686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040d6c80 0x7fa30410cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa308aa3700 1 -- 172.21.15.32:0/928069686 >> v1:172.21.15.86:6789/0 conn(0x7fa3040d7800 legacy=0x7fa3040d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa308aa3700 1 -- 172.21.15.32:0/928069686 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3040d05a0 con 0x7fa3040cffa0 2021-04-09T18:19:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa3092a4700 1 --2- 172.21.15.32:0/928069686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040d6c80 0x7fa30410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:19:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.527 7fa308aa3700 1 --2- 172.21.15.32:0/928069686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3040cffa0 0x7fa3040d2440 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fa2ec006650 tx=0x7fa2ec009a80).ready entity=mon.1 client_cookie=31fc222df86021de server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa3037fe700 1 -- 172.21.15.32:0/928069686 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2ec0009b0 con 0x7fa3040cffa0 2021-04-09T18:19:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa3037fe700 1 -- 172.21.15.32:0/928069686 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec00c0c0 con 0x7fa3040cffa0 2021-04-09T18:19:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa3037fe700 1 -- 172.21.15.32:0/928069686 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2ec00c510 con 0x7fa3040cffa0 2021-04-09T18:19:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa30ac87700 1 -- 172.21.15.32:0/928069686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3040cffa0 msgr2=0x7fa3040d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:44.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa30ac87700 1 --2- 172.21.15.32:0/928069686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3040cffa0 0x7fa3040d2440 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fa2ec006650 tx=0x7fa2ec009a80).stop 2021-04-09T18:19:44.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa30ac87700 1 -- 172.21.15.32:0/928069686 shutdown_connections 2021-04-09T18:19:44.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa30ac87700 1 --2- 172.21.15.32:0/928069686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040d6c80 0x7fa30410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa30ac87700 1 --2- 172.21.15.32:0/928069686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3040cffa0 0x7fa3040d2440 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa30ac87700 1 -- 172.21.15.32:0/928069686 >> 172.21.15.32:0/928069686 conn(0x7fa3040ce5c0 msgr2=0x7fa3040cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:44.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa30ac87700 1 -- 172.21.15.32:0/928069686 shutdown_connections 2021-04-09T18:19:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.528 7fa30ac87700 1 -- 172.21.15.32:0/928069686 wait complete. 2021-04-09T18:19:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa30ac87700 1 Processor -- start 2021-04-09T18:19:44.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa30ac87700 1 -- start start 2021-04-09T18:19:44.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa30ac87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa304190320 0x7fa30418fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:44.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa30ac87700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30408ab30 0x7fa30408af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:44.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa30ac87700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040aa9b0 0x7fa3040aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:44.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa30ac87700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30408a650 con 0x7fa304190320 2021-04-09T18:19:44.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa30ac87700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30408b550 con 0x7fa30408ab30 2021-04-09T18:19:44.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa30ac87700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040ab3d0 con 0x7fa3040aa9b0 2021-04-09T18:19:44.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa308aa3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa304190320 0x7fa30418fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:44.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa308aa3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa304190320 0x7fa30418fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47442/0 (socket says 172.21.15.32:47442) 2021-04-09T18:19:44.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.549 7fa308aa3700 1 -- 172.21.15.32:0/3287794917 learned_addr learned my addr 172.21.15.32:0/3287794917 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:44.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa303fff700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30408ab30 0x7fa30408af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:44.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa3092a4700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040aa9b0 0x7fa3040aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:44.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa308aa3700 1 -- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040aa9b0 msgr2=0x7fa3040aadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:44.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa308aa3700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040aa9b0 0x7fa3040aadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa308aa3700 1 -- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30408ab30 msgr2=0x7fa30408af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:44.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa308aa3700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30408ab30 0x7fa30408af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa308aa3700 1 -- 172.21.15.32:0/3287794917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3040ab5f0 con 0x7fa304190320 2021-04-09T18:19:44.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa308aa3700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa304190320 0x7fa30418fb20 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7fa2ec009650 tx=0x7fa2ec00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:44.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa3017fa700 1 -- 172.21.15.32:0/3287794917 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2ec0093a0 con 0x7fa304190320 2021-04-09T18:19:44.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa3017fa700 1 -- 172.21.15.32:0/3287794917 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec006070 con 0x7fa304190320 2021-04-09T18:19:44.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa3017fa700 1 -- 172.21.15.32:0/3287794917 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2ec0052e0 con 0x7fa304190320 2021-04-09T18:19:44.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3040ab5f0 con 0x7fa304190320 2021-04-09T18:19:44.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.550 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3040ab5f0 con 0x7fa304190320 2021-04-09T18:19:44.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.551 7fa3017fa700 1 -- 172.21.15.32:0/3287794917 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa2ec009300 con 0x7fa304190320 2021-04-09T18:19:44.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.551 7fa3017fa700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2e8018930 0x7fa2e801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:44.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.551 7fa3017fa700 1 -- 172.21.15.32:0/3287794917 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa2ec032a00 con 0x7fa304190320 2021-04-09T18:19:44.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.551 7fa303fff700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2e8018930 0x7fa2e801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:44.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.551 7fa303fff700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2e8018930 0x7fa2e801ae10 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.570 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30408a650 con 0x7fa304190320 2021-04-09T18:19:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.574 7fa3017fa700 1 -- 172.21.15.32:0/3287794917 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa2ec006030 con 0x7fa304190320 2021-04-09T18:19:44.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.766 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa30408a650 con 0x7fa304190320 2021-04-09T18:19:44.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.766 7fa3017fa700 1 -- 172.21.15.32:0/3287794917 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fa2ec000940 con 0x7fa304190320 2021-04-09T18:19:44.810 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:44.811 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:44.811 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:44.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2e8018930 msgr2=0x7fa2e801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:44.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2e8018930 0x7fa2e801ae10 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa304190320 msgr2=0x7fa30418fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:44.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa304190320 0x7fa30418fb20 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7fa2ec009650 tx=0x7fa2ec00b8e0).stop 2021-04-09T18:19:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 shutdown_connections 2021-04-09T18:19:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa2e8018930 0x7fa2e801ae10 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3040aa9b0 0x7fa3040aadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30408ab30 0x7fa30408af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 --2- 172.21.15.32:0/3287794917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa304190320 0x7fa30418fb20 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:44.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 >> 172.21.15.32:0/3287794917 conn(0x7fa3041941b0 msgr2=0x7fa304183120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:44.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 shutdown_connections 2021-04-09T18:19:44.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:44.769 7fa30ac87700 1 -- 172.21.15.32:0/3287794917 wait complete. 2021-04-09T18:19:45.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:45.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:46.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:46.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.351 7f3486352700 1 Processor -- start 2021-04-09T18:19:46.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.351 7f3486352700 1 -- start start 2021-04-09T18:19:46.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.352 7f3486352700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f348010d010 0x7f34800d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:46.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.352 7f3486352700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34800d1da0 0x7f34800d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:46.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.352 7f3486352700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348010d480 con 0x7f34800d82c0 2021-04-09T18:19:46.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.352 7f3486352700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34800d9220 con 0x7f348010d010 2021-04-09T18:19:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.352 7f3486352700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34800d86c0 con 0x7f34800d1da0 2021-04-09T18:19:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.352 7f347ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f348010d010 0x7f34800d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.352 7f347ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f348010d010 0x7f34800d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47450/0 (socket says 172.21.15.32:47450) 2021-04-09T18:19:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.352 7f347ffff700 1 -- 172.21.15.32:0/1394326163 learned_addr learned my addr 172.21.15.32:0/1394326163 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.352 7f3477fff700 1 --2- 172.21.15.32:0/1394326163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34800d1da0 0x7f34800d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f347ffff700 1 -- 172.21.15.32:0/1394326163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34800d1da0 msgr2=0x7f34800d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f347ffff700 1 --2- 172.21.15.32:0/1394326163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34800d1da0 0x7f34800d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f347ffff700 1 -- 172.21.15.32:0/1394326163 >> v1:172.21.15.86:6789/0 conn(0x7f34800d82c0 legacy=0x7f348010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:46.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f347ffff700 1 -- 172.21.15.32:0/1394326163 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348010e1f0 con 0x7f348010d010 2021-04-09T18:19:46.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f347f7fe700 1 -- 172.21.15.32:0/1394326163 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246911592 0 0) 0x7f3478000d30 con 0x7f34800d82c0 2021-04-09T18:19:46.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f347ffff700 1 --2- 172.21.15.32:0/1394326163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348010d010 0x7f34800d1780 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f3470006700 tx=0x7f3470009b00).ready entity=mon.0 client_cookie=4aaf8be2eb54ebd3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:46.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f347f7fe700 1 -- 172.21.15.32:0/1394326163 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f347000c0c0 con 0x7f348010d010 2021-04-09T18:19:46.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f347f7fe700 1 -- 172.21.15.32:0/1394326163 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347000c400 con 0x7f348010d010 2021-04-09T18:19:46.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f347f7fe700 1 -- 172.21.15.32:0/1394326163 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f347000c8e0 con 0x7f348010d010 2021-04-09T18:19:46.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f3486352700 1 -- 172.21.15.32:0/1394326163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348010d010 msgr2=0x7f34800d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:46.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.353 7f3486352700 1 --2- 172.21.15.32:0/1394326163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348010d010 0x7f34800d1780 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f3470006700 tx=0x7f3470009b00).stop 2021-04-09T18:19:46.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.354 7f3486352700 1 -- 172.21.15.32:0/1394326163 shutdown_connections 2021-04-09T18:19:46.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.354 7f3486352700 1 --2- 172.21.15.32:0/1394326163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34800d1da0 0x7f34800d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.354 7f3486352700 1 --2- 172.21.15.32:0/1394326163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348010d010 0x7f34800d1780 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.354 7f3486352700 1 -- 172.21.15.32:0/1394326163 >> 172.21.15.32:0/1394326163 conn(0x7f34800ce5f0 msgr2=0x7f34800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:46.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.354 7f3486352700 1 -- 172.21.15.32:0/1394326163 shutdown_connections 2021-04-09T18:19:46.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.354 7f3486352700 1 -- 172.21.15.32:0/1394326163 wait complete. 2021-04-09T18:19:46.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.374 7f3486352700 1 Processor -- start 2021-04-09T18:19:46.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.374 7f3486352700 1 -- start start 2021-04-09T18:19:46.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f3486352700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34800a90c0 0x7f34801a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:46.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f3486352700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34801a76d0 0x7f34801a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:46.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f3486352700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348017bce0 0x7f348017c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:46.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f3486352700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34801a7c60 con 0x7f348017bce0 2021-04-09T18:19:46.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f3486352700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34801a8e20 con 0x7f34800a90c0 2021-04-09T18:19:46.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f3486352700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348017c700 con 0x7f34801a76d0 2021-04-09T18:19:46.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f348496f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348017bce0 0x7f348017c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:46.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f348496f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348017bce0 0x7f348017c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47460/0 (socket says 172.21.15.32:47460) 2021-04-09T18:19:46.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f348496f700 1 -- 172.21.15.32:0/702057235 learned_addr learned my addr 172.21.15.32:0/702057235 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:46.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f347ffff700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34800a90c0 0x7f34801a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:46.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f3477fff700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34801a76d0 0x7f34801a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:46.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f348496f700 1 -- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34801a76d0 msgr2=0x7f34801a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:46.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f348496f700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34801a76d0 0x7f34801a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f348496f700 1 -- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34800a90c0 msgr2=0x7f34801a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:46.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f348496f700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34800a90c0 0x7f34801a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f348496f700 1 -- 172.21.15.32:0/702057235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348017c920 con 0x7f348017bce0 2021-04-09T18:19:46.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f3477fff700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34801a76d0 0x7f34801a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:46.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f347ffff700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34800a90c0 0x7f34801a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:46.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f348496f700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348017bce0 0x7f348017c0e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f3478009980 tx=0x7f347800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:46.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f347d7fa700 1 -- 172.21.15.32:0/702057235 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3478005870 con 0x7f348017bce0 2021-04-09T18:19:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f347d7fa700 1 -- 172.21.15.32:0/702057235 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3478005b70 con 0x7f348017bce0 2021-04-09T18:19:46.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f3486352700 1 -- 172.21.15.32:0/702057235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348017c920 con 0x7f348017bce0 2021-04-09T18:19:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.375 7f347d7fa700 1 -- 172.21.15.32:0/702057235 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3478006030 con 0x7f348017bce0 2021-04-09T18:19:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.376 7f3486352700 1 -- 172.21.15.32:0/702057235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348017c920 con 0x7f348017bce0 2021-04-09T18:19:46.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.376 7f347d7fa700 1 -- 172.21.15.32:0/702057235 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3478006030 con 0x7f348017bce0 2021-04-09T18:19:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.376 7f347d7fa700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3464018930 0x7f346401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.376 7f347d7fa700 1 -- 172.21.15.32:0/702057235 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3478032e10 con 0x7f348017bce0 2021-04-09T18:19:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.376 7f347ffff700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3464018930 0x7f346401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:46.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.377 7f347ffff700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3464018930 0x7f346401adc0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.396 7f3486352700 1 -- 172.21.15.32:0/702057235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34801b7010 con 0x7f348017bce0 2021-04-09T18:19:46.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.400 7f347d7fa700 1 -- 172.21.15.32:0/702057235 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f347800be10 con 0x7f348017bce0 2021-04-09T18:19:46.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.590 7f3486352700 1 -- 172.21.15.32:0/702057235 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f348010d480 con 0x7f348017bce0 2021-04-09T18:19:46.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.591 7f347d7fa700 1 -- 172.21.15.32:0/702057235 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f347800be10 con 0x7f348017bce0 2021-04-09T18:19:46.635 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:46.636 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:46.636 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 -- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3464018930 msgr2=0x7f346401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:46.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3464018930 0x7f346401adc0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 -- 172.21.15.32:0/702057235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348017bce0 msgr2=0x7f348017c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:46.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348017bce0 0x7f348017c0e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f3478009980 tx=0x7f347800bcd0).stop 2021-04-09T18:19:46.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 -- 172.21.15.32:0/702057235 shutdown_connections 2021-04-09T18:19:46.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3464018930 0x7f346401adc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f348017bce0 0x7f348017c0e0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34801a76d0 0x7f34801a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 --2- 172.21.15.32:0/702057235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34800a90c0 0x7f34801a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:46.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 -- 172.21.15.32:0/702057235 >> 172.21.15.32:0/702057235 conn(0x7f34801acae0 msgr2=0x7f348017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:46.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.593 7f3486352700 1 -- 172.21.15.32:0/702057235 shutdown_connections 2021-04-09T18:19:46.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:46.594 7f3486352700 1 -- 172.21.15.32:0/702057235 wait complete. 2021-04-09T18:19:47.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:47.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:48.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:48.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.173 7f3afd5b9700 1 Processor -- start 2021-04-09T18:19:48.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.173 7f3afd5b9700 1 -- start start 2021-04-09T18:19:48.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.173 7f3afd5b9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3af810d010 0x7f3af80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:48.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.173 7f3afd5b9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af810d480 con 0x7f3af80d8490 2021-04-09T18:19:48.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.173 7f3afd5b9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af810e1f0 con 0x7f3af80d1da0 2021-04-09T18:19:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.173 7f3afd5b9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af80d8d30 con 0x7f3af810d010 2021-04-09T18:19:48.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.173 7f3af72bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3af810d010 0x7f3af80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.173 7f3af7ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3af80d8490 0x7f3af810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57056/0 (socket says 172.21.15.32:57056) 2021-04-09T18:19:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3af7ac0700 1 -- 172.21.15.32:0/221228318 learned_addr learned my addr 172.21.15.32:0/221228318 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:48.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3af72bf700 1 -- 172.21.15.32:0/221228318 >> v1:172.21.15.86:6789/0 conn(0x7f3af80d1da0 legacy=0x7f3af80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3af72bf700 1 -- 172.21.15.32:0/221228318 >> v1:172.21.15.32:6789/0 conn(0x7f3af80d8490 legacy=0x7f3af810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3af72bf700 1 -- 172.21.15.32:0/221228318 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af80d9220 con 0x7f3af810d010 2021-04-09T18:19:48.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3af62bd700 1 -- 172.21.15.32:0/221228318 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250598454 0 0) 0x7f3af0000d30 con 0x7f3af80d8490 2021-04-09T18:19:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3af72bf700 1 --2- 172.21.15.32:0/221228318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af810d010 0x7f3af80d1780 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f3ae8006650 tx=0x7f3ae8009a00).ready entity=mon.0 client_cookie=e7ad932aca7f0990 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3af62bd700 1 -- 172.21.15.32:0/221228318 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae8009df0 con 0x7f3af810d010 2021-04-09T18:19:48.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3af62bd700 1 -- 172.21.15.32:0/221228318 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae80009b0 con 0x7f3af810d010 2021-04-09T18:19:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3af62bd700 1 -- 172.21.15.32:0/221228318 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae800c2b0 con 0x7f3af810d010 2021-04-09T18:19:48.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3afd5b9700 1 -- 172.21.15.32:0/221228318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af810d010 msgr2=0x7f3af80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.174 7f3afd5b9700 1 --2- 172.21.15.32:0/221228318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af810d010 0x7f3af80d1780 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f3ae8006650 tx=0x7f3ae8009a00).stop 2021-04-09T18:19:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.175 7f3afd5b9700 1 -- 172.21.15.32:0/221228318 shutdown_connections 2021-04-09T18:19:48.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.175 7f3afd5b9700 1 --2- 172.21.15.32:0/221228318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af810d010 0x7f3af80d1780 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:48.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.175 7f3afd5b9700 1 -- 172.21.15.32:0/221228318 >> 172.21.15.32:0/221228318 conn(0x7f3af80ce5f0 msgr2=0x7f3af80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.175 7f3afd5b9700 1 -- 172.21.15.32:0/221228318 shutdown_connections 2021-04-09T18:19:48.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.175 7f3afd5b9700 1 -- 172.21.15.32:0/221228318 wait complete. 2021-04-09T18:19:48.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.195 7f3afd5b9700 1 Processor -- start 2021-04-09T18:19:48.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.195 7f3afd5b9700 1 -- start start 2021-04-09T18:19:48.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3afd5b9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af81a9a50 0x7f3af808a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:48.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3afd5b9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af808afb0 0x7f3af809d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:48.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3afd5b9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af809ddf0 0x7f3af8183b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:48.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3afd5b9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af81a9e50 con 0x7f3af809ddf0 2021-04-09T18:19:48.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3afd5b9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af809e410 con 0x7f3af808afb0 2021-04-09T18:19:48.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3afd5b9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af81841a0 con 0x7f3af81a9a50 2021-04-09T18:19:48.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af7ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af809ddf0 0x7f3af8183b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:48.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af7ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af809ddf0 0x7f3af8183b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47474/0 (socket says 172.21.15.32:47474) 2021-04-09T18:19:48.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af7ac0700 1 -- 172.21.15.32:0/454875730 learned_addr learned my addr 172.21.15.32:0/454875730 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:48.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af72bf700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af81a9a50 0x7f3af808a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:48.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af6abe700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af808afb0 0x7f3af809d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:48.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af7ac0700 1 -- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af81a9a50 msgr2=0x7f3af808a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:48.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af7ac0700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af81a9a50 0x7f3af808a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:48.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af7ac0700 1 -- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af808afb0 msgr2=0x7f3af809d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:48.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af7ac0700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af808afb0 0x7f3af809d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:48.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af7ac0700 1 -- 172.21.15.32:0/454875730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af8184380 con 0x7f3af809ddf0 2021-04-09T18:19:48.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af72bf700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af81a9a50 0x7f3af808a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:48.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af6abe700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af808afb0 0x7f3af809d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:48.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3af7ac0700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af809ddf0 0x7f3af8183b80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f3af0007f90 tx=0x7f3af000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:48.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3aeffff700 1 -- 172.21.15.32:0/454875730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3af0000cc0 con 0x7f3af809ddf0 2021-04-09T18:19:48.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.196 7f3aeffff700 1 -- 172.21.15.32:0/454875730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af0004930 con 0x7f3af809ddf0 2021-04-09T18:19:48.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.197 7f3aeffff700 1 -- 172.21.15.32:0/454875730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3af0004dc0 con 0x7f3af809ddf0 2021-04-09T18:19:48.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.197 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af8184380 con 0x7f3af809ddf0 2021-04-09T18:19:48.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.197 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af8184380 con 0x7f3af809ddf0 2021-04-09T18:19:48.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.198 7f3aeffff700 1 -- 172.21.15.32:0/454875730 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3af000ad90 con 0x7f3af809ddf0 2021-04-09T18:19:48.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.198 7f3aeffff700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ad40188c0 0x7f3ad401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:48.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.198 7f3aeffff700 1 -- 172.21.15.32:0/454875730 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3af0032b10 con 0x7f3af809ddf0 2021-04-09T18:19:48.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.198 7f3af72bf700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ad40188c0 0x7f3ad401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:48.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.199 7f3af72bf700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ad40188c0 0x7f3ad401adc0 crc :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:48.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.218 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af810d480 con 0x7f3af809ddf0 2021-04-09T18:19:48.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.221 7f3aeffff700 1 -- 172.21.15.32:0/454875730 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3af0004720 con 0x7f3af809ddf0 2021-04-09T18:19:48.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.412 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3af810d480 con 0x7f3af809ddf0 2021-04-09T18:19:48.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.413 7f3aeffff700 1 -- 172.21.15.32:0/454875730 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f3af000ad90 con 0x7f3af809ddf0 2021-04-09T18:19:48.457 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:48.458 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:48.458 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:48.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ad40188c0 msgr2=0x7f3ad401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:48.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ad40188c0 0x7f3ad401adc0 crc :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af809ddf0 msgr2=0x7f3af8183b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:48.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af809ddf0 0x7f3af8183b80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f3af0007f90 tx=0x7f3af000ac50).stop 2021-04-09T18:19:48.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 shutdown_connections 2021-04-09T18:19:48.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ad40188c0 0x7f3ad401adc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:48.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3af809ddf0 0x7f3af8183b80 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af808afb0 0x7f3af809d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:48.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 --2- 172.21.15.32:0/454875730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af81a9a50 0x7f3af808a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 >> 172.21.15.32:0/454875730 conn(0x7f3af81965c0 msgr2=0x7f3af81ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 shutdown_connections 2021-04-09T18:19:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:48.415 7f3afd5b9700 1 -- 172.21.15.32:0/454875730 wait complete. 2021-04-09T18:19:48.890 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:48.891 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:49.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:50.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.004 7f873de44700 1 Processor -- start 2021-04-09T18:19:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.004 7f873de44700 1 -- start start 2021-04-09T18:19:50.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f873de44700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87380cffa0 0x7f87380d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f873de44700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87380d7280 con 0x7f87380d6c80 2021-04-09T18:19:50.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f873de44700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87380d83f0 con 0x7f87380d7800 2021-04-09T18:19:50.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f873de44700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87380d7080 con 0x7f87380cffa0 2021-04-09T18:19:50.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f87377fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87380cffa0 0x7f87380d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:50.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f8737fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f87380d6c80 0x7f873810cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57070/0 (socket says 172.21.15.32:57070) 2021-04-09T18:19:50.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f8737fff700 1 -- 172.21.15.32:0/3711915374 learned_addr learned my addr 172.21.15.32:0/3711915374 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f87377fe700 1 -- 172.21.15.32:0/3711915374 >> v1:172.21.15.86:6789/0 conn(0x7f87380d7800 legacy=0x7f87380d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f87377fe700 1 -- 172.21.15.32:0/3711915374 >> v1:172.21.15.32:6789/0 conn(0x7f87380d6c80 legacy=0x7f873810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:50.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f87367fc700 1 -- 172.21.15.32:0/3711915374 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853671808 0 0) 0x7f872c000c60 con 0x7f87380d7800 2021-04-09T18:19:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f87377fe700 1 -- 172.21.15.32:0/3711915374 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380d91e0 con 0x7f87380cffa0 2021-04-09T18:19:50.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f87377fe700 1 --2- 172.21.15.32:0/3711915374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87380cffa0 0x7f87380d2440 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7f8728006650 tx=0x7f8728009a00).ready entity=mon.0 client_cookie=9bfc5406b5f24a92 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:50.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f87367fc700 1 -- 172.21.15.32:0/3711915374 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8728009df0 con 0x7f87380cffa0 2021-04-09T18:19:50.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f87367fc700 1 -- 172.21.15.32:0/3711915374 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87280009b0 con 0x7f87380cffa0 2021-04-09T18:19:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.005 7f87367fc700 1 -- 172.21.15.32:0/3711915374 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f872800c2b0 con 0x7f87380cffa0 2021-04-09T18:19:50.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.006 7f873de44700 1 -- 172.21.15.32:0/3711915374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87380cffa0 msgr2=0x7f87380d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:50.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.006 7f873de44700 1 --2- 172.21.15.32:0/3711915374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87380cffa0 0x7f87380d2440 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7f8728006650 tx=0x7f8728009a00).stop 2021-04-09T18:19:50.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.006 7f873de44700 1 -- 172.21.15.32:0/3711915374 shutdown_connections 2021-04-09T18:19:50.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.006 7f873de44700 1 --2- 172.21.15.32:0/3711915374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87380cffa0 0x7f87380d2440 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:50.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.006 7f873de44700 1 -- 172.21.15.32:0/3711915374 >> 172.21.15.32:0/3711915374 conn(0x7f87380ce5c0 msgr2=0x7f87380cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:50.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.006 7f873de44700 1 -- 172.21.15.32:0/3711915374 shutdown_connections 2021-04-09T18:19:50.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.006 7f873de44700 1 -- 172.21.15.32:0/3711915374 wait complete. 2021-04-09T18:19:50.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f873de44700 1 Processor -- start 2021-04-09T18:19:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f873de44700 1 -- start start 2021-04-09T18:19:50.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f873de44700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873817e470 0x7f873817dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:50.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f873de44700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87380955d0 0x7f87380962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:50.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f873de44700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87380968c0 0x7f87381a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:50.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f873de44700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8738095da0 con 0x7f873817e470 2021-04-09T18:19:50.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f873de44700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873817e870 con 0x7f87380955d0 2021-04-09T18:19:50.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f873de44700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8738096cc0 con 0x7f87380968c0 2021-04-09T18:19:50.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f87377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873817e470 0x7f873817dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:50.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f87377fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873817e470 0x7f873817dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47484/0 (socket says 172.21.15.32:47484) 2021-04-09T18:19:50.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.027 7f87377fe700 1 -- 172.21.15.32:0/1284915345 learned_addr learned my addr 172.21.15.32:0/1284915345 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:50.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f87377fe700 1 -- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87380968c0 msgr2=0x7f87381a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:50.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f87377fe700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87380968c0 0x7f87381a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:19:50.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f87377fe700 1 -- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87380955d0 msgr2=0x7f87380962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:50.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f8736ffd700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87380955d0 0x7f87380962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:50.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f87377fe700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87380955d0 0x7f87380962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:50.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f87377fe700 1 -- 172.21.15.32:0/1284915345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380959d0 con 0x7f873817e470 2021-04-09T18:19:50.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f87377fe700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873817e470 0x7f873817dce0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f872800b580 tx=0x7f872800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:50.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f871ffff700 1 -- 172.21.15.32:0/1284915345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8728000c90 con 0x7f873817e470 2021-04-09T18:19:50.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f871ffff700 1 -- 172.21.15.32:0/1284915345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8728002030 con 0x7f873817e470 2021-04-09T18:19:50.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f873de44700 1 -- 172.21.15.32:0/1284915345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87380959d0 con 0x7f873817e470 2021-04-09T18:19:50.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f871ffff700 1 -- 172.21.15.32:0/1284915345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8728002480 con 0x7f873817e470 2021-04-09T18:19:50.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.028 7f873de44700 1 -- 172.21.15.32:0/1284915345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87380959d0 con 0x7f873817e470 2021-04-09T18:19:50.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.029 7f871ffff700 1 -- 172.21.15.32:0/1284915345 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8728002210 con 0x7f873817e470 2021-04-09T18:19:50.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.029 7f871ffff700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8718018930 0x7f871801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:50.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.029 7f871ffff700 1 -- 172.21.15.32:0/1284915345 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8728032a60 con 0x7f873817e470 2021-04-09T18:19:50.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.029 7f8736ffd700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8718018930 0x7f871801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:50.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.029 7f8736ffd700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8718018930 0x7f871801ae10 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:50.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.049 7f873de44700 1 -- 172.21.15.32:0/1284915345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873817cae0 con 0x7f873817e470 2021-04-09T18:19:50.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.052 7f871ffff700 1 -- 172.21.15.32:0/1284915345 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8728002240 con 0x7f873817e470 2021-04-09T18:19:50.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.243 7f873de44700 1 -- 172.21.15.32:0/1284915345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f873817cae0 con 0x7f873817e470 2021-04-09T18:19:50.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.244 7f871ffff700 1 -- 172.21.15.32:0/1284915345 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f872800bd10 con 0x7f873817e470 2021-04-09T18:19:50.287 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:50.288 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:50.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 -- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8718018930 msgr2=0x7f871801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:50.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8718018930 0x7f871801ae10 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 -- 172.21.15.32:0/1284915345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873817e470 msgr2=0x7f873817dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:50.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873817e470 0x7f873817dce0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f872800b580 tx=0x7f872800b920).stop 2021-04-09T18:19:50.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 -- 172.21.15.32:0/1284915345 shutdown_connections 2021-04-09T18:19:50.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8718018930 0x7f871801ae10 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:50.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87380968c0 0x7f87381a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:19:50.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87380955d0 0x7f87380962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:50.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 --2- 172.21.15.32:0/1284915345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873817e470 0x7f873817dce0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:50.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.246 7f873de44700 1 -- 172.21.15.32:0/1284915345 >> 172.21.15.32:0/1284915345 conn(0x7f873818ce60 msgr2=0x7f8738094920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:50.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.247 7f873de44700 1 -- 172.21.15.32:0/1284915345 shutdown_connections 2021-04-09T18:19:50.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:50.247 7f873de44700 1 -- 172.21.15.32:0/1284915345 wait complete. 2021-04-09T18:19:50.705 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:50.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:51.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:51.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.818 7f15232cb700 1 Processor -- start 2021-04-09T18:19:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.818 7f15232cb700 1 -- start start 2021-04-09T18:19:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.818 7f15232cb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f151c0d7ca0 0x7f151c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.818 7f15232cb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c10d480 con 0x7f151c10d010 2021-04-09T18:19:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.818 7f15232cb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c0d86c0 con 0x7f151c10da00 2021-04-09T18:19:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.818 7f15232cb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151c0d9220 con 0x7f151c0d7ca0 2021-04-09T18:19:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f15208e6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f151c0d7ca0 0x7f151c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f15210e7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f151c10d010 0x7f151c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57082/0 (socket says 172.21.15.32:57082) 2021-04-09T18:19:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f15210e7700 1 -- 172.21.15.32:0/1256015202 learned_addr learned my addr 172.21.15.32:0/1256015202 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f15208e6700 1 -- 172.21.15.32:0/1256015202 >> v1:172.21.15.86:6789/0 conn(0x7f151c10da00 legacy=0x7f151c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:51.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f15208e6700 1 -- 172.21.15.32:0/1256015202 >> v1:172.21.15.32:6789/0 conn(0x7f151c10d010 legacy=0x7f151c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:51.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f1513fff700 1 -- 172.21.15.32:0/1256015202 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374314316 0 0) 0x7f1518000ce0 con 0x7f151c10d010 2021-04-09T18:19:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f15208e6700 1 -- 172.21.15.32:0/1256015202 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c0d1f70 con 0x7f151c0d7ca0 2021-04-09T18:19:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f15208e6700 1 --2- 172.21.15.32:0/1256015202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0d7ca0 0x7f151c0d80a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7f150c006650 tx=0x7f150c009a00).ready entity=mon.0 client_cookie=4bb988795478f3e5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f1513fff700 1 -- 172.21.15.32:0/1256015202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f150c009df0 con 0x7f151c0d7ca0 2021-04-09T18:19:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f1513fff700 1 -- 172.21.15.32:0/1256015202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c0009b0 con 0x7f151c0d7ca0 2021-04-09T18:19:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f1513fff700 1 -- 172.21.15.32:0/1256015202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f150c00c2b0 con 0x7f151c0d7ca0 2021-04-09T18:19:51.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f15232cb700 1 -- 172.21.15.32:0/1256015202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0d7ca0 msgr2=0x7f151c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:51.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.819 7f15232cb700 1 --2- 172.21.15.32:0/1256015202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0d7ca0 0x7f151c0d80a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7f150c006650 tx=0x7f150c009a00).stop 2021-04-09T18:19:51.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.820 7f15232cb700 1 -- 172.21.15.32:0/1256015202 shutdown_connections 2021-04-09T18:19:51.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.820 7f15232cb700 1 --2- 172.21.15.32:0/1256015202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0d7ca0 0x7f151c0d80a0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:51.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.820 7f15232cb700 1 -- 172.21.15.32:0/1256015202 >> 172.21.15.32:0/1256015202 conn(0x7f151c0ce5f0 msgr2=0x7f151c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:51.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.820 7f15232cb700 1 -- 172.21.15.32:0/1256015202 shutdown_connections 2021-04-09T18:19:51.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.820 7f15232cb700 1 -- 172.21.15.32:0/1256015202 wait complete. 2021-04-09T18:19:51.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.840 7f15232cb700 1 Processor -- start 2021-04-09T18:19:51.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.840 7f15232cb700 1 -- start start 2021-04-09T18:19:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15232cb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151c17e010 0x7f151c17e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15232cb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f151c09f280 0x7f151c09f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15232cb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0b1b00 0x7f151c0b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15232cb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c09ed00 con 0x7f151c0b1b00 2021-04-09T18:19:51.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15232cb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c09fca0 con 0x7f151c09f280 2021-04-09T18:19:51.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15232cb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c0b2520 con 0x7f151c17e010 2021-04-09T18:19:51.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15218e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0b1b00 0x7f151c0b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:51.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15218e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0b1b00 0x7f151c0b1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47502/0 (socket says 172.21.15.32:47502) 2021-04-09T18:19:51.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15218e8700 1 -- 172.21.15.32:0/3176183391 learned_addr learned my addr 172.21.15.32:0/3176183391 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:51.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15208e6700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f151c09f280 0x7f151c09f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:51.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.841 7f15210e7700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151c17e010 0x7f151c17e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:51.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f15218e8700 1 -- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151c17e010 msgr2=0x7f151c17e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:51.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f15218e8700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151c17e010 0x7f151c17e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:51.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f15218e8700 1 -- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f151c09f280 msgr2=0x7f151c09f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:51.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f15218e8700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f151c09f280 0x7f151c09f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:51.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f15218e8700 1 -- 172.21.15.32:0/3176183391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c0b2740 con 0x7f151c0b1b00 2021-04-09T18:19:51.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f15218e8700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0b1b00 0x7f151c0b1f00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f1514009980 tx=0x7f151400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:51.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f1511ffb700 1 -- 172.21.15.32:0/3176183391 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1514005870 con 0x7f151c0b1b00 2021-04-09T18:19:51.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f1511ffb700 1 -- 172.21.15.32:0/3176183391 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514005af0 con 0x7f151c0b1b00 2021-04-09T18:19:51.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f15232cb700 1 -- 172.21.15.32:0/3176183391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c0b2740 con 0x7f151c0b1b00 2021-04-09T18:19:51.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f1511ffb700 1 -- 172.21.15.32:0/3176183391 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1514006030 con 0x7f151c0b1b00 2021-04-09T18:19:51.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.842 7f15232cb700 1 -- 172.21.15.32:0/3176183391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c0b2740 con 0x7f151c0b1b00 2021-04-09T18:19:51.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.843 7f1511ffb700 1 -- 172.21.15.32:0/3176183391 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1514005cd0 con 0x7f151c0b1b00 2021-04-09T18:19:51.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.843 7f1511ffb700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14fc018930 0x7f14fc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:51.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.843 7f1511ffb700 1 -- 172.21.15.32:0/3176183391 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1514032e30 con 0x7f151c0b1b00 2021-04-09T18:19:51.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.843 7f15210e7700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14fc018930 0x7f14fc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:51.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.843 7f15210e7700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14fc018930 0x7f14fc01ae10 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:51.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.863 7f15232cb700 1 -- 172.21.15.32:0/3176183391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c17cb20 con 0x7f151c0b1b00 2021-04-09T18:19:51.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:51.866 7f1511ffb700 1 -- 172.21.15.32:0/3176183391 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f151400be10 con 0x7f151c0b1b00 2021-04-09T18:19:52.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.060 7f15232cb700 1 -- 172.21.15.32:0/3176183391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f151c10d480 con 0x7f151c0b1b00 2021-04-09T18:19:52.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.061 7f1511ffb700 1 -- 172.21.15.32:0/3176183391 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f151400be10 con 0x7f151c0b1b00 2021-04-09T18:19:52.105 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:52.106 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:52.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 -- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14fc018930 msgr2=0x7f14fc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:52.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14fc018930 0x7f14fc01ae10 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:52.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 -- 172.21.15.32:0/3176183391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0b1b00 msgr2=0x7f151c0b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:52.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0b1b00 0x7f151c0b1f00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f1514009980 tx=0x7f151400bcd0).stop 2021-04-09T18:19:52.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 -- 172.21.15.32:0/3176183391 shutdown_connections 2021-04-09T18:19:52.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14fc018930 0x7f14fc01ae10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:52.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f151c0b1b00 0x7f151c0b1f00 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:52.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f151c09f280 0x7f151c09f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:52.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 --2- 172.21.15.32:0/3176183391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151c17e010 0x7f151c17e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:52.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 -- 172.21.15.32:0/3176183391 >> 172.21.15.32:0/3176183391 conn(0x7f151c1965a0 msgr2=0x7f151c19dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:52.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 -- 172.21.15.32:0/3176183391 shutdown_connections 2021-04-09T18:19:52.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:52.063 7f15232cb700 1 -- 172.21.15.32:0/3176183391 wait complete. 2021-04-09T18:19:52.525 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:52.526 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:53.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.646 7f0274643700 1 Processor -- start 2021-04-09T18:19:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.646 7f0274643700 1 -- start start 2021-04-09T18:19:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f0274643700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f026c0d6c80 0x7f026c0d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f0274643700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c0d2440 0x7f026c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f0274643700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c0d7280 con 0x7f026c0cffa0 2021-04-09T18:19:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f0274643700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c0d7080 con 0x7f026c0d6c80 2021-04-09T18:19:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f0274643700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0d91e0 con 0x7f026c0d2440 2021-04-09T18:19:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f0271c5e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f026c0d6c80 0x7f026c0d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f0272c60700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c0d2440 0x7f026c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f027245f700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f026c0cffa0 0x7f026c0d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55022/0 (socket says 172.21.15.32:55022) 2021-04-09T18:19:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f0271c5e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f026c0d6c80 0x7f026c0d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:39268/0 (socket says 172.21.15.32:39268) 2021-04-09T18:19:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f0272c60700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c0d2440 0x7f026c10cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:54734/0 (socket says 172.21.15.32:54734) 2021-04-09T18:19:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.647 7f027245f700 1 -- 172.21.15.32:0/3067291123 learned_addr learned my addr 172.21.15.32:0/3067291123 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f0272c60700 1 -- 172.21.15.32:0/3067291123 >> v2:172.21.15.86:3300/0 conn(0x7f026c0d6c80 msgr2=0x7f026c0d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f0272c60700 1 --2- 172.21.15.32:0/3067291123 >> v2:172.21.15.86:3300/0 conn(0x7f026c0d6c80 0x7f026c0d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f0272c60700 1 -- 172.21.15.32:0/3067291123 >> v1:172.21.15.86:6789/0 conn(0x7f026c0cffa0 legacy=0x7f026c0d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f027145d700 1 -- 172.21.15.32:0/3067291123 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563008046 0 0) 0x7f02680010c0 con 0x7f026c0cffa0 2021-04-09T18:19:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f0272c60700 1 -- 172.21.15.32:0/3067291123 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0d2840 con 0x7f026c0d2440 2021-04-09T18:19:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f0271c5e700 1 --2- 172.21.15.32:0/3067291123 >> v2:172.21.15.86:3300/0 conn(0x7f026c0d6c80 0x7f026c0d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:19:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f0272c60700 1 --2- 172.21.15.32:0/3067291123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c0d2440 0x7f026c10cfd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0264006e50 tx=0x7f0264009e70).ready entity=mon.2 client_cookie=b3ab288105367dc6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f027145d700 1 -- 172.21.15.32:0/3067291123 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0264006a20 con 0x7f026c0d2440 2021-04-09T18:19:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f027145d700 1 -- 172.21.15.32:0/3067291123 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0264006790 con 0x7f026c0d2440 2021-04-09T18:19:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f027145d700 1 -- 172.21.15.32:0/3067291123 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f026400b700 con 0x7f026c0d2440 2021-04-09T18:19:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f0274643700 1 -- 172.21.15.32:0/3067291123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c0d2440 msgr2=0x7f026c10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:53.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.648 7f0274643700 1 --2- 172.21.15.32:0/3067291123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c0d2440 0x7f026c10cfd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0264006e50 tx=0x7f0264009e70).stop 2021-04-09T18:19:53.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.649 7f0274643700 1 -- 172.21.15.32:0/3067291123 shutdown_connections 2021-04-09T18:19:53.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.649 7f0274643700 1 --2- 172.21.15.32:0/3067291123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c0d2440 0x7f026c10cfd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.649 7f0274643700 1 --2- 172.21.15.32:0/3067291123 >> v2:172.21.15.86:3300/0 conn(0x7f026c0d6c80 0x7f026c0d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.649 7f0274643700 1 -- 172.21.15.32:0/3067291123 >> 172.21.15.32:0/3067291123 conn(0x7f026c0ce5c0 msgr2=0x7f026c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.649 7f0274643700 1 -- 172.21.15.32:0/3067291123 shutdown_connections 2021-04-09T18:19:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.649 7f0274643700 1 -- 172.21.15.32:0/3067291123 wait complete. 2021-04-09T18:19:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.669 7f0274643700 1 Processor -- start 2021-04-09T18:19:53.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.669 7f0274643700 1 -- start start 2021-04-09T18:19:53.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0274643700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f026c1acff0 0x7f026c08fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:53.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0274643700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c090150 0x7f026c08c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:53.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0274643700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f026c08cca0 0x7f026c08e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:53.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0274643700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c08c100 con 0x7f026c08cca0 2021-04-09T18:19:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0274643700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c08d0a0 con 0x7f026c1acff0 2021-04-09T18:19:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0274643700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c08e960 con 0x7f026c090150 2021-04-09T18:19:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0272c60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f026c08cca0 0x7f026c08e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0272c60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f026c08cca0 0x7f026c08e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47512/0 (socket says 172.21.15.32:47512) 2021-04-09T18:19:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0272c60700 1 -- 172.21.15.32:0/4136698748 learned_addr learned my addr 172.21.15.32:0/4136698748 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:53.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0271c5e700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c090150 0x7f026c08c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:53.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f027245f700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f026c1acff0 0x7f026c08fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:53.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0272c60700 1 -- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c090150 msgr2=0x7f026c08c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:53.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0272c60700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c090150 0x7f026c08c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0272c60700 1 -- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f026c1acff0 msgr2=0x7f026c08fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:53.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0272c60700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f026c1acff0 0x7f026c08fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0272c60700 1 -- 172.21.15.32:0/4136698748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c08eb80 con 0x7f026c08cca0 2021-04-09T18:19:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.670 7f0272c60700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f026c08cca0 0x7f026c08e340 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f0264006e60 tx=0x7f026400bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.671 7f0262ffd700 1 -- 172.21.15.32:0/4136698748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0264000df0 con 0x7f026c08cca0 2021-04-09T18:19:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.671 7f0262ffd700 1 -- 172.21.15.32:0/4136698748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026400bdb0 con 0x7f026c08cca0 2021-04-09T18:19:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.671 7f0262ffd700 1 -- 172.21.15.32:0/4136698748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0264001390 con 0x7f026c08cca0 2021-04-09T18:19:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.671 7f0274643700 1 -- 172.21.15.32:0/4136698748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c08eb80 con 0x7f026c08cca0 2021-04-09T18:19:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.671 7f0274643700 1 -- 172.21.15.32:0/4136698748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c08eb80 con 0x7f026c08cca0 2021-04-09T18:19:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.672 7f0262ffd700 1 -- 172.21.15.32:0/4136698748 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f02640011a0 con 0x7f026c08cca0 2021-04-09T18:19:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.672 7f0262ffd700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02540188d0 0x7f025401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.672 7f0262ffd700 1 -- 172.21.15.32:0/4136698748 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0264032d70 con 0x7f026c08cca0 2021-04-09T18:19:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.672 7f027245f700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02540188d0 0x7f025401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.672 7f027245f700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02540188d0 0x7f025401add0 crc :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:53.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.691 7f0274643700 1 -- 172.21.15.32:0/4136698748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026c182dc0 con 0x7f026c08cca0 2021-04-09T18:19:53.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.695 7f0262ffd700 1 -- 172.21.15.32:0/4136698748 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0264009130 con 0x7f026c08cca0 2021-04-09T18:19:53.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.886 7f0274643700 1 -- 172.21.15.32:0/4136698748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f026c0d7280 con 0x7f026c08cca0 2021-04-09T18:19:53.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.886 7f0262ffd700 1 -- 172.21.15.32:0/4136698748 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f02640011a0 con 0x7f026c08cca0 2021-04-09T18:19:53.930 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:53.931 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:53.931 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:53.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 -- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02540188d0 msgr2=0x7f025401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:53.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02540188d0 0x7f025401add0 crc :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 -- 172.21.15.32:0/4136698748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f026c08cca0 msgr2=0x7f026c08e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:53.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f026c08cca0 0x7f026c08e340 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f0264006e60 tx=0x7f026400bc70).stop 2021-04-09T18:19:53.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 -- 172.21.15.32:0/4136698748 shutdown_connections 2021-04-09T18:19:53.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02540188d0 0x7f025401add0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f026c08cca0 0x7f026c08e340 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f026c090150 0x7f026c08c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 --2- 172.21.15.32:0/4136698748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f026c1acff0 0x7f026c08fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:53.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 -- 172.21.15.32:0/4136698748 >> 172.21.15.32:0/4136698748 conn(0x7f026c09abb0 msgr2=0x7f026c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:53.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 -- 172.21.15.32:0/4136698748 shutdown_connections 2021-04-09T18:19:53.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:53.889 7f0274643700 1 -- 172.21.15.32:0/4136698748 wait complete. 2021-04-09T18:19:54.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:54.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:55.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:55.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8ca4fd700 1 Processor -- start 2021-04-09T18:19:55.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8ca4fd700 1 -- start start 2021-04-09T18:19:55.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8ca4fd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd8c40d7ca0 0x7fd8c40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:55.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8ca4fd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8c410da00 0x7fd8c410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:55.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8ca4fd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c410d480 con 0x7fd8c410d010 2021-04-09T18:19:55.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8ca4fd700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c40d9220 con 0x7fd8c410da00 2021-04-09T18:19:55.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8ca4fd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c40d86c0 con 0x7fd8c40d7ca0 2021-04-09T18:19:55.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8c8b1a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8c410da00 0x7fd8c410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:55.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8c8b1a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8c410da00 0x7fd8c410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47524/0 (socket says 172.21.15.32:47524) 2021-04-09T18:19:55.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.460 7fd8c8b1a700 1 -- 172.21.15.32:0/3264857476 learned_addr learned my addr 172.21.15.32:0/3264857476 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:55.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8bbfff700 1 --2- 172.21.15.32:0/3264857476 >> v2:172.21.15.86:3300/0 conn(0x7fd8c40d7ca0 0x7fd8c40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:55.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8c8b1a700 1 -- 172.21.15.32:0/3264857476 >> v2:172.21.15.86:3300/0 conn(0x7fd8c40d7ca0 msgr2=0x7fd8c40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:55.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8c8b1a700 1 --2- 172.21.15.32:0/3264857476 >> v2:172.21.15.86:3300/0 conn(0x7fd8c40d7ca0 0x7fd8c40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8c8b1a700 1 -- 172.21.15.32:0/3264857476 >> v1:172.21.15.86:6789/0 conn(0x7fd8c410d010 legacy=0x7fd8c40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:55.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8c8b1a700 1 -- 172.21.15.32:0/3264857476 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c40d1f70 con 0x7fd8c410da00 2021-04-09T18:19:55.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8c8b1a700 1 --2- 172.21.15.32:0/3264857476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c410da00 0x7fd8c410de00 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7fd8bc006f00 tx=0x7fd8bc009f10).ready entity=mon.0 client_cookie=94291025d7248081 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:55.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8c37fe700 1 -- 172.21.15.32:0/3264857476 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8bc00b3b0 con 0x7fd8c410da00 2021-04-09T18:19:55.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8c37fe700 1 -- 172.21.15.32:0/3264857476 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8bc00b730 con 0x7fd8c410da00 2021-04-09T18:19:55.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8c37fe700 1 -- 172.21.15.32:0/3264857476 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8bc00bc10 con 0x7fd8c410da00 2021-04-09T18:19:55.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8ca4fd700 1 -- 172.21.15.32:0/3264857476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c410da00 msgr2=0x7fd8c410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.461 7fd8ca4fd700 1 --2- 172.21.15.32:0/3264857476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c410da00 0x7fd8c410de00 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7fd8bc006f00 tx=0x7fd8bc009f10).stop 2021-04-09T18:19:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.462 7fd8ca4fd700 1 -- 172.21.15.32:0/3264857476 shutdown_connections 2021-04-09T18:19:55.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.462 7fd8ca4fd700 1 --2- 172.21.15.32:0/3264857476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c410da00 0x7fd8c410de00 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.462 7fd8ca4fd700 1 --2- 172.21.15.32:0/3264857476 >> v2:172.21.15.86:3300/0 conn(0x7fd8c40d7ca0 0x7fd8c40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.462 7fd8ca4fd700 1 -- 172.21.15.32:0/3264857476 >> 172.21.15.32:0/3264857476 conn(0x7fd8c40ce5f0 msgr2=0x7fd8c40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.462 7fd8ca4fd700 1 -- 172.21.15.32:0/3264857476 shutdown_connections 2021-04-09T18:19:55.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.462 7fd8ca4fd700 1 -- 172.21.15.32:0/3264857476 wait complete. 2021-04-09T18:19:55.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.482 7fd8ca4fd700 1 Processor -- start 2021-04-09T18:19:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.482 7fd8ca4fd700 1 -- start start 2021-04-09T18:19:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8ca4fd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8c4185060 0x7fd8c4185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:55.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8ca4fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c4186c00 0x7fd8c4187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8ca4fd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8c4191360 0x7fd8c4191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8ca4fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c41863a0 con 0x7fd8c4186c00 2021-04-09T18:19:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8ca4fd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4185a60 con 0x7fd8c4185060 2021-04-09T18:19:55.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8ca4fd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4187620 con 0x7fd8c4191360 2021-04-09T18:19:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8bbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c4186c00 0x7fd8c4187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8bbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c4186c00 0x7fd8c4187000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47526/0 (socket says 172.21.15.32:47526) 2021-04-09T18:19:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8bbfff700 1 -- 172.21.15.32:0/2209761500 learned_addr learned my addr 172.21.15.32:0/2209761500 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8c8b1a700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8c4191360 0x7fd8c4191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:55.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8c3fff700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8c4185060 0x7fd8c4185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8bbfff700 1 -- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8c4191360 msgr2=0x7fd8c4191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8bbfff700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8c4191360 0x7fd8c4191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8bbfff700 1 -- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8c4185060 msgr2=0x7fd8c4185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:55.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8bbfff700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8c4185060 0x7fd8c4185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8bbfff700 1 -- 172.21.15.32:0/2209761500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c4185460 con 0x7fd8c4186c00 2021-04-09T18:19:55.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8c3fff700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8c4185060 0x7fd8c4185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:19:55.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.483 7fd8bbfff700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c4186c00 0x7fd8c4187000 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7fd8ac00b580 tx=0x7fd8ac00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:55.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.484 7fd8c17fa700 1 -- 172.21.15.32:0/2209761500 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ac0064d0 con 0x7fd8c4186c00 2021-04-09T18:19:55.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.484 7fd8c17fa700 1 -- 172.21.15.32:0/2209761500 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac00bd10 con 0x7fd8c4186c00 2021-04-09T18:19:55.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.484 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c4185460 con 0x7fd8c4186c00 2021-04-09T18:19:55.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.484 7fd8c17fa700 1 -- 172.21.15.32:0/2209761500 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ac0064d0 con 0x7fd8c4186c00 2021-04-09T18:19:55.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.484 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c4185460 con 0x7fd8c4186c00 2021-04-09T18:19:55.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.484 7fd8c17fa700 1 -- 172.21.15.32:0/2209761500 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd8ac006c90 con 0x7fd8c4186c00 2021-04-09T18:19:55.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.484 7fd8c17fa700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8a8018930 0x7fd8a801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:55.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.484 7fd8c17fa700 1 -- 172.21.15.32:0/2209761500 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd8ac032a00 con 0x7fd8c4186c00 2021-04-09T18:19:55.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.485 7fd8c3fff700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8a8018930 0x7fd8a801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:55.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.485 7fd8c3fff700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8a8018930 0x7fd8a801adc0 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:55.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.504 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c41866d0 con 0x7fd8c4186c00 2021-04-09T18:19:55.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.508 7fd8c17fa700 1 -- 172.21.15.32:0/2209761500 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd8ac006820 con 0x7fd8c4186c00 2021-04-09T18:19:55.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.699 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8c41866d0 con 0x7fd8c4186c00 2021-04-09T18:19:55.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.699 7fd8c17fa700 1 -- 172.21.15.32:0/2209761500 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fd8ac006c90 con 0x7fd8c4186c00 2021-04-09T18:19:55.743 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:55.744 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:55.744 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:55.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8a8018930 msgr2=0x7fd8a801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:55.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8a8018930 0x7fd8a801adc0 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c4186c00 msgr2=0x7fd8c4187000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:55.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c4186c00 0x7fd8c4187000 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7fd8ac00b580 tx=0x7fd8ac00b920).stop 2021-04-09T18:19:55.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 shutdown_connections 2021-04-09T18:19:55.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8a8018930 0x7fd8a801adc0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8c4186c00 0x7fd8c4187000 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8c4191360 0x7fd8c4191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 --2- 172.21.15.32:0/2209761500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8c4185060 0x7fd8c4185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:55.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 >> 172.21.15.32:0/2209761500 conn(0x7fd8c41837d0 msgr2=0x7fd8c4195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:55.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 shutdown_connections 2021-04-09T18:19:55.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:55.702 7fd8ca4fd700 1 -- 172.21.15.32:0/2209761500 wait complete. 2021-04-09T18:19:56.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:56.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:57.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:57.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.276 7f1dc3743700 1 Processor -- start 2021-04-09T18:19:57.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.276 7f1dc3743700 1 -- start start 2021-04-09T18:19:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc3743700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1dbc10d010 0x7f1dbc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:57.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc3743700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1dbc0d8490 0x7f1dbc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:57.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc3743700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dbc10d480 con 0x7f1dbc0d1da0 2021-04-09T18:19:57.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc3743700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dbc0d8d30 con 0x7f1dbc10d010 2021-04-09T18:19:57.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc3743700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dbc0d9220 con 0x7f1dbc0d8490 2021-04-09T18:19:57.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc155f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1dbc10d010 0x7f1dbc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:57.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc155f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1dbc10d010 0x7f1dbc0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47534/0 (socket says 172.21.15.32:47534) 2021-04-09T18:19:57.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc155f700 1 -- 172.21.15.32:0/1530388116 learned_addr learned my addr 172.21.15.32:0/1530388116 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc1d60700 1 --2- 172.21.15.32:0/1530388116 >> v2:172.21.15.86:3300/0 conn(0x7f1dbc0d8490 0x7f1dbc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:57.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc155f700 1 -- 172.21.15.32:0/1530388116 >> v2:172.21.15.86:3300/0 conn(0x7f1dbc0d8490 msgr2=0x7f1dbc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:57.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc155f700 1 --2- 172.21.15.32:0/1530388116 >> v2:172.21.15.86:3300/0 conn(0x7f1dbc0d8490 0x7f1dbc10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc155f700 1 -- 172.21.15.32:0/1530388116 >> v1:172.21.15.86:6789/0 conn(0x7f1dbc0d1da0 legacy=0x7f1dbc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:57.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc155f700 1 -- 172.21.15.32:0/1530388116 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc10e020 con 0x7f1dbc10d010 2021-04-09T18:19:57.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.277 7f1dc1d60700 1 --2- 172.21.15.32:0/1530388116 >> v2:172.21.15.86:3300/0 conn(0x7f1dbc0d8490 0x7f1dbc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:57.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.278 7f1dc155f700 1 --2- 172.21.15.32:0/1530388116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc10d010 0x7f1dbc0d1780 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f1db8006700 tx=0x7f1db8009b80).ready entity=mon.0 client_cookie=791acb7bb7e1cf5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:57.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.278 7f1db3fff700 1 -- 172.21.15.32:0/1530388116 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1db800c0c0 con 0x7f1dbc10d010 2021-04-09T18:19:57.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.278 7f1db3fff700 1 -- 172.21.15.32:0/1530388116 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db800c440 con 0x7f1dbc10d010 2021-04-09T18:19:57.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.278 7f1db3fff700 1 -- 172.21.15.32:0/1530388116 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1db800c920 con 0x7f1dbc10d010 2021-04-09T18:19:57.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.278 7f1dc3743700 1 -- 172.21.15.32:0/1530388116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc10d010 msgr2=0x7f1dbc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:57.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.278 7f1dc3743700 1 --2- 172.21.15.32:0/1530388116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc10d010 0x7f1dbc0d1780 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f1db8006700 tx=0x7f1db8009b80).stop 2021-04-09T18:19:57.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.278 7f1dc3743700 1 -- 172.21.15.32:0/1530388116 shutdown_connections 2021-04-09T18:19:57.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.278 7f1dc3743700 1 --2- 172.21.15.32:0/1530388116 >> v2:172.21.15.86:3300/0 conn(0x7f1dbc0d8490 0x7f1dbc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.279 7f1dc3743700 1 --2- 172.21.15.32:0/1530388116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc10d010 0x7f1dbc0d1780 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.279 7f1dc3743700 1 -- 172.21.15.32:0/1530388116 >> 172.21.15.32:0/1530388116 conn(0x7f1dbc0ce5f0 msgr2=0x7f1dbc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:57.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.279 7f1dc3743700 1 -- 172.21.15.32:0/1530388116 shutdown_connections 2021-04-09T18:19:57.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.279 7f1dc3743700 1 -- 172.21.15.32:0/1530388116 wait complete. 2021-04-09T18:19:57.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.299 7f1dc3743700 1 Processor -- start 2021-04-09T18:19:57.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.299 7f1dc3743700 1 -- start start 2021-04-09T18:19:57.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.299 7f1dc3743700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc179fd0 0x7f1dbc0976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:57.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc3743700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dbc097cc0 0x7f1dbc1a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:57.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc3743700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dbc1a4b10 0x7f1dbc1a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc3743700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc098140 con 0x7f1dbc179fd0 2021-04-09T18:19:57.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc3743700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc17a3d0 con 0x7f1dbc1a4b10 2021-04-09T18:19:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc3743700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dbc17c870 con 0x7f1dbc097cc0 2021-04-09T18:19:57.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc155f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc179fd0 0x7f1dbc0976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:57.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc155f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc179fd0 0x7f1dbc0976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47540/0 (socket says 172.21.15.32:47540) 2021-04-09T18:19:57.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc155f700 1 -- 172.21.15.32:0/2066779591 learned_addr learned my addr 172.21.15.32:0/2066779591 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:57.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc0d5e700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dbc097cc0 0x7f1dbc1a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:57.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc1d60700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dbc1a4b10 0x7f1dbc1a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:57.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc155f700 1 -- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dbc097cc0 msgr2=0x7f1dbc1a44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:57.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc155f700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dbc097cc0 0x7f1dbc1a44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc155f700 1 -- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dbc1a4b10 msgr2=0x7f1dbc1a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:57.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc155f700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dbc1a4b10 0x7f1dbc1a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc155f700 1 -- 172.21.15.32:0/2066779591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc179bf0 con 0x7f1dbc179fd0 2021-04-09T18:19:57.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc155f700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc179fd0 0x7f1dbc0976a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f1db800b5e0 tx=0x7f1db800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1db1ffb700 1 -- 172.21.15.32:0/2066779591 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1db800bd70 con 0x7f1dbc179fd0 2021-04-09T18:19:57.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1db1ffb700 1 -- 172.21.15.32:0/2066779591 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db8002030 con 0x7f1dbc179fd0 2021-04-09T18:19:57.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dbc179bf0 con 0x7f1dbc179fd0 2021-04-09T18:19:57.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1db1ffb700 1 -- 172.21.15.32:0/2066779591 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1db8002550 con 0x7f1dbc179fd0 2021-04-09T18:19:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.300 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dbc179bf0 con 0x7f1dbc179fd0 2021-04-09T18:19:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.301 7f1db1ffb700 1 -- 172.21.15.32:0/2066779591 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1db8002210 con 0x7f1dbc179fd0 2021-04-09T18:19:57.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.301 7f1db1ffb700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1d9c018930 0x7f1d9c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:57.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.301 7f1db1ffb700 1 -- 172.21.15.32:0/2066779591 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1db8032a80 con 0x7f1dbc179fd0 2021-04-09T18:19:57.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.301 7f1dc0d5e700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1d9c018930 0x7f1d9c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:57.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.301 7f1dc0d5e700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1d9c018930 0x7f1d9c01ae10 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:57.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.321 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dbc1b7050 con 0x7f1dbc179fd0 2021-04-09T18:19:57.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.325 7f1db1ffb700 1 -- 172.21.15.32:0/2066779591 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1db8000c90 con 0x7f1dbc179fd0 2021-04-09T18:19:57.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.515 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dbc10d480 con 0x7f1dbc179fd0 2021-04-09T18:19:57.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.515 7f1db1ffb700 1 -- 172.21.15.32:0/2066779591 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f1db8002030 con 0x7f1dbc179fd0 2021-04-09T18:19:57.559 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:57.560 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:57.560 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:57.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1d9c018930 msgr2=0x7f1d9c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:57.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1d9c018930 0x7f1d9c01ae10 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc179fd0 msgr2=0x7f1dbc0976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:57.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc179fd0 0x7f1dbc0976a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f1db800b5e0 tx=0x7f1db800b980).stop 2021-04-09T18:19:57.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 shutdown_connections 2021-04-09T18:19:57.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1d9c018930 0x7f1d9c01ae10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dbc1a4b10 0x7f1dbc1a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dbc097cc0 0x7f1dbc1a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 --2- 172.21.15.32:0/2066779591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dbc179fd0 0x7f1dbc0976a0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:57.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 >> 172.21.15.32:0/2066779591 conn(0x7f1dbc199680 msgr2=0x7f1dbc1a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:57.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 shutdown_connections 2021-04-09T18:19:57.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:57.518 7f1dc3743700 1 -- 172.21.15.32:0/2066779591 wait complete. 2021-04-09T18:19:57.978 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:57.978 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:19:58.980 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:19:59.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.099 7f67a5eea700 1 Processor -- start 2021-04-09T18:19:59.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.099 7f67a5eea700 1 -- start start 2021-04-09T18:19:59.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.099 7f67a5eea700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67a00d7ca0 0x7f67a00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:59.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.099 7f67a5eea700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f67a010da00 0x7f67a010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:59.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f67a5eea700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a010d480 con 0x7f67a010d010 2021-04-09T18:19:59.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f67a5eea700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a00d9220 con 0x7f67a00d7ca0 2021-04-09T18:19:59.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f67a5eea700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a00d86c0 con 0x7f67a010da00 2021-04-09T18:19:59.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679effd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f67a00d7ca0 0x7f67a00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:59.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679f7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f67a010d010 0x7f67a00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57138/0 (socket says 172.21.15.32:57138) 2021-04-09T18:19:59.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679f7fe700 1 -- 172.21.15.32:0/3185381492 learned_addr learned my addr 172.21.15.32:0/3185381492 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:19:59.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679ffff700 1 --2- 172.21.15.32:0/3185381492 >> v2:172.21.15.86:3300/0 conn(0x7f67a010da00 0x7f67a010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:59.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679effd700 1 -- 172.21.15.32:0/3185381492 >> v2:172.21.15.86:3300/0 conn(0x7f67a010da00 msgr2=0x7f67a010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:19:59.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679effd700 1 --2- 172.21.15.32:0/3185381492 >> v2:172.21.15.86:3300/0 conn(0x7f67a010da00 0x7f67a010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679effd700 1 -- 172.21.15.32:0/3185381492 >> v1:172.21.15.32:6789/0 conn(0x7f67a010d010 legacy=0x7f67a00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:59.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679effd700 1 -- 172.21.15.32:0/3185381492 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a00d1f70 con 0x7f67a00d7ca0 2021-04-09T18:19:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679e7fc700 1 -- 172.21.15.32:0/3185381492 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304392713 0 0) 0x7f6790000ce0 con 0x7f67a010d010 2021-04-09T18:19:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679ffff700 1 --2- 172.21.15.32:0/3185381492 >> v2:172.21.15.86:3300/0 conn(0x7f67a010da00 0x7f67a010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:19:59.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679effd700 1 --2- 172.21.15.32:0/3185381492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a00d7ca0 0x7f67a00d80a0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f6788006650 tx=0x7f6788009a80).ready entity=mon.0 client_cookie=56d22bc2a526cf06 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:59.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679e7fc700 1 -- 172.21.15.32:0/3185381492 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67880009b0 con 0x7f67a00d7ca0 2021-04-09T18:19:59.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679e7fc700 1 -- 172.21.15.32:0/3185381492 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678800c0c0 con 0x7f67a00d7ca0 2021-04-09T18:19:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f679e7fc700 1 -- 172.21.15.32:0/3185381492 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f678800c510 con 0x7f67a00d7ca0 2021-04-09T18:19:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f67a5eea700 1 -- 172.21.15.32:0/3185381492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a00d7ca0 msgr2=0x7f67a00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.100 7f67a5eea700 1 --2- 172.21.15.32:0/3185381492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a00d7ca0 0x7f67a00d80a0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f6788006650 tx=0x7f6788009a80).stop 2021-04-09T18:19:59.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.101 7f67a5eea700 1 -- 172.21.15.32:0/3185381492 shutdown_connections 2021-04-09T18:19:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.101 7f67a5eea700 1 --2- 172.21.15.32:0/3185381492 >> v2:172.21.15.86:3300/0 conn(0x7f67a010da00 0x7f67a010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.101 7f67a5eea700 1 --2- 172.21.15.32:0/3185381492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a00d7ca0 0x7f67a00d80a0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.101 7f67a5eea700 1 -- 172.21.15.32:0/3185381492 >> 172.21.15.32:0/3185381492 conn(0x7f67a00ce5f0 msgr2=0x7f67a00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.101 7f67a5eea700 1 -- 172.21.15.32:0/3185381492 shutdown_connections 2021-04-09T18:19:59.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.101 7f67a5eea700 1 -- 172.21.15.32:0/3185381492 wait complete. 2021-04-09T18:19:59.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.121 7f67a5eea700 1 Processor -- start 2021-04-09T18:19:59.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.121 7f67a5eea700 1 -- start start 2021-04-09T18:19:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f67a5eea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67a0179fd0 0x7f67a00976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:59.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f67a5eea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a0097cc0 0x7f67a01a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f67a5eea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67a01a4b10 0x7f67a01a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f67a5eea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a0098140 con 0x7f67a0097cc0 2021-04-09T18:19:59.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f67a5eea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a017a3d0 con 0x7f67a01a4b10 2021-04-09T18:19:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f67a5eea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a017c870 con 0x7f67a0179fd0 2021-04-09T18:19:59.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f679effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a0097cc0 0x7f67a01a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:59.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f679effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a0097cc0 0x7f67a01a44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47556/0 (socket says 172.21.15.32:47556) 2021-04-09T18:19:59.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f679effd700 1 -- 172.21.15.32:0/2240091148 learned_addr learned my addr 172.21.15.32:0/2240091148 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f679f7fe700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67a0179fd0 0x7f67a00976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f679ffff700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67a01a4b10 0x7f67a01a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f679effd700 1 -- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67a0179fd0 msgr2=0x7f67a00976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:59.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f679effd700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67a0179fd0 0x7f67a00976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f679effd700 1 -- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67a01a4b10 msgr2=0x7f67a01a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.122 7f679effd700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67a01a4b10 0x7f67a01a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.123 7f679effd700 1 -- 172.21.15.32:0/2240091148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a0179bf0 con 0x7f67a0097cc0 2021-04-09T18:19:59.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.123 7f679effd700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a0097cc0 0x7f67a01a44f0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f678800b580 tx=0x7f678800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:59.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.123 7f6797fff700 1 -- 172.21.15.32:0/2240091148 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6788006440 con 0x7f67a0097cc0 2021-04-09T18:19:59.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.123 7f6797fff700 1 -- 172.21.15.32:0/2240091148 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6788006760 con 0x7f67a0097cc0 2021-04-09T18:19:59.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.123 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a0179bf0 con 0x7f67a0097cc0 2021-04-09T18:19:59.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.123 7f6797fff700 1 -- 172.21.15.32:0/2240091148 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67880052e0 con 0x7f67a0097cc0 2021-04-09T18:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.124 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a0179bf0 con 0x7f67a0097cc0 2021-04-09T18:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.125 7f6797fff700 1 -- 172.21.15.32:0/2240091148 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6788006030 con 0x7f67a0097cc0 2021-04-09T18:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.125 7f6797fff700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67840188c0 0x7f678401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.125 7f6797fff700 1 -- 172.21.15.32:0/2240091148 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f67880327c0 con 0x7f67a0097cc0 2021-04-09T18:19:59.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.125 7f679f7fe700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67840188c0 0x7f678401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:19:59.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.125 7f679f7fe700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67840188c0 0x7f678401adc0 crc :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:19:59.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.143 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a01b7050 con 0x7f67a0097cc0 2021-04-09T18:19:59.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.147 7f6797fff700 1 -- 172.21.15.32:0/2240091148 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6788006440 con 0x7f67a0097cc0 2021-04-09T18:19:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.339 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67a010d480 con 0x7f67a0097cc0 2021-04-09T18:19:59.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.339 7f6797fff700 1 -- 172.21.15.32:0/2240091148 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f6788006030 con 0x7f67a0097cc0 2021-04-09T18:19:59.384 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:19:59.385 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:19:59.385 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:19:59.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67840188c0 msgr2=0x7f678401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:59.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67840188c0 0x7f678401adc0 crc :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a0097cc0 msgr2=0x7f67a01a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:19:59.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a0097cc0 0x7f67a01a44f0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f678800b580 tx=0x7f678800b920).stop 2021-04-09T18:19:59.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 shutdown_connections 2021-04-09T18:19:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f67840188c0 0x7f678401adc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67a0097cc0 0x7f67a01a44f0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67a01a4b10 0x7f67a01a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 --2- 172.21.15.32:0/2240091148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67a0179fd0 0x7f67a00976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:19:59.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 >> 172.21.15.32:0/2240091148 conn(0x7f67a0199680 msgr2=0x7f67a01a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:19:59.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 shutdown_connections 2021-04-09T18:19:59.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:19:59.342 7f67a5eea700 1 -- 172.21.15.32:0/2240091148 wait complete. 2021-04-09T18:19:59.802 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:19:59.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:00.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:00.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.923 7fefd364a700 1 Processor -- start 2021-04-09T18:20:00.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.923 7fefd364a700 1 -- start start 2021-04-09T18:20:00.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd364a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fefcc10d010 0x7fefcc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:00.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd364a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fefcc0d1da0 0x7fefcc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:00.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd364a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc0d82c0 0x7fefcc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:00.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd364a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc0d8d30 con 0x7fefcc0d1da0 2021-04-09T18:20:00.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd364a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc0d9220 con 0x7fefcc10d010 2021-04-09T18:20:00.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd364a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc10d4b0 con 0x7fefcc0d82c0 2021-04-09T18:20:00.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd0c65700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fefcc0d1da0 0x7fefcc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:00.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd0c65700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fefcc0d1da0 0x7fefcc0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47566/0 (socket says 172.21.15.32:47566) 2021-04-09T18:20:01.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd0c65700 1 -- 172.21.15.32:0/3935482490 learned_addr learned my addr 172.21.15.32:0/3935482490 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:01.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd1466700 1 --2- 172.21.15.32:0/3935482490 >> v2:172.21.15.86:3300/0 conn(0x7fefcc10d010 0x7fefcc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:01.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.924 7fefd1c67700 1 --2- 172.21.15.32:0/3935482490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc0d82c0 0x7fefcc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:01.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefd0c65700 1 -- 172.21.15.32:0/3935482490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc0d82c0 msgr2=0x7fefcc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:01.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefd0c65700 1 --2- 172.21.15.32:0/3935482490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc0d82c0 0x7fefcc10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefd0c65700 1 -- 172.21.15.32:0/3935482490 >> v2:172.21.15.86:3300/0 conn(0x7fefcc10d010 msgr2=0x7fefcc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:01.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefd0c65700 1 --2- 172.21.15.32:0/3935482490 >> v2:172.21.15.86:3300/0 conn(0x7fefcc10d010 0x7fefcc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefd0c65700 1 -- 172.21.15.32:0/3935482490 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc0d86c0 con 0x7fefcc0d1da0 2021-04-09T18:20:01.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefd1466700 1 --2- 172.21.15.32:0/3935482490 >> v2:172.21.15.86:3300/0 conn(0x7fefcc10d010 0x7fefcc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:01.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefd0c65700 1 --2- 172.21.15.32:0/3935482490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc0d1da0 0x7fefcc0d7ca0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7fefbc006700 tx=0x7fefbc009b80).ready entity=mon.0 client_cookie=d3479264b29d27c2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:01.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefc3fff700 1 -- 172.21.15.32:0/3935482490 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefbc00c0c0 con 0x7fefcc0d1da0 2021-04-09T18:20:01.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefc3fff700 1 -- 172.21.15.32:0/3935482490 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefbc00c440 con 0x7fefcc0d1da0 2021-04-09T18:20:01.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefc3fff700 1 -- 172.21.15.32:0/3935482490 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefbc00c920 con 0x7fefcc0d1da0 2021-04-09T18:20:01.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefd364a700 1 -- 172.21.15.32:0/3935482490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc0d1da0 msgr2=0x7fefcc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:01.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.925 7fefd364a700 1 --2- 172.21.15.32:0/3935482490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc0d1da0 0x7fefcc0d7ca0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7fefbc006700 tx=0x7fefbc009b80).stop 2021-04-09T18:20:01.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.926 7fefd364a700 1 -- 172.21.15.32:0/3935482490 shutdown_connections 2021-04-09T18:20:01.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.926 7fefd364a700 1 --2- 172.21.15.32:0/3935482490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc0d82c0 0x7fefcc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.926 7fefd364a700 1 --2- 172.21.15.32:0/3935482490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc0d1da0 0x7fefcc0d7ca0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.926 7fefd364a700 1 --2- 172.21.15.32:0/3935482490 >> v2:172.21.15.86:3300/0 conn(0x7fefcc10d010 0x7fefcc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.926 7fefd364a700 1 -- 172.21.15.32:0/3935482490 >> 172.21.15.32:0/3935482490 conn(0x7fefcc0ce5f0 msgr2=0x7fefcc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:01.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.926 7fefd364a700 1 -- 172.21.15.32:0/3935482490 shutdown_connections 2021-04-09T18:20:01.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.926 7fefd364a700 1 -- 172.21.15.32:0/3935482490 wait complete. 2021-04-09T18:20:01.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.946 7fefd364a700 1 Processor -- start 2021-04-09T18:20:01.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.946 7fefd364a700 1 -- start start 2021-04-09T18:20:01.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd364a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc1aca20 0x7fefcc1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:01.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd364a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefcc18bd40 0x7fefcc18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:01.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd364a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc08fba0 0x7fefcc08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:01.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd364a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc18c7e0 con 0x7fefcc08fba0 2021-04-09T18:20:01.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd364a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc18ca30 con 0x7fefcc18bd40 2021-04-09T18:20:01.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd364a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0905f0 con 0x7fefcc1aca20 2021-04-09T18:20:01.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd1c67700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc08fba0 0x7fefcc08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:01.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd1c67700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc08fba0 0x7fefcc08ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47568/0 (socket says 172.21.15.32:47568) 2021-04-09T18:20:01.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd1c67700 1 -- 172.21.15.32:0/3923324520 learned_addr learned my addr 172.21.15.32:0/3923324520 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:01.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd1466700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc1aca20 0x7fefcc1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:01.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd0c65700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefcc18bd40 0x7fefcc18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:01.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd1c67700 1 -- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc1aca20 msgr2=0x7fefcc1ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:01.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd1c67700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc1aca20 0x7fefcc1ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd1c67700 1 -- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefcc18bd40 msgr2=0x7fefcc18c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:01.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd1c67700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefcc18bd40 0x7fefcc18c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.947 7fefd1c67700 1 -- 172.21.15.32:0/3923324520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc0b4300 con 0x7fefcc08fba0 2021-04-09T18:20:01.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.948 7fefd1c67700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc08fba0 0x7fefcc08ffa0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7fefc4005fd0 tx=0x7fefc400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:01.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.948 7fefc1ffb700 1 -- 172.21.15.32:0/3923324520 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc400c300 con 0x7fefcc08fba0 2021-04-09T18:20:01.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.948 7fefc1ffb700 1 -- 172.21.15.32:0/3923324520 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc4003030 con 0x7fefcc08fba0 2021-04-09T18:20:01.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.948 7fefd364a700 1 -- 172.21.15.32:0/3923324520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc0b4300 con 0x7fefcc08fba0 2021-04-09T18:20:01.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.948 7fefc1ffb700 1 -- 172.21.15.32:0/3923324520 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc400c7b0 con 0x7fefcc08fba0 2021-04-09T18:20:01.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.948 7fefd364a700 1 -- 172.21.15.32:0/3923324520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc0b4300 con 0x7fefcc08fba0 2021-04-09T18:20:01.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.948 7fefc1ffb700 1 -- 172.21.15.32:0/3923324520 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fefc40011f0 con 0x7fefcc08fba0 2021-04-09T18:20:01.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.949 7fefc1ffb700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fefac018930 0x7fefac01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:01.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.949 7fefc1ffb700 1 -- 172.21.15.32:0/3923324520 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fefc4032da0 con 0x7fefcc08fba0 2021-04-09T18:20:01.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.949 7fefd1466700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fefac018930 0x7fefac01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:01.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.949 7fefd1466700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fefac018930 0x7fefac01ae10 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:01.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.968 7fefd364a700 1 -- 172.21.15.32:0/3923324520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc182d70 con 0x7fefcc08fba0 2021-04-09T18:20:01.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:00.971 7fefc1ffb700 1 -- 172.21.15.32:0/3923324520 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fefc400ad90 con 0x7fefcc08fba0 2021-04-09T18:20:01.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.163 7fefd364a700 1 -- 172.21.15.32:0/3923324520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefcc18c760 con 0x7fefcc08fba0 2021-04-09T18:20:01.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.163 7fefc1ffb700 1 -- 172.21.15.32:0/3923324520 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fefc40011f0 con 0x7fefcc08fba0 2021-04-09T18:20:01.207 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:01.208 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:01.209 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:01.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 -- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fefac018930 msgr2=0x7fefac01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fefac018930 0x7fefac01ae10 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 -- 172.21.15.32:0/3923324520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc08fba0 msgr2=0x7fefcc08ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc08fba0 0x7fefcc08ffa0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7fefc4005fd0 tx=0x7fefc400ac50).stop 2021-04-09T18:20:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 -- 172.21.15.32:0/3923324520 shutdown_connections 2021-04-09T18:20:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fefac018930 0x7fefac01ae10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefcc08fba0 0x7fefcc08ffa0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefcc18bd40 0x7fefcc18c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 --2- 172.21.15.32:0/3923324520 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefcc1aca20 0x7fefcc1ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:01.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 -- 172.21.15.32:0/3923324520 >> 172.21.15.32:0/3923324520 conn(0x7fefcc19b460 msgr2=0x7fefcc19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:01.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 -- 172.21.15.32:0/3923324520 shutdown_connections 2021-04-09T18:20:01.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:01.166 7fefd364a700 1 -- 172.21.15.32:0/3923324520 wait complete. 2021-04-09T18:20:01.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:01.644 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:02.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:02.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.777 7fcd4354f700 1 Processor -- start 2021-04-09T18:20:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.777 7fcd4354f700 1 -- start start 2021-04-09T18:20:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd4354f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcd3c0d7ca0 0x7fcd3c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:02.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd4354f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c10da00 0x7fcd3c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:02.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd4354f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3c10d480 con 0x7fcd3c10d010 2021-04-09T18:20:02.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd4354f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3c0d9220 con 0x7fcd3c0d7ca0 2021-04-09T18:20:02.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd4354f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c0d86c0 con 0x7fcd3c10da00 2021-04-09T18:20:02.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd41b6c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c10da00 0x7fcd3c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd40b6a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcd3c0d7ca0 0x7fcd3c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd4136b700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcd3c10d010 0x7fcd3c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55092/0 (socket says 172.21.15.32:55092) 2021-04-09T18:20:02.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd41b6c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c10da00 0x7fcd3c10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:54804/0 (socket says 172.21.15.32:54804) 2021-04-09T18:20:02.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd40b6a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcd3c0d7ca0 0x7fcd3c0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:39338/0 (socket says 172.21.15.32:39338) 2021-04-09T18:20:02.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd4136b700 1 -- 172.21.15.32:0/3704021699 learned_addr learned my addr 172.21.15.32:0/3704021699 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:02.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd41b6c700 1 -- 172.21.15.32:0/3704021699 >> v2:172.21.15.86:3300/0 conn(0x7fcd3c0d7ca0 msgr2=0x7fcd3c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:02.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd41b6c700 1 --2- 172.21.15.32:0/3704021699 >> v2:172.21.15.86:3300/0 conn(0x7fcd3c0d7ca0 0x7fcd3c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:02.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd41b6c700 1 -- 172.21.15.32:0/3704021699 >> v1:172.21.15.86:6789/0 conn(0x7fcd3c10d010 legacy=0x7fcd3c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:02.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd41b6c700 1 -- 172.21.15.32:0/3704021699 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c0d1f70 con 0x7fcd3c10da00 2021-04-09T18:20:02.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd33fff700 1 -- 172.21.15.32:0/3704021699 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223858359 0 0) 0x7fcd38001440 con 0x7fcd3c10d010 2021-04-09T18:20:02.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd40b6a700 1 --2- 172.21.15.32:0/3704021699 >> v2:172.21.15.86:3300/0 conn(0x7fcd3c0d7ca0 0x7fcd3c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:02.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.778 7fcd41b6c700 1 --2- 172.21.15.32:0/3704021699 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c10da00 0x7fcd3c10de00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fcd34006ad0 tx=0x7fcd34009ad0).ready entity=mon.2 client_cookie=172edd952eefe726 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:02.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.779 7fcd33fff700 1 -- 172.21.15.32:0/3704021699 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd340022e0 con 0x7fcd3c10da00 2021-04-09T18:20:02.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.779 7fcd33fff700 1 -- 172.21.15.32:0/3704021699 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd34006410 con 0x7fcd3c10da00 2021-04-09T18:20:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.779 7fcd33fff700 1 -- 172.21.15.32:0/3704021699 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd34002620 con 0x7fcd3c10da00 2021-04-09T18:20:02.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.779 7fcd4354f700 1 -- 172.21.15.32:0/3704021699 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c10da00 msgr2=0x7fcd3c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.779 7fcd4354f700 1 --2- 172.21.15.32:0/3704021699 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c10da00 0x7fcd3c10de00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fcd34006ad0 tx=0x7fcd34009ad0).stop 2021-04-09T18:20:02.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.779 7fcd4354f700 1 -- 172.21.15.32:0/3704021699 shutdown_connections 2021-04-09T18:20:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.779 7fcd4354f700 1 --2- 172.21.15.32:0/3704021699 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c10da00 0x7fcd3c10de00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:02.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.779 7fcd4354f700 1 --2- 172.21.15.32:0/3704021699 >> v2:172.21.15.86:3300/0 conn(0x7fcd3c0d7ca0 0x7fcd3c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:02.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.779 7fcd4354f700 1 -- 172.21.15.32:0/3704021699 >> 172.21.15.32:0/3704021699 conn(0x7fcd3c0ce5f0 msgr2=0x7fcd3c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:02.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.780 7fcd4354f700 1 -- 172.21.15.32:0/3704021699 shutdown_connections 2021-04-09T18:20:02.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.780 7fcd4354f700 1 -- 172.21.15.32:0/3704021699 wait complete. 2021-04-09T18:20:02.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.800 7fcd4354f700 1 Processor -- start 2021-04-09T18:20:02.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.800 7fcd4354f700 1 -- start start 2021-04-09T18:20:02.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.800 7fcd4354f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3c179fd0 0x7fcd3c0976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:02.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.800 7fcd4354f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd3c097cc0 0x7fcd3c1a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:02.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.800 7fcd4354f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c1a4b10 0x7fcd3c1a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:02.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd4354f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c098140 con 0x7fcd3c097cc0 2021-04-09T18:20:02.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd4354f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c17a3d0 con 0x7fcd3c179fd0 2021-04-09T18:20:02.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd4354f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c17c870 con 0x7fcd3c1a4b10 2021-04-09T18:20:02.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd40b6a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd3c097cc0 0x7fcd3c1a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:02.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd40b6a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd3c097cc0 0x7fcd3c1a44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47582/0 (socket says 172.21.15.32:47582) 2021-04-09T18:20:02.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd40b6a700 1 -- 172.21.15.32:0/4200928036 learned_addr learned my addr 172.21.15.32:0/4200928036 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:02.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd41b6c700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c1a4b10 0x7fcd3c1a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:02.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd4136b700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3c179fd0 0x7fcd3c0976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:02.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd40b6a700 1 -- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c1a4b10 msgr2=0x7fcd3c1a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:02.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd40b6a700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c1a4b10 0x7fcd3c1a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:02.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd40b6a700 1 -- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3c179fd0 msgr2=0x7fcd3c0976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:02.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd40b6a700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3c179fd0 0x7fcd3c0976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:02.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd40b6a700 1 -- 172.21.15.32:0/4200928036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c179bf0 con 0x7fcd3c097cc0 2021-04-09T18:20:02.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd40b6a700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd3c097cc0 0x7fcd3c1a44f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7fcd2c00b580 tx=0x7fcd2c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:02.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd31ffb700 1 -- 172.21.15.32:0/4200928036 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd2c00bd10 con 0x7fcd3c097cc0 2021-04-09T18:20:02.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd31ffb700 1 -- 172.21.15.32:0/4200928036 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c000c90 con 0x7fcd3c097cc0 2021-04-09T18:20:02.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3c179bf0 con 0x7fcd3c097cc0 2021-04-09T18:20:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd31ffb700 1 -- 172.21.15.32:0/4200928036 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd2c002220 con 0x7fcd3c097cc0 2021-04-09T18:20:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.801 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3c179bf0 con 0x7fcd3c097cc0 2021-04-09T18:20:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.803 7fcd31ffb700 1 -- 172.21.15.32:0/4200928036 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcd2c000c90 con 0x7fcd3c097cc0 2021-04-09T18:20:02.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.803 7fcd31ffb700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd1c0188c0 0x7fcd1c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:02.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.803 7fcd31ffb700 1 -- 172.21.15.32:0/4200928036 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcd2c0327c0 con 0x7fcd3c097cc0 2021-04-09T18:20:02.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.803 7fcd4136b700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd1c0188c0 0x7fcd1c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:02.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.803 7fcd4136b700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd1c0188c0 0x7fcd1c01adc0 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:02.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.822 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3c1b7050 con 0x7fcd3c097cc0 2021-04-09T18:20:02.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:02.825 7fcd31ffb700 1 -- 172.21.15.32:0/4200928036 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcd2c006b00 con 0x7fcd3c097cc0 2021-04-09T18:20:03.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.016 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd3c10d480 con 0x7fcd3c097cc0 2021-04-09T18:20:03.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.017 7fcd31ffb700 1 -- 172.21.15.32:0/4200928036 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fcd2c00bd10 con 0x7fcd3c097cc0 2021-04-09T18:20:03.061 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:03.061 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:03.061 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:03.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd1c0188c0 msgr2=0x7fcd1c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:03.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd1c0188c0 0x7fcd1c01adc0 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:03.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd3c097cc0 msgr2=0x7fcd3c1a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:03.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd3c097cc0 0x7fcd3c1a44f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7fcd2c00b580 tx=0x7fcd2c00b920).stop 2021-04-09T18:20:03.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 shutdown_connections 2021-04-09T18:20:03.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd1c0188c0 0x7fcd1c01adc0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd3c097cc0 0x7fcd3c1a44f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3c1a4b10 0x7fcd3c1a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 --2- 172.21.15.32:0/4200928036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3c179fd0 0x7fcd3c0976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 >> 172.21.15.32:0/4200928036 conn(0x7fcd3c199680 msgr2=0x7fcd3c1a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 shutdown_connections 2021-04-09T18:20:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:03.019 7fcd4354f700 1 -- 172.21.15.32:0/4200928036 wait complete. 2021-04-09T18:20:03.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:03.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:04.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:04.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.591 7f825769f700 1 Processor -- start 2021-04-09T18:20:04.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.592 7f825769f700 1 -- start start 2021-04-09T18:20:04.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.592 7f825769f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f825010d010 0x7f82500d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:04.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.592 7f825769f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82500d8490 0x7f825010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:04.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.592 7f825769f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825010d480 con 0x7f82500d1da0 2021-04-09T18:20:04.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.592 7f825769f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825010e780 con 0x7f82500d8490 2021-04-09T18:20:04.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.592 7f825769f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82500d8d30 con 0x7f825010d010 2021-04-09T18:20:04.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.592 7f8255cbc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82500d8490 0x7f825010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:04.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.592 7f8255cbc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82500d8490 0x7f825010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47592/0 (socket says 172.21.15.32:47592) 2021-04-09T18:20:04.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.592 7f8255cbc700 1 -- 172.21.15.32:0/20302170 learned_addr learned my addr 172.21.15.32:0/20302170 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:04.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f82554bb700 1 --2- 172.21.15.32:0/20302170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f825010d010 0x7f82500d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:04.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f8255cbc700 1 -- 172.21.15.32:0/20302170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f825010d010 msgr2=0x7f82500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:04.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f8255cbc700 1 --2- 172.21.15.32:0/20302170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f825010d010 0x7f82500d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:04.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f8255cbc700 1 -- 172.21.15.32:0/20302170 >> v1:172.21.15.86:6789/0 conn(0x7f82500d1da0 legacy=0x7f82500d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:04.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f8255cbc700 1 -- 172.21.15.32:0/20302170 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82500d9220 con 0x7f82500d8490 2021-04-09T18:20:04.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f8255cbc700 1 --2- 172.21.15.32:0/20302170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500d8490 0x7f825010da00 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f8248006b80 tx=0x7f8248009b10).ready entity=mon.0 client_cookie=4e57e4611f46fdff server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:04.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f8247fff700 1 -- 172.21.15.32:0/20302170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82480022e0 con 0x7f82500d8490 2021-04-09T18:20:04.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f8247fff700 1 -- 172.21.15.32:0/20302170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8248002620 con 0x7f82500d8490 2021-04-09T18:20:04.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f8247fff700 1 -- 172.21.15.32:0/20302170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8248002b00 con 0x7f82500d8490 2021-04-09T18:20:04.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f825769f700 1 -- 172.21.15.32:0/20302170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500d8490 msgr2=0x7f825010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:04.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.593 7f825769f700 1 --2- 172.21.15.32:0/20302170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500d8490 0x7f825010da00 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f8248006b80 tx=0x7f8248009b10).stop 2021-04-09T18:20:04.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.594 7f825769f700 1 -- 172.21.15.32:0/20302170 shutdown_connections 2021-04-09T18:20:04.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.594 7f825769f700 1 --2- 172.21.15.32:0/20302170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500d8490 0x7f825010da00 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:04.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.594 7f825769f700 1 --2- 172.21.15.32:0/20302170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f825010d010 0x7f82500d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:04.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.594 7f825769f700 1 -- 172.21.15.32:0/20302170 >> 172.21.15.32:0/20302170 conn(0x7f82500ce5f0 msgr2=0x7f82500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:04.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.594 7f825769f700 1 -- 172.21.15.32:0/20302170 shutdown_connections 2021-04-09T18:20:04.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.594 7f825769f700 1 -- 172.21.15.32:0/20302170 wait complete. 2021-04-09T18:20:04.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.614 7f825769f700 1 Processor -- start 2021-04-09T18:20:04.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.614 7f825769f700 1 -- start start 2021-04-09T18:20:04.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f825769f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500ad160 0x7f82500ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:04.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f825769f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8250179650 0x7f8250179a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:04.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f825769f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f825018dd40 0x7f825018e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:04.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f825769f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825017a070 con 0x7f82500ad160 2021-04-09T18:20:04.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f825769f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8250182e30 con 0x7f825018dd40 2021-04-09T18:20:04.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f825769f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8250182920 con 0x7f8250179650 2021-04-09T18:20:04.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f82554bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500ad160 0x7f82500ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:04.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f82554bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500ad160 0x7f82500ad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47596/0 (socket says 172.21.15.32:47596) 2021-04-09T18:20:04.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f82554bb700 1 -- 172.21.15.32:0/1892446675 learned_addr learned my addr 172.21.15.32:0/1892446675 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:04.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f8255cbc700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f825018dd40 0x7f825018e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:04.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f82554bb700 1 -- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8250179650 msgr2=0x7f8250179a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:04.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f82554bb700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8250179650 0x7f8250179a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:20:04.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f82554bb700 1 -- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f825018dd40 msgr2=0x7f825018e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:04.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f82554bb700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f825018dd40 0x7f825018e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:04.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f82554bb700 1 -- 172.21.15.32:0/1892446675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825018e760 con 0x7f82500ad160 2021-04-09T18:20:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.615 7f82554bb700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500ad160 0x7f82500ad560 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f824c00b970 tx=0x7f824c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.616 7f8245ffb700 1 -- 172.21.15.32:0/1892446675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f824c006750 con 0x7f82500ad160 2021-04-09T18:20:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.616 7f8245ffb700 1 -- 172.21.15.32:0/1892446675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824c007070 con 0x7f82500ad160 2021-04-09T18:20:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.616 7f825769f700 1 -- 172.21.15.32:0/1892446675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825018e760 con 0x7f82500ad160 2021-04-09T18:20:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.616 7f8245ffb700 1 -- 172.21.15.32:0/1892446675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f824c007540 con 0x7f82500ad160 2021-04-09T18:20:04.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.616 7f825769f700 1 -- 172.21.15.32:0/1892446675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f825018e760 con 0x7f82500ad160 2021-04-09T18:20:04.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.616 7f8245ffb700 1 -- 172.21.15.32:0/1892446675 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f824c007250 con 0x7f82500ad160 2021-04-09T18:20:04.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.616 7f8245ffb700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8230018930 0x7f823001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:04.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.616 7f8245ffb700 1 -- 172.21.15.32:0/1892446675 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f824c032da0 con 0x7f82500ad160 2021-04-09T18:20:04.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.617 7f8254cba700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8230018930 0x7f823001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:04.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.617 7f8254cba700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8230018930 0x7f823001ae10 crc :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:04.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.636 7f825769f700 1 -- 172.21.15.32:0/1892446675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82501b7050 con 0x7f82500ad160 2021-04-09T18:20:04.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.640 7f8245ffb700 1 -- 172.21.15.32:0/1892446675 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f824c000cc0 con 0x7f82500ad160 2021-04-09T18:20:04.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.831 7f825769f700 1 -- 172.21.15.32:0/1892446675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82501b7050 con 0x7f82500ad160 2021-04-09T18:20:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.831 7f8245ffb700 1 -- 172.21.15.32:0/1892446675 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f824c007070 con 0x7f82500ad160 2021-04-09T18:20:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:04.877 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:04.878 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 -- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8230018930 msgr2=0x7f823001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8230018930 0x7f823001ae10 crc :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 -- 172.21.15.32:0/1892446675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500ad160 msgr2=0x7f82500ad560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500ad160 0x7f82500ad560 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f824c00b970 tx=0x7f824c00bd10).stop 2021-04-09T18:20:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 -- 172.21.15.32:0/1892446675 shutdown_connections 2021-04-09T18:20:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8230018930 0x7f823001ae10 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:04.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f825018dd40 0x7f825018e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:04.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8250179650 0x7f8250179a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:20:04.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 --2- 172.21.15.32:0/1892446675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82500ad160 0x7f82500ad560 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:04.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 -- 172.21.15.32:0/1892446675 >> 172.21.15.32:0/1892446675 conn(0x7f825008c950 msgr2=0x7f82501a4800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 -- 172.21.15.32:0/1892446675 shutdown_connections 2021-04-09T18:20:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:04.834 7f825769f700 1 -- 172.21.15.32:0/1892446675 wait complete. 2021-04-09T18:20:05.299 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:05.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:05.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:20:05.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:20:06.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:06.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.423 7faee191f700 1 Processor -- start 2021-04-09T18:20:06.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faee191f700 1 -- start start 2021-04-09T18:20:06.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faee191f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faedc10d010 0x7faedc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:06.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faee191f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc0d8490 0x7faedc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:06.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faee191f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc10d480 con 0x7faedc0d1da0 2021-04-09T18:20:06.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faee191f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faedc0d8d30 con 0x7faedc10d010 2021-04-09T18:20:06.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faee191f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0d9220 con 0x7faedc0d8490 2021-04-09T18:20:06.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faedb574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faedc10d010 0x7faedc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:06.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faedad73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faedc0d1da0 0x7faedc0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57198/0 (socket says 172.21.15.32:57198) 2021-04-09T18:20:06.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faedad73700 1 -- 172.21.15.32:0/2031613538 learned_addr learned my addr 172.21.15.32:0/2031613538 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:06.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.424 7faedbd75700 1 --2- 172.21.15.32:0/2031613538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc0d8490 0x7faedc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:06.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faeda572700 1 -- 172.21.15.32:0/2031613538 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508331577 0 0) 0x7faed00009b0 con 0x7faedc0d1da0 2021-04-09T18:20:06.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faeda572700 1 -- 172.21.15.32:0/2031613538 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec4002fe0 con 0x7faedc0d1da0 2021-04-09T18:20:06.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faedb574700 1 -- 172.21.15.32:0/2031613538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc0d8490 msgr2=0x7faedc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:06.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faedb574700 1 --2- 172.21.15.32:0/2031613538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc0d8490 0x7faedc10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faedb574700 1 -- 172.21.15.32:0/2031613538 >> v1:172.21.15.32:6789/0 conn(0x7faedc0d1da0 legacy=0x7faedc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:06.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faedb574700 1 -- 172.21.15.32:0/2031613538 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc10e020 con 0x7faedc10d010 2021-04-09T18:20:06.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faedbd75700 1 --2- 172.21.15.32:0/2031613538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc0d8490 0x7faedc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:06.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faedb574700 1 --2- 172.21.15.32:0/2031613538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc10d010 0x7faedc0d1780 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7faecc006650 tx=0x7faecc009a80).ready entity=mon.0 client_cookie=77809d165b91aed4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:06.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faeda572700 1 -- 172.21.15.32:0/2031613538 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faecc0009b0 con 0x7faedc10d010 2021-04-09T18:20:06.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faeda572700 1 -- 172.21.15.32:0/2031613538 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc00c0c0 con 0x7faedc10d010 2021-04-09T18:20:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faeda572700 1 -- 172.21.15.32:0/2031613538 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faecc00c510 con 0x7faedc10d010 2021-04-09T18:20:06.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faee191f700 1 -- 172.21.15.32:0/2031613538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc10d010 msgr2=0x7faedc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:06.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.425 7faee191f700 1 --2- 172.21.15.32:0/2031613538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc10d010 0x7faedc0d1780 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7faecc006650 tx=0x7faecc009a80).stop 2021-04-09T18:20:06.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.426 7faee191f700 1 -- 172.21.15.32:0/2031613538 shutdown_connections 2021-04-09T18:20:06.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.426 7faee191f700 1 --2- 172.21.15.32:0/2031613538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc0d8490 0x7faedc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.426 7faee191f700 1 --2- 172.21.15.32:0/2031613538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc10d010 0x7faedc0d1780 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.426 7faee191f700 1 -- 172.21.15.32:0/2031613538 >> 172.21.15.32:0/2031613538 conn(0x7faedc0ce5f0 msgr2=0x7faedc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.426 7faee191f700 1 -- 172.21.15.32:0/2031613538 shutdown_connections 2021-04-09T18:20:06.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.426 7faee191f700 1 -- 172.21.15.32:0/2031613538 wait complete. 2021-04-09T18:20:06.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.446 7faee191f700 1 Processor -- start 2021-04-09T18:20:06.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.446 7faee191f700 1 -- start start 2021-04-09T18:20:06.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faee191f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc0a90c0 0x7faedc1a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:06.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faee191f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faedc1a76d0 0x7faedc1a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:06.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faee191f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc17bce0 0x7faedc17c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:06.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faee191f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc1a7c60 con 0x7faedc0a90c0 2021-04-09T18:20:06.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faee191f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc1a8e20 con 0x7faedc1a76d0 2021-04-09T18:20:06.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faee191f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc17c700 con 0x7faedc17bce0 2021-04-09T18:20:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedb574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc0a90c0 0x7faedc1a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedb574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc0a90c0 0x7faedc1a81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47610/0 (socket says 172.21.15.32:47610) 2021-04-09T18:20:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedb574700 1 -- 172.21.15.32:0/1440278103 learned_addr learned my addr 172.21.15.32:0/1440278103 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedbd75700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc17bce0 0x7faedc17c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedad73700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faedc1a76d0 0x7faedc1a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedb574700 1 -- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc17bce0 msgr2=0x7faedc17c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedb574700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc17bce0 0x7faedc17c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedb574700 1 -- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faedc1a76d0 msgr2=0x7faedc1a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedb574700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faedc1a76d0 0x7faedc1a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedb574700 1 -- 172.21.15.32:0/1440278103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc17c920 con 0x7faedc0a90c0 2021-04-09T18:20:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.447 7faedb574700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc0a90c0 0x7faedc1a81e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7faecc009650 tx=0x7faecc00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.448 7faec3fff700 1 -- 172.21.15.32:0/1440278103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faecc0093a0 con 0x7faedc0a90c0 2021-04-09T18:20:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.448 7faec3fff700 1 -- 172.21.15.32:0/1440278103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc006070 con 0x7faedc0a90c0 2021-04-09T18:20:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.448 7faee191f700 1 -- 172.21.15.32:0/1440278103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc17c920 con 0x7faedc0a90c0 2021-04-09T18:20:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.448 7faec3fff700 1 -- 172.21.15.32:0/1440278103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faecc0052e0 con 0x7faedc0a90c0 2021-04-09T18:20:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.448 7faee191f700 1 -- 172.21.15.32:0/1440278103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faedc17c920 con 0x7faedc0a90c0 2021-04-09T18:20:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.448 7faec3fff700 1 -- 172.21.15.32:0/1440278103 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faecc009300 con 0x7faedc0a90c0 2021-04-09T18:20:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.448 7faec3fff700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faebc018930 0x7faebc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.448 7faec3fff700 1 -- 172.21.15.32:0/1440278103 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7faecc032a00 con 0x7faedc0a90c0 2021-04-09T18:20:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.449 7faedad73700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faebc018930 0x7faebc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:06.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.449 7faedad73700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faebc018930 0x7faebc01ae10 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.468 7faee191f700 1 -- 172.21.15.32:0/1440278103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faedc1b7010 con 0x7faedc0a90c0 2021-04-09T18:20:06.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.472 7faec3fff700 1 -- 172.21.15.32:0/1440278103 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faecc006030 con 0x7faedc0a90c0 2021-04-09T18:20:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.663 7faee191f700 1 -- 172.21.15.32:0/1440278103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faedc10d480 con 0x7faedc0a90c0 2021-04-09T18:20:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.664 7faec3fff700 1 -- 172.21.15.32:0/1440278103 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7faecc000940 con 0x7faedc0a90c0 2021-04-09T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:06.708 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 -- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faebc018930 msgr2=0x7faebc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faebc018930 0x7faebc01ae10 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 -- 172.21.15.32:0/1440278103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc0a90c0 msgr2=0x7faedc1a81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc0a90c0 0x7faedc1a81e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7faecc009650 tx=0x7faecc00b8e0).stop 2021-04-09T18:20:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 -- 172.21.15.32:0/1440278103 shutdown_connections 2021-04-09T18:20:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faebc018930 0x7faebc01ae10 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faedc17bce0 0x7faedc17c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faedc1a76d0 0x7faedc1a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 --2- 172.21.15.32:0/1440278103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faedc0a90c0 0x7faedc1a81e0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 -- 172.21.15.32:0/1440278103 >> 172.21.15.32:0/1440278103 conn(0x7faedc1acae0 msgr2=0x7faedc17ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 -- 172.21.15.32:0/1440278103 shutdown_connections 2021-04-09T18:20:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:06.666 7faee191f700 1 -- 172.21.15.32:0/1440278103 wait complete. 2021-04-09T18:20:07.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:07.123 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:08.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:08.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f995ac7c700 1 Processor -- start 2021-04-09T18:20:08.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f995ac7c700 1 -- start start 2021-04-09T18:20:08.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f995ac7c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99540d7ca0 0x7f99540d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:08.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f995ac7c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f995410da00 0x7f995410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:08.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f995ac7c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995410d480 con 0x7f995410d010 2021-04-09T18:20:08.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f995ac7c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99540d9220 con 0x7f995410da00 2021-04-09T18:20:08.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f995ac7c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99540d86c0 con 0x7f99540d7ca0 2021-04-09T18:20:08.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f9959299700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f995410da00 0x7f995410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:08.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f9959299700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f995410da00 0x7f995410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47618/0 (socket says 172.21.15.32:47618) 2021-04-09T18:20:08.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.247 7f9959299700 1 -- 172.21.15.32:0/2944781497 learned_addr learned my addr 172.21.15.32:0/2944781497 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:08.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f9953fff700 1 --2- 172.21.15.32:0/2944781497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99540d7ca0 0x7f99540d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:08.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f9959299700 1 -- 172.21.15.32:0/2944781497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99540d7ca0 msgr2=0x7f99540d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:08.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f9959299700 1 --2- 172.21.15.32:0/2944781497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99540d7ca0 0x7f99540d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f9959299700 1 -- 172.21.15.32:0/2944781497 >> v1:172.21.15.86:6789/0 conn(0x7f995410d010 legacy=0x7f99540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:08.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f9959299700 1 -- 172.21.15.32:0/2944781497 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99540d1f70 con 0x7f995410da00 2021-04-09T18:20:08.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f9959299700 1 --2- 172.21.15.32:0/2944781497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995410da00 0x7f995410de00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7f994c006b80 tx=0x7f994c009b90).ready entity=mon.0 client_cookie=a544f788109cdc07 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:08.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f99537fe700 1 -- 172.21.15.32:0/2944781497 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f994c0022e0 con 0x7f995410da00 2021-04-09T18:20:08.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f99537fe700 1 -- 172.21.15.32:0/2944781497 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994c002620 con 0x7f995410da00 2021-04-09T18:20:08.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f99537fe700 1 -- 172.21.15.32:0/2944781497 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f994c002b00 con 0x7f995410da00 2021-04-09T18:20:08.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f995ac7c700 1 -- 172.21.15.32:0/2944781497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995410da00 msgr2=0x7f995410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:08.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.248 7f995ac7c700 1 --2- 172.21.15.32:0/2944781497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995410da00 0x7f995410de00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7f994c006b80 tx=0x7f994c009b90).stop 2021-04-09T18:20:08.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.249 7f995ac7c700 1 -- 172.21.15.32:0/2944781497 shutdown_connections 2021-04-09T18:20:08.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.249 7f995ac7c700 1 --2- 172.21.15.32:0/2944781497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995410da00 0x7f995410de00 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.249 7f995ac7c700 1 --2- 172.21.15.32:0/2944781497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99540d7ca0 0x7f99540d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.249 7f995ac7c700 1 -- 172.21.15.32:0/2944781497 >> 172.21.15.32:0/2944781497 conn(0x7f99540ce5f0 msgr2=0x7f99540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:08.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.249 7f995ac7c700 1 -- 172.21.15.32:0/2944781497 shutdown_connections 2021-04-09T18:20:08.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.249 7f995ac7c700 1 -- 172.21.15.32:0/2944781497 wait complete. 2021-04-09T18:20:08.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.269 7f995ac7c700 1 Processor -- start 2021-04-09T18:20:08.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.269 7f995ac7c700 1 -- start start 2021-04-09T18:20:08.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f995ac7c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99541889c0 0x7f995409a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:08.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f995ac7c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995418e2c0 0x7f99541aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:08.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f995ac7c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99541aaad0 0x7f99540a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:08.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f995ac7c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99541aaed0 con 0x7f995418e2c0 2021-04-09T18:20:08.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f995ac7c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99541ab0f0 con 0x7f99541aaad0 2021-04-09T18:20:08.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f995ac7c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99540a3ed0 con 0x7f99541889c0 2021-04-09T18:20:08.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9953fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995418e2c0 0x7f99541aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:08.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9953fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995418e2c0 0x7f99541aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47622/0 (socket says 172.21.15.32:47622) 2021-04-09T18:20:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9953fff700 1 -- 172.21.15.32:0/3222465527 learned_addr learned my addr 172.21.15.32:0/3222465527 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9959299700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99541aaad0 0x7f99540a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:08.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9958a98700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99541889c0 0x7f995409a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:08.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9953fff700 1 -- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99541889c0 msgr2=0x7f995409a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:08.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9953fff700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99541889c0 0x7f995409a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9953fff700 1 -- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99541aaad0 msgr2=0x7f99540a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:08.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9953fff700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99541aaad0 0x7f99540a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9953fff700 1 -- 172.21.15.32:0/3222465527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99540a40f0 con 0x7f995418e2c0 2021-04-09T18:20:08.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.270 7f9953fff700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995418e2c0 0x7f99541aa4b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f993c006f60 tx=0x7f993c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:08.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.271 7f99517fa700 1 -- 172.21.15.32:0/3222465527 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f993c00bd60 con 0x7f995418e2c0 2021-04-09T18:20:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.271 7f99517fa700 1 -- 172.21.15.32:0/3222465527 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c000c90 con 0x7f995418e2c0 2021-04-09T18:20:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.271 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99540a40f0 con 0x7f995418e2c0 2021-04-09T18:20:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.271 7f99517fa700 1 -- 172.21.15.32:0/3222465527 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f993c002220 con 0x7f995418e2c0 2021-04-09T18:20:08.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.271 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99540a40f0 con 0x7f995418e2c0 2021-04-09T18:20:08.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.271 7f99517fa700 1 -- 172.21.15.32:0/3222465527 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f993c0052e0 con 0x7f995418e2c0 2021-04-09T18:20:08.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.271 7f99517fa700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9938018930 0x7f993801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:08.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.271 7f99517fa700 1 -- 172.21.15.32:0/3222465527 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f993c0329f0 con 0x7f995418e2c0 2021-04-09T18:20:08.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.272 7f9958a98700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9938018930 0x7f993801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:08.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.272 7f9958a98700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9938018930 0x7f993801adc0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:08.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.291 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995410d480 con 0x7f995418e2c0 2021-04-09T18:20:08.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.295 7f99517fa700 1 -- 172.21.15.32:0/3222465527 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f993c000c90 con 0x7f995418e2c0 2021-04-09T18:20:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.486 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f995410d480 con 0x7f995418e2c0 2021-04-09T18:20:08.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.486 7f99517fa700 1 -- 172.21.15.32:0/3222465527 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f993c006b00 con 0x7f995418e2c0 2021-04-09T18:20:08.531 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:08.532 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:08.533 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:08.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9938018930 msgr2=0x7f993801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:08.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9938018930 0x7f993801adc0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995418e2c0 msgr2=0x7f99541aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:08.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995418e2c0 0x7f99541aa4b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f993c006f60 tx=0x7f993c00b970).stop 2021-04-09T18:20:08.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 shutdown_connections 2021-04-09T18:20:08.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9938018930 0x7f993801adc0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f995418e2c0 0x7f99541aa4b0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99541aaad0 0x7f99540a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 --2- 172.21.15.32:0/3222465527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99541889c0 0x7f995409a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:08.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 >> 172.21.15.32:0/3222465527 conn(0x7f99541a0db0 msgr2=0x7f99541a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 shutdown_connections 2021-04-09T18:20:08.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:08.489 7f995ac7c700 1 -- 172.21.15.32:0/3222465527 wait complete. 2021-04-09T18:20:08.950 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:08.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:09.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:10.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.070 7fe84ff1f700 1 Processor -- start 2021-04-09T18:20:10.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.070 7fe84ff1f700 1 -- start start 2021-04-09T18:20:10.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.070 7fe84ff1f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe84810d010 0x7fe8480d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:10.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.070 7fe84ff1f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8480d8490 0x7fe84810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:10.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84ff1f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84810d480 con 0x7fe8480d1da0 2021-04-09T18:20:10.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84ff1f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8480d8d30 con 0x7fe84810d010 2021-04-09T18:20:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84ff1f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8480d9220 con 0x7fe8480d8490 2021-04-09T18:20:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84e53c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8480d8490 0x7fe84810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:10.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84dd3b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe84810d010 0x7fe8480d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84e53c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8480d8490 0x7fe84810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:54858/0 (socket says 172.21.15.32:54858) 2021-04-09T18:20:10.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84dd3b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe84810d010 0x7fe8480d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:39388/0 (socket says 172.21.15.32:39388) 2021-04-09T18:20:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84e53c700 1 -- 172.21.15.32:0/1812043012 learned_addr learned my addr 172.21.15.32:0/1812043012 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84cd39700 1 -- 172.21.15.32:0/1812043012 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004720537 0 0) 0x7fe838000c60 con 0x7fe8480d1da0 2021-04-09T18:20:10.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84cd39700 1 -- 172.21.15.32:0/1812043012 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe834002fe0 con 0x7fe8480d1da0 2021-04-09T18:20:10.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84dd3b700 1 -- 172.21.15.32:0/1812043012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8480d8490 msgr2=0x7fe84810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:10.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84dd3b700 1 --2- 172.21.15.32:0/1812043012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8480d8490 0x7fe84810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84dd3b700 1 -- 172.21.15.32:0/1812043012 >> v1:172.21.15.86:6789/0 conn(0x7fe8480d1da0 legacy=0x7fe8480d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84dd3b700 1 -- 172.21.15.32:0/1812043012 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84810e020 con 0x7fe84810d010 2021-04-09T18:20:10.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84e53c700 1 --2- 172.21.15.32:0/1812043012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8480d8490 0x7fe84810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:10.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.071 7fe84dd3b700 1 --2- 172.21.15.32:0/1812043012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe84810d010 0x7fe8480d1780 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fe844006650 tx=0x7fe844009a80).ready entity=mon.1 client_cookie=abe85e1019501e26 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:10.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.072 7fe84cd39700 1 -- 172.21.15.32:0/1812043012 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8440009b0 con 0x7fe84810d010 2021-04-09T18:20:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.072 7fe84cd39700 1 -- 172.21.15.32:0/1812043012 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84400c0c0 con 0x7fe84810d010 2021-04-09T18:20:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.072 7fe84cd39700 1 -- 172.21.15.32:0/1812043012 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe84400c510 con 0x7fe84810d010 2021-04-09T18:20:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.072 7fe84ff1f700 1 -- 172.21.15.32:0/1812043012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe84810d010 msgr2=0x7fe8480d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.072 7fe84ff1f700 1 --2- 172.21.15.32:0/1812043012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe84810d010 0x7fe8480d1780 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fe844006650 tx=0x7fe844009a80).stop 2021-04-09T18:20:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.072 7fe84ff1f700 1 -- 172.21.15.32:0/1812043012 shutdown_connections 2021-04-09T18:20:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.072 7fe84ff1f700 1 --2- 172.21.15.32:0/1812043012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8480d8490 0x7fe84810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.072 7fe84ff1f700 1 --2- 172.21.15.32:0/1812043012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe84810d010 0x7fe8480d1780 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.072 7fe84ff1f700 1 -- 172.21.15.32:0/1812043012 >> 172.21.15.32:0/1812043012 conn(0x7fe8480ce5f0 msgr2=0x7fe8480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:10.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.073 7fe84ff1f700 1 -- 172.21.15.32:0/1812043012 shutdown_connections 2021-04-09T18:20:10.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.073 7fe84ff1f700 1 -- 172.21.15.32:0/1812043012 wait complete. 2021-04-09T18:20:10.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.093 7fe84ff1f700 1 Processor -- start 2021-04-09T18:20:10.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.093 7fe84ff1f700 1 -- start start 2021-04-09T18:20:10.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.093 7fe84ff1f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848185060 0x7fe848185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:10.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.093 7fe84ff1f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe848186c00 0x7fe848187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:10.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.093 7fe84ff1f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe848191360 0x7fe848191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:10.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.093 7fe84ff1f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8481863a0 con 0x7fe848185060 2021-04-09T18:20:10.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.093 7fe84ff1f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe848185a60 con 0x7fe848191360 2021-04-09T18:20:10.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.093 7fe84ff1f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe848187620 con 0x7fe848186c00 2021-04-09T18:20:10.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84dd3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848185060 0x7fe848185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:10.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84dd3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848185060 0x7fe848185d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47636/0 (socket says 172.21.15.32:47636) 2021-04-09T18:20:10.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84dd3b700 1 -- 172.21.15.32:0/1253609996 learned_addr learned my addr 172.21.15.32:0/1253609996 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:10.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84e53c700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe848191360 0x7fe848191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:10.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84d53a700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe848186c00 0x7fe848187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84dd3b700 1 -- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe848186c00 msgr2=0x7fe848187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84dd3b700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe848186c00 0x7fe848187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84dd3b700 1 -- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe848191360 msgr2=0x7fe848191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:10.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84dd3b700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe848191360 0x7fe848191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84dd3b700 1 -- 172.21.15.32:0/1253609996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe848185460 con 0x7fe848185060 2021-04-09T18:20:10.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84e53c700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe848191360 0x7fe848191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84dd3b700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848185060 0x7fe848185d80 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7fe844009630 tx=0x7fe84400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe83e7fc700 1 -- 172.21.15.32:0/1253609996 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8440093a0 con 0x7fe848185060 2021-04-09T18:20:10.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe83e7fc700 1 -- 172.21.15.32:0/1253609996 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84400bcf0 con 0x7fe848185060 2021-04-09T18:20:10.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe848185460 con 0x7fe848185060 2021-04-09T18:20:10.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe83e7fc700 1 -- 172.21.15.32:0/1253609996 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8440054d0 con 0x7fe848185060 2021-04-09T18:20:10.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.094 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe848185460 con 0x7fe848185060 2021-04-09T18:20:10.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.095 7fe83e7fc700 1 -- 172.21.15.32:0/1253609996 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe8440054f0 con 0x7fe848185060 2021-04-09T18:20:10.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.095 7fe83e7fc700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:10.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.095 7fe83e7fc700 1 -- 172.21.15.32:0/1253609996 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe844032a10 con 0x7fe848185060 2021-04-09T18:20:10.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.095 7fe84d53a700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:10.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.095 7fe84d53a700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 crc :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:10.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.115 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8481866d0 con 0x7fe848185060 2021-04-09T18:20:10.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.118 7fe83e7fc700 1 -- 172.21.15.32:0/1253609996 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe8440052e0 con 0x7fe848185060 2021-04-09T18:20:10.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.309 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8481866d0 con 0x7fe848185060 2021-04-09T18:20:10.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.309 7fe83e7fc700 1 -- 172.21.15.32:0/1253609996 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fe8440054f0 con 0x7fe848185060 2021-04-09T18:20:10.355 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:10.356 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:10.357 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:10.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 msgr2=0x7fe83001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:10.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 crc :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848185060 msgr2=0x7fe848185d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:10.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848185060 0x7fe848185d80 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7fe844009630 tx=0x7fe84400b900).stop 2021-04-09T18:20:10.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 shutdown_connections 2021-04-09T18:20:10.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe848191360 0x7fe848191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe848186c00 0x7fe848187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 --2- 172.21.15.32:0/1253609996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe848185060 0x7fe848185d80 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:10.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 >> 172.21.15.32:0/1253609996 conn(0x7fe8481837d0 msgr2=0x7fe848195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:10.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 shutdown_connections 2021-04-09T18:20:10.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:10.312 7fe84ff1f700 1 -- 172.21.15.32:0/1253609996 wait complete. 2021-04-09T18:20:10.785 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:10.786 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:11.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:11.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.902 7f8d144a2700 1 Processor -- start 2021-04-09T18:20:11.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.902 7f8d144a2700 1 -- start start 2021-04-09T18:20:11.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d144a2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8d0c0d7ca0 0x7f8d0c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:11.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d144a2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c10da00 0x7f8d0c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:11.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d144a2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c10d480 con 0x7f8d0c10d010 2021-04-09T18:20:11.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d144a2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0c0d9220 con 0x7f8d0c0d7ca0 2021-04-09T18:20:11.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d144a2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0d86c0 con 0x7f8d0c10da00 2021-04-09T18:20:11.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d12abf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c10da00 0x7f8d0c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:11.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d11abd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8d0c0d7ca0 0x7f8d0c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:11.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d122be700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8d0c10d010 0x7f8d0c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55160/0 (socket says 172.21.15.32:55160) 2021-04-09T18:20:11.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d122be700 1 -- 172.21.15.32:0/3968614179 learned_addr learned my addr 172.21.15.32:0/3968614179 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:11.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d12abf700 1 -- 172.21.15.32:0/3968614179 >> v2:172.21.15.86:3300/0 conn(0x7f8d0c0d7ca0 msgr2=0x7f8d0c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:11.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d12abf700 1 --2- 172.21.15.32:0/3968614179 >> v2:172.21.15.86:3300/0 conn(0x7f8d0c0d7ca0 0x7f8d0c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:11.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d12abf700 1 -- 172.21.15.32:0/3968614179 >> v1:172.21.15.86:6789/0 conn(0x7f8d0c10d010 legacy=0x7f8d0c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:11.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d12abf700 1 -- 172.21.15.32:0/3968614179 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c0d1f70 con 0x7f8d0c10da00 2021-04-09T18:20:11.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.903 7f8d11abd700 1 --2- 172.21.15.32:0/3968614179 >> v2:172.21.15.86:3300/0 conn(0x7f8d0c0d7ca0 0x7f8d0c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:11.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.904 7f8d12abf700 1 --2- 172.21.15.32:0/3968614179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c10da00 0x7f8d0c10de00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8d04006e50 tx=0x7f8d04009e70).ready entity=mon.2 client_cookie=988b767ef5f5a5ba server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:11.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.904 7f8d112bc700 1 -- 172.21.15.32:0/3968614179 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d04006a20 con 0x7f8d0c10da00 2021-04-09T18:20:11.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.904 7f8d112bc700 1 -- 172.21.15.32:0/3968614179 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d04006790 con 0x7f8d0c10da00 2021-04-09T18:20:11.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.904 7f8d112bc700 1 -- 172.21.15.32:0/3968614179 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d0400b700 con 0x7f8d0c10da00 2021-04-09T18:20:11.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.904 7f8d144a2700 1 -- 172.21.15.32:0/3968614179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c10da00 msgr2=0x7f8d0c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:11.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.904 7f8d144a2700 1 --2- 172.21.15.32:0/3968614179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c10da00 0x7f8d0c10de00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8d04006e50 tx=0x7f8d04009e70).stop 2021-04-09T18:20:11.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.905 7f8d144a2700 1 -- 172.21.15.32:0/3968614179 shutdown_connections 2021-04-09T18:20:11.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.905 7f8d144a2700 1 --2- 172.21.15.32:0/3968614179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c10da00 0x7f8d0c10de00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:11.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.905 7f8d144a2700 1 --2- 172.21.15.32:0/3968614179 >> v2:172.21.15.86:3300/0 conn(0x7f8d0c0d7ca0 0x7f8d0c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:11.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.905 7f8d144a2700 1 -- 172.21.15.32:0/3968614179 >> 172.21.15.32:0/3968614179 conn(0x7f8d0c0ce5f0 msgr2=0x7f8d0c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:11.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.905 7f8d144a2700 1 -- 172.21.15.32:0/3968614179 shutdown_connections 2021-04-09T18:20:11.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.905 7f8d144a2700 1 -- 172.21.15.32:0/3968614179 wait complete. 2021-04-09T18:20:11.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.925 7f8d144a2700 1 Processor -- start 2021-04-09T18:20:11.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.925 7f8d144a2700 1 -- start start 2021-04-09T18:20:11.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.925 7f8d144a2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c0b42d0 0x7f8d0c183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:11.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.925 7f8d144a2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d0c1ac090 0x7f8d0c1ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:11.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.925 7f8d144a2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d0c184d30 0x7f8d0c185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.925 7f8d144a2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0a1a90 con 0x7f8d0c184d30 2021-04-09T18:20:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.925 7f8d144a2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c1acab0 con 0x7f8d0c1ac090 2021-04-09T18:20:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.925 7f8d144a2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c185a00 con 0x7f8d0c0b42d0 2021-04-09T18:20:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d12abf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d0c184d30 0x7f8d0c185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:11.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d12abf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d0c184d30 0x7f8d0c185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47654/0 (socket says 172.21.15.32:47654) 2021-04-09T18:20:11.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d12abf700 1 -- 172.21.15.32:0/2096627452 learned_addr learned my addr 172.21.15.32:0/2096627452 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:11.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d122be700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c0b42d0 0x7f8d0c183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:11.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d11abd700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d0c1ac090 0x7f8d0c1ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:11.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d12abf700 1 -- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c0b42d0 msgr2=0x7f8d0c183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d12abf700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c0b42d0 0x7f8d0c183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d12abf700 1 -- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d0c1ac090 msgr2=0x7f8d0c1ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d12abf700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d0c1ac090 0x7f8d0c1ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d12abf700 1 -- 172.21.15.32:0/2096627452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c185750 con 0x7f8d0c184d30 2021-04-09T18:20:11.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d12abf700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d0c184d30 0x7f8d0c185130 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f8d04006e60 tx=0x7f8d0400bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:11.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8cfeffd700 1 -- 172.21.15.32:0/2096627452 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d04000df0 con 0x7f8d0c184d30 2021-04-09T18:20:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8cfeffd700 1 -- 172.21.15.32:0/2096627452 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0400bdb0 con 0x7f8d0c184d30 2021-04-09T18:20:11.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c185750 con 0x7f8d0c184d30 2021-04-09T18:20:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8cfeffd700 1 -- 172.21.15.32:0/2096627452 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d04001390 con 0x7f8d0c184d30 2021-04-09T18:20:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.926 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0c185750 con 0x7f8d0c184d30 2021-04-09T18:20:11.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.927 7f8cfeffd700 1 -- 172.21.15.32:0/2096627452 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8d040011a0 con 0x7f8d0c184d30 2021-04-09T18:20:12.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.927 7f8cfeffd700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8cf4018930 0x7f8cf401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:12.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.927 7f8cfeffd700 1 -- 172.21.15.32:0/2096627452 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8d04032d70 con 0x7f8d0c184d30 2021-04-09T18:20:12.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.927 7f8d122be700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8cf4018930 0x7f8cf401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:12.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.927 7f8d122be700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8cf4018930 0x7f8cf401ae10 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:12.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.947 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0c182d10 con 0x7f8d0c184d30 2021-04-09T18:20:12.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:11.950 7f8cfeffd700 1 -- 172.21.15.32:0/2096627452 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8d04009130 con 0x7f8d0c184d30 2021-04-09T18:20:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.140 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d0c10d480 con 0x7f8d0c184d30 2021-04-09T18:20:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.141 7f8cfeffd700 1 -- 172.21.15.32:0/2096627452 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f8d040011a0 con 0x7f8d0c184d30 2021-04-09T18:20:12.185 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:12.186 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:12.186 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:12.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8cf4018930 msgr2=0x7f8cf401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8cf4018930 0x7f8cf401ae10 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d0c184d30 msgr2=0x7f8d0c185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:12.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d0c184d30 0x7f8d0c185130 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f8d04006e60 tx=0x7f8d0400bc70).stop 2021-04-09T18:20:12.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 shutdown_connections 2021-04-09T18:20:12.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8cf4018930 0x7f8cf401ae10 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:12.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d0c184d30 0x7f8d0c185130 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:12.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d0c1ac090 0x7f8d0c1ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:12.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 --2- 172.21.15.32:0/2096627452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d0c0b42d0 0x7f8d0c183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:12.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 >> 172.21.15.32:0/2096627452 conn(0x7f8d0c19b890 msgr2=0x7f8d0c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:12.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 shutdown_connections 2021-04-09T18:20:12.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:12.143 7f8d144a2700 1 -- 172.21.15.32:0/2096627452 wait complete. 2021-04-09T18:20:12.603 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:12.604 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:13.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:13.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.711 7f0923d67700 1 Processor -- start 2021-04-09T18:20:13.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.711 7f0923d67700 1 -- start start 2021-04-09T18:20:13.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.711 7f0923d67700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f091c10d010 0x7f091c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:13.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.711 7f0923d67700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f091c0d1da0 0x7f091c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:13.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.711 7f0923d67700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c10d480 con 0x7f091c0d82c0 2021-04-09T18:20:13.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.711 7f0923d67700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c0d8d30 con 0x7f091c0d1da0 2021-04-09T18:20:13.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.711 7f0923d67700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c0d9220 con 0x7f091c10d010 2021-04-09T18:20:13.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.711 7f0921382700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f091c0d1da0 0x7f091c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:13.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0921382700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f091c0d1da0 0x7f091c0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47662/0 (socket says 172.21.15.32:47662) 2021-04-09T18:20:13.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0921382700 1 -- 172.21.15.32:0/489860819 learned_addr learned my addr 172.21.15.32:0/489860819 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:13.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0921b83700 1 --2- 172.21.15.32:0/489860819 >> v2:172.21.15.86:3300/0 conn(0x7f091c10d010 0x7f091c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:13.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0921382700 1 -- 172.21.15.32:0/489860819 >> v2:172.21.15.86:3300/0 conn(0x7f091c10d010 msgr2=0x7f091c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:13.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0921382700 1 --2- 172.21.15.32:0/489860819 >> v2:172.21.15.86:3300/0 conn(0x7f091c10d010 0x7f091c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:13.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0921382700 1 -- 172.21.15.32:0/489860819 >> v1:172.21.15.86:6789/0 conn(0x7f091c0d82c0 legacy=0x7f091c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:13.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0921382700 1 -- 172.21.15.32:0/489860819 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c0d86f0 con 0x7f091c0d1da0 2021-04-09T18:20:13.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0921382700 1 --2- 172.21.15.32:0/489860819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0d1da0 0x7f091c0d7ca0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f090c006700 tx=0x7f090c009b00).ready entity=mon.0 client_cookie=15d107c94568b4c7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0920b81700 1 -- 172.21.15.32:0/489860819 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f090c00c0c0 con 0x7f091c0d1da0 2021-04-09T18:20:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0920b81700 1 -- 172.21.15.32:0/489860819 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090c00c400 con 0x7f091c0d1da0 2021-04-09T18:20:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0920b81700 1 -- 172.21.15.32:0/489860819 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f090c00c8e0 con 0x7f091c0d1da0 2021-04-09T18:20:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0923d67700 1 -- 172.21.15.32:0/489860819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0d1da0 msgr2=0x7f091c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.712 7f0923d67700 1 --2- 172.21.15.32:0/489860819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0d1da0 0x7f091c0d7ca0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f090c006700 tx=0x7f090c009b00).stop 2021-04-09T18:20:13.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.713 7f0923d67700 1 -- 172.21.15.32:0/489860819 shutdown_connections 2021-04-09T18:20:13.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.713 7f0923d67700 1 --2- 172.21.15.32:0/489860819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0d1da0 0x7f091c0d7ca0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:13.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.713 7f0923d67700 1 --2- 172.21.15.32:0/489860819 >> v2:172.21.15.86:3300/0 conn(0x7f091c10d010 0x7f091c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:13.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.713 7f0923d67700 1 -- 172.21.15.32:0/489860819 >> 172.21.15.32:0/489860819 conn(0x7f091c0ce5f0 msgr2=0x7f091c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:13.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.713 7f0923d67700 1 -- 172.21.15.32:0/489860819 shutdown_connections 2021-04-09T18:20:13.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.713 7f0923d67700 1 -- 172.21.15.32:0/489860819 wait complete. 2021-04-09T18:20:13.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.733 7f0923d67700 1 Processor -- start 2021-04-09T18:20:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.733 7f0923d67700 1 -- start start 2021-04-09T18:20:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0923d67700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0aa270 0x7f091c182670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0923d67700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f091c182c90 0x7f091c095650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0923d67700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f091c098d80 0x7f091c095c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:13.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0923d67700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c099180 con 0x7f091c0aa270 2021-04-09T18:20:13.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0923d67700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c17e800 con 0x7f091c182c90 2021-04-09T18:20:13.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0923d67700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c17e4f0 con 0x7f091c098d80 2021-04-09T18:20:13.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921b83700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0aa270 0x7f091c182670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:13.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921b83700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0aa270 0x7f091c182670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47664/0 (socket says 172.21.15.32:47664) 2021-04-09T18:20:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921b83700 1 -- 172.21.15.32:0/3135341614 learned_addr learned my addr 172.21.15.32:0/3135341614 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921382700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f091c182c90 0x7f091c095650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0922384700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f091c098d80 0x7f091c095c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921b83700 1 -- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f091c098d80 msgr2=0x7f091c095c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921b83700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f091c098d80 0x7f091c095c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921b83700 1 -- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f091c182c90 msgr2=0x7f091c095650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921b83700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f091c182c90 0x7f091c095650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921b83700 1 -- 172.21.15.32:0/3135341614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c179610 con 0x7f091c0aa270 2021-04-09T18:20:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921382700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f091c182c90 0x7f091c095650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:20:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.734 7f0921b83700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0aa270 0x7f091c182670 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f091800b580 tx=0x7f091800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.735 7f09127fc700 1 -- 172.21.15.32:0/3135341614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f091800bd10 con 0x7f091c0aa270 2021-04-09T18:20:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.735 7f09127fc700 1 -- 172.21.15.32:0/3135341614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0918000c90 con 0x7f091c0aa270 2021-04-09T18:20:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.735 7f09127fc700 1 -- 172.21.15.32:0/3135341614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09180022b0 con 0x7f091c0aa270 2021-04-09T18:20:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.735 7f0923d67700 1 -- 172.21.15.32:0/3135341614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c179610 con 0x7f091c0aa270 2021-04-09T18:20:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.735 7f0923d67700 1 -- 172.21.15.32:0/3135341614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c179610 con 0x7f091c0aa270 2021-04-09T18:20:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.735 7f09127fc700 1 -- 172.21.15.32:0/3135341614 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f09180052e0 con 0x7f091c0aa270 2021-04-09T18:20:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.735 7f09127fc700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0904018900 0x7f090401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.735 7f09127fc700 1 -- 172.21.15.32:0/3135341614 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0918032a20 con 0x7f091c0aa270 2021-04-09T18:20:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.736 7f0921382700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0904018900 0x7f090401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:13.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.736 7f0921382700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0904018900 0x7f090401adc0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.755 7f0923d67700 1 -- 172.21.15.32:0/3135341614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091c10d480 con 0x7f091c0aa270 2021-04-09T18:20:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.759 7f09127fc700 1 -- 172.21.15.32:0/3135341614 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0918006320 con 0x7f091c0aa270 2021-04-09T18:20:13.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.949 7f0923d67700 1 -- 172.21.15.32:0/3135341614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f091c10d480 con 0x7f091c0aa270 2021-04-09T18:20:13.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.950 7f09127fc700 1 -- 172.21.15.32:0/3135341614 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f0918006ba0 con 0x7f091c0aa270 2021-04-09T18:20:13.995 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:13.996 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:13.997 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:13.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 -- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0904018900 msgr2=0x7f090401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:13.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0904018900 0x7f090401adc0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:13.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 -- 172.21.15.32:0/3135341614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0aa270 msgr2=0x7f091c182670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:14.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0aa270 0x7f091c182670 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f091800b580 tx=0x7f091800b920).stop 2021-04-09T18:20:14.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 -- 172.21.15.32:0/3135341614 shutdown_connections 2021-04-09T18:20:14.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0904018900 0x7f090401adc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:14.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f091c098d80 0x7f091c095c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:14.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f091c182c90 0x7f091c095650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:14.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 --2- 172.21.15.32:0/3135341614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f091c0aa270 0x7f091c182670 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:14.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 -- 172.21.15.32:0/3135341614 >> 172.21.15.32:0/3135341614 conn(0x7f091c1915c0 msgr2=0x7f091c0947a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:14.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 -- 172.21.15.32:0/3135341614 shutdown_connections 2021-04-09T18:20:14.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:13.952 7f0923d67700 1 -- 172.21.15.32:0/3135341614 wait complete. 2021-04-09T18:20:14.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:14.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:15.420 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:15.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.540 7f49d5485700 1 Processor -- start 2021-04-09T18:20:15.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.540 7f49d5485700 1 -- start start 2021-04-09T18:20:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.541 7f49d5485700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49d010d010 0x7f49d00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:15.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.541 7f49d5485700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d010d480 con 0x7f49d00d8490 2021-04-09T18:20:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.541 7f49d5485700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d010e1f0 con 0x7f49d00d1da0 2021-04-09T18:20:15.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.541 7f49d5485700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d00d8d30 con 0x7f49d010d010 2021-04-09T18:20:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.541 7f49cf07d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49d010d010 0x7f49d00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:15.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.541 7f49cf87e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f49d00d8490 0x7f49d010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57264/0 (socket says 172.21.15.32:57264) 2021-04-09T18:20:15.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.541 7f49cf87e700 1 -- 172.21.15.32:0/4083273284 learned_addr learned my addr 172.21.15.32:0/4083273284 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:15.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49cf07d700 1 -- 172.21.15.32:0/4083273284 >> v1:172.21.15.86:6789/0 conn(0x7f49d00d1da0 legacy=0x7f49d00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:15.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49cf07d700 1 -- 172.21.15.32:0/4083273284 >> v1:172.21.15.32:6789/0 conn(0x7f49d00d8490 legacy=0x7f49d010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49cf07d700 1 -- 172.21.15.32:0/4083273284 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d00d9220 con 0x7f49d010d010 2021-04-09T18:20:15.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49ce07b700 1 -- 172.21.15.32:0/4083273284 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531135351 0 0) 0x7f49c0000d30 con 0x7f49d00d8490 2021-04-09T18:20:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49cf07d700 1 --2- 172.21.15.32:0/4083273284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d010d010 0x7f49d00d1780 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7f49c8006650 tx=0x7f49c8009a00).ready entity=mon.0 client_cookie=9dad5bc2ca13bca6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49ce07b700 1 -- 172.21.15.32:0/4083273284 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c8009df0 con 0x7f49d010d010 2021-04-09T18:20:15.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49ce07b700 1 -- 172.21.15.32:0/4083273284 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c80009b0 con 0x7f49d010d010 2021-04-09T18:20:15.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49ce07b700 1 -- 172.21.15.32:0/4083273284 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c800c2b0 con 0x7f49d010d010 2021-04-09T18:20:15.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49d5485700 1 -- 172.21.15.32:0/4083273284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d010d010 msgr2=0x7f49d00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.542 7f49d5485700 1 --2- 172.21.15.32:0/4083273284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d010d010 0x7f49d00d1780 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7f49c8006650 tx=0x7f49c8009a00).stop 2021-04-09T18:20:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.543 7f49d5485700 1 -- 172.21.15.32:0/4083273284 shutdown_connections 2021-04-09T18:20:15.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.543 7f49d5485700 1 --2- 172.21.15.32:0/4083273284 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d010d010 0x7f49d00d1780 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.543 7f49d5485700 1 -- 172.21.15.32:0/4083273284 >> 172.21.15.32:0/4083273284 conn(0x7f49d00ce5f0 msgr2=0x7f49d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:15.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.543 7f49d5485700 1 -- 172.21.15.32:0/4083273284 shutdown_connections 2021-04-09T18:20:15.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.543 7f49d5485700 1 -- 172.21.15.32:0/4083273284 wait complete. 2021-04-09T18:20:15.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.563 7f49d5485700 1 Processor -- start 2021-04-09T18:20:15.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.563 7f49d5485700 1 -- start start 2021-04-09T18:20:15.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49d5485700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d01a9a50 0x7f49d008a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:15.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49d5485700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d008afb0 0x7f49d009d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:15.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49d5485700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d009ddf0 0x7f49d0183b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:15.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49d5485700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d01a9e50 con 0x7f49d008afb0 2021-04-09T18:20:15.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49d5485700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d009e410 con 0x7f49d009ddf0 2021-04-09T18:20:15.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49d5485700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d01841a0 con 0x7f49d01a9a50 2021-04-09T18:20:15.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49ce87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d008afb0 0x7f49d009d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:15.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49ce87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d008afb0 0x7f49d009d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47680/0 (socket says 172.21.15.32:47680) 2021-04-09T18:20:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49ce87c700 1 -- 172.21.15.32:0/3889389654 learned_addr learned my addr 172.21.15.32:0/3889389654 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:15.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49cf07d700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d01a9a50 0x7f49d008a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49cf87e700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d009ddf0 0x7f49d0183b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49ce87c700 1 -- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d01a9a50 msgr2=0x7f49d008a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:15.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49ce87c700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d01a9a50 0x7f49d008a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:15.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49ce87c700 1 -- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d009ddf0 msgr2=0x7f49d0183b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:15.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49ce87c700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d009ddf0 0x7f49d0183b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:15.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49ce87c700 1 -- 172.21.15.32:0/3889389654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d0184380 con 0x7f49d008afb0 2021-04-09T18:20:15.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49ce87c700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d008afb0 0x7f49d009d7d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f49b800b530 tx=0x7f49b800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49c7fff700 1 -- 172.21.15.32:0/3889389654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b800bcc0 con 0x7f49d008afb0 2021-04-09T18:20:15.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49c7fff700 1 -- 172.21.15.32:0/3889389654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b8006020 con 0x7f49d008afb0 2021-04-09T18:20:15.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49d5485700 1 -- 172.21.15.32:0/3889389654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d0184380 con 0x7f49d008afb0 2021-04-09T18:20:15.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49c7fff700 1 -- 172.21.15.32:0/3889389654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b80054a0 con 0x7f49d008afb0 2021-04-09T18:20:15.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.564 7f49d5485700 1 -- 172.21.15.32:0/3889389654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d0184380 con 0x7f49d008afb0 2021-04-09T18:20:15.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.565 7f49c7fff700 1 -- 172.21.15.32:0/3889389654 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f49b8006200 con 0x7f49d008afb0 2021-04-09T18:20:15.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.565 7f49c7fff700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b4018930 0x7f49b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:15.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.565 7f49c7fff700 1 -- 172.21.15.32:0/3889389654 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f49b8032a30 con 0x7f49d008afb0 2021-04-09T18:20:15.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.565 7f49cf07d700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b4018930 0x7f49b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:15.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.566 7f49cf07d700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b4018930 0x7f49b401ae10 crc :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:15.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.585 7f49d5485700 1 -- 172.21.15.32:0/3889389654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d010d480 con 0x7f49d008afb0 2021-04-09T18:20:15.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.589 7f49c7fff700 1 -- 172.21.15.32:0/3889389654 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f49b800bcc0 con 0x7f49d008afb0 2021-04-09T18:20:15.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.781 7f49d5485700 1 -- 172.21.15.32:0/3889389654 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49d010d480 con 0x7f49d008afb0 2021-04-09T18:20:15.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.781 7f49c7fff700 1 -- 172.21.15.32:0/3889389654 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f49b8006020 con 0x7f49d008afb0 2021-04-09T18:20:15.826 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:15.826 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:15.827 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 -- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b4018930 msgr2=0x7f49b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b4018930 0x7f49b401ae10 crc :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 -- 172.21.15.32:0/3889389654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d008afb0 msgr2=0x7f49d009d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d008afb0 0x7f49d009d7d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f49b800b530 tx=0x7f49b800b8d0).stop 2021-04-09T18:20:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 -- 172.21.15.32:0/3889389654 shutdown_connections 2021-04-09T18:20:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49b4018930 0x7f49b401ae10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49d008afb0 0x7f49d009d7d0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49d009ddf0 0x7f49d0183b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 --2- 172.21.15.32:0/3889389654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49d01a9a50 0x7f49d008a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.783 7f49d5485700 1 -- 172.21.15.32:0/3889389654 >> 172.21.15.32:0/3889389654 conn(0x7f49d01965c0 msgr2=0x7f49d01ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.784 7f49d5485700 1 -- 172.21.15.32:0/3889389654 shutdown_connections 2021-04-09T18:20:15.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:15.784 7f49d5485700 1 -- 172.21.15.32:0/3889389654 wait complete. 2021-04-09T18:20:16.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:16.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:17.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:17.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.359 7fc44d82b700 1 Processor -- start 2021-04-09T18:20:17.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.359 7fc44d82b700 1 -- start start 2021-04-09T18:20:17.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.359 7fc44d82b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44810da00 0x7fc44810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.359 7fc44d82b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc44810d480 con 0x7fc44810d010 2021-04-09T18:20:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.359 7fc44d82b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4480d1f70 con 0x7fc4480d7ca0 2021-04-09T18:20:17.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.359 7fc44d82b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480d9220 con 0x7fc44810da00 2021-04-09T18:20:17.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc447574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc44810d010 0x7fc4480d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57276/0 (socket says 172.21.15.32:57276) 2021-04-09T18:20:17.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc447574700 1 -- 172.21.15.32:0/240303518 learned_addr learned my addr 172.21.15.32:0/240303518 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:17.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc447d75700 1 --2- 172.21.15.32:0/240303518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44810da00 0x7fc44810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc446572700 1 -- 172.21.15.32:0/240303518 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858938947 0 0) 0x7fc438000ce0 con 0x7fc44810d010 2021-04-09T18:20:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc446572700 1 -- 172.21.15.32:0/240303518 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc430002fe0 con 0x7fc44810d010 2021-04-09T18:20:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc446572700 1 -- 172.21.15.32:0/240303518 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415088184 0 0) 0x7fc43c000c60 con 0x7fc4480d7ca0 2021-04-09T18:20:17.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc446572700 1 -- 172.21.15.32:0/240303518 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc430003020 con 0x7fc4480d7ca0 2021-04-09T18:20:17.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc447d75700 1 -- 172.21.15.32:0/240303518 >> v1:172.21.15.86:6789/0 conn(0x7fc4480d7ca0 legacy=0x7fc4480d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:17.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc447d75700 1 -- 172.21.15.32:0/240303518 >> v1:172.21.15.32:6789/0 conn(0x7fc44810d010 legacy=0x7fc4480d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:17.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc447d75700 1 -- 172.21.15.32:0/240303518 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44810e420 con 0x7fc44810da00 2021-04-09T18:20:17.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc447d75700 1 --2- 172.21.15.32:0/240303518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44810da00 0x7fc44810de00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc440006e50 tx=0x7fc440009e70).ready entity=mon.2 client_cookie=5c982f04240e4a61 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:17.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc446572700 1 -- 172.21.15.32:0/240303518 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc440006a20 con 0x7fc44810da00 2021-04-09T18:20:17.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc446572700 1 -- 172.21.15.32:0/240303518 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc440006790 con 0x7fc44810da00 2021-04-09T18:20:17.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc446572700 1 -- 172.21.15.32:0/240303518 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc44000b700 con 0x7fc44810da00 2021-04-09T18:20:17.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc44d82b700 1 -- 172.21.15.32:0/240303518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44810da00 msgr2=0x7fc44810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:17.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.360 7fc44d82b700 1 --2- 172.21.15.32:0/240303518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44810da00 0x7fc44810de00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc440006e50 tx=0x7fc440009e70).stop 2021-04-09T18:20:17.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.361 7fc44d82b700 1 -- 172.21.15.32:0/240303518 shutdown_connections 2021-04-09T18:20:17.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.361 7fc44d82b700 1 --2- 172.21.15.32:0/240303518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44810da00 0x7fc44810de00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.361 7fc44d82b700 1 -- 172.21.15.32:0/240303518 >> 172.21.15.32:0/240303518 conn(0x7fc4480ce5f0 msgr2=0x7fc4480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.361 7fc44d82b700 1 -- 172.21.15.32:0/240303518 shutdown_connections 2021-04-09T18:20:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.361 7fc44d82b700 1 -- 172.21.15.32:0/240303518 wait complete. 2021-04-09T18:20:17.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.381 7fc44d82b700 1 Processor -- start 2021-04-09T18:20:17.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.381 7fc44d82b700 1 -- start start 2021-04-09T18:20:17.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc44d82b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc448185a00 0x7fc448097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:17.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc44d82b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4480983b0 0x7fc44817abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:17.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc44d82b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44817b1d0 0x7fc448179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:17.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc44d82b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480987b0 con 0x7fc448185a00 2021-04-09T18:20:17.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc44d82b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44817b5d0 con 0x7fc4480983b0 2021-04-09T18:20:17.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc44d82b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc448179c30 con 0x7fc44817b1d0 2021-04-09T18:20:17.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc447574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc448185a00 0x7fc448097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:17.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc447574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc448185a00 0x7fc448097d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47692/0 (socket says 172.21.15.32:47692) 2021-04-09T18:20:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc447574700 1 -- 172.21.15.32:0/1260437064 learned_addr learned my addr 172.21.15.32:0/1260437064 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc446d73700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4480983b0 0x7fc44817abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:17.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc447d75700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44817b1d0 0x7fc448179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:17.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc447d75700 1 -- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4480983b0 msgr2=0x7fc44817abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:17.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.382 7fc447d75700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4480983b0 0x7fc44817abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc447d75700 1 -- 172.21.15.32:0/1260437064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc448185a00 msgr2=0x7fc448097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc447d75700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc448185a00 0x7fc448097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc447d75700 1 -- 172.21.15.32:0/1260437064 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448179de0 con 0x7fc44817b1d0 2021-04-09T18:20:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc446d73700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4480983b0 0x7fc44817abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc447574700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc448185a00 0x7fc448097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:17.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc447d75700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44817b1d0 0x7fc448179610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc44000b880 tx=0x7fc44000bc20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:17.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc42ffff700 1 -- 172.21.15.32:0/1260437064 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc44000bd60 con 0x7fc44817b1d0 2021-04-09T18:20:17.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc42ffff700 1 -- 172.21.15.32:0/1260437064 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4400093f0 con 0x7fc44817b1d0 2021-04-09T18:20:17.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc448179c30 con 0x7fc44817b1d0 2021-04-09T18:20:17.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc42ffff700 1 -- 172.21.15.32:0/1260437064 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4400023c0 con 0x7fc44817b1d0 2021-04-09T18:20:17.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44817a370 con 0x7fc44817b1d0 2021-04-09T18:20:17.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.383 7fc42ffff700 1 -- 172.21.15.32:0/1260437064 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc440009680 con 0x7fc44817b1d0 2021-04-09T18:20:17.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.384 7fc42ffff700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc428018930 0x7fc42801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:17.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.384 7fc42ffff700 1 -- 172.21.15.32:0/1260437064 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc440032c90 con 0x7fc44817b1d0 2021-04-09T18:20:17.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.384 7fc447574700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc428018930 0x7fc42801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:17.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.384 7fc447574700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc428018930 0x7fc42801ae10 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:17.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.403 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4481b7010 con 0x7fc44817b1d0 2021-04-09T18:20:17.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.407 7fc42ffff700 1 -- 172.21.15.32:0/1260437064 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc44000bd60 con 0x7fc44817b1d0 2021-04-09T18:20:17.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.598 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc44810d480 con 0x7fc44817b1d0 2021-04-09T18:20:17.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.598 7fc42ffff700 1 -- 172.21.15.32:0/1260437064 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fc4400093f0 con 0x7fc44817b1d0 2021-04-09T18:20:17.646 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:17.646 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:17.647 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:17.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc428018930 msgr2=0x7fc42801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:17.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc428018930 0x7fc42801ae10 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44817b1d0 msgr2=0x7fc448179610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44817b1d0 0x7fc448179610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc44000b880 tx=0x7fc44000bc20).stop 2021-04-09T18:20:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 shutdown_connections 2021-04-09T18:20:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc428018930 0x7fc42801ae10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44817b1d0 0x7fc448179610 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:17.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4480983b0 0x7fc44817abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:17.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 --2- 172.21.15.32:0/1260437064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc448185a00 0x7fc448097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:17.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 >> 172.21.15.32:0/1260437064 conn(0x7fc448190550 msgr2=0x7fc4481b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:17.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 shutdown_connections 2021-04-09T18:20:17.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:17.601 7fc44d82b700 1 -- 172.21.15.32:0/1260437064 wait complete. 2021-04-09T18:20:18.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:18.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:19.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.182 7fd2b2c28700 1 Processor -- start 2021-04-09T18:20:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.182 7fd2b2c28700 1 -- start start 2021-04-09T18:20:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b2c28700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2ac10d010 0x7fd2ac0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b2c28700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac0d8490 0x7fd2ac10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b2c28700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac10d480 con 0x7fd2ac0d1da0 2021-04-09T18:20:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b2c28700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ac0d8d30 con 0x7fd2ac10d010 2021-04-09T18:20:19.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b2c28700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0d9220 con 0x7fd2ac0d8490 2021-04-09T18:20:19.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b0a44700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2ac10d010 0x7fd2ac0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b0a44700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2ac10d010 0x7fd2ac0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47700/0 (socket says 172.21.15.32:47700) 2021-04-09T18:20:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b0a44700 1 -- 172.21.15.32:0/2429426617 learned_addr learned my addr 172.21.15.32:0/2429426617 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b1245700 1 --2- 172.21.15.32:0/2429426617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac0d8490 0x7fd2ac10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b0a44700 1 -- 172.21.15.32:0/2429426617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac0d8490 msgr2=0x7fd2ac10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b0a44700 1 --2- 172.21.15.32:0/2429426617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac0d8490 0x7fd2ac10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b0a44700 1 -- 172.21.15.32:0/2429426617 >> v1:172.21.15.86:6789/0 conn(0x7fd2ac0d1da0 legacy=0x7fd2ac0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b0a44700 1 -- 172.21.15.32:0/2429426617 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac10e020 con 0x7fd2ac10d010 2021-04-09T18:20:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b0a44700 1 --2- 172.21.15.32:0/2429426617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac10d010 0x7fd2ac0d1780 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7fd29c006700 tx=0x7fd29c009b80).ready entity=mon.0 client_cookie=8a5cd80a81662c01 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:19.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2ab7fe700 1 -- 172.21.15.32:0/2429426617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd29c00c0c0 con 0x7fd2ac10d010 2021-04-09T18:20:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2ab7fe700 1 -- 172.21.15.32:0/2429426617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c00c440 con 0x7fd2ac10d010 2021-04-09T18:20:19.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2ab7fe700 1 -- 172.21.15.32:0/2429426617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd29c00c920 con 0x7fd2ac10d010 2021-04-09T18:20:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b2c28700 1 -- 172.21.15.32:0/2429426617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac10d010 msgr2=0x7fd2ac0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:19.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.183 7fd2b2c28700 1 --2- 172.21.15.32:0/2429426617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac10d010 0x7fd2ac0d1780 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7fd29c006700 tx=0x7fd29c009b80).stop 2021-04-09T18:20:19.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.184 7fd2b2c28700 1 -- 172.21.15.32:0/2429426617 shutdown_connections 2021-04-09T18:20:19.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.184 7fd2b2c28700 1 --2- 172.21.15.32:0/2429426617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac0d8490 0x7fd2ac10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.184 7fd2b2c28700 1 --2- 172.21.15.32:0/2429426617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac10d010 0x7fd2ac0d1780 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.184 7fd2b2c28700 1 -- 172.21.15.32:0/2429426617 >> 172.21.15.32:0/2429426617 conn(0x7fd2ac0ce5f0 msgr2=0x7fd2ac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:19.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.184 7fd2b2c28700 1 -- 172.21.15.32:0/2429426617 shutdown_connections 2021-04-09T18:20:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.184 7fd2b2c28700 1 -- 172.21.15.32:0/2429426617 wait complete. 2021-04-09T18:20:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.204 7fd2b2c28700 1 Processor -- start 2021-04-09T18:20:19.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2b2c28700 1 -- start start 2021-04-09T18:20:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2b2c28700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac186480 0x7fd2ac08c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:19.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2b2c28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac186880 0x7fd2ac08c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:19.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2b2c28700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ac08d680 0x7fd2ac08da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2b2c28700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac08bc60 con 0x7fd2ac186880 2021-04-09T18:20:19.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2b2c28700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac08e0a0 con 0x7fd2ac08d680 2021-04-09T18:20:19.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2b2c28700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac08e2c0 con 0x7fd2ac186480 2021-04-09T18:20:19.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2abfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac186880 0x7fd2ac08c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2abfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac186880 0x7fd2ac08c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47708/0 (socket says 172.21.15.32:47708) 2021-04-09T18:20:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2abfff700 1 -- 172.21.15.32:0/4095272225 learned_addr learned my addr 172.21.15.32:0/4095272225 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2b1245700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ac08d680 0x7fd2ac08da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.205 7fd2b0a44700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac186480 0x7fd2ac08c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:19.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2abfff700 1 -- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac186480 msgr2=0x7fd2ac08c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:19.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2abfff700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac186480 0x7fd2ac08c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2abfff700 1 -- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ac08d680 msgr2=0x7fd2ac08da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:19.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2abfff700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ac08d680 0x7fd2ac08da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2abfff700 1 -- 172.21.15.32:0/4095272225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac08e6e0 con 0x7fd2ac186880 2021-04-09T18:20:19.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2abfff700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac186880 0x7fd2ac08c800 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fd2a0006f10 tx=0x7fd2a0008960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:19.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2a97fa700 1 -- 172.21.15.32:0/4095272225 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2a000b020 con 0x7fd2ac186880 2021-04-09T18:20:19.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2a97fa700 1 -- 172.21.15.32:0/4095272225 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a0010450 con 0x7fd2ac186880 2021-04-09T18:20:19.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac08e6e0 con 0x7fd2ac186880 2021-04-09T18:20:19.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2a97fa700 1 -- 172.21.15.32:0/4095272225 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2a0010970 con 0x7fd2ac186880 2021-04-09T18:20:19.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.206 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac08e6e0 con 0x7fd2ac186880 2021-04-09T18:20:19.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.207 7fd2a97fa700 1 -- 172.21.15.32:0/4095272225 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd2a0007000 con 0x7fd2ac186880 2021-04-09T18:20:19.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.207 7fd2a97fa700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd28c018930 0x7fd28c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:19.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.207 7fd2a97fa700 1 -- 172.21.15.32:0/4095272225 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd2a0036e50 con 0x7fd2ac186880 2021-04-09T18:20:19.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.207 7fd2b0a44700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd28c018930 0x7fd28c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:19.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.208 7fd2b0a44700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd28c018930 0x7fd28c01ae10 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:19.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.227 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac186c80 con 0x7fd2ac186880 2021-04-09T18:20:19.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.230 7fd2a97fa700 1 -- 172.21.15.32:0/4095272225 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd2a0008d50 con 0x7fd2ac186880 2021-04-09T18:20:19.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.422 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2ac186c80 con 0x7fd2ac186880 2021-04-09T18:20:19.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.423 7fd2a97fa700 1 -- 172.21.15.32:0/4095272225 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fd2a0007000 con 0x7fd2ac186880 2021-04-09T18:20:19.469 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:19.470 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:19.470 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:19.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd28c018930 msgr2=0x7fd28c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:19.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd28c018930 0x7fd28c01ae10 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac186880 msgr2=0x7fd2ac08c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:19.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac186880 0x7fd2ac08c800 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fd2a0006f10 tx=0x7fd2a0008960).stop 2021-04-09T18:20:19.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 shutdown_connections 2021-04-09T18:20:19.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd28c018930 0x7fd28c01ae10 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ac186880 0x7fd2ac08c800 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ac08d680 0x7fd2ac08da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 --2- 172.21.15.32:0/4095272225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ac186480 0x7fd2ac08c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 >> 172.21.15.32:0/4095272225 conn(0x7fd2ac195e70 msgr2=0x7fd2ac1a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 shutdown_connections 2021-04-09T18:20:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:19.425 7fd2b2c28700 1 -- 172.21.15.32:0/4095272225 wait complete. 2021-04-09T18:20:19.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:19.884 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:20.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:21.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.002 7fe681fa1700 1 Processor -- start 2021-04-09T18:20:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.002 7fe681fa1700 1 -- start start 2021-04-09T18:20:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe681fa1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe67c10d010 0x7fe67c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe681fa1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe67c0d8490 0x7fe67c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe681fa1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c10d480 con 0x7fe67c0d1da0 2021-04-09T18:20:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe681fa1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c0d8d30 con 0x7fe67c0d8490 2021-04-09T18:20:21.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe681fa1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c0d9220 con 0x7fe67c10d010 2021-04-09T18:20:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe67bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe67c0d8490 0x7fe67c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe67bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe67c0d8490 0x7fe67c10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47716/0 (socket says 172.21.15.32:47716) 2021-04-09T18:20:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe67bfff700 1 -- 172.21.15.32:0/3280806396 learned_addr learned my addr 172.21.15.32:0/3280806396 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe67b7fe700 1 --2- 172.21.15.32:0/3280806396 >> v2:172.21.15.86:3300/0 conn(0x7fe67c10d010 0x7fe67c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe67bfff700 1 -- 172.21.15.32:0/3280806396 >> v2:172.21.15.86:3300/0 conn(0x7fe67c10d010 msgr2=0x7fe67c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe67bfff700 1 --2- 172.21.15.32:0/3280806396 >> v2:172.21.15.86:3300/0 conn(0x7fe67c10d010 0x7fe67c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.003 7fe67bfff700 1 -- 172.21.15.32:0/3280806396 >> v1:172.21.15.86:6789/0 conn(0x7fe67c0d1da0 legacy=0x7fe67c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.004 7fe67bfff700 1 -- 172.21.15.32:0/3280806396 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c10e020 con 0x7fe67c0d8490 2021-04-09T18:20:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.004 7fe67bfff700 1 --2- 172.21.15.32:0/3280806396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c0d8490 0x7fe67c10da00 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7fe674006b80 tx=0x7fe674009b10).ready entity=mon.0 client_cookie=d004debe1f6702c9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.004 7fe67a7fc700 1 -- 172.21.15.32:0/3280806396 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6740022e0 con 0x7fe67c0d8490 2021-04-09T18:20:21.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.004 7fe67a7fc700 1 -- 172.21.15.32:0/3280806396 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe674002620 con 0x7fe67c0d8490 2021-04-09T18:20:21.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.004 7fe67a7fc700 1 -- 172.21.15.32:0/3280806396 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe674002b00 con 0x7fe67c0d8490 2021-04-09T18:20:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.004 7fe681fa1700 1 -- 172.21.15.32:0/3280806396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c0d8490 msgr2=0x7fe67c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.004 7fe681fa1700 1 --2- 172.21.15.32:0/3280806396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c0d8490 0x7fe67c10da00 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7fe674006b80 tx=0x7fe674009b10).stop 2021-04-09T18:20:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.005 7fe681fa1700 1 -- 172.21.15.32:0/3280806396 shutdown_connections 2021-04-09T18:20:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.005 7fe681fa1700 1 --2- 172.21.15.32:0/3280806396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c0d8490 0x7fe67c10da00 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.005 7fe681fa1700 1 --2- 172.21.15.32:0/3280806396 >> v2:172.21.15.86:3300/0 conn(0x7fe67c10d010 0x7fe67c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.005 7fe681fa1700 1 -- 172.21.15.32:0/3280806396 >> 172.21.15.32:0/3280806396 conn(0x7fe67c0ce5f0 msgr2=0x7fe67c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.005 7fe681fa1700 1 -- 172.21.15.32:0/3280806396 shutdown_connections 2021-04-09T18:20:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.005 7fe681fa1700 1 -- 172.21.15.32:0/3280806396 wait complete. 2021-04-09T18:20:21.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.025 7fe681fa1700 1 Processor -- start 2021-04-09T18:20:21.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.025 7fe681fa1700 1 -- start start 2021-04-09T18:20:21.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe681fa1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c08f040 0x7fe67c1a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe681fa1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c08f440 0x7fe67c1a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:21.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe681fa1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c1a22e0 0x7fe67c1a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:21.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe681fa1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c1a3100 con 0x7fe67c08f040 2021-04-09T18:20:21.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe681fa1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c1a2d00 con 0x7fe67c1a22e0 2021-04-09T18:20:21.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe681fa1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c17c7e0 con 0x7fe67c08f440 2021-04-09T18:20:21.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c08f040 0x7fe67c1a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:21.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c08f040 0x7fe67c1a3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47720/0 (socket says 172.21.15.32:47720) 2021-04-09T18:20:21.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67b7fe700 1 -- 172.21.15.32:0/3526790502 learned_addr learned my addr 172.21.15.32:0/3526790502 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:21.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67bfff700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c1a22e0 0x7fe67c1a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:21.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67affd700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c08f440 0x7fe67c1a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:21.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67b7fe700 1 -- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c08f440 msgr2=0x7fe67c1a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:21.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67b7fe700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c08f440 0x7fe67c1a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67b7fe700 1 -- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c1a22e0 msgr2=0x7fe67c1a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:21.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67b7fe700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c1a22e0 0x7fe67c1a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67b7fe700 1 -- 172.21.15.32:0/3526790502 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0aee10 con 0x7fe67c08f040 2021-04-09T18:20:21.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe67b7fe700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c08f040 0x7fe67c1a3600 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7fe66c006f00 tx=0x7fe66c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:21.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe663fff700 1 -- 172.21.15.32:0/3526790502 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe66c00bd30 con 0x7fe67c08f040 2021-04-09T18:20:21.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe663fff700 1 -- 172.21.15.32:0/3526790502 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c002030 con 0x7fe67c08f040 2021-04-09T18:20:21.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe663fff700 1 -- 172.21.15.32:0/3526790502 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe66c002590 con 0x7fe67c08f040 2021-04-09T18:20:21.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c0aee10 con 0x7fe67c08f040 2021-04-09T18:20:21.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.026 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c0aee10 con 0x7fe67c08f040 2021-04-09T18:20:21.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.027 7fe663fff700 1 -- 172.21.15.32:0/3526790502 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe66c000c90 con 0x7fe67c08f040 2021-04-09T18:20:21.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.027 7fe663fff700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe65c018930 0x7fe65c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.027 7fe663fff700 1 -- 172.21.15.32:0/3526790502 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe66c032a40 con 0x7fe67c08f040 2021-04-09T18:20:21.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.027 7fe67affd700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe65c018930 0x7fe65c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:21.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.028 7fe67affd700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe65c018930 0x7fe65c01adc0 crc :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.047 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c1b7050 con 0x7fe67c08f040 2021-04-09T18:20:21.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.050 7fe663fff700 1 -- 172.21.15.32:0/3526790502 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe66c01f780 con 0x7fe67c08f040 2021-04-09T18:20:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.243 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe67c10d480 con 0x7fe67c08f040 2021-04-09T18:20:21.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.243 7fe663fff700 1 -- 172.21.15.32:0/3526790502 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fe66c00bd30 con 0x7fe67c08f040 2021-04-09T18:20:21.290 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:21.291 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe65c018930 msgr2=0x7fe65c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe65c018930 0x7fe65c01adc0 crc :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c08f040 msgr2=0x7fe67c1a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c08f040 0x7fe67c1a3600 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7fe66c006f00 tx=0x7fe66c00b940).stop 2021-04-09T18:20:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 shutdown_connections 2021-04-09T18:20:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe65c018930 0x7fe65c01adc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c1a22e0 0x7fe67c1a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c08f440 0x7fe67c1a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 --2- 172.21.15.32:0/3526790502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe67c08f040 0x7fe67c1a3600 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 >> 172.21.15.32:0/3526790502 conn(0x7fe67c199680 msgr2=0x7fe67c1a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 shutdown_connections 2021-04-09T18:20:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:21.246 7fe681fa1700 1 -- 172.21.15.32:0/3526790502 wait complete. 2021-04-09T18:20:21.708 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:21.709 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:22.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:22.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.819 7f1367f26700 1 Processor -- start 2021-04-09T18:20:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.819 7f1367f26700 1 -- start start 2021-04-09T18:20:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1367f26700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f136010d010 0x7f13600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1367f26700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13600d8490 0x7f136010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1367f26700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136010d480 con 0x7f13600d1da0 2021-04-09T18:20:22.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1367f26700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136010e780 con 0x7f13600d8490 2021-04-09T18:20:22.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1367f26700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13600d8d30 con 0x7f136010d010 2021-04-09T18:20:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1365541700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f13600d1da0 0x7f13600d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57322/0 (socket says 172.21.15.32:57322) 2021-04-09T18:20:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1365541700 1 -- 172.21.15.32:0/3142337501 learned_addr learned my addr 172.21.15.32:0/3142337501 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1365d42700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f136010d010 0x7f13600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1366543700 1 --2- 172.21.15.32:0/3142337501 >> v2:172.21.15.86:3300/0 conn(0x7f13600d8490 0x7f136010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1364d40700 1 -- 172.21.15.32:0/3142337501 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454524164 0 0) 0x7f1350000ce0 con 0x7f13600d1da0 2021-04-09T18:20:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1364d40700 1 -- 172.21.15.32:0/3142337501 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c002fe0 con 0x7f13600d1da0 2021-04-09T18:20:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1364d40700 1 -- 172.21.15.32:0/3142337501 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1193721127 0 0) 0x7f1350000ed0 con 0x7f13600d1da0 2021-04-09T18:20:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1364d40700 1 -- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f136010d010 msgr2=0x7f13600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1364d40700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f136010d010 0x7f13600d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1364d40700 1 -- 172.21.15.32:0/3142337501 >> v2:172.21.15.86:3300/0 conn(0x7f13600d8490 msgr2=0x7f136010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:22.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1364d40700 1 --2- 172.21.15.32:0/3142337501 >> v2:172.21.15.86:3300/0 conn(0x7f13600d8490 0x7f136010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.820 7f1364d40700 1 -- 172.21.15.32:0/3142337501 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13600d9220 con 0x7f13600d1da0 2021-04-09T18:20:22.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 -- 172.21.15.32:0/3142337501 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f1350001600 con 0x7f13600d1da0 2021-04-09T18:20:22.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 -- 172.21.15.32:0/3142337501 >> v1:172.21.15.32:6789/0 conn(0x7f13600d1da0 legacy=0x7f13600d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:22.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f134c003a70 0x7f134c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:22.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f134c04a7b0 0x7f134c04cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f134c04d520 0x7f134c04fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 -- 172.21.15.32:0/3142337501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0502f0 con 0x7f134c04d520 2021-04-09T18:20:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 -- 172.21.15.32:0/3142337501 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c050510 con 0x7f134c003a70 2021-04-09T18:20:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 -- 172.21.15.32:0/3142337501 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c050730 con 0x7f134c04a7b0 2021-04-09T18:20:22.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1366543700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f134c04d520 0x7f134c04fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:22.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1365d42700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f134c003a70 0x7f134c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:22.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1365541700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f134c04a7b0 0x7f134c04cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:22.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1366543700 1 -- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f134c04a7b0 msgr2=0x7f134c04cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:22.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1366543700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f134c04a7b0 0x7f134c04cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:22.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1366543700 1 -- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f134c003a70 msgr2=0x7f134c005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:22.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1366543700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f134c003a70 0x7f134c005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:22.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1366543700 1 -- 172.21.15.32:0/3142337501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0509d0 con 0x7f134c04d520 2021-04-09T18:20:22.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1366543700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f134c04d520 0x7f134c04fa40 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f1358006a40 tx=0x7f135800c0c0).ready entity=mon.0 client_cookie=a6bf496615dd21b7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:22.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 -- 172.21.15.32:0/3142337501 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f135800c4b0 con 0x7f134c04d520 2021-04-09T18:20:22.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 -- 172.21.15.32:0/3142337501 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135800c770 con 0x7f134c04d520 2021-04-09T18:20:22.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1364d40700 1 -- 172.21.15.32:0/3142337501 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f135800cbd0 con 0x7f134c04d520 2021-04-09T18:20:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1367f26700 1 -- 172.21.15.32:0/3142337501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f134c04d520 msgr2=0x7f134c04fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1367f26700 1 --2- 172.21.15.32:0/3142337501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f134c04d520 0x7f134c04fa40 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f1358006a40 tx=0x7f135800c0c0).stop 2021-04-09T18:20:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.821 7f1366543700 1 -- 172.21.15.32:0/3142337501 reap_dead start 2021-04-09T18:20:22.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.822 7f1367f26700 1 -- 172.21.15.32:0/3142337501 shutdown_connections 2021-04-09T18:20:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.822 7f1367f26700 1 -- 172.21.15.32:0/3142337501 >> 172.21.15.32:0/3142337501 conn(0x7f13600ce5f0 msgr2=0x7f13600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.822 7f1367f26700 1 -- 172.21.15.32:0/3142337501 shutdown_connections 2021-04-09T18:20:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.822 7f1367f26700 1 -- 172.21.15.32:0/3142337501 wait complete. 2021-04-09T18:20:22.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.842 7f1367f26700 1 Processor -- start 2021-04-09T18:20:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.842 7f1367f26700 1 -- start start 2021-04-09T18:20:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1367f26700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13600b5360 0x7f13601a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1367f26700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136018e140 0x7f13601aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:22.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1367f26700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13601ad040 0x7f136008fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1367f26700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13601a5360 con 0x7f13601ad040 2021-04-09T18:20:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1367f26700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13600902c0 con 0x7f136018e140 2021-04-09T18:20:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1367f26700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13600904e0 con 0x7f13600b5360 2021-04-09T18:20:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1366543700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13601ad040 0x7f136008fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:22.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1366543700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13601ad040 0x7f136008fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47744/0 (socket says 172.21.15.32:47744) 2021-04-09T18:20:22.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1366543700 1 -- 172.21.15.32:0/1190910064 learned_addr learned my addr 172.21.15.32:0/1190910064 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:22.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1365d42700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13600b5360 0x7f13601a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:22.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1365541700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136018e140 0x7f13601aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:22.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1366543700 1 -- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13600b5360 msgr2=0x7f13601a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1366543700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13600b5360 0x7f13601a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1366543700 1 -- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136018e140 msgr2=0x7f13601aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1366543700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136018e140 0x7f13601aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.843 7f1366543700 1 -- 172.21.15.32:0/1190910064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360183c80 con 0x7f13601ad040 2021-04-09T18:20:22.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.844 7f1366543700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13601ad040 0x7f136008fca0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f1358009fc0 tx=0x7f135800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.844 7f13567fc700 1 -- 172.21.15.32:0/1190910064 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1358009500 con 0x7f13601ad040 2021-04-09T18:20:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.844 7f13567fc700 1 -- 172.21.15.32:0/1190910064 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1358003030 con 0x7f13601ad040 2021-04-09T18:20:22.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.844 7f1367f26700 1 -- 172.21.15.32:0/1190910064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1360183c80 con 0x7f13601ad040 2021-04-09T18:20:22.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.844 7f13567fc700 1 -- 172.21.15.32:0/1190910064 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13580015a0 con 0x7f13601ad040 2021-04-09T18:20:22.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.844 7f1367f26700 1 -- 172.21.15.32:0/1190910064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1360183c80 con 0x7f13601ad040 2021-04-09T18:20:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.844 7f13567fc700 1 -- 172.21.15.32:0/1190910064 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f13580056b0 con 0x7f13601ad040 2021-04-09T18:20:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.845 7f13567fc700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:22.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.845 7f13567fc700 1 -- 172.21.15.32:0/1190910064 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1358032d70 con 0x7f13601ad040 2021-04-09T18:20:22.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.845 7f1365d42700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:22.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.845 7f1365d42700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.864 7f1367f26700 1 -- 172.21.15.32:0/1190910064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1360182d70 con 0x7f13601ad040 2021-04-09T18:20:22.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:22.867 7f13567fc700 1 -- 172.21.15.32:0/1190910064 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1358009500 con 0x7f13601ad040 2021-04-09T18:20:23.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.058 7f1367f26700 1 -- 172.21.15.32:0/1190910064 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f136010d480 con 0x7f13601ad040 2021-04-09T18:20:23.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.058 7f13567fc700 1 -- 172.21.15.32:0/1190910064 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f135801fab0 con 0x7f13601ad040 2021-04-09T18:20:23.105 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:23.106 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:23.106 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:23.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 -- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 msgr2=0x7f134801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:23.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:23.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 -- 172.21.15.32:0/1190910064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13601ad040 msgr2=0x7f136008fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:23.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13601ad040 0x7f136008fca0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f1358009fc0 tx=0x7f135800bd40).stop 2021-04-09T18:20:23.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 -- 172.21.15.32:0/1190910064 shutdown_connections 2021-04-09T18:20:23.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:23.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13601ad040 0x7f136008fca0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:23.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136018e140 0x7f13601aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:23.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 --2- 172.21.15.32:0/1190910064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13600b5360 0x7f13601a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:23.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 -- 172.21.15.32:0/1190910064 >> 172.21.15.32:0/1190910064 conn(0x7f136019b460 msgr2=0x7f136019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:23.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 -- 172.21.15.32:0/1190910064 shutdown_connections 2021-04-09T18:20:23.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:23.062 7f1367f26700 1 -- 172.21.15.32:0/1190910064 wait complete. 2021-04-09T18:20:23.524 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:23.526 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:24.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:24.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.641 7f13785ed700 1 Processor -- start 2021-04-09T18:20:24.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.641 7f13785ed700 1 -- start start 2021-04-09T18:20:24.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f13785ed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13700d6c80 0x7f13700d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f13785ed700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13700d2440 0x7f137010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:24.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f13785ed700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13700d7280 con 0x7f13700cffa0 2021-04-09T18:20:24.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f13785ed700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13700d7080 con 0x7f13700d2440 2021-04-09T18:20:24.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f13785ed700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13700d91e0 con 0x7f13700d6c80 2021-04-09T18:20:24.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1376c0a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13700d2440 0x7f137010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:24.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1375c08700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13700d6c80 0x7f13700d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:24.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1375c08700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13700d6c80 0x7f13700d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:54976/0 (socket says 172.21.15.32:54976) 2021-04-09T18:20:24.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1376c0a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13700d2440 0x7f137010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:39510/0 (socket says 172.21.15.32:39510) 2021-04-09T18:20:24.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1375c08700 1 -- 172.21.15.32:0/27376417 learned_addr learned my addr 172.21.15.32:0/27376417 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:24.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1376409700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f13700cffa0 0x7f13700d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55264/0 (socket says 172.21.15.32:55264) 2021-04-09T18:20:24.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1375c08700 1 -- 172.21.15.32:0/27376417 >> v2:172.21.15.86:3300/0 conn(0x7f13700d2440 msgr2=0x7f137010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:24.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1375c08700 1 --2- 172.21.15.32:0/27376417 >> v2:172.21.15.86:3300/0 conn(0x7f13700d2440 0x7f137010cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1375c08700 1 -- 172.21.15.32:0/27376417 >> v1:172.21.15.86:6789/0 conn(0x7f13700cffa0 legacy=0x7f13700d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:24.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1375407700 1 -- 172.21.15.32:0/27376417 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560711419 0 0) 0x7f136c000c60 con 0x7f13700cffa0 2021-04-09T18:20:24.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1375c08700 1 -- 172.21.15.32:0/27376417 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13700d2840 con 0x7f13700d6c80 2021-04-09T18:20:24.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1376c0a700 1 --2- 172.21.15.32:0/27376417 >> v2:172.21.15.86:3300/0 conn(0x7f13700d2440 0x7f137010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:24.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.642 7f1375c08700 1 --2- 172.21.15.32:0/27376417 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13700d6c80 0x7f13700d7ff0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1360006700 tx=0x7f1360009b80).ready entity=mon.2 client_cookie=ce283359a9e90853 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:24.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.643 7f1375407700 1 -- 172.21.15.32:0/27376417 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f136000c0c0 con 0x7f13700d6c80 2021-04-09T18:20:24.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.643 7f1375407700 1 -- 172.21.15.32:0/27376417 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136000c440 con 0x7f13700d6c80 2021-04-09T18:20:24.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.643 7f1375407700 1 -- 172.21.15.32:0/27376417 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f136000c920 con 0x7f13700d6c80 2021-04-09T18:20:24.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.643 7f13785ed700 1 -- 172.21.15.32:0/27376417 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13700d6c80 msgr2=0x7f13700d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:24.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.643 7f13785ed700 1 --2- 172.21.15.32:0/27376417 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13700d6c80 0x7f13700d7ff0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1360006700 tx=0x7f1360009b80).stop 2021-04-09T18:20:24.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.644 7f13785ed700 1 -- 172.21.15.32:0/27376417 shutdown_connections 2021-04-09T18:20:24.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.644 7f13785ed700 1 --2- 172.21.15.32:0/27376417 >> v2:172.21.15.86:3300/0 conn(0x7f13700d2440 0x7f137010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.644 7f13785ed700 1 --2- 172.21.15.32:0/27376417 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13700d6c80 0x7f13700d7ff0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.644 7f13785ed700 1 -- 172.21.15.32:0/27376417 >> 172.21.15.32:0/27376417 conn(0x7f13700ce5c0 msgr2=0x7f13700cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.644 7f13785ed700 1 -- 172.21.15.32:0/27376417 shutdown_connections 2021-04-09T18:20:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.644 7f13785ed700 1 -- 172.21.15.32:0/27376417 wait complete. 2021-04-09T18:20:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f13785ed700 1 Processor -- start 2021-04-09T18:20:24.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f13785ed700 1 -- start start 2021-04-09T18:20:24.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f13785ed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13701929f0 0x7f1370192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:24.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f13785ed700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137008ddc0 0x7f137008e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:24.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f13785ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f137008e7e0 0x7f137008c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:24.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f13785ed700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1370193410 con 0x7f137008e7e0 2021-04-09T18:20:24.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f13785ed700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13700a9140 con 0x7f137008ddc0 2021-04-09T18:20:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f13785ed700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137008ce80 con 0x7f13701929f0 2021-04-09T18:20:24.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f1376c0a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f137008e7e0 0x7f137008c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f1376c0a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f137008e7e0 0x7f137008c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47758/0 (socket says 172.21.15.32:47758) 2021-04-09T18:20:24.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f1376c0a700 1 -- 172.21.15.32:0/1213245370 learned_addr learned my addr 172.21.15.32:0/1213245370 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f1376409700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13701929f0 0x7f1370192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.664 7f1375c08700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137008ddc0 0x7f137008e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1376c0a700 1 -- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13701929f0 msgr2=0x7f1370192df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1376c0a700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13701929f0 0x7f1370192df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1376c0a700 1 -- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137008ddc0 msgr2=0x7f137008e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:24.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1376c0a700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137008ddc0 0x7f137008e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1376c0a700 1 -- 172.21.15.32:0/1213245370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137008d0a0 con 0x7f137008e7e0 2021-04-09T18:20:24.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1375c08700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137008ddc0 0x7f137008e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:24.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1376409700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13701929f0 0x7f1370192df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1376c0a700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f137008e7e0 0x7f137008c860 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f136800b970 tx=0x7f136800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1366ffd700 1 -- 172.21.15.32:0/1213245370 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1368000f90 con 0x7f137008e7e0 2021-04-09T18:20:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1366ffd700 1 -- 172.21.15.32:0/1213245370 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1368006aa0 con 0x7f137008e7e0 2021-04-09T18:20:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f13785ed700 1 -- 172.21.15.32:0/1213245370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137008d0a0 con 0x7f137008e7e0 2021-04-09T18:20:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f1366ffd700 1 -- 172.21.15.32:0/1213245370 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13680015f0 con 0x7f137008e7e0 2021-04-09T18:20:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.665 7f13785ed700 1 -- 172.21.15.32:0/1213245370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137008d0a0 con 0x7f137008e7e0 2021-04-09T18:20:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.666 7f1366ffd700 1 -- 172.21.15.32:0/1213245370 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1368000ed0 con 0x7f137008e7e0 2021-04-09T18:20:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.666 7f1366ffd700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1358018930 0x7f135801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.666 7f1366ffd700 1 -- 172.21.15.32:0/1213245370 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1368032da0 con 0x7f137008e7e0 2021-04-09T18:20:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.666 7f1376409700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1358018930 0x7f135801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:24.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.666 7f1376409700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1358018930 0x7f135801adc0 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:24.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.685 7f13785ed700 1 -- 172.21.15.32:0/1213245370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13701b70c0 con 0x7f137008e7e0 2021-04-09T18:20:24.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.689 7f1366ffd700 1 -- 172.21.15.32:0/1213245370 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f13680066f0 con 0x7f137008e7e0 2021-04-09T18:20:24.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.879 7f13785ed700 1 -- 172.21.15.32:0/1213245370 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13700d7280 con 0x7f137008e7e0 2021-04-09T18:20:24.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.879 7f1366ffd700 1 -- 172.21.15.32:0/1213245370 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f1368006aa0 con 0x7f137008e7e0 2021-04-09T18:20:24.931 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:24.932 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:24.932 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:24.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 -- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1358018930 msgr2=0x7f135801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:24.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1358018930 0x7f135801adc0 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 -- 172.21.15.32:0/1213245370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f137008e7e0 msgr2=0x7f137008c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f137008e7e0 0x7f137008c860 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f136800b970 tx=0x7f136800bd10).stop 2021-04-09T18:20:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 -- 172.21.15.32:0/1213245370 shutdown_connections 2021-04-09T18:20:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1358018930 0x7f135801adc0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f137008e7e0 0x7f137008c860 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137008ddc0 0x7f137008e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 --2- 172.21.15.32:0/1213245370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13701929f0 0x7f1370192df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 -- 172.21.15.32:0/1213245370 >> 172.21.15.32:0/1213245370 conn(0x7f1370189b00 msgr2=0x7f13701921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 -- 172.21.15.32:0/1213245370 shutdown_connections 2021-04-09T18:20:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:24.882 7f13785ed700 1 -- 172.21.15.32:0/1213245370 wait complete. 2021-04-09T18:20:25.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:25.359 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:26.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.476 7fd62b558700 1 Processor -- start 2021-04-09T18:20:26.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.476 7fd62b558700 1 -- start start 2021-04-09T18:20:26.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd62b558700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd62410d010 0x7fd6240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:26.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd62b558700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6240d8490 0x7fd62410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:26.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd62b558700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62410d480 con 0x7fd6240d1da0 2021-04-09T18:20:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd62b558700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6240d8d30 con 0x7fd62410d010 2021-04-09T18:20:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd62b558700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6240d9220 con 0x7fd6240d8490 2021-04-09T18:20:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd628b73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd6240d1da0 0x7fd6240d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57354/0 (socket says 172.21.15.32:57354) 2021-04-09T18:20:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd628b73700 1 -- 172.21.15.32:0/819326757 learned_addr learned my addr 172.21.15.32:0/819326757 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:26.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd629374700 1 --2- 172.21.15.32:0/819326757 >> v2:172.21.15.86:3300/0 conn(0x7fd62410d010 0x7fd6240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:26.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd629b75700 1 --2- 172.21.15.32:0/819326757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6240d8490 0x7fd62410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:26.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd61bfff700 1 -- 172.21.15.32:0/819326757 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532323110 0 0) 0x7fd614000ce0 con 0x7fd6240d1da0 2021-04-09T18:20:26.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd61bfff700 1 -- 172.21.15.32:0/819326757 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd610002fe0 con 0x7fd6240d1da0 2021-04-09T18:20:26.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd629b75700 1 -- 172.21.15.32:0/819326757 >> v2:172.21.15.86:3300/0 conn(0x7fd62410d010 msgr2=0x7fd6240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:26.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd629b75700 1 --2- 172.21.15.32:0/819326757 >> v2:172.21.15.86:3300/0 conn(0x7fd62410d010 0x7fd6240d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd629b75700 1 -- 172.21.15.32:0/819326757 >> v1:172.21.15.32:6789/0 conn(0x7fd6240d1da0 legacy=0x7fd6240d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:26.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd629b75700 1 -- 172.21.15.32:0/819326757 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62410e020 con 0x7fd6240d8490 2021-04-09T18:20:26.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd629374700 1 --2- 172.21.15.32:0/819326757 >> v2:172.21.15.86:3300/0 conn(0x7fd62410d010 0x7fd6240d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:26.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.477 7fd629b75700 1 --2- 172.21.15.32:0/819326757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6240d8490 0x7fd62410da00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fd61c006e50 tx=0x7fd61c009e70).ready entity=mon.2 client_cookie=73728e71fe03419e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:26.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.478 7fd61bfff700 1 -- 172.21.15.32:0/819326757 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd61c006a20 con 0x7fd6240d8490 2021-04-09T18:20:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.478 7fd61bfff700 1 -- 172.21.15.32:0/819326757 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61c006790 con 0x7fd6240d8490 2021-04-09T18:20:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.478 7fd61bfff700 1 -- 172.21.15.32:0/819326757 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd61c00b700 con 0x7fd6240d8490 2021-04-09T18:20:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.478 7fd62b558700 1 -- 172.21.15.32:0/819326757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6240d8490 msgr2=0x7fd62410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.478 7fd62b558700 1 --2- 172.21.15.32:0/819326757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6240d8490 0x7fd62410da00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fd61c006e50 tx=0x7fd61c009e70).stop 2021-04-09T18:20:26.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.478 7fd62b558700 1 -- 172.21.15.32:0/819326757 shutdown_connections 2021-04-09T18:20:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.478 7fd62b558700 1 --2- 172.21.15.32:0/819326757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6240d8490 0x7fd62410da00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.478 7fd62b558700 1 --2- 172.21.15.32:0/819326757 >> v2:172.21.15.86:3300/0 conn(0x7fd62410d010 0x7fd6240d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.478 7fd62b558700 1 -- 172.21.15.32:0/819326757 >> 172.21.15.32:0/819326757 conn(0x7fd6240ce5f0 msgr2=0x7fd6240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:26.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.479 7fd62b558700 1 -- 172.21.15.32:0/819326757 shutdown_connections 2021-04-09T18:20:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.479 7fd62b558700 1 -- 172.21.15.32:0/819326757 wait complete. 2021-04-09T18:20:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.499 7fd62b558700 1 Processor -- start 2021-04-09T18:20:26.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.499 7fd62b558700 1 -- start start 2021-04-09T18:20:26.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd62b558700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6241a5210 0x7fd6241a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd62b558700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6241a6340 0x7fd6241a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:26.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd62b558700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240abaa0 0x7fd6240abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd62b558700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6241a6d60 con 0x7fd6241a6340 2021-04-09T18:20:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd62b558700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6240ac4c0 con 0x7fd6240abaa0 2021-04-09T18:20:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd62b558700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6240ac6e0 con 0x7fd6241a5210 2021-04-09T18:20:26.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd628b73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6241a6340 0x7fd6241a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd628b73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6241a6340 0x7fd6241a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47770/0 (socket says 172.21.15.32:47770) 2021-04-09T18:20:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd628b73700 1 -- 172.21.15.32:0/1443892342 learned_addr learned my addr 172.21.15.32:0/1443892342 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd629374700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6241a5210 0x7fd6241a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd629b75700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240abaa0 0x7fd6240abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:26.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd629374700 1 -- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240abaa0 msgr2=0x7fd6240abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd629374700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240abaa0 0x7fd6240abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd629374700 1 -- 172.21.15.32:0/1443892342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6241a6340 msgr2=0x7fd6241a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:26.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd629374700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6241a6340 0x7fd6241a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd629374700 1 -- 172.21.15.32:0/1443892342 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62417a720 con 0x7fd6241a5210 2021-04-09T18:20:26.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd628b73700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6241a6340 0x7fd6241a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:26.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd629374700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6241a5210 0x7fd6241a56c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd620009540 tx=0x7fd62000b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:26.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd619ffb700 1 -- 172.21.15.32:0/1443892342 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd62000bcd0 con 0x7fd6241a5210 2021-04-09T18:20:26.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd619ffb700 1 -- 172.21.15.32:0/1443892342 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd620006740 con 0x7fd6241a5210 2021-04-09T18:20:26.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.500 7fd62b558700 1 -- 172.21.15.32:0/1443892342 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62417a9c0 con 0x7fd6241a5210 2021-04-09T18:20:26.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.501 7fd619ffb700 1 -- 172.21.15.32:0/1443892342 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd620002220 con 0x7fd6241a5210 2021-04-09T18:20:26.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.501 7fd62b558700 1 -- 172.21.15.32:0/1443892342 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62417a9c0 con 0x7fd6241a5210 2021-04-09T18:20:26.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.501 7fd619ffb700 1 -- 172.21.15.32:0/1443892342 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd620009fe0 con 0x7fd6241a5210 2021-04-09T18:20:26.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.501 7fd619ffb700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd604018930 0x7fd60401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:26.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.501 7fd619ffb700 1 -- 172.21.15.32:0/1443892342 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd620032960 con 0x7fd6241a5210 2021-04-09T18:20:26.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.501 7fd628b73700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd604018930 0x7fd60401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:26.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.502 7fd628b73700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd604018930 0x7fd60401ae10 crc :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:26.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.521 7fd62b558700 1 -- 172.21.15.32:0/1443892342 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6241b7070 con 0x7fd6241a5210 2021-04-09T18:20:26.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.525 7fd619ffb700 1 -- 172.21.15.32:0/1443892342 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd620006740 con 0x7fd6241a5210 2021-04-09T18:20:26.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.715 7fd62b558700 1 -- 172.21.15.32:0/1443892342 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd62410d480 con 0x7fd6241a5210 2021-04-09T18:20:26.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.716 7fd619ffb700 1 -- 172.21.15.32:0/1443892342 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fd620000c90 con 0x7fd6241a5210 2021-04-09T18:20:26.763 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:26.764 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:26.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.718 7fd62b558700 1 -- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd604018930 msgr2=0x7fd60401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:26.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.718 7fd62b558700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd604018930 0x7fd60401ae10 crc :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.718 7fd62b558700 1 -- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6241a5210 msgr2=0x7fd6241a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:26.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.718 7fd62b558700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6241a5210 0x7fd6241a56c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd620009540 tx=0x7fd62000b8e0).stop 2021-04-09T18:20:26.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.719 7fd62b558700 1 -- 172.21.15.32:0/1443892342 shutdown_connections 2021-04-09T18:20:26.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.719 7fd62b558700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd604018930 0x7fd60401ae10 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.719 7fd62b558700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6241a6340 0x7fd6241a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.719 7fd62b558700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240abaa0 0x7fd6240abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.719 7fd62b558700 1 --2- 172.21.15.32:0/1443892342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6241a5210 0x7fd6241a56c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:26.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.719 7fd62b558700 1 -- 172.21.15.32:0/1443892342 >> 172.21.15.32:0/1443892342 conn(0x7fd6241a99c0 msgr2=0x7fd62417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:26.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.719 7fd62b558700 1 -- 172.21.15.32:0/1443892342 shutdown_connections 2021-04-09T18:20:26.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:26.719 7fd62b558700 1 -- 172.21.15.32:0/1443892342 wait complete. 2021-04-09T18:20:27.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:27.213 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:28.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:28.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.331 7f82f7803700 1 Processor -- start 2021-04-09T18:20:28.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.331 7f82f7803700 1 -- start start 2021-04-09T18:20:28.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.332 7f82f7803700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f010d010 0x7f82f00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.332 7f82f7803700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82f00d1da0 0x7f82f00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.332 7f82f7803700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f010d480 con 0x7f82f00d82c0 2021-04-09T18:20:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.332 7f82f7803700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f010e780 con 0x7f82f00d1da0 2021-04-09T18:20:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.332 7f82f7803700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00d8d30 con 0x7f82f010d010 2021-04-09T18:20:28.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.332 7f82f4e1e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82f00d1da0 0x7f82f00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:28.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.332 7f82f4e1e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82f00d1da0 0x7f82f00d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47780/0 (socket says 172.21.15.32:47780) 2021-04-09T18:20:28.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.332 7f82f4e1e700 1 -- 172.21.15.32:0/3653905451 learned_addr learned my addr 172.21.15.32:0/3653905451 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:28.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.332 7f82f561f700 1 --2- 172.21.15.32:0/3653905451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f010d010 0x7f82f00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:28.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82f4e1e700 1 -- 172.21.15.32:0/3653905451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f010d010 msgr2=0x7f82f00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:28.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82f4e1e700 1 --2- 172.21.15.32:0/3653905451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f010d010 0x7f82f00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82f4e1e700 1 -- 172.21.15.32:0/3653905451 >> v1:172.21.15.86:6789/0 conn(0x7f82f00d82c0 legacy=0x7f82f010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:28.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82f4e1e700 1 -- 172.21.15.32:0/3653905451 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00d9220 con 0x7f82f00d1da0 2021-04-09T18:20:28.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82f4e1e700 1 --2- 172.21.15.32:0/3653905451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00d1da0 0x7f82f00d7ca0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f82e0006700 tx=0x7f82e0009b80).ready entity=mon.0 client_cookie=460b29c3759a6b75 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82e7fff700 1 -- 172.21.15.32:0/3653905451 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e000c0c0 con 0x7f82f00d1da0 2021-04-09T18:20:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82e7fff700 1 -- 172.21.15.32:0/3653905451 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e000c440 con 0x7f82f00d1da0 2021-04-09T18:20:28.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82e7fff700 1 -- 172.21.15.32:0/3653905451 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e000c920 con 0x7f82f00d1da0 2021-04-09T18:20:28.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82f7803700 1 -- 172.21.15.32:0/3653905451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00d1da0 msgr2=0x7f82f00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:28.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.333 7f82f7803700 1 --2- 172.21.15.32:0/3653905451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00d1da0 0x7f82f00d7ca0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f82e0006700 tx=0x7f82e0009b80).stop 2021-04-09T18:20:28.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.334 7f82f7803700 1 -- 172.21.15.32:0/3653905451 shutdown_connections 2021-04-09T18:20:28.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.334 7f82f7803700 1 --2- 172.21.15.32:0/3653905451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f00d1da0 0x7f82f00d7ca0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.334 7f82f7803700 1 --2- 172.21.15.32:0/3653905451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f010d010 0x7f82f00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.334 7f82f7803700 1 -- 172.21.15.32:0/3653905451 >> 172.21.15.32:0/3653905451 conn(0x7f82f00ce5f0 msgr2=0x7f82f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:28.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.334 7f82f7803700 1 -- 172.21.15.32:0/3653905451 shutdown_connections 2021-04-09T18:20:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.334 7f82f7803700 1 -- 172.21.15.32:0/3653905451 wait complete. 2021-04-09T18:20:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.354 7f82f7803700 1 Processor -- start 2021-04-09T18:20:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.354 7f82f7803700 1 -- start start 2021-04-09T18:20:28.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f7803700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01aca20 0x7f82f01ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:28.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f7803700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f018bd40 0x7f82f018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:28.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f7803700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f008fba0 0x7f82f008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f7803700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f018c7e0 con 0x7f82f018bd40 2021-04-09T18:20:28.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f7803700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f018ca30 con 0x7f82f01aca20 2021-04-09T18:20:28.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f7803700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00905f0 con 0x7f82f008fba0 2021-04-09T18:20:28.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f4e1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f018bd40 0x7f82f018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:28.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f4e1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f018bd40 0x7f82f018c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47786/0 (socket says 172.21.15.32:47786) 2021-04-09T18:20:28.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f4e1e700 1 -- 172.21.15.32:0/1182451547 learned_addr learned my addr 172.21.15.32:0/1182451547 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:28.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f561f700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01aca20 0x7f82f01ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:28.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f5e20700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f008fba0 0x7f82f008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:28.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f4e1e700 1 -- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f008fba0 msgr2=0x7f82f008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:28.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f4e1e700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f008fba0 0x7f82f008ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f4e1e700 1 -- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01aca20 msgr2=0x7f82f01ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:28.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f4e1e700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01aca20 0x7f82f01ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f4e1e700 1 -- 172.21.15.32:0/1182451547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00b4300 con 0x7f82f018bd40 2021-04-09T18:20:28.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f5e20700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f008fba0 0x7f82f008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:20:28.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82f4e1e700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f018bd40 0x7f82f018c140 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7f82e000b650 tx=0x7f82e000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:28.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82e5ffb700 1 -- 172.21.15.32:0/1182451547 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e000bde0 con 0x7f82f018bd40 2021-04-09T18:20:28.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.355 7f82e5ffb700 1 -- 172.21.15.32:0/1182451547 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e00091e0 con 0x7f82f018bd40 2021-04-09T18:20:28.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.356 7f82f7803700 1 -- 172.21.15.32:0/1182451547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f00b4300 con 0x7f82f018bd40 2021-04-09T18:20:28.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.356 7f82e5ffb700 1 -- 172.21.15.32:0/1182451547 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82e0002220 con 0x7f82f018bd40 2021-04-09T18:20:28.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.356 7f82f7803700 1 -- 172.21.15.32:0/1182451547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f00b4300 con 0x7f82f018bd40 2021-04-09T18:20:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.356 7f82e5ffb700 1 -- 172.21.15.32:0/1182451547 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f82e0000c90 con 0x7f82f018bd40 2021-04-09T18:20:28.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.356 7f82e5ffb700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82d0018930 0x7f82d001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:28.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.356 7f82e5ffb700 1 -- 172.21.15.32:0/1182451547 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f82e0032a90 con 0x7f82f018bd40 2021-04-09T18:20:28.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.357 7f82f561f700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82d0018930 0x7f82d001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:28.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.357 7f82f561f700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82d0018930 0x7f82d001ae10 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:28.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.376 7f82f7803700 1 -- 172.21.15.32:0/1182451547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f0182d70 con 0x7f82f018bd40 2021-04-09T18:20:28.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.379 7f82e5ffb700 1 -- 172.21.15.32:0/1182451547 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f82e0000940 con 0x7f82f018bd40 2021-04-09T18:20:28.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.572 7f82f7803700 1 -- 172.21.15.32:0/1182451547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82f018c760 con 0x7f82f018bd40 2021-04-09T18:20:28.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.572 7f82e5ffb700 1 -- 172.21.15.32:0/1182451547 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f82e000bde0 con 0x7f82f018bd40 2021-04-09T18:20:28.619 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:28.620 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:28.620 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 -- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82d0018930 msgr2=0x7f82d001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82d0018930 0x7f82d001ae10 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 -- 172.21.15.32:0/1182451547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f018bd40 msgr2=0x7f82f018c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f018bd40 0x7f82f018c140 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7f82e000b650 tx=0x7f82e000b9f0).stop 2021-04-09T18:20:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 -- 172.21.15.32:0/1182451547 shutdown_connections 2021-04-09T18:20:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f82d0018930 0x7f82d001ae10 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82f018bd40 0x7f82f018c140 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82f008fba0 0x7f82f008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 --2- 172.21.15.32:0/1182451547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82f01aca20 0x7f82f01ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:28.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 -- 172.21.15.32:0/1182451547 >> 172.21.15.32:0/1182451547 conn(0x7f82f019b460 msgr2=0x7f82f019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 -- 172.21.15.32:0/1182451547 shutdown_connections 2021-04-09T18:20:28.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:28.575 7f82f7803700 1 -- 172.21.15.32:0/1182451547 wait complete. 2021-04-09T18:20:29.040 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:29.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:30.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.158 7f5f3c81f700 1 Processor -- start 2021-04-09T18:20:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.158 7f5f3c81f700 1 -- start start 2021-04-09T18:20:30.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.159 7f5f3c81f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f3810da00 0x7f5f3810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:30.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.159 7f5f3c81f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3810d480 con 0x7f5f3810d010 2021-04-09T18:20:30.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.159 7f5f3c81f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f380d1f70 con 0x7f5f380d7ca0 2021-04-09T18:20:30.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.159 7f5f3c81f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f380d8d30 con 0x7f5f3810da00 2021-04-09T18:20:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.159 7f5f36e1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f3810da00 0x7f5f3810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.159 7f5f3661c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5f3810d010 0x7f5f380d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57380/0 (socket says 172.21.15.32:57380) 2021-04-09T18:20:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f3661c700 1 -- 172.21.15.32:0/4202221523 learned_addr learned my addr 172.21.15.32:0/4202221523 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:30.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f36e1d700 1 -- 172.21.15.32:0/4202221523 >> v1:172.21.15.86:6789/0 conn(0x7f5f380d7ca0 legacy=0x7f5f380d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:30.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f36e1d700 1 -- 172.21.15.32:0/4202221523 >> v1:172.21.15.32:6789/0 conn(0x7f5f3810d010 legacy=0x7f5f380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:30.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f36e1d700 1 -- 172.21.15.32:0/4202221523 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f380d9220 con 0x7f5f3810da00 2021-04-09T18:20:30.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f3561a700 1 -- 172.21.15.32:0/4202221523 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105674775 0 0) 0x7f5f28000ce0 con 0x7f5f3810d010 2021-04-09T18:20:30.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f36e1d700 1 --2- 172.21.15.32:0/4202221523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3810da00 0x7f5f3810de00 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f5f30006e30 tx=0x7f5f30009de0).ready entity=mon.0 client_cookie=17ba8631ce96ba72 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:30.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f3561a700 1 -- 172.21.15.32:0/4202221523 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f3000b300 con 0x7f5f3810da00 2021-04-09T18:20:30.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f3561a700 1 -- 172.21.15.32:0/4202221523 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3000b680 con 0x7f5f3810da00 2021-04-09T18:20:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f3561a700 1 -- 172.21.15.32:0/4202221523 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f3000bb60 con 0x7f5f3810da00 2021-04-09T18:20:30.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f3c81f700 1 -- 172.21.15.32:0/4202221523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3810da00 msgr2=0x7f5f3810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:30.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.160 7f5f3c81f700 1 --2- 172.21.15.32:0/4202221523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3810da00 0x7f5f3810de00 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f5f30006e30 tx=0x7f5f30009de0).stop 2021-04-09T18:20:30.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.161 7f5f3c81f700 1 -- 172.21.15.32:0/4202221523 shutdown_connections 2021-04-09T18:20:30.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.161 7f5f3c81f700 1 --2- 172.21.15.32:0/4202221523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f3810da00 0x7f5f3810de00 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:30.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.161 7f5f3c81f700 1 -- 172.21.15.32:0/4202221523 >> 172.21.15.32:0/4202221523 conn(0x7f5f380ce5f0 msgr2=0x7f5f380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:30.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.161 7f5f3c81f700 1 -- 172.21.15.32:0/4202221523 shutdown_connections 2021-04-09T18:20:30.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.161 7f5f3c81f700 1 -- 172.21.15.32:0/4202221523 wait complete. 2021-04-09T18:20:30.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.181 7f5f3c81f700 1 Processor -- start 2021-04-09T18:20:30.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.181 7f5f3c81f700 1 -- start start 2021-04-09T18:20:30.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3c81f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f38192a10 0x7f5f381ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:30.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3c81f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f381acaa0 0x7f5f381aa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:30.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3c81f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f381aad30 0x7f5f381ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3c81f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f381abf00 con 0x7f5f38192a10 2021-04-09T18:20:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3c81f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f381acea0 con 0x7f5f381aad30 2021-04-09T18:20:30.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3c81f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f38180860 con 0x7f5f381acaa0 2021-04-09T18:20:30.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3661c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f38192a10 0x7f5f381ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:30.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3661c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f38192a10 0x7f5f381ac480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47796/0 (socket says 172.21.15.32:47796) 2021-04-09T18:20:30.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3661c700 1 -- 172.21.15.32:0/3290118844 learned_addr learned my addr 172.21.15.32:0/3290118844 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:30.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f36e1d700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f381aad30 0x7f5f381ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:30.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f35e1b700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f381acaa0 0x7f5f381aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3661c700 1 -- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f381acaa0 msgr2=0x7f5f381aa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3661c700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f381acaa0 0x7f5f381aa710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3661c700 1 -- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f381aad30 msgr2=0x7f5f381ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3661c700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f381aad30 0x7f5f381ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3661c700 1 -- 172.21.15.32:0/3290118844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f381ab750 con 0x7f5f38192a10 2021-04-09T18:20:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f36e1d700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f381aad30 0x7f5f381ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:20:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f35e1b700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f381acaa0 0x7f5f381aa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:20:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.182 7f5f3661c700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f38192a10 0x7f5f381ac480 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f5f28006f00 tx=0x7f5f2800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.183 7f5f26ffd700 1 -- 172.21.15.32:0/3290118844 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f28006020 con 0x7f5f38192a10 2021-04-09T18:20:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.183 7f5f26ffd700 1 -- 172.21.15.32:0/3290118844 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f280063a0 con 0x7f5f38192a10 2021-04-09T18:20:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.183 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f381ab750 con 0x7f5f38192a10 2021-04-09T18:20:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.183 7f5f26ffd700 1 -- 172.21.15.32:0/3290118844 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f28004620 con 0x7f5f38192a10 2021-04-09T18:20:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.183 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f381abce0 con 0x7f5f38192a10 2021-04-09T18:20:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.183 7f5f26ffd700 1 -- 172.21.15.32:0/3290118844 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5f2800a000 con 0x7f5f38192a10 2021-04-09T18:20:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.183 7f5f26ffd700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f1c018930 0x7f5f1c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:30.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.183 7f5f26ffd700 1 -- 172.21.15.32:0/3290118844 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5f28032a10 con 0x7f5f38192a10 2021-04-09T18:20:30.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.184 7f5f35e1b700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f1c018930 0x7f5f1c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:30.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.184 7f5f35e1b700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f1c018930 0x7f5f1c01adc0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:30.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.203 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3817ca90 con 0x7f5f38192a10 2021-04-09T18:20:30.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.207 7f5f26ffd700 1 -- 172.21.15.32:0/3290118844 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5f28006b90 con 0x7f5f38192a10 2021-04-09T18:20:30.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.398 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f3810d480 con 0x7f5f38192a10 2021-04-09T18:20:30.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.399 7f5f26ffd700 1 -- 172.21.15.32:0/3290118844 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f5f2800a000 con 0x7f5f38192a10 2021-04-09T18:20:30.446 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:30.447 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:30.447 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:30.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f1c018930 msgr2=0x7f5f1c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:30.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f1c018930 0x7f5f1c01adc0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:30.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f38192a10 msgr2=0x7f5f381ac480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f38192a10 0x7f5f381ac480 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f5f28006f00 tx=0x7f5f2800b940).stop 2021-04-09T18:20:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 shutdown_connections 2021-04-09T18:20:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f1c018930 0x7f5f1c01adc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:30.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f381aad30 0x7f5f381ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:30.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f381acaa0 0x7f5f381aa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 --2- 172.21.15.32:0/3290118844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f38192a10 0x7f5f381ac480 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 >> 172.21.15.32:0/3290118844 conn(0x7f5f38188510 msgr2=0x7f5f381a7320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.401 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 shutdown_connections 2021-04-09T18:20:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:30.402 7f5f3c81f700 1 -- 172.21.15.32:0/3290118844 wait complete. 2021-04-09T18:20:30.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:30.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:31.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:32.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.978 7f8644b4d700 1 Processor -- start 2021-04-09T18:20:32.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.978 7f8644b4d700 1 -- start start 2021-04-09T18:20:32.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f8644b4d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400d7ca0 0x7f86400d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:32.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f8644b4d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f864010da00 0x7f864010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f8644b4d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864010d480 con 0x7f864010d010 2021-04-09T18:20:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f8644b4d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86400d9220 con 0x7f864010da00 2021-04-09T18:20:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f8644b4d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86400d86c0 con 0x7f86400d7ca0 2021-04-09T18:20:32.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863f05f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f864010da00 0x7f864010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863e85e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f864010d010 0x7f86400d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57394/0 (socket says 172.21.15.32:57394) 2021-04-09T18:20:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863e85e700 1 -- 172.21.15.32:0/2661548062 learned_addr learned my addr 172.21.15.32:0/2661548062 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863e05d700 1 --2- 172.21.15.32:0/2661548062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400d7ca0 0x7f86400d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863d85c700 1 -- 172.21.15.32:0/2661548062 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921891316 0 0) 0x7f8630000ce0 con 0x7f864010d010 2021-04-09T18:20:32.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863d85c700 1 -- 172.21.15.32:0/2661548062 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8624002fe0 con 0x7f864010d010 2021-04-09T18:20:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863f05f700 1 -- 172.21.15.32:0/2661548062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400d7ca0 msgr2=0x7f86400d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:32.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863f05f700 1 --2- 172.21.15.32:0/2661548062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400d7ca0 0x7f86400d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863f05f700 1 -- 172.21.15.32:0/2661548062 >> v1:172.21.15.32:6789/0 conn(0x7f864010d010 legacy=0x7f86400d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863f05f700 1 -- 172.21.15.32:0/2661548062 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400d1f70 con 0x7f864010da00 2021-04-09T18:20:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.979 7f863e05d700 1 --2- 172.21.15.32:0/2661548062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400d7ca0 0x7f86400d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:20:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.980 7f863f05f700 1 --2- 172.21.15.32:0/2661548062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f864010da00 0x7f864010de00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7f8638006e30 tx=0x7f8638009e60).ready entity=mon.0 client_cookie=d33ceeb13dc53cfa server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.980 7f863d85c700 1 -- 172.21.15.32:0/2661548062 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8638006a50 con 0x7f864010da00 2021-04-09T18:20:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.980 7f863d85c700 1 -- 172.21.15.32:0/2661548062 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863800b300 con 0x7f864010da00 2021-04-09T18:20:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.980 7f863d85c700 1 -- 172.21.15.32:0/2661548062 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f863800b750 con 0x7f864010da00 2021-04-09T18:20:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.980 7f8644b4d700 1 -- 172.21.15.32:0/2661548062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f864010da00 msgr2=0x7f864010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.980 7f8644b4d700 1 --2- 172.21.15.32:0/2661548062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f864010da00 0x7f864010de00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7f8638006e30 tx=0x7f8638009e60).stop 2021-04-09T18:20:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.981 7f8644b4d700 1 -- 172.21.15.32:0/2661548062 shutdown_connections 2021-04-09T18:20:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.981 7f8644b4d700 1 --2- 172.21.15.32:0/2661548062 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f864010da00 0x7f864010de00 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.981 7f8644b4d700 1 --2- 172.21.15.32:0/2661548062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400d7ca0 0x7f86400d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.981 7f8644b4d700 1 -- 172.21.15.32:0/2661548062 >> 172.21.15.32:0/2661548062 conn(0x7f86400ce5f0 msgr2=0x7f86400d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.981 7f8644b4d700 1 -- 172.21.15.32:0/2661548062 shutdown_connections 2021-04-09T18:20:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:31.981 7f8644b4d700 1 -- 172.21.15.32:0/2661548062 wait complete. 2021-04-09T18:20:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.001 7f8644b4d700 1 Processor -- start 2021-04-09T18:20:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.001 7f8644b4d700 1 -- start start 2021-04-09T18:20:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.001 7f8644b4d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86400a70b0 0x7f8640190100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.001 7f8644b4d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400a74b0 0x7f86401912a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.001 7f8644b4d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8640190720 0x7f86401918c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.001 7f8644b4d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8640190d20 con 0x7f8640190720 2021-04-09T18:20:32.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.001 7f8644b4d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864018a9a0 con 0x7f86400a70b0 2021-04-09T18:20:32.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.001 7f8644b4d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864018c820 con 0x7f86400a74b0 2021-04-09T18:20:32.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.001 7f863f05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8640190720 0x7f86401918c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:32.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863f05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8640190720 0x7f86401918c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47814/0 (socket says 172.21.15.32:47814) 2021-04-09T18:20:32.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863f05f700 1 -- 172.21.15.32:0/3416638850 learned_addr learned my addr 172.21.15.32:0/3416638850 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:32.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863e05d700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400a74b0 0x7f86401912a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:32.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863e85e700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86400a70b0 0x7f8640190100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:32.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863f05f700 1 -- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400a74b0 msgr2=0x7f86401912a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:32.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863f05f700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400a74b0 0x7f86401912a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863f05f700 1 -- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86400a70b0 msgr2=0x7f8640190100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863f05f700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86400a70b0 0x7f8640190100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863f05f700 1 -- 172.21.15.32:0/3416638850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864018ca40 con 0x7f8640190720 2021-04-09T18:20:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f863f05f700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8640190720 0x7f86401918c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f863800b910 tx=0x7f863800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:32.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f86377fe700 1 -- 172.21.15.32:0/3416638850 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f863800bdf0 con 0x7f8640190720 2021-04-09T18:20:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f86377fe700 1 -- 172.21.15.32:0/3416638850 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8638005820 con 0x7f8640190720 2021-04-09T18:20:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864018ca40 con 0x7f8640190720 2021-04-09T18:20:32.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f86377fe700 1 -- 172.21.15.32:0/3416638850 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8638005cf0 con 0x7f8640190720 2021-04-09T18:20:32.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.002 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864018ca40 con 0x7f8640190720 2021-04-09T18:20:32.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.003 7f86377fe700 1 -- 172.21.15.32:0/3416638850 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8638005d10 con 0x7f8640190720 2021-04-09T18:20:32.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.003 7f86377fe700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f861c0188d0 0x7f861c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:32.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.003 7f86377fe700 1 -- 172.21.15.32:0/3416638850 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8638032e30 con 0x7f8640190720 2021-04-09T18:20:32.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.003 7f863e85e700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f861c0188d0 0x7f861c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:32.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.003 7f863e85e700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f861c0188d0 0x7f861c01add0 crc :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:32.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.023 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864010d480 con 0x7f8640190720 2021-04-09T18:20:32.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.026 7f86377fe700 1 -- 172.21.15.32:0/3416638850 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8638005820 con 0x7f8640190720 2021-04-09T18:20:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.218 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f864010d480 con 0x7f8640190720 2021-04-09T18:20:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.219 7f86377fe700 1 -- 172.21.15.32:0/3416638850 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f8638005a30 con 0x7f8640190720 2021-04-09T18:20:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:32.267 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:32.267 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:32.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f861c0188d0 msgr2=0x7f861c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:32.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f861c0188d0 0x7f861c01add0 crc :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8640190720 msgr2=0x7f86401918c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8640190720 0x7f86401918c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f863800b910 tx=0x7f863800bcb0).stop 2021-04-09T18:20:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 shutdown_connections 2021-04-09T18:20:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f861c0188d0 0x7f861c01add0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8640190720 0x7f86401918c0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86400a74b0 0x7f86401912a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 --2- 172.21.15.32:0/3416638850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86400a70b0 0x7f8640190100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:32.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 >> 172.21.15.32:0/3416638850 conn(0x7f864008fb70 msgr2=0x7f864019ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:32.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 shutdown_connections 2021-04-09T18:20:32.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:32.221 7f8644b4d700 1 -- 172.21.15.32:0/3416638850 wait complete. 2021-04-09T18:20:32.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:32.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:33.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.814 7fb8a29ae700 1 Processor -- start 2021-04-09T18:20:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb8a29ae700 1 -- start start 2021-04-09T18:20:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb8a29ae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c10d010 0x7fb89c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb8a29ae700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb89c0d1da0 0x7fb89c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb8a29ae700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c10d480 con 0x7fb89c0d82c0 2021-04-09T18:20:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb8a29ae700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89c10e780 con 0x7fb89c0d1da0 2021-04-09T18:20:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb8a29ae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c0d8d30 con 0x7fb89c10d010 2021-04-09T18:20:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb89b7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb89c0d1da0 0x7fb89c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:33.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb8a0fcb700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb89c0d82c0 0x7fb89c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57410/0 (socket says 172.21.15.32:57410) 2021-04-09T18:20:33.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb89b7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb89c0d1da0 0x7fb89c0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47820/0 (socket says 172.21.15.32:47820) 2021-04-09T18:20:33.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb8a0fcb700 1 -- 172.21.15.32:0/3746180210 learned_addr learned my addr 172.21.15.32:0/3746180210 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.815 7fb89bfff700 1 --2- 172.21.15.32:0/3746180210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c10d010 0x7fb89c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb89b7fe700 1 -- 172.21.15.32:0/3746180210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c10d010 msgr2=0x7fb89c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb89b7fe700 1 --2- 172.21.15.32:0/3746180210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c10d010 0x7fb89c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb89b7fe700 1 -- 172.21.15.32:0/3746180210 >> v1:172.21.15.32:6789/0 conn(0x7fb89c0d82c0 legacy=0x7fb89c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:33.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb89b7fe700 1 -- 172.21.15.32:0/3746180210 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c0d9220 con 0x7fb89c0d1da0 2021-04-09T18:20:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb89affd700 1 -- 172.21.15.32:0/3746180210 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720200987 0 0) 0x7fb894001480 con 0x7fb89c0d82c0 2021-04-09T18:20:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb89b7fe700 1 --2- 172.21.15.32:0/3746180210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c0d1da0 0x7fb89c0d7ca0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7fb890006650 tx=0x7fb890009a00).ready entity=mon.0 client_cookie=15c6553d48f37444 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb89affd700 1 -- 172.21.15.32:0/3746180210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb890009df0 con 0x7fb89c0d1da0 2021-04-09T18:20:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb89affd700 1 -- 172.21.15.32:0/3746180210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8900009b0 con 0x7fb89c0d1da0 2021-04-09T18:20:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb89affd700 1 -- 172.21.15.32:0/3746180210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb89000c2b0 con 0x7fb89c0d1da0 2021-04-09T18:20:33.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb8a29ae700 1 -- 172.21.15.32:0/3746180210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c0d1da0 msgr2=0x7fb89c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:33.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.816 7fb8a29ae700 1 --2- 172.21.15.32:0/3746180210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c0d1da0 0x7fb89c0d7ca0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7fb890006650 tx=0x7fb890009a00).stop 2021-04-09T18:20:33.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.817 7fb8a29ae700 1 -- 172.21.15.32:0/3746180210 shutdown_connections 2021-04-09T18:20:33.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.817 7fb8a29ae700 1 --2- 172.21.15.32:0/3746180210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c0d1da0 0x7fb89c0d7ca0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:33.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.817 7fb8a29ae700 1 --2- 172.21.15.32:0/3746180210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c10d010 0x7fb89c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:33.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.817 7fb8a29ae700 1 -- 172.21.15.32:0/3746180210 >> 172.21.15.32:0/3746180210 conn(0x7fb89c0ce5f0 msgr2=0x7fb89c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:33.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.817 7fb8a29ae700 1 -- 172.21.15.32:0/3746180210 shutdown_connections 2021-04-09T18:20:33.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.817 7fb8a29ae700 1 -- 172.21.15.32:0/3746180210 wait complete. 2021-04-09T18:20:33.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.837 7fb8a29ae700 1 Processor -- start 2021-04-09T18:20:33.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.837 7fb8a29ae700 1 -- start start 2021-04-09T18:20:33.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb8a29ae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c1aca20 0x7fb89c1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:33.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb8a29ae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c18bd40 0x7fb89c18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:33.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb8a29ae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb89c08fba0 0x7fb89c08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:33.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb8a29ae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c18c7e0 con 0x7fb89c1aca20 2021-04-09T18:20:33.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb8a29ae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c18ca30 con 0x7fb89c08fba0 2021-04-09T18:20:33.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb8a29ae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c0905f0 con 0x7fb89c18bd40 2021-04-09T18:20:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c1aca20 0x7fb89c1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c1aca20 0x7fb89c1ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47822/0 (socket says 172.21.15.32:47822) 2021-04-09T18:20:33.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89bfff700 1 -- 172.21.15.32:0/2815117571 learned_addr learned my addr 172.21.15.32:0/2815117571 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:33.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb8a0fcb700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb89c08fba0 0x7fb89c08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:33.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89b7fe700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c18bd40 0x7fb89c18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:33.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89bfff700 1 -- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c18bd40 msgr2=0x7fb89c18c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:33.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89bfff700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c18bd40 0x7fb89c18c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:33.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89bfff700 1 -- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb89c08fba0 msgr2=0x7fb89c08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:33.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89bfff700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb89c08fba0 0x7fb89c08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:33.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89bfff700 1 -- 172.21.15.32:0/2815117571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c0b4300 con 0x7fb89c1aca20 2021-04-09T18:20:33.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.838 7fb89bfff700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c1aca20 0x7fb89c1ace20 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7fb88c006f60 tx=0x7fb88c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:33.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.839 7fb898ff9700 1 -- 172.21.15.32:0/2815117571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb88c00bd60 con 0x7fb89c1aca20 2021-04-09T18:20:33.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.839 7fb898ff9700 1 -- 172.21.15.32:0/2815117571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c000c90 con 0x7fb89c1aca20 2021-04-09T18:20:33.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.839 7fb898ff9700 1 -- 172.21.15.32:0/2815117571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb88c002220 con 0x7fb89c1aca20 2021-04-09T18:20:33.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.839 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89c0b4300 con 0x7fb89c1aca20 2021-04-09T18:20:33.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.839 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89c18d230 con 0x7fb89c1aca20 2021-04-09T18:20:33.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.839 7fb898ff9700 1 -- 172.21.15.32:0/2815117571 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb88c0052e0 con 0x7fb89c1aca20 2021-04-09T18:20:33.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.839 7fb898ff9700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb87c0188d0 0x7fb87c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:33.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.839 7fb898ff9700 1 -- 172.21.15.32:0/2815117571 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb88c0329d0 con 0x7fb89c1aca20 2021-04-09T18:20:33.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.840 7fb89b7fe700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb87c0188d0 0x7fb87c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:33.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.840 7fb89b7fe700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb87c0188d0 0x7fb87c01add0 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:33.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.859 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89c182d70 con 0x7fb89c1aca20 2021-04-09T18:20:33.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:33.863 7fb898ff9700 1 -- 172.21.15.32:0/2815117571 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb88c000c90 con 0x7fb89c1aca20 2021-04-09T18:20:34.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.054 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb89c10d480 con 0x7fb89c1aca20 2021-04-09T18:20:34.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.054 7fb898ff9700 1 -- 172.21.15.32:0/2815117571 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fb88c006b00 con 0x7fb89c1aca20 2021-04-09T18:20:34.101 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:34.102 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:34.102 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:34.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.056 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb87c0188d0 msgr2=0x7fb87c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:34.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.056 7fb8a29ae700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb87c0188d0 0x7fb87c01add0 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:34.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.056 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c1aca20 msgr2=0x7fb89c1ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:34.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.057 7fb8a29ae700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c1aca20 0x7fb89c1ace20 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7fb88c006f60 tx=0x7fb88c00b970).stop 2021-04-09T18:20:34.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.057 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 shutdown_connections 2021-04-09T18:20:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.057 7fb8a29ae700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb87c0188d0 0x7fb87c01add0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:34.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.057 7fb8a29ae700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb89c08fba0 0x7fb89c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.057 7fb8a29ae700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb89c18bd40 0x7fb89c18c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.057 7fb8a29ae700 1 --2- 172.21.15.32:0/2815117571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb89c1aca20 0x7fb89c1ace20 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:34.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.057 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 >> 172.21.15.32:0/2815117571 conn(0x7fb89c19b460 msgr2=0x7fb89c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:34.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.057 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 shutdown_connections 2021-04-09T18:20:34.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:34.057 7fb8a29ae700 1 -- 172.21.15.32:0/2815117571 wait complete. 2021-04-09T18:20:34.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:34.517 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:35.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:35.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.637 7f1f8c9f4700 1 Processor -- start 2021-04-09T18:20:35.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.637 7f1f8c9f4700 1 -- start start 2021-04-09T18:20:35.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8c9f4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f880d7ca0 0x7f1f880d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:35.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8c9f4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8810d480 con 0x7f1f8810d010 2021-04-09T18:20:35.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8c9f4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f880d86c0 con 0x7f1f8810da00 2021-04-09T18:20:35.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8c9f4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f880d9220 con 0x7f1f880d7ca0 2021-04-09T18:20:35.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8661c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1f8810d010 0x7f1f880d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57420/0 (socket says 172.21.15.32:57420) 2021-04-09T18:20:35.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8661c700 1 -- 172.21.15.32:0/1812944734 learned_addr learned my addr 172.21.15.32:0/1812944734 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:35.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f85e1b700 1 --2- 172.21.15.32:0/1812944734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f880d7ca0 0x7f1f880d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:35.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8561a700 1 -- 172.21.15.32:0/1812944734 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847572891 0 0) 0x7f1f78000ce0 con 0x7f1f8810d010 2021-04-09T18:20:35.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8561a700 1 -- 172.21.15.32:0/1812944734 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f70002fe0 con 0x7f1f8810d010 2021-04-09T18:20:35.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8561a700 1 -- 172.21.15.32:0/1812944734 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552576455 0 0) 0x7f1f80000d30 con 0x7f1f8810da00 2021-04-09T18:20:35.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f8561a700 1 -- 172.21.15.32:0/1812944734 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f700042c0 con 0x7f1f8810da00 2021-04-09T18:20:35.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f85e1b700 1 -- 172.21.15.32:0/1812944734 >> v1:172.21.15.86:6789/0 conn(0x7f1f8810da00 legacy=0x7f1f8810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:35.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f85e1b700 1 -- 172.21.15.32:0/1812944734 >> v1:172.21.15.32:6789/0 conn(0x7f1f8810d010 legacy=0x7f1f880d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:35.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f85e1b700 1 -- 172.21.15.32:0/1812944734 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f880d09c0 con 0x7f1f880d7ca0 2021-04-09T18:20:35.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.638 7f1f85e1b700 1 --2- 172.21.15.32:0/1812944734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f880d7ca0 0x7f1f880d80a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f1f7c006650 tx=0x7f1f7c009a80).ready entity=mon.2 client_cookie=7f9653274e030b09 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:35.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8561a700 1 -- 172.21.15.32:0/1812944734 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f7c0009b0 con 0x7f1f880d7ca0 2021-04-09T18:20:35.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8561a700 1 -- 172.21.15.32:0/1812944734 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c00c0c0 con 0x7f1f880d7ca0 2021-04-09T18:20:35.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8561a700 1 -- 172.21.15.32:0/1812944734 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f7c00c510 con 0x7f1f880d7ca0 2021-04-09T18:20:35.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8c9f4700 1 -- 172.21.15.32:0/1812944734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f880d7ca0 msgr2=0x7f1f880d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:35.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8c9f4700 1 --2- 172.21.15.32:0/1812944734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f880d7ca0 0x7f1f880d80a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f1f7c006650 tx=0x7f1f7c009a80).stop 2021-04-09T18:20:35.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8c9f4700 1 -- 172.21.15.32:0/1812944734 shutdown_connections 2021-04-09T18:20:35.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8c9f4700 1 --2- 172.21.15.32:0/1812944734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f880d7ca0 0x7f1f880d80a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:35.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8c9f4700 1 -- 172.21.15.32:0/1812944734 >> 172.21.15.32:0/1812944734 conn(0x7f1f880ce5f0 msgr2=0x7f1f880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:35.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8c9f4700 1 -- 172.21.15.32:0/1812944734 shutdown_connections 2021-04-09T18:20:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.639 7f1f8c9f4700 1 -- 172.21.15.32:0/1812944734 wait complete. 2021-04-09T18:20:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f8c9f4700 1 Processor -- start 2021-04-09T18:20:35.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f8c9f4700 1 -- start start 2021-04-09T18:20:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f8c9f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f880a80d0 0x7f1f881a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f8c9f4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f881a4ce0 0x7f1f881a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f8c9f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f881a5d70 0x7f1f8808ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:35.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f8c9f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881a5250 con 0x7f1f881a5d70 2021-04-09T18:20:35.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f8c9f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881a61b0 con 0x7f1f880a80d0 2021-04-09T18:20:35.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f8c9f4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8818be80 con 0x7f1f881a4ce0 2021-04-09T18:20:35.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f86e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f881a5d70 0x7f1f8808ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:35.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f85e1b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f881a4ce0 0x7f1f881a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:35.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f8661c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f880a80d0 0x7f1f881a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:35.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f86e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f881a5d70 0x7f1f8808ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47840/0 (socket says 172.21.15.32:47840) 2021-04-09T18:20:35.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.660 7f1f86e1d700 1 -- 172.21.15.32:0/215292740 learned_addr learned my addr 172.21.15.32:0/215292740 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:35.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f85e1b700 1 -- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f880a80d0 msgr2=0x7f1f881a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:35.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f85e1b700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f880a80d0 0x7f1f881a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:35.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f85e1b700 1 -- 172.21.15.32:0/215292740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f881a5d70 msgr2=0x7f1f8808ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:35.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f85e1b700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f881a5d70 0x7f1f8808ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f85e1b700 1 -- 172.21.15.32:0/215292740 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8808b2b0 con 0x7f1f881a4ce0 2021-04-09T18:20:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f86e1d700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f881a5d70 0x7f1f8808ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f85e1b700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f881a4ce0 0x7f1f881a5750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1f7c006fc0 tx=0x7f1f7c00daa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f76ffd700 1 -- 172.21.15.32:0/215292740 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f7c006740 con 0x7f1f881a4ce0 2021-04-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f76ffd700 1 -- 172.21.15.32:0/215292740 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c000c90 con 0x7f1f881a4ce0 2021-04-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8808b2b0 con 0x7f1f881a4ce0 2021-04-09T18:20:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f76ffd700 1 -- 172.21.15.32:0/215292740 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f7c00b2b0 con 0x7f1f881a4ce0 2021-04-09T18:20:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.661 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8808b2b0 con 0x7f1f881a4ce0 2021-04-09T18:20:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.662 7f1f76ffd700 1 -- 172.21.15.32:0/215292740 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1f7c00b020 con 0x7f1f881a4ce0 2021-04-09T18:20:35.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.662 7f1f76ffd700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f6c0188d0 0x7f1f6c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.662 7f1f76ffd700 1 -- 172.21.15.32:0/215292740 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1f7c034d00 con 0x7f1f881a4ce0 2021-04-09T18:20:35.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.662 7f1f8661c700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f6c0188d0 0x7f1f6c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:35.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.662 7f1f8661c700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f6c0188d0 0x7f1f6c01add0 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:35.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.682 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f881b7050 con 0x7f1f881a4ce0 2021-04-09T18:20:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.685 7f1f76ffd700 1 -- 172.21.15.32:0/215292740 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1f7c006390 con 0x7f1f881a4ce0 2021-04-09T18:20:35.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.873 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f881b7050 con 0x7f1f881a4ce0 2021-04-09T18:20:35.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.874 7f1f76ffd700 1 -- 172.21.15.32:0/215292740 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f1f7c000c90 con 0x7f1f881a4ce0 2021-04-09T18:20:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:35.923 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:35.924 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f6c0188d0 msgr2=0x7f1f6c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f6c0188d0 0x7f1f6c01add0 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f881a4ce0 msgr2=0x7f1f881a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f881a4ce0 0x7f1f881a5750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1f7c006fc0 tx=0x7f1f7c00daa0).stop 2021-04-09T18:20:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 shutdown_connections 2021-04-09T18:20:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f6c0188d0 0x7f1f6c01add0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f881a5d70 0x7f1f8808ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f881a4ce0 0x7f1f881a5750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:35.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 --2- 172.21.15.32:0/215292740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f880a80d0 0x7f1f881a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:35.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 >> 172.21.15.32:0/215292740 conn(0x7f1f88190570 msgr2=0x7f1f8818ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 shutdown_connections 2021-04-09T18:20:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:35.879 7f1f8c9f4700 1 -- 172.21.15.32:0/215292740 wait complete. 2021-04-09T18:20:36.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:20:36.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:20:36.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:36.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:37.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.457 7f51446ac700 1 Processor -- start 2021-04-09T18:20:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.457 7f51446ac700 1 -- start start 2021-04-09T18:20:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51446ac700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c10da00 0x7f513c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:37.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51446ac700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c10d480 con 0x7f513c10d010 2021-04-09T18:20:37.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51446ac700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c0d1f70 con 0x7f513c0d7ca0 2021-04-09T18:20:37.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51446ac700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0d9220 con 0x7f513c10da00 2021-04-09T18:20:37.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51424c8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f513c10d010 0x7f513c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57434/0 (socket says 172.21.15.32:57434) 2021-04-09T18:20:37.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51424c8700 1 -- 172.21.15.32:0/3716783095 learned_addr learned my addr 172.21.15.32:0/3716783095 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:37.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f5142cc9700 1 --2- 172.21.15.32:0/3716783095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c10da00 0x7f513c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:37.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51414c6700 1 -- 172.21.15.32:0/3716783095 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748476500 0 0) 0x7f5138000ce0 con 0x7f513c10d010 2021-04-09T18:20:37.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51414c6700 1 -- 172.21.15.32:0/3716783095 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5128002fe0 con 0x7f513c10d010 2021-04-09T18:20:37.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51414c6700 1 -- 172.21.15.32:0/3716783095 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166467694 0 0) 0x7f512c000c60 con 0x7f513c0d7ca0 2021-04-09T18:20:37.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f51414c6700 1 -- 172.21.15.32:0/3716783095 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51280042c0 con 0x7f513c0d7ca0 2021-04-09T18:20:37.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f5142cc9700 1 -- 172.21.15.32:0/3716783095 >> v1:172.21.15.86:6789/0 conn(0x7f513c0d7ca0 legacy=0x7f513c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:37.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f5142cc9700 1 -- 172.21.15.32:0/3716783095 >> v1:172.21.15.32:6789/0 conn(0x7f513c10d010 legacy=0x7f513c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:37.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f5142cc9700 1 -- 172.21.15.32:0/3716783095 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c10e420 con 0x7f513c10da00 2021-04-09T18:20:37.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.458 7f5142cc9700 1 --2- 172.21.15.32:0/3716783095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c10da00 0x7f513c10de00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5134006e50 tx=0x7f5134009e70).ready entity=mon.2 client_cookie=105882fc849d64e8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:37.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.459 7f51414c6700 1 -- 172.21.15.32:0/3716783095 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5134006a20 con 0x7f513c10da00 2021-04-09T18:20:37.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.459 7f51414c6700 1 -- 172.21.15.32:0/3716783095 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5134006790 con 0x7f513c10da00 2021-04-09T18:20:37.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.459 7f51414c6700 1 -- 172.21.15.32:0/3716783095 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f513400b700 con 0x7f513c10da00 2021-04-09T18:20:37.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.459 7f51446ac700 1 -- 172.21.15.32:0/3716783095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c10da00 msgr2=0x7f513c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:37.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.459 7f51446ac700 1 --2- 172.21.15.32:0/3716783095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c10da00 0x7f513c10de00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5134006e50 tx=0x7f5134009e70).stop 2021-04-09T18:20:37.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.459 7f51446ac700 1 -- 172.21.15.32:0/3716783095 shutdown_connections 2021-04-09T18:20:37.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.459 7f51446ac700 1 --2- 172.21.15.32:0/3716783095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c10da00 0x7f513c10de00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:37.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.459 7f51446ac700 1 -- 172.21.15.32:0/3716783095 >> 172.21.15.32:0/3716783095 conn(0x7f513c0ce5f0 msgr2=0x7f513c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:37.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.460 7f51446ac700 1 -- 172.21.15.32:0/3716783095 shutdown_connections 2021-04-09T18:20:37.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.460 7f51446ac700 1 -- 172.21.15.32:0/3716783095 wait complete. 2021-04-09T18:20:37.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f51446ac700 1 Processor -- start 2021-04-09T18:20:37.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f51446ac700 1 -- start start 2021-04-09T18:20:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f51446ac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f513c179610 0x7f513c179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f51446ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f513c17a6e0 0x7f513c17aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f51446ac700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c17b100 0x7f513c08db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f51446ac700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c17d720 con 0x7f513c17a6e0 2021-04-09T18:20:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f51446ac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c185ba0 con 0x7f513c179610 2021-04-09T18:20:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f51446ac700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c08e1b0 con 0x7f513c17b100 2021-04-09T18:20:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f5141cc7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f513c17a6e0 0x7f513c17aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f5141cc7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f513c17a6e0 0x7f513c17aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47850/0 (socket says 172.21.15.32:47850) 2021-04-09T18:20:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f5141cc7700 1 -- 172.21.15.32:0/4159337893 learned_addr learned my addr 172.21.15.32:0/4159337893 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f51424c8700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f513c179610 0x7f513c179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.480 7f5142cc9700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c17b100 0x7f513c08db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f5141cc7700 1 -- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c17b100 msgr2=0x7f513c08db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f5141cc7700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c17b100 0x7f513c08db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f5141cc7700 1 -- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f513c179610 msgr2=0x7f513c179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f5141cc7700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f513c179610 0x7f513c179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f5141cc7700 1 -- 172.21.15.32:0/4159337893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c08e360 con 0x7f513c17a6e0 2021-04-09T18:20:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f5141cc7700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f513c17a6e0 0x7f513c17aae0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f512c006f10 tx=0x7f512c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f5132ffd700 1 -- 172.21.15.32:0/4159337893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f512c00bd80 con 0x7f513c17a6e0 2021-04-09T18:20:37.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f5132ffd700 1 -- 172.21.15.32:0/4159337893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c000e20 con 0x7f513c17a6e0 2021-04-09T18:20:37.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f51446ac700 1 -- 172.21.15.32:0/4159337893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c08e1b0 con 0x7f513c17a6e0 2021-04-09T18:20:37.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f5132ffd700 1 -- 172.21.15.32:0/4159337893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f512c0054f0 con 0x7f513c17a6e0 2021-04-09T18:20:37.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.481 7f51446ac700 1 -- 172.21.15.32:0/4159337893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c08e8f0 con 0x7f513c17a6e0 2021-04-09T18:20:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.482 7f5132ffd700 1 -- 172.21.15.32:0/4159337893 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f512c009fe0 con 0x7f513c17a6e0 2021-04-09T18:20:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.482 7f5132ffd700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5124018900 0x7f512401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.482 7f5132ffd700 1 -- 172.21.15.32:0/4159337893 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f512c032a50 con 0x7f513c17a6e0 2021-04-09T18:20:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.482 7f51424c8700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5124018900 0x7f512401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.482 7f51424c8700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5124018900 0x7f512401adc0 crc :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:37.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.502 7f51446ac700 1 -- 172.21.15.32:0/4159337893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c17a1b0 con 0x7f513c17a6e0 2021-04-09T18:20:37.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.505 7f5132ffd700 1 -- 172.21.15.32:0/4159337893 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f512c000e20 con 0x7f513c17a6e0 2021-04-09T18:20:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.696 7f51446ac700 1 -- 172.21.15.32:0/4159337893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f513c17a1b0 con 0x7f513c17a6e0 2021-04-09T18:20:37.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.696 7f5132ffd700 1 -- 172.21.15.32:0/4159337893 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f512c009fe0 con 0x7f513c17a6e0 2021-04-09T18:20:37.743 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:37.744 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:37.744 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:37.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.698 7f51446ac700 1 -- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5124018900 msgr2=0x7f512401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:37.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.698 7f51446ac700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5124018900 0x7f512401adc0 crc :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:37.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.698 7f51446ac700 1 -- 172.21.15.32:0/4159337893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f513c17a6e0 msgr2=0x7f513c17aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.698 7f51446ac700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f513c17a6e0 0x7f513c17aae0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f512c006f10 tx=0x7f512c00b990).stop 2021-04-09T18:20:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.699 7f51446ac700 1 -- 172.21.15.32:0/4159337893 shutdown_connections 2021-04-09T18:20:37.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.699 7f51446ac700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5124018900 0x7f512401adc0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.699 7f51446ac700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f513c17a6e0 0x7f513c17aae0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.699 7f51446ac700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f513c17b100 0x7f513c08db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:37.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.699 7f51446ac700 1 --2- 172.21.15.32:0/4159337893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f513c179610 0x7f513c179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.699 7f51446ac700 1 -- 172.21.15.32:0/4159337893 >> 172.21.15.32:0/4159337893 conn(0x7f513c190550 msgr2=0x7f513c1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.699 7f51446ac700 1 -- 172.21.15.32:0/4159337893 shutdown_connections 2021-04-09T18:20:37.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:37.699 7f51446ac700 1 -- 172.21.15.32:0/4159337893 wait complete. 2021-04-09T18:20:38.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:38.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:39.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:39.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.271 7fe855680700 1 Processor -- start 2021-04-09T18:20:39.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.271 7fe855680700 1 -- start start 2021-04-09T18:20:39.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe855680700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe85010d010 0x7fe8500d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:39.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe855680700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85010d480 con 0x7fe8500d1da0 2021-04-09T18:20:39.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe855680700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85010e1f0 con 0x7fe8500d8490 2021-04-09T18:20:39.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe855680700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8500d8d30 con 0x7fe85010d010 2021-04-09T18:20:39.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe84f2bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe85010d010 0x7fe8500d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:39.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe84eabe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe8500d1da0 0x7fe8500d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57450/0 (socket says 172.21.15.32:57450) 2021-04-09T18:20:39.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe84eabe700 1 -- 172.21.15.32:0/2871703242 learned_addr learned my addr 172.21.15.32:0/2871703242 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:39.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe84f2bf700 1 -- 172.21.15.32:0/2871703242 >> v1:172.21.15.86:6789/0 conn(0x7fe8500d8490 legacy=0x7fe85010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:39.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe84f2bf700 1 -- 172.21.15.32:0/2871703242 >> v1:172.21.15.32:6789/0 conn(0x7fe8500d1da0 legacy=0x7fe8500d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:39.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe84f2bf700 1 -- 172.21.15.32:0/2871703242 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8500d9220 con 0x7fe85010d010 2021-04-09T18:20:39.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe84e2bd700 1 -- 172.21.15.32:0/2871703242 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438262849 0 0) 0x7fe8440009b0 con 0x7fe8500d1da0 2021-04-09T18:20:39.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe84f2bf700 1 --2- 172.21.15.32:0/2871703242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe85010d010 0x7fe8500d1780 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7fe840006650 tx=0x7fe840009a00).ready entity=mon.0 client_cookie=86479bf8f585d642 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:39.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.272 7fe84e2bd700 1 -- 172.21.15.32:0/2871703242 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe840009df0 con 0x7fe85010d010 2021-04-09T18:20:39.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.273 7fe84e2bd700 1 -- 172.21.15.32:0/2871703242 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8400009b0 con 0x7fe85010d010 2021-04-09T18:20:39.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.273 7fe84e2bd700 1 -- 172.21.15.32:0/2871703242 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe84000c2b0 con 0x7fe85010d010 2021-04-09T18:20:39.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.273 7fe855680700 1 -- 172.21.15.32:0/2871703242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe85010d010 msgr2=0x7fe8500d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:39.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.273 7fe855680700 1 --2- 172.21.15.32:0/2871703242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe85010d010 0x7fe8500d1780 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7fe840006650 tx=0x7fe840009a00).stop 2021-04-09T18:20:39.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.273 7fe855680700 1 -- 172.21.15.32:0/2871703242 shutdown_connections 2021-04-09T18:20:39.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.273 7fe855680700 1 --2- 172.21.15.32:0/2871703242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe85010d010 0x7fe8500d1780 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:39.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.273 7fe855680700 1 -- 172.21.15.32:0/2871703242 >> 172.21.15.32:0/2871703242 conn(0x7fe8500ce5f0 msgr2=0x7fe8500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:39.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.273 7fe855680700 1 -- 172.21.15.32:0/2871703242 shutdown_connections 2021-04-09T18:20:39.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.273 7fe855680700 1 -- 172.21.15.32:0/2871703242 wait complete. 2021-04-09T18:20:39.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.293 7fe855680700 1 Processor -- start 2021-04-09T18:20:39.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe855680700 1 -- start start 2021-04-09T18:20:39.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe855680700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe850188d40 0x7fe85017e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe855680700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe850182410 0x7fe8501a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:39.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe855680700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe850182810 0x7fe8501813a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe855680700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8501a7000 con 0x7fe850188d40 2021-04-09T18:20:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe855680700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe850189140 con 0x7fe850182810 2021-04-09T18:20:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe855680700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85017e800 con 0x7fe850182410 2021-04-09T18:20:39.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe84f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe850188d40 0x7fe85017e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe84f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe850188d40 0x7fe85017e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47862/0 (socket says 172.21.15.32:47862) 2021-04-09T18:20:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe84f2bf700 1 -- 172.21.15.32:0/3954498724 learned_addr learned my addr 172.21.15.32:0/3954498724 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:39.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe84eabe700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe850182410 0x7fe8501a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:39.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.294 7fe84fac0700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe850182810 0x7fe8501813a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:39.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe84f2bf700 1 -- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe850182410 msgr2=0x7fe8501a69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:39.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe84f2bf700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe850182410 0x7fe8501a69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:39.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe84f2bf700 1 -- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe850182810 msgr2=0x7fe8501813a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:39.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe84f2bf700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe850182810 0x7fe8501813a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe84f2bf700 1 -- 172.21.15.32:0/3954498724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8501819c0 con 0x7fe850188d40 2021-04-09T18:20:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe84f2bf700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe850188d40 0x7fe85017e010 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7fe8400096a0 tx=0x7fe84000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe837fff700 1 -- 172.21.15.32:0/3954498724 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe840000c90 con 0x7fe850188d40 2021-04-09T18:20:39.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe837fff700 1 -- 172.21.15.32:0/3954498724 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840002030 con 0x7fe850188d40 2021-04-09T18:20:39.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe837fff700 1 -- 172.21.15.32:0/3954498724 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe840002550 con 0x7fe850188d40 2021-04-09T18:20:39.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe855680700 1 -- 172.21.15.32:0/3954498724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8501819c0 con 0x7fe850188d40 2021-04-09T18:20:39.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.295 7fe855680700 1 -- 172.21.15.32:0/3954498724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8501819c0 con 0x7fe850188d40 2021-04-09T18:20:39.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.296 7fe837fff700 1 -- 172.21.15.32:0/3954498724 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe840002210 con 0x7fe850188d40 2021-04-09T18:20:39.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.296 7fe837fff700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.296 7fe837fff700 1 -- 172.21.15.32:0/3954498724 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe840032a40 con 0x7fe850188d40 2021-04-09T18:20:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.296 7fe84eabe700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:39.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.296 7fe84eabe700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 crc :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:39.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.316 7fe855680700 1 -- 172.21.15.32:0/3954498724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe850181c60 con 0x7fe850188d40 2021-04-09T18:20:39.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.319 7fe837fff700 1 -- 172.21.15.32:0/3954498724 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe8400009c0 con 0x7fe850188d40 2021-04-09T18:20:39.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.511 7fe855680700 1 -- 172.21.15.32:0/3954498724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe850181c60 con 0x7fe850188d40 2021-04-09T18:20:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.512 7fe837fff700 1 -- 172.21.15.32:0/3954498724 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fe84000bd80 con 0x7fe850188d40 2021-04-09T18:20:39.558 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:39.559 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:39.559 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:39.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 -- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 msgr2=0x7fe83001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:39.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 crc :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:39.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 -- 172.21.15.32:0/3954498724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe850188d40 msgr2=0x7fe85017e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:39.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe850188d40 0x7fe85017e010 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7fe8400096a0 tx=0x7fe84000b990).stop 2021-04-09T18:20:39.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 -- 172.21.15.32:0/3954498724 shutdown_connections 2021-04-09T18:20:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe830018930 0x7fe83001ae10 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe850182810 0x7fe8501813a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe850182410 0x7fe8501a69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:39.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 --2- 172.21.15.32:0/3954498724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe850188d40 0x7fe85017e010 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 -- 172.21.15.32:0/3954498724 >> 172.21.15.32:0/3954498724 conn(0x7fe8501965a0 msgr2=0x7fe85019dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 -- 172.21.15.32:0/3954498724 shutdown_connections 2021-04-09T18:20:39.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:39.514 7fe855680700 1 -- 172.21.15.32:0/3954498724 wait complete. 2021-04-09T18:20:39.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:39.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:40.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:41.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.094 7f0fedd49700 1 Processor -- start 2021-04-09T18:20:41.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.094 7f0fedd49700 1 -- start start 2021-04-09T18:20:41.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.095 7f0fedd49700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0fe80d7ca0 0x7f0fe80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:41.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.095 7f0fedd49700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe810d480 con 0x7f0fe810d010 2021-04-09T18:20:41.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.095 7f0fedd49700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe80d86c0 con 0x7f0fe810da00 2021-04-09T18:20:41.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.095 7f0fedd49700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe80d9220 con 0x7f0fe80d7ca0 2021-04-09T18:20:41.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.095 7f0fe77fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0fe810d010 0x7f0fe80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57460/0 (socket says 172.21.15.32:57460) 2021-04-09T18:20:41.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.095 7f0fe77fe700 1 -- 172.21.15.32:0/1250236260 learned_addr learned my addr 172.21.15.32:0/1250236260 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:41.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.095 7f0fe6ffd700 1 --2- 172.21.15.32:0/1250236260 >> v2:172.21.15.86:3300/0 conn(0x7f0fe80d7ca0 0x7f0fe80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:41.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe67fc700 1 -- 172.21.15.32:0/1250236260 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531497908 0 0) 0x7f0fd8000ce0 con 0x7f0fe810d010 2021-04-09T18:20:41.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe67fc700 1 -- 172.21.15.32:0/1250236260 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd4002fe0 con 0x7f0fe810d010 2021-04-09T18:20:41.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe67fc700 1 -- 172.21.15.32:0/1250236260 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655191289 0 0) 0x7f0fe0001400 con 0x7f0fe810da00 2021-04-09T18:20:41.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe67fc700 1 -- 172.21.15.32:0/1250236260 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd4003210 con 0x7f0fe810da00 2021-04-09T18:20:41.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe67fc700 1 -- 172.21.15.32:0/1250236260 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3854957536 0 0) 0x7f0fd8000ed0 con 0x7f0fe810d010 2021-04-09T18:20:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe6ffd700 1 -- 172.21.15.32:0/1250236260 >> v1:172.21.15.86:6789/0 conn(0x7f0fe810da00 legacy=0x7f0fe810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe6ffd700 1 -- 172.21.15.32:0/1250236260 >> v1:172.21.15.32:6789/0 conn(0x7f0fe810d010 legacy=0x7f0fe80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:41.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe6ffd700 1 -- 172.21.15.32:0/1250236260 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe80d1f70 con 0x7f0fe80d7ca0 2021-04-09T18:20:41.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe6ffd700 1 --2- 172.21.15.32:0/1250236260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fe80d7ca0 0x7f0fe80d80a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f0fd0006650 tx=0x7f0fd0009a80).ready entity=mon.1 client_cookie=43787a82a7857d07 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:41.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe67fc700 1 -- 172.21.15.32:0/1250236260 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fd00009b0 con 0x7f0fe80d7ca0 2021-04-09T18:20:41.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe67fc700 1 -- 172.21.15.32:0/1250236260 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd000c0c0 con 0x7f0fe80d7ca0 2021-04-09T18:20:41.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fe67fc700 1 -- 172.21.15.32:0/1250236260 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fd000c510 con 0x7f0fe80d7ca0 2021-04-09T18:20:41.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fedd49700 1 -- 172.21.15.32:0/1250236260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fe80d7ca0 msgr2=0x7f0fe80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:41.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.096 7f0fedd49700 1 --2- 172.21.15.32:0/1250236260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fe80d7ca0 0x7f0fe80d80a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f0fd0006650 tx=0x7f0fd0009a80).stop 2021-04-09T18:20:41.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.097 7f0fedd49700 1 -- 172.21.15.32:0/1250236260 shutdown_connections 2021-04-09T18:20:41.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.097 7f0fedd49700 1 --2- 172.21.15.32:0/1250236260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fe80d7ca0 0x7f0fe80d80a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:41.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.097 7f0fedd49700 1 -- 172.21.15.32:0/1250236260 >> 172.21.15.32:0/1250236260 conn(0x7f0fe80ce5f0 msgr2=0x7f0fe80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:41.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.097 7f0fedd49700 1 -- 172.21.15.32:0/1250236260 shutdown_connections 2021-04-09T18:20:41.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.097 7f0fedd49700 1 -- 172.21.15.32:0/1250236260 wait complete. 2021-04-09T18:20:41.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.117 7f0fedd49700 1 Processor -- start 2021-04-09T18:20:41.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.117 7f0fedd49700 1 -- start start 2021-04-09T18:20:41.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fedd49700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fe81ac670 0x7f0fe81a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:41.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fedd49700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe81a5dc0 0x7f0fe80ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:41.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fedd49700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0fe81aca70 0x7f0fe80ba800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:41.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fedd49700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe81a5220 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fedd49700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe818e100 con 0x7f0fe81ac670 2021-04-09T18:20:41.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fedd49700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe819ccc0 con 0x7f0fe81aca70 2021-04-09T18:20:41.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe81a5dc0 0x7f0fe80ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:41.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe81a5dc0 0x7f0fe80ba1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47878/0 (socket says 172.21.15.32:47878) 2021-04-09T18:20:41.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe6ffd700 1 -- 172.21.15.32:0/599548924 learned_addr learned my addr 172.21.15.32:0/599548924 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:41.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe7fff700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0fe81aca70 0x7f0fe80ba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:41.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe77fe700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fe81ac670 0x7f0fe81a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:41.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe6ffd700 1 -- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0fe81aca70 msgr2=0x7f0fe80ba800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:41.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe6ffd700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0fe81aca70 0x7f0fe80ba800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:41.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe6ffd700 1 -- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fe81ac670 msgr2=0x7f0fe81a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:41.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe6ffd700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fe81ac670 0x7f0fe81a57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.118 7f0fe6ffd700 1 -- 172.21.15.32:0/599548924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe80bae20 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.119 7f0fe6ffd700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe81a5dc0 0x7f0fe80ba1e0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f0fd0009650 tx=0x7f0fd000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.119 7f0fdffff700 1 -- 172.21.15.32:0/599548924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fd00093a0 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.119 7f0fdffff700 1 -- 172.21.15.32:0/599548924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd000bd50 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.119 7f0fedd49700 1 -- 172.21.15.32:0/599548924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe80bae20 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.119 7f0fdffff700 1 -- 172.21.15.32:0/599548924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fd00052e0 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.119 7f0fedd49700 1 -- 172.21.15.32:0/599548924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe80bae20 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.119 7f0fdffff700 1 -- 172.21.15.32:0/599548924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0fd0006030 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.119 7f0fdffff700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fcc018930 0x7f0fcc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:41.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.119 7f0fdffff700 1 -- 172.21.15.32:0/599548924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0fd00329e0 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.120 7f0fe77fe700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fcc018930 0x7f0fcc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:41.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.120 7f0fe77fe700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fcc018930 0x7f0fcc01adc0 crc :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:41.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.139 7f0fedd49700 1 -- 172.21.15.32:0/599548924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe8182d10 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.143 7f0fdffff700 1 -- 172.21.15.32:0/599548924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0fd00009c0 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.334 7f0fedd49700 1 -- 172.21.15.32:0/599548924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fe8182d10 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.335 7f0fdffff700 1 -- 172.21.15.32:0/599548924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f0fd000bd50 con 0x7f0fe81a5dc0 2021-04-09T18:20:41.384 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:41.384 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:41.385 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 -- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fcc018930 msgr2=0x7f0fcc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fcc018930 0x7f0fcc01adc0 crc :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 -- 172.21.15.32:0/599548924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe81a5dc0 msgr2=0x7f0fe80ba1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe81a5dc0 0x7f0fe80ba1e0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f0fd0009650 tx=0x7f0fd000b960).stop 2021-04-09T18:20:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 -- 172.21.15.32:0/599548924 shutdown_connections 2021-04-09T18:20:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fcc018930 0x7f0fcc01adc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:41.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe81a5dc0 0x7f0fe80ba1e0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:41.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0fe81aca70 0x7f0fe80ba800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 --2- 172.21.15.32:0/599548924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fe81ac670 0x7f0fe81a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 -- 172.21.15.32:0/599548924 >> 172.21.15.32:0/599548924 conn(0x7f0fe819b890 msgr2=0x7f0fe81a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:41.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 -- 172.21.15.32:0/599548924 shutdown_connections 2021-04-09T18:20:41.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:41.338 7f0fedd49700 1 -- 172.21.15.32:0/599548924 wait complete. 2021-04-09T18:20:41.805 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:41.806 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:42.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:43.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.924 7f255bbc2700 1 Processor -- start 2021-04-09T18:20:43.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.925 7f255bbc2700 1 -- start start 2021-04-09T18:20:43.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.925 7f255bbc2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25540d7ca0 0x7f25540d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:43.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.925 7f255bbc2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f255410da00 0x7f255410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.925 7f255bbc2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f255410d480 con 0x7f255410d010 2021-04-09T18:20:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.925 7f255bbc2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25540d9220 con 0x7f255410da00 2021-04-09T18:20:43.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.925 7f255bbc2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2554112100 con 0x7f25540d7ca0 2021-04-09T18:20:43.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f25591dd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25540d7ca0 0x7f25540d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:43.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f255a1df700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f255410da00 0x7f255410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:43.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f255a1df700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f255410da00 0x7f255410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:39644/0 (socket says 172.21.15.32:39644) 2021-04-09T18:20:43.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f25591dd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25540d7ca0 0x7f25540d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:55114/0 (socket says 172.21.15.32:55114) 2021-04-09T18:20:43.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f255a1df700 1 -- 172.21.15.32:0/2848134085 learned_addr learned my addr 172.21.15.32:0/2848134085 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:43.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f25599de700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f255410d010 0x7f25540d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55400/0 (socket says 172.21.15.32:55400) 2021-04-09T18:20:43.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f255a1df700 1 -- 172.21.15.32:0/2848134085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25540d7ca0 msgr2=0x7f25540d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:43.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f255a1df700 1 --2- 172.21.15.32:0/2848134085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25540d7ca0 0x7f25540d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f255a1df700 1 -- 172.21.15.32:0/2848134085 >> v1:172.21.15.86:6789/0 conn(0x7f255410d010 legacy=0x7f25540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:43.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f255a1df700 1 -- 172.21.15.32:0/2848134085 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25540d86c0 con 0x7f255410da00 2021-04-09T18:20:43.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f25589dc700 1 -- 172.21.15.32:0/2848134085 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738869615 0 0) 0x7f254c000fe0 con 0x7f255410d010 2021-04-09T18:20:43.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f25591dd700 1 --2- 172.21.15.32:0/2848134085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25540d7ca0 0x7f25540d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:43.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f255a1df700 1 --2- 172.21.15.32:0/2848134085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f255410da00 0x7f255410de00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f2544006720 tx=0x7f2544009ba0).ready entity=mon.1 client_cookie=718bd3195f72ce96 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:43.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.926 7f25589dc700 1 -- 172.21.15.32:0/2848134085 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25440022e0 con 0x7f255410da00 2021-04-09T18:20:43.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.927 7f25589dc700 1 -- 172.21.15.32:0/2848134085 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2544002660 con 0x7f255410da00 2021-04-09T18:20:43.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.927 7f25589dc700 1 -- 172.21.15.32:0/2848134085 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2544002b40 con 0x7f255410da00 2021-04-09T18:20:43.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.927 7f255bbc2700 1 -- 172.21.15.32:0/2848134085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f255410da00 msgr2=0x7f255410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:43.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.927 7f255bbc2700 1 --2- 172.21.15.32:0/2848134085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f255410da00 0x7f255410de00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f2544006720 tx=0x7f2544009ba0).stop 2021-04-09T18:20:43.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.927 7f255bbc2700 1 -- 172.21.15.32:0/2848134085 shutdown_connections 2021-04-09T18:20:43.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.927 7f255bbc2700 1 --2- 172.21.15.32:0/2848134085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f255410da00 0x7f255410de00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.927 7f255bbc2700 1 --2- 172.21.15.32:0/2848134085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25540d7ca0 0x7f25540d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.927 7f255bbc2700 1 -- 172.21.15.32:0/2848134085 >> 172.21.15.32:0/2848134085 conn(0x7f25540ce5f0 msgr2=0x7f25540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.928 7f255bbc2700 1 -- 172.21.15.32:0/2848134085 shutdown_connections 2021-04-09T18:20:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.928 7f255bbc2700 1 -- 172.21.15.32:0/2848134085 wait complete. 2021-04-09T18:20:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.948 7f255bbc2700 1 Processor -- start 2021-04-09T18:20:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.948 7f255bbc2700 1 -- start start 2021-04-09T18:20:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.948 7f255bbc2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2554184a30 0x7f25541acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.948 7f255bbc2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25541a6b90 0x7f25541a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:43.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.948 7f255bbc2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25541a5150 0x7f25541a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:43.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.948 7f255bbc2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25541a75b0 con 0x7f25541a5150 2021-04-09T18:20:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.948 7f255bbc2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2554184590 con 0x7f25541a6b90 2021-04-09T18:20:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.948 7f255bbc2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25541a5b70 con 0x7f2554184a30 2021-04-09T18:20:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.948 7f255a1df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25541a5150 0x7f25541a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255a1df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25541a5150 0x7f25541a5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47892/0 (socket says 172.21.15.32:47892) 2021-04-09T18:20:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255a1df700 1 -- 172.21.15.32:0/1421298952 learned_addr learned my addr 172.21.15.32:0/1421298952 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:43.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f25599de700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2554184a30 0x7f25541acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:43.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f25591dd700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25541a6b90 0x7f25541a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:43.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255a1df700 1 -- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2554184a30 msgr2=0x7f25541acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:43.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255a1df700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2554184a30 0x7f25541acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255a1df700 1 -- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25541a6b90 msgr2=0x7f25541a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:43.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255a1df700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25541a6b90 0x7f25541a6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255a1df700 1 -- 172.21.15.32:0/1421298952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25541a5d90 con 0x7f25541a5150 2021-04-09T18:20:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255a1df700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25541a5150 0x7f25541a5550 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7f254400b560 tx=0x7f254400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f25527fc700 1 -- 172.21.15.32:0/1421298952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2544000a70 con 0x7f25541a5150 2021-04-09T18:20:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f25527fc700 1 -- 172.21.15.32:0/1421298952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2544000db0 con 0x7f25541a5150 2021-04-09T18:20:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25541a5d90 con 0x7f25541a5150 2021-04-09T18:20:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f25527fc700 1 -- 172.21.15.32:0/1421298952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25440022b0 con 0x7f25541a5150 2021-04-09T18:20:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.949 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25541a5d90 con 0x7f25541a5150 2021-04-09T18:20:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.950 7f25527fc700 1 -- 172.21.15.32:0/1421298952 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f254400bcf0 con 0x7f25541a5150 2021-04-09T18:20:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.950 7f25527fc700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2534018930 0x7f253401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.950 7f25527fc700 1 -- 172.21.15.32:0/1421298952 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2544034bf0 con 0x7f25541a5150 2021-04-09T18:20:43.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.950 7f25599de700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2534018930 0x7f253401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:43.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.951 7f25599de700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2534018930 0x7f253401ae10 crc :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:43.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.969 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25541b7080 con 0x7f25541a5150 2021-04-09T18:20:43.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:42.973 7f25527fc700 1 -- 172.21.15.32:0/1421298952 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2544034e00 con 0x7f25541a5150 2021-04-09T18:20:43.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.165 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f255410d480 con 0x7f25541a5150 2021-04-09T18:20:43.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.165 7f25527fc700 1 -- 172.21.15.32:0/1421298952 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f254400bcf0 con 0x7f25541a5150 2021-04-09T18:20:43.213 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:43.214 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:43.215 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2534018930 msgr2=0x7f253401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:43.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2534018930 0x7f253401ae10 crc :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25541a5150 msgr2=0x7f25541a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25541a5150 0x7f25541a5550 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7f254400b560 tx=0x7f254400b900).stop 2021-04-09T18:20:43.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 shutdown_connections 2021-04-09T18:20:43.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2534018930 0x7f253401ae10 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25541a5150 0x7f25541a5550 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25541a6b90 0x7f25541a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 --2- 172.21.15.32:0/1421298952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2554184a30 0x7f25541acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:43.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 >> 172.21.15.32:0/1421298952 conn(0x7f2554191560 msgr2=0x7f25541b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 shutdown_connections 2021-04-09T18:20:43.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:43.168 7f255bbc2700 1 -- 172.21.15.32:0/1421298952 wait complete. 2021-04-09T18:20:43.641 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:43.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:44.644 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:44.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.752 7f47176d9700 1 Processor -- start 2021-04-09T18:20:44.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.752 7f47176d9700 1 -- start start 2021-04-09T18:20:44.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f47176d9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:44.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f47176d9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47100d1da0 0x7f47100d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:44.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f47176d9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471010d480 con 0x7f47100d82c0 2021-04-09T18:20:44.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f47176d9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471010e780 con 0x7f47100d1da0 2021-04-09T18:20:44.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f47176d9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100d8d30 con 0x7f471010d010 2021-04-09T18:20:44.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f4714cf4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47100d1da0 0x7f47100d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:44.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f47154f5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:44.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f4714cf4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47100d1da0 0x7f47100d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:39660/0 (socket says 172.21.15.32:39660) 2021-04-09T18:20:44.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f47154f5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:55124/0 (socket says 172.21.15.32:55124) 2021-04-09T18:20:44.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f4714cf4700 1 -- 172.21.15.32:0/2815133202 learned_addr learned my addr 172.21.15.32:0/2815133202 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:44.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.753 7f4715cf6700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f47100d82c0 0x7f471010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55416/0 (socket says 172.21.15.32:55416) 2021-04-09T18:20:44.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f4714cf4700 1 -- 172.21.15.32:0/2815133202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 msgr2=0x7f47100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:44.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f4714cf4700 1 --2- 172.21.15.32:0/2815133202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:44.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f4714cf4700 1 -- 172.21.15.32:0/2815133202 >> v1:172.21.15.86:6789/0 conn(0x7f47100d82c0 legacy=0x7f471010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:44.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f4714cf4700 1 -- 172.21.15.32:0/2815133202 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100d9220 con 0x7f47100d1da0 2021-04-09T18:20:44.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f4707fff700 1 -- 172.21.15.32:0/2815133202 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981254554 0 0) 0x7f4708000d30 con 0x7f47100d82c0 2021-04-09T18:20:44.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f47154f5700 1 --2- 172.21.15.32:0/2815133202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:44.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f4714cf4700 1 --2- 172.21.15.32:0/2815133202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47100d1da0 0x7f47100d7ca0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f4700006700 tx=0x7f4700009b80).ready entity=mon.1 client_cookie=2c196bcdc184c77 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:44.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f4707fff700 1 -- 172.21.15.32:0/2815133202 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470000c0c0 con 0x7f47100d1da0 2021-04-09T18:20:44.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f4707fff700 1 -- 172.21.15.32:0/2815133202 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470000c440 con 0x7f47100d1da0 2021-04-09T18:20:44.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f4707fff700 1 -- 172.21.15.32:0/2815133202 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470000c920 con 0x7f47100d1da0 2021-04-09T18:20:44.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f47176d9700 1 -- 172.21.15.32:0/2815133202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47100d1da0 msgr2=0x7f47100d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:44.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.754 7f47176d9700 1 --2- 172.21.15.32:0/2815133202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47100d1da0 0x7f47100d7ca0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f4700006700 tx=0x7f4700009b80).stop 2021-04-09T18:20:44.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.755 7f47176d9700 1 -- 172.21.15.32:0/2815133202 shutdown_connections 2021-04-09T18:20:44.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.755 7f47176d9700 1 --2- 172.21.15.32:0/2815133202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47100d1da0 0x7f47100d7ca0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:44.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.755 7f47176d9700 1 --2- 172.21.15.32:0/2815133202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:44.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.755 7f47176d9700 1 -- 172.21.15.32:0/2815133202 >> 172.21.15.32:0/2815133202 conn(0x7f47100ce5f0 msgr2=0x7f47100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:44.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.755 7f47176d9700 1 -- 172.21.15.32:0/2815133202 shutdown_connections 2021-04-09T18:20:44.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.755 7f47176d9700 1 -- 172.21.15.32:0/2815133202 wait complete. 2021-04-09T18:20:44.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.775 7f47176d9700 1 Processor -- start 2021-04-09T18:20:44.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.775 7f47176d9700 1 -- start start 2021-04-09T18:20:44.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47176d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47101a5210 0x7f47101a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:44.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47176d9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a6340 0x7f47101a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:44.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47176d9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47100abaa0 0x7f47100abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:44.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47176d9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47101a6d60 con 0x7f47101a5210 2021-04-09T18:20:44.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47176d9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47100ac4c0 con 0x7f47101a6340 2021-04-09T18:20:44.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47176d9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100ac670 con 0x7f47100abaa0 2021-04-09T18:20:44.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47154f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47101a5210 0x7f47101a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:44.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47154f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47101a5210 0x7f47101a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47906/0 (socket says 172.21.15.32:47906) 2021-04-09T18:20:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47154f5700 1 -- 172.21.15.32:0/3308565881 learned_addr learned my addr 172.21.15.32:0/3308565881 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f4714cf4700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a6340 0x7f47101a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f4715cf6700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47100abaa0 0x7f47100abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47154f5700 1 -- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47100abaa0 msgr2=0x7f47100abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:44.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47154f5700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47100abaa0 0x7f47100abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:44.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47154f5700 1 -- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a6340 msgr2=0x7f47101a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:44.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47154f5700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a6340 0x7f47101a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:44.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47154f5700 1 -- 172.21.15.32:0/3308565881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100ac890 con 0x7f47101a5210 2021-04-09T18:20:44.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.776 7f47154f5700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47101a5210 0x7f47101a56c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f470c006f40 tx=0x7f470c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:44.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.777 7f4705ffb700 1 -- 172.21.15.32:0/3308565881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470c00bd80 con 0x7f47101a5210 2021-04-09T18:20:44.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.777 7f4705ffb700 1 -- 172.21.15.32:0/3308565881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c000c90 con 0x7f47101a5210 2021-04-09T18:20:44.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.777 7f47176d9700 1 -- 172.21.15.32:0/3308565881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47100ac890 con 0x7f47101a5210 2021-04-09T18:20:44.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.777 7f4705ffb700 1 -- 172.21.15.32:0/3308565881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470c002220 con 0x7f47101a5210 2021-04-09T18:20:44.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.777 7f47176d9700 1 -- 172.21.15.32:0/3308565881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47100ac890 con 0x7f47101a5210 2021-04-09T18:20:44.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.778 7f4705ffb700 1 -- 172.21.15.32:0/3308565881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f470c00a000 con 0x7f47101a5210 2021-04-09T18:20:44.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.778 7f4705ffb700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f00188d0 0x7f46f001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:44.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.778 7f4705ffb700 1 -- 172.21.15.32:0/3308565881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f470c0329e0 con 0x7f47101a5210 2021-04-09T18:20:44.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.778 7f4714cf4700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f00188d0 0x7f46f001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:44.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.778 7f4714cf4700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f00188d0 0x7f46f001add0 crc :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:44.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.797 7f47176d9700 1 -- 172.21.15.32:0/3308565881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47101a6d60 con 0x7f47101a5210 2021-04-09T18:20:44.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.801 7f4705ffb700 1 -- 172.21.15.32:0/3308565881 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f470c000c90 con 0x7f47101a5210 2021-04-09T18:20:45.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.993 7f47176d9700 1 -- 172.21.15.32:0/3308565881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47101a6d60 con 0x7f47101a5210 2021-04-09T18:20:45.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.994 7f4705ffb700 1 -- 172.21.15.32:0/3308565881 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f470c006b00 con 0x7f47101a5210 2021-04-09T18:20:45.041 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:45.042 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:45.043 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:45.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 -- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f00188d0 msgr2=0x7f46f001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:45.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f00188d0 0x7f46f001add0 crc :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:45.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 -- 172.21.15.32:0/3308565881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47101a5210 msgr2=0x7f47101a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47101a5210 0x7f47101a56c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f470c006f40 tx=0x7f470c00b990).stop 2021-04-09T18:20:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 -- 172.21.15.32:0/3308565881 shutdown_connections 2021-04-09T18:20:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f00188d0 0x7f46f001add0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47100abaa0 0x7f47100abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a6340 0x7f47101a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 --2- 172.21.15.32:0/3308565881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47101a5210 0x7f47101a56c0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 -- 172.21.15.32:0/3308565881 >> 172.21.15.32:0/3308565881 conn(0x7f47101a99c0 msgr2=0x7f471017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 -- 172.21.15.32:0/3308565881 shutdown_connections 2021-04-09T18:20:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:44.996 7f47176d9700 1 -- 172.21.15.32:0/3308565881 wait complete. 2021-04-09T18:20:45.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:45.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:46.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:46.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.576 7f6a3dc32700 1 Processor -- start 2021-04-09T18:20:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.576 7f6a3dc32700 1 -- start start 2021-04-09T18:20:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a3dc32700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a3810d010 0x7f6a380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:46.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a3dc32700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380d1da0 0x7f6a380d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:46.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a3dc32700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a380d82c0 0x7f6a3810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:46.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a3dc32700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a380d8d30 con 0x7f6a3810d010 2021-04-09T18:20:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a3dc32700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a380d9220 con 0x7f6a380d82c0 2021-04-09T18:20:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a3dc32700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3810d480 con 0x7f6a380d1da0 2021-04-09T18:20:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a377fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a3810d010 0x7f6a380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:46.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a377fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a3810d010 0x7f6a380d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47912/0 (socket says 172.21.15.32:47912) 2021-04-09T18:20:46.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a377fe700 1 -- 172.21.15.32:0/3894453969 learned_addr learned my addr 172.21.15.32:0/3894453969 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:46.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a36ffd700 1 --2- 172.21.15.32:0/3894453969 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380d1da0 0x7f6a380d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:46.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a37fff700 1 --2- 172.21.15.32:0/3894453969 >> v2:172.21.15.86:3300/0 conn(0x7f6a380d82c0 0x7f6a3810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:46.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a377fe700 1 -- 172.21.15.32:0/3894453969 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380d1da0 msgr2=0x7f6a380d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a377fe700 1 --2- 172.21.15.32:0/3894453969 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380d1da0 0x7f6a380d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a377fe700 1 -- 172.21.15.32:0/3894453969 >> v2:172.21.15.86:3300/0 conn(0x7f6a380d82c0 msgr2=0x7f6a3810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:46.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a377fe700 1 --2- 172.21.15.32:0/3894453969 >> v2:172.21.15.86:3300/0 conn(0x7f6a380d82c0 0x7f6a3810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a377fe700 1 -- 172.21.15.32:0/3894453969 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a380d86c0 con 0x7f6a3810d010 2021-04-09T18:20:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a36ffd700 1 --2- 172.21.15.32:0/3894453969 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380d1da0 0x7f6a380d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:20:46.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a377fe700 1 --2- 172.21.15.32:0/3894453969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a3810d010 0x7f6a380d1780 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f6a30006a80 tx=0x7f6a30009f00).ready entity=mon.0 client_cookie=4bbf9c30f2518d76 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:46.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a367fc700 1 -- 172.21.15.32:0/3894453969 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a3000b5c0 con 0x7f6a3810d010 2021-04-09T18:20:46.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a367fc700 1 -- 172.21.15.32:0/3894453969 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3000b8c0 con 0x7f6a3810d010 2021-04-09T18:20:46.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a367fc700 1 -- 172.21.15.32:0/3894453969 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a3000bda0 con 0x7f6a3810d010 2021-04-09T18:20:46.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a3dc32700 1 -- 172.21.15.32:0/3894453969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a3810d010 msgr2=0x7f6a380d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:46.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.577 7f6a3dc32700 1 --2- 172.21.15.32:0/3894453969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a3810d010 0x7f6a380d1780 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f6a30006a80 tx=0x7f6a30009f00).stop 2021-04-09T18:20:46.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.578 7f6a3dc32700 1 -- 172.21.15.32:0/3894453969 shutdown_connections 2021-04-09T18:20:46.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.578 7f6a3dc32700 1 --2- 172.21.15.32:0/3894453969 >> v2:172.21.15.86:3300/0 conn(0x7f6a380d82c0 0x7f6a3810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.578 7f6a3dc32700 1 --2- 172.21.15.32:0/3894453969 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380d1da0 0x7f6a380d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.578 7f6a3dc32700 1 --2- 172.21.15.32:0/3894453969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a3810d010 0x7f6a380d1780 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.578 7f6a3dc32700 1 -- 172.21.15.32:0/3894453969 >> 172.21.15.32:0/3894453969 conn(0x7f6a380ce5f0 msgr2=0x7f6a380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:46.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.578 7f6a3dc32700 1 -- 172.21.15.32:0/3894453969 shutdown_connections 2021-04-09T18:20:46.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.578 7f6a3dc32700 1 -- 172.21.15.32:0/3894453969 wait complete. 2021-04-09T18:20:46.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.598 7f6a3dc32700 1 Processor -- start 2021-04-09T18:20:46.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a3dc32700 1 -- start start 2021-04-09T18:20:46.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a3dc32700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380b42d0 0x7f6a38183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:46.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a3dc32700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a381ac090 0x7f6a381ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:46.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a3dc32700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a38184d30 0x7f6a38185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:46.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a3dc32700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380a1a90 con 0x7f6a38184d30 2021-04-09T18:20:46.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a3dc32700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a381acab0 con 0x7f6a381ac090 2021-04-09T18:20:46.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a3dc32700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38185a00 con 0x7f6a380b42d0 2021-04-09T18:20:46.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a38184d30 0x7f6a38185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:46.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a37fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a38184d30 0x7f6a38185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47922/0 (socket says 172.21.15.32:47922) 2021-04-09T18:20:46.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a37fff700 1 -- 172.21.15.32:0/996255722 learned_addr learned my addr 172.21.15.32:0/996255722 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:46.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a36ffd700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a381ac090 0x7f6a381ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:46.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.599 7f6a377fe700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380b42d0 0x7f6a38183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:46.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a37fff700 1 -- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380b42d0 msgr2=0x7f6a38183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:46.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a37fff700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380b42d0 0x7f6a38183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a37fff700 1 -- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a381ac090 msgr2=0x7f6a381ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:46.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a37fff700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a381ac090 0x7f6a381ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a37fff700 1 -- 172.21.15.32:0/996255722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a38185750 con 0x7f6a38184d30 2021-04-09T18:20:46.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a37fff700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a38184d30 0x7f6a38185130 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f6a28006f10 tx=0x7f6a2800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:46.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a2ffff700 1 -- 172.21.15.32:0/996255722 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a28000a70 con 0x7f6a38184d30 2021-04-09T18:20:46.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a2ffff700 1 -- 172.21.15.32:0/996255722 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2800bd80 con 0x7f6a38184d30 2021-04-09T18:20:46.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a38185750 con 0x7f6a38184d30 2021-04-09T18:20:46.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a2ffff700 1 -- 172.21.15.32:0/996255722 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a28002220 con 0x7f6a38184d30 2021-04-09T18:20:46.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.600 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a38185750 con 0x7f6a38184d30 2021-04-09T18:20:46.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.601 7f6a2ffff700 1 -- 172.21.15.32:0/996255722 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6a28006470 con 0x7f6a38184d30 2021-04-09T18:20:46.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.601 7f6a2ffff700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a1c018930 0x7f6a1c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:46.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.601 7f6a2ffff700 1 -- 172.21.15.32:0/996255722 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6a28032a10 con 0x7f6a38184d30 2021-04-09T18:20:46.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.601 7f6a377fe700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a1c018930 0x7f6a1c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:46.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.601 7f6a377fe700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a1c018930 0x7f6a1c01ae10 crc :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:46.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.621 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a38182d10 con 0x7f6a38184d30 2021-04-09T18:20:46.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.624 7f6a2ffff700 1 -- 172.21.15.32:0/996255722 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6a28006ba0 con 0x7f6a38184d30 2021-04-09T18:20:46.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.814 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a3810d480 con 0x7f6a38184d30 2021-04-09T18:20:46.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.815 7f6a2ffff700 1 -- 172.21.15.32:0/996255722 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f6a2800bd80 con 0x7f6a38184d30 2021-04-09T18:20:46.862 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:46.863 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:46.864 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:46.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a1c018930 msgr2=0x7f6a1c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:46.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a1c018930 0x7f6a1c01ae10 crc :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a38184d30 msgr2=0x7f6a38185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:46.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a38184d30 0x7f6a38185130 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f6a28006f10 tx=0x7f6a2800b990).stop 2021-04-09T18:20:46.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 shutdown_connections 2021-04-09T18:20:46.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a1c018930 0x7f6a1c01ae10 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a38184d30 0x7f6a38185130 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a381ac090 0x7f6a381ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 --2- 172.21.15.32:0/996255722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a380b42d0 0x7f6a38183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:46.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 >> 172.21.15.32:0/996255722 conn(0x7f6a3819b890 msgr2=0x7f6a381a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:46.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 shutdown_connections 2021-04-09T18:20:46.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:46.817 7f6a3dc32700 1 -- 172.21.15.32:0/996255722 wait complete. 2021-04-09T18:20:47.286 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:47.286 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:48.289 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:48.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.403 7f35ccb02700 1 Processor -- start 2021-04-09T18:20:48.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.403 7f35ccb02700 1 -- start start 2021-04-09T18:20:48.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35ccb02700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c810d010 0x7f35c80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:48.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35ccb02700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f35c80d8490 0x7f35c810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35ccb02700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c810d480 con 0x7f35c80d1da0 2021-04-09T18:20:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35ccb02700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c810e780 con 0x7f35c80d8490 2021-04-09T18:20:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35ccb02700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80d8d30 con 0x7f35c810d010 2021-04-09T18:20:48.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c705f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f35c80d8490 0x7f35c810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c605d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f35c80d1da0 0x7f35c80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57520/0 (socket says 172.21.15.32:57520) 2021-04-09T18:20:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c605d700 1 -- 172.21.15.32:0/1088523630 learned_addr learned my addr 172.21.15.32:0/1088523630 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c685e700 1 --2- 172.21.15.32:0/1088523630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c810d010 0x7f35c80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:48.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c585c700 1 -- 172.21.15.32:0/1088523630 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056699605 0 0) 0x7f35bc000ce0 con 0x7f35c80d1da0 2021-04-09T18:20:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c585c700 1 -- 172.21.15.32:0/1088523630 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b0002fe0 con 0x7f35c80d1da0 2021-04-09T18:20:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c705f700 1 -- 172.21.15.32:0/1088523630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c810d010 msgr2=0x7f35c80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c705f700 1 --2- 172.21.15.32:0/1088523630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c810d010 0x7f35c80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c705f700 1 -- 172.21.15.32:0/1088523630 >> v1:172.21.15.32:6789/0 conn(0x7f35c80d1da0 legacy=0x7f35c80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:48.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.404 7f35c705f700 1 -- 172.21.15.32:0/1088523630 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80d9220 con 0x7f35c80d8490 2021-04-09T18:20:48.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.405 7f35c705f700 1 --2- 172.21.15.32:0/1088523630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c80d8490 0x7f35c810da00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f35c0006e30 tx=0x7f35c0009e60).ready entity=mon.0 client_cookie=82a5a90989650acd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:48.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.405 7f35c585c700 1 -- 172.21.15.32:0/1088523630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35c0006a50 con 0x7f35c80d8490 2021-04-09T18:20:48.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.405 7f35c585c700 1 -- 172.21.15.32:0/1088523630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c000b300 con 0x7f35c80d8490 2021-04-09T18:20:48.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.405 7f35c585c700 1 -- 172.21.15.32:0/1088523630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35c000b750 con 0x7f35c80d8490 2021-04-09T18:20:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.405 7f35ccb02700 1 -- 172.21.15.32:0/1088523630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c80d8490 msgr2=0x7f35c810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.405 7f35ccb02700 1 --2- 172.21.15.32:0/1088523630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c80d8490 0x7f35c810da00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f35c0006e30 tx=0x7f35c0009e60).stop 2021-04-09T18:20:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.406 7f35ccb02700 1 -- 172.21.15.32:0/1088523630 shutdown_connections 2021-04-09T18:20:48.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.406 7f35ccb02700 1 --2- 172.21.15.32:0/1088523630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c80d8490 0x7f35c810da00 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.406 7f35ccb02700 1 --2- 172.21.15.32:0/1088523630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c810d010 0x7f35c80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.406 7f35ccb02700 1 -- 172.21.15.32:0/1088523630 >> 172.21.15.32:0/1088523630 conn(0x7f35c80ce5f0 msgr2=0x7f35c80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:48.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.406 7f35ccb02700 1 -- 172.21.15.32:0/1088523630 shutdown_connections 2021-04-09T18:20:48.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.406 7f35ccb02700 1 -- 172.21.15.32:0/1088523630 wait complete. 2021-04-09T18:20:48.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.426 7f35ccb02700 1 Processor -- start 2021-04-09T18:20:48.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.426 7f35ccb02700 1 -- start start 2021-04-09T18:20:48.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35ccb02700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c80b5360 0x7f35c81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:48.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35ccb02700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c818e140 0x7f35c81aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:48.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35ccb02700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c81ad040 0x7f35c808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:48.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35ccb02700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c81a5360 con 0x7f35c81ad040 2021-04-09T18:20:48.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35ccb02700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80902c0 con 0x7f35c818e140 2021-04-09T18:20:48.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35ccb02700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80904e0 con 0x7f35c80b5360 2021-04-09T18:20:48.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c81ad040 0x7f35c808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:48.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c81ad040 0x7f35c808fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47934/0 (socket says 172.21.15.32:47934) 2021-04-09T18:20:48.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c705f700 1 -- 172.21.15.32:0/1780215216 learned_addr learned my addr 172.21.15.32:0/1780215216 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:48.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c685e700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c80b5360 0x7f35c81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:48.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c605d700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c818e140 0x7f35c81aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:48.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c705f700 1 -- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c80b5360 msgr2=0x7f35c81a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:48.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c705f700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c80b5360 0x7f35c81a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c705f700 1 -- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c818e140 msgr2=0x7f35c81aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:48.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c705f700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c818e140 0x7f35c81aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c705f700 1 -- 172.21.15.32:0/1780215216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8183c80 con 0x7f35c81ad040 2021-04-09T18:20:48.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c605d700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c818e140 0x7f35c81aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:20:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35c705f700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c81ad040 0x7f35c808fca0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f35c0001070 tx=0x7f35c000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35b77fe700 1 -- 172.21.15.32:0/1780215216 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35c0001270 con 0x7f35c81ad040 2021-04-09T18:20:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35b77fe700 1 -- 172.21.15.32:0/1780215216 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c0006df0 con 0x7f35c81ad040 2021-04-09T18:20:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c8183c80 con 0x7f35c81ad040 2021-04-09T18:20:48.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35b77fe700 1 -- 172.21.15.32:0/1780215216 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35c0001720 con 0x7f35c81ad040 2021-04-09T18:20:48.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.427 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c8183c80 con 0x7f35c81ad040 2021-04-09T18:20:48.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.428 7f35b77fe700 1 -- 172.21.15.32:0/1780215216 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f35c00014f0 con 0x7f35c81ad040 2021-04-09T18:20:48.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.428 7f35b77fe700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35ac018930 0x7f35ac01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:48.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.428 7f35b77fe700 1 -- 172.21.15.32:0/1780215216 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f35c0032db0 con 0x7f35c81ad040 2021-04-09T18:20:48.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.428 7f35c685e700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35ac018930 0x7f35ac01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:48.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.429 7f35c685e700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35ac018930 0x7f35ac01ae10 crc :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:48.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.448 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c8182d70 con 0x7f35c81ad040 2021-04-09T18:20:48.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.451 7f35b77fe700 1 -- 172.21.15.32:0/1780215216 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f35c0006dd0 con 0x7f35c81ad040 2021-04-09T18:20:48.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.643 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35c8182d70 con 0x7f35c81ad040 2021-04-09T18:20:48.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.644 7f35b77fe700 1 -- 172.21.15.32:0/1780215216 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f35c0006dd0 con 0x7f35c81ad040 2021-04-09T18:20:48.692 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:48.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:48.693 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35ac018930 msgr2=0x7f35ac01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:48.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35ac018930 0x7f35ac01ae10 crc :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c81ad040 msgr2=0x7f35c808fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c81ad040 0x7f35c808fca0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f35c0001070 tx=0x7f35c000bd40).stop 2021-04-09T18:20:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 shutdown_connections 2021-04-09T18:20:48.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35ac018930 0x7f35ac01ae10 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c81ad040 0x7f35c808fca0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c818e140 0x7f35c81aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 --2- 172.21.15.32:0/1780215216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c80b5360 0x7f35c81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:48.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 >> 172.21.15.32:0/1780215216 conn(0x7f35c819b460 msgr2=0x7f35c819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:48.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 shutdown_connections 2021-04-09T18:20:48.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:48.646 7f35ccb02700 1 -- 172.21.15.32:0/1780215216 wait complete. 2021-04-09T18:20:49.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:49.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:50.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:50.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.228 7f6cdfe9e700 1 Processor -- start 2021-04-09T18:20:50.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.228 7f6cdfe9e700 1 -- start start 2021-04-09T18:20:50.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cdfe9e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6cd810d010 0x7f6cd80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:50.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cdfe9e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6cd80d8490 0x7f6cd810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:50.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cdfe9e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd810d480 con 0x7f6cd80d1da0 2021-04-09T18:20:50.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cdfe9e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd80d8d30 con 0x7f6cd80d8490 2021-04-09T18:20:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cdfe9e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd80d9220 con 0x7f6cd810d010 2021-04-09T18:20:50.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cde4bb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6cd80d8490 0x7f6cd810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:50.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cde4bb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6cd80d8490 0x7f6cd810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47942/0 (socket says 172.21.15.32:47942) 2021-04-09T18:20:50.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cde4bb700 1 -- 172.21.15.32:0/3726175151 learned_addr learned my addr 172.21.15.32:0/3726175151 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cddcba700 1 --2- 172.21.15.32:0/3726175151 >> v2:172.21.15.86:3300/0 conn(0x7f6cd810d010 0x7f6cd80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:50.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cde4bb700 1 -- 172.21.15.32:0/3726175151 >> v2:172.21.15.86:3300/0 conn(0x7f6cd810d010 msgr2=0x7f6cd80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:50.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cde4bb700 1 --2- 172.21.15.32:0/3726175151 >> v2:172.21.15.86:3300/0 conn(0x7f6cd810d010 0x7f6cd80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cde4bb700 1 -- 172.21.15.32:0/3726175151 >> v1:172.21.15.86:6789/0 conn(0x7f6cd80d1da0 legacy=0x7f6cd80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:50.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cde4bb700 1 -- 172.21.15.32:0/3726175151 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd810e020 con 0x7f6cd80d8490 2021-04-09T18:20:50.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.229 7f6cde4bb700 1 --2- 172.21.15.32:0/3726175151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd80d8490 0x7f6cd810da00 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f6cd0006b80 tx=0x7f6cd0009b10).ready entity=mon.0 client_cookie=3940cae4b48e9339 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:50.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.230 7f6cdccb8700 1 -- 172.21.15.32:0/3726175151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd00022e0 con 0x7f6cd80d8490 2021-04-09T18:20:50.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.230 7f6cdccb8700 1 -- 172.21.15.32:0/3726175151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd0002620 con 0x7f6cd80d8490 2021-04-09T18:20:50.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.230 7f6cdccb8700 1 -- 172.21.15.32:0/3726175151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd0002b00 con 0x7f6cd80d8490 2021-04-09T18:20:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.230 7f6cdfe9e700 1 -- 172.21.15.32:0/3726175151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd80d8490 msgr2=0x7f6cd810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:50.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.230 7f6cdfe9e700 1 --2- 172.21.15.32:0/3726175151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd80d8490 0x7f6cd810da00 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f6cd0006b80 tx=0x7f6cd0009b10).stop 2021-04-09T18:20:50.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.230 7f6cdfe9e700 1 -- 172.21.15.32:0/3726175151 shutdown_connections 2021-04-09T18:20:50.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.230 7f6cdfe9e700 1 --2- 172.21.15.32:0/3726175151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd80d8490 0x7f6cd810da00 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.230 7f6cdfe9e700 1 --2- 172.21.15.32:0/3726175151 >> v2:172.21.15.86:3300/0 conn(0x7f6cd810d010 0x7f6cd80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.230 7f6cdfe9e700 1 -- 172.21.15.32:0/3726175151 >> 172.21.15.32:0/3726175151 conn(0x7f6cd80ce5f0 msgr2=0x7f6cd80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:50.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.231 7f6cdfe9e700 1 -- 172.21.15.32:0/3726175151 shutdown_connections 2021-04-09T18:20:50.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.231 7f6cdfe9e700 1 -- 172.21.15.32:0/3726175151 wait complete. 2021-04-09T18:20:50.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cdfe9e700 1 Processor -- start 2021-04-09T18:20:50.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cdfe9e700 1 -- start start 2021-04-09T18:20:50.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cdfe9e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cd81a91e0 0x7f6cd80a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:50.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cdfe9e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cd81a95e0 0x7f6cd80a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:50.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cdfe9e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd81ac850 0x7f6cd81acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:50.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cdfe9e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd81ad270 con 0x7f6cd81ac850 2021-04-09T18:20:50.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cdfe9e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80b7640 con 0x7f6cd81a95e0 2021-04-09T18:20:50.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cdfe9e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8183c80 con 0x7f6cd81a91e0 2021-04-09T18:20:50.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cde4bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd81ac850 0x7f6cd81acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:50.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cde4bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd81ac850 0x7f6cd81acc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47946/0 (socket says 172.21.15.32:47946) 2021-04-09T18:20:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.251 7f6cde4bb700 1 -- 172.21.15.32:0/1586070057 learned_addr learned my addr 172.21.15.32:0/1586070057 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:50.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cddcba700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cd81a91e0 0x7f6cd80a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:50.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cde4bb700 1 -- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cd81a91e0 msgr2=0x7f6cd80a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:50.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cde4bb700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cd81a91e0 0x7f6cd80a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cdd4b9700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cd81a95e0 0x7f6cd80a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:50.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cde4bb700 1 -- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cd81a95e0 msgr2=0x7f6cd80a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:50.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cde4bb700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cd81a95e0 0x7f6cd80a5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cde4bb700 1 -- 172.21.15.32:0/1586070057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd8183ea0 con 0x7f6cd81ac850 2021-04-09T18:20:50.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cde4bb700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd81ac850 0x7f6cd81acc50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7f6cd0006f10 tx=0x7f6cd000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:50.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cce7fc700 1 -- 172.21.15.32:0/1586070057 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd0000b90 con 0x7f6cd81ac850 2021-04-09T18:20:50.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cce7fc700 1 -- 172.21.15.32:0/1586070057 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd000bcd0 con 0x7f6cd81ac850 2021-04-09T18:20:50.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd8183ea0 con 0x7f6cd81ac850 2021-04-09T18:20:50.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cce7fc700 1 -- 172.21.15.32:0/1586070057 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd0002220 con 0x7f6cd81ac850 2021-04-09T18:20:50.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.252 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd8183ea0 con 0x7f6cd81ac850 2021-04-09T18:20:50.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.253 7f6cce7fc700 1 -- 172.21.15.32:0/1586070057 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6cd00054a0 con 0x7f6cd81ac850 2021-04-09T18:20:50.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.253 7f6cce7fc700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6cc0018930 0x7f6cc001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:50.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.253 7f6cce7fc700 1 -- 172.21.15.32:0/1586070057 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6cd0032ba0 con 0x7f6cd81ac850 2021-04-09T18:20:50.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.253 7f6cddcba700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6cc0018930 0x7f6cc001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:50.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.254 7f6cddcba700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6cc0018930 0x7f6cc001ae10 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:50.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.273 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd8182d70 con 0x7f6cd81ac850 2021-04-09T18:20:50.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.276 7f6cce7fc700 1 -- 172.21.15.32:0/1586070057 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6cd0009310 con 0x7f6cd81ac850 2021-04-09T18:20:50.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.468 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cd810d480 con 0x7f6cd81ac850 2021-04-09T18:20:50.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.469 7f6cce7fc700 1 -- 172.21.15.32:0/1586070057 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f6cd00054a0 con 0x7f6cd81ac850 2021-04-09T18:20:50.517 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:50.517 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:50.518 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:50.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6cc0018930 msgr2=0x7f6cc001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:50.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6cc0018930 0x7f6cc001ae10 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd81ac850 msgr2=0x7f6cd81acc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:50.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd81ac850 0x7f6cd81acc50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7f6cd0006f10 tx=0x7f6cd000b8e0).stop 2021-04-09T18:20:50.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 shutdown_connections 2021-04-09T18:20:50.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6cc0018930 0x7f6cc001ae10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6cd81ac850 0x7f6cd81acc50 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cd81a95e0 0x7f6cd80a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 --2- 172.21.15.32:0/1586070057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cd81a91e0 0x7f6cd80a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:50.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.471 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 >> 172.21.15.32:0/1586070057 conn(0x7f6cd819b460 msgr2=0x7f6cd819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:50.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.472 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 shutdown_connections 2021-04-09T18:20:50.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:50.472 7f6cdfe9e700 1 -- 172.21.15.32:0/1586070057 wait complete. 2021-04-09T18:20:50.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:50.939 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:51.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:52.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.046 7f07df6b0700 1 Processor -- start 2021-04-09T18:20:52.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.046 7f07df6b0700 1 -- start start 2021-04-09T18:20:52.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07df6b0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d80d7ca0 0x7f07d80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:52.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07df6b0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f07d810da00 0x7f07d810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:52.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07df6b0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d810d480 con 0x7f07d810d010 2021-04-09T18:20:52.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07df6b0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d80d9220 con 0x7f07d810da00 2021-04-09T18:20:52.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07df6b0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80d86c0 con 0x7f07d80d7ca0 2021-04-09T18:20:52.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07dd4cc700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f07d810d010 0x7f07d80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57544/0 (socket says 172.21.15.32:57544) 2021-04-09T18:20:52.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07dd4cc700 1 -- 172.21.15.32:0/2406198153 learned_addr learned my addr 172.21.15.32:0/2406198153 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:52.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07dcccb700 1 --2- 172.21.15.32:0/2406198153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d80d7ca0 0x7f07d80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:52.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07ddccd700 1 --2- 172.21.15.32:0/2406198153 >> v2:172.21.15.86:3300/0 conn(0x7f07d810da00 0x7f07d810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:52.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07cbfff700 1 -- 172.21.15.32:0/2406198153 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102506369 0 0) 0x7f07d4000ce0 con 0x7f07d810d010 2021-04-09T18:20:52.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07cbfff700 1 -- 172.21.15.32:0/2406198153 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07c4002fe0 con 0x7f07d810d010 2021-04-09T18:20:52.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07dcccb700 1 -- 172.21.15.32:0/2406198153 >> v2:172.21.15.86:3300/0 conn(0x7f07d810da00 msgr2=0x7f07d810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:52.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07cbfff700 1 -- 172.21.15.32:0/2406198153 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1511841715 0 0) 0x7f07d40010e0 con 0x7f07d810d010 2021-04-09T18:20:52.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07dcccb700 1 --2- 172.21.15.32:0/2406198153 >> v2:172.21.15.86:3300/0 conn(0x7f07d810da00 0x7f07d810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07dcccb700 1 -- 172.21.15.32:0/2406198153 >> v1:172.21.15.32:6789/0 conn(0x7f07d810d010 legacy=0x7f07d80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:52.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.047 7f07dcccb700 1 -- 172.21.15.32:0/2406198153 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d80d1f70 con 0x7f07d80d7ca0 2021-04-09T18:20:52.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.048 7f07ddccd700 1 --2- 172.21.15.32:0/2406198153 >> v2:172.21.15.86:3300/0 conn(0x7f07d810da00 0x7f07d810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:52.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.048 7f07dcccb700 1 --2- 172.21.15.32:0/2406198153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d80d7ca0 0x7f07d80d80a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f07cc006650 tx=0x7f07cc009a80).ready entity=mon.2 client_cookie=4eb6d4464f7ac738 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:52.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.048 7f07cbfff700 1 -- 172.21.15.32:0/2406198153 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07cc0009b0 con 0x7f07d80d7ca0 2021-04-09T18:20:52.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.048 7f07cbfff700 1 -- 172.21.15.32:0/2406198153 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc00c0c0 con 0x7f07d80d7ca0 2021-04-09T18:20:52.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.048 7f07cbfff700 1 -- 172.21.15.32:0/2406198153 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07cc00c510 con 0x7f07d80d7ca0 2021-04-09T18:20:52.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.048 7f07df6b0700 1 -- 172.21.15.32:0/2406198153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d80d7ca0 msgr2=0x7f07d80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:52.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.048 7f07df6b0700 1 --2- 172.21.15.32:0/2406198153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d80d7ca0 0x7f07d80d80a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f07cc006650 tx=0x7f07cc009a80).stop 2021-04-09T18:20:52.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.049 7f07df6b0700 1 -- 172.21.15.32:0/2406198153 shutdown_connections 2021-04-09T18:20:52.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.049 7f07df6b0700 1 --2- 172.21.15.32:0/2406198153 >> v2:172.21.15.86:3300/0 conn(0x7f07d810da00 0x7f07d810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.049 7f07df6b0700 1 --2- 172.21.15.32:0/2406198153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d80d7ca0 0x7f07d80d80a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.049 7f07df6b0700 1 -- 172.21.15.32:0/2406198153 >> 172.21.15.32:0/2406198153 conn(0x7f07d80ce5f0 msgr2=0x7f07d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:52.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.049 7f07df6b0700 1 -- 172.21.15.32:0/2406198153 shutdown_connections 2021-04-09T18:20:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.049 7f07df6b0700 1 -- 172.21.15.32:0/2406198153 wait complete. 2021-04-09T18:20:52.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.069 7f07df6b0700 1 Processor -- start 2021-04-09T18:20:52.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.069 7f07df6b0700 1 -- start start 2021-04-09T18:20:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07df6b0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d8179bf0 0x7f07d8179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:52.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07df6b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d817c540 0x7f07d817c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07df6b0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d80aca60 0x7f07d80ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07df6b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d817cf60 con 0x7f07d817c540 2021-04-09T18:20:52.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07df6b0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80ad480 con 0x7f07d80aca60 2021-04-09T18:20:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07df6b0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80ad6a0 con 0x7f07d8179bf0 2021-04-09T18:20:52.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dcccb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d817c540 0x7f07d817c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dcccb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d817c540 0x7f07d817c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47962/0 (socket says 172.21.15.32:47962) 2021-04-09T18:20:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dcccb700 1 -- 172.21.15.32:0/801916796 learned_addr learned my addr 172.21.15.32:0/801916796 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:52.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dd4cc700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d8179bf0 0x7f07d8179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07ddccd700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d80aca60 0x7f07d80ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:52.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dcccb700 1 -- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d8179bf0 msgr2=0x7f07d8179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:52.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dcccb700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d8179bf0 0x7f07d8179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dcccb700 1 -- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d80aca60 msgr2=0x7f07d80ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:52.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dcccb700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d80aca60 0x7f07d80ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dcccb700 1 -- 172.21.15.32:0/801916796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d80ad8c0 con 0x7f07d817c540 2021-04-09T18:20:52.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dd4cc700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d8179bf0 0x7f07d8179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:52.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07dcccb700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d817c540 0x7f07d817c940 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f07cc009650 tx=0x7f07cc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:52.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07c9ffb700 1 -- 172.21.15.32:0/801916796 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07cc0093a0 con 0x7f07d817c540 2021-04-09T18:20:52.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07c9ffb700 1 -- 172.21.15.32:0/801916796 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc00bd50 con 0x7f07d817c540 2021-04-09T18:20:52.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07df6b0700 1 -- 172.21.15.32:0/801916796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d80ad8c0 con 0x7f07d817c540 2021-04-09T18:20:52.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07c9ffb700 1 -- 172.21.15.32:0/801916796 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07cc0052e0 con 0x7f07d817c540 2021-04-09T18:20:52.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.070 7f07df6b0700 1 -- 172.21.15.32:0/801916796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d80ad8c0 con 0x7f07d817c540 2021-04-09T18:20:52.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.071 7f07c9ffb700 1 -- 172.21.15.32:0/801916796 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f07cc006030 con 0x7f07d817c540 2021-04-09T18:20:52.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.071 7f07c9ffb700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b8018930 0x7f07b801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:52.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.071 7f07c9ffb700 1 -- 172.21.15.32:0/801916796 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f07cc0329e0 con 0x7f07d817c540 2021-04-09T18:20:52.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.071 7f07dd4cc700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b8018930 0x7f07b801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:52.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.072 7f07dd4cc700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b8018930 0x7f07b801ae10 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.091 7f07df6b0700 1 -- 172.21.15.32:0/801916796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d817cf60 con 0x7f07d817c540 2021-04-09T18:20:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.094 7f07c9ffb700 1 -- 172.21.15.32:0/801916796 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f07cc0009c0 con 0x7f07d817c540 2021-04-09T18:20:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.286 7f07df6b0700 1 -- 172.21.15.32:0/801916796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07d817cf60 con 0x7f07d817c540 2021-04-09T18:20:52.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.287 7f07c9ffb700 1 -- 172.21.15.32:0/801916796 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f07cc00bd50 con 0x7f07d817c540 2021-04-09T18:20:52.335 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:52.337 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:52.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 -- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b8018930 msgr2=0x7f07b801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:52.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b8018930 0x7f07b801ae10 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 -- 172.21.15.32:0/801916796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d817c540 msgr2=0x7f07d817c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:52.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d817c540 0x7f07d817c940 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f07cc009650 tx=0x7f07cc00b960).stop 2021-04-09T18:20:52.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 -- 172.21.15.32:0/801916796 shutdown_connections 2021-04-09T18:20:52.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07b8018930 0x7f07b801ae10 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07d817c540 0x7f07d817c940 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d80aca60 0x7f07d80ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 --2- 172.21.15.32:0/801916796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d8179bf0 0x7f07d8179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:52.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.289 7f07df6b0700 1 -- 172.21.15.32:0/801916796 >> 172.21.15.32:0/801916796 conn(0x7f07d81acae0 msgr2=0x7f07d817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:52.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.290 7f07df6b0700 1 -- 172.21.15.32:0/801916796 shutdown_connections 2021-04-09T18:20:52.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:52.290 7f07df6b0700 1 -- 172.21.15.32:0/801916796 wait complete. 2021-04-09T18:20:52.763 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:52.763 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:53.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:53.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.880 7ff07a939700 1 Processor -- start 2021-04-09T18:20:53.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.880 7ff07a939700 1 -- start start 2021-04-09T18:20:53.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff07a939700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff0740cffa0 0x7ff0740d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:53.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff07a939700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0740d7800 0x7ff0740d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:53.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff07a939700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0740d7280 con 0x7ff0740d8220 2021-04-09T18:20:53.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff07a939700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0740d91e0 con 0x7ff0740d7800 2021-04-09T18:20:53.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff07a939700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07410dfe0 con 0x7ff0740cffa0 2021-04-09T18:20:53.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff06bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0740d7800 0x7ff0740d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:53.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff06bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0740d7800 0x7ff0740d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47972/0 (socket says 172.21.15.32:47972) 2021-04-09T18:20:53.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff06bfff700 1 -- 172.21.15.32:0/3965420812 learned_addr learned my addr 172.21.15.32:0/3965420812 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:53.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff073fff700 1 --2- 172.21.15.32:0/3965420812 >> v2:172.21.15.86:3300/0 conn(0x7ff0740cffa0 0x7ff0740d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:53.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff06bfff700 1 -- 172.21.15.32:0/3965420812 >> v2:172.21.15.86:3300/0 conn(0x7ff0740cffa0 msgr2=0x7ff0740d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:53.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff06bfff700 1 --2- 172.21.15.32:0/3965420812 >> v2:172.21.15.86:3300/0 conn(0x7ff0740cffa0 0x7ff0740d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:53.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff06bfff700 1 -- 172.21.15.32:0/3965420812 >> v1:172.21.15.86:6789/0 conn(0x7ff0740d8220 legacy=0x7ff0740d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:53.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff06bfff700 1 -- 172.21.15.32:0/3965420812 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07410e200 con 0x7ff0740d7800 2021-04-09T18:20:53.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff073fff700 1 --2- 172.21.15.32:0/3965420812 >> v2:172.21.15.86:3300/0 conn(0x7ff0740cffa0 0x7ff0740d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:20:53.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.881 7ff06bfff700 1 --2- 172.21.15.32:0/3965420812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0740d7800 0x7ff0740d7c00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7ff05c006700 tx=0x7ff05c009b80).ready entity=mon.0 client_cookie=630e095750d7e5ff server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:53.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff0737fe700 1 -- 172.21.15.32:0/3965420812 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff05c00c0c0 con 0x7ff0740d7800 2021-04-09T18:20:53.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff0737fe700 1 -- 172.21.15.32:0/3965420812 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c00c440 con 0x7ff0740d7800 2021-04-09T18:20:53.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff0737fe700 1 -- 172.21.15.32:0/3965420812 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff05c00c920 con 0x7ff0740d7800 2021-04-09T18:20:53.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff07a939700 1 -- 172.21.15.32:0/3965420812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0740d7800 msgr2=0x7ff0740d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:53.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff07a939700 1 --2- 172.21.15.32:0/3965420812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0740d7800 0x7ff0740d7c00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7ff05c006700 tx=0x7ff05c009b80).stop 2021-04-09T18:20:53.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff07a939700 1 -- 172.21.15.32:0/3965420812 shutdown_connections 2021-04-09T18:20:53.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff07a939700 1 --2- 172.21.15.32:0/3965420812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0740d7800 0x7ff0740d7c00 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:53.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff07a939700 1 --2- 172.21.15.32:0/3965420812 >> v2:172.21.15.86:3300/0 conn(0x7ff0740cffa0 0x7ff0740d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:53.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff07a939700 1 -- 172.21.15.32:0/3965420812 >> 172.21.15.32:0/3965420812 conn(0x7ff0740ce5c0 msgr2=0x7ff0740cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:53.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.882 7ff07a939700 1 -- 172.21.15.32:0/3965420812 shutdown_connections 2021-04-09T18:20:53.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.883 7ff07a939700 1 -- 172.21.15.32:0/3965420812 wait complete. 2021-04-09T18:20:53.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff07a939700 1 Processor -- start 2021-04-09T18:20:53.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff07a939700 1 -- start start 2021-04-09T18:20:53.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff07a939700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0741929f0 0x7ff074192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:53.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff07a939700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07408ddc0 0x7ff07408e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:53.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff07a939700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07408e7e0 0x7ff07408c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:53.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff07a939700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074193410 con 0x7ff0741929f0 2021-04-09T18:20:53.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff07a939700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740a9140 con 0x7ff07408ddc0 2021-04-09T18:20:53.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff07a939700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07408ce80 con 0x7ff07408e7e0 2021-04-09T18:20:53.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff073fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0741929f0 0x7ff074192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:53.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff073fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0741929f0 0x7ff074192df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47974/0 (socket says 172.21.15.32:47974) 2021-04-09T18:20:53.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff073fff700 1 -- 172.21.15.32:0/3723945095 learned_addr learned my addr 172.21.15.32:0/3723945095 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:53.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff06bfff700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07408ddc0 0x7ff07408e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:53.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.903 7ff078f56700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07408e7e0 0x7ff07408c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:53.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff073fff700 1 -- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07408e7e0 msgr2=0x7ff07408c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:53.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff073fff700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07408e7e0 0x7ff07408c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:53.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff073fff700 1 -- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07408ddc0 msgr2=0x7ff07408e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:53.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff073fff700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07408ddc0 0x7ff07408e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:53.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff073fff700 1 -- 172.21.15.32:0/3723945095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07408d0a0 con 0x7ff0741929f0 2021-04-09T18:20:53.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff073fff700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0741929f0 0x7ff074192df0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7ff064006f20 tx=0x7ff06400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:53.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff0717fa700 1 -- 172.21.15.32:0/3723945095 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff06400bd80 con 0x7ff0741929f0 2021-04-09T18:20:53.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff0717fa700 1 -- 172.21.15.32:0/3723945095 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff064000c90 con 0x7ff0741929f0 2021-04-09T18:20:53.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff07a939700 1 -- 172.21.15.32:0/3723945095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07408d0a0 con 0x7ff0741929f0 2021-04-09T18:20:53.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff0717fa700 1 -- 172.21.15.32:0/3723945095 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0640022b0 con 0x7ff0741929f0 2021-04-09T18:20:53.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.904 7ff07a939700 1 -- 172.21.15.32:0/3723945095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07408d0a0 con 0x7ff0741929f0 2021-04-09T18:20:53.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.905 7ff0717fa700 1 -- 172.21.15.32:0/3723945095 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff06400a000 con 0x7ff0741929f0 2021-04-09T18:20:53.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.905 7ff0717fa700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff058018900 0x7ff05801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:53.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.905 7ff0717fa700 1 -- 172.21.15.32:0/3723945095 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff064032a30 con 0x7ff0741929f0 2021-04-09T18:20:53.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.905 7ff06bfff700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff058018900 0x7ff05801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:53.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.905 7ff06bfff700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff058018900 0x7ff05801adc0 crc :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:53.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.924 7ff07a939700 1 -- 172.21.15.32:0/3723945095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0741b70c0 con 0x7ff0741929f0 2021-04-09T18:20:53.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:53.928 7ff0717fa700 1 -- 172.21.15.32:0/3723945095 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff064006bb0 con 0x7ff0741929f0 2021-04-09T18:20:54.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.119 7ff07a939700 1 -- 172.21.15.32:0/3723945095 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0740d7280 con 0x7ff0741929f0 2021-04-09T18:20:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.119 7ff0717fa700 1 -- 172.21.15.32:0/3723945095 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7ff064006bb0 con 0x7ff0741929f0 2021-04-09T18:20:54.167 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:54.168 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:54.168 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 -- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff058018900 msgr2=0x7ff05801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:54.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff058018900 0x7ff05801adc0 crc :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:54.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 -- 172.21.15.32:0/3723945095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0741929f0 msgr2=0x7ff074192df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:54.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0741929f0 0x7ff074192df0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7ff064006f20 tx=0x7ff06400b990).stop 2021-04-09T18:20:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 -- 172.21.15.32:0/3723945095 shutdown_connections 2021-04-09T18:20:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff058018900 0x7ff05801adc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:54.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07408e7e0 0x7ff07408c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:54.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07408ddc0 0x7ff07408e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:54.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 --2- 172.21.15.32:0/3723945095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0741929f0 0x7ff074192df0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:54.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 -- 172.21.15.32:0/3723945095 >> 172.21.15.32:0/3723945095 conn(0x7ff074189b00 msgr2=0x7ff0741921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:54.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 -- 172.21.15.32:0/3723945095 shutdown_connections 2021-04-09T18:20:54.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:54.122 7ff07a939700 1 -- 172.21.15.32:0/3723945095 wait complete. 2021-04-09T18:20:54.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:54.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:55.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.699 7f83b506e700 1 Processor -- start 2021-04-09T18:20:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.699 7f83b506e700 1 -- start start 2021-04-09T18:20:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.699 7f83b506e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f83b00cffa0 0x7f83b00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.699 7f83b506e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00d7800 0x7f83b00d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83b506e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b00d7280 con 0x7f83b00d8220 2021-04-09T18:20:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83b506e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b00d91e0 con 0x7f83b00cffa0 2021-04-09T18:20:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83b506e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b010dfe0 con 0x7f83b00d7800 2021-04-09T18:20:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83aed9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f83b00cffa0 0x7f83b00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83ae59c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00d7800 0x7f83b00d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83aed9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f83b00cffa0 0x7f83b00d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47982/0 (socket says 172.21.15.32:47982) 2021-04-09T18:20:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83aed9d700 1 -- 172.21.15.32:0/302788382 learned_addr learned my addr 172.21.15.32:0/302788382 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83af59e700 1 --1- 172.21.15.32:0/302788382 >> v1:172.21.15.86:6789/0 conn(0x7f83b00d8220 0x7f83b00d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55500/0 (socket says 172.21.15.32:55500) 2021-04-09T18:20:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83add9b700 1 -- 172.21.15.32:0/302788382 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970794975 0 0) 0x7f83a80009b0 con 0x7f83b00d8220 2021-04-09T18:20:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83aed9d700 1 -- 172.21.15.32:0/302788382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00d7800 msgr2=0x7f83b00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83aed9d700 1 --2- 172.21.15.32:0/302788382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00d7800 0x7f83b00d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83aed9d700 1 -- 172.21.15.32:0/302788382 >> v1:172.21.15.86:6789/0 conn(0x7f83b00d8220 legacy=0x7f83b00d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83aed9d700 1 -- 172.21.15.32:0/302788382 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b010e200 con 0x7f83b00cffa0 2021-04-09T18:20:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83ae59c700 1 --2- 172.21.15.32:0/302788382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00d7800 0x7f83b00d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83aed9d700 1 --2- 172.21.15.32:0/302788382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b00cffa0 0x7f83b00d2440 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7f83a0006a80 tx=0x7f83a0009f00).ready entity=mon.0 client_cookie=7f7fc3343bd415b4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:55.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83add9b700 1 -- 172.21.15.32:0/302788382 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a000b5c0 con 0x7f83b00cffa0 2021-04-09T18:20:55.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83add9b700 1 -- 172.21.15.32:0/302788382 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a000b8c0 con 0x7f83b00cffa0 2021-04-09T18:20:55.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83add9b700 1 -- 172.21.15.32:0/302788382 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a000bda0 con 0x7f83b00cffa0 2021-04-09T18:20:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83b506e700 1 -- 172.21.15.32:0/302788382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b00cffa0 msgr2=0x7f83b00d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.700 7f83b506e700 1 --2- 172.21.15.32:0/302788382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b00cffa0 0x7f83b00d2440 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7f83a0006a80 tx=0x7f83a0009f00).stop 2021-04-09T18:20:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.701 7f83b506e700 1 -- 172.21.15.32:0/302788382 shutdown_connections 2021-04-09T18:20:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.701 7f83b506e700 1 --2- 172.21.15.32:0/302788382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00d7800 0x7f83b00d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.701 7f83b506e700 1 --2- 172.21.15.32:0/302788382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b00cffa0 0x7f83b00d2440 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.701 7f83b506e700 1 -- 172.21.15.32:0/302788382 >> 172.21.15.32:0/302788382 conn(0x7f83b00ce5c0 msgr2=0x7f83b00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.701 7f83b506e700 1 -- 172.21.15.32:0/302788382 shutdown_connections 2021-04-09T18:20:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.701 7f83b506e700 1 -- 172.21.15.32:0/302788382 wait complete. 2021-04-09T18:20:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.721 7f83b506e700 1 Processor -- start 2021-04-09T18:20:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.721 7f83b506e700 1 -- start start 2021-04-09T18:20:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83b506e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b01acff0 0x7f83b01847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83b506e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b0183fc0 0x7f83b01a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:55.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83b506e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b0186ff0 0x7f83b01873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83b506e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b0188c30 con 0x7f83b01acff0 2021-04-09T18:20:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83b506e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b01843c0 con 0x7f83b0183fc0 2021-04-09T18:20:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83b506e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b01a5c20 con 0x7f83b0186ff0 2021-04-09T18:20:55.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83aed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b01acff0 0x7f83b01847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83aed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b01acff0 0x7f83b01847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47990/0 (socket says 172.21.15.32:47990) 2021-04-09T18:20:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83aed9d700 1 -- 172.21.15.32:0/3064055785 learned_addr learned my addr 172.21.15.32:0/3064055785 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83af59e700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b0186ff0 0x7f83b01873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83ae59c700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b0183fc0 0x7f83b01a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:55.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83aed9d700 1 -- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b0186ff0 msgr2=0x7f83b01873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83aed9d700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b0186ff0 0x7f83b01873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83aed9d700 1 -- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b0183fc0 msgr2=0x7f83b01a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83aed9d700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b0183fc0 0x7f83b01a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83aed9d700 1 -- 172.21.15.32:0/3064055785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b0187a10 con 0x7f83b01acff0 2021-04-09T18:20:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83af59e700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b0186ff0 0x7f83b01873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:20:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.722 7f83aed9d700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b01acff0 0x7f83b01847a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7f83a0006a90 tx=0x7f83a000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:55.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.723 7f839f7fe700 1 -- 172.21.15.32:0/3064055785 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a0000d30 con 0x7f83b01acff0 2021-04-09T18:20:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.723 7f839f7fe700 1 -- 172.21.15.32:0/3064055785 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a0006710 con 0x7f83b01acff0 2021-04-09T18:20:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.723 7f839f7fe700 1 -- 172.21.15.32:0/3064055785 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a000c4e0 con 0x7f83b01acff0 2021-04-09T18:20:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.723 7f83b506e700 1 -- 172.21.15.32:0/3064055785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b0187a10 con 0x7f83b01acff0 2021-04-09T18:20:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.723 7f83b506e700 1 -- 172.21.15.32:0/3064055785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b0187a10 con 0x7f83b01acff0 2021-04-09T18:20:55.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.723 7f839f7fe700 1 -- 172.21.15.32:0/3064055785 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f83a0007000 con 0x7f83b01acff0 2021-04-09T18:20:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.723 7f839f7fe700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83940188c0 0x7f839401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.723 7f839f7fe700 1 -- 172.21.15.32:0/3064055785 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f83a0032de0 con 0x7f83b01acff0 2021-04-09T18:20:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.724 7f83ae59c700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83940188c0 0x7f839401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:55.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.724 7f83ae59c700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83940188c0 0x7f839401adc0 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:55.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.743 7f83b506e700 1 -- 172.21.15.32:0/3064055785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b0182dc0 con 0x7f83b01acff0 2021-04-09T18:20:55.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.747 7f839f7fe700 1 -- 172.21.15.32:0/3064055785 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f83a0007330 con 0x7f83b01acff0 2021-04-09T18:20:55.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.938 7f83b506e700 1 -- 172.21.15.32:0/3064055785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b00d7280 con 0x7f83b01acff0 2021-04-09T18:20:55.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.939 7f839f7fe700 1 -- 172.21.15.32:0/3064055785 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f83a0007000 con 0x7f83b01acff0 2021-04-09T18:20:55.987 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:55.987 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:55.987 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:55.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.941 7f83b506e700 1 -- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83940188c0 msgr2=0x7f839401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:55.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.941 7f83b506e700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83940188c0 0x7f839401adc0 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.941 7f83b506e700 1 -- 172.21.15.32:0/3064055785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b01acff0 msgr2=0x7f83b01847a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:55.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.941 7f83b506e700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b01acff0 0x7f83b01847a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7f83a0006a90 tx=0x7f83a000bd10).stop 2021-04-09T18:20:55.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.942 7f83b506e700 1 -- 172.21.15.32:0/3064055785 shutdown_connections 2021-04-09T18:20:55.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.942 7f83b506e700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83940188c0 0x7f839401adc0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.942 7f83b506e700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b0186ff0 0x7f83b01873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.942 7f83b506e700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b0183fc0 0x7f83b01a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.942 7f83b506e700 1 --2- 172.21.15.32:0/3064055785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83b01acff0 0x7f83b01847a0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:55.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.942 7f83b506e700 1 -- 172.21.15.32:0/3064055785 >> 172.21.15.32:0/3064055785 conn(0x7f83b009abb0 msgr2=0x7f83b00a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:55.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.942 7f83b506e700 1 -- 172.21.15.32:0/3064055785 shutdown_connections 2021-04-09T18:20:55.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:55.942 7f83b506e700 1 -- 172.21.15.32:0/3064055785 wait complete. 2021-04-09T18:20:56.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:56.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:57.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:57.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.518 7fd6d39fd700 1 Processor -- start 2021-04-09T18:20:57.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.518 7fd6d39fd700 1 -- start start 2021-04-09T18:20:57.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.519 7fd6d39fd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd6cc0d7ca0 0x7fd6cc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:57.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.519 7fd6d39fd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc10d480 con 0x7fd6cc10d010 2021-04-09T18:20:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.519 7fd6d39fd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc0d86c0 con 0x7fd6cc10da00 2021-04-09T18:20:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.519 7fd6d39fd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc0d9220 con 0x7fd6cc0d7ca0 2021-04-09T18:20:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.519 7fd6d1819700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd6cc10d010 0x7fd6cc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57586/0 (socket says 172.21.15.32:57586) 2021-04-09T18:20:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.519 7fd6d1819700 1 -- 172.21.15.32:0/506914662 learned_addr learned my addr 172.21.15.32:0/506914662 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:57.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.519 7fd6d1018700 1 --2- 172.21.15.32:0/506914662 >> v2:172.21.15.86:3300/0 conn(0x7fd6cc0d7ca0 0x7fd6cc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.519 7fd6d0817700 1 -- 172.21.15.32:0/506914662 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638544614 0 0) 0x7fd6c8000ce0 con 0x7fd6cc10d010 2021-04-09T18:20:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.519 7fd6d0817700 1 -- 172.21.15.32:0/506914662 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b8002fe0 con 0x7fd6cc10d010 2021-04-09T18:20:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2657635179 0 0) 0x7fd6c80010e0 con 0x7fd6cc10d010 2021-04-09T18:20:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 >> v2:172.21.15.86:3300/0 conn(0x7fd6cc0d7ca0 msgr2=0x7fd6cc0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 --2- 172.21.15.32:0/506914662 >> v2:172.21.15.86:3300/0 conn(0x7fd6cc0d7ca0 0x7fd6cc0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 >> v1:172.21.15.86:6789/0 conn(0x7fd6cc10da00 legacy=0x7fd6cc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc0d1f70 con 0x7fd6cc10d010 2021-04-09T18:20:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fd6c8001600 con 0x7fd6cc10d010 2021-04-09T18:20:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 >> v1:172.21.15.32:6789/0 conn(0x7fd6cc10d010 legacy=0x7fd6cc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b8003a70 0x7fd6b8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b80067a0 0x7fd6b8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6b80095a0 0x7fd6b800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b800c310 con 0x7fd6b80095a0 2021-04-09T18:20:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b800c530 con 0x7fd6b8003a70 2021-04-09T18:20:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b8050760 con 0x7fd6b80067a0 2021-04-09T18:20:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d201a700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6b80095a0 0x7fd6b800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d1819700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b8003a70 0x7fd6b8005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d1018700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b80067a0 0x7fd6b8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d201a700 1 -- 172.21.15.32:0/506914662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b80067a0 msgr2=0x7fd6b8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d201a700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b80067a0 0x7fd6b8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d201a700 1 -- 172.21.15.32:0/506914662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b8003a70 msgr2=0x7fd6b8005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:20:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d201a700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6b8003a70 0x7fd6b8005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d201a700 1 -- 172.21.15.32:0/506914662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b80509c0 con 0x7fd6b80095a0 2021-04-09T18:20:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d1018700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6b80067a0 0x7fd6b8008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:20:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d201a700 1 -- 172.21.15.32:0/506914662 reap_dead start 2021-04-09T18:20:57.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d201a700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6b80095a0 0x7fd6b800bac0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7fd6c4006e40 tx=0x7fd6c4009e30).ready entity=mon.0 client_cookie=42343869dbf79580 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:57.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.520 7fd6d0817700 1 -- 172.21.15.32:0/506914662 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c4006a30 con 0x7fd6b80095a0 2021-04-09T18:20:57.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.521 7fd6d0817700 1 -- 172.21.15.32:0/506914662 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c400b300 con 0x7fd6b80095a0 2021-04-09T18:20:57.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.521 7fd6d0817700 1 -- 172.21.15.32:0/506914662 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c400b7e0 con 0x7fd6b80095a0 2021-04-09T18:20:57.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.521 7fd6d39fd700 1 -- 172.21.15.32:0/506914662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6b80095a0 msgr2=0x7fd6b800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:57.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.521 7fd6d39fd700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6b80095a0 0x7fd6b800bac0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7fd6c4006e40 tx=0x7fd6c4009e30).stop 2021-04-09T18:20:57.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.521 7fd6d39fd700 1 -- 172.21.15.32:0/506914662 shutdown_connections 2021-04-09T18:20:57.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.521 7fd6d39fd700 1 --2- 172.21.15.32:0/506914662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6b80095a0 0x7fd6b800bac0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.521 7fd6d39fd700 1 -- 172.21.15.32:0/506914662 >> 172.21.15.32:0/506914662 conn(0x7fd6cc0ce5f0 msgr2=0x7fd6cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:57.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.521 7fd6d39fd700 1 -- 172.21.15.32:0/506914662 shutdown_connections 2021-04-09T18:20:57.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.521 7fd6d39fd700 1 -- 172.21.15.32:0/506914662 wait complete. 2021-04-09T18:20:57.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d39fd700 1 Processor -- start 2021-04-09T18:20:57.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d39fd700 1 -- start start 2021-04-09T18:20:57.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d39fd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac380 0x7fd6cc18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d39fd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1ac780 0x7fd6cc18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d39fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc183520 0x7fd6cc183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:57.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d39fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc1ace10 con 0x7fd6cc183520 2021-04-09T18:20:57.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d39fd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc189cf0 con 0x7fd6cc1ac780 2021-04-09T18:20:57.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d39fd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc183f40 con 0x7fd6cc1ac380 2021-04-09T18:20:57.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d201a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc183520 0x7fd6cc183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:57.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d201a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc183520 0x7fd6cc183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48010/0 (socket says 172.21.15.32:48010) 2021-04-09T18:20:57.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d201a700 1 -- 172.21.15.32:0/465271908 learned_addr learned my addr 172.21.15.32:0/465271908 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:57.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d1819700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac380 0x7fd6cc18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:57.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.542 7fd6d1018700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1ac780 0x7fd6cc18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:57.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6d201a700 1 -- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac380 msgr2=0x7fd6cc18a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:57.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6d201a700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac380 0x7fd6cc18a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6d201a700 1 -- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1ac780 msgr2=0x7fd6cc18a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:57.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6d201a700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1ac780 0x7fd6cc18a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6d201a700 1 -- 172.21.15.32:0/465271908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc184160 con 0x7fd6cc183520 2021-04-09T18:20:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6d201a700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc183520 0x7fd6cc183920 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7fd6c4006fc0 tx=0x7fd6c400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:57.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6c27fc700 1 -- 172.21.15.32:0/465271908 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c4003080 con 0x7fd6cc183520 2021-04-09T18:20:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6c27fc700 1 -- 172.21.15.32:0/465271908 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c4000e70 con 0x7fd6cc183520 2021-04-09T18:20:57.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6c27fc700 1 -- 172.21.15.32:0/465271908 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c40016b0 con 0x7fd6cc183520 2021-04-09T18:20:57.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc184160 con 0x7fd6cc183520 2021-04-09T18:20:57.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.543 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc184160 con 0x7fd6cc183520 2021-04-09T18:20:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.544 7fd6c27fc700 1 -- 172.21.15.32:0/465271908 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd6c4001220 con 0x7fd6cc183520 2021-04-09T18:20:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.544 7fd6c27fc700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6b40188d0 0x7fd6b401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:57.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.544 7fd6c27fc700 1 -- 172.21.15.32:0/465271908 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd6c4032dd0 con 0x7fd6cc183520 2021-04-09T18:20:57.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.544 7fd6d1819700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6b40188d0 0x7fd6b401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:57.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.544 7fd6d1819700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6b40188d0 0x7fd6b401add0 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:57.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.563 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc189cf0 con 0x7fd6cc183520 2021-04-09T18:20:57.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.566 7fd6c27fc700 1 -- 172.21.15.32:0/465271908 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd6c4003000 con 0x7fd6cc183520 2021-04-09T18:20:57.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.758 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6cc189cf0 con 0x7fd6cc183520 2021-04-09T18:20:57.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.759 7fd6c27fc700 1 -- 172.21.15.32:0/465271908 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fd6c4001220 con 0x7fd6cc183520 2021-04-09T18:20:57.807 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:57.808 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:57.809 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:57.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6b40188d0 msgr2=0x7fd6b401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:57.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6b40188d0 0x7fd6b401add0 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc183520 msgr2=0x7fd6cc183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:57.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc183520 0x7fd6cc183920 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7fd6c4006fc0 tx=0x7fd6c400bd10).stop 2021-04-09T18:20:57.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 shutdown_connections 2021-04-09T18:20:57.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6b40188d0 0x7fd6b401add0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc183520 0x7fd6cc183920 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1ac780 0x7fd6cc18a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 --2- 172.21.15.32:0/465271908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac380 0x7fd6cc18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:57.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 >> 172.21.15.32:0/465271908 conn(0x7fd6cc19b890 msgr2=0x7fd6cc1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:57.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 shutdown_connections 2021-04-09T18:20:57.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:57.761 7fd6d39fd700 1 -- 172.21.15.32:0/465271908 wait complete. 2021-04-09T18:20:58.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:20:58.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:20:59.230 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:20:59.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.343 7fbae98a5700 1 Processor -- start 2021-04-09T18:20:59.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.343 7fbae98a5700 1 -- start start 2021-04-09T18:20:59.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.344 7fbae98a5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbae410da00 0x7fbae410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:59.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.344 7fbae98a5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae410d480 con 0x7fbae40d7ca0 2021-04-09T18:20:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.344 7fbae98a5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae40d1f70 con 0x7fbae410d010 2021-04-09T18:20:59.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.344 7fbae98a5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae40d8d30 con 0x7fbae410da00 2021-04-09T18:20:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.344 7fbae3d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbae410da00 0x7fbae410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.344 7fbae2d73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbae40d7ca0 0x7fbae40d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57610/0 (socket says 172.21.15.32:57610) 2021-04-09T18:20:59.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.344 7fbae2d73700 1 -- 172.21.15.32:0/1137543773 learned_addr learned my addr 172.21.15.32:0/1137543773 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:20:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae2572700 1 -- 172.21.15.32:0/1137543773 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810858837 0 0) 0x7fbad4000c60 con 0x7fbae410d010 2021-04-09T18:20:59.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae3d75700 1 -- 172.21.15.32:0/1137543773 >> v1:172.21.15.86:6789/0 conn(0x7fbae410d010 legacy=0x7fbae40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:59.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae3d75700 1 -- 172.21.15.32:0/1137543773 >> v1:172.21.15.32:6789/0 conn(0x7fbae40d7ca0 legacy=0x7fbae40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:59.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae3d75700 1 -- 172.21.15.32:0/1137543773 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae40d9220 con 0x7fbae410da00 2021-04-09T18:20:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae3d75700 1 --2- 172.21.15.32:0/1137543773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae410da00 0x7fbae410de00 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7fbadc006e30 tx=0x7fbadc009e60).ready entity=mon.0 client_cookie=4066a22ba2a86bbb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae2572700 1 -- 172.21.15.32:0/1137543773 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbadc006a50 con 0x7fbae410da00 2021-04-09T18:20:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae2572700 1 -- 172.21.15.32:0/1137543773 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc00b300 con 0x7fbae410da00 2021-04-09T18:20:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae2572700 1 -- 172.21.15.32:0/1137543773 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbadc00b750 con 0x7fbae410da00 2021-04-09T18:20:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae98a5700 1 -- 172.21.15.32:0/1137543773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae410da00 msgr2=0x7fbae410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:59.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.345 7fbae98a5700 1 --2- 172.21.15.32:0/1137543773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae410da00 0x7fbae410de00 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7fbadc006e30 tx=0x7fbadc009e60).stop 2021-04-09T18:20:59.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.346 7fbae98a5700 1 -- 172.21.15.32:0/1137543773 shutdown_connections 2021-04-09T18:20:59.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.346 7fbae98a5700 1 --2- 172.21.15.32:0/1137543773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae410da00 0x7fbae410de00 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.346 7fbae98a5700 1 -- 172.21.15.32:0/1137543773 >> 172.21.15.32:0/1137543773 conn(0x7fbae40ce5f0 msgr2=0x7fbae40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.346 7fbae98a5700 1 -- 172.21.15.32:0/1137543773 shutdown_connections 2021-04-09T18:20:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.346 7fbae98a5700 1 -- 172.21.15.32:0/1137543773 wait complete. 2021-04-09T18:20:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.366 7fbae98a5700 1 Processor -- start 2021-04-09T18:20:59.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.366 7fbae98a5700 1 -- start start 2021-04-09T18:20:59.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.366 7fbae98a5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae417dc90 0x7fbae417e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:59.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.366 7fbae98a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae41ac720 0x7fbae41acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:59.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.366 7fbae98a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae409bb70 0x7fbae409bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:59.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.366 7fbae98a5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae409b690 con 0x7fbae41ac720 2021-04-09T18:20:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.366 7fbae98a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae4093fc0 con 0x7fbae409bb70 2021-04-09T18:20:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.366 7fbae98a5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae417e6b0 con 0x7fbae417dc90 2021-04-09T18:20:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae2d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae41ac720 0x7fbae41acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae2d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae41ac720 0x7fbae41acb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48024/0 (socket says 172.21.15.32:48024) 2021-04-09T18:20:59.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae2d73700 1 -- 172.21.15.32:0/758090550 learned_addr learned my addr 172.21.15.32:0/758090550 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:20:59.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae3d75700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae409bb70 0x7fbae409bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:59.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae3574700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae417dc90 0x7fbae417e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:59.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae2d73700 1 -- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae417dc90 msgr2=0x7fbae417e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:59.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae2d73700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae417dc90 0x7fbae417e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:59.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae2d73700 1 -- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae409bb70 msgr2=0x7fbae409bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae2d73700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae409bb70 0x7fbae409bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae2d73700 1 -- 172.21.15.32:0/758090550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae41807d0 con 0x7fbae41ac720 2021-04-09T18:20:59.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae3d75700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae409bb70 0x7fbae409bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:20:59.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae2d73700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae41ac720 0x7fbae41acb20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7fbad800a5a0 tx=0x7fbad800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:59.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbacbfff700 1 -- 172.21.15.32:0/758090550 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad8000d10 con 0x7fbae41ac720 2021-04-09T18:20:59.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbacbfff700 1 -- 172.21.15.32:0/758090550 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad80043a0 con 0x7fbae41ac720 2021-04-09T18:20:59.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae98a5700 1 -- 172.21.15.32:0/758090550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae41807d0 con 0x7fbae41ac720 2021-04-09T18:20:59.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbacbfff700 1 -- 172.21.15.32:0/758090550 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad8004880 con 0x7fbae41ac720 2021-04-09T18:20:59.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.367 7fbae98a5700 1 -- 172.21.15.32:0/758090550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae41807d0 con 0x7fbae41ac720 2021-04-09T18:20:59.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.368 7fbacbfff700 1 -- 172.21.15.32:0/758090550 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbad80047f0 con 0x7fbae41ac720 2021-04-09T18:20:59.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.368 7fbacbfff700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac40188d0 0x7fbac401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:20:59.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.368 7fbacbfff700 1 -- 172.21.15.32:0/758090550 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbad80329e0 con 0x7fbae41ac720 2021-04-09T18:20:59.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.368 7fbae3574700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac40188d0 0x7fbac401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:20:59.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.369 7fbae3574700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac40188d0 0x7fbac401add0 crc :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:20:59.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.388 7fbae98a5700 1 -- 172.21.15.32:0/758090550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae409c880 con 0x7fbae41ac720 2021-04-09T18:20:59.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.392 7fbacbfff700 1 -- 172.21.15.32:0/758090550 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbad8000df0 con 0x7fbae41ac720 2021-04-09T18:20:59.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.585 7fbae98a5700 1 -- 172.21.15.32:0/758090550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbae409c880 con 0x7fbae41ac720 2021-04-09T18:20:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.585 7fbacbfff700 1 -- 172.21.15.32:0/758090550 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fbad800ad30 con 0x7fbae41ac720 2021-04-09T18:20:59.633 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:20:59.634 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:20:59.634 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:20:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.587 7fbae98a5700 1 -- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac40188d0 msgr2=0x7fbac401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.587 7fbae98a5700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac40188d0 0x7fbac401add0 crc :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:59.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.587 7fbae98a5700 1 -- 172.21.15.32:0/758090550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae41ac720 msgr2=0x7fbae41acb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:20:59.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.587 7fbae98a5700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae41ac720 0x7fbae41acb20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7fbad800a5a0 tx=0x7fbad800a940).stop 2021-04-09T18:20:59.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.588 7fbae98a5700 1 -- 172.21.15.32:0/758090550 shutdown_connections 2021-04-09T18:20:59.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.588 7fbae98a5700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbac40188d0 0x7fbac401add0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:59.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.588 7fbae98a5700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbae41ac720 0x7fbae41acb20 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:59.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.588 7fbae98a5700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbae409bb70 0x7fbae409bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:59.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.588 7fbae98a5700 1 --2- 172.21.15.32:0/758090550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbae417dc90 0x7fbae417e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:20:59.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.588 7fbae98a5700 1 -- 172.21.15.32:0/758090550 >> 172.21.15.32:0/758090550 conn(0x7fbae4188510 msgr2=0x7fbae41a7320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:20:59.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.588 7fbae98a5700 1 -- 172.21.15.32:0/758090550 shutdown_connections 2021-04-09T18:20:59.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:20:59.588 7fbae98a5700 1 -- 172.21.15.32:0/758090550 wait complete. 2021-04-09T18:21:00.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:00.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:01.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:01.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.162 7fa9582e9700 1 Processor -- start 2021-04-09T18:21:01.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.163 7fa9582e9700 1 -- start start 2021-04-09T18:21:01.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.163 7fa9582e9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9500d7ca0 0x7fa9500d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:01.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.163 7fa9582e9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa95010da00 0x7fa95010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:01.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.163 7fa9582e9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95010d480 con 0x7fa95010d010 2021-04-09T18:21:01.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.163 7fa9582e9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9500d9220 con 0x7fa95010da00 2021-04-09T18:21:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.163 7fa9582e9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9500d86c0 con 0x7fa9500d7ca0 2021-04-09T18:21:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.163 7fa956906700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa95010da00 0x7fa95010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:01.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa956906700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa95010da00 0x7fa95010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48032/0 (socket says 172.21.15.32:48032) 2021-04-09T18:21:01.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa956906700 1 -- 172.21.15.32:0/3165790876 learned_addr learned my addr 172.21.15.32:0/3165790876 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:01.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa956105700 1 --1- 172.21.15.32:0/3165790876 >> v1:172.21.15.86:6789/0 conn(0x7fa95010d010 0x7fa9500d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55546/0 (socket says 172.21.15.32:55546) 2021-04-09T18:21:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa955904700 1 --2- 172.21.15.32:0/3165790876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9500d7ca0 0x7fa9500d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa956906700 1 -- 172.21.15.32:0/3165790876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9500d7ca0 msgr2=0x7fa9500d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa956906700 1 --2- 172.21.15.32:0/3165790876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9500d7ca0 0x7fa9500d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa956906700 1 -- 172.21.15.32:0/3165790876 >> v1:172.21.15.86:6789/0 conn(0x7fa95010d010 legacy=0x7fa9500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:01.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa955103700 1 -- 172.21.15.32:0/3165790876 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922700 0 0) 0x7fa94c000c60 con 0x7fa95010d010 2021-04-09T18:21:01.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa956906700 1 -- 172.21.15.32:0/3165790876 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9500d1f70 con 0x7fa95010da00 2021-04-09T18:21:01.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa956906700 1 --2- 172.21.15.32:0/3165790876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95010da00 0x7fa95010de00 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7fa948006f00 tx=0x7fa948009f10).ready entity=mon.0 client_cookie=40f472ee5d7a4c05 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:01.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa955103700 1 -- 172.21.15.32:0/3165790876 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa94800b3b0 con 0x7fa95010da00 2021-04-09T18:21:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa955103700 1 -- 172.21.15.32:0/3165790876 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94800b730 con 0x7fa95010da00 2021-04-09T18:21:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa955103700 1 -- 172.21.15.32:0/3165790876 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa948000f90 con 0x7fa95010da00 2021-04-09T18:21:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa9582e9700 1 -- 172.21.15.32:0/3165790876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95010da00 msgr2=0x7fa95010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.164 7fa9582e9700 1 --2- 172.21.15.32:0/3165790876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95010da00 0x7fa95010de00 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7fa948006f00 tx=0x7fa948009f10).stop 2021-04-09T18:21:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.165 7fa9582e9700 1 -- 172.21.15.32:0/3165790876 shutdown_connections 2021-04-09T18:21:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.165 7fa9582e9700 1 --2- 172.21.15.32:0/3165790876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95010da00 0x7fa95010de00 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.165 7fa9582e9700 1 --2- 172.21.15.32:0/3165790876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9500d7ca0 0x7fa9500d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.165 7fa9582e9700 1 -- 172.21.15.32:0/3165790876 >> 172.21.15.32:0/3165790876 conn(0x7fa9500ce5f0 msgr2=0x7fa9500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.165 7fa9582e9700 1 -- 172.21.15.32:0/3165790876 shutdown_connections 2021-04-09T18:21:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.165 7fa9582e9700 1 -- 172.21.15.32:0/3165790876 wait complete. 2021-04-09T18:21:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.185 7fa9582e9700 1 Processor -- start 2021-04-09T18:21:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.185 7fa9582e9700 1 -- start start 2021-04-09T18:21:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa9582e9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9501aca20 0x7fa9501ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa9582e9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95018bd40 0x7fa95018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa9582e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95008fba0 0x7fa95008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa9582e9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95018c7e0 con 0x7fa95008fba0 2021-04-09T18:21:01.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa9582e9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95018ca30 con 0x7fa95018bd40 2021-04-09T18:21:01.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa9582e9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9500905f0 con 0x7fa9501aca20 2021-04-09T18:21:01.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa956906700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95008fba0 0x7fa95008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa956906700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95008fba0 0x7fa95008ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48040/0 (socket says 172.21.15.32:48040) 2021-04-09T18:21:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa956906700 1 -- 172.21.15.32:0/3425795887 learned_addr learned my addr 172.21.15.32:0/3425795887 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa956105700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9501aca20 0x7fa9501ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa955904700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95018bd40 0x7fa95018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:01.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa956906700 1 -- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9501aca20 msgr2=0x7fa9501ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:01.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa956906700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9501aca20 0x7fa9501ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa956906700 1 -- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95018bd40 msgr2=0x7fa95018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:01.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa956906700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95018bd40 0x7fa95018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa956906700 1 -- 172.21.15.32:0/3425795887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9500b4300 con 0x7fa95008fba0 2021-04-09T18:21:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.186 7fa955904700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95018bd40 0x7fa95018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.187 7fa956906700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95008fba0 0x7fa95008ffa0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7fa948009f90 tx=0x7fa94800bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.187 7fa946ffd700 1 -- 172.21.15.32:0/3425795887 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa94800bd80 con 0x7fa95008fba0 2021-04-09T18:21:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.187 7fa946ffd700 1 -- 172.21.15.32:0/3425795887 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa948005820 con 0x7fa95008fba0 2021-04-09T18:21:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.187 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9500b4300 con 0x7fa95008fba0 2021-04-09T18:21:01.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.187 7fa946ffd700 1 -- 172.21.15.32:0/3425795887 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa948005c70 con 0x7fa95008fba0 2021-04-09T18:21:01.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.187 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9500b4300 con 0x7fa95008fba0 2021-04-09T18:21:01.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.187 7fa946ffd700 1 -- 172.21.15.32:0/3425795887 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa948009450 con 0x7fa95008fba0 2021-04-09T18:21:01.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.187 7fa946ffd700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa938018930 0x7fa93801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:01.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.187 7fa946ffd700 1 -- 172.21.15.32:0/3425795887 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa948034f80 con 0x7fa95008fba0 2021-04-09T18:21:01.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.188 7fa956105700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa938018930 0x7fa93801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:01.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.188 7fa956105700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa938018930 0x7fa93801ae10 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:01.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.207 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa950182d70 con 0x7fa95008fba0 2021-04-09T18:21:01.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.211 7fa946ffd700 1 -- 172.21.15.32:0/3425795887 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa948000f90 con 0x7fa95008fba0 2021-04-09T18:21:01.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.401 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa95018c760 con 0x7fa95008fba0 2021-04-09T18:21:01.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.402 7fa946ffd700 1 -- 172.21.15.32:0/3425795887 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fa948005820 con 0x7fa95008fba0 2021-04-09T18:21:01.450 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:01.450 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:01.451 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:01.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa938018930 msgr2=0x7fa93801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:01.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa938018930 0x7fa93801ae10 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95008fba0 msgr2=0x7fa95008ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:01.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95008fba0 0x7fa95008ffa0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7fa948009f90 tx=0x7fa94800bc40).stop 2021-04-09T18:21:01.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 shutdown_connections 2021-04-09T18:21:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa938018930 0x7fa93801ae10 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95008fba0 0x7fa95008ffa0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95018bd40 0x7fa95018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 --2- 172.21.15.32:0/3425795887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9501aca20 0x7fa9501ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:01.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 >> 172.21.15.32:0/3425795887 conn(0x7fa95019b460 msgr2=0x7fa95019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:01.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 shutdown_connections 2021-04-09T18:21:01.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:01.404 7fa9582e9700 1 -- 172.21.15.32:0/3425795887 wait complete. 2021-04-09T18:21:01.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:01.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:02.867 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:03.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.982 7ff10717f700 1 Processor -- start 2021-04-09T18:21:03.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.982 7ff10717f700 1 -- start start 2021-04-09T18:21:03.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff10717f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff10010d010 0x7ff1000d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:03.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff10717f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10010d480 con 0x7ff1000d8490 2021-04-09T18:21:03.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff10717f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10010e1f0 con 0x7ff1000d1da0 2021-04-09T18:21:03.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff10717f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1000d8d30 con 0x7ff10010d010 2021-04-09T18:21:03.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff104f9b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff10010d010 0x7ff1000d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:03.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff10579c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff1000d8490 0x7ff10010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57634/0 (socket says 172.21.15.32:57634) 2021-04-09T18:21:03.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff10579c700 1 -- 172.21.15.32:0/152945280 learned_addr learned my addr 172.21.15.32:0/152945280 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:03.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff104f9b700 1 -- 172.21.15.32:0/152945280 >> v1:172.21.15.86:6789/0 conn(0x7ff1000d1da0 legacy=0x7ff1000d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:03.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff104f9b700 1 -- 172.21.15.32:0/152945280 >> v1:172.21.15.32:6789/0 conn(0x7ff1000d8490 legacy=0x7ff10010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:03.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff0ff7fe700 1 -- 172.21.15.32:0/152945280 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460729580 0 0) 0x7ff0f80009b0 con 0x7ff1000d8490 2021-04-09T18:21:03.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.983 7ff104f9b700 1 -- 172.21.15.32:0/152945280 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1000d9220 con 0x7ff10010d010 2021-04-09T18:21:03.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.984 7ff104f9b700 1 --2- 172.21.15.32:0/152945280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff10010d010 0x7ff1000d1780 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7ff0e8006650 tx=0x7ff0e8009a00).ready entity=mon.0 client_cookie=afd7fd3bdb07a8e8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:03.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.984 7ff0ff7fe700 1 -- 172.21.15.32:0/152945280 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0e8009df0 con 0x7ff10010d010 2021-04-09T18:21:03.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.984 7ff0ff7fe700 1 -- 172.21.15.32:0/152945280 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e80009b0 con 0x7ff10010d010 2021-04-09T18:21:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.984 7ff0ff7fe700 1 -- 172.21.15.32:0/152945280 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0e800c2b0 con 0x7ff10010d010 2021-04-09T18:21:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.984 7ff10717f700 1 -- 172.21.15.32:0/152945280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff10010d010 msgr2=0x7ff1000d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:03.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.984 7ff10717f700 1 --2- 172.21.15.32:0/152945280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff10010d010 0x7ff1000d1780 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7ff0e8006650 tx=0x7ff0e8009a00).stop 2021-04-09T18:21:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.985 7ff10717f700 1 -- 172.21.15.32:0/152945280 shutdown_connections 2021-04-09T18:21:03.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.985 7ff10717f700 1 --2- 172.21.15.32:0/152945280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff10010d010 0x7ff1000d1780 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.985 7ff10717f700 1 -- 172.21.15.32:0/152945280 >> 172.21.15.32:0/152945280 conn(0x7ff1000ce5f0 msgr2=0x7ff1000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:03.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.985 7ff10717f700 1 -- 172.21.15.32:0/152945280 shutdown_connections 2021-04-09T18:21:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:02.985 7ff10717f700 1 -- 172.21.15.32:0/152945280 wait complete. 2021-04-09T18:21:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.005 7ff10717f700 1 Processor -- start 2021-04-09T18:21:03.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.005 7ff10717f700 1 -- start start 2021-04-09T18:21:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.005 7ff10717f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1001ac5c0 0x7ff1001ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:03.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.005 7ff10717f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1001a83d0 0x7ff1001a87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.005 7ff10717f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1001aaa40 0x7ff1001aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:03.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.005 7ff10717f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1001932a0 con 0x7ff1001ac5c0 2021-04-09T18:21:03.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff10717f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1001a8df0 con 0x7ff1001aaa40 2021-04-09T18:21:03.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff10717f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000af1f0 con 0x7ff1001a83d0 2021-04-09T18:21:03.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff104f9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1001ac5c0 0x7ff1001ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:03.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff104f9b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1001ac5c0 0x7ff1001ac9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48048/0 (socket says 172.21.15.32:48048) 2021-04-09T18:21:03.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff104f9b700 1 -- 172.21.15.32:0/2279336842 learned_addr learned my addr 172.21.15.32:0/2279336842 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff10579c700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1001aaa40 0x7ff1001aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:03.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff0fffff700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1001a83d0 0x7ff1001a87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:03.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff104f9b700 1 -- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1001a83d0 msgr2=0x7ff1001a87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:03.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff104f9b700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1001a83d0 0x7ff1001a87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:03.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff104f9b700 1 -- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1001aaa40 msgr2=0x7ff1001aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:03.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff104f9b700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1001aaa40 0x7ff1001aae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:03.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff104f9b700 1 -- 172.21.15.32:0/2279336842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1001ab460 con 0x7ff1001ac5c0 2021-04-09T18:21:03.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff0fffff700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1001a83d0 0x7ff1001a87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:03.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff10579c700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1001aaa40 0x7ff1001aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:03.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff104f9b700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1001ac5c0 0x7ff1001ac9c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7ff0e80096a0 tx=0x7ff0e800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:03.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff0fd7fa700 1 -- 172.21.15.32:0/2279336842 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0e800bd80 con 0x7ff1001ac5c0 2021-04-09T18:21:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff0fd7fa700 1 -- 172.21.15.32:0/2279336842 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e8006030 con 0x7ff1001ac5c0 2021-04-09T18:21:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff0fd7fa700 1 -- 172.21.15.32:0/2279336842 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0e8005560 con 0x7ff1001ac5c0 2021-04-09T18:21:03.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff10717f700 1 -- 172.21.15.32:0/2279336842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1001ab460 con 0x7ff1001ac5c0 2021-04-09T18:21:03.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.006 7ff10717f700 1 -- 172.21.15.32:0/2279336842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1001ab460 con 0x7ff1001ac5c0 2021-04-09T18:21:03.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.007 7ff0fd7fa700 1 -- 172.21.15.32:0/2279336842 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff0e80052e0 con 0x7ff1001ac5c0 2021-04-09T18:21:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.007 7ff0fd7fa700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0e40188c0 0x7ff0e401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.007 7ff0fd7fa700 1 -- 172.21.15.32:0/2279336842 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff0e80327b0 con 0x7ff1001ac5c0 2021-04-09T18:21:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.007 7ff0fffff700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0e40188c0 0x7ff0e401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.008 7ff0fffff700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0e40188c0 0x7ff0e401adc0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:03.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.027 7ff10717f700 1 -- 172.21.15.32:0/2279336842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1001a7ea0 con 0x7ff1001ac5c0 2021-04-09T18:21:03.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.030 7ff0fd7fa700 1 -- 172.21.15.32:0/2279336842 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff0e8025f00 con 0x7ff1001ac5c0 2021-04-09T18:21:03.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.222 7ff10717f700 1 -- 172.21.15.32:0/2279336842 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1001a7ea0 con 0x7ff1001ac5c0 2021-04-09T18:21:03.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.222 7ff0fd7fa700 1 -- 172.21.15.32:0/2279336842 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7ff0e8025f00 con 0x7ff1001ac5c0 2021-04-09T18:21:03.272 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:03.272 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:03.273 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:03.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 -- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0e40188c0 msgr2=0x7ff0e401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:03.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0e40188c0 0x7ff0e401adc0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:03.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 -- 172.21.15.32:0/2279336842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1001ac5c0 msgr2=0x7ff1001ac9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:03.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1001ac5c0 0x7ff1001ac9c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7ff0e80096a0 tx=0x7ff0e800b990).stop 2021-04-09T18:21:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 -- 172.21.15.32:0/2279336842 shutdown_connections 2021-04-09T18:21:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0e40188c0 0x7ff0e401adc0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1001aaa40 0x7ff1001aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:03.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1001a83d0 0x7ff1001a87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:03.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 --2- 172.21.15.32:0/2279336842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1001ac5c0 0x7ff1001ac9c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:03.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 -- 172.21.15.32:0/2279336842 >> 172.21.15.32:0/2279336842 conn(0x7ff1001a9b80 msgr2=0x7ff100098040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:03.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 -- 172.21.15.32:0/2279336842 shutdown_connections 2021-04-09T18:21:03.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:03.225 7ff10717f700 1 -- 172.21.15.32:0/2279336842 wait complete. 2021-04-09T18:21:03.687 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:03.688 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:04.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.803 7fdc064f0700 1 Processor -- start 2021-04-09T18:21:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.803 7fdc064f0700 1 -- start start 2021-04-09T18:21:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.803 7fdc064f0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdc0010da00 0x7fdc0010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.803 7fdc064f0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0010d480 con 0x7fdc000d7ca0 2021-04-09T18:21:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdc064f0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc000d1f70 con 0x7fdc0010d010 2021-04-09T18:21:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdc064f0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc000d8d30 con 0x7fdc0010da00 2021-04-09T18:21:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbf7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdc000d7ca0 0x7fdc000d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57648/0 (socket says 172.21.15.32:57648) 2021-04-09T18:21:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbf7fff700 1 -- 172.21.15.32:0/1510359809 learned_addr learned my addr 172.21.15.32:0/1510359809 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:04.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdc04b0d700 1 --2- 172.21.15.32:0/1510359809 >> v2:172.21.15.86:3300/0 conn(0x7fdc0010da00 0x7fdc0010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971202078 0 0) 0x7fdbe8000ce0 con 0x7fdc000d7ca0 2021-04-09T18:21:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbec002fe0 con 0x7fdc000d7ca0 2021-04-09T18:21:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2910727267 0 0) 0x7fdbe80010e0 con 0x7fdc000d7ca0 2021-04-09T18:21:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 >> v2:172.21.15.86:3300/0 conn(0x7fdc0010da00 msgr2=0x7fdc0010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:04.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 --2- 172.21.15.32:0/1510359809 >> v2:172.21.15.86:3300/0 conn(0x7fdc0010da00 0x7fdc0010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 >> v1:172.21.15.86:6789/0 conn(0x7fdc0010d010 legacy=0x7fdc000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc000d9220 con 0x7fdc000d7ca0 2021-04-09T18:21:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fdbe8001600 con 0x7fdc000d7ca0 2021-04-09T18:21:04.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 >> v1:172.21.15.32:6789/0 conn(0x7fdc000d7ca0 legacy=0x7fdc000d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbec003a40 0x7fdbec005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbec006790 0x7fdbec008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbec009590 0x7fdbec00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec00c300 con 0x7fdbec003a40 2021-04-09T18:21:04.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec00c520 con 0x7fdbec009590 2021-04-09T18:21:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec050750 con 0x7fdbec006790 2021-04-09T18:21:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbfffff700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbec003a40 0x7fdbec005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdbf7fff700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbec006790 0x7fdbec008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.804 7fdc04b0d700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbec009590 0x7fdbec00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:04.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbfffff700 1 -- 172.21.15.32:0/1510359809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbec006790 msgr2=0x7fdbec008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:04.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbfffff700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbec006790 0x7fdbec008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:04.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbfffff700 1 -- 172.21.15.32:0/1510359809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbec009590 msgr2=0x7fdbec00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:04.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbfffff700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbec009590 0x7fdbec00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:04.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbfffff700 1 -- 172.21.15.32:0/1510359809 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec0509b0 con 0x7fdbec003a40 2021-04-09T18:21:04.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdc04b0d700 1 -- 172.21.15.32:0/1510359809 reap_dead start 2021-04-09T18:21:04.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbf7fff700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbec006790 0x7fdbec008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:04.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbfffff700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbec003a40 0x7fdbec005ef0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fdbf0006ab0 tx=0x7fdbf0009b40).ready entity=mon.0 client_cookie=b14593a758ae27f2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbf000b2e0 con 0x7fdbec003a40 2021-04-09T18:21:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf000b660 con 0x7fdbec003a40 2021-04-09T18:21:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdbff7fe700 1 -- 172.21.15.32:0/1510359809 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbf000bb40 con 0x7fdbec003a40 2021-04-09T18:21:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdc064f0700 1 -- 172.21.15.32:0/1510359809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbec003a40 msgr2=0x7fdbec005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:04.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.805 7fdc064f0700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbec003a40 0x7fdbec005ef0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fdbf0006ab0 tx=0x7fdbf0009b40).stop 2021-04-09T18:21:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.806 7fdc064f0700 1 -- 172.21.15.32:0/1510359809 shutdown_connections 2021-04-09T18:21:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.806 7fdc064f0700 1 --2- 172.21.15.32:0/1510359809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbec003a40 0x7fdbec005ef0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.806 7fdc064f0700 1 -- 172.21.15.32:0/1510359809 >> 172.21.15.32:0/1510359809 conn(0x7fdc000ce5f0 msgr2=0x7fdc000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.806 7fdc064f0700 1 -- 172.21.15.32:0/1510359809 shutdown_connections 2021-04-09T18:21:04.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.806 7fdc064f0700 1 -- 172.21.15.32:0/1510359809 wait complete. 2021-04-09T18:21:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.826 7fdc064f0700 1 Processor -- start 2021-04-09T18:21:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.826 7fdc064f0700 1 -- start start 2021-04-09T18:21:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc064f0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc001ac670 0x7fdc001a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:04.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc064f0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc001a5dc0 0x7fdc000ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc064f0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc001aca70 0x7fdc000ba800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc064f0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc001a5220 con 0x7fdc001aca70 2021-04-09T18:21:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc064f0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0018e100 con 0x7fdc001ac670 2021-04-09T18:21:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc064f0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0019ccc0 con 0x7fdc001a5dc0 2021-04-09T18:21:04.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc04b0d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc001aca70 0x7fdc000ba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:04.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc04b0d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc001aca70 0x7fdc000ba800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48072/0 (socket says 172.21.15.32:48072) 2021-04-09T18:21:04.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc04b0d700 1 -- 172.21.15.32:0/3861637970 learned_addr learned my addr 172.21.15.32:0/3861637970 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:04.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdbfffff700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc001ac670 0x7fdc001a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:04.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdbf7fff700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc001a5dc0 0x7fdc000ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:04.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc04b0d700 1 -- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc001a5dc0 msgr2=0x7fdc000ba1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:04.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc04b0d700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc001a5dc0 0x7fdc000ba1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:04.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc04b0d700 1 -- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc001ac670 msgr2=0x7fdc001a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:04.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc04b0d700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc001ac670 0x7fdc001a57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:04.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc04b0d700 1 -- 172.21.15.32:0/3861637970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc000bae20 con 0x7fdc001aca70 2021-04-09T18:21:04.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc04b0d700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc001aca70 0x7fdc000ba800 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fdbf800b910 tx=0x7fdbf800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:04.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdbfd7fa700 1 -- 172.21.15.32:0/3861637970 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbf8000f90 con 0x7fdc001aca70 2021-04-09T18:21:04.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdbfd7fa700 1 -- 172.21.15.32:0/3861637970 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf8006720 con 0x7fdc001aca70 2021-04-09T18:21:04.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc000bae20 con 0x7fdc001aca70 2021-04-09T18:21:04.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdbfd7fa700 1 -- 172.21.15.32:0/3861637970 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbf8001470 con 0x7fdc001aca70 2021-04-09T18:21:04.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.827 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc000bae20 con 0x7fdc001aca70 2021-04-09T18:21:04.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.828 7fdbfd7fa700 1 -- 172.21.15.32:0/3861637970 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdbf8000df0 con 0x7fdc001aca70 2021-04-09T18:21:04.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.828 7fdbfd7fa700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdbe40188c0 0x7fdbe401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:04.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.828 7fdbfd7fa700 1 -- 172.21.15.32:0/3861637970 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdbf8032b60 con 0x7fdc001aca70 2021-04-09T18:21:04.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.828 7fdbfffff700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdbe40188c0 0x7fdbe401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:04.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.829 7fdbfffff700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdbe40188c0 0x7fdbe401adc0 crc :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:04.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.847 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc00182d10 con 0x7fdc001aca70 2021-04-09T18:21:04.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:04.851 7fdbfd7fa700 1 -- 172.21.15.32:0/3861637970 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdbf800bdf0 con 0x7fdc001aca70 2021-04-09T18:21:05.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.042 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc00182d10 con 0x7fdc001aca70 2021-04-09T18:21:05.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.042 7fdbfd7fa700 1 -- 172.21.15.32:0/3861637970 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fdbf8000df0 con 0x7fdc001aca70 2021-04-09T18:21:05.091 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:05.091 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:05.091 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:05.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdbe40188c0 msgr2=0x7fdbe401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:05.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdbe40188c0 0x7fdbe401adc0 crc :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:05.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc001aca70 msgr2=0x7fdc000ba800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:05.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc001aca70 0x7fdc000ba800 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fdbf800b910 tx=0x7fdbf800bcb0).stop 2021-04-09T18:21:05.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 shutdown_connections 2021-04-09T18:21:05.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdbe40188c0 0x7fdbe401adc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:05.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc001aca70 0x7fdc000ba800 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:05.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc001a5dc0 0x7fdc000ba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:05.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 --2- 172.21.15.32:0/3861637970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc001ac670 0x7fdc001a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:05.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 >> 172.21.15.32:0/3861637970 conn(0x7fdc0019b890 msgr2=0x7fdc001a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:05.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 shutdown_connections 2021-04-09T18:21:05.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:05.045 7fdc064f0700 1 -- 172.21.15.32:0/3861637970 wait complete. 2021-04-09T18:21:05.506 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:05.507 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:06.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:21:06.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:21:06.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:06.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.611 7fd16549c700 1 Processor -- start 2021-04-09T18:21:06.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.611 7fd16549c700 1 -- start start 2021-04-09T18:21:06.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd16549c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd16010d010 0x7fd1600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:06.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd16549c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16010d480 con 0x7fd1600d1da0 2021-04-09T18:21:06.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd16549c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16010e1f0 con 0x7fd1600d8490 2021-04-09T18:21:06.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd16549c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1600d8d30 con 0x7fd16010d010 2021-04-09T18:21:06.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15e87c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd1600d1da0 0x7fd1600d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57666/0 (socket says 172.21.15.32:57666) 2021-04-09T18:21:06.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15e87c700 1 -- 172.21.15.32:0/3380206155 learned_addr learned my addr 172.21.15.32:0/3380206155 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:06.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15f07d700 1 --2- 172.21.15.32:0/3380206155 >> v2:172.21.15.86:3300/0 conn(0x7fd16010d010 0x7fd1600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:06.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15e07b700 1 -- 172.21.15.32:0/3380206155 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128145840 0 0) 0x7fd148000ce0 con 0x7fd1600d1da0 2021-04-09T18:21:06.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15e07b700 1 -- 172.21.15.32:0/3380206155 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd14c002fe0 con 0x7fd1600d1da0 2021-04-09T18:21:06.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15e07b700 1 -- 172.21.15.32:0/3380206155 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494673283 0 0) 0x7fd1500009b0 con 0x7fd1600d8490 2021-04-09T18:21:06.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15e07b700 1 -- 172.21.15.32:0/3380206155 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd14c003210 con 0x7fd1600d8490 2021-04-09T18:21:06.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15f07d700 1 -- 172.21.15.32:0/3380206155 >> v1:172.21.15.86:6789/0 conn(0x7fd1600d8490 legacy=0x7fd16010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:06.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15f07d700 1 -- 172.21.15.32:0/3380206155 >> v1:172.21.15.32:6789/0 conn(0x7fd1600d1da0 legacy=0x7fd1600d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:06.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15f07d700 1 -- 172.21.15.32:0/3380206155 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1600d9220 con 0x7fd16010d010 2021-04-09T18:21:06.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15f07d700 1 --2- 172.21.15.32:0/3380206155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd16010d010 0x7fd1600d1780 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fd1580069d0 tx=0x7fd158009e40).ready entity=mon.1 client_cookie=18eb423616e320d2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:06.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15e07b700 1 -- 172.21.15.32:0/3380206155 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd15800b510 con 0x7fd16010d010 2021-04-09T18:21:06.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15e07b700 1 -- 172.21.15.32:0/3380206155 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15800b890 con 0x7fd16010d010 2021-04-09T18:21:06.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.612 7fd15e07b700 1 -- 172.21.15.32:0/3380206155 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd15800bd70 con 0x7fd16010d010 2021-04-09T18:21:06.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.613 7fd16549c700 1 -- 172.21.15.32:0/3380206155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd16010d010 msgr2=0x7fd1600d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:06.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.613 7fd16549c700 1 --2- 172.21.15.32:0/3380206155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd16010d010 0x7fd1600d1780 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fd1580069d0 tx=0x7fd158009e40).stop 2021-04-09T18:21:06.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.613 7fd16549c700 1 -- 172.21.15.32:0/3380206155 shutdown_connections 2021-04-09T18:21:06.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.613 7fd16549c700 1 --2- 172.21.15.32:0/3380206155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd16010d010 0x7fd1600d1780 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:06.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.613 7fd16549c700 1 -- 172.21.15.32:0/3380206155 >> 172.21.15.32:0/3380206155 conn(0x7fd1600ce5f0 msgr2=0x7fd1600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:06.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.613 7fd16549c700 1 -- 172.21.15.32:0/3380206155 shutdown_connections 2021-04-09T18:21:06.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.613 7fd16549c700 1 -- 172.21.15.32:0/3380206155 wait complete. 2021-04-09T18:21:06.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.633 7fd16549c700 1 Processor -- start 2021-04-09T18:21:06.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.633 7fd16549c700 1 -- start start 2021-04-09T18:21:06.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd16549c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd160179610 0x7fd160179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:06.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd16549c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd16017a6e0 0x7fd16017aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:06.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd16549c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd16017b100 0x7fd16008db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:06.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd16549c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16017d720 con 0x7fd16017b100 2021-04-09T18:21:06.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd16549c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd160185ba0 con 0x7fd16017a6e0 2021-04-09T18:21:06.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd16549c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16008e1b0 con 0x7fd160179610 2021-04-09T18:21:06.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15f87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd16017b100 0x7fd16008db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:06.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15f87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd16017b100 0x7fd16008db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48082/0 (socket says 172.21.15.32:48082) 2021-04-09T18:21:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15f87e700 1 -- 172.21.15.32:0/2793398402 learned_addr learned my addr 172.21.15.32:0/2793398402 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15e87c700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd16017a6e0 0x7fd16017aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15f07d700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd160179610 0x7fd160179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15f87e700 1 -- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd160179610 msgr2=0x7fd160179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15f87e700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd160179610 0x7fd160179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:06.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15f87e700 1 -- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd16017a6e0 msgr2=0x7fd16017aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:06.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15f87e700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd16017a6e0 0x7fd16017aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:06.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.634 7fd15f87e700 1 -- 172.21.15.32:0/2793398402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16008e3d0 con 0x7fd16017b100 2021-04-09T18:21:06.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.635 7fd15f87e700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd16017b100 0x7fd16008db90 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7fd15000b580 tx=0x7fd15000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:06.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.635 7fd157fff700 1 -- 172.21.15.32:0/2793398402 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd15000bd10 con 0x7fd16017b100 2021-04-09T18:21:06.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.635 7fd157fff700 1 -- 172.21.15.32:0/2793398402 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd150000e20 con 0x7fd16017b100 2021-04-09T18:21:06.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.635 7fd16549c700 1 -- 172.21.15.32:0/2793398402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16008e3d0 con 0x7fd16017b100 2021-04-09T18:21:06.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.635 7fd157fff700 1 -- 172.21.15.32:0/2793398402 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd150000940 con 0x7fd16017b100 2021-04-09T18:21:06.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.635 7fd16549c700 1 -- 172.21.15.32:0/2793398402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16008e3d0 con 0x7fd16017b100 2021-04-09T18:21:06.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.635 7fd157fff700 1 -- 172.21.15.32:0/2793398402 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd150006030 con 0x7fd16017b100 2021-04-09T18:21:06.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.635 7fd157fff700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd144018930 0x7fd14401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:06.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.636 7fd157fff700 1 -- 172.21.15.32:0/2793398402 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd150032a10 con 0x7fd16017b100 2021-04-09T18:21:06.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.636 7fd15f07d700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd144018930 0x7fd14401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:06.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.636 7fd15f07d700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd144018930 0x7fd14401ae10 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:06.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.656 7fd16549c700 1 -- 172.21.15.32:0/2793398402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1601b7010 con 0x7fd16017b100 2021-04-09T18:21:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.659 7fd157fff700 1 -- 172.21.15.32:0/2793398402 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd150000e00 con 0x7fd16017b100 2021-04-09T18:21:06.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.850 7fd16549c700 1 -- 172.21.15.32:0/2793398402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd16010d480 con 0x7fd16017b100 2021-04-09T18:21:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.850 7fd157fff700 1 -- 172.21.15.32:0/2793398402 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fd150000e00 con 0x7fd16017b100 2021-04-09T18:21:06.902 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:06.903 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:06.904 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:06.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.852 7fd16549c700 1 -- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd144018930 msgr2=0x7fd14401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:06.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.852 7fd16549c700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd144018930 0x7fd14401ae10 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:06.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.852 7fd16549c700 1 -- 172.21.15.32:0/2793398402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd16017b100 msgr2=0x7fd16008db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:06.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.852 7fd16549c700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd16017b100 0x7fd16008db90 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7fd15000b580 tx=0x7fd15000b920).stop 2021-04-09T18:21:06.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.852 7fd16549c700 1 -- 172.21.15.32:0/2793398402 shutdown_connections 2021-04-09T18:21:06.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.852 7fd16549c700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd144018930 0x7fd14401ae10 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:06.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.852 7fd16549c700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd16017b100 0x7fd16008db90 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:06.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.852 7fd16549c700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd16017a6e0 0x7fd16017aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:06.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.852 7fd16549c700 1 --2- 172.21.15.32:0/2793398402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd160179610 0x7fd160179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:06.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.853 7fd16549c700 1 -- 172.21.15.32:0/2793398402 >> 172.21.15.32:0/2793398402 conn(0x7fd160190550 msgr2=0x7fd1601b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:06.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.853 7fd16549c700 1 -- 172.21.15.32:0/2793398402 shutdown_connections 2021-04-09T18:21:06.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:06.853 7fd16549c700 1 -- 172.21.15.32:0/2793398402 wait complete. 2021-04-09T18:21:07.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:07.316 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:08.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:08.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.420 7fed839fe700 1 Processor -- start 2021-04-09T18:21:08.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.420 7fed839fe700 1 -- start start 2021-04-09T18:21:08.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed839fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fed7c10d010 0x7fed7c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:08.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed839fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c0d8490 0x7fed7c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:08.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed839fe700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c10d480 con 0x7fed7c0d1da0 2021-04-09T18:21:08.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed839fe700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7c0d8d30 con 0x7fed7c10d010 2021-04-09T18:21:08.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed839fe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0d9220 con 0x7fed7c0d8490 2021-04-09T18:21:08.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed8181a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fed7c10d010 0x7fed7c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:08.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed8181a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fed7c10d010 0x7fed7c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48090/0 (socket says 172.21.15.32:48090) 2021-04-09T18:21:08.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed8181a700 1 -- 172.21.15.32:0/1463549639 learned_addr learned my addr 172.21.15.32:0/1463549639 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:08.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed8201b700 1 --2- 172.21.15.32:0/1463549639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c0d8490 0x7fed7c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:08.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed8181a700 1 -- 172.21.15.32:0/1463549639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c0d8490 msgr2=0x7fed7c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:08.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed8181a700 1 --2- 172.21.15.32:0/1463549639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c0d8490 0x7fed7c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed8181a700 1 -- 172.21.15.32:0/1463549639 >> v1:172.21.15.86:6789/0 conn(0x7fed7c0d1da0 legacy=0x7fed7c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:08.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed8181a700 1 -- 172.21.15.32:0/1463549639 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c10e020 con 0x7fed7c10d010 2021-04-09T18:21:08.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.421 7fed8181a700 1 --2- 172.21.15.32:0/1463549639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c10d010 0x7fed7c0d1780 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7fed78006700 tx=0x7fed78009b00).ready entity=mon.0 client_cookie=78f37ea990351dce server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:08.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.422 7fed80818700 1 -- 172.21.15.32:0/1463549639 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed7800c0c0 con 0x7fed7c10d010 2021-04-09T18:21:08.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.422 7fed80818700 1 -- 172.21.15.32:0/1463549639 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7800c400 con 0x7fed7c10d010 2021-04-09T18:21:08.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.422 7fed80818700 1 -- 172.21.15.32:0/1463549639 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed7800c8e0 con 0x7fed7c10d010 2021-04-09T18:21:08.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.422 7fed839fe700 1 -- 172.21.15.32:0/1463549639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c10d010 msgr2=0x7fed7c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:08.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.422 7fed839fe700 1 --2- 172.21.15.32:0/1463549639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c10d010 0x7fed7c0d1780 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7fed78006700 tx=0x7fed78009b00).stop 2021-04-09T18:21:08.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.422 7fed839fe700 1 -- 172.21.15.32:0/1463549639 shutdown_connections 2021-04-09T18:21:08.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.422 7fed839fe700 1 --2- 172.21.15.32:0/1463549639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c0d8490 0x7fed7c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.422 7fed839fe700 1 --2- 172.21.15.32:0/1463549639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c10d010 0x7fed7c0d1780 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.422 7fed839fe700 1 -- 172.21.15.32:0/1463549639 >> 172.21.15.32:0/1463549639 conn(0x7fed7c0ce5f0 msgr2=0x7fed7c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.423 7fed839fe700 1 -- 172.21.15.32:0/1463549639 shutdown_connections 2021-04-09T18:21:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.423 7fed839fe700 1 -- 172.21.15.32:0/1463549639 wait complete. 2021-04-09T18:21:08.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.443 7fed839fe700 1 Processor -- start 2021-04-09T18:21:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.443 7fed839fe700 1 -- start start 2021-04-09T18:21:08.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.443 7fed839fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c0b42d0 0x7fed7c183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.443 7fed839fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed7c1ac090 0x7fed7c1ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.443 7fed839fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c184d30 0x7fed7c185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:08.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.443 7fed839fe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0a1a90 con 0x7fed7c0b42d0 2021-04-09T18:21:08.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.443 7fed839fe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c1acab0 con 0x7fed7c1ac090 2021-04-09T18:21:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.443 7fed839fe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c185a00 con 0x7fed7c184d30 2021-04-09T18:21:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8181a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c0b42d0 0x7fed7c183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:08.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8181a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c0b42d0 0x7fed7c183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48098/0 (socket says 172.21.15.32:48098) 2021-04-09T18:21:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8181a700 1 -- 172.21.15.32:0/1694373474 learned_addr learned my addr 172.21.15.32:0/1694373474 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8201b700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c184d30 0x7fed7c185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:08.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed81019700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed7c1ac090 0x7fed7c1ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:08.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8181a700 1 -- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c184d30 msgr2=0x7fed7c185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:08.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8181a700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c184d30 0x7fed7c185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8181a700 1 -- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed7c1ac090 msgr2=0x7fed7c1ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8181a700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed7c1ac090 0x7fed7c1ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8181a700 1 -- 172.21.15.32:0/1694373474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c185750 con 0x7fed7c0b42d0 2021-04-09T18:21:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed8181a700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c0b42d0 0x7fed7c183520 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fed7800b660 tx=0x7fed7800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:08.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed727fc700 1 -- 172.21.15.32:0/1694373474 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed78009440 con 0x7fed7c0b42d0 2021-04-09T18:21:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed727fc700 1 -- 172.21.15.32:0/1694373474 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed78006390 con 0x7fed7c0b42d0 2021-04-09T18:21:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed839fe700 1 -- 172.21.15.32:0/1694373474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7c185750 con 0x7fed7c0b42d0 2021-04-09T18:21:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed727fc700 1 -- 172.21.15.32:0/1694373474 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed78002030 con 0x7fed7c0b42d0 2021-04-09T18:21:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.444 7fed839fe700 1 -- 172.21.15.32:0/1694373474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c185750 con 0x7fed7c0b42d0 2021-04-09T18:21:08.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.445 7fed727fc700 1 -- 172.21.15.32:0/1694373474 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fed780093e0 con 0x7fed7c0b42d0 2021-04-09T18:21:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.445 7fed727fc700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:08.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.445 7fed727fc700 1 -- 172.21.15.32:0/1694373474 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fed780329f0 con 0x7fed7c0b42d0 2021-04-09T18:21:08.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.445 7fed81019700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:08.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.445 7fed81019700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401ae10 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:08.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.465 7fed839fe700 1 -- 172.21.15.32:0/1694373474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7c182d10 con 0x7fed7c0b42d0 2021-04-09T18:21:08.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.468 7fed727fc700 1 -- 172.21.15.32:0/1694373474 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fed780090c0 con 0x7fed7c0b42d0 2021-04-09T18:21:08.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.660 7fed839fe700 1 -- 172.21.15.32:0/1694373474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed7c10d480 con 0x7fed7c0b42d0 2021-04-09T18:21:08.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.660 7fed727fc700 1 -- 172.21.15.32:0/1694373474 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fed7800bdf0 con 0x7fed7c0b42d0 2021-04-09T18:21:08.709 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:08.709 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:08.710 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 -- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 msgr2=0x7fed6401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:08.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401ae10 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 -- 172.21.15.32:0/1694373474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c0b42d0 msgr2=0x7fed7c183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:08.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c0b42d0 0x7fed7c183520 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fed7800b660 tx=0x7fed7800ba00).stop 2021-04-09T18:21:08.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 -- 172.21.15.32:0/1694373474 shutdown_connections 2021-04-09T18:21:08.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401ae10 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed7c184d30 0x7fed7c185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed7c1ac090 0x7fed7c1ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 --2- 172.21.15.32:0/1694373474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7c0b42d0 0x7fed7c183520 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:08.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 -- 172.21.15.32:0/1694373474 >> 172.21.15.32:0/1694373474 conn(0x7fed7c19b890 msgr2=0x7fed7c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:08.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 -- 172.21.15.32:0/1694373474 shutdown_connections 2021-04-09T18:21:08.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:08.663 7fed839fe700 1 -- 172.21.15.32:0/1694373474 wait complete. 2021-04-09T18:21:09.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:09.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:10.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:10.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.240 7f74a1f58700 1 Processor -- start 2021-04-09T18:21:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.240 7f74a1f58700 1 -- start start 2021-04-09T18:21:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.241 7f74a1f58700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c0cffa0 0x7f749c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.241 7f74a1f58700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f749c0d6c80 0x7f749c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:10.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.241 7f74a1f58700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c0d7280 con 0x7f749c0d7800 2021-04-09T18:21:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.241 7f74a1f58700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c0d7080 con 0x7f749c0d6c80 2021-04-09T18:21:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.241 7f74a1f58700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0d8580 con 0x7f749c0cffa0 2021-04-09T18:21:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.241 7f749b7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c0cffa0 0x7f749c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:10.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.241 7f749bfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f749c0d6c80 0x7f749c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.241 7f749affd700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f749c0d7800 0x7f749c0d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55624/0 (socket says 172.21.15.32:55624) 2021-04-09T18:21:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.241 7f749affd700 1 -- 172.21.15.32:0/68151386 learned_addr learned my addr 172.21.15.32:0/68151386 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f749a7fc700 1 -- 172.21.15.32:0/68151386 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392041677 0 0) 0x7f74900009b0 con 0x7f749c0d7800 2021-04-09T18:21:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f749b7fe700 1 -- 172.21.15.32:0/68151386 >> v2:172.21.15.86:3300/0 conn(0x7f749c0d6c80 msgr2=0x7f749c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:10.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f749b7fe700 1 --2- 172.21.15.32:0/68151386 >> v2:172.21.15.86:3300/0 conn(0x7f749c0d6c80 0x7f749c10cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f749b7fe700 1 -- 172.21.15.32:0/68151386 >> v1:172.21.15.86:6789/0 conn(0x7f749c0d7800 legacy=0x7f749c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:10.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f749b7fe700 1 -- 172.21.15.32:0/68151386 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c0d91e0 con 0x7f749c0cffa0 2021-04-09T18:21:10.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f749b7fe700 1 --2- 172.21.15.32:0/68151386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c0cffa0 0x7f749c0d2440 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f748c006650 tx=0x7f748c009a80).ready entity=mon.2 client_cookie=80e9ffc501880db9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:10.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f749a7fc700 1 -- 172.21.15.32:0/68151386 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748c0009b0 con 0x7f749c0cffa0 2021-04-09T18:21:10.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f749a7fc700 1 -- 172.21.15.32:0/68151386 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748c00c0c0 con 0x7f749c0cffa0 2021-04-09T18:21:10.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f749a7fc700 1 -- 172.21.15.32:0/68151386 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748c00c510 con 0x7f749c0cffa0 2021-04-09T18:21:10.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f74a1f58700 1 -- 172.21.15.32:0/68151386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c0cffa0 msgr2=0x7f749c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:10.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.242 7f74a1f58700 1 --2- 172.21.15.32:0/68151386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c0cffa0 0x7f749c0d2440 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f748c006650 tx=0x7f748c009a80).stop 2021-04-09T18:21:10.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.243 7f74a1f58700 1 -- 172.21.15.32:0/68151386 shutdown_connections 2021-04-09T18:21:10.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.243 7f74a1f58700 1 --2- 172.21.15.32:0/68151386 >> v2:172.21.15.86:3300/0 conn(0x7f749c0d6c80 0x7f749c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.243 7f74a1f58700 1 --2- 172.21.15.32:0/68151386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c0cffa0 0x7f749c0d2440 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.243 7f74a1f58700 1 -- 172.21.15.32:0/68151386 >> 172.21.15.32:0/68151386 conn(0x7f749c0ce5c0 msgr2=0x7f749c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:10.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.243 7f74a1f58700 1 -- 172.21.15.32:0/68151386 shutdown_connections 2021-04-09T18:21:10.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.243 7f74a1f58700 1 -- 172.21.15.32:0/68151386 wait complete. 2021-04-09T18:21:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.263 7f74a1f58700 1 Processor -- start 2021-04-09T18:21:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.263 7f74a1f58700 1 -- start start 2021-04-09T18:21:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f74a1f58700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f749c0a80c0 0x7f749c17bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f74a1f58700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749c17b1d0 0x7f749c17b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f74a1f58700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c08dc50 0x7f749c08e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f74a1f58700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c17bc70 con 0x7f749c17b1d0 2021-04-09T18:21:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f74a1f58700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c08e670 con 0x7f749c0a80c0 2021-04-09T18:21:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f74a1f58700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c08e820 con 0x7f749c08dc50 2021-04-09T18:21:10.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749c17b1d0 0x7f749c17b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:10.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749c17b1d0 0x7f749c17b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48112/0 (socket says 172.21.15.32:48112) 2021-04-09T18:21:10.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749affd700 1 -- 172.21.15.32:0/999347478 learned_addr learned my addr 172.21.15.32:0/999347478 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:10.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749b7fe700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f749c0a80c0 0x7f749c17bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:10.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749bfff700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c08dc50 0x7f749c08e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:10.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749affd700 1 -- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c08dc50 msgr2=0x7f749c08e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:10.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749affd700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c08dc50 0x7f749c08e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749affd700 1 -- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f749c0a80c0 msgr2=0x7f749c17bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:10.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749affd700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f749c0a80c0 0x7f749c17bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749affd700 1 -- 172.21.15.32:0/999347478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c08c660 con 0x7f749c17b1d0 2021-04-09T18:21:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.264 7f749affd700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749c17b1d0 0x7f749c17b5d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f7490001c60 tx=0x7f749000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.265 7f7483fff700 1 -- 172.21.15.32:0/999347478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f749000acc0 con 0x7f749c17b1d0 2021-04-09T18:21:10.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.265 7f7483fff700 1 -- 172.21.15.32:0/999347478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7490004140 con 0x7f749c17b1d0 2021-04-09T18:21:10.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.265 7f74a1f58700 1 -- 172.21.15.32:0/999347478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c08c660 con 0x7f749c17b1d0 2021-04-09T18:21:10.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.265 7f7483fff700 1 -- 172.21.15.32:0/999347478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7490004620 con 0x7f749c17b1d0 2021-04-09T18:21:10.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.265 7f74a1f58700 1 -- 172.21.15.32:0/999347478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c08c660 con 0x7f749c17b1d0 2021-04-09T18:21:10.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.265 7f7483fff700 1 -- 172.21.15.32:0/999347478 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7490000dd0 con 0x7f749c17b1d0 2021-04-09T18:21:10.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.265 7f7483fff700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f747c018930 0x7f747c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.265 7f7483fff700 1 -- 172.21.15.32:0/999347478 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7490032a00 con 0x7f749c17b1d0 2021-04-09T18:21:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.266 7f749b7fe700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f747c018930 0x7f747c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:10.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.266 7f749b7fe700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f747c018930 0x7f747c01adc0 crc :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:10.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.285 7f74a1f58700 1 -- 172.21.15.32:0/999347478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c1b70c0 con 0x7f749c17b1d0 2021-04-09T18:21:10.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.288 7f7483fff700 1 -- 172.21.15.32:0/999347478 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f74900009b0 con 0x7f749c17b1d0 2021-04-09T18:21:10.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.481 7f74a1f58700 1 -- 172.21.15.32:0/999347478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f749c0d7280 con 0x7f749c17b1d0 2021-04-09T18:21:10.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.482 7f7483fff700 1 -- 172.21.15.32:0/999347478 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f7490000dd0 con 0x7f749c17b1d0 2021-04-09T18:21:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:10.532 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:10.533 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:10.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 -- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f747c018930 msgr2=0x7f747c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:10.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f747c018930 0x7f747c01adc0 crc :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 -- 172.21.15.32:0/999347478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749c17b1d0 msgr2=0x7f749c17b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:10.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749c17b1d0 0x7f749c17b5d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f7490001c60 tx=0x7f749000a8d0).stop 2021-04-09T18:21:10.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 -- 172.21.15.32:0/999347478 shutdown_connections 2021-04-09T18:21:10.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f747c018930 0x7f747c01adc0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f749c17b1d0 0x7f749c17b5d0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f749c08dc50 0x7f749c08e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 --2- 172.21.15.32:0/999347478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f749c0a80c0 0x7f749c17bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.484 7f74a1f58700 1 -- 172.21.15.32:0/999347478 >> 172.21.15.32:0/999347478 conn(0x7f749c189b00 msgr2=0x7f749c1921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.485 7f74a1f58700 1 -- 172.21.15.32:0/999347478 shutdown_connections 2021-04-09T18:21:10.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:10.485 7f74a1f58700 1 -- 172.21.15.32:0/999347478 wait complete. 2021-04-09T18:21:10.946 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:10.947 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:11.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:12.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.052 7fceb053d700 1 Processor -- start 2021-04-09T18:21:12.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.052 7fceb053d700 1 -- start start 2021-04-09T18:21:12.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.052 7fceb053d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcea810d010 0x7fcea80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:12.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.052 7fceb053d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea80d8490 0x7fcea810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:12.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.052 7fceb053d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea810d480 con 0x7fcea80d1da0 2021-04-09T18:21:12.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.052 7fceb053d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea80d8d30 con 0x7fcea810d010 2021-04-09T18:21:12.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.052 7fceb053d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80d9220 con 0x7fcea80d8490 2021-04-09T18:21:12.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceae359700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcea810d010 0x7fcea80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:12.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceadb58700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcea80d1da0 0x7fcea80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57712/0 (socket says 172.21.15.32:57712) 2021-04-09T18:21:12.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceadb58700 1 -- 172.21.15.32:0/886558605 learned_addr learned my addr 172.21.15.32:0/886558605 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:12.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceaeb5a700 1 --2- 172.21.15.32:0/886558605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea80d8490 0x7fcea810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:12.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceae359700 1 -- 172.21.15.32:0/886558605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea80d8490 msgr2=0x7fcea810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceae359700 1 --2- 172.21.15.32:0/886558605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea80d8490 0x7fcea810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:12.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceae359700 1 -- 172.21.15.32:0/886558605 >> v1:172.21.15.32:6789/0 conn(0x7fcea80d1da0 legacy=0x7fcea80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:12.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceae359700 1 -- 172.21.15.32:0/886558605 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea810e020 con 0x7fcea810d010 2021-04-09T18:21:12.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fcead357700 1 -- 172.21.15.32:0/886558605 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261050280 0 0) 0x7fce98000ce0 con 0x7fcea80d1da0 2021-04-09T18:21:12.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceaeb5a700 1 --2- 172.21.15.32:0/886558605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea80d8490 0x7fcea810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:12.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceae359700 1 --2- 172.21.15.32:0/886558605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea810d010 0x7fcea80d1780 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7fcea4006650 tx=0x7fcea4009a80).ready entity=mon.0 client_cookie=1ddd4925de65f745 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:12.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fcead357700 1 -- 172.21.15.32:0/886558605 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcea40009b0 con 0x7fcea810d010 2021-04-09T18:21:12.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fcead357700 1 -- 172.21.15.32:0/886558605 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea400c0c0 con 0x7fcea810d010 2021-04-09T18:21:12.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fcead357700 1 -- 172.21.15.32:0/886558605 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcea400c510 con 0x7fcea810d010 2021-04-09T18:21:12.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceb053d700 1 -- 172.21.15.32:0/886558605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea810d010 msgr2=0x7fcea80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:12.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.053 7fceb053d700 1 --2- 172.21.15.32:0/886558605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea810d010 0x7fcea80d1780 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7fcea4006650 tx=0x7fcea4009a80).stop 2021-04-09T18:21:12.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.054 7fceb053d700 1 -- 172.21.15.32:0/886558605 shutdown_connections 2021-04-09T18:21:12.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.054 7fceb053d700 1 --2- 172.21.15.32:0/886558605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea80d8490 0x7fcea810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:12.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.054 7fceb053d700 1 --2- 172.21.15.32:0/886558605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea810d010 0x7fcea80d1780 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:12.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.054 7fceb053d700 1 -- 172.21.15.32:0/886558605 >> 172.21.15.32:0/886558605 conn(0x7fcea80ce5f0 msgr2=0x7fcea80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:12.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.054 7fceb053d700 1 -- 172.21.15.32:0/886558605 shutdown_connections 2021-04-09T18:21:12.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.054 7fceb053d700 1 -- 172.21.15.32:0/886558605 wait complete. 2021-04-09T18:21:12.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.074 7fceb053d700 1 Processor -- start 2021-04-09T18:21:12.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.074 7fceb053d700 1 -- start start 2021-04-09T18:21:12.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceb053d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8185060 0x7fcea8185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:12.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceb053d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea8186c00 0x7fcea8187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:12.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceb053d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8191360 0x7fcea8191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceb053d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea81863a0 con 0x7fcea8185060 2021-04-09T18:21:12.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceb053d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea8185a60 con 0x7fcea8191360 2021-04-09T18:21:12.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceb053d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea8187620 con 0x7fcea8186c00 2021-04-09T18:21:12.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceae359700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8185060 0x7fcea8185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:12.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceae359700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8185060 0x7fcea8185d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48124/0 (socket says 172.21.15.32:48124) 2021-04-09T18:21:12.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceae359700 1 -- 172.21.15.32:0/89602400 learned_addr learned my addr 172.21.15.32:0/89602400 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:12.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceae359700 1 -- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea8186c00 msgr2=0x7fcea8187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:12.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceae359700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea8186c00 0x7fcea8187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:21:12.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceae359700 1 -- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8191360 msgr2=0x7fcea8191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:12.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceae359700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8191360 0x7fcea8191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:21:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceae359700 1 -- 172.21.15.32:0/89602400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea8185460 con 0x7fcea8185060 2021-04-09T18:21:12.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.075 7fceae359700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8185060 0x7fcea8185d80 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fcea400b530 tx=0x7fcea400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:12.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.076 7fce9effd700 1 -- 172.21.15.32:0/89602400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcea40093a0 con 0x7fcea8185060 2021-04-09T18:21:12.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.076 7fce9effd700 1 -- 172.21.15.32:0/89602400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea4006070 con 0x7fcea8185060 2021-04-09T18:21:12.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.076 7fce9effd700 1 -- 172.21.15.32:0/89602400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcea40052e0 con 0x7fcea8185060 2021-04-09T18:21:12.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.076 7fceb053d700 1 -- 172.21.15.32:0/89602400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea8185460 con 0x7fcea8185060 2021-04-09T18:21:12.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.076 7fceb053d700 1 -- 172.21.15.32:0/89602400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea8183c60 con 0x7fcea8185060 2021-04-09T18:21:12.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.076 7fce9effd700 1 -- 172.21.15.32:0/89602400 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcea4009300 con 0x7fcea8185060 2021-04-09T18:21:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.076 7fce9effd700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce900188d0 0x7fce9001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.076 7fce9effd700 1 -- 172.21.15.32:0/89602400 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcea4034bf0 con 0x7fcea8185060 2021-04-09T18:21:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.077 7fceadb58700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce900188d0 0x7fce9001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.077 7fceadb58700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce900188d0 0x7fce9001add0 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:12.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.096 7fceb053d700 1 -- 172.21.15.32:0/89602400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea81866d0 con 0x7fcea8185060 2021-04-09T18:21:12.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.100 7fce9effd700 1 -- 172.21.15.32:0/89602400 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcea4034e00 con 0x7fcea8185060 2021-04-09T18:21:12.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.292 7fceb053d700 1 -- 172.21.15.32:0/89602400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcea81866d0 con 0x7fcea8185060 2021-04-09T18:21:12.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.292 7fce9effd700 1 -- 172.21.15.32:0/89602400 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fcea4006050 con 0x7fcea8185060 2021-04-09T18:21:12.342 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:12.342 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:12.343 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.294 7fceb053d700 1 -- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce900188d0 msgr2=0x7fce9001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.294 7fceb053d700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce900188d0 0x7fce9001add0 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:12.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.294 7fceb053d700 1 -- 172.21.15.32:0/89602400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8185060 msgr2=0x7fcea8185d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.294 7fceb053d700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8185060 0x7fcea8185d80 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fcea400b530 tx=0x7fcea400b8d0).stop 2021-04-09T18:21:12.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.295 7fceb053d700 1 -- 172.21.15.32:0/89602400 shutdown_connections 2021-04-09T18:21:12.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.295 7fceb053d700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce900188d0 0x7fce9001add0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:12.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.295 7fceb053d700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea8191360 0x7fcea8191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:21:12.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.295 7fceb053d700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea8186c00 0x7fcea8187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:21:12.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.295 7fceb053d700 1 --2- 172.21.15.32:0/89602400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea8185060 0x7fcea8185d80 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:12.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.295 7fceb053d700 1 -- 172.21.15.32:0/89602400 >> 172.21.15.32:0/89602400 conn(0x7fcea81837d0 msgr2=0x7fcea8195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:12.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.295 7fceb053d700 1 -- 172.21.15.32:0/89602400 shutdown_connections 2021-04-09T18:21:12.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:12.295 7fceb053d700 1 -- 172.21.15.32:0/89602400 wait complete. 2021-04-09T18:21:12.764 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:12.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:13.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:13.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.878 7f9aec67d700 1 Processor -- start 2021-04-09T18:21:13.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.878 7f9aec67d700 1 -- start start 2021-04-09T18:21:13.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9aec67d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9ae410d010 0x7f9ae40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9aec67d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ae40d8490 0x7f9ae410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:13.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9aec67d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae410d480 con 0x7f9ae40d1da0 2021-04-09T18:21:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9aec67d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae40d8d30 con 0x7f9ae410d010 2021-04-09T18:21:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9aec67d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40d9220 con 0x7f9ae40d8490 2021-04-09T18:21:13.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9c98700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9ae40d1da0 0x7f9ae40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57726/0 (socket says 172.21.15.32:57726) 2021-04-09T18:21:13.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9c98700 1 -- 172.21.15.32:0/1284073601 learned_addr learned my addr 172.21.15.32:0/1284073601 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:13.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9aea499700 1 --2- 172.21.15.32:0/1284073601 >> v2:172.21.15.86:3300/0 conn(0x7f9ae410d010 0x7f9ae40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:13.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9aeac9a700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ae40d8490 0x7f9ae410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:13.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466725130 0 0) 0x7f9ad4000ce0 con 0x7f9ae40d1da0 2021-04-09T18:21:13.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad0002fe0 con 0x7f9ae40d1da0 2021-04-09T18:21:13.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 557564089 0 0) 0x7f9ad40010e0 con 0x7f9ae40d1da0 2021-04-09T18:21:13.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ae40d8490 msgr2=0x7f9ae410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:13.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ae40d8490 0x7f9ae410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:13.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 >> v2:172.21.15.86:3300/0 conn(0x7f9ae410d010 msgr2=0x7f9ae40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:13.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 --2- 172.21.15.32:0/1284073601 >> v2:172.21.15.86:3300/0 conn(0x7f9ae410d010 0x7f9ae40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:13.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae410e020 con 0x7f9ae40d1da0 2021-04-09T18:21:13.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f9ad4001600 con 0x7f9ae40d1da0 2021-04-09T18:21:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.879 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 >> v1:172.21.15.32:6789/0 conn(0x7f9ae40d1da0 legacy=0x7f9ae40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9497700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ad0003a60 0x7f9ad0005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:13.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9497700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ad004a760 0x7f9ad004cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:13.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9497700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ad004d500 0x7f9ad004fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:13.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00502d0 con 0x7f9ad004a760 2021-04-09T18:21:13.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00504f0 con 0x7f9ad004d500 2021-04-09T18:21:13.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad0050710 con 0x7f9ad0003a60 2021-04-09T18:21:13.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9c98700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ad004a760 0x7f9ad004cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:13.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9aea499700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ad0003a60 0x7f9ad0005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9aeac9a700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ad004d500 0x7f9ad004fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9c98700 1 -- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ad0003a60 msgr2=0x7f9ad0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:13.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9c98700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ad0003a60 0x7f9ad0005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9c98700 1 -- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ad004d500 msgr2=0x7f9ad004fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:13.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9c98700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ad004d500 0x7f9ad004fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:13.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9c98700 1 -- 172.21.15.32:0/1284073601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad00509b0 con 0x7f9ad004a760 2021-04-09T18:21:13.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9c98700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ad004a760 0x7f9ad004cc60 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f9ad4001900 tx=0x7f9ad4007cf0).ready entity=mon.0 client_cookie=fb36cfa8aa1f55ac server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:13.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ad4001470 con 0x7f9ad004a760 2021-04-09T18:21:13.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad40070c0 con 0x7f9ad004a760 2021-04-09T18:21:13.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9ae9497700 1 -- 172.21.15.32:0/1284073601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ad4004590 con 0x7f9ad004a760 2021-04-09T18:21:13.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9aec67d700 1 -- 172.21.15.32:0/1284073601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ad004a760 msgr2=0x7f9ad004cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:13.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9aec67d700 1 --2- 172.21.15.32:0/1284073601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ad004a760 0x7f9ad004cc60 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f9ad4001900 tx=0x7f9ad4007cf0).stop 2021-04-09T18:21:13.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.880 7f9aeac9a700 1 -- 172.21.15.32:0/1284073601 reap_dead start 2021-04-09T18:21:14.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.881 7f9aec67d700 1 -- 172.21.15.32:0/1284073601 shutdown_connections 2021-04-09T18:21:14.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.881 7f9aec67d700 1 -- 172.21.15.32:0/1284073601 >> 172.21.15.32:0/1284073601 conn(0x7f9ae40ce5f0 msgr2=0x7f9ae40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:14.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.881 7f9aec67d700 1 -- 172.21.15.32:0/1284073601 shutdown_connections 2021-04-09T18:21:14.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.881 7f9aec67d700 1 -- 172.21.15.32:0/1284073601 wait complete. 2021-04-09T18:21:14.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aec67d700 1 Processor -- start 2021-04-09T18:21:14.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aec67d700 1 -- start start 2021-04-09T18:21:14.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aec67d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ae41a9360 0x7f9ae41918f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:14.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aec67d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ae418c440 0x7f9ae418c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:14.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aec67d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae41a80f0 0x7f9ae41a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:14.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aec67d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4191f10 con 0x7f9ae41a80f0 2021-04-09T18:21:14.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aec67d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae41a9760 con 0x7f9ae418c440 2021-04-09T18:21:14.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aec67d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae418ce60 con 0x7f9ae41a9360 2021-04-09T18:21:14.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aeac9a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae41a80f0 0x7f9ae41a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:14.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aeac9a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae41a80f0 0x7f9ae41a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48146/0 (socket says 172.21.15.32:48146) 2021-04-09T18:21:14.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aeac9a700 1 -- 172.21.15.32:0/3161331314 learned_addr learned my addr 172.21.15.32:0/3161331314 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9aea499700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ae41a9360 0x7f9ae41918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.902 7f9ae9c98700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ae418c440 0x7f9ae418c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:14.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9aeac9a700 1 -- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ae41a9360 msgr2=0x7f9ae41918f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:14.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9aeac9a700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ae41a9360 0x7f9ae41918f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:14.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9aeac9a700 1 -- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ae418c440 msgr2=0x7f9ae418c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:14.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9aeac9a700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ae418c440 0x7f9ae418c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:14.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9aeac9a700 1 -- 172.21.15.32:0/3161331314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae41a8b10 con 0x7f9ae41a80f0 2021-04-09T18:21:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9aeac9a700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae41a80f0 0x7f9ae41a84f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7f9adc00a940 tx=0x7f9adc00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9adaffd700 1 -- 172.21.15.32:0/3161331314 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9adc001400 con 0x7f9ae41a80f0 2021-04-09T18:21:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9adaffd700 1 -- 172.21.15.32:0/3161331314 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc003030 con 0x7f9ae41a80f0 2021-04-09T18:21:14.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9adaffd700 1 -- 172.21.15.32:0/3161331314 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9adc0018f0 con 0x7f9ae41a80f0 2021-04-09T18:21:14.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae41a8b10 con 0x7f9ae41a80f0 2021-04-09T18:21:14.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.903 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae41a8b10 con 0x7f9ae41a80f0 2021-04-09T18:21:14.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.904 7f9adaffd700 1 -- 172.21.15.32:0/3161331314 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9adc0011f0 con 0x7f9ae41a80f0 2021-04-09T18:21:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.904 7f9adaffd700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9acc018930 0x7f9acc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.904 7f9adaffd700 1 -- 172.21.15.32:0/3161331314 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9adc032d80 con 0x7f9ae41a80f0 2021-04-09T18:21:14.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.904 7f9aea499700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9acc018930 0x7f9acc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:14.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.904 7f9aea499700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9acc018930 0x7f9acc01ae10 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:14.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.923 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae41a9760 con 0x7f9ae41a80f0 2021-04-09T18:21:14.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:13.926 7f9adaffd700 1 -- 172.21.15.32:0/3161331314 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9adc0066f0 con 0x7f9ae41a80f0 2021-04-09T18:21:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.117 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ae418bf60 con 0x7f9ae41a80f0 2021-04-09T18:21:14.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.117 7f9adaffd700 1 -- 172.21.15.32:0/3161331314 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f9adc0011f0 con 0x7f9ae41a80f0 2021-04-09T18:21:14.167 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:14.168 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:14.168 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:14.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9acc018930 msgr2=0x7f9acc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:14.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9acc018930 0x7f9acc01ae10 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:14.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae41a80f0 msgr2=0x7f9ae41a84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:14.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae41a80f0 0x7f9ae41a84f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7f9adc00a940 tx=0x7f9adc00ace0).stop 2021-04-09T18:21:14.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 shutdown_connections 2021-04-09T18:21:14.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9acc018930 0x7f9acc01ae10 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:14.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ae41a80f0 0x7f9ae41a84f0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:14.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ae418c440 0x7f9ae418c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:14.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 --2- 172.21.15.32:0/3161331314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ae41a9360 0x7f9ae41918f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:14.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 >> 172.21.15.32:0/3161331314 conn(0x7f9ae41837d0 msgr2=0x7f9ae4195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:14.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 shutdown_connections 2021-04-09T18:21:14.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:14.120 7f9aec67d700 1 -- 172.21.15.32:0/3161331314 wait complete. 2021-04-09T18:21:14.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:14.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:15.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:15.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.702 7f15110ad700 1 Processor -- start 2021-04-09T18:21:15.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.702 7f15110ad700 1 -- start start 2021-04-09T18:21:15.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.703 7f15110ad700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f150c0cffa0 0x7f150c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.703 7f15110ad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c0d6c80 0x7f150c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.703 7f15110ad700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c0d7280 con 0x7f150c0d7800 2021-04-09T18:21:15.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.703 7f15110ad700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f150c0d7080 con 0x7f150c0cffa0 2021-04-09T18:21:15.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.703 7f15110ad700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150c0d91e0 con 0x7f150c0d6c80 2021-04-09T18:21:15.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.703 7f150ad9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f150c0cffa0 0x7f150c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:15.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.703 7f150a59c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f150c0d7800 0x7f150c0d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57742/0 (socket says 172.21.15.32:57742) 2021-04-09T18:21:15.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.703 7f150a59c700 1 -- 172.21.15.32:0/895438861 learned_addr learned my addr 172.21.15.32:0/895438861 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:15.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f150b59e700 1 --2- 172.21.15.32:0/895438861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c0d6c80 0x7f150c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:15.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f150ad9d700 1 -- 172.21.15.32:0/895438861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c0d6c80 msgr2=0x7f150c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:15.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f150ad9d700 1 --2- 172.21.15.32:0/895438861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c0d6c80 0x7f150c10cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:15.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f150ad9d700 1 -- 172.21.15.32:0/895438861 >> v1:172.21.15.32:6789/0 conn(0x7f150c0d7800 legacy=0x7f150c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:15.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f150ad9d700 1 -- 172.21.15.32:0/895438861 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c0d83f0 con 0x7f150c0cffa0 2021-04-09T18:21:15.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f1509d9b700 1 -- 172.21.15.32:0/895438861 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330258972 0 0) 0x7f1500000ce0 con 0x7f150c0d7800 2021-04-09T18:21:15.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f150b59e700 1 --2- 172.21.15.32:0/895438861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c0d6c80 0x7f150c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:15.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f150ad9d700 1 --2- 172.21.15.32:0/895438861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0cffa0 0x7f150c0d2440 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f14fc006650 tx=0x7f14fc009a80).ready entity=mon.0 client_cookie=2701974916e3d54 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f1509d9b700 1 -- 172.21.15.32:0/895438861 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14fc0009b0 con 0x7f150c0cffa0 2021-04-09T18:21:15.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f1509d9b700 1 -- 172.21.15.32:0/895438861 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc00c0c0 con 0x7f150c0cffa0 2021-04-09T18:21:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f1509d9b700 1 -- 172.21.15.32:0/895438861 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14fc00c510 con 0x7f150c0cffa0 2021-04-09T18:21:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f15110ad700 1 -- 172.21.15.32:0/895438861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0cffa0 msgr2=0x7f150c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.704 7f15110ad700 1 --2- 172.21.15.32:0/895438861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0cffa0 0x7f150c0d2440 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f14fc006650 tx=0x7f14fc009a80).stop 2021-04-09T18:21:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.705 7f15110ad700 1 -- 172.21.15.32:0/895438861 shutdown_connections 2021-04-09T18:21:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.705 7f15110ad700 1 --2- 172.21.15.32:0/895438861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c0d6c80 0x7f150c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.705 7f15110ad700 1 --2- 172.21.15.32:0/895438861 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0cffa0 0x7f150c0d2440 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.705 7f15110ad700 1 -- 172.21.15.32:0/895438861 >> 172.21.15.32:0/895438861 conn(0x7f150c0ce5c0 msgr2=0x7f150c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.705 7f15110ad700 1 -- 172.21.15.32:0/895438861 shutdown_connections 2021-04-09T18:21:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.705 7f15110ad700 1 -- 172.21.15.32:0/895438861 wait complete. 2021-04-09T18:21:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.725 7f15110ad700 1 Processor -- start 2021-04-09T18:21:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.725 7f15110ad700 1 -- start start 2021-04-09T18:21:15.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f15110ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0b72f0 0x7f150c0b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f15110ad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c19c860 0x7f150c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f15110ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f150c19fdd0 0x7f150c19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f15110ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c19f230 con 0x7f150c0b72f0 2021-04-09T18:21:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f15110ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c08a6a0 con 0x7f150c19fdd0 2021-04-09T18:21:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f15110ad700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150c18dce0 con 0x7f150c19c860 2021-04-09T18:21:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0b72f0 0x7f150c0b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0b72f0 0x7f150c0b76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48158/0 (socket says 172.21.15.32:48158) 2021-04-09T18:21:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150ad9d700 1 -- 172.21.15.32:0/3303316557 learned_addr learned my addr 172.21.15.32:0/3303316557 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150a59c700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c19c860 0x7f150c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:15.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150b59e700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f150c19fdd0 0x7f150c19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:15.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150ad9d700 1 -- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c19c860 msgr2=0x7f150c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150ad9d700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c19c860 0x7f150c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150ad9d700 1 -- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f150c19fdd0 msgr2=0x7f150c19da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150ad9d700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f150c19fdd0 0x7f150c19da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150ad9d700 1 -- 172.21.15.32:0/3303316557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c18e150 con 0x7f150c0b72f0 2021-04-09T18:21:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.726 7f150ad9d700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0b72f0 0x7f150c0b76f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7f14fc009650 tx=0x7f14fc00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.727 7f14fb7fe700 1 -- 172.21.15.32:0/3303316557 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14fc0093a0 con 0x7f150c0b72f0 2021-04-09T18:21:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.727 7f14fb7fe700 1 -- 172.21.15.32:0/3303316557 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc006070 con 0x7f150c0b72f0 2021-04-09T18:21:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.727 7f15110ad700 1 -- 172.21.15.32:0/3303316557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c18e150 con 0x7f150c0b72f0 2021-04-09T18:21:15.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.727 7f14fb7fe700 1 -- 172.21.15.32:0/3303316557 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14fc0052e0 con 0x7f150c0b72f0 2021-04-09T18:21:15.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.727 7f15110ad700 1 -- 172.21.15.32:0/3303316557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150c18e150 con 0x7f150c0b72f0 2021-04-09T18:21:15.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.727 7f14fb7fe700 1 -- 172.21.15.32:0/3303316557 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f14fc009300 con 0x7f150c0b72f0 2021-04-09T18:21:15.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.727 7f14fb7fe700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14f00188d0 0x7f14f001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:15.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.727 7f14fb7fe700 1 -- 172.21.15.32:0/3303316557 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f14fc032a00 con 0x7f150c0b72f0 2021-04-09T18:21:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.728 7f150a59c700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14f00188d0 0x7f14f001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.728 7f150a59c700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14f00188d0 0x7f14f001add0 crc :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.747 7f15110ad700 1 -- 172.21.15.32:0/3303316557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150c182e10 con 0x7f150c0b72f0 2021-04-09T18:21:15.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.751 7f14fb7fe700 1 -- 172.21.15.32:0/3303316557 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f14fc006030 con 0x7f150c0b72f0 2021-04-09T18:21:15.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.944 7f15110ad700 1 -- 172.21.15.32:0/3303316557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f150c182e10 con 0x7f150c0b72f0 2021-04-09T18:21:15.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.944 7f14fb7fe700 1 -- 172.21.15.32:0/3303316557 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f14fc000940 con 0x7f150c0b72f0 2021-04-09T18:21:15.994 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:15.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:15.995 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:16.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 -- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14f00188d0 msgr2=0x7f14f001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:16.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14f00188d0 0x7f14f001add0 crc :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:16.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 -- 172.21.15.32:0/3303316557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0b72f0 msgr2=0x7f150c0b76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:16.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0b72f0 0x7f150c0b76f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7f14fc009650 tx=0x7f14fc00b8e0).stop 2021-04-09T18:21:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 -- 172.21.15.32:0/3303316557 shutdown_connections 2021-04-09T18:21:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14f00188d0 0x7f14f001add0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f150c19fdd0 0x7f150c19da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:16.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f150c19c860 0x7f150c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:16.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 --2- 172.21.15.32:0/3303316557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c0b72f0 0x7f150c0b76f0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:16.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 -- 172.21.15.32:0/3303316557 >> 172.21.15.32:0/3303316557 conn(0x7f150c097550 msgr2=0x7f150c18b920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 -- 172.21.15.32:0/3303316557 shutdown_connections 2021-04-09T18:21:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:15.947 7f15110ad700 1 -- 172.21.15.32:0/3303316557 wait complete. 2021-04-09T18:21:16.420 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:16.420 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:17.422 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:17.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.533 7f01ae031700 1 Processor -- start 2021-04-09T18:21:17.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.533 7f01ae031700 1 -- start start 2021-04-09T18:21:17.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.533 7f01ae031700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a810d010 0x7f01a80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:17.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.533 7f01ae031700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f01a80d8490 0x7f01a810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:17.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.533 7f01ae031700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a810d480 con 0x7f01a80d1da0 2021-04-09T18:21:17.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.533 7f01ae031700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a810e780 con 0x7f01a80d8490 2021-04-09T18:21:17.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.533 7f01ae031700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80d8d30 con 0x7f01a810d010 2021-04-09T18:21:17.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f01a80d8490 0x7f01a810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:17.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f01a80d8490 0x7f01a810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48168/0 (socket says 172.21.15.32:48168) 2021-04-09T18:21:17.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a7fff700 1 -- 172.21.15.32:0/380064162 learned_addr learned my addr 172.21.15.32:0/380064162 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:17.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a77fe700 1 --2- 172.21.15.32:0/380064162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a810d010 0x7f01a80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:17.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a7fff700 1 -- 172.21.15.32:0/380064162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a810d010 msgr2=0x7f01a80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:17.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a7fff700 1 --2- 172.21.15.32:0/380064162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a810d010 0x7f01a80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a7fff700 1 -- 172.21.15.32:0/380064162 >> v1:172.21.15.86:6789/0 conn(0x7f01a80d1da0 legacy=0x7f01a80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:17.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a7fff700 1 -- 172.21.15.32:0/380064162 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80d9220 con 0x7f01a80d8490 2021-04-09T18:21:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a7fff700 1 --2- 172.21.15.32:0/380064162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a80d8490 0x7f01a810da00 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f01a0006b80 tx=0x7f01a0009b90).ready entity=mon.0 client_cookie=3ad829fd9ff9dd15 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a67fc700 1 -- 172.21.15.32:0/380064162 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01a00022e0 con 0x7f01a80d8490 2021-04-09T18:21:17.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a67fc700 1 -- 172.21.15.32:0/380064162 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a0002620 con 0x7f01a80d8490 2021-04-09T18:21:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01a67fc700 1 -- 172.21.15.32:0/380064162 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01a0002b00 con 0x7f01a80d8490 2021-04-09T18:21:17.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01ae031700 1 -- 172.21.15.32:0/380064162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a80d8490 msgr2=0x7f01a810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:17.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.534 7f01ae031700 1 --2- 172.21.15.32:0/380064162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a80d8490 0x7f01a810da00 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f01a0006b80 tx=0x7f01a0009b90).stop 2021-04-09T18:21:17.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.535 7f01ae031700 1 -- 172.21.15.32:0/380064162 shutdown_connections 2021-04-09T18:21:17.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.535 7f01ae031700 1 --2- 172.21.15.32:0/380064162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a80d8490 0x7f01a810da00 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.535 7f01ae031700 1 --2- 172.21.15.32:0/380064162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a810d010 0x7f01a80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.535 7f01ae031700 1 -- 172.21.15.32:0/380064162 >> 172.21.15.32:0/380064162 conn(0x7f01a80ce5f0 msgr2=0x7f01a80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:17.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.535 7f01ae031700 1 -- 172.21.15.32:0/380064162 shutdown_connections 2021-04-09T18:21:17.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.535 7f01ae031700 1 -- 172.21.15.32:0/380064162 wait complete. 2021-04-09T18:21:17.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.555 7f01ae031700 1 Processor -- start 2021-04-09T18:21:17.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.555 7f01ae031700 1 -- start start 2021-04-09T18:21:17.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01ae031700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a8099460 0x7f01a819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:17.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01ae031700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a819a760 0x7f01a80b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:17.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01ae031700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a819ab60 0x7f01a80b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:17.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01ae031700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80b3c80 con 0x7f01a8099460 2021-04-09T18:21:17.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01ae031700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80b3e70 con 0x7f01a819ab60 2021-04-09T18:21:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01ae031700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a8198930 con 0x7f01a819a760 2021-04-09T18:21:17.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01a77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a8099460 0x7f01a819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:17.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01a77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a8099460 0x7f01a819a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48170/0 (socket says 172.21.15.32:48170) 2021-04-09T18:21:17.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01a77fe700 1 -- 172.21.15.32:0/3074139427 learned_addr learned my addr 172.21.15.32:0/3074139427 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:17.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01a7fff700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a819ab60 0x7f01a80b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:17.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.556 7f01a6ffd700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a819a760 0x7f01a80b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:17.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f01a77fe700 1 -- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a819a760 msgr2=0x7f01a80b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:17.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f01a77fe700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a819a760 0x7f01a80b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f01a77fe700 1 -- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a819ab60 msgr2=0x7f01a80b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:17.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f01a77fe700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a819ab60 0x7f01a80b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f01a77fe700 1 -- 172.21.15.32:0/3074139427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a8198b10 con 0x7f01a8099460 2021-04-09T18:21:17.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f01a77fe700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a8099460 0x7f01a819a140 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f0198009980 tx=0x7f019800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:17.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f018ffff700 1 -- 172.21.15.32:0/3074139427 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0198006030 con 0x7f01a8099460 2021-04-09T18:21:17.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f018ffff700 1 -- 172.21.15.32:0/3074139427 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01980062f0 con 0x7f01a8099460 2021-04-09T18:21:17.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f01ae031700 1 -- 172.21.15.32:0/3074139427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a8198b10 con 0x7f01a8099460 2021-04-09T18:21:17.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f018ffff700 1 -- 172.21.15.32:0/3074139427 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0198005660 con 0x7f01a8099460 2021-04-09T18:21:17.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f01ae031700 1 -- 172.21.15.32:0/3074139427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a8198b10 con 0x7f01a8099460 2021-04-09T18:21:17.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.557 7f018ffff700 1 -- 172.21.15.32:0/3074139427 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f01980064d0 con 0x7f01a8099460 2021-04-09T18:21:17.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.558 7f018ffff700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0188018930 0x7f018801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:17.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.558 7f018ffff700 1 -- 172.21.15.32:0/3074139427 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0198032dd0 con 0x7f01a8099460 2021-04-09T18:21:17.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.558 7f01a6ffd700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0188018930 0x7f018801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:17.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.558 7f01a6ffd700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0188018930 0x7f018801ae10 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.578 7f01ae031700 1 -- 172.21.15.32:0/3074139427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a81b7080 con 0x7f01a8099460 2021-04-09T18:21:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.581 7f018ffff700 1 -- 172.21.15.32:0/3074139427 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f019800be10 con 0x7f01a8099460 2021-04-09T18:21:17.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.773 7f01ae031700 1 -- 172.21.15.32:0/3074139427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01a81b7080 con 0x7f01a8099460 2021-04-09T18:21:17.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.773 7f018ffff700 1 -- 172.21.15.32:0/3074139427 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f019800be10 con 0x7f01a8099460 2021-04-09T18:21:17.823 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:17.824 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:17.824 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:17.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 -- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0188018930 msgr2=0x7f018801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:17.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0188018930 0x7f018801ae10 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 -- 172.21.15.32:0/3074139427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a8099460 msgr2=0x7f01a819a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:17.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a8099460 0x7f01a819a140 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f0198009980 tx=0x7f019800bcd0).stop 2021-04-09T18:21:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 -- 172.21.15.32:0/3074139427 shutdown_connections 2021-04-09T18:21:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0188018930 0x7f018801ae10 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a819ab60 0x7f01a80b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a819a760 0x7f01a80b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 --2- 172.21.15.32:0/3074139427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a8099460 0x7f01a819a140 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:17.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 -- 172.21.15.32:0/3074139427 >> 172.21.15.32:0/3074139427 conn(0x7f01a8191560 msgr2=0x7f01a81b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:17.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 -- 172.21.15.32:0/3074139427 shutdown_connections 2021-04-09T18:21:17.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:17.776 7f01ae031700 1 -- 172.21.15.32:0/3074139427 wait complete. 2021-04-09T18:21:18.238 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:18.239 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:19.241 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:19.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.353 7f242e266700 1 Processor -- start 2021-04-09T18:21:19.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.353 7f242e266700 1 -- start start 2021-04-09T18:21:19.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f242e266700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f242810da00 0x7f242810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:19.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f242e266700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242810d480 con 0x7f242810d010 2021-04-09T18:21:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f242e266700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24280d1f70 con 0x7f24280d7ca0 2021-04-09T18:21:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f242e266700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24280d8d30 con 0x7f242810da00 2021-04-09T18:21:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f2427fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f242810d010 0x7f24280d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57766/0 (socket says 172.21.15.32:57766) 2021-04-09T18:21:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f2427fff700 1 -- 172.21.15.32:0/3707421563 learned_addr learned my addr 172.21.15.32:0/3707421563 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f242c883700 1 --2- 172.21.15.32:0/3707421563 >> v2:172.21.15.86:3300/0 conn(0x7f242810da00 0x7f242810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:19.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f24277fe700 1 -- 172.21.15.32:0/3707421563 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076310141 0 0) 0x7f2418001060 con 0x7f242810d010 2021-04-09T18:21:19.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f24277fe700 1 -- 172.21.15.32:0/3707421563 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2414002fe0 con 0x7f242810d010 2021-04-09T18:21:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f24277fe700 1 -- 172.21.15.32:0/3707421563 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706620676 0 0) 0x7f2410000c60 con 0x7f24280d7ca0 2021-04-09T18:21:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f24277fe700 1 -- 172.21.15.32:0/3707421563 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2414003210 con 0x7f24280d7ca0 2021-04-09T18:21:19.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f242c883700 1 -- 172.21.15.32:0/3707421563 >> v1:172.21.15.86:6789/0 conn(0x7f24280d7ca0 legacy=0x7f24280d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:19.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f242c883700 1 -- 172.21.15.32:0/3707421563 >> v1:172.21.15.32:6789/0 conn(0x7f242810d010 legacy=0x7f24280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f242c883700 1 -- 172.21.15.32:0/3707421563 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24280d9220 con 0x7f242810da00 2021-04-09T18:21:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f242c883700 1 --2- 172.21.15.32:0/3707421563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242810da00 0x7f242810de00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f2420006ad0 tx=0x7f2420009ad0).ready entity=mon.1 client_cookie=768c70f12265ba82 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.354 7f24277fe700 1 -- 172.21.15.32:0/3707421563 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24200022e0 con 0x7f242810da00 2021-04-09T18:21:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.355 7f24277fe700 1 -- 172.21.15.32:0/3707421563 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2420006410 con 0x7f242810da00 2021-04-09T18:21:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.355 7f24277fe700 1 -- 172.21.15.32:0/3707421563 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2420002620 con 0x7f242810da00 2021-04-09T18:21:19.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.355 7f242e266700 1 -- 172.21.15.32:0/3707421563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242810da00 msgr2=0x7f242810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.355 7f242e266700 1 --2- 172.21.15.32:0/3707421563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242810da00 0x7f242810de00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f2420006ad0 tx=0x7f2420009ad0).stop 2021-04-09T18:21:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.355 7f242e266700 1 -- 172.21.15.32:0/3707421563 shutdown_connections 2021-04-09T18:21:19.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.355 7f242e266700 1 --2- 172.21.15.32:0/3707421563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242810da00 0x7f242810de00 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:19.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.355 7f242e266700 1 -- 172.21.15.32:0/3707421563 >> 172.21.15.32:0/3707421563 conn(0x7f24280ce5f0 msgr2=0x7f24280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:19.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.355 7f242e266700 1 -- 172.21.15.32:0/3707421563 shutdown_connections 2021-04-09T18:21:19.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.355 7f242e266700 1 -- 172.21.15.32:0/3707421563 wait complete. 2021-04-09T18:21:19.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f242e266700 1 Processor -- start 2021-04-09T18:21:19.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f242e266700 1 -- start start 2021-04-09T18:21:19.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f242e266700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24280ab300 0x7f24281a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f242e266700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24281a8ad0 0x7f24281a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f242e266700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2428180db0 0x7f24281811b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:19.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f242e266700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24281817d0 con 0x7f24280ab300 2021-04-09T18:21:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f242e266700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24281819f0 con 0x7f24281a8ad0 2021-04-09T18:21:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f242e266700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24281a1f80 con 0x7f2428180db0 2021-04-09T18:21:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f2427fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24280ab300 0x7f24281a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f2427fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24280ab300 0x7f24281a12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48182/0 (socket says 172.21.15.32:48182) 2021-04-09T18:21:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f2427fff700 1 -- 172.21.15.32:0/765819972 learned_addr learned my addr 172.21.15.32:0/765819972 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f241ffff700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24281a8ad0 0x7f24281a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.376 7f242c883700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2428180db0 0x7f24281811b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:19.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f2427fff700 1 -- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2428180db0 msgr2=0x7f24281811b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:19.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f2427fff700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2428180db0 0x7f24281811b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:19.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f2427fff700 1 -- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24281a8ad0 msgr2=0x7f24281a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:19.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f2427fff700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24281a8ad0 0x7f24281a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f2427fff700 1 -- 172.21.15.32:0/765819972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24281a21a0 con 0x7f24280ab300 2021-04-09T18:21:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f242c883700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2428180db0 0x7f24281811b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f2427fff700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24280ab300 0x7f24281a12d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f2418009980 tx=0x7f241800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f24257fa700 1 -- 172.21.15.32:0/765819972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2418006020 con 0x7f24280ab300 2021-04-09T18:21:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f24257fa700 1 -- 172.21.15.32:0/765819972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2418006320 con 0x7f24280ab300 2021-04-09T18:21:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f24257fa700 1 -- 172.21.15.32:0/765819972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2418004720 con 0x7f24280ab300 2021-04-09T18:21:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f242e266700 1 -- 172.21.15.32:0/765819972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24281a21a0 con 0x7f24280ab300 2021-04-09T18:21:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.377 7f242e266700 1 -- 172.21.15.32:0/765819972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24281a21a0 con 0x7f24280ab300 2021-04-09T18:21:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.378 7f24257fa700 1 -- 172.21.15.32:0/765819972 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2418006500 con 0x7f24280ab300 2021-04-09T18:21:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.378 7f24257fa700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f240c018930 0x7f240c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.378 7f24257fa700 1 -- 172.21.15.32:0/765819972 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2418032d70 con 0x7f24280ab300 2021-04-09T18:21:19.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.378 7f241ffff700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f240c018930 0x7f240c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:19.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.378 7f241ffff700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f240c018930 0x7f240c01ae10 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:19.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.397 7f242e266700 1 -- 172.21.15.32:0/765819972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24281b7070 con 0x7f24280ab300 2021-04-09T18:21:19.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.401 7f24257fa700 1 -- 172.21.15.32:0/765819972 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f241800be10 con 0x7f24280ab300 2021-04-09T18:21:19.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.594 7f242e266700 1 -- 172.21.15.32:0/765819972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f242810d480 con 0x7f24280ab300 2021-04-09T18:21:19.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.595 7f24257fa700 1 -- 172.21.15.32:0/765819972 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f2418006220 con 0x7f24280ab300 2021-04-09T18:21:19.643 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:19.644 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:19.644 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:19.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 -- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f240c018930 msgr2=0x7f240c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:19.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f240c018930 0x7f240c01ae10 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:19.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 -- 172.21.15.32:0/765819972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24280ab300 msgr2=0x7f24281a12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:19.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24280ab300 0x7f24281a12d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f2418009980 tx=0x7f241800bcd0).stop 2021-04-09T18:21:19.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 -- 172.21.15.32:0/765819972 shutdown_connections 2021-04-09T18:21:19.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f240c018930 0x7f240c01ae10 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:19.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2428180db0 0x7f24281811b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:19.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24281a8ad0 0x7f24281a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:19.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 --2- 172.21.15.32:0/765819972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24280ab300 0x7f24281a12d0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:19.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 -- 172.21.15.32:0/765819972 >> 172.21.15.32:0/765819972 conn(0x7f24281aa2b0 msgr2=0x7f24281a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:19.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 -- 172.21.15.32:0/765819972 shutdown_connections 2021-04-09T18:21:19.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:19.597 7f242e266700 1 -- 172.21.15.32:0/765819972 wait complete. 2021-04-09T18:21:20.058 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:20.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:21.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:21.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.179 7f9fe7a2b700 1 Processor -- start 2021-04-09T18:21:21.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.179 7f9fe7a2b700 1 -- start start 2021-04-09T18:21:21.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.180 7f9fe7a2b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9fe010d010 0x7f9fe00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:21.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.180 7f9fe7a2b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe010d480 con 0x7f9fe00d8490 2021-04-09T18:21:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.180 7f9fe7a2b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe010e1f0 con 0x7f9fe00d1da0 2021-04-09T18:21:21.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.180 7f9fe7a2b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe00d8d30 con 0x7f9fe010d010 2021-04-09T18:21:21.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.180 7f9fe5847700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9fe010d010 0x7f9fe00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:21.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.180 7f9fe6048700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9fe00d8490 0x7f9fe010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57780/0 (socket says 172.21.15.32:57780) 2021-04-09T18:21:21.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.180 7f9fe6048700 1 -- 172.21.15.32:0/1900646898 learned_addr learned my addr 172.21.15.32:0/1900646898 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:21.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe4845700 1 -- 172.21.15.32:0/1900646898 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828171887 0 0) 0x7f9fd8000c60 con 0x7f9fe00d1da0 2021-04-09T18:21:21.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe4845700 1 -- 172.21.15.32:0/1900646898 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc8002fe0 con 0x7f9fe00d1da0 2021-04-09T18:21:21.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe5847700 1 -- 172.21.15.32:0/1900646898 >> v1:172.21.15.86:6789/0 conn(0x7f9fe00d1da0 legacy=0x7f9fe00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:21.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe4845700 1 -- 172.21.15.32:0/1900646898 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075930887 0 0) 0x7f9fd00009b0 con 0x7f9fe00d8490 2021-04-09T18:21:21.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe5847700 1 -- 172.21.15.32:0/1900646898 >> v1:172.21.15.32:6789/0 conn(0x7f9fe00d8490 legacy=0x7f9fe010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:21.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe5847700 1 -- 172.21.15.32:0/1900646898 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe00d9220 con 0x7f9fe010d010 2021-04-09T18:21:21.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe5847700 1 --2- 172.21.15.32:0/1900646898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe010d010 0x7f9fe00d1780 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7f9fd40069d0 tx=0x7f9fd4009dc0).ready entity=mon.0 client_cookie=2c2a59e0f03ccea4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:21.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe4845700 1 -- 172.21.15.32:0/1900646898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fd400b510 con 0x7f9fe010d010 2021-04-09T18:21:21.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe4845700 1 -- 172.21.15.32:0/1900646898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd4006ac0 con 0x7f9fe010d010 2021-04-09T18:21:21.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe4845700 1 -- 172.21.15.32:0/1900646898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fd400ba80 con 0x7f9fe010d010 2021-04-09T18:21:21.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe7a2b700 1 -- 172.21.15.32:0/1900646898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe010d010 msgr2=0x7f9fe00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:21.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.181 7f9fe7a2b700 1 --2- 172.21.15.32:0/1900646898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe010d010 0x7f9fe00d1780 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7f9fd40069d0 tx=0x7f9fd4009dc0).stop 2021-04-09T18:21:21.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.182 7f9fe7a2b700 1 -- 172.21.15.32:0/1900646898 shutdown_connections 2021-04-09T18:21:21.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.182 7f9fe7a2b700 1 --2- 172.21.15.32:0/1900646898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe010d010 0x7f9fe00d1780 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:21.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.182 7f9fe7a2b700 1 -- 172.21.15.32:0/1900646898 >> 172.21.15.32:0/1900646898 conn(0x7f9fe00ce5f0 msgr2=0x7f9fe00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:21.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.182 7f9fe7a2b700 1 -- 172.21.15.32:0/1900646898 shutdown_connections 2021-04-09T18:21:21.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.182 7f9fe7a2b700 1 -- 172.21.15.32:0/1900646898 wait complete. 2021-04-09T18:21:21.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.202 7f9fe7a2b700 1 Processor -- start 2021-04-09T18:21:21.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.202 7f9fe7a2b700 1 -- start start 2021-04-09T18:21:21.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe7a2b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fe018fb40 0x7f9fe01a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:21.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe7a2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe01a17f0 0x7f9fe017feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:21.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe7a2b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe017c7b0 0x7f9fe01804d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:21.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe7a2b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe01a0e70 con 0x7f9fe01a17f0 2021-04-09T18:21:21.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe7a2b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe00a9140 con 0x7f9fe017c7b0 2021-04-09T18:21:21.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe7a2b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe01b7fc0 con 0x7f9fe018fb40 2021-04-09T18:21:21.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5046700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe01a17f0 0x7f9fe017feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:21.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5046700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe01a17f0 0x7f9fe017feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48196/0 (socket says 172.21.15.32:48196) 2021-04-09T18:21:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5046700 1 -- 172.21.15.32:0/876604161 learned_addr learned my addr 172.21.15.32:0/876604161 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe6048700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe017c7b0 0x7f9fe01804d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5847700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fe018fb40 0x7f9fe01a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5046700 1 -- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fe018fb40 msgr2=0x7f9fe01a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5046700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fe018fb40 0x7f9fe01a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5046700 1 -- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe017c7b0 msgr2=0x7f9fe01804d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5046700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe017c7b0 0x7f9fe01804d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5046700 1 -- 172.21.15.32:0/876604161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe01b81e0 con 0x7f9fe01a17f0 2021-04-09T18:21:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5847700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fe018fb40 0x7f9fe01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.203 7f9fe5046700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe01a17f0 0x7f9fe017feb0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7f9fd8006f20 tx=0x7f9fd800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.204 7f9fde7fc700 1 -- 172.21.15.32:0/876604161 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fd800bd80 con 0x7f9fe01a17f0 2021-04-09T18:21:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.204 7f9fde7fc700 1 -- 172.21.15.32:0/876604161 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd8006030 con 0x7f9fe01a17f0 2021-04-09T18:21:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.204 7f9fde7fc700 1 -- 172.21.15.32:0/876604161 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fd8005520 con 0x7f9fe01a17f0 2021-04-09T18:21:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.204 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe01b81e0 con 0x7f9fe01a17f0 2021-04-09T18:21:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.204 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe01b81e0 con 0x7f9fe01a17f0 2021-04-09T18:21:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.204 7f9fde7fc700 1 -- 172.21.15.32:0/876604161 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9fd8006210 con 0x7f9fe01a17f0 2021-04-09T18:21:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.204 7f9fde7fc700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9fc0018930 0x7f9fc001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.204 7f9fde7fc700 1 -- 172.21.15.32:0/876604161 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9fd80329f0 con 0x7f9fe01a17f0 2021-04-09T18:21:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.205 7f9fe5847700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9fc0018930 0x7f9fc001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.205 7f9fe5847700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9fc0018930 0x7f9fc001ae10 crc :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.224 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe010d480 con 0x7f9fe01a17f0 2021-04-09T18:21:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.227 7f9fde7fc700 1 -- 172.21.15.32:0/876604161 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9fd8006bb0 con 0x7f9fe01a17f0 2021-04-09T18:21:21.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.419 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fe010d480 con 0x7f9fe01a17f0 2021-04-09T18:21:21.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.419 7f9fde7fc700 1 -- 172.21.15.32:0/876604161 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f9fd8006bb0 con 0x7f9fe01a17f0 2021-04-09T18:21:21.470 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:21.471 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:21.471 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:21.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9fc0018930 msgr2=0x7f9fc001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:21.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9fc0018930 0x7f9fc001ae10 crc :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:21.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe01a17f0 msgr2=0x7f9fe017feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:21.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe01a17f0 0x7f9fe017feb0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7f9fd8006f20 tx=0x7f9fd800b990).stop 2021-04-09T18:21:21.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 shutdown_connections 2021-04-09T18:21:21.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9fc0018930 0x7f9fc001ae10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:21.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9fe01a17f0 0x7f9fe017feb0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:21.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe017c7b0 0x7f9fe01804d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:21.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 --2- 172.21.15.32:0/876604161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fe018fb40 0x7f9fe01a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:21.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 >> 172.21.15.32:0/876604161 conn(0x7f9fe01905b0 msgr2=0x7f9fe01a3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:21.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 shutdown_connections 2021-04-09T18:21:21.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:21.422 7f9fe7a2b700 1 -- 172.21.15.32:0/876604161 wait complete. 2021-04-09T18:21:21.887 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:21.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:22.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:23.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.003 7f6f466c8700 1 Processor -- start 2021-04-09T18:21:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.003 7f6f466c8700 1 -- start start 2021-04-09T18:21:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f466c8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6f4010d010 0x7f6f400d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f466c8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f400d1da0 0x7f6f400d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:23.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f466c8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4010d480 con 0x7f6f400d82c0 2021-04-09T18:21:23.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f466c8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f400d8d30 con 0x7f6f400d1da0 2021-04-09T18:21:23.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f466c8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f400d9220 con 0x7f6f4010d010 2021-04-09T18:21:23.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f37fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f400d1da0 0x7f6f400d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:23.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f37fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f400d1da0 0x7f6f400d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48208/0 (socket says 172.21.15.32:48208) 2021-04-09T18:21:23.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f37fff700 1 -- 172.21.15.32:0/2614065207 learned_addr learned my addr 172.21.15.32:0/2614065207 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:23.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f3ffff700 1 --2- 172.21.15.32:0/2614065207 >> v2:172.21.15.86:3300/0 conn(0x7f6f4010d010 0x7f6f400d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:23.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f37fff700 1 -- 172.21.15.32:0/2614065207 >> v2:172.21.15.86:3300/0 conn(0x7f6f4010d010 msgr2=0x7f6f400d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:23.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f37fff700 1 --2- 172.21.15.32:0/2614065207 >> v2:172.21.15.86:3300/0 conn(0x7f6f4010d010 0x7f6f400d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f37fff700 1 -- 172.21.15.32:0/2614065207 >> v1:172.21.15.86:6789/0 conn(0x7f6f400d82c0 legacy=0x7f6f4010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:23.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f37fff700 1 -- 172.21.15.32:0/2614065207 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f400d86f0 con 0x7f6f400d1da0 2021-04-09T18:21:23.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f3ffff700 1 --2- 172.21.15.32:0/2614065207 >> v2:172.21.15.86:3300/0 conn(0x7f6f4010d010 0x7f6f400d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:21:23.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.004 7f6f37fff700 1 --2- 172.21.15.32:0/2614065207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f400d1da0 0x7f6f400d7ca0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7f6f28006700 tx=0x7f6f28009b80).ready entity=mon.0 client_cookie=4928a8db742a7397 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:23.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f3f7fe700 1 -- 172.21.15.32:0/2614065207 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f2800c0c0 con 0x7f6f400d1da0 2021-04-09T18:21:23.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f3f7fe700 1 -- 172.21.15.32:0/2614065207 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2800c440 con 0x7f6f400d1da0 2021-04-09T18:21:23.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f3f7fe700 1 -- 172.21.15.32:0/2614065207 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f2800c920 con 0x7f6f400d1da0 2021-04-09T18:21:23.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f466c8700 1 -- 172.21.15.32:0/2614065207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f400d1da0 msgr2=0x7f6f400d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:23.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f466c8700 1 --2- 172.21.15.32:0/2614065207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f400d1da0 0x7f6f400d7ca0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7f6f28006700 tx=0x7f6f28009b80).stop 2021-04-09T18:21:23.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f466c8700 1 -- 172.21.15.32:0/2614065207 shutdown_connections 2021-04-09T18:21:23.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f466c8700 1 --2- 172.21.15.32:0/2614065207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f400d1da0 0x7f6f400d7ca0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f466c8700 1 --2- 172.21.15.32:0/2614065207 >> v2:172.21.15.86:3300/0 conn(0x7f6f4010d010 0x7f6f400d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f466c8700 1 -- 172.21.15.32:0/2614065207 >> 172.21.15.32:0/2614065207 conn(0x7f6f400ce5f0 msgr2=0x7f6f400d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:23.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f466c8700 1 -- 172.21.15.32:0/2614065207 shutdown_connections 2021-04-09T18:21:23.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.005 7f6f466c8700 1 -- 172.21.15.32:0/2614065207 wait complete. 2021-04-09T18:21:23.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f466c8700 1 Processor -- start 2021-04-09T18:21:23.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f466c8700 1 -- start start 2021-04-09T18:21:23.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f466c8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f401a5210 0x7f6f40185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:23.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f466c8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40185a70 0x7f6f400acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:23.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f466c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f40185e70 0x7f6f400ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:23.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f466c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4018f7b0 con 0x7f6f40185e70 2021-04-09T18:21:23.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f466c8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400ad7f0 con 0x7f6f401a5210 2021-04-09T18:21:23.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f466c8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400ada10 con 0x7f6f40185a70 2021-04-09T18:21:23.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f44ce5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f40185e70 0x7f6f400ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:23.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f44ce5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f40185e70 0x7f6f400ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48210/0 (socket says 172.21.15.32:48210) 2021-04-09T18:21:23.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f44ce5700 1 -- 172.21.15.32:0/3990928371 learned_addr learned my addr 172.21.15.32:0/3990928371 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:23.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f37fff700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40185a70 0x7f6f400acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:23.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.026 7f6f3ffff700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f401a5210 0x7f6f40185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:23.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f44ce5700 1 -- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40185a70 msgr2=0x7f6f400acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:23.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f44ce5700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40185a70 0x7f6f400acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f44ce5700 1 -- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f401a5210 msgr2=0x7f6f40185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:23.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f44ce5700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f401a5210 0x7f6f40185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f44ce5700 1 -- 172.21.15.32:0/3990928371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f40179610 con 0x7f6f40185e70 2021-04-09T18:21:23.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f3ffff700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f401a5210 0x7f6f40185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:23.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f44ce5700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f40185e70 0x7f6f400ad1d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f6f38009940 tx=0x7f6f3800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f3d7fa700 1 -- 172.21.15.32:0/3990928371 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f38000d30 con 0x7f6f40185e70 2021-04-09T18:21:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f3d7fa700 1 -- 172.21.15.32:0/3990928371 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3800c2f0 con 0x7f6f40185e70 2021-04-09T18:21:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f3d7fa700 1 -- 172.21.15.32:0/3990928371 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3800c7d0 con 0x7f6f40185e70 2021-04-09T18:21:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f40179610 con 0x7f6f40185e70 2021-04-09T18:21:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.027 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f40179610 con 0x7f6f40185e70 2021-04-09T18:21:23.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.028 7f6f3d7fa700 1 -- 172.21.15.32:0/3990928371 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6f38001220 con 0x7f6f40185e70 2021-04-09T18:21:23.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.028 7f6f3d7fa700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f240188c0 0x7f6f2401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:23.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.028 7f6f3d7fa700 1 -- 172.21.15.32:0/3990928371 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6f38032b80 con 0x7f6f40185e70 2021-04-09T18:21:23.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.028 7f6f3ffff700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f240188c0 0x7f6f2401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:23.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.028 7f6f3ffff700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f240188c0 0x7f6f2401adc0 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:23.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.048 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4018f7b0 con 0x7f6f40185e70 2021-04-09T18:21:23.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.051 7f6f3d7fa700 1 -- 172.21.15.32:0/3990928371 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6f3802df50 con 0x7f6f40185e70 2021-04-09T18:21:23.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.255 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f4018f7b0 con 0x7f6f40185e70 2021-04-09T18:21:23.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.255 7f6f3d7fa700 1 -- 172.21.15.32:0/3990928371 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f6f3802df50 con 0x7f6f40185e70 2021-04-09T18:21:23.305 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:23.306 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:23.306 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:23.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f240188c0 msgr2=0x7f6f2401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:23.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f240188c0 0x7f6f2401adc0 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f40185e70 msgr2=0x7f6f400ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f40185e70 0x7f6f400ad1d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f6f38009940 tx=0x7f6f3800bcf0).stop 2021-04-09T18:21:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 shutdown_connections 2021-04-09T18:21:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f240188c0 0x7f6f2401adc0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f40185e70 0x7f6f400ad1d0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40185a70 0x7f6f400acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 --2- 172.21.15.32:0/3990928371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f401a5210 0x7f6f40185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 >> 172.21.15.32:0/3990928371 conn(0x7f6f401a99c0 msgr2=0x7f6f4017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 shutdown_connections 2021-04-09T18:21:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:23.258 7f6f466c8700 1 -- 172.21.15.32:0/3990928371 wait complete. 2021-04-09T18:21:23.721 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:23.721 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:24.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:24.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.865 7f473ec30700 1 Processor -- start 2021-04-09T18:21:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.865 7f473ec30700 1 -- start start 2021-04-09T18:21:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.865 7f473ec30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f473810d010 0x7f47380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:24.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.865 7f473ec30700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47380d1da0 0x7f47380d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.865 7f473ec30700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47380d82c0 0x7f473810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.865 7f473ec30700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473810e780 con 0x7f47380d82c0 2021-04-09T18:21:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.865 7f473ec30700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47380d8d30 con 0x7f47380d1da0 2021-04-09T18:21:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.865 7f473ec30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47380d9220 con 0x7f473810d010 2021-04-09T18:21:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473d24d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47380d82c0 0x7f473810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473d24d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47380d82c0 0x7f473810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48218/0 (socket says 172.21.15.32:48218) 2021-04-09T18:21:24.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473d24d700 1 -- 172.21.15.32:0/1967768399 learned_addr learned my addr 172.21.15.32:0/1967768399 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:24.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f4737fff700 1 --2- 172.21.15.32:0/1967768399 >> v2:172.21.15.86:3300/0 conn(0x7f47380d1da0 0x7f47380d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473ca4c700 1 --2- 172.21.15.32:0/1967768399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f473810d010 0x7f47380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473d24d700 1 -- 172.21.15.32:0/1967768399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f473810d010 msgr2=0x7f47380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473d24d700 1 --2- 172.21.15.32:0/1967768399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f473810d010 0x7f47380d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473d24d700 1 -- 172.21.15.32:0/1967768399 >> v2:172.21.15.86:3300/0 conn(0x7f47380d1da0 msgr2=0x7f47380d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473d24d700 1 --2- 172.21.15.32:0/1967768399 >> v2:172.21.15.86:3300/0 conn(0x7f47380d1da0 0x7f47380d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473d24d700 1 -- 172.21.15.32:0/1967768399 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473810d480 con 0x7f47380d82c0 2021-04-09T18:21:24.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f4737fff700 1 --2- 172.21.15.32:0/1967768399 >> v2:172.21.15.86:3300/0 conn(0x7f47380d1da0 0x7f47380d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:24.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473ca4c700 1 --2- 172.21.15.32:0/1967768399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f473810d010 0x7f47380d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473d24d700 1 --2- 172.21.15.32:0/1967768399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47380d82c0 0x7f473810da00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f4730006f00 tx=0x7f4730009f10).ready entity=mon.0 client_cookie=ad584c6ccaa49d82 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f47377fe700 1 -- 172.21.15.32:0/1967768399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f473000b3b0 con 0x7f47380d82c0 2021-04-09T18:21:24.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f47377fe700 1 -- 172.21.15.32:0/1967768399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473000b730 con 0x7f47380d82c0 2021-04-09T18:21:24.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f47377fe700 1 -- 172.21.15.32:0/1967768399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f473000bc10 con 0x7f47380d82c0 2021-04-09T18:21:24.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473ec30700 1 -- 172.21.15.32:0/1967768399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47380d82c0 msgr2=0x7f473810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:24.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.866 7f473ec30700 1 --2- 172.21.15.32:0/1967768399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47380d82c0 0x7f473810da00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f4730006f00 tx=0x7f4730009f10).stop 2021-04-09T18:21:24.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.867 7f473ec30700 1 -- 172.21.15.32:0/1967768399 shutdown_connections 2021-04-09T18:21:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.867 7f473ec30700 1 --2- 172.21.15.32:0/1967768399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47380d82c0 0x7f473810da00 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.867 7f473ec30700 1 --2- 172.21.15.32:0/1967768399 >> v2:172.21.15.86:3300/0 conn(0x7f47380d1da0 0x7f47380d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.867 7f473ec30700 1 --2- 172.21.15.32:0/1967768399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f473810d010 0x7f47380d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.867 7f473ec30700 1 -- 172.21.15.32:0/1967768399 >> 172.21.15.32:0/1967768399 conn(0x7f47380ce5f0 msgr2=0x7f47380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.867 7f473ec30700 1 -- 172.21.15.32:0/1967768399 shutdown_connections 2021-04-09T18:21:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.867 7f473ec30700 1 -- 172.21.15.32:0/1967768399 wait complete. 2021-04-09T18:21:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.887 7f473ec30700 1 Processor -- start 2021-04-09T18:21:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.887 7f473ec30700 1 -- start start 2021-04-09T18:21:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473ec30700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47381a91e0 0x7f47380a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473ec30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47381a95e0 0x7f47380a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473ec30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47381ac850 0x7f47381acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473ec30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47381ad270 con 0x7f47381ac850 2021-04-09T18:21:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473ec30700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47380b7640 con 0x7f47381a91e0 2021-04-09T18:21:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473ec30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4738183c80 con 0x7f47381a95e0 2021-04-09T18:21:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473d24d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47381ac850 0x7f47381acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473d24d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47381ac850 0x7f47381acc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48224/0 (socket says 172.21.15.32:48224) 2021-04-09T18:21:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473d24d700 1 -- 172.21.15.32:0/3383460303 learned_addr learned my addr 172.21.15.32:0/3383460303 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f473ca4c700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47381a91e0 0x7f47380a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.888 7f4737fff700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47381a95e0 0x7f47380a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:24.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f473d24d700 1 -- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47381a95e0 msgr2=0x7f47380a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:24.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f473d24d700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47381a95e0 0x7f47380a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:24.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f473d24d700 1 -- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47381a91e0 msgr2=0x7f47380a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:24.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f473d24d700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47381a91e0 0x7f47380a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:24.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f473d24d700 1 -- 172.21.15.32:0/3383460303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4738183ea0 con 0x7f47381ac850 2021-04-09T18:21:24.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f473ca4c700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47381a91e0 0x7f47380a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:24.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f473d24d700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47381ac850 0x7f47381acc50 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7f473000b8c0 tx=0x7f473000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:24.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f47357fa700 1 -- 172.21.15.32:0/3383460303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4730000f90 con 0x7f47381ac850 2021-04-09T18:21:24.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f47357fa700 1 -- 172.21.15.32:0/3383460303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473000bda0 con 0x7f47381ac850 2021-04-09T18:21:24.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f473ec30700 1 -- 172.21.15.32:0/3383460303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4738183ea0 con 0x7f47381ac850 2021-04-09T18:21:24.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f47357fa700 1 -- 172.21.15.32:0/3383460303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4730001460 con 0x7f47381ac850 2021-04-09T18:21:24.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.889 7f473ec30700 1 -- 172.21.15.32:0/3383460303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4738183ea0 con 0x7f47381ac850 2021-04-09T18:21:24.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.890 7f47357fa700 1 -- 172.21.15.32:0/3383460303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4730005820 con 0x7f47381ac850 2021-04-09T18:21:24.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.890 7f47357fa700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f471c018930 0x7f471c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:24.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.890 7f47357fa700 1 -- 172.21.15.32:0/3383460303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4730032da0 con 0x7f47381ac850 2021-04-09T18:21:24.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.890 7f473ca4c700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f471c018930 0x7f471c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:24.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.890 7f473ca4c700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f471c018930 0x7f471c01ae10 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:24.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.910 7f473ec30700 1 -- 172.21.15.32:0/3383460303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4738182d70 con 0x7f47381ac850 2021-04-09T18:21:24.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:24.913 7f47357fa700 1 -- 172.21.15.32:0/3383460303 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4730000cc0 con 0x7f47381ac850 2021-04-09T18:21:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.104 7f473ec30700 1 -- 172.21.15.32:0/3383460303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f473810d480 con 0x7f47381ac850 2021-04-09T18:21:25.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.105 7f47357fa700 1 -- 172.21.15.32:0/3383460303 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f473000bda0 con 0x7f47381ac850 2021-04-09T18:21:25.156 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:25.157 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:25.157 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:25.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.107 7f473ec30700 1 -- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f471c018930 msgr2=0x7f471c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.107 7f473ec30700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f471c018930 0x7f471c01ae10 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.107 7f473ec30700 1 -- 172.21.15.32:0/3383460303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47381ac850 msgr2=0x7f47381acc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.107 7f473ec30700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47381ac850 0x7f47381acc50 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7f473000b8c0 tx=0x7f473000bc60).stop 2021-04-09T18:21:25.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.108 7f473ec30700 1 -- 172.21.15.32:0/3383460303 shutdown_connections 2021-04-09T18:21:25.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.108 7f473ec30700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f471c018930 0x7f471c01ae10 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:25.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.108 7f473ec30700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47381ac850 0x7f47381acc50 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:25.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.108 7f473ec30700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47381a95e0 0x7f47380a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.108 7f473ec30700 1 --2- 172.21.15.32:0/3383460303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47381a91e0 0x7f47380a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:25.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.108 7f473ec30700 1 -- 172.21.15.32:0/3383460303 >> 172.21.15.32:0/3383460303 conn(0x7f473819b460 msgr2=0x7f473819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:25.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.108 7f473ec30700 1 -- 172.21.15.32:0/3383460303 shutdown_connections 2021-04-09T18:21:25.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:25.108 7f473ec30700 1 -- 172.21.15.32:0/3383460303 wait complete. 2021-04-09T18:21:25.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:25.573 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:26.574 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77af71700 1 Processor -- start 2021-04-09T18:21:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77af71700 1 -- start start 2021-04-09T18:21:26.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77af71700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd77410d010 0x7fd7740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:26.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77af71700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740d1da0 0x7fd7740d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:26.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77af71700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd7740d82c0 0x7fd77410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:26.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77af71700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7740d8d30 con 0x7fd7740d82c0 2021-04-09T18:21:26.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77af71700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7740d9220 con 0x7fd77410d010 2021-04-09T18:21:26.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77af71700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77410d480 con 0x7fd7740d1da0 2021-04-09T18:21:26.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77958e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd7740d82c0 0x7fd77410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:26.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77958e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd7740d82c0 0x7fd77410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48230/0 (socket says 172.21.15.32:48230) 2021-04-09T18:21:26.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.681 7fd77958e700 1 -- 172.21.15.32:0/4098005260 learned_addr learned my addr 172.21.15.32:0/4098005260 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd773fff700 1 --2- 172.21.15.32:0/4098005260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740d1da0 0x7fd7740d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd778d8d700 1 --2- 172.21.15.32:0/4098005260 >> v2:172.21.15.86:3300/0 conn(0x7fd77410d010 0x7fd7740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd77958e700 1 -- 172.21.15.32:0/4098005260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740d1da0 msgr2=0x7fd7740d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd77958e700 1 --2- 172.21.15.32:0/4098005260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740d1da0 0x7fd7740d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd77958e700 1 -- 172.21.15.32:0/4098005260 >> v2:172.21.15.86:3300/0 conn(0x7fd77410d010 msgr2=0x7fd7740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:26.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd77958e700 1 --2- 172.21.15.32:0/4098005260 >> v2:172.21.15.86:3300/0 conn(0x7fd77410d010 0x7fd7740d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd77958e700 1 -- 172.21.15.32:0/4098005260 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7740d86c0 con 0x7fd7740d82c0 2021-04-09T18:21:26.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd773fff700 1 --2- 172.21.15.32:0/4098005260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740d1da0 0x7fd7740d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:26.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd778d8d700 1 --2- 172.21.15.32:0/4098005260 >> v2:172.21.15.86:3300/0 conn(0x7fd77410d010 0x7fd7740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:26.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd77958e700 1 --2- 172.21.15.32:0/4098005260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740d82c0 0x7fd77410da00 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7fd76c006b80 tx=0x7fd76c009b90).ready entity=mon.0 client_cookie=6e881c2a19c35bc3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:26.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd7737fe700 1 -- 172.21.15.32:0/4098005260 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd76c0022e0 con 0x7fd7740d82c0 2021-04-09T18:21:26.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd7737fe700 1 -- 172.21.15.32:0/4098005260 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c002620 con 0x7fd7740d82c0 2021-04-09T18:21:26.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd7737fe700 1 -- 172.21.15.32:0/4098005260 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd76c002b00 con 0x7fd7740d82c0 2021-04-09T18:21:26.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd77af71700 1 -- 172.21.15.32:0/4098005260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740d82c0 msgr2=0x7fd77410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:26.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.682 7fd77af71700 1 --2- 172.21.15.32:0/4098005260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740d82c0 0x7fd77410da00 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7fd76c006b80 tx=0x7fd76c009b90).stop 2021-04-09T18:21:26.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.683 7fd77af71700 1 -- 172.21.15.32:0/4098005260 shutdown_connections 2021-04-09T18:21:26.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.683 7fd77af71700 1 --2- 172.21.15.32:0/4098005260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740d82c0 0x7fd77410da00 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.683 7fd77af71700 1 --2- 172.21.15.32:0/4098005260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740d1da0 0x7fd7740d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.683 7fd77af71700 1 --2- 172.21.15.32:0/4098005260 >> v2:172.21.15.86:3300/0 conn(0x7fd77410d010 0x7fd7740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.683 7fd77af71700 1 -- 172.21.15.32:0/4098005260 >> 172.21.15.32:0/4098005260 conn(0x7fd7740ce5f0 msgr2=0x7fd7740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:26.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.683 7fd77af71700 1 -- 172.21.15.32:0/4098005260 shutdown_connections 2021-04-09T18:21:26.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.683 7fd77af71700 1 -- 172.21.15.32:0/4098005260 wait complete. 2021-04-09T18:21:26.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.703 7fd77af71700 1 Processor -- start 2021-04-09T18:21:26.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.703 7fd77af71700 1 -- start start 2021-04-09T18:21:26.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77af71700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740aa2b0 0x7fd774187620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:26.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77af71700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd774196f70 0x7fd774187c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:26.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77af71700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740b7e90 0x7fd7740b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:26.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77af71700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7740aa6f0 con 0x7fd7740b7e90 2021-04-09T18:21:26.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77af71700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7741870a0 con 0x7fd774196f70 2021-04-09T18:21:26.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77af71700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7740b88b0 con 0x7fd7740aa2b0 2021-04-09T18:21:26.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77958e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740b7e90 0x7fd7740b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:26.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77958e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740b7e90 0x7fd7740b8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48238/0 (socket says 172.21.15.32:48238) 2021-04-09T18:21:26.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77958e700 1 -- 172.21.15.32:0/599641205 learned_addr learned my addr 172.21.15.32:0/599641205 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:26.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd778d8d700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740aa2b0 0x7fd774187620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:26.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd773fff700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd774196f70 0x7fd774187c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:26.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77958e700 1 -- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740aa2b0 msgr2=0x7fd774187620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:26.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77958e700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740aa2b0 0x7fd774187620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77958e700 1 -- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd774196f70 msgr2=0x7fd774187c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:26.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77958e700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd774196f70 0x7fd774187c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77958e700 1 -- 172.21.15.32:0/599641205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7740b8a90 con 0x7fd7740b7e90 2021-04-09T18:21:26.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd778d8d700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740aa2b0 0x7fd774187620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:26.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77958e700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740b7e90 0x7fd7740b8290 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7fd76c006f30 tx=0x7fd76c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:26.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd7717fa700 1 -- 172.21.15.32:0/599641205 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd76c006240 con 0x7fd7740b7e90 2021-04-09T18:21:26.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd7717fa700 1 -- 172.21.15.32:0/599641205 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c000940 con 0x7fd7740b7e90 2021-04-09T18:21:26.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd77af71700 1 -- 172.21.15.32:0/599641205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7740b8a90 con 0x7fd7740b7e90 2021-04-09T18:21:26.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.704 7fd7717fa700 1 -- 172.21.15.32:0/599641205 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd76c0054a0 con 0x7fd7740b7e90 2021-04-09T18:21:26.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.705 7fd77af71700 1 -- 172.21.15.32:0/599641205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7740b8a90 con 0x7fd7740b7e90 2021-04-09T18:21:26.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.705 7fd7717fa700 1 -- 172.21.15.32:0/599641205 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd76c006030 con 0x7fd7740b7e90 2021-04-09T18:21:26.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.705 7fd7717fa700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd758018930 0x7fd75801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:26.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.705 7fd7717fa700 1 -- 172.21.15.32:0/599641205 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd76c032a40 con 0x7fd7740b7e90 2021-04-09T18:21:26.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.705 7fd778d8d700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd758018930 0x7fd75801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:26.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.706 7fd778d8d700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd758018930 0x7fd75801ae10 crc :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:26.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.725 7fd77af71700 1 -- 172.21.15.32:0/599641205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd774182d10 con 0x7fd7740b7e90 2021-04-09T18:21:26.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.729 7fd7717fa700 1 -- 172.21.15.32:0/599641205 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd76c000ba0 con 0x7fd7740b7e90 2021-04-09T18:21:26.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.921 7fd77af71700 1 -- 172.21.15.32:0/599641205 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd77410d480 con 0x7fd7740b7e90 2021-04-09T18:21:26.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.921 7fd7717fa700 1 -- 172.21.15.32:0/599641205 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fd76c00bd50 con 0x7fd7740b7e90 2021-04-09T18:21:26.972 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:26.973 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:26.973 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 -- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd758018930 msgr2=0x7fd75801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd758018930 0x7fd75801ae10 crc :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 -- 172.21.15.32:0/599641205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740b7e90 msgr2=0x7fd7740b8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740b7e90 0x7fd7740b8290 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7fd76c006f30 tx=0x7fd76c00b960).stop 2021-04-09T18:21:26.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 -- 172.21.15.32:0/599641205 shutdown_connections 2021-04-09T18:21:26.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd758018930 0x7fd75801ae10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7740b7e90 0x7fd7740b8290 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd774196f70 0x7fd774187c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 --2- 172.21.15.32:0/599641205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7740aa2b0 0x7fd774187620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:26.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 -- 172.21.15.32:0/599641205 >> 172.21.15.32:0/599641205 conn(0x7fd7741a13a0 msgr2=0x7fd774094bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:26.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 -- 172.21.15.32:0/599641205 shutdown_connections 2021-04-09T18:21:26.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:26.924 7fd77af71700 1 -- 172.21.15.32:0/599641205 wait complete. 2021-04-09T18:21:27.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:27.396 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:28.399 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:28.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.511 7fa528b34700 1 Processor -- start 2021-04-09T18:21:28.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.511 7fa528b34700 1 -- start start 2021-04-09T18:21:28.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.511 7fa528b34700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa52410d010 0x7fa5240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:28.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.511 7fa528b34700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5240d1da0 0x7fa5240d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:28.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.511 7fa528b34700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52410d480 con 0x7fa5240d82c0 2021-04-09T18:21:28.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.511 7fa528b34700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5240d8d30 con 0x7fa52410d010 2021-04-09T18:21:28.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.511 7fa528b34700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5240d9220 con 0x7fa5240d1da0 2021-04-09T18:21:28.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52285e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa52410d010 0x7fa5240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:28.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52285e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa52410d010 0x7fa5240d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48242/0 (socket says 172.21.15.32:48242) 2021-04-09T18:21:28.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52285e700 1 -- 172.21.15.32:0/1943928149 learned_addr learned my addr 172.21.15.32:0/1943928149 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:28.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52205d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5240d1da0 0x7fa5240d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:28.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52285e700 1 -- 172.21.15.32:0/1943928149 >> v2:172.21.15.86:3300/0 conn(0x7fa5240d1da0 msgr2=0x7fa5240d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:28.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52285e700 1 --2- 172.21.15.32:0/1943928149 >> v2:172.21.15.86:3300/0 conn(0x7fa5240d1da0 0x7fa5240d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52285e700 1 -- 172.21.15.32:0/1943928149 >> v1:172.21.15.86:6789/0 conn(0x7fa5240d82c0 legacy=0x7fa52410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:28.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52285e700 1 -- 172.21.15.32:0/1943928149 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5240d86f0 con 0x7fa52410d010 2021-04-09T18:21:28.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52205d700 1 --2- 172.21.15.32:0/1943928149 >> v2:172.21.15.86:3300/0 conn(0x7fa5240d1da0 0x7fa5240d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:28.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52185c700 1 -- 172.21.15.32:0/1943928149 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718714946 0 0) 0x7fa51c000d30 con 0x7fa5240d82c0 2021-04-09T18:21:28.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52285e700 1 --2- 172.21.15.32:0/1943928149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52410d010 0x7fa5240d1780 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7fa514006700 tx=0x7fa514009b80).ready entity=mon.0 client_cookie=cbf40bf1d6ababa6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:28.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52185c700 1 -- 172.21.15.32:0/1943928149 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa51400c0c0 con 0x7fa52410d010 2021-04-09T18:21:28.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52185c700 1 -- 172.21.15.32:0/1943928149 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51400c440 con 0x7fa52410d010 2021-04-09T18:21:28.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa52185c700 1 -- 172.21.15.32:0/1943928149 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa51400c920 con 0x7fa52410d010 2021-04-09T18:21:28.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa528b34700 1 -- 172.21.15.32:0/1943928149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52410d010 msgr2=0x7fa5240d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:28.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.512 7fa528b34700 1 --2- 172.21.15.32:0/1943928149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52410d010 0x7fa5240d1780 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7fa514006700 tx=0x7fa514009b80).stop 2021-04-09T18:21:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.513 7fa528b34700 1 -- 172.21.15.32:0/1943928149 shutdown_connections 2021-04-09T18:21:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.513 7fa528b34700 1 --2- 172.21.15.32:0/1943928149 >> v2:172.21.15.86:3300/0 conn(0x7fa5240d1da0 0x7fa5240d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.513 7fa528b34700 1 --2- 172.21.15.32:0/1943928149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52410d010 0x7fa5240d1780 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.513 7fa528b34700 1 -- 172.21.15.32:0/1943928149 >> 172.21.15.32:0/1943928149 conn(0x7fa5240ce5f0 msgr2=0x7fa5240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:28.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.513 7fa528b34700 1 -- 172.21.15.32:0/1943928149 shutdown_connections 2021-04-09T18:21:28.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.513 7fa528b34700 1 -- 172.21.15.32:0/1943928149 wait complete. 2021-04-09T18:21:28.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa528b34700 1 Processor -- start 2021-04-09T18:21:28.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa528b34700 1 -- start start 2021-04-09T18:21:28.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa528b34700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa524182980 0x7fa524186480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:28.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa528b34700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5241934f0 0x7fa5241938f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:28.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa528b34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52417dc50 0x7fa52417e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:28.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa528b34700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa524193f10 con 0x7fa52417dc50 2021-04-09T18:21:28.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa528b34700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52417e670 con 0x7fa5241934f0 2021-04-09T18:21:28.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa528b34700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa524186aa0 con 0x7fa524182980 2021-04-09T18:21:28.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa52305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52417dc50 0x7fa52417e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:28.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa52305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52417dc50 0x7fa52417e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48252/0 (socket says 172.21.15.32:48252) 2021-04-09T18:21:28.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa52305f700 1 -- 172.21.15.32:0/1703139952 learned_addr learned my addr 172.21.15.32:0/1703139952 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:28.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa52285e700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa524182980 0x7fa524186480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:28.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.534 7fa52205d700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5241934f0 0x7fa5241938f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:28.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa52305f700 1 -- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa524182980 msgr2=0x7fa524186480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:28.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa52305f700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa524182980 0x7fa524186480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa52305f700 1 -- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5241934f0 msgr2=0x7fa5241938f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:28.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa52305f700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5241934f0 0x7fa5241938f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa52305f700 1 -- 172.21.15.32:0/1703139952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa524181920 con 0x7fa52417dc50 2021-04-09T18:21:28.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa52285e700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa524182980 0x7fa524186480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:28.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa52205d700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5241934f0 0x7fa5241938f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:28.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa52305f700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52417dc50 0x7fa52417e050 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7fa51c009980 tx=0x7fa51c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa5137fe700 1 -- 172.21.15.32:0/1703139952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa51c005870 con 0x7fa52417dc50 2021-04-09T18:21:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa5137fe700 1 -- 172.21.15.32:0/1703139952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c005b70 con 0x7fa52417dc50 2021-04-09T18:21:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa528b34700 1 -- 172.21.15.32:0/1703139952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa524181920 con 0x7fa52417dc50 2021-04-09T18:21:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa5137fe700 1 -- 172.21.15.32:0/1703139952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa51c006030 con 0x7fa52417dc50 2021-04-09T18:21:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.535 7fa528b34700 1 -- 172.21.15.32:0/1703139952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa524181920 con 0x7fa52417dc50 2021-04-09T18:21:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.536 7fa5137fe700 1 -- 172.21.15.32:0/1703139952 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa51c006030 con 0x7fa52417dc50 2021-04-09T18:21:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.536 7fa5137fe700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa508018930 0x7fa50801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.536 7fa5137fe700 1 -- 172.21.15.32:0/1703139952 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa51c032e10 con 0x7fa52417dc50 2021-04-09T18:21:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.536 7fa52285e700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa508018930 0x7fa50801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.536 7fa52285e700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa508018930 0x7fa50801adc0 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:28.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.556 7fa528b34700 1 -- 172.21.15.32:0/1703139952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52410d480 con 0x7fa52417dc50 2021-04-09T18:21:28.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.559 7fa5137fe700 1 -- 172.21.15.32:0/1703139952 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa51c00be10 con 0x7fa52417dc50 2021-04-09T18:21:28.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.751 7fa528b34700 1 -- 172.21.15.32:0/1703139952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa52410d480 con 0x7fa52417dc50 2021-04-09T18:21:28.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.751 7fa5137fe700 1 -- 172.21.15.32:0/1703139952 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fa51c00be10 con 0x7fa52417dc50 2021-04-09T18:21:28.802 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:28.803 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:28.803 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:28.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 -- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa508018930 msgr2=0x7fa50801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:28.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa508018930 0x7fa50801adc0 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 -- 172.21.15.32:0/1703139952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52417dc50 msgr2=0x7fa52417e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52417dc50 0x7fa52417e050 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7fa51c009980 tx=0x7fa51c00bcd0).stop 2021-04-09T18:21:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 -- 172.21.15.32:0/1703139952 shutdown_connections 2021-04-09T18:21:28.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa508018930 0x7fa50801adc0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa52417dc50 0x7fa52417e050 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5241934f0 0x7fa5241938f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 --2- 172.21.15.32:0/1703139952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa524182980 0x7fa524186480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:28.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 -- 172.21.15.32:0/1703139952 >> 172.21.15.32:0/1703139952 conn(0x7fa524195e70 msgr2=0x7fa5241a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:28.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 -- 172.21.15.32:0/1703139952 shutdown_connections 2021-04-09T18:21:28.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:28.754 7fa528b34700 1 -- 172.21.15.32:0/1703139952 wait complete. 2021-04-09T18:21:29.222 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:29.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:30.224 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:30.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.346 7f121d999700 1 Processor -- start 2021-04-09T18:21:30.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.346 7f121d999700 1 -- start start 2021-04-09T18:21:30.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.346 7f121d999700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f12180d7ca0 0x7f12180d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:30.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.346 7f121d999700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121810d480 con 0x7f121810da00 2021-04-09T18:21:30.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.346 7f121d999700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12180d86c0 con 0x7f121810d010 2021-04-09T18:21:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.346 7f121d999700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12180d9220 con 0x7f12180d7ca0 2021-04-09T18:21:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.346 7f1217d75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f121810da00 0x7f121810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57848/0 (socket says 172.21.15.32:57848) 2021-04-09T18:21:30.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.346 7f1217d75700 1 -- 172.21.15.32:0/2707016972 learned_addr learned my addr 172.21.15.32:0/2707016972 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216d73700 1 --2- 172.21.15.32:0/2707016972 >> v2:172.21.15.86:3300/0 conn(0x7f12180d7ca0 0x7f12180d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:30.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344477882 0 0) 0x7f1210000d30 con 0x7f121810da00 2021-04-09T18:21:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1200002fe0 con 0x7f121810da00 2021-04-09T18:21:30.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963450632 0 0) 0x7f1208000c60 con 0x7f121810d010 2021-04-09T18:21:30.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1200003210 con 0x7f121810d010 2021-04-09T18:21:30.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3949526967 0 0) 0x7f1210001460 con 0x7f121810da00 2021-04-09T18:21:30.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 >> v2:172.21.15.86:3300/0 conn(0x7f12180d7ca0 msgr2=0x7f12180d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 --2- 172.21.15.32:0/2707016972 >> v2:172.21.15.86:3300/0 conn(0x7f12180d7ca0 0x7f12180d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 >> v1:172.21.15.86:6789/0 conn(0x7f121810d010 legacy=0x7f12180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1218112100 con 0x7f121810da00 2021-04-09T18:21:30.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f1210001980 con 0x7f121810da00 2021-04-09T18:21:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 >> v1:172.21.15.32:6789/0 conn(0x7f121810da00 legacy=0x7f121810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1200003b90 0x7f1200005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:30.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000067a0 0x7f1200008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:30.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000095a0 0x7f120000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:30.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216d73700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000067a0 0x7f1200008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:30.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1200050320 con 0x7f12000067a0 2021-04-09T18:21:30.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1200050500 con 0x7f12000095a0 2021-04-09T18:21:30.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1216572700 1 -- 172.21.15.32:0/2707016972 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1200050720 con 0x7f1200003b90 2021-04-09T18:21:30.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1217574700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1200003b90 0x7f1200005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:30.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.347 7f1217d75700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000095a0 0x7f120000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:30.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1216d73700 1 -- 172.21.15.32:0/2707016972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1200003b90 msgr2=0x7f1200005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:30.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1216d73700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1200003b90 0x7f1200005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1216d73700 1 -- 172.21.15.32:0/2707016972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000095a0 msgr2=0x7f120000bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:30.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1216d73700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000095a0 0x7f120000bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1216d73700 1 -- 172.21.15.32:0/2707016972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12000509c0 con 0x7f12000067a0 2021-04-09T18:21:30.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1217d75700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000095a0 0x7f120000bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:30.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1217d75700 1 -- 172.21.15.32:0/2707016972 reap_dead start 2021-04-09T18:21:30.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1216d73700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000067a0 0x7f1200008ca0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7f120c00c870 tx=0x7f120c00cc10).ready entity=mon.0 client_cookie=73c9c36337502447 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:30.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1216572700 1 -- 172.21.15.32:0/2707016972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f120c0090d0 con 0x7f12000067a0 2021-04-09T18:21:30.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1216572700 1 -- 172.21.15.32:0/2707016972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c009450 con 0x7f12000067a0 2021-04-09T18:21:30.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f1216572700 1 -- 172.21.15.32:0/2707016972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f120c009930 con 0x7f12000067a0 2021-04-09T18:21:30.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f121d999700 1 -- 172.21.15.32:0/2707016972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000067a0 msgr2=0x7f1200008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:30.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.348 7f121d999700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000067a0 0x7f1200008ca0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7f120c00c870 tx=0x7f120c00cc10).stop 2021-04-09T18:21:30.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.349 7f121d999700 1 -- 172.21.15.32:0/2707016972 shutdown_connections 2021-04-09T18:21:30.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.349 7f121d999700 1 --2- 172.21.15.32:0/2707016972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000067a0 0x7f1200008ca0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.349 7f121d999700 1 -- 172.21.15.32:0/2707016972 >> 172.21.15.32:0/2707016972 conn(0x7f12180ce5f0 msgr2=0x7f12180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:30.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.349 7f121d999700 1 -- 172.21.15.32:0/2707016972 shutdown_connections 2021-04-09T18:21:30.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.349 7f121d999700 1 -- 172.21.15.32:0/2707016972 wait complete. 2021-04-09T18:21:30.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.369 7f121d999700 1 Processor -- start 2021-04-09T18:21:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.369 7f121d999700 1 -- start start 2021-04-09T18:21:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f121d999700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12181ac380 0x7f121818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:30.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f121d999700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12181ac780 0x7f121818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:30.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f121d999700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1218183520 0x7f1218183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:30.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f121d999700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12181ace10 con 0x7f12181ac380 2021-04-09T18:21:30.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f121d999700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1218189cf0 con 0x7f1218183520 2021-04-09T18:21:30.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f121d999700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1218183f40 con 0x7f12181ac780 2021-04-09T18:21:30.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12181ac380 0x7f121818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:30.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12181ac380 0x7f121818a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48268/0 (socket says 172.21.15.32:48268) 2021-04-09T18:21:30.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217574700 1 -- 172.21.15.32:0/1679012898 learned_addr learned my addr 172.21.15.32:0/1679012898 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1216d73700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12181ac780 0x7f121818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:30.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217d75700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1218183520 0x7f1218183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217574700 1 -- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12181ac780 msgr2=0x7f121818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:30.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217574700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12181ac780 0x7f121818a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217574700 1 -- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1218183520 msgr2=0x7f1218183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:30.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217574700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1218183520 0x7f1218183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217574700 1 -- 172.21.15.32:0/1679012898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1218184160 con 0x7f12181ac380 2021-04-09T18:21:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217d75700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1218183520 0x7f1218183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:21:30.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1216d73700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12181ac780 0x7f121818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:21:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.370 7f1217574700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12181ac380 0x7f121818a230 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7f12080059f0 tx=0x7f120800a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.371 7f11fffff700 1 -- 172.21.15.32:0/1679012898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f120800ad10 con 0x7f12181ac380 2021-04-09T18:21:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.371 7f11fffff700 1 -- 172.21.15.32:0/1679012898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1208000c10 con 0x7f12181ac380 2021-04-09T18:21:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.371 7f121d999700 1 -- 172.21.15.32:0/1679012898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1218184160 con 0x7f12181ac380 2021-04-09T18:21:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.371 7f11fffff700 1 -- 172.21.15.32:0/1679012898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1208002220 con 0x7f12181ac380 2021-04-09T18:21:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.371 7f121d999700 1 -- 172.21.15.32:0/1679012898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1218184160 con 0x7f12181ac380 2021-04-09T18:21:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.371 7f11fffff700 1 -- 172.21.15.32:0/1679012898 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1208006760 con 0x7f12181ac380 2021-04-09T18:21:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.371 7f11fffff700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11f8018930 0x7f11f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.371 7f11fffff700 1 -- 172.21.15.32:0/1679012898 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f12080329c0 con 0x7f12181ac380 2021-04-09T18:21:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.372 7f1216d73700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11f8018930 0x7f11f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.372 7f1216d73700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11f8018930 0x7f11f801ae10 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.390 7f121d999700 1 -- 172.21.15.32:0/1679012898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1218182d10 con 0x7f12181ac380 2021-04-09T18:21:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.394 7f11fffff700 1 -- 172.21.15.32:0/1679012898 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1208000c10 con 0x7f12181ac380 2021-04-09T18:21:30.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.586 7f121d999700 1 -- 172.21.15.32:0/1679012898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1218182d10 con 0x7f12181ac380 2021-04-09T18:21:30.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.587 7f11fffff700 1 -- 172.21.15.32:0/1679012898 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f1208000e20 con 0x7f12181ac380 2021-04-09T18:21:30.636 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:30.636 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:30.637 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:30.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 -- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11f8018930 msgr2=0x7f11f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:30.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11f8018930 0x7f11f801ae10 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 -- 172.21.15.32:0/1679012898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12181ac380 msgr2=0x7f121818a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:30.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12181ac380 0x7f121818a230 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7f12080059f0 tx=0x7f120800a920).stop 2021-04-09T18:21:30.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 -- 172.21.15.32:0/1679012898 shutdown_connections 2021-04-09T18:21:30.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11f8018930 0x7f11f801ae10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1218183520 0x7f1218183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12181ac780 0x7f121818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 --2- 172.21.15.32:0/1679012898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12181ac380 0x7f121818a230 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:30.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 -- 172.21.15.32:0/1679012898 >> 172.21.15.32:0/1679012898 conn(0x7f121819b890 msgr2=0x7f12181a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:30.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 -- 172.21.15.32:0/1679012898 shutdown_connections 2021-04-09T18:21:30.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:30.589 7f121d999700 1 -- 172.21.15.32:0/1679012898 wait complete. 2021-04-09T18:21:31.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:31.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:32.054 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:32.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.156 7fcf8fafa700 1 Processor -- start 2021-04-09T18:21:32.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.156 7fcf8fafa700 1 -- start start 2021-04-09T18:21:32.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8fafa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8810d010 0x7fcf880d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:32.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8fafa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcf880d8490 0x7fcf8810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:32.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8fafa700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf8810d480 con 0x7fcf880d1da0 2021-04-09T18:21:32.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8fafa700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8810e780 con 0x7fcf880d8490 2021-04-09T18:21:32.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8fafa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880d8d30 con 0x7fcf8810d010 2021-04-09T18:21:32.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8e117700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcf880d8490 0x7fcf8810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:32.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8d115700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcf880d1da0 0x7fcf880d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57870/0 (socket says 172.21.15.32:57870) 2021-04-09T18:21:32.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8d115700 1 -- 172.21.15.32:0/247760581 learned_addr learned my addr 172.21.15.32:0/247760581 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:32.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8d916700 1 --2- 172.21.15.32:0/247760581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8810d010 0x7fcf880d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:32.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8e117700 1 -- 172.21.15.32:0/247760581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8810d010 msgr2=0x7fcf880d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:32.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8e117700 1 --2- 172.21.15.32:0/247760581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8810d010 0x7fcf880d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8e117700 1 -- 172.21.15.32:0/247760581 >> v1:172.21.15.32:6789/0 conn(0x7fcf880d1da0 legacy=0x7fcf880d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:32.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8c914700 1 -- 172.21.15.32:0/247760581 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299625409 0 0) 0x7fcf78000ce0 con 0x7fcf880d1da0 2021-04-09T18:21:32.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8e117700 1 -- 172.21.15.32:0/247760581 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880d9220 con 0x7fcf880d8490 2021-04-09T18:21:32.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8d916700 1 --2- 172.21.15.32:0/247760581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8810d010 0x7fcf880d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:32.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8e117700 1 --2- 172.21.15.32:0/247760581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf880d8490 0x7fcf8810da00 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7fcf80006e30 tx=0x7fcf80009e60).ready entity=mon.0 client_cookie=3c68983dfc247d2f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:32.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8c914700 1 -- 172.21.15.32:0/247760581 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf80006a50 con 0x7fcf880d8490 2021-04-09T18:21:32.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.157 7fcf8c914700 1 -- 172.21.15.32:0/247760581 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8000b300 con 0x7fcf880d8490 2021-04-09T18:21:32.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.158 7fcf8c914700 1 -- 172.21.15.32:0/247760581 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf8000b750 con 0x7fcf880d8490 2021-04-09T18:21:32.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.158 7fcf8fafa700 1 -- 172.21.15.32:0/247760581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf880d8490 msgr2=0x7fcf8810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:32.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.158 7fcf8fafa700 1 --2- 172.21.15.32:0/247760581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf880d8490 0x7fcf8810da00 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7fcf80006e30 tx=0x7fcf80009e60).stop 2021-04-09T18:21:32.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.158 7fcf8fafa700 1 -- 172.21.15.32:0/247760581 shutdown_connections 2021-04-09T18:21:32.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.158 7fcf8fafa700 1 --2- 172.21.15.32:0/247760581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf880d8490 0x7fcf8810da00 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.158 7fcf8fafa700 1 --2- 172.21.15.32:0/247760581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8810d010 0x7fcf880d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.158 7fcf8fafa700 1 -- 172.21.15.32:0/247760581 >> 172.21.15.32:0/247760581 conn(0x7fcf880ce5f0 msgr2=0x7fcf880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:32.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.158 7fcf8fafa700 1 -- 172.21.15.32:0/247760581 shutdown_connections 2021-04-09T18:21:32.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.158 7fcf8fafa700 1 -- 172.21.15.32:0/247760581 wait complete. 2021-04-09T18:21:32.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.178 7fcf8fafa700 1 Processor -- start 2021-04-09T18:21:32.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8fafa700 1 -- start start 2021-04-09T18:21:32.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8fafa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf880b5360 0x7fcf881a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:32.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8fafa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8818e140 0x7fcf881aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:32.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8fafa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf881ad040 0x7fcf8808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:32.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8fafa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881a5360 con 0x7fcf881ad040 2021-04-09T18:21:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8fafa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880902c0 con 0x7fcf8818e140 2021-04-09T18:21:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8fafa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880904e0 con 0x7fcf880b5360 2021-04-09T18:21:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8e117700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf881ad040 0x7fcf8808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:32.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8e117700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf881ad040 0x7fcf8808fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48284/0 (socket says 172.21.15.32:48284) 2021-04-09T18:21:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8e117700 1 -- 172.21.15.32:0/2042969086 learned_addr learned my addr 172.21.15.32:0/2042969086 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8d115700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8818e140 0x7fcf881aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.179 7fcf8d916700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf880b5360 0x7fcf881a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:32.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8e117700 1 -- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf880b5360 msgr2=0x7fcf881a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8e117700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf880b5360 0x7fcf881a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8e117700 1 -- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8818e140 msgr2=0x7fcf881aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8e117700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8818e140 0x7fcf881aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8e117700 1 -- 172.21.15.32:0/2042969086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88183c80 con 0x7fcf881ad040 2021-04-09T18:21:32.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8d916700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf880b5360 0x7fcf881a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:32.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8d115700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8818e140 0x7fcf881aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:32.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8e117700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf881ad040 0x7fcf8808fca0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7fcf8000b970 tx=0x7fcf8000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:32.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf7e7fc700 1 -- 172.21.15.32:0/2042969086 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf80009380 con 0x7fcf881ad040 2021-04-09T18:21:32.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf7e7fc700 1 -- 172.21.15.32:0/2042969086 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf80005820 con 0x7fcf881ad040 2021-04-09T18:21:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf88183c80 con 0x7fcf881ad040 2021-04-09T18:21:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf7e7fc700 1 -- 172.21.15.32:0/2042969086 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf80005d00 con 0x7fcf881ad040 2021-04-09T18:21:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.180 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf88183c80 con 0x7fcf881ad040 2021-04-09T18:21:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.181 7fcf7e7fc700 1 -- 172.21.15.32:0/2042969086 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcf80005a00 con 0x7fcf881ad040 2021-04-09T18:21:32.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.181 7fcf7e7fc700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcf70018930 0x7fcf7001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:32.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.181 7fcf7e7fc700 1 -- 172.21.15.32:0/2042969086 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcf80032dd0 con 0x7fcf881ad040 2021-04-09T18:21:32.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.181 7fcf8d916700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcf70018930 0x7fcf7001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:32.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.181 7fcf8d916700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcf70018930 0x7fcf7001ae10 crc :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:32.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.201 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf88182d70 con 0x7fcf881ad040 2021-04-09T18:21:32.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.204 7fcf7e7fc700 1 -- 172.21.15.32:0/2042969086 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcf80009380 con 0x7fcf881ad040 2021-04-09T18:21:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.396 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf8810d480 con 0x7fcf881ad040 2021-04-09T18:21:32.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.396 7fcf7e7fc700 1 -- 172.21.15.32:0/2042969086 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fcf80005820 con 0x7fcf881ad040 2021-04-09T18:21:32.447 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:32.447 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:32.448 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:32.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcf70018930 msgr2=0x7fcf7001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:32.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcf70018930 0x7fcf7001ae10 crc :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf881ad040 msgr2=0x7fcf8808fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:32.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf881ad040 0x7fcf8808fca0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7fcf8000b970 tx=0x7fcf8000bd10).stop 2021-04-09T18:21:32.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 shutdown_connections 2021-04-09T18:21:32.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcf70018930 0x7fcf7001ae10 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf881ad040 0x7fcf8808fca0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8818e140 0x7fcf881aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 --2- 172.21.15.32:0/2042969086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf880b5360 0x7fcf881a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:32.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 >> 172.21.15.32:0/2042969086 conn(0x7fcf8819b460 msgr2=0x7fcf8819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:32.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 shutdown_connections 2021-04-09T18:21:32.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:32.399 7fcf8fafa700 1 -- 172.21.15.32:0/2042969086 wait complete. 2021-04-09T18:21:32.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:32.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:33.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:34.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.974 7fae9f5c2700 1 Processor -- start 2021-04-09T18:21:34.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.974 7fae9f5c2700 1 -- start start 2021-04-09T18:21:34.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.975 7fae9f5c2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae9810d010 0x7fae980d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:34.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.975 7fae9f5c2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae980d1da0 0x7fae980d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:34.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.975 7fae9f5c2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9810d480 con 0x7fae980d82c0 2021-04-09T18:21:34.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.975 7fae9f5c2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9810e780 con 0x7fae980d1da0 2021-04-09T18:21:34.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.975 7fae9f5c2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae980d8d30 con 0x7fae9810d010 2021-04-09T18:21:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.975 7fae9cbdd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae980d1da0 0x7fae980d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.975 7fae9d3de700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae9810d010 0x7fae980d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.975 7fae9dbdf700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fae980d82c0 0x7fae9810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:55808/0 (socket says 172.21.15.32:55808) 2021-04-09T18:21:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae9dbdf700 1 -- 172.21.15.32:0/3897810508 learned_addr learned my addr 172.21.15.32:0/3897810508 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:34.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae9d3de700 1 -- 172.21.15.32:0/3897810508 >> v2:172.21.15.86:3300/0 conn(0x7fae980d1da0 msgr2=0x7fae980d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae9d3de700 1 --2- 172.21.15.32:0/3897810508 >> v2:172.21.15.86:3300/0 conn(0x7fae980d1da0 0x7fae980d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae9d3de700 1 -- 172.21.15.32:0/3897810508 >> v1:172.21.15.86:6789/0 conn(0x7fae980d82c0 legacy=0x7fae9810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:34.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae8ffff700 1 -- 172.21.15.32:0/3897810508 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921774327 0 0) 0x7fae90001440 con 0x7fae980d82c0 2021-04-09T18:21:34.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae9d3de700 1 -- 172.21.15.32:0/3897810508 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae980d9220 con 0x7fae9810d010 2021-04-09T18:21:34.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae9cbdd700 1 --2- 172.21.15.32:0/3897810508 >> v2:172.21.15.86:3300/0 conn(0x7fae980d1da0 0x7fae980d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae9d3de700 1 --2- 172.21.15.32:0/3897810508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae9810d010 0x7fae980d1780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fae94006650 tx=0x7fae94009a80).ready entity=mon.2 client_cookie=dd11176125eb19ef server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae8ffff700 1 -- 172.21.15.32:0/3897810508 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae940009b0 con 0x7fae9810d010 2021-04-09T18:21:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae8ffff700 1 -- 172.21.15.32:0/3897810508 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9400c0c0 con 0x7fae9810d010 2021-04-09T18:21:34.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae8ffff700 1 -- 172.21.15.32:0/3897810508 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae9400c510 con 0x7fae9810d010 2021-04-09T18:21:34.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae9f5c2700 1 -- 172.21.15.32:0/3897810508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae9810d010 msgr2=0x7fae980d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:34.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.976 7fae9f5c2700 1 --2- 172.21.15.32:0/3897810508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae9810d010 0x7fae980d1780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fae94006650 tx=0x7fae94009a80).stop 2021-04-09T18:21:34.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.977 7fae9f5c2700 1 -- 172.21.15.32:0/3897810508 shutdown_connections 2021-04-09T18:21:34.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.977 7fae9f5c2700 1 --2- 172.21.15.32:0/3897810508 >> v2:172.21.15.86:3300/0 conn(0x7fae980d1da0 0x7fae980d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.977 7fae9f5c2700 1 --2- 172.21.15.32:0/3897810508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae9810d010 0x7fae980d1780 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.977 7fae9f5c2700 1 -- 172.21.15.32:0/3897810508 >> 172.21.15.32:0/3897810508 conn(0x7fae980ce5f0 msgr2=0x7fae980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:34.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.977 7fae9f5c2700 1 -- 172.21.15.32:0/3897810508 shutdown_connections 2021-04-09T18:21:34.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.977 7fae9f5c2700 1 -- 172.21.15.32:0/3897810508 wait complete. 2021-04-09T18:21:34.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.997 7fae9f5c2700 1 Processor -- start 2021-04-09T18:21:34.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9f5c2700 1 -- start start 2021-04-09T18:21:34.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9f5c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae981a91e0 0x7fae980a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:34.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9f5c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae981a95e0 0x7fae980a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:34.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9f5c2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae981ac850 0x7fae981acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:34.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9f5c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae981ad270 con 0x7fae981a91e0 2021-04-09T18:21:34.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9f5c2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae980b7640 con 0x7fae981a95e0 2021-04-09T18:21:34.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9f5c2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae98183c80 con 0x7fae981ac850 2021-04-09T18:21:34.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9d3de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae981a91e0 0x7fae980a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:34.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9d3de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae981a91e0 0x7fae980a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48296/0 (socket says 172.21.15.32:48296) 2021-04-09T18:21:34.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9d3de700 1 -- 172.21.15.32:0/2665704735 learned_addr learned my addr 172.21.15.32:0/2665704735 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:34.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9cbdd700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae981a95e0 0x7fae980a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:34.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.998 7fae9dbdf700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae981ac850 0x7fae981acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:34.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae9cbdd700 1 -- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae981ac850 msgr2=0x7fae981acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:34.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae9cbdd700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae981ac850 0x7fae981acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae9cbdd700 1 -- 172.21.15.32:0/2665704735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae981a91e0 msgr2=0x7fae980a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:34.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae9cbdd700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae981a91e0 0x7fae980a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae9cbdd700 1 -- 172.21.15.32:0/2665704735 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae98183ea0 con 0x7fae981a95e0 2021-04-09T18:21:34.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae9d3de700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae981a91e0 0x7fae980a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:34.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae9cbdd700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae981a95e0 0x7fae980a5080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fae88009540 tx=0x7fae8800b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:34.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae8dffb700 1 -- 172.21.15.32:0/2665704735 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae8800bcd0 con 0x7fae981a95e0 2021-04-09T18:21:34.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae8dffb700 1 -- 172.21.15.32:0/2665704735 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae880063e0 con 0x7fae981a95e0 2021-04-09T18:21:34.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae98183ea0 con 0x7fae981a95e0 2021-04-09T18:21:34.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae8dffb700 1 -- 172.21.15.32:0/2665704735 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae88002030 con 0x7fae981a95e0 2021-04-09T18:21:34.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:33.999 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae98183ea0 con 0x7fae981a95e0 2021-04-09T18:21:34.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.000 7fae8dffb700 1 -- 172.21.15.32:0/2665704735 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fae88009fe0 con 0x7fae981a95e0 2021-04-09T18:21:34.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.000 7fae8dffb700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae780188d0 0x7fae7801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:34.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.000 7fae8dffb700 1 -- 172.21.15.32:0/2665704735 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fae88032910 con 0x7fae981a95e0 2021-04-09T18:21:34.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.000 7fae9d3de700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae780188d0 0x7fae7801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:34.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.001 7fae9d3de700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae780188d0 0x7fae7801add0 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:34.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.019 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae98182d70 con 0x7fae981a95e0 2021-04-09T18:21:34.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.023 7fae8dffb700 1 -- 172.21.15.32:0/2665704735 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fae88000c90 con 0x7fae981a95e0 2021-04-09T18:21:34.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.214 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae9810d480 con 0x7fae981a95e0 2021-04-09T18:21:34.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.215 7fae8dffb700 1 -- 172.21.15.32:0/2665704735 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fae88009fe0 con 0x7fae981a95e0 2021-04-09T18:21:34.265 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:34.265 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:34.266 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:34.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae780188d0 msgr2=0x7fae7801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:34.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae780188d0 0x7fae7801add0 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae981a95e0 msgr2=0x7fae980a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:34.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae981a95e0 0x7fae980a5080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fae88009540 tx=0x7fae8800b8e0).stop 2021-04-09T18:21:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 shutdown_connections 2021-04-09T18:21:34.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae780188d0 0x7fae7801add0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae981ac850 0x7fae981acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae981a95e0 0x7fae980a5080 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 --2- 172.21.15.32:0/2665704735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae981a91e0 0x7fae980a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 >> 172.21.15.32:0/2665704735 conn(0x7fae9819b460 msgr2=0x7fae9819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:34.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 shutdown_connections 2021-04-09T18:21:34.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:34.217 7fae9f5c2700 1 -- 172.21.15.32:0/2665704735 wait complete. 2021-04-09T18:21:34.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:34.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:35.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:35.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.793 7fa218375700 1 Processor -- start 2021-04-09T18:21:35.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.793 7fa218375700 1 -- start start 2021-04-09T18:21:35.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.794 7fa218375700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa2100d7ca0 0x7fa2100d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:35.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.794 7fa218375700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21010d480 con 0x7fa21010d010 2021-04-09T18:21:35.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.794 7fa218375700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2100d86c0 con 0x7fa21010da00 2021-04-09T18:21:35.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.794 7fa218375700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2100d9220 con 0x7fa2100d7ca0 2021-04-09T18:21:35.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.794 7fa215990700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa2100d7ca0 0x7fa2100d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:35.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.794 7fa216191700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa21010d010 0x7fa2100d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57894/0 (socket says 172.21.15.32:57894) 2021-04-09T18:21:35.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.794 7fa216191700 1 -- 172.21.15.32:0/2924276548 learned_addr learned my addr 172.21.15.32:0/2924276548 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:35.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.794 7fa215990700 1 -- 172.21.15.32:0/2924276548 >> v1:172.21.15.86:6789/0 conn(0x7fa21010da00 legacy=0x7fa21010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:35.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.795 7fa215990700 1 -- 172.21.15.32:0/2924276548 >> v1:172.21.15.32:6789/0 conn(0x7fa21010d010 legacy=0x7fa2100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:35.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.795 7fa215990700 1 -- 172.21.15.32:0/2924276548 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2100d1f70 con 0x7fa2100d7ca0 2021-04-09T18:21:35.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.795 7fa21518f700 1 -- 172.21.15.32:0/2924276548 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337981623 0 0) 0x7fa20c000ce0 con 0x7fa21010d010 2021-04-09T18:21:35.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.795 7fa215990700 1 --2- 172.21.15.32:0/2924276548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100d7ca0 0x7fa2100d80a0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7fa204006650 tx=0x7fa204009a00).ready entity=mon.0 client_cookie=a2073e848b189331 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:35.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.795 7fa21518f700 1 -- 172.21.15.32:0/2924276548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa204009df0 con 0x7fa2100d7ca0 2021-04-09T18:21:35.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.795 7fa21518f700 1 -- 172.21.15.32:0/2924276548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2040009b0 con 0x7fa2100d7ca0 2021-04-09T18:21:35.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.795 7fa21518f700 1 -- 172.21.15.32:0/2924276548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa20400c2b0 con 0x7fa2100d7ca0 2021-04-09T18:21:35.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.795 7fa218375700 1 -- 172.21.15.32:0/2924276548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100d7ca0 msgr2=0x7fa2100d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:35.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.795 7fa218375700 1 --2- 172.21.15.32:0/2924276548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100d7ca0 0x7fa2100d80a0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7fa204006650 tx=0x7fa204009a00).stop 2021-04-09T18:21:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.796 7fa218375700 1 -- 172.21.15.32:0/2924276548 shutdown_connections 2021-04-09T18:21:35.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.796 7fa218375700 1 --2- 172.21.15.32:0/2924276548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2100d7ca0 0x7fa2100d80a0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.796 7fa218375700 1 -- 172.21.15.32:0/2924276548 >> 172.21.15.32:0/2924276548 conn(0x7fa2100ce5f0 msgr2=0x7fa2100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.796 7fa218375700 1 -- 172.21.15.32:0/2924276548 shutdown_connections 2021-04-09T18:21:35.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.796 7fa218375700 1 -- 172.21.15.32:0/2924276548 wait complete. 2021-04-09T18:21:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.816 7fa218375700 1 Processor -- start 2021-04-09T18:21:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.816 7fa218375700 1 -- start start 2021-04-09T18:21:35.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa218375700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2100c10d0 0x7fa2101bffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa218375700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2101c05d0 0x7fa2100bdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa218375700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2100be590 0x7fa2101a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:35.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa218375700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2101abd30 con 0x7fa2101c05d0 2021-04-09T18:21:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa218375700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100bd9f0 con 0x7fa2100c10d0 2021-04-09T18:21:35.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa218375700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100c14d0 con 0x7fa2100be590 2021-04-09T18:21:35.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa215990700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2101c05d0 0x7fa2100bdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:35.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa215990700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2101c05d0 0x7fa2100bdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48312/0 (socket says 172.21.15.32:48312) 2021-04-09T18:21:35.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa215990700 1 -- 172.21.15.32:0/4090612150 learned_addr learned my addr 172.21.15.32:0/4090612150 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa216191700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2100c10d0 0x7fa2101bffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:35.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa216992700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2100be590 0x7fa2101a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa215990700 1 -- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2100be590 msgr2=0x7fa2101a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa215990700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2100be590 0x7fa2101a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:35.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa215990700 1 -- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2100c10d0 msgr2=0x7fa2101bffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa215990700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2100c10d0 0x7fa2101bffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa215990700 1 -- 172.21.15.32:0/4090612150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2100be990 con 0x7fa2101c05d0 2021-04-09T18:21:35.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa216191700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2100c10d0 0x7fa2101bffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa215990700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2101c05d0 0x7fa2100bdf70 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7fa20400b580 tx=0x7fa20400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:35.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa202ffd700 1 -- 172.21.15.32:0/4090612150 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa204000c90 con 0x7fa2101c05d0 2021-04-09T18:21:35.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa202ffd700 1 -- 172.21.15.32:0/4090612150 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa204002030 con 0x7fa2101c05d0 2021-04-09T18:21:35.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa218375700 1 -- 172.21.15.32:0/4090612150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2100be990 con 0x7fa2101c05d0 2021-04-09T18:21:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa202ffd700 1 -- 172.21.15.32:0/4090612150 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2040024c0 con 0x7fa2101c05d0 2021-04-09T18:21:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.817 7fa218375700 1 -- 172.21.15.32:0/4090612150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2100be990 con 0x7fa2101c05d0 2021-04-09T18:21:35.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.818 7fa202ffd700 1 -- 172.21.15.32:0/4090612150 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa2040024a0 con 0x7fa2101c05d0 2021-04-09T18:21:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.818 7fa202ffd700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1f80188d0 0x7fa1f801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.818 7fa202ffd700 1 -- 172.21.15.32:0/4090612150 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa204032a50 con 0x7fa2101c05d0 2021-04-09T18:21:35.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.818 7fa216191700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1f80188d0 0x7fa1f801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:35.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.819 7fa216191700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1f80188d0 0x7fa1f801add0 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.838 7fa218375700 1 -- 172.21.15.32:0/4090612150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2101abd30 con 0x7fa2101c05d0 2021-04-09T18:21:35.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:35.841 7fa202ffd700 1 -- 172.21.15.32:0/4090612150 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa204002290 con 0x7fa2101c05d0 2021-04-09T18:21:36.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.032 7fa218375700 1 -- 172.21.15.32:0/4090612150 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2101abd30 con 0x7fa2101c05d0 2021-04-09T18:21:36.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.033 7fa202ffd700 1 -- 172.21.15.32:0/4090612150 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7fa2040024a0 con 0x7fa2101c05d0 2021-04-09T18:21:36.083 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:36.084 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:36.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 -- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1f80188d0 msgr2=0x7fa1f801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1f80188d0 0x7fa1f801add0 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 -- 172.21.15.32:0/4090612150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2101c05d0 msgr2=0x7fa2100bdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2101c05d0 0x7fa2100bdf70 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7fa20400b580 tx=0x7fa20400b920).stop 2021-04-09T18:21:36.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 -- 172.21.15.32:0/4090612150 shutdown_connections 2021-04-09T18:21:36.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1f80188d0 0x7fa1f801add0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:36.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2101c05d0 0x7fa2100bdf70 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:36.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2100be590 0x7fa2101a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:36.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 --2- 172.21.15.32:0/4090612150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2100c10d0 0x7fa2101bffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.035 7fa218375700 1 -- 172.21.15.32:0/4090612150 >> 172.21.15.32:0/4090612150 conn(0x7fa2101acbe0 msgr2=0x7fa21008dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.036 7fa218375700 1 -- 172.21.15.32:0/4090612150 shutdown_connections 2021-04-09T18:21:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:36.036 7fa218375700 1 -- 172.21.15.32:0/4090612150 wait complete. 2021-04-09T18:21:36.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:21:36.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:21:36.499 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 19, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:18:33.664336', 'modified': '2021-04-09 18:18:54.784269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 6526}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_6526': {'gid': 6526, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2067624954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2067624954}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2067624954}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:36.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2021-04-09T18:21:37.502 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 101, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 7, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:18:33.664336", "damaged": [], "data_pools": [ 15 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 19, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6526": { "addr": "172.21.15.32:6835/2067624954", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6833", "nonce": 2067624954, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 2067624954, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6526, "incarnation": 88, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 2, "metadata_pool": 14, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:18:54.784269", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [ 1 ], "tableserver": 0, "up": { "mds_0": 6526 } } } ], "standbys": [ { "addr": "172.21.15.32:6834/34806106", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 34806106, "type": "v2" }, { "addr": "172.21.15.32:6834", "nonce": 34806106, "type": "v1" } ] }, "epoch": 101, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6757, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6836/2928911362", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 2928911362, "type": "v2" }, { "addr": "172.21.15.86:6836", "nonce": 2928911362, "type": "v1" } ] }, "epoch": 87, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55799, "incarnation": 0, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6837/3460852521", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6835", "nonce": 3460852521, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 3460852521, "type": "v1" } ] }, "epoch": 87, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55802, "incarnation": 0, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:21:37.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:37.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.601 7f2755b4a700 1 Processor -- start 2021-04-09T18:21:37.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.601 7f2755b4a700 1 -- start start 2021-04-09T18:21:37.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f2755b4a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f275010d010 0x7f27500d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:37.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f2755b4a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f27500d1da0 0x7f27500d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:37.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f2755b4a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27500d82c0 0x7f275010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:37.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f2755b4a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27500d8d30 con 0x7f275010d010 2021-04-09T18:21:37.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f2755b4a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27500d9220 con 0x7f27500d1da0 2021-04-09T18:21:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f2755b4a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275010d4b0 con 0x7f27500d82c0 2021-04-09T18:21:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f275010d010 0x7f27500d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:37.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f275010d010 0x7f27500d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48318/0 (socket says 172.21.15.32:48318) 2021-04-09T18:21:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274f7fe700 1 -- 172.21.15.32:0/3612802422 learned_addr learned my addr 172.21.15.32:0/3612802422 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274ffff700 1 --2- 172.21.15.32:0/3612802422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27500d82c0 0x7f275010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274effd700 1 --2- 172.21.15.32:0/3612802422 >> v2:172.21.15.86:3300/0 conn(0x7f27500d1da0 0x7f27500d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:37.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274f7fe700 1 -- 172.21.15.32:0/3612802422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27500d82c0 msgr2=0x7f275010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274f7fe700 1 --2- 172.21.15.32:0/3612802422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27500d82c0 0x7f275010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274f7fe700 1 -- 172.21.15.32:0/3612802422 >> v2:172.21.15.86:3300/0 conn(0x7f27500d1da0 msgr2=0x7f27500d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274f7fe700 1 --2- 172.21.15.32:0/3612802422 >> v2:172.21.15.86:3300/0 conn(0x7f27500d1da0 0x7f27500d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274f7fe700 1 -- 172.21.15.32:0/3612802422 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27500d86c0 con 0x7f275010d010 2021-04-09T18:21:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274ffff700 1 --2- 172.21.15.32:0/3612802422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27500d82c0 0x7f275010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274effd700 1 --2- 172.21.15.32:0/3612802422 >> v2:172.21.15.86:3300/0 conn(0x7f27500d1da0 0x7f27500d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.602 7f274f7fe700 1 --2- 172.21.15.32:0/3612802422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f275010d010 0x7f27500d1780 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f2740006700 tx=0x7f2740009b80).ready entity=mon.0 client_cookie=c3e82b846f647998 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:37.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f274e7fc700 1 -- 172.21.15.32:0/3612802422 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f274000c0c0 con 0x7f275010d010 2021-04-09T18:21:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f274e7fc700 1 -- 172.21.15.32:0/3612802422 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274000c440 con 0x7f275010d010 2021-04-09T18:21:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f274e7fc700 1 -- 172.21.15.32:0/3612802422 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f274000c920 con 0x7f275010d010 2021-04-09T18:21:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f2755b4a700 1 -- 172.21.15.32:0/3612802422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f275010d010 msgr2=0x7f27500d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f2755b4a700 1 --2- 172.21.15.32:0/3612802422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f275010d010 0x7f27500d1780 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f2740006700 tx=0x7f2740009b80).stop 2021-04-09T18:21:37.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f2755b4a700 1 -- 172.21.15.32:0/3612802422 shutdown_connections 2021-04-09T18:21:37.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f2755b4a700 1 --2- 172.21.15.32:0/3612802422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27500d82c0 0x7f275010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f2755b4a700 1 --2- 172.21.15.32:0/3612802422 >> v2:172.21.15.86:3300/0 conn(0x7f27500d1da0 0x7f27500d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f2755b4a700 1 --2- 172.21.15.32:0/3612802422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f275010d010 0x7f27500d1780 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f2755b4a700 1 -- 172.21.15.32:0/3612802422 >> 172.21.15.32:0/3612802422 conn(0x7f27500ce5f0 msgr2=0x7f27500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:37.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f2755b4a700 1 -- 172.21.15.32:0/3612802422 shutdown_connections 2021-04-09T18:21:37.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.603 7f2755b4a700 1 -- 172.21.15.32:0/3612802422 wait complete. 2021-04-09T18:21:37.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.623 7f2755b4a700 1 Processor -- start 2021-04-09T18:21:37.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f2755b4a700 1 -- start start 2021-04-09T18:21:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f2755b4a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27500b42d0 0x7f2750183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f2755b4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27501ac090 0x7f27501ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f2755b4a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2750184d30 0x7f2750185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f2755b4a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27500a1a90 con 0x7f27501ac090 2021-04-09T18:21:37.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f2755b4a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27501acab0 con 0x7f27500b42d0 2021-04-09T18:21:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f2755b4a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2750185a00 con 0x7f2750184d30 2021-04-09T18:21:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f274effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27501ac090 0x7f27501ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f274effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27501ac090 0x7f27501ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48324/0 (socket says 172.21.15.32:48324) 2021-04-09T18:21:37.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.624 7f274effd700 1 -- 172.21.15.32:0/1746052694 learned_addr learned my addr 172.21.15.32:0/1746052694 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f274f7fe700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27500b42d0 0x7f2750183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f274ffff700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2750184d30 0x7f2750185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f274effd700 1 -- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2750184d30 msgr2=0x7f2750185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:37.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f274effd700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2750184d30 0x7f2750185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f274effd700 1 -- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27500b42d0 msgr2=0x7f2750183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f274effd700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27500b42d0 0x7f2750183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f274effd700 1 -- 172.21.15.32:0/1746052694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2750185750 con 0x7f27501ac090 2021-04-09T18:21:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f274f7fe700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27500b42d0 0x7f2750183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:37.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f274effd700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27501ac090 0x7f27501ac490 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7f273c006f40 tx=0x7f273c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f2747fff700 1 -- 172.21.15.32:0/1746052694 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f273c00bd80 con 0x7f27501ac090 2021-04-09T18:21:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f2747fff700 1 -- 172.21.15.32:0/1746052694 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c000c90 con 0x7f27501ac090 2021-04-09T18:21:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2750185750 con 0x7f27501ac090 2021-04-09T18:21:37.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f2747fff700 1 -- 172.21.15.32:0/1746052694 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f273c002220 con 0x7f27501ac090 2021-04-09T18:21:37.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.625 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2750185750 con 0x7f27501ac090 2021-04-09T18:21:37.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.626 7f2747fff700 1 -- 172.21.15.32:0/1746052694 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f273c000c90 con 0x7f27501ac090 2021-04-09T18:21:37.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.626 7f2747fff700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f272c0188c0 0x7f272c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:37.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.627 7f2747fff700 1 -- 172.21.15.32:0/1746052694 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f273c0327c0 con 0x7f27501ac090 2021-04-09T18:21:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.627 7f274f7fe700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f272c0188c0 0x7f272c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:37.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.627 7f274f7fe700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f272c0188c0 0x7f272c01adc0 crc :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:37.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.646 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2750182d10 con 0x7f27501ac090 2021-04-09T18:21:37.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.649 7f2747fff700 1 -- 172.21.15.32:0/1746052694 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f273c006b00 con 0x7f27501ac090 2021-04-09T18:21:37.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.840 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f275010d480 con 0x7f27501ac090 2021-04-09T18:21:37.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.841 7f2747fff700 1 -- 172.21.15.32:0/1746052694 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f273c00bd80 con 0x7f27501ac090 2021-04-09T18:21:37.891 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:37.892 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:37.892 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f272c0188c0 msgr2=0x7f272c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f272c0188c0 0x7f272c01adc0 crc :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27501ac090 msgr2=0x7f27501ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:37.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27501ac090 0x7f27501ac490 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7f273c006f40 tx=0x7f273c00b990).stop 2021-04-09T18:21:37.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 shutdown_connections 2021-04-09T18:21:37.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f272c0188c0 0x7f272c01adc0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27501ac090 0x7f27501ac490 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2750184d30 0x7f2750185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 --2- 172.21.15.32:0/1746052694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27500b42d0 0x7f2750183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:37.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.843 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 >> 172.21.15.32:0/1746052694 conn(0x7f275019b890 msgr2=0x7f27501a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:37.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.844 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 shutdown_connections 2021-04-09T18:21:37.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:37.844 7f2755b4a700 1 -- 172.21.15.32:0/1746052694 wait complete. 2021-04-09T18:21:37.904 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 101, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 7, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:18:33.664336", "damaged": [], "data_pools": [ 15 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 19, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6526": { "addr": "172.21.15.32:6835/2067624954", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6833", "nonce": 2067624954, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 2067624954, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6526, "incarnation": 88, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 2, "metadata_pool": 14, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:18:54.784269", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [ 1 ], "tableserver": 0, "up": { "mds_0": 6526 } } } ], "standbys": [ { "addr": "172.21.15.32:6834/34806106", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 34806106, "type": "v2" }, { "addr": "172.21.15.32:6834", "nonce": 34806106, "type": "v1" } ] }, "epoch": 101, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 6757, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6836/2928911362", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 2928911362, "type": "v2" }, { "addr": "172.21.15.86:6836", "nonce": 2928911362, "type": "v1" } ] }, "epoch": 87, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55799, "incarnation": 0, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.86:6837/3460852521", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6835", "nonce": 3460852521, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 3460852521, "type": "v1" } ] }, "epoch": 87, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 55802, "incarnation": 0, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:21:37.905 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:21:37.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:21:37.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:21:38.022 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:21:38.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:21:38.023 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:21:38.024 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:21:38.070 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:21:38.070 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:21:38.071 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:21:38.072 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:21:38.073 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:21:38.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[50636]: fuse finished with error 0 and tester_r 0 2021-04-09T18:21:38.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:21:38.144 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:21:38.144 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:21:38.188 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:21:38.188 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:21:38.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:21:38.195 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:21:38.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:21:38.278 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:21:38.279 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:21:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:21:38.322 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:21:38.322 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:21:38.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:21:38.323 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:21:38.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:21:38.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.502 7fa4be1c8700 1 Processor -- start 2021-04-09T18:21:38.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.502 7fa4be1c8700 1 -- start start 2021-04-09T18:21:38.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.503 7fa4be1c8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4b810d010 0x7fa4b80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:38.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.503 7fa4be1c8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b810d480 con 0x7fa4b80d1da0 2021-04-09T18:21:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.503 7fa4be1c8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b810e1f0 con 0x7fa4b80d8490 2021-04-09T18:21:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.503 7fa4be1c8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b80d8d30 con 0x7fa4b810d010 2021-04-09T18:21:38.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.503 7fa4b6ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa4b80d1da0 0x7fa4b80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57926/0 (socket says 172.21.15.32:57926) 2021-04-09T18:21:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.503 7fa4b6ffd700 1 -- 172.21.15.32:0/628696296 learned_addr learned my addr 172.21.15.32:0/628696296 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:38.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.503 7fa4b77fe700 1 --2- 172.21.15.32:0/628696296 >> v2:172.21.15.86:3300/0 conn(0x7fa4b810d010 0x7fa4b80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:38.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.503 7fa4b67fc700 1 -- 172.21.15.32:0/628696296 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762428881 0 0) 0x7fa4a0000ce0 con 0x7fa4b80d1da0 2021-04-09T18:21:38.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.503 7fa4b67fc700 1 -- 172.21.15.32:0/628696296 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4002fe0 con 0x7fa4b80d1da0 2021-04-09T18:21:38.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4b67fc700 1 -- 172.21.15.32:0/628696296 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838801483 0 0) 0x7fa4b0000d30 con 0x7fa4b80d8490 2021-04-09T18:21:38.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4b67fc700 1 -- 172.21.15.32:0/628696296 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4003210 con 0x7fa4b80d8490 2021-04-09T18:21:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4b77fe700 1 -- 172.21.15.32:0/628696296 >> v1:172.21.15.86:6789/0 conn(0x7fa4b80d8490 legacy=0x7fa4b810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4b77fe700 1 -- 172.21.15.32:0/628696296 >> v1:172.21.15.32:6789/0 conn(0x7fa4b80d1da0 legacy=0x7fa4b80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4b77fe700 1 -- 172.21.15.32:0/628696296 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b80d9220 con 0x7fa4b810d010 2021-04-09T18:21:38.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4b77fe700 1 --2- 172.21.15.32:0/628696296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b810d010 0x7fa4b80d1780 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fa4a8006650 tx=0x7fa4a8009a80).ready entity=mon.1 client_cookie=d025f43516f31c51 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4b67fc700 1 -- 172.21.15.32:0/628696296 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4a80009b0 con 0x7fa4b810d010 2021-04-09T18:21:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4b67fc700 1 -- 172.21.15.32:0/628696296 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a800c0c0 con 0x7fa4b810d010 2021-04-09T18:21:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4b67fc700 1 -- 172.21.15.32:0/628696296 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4a800c510 con 0x7fa4b810d010 2021-04-09T18:21:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4be1c8700 1 -- 172.21.15.32:0/628696296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b810d010 msgr2=0x7fa4b80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:38.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.504 7fa4be1c8700 1 --2- 172.21.15.32:0/628696296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b810d010 0x7fa4b80d1780 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fa4a8006650 tx=0x7fa4a8009a80).stop 2021-04-09T18:21:38.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.505 7fa4be1c8700 1 -- 172.21.15.32:0/628696296 shutdown_connections 2021-04-09T18:21:38.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.505 7fa4be1c8700 1 --2- 172.21.15.32:0/628696296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b810d010 0x7fa4b80d1780 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:38.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.505 7fa4be1c8700 1 -- 172.21.15.32:0/628696296 >> 172.21.15.32:0/628696296 conn(0x7fa4b80ce5f0 msgr2=0x7fa4b80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:38.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.505 7fa4be1c8700 1 -- 172.21.15.32:0/628696296 shutdown_connections 2021-04-09T18:21:38.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.505 7fa4be1c8700 1 -- 172.21.15.32:0/628696296 wait complete. 2021-04-09T18:21:38.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.525 7fa4be1c8700 1 Processor -- start 2021-04-09T18:21:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.525 7fa4be1c8700 1 -- start start 2021-04-09T18:21:38.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4be1c8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b80a8b10 0x7fa4b81b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:38.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4be1c8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b81b8b60 0x7fa4b81ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4be1c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b80a8f10 0x7fa4b81abc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:38.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4be1c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b81ac220 con 0x7fa4b80a8f10 2021-04-09T18:21:38.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4be1c8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b808cf40 con 0x7fa4b80a8b10 2021-04-09T18:21:38.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4be1c8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b808d160 con 0x7fa4b81b8b60 2021-04-09T18:21:38.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b80a8f10 0x7fa4b81abc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b80a8f10 0x7fa4b81abc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48342/0 (socket says 172.21.15.32:48342) 2021-04-09T18:21:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b7fff700 1 -- 172.21.15.32:0/1601702316 learned_addr learned my addr 172.21.15.32:0/1601702316 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:38.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b6ffd700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b81b8b60 0x7fa4b81ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:38.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b77fe700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b80a8b10 0x7fa4b81b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:38.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b7fff700 1 -- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b81b8b60 msgr2=0x7fa4b81ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b7fff700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b81b8b60 0x7fa4b81ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b7fff700 1 -- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b80a8b10 msgr2=0x7fa4b81b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:38.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b7fff700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b80a8b10 0x7fa4b81b8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b7fff700 1 -- 172.21.15.32:0/1601702316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b808d380 con 0x7fa4b80a8f10 2021-04-09T18:21:38.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4b7fff700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b80a8f10 0x7fa4b81abc00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7fa4b0009980 tx=0x7fa4b000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4affff700 1 -- 172.21.15.32:0/1601702316 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4b000be10 con 0x7fa4b80a8f10 2021-04-09T18:21:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4affff700 1 -- 172.21.15.32:0/1601702316 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b0005870 con 0x7fa4b80a8f10 2021-04-09T18:21:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b808d380 con 0x7fa4b80a8f10 2021-04-09T18:21:38.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4affff700 1 -- 172.21.15.32:0/1601702316 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4b0005d50 con 0x7fa4b80a8f10 2021-04-09T18:21:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.526 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b808d380 con 0x7fa4b80a8f10 2021-04-09T18:21:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.527 7fa4affff700 1 -- 172.21.15.32:0/1601702316 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa4b0005a50 con 0x7fa4b80a8f10 2021-04-09T18:21:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.527 7fa4affff700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa49c0188d0 0x7fa49c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.527 7fa4affff700 1 -- 172.21.15.32:0/1601702316 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa4b0032d80 con 0x7fa4b80a8f10 2021-04-09T18:21:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.527 7fa4b77fe700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa49c0188d0 0x7fa49c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.528 7fa4b77fe700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa49c0188d0 0x7fa49c01add0 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.547 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b81ab0b0 con 0x7fa4b80a8f10 2021-04-09T18:21:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.551 7fa4affff700 1 -- 172.21.15.32:0/1601702316 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa4b000be10 con 0x7fa4b80a8f10 2021-04-09T18:21:38.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.738 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa4b81ab0b0 con 0x7fa4b80a8f10 2021-04-09T18:21:38.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.739 7fa4affff700 1 -- 172.21.15.32:0/1601702316 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v84) v1 ==== 81+0+30363 (secure 0 0 0) 0x7fa4b0006f20 con 0x7fa4b80a8f10 2021-04-09T18:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:21:38.792 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 84, 2021-04-09T18:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:18:33.664214", 2021-04-09T18:21:38.793 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:21:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:21:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:21:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:21:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:21:38.794 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:21:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:21:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:21:38.795 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:21:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:21:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:21:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:21:38.796 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:21:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:21:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:21:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 15, 2021-04-09T18:21:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:21:38.797 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:21:38.798 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:21:38.798 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:21:38.798 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:21:38.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.799 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:21:38.799 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:21:38.799 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:21:38.799 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:21:38.799 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:21:38.800 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:21:38.800 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:21:38.800 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:21:38.801 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:21:38.801 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:21:38.802 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:21:38.802 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:21:38.802 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:21:38.802 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:21:38.803 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:21:38.803 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:21:38.803 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:21:38.803 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:21:38.804 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:21:38.804 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:21:38.804 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:21:38.804 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:21:38.804 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:21:38.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.805 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:21:38.805 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:21:38.805 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:21:38.806 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:21:38.806 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:21:38.806 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:21:38.806 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:21:38.806 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:21:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:21:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:21:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:21:38.807 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:21:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:21:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:21:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:21:38.808 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:21:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:21:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:21:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:21:38.809 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:21:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:21:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:21:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:21:38.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:21:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:21:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:21:38.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:21:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:21:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:21:38.812 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:21:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:21:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:21:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:21:38.813 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:21:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:21:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:21:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:21:38.814 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:21:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:21:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:21:38.815 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:21:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 14, 2021-04-09T18:21:38.816 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:21:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:18:32.041217", 2021-04-09T18:21:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:21:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:21:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:21:38.817 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:21:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:21:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:21:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:21:38.818 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:21:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:21:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:21:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:21:38.819 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:21:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:21:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:21:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:21:38.820 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:21:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:21:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:21:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:21:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:21:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "84", 2021-04-09T18:21:38.822 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:21:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:21:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:21:38.823 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:21:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:21:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:21:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:21:38.824 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:21:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:21:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:21:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:21:38.825 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:21:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:21:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:21:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:21:38.826 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:21:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:21:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:21:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:21:38.827 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:21:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:21:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:21:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:21:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:21:38.828 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:21:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:21:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:21:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:21:38.829 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:21:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:21:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:21:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:21:38.830 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:21:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:21:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:21:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:21:38.831 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:21:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:21:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:21:38.832 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:21:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:21:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:21:38.833 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:21:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:21:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 15, 2021-04-09T18:21:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:21:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:18:33.125737", 2021-04-09T18:21:38.835 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:21:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:21:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:21:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:21:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:21:38.836 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:21:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:21:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:21:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:21:38.837 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:21:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:21:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:21:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:21:38.838 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:21:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:21:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:21:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:21:38.839 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:21:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:21:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:21:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.840 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "84", 2021-04-09T18:21:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:21:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:21:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:21:38.841 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:21:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:21:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:21:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:21:38.842 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:21:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:21:38.843 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:21:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:21:38.844 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:21:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:21:38.845 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:21:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:21:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:21:38.846 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:21:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:21:38.847 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:21:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:21:38.848 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:21:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:21:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:21:38.849 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:21:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:21:38.850 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:21:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.851 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:21:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:21:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:21:38.852 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:21:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:21:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:21:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:21:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:21:38.854 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:21:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:21:38.855 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:21:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:21:38.856 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:21:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:21:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:21:38.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.858 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:21:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:21:38.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:21:38.860 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:21:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:38.861 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:38.862 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:38.863 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:38.864 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:38.865 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:38.865 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.866 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:21:38.867 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:21:38.869 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.870 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.871 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:21:38.873 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.874 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:21:38.875 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:21:38.878 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.879 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:21:38.880 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:38.882 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.883 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:21:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:38.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:21:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:38.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:21:38.888 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:21:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:21:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:21:38.889 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:38.890 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:21:38.892 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:21:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:38.893 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:38.894 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:38.895 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:38.896 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:38.897 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.898 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:21:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:21:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:38.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.904 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:21:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:21:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.908 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:21:38.910 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:21:38.912 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.913 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:38.914 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.915 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.916 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:21:38.916 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:38.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.917 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.918 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:21:38.918 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:38.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.919 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.920 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:21:38.920 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:21:38.921 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:21:38.921 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:21:38.922 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:38.922 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:38.922 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:38.923 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.924 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:21:38.925 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:21:38.925 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:38.925 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:38.926 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:38.926 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:38.927 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:38.927 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:38.928 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:38.928 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:38.929 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:38.929 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:38.930 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:38.930 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.931 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:21:38.932 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:38.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.933 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:21:38.934 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:38.934 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.935 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.935 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:38.936 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.937 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:21:38.937 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:38.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.938 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.939 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:21:38.939 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:38.939 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.940 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.940 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:38.941 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:21:38.942 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:38.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.944 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:21:38.944 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:38.945 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.945 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.946 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:38.946 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.947 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:21:38.947 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:38.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.948 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:21:38.948 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:38.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.948 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.949 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:21:38.949 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:21:38.949 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:21:38.950 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:21:38.950 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:38.950 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:38.950 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:38.951 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.951 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:21:38.951 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:21:38.952 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:38.952 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:38.952 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:38.952 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:38.953 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:38.953 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:38.953 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:38.953 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 79, 2021-04-09T18:21:38.953 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:38.954 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:38.954 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:38.954 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.955 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:21:38.955 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:38.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.956 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.956 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:21:38.956 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:38.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.957 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.958 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:38.958 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.958 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.959 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:21:38.959 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:38.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.960 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:21:38.960 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:38.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.960 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.961 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:38.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:21:38.962 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:38.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:21:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:21:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:21:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:21:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:21:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:21:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:21:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:21:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:21:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:21:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:21:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:21:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:21:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.978 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:38.978 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.979 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:21:38.979 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:38.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.980 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:21:38.980 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:38.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.981 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.981 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:38.981 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.982 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:21:38.982 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:38.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.983 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.983 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:21:38.983 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:38.983 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.984 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.984 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:21:38.984 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:21:38.984 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:21:38.985 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:21:38.985 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:38.985 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:38.985 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:38.985 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.986 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:21:38.986 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:21:38.987 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:38.987 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:38.987 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:38.987 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 79, 2021-04-09T18:21:38.988 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:38.989 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:38.989 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:38.989 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.989 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.990 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:21:38.990 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:38.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.991 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.991 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:21:38.991 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:38.991 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.992 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:38.992 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.993 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.993 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:21:38.993 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:38.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.993 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.994 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.994 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:21:38.994 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:38.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.994 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.995 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:38.995 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.996 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.996 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:21:38.996 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:38.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.997 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:38.997 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:21:38.997 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:38.997 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:38.998 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:38.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.998 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:38.998 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:38.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:38.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:38.999 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:21:38.999 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:38.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:38.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:39.000 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:21:39.000 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:39.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.001 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.001 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:21:39.001 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:21:39.001 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:21:39.002 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:21:39.002 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:39.002 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:39.002 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:39.002 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.003 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.003 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.003 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:21:39.003 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:21:39.004 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:39.004 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:39.004 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:39.004 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:39.004 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:39.005 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:39.005 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:39.005 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:39.005 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:39.006 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:39.006 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:39.006 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:39.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.006 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:39.007 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:21:39.007 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:39.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.008 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:39.008 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:21:39.008 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:39.008 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.008 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.009 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:39.009 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:39.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.009 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:39.010 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:21:39.010 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:39.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.010 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.011 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:39.011 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:21:39.011 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:39.011 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.011 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.012 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:39.012 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:39.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.012 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:39.013 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:21:39.013 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:39.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.014 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:39.014 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:21:39.014 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:39.014 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.014 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.015 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:39.015 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:39.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.016 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:39.016 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:21:39.016 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:39.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.017 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:39.017 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:21:39.017 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:39.017 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.018 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.018 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:21:39.018 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:21:39.018 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:21:39.019 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:21:39.019 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:39.019 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:39.019 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:39.020 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.020 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:21:39.020 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:21:39.021 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:39.021 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:39.021 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:39.021 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:39.022 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:39.022 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:39.022 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:39.022 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:21:39.022 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:39.023 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:39.023 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:39.023 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:39.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.023 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:39.024 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:21:39.024 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:39.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.025 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:39.025 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:21:39.025 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:39.025 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.026 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:39.026 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:39.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.026 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:39.027 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:21:39.027 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:39.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.027 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.028 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:39.028 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:21:39.028 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:39.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.028 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.029 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:39.029 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:39.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.030 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:39.030 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:21:39.030 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:39.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.031 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:39.031 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:21:39.031 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:39.031 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.031 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.032 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:39.032 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:39.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:39.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:21:39.033 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:39.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.034 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:39.034 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:21:39.034 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:39.034 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.034 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.035 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:21:39.035 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:21:39.035 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:21:39.036 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:21:39.036 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:39.036 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:39.036 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:39.036 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:39.037 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.037 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:21:39.037 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:21:39.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.037 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:21:39.038 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:39.038 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:39.038 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:39.038 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:39.039 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:39.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.039 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:21:39.039 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:39.040 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:39.040 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:39.040 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:39.040 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:39.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.041 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:21:39.041 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:39.041 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:39.042 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:39.042 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:39.042 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:39.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.043 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:21:39.043 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:39.043 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:39.043 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:39.044 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:39.044 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:39.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:21:39.045 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:39.045 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:39.045 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:39.045 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:39.045 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:39.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.046 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:21:39.046 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:39.047 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:39.047 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:39.047 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:39.047 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:39.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:21:39.048 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:39.048 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:39.048 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:39.049 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:39.049 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:39.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:39.050 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:21:39.050 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:39.050 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:39.050 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:39.050 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:39.051 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:39.051 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.051 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:21:39.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:21:39.052 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:21:39.052 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:21:39.052 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:21:39.052 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:21:39.052 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:21:39.053 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:21:39.053 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:21:39.053 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:21:39.053 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:21:39.053 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:21:39.054 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:21:39.054 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:21:39.054 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:39.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:39.054 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:21:39.055 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:21:39.055 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:21:39.055 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:21:39.055 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:21:39.056 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:21:39.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa49c0188d0 msgr2=0x7fa49c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa49c0188d0 0x7fa49c01add0 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b80a8f10 msgr2=0x7fa4b81abc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b80a8f10 0x7fa4b81abc00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7fa4b0009980 tx=0x7fa4b000bcd0).stop 2021-04-09T18:21:39.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 shutdown_connections 2021-04-09T18:21:39.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa49c0188d0 0x7fa49c01add0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b80a8f10 0x7fa4b81abc00 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b81b8b60 0x7fa4b81ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 --2- 172.21.15.32:0/1601702316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b80a8b10 0x7fa4b81b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.743 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 >> 172.21.15.32:0/1601702316 conn(0x7fa4b8190570 msgr2=0x7fa4b818ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:39.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.744 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 shutdown_connections 2021-04-09T18:21:39.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:38.744 7fa4be1c8700 1 -- 172.21.15.32:0/1601702316 wait complete. 2021-04-09T18:21:39.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:39.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.161 7ff7308d4700 1 Processor -- start 2021-04-09T18:21:39.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.161 7ff7308d4700 1 -- start start 2021-04-09T18:21:39.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.161 7ff7308d4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff72c0d7ca0 0x7ff72c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.161 7ff7308d4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c10da00 0x7ff72c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.161 7ff7308d4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c10d480 con 0x7ff72c10d010 2021-04-09T18:21:39.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.161 7ff7308d4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72c0d9220 con 0x7ff72c0d7ca0 2021-04-09T18:21:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.161 7ff7308d4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0d86c0 con 0x7ff72c10da00 2021-04-09T18:21:39.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff72a61c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff72c10d010 0x7ff72c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57936/0 (socket says 172.21.15.32:57936) 2021-04-09T18:21:39.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff72a61c700 1 -- 172.21.15.32:0/766283157 learned_addr learned my addr 172.21.15.32:0/766283157 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:39.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff729e1b700 1 --2- 172.21.15.32:0/766283157 >> v2:172.21.15.86:3300/0 conn(0x7ff72c0d7ca0 0x7ff72c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff72ae1d700 1 --2- 172.21.15.32:0/766283157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c10da00 0x7ff72c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff72961a700 1 -- 172.21.15.32:0/766283157 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579159430 0 0) 0x7ff71c001060 con 0x7ff72c10d010 2021-04-09T18:21:39.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff72961a700 1 -- 172.21.15.32:0/766283157 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff714002fe0 con 0x7ff72c10d010 2021-04-09T18:21:39.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff729e1b700 1 -- 172.21.15.32:0/766283157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c10da00 msgr2=0x7ff72c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:39.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff729e1b700 1 --2- 172.21.15.32:0/766283157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c10da00 0x7ff72c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff729e1b700 1 -- 172.21.15.32:0/766283157 >> v1:172.21.15.32:6789/0 conn(0x7ff72c10d010 legacy=0x7ff72c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff729e1b700 1 -- 172.21.15.32:0/766283157 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c0d1f70 con 0x7ff72c0d7ca0 2021-04-09T18:21:39.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff729e1b700 1 --2- 172.21.15.32:0/766283157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c0d7ca0 0x7ff72c0d80a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7ff720006650 tx=0x7ff720009a80).ready entity=mon.1 client_cookie=d2c320bcc62b9650 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:39.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff72961a700 1 -- 172.21.15.32:0/766283157 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7200009b0 con 0x7ff72c0d7ca0 2021-04-09T18:21:39.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff72961a700 1 -- 172.21.15.32:0/766283157 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72000c0c0 con 0x7ff72c0d7ca0 2021-04-09T18:21:39.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff72961a700 1 -- 172.21.15.32:0/766283157 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff72000c510 con 0x7ff72c0d7ca0 2021-04-09T18:21:39.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff7308d4700 1 -- 172.21.15.32:0/766283157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c0d7ca0 msgr2=0x7ff72c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.162 7ff7308d4700 1 --2- 172.21.15.32:0/766283157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c0d7ca0 0x7ff72c0d80a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7ff720006650 tx=0x7ff720009a80).stop 2021-04-09T18:21:39.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.163 7ff7308d4700 1 -- 172.21.15.32:0/766283157 shutdown_connections 2021-04-09T18:21:39.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.163 7ff7308d4700 1 --2- 172.21.15.32:0/766283157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c10da00 0x7ff72c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.163 7ff7308d4700 1 --2- 172.21.15.32:0/766283157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c0d7ca0 0x7ff72c0d80a0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.163 7ff7308d4700 1 -- 172.21.15.32:0/766283157 >> 172.21.15.32:0/766283157 conn(0x7ff72c0ce5f0 msgr2=0x7ff72c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:39.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.163 7ff7308d4700 1 -- 172.21.15.32:0/766283157 shutdown_connections 2021-04-09T18:21:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.163 7ff7308d4700 1 -- 172.21.15.32:0/766283157 wait complete. 2021-04-09T18:21:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.183 7ff7308d4700 1 Processor -- start 2021-04-09T18:21:39.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.183 7ff7308d4700 1 -- start start 2021-04-09T18:21:39.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff7308d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c1ac380 0x7ff72c1ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff7308d4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c186c90 0x7ff72c187090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff7308d4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c185530 0x7ff72c185930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff7308d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c186790 con 0x7ff72c1ac380 2021-04-09T18:21:39.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff7308d4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c1acda0 con 0x7ff72c185530 2021-04-09T18:21:39.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff7308d4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c18e100 con 0x7ff72c186c90 2021-04-09T18:21:39.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72a61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c1ac380 0x7ff72c1ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72a61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c1ac380 0x7ff72c1ac780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48352/0 (socket says 172.21.15.32:48352) 2021-04-09T18:21:39.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72a61c700 1 -- 172.21.15.32:0/2060694437 learned_addr learned my addr 172.21.15.32:0/2060694437 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:39.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72ae1d700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c185530 0x7ff72c185930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff729e1b700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c186c90 0x7ff72c187090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72a61c700 1 -- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c186c90 msgr2=0x7ff72c187090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72a61c700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c186c90 0x7ff72c187090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72a61c700 1 -- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c185530 msgr2=0x7ff72c185930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72a61c700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c185530 0x7ff72c185930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72a61c700 1 -- 172.21.15.32:0/2060694437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c1876b0 con 0x7ff72c1ac380 2021-04-09T18:21:39.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.184 7ff72ae1d700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c185530 0x7ff72c185930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:21:39.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.185 7ff72a61c700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c1ac380 0x7ff72c1ac780 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7ff71c007fc0 tx=0x7ff71c00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:39.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.185 7ff71affd700 1 -- 172.21.15.32:0/2060694437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff71c001270 con 0x7ff72c1ac380 2021-04-09T18:21:39.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.185 7ff71affd700 1 -- 172.21.15.32:0/2060694437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c007910 con 0x7ff72c1ac380 2021-04-09T18:21:39.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.185 7ff71affd700 1 -- 172.21.15.32:0/2060694437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff71c004bb0 con 0x7ff72c1ac380 2021-04-09T18:21:39.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.185 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72c1876b0 con 0x7ff72c1ac380 2021-04-09T18:21:39.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.185 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c1876b0 con 0x7ff72c1ac380 2021-04-09T18:21:39.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.185 7ff71affd700 1 -- 172.21.15.32:0/2060694437 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff71c00adc0 con 0x7ff72c1ac380 2021-04-09T18:21:39.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.186 7ff71affd700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7100188c0 0x7ff71001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.186 7ff71affd700 1 -- 172.21.15.32:0/2060694437 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff71c032b30 con 0x7ff72c1ac380 2021-04-09T18:21:39.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.186 7ff729e1b700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7100188c0 0x7ff71001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.186 7ff729e1b700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7100188c0 0x7ff71001adc0 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:39.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.206 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c1acda0 con 0x7ff72c1ac380 2021-04-09T18:21:39.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.209 7ff71affd700 1 -- 172.21.15.32:0/2060694437 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff71c004720 con 0x7ff72c1ac380 2021-04-09T18:21:39.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.401 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff72c1acda0 con 0x7ff72c1ac380 2021-04-09T18:21:39.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.401 7ff71affd700 1 -- 172.21.15.32:0/2060694437 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2777 (secure 0 0 0) 0x7ff71c01fa90 con 0x7ff72c1ac380 2021-04-09T18:21:39.452 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 101 2021-04-09T18:21:39.453 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:39.453 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":101,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":100,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:18:54.784269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{"mds_0":6526},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_6526":{"gid":6526,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2067624954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2067624954},{"type":"v1","addr":"172.21.15.32:6835","nonce":2067624954}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:39.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7100188c0 msgr2=0x7ff71001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7100188c0 0x7ff71001adc0 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c1ac380 msgr2=0x7ff72c1ac780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c1ac380 0x7ff72c1ac780 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7ff71c007fc0 tx=0x7ff71c00ac80).stop 2021-04-09T18:21:39.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 shutdown_connections 2021-04-09T18:21:39.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff7100188c0 0x7ff71001adc0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c185530 0x7ff72c185930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c186c90 0x7ff72c187090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 --2- 172.21.15.32:0/2060694437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c1ac380 0x7ff72c1ac780 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 >> 172.21.15.32:0/2060694437 conn(0x7ff72c19b890 msgr2=0x7ff72c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:39.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 shutdown_connections 2021-04-09T18:21:39.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.404 7ff7308d4700 1 -- 172.21.15.32:0/2060694437 wait complete. 2021-04-09T18:21:39.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:21:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.566 7f135d548700 1 Processor -- start 2021-04-09T18:21:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.566 7f135d548700 1 -- start start 2021-04-09T18:21:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.567 7f135d548700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f13580cffb0 0x7f13580d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.567 7f135d548700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13580d7810 0x7f13580d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.567 7f135d548700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13580d7290 con 0x7f13580d8230 2021-04-09T18:21:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.567 7f135d548700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13580d91f0 con 0x7f13580cffb0 2021-04-09T18:21:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.567 7f135d548700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135810dff0 con 0x7f13580d7810 2021-04-09T18:21:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.567 7f13572bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f13580cffb0 0x7f13580d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.567 7f1357ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f13580d8230 0x7f13580d5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57952/0 (socket says 172.21.15.32:57952) 2021-04-09T18:21:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.567 7f1357ac0700 1 -- 172.21.15.32:0/3599445594 learned_addr learned my addr 172.21.15.32:0/3599445594 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.567 7f1356abe700 1 --2- 172.21.15.32:0/3599445594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13580d7810 0x7f13580d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f13572bf700 1 -- 172.21.15.32:0/3599445594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13580d7810 msgr2=0x7f13580d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f13572bf700 1 --2- 172.21.15.32:0/3599445594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13580d7810 0x7f13580d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f13572bf700 1 -- 172.21.15.32:0/3599445594 >> v1:172.21.15.32:6789/0 conn(0x7f13580d8230 legacy=0x7f13580d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f13572bf700 1 -- 172.21.15.32:0/3599445594 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135810e210 con 0x7f13580cffb0 2021-04-09T18:21:39.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f13562bd700 1 -- 172.21.15.32:0/3599445594 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657316895 0 0) 0x7f1350000d30 con 0x7f13580d8230 2021-04-09T18:21:39.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f13572bf700 1 --2- 172.21.15.32:0/3599445594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13580cffb0 0x7f13580d2450 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f1348006650 tx=0x7f1348009a00).ready entity=mon.0 client_cookie=92de6267c0a60c42 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:39.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f13562bd700 1 -- 172.21.15.32:0/3599445594 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1348009df0 con 0x7f13580cffb0 2021-04-09T18:21:39.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f13562bd700 1 -- 172.21.15.32:0/3599445594 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13480009b0 con 0x7f13580cffb0 2021-04-09T18:21:39.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f13562bd700 1 -- 172.21.15.32:0/3599445594 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f134800c2b0 con 0x7f13580cffb0 2021-04-09T18:21:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f135d548700 1 -- 172.21.15.32:0/3599445594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13580cffb0 msgr2=0x7f13580d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.568 7f135d548700 1 --2- 172.21.15.32:0/3599445594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13580cffb0 0x7f13580d2450 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f1348006650 tx=0x7f1348009a00).stop 2021-04-09T18:21:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.569 7f135d548700 1 -- 172.21.15.32:0/3599445594 shutdown_connections 2021-04-09T18:21:39.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.569 7f135d548700 1 --2- 172.21.15.32:0/3599445594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13580d7810 0x7f13580d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.569 7f135d548700 1 --2- 172.21.15.32:0/3599445594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13580cffb0 0x7f13580d2450 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.569 7f135d548700 1 -- 172.21.15.32:0/3599445594 >> 172.21.15.32:0/3599445594 conn(0x7f13580ce5d0 msgr2=0x7f13580cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.569 7f135d548700 1 -- 172.21.15.32:0/3599445594 shutdown_connections 2021-04-09T18:21:39.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.569 7f135d548700 1 -- 172.21.15.32:0/3599445594 wait complete. 2021-04-09T18:21:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.589 7f135d548700 1 Processor -- start 2021-04-09T18:21:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.589 7f135d548700 1 -- start start 2021-04-09T18:21:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.589 7f135d548700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1358192320 0x7f1358192720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f135d548700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1358092ad0 0x7f1358092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f135d548700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13580934f0 0x7f13580938f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f135d548700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13581930a0 con 0x7f1358192320 2021-04-09T18:21:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f135d548700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13580a5e50 con 0x7f13580934f0 2021-04-09T18:21:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f135d548700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1358191dd0 con 0x7f1358092ad0 2021-04-09T18:21:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f13572bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1358192320 0x7f1358192720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f13572bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1358192320 0x7f1358192720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48366/0 (socket says 172.21.15.32:48366) 2021-04-09T18:21:39.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f13572bf700 1 -- 172.21.15.32:0/728589454 learned_addr learned my addr 172.21.15.32:0/728589454 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:39.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f1357ac0700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13580934f0 0x7f13580938f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f1356abe700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1358092ad0 0x7f1358092ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f13572bf700 1 -- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1358092ad0 msgr2=0x7f1358092ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f13572bf700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1358092ad0 0x7f1358092ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f13572bf700 1 -- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13580934f0 msgr2=0x7f13580938f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:39.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f13572bf700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13580934f0 0x7f13580938f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:39.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f13572bf700 1 -- 172.21.15.32:0/728589454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1358093f10 con 0x7f1358192320 2021-04-09T18:21:39.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f13572bf700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1358192320 0x7f1358192720 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7f13480096a0 tx=0x7f134800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:39.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f133ffff700 1 -- 172.21.15.32:0/728589454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1348000c90 con 0x7f1358192320 2021-04-09T18:21:39.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f133ffff700 1 -- 172.21.15.32:0/728589454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1348002030 con 0x7f1358192320 2021-04-09T18:21:39.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f133ffff700 1 -- 172.21.15.32:0/728589454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1348002550 con 0x7f1358192320 2021-04-09T18:21:39.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f135d548700 1 -- 172.21.15.32:0/728589454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1358093f10 con 0x7f1358192320 2021-04-09T18:21:39.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.590 7f135d548700 1 -- 172.21.15.32:0/728589454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1358093f10 con 0x7f1358192320 2021-04-09T18:21:39.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.591 7f133ffff700 1 -- 172.21.15.32:0/728589454 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1348002210 con 0x7f1358192320 2021-04-09T18:21:39.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.591 7f133ffff700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1338018930 0x7f133801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:39.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.591 7f133ffff700 1 -- 172.21.15.32:0/728589454 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1348032a70 con 0x7f1358192320 2021-04-09T18:21:39.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.591 7f1356abe700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1338018930 0x7f133801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:39.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.592 7f1356abe700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1338018930 0x7f133801ae10 crc :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:39.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.611 7f135d548700 1 -- 172.21.15.32:0/728589454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13580a5e50 con 0x7f1358192320 2021-04-09T18:21:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.615 7f133ffff700 1 -- 172.21.15.32:0/728589454 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f13480009c0 con 0x7f1358192320 2021-04-09T18:21:39.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:39.796 7f135d548700 1 -- 172.21.15.32:0/728589454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f13580a5e50 con 0x7f1358192320 2021-04-09T18:21:40.329 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:21:40.276 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:21:40.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.278 7f133ffff700 1 -- 172.21.15.32:0/728589454 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v102) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1348002010 con 0x7f1358192320 2021-04-09T18:21:40.331 INFO:tasks.ceph.mds.a.smithi032.stderr: -4239> 2021-04-09 18:18:29.207 7fa99b826700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2021-04-09T18:21:40.361 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:21:40.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.313 7f135d548700 1 -- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1338018930 msgr2=0x7f133801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:40.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.313 7f135d548700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1338018930 0x7f133801ae10 crc :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.313 7f135d548700 1 -- 172.21.15.32:0/728589454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1358192320 msgr2=0x7f1358192720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:40.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.313 7f135d548700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1358192320 0x7f1358192720 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7f13480096a0 tx=0x7f134800b990).stop 2021-04-09T18:21:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.314 7f135d548700 1 -- 172.21.15.32:0/728589454 shutdown_connections 2021-04-09T18:21:40.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.314 7f135d548700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1338018930 0x7f133801ae10 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.314 7f135d548700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13580934f0 0x7f13580938f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.314 7f135d548700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1358092ad0 0x7f1358092ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.314 7f135d548700 1 --2- 172.21.15.32:0/728589454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1358192320 0x7f1358192720 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.314 7f135d548700 1 -- 172.21.15.32:0/728589454 >> 172.21.15.32:0/728589454 conn(0x7f135809cb30 msgr2=0x7f1358195df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:40.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.314 7f135d548700 1 -- 172.21.15.32:0/728589454 shutdown_connections 2021-04-09T18:21:40.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.314 7f135d548700 1 -- 172.21.15.32:0/728589454 wait complete. 2021-04-09T18:21:40.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:40.393 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:21:40.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.479 7f0356426700 1 Processor -- start 2021-04-09T18:21:40.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.479 7f0356426700 1 -- start start 2021-04-09T18:21:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f0356426700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f035010da00 0x7f035010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:40.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f0356426700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035010d480 con 0x7f035010d010 2021-04-09T18:21:40.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f0356426700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03500d1f70 con 0x7f03500d7ca0 2021-04-09T18:21:40.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f0356426700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03500d9220 con 0x7f035010da00 2021-04-09T18:21:40.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f035010d010 0x7f03500d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57974/0 (socket says 172.21.15.32:57974) 2021-04-09T18:21:40.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034ffff700 1 -- 172.21.15.32:0/371191581 learned_addr learned my addr 172.21.15.32:0/371191581 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f0354a43700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f035010da00 0x7f035010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106426517 0 0) 0x7f0340000ce0 con 0x7f035010d010 2021-04-09T18:21:40.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f033c002fe0 con 0x7f035010d010 2021-04-09T18:21:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111552052 0 0) 0x7f0338001080 con 0x7f03500d7ca0 2021-04-09T18:21:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f033c003210 con 0x7f03500d7ca0 2021-04-09T18:21:40.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 68978453 0 0) 0x7f03400010e0 con 0x7f035010d010 2021-04-09T18:21:40.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f035010da00 msgr2=0x7f035010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:40.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f035010da00 0x7f035010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 >> v1:172.21.15.86:6789/0 conn(0x7f03500d7ca0 legacy=0x7f03500d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:40.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035010e420 con 0x7f035010d010 2021-04-09T18:21:40.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f0340001600 con 0x7f035010d010 2021-04-09T18:21:40.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 -- 172.21.15.32:0/371191581 >> v1:172.21.15.32:6789/0 conn(0x7f035010d010 legacy=0x7f03500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:40.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f033c003b90 0x7f033c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:40.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f033c0067a0 0x7f033c008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:40.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.480 7f034f7fe700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f033c009510 0x7f033c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:40.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034ffff700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f033c003b90 0x7f033c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:40.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034f7fe700 1 -- 172.21.15.32:0/371191581 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c0502f0 con 0x7f033c003b90 2021-04-09T18:21:40.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034f7fe700 1 -- 172.21.15.32:0/371191581 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c0504d0 con 0x7f033c009510 2021-04-09T18:21:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034f7fe700 1 -- 172.21.15.32:0/371191581 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033c0506f0 con 0x7f033c0067a0 2021-04-09T18:21:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f0354a43700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f033c009510 0x7f033c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:40.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f0347fff700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f033c0067a0 0x7f033c008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:40.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034ffff700 1 -- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f033c0067a0 msgr2=0x7f033c008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:40.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034ffff700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f033c0067a0 0x7f033c008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034ffff700 1 -- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f033c009510 msgr2=0x7f033c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:40.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034ffff700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f033c009510 0x7f033c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034ffff700 1 -- 172.21.15.32:0/371191581 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c050990 con 0x7f033c003b90 2021-04-09T18:21:40.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034ffff700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f033c003b90 0x7f033c005fa0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f03400018d0 tx=0x7f0340007c70).ready entity=mon.0 client_cookie=7fb72c27a68fd90 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:40.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034f7fe700 1 -- 172.21.15.32:0/371191581 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03400014f0 con 0x7f033c003b90 2021-04-09T18:21:40.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034f7fe700 1 -- 172.21.15.32:0/371191581 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0340007140 con 0x7f033c003b90 2021-04-09T18:21:40.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f034f7fe700 1 -- 172.21.15.32:0/371191581 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03400045d0 con 0x7f033c003b90 2021-04-09T18:21:40.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f0356426700 1 -- 172.21.15.32:0/371191581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f033c003b90 msgr2=0x7f033c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:40.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f0356426700 1 --2- 172.21.15.32:0/371191581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f033c003b90 0x7f033c005fa0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f03400018d0 tx=0x7f0340007c70).stop 2021-04-09T18:21:40.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.481 7f0354a43700 1 -- 172.21.15.32:0/371191581 reap_dead start 2021-04-09T18:21:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.482 7f0356426700 1 -- 172.21.15.32:0/371191581 shutdown_connections 2021-04-09T18:21:40.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.482 7f0356426700 1 -- 172.21.15.32:0/371191581 >> 172.21.15.32:0/371191581 conn(0x7f03500ce5f0 msgr2=0x7f03500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.482 7f0356426700 1 -- 172.21.15.32:0/371191581 shutdown_connections 2021-04-09T18:21:40.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.482 7f0356426700 1 -- 172.21.15.32:0/371191581 wait complete. 2021-04-09T18:21:40.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.502 7f0356426700 1 Processor -- start 2021-04-09T18:21:40.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.502 7f0356426700 1 -- start start 2021-04-09T18:21:40.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0356426700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03500b42d0 0x7f0350183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0356426700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03501ac090 0x7f03501ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:40.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0356426700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0350184d30 0x7f0350185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:40.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0356426700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03500a1a90 con 0x7f0350184d30 2021-04-09T18:21:40.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0356426700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03501acab0 con 0x7f03501ac090 2021-04-09T18:21:40.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0356426700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0350185a00 con 0x7f03500b42d0 2021-04-09T18:21:40.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0354a43700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0350184d30 0x7f0350185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:40.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0354a43700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0350184d30 0x7f0350185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48400/0 (socket says 172.21.15.32:48400) 2021-04-09T18:21:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0354a43700 1 -- 172.21.15.32:0/863252899 learned_addr learned my addr 172.21.15.32:0/863252899 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0347fff700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03501ac090 0x7f03501ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f034ffff700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03500b42d0 0x7f0350183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0354a43700 1 -- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03500b42d0 msgr2=0x7f0350183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0354a43700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03500b42d0 0x7f0350183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0354a43700 1 -- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03501ac090 msgr2=0x7f03501ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0354a43700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03501ac090 0x7f03501ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0354a43700 1 -- 172.21.15.32:0/863252899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0350185750 con 0x7f0350184d30 2021-04-09T18:21:40.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f0354a43700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0350184d30 0x7f0350185130 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f034800a940 tx=0x7f034800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:40.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.503 7f034d7fa700 1 -- 172.21.15.32:0/863252899 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0348001400 con 0x7f0350184d30 2021-04-09T18:21:40.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.504 7f034d7fa700 1 -- 172.21.15.32:0/863252899 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0348003030 con 0x7f0350184d30 2021-04-09T18:21:40.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.504 7f0356426700 1 -- 172.21.15.32:0/863252899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0350185750 con 0x7f0350184d30 2021-04-09T18:21:40.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.504 7f034d7fa700 1 -- 172.21.15.32:0/863252899 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03480018f0 con 0x7f0350184d30 2021-04-09T18:21:40.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.504 7f0356426700 1 -- 172.21.15.32:0/863252899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0350185750 con 0x7f0350184d30 2021-04-09T18:21:40.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.504 7f034d7fa700 1 -- 172.21.15.32:0/863252899 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f03480011f0 con 0x7f0350184d30 2021-04-09T18:21:40.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.504 7f034d7fa700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0334018930 0x7f033401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:40.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.504 7f034d7fa700 1 -- 172.21.15.32:0/863252899 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f0348032da0 con 0x7f0350184d30 2021-04-09T18:21:40.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.505 7f034ffff700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0334018930 0x7f033401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:40.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.505 7f034ffff700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0334018930 0x7f033401ae10 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:40.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.523 7f0356426700 1 -- 172.21.15.32:0/863252899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0350182d10 con 0x7f0350184d30 2021-04-09T18:21:40.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.527 7f034d7fa700 1 -- 172.21.15.32:0/863252899 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f03480066f0 con 0x7f0350184d30 2021-04-09T18:21:40.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.718 7f0356426700 1 -- 172.21.15.32:0/863252899 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f035010d480 con 0x7f0350184d30 2021-04-09T18:21:40.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.719 7f034d7fa700 1 -- 172.21.15.32:0/863252899 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2432 (secure 0 0 0) 0x7f03480011f0 con 0x7f0350184d30 2021-04-09T18:21:40.770 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 102 2021-04-09T18:21:40.771 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:40.771 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":102,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":102,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:18:33.664336","modified":"2021-04-09 18:21:40.276771","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[1],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-04-09T18:21:40.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.721 7f0356426700 1 -- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0334018930 msgr2=0x7f033401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:40.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.721 7f0356426700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0334018930 0x7f033401ae10 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.721 7f0356426700 1 -- 172.21.15.32:0/863252899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0350184d30 msgr2=0x7f0350185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:40.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.721 7f0356426700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0350184d30 0x7f0350185130 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f034800a940 tx=0x7f034800ace0).stop 2021-04-09T18:21:40.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.722 7f0356426700 1 -- 172.21.15.32:0/863252899 shutdown_connections 2021-04-09T18:21:40.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.722 7f0356426700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0334018930 0x7f033401ae10 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.722 7f0356426700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0350184d30 0x7f0350185130 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.722 7f0356426700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03501ac090 0x7f03501ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.722 7f0356426700 1 --2- 172.21.15.32:0/863252899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03500b42d0 0x7f0350183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:40.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.722 7f0356426700 1 -- 172.21.15.32:0/863252899 >> 172.21.15.32:0/863252899 conn(0x7f035019b890 msgr2=0x7f03501a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:40.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.722 7f0356426700 1 -- 172.21.15.32:0/863252899 shutdown_connections 2021-04-09T18:21:40.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.722 7f0356426700 1 -- 172.21.15.32:0/863252899 wait complete. 2021-04-09T18:21:40.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:21:41.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.928 7f491b9bb700 1 Processor -- start 2021-04-09T18:21:41.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.928 7f491b9bb700 1 -- start start 2021-04-09T18:21:41.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.928 7f491b9bb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49140cffa0 0x7f49140d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f491b9bb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f49140d7800 0x7f49140d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f491b9bb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49140d7280 con 0x7f49140d8220 2021-04-09T18:21:41.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f491b9bb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49140d91e0 con 0x7f49140cffa0 2021-04-09T18:21:41.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f491b9bb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491410dfe0 con 0x7f49140d7800 2021-04-09T18:21:41.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f49197d7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49140cffa0 0x7f49140d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f4919fd8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f49140d8220 0x7f49140d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57996/0 (socket says 172.21.15.32:57996) 2021-04-09T18:21:41.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f49197d7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f49140cffa0 0x7f49140d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48404/0 (socket says 172.21.15.32:48404) 2021-04-09T18:21:41.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f4919fd8700 1 -- 172.21.15.32:0/1812996289 learned_addr learned my addr 172.21.15.32:0/1812996289 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:41.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f4918fd6700 1 --2- 172.21.15.32:0/1812996289 >> v2:172.21.15.86:3300/0 conn(0x7f49140d7800 0x7f49140d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f490bfff700 1 -- 172.21.15.32:0/1812996289 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224902811 0 0) 0x7f490c000d30 con 0x7f49140d8220 2021-04-09T18:21:41.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f490bfff700 1 -- 172.21.15.32:0/1812996289 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4900002fe0 con 0x7f49140d8220 2021-04-09T18:21:41.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f49197d7700 1 -- 172.21.15.32:0/1812996289 >> v2:172.21.15.86:3300/0 conn(0x7f49140d7800 msgr2=0x7f49140d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:41.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f49197d7700 1 --2- 172.21.15.32:0/1812996289 >> v2:172.21.15.86:3300/0 conn(0x7f49140d7800 0x7f49140d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f49197d7700 1 -- 172.21.15.32:0/1812996289 >> v1:172.21.15.32:6789/0 conn(0x7f49140d8220 legacy=0x7f49140d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:41.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f49197d7700 1 -- 172.21.15.32:0/1812996289 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491410e200 con 0x7f49140cffa0 2021-04-09T18:21:41.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f4918fd6700 1 --2- 172.21.15.32:0/1812996289 >> v2:172.21.15.86:3300/0 conn(0x7f49140d7800 0x7f49140d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:41.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f49197d7700 1 --2- 172.21.15.32:0/1812996289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49140cffa0 0x7f49140d2440 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f4910006650 tx=0x7f4910009a80).ready entity=mon.0 client_cookie=618948c771ee8ad0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:41.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f490bfff700 1 -- 172.21.15.32:0/1812996289 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49100009b0 con 0x7f49140cffa0 2021-04-09T18:21:41.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f490bfff700 1 -- 172.21.15.32:0/1812996289 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491000c0c0 con 0x7f49140cffa0 2021-04-09T18:21:41.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.929 7f490bfff700 1 -- 172.21.15.32:0/1812996289 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f491000c510 con 0x7f49140cffa0 2021-04-09T18:21:41.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.930 7f491b9bb700 1 -- 172.21.15.32:0/1812996289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49140cffa0 msgr2=0x7f49140d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:41.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.930 7f491b9bb700 1 --2- 172.21.15.32:0/1812996289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49140cffa0 0x7f49140d2440 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f4910006650 tx=0x7f4910009a80).stop 2021-04-09T18:21:41.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.930 7f491b9bb700 1 -- 172.21.15.32:0/1812996289 shutdown_connections 2021-04-09T18:21:41.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.930 7f491b9bb700 1 --2- 172.21.15.32:0/1812996289 >> v2:172.21.15.86:3300/0 conn(0x7f49140d7800 0x7f49140d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.930 7f491b9bb700 1 --2- 172.21.15.32:0/1812996289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49140cffa0 0x7f49140d2440 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.930 7f491b9bb700 1 -- 172.21.15.32:0/1812996289 >> 172.21.15.32:0/1812996289 conn(0x7f49140ce5c0 msgr2=0x7f49140cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:41.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.930 7f491b9bb700 1 -- 172.21.15.32:0/1812996289 shutdown_connections 2021-04-09T18:21:41.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.930 7f491b9bb700 1 -- 172.21.15.32:0/1812996289 wait complete. 2021-04-09T18:21:41.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.950 7f491b9bb700 1 Processor -- start 2021-04-09T18:21:41.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.950 7f491b9bb700 1 -- start start 2021-04-09T18:21:41.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f491b9bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f491408d210 0x7f491408fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f491b9bb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49140903f0 0x7f49141a5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f491b9bb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49141a6070 0x7f49141a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f491b9bb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491408d690 con 0x7f491408d210 2021-04-09T18:21:41.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f491b9bb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49140b0fd0 con 0x7f49140903f0 2021-04-09T18:21:41.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f491b9bb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49141a6470 con 0x7f49141a6070 2021-04-09T18:21:41.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f49197d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f491408d210 0x7f491408fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f49197d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f491408d210 0x7f491408fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48410/0 (socket says 172.21.15.32:48410) 2021-04-09T18:21:41.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f49197d7700 1 -- 172.21.15.32:0/1619556464 learned_addr learned my addr 172.21.15.32:0/1619556464 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:41.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f4918fd6700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49140903f0 0x7f49141a5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.951 7f4919fd8700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49141a6070 0x7f49141a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f49197d7700 1 -- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49141a6070 msgr2=0x7f49141a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:41.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f49197d7700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49141a6070 0x7f49141a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f49197d7700 1 -- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49140903f0 msgr2=0x7f49141a5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:41.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f49197d7700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49140903f0 0x7f49141a5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f49197d7700 1 -- 172.21.15.32:0/1619556464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491408d8b0 con 0x7f491408d210 2021-04-09T18:21:41.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f49197d7700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f491408d210 0x7f491408fdd0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f4910009630 tx=0x7f491000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:41.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f4909ffb700 1 -- 172.21.15.32:0/1619556464 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49100093a0 con 0x7f491408d210 2021-04-09T18:21:41.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f4909ffb700 1 -- 172.21.15.32:0/1619556464 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491000bd70 con 0x7f491408d210 2021-04-09T18:21:41.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f4909ffb700 1 -- 172.21.15.32:0/1619556464 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4910005560 con 0x7f491408d210 2021-04-09T18:21:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491408d8b0 con 0x7f491408d210 2021-04-09T18:21:41.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.952 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491408d8b0 con 0x7f491408d210 2021-04-09T18:21:41.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.953 7f4909ffb700 1 -- 172.21.15.32:0/1619556464 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f49100052e0 con 0x7f491408d210 2021-04-09T18:21:41.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.953 7f4909ffb700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48f40188c0 0x7f48f401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.953 7f4909ffb700 1 -- 172.21.15.32:0/1619556464 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f4910032ab0 con 0x7f491408d210 2021-04-09T18:21:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.953 7f4918fd6700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48f40188c0 0x7f48f401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.953 7f4918fd6700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48f40188c0 0x7f48f401adc0 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:41.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.972 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49141a52a0 con 0x7f491408d210 2021-04-09T18:21:41.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:40.976 7f4909ffb700 1 -- 172.21.15.32:0/1619556464 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f491001f7a0 con 0x7f491408d210 2021-04-09T18:21:41.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.170 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f49141a52a0 con 0x7f491408d210 2021-04-09T18:21:41.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.287 7f4909ffb700 1 -- 172.21.15.32:0/1619556464 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v103) v1 ==== 104+0+0 (secure 0 0 0) 0x7f49100052e0 con 0x7f491408d210 2021-04-09T18:21:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48f40188c0 msgr2=0x7f48f401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48f40188c0 0x7f48f401adc0 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f491408d210 msgr2=0x7f491408fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f491408d210 0x7f491408fdd0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f4910009630 tx=0x7f491000b980).stop 2021-04-09T18:21:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 shutdown_connections 2021-04-09T18:21:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48f40188c0 0x7f48f401adc0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49141a6070 0x7f49141a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49140903f0 0x7f49141a5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 --2- 172.21.15.32:0/1619556464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f491408d210 0x7f491408fdd0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 >> 172.21.15.32:0/1619556464 conn(0x7f49141a8f40 msgr2=0x7f4914188050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 shutdown_connections 2021-04-09T18:21:41.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.336 7f491b9bb700 1 -- 172.21.15.32:0/1619556464 wait complete. 2021-04-09T18:21:41.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:21:41.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.510 7fedbdabe700 1 Processor -- start 2021-04-09T18:21:41.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.510 7fedbdabe700 1 -- start start 2021-04-09T18:21:41.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.510 7fedbdabe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fedb810d050 0x7fedb80cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.510 7fedbdabe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedb80d0160 0x7fedb80d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.510 7fedbdabe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80d82e0 0x7fedb810da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.510 7fedbdabe700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb80d8d70 con 0x7fedb80d0160 2021-04-09T18:21:41.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.510 7fedbdabe700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb80d9260 con 0x7fedb810d050 2021-04-09T18:21:41.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.510 7fedbdabe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb810d530 con 0x7fedb80d82e0 2021-04-09T18:21:41.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb6ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedb80d0160 0x7fedb80d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb6ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedb80d0160 0x7fedb80d7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48422/0 (socket says 172.21.15.32:48422) 2021-04-09T18:21:41.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb6ffd700 1 -- 172.21.15.32:0/387066077 learned_addr learned my addr 172.21.15.32:0/387066077 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:41.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb77fe700 1 --2- 172.21.15.32:0/387066077 >> v2:172.21.15.86:3300/0 conn(0x7fedb810d050 0x7fedb80cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb7fff700 1 --2- 172.21.15.32:0/387066077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80d82e0 0x7fedb810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb6ffd700 1 -- 172.21.15.32:0/387066077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80d82e0 msgr2=0x7fedb810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:41.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb6ffd700 1 --2- 172.21.15.32:0/387066077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80d82e0 0x7fedb810da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb6ffd700 1 -- 172.21.15.32:0/387066077 >> v2:172.21.15.86:3300/0 conn(0x7fedb810d050 msgr2=0x7fedb80cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:41.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb6ffd700 1 --2- 172.21.15.32:0/387066077 >> v2:172.21.15.86:3300/0 conn(0x7fedb810d050 0x7fedb80cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb6ffd700 1 -- 172.21.15.32:0/387066077 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb80d86e0 con 0x7fedb80d0160 2021-04-09T18:21:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb77fe700 1 --2- 172.21.15.32:0/387066077 >> v2:172.21.15.86:3300/0 conn(0x7fedb810d050 0x7fedb80cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb6ffd700 1 --2- 172.21.15.32:0/387066077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb80d0160 0x7fedb80d7cc0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7fedac006700 tx=0x7fedac009b00).ready entity=mon.0 client_cookie=c299206c9bdd781 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb67fc700 1 -- 172.21.15.32:0/387066077 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedac00c0c0 con 0x7fedb80d0160 2021-04-09T18:21:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb67fc700 1 -- 172.21.15.32:0/387066077 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac00c400 con 0x7fedb80d0160 2021-04-09T18:21:41.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedb67fc700 1 -- 172.21.15.32:0/387066077 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedac00c8e0 con 0x7fedb80d0160 2021-04-09T18:21:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedbdabe700 1 -- 172.21.15.32:0/387066077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb80d0160 msgr2=0x7fedb80d7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.511 7fedbdabe700 1 --2- 172.21.15.32:0/387066077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb80d0160 0x7fedb80d7cc0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7fedac006700 tx=0x7fedac009b00).stop 2021-04-09T18:21:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.512 7fedbdabe700 1 -- 172.21.15.32:0/387066077 shutdown_connections 2021-04-09T18:21:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.512 7fedbdabe700 1 --2- 172.21.15.32:0/387066077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80d82e0 0x7fedb810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.512 7fedbdabe700 1 --2- 172.21.15.32:0/387066077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb80d0160 0x7fedb80d7cc0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.512 7fedbdabe700 1 --2- 172.21.15.32:0/387066077 >> v2:172.21.15.86:3300/0 conn(0x7fedb810d050 0x7fedb80cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.512 7fedbdabe700 1 -- 172.21.15.32:0/387066077 >> 172.21.15.32:0/387066077 conn(0x7fedb80ce620 msgr2=0x7fedb80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.512 7fedbdabe700 1 -- 172.21.15.32:0/387066077 shutdown_connections 2021-04-09T18:21:41.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.512 7fedbdabe700 1 -- 172.21.15.32:0/387066077 wait complete. 2021-04-09T18:21:41.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.532 7fedbdabe700 1 Processor -- start 2021-04-09T18:21:41.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedbdabe700 1 -- start start 2021-04-09T18:21:41.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedbdabe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80ad600 0x7fedb809ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedbdabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb809b1b0 0x7fedb80ba220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedbdabe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb80ba840 0x7fedb80a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedbdabe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb809fa80 con 0x7fedb809b1b0 2021-04-09T18:21:41.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedbdabe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb809b5b0 con 0x7fedb80ba840 2021-04-09T18:21:41.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedbdabe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb80bac40 con 0x7fedb80ad600 2021-04-09T18:21:41.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedb6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb809b1b0 0x7fedb80ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedb6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb809b1b0 0x7fedb80ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48426/0 (socket says 172.21.15.32:48426) 2021-04-09T18:21:41.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedb6ffd700 1 -- 172.21.15.32:0/1317012835 learned_addr learned my addr 172.21.15.32:0/1317012835 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:41.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedb7fff700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb80ba840 0x7fedb80a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.533 7fedb77fe700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80ad600 0x7fedb809ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fedb6ffd700 1 -- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80ad600 msgr2=0x7fedb809ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:41.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fedb6ffd700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80ad600 0x7fedb809ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fedb6ffd700 1 -- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb80ba840 msgr2=0x7fedb80a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:41.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fedb6ffd700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb80ba840 0x7fedb80a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:41.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fedb6ffd700 1 -- 172.21.15.32:0/1317012835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb80a97c0 con 0x7fedb809b1b0 2021-04-09T18:21:41.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fedb7fff700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb80ba840 0x7fedb80a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:41.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fedb6ffd700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb809b1b0 0x7fedb80ba220 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7fedac006f20 tx=0x7fedac00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:41.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fed9ffff700 1 -- 172.21.15.32:0/1317012835 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedac00bdd0 con 0x7fedb809b1b0 2021-04-09T18:21:41.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fed9ffff700 1 -- 172.21.15.32:0/1317012835 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac006440 con 0x7fedb809b1b0 2021-04-09T18:21:41.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb80a97c0 con 0x7fedb809b1b0 2021-04-09T18:21:41.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fed9ffff700 1 -- 172.21.15.32:0/1317012835 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedac0052e0 con 0x7fedb809b1b0 2021-04-09T18:21:41.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.534 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb80a97c0 con 0x7fedb809b1b0 2021-04-09T18:21:41.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.535 7fed9ffff700 1 -- 172.21.15.32:0/1317012835 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fedac0052e0 con 0x7fedb809b1b0 2021-04-09T18:21:41.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.535 7fed9ffff700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed98018930 0x7fed9801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:41.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.535 7fed9ffff700 1 -- 172.21.15.32:0/1317012835 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fedac032a00 con 0x7fedb809b1b0 2021-04-09T18:21:41.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.535 7fedb77fe700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed98018930 0x7fed9801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:41.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.535 7fedb77fe700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed98018930 0x7fed9801ae10 crc :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:41.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.554 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb80aa330 con 0x7fedb809b1b0 2021-04-09T18:21:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.558 7fed9ffff700 1 -- 172.21.15.32:0/1317012835 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fedac006440 con 0x7fedb809b1b0 2021-04-09T18:21:41.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:41.740 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fedb80aa330 con 0x7fedb809b1b0 2021-04-09T18:21:42.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.292 7fed9ffff700 1 -- 172.21.15.32:0/1317012835 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v86) v1 ==== 185+0+0 (secure 0 0 0) 0x7fedac006030 con 0x7fedb809b1b0 2021-04-09T18:21:42.356 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:21:42.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed98018930 msgr2=0x7fed9801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:42.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed98018930 0x7fed9801ae10 crc :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb809b1b0 msgr2=0x7fedb80ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:42.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb809b1b0 0x7fedb80ba220 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7fedac006f20 tx=0x7fedac00b9e0).stop 2021-04-09T18:21:42.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 shutdown_connections 2021-04-09T18:21:42.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed98018930 0x7fed9801ae10 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb809b1b0 0x7fedb80ba220 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb80ba840 0x7fedb80a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 --2- 172.21.15.32:0/1317012835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedb80ad600 0x7fedb809ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 >> 172.21.15.32:0/1317012835 conn(0x7fedb80ae6f0 msgr2=0x7fedb8097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:42.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 shutdown_connections 2021-04-09T18:21:42.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.306 7fedbdabe700 1 -- 172.21.15.32:0/1317012835 wait complete. 2021-04-09T18:21:42.367 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:21:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.479 7f92e29a1700 1 Processor -- start 2021-04-09T18:21:42.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.479 7f92e29a1700 1 -- start start 2021-04-09T18:21:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.480 7f92e29a1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0d7cc0 0x7f92dc0d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.480 7f92e29a1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f92dc10da80 0x7f92dc10de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:42.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.480 7f92e29a1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92dc10d500 con 0x7f92dc10d050 2021-04-09T18:21:42.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.480 7f92e29a1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92dc0d9260 con 0x7f92dc10da80 2021-04-09T18:21:42.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.480 7f92e29a1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0d0330 con 0x7f92dc0d7cc0 2021-04-09T18:21:42.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.480 7f92dbfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f92dc10d050 0x7f92dc0cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58022/0 (socket says 172.21.15.32:58022) 2021-04-09T18:21:42.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.480 7f92dbfff700 1 -- 172.21.15.32:0/2635647747 learned_addr learned my addr 172.21.15.32:0/2635647747 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:42.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.480 7f92e0fbe700 1 --2- 172.21.15.32:0/2635647747 >> v2:172.21.15.86:3300/0 conn(0x7f92dc10da80 0x7f92dc10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:42.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.480 7f92d3fff700 1 --2- 172.21.15.32:0/2635647747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0d7cc0 0x7f92dc0d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:42.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92db7fe700 1 -- 172.21.15.32:0/2635647747 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682663690 0 0) 0x7f92cc000ce0 con 0x7f92dc10d050 2021-04-09T18:21:42.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92db7fe700 1 -- 172.21.15.32:0/2635647747 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c8002fe0 con 0x7f92dc10d050 2021-04-09T18:21:42.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92d3fff700 1 -- 172.21.15.32:0/2635647747 >> v2:172.21.15.86:3300/0 conn(0x7f92dc10da80 msgr2=0x7f92dc10de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92d3fff700 1 --2- 172.21.15.32:0/2635647747 >> v2:172.21.15.86:3300/0 conn(0x7f92dc10da80 0x7f92dc10de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92d3fff700 1 -- 172.21.15.32:0/2635647747 >> v1:172.21.15.32:6789/0 conn(0x7f92dc10d050 legacy=0x7f92dc0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:42.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92d3fff700 1 -- 172.21.15.32:0/2635647747 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc0d86e0 con 0x7f92dc0d7cc0 2021-04-09T18:21:42.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92e0fbe700 1 --2- 172.21.15.32:0/2635647747 >> v2:172.21.15.86:3300/0 conn(0x7f92dc10da80 0x7f92dc10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:42.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92d3fff700 1 --2- 172.21.15.32:0/2635647747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0d7cc0 0x7f92dc0d80c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f92c4006650 tx=0x7f92c4009a80).ready entity=mon.2 client_cookie=a19ba92f5e94a78b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:42.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92db7fe700 1 -- 172.21.15.32:0/2635647747 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92c40009b0 con 0x7f92dc0d7cc0 2021-04-09T18:21:42.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92db7fe700 1 -- 172.21.15.32:0/2635647747 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c400c0c0 con 0x7f92dc0d7cc0 2021-04-09T18:21:42.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92db7fe700 1 -- 172.21.15.32:0/2635647747 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92c400c510 con 0x7f92dc0d7cc0 2021-04-09T18:21:42.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92e29a1700 1 -- 172.21.15.32:0/2635647747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0d7cc0 msgr2=0x7f92dc0d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:42.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.481 7f92e29a1700 1 --2- 172.21.15.32:0/2635647747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0d7cc0 0x7f92dc0d80c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f92c4006650 tx=0x7f92c4009a80).stop 2021-04-09T18:21:42.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.482 7f92e29a1700 1 -- 172.21.15.32:0/2635647747 shutdown_connections 2021-04-09T18:21:42.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.482 7f92e29a1700 1 --2- 172.21.15.32:0/2635647747 >> v2:172.21.15.86:3300/0 conn(0x7f92dc10da80 0x7f92dc10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.482 7f92e29a1700 1 --2- 172.21.15.32:0/2635647747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0d7cc0 0x7f92dc0d80c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.482 7f92e29a1700 1 -- 172.21.15.32:0/2635647747 >> 172.21.15.32:0/2635647747 conn(0x7f92dc0ce620 msgr2=0x7f92dc0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:42.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.482 7f92e29a1700 1 -- 172.21.15.32:0/2635647747 shutdown_connections 2021-04-09T18:21:42.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.482 7f92e29a1700 1 -- 172.21.15.32:0/2635647747 wait complete. 2021-04-09T18:21:42.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.502 7f92e29a1700 1 Processor -- start 2021-04-09T18:21:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.502 7f92e29a1700 1 -- start start 2021-04-09T18:21:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e29a1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92dc0b84d0 0x7f92dc0c2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:42.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e29a1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0c2e20 0x7f92dc0a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:42.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e29a1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92dc0a1b60 0x7f92dc0a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e29a1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc18e9f0 con 0x7f92dc0a1b60 2021-04-09T18:21:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e29a1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0a8420 con 0x7f92dc0b84d0 2021-04-09T18:21:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e29a1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0b3d90 con 0x7f92dc0c2e20 2021-04-09T18:21:42.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e0fbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92dc0a1b60 0x7f92dc0a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:42.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e0fbe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92dc0a1b60 0x7f92dc0a7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48440/0 (socket says 172.21.15.32:48440) 2021-04-09T18:21:42.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e0fbe700 1 -- 172.21.15.32:0/1616818709 learned_addr learned my addr 172.21.15.32:0/1616818709 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:42.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92dbfff700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92dc0b84d0 0x7f92dc0c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92d3fff700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0c2e20 0x7f92dc0a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:42.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e0fbe700 1 -- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0c2e20 msgr2=0x7f92dc0a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e0fbe700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0c2e20 0x7f92dc0a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e0fbe700 1 -- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92dc0b84d0 msgr2=0x7f92dc0c2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e0fbe700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92dc0b84d0 0x7f92dc0c2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:42.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e0fbe700 1 -- 172.21.15.32:0/1616818709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc0a7cb0 con 0x7f92dc0a1b60 2021-04-09T18:21:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e0fbe700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92dc0a1b60 0x7f92dc0a7690 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7f92d400b970 tx=0x7f92d400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:42.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92d97fa700 1 -- 172.21.15.32:0/1616818709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92d4000f90 con 0x7f92dc0a1b60 2021-04-09T18:21:42.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92d97fa700 1 -- 172.21.15.32:0/1616818709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d4006aa0 con 0x7f92dc0a1b60 2021-04-09T18:21:42.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92dc0a7cb0 con 0x7f92dc0a1b60 2021-04-09T18:21:42.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.503 7f92d97fa700 1 -- 172.21.15.32:0/1616818709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92d40015f0 con 0x7f92dc0a1b60 2021-04-09T18:21:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.504 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92dc0a7cb0 con 0x7f92dc0a1b60 2021-04-09T18:21:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.504 7f92d97fa700 1 -- 172.21.15.32:0/1616818709 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f92d4000e70 con 0x7f92dc0a1b60 2021-04-09T18:21:42.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.504 7f92d97fa700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f92c00188d0 0x7f92c001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:42.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.504 7f92d97fa700 1 -- 172.21.15.32:0/1616818709 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5225+0+0 (secure 0 0 0) 0x7f92d4032c10 con 0x7f92dc0a1b60 2021-04-09T18:21:42.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.504 7f92dbfff700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f92c00188d0 0x7f92c001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:42.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.505 7f92dbfff700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f92c00188d0 0x7f92c001add0 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:42.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.524 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92dc1b7180 con 0x7f92dc0a1b60 2021-04-09T18:21:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.528 7f92d97fa700 1 -- 172.21.15.32:0/1616818709 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f92d40066f0 con 0x7f92dc0a1b60 2021-04-09T18:21:42.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:42.710 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f92dc10d500 con 0x7f92dc0a1b60 2021-04-09T18:21:43.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.295 7f92d97fa700 1 -- 172.21.15.32:0/1616818709 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v87) v1 ==== 173+0+0 (secure 0 0 0) 0x7f92d4006aa0 con 0x7f92dc0a1b60 2021-04-09T18:21:43.375 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:21:43.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f92c00188d0 msgr2=0x7f92c001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f92c00188d0 0x7f92c001add0 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92dc0a1b60 msgr2=0x7f92dc0a7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92dc0a1b60 0x7f92dc0a7690 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7f92d400b970 tx=0x7f92d400bd10).stop 2021-04-09T18:21:43.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 shutdown_connections 2021-04-09T18:21:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f92c00188d0 0x7f92c001add0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92dc0a1b60 0x7f92dc0a7690 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92dc0c2e20 0x7f92dc0a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 --2- 172.21.15.32:0/1616818709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92dc0b84d0 0x7f92dc0c2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 >> 172.21.15.32:0/1616818709 conn(0x7f92dc0b6b70 msgr2=0x7f92dc0b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 shutdown_connections 2021-04-09T18:21:43.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.328 7f92e29a1700 1 -- 172.21.15.32:0/1616818709 wait complete. 2021-04-09T18:21:43.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:21:43.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.493 7f3d37dec700 1 Processor -- start 2021-04-09T18:21:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.494 7f3d37dec700 1 -- start start 2021-04-09T18:21:43.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.494 7f3d37dec700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d3010e7e0 0x7f3d300d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:43.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.494 7f3d37dec700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d300d7290 con 0x7f3d300cffb0 2021-04-09T18:21:43.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.494 7f3d37dec700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d300d2450 con 0x7f3d300d6c90 2021-04-09T18:21:43.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.494 7f3d37dec700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d300d7090 con 0x7f3d3010e7e0 2021-04-09T18:21:43.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.494 7f3d35c08700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d300cffb0 0x7f3d300d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58036/0 (socket says 172.21.15.32:58036) 2021-04-09T18:21:43.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.494 7f3d35c08700 1 -- 172.21.15.32:0/1006079188 learned_addr learned my addr 172.21.15.32:0/1006079188 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d36409700 1 --2- 172.21.15.32:0/1006079188 >> v2:172.21.15.86:3300/0 conn(0x7f3d3010e7e0 0x7f3d300d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342686833 0 0) 0x7f3d2c000ce0 con 0x7f3d300cffb0 2021-04-09T18:21:43.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c002fe0 con 0x7f3d300cffb0 2021-04-09T18:21:43.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588631037 0 0) 0x7f3d20000c60 con 0x7f3d300d6c90 2021-04-09T18:21:43.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c003210 con 0x7f3d300d6c90 2021-04-09T18:21:43.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 701085713 0 0) 0x7f3d2c0010e0 con 0x7f3d300cffb0 2021-04-09T18:21:43.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 >> v2:172.21.15.86:3300/0 conn(0x7f3d3010e7e0 msgr2=0x7f3d300d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:43.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 --2- 172.21.15.32:0/1006079188 >> v2:172.21.15.86:3300/0 conn(0x7f3d3010e7e0 0x7f3d300d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 >> v1:172.21.15.86:6789/0 conn(0x7f3d300d6c90 legacy=0x7f3d3010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:43.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d300d91f0 con 0x7f3d300cffb0 2021-04-09T18:21:43.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f3d2c001600 con 0x7f3d300cffb0 2021-04-09T18:21:43.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 >> v1:172.21.15.32:6789/0 conn(0x7f3d300cffb0 legacy=0x7f3d300d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:43.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1c003b90 0x7f3d1c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:43.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0067a0 0x7f3d1c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:43.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c0095a0 0x7f3d1c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:43.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d35c08700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1c003b90 0x7f3d1c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:43.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c050320 con 0x7f3d1c003b90 2021-04-09T18:21:43.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c050500 con 0x7f3d1c0067a0 2021-04-09T18:21:43.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c050720 con 0x7f3d1c0095a0 2021-04-09T18:21:43.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d35407700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0067a0 0x7f3d1c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:43.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d36409700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c0095a0 0x7f3d1c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:43.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d35c08700 1 -- 172.21.15.32:0/1006079188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c0095a0 msgr2=0x7f3d1c00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:43.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d35c08700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c0095a0 0x7f3d1c00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d35c08700 1 -- 172.21.15.32:0/1006079188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0067a0 msgr2=0x7f3d1c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:43.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d35c08700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0067a0 0x7f3d1c008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d35c08700 1 -- 172.21.15.32:0/1006079188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0509c0 con 0x7f3d1c003b90 2021-04-09T18:21:43.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.495 7f3d36409700 1 -- 172.21.15.32:0/1006079188 reap_dead start 2021-04-09T18:21:43.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.496 7f3d35c08700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1c003b90 0x7f3d1c005fa0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7f3d2c007970 tx=0x7f3d2c007d10).ready entity=mon.0 client_cookie=452eb7dde798cc16 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:43.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.496 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d2c0010e0 con 0x7f3d1c003b90 2021-04-09T18:21:43.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.496 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c0070c0 con 0x7f3d1c003b90 2021-04-09T18:21:43.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.496 7f3d34c06700 1 -- 172.21.15.32:0/1006079188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d2c00a710 con 0x7f3d1c003b90 2021-04-09T18:21:43.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.496 7f3d37dec700 1 -- 172.21.15.32:0/1006079188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1c003b90 msgr2=0x7f3d1c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:43.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.496 7f3d37dec700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1c003b90 0x7f3d1c005fa0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7f3d2c007970 tx=0x7f3d2c007d10).stop 2021-04-09T18:21:43.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.496 7f3d37dec700 1 -- 172.21.15.32:0/1006079188 shutdown_connections 2021-04-09T18:21:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.496 7f3d37dec700 1 --2- 172.21.15.32:0/1006079188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d1c003b90 0x7f3d1c005fa0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.496 7f3d37dec700 1 -- 172.21.15.32:0/1006079188 >> 172.21.15.32:0/1006079188 conn(0x7f3d300ce5d0 msgr2=0x7f3d300cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:43.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.497 7f3d37dec700 1 -- 172.21.15.32:0/1006079188 shutdown_connections 2021-04-09T18:21:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.497 7f3d37dec700 1 -- 172.21.15.32:0/1006079188 wait complete. 2021-04-09T18:21:43.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.517 7f3d37dec700 1 Processor -- start 2021-04-09T18:21:43.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.517 7f3d37dec700 1 -- start start 2021-04-09T18:21:43.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d37dec700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d30184080 0x7f3d301955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:43.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d37dec700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d30195c00 0x7f3d3008ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:43.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d37dec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d3008f550 0x7f3d3008f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:43.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d37dec700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30090570 con 0x7f3d3008f550 2021-04-09T18:21:43.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d37dec700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30184480 con 0x7f3d30195c00 2021-04-09T18:21:43.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d37dec700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30196000 con 0x7f3d30184080 2021-04-09T18:21:43.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d36409700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d3008f550 0x7f3d3008f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:43.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d36409700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d3008f550 0x7f3d3008f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48460/0 (socket says 172.21.15.32:48460) 2021-04-09T18:21:43.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d36409700 1 -- 172.21.15.32:0/385662767 learned_addr learned my addr 172.21.15.32:0/385662767 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:43.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d35407700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d30195c00 0x7f3d3008ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:43.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d35c08700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d30184080 0x7f3d301955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:43.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d36409700 1 -- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d30184080 msgr2=0x7f3d301955e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d36409700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d30184080 0x7f3d301955e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d36409700 1 -- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d30195c00 msgr2=0x7f3d3008ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d36409700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d30195c00 0x7f3d3008ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d36409700 1 -- 172.21.15.32:0/385662767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d301847b0 con 0x7f3d3008f550 2021-04-09T18:21:43.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d35407700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d30195c00 0x7f3d3008ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:43.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d36409700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d3008f550 0x7f3d3008f950 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7f3d28006fa0 tx=0x7f3d2800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:43.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d267fc700 1 -- 172.21.15.32:0/385662767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d28000f60 con 0x7f3d3008f550 2021-04-09T18:21:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d267fc700 1 -- 172.21.15.32:0/385662767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2800c2f0 con 0x7f3d3008f550 2021-04-09T18:21:43.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.518 7f3d37dec700 1 -- 172.21.15.32:0/385662767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d301847b0 con 0x7f3d3008f550 2021-04-09T18:21:43.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.519 7f3d267fc700 1 -- 172.21.15.32:0/385662767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d2800c7d0 con 0x7f3d3008f550 2021-04-09T18:21:43.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.519 7f3d37dec700 1 -- 172.21.15.32:0/385662767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d301847b0 con 0x7f3d3008f550 2021-04-09T18:21:43.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.519 7f3d267fc700 1 -- 172.21.15.32:0/385662767 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3d28000df0 con 0x7f3d3008f550 2021-04-09T18:21:43.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.519 7f3d267fc700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d18018930 0x7f3d1801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:43.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.519 7f3d267fc700 1 -- 172.21.15.32:0/385662767 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f3d28032c20 con 0x7f3d3008f550 2021-04-09T18:21:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.520 7f3d35c08700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d18018930 0x7f3d1801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:43.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.520 7f3d35c08700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d18018930 0x7f3d1801ae10 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:43.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.538 7f3d37dec700 1 -- 172.21.15.32:0/385662767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d30182e80 con 0x7f3d3008f550 2021-04-09T18:21:43.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.542 7f3d267fc700 1 -- 172.21.15.32:0/385662767 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3d2800be30 con 0x7f3d3008f550 2021-04-09T18:21:43.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.754 7f3d37dec700 1 -- 172.21.15.32:0/385662767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3d300d7290 con 0x7f3d3008f550 2021-04-09T18:21:43.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.757 7f3d267fc700 1 -- 172.21.15.32:0/385662767 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3d28000df0 con 0x7f3d3008f550 2021-04-09T18:21:43.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 -- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d18018930 msgr2=0x7f3d1801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:43.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d18018930 0x7f3d1801ae10 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 -- 172.21.15.32:0/385662767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d3008f550 msgr2=0x7f3d3008f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:43.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d3008f550 0x7f3d3008f950 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7f3d28006fa0 tx=0x7f3d2800bcf0).stop 2021-04-09T18:21:43.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 -- 172.21.15.32:0/385662767 shutdown_connections 2021-04-09T18:21:43.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d18018930 0x7f3d1801ae10 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d3008f550 0x7f3d3008f950 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d30195c00 0x7f3d3008ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 --2- 172.21.15.32:0/385662767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d30184080 0x7f3d301955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:43.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 -- 172.21.15.32:0/385662767 >> 172.21.15.32:0/385662767 conn(0x7f3d301a8f50 msgr2=0x7f3d30188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:43.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 -- 172.21.15.32:0/385662767 shutdown_connections 2021-04-09T18:21:43.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.763 7f3d37dec700 1 -- 172.21.15.32:0/385662767 wait complete. 2021-04-09T18:21:43.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_hole' 2021-04-09T18:21:44.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.925 7f00f0910700 1 Processor -- start 2021-04-09T18:21:44.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.925 7f00f0910700 1 -- start start 2021-04-09T18:21:44.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00f0910700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00ec10d010 0x7f00ec0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00f0910700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec0d1da0 0x7f00ec0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00f0910700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ec10d480 con 0x7f00ec0d82c0 2021-04-09T18:21:44.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00f0910700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ec0d9220 con 0x7f00ec10d010 2021-04-09T18:21:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00f0910700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0d86c0 con 0x7f00ec0d1da0 2021-04-09T18:21:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00ea61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00ec10d010 0x7f00ec0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00ea61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00ec10d010 0x7f00ec0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48464/0 (socket says 172.21.15.32:48464) 2021-04-09T18:21:44.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00ea61c700 1 -- 172.21.15.32:0/132192216 learned_addr learned my addr 172.21.15.32:0/132192216 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:44.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00e9e1b700 1 --2- 172.21.15.32:0/132192216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec0d1da0 0x7f00ec0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00ea61c700 1 -- 172.21.15.32:0/132192216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec0d1da0 msgr2=0x7f00ec0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:44.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00ea61c700 1 --2- 172.21.15.32:0/132192216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec0d1da0 0x7f00ec0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00ea61c700 1 -- 172.21.15.32:0/132192216 >> v1:172.21.15.86:6789/0 conn(0x7f00ec0d82c0 legacy=0x7f00ec10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00ea61c700 1 -- 172.21.15.32:0/132192216 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec10e1f0 con 0x7f00ec10d010 2021-04-09T18:21:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00ea61c700 1 --2- 172.21.15.32:0/132192216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec10d010 0x7f00ec0d1780 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f00dc006700 tx=0x7f00dc009b80).ready entity=mon.0 client_cookie=eda4ee8d17c2e1a8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00e961a700 1 -- 172.21.15.32:0/132192216 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00dc00c0c0 con 0x7f00ec10d010 2021-04-09T18:21:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.926 7f00e961a700 1 -- 172.21.15.32:0/132192216 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00dc00c440 con 0x7f00ec10d010 2021-04-09T18:21:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.927 7f00e961a700 1 -- 172.21.15.32:0/132192216 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00dc00c920 con 0x7f00ec10d010 2021-04-09T18:21:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.927 7f00f0910700 1 -- 172.21.15.32:0/132192216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec10d010 msgr2=0x7f00ec0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.927 7f00f0910700 1 --2- 172.21.15.32:0/132192216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec10d010 0x7f00ec0d1780 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f00dc006700 tx=0x7f00dc009b80).stop 2021-04-09T18:21:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.927 7f00f0910700 1 -- 172.21.15.32:0/132192216 shutdown_connections 2021-04-09T18:21:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.927 7f00f0910700 1 --2- 172.21.15.32:0/132192216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec0d1da0 0x7f00ec0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.927 7f00f0910700 1 --2- 172.21.15.32:0/132192216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec10d010 0x7f00ec0d1780 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.927 7f00f0910700 1 -- 172.21.15.32:0/132192216 >> 172.21.15.32:0/132192216 conn(0x7f00ec0ce5f0 msgr2=0x7f00ec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.927 7f00f0910700 1 -- 172.21.15.32:0/132192216 shutdown_connections 2021-04-09T18:21:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.927 7f00f0910700 1 -- 172.21.15.32:0/132192216 wait complete. 2021-04-09T18:21:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.947 7f00f0910700 1 Processor -- start 2021-04-09T18:21:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00f0910700 1 -- start start 2021-04-09T18:21:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00f0910700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec179bf0 0x7f00ec179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00f0910700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec17c540 0x7f00ec17c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00f0910700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0aca60 0x7f00ec0ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00f0910700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec17cf60 con 0x7f00ec0aca60 2021-04-09T18:21:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00f0910700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0ad480 con 0x7f00ec17c540 2021-04-09T18:21:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00f0910700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0ad6a0 con 0x7f00ec179bf0 2021-04-09T18:21:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00eae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0aca60 0x7f00ec0ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00eae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0aca60 0x7f00ec0ace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48472/0 (socket says 172.21.15.32:48472) 2021-04-09T18:21:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00eae1d700 1 -- 172.21.15.32:0/3020011002 learned_addr learned my addr 172.21.15.32:0/3020011002 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00e9e1b700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec17c540 0x7f00ec17c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.948 7f00ea61c700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec179bf0 0x7f00ec179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00eae1d700 1 -- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec179bf0 msgr2=0x7f00ec179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00eae1d700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec179bf0 0x7f00ec179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00eae1d700 1 -- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec17c540 msgr2=0x7f00ec17c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00eae1d700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec17c540 0x7f00ec17c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00eae1d700 1 -- 172.21.15.32:0/3020011002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec0ad8c0 con 0x7f00ec0aca60 2021-04-09T18:21:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00ea61c700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec179bf0 0x7f00ec179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00e9e1b700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec17c540 0x7f00ec17c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00eae1d700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0aca60 0x7f00ec0ace60 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f00e4009940 tx=0x7f00e400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:44.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00daffd700 1 -- 172.21.15.32:0/3020011002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00e4000d30 con 0x7f00ec0aca60 2021-04-09T18:21:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00daffd700 1 -- 172.21.15.32:0/3020011002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e400c2f0 con 0x7f00ec0aca60 2021-04-09T18:21:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00f0910700 1 -- 172.21.15.32:0/3020011002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ec0ad8c0 con 0x7f00ec0aca60 2021-04-09T18:21:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00daffd700 1 -- 172.21.15.32:0/3020011002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00e400c7d0 con 0x7f00ec0aca60 2021-04-09T18:21:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.949 7f00f0910700 1 -- 172.21.15.32:0/3020011002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec0ad8c0 con 0x7f00ec0aca60 2021-04-09T18:21:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.950 7f00daffd700 1 -- 172.21.15.32:0/3020011002 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f00e4001220 con 0x7f00ec0aca60 2021-04-09T18:21:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.950 7f00daffd700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00d00188d0 0x7f00d001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.950 7f00daffd700 1 -- 172.21.15.32:0/3020011002 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f00e4032ad0 con 0x7f00ec0aca60 2021-04-09T18:21:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.950 7f00ea61c700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00d00188d0 0x7f00d001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.950 7f00ea61c700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00d00188d0 0x7f00d001add0 crc :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:44.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.970 7f00f0910700 1 -- 172.21.15.32:0/3020011002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ec17cf60 con 0x7f00ec0aca60 2021-04-09T18:21:44.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:43.973 7f00daffd700 1 -- 172.21.15.32:0/3020011002 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f00e400be30 con 0x7f00ec0aca60 2021-04-09T18:21:44.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.159 7f00f0910700 1 -- 172.21.15.32:0/3020011002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_hole"]} v 0) v1 -- 0x7f00ec17cf60 con 0x7f00ec0aca60 2021-04-09T18:21:44.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.296 7f00daffd700 1 -- 172.21.15.32:0/3020011002 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_hole"]}]=0 v405) v1 ==== 133+0+0 (secure 0 0 0) 0x7f00e4001220 con 0x7f00ec0aca60 2021-04-09T18:21:44.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.325 7f00f0910700 1 -- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00d00188d0 msgr2=0x7f00d001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.325 7f00f0910700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00d00188d0 0x7f00d001add0 crc :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.325 7f00f0910700 1 -- 172.21.15.32:0/3020011002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0aca60 msgr2=0x7f00ec0ace60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.325 7f00f0910700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0aca60 0x7f00ec0ace60 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f00e4009940 tx=0x7f00e400bcf0).stop 2021-04-09T18:21:44.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.326 7f00f0910700 1 -- 172.21.15.32:0/3020011002 shutdown_connections 2021-04-09T18:21:44.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.326 7f00f0910700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00d00188d0 0x7f00d001add0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.326 7f00f0910700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00ec0aca60 0x7f00ec0ace60 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.326 7f00f0910700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ec17c540 0x7f00ec17c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.326 7f00f0910700 1 --2- 172.21.15.32:0/3020011002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ec179bf0 0x7f00ec179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.326 7f00f0910700 1 -- 172.21.15.32:0/3020011002 >> 172.21.15.32:0/3020011002 conn(0x7f00ec1acae0 msgr2=0x7f00ec17ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.326 7f00f0910700 1 -- 172.21.15.32:0/3020011002 shutdown_connections 2021-04-09T18:21:44.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.326 7f00f0910700 1 -- 172.21.15.32:0/3020011002 wait complete. 2021-04-09T18:21:44.387 INFO:tasks.cephfs_test_runner:test_hole (tasks.cephfs.test_failover.TestClusterResize) ... ok 2021-04-09T18:21:44.388 INFO:tasks.cephfs_test_runner:Starting test: test_shrink (tasks.cephfs.test_failover.TestClusterResize) 2021-04-09T18:21:44.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_shrink' 2021-04-09T18:21:44.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.486 7fd9de3f8700 1 Processor -- start 2021-04-09T18:21:44.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.486 7fd9de3f8700 1 -- start start 2021-04-09T18:21:44.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9de3f8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd9d810da00 0x7fd9d810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9de3f8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d810d480 con 0x7fd9d80d7ca0 2021-04-09T18:21:44.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9de3f8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d80d1f70 con 0x7fd9d810d010 2021-04-09T18:21:44.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9de3f8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d80d8d30 con 0x7fd9d810da00 2021-04-09T18:21:44.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9cffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd9d80d7ca0 0x7fd9d80d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58068/0 (socket says 172.21.15.32:58068) 2021-04-09T18:21:44.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9cffff700 1 -- 172.21.15.32:0/1830280585 learned_addr learned my addr 172.21.15.32:0/1830280585 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9dca15700 1 --2- 172.21.15.32:0/1830280585 >> v2:172.21.15.86:3300/0 conn(0x7fd9d810da00 0x7fd9d810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9d77fe700 1 -- 172.21.15.32:0/1830280585 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862754802 0 0) 0x7fd9c0000ce0 con 0x7fd9d80d7ca0 2021-04-09T18:21:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9d77fe700 1 -- 172.21.15.32:0/1830280585 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c4002fe0 con 0x7fd9d80d7ca0 2021-04-09T18:21:44.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9d77fe700 1 -- 172.21.15.32:0/1830280585 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423174891 0 0) 0x7fd9c8000c60 con 0x7fd9d810d010 2021-04-09T18:21:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.487 7fd9d77fe700 1 -- 172.21.15.32:0/1830280585 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c40042c0 con 0x7fd9d810d010 2021-04-09T18:21:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.488 7fd9dca15700 1 -- 172.21.15.32:0/1830280585 >> v1:172.21.15.86:6789/0 conn(0x7fd9d810d010 legacy=0x7fd9d80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.488 7fd9dca15700 1 -- 172.21.15.32:0/1830280585 >> v1:172.21.15.32:6789/0 conn(0x7fd9d80d7ca0 legacy=0x7fd9d80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.488 7fd9dca15700 1 -- 172.21.15.32:0/1830280585 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d80d9220 con 0x7fd9d810da00 2021-04-09T18:21:44.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.488 7fd9dca15700 1 --2- 172.21.15.32:0/1830280585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d810da00 0x7fd9d810de00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fd9d0006e50 tx=0x7fd9d0009e70).ready entity=mon.1 client_cookie=4069f66ed7bd101c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:44.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.488 7fd9d77fe700 1 -- 172.21.15.32:0/1830280585 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d0006a20 con 0x7fd9d810da00 2021-04-09T18:21:44.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.488 7fd9d77fe700 1 -- 172.21.15.32:0/1830280585 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d0006790 con 0x7fd9d810da00 2021-04-09T18:21:44.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.488 7fd9d77fe700 1 -- 172.21.15.32:0/1830280585 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d000b700 con 0x7fd9d810da00 2021-04-09T18:21:44.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.488 7fd9de3f8700 1 -- 172.21.15.32:0/1830280585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d810da00 msgr2=0x7fd9d810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.488 7fd9de3f8700 1 --2- 172.21.15.32:0/1830280585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d810da00 0x7fd9d810de00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fd9d0006e50 tx=0x7fd9d0009e70).stop 2021-04-09T18:21:44.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.489 7fd9de3f8700 1 -- 172.21.15.32:0/1830280585 shutdown_connections 2021-04-09T18:21:44.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.489 7fd9de3f8700 1 --2- 172.21.15.32:0/1830280585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d810da00 0x7fd9d810de00 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.489 7fd9de3f8700 1 -- 172.21.15.32:0/1830280585 >> 172.21.15.32:0/1830280585 conn(0x7fd9d80ce5f0 msgr2=0x7fd9d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:44.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.489 7fd9de3f8700 1 -- 172.21.15.32:0/1830280585 shutdown_connections 2021-04-09T18:21:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.489 7fd9de3f8700 1 -- 172.21.15.32:0/1830280585 wait complete. 2021-04-09T18:21:44.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.509 7fd9de3f8700 1 Processor -- start 2021-04-09T18:21:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.509 7fd9de3f8700 1 -- start start 2021-04-09T18:21:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.509 7fd9de3f8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d8185170 0x7fd9d8185570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.509 7fd9de3f8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9d808c520 0x7fd9d808c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.509 7fd9de3f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d808a650 0x7fd9d808aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.509 7fd9de3f8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d808e6e0 con 0x7fd9d808a650 2021-04-09T18:21:44.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.509 7fd9de3f8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d8185b90 con 0x7fd9d8185170 2021-04-09T18:21:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.509 7fd9de3f8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d808cf40 con 0x7fd9d808c520 2021-04-09T18:21:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9dca15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d808a650 0x7fd9d808aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9dca15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d808a650 0x7fd9d808aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48486/0 (socket says 172.21.15.32:48486) 2021-04-09T18:21:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9dca15700 1 -- 172.21.15.32:0/1770772830 learned_addr learned my addr 172.21.15.32:0/1770772830 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:44.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9d7fff700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d8185170 0x7fd9d8185570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9cffff700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9d808c520 0x7fd9d808c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9dca15700 1 -- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9d808c520 msgr2=0x7fd9d808c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9dca15700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9d808c520 0x7fd9d808c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9dca15700 1 -- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d8185170 msgr2=0x7fd9d8185570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9dca15700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d8185170 0x7fd9d8185570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9dca15700 1 -- 172.21.15.32:0/1770772830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d808b070 con 0x7fd9d808a650 2021-04-09T18:21:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9d7fff700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d8185170 0x7fd9d8185570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:21:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9cffff700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9d808c520 0x7fd9d808c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:21:44.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9dca15700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d808a650 0x7fd9d808aa50 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7fd9d00099f0 tx=0x7fd9d000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9d57fa700 1 -- 172.21.15.32:0/1770772830 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d00093d0 con 0x7fd9d808a650 2021-04-09T18:21:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9d57fa700 1 -- 172.21.15.32:0/1770772830 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d000be50 con 0x7fd9d808a650 2021-04-09T18:21:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d808b070 con 0x7fd9d808a650 2021-04-09T18:21:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9d57fa700 1 -- 172.21.15.32:0/1770772830 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d00090c0 con 0x7fd9d808a650 2021-04-09T18:21:44.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.510 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d808b070 con 0x7fd9d808a650 2021-04-09T18:21:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.511 7fd9d57fa700 1 -- 172.21.15.32:0/1770772830 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd9d00011a0 con 0x7fd9d808a650 2021-04-09T18:21:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.511 7fd9d57fa700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd9bc0188d0 0x7fd9bc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.511 7fd9d57fa700 1 -- 172.21.15.32:0/1770772830 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fd9d0032ac0 con 0x7fd9d808a650 2021-04-09T18:21:44.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.511 7fd9d7fff700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd9bc0188d0 0x7fd9bc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:44.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.511 7fd9d7fff700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd9bc0188d0 0x7fd9bc01add0 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.531 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d808e6e0 con 0x7fd9d808a650 2021-04-09T18:21:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.534 7fd9d57fa700 1 -- 172.21.15.32:0/1770772830 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd9d0000df0 con 0x7fd9d808a650 2021-04-09T18:21:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:44.722 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_shrink"]} v 0) v1 -- 0x7fd9d808e6e0 con 0x7fd9d808a650 2021-04-09T18:21:45.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.301 7fd9d57fa700 1 -- 172.21.15.32:0/1770772830 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_shrink"]}]=0 v406) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd9d00011a0 con 0x7fd9d808a650 2021-04-09T18:21:45.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd9bc0188d0 msgr2=0x7fd9bc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:45.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd9bc0188d0 0x7fd9bc01add0 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d808a650 msgr2=0x7fd9d808aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:45.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d808a650 0x7fd9d808aa50 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7fd9d00099f0 tx=0x7fd9d000bd10).stop 2021-04-09T18:21:45.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 shutdown_connections 2021-04-09T18:21:45.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd9bc0188d0 0x7fd9bc01add0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9d808a650 0x7fd9d808aa50 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9d808c520 0x7fd9d808c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 --2- 172.21.15.32:0/1770772830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9d8185170 0x7fd9d8185570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 >> 172.21.15.32:0/1770772830 conn(0x7fd9d8190550 msgr2=0x7fd9d81b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:45.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 shutdown_connections 2021-04-09T18:21:45.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.340 7fd9de3f8700 1 -- 172.21.15.32:0/1770772830 wait complete. 2021-04-09T18:21:45.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:21:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.510 7fbcf099b700 1 Processor -- start 2021-04-09T18:21:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.510 7fbcf099b700 1 -- start start 2021-04-09T18:21:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.511 7fbcf099b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbcec0d7ca0 0x7fbcec0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.511 7fbcf099b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec10da00 0x7fbcec10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.511 7fbcf099b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec10d480 con 0x7fbcec10d010 2021-04-09T18:21:45.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.511 7fbcf099b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcec0d9220 con 0x7fbcec0d7ca0 2021-04-09T18:21:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.511 7fbcf099b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0d86c0 con 0x7fbcec10da00 2021-04-09T18:21:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.511 7fbce9e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbcec0d7ca0 0x7fbcec0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.511 7fbcea61c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbcec10d010 0x7fbcec0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58082/0 (socket says 172.21.15.32:58082) 2021-04-09T18:21:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.511 7fbcea61c700 1 -- 172.21.15.32:0/240535364 learned_addr learned my addr 172.21.15.32:0/240535364 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:45.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.511 7fbceae1d700 1 --2- 172.21.15.32:0/240535364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec10da00 0x7fbcec10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:45.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbce9e1b700 1 -- 172.21.15.32:0/240535364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec10da00 msgr2=0x7fbcec10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:45.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbce9e1b700 1 --2- 172.21.15.32:0/240535364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec10da00 0x7fbcec10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbce9e1b700 1 -- 172.21.15.32:0/240535364 >> v1:172.21.15.32:6789/0 conn(0x7fbcec10d010 legacy=0x7fbcec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:45.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbce9e1b700 1 -- 172.21.15.32:0/240535364 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec0d1f70 con 0x7fbcec0d7ca0 2021-04-09T18:21:45.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbce961a700 1 -- 172.21.15.32:0/240535364 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235049816 0 0) 0x7fbcdc000ce0 con 0x7fbcec10d010 2021-04-09T18:21:45.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbce9e1b700 1 --2- 172.21.15.32:0/240535364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec0d7ca0 0x7fbcec0d80a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7fbcd4006650 tx=0x7fbcd4009a00).ready entity=mon.0 client_cookie=b1ac302078694e55 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:45.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbce961a700 1 -- 172.21.15.32:0/240535364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcd4009df0 con 0x7fbcec0d7ca0 2021-04-09T18:21:45.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbce961a700 1 -- 172.21.15.32:0/240535364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd40009b0 con 0x7fbcec0d7ca0 2021-04-09T18:21:45.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbce961a700 1 -- 172.21.15.32:0/240535364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcd400c2b0 con 0x7fbcec0d7ca0 2021-04-09T18:21:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbcf099b700 1 -- 172.21.15.32:0/240535364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec0d7ca0 msgr2=0x7fbcec0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.512 7fbcf099b700 1 --2- 172.21.15.32:0/240535364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec0d7ca0 0x7fbcec0d80a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7fbcd4006650 tx=0x7fbcd4009a00).stop 2021-04-09T18:21:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.513 7fbcf099b700 1 -- 172.21.15.32:0/240535364 shutdown_connections 2021-04-09T18:21:45.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.513 7fbcf099b700 1 --2- 172.21.15.32:0/240535364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec10da00 0x7fbcec10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.513 7fbcf099b700 1 --2- 172.21.15.32:0/240535364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec0d7ca0 0x7fbcec0d80a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.513 7fbcf099b700 1 -- 172.21.15.32:0/240535364 >> 172.21.15.32:0/240535364 conn(0x7fbcec0ce5f0 msgr2=0x7fbcec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:45.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.513 7fbcf099b700 1 -- 172.21.15.32:0/240535364 shutdown_connections 2021-04-09T18:21:45.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.513 7fbcf099b700 1 -- 172.21.15.32:0/240535364 wait complete. 2021-04-09T18:21:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.533 7fbcf099b700 1 Processor -- start 2021-04-09T18:21:45.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.533 7fbcf099b700 1 -- start start 2021-04-09T18:21:45.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbcf099b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcec09c5b0 0x7fbcec09d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:45.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbcf099b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec09c9b0 0x7fbcec09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:45.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbcf099b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec090390 0x7fbcec090790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbcf099b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec09e290 con 0x7fbcec09c9b0 2021-04-09T18:21:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbcf099b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec09cf90 con 0x7fbcec09c5b0 2021-04-09T18:21:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbcf099b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec090db0 con 0x7fbcec090390 2021-04-09T18:21:45.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce9e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec09c9b0 0x7fbcec09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:45.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce9e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec09c9b0 0x7fbcec09db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48500/0 (socket says 172.21.15.32:48500) 2021-04-09T18:21:45.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce9e1b700 1 -- 172.21.15.32:0/3132994550 learned_addr learned my addr 172.21.15.32:0/3132994550 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:45.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbcea61c700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcec09c5b0 0x7fbcec09d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:45.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbceae1d700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec090390 0x7fbcec090790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce9e1b700 1 -- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec090390 msgr2=0x7fbcec090790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce9e1b700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec090390 0x7fbcec090790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce9e1b700 1 -- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcec09c5b0 msgr2=0x7fbcec09d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:45.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce9e1b700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcec09c5b0 0x7fbcec09d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce9e1b700 1 -- 172.21.15.32:0/3132994550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec090f90 con 0x7fbcec09c9b0 2021-04-09T18:21:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce9e1b700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec09c9b0 0x7fbcec09db30 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7fbcd40096a0 tx=0x7fbcd400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce2ffd700 1 -- 172.21.15.32:0/3132994550 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcd4000c90 con 0x7fbcec09c9b0 2021-04-09T18:21:45.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce2ffd700 1 -- 172.21.15.32:0/3132994550 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd4002030 con 0x7fbcec09c9b0 2021-04-09T18:21:45.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec090f90 con 0x7fbcec09c9b0 2021-04-09T18:21:45.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.534 7fbce2ffd700 1 -- 172.21.15.32:0/3132994550 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcd4002480 con 0x7fbcec09c9b0 2021-04-09T18:21:45.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.535 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcec090f90 con 0x7fbcec09c9b0 2021-04-09T18:21:45.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.535 7fbce2ffd700 1 -- 172.21.15.32:0/3132994550 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbcd4002210 con 0x7fbcec09c9b0 2021-04-09T18:21:45.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.535 7fbce2ffd700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:45.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.535 7fbce2ffd700 1 -- 172.21.15.32:0/3132994550 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fbcd4032720 con 0x7fbcec09c9b0 2021-04-09T18:21:45.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.535 7fbcea61c700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:45.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.536 7fbcea61c700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001ae10 crc :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:45.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.555 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcec10d480 con 0x7fbcec09c9b0 2021-04-09T18:21:45.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.559 7fbce2ffd700 1 -- 172.21.15.32:0/3132994550 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbcd40009c0 con 0x7fbcec09c9b0 2021-04-09T18:21:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.772 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbcec10d480 con 0x7fbcec09c9b0 2021-04-09T18:21:45.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.775 7fbce2ffd700 1 -- 172.21.15.32:0/3132994550 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbcd4002030 con 0x7fbcec09c9b0 2021-04-09T18:21:45.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 msgr2=0x7fbcd001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001ae10 crc :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec09c9b0 msgr2=0x7fbcec09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:45.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec09c9b0 0x7fbcec09db30 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7fbcd40096a0 tx=0x7fbcd400b990).stop 2021-04-09T18:21:45.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 shutdown_connections 2021-04-09T18:21:45.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001ae10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcec09c9b0 0x7fbcec09db30 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcec090390 0x7fbcec090790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 --2- 172.21.15.32:0/3132994550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcec09c5b0 0x7fbcec09d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:45.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 >> 172.21.15.32:0/3132994550 conn(0x7fbcec0b2fa0 msgr2=0x7fbcec0a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:45.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 shutdown_connections 2021-04-09T18:21:45.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:45.781 7fbcf099b700 1 -- 172.21.15.32:0/3132994550 wait complete. 2021-04-09T18:21:45.843 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:21:45.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:21:45.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:21:45.963 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:21:45.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:21:45.964 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:21:45.964 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:21:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:21:46.009 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:21:46.010 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:21:46.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:21:46.011 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:21:46.011 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:21:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:21:46.054 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:21:46.055 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:21:46.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:21:46.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.157 7f1995d8f700 1 Processor -- start 2021-04-09T18:21:46.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.157 7f1995d8f700 1 -- start start 2021-04-09T18:21:46.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.157 7f1995d8f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19900cffa0 0x7f19900d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:46.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.157 7f1995d8f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19900d6c80 0x7f199010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:46.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.157 7f1995d8f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19900d7280 con 0x7f19900d7800 2021-04-09T18:21:46.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.157 7f1995d8f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19900d7080 con 0x7f19900d6c80 2021-04-09T18:21:46.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.157 7f1995d8f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19900d91e0 con 0x7f19900cffa0 2021-04-09T18:21:46.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19900d6c80 0x7f199010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:46.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198effd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f19900d7800 0x7f19900d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58100/0 (socket says 172.21.15.32:58100) 2021-04-09T18:21:46.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198effd700 1 -- 172.21.15.32:0/2639111271 learned_addr learned my addr 172.21.15.32:0/2639111271 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:46.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198f7fe700 1 --2- 172.21.15.32:0/2639111271 >> v2:172.21.15.86:3300/0 conn(0x7f19900cffa0 0x7f19900d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:46.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198ffff700 1 -- 172.21.15.32:0/2639111271 >> v2:172.21.15.86:3300/0 conn(0x7f19900cffa0 msgr2=0x7f19900d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:46.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198ffff700 1 --2- 172.21.15.32:0/2639111271 >> v2:172.21.15.86:3300/0 conn(0x7f19900cffa0 0x7f19900d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198ffff700 1 -- 172.21.15.32:0/2639111271 >> v1:172.21.15.32:6789/0 conn(0x7f19900d7800 legacy=0x7f19900d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:46.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198e7fc700 1 -- 172.21.15.32:0/2639111271 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152129952 0 0) 0x7f19840009b0 con 0x7f19900d7800 2021-04-09T18:21:46.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198ffff700 1 -- 172.21.15.32:0/2639111271 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19900d83f0 con 0x7f19900d6c80 2021-04-09T18:21:46.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198ffff700 1 --2- 172.21.15.32:0/2639111271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19900d6c80 0x7f199010cfd0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f1988006e30 tx=0x7f1988009de0).ready entity=mon.0 client_cookie=94ff4ce9b62a5e05 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:46.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198e7fc700 1 -- 172.21.15.32:0/2639111271 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198800b300 con 0x7f19900d6c80 2021-04-09T18:21:46.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198e7fc700 1 -- 172.21.15.32:0/2639111271 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198800b680 con 0x7f19900d6c80 2021-04-09T18:21:46.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f198e7fc700 1 -- 172.21.15.32:0/2639111271 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198800bb60 con 0x7f19900d6c80 2021-04-09T18:21:46.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f1995d8f700 1 -- 172.21.15.32:0/2639111271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19900d6c80 msgr2=0x7f199010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.158 7f1995d8f700 1 --2- 172.21.15.32:0/2639111271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19900d6c80 0x7f199010cfd0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f1988006e30 tx=0x7f1988009de0).stop 2021-04-09T18:21:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.159 7f1995d8f700 1 -- 172.21.15.32:0/2639111271 shutdown_connections 2021-04-09T18:21:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.159 7f1995d8f700 1 --2- 172.21.15.32:0/2639111271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19900d6c80 0x7f199010cfd0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.159 7f1995d8f700 1 --2- 172.21.15.32:0/2639111271 >> v2:172.21.15.86:3300/0 conn(0x7f19900cffa0 0x7f19900d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.159 7f1995d8f700 1 -- 172.21.15.32:0/2639111271 >> 172.21.15.32:0/2639111271 conn(0x7f19900ce5c0 msgr2=0x7f19900cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:46.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.159 7f1995d8f700 1 -- 172.21.15.32:0/2639111271 shutdown_connections 2021-04-09T18:21:46.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.159 7f1995d8f700 1 -- 172.21.15.32:0/2639111271 wait complete. 2021-04-09T18:21:46.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.179 7f1995d8f700 1 Processor -- start 2021-04-09T18:21:46.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.179 7f1995d8f700 1 -- start start 2021-04-09T18:21:46.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f1995d8f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f199018e990 0x7f199018ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:46.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f1995d8f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f199008bd20 0x7f199008c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:46.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f1995d8f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f199008a650 0x7f199008aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:46.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f1995d8f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199008c740 con 0x7f199008bd20 2021-04-09T18:21:46.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f1995d8f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199008b070 con 0x7f199018e990 2021-04-09T18:21:46.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f1995d8f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199008b290 con 0x7f199008a650 2021-04-09T18:21:46.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f199008bd20 0x7f199008c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:46.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f199008bd20 0x7f199008c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48512/0 (socket says 172.21.15.32:48512) 2021-04-09T18:21:46.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198effd700 1 -- 172.21.15.32:0/177124679 learned_addr learned my addr 172.21.15.32:0/177124679 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:46.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198f7fe700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f199018e990 0x7f199018ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:46.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198ffff700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f199008a650 0x7f199008aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:46.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198effd700 1 -- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f199008a650 msgr2=0x7f199008aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:46.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198effd700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f199008a650 0x7f199008aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198effd700 1 -- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f199018e990 msgr2=0x7f199018ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:46.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198effd700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f199018e990 0x7f199018ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198effd700 1 -- 172.21.15.32:0/177124679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199008b4b0 con 0x7f199008bd20 2021-04-09T18:21:46.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.180 7f198f7fe700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f199018e990 0x7f199018ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:46.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.181 7f198effd700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f199008bd20 0x7f199008c120 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f1984001c60 tx=0x7f198400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:46.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.181 7f1977fff700 1 -- 172.21.15.32:0/177124679 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198400acc0 con 0x7f199008bd20 2021-04-09T18:21:46.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.181 7f1977fff700 1 -- 172.21.15.32:0/177124679 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1984004140 con 0x7f199008bd20 2021-04-09T18:21:46.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.181 7f1995d8f700 1 -- 172.21.15.32:0/177124679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199008b4b0 con 0x7f199008bd20 2021-04-09T18:21:46.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.181 7f1977fff700 1 -- 172.21.15.32:0/177124679 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1984004620 con 0x7f199008bd20 2021-04-09T18:21:46.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.181 7f1995d8f700 1 -- 172.21.15.32:0/177124679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199008b4b0 con 0x7f199008bd20 2021-04-09T18:21:46.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.181 7f1977fff700 1 -- 172.21.15.32:0/177124679 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1984000dd0 con 0x7f199008bd20 2021-04-09T18:21:46.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.182 7f1977fff700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1970018930 0x7f197001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:46.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.182 7f1977fff700 1 -- 172.21.15.32:0/177124679 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f19840328a0 con 0x7f199008bd20 2021-04-09T18:21:46.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.182 7f198f7fe700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1970018930 0x7f197001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:46.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.182 7f198f7fe700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1970018930 0x7f197001ae10 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:46.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.201 7f1995d8f700 1 -- 172.21.15.32:0/177124679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19901b70c0 con 0x7f199008bd20 2021-04-09T18:21:46.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.205 7f1977fff700 1 -- 172.21.15.32:0/177124679 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f19840009b0 con 0x7f199008bd20 2021-04-09T18:21:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.393 7f1995d8f700 1 -- 172.21.15.32:0/177124679 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f19900d7280 con 0x7f199008bd20 2021-04-09T18:21:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.394 7f1977fff700 1 -- 172.21.15.32:0/177124679 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v87) v1 ==== 81+0+25541 (secure 0 0 0) 0x7f1984000dd0 con 0x7f199008bd20 2021-04-09T18:21:46.447 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:46.447 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:21:46.447 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 87, 2021-04-09T18:21:46.448 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:21:46.448 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:21:46.455 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:21:43.291959", 2021-04-09T18:21:46.456 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:21:46.456 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:21:46.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:21:46.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:21:46.458 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:21:46.458 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:21:46.459 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:21:46.459 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:21:46.460 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:21:46.460 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:21:46.461 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:21:46.461 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:21:46.462 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:21:46.462 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:21:46.462 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:21:46.463 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 15, 2021-04-09T18:21:46.463 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:21:46.464 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:21:46.464 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:21:46.465 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:21:46.465 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:21:46.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.466 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:21:46.467 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:21:46.467 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:21:46.467 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:21:46.468 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:21:46.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:21:46.469 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:21:46.469 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:21:46.469 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:21:46.470 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:21:46.470 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:21:46.471 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:21:46.471 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:21:46.471 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:21:46.472 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:21:46.472 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:21:46.473 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:21:46.473 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:21:46.473 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:21:46.474 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:21:46.474 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:21:46.474 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:21:46.475 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:21:46.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.476 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:21:46.476 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:21:46.476 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:21:46.477 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:21:46.477 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:21:46.478 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:21:46.478 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:21:46.479 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:21:46.479 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:21:46.480 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:21:46.480 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:21:46.481 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:21:46.481 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:21:46.482 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:21:46.482 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:21:46.482 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:21:46.483 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:21:46.483 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:21:46.484 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:21:46.484 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:21:46.484 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:21:46.485 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:21:46.485 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:21:46.486 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:21:46.486 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:21:46.487 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:21:46.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:21:46.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.488 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:21:46.488 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:21:46.489 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:21:46.489 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:21:46.490 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:21:46.490 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:21:46.490 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:21:46.491 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:21:46.491 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:21:46.492 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:21:46.492 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:21:46.493 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:21:46.493 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:21:46.493 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:21:46.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.495 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:21:46.495 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:21:46.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.496 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:21:46.496 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:21:46.497 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:46.497 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:46.498 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:46.498 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:46.499 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:46.499 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:46.500 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:46.500 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:46.500 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:46.501 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:46.501 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:46.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.513 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.513 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:21:46.513 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:46.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.515 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.516 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:21:46.516 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:46.516 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.517 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.518 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:46.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:21:46.520 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:46.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.522 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:21:46.523 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:46.523 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.524 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.525 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:46.525 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.526 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:21:46.527 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:46.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.528 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.529 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:21:46.529 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:46.529 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.530 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.531 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:46.532 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:21:46.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:46.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:21:46.536 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:21:46.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.537 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.539 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:21:46.540 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:21:46.541 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:21:46.541 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:21:46.542 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:46.542 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:46.543 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:46.543 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.547 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:21:46.547 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:21:46.549 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:46.549 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:46.550 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:46.551 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:46.551 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:46.553 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:46.553 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:46.554 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:46.554 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:46.555 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:46.555 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:46.555 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.556 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:21:46.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:46.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.558 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.559 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:21:46.559 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:46.560 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.560 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.561 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:46.561 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:21:46.563 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:46.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.564 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.564 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:21:46.565 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:46.565 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.565 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.566 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:46.566 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.567 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:21:46.568 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:46.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.569 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:21:46.570 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:46.570 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.570 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.571 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:46.572 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.573 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:21:46.573 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:46.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.575 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:21:46.575 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:21:46.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.576 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.577 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:21:46.577 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:21:46.578 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:21:46.578 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:21:46.578 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:46.579 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:46.579 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:46.580 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.581 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:21:46.581 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:21:46.582 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:46.582 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:46.583 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:46.583 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:46.584 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:46.584 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:46.585 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:46.585 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:46.585 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:46.586 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:46.586 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:46.587 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.587 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.588 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:21:46.591 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:46.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.592 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.593 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:21:46.593 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:46.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.594 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.595 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:46.608 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.608 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.608 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.609 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:21:46.609 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:46.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.612 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:21:46.612 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:46.613 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.613 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.614 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:46.614 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.615 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:21:46.616 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:46.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.617 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:21:46.618 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:46.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.618 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.619 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:46.619 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.620 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:21:46.621 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:46.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.622 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:21:46.623 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:21:46.623 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.623 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.624 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:21:46.625 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:21:46.625 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:21:46.626 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:21:46.626 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:46.626 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:46.627 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:46.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.640 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:21:46.640 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:21:46.641 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:46.641 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:46.641 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:46.642 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:46.643 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:46.644 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:46.644 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:46.644 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 79, 2021-04-09T18:21:46.645 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:46.645 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:46.646 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:46.646 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:21:46.647 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:46.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.649 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.649 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:21:46.649 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:46.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.653 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:46.653 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.654 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:21:46.655 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:46.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.657 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:21:46.657 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:46.661 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.661 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.662 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:46.662 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.664 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:21:46.664 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:46.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.665 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.666 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:21:46.666 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:46.667 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.667 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.668 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:46.668 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.669 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:21:46.670 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:46.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.671 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.672 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:21:46.672 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:21:46.672 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.673 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.674 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:21:46.675 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:21:46.675 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:21:46.676 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:21:46.676 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:46.678 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:46.678 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:46.680 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.685 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:21:46.686 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:21:46.686 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:46.686 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:46.687 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:46.687 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:46.691 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:46.692 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:46.694 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:46.695 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:46.695 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:46.700 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:46.702 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.703 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:21:46.704 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:46.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.705 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:21:46.710 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:46.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.710 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.711 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:46.712 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.714 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.715 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:21:46.717 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:46.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.719 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:21:46.721 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:46.721 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.722 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.724 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:46.724 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:21:46.730 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:46.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:21:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:46.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:46.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:21:46.737 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.738 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:21:46.739 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:21:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.741 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:21:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:21:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:21:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:21:46.742 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:21:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:21:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:46.744 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:46.745 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 79, 2021-04-09T18:21:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:46.746 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:46.747 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.747 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.747 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:21:46.748 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:46.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.749 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:21:46.749 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:46.749 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.749 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.750 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:46.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.751 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.752 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:21:46.753 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:46.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.755 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:21:46.755 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:46.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.756 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.757 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:46.757 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.758 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.758 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:21:46.759 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:46.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.759 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.760 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:21:46.764 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:46.764 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.765 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.767 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:46.767 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.776 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:21:46.778 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:46.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.779 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:21:46.779 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:21:46.780 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.780 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.781 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:21:46.781 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:21:46.783 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:21:46.784 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:21:46.799 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:46.799 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:46.800 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:46.800 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.801 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:21:46.801 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:21:46.802 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:46.802 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:46.803 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:46.803 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:46.803 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:46.804 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:46.804 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:46.805 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:21:46.805 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:46.805 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:46.806 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:46.806 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:21:46.808 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:46.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.813 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:21:46.813 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:46.814 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.814 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.814 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:46.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:21:46.816 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:46.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.817 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:21:46.817 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:46.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.818 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.818 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:46.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.820 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:21:46.821 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:46.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.822 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:21:46.827 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:46.827 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.828 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.828 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:46.829 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:21:46.832 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:46.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.833 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.833 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:21:46.834 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:21:46.834 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.835 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.837 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:21:46.837 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:21:46.838 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:21:46.838 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:21:46.838 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:46.838 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:46.839 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:46.840 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.841 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:21:46.841 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:21:46.841 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:21:46.842 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:21:46.842 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:21:46.842 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:21:46.842 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:21:46.842 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:21:46.843 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:21:46.843 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 72, 2021-04-09T18:21:46.843 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:21:46.843 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:21:46.844 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:21:46.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:21:46.845 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:46.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.846 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:21:46.846 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:46.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.847 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.848 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:21:46.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:21:46.849 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:46.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.849 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.850 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:21:46.850 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:46.851 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.851 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.852 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:21:46.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:21:46.853 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:46.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.853 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:21:46.854 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:46.854 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.854 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.855 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:21:46.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:21:46.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:21:46.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:21:46.856 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:46.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:21:46.858 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:21:46.859 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:21:46.859 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.860 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.862 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:21:46.862 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:21:46.862 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:21:46.863 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:21:46.863 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:21:46.864 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:21:46.864 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:21:46.864 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:46.865 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.865 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:21:46.866 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:21:46.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.867 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:21:46.867 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:46.868 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:46.870 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:46.871 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:46.871 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:46.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.873 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:21:46.873 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:46.874 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:46.874 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:46.875 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:46.875 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:46.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.877 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:21:46.877 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:46.878 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:46.878 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:46.879 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:46.879 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:46.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.880 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:21:46.881 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:46.881 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:46.882 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:46.882 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:46.882 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:46.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.884 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:21:46.884 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:46.885 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:46.885 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:46.886 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:46.886 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:46.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.891 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:21:46.891 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:46.892 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:46.892 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:46.893 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:46.893 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:46.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.894 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:21:46.895 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:46.895 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:46.896 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:46.896 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:46.899 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:46.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:46.901 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:21:46.901 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:21:46.901 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:21:46.902 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:21:46.902 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:21:46.903 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:21:46.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.904 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:21:46.904 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:21:46.905 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:21:46.907 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:21:46.907 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:21:46.909 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:21:46.909 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/2067624954": "2021-04-10 18:21:39.798271", 2021-04-09T18:21:46.910 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/2067624954": "2021-04-10 18:21:39.798271" 2021-04-09T18:21:46.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.911 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:21:46.911 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:21:46.912 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:21:46.912 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:21:46.913 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:21:46.913 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:21:46.914 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:21:46.914 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:21:46.915 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:46.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:46.916 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:21:46.916 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:21:46.916 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:21:46.917 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:21:46.918 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:21:46.918 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:21:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 -- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1970018930 msgr2=0x7f197001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:46.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1970018930 0x7f197001ae10 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 -- 172.21.15.32:0/177124679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f199008bd20 msgr2=0x7f199008c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f199008bd20 0x7f199008c120 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f1984001c60 tx=0x7f198400a8d0).stop 2021-04-09T18:21:46.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 -- 172.21.15.32:0/177124679 shutdown_connections 2021-04-09T18:21:46.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1970018930 0x7f197001ae10 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f199008bd20 0x7f199008c120 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f199008a650 0x7f199008aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 --2- 172.21.15.32:0/177124679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f199018e990 0x7f199018ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 -- 172.21.15.32:0/177124679 >> 172.21.15.32:0/177124679 conn(0x7f1990189b00 msgr2=0x7f19901921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 -- 172.21.15.32:0/177124679 shutdown_connections 2021-04-09T18:21:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:46.396 7f1995d8f700 1 -- 172.21.15.32:0/177124679 wait complete. 2021-04-09T18:21:46.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.027 7fc1bc2f0700 1 Processor -- start 2021-04-09T18:21:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.027 7fc1bc2f0700 1 -- start start 2021-04-09T18:21:47.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.027 7fc1bc2f0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b410d010 0x7fc1b40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.027 7fc1bc2f0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc1b40d8490 0x7fc1b410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.027 7fc1bc2f0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b410d480 con 0x7fc1b40d1da0 2021-04-09T18:21:47.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.027 7fc1bc2f0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b410e780 con 0x7fc1b40d8490 2021-04-09T18:21:47.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.027 7fc1bc2f0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40d8d30 con 0x7fc1b410d010 2021-04-09T18:21:47.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1b990b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc1b40d1da0 0x7fc1b40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58114/0 (socket says 172.21.15.32:58114) 2021-04-09T18:21:47.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1b990b700 1 -- 172.21.15.32:0/3903467213 learned_addr learned my addr 172.21.15.32:0/3903467213 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:47.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1ba90d700 1 --2- 172.21.15.32:0/3903467213 >> v2:172.21.15.86:3300/0 conn(0x7fc1b40d8490 0x7fc1b410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1ba10c700 1 --2- 172.21.15.32:0/3903467213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b410d010 0x7fc1b40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1b910a700 1 -- 172.21.15.32:0/3903467213 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26840512 0 0) 0x7fc1a4000ce0 con 0x7fc1b40d1da0 2021-04-09T18:21:47.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1b910a700 1 -- 172.21.15.32:0/3903467213 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a0002fe0 con 0x7fc1b40d1da0 2021-04-09T18:21:47.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1ba10c700 1 -- 172.21.15.32:0/3903467213 >> v2:172.21.15.86:3300/0 conn(0x7fc1b40d8490 msgr2=0x7fc1b410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:47.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1ba10c700 1 --2- 172.21.15.32:0/3903467213 >> v2:172.21.15.86:3300/0 conn(0x7fc1b40d8490 0x7fc1b410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1ba10c700 1 -- 172.21.15.32:0/3903467213 >> v1:172.21.15.32:6789/0 conn(0x7fc1b40d1da0 legacy=0x7fc1b40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1ba10c700 1 -- 172.21.15.32:0/3903467213 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b40d9220 con 0x7fc1b410d010 2021-04-09T18:21:47.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1ba90d700 1 --2- 172.21.15.32:0/3903467213 >> v2:172.21.15.86:3300/0 conn(0x7fc1b40d8490 0x7fc1b410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:47.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1ba10c700 1 --2- 172.21.15.32:0/3903467213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b410d010 0x7fc1b40d1780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc1b00069d0 tx=0x7fc1b0009e40).ready entity=mon.2 client_cookie=96553dd5134a4451 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:47.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1b910a700 1 -- 172.21.15.32:0/3903467213 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1b000b510 con 0x7fc1b410d010 2021-04-09T18:21:47.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1b910a700 1 -- 172.21.15.32:0/3903467213 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b000b890 con 0x7fc1b410d010 2021-04-09T18:21:47.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.028 7fc1b910a700 1 -- 172.21.15.32:0/3903467213 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1b000bd70 con 0x7fc1b410d010 2021-04-09T18:21:47.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.029 7fc1bc2f0700 1 -- 172.21.15.32:0/3903467213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b410d010 msgr2=0x7fc1b40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.029 7fc1bc2f0700 1 --2- 172.21.15.32:0/3903467213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b410d010 0x7fc1b40d1780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc1b00069d0 tx=0x7fc1b0009e40).stop 2021-04-09T18:21:47.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.029 7fc1bc2f0700 1 -- 172.21.15.32:0/3903467213 shutdown_connections 2021-04-09T18:21:47.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.029 7fc1bc2f0700 1 --2- 172.21.15.32:0/3903467213 >> v2:172.21.15.86:3300/0 conn(0x7fc1b40d8490 0x7fc1b410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.029 7fc1bc2f0700 1 --2- 172.21.15.32:0/3903467213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b410d010 0x7fc1b40d1780 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.029 7fc1bc2f0700 1 -- 172.21.15.32:0/3903467213 >> 172.21.15.32:0/3903467213 conn(0x7fc1b40ce5f0 msgr2=0x7fc1b40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:47.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.029 7fc1bc2f0700 1 -- 172.21.15.32:0/3903467213 shutdown_connections 2021-04-09T18:21:47.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.029 7fc1bc2f0700 1 -- 172.21.15.32:0/3903467213 wait complete. 2021-04-09T18:21:47.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.049 7fc1bc2f0700 1 Processor -- start 2021-04-09T18:21:47.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1bc2f0700 1 -- start start 2021-04-09T18:21:47.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1bc2f0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b41a22e0 0x7fc1b41a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1bc2f0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b41a26e0 0x7fc1b41a3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1bc2f0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b41a5e50 0x7fc1b41a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1bc2f0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b41a2bb0 con 0x7fc1b41a26e0 2021-04-09T18:21:47.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1bc2f0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b41a6360 con 0x7fc1b41a22e0 2021-04-09T18:21:47.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1bc2f0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b41a6540 con 0x7fc1b41a5e50 2021-04-09T18:21:47.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1b990b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b41a26e0 0x7fc1b41a3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1b990b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b41a26e0 0x7fc1b41a3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48526/0 (socket says 172.21.15.32:48526) 2021-04-09T18:21:47.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1b990b700 1 -- 172.21.15.32:0/1862676075 learned_addr learned my addr 172.21.15.32:0/1862676075 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:47.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1ba90d700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b41a5e50 0x7fc1b41a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.050 7fc1ba10c700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b41a22e0 0x7fc1b41a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1b990b700 1 -- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b41a5e50 msgr2=0x7fc1b41a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1b990b700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b41a5e50 0x7fc1b41a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1b990b700 1 -- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b41a22e0 msgr2=0x7fc1b41a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1b990b700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b41a22e0 0x7fc1b41a5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1b990b700 1 -- 172.21.15.32:0/1862676075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b41a6760 con 0x7fc1b41a26e0 2021-04-09T18:21:47.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1ba10c700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b41a22e0 0x7fc1b41a5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:47.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1b990b700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b41a26e0 0x7fc1b41a3130 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7fc1a400b530 tx=0x7fc1a400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:47.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1aaffd700 1 -- 172.21.15.32:0/1862676075 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1a4006020 con 0x7fc1b41a26e0 2021-04-09T18:21:47.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1aaffd700 1 -- 172.21.15.32:0/1862676075 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a4006360 con 0x7fc1b41a26e0 2021-04-09T18:21:47.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b41a6760 con 0x7fc1b41a26e0 2021-04-09T18:21:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1aaffd700 1 -- 172.21.15.32:0/1862676075 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1a4004620 con 0x7fc1b41a26e0 2021-04-09T18:21:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.051 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b41a6760 con 0x7fc1b41a26e0 2021-04-09T18:21:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.052 7fc1aaffd700 1 -- 172.21.15.32:0/1862676075 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc1a400a000 con 0x7fc1b41a26e0 2021-04-09T18:21:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.052 7fc1aaffd700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc19c0188d0 0x7fc19c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.052 7fc1aaffd700 1 -- 172.21.15.32:0/1862676075 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fc1a4032730 con 0x7fc1b41a26e0 2021-04-09T18:21:47.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.052 7fc1ba10c700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc19c0188d0 0x7fc19c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.052 7fc1ba10c700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc19c0188d0 0x7fc19c01add0 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:47.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.072 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b410d480 con 0x7fc1b41a26e0 2021-04-09T18:21:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.075 7fc1aaffd700 1 -- 172.21.15.32:0/1862676075 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc1a4006b90 con 0x7fc1b41a26e0 2021-04-09T18:21:47.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.265 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1b410d480 con 0x7fc1b41a26e0 2021-04-09T18:21:47.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.266 7fc1aaffd700 1 -- 172.21.15.32:0/1862676075 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+1850 (secure 0 0 0) 0x7fc1a400a000 con 0x7fc1b41a26e0 2021-04-09T18:21:47.318 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 104 2021-04-09T18:21:47.319 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:47.319 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":104,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":7482,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2395614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2395614029},{"type":"v1","addr":"172.21.15.32:6835","nonce":2395614029}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":104},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[]} 2021-04-09T18:21:47.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc19c0188d0 msgr2=0x7fc19c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc19c0188d0 0x7fc19c01add0 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b41a26e0 msgr2=0x7fc1b41a3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b41a26e0 0x7fc1b41a3130 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7fc1a400b530 tx=0x7fc1a400b8d0).stop 2021-04-09T18:21:47.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 shutdown_connections 2021-04-09T18:21:47.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc19c0188d0 0x7fc19c01add0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1b41a26e0 0x7fc1b41a3130 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1b41a5e50 0x7fc1b41a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 --2- 172.21.15.32:0/1862676075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1b41a22e0 0x7fc1b41a5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 >> 172.21.15.32:0/1862676075 conn(0x7fc1b4179fd0 msgr2=0x7fc1b417f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:47.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 shutdown_connections 2021-04-09T18:21:47.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.268 7fc1bc2f0700 1 -- 172.21.15.32:0/1862676075 wait complete. 2021-04-09T18:21:47.337 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.440 7f371e277700 1 Processor -- start 2021-04-09T18:21:47.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.440 7f371e277700 1 -- start start 2021-04-09T18:21:47.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.441 7f371e277700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37180d7ca0 0x7f37180d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.441 7f371e277700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371810d480 con 0x7f371810d010 2021-04-09T18:21:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.441 7f371e277700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37180d86c0 con 0x7f371810da00 2021-04-09T18:21:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.441 7f371e277700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37180d9220 con 0x7f37180d7ca0 2021-04-09T18:21:47.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.441 7f3717fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f371810d010 0x7f37180d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58124/0 (socket says 172.21.15.32:58124) 2021-04-09T18:21:47.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.441 7f3717fff700 1 -- 172.21.15.32:0/524028167 learned_addr learned my addr 172.21.15.32:0/524028167 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:47.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.441 7f370ffff700 1 --2- 172.21.15.32:0/524028167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37180d7ca0 0x7f37180d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f37177fe700 1 -- 172.21.15.32:0/524028167 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778306046 0 0) 0x7f3708000ce0 con 0x7f371810d010 2021-04-09T18:21:47.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f37177fe700 1 -- 172.21.15.32:0/524028167 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3704002fe0 con 0x7f371810d010 2021-04-09T18:21:47.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f37177fe700 1 -- 172.21.15.32:0/524028167 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749153676 0 0) 0x7f3710000d30 con 0x7f371810da00 2021-04-09T18:21:47.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f37177fe700 1 -- 172.21.15.32:0/524028167 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37040042c0 con 0x7f371810da00 2021-04-09T18:21:47.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f370ffff700 1 -- 172.21.15.32:0/524028167 >> v1:172.21.15.86:6789/0 conn(0x7f371810da00 legacy=0x7f371810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f370ffff700 1 -- 172.21.15.32:0/524028167 >> v1:172.21.15.32:6789/0 conn(0x7f371810d010 legacy=0x7f37180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f370ffff700 1 -- 172.21.15.32:0/524028167 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37180d09c0 con 0x7f37180d7ca0 2021-04-09T18:21:47.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f370ffff700 1 --2- 172.21.15.32:0/524028167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37180d7ca0 0x7f37180d80a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f3700006650 tx=0x7f3700009a80).ready entity=mon.2 client_cookie=320cef68164022b3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:47.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f37177fe700 1 -- 172.21.15.32:0/524028167 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37000009b0 con 0x7f37180d7ca0 2021-04-09T18:21:47.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f37177fe700 1 -- 172.21.15.32:0/524028167 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370000c0c0 con 0x7f37180d7ca0 2021-04-09T18:21:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f37177fe700 1 -- 172.21.15.32:0/524028167 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f370000c510 con 0x7f37180d7ca0 2021-04-09T18:21:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f371e277700 1 -- 172.21.15.32:0/524028167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37180d7ca0 msgr2=0x7f37180d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.442 7f371e277700 1 --2- 172.21.15.32:0/524028167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37180d7ca0 0x7f37180d80a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f3700006650 tx=0x7f3700009a80).stop 2021-04-09T18:21:47.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.443 7f371e277700 1 -- 172.21.15.32:0/524028167 shutdown_connections 2021-04-09T18:21:47.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.443 7f371e277700 1 --2- 172.21.15.32:0/524028167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37180d7ca0 0x7f37180d80a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.443 7f371e277700 1 -- 172.21.15.32:0/524028167 >> 172.21.15.32:0/524028167 conn(0x7f37180ce5f0 msgr2=0x7f37180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:47.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.443 7f371e277700 1 -- 172.21.15.32:0/524028167 shutdown_connections 2021-04-09T18:21:47.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.443 7f371e277700 1 -- 172.21.15.32:0/524028167 wait complete. 2021-04-09T18:21:47.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.463 7f371e277700 1 Processor -- start 2021-04-09T18:21:47.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.463 7f371e277700 1 -- start start 2021-04-09T18:21:47.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f371e277700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37180ab300 0x7f37181a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f371e277700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37181a8ad0 0x7f37181a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f371e277700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3718180db0 0x7f37181811b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f371e277700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37181817d0 con 0x7f37181a8ad0 2021-04-09T18:21:47.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f371e277700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37181819f0 con 0x7f37180ab300 2021-04-09T18:21:47.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f371e277700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37181a1f80 con 0x7f3718180db0 2021-04-09T18:21:47.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f370ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37181a8ad0 0x7f37181a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f370ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37181a8ad0 0x7f37181a8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48544/0 (socket says 172.21.15.32:48544) 2021-04-09T18:21:47.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f370ffff700 1 -- 172.21.15.32:0/1403700832 learned_addr learned my addr 172.21.15.32:0/1403700832 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:47.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f3717fff700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37180ab300 0x7f37181a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f371c894700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3718180db0 0x7f37181811b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f370ffff700 1 -- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3718180db0 msgr2=0x7f37181811b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f370ffff700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3718180db0 0x7f37181811b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f370ffff700 1 -- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37180ab300 msgr2=0x7f37181a12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f370ffff700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37180ab300 0x7f37181a12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f370ffff700 1 -- 172.21.15.32:0/1403700832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37181a21a0 con 0x7f37181a8ad0 2021-04-09T18:21:47.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f370ffff700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37181a8ad0 0x7f37181a8ed0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f3700009630 tx=0x7f370000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:47.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f37157fa700 1 -- 172.21.15.32:0/1403700832 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37000093a0 con 0x7f37181a8ad0 2021-04-09T18:21:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f37157fa700 1 -- 172.21.15.32:0/1403700832 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3700002030 con 0x7f37181a8ad0 2021-04-09T18:21:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f37157fa700 1 -- 172.21.15.32:0/1403700832 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3700002550 con 0x7f37181a8ad0 2021-04-09T18:21:47.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f371e277700 1 -- 172.21.15.32:0/1403700832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37181a21a0 con 0x7f37181a8ad0 2021-04-09T18:21:47.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.464 7f371e277700 1 -- 172.21.15.32:0/1403700832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37181a21a0 con 0x7f37181a8ad0 2021-04-09T18:21:47.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.465 7f37157fa700 1 -- 172.21.15.32:0/1403700832 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3700000c90 con 0x7f37181a8ad0 2021-04-09T18:21:47.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.465 7f37157fa700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f36fc0188c0 0x7f36fc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.465 7f37157fa700 1 -- 172.21.15.32:0/1403700832 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f37000324c0 con 0x7f37181a8ad0 2021-04-09T18:21:47.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.465 7f3717fff700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f36fc0188c0 0x7f36fc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.466 7f3717fff700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f36fc0188c0 0x7f36fc01adc0 crc :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.485 7f371e277700 1 -- 172.21.15.32:0/1403700832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37181b7070 con 0x7f37181a8ad0 2021-04-09T18:21:47.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.489 7f37157fa700 1 -- 172.21.15.32:0/1403700832 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f370002cf70 con 0x7f37181a8ad0 2021-04-09T18:21:47.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.680 7f371e277700 1 -- 172.21.15.32:0/1403700832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f371810d480 con 0x7f37181a8ad0 2021-04-09T18:21:47.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.681 7f37157fa700 1 -- 172.21.15.32:0/1403700832 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+1850 (secure 0 0 0) 0x7f370002cf70 con 0x7f37181a8ad0 2021-04-09T18:21:47.733 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 104 2021-04-09T18:21:47.734 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:47.734 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":104,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/34806106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":34806106},{"type":"v1","addr":"172.21.15.32:6834","nonce":34806106}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":101},{"gid":7482,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2395614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2395614029},{"type":"v1","addr":"172.21.15.32:6835","nonce":2395614029}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":104},{"gid":55799,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2928911362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2928911362},{"type":"v1","addr":"172.21.15.86:6836","nonce":2928911362}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87},{"gid":55802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3460852521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3460852521},{"type":"v1","addr":"172.21.15.86:6837","nonce":3460852521}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[]} 2021-04-09T18:21:47.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 -- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f36fc0188c0 msgr2=0x7f36fc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f36fc0188c0 0x7f36fc01adc0 crc :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 -- 172.21.15.32:0/1403700832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37181a8ad0 msgr2=0x7f37181a8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37181a8ad0 0x7f37181a8ed0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f3700009630 tx=0x7f370000b970).stop 2021-04-09T18:21:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 -- 172.21.15.32:0/1403700832 shutdown_connections 2021-04-09T18:21:47.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f36fc0188c0 0x7f36fc01adc0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37181a8ad0 0x7f37181a8ed0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3718180db0 0x7f37181811b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 --2- 172.21.15.32:0/1403700832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37180ab300 0x7f37181a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 -- 172.21.15.32:0/1403700832 >> 172.21.15.32:0/1403700832 conn(0x7f37181aa2b0 msgr2=0x7f37181a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:47.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 -- 172.21.15.32:0/1403700832 shutdown_connections 2021-04-09T18:21:47.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.683 7f371e277700 1 -- 172.21.15.32:0/1403700832 wait complete. 2021-04-09T18:21:47.746 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:21:47.747 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:21:47.747 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:21:47.747 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:21:47.748 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:21:47.748 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:21:47.748 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:21:47.748 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:21:47.749 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:21:47.749 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:21:47.750 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:21:47.750 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:21:47.751 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:21:47.751 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:21:47.751 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:21:47.752 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:21:47.756 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:21:47.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:21:47.759 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:21:47.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:21:47.760 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:21:47.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:21:47.762 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:21:47.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:21:47.798 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:21:47.798 DEBUG:teuthology.parallel:result is None 2021-04-09T18:21:47.800 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:21:47.805 DEBUG:teuthology.parallel:result is None 2021-04-09T18:21:47.807 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:21:47.807 DEBUG:teuthology.parallel:result is None 2021-04-09T18:21:47.809 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:21:47.809 DEBUG:teuthology.parallel:result is None 2021-04-09T18:21:47.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:21:47.942 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:21:47.943 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:21:47.949 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:21:47.950 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:21:47.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.915 7f58d5f8d700 1 Processor -- start 2021-04-09T18:21:47.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.915 7f58d5f8d700 1 -- start start 2021-04-09T18:21:47.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.915 7f58d5f8d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f58d00d6c90 0x7f58d00d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.915 7f58d5f8d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d00d2450 0x7f58d010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:47.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.915 7f58d5f8d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d00d7290 con 0x7f58d00cffb0 2021-04-09T18:21:47.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.915 7f58d5f8d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d00d7090 con 0x7f58d00d6c90 2021-04-09T18:21:47.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.915 7f58d5f8d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00d91f0 con 0x7f58d00d2450 2021-04-09T18:21:47.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.915 7f58cf7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f58d00cffb0 0x7f58d00d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58162/0 (socket says 172.21.15.32:58162) 2021-04-09T18:21:47.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.915 7f58cf7fe700 1 -- 172.21.15.32:0/627478842 learned_addr learned my addr 172.21.15.32:0/627478842 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:47.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58cffff700 1 --2- 172.21.15.32:0/627478842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d00d2450 0x7f58d010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58ceffd700 1 --2- 172.21.15.32:0/627478842 >> v2:172.21.15.86:3300/0 conn(0x7f58d00d6c90 0x7f58d00d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:47.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58ce7fc700 1 -- 172.21.15.32:0/627478842 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872289817 0 0) 0x7f58c0000ce0 con 0x7f58d00cffb0 2021-04-09T18:21:47.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58ce7fc700 1 -- 172.21.15.32:0/627478842 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b8002fe0 con 0x7f58d00cffb0 2021-04-09T18:21:47.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58cffff700 1 -- 172.21.15.32:0/627478842 >> v2:172.21.15.86:3300/0 conn(0x7f58d00d6c90 msgr2=0x7f58d00d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:47.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58cffff700 1 --2- 172.21.15.32:0/627478842 >> v2:172.21.15.86:3300/0 conn(0x7f58d00d6c90 0x7f58d00d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58cffff700 1 -- 172.21.15.32:0/627478842 >> v1:172.21.15.32:6789/0 conn(0x7f58d00cffb0 legacy=0x7f58d00d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58ce7fc700 1 -- 172.21.15.32:0/627478842 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1580636088 0 0) 0x7f58c0000ed0 con 0x7f58d00cffb0 2021-04-09T18:21:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58cffff700 1 -- 172.21.15.32:0/627478842 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d00d2850 con 0x7f58d00d2450 2021-04-09T18:21:47.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58cffff700 1 --2- 172.21.15.32:0/627478842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d00d2450 0x7f58d010cfe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f58c8006ad0 tx=0x7f58c8009a50).ready entity=mon.2 client_cookie=1c6cb5e65881e6f0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:47.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58ce7fc700 1 -- 172.21.15.32:0/627478842 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c80066a0 con 0x7f58d00d2450 2021-04-09T18:21:47.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58ce7fc700 1 -- 172.21.15.32:0/627478842 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c8006410 con 0x7f58d00d2450 2021-04-09T18:21:47.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58ce7fc700 1 -- 172.21.15.32:0/627478842 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c80024d0 con 0x7f58d00d2450 2021-04-09T18:21:47.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58d5f8d700 1 -- 172.21.15.32:0/627478842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d00d2450 msgr2=0x7f58d010cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:47.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.916 7f58d5f8d700 1 --2- 172.21.15.32:0/627478842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d00d2450 0x7f58d010cfe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f58c8006ad0 tx=0x7f58c8009a50).stop 2021-04-09T18:21:47.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.917 7f58d5f8d700 1 -- 172.21.15.32:0/627478842 shutdown_connections 2021-04-09T18:21:47.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.917 7f58d5f8d700 1 --2- 172.21.15.32:0/627478842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d00d2450 0x7f58d010cfe0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.917 7f58d5f8d700 1 --2- 172.21.15.32:0/627478842 >> v2:172.21.15.86:3300/0 conn(0x7f58d00d6c90 0x7f58d00d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:47.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.917 7f58d5f8d700 1 -- 172.21.15.32:0/627478842 >> 172.21.15.32:0/627478842 conn(0x7f58d00ce5d0 msgr2=0x7f58d00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:47.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.917 7f58d5f8d700 1 -- 172.21.15.32:0/627478842 shutdown_connections 2021-04-09T18:21:47.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.917 7f58d5f8d700 1 -- 172.21.15.32:0/627478842 wait complete. 2021-04-09T18:21:47.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.937 7f58d5f8d700 1 Processor -- start 2021-04-09T18:21:47.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58d5f8d700 1 -- start start 2021-04-09T18:21:48.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58d5f8d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d008d210 0x7f58d008fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58d5f8d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58d00903f0 0x7f58d01a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58d5f8d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d01a6080 0x7f58d01a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58d5f8d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d008d690 con 0x7f58d01a6080 2021-04-09T18:21:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58d5f8d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00b0fd0 con 0x7f58d00903f0 2021-04-09T18:21:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58d5f8d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d01a6480 con 0x7f58d008d210 2021-04-09T18:21:48.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d01a6080 0x7f58d01a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d01a6080 0x7f58d01a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48580/0 (socket says 172.21.15.32:48580) 2021-04-09T18:21:48.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58cf7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d008d210 0x7f58d008fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58cffff700 1 -- 172.21.15.32:0/1674024608 learned_addr learned my addr 172.21.15.32:0/1674024608 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:48.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.938 7f58ceffd700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58d00903f0 0x7f58d01a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58cffff700 1 -- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d008d210 msgr2=0x7f58d008fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:48.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58cffff700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d008d210 0x7f58d008fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58cffff700 1 -- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58d00903f0 msgr2=0x7f58d01a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:48.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58cffff700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58d00903f0 0x7f58d01a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58cffff700 1 -- 172.21.15.32:0/1674024608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d008d870 con 0x7f58d01a6080 2021-04-09T18:21:48.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58ceffd700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58d00903f0 0x7f58d01a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58cf7fe700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d008d210 0x7f58d008fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58cffff700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d01a6080 0x7f58d01a46b0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f58c800b690 tx=0x7f58c800ba30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58b7fff700 1 -- 172.21.15.32:0/1674024608 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c8006cd0 con 0x7f58d01a6080 2021-04-09T18:21:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58b7fff700 1 -- 172.21.15.32:0/1674024608 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c8000ab0 con 0x7f58d01a6080 2021-04-09T18:21:48.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d008d870 con 0x7f58d01a6080 2021-04-09T18:21:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58b7fff700 1 -- 172.21.15.32:0/1674024608 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c8002330 con 0x7f58d01a6080 2021-04-09T18:21:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.939 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d008d870 con 0x7f58d01a6080 2021-04-09T18:21:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.940 7f58b7fff700 1 -- 172.21.15.32:0/1674024608 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f58c8005330 con 0x7f58d01a6080 2021-04-09T18:21:48.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.940 7f58b7fff700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58b0018930 0x7f58b001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.940 7f58b7fff700 1 -- 172.21.15.32:0/1674024608 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f58c8032700 con 0x7f58d01a6080 2021-04-09T18:21:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.940 7f58cf7fe700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58b0018930 0x7f58b001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.940 7f58cf7fe700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58b0018930 0x7f58b001ae10 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:48.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.959 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d01a52b0 con 0x7f58d01a6080 2021-04-09T18:21:48.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:47.962 7f58b7fff700 1 -- 172.21.15.32:0/1674024608 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f58c8006560 con 0x7f58d01a6080 2021-04-09T18:21:48.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.165 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f58d01a52b0 con 0x7f58d01a6080 2021-04-09T18:21:48.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.308 7f58b7fff700 1 -- 172.21.15.32:0/1674024608 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v88) v1 ==== 97+0+0 (secure 0 0 0) 0x7f58c8000a70 con 0x7f58d01a6080 2021-04-09T18:21:48.379 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:21:48.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.330 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58b0018930 msgr2=0x7f58b001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.330 7f58d5f8d700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58b0018930 0x7f58b001ae10 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d01a6080 msgr2=0x7f58d01a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:48.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d01a6080 0x7f58d01a46b0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f58c800b690 tx=0x7f58c800ba30).stop 2021-04-09T18:21:48.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 shutdown_connections 2021-04-09T18:21:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58b0018930 0x7f58b001ae10 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58d01a6080 0x7f58d01a46b0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58d00903f0 0x7f58d01a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 --2- 172.21.15.32:0/1674024608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58d008d210 0x7f58d008fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 >> 172.21.15.32:0/1674024608 conn(0x7f58d01a8f50 msgr2=0x7f58d0188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:48.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 shutdown_connections 2021-04-09T18:21:48.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.331 7f58d5f8d700 1 -- 172.21.15.32:0/1674024608 wait complete. 2021-04-09T18:21:48.392 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:21:48.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:21:48.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.490 7fc48e78a700 1 Processor -- start 2021-04-09T18:21:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.491 7fc48e78a700 1 -- start start 2021-04-09T18:21:48.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.491 7fc48e78a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc4880cffa0 0x7fc4880d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.491 7fc48e78a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc4880d7800 0x7fc4880d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.491 7fc48e78a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4880d7280 con 0x7fc4880d8220 2021-04-09T18:21:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.491 7fc48e78a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4880d91e0 con 0x7fc4880cffa0 2021-04-09T18:21:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.491 7fc48e78a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48810dfe0 con 0x7fc4880d7800 2021-04-09T18:21:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.491 7fc487fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc4880cffa0 0x7fc4880d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.491 7fc487fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc4880cffa0 0x7fc4880d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48586/0 (socket says 172.21.15.32:48586) 2021-04-09T18:21:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.491 7fc487fff700 1 -- 172.21.15.32:0/3122524371 learned_addr learned my addr 172.21.15.32:0/3122524371 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:48.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc47ffff700 1 --2- 172.21.15.32:0/3122524371 >> v2:172.21.15.86:3300/0 conn(0x7fc4880d7800 0x7fc4880d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc487fff700 1 -- 172.21.15.32:0/3122524371 >> v2:172.21.15.86:3300/0 conn(0x7fc4880d7800 msgr2=0x7fc4880d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc487fff700 1 --2- 172.21.15.32:0/3122524371 >> v2:172.21.15.86:3300/0 conn(0x7fc4880d7800 0x7fc4880d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc487fff700 1 -- 172.21.15.32:0/3122524371 >> v1:172.21.15.86:6789/0 conn(0x7fc4880d8220 legacy=0x7fc4880d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:48.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc487fff700 1 -- 172.21.15.32:0/3122524371 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48810e200 con 0x7fc4880cffa0 2021-04-09T18:21:48.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc487fff700 1 --2- 172.21.15.32:0/3122524371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880cffa0 0x7fc4880d2440 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7fc478006700 tx=0x7fc478009b80).ready entity=mon.0 client_cookie=2e08b4a242f3cf58 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:48.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc4877fe700 1 -- 172.21.15.32:0/3122524371 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc47800c0c0 con 0x7fc4880cffa0 2021-04-09T18:21:48.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc4877fe700 1 -- 172.21.15.32:0/3122524371 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47800c440 con 0x7fc4880cffa0 2021-04-09T18:21:48.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc4877fe700 1 -- 172.21.15.32:0/3122524371 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc47800c920 con 0x7fc4880cffa0 2021-04-09T18:21:48.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc48e78a700 1 -- 172.21.15.32:0/3122524371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880cffa0 msgr2=0x7fc4880d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:48.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.492 7fc48e78a700 1 --2- 172.21.15.32:0/3122524371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880cffa0 0x7fc4880d2440 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7fc478006700 tx=0x7fc478009b80).stop 2021-04-09T18:21:48.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.493 7fc48e78a700 1 -- 172.21.15.32:0/3122524371 shutdown_connections 2021-04-09T18:21:48.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.493 7fc48e78a700 1 --2- 172.21.15.32:0/3122524371 >> v2:172.21.15.86:3300/0 conn(0x7fc4880d7800 0x7fc4880d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.493 7fc48e78a700 1 --2- 172.21.15.32:0/3122524371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880cffa0 0x7fc4880d2440 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.493 7fc48e78a700 1 -- 172.21.15.32:0/3122524371 >> 172.21.15.32:0/3122524371 conn(0x7fc4880ce5c0 msgr2=0x7fc4880cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:48.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.493 7fc48e78a700 1 -- 172.21.15.32:0/3122524371 shutdown_connections 2021-04-09T18:21:48.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.493 7fc48e78a700 1 -- 172.21.15.32:0/3122524371 wait complete. 2021-04-09T18:21:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.513 7fc48e78a700 1 Processor -- start 2021-04-09T18:21:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.513 7fc48e78a700 1 -- start start 2021-04-09T18:21:48.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48e78a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4881a0de0 0x7fc4881a3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48e78a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4881a40f0 0x7fc4880a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48e78a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880a91e0 0x7fc48808b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48e78a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48808bf90 con 0x7fc4880a91e0 2021-04-09T18:21:48.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48e78a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48808c1b0 con 0x7fc4881a0de0 2021-04-09T18:21:48.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48e78a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48808c3d0 con 0x7fc4881a40f0 2021-04-09T18:21:48.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48cda7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880a91e0 0x7fc48808b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48cda7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880a91e0 0x7fc48808b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48594/0 (socket says 172.21.15.32:48594) 2021-04-09T18:21:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48cda7700 1 -- 172.21.15.32:0/2886292330 learned_addr learned my addr 172.21.15.32:0/2886292330 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:48.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc47ffff700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4881a40f0 0x7fc4880a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc487fff700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4881a0de0 0x7fc4881a3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48cda7700 1 -- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4881a40f0 msgr2=0x7fc4880a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:48.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48cda7700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4881a40f0 0x7fc4880a8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48cda7700 1 -- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4881a0de0 msgr2=0x7fc4881a3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:48.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48cda7700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4881a0de0 0x7fc4881a3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48cda7700 1 -- 172.21.15.32:0/2886292330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48808a650 con 0x7fc4880a91e0 2021-04-09T18:21:48.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc48cda7700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880a91e0 0x7fc48808b970 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7fc480009940 tx=0x7fc48000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:48.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc4857fa700 1 -- 172.21.15.32:0/2886292330 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc480000f90 con 0x7fc4880a91e0 2021-04-09T18:21:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.514 7fc4857fa700 1 -- 172.21.15.32:0/2886292330 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48000c2f0 con 0x7fc4880a91e0 2021-04-09T18:21:48.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.515 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48808a650 con 0x7fc4880a91e0 2021-04-09T18:21:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.515 7fc4857fa700 1 -- 172.21.15.32:0/2886292330 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc48000c7d0 con 0x7fc4880a91e0 2021-04-09T18:21:48.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.515 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48808a650 con 0x7fc4880a91e0 2021-04-09T18:21:48.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.515 7fc4857fa700 1 -- 172.21.15.32:0/2886292330 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc480001220 con 0x7fc4880a91e0 2021-04-09T18:21:48.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.515 7fc4857fa700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc46c0188c0 0x7fc46c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:48.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.515 7fc4857fa700 1 -- 172.21.15.32:0/2886292330 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fc480032a50 con 0x7fc4880a91e0 2021-04-09T18:21:48.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.515 7fc487fff700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc46c0188c0 0x7fc46c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:48.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.516 7fc487fff700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc46c0188c0 0x7fc46c01adc0 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.535 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48808b1c0 con 0x7fc4880a91e0 2021-04-09T18:21:48.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.539 7fc4857fa700 1 -- 172.21.15.32:0/2886292330 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc48000be30 con 0x7fc4880a91e0 2021-04-09T18:21:48.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.724 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc48808b1c0 con 0x7fc4880a91e0 2021-04-09T18:21:48.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.724 7fc4857fa700 1 -- 172.21.15.32:0/2886292330 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) v1 ==== 80+0+4539 (secure 0 0 0) 0x7fc480001220 con 0x7fc4880a91e0 2021-04-09T18:21:48.776 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:48.776 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:21:48.776 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:21:48.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.777 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:21:48.778 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:21:48.778 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:21:48.779 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.780 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.780 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.781 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.783 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:21:48.783 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:21:48.783 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:21:48.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.784 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.785 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.786 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:21:48.786 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:21:48.786 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:21:48.786 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.787 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.787 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.787 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.788 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:21:48.789 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:21:48.789 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:21:48.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.790 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.790 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.790 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.791 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:21:48.791 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:21:48.792 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.794 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.794 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.796 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:21:48.796 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:21:48.796 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.797 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.797 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.798 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.798 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.800 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:21:48.801 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:21:48.801 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.802 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.802 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.806 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:21:48.809 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:21:48.809 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.810 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.810 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.812 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:21:48.813 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:21:48.813 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.814 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.815 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.818 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:21:48.818 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:21:48.819 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.819 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.820 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.821 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.822 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:21:48.823 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:21:48.823 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.824 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.824 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.824 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.826 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:21:48.826 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:21:48.826 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.827 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.827 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.827 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.829 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.829 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:21:48.831 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:21:48.831 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.832 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:21:48.832 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:21:48.832 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:21:48.833 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.834 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:21:48.835 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:21:48.835 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.836 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:21:48.836 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:21:48.837 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:21:48.838 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.839 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:21:48.840 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:21:48.840 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.841 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:21:48.841 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:21:48.842 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:21:48.842 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.842 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.843 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:21:48.843 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:21:48.844 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.844 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:21:48.844 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:21:48.844 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.845 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:21:48.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:48.845 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:21:48.845 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:21:48.846 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:21:48.846 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:21:48.846 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:21:48.847 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:21:48.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:48.848 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:21:48.848 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:21:48.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc46c0188c0 msgr2=0x7fc46c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc46c0188c0 0x7fc46c01adc0 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880a91e0 msgr2=0x7fc48808b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:48.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880a91e0 0x7fc48808b970 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7fc480009940 tx=0x7fc48000bcf0).stop 2021-04-09T18:21:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 shutdown_connections 2021-04-09T18:21:48.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc46c0188c0 0x7fc46c01adc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4880a91e0 0x7fc48808b970 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4881a40f0 0x7fc4880a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 --2- 172.21.15.32:0/2886292330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4881a0de0 0x7fc4881a3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 >> 172.21.15.32:0/2886292330 conn(0x7fc4881941b0 msgr2=0x7fc488183120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 shutdown_connections 2021-04-09T18:21:48.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.727 7fc48e78a700 1 -- 172.21.15.32:0/2886292330 wait complete. 2021-04-09T18:21:48.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:21:49.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.953 7f7ad62c1700 1 Processor -- start 2021-04-09T18:21:49.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.953 7f7ad62c1700 1 -- start start 2021-04-09T18:21:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.953 7f7ad62c1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ad00cffa0 0x7f7ad00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.953 7f7ad62c1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad00d7800 0x7f7ad00d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.953 7f7ad62c1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad00d7280 con 0x7f7ad00d8220 2021-04-09T18:21:49.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.953 7f7ad62c1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad00d91e0 con 0x7f7ad00cffa0 2021-04-09T18:21:49.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.953 7f7ad62c1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad010dfe0 con 0x7f7ad00d7800 2021-04-09T18:21:49.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ad00cffa0 0x7f7ad00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7ad00cffa0 0x7f7ad00d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48598/0 (socket says 172.21.15.32:48598) 2021-04-09T18:21:49.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acffff700 1 -- 172.21.15.32:0/2832200078 learned_addr learned my addr 172.21.15.32:0/2832200078 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7ac7fff700 1 --2- 172.21.15.32:0/2832200078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad00d7800 0x7f7ad00d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acffff700 1 -- 172.21.15.32:0/2832200078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad00d7800 msgr2=0x7f7ad00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:49.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acffff700 1 --2- 172.21.15.32:0/2832200078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad00d7800 0x7f7ad00d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acffff700 1 -- 172.21.15.32:0/2832200078 >> v1:172.21.15.86:6789/0 conn(0x7f7ad00d8220 legacy=0x7f7ad00d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acffff700 1 -- 172.21.15.32:0/2832200078 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad010e200 con 0x7f7ad00cffa0 2021-04-09T18:21:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acf7fe700 1 -- 172.21.15.32:0/2832200078 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171532503 0 0) 0x7f7ac8000d30 con 0x7f7ad00d8220 2021-04-09T18:21:49.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acffff700 1 --2- 172.21.15.32:0/2832200078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad00cffa0 0x7f7ad00d2440 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f7ac0006700 tx=0x7f7ac0009b00).ready entity=mon.0 client_cookie=852ecdad72c990cf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:49.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acf7fe700 1 -- 172.21.15.32:0/2832200078 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac000c0c0 con 0x7f7ad00cffa0 2021-04-09T18:21:49.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acf7fe700 1 -- 172.21.15.32:0/2832200078 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac000c400 con 0x7f7ad00cffa0 2021-04-09T18:21:49.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7acf7fe700 1 -- 172.21.15.32:0/2832200078 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac000c8e0 con 0x7f7ad00cffa0 2021-04-09T18:21:49.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7ad62c1700 1 -- 172.21.15.32:0/2832200078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad00cffa0 msgr2=0x7f7ad00d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.954 7f7ad62c1700 1 --2- 172.21.15.32:0/2832200078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad00cffa0 0x7f7ad00d2440 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f7ac0006700 tx=0x7f7ac0009b00).stop 2021-04-09T18:21:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.955 7f7ad62c1700 1 -- 172.21.15.32:0/2832200078 shutdown_connections 2021-04-09T18:21:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.955 7f7ad62c1700 1 --2- 172.21.15.32:0/2832200078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad00d7800 0x7f7ad00d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.955 7f7ad62c1700 1 --2- 172.21.15.32:0/2832200078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad00cffa0 0x7f7ad00d2440 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.955 7f7ad62c1700 1 -- 172.21.15.32:0/2832200078 >> 172.21.15.32:0/2832200078 conn(0x7f7ad00ce5c0 msgr2=0x7f7ad00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:49.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.955 7f7ad62c1700 1 -- 172.21.15.32:0/2832200078 shutdown_connections 2021-04-09T18:21:49.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.955 7f7ad62c1700 1 -- 172.21.15.32:0/2832200078 wait complete. 2021-04-09T18:21:49.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.975 7f7ad62c1700 1 Processor -- start 2021-04-09T18:21:49.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.975 7f7ad62c1700 1 -- start start 2021-04-09T18:21:49.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad62c1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ad01929f0 0x7f7ad0192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad62c1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad008ddc0 0x7f7ad008e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad62c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad008e7e0 0x7f7ad008c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad62c1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0193410 con 0x7f7ad008e7e0 2021-04-09T18:21:49.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad62c1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad00a9140 con 0x7f7ad01929f0 2021-04-09T18:21:49.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad62c1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad008ce80 con 0x7f7ad008ddc0 2021-04-09T18:21:49.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad48de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad008e7e0 0x7f7ad008c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad48de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad008e7e0 0x7f7ad008c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48608/0 (socket says 172.21.15.32:48608) 2021-04-09T18:21:49.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad48de700 1 -- 172.21.15.32:0/4180673060 learned_addr learned my addr 172.21.15.32:0/4180673060 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:49.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7acffff700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ad01929f0 0x7f7ad0192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ac7fff700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad008ddc0 0x7f7ad008e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad48de700 1 -- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad008ddc0 msgr2=0x7f7ad008e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad48de700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad008ddc0 0x7f7ad008e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad48de700 1 -- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ad01929f0 msgr2=0x7f7ad0192df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad48de700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ad01929f0 0x7f7ad0192df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad48de700 1 -- 172.21.15.32:0/4180673060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad008d0a0 con 0x7f7ad008e7e0 2021-04-09T18:21:49.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7ad48de700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad008e7e0 0x7f7ad008c860 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f7ac8009980 tx=0x7f7ac800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:49.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.976 7f7acd7fa700 1 -- 172.21.15.32:0/4180673060 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac8005870 con 0x7f7ad008e7e0 2021-04-09T18:21:49.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.977 7f7acd7fa700 1 -- 172.21.15.32:0/4180673060 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac8005b70 con 0x7f7ad008e7e0 2021-04-09T18:21:49.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.977 7f7acd7fa700 1 -- 172.21.15.32:0/4180673060 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac8006030 con 0x7f7ad008e7e0 2021-04-09T18:21:49.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.977 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad008d0a0 con 0x7f7ad008e7e0 2021-04-09T18:21:49.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.977 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad008d0a0 con 0x7f7ad008e7e0 2021-04-09T18:21:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.977 7f7acd7fa700 1 -- 172.21.15.32:0/4180673060 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7ac8006030 con 0x7f7ad008e7e0 2021-04-09T18:21:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.977 7f7acd7fa700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab40188c0 0x7f7ab401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.977 7f7acd7fa700 1 -- 172.21.15.32:0/4180673060 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f7ac80328c0 con 0x7f7ad008e7e0 2021-04-09T18:21:49.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.978 7f7acffff700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab40188c0 0x7f7ab401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.978 7f7acffff700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab40188c0 0x7f7ab401adc0 crc :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:49.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:48.997 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad01b70c0 con 0x7f7ad008e7e0 2021-04-09T18:21:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.001 7f7acd7fa700 1 -- 172.21.15.32:0/4180673060 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7ac800be10 con 0x7f7ad008e7e0 2021-04-09T18:21:49.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.190 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ad00d7280 con 0x7f7ad008e7e0 2021-04-09T18:21:49.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.191 7f7acd7fa700 1 -- 172.21.15.32:0/4180673060 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f7ac8005c30 con 0x7f7ad008e7e0 2021-04-09T18:21:49.243 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:49.244 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":88,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:21:48.306505","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:21:49.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab40188c0 msgr2=0x7f7ab401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab40188c0 0x7f7ab401adc0 crc :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad008e7e0 msgr2=0x7f7ad008c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad008e7e0 0x7f7ad008c860 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f7ac8009980 tx=0x7f7ac800bcd0).stop 2021-04-09T18:21:49.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 shutdown_connections 2021-04-09T18:21:49.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab40188c0 0x7f7ab401adc0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ad008e7e0 0x7f7ad008c860 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ad008ddc0 0x7f7ad008e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 --2- 172.21.15.32:0/4180673060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ad01929f0 0x7f7ad0192df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 >> 172.21.15.32:0/4180673060 conn(0x7f7ad0189b00 msgr2=0x7f7ad01921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:49.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 shutdown_connections 2021-04-09T18:21:49.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.193 7f7ad62c1700 1 -- 172.21.15.32:0/4180673060 wait complete. 2021-04-09T18:21:49.254 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:21:49.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:21:49.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.358 7fa867887700 1 Processor -- start 2021-04-09T18:21:49.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.358 7fa867887700 1 -- start start 2021-04-09T18:21:49.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa867887700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8600d6ca0 0x7fa8600d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa867887700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8600d2460 0x7fa86010cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa867887700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8600d72a0 con 0x7fa8600cffc0 2021-04-09T18:21:49.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa867887700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8600d70a0 con 0x7fa8600d2460 2021-04-09T18:21:49.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa867887700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8600d9200 con 0x7fa8600d6ca0 2021-04-09T18:21:49.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa865ea4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8600d2460 0x7fa86010cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa865ea4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa8600d2460 0x7fa86010cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48614/0 (socket says 172.21.15.32:48614) 2021-04-09T18:21:49.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa865ea4700 1 -- 172.21.15.32:0/687903468 learned_addr learned my addr 172.21.15.32:0/687903468 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:49.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa864ea2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8600d6ca0 0x7fa8600d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa865ea4700 1 -- 172.21.15.32:0/687903468 >> v2:172.21.15.86:3300/0 conn(0x7fa8600d6ca0 msgr2=0x7fa8600d8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:49.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa865ea4700 1 --2- 172.21.15.32:0/687903468 >> v2:172.21.15.86:3300/0 conn(0x7fa8600d6ca0 0x7fa8600d8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa865ea4700 1 -- 172.21.15.32:0/687903468 >> v1:172.21.15.86:6789/0 conn(0x7fa8600cffc0 legacy=0x7fa8600d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa865ea4700 1 -- 172.21.15.32:0/687903468 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8600d2860 con 0x7fa8600d2460 2021-04-09T18:21:49.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa85ffff700 1 -- 172.21.15.32:0/687903468 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776616640 0 0) 0x7fa854000fe0 con 0x7fa8600cffc0 2021-04-09T18:21:49.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa864ea2700 1 --2- 172.21.15.32:0/687903468 >> v2:172.21.15.86:3300/0 conn(0x7fa8600d6ca0 0x7fa8600d8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:49.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa865ea4700 1 --2- 172.21.15.32:0/687903468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600d2460 0x7fa86010cff0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7fa858006b80 tx=0x7fa858009b90).ready entity=mon.0 client_cookie=85b8f63c4d7a360d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:49.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa85ffff700 1 -- 172.21.15.32:0/687903468 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8580022e0 con 0x7fa8600d2460 2021-04-09T18:21:49.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa85ffff700 1 -- 172.21.15.32:0/687903468 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa858002620 con 0x7fa8600d2460 2021-04-09T18:21:49.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.359 7fa85ffff700 1 -- 172.21.15.32:0/687903468 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa858002b00 con 0x7fa8600d2460 2021-04-09T18:21:49.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.360 7fa867887700 1 -- 172.21.15.32:0/687903468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600d2460 msgr2=0x7fa86010cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.360 7fa867887700 1 --2- 172.21.15.32:0/687903468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600d2460 0x7fa86010cff0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7fa858006b80 tx=0x7fa858009b90).stop 2021-04-09T18:21:49.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.360 7fa867887700 1 -- 172.21.15.32:0/687903468 shutdown_connections 2021-04-09T18:21:49.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.360 7fa867887700 1 --2- 172.21.15.32:0/687903468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8600d2460 0x7fa86010cff0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.360 7fa867887700 1 --2- 172.21.15.32:0/687903468 >> v2:172.21.15.86:3300/0 conn(0x7fa8600d6ca0 0x7fa8600d8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.360 7fa867887700 1 -- 172.21.15.32:0/687903468 >> 172.21.15.32:0/687903468 conn(0x7fa8600ce5e0 msgr2=0x7fa8600cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:49.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.360 7fa867887700 1 -- 172.21.15.32:0/687903468 shutdown_connections 2021-04-09T18:21:49.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.360 7fa867887700 1 -- 172.21.15.32:0/687903468 wait complete. 2021-04-09T18:21:49.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.380 7fa867887700 1 Processor -- start 2021-04-09T18:21:49.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa867887700 1 -- start start 2021-04-09T18:21:49.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa867887700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa860182e60 0x7fa860183260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa867887700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86009d0f0 0x7fa86009d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa867887700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa860181330 0x7fa860181730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa867887700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86009cb70 con 0x7fa860182e60 2021-04-09T18:21:49.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa867887700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86009db10 con 0x7fa86009d0f0 2021-04-09T18:21:49.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa867887700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa860181d50 con 0x7fa860181330 2021-04-09T18:21:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa8656a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa860182e60 0x7fa860183260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa8656a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa860182e60 0x7fa860183260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48618/0 (socket says 172.21.15.32:48618) 2021-04-09T18:21:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa8656a3700 1 -- 172.21.15.32:0/2521635164 learned_addr learned my addr 172.21.15.32:0/2521635164 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:49.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa864ea2700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86009d0f0 0x7fa86009d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.381 7fa865ea4700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa860181330 0x7fa860181730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa8656a3700 1 -- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa860181330 msgr2=0x7fa860181730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa8656a3700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa860181330 0x7fa860181730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa8656a3700 1 -- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86009d0f0 msgr2=0x7fa86009d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa8656a3700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86009d0f0 0x7fa86009d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa8656a3700 1 -- 172.21.15.32:0/2521635164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa860181f70 con 0x7fa860182e60 2021-04-09T18:21:49.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa8656a3700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa860182e60 0x7fa860183260 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7fa85400db80 tx=0x7fa854009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:49.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa85dffb700 1 -- 172.21.15.32:0/2521635164 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa85400e3a0 con 0x7fa860182e60 2021-04-09T18:21:49.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa85dffb700 1 -- 172.21.15.32:0/2521635164 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85400e6a0 con 0x7fa860182e60 2021-04-09T18:21:49.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa867887700 1 -- 172.21.15.32:0/2521635164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa860181f70 con 0x7fa860182e60 2021-04-09T18:21:49.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa85dffb700 1 -- 172.21.15.32:0/2521635164 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa85400eb80 con 0x7fa860182e60 2021-04-09T18:21:49.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.382 7fa867887700 1 -- 172.21.15.32:0/2521635164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa860181f70 con 0x7fa860182e60 2021-04-09T18:21:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.383 7fa85dffb700 1 -- 172.21.15.32:0/2521635164 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa85400e880 con 0x7fa860182e60 2021-04-09T18:21:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.383 7fa85dffb700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa840018930 0x7fa84001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.383 7fa85dffb700 1 -- 172.21.15.32:0/2521635164 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fa854034c60 con 0x7fa860182e60 2021-04-09T18:21:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.383 7fa864ea2700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa840018930 0x7fa84001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.383 7fa864ea2700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa840018930 0x7fa84001ae10 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.402 7fa867887700 1 -- 172.21.15.32:0/2521635164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa860093df0 con 0x7fa860182e60 2021-04-09T18:21:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.406 7fa85dffb700 1 -- 172.21.15.32:0/2521635164 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa8540338c0 con 0x7fa860182e60 2021-04-09T18:21:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.585 7fa867887700 1 -- 172.21.15.32:0/2521635164 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fa8600d72a0 con 0x7fa860182e60 2021-04-09T18:21:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.585 7fa85dffb700 1 -- 172.21.15.32:0/2521635164 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v88) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa854021ee0 con 0x7fa860182e60 2021-04-09T18:21:49.636 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:21:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.587 7fa867887700 1 -- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa840018930 msgr2=0x7fa84001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.587 7fa867887700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa840018930 0x7fa84001ae10 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.587 7fa867887700 1 -- 172.21.15.32:0/2521635164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa860182e60 msgr2=0x7fa860183260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.587 7fa867887700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa860182e60 0x7fa860183260 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7fa85400db80 tx=0x7fa854009ee0).stop 2021-04-09T18:21:49.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.588 7fa867887700 1 -- 172.21.15.32:0/2521635164 shutdown_connections 2021-04-09T18:21:49.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.588 7fa867887700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa840018930 0x7fa84001ae10 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.588 7fa867887700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa860181330 0x7fa860181730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.588 7fa867887700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86009d0f0 0x7fa86009d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.588 7fa867887700 1 --2- 172.21.15.32:0/2521635164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa860182e60 0x7fa860183260 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.588 7fa867887700 1 -- 172.21.15.32:0/2521635164 >> 172.21.15.32:0/2521635164 conn(0x7fa86009fa00 msgr2=0x7fa8600b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:49.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.588 7fa867887700 1 -- 172.21.15.32:0/2521635164 shutdown_connections 2021-04-09T18:21:49.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.588 7fa867887700 1 -- 172.21.15.32:0/2521635164 wait complete. 2021-04-09T18:21:49.651 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:21:49.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.749 7f02cf8c7700 1 Processor -- start 2021-04-09T18:21:49.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.749 7f02cf8c7700 1 -- start start 2021-04-09T18:21:49.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.750 7f02cf8c7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c810d010 0x7f02c80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.750 7f02cf8c7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f02c80d8490 0x7f02c810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.750 7f02cf8c7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c810d480 con 0x7f02c80d1da0 2021-04-09T18:21:49.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.750 7f02cf8c7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c810e780 con 0x7f02c80d8490 2021-04-09T18:21:49.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.750 7f02cf8c7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80d8d30 con 0x7f02c810d010 2021-04-09T18:21:49.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.750 7f02ccee2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f02c80d1da0 0x7f02c80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58220/0 (socket says 172.21.15.32:58220) 2021-04-09T18:21:49.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.750 7f02ccee2700 1 -- 172.21.15.32:0/2668498105 learned_addr learned my addr 172.21.15.32:0/2668498105 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:49.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.750 7f02cd6e3700 1 --2- 172.21.15.32:0/2668498105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c810d010 0x7f02c80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.750 7f02cdee4700 1 --2- 172.21.15.32:0/2668498105 >> v2:172.21.15.86:3300/0 conn(0x7f02c80d8490 0x7f02c810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02c7fff700 1 -- 172.21.15.32:0/2668498105 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450162009 0 0) 0x7f02bc000ce0 con 0x7f02c80d1da0 2021-04-09T18:21:49.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02c7fff700 1 -- 172.21.15.32:0/2668498105 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b0002fe0 con 0x7f02c80d1da0 2021-04-09T18:21:49.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02cdee4700 1 -- 172.21.15.32:0/2668498105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c810d010 msgr2=0x7f02c80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:49.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02cdee4700 1 --2- 172.21.15.32:0/2668498105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c810d010 0x7f02c80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02cdee4700 1 -- 172.21.15.32:0/2668498105 >> v1:172.21.15.32:6789/0 conn(0x7f02c80d1da0 legacy=0x7f02c80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02cdee4700 1 -- 172.21.15.32:0/2668498105 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c80d9220 con 0x7f02c80d8490 2021-04-09T18:21:49.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02cd6e3700 1 --2- 172.21.15.32:0/2668498105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c810d010 0x7f02c80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:49.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02cdee4700 1 --2- 172.21.15.32:0/2668498105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02c80d8490 0x7f02c810da00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f02c0006e50 tx=0x7f02c0009e70).ready entity=mon.1 client_cookie=32007d95b468844d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:49.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02c7fff700 1 -- 172.21.15.32:0/2668498105 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02c0006a20 con 0x7f02c80d8490 2021-04-09T18:21:49.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02c7fff700 1 -- 172.21.15.32:0/2668498105 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c0006790 con 0x7f02c80d8490 2021-04-09T18:21:49.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02c7fff700 1 -- 172.21.15.32:0/2668498105 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02c0000f90 con 0x7f02c80d8490 2021-04-09T18:21:49.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02cf8c7700 1 -- 172.21.15.32:0/2668498105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02c80d8490 msgr2=0x7f02c810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.751 7f02cf8c7700 1 --2- 172.21.15.32:0/2668498105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02c80d8490 0x7f02c810da00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f02c0006e50 tx=0x7f02c0009e70).stop 2021-04-09T18:21:49.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.752 7f02cf8c7700 1 -- 172.21.15.32:0/2668498105 shutdown_connections 2021-04-09T18:21:49.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.752 7f02cf8c7700 1 --2- 172.21.15.32:0/2668498105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02c80d8490 0x7f02c810da00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.752 7f02cf8c7700 1 --2- 172.21.15.32:0/2668498105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c810d010 0x7f02c80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.752 7f02cf8c7700 1 -- 172.21.15.32:0/2668498105 >> 172.21.15.32:0/2668498105 conn(0x7f02c80ce5f0 msgr2=0x7f02c80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:49.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.752 7f02cf8c7700 1 -- 172.21.15.32:0/2668498105 shutdown_connections 2021-04-09T18:21:49.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.752 7f02cf8c7700 1 -- 172.21.15.32:0/2668498105 wait complete. 2021-04-09T18:21:49.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.772 7f02cf8c7700 1 Processor -- start 2021-04-09T18:21:49.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.772 7f02cf8c7700 1 -- start start 2021-04-09T18:21:49.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02cf8c7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c808f040 0x7f02c81a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02cf8c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c808f440 0x7f02c81a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02cf8c7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02c81a22e0 0x7f02c81a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02cf8c7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c81a3100 con 0x7f02c808f440 2021-04-09T18:21:49.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02cf8c7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c81a2d00 con 0x7f02c81a22e0 2021-04-09T18:21:49.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02cf8c7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c817c7e0 con 0x7f02c808f040 2021-04-09T18:21:49.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02ccee2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c808f440 0x7f02c81a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02ccee2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c808f440 0x7f02c81a3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48634/0 (socket says 172.21.15.32:48634) 2021-04-09T18:21:49.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02cd6e3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c808f040 0x7f02c81a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02ccee2700 1 -- 172.21.15.32:0/2389690381 learned_addr learned my addr 172.21.15.32:0/2389690381 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:49.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.773 7f02cdee4700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02c81a22e0 0x7f02c81a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02cd6e3700 1 -- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02c81a22e0 msgr2=0x7f02c81a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02cd6e3700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02c81a22e0 0x7f02c81a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02cd6e3700 1 -- 172.21.15.32:0/2389690381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c808f440 msgr2=0x7f02c81a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:49.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02cd6e3700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c808f440 0x7f02c81a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:49.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02cd6e3700 1 -- 172.21.15.32:0/2389690381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c80aee10 con 0x7f02c808f040 2021-04-09T18:21:49.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02ccee2700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c808f440 0x7f02c81a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:49.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02cd6e3700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c808f040 0x7f02c81a3600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f02b8009540 tx=0x7f02b800b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:49.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02c5ffb700 1 -- 172.21.15.32:0/2389690381 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02b800bcd0 con 0x7f02c808f040 2021-04-09T18:21:49.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02c5ffb700 1 -- 172.21.15.32:0/2389690381 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b8006740 con 0x7f02c808f040 2021-04-09T18:21:49.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c80aee10 con 0x7f02c808f040 2021-04-09T18:21:49.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02c5ffb700 1 -- 172.21.15.32:0/2389690381 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02b8002220 con 0x7f02c808f040 2021-04-09T18:21:49.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.774 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c80aee10 con 0x7f02c808f040 2021-04-09T18:21:49.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.775 7f02c5ffb700 1 -- 172.21.15.32:0/2389690381 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f02b8009fe0 con 0x7f02c808f040 2021-04-09T18:21:49.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.775 7f02c5ffb700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02a80188d0 0x7f02a801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:49.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.775 7f02c5ffb700 1 -- 172.21.15.32:0/2389690381 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f02b80327b0 con 0x7f02c808f040 2021-04-09T18:21:49.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.775 7f02ccee2700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02a80188d0 0x7f02a801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:49.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.775 7f02ccee2700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02a80188d0 0x7f02a801add0 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:49.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.794 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c81b7050 con 0x7f02c808f040 2021-04-09T18:21:49.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.798 7f02c5ffb700 1 -- 172.21.15.32:0/2389690381 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f02b8006740 con 0x7f02c808f040 2021-04-09T18:21:50.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.974 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f02c810d480 con 0x7f02c808f040 2021-04-09T18:21:50.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.975 7f02c5ffb700 1 -- 172.21.15.32:0/2389690381 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v88) v1 ==== 84+0+64 (secure 0 0 0) 0x7f02b8000c90 con 0x7f02c808f040 2021-04-09T18:21:50.027 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:50.028 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:21:50.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:21:50.028 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:21:50.029 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:21:50.037 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:21:50.038 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:21:50.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.977 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02a80188d0 msgr2=0x7f02a801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:50.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.977 7f02cf8c7700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02a80188d0 0x7f02a801add0 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.977 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c808f040 msgr2=0x7f02c81a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.977 7f02cf8c7700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c808f040 0x7f02c81a3600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f02b8009540 tx=0x7f02b800b8e0).stop 2021-04-09T18:21:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.978 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 shutdown_connections 2021-04-09T18:21:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.978 7f02cf8c7700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f02a80188d0 0x7f02a801add0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.978 7f02cf8c7700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c808f440 0x7f02c81a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.978 7f02cf8c7700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02c81a22e0 0x7f02c81a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.978 7f02cf8c7700 1 --2- 172.21.15.32:0/2389690381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02c808f040 0x7f02c81a3600 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.978 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 >> 172.21.15.32:0/2389690381 conn(0x7f02c8199680 msgr2=0x7f02c81a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.978 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 shutdown_connections 2021-04-09T18:21:50.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:49.978 7f02cf8c7700 1 -- 172.21.15.32:0/2389690381 wait complete. 2021-04-09T18:21:50.050 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:21:50.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:21:50.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb79dc51700 1 Processor -- start 2021-04-09T18:21:50.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb79dc51700 1 -- start start 2021-04-09T18:21:50.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb79dc51700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb79810d010 0x7fb7980d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:50.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb79dc51700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb7980d1da0 0x7fb7980d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:50.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb79dc51700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7980d82c0 0x7fb79810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:50.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb79dc51700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7980d8d30 con 0x7fb79810d010 2021-04-09T18:21:50.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb79dc51700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7980d9220 con 0x7fb7980d1da0 2021-04-09T18:21:50.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb79dc51700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79810d4b0 con 0x7fb7980d82c0 2021-04-09T18:21:50.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb7977fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb79810d010 0x7fb7980d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:50.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb7977fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb79810d010 0x7fb7980d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48640/0 (socket says 172.21.15.32:48640) 2021-04-09T18:21:50.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.152 7fb7977fe700 1 -- 172.21.15.32:0/1449156604 learned_addr learned my addr 172.21.15.32:0/1449156604 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:50.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb797fff700 1 --2- 172.21.15.32:0/1449156604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7980d82c0 0x7fb79810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:50.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb796ffd700 1 --2- 172.21.15.32:0/1449156604 >> v2:172.21.15.86:3300/0 conn(0x7fb7980d1da0 0x7fb7980d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:50.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb7977fe700 1 -- 172.21.15.32:0/1449156604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7980d82c0 msgr2=0x7fb79810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:50.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb7977fe700 1 --2- 172.21.15.32:0/1449156604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7980d82c0 0x7fb79810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb7977fe700 1 -- 172.21.15.32:0/1449156604 >> v2:172.21.15.86:3300/0 conn(0x7fb7980d1da0 msgr2=0x7fb7980d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:50.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb7977fe700 1 --2- 172.21.15.32:0/1449156604 >> v2:172.21.15.86:3300/0 conn(0x7fb7980d1da0 0x7fb7980d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb7977fe700 1 -- 172.21.15.32:0/1449156604 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7980d86c0 con 0x7fb79810d010 2021-04-09T18:21:50.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb796ffd700 1 --2- 172.21.15.32:0/1449156604 >> v2:172.21.15.86:3300/0 conn(0x7fb7980d1da0 0x7fb7980d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:50.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb7977fe700 1 --2- 172.21.15.32:0/1449156604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79810d010 0x7fb7980d1780 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7fb788006700 tx=0x7fb788009b00).ready entity=mon.0 client_cookie=c7582aa9bd546e76 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:50.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb7967fc700 1 -- 172.21.15.32:0/1449156604 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb78800c0c0 con 0x7fb79810d010 2021-04-09T18:21:50.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb7967fc700 1 -- 172.21.15.32:0/1449156604 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78800c400 con 0x7fb79810d010 2021-04-09T18:21:50.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb7967fc700 1 -- 172.21.15.32:0/1449156604 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb78800c8e0 con 0x7fb79810d010 2021-04-09T18:21:50.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb79dc51700 1 -- 172.21.15.32:0/1449156604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79810d010 msgr2=0x7fb7980d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:50.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.153 7fb79dc51700 1 --2- 172.21.15.32:0/1449156604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79810d010 0x7fb7980d1780 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7fb788006700 tx=0x7fb788009b00).stop 2021-04-09T18:21:50.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.154 7fb79dc51700 1 -- 172.21.15.32:0/1449156604 shutdown_connections 2021-04-09T18:21:50.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.154 7fb79dc51700 1 --2- 172.21.15.32:0/1449156604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7980d82c0 0x7fb79810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.154 7fb79dc51700 1 --2- 172.21.15.32:0/1449156604 >> v2:172.21.15.86:3300/0 conn(0x7fb7980d1da0 0x7fb7980d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.154 7fb79dc51700 1 --2- 172.21.15.32:0/1449156604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79810d010 0x7fb7980d1780 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.154 7fb79dc51700 1 -- 172.21.15.32:0/1449156604 >> 172.21.15.32:0/1449156604 conn(0x7fb7980ce5f0 msgr2=0x7fb7980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:50.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.154 7fb79dc51700 1 -- 172.21.15.32:0/1449156604 shutdown_connections 2021-04-09T18:21:50.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.154 7fb79dc51700 1 -- 172.21.15.32:0/1449156604 wait complete. 2021-04-09T18:21:50.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.174 7fb79dc51700 1 Processor -- start 2021-04-09T18:21:50.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.174 7fb79dc51700 1 -- start start 2021-04-09T18:21:50.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb79dc51700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7980c0ac0 0x7fb7980bfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:50.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb79dc51700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79808bfc0 0x7fb798092c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:50.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb79dc51700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb79808c3c0 0x7fb798093270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:50.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb79dc51700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980c0350 con 0x7fb79808bfc0 2021-04-09T18:21:50.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb79dc51700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7980c0570 con 0x7fb7980c0ac0 2021-04-09T18:21:50.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb79dc51700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb798093890 con 0x7fb79808c3c0 2021-04-09T18:21:50.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb796ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79808bfc0 0x7fb798092c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:50.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb796ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79808bfc0 0x7fb798092c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48646/0 (socket says 172.21.15.32:48646) 2021-04-09T18:21:50.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb796ffd700 1 -- 172.21.15.32:0/2502397278 learned_addr learned my addr 172.21.15.32:0/2502397278 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:50.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb7977fe700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7980c0ac0 0x7fb7980bfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb797fff700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb79808c3c0 0x7fb798093270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:50.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb796ffd700 1 -- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb79808c3c0 msgr2=0x7fb798093270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:50.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb796ffd700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb79808c3c0 0x7fb798093270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb796ffd700 1 -- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7980c0ac0 msgr2=0x7fb7980bfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:50.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb796ffd700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7980c0ac0 0x7fb7980bfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb796ffd700 1 -- 172.21.15.32:0/2502397278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb798093ab0 con 0x7fb79808bfc0 2021-04-09T18:21:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb7977fe700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7980c0ac0 0x7fb7980bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:50.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb797fff700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb79808c3c0 0x7fb798093270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.175 7fb796ffd700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79808bfc0 0x7fb798092c50 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7fb784006f20 tx=0x7fb78400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.176 7fb78ffff700 1 -- 172.21.15.32:0/2502397278 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb78400bd80 con 0x7fb79808bfc0 2021-04-09T18:21:50.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.176 7fb78ffff700 1 -- 172.21.15.32:0/2502397278 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb784002030 con 0x7fb79808bfc0 2021-04-09T18:21:50.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.176 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb798093ab0 con 0x7fb79808bfc0 2021-04-09T18:21:50.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.176 7fb78ffff700 1 -- 172.21.15.32:0/2502397278 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7840024c0 con 0x7fb79808bfc0 2021-04-09T18:21:50.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.176 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb798093ab0 con 0x7fb79808bfc0 2021-04-09T18:21:50.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.176 7fb78ffff700 1 -- 172.21.15.32:0/2502397278 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb784002210 con 0x7fb79808bfc0 2021-04-09T18:21:50.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.176 7fb78ffff700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb774018930 0x7fb77401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:50.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.176 7fb78ffff700 1 -- 172.21.15.32:0/2502397278 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fb784032710 con 0x7fb79808bfc0 2021-04-09T18:21:50.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.177 7fb7977fe700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb774018930 0x7fb77401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:50.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.177 7fb7977fe700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb774018930 0x7fb77401ae10 crc :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:50.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.196 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79810d480 con 0x7fb79808bfc0 2021-04-09T18:21:50.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.200 7fb78ffff700 1 -- 172.21.15.32:0/2502397278 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb784002240 con 0x7fb79808bfc0 2021-04-09T18:21:50.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:50.385 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb79810d480 con 0x7fb79808bfc0 2021-04-09T18:21:51.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.321 7fb78ffff700 1 -- 172.21.15.32:0/2502397278 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v89) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb784002030 con 0x7fb79808bfc0 2021-04-09T18:21:51.402 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:21:51.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb774018930 msgr2=0x7fb77401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:51.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb774018930 0x7fb77401ae10 crc :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79808bfc0 msgr2=0x7fb798092c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:51.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79808bfc0 0x7fb798092c50 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7fb784006f20 tx=0x7fb78400b990).stop 2021-04-09T18:21:51.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 shutdown_connections 2021-04-09T18:21:51.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb774018930 0x7fb77401ae10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb79808bfc0 0x7fb798092c50 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb79808c3c0 0x7fb798093270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 --2- 172.21.15.32:0/2502397278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7980c0ac0 0x7fb7980bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.353 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 >> 172.21.15.32:0/2502397278 conn(0x7fb798095040 msgr2=0x7fb7980bd220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:51.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.354 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 shutdown_connections 2021-04-09T18:21:51.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.354 7fb79dc51700 1 -- 172.21.15.32:0/2502397278 wait complete. 2021-04-09T18:21:51.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:21:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.523 7fdf20641700 1 Processor -- start 2021-04-09T18:21:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.523 7fdf20641700 1 -- start start 2021-04-09T18:21:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf20641700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1810da00 0x7fdf1810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf20641700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1810d480 con 0x7fdf1810d010 2021-04-09T18:21:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf20641700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf180d1f70 con 0x7fdf180d7ca0 2021-04-09T18:21:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf20641700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180d9220 con 0x7fdf1810da00 2021-04-09T18:21:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf1e45d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdf1810d010 0x7fdf180d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58244/0 (socket says 172.21.15.32:58244) 2021-04-09T18:21:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf1e45d700 1 -- 172.21.15.32:0/2000495714 learned_addr learned my addr 172.21.15.32:0/2000495714 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf1ec5e700 1 --2- 172.21.15.32:0/2000495714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1810da00 0x7fdf1810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf1d45b700 1 -- 172.21.15.32:0/2000495714 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511383707 0 0) 0x7fdf14000ce0 con 0x7fdf1810d010 2021-04-09T18:21:51.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf1d45b700 1 -- 172.21.15.32:0/2000495714 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf04002fe0 con 0x7fdf1810d010 2021-04-09T18:21:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.524 7fdf1d45b700 1 -- 172.21.15.32:0/2000495714 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397013562 0 0) 0x7fdf08000c60 con 0x7fdf180d7ca0 2021-04-09T18:21:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf1d45b700 1 -- 172.21.15.32:0/2000495714 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf04003020 con 0x7fdf180d7ca0 2021-04-09T18:21:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf1ec5e700 1 -- 172.21.15.32:0/2000495714 >> v1:172.21.15.86:6789/0 conn(0x7fdf180d7ca0 legacy=0x7fdf180d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:51.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf1ec5e700 1 -- 172.21.15.32:0/2000495714 >> v1:172.21.15.32:6789/0 conn(0x7fdf1810d010 legacy=0x7fdf180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:51.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf1ec5e700 1 -- 172.21.15.32:0/2000495714 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1810e420 con 0x7fdf1810da00 2021-04-09T18:21:51.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf1ec5e700 1 --2- 172.21.15.32:0/2000495714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1810da00 0x7fdf1810de00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fdf10006e50 tx=0x7fdf10009e70).ready entity=mon.2 client_cookie=ef1b0b90f4c05250 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:51.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf1d45b700 1 -- 172.21.15.32:0/2000495714 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf10006a20 con 0x7fdf1810da00 2021-04-09T18:21:51.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf1d45b700 1 -- 172.21.15.32:0/2000495714 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf10006790 con 0x7fdf1810da00 2021-04-09T18:21:51.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf1d45b700 1 -- 172.21.15.32:0/2000495714 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf1000b700 con 0x7fdf1810da00 2021-04-09T18:21:51.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf20641700 1 -- 172.21.15.32:0/2000495714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1810da00 msgr2=0x7fdf1810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:51.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.525 7fdf20641700 1 --2- 172.21.15.32:0/2000495714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1810da00 0x7fdf1810de00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fdf10006e50 tx=0x7fdf10009e70).stop 2021-04-09T18:21:51.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.526 7fdf20641700 1 -- 172.21.15.32:0/2000495714 shutdown_connections 2021-04-09T18:21:51.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.526 7fdf20641700 1 --2- 172.21.15.32:0/2000495714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1810da00 0x7fdf1810de00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.526 7fdf20641700 1 -- 172.21.15.32:0/2000495714 >> 172.21.15.32:0/2000495714 conn(0x7fdf180ce5f0 msgr2=0x7fdf180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.526 7fdf20641700 1 -- 172.21.15.32:0/2000495714 shutdown_connections 2021-04-09T18:21:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.526 7fdf20641700 1 -- 172.21.15.32:0/2000495714 wait complete. 2021-04-09T18:21:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.546 7fdf20641700 1 Processor -- start 2021-04-09T18:21:51.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.546 7fdf20641700 1 -- start start 2021-04-09T18:21:51.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf20641700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf180b8de0 0x7fdf180c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:51.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf20641700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf180c0640 0x7fdf181936d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:51.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf20641700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf18193cf0 0x7fdf181940f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:51.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf20641700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180c0a40 con 0x7fdf18193cf0 2021-04-09T18:21:51.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf20641700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180b89a0 con 0x7fdf180c0640 2021-04-09T18:21:51.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf20641700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf181b7b70 con 0x7fdf180b8de0 2021-04-09T18:21:51.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1ec5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf18193cf0 0x7fdf181940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1e45d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf180b8de0 0x7fdf180c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1ec5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf18193cf0 0x7fdf181940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48662/0 (socket says 172.21.15.32:48662) 2021-04-09T18:21:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1ec5e700 1 -- 172.21.15.32:0/2705694270 learned_addr learned my addr 172.21.15.32:0/2705694270 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:51.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1dc5c700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf180c0640 0x7fdf181936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1e45d700 1 -- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf180c0640 msgr2=0x7fdf181936d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1e45d700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf180c0640 0x7fdf181936d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1e45d700 1 -- 172.21.15.32:0/2705694270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf18193cf0 msgr2=0x7fdf181940f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1e45d700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf18193cf0 0x7fdf181940f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1e45d700 1 -- 172.21.15.32:0/2705694270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf180bb2d0 con 0x7fdf180b8de0 2021-04-09T18:21:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.547 7fdf1e45d700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf180b8de0 0x7fdf180c0020 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fdf14009fa0 tx=0x7fdf1400b840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:51.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.548 7fdf0effd700 1 -- 172.21.15.32:0/2705694270 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf140043a0 con 0x7fdf180b8de0 2021-04-09T18:21:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.548 7fdf0effd700 1 -- 172.21.15.32:0/2705694270 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf140063e0 con 0x7fdf180b8de0 2021-04-09T18:21:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.548 7fdf20641700 1 -- 172.21.15.32:0/2705694270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf180bb2d0 con 0x7fdf180b8de0 2021-04-09T18:21:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.548 7fdf0effd700 1 -- 172.21.15.32:0/2705694270 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf14004850 con 0x7fdf180b8de0 2021-04-09T18:21:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.548 7fdf20641700 1 -- 172.21.15.32:0/2705694270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf180bb2d0 con 0x7fdf180b8de0 2021-04-09T18:21:51.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.548 7fdf0effd700 1 -- 172.21.15.32:0/2705694270 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdf140043a0 con 0x7fdf180b8de0 2021-04-09T18:21:51.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.549 7fdf0effd700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf00018930 0x7fdf0001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:51.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.549 7fdf0effd700 1 -- 172.21.15.32:0/2705694270 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5111+0+0 (secure 0 0 0) 0x7fdf14032740 con 0x7fdf180b8de0 2021-04-09T18:21:51.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.549 7fdf1dc5c700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf00018930 0x7fdf0001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:51.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.549 7fdf1dc5c700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf00018930 0x7fdf0001ae10 crc :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:51.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.568 7fdf20641700 1 -- 172.21.15.32:0/2705694270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf180b89a0 con 0x7fdf180b8de0 2021-04-09T18:21:51.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.572 7fdf0effd700 1 -- 172.21.15.32:0/2705694270 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdf140063c0 con 0x7fdf180b8de0 2021-04-09T18:21:51.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.755 7fdf20641700 1 -- 172.21.15.32:0/2705694270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fdf180b89a0 con 0x7fdf180b8de0 2021-04-09T18:21:51.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.900 7fdf0effd700 1 -- 172.21.15.32:0/2705694270 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v90) v1 ==== 125+0+0 (secure 0 0 0) 0x7fdf140063c0 con 0x7fdf180b8de0 2021-04-09T18:21:51.969 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:21:51.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 -- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf00018930 msgr2=0x7fdf0001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:51.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf00018930 0x7fdf0001ae10 crc :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 -- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf180b8de0 msgr2=0x7fdf180c0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf180b8de0 0x7fdf180c0020 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fdf14009fa0 tx=0x7fdf1400b840).stop 2021-04-09T18:21:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 -- 172.21.15.32:0/2705694270 shutdown_connections 2021-04-09T18:21:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf00018930 0x7fdf0001ae10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf18193cf0 0x7fdf181940f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf180c0640 0x7fdf181936d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 --2- 172.21.15.32:0/2705694270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf180b8de0 0x7fdf180c0020 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:51.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.921 7fdf20641700 1 -- 172.21.15.32:0/2705694270 >> 172.21.15.32:0/2705694270 conn(0x7fdf18092a00 msgr2=0x7fdf181b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.922 7fdf20641700 1 -- 172.21.15.32:0/2705694270 shutdown_connections 2021-04-09T18:21:51.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:51.922 7fdf20641700 1 -- 172.21.15.32:0/2705694270 wait complete. 2021-04-09T18:21:51.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:21:52.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.095 7fc021917700 1 Processor -- start 2021-04-09T18:21:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.095 7fc021917700 1 -- start start 2021-04-09T18:21:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.096 7fc021917700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c10d020 0x7fc01c0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.096 7fc021917700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc01c0d84a0 0x7fc01c10da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.096 7fc021917700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01c10d490 con 0x7fc01c0d1db0 2021-04-09T18:21:52.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.096 7fc021917700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01c10e790 con 0x7fc01c0d84a0 2021-04-09T18:21:52.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.096 7fc021917700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0d8d40 con 0x7fc01c10d020 2021-04-09T18:21:52.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.096 7fc01bd75700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc01c0d84a0 0x7fc01c10da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.096 7fc01b574700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c10d020 0x7fc01c0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.096 7fc01ad73700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc01c0d1db0 0x7fc01c0d7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:56186/0 (socket says 172.21.15.32:56186) 2021-04-09T18:21:52.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01ad73700 1 -- 172.21.15.32:0/3428006774 learned_addr learned my addr 172.21.15.32:0/3428006774 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01bd75700 1 -- 172.21.15.32:0/3428006774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c10d020 msgr2=0x7fc01c0d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01bd75700 1 --2- 172.21.15.32:0/3428006774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c10d020 0x7fc01c0d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01bd75700 1 -- 172.21.15.32:0/3428006774 >> v1:172.21.15.86:6789/0 conn(0x7fc01c0d1db0 legacy=0x7fc01c0d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01bd75700 1 -- 172.21.15.32:0/3428006774 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c0d9230 con 0x7fc01c0d84a0 2021-04-09T18:21:52.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01b574700 1 --2- 172.21.15.32:0/3428006774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c10d020 0x7fc01c0d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01a572700 1 -- 172.21.15.32:0/3428006774 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416119843 0 0) 0x7fc010001440 con 0x7fc01c0d1db0 2021-04-09T18:21:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01bd75700 1 --2- 172.21.15.32:0/3428006774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0d84a0 0x7fc01c10da10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fc014006ad0 tx=0x7fc014009ad0).ready entity=mon.1 client_cookie=56400fffb6b5ff12 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:52.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01a572700 1 -- 172.21.15.32:0/3428006774 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0140022e0 con 0x7fc01c0d84a0 2021-04-09T18:21:52.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01a572700 1 -- 172.21.15.32:0/3428006774 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc014006410 con 0x7fc01c0d84a0 2021-04-09T18:21:52.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc01a572700 1 -- 172.21.15.32:0/3428006774 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc014000c10 con 0x7fc01c0d84a0 2021-04-09T18:21:52.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc021917700 1 -- 172.21.15.32:0/3428006774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0d84a0 msgr2=0x7fc01c10da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.097 7fc021917700 1 --2- 172.21.15.32:0/3428006774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0d84a0 0x7fc01c10da10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fc014006ad0 tx=0x7fc014009ad0).stop 2021-04-09T18:21:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.098 7fc021917700 1 -- 172.21.15.32:0/3428006774 shutdown_connections 2021-04-09T18:21:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.098 7fc021917700 1 --2- 172.21.15.32:0/3428006774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0d84a0 0x7fc01c10da10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.098 7fc021917700 1 --2- 172.21.15.32:0/3428006774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c10d020 0x7fc01c0d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.098 7fc021917700 1 -- 172.21.15.32:0/3428006774 >> 172.21.15.32:0/3428006774 conn(0x7fc01c0ce600 msgr2=0x7fc01c0d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.098 7fc021917700 1 -- 172.21.15.32:0/3428006774 shutdown_connections 2021-04-09T18:21:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.098 7fc021917700 1 -- 172.21.15.32:0/3428006774 wait complete. 2021-04-09T18:21:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.118 7fc021917700 1 Processor -- start 2021-04-09T18:21:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.118 7fc021917700 1 -- start start 2021-04-09T18:21:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc021917700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c0ad730 0x7fc01c0ab120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc021917700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0adb30 0x7fc01c0a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc021917700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0a0df0 0x7fc01c0a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc021917700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0a11f0 con 0x7fc01c0ad730 2021-04-09T18:21:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc021917700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0a1420 con 0x7fc01c0adb30 2021-04-09T18:21:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc021917700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0bce20 con 0x7fc01c0a0df0 2021-04-09T18:21:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c0ad730 0x7fc01c0ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01ad73700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0adb30 0x7fc01c0a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01bd75700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0a0df0 0x7fc01c0a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01ad73700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0adb30 0x7fc01c0a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:40430/0 (socket says 172.21.15.32:40430) 2021-04-09T18:21:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01ad73700 1 -- 172.21.15.32:0/2956122786 learned_addr learned my addr 172.21.15.32:0/2956122786 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c0ad730 0x7fc01c0ab120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48674/0 (socket says 172.21.15.32:48674) 2021-04-09T18:21:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01ad73700 1 -- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0a0df0 msgr2=0x7fc01c0a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01ad73700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0a0df0 0x7fc01c0a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01ad73700 1 -- 172.21.15.32:0/2956122786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c0ad730 msgr2=0x7fc01c0ab120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01ad73700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c0ad730 0x7fc01c0ab120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01ad73700 1 -- 172.21.15.32:0/2956122786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c0ab740 con 0x7fc01c0adb30 2021-04-09T18:21:52.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.119 7fc01ad73700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0adb30 0x7fc01c0a07d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fc010009980 tx=0x7fc01000bcd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:52.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.120 7fc003fff700 1 -- 172.21.15.32:0/2956122786 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc010006300 con 0x7fc01c0adb30 2021-04-09T18:21:52.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.120 7fc003fff700 1 -- 172.21.15.32:0/2956122786 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0100046a0 con 0x7fc01c0adb30 2021-04-09T18:21:52.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.120 7fc021917700 1 -- 172.21.15.32:0/2956122786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01c0ab740 con 0x7fc01c0adb30 2021-04-09T18:21:52.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.120 7fc003fff700 1 -- 172.21.15.32:0/2956122786 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc010004a70 con 0x7fc01c0adb30 2021-04-09T18:21:52.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.120 7fc021917700 1 -- 172.21.15.32:0/2956122786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01c0ab740 con 0x7fc01c0adb30 2021-04-09T18:21:52.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.120 7fc003fff700 1 -- 172.21.15.32:0/2956122786 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc010004880 con 0x7fc01c0adb30 2021-04-09T18:21:52.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.120 7fc003fff700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbffc018930 0x7fbffc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.120 7fc003fff700 1 -- 172.21.15.32:0/2956122786 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5461+0+0 (secure 0 0 0) 0x7fc010032d00 con 0x7fc01c0adb30 2021-04-09T18:21:52.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.121 7fc01b574700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbffc018930 0x7fbffc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.121 7fc01b574700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbffc018930 0x7fbffc01ae10 crc :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:52.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.140 7fc021917700 1 -- 172.21.15.32:0/2956122786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01c0b9010 con 0x7fc01c0adb30 2021-04-09T18:21:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.144 7fc003fff700 1 -- 172.21.15.32:0/2956122786 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc01000be10 con 0x7fc01c0adb30 2021-04-09T18:21:52.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.327 7fc021917700 1 -- 172.21.15.32:0/2956122786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fc01c10d490 con 0x7fc01c0adb30 2021-04-09T18:21:52.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.340 7fc003fff700 1 -- 172.21.15.32:0/2956122786 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 16 and data pool 17 v106) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc0100046a0 con 0x7fc01c0adb30 2021-04-09T18:21:52.396 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 16 and data pool 17 2021-04-09T18:21:52.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.345 7fc021917700 1 -- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbffc018930 msgr2=0x7fbffc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbffc018930 0x7fbffc01ae10 crc :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 -- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0adb30 msgr2=0x7fc01c0a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0adb30 0x7fc01c0a07d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fc010009980 tx=0x7fc01000bcd0).stop 2021-04-09T18:21:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 -- 172.21.15.32:0/2956122786 shutdown_connections 2021-04-09T18:21:52.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbffc018930 0x7fbffc01ae10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc01c0a0df0 0x7fc01c0a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc01c0adb30 0x7fc01c0a07d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 --2- 172.21.15.32:0/2956122786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc01c0ad730 0x7fc01c0ab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 -- 172.21.15.32:0/2956122786 >> 172.21.15.32:0/2956122786 conn(0x7fc01c0b9a90 msgr2=0x7fc01c0c1370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:52.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.346 7fc021917700 1 -- 172.21.15.32:0/2956122786 shutdown_connections 2021-04-09T18:21:52.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.347 7fc021917700 1 -- 172.21.15.32:0/2956122786 wait complete. 2021-04-09T18:21:52.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:21:52.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.505 7f214bd26700 1 Processor -- start 2021-04-09T18:21:52.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.505 7f214bd26700 1 -- start start 2021-04-09T18:21:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f214bd26700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f21440d7ca0 0x7f21440d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f214bd26700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214410d480 con 0x7f214410da00 2021-04-09T18:21:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f214bd26700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21440d86c0 con 0x7f214410d010 2021-04-09T18:21:52.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f214bd26700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21440d9220 con 0x7f21440d7ca0 2021-04-09T18:21:52.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f214a343700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f214410da00 0x7f214410de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58276/0 (socket says 172.21.15.32:58276) 2021-04-09T18:21:52.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f214a343700 1 -- 172.21.15.32:0/959490376 learned_addr learned my addr 172.21.15.32:0/959490376 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:52.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f2149341700 1 --2- 172.21.15.32:0/959490376 >> v2:172.21.15.86:3300/0 conn(0x7f21440d7ca0 0x7f21440d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f2148b40700 1 -- 172.21.15.32:0/959490376 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639375287 0 0) 0x7f213c000d30 con 0x7f214410da00 2021-04-09T18:21:52.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f2148b40700 1 -- 172.21.15.32:0/959490376 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2130002fe0 con 0x7f214410da00 2021-04-09T18:21:52.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f2148b40700 1 -- 172.21.15.32:0/959490376 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420020485 0 0) 0x7f2140000c60 con 0x7f214410d010 2021-04-09T18:21:52.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f2148b40700 1 -- 172.21.15.32:0/959490376 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21300042c0 con 0x7f214410d010 2021-04-09T18:21:52.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f2149341700 1 -- 172.21.15.32:0/959490376 >> v1:172.21.15.86:6789/0 conn(0x7f214410d010 legacy=0x7f21440d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f2149341700 1 -- 172.21.15.32:0/959490376 >> v1:172.21.15.32:6789/0 conn(0x7f214410da00 legacy=0x7f214410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.506 7f2149341700 1 -- 172.21.15.32:0/959490376 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2144112100 con 0x7f21440d7ca0 2021-04-09T18:21:52.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.507 7f2149341700 1 --2- 172.21.15.32:0/959490376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21440d7ca0 0x7f21440d80a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f2134006650 tx=0x7f2134009a80).ready entity=mon.1 client_cookie=195e76d2ac1efc9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:52.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.507 7f2148b40700 1 -- 172.21.15.32:0/959490376 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21340009b0 con 0x7f21440d7ca0 2021-04-09T18:21:52.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.507 7f2148b40700 1 -- 172.21.15.32:0/959490376 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213400c0c0 con 0x7f21440d7ca0 2021-04-09T18:21:52.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.507 7f2148b40700 1 -- 172.21.15.32:0/959490376 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f213400c510 con 0x7f21440d7ca0 2021-04-09T18:21:52.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.507 7f214bd26700 1 -- 172.21.15.32:0/959490376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21440d7ca0 msgr2=0x7f21440d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.507 7f214bd26700 1 --2- 172.21.15.32:0/959490376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21440d7ca0 0x7f21440d80a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f2134006650 tx=0x7f2134009a80).stop 2021-04-09T18:21:52.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.507 7f214bd26700 1 -- 172.21.15.32:0/959490376 shutdown_connections 2021-04-09T18:21:52.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.507 7f214bd26700 1 --2- 172.21.15.32:0/959490376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21440d7ca0 0x7f21440d80a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.507 7f214bd26700 1 -- 172.21.15.32:0/959490376 >> 172.21.15.32:0/959490376 conn(0x7f21440ce5f0 msgr2=0x7f21440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.508 7f214bd26700 1 -- 172.21.15.32:0/959490376 shutdown_connections 2021-04-09T18:21:52.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.508 7f214bd26700 1 -- 172.21.15.32:0/959490376 wait complete. 2021-04-09T18:21:52.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214bd26700 1 Processor -- start 2021-04-09T18:21:52.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214bd26700 1 -- start start 2021-04-09T18:21:52.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214bd26700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2144181e10 0x7f214419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214bd26700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21441b8500 0x7f21441b8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214bd26700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f214408ced0 0x7f214408d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214bd26700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214408c950 con 0x7f214408ced0 2021-04-09T18:21:52.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214bd26700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2144185ba0 con 0x7f21441b8500 2021-04-09T18:21:52.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214bd26700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214408d8f0 con 0x7f2144181e10 2021-04-09T18:21:52.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214a343700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f214408ced0 0x7f214408d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214a343700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f214408ced0 0x7f214408d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48692/0 (socket says 172.21.15.32:48692) 2021-04-09T18:21:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.528 7f214a343700 1 -- 172.21.15.32:0/310226254 learned_addr learned my addr 172.21.15.32:0/310226254 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f2149b42700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2144181e10 0x7f214419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f2149341700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21441b8500 0x7f21441b8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f214a343700 1 -- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2144181e10 msgr2=0x7f214419d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f214a343700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2144181e10 0x7f214419d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f214a343700 1 -- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21441b8500 msgr2=0x7f21441b8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f214a343700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21441b8500 0x7f21441b8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f214a343700 1 -- 172.21.15.32:0/310226254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214408db10 con 0x7f214408ced0 2021-04-09T18:21:52.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f2149341700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21441b8500 0x7f21441b8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:52.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f214a343700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f214408ced0 0x7f214408d2d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f213c009980 tx=0x7f213c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:52.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f213a7fc700 1 -- 172.21.15.32:0/310226254 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f213c00be10 con 0x7f214408ced0 2021-04-09T18:21:52.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f213a7fc700 1 -- 172.21.15.32:0/310226254 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c006080 con 0x7f214408ced0 2021-04-09T18:21:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f214bd26700 1 -- 172.21.15.32:0/310226254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214408db10 con 0x7f214408ced0 2021-04-09T18:21:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f213a7fc700 1 -- 172.21.15.32:0/310226254 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f213c004720 con 0x7f214408ced0 2021-04-09T18:21:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.529 7f214bd26700 1 -- 172.21.15.32:0/310226254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214408db10 con 0x7f214408ced0 2021-04-09T18:21:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.530 7f213a7fc700 1 -- 172.21.15.32:0/310226254 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f213c004720 con 0x7f214408ced0 2021-04-09T18:21:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.530 7f213a7fc700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f212c0188c0 0x7f212c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.530 7f213a7fc700 1 -- 172.21.15.32:0/310226254 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f213c032ab0 con 0x7f214408ced0 2021-04-09T18:21:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.531 7f2149b42700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f212c0188c0 0x7f212c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.531 7f2149b42700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f212c0188c0 0x7f212c01adc0 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:52.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.550 7f214bd26700 1 -- 172.21.15.32:0/310226254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21441b7010 con 0x7f214408ced0 2021-04-09T18:21:52.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.553 7f213a7fc700 1 -- 172.21.15.32:0/310226254 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f213c000f90 con 0x7f214408ced0 2021-04-09T18:21:52.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.740 7f214bd26700 1 -- 172.21.15.32:0/310226254 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21441b7010 con 0x7f214408ced0 2021-04-09T18:21:52.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.741 7f213a7fc700 1 -- 172.21.15.32:0/310226254 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f213c004720 con 0x7f214408ced0 2021-04-09T18:21:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:52.792 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":94,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:21:52.329511","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:21:50.420671","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-04-09 18:21:51.790105","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:21:52.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 -- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f212c0188c0 msgr2=0x7f212c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f212c0188c0 0x7f212c01adc0 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 -- 172.21.15.32:0/310226254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f214408ced0 msgr2=0x7f214408d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f214408ced0 0x7f214408d2d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f213c009980 tx=0x7f213c00bcd0).stop 2021-04-09T18:21:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 -- 172.21.15.32:0/310226254 shutdown_connections 2021-04-09T18:21:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f212c0188c0 0x7f212c01adc0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f214408ced0 0x7f214408d2d0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21441b8500 0x7f21441b8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 --2- 172.21.15.32:0/310226254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2144181e10 0x7f214419d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 -- 172.21.15.32:0/310226254 >> 172.21.15.32:0/310226254 conn(0x7f2144190550 msgr2=0x7f21441b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:52.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 -- 172.21.15.32:0/310226254 shutdown_connections 2021-04-09T18:21:52.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.743 7f214bd26700 1 -- 172.21.15.32:0/310226254 wait complete. 2021-04-09T18:21:52.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:21:52.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.902 7f50441dd700 1 Processor -- start 2021-04-09T18:21:52.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.902 7f50441dd700 1 -- start start 2021-04-09T18:21:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f50441dd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f503c0d7ca0 0x7f503c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f50441dd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c10d480 con 0x7f503c10d010 2021-04-09T18:21:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f50441dd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0d86c0 con 0x7f503c10da00 2021-04-09T18:21:52.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f50441dd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f503c0d9220 con 0x7f503c0d7ca0 2021-04-09T18:21:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f5041ff9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f503c10d010 0x7f503c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58288/0 (socket says 172.21.15.32:58288) 2021-04-09T18:21:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f5041ff9700 1 -- 172.21.15.32:0/2810081322 learned_addr learned my addr 172.21.15.32:0/2810081322 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f50417f8700 1 --2- 172.21.15.32:0/2810081322 >> v2:172.21.15.86:3300/0 conn(0x7f503c0d7ca0 0x7f503c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f5040ff7700 1 -- 172.21.15.32:0/2810081322 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667798003 0 0) 0x7f5034000d30 con 0x7f503c10da00 2021-04-09T18:21:52.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f5040ff7700 1 -- 172.21.15.32:0/2810081322 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5028002fe0 con 0x7f503c10da00 2021-04-09T18:21:52.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f5040ff7700 1 -- 172.21.15.32:0/2810081322 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603420252 0 0) 0x7f5038000ce0 con 0x7f503c10d010 2021-04-09T18:21:52.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f5040ff7700 1 -- 172.21.15.32:0/2810081322 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50280042c0 con 0x7f503c10d010 2021-04-09T18:21:52.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f50417f8700 1 -- 172.21.15.32:0/2810081322 >> v1:172.21.15.86:6789/0 conn(0x7f503c10da00 legacy=0x7f503c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f50417f8700 1 -- 172.21.15.32:0/2810081322 >> v1:172.21.15.32:6789/0 conn(0x7f503c10d010 legacy=0x7f503c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f50417f8700 1 -- 172.21.15.32:0/2810081322 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c0d1f70 con 0x7f503c0d7ca0 2021-04-09T18:21:52.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f50417f8700 1 --2- 172.21.15.32:0/2810081322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c0d7ca0 0x7f503c0d80a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f502c006650 tx=0x7f502c009a80).ready entity=mon.1 client_cookie=c7c6f7250ffebd1b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:52.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f5040ff7700 1 -- 172.21.15.32:0/2810081322 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f502c0009b0 con 0x7f503c0d7ca0 2021-04-09T18:21:52.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f5040ff7700 1 -- 172.21.15.32:0/2810081322 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502c00c0c0 con 0x7f503c0d7ca0 2021-04-09T18:21:52.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.903 7f5040ff7700 1 -- 172.21.15.32:0/2810081322 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f502c00c510 con 0x7f503c0d7ca0 2021-04-09T18:21:52.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.904 7f50441dd700 1 -- 172.21.15.32:0/2810081322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c0d7ca0 msgr2=0x7f503c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.904 7f50441dd700 1 --2- 172.21.15.32:0/2810081322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c0d7ca0 0x7f503c0d80a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f502c006650 tx=0x7f502c009a80).stop 2021-04-09T18:21:52.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.904 7f50441dd700 1 -- 172.21.15.32:0/2810081322 shutdown_connections 2021-04-09T18:21:52.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.904 7f50441dd700 1 --2- 172.21.15.32:0/2810081322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c0d7ca0 0x7f503c0d80a0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.904 7f50441dd700 1 -- 172.21.15.32:0/2810081322 >> 172.21.15.32:0/2810081322 conn(0x7f503c0ce5f0 msgr2=0x7f503c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.904 7f50441dd700 1 -- 172.21.15.32:0/2810081322 shutdown_connections 2021-04-09T18:21:52.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.904 7f50441dd700 1 -- 172.21.15.32:0/2810081322 wait complete. 2021-04-09T18:21:52.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.924 7f50441dd700 1 Processor -- start 2021-04-09T18:21:52.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.924 7f50441dd700 1 -- start start 2021-04-09T18:21:52.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50441dd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c181e10 0x7f503c19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50441dd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503c1b8500 0x7f503c1b8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50441dd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503c08ced0 0x7f503c08d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50441dd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c08c950 con 0x7f503c1b8500 2021-04-09T18:21:52.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50441dd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c185ba0 con 0x7f503c181e10 2021-04-09T18:21:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50441dd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c08d8f0 con 0x7f503c08ced0 2021-04-09T18:21:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50417f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503c1b8500 0x7f503c1b8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50417f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503c1b8500 0x7f503c1b8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48704/0 (socket says 172.21.15.32:48704) 2021-04-09T18:21:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50417f8700 1 -- 172.21.15.32:0/1209760101 learned_addr learned my addr 172.21.15.32:0/1209760101 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:52.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f5041ff9700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c181e10 0x7f503c19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50427fa700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503c08ced0 0x7f503c08d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50417f8700 1 -- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503c08ced0 msgr2=0x7f503c08d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50417f8700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503c08ced0 0x7f503c08d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.925 7f50417f8700 1 -- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c181e10 msgr2=0x7f503c19d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:52.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.926 7f50417f8700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c181e10 0x7f503c19d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:52.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.926 7f50417f8700 1 -- 172.21.15.32:0/1209760101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c08db10 con 0x7f503c1b8500 2021-04-09T18:21:52.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.926 7f5041ff9700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c181e10 0x7f503c19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:21:52.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.926 7f50417f8700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503c1b8500 0x7f503c1b8900 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7f502c009650 tx=0x7f502c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:52.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.926 7f50327fc700 1 -- 172.21.15.32:0/1209760101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f502c0093a0 con 0x7f503c1b8500 2021-04-09T18:21:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.926 7f50327fc700 1 -- 172.21.15.32:0/1209760101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502c006070 con 0x7f503c1b8500 2021-04-09T18:21:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.926 7f50441dd700 1 -- 172.21.15.32:0/1209760101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c08db10 con 0x7f503c1b8500 2021-04-09T18:21:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.926 7f50327fc700 1 -- 172.21.15.32:0/1209760101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f502c0052e0 con 0x7f503c1b8500 2021-04-09T18:21:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.926 7f50441dd700 1 -- 172.21.15.32:0/1209760101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c08db10 con 0x7f503c1b8500 2021-04-09T18:21:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.927 7f50327fc700 1 -- 172.21.15.32:0/1209760101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f502c009300 con 0x7f503c1b8500 2021-04-09T18:21:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.927 7f50327fc700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5024018930 0x7f502401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.927 7f50327fc700 1 -- 172.21.15.32:0/1209760101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f502c032a00 con 0x7f503c1b8500 2021-04-09T18:21:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.927 7f5041ff9700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5024018930 0x7f502401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:52.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.927 7f5041ff9700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5024018930 0x7f502401ae10 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:52.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.947 7f50441dd700 1 -- 172.21.15.32:0/1209760101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c1b7010 con 0x7f503c1b8500 2021-04-09T18:21:53.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:52.950 7f50327fc700 1 -- 172.21.15.32:0/1209760101 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f502c006030 con 0x7f503c1b8500 2021-04-09T18:21:53.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.139 7f50441dd700 1 -- 172.21.15.32:0/1209760101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f503c1b7010 con 0x7f503c1b8500 2021-04-09T18:21:53.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.140 7f50327fc700 1 -- 172.21.15.32:0/1209760101 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f502c000940 con 0x7f503c1b8500 2021-04-09T18:21:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:53.193 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":94,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:21:52.329511","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:21:50.420671","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-04-09 18:21:51.790105","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:21:53.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 -- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5024018930 msgr2=0x7f502401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5024018930 0x7f502401ae10 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 -- 172.21.15.32:0/1209760101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503c1b8500 msgr2=0x7f503c1b8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:53.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503c1b8500 0x7f503c1b8900 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7f502c009650 tx=0x7f502c00b8e0).stop 2021-04-09T18:21:53.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 -- 172.21.15.32:0/1209760101 shutdown_connections 2021-04-09T18:21:53.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5024018930 0x7f502401ae10 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503c1b8500 0x7f503c1b8900 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503c08ced0 0x7f503c08d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 --2- 172.21.15.32:0/1209760101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f503c181e10 0x7f503c19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.142 7f50441dd700 1 -- 172.21.15.32:0/1209760101 >> 172.21.15.32:0/1209760101 conn(0x7f503c190550 msgr2=0x7f503c1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:53.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.143 7f50441dd700 1 -- 172.21.15.32:0/1209760101 shutdown_connections 2021-04-09T18:21:53.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.143 7f50441dd700 1 -- 172.21.15.32:0/1209760101 wait complete. 2021-04-09T18:21:53.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:21:53.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.307 7f17549ef700 1 Processor -- start 2021-04-09T18:21:53.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.307 7f17549ef700 1 -- start start 2021-04-09T18:21:53.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.307 7f17549ef700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f175010d010 0x7f17500d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:53.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.307 7f17549ef700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f17500d8490 0x7f175010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:53.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.307 7f17549ef700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175010d480 con 0x7f17500d1da0 2021-04-09T18:21:53.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.307 7f17549ef700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17500d8d30 con 0x7f17500d8490 2021-04-09T18:21:53.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.307 7f17549ef700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17500d9220 con 0x7f175010d010 2021-04-09T18:21:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174de1b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f17500d1da0 0x7f17500d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58304/0 (socket says 172.21.15.32:58304) 2021-04-09T18:21:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174ee1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f17500d8490 0x7f175010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174de1b700 1 -- 172.21.15.32:0/3921375140 learned_addr learned my addr 172.21.15.32:0/3921375140 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174e61c700 1 --2- 172.21.15.32:0/3921375140 >> v2:172.21.15.86:3300/0 conn(0x7f175010d010 0x7f17500d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174d61a700 1 -- 172.21.15.32:0/3921375140 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703405398 0 0) 0x7f17380009b0 con 0x7f17500d1da0 2021-04-09T18:21:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174d61a700 1 -- 172.21.15.32:0/3921375140 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f173c002fe0 con 0x7f17500d1da0 2021-04-09T18:21:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174ee1d700 1 -- 172.21.15.32:0/3921375140 >> v2:172.21.15.86:3300/0 conn(0x7f175010d010 msgr2=0x7f17500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174ee1d700 1 --2- 172.21.15.32:0/3921375140 >> v2:172.21.15.86:3300/0 conn(0x7f175010d010 0x7f17500d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174ee1d700 1 -- 172.21.15.32:0/3921375140 >> v1:172.21.15.32:6789/0 conn(0x7f17500d1da0 legacy=0x7f17500d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:53.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174ee1d700 1 -- 172.21.15.32:0/3921375140 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175010e020 con 0x7f17500d8490 2021-04-09T18:21:53.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174ee1d700 1 --2- 172.21.15.32:0/3921375140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17500d8490 0x7f175010da00 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7f1748006ab0 tx=0x7f1748009ac0).ready entity=mon.0 client_cookie=fb82c00b1c1539f3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:53.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.308 7f174d61a700 1 -- 172.21.15.32:0/3921375140 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17480066d0 con 0x7f17500d8490 2021-04-09T18:21:53.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.309 7f174d61a700 1 -- 172.21.15.32:0/3921375140 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17480022e0 con 0x7f17500d8490 2021-04-09T18:21:53.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.309 7f174d61a700 1 -- 172.21.15.32:0/3921375140 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1748002740 con 0x7f17500d8490 2021-04-09T18:21:53.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.309 7f17549ef700 1 -- 172.21.15.32:0/3921375140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17500d8490 msgr2=0x7f175010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:53.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.309 7f17549ef700 1 --2- 172.21.15.32:0/3921375140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17500d8490 0x7f175010da00 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7f1748006ab0 tx=0x7f1748009ac0).stop 2021-04-09T18:21:53.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.309 7f17549ef700 1 -- 172.21.15.32:0/3921375140 shutdown_connections 2021-04-09T18:21:53.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.309 7f17549ef700 1 --2- 172.21.15.32:0/3921375140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17500d8490 0x7f175010da00 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.309 7f17549ef700 1 --2- 172.21.15.32:0/3921375140 >> v2:172.21.15.86:3300/0 conn(0x7f175010d010 0x7f17500d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.309 7f17549ef700 1 -- 172.21.15.32:0/3921375140 >> 172.21.15.32:0/3921375140 conn(0x7f17500ce5f0 msgr2=0x7f17500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:53.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.310 7f17549ef700 1 -- 172.21.15.32:0/3921375140 shutdown_connections 2021-04-09T18:21:53.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.310 7f17549ef700 1 -- 172.21.15.32:0/3921375140 wait complete. 2021-04-09T18:21:53.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.330 7f17549ef700 1 Processor -- start 2021-04-09T18:21:53.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.330 7f17549ef700 1 -- start start 2021-04-09T18:21:53.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f17549ef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17500c3c00 0x7f175009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:53.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f17549ef700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1750095240 0x7f17501a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:53.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f17549ef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17501a1270 0x7f17500b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:53.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f17549ef700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175009cda0 con 0x7f17501a1270 2021-04-09T18:21:53.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f17549ef700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17500c3680 con 0x7f17500c3c00 2021-04-09T18:21:53.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f17549ef700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1750095640 con 0x7f1750095240 2021-04-09T18:21:53.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174ee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17501a1270 0x7f17500b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:53.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174e61c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17500c3c00 0x7f175009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:53.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174ee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17501a1270 0x7f17500b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48718/0 (socket says 172.21.15.32:48718) 2021-04-09T18:21:53.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174ee1d700 1 -- 172.21.15.32:0/2281596050 learned_addr learned my addr 172.21.15.32:0/2281596050 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:53.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174de1b700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1750095240 0x7f17501a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:53.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174ee1d700 1 -- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1750095240 msgr2=0x7f17501a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:53.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174ee1d700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1750095240 0x7f17501a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174ee1d700 1 -- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17500c3c00 msgr2=0x7f175009c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:53.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174ee1d700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17500c3c00 0x7f175009c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:53.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174ee1d700 1 -- 172.21.15.32:0/2281596050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17501a1670 con 0x7f17501a1270 2021-04-09T18:21:53.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f174ee1d700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17501a1270 0x7f17500b0cf0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f1748002450 tx=0x7f17480027f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:53.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f1746ffd700 1 -- 172.21.15.32:0/2281596050 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f174800c350 con 0x7f17501a1270 2021-04-09T18:21:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f1746ffd700 1 -- 172.21.15.32:0/2281596050 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174800c610 con 0x7f17501a1270 2021-04-09T18:21:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f17549ef700 1 -- 172.21.15.32:0/2281596050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17501a1670 con 0x7f17501a1270 2021-04-09T18:21:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f1746ffd700 1 -- 172.21.15.32:0/2281596050 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f174800c9e0 con 0x7f17501a1270 2021-04-09T18:21:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.331 7f17549ef700 1 -- 172.21.15.32:0/2281596050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17501a1670 con 0x7f17501a1270 2021-04-09T18:21:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.332 7f1746ffd700 1 -- 172.21.15.32:0/2281596050 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1748002be0 con 0x7f17501a1270 2021-04-09T18:21:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.332 7f1746ffd700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1734018930 0x7f173401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.332 7f1746ffd700 1 -- 172.21.15.32:0/2281596050 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1748002be0 con 0x7f17501a1270 2021-04-09T18:21:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.332 7f174e61c700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1734018930 0x7f173401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.333 7f174e61c700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1734018930 0x7f173401adc0 crc :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.352 7f17549ef700 1 -- 172.21.15.32:0/2281596050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1750093d50 con 0x7f17501a1270 2021-04-09T18:21:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.355 7f1746ffd700 1 -- 172.21.15.32:0/2281596050 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1748006a70 con 0x7f17501a1270 2021-04-09T18:21:53.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:53.549 7f17549ef700 1 -- 172.21.15.32:0/2281596050 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f175010d480 con 0x7f17501a1270 2021-04-09T18:21:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.339 7f1746ffd700 1 -- 172.21.15.32:0/2281596050 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v108) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1748002be0 con 0x7f17501a1270 2021-04-09T18:21:54.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 -- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1734018930 msgr2=0x7f173401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:54.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1734018930 0x7f173401adc0 crc :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 -- 172.21.15.32:0/2281596050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17501a1270 msgr2=0x7f17500b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:54.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17501a1270 0x7f17500b0cf0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f1748002450 tx=0x7f17480027f0).stop 2021-04-09T18:21:54.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 -- 172.21.15.32:0/2281596050 shutdown_connections 2021-04-09T18:21:54.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1734018930 0x7f173401adc0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17501a1270 0x7f17500b0cf0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1750095240 0x7f17501a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 --2- 172.21.15.32:0/2281596050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17500c3c00 0x7f175009c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 -- 172.21.15.32:0/2281596050 >> 172.21.15.32:0/2281596050 conn(0x7f17500975f0 msgr2=0x7f1750096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:54.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 -- 172.21.15.32:0/2281596050 shutdown_connections 2021-04-09T18:21:54.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.367 7f17549ef700 1 -- 172.21.15.32:0/2281596050 wait complete. 2021-04-09T18:21:54.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:54.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.531 7f196b204700 1 Processor -- start 2021-04-09T18:21:54.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.531 7f196b204700 1 -- start start 2021-04-09T18:21:54.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196b204700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19640d7ca0 0x7f19640d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:54.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196b204700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196410d480 con 0x7f196410d010 2021-04-09T18:21:54.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196b204700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640d86c0 con 0x7f196410da00 2021-04-09T18:21:54.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196b204700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19640d9220 con 0x7f19640d7ca0 2021-04-09T18:21:54.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196881f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19640d7ca0 0x7f19640d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:54.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f1969020700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f196410d010 0x7f19640d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58314/0 (socket says 172.21.15.32:58314) 2021-04-09T18:21:54.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f1969020700 1 -- 172.21.15.32:0/1055889101 learned_addr learned my addr 172.21.15.32:0/1055889101 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:54.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196881f700 1 -- 172.21.15.32:0/1055889101 >> v1:172.21.15.86:6789/0 conn(0x7f196410da00 legacy=0x7f196410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:54.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196881f700 1 -- 172.21.15.32:0/1055889101 >> v1:172.21.15.32:6789/0 conn(0x7f196410d010 legacy=0x7f19640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:54.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f195bfff700 1 -- 172.21.15.32:0/1055889101 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943074045 0 0) 0x7f1960000ce0 con 0x7f196410d010 2021-04-09T18:21:54.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196881f700 1 -- 172.21.15.32:0/1055889101 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19640d1f70 con 0x7f19640d7ca0 2021-04-09T18:21:54.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196881f700 1 --2- 172.21.15.32:0/1055889101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19640d7ca0 0x7f19640d80a0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f1954006650 tx=0x7f1954009a80).ready entity=mon.0 client_cookie=1a904e0205a1ef7a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:54.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f195bfff700 1 -- 172.21.15.32:0/1055889101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19540009b0 con 0x7f19640d7ca0 2021-04-09T18:21:54.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f195bfff700 1 -- 172.21.15.32:0/1055889101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195400c0c0 con 0x7f19640d7ca0 2021-04-09T18:21:54.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f195bfff700 1 -- 172.21.15.32:0/1055889101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195400c510 con 0x7f19640d7ca0 2021-04-09T18:21:54.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196b204700 1 -- 172.21.15.32:0/1055889101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19640d7ca0 msgr2=0x7f19640d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:54.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.532 7f196b204700 1 --2- 172.21.15.32:0/1055889101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19640d7ca0 0x7f19640d80a0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f1954006650 tx=0x7f1954009a80).stop 2021-04-09T18:21:54.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.533 7f196b204700 1 -- 172.21.15.32:0/1055889101 shutdown_connections 2021-04-09T18:21:54.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.533 7f196b204700 1 --2- 172.21.15.32:0/1055889101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19640d7ca0 0x7f19640d80a0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.533 7f196b204700 1 -- 172.21.15.32:0/1055889101 >> 172.21.15.32:0/1055889101 conn(0x7f19640ce5f0 msgr2=0x7f19640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:54.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.533 7f196b204700 1 -- 172.21.15.32:0/1055889101 shutdown_connections 2021-04-09T18:21:54.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.533 7f196b204700 1 -- 172.21.15.32:0/1055889101 wait complete. 2021-04-09T18:21:54.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.553 7f196b204700 1 Processor -- start 2021-04-09T18:21:54.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.553 7f196b204700 1 -- start start 2021-04-09T18:21:54.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f196b204700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19641acea0 0x7f19641ac750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:54.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f196b204700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1964197ca0 0x7f19641980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:54.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f196b204700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1964198eb0 0x7f19641992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:54.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f196b204700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19641986c0 con 0x7f19641acea0 2021-04-09T18:21:54.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f196b204700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1964198930 con 0x7f1964197ca0 2021-04-09T18:21:54.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f196b204700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19641ad2a0 con 0x7f1964198eb0 2021-04-09T18:21:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f1969020700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19641acea0 0x7f19641ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:54.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f1969020700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19641acea0 0x7f19641ac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48728/0 (socket says 172.21.15.32:48728) 2021-04-09T18:21:54.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f1969020700 1 -- 172.21.15.32:0/1921241103 learned_addr learned my addr 172.21.15.32:0/1921241103 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:54.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f1969821700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1964198eb0 0x7f19641992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:54.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.554 7f196881f700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1964197ca0 0x7f19641980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:54.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f1969020700 1 -- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1964198eb0 msgr2=0x7f19641992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:54.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f1969020700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1964198eb0 0x7f19641992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f1969020700 1 -- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1964197ca0 msgr2=0x7f19641980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:54.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f1969020700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1964197ca0 0x7f19641980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f1969020700 1 -- 172.21.15.32:0/1921241103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19641998d0 con 0x7f19641acea0 2021-04-09T18:21:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f1969020700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19641acea0 0x7f19641ac750 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f196000b530 tx=0x7f196000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f1959ffb700 1 -- 172.21.15.32:0/1921241103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1960006020 con 0x7f19641acea0 2021-04-09T18:21:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f1959ffb700 1 -- 172.21.15.32:0/1921241103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19600063a0 con 0x7f19641acea0 2021-04-09T18:21:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f196b204700 1 -- 172.21.15.32:0/1921241103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19641998d0 con 0x7f19641acea0 2021-04-09T18:21:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f1959ffb700 1 -- 172.21.15.32:0/1921241103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1960004590 con 0x7f19641acea0 2021-04-09T18:21:54.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.555 7f196b204700 1 -- 172.21.15.32:0/1921241103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19641998d0 con 0x7f19641acea0 2021-04-09T18:21:54.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.556 7f1959ffb700 1 -- 172.21.15.32:0/1921241103 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f196000a000 con 0x7f19641acea0 2021-04-09T18:21:54.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.556 7f1959ffb700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19440188c0 0x7f194401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:54.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.556 7f1959ffb700 1 -- 172.21.15.32:0/1921241103 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f196001f4f0 con 0x7f19641acea0 2021-04-09T18:21:54.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.556 7f196881f700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19440188c0 0x7f194401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:54.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.557 7f196881f700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19440188c0 0x7f194401adc0 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:54.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.576 7f196b204700 1 -- 172.21.15.32:0/1921241103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1964184b30 con 0x7f19641acea0 2021-04-09T18:21:54.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.579 7f1959ffb700 1 -- 172.21.15.32:0/1921241103 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f196001f4c0 con 0x7f19641acea0 2021-04-09T18:21:54.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.773 7f196b204700 1 -- 172.21.15.32:0/1921241103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1964184b30 con 0x7f19641acea0 2021-04-09T18:21:54.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.773 7f1959ffb700 1 -- 172.21.15.32:0/1921241103 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2781 (secure 0 0 0) 0x7f196000a000 con 0x7f19641acea0 2021-04-09T18:21:54.825 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 108 2021-04-09T18:21:54.825 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:54.826 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":108,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":7575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:21:54.337303","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:21:54.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 -- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19440188c0 msgr2=0x7f194401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:54.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19440188c0 0x7f194401adc0 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 -- 172.21.15.32:0/1921241103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19641acea0 msgr2=0x7f19641ac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:54.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19641acea0 0x7f19641ac750 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f196000b530 tx=0x7f196000b8d0).stop 2021-04-09T18:21:54.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 -- 172.21.15.32:0/1921241103 shutdown_connections 2021-04-09T18:21:54.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19440188c0 0x7f194401adc0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1964198eb0 0x7f19641992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1964197ca0 0x7f19641980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 --2- 172.21.15.32:0/1921241103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19641acea0 0x7f19641ac750 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:54.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 -- 172.21.15.32:0/1921241103 >> 172.21.15.32:0/1921241103 conn(0x7f196419c7a0 msgr2=0x7f1964196a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:54.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 -- 172.21.15.32:0/1921241103 shutdown_connections 2021-04-09T18:21:54.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.776 7f196b204700 1 -- 172.21.15.32:0/1921241103 wait complete. 2021-04-09T18:21:54.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:21:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.935 7f9cf0401700 1 Processor -- start 2021-04-09T18:21:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.935 7f9cf0401700 1 -- start start 2021-04-09T18:21:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9cf0401700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce810d010 0x7f9ce80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9cf0401700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9ce80d1da0 0x7f9ce80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9cf0401700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce810d480 con 0x7f9ce80d82c0 2021-04-09T18:21:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9cf0401700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce810e780 con 0x7f9ce80d1da0 2021-04-09T18:21:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9cf0401700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80d8d30 con 0x7f9ce810d010 2021-04-09T18:21:55.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9ceda1c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9ce80d1da0 0x7f9ce80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9ceda1c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9ce80d1da0 0x7f9ce80d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48736/0 (socket says 172.21.15.32:48736) 2021-04-09T18:21:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9ceda1c700 1 -- 172.21.15.32:0/2647526756 learned_addr learned my addr 172.21.15.32:0/2647526756 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:55.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9cee21d700 1 --2- 172.21.15.32:0/2647526756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce810d010 0x7f9ce80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9ceda1c700 1 -- 172.21.15.32:0/2647526756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce810d010 msgr2=0x7f9ce80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:55.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9ceda1c700 1 --2- 172.21.15.32:0/2647526756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce810d010 0x7f9ce80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9ceda1c700 1 -- 172.21.15.32:0/2647526756 >> v1:172.21.15.86:6789/0 conn(0x7f9ce80d82c0 legacy=0x7f9ce810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9ceda1c700 1 -- 172.21.15.32:0/2647526756 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce80d9220 con 0x7f9ce80d1da0 2021-04-09T18:21:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.936 7f9ceda1c700 1 --2- 172.21.15.32:0/2647526756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce80d1da0 0x7f9ce80d7ca0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f9cd8006700 tx=0x7f9cd8009b80).ready entity=mon.0 client_cookie=5d1b242fcd617fd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:55.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.937 7f9ced21b700 1 -- 172.21.15.32:0/2647526756 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cd800c0c0 con 0x7f9ce80d1da0 2021-04-09T18:21:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.937 7f9ced21b700 1 -- 172.21.15.32:0/2647526756 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd800c440 con 0x7f9ce80d1da0 2021-04-09T18:21:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.937 7f9ced21b700 1 -- 172.21.15.32:0/2647526756 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cd800c920 con 0x7f9ce80d1da0 2021-04-09T18:21:55.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.937 7f9cf0401700 1 -- 172.21.15.32:0/2647526756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce80d1da0 msgr2=0x7f9ce80d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.937 7f9cf0401700 1 --2- 172.21.15.32:0/2647526756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce80d1da0 0x7f9ce80d7ca0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f9cd8006700 tx=0x7f9cd8009b80).stop 2021-04-09T18:21:55.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.937 7f9cf0401700 1 -- 172.21.15.32:0/2647526756 shutdown_connections 2021-04-09T18:21:55.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.937 7f9cf0401700 1 --2- 172.21.15.32:0/2647526756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce80d1da0 0x7f9ce80d7ca0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.938 7f9cf0401700 1 --2- 172.21.15.32:0/2647526756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce810d010 0x7f9ce80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.938 7f9cf0401700 1 -- 172.21.15.32:0/2647526756 >> 172.21.15.32:0/2647526756 conn(0x7f9ce80ce5f0 msgr2=0x7f9ce80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:55.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.938 7f9cf0401700 1 -- 172.21.15.32:0/2647526756 shutdown_connections 2021-04-09T18:21:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.938 7f9cf0401700 1 -- 172.21.15.32:0/2647526756 wait complete. 2021-04-09T18:21:55.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9cf0401700 1 Processor -- start 2021-04-09T18:21:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9cf0401700 1 -- start start 2021-04-09T18:21:55.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9cf0401700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce81a5210 0x7f9ce808eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9cf0401700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce81a8ab0 0x7f9ce81a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9cf0401700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce81a76d0 0x7f9ce81a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9cf0401700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce817ea50 con 0x7f9ce81a76d0 2021-04-09T18:21:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9cf0401700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce808f4f0 con 0x7f9ce81a5210 2021-04-09T18:21:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9cf0401700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce808ea60 con 0x7f9ce81a8ab0 2021-04-09T18:21:55.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9ceea1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce81a76d0 0x7f9ce81a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9ceea1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce81a76d0 0x7f9ce81a7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48744/0 (socket says 172.21.15.32:48744) 2021-04-09T18:21:55.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9ceea1e700 1 -- 172.21.15.32:0/4099281313 learned_addr learned my addr 172.21.15.32:0/4099281313 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9ceda1c700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce81a8ab0 0x7f9ce81a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.958 7f9cee21d700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce81a5210 0x7f9ce808eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9ceea1e700 1 -- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce81a8ab0 msgr2=0x7f9ce81a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9ceea1e700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce81a8ab0 0x7f9ce81a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9ceea1e700 1 -- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce81a5210 msgr2=0x7f9ce808eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9ceea1e700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce81a5210 0x7f9ce808eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9ceea1e700 1 -- 172.21.15.32:0/4099281313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce81a80f0 con 0x7f9ce81a76d0 2021-04-09T18:21:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9cee21d700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce81a5210 0x7f9ce808eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9ceea1e700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce81a76d0 0x7f9ce81a7ad0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7f9ce0006ed0 tx=0x7f9ce000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:55.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9cdeffd700 1 -- 172.21.15.32:0/4099281313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ce000bd50 con 0x7f9ce81a76d0 2021-04-09T18:21:55.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9cdeffd700 1 -- 172.21.15.32:0/4099281313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce0006020 con 0x7f9ce81a76d0 2021-04-09T18:21:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce81a80f0 con 0x7f9ce81a76d0 2021-04-09T18:21:55.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9cdeffd700 1 -- 172.21.15.32:0/4099281313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ce00054b0 con 0x7f9ce81a76d0 2021-04-09T18:21:55.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.959 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce81a80f0 con 0x7f9ce81a76d0 2021-04-09T18:21:55.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.960 7f9cdeffd700 1 -- 172.21.15.32:0/4099281313 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9ce0006200 con 0x7f9ce81a76d0 2021-04-09T18:21:55.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.960 7f9cdeffd700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9cd00188c0 0x7f9cd001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.960 7f9cdeffd700 1 -- 172.21.15.32:0/4099281313 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9ce0032790 con 0x7f9ce81a76d0 2021-04-09T18:21:55.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.960 7f9cee21d700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9cd00188c0 0x7f9cd001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.960 7f9cee21d700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9cd00188c0 0x7f9cd001adc0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.980 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce81b7070 con 0x7f9ce81a76d0 2021-04-09T18:21:55.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:54.983 7f9cdeffd700 1 -- 172.21.15.32:0/4099281313 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9ce00054b0 con 0x7f9ce81a76d0 2021-04-09T18:21:55.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.170 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ce810d480 con 0x7f9ce81a76d0 2021-04-09T18:21:55.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.171 7f9cdeffd700 1 -- 172.21.15.32:0/4099281313 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f9ce0006200 con 0x7f9ce81a76d0 2021-04-09T18:21:55.223 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:55.223 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":95,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:21:53.333784","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:21:50.420671","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-04-09 18:21:51.790105","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:21:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9cd00188c0 msgr2=0x7f9cd001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9cd00188c0 0x7f9cd001adc0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce81a76d0 msgr2=0x7f9ce81a7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce81a76d0 0x7f9ce81a7ad0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7f9ce0006ed0 tx=0x7f9ce000b960).stop 2021-04-09T18:21:55.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 shutdown_connections 2021-04-09T18:21:55.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9cd00188c0 0x7f9cd001adc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ce81a76d0 0x7f9ce81a7ad0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ce81a8ab0 0x7f9ce81a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 --2- 172.21.15.32:0/4099281313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce81a5210 0x7f9ce808eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 >> 172.21.15.32:0/4099281313 conn(0x7f9ce81a99c0 msgr2=0x7f9ce817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:55.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 shutdown_connections 2021-04-09T18:21:55.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.173 7f9cf0401700 1 -- 172.21.15.32:0/4099281313 wait complete. 2021-04-09T18:21:55.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:21:55.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.345 7ffa1293b700 1 Processor -- start 2021-04-09T18:21:55.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.345 7ffa1293b700 1 -- start start 2021-04-09T18:21:55.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.345 7ffa1293b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffa0c0cffb0 0x7ffa0c087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.345 7ffa1293b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0c088460 con 0x7ffa0c0d8050 2021-04-09T18:21:55.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.345 7ffa1293b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0c0d8620 con 0x7ffa0c0d7c50 2021-04-09T18:21:55.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.345 7ffa1293b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa0c1b0c80 con 0x7ffa0c0cffb0 2021-04-09T18:21:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.345 7ffa0bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffa0c0cffb0 0x7ffa0c087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa10f58700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffa0c0d8050 0x7ffa0c10d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58342/0 (socket says 172.21.15.32:58342) 2021-04-09T18:21:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa10f58700 1 -- 172.21.15.32:0/2142481440 learned_addr learned my addr 172.21.15.32:0/2142481440 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa03fff700 1 --1- 172.21.15.32:0/2142481440 >> v1:172.21.15.86:6789/0 conn(0x7ffa0c0d7c50 0x7ffa0c1b0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:56266/0 (socket says 172.21.15.32:56266) 2021-04-09T18:21:55.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0b7fe700 1 -- 172.21.15.32:0/2142481440 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252138680 0 0) 0x7ffa04000f90 con 0x7ffa0c0d8050 2021-04-09T18:21:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0b7fe700 1 -- 172.21.15.32:0/2142481440 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f8002fe0 con 0x7ffa0c0d8050 2021-04-09T18:21:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0b7fe700 1 -- 172.21.15.32:0/2142481440 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476068809 0 0) 0x7ff9f4000c60 con 0x7ffa0c0d7c50 2021-04-09T18:21:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0bfff700 1 -- 172.21.15.32:0/2142481440 >> v1:172.21.15.86:6789/0 conn(0x7ffa0c0d7c50 legacy=0x7ffa0c1b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0bfff700 1 -- 172.21.15.32:0/2142481440 >> v1:172.21.15.32:6789/0 conn(0x7ffa0c0d8050 legacy=0x7ffa0c10d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0bfff700 1 -- 172.21.15.32:0/2142481440 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c0d91f0 con 0x7ffa0c0cffb0 2021-04-09T18:21:55.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0bfff700 1 --2- 172.21.15.32:0/2142481440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c0cffb0 0x7ffa0c087e40 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7ff9fc006650 tx=0x7ff9fc009a80).ready entity=mon.0 client_cookie=6946d31c2c820e53 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:55.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0b7fe700 1 -- 172.21.15.32:0/2142481440 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9fc0009b0 con 0x7ffa0c0cffb0 2021-04-09T18:21:55.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0b7fe700 1 -- 172.21.15.32:0/2142481440 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9fc00c0c0 con 0x7ffa0c0cffb0 2021-04-09T18:21:55.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa0b7fe700 1 -- 172.21.15.32:0/2142481440 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9fc00c510 con 0x7ffa0c0cffb0 2021-04-09T18:21:55.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa1293b700 1 -- 172.21.15.32:0/2142481440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c0cffb0 msgr2=0x7ffa0c087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.346 7ffa1293b700 1 --2- 172.21.15.32:0/2142481440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c0cffb0 0x7ffa0c087e40 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7ff9fc006650 tx=0x7ff9fc009a80).stop 2021-04-09T18:21:55.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.347 7ffa1293b700 1 -- 172.21.15.32:0/2142481440 shutdown_connections 2021-04-09T18:21:55.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.347 7ffa1293b700 1 --2- 172.21.15.32:0/2142481440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c0cffb0 0x7ffa0c087e40 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.347 7ffa1293b700 1 -- 172.21.15.32:0/2142481440 >> 172.21.15.32:0/2142481440 conn(0x7ffa0c0ce5b0 msgr2=0x7ffa0c0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:55.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.347 7ffa1293b700 1 -- 172.21.15.32:0/2142481440 shutdown_connections 2021-04-09T18:21:55.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.347 7ffa1293b700 1 -- 172.21.15.32:0/2142481440 wait complete. 2021-04-09T18:21:55.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.367 7ffa1293b700 1 Processor -- start 2021-04-09T18:21:55.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.367 7ffa1293b700 1 -- start start 2021-04-09T18:21:55.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa1293b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa0c0b1950 0x7ffa0c183e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa1293b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa0c1795e0 0x7ffa0c094eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa1293b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c1799e0 0x7ffa0c0954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa1293b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c0b13d0 con 0x7ffa0c1799e0 2021-04-09T18:21:55.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa1293b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c0b1d50 con 0x7ffa0c1795e0 2021-04-09T18:21:55.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa1293b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c19e4a0 con 0x7ffa0c0b1950 2021-04-09T18:21:55.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa10f58700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c1799e0 0x7ffa0c0954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa10f58700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c1799e0 0x7ffa0c0954d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48758/0 (socket says 172.21.15.32:48758) 2021-04-09T18:21:55.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa10f58700 1 -- 172.21.15.32:0/407172008 learned_addr learned my addr 172.21.15.32:0/407172008 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:55.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa0bfff700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa0c0b1950 0x7ffa0c183e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa03fff700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa0c1795e0 0x7ffa0c094eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa10f58700 1 -- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa0c0b1950 msgr2=0x7ffa0c183e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa10f58700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa0c0b1950 0x7ffa0c183e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa10f58700 1 -- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa0c1795e0 msgr2=0x7ffa0c094eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa10f58700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa0c1795e0 0x7ffa0c094eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa10f58700 1 -- 172.21.15.32:0/407172008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c0b0180 con 0x7ffa0c1799e0 2021-04-09T18:21:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.368 7ffa10f58700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c1799e0 0x7ffa0c0954d0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7ffa04009980 tx=0x7ffa0400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.369 7ffa097fa700 1 -- 172.21.15.32:0/407172008 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa0400be10 con 0x7ffa0c1799e0 2021-04-09T18:21:55.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.369 7ffa097fa700 1 -- 172.21.15.32:0/407172008 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa04006030 con 0x7ffa0c1799e0 2021-04-09T18:21:55.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.369 7ffa1293b700 1 -- 172.21.15.32:0/407172008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa0c0b0180 con 0x7ffa0c1799e0 2021-04-09T18:21:55.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.369 7ffa097fa700 1 -- 172.21.15.32:0/407172008 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa040064c0 con 0x7ffa0c1799e0 2021-04-09T18:21:55.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.369 7ffa1293b700 1 -- 172.21.15.32:0/407172008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0c0b0180 con 0x7ffa0c1799e0 2021-04-09T18:21:55.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.369 7ffa097fa700 1 -- 172.21.15.32:0/407172008 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ffa04000d30 con 0x7ffa0c1799e0 2021-04-09T18:21:55.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.369 7ffa097fa700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9f0018930 0x7ff9f001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.369 7ffa097fa700 1 -- 172.21.15.32:0/407172008 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ffa04032d70 con 0x7ffa0c1799e0 2021-04-09T18:21:55.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.370 7ffa0bfff700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9f0018930 0x7ff9f001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.370 7ffa0bfff700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9f0018930 0x7ff9f001adc0 crc :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:55.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.389 7ffa1293b700 1 -- 172.21.15.32:0/407172008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0c093fc0 con 0x7ffa0c1799e0 2021-04-09T18:21:55.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.393 7ffa097fa700 1 -- 172.21.15.32:0/407172008 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ffa04000f40 con 0x7ffa0c1799e0 2021-04-09T18:21:55.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.579 7ffa1293b700 1 -- 172.21.15.32:0/407172008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ffa0c0d8620 con 0x7ffa0c1799e0 2021-04-09T18:21:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.582 7ffa097fa700 1 -- 172.21.15.32:0/407172008 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v26) v1 ==== 184+0+0 (secure 0 0 0) 0x7ffa04000d30 con 0x7ffa0c1799e0 2021-04-09T18:21:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:21:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 -- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9f0018930 msgr2=0x7ff9f001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9f0018930 0x7ff9f001adc0 crc :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 -- 172.21.15.32:0/407172008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c1799e0 msgr2=0x7ffa0c0954d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c1799e0 0x7ffa0c0954d0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7ffa04009980 tx=0x7ffa0400bcd0).stop 2021-04-09T18:21:55.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 -- 172.21.15.32:0/407172008 shutdown_connections 2021-04-09T18:21:55.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff9f0018930 0x7ff9f001adc0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa0c1799e0 0x7ffa0c0954d0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa0c1795e0 0x7ffa0c094eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 --2- 172.21.15.32:0/407172008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa0c0b1950 0x7ffa0c183e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 -- 172.21.15.32:0/407172008 >> 172.21.15.32:0/407172008 conn(0x7ffa0c1889b0 msgr2=0x7ffa0c189210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 -- 172.21.15.32:0/407172008 shutdown_connections 2021-04-09T18:21:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.584 7ffa1293b700 1 -- 172.21.15.32:0/407172008 wait complete. 2021-04-09T18:21:55.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:21:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.756 7f475aaf4700 1 Processor -- start 2021-04-09T18:21:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.756 7f475aaf4700 1 -- start start 2021-04-09T18:21:55.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f475aaf4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47540cffb0 0x7f4754087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f475aaf4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47540d8050 0x7f475410d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f475aaf4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4754088460 con 0x7f47540d7c50 2021-04-09T18:21:55.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f475aaf4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47541b0c80 con 0x7f47540d8050 2021-04-09T18:21:55.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f475aaf4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47540d91f0 con 0x7f47540cffb0 2021-04-09T18:21:55.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f4759111700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47540d8050 0x7f475410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f4753fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f47540d7c50 0x7f47541b0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58356/0 (socket says 172.21.15.32:58356) 2021-04-09T18:21:55.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f4753fff700 1 -- 172.21.15.32:0/664631656 learned_addr learned my addr 172.21.15.32:0/664631656 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:55.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f4758910700 1 --2- 172.21.15.32:0/664631656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47540cffb0 0x7f4754087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f4759111700 1 -- 172.21.15.32:0/664631656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47540cffb0 msgr2=0x7f4754087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:55.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f4759111700 1 --2- 172.21.15.32:0/664631656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47540cffb0 0x7f4754087e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f4759111700 1 -- 172.21.15.32:0/664631656 >> v1:172.21.15.32:6789/0 conn(0x7f47540d7c50 legacy=0x7f47541b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f4759111700 1 -- 172.21.15.32:0/664631656 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47540d8620 con 0x7f47540d8050 2021-04-09T18:21:55.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f47537fe700 1 -- 172.21.15.32:0/664631656 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606988239 0 0) 0x7f4748000ce0 con 0x7f47540d7c50 2021-04-09T18:21:55.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.757 7f4759111700 1 --2- 172.21.15.32:0/664631656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540d8050 0x7f475410d200 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f474c006e30 tx=0x7f474c009e60).ready entity=mon.0 client_cookie=b3b12c8a1b044366 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:55.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f47537fe700 1 -- 172.21.15.32:0/664631656 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f474c006a50 con 0x7f47540d8050 2021-04-09T18:21:55.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f47537fe700 1 -- 172.21.15.32:0/664631656 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c00b300 con 0x7f47540d8050 2021-04-09T18:21:55.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f47537fe700 1 -- 172.21.15.32:0/664631656 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f474c00b750 con 0x7f47540d8050 2021-04-09T18:21:55.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f475aaf4700 1 -- 172.21.15.32:0/664631656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540d8050 msgr2=0x7f475410d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f475aaf4700 1 --2- 172.21.15.32:0/664631656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540d8050 0x7f475410d200 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f474c006e30 tx=0x7f474c009e60).stop 2021-04-09T18:21:55.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f475aaf4700 1 -- 172.21.15.32:0/664631656 shutdown_connections 2021-04-09T18:21:55.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f475aaf4700 1 --2- 172.21.15.32:0/664631656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540d8050 0x7f475410d200 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f475aaf4700 1 --2- 172.21.15.32:0/664631656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47540cffb0 0x7f4754087e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f475aaf4700 1 -- 172.21.15.32:0/664631656 >> 172.21.15.32:0/664631656 conn(0x7f47540ce5b0 msgr2=0x7f47540cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:55.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f475aaf4700 1 -- 172.21.15.32:0/664631656 shutdown_connections 2021-04-09T18:21:55.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.758 7f475aaf4700 1 -- 172.21.15.32:0/664631656 wait complete. 2021-04-09T18:21:55.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.778 7f475aaf4700 1 Processor -- start 2021-04-09T18:21:55.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f475aaf4700 1 -- start start 2021-04-09T18:21:55.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f475aaf4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540c4080 0x7f47540958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f475aaf4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4754094f20 0x7f4754182540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f475aaf4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4754095320 0x7f4754182b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f475aaf4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47541832c0 con 0x7f47540c4080 2021-04-09T18:21:55.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f475aaf4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4754181fc0 con 0x7f4754095320 2021-04-09T18:21:55.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f475aaf4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47540c4480 con 0x7f4754094f20 2021-04-09T18:21:55.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f4758910700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540c4080 0x7f47540958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f4758910700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540c4080 0x7f47540958d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48768/0 (socket says 172.21.15.32:48768) 2021-04-09T18:21:55.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.779 7f4758910700 1 -- 172.21.15.32:0/132473650 learned_addr learned my addr 172.21.15.32:0/132473650 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:55.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f4753fff700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4754094f20 0x7f4754182540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f4759111700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4754095320 0x7f4754182b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f4758910700 1 -- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4754094f20 msgr2=0x7f4754182540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f4758910700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4754094f20 0x7f4754182540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f4758910700 1 -- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4754095320 msgr2=0x7f4754182b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f4758910700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4754095320 0x7f4754182b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f4758910700 1 -- 172.21.15.32:0/132473650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47540bc4a0 con 0x7f47540c4080 2021-04-09T18:21:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f4758910700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540c4080 0x7f47540958d0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f474400b580 tx=0x7f474400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f47517fa700 1 -- 172.21.15.32:0/132473650 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47440064d0 con 0x7f47540c4080 2021-04-09T18:21:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f47517fa700 1 -- 172.21.15.32:0/132473650 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474400bd10 con 0x7f47540c4080 2021-04-09T18:21:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f47517fa700 1 -- 172.21.15.32:0/132473650 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4744002030 con 0x7f47540c4080 2021-04-09T18:21:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f475aaf4700 1 -- 172.21.15.32:0/132473650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47540bc4a0 con 0x7f47540c4080 2021-04-09T18:21:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.780 7f475aaf4700 1 -- 172.21.15.32:0/132473650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47540bc4a0 con 0x7f47540c4080 2021-04-09T18:21:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.781 7f47517fa700 1 -- 172.21.15.32:0/132473650 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f47440052e0 con 0x7f47540c4080 2021-04-09T18:21:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.781 7f47517fa700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47340188d0 0x7f473401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:55.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.781 7f47517fa700 1 -- 172.21.15.32:0/132473650 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f47440329d0 con 0x7f47540c4080 2021-04-09T18:21:55.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.781 7f4753fff700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47340188d0 0x7f473401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:55.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.781 7f4753fff700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47340188d0 0x7f473401add0 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:55.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.801 7f475aaf4700 1 -- 172.21.15.32:0/132473650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4754093ff0 con 0x7f47540c4080 2021-04-09T18:21:55.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.804 7f47517fa700 1 -- 172.21.15.32:0/132473650 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4744006820 con 0x7f47540c4080 2021-04-09T18:21:56.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.990 7f475aaf4700 1 -- 172.21.15.32:0/132473650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4754093ff0 con 0x7f47540c4080 2021-04-09T18:21:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:55.993 7f47517fa700 1 -- 172.21.15.32:0/132473650 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v27) v1 ==== 184+0+0 (secure 0 0 0) 0x7f474401f710 con 0x7f47540c4080 2021-04-09T18:21:56.049 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:21:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 -- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47340188d0 msgr2=0x7f473401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47340188d0 0x7f473401add0 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 -- 172.21.15.32:0/132473650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540c4080 msgr2=0x7f47540958d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540c4080 0x7f47540958d0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f474400b580 tx=0x7f474400b920).stop 2021-04-09T18:21:56.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 -- 172.21.15.32:0/132473650 shutdown_connections 2021-04-09T18:21:56.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47340188d0 0x7f473401add0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4754095320 0x7f4754182b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4754094f20 0x7f4754182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 --2- 172.21.15.32:0/132473650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47540c4080 0x7f47540958d0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 -- 172.21.15.32:0/132473650 >> 172.21.15.32:0/132473650 conn(0x7f4754099610 msgr2=0x7f47540acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 -- 172.21.15.32:0/132473650 shutdown_connections 2021-04-09T18:21:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.000 7f475aaf4700 1 -- 172.21.15.32:0/132473650 wait complete. 2021-04-09T18:21:56.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:56.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.162 7fb4a30a5700 1 Processor -- start 2021-04-09T18:21:56.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.162 7fb4a30a5700 1 -- start start 2021-04-09T18:21:56.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.163 7fb4a30a5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb49c10da00 0x7fb49c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:56.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.163 7fb4a30a5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c10d480 con 0x7fb49c0d7ca0 2021-04-09T18:21:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.163 7fb4a30a5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c0d1f70 con 0x7fb49c10d010 2021-04-09T18:21:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.163 7fb4a30a5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c0d8d30 con 0x7fb49c10da00 2021-04-09T18:21:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.163 7fb49bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb49c0d7ca0 0x7fb49c0d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58366/0 (socket says 172.21.15.32:58366) 2021-04-09T18:21:56.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.163 7fb49bfff700 1 -- 172.21.15.32:0/3721055383 learned_addr learned my addr 172.21.15.32:0/3721055383 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:56.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.163 7fb4a16c2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb49c10da00 0x7fb49c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:56.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb49b7fe700 1 -- 172.21.15.32:0/3721055383 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729995468 0 0) 0x7fb484000c60 con 0x7fb49c10d010 2021-04-09T18:21:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb49b7fe700 1 -- 172.21.15.32:0/3721055383 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb488002fe0 con 0x7fb49c10d010 2021-04-09T18:21:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb49b7fe700 1 -- 172.21.15.32:0/3721055383 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734290917 0 0) 0x7fb48c001060 con 0x7fb49c0d7ca0 2021-04-09T18:21:56.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb49b7fe700 1 -- 172.21.15.32:0/3721055383 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb488003020 con 0x7fb49c0d7ca0 2021-04-09T18:21:56.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb4a16c2700 1 -- 172.21.15.32:0/3721055383 >> v1:172.21.15.86:6789/0 conn(0x7fb49c10d010 legacy=0x7fb49c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:56.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb4a16c2700 1 -- 172.21.15.32:0/3721055383 >> v1:172.21.15.32:6789/0 conn(0x7fb49c0d7ca0 legacy=0x7fb49c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb4a16c2700 1 -- 172.21.15.32:0/3721055383 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c0d9220 con 0x7fb49c10da00 2021-04-09T18:21:56.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb4a16c2700 1 --2- 172.21.15.32:0/3721055383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb49c10da00 0x7fb49c10de00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fb494006ad0 tx=0x7fb494009a50).ready entity=mon.1 client_cookie=635ccdc2b61fafe3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:56.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb49b7fe700 1 -- 172.21.15.32:0/3721055383 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4940066a0 con 0x7fb49c10da00 2021-04-09T18:21:56.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb49b7fe700 1 -- 172.21.15.32:0/3721055383 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb494006410 con 0x7fb49c10da00 2021-04-09T18:21:56.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb49b7fe700 1 -- 172.21.15.32:0/3721055383 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4940024d0 con 0x7fb49c10da00 2021-04-09T18:21:56.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb4a30a5700 1 -- 172.21.15.32:0/3721055383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb49c10da00 msgr2=0x7fb49c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:56.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.164 7fb4a30a5700 1 --2- 172.21.15.32:0/3721055383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb49c10da00 0x7fb49c10de00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fb494006ad0 tx=0x7fb494009a50).stop 2021-04-09T18:21:56.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.165 7fb4a30a5700 1 -- 172.21.15.32:0/3721055383 shutdown_connections 2021-04-09T18:21:56.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.165 7fb4a30a5700 1 --2- 172.21.15.32:0/3721055383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb49c10da00 0x7fb49c10de00 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.165 7fb4a30a5700 1 -- 172.21.15.32:0/3721055383 >> 172.21.15.32:0/3721055383 conn(0x7fb49c0ce5f0 msgr2=0x7fb49c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:56.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.165 7fb4a30a5700 1 -- 172.21.15.32:0/3721055383 shutdown_connections 2021-04-09T18:21:56.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.165 7fb4a30a5700 1 -- 172.21.15.32:0/3721055383 wait complete. 2021-04-09T18:21:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.185 7fb4a30a5700 1 Processor -- start 2021-04-09T18:21:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.185 7fb4a30a5700 1 -- start start 2021-04-09T18:21:56.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a30a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb49c0a7f50 0x7fb49c180be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a30a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb49c181200 0x7fb49c08ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a30a5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb49c08b3b0 0x7fb49c1a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a30a5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c08a890 con 0x7fb49c181200 2021-04-09T18:21:56.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a30a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c181640 con 0x7fb49c0a7f50 2021-04-09T18:21:56.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a30a5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c1819f0 con 0x7fb49c08b3b0 2021-04-09T18:21:56.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb49bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb49c181200 0x7fb49c08ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:56.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb49bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb49c181200 0x7fb49c08ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48782/0 (socket says 172.21.15.32:48782) 2021-04-09T18:21:56.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb49bfff700 1 -- 172.21.15.32:0/3242730171 learned_addr learned my addr 172.21.15.32:0/3242730171 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:56.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a0ec1700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb49c0a7f50 0x7fb49c180be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:56.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a16c2700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb49c08b3b0 0x7fb49c1a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:56.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb49bfff700 1 -- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb49c08b3b0 msgr2=0x7fb49c1a32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb49bfff700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb49c08b3b0 0x7fb49c1a32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb49bfff700 1 -- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb49c0a7f50 msgr2=0x7fb49c180be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb49bfff700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb49c0a7f50 0x7fb49c180be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb49bfff700 1 -- 172.21.15.32:0/3242730171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c08b7b0 con 0x7fb49c181200 2021-04-09T18:21:56.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a16c2700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb49c08b3b0 0x7fb49c1a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:21:56.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb49bfff700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb49c181200 0x7fb49c08ad90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7fb48c009960 tx=0x7fb48c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:56.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4997fa700 1 -- 172.21.15.32:0/3242730171 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb48c0062b0 con 0x7fb49c181200 2021-04-09T18:21:56.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4997fa700 1 -- 172.21.15.32:0/3242730171 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c004720 con 0x7fb49c181200 2021-04-09T18:21:56.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c08b7b0 con 0x7fb49c181200 2021-04-09T18:21:56.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4997fa700 1 -- 172.21.15.32:0/3242730171 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb48c004b70 con 0x7fb49c181200 2021-04-09T18:21:56.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.186 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c08b7b0 con 0x7fb49c181200 2021-04-09T18:21:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.187 7fb4997fa700 1 -- 172.21.15.32:0/3242730171 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb48c006570 con 0x7fb49c181200 2021-04-09T18:21:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.187 7fb4997fa700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb480018930 0x7fb48001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:56.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.187 7fb4997fa700 1 -- 172.21.15.32:0/3242730171 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb48c032db0 con 0x7fb49c181200 2021-04-09T18:21:56.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.187 7fb4a0ec1700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb480018930 0x7fb48001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:56.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.188 7fb4a0ec1700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb480018930 0x7fb48001ae10 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:56.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.207 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c1b7070 con 0x7fb49c181200 2021-04-09T18:21:56.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.211 7fb4997fa700 1 -- 172.21.15.32:0/3242730171 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb48c00be30 con 0x7fb49c181200 2021-04-09T18:21:56.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.405 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb49c1b7070 con 0x7fb49c181200 2021-04-09T18:21:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.405 7fb4997fa700 1 -- 172.21.15.32:0/3242730171 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2781 (secure 0 0 0) 0x7fb48c004720 con 0x7fb49c181200 2021-04-09T18:21:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 108 2021-04-09T18:21:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:56.458 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":108,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":7575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:21:54.337303","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:21:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb480018930 msgr2=0x7fb48001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb480018930 0x7fb48001ae10 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb49c181200 msgr2=0x7fb49c08ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb49c181200 0x7fb49c08ad90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7fb48c009960 tx=0x7fb48c00bcf0).stop 2021-04-09T18:21:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 shutdown_connections 2021-04-09T18:21:56.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb480018930 0x7fb48001ae10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb49c181200 0x7fb49c08ad90 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb49c08b3b0 0x7fb49c1a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 --2- 172.21.15.32:0/3242730171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb49c0a7f50 0x7fb49c180be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:56.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 >> 172.21.15.32:0/3242730171 conn(0x7fb49c1aa2b0 msgr2=0x7fb49c1a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:56.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 shutdown_connections 2021-04-09T18:21:56.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:56.408 7fb4a30a5700 1 -- 172.21.15.32:0/3242730171 wait complete. 2021-04-09T18:21:56.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:21:54.337303', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7387}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:56.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:21:56.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:21:57.002 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:21:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:21:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:21:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 7387, 2021-04-09T18:21:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:21:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:21:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.618502189, 2021-04-09T18:21:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 108, 2021-04-09T18:21:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 95, 2021-04-09T18:21:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 95, 2021-04-09T18:21:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 9.0580962710000001 2021-04-09T18:21:57.005 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:21:57.008 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 7387, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.618502189, "mdsmap_epoch": 108, "osdmap_epoch": 95, "osdmap_epoch_barrier": 95, "uptime": 9.0580962710000001 } 2021-04-09T18:21:57.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:21:57.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.112 7f6bd451e700 1 Processor -- start 2021-04-09T18:21:57.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.112 7f6bd451e700 1 -- start start 2021-04-09T18:21:57.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd451e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6bcc10d010 0x7f6bcc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd451e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6bcc0d1da0 0x7f6bcc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd451e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc0d82c0 0x7f6bcc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd451e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bcc0d8d30 con 0x7f6bcc0d1da0 2021-04-09T18:21:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd451e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bcc0d9220 con 0x7f6bcc10d010 2021-04-09T18:21:57.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd451e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc10d4b0 con 0x7f6bcc0d82c0 2021-04-09T18:21:57.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd1b39700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6bcc0d1da0 0x7f6bcc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd1b39700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6bcc0d1da0 0x7f6bcc0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48792/0 (socket says 172.21.15.32:48792) 2021-04-09T18:21:57.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd1b39700 1 -- 172.21.15.32:0/2354578491 learned_addr learned my addr 172.21.15.32:0/2354578491 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:57.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd2b3b700 1 --2- 172.21.15.32:0/2354578491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc0d82c0 0x7f6bcc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.113 7f6bd233a700 1 --2- 172.21.15.32:0/2354578491 >> v2:172.21.15.86:3300/0 conn(0x7f6bcc10d010 0x7f6bcc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd1b39700 1 -- 172.21.15.32:0/2354578491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc0d82c0 msgr2=0x7f6bcc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:57.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd1b39700 1 --2- 172.21.15.32:0/2354578491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc0d82c0 0x7f6bcc10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd1b39700 1 -- 172.21.15.32:0/2354578491 >> v2:172.21.15.86:3300/0 conn(0x7f6bcc10d010 msgr2=0x7f6bcc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:57.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd1b39700 1 --2- 172.21.15.32:0/2354578491 >> v2:172.21.15.86:3300/0 conn(0x7f6bcc10d010 0x7f6bcc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd1b39700 1 -- 172.21.15.32:0/2354578491 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0d86c0 con 0x7f6bcc0d1da0 2021-04-09T18:21:57.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd1b39700 1 --2- 172.21.15.32:0/2354578491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc0d1da0 0x7f6bcc0d7ca0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f6bbc006700 tx=0x7f6bbc009b00).ready entity=mon.0 client_cookie=98c7ba3c5e0b333f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:57.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd1338700 1 -- 172.21.15.32:0/2354578491 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bbc00c0c0 con 0x7f6bcc0d1da0 2021-04-09T18:21:57.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd1338700 1 -- 172.21.15.32:0/2354578491 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc00c400 con 0x7f6bcc0d1da0 2021-04-09T18:21:57.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd1338700 1 -- 172.21.15.32:0/2354578491 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bbc00c8e0 con 0x7f6bcc0d1da0 2021-04-09T18:21:57.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd451e700 1 -- 172.21.15.32:0/2354578491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc0d1da0 msgr2=0x7f6bcc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.114 7f6bd451e700 1 --2- 172.21.15.32:0/2354578491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc0d1da0 0x7f6bcc0d7ca0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f6bbc006700 tx=0x7f6bbc009b00).stop 2021-04-09T18:21:57.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.115 7f6bd451e700 1 -- 172.21.15.32:0/2354578491 shutdown_connections 2021-04-09T18:21:57.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.115 7f6bd451e700 1 --2- 172.21.15.32:0/2354578491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc0d82c0 0x7f6bcc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.115 7f6bd451e700 1 --2- 172.21.15.32:0/2354578491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc0d1da0 0x7f6bcc0d7ca0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.115 7f6bd451e700 1 --2- 172.21.15.32:0/2354578491 >> v2:172.21.15.86:3300/0 conn(0x7f6bcc10d010 0x7f6bcc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.115 7f6bd451e700 1 -- 172.21.15.32:0/2354578491 >> 172.21.15.32:0/2354578491 conn(0x7f6bcc0ce5f0 msgr2=0x7f6bcc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:57.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.115 7f6bd451e700 1 -- 172.21.15.32:0/2354578491 shutdown_connections 2021-04-09T18:21:57.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.115 7f6bd451e700 1 -- 172.21.15.32:0/2354578491 wait complete. 2021-04-09T18:21:57.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.135 7f6bd451e700 1 Processor -- start 2021-04-09T18:21:57.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.135 7f6bd451e700 1 -- start start 2021-04-09T18:21:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd451e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bcc0b8480 0x7f6bcc0b8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd451e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc18d800 0x7f6bcc18dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd451e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc1a9500 0x7f6bcc1a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd451e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc18d280 con 0x7f6bcc1a9500 2021-04-09T18:21:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd451e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc18e220 con 0x7f6bcc0b8480 2021-04-09T18:21:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd451e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc1a9f20 con 0x7f6bcc18d800 2021-04-09T18:21:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd2b3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc1a9500 0x7f6bcc1a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd2b3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc1a9500 0x7f6bcc1a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48794/0 (socket says 172.21.15.32:48794) 2021-04-09T18:21:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd2b3b700 1 -- 172.21.15.32:0/4263899039 learned_addr learned my addr 172.21.15.32:0/4263899039 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd233a700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bcc0b8480 0x7f6bcc0b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd1b39700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc18d800 0x7f6bcc18dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd2b3b700 1 -- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc18d800 msgr2=0x7f6bcc18dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd2b3b700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc18d800 0x7f6bcc18dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd2b3b700 1 -- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bcc0b8480 msgr2=0x7f6bcc0b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd2b3b700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bcc0b8480 0x7f6bcc0b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd2b3b700 1 -- 172.21.15.32:0/4263899039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc1aa140 con 0x7f6bcc1a9500 2021-04-09T18:21:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd1b39700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc18d800 0x7f6bcc18dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.136 7f6bd2b3b700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc1a9500 0x7f6bcc1a9900 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7f6bc400b5e0 tx=0x7f6bc400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.137 7f6bc2ffd700 1 -- 172.21.15.32:0/4263899039 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bc400bd70 con 0x7f6bcc1a9500 2021-04-09T18:21:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.137 7f6bc2ffd700 1 -- 172.21.15.32:0/4263899039 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc4006720 con 0x7f6bcc1a9500 2021-04-09T18:21:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.137 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bcc1aa140 con 0x7f6bcc1a9500 2021-04-09T18:21:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.137 7f6bc2ffd700 1 -- 172.21.15.32:0/4263899039 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bc4002220 con 0x7f6bcc1a9500 2021-04-09T18:21:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.137 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bcc1aa140 con 0x7f6bcc1a9500 2021-04-09T18:21:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.137 7f6bc2ffd700 1 -- 172.21.15.32:0/4263899039 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6bc4002020 con 0x7f6bcc1a9500 2021-04-09T18:21:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.137 7f6bc2ffd700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bb4018930 0x7f6bb401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.137 7f6bc2ffd700 1 -- 172.21.15.32:0/4263899039 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6bc40329f0 con 0x7f6bcc1a9500 2021-04-09T18:21:57.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.138 7f6bd233a700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bb4018930 0x7f6bb401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.138 7f6bd233a700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bb4018930 0x7f6bb401ae10 crc :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.157 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bcc18e220 con 0x7f6bcc1a9500 2021-04-09T18:21:57.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.161 7f6bc2ffd700 1 -- 172.21.15.32:0/4263899039 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6bc4006370 con 0x7f6bcc1a9500 2021-04-09T18:21:57.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.352 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bcc18e220 con 0x7f6bcc1a9500 2021-04-09T18:21:57.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.352 7f6bc2ffd700 1 -- 172.21.15.32:0/4263899039 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f6bc4006720 con 0x7f6bcc1a9500 2021-04-09T18:21:57.404 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:57.405 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":95,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:21:53.333784","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:21:50.420671","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-04-09 18:21:51.790105","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:21:57.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bb4018930 msgr2=0x7f6bb401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bb4018930 0x7f6bb401ae10 crc :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc1a9500 msgr2=0x7f6bcc1a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc1a9500 0x7f6bcc1a9900 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7f6bc400b5e0 tx=0x7f6bc400b980).stop 2021-04-09T18:21:57.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 shutdown_connections 2021-04-09T18:21:57.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6bb4018930 0x7f6bb401ae10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bcc1a9500 0x7f6bcc1a9900 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bcc18d800 0x7f6bcc18dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 --2- 172.21.15.32:0/4263899039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bcc0b8480 0x7f6bcc0b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 >> 172.21.15.32:0/4263899039 conn(0x7f6bcc08fb70 msgr2=0x7f6bcc19ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:57.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 shutdown_connections 2021-04-09T18:21:57.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.355 7f6bd451e700 1 -- 172.21.15.32:0/4263899039 wait complete. 2021-04-09T18:21:57.417 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:21:50.420671', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:21:51.790105', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:21:57.418 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:21:57.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.521 7f6e97d0a700 1 Processor -- start 2021-04-09T18:21:57.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.521 7f6e97d0a700 1 -- start start 2021-04-09T18:21:57.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e97d0a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6e9010d010 0x7f6e900d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e97d0a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e900d1da0 0x7f6e900d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e97d0a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6e900d82c0 0x7f6e9010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e97d0a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e900d8d30 con 0x7f6e900d82c0 2021-04-09T18:21:57.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e97d0a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e900d9220 con 0x7f6e9010d010 2021-04-09T18:21:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e97d0a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9010d480 con 0x7f6e900d1da0 2021-04-09T18:21:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e96327700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6e900d82c0 0x7f6e9010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e96327700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6e900d82c0 0x7f6e9010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48804/0 (socket says 172.21.15.32:48804) 2021-04-09T18:21:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e96327700 1 -- 172.21.15.32:0/1660491935 learned_addr learned my addr 172.21.15.32:0/1660491935 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:57.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e95325700 1 --2- 172.21.15.32:0/1660491935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e900d1da0 0x7f6e900d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e95b26700 1 --2- 172.21.15.32:0/1660491935 >> v2:172.21.15.86:3300/0 conn(0x7f6e9010d010 0x7f6e900d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e96327700 1 -- 172.21.15.32:0/1660491935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e900d1da0 msgr2=0x7f6e900d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:57.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e96327700 1 --2- 172.21.15.32:0/1660491935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e900d1da0 0x7f6e900d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e96327700 1 -- 172.21.15.32:0/1660491935 >> v2:172.21.15.86:3300/0 conn(0x7f6e9010d010 msgr2=0x7f6e900d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e96327700 1 --2- 172.21.15.32:0/1660491935 >> v2:172.21.15.86:3300/0 conn(0x7f6e9010d010 0x7f6e900d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e96327700 1 -- 172.21.15.32:0/1660491935 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e900d86c0 con 0x7f6e900d82c0 2021-04-09T18:21:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e95325700 1 --2- 172.21.15.32:0/1660491935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e900d1da0 0x7f6e900d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e95b26700 1 --2- 172.21.15.32:0/1660491935 >> v2:172.21.15.86:3300/0 conn(0x7f6e9010d010 0x7f6e900d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:21:57.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e96327700 1 --2- 172.21.15.32:0/1660491935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900d82c0 0x7f6e9010da00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f6e88006f00 tx=0x7f6e88009f10).ready entity=mon.0 client_cookie=79f6e6d599800136 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e94b24700 1 -- 172.21.15.32:0/1660491935 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e8800b3b0 con 0x7f6e900d82c0 2021-04-09T18:21:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e94b24700 1 -- 172.21.15.32:0/1660491935 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8800b730 con 0x7f6e900d82c0 2021-04-09T18:21:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e94b24700 1 -- 172.21.15.32:0/1660491935 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e8800bc10 con 0x7f6e900d82c0 2021-04-09T18:21:57.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e97d0a700 1 -- 172.21.15.32:0/1660491935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900d82c0 msgr2=0x7f6e9010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.522 7f6e97d0a700 1 --2- 172.21.15.32:0/1660491935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900d82c0 0x7f6e9010da00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f6e88006f00 tx=0x7f6e88009f10).stop 2021-04-09T18:21:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.523 7f6e97d0a700 1 -- 172.21.15.32:0/1660491935 shutdown_connections 2021-04-09T18:21:57.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.523 7f6e97d0a700 1 --2- 172.21.15.32:0/1660491935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900d82c0 0x7f6e9010da00 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.523 7f6e97d0a700 1 --2- 172.21.15.32:0/1660491935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e900d1da0 0x7f6e900d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.523 7f6e97d0a700 1 --2- 172.21.15.32:0/1660491935 >> v2:172.21.15.86:3300/0 conn(0x7f6e9010d010 0x7f6e900d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.523 7f6e97d0a700 1 -- 172.21.15.32:0/1660491935 >> 172.21.15.32:0/1660491935 conn(0x7f6e900ce5f0 msgr2=0x7f6e900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:57.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.523 7f6e97d0a700 1 -- 172.21.15.32:0/1660491935 shutdown_connections 2021-04-09T18:21:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.523 7f6e97d0a700 1 -- 172.21.15.32:0/1660491935 wait complete. 2021-04-09T18:21:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.543 7f6e97d0a700 1 Processor -- start 2021-04-09T18:21:57.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.543 7f6e97d0a700 1 -- start start 2021-04-09T18:21:57.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e97d0a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900c3c00 0x7f6e9009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e97d0a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e90095240 0x7f6e901a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e97d0a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e901a1270 0x7f6e900b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e97d0a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9009cda0 con 0x7f6e900c3c00 2021-04-09T18:21:57.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e97d0a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e900c3680 con 0x7f6e90095240 2021-04-09T18:21:57.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e97d0a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90095640 con 0x7f6e901a1270 2021-04-09T18:21:57.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e95b26700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900c3c00 0x7f6e9009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e95b26700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900c3c00 0x7f6e9009c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48808/0 (socket says 172.21.15.32:48808) 2021-04-09T18:21:57.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e95b26700 1 -- 172.21.15.32:0/1653353163 learned_addr learned my addr 172.21.15.32:0/1653353163 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:57.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e95325700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e90095240 0x7f6e901a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e96327700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e901a1270 0x7f6e900b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e95b26700 1 -- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e901a1270 msgr2=0x7f6e900b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e95b26700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e901a1270 0x7f6e900b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e95b26700 1 -- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e90095240 msgr2=0x7f6e901a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.544 7f6e95b26700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e90095240 0x7f6e901a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.545 7f6e95b26700 1 -- 172.21.15.32:0/1653353163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e901a1670 con 0x7f6e900c3c00 2021-04-09T18:21:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.545 7f6e95b26700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900c3c00 0x7f6e9009c600 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f6e8c00b580 tx=0x7f6e8c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.545 7f6e867fc700 1 -- 172.21.15.32:0/1653353163 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e8c00bd10 con 0x7f6e900c3c00 2021-04-09T18:21:57.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.545 7f6e867fc700 1 -- 172.21.15.32:0/1653353163 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8c000c90 con 0x7f6e900c3c00 2021-04-09T18:21:57.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.545 7f6e867fc700 1 -- 172.21.15.32:0/1653353163 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e8c002220 con 0x7f6e900c3c00 2021-04-09T18:21:57.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.545 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e901a1670 con 0x7f6e900c3c00 2021-04-09T18:21:57.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.545 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e901a1670 con 0x7f6e900c3c00 2021-04-09T18:21:57.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.545 7f6e867fc700 1 -- 172.21.15.32:0/1653353163 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6e8c00a000 con 0x7f6e900c3c00 2021-04-09T18:21:57.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.546 7f6e867fc700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e780188d0 0x7f6e7801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.546 7f6e867fc700 1 -- 172.21.15.32:0/1653353163 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6e8c032a20 con 0x7f6e900c3c00 2021-04-09T18:21:57.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.546 7f6e95325700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e780188d0 0x7f6e7801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.546 7f6e95325700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e780188d0 0x7f6e7801add0 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:57.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.566 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e90093d50 con 0x7f6e900c3c00 2021-04-09T18:21:57.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.569 7f6e867fc700 1 -- 172.21.15.32:0/1653353163 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6e8c006bb0 con 0x7f6e900c3c00 2021-04-09T18:21:57.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.747 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f6e9010d480 con 0x7f6e900c3c00 2021-04-09T18:21:57.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.748 7f6e867fc700 1 -- 172.21.15.32:0/1653353163 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v95) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6e8c000c90 con 0x7f6e900c3c00 2021-04-09T18:21:57.801 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:21:57.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e780188d0 msgr2=0x7f6e7801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e780188d0 0x7f6e7801add0 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900c3c00 msgr2=0x7f6e9009c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900c3c00 0x7f6e9009c600 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f6e8c00b580 tx=0x7f6e8c00b920).stop 2021-04-09T18:21:57.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 shutdown_connections 2021-04-09T18:21:57.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e780188d0 0x7f6e7801add0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e901a1270 0x7f6e900b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e90095240 0x7f6e901a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 --2- 172.21.15.32:0/1653353163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e900c3c00 0x7f6e9009c600 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.750 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 >> 172.21.15.32:0/1653353163 conn(0x7f6e900975f0 msgr2=0x7f6e90096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:57.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.751 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 shutdown_connections 2021-04-09T18:21:57.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.751 7f6e97d0a700 1 -- 172.21.15.32:0/1653353163 wait complete. 2021-04-09T18:21:57.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:21:57.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.914 7fed5f09b700 1 Processor -- start 2021-04-09T18:21:57.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.914 7fed5f09b700 1 -- start start 2021-04-09T18:21:57.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed5f09b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fed5810d010 0x7fed580d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed5f09b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5810d480 con 0x7fed580d1da0 2021-04-09T18:21:57.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed5f09b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5810e1f0 con 0x7fed580d8490 2021-04-09T18:21:57.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed5f09b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed580d8d30 con 0x7fed5810d010 2021-04-09T18:21:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed5ceb7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fed5810d010 0x7fed580d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed57fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fed580d1da0 0x7fed580d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58408/0 (socket says 172.21.15.32:58408) 2021-04-09T18:21:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed57fff700 1 -- 172.21.15.32:0/3070167171 learned_addr learned my addr 172.21.15.32:0/3070167171 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed5ceb7700 1 --2- 172.21.15.32:0/3070167171 >> v2:172.21.15.32:3300/0 conn(0x7fed5810d010 0x7fed580d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48816/0 (socket says 172.21.15.32:48816) 2021-04-09T18:21:57.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed5ceb7700 1 -- 172.21.15.32:0/3070167171 >> v1:172.21.15.86:6789/0 conn(0x7fed580d8490 legacy=0x7fed5810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed5ceb7700 1 -- 172.21.15.32:0/3070167171 >> v1:172.21.15.32:6789/0 conn(0x7fed580d1da0 legacy=0x7fed580d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed5ceb7700 1 -- 172.21.15.32:0/3070167171 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed580d9220 con 0x7fed5810d010 2021-04-09T18:21:57.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.915 7fed577fe700 1 -- 172.21.15.32:0/3070167171 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393773986 0 0) 0x7fed4c000ce0 con 0x7fed580d1da0 2021-04-09T18:21:57.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed5ceb7700 1 --2- 172.21.15.32:0/3070167171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed5810d010 0x7fed580d1780 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7fed48006650 tx=0x7fed48009a00).ready entity=mon.0 client_cookie=a6a1b02467e2502a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:57.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed577fe700 1 -- 172.21.15.32:0/3070167171 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed48009df0 con 0x7fed5810d010 2021-04-09T18:21:57.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed577fe700 1 -- 172.21.15.32:0/3070167171 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed480009b0 con 0x7fed5810d010 2021-04-09T18:21:57.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed577fe700 1 -- 172.21.15.32:0/3070167171 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed4800c2b0 con 0x7fed5810d010 2021-04-09T18:21:57.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed5f09b700 1 -- 172.21.15.32:0/3070167171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed5810d010 msgr2=0x7fed580d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed5f09b700 1 --2- 172.21.15.32:0/3070167171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed5810d010 0x7fed580d1780 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7fed48006650 tx=0x7fed48009a00).stop 2021-04-09T18:21:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed5f09b700 1 -- 172.21.15.32:0/3070167171 shutdown_connections 2021-04-09T18:21:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed5f09b700 1 --2- 172.21.15.32:0/3070167171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed5810d010 0x7fed580d1780 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:57.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed5f09b700 1 -- 172.21.15.32:0/3070167171 >> 172.21.15.32:0/3070167171 conn(0x7fed580ce5f0 msgr2=0x7fed580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:57.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.916 7fed5f09b700 1 -- 172.21.15.32:0/3070167171 shutdown_connections 2021-04-09T18:21:57.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.917 7fed5f09b700 1 -- 172.21.15.32:0/3070167171 wait complete. 2021-04-09T18:21:57.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.937 7fed5f09b700 1 Processor -- start 2021-04-09T18:21:57.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.937 7fed5f09b700 1 -- start start 2021-04-09T18:21:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.937 7fed5f09b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed5817d240 0x7fed58095350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.937 7fed5f09b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed5817d640 0x7fed58093c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5f09b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed58095970 0x7fed58094270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:57.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5f09b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed58095d70 con 0x7fed58095970 2021-04-09T18:21:57.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5f09b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed58094dd0 con 0x7fed5817d240 2021-04-09T18:21:57.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5f09b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed58094890 con 0x7fed5817d640 2021-04-09T18:21:58.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5d6b8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed58095970 0x7fed58094270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5d6b8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed58095970 0x7fed58094270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48822/0 (socket says 172.21.15.32:48822) 2021-04-09T18:21:58.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5d6b8700 1 -- 172.21.15.32:0/363467429 learned_addr learned my addr 172.21.15.32:0/363467429 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:58.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed57fff700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed5817d640 0x7fed58093c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5ceb7700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed5817d240 0x7fed58095350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5d6b8700 1 -- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed5817d640 msgr2=0x7fed58093c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5d6b8700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed5817d640 0x7fed58093c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5d6b8700 1 -- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed5817d240 msgr2=0x7fed58095350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5d6b8700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed5817d240 0x7fed58095350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5d6b8700 1 -- 172.21.15.32:0/363467429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed58094ab0 con 0x7fed58095970 2021-04-09T18:21:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5d6b8700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed58095970 0x7fed58094270 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7fed50009940 tx=0x7fed5000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed557fa700 1 -- 172.21.15.32:0/363467429 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed50000f40 con 0x7fed58095970 2021-04-09T18:21:58.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed557fa700 1 -- 172.21.15.32:0/363467429 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5000c2f0 con 0x7fed58095970 2021-04-09T18:21:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed557fa700 1 -- 172.21.15.32:0/363467429 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed5000c7d0 con 0x7fed58095970 2021-04-09T18:21:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5f09b700 1 -- 172.21.15.32:0/363467429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed58094ab0 con 0x7fed58095970 2021-04-09T18:21:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.938 7fed5f09b700 1 -- 172.21.15.32:0/363467429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed58094ab0 con 0x7fed58095970 2021-04-09T18:21:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.939 7fed557fa700 1 -- 172.21.15.32:0/363467429 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fed50001220 con 0x7fed58095970 2021-04-09T18:21:58.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.939 7fed557fa700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed380188d0 0x7fed3801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.939 7fed557fa700 1 -- 172.21.15.32:0/363467429 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fed50032f00 con 0x7fed58095970 2021-04-09T18:21:58.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.939 7fed5ceb7700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed380188d0 0x7fed3801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.940 7fed5ceb7700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed380188d0 0x7fed3801add0 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:58.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.959 7fed5f09b700 1 -- 172.21.15.32:0/363467429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5808dac0 con 0x7fed58095970 2021-04-09T18:21:58.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:57.962 7fed557fa700 1 -- 172.21.15.32:0/363467429 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fed5000be30 con 0x7fed58095970 2021-04-09T18:21:58.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.140 7fed5f09b700 1 -- 172.21.15.32:0/363467429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fed5810d480 con 0x7fed58095970 2021-04-09T18:21:58.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.141 7fed557fa700 1 -- 172.21.15.32:0/363467429 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v95) v1 ==== 104+0+10 (secure 0 0 0) 0x7fed50001220 con 0x7fed58095970 2021-04-09T18:21:58.193 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:21:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 -- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed380188d0 msgr2=0x7fed3801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed380188d0 0x7fed3801add0 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 -- 172.21.15.32:0/363467429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed58095970 msgr2=0x7fed58094270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed58095970 0x7fed58094270 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7fed50009940 tx=0x7fed5000bcf0).stop 2021-04-09T18:21:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 -- 172.21.15.32:0/363467429 shutdown_connections 2021-04-09T18:21:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed380188d0 0x7fed3801add0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed58095970 0x7fed58094270 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed5817d640 0x7fed58093c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 --2- 172.21.15.32:0/363467429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed5817d240 0x7fed58095350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 -- 172.21.15.32:0/363467429 >> 172.21.15.32:0/363467429 conn(0x7fed580bb410 msgr2=0x7fed580a13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 -- 172.21.15.32:0/363467429 shutdown_connections 2021-04-09T18:21:58.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.143 7fed5f09b700 1 -- 172.21.15.32:0/363467429 wait complete. 2021-04-09T18:21:58.209 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:21:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.310 7ff65c158700 1 Processor -- start 2021-04-09T18:21:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.310 7ff65c158700 1 -- start start 2021-04-09T18:21:58.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff65c158700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65410d010 0x7ff6540d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff65c158700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65410d480 con 0x7ff6540d8490 2021-04-09T18:21:58.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff65c158700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65410e1f0 con 0x7ff6540d1da0 2021-04-09T18:21:58.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff65c158700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65410e780 con 0x7ff65410d010 2021-04-09T18:21:58.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff65a775700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff6540d8490 0x7ff65410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58418/0 (socket says 172.21.15.32:58418) 2021-04-09T18:21:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff65a775700 1 -- 172.21.15.32:0/3334425516 learned_addr learned my addr 172.21.15.32:0/3334425516 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff659f74700 1 --2- 172.21.15.32:0/3334425516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65410d010 0x7ff6540d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff658f72700 1 -- 172.21.15.32:0/3334425516 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38340874 0 0) 0x7ff64c000d30 con 0x7ff6540d8490 2021-04-09T18:21:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff658f72700 1 -- 172.21.15.32:0/3334425516 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff640002fe0 con 0x7ff6540d8490 2021-04-09T18:21:58.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.311 7ff658f72700 1 -- 172.21.15.32:0/3334425516 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481787940 0 0) 0x7ff644000c60 con 0x7ff6540d1da0 2021-04-09T18:21:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff658f72700 1 -- 172.21.15.32:0/3334425516 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff640003210 con 0x7ff6540d1da0 2021-04-09T18:21:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff659f74700 1 -- 172.21.15.32:0/3334425516 >> v1:172.21.15.86:6789/0 conn(0x7ff6540d1da0 legacy=0x7ff6540d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff659f74700 1 -- 172.21.15.32:0/3334425516 >> v1:172.21.15.32:6789/0 conn(0x7ff6540d8490 legacy=0x7ff65410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff658f72700 1 -- 172.21.15.32:0/3334425516 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2787722359 0 0) 0x7ff64c001460 con 0x7ff6540d8490 2021-04-09T18:21:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff659f74700 1 -- 172.21.15.32:0/3334425516 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6540d8d30 con 0x7ff65410d010 2021-04-09T18:21:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff659f74700 1 --2- 172.21.15.32:0/3334425516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65410d010 0x7ff6540d1780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7ff650006650 tx=0x7ff650009a80).ready entity=mon.2 client_cookie=4b55411a66f880a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff658f72700 1 -- 172.21.15.32:0/3334425516 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6500009b0 con 0x7ff65410d010 2021-04-09T18:21:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff658f72700 1 -- 172.21.15.32:0/3334425516 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65000c0c0 con 0x7ff65410d010 2021-04-09T18:21:58.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff658f72700 1 -- 172.21.15.32:0/3334425516 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff65000c510 con 0x7ff65410d010 2021-04-09T18:21:58.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff65c158700 1 -- 172.21.15.32:0/3334425516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65410d010 msgr2=0x7ff6540d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.312 7ff65c158700 1 --2- 172.21.15.32:0/3334425516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65410d010 0x7ff6540d1780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7ff650006650 tx=0x7ff650009a80).stop 2021-04-09T18:21:58.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.313 7ff65c158700 1 -- 172.21.15.32:0/3334425516 shutdown_connections 2021-04-09T18:21:58.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.313 7ff65c158700 1 --2- 172.21.15.32:0/3334425516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65410d010 0x7ff6540d1780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.313 7ff65c158700 1 -- 172.21.15.32:0/3334425516 >> 172.21.15.32:0/3334425516 conn(0x7ff6540ce5f0 msgr2=0x7ff6540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:58.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.313 7ff65c158700 1 -- 172.21.15.32:0/3334425516 shutdown_connections 2021-04-09T18:21:58.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.313 7ff65c158700 1 -- 172.21.15.32:0/3334425516 wait complete. 2021-04-09T18:21:58.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.333 7ff65c158700 1 Processor -- start 2021-04-09T18:21:58.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.333 7ff65c158700 1 -- start start 2021-04-09T18:21:58.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff65c158700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6541a0050 0x7ff6541a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff65c158700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6540bdcd0 0x7ff6540bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff65c158700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6540bf080 0x7ff6540b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff65c158700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6540bf500 con 0x7ff6541a0050 2021-04-09T18:21:58.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff65c158700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6540bb7b0 con 0x7ff6540bf080 2021-04-09T18:21:58.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff65c158700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6540be0d0 con 0x7ff6540bdcd0 2021-04-09T18:21:58.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659f74700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6541a0050 0x7ff6541a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659f74700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6541a0050 0x7ff6541a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48832/0 (socket says 172.21.15.32:48832) 2021-04-09T18:21:58.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659f74700 1 -- 172.21.15.32:0/2283382151 learned_addr learned my addr 172.21.15.32:0/2283382151 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:58.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659773700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6540bdcd0 0x7ff6540bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff65a775700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6540bf080 0x7ff6540b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659f74700 1 -- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6540bdcd0 msgr2=0x7ff6540bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659f74700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6540bdcd0 0x7ff6540bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659f74700 1 -- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6540bf080 msgr2=0x7ff6540b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659f74700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6540bf080 0x7ff6540b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659f74700 1 -- 172.21.15.32:0/2283382151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6540b1310 con 0x7ff6541a0050 2021-04-09T18:21:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff659f74700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6541a0050 0x7ff6541a0450 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7ff650009650 tx=0x7ff65000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:58.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff64a7fc700 1 -- 172.21.15.32:0/2283382151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6500093a0 con 0x7ff6541a0050 2021-04-09T18:21:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff64a7fc700 1 -- 172.21.15.32:0/2283382151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65000bd50 con 0x7ff6541a0050 2021-04-09T18:21:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff64a7fc700 1 -- 172.21.15.32:0/2283382151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6500052e0 con 0x7ff6541a0050 2021-04-09T18:21:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.334 7ff65c158700 1 -- 172.21.15.32:0/2283382151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6540b1310 con 0x7ff6541a0050 2021-04-09T18:21:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.335 7ff65c158700 1 -- 172.21.15.32:0/2283382151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6540b1310 con 0x7ff6541a0050 2021-04-09T18:21:58.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.335 7ff64a7fc700 1 -- 172.21.15.32:0/2283382151 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff650006030 con 0x7ff6541a0050 2021-04-09T18:21:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.335 7ff64a7fc700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff63c018930 0x7ff63c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.335 7ff64a7fc700 1 -- 172.21.15.32:0/2283382151 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff6500329e0 con 0x7ff6541a0050 2021-04-09T18:21:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.335 7ff659773700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff63c018930 0x7ff63c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.335 7ff659773700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff63c018930 0x7ff63c01adc0 crc :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:58.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.355 7ff65c158700 1 -- 172.21.15.32:0/2283382151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff654093d50 con 0x7ff6541a0050 2021-04-09T18:21:58.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.358 7ff64a7fc700 1 -- 172.21.15.32:0/2283382151 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff6500009c0 con 0x7ff6541a0050 2021-04-09T18:21:58.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.538 7ff65c158700 1 -- 172.21.15.32:0/2283382151 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7ff654093d50 con 0x7ff6541a0050 2021-04-09T18:21:58.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.538 7ff64a7fc700 1 -- 172.21.15.32:0/2283382151 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v95) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff65000bd50 con 0x7ff6541a0050 2021-04-09T18:21:58.591 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:21:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 -- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff63c018930 msgr2=0x7ff63c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff63c018930 0x7ff63c01adc0 crc :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 -- 172.21.15.32:0/2283382151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6541a0050 msgr2=0x7ff6541a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6541a0050 0x7ff6541a0450 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7ff650009650 tx=0x7ff65000b960).stop 2021-04-09T18:21:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 -- 172.21.15.32:0/2283382151 shutdown_connections 2021-04-09T18:21:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff63c018930 0x7ff63c01adc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6540bf080 0x7ff6540b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6540bdcd0 0x7ff6540bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 --2- 172.21.15.32:0/2283382151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6541a0050 0x7ff6541a0450 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 -- 172.21.15.32:0/2283382151 >> 172.21.15.32:0/2283382151 conn(0x7ff6540975f0 msgr2=0x7ff654096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:58.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 -- 172.21.15.32:0/2283382151 shutdown_connections 2021-04-09T18:21:58.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.541 7ff65c158700 1 -- 172.21.15.32:0/2283382151 wait complete. 2021-04-09T18:21:58.603 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:21:58.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:21:58.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.700 7f63e12f1700 1 Processor -- start 2021-04-09T18:21:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.700 7f63e12f1700 1 -- start start 2021-04-09T18:21:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63e12f1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc0d7ca0 0x7f63dc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63e12f1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63dc10da00 0x7f63dc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63e12f1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63dc10d480 con 0x7f63dc10d010 2021-04-09T18:21:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63e12f1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63dc0d9220 con 0x7f63dc10da00 2021-04-09T18:21:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63e12f1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0d86c0 con 0x7f63dc0d7ca0 2021-04-09T18:21:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63db07d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f63dc10d010 0x7f63dc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58430/0 (socket says 172.21.15.32:58430) 2021-04-09T18:21:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63db07d700 1 -- 172.21.15.32:0/2710310931 learned_addr learned my addr 172.21.15.32:0/2710310931 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:21:58.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63da87c700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc0d7ca0 0x7f63dc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63db87e700 1 --2- 172.21.15.32:0/2710310931 >> v2:172.21.15.86:3300/0 conn(0x7f63dc10da00 0x7f63dc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63da07b700 1 -- 172.21.15.32:0/2710310931 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102947794 0 0) 0x7f63cc000ce0 con 0x7f63dc10d010 2021-04-09T18:21:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63da07b700 1 -- 172.21.15.32:0/2710310931 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c0002fe0 con 0x7f63dc10d010 2021-04-09T18:21:58.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63da07b700 1 -- 172.21.15.32:0/2710310931 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1249187512 0 0) 0x7f63cc0010e0 con 0x7f63dc10d010 2021-04-09T18:21:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63da07b700 1 -- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc0d7ca0 msgr2=0x7f63dc0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.701 7f63da07b700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc0d7ca0 0x7f63dc0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 >> v2:172.21.15.86:3300/0 conn(0x7f63dc10da00 msgr2=0x7f63dc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 --2- 172.21.15.32:0/2710310931 >> v2:172.21.15.86:3300/0 conn(0x7f63dc10da00 0x7f63dc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc0d1f70 con 0x7f63dc10d010 2021-04-09T18:21:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f63cc001600 con 0x7f63dc10d010 2021-04-09T18:21:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 >> v1:172.21.15.32:6789/0 conn(0x7f63dc10d010 legacy=0x7f63dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c0003a70 0x7f63c0005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c004a7b0 0x7f63c004ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c004d5b0 0x7f63c004faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c00502f0 con 0x7f63c0003a70 2021-04-09T18:21:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c0050510 con 0x7f63c004a7b0 2021-04-09T18:21:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c0050730 con 0x7f63c004d5b0 2021-04-09T18:21:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63db07d700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c0003a70 0x7f63c0005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63db87e700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c004d5b0 0x7f63c004faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da87c700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c004a7b0 0x7f63c004ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63db07d700 1 -- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c004d5b0 msgr2=0x7f63c004faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63db07d700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c004d5b0 0x7f63c004faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63db07d700 1 -- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c004a7b0 msgr2=0x7f63c004ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:21:58.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63db07d700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c004a7b0 0x7f63c004ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63db07d700 1 -- 172.21.15.32:0/2710310931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c00509d0 con 0x7f63c0003a70 2021-04-09T18:21:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63db07d700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c0003a70 0x7f63c0005f00 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f63cc007970 tx=0x7f63cc007d10).ready entity=mon.0 client_cookie=42dc12f47151a1b5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63cc0010e0 con 0x7f63c0003a70 2021-04-09T18:21:58.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc00a520 con 0x7f63c0003a70 2021-04-09T18:21:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63da07b700 1 -- 172.21.15.32:0/2710310931 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63cc00a930 con 0x7f63c0003a70 2021-04-09T18:21:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63e12f1700 1 -- 172.21.15.32:0/2710310931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c0003a70 msgr2=0x7f63c0005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63e12f1700 1 --2- 172.21.15.32:0/2710310931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c0003a70 0x7f63c0005f00 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f63cc007970 tx=0x7f63cc007d10).stop 2021-04-09T18:21:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.702 7f63db87e700 1 -- 172.21.15.32:0/2710310931 reap_dead start 2021-04-09T18:21:58.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.703 7f63e12f1700 1 -- 172.21.15.32:0/2710310931 shutdown_connections 2021-04-09T18:21:58.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.703 7f63e12f1700 1 -- 172.21.15.32:0/2710310931 >> 172.21.15.32:0/2710310931 conn(0x7f63dc0ce5f0 msgr2=0x7f63dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:58.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.703 7f63e12f1700 1 -- 172.21.15.32:0/2710310931 shutdown_connections 2021-04-09T18:21:58.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.703 7f63e12f1700 1 -- 172.21.15.32:0/2710310931 wait complete. 2021-04-09T18:21:58.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.723 7f63e12f1700 1 Processor -- start 2021-04-09T18:21:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.723 7f63e12f1700 1 -- start start 2021-04-09T18:21:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63e12f1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0aa2b0 0x7f63dc09d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63e12f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc09d9b0 0x7f63dc1943a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63e12f1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1949c0 0x7f63dc195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63e12f1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc1950a0 con 0x7f63dc09d9b0 2021-04-09T18:21:58.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63e12f1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc194dc0 con 0x7f63dc0aa2b0 2021-04-09T18:21:58.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63e12f1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc195c40 con 0x7f63dc1949c0 2021-04-09T18:21:58.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63da87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc09d9b0 0x7f63dc1943a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63da87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc09d9b0 0x7f63dc1943a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48850/0 (socket says 172.21.15.32:48850) 2021-04-09T18:21:58.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63da87c700 1 -- 172.21.15.32:0/3626028208 learned_addr learned my addr 172.21.15.32:0/3626028208 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:21:58.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63db87e700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1949c0 0x7f63dc195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63db07d700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0aa2b0 0x7f63dc09d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63da87c700 1 -- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1949c0 msgr2=0x7f63dc195620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63da87c700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1949c0 0x7f63dc195620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63da87c700 1 -- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0aa2b0 msgr2=0x7f63dc09d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63da87c700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0aa2b0 0x7f63dc09d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63da87c700 1 -- 172.21.15.32:0/3626028208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc195e60 con 0x7f63dc09d9b0 2021-04-09T18:21:58.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.724 7f63da87c700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc09d9b0 0x7f63dc1943a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f63c800b590 tx=0x7f63c800b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:58.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.725 7f63d3fff700 1 -- 172.21.15.32:0/3626028208 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c80009b0 con 0x7f63dc09d9b0 2021-04-09T18:21:58.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.725 7f63d3fff700 1 -- 172.21.15.32:0/3626028208 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c800bd20 con 0x7f63dc09d9b0 2021-04-09T18:21:58.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.725 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63dc195e60 con 0x7f63dc09d9b0 2021-04-09T18:21:58.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.725 7f63d3fff700 1 -- 172.21.15.32:0/3626028208 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c80022b0 con 0x7f63dc09d9b0 2021-04-09T18:21:58.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.725 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63dc195e60 con 0x7f63dc09d9b0 2021-04-09T18:21:58.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.725 7f63d3fff700 1 -- 172.21.15.32:0/3626028208 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f63c8009fe0 con 0x7f63dc09d9b0 2021-04-09T18:21:58.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.725 7f63d3fff700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b8018930 0x7f63b801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:21:58.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.725 7f63d3fff700 1 -- 172.21.15.32:0/3626028208 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f63c8032a70 con 0x7f63dc09d9b0 2021-04-09T18:21:58.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.726 7f63db07d700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b8018930 0x7f63b801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:21:58.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.726 7f63db07d700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b8018930 0x7f63b801adc0 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:21:58.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.745 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63dc182d10 con 0x7f63dc09d9b0 2021-04-09T18:21:58.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.748 7f63d3fff700 1 -- 172.21.15.32:0/3626028208 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f63c8000c90 con 0x7f63dc09d9b0 2021-04-09T18:21:58.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.939 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63dc10d480 con 0x7f63dc09d9b0 2021-04-09T18:21:58.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.939 7f63d3fff700 1 -- 172.21.15.32:0/3626028208 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2781 (secure 0 0 0) 0x7f63c800bd20 con 0x7f63dc09d9b0 2021-04-09T18:21:58.992 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 108 2021-04-09T18:21:58.992 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:21:58.992 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":108,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":7575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:21:54.337303","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:21:58.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.941 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b8018930 msgr2=0x7f63b801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.941 7f63e12f1700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b8018930 0x7f63b801adc0 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.941 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc09d9b0 msgr2=0x7f63dc1943a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:21:58.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.941 7f63e12f1700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc09d9b0 0x7f63dc1943a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f63c800b590 tx=0x7f63c800b930).stop 2021-04-09T18:21:58.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.942 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 shutdown_connections 2021-04-09T18:21:58.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.942 7f63e12f1700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b8018930 0x7f63b801adc0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.942 7f63e12f1700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc09d9b0 0x7f63dc1943a0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.942 7f63e12f1700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1949c0 0x7f63dc195620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.942 7f63e12f1700 1 --2- 172.21.15.32:0/3626028208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0aa2b0 0x7f63dc09d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:21:58.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.942 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 >> 172.21.15.32:0/3626028208 conn(0x7f63dc1a13a0 msgr2=0x7f63dc094bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:21:58.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.942 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 shutdown_connections 2021-04-09T18:21:58.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:21:58.942 7f63e12f1700 1 -- 172.21.15.32:0/3626028208 wait complete. 2021-04-09T18:21:59.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:21:54.337303', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7387}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:21:59.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:21:59.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:21:59.543 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:21:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:21:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:21:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 7387, 2021-04-09T18:21:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:21:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:21:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 7.1589185300000002, 2021-04-09T18:21:59.545 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 108, 2021-04-09T18:21:59.545 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 95, 2021-04-09T18:21:59.545 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 95, 2021-04-09T18:21:59.545 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 11.598512668 2021-04-09T18:21:59.546 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:21:59.549 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 7387, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.1589185300000002, "mdsmap_epoch": 108, "osdmap_epoch": 95, "osdmap_epoch_barrier": 95, "uptime": 11.598512668 } 2021-04-09T18:21:59.550 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:21:59.550 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:21:59.551 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:21:59.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:21:59.552 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:21:59.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:21:59.658 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:21:59.659 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:21:59.718 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:21:59.719 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:21:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:21:59.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:21:59.722 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:21:59.766 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:21:59.767 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:21:59.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:21:59.869 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:21:59.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:21:59.929 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:21:59.930 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:21:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:21:59.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:21:59.933 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:21:59.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:21:59.906 7f220659ef80 -1 init, newargv = 0x5577a0a25230 newargc=9 2021-04-09T18:21:59.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[59410]: starting ceph client 2021-04-09T18:21:59.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[59410]: starting fuse 2021-04-09T18:21:59.979 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:21:59.980 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:21:59.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:21:59.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:21:59.982 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:21:59.983 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:21:59.984 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:21:59.984 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:21:59.985 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:21:59.986 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:21:59.987 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:21:59.988 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:21:59.989 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:21:59.989 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:21:59.990 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:21:59.990 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:21:59.990 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:21:59.991 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:21:59.991 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:21:59.992 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:21:59.992 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:21:59.993 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:21:59.993 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:21:59.995 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:21:59.996 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:21:59.997 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:22:00.102 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.59410.asok 2021-04-09T18:22:00.106 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59410.asok 2021-04-09T18:22:00.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:00.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59410.asok status 2021-04-09T18:22:00.231 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:22:00.232 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:22:00.232 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:22:00.232 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:22:00.232 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:22:00.232 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:22:00.233 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:22:00.233 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "59410", 2021-04-09T18:22:00.233 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:22:00.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:00.234 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:22:00.234 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:22:00.234 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 7722, 2021-04-09T18:22:00.234 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:22:00.234 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:22:00.235 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:22:00.235 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 7722 2021-04-09T18:22:00.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:00.235 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:22:00.236 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:00.236 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:22:00.236 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3060726433 2021-04-09T18:22:00.237 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:00.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:00.238 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:22:00.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:00.239 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:22:00.239 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3060726433 2021-04-09T18:22:00.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:00.240 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.7722 172.21.15.32:0/3060726433", 2021-04-09T18:22:00.240 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/3060726433", 2021-04-09T18:22:00.240 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:22:00.241 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 108, 2021-04-09T18:22:00.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 95, 2021-04-09T18:22:00.241 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:22:00.241 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:22:00.241 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:22:00.242 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:22:00.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:00.243 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:22:00.244 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:00.289 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:22:00.289 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:22:00.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:00.290 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:00.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:22:00.479 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:22:00.479 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:22:00.479 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:22:00.485 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:22:00.486 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:00.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.584 7f1566226700 1 Processor -- start 2021-04-09T18:22:00.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.584 7f1566226700 1 -- start start 2021-04-09T18:22:00.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1566226700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f156010d010 0x7f15600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:00.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1566226700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f15600d8490 0x7f156010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:00.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1566226700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156010d480 con 0x7f15600d1da0 2021-04-09T18:22:00.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1566226700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156010e780 con 0x7f15600d8490 2021-04-09T18:22:00.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1566226700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15600d8d30 con 0x7f156010d010 2021-04-09T18:22:00.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1564843700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f15600d8490 0x7f156010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:00.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1564843700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f15600d8490 0x7f156010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48874/0 (socket says 172.21.15.32:48874) 2021-04-09T18:22:00.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1564843700 1 -- 172.21.15.32:0/893111110 learned_addr learned my addr 172.21.15.32:0/893111110 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:00.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f155ffff700 1 --2- 172.21.15.32:0/893111110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f156010d010 0x7f15600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:00.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1564843700 1 -- 172.21.15.32:0/893111110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f156010d010 msgr2=0x7f15600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:00.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1564843700 1 --2- 172.21.15.32:0/893111110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f156010d010 0x7f15600d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1564843700 1 -- 172.21.15.32:0/893111110 >> v1:172.21.15.86:6789/0 conn(0x7f15600d1da0 legacy=0x7f15600d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:00.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1564843700 1 -- 172.21.15.32:0/893111110 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15600d9220 con 0x7f15600d8490 2021-04-09T18:22:00.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f1564843700 1 --2- 172.21.15.32:0/893111110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15600d8490 0x7f156010da00 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f1558006f00 tx=0x7f1558009e90).ready entity=mon.0 client_cookie=8f88030196dd01b4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:00.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f155f7fe700 1 -- 172.21.15.32:0/893111110 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155800b300 con 0x7f15600d8490 2021-04-09T18:22:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f155f7fe700 1 -- 172.21.15.32:0/893111110 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155800b640 con 0x7f15600d8490 2021-04-09T18:22:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.585 7f155f7fe700 1 -- 172.21.15.32:0/893111110 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155800bb20 con 0x7f15600d8490 2021-04-09T18:22:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.586 7f1566226700 1 -- 172.21.15.32:0/893111110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15600d8490 msgr2=0x7f156010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:00.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.586 7f1566226700 1 --2- 172.21.15.32:0/893111110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15600d8490 0x7f156010da00 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f1558006f00 tx=0x7f1558009e90).stop 2021-04-09T18:22:00.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.586 7f1566226700 1 -- 172.21.15.32:0/893111110 shutdown_connections 2021-04-09T18:22:00.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.586 7f1566226700 1 --2- 172.21.15.32:0/893111110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15600d8490 0x7f156010da00 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.586 7f1566226700 1 --2- 172.21.15.32:0/893111110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f156010d010 0x7f15600d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.586 7f1566226700 1 -- 172.21.15.32:0/893111110 >> 172.21.15.32:0/893111110 conn(0x7f15600ce5f0 msgr2=0x7f15600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:00.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.586 7f1566226700 1 -- 172.21.15.32:0/893111110 shutdown_connections 2021-04-09T18:22:00.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.586 7f1566226700 1 -- 172.21.15.32:0/893111110 wait complete. 2021-04-09T18:22:00.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1566226700 1 Processor -- start 2021-04-09T18:22:00.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1566226700 1 -- start start 2021-04-09T18:22:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1566226700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1560185060 0x7f1560185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:00.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1566226700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1560186c00 0x7f1560187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1566226700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1560191360 0x7f1560191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1566226700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15601863a0 con 0x7f1560185060 2021-04-09T18:22:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1566226700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1560185a60 con 0x7f1560186c00 2021-04-09T18:22:00.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1566226700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1560187620 con 0x7f1560191360 2021-04-09T18:22:00.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f155ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1560185060 0x7f1560185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:00.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f155ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1560185060 0x7f1560185d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48878/0 (socket says 172.21.15.32:48878) 2021-04-09T18:22:00.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f155ffff700 1 -- 172.21.15.32:0/3781069734 learned_addr learned my addr 172.21.15.32:0/3781069734 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:00.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1564843700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1560191360 0x7f1560191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:00.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.607 7f1557fff700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1560186c00 0x7f1560187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:00.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f1564843700 1 -- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1560186c00 msgr2=0x7f1560187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:00.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f1564843700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1560186c00 0x7f1560187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f1564843700 1 -- 172.21.15.32:0/3781069734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1560185060 msgr2=0x7f1560185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:00.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f1564843700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1560185060 0x7f1560185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f1564843700 1 -- 172.21.15.32:0/3781069734 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560185460 con 0x7f1560191360 2021-04-09T18:22:00.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f155ffff700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1560185060 0x7f1560185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:00.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f1564843700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1560191360 0x7f1560191760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f155800b9a0 tx=0x7f1558008c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f155d7fa700 1 -- 172.21.15.32:0/3781069734 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1558003030 con 0x7f1560191360 2021-04-09T18:22:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f155d7fa700 1 -- 172.21.15.32:0/3781069734 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1558008d60 con 0x7f1560191360 2021-04-09T18:22:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f1566226700 1 -- 172.21.15.32:0/3781069734 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1560185460 con 0x7f1560191360 2021-04-09T18:22:00.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f155d7fa700 1 -- 172.21.15.32:0/3781069734 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1558005a10 con 0x7f1560191360 2021-04-09T18:22:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.608 7f1566226700 1 -- 172.21.15.32:0/3781069734 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1560185460 con 0x7f1560191360 2021-04-09T18:22:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.609 7f155d7fa700 1 -- 172.21.15.32:0/3781069734 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1558005820 con 0x7f1560191360 2021-04-09T18:22:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.609 7f155d7fa700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1544018930 0x7f154401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:00.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.609 7f155d7fa700 1 -- 172.21.15.32:0/3781069734 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1558032e50 con 0x7f1560191360 2021-04-09T18:22:00.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.609 7f155ffff700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1544018930 0x7f154401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:00.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.609 7f155ffff700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1544018930 0x7f154401adc0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:00.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.629 7f1566226700 1 -- 172.21.15.32:0/3781069734 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15601866d0 con 0x7f1560191360 2021-04-09T18:22:00.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.632 7f155d7fa700 1 -- 172.21.15.32:0/3781069734 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1558006d00 con 0x7f1560191360 2021-04-09T18:22:00.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.823 7f1566226700 1 -- 172.21.15.32:0/3781069734 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15601866d0 con 0x7f1560191360 2021-04-09T18:22:00.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.824 7f155d7fa700 1 -- 172.21.15.32:0/3781069734 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2781 (secure 0 0 0) 0x7f1558008d60 con 0x7f1560191360 2021-04-09T18:22:00.877 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 108 2021-04-09T18:22:00.878 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:00.878 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":108,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":7575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:21:54.337303","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:00.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 -- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1544018930 msgr2=0x7f154401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:00.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1544018930 0x7f154401adc0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 -- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1560191360 msgr2=0x7f1560191760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:00.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1560191360 0x7f1560191760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f155800b9a0 tx=0x7f1558008c20).stop 2021-04-09T18:22:00.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 -- 172.21.15.32:0/3781069734 shutdown_connections 2021-04-09T18:22:00.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1544018930 0x7f154401adc0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1560191360 0x7f1560191760 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1560186c00 0x7f1560187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 --2- 172.21.15.32:0/3781069734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1560185060 0x7f1560185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:00.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 -- 172.21.15.32:0/3781069734 >> 172.21.15.32:0/3781069734 conn(0x7f15601837d0 msgr2=0x7f1560195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:00.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 -- 172.21.15.32:0/3781069734 shutdown_connections 2021-04-09T18:22:00.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:00.826 7f1566226700 1 -- 172.21.15.32:0/3781069734 wait complete. 2021-04-09T18:22:00.896 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 108, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 8, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:21:52.329633", "damaged": [], "data_pools": [ 17 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7387": { "addr": "172.21.15.86:6835/3998950443", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 3998950443, "type": "v2" }, { "addr": "172.21.15.86:6835", "nonce": 3998950443, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 7387, "incarnation": 106, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 16, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:21:54.337303", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7387 } } } ], "standbys": [ { "addr": "172.21.15.86:6837/867563018", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6836", "nonce": 867563018, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 867563018, "type": "v1" } ] }, "epoch": 105, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 7390, "incarnation": 0, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.32:6834/4090970868", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 4090970868, "type": "v2" }, { "addr": "172.21.15.32:6834", "nonce": 4090970868, "type": "v1" } ] }, "epoch": 105, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 7575, "incarnation": 0, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.32:6835/2579974780", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6833", "nonce": 2579974780, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 2579974780, "type": "v1" } ] }, "epoch": 105, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 56423, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:22:00.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:01.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.006 7f1e3eddb700 1 Processor -- start 2021-04-09T18:22:01.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.006 7f1e3eddb700 1 -- start start 2021-04-09T18:22:01.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3eddb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e380d7ca0 0x7f1e380d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3eddb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1e3810da00 0x7f1e3810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3eddb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3810d480 con 0x7f1e3810d010 2021-04-09T18:22:01.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3eddb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e380d9220 con 0x7f1e3810da00 2021-04-09T18:22:01.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3eddb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e38112100 con 0x7f1e380d7ca0 2021-04-09T18:22:01.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3d3f8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1e3810da00 0x7f1e3810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3d3f8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1e3810da00 0x7f1e3810de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48888/0 (socket says 172.21.15.32:48888) 2021-04-09T18:22:01.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3d3f8700 1 -- 172.21.15.32:0/1377392905 learned_addr learned my addr 172.21.15.32:0/1377392905 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:01.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e37fff700 1 --2- 172.21.15.32:0/1377392905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e380d7ca0 0x7f1e380d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3d3f8700 1 -- 172.21.15.32:0/1377392905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e380d7ca0 msgr2=0x7f1e380d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:01.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3d3f8700 1 --2- 172.21.15.32:0/1377392905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e380d7ca0 0x7f1e380d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3d3f8700 1 -- 172.21.15.32:0/1377392905 >> v1:172.21.15.86:6789/0 conn(0x7f1e3810d010 legacy=0x7f1e380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3d3f8700 1 -- 172.21.15.32:0/1377392905 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e380d86c0 con 0x7f1e3810da00 2021-04-09T18:22:01.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e37fff700 1 --2- 172.21.15.32:0/1377392905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e380d7ca0 0x7f1e380d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:01.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.007 7f1e3d3f8700 1 --2- 172.21.15.32:0/1377392905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3810da00 0x7f1e3810de00 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f1e30006f00 tx=0x7f1e30009f10).ready entity=mon.0 client_cookie=a2a4410b10c62233 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:01.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e377fe700 1 -- 172.21.15.32:0/1377392905 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e3000b3b0 con 0x7f1e3810da00 2021-04-09T18:22:01.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e377fe700 1 -- 172.21.15.32:0/1377392905 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3000b730 con 0x7f1e3810da00 2021-04-09T18:22:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e377fe700 1 -- 172.21.15.32:0/1377392905 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e3000bc10 con 0x7f1e3810da00 2021-04-09T18:22:01.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e3eddb700 1 -- 172.21.15.32:0/1377392905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3810da00 msgr2=0x7f1e3810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e3eddb700 1 --2- 172.21.15.32:0/1377392905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3810da00 0x7f1e3810de00 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f1e30006f00 tx=0x7f1e30009f10).stop 2021-04-09T18:22:01.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e3eddb700 1 -- 172.21.15.32:0/1377392905 shutdown_connections 2021-04-09T18:22:01.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e3eddb700 1 --2- 172.21.15.32:0/1377392905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3810da00 0x7f1e3810de00 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e3eddb700 1 --2- 172.21.15.32:0/1377392905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e380d7ca0 0x7f1e380d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e3eddb700 1 -- 172.21.15.32:0/1377392905 >> 172.21.15.32:0/1377392905 conn(0x7f1e380ce5f0 msgr2=0x7f1e380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:01.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e3eddb700 1 -- 172.21.15.32:0/1377392905 shutdown_connections 2021-04-09T18:22:01.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.008 7f1e3eddb700 1 -- 172.21.15.32:0/1377392905 wait complete. 2021-04-09T18:22:01.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.028 7f1e3eddb700 1 Processor -- start 2021-04-09T18:22:01.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.028 7f1e3eddb700 1 -- start start 2021-04-09T18:22:01.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3eddb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e38179bf0 0x7f1e38179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3eddb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3817c540 0x7f1e3817c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3eddb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e380aca60 0x7f1e380ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3eddb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3817cf60 con 0x7f1e380aca60 2021-04-09T18:22:01.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3eddb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380ad480 con 0x7f1e38179bf0 2021-04-09T18:22:01.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3eddb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380ad6a0 con 0x7f1e3817c540 2021-04-09T18:22:01.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3d3f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e380aca60 0x7f1e380ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3d3f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e380aca60 0x7f1e380ace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48896/0 (socket says 172.21.15.32:48896) 2021-04-09T18:22:01.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3d3f8700 1 -- 172.21.15.32:0/4281333328 learned_addr learned my addr 172.21.15.32:0/4281333328 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:01.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e3cbf7700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e38179bf0 0x7f1e38179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.029 7f1e37fff700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3817c540 0x7f1e3817c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e3d3f8700 1 -- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3817c540 msgr2=0x7f1e3817c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e3d3f8700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3817c540 0x7f1e3817c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e3d3f8700 1 -- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e38179bf0 msgr2=0x7f1e38179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e3d3f8700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e38179bf0 0x7f1e38179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e3d3f8700 1 -- 172.21.15.32:0/4281333328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e380ad8c0 con 0x7f1e380aca60 2021-04-09T18:22:01.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e3d3f8700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e380aca60 0x7f1e380ace60 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f1e30009390 tx=0x7f1e3000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:01.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e357fa700 1 -- 172.21.15.32:0/4281333328 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e30003030 con 0x7f1e380aca60 2021-04-09T18:22:01.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e357fa700 1 -- 172.21.15.32:0/4281333328 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e300068e0 con 0x7f1e380aca60 2021-04-09T18:22:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e357fa700 1 -- 172.21.15.32:0/4281333328 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e3000c570 con 0x7f1e380aca60 2021-04-09T18:22:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e380ad8c0 con 0x7f1e380aca60 2021-04-09T18:22:01.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.030 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e380ad8c0 con 0x7f1e380aca60 2021-04-09T18:22:01.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.031 7f1e357fa700 1 -- 172.21.15.32:0/4281333328 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1e30005820 con 0x7f1e380aca60 2021-04-09T18:22:01.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.031 7f1e357fa700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188c0 0x7f1e1801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.031 7f1e357fa700 1 -- 172.21.15.32:0/4281333328 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1e30032d80 con 0x7f1e380aca60 2021-04-09T18:22:01.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.031 7f1e3cbf7700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188c0 0x7f1e1801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.031 7f1e3cbf7700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188c0 0x7f1e1801adc0 crc :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:01.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.051 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3817cf60 con 0x7f1e380aca60 2021-04-09T18:22:01.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.055 7f1e357fa700 1 -- 172.21.15.32:0/4281333328 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1e300068e0 con 0x7f1e380aca60 2021-04-09T18:22:01.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.245 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e3817cf60 con 0x7f1e380aca60 2021-04-09T18:22:01.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.245 7f1e357fa700 1 -- 172.21.15.32:0/4281333328 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2781 (secure 0 0 0) 0x7f1e300068e0 con 0x7f1e380aca60 2021-04-09T18:22:01.298 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 108 2021-04-09T18:22:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:01.298 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":108,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":7575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:21:54.337303","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:01.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188c0 msgr2=0x7f1e1801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188c0 0x7f1e1801adc0 crc :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e380aca60 msgr2=0x7f1e380ace60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e380aca60 0x7f1e380ace60 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f1e30009390 tx=0x7f1e3000bd40).stop 2021-04-09T18:22:01.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 shutdown_connections 2021-04-09T18:22:01.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e180188c0 0x7f1e1801adc0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e380aca60 0x7f1e380ace60 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e3817c540 0x7f1e3817c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 --2- 172.21.15.32:0/4281333328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e38179bf0 0x7f1e38179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 >> 172.21.15.32:0/4281333328 conn(0x7f1e381acae0 msgr2=0x7f1e3817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:01.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 shutdown_connections 2021-04-09T18:22:01.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.248 7f1e3eddb700 1 -- 172.21.15.32:0/4281333328 wait complete. 2021-04-09T18:22:01.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:22:01.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.418 7f48d0b9a700 1 Processor -- start 2021-04-09T18:22:01.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.418 7f48d0b9a700 1 -- start start 2021-04-09T18:22:01.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48d0b9a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f48cc10d010 0x7f48cc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48d0b9a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc0d8490 0x7f48cc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48d0b9a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48cc10d480 con 0x7f48cc0d1da0 2021-04-09T18:22:01.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48d0b9a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48cc0d8d30 con 0x7f48cc10d010 2021-04-09T18:22:01.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48d0b9a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc0d9220 con 0x7f48cc0d8490 2021-04-09T18:22:01.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48c205d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f48cc0d1da0 0x7f48cc0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58490/0 (socket says 172.21.15.32:58490) 2021-04-09T18:22:01.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48c205d700 1 -- 172.21.15.32:0/3003738926 learned_addr learned my addr 172.21.15.32:0/3003738926 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:01.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48cb05f700 1 --2- 172.21.15.32:0/3003738926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc0d8490 0x7f48cc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48ca85e700 1 --2- 172.21.15.32:0/3003738926 >> v2:172.21.15.86:3300/0 conn(0x7f48cc10d010 0x7f48cc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48ca05d700 1 -- 172.21.15.32:0/3003738926 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976824316 0 0) 0x7f48b4000ce0 con 0x7f48cc0d1da0 2021-04-09T18:22:01.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48ca05d700 1 -- 172.21.15.32:0/3003738926 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b8002fe0 con 0x7f48cc0d1da0 2021-04-09T18:22:01.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48ca85e700 1 -- 172.21.15.32:0/3003738926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc0d8490 msgr2=0x7f48cc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:01.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48ca85e700 1 --2- 172.21.15.32:0/3003738926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc0d8490 0x7f48cc10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48ca85e700 1 -- 172.21.15.32:0/3003738926 >> v1:172.21.15.32:6789/0 conn(0x7f48cc0d1da0 legacy=0x7f48cc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48ca85e700 1 -- 172.21.15.32:0/3003738926 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc112100 con 0x7f48cc10d010 2021-04-09T18:22:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.419 7f48cb05f700 1 --2- 172.21.15.32:0/3003738926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc0d8490 0x7f48cc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:01.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.420 7f48ca85e700 1 --2- 172.21.15.32:0/3003738926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48cc10d010 0x7f48cc0d1780 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f48bc006650 tx=0x7f48bc009a80).ready entity=mon.1 client_cookie=52cb2a10e5423770 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:01.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.422 7f48ca05d700 1 -- 172.21.15.32:0/3003738926 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48bc0009b0 con 0x7f48cc10d010 2021-04-09T18:22:01.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.422 7f48ca05d700 1 -- 172.21.15.32:0/3003738926 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc00c0c0 con 0x7f48cc10d010 2021-04-09T18:22:01.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.422 7f48ca05d700 1 -- 172.21.15.32:0/3003738926 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48bc00c5e0 con 0x7f48cc10d010 2021-04-09T18:22:01.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.422 7f48d0b9a700 1 -- 172.21.15.32:0/3003738926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48cc10d010 msgr2=0x7f48cc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.422 7f48d0b9a700 1 --2- 172.21.15.32:0/3003738926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48cc10d010 0x7f48cc0d1780 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f48bc006650 tx=0x7f48bc009a80).stop 2021-04-09T18:22:01.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.423 7f48d0b9a700 1 -- 172.21.15.32:0/3003738926 shutdown_connections 2021-04-09T18:22:01.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.423 7f48d0b9a700 1 --2- 172.21.15.32:0/3003738926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc0d8490 0x7f48cc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.423 7f48d0b9a700 1 --2- 172.21.15.32:0/3003738926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48cc10d010 0x7f48cc0d1780 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.423 7f48d0b9a700 1 -- 172.21.15.32:0/3003738926 >> 172.21.15.32:0/3003738926 conn(0x7f48cc0ce5f0 msgr2=0x7f48cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:01.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.423 7f48d0b9a700 1 -- 172.21.15.32:0/3003738926 shutdown_connections 2021-04-09T18:22:01.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.423 7f48d0b9a700 1 -- 172.21.15.32:0/3003738926 wait complete. 2021-04-09T18:22:01.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.443 7f48d0b9a700 1 Processor -- start 2021-04-09T18:22:01.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.443 7f48d0b9a700 1 -- start start 2021-04-09T18:22:01.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48d0b9a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc09c5b0 0x7f48cc09d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48d0b9a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48cc09c9b0 0x7f48cc09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48d0b9a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48cc090390 0x7f48cc090790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48d0b9a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc09e290 con 0x7f48cc09c9b0 2021-04-09T18:22:01.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48d0b9a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc09cf90 con 0x7f48cc090390 2021-04-09T18:22:01.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48d0b9a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc090db0 con 0x7f48cc09c5b0 2021-04-09T18:22:01.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48c205d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48cc09c9b0 0x7f48cc09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48c205d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48cc09c9b0 0x7f48cc09db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48908/0 (socket says 172.21.15.32:48908) 2021-04-09T18:22:01.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48c205d700 1 -- 172.21.15.32:0/1986745560 learned_addr learned my addr 172.21.15.32:0/1986745560 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:01.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48ca85e700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc09c5b0 0x7f48cc09d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48cb05f700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48cc090390 0x7f48cc090790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48c205d700 1 -- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc09c5b0 msgr2=0x7f48cc09d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48c205d700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc09c5b0 0x7f48cc09d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48c205d700 1 -- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48cc090390 msgr2=0x7f48cc090790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:01.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48c205d700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48cc090390 0x7f48cc090790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:01.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48c205d700 1 -- 172.21.15.32:0/1986745560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc090f90 con 0x7f48cc09c9b0 2021-04-09T18:22:01.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.444 7f48c205d700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48cc09c9b0 0x7f48cc09db30 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f48b4001400 tx=0x7f48b400a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:01.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.445 7f48c3fff700 1 -- 172.21.15.32:0/1986745560 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b4000ef0 con 0x7f48cc09c9b0 2021-04-09T18:22:01.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.445 7f48c3fff700 1 -- 172.21.15.32:0/1986745560 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b400acf0 con 0x7f48cc09c9b0 2021-04-09T18:22:01.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.445 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48cc090f90 con 0x7f48cc09c9b0 2021-04-09T18:22:01.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.445 7f48c3fff700 1 -- 172.21.15.32:0/1986745560 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b4004590 con 0x7f48cc09c9b0 2021-04-09T18:22:01.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.445 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48cc090f90 con 0x7f48cc09c9b0 2021-04-09T18:22:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.445 7f48c3fff700 1 -- 172.21.15.32:0/1986745560 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f48b40043a0 con 0x7f48cc09c9b0 2021-04-09T18:22:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.445 7f48c3fff700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48b0018930 0x7f48b001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:01.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.445 7f48c3fff700 1 -- 172.21.15.32:0/1986745560 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f48b40329d0 con 0x7f48cc09c9b0 2021-04-09T18:22:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.446 7f48ca85e700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48b0018930 0x7f48b001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:01.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.446 7f48ca85e700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48b0018930 0x7f48b001ae10 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:01.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.463 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48cc10d480 con 0x7f48cc09c9b0 2021-04-09T18:22:01.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.466 7f48c3fff700 1 -- 172.21.15.32:0/1986745560 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f48b40009b0 con 0x7f48cc09c9b0 2021-04-09T18:22:01.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:01.660 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f48cc10d480 con 0x7f48cc09c9b0 2021-04-09T18:22:02.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.424 7f48c3fff700 1 -- 172.21.15.32:0/1986745560 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v109) v1 ==== 105+0+0 (secure 0 0 0) 0x7f48b401f710 con 0x7f48cc09c9b0 2021-04-09T18:22:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48b0018930 msgr2=0x7f48b001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:02.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48b0018930 0x7f48b001ae10 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48cc09c9b0 msgr2=0x7f48cc09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:02.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48cc09c9b0 0x7f48cc09db30 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f48b4001400 tx=0x7f48b400a900).stop 2021-04-09T18:22:02.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 shutdown_connections 2021-04-09T18:22:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48b0018930 0x7f48b001ae10 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48cc09c9b0 0x7f48cc09db30 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48cc090390 0x7f48cc090790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 --2- 172.21.15.32:0/1986745560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48cc09c5b0 0x7f48cc09d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 >> 172.21.15.32:0/1986745560 conn(0x7f48cc0b2fa0 msgr2=0x7f48cc0a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 shutdown_connections 2021-04-09T18:22:02.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.429 7f48d0b9a700 1 -- 172.21.15.32:0/1986745560 wait complete. 2021-04-09T18:22:02.493 INFO:tasks.cephfs.test_failover:Waiting for cluster to grow. 2021-04-09T18:22:02.494 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:02.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.597 7f3aa52ba700 1 Processor -- start 2021-04-09T18:22:02.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.597 7f3aa52ba700 1 -- start start 2021-04-09T18:22:02.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3aa52ba700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa010da00 0x7f3aa010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:02.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3aa52ba700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa010d480 con 0x7f3aa00d7ca0 2021-04-09T18:22:02.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3aa52ba700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa00d1f70 con 0x7f3aa010d010 2021-04-09T18:22:02.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3aa52ba700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa00d9220 con 0x7f3aa010da00 2021-04-09T18:22:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9e87c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3aa00d7ca0 0x7f3aa00d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58504/0 (socket says 172.21.15.32:58504) 2021-04-09T18:22:02.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9e87c700 1 -- 172.21.15.32:0/687808118 learned_addr learned my addr 172.21.15.32:0/687808118 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9f87e700 1 --2- 172.21.15.32:0/687808118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa010da00 0x7f3aa010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9e07b700 1 -- 172.21.15.32:0/687808118 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814644229 0 0) 0x7f3a94001060 con 0x7f3aa00d7ca0 2021-04-09T18:22:02.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9e07b700 1 -- 172.21.15.32:0/687808118 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a88002fe0 con 0x7f3aa00d7ca0 2021-04-09T18:22:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9e07b700 1 -- 172.21.15.32:0/687808118 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449798000 0 0) 0x7f3a90000c60 con 0x7f3aa010d010 2021-04-09T18:22:02.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9e07b700 1 -- 172.21.15.32:0/687808118 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a880042c0 con 0x7f3aa010d010 2021-04-09T18:22:02.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9f87e700 1 -- 172.21.15.32:0/687808118 >> v1:172.21.15.86:6789/0 conn(0x7f3aa010d010 legacy=0x7f3aa00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:02.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9f87e700 1 -- 172.21.15.32:0/687808118 >> v1:172.21.15.32:6789/0 conn(0x7f3aa00d7ca0 legacy=0x7f3aa00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:02.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9f87e700 1 -- 172.21.15.32:0/687808118 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa010e420 con 0x7f3aa010da00 2021-04-09T18:22:02.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.598 7f3a9f87e700 1 --2- 172.21.15.32:0/687808118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa010da00 0x7f3aa010de00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3a98006ad0 tx=0x7f3a98009ad0).ready entity=mon.2 client_cookie=2a011016af8ad863 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3a9e07b700 1 -- 172.21.15.32:0/687808118 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a980022e0 con 0x7f3aa010da00 2021-04-09T18:22:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3a9e07b700 1 -- 172.21.15.32:0/687808118 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a98006410 con 0x7f3aa010da00 2021-04-09T18:22:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3a9e07b700 1 -- 172.21.15.32:0/687808118 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a98002620 con 0x7f3aa010da00 2021-04-09T18:22:02.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3aa52ba700 1 -- 172.21.15.32:0/687808118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa010da00 msgr2=0x7f3aa010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:02.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3aa52ba700 1 --2- 172.21.15.32:0/687808118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa010da00 0x7f3aa010de00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3a98006ad0 tx=0x7f3a98009ad0).stop 2021-04-09T18:22:02.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3aa52ba700 1 -- 172.21.15.32:0/687808118 shutdown_connections 2021-04-09T18:22:02.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3aa52ba700 1 --2- 172.21.15.32:0/687808118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa010da00 0x7f3aa010de00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3aa52ba700 1 -- 172.21.15.32:0/687808118 >> 172.21.15.32:0/687808118 conn(0x7f3aa00ce5f0 msgr2=0x7f3aa00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:02.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3aa52ba700 1 -- 172.21.15.32:0/687808118 shutdown_connections 2021-04-09T18:22:02.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.599 7f3aa52ba700 1 -- 172.21.15.32:0/687808118 wait complete. 2021-04-09T18:22:02.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.619 7f3aa52ba700 1 Processor -- start 2021-04-09T18:22:02.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3aa52ba700 1 -- start start 2021-04-09T18:22:02.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3aa52ba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aa00a80d0 0x7f3aa01a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:02.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3aa52ba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa01a4ce0 0x7f3aa01a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:02.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3aa52ba700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa01a5d70 0x7f3aa008ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:02.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3aa52ba700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa01a5250 con 0x7f3aa01a4ce0 2021-04-09T18:22:02.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3aa52ba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa01a61b0 con 0x7f3aa00a80d0 2021-04-09T18:22:02.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3aa52ba700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa018be80 con 0x7f3aa01a5d70 2021-04-09T18:22:02.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3a9e87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa01a4ce0 0x7f3aa01a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:02.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3a9e87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa01a4ce0 0x7f3aa01a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48916/0 (socket says 172.21.15.32:48916) 2021-04-09T18:22:02.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3a9e87c700 1 -- 172.21.15.32:0/3885445470 learned_addr learned my addr 172.21.15.32:0/3885445470 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:02.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3a9f87e700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa01a5d70 0x7f3aa008ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:02.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.620 7f3a9f07d700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aa00a80d0 0x7f3aa01a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:02.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a9f87e700 1 -- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aa00a80d0 msgr2=0x7f3aa01a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:02.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a9f87e700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aa00a80d0 0x7f3aa01a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a9f87e700 1 -- 172.21.15.32:0/3885445470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa01a4ce0 msgr2=0x7f3aa01a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:02.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a9f87e700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa01a4ce0 0x7f3aa01a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a9f87e700 1 -- 172.21.15.32:0/3885445470 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa008b2b0 con 0x7f3aa01a5d70 2021-04-09T18:22:02.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a9e87c700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa01a4ce0 0x7f3aa01a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:02.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a9f07d700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aa00a80d0 0x7f3aa01a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a9f87e700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa01a5d70 0x7f3aa008ac90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f3a98009060 tx=0x7f3a9800b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a87fff700 1 -- 172.21.15.32:0/3885445470 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a9800bcd0 con 0x7f3aa01a5d70 2021-04-09T18:22:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a87fff700 1 -- 172.21.15.32:0/3885445470 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a98000a70 con 0x7f3aa01a5d70 2021-04-09T18:22:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa008b2b0 con 0x7f3aa01a5d70 2021-04-09T18:22:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3a87fff700 1 -- 172.21.15.32:0/3885445470 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a98002240 con 0x7f3aa01a5d70 2021-04-09T18:22:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.621 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa008b2b0 con 0x7f3aa01a5d70 2021-04-09T18:22:02.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.622 7f3a87fff700 1 -- 172.21.15.32:0/3885445470 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3a980092e0 con 0x7f3aa01a5d70 2021-04-09T18:22:02.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.622 7f3a87fff700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a80018930 0x7f3a8001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:02.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.622 7f3a87fff700 1 -- 172.21.15.32:0/3885445470 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3a98032980 con 0x7f3aa01a5d70 2021-04-09T18:22:02.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.622 7f3a9f07d700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a80018930 0x7f3a8001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:02.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.623 7f3a9f07d700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a80018930 0x7f3a8001adc0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:02.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.642 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa01b7050 con 0x7f3aa01a5d70 2021-04-09T18:22:02.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.646 7f3a87fff700 1 -- 172.21.15.32:0/3885445470 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3a98000a70 con 0x7f3aa01a5d70 2021-04-09T18:22:02.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.835 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3aa01b7050 con 0x7f3aa01a5d70 2021-04-09T18:22:02.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.836 7f3a87fff700 1 -- 172.21.15.32:0/3885445470 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+2797 (secure 0 0 0) 0x7f3a980092e0 con 0x7f3aa01a5d70 2021-04-09T18:22:02.887 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 110 2021-04-09T18:22:02.888 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:02.888 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":110,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:02.424428","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1],"up":{"mds_0":7387,"mds_1":7390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:creating","state_seq":2,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:02.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a80018930 msgr2=0x7f3a8001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:02.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a80018930 0x7f3a8001adc0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa01a5d70 msgr2=0x7f3aa008ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:02.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa01a5d70 0x7f3aa008ac90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f3a98009060 tx=0x7f3a9800b8e0).stop 2021-04-09T18:22:02.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 shutdown_connections 2021-04-09T18:22:02.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a80018930 0x7f3a8001adc0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aa01a4ce0 0x7f3aa01a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aa01a5d70 0x7f3aa008ac90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 --2- 172.21.15.32:0/3885445470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aa00a80d0 0x7f3aa01a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:02.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.838 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 >> 172.21.15.32:0/3885445470 conn(0x7f3aa0190570 msgr2=0x7f3aa018ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:02.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.839 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 shutdown_connections 2021-04-09T18:22:02.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:02.839 7f3aa52ba700 1 -- 172.21.15.32:0/3885445470 wait complete. 2021-04-09T18:22:03.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 110, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:02.424428', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 7387, 'mds_1': 7390}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:creating', 'state_seq': 2, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:03.303 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7390:up:creating 2021-04-09T18:22:04.305 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:04.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.414 7f04a99c2700 1 Processor -- start 2021-04-09T18:22:04.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.414 7f04a99c2700 1 -- start start 2021-04-09T18:22:04.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.415 7f04a99c2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f04a410d010 0x7f04a40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:04.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.415 7f04a99c2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a410d480 con 0x7f04a40d8490 2021-04-09T18:22:04.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.415 7f04a99c2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a410e1f0 con 0x7f04a40d1da0 2021-04-09T18:22:04.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.415 7f04a99c2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a40d8d30 con 0x7f04a410d010 2021-04-09T18:22:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.415 7f04a3d75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f04a40d8490 0x7f04a410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58528/0 (socket says 172.21.15.32:58528) 2021-04-09T18:22:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.415 7f04a3d75700 1 -- 172.21.15.32:0/1827444734 learned_addr learned my addr 172.21.15.32:0/1827444734 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.415 7f04a3574700 1 --2- 172.21.15.32:0/1827444734 >> v2:172.21.15.86:3300/0 conn(0x7f04a410d010 0x7f04a40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.415 7f04a2572700 1 -- 172.21.15.32:0/1827444734 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920968020 0 0) 0x7f049c000d30 con 0x7f04a40d8490 2021-04-09T18:22:04.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.415 7f04a2572700 1 -- 172.21.15.32:0/1827444734 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c002fe0 con 0x7f04a40d8490 2021-04-09T18:22:04.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229157614 0 0) 0x7f0498001080 con 0x7f04a40d1da0 2021-04-09T18:22:04.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c003210 con 0x7f04a40d1da0 2021-04-09T18:22:04.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3392508364 0 0) 0x7f049c001460 con 0x7f04a40d8490 2021-04-09T18:22:04.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 >> v2:172.21.15.86:3300/0 conn(0x7f04a410d010 msgr2=0x7f04a40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:04.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 --2- 172.21.15.32:0/1827444734 >> v2:172.21.15.86:3300/0 conn(0x7f04a410d010 0x7f04a40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 >> v1:172.21.15.86:6789/0 conn(0x7f04a40d1da0 legacy=0x7f04a40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:04.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a40d9220 con 0x7f04a40d8490 2021-04-09T18:22:04.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f049c001980 con 0x7f04a40d8490 2021-04-09T18:22:04.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 >> v1:172.21.15.32:6789/0 conn(0x7f04a40d8490 legacy=0x7f04a410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:04.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048c003b90 0x7f048c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:04.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048c006770 0x7f048c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:04.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048c009590 0x7f048c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:04.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c050310 con 0x7f048c006770 2021-04-09T18:22:04.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c0504f0 con 0x7f048c003b90 2021-04-09T18:22:04.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048c050710 con 0x7f048c009590 2021-04-09T18:22:04.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2d73700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048c006770 0x7f048c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:04.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a3574700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048c003b90 0x7f048c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:04.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a3d75700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048c009590 0x7f048c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:04.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2d73700 1 -- 172.21.15.32:0/1827444734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048c009590 msgr2=0x7f048c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:04.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2d73700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048c009590 0x7f048c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2d73700 1 -- 172.21.15.32:0/1827444734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048c003b90 msgr2=0x7f048c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:04.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2d73700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048c003b90 0x7f048c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2d73700 1 -- 172.21.15.32:0/1827444734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c0509b0 con 0x7f048c006770 2021-04-09T18:22:04.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a3d75700 1 -- 172.21.15.32:0/1827444734 reap_dead start 2021-04-09T18:22:04.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2d73700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048c006770 0x7f048c008c90 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7f0498006ad0 tx=0x7f0498009b40).ready entity=mon.0 client_cookie=8c80e96e94b9611b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:04.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0498000c10 con 0x7f048c006770 2021-04-09T18:22:04.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049800b2e0 con 0x7f048c006770 2021-04-09T18:22:04.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a2572700 1 -- 172.21.15.32:0/1827444734 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f049800b7c0 con 0x7f048c006770 2021-04-09T18:22:04.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.416 7f04a99c2700 1 -- 172.21.15.32:0/1827444734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048c006770 msgr2=0x7f048c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:04.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.417 7f04a99c2700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048c006770 0x7f048c008c90 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7f0498006ad0 tx=0x7f0498009b40).stop 2021-04-09T18:22:04.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.417 7f04a99c2700 1 -- 172.21.15.32:0/1827444734 shutdown_connections 2021-04-09T18:22:04.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.417 7f04a99c2700 1 --2- 172.21.15.32:0/1827444734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048c006770 0x7f048c008c90 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.417 7f04a99c2700 1 -- 172.21.15.32:0/1827444734 >> 172.21.15.32:0/1827444734 conn(0x7f04a40ce5f0 msgr2=0x7f04a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:04.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.417 7f04a99c2700 1 -- 172.21.15.32:0/1827444734 shutdown_connections 2021-04-09T18:22:04.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.417 7f04a99c2700 1 -- 172.21.15.32:0/1827444734 wait complete. 2021-04-09T18:22:04.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.438 7f04a99c2700 1 Processor -- start 2021-04-09T18:22:04.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.438 7f04a99c2700 1 -- start start 2021-04-09T18:22:04.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.438 7f04a99c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a40b8460 0x7f04a40b8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:04.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.438 7f04a99c2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04a4191ee0 0x7f04a41922e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:04.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.438 7f04a99c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a418e640 0x7f04a418ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:04.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.438 7f04a99c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a4192900 con 0x7f04a40b8460 2021-04-09T18:22:04.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.438 7f04a99c2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a418f060 con 0x7f04a418e640 2021-04-09T18:22:04.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.438 7f04a99c2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a418f280 con 0x7f04a4191ee0 2021-04-09T18:22:04.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a40b8460 0x7f04a40b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:04.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a40b8460 0x7f04a40b8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48950/0 (socket says 172.21.15.32:48950) 2021-04-09T18:22:04.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3574700 1 -- 172.21.15.32:0/3733291966 learned_addr learned my addr 172.21.15.32:0/3733291966 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:04.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3d75700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a418e640 0x7f04a418ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a2d73700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04a4191ee0 0x7f04a41922e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3574700 1 -- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04a4191ee0 msgr2=0x7f04a41922e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3574700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04a4191ee0 0x7f04a41922e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3574700 1 -- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a418e640 msgr2=0x7f04a418ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3574700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a418e640 0x7f04a418ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3574700 1 -- 172.21.15.32:0/3733291966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a4190100 con 0x7f04a40b8460 2021-04-09T18:22:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a3574700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a40b8460 0x7f04a40b8860 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f0494006f60 tx=0x7f049400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f048bfff700 1 -- 172.21.15.32:0/3733291966 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04940009b0 con 0x7f04a40b8460 2021-04-09T18:22:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f048bfff700 1 -- 172.21.15.32:0/3733291966 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049400bd60 con 0x7f04a40b8460 2021-04-09T18:22:04.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a4190100 con 0x7f04a40b8460 2021-04-09T18:22:04.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f048bfff700 1 -- 172.21.15.32:0/3733291966 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0494002030 con 0x7f04a40b8460 2021-04-09T18:22:04.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.439 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a4190100 con 0x7f04a40b8460 2021-04-09T18:22:04.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.441 7f048bfff700 1 -- 172.21.15.32:0/3733291966 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f04940052e0 con 0x7f04a40b8460 2021-04-09T18:22:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.441 7f048bfff700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04840188c0 0x7f048401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.441 7f048bfff700 1 -- 172.21.15.32:0/3733291966 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0494032770 con 0x7f04a40b8460 2021-04-09T18:22:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.441 7f04a2d73700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04840188c0 0x7f048401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.441 7f04a2d73700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04840188c0 0x7f048401adc0 crc :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:04.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.459 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a41919b0 con 0x7f04a40b8460 2021-04-09T18:22:04.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.463 7f048bfff700 1 -- 172.21.15.32:0/3733291966 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0494006b00 con 0x7f04a40b8460 2021-04-09T18:22:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.654 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04a41919b0 con 0x7f04a40b8460 2021-04-09T18:22:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.655 7f048bfff700 1 -- 172.21.15.32:0/3733291966 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2809 (secure 0 0 0) 0x7f049400bd60 con 0x7f04a40b8460 2021-04-09T18:22:04.707 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 113 2021-04-09T18:22:04.708 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:04.708 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":113,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:04.434224","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7387,"mds_1":7390,"mds_2":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7575":{"gid":7575,"name":"a","rank":2,"incarnation":112,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:04.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04840188c0 msgr2=0x7f048401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:04.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04840188c0 0x7f048401adc0 crc :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a40b8460 msgr2=0x7f04a40b8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:04.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a40b8460 0x7f04a40b8860 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f0494006f60 tx=0x7f049400b970).stop 2021-04-09T18:22:04.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 shutdown_connections 2021-04-09T18:22:04.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04840188c0 0x7f048401adc0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a418e640 0x7f04a418ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04a4191ee0 0x7f04a41922e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 --2- 172.21.15.32:0/3733291966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a40b8460 0x7f04a40b8860 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:04.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.657 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 >> 172.21.15.32:0/3733291966 conn(0x7f04a408fb70 msgr2=0x7f04a419ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:04.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.658 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 shutdown_connections 2021-04-09T18:22:04.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:04.658 7f04a99c2700 1 -- 172.21.15.32:0/3733291966 wait complete. 2021-04-09T18:22:05.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 113, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:04.434224', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 7387, 'mds_1': 7390, 'mds_2': 7575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7575': {'gid': 7575, 'name': 'a', 'rank': 2, 'incarnation': 112, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6834/4090970868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 4090970868}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 4090970868}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:05.123 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2021-04-09T18:22:05.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:22:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:22:05.254 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:22:05.254 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:22:05.255 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 7387, 2021-04-09T18:22:05.255 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:22:05.255 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:22:05.256 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 12.869230443999999, 2021-04-09T18:22:05.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 113, 2021-04-09T18:22:05.256 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 95, 2021-04-09T18:22:05.256 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 95, 2021-04-09T18:22:05.257 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 17.308828046999999 2021-04-09T18:22:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:22:05.259 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 7387, "want_state": "up:active", "state": "up:active", "rank_uptime": 12.869230443999999, "mdsmap_epoch": 113, "osdmap_epoch": 95, "osdmap_epoch_barrier": 95, "uptime": 17.308828046999999 } 2021-04-09T18:22:05.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:22:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:22:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:22:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 1, 2021-04-09T18:22:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 7390, 2021-04-09T18:22:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:22:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:22:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 2.9045632690000001, 2021-04-09T18:22:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 113, 2021-04-09T18:22:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 95, 2021-04-09T18:22:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 95, 2021-04-09T18:22:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 17.439768938 2021-04-09T18:22:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:22:05.392 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 7390, "want_state": "up:active", "state": "up:active", "rank_uptime": 2.9045632690000001, "mdsmap_epoch": 113, "osdmap_epoch": 95, "osdmap_epoch_barrier": 95, "uptime": 17.439768938 } 2021-04-09T18:22:05.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:22:05.538 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:22:05.538 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:22:05.538 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 2, 2021-04-09T18:22:05.539 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 7575, 2021-04-09T18:22:05.539 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:22:05.539 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:22:05.540 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 2.052688142, 2021-04-09T18:22:05.540 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 113, 2021-04-09T18:22:05.541 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 95, 2021-04-09T18:22:05.541 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 95, 2021-04-09T18:22:05.541 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 17.585288444 2021-04-09T18:22:05.542 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:22:05.544 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 2, "id": 7575, "want_state": "up:active", "state": "up:active", "rank_uptime": 2.052688142, "mdsmap_epoch": 113, "osdmap_epoch": 95, "osdmap_epoch_barrier": 95, "uptime": 17.585288444 } 2021-04-09T18:22:05.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:22:05.671 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:22:05.672 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:22:05.672 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:22:05.678 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:22:05.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:05.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.775 7fb3c2802700 1 Processor -- start 2021-04-09T18:22:05.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.775 7fb3c2802700 1 -- start start 2021-04-09T18:22:05.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3c2802700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc10d010 0x7fb3bc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:05.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3c2802700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3bc0d1da0 0x7fb3bc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:05.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3c2802700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3bc10d480 con 0x7fb3bc0d82c0 2021-04-09T18:22:05.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3c2802700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3bc10e780 con 0x7fb3bc0d1da0 2021-04-09T18:22:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3c2802700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0d8d30 con 0x7fb3bc10d010 2021-04-09T18:22:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3c0e1f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb3bc0d82c0 0x7fb3bc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58550/0 (socket says 172.21.15.32:58550) 2021-04-09T18:22:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3c0e1f700 1 -- 172.21.15.32:0/1589690639 learned_addr learned my addr 172.21.15.32:0/1589690639 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3b3fff700 1 --2- 172.21.15.32:0/1589690639 >> v2:172.21.15.86:3300/0 conn(0x7fb3bc0d1da0 0x7fb3bc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3bbfff700 1 --2- 172.21.15.32:0/1589690639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc10d010 0x7fb3bc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.776 7fb3bb7fe700 1 -- 172.21.15.32:0/1589690639 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635016557 0 0) 0x7fb3b4000d30 con 0x7fb3bc0d82c0 2021-04-09T18:22:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3bb7fe700 1 -- 172.21.15.32:0/1589690639 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a8002fe0 con 0x7fb3bc0d82c0 2021-04-09T18:22:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3b3fff700 1 -- 172.21.15.32:0/1589690639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc10d010 msgr2=0x7fb3bc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3b3fff700 1 --2- 172.21.15.32:0/1589690639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc10d010 0x7fb3bc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3b3fff700 1 -- 172.21.15.32:0/1589690639 >> v1:172.21.15.32:6789/0 conn(0x7fb3bc0d82c0 legacy=0x7fb3bc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3b3fff700 1 -- 172.21.15.32:0/1589690639 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc0d9220 con 0x7fb3bc0d1da0 2021-04-09T18:22:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3bbfff700 1 --2- 172.21.15.32:0/1589690639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc10d010 0x7fb3bc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3b3fff700 1 --2- 172.21.15.32:0/1589690639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0d1da0 0x7fb3bc0d7ca0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fb3a4006650 tx=0x7fb3a4009a80).ready entity=mon.1 client_cookie=4a2437e8f715b700 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3bb7fe700 1 -- 172.21.15.32:0/1589690639 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3a40009b0 con 0x7fb3bc0d1da0 2021-04-09T18:22:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3bb7fe700 1 -- 172.21.15.32:0/1589690639 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a400c0c0 con 0x7fb3bc0d1da0 2021-04-09T18:22:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3bb7fe700 1 -- 172.21.15.32:0/1589690639 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3a400c510 con 0x7fb3bc0d1da0 2021-04-09T18:22:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3c2802700 1 -- 172.21.15.32:0/1589690639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0d1da0 msgr2=0x7fb3bc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.777 7fb3c2802700 1 --2- 172.21.15.32:0/1589690639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0d1da0 0x7fb3bc0d7ca0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fb3a4006650 tx=0x7fb3a4009a80).stop 2021-04-09T18:22:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.778 7fb3c2802700 1 -- 172.21.15.32:0/1589690639 shutdown_connections 2021-04-09T18:22:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.778 7fb3c2802700 1 --2- 172.21.15.32:0/1589690639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0d1da0 0x7fb3bc0d7ca0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.778 7fb3c2802700 1 --2- 172.21.15.32:0/1589690639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc10d010 0x7fb3bc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.778 7fb3c2802700 1 -- 172.21.15.32:0/1589690639 >> 172.21.15.32:0/1589690639 conn(0x7fb3bc0ce5f0 msgr2=0x7fb3bc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.778 7fb3c2802700 1 -- 172.21.15.32:0/1589690639 shutdown_connections 2021-04-09T18:22:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.778 7fb3c2802700 1 -- 172.21.15.32:0/1589690639 wait complete. 2021-04-09T18:22:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.798 7fb3c2802700 1 Processor -- start 2021-04-09T18:22:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.798 7fb3c2802700 1 -- start start 2021-04-09T18:22:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3c2802700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0a90c0 0x7fb3bc1a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3c2802700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3bc1a76d0 0x7fb3bc1a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3c2802700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc17bce0 0x7fb3bc17c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3c2802700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc1a7c60 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3c2802700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc1a8e20 con 0x7fb3bc0a90c0 2021-04-09T18:22:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3c2802700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc17db80 con 0x7fb3bc17bce0 2021-04-09T18:22:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3bc1a76d0 0x7fb3bc1a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3b3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3bc1a76d0 0x7fb3bc1a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48966/0 (socket says 172.21.15.32:48966) 2021-04-09T18:22:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3b3fff700 1 -- 172.21.15.32:0/392529945 learned_addr learned my addr 172.21.15.32:0/392529945 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3c0e1f700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc17bce0 0x7fb3bc17c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3bbfff700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0a90c0 0x7fb3bc1a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:05.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3b3fff700 1 -- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc17bce0 msgr2=0x7fb3bc17c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:05.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3b3fff700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc17bce0 0x7fb3bc17c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:05.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3b3fff700 1 -- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0a90c0 msgr2=0x7fb3bc1a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:05.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3b3fff700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0a90c0 0x7fb3bc1a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3b3fff700 1 -- 172.21.15.32:0/392529945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc179610 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3bbfff700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0a90c0 0x7fb3bc1a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:22:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3c0e1f700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc17bce0 0x7fb3bc17c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:22:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.799 7fb3b3fff700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3bc1a76d0 0x7fb3bc1a8800 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fb3a4009650 tx=0x7fb3a400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:05.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.800 7fb3b97fa700 1 -- 172.21.15.32:0/392529945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3a40093a0 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.800 7fb3b97fa700 1 -- 172.21.15.32:0/392529945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a400bd50 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.800 7fb3c2802700 1 -- 172.21.15.32:0/392529945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc179610 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.800 7fb3b97fa700 1 -- 172.21.15.32:0/392529945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3a40052e0 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.800 7fb3c2802700 1 -- 172.21.15.32:0/392529945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc179610 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.800 7fb3b97fa700 1 -- 172.21.15.32:0/392529945 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb3a4006030 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.800 7fb3b97fa700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3a00188d0 0x7fb3a001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.800 7fb3b97fa700 1 -- 172.21.15.32:0/392529945 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb3a40329e0 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.800 7fb3bbfff700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3a00188d0 0x7fb3a001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:05.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.801 7fb3bbfff700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3a00188d0 0x7fb3a001add0 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:05.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.820 7fb3c2802700 1 -- 172.21.15.32:0/392529945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc1b7010 con 0x7fb3bc1a76d0 2021-04-09T18:22:05.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:05.824 7fb3b97fa700 1 -- 172.21.15.32:0/392529945 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb3a40009c0 con 0x7fb3bc1a76d0 2021-04-09T18:22:06.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.017 7fb3c2802700 1 -- 172.21.15.32:0/392529945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3bc10d480 con 0x7fb3bc1a76d0 2021-04-09T18:22:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.018 7fb3b97fa700 1 -- 172.21.15.32:0/392529945 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2809 (secure 0 0 0) 0x7fb3a400bd50 con 0x7fb3bc1a76d0 2021-04-09T18:22:06.070 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 113 2021-04-09T18:22:06.071 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:06.071 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":113,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:04.434224","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7387,"mds_1":7390,"mds_2":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7575":{"gid":7575,"name":"a","rank":2,"incarnation":112,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 -- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3a00188d0 msgr2=0x7fb3a001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3a00188d0 0x7fb3a001add0 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 -- 172.21.15.32:0/392529945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3bc1a76d0 msgr2=0x7fb3bc1a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3bc1a76d0 0x7fb3bc1a8800 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fb3a4009650 tx=0x7fb3a400b960).stop 2021-04-09T18:22:06.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 -- 172.21.15.32:0/392529945 shutdown_connections 2021-04-09T18:22:06.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3a00188d0 0x7fb3a001add0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3bc1a76d0 0x7fb3bc1a8800 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc17bce0 0x7fb3bc17c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 --2- 172.21.15.32:0/392529945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc0a90c0 0x7fb3bc1a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 -- 172.21.15.32:0/392529945 >> 172.21.15.32:0/392529945 conn(0x7fb3bc1acae0 msgr2=0x7fb3bc17ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 -- 172.21.15.32:0/392529945 shutdown_connections 2021-04-09T18:22:06.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.020 7fb3c2802700 1 -- 172.21.15.32:0/392529945 wait complete. 2021-04-09T18:22:06.083 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 113, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 8, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:21:52.329633", "damaged": [], "data_pools": [ 17 ], "enabled": true, "epoch": 113, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_7387": { "addr": "172.21.15.86:6835/3998950443", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 3998950443, "type": "v2" }, { "addr": "172.21.15.86:6835", "nonce": 3998950443, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 7387, "incarnation": 106, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 }, "gid_7390": { "addr": "172.21.15.86:6837/867563018", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6836", "nonce": 867563018, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 867563018, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 7390, "incarnation": 110, "name": "d", "rank": 1, "state": "up:active", "state_seq": 5 }, "gid_7575": { "addr": "172.21.15.32:6834/4090970868", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 4090970868, "type": "v2" }, { "addr": "172.21.15.32:6834", "nonce": 4090970868, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 7575, "incarnation": 112, "name": "a", "rank": 2, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 3, "metadata_pool": 16, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:22:04.434224", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7387, "mds_1": 7390, "mds_2": 7575 } } } ], "standbys": [ { "addr": "172.21.15.32:6835/2579974780", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6833", "nonce": 2579974780, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 2579974780, "type": "v1" } ] }, "epoch": 105, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 56423, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:22:06.084 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:06.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.180 7f378e23e700 1 Processor -- start 2021-04-09T18:22:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.180 7f378e23e700 1 -- start start 2021-04-09T18:22:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f378e23e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f378810da00 0x7f378810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f378e23e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378810d480 con 0x7f378810d010 2021-04-09T18:22:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f378e23e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37880d1f70 con 0x7f37880d7ca0 2021-04-09T18:22:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f378e23e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37880d9220 con 0x7f378810da00 2021-04-09T18:22:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f3787fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f378810d010 0x7f37880d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58562/0 (socket says 172.21.15.32:58562) 2021-04-09T18:22:06.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f3787fff700 1 -- 172.21.15.32:0/2320655534 learned_addr learned my addr 172.21.15.32:0/2320655534 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f378c85b700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f378810da00 0x7f378810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f37877fe700 1 -- 172.21.15.32:0/2320655534 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551225749 0 0) 0x7f3778000ce0 con 0x7f378810d010 2021-04-09T18:22:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f37877fe700 1 -- 172.21.15.32:0/2320655534 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3774002fe0 con 0x7f378810d010 2021-04-09T18:22:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f37877fe700 1 -- 172.21.15.32:0/2320655534 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3045653377 0 0) 0x7f3778000ed0 con 0x7f378810d010 2021-04-09T18:22:06.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f37877fe700 1 -- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f378810da00 msgr2=0x7f378810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f37877fe700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f378810da00 0x7f378810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f37877fe700 1 -- 172.21.15.32:0/2320655534 >> v1:172.21.15.86:6789/0 conn(0x7f37880d7ca0 legacy=0x7f37880d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f37877fe700 1 -- 172.21.15.32:0/2320655534 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378810e420 con 0x7f378810d010 2021-04-09T18:22:06.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f37877fe700 1 -- 172.21.15.32:0/2320655534 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f3778001600 con 0x7f378810d010 2021-04-09T18:22:06.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.181 7f37877fe700 1 -- 172.21.15.32:0/2320655534 >> v1:172.21.15.32:6789/0 conn(0x7f378810d010 legacy=0x7f37880d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f37877fe700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3774003a70 0x7f3774005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f37877fe700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3774006780 0x7f3774008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f37877fe700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3774009510 0x7f377400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f37877fe700 1 -- 172.21.15.32:0/2320655534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377400c2e0 con 0x7f3774003a70 2021-04-09T18:22:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f37877fe700 1 -- 172.21.15.32:0/2320655534 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377400c500 con 0x7f3774009510 2021-04-09T18:22:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f37877fe700 1 -- 172.21.15.32:0/2320655534 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3774050730 con 0x7f3774006780 2021-04-09T18:22:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f3787fff700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3774003a70 0x7f3774005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f378c85b700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3774009510 0x7f377400ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f377ffff700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3774006780 0x7f3774008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f3787fff700 1 -- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3774006780 msgr2=0x7f3774008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f3787fff700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3774006780 0x7f3774008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f3787fff700 1 -- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3774009510 msgr2=0x7f377400ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f3787fff700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3774009510 0x7f377400ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f3787fff700 1 -- 172.21.15.32:0/2320655534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3774050990 con 0x7f3774003a70 2021-04-09T18:22:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f3787fff700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3774003a70 0x7f3774005f00 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f3778000b80 tx=0x7f3778007c70).ready entity=mon.0 client_cookie=db366b7c36bf38d6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f37877fe700 1 -- 172.21.15.32:0/2320655534 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37780070c0 con 0x7f3774003a70 2021-04-09T18:22:06.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f37877fe700 1 -- 172.21.15.32:0/2320655534 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37780010e0 con 0x7f3774003a70 2021-04-09T18:22:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f37877fe700 1 -- 172.21.15.32:0/2320655534 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f377800a710 con 0x7f3774003a70 2021-04-09T18:22:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f378e23e700 1 -- 172.21.15.32:0/2320655534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3774003a70 msgr2=0x7f3774005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f378e23e700 1 --2- 172.21.15.32:0/2320655534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3774003a70 0x7f3774005f00 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f3778000b80 tx=0x7f3778007c70).stop 2021-04-09T18:22:06.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.182 7f378c85b700 1 -- 172.21.15.32:0/2320655534 reap_dead start 2021-04-09T18:22:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.183 7f378e23e700 1 -- 172.21.15.32:0/2320655534 shutdown_connections 2021-04-09T18:22:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.183 7f378e23e700 1 -- 172.21.15.32:0/2320655534 >> 172.21.15.32:0/2320655534 conn(0x7f37880ce5f0 msgr2=0x7f37880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.183 7f378e23e700 1 -- 172.21.15.32:0/2320655534 shutdown_connections 2021-04-09T18:22:06.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.183 7f378e23e700 1 -- 172.21.15.32:0/2320655534 wait complete. 2021-04-09T18:22:06.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.203 7f378e23e700 1 Processor -- start 2021-04-09T18:22:06.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.203 7f378e23e700 1 -- start start 2021-04-09T18:22:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f378e23e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3788188780 0x7f3788188b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f378e23e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37881996c0 0x7f3788199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f378e23e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f378808de20 0x7f378808e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f378e23e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378819a0e0 con 0x7f3788188780 2021-04-09T18:22:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f378e23e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378808e840 con 0x7f378808de20 2021-04-09T18:22:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f378e23e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378808ea60 con 0x7f37881996c0 2021-04-09T18:22:06.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f3787fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3788188780 0x7f3788188b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f3787fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3788188780 0x7f3788188b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48984/0 (socket says 172.21.15.32:48984) 2021-04-09T18:22:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f3787fff700 1 -- 172.21.15.32:0/2443142881 learned_addr learned my addr 172.21.15.32:0/2443142881 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f377ffff700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37881996c0 0x7f3788199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f378c85b700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f378808de20 0x7f378808e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f3787fff700 1 -- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37881996c0 msgr2=0x7f3788199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f3787fff700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37881996c0 0x7f3788199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f3787fff700 1 -- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f378808de20 msgr2=0x7f378808e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f3787fff700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f378808de20 0x7f378808e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f3787fff700 1 -- 172.21.15.32:0/2443142881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378808ec80 con 0x7f3788188780 2021-04-09T18:22:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.204 7f3787fff700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3788188780 0x7f3788188b80 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f3778009610 tx=0x7f37780099b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.205 7f37857fa700 1 -- 172.21.15.32:0/2443142881 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3778000c90 con 0x7f3788188780 2021-04-09T18:22:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.205 7f37857fa700 1 -- 172.21.15.32:0/2443142881 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37780070d0 con 0x7f3788188780 2021-04-09T18:22:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.205 7f378e23e700 1 -- 172.21.15.32:0/2443142881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378808ec80 con 0x7f3788188780 2021-04-09T18:22:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.205 7f37857fa700 1 -- 172.21.15.32:0/2443142881 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37780043a0 con 0x7f3788188780 2021-04-09T18:22:06.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.205 7f378e23e700 1 -- 172.21.15.32:0/2443142881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378808ec80 con 0x7f3788188780 2021-04-09T18:22:06.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.205 7f37857fa700 1 -- 172.21.15.32:0/2443142881 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f37780043a0 con 0x7f3788188780 2021-04-09T18:22:06.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.206 7f37857fa700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f376c018930 0x7f376c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.206 7f37857fa700 1 -- 172.21.15.32:0/2443142881 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3778032a40 con 0x7f3788188780 2021-04-09T18:22:06.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.206 7f377ffff700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f376c018930 0x7f376c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.206 7f377ffff700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f376c018930 0x7f376c01adc0 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:06.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.225 7f378e23e700 1 -- 172.21.15.32:0/2443142881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378808e840 con 0x7f3788188780 2021-04-09T18:22:06.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.228 7f37857fa700 1 -- 172.21.15.32:0/2443142881 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f37780070d0 con 0x7f3788188780 2021-04-09T18:22:06.328 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:22:06.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:22:06.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.421 7f378e23e700 1 -- 172.21.15.32:0/2443142881 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f378808e840 con 0x7f3788188780 2021-04-09T18:22:06.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.422 7f37857fa700 1 -- 172.21.15.32:0/2443142881 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2809 (secure 0 0 0) 0x7f37780070d0 con 0x7f3788188780 2021-04-09T18:22:06.475 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 113 2021-04-09T18:22:06.476 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:06.476 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":113,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:04.434224","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7387,"mds_1":7390,"mds_2":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7575":{"gid":7575,"name":"a","rank":2,"incarnation":112,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 -- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f376c018930 msgr2=0x7f376c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f376c018930 0x7f376c01adc0 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 -- 172.21.15.32:0/2443142881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3788188780 msgr2=0x7f3788188b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3788188780 0x7f3788188b80 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f3778009610 tx=0x7f37780099b0).stop 2021-04-09T18:22:06.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 -- 172.21.15.32:0/2443142881 shutdown_connections 2021-04-09T18:22:06.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f376c018930 0x7f376c01adc0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f378808de20 0x7f378808e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37881996c0 0x7f3788199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 --2- 172.21.15.32:0/2443142881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3788188780 0x7f3788188b80 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 -- 172.21.15.32:0/2443142881 >> 172.21.15.32:0/2443142881 conn(0x7f37881a8e60 msgr2=0x7f37881a0670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:06.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 -- 172.21.15.32:0/2443142881 shutdown_connections 2021-04-09T18:22:06.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.424 7f378e23e700 1 -- 172.21.15.32:0/2443142881 wait complete. 2021-04-09T18:22:06.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:22:06.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.588 7f27f7b7d700 1 Processor -- start 2021-04-09T18:22:06.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.589 7f27f7b7d700 1 -- start start 2021-04-09T18:22:06.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.589 7f27f7b7d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27f00d6ca0 0x7f27f00d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.589 7f27f7b7d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00d2460 0x7f27f010cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.589 7f27f7b7d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f00d72a0 con 0x7f27f00cffc0 2021-04-09T18:22:06.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.589 7f27f7b7d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f00d70a0 con 0x7f27f00d6ca0 2021-04-09T18:22:06.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.589 7f27f7b7d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00d9200 con 0x7f27f00d2460 2021-04-09T18:22:06.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.589 7f27f5198700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27f00d6ca0 0x7f27f00d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.589 7f27f5999700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f27f00cffc0 0x7f27f00d7820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58582/0 (socket says 172.21.15.32:58582) 2021-04-09T18:22:06.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.589 7f27f5999700 1 -- 172.21.15.32:0/1546895380 learned_addr learned my addr 172.21.15.32:0/1546895380 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:06.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f619a700 1 --2- 172.21.15.32:0/1546895380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00d2460 0x7f27f010cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f5198700 1 -- 172.21.15.32:0/1546895380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00d2460 msgr2=0x7f27f010cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:06.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f5198700 1 --2- 172.21.15.32:0/1546895380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00d2460 0x7f27f010cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f5198700 1 -- 172.21.15.32:0/1546895380 >> v1:172.21.15.32:6789/0 conn(0x7f27f00cffc0 legacy=0x7f27f00d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f5198700 1 -- 172.21.15.32:0/1546895380 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f00d2860 con 0x7f27f00d6ca0 2021-04-09T18:22:06.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f4997700 1 -- 172.21.15.32:0/1546895380 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510540261 0 0) 0x7f27ec000ce0 con 0x7f27f00cffc0 2021-04-09T18:22:06.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f5198700 1 --2- 172.21.15.32:0/1546895380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00d6ca0 0x7f27f00d8010 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f27e0006650 tx=0x7f27e0009a00).ready entity=mon.0 client_cookie=7f0691de148cd05c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:06.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f4997700 1 -- 172.21.15.32:0/1546895380 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e0009df0 con 0x7f27f00d6ca0 2021-04-09T18:22:06.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f4997700 1 -- 172.21.15.32:0/1546895380 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e00009b0 con 0x7f27f00d6ca0 2021-04-09T18:22:06.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f4997700 1 -- 172.21.15.32:0/1546895380 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e000c2b0 con 0x7f27f00d6ca0 2021-04-09T18:22:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f7b7d700 1 -- 172.21.15.32:0/1546895380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00d6ca0 msgr2=0x7f27f00d8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.590 7f27f7b7d700 1 --2- 172.21.15.32:0/1546895380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00d6ca0 0x7f27f00d8010 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f27e0006650 tx=0x7f27e0009a00).stop 2021-04-09T18:22:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.591 7f27f7b7d700 1 -- 172.21.15.32:0/1546895380 shutdown_connections 2021-04-09T18:22:06.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.591 7f27f7b7d700 1 --2- 172.21.15.32:0/1546895380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00d2460 0x7f27f010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.591 7f27f7b7d700 1 --2- 172.21.15.32:0/1546895380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00d6ca0 0x7f27f00d8010 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.591 7f27f7b7d700 1 -- 172.21.15.32:0/1546895380 >> 172.21.15.32:0/1546895380 conn(0x7f27f00ce5e0 msgr2=0x7f27f00cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:06.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.591 7f27f7b7d700 1 -- 172.21.15.32:0/1546895380 shutdown_connections 2021-04-09T18:22:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.591 7f27f7b7d700 1 -- 172.21.15.32:0/1546895380 wait complete. 2021-04-09T18:22:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.611 7f27f7b7d700 1 Processor -- start 2021-04-09T18:22:06.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.611 7f27f7b7d700 1 -- start start 2021-04-09T18:22:06.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f7b7d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00b89a0 0x7f27f00c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f7b7d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f00b8da0 0x7f27f00be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f7b7d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00b91a0 0x7f27f00bead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f7b7d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00c39b0 con 0x7f27f00b89a0 2021-04-09T18:22:06.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f7b7d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00c0d60 con 0x7f27f00b8da0 2021-04-09T18:22:06.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f7b7d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00bf0f0 con 0x7f27f00b91a0 2021-04-09T18:22:06.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f5999700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00b89a0 0x7f27f00c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f5999700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00b89a0 0x7f27f00c31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48998/0 (socket says 172.21.15.32:48998) 2021-04-09T18:22:06.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f5999700 1 -- 172.21.15.32:0/443370463 learned_addr learned my addr 172.21.15.32:0/443370463 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:06.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f5198700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f00b8da0 0x7f27f00be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f619a700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00b91a0 0x7f27f00bead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f5999700 1 -- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00b91a0 msgr2=0x7f27f00bead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f5999700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00b91a0 0x7f27f00bead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f5999700 1 -- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f00b8da0 msgr2=0x7f27f00be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:06.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f5999700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f00b8da0 0x7f27f00be4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:06.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.612 7f27f5999700 1 -- 172.21.15.32:0/443370463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f00bf2a0 con 0x7f27f00b89a0 2021-04-09T18:22:06.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.613 7f27f5999700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00b89a0 0x7f27f00c31a0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7f27ec00b530 tx=0x7f27ec00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:06.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.613 7f27e67fc700 1 -- 172.21.15.32:0/443370463 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27ec006020 con 0x7f27f00b89a0 2021-04-09T18:22:06.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.613 7f27e67fc700 1 -- 172.21.15.32:0/443370463 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec0063a0 con 0x7f27f00b89a0 2021-04-09T18:22:06.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.613 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f00bf0f0 con 0x7f27f00b89a0 2021-04-09T18:22:06.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.613 7f27e67fc700 1 -- 172.21.15.32:0/443370463 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27ec004590 con 0x7f27f00b89a0 2021-04-09T18:22:06.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.613 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f00bf0f0 con 0x7f27f00b89a0 2021-04-09T18:22:06.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.613 7f27e67fc700 1 -- 172.21.15.32:0/443370463 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f27ec00a000 con 0x7f27f00b89a0 2021-04-09T18:22:06.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.614 7f27e67fc700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d8018930 0x7f27d801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:06.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.614 7f27e67fc700 1 -- 172.21.15.32:0/443370463 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f27ec032b80 con 0x7f27f00b89a0 2021-04-09T18:22:06.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.614 7f27f5198700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d8018930 0x7f27d801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:06.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.614 7f27f5198700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d8018930 0x7f27d801ae10 crc :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:06.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.633 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f01b70c0 con 0x7f27f00b89a0 2021-04-09T18:22:06.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.637 7f27e67fc700 1 -- 172.21.15.32:0/443370463 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f27ec006b90 con 0x7f27f00b89a0 2021-04-09T18:22:06.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:06.832 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f27f01b70c0 con 0x7f27f00b89a0 2021-04-09T18:22:07.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.450 7f27e67fc700 1 -- 172.21.15.32:0/443370463 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v114) v1 ==== 105+0+0 (secure 0 0 0) 0x7f27ec00a000 con 0x7f27f00b89a0 2021-04-09T18:22:07.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d8018930 msgr2=0x7f27d801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:07.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d8018930 0x7f27d801ae10 crc :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00b89a0 msgr2=0x7f27f00c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:07.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00b89a0 0x7f27f00c31a0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7f27ec00b530 tx=0x7f27ec00b8d0).stop 2021-04-09T18:22:07.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 shutdown_connections 2021-04-09T18:22:07.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d8018930 0x7f27d801ae10 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00b91a0 0x7f27f00bead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f00b8da0 0x7f27f00be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 --2- 172.21.15.32:0/443370463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00b89a0 0x7f27f00c31a0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 >> 172.21.15.32:0/443370463 conn(0x7f27f00a06a0 msgr2=0x7f27f008fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 shutdown_connections 2021-04-09T18:22:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.500 7f27f7b7d700 1 -- 172.21.15.32:0/443370463 wait complete. 2021-04-09T18:22:07.563 INFO:tasks.cephfs.test_failover:Waiting for cluster to shink. 2021-04-09T18:22:07.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:07.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.661 7fe0055b0700 1 Processor -- start 2021-04-09T18:22:07.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.661 7fe0055b0700 1 -- start start 2021-04-09T18:22:07.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.661 7fe0055b0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe00010da00 0x7fe00010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:07.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.661 7fe0055b0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00010d480 con 0x7fe00010d010 2021-04-09T18:22:07.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.661 7fe0055b0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0000d1f70 con 0x7fe0000d7ca0 2021-04-09T18:22:07.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.661 7fe0055b0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0000d9220 con 0x7fe00010da00 2021-04-09T18:22:07.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdfff2bf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe00010d010 0x7fe0000d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58596/0 (socket says 172.21.15.32:58596) 2021-04-09T18:22:07.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdfff2bf700 1 -- 172.21.15.32:0/4079796009 learned_addr learned my addr 172.21.15.32:0/4079796009 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:07.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdfffac0700 1 --2- 172.21.15.32:0/4079796009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe00010da00 0x7fe00010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:07.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdffe2bd700 1 -- 172.21.15.32:0/4079796009 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541071342 0 0) 0x7fdff0001060 con 0x7fe00010d010 2021-04-09T18:22:07.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdffe2bd700 1 -- 172.21.15.32:0/4079796009 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe8002fe0 con 0x7fe00010d010 2021-04-09T18:22:07.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdffe2bd700 1 -- 172.21.15.32:0/4079796009 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288930465 0 0) 0x7fdff4000c60 con 0x7fe0000d7ca0 2021-04-09T18:22:07.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdffe2bd700 1 -- 172.21.15.32:0/4079796009 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe8003210 con 0x7fe0000d7ca0 2021-04-09T18:22:07.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdfffac0700 1 -- 172.21.15.32:0/4079796009 >> v1:172.21.15.86:6789/0 conn(0x7fe0000d7ca0 legacy=0x7fe0000d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:07.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdfffac0700 1 -- 172.21.15.32:0/4079796009 >> v1:172.21.15.32:6789/0 conn(0x7fe00010d010 legacy=0x7fe0000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:07.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdffe2bd700 1 -- 172.21.15.32:0/4079796009 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 844039726 0 0) 0x7fdff0001460 con 0x7fe00010d010 2021-04-09T18:22:07.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdfffac0700 1 -- 172.21.15.32:0/4079796009 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00010e420 con 0x7fe00010da00 2021-04-09T18:22:07.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdfffac0700 1 --2- 172.21.15.32:0/4079796009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe00010da00 0x7fe00010de00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fdff8006ad0 tx=0x7fdff8009a50).ready entity=mon.2 client_cookie=3f99f4202a6df76f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:07.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdffe2bd700 1 -- 172.21.15.32:0/4079796009 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdff80066a0 con 0x7fe00010da00 2021-04-09T18:22:07.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdffe2bd700 1 -- 172.21.15.32:0/4079796009 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff8006410 con 0x7fe00010da00 2021-04-09T18:22:07.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.662 7fdffe2bd700 1 -- 172.21.15.32:0/4079796009 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdff80024d0 con 0x7fe00010da00 2021-04-09T18:22:07.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.663 7fe0055b0700 1 -- 172.21.15.32:0/4079796009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe00010da00 msgr2=0x7fe00010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:07.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.663 7fe0055b0700 1 --2- 172.21.15.32:0/4079796009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe00010da00 0x7fe00010de00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fdff8006ad0 tx=0x7fdff8009a50).stop 2021-04-09T18:22:07.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.663 7fe0055b0700 1 -- 172.21.15.32:0/4079796009 shutdown_connections 2021-04-09T18:22:07.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.663 7fe0055b0700 1 --2- 172.21.15.32:0/4079796009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe00010da00 0x7fe00010de00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.663 7fe0055b0700 1 -- 172.21.15.32:0/4079796009 >> 172.21.15.32:0/4079796009 conn(0x7fe0000ce5f0 msgr2=0x7fe0000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:07.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.663 7fe0055b0700 1 -- 172.21.15.32:0/4079796009 shutdown_connections 2021-04-09T18:22:07.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.663 7fe0055b0700 1 -- 172.21.15.32:0/4079796009 wait complete. 2021-04-09T18:22:07.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.683 7fe0055b0700 1 Processor -- start 2021-04-09T18:22:07.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.683 7fe0055b0700 1 -- start start 2021-04-09T18:22:07.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fe0055b0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0000b42d0 0x7fe000183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:07.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fe0055b0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0001ac090 0x7fe0001ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:07.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fe0055b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe000184d30 0x7fe000185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:07.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fe0055b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0000a1a90 con 0x7fe000184d30 2021-04-09T18:22:07.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fe0055b0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0001acab0 con 0x7fe0001ac090 2021-04-09T18:22:07.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fe0055b0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe000185a00 con 0x7fe0000b42d0 2021-04-09T18:22:07.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fdfffac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe000184d30 0x7fe000185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:07.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fdfffac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe000184d30 0x7fe000185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49016/0 (socket says 172.21.15.32:49016) 2021-04-09T18:22:07.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fdfffac0700 1 -- 172.21.15.32:0/3347491364 learned_addr learned my addr 172.21.15.32:0/3347491364 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:07.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fdfff2bf700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0000b42d0 0x7fe000183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:07.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.684 7fdffeabe700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0001ac090 0x7fe0001ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:07.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfffac0700 1 -- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0000b42d0 msgr2=0x7fe000183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:07.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfffac0700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0000b42d0 0x7fe000183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfffac0700 1 -- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0001ac090 msgr2=0x7fe0001ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:07.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfffac0700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0001ac090 0x7fe0001ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfffac0700 1 -- 172.21.15.32:0/3347491364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe000185750 con 0x7fe000184d30 2021-04-09T18:22:07.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfffac0700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe000184d30 0x7fe000185130 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7fdff8009fc0 tx=0x7fdff800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:07.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfe7fff700 1 -- 172.21.15.32:0/3347491364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdff8006cd0 con 0x7fe000184d30 2021-04-09T18:22:07.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfe7fff700 1 -- 172.21.15.32:0/3347491364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff8000a70 con 0x7fe000184d30 2021-04-09T18:22:07.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe000185750 con 0x7fe000184d30 2021-04-09T18:22:07.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfe7fff700 1 -- 172.21.15.32:0/3347491364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdff8002080 con 0x7fe000184d30 2021-04-09T18:22:07.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe000185750 con 0x7fe000184d30 2021-04-09T18:22:07.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.685 7fdfe7fff700 1 -- 172.21.15.32:0/3347491364 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdff8006cd0 con 0x7fe000184d30 2021-04-09T18:22:07.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.686 7fdfe7fff700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe0018930 0x7fdfe001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:07.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.686 7fdfe7fff700 1 -- 172.21.15.32:0/3347491364 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdff8032a10 con 0x7fe000184d30 2021-04-09T18:22:07.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.686 7fdfff2bf700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe0018930 0x7fdfe001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:07.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.686 7fdfff2bf700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe0018930 0x7fdfe001ae10 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:07.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.705 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe000182d10 con 0x7fe000184d30 2021-04-09T18:22:07.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.709 7fdfe7fff700 1 -- 172.21.15.32:0/3347491364 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdff8006560 con 0x7fe000184d30 2021-04-09T18:22:07.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.900 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe00010d480 con 0x7fe000184d30 2021-04-09T18:22:07.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.901 7fdfe7fff700 1 -- 172.21.15.32:0/3347491364 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+2811 (secure 0 0 0) 0x7fdff8000a70 con 0x7fe000184d30 2021-04-09T18:22:07.953 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 115 2021-04-09T18:22:07.954 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:07.954 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":115,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:07.452036","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":7387,"mds_1":7390,"mds_2":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7575":{"gid":7575,"name":"a","rank":2,"incarnation":112,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:07.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.903 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe0018930 msgr2=0x7fdfe001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:07.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.903 7fe0055b0700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe0018930 0x7fdfe001ae10 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.903 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe000184d30 msgr2=0x7fe000185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:07.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.903 7fe0055b0700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe000184d30 0x7fe000185130 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7fdff8009fc0 tx=0x7fdff800b940).stop 2021-04-09T18:22:07.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.904 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 shutdown_connections 2021-04-09T18:22:07.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.904 7fe0055b0700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe0018930 0x7fdfe001ae10 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.904 7fe0055b0700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe000184d30 0x7fe000185130 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.904 7fe0055b0700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0001ac090 0x7fe0001ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.904 7fe0055b0700 1 --2- 172.21.15.32:0/3347491364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0000b42d0 0x7fe000183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:07.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.904 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 >> 172.21.15.32:0/3347491364 conn(0x7fe00019b890 msgr2=0x7fe0001a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:07.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.904 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 shutdown_connections 2021-04-09T18:22:07.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:07.904 7fe0055b0700 1 -- 172.21.15.32:0/3347491364 wait complete. 2021-04-09T18:22:08.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:07.452036', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1, 2], 'up': {'mds_0': 7387, 'mds_1': 7390, 'mds_2': 7575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7575': {'gid': 7575, 'name': 'a', 'rank': 2, 'incarnation': 112, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6834/4090970868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 4090970868}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 4090970868}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:08.369 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7575:up:stopping 2021-04-09T18:22:09.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:09.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.480 7f34678d9700 1 Processor -- start 2021-04-09T18:22:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.480 7f34678d9700 1 -- start start 2021-04-09T18:22:09.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f34678d9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f346010d010 0x7f34600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:09.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f34678d9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346010d480 con 0x7f34600d8490 2021-04-09T18:22:09.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f34678d9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346010e1f0 con 0x7f34600d1da0 2021-04-09T18:22:09.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f34678d9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34600d8d30 con 0x7f346010d010 2021-04-09T18:22:09.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f3465ef6700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f34600d8490 0x7f346010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58612/0 (socket says 172.21.15.32:58612) 2021-04-09T18:22:09.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f3465ef6700 1 -- 172.21.15.32:0/350552864 learned_addr learned my addr 172.21.15.32:0/350552864 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:09.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f34656f5700 1 --2- 172.21.15.32:0/350552864 >> v2:172.21.15.86:3300/0 conn(0x7f346010d010 0x7f34600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:09.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f3453fff700 1 -- 172.21.15.32:0/350552864 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185970097 0 0) 0x7f3458000d30 con 0x7f34600d8490 2021-04-09T18:22:09.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f3453fff700 1 -- 172.21.15.32:0/350552864 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f344c002fe0 con 0x7f34600d8490 2021-04-09T18:22:09.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f3453fff700 1 -- 172.21.15.32:0/350552864 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580453447 0 0) 0x7f3454000c60 con 0x7f34600d1da0 2021-04-09T18:22:09.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f3453fff700 1 -- 172.21.15.32:0/350552864 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f344c003210 con 0x7f34600d1da0 2021-04-09T18:22:09.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f34656f5700 1 -- 172.21.15.32:0/350552864 >> v1:172.21.15.86:6789/0 conn(0x7f34600d1da0 legacy=0x7f34600d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:09.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f34656f5700 1 -- 172.21.15.32:0/350552864 >> v1:172.21.15.32:6789/0 conn(0x7f34600d8490 legacy=0x7f346010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:09.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f3453fff700 1 -- 172.21.15.32:0/350552864 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1442291735 0 0) 0x7f3458001270 con 0x7f34600d8490 2021-04-09T18:22:09.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.481 7f34656f5700 1 -- 172.21.15.32:0/350552864 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600d9220 con 0x7f346010d010 2021-04-09T18:22:09.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.482 7f34656f5700 1 --2- 172.21.15.32:0/350552864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f346010d010 0x7f34600d1780 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f345c006650 tx=0x7f345c009a00).ready entity=mon.1 client_cookie=836f3693671551da server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:09.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.482 7f3453fff700 1 -- 172.21.15.32:0/350552864 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f345c009df0 con 0x7f346010d010 2021-04-09T18:22:09.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.482 7f3453fff700 1 -- 172.21.15.32:0/350552864 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c0009b0 con 0x7f346010d010 2021-04-09T18:22:09.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.482 7f3453fff700 1 -- 172.21.15.32:0/350552864 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f345c00c2b0 con 0x7f346010d010 2021-04-09T18:22:09.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.482 7f34678d9700 1 -- 172.21.15.32:0/350552864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f346010d010 msgr2=0x7f34600d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:09.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.482 7f34678d9700 1 --2- 172.21.15.32:0/350552864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f346010d010 0x7f34600d1780 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f345c006650 tx=0x7f345c009a00).stop 2021-04-09T18:22:09.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.483 7f34678d9700 1 -- 172.21.15.32:0/350552864 shutdown_connections 2021-04-09T18:22:09.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.483 7f34678d9700 1 --2- 172.21.15.32:0/350552864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f346010d010 0x7f34600d1780 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:09.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.483 7f34678d9700 1 -- 172.21.15.32:0/350552864 >> 172.21.15.32:0/350552864 conn(0x7f34600ce5f0 msgr2=0x7f34600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:09.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.483 7f34678d9700 1 -- 172.21.15.32:0/350552864 shutdown_connections 2021-04-09T18:22:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.483 7f34678d9700 1 -- 172.21.15.32:0/350552864 wait complete. 2021-04-09T18:22:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.503 7f34678d9700 1 Processor -- start 2021-04-09T18:22:09.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.503 7f34678d9700 1 -- start start 2021-04-09T18:22:09.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34678d9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34601a91e0 0x7f34600a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:09.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34678d9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34601a95e0 0x7f34600a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:09.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34678d9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34601ac850 0x7f34601acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:09.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34678d9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34601ad270 con 0x7f34601a91e0 2021-04-09T18:22:09.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34678d9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34600b7640 con 0x7f34601ac850 2021-04-09T18:22:09.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34678d9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3460183c80 con 0x7f34601a95e0 2021-04-09T18:22:09.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34656f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34601a91e0 0x7f34600a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:09.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34656f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34601a91e0 0x7f34600a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49026/0 (socket says 172.21.15.32:49026) 2021-04-09T18:22:09.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34656f5700 1 -- 172.21.15.32:0/1272451696 learned_addr learned my addr 172.21.15.32:0/1272451696 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:09.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f3464ef4700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34601a95e0 0x7f34600a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:09.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f3465ef6700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34601ac850 0x7f34601acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:09.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34656f5700 1 -- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34601a95e0 msgr2=0x7f34600a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:09.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34656f5700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34601a95e0 0x7f34600a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:09.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34656f5700 1 -- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34601ac850 msgr2=0x7f34601acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:09.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34656f5700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34601ac850 0x7f34601acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:09.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34656f5700 1 -- 172.21.15.32:0/1272451696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3460183ea0 con 0x7f34601a91e0 2021-04-09T18:22:09.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f34656f5700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34601a91e0 0x7f34600a4a60 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7f345c0096a0 tx=0x7f345c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:09.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.504 7f3451ffb700 1 -- 172.21.15.32:0/1272451696 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f345c000c90 con 0x7f34601a91e0 2021-04-09T18:22:09.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.505 7f3451ffb700 1 -- 172.21.15.32:0/1272451696 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c002030 con 0x7f34601a91e0 2021-04-09T18:22:09.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.505 7f34678d9700 1 -- 172.21.15.32:0/1272451696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3460183ea0 con 0x7f34601a91e0 2021-04-09T18:22:09.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.505 7f3451ffb700 1 -- 172.21.15.32:0/1272451696 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f345c002550 con 0x7f34601a91e0 2021-04-09T18:22:09.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.505 7f34678d9700 1 -- 172.21.15.32:0/1272451696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3460183ea0 con 0x7f34601a91e0 2021-04-09T18:22:09.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.506 7f3451ffb700 1 -- 172.21.15.32:0/1272451696 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f345c00a000 con 0x7f34601a91e0 2021-04-09T18:22:09.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.506 7f3451ffb700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34400188c0 0x7f344001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:09.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.506 7f3451ffb700 1 -- 172.21.15.32:0/1272451696 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f345c0329b0 con 0x7f34601a91e0 2021-04-09T18:22:09.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.506 7f3464ef4700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34400188c0 0x7f344001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:09.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.506 7f3464ef4700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34400188c0 0x7f344001adc0 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:09.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.525 7f34678d9700 1 -- 172.21.15.32:0/1272451696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3460182d70 con 0x7f34601a91e0 2021-04-09T18:22:09.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.528 7f3451ffb700 1 -- 172.21.15.32:0/1272451696 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f345c0009a0 con 0x7f34601a91e0 2021-04-09T18:22:09.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.721 7f34678d9700 1 -- 172.21.15.32:0/1272451696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f346010d480 con 0x7f34601a91e0 2021-04-09T18:22:09.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.721 7f3451ffb700 1 -- 172.21.15.32:0/1272451696 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+2811 (secure 0 0 0) 0x7f345c00bd80 con 0x7f34601a91e0 2021-04-09T18:22:09.774 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 115 2021-04-09T18:22:09.775 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:09.775 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":115,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:07.452036","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":7387,"mds_1":7390,"mds_2":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7575":{"gid":7575,"name":"a","rank":2,"incarnation":112,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:09.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 -- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34400188c0 msgr2=0x7f344001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:09.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34400188c0 0x7f344001adc0 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:09.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 -- 172.21.15.32:0/1272451696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34601a91e0 msgr2=0x7f34600a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:09.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34601a91e0 0x7f34600a4a60 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7f345c0096a0 tx=0x7f345c00b990).stop 2021-04-09T18:22:09.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 -- 172.21.15.32:0/1272451696 shutdown_connections 2021-04-09T18:22:09.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34400188c0 0x7f344001adc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:09.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34601ac850 0x7f34601acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:09.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34601a95e0 0x7f34600a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:09.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 --2- 172.21.15.32:0/1272451696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34601a91e0 0x7f34600a4a60 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:09.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 -- 172.21.15.32:0/1272451696 >> 172.21.15.32:0/1272451696 conn(0x7f346019b460 msgr2=0x7f346019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:09.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 -- 172.21.15.32:0/1272451696 shutdown_connections 2021-04-09T18:22:09.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:09.724 7f34678d9700 1 -- 172.21.15.32:0/1272451696 wait complete. 2021-04-09T18:22:10.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:07.452036', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1, 2], 'up': {'mds_0': 7387, 'mds_1': 7390, 'mds_2': 7575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7575': {'gid': 7575, 'name': 'a', 'rank': 2, 'incarnation': 112, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6834/4090970868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 4090970868}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 4090970868}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:10.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7575:up:stopping 2021-04-09T18:22:11.191 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:11.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.295 7ff074137700 1 Processor -- start 2021-04-09T18:22:11.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.296 7ff074137700 1 -- start start 2021-04-09T18:22:11.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.296 7ff074137700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff06c0d6c80 0x7ff06c0d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:11.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.296 7ff074137700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff06c0d2440 0x7ff06c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:11.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.296 7ff074137700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06c0d7280 con 0x7ff06c0cffa0 2021-04-09T18:22:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.296 7ff074137700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06c0d7080 con 0x7ff06c0d2440 2021-04-09T18:22:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.296 7ff074137700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06c0d91e0 con 0x7ff06c0d6c80 2021-04-09T18:22:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.296 7ff072754700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff06c0d2440 0x7ff06c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.296 7ff071f53700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff06c0cffa0 0x7ff06c0d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58624/0 (socket says 172.21.15.32:58624) 2021-04-09T18:22:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.296 7ff071f53700 1 -- 172.21.15.32:0/4137324663 learned_addr learned my addr 172.21.15.32:0/4137324663 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff071752700 1 --2- 172.21.15.32:0/4137324663 >> v2:172.21.15.86:3300/0 conn(0x7ff06c0d6c80 0x7ff06c0d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff072754700 1 -- 172.21.15.32:0/4137324663 >> v2:172.21.15.86:3300/0 conn(0x7ff06c0d6c80 msgr2=0x7ff06c0d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff072754700 1 --2- 172.21.15.32:0/4137324663 >> v2:172.21.15.86:3300/0 conn(0x7ff06c0d6c80 0x7ff06c0d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff072754700 1 -- 172.21.15.32:0/4137324663 >> v1:172.21.15.32:6789/0 conn(0x7ff06c0cffa0 legacy=0x7ff06c0d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff070f51700 1 -- 172.21.15.32:0/4137324663 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505021887 0 0) 0x7ff068000ce0 con 0x7ff06c0cffa0 2021-04-09T18:22:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff072754700 1 -- 172.21.15.32:0/4137324663 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c0d2840 con 0x7ff06c0d2440 2021-04-09T18:22:11.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff071752700 1 --2- 172.21.15.32:0/4137324663 >> v2:172.21.15.86:3300/0 conn(0x7ff06c0d6c80 0x7ff06c0d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:11.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff072754700 1 --2- 172.21.15.32:0/4137324663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c0d2440 0x7ff06c10cfd0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7ff064006e30 tx=0x7ff064009e60).ready entity=mon.0 client_cookie=3112958131e0efec server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:11.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff070f51700 1 -- 172.21.15.32:0/4137324663 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff064006a50 con 0x7ff06c0d2440 2021-04-09T18:22:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff070f51700 1 -- 172.21.15.32:0/4137324663 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06400b300 con 0x7ff06c0d2440 2021-04-09T18:22:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff070f51700 1 -- 172.21.15.32:0/4137324663 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff06400b750 con 0x7ff06c0d2440 2021-04-09T18:22:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff074137700 1 -- 172.21.15.32:0/4137324663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c0d2440 msgr2=0x7ff06c10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.297 7ff074137700 1 --2- 172.21.15.32:0/4137324663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c0d2440 0x7ff06c10cfd0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7ff064006e30 tx=0x7ff064009e60).stop 2021-04-09T18:22:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.298 7ff074137700 1 -- 172.21.15.32:0/4137324663 shutdown_connections 2021-04-09T18:22:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.298 7ff074137700 1 --2- 172.21.15.32:0/4137324663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c0d2440 0x7ff06c10cfd0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.298 7ff074137700 1 --2- 172.21.15.32:0/4137324663 >> v2:172.21.15.86:3300/0 conn(0x7ff06c0d6c80 0x7ff06c0d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.298 7ff074137700 1 -- 172.21.15.32:0/4137324663 >> 172.21.15.32:0/4137324663 conn(0x7ff06c0ce5c0 msgr2=0x7ff06c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:11.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.298 7ff074137700 1 -- 172.21.15.32:0/4137324663 shutdown_connections 2021-04-09T18:22:11.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.298 7ff074137700 1 -- 172.21.15.32:0/4137324663 wait complete. 2021-04-09T18:22:11.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.318 7ff074137700 1 Processor -- start 2021-04-09T18:22:11.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.318 7ff074137700 1 -- start start 2021-04-09T18:22:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff074137700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c1acff0 0x7ff06c1847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff074137700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff06c183fc0 0x7ff06c1a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff074137700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff06c186ff0 0x7ff06c1873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:11.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff074137700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c188c30 con 0x7ff06c1acff0 2021-04-09T18:22:11.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff074137700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c1843c0 con 0x7ff06c186ff0 2021-04-09T18:22:11.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff074137700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06c1a5c20 con 0x7ff06c183fc0 2021-04-09T18:22:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071f53700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c1acff0 0x7ff06c1847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071f53700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c1acff0 0x7ff06c1847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49040/0 (socket says 172.21.15.32:49040) 2021-04-09T18:22:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071f53700 1 -- 172.21.15.32:0/1503053287 learned_addr learned my addr 172.21.15.32:0/1503053287 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071752700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff06c183fc0 0x7ff06c1a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff072754700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff06c186ff0 0x7ff06c1873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:11.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071f53700 1 -- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff06c183fc0 msgr2=0x7ff06c1a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:11.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071f53700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff06c183fc0 0x7ff06c1a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071f53700 1 -- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff06c186ff0 msgr2=0x7ff06c1873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:11.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071f53700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff06c186ff0 0x7ff06c1873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071f53700 1 -- 172.21.15.32:0/1503053287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c187a10 con 0x7ff06c1acff0 2021-04-09T18:22:11.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff071f53700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c1acff0 0x7ff06c1847a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7ff068006f00 tx=0x7ff06800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:11.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.319 7ff0627fc700 1 -- 172.21.15.32:0/1503053287 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff068006020 con 0x7ff06c1acff0 2021-04-09T18:22:11.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.320 7ff0627fc700 1 -- 172.21.15.32:0/1503053287 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0680063a0 con 0x7ff06c1acff0 2021-04-09T18:22:11.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.320 7ff074137700 1 -- 172.21.15.32:0/1503053287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06c187a10 con 0x7ff06c1acff0 2021-04-09T18:22:11.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.320 7ff0627fc700 1 -- 172.21.15.32:0/1503053287 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff068004620 con 0x7ff06c1acff0 2021-04-09T18:22:11.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.320 7ff074137700 1 -- 172.21.15.32:0/1503053287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06c187a10 con 0x7ff06c1acff0 2021-04-09T18:22:11.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.320 7ff0627fc700 1 -- 172.21.15.32:0/1503053287 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff06800a000 con 0x7ff06c1acff0 2021-04-09T18:22:11.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.320 7ff0627fc700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff054018930 0x7ff05401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:11.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.320 7ff0627fc700 1 -- 172.21.15.32:0/1503053287 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff0680329e0 con 0x7ff06c1acff0 2021-04-09T18:22:11.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.320 7ff071752700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff054018930 0x7ff05401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:11.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.321 7ff071752700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff054018930 0x7ff05401ae10 crc :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:11.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.340 7ff074137700 1 -- 172.21.15.32:0/1503053287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06c182dc0 con 0x7ff06c1acff0 2021-04-09T18:22:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.344 7ff0627fc700 1 -- 172.21.15.32:0/1503053287 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff068006b90 con 0x7ff06c1acff0 2021-04-09T18:22:11.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.534 7ff074137700 1 -- 172.21.15.32:0/1503053287 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff06c0d7280 con 0x7ff06c1acff0 2021-04-09T18:22:11.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.534 7ff0627fc700 1 -- 172.21.15.32:0/1503053287 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+2811 (secure 0 0 0) 0x7ff06800a000 con 0x7ff06c1acff0 2021-04-09T18:22:11.587 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 115 2021-04-09T18:22:11.588 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:11.588 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":115,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:07.452036","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":7387,"mds_1":7390,"mds_2":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7575":{"gid":7575,"name":"a","rank":2,"incarnation":112,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:11.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 -- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff054018930 msgr2=0x7ff05401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:11.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff054018930 0x7ff05401ae10 crc :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 -- 172.21.15.32:0/1503053287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c1acff0 msgr2=0x7ff06c1847a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:11.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c1acff0 0x7ff06c1847a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7ff068006f00 tx=0x7ff06800b940).stop 2021-04-09T18:22:11.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 -- 172.21.15.32:0/1503053287 shutdown_connections 2021-04-09T18:22:11.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff054018930 0x7ff05401ae10 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff06c186ff0 0x7ff06c1873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff06c183fc0 0x7ff06c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 --2- 172.21.15.32:0/1503053287 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff06c1acff0 0x7ff06c1847a0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:11.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 -- 172.21.15.32:0/1503053287 >> 172.21.15.32:0/1503053287 conn(0x7ff06c09abb0 msgr2=0x7ff06c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 -- 172.21.15.32:0/1503053287 shutdown_connections 2021-04-09T18:22:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:11.537 7ff074137700 1 -- 172.21.15.32:0/1503053287 wait complete. 2021-04-09T18:22:12.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:07.452036', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1, 2], 'up': {'mds_0': 7387, 'mds_1': 7390, 'mds_2': 7575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7575': {'gid': 7575, 'name': 'a', 'rank': 2, 'incarnation': 112, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6834/4090970868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 4090970868}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 4090970868}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:12.003 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7575:up:stopping 2021-04-09T18:22:13.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:13.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.100 7fe579221700 1 Processor -- start 2021-04-09T18:22:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.100 7fe579221700 1 -- start start 2021-04-09T18:22:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe579221700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe57410d010 0x7fe5740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:13.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe579221700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57410d480 con 0x7fe5740d1da0 2021-04-09T18:22:13.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe579221700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57410e1f0 con 0x7fe5740d8490 2021-04-09T18:22:13.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe579221700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5740d8d30 con 0x7fe57410d010 2021-04-09T18:22:13.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe57259c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe5740d1da0 0x7fe5740d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58642/0 (socket says 172.21.15.32:58642) 2021-04-09T18:22:13.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe57259c700 1 -- 172.21.15.32:0/1831711098 learned_addr learned my addr 172.21.15.32:0/1831711098 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:13.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe572d9d700 1 --2- 172.21.15.32:0/1831711098 >> v2:172.21.15.86:3300/0 conn(0x7fe57410d010 0x7fe5740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:13.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144207631 0 0) 0x7fe5680009b0 con 0x7fe5740d1da0 2021-04-09T18:22:13.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c002fe0 con 0x7fe5740d1da0 2021-04-09T18:22:13.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223612495 0 0) 0x7fe564001080 con 0x7fe5740d8490 2021-04-09T18:22:13.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c003210 con 0x7fe5740d8490 2021-04-09T18:22:13.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3992972375 0 0) 0x7fe568001010 con 0x7fe5740d1da0 2021-04-09T18:22:13.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 >> v2:172.21.15.86:3300/0 conn(0x7fe57410d010 msgr2=0x7fe5740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:13.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 --2- 172.21.15.32:0/1831711098 >> v2:172.21.15.86:3300/0 conn(0x7fe57410d010 0x7fe5740d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 >> v1:172.21.15.86:6789/0 conn(0x7fe5740d8490 legacy=0x7fe57410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5740d9220 con 0x7fe5740d1da0 2021-04-09T18:22:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fe568001530 con 0x7fe5740d1da0 2021-04-09T18:22:13.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 >> v1:172.21.15.32:6789/0 conn(0x7fe5740d1da0 legacy=0x7fe5740d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.101 7fe571d9b700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe55c003ba0 0x7fe55c005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe571d9b700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe55c0067b0 0x7fe55c008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe571d9b700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe55c0095b0 0x7fe55c00bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c050330 con 0x7fe55c0095b0 2021-04-09T18:22:13.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c050510 con 0x7fe55c0067b0 2021-04-09T18:22:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c050730 con 0x7fe55c003ba0 2021-04-09T18:22:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe572d9d700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe55c003ba0 0x7fe55c005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe57359e700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe55c0095b0 0x7fe55c00bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:13.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe57259c700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe55c0067b0 0x7fe55c008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:13.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe572d9d700 1 -- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe55c0067b0 msgr2=0x7fe55c008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:13.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe572d9d700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe55c0067b0 0x7fe55c008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe572d9d700 1 -- 172.21.15.32:0/1831711098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe55c0095b0 msgr2=0x7fe55c00bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:13.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe572d9d700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe55c0095b0 0x7fe55c00bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe572d9d700 1 -- 172.21.15.32:0/1831711098 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c0509d0 con 0x7fe55c003ba0 2021-04-09T18:22:13.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe57359e700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe55c0095b0 0x7fe55c00bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:13.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe57359e700 1 -- 172.21.15.32:0/1831711098 reap_dead start 2021-04-09T18:22:13.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe572d9d700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe55c003ba0 0x7fe55c005fb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe56c006fc0 tx=0x7fe56c00c0c0).ready entity=mon.2 client_cookie=19000d86c0f05952 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:13.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe56c00c4b0 con 0x7fe55c003ba0 2021-04-09T18:22:13.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c009a70 con 0x7fe55c003ba0 2021-04-09T18:22:13.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe571d9b700 1 -- 172.21.15.32:0/1831711098 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe56c006ac0 con 0x7fe55c003ba0 2021-04-09T18:22:13.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe579221700 1 -- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe55c003ba0 msgr2=0x7fe55c005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:13.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.102 7fe579221700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe55c003ba0 0x7fe55c005fb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe56c006fc0 tx=0x7fe56c00c0c0).stop 2021-04-09T18:22:13.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.103 7fe579221700 1 -- 172.21.15.32:0/1831711098 shutdown_connections 2021-04-09T18:22:13.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.103 7fe579221700 1 --2- 172.21.15.32:0/1831711098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe55c003ba0 0x7fe55c005fb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.103 7fe579221700 1 -- 172.21.15.32:0/1831711098 >> 172.21.15.32:0/1831711098 conn(0x7fe5740ce5f0 msgr2=0x7fe5740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.103 7fe579221700 1 -- 172.21.15.32:0/1831711098 shutdown_connections 2021-04-09T18:22:13.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.103 7fe579221700 1 -- 172.21.15.32:0/1831711098 wait complete. 2021-04-09T18:22:13.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.123 7fe579221700 1 Processor -- start 2021-04-09T18:22:13.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.123 7fe579221700 1 -- start start 2021-04-09T18:22:13.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe579221700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5740b5360 0x7fe5741a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:13.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe579221700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe57418e140 0x7fe5741aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:13.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe579221700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5741ad040 0x7fe57408fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:13.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe579221700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5741a5360 con 0x7fe5741ad040 2021-04-09T18:22:13.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe579221700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5740902c0 con 0x7fe57418e140 2021-04-09T18:22:13.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe579221700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5740904e0 con 0x7fe5740b5360 2021-04-09T18:22:13.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe57359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5741ad040 0x7fe57408fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:13.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe57359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5741ad040 0x7fe57408fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49064/0 (socket says 172.21.15.32:49064) 2021-04-09T18:22:13.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe57359e700 1 -- 172.21.15.32:0/2230007695 learned_addr learned my addr 172.21.15.32:0/2230007695 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:13.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe572d9d700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5740b5360 0x7fe5741a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe57259c700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe57418e140 0x7fe5741aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe57359e700 1 -- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5740b5360 msgr2=0x7fe5741a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe57359e700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5740b5360 0x7fe5741a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe57359e700 1 -- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe57418e140 msgr2=0x7fe5741aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe57359e700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe57418e140 0x7fe5741aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.124 7fe57359e700 1 -- 172.21.15.32:0/2230007695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe574183c80 con 0x7fe5741ad040 2021-04-09T18:22:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.125 7fe57359e700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5741ad040 0x7fe57408fca0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7fe564006fc0 tx=0x7fe56400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.125 7fe5637fe700 1 -- 172.21.15.32:0/2230007695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe56400bd50 con 0x7fe5741ad040 2021-04-09T18:22:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.125 7fe5637fe700 1 -- 172.21.15.32:0/2230007695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe564006240 con 0x7fe5741ad040 2021-04-09T18:22:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.125 7fe5637fe700 1 -- 172.21.15.32:0/2230007695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5640057c0 con 0x7fe5741ad040 2021-04-09T18:22:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.125 7fe579221700 1 -- 172.21.15.32:0/2230007695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe574183c80 con 0x7fe5741ad040 2021-04-09T18:22:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.125 7fe579221700 1 -- 172.21.15.32:0/2230007695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe574183c80 con 0x7fe5741ad040 2021-04-09T18:22:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.125 7fe5637fe700 1 -- 172.21.15.32:0/2230007695 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe564005540 con 0x7fe5741ad040 2021-04-09T18:22:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.125 7fe5637fe700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5580188d0 0x7fe55801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:13.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.126 7fe5637fe700 1 -- 172.21.15.32:0/2230007695 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe5640329d0 con 0x7fe5741ad040 2021-04-09T18:22:13.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.126 7fe572d9d700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5580188d0 0x7fe55801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:13.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.126 7fe572d9d700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5580188d0 0x7fe55801add0 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.145 7fe579221700 1 -- 172.21.15.32:0/2230007695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe574182d70 con 0x7fe5741ad040 2021-04-09T18:22:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.149 7fe5637fe700 1 -- 172.21.15.32:0/2230007695 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe564006030 con 0x7fe5741ad040 2021-04-09T18:22:13.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.339 7fe579221700 1 -- 172.21.15.32:0/2230007695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe574182d70 con 0x7fe5741ad040 2021-04-09T18:22:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.339 7fe5637fe700 1 -- 172.21.15.32:0/2230007695 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+2811 (secure 0 0 0) 0x7fe564006240 con 0x7fe5741ad040 2021-04-09T18:22:13.391 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 115 2021-04-09T18:22:13.392 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:13.392 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":115,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:07.452036","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":7387,"mds_1":7390,"mds_2":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7575":{"gid":7575,"name":"a","rank":2,"incarnation":112,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6834/4090970868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4090970868},{"type":"v1","addr":"172.21.15.32:6834","nonce":4090970868}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:13.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 -- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5580188d0 msgr2=0x7fe55801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:13.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5580188d0 0x7fe55801add0 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 -- 172.21.15.32:0/2230007695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5741ad040 msgr2=0x7fe57408fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:13.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5741ad040 0x7fe57408fca0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7fe564006fc0 tx=0x7fe56400b960).stop 2021-04-09T18:22:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 -- 172.21.15.32:0/2230007695 shutdown_connections 2021-04-09T18:22:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5580188d0 0x7fe55801add0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5741ad040 0x7fe57408fca0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe57418e140 0x7fe5741aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 --2- 172.21.15.32:0/2230007695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5740b5360 0x7fe5741a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 -- 172.21.15.32:0/2230007695 >> 172.21.15.32:0/2230007695 conn(0x7fe57419b460 msgr2=0x7fe57419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 -- 172.21.15.32:0/2230007695 shutdown_connections 2021-04-09T18:22:13.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:13.342 7fe579221700 1 -- 172.21.15.32:0/2230007695 wait complete. 2021-04-09T18:22:13.806 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:07.452036', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1, 2], 'up': {'mds_0': 7387, 'mds_1': 7390, 'mds_2': 7575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7575': {'gid': 7575, 'name': 'a', 'rank': 2, 'incarnation': 112, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6834/4090970868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 4090970868}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 4090970868}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:13.806 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7575:up:stopping 2021-04-09T18:22:13.896 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:22:14.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:14.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.912 7f4f23b51700 1 Processor -- start 2021-04-09T18:22:14.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.912 7f4f23b51700 1 -- start start 2021-04-09T18:22:14.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f23b51700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c10d010 0x7f4f1c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:14.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f23b51700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c10d480 con 0x7f4f1c0d8490 2021-04-09T18:22:14.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f23b51700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c10e1f0 con 0x7f4f1c0d1da0 2021-04-09T18:22:14.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f23b51700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c10e780 con 0x7f4f1c10d010 2021-04-09T18:22:14.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2216e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4f1c0d8490 0x7f4f1c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58674/0 (socket says 172.21.15.32:58674) 2021-04-09T18:22:14.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2216e700 1 -- 172.21.15.32:0/1444234705 learned_addr learned my addr 172.21.15.32:0/1444234705 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:14.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2196d700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c10d010 0x7f4f1c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:14.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798228500 0 0) 0x7f4f14000c10 con 0x7f4f1c0d8490 2021-04-09T18:22:14.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f04002fe0 con 0x7f4f1c0d8490 2021-04-09T18:22:14.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483087203 0 0) 0x7f4f10001080 con 0x7f4f1c0d1da0 2021-04-09T18:22:14.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f04003210 con 0x7f4f1c0d1da0 2021-04-09T18:22:14.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 442515097 0 0) 0x7f4f140010e0 con 0x7f4f1c0d8490 2021-04-09T18:22:14.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c10d010 msgr2=0x7f4f1c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:14.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c10d010 0x7f4f1c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:14.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 >> v1:172.21.15.86:6789/0 conn(0x7f4f1c0d1da0 legacy=0x7f4f1c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:14.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c0d8d30 con 0x7f4f1c0d8490 2021-04-09T18:22:14.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f4f14001600 con 0x7f4f1c0d8490 2021-04-09T18:22:14.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2196d700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c10d010 0x7f4f1c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:14.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 >> v1:172.21.15.32:6789/0 conn(0x7f4f1c0d8490 legacy=0x7f4f1c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:14.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f04003b90 0x7f4f04005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:14.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f04006770 0x7f4f04008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:14.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.913 7f4f2096b700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f04009500 0x7f4f0400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:14.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f040502e0 con 0x7f4f04003b90 2021-04-09T18:22:14.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2196d700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f04003b90 0x7f4f04005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:14.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04050500 con 0x7f4f04009500 2021-04-09T18:22:15.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04050720 con 0x7f4f04006770 2021-04-09T18:22:15.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2116c700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f04006770 0x7f4f04008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:15.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2216e700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f04009500 0x7f4f0400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:15.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2196d700 1 -- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f04006770 msgr2=0x7f4f04008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:15.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2196d700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f04006770 0x7f4f04008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:15.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2196d700 1 -- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f04009500 msgr2=0x7f4f0400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:15.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2196d700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f04009500 0x7f4f0400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:15.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2196d700 1 -- 172.21.15.32:0/1444234705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f040509c0 con 0x7f4f04003b90 2021-04-09T18:22:15.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2216e700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f04009500 0x7f4f0400ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:15.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2196d700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f04003b90 0x7f4f04005fa0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f4f0c006f80 tx=0x7f4f0c009e20).ready entity=mon.0 client_cookie=13f37ecede46eaf4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:15.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f0c00b510 con 0x7f4f04003b90 2021-04-09T18:22:15.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c00b850 con 0x7f4f04003b90 2021-04-09T18:22:15.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2096b700 1 -- 172.21.15.32:0/1444234705 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f0c00bd30 con 0x7f4f04003b90 2021-04-09T18:22:15.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f23b51700 1 -- 172.21.15.32:0/1444234705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f04003b90 msgr2=0x7f4f04005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:15.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f23b51700 1 --2- 172.21.15.32:0/1444234705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f04003b90 0x7f4f04005fa0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f4f0c006f80 tx=0x7f4f0c009e20).stop 2021-04-09T18:22:15.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.914 7f4f2216e700 1 -- 172.21.15.32:0/1444234705 reap_dead start 2021-04-09T18:22:15.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.915 7f4f23b51700 1 -- 172.21.15.32:0/1444234705 shutdown_connections 2021-04-09T18:22:15.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.915 7f4f23b51700 1 -- 172.21.15.32:0/1444234705 >> 172.21.15.32:0/1444234705 conn(0x7f4f1c0ce5f0 msgr2=0x7f4f1c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:15.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.915 7f4f23b51700 1 -- 172.21.15.32:0/1444234705 shutdown_connections 2021-04-09T18:22:15.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.915 7f4f23b51700 1 -- 172.21.15.32:0/1444234705 wait complete. 2021-04-09T18:22:15.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.935 7f4f23b51700 1 Processor -- start 2021-04-09T18:22:15.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.935 7f4f23b51700 1 -- start start 2021-04-09T18:22:15.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f23b51700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c1aca20 0x7f4f1c1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:15.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f23b51700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c18bd40 0x7f4f1c18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:15.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f23b51700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f1c08fba0 0x7f4f1c08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:15.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f23b51700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c18c7e0 con 0x7f4f1c1aca20 2021-04-09T18:22:15.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f23b51700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c18ca30 con 0x7f4f1c08fba0 2021-04-09T18:22:15.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f23b51700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0905f0 con 0x7f4f1c18bd40 2021-04-09T18:22:15.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2196d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c1aca20 0x7f4f1c1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:15.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2196d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c1aca20 0x7f4f1c1ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49094/0 (socket says 172.21.15.32:49094) 2021-04-09T18:22:15.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2196d700 1 -- 172.21.15.32:0/2331229056 learned_addr learned my addr 172.21.15.32:0/2331229056 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2116c700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c18bd40 0x7f4f1c18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2216e700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f1c08fba0 0x7f4f1c08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2196d700 1 -- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c18bd40 msgr2=0x7f4f1c18c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:15.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2196d700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c18bd40 0x7f4f1c18c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2196d700 1 -- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f1c08fba0 msgr2=0x7f4f1c08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2196d700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f1c08fba0 0x7f4f1c08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2196d700 1 -- 172.21.15.32:0/2331229056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c0b4300 con 0x7f4f1c1aca20 2021-04-09T18:22:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2216e700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f1c08fba0 0x7f4f1c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:22:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f2196d700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c1aca20 0x7f4f1c1ace20 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0x7f4f0c006e40 tx=0x7f4f0c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.936 7f4f1a7fc700 1 -- 172.21.15.32:0/2331229056 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f0c00c2f0 con 0x7f4f1c1aca20 2021-04-09T18:22:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.937 7f4f1a7fc700 1 -- 172.21.15.32:0/2331229056 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c00c670 con 0x7f4f1c1aca20 2021-04-09T18:22:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.937 7f4f1a7fc700 1 -- 172.21.15.32:0/2331229056 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f0c00cb50 con 0x7f4f1c1aca20 2021-04-09T18:22:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.937 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c0b4300 con 0x7f4f1c1aca20 2021-04-09T18:22:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.937 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c0b4300 con 0x7f4f1c1aca20 2021-04-09T18:22:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.937 7f4f1a7fc700 1 -- 172.21.15.32:0/2331229056 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4f0c009fb0 con 0x7f4f1c1aca20 2021-04-09T18:22:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.937 7f4f1a7fc700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4efc0188c0 0x7f4efc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.937 7f4f1a7fc700 1 -- 172.21.15.32:0/2331229056 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4f0c032ac0 con 0x7f4f1c1aca20 2021-04-09T18:22:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.938 7f4f2116c700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4efc0188c0 0x7f4efc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.938 7f4f2116c700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4efc0188c0 0x7f4efc01adc0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.957 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c182d70 con 0x7f4f1c1aca20 2021-04-09T18:22:15.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:14.960 7f4f1a7fc700 1 -- 172.21.15.32:0/2331229056 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4f0c00be30 con 0x7f4f1c1aca20 2021-04-09T18:22:15.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.151 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f1c18c760 con 0x7f4f1c1aca20 2021-04-09T18:22:15.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.152 7f4f1a7fc700 1 -- 172.21.15.32:0/2331229056 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2460 (secure 0 0 0) 0x7f4f0c00c850 con 0x7f4f1c1aca20 2021-04-09T18:22:15.205 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 117 2021-04-09T18:22:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:15.206 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":117,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:13.785899","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7387,"mds_1":7390},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:stopping","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:15.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4efc0188c0 msgr2=0x7f4efc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:15.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4efc0188c0 0x7f4efc01adc0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c1aca20 msgr2=0x7f4f1c1ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c1aca20 0x7f4f1c1ace20 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0x7f4f0c006e40 tx=0x7f4f0c00bcf0).stop 2021-04-09T18:22:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 shutdown_connections 2021-04-09T18:22:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4efc0188c0 0x7f4efc01adc0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f1c08fba0 0x7f4f1c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1c18bd40 0x7f4f1c18c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 --2- 172.21.15.32:0/2331229056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f1c1aca20 0x7f4f1c1ace20 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:15.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 >> 172.21.15.32:0/2331229056 conn(0x7f4f1c19b460 msgr2=0x7f4f1c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:15.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 shutdown_connections 2021-04-09T18:22:15.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:15.154 7f4f23b51700 1 -- 172.21.15.32:0/2331229056 wait complete. 2021-04-09T18:22:15.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:13.785899', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7387, 'mds_1': 7390}, 'failed': [], 'damaged': [], 'stopped': [2], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:15.620 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7390:up:stopping 2021-04-09T18:22:16.621 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:16.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.724 7faadf345700 1 Processor -- start 2021-04-09T18:22:16.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.724 7faadf345700 1 -- start start 2021-04-09T18:22:16.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faadf345700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faad810da00 0x7faad810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:16.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faadf345700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad810d480 con 0x7faad810d010 2021-04-09T18:22:16.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faadf345700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad80d1f70 con 0x7faad80d7ca0 2021-04-09T18:22:16.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faadf345700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad80d8d30 con 0x7faad810da00 2021-04-09T18:22:16.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faadd161700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faad810d010 0x7faad80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58690/0 (socket says 172.21.15.32:58690) 2021-04-09T18:22:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faadd161700 1 -- 172.21.15.32:0/2696731486 learned_addr learned my addr 172.21.15.32:0/2696731486 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faadd962700 1 --2- 172.21.15.32:0/2696731486 >> v2:172.21.15.86:3300/0 conn(0x7faad810da00 0x7faad810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faacffff700 1 -- 172.21.15.32:0/2696731486 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605378855 0 0) 0x7faad4001060 con 0x7faad810d010 2021-04-09T18:22:16.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faacffff700 1 -- 172.21.15.32:0/2696731486 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faac4002fe0 con 0x7faad810d010 2021-04-09T18:22:16.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faacffff700 1 -- 172.21.15.32:0/2696731486 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2540875228 0 0) 0x7faad4001250 con 0x7faad810d010 2021-04-09T18:22:16.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faacffff700 1 -- 172.21.15.32:0/2696731486 >> v2:172.21.15.86:3300/0 conn(0x7faad810da00 msgr2=0x7faad810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:16.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faacffff700 1 --2- 172.21.15.32:0/2696731486 >> v2:172.21.15.86:3300/0 conn(0x7faad810da00 0x7faad810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:16.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.725 7faacffff700 1 -- 172.21.15.32:0/2696731486 >> v1:172.21.15.86:6789/0 conn(0x7faad80d7ca0 legacy=0x7faad80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:16.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 -- 172.21.15.32:0/2696731486 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad80d9220 con 0x7faad810d010 2021-04-09T18:22:16.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 -- 172.21.15.32:0/2696731486 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7faad4001980 con 0x7faad810d010 2021-04-09T18:22:16.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 -- 172.21.15.32:0/2696731486 >> v1:172.21.15.32:6789/0 conn(0x7faad810d010 legacy=0x7faad80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:16.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faac4003a40 0x7faac4005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:16.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faac4006780 0x7faac4008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faac4009580 0x7faac400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 -- 172.21.15.32:0/2696731486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac400c2f0 con 0x7faac4003a40 2021-04-09T18:22:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 -- 172.21.15.32:0/2696731486 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac400c510 con 0x7faac4009580 2021-04-09T18:22:16.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 -- 172.21.15.32:0/2696731486 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac400c730 con 0x7faac4006780 2021-04-09T18:22:16.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadd161700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faac4003a40 0x7faac4005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:16.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadd962700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faac4009580 0x7faac400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:16.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadc960700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faac4006780 0x7faac4008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:16.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadd161700 1 -- 172.21.15.32:0/2696731486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faac4006780 msgr2=0x7faac4008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:16.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadd161700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faac4006780 0x7faac4008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:16.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadd161700 1 -- 172.21.15.32:0/2696731486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faac4009580 msgr2=0x7faac400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:16.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadd161700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faac4009580 0x7faac400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadd161700 1 -- 172.21.15.32:0/2696731486 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac40509a0 con 0x7faac4003a40 2021-04-09T18:22:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadd962700 1 -- 172.21.15.32:0/2696731486 reap_dead start 2021-04-09T18:22:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadd161700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faac4003a40 0x7faac4005f60 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7faad4001c30 tx=0x7faad400a310).ready entity=mon.0 client_cookie=104b9523632a9921 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 -- 172.21.15.32:0/2696731486 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad40017d0 con 0x7faac4003a40 2021-04-09T18:22:16.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 -- 172.21.15.32:0/2696731486 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad40070c0 con 0x7faac4003a40 2021-04-09T18:22:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faacffff700 1 -- 172.21.15.32:0/2696731486 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad4007510 con 0x7faac4003a40 2021-04-09T18:22:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadf345700 1 -- 172.21.15.32:0/2696731486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faac4003a40 msgr2=0x7faac4005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.726 7faadf345700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faac4003a40 0x7faac4005f60 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7faad4001c30 tx=0x7faad400a310).stop 2021-04-09T18:22:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.727 7faadf345700 1 -- 172.21.15.32:0/2696731486 shutdown_connections 2021-04-09T18:22:16.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.727 7faadf345700 1 --2- 172.21.15.32:0/2696731486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faac4003a40 0x7faac4005f60 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:16.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.727 7faadf345700 1 -- 172.21.15.32:0/2696731486 >> 172.21.15.32:0/2696731486 conn(0x7faad80ce5f0 msgr2=0x7faad80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:16.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.727 7faadf345700 1 -- 172.21.15.32:0/2696731486 shutdown_connections 2021-04-09T18:22:16.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.727 7faadf345700 1 -- 172.21.15.32:0/2696731486 wait complete. 2021-04-09T18:22:16.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.747 7faadf345700 1 Processor -- start 2021-04-09T18:22:16.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.747 7faadf345700 1 -- start start 2021-04-09T18:22:16.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadf345700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad81ac380 0x7faad818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:16.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadf345700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad81ac780 0x7faad818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:16.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadf345700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad8183520 0x7faad8183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:16.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadf345700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad81ace10 con 0x7faad8183520 2021-04-09T18:22:16.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadf345700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad8189cf0 con 0x7faad81ac780 2021-04-09T18:22:16.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadf345700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad8183f40 con 0x7faad81ac380 2021-04-09T18:22:16.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd962700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad8183520 0x7faad8183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:16.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd962700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad8183520 0x7faad8183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49116/0 (socket says 172.21.15.32:49116) 2021-04-09T18:22:16.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd962700 1 -- 172.21.15.32:0/1819187812 learned_addr learned my addr 172.21.15.32:0/1819187812 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:16.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd161700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad81ac380 0x7faad818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:16.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadc960700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad81ac780 0x7faad818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:16.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd962700 1 -- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad81ac380 msgr2=0x7faad818a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:16.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd962700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad81ac380 0x7faad818a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:16.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd962700 1 -- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad81ac780 msgr2=0x7faad818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:16.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd962700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad81ac780 0x7faad818a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:16.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd962700 1 -- 172.21.15.32:0/1819187812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad8184160 con 0x7faad8183520 2021-04-09T18:22:16.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.748 7faadd962700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad8183520 0x7faad8183920 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7faad000b5e0 tx=0x7faad000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:16.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.749 7faacdffb700 1 -- 172.21.15.32:0/1819187812 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad000bd70 con 0x7faad8183520 2021-04-09T18:22:16.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.749 7faacdffb700 1 -- 172.21.15.32:0/1819187812 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad0006730 con 0x7faad8183520 2021-04-09T18:22:16.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.749 7faadf345700 1 -- 172.21.15.32:0/1819187812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad8184160 con 0x7faad8183520 2021-04-09T18:22:16.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.749 7faacdffb700 1 -- 172.21.15.32:0/1819187812 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad00022b0 con 0x7faad8183520 2021-04-09T18:22:16.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.749 7faadf345700 1 -- 172.21.15.32:0/1819187812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad8184160 con 0x7faad8183520 2021-04-09T18:22:16.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.749 7faacdffb700 1 -- 172.21.15.32:0/1819187812 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faad0000a70 con 0x7faad8183520 2021-04-09T18:22:16.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.749 7faacdffb700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab8018930 0x7faab801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:16.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.750 7faacdffb700 1 -- 172.21.15.32:0/1819187812 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7faad0032a20 con 0x7faad8183520 2021-04-09T18:22:16.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.750 7faadd161700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab8018930 0x7faab801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:16.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.750 7faadd161700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab8018930 0x7faab801adc0 crc :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:16.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.769 7faadf345700 1 -- 172.21.15.32:0/1819187812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad8189cf0 con 0x7faad8183520 2021-04-09T18:22:16.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.772 7faacdffb700 1 -- 172.21.15.32:0/1819187812 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faad00063a0 con 0x7faad8183520 2021-04-09T18:22:17.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.963 7faadf345700 1 -- 172.21.15.32:0/1819187812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faad8189cf0 con 0x7faad8183520 2021-04-09T18:22:17.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.963 7faacdffb700 1 -- 172.21.15.32:0/1819187812 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2460 (secure 0 0 0) 0x7faad0006730 con 0x7faad8183520 2021-04-09T18:22:17.017 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 117 2021-04-09T18:22:17.017 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:17.018 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":117,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:13.785899","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7387,"mds_1":7390},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:stopping","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:17.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.965 7faadf345700 1 -- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab8018930 msgr2=0x7faab801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:17.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.965 7faadf345700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab8018930 0x7faab801adc0 crc :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:17.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.965 7faadf345700 1 -- 172.21.15.32:0/1819187812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad8183520 msgr2=0x7faad8183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:17.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.965 7faadf345700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad8183520 0x7faad8183920 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7faad000b5e0 tx=0x7faad000b980).stop 2021-04-09T18:22:17.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.966 7faadf345700 1 -- 172.21.15.32:0/1819187812 shutdown_connections 2021-04-09T18:22:17.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.966 7faadf345700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faab8018930 0x7faab801adc0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:17.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.966 7faadf345700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faad8183520 0x7faad8183920 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:17.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.966 7faadf345700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad81ac780 0x7faad818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:17.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.966 7faadf345700 1 --2- 172.21.15.32:0/1819187812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad81ac380 0x7faad818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:17.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.966 7faadf345700 1 -- 172.21.15.32:0/1819187812 >> 172.21.15.32:0/1819187812 conn(0x7faad819b890 msgr2=0x7faad81a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:17.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.966 7faadf345700 1 -- 172.21.15.32:0/1819187812 shutdown_connections 2021-04-09T18:22:17.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:16.966 7faadf345700 1 -- 172.21.15.32:0/1819187812 wait complete. 2021-04-09T18:22:17.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:13.785899', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7387, 'mds_1': 7390}, 'failed': [], 'damaged': [], 'stopped': [2], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:17.432 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7390:up:stopping 2021-04-09T18:22:18.434 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:18.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.531 7f00eafd7700 1 Processor -- start 2021-04-09T18:22:18.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.531 7f00eafd7700 1 -- start start 2021-04-09T18:22:18.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.531 7f00eafd7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00e40d7ca0 0x7f00e40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:18.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.531 7f00eafd7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f00e410da00 0x7f00e410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.531 7f00eafd7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e410d480 con 0x7f00e410d010 2021-04-09T18:22:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.531 7f00eafd7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00e40d9220 con 0x7f00e40d7ca0 2021-04-09T18:22:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.531 7f00eafd7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00e40d86c0 con 0x7f00e410da00 2021-04-09T18:22:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00e40d7ca0 0x7f00e40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00e40d7ca0 0x7f00e40d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49120/0 (socket says 172.21.15.32:49120) 2021-04-09T18:22:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e3fff700 1 -- 172.21.15.32:0/2581245650 learned_addr learned my addr 172.21.15.32:0/2581245650 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e95f4700 1 --2- 172.21.15.32:0/2581245650 >> v2:172.21.15.86:3300/0 conn(0x7f00e410da00 0x7f00e410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e3fff700 1 -- 172.21.15.32:0/2581245650 >> v2:172.21.15.86:3300/0 conn(0x7f00e410da00 msgr2=0x7f00e410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e3fff700 1 --2- 172.21.15.32:0/2581245650 >> v2:172.21.15.86:3300/0 conn(0x7f00e410da00 0x7f00e410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e3fff700 1 -- 172.21.15.32:0/2581245650 >> v1:172.21.15.86:6789/0 conn(0x7f00e410d010 legacy=0x7f00e40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:18.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e37fe700 1 -- 172.21.15.32:0/2581245650 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281902273 0 0) 0x7f00cc000c60 con 0x7f00e410d010 2021-04-09T18:22:18.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e3fff700 1 -- 172.21.15.32:0/2581245650 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e40d1f70 con 0x7f00e40d7ca0 2021-04-09T18:22:18.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e3fff700 1 --2- 172.21.15.32:0/2581245650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e40d7ca0 0x7f00e40d80a0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f00d4006700 tx=0x7f00d4009b80).ready entity=mon.0 client_cookie=f70e7eaed507fb24 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:18.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e37fe700 1 -- 172.21.15.32:0/2581245650 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d400c0c0 con 0x7f00e40d7ca0 2021-04-09T18:22:18.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e37fe700 1 -- 172.21.15.32:0/2581245650 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d400c440 con 0x7f00e40d7ca0 2021-04-09T18:22:18.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00e37fe700 1 -- 172.21.15.32:0/2581245650 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00d400c920 con 0x7f00e40d7ca0 2021-04-09T18:22:18.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00eafd7700 1 -- 172.21.15.32:0/2581245650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e40d7ca0 msgr2=0x7f00e40d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:18.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.532 7f00eafd7700 1 --2- 172.21.15.32:0/2581245650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e40d7ca0 0x7f00e40d80a0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f00d4006700 tx=0x7f00d4009b80).stop 2021-04-09T18:22:18.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.533 7f00eafd7700 1 -- 172.21.15.32:0/2581245650 shutdown_connections 2021-04-09T18:22:18.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.533 7f00eafd7700 1 --2- 172.21.15.32:0/2581245650 >> v2:172.21.15.86:3300/0 conn(0x7f00e410da00 0x7f00e410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.533 7f00eafd7700 1 --2- 172.21.15.32:0/2581245650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e40d7ca0 0x7f00e40d80a0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.533 7f00eafd7700 1 -- 172.21.15.32:0/2581245650 >> 172.21.15.32:0/2581245650 conn(0x7f00e40ce5f0 msgr2=0x7f00e40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:18.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.533 7f00eafd7700 1 -- 172.21.15.32:0/2581245650 shutdown_connections 2021-04-09T18:22:18.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.533 7f00eafd7700 1 -- 172.21.15.32:0/2581245650 wait complete. 2021-04-09T18:22:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.553 7f00eafd7700 1 Processor -- start 2021-04-09T18:22:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.553 7f00eafd7700 1 -- start start 2021-04-09T18:22:18.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00eafd7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e41a5210 0x7f00e41a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:18.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00eafd7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00e41a6340 0x7f00e41a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:18.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00eafd7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00e40abaa0 0x7f00e40abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00eafd7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e41a6d60 con 0x7f00e41a5210 2021-04-09T18:22:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00eafd7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e40ac4c0 con 0x7f00e41a6340 2021-04-09T18:22:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00eafd7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e40ac670 con 0x7f00e40abaa0 2021-04-09T18:22:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e8df3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e41a5210 0x7f00e41a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e3fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00e41a6340 0x7f00e41a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:18.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e8df3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e41a5210 0x7f00e41a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49128/0 (socket says 172.21.15.32:49128) 2021-04-09T18:22:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e8df3700 1 -- 172.21.15.32:0/3171242261 learned_addr learned my addr 172.21.15.32:0/3171242261 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e95f4700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00e40abaa0 0x7f00e40abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e8df3700 1 -- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00e40abaa0 msgr2=0x7f00e40abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e8df3700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00e40abaa0 0x7f00e40abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e8df3700 1 -- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00e41a6340 msgr2=0x7f00e41a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e8df3700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00e41a6340 0x7f00e41a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e8df3700 1 -- 172.21.15.32:0/3171242261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e40ac890 con 0x7f00e41a5210 2021-04-09T18:22:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.554 7f00e3fff700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00e41a6340 0x7f00e41a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.555 7f00e8df3700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e41a5210 0x7f00e41a56c0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f00cc006f20 tx=0x7f00cc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.555 7f00e17fa700 1 -- 172.21.15.32:0/3171242261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc00bd80 con 0x7f00e41a5210 2021-04-09T18:22:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.555 7f00e17fa700 1 -- 172.21.15.32:0/3171242261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc001080 con 0x7f00e41a5210 2021-04-09T18:22:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.555 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e40ac890 con 0x7f00e41a5210 2021-04-09T18:22:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.555 7f00e17fa700 1 -- 172.21.15.32:0/3171242261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc001550 con 0x7f00e41a5210 2021-04-09T18:22:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.555 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e40ac890 con 0x7f00e41a5210 2021-04-09T18:22:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.556 7f00e17fa700 1 -- 172.21.15.32:0/3171242261 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f00cc001080 con 0x7f00e41a5210 2021-04-09T18:22:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.556 7f00e17fa700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00c80188c0 0x7f00c801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.556 7f00e17fa700 1 -- 172.21.15.32:0/3171242261 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f00cc0327c0 con 0x7f00e41a5210 2021-04-09T18:22:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.556 7f00e3fff700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00c80188c0 0x7f00c801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.557 7f00e3fff700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00c80188c0 0x7f00c801adc0 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:18.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.576 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e41a6d60 con 0x7f00e41a5210 2021-04-09T18:22:18.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.579 7f00e17fa700 1 -- 172.21.15.32:0/3171242261 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f00cc006bf0 con 0x7f00e41a5210 2021-04-09T18:22:18.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.769 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00e41a6d60 con 0x7f00e41a5210 2021-04-09T18:22:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.770 7f00e17fa700 1 -- 172.21.15.32:0/3171242261 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2460 (secure 0 0 0) 0x7f00cc00bd80 con 0x7f00e41a5210 2021-04-09T18:22:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 117 2021-04-09T18:22:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:18.824 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":117,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:13.785899","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7387,"mds_1":7390},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:stopping","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:18.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00c80188c0 msgr2=0x7f00c801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00c80188c0 0x7f00c801adc0 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e41a5210 msgr2=0x7f00e41a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e41a5210 0x7f00e41a56c0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f00cc006f20 tx=0x7f00cc00b990).stop 2021-04-09T18:22:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 shutdown_connections 2021-04-09T18:22:18.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00c80188c0 0x7f00c801adc0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00e40abaa0 0x7f00e40abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00e41a6340 0x7f00e41a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 --2- 172.21.15.32:0/3171242261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00e41a5210 0x7f00e41a56c0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:18.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 >> 172.21.15.32:0/3171242261 conn(0x7f00e41a99c0 msgr2=0x7f00e417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:18.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 shutdown_connections 2021-04-09T18:22:18.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:18.772 7f00eafd7700 1 -- 172.21.15.32:0/3171242261 wait complete. 2021-04-09T18:22:19.238 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:13.785899', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7387, 'mds_1': 7390}, 'failed': [], 'damaged': [], 'stopped': [2], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:19.239 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7390:up:stopping 2021-04-09T18:22:20.240 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:20.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.347 7fb98b0f6700 1 Processor -- start 2021-04-09T18:22:20.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.347 7fb98b0f6700 1 -- start start 2021-04-09T18:22:20.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.348 7fb98b0f6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb98410d010 0x7fb9840d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:20.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.348 7fb98b0f6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98410d480 con 0x7fb9840d1da0 2021-04-09T18:22:20.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.348 7fb98b0f6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98410e1f0 con 0x7fb9840d8490 2021-04-09T18:22:20.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.348 7fb98b0f6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9840d8d30 con 0x7fb98410d010 2021-04-09T18:22:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.348 7fb988f12700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb98410d010 0x7fb9840d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.348 7fb983fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb9840d1da0 0x7fb9840d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58724/0 (socket says 172.21.15.32:58724) 2021-04-09T18:22:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.348 7fb983fff700 1 -- 172.21.15.32:0/3281489200 learned_addr learned my addr 172.21.15.32:0/3281489200 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb9837fe700 1 -- 172.21.15.32:0/3281489200 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019560472 0 0) 0x7fb974001060 con 0x7fb9840d1da0 2021-04-09T18:22:20.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb9837fe700 1 -- 172.21.15.32:0/3281489200 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb970002fe0 con 0x7fb9840d1da0 2021-04-09T18:22:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb988f12700 1 -- 172.21.15.32:0/3281489200 >> v1:172.21.15.86:6789/0 conn(0x7fb9840d8490 legacy=0x7fb98410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb988f12700 1 -- 172.21.15.32:0/3281489200 >> v1:172.21.15.32:6789/0 conn(0x7fb9840d1da0 legacy=0x7fb9840d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb9837fe700 1 -- 172.21.15.32:0/3281489200 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424689042 0 0) 0x7fb97c0009b0 con 0x7fb9840d8490 2021-04-09T18:22:20.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb988f12700 1 -- 172.21.15.32:0/3281489200 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9840d9220 con 0x7fb98410d010 2021-04-09T18:22:20.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb988f12700 1 --2- 172.21.15.32:0/3281489200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb98410d010 0x7fb9840d1780 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7fb96c006650 tx=0x7fb96c009a80).ready entity=mon.0 client_cookie=2672cdea5c8eee60 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:20.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb9837fe700 1 -- 172.21.15.32:0/3281489200 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb96c00c0c0 con 0x7fb98410d010 2021-04-09T18:22:20.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb9837fe700 1 -- 172.21.15.32:0/3281489200 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96c00c3c0 con 0x7fb98410d010 2021-04-09T18:22:20.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb9837fe700 1 -- 172.21.15.32:0/3281489200 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb96c00c8a0 con 0x7fb98410d010 2021-04-09T18:22:20.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb98b0f6700 1 -- 172.21.15.32:0/3281489200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb98410d010 msgr2=0x7fb9840d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:20.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.349 7fb98b0f6700 1 --2- 172.21.15.32:0/3281489200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb98410d010 0x7fb9840d1780 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7fb96c006650 tx=0x7fb96c009a80).stop 2021-04-09T18:22:20.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.350 7fb98b0f6700 1 -- 172.21.15.32:0/3281489200 shutdown_connections 2021-04-09T18:22:20.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.350 7fb98b0f6700 1 --2- 172.21.15.32:0/3281489200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb98410d010 0x7fb9840d1780 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:20.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.350 7fb98b0f6700 1 -- 172.21.15.32:0/3281489200 >> 172.21.15.32:0/3281489200 conn(0x7fb9840ce5f0 msgr2=0x7fb9840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:20.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.350 7fb98b0f6700 1 -- 172.21.15.32:0/3281489200 shutdown_connections 2021-04-09T18:22:20.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.350 7fb98b0f6700 1 -- 172.21.15.32:0/3281489200 wait complete. 2021-04-09T18:22:20.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.370 7fb98b0f6700 1 Processor -- start 2021-04-09T18:22:20.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.370 7fb98b0f6700 1 -- start start 2021-04-09T18:22:20.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb98b0f6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9841a17b0 0x7fb98417ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:20.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb98b0f6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb98417c5b0 0x7fb98417e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:20.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb98b0f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9840addd0 0x7fb9840ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:20.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb98b0f6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98417eb70 con 0x7fb9840addd0 2021-04-09T18:22:20.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb98b0f6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98417ca90 con 0x7fb9841a17b0 2021-04-09T18:22:20.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb98b0f6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98417d080 con 0x7fb98417c5b0 2021-04-09T18:22:20.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb989713700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9840addd0 0x7fb9840ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:20.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb989713700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9840addd0 0x7fb9840ae1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49142/0 (socket says 172.21.15.32:49142) 2021-04-09T18:22:20.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb989713700 1 -- 172.21.15.32:0/4152939786 learned_addr learned my addr 172.21.15.32:0/4152939786 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:20.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb988f12700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9841a17b0 0x7fb98417ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:20.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb983fff700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb98417c5b0 0x7fb98417e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:20.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb989713700 1 -- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb98417c5b0 msgr2=0x7fb98417e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:20.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb989713700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb98417c5b0 0x7fb98417e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:20.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb989713700 1 -- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9841a17b0 msgr2=0x7fb98417ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:20.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb989713700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9841a17b0 0x7fb98417ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:20.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb989713700 1 -- 172.21.15.32:0/4152939786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9840ae7f0 con 0x7fb9840addd0 2021-04-09T18:22:20.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb988f12700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9841a17b0 0x7fb98417ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:20.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb989713700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9840addd0 0x7fb9840ae1d0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7fb97c00b530 tx=0x7fb97c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:20.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb9817fa700 1 -- 172.21.15.32:0/4152939786 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97c006030 con 0x7fb9840addd0 2021-04-09T18:22:20.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.371 7fb9817fa700 1 -- 172.21.15.32:0/4152939786 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c0062f0 con 0x7fb9840addd0 2021-04-09T18:22:20.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.372 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9840ae7f0 con 0x7fb9840addd0 2021-04-09T18:22:20.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.372 7fb9817fa700 1 -- 172.21.15.32:0/4152939786 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97c005770 con 0x7fb9840addd0 2021-04-09T18:22:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.372 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9840ae7f0 con 0x7fb9840addd0 2021-04-09T18:22:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.372 7fb9817fa700 1 -- 172.21.15.32:0/4152939786 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb97c0054f0 con 0x7fb9840addd0 2021-04-09T18:22:20.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.372 7fb9817fa700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.372 7fb9817fa700 1 -- 172.21.15.32:0/4152939786 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb97c032b80 con 0x7fb9840addd0 2021-04-09T18:22:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.372 7fb988f12700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.373 7fb988f12700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 crc :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:20.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.392 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9841b7070 con 0x7fb9840addd0 2021-04-09T18:22:20.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.396 7fb9817fa700 1 -- 172.21.15.32:0/4152939786 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb97c006b90 con 0x7fb9840addd0 2021-04-09T18:22:20.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.587 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb98410d480 con 0x7fb9840addd0 2021-04-09T18:22:20.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.588 7fb9817fa700 1 -- 172.21.15.32:0/4152939786 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2795 (secure 0 0 0) 0x7fb97c006210 con 0x7fb9840addd0 2021-04-09T18:22:20.642 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 118 2021-04-09T18:22:20.642 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:20.642 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":118,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7809,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/633563485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":633563485},{"type":"v1","addr":"172.21.15.32:6834","nonce":633563485}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":118},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:13.785899","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7387,"mds_1":7390},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:stopping","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:20.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 msgr2=0x7fb96801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:20.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 crc :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:20.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9840addd0 msgr2=0x7fb9840ae1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9840addd0 0x7fb9840ae1d0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7fb97c00b530 tx=0x7fb97c00b8d0).stop 2021-04-09T18:22:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 shutdown_connections 2021-04-09T18:22:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9840addd0 0x7fb9840ae1d0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb98417c5b0 0x7fb98417e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 --2- 172.21.15.32:0/4152939786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9841a17b0 0x7fb98417ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 >> 172.21.15.32:0/4152939786 conn(0x7fb9841905b0 msgr2=0x7fb9841b75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 shutdown_connections 2021-04-09T18:22:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:20.590 7fb98b0f6700 1 -- 172.21.15.32:0/4152939786 wait complete. 2021-04-09T18:22:21.056 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:13.785899', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7387, 'mds_1': 7390}, 'failed': [], 'damaged': [], 'stopped': [2], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:21.057 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7390:up:stopping 2021-04-09T18:22:22.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:22.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.165 7f9dfda8e700 1 Processor -- start 2021-04-09T18:22:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.165 7f9dfda8e700 1 -- start start 2021-04-09T18:22:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9dfda8e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9df80cffa0 0x7f9df80d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9dfda8e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df80d7280 con 0x7f9df80d6c80 2021-04-09T18:22:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9dfda8e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df80d83f0 con 0x7f9df80d7800 2021-04-09T18:22:22.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9dfda8e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df80d7080 con 0x7f9df80cffa0 2021-04-09T18:22:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9df77fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9df80cffa0 0x7f9df80d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9df7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9df80d6c80 0x7f9df810cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58738/0 (socket says 172.21.15.32:58738) 2021-04-09T18:22:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9df7fff700 1 -- 172.21.15.32:0/1904631723 learned_addr learned my addr 172.21.15.32:0/1904631723 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:22.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9df67fc700 1 -- 172.21.15.32:0/1904631723 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716884266 0 0) 0x7f9df0000d30 con 0x7f9df80d6c80 2021-04-09T18:22:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9df77fe700 1 -- 172.21.15.32:0/1904631723 >> v1:172.21.15.86:6789/0 conn(0x7f9df80d7800 legacy=0x7f9df80d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9df77fe700 1 -- 172.21.15.32:0/1904631723 >> v1:172.21.15.32:6789/0 conn(0x7f9df80d6c80 legacy=0x7f9df810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.166 7f9df77fe700 1 -- 172.21.15.32:0/1904631723 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df80d91e0 con 0x7f9df80cffa0 2021-04-09T18:22:22.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9df77fe700 1 --2- 172.21.15.32:0/1904631723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df80cffa0 0x7f9df80d2440 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f9de8006650 tx=0x7f9de8009a80).ready entity=mon.0 client_cookie=eaa9e304444a6dff server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:22.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9df67fc700 1 -- 172.21.15.32:0/1904631723 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9de80009b0 con 0x7f9df80cffa0 2021-04-09T18:22:22.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9df67fc700 1 -- 172.21.15.32:0/1904631723 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de800c0c0 con 0x7f9df80cffa0 2021-04-09T18:22:22.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9df67fc700 1 -- 172.21.15.32:0/1904631723 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9de800c510 con 0x7f9df80cffa0 2021-04-09T18:22:22.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9dfda8e700 1 -- 172.21.15.32:0/1904631723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df80cffa0 msgr2=0x7f9df80d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:22.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9dfda8e700 1 --2- 172.21.15.32:0/1904631723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df80cffa0 0x7f9df80d2440 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f9de8006650 tx=0x7f9de8009a80).stop 2021-04-09T18:22:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9dfda8e700 1 -- 172.21.15.32:0/1904631723 shutdown_connections 2021-04-09T18:22:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9dfda8e700 1 --2- 172.21.15.32:0/1904631723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df80cffa0 0x7f9df80d2440 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9dfda8e700 1 -- 172.21.15.32:0/1904631723 >> 172.21.15.32:0/1904631723 conn(0x7f9df80ce5c0 msgr2=0x7f9df80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.167 7f9dfda8e700 1 -- 172.21.15.32:0/1904631723 shutdown_connections 2021-04-09T18:22:22.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.168 7f9dfda8e700 1 -- 172.21.15.32:0/1904631723 wait complete. 2021-04-09T18:22:22.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9dfda8e700 1 Processor -- start 2021-04-09T18:22:22.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9dfda8e700 1 -- start start 2021-04-09T18:22:22.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9dfda8e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df81a6250 0x7f9df81a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:22.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9dfda8e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df81a7470 0x7f9df81a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:22.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9dfda8e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df81aba60 0x7f9df81abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:22.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9dfda8e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81a6c70 con 0x7f9df81aba60 2021-04-09T18:22:22.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9dfda8e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81a6ef0 con 0x7f9df81a7470 2021-04-09T18:22:22.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9dfda8e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81a7e90 con 0x7f9df81a6250 2021-04-09T18:22:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9df7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df81aba60 0x7f9df81abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.188 7f9df7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df81aba60 0x7f9df81abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49154/0 (socket says 172.21.15.32:49154) 2021-04-09T18:22:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df7fff700 1 -- 172.21.15.32:0/973841565 learned_addr learned my addr 172.21.15.32:0/973841565 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df77fe700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df81a6250 0x7f9df81a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:22.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df6ffd700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df81a7470 0x7f9df81a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df7fff700 1 -- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df81a6250 msgr2=0x7f9df81a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df7fff700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df81a6250 0x7f9df81a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df7fff700 1 -- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df81a7470 msgr2=0x7f9df81a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:22.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df7fff700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df81a7470 0x7f9df81a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:22.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df7fff700 1 -- 172.21.15.32:0/973841565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df8092900 con 0x7f9df81aba60 2021-04-09T18:22:22.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df6ffd700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df81a7470 0x7f9df81a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:22.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9df7fff700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df81aba60 0x7f9df81abe60 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f9df0007f90 tx=0x7f9df000abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:22.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9ddffff700 1 -- 172.21.15.32:0/973841565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df0000cc0 con 0x7f9df81aba60 2021-04-09T18:22:22.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9ddffff700 1 -- 172.21.15.32:0/973841565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df00070d0 con 0x7f9df81aba60 2021-04-09T18:22:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df8092900 con 0x7f9df81aba60 2021-04-09T18:22:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9ddffff700 1 -- 172.21.15.32:0/973841565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df0007520 con 0x7f9df81aba60 2021-04-09T18:22:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.189 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df8092900 con 0x7f9df81aba60 2021-04-09T18:22:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.190 7f9ddffff700 1 -- 172.21.15.32:0/973841565 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9df00072b0 con 0x7f9df81aba60 2021-04-09T18:22:22.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.190 7f9ddffff700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9dd8018930 0x7f9dd801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.190 7f9ddffff700 1 -- 172.21.15.32:0/973841565 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9df0032d30 con 0x7f9df81aba60 2021-04-09T18:22:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.190 7f9df77fe700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9dd8018930 0x7f9dd801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:22.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.191 7f9df77fe700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9dd8018930 0x7f9dd801adc0 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:22.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.210 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df817cae0 con 0x7f9df81aba60 2021-04-09T18:22:22.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.214 7f9ddffff700 1 -- 172.21.15.32:0/973841565 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9df0007d00 con 0x7f9df81aba60 2021-04-09T18:22:22.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.406 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9df80d7280 con 0x7f9df81aba60 2021-04-09T18:22:22.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.406 7f9ddffff700 1 -- 172.21.15.32:0/973841565 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2795 (secure 0 0 0) 0x7f9df00070d0 con 0x7f9df81aba60 2021-04-09T18:22:22.460 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 118 2021-04-09T18:22:22.460 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:22.461 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":118,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7809,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/633563485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":633563485},{"type":"v1","addr":"172.21.15.32:6834","nonce":633563485}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":118},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:13.785899","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7387,"mds_1":7390},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_7390":{"gid":7390,"name":"d","rank":1,"incarnation":110,"state":"up:stopping","state_seq":5,"addr":"172.21.15.86:6837/867563018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":867563018},{"type":"v1","addr":"172.21.15.86:6837","nonce":867563018}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:22.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.408 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9dd8018930 msgr2=0x7f9dd801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:22.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.408 7f9dfda8e700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9dd8018930 0x7f9dd801adc0 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:22.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.408 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df81aba60 msgr2=0x7f9df81abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:22.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.408 7f9dfda8e700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df81aba60 0x7f9df81abe60 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f9df0007f90 tx=0x7f9df000abd0).stop 2021-04-09T18:22:22.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.409 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 shutdown_connections 2021-04-09T18:22:22.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.409 7f9dfda8e700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9dd8018930 0x7f9dd801adc0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:22.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.409 7f9dfda8e700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9df81aba60 0x7f9df81abe60 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:22.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.409 7f9dfda8e700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df81a7470 0x7f9df81a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:22.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.409 7f9dfda8e700 1 --2- 172.21.15.32:0/973841565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df81a6250 0x7f9df81a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:22.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.409 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 >> 172.21.15.32:0/973841565 conn(0x7f9df818ce60 msgr2=0x7f9df8094920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:22.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.409 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 shutdown_connections 2021-04-09T18:22:22.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:22.409 7f9dfda8e700 1 -- 172.21.15.32:0/973841565 wait complete. 2021-04-09T18:22:22.874 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:13.785899', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7387, 'mds_1': 7390}, 'failed': [], 'damaged': [], 'stopped': [2], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_7390': {'gid': 7390, 'name': 'd', 'rank': 1, 'incarnation': 110, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.86:6837/867563018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 867563018}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 867563018}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:22.875 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7390:up:stopping 2021-04-09T18:22:23.543 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:22:23.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:24.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.983 7f8d3f89b700 1 Processor -- start 2021-04-09T18:22:24.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.984 7f8d3f89b700 1 -- start start 2021-04-09T18:22:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.984 7f8d3f89b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8d380cffa0 0x7f8d380d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.984 7f8d3f89b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380d6c80 0x7f8d3810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.984 7f8d3f89b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d380d7280 con 0x7f8d380d7800 2021-04-09T18:22:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.984 7f8d3f89b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d380d7080 con 0x7f8d380cffa0 2021-04-09T18:22:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.984 7f8d3f89b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d380d91e0 con 0x7f8d380d6c80 2021-04-09T18:22:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.984 7f8d3d6b7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8d380cffa0 0x7f8d380d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3d6b7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8d380cffa0 0x7f8d380d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49160/0 (socket says 172.21.15.32:49160) 2021-04-09T18:22:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3d6b7700 1 -- 172.21.15.32:0/3491897675 learned_addr learned my addr 172.21.15.32:0/3491897675 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:24.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3deb8700 1 --2- 172.21.15.32:0/3491897675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380d6c80 0x7f8d3810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:24.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3d6b7700 1 -- 172.21.15.32:0/3491897675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380d6c80 msgr2=0x7f8d3810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:24.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3d6b7700 1 --2- 172.21.15.32:0/3491897675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380d6c80 0x7f8d3810cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3d6b7700 1 -- 172.21.15.32:0/3491897675 >> v1:172.21.15.86:6789/0 conn(0x7f8d380d7800 legacy=0x7f8d380d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3d6b7700 1 -- 172.21.15.32:0/3491897675 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d380d83f0 con 0x7f8d380cffa0 2021-04-09T18:22:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3deb8700 1 --2- 172.21.15.32:0/3491897675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380d6c80 0x7f8d3810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:24.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3d6b7700 1 --2- 172.21.15.32:0/3491897675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d380cffa0 0x7f8d380d2440 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f8d30006700 tx=0x7f8d30009b80).ready entity=mon.0 client_cookie=292d24a480cd7296 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d37fff700 1 -- 172.21.15.32:0/3491897675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d3000c0c0 con 0x7f8d380cffa0 2021-04-09T18:22:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d37fff700 1 -- 172.21.15.32:0/3491897675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3000c440 con 0x7f8d380cffa0 2021-04-09T18:22:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d37fff700 1 -- 172.21.15.32:0/3491897675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d3000c920 con 0x7f8d380cffa0 2021-04-09T18:22:24.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3f89b700 1 -- 172.21.15.32:0/3491897675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d380cffa0 msgr2=0x7f8d380d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.985 7f8d3f89b700 1 --2- 172.21.15.32:0/3491897675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d380cffa0 0x7f8d380d2440 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f8d30006700 tx=0x7f8d30009b80).stop 2021-04-09T18:22:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.986 7f8d3f89b700 1 -- 172.21.15.32:0/3491897675 shutdown_connections 2021-04-09T18:22:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.986 7f8d3f89b700 1 --2- 172.21.15.32:0/3491897675 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380d6c80 0x7f8d3810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.986 7f8d3f89b700 1 --2- 172.21.15.32:0/3491897675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d380cffa0 0x7f8d380d2440 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.986 7f8d3f89b700 1 -- 172.21.15.32:0/3491897675 >> 172.21.15.32:0/3491897675 conn(0x7f8d380ce5c0 msgr2=0x7f8d380cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.986 7f8d3f89b700 1 -- 172.21.15.32:0/3491897675 shutdown_connections 2021-04-09T18:22:24.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:23.986 7f8d3f89b700 1 -- 172.21.15.32:0/3491897675 wait complete. 2021-04-09T18:22:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.006 7f8d3f89b700 1 Processor -- start 2021-04-09T18:22:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.006 7f8d3f89b700 1 -- start start 2021-04-09T18:22:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3f89b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380a80c0 0x7f8d3817bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3f89b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d3817b1d0 0x7f8d3817b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3f89b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d3808dc50 0x7f8d3808e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3f89b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3817bc70 con 0x7f8d3817b1d0 2021-04-09T18:22:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3f89b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3808e670 con 0x7f8d3808dc50 2021-04-09T18:22:24.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3f89b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3808e850 con 0x7f8d380a80c0 2021-04-09T18:22:24.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3ceb6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d3817b1d0 0x7f8d3817b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:24.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3ceb6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d3817b1d0 0x7f8d3817b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49168/0 (socket says 172.21.15.32:49168) 2021-04-09T18:22:24.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3ceb6700 1 -- 172.21.15.32:0/3223856503 learned_addr learned my addr 172.21.15.32:0/3223856503 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:24.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3d6b7700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380a80c0 0x7f8d3817bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:24.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.007 7f8d3deb8700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d3808dc50 0x7f8d3808e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:24.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d3ceb6700 1 -- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380a80c0 msgr2=0x7f8d3817bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d3ceb6700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380a80c0 0x7f8d3817bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d3ceb6700 1 -- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d3808dc50 msgr2=0x7f8d3808e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d3ceb6700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d3808dc50 0x7f8d3808e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d3ceb6700 1 -- 172.21.15.32:0/3223856503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3808c660 con 0x7f8d3817b1d0 2021-04-09T18:22:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d3ceb6700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d3817b1d0 0x7f8d3817b5d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f8d2c006ed0 tx=0x7f8d2c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d35ffb700 1 -- 172.21.15.32:0/3223856503 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d2c006020 con 0x7f8d3817b1d0 2021-04-09T18:22:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d35ffb700 1 -- 172.21.15.32:0/3223856503 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c006360 con 0x7f8d3817b1d0 2021-04-09T18:22:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3808c660 con 0x7f8d3817b1d0 2021-04-09T18:22:24.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d35ffb700 1 -- 172.21.15.32:0/3223856503 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d2c005730 con 0x7f8d3817b1d0 2021-04-09T18:22:24.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.008 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3808c660 con 0x7f8d3817b1d0 2021-04-09T18:22:24.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.009 7f8d35ffb700 1 -- 172.21.15.32:0/3223856503 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8d2c0054b0 con 0x7f8d3817b1d0 2021-04-09T18:22:24.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.009 7f8d35ffb700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d18018930 0x7f8d1801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:24.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.009 7f8d35ffb700 1 -- 172.21.15.32:0/3223856503 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8d2c032a40 con 0x7f8d3817b1d0 2021-04-09T18:22:24.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.009 7f8d3d6b7700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d18018930 0x7f8d1801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:24.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.009 7f8d3d6b7700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d18018930 0x7f8d1801adc0 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.029 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d381b70c0 con 0x7f8d3817b1d0 2021-04-09T18:22:24.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.033 7f8d35ffb700 1 -- 172.21.15.32:0/3223856503 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8d2c006000 con 0x7f8d3817b1d0 2021-04-09T18:22:24.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.226 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d381b70c0 con 0x7f8d3817b1d0 2021-04-09T18:22:24.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.226 7f8d35ffb700 1 -- 172.21.15.32:0/3223856503 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+2446 (secure 0 0 0) 0x7f8d2c00bd50 con 0x7f8d3817b1d0 2021-04-09T18:22:24.279 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 119 2021-04-09T18:22:24.280 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:24.280 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":119,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7809,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/633563485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":633563485},{"type":"v1","addr":"172.21.15.32:6834","nonce":633563485}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":118},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:23.428551","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d18018930 msgr2=0x7f8d1801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d18018930 0x7f8d1801adc0 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d3817b1d0 msgr2=0x7f8d3817b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d3817b1d0 0x7f8d3817b5d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f8d2c006ed0 tx=0x7f8d2c00b960).stop 2021-04-09T18:22:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 shutdown_connections 2021-04-09T18:22:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d18018930 0x7f8d1801adc0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d3817b1d0 0x7f8d3817b5d0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d3808dc50 0x7f8d3808e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 --2- 172.21.15.32:0/3223856503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d380a80c0 0x7f8d3817bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 >> 172.21.15.32:0/3223856503 conn(0x7f8d38189b00 msgr2=0x7f8d381921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 shutdown_connections 2021-04-09T18:22:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:24.229 7f8d3f89b700 1 -- 172.21.15.32:0/3223856503 wait complete. 2021-04-09T18:22:24.693 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 119, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:21:52.329633', 'modified': '2021-04-09 18:22:23.428551', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7387}, 'failed': [], 'damaged': [], 'stopped': [1, 2], 'info': {'gid_7387': {'gid': 7387, 'name': 'b', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6835/3998950443', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3998950443}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 3998950443}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:24.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:22:24.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:22:24.821 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:22:24.821 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:22:24.821 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:22:24.822 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 7387, 2021-04-09T18:22:24.822 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:22:24.822 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:22:24.822 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 32.435107979000001, 2021-04-09T18:22:24.823 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 119, 2021-04-09T18:22:24.823 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 95, 2021-04-09T18:22:24.823 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 95, 2021-04-09T18:22:24.824 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 36.874703615000001 2021-04-09T18:22:24.824 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:22:24.826 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 7387, "want_state": "up:active", "state": "up:active", "rank_uptime": 32.435107979000001, "mdsmap_epoch": 119, "osdmap_epoch": 95, "osdmap_epoch_barrier": 95, "uptime": 36.874703615000001 } 2021-04-09T18:22:24.827 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:22:24.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:22:24.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:22:24.945 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:22:24.946 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:24.947 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:22:24.947 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:24.992 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:22:24.993 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:22:24.993 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:22:24.993 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:22:24.994 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:25.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[59410]: fuse finished with error 0 and tester_r 0 2021-04-09T18:22:25.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:25.067 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:22:25.068 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:25.112 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:22:25.113 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:22:25.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:22:25.146 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:25.228 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:25.229 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:22:25.230 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:22:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:22:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:22:25.274 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:22:25.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:22:25.275 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:22:25.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:22:25.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.450 7f4531d9c700 1 Processor -- start 2021-04-09T18:22:25.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.450 7f4531d9c700 1 -- start start 2021-04-09T18:22:25.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f4531d9c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f452c0d7ca0 0x7f452c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:25.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f4531d9c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f452c10da00 0x7f452c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:25.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f4531d9c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c10d480 con 0x7f452c10d010 2021-04-09T18:22:25.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f4531d9c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c0d9220 con 0x7f452c0d7ca0 2021-04-09T18:22:25.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f4531d9c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c0d86c0 con 0x7f452c10da00 2021-04-09T18:22:25.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452affd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f452c0d7ca0 0x7f452c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:25.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452b7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f452c10d010 0x7f452c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58764/0 (socket says 172.21.15.32:58764) 2021-04-09T18:22:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452b7fe700 1 -- 172.21.15.32:0/2291006197 learned_addr learned my addr 172.21.15.32:0/2291006197 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452bfff700 1 --2- 172.21.15.32:0/2291006197 >> v2:172.21.15.86:3300/0 conn(0x7f452c10da00 0x7f452c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452affd700 1 -- 172.21.15.32:0/2291006197 >> v2:172.21.15.86:3300/0 conn(0x7f452c10da00 msgr2=0x7f452c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452affd700 1 --2- 172.21.15.32:0/2291006197 >> v2:172.21.15.86:3300/0 conn(0x7f452c10da00 0x7f452c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452affd700 1 -- 172.21.15.32:0/2291006197 >> v1:172.21.15.32:6789/0 conn(0x7f452c10d010 legacy=0x7f452c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452affd700 1 -- 172.21.15.32:0/2291006197 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c0d1f70 con 0x7f452c0d7ca0 2021-04-09T18:22:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452a7fc700 1 -- 172.21.15.32:0/2291006197 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806753322 0 0) 0x7f451c000ce0 con 0x7f452c10d010 2021-04-09T18:22:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452bfff700 1 --2- 172.21.15.32:0/2291006197 >> v2:172.21.15.86:3300/0 conn(0x7f452c10da00 0x7f452c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.451 7f452affd700 1 --2- 172.21.15.32:0/2291006197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0d7ca0 0x7f452c0d80a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f4520006650 tx=0x7f4520009a80).ready entity=mon.0 client_cookie=6ad9de465fba09ba server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f452a7fc700 1 -- 172.21.15.32:0/2291006197 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45200009b0 con 0x7f452c0d7ca0 2021-04-09T18:22:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f452a7fc700 1 -- 172.21.15.32:0/2291006197 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452000c0c0 con 0x7f452c0d7ca0 2021-04-09T18:22:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f452a7fc700 1 -- 172.21.15.32:0/2291006197 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f452000c510 con 0x7f452c0d7ca0 2021-04-09T18:22:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f4531d9c700 1 -- 172.21.15.32:0/2291006197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0d7ca0 msgr2=0x7f452c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f4531d9c700 1 --2- 172.21.15.32:0/2291006197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0d7ca0 0x7f452c0d80a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f4520006650 tx=0x7f4520009a80).stop 2021-04-09T18:22:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f4531d9c700 1 -- 172.21.15.32:0/2291006197 shutdown_connections 2021-04-09T18:22:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f4531d9c700 1 --2- 172.21.15.32:0/2291006197 >> v2:172.21.15.86:3300/0 conn(0x7f452c10da00 0x7f452c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f4531d9c700 1 --2- 172.21.15.32:0/2291006197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c0d7ca0 0x7f452c0d80a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f4531d9c700 1 -- 172.21.15.32:0/2291006197 >> 172.21.15.32:0/2291006197 conn(0x7f452c0ce5f0 msgr2=0x7f452c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f4531d9c700 1 -- 172.21.15.32:0/2291006197 shutdown_connections 2021-04-09T18:22:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.452 7f4531d9c700 1 -- 172.21.15.32:0/2291006197 wait complete. 2021-04-09T18:22:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.472 7f4531d9c700 1 Processor -- start 2021-04-09T18:22:25.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f4531d9c700 1 -- start start 2021-04-09T18:22:25.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f4531d9c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c0aa2b0 0x7f452c09d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:25.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f4531d9c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c09d9b0 0x7f452c1943a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:25.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f4531d9c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c1949c0 0x7f452c195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:25.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f4531d9c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c1950a0 con 0x7f452c1949c0 2021-04-09T18:22:25.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f4531d9c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c194dc0 con 0x7f452c0aa2b0 2021-04-09T18:22:25.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f4531d9c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c195c40 con 0x7f452c09d9b0 2021-04-09T18:22:25.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f452bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c1949c0 0x7f452c195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:25.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f452bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c1949c0 0x7f452c195620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49184/0 (socket says 172.21.15.32:49184) 2021-04-09T18:22:25.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.473 7f452bfff700 1 -- 172.21.15.32:0/265841559 learned_addr learned my addr 172.21.15.32:0/265841559 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:25.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f452affd700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c09d9b0 0x7f452c1943a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:25.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f452b7fe700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c0aa2b0 0x7f452c09d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:25.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f452bfff700 1 -- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c09d9b0 msgr2=0x7f452c1943a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:25.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f452bfff700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c09d9b0 0x7f452c1943a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:25.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f452bfff700 1 -- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c0aa2b0 msgr2=0x7f452c09d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:25.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f452bfff700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c0aa2b0 0x7f452c09d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f452bfff700 1 -- 172.21.15.32:0/265841559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c195e60 con 0x7f452c1949c0 2021-04-09T18:22:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f452affd700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c09d9b0 0x7f452c1943a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f452bfff700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c1949c0 0x7f452c195620 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f4524009980 tx=0x7f452400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f4513fff700 1 -- 172.21.15.32:0/265841559 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f452400c2f0 con 0x7f452c1949c0 2021-04-09T18:22:25.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f4513fff700 1 -- 172.21.15.32:0/265841559 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4524006790 con 0x7f452c1949c0 2021-04-09T18:22:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f4531d9c700 1 -- 172.21.15.32:0/265841559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c195e60 con 0x7f452c1949c0 2021-04-09T18:22:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f4513fff700 1 -- 172.21.15.32:0/265841559 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f452400c7e0 con 0x7f452c1949c0 2021-04-09T18:22:25.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.474 7f4531d9c700 1 -- 172.21.15.32:0/265841559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c195e60 con 0x7f452c1949c0 2021-04-09T18:22:25.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.475 7f4513fff700 1 -- 172.21.15.32:0/265841559 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f45240011f0 con 0x7f452c1949c0 2021-04-09T18:22:25.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.475 7f4513fff700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f450c018930 0x7f450c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:25.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.475 7f4513fff700 1 -- 172.21.15.32:0/265841559 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4524032d70 con 0x7f452c1949c0 2021-04-09T18:22:25.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.475 7f452b7fe700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f450c018930 0x7f450c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:25.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.475 7f452b7fe700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f450c018930 0x7f450c01ae10 crc :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:25.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.495 7f4531d9c700 1 -- 172.21.15.32:0/265841559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452c182d10 con 0x7f452c1949c0 2021-04-09T18:22:25.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.498 7f4513fff700 1 -- 172.21.15.32:0/265841559 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f452400be10 con 0x7f452c1949c0 2021-04-09T18:22:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.687 7f4531d9c700 1 -- 172.21.15.32:0/265841559 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f452c10d480 con 0x7f452c1949c0 2021-04-09T18:22:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.688 7f4513fff700 1 -- 172.21.15.32:0/265841559 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v95) v1 ==== 81+0+30363 (secure 0 0 0) 0x7f45240011f0 con 0x7f452c1949c0 2021-04-09T18:22:25.744 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:25.744 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:22:25.744 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 95, 2021-04-09T18:22:25.744 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:22:25.745 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:22:25.745 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:21:53.333784", 2021-04-09T18:22:25.745 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:22:25.745 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:22:25.745 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:22:25.746 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:22:25.746 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:22:25.746 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:22:25.747 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:22:25.747 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:22:25.748 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:22:25.748 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:22:25.748 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:22:25.749 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:22:25.749 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:22:25.750 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:22:25.750 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:22:25.750 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 17, 2021-04-09T18:22:25.751 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:22:25.751 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:22:25.751 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:22:25.752 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:22:25.752 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:22:25.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.753 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:22:25.754 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:22:25.754 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:22:25.754 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:22:25.754 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:22:25.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:22:25.756 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:22:25.756 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:22:25.757 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:22:25.757 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:22:25.757 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:22:25.758 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:22:25.758 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:22:25.758 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:22:25.759 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:22:25.759 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:22:25.759 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:22:25.759 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:22:25.760 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:22:25.760 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:22:25.760 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:22:25.761 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:22:25.761 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:22:25.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.763 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:22:25.764 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:22:25.764 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:22:25.764 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:22:25.765 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:22:25.765 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:22:25.765 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:22:25.765 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:22:25.766 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:22:25.766 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:22:25.766 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:22:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:22:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:22:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:22:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:22:25.768 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:22:25.768 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:22:25.768 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:22:25.769 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:22:25.769 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:22:25.769 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:22:25.769 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:22:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:22:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:22:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:22:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:22:25.771 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:22:25.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.771 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:22:25.771 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:22:25.772 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:22:25.772 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:22:25.772 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:22:25.773 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:22:25.773 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:22:25.773 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:22:25.773 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:22:25.774 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:22:25.774 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:22:25.774 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:22:25.774 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:22:25.775 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:22:25.775 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.776 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 16, 2021-04-09T18:22:25.776 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:22:25.776 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:21:50.420671", 2021-04-09T18:22:25.777 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:22:25.777 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:22:25.777 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:22:25.777 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:22:25.778 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:22:25.778 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:22:25.778 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:22:25.778 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:22:25.779 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:22:25.779 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:22:25.780 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:22:25.780 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:22:25.780 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:22:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:22:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:22:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:22:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:22:25.782 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:22:25.782 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:22:25.782 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:22:25.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.783 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "94", 2021-04-09T18:22:25.783 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:22:25.783 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:22:25.784 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:22:25.784 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:22:25.784 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:22:25.785 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:22:25.785 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:22:25.785 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:22:25.786 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:22:25.786 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:22:25.786 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:22:25.787 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:22:25.787 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:22:25.787 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:22:25.787 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:22:25.788 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:22:25.788 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:22:25.788 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:22:25.788 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:22:25.789 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:22:25.789 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:22:25.789 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:22:25.789 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:22:25.790 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:22:25.790 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:22:25.790 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:22:25.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.791 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:22:25.791 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:22:25.791 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:22:25.791 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:22:25.792 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:22:25.792 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:22:25.792 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:22:25.792 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:22:25.793 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:22:25.793 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:22:25.793 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:22:25.794 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:22:25.794 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:22:25.794 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:22:25.795 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:22:25.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.796 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:22:25.796 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:22:25.797 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:22:25.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.798 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 17, 2021-04-09T18:22:25.798 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:22:25.798 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:21:51.790105", 2021-04-09T18:22:25.799 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:22:25.799 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:22:25.799 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:22:25.800 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:22:25.800 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:22:25.801 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:22:25.801 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:22:25.801 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:22:25.801 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:22:25.802 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:22:25.802 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:22:25.802 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:22:25.802 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:22:25.803 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:22:25.803 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:22:25.803 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:22:25.803 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:22:25.804 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:22:25.804 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:22:25.804 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:22:25.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.805 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "95", 2021-04-09T18:22:25.805 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:22:25.805 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:22:25.805 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:22:25.806 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:22:25.806 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:22:25.806 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:22:25.806 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:22:25.807 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:22:25.807 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:22:25.807 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:22:25.807 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:22:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:22:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:22:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:22:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:22:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:22:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:22:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:22:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:22:25.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:22:25.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:22:25.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:22:25.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:22:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:22:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:22:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:22:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:22:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:22:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:22:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:22:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:22:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:22:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:22:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:22:25.814 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:22:25.814 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:22:25.814 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:22:25.815 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:22:25.815 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:22:25.815 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:22:25.816 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:22:25.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.817 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:22:25.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:22:25.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:22:25.818 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:22:25.818 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:25.818 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:25.818 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:25.819 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:25.819 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:25.819 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:25.820 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:25.820 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 90, 2021-04-09T18:22:25.820 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:25.820 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:22:25.822 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:25.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.823 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:22:25.823 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:25.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.823 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.824 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:25.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:22:25.825 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:25.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:22:25.827 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:25.827 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.827 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.828 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:25.828 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.829 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:22:25.829 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:25.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:22:25.830 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:25.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.831 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.831 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:25.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.832 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:22:25.832 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:25.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.833 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.833 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:22:25.833 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:22:25.835 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:22:25.835 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:22:25.835 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:22:25.835 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:25.836 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:25.836 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:25.836 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.837 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:22:25.837 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:22:25.837 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:25.838 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:25.838 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:25.838 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:25.838 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:25.839 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:25.839 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:25.839 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:22:25.839 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:25.840 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:25.840 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:25.840 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:22:25.841 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:25.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.842 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.842 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:22:25.842 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:25.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.843 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.843 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:25.843 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:22:25.844 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:25.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.845 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:22:25.846 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:25.846 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.846 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.847 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:25.847 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.847 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:22:25.848 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:25.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.849 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.849 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:22:25.849 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:25.849 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.850 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.850 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:25.850 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:22:25.851 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:25.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:22:25.853 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:25.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.853 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.854 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:22:25.854 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:22:25.854 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:22:25.855 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:22:25.855 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:25.855 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:25.855 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:25.856 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.856 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:22:25.857 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:22:25.857 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:25.857 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:25.857 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:25.858 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:25.858 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:25.858 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:25.858 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:25.859 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 89, 2021-04-09T18:22:25.859 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:25.859 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:25.859 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:25.860 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.860 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:22:25.861 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:25.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.861 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.862 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:22:25.862 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:25.862 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.862 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.863 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:25.863 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.864 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:22:25.864 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:25.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.865 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:22:25.865 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:25.866 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.866 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.866 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:25.867 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.867 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.867 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:22:25.868 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:25.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.868 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.869 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:22:25.869 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:25.869 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.869 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.870 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:25.870 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.871 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.871 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:22:25.871 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:25.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.872 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:22:25.872 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:25.873 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.873 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.873 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:22:25.874 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:22:25.874 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:22:25.874 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:22:25.874 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:25.875 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:25.875 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:25.875 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.876 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:22:25.876 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:22:25.876 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:25.876 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:25.877 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:25.877 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:25.877 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:25.877 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:25.878 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:25.878 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 89, 2021-04-09T18:22:25.878 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:25.878 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:25.879 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:25.879 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.879 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.880 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:22:25.880 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:25.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.881 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:22:25.881 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:25.882 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.882 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.883 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:25.883 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.884 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.884 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:22:25.884 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:25.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.885 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:22:25.885 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:25.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.886 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.886 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:25.887 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.887 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.887 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:22:25.888 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:25.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.888 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.889 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:22:25.889 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:25.889 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.889 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.890 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:25.890 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.891 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:22:25.891 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:25.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.892 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.892 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:22:25.892 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:25.893 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.893 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.893 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:22:25.893 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:22:25.894 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:22:25.894 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:22:25.894 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:25.894 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:25.895 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:25.895 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.896 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:22:25.896 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:22:25.896 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:25.896 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:25.897 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:25.897 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:25.897 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:25.897 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:25.898 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:25.898 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 90, 2021-04-09T18:22:25.898 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:25.898 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:25.899 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:25.899 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:22:25.900 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:25.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:22:25.901 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:25.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.902 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.903 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:25.903 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.903 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.904 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:22:25.904 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:25.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.905 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:22:25.905 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:25.905 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.906 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.906 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:25.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.907 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.907 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:22:25.907 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:25.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.908 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:22:25.909 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:25.909 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.909 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.910 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:25.910 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.910 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.911 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:22:25.911 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:25.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.912 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:22:25.912 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:25.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.913 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.913 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:22:25.914 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:22:25.914 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:22:25.914 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:22:25.914 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:25.915 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:25.915 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:25.915 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.916 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:22:25.916 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:22:25.916 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:25.916 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:25.917 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:25.917 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:25.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:25.917 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:25.918 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:25.918 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 89, 2021-04-09T18:22:25.918 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:25.918 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:25.919 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:25.919 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.920 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:22:25.920 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:25.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.921 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.921 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:22:25.921 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:25.921 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.922 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.922 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:25.922 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.923 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:22:25.923 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:25.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.924 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:22:25.925 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:25.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.925 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.926 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:25.926 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.927 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:22:25.927 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:25.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.928 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.928 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:22:25.928 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:25.928 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.929 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.929 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:25.929 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.930 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.930 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:22:25.930 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:25.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.931 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:22:25.932 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:25.932 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.932 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.932 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:22:25.933 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:22:25.933 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:22:25.933 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:22:25.933 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:25.934 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:25.934 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:25.934 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.935 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:22:25.935 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:22:25.935 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:25.936 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:25.936 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:25.936 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:25.936 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:25.937 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:25.937 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:25.937 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 90, 2021-04-09T18:22:25.937 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:25.938 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:25.938 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:25.938 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.939 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:22:25.939 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:25.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.940 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:22:25.940 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:25.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.941 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.941 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:25.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:22:25.942 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:25.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.944 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:22:25.944 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:25.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.944 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.945 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:25.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.946 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:22:25.946 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:25.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.947 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:22:25.947 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:25.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.948 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.948 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:25.949 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.949 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.949 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.950 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:22:25.950 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:25.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.951 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:22:25.951 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:25.951 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.952 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.952 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:22:25.952 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:22:25.953 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:22:25.953 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:22:25.953 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:25.953 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:25.954 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:25.954 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.955 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:22:25.955 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:22:25.955 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:25.955 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:25.956 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:25.956 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:25.956 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:25.957 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:25.957 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:25.957 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 92, 2021-04-09T18:22:25.957 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:25.958 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:25.958 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:25.958 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.959 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:22:25.959 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:25.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.960 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.960 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:22:25.960 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:25.961 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.961 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.961 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:25.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:22:25.962 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:25.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.964 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:22:25.964 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:25.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.964 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.965 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:25.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.966 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:22:25.966 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:25.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.967 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:22:25.967 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:25.968 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.968 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.968 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:25.969 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:25.969 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.969 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:25.969 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:22:25.970 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:25.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:25.971 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:22:25.971 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:25.971 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.971 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.972 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:22:25.972 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:22:25.972 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:22:25.973 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:22:25.973 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:25.973 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:25.973 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:25.973 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:25.974 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.974 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:22:25.974 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:22:25.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.975 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:22:25.975 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:25.975 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:25.975 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:25.976 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:25.976 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:25.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.977 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:22:25.977 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:25.977 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:25.977 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:25.978 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:25.978 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:25.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.979 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:22:25.979 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:25.979 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:25.979 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:25.980 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:25.980 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:25.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.980 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.981 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:22:25.981 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:25.981 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:25.981 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:25.981 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:25.982 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:25.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.982 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:22:25.983 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:25.983 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:25.983 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:25.983 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:25.984 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:25.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.984 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:22:25.985 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:25.985 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:25.985 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:25.985 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:25.986 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:25.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.986 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:22:25.987 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:25.987 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:25.987 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:25.987 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:25.988 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:25.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:25.988 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:22:25.989 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:25.989 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:25.989 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:25.989 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:25.990 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:25.990 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.990 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:22:25.990 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:22:25.991 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:22:25.991 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:22:25.991 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:22:25.991 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:22:25.992 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:22:25.992 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:22:25.992 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:22:25.993 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:22:25.993 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:22:25.993 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:22:25.993 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:22:25.994 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:22:25.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:25.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:25.994 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:22:25.995 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:22:25.995 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:22:25.995 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:22:25.995 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:22:25.996 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:22:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 -- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f450c018930 msgr2=0x7f450c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f450c018930 0x7f450c01ae10 crc :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:25.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 -- 172.21.15.32:0/265841559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c1949c0 msgr2=0x7f452c195620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c1949c0 0x7f452c195620 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f4524009980 tx=0x7f452400bcd0).stop 2021-04-09T18:22:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 -- 172.21.15.32:0/265841559 shutdown_connections 2021-04-09T18:22:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f450c018930 0x7f450c01ae10 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:25.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c1949c0 0x7f452c195620 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f452c09d9b0 0x7f452c1943a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 --2- 172.21.15.32:0/265841559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f452c0aa2b0 0x7f452c09d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 -- 172.21.15.32:0/265841559 >> 172.21.15.32:0/265841559 conn(0x7f452c1a13a0 msgr2=0x7f452c094bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:26.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 -- 172.21.15.32:0/265841559 shutdown_connections 2021-04-09T18:22:26.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:25.692 7f4531d9c700 1 -- 172.21.15.32:0/265841559 wait complete. 2021-04-09T18:22:26.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:26.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.097 7f25d9915700 1 Processor -- start 2021-04-09T18:22:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.098 7f25d9915700 1 -- start start 2021-04-09T18:22:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.098 7f25d9915700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25d40d7ca0 0x7f25d40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.098 7f25d9915700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d410d480 con 0x7f25d410d010 2021-04-09T18:22:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.098 7f25d9915700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d40d86c0 con 0x7f25d410da00 2021-04-09T18:22:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.098 7f25d9915700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d40d9220 con 0x7f25d40d7ca0 2021-04-09T18:22:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.098 7f25d2d73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25d40d7ca0 0x7f25d40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.098 7f25d3574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f25d410d010 0x7f25d40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58778/0 (socket says 172.21.15.32:58778) 2021-04-09T18:22:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.098 7f25d3574700 1 -- 172.21.15.32:0/1634707661 learned_addr learned my addr 172.21.15.32:0/1634707661 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d2d73700 1 -- 172.21.15.32:0/1634707661 >> v1:172.21.15.86:6789/0 conn(0x7f25d410da00 legacy=0x7f25d410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d2d73700 1 -- 172.21.15.32:0/1634707661 >> v1:172.21.15.32:6789/0 conn(0x7f25d410d010 legacy=0x7f25d40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d2d73700 1 -- 172.21.15.32:0/1634707661 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d40d1f70 con 0x7f25d40d7ca0 2021-04-09T18:22:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d2572700 1 -- 172.21.15.32:0/1634707661 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524827697 0 0) 0x7f25c4000ce0 con 0x7f25d410d010 2021-04-09T18:22:26.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d2d73700 1 --2- 172.21.15.32:0/1634707661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d40d7ca0 0x7f25d40d80a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f25c8006650 tx=0x7f25c8009a00).ready entity=mon.0 client_cookie=d6659ee479b247f5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:26.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d2572700 1 -- 172.21.15.32:0/1634707661 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25c8009df0 con 0x7f25d40d7ca0 2021-04-09T18:22:26.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d2572700 1 -- 172.21.15.32:0/1634707661 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c80009b0 con 0x7f25d40d7ca0 2021-04-09T18:22:26.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d2572700 1 -- 172.21.15.32:0/1634707661 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25c800c2b0 con 0x7f25d40d7ca0 2021-04-09T18:22:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d9915700 1 -- 172.21.15.32:0/1634707661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d40d7ca0 msgr2=0x7f25d40d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.099 7f25d9915700 1 --2- 172.21.15.32:0/1634707661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d40d7ca0 0x7f25d40d80a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f25c8006650 tx=0x7f25c8009a00).stop 2021-04-09T18:22:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.100 7f25d9915700 1 -- 172.21.15.32:0/1634707661 shutdown_connections 2021-04-09T18:22:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.100 7f25d9915700 1 --2- 172.21.15.32:0/1634707661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d40d7ca0 0x7f25d40d80a0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.100 7f25d9915700 1 -- 172.21.15.32:0/1634707661 >> 172.21.15.32:0/1634707661 conn(0x7f25d40ce5f0 msgr2=0x7f25d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.100 7f25d9915700 1 -- 172.21.15.32:0/1634707661 shutdown_connections 2021-04-09T18:22:26.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.100 7f25d9915700 1 -- 172.21.15.32:0/1634707661 wait complete. 2021-04-09T18:22:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.120 7f25d9915700 1 Processor -- start 2021-04-09T18:22:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.120 7f25d9915700 1 -- start start 2021-04-09T18:22:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.120 7f25d9915700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d417e010 0x7f25d417e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.120 7f25d9915700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25d409f280 0x7f25d409f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.120 7f25d9915700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25d40b1b00 0x7f25d40b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.120 7f25d9915700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d409ed00 con 0x7f25d417e010 2021-04-09T18:22:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.120 7f25d9915700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d409fca0 con 0x7f25d409f280 2021-04-09T18:22:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d9915700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d40b2520 con 0x7f25d40b1b00 2021-04-09T18:22:26.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d417e010 0x7f25d417e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d417e010 0x7f25d417e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49194/0 (socket says 172.21.15.32:49194) 2021-04-09T18:22:26.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3574700 1 -- 172.21.15.32:0/1182220505 learned_addr learned my addr 172.21.15.32:0/1182220505 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:26.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3d75700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25d40b1b00 0x7f25d40b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d2d73700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25d409f280 0x7f25d409f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3574700 1 -- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25d40b1b00 msgr2=0x7f25d40b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3574700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25d40b1b00 0x7f25d40b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3574700 1 -- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25d409f280 msgr2=0x7f25d409f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3574700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25d409f280 0x7f25d409f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3574700 1 -- 172.21.15.32:0/1182220505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d40b2740 con 0x7f25d417e010 2021-04-09T18:22:26.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3d75700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25d40b1b00 0x7f25d40b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:26.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d2d73700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25d409f280 0x7f25d409f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:26.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d3574700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d417e010 0x7f25d417e410 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f25c4006f00 tx=0x7f25c400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:26.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25bbfff700 1 -- 172.21.15.32:0/1182220505 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25c400bd30 con 0x7f25d417e010 2021-04-09T18:22:26.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25bbfff700 1 -- 172.21.15.32:0/1182220505 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c40060d0 con 0x7f25d417e010 2021-04-09T18:22:26.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d9915700 1 -- 172.21.15.32:0/1182220505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d40b2740 con 0x7f25d417e010 2021-04-09T18:22:26.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25bbfff700 1 -- 172.21.15.32:0/1182220505 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25c40043a0 con 0x7f25d417e010 2021-04-09T18:22:26.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.121 7f25d9915700 1 -- 172.21.15.32:0/1182220505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d40b2740 con 0x7f25d417e010 2021-04-09T18:22:26.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.122 7f25bbfff700 1 -- 172.21.15.32:0/1182220505 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f25c4007000 con 0x7f25d417e010 2021-04-09T18:22:26.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.122 7f25bbfff700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25b40188d0 0x7f25b401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.122 7f25bbfff700 1 -- 172.21.15.32:0/1182220505 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f25c40329e0 con 0x7f25d417e010 2021-04-09T18:22:26.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.122 7f25d2d73700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25b40188d0 0x7f25b401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.123 7f25d2d73700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25b40188d0 0x7f25b401add0 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:26.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.142 7f25d9915700 1 -- 172.21.15.32:0/1182220505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d417cb20 con 0x7f25d417e010 2021-04-09T18:22:26.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.146 7f25bbfff700 1 -- 172.21.15.32:0/1182220505 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f25c4007ee0 con 0x7f25d417e010 2021-04-09T18:22:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.340 7f25d9915700 1 -- 172.21.15.32:0/1182220505 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25d410d480 con 0x7f25d417e010 2021-04-09T18:22:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.341 7f25bbfff700 1 -- 172.21.15.32:0/1182220505 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+2446 (secure 0 0 0) 0x7f25c4007000 con 0x7f25d417e010 2021-04-09T18:22:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 119 2021-04-09T18:22:26.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:26.396 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":119,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7809,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/633563485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":633563485},{"type":"v1","addr":"172.21.15.32:6834","nonce":633563485}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":118},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:23.428551","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7387},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7387":{"gid":7387,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6835/3998950443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3998950443},{"type":"v1","addr":"172.21.15.86:6835","nonce":3998950443}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 -- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25b40188d0 msgr2=0x7f25b401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25b40188d0 0x7f25b401add0 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 -- 172.21.15.32:0/1182220505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d417e010 msgr2=0x7f25d417e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d417e010 0x7f25d417e410 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f25c4006f00 tx=0x7f25c400b940).stop 2021-04-09T18:22:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 -- 172.21.15.32:0/1182220505 shutdown_connections 2021-04-09T18:22:26.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f25b40188d0 0x7f25b401add0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25d40b1b00 0x7f25d40b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25d409f280 0x7f25d409f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 --2- 172.21.15.32:0/1182220505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25d417e010 0x7f25d417e410 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.343 7f25d9915700 1 -- 172.21.15.32:0/1182220505 >> 172.21.15.32:0/1182220505 conn(0x7f25d41965a0 msgr2=0x7f25d419dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:26.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.344 7f25d9915700 1 -- 172.21.15.32:0/1182220505 shutdown_connections 2021-04-09T18:22:26.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.344 7f25d9915700 1 -- 172.21.15.32:0/1182220505 wait complete. 2021-04-09T18:22:26.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:22:26.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.506 7f2e0c157700 1 Processor -- start 2021-04-09T18:22:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.506 7f2e0c157700 1 -- start start 2021-04-09T18:22:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e0c157700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e040d6c90 0x7f2e040d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e0c157700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e040d2450 0x7f2e0410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e0c157700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e040d7290 con 0x7f2e040cffb0 2021-04-09T18:22:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e0c157700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e040d7090 con 0x7f2e040d6c90 2021-04-09T18:22:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e0c157700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e040d91f0 con 0x7f2e040d2450 2021-04-09T18:22:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e09772700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e040d6c90 0x7f2e040d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e09772700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e040d6c90 0x7f2e040d8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49206/0 (socket says 172.21.15.32:49206) 2021-04-09T18:22:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e09772700 1 -- 172.21.15.32:0/957583818 learned_addr learned my addr 172.21.15.32:0/957583818 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e0a774700 1 --2- 172.21.15.32:0/957583818 >> v2:172.21.15.86:3300/0 conn(0x7f2e040d2450 0x7f2e0410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e09772700 1 -- 172.21.15.32:0/957583818 >> v2:172.21.15.86:3300/0 conn(0x7f2e040d2450 msgr2=0x7f2e0410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:26.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e09772700 1 --2- 172.21.15.32:0/957583818 >> v2:172.21.15.86:3300/0 conn(0x7f2e040d2450 0x7f2e0410cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.507 7f2e09772700 1 -- 172.21.15.32:0/957583818 >> v1:172.21.15.86:6789/0 conn(0x7f2e040cffb0 legacy=0x7f2e040d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.508 7f2e09772700 1 -- 172.21.15.32:0/957583818 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e040d2850 con 0x7f2e040d6c90 2021-04-09T18:22:26.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.508 7f2e0a774700 1 --2- 172.21.15.32:0/957583818 >> v2:172.21.15.86:3300/0 conn(0x7f2e040d2450 0x7f2e0410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:26.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.508 7f2e09772700 1 --2- 172.21.15.32:0/957583818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e040d6c90 0x7f2e040d8000 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0x7f2df4006700 tx=0x7f2df4009b80).ready entity=mon.0 client_cookie=c01d0b7ae5485675 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:26.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.508 7f2e08f71700 1 -- 172.21.15.32:0/957583818 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df400c0c0 con 0x7f2e040d6c90 2021-04-09T18:22:26.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.508 7f2e08f71700 1 -- 172.21.15.32:0/957583818 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df400c440 con 0x7f2e040d6c90 2021-04-09T18:22:26.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.508 7f2e08f71700 1 -- 172.21.15.32:0/957583818 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df400c920 con 0x7f2e040d6c90 2021-04-09T18:22:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.508 7f2e0c157700 1 -- 172.21.15.32:0/957583818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e040d6c90 msgr2=0x7f2e040d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.508 7f2e0c157700 1 --2- 172.21.15.32:0/957583818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e040d6c90 0x7f2e040d8000 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0x7f2df4006700 tx=0x7f2df4009b80).stop 2021-04-09T18:22:26.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.509 7f2e0c157700 1 -- 172.21.15.32:0/957583818 shutdown_connections 2021-04-09T18:22:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.509 7f2e0c157700 1 --2- 172.21.15.32:0/957583818 >> v2:172.21.15.86:3300/0 conn(0x7f2e040d2450 0x7f2e0410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.509 7f2e0c157700 1 --2- 172.21.15.32:0/957583818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e040d6c90 0x7f2e040d8000 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.509 7f2e0c157700 1 -- 172.21.15.32:0/957583818 >> 172.21.15.32:0/957583818 conn(0x7f2e040ce5d0 msgr2=0x7f2e040cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:26.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.509 7f2e0c157700 1 -- 172.21.15.32:0/957583818 shutdown_connections 2021-04-09T18:22:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.509 7f2e0c157700 1 -- 172.21.15.32:0/957583818 wait complete. 2021-04-09T18:22:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.529 7f2e0c157700 1 Processor -- start 2021-04-09T18:22:26.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.529 7f2e0c157700 1 -- start start 2021-04-09T18:22:26.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.529 7f2e0c157700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0417d490 0x7f2e04181380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.529 7f2e0c157700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e040a7b50 0x7f2e040a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.529 7f2e0c157700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e041a1340 0x7f2e041a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.529 7f2e0c157700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e041a1e70 con 0x7f2e0417d490 2021-04-09T18:22:26.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.529 7f2e0c157700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e04180f20 con 0x7f2e040a7b50 2021-04-09T18:22:26.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.529 7f2e0c157700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e041819a0 con 0x7f2e041a1340 2021-04-09T18:22:26.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09f73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0417d490 0x7f2e04181380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09f73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0417d490 0x7f2e04181380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49208/0 (socket says 172.21.15.32:49208) 2021-04-09T18:22:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09f73700 1 -- 172.21.15.32:0/3902899962 learned_addr learned my addr 172.21.15.32:0/3902899962 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e0a774700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e041a1340 0x7f2e041a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09772700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e040a7b50 0x7f2e040a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09f73700 1 -- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e041a1340 msgr2=0x7f2e041a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09f73700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e041a1340 0x7f2e041a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09f73700 1 -- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e040a7b50 msgr2=0x7f2e040a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09f73700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e040a7b50 0x7f2e040a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09f73700 1 -- 172.21.15.32:0/3902899962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e041a2370 con 0x7f2e0417d490 2021-04-09T18:22:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e09f73700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0417d490 0x7f2e04181380 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f2e0000b8e0 tx=0x7f2e0000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2dfa7fc700 1 -- 172.21.15.32:0/3902899962 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e00000ce0 con 0x7f2e0417d490 2021-04-09T18:22:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2dfa7fc700 1 -- 172.21.15.32:0/3902899962 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0000c2f0 con 0x7f2e0417d490 2021-04-09T18:22:26.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e041a2370 con 0x7f2e0417d490 2021-04-09T18:22:26.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2dfa7fc700 1 -- 172.21.15.32:0/3902899962 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e0000c7d0 con 0x7f2e0417d490 2021-04-09T18:22:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.530 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e041a2370 con 0x7f2e0417d490 2021-04-09T18:22:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.531 7f2dfa7fc700 1 -- 172.21.15.32:0/3902899962 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2e0000c4d0 con 0x7f2e0417d490 2021-04-09T18:22:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.531 7f2dfa7fc700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dec018930 0x7f2dec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:26.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.531 7f2dfa7fc700 1 -- 172.21.15.32:0/3902899962 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2e00032de0 con 0x7f2e0417d490 2021-04-09T18:22:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.531 7f2e09772700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dec018930 0x7f2dec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.532 7f2e09772700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dec018930 0x7f2dec01ae10 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:26.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.551 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e041b7110 con 0x7f2e0417d490 2021-04-09T18:22:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.554 7f2dfa7fc700 1 -- 172.21.15.32:0/3902899962 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2e0000bdc0 con 0x7f2e0417d490 2021-04-09T18:22:26.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:26.735 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2e04180ee0 con 0x7f2e0417d490 2021-04-09T18:22:27.679 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:22:27.625 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:22:27.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.631 7f2dfa7fc700 1 -- 172.21.15.32:0/3902899962 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v120) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2e0000c2f0 con 0x7f2e0417d490 2021-04-09T18:22:27.704 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:22:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dec018930 msgr2=0x7f2dec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dec018930 0x7f2dec01ae10 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0417d490 msgr2=0x7f2e04181380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0417d490 0x7f2e04181380 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f2e0000b8e0 tx=0x7f2e0000bc80).stop 2021-04-09T18:22:27.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 shutdown_connections 2021-04-09T18:22:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dec018930 0x7f2dec01ae10 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e041a1340 0x7f2e041a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e040a7b50 0x7f2e040a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 --2- 172.21.15.32:0/3902899962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0417d490 0x7f2e04181380 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 >> 172.21.15.32:0/3902899962 conn(0x7f2e04198770 msgr2=0x7f2e041ac0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 shutdown_connections 2021-04-09T18:22:27.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.654 7f2e0c157700 1 -- 172.21.15.32:0/3902899962 wait complete. 2021-04-09T18:22:27.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:27.742 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:22:27.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe079df3700 1 Processor -- start 2021-04-09T18:22:27.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe079df3700 1 -- start start 2021-04-09T18:22:27.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe079df3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe07410d010 0x7fe0740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:27.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe079df3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0740d1da0 0x7fe0740d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:27.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe079df3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07410d480 con 0x7fe0740d82c0 2021-04-09T18:22:27.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe079df3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0740d8d30 con 0x7fe0740d1da0 2021-04-09T18:22:27.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe079df3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0740d9220 con 0x7fe07410d010 2021-04-09T18:22:27.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe072ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0740d1da0 0x7fe0740d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:27.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe072ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0740d1da0 0x7fe0740d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49220/0 (socket says 172.21.15.32:49220) 2021-04-09T18:22:27.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.824 7fe072ffd700 1 -- 172.21.15.32:0/4186810063 learned_addr learned my addr 172.21.15.32:0/4186810063 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:27.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe0737fe700 1 --2- 172.21.15.32:0/4186810063 >> v2:172.21.15.86:3300/0 conn(0x7fe07410d010 0x7fe0740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:27.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe072ffd700 1 -- 172.21.15.32:0/4186810063 >> v2:172.21.15.86:3300/0 conn(0x7fe07410d010 msgr2=0x7fe0740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:27.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe072ffd700 1 --2- 172.21.15.32:0/4186810063 >> v2:172.21.15.86:3300/0 conn(0x7fe07410d010 0x7fe0740d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe072ffd700 1 -- 172.21.15.32:0/4186810063 >> v1:172.21.15.86:6789/0 conn(0x7fe0740d82c0 legacy=0x7fe07410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:27.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe072ffd700 1 -- 172.21.15.32:0/4186810063 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0740d86f0 con 0x7fe0740d1da0 2021-04-09T18:22:27.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe072ffd700 1 --2- 172.21.15.32:0/4186810063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0740d1da0 0x7fe0740d7ca0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7fe068006700 tx=0x7fe068009b80).ready entity=mon.0 client_cookie=a213d7e101373d7b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:27.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe0727fc700 1 -- 172.21.15.32:0/4186810063 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe06800c0c0 con 0x7fe0740d1da0 2021-04-09T18:22:27.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe0727fc700 1 -- 172.21.15.32:0/4186810063 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06800c440 con 0x7fe0740d1da0 2021-04-09T18:22:27.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe0727fc700 1 -- 172.21.15.32:0/4186810063 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe06800c920 con 0x7fe0740d1da0 2021-04-09T18:22:27.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe079df3700 1 -- 172.21.15.32:0/4186810063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0740d1da0 msgr2=0x7fe0740d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:27.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.825 7fe079df3700 1 --2- 172.21.15.32:0/4186810063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0740d1da0 0x7fe0740d7ca0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7fe068006700 tx=0x7fe068009b80).stop 2021-04-09T18:22:27.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.826 7fe079df3700 1 -- 172.21.15.32:0/4186810063 shutdown_connections 2021-04-09T18:22:27.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.826 7fe079df3700 1 --2- 172.21.15.32:0/4186810063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0740d1da0 0x7fe0740d7ca0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.826 7fe079df3700 1 --2- 172.21.15.32:0/4186810063 >> v2:172.21.15.86:3300/0 conn(0x7fe07410d010 0x7fe0740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.826 7fe079df3700 1 -- 172.21.15.32:0/4186810063 >> 172.21.15.32:0/4186810063 conn(0x7fe0740ce5f0 msgr2=0x7fe0740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:27.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.826 7fe079df3700 1 -- 172.21.15.32:0/4186810063 shutdown_connections 2021-04-09T18:22:27.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.826 7fe079df3700 1 -- 172.21.15.32:0/4186810063 wait complete. 2021-04-09T18:22:27.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.846 7fe079df3700 1 Processor -- start 2021-04-09T18:22:27.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.846 7fe079df3700 1 -- start start 2021-04-09T18:22:27.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe079df3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07418c5c0 0x7fe0741a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe079df3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe074191880 0x7fe07418cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe079df3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe074191c80 0x7fe07418d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe079df3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07418c040 con 0x7fe074191c80 2021-04-09T18:22:27.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe079df3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0741aa5c0 con 0x7fe074191880 2021-04-09T18:22:27.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe079df3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07418bd20 con 0x7fe07418c5c0 2021-04-09T18:22:27.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe073fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe074191c80 0x7fe07418d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:27.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe073fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe074191c80 0x7fe07418d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49222/0 (socket says 172.21.15.32:49222) 2021-04-09T18:22:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe073fff700 1 -- 172.21.15.32:0/667574980 learned_addr learned my addr 172.21.15.32:0/667574980 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe072ffd700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe074191880 0x7fe07418cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe0737fe700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07418c5c0 0x7fe0741a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe073fff700 1 -- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07418c5c0 msgr2=0x7fe0741a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe073fff700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07418c5c0 0x7fe0741a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe073fff700 1 -- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe074191880 msgr2=0x7fe07418cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe073fff700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe074191880 0x7fe07418cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe073fff700 1 -- 172.21.15.32:0/667574980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07418da30 con 0x7fe074191c80 2021-04-09T18:22:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.847 7fe073fff700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe074191c80 0x7fe07418d410 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7fe06c006ed0 tx=0x7fe06c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.848 7fe05bfff700 1 -- 172.21.15.32:0/667574980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe06c00bd50 con 0x7fe074191c80 2021-04-09T18:22:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.848 7fe05bfff700 1 -- 172.21.15.32:0/667574980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06c006020 con 0x7fe074191c80 2021-04-09T18:22:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.848 7fe079df3700 1 -- 172.21.15.32:0/667574980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07418da30 con 0x7fe074191c80 2021-04-09T18:22:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.848 7fe05bfff700 1 -- 172.21.15.32:0/667574980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe06c0054b0 con 0x7fe074191c80 2021-04-09T18:22:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.848 7fe079df3700 1 -- 172.21.15.32:0/667574980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07418da30 con 0x7fe074191c80 2021-04-09T18:22:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.848 7fe05bfff700 1 -- 172.21.15.32:0/667574980 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe06c006200 con 0x7fe074191c80 2021-04-09T18:22:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.848 7fe05bfff700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe054018930 0x7fe05401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.848 7fe05bfff700 1 -- 172.21.15.32:0/667574980 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fe06c032a20 con 0x7fe074191c80 2021-04-09T18:22:27.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.849 7fe0737fe700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe054018930 0x7fe05401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:27.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.849 7fe0737fe700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe054018930 0x7fe05401adc0 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.868 7fe079df3700 1 -- 172.21.15.32:0/667574980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07418dfc0 con 0x7fe074191c80 2021-04-09T18:22:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:27.872 7fe05bfff700 1 -- 172.21.15.32:0/667574980 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe06c006230 con 0x7fe074191c80 2021-04-09T18:22:28.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.064 7fe079df3700 1 -- 172.21.15.32:0/667574980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe07418dfc0 con 0x7fe074191c80 2021-04-09T18:22:28.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.065 7fe05bfff700 1 -- 172.21.15.32:0/667574980 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+2438 (secure 0 0 0) 0x7fe06c00bd50 con 0x7fe074191c80 2021-04-09T18:22:28.119 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 120 2021-04-09T18:22:28.119 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:28.119 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":120,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7657,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3983752195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3983752195},{"type":"v1","addr":"172.21.15.86:6837","nonce":3983752195}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":120},{"gid":7809,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/633563485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":633563485},{"type":"v1","addr":"172.21.15.32:6834","nonce":633563485}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":118},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105}],"filesystems":[{"mdsmap":{"epoch":120,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:21:52.329633","modified":"2021-04-09 18:22:27.626898","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[1,2],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-04-09T18:22:28.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 -- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe054018930 msgr2=0x7fe05401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe054018930 0x7fe05401adc0 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 -- 172.21.15.32:0/667574980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe074191c80 msgr2=0x7fe07418d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe074191c80 0x7fe07418d410 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7fe06c006ed0 tx=0x7fe06c00b960).stop 2021-04-09T18:22:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 -- 172.21.15.32:0/667574980 shutdown_connections 2021-04-09T18:22:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe054018930 0x7fe05401adc0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe074191c80 0x7fe07418d410 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe074191880 0x7fe07418cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 --2- 172.21.15.32:0/667574980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07418c5c0 0x7fe0741a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 -- 172.21.15.32:0/667574980 >> 172.21.15.32:0/667574980 conn(0x7fe0741837d0 msgr2=0x7fe074195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:28.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 -- 172.21.15.32:0/667574980 shutdown_connections 2021-04-09T18:22:28.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.067 7fe079df3700 1 -- 172.21.15.32:0/667574980 wait complete. 2021-04-09T18:22:28.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:22:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.230 7f78d69cf700 1 Processor -- start 2021-04-09T18:22:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.230 7f78d69cf700 1 -- start start 2021-04-09T18:22:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78d69cf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00cffb0 0x7f78d00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78d69cf700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f78d00d6c90 0x7f78d010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78d69cf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d00d7290 con 0x7f78d00d7810 2021-04-09T18:22:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78d69cf700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d00d7090 con 0x7f78d00d6c90 2021-04-09T18:22:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78d69cf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00d8590 con 0x7f78d00cffb0 2021-04-09T18:22:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78c7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f78d00d7810 0x7f78d00d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58824/0 (socket says 172.21.15.32:58824) 2021-04-09T18:22:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78c7fff700 1 -- 172.21.15.32:0/2831882636 learned_addr learned my addr 172.21.15.32:0/2831882636 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78cffff700 1 --2- 172.21.15.32:0/2831882636 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00cffb0 0x7f78d00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78d4fec700 1 --2- 172.21.15.32:0/2831882636 >> v2:172.21.15.86:3300/0 conn(0x7f78d00d6c90 0x7f78d010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78cf7fe700 1 -- 172.21.15.32:0/2831882636 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247968609 0 0) 0x7f78b80009b0 con 0x7f78d00d7810 2021-04-09T18:22:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.231 7f78cf7fe700 1 -- 172.21.15.32:0/2831882636 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78bc002fe0 con 0x7f78d00d7810 2021-04-09T18:22:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78cffff700 1 -- 172.21.15.32:0/2831882636 >> v2:172.21.15.86:3300/0 conn(0x7f78d00d6c90 msgr2=0x7f78d010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78cffff700 1 --2- 172.21.15.32:0/2831882636 >> v2:172.21.15.86:3300/0 conn(0x7f78d00d6c90 0x7f78d010cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78cffff700 1 -- 172.21.15.32:0/2831882636 >> v1:172.21.15.32:6789/0 conn(0x7f78d00d7810 legacy=0x7f78d00d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78cffff700 1 -- 172.21.15.32:0/2831882636 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d00d91f0 con 0x7f78d00cffb0 2021-04-09T18:22:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78cffff700 1 --2- 172.21.15.32:0/2831882636 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00cffb0 0x7f78d00d2450 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f78c0006650 tx=0x7f78c0009a80).ready entity=mon.2 client_cookie=3fb05c93285b01c4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78cf7fe700 1 -- 172.21.15.32:0/2831882636 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c00009b0 con 0x7f78d00cffb0 2021-04-09T18:22:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78cf7fe700 1 -- 172.21.15.32:0/2831882636 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c000c0c0 con 0x7f78d00cffb0 2021-04-09T18:22:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78cf7fe700 1 -- 172.21.15.32:0/2831882636 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c000c510 con 0x7f78d00cffb0 2021-04-09T18:22:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78d69cf700 1 -- 172.21.15.32:0/2831882636 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00cffb0 msgr2=0x7f78d00d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.232 7f78d69cf700 1 --2- 172.21.15.32:0/2831882636 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00cffb0 0x7f78d00d2450 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f78c0006650 tx=0x7f78c0009a80).stop 2021-04-09T18:22:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.233 7f78d69cf700 1 -- 172.21.15.32:0/2831882636 shutdown_connections 2021-04-09T18:22:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.233 7f78d69cf700 1 --2- 172.21.15.32:0/2831882636 >> v2:172.21.15.86:3300/0 conn(0x7f78d00d6c90 0x7f78d010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.233 7f78d69cf700 1 --2- 172.21.15.32:0/2831882636 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00cffb0 0x7f78d00d2450 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.233 7f78d69cf700 1 -- 172.21.15.32:0/2831882636 >> 172.21.15.32:0/2831882636 conn(0x7f78d00ce5d0 msgr2=0x7f78d00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:28.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.233 7f78d69cf700 1 -- 172.21.15.32:0/2831882636 shutdown_connections 2021-04-09T18:22:28.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.233 7f78d69cf700 1 -- 172.21.15.32:0/2831882636 wait complete. 2021-04-09T18:22:28.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.253 7f78d69cf700 1 Processor -- start 2021-04-09T18:22:28.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.253 7f78d69cf700 1 -- start start 2021-04-09T18:22:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.253 7f78d69cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d008ac00 0x7f78d00b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.253 7f78d69cf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d008b000 0x7f78d01ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.253 7f78d69cf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00a0df0 0x7f78d01abc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.253 7f78d69cf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d01ab130 con 0x7f78d008ac00 2021-04-09T18:22:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.253 7f78d69cf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00b43e0 con 0x7f78d008b000 2021-04-09T18:22:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.253 7f78d69cf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00b4600 con 0x7f78d00a0df0 2021-04-09T18:22:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d008ac00 0x7f78d00b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d008ac00 0x7f78d00b3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49238/0 (socket says 172.21.15.32:49238) 2021-04-09T18:22:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cffff700 1 -- 172.21.15.32:0/1992523232 learned_addr learned my addr 172.21.15.32:0/1992523232 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78d4fec700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00a0df0 0x7f78d01abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78c7fff700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d008b000 0x7f78d01ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cffff700 1 -- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00a0df0 msgr2=0x7f78d01abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cffff700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00a0df0 0x7f78d01abc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cffff700 1 -- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d008b000 msgr2=0x7f78d01ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cffff700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d008b000 0x7f78d01ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cffff700 1 -- 172.21.15.32:0/1992523232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d01a9e60 con 0x7f78d008ac00 2021-04-09T18:22:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78d4fec700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00a0df0 0x7f78d01abc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cffff700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d008ac00 0x7f78d00b3d80 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7f78c0009630 tx=0x7f78c000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cd7fa700 1 -- 172.21.15.32:0/1992523232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c00093a0 con 0x7f78d008ac00 2021-04-09T18:22:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cd7fa700 1 -- 172.21.15.32:0/1992523232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c000bd70 con 0x7f78d008ac00 2021-04-09T18:22:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d01a9e60 con 0x7f78d008ac00 2021-04-09T18:22:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78cd7fa700 1 -- 172.21.15.32:0/1992523232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c0005560 con 0x7f78d008ac00 2021-04-09T18:22:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.254 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d01a9e60 con 0x7f78d008ac00 2021-04-09T18:22:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.255 7f78cd7fa700 1 -- 172.21.15.32:0/1992523232 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f78c0000c90 con 0x7f78d008ac00 2021-04-09T18:22:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.255 7f78cd7fa700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b40188c0 0x7f78b401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.255 7f78cd7fa700 1 -- 172.21.15.32:0/1992523232 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f78c0032c40 con 0x7f78d008ac00 2021-04-09T18:22:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.255 7f78c7fff700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b40188c0 0x7f78b401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.256 7f78c7fff700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b40188c0 0x7f78b401adc0 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.275 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d0182e10 con 0x7f78d008ac00 2021-04-09T18:22:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.278 7f78cd7fa700 1 -- 172.21.15.32:0/1992523232 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f78c000a000 con 0x7f78d008ac00 2021-04-09T18:22:28.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.472 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f78d0182e10 con 0x7f78d008ac00 2021-04-09T18:22:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.633 7f78cd7fa700 1 -- 172.21.15.32:0/1992523232 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v121) v1 ==== 104+0+0 (secure 0 0 0) 0x7f78c0000c90 con 0x7f78d008ac00 2021-04-09T18:22:28.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b40188c0 msgr2=0x7f78b401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b40188c0 0x7f78b401adc0 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d008ac00 msgr2=0x7f78d00b3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d008ac00 0x7f78d00b3d80 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7f78c0009630 tx=0x7f78c000b980).stop 2021-04-09T18:22:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 shutdown_connections 2021-04-09T18:22:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b40188c0 0x7f78b401adc0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00a0df0 0x7f78d01abc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d008b000 0x7f78d01ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 --2- 172.21.15.32:0/1992523232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d008ac00 0x7f78d00b3d80 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 >> 172.21.15.32:0/1992523232 conn(0x7f78d0183890 msgr2=0x7f78d019b900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:28.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 shutdown_connections 2021-04-09T18:22:28.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.638 7f78d69cf700 1 -- 172.21.15.32:0/1992523232 wait complete. 2021-04-09T18:22:28.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:22:28.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.809 7fc33136c700 1 Processor -- start 2021-04-09T18:22:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.809 7fc33136c700 1 -- start start 2021-04-09T18:22:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc33136c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc32c0d7cc0 0x7fc32c0d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc33136c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c10da80 0x7fc32c10de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc33136c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32c10d500 con 0x7fc32c10d050 2021-04-09T18:22:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc33136c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc32c0d9260 con 0x7fc32c0d7cc0 2021-04-09T18:22:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc33136c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c112140 con 0x7fc32c10da80 2021-04-09T18:22:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32b87e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c10da80 0x7fc32c10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32a87c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc32c0d7cc0 0x7fc32c0d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32b07d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc32c10d050 0x7fc32c0cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:56760/0 (socket says 172.21.15.32:56760) 2021-04-09T18:22:28.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32b07d700 1 -- 172.21.15.32:0/2837954080 learned_addr learned my addr 172.21.15.32:0/2837954080 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:28.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32a87c700 1 -- 172.21.15.32:0/2837954080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c10da80 msgr2=0x7fc32c10de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:28.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32a87c700 1 --2- 172.21.15.32:0/2837954080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c10da80 0x7fc32c10de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32a87c700 1 -- 172.21.15.32:0/2837954080 >> v1:172.21.15.86:6789/0 conn(0x7fc32c10d050 legacy=0x7fc32c0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32a87c700 1 -- 172.21.15.32:0/2837954080 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32c0d0330 con 0x7fc32c0d7cc0 2021-04-09T18:22:28.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32b87e700 1 --2- 172.21.15.32:0/2837954080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c10da80 0x7fc32c10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:28.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.810 7fc32a07b700 1 -- 172.21.15.32:0/2837954080 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170626890 0 0) 0x7fc31c0010c0 con 0x7fc32c10d050 2021-04-09T18:22:28.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc32a87c700 1 --2- 172.21.15.32:0/2837954080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c0d7cc0 0x7fc32c0d80c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fc320006650 tx=0x7fc320009a80).ready entity=mon.1 client_cookie=b32572de729dd4c4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:28.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc32a07b700 1 -- 172.21.15.32:0/2837954080 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3200009b0 con 0x7fc32c0d7cc0 2021-04-09T18:22:28.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc32a07b700 1 -- 172.21.15.32:0/2837954080 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32000c0c0 con 0x7fc32c0d7cc0 2021-04-09T18:22:28.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc32a07b700 1 -- 172.21.15.32:0/2837954080 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc32000c510 con 0x7fc32c0d7cc0 2021-04-09T18:22:28.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc33136c700 1 -- 172.21.15.32:0/2837954080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c0d7cc0 msgr2=0x7fc32c0d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc33136c700 1 --2- 172.21.15.32:0/2837954080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c0d7cc0 0x7fc32c0d80c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fc320006650 tx=0x7fc320009a80).stop 2021-04-09T18:22:28.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc33136c700 1 -- 172.21.15.32:0/2837954080 shutdown_connections 2021-04-09T18:22:28.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc33136c700 1 --2- 172.21.15.32:0/2837954080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c10da80 0x7fc32c10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc33136c700 1 --2- 172.21.15.32:0/2837954080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c0d7cc0 0x7fc32c0d80c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.811 7fc33136c700 1 -- 172.21.15.32:0/2837954080 >> 172.21.15.32:0/2837954080 conn(0x7fc32c0ce620 msgr2=0x7fc32c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:28.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.812 7fc33136c700 1 -- 172.21.15.32:0/2837954080 shutdown_connections 2021-04-09T18:22:28.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.812 7fc33136c700 1 -- 172.21.15.32:0/2837954080 wait complete. 2021-04-09T18:22:28.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.831 7fc33136c700 1 Processor -- start 2021-04-09T18:22:28.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc33136c700 1 -- start start 2021-04-09T18:22:28.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc33136c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c09b2f0 0x7fc32c09b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc33136c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32c09a8b0 0x7fc32c09e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc33136c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c09e9c0 0x7fc32c09edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc33136c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c0bde80 con 0x7fc32c09a8b0 2021-04-09T18:22:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc33136c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c09ad70 con 0x7fc32c09e9c0 2021-04-09T18:22:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc33136c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c09bd10 con 0x7fc32c09b2f0 2021-04-09T18:22:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc32a87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32c09a8b0 0x7fc32c09e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc32a87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32c09a8b0 0x7fc32c09e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49250/0 (socket says 172.21.15.32:49250) 2021-04-09T18:22:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.832 7fc32a87c700 1 -- 172.21.15.32:0/2084536209 learned_addr learned my addr 172.21.15.32:0/2084536209 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc32b07d700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c09b2f0 0x7fc32c09b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc32b87e700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c09e9c0 0x7fc32c09edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc32a87c700 1 -- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c09b2f0 msgr2=0x7fc32c09b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc32a87c700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c09b2f0 0x7fc32c09b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc32a87c700 1 -- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c09e9c0 msgr2=0x7fc32c09edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:28.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc32a87c700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c09e9c0 0x7fc32c09edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:28.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc32a87c700 1 -- 172.21.15.32:0/2084536209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32c09fbd0 con 0x7fc32c09a8b0 2021-04-09T18:22:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc32b87e700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c09e9c0 0x7fc32c09edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc32a87c700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32c09a8b0 0x7fc32c09e3a0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7fc32000b470 tx=0x7fc32000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc313fff700 1 -- 172.21.15.32:0/2084536209 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3200093a0 con 0x7fc32c09a8b0 2021-04-09T18:22:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc313fff700 1 -- 172.21.15.32:0/2084536209 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32000bcf0 con 0x7fc32c09a8b0 2021-04-09T18:22:28.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc33136c700 1 -- 172.21.15.32:0/2084536209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32c09fbd0 con 0x7fc32c09a8b0 2021-04-09T18:22:28.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc313fff700 1 -- 172.21.15.32:0/2084536209 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3200054d0 con 0x7fc32c09a8b0 2021-04-09T18:22:28.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.833 7fc33136c700 1 -- 172.21.15.32:0/2084536209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32c09fbd0 con 0x7fc32c09a8b0 2021-04-09T18:22:28.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.834 7fc313fff700 1 -- 172.21.15.32:0/2084536209 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc32000a000 con 0x7fc32c09a8b0 2021-04-09T18:22:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.834 7fc313fff700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc30c018930 0x7fc30c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.834 7fc313fff700 1 -- 172.21.15.32:0/2084536209 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fc320034c70 con 0x7fc32c09a8b0 2021-04-09T18:22:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.834 7fc32b07d700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc30c018930 0x7fc30c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.834 7fc32b07d700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc30c018930 0x7fc30c01adc0 crc :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:28.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.854 7fc33136c700 1 -- 172.21.15.32:0/2084536209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32c0b9040 con 0x7fc32c09a8b0 2021-04-09T18:22:28.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:28.857 7fc313fff700 1 -- 172.21.15.32:0/2084536209 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc32000a000 con 0x7fc32c09a8b0 2021-04-09T18:22:29.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.042 7fc33136c700 1 -- 172.21.15.32:0/2084536209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc32c10d500 con 0x7fc32c09a8b0 2021-04-09T18:22:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.636 7fc313fff700 1 -- 172.21.15.32:0/2084536209 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v97) v1 ==== 185+0+0 (secure 0 0 0) 0x7fc32000a350 con 0x7fc32c09a8b0 2021-04-09T18:22:29.711 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:22:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 -- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc30c018930 msgr2=0x7fc30c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc30c018930 0x7fc30c01adc0 crc :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 -- 172.21.15.32:0/2084536209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32c09a8b0 msgr2=0x7fc32c09e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32c09a8b0 0x7fc32c09e3a0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7fc32000b470 tx=0x7fc32000b900).stop 2021-04-09T18:22:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 -- 172.21.15.32:0/2084536209 shutdown_connections 2021-04-09T18:22:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc30c018930 0x7fc30c01adc0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32c09a8b0 0x7fc32c09e3a0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32c09e9c0 0x7fc32c09edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 --2- 172.21.15.32:0/2084536209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc32c09b2f0 0x7fc32c09b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.659 7fc33136c700 1 -- 172.21.15.32:0/2084536209 >> 172.21.15.32:0/2084536209 conn(0x7fc32c0ae6f0 msgr2=0x7fc32c097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.660 7fc33136c700 1 -- 172.21.15.32:0/2084536209 shutdown_connections 2021-04-09T18:22:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.660 7fc33136c700 1 -- 172.21.15.32:0/2084536209 wait complete. 2021-04-09T18:22:29.724 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:22:29.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.831 7f7cc8fc9700 1 Processor -- start 2021-04-09T18:22:29.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.831 7f7cc8fc9700 1 -- start start 2021-04-09T18:22:29.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc8fc9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7cc410d050 0x7f7cc40cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:29.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc8fc9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cc40d0160 0x7f7cc40d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:29.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc8fc9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc410d500 con 0x7f7cc40d82e0 2021-04-09T18:22:29.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc8fc9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc40d9260 con 0x7f7cc410d050 2021-04-09T18:22:29.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc8fc9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40d86e0 con 0x7f7cc40d0160 2021-04-09T18:22:29.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc359e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7cc40d82e0 0x7f7cc410da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58850/0 (socket says 172.21.15.32:58850) 2021-04-09T18:22:29.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc359e700 1 -- 172.21.15.32:0/1919993016 learned_addr learned my addr 172.21.15.32:0/1919993016 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:29.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc259c700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cc40d0160 0x7f7cc40d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:29.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc2d9d700 1 --2- 172.21.15.32:0/1919993016 >> v2:172.21.15.86:3300/0 conn(0x7f7cc410d050 0x7f7cc40cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:29.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634496230 0 0) 0x7f7cbc000d30 con 0x7f7cc40d82e0 2021-04-09T18:22:29.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cac002fe0 con 0x7f7cc40d82e0 2021-04-09T18:22:29.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.832 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4225445613 0 0) 0x7f7cbc001270 con 0x7f7cc40d82e0 2021-04-09T18:22:29.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cc40d0160 msgr2=0x7f7cc40d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:29.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cc40d0160 0x7f7cc40d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 >> v2:172.21.15.86:3300/0 conn(0x7f7cc410d050 msgr2=0x7f7cc40cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:29.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 --2- 172.21.15.32:0/1919993016 >> v2:172.21.15.86:3300/0 conn(0x7f7cc410d050 0x7f7cc40cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc410e270 con 0x7f7cc40d82e0 2021-04-09T18:22:29.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f7cbc001980 con 0x7f7cc40d82e0 2021-04-09T18:22:29.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 >> v1:172.21.15.32:6789/0 conn(0x7f7cc40d82e0 legacy=0x7f7cc410da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:29.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cac003a60 0x7f7cac005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:29.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cac04a780 0x7f7cac04cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:29.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cac04d510 0x7f7cac04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:29.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac0502e0 con 0x7f7cac04d510 2021-04-09T18:22:29.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac050500 con 0x7f7cac003a60 2021-04-09T18:22:29.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac050720 con 0x7f7cac04a780 2021-04-09T18:22:29.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc359e700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cac04d510 0x7f7cac04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:29.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc259c700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cac04a780 0x7f7cac04cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:29.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc2d9d700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cac003a60 0x7f7cac005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:29.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc359e700 1 -- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cac04a780 msgr2=0x7f7cac04cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:29.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc359e700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cac04a780 0x7f7cac04cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc359e700 1 -- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cac003a60 msgr2=0x7f7cac005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:29.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc359e700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cac003a60 0x7f7cac005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc359e700 1 -- 172.21.15.32:0/1919993016 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0509c0 con 0x7f7cac04d510 2021-04-09T18:22:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc359e700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cac04d510 0x7f7cac04fa30 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7f7cbc007a10 tx=0x7f7cbc00a310).ready entity=mon.0 client_cookie=ae43d4af1576581 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cbc0070c0 con 0x7f7cac04d510 2021-04-09T18:22:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cbc001de0 con 0x7f7cac04d510 2021-04-09T18:22:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc1d9b700 1 -- 172.21.15.32:0/1919993016 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cbc004720 con 0x7f7cac04d510 2021-04-09T18:22:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc8fc9700 1 -- 172.21.15.32:0/1919993016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cac04d510 msgr2=0x7f7cac04fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc8fc9700 1 --2- 172.21.15.32:0/1919993016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cac04d510 0x7f7cac04fa30 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7f7cbc007a10 tx=0x7f7cbc00a310).stop 2021-04-09T18:22:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.833 7f7cc359e700 1 -- 172.21.15.32:0/1919993016 reap_dead start 2021-04-09T18:22:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.834 7f7cc8fc9700 1 -- 172.21.15.32:0/1919993016 shutdown_connections 2021-04-09T18:22:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.834 7f7cc8fc9700 1 -- 172.21.15.32:0/1919993016 >> 172.21.15.32:0/1919993016 conn(0x7f7cc40ce620 msgr2=0x7f7cc40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.834 7f7cc8fc9700 1 -- 172.21.15.32:0/1919993016 shutdown_connections 2021-04-09T18:22:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.834 7f7cc8fc9700 1 -- 172.21.15.32:0/1919993016 wait complete. 2021-04-09T18:22:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.854 7f7cc8fc9700 1 Processor -- start 2021-04-09T18:22:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc8fc9700 1 -- start start 2021-04-09T18:22:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc8fc9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cc40ad5b0 0x7f7cc41926b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc8fc9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cc40990f0 0x7f7cc40994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:29.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc8fc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc41a30b0 0x7f7cc41a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:29.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc8fc9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc4098b70 con 0x7f7cc41a30b0 2021-04-09T18:22:29.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc8fc9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40988a0 con 0x7f7cc40990f0 2021-04-09T18:22:29.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc8fc9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc4099b10 con 0x7f7cc40ad5b0 2021-04-09T18:22:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc41a30b0 0x7f7cc41a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc41a30b0 0x7f7cc41a34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49274/0 (socket says 172.21.15.32:49274) 2021-04-09T18:22:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc359e700 1 -- 172.21.15.32:0/1907865450 learned_addr learned my addr 172.21.15.32:0/1907865450 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc2d9d700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cc40ad5b0 0x7f7cc41926b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.855 7f7cc259c700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cc40990f0 0x7f7cc40994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cc359e700 1 -- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cc40ad5b0 msgr2=0x7f7cc41926b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cc359e700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cc40ad5b0 0x7f7cc41926b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cc359e700 1 -- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cc40990f0 msgr2=0x7f7cc40994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cc359e700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cc40990f0 0x7f7cc40994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cc359e700 1 -- 172.21.15.32:0/1907865450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc41a3ad0 con 0x7f7cc41a30b0 2021-04-09T18:22:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cc359e700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc41a30b0 0x7f7cc41a34b0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7f7cbc001c40 tx=0x7f7cbc00acb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cb37fe700 1 -- 172.21.15.32:0/1907865450 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cbc0014d0 con 0x7f7cc41a30b0 2021-04-09T18:22:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cb37fe700 1 -- 172.21.15.32:0/1907865450 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cbc004930 con 0x7f7cc41a30b0 2021-04-09T18:22:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc41a3ad0 con 0x7f7cc41a30b0 2021-04-09T18:22:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cb37fe700 1 -- 172.21.15.32:0/1907865450 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cbc004dc0 con 0x7f7cc41a30b0 2021-04-09T18:22:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc41a3ad0 con 0x7f7cc41a30b0 2021-04-09T18:22:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.856 7f7cb37fe700 1 -- 172.21.15.32:0/1907865450 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7cbc00adf0 con 0x7f7cc41a30b0 2021-04-09T18:22:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.857 7f7cb37fe700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ca8018900 0x7f7ca801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.857 7f7cb37fe700 1 -- 172.21.15.32:0/1907865450 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5225+0+0 (secure 0 0 0) 0x7f7cbc032be0 con 0x7f7cc41a30b0 2021-04-09T18:22:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.857 7f7cc2d9d700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ca8018900 0x7f7ca801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.857 7f7cc2d9d700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ca8018900 0x7f7ca801adc0 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:29.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.876 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc40b9040 con 0x7f7cc41a30b0 2021-04-09T18:22:29.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:29.879 7f7cb37fe700 1 -- 172.21.15.32:0/1907865450 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7cbc032df0 con 0x7f7cc41a30b0 2021-04-09T18:22:30.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.065 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7cc410d500 con 0x7f7cc41a30b0 2021-04-09T18:22:30.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.642 7f7cb37fe700 1 -- 172.21.15.32:0/1907865450 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v98) v1 ==== 173+0+0 (secure 0 0 0) 0x7f7cbc004720 con 0x7f7cc41a30b0 2021-04-09T18:22:30.733 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:22:30.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ca8018900 msgr2=0x7f7ca801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ca8018900 0x7f7ca801adc0 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc41a30b0 msgr2=0x7f7cc41a34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc41a30b0 0x7f7cc41a34b0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7f7cbc001c40 tx=0x7f7cbc00acb0).stop 2021-04-09T18:22:30.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 shutdown_connections 2021-04-09T18:22:30.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ca8018900 0x7f7ca801adc0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7cc41a30b0 0x7f7cc41a34b0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cc40990f0 0x7f7cc40994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 --2- 172.21.15.32:0/1907865450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cc40ad5b0 0x7f7cc41926b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 >> 172.21.15.32:0/1907865450 conn(0x7f7cc40ae6f0 msgr2=0x7f7cc4097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:30.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 shutdown_connections 2021-04-09T18:22:30.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.682 7f7cc8fc9700 1 -- 172.21.15.32:0/1907865450 wait complete. 2021-04-09T18:22:30.746 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:22:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.844 7f212c7f3700 1 Processor -- start 2021-04-09T18:22:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.844 7f212c7f3700 1 -- start start 2021-04-09T18:22:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.845 7f212c7f3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21240cffb0 0x7f21240d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.845 7f212c7f3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21240d6c90 0x7f212410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.845 7f212c7f3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21240d7290 con 0x7f21240d7810 2021-04-09T18:22:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.845 7f212c7f3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21240d7090 con 0x7f21240d6c90 2021-04-09T18:22:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.845 7f212c7f3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21240d8590 con 0x7f21240cffb0 2021-04-09T18:22:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.845 7f212ae10700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21240d6c90 0x7f212410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.845 7f2129e0e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f21240d7810 0x7f21240d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58872/0 (socket says 172.21.15.32:58872) 2021-04-09T18:22:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.845 7f212ae10700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f21240d6c90 0x7f212410cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49280/0 (socket says 172.21.15.32:49280) 2021-04-09T18:22:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.845 7f2129e0e700 1 -- 172.21.15.32:0/837085875 learned_addr learned my addr 172.21.15.32:0/837085875 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212a60f700 1 --2- 172.21.15.32:0/837085875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21240cffb0 0x7f21240d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212ae10700 1 -- 172.21.15.32:0/837085875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21240cffb0 msgr2=0x7f21240d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212ae10700 1 --2- 172.21.15.32:0/837085875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21240cffb0 0x7f21240d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212ae10700 1 -- 172.21.15.32:0/837085875 >> v1:172.21.15.32:6789/0 conn(0x7f21240d7810 legacy=0x7f21240d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212960d700 1 -- 172.21.15.32:0/837085875 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640607059 0 0) 0x7f211c000d30 con 0x7f21240d7810 2021-04-09T18:22:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212ae10700 1 -- 172.21.15.32:0/837085875 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21240d91f0 con 0x7f21240d6c90 2021-04-09T18:22:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212ae10700 1 --2- 172.21.15.32:0/837085875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21240d6c90 0x7f212410cfe0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7f2114006ab0 tx=0x7f2114009ac0).ready entity=mon.0 client_cookie=d3a15a873dfaeb3d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212960d700 1 -- 172.21.15.32:0/837085875 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21140066d0 con 0x7f21240d6c90 2021-04-09T18:22:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212960d700 1 -- 172.21.15.32:0/837085875 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21140022e0 con 0x7f21240d6c90 2021-04-09T18:22:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212960d700 1 -- 172.21.15.32:0/837085875 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21140027c0 con 0x7f21240d6c90 2021-04-09T18:22:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212c7f3700 1 -- 172.21.15.32:0/837085875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21240d6c90 msgr2=0x7f212410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.846 7f212c7f3700 1 --2- 172.21.15.32:0/837085875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21240d6c90 0x7f212410cfe0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7f2114006ab0 tx=0x7f2114009ac0).stop 2021-04-09T18:22:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.847 7f212c7f3700 1 -- 172.21.15.32:0/837085875 shutdown_connections 2021-04-09T18:22:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.847 7f212c7f3700 1 --2- 172.21.15.32:0/837085875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21240d6c90 0x7f212410cfe0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.847 7f212c7f3700 1 --2- 172.21.15.32:0/837085875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21240cffb0 0x7f21240d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.847 7f212c7f3700 1 -- 172.21.15.32:0/837085875 >> 172.21.15.32:0/837085875 conn(0x7f21240ce5d0 msgr2=0x7f21240cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:30.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.847 7f212c7f3700 1 -- 172.21.15.32:0/837085875 shutdown_connections 2021-04-09T18:22:30.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.847 7f212c7f3700 1 -- 172.21.15.32:0/837085875 wait complete. 2021-04-09T18:22:30.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.867 7f212c7f3700 1 Processor -- start 2021-04-09T18:22:30.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.867 7f212c7f3700 1 -- start start 2021-04-09T18:22:30.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212c7f3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2124188cf0 0x7f21241a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:30.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212c7f3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21241a73d0 0x7f21241a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212c7f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21241890f0 0x7f21241a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212c7f3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21240b10d0 con 0x7f21241890f0 2021-04-09T18:22:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212c7f3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21241a7890 con 0x7f2124188cf0 2021-04-09T18:22:30.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212c7f3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21241a3fd0 con 0x7f21241a73d0 2021-04-09T18:22:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212ae10700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21241890f0 0x7f21241a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212ae10700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21241890f0 0x7f21241a8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49288/0 (socket says 172.21.15.32:49288) 2021-04-09T18:22:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212ae10700 1 -- 172.21.15.32:0/4088583350 learned_addr learned my addr 172.21.15.32:0/4088583350 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212a60f700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2124188cf0 0x7f21241a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f2129e0e700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21241a73d0 0x7f21241a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212ae10700 1 -- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21241a73d0 msgr2=0x7f21241a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212ae10700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21241a73d0 0x7f21241a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212ae10700 1 -- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2124188cf0 msgr2=0x7f21241a6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212ae10700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2124188cf0 0x7f21241a6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212ae10700 1 -- 172.21.15.32:0/4088583350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21241a41b0 con 0x7f21241890f0 2021-04-09T18:22:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f2129e0e700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21241a73d0 0x7f21241a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f212ae10700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21241890f0 0x7f21241a8430 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7f2114000e10 tx=0x7f2114002880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.868 7f2122ffd700 1 -- 172.21.15.32:0/4088583350 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f211400c2c0 con 0x7f21241890f0 2021-04-09T18:22:30.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.869 7f2122ffd700 1 -- 172.21.15.32:0/4088583350 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211400c540 con 0x7f21241890f0 2021-04-09T18:22:30.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.869 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21241a41b0 con 0x7f21241890f0 2021-04-09T18:22:30.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.869 7f2122ffd700 1 -- 172.21.15.32:0/4088583350 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f211400c910 con 0x7f21241890f0 2021-04-09T18:22:30.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.869 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21241a41b0 con 0x7f21241890f0 2021-04-09T18:22:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.869 7f2122ffd700 1 -- 172.21.15.32:0/4088583350 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2114008030 con 0x7f21241890f0 2021-04-09T18:22:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.869 7f2122ffd700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2104018930 0x7f210401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.869 7f2122ffd700 1 -- 172.21.15.32:0/4088583350 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f2114032790 con 0x7f21241890f0 2021-04-09T18:22:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.870 7f212a60f700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2104018930 0x7f210401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.870 7f212a60f700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2104018930 0x7f210401ae10 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:30.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.889 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2124182e20 con 0x7f21241890f0 2021-04-09T18:22:30.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:30.893 7f2122ffd700 1 -- 172.21.15.32:0/4088583350 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2114002c70 con 0x7f21241890f0 2021-04-09T18:22:31.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.105 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2124182e20 con 0x7f21241890f0 2021-04-09T18:22:31.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.108 7f2122ffd700 1 -- 172.21.15.32:0/4088583350 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2114008030 con 0x7f21241890f0 2021-04-09T18:22:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2104018930 msgr2=0x7f210401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2104018930 0x7f210401ae10 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21241890f0 msgr2=0x7f21241a8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21241890f0 0x7f21241a8430 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7f2114000e10 tx=0x7f2114002880).stop 2021-04-09T18:22:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 shutdown_connections 2021-04-09T18:22:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2104018930 0x7f210401ae10 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f21241890f0 0x7f21241a8430 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21241a73d0 0x7f21241a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 --2- 172.21.15.32:0/4088583350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2124188cf0 0x7f21241a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 >> 172.21.15.32:0/4088583350 conn(0x7f21241a36a0 msgr2=0x7f2124185120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 shutdown_connections 2021-04-09T18:22:31.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.115 7f212c7f3700 1 -- 172.21.15.32:0/4088583350 wait complete. 2021-04-09T18:22:31.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_shrink' 2021-04-09T18:22:31.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.273 7fdd0e978700 1 Processor -- start 2021-04-09T18:22:31.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.273 7fdd0e978700 1 -- start start 2021-04-09T18:22:31.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdd0e978700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdd0810d010 0x7fdd080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdd0e978700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdd080d1da0 0x7fdd080d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdd0e978700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0810d480 con 0x7fdd080d82c0 2021-04-09T18:22:31.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdd0e978700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd080d8d30 con 0x7fdd080d1da0 2021-04-09T18:22:31.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdd0e978700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd080d9220 con 0x7fdd0810d010 2021-04-09T18:22:31.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdcfffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdd080d1da0 0x7fdd080d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdd0cf95700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdd080d82c0 0x7fdd0810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58884/0 (socket says 172.21.15.32:58884) 2021-04-09T18:22:31.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdd0cf95700 1 -- 172.21.15.32:0/999651188 learned_addr learned my addr 172.21.15.32:0/999651188 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:31.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdd07fff700 1 --2- 172.21.15.32:0/999651188 >> v2:172.21.15.86:3300/0 conn(0x7fdd0810d010 0x7fdd080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdcfffff700 1 -- 172.21.15.32:0/999651188 >> v2:172.21.15.86:3300/0 conn(0x7fdd0810d010 msgr2=0x7fdd080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:31.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.274 7fdcfffff700 1 --2- 172.21.15.32:0/999651188 >> v2:172.21.15.86:3300/0 conn(0x7fdd0810d010 0x7fdd080d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.275 7fdcfffff700 1 -- 172.21.15.32:0/999651188 >> v1:172.21.15.32:6789/0 conn(0x7fdd080d82c0 legacy=0x7fdd0810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.275 7fdcfffff700 1 -- 172.21.15.32:0/999651188 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd080d86f0 con 0x7fdd080d1da0 2021-04-09T18:22:31.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.275 7fdd077fe700 1 -- 172.21.15.32:0/999651188 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997339827 0 0) 0x7fdd00000d30 con 0x7fdd080d82c0 2021-04-09T18:22:31.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.275 7fdcfffff700 1 --2- 172.21.15.32:0/999651188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080d1da0 0x7fdd080d7ca0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7fdcf0006650 tx=0x7fdcf0009a00).ready entity=mon.0 client_cookie=90e4755f4a1c2c46 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:31.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.275 7fdd077fe700 1 -- 172.21.15.32:0/999651188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf0009df0 con 0x7fdd080d1da0 2021-04-09T18:22:31.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.275 7fdd077fe700 1 -- 172.21.15.32:0/999651188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf00009b0 con 0x7fdd080d1da0 2021-04-09T18:22:31.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.275 7fdd077fe700 1 -- 172.21.15.32:0/999651188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf000c2b0 con 0x7fdd080d1da0 2021-04-09T18:22:31.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.275 7fdd0e978700 1 -- 172.21.15.32:0/999651188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080d1da0 msgr2=0x7fdd080d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.275 7fdd0e978700 1 --2- 172.21.15.32:0/999651188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080d1da0 0x7fdd080d7ca0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7fdcf0006650 tx=0x7fdcf0009a00).stop 2021-04-09T18:22:31.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.276 7fdd0e978700 1 -- 172.21.15.32:0/999651188 shutdown_connections 2021-04-09T18:22:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.276 7fdd0e978700 1 --2- 172.21.15.32:0/999651188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080d1da0 0x7fdd080d7ca0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.276 7fdd0e978700 1 --2- 172.21.15.32:0/999651188 >> v2:172.21.15.86:3300/0 conn(0x7fdd0810d010 0x7fdd080d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.276 7fdd0e978700 1 -- 172.21.15.32:0/999651188 >> 172.21.15.32:0/999651188 conn(0x7fdd080ce5f0 msgr2=0x7fdd080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.276 7fdd0e978700 1 -- 172.21.15.32:0/999651188 shutdown_connections 2021-04-09T18:22:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.276 7fdd0e978700 1 -- 172.21.15.32:0/999651188 wait complete. 2021-04-09T18:22:31.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.296 7fdd0e978700 1 Processor -- start 2021-04-09T18:22:31.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.296 7fdd0e978700 1 -- start start 2021-04-09T18:22:31.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.296 7fdd0e978700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd081a5210 0x7fdd081a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.296 7fdd0e978700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081a6340 0x7fdd081a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.296 7fdd0e978700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080abaa0 0x7fdd080abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.296 7fdd0e978700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd081a6d60 con 0x7fdd080abaa0 2021-04-09T18:22:31.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.296 7fdd0e978700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080ac4c0 con 0x7fdd081a6340 2021-04-09T18:22:31.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.296 7fdd0e978700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080ac6e0 con 0x7fdd081a5210 2021-04-09T18:22:31.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0cf95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080abaa0 0x7fdd080abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0cf95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080abaa0 0x7fdd080abea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49300/0 (socket says 172.21.15.32:49300) 2021-04-09T18:22:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0cf95700 1 -- 172.21.15.32:0/983023105 learned_addr learned my addr 172.21.15.32:0/983023105 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdcfffff700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081a6340 0x7fdd081a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd07fff700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd081a5210 0x7fdd081a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0cf95700 1 -- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd081a5210 msgr2=0x7fdd081a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0cf95700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd081a5210 0x7fdd081a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0cf95700 1 -- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081a6340 msgr2=0x7fdd081a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0cf95700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081a6340 0x7fdd081a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0cf95700 1 -- 172.21.15.32:0/983023105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0817a720 con 0x7fdd080abaa0 2021-04-09T18:22:31.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdcfffff700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081a6340 0x7fdd081a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:31.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0cf95700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080abaa0 0x7fdd080abea0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7fdd0000cfc0 tx=0x7fdd00002100).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:31.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd057fa700 1 -- 172.21.15.32:0/983023105 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd00000cc0 con 0x7fdd080abaa0 2021-04-09T18:22:31.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd057fa700 1 -- 172.21.15.32:0/983023105 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd000070d0 con 0x7fdd080abaa0 2021-04-09T18:22:31.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0e978700 1 -- 172.21.15.32:0/983023105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0817a9c0 con 0x7fdd080abaa0 2021-04-09T18:22:31.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd057fa700 1 -- 172.21.15.32:0/983023105 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd00004720 con 0x7fdd080abaa0 2021-04-09T18:22:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.297 7fdd0e978700 1 -- 172.21.15.32:0/983023105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0817a9c0 con 0x7fdd080abaa0 2021-04-09T18:22:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.298 7fdd057fa700 1 -- 172.21.15.32:0/983023105 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdd000072b0 con 0x7fdd080abaa0 2021-04-09T18:22:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.298 7fdd057fa700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdcec018900 0x7fdcec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.298 7fdd057fa700 1 -- 172.21.15.32:0/983023105 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fdd00034c90 con 0x7fdd080abaa0 2021-04-09T18:22:31.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.298 7fdd07fff700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdcec018900 0x7fdcec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.298 7fdd07fff700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdcec018900 0x7fdcec01adc0 crc :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:31.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.318 7fdd0e978700 1 -- 172.21.15.32:0/983023105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd081a6d60 con 0x7fdd080abaa0 2021-04-09T18:22:31.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.321 7fdd057fa700 1 -- 172.21.15.32:0/983023105 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdd00007d00 con 0x7fdd080abaa0 2021-04-09T18:22:31.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.512 7fdd0e978700 1 -- 172.21.15.32:0/983023105 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_shrink"]} v 0) v1 -- 0x7fdd081a6d60 con 0x7fdd080abaa0 2021-04-09T18:22:31.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.643 7fdd057fa700 1 -- 172.21.15.32:0/983023105 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_shrink"]}]=0 v446) v1 ==== 135+0+0 (secure 0 0 0) 0x7fdd000070d0 con 0x7fdd080abaa0 2021-04-09T18:22:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 -- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdcec018900 msgr2=0x7fdcec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdcec018900 0x7fdcec01adc0 crc :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 -- 172.21.15.32:0/983023105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080abaa0 msgr2=0x7fdd080abea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080abaa0 0x7fdd080abea0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7fdd0000cfc0 tx=0x7fdd00002100).stop 2021-04-09T18:22:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 -- 172.21.15.32:0/983023105 shutdown_connections 2021-04-09T18:22:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdcec018900 0x7fdcec01adc0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd080abaa0 0x7fdd080abea0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081a6340 0x7fdd081a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 --2- 172.21.15.32:0/983023105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd081a5210 0x7fdd081a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 -- 172.21.15.32:0/983023105 >> 172.21.15.32:0/983023105 conn(0x7fdd081a99c0 msgr2=0x7fdd0817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 -- 172.21.15.32:0/983023105 shutdown_connections 2021-04-09T18:22:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.678 7fdd0e978700 1 -- 172.21.15.32:0/983023105 wait complete. 2021-04-09T18:22:31.743 INFO:tasks.cephfs_test_runner:test_shrink (tasks.cephfs.test_failover.TestClusterResize) ... ok 2021-04-09T18:22:31.743 INFO:tasks.cephfs_test_runner:Starting test: test_thrash (tasks.cephfs.test_failover.TestClusterResize) 2021-04-09T18:22:31.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_thrash' 2021-04-09T18:22:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.837 7f5850d26700 1 Processor -- start 2021-04-09T18:22:31.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.837 7f5850d26700 1 -- start start 2021-04-09T18:22:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f5850d26700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f584c0cffa0 0x7f584c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f5850d26700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c0d7280 con 0x7f584c0d6c80 2021-04-09T18:22:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f5850d26700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c0d83f0 con 0x7f584c0d7800 2021-04-09T18:22:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f5850d26700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f584c0d7080 con 0x7f584c0cffa0 2021-04-09T18:22:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f584ab13700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f584c0cffa0 0x7f584c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f584b314700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f584c0d6c80 0x7f584c10cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58898/0 (socket says 172.21.15.32:58898) 2021-04-09T18:22:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f584b314700 1 -- 172.21.15.32:0/3630653504 learned_addr learned my addr 172.21.15.32:0/3630653504 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f5849b11700 1 -- 172.21.15.32:0/3630653504 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016829022 0 0) 0x7f5844000d30 con 0x7f584c0d6c80 2021-04-09T18:22:31.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f5849b11700 1 -- 172.21.15.32:0/3630653504 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5834002fe0 con 0x7f584c0d6c80 2021-04-09T18:22:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f584ab13700 1 -- 172.21.15.32:0/3630653504 >> v1:172.21.15.86:6789/0 conn(0x7f584c0d7800 legacy=0x7f584c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f584ab13700 1 -- 172.21.15.32:0/3630653504 >> v1:172.21.15.32:6789/0 conn(0x7f584c0d6c80 legacy=0x7f584c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.838 7f584ab13700 1 -- 172.21.15.32:0/3630653504 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c0d91e0 con 0x7f584c0cffa0 2021-04-09T18:22:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.839 7f584ab13700 1 --2- 172.21.15.32:0/3630653504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c0cffa0 0x7f584c0d2440 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f583c006650 tx=0x7f583c009a00).ready entity=mon.0 client_cookie=13d8eab9bcd15dc7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:31.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.839 7f5849b11700 1 -- 172.21.15.32:0/3630653504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f583c009df0 con 0x7f584c0cffa0 2021-04-09T18:22:31.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.839 7f5849b11700 1 -- 172.21.15.32:0/3630653504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c0009b0 con 0x7f584c0cffa0 2021-04-09T18:22:31.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.839 7f5849b11700 1 -- 172.21.15.32:0/3630653504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f583c00c2b0 con 0x7f584c0cffa0 2021-04-09T18:22:31.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.839 7f5850d26700 1 -- 172.21.15.32:0/3630653504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c0cffa0 msgr2=0x7f584c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.839 7f5850d26700 1 --2- 172.21.15.32:0/3630653504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c0cffa0 0x7f584c0d2440 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f583c006650 tx=0x7f583c009a00).stop 2021-04-09T18:22:31.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.840 7f5850d26700 1 -- 172.21.15.32:0/3630653504 shutdown_connections 2021-04-09T18:22:31.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.840 7f5850d26700 1 --2- 172.21.15.32:0/3630653504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c0cffa0 0x7f584c0d2440 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.840 7f5850d26700 1 -- 172.21.15.32:0/3630653504 >> 172.21.15.32:0/3630653504 conn(0x7f584c0ce5c0 msgr2=0x7f584c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.840 7f5850d26700 1 -- 172.21.15.32:0/3630653504 shutdown_connections 2021-04-09T18:22:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.840 7f5850d26700 1 -- 172.21.15.32:0/3630653504 wait complete. 2021-04-09T18:22:31.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f5850d26700 1 Processor -- start 2021-04-09T18:22:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f5850d26700 1 -- start start 2021-04-09T18:22:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f5850d26700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f584c17dcf0 0x7f584c17e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f5850d26700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c1a0820 0x7f584c1a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f5850d26700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f584c19f050 0x7f584c19f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f5850d26700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c1a02a0 con 0x7f584c1a0820 2021-04-09T18:22:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f5850d26700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c1a1240 con 0x7f584c17dcf0 2021-04-09T18:22:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f5850d26700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584c0a1aa0 con 0x7f584c19f050 2021-04-09T18:22:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f584a312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c1a0820 0x7f584c1a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f584a312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c1a0820 0x7f584c1a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49312/0 (socket says 172.21.15.32:49312) 2021-04-09T18:22:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.860 7f584a312700 1 -- 172.21.15.32:0/718046646 learned_addr learned my addr 172.21.15.32:0/718046646 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f584b314700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f584c19f050 0x7f584c19f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f584ab13700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f584c17dcf0 0x7f584c17e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f584a312700 1 -- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f584c19f050 msgr2=0x7f584c19f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f584a312700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f584c19f050 0x7f584c19f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f584a312700 1 -- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f584c17dcf0 msgr2=0x7f584c17e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f584a312700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f584c17dcf0 0x7f584c17e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f584a312700 1 -- 172.21.15.32:0/718046646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c19fa70 con 0x7f584c1a0820 2021-04-09T18:22:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f584ab13700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f584c17dcf0 0x7f584c17e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:22:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f584a312700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c1a0820 0x7f584c1a0c20 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f584000b5b0 tx=0x7f584000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f583b7fe700 1 -- 172.21.15.32:0/718046646 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f584000bd40 con 0x7f584c1a0820 2021-04-09T18:22:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f583b7fe700 1 -- 172.21.15.32:0/718046646 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5840006020 con 0x7f584c1a0820 2021-04-09T18:22:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f5850d26700 1 -- 172.21.15.32:0/718046646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584c19fa70 con 0x7f584c1a0820 2021-04-09T18:22:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f583b7fe700 1 -- 172.21.15.32:0/718046646 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58400054a0 con 0x7f584c1a0820 2021-04-09T18:22:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.861 7f5850d26700 1 -- 172.21.15.32:0/718046646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584c19fa70 con 0x7f584c1a0820 2021-04-09T18:22:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.862 7f583b7fe700 1 -- 172.21.15.32:0/718046646 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5840006200 con 0x7f584c1a0820 2021-04-09T18:22:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.862 7f583b7fe700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58300188d0 0x7f583001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.862 7f583b7fe700 1 -- 172.21.15.32:0/718046646 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f5840032720 con 0x7f584c1a0820 2021-04-09T18:22:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.862 7f584ab13700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58300188d0 0x7f583001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.862 7f584ab13700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58300188d0 0x7f583001add0 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.882 7f5850d26700 1 -- 172.21.15.32:0/718046646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584c17caf0 con 0x7f584c1a0820 2021-04-09T18:22:31.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:31.885 7f583b7fe700 1 -- 172.21.15.32:0/718046646 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f584000bd40 con 0x7f584c1a0820 2021-04-09T18:22:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.071 7f5850d26700 1 -- 172.21.15.32:0/718046646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_thrash"]} v 0) v1 -- 0x7f584c17caf0 con 0x7f584c1a0820 2021-04-09T18:22:32.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.644 7f583b7fe700 1 -- 172.21.15.32:0/718046646 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_thrash"]}]=0 v447) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5840006000 con 0x7f584c1a0820 2021-04-09T18:22:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 -- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58300188d0 msgr2=0x7f583001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:32.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58300188d0 0x7f583001add0 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 -- 172.21.15.32:0/718046646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c1a0820 msgr2=0x7f584c1a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c1a0820 0x7f584c1a0c20 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f584000b5b0 tx=0x7f584000b950).stop 2021-04-09T18:22:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 -- 172.21.15.32:0/718046646 shutdown_connections 2021-04-09T18:22:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f58300188d0 0x7f583001add0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f584c1a0820 0x7f584c1a0c20 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f584c19f050 0x7f584c19f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 --2- 172.21.15.32:0/718046646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f584c17dcf0 0x7f584c17e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 -- 172.21.15.32:0/718046646 >> 172.21.15.32:0/718046646 conn(0x7f584c099f20 msgr2=0x7f584c094920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:32.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 -- 172.21.15.32:0/718046646 shutdown_connections 2021-04-09T18:22:32.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.689 7f5850d26700 1 -- 172.21.15.32:0/718046646 wait complete. 2021-04-09T18:22:32.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:22:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.857 7fe8f3b53700 1 Processor -- start 2021-04-09T18:22:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.857 7fe8f3b53700 1 -- start start 2021-04-09T18:22:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f3b53700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe8ec0cffc0 0x7fe8ec0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f3b53700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8ec0d7820 0x7fe8ec0d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f3b53700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ec0d72a0 con 0x7fe8ec0d8240 2021-04-09T18:22:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f3b53700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8ec0d9200 con 0x7fe8ec0d7820 2021-04-09T18:22:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f3b53700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8ec10e000 con 0x7fe8ec0cffc0 2021-04-09T18:22:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f116e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe8ec0d7820 0x7fe8ec0d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f2170700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe8ec0d8240 0x7fe8ec0d5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58912/0 (socket says 172.21.15.32:58912) 2021-04-09T18:22:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f2170700 1 -- 172.21.15.32:0/3670769016 learned_addr learned my addr 172.21.15.32:0/3670769016 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f196f700 1 --2- 172.21.15.32:0/3670769016 >> v2:172.21.15.86:3300/0 conn(0x7fe8ec0cffc0 0x7fe8ec0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f116e700 1 -- 172.21.15.32:0/3670769016 >> v2:172.21.15.86:3300/0 conn(0x7fe8ec0cffc0 msgr2=0x7fe8ec0d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f116e700 1 --2- 172.21.15.32:0/3670769016 >> v2:172.21.15.86:3300/0 conn(0x7fe8ec0cffc0 0x7fe8ec0d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f116e700 1 -- 172.21.15.32:0/3670769016 >> v1:172.21.15.32:6789/0 conn(0x7fe8ec0d8240 legacy=0x7fe8ec0d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f096d700 1 -- 172.21.15.32:0/3670769016 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930573525 0 0) 0x7fe8e4000d30 con 0x7fe8ec0d8240 2021-04-09T18:22:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f116e700 1 -- 172.21.15.32:0/3670769016 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec10e220 con 0x7fe8ec0d7820 2021-04-09T18:22:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.858 7fe8f196f700 1 --2- 172.21.15.32:0/3670769016 >> v2:172.21.15.86:3300/0 conn(0x7fe8ec0cffc0 0x7fe8ec0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.859 7fe8f116e700 1 --2- 172.21.15.32:0/3670769016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec0d7820 0x7fe8ec0d7c20 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7fe8dc006650 tx=0x7fe8dc009a80).ready entity=mon.0 client_cookie=5062e15c55e8d34b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.859 7fe8f096d700 1 -- 172.21.15.32:0/3670769016 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc0009b0 con 0x7fe8ec0d7820 2021-04-09T18:22:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.859 7fe8f096d700 1 -- 172.21.15.32:0/3670769016 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc00c0c0 con 0x7fe8ec0d7820 2021-04-09T18:22:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.859 7fe8f096d700 1 -- 172.21.15.32:0/3670769016 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc00c510 con 0x7fe8ec0d7820 2021-04-09T18:22:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.859 7fe8f3b53700 1 -- 172.21.15.32:0/3670769016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec0d7820 msgr2=0x7fe8ec0d7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.859 7fe8f3b53700 1 --2- 172.21.15.32:0/3670769016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec0d7820 0x7fe8ec0d7c20 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7fe8dc006650 tx=0x7fe8dc009a80).stop 2021-04-09T18:22:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.860 7fe8f3b53700 1 -- 172.21.15.32:0/3670769016 shutdown_connections 2021-04-09T18:22:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.860 7fe8f3b53700 1 --2- 172.21.15.32:0/3670769016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec0d7820 0x7fe8ec0d7c20 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.860 7fe8f3b53700 1 --2- 172.21.15.32:0/3670769016 >> v2:172.21.15.86:3300/0 conn(0x7fe8ec0cffc0 0x7fe8ec0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.860 7fe8f3b53700 1 -- 172.21.15.32:0/3670769016 >> 172.21.15.32:0/3670769016 conn(0x7fe8ec0ce5e0 msgr2=0x7fe8ec0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:32.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.860 7fe8f3b53700 1 -- 172.21.15.32:0/3670769016 shutdown_connections 2021-04-09T18:22:32.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.860 7fe8f3b53700 1 -- 172.21.15.32:0/3670769016 wait complete. 2021-04-09T18:22:32.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f3b53700 1 Processor -- start 2021-04-09T18:22:32.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f3b53700 1 -- start start 2021-04-09T18:22:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f3b53700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec182e60 0x7fe8ec183260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f3b53700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec09d0f0 0x7fe8ec09d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f3b53700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec181330 0x7fe8ec181730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f3b53700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec09cb70 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f3b53700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec09db10 con 0x7fe8ec181330 2021-04-09T18:22:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f3b53700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec181d50 con 0x7fe8ec182e60 2021-04-09T18:22:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f116e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec09d0f0 0x7fe8ec09d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.880 7fe8f116e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec09d0f0 0x7fe8ec09d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49330/0 (socket says 172.21.15.32:49330) 2021-04-09T18:22:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f116e700 1 -- 172.21.15.32:0/987306838 learned_addr learned my addr 172.21.15.32:0/987306838 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f2170700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec181330 0x7fe8ec181730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f196f700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec182e60 0x7fe8ec183260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f116e700 1 -- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec182e60 msgr2=0x7fe8ec183260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f116e700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec182e60 0x7fe8ec183260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f116e700 1 -- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec181330 msgr2=0x7fe8ec181730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f116e700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec181330 0x7fe8ec181730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f116e700 1 -- 172.21.15.32:0/987306838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec181f70 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f2170700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec181330 0x7fe8ec181730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:32.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f116e700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec09d0f0 0x7fe8ec09d4f0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fe8dc00d760 tx=0x7fe8dc00db00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:32.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8e27fc700 1 -- 172.21.15.32:0/987306838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc000c90 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8e27fc700 1 -- 172.21.15.32:0/987306838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc00b030 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ec181f70 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8e27fc700 1 -- 172.21.15.32:0/987306838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc002020 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.881 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ec181f70 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.882 7fe8e27fc700 1 -- 172.21.15.32:0/987306838 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe8dc009300 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.882 7fe8e27fc700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8d4018930 0x7fe8d401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:32.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.882 7fe8e27fc700 1 -- 172.21.15.32:0/987306838 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fe8dc034990 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.882 7fe8f196f700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8d4018930 0x7fe8d401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:32.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.882 7fe8f196f700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8d4018930 0x7fe8d401ae10 crc :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.902 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ec093df0 con 0x7fe8ec09d0f0 2021-04-09T18:22:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:32.906 7fe8e27fc700 1 -- 172.21.15.32:0/987306838 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe8dc006490 con 0x7fe8ec09d0f0 2021-04-09T18:22:33.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.120 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe8ec0d72a0 con 0x7fe8ec09d0f0 2021-04-09T18:22:33.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.126 7fe8e27fc700 1 -- 172.21.15.32:0/987306838 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe8dc009ee0 con 0x7fe8ec09d0f0 2021-04-09T18:22:33.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8d4018930 msgr2=0x7fe8d401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:33.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8d4018930 0x7fe8d401ae10 crc :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec09d0f0 msgr2=0x7fe8ec09d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec09d0f0 0x7fe8ec09d4f0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fe8dc00d760 tx=0x7fe8dc00db00).stop 2021-04-09T18:22:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 shutdown_connections 2021-04-09T18:22:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8d4018930 0x7fe8d401ae10 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8ec09d0f0 0x7fe8ec09d4f0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8ec181330 0x7fe8ec181730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 --2- 172.21.15.32:0/987306838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8ec182e60 0x7fe8ec183260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 >> 172.21.15.32:0/987306838 conn(0x7fe8ec09fa00 msgr2=0x7fe8ec0b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:33.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 shutdown_connections 2021-04-09T18:22:33.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.130 7fe8f3b53700 1 -- 172.21.15.32:0/987306838 wait complete. 2021-04-09T18:22:33.194 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:22:33.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:22:33.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:22:33.311 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:22:33.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:33.312 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:22:33.313 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:22:33.357 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:22:33.357 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:22:33.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:33.358 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:22:33.359 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:22:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:22:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:22:33.401 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:22:33.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:22:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.496 7ff4339db700 1 Processor -- start 2021-04-09T18:22:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.496 7ff4339db700 1 -- start start 2021-04-09T18:22:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4339db700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff42c10d010 0x7ff42c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:33.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4339db700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0d1da0 0x7ff42c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:33.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4339db700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff42c0d82c0 0x7ff42c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:33.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4339db700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff42c0d8d30 con 0x7ff42c10d010 2021-04-09T18:22:33.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4339db700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff42c0d9220 con 0x7ff42c0d82c0 2021-04-09T18:22:33.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4339db700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c10d480 con 0x7ff42c0d1da0 2021-04-09T18:22:33.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4317f7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff42c10d010 0x7ff42c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:33.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4317f7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff42c10d010 0x7ff42c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49334/0 (socket says 172.21.15.32:49334) 2021-04-09T18:22:33.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4317f7700 1 -- 172.21.15.32:0/472694887 learned_addr learned my addr 172.21.15.32:0/472694887 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff430ff6700 1 --2- 172.21.15.32:0/472694887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0d1da0 0x7ff42c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff431ff8700 1 --2- 172.21.15.32:0/472694887 >> v2:172.21.15.86:3300/0 conn(0x7ff42c0d82c0 0x7ff42c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4317f7700 1 -- 172.21.15.32:0/472694887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0d1da0 msgr2=0x7ff42c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4317f7700 1 --2- 172.21.15.32:0/472694887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0d1da0 0x7ff42c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4317f7700 1 -- 172.21.15.32:0/472694887 >> v2:172.21.15.86:3300/0 conn(0x7ff42c0d82c0 msgr2=0x7ff42c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.497 7ff4317f7700 1 --2- 172.21.15.32:0/472694887 >> v2:172.21.15.86:3300/0 conn(0x7ff42c0d82c0 0x7ff42c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.498 7ff4317f7700 1 -- 172.21.15.32:0/472694887 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c0d86c0 con 0x7ff42c10d010 2021-04-09T18:22:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.498 7ff430ff6700 1 --2- 172.21.15.32:0/472694887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0d1da0 0x7ff42c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.498 7ff4317f7700 1 --2- 172.21.15.32:0/472694887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c10d010 0x7ff42c0d1780 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7ff428006a80 tx=0x7ff428009f00).ready entity=mon.0 client_cookie=45ed0e8204f405e7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.498 7ff423fff700 1 -- 172.21.15.32:0/472694887 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff42800b5c0 con 0x7ff42c10d010 2021-04-09T18:22:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.498 7ff423fff700 1 -- 172.21.15.32:0/472694887 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42800b8c0 con 0x7ff42c10d010 2021-04-09T18:22:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.498 7ff423fff700 1 -- 172.21.15.32:0/472694887 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff42800bda0 con 0x7ff42c10d010 2021-04-09T18:22:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.498 7ff4339db700 1 -- 172.21.15.32:0/472694887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c10d010 msgr2=0x7ff42c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:33.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.498 7ff4339db700 1 --2- 172.21.15.32:0/472694887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c10d010 0x7ff42c0d1780 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7ff428006a80 tx=0x7ff428009f00).stop 2021-04-09T18:22:33.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.499 7ff4339db700 1 -- 172.21.15.32:0/472694887 shutdown_connections 2021-04-09T18:22:33.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.499 7ff4339db700 1 --2- 172.21.15.32:0/472694887 >> v2:172.21.15.86:3300/0 conn(0x7ff42c0d82c0 0x7ff42c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.499 7ff4339db700 1 --2- 172.21.15.32:0/472694887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0d1da0 0x7ff42c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.499 7ff4339db700 1 --2- 172.21.15.32:0/472694887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c10d010 0x7ff42c0d1780 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.499 7ff4339db700 1 -- 172.21.15.32:0/472694887 >> 172.21.15.32:0/472694887 conn(0x7ff42c0ce5f0 msgr2=0x7ff42c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:33.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.499 7ff4339db700 1 -- 172.21.15.32:0/472694887 shutdown_connections 2021-04-09T18:22:33.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.499 7ff4339db700 1 -- 172.21.15.32:0/472694887 wait complete. 2021-04-09T18:22:33.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.519 7ff4339db700 1 Processor -- start 2021-04-09T18:22:33.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.519 7ff4339db700 1 -- start start 2021-04-09T18:22:33.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4339db700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c09a5e0 0x7ff42c1884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:33.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4339db700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0a3350 0x7ff42c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:33.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4339db700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff42c0a15b0 0x7ff42c0a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:33.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4339db700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c0a40d0 con 0x7ff42c09a5e0 2021-04-09T18:22:33.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4339db700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c0a2dd0 con 0x7ff42c0a15b0 2021-04-09T18:22:33.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4339db700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c188ad0 con 0x7ff42c0a3350 2021-04-09T18:22:33.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4317f7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c09a5e0 0x7ff42c1884b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:33.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4317f7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c09a5e0 0x7ff42c1884b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49340/0 (socket says 172.21.15.32:49340) 2021-04-09T18:22:33.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4317f7700 1 -- 172.21.15.32:0/3989912722 learned_addr learned my addr 172.21.15.32:0/3989912722 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:33.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff431ff8700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff42c0a15b0 0x7ff42c0a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:33.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff430ff6700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0a3350 0x7ff42c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:33.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4317f7700 1 -- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0a3350 msgr2=0x7ff42c0a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4317f7700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0a3350 0x7ff42c0a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4317f7700 1 -- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff42c0a15b0 msgr2=0x7ff42c0a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4317f7700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff42c0a15b0 0x7ff42c0a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4317f7700 1 -- 172.21.15.32:0/3989912722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c0a3d70 con 0x7ff42c09a5e0 2021-04-09T18:22:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4317f7700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c09a5e0 0x7ff42c1884b0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7ff428009450 tx=0x7ff42800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff421ffb700 1 -- 172.21.15.32:0/3989912722 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff428006e40 con 0x7ff42c09a5e0 2021-04-09T18:22:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff421ffb700 1 -- 172.21.15.32:0/3989912722 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42800be60 con 0x7ff42c09a5e0 2021-04-09T18:22:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff4339db700 1 -- 172.21.15.32:0/3989912722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c0a3d70 con 0x7ff42c09a5e0 2021-04-09T18:22:33.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.520 7ff421ffb700 1 -- 172.21.15.32:0/3989912722 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff42800c4e0 con 0x7ff42c09a5e0 2021-04-09T18:22:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.521 7ff4339db700 1 -- 172.21.15.32:0/3989912722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42c0a3d70 con 0x7ff42c09a5e0 2021-04-09T18:22:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.521 7ff421ffb700 1 -- 172.21.15.32:0/3989912722 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff428008020 con 0x7ff42c09a5e0 2021-04-09T18:22:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.521 7ff421ffb700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff40c018900 0x7ff40c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:33.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.521 7ff421ffb700 1 -- 172.21.15.32:0/3989912722 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4843+0+0 (secure 0 0 0) 0x7ff428032ad0 con 0x7ff42c09a5e0 2021-04-09T18:22:33.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.521 7ff430ff6700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff40c018900 0x7ff40c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:33.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.522 7ff430ff6700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff40c018900 0x7ff40c01adc0 crc :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:33.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.541 7ff4339db700 1 -- 172.21.15.32:0/3989912722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42c10d480 con 0x7ff42c09a5e0 2021-04-09T18:22:33.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.545 7ff421ffb700 1 -- 172.21.15.32:0/3989912722 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff42801faf0 con 0x7ff42c09a5e0 2021-04-09T18:22:33.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.734 7ff4339db700 1 -- 172.21.15.32:0/3989912722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff42c10d480 con 0x7ff42c09a5e0 2021-04-09T18:22:33.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.735 7ff421ffb700 1 -- 172.21.15.32:0/3989912722 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v98) v1 ==== 81+0+25541 (secure 0 0 0) 0x7ff428007fe0 con 0x7ff42c09a5e0 2021-04-09T18:22:33.790 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:33.790 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:22:33.790 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 98, 2021-04-09T18:22:33.790 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:22:33.791 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:22:33.791 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:22:30.635981", 2021-04-09T18:22:33.791 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:22:33.791 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:22:33.792 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:22:33.792 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:22:33.792 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:22:33.792 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:22:33.792 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:22:33.793 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:22:33.793 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:22:33.793 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:22:33.793 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:22:33.793 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:22:33.794 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:22:33.794 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:22:33.794 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:22:33.794 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 17, 2021-04-09T18:22:33.795 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:22:33.795 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:22:33.795 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:22:33.795 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:22:33.796 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:22:33.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.796 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:22:33.796 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:22:33.796 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:22:33.797 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:22:33.797 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:22:33.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:22:33.797 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:22:33.797 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:22:33.798 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:22:33.798 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:22:33.798 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:22:33.798 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:22:33.799 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:22:33.799 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:22:33.799 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:22:33.799 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:22:33.800 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:22:33.800 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:22:33.800 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:22:33.800 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:22:33.800 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:22:33.801 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:22:33.801 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:22:33.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.801 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:22:33.801 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:22:33.802 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:22:33.802 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:22:33.802 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:22:33.802 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:22:33.803 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:22:33.803 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:22:33.803 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:22:33.803 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:22:33.803 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:22:33.804 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:22:33.804 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:22:33.804 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:22:33.804 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:22:33.804 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:22:33.805 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:22:33.805 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:22:33.805 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:22:33.805 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:22:33.806 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:22:33.806 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:22:33.806 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:22:33.806 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:22:33.806 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:22:33.807 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:22:33.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:22:33.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.807 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:22:33.807 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:22:33.808 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:22:33.808 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:22:33.808 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:22:33.808 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:22:33.809 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:22:33.809 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:22:33.809 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:22:33.809 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:22:33.809 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:22:33.810 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:22:33.810 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:22:33.810 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:22:33.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.811 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:22:33.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:22:33.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:22:33.812 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:22:33.812 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:33.812 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:33.812 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:33.812 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:33.813 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:33.813 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:33.813 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:33.813 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 90, 2021-04-09T18:22:33.814 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:33.814 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:33.814 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:33.814 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:22:33.815 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:33.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.816 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.816 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:22:33.816 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:33.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.817 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.817 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:33.817 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:22:33.818 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:33.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.819 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:22:33.819 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:33.820 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.820 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.820 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:33.820 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:22:33.821 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:33.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.822 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:22:33.822 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:33.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.823 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.823 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:33.823 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:22:33.824 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:33.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:22:33.825 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:22:33.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.826 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.826 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:22:33.826 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:22:33.827 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:22:33.827 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:22:33.827 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:33.827 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:33.827 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:33.828 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.828 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:22:33.828 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:22:33.829 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:33.829 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:33.829 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:33.829 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:33.829 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:33.830 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:33.830 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:33.830 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 80, 2021-04-09T18:22:33.830 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:33.831 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:33.831 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:33.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.832 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:22:33.832 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:33.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.833 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.833 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:22:33.833 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:33.833 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.833 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.834 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:33.834 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.834 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.835 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:22:33.835 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:33.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.835 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.836 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:22:33.836 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:33.836 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.836 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.837 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:33.837 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:22:33.838 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:33.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.839 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:22:33.839 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:33.839 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.839 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.840 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:33.840 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.840 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.840 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:22:33.841 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:33.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.842 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:22:33.842 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:22:33.842 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.842 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.843 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:22:33.843 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:22:33.843 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:22:33.843 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:22:33.844 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:33.844 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:33.844 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:33.844 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.845 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:22:33.845 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:22:33.845 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:33.845 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:33.846 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:33.846 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:33.846 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:33.846 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:33.846 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:33.847 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 89, 2021-04-09T18:22:33.847 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:33.847 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:33.847 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:33.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:22:33.848 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:22:33.849 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:33.850 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.850 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.850 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:22:33.851 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:22:33.852 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:33.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.853 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.853 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:22:33.854 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:33.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:22:33.855 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:33.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.856 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.856 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:33.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.857 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.857 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:22:33.857 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:33.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.858 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:22:33.858 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:22:33.859 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.859 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.859 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:22:33.859 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:22:33.860 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:22:33.860 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:22:33.860 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:33.860 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:33.861 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:33.861 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.861 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:22:33.862 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:22:33.862 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:33.862 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:33.862 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:33.862 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:33.863 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:33.863 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:33.863 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:33.863 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 89, 2021-04-09T18:22:33.864 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:33.864 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:33.864 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:33.864 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.865 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:22:33.865 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:33.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.866 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.866 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:22:33.866 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:33.866 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.866 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.867 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:33.867 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.868 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.868 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:22:33.868 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:33.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.869 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:22:33.869 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:33.869 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.869 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.870 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:33.870 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.871 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:22:33.871 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:33.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.872 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:22:33.872 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:33.872 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.872 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.873 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:33.873 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.873 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.874 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:22:33.874 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:33.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.874 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.875 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:22:33.875 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:22:33.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.875 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.876 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:22:33.876 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:22:33.876 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:22:33.876 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:22:33.877 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:33.877 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:33.877 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:33.877 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.878 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:22:33.878 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:22:33.878 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:33.878 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:33.879 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:33.879 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:33.879 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:33.879 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:33.880 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:33.880 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 90, 2021-04-09T18:22:33.880 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:33.880 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:33.880 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:33.881 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.881 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:22:33.882 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:33.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.882 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.882 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:22:33.883 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:33.883 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.883 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.884 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:33.884 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.884 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.884 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:22:33.885 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:33.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.885 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:22:33.886 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:33.886 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.886 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.887 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:33.887 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.887 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.887 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:22:33.888 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:33.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.888 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.889 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:22:33.889 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:33.889 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.889 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.890 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:33.890 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.890 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:22:33.891 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:33.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.892 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:22:33.892 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:22:33.892 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.892 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.893 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:22:33.893 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:22:33.893 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:22:33.894 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:22:33.894 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:33.894 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:33.894 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:33.894 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.895 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:22:33.895 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:22:33.896 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:33.896 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:33.896 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:33.896 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:33.897 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:33.897 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:33.897 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:33.897 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 89, 2021-04-09T18:22:33.898 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:33.898 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:33.898 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:33.898 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.899 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:22:33.899 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:33.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.900 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:22:33.900 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:33.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:22:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:33.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:22:33.903 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:33.904 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.904 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.904 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:33.904 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.905 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:22:33.905 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:33.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:22:33.906 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:33.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.907 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:33.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.908 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:22:33.908 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:33.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:22:33.909 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:22:33.910 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.910 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.910 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:22:33.910 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:22:33.911 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:22:33.911 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:22:33.911 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:33.911 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:33.911 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:33.912 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.912 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:22:33.913 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:22:33.913 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:33.913 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:33.913 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:33.913 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:33.914 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:33.914 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:33.914 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:33.914 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 90, 2021-04-09T18:22:33.914 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:33.915 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:33.915 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:33.915 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.915 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.916 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:22:33.916 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:33.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.917 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.917 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:22:33.917 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:33.917 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.917 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.918 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:33.918 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.919 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:22:33.919 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:33.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.920 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.920 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:22:33.920 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:33.920 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.920 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.921 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:33.921 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.921 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.922 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:22:33.922 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:33.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.922 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.923 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:22:33.923 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:33.923 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.923 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.924 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:33.924 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.924 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.925 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:22:33.925 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:33.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.925 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.926 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:22:33.926 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:22:33.926 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.926 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.927 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:22:33.927 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:22:33.927 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:22:33.927 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:22:33.928 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:33.928 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:33.928 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:33.928 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.929 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.929 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:22:33.929 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:22:33.929 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:22:33.929 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:22:33.930 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:22:33.930 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:22:33.930 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:22:33.930 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:22:33.930 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:22:33.931 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 92, 2021-04-09T18:22:33.931 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:22:33.931 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:22:33.931 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:22:33.932 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.932 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.932 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:22:33.932 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:33.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.933 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.933 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:22:33.933 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:33.934 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.934 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.934 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:22:33.935 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.935 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.935 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:22:33.935 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:33.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.936 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:22:33.936 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:33.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.937 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.937 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:22:33.937 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.938 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.938 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:22:33.938 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:33.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.939 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:22:33.939 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:33.940 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.940 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.940 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:22:33.941 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:22:33.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:22:33.941 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:22:33.942 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:33.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:33.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:22:33.943 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:22:33.943 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.943 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.944 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:22:33.944 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:22:33.944 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:22:33.944 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:22:33.945 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:22:33.945 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:22:33.945 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:22:33.945 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:33.945 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.946 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:22:33.946 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:22:33.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.946 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:22:33.947 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:33.947 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:33.947 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:33.947 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:33.948 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:33.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.948 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:22:33.948 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:33.949 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:33.949 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:33.949 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:33.949 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:33.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.950 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:22:33.950 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:33.950 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:22:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:33.952 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:33.953 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:33.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.953 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:22:33.953 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:33.954 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:33.954 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:33.954 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:33.954 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:33.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.955 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:22:33.955 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:33.955 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:22:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:22:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:22:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:22:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:22:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:22:33.960 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:22:33.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.960 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:22:33.960 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:22:33.960 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:22:33.961 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:22:33.961 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:22:33.961 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:22:33.961 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/3998950443": "2021-04-10 18:22:26.737257", 2021-04-09T18:22:33.961 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/3998950443": "2021-04-10 18:22:26.737257" 2021-04-09T18:22:33.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.962 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:22:33.962 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:22:33.962 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:22:33.963 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:22:33.963 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:22:33.963 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:22:33.963 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:22:33.963 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:22:33.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:33.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:33.964 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:22:33.964 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:22:33.964 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:22:33.965 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:22:33.965 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:22:33.965 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:22:33.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 -- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff40c018900 msgr2=0x7ff40c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:33.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff40c018900 0x7ff40c01adc0 crc :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 -- 172.21.15.32:0/3989912722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c09a5e0 msgr2=0x7ff42c1884b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:33.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c09a5e0 0x7ff42c1884b0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7ff428009450 tx=0x7ff42800bd20).stop 2021-04-09T18:22:33.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 -- 172.21.15.32:0/3989912722 shutdown_connections 2021-04-09T18:22:33.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff40c018900 0x7ff40c01adc0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff42c0a15b0 0x7ff42c0a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff42c0a3350 0x7ff42c0a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 --2- 172.21.15.32:0/3989912722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff42c09a5e0 0x7ff42c1884b0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 -- 172.21.15.32:0/3989912722 >> 172.21.15.32:0/3989912722 conn(0x7ff42c1a0db0 msgr2=0x7ff42c1a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 -- 172.21.15.32:0/3989912722 shutdown_connections 2021-04-09T18:22:33.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:33.737 7ff4339db700 1 -- 172.21.15.32:0/3989912722 wait complete. 2021-04-09T18:22:33.970 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:34.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.064 7f91d7ad3700 1 Processor -- start 2021-04-09T18:22:34.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.064 7f91d7ad3700 1 -- start start 2021-04-09T18:22:34.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d7ad3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d00d7ca0 0x7f91d00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d7ad3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f91d010da00 0x7f91d010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d7ad3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d010d480 con 0x7f91d010d010 2021-04-09T18:22:34.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d7ad3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d00d9220 con 0x7f91d010da00 2021-04-09T18:22:34.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d7ad3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d00d86c0 con 0x7f91d00d7ca0 2021-04-09T18:22:34.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d58ef700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f91d010d010 0x7f91d00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58938/0 (socket says 172.21.15.32:58938) 2021-04-09T18:22:34.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d58ef700 1 -- 172.21.15.32:0/785810963 learned_addr learned my addr 172.21.15.32:0/785810963 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d50ee700 1 --2- 172.21.15.32:0/785810963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d00d7ca0 0x7f91d00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d60f0700 1 --2- 172.21.15.32:0/785810963 >> v2:172.21.15.86:3300/0 conn(0x7f91d010da00 0x7f91d010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d48ed700 1 -- 172.21.15.32:0/785810963 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159083973 0 0) 0x7f91cc000ce0 con 0x7f91d010d010 2021-04-09T18:22:34.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d48ed700 1 -- 172.21.15.32:0/785810963 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91bc002fe0 con 0x7f91d010d010 2021-04-09T18:22:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d50ee700 1 -- 172.21.15.32:0/785810963 >> v2:172.21.15.86:3300/0 conn(0x7f91d010da00 msgr2=0x7f91d010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d50ee700 1 --2- 172.21.15.32:0/785810963 >> v2:172.21.15.86:3300/0 conn(0x7f91d010da00 0x7f91d010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d50ee700 1 -- 172.21.15.32:0/785810963 >> v1:172.21.15.32:6789/0 conn(0x7f91d010d010 legacy=0x7f91d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d50ee700 1 -- 172.21.15.32:0/785810963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d00d1f70 con 0x7f91d00d7ca0 2021-04-09T18:22:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d60f0700 1 --2- 172.21.15.32:0/785810963 >> v2:172.21.15.86:3300/0 conn(0x7f91d010da00 0x7f91d010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:34.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.065 7f91d50ee700 1 --2- 172.21.15.32:0/785810963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d00d7ca0 0x7f91d00d80a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f91c0006650 tx=0x7f91c0009a80).ready entity=mon.2 client_cookie=1ff2841d55b5ab73 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:34.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.066 7f91d48ed700 1 -- 172.21.15.32:0/785810963 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91c00009b0 con 0x7f91d00d7ca0 2021-04-09T18:22:34.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.066 7f91d48ed700 1 -- 172.21.15.32:0/785810963 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c000c0c0 con 0x7f91d00d7ca0 2021-04-09T18:22:34.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.066 7f91d48ed700 1 -- 172.21.15.32:0/785810963 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91c000c520 con 0x7f91d00d7ca0 2021-04-09T18:22:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.066 7f91d7ad3700 1 -- 172.21.15.32:0/785810963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d00d7ca0 msgr2=0x7f91d00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.066 7f91d7ad3700 1 --2- 172.21.15.32:0/785810963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d00d7ca0 0x7f91d00d80a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f91c0006650 tx=0x7f91c0009a80).stop 2021-04-09T18:22:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.066 7f91d7ad3700 1 -- 172.21.15.32:0/785810963 shutdown_connections 2021-04-09T18:22:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.066 7f91d7ad3700 1 --2- 172.21.15.32:0/785810963 >> v2:172.21.15.86:3300/0 conn(0x7f91d010da00 0x7f91d010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.066 7f91d7ad3700 1 --2- 172.21.15.32:0/785810963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d00d7ca0 0x7f91d00d80a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.066 7f91d7ad3700 1 -- 172.21.15.32:0/785810963 >> 172.21.15.32:0/785810963 conn(0x7f91d00ce5f0 msgr2=0x7f91d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.067 7f91d7ad3700 1 -- 172.21.15.32:0/785810963 shutdown_connections 2021-04-09T18:22:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.067 7f91d7ad3700 1 -- 172.21.15.32:0/785810963 wait complete. 2021-04-09T18:22:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d7ad3700 1 Processor -- start 2021-04-09T18:22:34.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d7ad3700 1 -- start start 2021-04-09T18:22:34.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d7ad3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d00a90c0 0x7f91d01a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d7ad3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d01a76d0 0x7f91d01a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d7ad3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d017bce0 0x7f91d017c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d7ad3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d01a7c60 con 0x7f91d00a90c0 2021-04-09T18:22:34.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d7ad3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d01a8e20 con 0x7f91d017bce0 2021-04-09T18:22:34.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d7ad3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d017c700 con 0x7f91d01a76d0 2021-04-09T18:22:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d58ef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d00a90c0 0x7f91d01a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d58ef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d00a90c0 0x7f91d01a81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49352/0 (socket says 172.21.15.32:49352) 2021-04-09T18:22:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d58ef700 1 -- 172.21.15.32:0/699974696 learned_addr learned my addr 172.21.15.32:0/699974696 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.087 7f91d50ee700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d01a76d0 0x7f91d01a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91d60f0700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d017bce0 0x7f91d017c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91d58ef700 1 -- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d01a76d0 msgr2=0x7f91d01a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91d58ef700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d01a76d0 0x7f91d01a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91d58ef700 1 -- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d017bce0 msgr2=0x7f91d017c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91d58ef700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d017bce0 0x7f91d017c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91d58ef700 1 -- 172.21.15.32:0/699974696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d017c920 con 0x7f91d00a90c0 2021-04-09T18:22:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91d58ef700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d00a90c0 0x7f91d01a81e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f91cc006d90 tx=0x7f91cc00b8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91c67fc700 1 -- 172.21.15.32:0/699974696 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91cc00bcb0 con 0x7f91d00a90c0 2021-04-09T18:22:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91c67fc700 1 -- 172.21.15.32:0/699974696 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc006030 con 0x7f91d00a90c0 2021-04-09T18:22:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d017c920 con 0x7f91d00a90c0 2021-04-09T18:22:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91c67fc700 1 -- 172.21.15.32:0/699974696 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91cc004620 con 0x7f91d00a90c0 2021-04-09T18:22:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.088 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d017c920 con 0x7f91d00a90c0 2021-04-09T18:22:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.089 7f91c67fc700 1 -- 172.21.15.32:0/699974696 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f91cc0043a0 con 0x7f91d00a90c0 2021-04-09T18:22:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.089 7f91c67fc700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b8018930 0x7f91b801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.089 7f91c67fc700 1 -- 172.21.15.32:0/699974696 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f91cc034930 con 0x7f91d00a90c0 2021-04-09T18:22:34.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.089 7f91d50ee700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b8018930 0x7f91b801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.089 7f91d50ee700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b8018930 0x7f91b801ae10 crc :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:34.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.109 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d01b7010 con 0x7f91d00a90c0 2021-04-09T18:22:34.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.112 7f91c67fc700 1 -- 172.21.15.32:0/699974696 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f91cc0009b0 con 0x7f91d00a90c0 2021-04-09T18:22:34.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.305 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91d010d480 con 0x7f91d00a90c0 2021-04-09T18:22:34.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.305 7f91c67fc700 1 -- 172.21.15.32:0/699974696 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+1849 (secure 0 0 0) 0x7f91cc00be10 con 0x7f91d00a90c0 2021-04-09T18:22:34.360 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 122 2021-04-09T18:22:34.360 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:34.361 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":122,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7657,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3983752195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3983752195},{"type":"v1","addr":"172.21.15.86:6837","nonce":3983752195}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":120},{"gid":7809,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/633563485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":633563485},{"type":"v1","addr":"172.21.15.32:6834","nonce":633563485}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":118},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":56624,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/24772624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":24772624},{"type":"v1","addr":"172.21.15.86:6835","nonce":24772624}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":122}],"filesystems":[]} 2021-04-09T18:22:34.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b8018930 msgr2=0x7f91b801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b8018930 0x7f91b801ae10 crc :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d00a90c0 msgr2=0x7f91d01a81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d00a90c0 0x7f91d01a81e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f91cc006d90 tx=0x7f91cc00b8c0).stop 2021-04-09T18:22:34.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 shutdown_connections 2021-04-09T18:22:34.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b8018930 0x7f91b801ae10 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d017bce0 0x7f91d017c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d01a76d0 0x7f91d01a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 --2- 172.21.15.32:0/699974696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d00a90c0 0x7f91d01a81e0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 >> 172.21.15.32:0/699974696 conn(0x7f91d01acae0 msgr2=0x7f91d017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:34.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 shutdown_connections 2021-04-09T18:22:34.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.308 7f91d7ad3700 1 -- 172.21.15.32:0/699974696 wait complete. 2021-04-09T18:22:34.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:34.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.467 7f09e485a700 1 Processor -- start 2021-04-09T18:22:34.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.467 7f09e485a700 1 -- start start 2021-04-09T18:22:34.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09e485a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f09e00d6c80 0x7f09e00d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09e485a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f09e00d2440 0x7f09e010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09e485a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e00d7280 con 0x7f09e00cffa0 2021-04-09T18:22:34.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09e485a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e00d7080 con 0x7f09e00d2440 2021-04-09T18:22:34.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09e485a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e00d91e0 con 0x7f09e00d6c80 2021-04-09T18:22:34.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dee1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f09e00d2440 0x7f09e010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dee1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f09e00d2440 0x7f09e010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49362/0 (socket says 172.21.15.32:49362) 2021-04-09T18:22:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dee1d700 1 -- 172.21.15.32:0/3251242378 learned_addr learned my addr 172.21.15.32:0/3251242378 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dde1b700 1 --2- 172.21.15.32:0/3251242378 >> v2:172.21.15.86:3300/0 conn(0x7f09e00d6c80 0x7f09e00d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dee1d700 1 -- 172.21.15.32:0/3251242378 >> v2:172.21.15.86:3300/0 conn(0x7f09e00d6c80 msgr2=0x7f09e00d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:34.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dee1d700 1 --2- 172.21.15.32:0/3251242378 >> v2:172.21.15.86:3300/0 conn(0x7f09e00d6c80 0x7f09e00d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dd61a700 1 -- 172.21.15.32:0/3251242378 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527397202 0 0) 0x7f09d8000fe0 con 0x7f09e00cffa0 2021-04-09T18:22:34.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dee1d700 1 -- 172.21.15.32:0/3251242378 >> v1:172.21.15.86:6789/0 conn(0x7f09e00cffa0 legacy=0x7f09e00d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dee1d700 1 -- 172.21.15.32:0/3251242378 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00d2840 con 0x7f09e00d2440 2021-04-09T18:22:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dde1b700 1 --2- 172.21.15.32:0/3251242378 >> v2:172.21.15.86:3300/0 conn(0x7f09e00d6c80 0x7f09e00d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dee1d700 1 --2- 172.21.15.32:0/3251242378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e00d2440 0x7f09e010cfd0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f09d0006b80 tx=0x7f09d0009b90).ready entity=mon.0 client_cookie=a91f08d84481d11a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dd61a700 1 -- 172.21.15.32:0/3251242378 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09d00022e0 con 0x7f09e00d2440 2021-04-09T18:22:34.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dd61a700 1 -- 172.21.15.32:0/3251242378 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d0002620 con 0x7f09e00d2440 2021-04-09T18:22:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.468 7f09dd61a700 1 -- 172.21.15.32:0/3251242378 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09d0000c10 con 0x7f09e00d2440 2021-04-09T18:22:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.469 7f09e485a700 1 -- 172.21.15.32:0/3251242378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e00d2440 msgr2=0x7f09e010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.469 7f09e485a700 1 --2- 172.21.15.32:0/3251242378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e00d2440 0x7f09e010cfd0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f09d0006b80 tx=0x7f09d0009b90).stop 2021-04-09T18:22:34.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.469 7f09e485a700 1 -- 172.21.15.32:0/3251242378 shutdown_connections 2021-04-09T18:22:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.469 7f09e485a700 1 --2- 172.21.15.32:0/3251242378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e00d2440 0x7f09e010cfd0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.469 7f09e485a700 1 --2- 172.21.15.32:0/3251242378 >> v2:172.21.15.86:3300/0 conn(0x7f09e00d6c80 0x7f09e00d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.469 7f09e485a700 1 -- 172.21.15.32:0/3251242378 >> 172.21.15.32:0/3251242378 conn(0x7f09e00ce5c0 msgr2=0x7f09e00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.469 7f09e485a700 1 -- 172.21.15.32:0/3251242378 shutdown_connections 2021-04-09T18:22:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.469 7f09e485a700 1 -- 172.21.15.32:0/3251242378 wait complete. 2021-04-09T18:22:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.489 7f09e485a700 1 Processor -- start 2021-04-09T18:22:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09e485a700 1 -- start start 2021-04-09T18:22:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09e485a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e00a80c0 0x7f09e017bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09e485a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e017b1d0 0x7f09e017b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09e485a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e008dc50 0x7f09e008e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09e485a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e017bc70 con 0x7f09e008dc50 2021-04-09T18:22:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09e485a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e008e670 con 0x7f09e017b1d0 2021-04-09T18:22:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09e485a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e008e850 con 0x7f09e00a80c0 2021-04-09T18:22:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09dee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e008dc50 0x7f09e008e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09dee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e008dc50 0x7f09e008e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49370/0 (socket says 172.21.15.32:49370) 2021-04-09T18:22:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09dee1d700 1 -- 172.21.15.32:0/587548860 learned_addr learned my addr 172.21.15.32:0/587548860 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09dde1b700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e017b1d0 0x7f09e017b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.490 7f09de61c700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e00a80c0 0x7f09e017bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09dee1d700 1 -- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e00a80c0 msgr2=0x7f09e017bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09dee1d700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e00a80c0 0x7f09e017bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09dee1d700 1 -- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e017b1d0 msgr2=0x7f09e017b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09dee1d700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e017b1d0 0x7f09e017b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09dee1d700 1 -- 172.21.15.32:0/587548860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e008c660 con 0x7f09e008dc50 2021-04-09T18:22:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09dee1d700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e008dc50 0x7f09e008e050 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7f09d0006f10 tx=0x7f09d000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09d6ffd700 1 -- 172.21.15.32:0/587548860 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09d0000b90 con 0x7f09e008dc50 2021-04-09T18:22:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09d6ffd700 1 -- 172.21.15.32:0/587548860 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d000bd50 con 0x7f09e008dc50 2021-04-09T18:22:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09e485a700 1 -- 172.21.15.32:0/587548860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e008c660 con 0x7f09e008dc50 2021-04-09T18:22:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09d6ffd700 1 -- 172.21.15.32:0/587548860 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09d0002220 con 0x7f09e008dc50 2021-04-09T18:22:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.491 7f09e485a700 1 -- 172.21.15.32:0/587548860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e008c660 con 0x7f09e008dc50 2021-04-09T18:22:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.492 7f09d6ffd700 1 -- 172.21.15.32:0/587548860 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f09d0009100 con 0x7f09e008dc50 2021-04-09T18:22:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.492 7f09d6ffd700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09c40188c0 0x7f09c401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.492 7f09d6ffd700 1 -- 172.21.15.32:0/587548860 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f09d00324e0 con 0x7f09e008dc50 2021-04-09T18:22:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.492 7f09de61c700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09c40188c0 0x7f09c401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:34.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.492 7f09de61c700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09c40188c0 0x7f09c401adc0 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:34.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.512 7f09e485a700 1 -- 172.21.15.32:0/587548860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e008e850 con 0x7f09e008dc50 2021-04-09T18:22:34.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.515 7f09d6ffd700 1 -- 172.21.15.32:0/587548860 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f09d0000b90 con 0x7f09e008dc50 2021-04-09T18:22:34.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.707 7f09e485a700 1 -- 172.21.15.32:0/587548860 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09e008e850 con 0x7f09e008dc50 2021-04-09T18:22:34.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.707 7f09d6ffd700 1 -- 172.21.15.32:0/587548860 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+1849 (secure 0 0 0) 0x7f09d0009100 con 0x7f09e008dc50 2021-04-09T18:22:34.762 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 122 2021-04-09T18:22:34.762 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:34.763 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":122,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7657,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3983752195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3983752195},{"type":"v1","addr":"172.21.15.86:6837","nonce":3983752195}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":120},{"gid":7809,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/633563485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":633563485},{"type":"v1","addr":"172.21.15.32:6834","nonce":633563485}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":118},{"gid":56423,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2579974780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2579974780},{"type":"v1","addr":"172.21.15.32:6835","nonce":2579974780}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":105},{"gid":56624,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/24772624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":24772624},{"type":"v1","addr":"172.21.15.86:6835","nonce":24772624}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":122}],"filesystems":[]} 2021-04-09T18:22:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 -- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09c40188c0 msgr2=0x7f09c401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09c40188c0 0x7f09c401adc0 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 -- 172.21.15.32:0/587548860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e008dc50 msgr2=0x7f09e008e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e008dc50 0x7f09e008e050 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7f09d0006f10 tx=0x7f09d000b960).stop 2021-04-09T18:22:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 -- 172.21.15.32:0/587548860 shutdown_connections 2021-04-09T18:22:34.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f09c40188c0 0x7f09c401adc0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09e008dc50 0x7f09e008e050 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e017b1d0 0x7f09e017b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 --2- 172.21.15.32:0/587548860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e00a80c0 0x7f09e017bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:34.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 -- 172.21.15.32:0/587548860 >> 172.21.15.32:0/587548860 conn(0x7f09e0189b00 msgr2=0x7f09e01921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:34.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 -- 172.21.15.32:0/587548860 shutdown_connections 2021-04-09T18:22:34.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.710 7f09e485a700 1 -- 172.21.15.32:0/587548860 wait complete. 2021-04-09T18:22:34.775 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:22:34.775 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:22:34.775 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:22:34.776 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:22:34.776 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:22:34.776 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:22:34.777 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:22:34.777 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:22:34.777 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:22:34.777 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:22:34.778 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:22:34.778 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:22:34.778 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:22:34.779 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:22:34.779 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:22:34.780 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:22:34.785 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:22:34.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:22:34.787 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:22:34.788 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:22:34.790 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:22:34.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:22:34.792 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:22:34.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:22:34.827 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:22:34.828 DEBUG:teuthology.parallel:result is None 2021-04-09T18:22:34.828 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:22:34.829 DEBUG:teuthology.parallel:result is None 2021-04-09T18:22:34.832 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:22:34.832 DEBUG:teuthology.parallel:result is None 2021-04-09T18:22:34.833 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:22:34.834 DEBUG:teuthology.parallel:result is None 2021-04-09T18:22:34.834 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:22:34.973 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:22:34.974 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:22:34.974 INFO:tasks.ceph.mds.b.smithi086.stderr:2021-04-09 18:22:34.920 7fb9c5d9d700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2021-04-09T18:22:34.979 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:22:34.980 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:22:35.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.935 7ffa22a81700 1 Processor -- start 2021-04-09T18:22:35.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.935 7ffa22a81700 1 -- start start 2021-04-09T18:22:35.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa22a81700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c0cffb0 0x7ffa1c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa22a81700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ffa1c0d6c90 0x7ffa1c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa22a81700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c0d7290 con 0x7ffa1c0d7810 2021-04-09T18:22:35.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa22a81700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1c0d7090 con 0x7ffa1c0d6c90 2021-04-09T18:22:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa22a81700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c0d8590 con 0x7ffa1c0cffb0 2021-04-09T18:22:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa1bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffa1c0d7810 0x7ffa1c0d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58988/0 (socket says 172.21.15.32:58988) 2021-04-09T18:22:35.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa1bfff700 1 -- 172.21.15.32:0/4238368249 learned_addr learned my addr 172.21.15.32:0/4238368249 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:35.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa2089d700 1 --2- 172.21.15.32:0/4238368249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c0cffb0 0x7ffa1c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa2109e700 1 --2- 172.21.15.32:0/4238368249 >> v2:172.21.15.86:3300/0 conn(0x7ffa1c0d6c90 0x7ffa1c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa1b7fe700 1 -- 172.21.15.32:0/4238368249 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044070952 0 0) 0x7ffa0c000ce0 con 0x7ffa1c0d7810 2021-04-09T18:22:35.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa1b7fe700 1 -- 172.21.15.32:0/4238368249 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa08002fe0 con 0x7ffa1c0d7810 2021-04-09T18:22:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa2089d700 1 -- 172.21.15.32:0/4238368249 >> v2:172.21.15.86:3300/0 conn(0x7ffa1c0d6c90 msgr2=0x7ffa1c10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa2089d700 1 --2- 172.21.15.32:0/4238368249 >> v2:172.21.15.86:3300/0 conn(0x7ffa1c0d6c90 0x7ffa1c10cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa2089d700 1 -- 172.21.15.32:0/4238368249 >> v1:172.21.15.32:6789/0 conn(0x7ffa1c0d7810 legacy=0x7ffa1c0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa2089d700 1 -- 172.21.15.32:0/4238368249 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c0d91f0 con 0x7ffa1c0cffb0 2021-04-09T18:22:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.936 7ffa2109e700 1 --2- 172.21.15.32:0/4238368249 >> v2:172.21.15.86:3300/0 conn(0x7ffa1c0d6c90 0x7ffa1c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:35.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.937 7ffa2089d700 1 --2- 172.21.15.32:0/4238368249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c0cffb0 0x7ffa1c0d2450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ffa04006650 tx=0x7ffa04009a80).ready entity=mon.2 client_cookie=8c02e64bf4b1d121 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.937 7ffa1b7fe700 1 -- 172.21.15.32:0/4238368249 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa040009b0 con 0x7ffa1c0cffb0 2021-04-09T18:22:35.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.937 7ffa1b7fe700 1 -- 172.21.15.32:0/4238368249 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0400c0c0 con 0x7ffa1c0cffb0 2021-04-09T18:22:35.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.937 7ffa1b7fe700 1 -- 172.21.15.32:0/4238368249 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa0400c510 con 0x7ffa1c0cffb0 2021-04-09T18:22:35.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.937 7ffa22a81700 1 -- 172.21.15.32:0/4238368249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c0cffb0 msgr2=0x7ffa1c0d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.937 7ffa22a81700 1 --2- 172.21.15.32:0/4238368249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c0cffb0 0x7ffa1c0d2450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ffa04006650 tx=0x7ffa04009a80).stop 2021-04-09T18:22:35.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.938 7ffa22a81700 1 -- 172.21.15.32:0/4238368249 shutdown_connections 2021-04-09T18:22:35.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.938 7ffa22a81700 1 --2- 172.21.15.32:0/4238368249 >> v2:172.21.15.86:3300/0 conn(0x7ffa1c0d6c90 0x7ffa1c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.938 7ffa22a81700 1 --2- 172.21.15.32:0/4238368249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c0cffb0 0x7ffa1c0d2450 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.938 7ffa22a81700 1 -- 172.21.15.32:0/4238368249 >> 172.21.15.32:0/4238368249 conn(0x7ffa1c0ce5d0 msgr2=0x7ffa1c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:35.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.938 7ffa22a81700 1 -- 172.21.15.32:0/4238368249 shutdown_connections 2021-04-09T18:22:35.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.938 7ffa22a81700 1 -- 172.21.15.32:0/4238368249 wait complete. 2021-04-09T18:22:35.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.958 7ffa22a81700 1 Processor -- start 2021-04-09T18:22:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.958 7ffa22a81700 1 -- start start 2021-04-09T18:22:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa22a81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa1c1814c0 0x7ffa1c092d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa22a81700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c093390 0x7ffa1c0abbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa22a81700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa1c0ac200 0x7ffa1c1a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa22a81700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c093790 con 0x7ffa1c1814c0 2021-04-09T18:22:35.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa22a81700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c1818c0 con 0x7ffa1c0ac200 2021-04-09T18:22:35.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa22a81700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c0ac600 con 0x7ffa1c093390 2021-04-09T18:22:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2089d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa1c1814c0 0x7ffa1c092d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2089d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa1c1814c0 0x7ffa1c092d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49404/0 (socket says 172.21.15.32:49404) 2021-04-09T18:22:35.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2089d700 1 -- 172.21.15.32:0/632956504 learned_addr learned my addr 172.21.15.32:0/632956504 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:35.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa1bfff700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c093390 0x7ffa1c0abbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2109e700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa1c0ac200 0x7ffa1c1a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2089d700 1 -- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c093390 msgr2=0x7ffa1c0abbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2089d700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c093390 0x7ffa1c0abbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2089d700 1 -- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa1c0ac200 msgr2=0x7ffa1c1a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2089d700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa1c0ac200 0x7ffa1c1a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2089d700 1 -- 172.21.15.32:0/632956504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c0ac820 con 0x7ffa1c1814c0 2021-04-09T18:22:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2109e700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa1c0ac200 0x7ffa1c1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa2089d700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa1c1814c0 0x7ffa1c092d70 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7ffa04009630 tx=0x7ffa0400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa197fa700 1 -- 172.21.15.32:0/632956504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa040093a0 con 0x7ffa1c1814c0 2021-04-09T18:22:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa197fa700 1 -- 172.21.15.32:0/632956504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0400bd70 con 0x7ffa1c1814c0 2021-04-09T18:22:35.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa197fa700 1 -- 172.21.15.32:0/632956504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa04005560 con 0x7ffa1c1814c0 2021-04-09T18:22:35.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa22a81700 1 -- 172.21.15.32:0/632956504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1c0ac820 con 0x7ffa1c1814c0 2021-04-09T18:22:35.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.959 7ffa22a81700 1 -- 172.21.15.32:0/632956504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1c0ac820 con 0x7ffa1c1814c0 2021-04-09T18:22:35.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.960 7ffa197fa700 1 -- 172.21.15.32:0/632956504 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ffa04005700 con 0x7ffa1c1814c0 2021-04-09T18:22:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.960 7ffa197fa700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa00018930 0x7ffa0001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.960 7ffa197fa700 1 -- 172.21.15.32:0/632956504 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 4843+0+0 (secure 0 0 0) 0x7ffa040326f0 con 0x7ffa1c1814c0 2021-04-09T18:22:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.960 7ffa1bfff700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa00018930 0x7ffa0001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.961 7ffa1bfff700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa00018930 0x7ffa0001ae10 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:35.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.979 7ffa22a81700 1 -- 172.21.15.32:0/632956504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1c093790 con 0x7ffa1c1814c0 2021-04-09T18:22:35.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:34.983 7ffa197fa700 1 -- 172.21.15.32:0/632956504 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ffa040054f0 con 0x7ffa1c1814c0 2021-04-09T18:22:35.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.186 7ffa22a81700 1 -- 172.21.15.32:0/632956504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7ffa1c093790 con 0x7ffa1c1814c0 2021-04-09T18:22:35.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.654 7ffa197fa700 1 -- 172.21.15.32:0/632956504 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v99) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffa04005700 con 0x7ffa1c1814c0 2021-04-09T18:22:35.754 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:22:35.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 -- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa00018930 msgr2=0x7ffa0001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa00018930 0x7ffa0001ae10 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 -- 172.21.15.32:0/632956504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa1c1814c0 msgr2=0x7ffa1c092d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa1c1814c0 0x7ffa1c092d70 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7ffa04009630 tx=0x7ffa0400b980).stop 2021-04-09T18:22:35.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 -- 172.21.15.32:0/632956504 shutdown_connections 2021-04-09T18:22:35.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa00018930 0x7ffa0001ae10 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa1c0ac200 0x7ffa1c1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa1c093390 0x7ffa1c0abbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 --2- 172.21.15.32:0/632956504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa1c1814c0 0x7ffa1c092d70 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 -- 172.21.15.32:0/632956504 >> 172.21.15.32:0/632956504 conn(0x7ffa1c198770 msgr2=0x7ffa1c1ac0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:35.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 -- 172.21.15.32:0/632956504 shutdown_connections 2021-04-09T18:22:35.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.703 7ffa22a81700 1 -- 172.21.15.32:0/632956504 wait complete. 2021-04-09T18:22:35.767 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:22:35.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:22:35.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.863 7fbc6fd59700 1 Processor -- start 2021-04-09T18:22:35.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.863 7fbc6fd59700 1 -- start start 2021-04-09T18:22:35.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.864 7fbc6fd59700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc6810d010 0x7fbc680d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.864 7fbc6fd59700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680d1da0 0x7fbc680d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.864 7fbc6fd59700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6810d480 con 0x7fbc680d82c0 2021-04-09T18:22:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.864 7fbc6fd59700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc680d9220 con 0x7fbc6810d010 2021-04-09T18:22:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.864 7fbc6fd59700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc680d86c0 con 0x7fbc680d1da0 2021-04-09T18:22:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.864 7fbc6db75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc6810d010 0x7fbc680d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.864 7fbc6db75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc6810d010 0x7fbc680d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49412/0 (socket says 172.21.15.32:49412) 2021-04-09T18:22:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.864 7fbc6db75700 1 -- 172.21.15.32:0/2547132776 learned_addr learned my addr 172.21.15.32:0/2547132776 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.864 7fbc6d374700 1 --2- 172.21.15.32:0/2547132776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680d1da0 0x7fbc680d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6db75700 1 -- 172.21.15.32:0/2547132776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680d1da0 msgr2=0x7fbc680d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6db75700 1 --2- 172.21.15.32:0/2547132776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680d1da0 0x7fbc680d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6db75700 1 -- 172.21.15.32:0/2547132776 >> v1:172.21.15.86:6789/0 conn(0x7fbc680d82c0 legacy=0x7fbc6810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6cb73700 1 -- 172.21.15.32:0/2547132776 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318160086 0 0) 0x7fbc60000d30 con 0x7fbc680d82c0 2021-04-09T18:22:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6db75700 1 -- 172.21.15.32:0/2547132776 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6810e1f0 con 0x7fbc6810d010 2021-04-09T18:22:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6db75700 1 --2- 172.21.15.32:0/2547132776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc6810d010 0x7fbc680d1780 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7fbc64006700 tx=0x7fbc64009b80).ready entity=mon.0 client_cookie=c412f4eb60d2668a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6cb73700 1 -- 172.21.15.32:0/2547132776 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc6400c0c0 con 0x7fbc6810d010 2021-04-09T18:22:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6cb73700 1 -- 172.21.15.32:0/2547132776 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6400c440 con 0x7fbc6810d010 2021-04-09T18:22:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6cb73700 1 -- 172.21.15.32:0/2547132776 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc640009b0 con 0x7fbc6810d010 2021-04-09T18:22:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6fd59700 1 -- 172.21.15.32:0/2547132776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc6810d010 msgr2=0x7fbc680d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.865 7fbc6fd59700 1 --2- 172.21.15.32:0/2547132776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc6810d010 0x7fbc680d1780 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7fbc64006700 tx=0x7fbc64009b80).stop 2021-04-09T18:22:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.866 7fbc6fd59700 1 -- 172.21.15.32:0/2547132776 shutdown_connections 2021-04-09T18:22:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.866 7fbc6fd59700 1 --2- 172.21.15.32:0/2547132776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680d1da0 0x7fbc680d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.866 7fbc6fd59700 1 --2- 172.21.15.32:0/2547132776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc6810d010 0x7fbc680d1780 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.866 7fbc6fd59700 1 -- 172.21.15.32:0/2547132776 >> 172.21.15.32:0/2547132776 conn(0x7fbc680ce5f0 msgr2=0x7fbc680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.866 7fbc6fd59700 1 -- 172.21.15.32:0/2547132776 shutdown_connections 2021-04-09T18:22:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.866 7fbc6fd59700 1 -- 172.21.15.32:0/2547132776 wait complete. 2021-04-09T18:22:35.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.886 7fbc6fd59700 1 Processor -- start 2021-04-09T18:22:35.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.886 7fbc6fd59700 1 -- start start 2021-04-09T18:22:35.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6fd59700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc68186480 0x7fbc6808c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6fd59700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68186880 0x7fbc6808c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6fd59700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc6808d680 0x7fbc6808da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6fd59700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6808bc60 con 0x7fbc68186480 2021-04-09T18:22:35.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6fd59700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6808e0a0 con 0x7fbc68186880 2021-04-09T18:22:35.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6fd59700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6808e2c0 con 0x7fbc6808d680 2021-04-09T18:22:35.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6db75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc68186480 0x7fbc6808c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6db75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc68186480 0x7fbc6808c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49418/0 (socket says 172.21.15.32:49418) 2021-04-09T18:22:35.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6db75700 1 -- 172.21.15.32:0/1323265325 learned_addr learned my addr 172.21.15.32:0/1323265325 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:35.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6e376700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc6808d680 0x7fbc6808da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6d374700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68186880 0x7fbc6808c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6db75700 1 -- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc6808d680 msgr2=0x7fbc6808da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6db75700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc6808d680 0x7fbc6808da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6db75700 1 -- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68186880 msgr2=0x7fbc6808c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:35.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6db75700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68186880 0x7fbc6808c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:35.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6db75700 1 -- 172.21.15.32:0/1323265325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6808e6e0 con 0x7fbc68186480 2021-04-09T18:22:35.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6db75700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc68186480 0x7fbc6808c1e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7fbc6400b650 tx=0x7fbc6400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:35.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc5a7fc700 1 -- 172.21.15.32:0/1323265325 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc6400bde0 con 0x7fbc68186480 2021-04-09T18:22:35.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc5a7fc700 1 -- 172.21.15.32:0/1323265325 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc64006570 con 0x7fbc68186480 2021-04-09T18:22:35.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6808e6e0 con 0x7fbc68186480 2021-04-09T18:22:35.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.887 7fbc5a7fc700 1 -- 172.21.15.32:0/1323265325 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc640022b0 con 0x7fbc68186480 2021-04-09T18:22:35.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.888 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6808e6e0 con 0x7fbc68186480 2021-04-09T18:22:35.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.888 7fbc5a7fc700 1 -- 172.21.15.32:0/1323265325 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbc64002020 con 0x7fbc68186480 2021-04-09T18:22:35.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.888 7fbc5a7fc700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc50018930 0x7fbc5001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:35.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.888 7fbc5a7fc700 1 -- 172.21.15.32:0/1323265325 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fbc640326f0 con 0x7fbc68186480 2021-04-09T18:22:35.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.888 7fbc6d374700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc50018930 0x7fbc5001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:35.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.889 7fbc6d374700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc50018930 0x7fbc5001ae10 crc :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:35.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.908 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc68186c80 con 0x7fbc68186480 2021-04-09T18:22:35.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:35.912 7fbc5a7fc700 1 -- 172.21.15.32:0/1323265325 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbc64000c90 con 0x7fbc68186480 2021-04-09T18:22:36.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.097 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbc68186c80 con 0x7fbc68186480 2021-04-09T18:22:36.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.097 7fbc5a7fc700 1 -- 172.21.15.32:0/1323265325 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v27) v1 ==== 80+0+4539 (secure 0 0 0) 0x7fbc6400bde0 con 0x7fbc68186480 2021-04-09T18:22:36.151 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:36.151 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:22:36.152 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:22:36.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.152 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:22:36.152 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:22:36.152 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.153 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:22:36.153 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.153 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.153 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.154 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:22:36.154 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:22:36.155 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.155 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:22:36.155 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.155 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.156 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.156 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.156 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:22:36.157 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:22:36.157 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.157 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:22:36.157 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.157 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.158 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.158 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.159 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:22:36.159 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:22:36.159 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.159 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:22:36.159 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.160 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.160 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.161 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:22:36.161 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:22:36.161 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.161 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.162 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.162 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.162 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.163 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:22:36.163 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:22:36.163 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.164 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.164 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.164 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.165 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:22:36.165 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:22:36.165 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.166 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.166 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.166 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.166 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.167 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:22:36.167 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:22:36.167 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.168 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.168 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.168 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.168 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.169 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:22:36.169 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:22:36.169 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.170 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.170 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.170 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.170 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.171 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:22:36.171 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:22:36.171 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.172 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.172 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.172 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.173 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:22:36.173 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:22:36.173 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.174 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.174 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.175 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:22:36.175 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:22:36.176 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.176 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.177 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.177 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:22:36.178 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:22:36.178 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:22:36.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:22:36.179 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:22:36.179 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.180 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:22:36.180 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:22:36.180 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:22:36.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:22:36.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:22:36.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.182 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:22:36.182 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:22:36.183 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:22:36.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:22:36.183 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:22:36.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.184 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.185 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:22:36.185 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:22:36.185 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.185 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:22:36.186 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:22:36.186 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.186 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:36.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:36.187 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:22:36.187 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:22:36.187 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:22:36.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:22:36.188 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:22:36.188 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:22:36.188 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.189 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:36.189 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:22:36.189 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:22:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc50018930 msgr2=0x7fbc5001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc50018930 0x7fbc5001ae10 crc :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc68186480 msgr2=0x7fbc6808c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc68186480 0x7fbc6808c1e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7fbc6400b650 tx=0x7fbc6400b9f0).stop 2021-04-09T18:22:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 shutdown_connections 2021-04-09T18:22:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc50018930 0x7fbc5001ae10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc6808d680 0x7fbc6808da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68186880 0x7fbc6808c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 --2- 172.21.15.32:0/1323265325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc68186480 0x7fbc6808c1e0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 >> 172.21.15.32:0/1323265325 conn(0x7fbc68195e70 msgr2=0x7fbc681a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 shutdown_connections 2021-04-09T18:22:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.100 7fbc6fd59700 1 -- 172.21.15.32:0/1323265325 wait complete. 2021-04-09T18:22:36.194 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:22:36.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.289 7f754d130700 1 Processor -- start 2021-04-09T18:22:36.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.289 7f754d130700 1 -- start start 2021-04-09T18:22:36.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754d130700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75480d7ca0 0x7f75480d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754d130700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754810da00 0x7f754810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754d130700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754810d480 con 0x7f754810d010 2021-04-09T18:22:36.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754d130700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75480d9220 con 0x7f75480d7ca0 2021-04-09T18:22:36.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754d130700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75480d86c0 con 0x7f754810da00 2021-04-09T18:22:36.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754659c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75480d7ca0 0x7f75480d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f7546d9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f754810d010 0x7f75480d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59016/0 (socket says 172.21.15.32:59016) 2021-04-09T18:22:36.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f7546d9d700 1 -- 172.21.15.32:0/3079823973 learned_addr learned my addr 172.21.15.32:0/3079823973 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:36.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754759e700 1 --2- 172.21.15.32:0/3079823973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754810da00 0x7f754810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f7545d9b700 1 -- 172.21.15.32:0/3079823973 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906871745 0 0) 0x7f7538000ce0 con 0x7f754810d010 2021-04-09T18:22:36.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f7545d9b700 1 -- 172.21.15.32:0/3079823973 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7530002fe0 con 0x7f754810d010 2021-04-09T18:22:36.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754659c700 1 -- 172.21.15.32:0/3079823973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754810da00 msgr2=0x7f754810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:36.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754659c700 1 --2- 172.21.15.32:0/3079823973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754810da00 0x7f754810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754659c700 1 -- 172.21.15.32:0/3079823973 >> v1:172.21.15.32:6789/0 conn(0x7f754810d010 legacy=0x7f75480d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.290 7f754659c700 1 -- 172.21.15.32:0/3079823973 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75480d1f70 con 0x7f75480d7ca0 2021-04-09T18:22:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.291 7f754659c700 1 --2- 172.21.15.32:0/3079823973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75480d7ca0 0x7f75480d80a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f753c0069d0 tx=0x7f753c009e40).ready entity=mon.0 client_cookie=762626ecacf448f2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.291 7f7545d9b700 1 -- 172.21.15.32:0/3079823973 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f753c00b510 con 0x7f75480d7ca0 2021-04-09T18:22:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.291 7f7545d9b700 1 -- 172.21.15.32:0/3079823973 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c00b890 con 0x7f75480d7ca0 2021-04-09T18:22:36.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.291 7f7545d9b700 1 -- 172.21.15.32:0/3079823973 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f753c00bd70 con 0x7f75480d7ca0 2021-04-09T18:22:36.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.291 7f754d130700 1 -- 172.21.15.32:0/3079823973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75480d7ca0 msgr2=0x7f75480d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.291 7f754d130700 1 --2- 172.21.15.32:0/3079823973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75480d7ca0 0x7f75480d80a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f753c0069d0 tx=0x7f753c009e40).stop 2021-04-09T18:22:36.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.292 7f754d130700 1 -- 172.21.15.32:0/3079823973 shutdown_connections 2021-04-09T18:22:36.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.292 7f754d130700 1 --2- 172.21.15.32:0/3079823973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754810da00 0x7f754810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.292 7f754d130700 1 --2- 172.21.15.32:0/3079823973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75480d7ca0 0x7f75480d80a0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.292 7f754d130700 1 -- 172.21.15.32:0/3079823973 >> 172.21.15.32:0/3079823973 conn(0x7f75480ce5f0 msgr2=0x7f75480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:36.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.292 7f754d130700 1 -- 172.21.15.32:0/3079823973 shutdown_connections 2021-04-09T18:22:36.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.292 7f754d130700 1 -- 172.21.15.32:0/3079823973 wait complete. 2021-04-09T18:22:36.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754d130700 1 Processor -- start 2021-04-09T18:22:36.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754d130700 1 -- start start 2021-04-09T18:22:36.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754d130700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75481aca20 0x7f75481ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754d130700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f754818bd40 0x7f754818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754d130700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754808fba0 0x7f754808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754d130700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754818c7e0 con 0x7f754818bd40 2021-04-09T18:22:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754d130700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754818ca30 con 0x7f75481aca20 2021-04-09T18:22:36.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754d130700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75480905f0 con 0x7f754808fba0 2021-04-09T18:22:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f754818bd40 0x7f754818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f754818bd40 0x7f754818c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49434/0 (socket says 172.21.15.32:49434) 2021-04-09T18:22:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.312 7f754659c700 1 -- 172.21.15.32:0/897163695 learned_addr learned my addr 172.21.15.32:0/897163695 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:36.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f754759e700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754808fba0 0x7f754808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f7546d9d700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75481aca20 0x7f75481ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f754659c700 1 -- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754808fba0 msgr2=0x7f754808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f754659c700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754808fba0 0x7f754808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f754659c700 1 -- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75481aca20 msgr2=0x7f75481ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f754659c700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75481aca20 0x7f75481ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f754659c700 1 -- 172.21.15.32:0/897163695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75480b4300 con 0x7f754818bd40 2021-04-09T18:22:36.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f754659c700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f754818bd40 0x7f754818c140 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f753c009ee0 tx=0x7f753c00bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:36.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f75377fe700 1 -- 172.21.15.32:0/897163695 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f753c00e500 con 0x7f754818bd40 2021-04-09T18:22:36.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f75377fe700 1 -- 172.21.15.32:0/897163695 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c009020 con 0x7f754818bd40 2021-04-09T18:22:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f754d130700 1 -- 172.21.15.32:0/897163695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75480b4300 con 0x7f754818bd40 2021-04-09T18:22:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f75377fe700 1 -- 172.21.15.32:0/897163695 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f753c00ea80 con 0x7f754818bd40 2021-04-09T18:22:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.313 7f754d130700 1 -- 172.21.15.32:0/897163695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75480b4300 con 0x7f754818bd40 2021-04-09T18:22:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.314 7f75377fe700 1 -- 172.21.15.32:0/897163695 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f753c006020 con 0x7f754818bd40 2021-04-09T18:22:36.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.314 7f75377fe700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f752c018930 0x7f752c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.314 7f75377fe700 1 -- 172.21.15.32:0/897163695 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f753c034c60 con 0x7f754818bd40 2021-04-09T18:22:36.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.314 7f7546d9d700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f752c018930 0x7f752c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.314 7f7546d9d700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f752c018930 0x7f752c01ae10 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:36.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.334 7f754d130700 1 -- 172.21.15.32:0/897163695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7548182d70 con 0x7f754818bd40 2021-04-09T18:22:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.337 7f75377fe700 1 -- 172.21.15.32:0/897163695 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f753c00bdb0 con 0x7f754818bd40 2021-04-09T18:22:36.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:22:36.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:22:36.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.527 7f754d130700 1 -- 172.21.15.32:0/897163695 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f754818c760 con 0x7f754818bd40 2021-04-09T18:22:36.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.528 7f75377fe700 1 -- 172.21.15.32:0/897163695 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f753c006000 con 0x7f754818bd40 2021-04-09T18:22:36.582 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:36.583 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":99,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:22:35.651588","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:22:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 -- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f752c018930 msgr2=0x7f752c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f752c018930 0x7f752c01ae10 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 -- 172.21.15.32:0/897163695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f754818bd40 msgr2=0x7f754818c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f754818bd40 0x7f754818c140 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f753c009ee0 tx=0x7f753c00bc70).stop 2021-04-09T18:22:36.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 -- 172.21.15.32:0/897163695 shutdown_connections 2021-04-09T18:22:36.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f752c018930 0x7f752c01ae10 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f754818bd40 0x7f754818c140 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f754808fba0 0x7f754808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 --2- 172.21.15.32:0/897163695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75481aca20 0x7f75481ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 -- 172.21.15.32:0/897163695 >> 172.21.15.32:0/897163695 conn(0x7f754819b460 msgr2=0x7f754819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:36.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 -- 172.21.15.32:0/897163695 shutdown_connections 2021-04-09T18:22:36.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.530 7f754d130700 1 -- 172.21.15.32:0/897163695 wait complete. 2021-04-09T18:22:36.595 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:22:36.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:22:36.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.694 7f5298719700 1 Processor -- start 2021-04-09T18:22:36.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.694 7f5298719700 1 -- start start 2021-04-09T18:22:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.695 7f5298719700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f529010d010 0x7f52900d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.695 7f5298719700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900d8490 0x7f529010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.695 7f5298719700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529010d480 con 0x7f52900d1da0 2021-04-09T18:22:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.695 7f5298719700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52900d8d30 con 0x7f529010d010 2021-04-09T18:22:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.695 7f5298719700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52900d9220 con 0x7f52900d8490 2021-04-09T18:22:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.695 7f5296535700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f529010d010 0x7f52900d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.695 7f5295d34700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f52900d1da0 0x7f52900d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59034/0 (socket says 172.21.15.32:59034) 2021-04-09T18:22:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.695 7f5295d34700 1 -- 172.21.15.32:0/3055218986 learned_addr learned my addr 172.21.15.32:0/3055218986 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.695 7f5296d36700 1 --2- 172.21.15.32:0/3055218986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900d8490 0x7f529010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5296535700 1 -- 172.21.15.32:0/3055218986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900d8490 msgr2=0x7f529010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5296535700 1 --2- 172.21.15.32:0/3055218986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900d8490 0x7f529010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5296535700 1 -- 172.21.15.32:0/3055218986 >> v1:172.21.15.32:6789/0 conn(0x7f52900d1da0 legacy=0x7f52900d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5296535700 1 -- 172.21.15.32:0/3055218986 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529010e020 con 0x7f529010d010 2021-04-09T18:22:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5295533700 1 -- 172.21.15.32:0/3055218986 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193644049 0 0) 0x7f52800009b0 con 0x7f52900d1da0 2021-04-09T18:22:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5296d36700 1 --2- 172.21.15.32:0/3055218986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900d8490 0x7f529010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5296535700 1 --2- 172.21.15.32:0/3055218986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f529010d010 0x7f52900d1780 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f528c006650 tx=0x7f528c009a80).ready entity=mon.0 client_cookie=fe2c02737dfda6bd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5295533700 1 -- 172.21.15.32:0/3055218986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f528c0009b0 con 0x7f529010d010 2021-04-09T18:22:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5295533700 1 -- 172.21.15.32:0/3055218986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c00c0c0 con 0x7f529010d010 2021-04-09T18:22:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5295533700 1 -- 172.21.15.32:0/3055218986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f528c00c510 con 0x7f529010d010 2021-04-09T18:22:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5298719700 1 -- 172.21.15.32:0/3055218986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f529010d010 msgr2=0x7f52900d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.696 7f5298719700 1 --2- 172.21.15.32:0/3055218986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f529010d010 0x7f52900d1780 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f528c006650 tx=0x7f528c009a80).stop 2021-04-09T18:22:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.697 7f5298719700 1 -- 172.21.15.32:0/3055218986 shutdown_connections 2021-04-09T18:22:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.697 7f5298719700 1 --2- 172.21.15.32:0/3055218986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900d8490 0x7f529010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.697 7f5298719700 1 --2- 172.21.15.32:0/3055218986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f529010d010 0x7f52900d1780 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.697 7f5298719700 1 -- 172.21.15.32:0/3055218986 >> 172.21.15.32:0/3055218986 conn(0x7f52900ce5f0 msgr2=0x7f52900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:36.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.697 7f5298719700 1 -- 172.21.15.32:0/3055218986 shutdown_connections 2021-04-09T18:22:36.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.697 7f5298719700 1 -- 172.21.15.32:0/3055218986 wait complete. 2021-04-09T18:22:36.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.717 7f5298719700 1 Processor -- start 2021-04-09T18:22:36.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.717 7f5298719700 1 -- start start 2021-04-09T18:22:36.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5298719700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900c3c00 0x7f529009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5298719700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290095240 0x7f52901a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5298719700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52901a1270 0x7f52900b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5298719700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529009cda0 con 0x7f52901a1270 2021-04-09T18:22:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5298719700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52900c3680 con 0x7f5290095240 2021-04-09T18:22:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5298719700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5290095640 con 0x7f52900c3c00 2021-04-09T18:22:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296d36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52901a1270 0x7f52900b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296d36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52901a1270 0x7f52900b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49448/0 (socket says 172.21.15.32:49448) 2021-04-09T18:22:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296d36700 1 -- 172.21.15.32:0/2632060865 learned_addr learned my addr 172.21.15.32:0/2632060865 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296535700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900c3c00 0x7f529009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5295d34700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290095240 0x7f52901a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296d36700 1 -- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900c3c00 msgr2=0x7f529009c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296d36700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900c3c00 0x7f529009c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296d36700 1 -- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290095240 msgr2=0x7f52901a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296d36700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290095240 0x7f52901a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296d36700 1 -- 172.21.15.32:0/2632060865 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52901a1670 con 0x7f52901a1270 2021-04-09T18:22:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5296d36700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52901a1270 0x7f52900b0cf0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f528800b970 tx=0x7f528800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5286ffd700 1 -- 172.21.15.32:0/2632060865 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5288000f90 con 0x7f52901a1270 2021-04-09T18:22:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5286ffd700 1 -- 172.21.15.32:0/2632060865 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5288006aa0 con 0x7f52901a1270 2021-04-09T18:22:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5286ffd700 1 -- 172.21.15.32:0/2632060865 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52880015f0 con 0x7f52901a1270 2021-04-09T18:22:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5298719700 1 -- 172.21.15.32:0/2632060865 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52901a1670 con 0x7f52901a1270 2021-04-09T18:22:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.718 7f5298719700 1 -- 172.21.15.32:0/2632060865 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52901a1670 con 0x7f52901a1270 2021-04-09T18:22:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.719 7f5286ffd700 1 -- 172.21.15.32:0/2632060865 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5288000ed0 con 0x7f52901a1270 2021-04-09T18:22:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.719 7f5286ffd700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52780188d0 0x7f527801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.719 7f5286ffd700 1 -- 172.21.15.32:0/2632060865 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f5288032a70 con 0x7f52901a1270 2021-04-09T18:22:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.719 7f5296535700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52780188d0 0x7f527801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.720 7f5296535700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52780188d0 0x7f527801add0 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:36.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.739 7f5298719700 1 -- 172.21.15.32:0/2632060865 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5290093d50 con 0x7f52901a1270 2021-04-09T18:22:36.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.742 7f5286ffd700 1 -- 172.21.15.32:0/2632060865 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f52880066f0 con 0x7f52901a1270 2021-04-09T18:22:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.922 7f5298719700 1 -- 172.21.15.32:0/2632060865 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f529010d480 con 0x7f52901a1270 2021-04-09T18:22:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.923 7f5286ffd700 1 -- 172.21.15.32:0/2632060865 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v99) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5288006aa0 con 0x7f52901a1270 2021-04-09T18:22:36.977 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:22:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 -- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52780188d0 msgr2=0x7f527801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52780188d0 0x7f527801add0 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 -- 172.21.15.32:0/2632060865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52901a1270 msgr2=0x7f52900b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52901a1270 0x7f52900b0cf0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f528800b970 tx=0x7f528800bd10).stop 2021-04-09T18:22:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 -- 172.21.15.32:0/2632060865 shutdown_connections 2021-04-09T18:22:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52780188d0 0x7f527801add0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52901a1270 0x7f52900b0cf0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290095240 0x7f52901a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 --2- 172.21.15.32:0/2632060865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52900c3c00 0x7f529009c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 -- 172.21.15.32:0/2632060865 >> 172.21.15.32:0/2632060865 conn(0x7f52900975f0 msgr2=0x7f5290096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 -- 172.21.15.32:0/2632060865 shutdown_connections 2021-04-09T18:22:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:36.925 7f5298719700 1 -- 172.21.15.32:0/2632060865 wait complete. 2021-04-09T18:22:36.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:22:37.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.083 7f19f1de4700 1 Processor -- start 2021-04-09T18:22:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.083 7f19f1de4700 1 -- start start 2021-04-09T18:22:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19f1de4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19ec0cffa0 0x7f19ec0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19f1de4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec0d7800 0x7f19ec0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19f1de4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19ec0d8220 0x7f19ec0d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19f1de4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0d91e0 con 0x7f19ec0d8220 2021-04-09T18:22:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19f1de4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0d7280 con 0x7f19ec0cffa0 2021-04-09T18:22:37.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19f1de4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0d6290 con 0x7f19ec0d7800 2021-04-09T18:22:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19ebfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19ec0d8220 0x7f19ec0d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19ebfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19ec0d8220 0x7f19ec0d5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49456/0 (socket says 172.21.15.32:49456) 2021-04-09T18:22:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19ebfff700 1 -- 172.21.15.32:0/1460477862 learned_addr learned my addr 172.21.15.32:0/1460477862 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:37.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19eaffd700 1 --2- 172.21.15.32:0/1460477862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec0d7800 0x7f19ec0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19eb7fe700 1 --2- 172.21.15.32:0/1460477862 >> v2:172.21.15.86:3300/0 conn(0x7f19ec0cffa0 0x7f19ec0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19ebfff700 1 -- 172.21.15.32:0/1460477862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec0d7800 msgr2=0x7f19ec0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.084 7f19ebfff700 1 --2- 172.21.15.32:0/1460477862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec0d7800 0x7f19ec0d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19ebfff700 1 -- 172.21.15.32:0/1460477862 >> v2:172.21.15.86:3300/0 conn(0x7f19ec0cffa0 msgr2=0x7f19ec0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:37.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19ebfff700 1 --2- 172.21.15.32:0/1460477862 >> v2:172.21.15.86:3300/0 conn(0x7f19ec0cffa0 0x7f19ec0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19ebfff700 1 -- 172.21.15.32:0/1460477862 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0d64b0 con 0x7f19ec0d8220 2021-04-09T18:22:37.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19eb7fe700 1 --2- 172.21.15.32:0/1460477862 >> v2:172.21.15.86:3300/0 conn(0x7f19ec0cffa0 0x7f19ec0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:37.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19ebfff700 1 --2- 172.21.15.32:0/1460477862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec0d8220 0x7f19ec0d5c70 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f19e4006f00 tx=0x7f19e4009e90).ready entity=mon.0 client_cookie=dc8787e619bbaed4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:37.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19ea7fc700 1 -- 172.21.15.32:0/1460477862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19e400b300 con 0x7f19ec0d8220 2021-04-09T18:22:37.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19ea7fc700 1 -- 172.21.15.32:0/1460477862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e400b640 con 0x7f19ec0d8220 2021-04-09T18:22:37.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19ea7fc700 1 -- 172.21.15.32:0/1460477862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19e400baa0 con 0x7f19ec0d8220 2021-04-09T18:22:37.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19f1de4700 1 -- 172.21.15.32:0/1460477862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec0d8220 msgr2=0x7f19ec0d5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.085 7f19f1de4700 1 --2- 172.21.15.32:0/1460477862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec0d8220 0x7f19ec0d5c70 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f19e4006f00 tx=0x7f19e4009e90).stop 2021-04-09T18:22:37.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.086 7f19f1de4700 1 -- 172.21.15.32:0/1460477862 shutdown_connections 2021-04-09T18:22:37.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.086 7f19f1de4700 1 --2- 172.21.15.32:0/1460477862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec0d8220 0x7f19ec0d5c70 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.086 7f19f1de4700 1 --2- 172.21.15.32:0/1460477862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec0d7800 0x7f19ec0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.086 7f19f1de4700 1 --2- 172.21.15.32:0/1460477862 >> v2:172.21.15.86:3300/0 conn(0x7f19ec0cffa0 0x7f19ec0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.086 7f19f1de4700 1 -- 172.21.15.32:0/1460477862 >> 172.21.15.32:0/1460477862 conn(0x7f19ec0ce5c0 msgr2=0x7f19ec0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:37.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.086 7f19f1de4700 1 -- 172.21.15.32:0/1460477862 shutdown_connections 2021-04-09T18:22:37.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.086 7f19f1de4700 1 -- 172.21.15.32:0/1460477862 wait complete. 2021-04-09T18:22:37.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19f1de4700 1 Processor -- start 2021-04-09T18:22:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19f1de4700 1 -- start start 2021-04-09T18:22:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19f1de4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19ec1acff0 0x7f19ec08fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19f1de4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec090150 0x7f19ec08c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19f1de4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec08cca0 0x7f19ec08e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19f1de4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec08c100 con 0x7f19ec090150 2021-04-09T18:22:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19f1de4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec08d0a0 con 0x7f19ec1acff0 2021-04-09T18:22:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19f1de4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec08e960 con 0x7f19ec08cca0 2021-04-09T18:22:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec090150 0x7f19ec08c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19eaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec090150 0x7f19ec08c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49462/0 (socket says 172.21.15.32:49462) 2021-04-09T18:22:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.107 7f19eaffd700 1 -- 172.21.15.32:0/399401945 learned_addr learned my addr 172.21.15.32:0/399401945 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19eb7fe700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19ec1acff0 0x7f19ec08fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19ebfff700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec08cca0 0x7f19ec08e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19eaffd700 1 -- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec08cca0 msgr2=0x7f19ec08e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19eaffd700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec08cca0 0x7f19ec08e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19eaffd700 1 -- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19ec1acff0 msgr2=0x7f19ec08fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19eaffd700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19ec1acff0 0x7f19ec08fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19eaffd700 1 -- 172.21.15.32:0/399401945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec08eb80 con 0x7f19ec090150 2021-04-09T18:22:37.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19eaffd700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec090150 0x7f19ec08c680 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7f19e000b560 tx=0x7f19e000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19d3fff700 1 -- 172.21.15.32:0/399401945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19e000bcf0 con 0x7f19ec090150 2021-04-09T18:22:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19d3fff700 1 -- 172.21.15.32:0/399401945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0000c90 con 0x7f19ec090150 2021-04-09T18:22:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19d3fff700 1 -- 172.21.15.32:0/399401945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19e0002220 con 0x7f19ec090150 2021-04-09T18:22:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19f1de4700 1 -- 172.21.15.32:0/399401945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec08eb80 con 0x7f19ec090150 2021-04-09T18:22:37.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.108 7f19f1de4700 1 -- 172.21.15.32:0/399401945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec08eb80 con 0x7f19ec090150 2021-04-09T18:22:37.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.109 7f19d3fff700 1 -- 172.21.15.32:0/399401945 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f19e00052e0 con 0x7f19ec090150 2021-04-09T18:22:37.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.109 7f19d3fff700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19cc018930 0x7f19cc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.109 7f19d3fff700 1 -- 172.21.15.32:0/399401945 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f19e00326f0 con 0x7f19ec090150 2021-04-09T18:22:37.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.109 7f19eb7fe700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19cc018930 0x7f19cc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.109 7f19eb7fe700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19cc018930 0x7f19cc01ae10 crc :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:37.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.128 7f19f1de4700 1 -- 172.21.15.32:0/399401945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ec182dc0 con 0x7f19ec090150 2021-04-09T18:22:37.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.131 7f19d3fff700 1 -- 172.21.15.32:0/399401945 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f19e0000c90 con 0x7f19ec090150 2021-04-09T18:22:37.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.307 7f19f1de4700 1 -- 172.21.15.32:0/399401945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f19ec0d7280 con 0x7f19ec090150 2021-04-09T18:22:37.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.307 7f19d3fff700 1 -- 172.21.15.32:0/399401945 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v99) v1 ==== 84+0+64 (secure 0 0 0) 0x7f19e0006b00 con 0x7f19ec090150 2021-04-09T18:22:37.362 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:37.362 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:22:37.363 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:22:37.363 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:22:37.363 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:22:37.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:37.364 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:22:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 -- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19cc018930 msgr2=0x7f19cc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19cc018930 0x7f19cc01ae10 crc :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 -- 172.21.15.32:0/399401945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec090150 msgr2=0x7f19ec08c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec090150 0x7f19ec08c680 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7f19e000b560 tx=0x7f19e000b900).stop 2021-04-09T18:22:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 -- 172.21.15.32:0/399401945 shutdown_connections 2021-04-09T18:22:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19cc018930 0x7f19cc01ae10 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19ec090150 0x7f19ec08c680 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19ec08cca0 0x7f19ec08e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 --2- 172.21.15.32:0/399401945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19ec1acff0 0x7f19ec08fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 -- 172.21.15.32:0/399401945 >> 172.21.15.32:0/399401945 conn(0x7f19ec09abb0 msgr2=0x7f19ec0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:37.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 -- 172.21.15.32:0/399401945 shutdown_connections 2021-04-09T18:22:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.310 7f19f1de4700 1 -- 172.21.15.32:0/399401945 wait complete. 2021-04-09T18:22:37.374 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:22:37.375 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:22:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.473 7f10bf0f6700 1 Processor -- start 2021-04-09T18:22:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.473 7f10bf0f6700 1 -- start start 2021-04-09T18:22:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.474 7f10bf0f6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f10b810da00 0x7f10b810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.474 7f10bf0f6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b810d480 con 0x7f10b80d7ca0 2021-04-09T18:22:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.474 7f10bf0f6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b80d1f70 con 0x7f10b810d010 2021-04-09T18:22:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.474 7f10bf0f6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b80d8d30 con 0x7f10b810da00 2021-04-09T18:22:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.474 7f10b7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f10b80d7ca0 0x7f10b80d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59058/0 (socket says 172.21.15.32:59058) 2021-04-09T18:22:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.474 7f10b7fff700 1 -- 172.21.15.32:0/1936647336 learned_addr learned my addr 172.21.15.32:0/1936647336 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.474 7f10bd713700 1 --2- 172.21.15.32:0/1936647336 >> v2:172.21.15.86:3300/0 conn(0x7f10b810da00 0x7f10b810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10b77fe700 1 -- 172.21.15.32:0/1936647336 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544858300 0 0) 0x7f10a8000ce0 con 0x7f10b80d7ca0 2021-04-09T18:22:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10b77fe700 1 -- 172.21.15.32:0/1936647336 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a4002fe0 con 0x7f10b80d7ca0 2021-04-09T18:22:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10b77fe700 1 -- 172.21.15.32:0/1936647336 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092278029 0 0) 0x7f10b0000fe0 con 0x7f10b810d010 2021-04-09T18:22:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10b77fe700 1 -- 172.21.15.32:0/1936647336 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a4003210 con 0x7f10b810d010 2021-04-09T18:22:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10bd713700 1 -- 172.21.15.32:0/1936647336 >> v1:172.21.15.86:6789/0 conn(0x7f10b810d010 legacy=0x7f10b80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10bd713700 1 -- 172.21.15.32:0/1936647336 >> v1:172.21.15.32:6789/0 conn(0x7f10b80d7ca0 legacy=0x7f10b80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10b77fe700 1 -- 172.21.15.32:0/1936647336 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1535688654 0 0) 0x7f10a80010e0 con 0x7f10b80d7ca0 2021-04-09T18:22:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10bd713700 1 -- 172.21.15.32:0/1936647336 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80d9220 con 0x7f10b810da00 2021-04-09T18:22:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10bd713700 1 --2- 172.21.15.32:0/1936647336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b810da00 0x7f10b810de00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f10a0006ad0 tx=0x7f10a0009a50).ready entity=mon.1 client_cookie=df1813d04f1ce119 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10b77fe700 1 -- 172.21.15.32:0/1936647336 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10a00066a0 con 0x7f10b810da00 2021-04-09T18:22:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10b77fe700 1 -- 172.21.15.32:0/1936647336 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a0006410 con 0x7f10b810da00 2021-04-09T18:22:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10b77fe700 1 -- 172.21.15.32:0/1936647336 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10a00024d0 con 0x7f10b810da00 2021-04-09T18:22:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10bf0f6700 1 -- 172.21.15.32:0/1936647336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b810da00 msgr2=0x7f10b810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.475 7f10bf0f6700 1 --2- 172.21.15.32:0/1936647336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b810da00 0x7f10b810de00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f10a0006ad0 tx=0x7f10a0009a50).stop 2021-04-09T18:22:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.476 7f10bf0f6700 1 -- 172.21.15.32:0/1936647336 shutdown_connections 2021-04-09T18:22:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.476 7f10bf0f6700 1 --2- 172.21.15.32:0/1936647336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b810da00 0x7f10b810de00 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.476 7f10bf0f6700 1 -- 172.21.15.32:0/1936647336 >> 172.21.15.32:0/1936647336 conn(0x7f10b80ce5f0 msgr2=0x7f10b80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.476 7f10bf0f6700 1 -- 172.21.15.32:0/1936647336 shutdown_connections 2021-04-09T18:22:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.476 7f10bf0f6700 1 -- 172.21.15.32:0/1936647336 wait complete. 2021-04-09T18:22:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.496 7f10bf0f6700 1 Processor -- start 2021-04-09T18:22:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.496 7f10bf0f6700 1 -- start start 2021-04-09T18:22:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10bf0f6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b8179610 0x7f10b8179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10bf0f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10b817a820 0x7f10b817ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10bf0f6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b81949f0 0x7f10b8194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10bf0f6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b81b8d50 con 0x7f10b817a820 2021-04-09T18:22:37.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10bf0f6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b817b240 con 0x7f10b8179610 2021-04-09T18:22:37.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10bf0f6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b8195410 con 0x7f10b81949f0 2021-04-09T18:22:37.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10b817a820 0x7f10b817ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10b817a820 0x7f10b817ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49472/0 (socket says 172.21.15.32:49472) 2021-04-09T18:22:37.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10b7fff700 1 -- 172.21.15.32:0/2162935911 learned_addr learned my addr 172.21.15.32:0/2162935911 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:37.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10bcf12700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b8179610 0x7f10b8179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10bd713700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b81949f0 0x7f10b8194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10b7fff700 1 -- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b81949f0 msgr2=0x7f10b8194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10b7fff700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b81949f0 0x7f10b8194df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10b7fff700 1 -- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b8179610 msgr2=0x7f10b8179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:37.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10b7fff700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b8179610 0x7f10b8179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:37.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10b7fff700 1 -- 172.21.15.32:0/2162935911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b8195630 con 0x7f10b817a820 2021-04-09T18:22:37.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10bd713700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b81949f0 0x7f10b8194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:37.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.497 7f10b7fff700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10b817a820 0x7f10b817ac20 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f10a800a590 tx=0x7f10a800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:37.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.498 7f10b57fa700 1 -- 172.21.15.32:0/2162935911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10a8000ef0 con 0x7f10b817a820 2021-04-09T18:22:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.498 7f10b57fa700 1 -- 172.21.15.32:0/2162935911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a800ad20 con 0x7f10b817a820 2021-04-09T18:22:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.498 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b8195630 con 0x7f10b817a820 2021-04-09T18:22:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.498 7f10b57fa700 1 -- 172.21.15.32:0/2162935911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10a8004590 con 0x7f10b817a820 2021-04-09T18:22:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.498 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b8195630 con 0x7f10b817a820 2021-04-09T18:22:37.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.498 7f10b57fa700 1 -- 172.21.15.32:0/2162935911 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f10a80043a0 con 0x7f10b817a820 2021-04-09T18:22:37.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.498 7f10b57fa700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f109c018930 0x7f109c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:37.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.498 7f10b57fa700 1 -- 172.21.15.32:0/2162935911 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f10a8032710 con 0x7f10b817a820 2021-04-09T18:22:37.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.498 7f10bcf12700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f109c018930 0x7f109c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:37.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.499 7f10bcf12700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f109c018930 0x7f109c01adc0 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:37.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.518 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b817a2f0 con 0x7f10b817a820 2021-04-09T18:22:37.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.522 7f10b57fa700 1 -- 172.21.15.32:0/2162935911 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f10a8007070 con 0x7f10b817a820 2021-04-09T18:22:37.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:37.703 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f10b817a2f0 con 0x7f10b817a820 2021-04-09T18:22:38.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.677 7f10b57fa700 1 -- 172.21.15.32:0/2162935911 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v100) v1 ==== 133+0+0 (secure 0 0 0) 0x7f10a801f780 con 0x7f10b817a820 2021-04-09T18:22:38.773 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:22:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.721 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f109c018930 msgr2=0x7f109c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:38.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.721 7f10bf0f6700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f109c018930 0x7f109c01adc0 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.721 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10b817a820 msgr2=0x7f10b817ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.721 7f10bf0f6700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10b817a820 0x7f10b817ac20 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f10a800a590 tx=0x7f10a800a930).stop 2021-04-09T18:22:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.722 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 shutdown_connections 2021-04-09T18:22:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.722 7f10bf0f6700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f109c018930 0x7f109c01adc0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.722 7f10bf0f6700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10b817a820 0x7f10b817ac20 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.722 7f10bf0f6700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b81949f0 0x7f10b8194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.722 7f10bf0f6700 1 --2- 172.21.15.32:0/2162935911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b8179610 0x7f10b8179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.722 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 >> 172.21.15.32:0/2162935911 conn(0x7f10b8092a00 msgr2=0x7f10b81b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:38.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.722 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 shutdown_connections 2021-04-09T18:22:38.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.722 7f10bf0f6700 1 -- 172.21.15.32:0/2162935911 wait complete. 2021-04-09T18:22:38.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:22:38.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.885 7fa0ce18f700 1 Processor -- start 2021-04-09T18:22:38.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.885 7fa0ce18f700 1 -- start start 2021-04-09T18:22:38.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0ce18f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa0c810d010 0x7fa0c80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:38.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0ce18f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c80d8490 0x7fa0c810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:38.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0ce18f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c810d480 con 0x7fa0c80d1da0 2021-04-09T18:22:38.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0ce18f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c80d8d30 con 0x7fa0c810d010 2021-04-09T18:22:38.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0ce18f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80d9220 con 0x7fa0c80d8490 2021-04-09T18:22:38.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c6ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0c80d1da0 0x7fa0c80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59074/0 (socket says 172.21.15.32:59074) 2021-04-09T18:22:38.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c6ffd700 1 -- 172.21.15.32:0/3841831733 learned_addr learned my addr 172.21.15.32:0/3841831733 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:38.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c77fe700 1 --2- 172.21.15.32:0/3841831733 >> v2:172.21.15.86:3300/0 conn(0x7fa0c810d010 0x7fa0c80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:38.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c7fff700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c80d8490 0x7fa0c810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:38.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658058997 0 0) 0x7fa0bc0009b0 con 0x7fa0c80d1da0 2021-04-09T18:22:38.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b0002fe0 con 0x7fa0c80d1da0 2021-04-09T18:22:38.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 81819644 0 0) 0x7fa0bc001010 con 0x7fa0c80d1da0 2021-04-09T18:22:38.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c80d8490 msgr2=0x7fa0c810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:38.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c67fc700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c80d8490 0x7fa0c810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 >> v2:172.21.15.86:3300/0 conn(0x7fa0c810d010 msgr2=0x7fa0c80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:38.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c67fc700 1 --2- 172.21.15.32:0/3841831733 >> v2:172.21.15.86:3300/0 conn(0x7fa0c810d010 0x7fa0c80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.886 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c810e020 con 0x7fa0c80d1da0 2021-04-09T18:22:38.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fa0bc001530 con 0x7fa0c80d1da0 2021-04-09T18:22:38.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 >> v1:172.21.15.32:6789/0 conn(0x7fa0c80d1da0 legacy=0x7fa0c80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:38.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0b0003a60 0x7fa0b0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:38.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0b004a7a0 0x7fa0b004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:38.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0b004d510 0x7fa0b004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:38.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00502e0 con 0x7fa0b004d510 2021-04-09T18:22:38.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0050500 con 0x7fa0b0003a60 2021-04-09T18:22:38.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0050720 con 0x7fa0b004a7a0 2021-04-09T18:22:38.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c7fff700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0b004d510 0x7fa0b004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c77fe700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0b0003a60 0x7fa0b0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c6ffd700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0b004a7a0 0x7fa0b004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c7fff700 1 -- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0b004a7a0 msgr2=0x7fa0b004cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c7fff700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0b004a7a0 0x7fa0b004cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c7fff700 1 -- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0b0003a60 msgr2=0x7fa0b0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c7fff700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0b0003a60 0x7fa0b0005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c7fff700 1 -- 172.21.15.32:0/3841831733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00509c0 con 0x7fa0b004d510 2021-04-09T18:22:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c7fff700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0b004d510 0x7fa0b004fa30 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7fa0c0009a40 tx=0x7fa0c0009de0).ready entity=mon.0 client_cookie=811ed8161b800859 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c0006a90 con 0x7fa0b004d510 2021-04-09T18:22:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c0003030 con 0x7fa0b004d510 2021-04-09T18:22:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c67fc700 1 -- 172.21.15.32:0/3841831733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c000b790 con 0x7fa0b004d510 2021-04-09T18:22:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0ce18f700 1 -- 172.21.15.32:0/3841831733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0b004d510 msgr2=0x7fa0b004fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0ce18f700 1 --2- 172.21.15.32:0/3841831733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0b004d510 0x7fa0b004fa30 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7fa0c0009a40 tx=0x7fa0c0009de0).stop 2021-04-09T18:22:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.887 7fa0c7fff700 1 -- 172.21.15.32:0/3841831733 reap_dead start 2021-04-09T18:22:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.888 7fa0ce18f700 1 -- 172.21.15.32:0/3841831733 shutdown_connections 2021-04-09T18:22:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.888 7fa0ce18f700 1 -- 172.21.15.32:0/3841831733 >> 172.21.15.32:0/3841831733 conn(0x7fa0c80ce5f0 msgr2=0x7fa0c80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.888 7fa0ce18f700 1 -- 172.21.15.32:0/3841831733 shutdown_connections 2021-04-09T18:22:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.888 7fa0ce18f700 1 -- 172.21.15.32:0/3841831733 wait complete. 2021-04-09T18:22:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.908 7fa0ce18f700 1 Processor -- start 2021-04-09T18:22:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0ce18f700 1 -- start start 2021-04-09T18:22:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0ce18f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c81a0050 0x7fa0c81a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0ce18f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c80bdcd0 0x7fa0c80bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0ce18f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c80bf080 0x7fa0c80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0ce18f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80bf500 con 0x7fa0c80bf080 2021-04-09T18:22:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0ce18f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80bb7b0 con 0x7fa0c81a0050 2021-04-09T18:22:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0ce18f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80be0d0 con 0x7fa0c80bdcd0 2021-04-09T18:22:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c80bf080 0x7fa0c80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c80bf080 0x7fa0c80b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49496/0 (socket says 172.21.15.32:49496) 2021-04-09T18:22:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0c7fff700 1 -- 172.21.15.32:0/2931968017 learned_addr learned my addr 172.21.15.32:0/2931968017 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0c77fe700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c81a0050 0x7fa0c81a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0c6ffd700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c80bdcd0 0x7fa0c80bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0c7fff700 1 -- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c80bdcd0 msgr2=0x7fa0c80bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0c7fff700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c80bdcd0 0x7fa0c80bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0c7fff700 1 -- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c81a0050 msgr2=0x7fa0c81a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:38.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.909 7fa0c7fff700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c81a0050 0x7fa0c81a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:38.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0c7fff700 1 -- 172.21.15.32:0/2931968017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c80b1310 con 0x7fa0c80bf080 2021-04-09T18:22:38.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0c77fe700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c81a0050 0x7fa0c81a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:38.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0c7fff700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c80bf080 0x7fa0c80b0cf0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7fa0c000b910 tx=0x7fa0c000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:38.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0affff700 1 -- 172.21.15.32:0/2931968017 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c0000df0 con 0x7fa0c80bf080 2021-04-09T18:22:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0affff700 1 -- 172.21.15.32:0/2931968017 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c00011a0 con 0x7fa0c80bf080 2021-04-09T18:22:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c80b1310 con 0x7fa0c80bf080 2021-04-09T18:22:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0affff700 1 -- 172.21.15.32:0/2931968017 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c0001680 con 0x7fa0c80bf080 2021-04-09T18:22:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c80b1310 con 0x7fa0c80bf080 2021-04-09T18:22:38.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0affff700 1 -- 172.21.15.32:0/2931968017 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa0c0001380 con 0x7fa0c80bf080 2021-04-09T18:22:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.910 7fa0affff700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a8018930 0x7fa0a801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.911 7fa0affff700 1 -- 172.21.15.32:0/2931968017 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5111+0+0 (secure 0 0 0) 0x7fa0c0032bd0 con 0x7fa0c80bf080 2021-04-09T18:22:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.911 7fa0c77fe700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a8018930 0x7fa0a801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:38.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.911 7fa0c77fe700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a8018930 0x7fa0a801ae10 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:38.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.930 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c80bb7b0 con 0x7fa0c80bf080 2021-04-09T18:22:38.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:38.935 7fa0affff700 1 -- 172.21.15.32:0/2931968017 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa0c000bdf0 con 0x7fa0c80bf080 2021-04-09T18:22:39.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.117 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fa0c80bb7b0 con 0x7fa0c80bf080 2021-04-09T18:22:39.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.945 7fa0affff700 1 -- 172.21.15.32:0/2931968017 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v105) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa0c00011a0 con 0x7fa0c80bf080 2021-04-09T18:22:40.036 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:22:40.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.984 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a8018930 msgr2=0x7fa0a801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.984 7fa0ce18f700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a8018930 0x7fa0a801ae10 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.984 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c80bf080 msgr2=0x7fa0c80b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.984 7fa0ce18f700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c80bf080 0x7fa0c80b0cf0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7fa0c000b910 tx=0x7fa0c000bcb0).stop 2021-04-09T18:22:40.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.985 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 shutdown_connections 2021-04-09T18:22:40.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.985 7fa0ce18f700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a8018930 0x7fa0a801ae10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.985 7fa0ce18f700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c80bf080 0x7fa0c80b0cf0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.985 7fa0ce18f700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c80bdcd0 0x7fa0c80bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.985 7fa0ce18f700 1 --2- 172.21.15.32:0/2931968017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c81a0050 0x7fa0c81a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.985 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 >> 172.21.15.32:0/2931968017 conn(0x7fa0c80975f0 msgr2=0x7fa0c8096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:40.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.985 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 shutdown_connections 2021-04-09T18:22:40.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:39.985 7fa0ce18f700 1 -- 172.21.15.32:0/2931968017 wait complete. 2021-04-09T18:22:40.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:22:40.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.154 7f078fba2700 1 Processor -- start 2021-04-09T18:22:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.155 7f078fba2700 1 -- start start 2021-04-09T18:22:40.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.155 7f078fba2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f078810d050 0x7f07880cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.155 7f078fba2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07880d0160 0x7f07880d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.155 7f078fba2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078810d500 con 0x7f07880d82e0 2021-04-09T18:22:40.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.155 7f078fba2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07880d9260 con 0x7f078810d050 2021-04-09T18:22:40.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.155 7f078fba2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07880d86e0 con 0x7f07880d0160 2021-04-09T18:22:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.155 7f078d1bd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07880d0160 0x7f07880d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078d9be700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f078810d050 0x7f07880cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078e1bf700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f07880d82e0 0x7f078810da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:57022/0 (socket says 172.21.15.32:57022) 2021-04-09T18:22:40.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078e1bf700 1 -- 172.21.15.32:0/1523016247 learned_addr learned my addr 172.21.15.32:0/1523016247 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:40.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078d9be700 1 -- 172.21.15.32:0/1523016247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07880d0160 msgr2=0x7f07880d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:40.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078d9be700 1 --2- 172.21.15.32:0/1523016247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07880d0160 0x7f07880d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078d9be700 1 -- 172.21.15.32:0/1523016247 >> v1:172.21.15.86:6789/0 conn(0x7f07880d82e0 legacy=0x7f078810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078c9bc700 1 -- 172.21.15.32:0/1523016247 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717605087 0 0) 0x7f0780001440 con 0x7f07880d82e0 2021-04-09T18:22:40.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078d9be700 1 -- 172.21.15.32:0/1523016247 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078810e270 con 0x7f078810d050 2021-04-09T18:22:40.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078d1bd700 1 --2- 172.21.15.32:0/1523016247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07880d0160 0x7f07880d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:40.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078d9be700 1 --2- 172.21.15.32:0/1523016247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f078810d050 0x7f07880cfb40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f0784006650 tx=0x7f0784009a80).ready entity=mon.1 client_cookie=22104231aa931d0c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:40.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078c9bc700 1 -- 172.21.15.32:0/1523016247 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07840009b0 con 0x7f078810d050 2021-04-09T18:22:40.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078c9bc700 1 -- 172.21.15.32:0/1523016247 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078400c0c0 con 0x7f078810d050 2021-04-09T18:22:40.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078c9bc700 1 -- 172.21.15.32:0/1523016247 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f078400c510 con 0x7f078810d050 2021-04-09T18:22:40.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078fba2700 1 -- 172.21.15.32:0/1523016247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f078810d050 msgr2=0x7f07880cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.156 7f078fba2700 1 --2- 172.21.15.32:0/1523016247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f078810d050 0x7f07880cfb40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f0784006650 tx=0x7f0784009a80).stop 2021-04-09T18:22:40.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.157 7f078fba2700 1 -- 172.21.15.32:0/1523016247 shutdown_connections 2021-04-09T18:22:40.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.157 7f078fba2700 1 --2- 172.21.15.32:0/1523016247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07880d0160 0x7f07880d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.157 7f078fba2700 1 --2- 172.21.15.32:0/1523016247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f078810d050 0x7f07880cfb40 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.157 7f078fba2700 1 -- 172.21.15.32:0/1523016247 >> 172.21.15.32:0/1523016247 conn(0x7f07880ce620 msgr2=0x7f07880d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:40.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.157 7f078fba2700 1 -- 172.21.15.32:0/1523016247 shutdown_connections 2021-04-09T18:22:40.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.157 7f078fba2700 1 -- 172.21.15.32:0/1523016247 wait complete. 2021-04-09T18:22:40.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.177 7f078fba2700 1 Processor -- start 2021-04-09T18:22:40.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.177 7f078fba2700 1 -- start start 2021-04-09T18:22:40.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078fba2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f078809b2f0 0x7f078809b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078fba2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f078809a8b0 0x7f078809e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078fba2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f078809e9c0 0x7f078809edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078fba2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07880bde80 con 0x7f078809a8b0 2021-04-09T18:22:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078fba2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078809ad70 con 0x7f078809b2f0 2021-04-09T18:22:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078fba2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078809bd10 con 0x7f078809e9c0 2021-04-09T18:22:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078d1bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f078809a8b0 0x7f078809e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078d1bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f078809a8b0 0x7f078809e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49510/0 (socket says 172.21.15.32:49510) 2021-04-09T18:22:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078d1bd700 1 -- 172.21.15.32:0/953978378 learned_addr learned my addr 172.21.15.32:0/953978378 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:40.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078e1bf700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f078809e9c0 0x7f078809edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078d9be700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f078809b2f0 0x7f078809b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078d1bd700 1 -- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f078809e9c0 msgr2=0x7f078809edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078d1bd700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f078809e9c0 0x7f078809edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078d1bd700 1 -- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f078809b2f0 msgr2=0x7f078809b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078d1bd700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f078809b2f0 0x7f078809b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.178 7f078d1bd700 1 -- 172.21.15.32:0/953978378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078809fbd0 con 0x7f078809a8b0 2021-04-09T18:22:40.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.179 7f078d1bd700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f078809a8b0 0x7f078809e3a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f0778006f40 tx=0x7f077800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:40.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.179 7f077e7fc700 1 -- 172.21.15.32:0/953978378 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f077800bd80 con 0x7f078809a8b0 2021-04-09T18:22:40.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.179 7f077e7fc700 1 -- 172.21.15.32:0/953978378 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0778000c90 con 0x7f078809a8b0 2021-04-09T18:22:40.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.179 7f077e7fc700 1 -- 172.21.15.32:0/953978378 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0778002220 con 0x7f078809a8b0 2021-04-09T18:22:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.179 7f078fba2700 1 -- 172.21.15.32:0/953978378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078809fbd0 con 0x7f078809a8b0 2021-04-09T18:22:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.179 7f078fba2700 1 -- 172.21.15.32:0/953978378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078809fbd0 con 0x7f078809a8b0 2021-04-09T18:22:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.180 7f077e7fc700 1 -- 172.21.15.32:0/953978378 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f077800a000 con 0x7f078809a8b0 2021-04-09T18:22:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.180 7f077e7fc700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07700188c0 0x7f077001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.180 7f077e7fc700 1 -- 172.21.15.32:0/953978378 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f0778032750 con 0x7f078809a8b0 2021-04-09T18:22:40.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.180 7f078d9be700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07700188c0 0x7f077001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.180 7f078d9be700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07700188c0 0x7f077001adc0 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:40.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.199 7f078fba2700 1 -- 172.21.15.32:0/953978378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07880b9040 con 0x7f078809a8b0 2021-04-09T18:22:40.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.203 7f077e7fc700 1 -- 172.21.15.32:0/953978378 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0778006b00 con 0x7f078809a8b0 2021-04-09T18:22:40.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.387 7f078fba2700 1 -- 172.21.15.32:0/953978378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f078810d500 con 0x7f078809a8b0 2021-04-09T18:22:40.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.391 7f077e7fc700 1 -- 172.21.15.32:0/953978378 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 18 and data pool 19 v124) v1 ==== 189+0+0 (secure 0 0 0) 0x7f077800a3b0 con 0x7f078809a8b0 2021-04-09T18:22:40.449 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 18 and data pool 19 2021-04-09T18:22:40.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 -- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07700188c0 msgr2=0x7f077001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07700188c0 0x7f077001adc0 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 -- 172.21.15.32:0/953978378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f078809a8b0 msgr2=0x7f078809e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f078809a8b0 0x7f078809e3a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f0778006f40 tx=0x7f077800b990).stop 2021-04-09T18:22:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 -- 172.21.15.32:0/953978378 shutdown_connections 2021-04-09T18:22:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07700188c0 0x7f077001adc0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f078809a8b0 0x7f078809e3a0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f078809e9c0 0x7f078809edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 --2- 172.21.15.32:0/953978378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f078809b2f0 0x7f078809b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 -- 172.21.15.32:0/953978378 >> 172.21.15.32:0/953978378 conn(0x7f07880ae6f0 msgr2=0x7f0788097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 -- 172.21.15.32:0/953978378 shutdown_connections 2021-04-09T18:22:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.399 7f078fba2700 1 -- 172.21.15.32:0/953978378 wait complete. 2021-04-09T18:22:40.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:22:40.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.562 7fd36d624700 1 Processor -- start 2021-04-09T18:22:40.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.562 7fd36d624700 1 -- start start 2021-04-09T18:22:40.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd36d624700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd36810d010 0x7fd3680d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd36d624700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3680d1da0 0x7fd3680d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd36d624700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36810d480 con 0x7fd3680d82c0 2021-04-09T18:22:40.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd36d624700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3680d9220 con 0x7fd36810d010 2021-04-09T18:22:40.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd36d624700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680d86c0 con 0x7fd3680d1da0 2021-04-09T18:22:40.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd3672bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd36810d010 0x7fd3680d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd367ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd3680d82c0 0x7fd36810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59112/0 (socket says 172.21.15.32:59112) 2021-04-09T18:22:40.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd367ac0700 1 -- 172.21.15.32:0/3978989091 learned_addr learned my addr 172.21.15.32:0/3978989091 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:40.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd366abe700 1 --2- 172.21.15.32:0/3978989091 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3680d1da0 0x7fd3680d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd3672bf700 1 -- 172.21.15.32:0/3978989091 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3680d1da0 msgr2=0x7fd3680d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:40.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd3672bf700 1 --2- 172.21.15.32:0/3978989091 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3680d1da0 0x7fd3680d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd3672bf700 1 -- 172.21.15.32:0/3978989091 >> v1:172.21.15.32:6789/0 conn(0x7fd3680d82c0 legacy=0x7fd36810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd3662bd700 1 -- 172.21.15.32:0/3978989091 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773590794 0 0) 0x7fd360000d30 con 0x7fd3680d82c0 2021-04-09T18:22:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd3672bf700 1 -- 172.21.15.32:0/3978989091 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36810e1f0 con 0x7fd36810d010 2021-04-09T18:22:40.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd366abe700 1 --2- 172.21.15.32:0/3978989091 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3680d1da0 0x7fd3680d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd3672bf700 1 --2- 172.21.15.32:0/3978989091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd36810d010 0x7fd3680d1780 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7fd358006650 tx=0x7fd358009a80).ready entity=mon.0 client_cookie=f2e70c1771fd5096 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.563 7fd3662bd700 1 -- 172.21.15.32:0/3978989091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3580009b0 con 0x7fd36810d010 2021-04-09T18:22:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd3662bd700 1 -- 172.21.15.32:0/3978989091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35800c0c0 con 0x7fd36810d010 2021-04-09T18:22:40.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd3662bd700 1 -- 172.21.15.32:0/3978989091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3580009b0 con 0x7fd36810d010 2021-04-09T18:22:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd36d624700 1 -- 172.21.15.32:0/3978989091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd36810d010 msgr2=0x7fd3680d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd36d624700 1 --2- 172.21.15.32:0/3978989091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd36810d010 0x7fd3680d1780 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7fd358006650 tx=0x7fd358009a80).stop 2021-04-09T18:22:40.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd36d624700 1 -- 172.21.15.32:0/3978989091 shutdown_connections 2021-04-09T18:22:40.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd36d624700 1 --2- 172.21.15.32:0/3978989091 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3680d1da0 0x7fd3680d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd36d624700 1 --2- 172.21.15.32:0/3978989091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd36810d010 0x7fd3680d1780 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd36d624700 1 -- 172.21.15.32:0/3978989091 >> 172.21.15.32:0/3978989091 conn(0x7fd3680ce5f0 msgr2=0x7fd3680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd36d624700 1 -- 172.21.15.32:0/3978989091 shutdown_connections 2021-04-09T18:22:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.564 7fd36d624700 1 -- 172.21.15.32:0/3978989091 wait complete. 2021-04-09T18:22:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.585 7fd36d624700 1 Processor -- start 2021-04-09T18:22:40.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.585 7fd36d624700 1 -- start start 2021-04-09T18:22:40.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.585 7fd36d624700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3681ac380 0x7fd36818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.585 7fd36d624700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3681ac780 0x7fd36818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.585 7fd36d624700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd368183520 0x7fd368183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.585 7fd36d624700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681ace10 con 0x7fd3681ac780 2021-04-09T18:22:40.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.585 7fd36d624700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368189cf0 con 0x7fd368183520 2021-04-09T18:22:40.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.585 7fd36d624700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd368183f40 con 0x7fd3681ac380 2021-04-09T18:22:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd366abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3681ac780 0x7fd36818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd366abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3681ac780 0x7fd36818a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49526/0 (socket says 172.21.15.32:49526) 2021-04-09T18:22:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd366abe700 1 -- 172.21.15.32:0/1814170122 learned_addr learned my addr 172.21.15.32:0/1814170122 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd3672bf700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3681ac380 0x7fd36818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd367ac0700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd368183520 0x7fd368183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd366abe700 1 -- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3681ac380 msgr2=0x7fd36818a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd366abe700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3681ac380 0x7fd36818a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd366abe700 1 -- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd368183520 msgr2=0x7fd368183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd366abe700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd368183520 0x7fd368183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd366abe700 1 -- 172.21.15.32:0/1814170122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368184160 con 0x7fd3681ac780 2021-04-09T18:22:40.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd366abe700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3681ac780 0x7fd36818a850 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7fd35c006f20 tx=0x7fd35c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:40.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd34ffff700 1 -- 172.21.15.32:0/1814170122 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35c00bd80 con 0x7fd3681ac780 2021-04-09T18:22:40.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd34ffff700 1 -- 172.21.15.32:0/1814170122 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c000c90 con 0x7fd3681ac780 2021-04-09T18:22:40.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd34ffff700 1 -- 172.21.15.32:0/1814170122 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35c0022b0 con 0x7fd3681ac780 2021-04-09T18:22:40.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd36d624700 1 -- 172.21.15.32:0/1814170122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd368184160 con 0x7fd3681ac780 2021-04-09T18:22:40.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.586 7fd36d624700 1 -- 172.21.15.32:0/1814170122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd368184160 con 0x7fd3681ac780 2021-04-09T18:22:40.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.587 7fd34ffff700 1 -- 172.21.15.32:0/1814170122 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd35c00a000 con 0x7fd3681ac780 2021-04-09T18:22:40.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.587 7fd34ffff700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd348018930 0x7fd34801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:40.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.587 7fd34ffff700 1 -- 172.21.15.32:0/1814170122 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd35c032a30 con 0x7fd3681ac780 2021-04-09T18:22:40.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.587 7fd3672bf700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd348018930 0x7fd34801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:40.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.587 7fd3672bf700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd348018930 0x7fd34801ae10 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:40.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.607 7fd36d624700 1 -- 172.21.15.32:0/1814170122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd368189cf0 con 0x7fd3681ac780 2021-04-09T18:22:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.611 7fd34ffff700 1 -- 172.21.15.32:0/1814170122 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd35c006bb0 con 0x7fd3681ac780 2021-04-09T18:22:40.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.800 7fd36d624700 1 -- 172.21.15.32:0/1814170122 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd368189cf0 con 0x7fd3681ac780 2021-04-09T18:22:40.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.800 7fd34ffff700 1 -- 172.21.15.32:0/1814170122 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+14019 (secure 0 0 0) 0x7fd35c00a000 con 0x7fd3681ac780 2021-04-09T18:22:40.856 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:40.856 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":106,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:22:40.389257","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:22:37.739108","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-04-09 18:22:39.152543","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:22:40.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 -- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd348018930 msgr2=0x7fd34801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd348018930 0x7fd34801ae10 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 -- 172.21.15.32:0/1814170122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3681ac780 msgr2=0x7fd36818a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:40.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3681ac780 0x7fd36818a850 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7fd35c006f20 tx=0x7fd35c00b990).stop 2021-04-09T18:22:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 -- 172.21.15.32:0/1814170122 shutdown_connections 2021-04-09T18:22:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd348018930 0x7fd34801ae10 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3681ac780 0x7fd36818a850 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd368183520 0x7fd368183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 --2- 172.21.15.32:0/1814170122 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3681ac380 0x7fd36818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:40.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 -- 172.21.15.32:0/1814170122 >> 172.21.15.32:0/1814170122 conn(0x7fd36819b890 msgr2=0x7fd3681a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:40.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 -- 172.21.15.32:0/1814170122 shutdown_connections 2021-04-09T18:22:40.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.803 7fd36d624700 1 -- 172.21.15.32:0/1814170122 wait complete. 2021-04-09T18:22:40.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:22:41.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.961 7f598d925700 1 Processor -- start 2021-04-09T18:22:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.961 7f598d925700 1 -- start start 2021-04-09T18:22:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f598d925700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f59880d7ca0 0x7f59880d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f598d925700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f598810da00 0x7f598810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f598d925700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598810d480 con 0x7f598810d010 2021-04-09T18:22:41.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f598d925700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59880d9220 con 0x7f598810da00 2021-04-09T18:22:41.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f598d925700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59880d86c0 con 0x7f59880d7ca0 2021-04-09T18:22:41.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5987d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f598810da00 0x7f598810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5987574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f598810d010 0x7f59880d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59122/0 (socket says 172.21.15.32:59122) 2021-04-09T18:22:41.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5987574700 1 -- 172.21.15.32:0/249430261 learned_addr learned my addr 172.21.15.32:0/249430261 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:41.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5986d73700 1 --2- 172.21.15.32:0/249430261 >> v2:172.21.15.86:3300/0 conn(0x7f59880d7ca0 0x7f59880d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5986572700 1 -- 172.21.15.32:0/249430261 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882737726 0 0) 0x7f5978001060 con 0x7f598810d010 2021-04-09T18:22:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5986572700 1 -- 172.21.15.32:0/249430261 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5970002fe0 con 0x7f598810d010 2021-04-09T18:22:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5987d75700 1 -- 172.21.15.32:0/249430261 >> v2:172.21.15.86:3300/0 conn(0x7f59880d7ca0 msgr2=0x7f59880d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:41.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5987d75700 1 --2- 172.21.15.32:0/249430261 >> v2:172.21.15.86:3300/0 conn(0x7f59880d7ca0 0x7f59880d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5987d75700 1 -- 172.21.15.32:0/249430261 >> v1:172.21.15.32:6789/0 conn(0x7f598810d010 legacy=0x7f59880d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5987d75700 1 -- 172.21.15.32:0/249430261 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59880d1f70 con 0x7f598810da00 2021-04-09T18:22:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5987d75700 1 --2- 172.21.15.32:0/249430261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f598810da00 0x7f598810de00 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7f5980006ab0 tx=0x7f5980009b40).ready entity=mon.0 client_cookie=a0f77e162d1266a9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:41.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5986572700 1 -- 172.21.15.32:0/249430261 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59800022e0 con 0x7f598810da00 2021-04-09T18:22:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5986572700 1 -- 172.21.15.32:0/249430261 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5980002660 con 0x7f598810da00 2021-04-09T18:22:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f5986572700 1 -- 172.21.15.32:0/249430261 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5980002b40 con 0x7f598810da00 2021-04-09T18:22:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f598d925700 1 -- 172.21.15.32:0/249430261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f598810da00 msgr2=0x7f598810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.962 7f598d925700 1 --2- 172.21.15.32:0/249430261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f598810da00 0x7f598810de00 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7f5980006ab0 tx=0x7f5980009b40).stop 2021-04-09T18:22:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.963 7f598d925700 1 -- 172.21.15.32:0/249430261 shutdown_connections 2021-04-09T18:22:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.963 7f598d925700 1 --2- 172.21.15.32:0/249430261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f598810da00 0x7f598810de00 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.963 7f598d925700 1 --2- 172.21.15.32:0/249430261 >> v2:172.21.15.86:3300/0 conn(0x7f59880d7ca0 0x7f59880d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.963 7f598d925700 1 -- 172.21.15.32:0/249430261 >> 172.21.15.32:0/249430261 conn(0x7f59880ce5f0 msgr2=0x7f59880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:41.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.963 7f598d925700 1 -- 172.21.15.32:0/249430261 shutdown_connections 2021-04-09T18:22:41.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.963 7f598d925700 1 -- 172.21.15.32:0/249430261 wait complete. 2021-04-09T18:22:41.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.983 7f598d925700 1 Processor -- start 2021-04-09T18:22:41.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.983 7f598d925700 1 -- start start 2021-04-09T18:22:41.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f598d925700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59881ac670 0x7f59881a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f598d925700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59881a5dc0 0x7f59880ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f598d925700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59881aca70 0x7f59880ba800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f598d925700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59881a5220 con 0x7f59881a5dc0 2021-04-09T18:22:41.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f598d925700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598818e100 con 0x7f59881ac670 2021-04-09T18:22:41.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f598d925700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598819ccc0 con 0x7f59881aca70 2021-04-09T18:22:41.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5986d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59881a5dc0 0x7f59880ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5986d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59881a5dc0 0x7f59880ba1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49540/0 (socket says 172.21.15.32:49540) 2021-04-09T18:22:41.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5986d73700 1 -- 172.21.15.32:0/2468793856 learned_addr learned my addr 172.21.15.32:0/2468793856 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:41.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5987574700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59881ac670 0x7f59881a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5987d75700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59881aca70 0x7f59880ba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5986d73700 1 -- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59881aca70 msgr2=0x7f59880ba800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5986d73700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59881aca70 0x7f59880ba800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5986d73700 1 -- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59881ac670 msgr2=0x7f59881a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5986d73700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59881ac670 0x7f59881a57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5986d73700 1 -- 172.21.15.32:0/2468793856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59880bae20 con 0x7f59881a5dc0 2021-04-09T18:22:41.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.984 7f5986d73700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59881a5dc0 0x7f59880ba1e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0x7f597c006f60 tx=0x7f597c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:41.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.985 7f596ffff700 1 -- 172.21.15.32:0/2468793856 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f597c00bd60 con 0x7f59881a5dc0 2021-04-09T18:22:41.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.985 7f596ffff700 1 -- 172.21.15.32:0/2468793856 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c000c90 con 0x7f59881a5dc0 2021-04-09T18:22:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.985 7f598d925700 1 -- 172.21.15.32:0/2468793856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59880bae20 con 0x7f59881a5dc0 2021-04-09T18:22:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.985 7f596ffff700 1 -- 172.21.15.32:0/2468793856 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f597c002220 con 0x7f59881a5dc0 2021-04-09T18:22:41.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.985 7f598d925700 1 -- 172.21.15.32:0/2468793856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59880bae20 con 0x7f59881a5dc0 2021-04-09T18:22:41.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.985 7f596ffff700 1 -- 172.21.15.32:0/2468793856 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f597c0052e0 con 0x7f59881a5dc0 2021-04-09T18:22:41.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.985 7f596ffff700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5968018930 0x7f596801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.985 7f596ffff700 1 -- 172.21.15.32:0/2468793856 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f597c0329f0 con 0x7f59881a5dc0 2021-04-09T18:22:41.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.985 7f5987574700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5968018930 0x7f596801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:40.986 7f5987574700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5968018930 0x7f596801ae10 crc :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:41.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.006 7f598d925700 1 -- 172.21.15.32:0/2468793856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5988182d10 con 0x7f59881a5dc0 2021-04-09T18:22:41.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.009 7f596ffff700 1 -- 172.21.15.32:0/2468793856 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f597c000c90 con 0x7f59881a5dc0 2021-04-09T18:22:41.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.198 7f598d925700 1 -- 172.21.15.32:0/2468793856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5988182d10 con 0x7f59881a5dc0 2021-04-09T18:22:41.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.199 7f596ffff700 1 -- 172.21.15.32:0/2468793856 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+14019 (secure 0 0 0) 0x7f597c006b00 con 0x7f59881a5dc0 2021-04-09T18:22:41.254 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:41.254 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":106,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:22:40.389257","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:22:37.739108","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-04-09 18:22:39.152543","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:22:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 -- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5968018930 msgr2=0x7f596801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5968018930 0x7f596801ae10 crc :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 -- 172.21.15.32:0/2468793856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59881a5dc0 msgr2=0x7f59880ba1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59881a5dc0 0x7f59880ba1e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0x7f597c006f60 tx=0x7f597c00b970).stop 2021-04-09T18:22:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 -- 172.21.15.32:0/2468793856 shutdown_connections 2021-04-09T18:22:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5968018930 0x7f596801ae10 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59881a5dc0 0x7f59880ba1e0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59881aca70 0x7f59880ba800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 --2- 172.21.15.32:0/2468793856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59881ac670 0x7f59881a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 -- 172.21.15.32:0/2468793856 >> 172.21.15.32:0/2468793856 conn(0x7f598819b890 msgr2=0x7f59881a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 -- 172.21.15.32:0/2468793856 shutdown_connections 2021-04-09T18:22:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.201 7f598d925700 1 -- 172.21.15.32:0/2468793856 wait complete. 2021-04-09T18:22:41.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:22:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.364 7ff6e3220700 1 Processor -- start 2021-04-09T18:22:41.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.364 7ff6e3220700 1 -- start start 2021-04-09T18:22:41.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e3220700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0d7ca0 0x7ff6dc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e3220700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff6dc10da00 0x7ff6dc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e3220700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc10d480 con 0x7ff6dc10d010 2021-04-09T18:22:41.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e3220700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc0d9220 con 0x7ff6dc10da00 2021-04-09T18:22:41.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e3220700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0d86c0 con 0x7ff6dc0d7ca0 2021-04-09T18:22:41.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e083b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0d7ca0 0x7ff6dc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e183d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff6dc10da00 0x7ff6dc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e103c700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff6dc10d010 0x7ff6dc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:57062/0 (socket says 172.21.15.32:57062) 2021-04-09T18:22:41.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e103c700 1 -- 172.21.15.32:0/912933512 learned_addr learned my addr 172.21.15.32:0/912933512 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:41.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e083b700 1 -- 172.21.15.32:0/912933512 >> v2:172.21.15.86:3300/0 conn(0x7ff6dc10da00 msgr2=0x7ff6dc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:41.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e083b700 1 --2- 172.21.15.32:0/912933512 >> v2:172.21.15.86:3300/0 conn(0x7ff6dc10da00 0x7ff6dc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e083b700 1 -- 172.21.15.32:0/912933512 >> v1:172.21.15.86:6789/0 conn(0x7ff6dc10d010 legacy=0x7ff6dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e083b700 1 -- 172.21.15.32:0/912933512 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc0d1f70 con 0x7ff6dc0d7ca0 2021-04-09T18:22:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6d3fff700 1 -- 172.21.15.32:0/912933512 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270403981 0 0) 0x7ff6d80010c0 con 0x7ff6dc10d010 2021-04-09T18:22:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e183d700 1 --2- 172.21.15.32:0/912933512 >> v2:172.21.15.86:3300/0 conn(0x7ff6dc10da00 0x7ff6dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:41.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.365 7ff6e083b700 1 --2- 172.21.15.32:0/912933512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0d7ca0 0x7ff6dc0d80a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff6cc006650 tx=0x7ff6cc009a80).ready entity=mon.2 client_cookie=5bf991ad4bb14998 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:41.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.366 7ff6d3fff700 1 -- 172.21.15.32:0/912933512 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6cc0009b0 con 0x7ff6dc0d7ca0 2021-04-09T18:22:41.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.366 7ff6d3fff700 1 -- 172.21.15.32:0/912933512 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc00c0c0 con 0x7ff6dc0d7ca0 2021-04-09T18:22:41.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.366 7ff6d3fff700 1 -- 172.21.15.32:0/912933512 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6cc00c510 con 0x7ff6dc0d7ca0 2021-04-09T18:22:41.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.366 7ff6e3220700 1 -- 172.21.15.32:0/912933512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0d7ca0 msgr2=0x7ff6dc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.366 7ff6e3220700 1 --2- 172.21.15.32:0/912933512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0d7ca0 0x7ff6dc0d80a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff6cc006650 tx=0x7ff6cc009a80).stop 2021-04-09T18:22:41.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.366 7ff6e3220700 1 -- 172.21.15.32:0/912933512 shutdown_connections 2021-04-09T18:22:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.366 7ff6e3220700 1 --2- 172.21.15.32:0/912933512 >> v2:172.21.15.86:3300/0 conn(0x7ff6dc10da00 0x7ff6dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.366 7ff6e3220700 1 --2- 172.21.15.32:0/912933512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0d7ca0 0x7ff6dc0d80a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.366 7ff6e3220700 1 -- 172.21.15.32:0/912933512 >> 172.21.15.32:0/912933512 conn(0x7ff6dc0ce5f0 msgr2=0x7ff6dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:41.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.367 7ff6e3220700 1 -- 172.21.15.32:0/912933512 shutdown_connections 2021-04-09T18:22:41.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.367 7ff6e3220700 1 -- 172.21.15.32:0/912933512 wait complete. 2021-04-09T18:22:41.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.387 7ff6e3220700 1 Processor -- start 2021-04-09T18:22:41.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.387 7ff6e3220700 1 -- start start 2021-04-09T18:22:41.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.387 7ff6e3220700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0c3c00 0x7ff6dc09c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.387 7ff6e3220700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc095240 0x7ff6dc1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.387 7ff6e3220700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6dc1a1270 0x7ff6dc0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.387 7ff6e3220700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc09cda0 con 0x7ff6dc095240 2021-04-09T18:22:41.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.387 7ff6e3220700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0c3680 con 0x7ff6dc1a1270 2021-04-09T18:22:41.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.387 7ff6e3220700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc095640 con 0x7ff6dc0c3c00 2021-04-09T18:22:41.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e083b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc095240 0x7ff6dc1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e083b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc095240 0x7ff6dc1a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49552/0 (socket says 172.21.15.32:49552) 2021-04-09T18:22:41.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e083b700 1 -- 172.21.15.32:0/452232718 learned_addr learned my addr 172.21.15.32:0/452232718 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e183d700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6dc1a1270 0x7ff6dc0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e103c700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0c3c00 0x7ff6dc09c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e083b700 1 -- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0c3c00 msgr2=0x7ff6dc09c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e083b700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0c3c00 0x7ff6dc09c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e083b700 1 -- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6dc1a1270 msgr2=0x7ff6dc0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:41.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e083b700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6dc1a1270 0x7ff6dc0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:41.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e083b700 1 -- 172.21.15.32:0/452232718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc1a1670 con 0x7ff6dc095240 2021-04-09T18:22:41.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e083b700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc095240 0x7ff6dc1a0c50 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7ff6cc009630 tx=0x7ff6cc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:41.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6d1ffb700 1 -- 172.21.15.32:0/452232718 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6cc0093a0 con 0x7ff6dc095240 2021-04-09T18:22:41.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6d1ffb700 1 -- 172.21.15.32:0/452232718 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc00bd70 con 0x7ff6dc095240 2021-04-09T18:22:41.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e3220700 1 -- 172.21.15.32:0/452232718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc1a1670 con 0x7ff6dc095240 2021-04-09T18:22:41.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6d1ffb700 1 -- 172.21.15.32:0/452232718 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6cc0054d0 con 0x7ff6dc095240 2021-04-09T18:22:41.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.388 7ff6e3220700 1 -- 172.21.15.32:0/452232718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc1a1670 con 0x7ff6dc095240 2021-04-09T18:22:41.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.389 7ff6d1ffb700 1 -- 172.21.15.32:0/452232718 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff6cc0054f0 con 0x7ff6dc095240 2021-04-09T18:22:41.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.389 7ff6d1ffb700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6bc018930 0x7ff6bc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:41.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.389 7ff6d1ffb700 1 -- 172.21.15.32:0/452232718 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff6cc032b80 con 0x7ff6dc095240 2021-04-09T18:22:41.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.389 7ff6e103c700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6bc018930 0x7ff6bc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:41.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.390 7ff6e103c700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6bc018930 0x7ff6bc01ae10 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:41.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.409 7ff6e3220700 1 -- 172.21.15.32:0/452232718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc093d50 con 0x7ff6dc095240 2021-04-09T18:22:41.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.412 7ff6d1ffb700 1 -- 172.21.15.32:0/452232718 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff6cc0052e0 con 0x7ff6dc095240 2021-04-09T18:22:41.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:41.607 7ff6e3220700 1 -- 172.21.15.32:0/452232718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7ff6dc10d480 con 0x7ff6dc095240 2021-04-09T18:22:42.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.407 7ff6d1ffb700 1 -- 172.21.15.32:0/452232718 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v127) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff6cc0054f0 con 0x7ff6dc095240 2021-04-09T18:22:42.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 -- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6bc018930 msgr2=0x7ff6bc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:42.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6bc018930 0x7ff6bc01ae10 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 -- 172.21.15.32:0/452232718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc095240 msgr2=0x7ff6dc1a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc095240 0x7ff6dc1a0c50 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7ff6cc009630 tx=0x7ff6cc00b980).stop 2021-04-09T18:22:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 -- 172.21.15.32:0/452232718 shutdown_connections 2021-04-09T18:22:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6bc018930 0x7ff6bc01ae10 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6dc095240 0x7ff6dc1a0c50 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6dc1a1270 0x7ff6dc0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 --2- 172.21.15.32:0/452232718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6dc0c3c00 0x7ff6dc09c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 -- 172.21.15.32:0/452232718 >> 172.21.15.32:0/452232718 conn(0x7ff6dc0975f0 msgr2=0x7ff6dc096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 -- 172.21.15.32:0/452232718 shutdown_connections 2021-04-09T18:22:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.425 7ff6e3220700 1 -- 172.21.15.32:0/452232718 wait complete. 2021-04-09T18:22:42.490 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:42.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.585 7f0b5d8f4700 1 Processor -- start 2021-04-09T18:22:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.585 7f0b5d8f4700 1 -- start start 2021-04-09T18:22:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b5d8f4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b5810da00 0x7f0b5810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:42.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b5d8f4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5810d480 con 0x7f0b580d7ca0 2021-04-09T18:22:42.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b5d8f4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580d1f70 con 0x7f0b5810d010 2021-04-09T18:22:42.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b5d8f4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b580d8d30 con 0x7f0b5810da00 2021-04-09T18:22:42.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b56d73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0b580d7ca0 0x7f0b580d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59150/0 (socket says 172.21.15.32:59150) 2021-04-09T18:22:42.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b56d73700 1 -- 172.21.15.32:0/1392506024 learned_addr learned my addr 172.21.15.32:0/1392506024 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b57d75700 1 --2- 172.21.15.32:0/1392506024 >> v2:172.21.15.86:3300/0 conn(0x7f0b5810da00 0x7f0b5810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b56572700 1 -- 172.21.15.32:0/1392506024 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523435597 0 0) 0x7f0b4c000ce0 con 0x7f0b580d7ca0 2021-04-09T18:22:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b56572700 1 -- 172.21.15.32:0/1392506024 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b40002fe0 con 0x7f0b580d7ca0 2021-04-09T18:22:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b56572700 1 -- 172.21.15.32:0/1392506024 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285515280 0 0) 0x7f0b48000c60 con 0x7f0b5810d010 2021-04-09T18:22:42.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b56572700 1 -- 172.21.15.32:0/1392506024 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b40003020 con 0x7f0b5810d010 2021-04-09T18:22:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b57d75700 1 -- 172.21.15.32:0/1392506024 >> v1:172.21.15.86:6789/0 conn(0x7f0b5810d010 legacy=0x7f0b580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b57d75700 1 -- 172.21.15.32:0/1392506024 >> v1:172.21.15.32:6789/0 conn(0x7f0b580d7ca0 legacy=0x7f0b580d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.586 7f0b57d75700 1 -- 172.21.15.32:0/1392506024 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580d9220 con 0x7f0b5810da00 2021-04-09T18:22:42.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b57d75700 1 --2- 172.21.15.32:0/1392506024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5810da00 0x7f0b5810de00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f0b50006e50 tx=0x7f0b50009e70).ready entity=mon.1 client_cookie=5c90dc27e289811c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b56572700 1 -- 172.21.15.32:0/1392506024 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b50006a20 con 0x7f0b5810da00 2021-04-09T18:22:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b56572700 1 -- 172.21.15.32:0/1392506024 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b50006790 con 0x7f0b5810da00 2021-04-09T18:22:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b56572700 1 -- 172.21.15.32:0/1392506024 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b5000b700 con 0x7f0b5810da00 2021-04-09T18:22:42.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b5d8f4700 1 -- 172.21.15.32:0/1392506024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5810da00 msgr2=0x7f0b5810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b5d8f4700 1 --2- 172.21.15.32:0/1392506024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5810da00 0x7f0b5810de00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f0b50006e50 tx=0x7f0b50009e70).stop 2021-04-09T18:22:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b5d8f4700 1 -- 172.21.15.32:0/1392506024 shutdown_connections 2021-04-09T18:22:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b5d8f4700 1 --2- 172.21.15.32:0/1392506024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5810da00 0x7f0b5810de00 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b5d8f4700 1 -- 172.21.15.32:0/1392506024 >> 172.21.15.32:0/1392506024 conn(0x7f0b580ce5f0 msgr2=0x7f0b580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:42.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b5d8f4700 1 -- 172.21.15.32:0/1392506024 shutdown_connections 2021-04-09T18:22:42.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.587 7f0b5d8f4700 1 -- 172.21.15.32:0/1392506024 wait complete. 2021-04-09T18:22:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.607 7f0b5d8f4700 1 Processor -- start 2021-04-09T18:22:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b5d8f4700 1 -- start start 2021-04-09T18:22:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b5d8f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b58099920 0x7f0b58179f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:42.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b5d8f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5817a560 0x7f0b581b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b5d8f4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b581b85a0 0x7f0b581b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b5d8f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5817a960 con 0x7f0b58099920 2021-04-09T18:22:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b5d8f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b58099d20 con 0x7f0b5817a560 2021-04-09T18:22:42.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b5d8f4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b58179610 con 0x7f0b581b85a0 2021-04-09T18:22:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b57574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b58099920 0x7f0b58179f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b57574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b58099920 0x7f0b58179f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49564/0 (socket says 172.21.15.32:49564) 2021-04-09T18:22:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b57574700 1 -- 172.21.15.32:0/836257498 learned_addr learned my addr 172.21.15.32:0/836257498 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:42.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b57d75700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b581b85a0 0x7f0b581b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:42.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.608 7f0b56d73700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5817a560 0x7f0b581b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:42.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b57574700 1 -- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b581b85a0 msgr2=0x7f0b581b89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:42.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b57574700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b581b85a0 0x7f0b581b89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b57574700 1 -- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5817a560 msgr2=0x7f0b581b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:42.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b57574700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5817a560 0x7f0b581b7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b57574700 1 -- 172.21.15.32:0/836257498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58180be0 con 0x7f0b58099920 2021-04-09T18:22:42.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b57574700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b58099920 0x7f0b58179f40 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f0b48006f10 tx=0x7f0b4800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:42.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b3ffff700 1 -- 172.21.15.32:0/836257498 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b4800bd80 con 0x7f0b58099920 2021-04-09T18:22:42.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b3ffff700 1 -- 172.21.15.32:0/836257498 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b48000e20 con 0x7f0b58099920 2021-04-09T18:22:42.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b58180be0 con 0x7f0b58099920 2021-04-09T18:22:42.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b3ffff700 1 -- 172.21.15.32:0/836257498 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b480054f0 con 0x7f0b58099920 2021-04-09T18:22:42.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.609 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b58180be0 con 0x7f0b58099920 2021-04-09T18:22:42.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.610 7f0b3ffff700 1 -- 172.21.15.32:0/836257498 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0b48009fe0 con 0x7f0b58099920 2021-04-09T18:22:42.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.610 7f0b3ffff700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b38018930 0x7f0b3801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:42.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.610 7f0b3ffff700 1 -- 172.21.15.32:0/836257498 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0b48032a10 con 0x7f0b58099920 2021-04-09T18:22:42.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.610 7f0b56d73700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b38018930 0x7f0b3801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.610 7f0b56d73700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b38018930 0x7f0b3801ae10 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:42.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.630 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b581b7070 con 0x7f0b58099920 2021-04-09T18:22:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.633 7f0b3ffff700 1 -- 172.21.15.32:0/836257498 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0b48000e20 con 0x7f0b58099920 2021-04-09T18:22:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.825 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b581b7070 con 0x7f0b58099920 2021-04-09T18:22:42.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.826 7f0b3ffff700 1 -- 172.21.15.32:0/836257498 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f0b48009fe0 con 0x7f0b58099920 2021-04-09T18:22:42.881 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 127 2021-04-09T18:22:42.881 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:42.882 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":127,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56684,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1833188051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1833188051},{"type":"v1","addr":"172.21.15.86:6836","nonce":1833188051}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":123},{"gid":56690,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/46738575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46738575},{"type":"v1","addr":"172.21.15.32:6834","nonce":46738575}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":123},{"gid":56693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/647788630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":647788630},{"type":"v1","addr":"172.21.15.32:6835","nonce":647788630}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:22:42.405857","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:22:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b38018930 msgr2=0x7f0b3801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b38018930 0x7f0b3801ae10 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b58099920 msgr2=0x7f0b58179f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b58099920 0x7f0b58179f40 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f0b48006f10 tx=0x7f0b4800b990).stop 2021-04-09T18:22:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 shutdown_connections 2021-04-09T18:22:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b38018930 0x7f0b3801ae10 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b581b85a0 0x7f0b581b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5817a560 0x7f0b581b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 --2- 172.21.15.32:0/836257498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b58099920 0x7f0b58179f40 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 >> 172.21.15.32:0/836257498 conn(0x7f0b581aa2b0 msgr2=0x7f0b581a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:42.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 shutdown_connections 2021-04-09T18:22:42.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.828 7f0b5d8f4700 1 -- 172.21.15.32:0/836257498 wait complete. 2021-04-09T18:22:42.893 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:22:43.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.987 7fae84edf700 1 Processor -- start 2021-04-09T18:22:43.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.987 7fae84edf700 1 -- start start 2021-04-09T18:22:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae84edf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae800d7ca0 0x7fae800d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae84edf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8010da00 0x7fae8010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae84edf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8010d480 con 0x7fae8010d010 2021-04-09T18:22:43.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae84edf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae800d9220 con 0x7fae800d7ca0 2021-04-09T18:22:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae84edf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae800d86c0 con 0x7fae8010da00 2021-04-09T18:22:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7e312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae800d7ca0 0x7fae800d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7eb13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fae8010d010 0x7fae800d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59162/0 (socket says 172.21.15.32:59162) 2021-04-09T18:22:43.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7eb13700 1 -- 172.21.15.32:0/325891117 learned_addr learned my addr 172.21.15.32:0/325891117 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7f314700 1 --2- 172.21.15.32:0/325891117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8010da00 0x7fae8010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7e312700 1 -- 172.21.15.32:0/325891117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8010da00 msgr2=0x7fae8010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:43.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7e312700 1 --2- 172.21.15.32:0/325891117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8010da00 0x7fae8010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7e312700 1 -- 172.21.15.32:0/325891117 >> v1:172.21.15.32:6789/0 conn(0x7fae8010d010 legacy=0x7fae800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7e312700 1 -- 172.21.15.32:0/325891117 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae800d1f70 con 0x7fae800d7ca0 2021-04-09T18:22:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7db11700 1 -- 172.21.15.32:0/325891117 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803562168 0 0) 0x7fae70000ce0 con 0x7fae8010d010 2021-04-09T18:22:43.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7f314700 1 --2- 172.21.15.32:0/325891117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8010da00 0x7fae8010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:43.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7e312700 1 --2- 172.21.15.32:0/325891117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae800d7ca0 0x7fae800d80a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7fae74006650 tx=0x7fae74009a80).ready entity=mon.0 client_cookie=3d65e686646c38a8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:43.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7db11700 1 -- 172.21.15.32:0/325891117 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae740009b0 con 0x7fae800d7ca0 2021-04-09T18:22:43.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7db11700 1 -- 172.21.15.32:0/325891117 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7400c0c0 con 0x7fae800d7ca0 2021-04-09T18:22:43.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae7db11700 1 -- 172.21.15.32:0/325891117 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae7400c510 con 0x7fae800d7ca0 2021-04-09T18:22:43.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.988 7fae84edf700 1 -- 172.21.15.32:0/325891117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae800d7ca0 msgr2=0x7fae800d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.989 7fae84edf700 1 --2- 172.21.15.32:0/325891117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae800d7ca0 0x7fae800d80a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7fae74006650 tx=0x7fae74009a80).stop 2021-04-09T18:22:43.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.989 7fae84edf700 1 -- 172.21.15.32:0/325891117 shutdown_connections 2021-04-09T18:22:43.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.989 7fae84edf700 1 --2- 172.21.15.32:0/325891117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8010da00 0x7fae8010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.989 7fae84edf700 1 --2- 172.21.15.32:0/325891117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae800d7ca0 0x7fae800d80a0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.989 7fae84edf700 1 -- 172.21.15.32:0/325891117 >> 172.21.15.32:0/325891117 conn(0x7fae800ce5f0 msgr2=0x7fae800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:43.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.989 7fae84edf700 1 -- 172.21.15.32:0/325891117 shutdown_connections 2021-04-09T18:22:43.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:42.989 7fae84edf700 1 -- 172.21.15.32:0/325891117 wait complete. 2021-04-09T18:22:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.009 7fae84edf700 1 Processor -- start 2021-04-09T18:22:43.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.009 7fae84edf700 1 -- start start 2021-04-09T18:22:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae84edf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae801aca20 0x7fae801ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae84edf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8018bd40 0x7fae8018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae84edf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae8008fba0 0x7fae8008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae84edf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8018c7e0 con 0x7fae8008fba0 2021-04-09T18:22:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae84edf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8018ca30 con 0x7fae801aca20 2021-04-09T18:22:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae84edf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae800905f0 con 0x7fae8018bd40 2021-04-09T18:22:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7f314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae8008fba0 0x7fae8008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7f314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae8008fba0 0x7fae8008ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49580/0 (socket says 172.21.15.32:49580) 2021-04-09T18:22:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7f314700 1 -- 172.21.15.32:0/4091452524 learned_addr learned my addr 172.21.15.32:0/4091452524 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7e312700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8018bd40 0x7fae8018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7eb13700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae801aca20 0x7fae801ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7f314700 1 -- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8018bd40 msgr2=0x7fae8018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7f314700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8018bd40 0x7fae8018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7f314700 1 -- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae801aca20 msgr2=0x7fae801ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7f314700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae801aca20 0x7fae801ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7f314700 1 -- 172.21.15.32:0/4091452524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae800b4300 con 0x7fae8008fba0 2021-04-09T18:22:43.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.010 7fae7e312700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8018bd40 0x7fae8018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:43.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.011 7fae7f314700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae8008fba0 0x7fae8008ffa0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7fae78009980 tx=0x7fae7800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:43.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.011 7fae6f7fe700 1 -- 172.21.15.32:0/4091452524 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae7800c2f0 con 0x7fae8008fba0 2021-04-09T18:22:43.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.011 7fae6f7fe700 1 -- 172.21.15.32:0/4091452524 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae78006790 con 0x7fae8008fba0 2021-04-09T18:22:43.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.011 7fae84edf700 1 -- 172.21.15.32:0/4091452524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae800b4300 con 0x7fae8008fba0 2021-04-09T18:22:43.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.011 7fae6f7fe700 1 -- 172.21.15.32:0/4091452524 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae7800c7e0 con 0x7fae8008fba0 2021-04-09T18:22:43.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.011 7fae84edf700 1 -- 172.21.15.32:0/4091452524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae800b4300 con 0x7fae8008fba0 2021-04-09T18:22:43.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.012 7fae6f7fe700 1 -- 172.21.15.32:0/4091452524 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fae780011f0 con 0x7fae8008fba0 2021-04-09T18:22:43.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.012 7fae6f7fe700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae640188c0 0x7fae6401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.012 7fae6f7fe700 1 -- 172.21.15.32:0/4091452524 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fae78032b50 con 0x7fae8008fba0 2021-04-09T18:22:43.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.012 7fae7eb13700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae640188c0 0x7fae6401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.012 7fae7eb13700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae640188c0 0x7fae6401adc0 crc :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:43.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.032 7fae84edf700 1 -- 172.21.15.32:0/4091452524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae80182d70 con 0x7fae8008fba0 2021-04-09T18:22:43.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.035 7fae6f7fe700 1 -- 172.21.15.32:0/4091452524 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fae7800be10 con 0x7fae8008fba0 2021-04-09T18:22:43.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.222 7fae84edf700 1 -- 172.21.15.32:0/4091452524 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae8018c760 con 0x7fae8008fba0 2021-04-09T18:22:43.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.223 7fae6f7fe700 1 -- 172.21.15.32:0/4091452524 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+14006 (secure 0 0 0) 0x7fae780011f0 con 0x7fae8008fba0 2021-04-09T18:22:43.278 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:43.278 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":107,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:22:41.390553","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:22:37.739108","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-04-09 18:22:39.152543","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:22:43.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 -- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae640188c0 msgr2=0x7fae6401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae640188c0 0x7fae6401adc0 crc :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 -- 172.21.15.32:0/4091452524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae8008fba0 msgr2=0x7fae8008ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae8008fba0 0x7fae8008ffa0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7fae78009980 tx=0x7fae7800bcd0).stop 2021-04-09T18:22:43.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 -- 172.21.15.32:0/4091452524 shutdown_connections 2021-04-09T18:22:43.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae640188c0 0x7fae6401adc0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae8008fba0 0x7fae8008ffa0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8018bd40 0x7fae8018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 --2- 172.21.15.32:0/4091452524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae801aca20 0x7fae801ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 -- 172.21.15.32:0/4091452524 >> 172.21.15.32:0/4091452524 conn(0x7fae8019b460 msgr2=0x7fae8019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:43.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 -- 172.21.15.32:0/4091452524 shutdown_connections 2021-04-09T18:22:43.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.225 7fae84edf700 1 -- 172.21.15.32:0/4091452524 wait complete. 2021-04-09T18:22:43.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:22:43.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.395 7f0d8168e700 1 Processor -- start 2021-04-09T18:22:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.395 7f0d8168e700 1 -- start start 2021-04-09T18:22:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.396 7f0d8168e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c0d7c50 0x7f0d7c1b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.396 7f0d8168e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c0d8620 con 0x7f0d7c0cffb0 2021-04-09T18:22:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.396 7f0d8168e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c0d2800 con 0x7f0d7c0d8050 2021-04-09T18:22:43.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.396 7f0d8168e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0d91f0 con 0x7f0d7c0d7c50 2021-04-09T18:22:43.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.396 7f0d7b2bf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0d7c0cffb0 0x7f0d7c087e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59176/0 (socket says 172.21.15.32:59176) 2021-04-09T18:22:43.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.396 7f0d7b2bf700 1 -- 172.21.15.32:0/548643269 learned_addr learned my addr 172.21.15.32:0/548643269 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:43.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.396 7f0d7aabe700 1 --2- 172.21.15.32:0/548643269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c0d7c50 0x7f0d7c1b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.396 7f0d7a2bd700 1 -- 172.21.15.32:0/548643269 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434858593 0 0) 0x7f0d6c000ce0 con 0x7f0d7c0cffb0 2021-04-09T18:22:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.396 7f0d7a2bd700 1 -- 172.21.15.32:0/548643269 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d64002fe0 con 0x7f0d7c0cffb0 2021-04-09T18:22:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7a2bd700 1 -- 172.21.15.32:0/548643269 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106783322 0 0) 0x7f0d74000d30 con 0x7f0d7c0d8050 2021-04-09T18:22:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7a2bd700 1 -- 172.21.15.32:0/548643269 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d64003210 con 0x7f0d7c0d8050 2021-04-09T18:22:43.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7a2bd700 1 -- 172.21.15.32:0/548643269 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 506397018 0 0) 0x7f0d6c0010e0 con 0x7f0d7c0cffb0 2021-04-09T18:22:43.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7aabe700 1 -- 172.21.15.32:0/548643269 >> v1:172.21.15.86:6789/0 conn(0x7f0d7c0d8050 legacy=0x7f0d7c10d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7aabe700 1 -- 172.21.15.32:0/548643269 >> v1:172.21.15.32:6789/0 conn(0x7f0d7c0cffb0 legacy=0x7f0d7c087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7aabe700 1 -- 172.21.15.32:0/548643269 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c1b0ab0 con 0x7f0d7c0d7c50 2021-04-09T18:22:43.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7aabe700 1 --2- 172.21.15.32:0/548643269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c0d7c50 0x7f0d7c1b0490 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f0d70006650 tx=0x7f0d70009a80).ready entity=mon.2 client_cookie=a6326c2357ebe2c8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:43.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7a2bd700 1 -- 172.21.15.32:0/548643269 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d700009b0 con 0x7f0d7c0d7c50 2021-04-09T18:22:43.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7a2bd700 1 -- 172.21.15.32:0/548643269 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7000c0c0 con 0x7f0d7c0d7c50 2021-04-09T18:22:43.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d7a2bd700 1 -- 172.21.15.32:0/548643269 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d7000c510 con 0x7f0d7c0d7c50 2021-04-09T18:22:43.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d8168e700 1 -- 172.21.15.32:0/548643269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c0d7c50 msgr2=0x7f0d7c1b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.397 7f0d8168e700 1 --2- 172.21.15.32:0/548643269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c0d7c50 0x7f0d7c1b0490 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f0d70006650 tx=0x7f0d70009a80).stop 2021-04-09T18:22:43.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.398 7f0d8168e700 1 -- 172.21.15.32:0/548643269 shutdown_connections 2021-04-09T18:22:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.398 7f0d8168e700 1 --2- 172.21.15.32:0/548643269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c0d7c50 0x7f0d7c1b0490 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.398 7f0d8168e700 1 -- 172.21.15.32:0/548643269 >> 172.21.15.32:0/548643269 conn(0x7f0d7c0ce5b0 msgr2=0x7f0d7c0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.398 7f0d8168e700 1 -- 172.21.15.32:0/548643269 shutdown_connections 2021-04-09T18:22:43.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.398 7f0d8168e700 1 -- 172.21.15.32:0/548643269 wait complete. 2021-04-09T18:22:43.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.418 7f0d8168e700 1 Processor -- start 2021-04-09T18:22:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.418 7f0d8168e700 1 -- start start 2021-04-09T18:22:43.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d8168e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d7c09d680 0x7f0d7c09afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d8168e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d7c09da80 0x7f0d7c191dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d8168e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c1923f0 0x7f0d7c09be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d8168e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c09cbe0 con 0x7f0d7c09da80 2021-04-09T18:22:43.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d8168e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c09b8e0 con 0x7f0d7c09d680 2021-04-09T18:22:43.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d8168e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c09b5c0 con 0x7f0d7c1923f0 2021-04-09T18:22:43.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7aabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d7c09da80 0x7f0d7c191dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7aabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d7c09da80 0x7f0d7c191dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49592/0 (socket says 172.21.15.32:49592) 2021-04-09T18:22:43.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7aabe700 1 -- 172.21.15.32:0/3241088311 learned_addr learned my addr 172.21.15.32:0/3241088311 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7bac0700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c1923f0 0x7f0d7c09be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7b2bf700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d7c09d680 0x7f0d7c09afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7aabe700 1 -- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c1923f0 msgr2=0x7f0d7c09be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7aabe700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c1923f0 0x7f0d7c09be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7aabe700 1 -- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d7c09d680 msgr2=0x7f0d7c09afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7aabe700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d7c09d680 0x7f0d7c09afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7aabe700 1 -- 172.21.15.32:0/3241088311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c09c480 con 0x7f0d7c09da80 2021-04-09T18:22:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d7aabe700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d7c09da80 0x7f0d7c191dd0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f0d70009650 tx=0x7f0d7000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:43.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d63fff700 1 -- 172.21.15.32:0/3241088311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d700093a0 con 0x7f0d7c09da80 2021-04-09T18:22:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d63fff700 1 -- 172.21.15.32:0/3241088311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7000bd50 con 0x7f0d7c09da80 2021-04-09T18:22:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d63fff700 1 -- 172.21.15.32:0/3241088311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d700052e0 con 0x7f0d7c09da80 2021-04-09T18:22:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7c09c480 con 0x7f0d7c09da80 2021-04-09T18:22:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.419 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7c09c480 con 0x7f0d7c09da80 2021-04-09T18:22:43.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.420 7f0d63fff700 1 -- 172.21.15.32:0/3241088311 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0d70006030 con 0x7f0d7c09da80 2021-04-09T18:22:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.420 7f0d63fff700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d5c018930 0x7f0d5c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.420 7f0d63fff700 1 -- 172.21.15.32:0/3241088311 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0d700329e0 con 0x7f0d7c09da80 2021-04-09T18:22:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.420 7f0d7b2bf700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d5c018930 0x7f0d5c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.421 7f0d7b2bf700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d5c018930 0x7f0d5c01ae10 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:43.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.440 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7c093fc0 con 0x7f0d7c09da80 2021-04-09T18:22:43.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.444 7f0d63fff700 1 -- 172.21.15.32:0/3241088311 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0d700009c0 con 0x7f0d7c09da80 2021-04-09T18:22:43.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.630 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0d7c093fc0 con 0x7f0d7c09da80 2021-04-09T18:22:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.633 7f0d63fff700 1 -- 172.21.15.32:0/3241088311 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v28) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0d70006030 con 0x7f0d7c09da80 2021-04-09T18:22:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:22:43.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d5c018930 msgr2=0x7f0d5c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d5c018930 0x7f0d5c01ae10 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d7c09da80 msgr2=0x7f0d7c191dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d7c09da80 0x7f0d7c191dd0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f0d70009650 tx=0x7f0d7000b960).stop 2021-04-09T18:22:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 shutdown_connections 2021-04-09T18:22:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d5c018930 0x7f0d5c01ae10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d7c09da80 0x7f0d7c191dd0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d7c1923f0 0x7f0d7c09be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 --2- 172.21.15.32:0/3241088311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d7c09d680 0x7f0d7c09afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 >> 172.21.15.32:0/3241088311 conn(0x7f0d7c0adaf0 msgr2=0x7f0d7c0bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.639 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 shutdown_connections 2021-04-09T18:22:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.640 7f0d8168e700 1 -- 172.21.15.32:0/3241088311 wait complete. 2021-04-09T18:22:43.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:22:43.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.810 7fc939562700 1 Processor -- start 2021-04-09T18:22:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.810 7fc939562700 1 -- start start 2021-04-09T18:22:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc939562700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340d7c50 0x7fc9341b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc939562700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc9340d8050 0x7fc93410d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc939562700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9340d8620 con 0x7fc9340cffb0 2021-04-09T18:22:43.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc939562700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9340d91f0 con 0x7fc9340d8050 2021-04-09T18:22:43.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc939562700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340d2800 con 0x7fc9340d7c50 2021-04-09T18:22:43.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc933ac0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc9340d8050 0x7fc93410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc932abe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340d7c50 0x7fc9341b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc9332bf700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc9340cffb0 0x7fc934087e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:57116/0 (socket says 172.21.15.32:57116) 2021-04-09T18:22:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc9332bf700 1 -- 172.21.15.32:0/2136084177 learned_addr learned my addr 172.21.15.32:0/2136084177 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc933ac0700 1 -- 172.21.15.32:0/2136084177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340d7c50 msgr2=0x7fc9341b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc933ac0700 1 --2- 172.21.15.32:0/2136084177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340d7c50 0x7fc9341b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc933ac0700 1 -- 172.21.15.32:0/2136084177 >> v1:172.21.15.86:6789/0 conn(0x7fc9340cffb0 legacy=0x7fc934087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.811 7fc9322bd700 1 -- 172.21.15.32:0/2136084177 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848899841 0 0) 0x7fc924001440 con 0x7fc9340cffb0 2021-04-09T18:22:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.812 7fc933ac0700 1 -- 172.21.15.32:0/2136084177 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9341b0ab0 con 0x7fc9340d8050 2021-04-09T18:22:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.812 7fc932abe700 1 --2- 172.21.15.32:0/2136084177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340d7c50 0x7fc9341b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.812 7fc933ac0700 1 --2- 172.21.15.32:0/2136084177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9340d8050 0x7fc93410d200 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fc92c006ad0 tx=0x7fc92c009ad0).ready entity=mon.1 client_cookie=b1dce4c6f2eeba28 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:43.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.812 7fc9322bd700 1 -- 172.21.15.32:0/2136084177 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc92c0022e0 con 0x7fc9340d8050 2021-04-09T18:22:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.812 7fc9322bd700 1 -- 172.21.15.32:0/2136084177 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92c006410 con 0x7fc9340d8050 2021-04-09T18:22:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.812 7fc939562700 1 -- 172.21.15.32:0/2136084177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9340d8050 msgr2=0x7fc93410d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.812 7fc939562700 1 --2- 172.21.15.32:0/2136084177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9340d8050 0x7fc93410d200 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fc92c006ad0 tx=0x7fc92c009ad0).stop 2021-04-09T18:22:43.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.812 7fc9322bd700 1 -- 172.21.15.32:0/2136084177 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc92c000c10 con 0x7fc9340d8050 2021-04-09T18:22:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.813 7fc939562700 1 -- 172.21.15.32:0/2136084177 shutdown_connections 2021-04-09T18:22:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.813 7fc939562700 1 --2- 172.21.15.32:0/2136084177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9340d8050 0x7fc93410d200 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.813 7fc939562700 1 --2- 172.21.15.32:0/2136084177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340d7c50 0x7fc9341b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.813 7fc939562700 1 -- 172.21.15.32:0/2136084177 >> 172.21.15.32:0/2136084177 conn(0x7fc9340ce5b0 msgr2=0x7fc9340cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:43.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.813 7fc939562700 1 -- 172.21.15.32:0/2136084177 shutdown_connections 2021-04-09T18:22:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.813 7fc939562700 1 -- 172.21.15.32:0/2136084177 wait complete. 2021-04-09T18:22:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.833 7fc939562700 1 Processor -- start 2021-04-09T18:22:43.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.833 7fc939562700 1 -- start start 2021-04-09T18:22:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.833 7fc939562700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340c0cd0 0x7fc9340a2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.833 7fc939562700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc93409afb0 0x7fc93409b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.833 7fc939562700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc934182890 0x7fc934182c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc939562700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9341832b0 con 0x7fc934182890 2021-04-09T18:22:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc939562700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc934183510 con 0x7fc93409afb0 2021-04-09T18:22:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc939562700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340a24b0 con 0x7fc9340c0cd0 2021-04-09T18:22:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc933ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc934182890 0x7fc934182c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc933ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc934182890 0x7fc934182c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49608/0 (socket says 172.21.15.32:49608) 2021-04-09T18:22:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc933ac0700 1 -- 172.21.15.32:0/2666625289 learned_addr learned my addr 172.21.15.32:0/2666625289 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc932abe700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc93409afb0 0x7fc93409b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc9332bf700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340c0cd0 0x7fc9340a2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc933ac0700 1 -- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340c0cd0 msgr2=0x7fc9340a2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc933ac0700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340c0cd0 0x7fc9340a2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc933ac0700 1 -- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc93409afb0 msgr2=0x7fc93409b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc933ac0700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc93409afb0 0x7fc93409b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc933ac0700 1 -- 172.21.15.32:0/2666625289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9340a2f90 con 0x7fc934182890 2021-04-09T18:22:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc933ac0700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc934182890 0x7fc934182c90 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7fc92c00b660 tx=0x7fc92c00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:43.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc92bfff700 1 -- 172.21.15.32:0/2666625289 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc92c002280 con 0x7fc934182890 2021-04-09T18:22:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc92bfff700 1 -- 172.21.15.32:0/2666625289 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92c006370 con 0x7fc934182890 2021-04-09T18:22:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc939562700 1 -- 172.21.15.32:0/2666625289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9340a2f90 con 0x7fc934182890 2021-04-09T18:22:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc92bfff700 1 -- 172.21.15.32:0/2666625289 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc92c0025c0 con 0x7fc934182890 2021-04-09T18:22:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.834 7fc939562700 1 -- 172.21.15.32:0/2666625289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9340a2f90 con 0x7fc934182890 2021-04-09T18:22:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.835 7fc92bfff700 1 -- 172.21.15.32:0/2666625289 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc92c000c10 con 0x7fc934182890 2021-04-09T18:22:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.835 7fc92bfff700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9180188d0 0x7fc91801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.835 7fc92bfff700 1 -- 172.21.15.32:0/2666625289 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc92c032a40 con 0x7fc934182890 2021-04-09T18:22:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.835 7fc9332bf700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9180188d0 0x7fc91801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:43.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.836 7fc9332bf700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9180188d0 0x7fc91801add0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.855 7fc939562700 1 -- 172.21.15.32:0/2666625289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc934183510 con 0x7fc934182890 2021-04-09T18:22:43.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:43.859 7fc92bfff700 1 -- 172.21.15.32:0/2666625289 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc92c01f780 con 0x7fc934182890 2021-04-09T18:22:44.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.043 7fc939562700 1 -- 172.21.15.32:0/2666625289 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc934183510 con 0x7fc934182890 2021-04-09T18:22:44.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.046 7fc92bfff700 1 -- 172.21.15.32:0/2666625289 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v29) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc92c000c10 con 0x7fc934182890 2021-04-09T18:22:44.104 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:22:44.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 -- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9180188d0 msgr2=0x7fc91801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:44.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9180188d0 0x7fc91801add0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 -- 172.21.15.32:0/2666625289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc934182890 msgr2=0x7fc934182c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:44.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc934182890 0x7fc934182c90 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7fc92c00b660 tx=0x7fc92c00ba00).stop 2021-04-09T18:22:44.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 -- 172.21.15.32:0/2666625289 shutdown_connections 2021-04-09T18:22:44.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9180188d0 0x7fc91801add0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc934182890 0x7fc934182c90 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc93409afb0 0x7fc93409b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 --2- 172.21.15.32:0/2666625289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9340c0cd0 0x7fc9340a2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 -- 172.21.15.32:0/2666625289 >> 172.21.15.32:0/2666625289 conn(0x7fc9340adaf0 msgr2=0x7fc9340bdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 -- 172.21.15.32:0/2666625289 shutdown_connections 2021-04-09T18:22:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.052 7fc939562700 1 -- 172.21.15.32:0/2666625289 wait complete. 2021-04-09T18:22:44.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:44.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.211 7f1b601ff700 1 Processor -- start 2021-04-09T18:22:44.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.211 7f1b601ff700 1 -- start start 2021-04-09T18:22:44.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b601ff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b580d6c80 0x7f1b580d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:44.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b601ff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1b580d2440 0x7f1b5810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:44.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b601ff700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b580d7280 con 0x7f1b580cffa0 2021-04-09T18:22:44.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b601ff700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b580d7080 con 0x7f1b580d6c80 2021-04-09T18:22:44.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b601ff700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b580d91e0 con 0x7f1b580d2440 2021-04-09T18:22:44.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b5d81a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b580d6c80 0x7f1b580d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:44.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b5e01b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1b580cffa0 0x7f1b580d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59204/0 (socket says 172.21.15.32:59204) 2021-04-09T18:22:44.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b5e01b700 1 -- 172.21.15.32:0/186236694 learned_addr learned my addr 172.21.15.32:0/186236694 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:44.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b5e81c700 1 --2- 172.21.15.32:0/186236694 >> v2:172.21.15.86:3300/0 conn(0x7f1b580d2440 0x7f1b5810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:44.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b5d81a700 1 -- 172.21.15.32:0/186236694 >> v2:172.21.15.86:3300/0 conn(0x7f1b580d2440 msgr2=0x7f1b5810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:44.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b5d81a700 1 --2- 172.21.15.32:0/186236694 >> v2:172.21.15.86:3300/0 conn(0x7f1b580d2440 0x7f1b5810cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.212 7f1b5d81a700 1 -- 172.21.15.32:0/186236694 >> v1:172.21.15.32:6789/0 conn(0x7f1b580cffa0 legacy=0x7f1b580d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:44.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.213 7f1b5d81a700 1 -- 172.21.15.32:0/186236694 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b580d2840 con 0x7f1b580d6c80 2021-04-09T18:22:44.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.213 7f1b5d019700 1 -- 172.21.15.32:0/186236694 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900748394 0 0) 0x7f1b54001060 con 0x7f1b580cffa0 2021-04-09T18:22:44.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.213 7f1b5d81a700 1 --2- 172.21.15.32:0/186236694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b580d6c80 0x7f1b580d7ff0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f1b48006650 tx=0x7f1b48009a00).ready entity=mon.0 client_cookie=b26d872c9fd79c93 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:44.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.213 7f1b5d019700 1 -- 172.21.15.32:0/186236694 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b48009df0 con 0x7f1b580d6c80 2021-04-09T18:22:44.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.213 7f1b5d019700 1 -- 172.21.15.32:0/186236694 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b480009b0 con 0x7f1b580d6c80 2021-04-09T18:22:44.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.213 7f1b5d019700 1 -- 172.21.15.32:0/186236694 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b4800c2b0 con 0x7f1b580d6c80 2021-04-09T18:22:44.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.213 7f1b601ff700 1 -- 172.21.15.32:0/186236694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b580d6c80 msgr2=0x7f1b580d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:44.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.213 7f1b601ff700 1 --2- 172.21.15.32:0/186236694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b580d6c80 0x7f1b580d7ff0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f1b48006650 tx=0x7f1b48009a00).stop 2021-04-09T18:22:44.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.214 7f1b601ff700 1 -- 172.21.15.32:0/186236694 shutdown_connections 2021-04-09T18:22:44.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.214 7f1b601ff700 1 --2- 172.21.15.32:0/186236694 >> v2:172.21.15.86:3300/0 conn(0x7f1b580d2440 0x7f1b5810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.214 7f1b601ff700 1 --2- 172.21.15.32:0/186236694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b580d6c80 0x7f1b580d7ff0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.214 7f1b601ff700 1 -- 172.21.15.32:0/186236694 >> 172.21.15.32:0/186236694 conn(0x7f1b580ce5c0 msgr2=0x7f1b580cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:44.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.214 7f1b601ff700 1 -- 172.21.15.32:0/186236694 shutdown_connections 2021-04-09T18:22:44.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.214 7f1b601ff700 1 -- 172.21.15.32:0/186236694 wait complete. 2021-04-09T18:22:44.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.234 7f1b601ff700 1 Processor -- start 2021-04-09T18:22:44.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.234 7f1b601ff700 1 -- start start 2021-04-09T18:22:44.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.234 7f1b601ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b581929f0 0x7f1b58192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:44.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.234 7f1b601ff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b5808ddc0 0x7f1b5808e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:44.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.234 7f1b601ff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5808e7e0 0x7f1b5808c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:44.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.234 7f1b601ff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58193410 con 0x7f1b581929f0 2021-04-09T18:22:44.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.234 7f1b601ff700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580a9140 con 0x7f1b5808e7e0 2021-04-09T18:22:44.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.234 7f1b601ff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5808ce80 con 0x7f1b5808ddc0 2021-04-09T18:22:44.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e01b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b581929f0 0x7f1b58192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:44.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e01b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b581929f0 0x7f1b58192df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49620/0 (socket says 172.21.15.32:49620) 2021-04-09T18:22:44.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e01b700 1 -- 172.21.15.32:0/88436442 learned_addr learned my addr 172.21.15.32:0/88436442 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:44.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e81c700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5808e7e0 0x7f1b5808c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:44.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5d81a700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b5808ddc0 0x7f1b5808e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:44.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e01b700 1 -- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b5808ddc0 msgr2=0x7f1b5808e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:44.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e01b700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b5808ddc0 0x7f1b5808e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e01b700 1 -- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5808e7e0 msgr2=0x7f1b5808c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e01b700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5808e7e0 0x7f1b5808c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e01b700 1 -- 172.21.15.32:0/88436442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5808d0a0 con 0x7f1b581929f0 2021-04-09T18:22:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e81c700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5808e7e0 0x7f1b5808c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5d81a700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b5808ddc0 0x7f1b5808e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:44.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b5e01b700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b581929f0 0x7f1b58192df0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f1b54009980 tx=0x7f1b5400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:44.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b4effd700 1 -- 172.21.15.32:0/88436442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b54006340 con 0x7f1b581929f0 2021-04-09T18:22:44.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b4effd700 1 -- 172.21.15.32:0/88436442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b54006640 con 0x7f1b581929f0 2021-04-09T18:22:44.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b601ff700 1 -- 172.21.15.32:0/88436442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5808d0a0 con 0x7f1b581929f0 2021-04-09T18:22:44.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b4effd700 1 -- 172.21.15.32:0/88436442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b54004910 con 0x7f1b581929f0 2021-04-09T18:22:44.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.235 7f1b601ff700 1 -- 172.21.15.32:0/88436442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5808d0a0 con 0x7f1b581929f0 2021-04-09T18:22:44.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.236 7f1b4effd700 1 -- 172.21.15.32:0/88436442 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1b5400a000 con 0x7f1b581929f0 2021-04-09T18:22:44.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.236 7f1b4effd700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b400188c0 0x7f1b4001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:44.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.236 7f1b4effd700 1 -- 172.21.15.32:0/88436442 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1b54032c50 con 0x7f1b581929f0 2021-04-09T18:22:44.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.236 7f1b5d81a700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b400188c0 0x7f1b4001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:44.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.236 7f1b5d81a700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b400188c0 0x7f1b4001adc0 crc :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:44.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.256 7f1b601ff700 1 -- 172.21.15.32:0/88436442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b581b70c0 con 0x7f1b581929f0 2021-04-09T18:22:44.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.260 7f1b4effd700 1 -- 172.21.15.32:0/88436442 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1b5400be10 con 0x7f1b581929f0 2021-04-09T18:22:44.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.453 7f1b601ff700 1 -- 172.21.15.32:0/88436442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b580d7280 con 0x7f1b581929f0 2021-04-09T18:22:44.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.453 7f1b4effd700 1 -- 172.21.15.32:0/88436442 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f1b5400a000 con 0x7f1b581929f0 2021-04-09T18:22:44.508 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 127 2021-04-09T18:22:44.509 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:44.509 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":127,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56684,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1833188051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1833188051},{"type":"v1","addr":"172.21.15.86:6836","nonce":1833188051}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":123},{"gid":56690,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/46738575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46738575},{"type":"v1","addr":"172.21.15.32:6834","nonce":46738575}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":123},{"gid":56693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/647788630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":647788630},{"type":"v1","addr":"172.21.15.32:6835","nonce":647788630}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:22:42.405857","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:22:44.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 -- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b400188c0 msgr2=0x7f1b4001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:44.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b400188c0 0x7f1b4001adc0 crc :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 -- 172.21.15.32:0/88436442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b581929f0 msgr2=0x7f1b58192df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:44.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b581929f0 0x7f1b58192df0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f1b54009980 tx=0x7f1b5400bcd0).stop 2021-04-09T18:22:44.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 -- 172.21.15.32:0/88436442 shutdown_connections 2021-04-09T18:22:44.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b400188c0 0x7f1b4001adc0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5808e7e0 0x7f1b5808c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b5808ddc0 0x7f1b5808e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 --2- 172.21.15.32:0/88436442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b581929f0 0x7f1b58192df0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:44.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 -- 172.21.15.32:0/88436442 >> 172.21.15.32:0/88436442 conn(0x7f1b58189b00 msgr2=0x7f1b581921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:44.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 -- 172.21.15.32:0/88436442 shutdown_connections 2021-04-09T18:22:44.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:44.456 7f1b601ff700 1 -- 172.21.15.32:0/88436442 wait complete. 2021-04-09T18:22:44.923 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 127, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:22:40.389392', 'modified': '2021-04-09 18:22:42.405857', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7777}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7777': {'gid': 7777, 'name': 'b', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/1938812020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1938812020}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1938812020}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:44.923 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:22:44.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:22:45.052 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:22:45.052 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:22:45.053 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:22:45.053 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 7777, 2021-04-09T18:22:45.053 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:22:45.053 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:22:45.053 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.6076345549999997, 2021-04-09T18:22:45.054 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 127, 2021-04-09T18:22:45.054 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 107, 2021-04-09T18:22:45.054 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 107, 2021-04-09T18:22:45.054 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10.076409439000001 2021-04-09T18:22:45.055 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:22:45.058 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 7777, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.6076345549999997, "mdsmap_epoch": 127, "osdmap_epoch": 107, "osdmap_epoch_barrier": 107, "uptime": 10.076409439000001 } 2021-04-09T18:22:45.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:22:45.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.154 7fefb9739700 1 Processor -- start 2021-04-09T18:22:45.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.154 7fefb9739700 1 -- start start 2021-04-09T18:22:45.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb9739700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fefb410d010 0x7fefb40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb9739700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb40d1da0 0x7fefb40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb9739700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb410d480 con 0x7fefb40d82c0 2021-04-09T18:22:45.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb9739700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb40d9220 con 0x7fefb410d010 2021-04-09T18:22:45.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb9739700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40d86c0 con 0x7fefb40d1da0 2021-04-09T18:22:45.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb3d4a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fefb40d82c0 0x7fefb410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59220/0 (socket says 172.21.15.32:59220) 2021-04-09T18:22:45.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb3d4a700 1 -- 172.21.15.32:0/1736515231 learned_addr learned my addr 172.21.15.32:0/1736515231 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:45.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb3549700 1 --2- 172.21.15.32:0/1736515231 >> v2:172.21.15.86:3300/0 conn(0x7fefb410d010 0x7fefb40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb2d48700 1 --2- 172.21.15.32:0/1736515231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb40d1da0 0x7fefb40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb2547700 1 -- 172.21.15.32:0/1736515231 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746886763 0 0) 0x7fefac000d30 con 0x7fefb40d82c0 2021-04-09T18:22:45.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb2547700 1 -- 172.21.15.32:0/1736515231 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c002fe0 con 0x7fefb40d82c0 2021-04-09T18:22:45.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb3549700 1 -- 172.21.15.32:0/1736515231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb40d1da0 msgr2=0x7fefb40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:45.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb3549700 1 --2- 172.21.15.32:0/1736515231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb40d1da0 0x7fefb40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb3549700 1 -- 172.21.15.32:0/1736515231 >> v1:172.21.15.32:6789/0 conn(0x7fefb40d82c0 legacy=0x7fefb410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb3549700 1 -- 172.21.15.32:0/1736515231 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb410e1f0 con 0x7fefb410d010 2021-04-09T18:22:45.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb2d48700 1 --2- 172.21.15.32:0/1736515231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb40d1da0 0x7fefb40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:45.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.155 7fefb3549700 1 --2- 172.21.15.32:0/1736515231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb410d010 0x7fefb40d1780 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7fefa4006650 tx=0x7fefa4009a80).ready entity=mon.1 client_cookie=bcdef794b1271bc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:45.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb2547700 1 -- 172.21.15.32:0/1736515231 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefa40009b0 con 0x7fefb410d010 2021-04-09T18:22:45.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb2547700 1 -- 172.21.15.32:0/1736515231 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa400c0c0 con 0x7fefb410d010 2021-04-09T18:22:45.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb2547700 1 -- 172.21.15.32:0/1736515231 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefa400c510 con 0x7fefb410d010 2021-04-09T18:22:45.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb9739700 1 -- 172.21.15.32:0/1736515231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb410d010 msgr2=0x7fefb40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb9739700 1 --2- 172.21.15.32:0/1736515231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb410d010 0x7fefb40d1780 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7fefa4006650 tx=0x7fefa4009a80).stop 2021-04-09T18:22:45.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb9739700 1 -- 172.21.15.32:0/1736515231 shutdown_connections 2021-04-09T18:22:45.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb9739700 1 --2- 172.21.15.32:0/1736515231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb40d1da0 0x7fefb40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb9739700 1 --2- 172.21.15.32:0/1736515231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb410d010 0x7fefb40d1780 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb9739700 1 -- 172.21.15.32:0/1736515231 >> 172.21.15.32:0/1736515231 conn(0x7fefb40ce5f0 msgr2=0x7fefb40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb9739700 1 -- 172.21.15.32:0/1736515231 shutdown_connections 2021-04-09T18:22:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.156 7fefb9739700 1 -- 172.21.15.32:0/1736515231 wait complete. 2021-04-09T18:22:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.176 7fefb9739700 1 Processor -- start 2021-04-09T18:22:45.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb9739700 1 -- start start 2021-04-09T18:22:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb9739700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb41a5210 0x7fefb408eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb9739700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb41a8ab0 0x7fefb41a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb9739700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefb41a76d0 0x7fefb41a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb9739700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb417ea50 con 0x7fefb41a76d0 2021-04-09T18:22:45.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb9739700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb408f4f0 con 0x7fefb41a8ab0 2021-04-09T18:22:45.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb9739700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb408ea60 con 0x7fefb41a5210 2021-04-09T18:22:45.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb3d4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefb41a76d0 0x7fefb41a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb3d4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefb41a76d0 0x7fefb41a7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49638/0 (socket says 172.21.15.32:49638) 2021-04-09T18:22:45.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb3d4a700 1 -- 172.21.15.32:0/2276943274 learned_addr learned my addr 172.21.15.32:0/2276943274 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:45.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb2d48700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb41a8ab0 0x7fefb41a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.177 7fefb3549700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb41a5210 0x7fefb408eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fefb3d4a700 1 -- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb41a5210 msgr2=0x7fefb408eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fefb3d4a700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb41a5210 0x7fefb408eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fefb3d4a700 1 -- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb41a8ab0 msgr2=0x7fefb41a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fefb3d4a700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb41a8ab0 0x7fefb41a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fefb3d4a700 1 -- 172.21.15.32:0/2276943274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb41a80f0 con 0x7fefb41a76d0 2021-04-09T18:22:45.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fefb3d4a700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefb41a76d0 0x7fefb41a7ad0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7fefac009980 tx=0x7fefac00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fef9bfff700 1 -- 172.21.15.32:0/2276943274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefac00be10 con 0x7fefb41a76d0 2021-04-09T18:22:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fef9bfff700 1 -- 172.21.15.32:0/2276943274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac006030 con 0x7fefb41a76d0 2021-04-09T18:22:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fefb9739700 1 -- 172.21.15.32:0/2276943274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb41a80f0 con 0x7fefb41a76d0 2021-04-09T18:22:45.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fef9bfff700 1 -- 172.21.15.32:0/2276943274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefac0064c0 con 0x7fefb41a76d0 2021-04-09T18:22:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.178 7fefb9739700 1 -- 172.21.15.32:0/2276943274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb41a80f0 con 0x7fefb41a76d0 2021-04-09T18:22:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.179 7fef9bfff700 1 -- 172.21.15.32:0/2276943274 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fefac006210 con 0x7fefb41a76d0 2021-04-09T18:22:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.179 7fef9bfff700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef94018930 0x7fef9401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.179 7fef9bfff700 1 -- 172.21.15.32:0/2276943274 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fefac032d70 con 0x7fefb41a76d0 2021-04-09T18:22:45.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.179 7fefb3549700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef94018930 0x7fef9401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.179 7fefb3549700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef94018930 0x7fef9401ae10 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:45.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.199 7fefb9739700 1 -- 172.21.15.32:0/2276943274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb41b7070 con 0x7fefb41a76d0 2021-04-09T18:22:45.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.202 7fef9bfff700 1 -- 172.21.15.32:0/2276943274 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fefac000f90 con 0x7fefb41a76d0 2021-04-09T18:22:45.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.391 7fefb9739700 1 -- 172.21.15.32:0/2276943274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefb410d480 con 0x7fefb41a76d0 2021-04-09T18:22:45.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.392 7fef9bfff700 1 -- 172.21.15.32:0/2276943274 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+14006 (secure 0 0 0) 0x7fefac006030 con 0x7fefb41a76d0 2021-04-09T18:22:45.447 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:45.448 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":107,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:22:41.390553","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:22:37.739108","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-04-09 18:22:39.152543","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:22:45.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 -- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef94018930 msgr2=0x7fef9401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef94018930 0x7fef9401ae10 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 -- 172.21.15.32:0/2276943274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefb41a76d0 msgr2=0x7fefb41a7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefb41a76d0 0x7fefb41a7ad0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7fefac009980 tx=0x7fefac00bcd0).stop 2021-04-09T18:22:45.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 -- 172.21.15.32:0/2276943274 shutdown_connections 2021-04-09T18:22:45.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef94018930 0x7fef9401ae10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefb41a76d0 0x7fefb41a7ad0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb41a8ab0 0x7fefb41a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 --2- 172.21.15.32:0/2276943274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefb41a5210 0x7fefb408eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 -- 172.21.15.32:0/2276943274 >> 172.21.15.32:0/2276943274 conn(0x7fefb41a99c0 msgr2=0x7fefb417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:45.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 -- 172.21.15.32:0/2276943274 shutdown_connections 2021-04-09T18:22:45.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.394 7fefb9739700 1 -- 172.21.15.32:0/2276943274 wait complete. 2021-04-09T18:22:45.460 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:22:37.739108', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:22:39.152543', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:22:45.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:22:45.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.558 7f7f49597700 1 Processor -- start 2021-04-09T18:22:45.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.558 7f7f49597700 1 -- start start 2021-04-09T18:22:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f49597700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7f440d7ca0 0x7f7f440d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f49597700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7f4410da00 0x7f7f4410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f49597700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4410d480 con 0x7f7f4410d010 2021-04-09T18:22:45.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f49597700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f440d9220 con 0x7f7f440d7ca0 2021-04-09T18:22:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f49597700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f440d86c0 con 0x7f7f4410da00 2021-04-09T18:22:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f42abe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7f440d7ca0 0x7f7f440d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f432bf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7f4410d010 0x7f7f440d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59232/0 (socket says 172.21.15.32:59232) 2021-04-09T18:22:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f432bf700 1 -- 172.21.15.32:0/334156710 learned_addr learned my addr 172.21.15.32:0/334156710 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f43ac0700 1 --2- 172.21.15.32:0/334156710 >> v2:172.21.15.86:3300/0 conn(0x7f7f4410da00 0x7f7f4410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f42abe700 1 -- 172.21.15.32:0/334156710 >> v2:172.21.15.86:3300/0 conn(0x7f7f4410da00 msgr2=0x7f7f4410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f42abe700 1 --2- 172.21.15.32:0/334156710 >> v2:172.21.15.86:3300/0 conn(0x7f7f4410da00 0x7f7f4410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f42abe700 1 -- 172.21.15.32:0/334156710 >> v1:172.21.15.32:6789/0 conn(0x7f7f4410d010 legacy=0x7f7f440d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f42abe700 1 -- 172.21.15.32:0/334156710 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f440d1f70 con 0x7f7f440d7ca0 2021-04-09T18:22:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f422bd700 1 -- 172.21.15.32:0/334156710 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332660068 0 0) 0x7f7f34000ce0 con 0x7f7f4410d010 2021-04-09T18:22:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f43ac0700 1 --2- 172.21.15.32:0/334156710 >> v2:172.21.15.86:3300/0 conn(0x7f7f4410da00 0x7f7f4410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f42abe700 1 --2- 172.21.15.32:0/334156710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440d7ca0 0x7f7f440d80a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f7f38006650 tx=0x7f7f38009a80).ready entity=mon.0 client_cookie=48805b2b23c2ac65 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:45.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f422bd700 1 -- 172.21.15.32:0/334156710 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f380009b0 con 0x7f7f440d7ca0 2021-04-09T18:22:45.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f422bd700 1 -- 172.21.15.32:0/334156710 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3800c0c0 con 0x7f7f440d7ca0 2021-04-09T18:22:45.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.559 7f7f422bd700 1 -- 172.21.15.32:0/334156710 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f3800c510 con 0x7f7f440d7ca0 2021-04-09T18:22:45.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.560 7f7f49597700 1 -- 172.21.15.32:0/334156710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440d7ca0 msgr2=0x7f7f440d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.560 7f7f49597700 1 --2- 172.21.15.32:0/334156710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440d7ca0 0x7f7f440d80a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f7f38006650 tx=0x7f7f38009a80).stop 2021-04-09T18:22:45.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.560 7f7f49597700 1 -- 172.21.15.32:0/334156710 shutdown_connections 2021-04-09T18:22:45.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.560 7f7f49597700 1 --2- 172.21.15.32:0/334156710 >> v2:172.21.15.86:3300/0 conn(0x7f7f4410da00 0x7f7f4410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.560 7f7f49597700 1 --2- 172.21.15.32:0/334156710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f440d7ca0 0x7f7f440d80a0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.560 7f7f49597700 1 -- 172.21.15.32:0/334156710 >> 172.21.15.32:0/334156710 conn(0x7f7f440ce5f0 msgr2=0x7f7f440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:45.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.560 7f7f49597700 1 -- 172.21.15.32:0/334156710 shutdown_connections 2021-04-09T18:22:45.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.560 7f7f49597700 1 -- 172.21.15.32:0/334156710 wait complete. 2021-04-09T18:22:45.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.580 7f7f49597700 1 Processor -- start 2021-04-09T18:22:45.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.580 7f7f49597700 1 -- start start 2021-04-09T18:22:45.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f49597700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f440a0a10 0x7f7f440bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f49597700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f4418ad00 0x7f7f4419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f49597700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f441a03f0 0x7f7f44095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f49597700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4418b100 con 0x7f7f441a03f0 2021-04-09T18:22:45.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f49597700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f44095860 con 0x7f7f4418ad00 2021-04-09T18:22:45.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f49597700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f44095a80 con 0x7f7f440a0a10 2021-04-09T18:22:45.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f43ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f441a03f0 0x7f7f44095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f43ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f441a03f0 0x7f7f44095240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49652/0 (socket says 172.21.15.32:49652) 2021-04-09T18:22:45.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f43ac0700 1 -- 172.21.15.32:0/3699543924 learned_addr learned my addr 172.21.15.32:0/3699543924 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:45.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f432bf700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f440a0a10 0x7f7f440bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f42abe700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f4418ad00 0x7f7f4419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f43ac0700 1 -- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f440a0a10 msgr2=0x7f7f440bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f43ac0700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f440a0a10 0x7f7f440bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f43ac0700 1 -- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f4418ad00 msgr2=0x7f7f4419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f43ac0700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f4418ad00 0x7f7f4419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.581 7f7f43ac0700 1 -- 172.21.15.32:0/3699543924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f44189250 con 0x7f7f441a03f0 2021-04-09T18:22:45.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.582 7f7f43ac0700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f441a03f0 0x7f7f44095240 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f7f3c00b970 tx=0x7f7f3c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:45.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.582 7f7f2bfff700 1 -- 172.21.15.32:0/3699543924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f3c000f90 con 0x7f7f441a03f0 2021-04-09T18:22:45.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.582 7f7f2bfff700 1 -- 172.21.15.32:0/3699543924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3c006aa0 con 0x7f7f441a03f0 2021-04-09T18:22:45.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.582 7f7f49597700 1 -- 172.21.15.32:0/3699543924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f44189250 con 0x7f7f441a03f0 2021-04-09T18:22:45.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.582 7f7f2bfff700 1 -- 172.21.15.32:0/3699543924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f3c0015f0 con 0x7f7f441a03f0 2021-04-09T18:22:45.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.582 7f7f49597700 1 -- 172.21.15.32:0/3699543924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f44189250 con 0x7f7f441a03f0 2021-04-09T18:22:45.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.582 7f7f2bfff700 1 -- 172.21.15.32:0/3699543924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7f3c000ed0 con 0x7f7f441a03f0 2021-04-09T18:22:45.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.583 7f7f2bfff700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7f24018930 0x7f7f2401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:45.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.583 7f7f2bfff700 1 -- 172.21.15.32:0/3699543924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7f3c032da0 con 0x7f7f441a03f0 2021-04-09T18:22:45.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.583 7f7f432bf700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7f24018930 0x7f7f2401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:45.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.583 7f7f432bf700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7f24018930 0x7f7f2401adc0 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:45.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.603 7f7f49597700 1 -- 172.21.15.32:0/3699543924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f44093d50 con 0x7f7f441a03f0 2021-04-09T18:22:45.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.606 7f7f2bfff700 1 -- 172.21.15.32:0/3699543924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7f3c0066f0 con 0x7f7f441a03f0 2021-04-09T18:22:45.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.786 7f7f49597700 1 -- 172.21.15.32:0/3699543924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f7f4410d480 con 0x7f7f441a03f0 2021-04-09T18:22:45.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.787 7f7f2bfff700 1 -- 172.21.15.32:0/3699543924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v107) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7f3c006aa0 con 0x7f7f441a03f0 2021-04-09T18:22:45.842 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:22:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 -- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7f24018930 msgr2=0x7f7f2401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7f24018930 0x7f7f2401adc0 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 -- 172.21.15.32:0/3699543924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f441a03f0 msgr2=0x7f7f44095240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f441a03f0 0x7f7f44095240 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f7f3c00b970 tx=0x7f7f3c00bd10).stop 2021-04-09T18:22:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 -- 172.21.15.32:0/3699543924 shutdown_connections 2021-04-09T18:22:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7f24018930 0x7f7f2401adc0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f441a03f0 0x7f7f44095240 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f4418ad00 0x7f7f4419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 --2- 172.21.15.32:0/3699543924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f440a0a10 0x7f7f440bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 -- 172.21.15.32:0/3699543924 >> 172.21.15.32:0/3699543924 conn(0x7f7f440975f0 msgr2=0x7f7f44096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:45.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 -- 172.21.15.32:0/3699543924 shutdown_connections 2021-04-09T18:22:45.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.790 7f7f49597700 1 -- 172.21.15.32:0/3699543924 wait complete. 2021-04-09T18:22:45.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:22:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.952 7fe4cf358700 1 Processor -- start 2021-04-09T18:22:46.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.952 7fe4cf358700 1 -- start start 2021-04-09T18:22:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.952 7fe4cf358700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c810d010 0x7fe4c80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cf358700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4c80d8490 0x7fe4c810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cf358700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c810d480 con 0x7fe4c80d1da0 2021-04-09T18:22:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cf358700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c810e780 con 0x7fe4c80d8490 2021-04-09T18:22:46.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cf358700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80d8d30 con 0x7fe4c810d010 2021-04-09T18:22:46.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cd975700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4c80d8490 0x7fe4c810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cd975700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4c80d8490 0x7fe4c810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49658/0 (socket says 172.21.15.32:49658) 2021-04-09T18:22:46.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cd975700 1 -- 172.21.15.32:0/2431977333 learned_addr learned my addr 172.21.15.32:0/2431977333 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:46.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cd174700 1 --2- 172.21.15.32:0/2431977333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c810d010 0x7fe4c80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cd975700 1 -- 172.21.15.32:0/2431977333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c810d010 msgr2=0x7fe4c80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:46.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cd975700 1 --2- 172.21.15.32:0/2431977333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c810d010 0x7fe4c80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cd975700 1 -- 172.21.15.32:0/2431977333 >> v1:172.21.15.86:6789/0 conn(0x7fe4c80d1da0 legacy=0x7fe4c80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cd975700 1 -- 172.21.15.32:0/2431977333 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c80d9220 con 0x7fe4c80d8490 2021-04-09T18:22:46.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cd975700 1 --2- 172.21.15.32:0/2431977333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c80d8490 0x7fe4c810da00 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7fe4b8006b80 tx=0x7fe4b8009b90).ready entity=mon.0 client_cookie=38e56e34d9b7b7fb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:46.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4c7fff700 1 -- 172.21.15.32:0/2431977333 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4b80022e0 con 0x7fe4c80d8490 2021-04-09T18:22:46.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4c7fff700 1 -- 172.21.15.32:0/2431977333 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b8002620 con 0x7fe4c80d8490 2021-04-09T18:22:46.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4c7fff700 1 -- 172.21.15.32:0/2431977333 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4b8002b00 con 0x7fe4c80d8490 2021-04-09T18:22:46.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cf358700 1 -- 172.21.15.32:0/2431977333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c80d8490 msgr2=0x7fe4c810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.953 7fe4cf358700 1 --2- 172.21.15.32:0/2431977333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c80d8490 0x7fe4c810da00 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7fe4b8006b80 tx=0x7fe4b8009b90).stop 2021-04-09T18:22:46.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.954 7fe4cf358700 1 -- 172.21.15.32:0/2431977333 shutdown_connections 2021-04-09T18:22:46.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.954 7fe4cf358700 1 --2- 172.21.15.32:0/2431977333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c80d8490 0x7fe4c810da00 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.954 7fe4cf358700 1 --2- 172.21.15.32:0/2431977333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c810d010 0x7fe4c80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.954 7fe4cf358700 1 -- 172.21.15.32:0/2431977333 >> 172.21.15.32:0/2431977333 conn(0x7fe4c80ce5f0 msgr2=0x7fe4c80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:46.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.954 7fe4cf358700 1 -- 172.21.15.32:0/2431977333 shutdown_connections 2021-04-09T18:22:46.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.954 7fe4cf358700 1 -- 172.21.15.32:0/2431977333 wait complete. 2021-04-09T18:22:46.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.974 7fe4cf358700 1 Processor -- start 2021-04-09T18:22:46.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.974 7fe4cf358700 1 -- start start 2021-04-09T18:22:46.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cf358700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4c808eeb0 0x7fe4c80c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cf358700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c808f2b0 0x7fe4c80c3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cf358700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c80c1b80 0x7fe4c80c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cf358700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c808d740 con 0x7fe4c808f2b0 2021-04-09T18:22:46.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cf358700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80c2920 con 0x7fe4c808eeb0 2021-04-09T18:22:46.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cf358700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80a2450 con 0x7fe4c80c1b80 2021-04-09T18:22:46.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cc973700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c808f2b0 0x7fe4c80c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cc973700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c808f2b0 0x7fe4c80c3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49662/0 (socket says 172.21.15.32:49662) 2021-04-09T18:22:46.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cc973700 1 -- 172.21.15.32:0/2063353980 learned_addr learned my addr 172.21.15.32:0/2063353980 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:46.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cd174700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4c808eeb0 0x7fe4c80c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cd975700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c80c1b80 0x7fe4c80c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cc973700 1 -- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c80c1b80 msgr2=0x7fe4c80c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cc973700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c80c1b80 0x7fe4c80c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cc973700 1 -- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4c808eeb0 msgr2=0x7fe4c80c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.975 7fe4cc973700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4c808eeb0 0x7fe4c80c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.976 7fe4cc973700 1 -- 172.21.15.32:0/2063353980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c80ac2f0 con 0x7fe4c808f2b0 2021-04-09T18:22:46.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.976 7fe4cc973700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c808f2b0 0x7fe4c80c3490 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fe4bc00b5b0 tx=0x7fe4bc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:46.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.976 7fe4c5ffb700 1 -- 172.21.15.32:0/2063353980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4bc00bd40 con 0x7fe4c808f2b0 2021-04-09T18:22:46.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.976 7fe4c5ffb700 1 -- 172.21.15.32:0/2063353980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc006240 con 0x7fe4c808f2b0 2021-04-09T18:22:46.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.976 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c80ac2f0 con 0x7fe4c808f2b0 2021-04-09T18:22:46.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.976 7fe4c5ffb700 1 -- 172.21.15.32:0/2063353980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4bc005690 con 0x7fe4c808f2b0 2021-04-09T18:22:46.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.976 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c80ac2f0 con 0x7fe4c808f2b0 2021-04-09T18:22:46.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.976 7fe4c5ffb700 1 -- 172.21.15.32:0/2063353980 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe4bc0054a0 con 0x7fe4c808f2b0 2021-04-09T18:22:46.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.977 7fe4c5ffb700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4a8018930 0x7fe4a801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.977 7fe4c5ffb700 1 -- 172.21.15.32:0/2063353980 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe4bc032a30 con 0x7fe4c808f2b0 2021-04-09T18:22:46.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.977 7fe4cd174700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4a8018930 0x7fe4a801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.977 7fe4cd174700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4a8018930 0x7fe4a801adc0 crc :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:46.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:45.997 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c808d740 con 0x7fe4c808f2b0 2021-04-09T18:22:46.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.000 7fe4c5ffb700 1 -- 172.21.15.32:0/2063353980 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe4bc000c60 con 0x7fe4c808f2b0 2021-04-09T18:22:46.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.181 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe4c808d740 con 0x7fe4c808f2b0 2021-04-09T18:22:46.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.181 7fe4c5ffb700 1 -- 172.21.15.32:0/2063353980 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v107) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe4bc0054a0 con 0x7fe4c808f2b0 2021-04-09T18:22:46.236 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:22:46.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4a8018930 msgr2=0x7fe4a801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4a8018930 0x7fe4a801adc0 crc :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c808f2b0 msgr2=0x7fe4c80c3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c808f2b0 0x7fe4c80c3490 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fe4bc00b5b0 tx=0x7fe4bc00b950).stop 2021-04-09T18:22:46.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 shutdown_connections 2021-04-09T18:22:46.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4a8018930 0x7fe4a801adc0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4c808f2b0 0x7fe4c80c3490 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4c80c1b80 0x7fe4c80c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 --2- 172.21.15.32:0/2063353980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4c808eeb0 0x7fe4c80c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 >> 172.21.15.32:0/2063353980 conn(0x7fe4c809d710 msgr2=0x7fe4c8186570 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:46.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 shutdown_connections 2021-04-09T18:22:46.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.184 7fe4cf358700 1 -- 172.21.15.32:0/2063353980 wait complete. 2021-04-09T18:22:46.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:22:46.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.347 7f58ab1c5700 1 Processor -- start 2021-04-09T18:22:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.347 7f58ab1c5700 1 -- start start 2021-04-09T18:22:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.347 7f58ab1c5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f58a410d010 0x7f58a40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.347 7f58ab1c5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a410d480 con 0x7f58a40d8490 2021-04-09T18:22:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.347 7f58ab1c5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a410e1f0 con 0x7f58a40d1da0 2021-04-09T18:22:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.347 7f58ab1c5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a40d8d30 con 0x7f58a410d010 2021-04-09T18:22:46.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a97e2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f58a40d8490 0x7f58a410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59260/0 (socket says 172.21.15.32:59260) 2021-04-09T18:22:46.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a97e2700 1 -- 172.21.15.32:0/1407751936 learned_addr learned my addr 172.21.15.32:0/1407751936 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:46.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a8fe1700 1 --2- 172.21.15.32:0/1407751936 >> v2:172.21.15.86:3300/0 conn(0x7f58a410d010 0x7f58a40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075651972 0 0) 0x7f589c0009b0 con 0x7f58a40d8490 2021-04-09T18:22:46.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5890002fe0 con 0x7f58a40d8490 2021-04-09T18:22:46.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1965452304 0 0) 0x7f589c0010e0 con 0x7f58a40d8490 2021-04-09T18:22:46.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 >> v2:172.21.15.86:3300/0 conn(0x7f58a410d010 msgr2=0x7f58a40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:46.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 --2- 172.21.15.32:0/1407751936 >> v2:172.21.15.86:3300/0 conn(0x7f58a410d010 0x7f58a40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 >> v1:172.21.15.86:6789/0 conn(0x7f58a40d1da0 legacy=0x7f58a40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a40d9220 con 0x7f58a40d8490 2021-04-09T18:22:46.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f589c001600 con 0x7f58a40d8490 2021-04-09T18:22:46.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 >> v1:172.21.15.32:6789/0 conn(0x7f58a40d8490 legacy=0x7f58a410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5890003a70 0x7f5890005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58900067a0 0x7f5890008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58900095a0 0x7f589000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589000c310 con 0x7f58900095a0 2021-04-09T18:22:46.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589000c530 con 0x7f58900067a0 2021-04-09T18:22:46.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a97e2700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58900095a0 0x7f589000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5890050760 con 0x7f5890003a70 2021-04-09T18:22:46.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a3fff700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58900067a0 0x7f5890008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.348 7f58a8fe1700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5890003a70 0x7f5890005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a97e2700 1 -- 172.21.15.32:0/1407751936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5890003a70 msgr2=0x7f5890005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:46.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a97e2700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5890003a70 0x7f5890005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a97e2700 1 -- 172.21.15.32:0/1407751936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58900067a0 msgr2=0x7f5890008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:46.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a97e2700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58900067a0 0x7f5890008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a97e2700 1 -- 172.21.15.32:0/1407751936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58900509c0 con 0x7f58900095a0 2021-04-09T18:22:46.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a97e2700 1 -- 172.21.15.32:0/1407751936 reap_dead start 2021-04-09T18:22:46.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a97e2700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58900095a0 0x7f589000bac0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7f589c007970 tx=0x7f589c007d10).ready entity=mon.0 client_cookie=3197a4336edc878e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:46.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c0070c0 con 0x7f58900095a0 2021-04-09T18:22:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c0043a0 con 0x7f58900095a0 2021-04-09T18:22:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58a37fe700 1 -- 172.21.15.32:0/1407751936 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c004880 con 0x7f58900095a0 2021-04-09T18:22:46.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58ab1c5700 1 -- 172.21.15.32:0/1407751936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58900095a0 msgr2=0x7f589000bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.349 7f58ab1c5700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58900095a0 0x7f589000bac0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7f589c007970 tx=0x7f589c007d10).stop 2021-04-09T18:22:46.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.350 7f58ab1c5700 1 -- 172.21.15.32:0/1407751936 shutdown_connections 2021-04-09T18:22:46.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.350 7f58ab1c5700 1 --2- 172.21.15.32:0/1407751936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58900095a0 0x7f589000bac0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.350 7f58ab1c5700 1 -- 172.21.15.32:0/1407751936 >> 172.21.15.32:0/1407751936 conn(0x7f58a40ce5f0 msgr2=0x7f58a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.350 7f58ab1c5700 1 -- 172.21.15.32:0/1407751936 shutdown_connections 2021-04-09T18:22:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.350 7f58ab1c5700 1 -- 172.21.15.32:0/1407751936 wait complete. 2021-04-09T18:22:46.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.370 7f58ab1c5700 1 Processor -- start 2021-04-09T18:22:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.370 7f58ab1c5700 1 -- start start 2021-04-09T18:22:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58ab1c5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a40c3c00 0x7f58a409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58ab1c5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4095240 0x7f58a41a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58ab1c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41a1270 0x7f58a40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58ab1c5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a409cda0 con 0x7f58a41a1270 2021-04-09T18:22:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58ab1c5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a40c3680 con 0x7f58a4095240 2021-04-09T18:22:46.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58ab1c5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4095640 con 0x7f58a40c3c00 2021-04-09T18:22:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a97e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41a1270 0x7f58a40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a97e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41a1270 0x7f58a40b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49684/0 (socket says 172.21.15.32:49684) 2021-04-09T18:22:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a97e2700 1 -- 172.21.15.32:0/3948832399 learned_addr learned my addr 172.21.15.32:0/3948832399 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:46.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a8fe1700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a40c3c00 0x7f58a409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a3fff700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4095240 0x7f58a41a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a97e2700 1 -- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a40c3c00 msgr2=0x7f58a409c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a97e2700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a40c3c00 0x7f58a409c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a97e2700 1 -- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4095240 msgr2=0x7f58a41a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a97e2700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4095240 0x7f58a41a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a97e2700 1 -- 172.21.15.32:0/3948832399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a41a1670 con 0x7f58a41a1270 2021-04-09T18:22:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a97e2700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41a1270 0x7f58a40b0cf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f589c007fc0 tx=0x7f589c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a17fa700 1 -- 172.21.15.32:0/3948832399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c0010e0 con 0x7f58a41a1270 2021-04-09T18:22:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a17fa700 1 -- 172.21.15.32:0/3948832399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c00ad20 con 0x7f58a41a1270 2021-04-09T18:22:46.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a41a1670 con 0x7f58a41a1270 2021-04-09T18:22:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58a17fa700 1 -- 172.21.15.32:0/3948832399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c004620 con 0x7f58a41a1270 2021-04-09T18:22:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.371 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a41a1670 con 0x7f58a41a1270 2021-04-09T18:22:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.372 7f58a17fa700 1 -- 172.21.15.32:0/3948832399 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f589c0043a0 con 0x7f58a41a1270 2021-04-09T18:22:46.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.372 7f58a17fa700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5888018930 0x7f588801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.372 7f58a17fa700 1 -- 172.21.15.32:0/3948832399 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f589c0329d0 con 0x7f58a41a1270 2021-04-09T18:22:46.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.372 7f58a8fe1700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5888018930 0x7f588801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.372 7f58a8fe1700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5888018930 0x7f588801ae10 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.391 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a4093d50 con 0x7f58a41a1270 2021-04-09T18:22:46.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.395 7f58a17fa700 1 -- 172.21.15.32:0/3948832399 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f589c0071c0 con 0x7f58a41a1270 2021-04-09T18:22:46.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.575 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f58a410d480 con 0x7f58a41a1270 2021-04-09T18:22:46.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.575 7f58a17fa700 1 -- 172.21.15.32:0/3948832399 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v107) v1 ==== 100+0+10 (secure 0 0 0) 0x7f589c0071c0 con 0x7f58a41a1270 2021-04-09T18:22:46.630 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:22:46.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5888018930 msgr2=0x7f588801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5888018930 0x7f588801ae10 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41a1270 msgr2=0x7f58a40b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41a1270 0x7f58a40b0cf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f589c007fc0 tx=0x7f589c00a930).stop 2021-04-09T18:22:46.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 shutdown_connections 2021-04-09T18:22:46.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5888018930 0x7f588801ae10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58a41a1270 0x7f58a40b0cf0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4095240 0x7f58a41a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 --2- 172.21.15.32:0/3948832399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58a40c3c00 0x7f58a409c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 >> 172.21.15.32:0/3948832399 conn(0x7f58a40975f0 msgr2=0x7f58a4096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:46.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 shutdown_connections 2021-04-09T18:22:46.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.578 7f58ab1c5700 1 -- 172.21.15.32:0/3948832399 wait complete. 2021-04-09T18:22:46.642 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:22:46.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:22:46.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.737 7f0ff5ec7700 1 Processor -- start 2021-04-09T18:22:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.737 7f0ff5ec7700 1 -- start start 2021-04-09T18:22:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0ff5ec7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0ff010d010 0x7f0ff00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0ff5ec7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ff00d1da0 0x7f0ff00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0ff5ec7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff010d480 con 0x7f0ff00d82c0 2021-04-09T18:22:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0ff5ec7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff00d8d30 con 0x7f0ff00d1da0 2021-04-09T18:22:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0ff5ec7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff00d9220 con 0x7f0ff010d010 2021-04-09T18:22:46.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0feffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0ff00d82c0 0x7f0ff010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59280/0 (socket says 172.21.15.32:59280) 2021-04-09T18:22:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0feeffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0ff00d1da0 0x7f0ff00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0feffff700 1 -- 172.21.15.32:0/1326811908 learned_addr learned my addr 172.21.15.32:0/1326811908 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0fef7fe700 1 --2- 172.21.15.32:0/1326811908 >> v2:172.21.15.86:3300/0 conn(0x7f0ff010d010 0x7f0ff00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0feeffd700 1 -- 172.21.15.32:0/1326811908 >> v2:172.21.15.86:3300/0 conn(0x7f0ff010d010 msgr2=0x7f0ff00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:46.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0feeffd700 1 --2- 172.21.15.32:0/1326811908 >> v2:172.21.15.86:3300/0 conn(0x7f0ff010d010 0x7f0ff00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0feeffd700 1 -- 172.21.15.32:0/1326811908 >> v1:172.21.15.32:6789/0 conn(0x7f0ff00d82c0 legacy=0x7f0ff010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0feeffd700 1 -- 172.21.15.32:0/1326811908 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff00d86f0 con 0x7f0ff00d1da0 2021-04-09T18:22:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.738 7f0fee7fc700 1 -- 172.21.15.32:0/1326811908 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135319166 0 0) 0x7f0fe8000d30 con 0x7f0ff00d82c0 2021-04-09T18:22:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.739 7f0feeffd700 1 --2- 172.21.15.32:0/1326811908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00d1da0 0x7f0ff00d7ca0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7f0fe4006650 tx=0x7f0fe4009a80).ready entity=mon.0 client_cookie=e462cb6c12aacd9f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:46.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.739 7f0fee7fc700 1 -- 172.21.15.32:0/1326811908 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe40009b0 con 0x7f0ff00d1da0 2021-04-09T18:22:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.739 7f0fee7fc700 1 -- 172.21.15.32:0/1326811908 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe400c0c0 con 0x7f0ff00d1da0 2021-04-09T18:22:46.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.739 7f0fee7fc700 1 -- 172.21.15.32:0/1326811908 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe400c510 con 0x7f0ff00d1da0 2021-04-09T18:22:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.739 7f0ff5ec7700 1 -- 172.21.15.32:0/1326811908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00d1da0 msgr2=0x7f0ff00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.739 7f0ff5ec7700 1 --2- 172.21.15.32:0/1326811908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00d1da0 0x7f0ff00d7ca0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7f0fe4006650 tx=0x7f0fe4009a80).stop 2021-04-09T18:22:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.739 7f0ff5ec7700 1 -- 172.21.15.32:0/1326811908 shutdown_connections 2021-04-09T18:22:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.739 7f0ff5ec7700 1 --2- 172.21.15.32:0/1326811908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00d1da0 0x7f0ff00d7ca0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.739 7f0ff5ec7700 1 --2- 172.21.15.32:0/1326811908 >> v2:172.21.15.86:3300/0 conn(0x7f0ff010d010 0x7f0ff00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.740 7f0ff5ec7700 1 -- 172.21.15.32:0/1326811908 >> 172.21.15.32:0/1326811908 conn(0x7f0ff00ce5f0 msgr2=0x7f0ff00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:46.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.740 7f0ff5ec7700 1 -- 172.21.15.32:0/1326811908 shutdown_connections 2021-04-09T18:22:46.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.740 7f0ff5ec7700 1 -- 172.21.15.32:0/1326811908 wait complete. 2021-04-09T18:22:46.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.760 7f0ff5ec7700 1 Processor -- start 2021-04-09T18:22:46.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.760 7f0ff5ec7700 1 -- start start 2021-04-09T18:22:46.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.760 7f0ff5ec7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00b5360 0x7f0ff01a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.760 7f0ff5ec7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ff018e140 0x7f0ff01aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.760 7f0ff5ec7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ff01ad040 0x7f0ff008fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.760 7f0ff5ec7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff01a5360 con 0x7f0ff00b5360 2021-04-09T18:22:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.760 7f0ff5ec7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff00902c0 con 0x7f0ff01ad040 2021-04-09T18:22:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.760 7f0ff5ec7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff00904e0 con 0x7f0ff018e140 2021-04-09T18:22:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00b5360 0x7f0ff01a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00b5360 0x7f0ff01a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49696/0 (socket says 172.21.15.32:49696) 2021-04-09T18:22:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fef7fe700 1 -- 172.21.15.32:0/2560718593 learned_addr learned my addr 172.21.15.32:0/2560718593 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0feffff700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ff01ad040 0x7f0ff008fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0feeffd700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ff018e140 0x7f0ff01aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fef7fe700 1 -- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ff018e140 msgr2=0x7f0ff01aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fef7fe700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ff018e140 0x7f0ff01aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fef7fe700 1 -- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ff01ad040 msgr2=0x7f0ff008fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:46.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fef7fe700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ff01ad040 0x7f0ff008fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fef7fe700 1 -- 172.21.15.32:0/2560718593 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff0183c80 con 0x7f0ff00b5360 2021-04-09T18:22:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0feeffd700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ff018e140 0x7f0ff01aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fef7fe700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00b5360 0x7f0ff01a4d40 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0x7f0fe0006f00 tx=0x7f0fe000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:46.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fd7fff700 1 -- 172.21.15.32:0/2560718593 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe000bd30 con 0x7f0ff00b5360 2021-04-09T18:22:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fd7fff700 1 -- 172.21.15.32:0/2560718593 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe0002030 con 0x7f0ff00b5360 2021-04-09T18:22:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff0183c80 con 0x7f0ff00b5360 2021-04-09T18:22:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0fd7fff700 1 -- 172.21.15.32:0/2560718593 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe0002590 con 0x7f0ff00b5360 2021-04-09T18:22:46.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.761 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff0183c80 con 0x7f0ff00b5360 2021-04-09T18:22:46.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.762 7f0fd7fff700 1 -- 172.21.15.32:0/2560718593 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0fe0000c90 con 0x7f0ff00b5360 2021-04-09T18:22:46.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.762 7f0fd7fff700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fd0018930 0x7f0fd001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:46.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.762 7f0fd7fff700 1 -- 172.21.15.32:0/2560718593 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0fe0032a40 con 0x7f0ff00b5360 2021-04-09T18:22:46.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.762 7f0feeffd700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fd0018930 0x7f0fd001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:46.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.762 7f0feeffd700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fd0018930 0x7f0fd001ae10 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:46.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.782 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff0182d70 con 0x7f0ff00b5360 2021-04-09T18:22:46.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.785 7f0fd7fff700 1 -- 172.21.15.32:0/2560718593 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0fe001f780 con 0x7f0ff00b5360 2021-04-09T18:22:47.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.978 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ff0182d70 con 0x7f0ff00b5360 2021-04-09T18:22:47.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.978 7f0fd7fff700 1 -- 172.21.15.32:0/2560718593 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2777 (secure 0 0 0) 0x7f0fe000bd30 con 0x7f0ff00b5360 2021-04-09T18:22:47.033 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 127 2021-04-09T18:22:47.034 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:22:47.034 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":127,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":56684,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1833188051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1833188051},{"type":"v1","addr":"172.21.15.86:6836","nonce":1833188051}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":123},{"gid":56690,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/46738575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46738575},{"type":"v1","addr":"172.21.15.32:6834","nonce":46738575}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":123},{"gid":56693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/647788630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":647788630},{"type":"v1","addr":"172.21.15.32:6835","nonce":647788630}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:22:42.405857","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:22:47.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fd0018930 msgr2=0x7f0fd001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fd0018930 0x7f0fd001ae10 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00b5360 msgr2=0x7f0ff01a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00b5360 0x7f0ff01a4d40 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0x7f0fe0006f00 tx=0x7f0fe000b940).stop 2021-04-09T18:22:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 shutdown_connections 2021-04-09T18:22:47.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0fd0018930 0x7f0fd001ae10 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:47.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ff01ad040 0x7f0ff008fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:47.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ff018e140 0x7f0ff01aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:47.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 --2- 172.21.15.32:0/2560718593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff00b5360 0x7f0ff01a4d40 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:47.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 >> 172.21.15.32:0/2560718593 conn(0x7f0ff019b460 msgr2=0x7f0ff019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:47.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 shutdown_connections 2021-04-09T18:22:47.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:46.981 7f0ff5ec7700 1 -- 172.21.15.32:0/2560718593 wait complete. 2021-04-09T18:22:47.447 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 127, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:22:40.389392', 'modified': '2021-04-09 18:22:42.405857', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7777}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7777': {'gid': 7777, 'name': 'b', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/1938812020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1938812020}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1938812020}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:22:47.448 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:22:47.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:22:47.577 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:22:47.577 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:22:47.577 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:22:47.577 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 7777, 2021-04-09T18:22:47.578 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:22:47.578 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:22:47.578 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 7.13210064, 2021-04-09T18:22:47.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 127, 2021-04-09T18:22:47.579 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 107, 2021-04-09T18:22:47.579 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 107, 2021-04-09T18:22:47.579 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12.600874504 2021-04-09T18:22:47.579 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:22:47.583 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 7777, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.13210064, "mdsmap_epoch": 127, "osdmap_epoch": 107, "osdmap_epoch_barrier": 107, "uptime": 12.600874504 } 2021-04-09T18:22:47.583 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:22:47.584 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:22:47.584 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:22:47.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:47.585 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:47.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:22:47.693 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:22:47.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:22:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:22:47.753 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:22:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:22:47.754 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:47.755 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:22:47.798 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:22:47.799 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:47.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:22:47.900 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:22:47.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:22:47.959 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:22:47.959 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:22:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:22:47.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:47.961 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:22:47.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:22:47.934 7f66c0beaf80 -1 init, newargv = 0x5606178563d0 newargc=9 2021-04-09T18:22:47.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[63257]: starting ceph client 2021-04-09T18:22:47.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[63257]: starting fuse 2021-04-09T18:22:48.006 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:22:48.006 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:22:48.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:48.007 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:22:48.007 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:22:48.008 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:22:48.008 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:22:48.009 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:22:48.009 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:22:48.010 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:22:48.010 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:22:48.010 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:22:48.011 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:22:48.011 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:22:48.011 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:22:48.012 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:22:48.012 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:22:48.013 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:22:48.013 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:22:48.013 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:22:48.014 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:22:48.014 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:22:48.015 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:22:48.015 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:22:48.016 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:22:48.016 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:22:48.127 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.63257.asok 2021-04-09T18:22:48.130 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.63257.asok 2021-04-09T18:22:48.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:48.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.63257.asok status 2021-04-09T18:22:48.255 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:22:48.256 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:22:48.256 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:22:48.256 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:22:48.256 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:22:48.257 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:22:48.257 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:22:48.257 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "63257", 2021-04-09T18:22:48.258 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:22:48.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:48.258 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:22:48.258 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:22:48.259 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 8136, 2021-04-09T18:22:48.259 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:22:48.259 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:22:48.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:22:48.260 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 8136 2021-04-09T18:22:48.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:48.260 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:22:48.261 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:48.261 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:22:48.261 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1114454559 2021-04-09T18:22:48.262 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:22:48.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:48.262 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:22:48.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:22:48.263 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:22:48.263 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1114454559 2021-04-09T18:22:48.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:22:48.264 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.8136 172.21.15.32:0/1114454559", 2021-04-09T18:22:48.264 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/1114454559", 2021-04-09T18:22:48.265 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:22:48.265 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 127, 2021-04-09T18:22:48.265 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 107, 2021-04-09T18:22:48.266 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:22:48.266 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:22:48.266 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:22:48.266 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:22:48.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:48.269 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:22:48.269 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:48.314 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:22:48.314 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:22:48.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:22:48.316 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:22:48.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:22:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.472 7f8c13a35700 1 Processor -- start 2021-04-09T18:22:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.473 7f8c13a35700 1 -- start start 2021-04-09T18:22:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.473 7f8c13a35700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8c0c10d010 0x7f8c0c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.473 7f8c13a35700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0d8490 0x7f8c0c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:48.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.473 7f8c13a35700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c10d480 con 0x7f8c0c0d1da0 2021-04-09T18:22:48.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.473 7f8c13a35700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0c0d8d30 con 0x7f8c0c10d010 2021-04-09T18:22:48.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.473 7f8c13a35700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0d9220 con 0x7f8c0c0d8490 2021-04-09T18:22:48.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c12052700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0d8490 0x7f8c0c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:48.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c11851700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8c0c10d010 0x7f8c0c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:48.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c12052700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0d8490 0x7f8c0c10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:56944/0 (socket says 172.21.15.32:56944) 2021-04-09T18:22:48.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c12052700 1 -- 172.21.15.32:0/2574323256 learned_addr learned my addr 172.21.15.32:0/2574323256 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:48.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c11050700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8c0c0d1da0 0x7f8c0c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:57236/0 (socket says 172.21.15.32:57236) 2021-04-09T18:22:48.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c1084f700 1 -- 172.21.15.32:0/2574323256 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299279832 0 0) 0x7f8c00000c60 con 0x7f8c0c0d1da0 2021-04-09T18:22:48.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c11851700 1 -- 172.21.15.32:0/2574323256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0d8490 msgr2=0x7f8c0c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:48.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c11851700 1 --2- 172.21.15.32:0/2574323256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0d8490 0x7f8c0c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:48.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c11851700 1 -- 172.21.15.32:0/2574323256 >> v1:172.21.15.86:6789/0 conn(0x7f8c0c0d1da0 legacy=0x7f8c0c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:48.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c11851700 1 -- 172.21.15.32:0/2574323256 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c10e020 con 0x7f8c0c10d010 2021-04-09T18:22:48.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c12052700 1 --2- 172.21.15.32:0/2574323256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0d8490 0x7f8c0c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:48.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c11851700 1 --2- 172.21.15.32:0/2574323256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c0c10d010 0x7f8c0c0d1780 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f8c08006650 tx=0x7f8c08009a80).ready entity=mon.1 client_cookie=94a63d88898d06fa server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:48.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c1084f700 1 -- 172.21.15.32:0/2574323256 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c080009b0 con 0x7f8c0c10d010 2021-04-09T18:22:48.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c1084f700 1 -- 172.21.15.32:0/2574323256 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0800c0c0 con 0x7f8c0c10d010 2021-04-09T18:22:48.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c1084f700 1 -- 172.21.15.32:0/2574323256 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c0800c510 con 0x7f8c0c10d010 2021-04-09T18:22:48.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c13a35700 1 -- 172.21.15.32:0/2574323256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c0c10d010 msgr2=0x7f8c0c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:48.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.474 7f8c13a35700 1 --2- 172.21.15.32:0/2574323256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c0c10d010 0x7f8c0c0d1780 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f8c08006650 tx=0x7f8c08009a80).stop 2021-04-09T18:22:48.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.475 7f8c13a35700 1 -- 172.21.15.32:0/2574323256 shutdown_connections 2021-04-09T18:22:48.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.475 7f8c13a35700 1 --2- 172.21.15.32:0/2574323256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0d8490 0x7f8c0c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:48.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.475 7f8c13a35700 1 --2- 172.21.15.32:0/2574323256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c0c10d010 0x7f8c0c0d1780 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:48.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.475 7f8c13a35700 1 -- 172.21.15.32:0/2574323256 >> 172.21.15.32:0/2574323256 conn(0x7f8c0c0ce5f0 msgr2=0x7f8c0c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:48.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.475 7f8c13a35700 1 -- 172.21.15.32:0/2574323256 shutdown_connections 2021-04-09T18:22:48.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.475 7f8c13a35700 1 -- 172.21.15.32:0/2574323256 wait complete. 2021-04-09T18:22:48.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.495 7f8c13a35700 1 Processor -- start 2021-04-09T18:22:48.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.495 7f8c13a35700 1 -- start start 2021-04-09T18:22:48.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c13a35700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0b7010 0x7f8c0c09f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:48.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c13a35700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c0c0b7410 0x7f8c0c09f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:48.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c13a35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0c09d590 0x7f8c0c09d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:48.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c13a35700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0b7810 con 0x7f8c0c09d590 2021-04-09T18:22:48.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c13a35700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c19d160 con 0x7f8c0c0b7410 2021-04-09T18:22:48.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c13a35700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c09ea90 con 0x7f8c0c0b7010 2021-04-09T18:22:48.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c12052700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0c09d590 0x7f8c0c09d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:48.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c12052700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0c09d590 0x7f8c0c09d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49726/0 (socket says 172.21.15.32:49726) 2021-04-09T18:22:48.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c12052700 1 -- 172.21.15.32:0/1515205399 learned_addr learned my addr 172.21.15.32:0/1515205399 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:48.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c11851700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0b7010 0x7f8c0c09f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:48.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c11050700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c0c0b7410 0x7f8c0c09f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:48.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c12052700 1 -- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0b7010 msgr2=0x7f8c0c09f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:48.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c12052700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0b7010 0x7f8c0c09f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:48.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c12052700 1 -- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c0c0b7410 msgr2=0x7f8c0c09f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:48.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c12052700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c0c0b7410 0x7f8c0c09f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:48.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.496 7f8c12052700 1 -- 172.21.15.32:0/1515205399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c09dfb0 con 0x7f8c0c09d590 2021-04-09T18:22:48.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.497 7f8c11851700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0b7010 0x7f8c0c09f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:48.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.497 7f8c12052700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0c09d590 0x7f8c0c09d990 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7f8c04009940 tx=0x7f8c0400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:48.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.497 7f8bfe7fc700 1 -- 172.21.15.32:0/1515205399 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c04000f90 con 0x7f8c0c09d590 2021-04-09T18:22:48.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.497 7f8bfe7fc700 1 -- 172.21.15.32:0/1515205399 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04006790 con 0x7f8c0c09d590 2021-04-09T18:22:48.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.497 7f8bfe7fc700 1 -- 172.21.15.32:0/1515205399 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c0400c4e0 con 0x7f8c0c09d590 2021-04-09T18:22:48.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.497 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c09dfb0 con 0x7f8c0c09d590 2021-04-09T18:22:48.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.497 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c09dfb0 con 0x7f8c0c09d590 2021-04-09T18:22:48.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.498 7f8bfe7fc700 1 -- 172.21.15.32:0/1515205399 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8c040011f0 con 0x7f8c0c09d590 2021-04-09T18:22:48.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.498 7f8bfe7fc700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bf4018930 0x7f8bf401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:48.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.498 7f8bfe7fc700 1 -- 172.21.15.32:0/1515205399 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8c04032d70 con 0x7f8c0c09d590 2021-04-09T18:22:48.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.498 7f8c11851700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bf4018930 0x7f8bf401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:48.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.498 7f8c11851700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bf4018930 0x7f8bf401adc0 crc :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:48.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.517 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c0b7810 con 0x7f8c0c09d590 2021-04-09T18:22:48.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.521 7f8bfe7fc700 1 -- 172.21.15.32:0/1515205399 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8c04006770 con 0x7f8c0c09d590 2021-04-09T18:22:48.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:48.713 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f8c0c0b7810 con 0x7f8c0c09d590 2021-04-09T18:22:49.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.420 7f8bfe7fc700 1 -- 172.21.15.32:0/1515205399 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v128) v1 ==== 105+0+0 (secure 0 0 0) 0x7f8c04006770 con 0x7f8c0c09d590 2021-04-09T18:22:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bf4018930 msgr2=0x7f8bf401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bf4018930 0x7f8bf401adc0 crc :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0c09d590 msgr2=0x7f8c0c09d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0c09d590 0x7f8c0c09d990 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7f8c04009940 tx=0x7f8c0400bcf0).stop 2021-04-09T18:22:49.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 shutdown_connections 2021-04-09T18:22:49.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8bf4018930 0x7f8bf401adc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:49.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c0c09d590 0x7f8c0c09d990 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:49.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c0c0b7410 0x7f8c0c09f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:49.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 --2- 172.21.15.32:0/1515205399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c0c0b7010 0x7f8c0c09f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:49.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 >> 172.21.15.32:0/1515205399 conn(0x7f8c0c0b6770 msgr2=0x7f8c0c1818c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:49.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.431 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 shutdown_connections 2021-04-09T18:22:49.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.432 7f8c13a35700 1 -- 172.21.15.32:0/1515205399 wait complete. 2021-04-09T18:22:49.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:22:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.929 7f6598200700 1 Processor -- start 2021-04-09T18:22:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.929 7f6598200700 1 -- start start 2021-04-09T18:22:50.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.930 7f6598200700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f65900cffc0 0x7f65900d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.930 7f6598200700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65900d7820 0x7f65900d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.930 7f6598200700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65900d72a0 con 0x7f65900d8240 2021-04-09T18:22:50.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.930 7f6598200700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65900d9200 con 0x7f65900d7820 2021-04-09T18:22:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.930 7f6598200700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659010e000 con 0x7f65900cffc0 2021-04-09T18:22:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.930 7f659581b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65900d7820 0x7f65900d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.930 7f659581b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65900d7820 0x7f65900d7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49734/0 (socket says 172.21.15.32:49734) 2021-04-09T18:22:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.930 7f659581b700 1 -- 172.21.15.32:0/3218092369 learned_addr learned my addr 172.21.15.32:0/3218092369 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.930 7f659601c700 1 --2- 172.21.15.32:0/3218092369 >> v2:172.21.15.86:3300/0 conn(0x7f65900cffc0 0x7f65900d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f659581b700 1 -- 172.21.15.32:0/3218092369 >> v2:172.21.15.86:3300/0 conn(0x7f65900cffc0 msgr2=0x7f65900d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:50.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f659581b700 1 --2- 172.21.15.32:0/3218092369 >> v2:172.21.15.86:3300/0 conn(0x7f65900cffc0 0x7f65900d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f659581b700 1 -- 172.21.15.32:0/3218092369 >> v1:172.21.15.86:6789/0 conn(0x7f65900d8240 legacy=0x7f65900d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:50.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f659581b700 1 -- 172.21.15.32:0/3218092369 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659010e220 con 0x7f65900d7820 2021-04-09T18:22:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f659581b700 1 --2- 172.21.15.32:0/3218092369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65900d7820 0x7f65900d7c20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f6580006700 tx=0x7f6580009b80).ready entity=mon.0 client_cookie=d8d7211d442519f1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f659501a700 1 -- 172.21.15.32:0/3218092369 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f658000c0c0 con 0x7f65900d7820 2021-04-09T18:22:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f659501a700 1 -- 172.21.15.32:0/3218092369 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658000c440 con 0x7f65900d7820 2021-04-09T18:22:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f659501a700 1 -- 172.21.15.32:0/3218092369 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f658000c920 con 0x7f65900d7820 2021-04-09T18:22:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f6598200700 1 -- 172.21.15.32:0/3218092369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65900d7820 msgr2=0x7f65900d7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.931 7f6598200700 1 --2- 172.21.15.32:0/3218092369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65900d7820 0x7f65900d7c20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f6580006700 tx=0x7f6580009b80).stop 2021-04-09T18:22:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.932 7f6598200700 1 -- 172.21.15.32:0/3218092369 shutdown_connections 2021-04-09T18:22:50.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.932 7f6598200700 1 --2- 172.21.15.32:0/3218092369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65900d7820 0x7f65900d7c20 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.932 7f6598200700 1 --2- 172.21.15.32:0/3218092369 >> v2:172.21.15.86:3300/0 conn(0x7f65900cffc0 0x7f65900d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.932 7f6598200700 1 -- 172.21.15.32:0/3218092369 >> 172.21.15.32:0/3218092369 conn(0x7f65900ce5e0 msgr2=0x7f65900cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:50.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.932 7f6598200700 1 -- 172.21.15.32:0/3218092369 shutdown_connections 2021-04-09T18:22:50.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.932 7f6598200700 1 -- 172.21.15.32:0/3218092369 wait complete. 2021-04-09T18:22:50.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.952 7f6598200700 1 Processor -- start 2021-04-09T18:22:50.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.952 7f6598200700 1 -- start start 2021-04-09T18:22:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6598200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65901833a0 0x7f65900a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6598200700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65900a2550 0x7f65900a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6598200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590094d00 0x7f6590095100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6598200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65900a1fd0 con 0x7f65901833a0 2021-04-09T18:22:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6598200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65900a2f70 con 0x7f6590094d00 2021-04-09T18:22:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6598200700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6590095720 con 0x7f65900a2550 2021-04-09T18:22:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659601c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65901833a0 0x7f65900a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659601c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65901833a0 0x7f65900a3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49736/0 (socket says 172.21.15.32:49736) 2021-04-09T18:22:50.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659601c700 1 -- 172.21.15.32:0/2667040703 learned_addr learned my addr 172.21.15.32:0/2667040703 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:50.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659581b700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65900a2550 0x7f65900a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659601c700 1 -- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65900a2550 msgr2=0x7f65900a2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:50.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659601c700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65900a2550 0x7f65900a2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659681d700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590094d00 0x7f6590095100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659601c700 1 -- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590094d00 msgr2=0x7f6590095100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:50.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659601c700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590094d00 0x7f6590095100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659601c700 1 -- 172.21.15.32:0/2667040703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6590095940 con 0x7f65901833a0 2021-04-09T18:22:50.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659581b700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65900a2550 0x7f65900a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:22:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659681d700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590094d00 0x7f6590095100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:22:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f659601c700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65901833a0 0x7f65900a3a10 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f658c006f60 tx=0x7f658c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6586ffd700 1 -- 172.21.15.32:0/2667040703 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f658c00bd60 con 0x7f65901833a0 2021-04-09T18:22:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6586ffd700 1 -- 172.21.15.32:0/2667040703 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658c000c90 con 0x7f65901833a0 2021-04-09T18:22:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6598200700 1 -- 172.21.15.32:0/2667040703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6590095940 con 0x7f65901833a0 2021-04-09T18:22:50.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.953 7f6586ffd700 1 -- 172.21.15.32:0/2667040703 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f658c002220 con 0x7f65901833a0 2021-04-09T18:22:50.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.954 7f6598200700 1 -- 172.21.15.32:0/2667040703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6590095940 con 0x7f65901833a0 2021-04-09T18:22:50.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.954 7f6586ffd700 1 -- 172.21.15.32:0/2667040703 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f658c0052e0 con 0x7f65901833a0 2021-04-09T18:22:50.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.954 7f6586ffd700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6578018930 0x7f657801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.954 7f6586ffd700 1 -- 172.21.15.32:0/2667040703 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f658c0329d0 con 0x7f65901833a0 2021-04-09T18:22:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.954 7f659581b700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6578018930 0x7f657801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.955 7f659581b700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6578018930 0x7f657801ae10 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:50.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.974 7f6598200700 1 -- 172.21.15.32:0/2667040703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6590095be0 con 0x7f65901833a0 2021-04-09T18:22:50.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:49.978 7f6586ffd700 1 -- 172.21.15.32:0/2667040703 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f658c000c90 con 0x7f65901833a0 2021-04-09T18:22:50.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.170 7f6598200700 1 -- 172.21.15.32:0/2667040703 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f65900d72a0 con 0x7f65901833a0 2021-04-09T18:22:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.430 7f6586ffd700 1 -- 172.21.15.32:0/2667040703 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v130) v1 ==== 105+0+0 (secure 0 0 0) 0x7f658c0052e0 con 0x7f65901833a0 2021-04-09T18:22:50.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 -- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6578018930 msgr2=0x7f657801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:50.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6578018930 0x7f657801ae10 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 -- 172.21.15.32:0/2667040703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65901833a0 msgr2=0x7f65900a3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65901833a0 0x7f65900a3a10 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f658c006f60 tx=0x7f658c00b970).stop 2021-04-09T18:22:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 -- 172.21.15.32:0/2667040703 shutdown_connections 2021-04-09T18:22:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6578018930 0x7f657801ae10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590094d00 0x7f6590095100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65900a2550 0x7f65900a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 --2- 172.21.15.32:0/2667040703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65901833a0 0x7f65900a3a10 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 -- 172.21.15.32:0/2667040703 >> 172.21.15.32:0/2667040703 conn(0x7f659009fa00 msgr2=0x7f65900b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 -- 172.21.15.32:0/2667040703 shutdown_connections 2021-04-09T18:22:50.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.436 7f6598200700 1 -- 172.21.15.32:0/2667040703 wait complete. 2021-04-09T18:22:50.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:22:50.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.599 7fa020f92700 1 Processor -- start 2021-04-09T18:22:50.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.599 7fa020f92700 1 -- start start 2021-04-09T18:22:50.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa020f92700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa01c10d010 0x7fa01c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa020f92700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa01c0d1da0 0x7fa01c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa020f92700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0d82c0 0x7fa01c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa020f92700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c0d8d30 con 0x7fa01c0d1da0 2021-04-09T18:22:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa020f92700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c0d9220 con 0x7fa01c10d010 2021-04-09T18:22:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa020f92700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c10d4b0 con 0x7fa01c0d82c0 2021-04-09T18:22:50.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01a59c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa01c0d1da0 0x7fa01c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01a59c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa01c0d1da0 0x7fa01c0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49744/0 (socket says 172.21.15.32:49744) 2021-04-09T18:22:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01a59c700 1 -- 172.21.15.32:0/3695818640 learned_addr learned my addr 172.21.15.32:0/3695818640 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01ad9d700 1 --2- 172.21.15.32:0/3695818640 >> v2:172.21.15.86:3300/0 conn(0x7fa01c10d010 0x7fa01c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01b59e700 1 --2- 172.21.15.32:0/3695818640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0d82c0 0x7fa01c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01a59c700 1 -- 172.21.15.32:0/3695818640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0d82c0 msgr2=0x7fa01c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01a59c700 1 --2- 172.21.15.32:0/3695818640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0d82c0 0x7fa01c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01a59c700 1 -- 172.21.15.32:0/3695818640 >> v2:172.21.15.86:3300/0 conn(0x7fa01c10d010 msgr2=0x7fa01c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01a59c700 1 --2- 172.21.15.32:0/3695818640 >> v2:172.21.15.86:3300/0 conn(0x7fa01c10d010 0x7fa01c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01a59c700 1 -- 172.21.15.32:0/3695818640 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c0d86c0 con 0x7fa01c0d1da0 2021-04-09T18:22:50.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01ad9d700 1 --2- 172.21.15.32:0/3695818640 >> v2:172.21.15.86:3300/0 conn(0x7fa01c10d010 0x7fa01c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:50.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa01a59c700 1 --2- 172.21.15.32:0/3695818640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c0d1da0 0x7fa01c0d7ca0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7fa004006700 tx=0x7fa004009b00).ready entity=mon.0 client_cookie=6c970554ab637f8e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:50.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.600 7fa019d9b700 1 -- 172.21.15.32:0/3695818640 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa00400c0c0 con 0x7fa01c0d1da0 2021-04-09T18:22:50.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa019d9b700 1 -- 172.21.15.32:0/3695818640 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00400c400 con 0x7fa01c0d1da0 2021-04-09T18:22:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa019d9b700 1 -- 172.21.15.32:0/3695818640 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa00400c8e0 con 0x7fa01c0d1da0 2021-04-09T18:22:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa020f92700 1 -- 172.21.15.32:0/3695818640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c0d1da0 msgr2=0x7fa01c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa020f92700 1 --2- 172.21.15.32:0/3695818640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c0d1da0 0x7fa01c0d7ca0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7fa004006700 tx=0x7fa004009b00).stop 2021-04-09T18:22:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa020f92700 1 -- 172.21.15.32:0/3695818640 shutdown_connections 2021-04-09T18:22:50.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa020f92700 1 --2- 172.21.15.32:0/3695818640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0d82c0 0x7fa01c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa020f92700 1 --2- 172.21.15.32:0/3695818640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c0d1da0 0x7fa01c0d7ca0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa020f92700 1 --2- 172.21.15.32:0/3695818640 >> v2:172.21.15.86:3300/0 conn(0x7fa01c10d010 0x7fa01c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa020f92700 1 -- 172.21.15.32:0/3695818640 >> 172.21.15.32:0/3695818640 conn(0x7fa01c0ce5f0 msgr2=0x7fa01c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:50.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.601 7fa020f92700 1 -- 172.21.15.32:0/3695818640 shutdown_connections 2021-04-09T18:22:50.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.602 7fa020f92700 1 -- 172.21.15.32:0/3695818640 wait complete. 2021-04-09T18:22:50.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa020f92700 1 Processor -- start 2021-04-09T18:22:50.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa020f92700 1 -- start start 2021-04-09T18:22:50.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa020f92700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01c093dc0 0x7fa01c0941c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa020f92700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0b5710 0x7fa01c0b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa020f92700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c092670 0x7fa01c092a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa020f92700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0b6130 con 0x7fa01c092670 2021-04-09T18:22:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa020f92700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c099430 con 0x7fa01c093dc0 2021-04-09T18:22:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa020f92700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c093090 con 0x7fa01c0b5710 2021-04-09T18:22:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa01b59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c092670 0x7fa01c092a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa01b59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c092670 0x7fa01c092a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49754/0 (socket says 172.21.15.32:49754) 2021-04-09T18:22:50.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa01b59e700 1 -- 172.21.15.32:0/1456276146 learned_addr learned my addr 172.21.15.32:0/1456276146 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:50.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa01a59c700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0b5710 0x7fa01c0b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.622 7fa01ad9d700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01c093dc0 0x7fa01c0941c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa01b59e700 1 -- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0b5710 msgr2=0x7fa01c0b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:50.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa01b59e700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0b5710 0x7fa01c0b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa01b59e700 1 -- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01c093dc0 msgr2=0x7fa01c0941c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa01b59e700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01c093dc0 0x7fa01c0941c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa01b59e700 1 -- 172.21.15.32:0/1456276146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c0932b0 con 0x7fa01c092670 2021-04-09T18:22:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa01b59e700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c092670 0x7fa01c092a70 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7fa01400b8f0 tx=0x7fa01400bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa0137fe700 1 -- 172.21.15.32:0/1456276146 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa014000f90 con 0x7fa01c092670 2021-04-09T18:22:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa0137fe700 1 -- 172.21.15.32:0/1456276146 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01400bdd0 con 0x7fa01c092670 2021-04-09T18:22:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa020f92700 1 -- 172.21.15.32:0/1456276146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c0932b0 con 0x7fa01c092670 2021-04-09T18:22:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa0137fe700 1 -- 172.21.15.32:0/1456276146 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa014001470 con 0x7fa01c092670 2021-04-09T18:22:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.623 7fa020f92700 1 -- 172.21.15.32:0/1456276146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c0932b0 con 0x7fa01c092670 2021-04-09T18:22:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.624 7fa0137fe700 1 -- 172.21.15.32:0/1456276146 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa0140011f0 con 0x7fa01c092670 2021-04-09T18:22:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.624 7fa0137fe700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa000018930 0x7fa00001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.624 7fa0137fe700 1 -- 172.21.15.32:0/1456276146 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa014034f80 con 0x7fa01c092670 2021-04-09T18:22:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.624 7fa01ad9d700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa000018930 0x7fa00001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:50.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.624 7fa01ad9d700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa000018930 0x7fa00001ae10 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:50.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.644 7fa020f92700 1 -- 172.21.15.32:0/1456276146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c099410 con 0x7fa01c092670 2021-04-09T18:22:50.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.647 7fa0137fe700 1 -- 172.21.15.32:0/1456276146 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa0140066f0 con 0x7fa01c092670 2021-04-09T18:22:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:50.840 7fa020f92700 1 -- 172.21.15.32:0/1456276146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fa01c0b51e0 con 0x7fa01c092670 2021-04-09T18:22:51.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.439 7fa0137fe700 1 -- 172.21.15.32:0/1456276146 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v132) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa0140011f0 con 0x7fa01c092670 2021-04-09T18:22:51.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.457 7fa020f92700 1 -- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa000018930 msgr2=0x7fa00001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:51.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.457 7fa020f92700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa000018930 0x7fa00001ae10 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.457 7fa020f92700 1 -- 172.21.15.32:0/1456276146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c092670 msgr2=0x7fa01c092a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:51.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.457 7fa020f92700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c092670 0x7fa01c092a70 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7fa01400b8f0 tx=0x7fa01400bc90).stop 2021-04-09T18:22:51.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.457 7fa020f92700 1 -- 172.21.15.32:0/1456276146 shutdown_connections 2021-04-09T18:22:51.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.457 7fa020f92700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa000018930 0x7fa00001ae10 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.458 7fa020f92700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa01c092670 0x7fa01c092a70 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.458 7fa020f92700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01c0b5710 0x7fa01c0b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.458 7fa020f92700 1 --2- 172.21.15.32:0/1456276146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01c093dc0 0x7fa01c0941c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.458 7fa020f92700 1 -- 172.21.15.32:0/1456276146 >> 172.21.15.32:0/1456276146 conn(0x7fa01c0950b0 msgr2=0x7fa01c0b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:51.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.458 7fa020f92700 1 -- 172.21.15.32:0/1456276146 shutdown_connections 2021-04-09T18:22:51.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.458 7fa020f92700 1 -- 172.21.15.32:0/1456276146 wait complete. 2021-04-09T18:22:51.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:22:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.620 7fbea30b3700 1 Processor -- start 2021-04-09T18:22:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.620 7fbea30b3700 1 -- start start 2021-04-09T18:22:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbea30b3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbe9c10d010 0x7fbe9c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbea30b3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbe9c0d1da0 0x7fbe9c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbea30b3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c10d480 con 0x7fbe9c0d82c0 2021-04-09T18:22:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbea30b3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c0d8d30 con 0x7fbe9c0d1da0 2021-04-09T18:22:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbea30b3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c0d9220 con 0x7fbe9c10d010 2021-04-09T18:22:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbe9bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbe9c0d1da0 0x7fbe9c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbea0ecf700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbe9c10d010 0x7fbe9c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:51.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbe9bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbe9c0d1da0 0x7fbe9c0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49758/0 (socket says 172.21.15.32:49758) 2021-04-09T18:22:51.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbea16d0700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbe9c0d82c0 0x7fbe9c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:57278/0 (socket says 172.21.15.32:57278) 2021-04-09T18:22:51.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbea0ecf700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbe9c10d010 0x7fbe9c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:41516/0 (socket says 172.21.15.32:41516) 2021-04-09T18:22:51.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.621 7fbe9bfff700 1 -- 172.21.15.32:0/3782773159 learned_addr learned my addr 172.21.15.32:0/3782773159 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:51.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbe9bfff700 1 -- 172.21.15.32:0/3782773159 >> v2:172.21.15.86:3300/0 conn(0x7fbe9c10d010 msgr2=0x7fbe9c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:51.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbe9bfff700 1 --2- 172.21.15.32:0/3782773159 >> v2:172.21.15.86:3300/0 conn(0x7fbe9c10d010 0x7fbe9c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbe9bfff700 1 -- 172.21.15.32:0/3782773159 >> v1:172.21.15.86:6789/0 conn(0x7fbe9c0d82c0 legacy=0x7fbe9c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:51.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbe9bfff700 1 -- 172.21.15.32:0/3782773159 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0d86f0 con 0x7fbe9c0d1da0 2021-04-09T18:22:51.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbea0ecf700 1 --2- 172.21.15.32:0/3782773159 >> v2:172.21.15.86:3300/0 conn(0x7fbe9c10d010 0x7fbe9c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:51.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbe9bfff700 1 --2- 172.21.15.32:0/3782773159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0d1da0 0x7fbe9c0d7ca0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7fbe8c006700 tx=0x7fbe8c009b80).ready entity=mon.0 client_cookie=81b8a5457bf4a732 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:51.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbe9b7fe700 1 -- 172.21.15.32:0/3782773159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe8c00c0c0 con 0x7fbe9c0d1da0 2021-04-09T18:22:51.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbe9b7fe700 1 -- 172.21.15.32:0/3782773159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c00c440 con 0x7fbe9c0d1da0 2021-04-09T18:22:51.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbe9b7fe700 1 -- 172.21.15.32:0/3782773159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe8c00c920 con 0x7fbe9c0d1da0 2021-04-09T18:22:51.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbea30b3700 1 -- 172.21.15.32:0/3782773159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0d1da0 msgr2=0x7fbe9c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:51.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.622 7fbea30b3700 1 --2- 172.21.15.32:0/3782773159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0d1da0 0x7fbe9c0d7ca0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7fbe8c006700 tx=0x7fbe8c009b80).stop 2021-04-09T18:22:51.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.623 7fbea30b3700 1 -- 172.21.15.32:0/3782773159 shutdown_connections 2021-04-09T18:22:51.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.623 7fbea30b3700 1 --2- 172.21.15.32:0/3782773159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0d1da0 0x7fbe9c0d7ca0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.623 7fbea30b3700 1 --2- 172.21.15.32:0/3782773159 >> v2:172.21.15.86:3300/0 conn(0x7fbe9c10d010 0x7fbe9c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.623 7fbea30b3700 1 -- 172.21.15.32:0/3782773159 >> 172.21.15.32:0/3782773159 conn(0x7fbe9c0ce5f0 msgr2=0x7fbe9c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.623 7fbea30b3700 1 -- 172.21.15.32:0/3782773159 shutdown_connections 2021-04-09T18:22:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.623 7fbea30b3700 1 -- 172.21.15.32:0/3782773159 wait complete. 2021-04-09T18:22:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.643 7fbea30b3700 1 Processor -- start 2021-04-09T18:22:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.643 7fbea30b3700 1 -- start start 2021-04-09T18:22:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.643 7fbea30b3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe9c0a8b00 0x7fbe9c18c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.643 7fbea30b3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c0a8f00 0x7fbe9c18cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.643 7fbea30b3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0c2dd0 0x7fbe9c0c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.643 7fbea30b3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c18c300 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.643 7fbea30b3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c37f0 con 0x7fbe9c0a8f00 2021-04-09T18:22:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.643 7fbea30b3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c39d0 con 0x7fbe9c0a8b00 2021-04-09T18:22:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea16d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0c2dd0 0x7fbe9c0c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea16d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0c2dd0 0x7fbe9c0c31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49768/0 (socket says 172.21.15.32:49768) 2021-04-09T18:22:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea16d0700 1 -- 172.21.15.32:0/3236759303 learned_addr learned my addr 172.21.15.32:0/3236759303 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea0ecf700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe9c0a8b00 0x7fbe9c18c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbe9bfff700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c0a8f00 0x7fbe9c18cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea16d0700 1 -- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe9c0a8b00 msgr2=0x7fbe9c18c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea16d0700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe9c0a8b00 0x7fbe9c18c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea16d0700 1 -- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c0a8f00 msgr2=0x7fbe9c18cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea16d0700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c0a8f00 0x7fbe9c18cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea16d0700 1 -- 172.21.15.32:0/3236759303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0c3bf0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea16d0700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0c2dd0 0x7fbe9c0c31d0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7fbe9400a8d0 tx=0x7fbe9400ac70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbe997fa700 1 -- 172.21.15.32:0/3236759303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe940011f0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbe997fa700 1 -- 172.21.15.32:0/3236759303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe940070d0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c0c3bf0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbe997fa700 1 -- 172.21.15.32:0/3236759303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe940046a0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.644 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c0c3bf0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.645 7fbe997fa700 1 -- 172.21.15.32:0/3236759303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbe940072b0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.645 7fbe997fa700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe80018930 0x7fbe8001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.645 7fbe997fa700 1 -- 172.21.15.32:0/3236759303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbe94032db0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.645 7fbea0ecf700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe80018930 0x7fbe8001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:51.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.645 7fbea0ecf700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe80018930 0x7fbe8001ae10 crc :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:51.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.665 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c10d480 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.668 7fbe997fa700 1 -- 172.21.15.32:0/3236759303 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbe94000cc0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:51.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:51.862 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fbe9c10d480 con 0x7fbe9c0c2dd0 2021-04-09T18:22:52.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.445 7fbe997fa700 1 -- 172.21.15.32:0/3236759303 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v133) v1 ==== 105+0+0 (secure 0 0 0) 0x7fbe94006fc0 con 0x7fbe9c0c2dd0 2021-04-09T18:22:52.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe80018930 msgr2=0x7fbe8001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe80018930 0x7fbe8001ae10 crc :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0c2dd0 msgr2=0x7fbe9c0c31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:52.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0c2dd0 0x7fbe9c0c31d0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7fbe9400a8d0 tx=0x7fbe9400ac70).stop 2021-04-09T18:22:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 shutdown_connections 2021-04-09T18:22:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbe80018930 0x7fbe8001ae10 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe9c0c2dd0 0x7fbe9c0c31d0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c0a8f00 0x7fbe9c18cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 --2- 172.21.15.32:0/3236759303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe9c0a8b00 0x7fbe9c18c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 >> 172.21.15.32:0/3236759303 conn(0x7fbe9c0b93a0 msgr2=0x7fbe9c0b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 shutdown_connections 2021-04-09T18:22:52.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.479 7fbea30b3700 1 -- 172.21.15.32:0/3236759303 wait complete. 2021-04-09T18:22:52.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:22:52.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.642 7fcfdadeb700 1 Processor -- start 2021-04-09T18:22:52.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.642 7fcfdadeb700 1 -- start start 2021-04-09T18:22:52.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfdadeb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcfd410d010 0x7fcfd40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:52.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfdadeb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcfd40d1da0 0x7fcfd40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:52.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfdadeb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd410d480 con 0x7fcfd40d82c0 2021-04-09T18:22:52.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfdadeb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd40d8d30 con 0x7fcfd40d1da0 2021-04-09T18:22:52.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfdadeb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd40d9220 con 0x7fcfd410d010 2021-04-09T18:22:52.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcfd40d1da0 0x7fcfd40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:52.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd9408700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcfd40d82c0 0x7fcfd410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59362/0 (socket says 172.21.15.32:59362) 2021-04-09T18:22:52.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd9408700 1 -- 172.21.15.32:0/2933123692 learned_addr learned my addr 172.21.15.32:0/2933123692 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:52.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd8c07700 1 --2- 172.21.15.32:0/2933123692 >> v2:172.21.15.86:3300/0 conn(0x7fcfd410d010 0x7fcfd40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:52.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd3fff700 1 -- 172.21.15.32:0/2933123692 >> v2:172.21.15.86:3300/0 conn(0x7fcfd410d010 msgr2=0x7fcfd40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:52.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd3fff700 1 --2- 172.21.15.32:0/2933123692 >> v2:172.21.15.86:3300/0 conn(0x7fcfd410d010 0x7fcfd40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd3fff700 1 -- 172.21.15.32:0/2933123692 >> v1:172.21.15.32:6789/0 conn(0x7fcfd40d82c0 legacy=0x7fcfd410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:52.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd3fff700 1 -- 172.21.15.32:0/2933123692 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd40d86f0 con 0x7fcfd40d1da0 2021-04-09T18:22:52.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd37fe700 1 -- 172.21.15.32:0/2933123692 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257157799 0 0) 0x7fcfcc0009b0 con 0x7fcfd40d82c0 2021-04-09T18:22:52.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd8c07700 1 --2- 172.21.15.32:0/2933123692 >> v2:172.21.15.86:3300/0 conn(0x7fcfd410d010 0x7fcfd40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:52.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd3fff700 1 --2- 172.21.15.32:0/2933123692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd40d1da0 0x7fcfd40d7ca0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7fcfc40069d0 tx=0x7fcfc4009e40).ready entity=mon.0 client_cookie=4a258bc8a563541c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:52.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd37fe700 1 -- 172.21.15.32:0/2933123692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfc400b510 con 0x7fcfd40d1da0 2021-04-09T18:22:52.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd37fe700 1 -- 172.21.15.32:0/2933123692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc400b890 con 0x7fcfd40d1da0 2021-04-09T18:22:52.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfd37fe700 1 -- 172.21.15.32:0/2933123692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfc400bd70 con 0x7fcfd40d1da0 2021-04-09T18:22:52.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfdadeb700 1 -- 172.21.15.32:0/2933123692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd40d1da0 msgr2=0x7fcfd40d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.643 7fcfdadeb700 1 --2- 172.21.15.32:0/2933123692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd40d1da0 0x7fcfd40d7ca0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7fcfc40069d0 tx=0x7fcfc4009e40).stop 2021-04-09T18:22:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.644 7fcfdadeb700 1 -- 172.21.15.32:0/2933123692 shutdown_connections 2021-04-09T18:22:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.644 7fcfdadeb700 1 --2- 172.21.15.32:0/2933123692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd40d1da0 0x7fcfd40d7ca0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.644 7fcfdadeb700 1 --2- 172.21.15.32:0/2933123692 >> v2:172.21.15.86:3300/0 conn(0x7fcfd410d010 0x7fcfd40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.644 7fcfdadeb700 1 -- 172.21.15.32:0/2933123692 >> 172.21.15.32:0/2933123692 conn(0x7fcfd40ce5f0 msgr2=0x7fcfd40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.644 7fcfdadeb700 1 -- 172.21.15.32:0/2933123692 shutdown_connections 2021-04-09T18:22:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.644 7fcfdadeb700 1 -- 172.21.15.32:0/2933123692 wait complete. 2021-04-09T18:22:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.664 7fcfdadeb700 1 Processor -- start 2021-04-09T18:22:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.664 7fcfdadeb700 1 -- start start 2021-04-09T18:22:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfdadeb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfd40c3c00 0x7fcfd409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfdadeb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd4095240 0x7fcfd41a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfdadeb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfd41a1270 0x7fcfd40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfdadeb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd409cda0 con 0x7fcfd4095240 2021-04-09T18:22:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfdadeb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd40c3680 con 0x7fcfd40c3c00 2021-04-09T18:22:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfdadeb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd4095640 con 0x7fcfd41a1270 2021-04-09T18:22:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfd3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd4095240 0x7fcfd41a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfd3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd4095240 0x7fcfd41a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49778/0 (socket says 172.21.15.32:49778) 2021-04-09T18:22:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfd3fff700 1 -- 172.21.15.32:0/1761655482 learned_addr learned my addr 172.21.15.32:0/1761655482 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfd9408700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfd41a1270 0x7fcfd40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.665 7fcfd8c07700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfd40c3c00 0x7fcfd409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfd3fff700 1 -- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfd41a1270 msgr2=0x7fcfd40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfd3fff700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfd41a1270 0x7fcfd40b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfd3fff700 1 -- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfd40c3c00 msgr2=0x7fcfd409c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfd3fff700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfd40c3c00 0x7fcfd409c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfd3fff700 1 -- 172.21.15.32:0/1761655482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd41a1670 con 0x7fcfd4095240 2021-04-09T18:22:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfd3fff700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd4095240 0x7fcfd41a0c50 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7fcfc40099e0 tx=0x7fcfc400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfd17fa700 1 -- 172.21.15.32:0/1761655482 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfc4006240 con 0x7fcfd4095240 2021-04-09T18:22:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfd17fa700 1 -- 172.21.15.32:0/1761655482 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc400be60 con 0x7fcfd4095240 2021-04-09T18:22:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfd17fa700 1 -- 172.21.15.32:0/1761655482 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfc4005660 con 0x7fcfd4095240 2021-04-09T18:22:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd41a1670 con 0x7fcfd4095240 2021-04-09T18:22:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.666 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd41a1670 con 0x7fcfd4095240 2021-04-09T18:22:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.667 7fcfd17fa700 1 -- 172.21.15.32:0/1761655482 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcfc4005660 con 0x7fcfd4095240 2021-04-09T18:22:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.667 7fcfd17fa700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfb80188c0 0x7fcfb801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.667 7fcfd17fa700 1 -- 172.21.15.32:0/1761655482 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcfc4032d70 con 0x7fcfd4095240 2021-04-09T18:22:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.667 7fcfd8c07700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfb80188c0 0x7fcfb801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.667 7fcfd8c07700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfb80188c0 0x7fcfb801adc0 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:52.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.687 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd4093d50 con 0x7fcfd4095240 2021-04-09T18:22:52.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.690 7fcfd17fa700 1 -- 172.21.15.32:0/1761655482 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcfc40064a0 con 0x7fcfd4095240 2021-04-09T18:22:52.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:52.881 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7fcfd410d480 con 0x7fcfd4095240 2021-04-09T18:22:53.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.446 7fcfd17fa700 1 -- 172.21.15.32:0/1761655482 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v134) v1 ==== 105+0+0 (secure 0 0 0) 0x7fcfc4005660 con 0x7fcfd4095240 2021-04-09T18:22:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.498 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfb80188c0 msgr2=0x7fcfb801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.498 7fcfdadeb700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfb80188c0 0x7fcfb801adc0 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.498 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd4095240 msgr2=0x7fcfd41a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:53.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.498 7fcfdadeb700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd4095240 0x7fcfd41a0c50 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7fcfc40099e0 tx=0x7fcfc400bd20).stop 2021-04-09T18:22:53.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.499 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 shutdown_connections 2021-04-09T18:22:53.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.499 7fcfdadeb700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfb80188c0 0x7fcfb801adc0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.499 7fcfdadeb700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcfd4095240 0x7fcfd41a0c50 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.499 7fcfdadeb700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfd41a1270 0x7fcfd40b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.499 7fcfdadeb700 1 --2- 172.21.15.32:0/1761655482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfd40c3c00 0x7fcfd409c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.499 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 >> 172.21.15.32:0/1761655482 conn(0x7fcfd40975f0 msgr2=0x7fcfd4096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:53.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.499 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 shutdown_connections 2021-04-09T18:22:53.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.499 7fcfdadeb700 1 -- 172.21.15.32:0/1761655482 wait complete. 2021-04-09T18:22:53.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:22:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.664 7fe852cba700 1 Processor -- start 2021-04-09T18:22:53.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.664 7fe852cba700 1 -- start start 2021-04-09T18:22:53.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.665 7fe852cba700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe84c10d010 0x7fe84c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:53.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.665 7fe852cba700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0d1da0 0x7fe84c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:53.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.665 7fe852cba700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84c10d480 con 0x7fe84c0d82c0 2021-04-09T18:22:53.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.665 7fe852cba700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84c0d9220 con 0x7fe84c10d010 2021-04-09T18:22:53.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.665 7fe852cba700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c0d86c0 con 0x7fe84c0d1da0 2021-04-09T18:22:53.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.665 7fe850ad6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe84c10d010 0x7fe84c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:53.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.665 7fe850ad6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe84c10d010 0x7fe84c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49784/0 (socket says 172.21.15.32:49784) 2021-04-09T18:22:53.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.665 7fe850ad6700 1 -- 172.21.15.32:0/3869830070 learned_addr learned my addr 172.21.15.32:0/3869830070 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:53.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.665 7fe84bfff700 1 --2- 172.21.15.32:0/3869830070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0d1da0 0x7fe84c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:53.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe850ad6700 1 -- 172.21.15.32:0/3869830070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0d1da0 msgr2=0x7fe84c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:53.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe850ad6700 1 --2- 172.21.15.32:0/3869830070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0d1da0 0x7fe84c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe850ad6700 1 -- 172.21.15.32:0/3869830070 >> v1:172.21.15.86:6789/0 conn(0x7fe84c0d82c0 legacy=0x7fe84c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:53.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe850ad6700 1 -- 172.21.15.32:0/3869830070 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c10e1f0 con 0x7fe84c10d010 2021-04-09T18:22:53.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe84bfff700 1 --2- 172.21.15.32:0/3869830070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0d1da0 0x7fe84c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:53.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe850ad6700 1 --2- 172.21.15.32:0/3869830070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c10d010 0x7fe84c0d1780 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fe83c006700 tx=0x7fe83c009b80).ready entity=mon.0 client_cookie=6d1973fd0fb7b649 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:53.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe84b7fe700 1 -- 172.21.15.32:0/3869830070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe83c00c0c0 con 0x7fe84c10d010 2021-04-09T18:22:53.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe84b7fe700 1 -- 172.21.15.32:0/3869830070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c00c440 con 0x7fe84c10d010 2021-04-09T18:22:53.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe84b7fe700 1 -- 172.21.15.32:0/3869830070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe83c00c920 con 0x7fe84c10d010 2021-04-09T18:22:53.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe852cba700 1 -- 172.21.15.32:0/3869830070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c10d010 msgr2=0x7fe84c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:53.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.666 7fe852cba700 1 --2- 172.21.15.32:0/3869830070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c10d010 0x7fe84c0d1780 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fe83c006700 tx=0x7fe83c009b80).stop 2021-04-09T18:22:53.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.667 7fe852cba700 1 -- 172.21.15.32:0/3869830070 shutdown_connections 2021-04-09T18:22:53.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.667 7fe852cba700 1 --2- 172.21.15.32:0/3869830070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0d1da0 0x7fe84c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.667 7fe852cba700 1 --2- 172.21.15.32:0/3869830070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c10d010 0x7fe84c0d1780 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.667 7fe852cba700 1 -- 172.21.15.32:0/3869830070 >> 172.21.15.32:0/3869830070 conn(0x7fe84c0ce5f0 msgr2=0x7fe84c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:53.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.667 7fe852cba700 1 -- 172.21.15.32:0/3869830070 shutdown_connections 2021-04-09T18:22:53.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.667 7fe852cba700 1 -- 172.21.15.32:0/3869830070 wait complete. 2021-04-09T18:22:53.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.687 7fe852cba700 1 Processor -- start 2021-04-09T18:22:53.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.687 7fe852cba700 1 -- start start 2021-04-09T18:22:53.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.687 7fe852cba700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0a0cb0 0x7fe84c092550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:53.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.687 7fe852cba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe84c0932e0 0x7fe84c08a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:53.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.687 7fe852cba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c0936e0 0x7fe84c08ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:53.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.687 7fe852cba700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c0b34f0 con 0x7fe84c0936e0 2021-04-09T18:22:53.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.687 7fe852cba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c092d60 con 0x7fe84c0932e0 2021-04-09T18:22:53.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.687 7fe852cba700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c0900a0 con 0x7fe84c0a0cb0 2021-04-09T18:22:53.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8512d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c0936e0 0x7fe84c08ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:53.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8512d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c0936e0 0x7fe84c08ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49794/0 (socket says 172.21.15.32:49794) 2021-04-09T18:22:53.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8512d7700 1 -- 172.21.15.32:0/1321909388 learned_addr learned my addr 172.21.15.32:0/1321909388 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:53.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe850ad6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0a0cb0 0x7fe84c092550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:53.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe84bfff700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe84c0932e0 0x7fe84c08a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:53.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8512d7700 1 -- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0a0cb0 msgr2=0x7fe84c092550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:53.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8512d7700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0a0cb0 0x7fe84c092550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8512d7700 1 -- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe84c0932e0 msgr2=0x7fe84c08a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:53.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8512d7700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe84c0932e0 0x7fe84c08a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:53.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8512d7700 1 -- 172.21.15.32:0/1321909388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c0a10b0 con 0x7fe84c0936e0 2021-04-09T18:22:53.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8512d7700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c0936e0 0x7fe84c08ac40 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fe844009940 tx=0x7fe84400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:53.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8497fa700 1 -- 172.21.15.32:0/1321909388 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe844000f40 con 0x7fe84c0936e0 2021-04-09T18:22:53.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8497fa700 1 -- 172.21.15.32:0/1321909388 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84400c2f0 con 0x7fe84c0936e0 2021-04-09T18:22:53.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe8497fa700 1 -- 172.21.15.32:0/1321909388 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe84400c7d0 con 0x7fe84c0936e0 2021-04-09T18:22:53.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe852cba700 1 -- 172.21.15.32:0/1321909388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84c0a10b0 con 0x7fe84c0936e0 2021-04-09T18:22:53.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.688 7fe852cba700 1 -- 172.21.15.32:0/1321909388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84c0a10b0 con 0x7fe84c0936e0 2021-04-09T18:22:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.689 7fe8497fa700 1 -- 172.21.15.32:0/1321909388 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe844001220 con 0x7fe84c0936e0 2021-04-09T18:22:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.689 7fe8497fa700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe82c0188c0 0x7fe82c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.689 7fe8497fa700 1 -- 172.21.15.32:0/1321909388 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe844032f00 con 0x7fe84c0936e0 2021-04-09T18:22:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.689 7fe850ad6700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe82c0188c0 0x7fe82c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.690 7fe850ad6700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe82c0188c0 0x7fe82c01adc0 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.709 7fe852cba700 1 -- 172.21.15.32:0/1321909388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84c10d480 con 0x7fe84c0936e0 2021-04-09T18:22:53.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.713 7fe8497fa700 1 -- 172.21.15.32:0/1321909388 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe84400be30 con 0x7fe84c0936e0 2021-04-09T18:22:53.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:53.903 7fe852cba700 1 -- 172.21.15.32:0/1321909388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fe84c10d480 con 0x7fe84c0936e0 2021-04-09T18:22:54.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.451 7fe8497fa700 1 -- 172.21.15.32:0/1321909388 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v135) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe844001220 con 0x7fe84c0936e0 2021-04-09T18:22:54.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 -- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe82c0188c0 msgr2=0x7fe82c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:54.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe82c0188c0 0x7fe82c01adc0 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 -- 172.21.15.32:0/1321909388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c0936e0 msgr2=0x7fe84c08ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:54.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c0936e0 0x7fe84c08ac40 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fe844009940 tx=0x7fe84400bcf0).stop 2021-04-09T18:22:54.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 -- 172.21.15.32:0/1321909388 shutdown_connections 2021-04-09T18:22:54.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe82c0188c0 0x7fe82c01adc0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe84c0936e0 0x7fe84c08ac40 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe84c0932e0 0x7fe84c08a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 --2- 172.21.15.32:0/1321909388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe84c0a0cb0 0x7fe84c092550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 -- 172.21.15.32:0/1321909388 >> 172.21.15.32:0/1321909388 conn(0x7fe84c0b2fa0 msgr2=0x7fe84c0a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:54.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 -- 172.21.15.32:0/1321909388 shutdown_connections 2021-04-09T18:22:54.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.471 7fe852cba700 1 -- 172.21.15.32:0/1321909388 wait complete. 2021-04-09T18:22:54.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:22:54.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.639 7f22a2deb700 1 Processor -- start 2021-04-09T18:22:54.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.639 7f22a2deb700 1 -- start start 2021-04-09T18:22:54.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.640 7f22a2deb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f229c10d010 0x7f229c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:54.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.640 7f22a2deb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f229c0d8490 0x7f229c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.640 7f22a2deb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c10d480 con 0x7f229c0d1da0 2021-04-09T18:22:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.640 7f22a2deb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229c0d8d30 con 0x7f229c10d010 2021-04-09T18:22:54.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.640 7f22a2deb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229c0d9220 con 0x7f229c0d8490 2021-04-09T18:22:54.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.640 7f22a0c07700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f229c10d010 0x7f229c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:54.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.640 7f22a0c07700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f229c10d010 0x7f229c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49798/0 (socket says 172.21.15.32:49798) 2021-04-09T18:22:54.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.640 7f22a0c07700 1 -- 172.21.15.32:0/4255627434 learned_addr learned my addr 172.21.15.32:0/4255627434 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:54.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.640 7f22a1408700 1 --2- 172.21.15.32:0/4255627434 >> v2:172.21.15.86:3300/0 conn(0x7f229c0d8490 0x7f229c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:54.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f22a0c07700 1 -- 172.21.15.32:0/4255627434 >> v2:172.21.15.86:3300/0 conn(0x7f229c0d8490 msgr2=0x7f229c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:54.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f22a0c07700 1 --2- 172.21.15.32:0/4255627434 >> v2:172.21.15.86:3300/0 conn(0x7f229c0d8490 0x7f229c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f22a0c07700 1 -- 172.21.15.32:0/4255627434 >> v1:172.21.15.86:6789/0 conn(0x7f229c0d1da0 legacy=0x7f229c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:54.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f22a0c07700 1 -- 172.21.15.32:0/4255627434 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c10e020 con 0x7f229c10d010 2021-04-09T18:22:54.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f22a0c07700 1 --2- 172.21.15.32:0/4255627434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c10d010 0x7f229c0d1780 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7f228c006700 tx=0x7f228c009b80).ready entity=mon.0 client_cookie=f9799dd3d8eeed51 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:54.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f229b7fe700 1 -- 172.21.15.32:0/4255627434 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228c00c0c0 con 0x7f229c10d010 2021-04-09T18:22:54.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f229b7fe700 1 -- 172.21.15.32:0/4255627434 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c00c440 con 0x7f229c10d010 2021-04-09T18:22:54.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f229b7fe700 1 -- 172.21.15.32:0/4255627434 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228c00c920 con 0x7f229c10d010 2021-04-09T18:22:54.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f22a2deb700 1 -- 172.21.15.32:0/4255627434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c10d010 msgr2=0x7f229c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:54.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.641 7f22a2deb700 1 --2- 172.21.15.32:0/4255627434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c10d010 0x7f229c0d1780 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7f228c006700 tx=0x7f228c009b80).stop 2021-04-09T18:22:54.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.642 7f22a2deb700 1 -- 172.21.15.32:0/4255627434 shutdown_connections 2021-04-09T18:22:54.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.642 7f22a2deb700 1 --2- 172.21.15.32:0/4255627434 >> v2:172.21.15.86:3300/0 conn(0x7f229c0d8490 0x7f229c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.642 7f22a2deb700 1 --2- 172.21.15.32:0/4255627434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c10d010 0x7f229c0d1780 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.642 7f22a2deb700 1 -- 172.21.15.32:0/4255627434 >> 172.21.15.32:0/4255627434 conn(0x7f229c0ce5f0 msgr2=0x7f229c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:54.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.642 7f22a2deb700 1 -- 172.21.15.32:0/4255627434 shutdown_connections 2021-04-09T18:22:54.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.642 7f22a2deb700 1 -- 172.21.15.32:0/4255627434 wait complete. 2021-04-09T18:22:54.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.662 7f22a2deb700 1 Processor -- start 2021-04-09T18:22:54.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.662 7f22a2deb700 1 -- start start 2021-04-09T18:22:54.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a2deb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f229c0c3c00 0x7f229c09c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:54.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a2deb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f229c095240 0x7f229c1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:54.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a2deb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c1a1270 0x7f229c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:54.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a2deb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c09cda0 con 0x7f229c1a1270 2021-04-09T18:22:54.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a2deb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c0c3680 con 0x7f229c0c3c00 2021-04-09T18:22:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a2deb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c095640 con 0x7f229c095240 2021-04-09T18:22:54.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a1408700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c1a1270 0x7f229c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:54.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a1408700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c1a1270 0x7f229c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49808/0 (socket says 172.21.15.32:49808) 2021-04-09T18:22:54.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a1408700 1 -- 172.21.15.32:0/2454093933 learned_addr learned my addr 172.21.15.32:0/2454093933 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:54.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f229bfff700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f229c095240 0x7f229c1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:54.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a0c07700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f229c0c3c00 0x7f229c09c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:54.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a1408700 1 -- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f229c095240 msgr2=0x7f229c1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:54.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a1408700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f229c095240 0x7f229c1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a1408700 1 -- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f229c0c3c00 msgr2=0x7f229c09c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:54.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a1408700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f229c0c3c00 0x7f229c09c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:54.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a1408700 1 -- 172.21.15.32:0/2454093933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c1a1670 con 0x7f229c1a1270 2021-04-09T18:22:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a0c07700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f229c0c3c00 0x7f229c09c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:22:54.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f229bfff700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f229c095240 0x7f229c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:22:54.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a1408700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c1a1270 0x7f229c0b0cf0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f229400b970 tx=0x7f229400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:54.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22997fa700 1 -- 172.21.15.32:0/2454093933 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2294000f90 con 0x7f229c1a1270 2021-04-09T18:22:54.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22997fa700 1 -- 172.21.15.32:0/2454093933 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2294006aa0 con 0x7f229c1a1270 2021-04-09T18:22:54.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229c1a1670 con 0x7f229c1a1270 2021-04-09T18:22:54.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22997fa700 1 -- 172.21.15.32:0/2454093933 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22940015f0 con 0x7f229c1a1270 2021-04-09T18:22:54.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.663 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229c1a1670 con 0x7f229c1a1270 2021-04-09T18:22:54.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.664 7f22997fa700 1 -- 172.21.15.32:0/2454093933 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2294000ed0 con 0x7f229c1a1270 2021-04-09T18:22:54.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.664 7f22997fa700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f227c0188d0 0x7f227c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:54.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.664 7f22997fa700 1 -- 172.21.15.32:0/2454093933 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2294032da0 con 0x7f229c1a1270 2021-04-09T18:22:54.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.665 7f22a0c07700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f227c0188d0 0x7f227c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:54.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.665 7f22a0c07700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f227c0188d0 0x7f227c01add0 crc :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:54.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.684 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229c093d50 con 0x7f229c1a1270 2021-04-09T18:22:54.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.688 7f22997fa700 1 -- 172.21.15.32:0/2454093933 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f22940066f0 con 0x7f229c1a1270 2021-04-09T18:22:54.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:54.880 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f229c10d480 con 0x7f229c1a1270 2021-04-09T18:22:55.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.453 7f22997fa700 1 -- 172.21.15.32:0/2454093933 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v136) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2294000ed0 con 0x7f229c1a1270 2021-04-09T18:22:55.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f227c0188d0 msgr2=0x7f227c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f227c0188d0 0x7f227c01add0 crc :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c1a1270 msgr2=0x7f229c0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c1a1270 0x7f229c0b0cf0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f229400b970 tx=0x7f229400bd10).stop 2021-04-09T18:22:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 shutdown_connections 2021-04-09T18:22:55.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f227c0188d0 0x7f227c01add0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f229c1a1270 0x7f229c0b0cf0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f229c095240 0x7f229c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 --2- 172.21.15.32:0/2454093933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f229c0c3c00 0x7f229c09c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 >> 172.21.15.32:0/2454093933 conn(0x7f229c0975f0 msgr2=0x7f229c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:55.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 shutdown_connections 2021-04-09T18:22:55.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.498 7f22a2deb700 1 -- 172.21.15.32:0/2454093933 wait complete. 2021-04-09T18:22:55.564 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:22:55.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.675 7ff165685700 1 Processor -- start 2021-04-09T18:22:55.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.675 7ff165685700 1 -- start start 2021-04-09T18:22:55.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.675 7ff165685700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff1600cffc0 0x7ff1600d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff165685700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff1600d6ca0 0x7ff16010cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff165685700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1600d72a0 con 0x7ff1600d7820 2021-04-09T18:22:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff165685700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1600d70a0 con 0x7ff1600cffc0 2021-04-09T18:22:55.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff165685700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1600d9200 con 0x7ff1600d6ca0 2021-04-09T18:22:55.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15f2bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff1600cffc0 0x7ff1600d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:55.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15f2bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff1600cffc0 0x7ff1600d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49812/0 (socket says 172.21.15.32:49812) 2021-04-09T18:22:55.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15f2bf700 1 -- 172.21.15.32:0/1274705110 learned_addr learned my addr 172.21.15.32:0/1274705110 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15fac0700 1 --2- 172.21.15.32:0/1274705110 >> v2:172.21.15.86:3300/0 conn(0x7ff1600d6ca0 0x7ff16010cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15f2bf700 1 -- 172.21.15.32:0/1274705110 >> v2:172.21.15.86:3300/0 conn(0x7ff1600d6ca0 msgr2=0x7ff16010cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15f2bf700 1 --2- 172.21.15.32:0/1274705110 >> v2:172.21.15.86:3300/0 conn(0x7ff1600d6ca0 0x7ff16010cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15f2bf700 1 -- 172.21.15.32:0/1274705110 >> v1:172.21.15.86:6789/0 conn(0x7ff1600d7820 legacy=0x7ff1600d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15f2bf700 1 -- 172.21.15.32:0/1274705110 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1600d8410 con 0x7ff1600cffc0 2021-04-09T18:22:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15f2bf700 1 --2- 172.21.15.32:0/1274705110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600cffc0 0x7ff1600d2460 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7ff150006700 tx=0x7ff150009b80).ready entity=mon.0 client_cookie=86208ce06d81ef85 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15e2bd700 1 -- 172.21.15.32:0/1274705110 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff15000c0c0 con 0x7ff1600cffc0 2021-04-09T18:22:55.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15e2bd700 1 -- 172.21.15.32:0/1274705110 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15000c440 con 0x7ff1600cffc0 2021-04-09T18:22:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff15e2bd700 1 -- 172.21.15.32:0/1274705110 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff15000c920 con 0x7ff1600cffc0 2021-04-09T18:22:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff165685700 1 -- 172.21.15.32:0/1274705110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600cffc0 msgr2=0x7ff1600d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:55.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.676 7ff165685700 1 --2- 172.21.15.32:0/1274705110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600cffc0 0x7ff1600d2460 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7ff150006700 tx=0x7ff150009b80).stop 2021-04-09T18:22:55.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.677 7ff165685700 1 -- 172.21.15.32:0/1274705110 shutdown_connections 2021-04-09T18:22:55.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.677 7ff165685700 1 --2- 172.21.15.32:0/1274705110 >> v2:172.21.15.86:3300/0 conn(0x7ff1600d6ca0 0x7ff16010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.677 7ff165685700 1 --2- 172.21.15.32:0/1274705110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600cffc0 0x7ff1600d2460 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.677 7ff165685700 1 -- 172.21.15.32:0/1274705110 >> 172.21.15.32:0/1274705110 conn(0x7ff1600ce5e0 msgr2=0x7ff1600cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:55.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.677 7ff165685700 1 -- 172.21.15.32:0/1274705110 shutdown_connections 2021-04-09T18:22:55.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.677 7ff165685700 1 -- 172.21.15.32:0/1274705110 wait complete. 2021-04-09T18:22:55.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.697 7ff165685700 1 Processor -- start 2021-04-09T18:22:55.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.697 7ff165685700 1 -- start start 2021-04-09T18:22:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff165685700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1600b89a0 0x7ff1600c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff165685700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1600b8da0 0x7ff1600be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:55.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff165685700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600b91a0 0x7ff1600bead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff165685700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1600c39b0 con 0x7ff1600b91a0 2021-04-09T18:22:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff165685700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1600c0d60 con 0x7ff1600b8da0 2021-04-09T18:22:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff165685700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1600bf0f0 con 0x7ff1600b89a0 2021-04-09T18:22:55.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15fac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600b91a0 0x7ff1600bead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15fac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600b91a0 0x7ff1600bead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49822/0 (socket says 172.21.15.32:49822) 2021-04-09T18:22:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15fac0700 1 -- 172.21.15.32:0/1742840478 learned_addr learned my addr 172.21.15.32:0/1742840478 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:55.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15f2bf700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1600b89a0 0x7ff1600c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15eabe700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1600b8da0 0x7ff1600be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15fac0700 1 -- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1600b89a0 msgr2=0x7ff1600c31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:55.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15fac0700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1600b89a0 0x7ff1600c31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15fac0700 1 -- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1600b8da0 msgr2=0x7ff1600be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15fac0700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1600b8da0 0x7ff1600be4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:55.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15fac0700 1 -- 172.21.15.32:0/1742840478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1600bf310 con 0x7ff1600b91a0 2021-04-09T18:22:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.698 7ff15fac0700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600b91a0 0x7ff1600bead0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7ff15800b810 tx=0x7ff15800bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.699 7ff147fff700 1 -- 172.21.15.32:0/1742840478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff15800bdb0 con 0x7ff1600b91a0 2021-04-09T18:22:55.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.699 7ff147fff700 1 -- 172.21.15.32:0/1742840478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff158003030 con 0x7ff1600b91a0 2021-04-09T18:22:55.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.699 7ff165685700 1 -- 172.21.15.32:0/1742840478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1600bf310 con 0x7ff1600b91a0 2021-04-09T18:22:55.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.699 7ff147fff700 1 -- 172.21.15.32:0/1742840478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff158001470 con 0x7ff1600b91a0 2021-04-09T18:22:55.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.699 7ff165685700 1 -- 172.21.15.32:0/1742840478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1600bf310 con 0x7ff1600b91a0 2021-04-09T18:22:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.699 7ff147fff700 1 -- 172.21.15.32:0/1742840478 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff1580011f0 con 0x7ff1600b91a0 2021-04-09T18:22:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.699 7ff147fff700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff140018930 0x7ff14001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:55.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.699 7ff147fff700 1 -- 172.21.15.32:0/1742840478 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff158035100 con 0x7ff1600b91a0 2021-04-09T18:22:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.700 7ff15f2bf700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff140018930 0x7ff14001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:55.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.700 7ff15f2bf700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff140018930 0x7ff14001ae10 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:55.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.720 7ff165685700 1 -- 172.21.15.32:0/1742840478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1601b70c0 con 0x7ff1600b91a0 2021-04-09T18:22:55.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.723 7ff147fff700 1 -- 172.21.15.32:0/1742840478 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff1580066f0 con 0x7ff1600b91a0 2021-04-09T18:22:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:55.917 7ff165685700 1 -- 172.21.15.32:0/1742840478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7ff1601b70c0 con 0x7ff1600b91a0 2021-04-09T18:22:56.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.464 7ff147fff700 1 -- 172.21.15.32:0/1742840478 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v137) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff158021cc0 con 0x7ff1600b91a0 2021-04-09T18:22:56.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 -- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff140018930 msgr2=0x7ff14001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:56.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff140018930 0x7ff14001ae10 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 -- 172.21.15.32:0/1742840478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600b91a0 msgr2=0x7ff1600bead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:56.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600b91a0 0x7ff1600bead0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7ff15800b810 tx=0x7ff15800bc70).stop 2021-04-09T18:22:56.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 -- 172.21.15.32:0/1742840478 shutdown_connections 2021-04-09T18:22:56.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff140018930 0x7ff14001ae10 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1600b91a0 0x7ff1600bead0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1600b8da0 0x7ff1600be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 --2- 172.21.15.32:0/1742840478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1600b89a0 0x7ff1600c31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 -- 172.21.15.32:0/1742840478 >> 172.21.15.32:0/1742840478 conn(0x7ff1600a06a0 msgr2=0x7ff16008fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 -- 172.21.15.32:0/1742840478 shutdown_connections 2021-04-09T18:22:56.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.484 7ff165685700 1 -- 172.21.15.32:0/1742840478 wait complete. 2021-04-09T18:22:56.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:22:56.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.656 7f5b28a1c700 1 Processor -- start 2021-04-09T18:22:56.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.656 7f5b28a1c700 1 -- start start 2021-04-09T18:22:56.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.657 7f5b28a1c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5b240d7ca0 0x7f5b240d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:56.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.657 7f5b28a1c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5b2410da00 0x7f5b2410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:56.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.657 7f5b28a1c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2410d480 con 0x7f5b2410d010 2021-04-09T18:22:56.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.657 7f5b28a1c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240d9220 con 0x7f5b240d7ca0 2021-04-09T18:22:56.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.657 7f5b28a1c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240d86c0 con 0x7f5b2410da00 2021-04-09T18:22:56.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.657 7f5b21e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5b240d7ca0 0x7f5b240d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:56.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.657 7f5b2261c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5b2410d010 0x7f5b240d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59416/0 (socket says 172.21.15.32:59416) 2021-04-09T18:22:56.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.657 7f5b2261c700 1 -- 172.21.15.32:0/4195796483 learned_addr learned my addr 172.21.15.32:0/4195796483 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:56.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.657 7f5b22e1d700 1 --2- 172.21.15.32:0/4195796483 >> v2:172.21.15.86:3300/0 conn(0x7f5b2410da00 0x7f5b2410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:56.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b21e1b700 1 -- 172.21.15.32:0/4195796483 >> v2:172.21.15.86:3300/0 conn(0x7f5b2410da00 msgr2=0x7f5b2410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:56.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b21e1b700 1 --2- 172.21.15.32:0/4195796483 >> v2:172.21.15.86:3300/0 conn(0x7f5b2410da00 0x7f5b2410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b21e1b700 1 -- 172.21.15.32:0/4195796483 >> v1:172.21.15.32:6789/0 conn(0x7f5b2410d010 legacy=0x7f5b240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:56.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b2161a700 1 -- 172.21.15.32:0/4195796483 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658342142 0 0) 0x7f5b14000ce0 con 0x7f5b2410d010 2021-04-09T18:22:56.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b21e1b700 1 -- 172.21.15.32:0/4195796483 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240d1f70 con 0x7f5b240d7ca0 2021-04-09T18:22:56.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b21e1b700 1 --2- 172.21.15.32:0/4195796483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240d7ca0 0x7f5b240d80a0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f5b10006650 tx=0x7f5b10009a80).ready entity=mon.0 client_cookie=2ab4e22c6806d578 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:56.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b2161a700 1 -- 172.21.15.32:0/4195796483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b100009b0 con 0x7f5b240d7ca0 2021-04-09T18:22:56.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b2161a700 1 -- 172.21.15.32:0/4195796483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1000c0c0 con 0x7f5b240d7ca0 2021-04-09T18:22:56.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b2161a700 1 -- 172.21.15.32:0/4195796483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b1000c510 con 0x7f5b240d7ca0 2021-04-09T18:22:56.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b28a1c700 1 -- 172.21.15.32:0/4195796483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240d7ca0 msgr2=0x7f5b240d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:56.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.658 7f5b28a1c700 1 --2- 172.21.15.32:0/4195796483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240d7ca0 0x7f5b240d80a0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f5b10006650 tx=0x7f5b10009a80).stop 2021-04-09T18:22:56.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.659 7f5b28a1c700 1 -- 172.21.15.32:0/4195796483 shutdown_connections 2021-04-09T18:22:56.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.659 7f5b28a1c700 1 --2- 172.21.15.32:0/4195796483 >> v2:172.21.15.86:3300/0 conn(0x7f5b2410da00 0x7f5b2410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.659 7f5b28a1c700 1 --2- 172.21.15.32:0/4195796483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240d7ca0 0x7f5b240d80a0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.659 7f5b28a1c700 1 -- 172.21.15.32:0/4195796483 >> 172.21.15.32:0/4195796483 conn(0x7f5b240ce5f0 msgr2=0x7f5b240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:56.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.659 7f5b28a1c700 1 -- 172.21.15.32:0/4195796483 shutdown_connections 2021-04-09T18:22:56.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.659 7f5b28a1c700 1 -- 172.21.15.32:0/4195796483 wait complete. 2021-04-09T18:22:56.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.679 7f5b28a1c700 1 Processor -- start 2021-04-09T18:22:56.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.679 7f5b28a1c700 1 -- start start 2021-04-09T18:22:56.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b28a1c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b240b1e50 0x7f5b240b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:56.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b28a1c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240b9fd0 0x7f5b240b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:56.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b28a1c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b240b5510 0x7f5b240b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:56.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b28a1c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240b2250 con 0x7f5b240b9fd0 2021-04-09T18:22:56.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b28a1c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240a1220 con 0x7f5b240b5510 2021-04-09T18:22:56.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b28a1c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240b6b30 con 0x7f5b240b1e50 2021-04-09T18:22:56.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b21e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240b9fd0 0x7f5b240b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:56.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b21e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240b9fd0 0x7f5b240b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49836/0 (socket says 172.21.15.32:49836) 2021-04-09T18:22:56.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b21e1b700 1 -- 172.21.15.32:0/2866328864 learned_addr learned my addr 172.21.15.32:0/2866328864 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:56.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b2261c700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b240b1e50 0x7f5b240b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:56.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b22e1d700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b240b5510 0x7f5b240b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:56.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b21e1b700 1 -- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b240b1e50 msgr2=0x7f5b240b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:56.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b21e1b700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b240b1e50 0x7f5b240b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b21e1b700 1 -- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b240b5510 msgr2=0x7f5b240b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:56.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b21e1b700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b240b5510 0x7f5b240b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:56.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b21e1b700 1 -- 172.21.15.32:0/2866328864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240b5f30 con 0x7f5b240b9fd0 2021-04-09T18:22:56.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b21e1b700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240b9fd0 0x7f5b240b4ef0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7f5b10009650 tx=0x7f5b1000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:56.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b1affd700 1 -- 172.21.15.32:0/2866328864 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b100093a0 con 0x7f5b240b9fd0 2021-04-09T18:22:56.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b1affd700 1 -- 172.21.15.32:0/2866328864 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1000bd50 con 0x7f5b240b9fd0 2021-04-09T18:22:56.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b240b5f30 con 0x7f5b240b9fd0 2021-04-09T18:22:56.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b1affd700 1 -- 172.21.15.32:0/2866328864 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b100052e0 con 0x7f5b240b9fd0 2021-04-09T18:22:56.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.680 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b240b5f30 con 0x7f5b240b9fd0 2021-04-09T18:22:56.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.681 7f5b1affd700 1 -- 172.21.15.32:0/2866328864 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5b10006030 con 0x7f5b240b9fd0 2021-04-09T18:22:56.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.681 7f5b1affd700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b00018930 0x7f5b0001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:56.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.681 7f5b1affd700 1 -- 172.21.15.32:0/2866328864 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5b100329e0 con 0x7f5b240b9fd0 2021-04-09T18:22:56.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.681 7f5b2261c700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b00018930 0x7f5b0001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:56.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.682 7f5b2261c700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b00018930 0x7f5b0001adc0 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:56.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.701 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b241b7040 con 0x7f5b240b9fd0 2021-04-09T18:22:56.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.705 7f5b1affd700 1 -- 172.21.15.32:0/2866328864 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5b100009c0 con 0x7f5b240b9fd0 2021-04-09T18:22:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:56.900 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f5b2410d480 con 0x7f5b240b9fd0 2021-04-09T18:22:57.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.467 7f5b1affd700 1 -- 172.21.15.32:0/2866328864 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v138) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5b10006030 con 0x7f5b240b9fd0 2021-04-09T18:22:57.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.517 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b00018930 msgr2=0x7f5b0001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:57.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.517 7f5b28a1c700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b00018930 0x7f5b0001adc0 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240b9fd0 msgr2=0x7f5b240b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:57.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240b9fd0 0x7f5b240b4ef0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7f5b10009650 tx=0x7f5b1000b960).stop 2021-04-09T18:22:57.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 shutdown_connections 2021-04-09T18:22:57.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b00018930 0x7f5b0001adc0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b240b9fd0 0x7f5b240b4ef0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b240b5510 0x7f5b240b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 --2- 172.21.15.32:0/2866328864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b240b1e50 0x7f5b240b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 >> 172.21.15.32:0/2866328864 conn(0x7f5b240b27e0 msgr2=0x7f5b241b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:57.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 shutdown_connections 2021-04-09T18:22:57.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.518 7f5b28a1c700 1 -- 172.21.15.32:0/2866328864 wait complete. 2021-04-09T18:22:57.583 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:22:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd55311d700 1 Processor -- start 2021-04-09T18:22:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd55311d700 1 -- start start 2021-04-09T18:22:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd55311d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c10d010 0x7fd54c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:57.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd55311d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd54c0d1da0 0x7fd54c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd55311d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c10d480 con 0x7fd54c0d82c0 2021-04-09T18:22:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd55311d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c10e780 con 0x7fd54c0d1da0 2021-04-09T18:22:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd55311d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0d8d30 con 0x7fd54c10d010 2021-04-09T18:22:57.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd550f39700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c10d010 0x7fd54c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:57.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd54bfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd54c0d1da0 0x7fd54c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:57.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.685 7fd55173a700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd54c0d82c0 0x7fd54c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:57358/0 (socket says 172.21.15.32:57358) 2021-04-09T18:22:57.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd55173a700 1 -- 172.21.15.32:0/3606755352 learned_addr learned my addr 172.21.15.32:0/3606755352 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:57.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd54bfff700 1 -- 172.21.15.32:0/3606755352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c10d010 msgr2=0x7fd54c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:57.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd54bfff700 1 --2- 172.21.15.32:0/3606755352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c10d010 0x7fd54c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd54bfff700 1 -- 172.21.15.32:0/3606755352 >> v1:172.21.15.86:6789/0 conn(0x7fd54c0d82c0 legacy=0x7fd54c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:57.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd54bfff700 1 -- 172.21.15.32:0/3606755352 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0d9220 con 0x7fd54c0d1da0 2021-04-09T18:22:57.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd550f39700 1 --2- 172.21.15.32:0/3606755352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c10d010 0x7fd54c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:57.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd54b7fe700 1 -- 172.21.15.32:0/3606755352 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159378719 0 0) 0x7fd544001440 con 0x7fd54c0d82c0 2021-04-09T18:22:57.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd54bfff700 1 --2- 172.21.15.32:0/3606755352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c0d1da0 0x7fd54c0d7ca0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7fd540006650 tx=0x7fd540009a80).ready entity=mon.1 client_cookie=e3fce1b9ce9ee81d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd54b7fe700 1 -- 172.21.15.32:0/3606755352 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5400009b0 con 0x7fd54c0d1da0 2021-04-09T18:22:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd54b7fe700 1 -- 172.21.15.32:0/3606755352 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54000c0c0 con 0x7fd54c0d1da0 2021-04-09T18:22:57.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd54b7fe700 1 -- 172.21.15.32:0/3606755352 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd54000c510 con 0x7fd54c0d1da0 2021-04-09T18:22:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd55311d700 1 -- 172.21.15.32:0/3606755352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c0d1da0 msgr2=0x7fd54c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.686 7fd55311d700 1 --2- 172.21.15.32:0/3606755352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c0d1da0 0x7fd54c0d7ca0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7fd540006650 tx=0x7fd540009a80).stop 2021-04-09T18:22:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.687 7fd55311d700 1 -- 172.21.15.32:0/3606755352 shutdown_connections 2021-04-09T18:22:57.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.687 7fd55311d700 1 --2- 172.21.15.32:0/3606755352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c0d1da0 0x7fd54c0d7ca0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.687 7fd55311d700 1 --2- 172.21.15.32:0/3606755352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c10d010 0x7fd54c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.687 7fd55311d700 1 -- 172.21.15.32:0/3606755352 >> 172.21.15.32:0/3606755352 conn(0x7fd54c0ce5f0 msgr2=0x7fd54c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.687 7fd55311d700 1 -- 172.21.15.32:0/3606755352 shutdown_connections 2021-04-09T18:22:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.687 7fd55311d700 1 -- 172.21.15.32:0/3606755352 wait complete. 2021-04-09T18:22:57.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.707 7fd55311d700 1 Processor -- start 2021-04-09T18:22:57.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.707 7fd55311d700 1 -- start start 2021-04-09T18:22:57.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd55311d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c1a0050 0x7fd54c1a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:57.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd55311d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c0bdcd0 0x7fd54c0bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:57.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd55311d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c0bf080 0x7fd54c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:57.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd55311d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0bf500 con 0x7fd54c0bdcd0 2021-04-09T18:22:57.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd55311d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0bb7b0 con 0x7fd54c1a0050 2021-04-09T18:22:57.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd55311d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0be0d0 con 0x7fd54c0bf080 2021-04-09T18:22:57.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd54bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c0bdcd0 0x7fd54c0bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:57.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd54bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c0bdcd0 0x7fd54c0bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49848/0 (socket says 172.21.15.32:49848) 2021-04-09T18:22:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd54bfff700 1 -- 172.21.15.32:0/3874488262 learned_addr learned my addr 172.21.15.32:0/3874488262 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd550f39700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c1a0050 0x7fd54c1a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd55173a700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c0bf080 0x7fd54c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:57.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd550f39700 1 -- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c0bf080 msgr2=0x7fd54c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:57.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd550f39700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c0bf080 0x7fd54c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd550f39700 1 -- 172.21.15.32:0/3874488262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c0bdcd0 msgr2=0x7fd54c0bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:57.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd550f39700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c0bdcd0 0x7fd54c0bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:57.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd550f39700 1 -- 172.21.15.32:0/3874488262 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0b1310 con 0x7fd54c1a0050 2021-04-09T18:22:57.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd54bfff700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c0bdcd0 0x7fd54c0bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:22:57.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.708 7fd550f39700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c1a0050 0x7fd54c1a0450 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fd53c00b4e0 tx=0x7fd53c00b880).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:57.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.709 7fd5497fa700 1 -- 172.21.15.32:0/3874488262 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd53c00bc70 con 0x7fd54c1a0050 2021-04-09T18:22:57.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.709 7fd5497fa700 1 -- 172.21.15.32:0/3874488262 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c002030 con 0x7fd54c1a0050 2021-04-09T18:22:57.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.709 7fd55311d700 1 -- 172.21.15.32:0/3874488262 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c0b1310 con 0x7fd54c1a0050 2021-04-09T18:22:57.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.709 7fd5497fa700 1 -- 172.21.15.32:0/3874488262 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd53c002510 con 0x7fd54c1a0050 2021-04-09T18:22:57.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.709 7fd55311d700 1 -- 172.21.15.32:0/3874488262 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c0b1310 con 0x7fd54c1a0050 2021-04-09T18:22:57.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.709 7fd5497fa700 1 -- 172.21.15.32:0/3874488262 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd53c000c90 con 0x7fd54c1a0050 2021-04-09T18:22:57.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.709 7fd5497fa700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd52c0188d0 0x7fd52c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:57.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.709 7fd5497fa700 1 -- 172.21.15.32:0/3874488262 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd53c032970 con 0x7fd54c1a0050 2021-04-09T18:22:57.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.710 7fd54bfff700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd52c0188d0 0x7fd52c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:57.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.710 7fd54bfff700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd52c0188d0 0x7fd52c01add0 crc :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:57.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.729 7fd55311d700 1 -- 172.21.15.32:0/3874488262 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c0bb7b0 con 0x7fd54c1a0050 2021-04-09T18:22:57.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.733 7fd5497fa700 1 -- 172.21.15.32:0/3874488262 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd53c00bc70 con 0x7fd54c1a0050 2021-04-09T18:22:57.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:57.924 7fd55311d700 1 -- 172.21.15.32:0/3874488262 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fd54c0bb7b0 con 0x7fd54c1a0050 2021-04-09T18:22:58.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.476 7fd5497fa700 1 -- 172.21.15.32:0/3874488262 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v139) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd53c000c90 con 0x7fd54c1a0050 2021-04-09T18:22:58.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 -- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd52c0188d0 msgr2=0x7fd52c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd52c0188d0 0x7fd52c01add0 crc :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 -- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c1a0050 msgr2=0x7fd54c1a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c1a0050 0x7fd54c1a0450 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fd53c00b4e0 tx=0x7fd53c00b880).stop 2021-04-09T18:22:58.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 -- 172.21.15.32:0/3874488262 shutdown_connections 2021-04-09T18:22:58.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd52c0188d0 0x7fd52c01add0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd54c0bdcd0 0x7fd54c0bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd54c0bf080 0x7fd54c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 --2- 172.21.15.32:0/3874488262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd54c1a0050 0x7fd54c1a0450 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 -- 172.21.15.32:0/3874488262 >> 172.21.15.32:0/3874488262 conn(0x7fd54c0975f0 msgr2=0x7fd54c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 -- 172.21.15.32:0/3874488262 shutdown_connections 2021-04-09T18:22:58.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.491 7fd55311d700 1 -- 172.21.15.32:0/3874488262 wait complete. 2021-04-09T18:22:58.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:22:58.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febda51d700 1 Processor -- start 2021-04-09T18:22:58.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febda51d700 1 -- start start 2021-04-09T18:22:58.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febda51d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7febd40d7ca0 0x7febd40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:58.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febda51d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd410da00 0x7febd410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:58.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febda51d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd410d480 con 0x7febd410d010 2021-04-09T18:22:58.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febda51d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd40d9220 con 0x7febd40d7ca0 2021-04-09T18:22:58.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febda51d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd40d86c0 con 0x7febd410da00 2021-04-09T18:22:58.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febcbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7febd40d7ca0 0x7febd40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:58.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febcbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7febd40d7ca0 0x7febd40d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49858/0 (socket says 172.21.15.32:49858) 2021-04-09T18:22:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.655 7febcbfff700 1 -- 172.21.15.32:0/3741634418 learned_addr learned my addr 172.21.15.32:0/3741634418 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febd8b3a700 1 --2- 172.21.15.32:0/3741634418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd410da00 0x7febd410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febcbfff700 1 -- 172.21.15.32:0/3741634418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd410da00 msgr2=0x7febd410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febcbfff700 1 --2- 172.21.15.32:0/3741634418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd410da00 0x7febd410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febcbfff700 1 -- 172.21.15.32:0/3741634418 >> v1:172.21.15.86:6789/0 conn(0x7febd410d010 legacy=0x7febd40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febcbfff700 1 -- 172.21.15.32:0/3741634418 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd40d1f70 con 0x7febd40d7ca0 2021-04-09T18:22:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febd8b3a700 1 --2- 172.21.15.32:0/3741634418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd410da00 0x7febd410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febcbfff700 1 --2- 172.21.15.32:0/3741634418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40d7ca0 0x7febd40d80a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7febbc006700 tx=0x7febbc009b80).ready entity=mon.0 client_cookie=f11d65f23dae590f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febd37fe700 1 -- 172.21.15.32:0/3741634418 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febbc00c0c0 con 0x7febd40d7ca0 2021-04-09T18:22:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febd37fe700 1 -- 172.21.15.32:0/3741634418 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc00c440 con 0x7febd40d7ca0 2021-04-09T18:22:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febd37fe700 1 -- 172.21.15.32:0/3741634418 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febbc00c920 con 0x7febd40d7ca0 2021-04-09T18:22:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febda51d700 1 -- 172.21.15.32:0/3741634418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40d7ca0 msgr2=0x7febd40d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.656 7febda51d700 1 --2- 172.21.15.32:0/3741634418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40d7ca0 0x7febd40d80a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7febbc006700 tx=0x7febbc009b80).stop 2021-04-09T18:22:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.657 7febda51d700 1 -- 172.21.15.32:0/3741634418 shutdown_connections 2021-04-09T18:22:58.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.657 7febda51d700 1 --2- 172.21.15.32:0/3741634418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd410da00 0x7febd410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.657 7febda51d700 1 --2- 172.21.15.32:0/3741634418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40d7ca0 0x7febd40d80a0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.657 7febda51d700 1 -- 172.21.15.32:0/3741634418 >> 172.21.15.32:0/3741634418 conn(0x7febd40ce5f0 msgr2=0x7febd40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:58.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.657 7febda51d700 1 -- 172.21.15.32:0/3741634418 shutdown_connections 2021-04-09T18:22:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.657 7febda51d700 1 -- 172.21.15.32:0/3741634418 wait complete. 2021-04-09T18:22:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.677 7febda51d700 1 Processor -- start 2021-04-09T18:22:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.677 7febda51d700 1 -- start start 2021-04-09T18:22:58.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febda51d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febd41a0050 0x7febd41a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febda51d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40bdcd0 0x7febd40bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febda51d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd40bf080 0x7febd40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febda51d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd40bf500 con 0x7febd40bdcd0 2021-04-09T18:22:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febda51d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd40bb7b0 con 0x7febd41a0050 2021-04-09T18:22:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febda51d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd40be0d0 con 0x7febd40bf080 2021-04-09T18:22:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febcbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40bdcd0 0x7febd40bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febcbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40bdcd0 0x7febd40bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49862/0 (socket says 172.21.15.32:49862) 2021-04-09T18:22:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febcbfff700 1 -- 172.21.15.32:0/1941482114 learned_addr learned my addr 172.21.15.32:0/1941482114 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febd8b3a700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd40bf080 0x7febd40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febd3fff700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febd41a0050 0x7febd41a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febcbfff700 1 -- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd40bf080 msgr2=0x7febd40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febcbfff700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd40bf080 0x7febd40b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febcbfff700 1 -- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febd41a0050 msgr2=0x7febd41a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febcbfff700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febd41a0050 0x7febd41a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febcbfff700 1 -- 172.21.15.32:0/1941482114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd40b1310 con 0x7febd40bdcd0 2021-04-09T18:22:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febd3fff700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febd41a0050 0x7febd41a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febcbfff700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40bdcd0 0x7febd40bea60 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7febbc00b580 tx=0x7febbc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febd17fa700 1 -- 172.21.15.32:0/1941482114 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febbc00bd10 con 0x7febd40bdcd0 2021-04-09T18:22:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febd17fa700 1 -- 172.21.15.32:0/1941482114 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc000c90 con 0x7febd40bdcd0 2021-04-09T18:22:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.678 7febda51d700 1 -- 172.21.15.32:0/1941482114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd40b1310 con 0x7febd40bdcd0 2021-04-09T18:22:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.679 7febd17fa700 1 -- 172.21.15.32:0/1941482114 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febbc002220 con 0x7febd40bdcd0 2021-04-09T18:22:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.679 7febda51d700 1 -- 172.21.15.32:0/1941482114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd40b1310 con 0x7febd40bdcd0 2021-04-09T18:22:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.679 7febd17fa700 1 -- 172.21.15.32:0/1941482114 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7febbc009f90 con 0x7febd40bdcd0 2021-04-09T18:22:58.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.679 7febd17fa700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febb8018930 0x7febb801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:58.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.679 7febd17fa700 1 -- 172.21.15.32:0/1941482114 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7febbc032b60 con 0x7febd40bdcd0 2021-04-09T18:22:58.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.679 7febd3fff700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febb8018930 0x7febb801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:58.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.680 7febd3fff700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febb8018930 0x7febb801adc0 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:58.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.699 7febda51d700 1 -- 172.21.15.32:0/1941482114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd4093d50 con 0x7febd40bdcd0 2021-04-09T18:22:58.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.702 7febd17fa700 1 -- 172.21.15.32:0/1941482114 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7febbc0091e0 con 0x7febd40bdcd0 2021-04-09T18:22:58.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:58.896 7febda51d700 1 -- 172.21.15.32:0/1941482114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7febd410d480 con 0x7febd40bdcd0 2021-04-09T18:22:59.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.480 7febd17fa700 1 -- 172.21.15.32:0/1941482114 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v140) v1 ==== 105+0+0 (secure 0 0 0) 0x7febbc009f90 con 0x7febd40bdcd0 2021-04-09T18:22:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 -- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febb8018930 msgr2=0x7febb801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febb8018930 0x7febb801adc0 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 -- 172.21.15.32:0/1941482114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40bdcd0 msgr2=0x7febd40bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40bdcd0 0x7febd40bea60 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7febbc00b580 tx=0x7febbc00b920).stop 2021-04-09T18:22:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 -- 172.21.15.32:0/1941482114 shutdown_connections 2021-04-09T18:22:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7febb8018930 0x7febb801adc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febd40bdcd0 0x7febd40bea60 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febd40bf080 0x7febd40b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 --2- 172.21.15.32:0/1941482114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febd41a0050 0x7febd41a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 -- 172.21.15.32:0/1941482114 >> 172.21.15.32:0/1941482114 conn(0x7febd40975f0 msgr2=0x7febd4096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:59.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 -- 172.21.15.32:0/1941482114 shutdown_connections 2021-04-09T18:22:59.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.513 7febda51d700 1 -- 172.21.15.32:0/1941482114 wait complete. 2021-04-09T18:22:59.580 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:22:59.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.681 7f2938664700 1 Processor -- start 2021-04-09T18:22:59.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.681 7f2938664700 1 -- start start 2021-04-09T18:22:59.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f2938664700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f293010d010 0x7f29300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:59.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f2938664700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f29300d1da0 0x7f29300d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:59.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f2938664700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293010d480 con 0x7f29300d82c0 2021-04-09T18:22:59.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f2938664700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293010e780 con 0x7f29300d1da0 2021-04-09T18:22:59.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f2938664700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29300d8d30 con 0x7f293010d010 2021-04-09T18:22:59.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f2936c81700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f29300d82c0 0x7f293010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59460/0 (socket says 172.21.15.32:59460) 2021-04-09T18:22:59.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f2936c81700 1 -- 172.21.15.32:0/2559762942 learned_addr learned my addr 172.21.15.32:0/2559762942 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:22:59.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f2936480700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f293010d010 0x7f29300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:59.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f2935c7f700 1 --2- 172.21.15.32:0/2559762942 >> v2:172.21.15.86:3300/0 conn(0x7f29300d1da0 0x7f29300d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:59.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f293547e700 1 -- 172.21.15.32:0/2559762942 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029377228 0 0) 0x7f2928000d30 con 0x7f29300d82c0 2021-04-09T18:22:59.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.682 7f293547e700 1 -- 172.21.15.32:0/2559762942 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f291c002fe0 con 0x7f29300d82c0 2021-04-09T18:22:59.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1524327653 0 0) 0x7f2928001460 con 0x7f29300d82c0 2021-04-09T18:22:59.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f293010d010 msgr2=0x7f29300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:59.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f293010d010 0x7f29300d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 >> v2:172.21.15.86:3300/0 conn(0x7f29300d1da0 msgr2=0x7f29300d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:59.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 --2- 172.21.15.32:0/2559762942 >> v2:172.21.15.86:3300/0 conn(0x7f29300d1da0 0x7f29300d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300d9220 con 0x7f29300d82c0 2021-04-09T18:22:59.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f2928001980 con 0x7f29300d82c0 2021-04-09T18:22:59.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 >> v1:172.21.15.32:6789/0 conn(0x7f29300d82c0 legacy=0x7f293010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:59.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f291c003a60 0x7f291c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:59.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f291c04a7a0 0x7f291c04cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:59.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f291c04d510 0x7f291c04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:59.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c0502e0 con 0x7f291c003a60 2021-04-09T18:22:59.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c050500 con 0x7f291c04d510 2021-04-09T18:22:59.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c050720 con 0x7f291c04a7a0 2021-04-09T18:22:59.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936480700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f291c003a60 0x7f291c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:59.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2935c7f700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f291c04a7a0 0x7f291c04cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:59.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936c81700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f291c04d510 0x7f291c04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:59.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936480700 1 -- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f291c04a7a0 msgr2=0x7f291c04cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:59.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936480700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f291c04a7a0 0x7f291c04cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936480700 1 -- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f291c04d510 msgr2=0x7f291c04fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:22:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936480700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f291c04d510 0x7f291c04fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936480700 1 -- 172.21.15.32:0/2559762942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c0509c0 con 0x7f291c003a60 2021-04-09T18:22:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2935c7f700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f291c04a7a0 0x7f291c04cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936c81700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f291c04d510 0x7f291c04fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:22:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936480700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f291c003a60 0x7f291c005ef0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f292c006ac0 tx=0x7f292c009ab0).ready entity=mon.0 client_cookie=33e981c1fee680a0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:59.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f292c00c0c0 con 0x7f291c003a60 2021-04-09T18:22:59.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292c00c3c0 con 0x7f291c003a60 2021-04-09T18:22:59.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f293547e700 1 -- 172.21.15.32:0/2559762942 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f292c00c8a0 con 0x7f291c003a60 2021-04-09T18:22:59.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2938664700 1 -- 172.21.15.32:0/2559762942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f291c003a60 msgr2=0x7f291c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:59.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2938664700 1 --2- 172.21.15.32:0/2559762942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f291c003a60 0x7f291c005ef0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f292c006ac0 tx=0x7f292c009ab0).stop 2021-04-09T18:22:59.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.683 7f2936c81700 1 -- 172.21.15.32:0/2559762942 reap_dead start 2021-04-09T18:22:59.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.684 7f2938664700 1 -- 172.21.15.32:0/2559762942 shutdown_connections 2021-04-09T18:22:59.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.684 7f2938664700 1 -- 172.21.15.32:0/2559762942 >> 172.21.15.32:0/2559762942 conn(0x7f29300ce5f0 msgr2=0x7f29300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:22:59.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.684 7f2938664700 1 -- 172.21.15.32:0/2559762942 shutdown_connections 2021-04-09T18:22:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.684 7f2938664700 1 -- 172.21.15.32:0/2559762942 wait complete. 2021-04-09T18:22:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2938664700 1 Processor -- start 2021-04-09T18:22:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2938664700 1 -- start start 2021-04-09T18:22:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2938664700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300c3c00 0x7f293009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2938664700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930095240 0x7f29301a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:59.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2938664700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29301a1270 0x7f29300b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:59.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2938664700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293009cda0 con 0x7f29301a1270 2021-04-09T18:22:59.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2938664700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29300c3680 con 0x7f29300c3c00 2021-04-09T18:22:59.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2938664700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2930095640 con 0x7f2930095240 2021-04-09T18:22:59.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2936c81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29301a1270 0x7f29300b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:59.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2936c81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29301a1270 0x7f29300b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49884/0 (socket says 172.21.15.32:49884) 2021-04-09T18:22:59.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2936c81700 1 -- 172.21.15.32:0/4245066072 learned_addr learned my addr 172.21.15.32:0/4245066072 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:22:59.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2935c7f700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930095240 0x7f29301a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:59.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.705 7f2936480700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300c3c00 0x7f293009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2936c81700 1 -- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930095240 msgr2=0x7f29301a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2936c81700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930095240 0x7f29301a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2936c81700 1 -- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300c3c00 msgr2=0x7f293009c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:22:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2936c81700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300c3c00 0x7f293009c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:22:59.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2936c81700 1 -- 172.21.15.32:0/4245066072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29301a1670 con 0x7f29301a1270 2021-04-09T18:22:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2936c81700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29301a1270 0x7f29300b0cf0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f29280069f0 tx=0x7f292800bc10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2926ffd700 1 -- 172.21.15.32:0/4245066072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2928006030 con 0x7f29301a1270 2021-04-09T18:22:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2926ffd700 1 -- 172.21.15.32:0/4245066072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2928007d80 con 0x7f29301a1270 2021-04-09T18:22:59.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2938664700 1 -- 172.21.15.32:0/4245066072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29301a1670 con 0x7f29301a1270 2021-04-09T18:22:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2926ffd700 1 -- 172.21.15.32:0/4245066072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2928004720 con 0x7f29301a1270 2021-04-09T18:22:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.706 7f2938664700 1 -- 172.21.15.32:0/4245066072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29301a1670 con 0x7f29301a1270 2021-04-09T18:22:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.707 7f2926ffd700 1 -- 172.21.15.32:0/4245066072 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2928007800 con 0x7f29301a1270 2021-04-09T18:22:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.707 7f2926ffd700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2918018930 0x7f291801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:22:59.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.707 7f2926ffd700 1 -- 172.21.15.32:0/4245066072 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2928032d10 con 0x7f29301a1270 2021-04-09T18:22:59.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.707 7f2936480700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2918018930 0x7f291801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:22:59.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.707 7f2936480700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2918018930 0x7f291801ae10 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:22:59.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.726 7f2938664700 1 -- 172.21.15.32:0/4245066072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2930093d50 con 0x7f29301a1270 2021-04-09T18:22:59.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.729 7f2926ffd700 1 -- 172.21.15.32:0/4245066072 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2928007000 con 0x7f29301a1270 2021-04-09T18:22:59.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:22:59.922 7f2938664700 1 -- 172.21.15.32:0/4245066072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f293010d480 con 0x7f29301a1270 2021-04-09T18:23:00.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.485 7f2926ffd700 1 -- 172.21.15.32:0/4245066072 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v141) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2928007800 con 0x7f29301a1270 2021-04-09T18:23:00.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 -- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2918018930 msgr2=0x7f291801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:00.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2918018930 0x7f291801ae10 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 -- 172.21.15.32:0/4245066072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29301a1270 msgr2=0x7f29300b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:00.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29301a1270 0x7f29300b0cf0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f29280069f0 tx=0x7f292800bc10).stop 2021-04-09T18:23:00.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 -- 172.21.15.32:0/4245066072 shutdown_connections 2021-04-09T18:23:00.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2918018930 0x7f291801ae10 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29301a1270 0x7f29300b0cf0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930095240 0x7f29301a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 --2- 172.21.15.32:0/4245066072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300c3c00 0x7f293009c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.490 7f2938664700 1 -- 172.21.15.32:0/4245066072 >> 172.21.15.32:0/4245066072 conn(0x7f29300975f0 msgr2=0x7f2930096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.491 7f2938664700 1 -- 172.21.15.32:0/4245066072 shutdown_connections 2021-04-09T18:23:00.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.491 7f2938664700 1 -- 172.21.15.32:0/4245066072 wait complete. 2021-04-09T18:23:00.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:00.601 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:23:00.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.661 7f7dc3c8b700 1 Processor -- start 2021-04-09T18:23:00.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.661 7f7dc3c8b700 1 -- start start 2021-04-09T18:23:00.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.662 7f7dc3c8b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7dbc10d010 0x7f7dbc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:00.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.662 7f7dc3c8b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0d1da0 0x7f7dbc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:00.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.662 7f7dc3c8b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc10d480 con 0x7f7dbc0d82c0 2021-04-09T18:23:00.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.662 7f7dc3c8b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc0d9220 con 0x7f7dbc10d010 2021-04-09T18:23:00.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.662 7f7dc3c8b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0d86c0 con 0x7f7dbc0d1da0 2021-04-09T18:23:00.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.662 7f7dc1aa7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7dbc10d010 0x7f7dbc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:00.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.662 7f7dc1aa7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7dbc10d010 0x7f7dbc0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49894/0 (socket says 172.21.15.32:49894) 2021-04-09T18:23:00.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.662 7f7dc1aa7700 1 -- 172.21.15.32:0/1728990467 learned_addr learned my addr 172.21.15.32:0/1728990467 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:00.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.662 7f7dc12a6700 1 --2- 172.21.15.32:0/1728990467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0d1da0 0x7f7dbc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:00.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc1aa7700 1 -- 172.21.15.32:0/1728990467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0d1da0 msgr2=0x7f7dbc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:00.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc1aa7700 1 --2- 172.21.15.32:0/1728990467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0d1da0 0x7f7dbc0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc1aa7700 1 -- 172.21.15.32:0/1728990467 >> v1:172.21.15.86:6789/0 conn(0x7f7dbc0d82c0 legacy=0x7f7dbc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:00.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc1aa7700 1 -- 172.21.15.32:0/1728990467 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc10e1f0 con 0x7f7dbc10d010 2021-04-09T18:23:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc1aa7700 1 --2- 172.21.15.32:0/1728990467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc10d010 0x7f7dbc0d1780 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f7db8006700 tx=0x7f7db8009b00).ready entity=mon.0 client_cookie=be5019d20060bd1b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc0aa5700 1 -- 172.21.15.32:0/1728990467 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7db800c0c0 con 0x7f7dbc10d010 2021-04-09T18:23:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc0aa5700 1 -- 172.21.15.32:0/1728990467 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db800c400 con 0x7f7dbc10d010 2021-04-09T18:23:00.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc0aa5700 1 -- 172.21.15.32:0/1728990467 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7db800c8e0 con 0x7f7dbc10d010 2021-04-09T18:23:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc3c8b700 1 -- 172.21.15.32:0/1728990467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc10d010 msgr2=0x7f7dbc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.663 7f7dc3c8b700 1 --2- 172.21.15.32:0/1728990467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc10d010 0x7f7dbc0d1780 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f7db8006700 tx=0x7f7db8009b00).stop 2021-04-09T18:23:00.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.664 7f7dc3c8b700 1 -- 172.21.15.32:0/1728990467 shutdown_connections 2021-04-09T18:23:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.664 7f7dc3c8b700 1 --2- 172.21.15.32:0/1728990467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0d1da0 0x7f7dbc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.664 7f7dc3c8b700 1 --2- 172.21.15.32:0/1728990467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc10d010 0x7f7dbc0d1780 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.664 7f7dc3c8b700 1 -- 172.21.15.32:0/1728990467 >> 172.21.15.32:0/1728990467 conn(0x7f7dbc0ce5f0 msgr2=0x7f7dbc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:00.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.664 7f7dc3c8b700 1 -- 172.21.15.32:0/1728990467 shutdown_connections 2021-04-09T18:23:00.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.664 7f7dc3c8b700 1 -- 172.21.15.32:0/1728990467 wait complete. 2021-04-09T18:23:00.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.684 7f7dc3c8b700 1 Processor -- start 2021-04-09T18:23:00.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.684 7f7dc3c8b700 1 -- start start 2021-04-09T18:23:00.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc3c8b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0b2140 0x7f7dbc09d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:00.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc3c8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc09e010 0x7f7dbc194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc3c8b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dbc1950a0 0x7f7dbc0a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc3c8b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc09d470 con 0x7f7dbc09e010 2021-04-09T18:23:00.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc3c8b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc09e410 con 0x7f7dbc1950a0 2021-04-09T18:23:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc3c8b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc1954a0 con 0x7f7dbc0b2140 2021-04-09T18:23:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc12a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc09e010 0x7f7dbc194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc12a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc09e010 0x7f7dbc194a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49902/0 (socket says 172.21.15.32:49902) 2021-04-09T18:23:00.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc12a6700 1 -- 172.21.15.32:0/2815783684 learned_addr learned my addr 172.21.15.32:0/2815783684 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc1aa7700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0b2140 0x7f7dbc09d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc22a8700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dbc1950a0 0x7f7dbc0a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:00.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc12a6700 1 -- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0b2140 msgr2=0x7f7dbc09d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:00.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc12a6700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0b2140 0x7f7dbc09d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc12a6700 1 -- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dbc1950a0 msgr2=0x7f7dbc0a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:00.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc12a6700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dbc1950a0 0x7f7dbc0a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:00.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc12a6700 1 -- 172.21.15.32:0/2815783684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc1956c0 con 0x7f7dbc09e010 2021-04-09T18:23:00.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc1aa7700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0b2140 0x7f7dbc09d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:00.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc22a8700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dbc1950a0 0x7f7dbc0a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:00.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc12a6700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc09e010 0x7f7dbc194a80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7f7dac0067e0 tx=0x7f7dac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:00.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7db27fc700 1 -- 172.21.15.32:0/2815783684 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7dac006900 con 0x7f7dbc09e010 2021-04-09T18:23:00.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7db27fc700 1 -- 172.21.15.32:0/2815783684 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac00bd80 con 0x7f7dbc09e010 2021-04-09T18:23:00.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc1956c0 con 0x7f7dbc09e010 2021-04-09T18:23:00.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.685 7f7db27fc700 1 -- 172.21.15.32:0/2815783684 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7dac002030 con 0x7f7dbc09e010 2021-04-09T18:23:00.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.686 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc1956c0 con 0x7f7dbc09e010 2021-04-09T18:23:00.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.686 7f7db27fc700 1 -- 172.21.15.32:0/2815783684 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7dac0068b0 con 0x7f7dbc09e010 2021-04-09T18:23:00.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.686 7f7db27fc700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7da4018930 0x7f7da401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:00.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.686 7f7db27fc700 1 -- 172.21.15.32:0/2815783684 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7dac032a10 con 0x7f7dbc09e010 2021-04-09T18:23:00.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.686 7f7dc1aa7700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7da4018930 0x7f7da401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:00.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.687 7f7dc1aa7700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7da4018930 0x7f7da401ae10 crc :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:00.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.706 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dbc1b7040 con 0x7f7dbc09e010 2021-04-09T18:23:00.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.710 7f7db27fc700 1 -- 172.21.15.32:0/2815783684 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7dac000940 con 0x7f7dbc09e010 2021-04-09T18:23:00.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:00.905 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f7dbc1b7040 con 0x7f7dbc09e010 2021-04-09T18:23:01.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.494 7f7db27fc700 1 -- 172.21.15.32:0/2815783684 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v143) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7dac0068b0 con 0x7f7dbc09e010 2021-04-09T18:23:01.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7da4018930 msgr2=0x7f7da401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:01.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7da4018930 0x7f7da401ae10 crc :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc09e010 msgr2=0x7f7dbc194a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:01.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc09e010 0x7f7dbc194a80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7f7dac0067e0 tx=0x7f7dac00b990).stop 2021-04-09T18:23:01.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 shutdown_connections 2021-04-09T18:23:01.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7da4018930 0x7f7da401ae10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7dbc09e010 0x7f7dbc194a80 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dbc1950a0 0x7f7dbc0a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 --2- 172.21.15.32:0/2815783684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dbc0b2140 0x7f7dbc09d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 >> 172.21.15.32:0/2815783684 conn(0x7f7dbc0b27e0 msgr2=0x7f7dbc1b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:01.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 shutdown_connections 2021-04-09T18:23:01.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.522 7f7dc3c8b700 1 -- 172.21.15.32:0/2815783684 wait complete. 2021-04-09T18:23:01.588 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.684 7f27135ed700 1 Processor -- start 2021-04-09T18:23:01.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.684 7f27135ed700 1 -- start start 2021-04-09T18:23:01.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.685 7f27135ed700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f270c10d010 0x7f270c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.685 7f27135ed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c0d1da0 0x7f270c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.685 7f27135ed700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c10d480 con 0x7f270c0d82c0 2021-04-09T18:23:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.685 7f27135ed700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f270c0d9220 con 0x7f270c10d010 2021-04-09T18:23:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.685 7f27135ed700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c0d86c0 con 0x7f270c0d1da0 2021-04-09T18:23:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.685 7f2711c0a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f270c0d82c0 0x7f270c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59506/0 (socket says 172.21.15.32:59506) 2021-04-09T18:23:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.685 7f2711c0a700 1 -- 172.21.15.32:0/2340371707 learned_addr learned my addr 172.21.15.32:0/2340371707 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:01.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.685 7f2710c08700 1 --2- 172.21.15.32:0/2340371707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c0d1da0 0x7f270c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:01.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.685 7f2711409700 1 --2- 172.21.15.32:0/2340371707 >> v2:172.21.15.86:3300/0 conn(0x7f270c10d010 0x7f270c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:01.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2703fff700 1 -- 172.21.15.32:0/2340371707 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627019922 0 0) 0x7f2704000d30 con 0x7f270c0d82c0 2021-04-09T18:23:01.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2703fff700 1 -- 172.21.15.32:0/2340371707 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f8002fe0 con 0x7f270c0d82c0 2021-04-09T18:23:01.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2710c08700 1 -- 172.21.15.32:0/2340371707 >> v2:172.21.15.86:3300/0 conn(0x7f270c10d010 msgr2=0x7f270c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:01.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2710c08700 1 --2- 172.21.15.32:0/2340371707 >> v2:172.21.15.86:3300/0 conn(0x7f270c10d010 0x7f270c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2710c08700 1 -- 172.21.15.32:0/2340371707 >> v1:172.21.15.32:6789/0 conn(0x7f270c0d82c0 legacy=0x7f270c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2710c08700 1 -- 172.21.15.32:0/2340371707 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c10e1f0 con 0x7f270c0d1da0 2021-04-09T18:23:01.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2710c08700 1 --2- 172.21.15.32:0/2340371707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c0d1da0 0x7f270c0d7ca0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f26fc006650 tx=0x7f26fc009a80).ready entity=mon.2 client_cookie=b2583e61b570b0cf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2703fff700 1 -- 172.21.15.32:0/2340371707 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26fc0009b0 con 0x7f270c0d1da0 2021-04-09T18:23:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2703fff700 1 -- 172.21.15.32:0/2340371707 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc00c0c0 con 0x7f270c0d1da0 2021-04-09T18:23:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f2703fff700 1 -- 172.21.15.32:0/2340371707 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26fc00c510 con 0x7f270c0d1da0 2021-04-09T18:23:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f27135ed700 1 -- 172.21.15.32:0/2340371707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c0d1da0 msgr2=0x7f270c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:01.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.686 7f27135ed700 1 --2- 172.21.15.32:0/2340371707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c0d1da0 0x7f270c0d7ca0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f26fc006650 tx=0x7f26fc009a80).stop 2021-04-09T18:23:01.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.687 7f27135ed700 1 -- 172.21.15.32:0/2340371707 shutdown_connections 2021-04-09T18:23:01.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.687 7f27135ed700 1 --2- 172.21.15.32:0/2340371707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c0d1da0 0x7f270c0d7ca0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.687 7f27135ed700 1 --2- 172.21.15.32:0/2340371707 >> v2:172.21.15.86:3300/0 conn(0x7f270c10d010 0x7f270c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.687 7f27135ed700 1 -- 172.21.15.32:0/2340371707 >> 172.21.15.32:0/2340371707 conn(0x7f270c0ce5f0 msgr2=0x7f270c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:01.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.687 7f27135ed700 1 -- 172.21.15.32:0/2340371707 shutdown_connections 2021-04-09T18:23:01.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.687 7f27135ed700 1 -- 172.21.15.32:0/2340371707 wait complete. 2021-04-09T18:23:01.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.707 7f27135ed700 1 Processor -- start 2021-04-09T18:23:01.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.707 7f27135ed700 1 -- start start 2021-04-09T18:23:01.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f27135ed700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f270c0a0a10 0x7f270c0bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f27135ed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f270c18ad00 0x7f270c19fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f27135ed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c1a03f0 0x7f270c095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f27135ed700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c18b100 con 0x7f270c18ad00 2021-04-09T18:23:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f27135ed700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c095860 con 0x7f270c0a0a10 2021-04-09T18:23:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f27135ed700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c095a80 con 0x7f270c1a03f0 2021-04-09T18:23:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2710c08700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f270c18ad00 0x7f270c19fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2710c08700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f270c18ad00 0x7f270c19fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49922/0 (socket says 172.21.15.32:49922) 2021-04-09T18:23:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2710c08700 1 -- 172.21.15.32:0/456693015 learned_addr learned my addr 172.21.15.32:0/456693015 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:01.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2711c0a700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c1a03f0 0x7f270c095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:01.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2711409700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f270c0a0a10 0x7f270c0bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:01.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2710c08700 1 -- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c1a03f0 msgr2=0x7f270c095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:01.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2710c08700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c1a03f0 0x7f270c095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2710c08700 1 -- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f270c0a0a10 msgr2=0x7f270c0bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:01.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2710c08700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f270c0a0a10 0x7f270c0bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:01.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2710c08700 1 -- 172.21.15.32:0/456693015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c189250 con 0x7f270c18ad00 2021-04-09T18:23:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.708 7f2710c08700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f270c18ad00 0x7f270c19fdd0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f26fc009650 tx=0x7f26fc00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.709 7f2701ffb700 1 -- 172.21.15.32:0/456693015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26fc0093a0 con 0x7f270c18ad00 2021-04-09T18:23:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.709 7f2701ffb700 1 -- 172.21.15.32:0/456693015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc006070 con 0x7f270c18ad00 2021-04-09T18:23:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.709 7f2701ffb700 1 -- 172.21.15.32:0/456693015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26fc0052e0 con 0x7f270c18ad00 2021-04-09T18:23:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.709 7f27135ed700 1 -- 172.21.15.32:0/456693015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270c189250 con 0x7f270c18ad00 2021-04-09T18:23:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.709 7f27135ed700 1 -- 172.21.15.32:0/456693015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c189250 con 0x7f270c18ad00 2021-04-09T18:23:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.709 7f2701ffb700 1 -- 172.21.15.32:0/456693015 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f26fc009300 con 0x7f270c18ad00 2021-04-09T18:23:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.709 7f2701ffb700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f26ec018930 0x7f26ec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.709 7f2701ffb700 1 -- 172.21.15.32:0/456693015 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f26fc032a00 con 0x7f270c18ad00 2021-04-09T18:23:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.710 7f2711409700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f26ec018930 0x7f26ec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.710 7f2711409700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f26ec018930 0x7f26ec01ae10 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.729 7f27135ed700 1 -- 172.21.15.32:0/456693015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270c093d50 con 0x7f270c18ad00 2021-04-09T18:23:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.733 7f2701ffb700 1 -- 172.21.15.32:0/456693015 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f26fc006030 con 0x7f270c18ad00 2021-04-09T18:23:01.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:01.925 7f27135ed700 1 -- 172.21.15.32:0/456693015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f270c10d480 con 0x7f270c18ad00 2021-04-09T18:23:02.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.497 7f2701ffb700 1 -- 172.21.15.32:0/456693015 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v144) v1 ==== 105+0+0 (secure 0 0 0) 0x7f26fc009300 con 0x7f270c18ad00 2021-04-09T18:23:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.542 7f27135ed700 1 -- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f26ec018930 msgr2=0x7f26ec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.542 7f27135ed700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f26ec018930 0x7f26ec01ae10 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.542 7f27135ed700 1 -- 172.21.15.32:0/456693015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f270c18ad00 msgr2=0x7f270c19fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.542 7f27135ed700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f270c18ad00 0x7f270c19fdd0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f26fc009650 tx=0x7f26fc00b8e0).stop 2021-04-09T18:23:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.543 7f27135ed700 1 -- 172.21.15.32:0/456693015 shutdown_connections 2021-04-09T18:23:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.543 7f27135ed700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f26ec018930 0x7f26ec01ae10 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.543 7f27135ed700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f270c18ad00 0x7f270c19fdd0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.543 7f27135ed700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f270c1a03f0 0x7f270c095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.543 7f27135ed700 1 --2- 172.21.15.32:0/456693015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f270c0a0a10 0x7f270c0bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.543 7f27135ed700 1 -- 172.21.15.32:0/456693015 >> 172.21.15.32:0/456693015 conn(0x7f270c0975f0 msgr2=0x7f270c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.543 7f27135ed700 1 -- 172.21.15.32:0/456693015 shutdown_connections 2021-04-09T18:23:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.543 7f27135ed700 1 -- 172.21.15.32:0/456693015 wait complete. 2021-04-09T18:23:02.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.705 7fccfe785700 1 Processor -- start 2021-04-09T18:23:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.705 7fccfe785700 1 -- start start 2021-04-09T18:23:02.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.706 7fccfe785700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fccf810da00 0x7fccf810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.706 7fccfe785700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf810d480 con 0x7fccf80d7ca0 2021-04-09T18:23:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.706 7fccfe785700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf80d1f70 con 0x7fccf810d010 2021-04-09T18:23:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.706 7fccfe785700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf80d8d30 con 0x7fccf810da00 2021-04-09T18:23:02.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.706 7fccfcda2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fccf810da00 0x7fccf810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.706 7fcceffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fccf80d7ca0 0x7fccf80d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59520/0 (socket says 172.21.15.32:59520) 2021-04-09T18:23:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.706 7fcceffff700 1 -- 172.21.15.32:0/1329010612 learned_addr learned my addr 172.21.15.32:0/1329010612 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccfcda2700 1 -- 172.21.15.32:0/1329010612 >> v1:172.21.15.86:6789/0 conn(0x7fccf810d010 legacy=0x7fccf80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:02.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccfcda2700 1 -- 172.21.15.32:0/1329010612 >> v1:172.21.15.32:6789/0 conn(0x7fccf80d7ca0 legacy=0x7fccf80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccfcda2700 1 -- 172.21.15.32:0/1329010612 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf80d9220 con 0x7fccf810da00 2021-04-09T18:23:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccf77fe700 1 -- 172.21.15.32:0/1329010612 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282190412 0 0) 0x7fcce0000ce0 con 0x7fccf80d7ca0 2021-04-09T18:23:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccfcda2700 1 --2- 172.21.15.32:0/1329010612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf810da00 0x7fccf810de00 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7fccf0006e30 tx=0x7fccf0009de0).ready entity=mon.0 client_cookie=72b00dfa3e0cf7cc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:02.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccf77fe700 1 -- 172.21.15.32:0/1329010612 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccf000b300 con 0x7fccf810da00 2021-04-09T18:23:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccf77fe700 1 -- 172.21.15.32:0/1329010612 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf000b680 con 0x7fccf810da00 2021-04-09T18:23:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccf77fe700 1 -- 172.21.15.32:0/1329010612 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccf000bb60 con 0x7fccf810da00 2021-04-09T18:23:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccfe785700 1 -- 172.21.15.32:0/1329010612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf810da00 msgr2=0x7fccf810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.707 7fccfe785700 1 --2- 172.21.15.32:0/1329010612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf810da00 0x7fccf810de00 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7fccf0006e30 tx=0x7fccf0009de0).stop 2021-04-09T18:23:02.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.708 7fccfe785700 1 -- 172.21.15.32:0/1329010612 shutdown_connections 2021-04-09T18:23:02.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.708 7fccfe785700 1 --2- 172.21.15.32:0/1329010612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf810da00 0x7fccf810de00 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:02.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.708 7fccfe785700 1 -- 172.21.15.32:0/1329010612 >> 172.21.15.32:0/1329010612 conn(0x7fccf80ce5f0 msgr2=0x7fccf80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:02.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.708 7fccfe785700 1 -- 172.21.15.32:0/1329010612 shutdown_connections 2021-04-09T18:23:02.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.708 7fccfe785700 1 -- 172.21.15.32:0/1329010612 wait complete. 2021-04-09T18:23:02.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.728 7fccfe785700 1 Processor -- start 2021-04-09T18:23:02.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.728 7fccfe785700 1 -- start start 2021-04-09T18:23:02.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfe785700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf8097ff0 0x7fccf80ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:02.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfe785700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf80acf30 0x7fccf8090c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:02.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfe785700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf80912b0 0x7fccf808e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:02.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfe785700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf80a92f0 con 0x7fccf80912b0 2021-04-09T18:23:02.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfe785700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf80983f0 con 0x7fccf80acf30 2021-04-09T18:23:02.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfe785700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf80916b0 con 0x7fccf8097ff0 2021-04-09T18:23:02.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfcda2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf80912b0 0x7fccf808e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:02.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfcda2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf80912b0 0x7fccf808e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49938/0 (socket says 172.21.15.32:49938) 2021-04-09T18:23:02.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfcda2700 1 -- 172.21.15.32:0/364358509 learned_addr learned my addr 172.21.15.32:0/364358509 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:02.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fcceffff700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf80acf30 0x7fccf8090c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:02.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccf7fff700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf8097ff0 0x7fccf80ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:02.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfcda2700 1 -- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf8097ff0 msgr2=0x7fccf80ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:02.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfcda2700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf8097ff0 0x7fccf80ac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:02.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfcda2700 1 -- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf80acf30 msgr2=0x7fccf8090c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:02.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfcda2700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf80acf30 0x7fccf8090c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:02.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfcda2700 1 -- 172.21.15.32:0/364358509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf80aec50 con 0x7fccf80912b0 2021-04-09T18:23:02.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfcda2700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf80912b0 0x7fccf808e9a0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7fccf0009fc0 tx=0x7fccf000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:02.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccf57fa700 1 -- 172.21.15.32:0/364358509 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccf00093d0 con 0x7fccf80912b0 2021-04-09T18:23:02.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccf57fa700 1 -- 172.21.15.32:0/364358509 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf0009690 con 0x7fccf80912b0 2021-04-09T18:23:02.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfe785700 1 -- 172.21.15.32:0/364358509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf80aec50 con 0x7fccf80912b0 2021-04-09T18:23:02.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccf57fa700 1 -- 172.21.15.32:0/364358509 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccf0005a10 con 0x7fccf80912b0 2021-04-09T18:23:02.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.729 7fccfe785700 1 -- 172.21.15.32:0/364358509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf80aec50 con 0x7fccf80912b0 2021-04-09T18:23:02.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.730 7fccf57fa700 1 -- 172.21.15.32:0/364358509 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fccf0005820 con 0x7fccf80912b0 2021-04-09T18:23:02.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.730 7fccf57fa700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccdc018900 0x7fccdc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:02.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.730 7fccf57fa700 1 -- 172.21.15.32:0/364358509 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fccf0032da0 con 0x7fccf80912b0 2021-04-09T18:23:02.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.730 7fccf7fff700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccdc018900 0x7fccdc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:02.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.730 7fccf7fff700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccdc018900 0x7fccdc01adc0 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:02.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.750 7fccfe785700 1 -- 172.21.15.32:0/364358509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf808f890 con 0x7fccf80912b0 2021-04-09T18:23:02.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.753 7fccf57fa700 1 -- 172.21.15.32:0/364358509 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fccf0006bd0 con 0x7fccf80912b0 2021-04-09T18:23:03.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:02.947 7fccfe785700 1 -- 172.21.15.32:0/364358509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fccf808f890 con 0x7fccf80912b0 2021-04-09T18:23:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.501 7fccf57fa700 1 -- 172.21.15.32:0/364358509 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v146) v1 ==== 105+0+0 (secure 0 0 0) 0x7fccf0005820 con 0x7fccf80912b0 2021-04-09T18:23:03.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 -- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccdc018900 msgr2=0x7fccdc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccdc018900 0x7fccdc01adc0 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:03.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 -- 172.21.15.32:0/364358509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf80912b0 msgr2=0x7fccf808e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:03.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf80912b0 0x7fccf808e9a0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7fccf0009fc0 tx=0x7fccf000bd40).stop 2021-04-09T18:23:03.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 -- 172.21.15.32:0/364358509 shutdown_connections 2021-04-09T18:23:03.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fccdc018900 0x7fccdc01adc0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:03.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccf80912b0 0x7fccf808e9a0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:03.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccf80acf30 0x7fccf8090c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:03.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 --2- 172.21.15.32:0/364358509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccf8097ff0 0x7fccf80ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:03.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 -- 172.21.15.32:0/364358509 >> 172.21.15.32:0/364358509 conn(0x7fccf80bb410 msgr2=0x7fccf808dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:03.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 -- 172.21.15.32:0/364358509 shutdown_connections 2021-04-09T18:23:03.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.515 7fccfe785700 1 -- 172.21.15.32:0/364358509 wait complete. 2021-04-09T18:23:03.584 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:03.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.686 7f4c2dac6700 1 Processor -- start 2021-04-09T18:23:03.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.686 7f4c2dac6700 1 -- start start 2021-04-09T18:23:03.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.687 7f4c2dac6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c280d7ca0 0x7f4c280d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:03.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.687 7f4c2dac6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2810d480 con 0x7f4c2810da00 2021-04-09T18:23:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.687 7f4c2dac6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c280d86c0 con 0x7f4c2810d010 2021-04-09T18:23:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.687 7f4c2dac6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c280d9220 con 0x7f4c280d7ca0 2021-04-09T18:23:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.687 7f4c26ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c280d7ca0 0x7f4c280d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:03.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.687 7f4c27fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4c2810da00 0x7f4c2810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59534/0 (socket says 172.21.15.32:59534) 2021-04-09T18:23:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.687 7f4c27fff700 1 -- 172.21.15.32:0/1412530654 learned_addr learned my addr 172.21.15.32:0/1412530654 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c26ffd700 1 -- 172.21.15.32:0/1412530654 >> v1:172.21.15.86:6789/0 conn(0x7f4c2810d010 legacy=0x7f4c280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c26ffd700 1 -- 172.21.15.32:0/1412530654 >> v1:172.21.15.32:6789/0 conn(0x7f4c2810da00 legacy=0x7f4c2810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c26ffd700 1 -- 172.21.15.32:0/1412530654 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c28112100 con 0x7f4c280d7ca0 2021-04-09T18:23:03.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c267fc700 1 -- 172.21.15.32:0/1412530654 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063968341 0 0) 0x7f4c20000d30 con 0x7f4c2810da00 2021-04-09T18:23:03.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c26ffd700 1 --2- 172.21.15.32:0/1412530654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280d7ca0 0x7f4c280d80a0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f4c14006650 tx=0x7f4c14009a00).ready entity=mon.0 client_cookie=37da93a07eba8a7c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:03.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c267fc700 1 -- 172.21.15.32:0/1412530654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c14009df0 con 0x7f4c280d7ca0 2021-04-09T18:23:03.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c267fc700 1 -- 172.21.15.32:0/1412530654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c140009b0 con 0x7f4c280d7ca0 2021-04-09T18:23:03.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c267fc700 1 -- 172.21.15.32:0/1412530654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c1400c2b0 con 0x7f4c280d7ca0 2021-04-09T18:23:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c2dac6700 1 -- 172.21.15.32:0/1412530654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280d7ca0 msgr2=0x7f4c280d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.688 7f4c2dac6700 1 --2- 172.21.15.32:0/1412530654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280d7ca0 0x7f4c280d80a0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f4c14006650 tx=0x7f4c14009a00).stop 2021-04-09T18:23:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.689 7f4c2dac6700 1 -- 172.21.15.32:0/1412530654 shutdown_connections 2021-04-09T18:23:03.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.689 7f4c2dac6700 1 --2- 172.21.15.32:0/1412530654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280d7ca0 0x7f4c280d80a0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:03.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.689 7f4c2dac6700 1 -- 172.21.15.32:0/1412530654 >> 172.21.15.32:0/1412530654 conn(0x7f4c280ce5f0 msgr2=0x7f4c280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:03.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.689 7f4c2dac6700 1 -- 172.21.15.32:0/1412530654 shutdown_connections 2021-04-09T18:23:03.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.689 7f4c2dac6700 1 -- 172.21.15.32:0/1412530654 wait complete. 2021-04-09T18:23:03.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.709 7f4c2dac6700 1 Processor -- start 2021-04-09T18:23:03.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.709 7f4c2dac6700 1 -- start start 2021-04-09T18:23:03.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c2dac6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c280aece0 0x7f4c280ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:03.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c2dac6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280acf30 0x7f4c280aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:03.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c2dac6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c28196fa0 0x7f4c280ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:03.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c2dac6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c28092a70 con 0x7f4c280acf30 2021-04-09T18:23:03.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c2dac6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c280abba0 con 0x7f4c280aece0 2021-04-09T18:23:03.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c2dac6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c280aa8a0 con 0x7f4c28196fa0 2021-04-09T18:23:03.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c26ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280acf30 0x7f4c280aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:03.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c26ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280acf30 0x7f4c280aae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49948/0 (socket says 172.21.15.32:49948) 2021-04-09T18:23:03.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c26ffd700 1 -- 172.21.15.32:0/746371883 learned_addr learned my addr 172.21.15.32:0/746371883 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:03.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c27fff700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c28196fa0 0x7f4c280ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:03.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c277fe700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c280aece0 0x7f4c280ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:03.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c26ffd700 1 -- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c28196fa0 msgr2=0x7f4c280ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:03.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c26ffd700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c28196fa0 0x7f4c280ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:03.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c26ffd700 1 -- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c280aece0 msgr2=0x7f4c280ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:03.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c26ffd700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c280aece0 0x7f4c280ac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:03.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c26ffd700 1 -- 172.21.15.32:0/746371883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c280a9680 con 0x7f4c280acf30 2021-04-09T18:23:03.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c26ffd700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280acf30 0x7f4c280aae20 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7f4c1400b580 tx=0x7f4c1400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:03.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c1ffff700 1 -- 172.21.15.32:0/746371883 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c14000c90 con 0x7f4c280acf30 2021-04-09T18:23:03.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c1ffff700 1 -- 172.21.15.32:0/746371883 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c14002030 con 0x7f4c280acf30 2021-04-09T18:23:03.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c280a9680 con 0x7f4c280acf30 2021-04-09T18:23:03.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c1ffff700 1 -- 172.21.15.32:0/746371883 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c14002480 con 0x7f4c280acf30 2021-04-09T18:23:03.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.710 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c280a9680 con 0x7f4c280acf30 2021-04-09T18:23:03.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.712 7f4c1ffff700 1 -- 172.21.15.32:0/746371883 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4c14000c90 con 0x7f4c280acf30 2021-04-09T18:23:03.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.712 7f4c1ffff700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c040188c0 0x7f4c0401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:03.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.712 7f4c1ffff700 1 -- 172.21.15.32:0/746371883 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4c140327f0 con 0x7f4c280acf30 2021-04-09T18:23:03.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.712 7f4c277fe700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c040188c0 0x7f4c0401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:03.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.712 7f4c277fe700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c040188c0 0x7f4c0401adc0 crc :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:03.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.731 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2808dab0 con 0x7f4c280acf30 2021-04-09T18:23:03.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.735 7f4c1ffff700 1 -- 172.21.15.32:0/746371883 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4c140009c0 con 0x7f4c280acf30 2021-04-09T18:23:03.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:03.929 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f4c2810d480 con 0x7f4c280acf30 2021-04-09T18:23:04.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.505 7f4c1ffff700 1 -- 172.21.15.32:0/746371883 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v147) v1 ==== 105+0+0 (secure 0 0 0) 0x7f4c14000c90 con 0x7f4c280acf30 2021-04-09T18:23:04.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.545 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c040188c0 msgr2=0x7f4c0401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.545 7f4c2dac6700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c040188c0 0x7f4c0401adc0 crc :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.545 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280acf30 msgr2=0x7f4c280aae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.545 7f4c2dac6700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280acf30 0x7f4c280aae20 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7f4c1400b580 tx=0x7f4c1400b920).stop 2021-04-09T18:23:04.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.546 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 shutdown_connections 2021-04-09T18:23:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.546 7f4c2dac6700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c040188c0 0x7f4c0401adc0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.546 7f4c2dac6700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c280acf30 0x7f4c280aae20 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.546 7f4c2dac6700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c28196fa0 0x7f4c280ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.546 7f4c2dac6700 1 --2- 172.21.15.32:0/746371883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c280aece0 0x7f4c280ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.546 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 >> 172.21.15.32:0/746371883 conn(0x7f4c280bb410 msgr2=0x7f4c2808dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:04.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.546 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 shutdown_connections 2021-04-09T18:23:04.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.546 7f4c2dac6700 1 -- 172.21.15.32:0/746371883 wait complete. 2021-04-09T18:23:04.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:04.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.708 7f79b7da4700 1 Processor -- start 2021-04-09T18:23:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.708 7f79b7da4700 1 -- start start 2021-04-09T18:23:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b7da4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f79b010d010 0x7f79b00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b7da4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00d1da0 0x7f79b00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:04.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b7da4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b010d480 con 0x7f79b00d82c0 2021-04-09T18:23:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b7da4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b00d9220 con 0x7f79b010d010 2021-04-09T18:23:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b7da4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b0112100 con 0x7f79b00d1da0 2021-04-09T18:23:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b5bc0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f79b010d010 0x7f79b00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b53bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00d1da0 0x7f79b00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:04.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b63c1700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f79b00d82c0 0x7f79b010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:57474/0 (socket says 172.21.15.32:57474) 2021-04-09T18:23:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b63c1700 1 -- 172.21.15.32:0/1193527651 learned_addr learned my addr 172.21.15.32:0/1193527651 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b53bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00d1da0 0x7f79b00d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:57184/0 (socket says 172.21.15.32:57184) 2021-04-09T18:23:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.709 7f79b5bc0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f79b010d010 0x7f79b00d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:41714/0 (socket says 172.21.15.32:41714) 2021-04-09T18:23:04.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b5bc0700 1 -- 172.21.15.32:0/1193527651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00d1da0 msgr2=0x7f79b00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:04.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b5bc0700 1 --2- 172.21.15.32:0/1193527651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00d1da0 0x7f79b00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b5bc0700 1 -- 172.21.15.32:0/1193527651 >> v1:172.21.15.86:6789/0 conn(0x7f79b00d82c0 legacy=0x7f79b010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:04.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b5bc0700 1 -- 172.21.15.32:0/1193527651 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b00d86c0 con 0x7f79b010d010 2021-04-09T18:23:04.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b53bf700 1 --2- 172.21.15.32:0/1193527651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00d1da0 0x7f79b00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b4bbe700 1 -- 172.21.15.32:0/1193527651 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86311603 0 0) 0x7f79a8001440 con 0x7f79b00d82c0 2021-04-09T18:23:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b5bc0700 1 --2- 172.21.15.32:0/1193527651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b010d010 0x7f79b00d1780 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f79ac006650 tx=0x7f79ac009a80).ready entity=mon.1 client_cookie=f8d4686c1ed9863d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b4bbe700 1 -- 172.21.15.32:0/1193527651 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79ac0009b0 con 0x7f79b010d010 2021-04-09T18:23:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b4bbe700 1 -- 172.21.15.32:0/1193527651 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ac00c0c0 con 0x7f79b010d010 2021-04-09T18:23:04.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b4bbe700 1 -- 172.21.15.32:0/1193527651 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79ac00c510 con 0x7f79b010d010 2021-04-09T18:23:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b7da4700 1 -- 172.21.15.32:0/1193527651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b010d010 msgr2=0x7f79b00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.710 7f79b7da4700 1 --2- 172.21.15.32:0/1193527651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b010d010 0x7f79b00d1780 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f79ac006650 tx=0x7f79ac009a80).stop 2021-04-09T18:23:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.711 7f79b7da4700 1 -- 172.21.15.32:0/1193527651 shutdown_connections 2021-04-09T18:23:04.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.711 7f79b7da4700 1 --2- 172.21.15.32:0/1193527651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00d1da0 0x7f79b00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.711 7f79b7da4700 1 --2- 172.21.15.32:0/1193527651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b010d010 0x7f79b00d1780 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.711 7f79b7da4700 1 -- 172.21.15.32:0/1193527651 >> 172.21.15.32:0/1193527651 conn(0x7f79b00ce5f0 msgr2=0x7f79b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:04.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.711 7f79b7da4700 1 -- 172.21.15.32:0/1193527651 shutdown_connections 2021-04-09T18:23:04.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.711 7f79b7da4700 1 -- 172.21.15.32:0/1193527651 wait complete. 2021-04-09T18:23:04.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.731 7f79b7da4700 1 Processor -- start 2021-04-09T18:23:04.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.731 7f79b7da4700 1 -- start start 2021-04-09T18:23:04.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b7da4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b0191df0 0x7f79b0090ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:04.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b7da4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00914f0 0x7f79b008d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:04.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b7da4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b008da90 0x7f79b008b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:04.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b7da4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b008de90 con 0x7f79b008da90 2021-04-09T18:23:04.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b7da4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b008e080 con 0x7f79b0191df0 2021-04-09T18:23:04.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b7da4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b008a620 con 0x7f79b00914f0 2021-04-09T18:23:04.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b63c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b008da90 0x7f79b008b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:04.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b63c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b008da90 0x7f79b008b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49966/0 (socket says 172.21.15.32:49966) 2021-04-09T18:23:04.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b53bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00914f0 0x7f79b008d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:04.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b63c1700 1 -- 172.21.15.32:0/2133093190 learned_addr learned my addr 172.21.15.32:0/2133093190 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:04.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b5bc0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b0191df0 0x7f79b0090ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:04.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b63c1700 1 -- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00914f0 msgr2=0x7f79b008d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:04.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b63c1700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00914f0 0x7f79b008d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b63c1700 1 -- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b0191df0 msgr2=0x7f79b0090ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:04.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b63c1700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b0191df0 0x7f79b0090ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:04.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b63c1700 1 -- 172.21.15.32:0/2133093190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b008a810 con 0x7f79b008da90 2021-04-09T18:23:04.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b53bf700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00914f0 0x7f79b008d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:04.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b63c1700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b008da90 0x7f79b008b730 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f79a8001b30 tx=0x7f79a800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79a67fc700 1 -- 172.21.15.32:0/2133093190 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79a80048b0 con 0x7f79b008da90 2021-04-09T18:23:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79a67fc700 1 -- 172.21.15.32:0/2133093190 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a8004bf0 con 0x7f79b008da90 2021-04-09T18:23:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b008a810 con 0x7f79b008da90 2021-04-09T18:23:04.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.732 7f79a67fc700 1 -- 172.21.15.32:0/2133093190 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79a8007350 con 0x7f79b008da90 2021-04-09T18:23:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.733 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b008a810 con 0x7f79b008da90 2021-04-09T18:23:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.733 7f79a67fc700 1 -- 172.21.15.32:0/2133093190 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f79a80046a0 con 0x7f79b008da90 2021-04-09T18:23:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.733 7f79a67fc700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7998018930 0x7f799801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.733 7f79a67fc700 1 -- 172.21.15.32:0/2133093190 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f79a8032db0 con 0x7f79b008da90 2021-04-09T18:23:04.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.733 7f79b5bc0700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7998018930 0x7f799801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:04.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.734 7f79b5bc0700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7998018930 0x7f799801ae10 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:04.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.753 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b010d480 con 0x7f79b008da90 2021-04-09T18:23:04.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.757 7f79a67fc700 1 -- 172.21.15.32:0/2133093190 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f79a8004bd0 con 0x7f79b008da90 2021-04-09T18:23:05.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:04.951 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f79b010d480 con 0x7f79b008da90 2021-04-09T18:23:05.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.506 7f79a67fc700 1 -- 172.21.15.32:0/2133093190 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v148) v1 ==== 105+0+0 (secure 0 0 0) 0x7f79a80046a0 con 0x7f79b008da90 2021-04-09T18:23:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7998018930 msgr2=0x7f799801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7998018930 0x7f799801ae10 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b008da90 msgr2=0x7f79b008b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b008da90 0x7f79b008b730 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f79a8001b30 tx=0x7f79a800aca0).stop 2021-04-09T18:23:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 shutdown_connections 2021-04-09T18:23:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7998018930 0x7f799801ae10 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79b008da90 0x7f79b008b730 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b00914f0 0x7f79b008d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 --2- 172.21.15.32:0/2133093190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b0191df0 0x7f79b0090ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 >> 172.21.15.32:0/2133093190 conn(0x7f79b00b2fa0 msgr2=0x7f79b00a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 shutdown_connections 2021-04-09T18:23:05.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.519 7f79b7da4700 1 -- 172.21.15.32:0/2133093190 wait complete. 2021-04-09T18:23:05.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:05.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.687 7f43f6062700 1 Processor -- start 2021-04-09T18:23:05.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.687 7f43f6062700 1 -- start start 2021-04-09T18:23:05.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43f6062700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f43f010d010 0x7f43f00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:05.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43f6062700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f43f00d1da0 0x7f43f00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:05.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43f6062700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f010d480 con 0x7f43f00d82c0 2021-04-09T18:23:05.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43f6062700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f00d8d30 con 0x7f43f010d010 2021-04-09T18:23:05.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43f6062700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f00d9220 con 0x7f43f00d1da0 2021-04-09T18:23:05.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43ef7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f43f010d010 0x7f43f00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:05.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43ef7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f43f010d010 0x7f43f00d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49970/0 (socket says 172.21.15.32:49970) 2021-04-09T18:23:05.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43ef7fe700 1 -- 172.21.15.32:0/275275612 learned_addr learned my addr 172.21.15.32:0/275275612 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:05.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43effff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f43f00d82c0 0x7f43f010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59562/0 (socket says 172.21.15.32:59562) 2021-04-09T18:23:05.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43eeffd700 1 --2- 172.21.15.32:0/275275612 >> v2:172.21.15.86:3300/0 conn(0x7f43f00d1da0 0x7f43f00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:05.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43ef7fe700 1 -- 172.21.15.32:0/275275612 >> v2:172.21.15.86:3300/0 conn(0x7f43f00d1da0 msgr2=0x7f43f00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:05.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43ef7fe700 1 --2- 172.21.15.32:0/275275612 >> v2:172.21.15.86:3300/0 conn(0x7f43f00d1da0 0x7f43f00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43ef7fe700 1 -- 172.21.15.32:0/275275612 >> v1:172.21.15.32:6789/0 conn(0x7f43f00d82c0 legacy=0x7f43f010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:05.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43ef7fe700 1 -- 172.21.15.32:0/275275612 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f00d86f0 con 0x7f43f010d010 2021-04-09T18:23:05.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.688 7f43ee7fc700 1 -- 172.21.15.32:0/275275612 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959467207 0 0) 0x7f43e8000d30 con 0x7f43f00d82c0 2021-04-09T18:23:05.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43ef7fe700 1 --2- 172.21.15.32:0/275275612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f010d010 0x7f43f00d1780 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f43e0006700 tx=0x7f43e0009b00).ready entity=mon.0 client_cookie=49eb9d0def7ae278 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:05.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43ee7fc700 1 -- 172.21.15.32:0/275275612 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43e000c0c0 con 0x7f43f010d010 2021-04-09T18:23:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43ee7fc700 1 -- 172.21.15.32:0/275275612 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e000c400 con 0x7f43f010d010 2021-04-09T18:23:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43ee7fc700 1 -- 172.21.15.32:0/275275612 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43e000c8e0 con 0x7f43f010d010 2021-04-09T18:23:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43f6062700 1 -- 172.21.15.32:0/275275612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f010d010 msgr2=0x7f43f00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:05.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43f6062700 1 --2- 172.21.15.32:0/275275612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f010d010 0x7f43f00d1780 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f43e0006700 tx=0x7f43e0009b00).stop 2021-04-09T18:23:05.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43f6062700 1 -- 172.21.15.32:0/275275612 shutdown_connections 2021-04-09T18:23:05.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43f6062700 1 --2- 172.21.15.32:0/275275612 >> v2:172.21.15.86:3300/0 conn(0x7f43f00d1da0 0x7f43f00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43f6062700 1 --2- 172.21.15.32:0/275275612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f010d010 0x7f43f00d1780 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.689 7f43f6062700 1 -- 172.21.15.32:0/275275612 >> 172.21.15.32:0/275275612 conn(0x7f43f00ce5f0 msgr2=0x7f43f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:05.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.690 7f43f6062700 1 -- 172.21.15.32:0/275275612 shutdown_connections 2021-04-09T18:23:05.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.690 7f43f6062700 1 -- 172.21.15.32:0/275275612 wait complete. 2021-04-09T18:23:05.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.710 7f43f6062700 1 Processor -- start 2021-04-09T18:23:05.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.710 7f43f6062700 1 -- start start 2021-04-09T18:23:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.710 7f43f6062700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43f00b1680 0x7f43f00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.710 7f43f6062700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43f009e330 0x7f43f00b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.710 7f43f6062700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f00b2190 0x7f43f009f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:05.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.710 7f43f6062700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f009e730 con 0x7f43f00b2190 2021-04-09T18:23:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.710 7f43f6062700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f009f6c0 con 0x7f43f00b1680 2021-04-09T18:23:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.710 7f43f6062700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f009f8e0 con 0x7f43f009e330 2021-04-09T18:23:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f00b2190 0x7f43f009f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:05.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43effff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f00b2190 0x7f43f009f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49978/0 (socket says 172.21.15.32:49978) 2021-04-09T18:23:05.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43effff700 1 -- 172.21.15.32:0/4169643882 learned_addr learned my addr 172.21.15.32:0/4169643882 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:05.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43eeffd700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43f009e330 0x7f43f00b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:05.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43ef7fe700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43f00b1680 0x7f43f00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:05.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43effff700 1 -- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43f009e330 msgr2=0x7f43f00b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:05.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43effff700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43f009e330 0x7f43f00b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43effff700 1 -- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43f00b1680 msgr2=0x7f43f00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43effff700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43f00b1680 0x7f43f00b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43effff700 1 -- 172.21.15.32:0/4169643882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f009fb00 con 0x7f43f00b2190 2021-04-09T18:23:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43effff700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f00b2190 0x7f43f009f0a0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f43e8007f90 tx=0x7f43e800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43d7fff700 1 -- 172.21.15.32:0/4169643882 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43e8000cc0 con 0x7f43f00b2190 2021-04-09T18:23:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43d7fff700 1 -- 172.21.15.32:0/4169643882 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e8007020 con 0x7f43f00b2190 2021-04-09T18:23:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43f6062700 1 -- 172.21.15.32:0/4169643882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f009fb00 con 0x7f43f00b2190 2021-04-09T18:23:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43d7fff700 1 -- 172.21.15.32:0/4169643882 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43e8007430 con 0x7f43f00b2190 2021-04-09T18:23:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.711 7f43f6062700 1 -- 172.21.15.32:0/4169643882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f009fb00 con 0x7f43f00b2190 2021-04-09T18:23:05.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.712 7f43d7fff700 1 -- 172.21.15.32:0/4169643882 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f43e8007200 con 0x7f43f00b2190 2021-04-09T18:23:05.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.712 7f43d7fff700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43d0018930 0x7f43d001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:05.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.712 7f43d7fff700 1 -- 172.21.15.32:0/4169643882 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f43e8032d70 con 0x7f43f00b2190 2021-04-09T18:23:05.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.712 7f43ef7fe700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43d0018930 0x7f43d001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:05.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.712 7f43ef7fe700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43d0018930 0x7f43d001adc0 crc :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:05.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.732 7f43f6062700 1 -- 172.21.15.32:0/4169643882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f010d480 con 0x7f43f00b2190 2021-04-09T18:23:05.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.735 7f43d7fff700 1 -- 172.21.15.32:0/4169643882 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f43e8007d00 con 0x7f43f00b2190 2021-04-09T18:23:05.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:05.928 7f43f6062700 1 -- 172.21.15.32:0/4169643882 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f43f010d480 con 0x7f43f00b2190 2021-04-09T18:23:06.491 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:23:06.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:23:06.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.510 7f43d7fff700 1 -- 172.21.15.32:0/4169643882 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v149) v1 ==== 105+0+0 (secure 0 0 0) 0x7f43e8007000 con 0x7f43f00b2190 2021-04-09T18:23:06.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.545 7f43f6062700 1 -- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43d0018930 msgr2=0x7f43d001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.545 7f43f6062700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43d0018930 0x7f43d001adc0 crc :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.545 7f43f6062700 1 -- 172.21.15.32:0/4169643882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f00b2190 msgr2=0x7f43f009f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.545 7f43f6062700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f00b2190 0x7f43f009f0a0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f43e8007f90 tx=0x7f43e800ac50).stop 2021-04-09T18:23:06.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.546 7f43f6062700 1 -- 172.21.15.32:0/4169643882 shutdown_connections 2021-04-09T18:23:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.546 7f43f6062700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43d0018930 0x7f43d001adc0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.546 7f43f6062700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43f00b2190 0x7f43f009f0a0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.546 7f43f6062700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43f009e330 0x7f43f00b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.546 7f43f6062700 1 --2- 172.21.15.32:0/4169643882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43f00b1680 0x7f43f00b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.546 7f43f6062700 1 -- 172.21.15.32:0/4169643882 >> 172.21.15.32:0/4169643882 conn(0x7f43f00c4290 msgr2=0x7f43f00c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.546 7f43f6062700 1 -- 172.21.15.32:0/4169643882 shutdown_connections 2021-04-09T18:23:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.546 7f43f6062700 1 -- 172.21.15.32:0/4169643882 wait complete. 2021-04-09T18:23:06.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:06.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.750 7fe180762700 1 Processor -- start 2021-04-09T18:23:06.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.750 7fe180762700 1 -- start start 2021-04-09T18:23:06.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe180762700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1780d7ca0 0x7fe1780d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:06.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe180762700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe17810da00 0x7fe17810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:06.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe180762700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17810d480 con 0x7fe17810d010 2021-04-09T18:23:06.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe180762700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1780d9220 con 0x7fe1780d7ca0 2021-04-09T18:23:06.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe180762700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1780d86c0 con 0x7fe17810da00 2021-04-09T18:23:06.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe17dd7d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1780d7ca0 0x7fe1780d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:06.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe17dd7d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1780d7ca0 0x7fe1780d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49988/0 (socket says 172.21.15.32:49988) 2021-04-09T18:23:06.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe17dd7d700 1 -- 172.21.15.32:0/3122010303 learned_addr learned my addr 172.21.15.32:0/3122010303 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:06.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe17ed7f700 1 --2- 172.21.15.32:0/3122010303 >> v2:172.21.15.86:3300/0 conn(0x7fe17810da00 0x7fe17810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:06.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe17dd7d700 1 -- 172.21.15.32:0/3122010303 >> v2:172.21.15.86:3300/0 conn(0x7fe17810da00 msgr2=0x7fe17810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:06.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe17dd7d700 1 --2- 172.21.15.32:0/3122010303 >> v2:172.21.15.86:3300/0 conn(0x7fe17810da00 0x7fe17810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe17dd7d700 1 -- 172.21.15.32:0/3122010303 >> v1:172.21.15.86:6789/0 conn(0x7fe17810d010 legacy=0x7fe1780d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:06.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe17dd7d700 1 -- 172.21.15.32:0/3122010303 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe178112100 con 0x7fe1780d7ca0 2021-04-09T18:23:06.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.751 7fe17dd7d700 1 --2- 172.21.15.32:0/3122010303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1780d7ca0 0x7fe1780d80a0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7fe16c006700 tx=0x7fe16c009b80).ready entity=mon.0 client_cookie=4e7e9c927ab990f6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:06.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.752 7fe17d57c700 1 -- 172.21.15.32:0/3122010303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe16c00c0c0 con 0x7fe1780d7ca0 2021-04-09T18:23:06.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.752 7fe17d57c700 1 -- 172.21.15.32:0/3122010303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16c00c440 con 0x7fe1780d7ca0 2021-04-09T18:23:06.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.752 7fe17d57c700 1 -- 172.21.15.32:0/3122010303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe16c00c920 con 0x7fe1780d7ca0 2021-04-09T18:23:06.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.752 7fe180762700 1 -- 172.21.15.32:0/3122010303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1780d7ca0 msgr2=0x7fe1780d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:06.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.752 7fe180762700 1 --2- 172.21.15.32:0/3122010303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1780d7ca0 0x7fe1780d80a0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7fe16c006700 tx=0x7fe16c009b80).stop 2021-04-09T18:23:06.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.753 7fe180762700 1 -- 172.21.15.32:0/3122010303 shutdown_connections 2021-04-09T18:23:06.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.753 7fe180762700 1 --2- 172.21.15.32:0/3122010303 >> v2:172.21.15.86:3300/0 conn(0x7fe17810da00 0x7fe17810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.753 7fe180762700 1 --2- 172.21.15.32:0/3122010303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1780d7ca0 0x7fe1780d80a0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.753 7fe180762700 1 -- 172.21.15.32:0/3122010303 >> 172.21.15.32:0/3122010303 conn(0x7fe1780ce5f0 msgr2=0x7fe1780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:06.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.753 7fe180762700 1 -- 172.21.15.32:0/3122010303 shutdown_connections 2021-04-09T18:23:06.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.753 7fe180762700 1 -- 172.21.15.32:0/3122010303 wait complete. 2021-04-09T18:23:06.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe180762700 1 Processor -- start 2021-04-09T18:23:06.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe180762700 1 -- start start 2021-04-09T18:23:06.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe180762700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1780c3c00 0x7fe17809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:06.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe180762700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe178095240 0x7fe1781a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:06.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe180762700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1781a1270 0x7fe1780b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:06.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe180762700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17809cda0 con 0x7fe1781a1270 2021-04-09T18:23:06.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe180762700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780c3680 con 0x7fe1780c3c00 2021-04-09T18:23:06.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe180762700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe178095640 con 0x7fe178095240 2021-04-09T18:23:06.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe17ed7f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1781a1270 0x7fe1780b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe17ed7f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1781a1270 0x7fe1780b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49994/0 (socket says 172.21.15.32:49994) 2021-04-09T18:23:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe17ed7f700 1 -- 172.21.15.32:0/678720368 learned_addr learned my addr 172.21.15.32:0/678720368 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:06.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe17dd7d700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe178095240 0x7fe1781a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:06.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.773 7fe17e57e700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1780c3c00 0x7fe17809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:06.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe17ed7f700 1 -- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe178095240 msgr2=0x7fe1781a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:06.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe17ed7f700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe178095240 0x7fe1781a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe17ed7f700 1 -- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1780c3c00 msgr2=0x7fe17809c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:06.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe17ed7f700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1780c3c00 0x7fe17809c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:06.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe17ed7f700 1 -- 172.21.15.32:0/678720368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1781a1670 con 0x7fe1781a1270 2021-04-09T18:23:06.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe17ed7f700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1781a1270 0x7fe1780b0cf0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7fe170009980 tx=0x7fe17000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe16affd700 1 -- 172.21.15.32:0/678720368 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe17000c2f0 con 0x7fe1781a1270 2021-04-09T18:23:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe16affd700 1 -- 172.21.15.32:0/678720368 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17000c5b0 con 0x7fe1781a1270 2021-04-09T18:23:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe180762700 1 -- 172.21.15.32:0/678720368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1781a1670 con 0x7fe1781a1270 2021-04-09T18:23:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe16affd700 1 -- 172.21.15.32:0/678720368 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe17000ca90 con 0x7fe1781a1270 2021-04-09T18:23:06.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.774 7fe180762700 1 -- 172.21.15.32:0/678720368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1781a1670 con 0x7fe1781a1270 2021-04-09T18:23:06.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.775 7fe16affd700 1 -- 172.21.15.32:0/678720368 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe1700011f0 con 0x7fe1781a1270 2021-04-09T18:23:06.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.775 7fe16affd700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1600188d0 0x7fe16001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:06.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.775 7fe16affd700 1 -- 172.21.15.32:0/678720368 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe170032db0 con 0x7fe1781a1270 2021-04-09T18:23:06.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.775 7fe17e57e700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1600188d0 0x7fe16001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:06.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.775 7fe17e57e700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1600188d0 0x7fe16001add0 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:06.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.795 7fe180762700 1 -- 172.21.15.32:0/678720368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe178093d50 con 0x7fe1781a1270 2021-04-09T18:23:06.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.798 7fe16affd700 1 -- 172.21.15.32:0/678720368 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe17000be10 con 0x7fe1781a1270 2021-04-09T18:23:07.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:06.989 7fe180762700 1 -- 172.21.15.32:0/678720368 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fe17810d480 con 0x7fe1781a1270 2021-04-09T18:23:07.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.513 7fe16affd700 1 -- 172.21.15.32:0/678720368 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v150) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe1700011f0 con 0x7fe1781a1270 2021-04-09T18:23:07.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 -- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1600188d0 msgr2=0x7fe16001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:07.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1600188d0 0x7fe16001add0 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 -- 172.21.15.32:0/678720368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1781a1270 msgr2=0x7fe1780b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:07.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1781a1270 0x7fe1780b0cf0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7fe170009980 tx=0x7fe17000bcd0).stop 2021-04-09T18:23:07.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 -- 172.21.15.32:0/678720368 shutdown_connections 2021-04-09T18:23:07.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe1600188d0 0x7fe16001add0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1781a1270 0x7fe1780b0cf0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe178095240 0x7fe1781a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 --2- 172.21.15.32:0/678720368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1780c3c00 0x7fe17809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 -- 172.21.15.32:0/678720368 >> 172.21.15.32:0/678720368 conn(0x7fe1780975f0 msgr2=0x7fe178096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:07.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 -- 172.21.15.32:0/678720368 shutdown_connections 2021-04-09T18:23:07.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.558 7fe180762700 1 -- 172.21.15.32:0/678720368 wait complete. 2021-04-09T18:23:07.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:07.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.724 7f1c01299700 1 Processor -- start 2021-04-09T18:23:07.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.724 7f1c01299700 1 -- start start 2021-04-09T18:23:07.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.725 7f1c01299700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc10d010 0x7f1bfc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:07.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.725 7f1c01299700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1bfc0d1da0 0x7f1bfc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:07.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.725 7f1c01299700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1bfc0d82c0 0x7f1bfc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:07.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.725 7f1c01299700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc10e780 con 0x7f1bfc0d1da0 2021-04-09T18:23:07.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.725 7f1c01299700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc0d8d30 con 0x7f1bfc0d82c0 2021-04-09T18:23:07.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.725 7f1c01299700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc0d9220 con 0x7f1bfc10d010 2021-04-09T18:23:07.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.725 7f1bfa87c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1bfc0d1da0 0x7f1bfc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:07.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.725 7f1bfa87c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1bfc0d1da0 0x7f1bfc0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50000/0 (socket says 172.21.15.32:50000) 2021-04-09T18:23:07.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.725 7f1bfa87c700 1 -- 172.21.15.32:0/1292297858 learned_addr learned my addr 172.21.15.32:0/1292297858 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:07.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfb07d700 1 --2- 172.21.15.32:0/1292297858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc10d010 0x7f1bfc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:07.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfb87e700 1 --2- 172.21.15.32:0/1292297858 >> v2:172.21.15.86:3300/0 conn(0x7f1bfc0d82c0 0x7f1bfc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:07.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfa87c700 1 -- 172.21.15.32:0/1292297858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc10d010 msgr2=0x7f1bfc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:07.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfa87c700 1 --2- 172.21.15.32:0/1292297858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc10d010 0x7f1bfc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfa87c700 1 -- 172.21.15.32:0/1292297858 >> v2:172.21.15.86:3300/0 conn(0x7f1bfc0d82c0 msgr2=0x7f1bfc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:07.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfa87c700 1 --2- 172.21.15.32:0/1292297858 >> v2:172.21.15.86:3300/0 conn(0x7f1bfc0d82c0 0x7f1bfc10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfa87c700 1 -- 172.21.15.32:0/1292297858 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc10d480 con 0x7f1bfc0d1da0 2021-04-09T18:23:07.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfa87c700 1 --2- 172.21.15.32:0/1292297858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc0d1da0 0x7f1bfc0d7ca0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f1bf0006700 tx=0x7f1bf0009b00).ready entity=mon.0 client_cookie=3fbd469747d6b2f9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:07.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfa07b700 1 -- 172.21.15.32:0/1292297858 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bf000c0c0 con 0x7f1bfc0d1da0 2021-04-09T18:23:07.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfa07b700 1 -- 172.21.15.32:0/1292297858 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf000c400 con 0x7f1bfc0d1da0 2021-04-09T18:23:07.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1bfa07b700 1 -- 172.21.15.32:0/1292297858 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bf000c8e0 con 0x7f1bfc0d1da0 2021-04-09T18:23:07.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1c01299700 1 -- 172.21.15.32:0/1292297858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc0d1da0 msgr2=0x7f1bfc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:07.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.726 7f1c01299700 1 --2- 172.21.15.32:0/1292297858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc0d1da0 0x7f1bfc0d7ca0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f1bf0006700 tx=0x7f1bf0009b00).stop 2021-04-09T18:23:07.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.727 7f1c01299700 1 -- 172.21.15.32:0/1292297858 shutdown_connections 2021-04-09T18:23:07.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.727 7f1c01299700 1 --2- 172.21.15.32:0/1292297858 >> v2:172.21.15.86:3300/0 conn(0x7f1bfc0d82c0 0x7f1bfc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.727 7f1c01299700 1 --2- 172.21.15.32:0/1292297858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc0d1da0 0x7f1bfc0d7ca0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.727 7f1c01299700 1 --2- 172.21.15.32:0/1292297858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc10d010 0x7f1bfc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.727 7f1c01299700 1 -- 172.21.15.32:0/1292297858 >> 172.21.15.32:0/1292297858 conn(0x7f1bfc0ce5f0 msgr2=0x7f1bfc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:07.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.727 7f1c01299700 1 -- 172.21.15.32:0/1292297858 shutdown_connections 2021-04-09T18:23:07.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.727 7f1c01299700 1 -- 172.21.15.32:0/1292297858 wait complete. 2021-04-09T18:23:07.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.747 7f1c01299700 1 Processor -- start 2021-04-09T18:23:07.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.747 7f1c01299700 1 -- start start 2021-04-09T18:23:07.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1c01299700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bfc099ee0 0x7f1bfc095650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:07.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1c01299700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc09a2e0 0x7f1bfc0964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1c01299700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc096af0 0x7f1bfc0a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1c01299700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc095f50 con 0x7f1bfc096af0 2021-04-09T18:23:07.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1c01299700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc095c70 con 0x7f1bfc099ee0 2021-04-09T18:23:07.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1c01299700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc096ef0 con 0x7f1bfc09a2e0 2021-04-09T18:23:07.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc096af0 0x7f1bfc0a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:07.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc096af0 0x7f1bfc0a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50004/0 (socket says 172.21.15.32:50004) 2021-04-09T18:23:07.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb87e700 1 -- 172.21.15.32:0/1846587125 learned_addr learned my addr 172.21.15.32:0/1846587125 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:07.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfa87c700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc09a2e0 0x7f1bfc0964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:07.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb07d700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bfc099ee0 0x7f1bfc095650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:07.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb87e700 1 -- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc09a2e0 msgr2=0x7f1bfc0964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:07.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb87e700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc09a2e0 0x7f1bfc0964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb87e700 1 -- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bfc099ee0 msgr2=0x7f1bfc095650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:07.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb87e700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bfc099ee0 0x7f1bfc095650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:07.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb87e700 1 -- 172.21.15.32:0/1846587125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc0a4a20 con 0x7f1bfc096af0 2021-04-09T18:23:07.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1bfb87e700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc096af0 0x7f1bfc0a4400 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f1bf40099a0 tx=0x7f1bf400bc50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:07.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.748 7f1be3fff700 1 -- 172.21.15.32:0/1846587125 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bf4006030 con 0x7f1bfc096af0 2021-04-09T18:23:07.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.749 7f1be3fff700 1 -- 172.21.15.32:0/1846587125 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf4000fc0 con 0x7f1bfc096af0 2021-04-09T18:23:07.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.749 7f1be3fff700 1 -- 172.21.15.32:0/1846587125 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bf4006520 con 0x7f1bfc096af0 2021-04-09T18:23:07.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.749 7f1c01299700 1 -- 172.21.15.32:0/1846587125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc0a4a20 con 0x7f1bfc096af0 2021-04-09T18:23:07.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.749 7f1c01299700 1 -- 172.21.15.32:0/1846587125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc0a4a20 con 0x7f1bfc096af0 2021-04-09T18:23:07.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.749 7f1be3fff700 1 -- 172.21.15.32:0/1846587125 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1bf40062b0 con 0x7f1bfc096af0 2021-04-09T18:23:07.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.749 7f1be3fff700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bdc018930 0x7f1bdc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:07.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.749 7f1be3fff700 1 -- 172.21.15.32:0/1846587125 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1bf4032da0 con 0x7f1bfc096af0 2021-04-09T18:23:07.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.750 7f1bfb07d700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bdc018930 0x7f1bdc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:07.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.750 7f1bfb07d700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bdc018930 0x7f1bdc01ae10 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:07.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.769 7f1c01299700 1 -- 172.21.15.32:0/1846587125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc0a5590 con 0x7f1bfc096af0 2021-04-09T18:23:07.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.773 7f1be3fff700 1 -- 172.21.15.32:0/1846587125 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1bf400bd90 con 0x7f1bfc096af0 2021-04-09T18:23:08.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:07.964 7f1c01299700 1 -- 172.21.15.32:0/1846587125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f1bfc0a5590 con 0x7f1bfc096af0 2021-04-09T18:23:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.525 7f1be3fff700 1 -- 172.21.15.32:0/1846587125 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v151) v1 ==== 105+0+0 (secure 0 0 0) 0x7f1bf40062b0 con 0x7f1bfc096af0 2021-04-09T18:23:08.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 -- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bdc018930 msgr2=0x7f1bdc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:08.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bdc018930 0x7f1bdc01ae10 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 -- 172.21.15.32:0/1846587125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc096af0 msgr2=0x7f1bfc0a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:08.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc096af0 0x7f1bfc0a4400 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f1bf40099a0 tx=0x7f1bf400bc50).stop 2021-04-09T18:23:08.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 -- 172.21.15.32:0/1846587125 shutdown_connections 2021-04-09T18:23:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1bdc018930 0x7f1bdc01ae10 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bfc096af0 0x7f1bfc0a4400 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bfc09a2e0 0x7f1bfc0964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 --2- 172.21.15.32:0/1846587125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bfc099ee0 0x7f1bfc095650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.531 7f1c01299700 1 -- 172.21.15.32:0/1846587125 >> 172.21.15.32:0/1846587125 conn(0x7f1bfc0ab0a0 msgr2=0x7f1bfc0ab8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:08.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.532 7f1c01299700 1 -- 172.21.15.32:0/1846587125 shutdown_connections 2021-04-09T18:23:08.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.532 7f1c01299700 1 -- 172.21.15.32:0/1846587125 wait complete. 2021-04-09T18:23:08.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:08.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.696 7fc0c5b0e700 1 Processor -- start 2021-04-09T18:23:08.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.696 7fc0c5b0e700 1 -- start start 2021-04-09T18:23:08.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0c5b0e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc0c010da00 0x7fc0c010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:08.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0c5b0e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c010d480 con 0x7fc0c00d7ca0 2021-04-09T18:23:08.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0c5b0e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00d1f70 con 0x7fc0c010d010 2021-04-09T18:23:08.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0c5b0e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c00d8d30 con 0x7fc0c010da00 2021-04-09T18:23:08.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0beffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc0c00d7ca0 0x7fc0c00d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59604/0 (socket says 172.21.15.32:59604) 2021-04-09T18:23:08.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0beffd700 1 -- 172.21.15.32:0/3340022008 learned_addr learned my addr 172.21.15.32:0/3340022008 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:08.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0bffff700 1 --2- 172.21.15.32:0/3340022008 >> v2:172.21.15.86:3300/0 conn(0x7fc0c010da00 0x7fc0c010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:08.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212057167 0 0) 0x7fc0b4000ce0 con 0x7fc0c00d7ca0 2021-04-09T18:23:08.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a8002fe0 con 0x7fc0c00d7ca0 2021-04-09T18:23:08.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2549727415 0 0) 0x7fc0b40010e0 con 0x7fc0c00d7ca0 2021-04-09T18:23:08.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 >> v2:172.21.15.86:3300/0 conn(0x7fc0c010da00 msgr2=0x7fc0c010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:08.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0be7fc700 1 --2- 172.21.15.32:0/3340022008 >> v2:172.21.15.86:3300/0 conn(0x7fc0c010da00 0x7fc0c010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 >> v1:172.21.15.86:6789/0 conn(0x7fc0c010d010 legacy=0x7fc0c00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:08.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.697 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c00d9220 con 0x7fc0c00d7ca0 2021-04-09T18:23:08.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fc0b4001600 con 0x7fc0c00d7ca0 2021-04-09T18:23:08.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 >> v1:172.21.15.32:6789/0 conn(0x7fc0c00d7ca0 legacy=0x7fc0c00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:08.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0a8003a70 0x7fc0a8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:08.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a80067a0 0x7fc0a8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:08.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a80095a0 0x7fc0a800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:08.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a800c310 con 0x7fc0a80067a0 2021-04-09T18:23:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a800c530 con 0x7fc0a80095a0 2021-04-09T18:23:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0beffd700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a80067a0 0x7fc0a8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a8050760 con 0x7fc0a8003a70 2021-04-09T18:23:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0bf7fe700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0a8003a70 0x7fc0a8005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:08.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0bffff700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a80095a0 0x7fc0a800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:08.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0beffd700 1 -- 172.21.15.32:0/3340022008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0a8003a70 msgr2=0x7fc0a8005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:08.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0beffd700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0a8003a70 0x7fc0a8005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0beffd700 1 -- 172.21.15.32:0/3340022008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a80095a0 msgr2=0x7fc0a800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0beffd700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a80095a0 0x7fc0a800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0beffd700 1 -- 172.21.15.32:0/3340022008 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80509c0 con 0x7fc0a80067a0 2021-04-09T18:23:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0bffff700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0a80095a0 0x7fc0a800bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0bffff700 1 -- 172.21.15.32:0/3340022008 reap_dead start 2021-04-09T18:23:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0beffd700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a80067a0 0x7fc0a8008ca0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7fc0b40078f0 tx=0x7fc0b4007c90).ready entity=mon.0 client_cookie=38a7fa04557f5037 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:08.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0b4000c90 con 0x7fc0a80067a0 2021-04-09T18:23:08.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b40070c0 con 0x7fc0a80067a0 2021-04-09T18:23:08.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0be7fc700 1 -- 172.21.15.32:0/3340022008 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0b400a710 con 0x7fc0a80067a0 2021-04-09T18:23:08.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0c5b0e700 1 -- 172.21.15.32:0/3340022008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a80067a0 msgr2=0x7fc0a8008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.698 7fc0c5b0e700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a80067a0 0x7fc0a8008ca0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7fc0b40078f0 tx=0x7fc0b4007c90).stop 2021-04-09T18:23:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.699 7fc0c5b0e700 1 -- 172.21.15.32:0/3340022008 shutdown_connections 2021-04-09T18:23:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.699 7fc0c5b0e700 1 --2- 172.21.15.32:0/3340022008 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0a80067a0 0x7fc0a8008ca0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.699 7fc0c5b0e700 1 -- 172.21.15.32:0/3340022008 >> 172.21.15.32:0/3340022008 conn(0x7fc0c00ce5f0 msgr2=0x7fc0c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.699 7fc0c5b0e700 1 -- 172.21.15.32:0/3340022008 shutdown_connections 2021-04-09T18:23:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.699 7fc0c5b0e700 1 -- 172.21.15.32:0/3340022008 wait complete. 2021-04-09T18:23:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.719 7fc0c5b0e700 1 Processor -- start 2021-04-09T18:23:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.719 7fc0c5b0e700 1 -- start start 2021-04-09T18:23:08.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0c5b0e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c00a0a10 0x7fc0c00bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:08.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0c5b0e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0c018ad00 0x7fc0c019fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:08.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0c5b0e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c01a03f0 0x7fc0c0095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:08.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0c5b0e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c018b100 con 0x7fc0c018ad00 2021-04-09T18:23:08.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0c5b0e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c0095860 con 0x7fc0c01a03f0 2021-04-09T18:23:08.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0c5b0e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c0095a80 con 0x7fc0c00a0a10 2021-04-09T18:23:08.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0beffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0c018ad00 0x7fc0c019fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:08.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0beffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0c018ad00 0x7fc0c019fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50026/0 (socket says 172.21.15.32:50026) 2021-04-09T18:23:08.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0beffd700 1 -- 172.21.15.32:0/2489496229 learned_addr learned my addr 172.21.15.32:0/2489496229 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:08.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0bffff700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c01a03f0 0x7fc0c0095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:08.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0bf7fe700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c00a0a10 0x7fc0c00bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:08.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0beffd700 1 -- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c00a0a10 msgr2=0x7fc0c00bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:08.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0beffd700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c00a0a10 0x7fc0c00bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0beffd700 1 -- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c01a03f0 msgr2=0x7fc0c0095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:08.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0beffd700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c01a03f0 0x7fc0c0095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:08.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0beffd700 1 -- 172.21.15.32:0/2489496229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c0189250 con 0x7fc0c018ad00 2021-04-09T18:23:08.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.720 7fc0beffd700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0c018ad00 0x7fc0c019fdd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7fc0b4001c60 tx=0x7fc0b400b870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:08.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.721 7fc0a7fff700 1 -- 172.21.15.32:0/2489496229 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0b4007020 con 0x7fc0c018ad00 2021-04-09T18:23:08.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.721 7fc0a7fff700 1 -- 172.21.15.32:0/2489496229 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b40072e0 con 0x7fc0c018ad00 2021-04-09T18:23:08.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.721 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c0189250 con 0x7fc0c018ad00 2021-04-09T18:23:08.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.721 7fc0a7fff700 1 -- 172.21.15.32:0/2489496229 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0b4004620 con 0x7fc0c018ad00 2021-04-09T18:23:08.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.721 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c0189250 con 0x7fc0c018ad00 2021-04-09T18:23:08.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.721 7fc0a7fff700 1 -- 172.21.15.32:0/2489496229 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc0b4008fc0 con 0x7fc0c018ad00 2021-04-09T18:23:08.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.721 7fc0a7fff700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0a0018930 0x7fc0a001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:08.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.721 7fc0a7fff700 1 -- 172.21.15.32:0/2489496229 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc0b4032a00 con 0x7fc0c018ad00 2021-04-09T18:23:08.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.722 7fc0bf7fe700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0a0018930 0x7fc0a001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:08.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.722 7fc0bf7fe700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0a0018930 0x7fc0a001ae10 crc :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:08.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.741 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c0093d50 con 0x7fc0c018ad00 2021-04-09T18:23:08.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.744 7fc0a7fff700 1 -- 172.21.15.32:0/2489496229 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc0b40070d0 con 0x7fc0c018ad00 2021-04-09T18:23:09.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:08.944 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fc0c010d480 con 0x7fc0c018ad00 2021-04-09T18:23:09.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.525 7fc0a7fff700 1 -- 172.21.15.32:0/2489496229 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v152) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc0b4008fc0 con 0x7fc0c018ad00 2021-04-09T18:23:09.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0a0018930 msgr2=0x7fc0a001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:09.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0a0018930 0x7fc0a001ae10 crc :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:09.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0c018ad00 msgr2=0x7fc0c019fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:09.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0c018ad00 0x7fc0c019fdd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7fc0b4001c60 tx=0x7fc0b400b870).stop 2021-04-09T18:23:09.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 shutdown_connections 2021-04-09T18:23:09.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0a0018930 0x7fc0a001ae10 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:09.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0c018ad00 0x7fc0c019fdd0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:09.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c01a03f0 0x7fc0c0095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:09.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 --2- 172.21.15.32:0/2489496229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c00a0a10 0x7fc0c00bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:09.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 >> 172.21.15.32:0/2489496229 conn(0x7fc0c00975f0 msgr2=0x7fc0c0096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:09.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 shutdown_connections 2021-04-09T18:23:09.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.562 7fc0c5b0e700 1 -- 172.21.15.32:0/2489496229 wait complete. 2021-04-09T18:23:09.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:09.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.724 7f72fcab5700 1 Processor -- start 2021-04-09T18:23:09.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.724 7f72fcab5700 1 -- start start 2021-04-09T18:23:09.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72fcab5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72f810d010 0x7f72f80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:09.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72fcab5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f810d480 con 0x7f72f80d1da0 2021-04-09T18:23:09.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72fcab5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f810e1f0 con 0x7f72f80d8490 2021-04-09T18:23:09.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72fcab5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f80d8d30 con 0x7f72f810d010 2021-04-09T18:23:09.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72f685e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f72f810d010 0x7f72f80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:09.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72f605d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f72f80d1da0 0x7f72f80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59626/0 (socket says 172.21.15.32:59626) 2021-04-09T18:23:09.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72f605d700 1 -- 172.21.15.32:0/1433479431 learned_addr learned my addr 172.21.15.32:0/1433479431 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:09.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72f585c700 1 -- 172.21.15.32:0/1433479431 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740780288 0 0) 0x7f72f0000d30 con 0x7f72f80d8490 2021-04-09T18:23:09.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72f685e700 1 -- 172.21.15.32:0/1433479431 >> v1:172.21.15.86:6789/0 conn(0x7f72f80d8490 legacy=0x7f72f810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:09.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72f685e700 1 -- 172.21.15.32:0/1433479431 >> v1:172.21.15.32:6789/0 conn(0x7f72f80d1da0 legacy=0x7f72f80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:09.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.725 7f72f685e700 1 -- 172.21.15.32:0/1433479431 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80d9220 con 0x7f72f810d010 2021-04-09T18:23:09.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.726 7f72f685e700 1 --2- 172.21.15.32:0/1433479431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f810d010 0x7f72f80d1780 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f72e8006650 tx=0x7f72e8009a80).ready entity=mon.0 client_cookie=701a0dedbcaa4a62 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:09.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.726 7f72f585c700 1 -- 172.21.15.32:0/1433479431 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e80009b0 con 0x7f72f810d010 2021-04-09T18:23:09.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.726 7f72f585c700 1 -- 172.21.15.32:0/1433479431 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e800c0c0 con 0x7f72f810d010 2021-04-09T18:23:09.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.726 7f72f585c700 1 -- 172.21.15.32:0/1433479431 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e800c510 con 0x7f72f810d010 2021-04-09T18:23:09.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.726 7f72fcab5700 1 -- 172.21.15.32:0/1433479431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f810d010 msgr2=0x7f72f80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:09.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.726 7f72fcab5700 1 --2- 172.21.15.32:0/1433479431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f810d010 0x7f72f80d1780 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f72e8006650 tx=0x7f72e8009a80).stop 2021-04-09T18:23:09.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.727 7f72fcab5700 1 -- 172.21.15.32:0/1433479431 shutdown_connections 2021-04-09T18:23:09.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.727 7f72fcab5700 1 --2- 172.21.15.32:0/1433479431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f810d010 0x7f72f80d1780 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:09.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.727 7f72fcab5700 1 -- 172.21.15.32:0/1433479431 >> 172.21.15.32:0/1433479431 conn(0x7f72f80ce5f0 msgr2=0x7f72f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:09.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.727 7f72fcab5700 1 -- 172.21.15.32:0/1433479431 shutdown_connections 2021-04-09T18:23:09.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.727 7f72fcab5700 1 -- 172.21.15.32:0/1433479431 wait complete. 2021-04-09T18:23:09.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.747 7f72fcab5700 1 Processor -- start 2021-04-09T18:23:09.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.747 7f72fcab5700 1 -- start start 2021-04-09T18:23:09.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.747 7f72fcab5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f817ff00 0x7f72f8090a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:09.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.747 7f72fcab5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72f80acef0 0x7f72f819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:09.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.747 7f72fcab5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72f819b0f0 0x7f72f80ab890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:09.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.747 7f72fcab5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f817f980 con 0x7f72f817ff00 2021-04-09T18:23:09.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.747 7f72fcab5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80aee20 con 0x7f72f819b0f0 2021-04-09T18:23:09.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.747 7f72fcab5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8091080 con 0x7f72f80acef0 2021-04-09T18:23:09.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f817ff00 0x7f72f8090a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:09.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f817ff00 0x7f72f8090a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50038/0 (socket says 172.21.15.32:50038) 2021-04-09T18:23:09.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f685e700 1 -- 172.21.15.32:0/1642083423 learned_addr learned my addr 172.21.15.32:0/1642083423 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:09.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f605d700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72f80acef0 0x7f72f819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:09.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f705f700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72f819b0f0 0x7f72f80ab890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:09.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f685e700 1 -- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72f80acef0 msgr2=0x7f72f819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:09.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f685e700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72f80acef0 0x7f72f819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:09.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f685e700 1 -- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72f819b0f0 msgr2=0x7f72f80ab890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:09.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f685e700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72f819b0f0 0x7f72f80ab890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:09.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f685e700 1 -- 172.21.15.32:0/1642083423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f819b4f0 con 0x7f72f817ff00 2021-04-09T18:23:09.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72f685e700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f817ff00 0x7f72f8090a60 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f72e8009650 tx=0x7f72e800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:09.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72ef7fe700 1 -- 172.21.15.32:0/1642083423 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e80093a0 con 0x7f72f817ff00 2021-04-09T18:23:09.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72ef7fe700 1 -- 172.21.15.32:0/1642083423 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e800bd50 con 0x7f72f817ff00 2021-04-09T18:23:09.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f819b4f0 con 0x7f72f817ff00 2021-04-09T18:23:09.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72ef7fe700 1 -- 172.21.15.32:0/1642083423 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72e80052e0 con 0x7f72f817ff00 2021-04-09T18:23:09.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.748 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f819b4f0 con 0x7f72f817ff00 2021-04-09T18:23:09.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.749 7f72ef7fe700 1 -- 172.21.15.32:0/1642083423 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f72e8006030 con 0x7f72f817ff00 2021-04-09T18:23:09.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.749 7f72ef7fe700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72dc018900 0x7f72dc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:09.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.749 7f72ef7fe700 1 -- 172.21.15.32:0/1642083423 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f72e80329e0 con 0x7f72f817ff00 2021-04-09T18:23:09.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.749 7f72f605d700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72dc018900 0x7f72dc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:09.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.749 7f72f605d700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72dc018900 0x7f72dc01adc0 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:09.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.769 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f8091080 con 0x7f72f817ff00 2021-04-09T18:23:09.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.773 7f72ef7fe700 1 -- 172.21.15.32:0/1642083423 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f72e80009c0 con 0x7f72f817ff00 2021-04-09T18:23:10.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:09.967 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f72f8091080 con 0x7f72f817ff00 2021-04-09T18:23:10.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.535 7f72ef7fe700 1 -- 172.21.15.32:0/1642083423 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v153) v1 ==== 105+0+0 (secure 0 0 0) 0x7f72e8006030 con 0x7f72f817ff00 2021-04-09T18:23:10.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72dc018900 msgr2=0x7f72dc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:10.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72dc018900 0x7f72dc01adc0 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f817ff00 msgr2=0x7f72f8090a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:10.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f817ff00 0x7f72f8090a60 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f72e8009650 tx=0x7f72e800b960).stop 2021-04-09T18:23:10.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 shutdown_connections 2021-04-09T18:23:10.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72dc018900 0x7f72dc01adc0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72f819b0f0 0x7f72f80ab890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72f80acef0 0x7f72f819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 --2- 172.21.15.32:0/1642083423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72f817ff00 0x7f72f8090a60 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 >> 172.21.15.32:0/1642083423 conn(0x7f72f80bb410 msgr2=0x7f72f808dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:10.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 shutdown_connections 2021-04-09T18:23:10.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.586 7f72fcab5700 1 -- 172.21.15.32:0/1642083423 wait complete. 2021-04-09T18:23:10.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:10.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.761 7fe5e69e2700 1 Processor -- start 2021-04-09T18:23:10.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.761 7fe5e69e2700 1 -- start start 2021-04-09T18:23:10.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e69e2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe5e010d010 0x7fe5e00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:10.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e69e2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5e00d8490 0x7fe5e010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:10.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e69e2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e010d480 con 0x7fe5e00d1da0 2021-04-09T18:23:10.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e69e2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e00d8d30 con 0x7fe5e00d8490 2021-04-09T18:23:10.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e69e2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e00d9220 con 0x7fe5e010d010 2021-04-09T18:23:10.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e4fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5e00d8490 0x7fe5e010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:10.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e4fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5e00d8490 0x7fe5e010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50048/0 (socket says 172.21.15.32:50048) 2021-04-09T18:23:10.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e4fff700 1 -- 172.21.15.32:0/3390163093 learned_addr learned my addr 172.21.15.32:0/3390163093 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:10.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5dffff700 1 --2- 172.21.15.32:0/3390163093 >> v2:172.21.15.86:3300/0 conn(0x7fe5e010d010 0x7fe5e00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e4fff700 1 -- 172.21.15.32:0/3390163093 >> v2:172.21.15.86:3300/0 conn(0x7fe5e010d010 msgr2=0x7fe5e00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e4fff700 1 --2- 172.21.15.32:0/3390163093 >> v2:172.21.15.86:3300/0 conn(0x7fe5e010d010 0x7fe5e00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e4fff700 1 -- 172.21.15.32:0/3390163093 >> v1:172.21.15.86:6789/0 conn(0x7fe5e00d1da0 legacy=0x7fe5e00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5df7fe700 1 -- 172.21.15.32:0/3390163093 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912841189 0 0) 0x7fe5c8000c60 con 0x7fe5e00d1da0 2021-04-09T18:23:10.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5e4fff700 1 -- 172.21.15.32:0/3390163093 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e010e020 con 0x7fe5e00d8490 2021-04-09T18:23:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.762 7fe5dffff700 1 --2- 172.21.15.32:0/3390163093 >> v2:172.21.15.86:3300/0 conn(0x7fe5e010d010 0x7fe5e00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.763 7fe5e4fff700 1 --2- 172.21.15.32:0/3390163093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e00d8490 0x7fe5e010da00 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7fe5d8006f00 tx=0x7fe5d8009f10).ready entity=mon.0 client_cookie=86e3cc2a8f5519b1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.763 7fe5df7fe700 1 -- 172.21.15.32:0/3390163093 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5d800b3b0 con 0x7fe5e00d8490 2021-04-09T18:23:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.763 7fe5df7fe700 1 -- 172.21.15.32:0/3390163093 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d800b730 con 0x7fe5e00d8490 2021-04-09T18:23:10.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.763 7fe5df7fe700 1 -- 172.21.15.32:0/3390163093 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5d8000f90 con 0x7fe5e00d8490 2021-04-09T18:23:10.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.763 7fe5e69e2700 1 -- 172.21.15.32:0/3390163093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e00d8490 msgr2=0x7fe5e010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:10.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.763 7fe5e69e2700 1 --2- 172.21.15.32:0/3390163093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e00d8490 0x7fe5e010da00 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7fe5d8006f00 tx=0x7fe5d8009f10).stop 2021-04-09T18:23:10.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.764 7fe5e69e2700 1 -- 172.21.15.32:0/3390163093 shutdown_connections 2021-04-09T18:23:10.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.764 7fe5e69e2700 1 --2- 172.21.15.32:0/3390163093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e00d8490 0x7fe5e010da00 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.764 7fe5e69e2700 1 --2- 172.21.15.32:0/3390163093 >> v2:172.21.15.86:3300/0 conn(0x7fe5e010d010 0x7fe5e00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.764 7fe5e69e2700 1 -- 172.21.15.32:0/3390163093 >> 172.21.15.32:0/3390163093 conn(0x7fe5e00ce5f0 msgr2=0x7fe5e00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.764 7fe5e69e2700 1 -- 172.21.15.32:0/3390163093 shutdown_connections 2021-04-09T18:23:10.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.764 7fe5e69e2700 1 -- 172.21.15.32:0/3390163093 wait complete. 2021-04-09T18:23:10.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.784 7fe5e69e2700 1 Processor -- start 2021-04-09T18:23:10.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.784 7fe5e69e2700 1 -- start start 2021-04-09T18:23:10.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.784 7fe5e69e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5e00b1680 0x7fe5e00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:10.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.784 7fe5e69e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e009e330 0x7fe5e00b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:10.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.784 7fe5e69e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e00b2190 0x7fe5e009f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:10.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.784 7fe5e69e2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e009e730 con 0x7fe5e009e330 2021-04-09T18:23:10.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5e69e2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e009f6c0 con 0x7fe5e00b2190 2021-04-09T18:23:10.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5e69e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e009f8e0 con 0x7fe5e00b1680 2021-04-09T18:23:10.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5d7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e009e330 0x7fe5e00b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:10.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5d7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e009e330 0x7fe5e00b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50050/0 (socket says 172.21.15.32:50050) 2021-04-09T18:23:10.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5d7fff700 1 -- 172.21.15.32:0/1912845590 learned_addr learned my addr 172.21.15.32:0/1912845590 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:10.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5dffff700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5e00b1680 0x7fe5e00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:10.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5e4fff700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e00b2190 0x7fe5e009f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:10.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5d7fff700 1 -- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5e00b1680 msgr2=0x7fe5e00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:10.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5d7fff700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5e00b1680 0x7fe5e00b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5d7fff700 1 -- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e00b2190 msgr2=0x7fe5e009f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:10.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5d7fff700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e00b2190 0x7fe5e009f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:10.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5d7fff700 1 -- 172.21.15.32:0/1912845590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e009fb00 con 0x7fe5e009e330 2021-04-09T18:23:10.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5d7fff700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e009e330 0x7fe5e00b1b70 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7fe5c800b580 tx=0x7fe5c800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:10.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5dd7fa700 1 -- 172.21.15.32:0/1912845590 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5c800bd10 con 0x7fe5e009e330 2021-04-09T18:23:10.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5dd7fa700 1 -- 172.21.15.32:0/1912845590 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c8001080 con 0x7fe5e009e330 2021-04-09T18:23:10.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e009fb00 con 0x7fe5e009e330 2021-04-09T18:23:10.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5dd7fa700 1 -- 172.21.15.32:0/1912845590 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5c8001550 con 0x7fe5e009e330 2021-04-09T18:23:10.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.785 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e009fb00 con 0x7fe5e009e330 2021-04-09T18:23:10.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.786 7fe5dd7fa700 1 -- 172.21.15.32:0/1912845590 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe5c800a000 con 0x7fe5e009e330 2021-04-09T18:23:10.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.786 7fe5dd7fa700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5c40188c0 0x7fe5c401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:10.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.786 7fe5dd7fa700 1 -- 172.21.15.32:0/1912845590 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe5c8032830 con 0x7fe5e009e330 2021-04-09T18:23:10.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.786 7fe5dffff700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5c40188c0 0x7fe5c401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:10.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.787 7fe5dffff700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5c40188c0 0x7fe5c401adc0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:10.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.805 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e010d480 con 0x7fe5e009e330 2021-04-09T18:23:10.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:10.809 7fe5dd7fa700 1 -- 172.21.15.32:0/1912845590 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe5c8006bf0 con 0x7fe5e009e330 2021-04-09T18:23:11.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.003 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7fe5e010d480 con 0x7fe5e009e330 2021-04-09T18:23:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.539 7fe5dd7fa700 1 -- 172.21.15.32:0/1912845590 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v154) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe5c800a000 con 0x7fe5e009e330 2021-04-09T18:23:11.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5c40188c0 msgr2=0x7fe5c401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:11.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5c40188c0 0x7fe5c401adc0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e009e330 msgr2=0x7fe5e00b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e009e330 0x7fe5e00b1b70 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7fe5c800b580 tx=0x7fe5c800b920).stop 2021-04-09T18:23:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 shutdown_connections 2021-04-09T18:23:11.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5c40188c0 0x7fe5c401adc0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5e009e330 0x7fe5e00b1b70 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e00b2190 0x7fe5e009f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 --2- 172.21.15.32:0/1912845590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5e00b1680 0x7fe5e00b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 >> 172.21.15.32:0/1912845590 conn(0x7fe5e00c4290 msgr2=0x7fe5e00c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:11.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 shutdown_connections 2021-04-09T18:23:11.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.570 7fe5e69e2700 1 -- 172.21.15.32:0/1912845590 wait complete. 2021-04-09T18:23:11.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.733 7f71f0661700 1 Processor -- start 2021-04-09T18:23:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.733 7f71f0661700 1 -- start start 2021-04-09T18:23:11.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71f0661700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f71e810d010 0x7f71e80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:11.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71f0661700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80d1da0 0x7f71e80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:11.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71f0661700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71e80d82c0 0x7f71e810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:11.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71f0661700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e80d8d30 con 0x7f71e80d82c0 2021-04-09T18:23:11.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71f0661700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e80d9220 con 0x7f71e810d010 2021-04-09T18:23:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71f0661700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e810d480 con 0x7f71e80d1da0 2021-04-09T18:23:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71eec7e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71e80d82c0 0x7f71e810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71eec7e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71e80d82c0 0x7f71e810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50060/0 (socket says 172.21.15.32:50060) 2021-04-09T18:23:11.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71eec7e700 1 -- 172.21.15.32:0/1712651607 learned_addr learned my addr 172.21.15.32:0/1712651607 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71ee47d700 1 --2- 172.21.15.32:0/1712651607 >> v2:172.21.15.86:3300/0 conn(0x7f71e810d010 0x7f71e80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.734 7f71edc7c700 1 --2- 172.21.15.32:0/1712651607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80d1da0 0x7f71e80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71eec7e700 1 -- 172.21.15.32:0/1712651607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80d1da0 msgr2=0x7f71e80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:11.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71eec7e700 1 --2- 172.21.15.32:0/1712651607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80d1da0 0x7f71e80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71eec7e700 1 -- 172.21.15.32:0/1712651607 >> v2:172.21.15.86:3300/0 conn(0x7f71e810d010 msgr2=0x7f71e80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71eec7e700 1 --2- 172.21.15.32:0/1712651607 >> v2:172.21.15.86:3300/0 conn(0x7f71e810d010 0x7f71e80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71eec7e700 1 -- 172.21.15.32:0/1712651607 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e80d86c0 con 0x7f71e80d82c0 2021-04-09T18:23:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71eec7e700 1 --2- 172.21.15.32:0/1712651607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e80d82c0 0x7f71e810da00 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f71d8006b80 tx=0x7f71d8009b10).ready entity=mon.0 client_cookie=cba6c81d36a010c4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:11.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71ed47b700 1 -- 172.21.15.32:0/1712651607 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71d80022e0 con 0x7f71e80d82c0 2021-04-09T18:23:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71ed47b700 1 -- 172.21.15.32:0/1712651607 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d8002620 con 0x7f71e80d82c0 2021-04-09T18:23:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71ed47b700 1 -- 172.21.15.32:0/1712651607 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71d8002b00 con 0x7f71e80d82c0 2021-04-09T18:23:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71f0661700 1 -- 172.21.15.32:0/1712651607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e80d82c0 msgr2=0x7f71e810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.735 7f71f0661700 1 --2- 172.21.15.32:0/1712651607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e80d82c0 0x7f71e810da00 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f71d8006b80 tx=0x7f71d8009b10).stop 2021-04-09T18:23:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.736 7f71f0661700 1 -- 172.21.15.32:0/1712651607 shutdown_connections 2021-04-09T18:23:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.736 7f71f0661700 1 --2- 172.21.15.32:0/1712651607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e80d82c0 0x7f71e810da00 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.736 7f71f0661700 1 --2- 172.21.15.32:0/1712651607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80d1da0 0x7f71e80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.736 7f71f0661700 1 --2- 172.21.15.32:0/1712651607 >> v2:172.21.15.86:3300/0 conn(0x7f71e810d010 0x7f71e80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.736 7f71f0661700 1 -- 172.21.15.32:0/1712651607 >> 172.21.15.32:0/1712651607 conn(0x7f71e80ce5f0 msgr2=0x7f71e80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:11.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.736 7f71f0661700 1 -- 172.21.15.32:0/1712651607 shutdown_connections 2021-04-09T18:23:11.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.736 7f71f0661700 1 -- 172.21.15.32:0/1712651607 wait complete. 2021-04-09T18:23:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.756 7f71f0661700 1 Processor -- start 2021-04-09T18:23:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.756 7f71f0661700 1 -- start start 2021-04-09T18:23:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71f0661700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e81a0050 0x7f71e81a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71f0661700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71e80bdcd0 0x7f71e80bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71f0661700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80bf080 0x7f71e80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71f0661700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80bf500 con 0x7f71e81a0050 2021-04-09T18:23:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71f0661700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80bb7b0 con 0x7f71e80bdcd0 2021-04-09T18:23:11.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71f0661700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80be0d0 con 0x7f71e80bf080 2021-04-09T18:23:11.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71ee47d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e81a0050 0x7f71e81a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:11.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71ee47d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e81a0050 0x7f71e81a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50064/0 (socket says 172.21.15.32:50064) 2021-04-09T18:23:11.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71ee47d700 1 -- 172.21.15.32:0/967360909 learned_addr learned my addr 172.21.15.32:0/967360909 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:11.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71eec7e700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80bf080 0x7f71e80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:11.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71edc7c700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71e80bdcd0 0x7f71e80bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:11.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71ee47d700 1 -- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80bf080 msgr2=0x7f71e80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:11.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71ee47d700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80bf080 0x7f71e80b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71ee47d700 1 -- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71e80bdcd0 msgr2=0x7f71e80bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71ee47d700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71e80bdcd0 0x7f71e80bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:11.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71ee47d700 1 -- 172.21.15.32:0/967360909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e80b1310 con 0x7f71e81a0050 2021-04-09T18:23:11.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71ee47d700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e81a0050 0x7f71e81a0450 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7f71e000b930 tx=0x7f71e000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:11.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71e6ffd700 1 -- 172.21.15.32:0/967360909 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71e000c2f0 con 0x7f71e81a0050 2021-04-09T18:23:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71e6ffd700 1 -- 172.21.15.32:0/967360909 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e000c5f0 con 0x7f71e81a0050 2021-04-09T18:23:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71f0661700 1 -- 172.21.15.32:0/967360909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e80b1310 con 0x7f71e81a0050 2021-04-09T18:23:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71e6ffd700 1 -- 172.21.15.32:0/967360909 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71e000ca40 con 0x7f71e81a0050 2021-04-09T18:23:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.757 7f71f0661700 1 -- 172.21.15.32:0/967360909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e80b1310 con 0x7f71e81a0050 2021-04-09T18:23:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.758 7f71e6ffd700 1 -- 172.21.15.32:0/967360909 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f71e0001010 con 0x7f71e81a0050 2021-04-09T18:23:11.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.758 7f71e6ffd700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71c80188d0 0x7f71c801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:11.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.758 7f71e6ffd700 1 -- 172.21.15.32:0/967360909 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f71e0032df0 con 0x7f71e81a0050 2021-04-09T18:23:11.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.758 7f71edc7c700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71c80188d0 0x7f71c801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:11.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.759 7f71edc7c700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71c80188d0 0x7f71c801add0 crc :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:11.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.778 7f71f0661700 1 -- 172.21.15.32:0/967360909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e80bb7b0 con 0x7f71e81a0050 2021-04-09T18:23:11.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.781 7f71e6ffd700 1 -- 172.21.15.32:0/967360909 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f71e000be10 con 0x7f71e81a0050 2021-04-09T18:23:12.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:11.974 7f71f0661700 1 -- 172.21.15.32:0/967360909 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f71e80bb7b0 con 0x7f71e81a0050 2021-04-09T18:23:12.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.543 7f71e6ffd700 1 -- 172.21.15.32:0/967360909 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v155) v1 ==== 105+0+0 (secure 0 0 0) 0x7f71e0001010 con 0x7f71e81a0050 2021-04-09T18:23:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 -- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71c80188d0 msgr2=0x7f71c801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71c80188d0 0x7f71c801add0 crc :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 -- 172.21.15.32:0/967360909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e81a0050 msgr2=0x7f71e81a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e81a0050 0x7f71e81a0450 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7f71e000b930 tx=0x7f71e000bcd0).stop 2021-04-09T18:23:12.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 -- 172.21.15.32:0/967360909 shutdown_connections 2021-04-09T18:23:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f71c80188d0 0x7f71c801add0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71e80bf080 0x7f71e80b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71e80bdcd0 0x7f71e80bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 --2- 172.21.15.32:0/967360909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71e81a0050 0x7f71e81a0450 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 -- 172.21.15.32:0/967360909 >> 172.21.15.32:0/967360909 conn(0x7f71e80975f0 msgr2=0x7f71e8096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 -- 172.21.15.32:0/967360909 shutdown_connections 2021-04-09T18:23:12.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.592 7f71f0661700 1 -- 172.21.15.32:0/967360909 wait complete. 2021-04-09T18:23:12.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:12.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.762 7f9098b22700 1 Processor -- start 2021-04-09T18:23:12.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.762 7f9098b22700 1 -- start start 2021-04-09T18:23:12.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.762 7f9098b22700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f90940d7ca0 0x7f90940d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:12.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.762 7f9098b22700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909410da00 0x7f909410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:12.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.762 7f9098b22700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909410d480 con 0x7f909410d010 2021-04-09T18:23:12.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.762 7f9098b22700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90940d9220 con 0x7f90940d7ca0 2021-04-09T18:23:12.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.762 7f9098b22700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90940d86c0 con 0x7f909410da00 2021-04-09T18:23:12.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909305f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909410da00 0x7f909410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909205d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f90940d7ca0 0x7f90940d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909305f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909410da00 0x7f909410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:57300/0 (socket says 172.21.15.32:57300) 2021-04-09T18:23:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909305f700 1 -- 172.21.15.32:0/3628425563 learned_addr learned my addr 172.21.15.32:0/3628425563 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909305f700 1 -- 172.21.15.32:0/3628425563 >> v2:172.21.15.86:3300/0 conn(0x7f90940d7ca0 msgr2=0x7f90940d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909305f700 1 --2- 172.21.15.32:0/3628425563 >> v2:172.21.15.86:3300/0 conn(0x7f90940d7ca0 0x7f90940d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909305f700 1 -- 172.21.15.32:0/3628425563 >> v1:172.21.15.86:6789/0 conn(0x7f909410d010 legacy=0x7f90940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909305f700 1 -- 172.21.15.32:0/3628425563 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940d1f70 con 0x7f909410da00 2021-04-09T18:23:12.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909205d700 1 --2- 172.21.15.32:0/3628425563 >> v2:172.21.15.86:3300/0 conn(0x7f90940d7ca0 0x7f90940d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909305f700 1 --2- 172.21.15.32:0/3628425563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909410da00 0x7f909410de00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f908c006aa0 tx=0x7f908c009f20).ready entity=mon.2 client_cookie=7507a0dc3b871587 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.763 7f909185c700 1 -- 172.21.15.32:0/3628425563 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f908c00b5c0 con 0x7f909410da00 2021-04-09T18:23:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.764 7f909185c700 1 -- 172.21.15.32:0/3628425563 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908c003030 con 0x7f909410da00 2021-04-09T18:23:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.764 7f909185c700 1 -- 172.21.15.32:0/3628425563 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f908c00bb40 con 0x7f909410da00 2021-04-09T18:23:12.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.764 7f9098b22700 1 -- 172.21.15.32:0/3628425563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909410da00 msgr2=0x7f909410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.764 7f9098b22700 1 --2- 172.21.15.32:0/3628425563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909410da00 0x7f909410de00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f908c006aa0 tx=0x7f908c009f20).stop 2021-04-09T18:23:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.764 7f9098b22700 1 -- 172.21.15.32:0/3628425563 shutdown_connections 2021-04-09T18:23:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.764 7f9098b22700 1 --2- 172.21.15.32:0/3628425563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909410da00 0x7f909410de00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.764 7f9098b22700 1 --2- 172.21.15.32:0/3628425563 >> v2:172.21.15.86:3300/0 conn(0x7f90940d7ca0 0x7f90940d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.764 7f9098b22700 1 -- 172.21.15.32:0/3628425563 >> 172.21.15.32:0/3628425563 conn(0x7f90940ce5f0 msgr2=0x7f90940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:12.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.764 7f9098b22700 1 -- 172.21.15.32:0/3628425563 shutdown_connections 2021-04-09T18:23:12.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.765 7f9098b22700 1 -- 172.21.15.32:0/3628425563 wait complete. 2021-04-09T18:23:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.784 7f9098b22700 1 Processor -- start 2021-04-09T18:23:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f9098b22700 1 -- start start 2021-04-09T18:23:12.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f9098b22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90941a0050 0x7f90941a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f9098b22700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90940bdcd0 0x7f90940bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f9098b22700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90940bf080 0x7f90940b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:12.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f9098b22700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90940bf500 con 0x7f90941a0050 2021-04-09T18:23:12.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f9098b22700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90940bb7b0 con 0x7f90940bdcd0 2021-04-09T18:23:12.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f9098b22700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90940be0d0 con 0x7f90940bf080 2021-04-09T18:23:12.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f909285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90941a0050 0x7f90941a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:12.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f909285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90941a0050 0x7f90941a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50082/0 (socket says 172.21.15.32:50082) 2021-04-09T18:23:12.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f909285e700 1 -- 172.21.15.32:0/1173529509 learned_addr learned my addr 172.21.15.32:0/1173529509 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:12.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f909205d700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90940bdcd0 0x7f90940bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:12.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.785 7f909305f700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90940bf080 0x7f90940b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:12.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f909285e700 1 -- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90940bf080 msgr2=0x7f90940b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:12.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f909285e700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90940bf080 0x7f90940b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f909285e700 1 -- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90940bdcd0 msgr2=0x7f90940bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:12.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f909285e700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90940bdcd0 0x7f90940bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:12.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f909285e700 1 -- 172.21.15.32:0/1173529509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940b1310 con 0x7f90941a0050 2021-04-09T18:23:12.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f909285e700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90941a0050 0x7f90941a0450 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7f908400b5b0 tx=0x7f908400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:12.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f90837fe700 1 -- 172.21.15.32:0/1173529509 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f908400bd40 con 0x7f90941a0050 2021-04-09T18:23:12.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f90837fe700 1 -- 172.21.15.32:0/1173529509 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084006240 con 0x7f90941a0050 2021-04-09T18:23:12.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f9098b22700 1 -- 172.21.15.32:0/1173529509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90940b1310 con 0x7f90941a0050 2021-04-09T18:23:12.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f90837fe700 1 -- 172.21.15.32:0/1173529509 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9084005690 con 0x7f90941a0050 2021-04-09T18:23:12.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.786 7f9098b22700 1 -- 172.21.15.32:0/1173529509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90940b1310 con 0x7f90941a0050 2021-04-09T18:23:12.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.787 7f90837fe700 1 -- 172.21.15.32:0/1173529509 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f90840054a0 con 0x7f90941a0050 2021-04-09T18:23:12.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.787 7f90837fe700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9078018930 0x7f907801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:12.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.787 7f90837fe700 1 -- 172.21.15.32:0/1173529509 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9084032a30 con 0x7f90941a0050 2021-04-09T18:23:12.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.787 7f909205d700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9078018930 0x7f907801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:12.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.787 7f909205d700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9078018930 0x7f907801ae10 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:12.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.806 7f9098b22700 1 -- 172.21.15.32:0/1173529509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90940bb7b0 con 0x7f90941a0050 2021-04-09T18:23:12.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:12.810 7f90837fe700 1 -- 172.21.15.32:0/1173529509 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9084000c60 con 0x7f90941a0050 2021-04-09T18:23:13.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.002 7f9098b22700 1 -- 172.21.15.32:0/1173529509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f90940bb7b0 con 0x7f90941a0050 2021-04-09T18:23:13.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.544 7f90837fe700 1 -- 172.21.15.32:0/1173529509 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v156) v1 ==== 105+0+0 (secure 0 0 0) 0x7f90840054a0 con 0x7f90941a0050 2021-04-09T18:23:13.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 -- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9078018930 msgr2=0x7f907801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:13.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9078018930 0x7f907801ae10 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:13.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 -- 172.21.15.32:0/1173529509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90941a0050 msgr2=0x7f90941a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:13.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90941a0050 0x7f90941a0450 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7f908400b5b0 tx=0x7f908400b950).stop 2021-04-09T18:23:13.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 -- 172.21.15.32:0/1173529509 shutdown_connections 2021-04-09T18:23:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9078018930 0x7f907801ae10 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90940bf080 0x7f90940b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90940bdcd0 0x7f90940bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:13.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 --2- 172.21.15.32:0/1173529509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90941a0050 0x7f90941a0450 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 -- 172.21.15.32:0/1173529509 >> 172.21.15.32:0/1173529509 conn(0x7f90940975f0 msgr2=0x7f9094096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 -- 172.21.15.32:0/1173529509 shutdown_connections 2021-04-09T18:23:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.569 7f9098b22700 1 -- 172.21.15.32:0/1173529509 wait complete. 2021-04-09T18:23:13.636 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.733 7f6578466700 1 Processor -- start 2021-04-09T18:23:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.733 7f6578466700 1 -- start start 2021-04-09T18:23:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.734 7f6578466700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f657010d010 0x7f65700d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.734 7f6578466700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657010d480 con 0x7f65700d1da0 2021-04-09T18:23:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.734 7f6578466700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657010e1f0 con 0x7f65700d8490 2021-04-09T18:23:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.734 7f6578466700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65700d8d30 con 0x7f657010d010 2021-04-09T18:23:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.734 7f6576282700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f657010d010 0x7f65700d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.734 7f6575a81700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f65700d1da0 0x7f65700d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59680/0 (socket says 172.21.15.32:59680) 2021-04-09T18:23:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.734 7f6575a81700 1 -- 172.21.15.32:0/4246591704 learned_addr learned my addr 172.21.15.32:0/4246591704 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.734 7f6576282700 1 -- 172.21.15.32:0/4246591704 >> v1:172.21.15.86:6789/0 conn(0x7f65700d8490 legacy=0x7f657010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:13.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.735 7f6576282700 1 -- 172.21.15.32:0/4246591704 >> v1:172.21.15.32:6789/0 conn(0x7f65700d1da0 legacy=0x7f65700d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:13.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.735 7f6575280700 1 -- 172.21.15.32:0/4246591704 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984050458 0 0) 0x7f6560000ce0 con 0x7f65700d1da0 2021-04-09T18:23:13.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.735 7f6576282700 1 -- 172.21.15.32:0/4246591704 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65700d9220 con 0x7f657010d010 2021-04-09T18:23:13.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.735 7f6576282700 1 --2- 172.21.15.32:0/4246591704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f657010d010 0x7f65700d1780 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f656c006650 tx=0x7f656c009a80).ready entity=mon.0 client_cookie=c44f653fdfc61cb5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.735 7f6575280700 1 -- 172.21.15.32:0/4246591704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f656c0009b0 con 0x7f657010d010 2021-04-09T18:23:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.735 7f6575280700 1 -- 172.21.15.32:0/4246591704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c00c0c0 con 0x7f657010d010 2021-04-09T18:23:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.735 7f6575280700 1 -- 172.21.15.32:0/4246591704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f656c00c510 con 0x7f657010d010 2021-04-09T18:23:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.735 7f6578466700 1 -- 172.21.15.32:0/4246591704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f657010d010 msgr2=0x7f65700d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.735 7f6578466700 1 --2- 172.21.15.32:0/4246591704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f657010d010 0x7f65700d1780 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f656c006650 tx=0x7f656c009a80).stop 2021-04-09T18:23:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.736 7f6578466700 1 -- 172.21.15.32:0/4246591704 shutdown_connections 2021-04-09T18:23:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.736 7f6578466700 1 --2- 172.21.15.32:0/4246591704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f657010d010 0x7f65700d1780 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.736 7f6578466700 1 -- 172.21.15.32:0/4246591704 >> 172.21.15.32:0/4246591704 conn(0x7f65700ce5f0 msgr2=0x7f65700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.736 7f6578466700 1 -- 172.21.15.32:0/4246591704 shutdown_connections 2021-04-09T18:23:13.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.736 7f6578466700 1 -- 172.21.15.32:0/4246591704 wait complete. 2021-04-09T18:23:13.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.756 7f6578466700 1 Processor -- start 2021-04-09T18:23:13.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.756 7f6578466700 1 -- start start 2021-04-09T18:23:13.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.756 7f6578466700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65700b84a0 0x7f65700b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:13.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.756 7f6578466700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65700b4610 0x7f65700b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:13.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6578466700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6570199b50 0x7f6570199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:13.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6578466700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65700b8920 con 0x7f65700b84a0 2021-04-09T18:23:13.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6578466700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65700b40c0 con 0x7f6570199b50 2021-04-09T18:23:13.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6578466700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65700b5030 con 0x7f65700b4610 2021-04-09T18:23:13.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576282700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65700b84a0 0x7f65700b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:13.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576282700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65700b84a0 0x7f65700b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50094/0 (socket says 172.21.15.32:50094) 2021-04-09T18:23:13.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576282700 1 -- 172.21.15.32:0/1426812430 learned_addr learned my addr 172.21.15.32:0/1426812430 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:13.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6575a81700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65700b4610 0x7f65700b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:13.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576a83700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6570199b50 0x7f6570199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:13.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576282700 1 -- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65700b4610 msgr2=0x7f65700b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576282700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65700b4610 0x7f65700b4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576282700 1 -- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6570199b50 msgr2=0x7f6570199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576282700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6570199b50 0x7f6570199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576282700 1 -- 172.21.15.32:0/1426812430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657019a570 con 0x7f65700b84a0 2021-04-09T18:23:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576a83700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6570199b50 0x7f6570199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6576282700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65700b84a0 0x7f65700b3880 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7f656c009650 tx=0x7f656c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6566ffd700 1 -- 172.21.15.32:0/1426812430 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f656c0093a0 con 0x7f65700b84a0 2021-04-09T18:23:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6566ffd700 1 -- 172.21.15.32:0/1426812430 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c006070 con 0x7f65700b84a0 2021-04-09T18:23:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6566ffd700 1 -- 172.21.15.32:0/1426812430 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f656c0052e0 con 0x7f65700b84a0 2021-04-09T18:23:13.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6578466700 1 -- 172.21.15.32:0/1426812430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657019a570 con 0x7f65700b84a0 2021-04-09T18:23:13.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.757 7f6578466700 1 -- 172.21.15.32:0/1426812430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657019a570 con 0x7f65700b84a0 2021-04-09T18:23:13.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.759 7f6566ffd700 1 -- 172.21.15.32:0/1426812430 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f656c009300 con 0x7f65700b84a0 2021-04-09T18:23:13.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.759 7f6566ffd700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65580188c0 0x7f655801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:13.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.759 7f6566ffd700 1 -- 172.21.15.32:0/1426812430 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f656c0327b0 con 0x7f65700b84a0 2021-04-09T18:23:13.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.759 7f6575a81700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65580188c0 0x7f655801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:13.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.759 7f6575a81700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65580188c0 0x7f655801adc0 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:13.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.778 7f6578466700 1 -- 172.21.15.32:0/1426812430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65700b5030 con 0x7f65700b84a0 2021-04-09T18:23:13.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.781 7f6566ffd700 1 -- 172.21.15.32:0/1426812430 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f656c0009a0 con 0x7f65700b84a0 2021-04-09T18:23:14.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:13.974 7f6578466700 1 -- 172.21.15.32:0/1426812430 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f65700b5030 con 0x7f65700b84a0 2021-04-09T18:23:14.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.551 7f6566ffd700 1 -- 172.21.15.32:0/1426812430 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v157) v1 ==== 105+0+0 (secure 0 0 0) 0x7f656c009300 con 0x7f65700b84a0 2021-04-09T18:23:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 -- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65580188c0 msgr2=0x7f655801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65580188c0 0x7f655801adc0 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 -- 172.21.15.32:0/1426812430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65700b84a0 msgr2=0x7f65700b3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:14.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65700b84a0 0x7f65700b3880 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7f656c009650 tx=0x7f656c00b8e0).stop 2021-04-09T18:23:14.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 -- 172.21.15.32:0/1426812430 shutdown_connections 2021-04-09T18:23:14.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65580188c0 0x7f655801adc0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6570199b50 0x7f6570199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65700b4610 0x7f65700b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 --2- 172.21.15.32:0/1426812430 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65700b84a0 0x7f65700b3880 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 -- 172.21.15.32:0/1426812430 >> 172.21.15.32:0/1426812430 conn(0x7f65700aeb60 msgr2=0x7f657008dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:14.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 -- 172.21.15.32:0/1426812430 shutdown_connections 2021-04-09T18:23:14.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.592 7f6578466700 1 -- 172.21.15.32:0/1426812430 wait complete. 2021-04-09T18:23:14.660 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:14.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.762 7fa81fc97700 1 Processor -- start 2021-04-09T18:23:14.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81fc97700 1 -- start start 2021-04-09T18:23:14.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81fc97700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180d7ca0 0x7fa8180d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81fc97700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa81810da00 0x7fa81810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81fc97700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81810d480 con 0x7fa81810d010 2021-04-09T18:23:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81fc97700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8180d9220 con 0x7fa81810da00 2021-04-09T18:23:14.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81fc97700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8180d86c0 con 0x7fa8180d7ca0 2021-04-09T18:23:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81e2b4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa81810da00 0x7fa81810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81dab3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa81810d010 0x7fa8180d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59690/0 (socket says 172.21.15.32:59690) 2021-04-09T18:23:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81dab3700 1 -- 172.21.15.32:0/1447933204 learned_addr learned my addr 172.21.15.32:0/1447933204 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:14.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.763 7fa81d2b2700 1 --2- 172.21.15.32:0/1447933204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180d7ca0 0x7fa8180d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81e2b4700 1 -- 172.21.15.32:0/1447933204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180d7ca0 msgr2=0x7fa8180d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81e2b4700 1 --2- 172.21.15.32:0/1447933204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180d7ca0 0x7fa8180d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81e2b4700 1 -- 172.21.15.32:0/1447933204 >> v1:172.21.15.32:6789/0 conn(0x7fa81810d010 legacy=0x7fa8180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81e2b4700 1 -- 172.21.15.32:0/1447933204 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180d1f70 con 0x7fa81810da00 2021-04-09T18:23:14.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81cab1700 1 -- 172.21.15.32:0/1447933204 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877610479 0 0) 0x7fa814000ce0 con 0x7fa81810d010 2021-04-09T18:23:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81e2b4700 1 --2- 172.21.15.32:0/1447933204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81810da00 0x7fa81810de00 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7fa810006e30 tx=0x7fa810009e60).ready entity=mon.0 client_cookie=1483d56ee453d5c8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81cab1700 1 -- 172.21.15.32:0/1447933204 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa810006a50 con 0x7fa81810da00 2021-04-09T18:23:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81cab1700 1 -- 172.21.15.32:0/1447933204 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81000b300 con 0x7fa81810da00 2021-04-09T18:23:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81cab1700 1 -- 172.21.15.32:0/1447933204 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa81000b750 con 0x7fa81810da00 2021-04-09T18:23:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81fc97700 1 -- 172.21.15.32:0/1447933204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81810da00 msgr2=0x7fa81810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.764 7fa81fc97700 1 --2- 172.21.15.32:0/1447933204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81810da00 0x7fa81810de00 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7fa810006e30 tx=0x7fa810009e60).stop 2021-04-09T18:23:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.765 7fa81fc97700 1 -- 172.21.15.32:0/1447933204 shutdown_connections 2021-04-09T18:23:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.765 7fa81fc97700 1 --2- 172.21.15.32:0/1447933204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81810da00 0x7fa81810de00 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.765 7fa81fc97700 1 --2- 172.21.15.32:0/1447933204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180d7ca0 0x7fa8180d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.765 7fa81fc97700 1 -- 172.21.15.32:0/1447933204 >> 172.21.15.32:0/1447933204 conn(0x7fa8180ce5f0 msgr2=0x7fa8180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.765 7fa81fc97700 1 -- 172.21.15.32:0/1447933204 shutdown_connections 2021-04-09T18:23:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.765 7fa81fc97700 1 -- 172.21.15.32:0/1447933204 wait complete. 2021-04-09T18:23:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.785 7fa81fc97700 1 Processor -- start 2021-04-09T18:23:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.785 7fa81fc97700 1 -- start start 2021-04-09T18:23:14.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81fc97700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180b1680 0x7fa8180b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:14.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81fc97700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81809e330 0x7fa8180b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:14.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81fc97700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8180b2190 0x7fa81809f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81fc97700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81809e730 con 0x7fa81809e330 2021-04-09T18:23:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81fc97700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81809f6c0 con 0x7fa8180b2190 2021-04-09T18:23:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81fc97700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81809f8e0 con 0x7fa8180b1680 2021-04-09T18:23:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81d2b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81809e330 0x7fa8180b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81d2b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81809e330 0x7fa8180b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50104/0 (socket says 172.21.15.32:50104) 2021-04-09T18:23:14.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81d2b2700 1 -- 172.21.15.32:0/1699146922 learned_addr learned my addr 172.21.15.32:0/1699146922 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:14.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81e2b4700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8180b2190 0x7fa81809f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:14.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81dab3700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180b1680 0x7fa8180b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:14.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81d2b2700 1 -- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180b1680 msgr2=0x7fa8180b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:14.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81d2b2700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180b1680 0x7fa8180b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81d2b2700 1 -- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8180b2190 msgr2=0x7fa81809f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:14.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81d2b2700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8180b2190 0x7fa81809f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:14.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81d2b2700 1 -- 172.21.15.32:0/1699146922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81809fb00 con 0x7fa81809e330 2021-04-09T18:23:14.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81e2b4700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8180b2190 0x7fa81809f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:14.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81d2b2700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81809e330 0x7fa8180b1b70 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7fa808006f60 tx=0x7fa80800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:14.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa80e7fc700 1 -- 172.21.15.32:0/1699146922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa80800bd60 con 0x7fa81809e330 2021-04-09T18:23:14.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa80e7fc700 1 -- 172.21.15.32:0/1699146922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa808000c90 con 0x7fa81809e330 2021-04-09T18:23:14.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81809fb00 con 0x7fa81809e330 2021-04-09T18:23:14.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.786 7fa80e7fc700 1 -- 172.21.15.32:0/1699146922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa808002220 con 0x7fa81809e330 2021-04-09T18:23:14.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.787 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81809fb00 con 0x7fa81809e330 2021-04-09T18:23:14.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.787 7fa80e7fc700 1 -- 172.21.15.32:0/1699146922 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa8080052e0 con 0x7fa81809e330 2021-04-09T18:23:14.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.787 7fa80e7fc700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa800018930 0x7fa80001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:14.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.787 7fa80e7fc700 1 -- 172.21.15.32:0/1699146922 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa8080329f0 con 0x7fa81809e330 2021-04-09T18:23:14.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.787 7fa81dab3700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa800018930 0x7fa80001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:14.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.788 7fa81dab3700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa800018930 0x7fa80001ae10 crc :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:14.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.807 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81809eb70 con 0x7fa81809e330 2021-04-09T18:23:14.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:14.811 7fa80e7fc700 1 -- 172.21.15.32:0/1699146922 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa808000c90 con 0x7fa81809e330 2021-04-09T18:23:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.004 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fa81809eb70 con 0x7fa81809e330 2021-04-09T18:23:15.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.553 7fa80e7fc700 1 -- 172.21.15.32:0/1699146922 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v158) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa8080052e0 con 0x7fa81809e330 2021-04-09T18:23:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa800018930 msgr2=0x7fa80001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:15.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa800018930 0x7fa80001ae10 crc :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81809e330 msgr2=0x7fa8180b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:15.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81809e330 0x7fa8180b1b70 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7fa808006f60 tx=0x7fa80800b970).stop 2021-04-09T18:23:15.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 shutdown_connections 2021-04-09T18:23:15.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa800018930 0x7fa80001ae10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81809e330 0x7fa8180b1b70 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8180b2190 0x7fa81809f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 --2- 172.21.15.32:0/1699146922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8180b1680 0x7fa8180b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 >> 172.21.15.32:0/1699146922 conn(0x7fa8180c4290 msgr2=0x7fa8180c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:15.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 shutdown_connections 2021-04-09T18:23:15.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.572 7fa81fc97700 1 -- 172.21.15.32:0/1699146922 wait complete. 2021-04-09T18:23:15.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:15.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.739 7f8991903700 1 Processor -- start 2021-04-09T18:23:15.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.739 7f8991903700 1 -- start start 2021-04-09T18:23:15.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f8991903700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f898c0d7ca0 0x7f898c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:15.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f8991903700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f898c10da00 0x7f898c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f8991903700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c10d480 con 0x7f898c10d010 2021-04-09T18:23:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f8991903700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f898c0d9220 con 0x7f898c10da00 2021-04-09T18:23:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f8991903700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f898c0d86c0 con 0x7f898c0d7ca0 2021-04-09T18:23:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f898bd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f898c10da00 0x7f898c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:15.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f898b574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f898c10d010 0x7f898c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59702/0 (socket says 172.21.15.32:59702) 2021-04-09T18:23:15.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f898b574700 1 -- 172.21.15.32:0/708393757 learned_addr learned my addr 172.21.15.32:0/708393757 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:15.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f898bd75700 1 --2- 172.21.15.32:0/708393757 >> v2:172.21.15.32:3300/0 conn(0x7f898c10da00 0x7f898c10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50114/0 (socket says 172.21.15.32:50114) 2021-04-09T18:23:15.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.740 7f898ad73700 1 --2- 172.21.15.32:0/708393757 >> v2:172.21.15.86:3300/0 conn(0x7f898c0d7ca0 0x7f898c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:15.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f898bd75700 1 -- 172.21.15.32:0/708393757 >> v2:172.21.15.86:3300/0 conn(0x7f898c0d7ca0 msgr2=0x7f898c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f898bd75700 1 --2- 172.21.15.32:0/708393757 >> v2:172.21.15.86:3300/0 conn(0x7f898c0d7ca0 0x7f898c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f898bd75700 1 -- 172.21.15.32:0/708393757 >> v1:172.21.15.32:6789/0 conn(0x7f898c10d010 legacy=0x7f898c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f898a572700 1 -- 172.21.15.32:0/708393757 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270027441 0 0) 0x7f897c000ce0 con 0x7f898c10d010 2021-04-09T18:23:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f898bd75700 1 -- 172.21.15.32:0/708393757 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c0d1f70 con 0x7f898c10da00 2021-04-09T18:23:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f898bd75700 1 --2- 172.21.15.32:0/708393757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c10da00 0x7f898c10de00 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f8984006e30 tx=0x7f8984009e60).ready entity=mon.0 client_cookie=abe835bb04a0f694 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:15.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f898a572700 1 -- 172.21.15.32:0/708393757 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8984006a50 con 0x7f898c10da00 2021-04-09T18:23:15.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f898a572700 1 -- 172.21.15.32:0/708393757 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898400b300 con 0x7f898c10da00 2021-04-09T18:23:15.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f898a572700 1 -- 172.21.15.32:0/708393757 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f898400b750 con 0x7f898c10da00 2021-04-09T18:23:15.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f8991903700 1 -- 172.21.15.32:0/708393757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c10da00 msgr2=0x7f898c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:15.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.741 7f8991903700 1 --2- 172.21.15.32:0/708393757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c10da00 0x7f898c10de00 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f8984006e30 tx=0x7f8984009e60).stop 2021-04-09T18:23:15.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.742 7f8991903700 1 -- 172.21.15.32:0/708393757 shutdown_connections 2021-04-09T18:23:15.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.742 7f8991903700 1 --2- 172.21.15.32:0/708393757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c10da00 0x7f898c10de00 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.742 7f8991903700 1 --2- 172.21.15.32:0/708393757 >> v2:172.21.15.86:3300/0 conn(0x7f898c0d7ca0 0x7f898c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.742 7f8991903700 1 -- 172.21.15.32:0/708393757 >> 172.21.15.32:0/708393757 conn(0x7f898c0ce5f0 msgr2=0x7f898c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:15.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.742 7f8991903700 1 -- 172.21.15.32:0/708393757 shutdown_connections 2021-04-09T18:23:15.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.742 7f8991903700 1 -- 172.21.15.32:0/708393757 wait complete. 2021-04-09T18:23:15.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.762 7f8991903700 1 Processor -- start 2021-04-09T18:23:15.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.762 7f8991903700 1 -- start start 2021-04-09T18:23:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.762 7f8991903700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c09e530 0x7f898c09b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.762 7f8991903700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898c09e930 0x7f898c09f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:15.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.762 7f8991903700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c09f990 0x7f898c094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.762 7f8991903700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c09edf0 con 0x7f898c09e530 2021-04-09T18:23:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.762 7f8991903700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c09fd90 con 0x7f898c09f990 2021-04-09T18:23:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.762 7f8991903700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898c09bd40 con 0x7f898c09e930 2021-04-09T18:23:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c09e530 0x7f898c09b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:15.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c09e530 0x7f898c09b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50118/0 (socket says 172.21.15.32:50118) 2021-04-09T18:23:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898b574700 1 -- 172.21.15.32:0/902932186 learned_addr learned my addr 172.21.15.32:0/902932186 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898bd75700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c09f990 0x7f898c094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898ad73700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898c09e930 0x7f898c09f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898b574700 1 -- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898c09e930 msgr2=0x7f898c09f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:15.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898b574700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898c09e930 0x7f898c09f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898b574700 1 -- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c09f990 msgr2=0x7f898c094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:15.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898b574700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c09f990 0x7f898c094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:15.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898b574700 1 -- 172.21.15.32:0/902932186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c095290 con 0x7f898c09e530 2021-04-09T18:23:15.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f898b574700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c09e530 0x7f898c09b720 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f897c006f10 tx=0x7f897c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f8983fff700 1 -- 172.21.15.32:0/902932186 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f897c006020 con 0x7f898c09e530 2021-04-09T18:23:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f8983fff700 1 -- 172.21.15.32:0/902932186 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897c006360 con 0x7f898c09e530 2021-04-09T18:23:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f8991903700 1 -- 172.21.15.32:0/902932186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898c095290 con 0x7f898c09e530 2021-04-09T18:23:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f8983fff700 1 -- 172.21.15.32:0/902932186 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f897c004590 con 0x7f898c09e530 2021-04-09T18:23:15.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.763 7f8991903700 1 -- 172.21.15.32:0/902932186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c095290 con 0x7f898c09e530 2021-04-09T18:23:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.764 7f8983fff700 1 -- 172.21.15.32:0/902932186 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f897c009fe0 con 0x7f898c09e530 2021-04-09T18:23:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.764 7f8983fff700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8968018930 0x7f896801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.764 7f8983fff700 1 -- 172.21.15.32:0/902932186 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f897c032a30 con 0x7f898c09e530 2021-04-09T18:23:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.764 7f898ad73700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8968018930 0x7f896801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:15.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.764 7f898ad73700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8968018930 0x7f896801ae10 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:15.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.784 7f8991903700 1 -- 172.21.15.32:0/902932186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898c095b10 con 0x7f898c09e530 2021-04-09T18:23:15.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.788 7f8983fff700 1 -- 172.21.15.32:0/902932186 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f897c006ba0 con 0x7f898c09e530 2021-04-09T18:23:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:15.981 7f8991903700 1 -- 172.21.15.32:0/902932186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f898c095b10 con 0x7f898c09e530 2021-04-09T18:23:16.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.560 7f8983fff700 1 -- 172.21.15.32:0/902932186 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v159) v1 ==== 105+0+0 (secure 0 0 0) 0x7f897c009fe0 con 0x7f898c09e530 2021-04-09T18:23:16.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 -- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8968018930 msgr2=0x7f896801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:16.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8968018930 0x7f896801ae10 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 -- 172.21.15.32:0/902932186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c09e530 msgr2=0x7f898c09b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:16.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c09e530 0x7f898c09b720 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f897c006f10 tx=0x7f897c00b990).stop 2021-04-09T18:23:16.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 -- 172.21.15.32:0/902932186 shutdown_connections 2021-04-09T18:23:16.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8968018930 0x7f896801ae10 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c09f990 0x7f898c094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898c09e930 0x7f898c09f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 --2- 172.21.15.32:0/902932186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f898c09e530 0x7f898c09b720 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 -- 172.21.15.32:0/902932186 >> 172.21.15.32:0/902932186 conn(0x7f898c0c4290 msgr2=0x7f898c0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 -- 172.21.15.32:0/902932186 shutdown_connections 2021-04-09T18:23:16.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.599 7f8991903700 1 -- 172.21.15.32:0/902932186 wait complete. 2021-04-09T18:23:16.667 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.768 7f4c58f36700 1 Processor -- start 2021-04-09T18:23:16.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.769 7f4c58f36700 1 -- start start 2021-04-09T18:23:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.769 7f4c58f36700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5410d010 0x7f4c540d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.769 7f4c58f36700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4c540d8490 0x7f4c5410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.769 7f4c58f36700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5410d480 con 0x7f4c540d1da0 2021-04-09T18:23:16.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.769 7f4c58f36700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c5410e780 con 0x7f4c540d8490 2021-04-09T18:23:16.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.769 7f4c58f36700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540d8d30 con 0x7f4c5410d010 2021-04-09T18:23:16.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c52b13700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5410d010 0x7f4c540d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:16.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c53314700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4c540d8490 0x7f4c5410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:16.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c52b13700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5410d010 0x7f4c540d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:57352/0 (socket says 172.21.15.32:57352) 2021-04-09T18:23:16.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c52b13700 1 -- 172.21.15.32:0/2298389890 learned_addr learned my addr 172.21.15.32:0/2298389890 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:16.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c53314700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4c540d8490 0x7f4c5410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:41886/0 (socket says 172.21.15.32:41886) 2021-04-09T18:23:16.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c52b13700 1 -- 172.21.15.32:0/2298389890 >> v2:172.21.15.86:3300/0 conn(0x7f4c540d8490 msgr2=0x7f4c5410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:16.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c52b13700 1 --2- 172.21.15.32:0/2298389890 >> v2:172.21.15.86:3300/0 conn(0x7f4c540d8490 0x7f4c5410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c52b13700 1 -- 172.21.15.32:0/2298389890 >> v1:172.21.15.86:6789/0 conn(0x7f4c540d1da0 legacy=0x7f4c540d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:16.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c52b13700 1 -- 172.21.15.32:0/2298389890 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540d9220 con 0x7f4c5410d010 2021-04-09T18:23:16.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c53314700 1 --2- 172.21.15.32:0/2298389890 >> v2:172.21.15.86:3300/0 conn(0x7f4c540d8490 0x7f4c5410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:16.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c51b11700 1 -- 172.21.15.32:0/2298389890 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568374422 0 0) 0x7f4c48000c60 con 0x7f4c540d1da0 2021-04-09T18:23:16.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c52b13700 1 --2- 172.21.15.32:0/2298389890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5410d010 0x7f4c540d1780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f4c44006700 tx=0x7f4c44009b80).ready entity=mon.2 client_cookie=8da039a3edd1f239 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:16.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c51b11700 1 -- 172.21.15.32:0/2298389890 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c4400c0c0 con 0x7f4c5410d010 2021-04-09T18:23:16.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.770 7f4c51b11700 1 -- 172.21.15.32:0/2298389890 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4400c440 con 0x7f4c5410d010 2021-04-09T18:23:16.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.771 7f4c51b11700 1 -- 172.21.15.32:0/2298389890 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c4400c920 con 0x7f4c5410d010 2021-04-09T18:23:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.771 7f4c58f36700 1 -- 172.21.15.32:0/2298389890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5410d010 msgr2=0x7f4c540d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.771 7f4c58f36700 1 --2- 172.21.15.32:0/2298389890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5410d010 0x7f4c540d1780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f4c44006700 tx=0x7f4c44009b80).stop 2021-04-09T18:23:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.771 7f4c58f36700 1 -- 172.21.15.32:0/2298389890 shutdown_connections 2021-04-09T18:23:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.771 7f4c58f36700 1 --2- 172.21.15.32:0/2298389890 >> v2:172.21.15.86:3300/0 conn(0x7f4c540d8490 0x7f4c5410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.771 7f4c58f36700 1 --2- 172.21.15.32:0/2298389890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5410d010 0x7f4c540d1780 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.771 7f4c58f36700 1 -- 172.21.15.32:0/2298389890 >> 172.21.15.32:0/2298389890 conn(0x7f4c540ce5f0 msgr2=0x7f4c540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.771 7f4c58f36700 1 -- 172.21.15.32:0/2298389890 shutdown_connections 2021-04-09T18:23:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.772 7f4c58f36700 1 -- 172.21.15.32:0/2298389890 wait complete. 2021-04-09T18:23:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c58f36700 1 Processor -- start 2021-04-09T18:23:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c58f36700 1 -- start start 2021-04-09T18:23:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c58f36700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c541a0050 0x7f4c541a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c58f36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540bdcd0 0x7f4c540bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c58f36700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c540bf080 0x7f4c540b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c58f36700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540bf500 con 0x7f4c540bdcd0 2021-04-09T18:23:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c58f36700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540bb7b0 con 0x7f4c540bf080 2021-04-09T18:23:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c58f36700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540be0d0 con 0x7f4c541a0050 2021-04-09T18:23:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c52312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540bdcd0 0x7f4c540bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c52312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540bdcd0 0x7f4c540bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50132/0 (socket says 172.21.15.32:50132) 2021-04-09T18:23:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c52312700 1 -- 172.21.15.32:0/1829755582 learned_addr learned my addr 172.21.15.32:0/1829755582 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c52b13700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c541a0050 0x7f4c541a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.792 7f4c53314700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c540bf080 0x7f4c540b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c52312700 1 -- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c541a0050 msgr2=0x7f4c541a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c52312700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c541a0050 0x7f4c541a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c52312700 1 -- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c540bf080 msgr2=0x7f4c540b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c52312700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c540bf080 0x7f4c540b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c52312700 1 -- 172.21.15.32:0/1829755582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540b1310 con 0x7f4c540bdcd0 2021-04-09T18:23:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c52312700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540bdcd0 0x7f4c540bea60 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0x7f4c4800b560 tx=0x7f4c4800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c437fe700 1 -- 172.21.15.32:0/1829755582 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c48006bf0 con 0x7f4c540bdcd0 2021-04-09T18:23:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c437fe700 1 -- 172.21.15.32:0/1829755582 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c48006030 con 0x7f4c540bdcd0 2021-04-09T18:23:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c540b1310 con 0x7f4c540bdcd0 2021-04-09T18:23:16.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c437fe700 1 -- 172.21.15.32:0/1829755582 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c48005540 con 0x7f4c540bdcd0 2021-04-09T18:23:16.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.793 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c540b1310 con 0x7f4c540bdcd0 2021-04-09T18:23:16.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.794 7f4c437fe700 1 -- 172.21.15.32:0/1829755582 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4c48000af0 con 0x7f4c540bdcd0 2021-04-09T18:23:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.794 7f4c437fe700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c38018930 0x7f4c3801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.794 7f4c437fe700 1 -- 172.21.15.32:0/1829755582 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4c48034c60 con 0x7f4c540bdcd0 2021-04-09T18:23:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.794 7f4c52b13700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c38018930 0x7f4c3801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:16.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.794 7f4c52b13700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c38018930 0x7f4c3801adc0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:16.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.813 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c54093d50 con 0x7f4c540bdcd0 2021-04-09T18:23:16.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:16.817 7f4c437fe700 1 -- 172.21.15.32:0/1829755582 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4c48006bf0 con 0x7f4c540bdcd0 2021-04-09T18:23:17.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.015 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f4c5410d480 con 0x7f4c540bdcd0 2021-04-09T18:23:17.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.561 7f4c437fe700 1 -- 172.21.15.32:0/1829755582 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v160) v1 ==== 105+0+0 (secure 0 0 0) 0x7f4c480219a0 con 0x7f4c540bdcd0 2021-04-09T18:23:17.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c38018930 msgr2=0x7f4c3801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:17.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c38018930 0x7f4c3801adc0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540bdcd0 msgr2=0x7f4c540bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:17.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540bdcd0 0x7f4c540bea60 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0x7f4c4800b560 tx=0x7f4c4800b900).stop 2021-04-09T18:23:17.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 shutdown_connections 2021-04-09T18:23:17.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c38018930 0x7f4c3801adc0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c540bdcd0 0x7f4c540bea60 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c540bf080 0x7f4c540b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 --2- 172.21.15.32:0/1829755582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c541a0050 0x7f4c541a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 >> 172.21.15.32:0/1829755582 conn(0x7f4c540975f0 msgr2=0x7f4c54096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:17.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 shutdown_connections 2021-04-09T18:23:17.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.582 7f4c58f36700 1 -- 172.21.15.32:0/1829755582 wait complete. 2021-04-09T18:23:17.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:17.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.745 7f9d0e196700 1 Processor -- start 2021-04-09T18:23:17.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.745 7f9d0e196700 1 -- start start 2021-04-09T18:23:17.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d0e196700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d0810d010 0x7f9d080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d0e196700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0810d480 con 0x7f9d080d8490 2021-04-09T18:23:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d0e196700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0810e1f0 con 0x7f9d080d1da0 2021-04-09T18:23:17.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d0e196700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0810e780 con 0x7f9d0810d010 2021-04-09T18:23:17.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d07fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9d080d8490 0x7f9d0810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59732/0 (socket says 172.21.15.32:59732) 2021-04-09T18:23:17.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d07fff700 1 -- 172.21.15.32:0/1129864340 learned_addr learned my addr 172.21.15.32:0/1129864340 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d077fe700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d0810d010 0x7f9d080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013260752 0 0) 0x7f9d00000d30 con 0x7f9d080d8490 2021-04-09T18:23:17.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cec002fe0 con 0x7f9d080d8490 2021-04-09T18:23:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698891735 0 0) 0x7f9cf4001080 con 0x7f9d080d1da0 2021-04-09T18:23:17.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cec003210 con 0x7f9d080d1da0 2021-04-09T18:23:17.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 299330361 0 0) 0x7f9d00001460 con 0x7f9d080d8490 2021-04-09T18:23:17.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d0810d010 msgr2=0x7f9d080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:17.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d0810d010 0x7f9d080d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 >> v1:172.21.15.86:6789/0 conn(0x7f9d080d1da0 legacy=0x7f9d080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:17.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d080d8d30 con 0x7f9d080d8490 2021-04-09T18:23:17.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f9d00001980 con 0x7f9d080d8490 2021-04-09T18:23:17.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 >> v1:172.21.15.32:6789/0 conn(0x7f9d080d8490 legacy=0x7f9d0810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.746 7f9d067fc700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec003b10 0x7f9cec005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d067fc700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cec0067a0 0x7f9cec008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:17.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d067fc700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cec0095a0 0x7f9cec00ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:17.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0502f0 con 0x7f9cec003b10 2021-04-09T18:23:17.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0504d0 con 0x7f9cec0067a0 2021-04-09T18:23:17.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec0506f0 con 0x7f9cec0095a0 2021-04-09T18:23:17.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d077fe700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec003b10 0x7f9cec005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:17.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d07fff700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cec0095a0 0x7f9cec00ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:17.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d06ffd700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cec0067a0 0x7f9cec008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:17.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d077fe700 1 -- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cec0095a0 msgr2=0x7f9cec00ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d077fe700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cec0095a0 0x7f9cec00ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d077fe700 1 -- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cec0067a0 msgr2=0x7f9cec008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:17.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d077fe700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cec0067a0 0x7f9cec008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d077fe700 1 -- 172.21.15.32:0/1129864340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec050990 con 0x7f9cec003b10 2021-04-09T18:23:17.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d06ffd700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cec0067a0 0x7f9cec008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:17.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d077fe700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec003b10 0x7f9cec005f20 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f9cf8006ac0 tx=0x7f9cf8009ab0).ready entity=mon.0 client_cookie=39aa3431c3e96ae7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:17.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cf800c0c0 con 0x7f9cec003b10 2021-04-09T18:23:17.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf800c3c0 con 0x7f9cec003b10 2021-04-09T18:23:17.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d067fc700 1 -- 172.21.15.32:0/1129864340 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cf800c8a0 con 0x7f9cec003b10 2021-04-09T18:23:17.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d0e196700 1 -- 172.21.15.32:0/1129864340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec003b10 msgr2=0x7f9cec005f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:17.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d0e196700 1 --2- 172.21.15.32:0/1129864340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec003b10 0x7f9cec005f20 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f9cf8006ac0 tx=0x7f9cf8009ab0).stop 2021-04-09T18:23:17.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.747 7f9d07fff700 1 -- 172.21.15.32:0/1129864340 reap_dead start 2021-04-09T18:23:17.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.748 7f9d0e196700 1 -- 172.21.15.32:0/1129864340 shutdown_connections 2021-04-09T18:23:17.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.748 7f9d0e196700 1 -- 172.21.15.32:0/1129864340 >> 172.21.15.32:0/1129864340 conn(0x7f9d080ce5f0 msgr2=0x7f9d080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:17.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.748 7f9d0e196700 1 -- 172.21.15.32:0/1129864340 shutdown_connections 2021-04-09T18:23:17.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.748 7f9d0e196700 1 -- 172.21.15.32:0/1129864340 wait complete. 2021-04-09T18:23:17.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.768 7f9d0e196700 1 Processor -- start 2021-04-09T18:23:17.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.768 7f9d0e196700 1 -- start start 2021-04-09T18:23:17.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d0e196700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080c3c00 0x7f9d0809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d0e196700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d08095240 0x7f9d081a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d0e196700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d081a1270 0x7f9d080b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d0e196700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0809cda0 con 0x7f9d080c3c00 2021-04-09T18:23:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d0e196700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080c3680 con 0x7f9d08095240 2021-04-09T18:23:17.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d0e196700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d08095640 con 0x7f9d081a1270 2021-04-09T18:23:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080c3c00 0x7f9d0809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080c3c00 0x7f9d0809c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50152/0 (socket says 172.21.15.32:50152) 2021-04-09T18:23:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d077fe700 1 -- 172.21.15.32:0/3998465678 learned_addr learned my addr 172.21.15.32:0/3998465678 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:17.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d07fff700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d081a1270 0x7f9d080b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d06ffd700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d08095240 0x7f9d081a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d077fe700 1 -- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d081a1270 msgr2=0x7f9d080b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d077fe700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d081a1270 0x7f9d080b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d077fe700 1 -- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d08095240 msgr2=0x7f9d081a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:17.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d077fe700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d08095240 0x7f9d081a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d077fe700 1 -- 172.21.15.32:0/3998465678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d081a1670 con 0x7f9d080c3c00 2021-04-09T18:23:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d06ffd700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d08095240 0x7f9d081a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.769 7f9d077fe700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080c3c00 0x7f9d0809c600 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7f9cf80066c0 tx=0x7f9cf800b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:17.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.770 7f9cfffff700 1 -- 172.21.15.32:0/3998465678 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cf800bdb0 con 0x7f9d080c3c00 2021-04-09T18:23:17.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.770 7f9cfffff700 1 -- 172.21.15.32:0/3998465678 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf8000940 con 0x7f9d080c3c00 2021-04-09T18:23:17.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.770 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d081a1670 con 0x7f9d080c3c00 2021-04-09T18:23:17.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.770 7f9cfffff700 1 -- 172.21.15.32:0/3998465678 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cf8002030 con 0x7f9d080c3c00 2021-04-09T18:23:17.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.770 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d081a1670 con 0x7f9d080c3c00 2021-04-09T18:23:17.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.771 7f9cfffff700 1 -- 172.21.15.32:0/3998465678 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9cf8009360 con 0x7f9d080c3c00 2021-04-09T18:23:17.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.771 7f9cfffff700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ce40188c0 0x7f9ce401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:17.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.771 7f9cfffff700 1 -- 172.21.15.32:0/3998465678 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9cf8032780 con 0x7f9d080c3c00 2021-04-09T18:23:17.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.771 7f9d06ffd700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ce40188c0 0x7f9ce401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:17.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.772 7f9d06ffd700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ce40188c0 0x7f9ce401adc0 crc :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:17.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.790 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d08093d50 con 0x7f9d080c3c00 2021-04-09T18:23:17.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.793 7f9cfffff700 1 -- 172.21.15.32:0/3998465678 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9cf8009140 con 0x7f9d080c3c00 2021-04-09T18:23:18.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:17.986 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f9d0810d480 con 0x7f9d080c3c00 2021-04-09T18:23:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.573 7f9cfffff700 1 -- 172.21.15.32:0/3998465678 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v161) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9cf8009350 con 0x7f9d080c3c00 2021-04-09T18:23:18.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ce40188c0 msgr2=0x7f9ce401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:18.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ce40188c0 0x7f9ce401adc0 crc :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080c3c00 msgr2=0x7f9d0809c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:18.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080c3c00 0x7f9d0809c600 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7f9cf80066c0 tx=0x7f9cf800b9c0).stop 2021-04-09T18:23:18.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 shutdown_connections 2021-04-09T18:23:18.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ce40188c0 0x7f9ce401adc0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d081a1270 0x7f9d080b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d08095240 0x7f9d081a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 --2- 172.21.15.32:0/3998465678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d080c3c00 0x7f9d0809c600 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.605 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 >> 172.21.15.32:0/3998465678 conn(0x7f9d080975f0 msgr2=0x7f9d08096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:18.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.606 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 shutdown_connections 2021-04-09T18:23:18.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.606 7f9d0e196700 1 -- 172.21.15.32:0/3998465678 wait complete. 2021-04-09T18:23:18.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.775 7fd810180700 1 Processor -- start 2021-04-09T18:23:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.775 7fd810180700 1 -- start start 2021-04-09T18:23:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd810180700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd80810d010 0x7fd8080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd810180700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd8080d1da0 0x7fd8080d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd810180700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80810d480 con 0x7fd8080d82c0 2021-04-09T18:23:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd810180700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8080d8d30 con 0x7fd80810d010 2021-04-09T18:23:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd810180700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8080d9220 con 0x7fd8080d1da0 2021-04-09T18:23:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd80df9c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd80810d010 0x7fd8080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd80df9c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd80810d010 0x7fd8080d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50160/0 (socket says 172.21.15.32:50160) 2021-04-09T18:23:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd80df9c700 1 -- 172.21.15.32:0/1931607940 learned_addr learned my addr 172.21.15.32:0/1931607940 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd80d79b700 1 --2- 172.21.15.32:0/1931607940 >> v2:172.21.15.86:3300/0 conn(0x7fd8080d1da0 0x7fd8080d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd80df9c700 1 -- 172.21.15.32:0/1931607940 >> v2:172.21.15.86:3300/0 conn(0x7fd8080d1da0 msgr2=0x7fd8080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd80df9c700 1 --2- 172.21.15.32:0/1931607940 >> v2:172.21.15.86:3300/0 conn(0x7fd8080d1da0 0x7fd8080d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd80df9c700 1 -- 172.21.15.32:0/1931607940 >> v1:172.21.15.86:6789/0 conn(0x7fd8080d82c0 legacy=0x7fd80810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd80df9c700 1 -- 172.21.15.32:0/1931607940 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8080d86f0 con 0x7fd80810d010 2021-04-09T18:23:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.776 7fd80df9c700 1 --2- 172.21.15.32:0/1931607940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80810d010 0x7fd8080d1780 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7fd804006700 tx=0x7fd804009b80).ready entity=mon.0 client_cookie=6e620b80943bce1d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.777 7fd80cf9a700 1 -- 172.21.15.32:0/1931607940 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd80400c0c0 con 0x7fd80810d010 2021-04-09T18:23:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.777 7fd80cf9a700 1 -- 172.21.15.32:0/1931607940 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80400c440 con 0x7fd80810d010 2021-04-09T18:23:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.777 7fd80cf9a700 1 -- 172.21.15.32:0/1931607940 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd80400c920 con 0x7fd80810d010 2021-04-09T18:23:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.777 7fd810180700 1 -- 172.21.15.32:0/1931607940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80810d010 msgr2=0x7fd8080d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.777 7fd810180700 1 --2- 172.21.15.32:0/1931607940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80810d010 0x7fd8080d1780 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7fd804006700 tx=0x7fd804009b80).stop 2021-04-09T18:23:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.777 7fd810180700 1 -- 172.21.15.32:0/1931607940 shutdown_connections 2021-04-09T18:23:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.777 7fd810180700 1 --2- 172.21.15.32:0/1931607940 >> v2:172.21.15.86:3300/0 conn(0x7fd8080d1da0 0x7fd8080d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.777 7fd810180700 1 --2- 172.21.15.32:0/1931607940 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80810d010 0x7fd8080d1780 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.777 7fd810180700 1 -- 172.21.15.32:0/1931607940 >> 172.21.15.32:0/1931607940 conn(0x7fd8080ce5f0 msgr2=0x7fd8080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:18.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.778 7fd810180700 1 -- 172.21.15.32:0/1931607940 shutdown_connections 2021-04-09T18:23:18.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.778 7fd810180700 1 -- 172.21.15.32:0/1931607940 wait complete. 2021-04-09T18:23:18.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd810180700 1 Processor -- start 2021-04-09T18:23:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd810180700 1 -- start start 2021-04-09T18:23:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd810180700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80808a910 0x7fd80809e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd810180700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80808ad10 0x7fd80809ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd810180700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8080922e0 0x7fd8080926e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd810180700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80818fc90 con 0x7fd80808ad10 2021-04-09T18:23:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd810180700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80809e090 con 0x7fd80808a910 2021-04-09T18:23:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd810180700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80808b400 con 0x7fd8080922e0 2021-04-09T18:23:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd80d79b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80808ad10 0x7fd80809ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd80d79b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80808ad10 0x7fd80809ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50166/0 (socket says 172.21.15.32:50166) 2021-04-09T18:23:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd80d79b700 1 -- 172.21.15.32:0/3272989663 learned_addr learned my addr 172.21.15.32:0/3272989663 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.798 7fd80e79d700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8080922e0 0x7fd8080926e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd80df9c700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80808a910 0x7fd80809e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd80d79b700 1 -- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8080922e0 msgr2=0x7fd8080926e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd80d79b700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8080922e0 0x7fd8080926e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd80d79b700 1 -- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80808a910 msgr2=0x7fd80809e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd80d79b700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80808a910 0x7fd80809e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd80d79b700 1 -- 172.21.15.32:0/3272989663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd808092d00 con 0x7fd80808ad10 2021-04-09T18:23:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd80e79d700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8080922e0 0x7fd8080926e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd80df9c700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80808a910 0x7fd80809e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd80d79b700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80808ad10 0x7fd80809ec30 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7fd7f8006f00 tx=0x7fd7f800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd7fe7fc700 1 -- 172.21.15.32:0/3272989663 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7f800bd30 con 0x7fd80808ad10 2021-04-09T18:23:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd7fe7fc700 1 -- 172.21.15.32:0/3272989663 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f8002030 con 0x7fd80808ad10 2021-04-09T18:23:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd810180700 1 -- 172.21.15.32:0/3272989663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd808092d00 con 0x7fd80808ad10 2021-04-09T18:23:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd7fe7fc700 1 -- 172.21.15.32:0/3272989663 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7f8002590 con 0x7fd80808ad10 2021-04-09T18:23:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.799 7fd810180700 1 -- 172.21.15.32:0/3272989663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd808092d00 con 0x7fd80808ad10 2021-04-09T18:23:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.800 7fd7fe7fc700 1 -- 172.21.15.32:0/3272989663 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd7f8000c90 con 0x7fd80808ad10 2021-04-09T18:23:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.800 7fd7fe7fc700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7f0018930 0x7fd7f001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:18.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.800 7fd7fe7fc700 1 -- 172.21.15.32:0/3272989663 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd7f8032a70 con 0x7fd80808ad10 2021-04-09T18:23:18.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.800 7fd80df9c700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7f0018930 0x7fd7f001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:18.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.801 7fd80df9c700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7f0018930 0x7fd7f001ae10 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:18.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.820 7fd810180700 1 -- 172.21.15.32:0/3272989663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80818fc90 con 0x7fd80808ad10 2021-04-09T18:23:18.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:18.824 7fd7fe7fc700 1 -- 172.21.15.32:0/3272989663 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd7f800bd30 con 0x7fd80808ad10 2021-04-09T18:23:19.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.017 7fd810180700 1 -- 172.21.15.32:0/3272989663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fd80818fc90 con 0x7fd80808ad10 2021-04-09T18:23:19.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.576 7fd7fe7fc700 1 -- 172.21.15.32:0/3272989663 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v162) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd7f8000c90 con 0x7fd80808ad10 2021-04-09T18:23:19.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.583 7fd810180700 1 -- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7f0018930 msgr2=0x7fd7f001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:19.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.583 7fd810180700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7f0018930 0x7fd7f001ae10 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.583 7fd810180700 1 -- 172.21.15.32:0/3272989663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80808ad10 msgr2=0x7fd80809ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:19.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.583 7fd810180700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80808ad10 0x7fd80809ec30 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7fd7f8006f00 tx=0x7fd7f800b940).stop 2021-04-09T18:23:19.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.584 7fd810180700 1 -- 172.21.15.32:0/3272989663 shutdown_connections 2021-04-09T18:23:19.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.584 7fd810180700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7f0018930 0x7fd7f001ae10 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.584 7fd810180700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd80808ad10 0x7fd80809ec30 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.584 7fd810180700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8080922e0 0x7fd8080926e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.584 7fd810180700 1 --2- 172.21.15.32:0/3272989663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80808a910 0x7fd80809e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.584 7fd810180700 1 -- 172.21.15.32:0/3272989663 >> 172.21.15.32:0/3272989663 conn(0x7fd8080b2fa0 msgr2=0x7fd8080a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:19.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.584 7fd810180700 1 -- 172.21.15.32:0/3272989663 shutdown_connections 2021-04-09T18:23:19.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.584 7fd810180700 1 -- 172.21.15.32:0/3272989663 wait complete. 2021-04-09T18:23:19.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.749 7fbdd8f15700 1 Processor -- start 2021-04-09T18:23:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.749 7fbdd8f15700 1 -- start start 2021-04-09T18:23:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.750 7fbdd8f15700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbdd410d010 0x7fbdd40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:19.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.750 7fbdd8f15700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40d8490 0x7fbdd410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.750 7fbdd8f15700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd410d480 con 0x7fbdd40d1da0 2021-04-09T18:23:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.750 7fbdd8f15700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd40d8d30 con 0x7fbdd410d010 2021-04-09T18:23:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.750 7fbdd8f15700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd40d9220 con 0x7fbdd40d8490 2021-04-09T18:23:19.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.750 7fbdd2b13700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbdd410d010 0x7fbdd40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.750 7fbdd2312700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbdd40d1da0 0x7fbdd40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59764/0 (socket says 172.21.15.32:59764) 2021-04-09T18:23:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.750 7fbdd2312700 1 -- 172.21.15.32:0/2518662792 learned_addr learned my addr 172.21.15.32:0/2518662792 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.750 7fbdd3314700 1 --2- 172.21.15.32:0/2518662792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40d8490 0x7fbdd410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd1b11700 1 -- 172.21.15.32:0/2518662792 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497418012 0 0) 0x7fbdc8000ce0 con 0x7fbdd40d1da0 2021-04-09T18:23:19.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd1b11700 1 -- 172.21.15.32:0/2518662792 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdbc002fe0 con 0x7fbdd40d1da0 2021-04-09T18:23:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd2b13700 1 -- 172.21.15.32:0/2518662792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40d8490 msgr2=0x7fbdd410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd2b13700 1 --2- 172.21.15.32:0/2518662792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40d8490 0x7fbdd410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd2b13700 1 -- 172.21.15.32:0/2518662792 >> v1:172.21.15.32:6789/0 conn(0x7fbdd40d1da0 legacy=0x7fbdd40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:19.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd2b13700 1 -- 172.21.15.32:0/2518662792 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd410e020 con 0x7fbdd410d010 2021-04-09T18:23:19.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd2b13700 1 --2- 172.21.15.32:0/2518662792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd410d010 0x7fbdd40d1780 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7fbdc4006650 tx=0x7fbdc4009a80).ready entity=mon.0 client_cookie=20610acc84946920 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:19.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd1b11700 1 -- 172.21.15.32:0/2518662792 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdc40009b0 con 0x7fbdd410d010 2021-04-09T18:23:19.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd1b11700 1 -- 172.21.15.32:0/2518662792 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc400c0c0 con 0x7fbdd410d010 2021-04-09T18:23:19.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd1b11700 1 -- 172.21.15.32:0/2518662792 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdc400c510 con 0x7fbdd410d010 2021-04-09T18:23:19.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd8f15700 1 -- 172.21.15.32:0/2518662792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd410d010 msgr2=0x7fbdd40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.751 7fbdd8f15700 1 --2- 172.21.15.32:0/2518662792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd410d010 0x7fbdd40d1780 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7fbdc4006650 tx=0x7fbdc4009a80).stop 2021-04-09T18:23:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.752 7fbdd8f15700 1 -- 172.21.15.32:0/2518662792 shutdown_connections 2021-04-09T18:23:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.752 7fbdd8f15700 1 --2- 172.21.15.32:0/2518662792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40d8490 0x7fbdd410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.752 7fbdd8f15700 1 --2- 172.21.15.32:0/2518662792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd410d010 0x7fbdd40d1780 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.752 7fbdd8f15700 1 -- 172.21.15.32:0/2518662792 >> 172.21.15.32:0/2518662792 conn(0x7fbdd40ce5f0 msgr2=0x7fbdd40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.752 7fbdd8f15700 1 -- 172.21.15.32:0/2518662792 shutdown_connections 2021-04-09T18:23:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.752 7fbdd8f15700 1 -- 172.21.15.32:0/2518662792 wait complete. 2021-04-09T18:23:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.772 7fbdd8f15700 1 Processor -- start 2021-04-09T18:23:19.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.772 7fbdd8f15700 1 -- start start 2021-04-09T18:23:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd8f15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd41a0050 0x7fbdd41a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd8f15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd40bdcd0 0x7fbdd40bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd8f15700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40bf080 0x7fbdd40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:19.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd8f15700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd40bf500 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd8f15700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd40bb7b0 con 0x7fbdd41a0050 2021-04-09T18:23:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd8f15700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd40be0d0 con 0x7fbdd40bf080 2021-04-09T18:23:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd40bdcd0 0x7fbdd40bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd40bdcd0 0x7fbdd40bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50180/0 (socket says 172.21.15.32:50180) 2021-04-09T18:23:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2312700 1 -- 172.21.15.32:0/4114943805 learned_addr learned my addr 172.21.15.32:0/4114943805 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2b13700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd41a0050 0x7fbdd41a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd3314700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40bf080 0x7fbdd40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2312700 1 -- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40bf080 msgr2=0x7fbdd40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2312700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40bf080 0x7fbdd40b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2312700 1 -- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd41a0050 msgr2=0x7fbdd41a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2312700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd41a0050 0x7fbdd41a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2312700 1 -- 172.21.15.32:0/4114943805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd40b1310 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.773 7fbdd2312700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd40bdcd0 0x7fbdd40bea60 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7fbdc8006f00 tx=0x7fbdc800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.774 7fbdc37fe700 1 -- 172.21.15.32:0/4114943805 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdc800bd30 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.774 7fbdc37fe700 1 -- 172.21.15.32:0/4114943805 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc8006030 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.774 7fbdc37fe700 1 -- 172.21.15.32:0/4114943805 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdc8004590 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.774 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd40b1310 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.774 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd40b1310 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.774 7fbdc37fe700 1 -- 172.21.15.32:0/4114943805 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbdc80043a0 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.774 7fbdc37fe700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb80188d0 0x7fbdb801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:19.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.774 7fbdc37fe700 1 -- 172.21.15.32:0/4114943805 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbdc80329c0 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.775 7fbdd2b13700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb80188d0 0x7fbdb801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.775 7fbdd2b13700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb80188d0 0x7fbdb801add0 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.794 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd4093d50 con 0x7fbdd40bdcd0 2021-04-09T18:23:19.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.798 7fbdc37fe700 1 -- 172.21.15.32:0/4114943805 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbdc80009b0 con 0x7fbdd40bdcd0 2021-04-09T18:23:20.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:19.993 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7fbdd410d480 con 0x7fbdd40bdcd0 2021-04-09T18:23:20.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.583 7fbdc37fe700 1 -- 172.21.15.32:0/4114943805 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v163) v1 ==== 105+0+0 (secure 0 0 0) 0x7fbdc80043a0 con 0x7fbdd40bdcd0 2021-04-09T18:23:20.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb80188d0 msgr2=0x7fbdb801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:20.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb80188d0 0x7fbdb801add0 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd40bdcd0 msgr2=0x7fbdd40bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:20.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd40bdcd0 0x7fbdd40bea60 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7fbdc8006f00 tx=0x7fbdc800b940).stop 2021-04-09T18:23:20.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 shutdown_connections 2021-04-09T18:23:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb80188d0 0x7fbdb801add0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd40bdcd0 0x7fbdd40bea60 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd40bf080 0x7fbdd40b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 --2- 172.21.15.32:0/4114943805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd41a0050 0x7fbdd41a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.610 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 >> 172.21.15.32:0/4114943805 conn(0x7fbdd40975f0 msgr2=0x7fbdd4096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:20.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.611 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 shutdown_connections 2021-04-09T18:23:20.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.611 7fbdd8f15700 1 -- 172.21.15.32:0/4114943805 wait complete. 2021-04-09T18:23:20.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:20.705 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:23:20.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.776 7fae26383700 1 Processor -- start 2021-04-09T18:23:20.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.776 7fae26383700 1 -- start start 2021-04-09T18:23:20.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.776 7fae26383700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2010d010 0x7fae200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:20.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.776 7fae26383700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae200d8490 0x7fae2010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:20.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.776 7fae26383700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2010d480 con 0x7fae200d1da0 2021-04-09T18:23:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.776 7fae26383700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae2010e780 con 0x7fae200d8490 2021-04-09T18:23:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.776 7fae26383700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae200d8d30 con 0x7fae2010d010 2021-04-09T18:23:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae249a0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae200d8490 0x7fae2010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:20.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae249a0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fae200d8490 0x7fae2010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50200/0 (socket says 172.21.15.32:50200) 2021-04-09T18:23:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae249a0700 1 -- 172.21.15.32:0/2961473216 learned_addr learned my addr 172.21.15.32:0/2961473216 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae1ffff700 1 --2- 172.21.15.32:0/2961473216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2010d010 0x7fae200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae249a0700 1 -- 172.21.15.32:0/2961473216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2010d010 msgr2=0x7fae200d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:20.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae249a0700 1 --2- 172.21.15.32:0/2961473216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2010d010 0x7fae200d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae249a0700 1 -- 172.21.15.32:0/2961473216 >> v1:172.21.15.86:6789/0 conn(0x7fae200d1da0 legacy=0x7fae200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:20.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae249a0700 1 -- 172.21.15.32:0/2961473216 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae200d9220 con 0x7fae200d8490 2021-04-09T18:23:20.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae249a0700 1 --2- 172.21.15.32:0/2961473216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae200d8490 0x7fae2010da00 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7fae18006f00 tx=0x7fae18009f10).ready entity=mon.0 client_cookie=409cb4625d9b9333 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:20.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae1f7fe700 1 -- 172.21.15.32:0/2961473216 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae1800b3b0 con 0x7fae200d8490 2021-04-09T18:23:20.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae1f7fe700 1 -- 172.21.15.32:0/2961473216 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1800b730 con 0x7fae200d8490 2021-04-09T18:23:20.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae1f7fe700 1 -- 172.21.15.32:0/2961473216 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae1800bc10 con 0x7fae200d8490 2021-04-09T18:23:20.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae26383700 1 -- 172.21.15.32:0/2961473216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae200d8490 msgr2=0x7fae2010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:20.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.777 7fae26383700 1 --2- 172.21.15.32:0/2961473216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae200d8490 0x7fae2010da00 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7fae18006f00 tx=0x7fae18009f10).stop 2021-04-09T18:23:20.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.778 7fae26383700 1 -- 172.21.15.32:0/2961473216 shutdown_connections 2021-04-09T18:23:20.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.778 7fae26383700 1 --2- 172.21.15.32:0/2961473216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae200d8490 0x7fae2010da00 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.778 7fae26383700 1 --2- 172.21.15.32:0/2961473216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2010d010 0x7fae200d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.778 7fae26383700 1 -- 172.21.15.32:0/2961473216 >> 172.21.15.32:0/2961473216 conn(0x7fae200ce5f0 msgr2=0x7fae200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:20.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.778 7fae26383700 1 -- 172.21.15.32:0/2961473216 shutdown_connections 2021-04-09T18:23:20.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.778 7fae26383700 1 -- 172.21.15.32:0/2961473216 wait complete. 2021-04-09T18:23:20.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.798 7fae26383700 1 Processor -- start 2021-04-09T18:23:20.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.798 7fae26383700 1 -- start start 2021-04-09T18:23:20.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae26383700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae201a33f0 0x7fae201a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:20.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae26383700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2017bad0 0x7fae2017bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:20.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae26383700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae20179610 0x7fae20179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:20.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae26383700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2017b5d0 con 0x7fae2017bad0 2021-04-09T18:23:20.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae26383700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2017c4f0 con 0x7fae20179610 2021-04-09T18:23:20.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae26383700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae200a1800 con 0x7fae201a33f0 2021-04-09T18:23:20.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2017bad0 0x7fae2017bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:20.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2017bad0 0x7fae2017bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50206/0 (socket says 172.21.15.32:50206) 2021-04-09T18:23:20.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae17fff700 1 -- 172.21.15.32:0/2177568390 learned_addr learned my addr 172.21.15.32:0/2177568390 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:20.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae1ffff700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae201a33f0 0x7fae201a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:20.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae249a0700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae20179610 0x7fae20179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:20.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae17fff700 1 -- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae201a33f0 msgr2=0x7fae201a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:20.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae17fff700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae201a33f0 0x7fae201a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae17fff700 1 -- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae20179610 msgr2=0x7fae20179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:20.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae17fff700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae20179610 0x7fae20179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:20.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.799 7fae17fff700 1 -- 172.21.15.32:0/2177568390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2017a030 con 0x7fae2017bad0 2021-04-09T18:23:20.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.800 7fae17fff700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2017bad0 0x7fae2017bed0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7fae08006ed0 tx=0x7fae0800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:20.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.800 7fae1d7fa700 1 -- 172.21.15.32:0/2177568390 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae08006020 con 0x7fae2017bad0 2021-04-09T18:23:20.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.800 7fae1d7fa700 1 -- 172.21.15.32:0/2177568390 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae08006360 con 0x7fae2017bad0 2021-04-09T18:23:20.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.800 7fae26383700 1 -- 172.21.15.32:0/2177568390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2017a030 con 0x7fae2017bad0 2021-04-09T18:23:20.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.800 7fae1d7fa700 1 -- 172.21.15.32:0/2177568390 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae08005730 con 0x7fae2017bad0 2021-04-09T18:23:20.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.800 7fae26383700 1 -- 172.21.15.32:0/2177568390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2017a030 con 0x7fae2017bad0 2021-04-09T18:23:20.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.801 7fae1d7fa700 1 -- 172.21.15.32:0/2177568390 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fae080054b0 con 0x7fae2017bad0 2021-04-09T18:23:20.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.801 7fae1d7fa700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae040188c0 0x7fae0401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:20.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.801 7fae1d7fa700 1 -- 172.21.15.32:0/2177568390 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fae080327e0 con 0x7fae2017bad0 2021-04-09T18:23:20.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.801 7fae1ffff700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae040188c0 0x7fae0401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:20.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.802 7fae1ffff700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae040188c0 0x7fae0401adc0 crc :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:20.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.820 7fae26383700 1 -- 172.21.15.32:0/2177568390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae20093d50 con 0x7fae2017bad0 2021-04-09T18:23:20.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:20.824 7fae1d7fa700 1 -- 172.21.15.32:0/2177568390 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fae08005fe0 con 0x7fae2017bad0 2021-04-09T18:23:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.019 7fae26383700 1 -- 172.21.15.32:0/2177568390 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fae2010d480 con 0x7fae2017bad0 2021-04-09T18:23:21.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.589 7fae1d7fa700 1 -- 172.21.15.32:0/2177568390 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v164) v1 ==== 105+0+0 (secure 0 0 0) 0x7fae080054b0 con 0x7fae2017bad0 2021-04-09T18:23:21.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 -- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae040188c0 msgr2=0x7fae0401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:21.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae040188c0 0x7fae0401adc0 crc :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 -- 172.21.15.32:0/2177568390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2017bad0 msgr2=0x7fae2017bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:21.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2017bad0 0x7fae2017bed0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7fae08006ed0 tx=0x7fae0800b8e0).stop 2021-04-09T18:23:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 -- 172.21.15.32:0/2177568390 shutdown_connections 2021-04-09T18:23:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae040188c0 0x7fae0401adc0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae2017bad0 0x7fae2017bed0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae20179610 0x7fae20179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 --2- 172.21.15.32:0/2177568390 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae201a33f0 0x7fae201a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 -- 172.21.15.32:0/2177568390 >> 172.21.15.32:0/2177568390 conn(0x7fae200a22e0 msgr2=0x7fae20094c80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 -- 172.21.15.32:0/2177568390 shutdown_connections 2021-04-09T18:23:21.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.636 7fae26383700 1 -- 172.21.15.32:0/2177568390 wait complete. 2021-04-09T18:23:21.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:21.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.838 7fd31090c700 1 Processor -- start 2021-04-09T18:23:21.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.838 7fd31090c700 1 -- start start 2021-04-09T18:23:21.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd31090c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c10d010 0x7fd30c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:21.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd31090c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd30c0d8490 0x7fd30c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd31090c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c10d480 con 0x7fd30c0d1da0 2021-04-09T18:23:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd31090c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd30c10e780 con 0x7fd30c0d8490 2021-04-09T18:23:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd31090c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c0d8d30 con 0x7fd30c10d010 2021-04-09T18:23:21.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd30ae1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd30c0d8490 0x7fd30c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:21.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd30a61c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c10d010 0x7fd30c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:21.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd30ae1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd30c0d8490 0x7fd30c10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50222/0 (socket says 172.21.15.32:50222) 2021-04-09T18:23:21.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd30ae1d700 1 -- 172.21.15.32:0/3312383404 learned_addr learned my addr 172.21.15.32:0/3312383404 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd309e1b700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd30c0d1da0 0x7fd30c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:57734/0 (socket says 172.21.15.32:57734) 2021-04-09T18:23:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.839 7fd30961a700 1 -- 172.21.15.32:0/3312383404 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486143990 0 0) 0x7fd300000c60 con 0x7fd30c0d1da0 2021-04-09T18:23:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd30ae1d700 1 -- 172.21.15.32:0/3312383404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c10d010 msgr2=0x7fd30c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:21.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd30ae1d700 1 --2- 172.21.15.32:0/3312383404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c10d010 0x7fd30c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd30ae1d700 1 -- 172.21.15.32:0/3312383404 >> v1:172.21.15.86:6789/0 conn(0x7fd30c0d1da0 legacy=0x7fd30c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd30ae1d700 1 -- 172.21.15.32:0/3312383404 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c0d9220 con 0x7fd30c0d8490 2021-04-09T18:23:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd30a61c700 1 --2- 172.21.15.32:0/3312383404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c10d010 0x7fd30c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:21.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd30ae1d700 1 --2- 172.21.15.32:0/3312383404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c0d8490 0x7fd30c10da00 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fd304006f00 tx=0x7fd304009f10).ready entity=mon.0 client_cookie=9acc51d9ccf4a158 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd30961a700 1 -- 172.21.15.32:0/3312383404 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd30400b3b0 con 0x7fd30c0d8490 2021-04-09T18:23:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd30961a700 1 -- 172.21.15.32:0/3312383404 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30400b730 con 0x7fd30c0d8490 2021-04-09T18:23:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd30961a700 1 -- 172.21.15.32:0/3312383404 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd304000f90 con 0x7fd30c0d8490 2021-04-09T18:23:21.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd31090c700 1 -- 172.21.15.32:0/3312383404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c0d8490 msgr2=0x7fd30c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:21.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.840 7fd31090c700 1 --2- 172.21.15.32:0/3312383404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c0d8490 0x7fd30c10da00 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fd304006f00 tx=0x7fd304009f10).stop 2021-04-09T18:23:21.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.841 7fd31090c700 1 -- 172.21.15.32:0/3312383404 shutdown_connections 2021-04-09T18:23:21.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.841 7fd31090c700 1 --2- 172.21.15.32:0/3312383404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c0d8490 0x7fd30c10da00 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.841 7fd31090c700 1 --2- 172.21.15.32:0/3312383404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c10d010 0x7fd30c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.841 7fd31090c700 1 -- 172.21.15.32:0/3312383404 >> 172.21.15.32:0/3312383404 conn(0x7fd30c0ce5f0 msgr2=0x7fd30c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:21.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.841 7fd31090c700 1 -- 172.21.15.32:0/3312383404 shutdown_connections 2021-04-09T18:23:21.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.841 7fd31090c700 1 -- 172.21.15.32:0/3312383404 wait complete. 2021-04-09T18:23:21.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.861 7fd31090c700 1 Processor -- start 2021-04-09T18:23:21.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.861 7fd31090c700 1 -- start start 2021-04-09T18:23:21.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.861 7fd31090c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30c09e530 0x7fd30c09b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:21.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.861 7fd31090c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c09e930 0x7fd30c09f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:21.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.861 7fd31090c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c09f990 0x7fd30c094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:21.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.861 7fd31090c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c09edf0 con 0x7fd30c09f990 2021-04-09T18:23:21.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.861 7fd31090c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c09fd90 con 0x7fd30c09e530 2021-04-09T18:23:21.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.861 7fd31090c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c09bd40 con 0x7fd30c09e930 2021-04-09T18:23:21.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30ae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c09f990 0x7fd30c094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:21.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30ae1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c09f990 0x7fd30c094c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50228/0 (socket says 172.21.15.32:50228) 2021-04-09T18:23:21.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30ae1d700 1 -- 172.21.15.32:0/3022558923 learned_addr learned my addr 172.21.15.32:0/3022558923 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:21.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd309e1b700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c09e930 0x7fd30c09f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30a61c700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30c09e530 0x7fd30c09b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30ae1d700 1 -- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c09e930 msgr2=0x7fd30c09f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30ae1d700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c09e930 0x7fd30c09f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30ae1d700 1 -- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30c09e530 msgr2=0x7fd30c09b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30ae1d700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30c09e530 0x7fd30c09b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30ae1d700 1 -- 172.21.15.32:0/3022558923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c095290 con 0x7fd30c09f990 2021-04-09T18:23:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd30ae1d700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c09f990 0x7fd30c094c70 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fd30400b9b0 tx=0x7fd30400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd2faffd700 1 -- 172.21.15.32:0/3022558923 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd304005820 con 0x7fd30c09f990 2021-04-09T18:23:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd2faffd700 1 -- 172.21.15.32:0/3022558923 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd304005aa0 con 0x7fd30c09f990 2021-04-09T18:23:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd31090c700 1 -- 172.21.15.32:0/3022558923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30c095290 con 0x7fd30c09f990 2021-04-09T18:23:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd2faffd700 1 -- 172.21.15.32:0/3022558923 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd304006030 con 0x7fd30c09f990 2021-04-09T18:23:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.862 7fd31090c700 1 -- 172.21.15.32:0/3022558923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30c095290 con 0x7fd30c09f990 2021-04-09T18:23:21.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.863 7fd2faffd700 1 -- 172.21.15.32:0/3022558923 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd304006030 con 0x7fd30c09f990 2021-04-09T18:23:21.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.863 7fd2faffd700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2f0018930 0x7fd2f001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:21.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.863 7fd2faffd700 1 -- 172.21.15.32:0/3022558923 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd304032da0 con 0x7fd30c09f990 2021-04-09T18:23:21.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.863 7fd30a61c700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2f0018930 0x7fd2f001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:21.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.863 7fd30a61c700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2f0018930 0x7fd2f001adc0 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.883 7fd31090c700 1 -- 172.21.15.32:0/3022558923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30c095b10 con 0x7fd30c09f990 2021-04-09T18:23:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:21.887 7fd2faffd700 1 -- 172.21.15.32:0/3022558923 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd304000f90 con 0x7fd30c09f990 2021-04-09T18:23:22.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.079 7fd31090c700 1 -- 172.21.15.32:0/3022558923 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fd30c095b10 con 0x7fd30c09f990 2021-04-09T18:23:22.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.603 7fd2faffd700 1 -- 172.21.15.32:0/3022558923 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v166) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd304006030 con 0x7fd30c09f990 2021-04-09T18:23:22.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 -- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2f0018930 msgr2=0x7fd2f001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:22.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2f0018930 0x7fd2f001adc0 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 -- 172.21.15.32:0/3022558923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c09f990 msgr2=0x7fd30c094c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:22.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c09f990 0x7fd30c094c70 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fd30400b9b0 tx=0x7fd30400bd50).stop 2021-04-09T18:23:22.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 -- 172.21.15.32:0/3022558923 shutdown_connections 2021-04-09T18:23:22.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2f0018930 0x7fd2f001adc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd30c09f990 0x7fd30c094c70 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30c09e930 0x7fd30c09f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 --2- 172.21.15.32:0/3022558923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30c09e530 0x7fd30c09b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 -- 172.21.15.32:0/3022558923 >> 172.21.15.32:0/3022558923 conn(0x7fd30c0c4290 msgr2=0x7fd30c0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:22.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 -- 172.21.15.32:0/3022558923 shutdown_connections 2021-04-09T18:23:22.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.646 7fd31090c700 1 -- 172.21.15.32:0/3022558923 wait complete. 2021-04-09T18:23:22.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:22.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.811 7fa087c7e700 1 Processor -- start 2021-04-09T18:23:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.811 7fa087c7e700 1 -- start start 2021-04-09T18:23:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.811 7fa087c7e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa08010d010 0x7fa0800d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:22.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.811 7fa087c7e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0800d8490 0x7fa08010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:22.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.811 7fa087c7e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08010d480 con 0x7fa0800d1da0 2021-04-09T18:23:22.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.811 7fa087c7e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0800d8d30 con 0x7fa08010d010 2021-04-09T18:23:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa087c7e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0800d9220 con 0x7fa0800d8490 2021-04-09T18:23:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa085a9a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa08010d010 0x7fa0800d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa08629b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0800d8490 0x7fa08010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa085299700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa0800d1da0 0x7fa0800d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:57758/0 (socket says 172.21.15.32:57758) 2021-04-09T18:23:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa085299700 1 -- 172.21.15.32:0/2402753080 learned_addr learned my addr 172.21.15.32:0/2402753080 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa084a98700 1 -- 172.21.15.32:0/2402753080 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672966483 0 0) 0x7fa0740010c0 con 0x7fa0800d1da0 2021-04-09T18:23:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa084a98700 1 -- 172.21.15.32:0/2402753080 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa068002fe0 con 0x7fa0800d1da0 2021-04-09T18:23:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa08629b700 1 -- 172.21.15.32:0/2402753080 >> v2:172.21.15.86:3300/0 conn(0x7fa08010d010 msgr2=0x7fa0800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa08629b700 1 --2- 172.21.15.32:0/2402753080 >> v2:172.21.15.86:3300/0 conn(0x7fa08010d010 0x7fa0800d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa08629b700 1 -- 172.21.15.32:0/2402753080 >> v1:172.21.15.86:6789/0 conn(0x7fa0800d1da0 legacy=0x7fa0800d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:22.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa08629b700 1 -- 172.21.15.32:0/2402753080 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08010e020 con 0x7fa0800d8490 2021-04-09T18:23:22.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa085a9a700 1 --2- 172.21.15.32:0/2402753080 >> v2:172.21.15.86:3300/0 conn(0x7fa08010d010 0x7fa0800d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:22.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.812 7fa08629b700 1 --2- 172.21.15.32:0/2402753080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0800d8490 0x7fa08010da00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fa070006ad0 tx=0x7fa070009ad0).ready entity=mon.2 client_cookie=e9ba682d73352789 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:22.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.813 7fa084a98700 1 -- 172.21.15.32:0/2402753080 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0700022e0 con 0x7fa0800d8490 2021-04-09T18:23:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.813 7fa084a98700 1 -- 172.21.15.32:0/2402753080 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa070006410 con 0x7fa0800d8490 2021-04-09T18:23:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.813 7fa084a98700 1 -- 172.21.15.32:0/2402753080 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa070002620 con 0x7fa0800d8490 2021-04-09T18:23:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.813 7fa087c7e700 1 -- 172.21.15.32:0/2402753080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0800d8490 msgr2=0x7fa08010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.813 7fa087c7e700 1 --2- 172.21.15.32:0/2402753080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0800d8490 0x7fa08010da00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fa070006ad0 tx=0x7fa070009ad0).stop 2021-04-09T18:23:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.813 7fa087c7e700 1 -- 172.21.15.32:0/2402753080 shutdown_connections 2021-04-09T18:23:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.813 7fa087c7e700 1 --2- 172.21.15.32:0/2402753080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0800d8490 0x7fa08010da00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.813 7fa087c7e700 1 --2- 172.21.15.32:0/2402753080 >> v2:172.21.15.86:3300/0 conn(0x7fa08010d010 0x7fa0800d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.813 7fa087c7e700 1 -- 172.21.15.32:0/2402753080 >> 172.21.15.32:0/2402753080 conn(0x7fa0800ce5f0 msgr2=0x7fa0800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.814 7fa087c7e700 1 -- 172.21.15.32:0/2402753080 shutdown_connections 2021-04-09T18:23:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.814 7fa087c7e700 1 -- 172.21.15.32:0/2402753080 wait complete. 2021-04-09T18:23:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa087c7e700 1 Processor -- start 2021-04-09T18:23:22.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa087c7e700 1 -- start start 2021-04-09T18:23:22.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa087c7e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0800b2140 0x7fa08009d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:22.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa087c7e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa08009e010 0x7fa080194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:22.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa087c7e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0801950a0 0x7fa0800a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:22.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa087c7e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08009d470 con 0x7fa0800b2140 2021-04-09T18:23:22.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa087c7e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08009e410 con 0x7fa0801950a0 2021-04-09T18:23:22.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa087c7e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0801954a0 con 0x7fa08009e010 2021-04-09T18:23:22.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa085a9a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0800b2140 0x7fa08009d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa085a9a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0800b2140 0x7fa08009d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50244/0 (socket says 172.21.15.32:50244) 2021-04-09T18:23:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa085a9a700 1 -- 172.21.15.32:0/1267088236 learned_addr learned my addr 172.21.15.32:0/1267088236 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa08629b700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0801950a0 0x7fa0800a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.834 7fa085299700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa08009e010 0x7fa080194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:22.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa085a9a700 1 -- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa08009e010 msgr2=0x7fa080194a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:22.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa085a9a700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa08009e010 0x7fa080194a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa085a9a700 1 -- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0801950a0 msgr2=0x7fa0800a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:22.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa085a9a700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0801950a0 0x7fa0800a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:22.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa085a9a700 1 -- 172.21.15.32:0/1267088236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0801956c0 con 0x7fa0800b2140 2021-04-09T18:23:22.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa085a9a700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0800b2140 0x7fa08009d9f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7fa07800b910 tx=0x7fa07800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:22.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa07e7fc700 1 -- 172.21.15.32:0/1267088236 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa07800c2f0 con 0x7fa0800b2140 2021-04-09T18:23:22.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa07e7fc700 1 -- 172.21.15.32:0/1267088236 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07800c5f0 con 0x7fa0800b2140 2021-04-09T18:23:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa07e7fc700 1 -- 172.21.15.32:0/1267088236 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa07800cad0 con 0x7fa0800b2140 2021-04-09T18:23:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0801956c0 con 0x7fa0800b2140 2021-04-09T18:23:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.835 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0801956c0 con 0x7fa0800b2140 2021-04-09T18:23:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.836 7fa07e7fc700 1 -- 172.21.15.32:0/1267088236 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa078001010 con 0x7fa0800b2140 2021-04-09T18:23:22.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.836 7fa07e7fc700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0600188c0 0x7fa06001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:22.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.836 7fa07e7fc700 1 -- 172.21.15.32:0/1267088236 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa078032b70 con 0x7fa0800b2140 2021-04-09T18:23:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.836 7fa085299700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0600188c0 0x7fa06001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.836 7fa085299700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0600188c0 0x7fa06001adc0 crc :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:22.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.856 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0801b7040 con 0x7fa0800b2140 2021-04-09T18:23:22.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:22.859 7fa07e7fc700 1 -- 172.21.15.32:0/1267088236 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa07800bdf0 con 0x7fa0800b2140 2021-04-09T18:23:23.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.051 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7fa08010d480 con 0x7fa0800b2140 2021-04-09T18:23:23.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.604 7fa07e7fc700 1 -- 172.21.15.32:0/1267088236 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v167) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa078001010 con 0x7fa0800b2140 2021-04-09T18:23:23.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0600188c0 msgr2=0x7fa06001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:23.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0600188c0 0x7fa06001adc0 crc :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0800b2140 msgr2=0x7fa08009d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:23.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0800b2140 0x7fa08009d9f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7fa07800b910 tx=0x7fa07800bcb0).stop 2021-04-09T18:23:23.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 shutdown_connections 2021-04-09T18:23:23.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0600188c0 0x7fa06001adc0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0801950a0 0x7fa0800a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa08009e010 0x7fa080194a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 --2- 172.21.15.32:0/1267088236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0800b2140 0x7fa08009d9f0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 >> 172.21.15.32:0/1267088236 conn(0x7fa0800b27e0 msgr2=0x7fa0801b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 shutdown_connections 2021-04-09T18:23:23.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.618 7fa087c7e700 1 -- 172.21.15.32:0/1267088236 wait complete. 2021-04-09T18:23:23.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.780 7fdb466da700 1 Processor -- start 2021-04-09T18:23:23.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.780 7fdb466da700 1 -- start start 2021-04-09T18:23:23.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb466da700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400d7ca0 0x7fdb400d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:23.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb466da700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb4010da00 0x7fdb4010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:23.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb466da700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4010d480 con 0x7fdb4010d010 2021-04-09T18:23:23.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb466da700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb400d9220 con 0x7fdb4010da00 2021-04-09T18:23:23.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb466da700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb400d86c0 con 0x7fdb400d7ca0 2021-04-09T18:23:23.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb44cf7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb4010da00 0x7fdb4010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:23.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb44cf7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb4010da00 0x7fdb4010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50254/0 (socket says 172.21.15.32:50254) 2021-04-09T18:23:23.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb44cf7700 1 -- 172.21.15.32:0/1807139874 learned_addr learned my addr 172.21.15.32:0/1807139874 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:23.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb37fff700 1 --2- 172.21.15.32:0/1807139874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400d7ca0 0x7fdb400d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb44cf7700 1 -- 172.21.15.32:0/1807139874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400d7ca0 msgr2=0x7fdb400d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.781 7fdb44cf7700 1 --2- 172.21.15.32:0/1807139874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400d7ca0 0x7fdb400d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.782 7fdb44cf7700 1 -- 172.21.15.32:0/1807139874 >> v1:172.21.15.86:6789/0 conn(0x7fdb4010d010 legacy=0x7fdb400d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.782 7fdb44cf7700 1 -- 172.21.15.32:0/1807139874 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400d1f70 con 0x7fdb4010da00 2021-04-09T18:23:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.782 7fdb44cf7700 1 --2- 172.21.15.32:0/1807139874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb4010da00 0x7fdb4010de00 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7fdb38006f00 tx=0x7fdb38009f10).ready entity=mon.0 client_cookie=9ef2a85b1e4deb89 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.782 7fdb3f7fe700 1 -- 172.21.15.32:0/1807139874 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb3800b3b0 con 0x7fdb4010da00 2021-04-09T18:23:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.782 7fdb3f7fe700 1 -- 172.21.15.32:0/1807139874 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3800b730 con 0x7fdb4010da00 2021-04-09T18:23:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.782 7fdb3f7fe700 1 -- 172.21.15.32:0/1807139874 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb3800bc10 con 0x7fdb4010da00 2021-04-09T18:23:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.782 7fdb466da700 1 -- 172.21.15.32:0/1807139874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb4010da00 msgr2=0x7fdb4010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:23.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.782 7fdb466da700 1 --2- 172.21.15.32:0/1807139874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb4010da00 0x7fdb4010de00 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7fdb38006f00 tx=0x7fdb38009f10).stop 2021-04-09T18:23:23.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.783 7fdb466da700 1 -- 172.21.15.32:0/1807139874 shutdown_connections 2021-04-09T18:23:23.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.783 7fdb466da700 1 --2- 172.21.15.32:0/1807139874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb4010da00 0x7fdb4010de00 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.783 7fdb466da700 1 --2- 172.21.15.32:0/1807139874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400d7ca0 0x7fdb400d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.783 7fdb466da700 1 -- 172.21.15.32:0/1807139874 >> 172.21.15.32:0/1807139874 conn(0x7fdb400ce5f0 msgr2=0x7fdb400d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:23.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.783 7fdb466da700 1 -- 172.21.15.32:0/1807139874 shutdown_connections 2021-04-09T18:23:23.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.783 7fdb466da700 1 -- 172.21.15.32:0/1807139874 wait complete. 2021-04-09T18:23:23.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.803 7fdb466da700 1 Processor -- start 2021-04-09T18:23:23.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.803 7fdb466da700 1 -- start start 2021-04-09T18:23:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.803 7fdb466da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400b2140 0x7fdb4009d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.803 7fdb466da700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb4009e010 0x7fdb40194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.803 7fdb466da700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb401950a0 0x7fdb400a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:23.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.803 7fdb466da700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4009d470 con 0x7fdb400b2140 2021-04-09T18:23:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.803 7fdb466da700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4009e410 con 0x7fdb4009e010 2021-04-09T18:23:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.803 7fdb466da700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb401954a0 con 0x7fdb401950a0 2021-04-09T18:23:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400b2140 0x7fdb4009d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:23.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400b2140 0x7fdb4009d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50258/0 (socket says 172.21.15.32:50258) 2021-04-09T18:23:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3ffff700 1 -- 172.21.15.32:0/406322221 learned_addr learned my addr 172.21.15.32:0/406322221 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb44cf7700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb401950a0 0x7fdb400a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb37fff700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb4009e010 0x7fdb40194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:23.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3ffff700 1 -- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb401950a0 msgr2=0x7fdb400a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3ffff700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb401950a0 0x7fdb400a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3ffff700 1 -- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb4009e010 msgr2=0x7fdb40194a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3ffff700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb4009e010 0x7fdb40194a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3ffff700 1 -- 172.21.15.32:0/406322221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb401956c0 con 0x7fdb400b2140 2021-04-09T18:23:23.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3ffff700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400b2140 0x7fdb4009d9f0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7fdb3000b5b0 tx=0x7fdb3000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3d7fa700 1 -- 172.21.15.32:0/406322221 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb3000bd40 con 0x7fdb400b2140 2021-04-09T18:23:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3d7fa700 1 -- 172.21.15.32:0/406322221 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb30006240 con 0x7fdb400b2140 2021-04-09T18:23:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb466da700 1 -- 172.21.15.32:0/406322221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb401956c0 con 0x7fdb400b2140 2021-04-09T18:23:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb3d7fa700 1 -- 172.21.15.32:0/406322221 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb30005690 con 0x7fdb400b2140 2021-04-09T18:23:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.804 7fdb466da700 1 -- 172.21.15.32:0/406322221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb401956c0 con 0x7fdb400b2140 2021-04-09T18:23:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.805 7fdb3d7fa700 1 -- 172.21.15.32:0/406322221 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdb300054a0 con 0x7fdb400b2140 2021-04-09T18:23:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.805 7fdb3d7fa700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb240188d0 0x7fdb2401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:23.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.805 7fdb3d7fa700 1 -- 172.21.15.32:0/406322221 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdb30032a30 con 0x7fdb400b2140 2021-04-09T18:23:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.805 7fdb37fff700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb240188d0 0x7fdb2401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:23.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.805 7fdb37fff700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb240188d0 0x7fdb2401add0 crc :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:23.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.825 7fdb466da700 1 -- 172.21.15.32:0/406322221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb401b7040 con 0x7fdb400b2140 2021-04-09T18:23:23.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:23.828 7fdb3d7fa700 1 -- 172.21.15.32:0/406322221 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdb300009b0 con 0x7fdb400b2140 2021-04-09T18:23:24.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.019 7fdb466da700 1 -- 172.21.15.32:0/406322221 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fdb401b7040 con 0x7fdb400b2140 2021-04-09T18:23:24.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.613 7fdb3d7fa700 1 -- 172.21.15.32:0/406322221 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v169) v1 ==== 105+0+0 (secure 0 0 0) 0x7fdb300054a0 con 0x7fdb400b2140 2021-04-09T18:23:24.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 -- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb240188d0 msgr2=0x7fdb2401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb240188d0 0x7fdb2401add0 crc :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 -- 172.21.15.32:0/406322221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400b2140 msgr2=0x7fdb4009d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400b2140 0x7fdb4009d9f0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7fdb3000b5b0 tx=0x7fdb3000b950).stop 2021-04-09T18:23:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 -- 172.21.15.32:0/406322221 shutdown_connections 2021-04-09T18:23:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb240188d0 0x7fdb2401add0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb401950a0 0x7fdb400a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb4009e010 0x7fdb40194a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:24.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 --2- 172.21.15.32:0/406322221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400b2140 0x7fdb4009d9f0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:24.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 -- 172.21.15.32:0/406322221 >> 172.21.15.32:0/406322221 conn(0x7fdb400b27e0 msgr2=0x7fdb401b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 -- 172.21.15.32:0/406322221 shutdown_connections 2021-04-09T18:23:24.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.637 7fdb466da700 1 -- 172.21.15.32:0/406322221 wait complete. 2021-04-09T18:23:24.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:24.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.802 7fa79b28f700 1 Processor -- start 2021-04-09T18:23:24.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.802 7fa79b28f700 1 -- start start 2021-04-09T18:23:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa79b28f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7940d6ca0 0x7fa7940d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa79b28f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940d72a0 con 0x7fa7940d2460 2021-04-09T18:23:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa79b28f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940d2860 con 0x7fa7940cffc0 2021-04-09T18:23:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa79b28f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7940d70a0 con 0x7fa7940d6ca0 2021-04-09T18:23:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa7988aa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7940d6ca0 0x7fa7940d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa7998ac700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa7940d2460 0x7fa79410cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59858/0 (socket says 172.21.15.32:59858) 2021-04-09T18:23:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa7988aa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7940d6ca0 0x7fa7940d8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50270/0 (socket says 172.21.15.32:50270) 2021-04-09T18:23:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa7998ac700 1 -- 172.21.15.32:0/2767199177 learned_addr learned my addr 172.21.15.32:0/2767199177 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa7988aa700 1 -- 172.21.15.32:0/2767199177 >> v1:172.21.15.86:6789/0 conn(0x7fa7940cffc0 legacy=0x7fa7940d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa7988aa700 1 -- 172.21.15.32:0/2767199177 >> v1:172.21.15.32:6789/0 conn(0x7fa7940d2460 legacy=0x7fa79410cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa7988aa700 1 -- 172.21.15.32:0/2767199177 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7940d9200 con 0x7fa7940d6ca0 2021-04-09T18:23:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa78bfff700 1 -- 172.21.15.32:0/2767199177 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130449487 0 0) 0x7fa78c000d30 con 0x7fa7940d2460 2021-04-09T18:23:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa7988aa700 1 --2- 172.21.15.32:0/2767199177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940d6ca0 0x7fa7940d8010 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7fa784006650 tx=0x7fa784009a00).ready entity=mon.0 client_cookie=a8ff31e2704e0d83 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:24.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa78bfff700 1 -- 172.21.15.32:0/2767199177 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa784009df0 con 0x7fa7940d6ca0 2021-04-09T18:23:24.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa78bfff700 1 -- 172.21.15.32:0/2767199177 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7840009b0 con 0x7fa7940d6ca0 2021-04-09T18:23:24.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa78bfff700 1 -- 172.21.15.32:0/2767199177 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa78400c2b0 con 0x7fa7940d6ca0 2021-04-09T18:23:24.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa79b28f700 1 -- 172.21.15.32:0/2767199177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940d6ca0 msgr2=0x7fa7940d8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:24.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.803 7fa79b28f700 1 --2- 172.21.15.32:0/2767199177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940d6ca0 0x7fa7940d8010 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7fa784006650 tx=0x7fa784009a00).stop 2021-04-09T18:23:24.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.804 7fa79b28f700 1 -- 172.21.15.32:0/2767199177 shutdown_connections 2021-04-09T18:23:24.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.804 7fa79b28f700 1 --2- 172.21.15.32:0/2767199177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940d6ca0 0x7fa7940d8010 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:24.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.804 7fa79b28f700 1 -- 172.21.15.32:0/2767199177 >> 172.21.15.32:0/2767199177 conn(0x7fa7940ce5e0 msgr2=0x7fa7940cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:24.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.804 7fa79b28f700 1 -- 172.21.15.32:0/2767199177 shutdown_connections 2021-04-09T18:23:24.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.804 7fa79b28f700 1 -- 172.21.15.32:0/2767199177 wait complete. 2021-04-09T18:23:24.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.824 7fa79b28f700 1 Processor -- start 2021-04-09T18:23:24.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.824 7fa79b28f700 1 -- start start 2021-04-09T18:23:24.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa79b28f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940afd80 0x7fa7940a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:24.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa79b28f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7940b0180 0x7fa7940a25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:24.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa79b28f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7940a0b30 0x7fa7940a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa79b28f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940a1a00 con 0x7fa7940afd80 2021-04-09T18:23:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa79b28f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940a1550 con 0x7fa7940a0b30 2021-04-09T18:23:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa79b28f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940a1730 con 0x7fa7940b0180 2021-04-09T18:23:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa7990ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940afd80 0x7fa7940a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa7990ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940afd80 0x7fa7940a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50274/0 (socket says 172.21.15.32:50274) 2021-04-09T18:23:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa7990ab700 1 -- 172.21.15.32:0/1342672544 learned_addr learned my addr 172.21.15.32:0/1342672544 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa7988aa700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7940b0180 0x7fa7940a25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.825 7fa7998ac700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7940a0b30 0x7fa7940a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa7990ab700 1 -- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7940b0180 msgr2=0x7fa7940a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa7990ab700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7940b0180 0x7fa7940a25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa7990ab700 1 -- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7940a0b30 msgr2=0x7fa7940a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa7990ab700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7940a0b30 0x7fa7940a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa7990ab700 1 -- 172.21.15.32:0/1342672544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79408eab0 con 0x7fa7940afd80 2021-04-09T18:23:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa7988aa700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7940b0180 0x7fa7940a25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa7990ab700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940afd80 0x7fa7940a1f80 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fa79000b5b0 tx=0x7fa79000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa789ffb700 1 -- 172.21.15.32:0/1342672544 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa79000bd40 con 0x7fa7940afd80 2021-04-09T18:23:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa789ffb700 1 -- 172.21.15.32:0/1342672544 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa790006240 con 0x7fa7940afd80 2021-04-09T18:23:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79408eab0 con 0x7fa7940afd80 2021-04-09T18:23:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa789ffb700 1 -- 172.21.15.32:0/1342672544 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa790005690 con 0x7fa7940afd80 2021-04-09T18:23:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.826 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79408eab0 con 0x7fa7940afd80 2021-04-09T18:23:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.827 7fa789ffb700 1 -- 172.21.15.32:0/1342672544 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa7900054a0 con 0x7fa7940afd80 2021-04-09T18:23:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.827 7fa789ffb700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7740188d0 0x7fa77401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.827 7fa789ffb700 1 -- 172.21.15.32:0/1342672544 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa790032a30 con 0x7fa7940afd80 2021-04-09T18:23:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.827 7fa7988aa700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7740188d0 0x7fa77401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.827 7fa7988aa700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7740188d0 0x7fa77401add0 crc :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.847 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79408dba0 con 0x7fa7940afd80 2021-04-09T18:23:24.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:24.850 7fa789ffb700 1 -- 172.21.15.32:0/1342672544 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa790000c60 con 0x7fa7940afd80 2021-04-09T18:23:25.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.044 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fa7940935d0 con 0x7fa7940afd80 2021-04-09T18:23:25.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.614 7fa789ffb700 1 -- 172.21.15.32:0/1342672544 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v170) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa7900054a0 con 0x7fa7940afd80 2021-04-09T18:23:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.661 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7740188d0 msgr2=0x7fa77401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.661 7fa79b28f700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7740188d0 0x7fa77401add0 crc :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.661 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940afd80 msgr2=0x7fa7940a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:25.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.661 7fa79b28f700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940afd80 0x7fa7940a1f80 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fa79000b5b0 tx=0x7fa79000b950).stop 2021-04-09T18:23:25.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.662 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 shutdown_connections 2021-04-09T18:23:25.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.662 7fa79b28f700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7740188d0 0x7fa77401add0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.662 7fa79b28f700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7940a0b30 0x7fa7940a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.662 7fa79b28f700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7940b0180 0x7fa7940a25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.662 7fa79b28f700 1 --2- 172.21.15.32:0/1342672544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7940afd80 0x7fa7940a1f80 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.662 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 >> 172.21.15.32:0/1342672544 conn(0x7fa7940a3640 msgr2=0x7fa79408ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:25.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.662 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 shutdown_connections 2021-04-09T18:23:25.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.662 7fa79b28f700 1 -- 172.21.15.32:0/1342672544 wait complete. 2021-04-09T18:23:25.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.826 7f08b58ee700 1 Processor -- start 2021-04-09T18:23:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.826 7f08b58ee700 1 -- start start 2021-04-09T18:23:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.827 7f08b58ee700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08b010d010 0x7f08b00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.827 7f08b58ee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8490 0x7f08b010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.827 7f08b58ee700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b010d480 con 0x7f08b00d1da0 2021-04-09T18:23:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.827 7f08b58ee700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b00d8d30 con 0x7f08b010d010 2021-04-09T18:23:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.827 7f08b58ee700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00d9220 con 0x7f08b00d8490 2021-04-09T18:23:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.827 7f08af574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08b010d010 0x7f08b00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.827 7f08aed73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f08b00d1da0 0x7f08b00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59876/0 (socket says 172.21.15.32:59876) 2021-04-09T18:23:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.827 7f08aed73700 1 -- 172.21.15.32:0/1184119639 learned_addr learned my addr 172.21.15.32:0/1184119639 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.827 7f08afd75700 1 --2- 172.21.15.32:0/1184119639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8490 0x7f08b010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08af574700 1 -- 172.21.15.32:0/1184119639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8490 msgr2=0x7f08b010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08af574700 1 --2- 172.21.15.32:0/1184119639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8490 0x7f08b010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08af574700 1 -- 172.21.15.32:0/1184119639 >> v1:172.21.15.32:6789/0 conn(0x7f08b00d1da0 legacy=0x7f08b00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08af574700 1 -- 172.21.15.32:0/1184119639 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b010e020 con 0x7f08b010d010 2021-04-09T18:23:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08ae572700 1 -- 172.21.15.32:0/1184119639 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185491032 0 0) 0x7f08a4000ce0 con 0x7f08b00d1da0 2021-04-09T18:23:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08afd75700 1 --2- 172.21.15.32:0/1184119639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8490 0x7f08b010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08af574700 1 --2- 172.21.15.32:0/1184119639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b010d010 0x7f08b00d1780 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7f08a00069d0 tx=0x7f08a0009e40).ready entity=mon.0 client_cookie=aaeffb0936d6e821 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08ae572700 1 -- 172.21.15.32:0/1184119639 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a000b510 con 0x7f08b010d010 2021-04-09T18:23:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08ae572700 1 -- 172.21.15.32:0/1184119639 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a000b890 con 0x7f08b010d010 2021-04-09T18:23:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08ae572700 1 -- 172.21.15.32:0/1184119639 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a000bd70 con 0x7f08b010d010 2021-04-09T18:23:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08b58ee700 1 -- 172.21.15.32:0/1184119639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b010d010 msgr2=0x7f08b00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.828 7f08b58ee700 1 --2- 172.21.15.32:0/1184119639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b010d010 0x7f08b00d1780 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7f08a00069d0 tx=0x7f08a0009e40).stop 2021-04-09T18:23:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.829 7f08b58ee700 1 -- 172.21.15.32:0/1184119639 shutdown_connections 2021-04-09T18:23:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.829 7f08b58ee700 1 --2- 172.21.15.32:0/1184119639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00d8490 0x7f08b010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.829 7f08b58ee700 1 --2- 172.21.15.32:0/1184119639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b010d010 0x7f08b00d1780 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.829 7f08b58ee700 1 -- 172.21.15.32:0/1184119639 >> 172.21.15.32:0/1184119639 conn(0x7f08b00ce5f0 msgr2=0x7f08b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.829 7f08b58ee700 1 -- 172.21.15.32:0/1184119639 shutdown_connections 2021-04-09T18:23:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.829 7f08b58ee700 1 -- 172.21.15.32:0/1184119639 wait complete. 2021-04-09T18:23:25.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.849 7f08b58ee700 1 Processor -- start 2021-04-09T18:23:25.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.849 7f08b58ee700 1 -- start start 2021-04-09T18:23:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.849 7f08b58ee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00a8b00 0x7f08b018c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08b58ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00a8f00 0x7f08b018cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08b58ee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b00c2dd0 0x7f08b00c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08b58ee700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b018c300 con 0x7f08b00a8f00 2021-04-09T18:23:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08b58ee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c37f0 con 0x7f08b00c2dd0 2021-04-09T18:23:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08b58ee700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c39d0 con 0x7f08b00a8b00 2021-04-09T18:23:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08aed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00a8f00 0x7f08b018cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08aed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00a8f00 0x7f08b018cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50290/0 (socket says 172.21.15.32:50290) 2021-04-09T18:23:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08aed73700 1 -- 172.21.15.32:0/3936017099 learned_addr learned my addr 172.21.15.32:0/3936017099 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08af574700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00a8b00 0x7f08b018c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08afd75700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b00c2dd0 0x7f08b00c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08aed73700 1 -- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00a8b00 msgr2=0x7f08b018c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08aed73700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00a8b00 0x7f08b018c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08aed73700 1 -- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b00c2dd0 msgr2=0x7f08b00c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08aed73700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b00c2dd0 0x7f08b00c31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08aed73700 1 -- 172.21.15.32:0/3936017099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b00c3bf0 con 0x7f08b00a8f00 2021-04-09T18:23:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08af574700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00a8b00 0x7f08b018c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08afd75700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b00c2dd0 0x7f08b00c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:25.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08aed73700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00a8f00 0x7f08b018cea0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f08a4006f00 tx=0x7f08a400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:25.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f0897fff700 1 -- 172.21.15.32:0/3936017099 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a4006bd0 con 0x7f08b00a8f00 2021-04-09T18:23:25.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f0897fff700 1 -- 172.21.15.32:0/3936017099 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a400bd30 con 0x7f08b00a8f00 2021-04-09T18:23:25.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f0897fff700 1 -- 172.21.15.32:0/3936017099 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a4004590 con 0x7f08b00a8f00 2021-04-09T18:23:25.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b00c3bf0 con 0x7f08b00a8f00 2021-04-09T18:23:25.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.850 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b00c3bf0 con 0x7f08b00a8f00 2021-04-09T18:23:25.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.851 7f0897fff700 1 -- 172.21.15.32:0/3936017099 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f08a40043a0 con 0x7f08b00a8f00 2021-04-09T18:23:25.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.851 7f0897fff700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08900188c0 0x7f089001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:25.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.851 7f0897fff700 1 -- 172.21.15.32:0/3936017099 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f08a4032a20 con 0x7f08b00a8f00 2021-04-09T18:23:25.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.851 7f08af574700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08900188c0 0x7f089001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:25.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.851 7f08af574700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08900188c0 0x7f089001adc0 crc :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.871 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b010d480 con 0x7f08b00a8f00 2021-04-09T18:23:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:25.875 7f0897fff700 1 -- 172.21.15.32:0/3936017099 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f08a4006070 con 0x7f08b00a8f00 2021-04-09T18:23:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.071 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f08b010d480 con 0x7f08b00a8f00 2021-04-09T18:23:26.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.622 7f0897fff700 1 -- 172.21.15.32:0/3936017099 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v171) v1 ==== 105+0+0 (secure 0 0 0) 0x7f08a401f760 con 0x7f08b00a8f00 2021-04-09T18:23:26.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08900188c0 msgr2=0x7f089001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08900188c0 0x7f089001adc0 crc :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00a8f00 msgr2=0x7f08b018cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00a8f00 0x7f08b018cea0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f08a4006f00 tx=0x7f08a400b940).stop 2021-04-09T18:23:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 shutdown_connections 2021-04-09T18:23:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08900188c0 0x7f089001adc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08b00a8f00 0x7f08b018cea0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08b00c2dd0 0x7f08b00c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 --2- 172.21.15.32:0/3936017099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08b00a8b00 0x7f08b018c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 >> 172.21.15.32:0/3936017099 conn(0x7f08b00b93a0 msgr2=0x7f08b00b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:26.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.639 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 shutdown_connections 2021-04-09T18:23:26.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.640 7f08b58ee700 1 -- 172.21.15.32:0/3936017099 wait complete. 2021-04-09T18:23:26.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:26.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.811 7f62e32e4700 1 Processor -- start 2021-04-09T18:23:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.811 7f62e32e4700 1 -- start start 2021-04-09T18:23:26.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e32e4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62dc0d7ca0 0x7f62dc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:26.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e32e4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc10da00 0x7f62dc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:26.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e32e4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62dc10d480 con 0x7f62dc10d010 2021-04-09T18:23:26.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e32e4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc0d9220 con 0x7f62dc0d7ca0 2021-04-09T18:23:26.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e32e4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0d86c0 con 0x7f62dc10da00 2021-04-09T18:23:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e08ff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62dc0d7ca0 0x7f62dc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e1100700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f62dc10d010 0x7f62dc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59886/0 (socket says 172.21.15.32:59886) 2021-04-09T18:23:26.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e1100700 1 -- 172.21.15.32:0/2063901870 learned_addr learned my addr 172.21.15.32:0/2063901870 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:26.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e1901700 1 --2- 172.21.15.32:0/2063901870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc10da00 0x7f62dc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:26.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e08ff700 1 -- 172.21.15.32:0/2063901870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc10da00 msgr2=0x7f62dc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:26.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e08ff700 1 --2- 172.21.15.32:0/2063901870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc10da00 0x7f62dc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e08ff700 1 -- 172.21.15.32:0/2063901870 >> v1:172.21.15.32:6789/0 conn(0x7f62dc10d010 legacy=0x7f62dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:26.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e08ff700 1 -- 172.21.15.32:0/2063901870 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc0d1f70 con 0x7f62dc0d7ca0 2021-04-09T18:23:26.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62d3fff700 1 -- 172.21.15.32:0/2063901870 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618376396 0 0) 0x7f62d8000ce0 con 0x7f62dc10d010 2021-04-09T18:23:26.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.812 7f62e1901700 1 --2- 172.21.15.32:0/2063901870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc10da00 0x7f62dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:26.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62e08ff700 1 --2- 172.21.15.32:0/2063901870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc0d7ca0 0x7f62dc0d80a0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7f62cc006650 tx=0x7f62cc009a80).ready entity=mon.0 client_cookie=88215c316bd451fd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62d3fff700 1 -- 172.21.15.32:0/2063901870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62cc0009b0 con 0x7f62dc0d7ca0 2021-04-09T18:23:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62d3fff700 1 -- 172.21.15.32:0/2063901870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc00c0c0 con 0x7f62dc0d7ca0 2021-04-09T18:23:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62d3fff700 1 -- 172.21.15.32:0/2063901870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62cc00c510 con 0x7f62dc0d7ca0 2021-04-09T18:23:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62e32e4700 1 -- 172.21.15.32:0/2063901870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc0d7ca0 msgr2=0x7f62dc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62e32e4700 1 --2- 172.21.15.32:0/2063901870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc0d7ca0 0x7f62dc0d80a0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7f62cc006650 tx=0x7f62cc009a80).stop 2021-04-09T18:23:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62e32e4700 1 -- 172.21.15.32:0/2063901870 shutdown_connections 2021-04-09T18:23:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62e32e4700 1 --2- 172.21.15.32:0/2063901870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc10da00 0x7f62dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62e32e4700 1 --2- 172.21.15.32:0/2063901870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc0d7ca0 0x7f62dc0d80a0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.813 7f62e32e4700 1 -- 172.21.15.32:0/2063901870 >> 172.21.15.32:0/2063901870 conn(0x7f62dc0ce5f0 msgr2=0x7f62dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.814 7f62e32e4700 1 -- 172.21.15.32:0/2063901870 shutdown_connections 2021-04-09T18:23:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.814 7f62e32e4700 1 -- 172.21.15.32:0/2063901870 wait complete. 2021-04-09T18:23:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.834 7f62e32e4700 1 Processor -- start 2021-04-09T18:23:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.834 7f62e32e4700 1 -- start start 2021-04-09T18:23:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.834 7f62e32e4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc0c3c00 0x7f62dc09c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.834 7f62e32e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc095240 0x7f62dc1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.834 7f62e32e4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62dc1a1270 0x7f62dc0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.834 7f62e32e4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc09cda0 con 0x7f62dc095240 2021-04-09T18:23:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.834 7f62e32e4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0c3680 con 0x7f62dc1a1270 2021-04-09T18:23:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e32e4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc095640 con 0x7f62dc0c3c00 2021-04-09T18:23:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e08ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc095240 0x7f62dc1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e08ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc095240 0x7f62dc1a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50302/0 (socket says 172.21.15.32:50302) 2021-04-09T18:23:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e08ff700 1 -- 172.21.15.32:0/2466649206 learned_addr learned my addr 172.21.15.32:0/2466649206 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e1100700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc0c3c00 0x7f62dc09c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e1901700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62dc1a1270 0x7f62dc0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e08ff700 1 -- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc0c3c00 msgr2=0x7f62dc09c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e08ff700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc0c3c00 0x7f62dc09c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e08ff700 1 -- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62dc1a1270 msgr2=0x7f62dc0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e08ff700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62dc1a1270 0x7f62dc0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e08ff700 1 -- 172.21.15.32:0/2466649206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc1a1670 con 0x7f62dc095240 2021-04-09T18:23:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e1901700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62dc1a1270 0x7f62dc0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e1100700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc0c3c00 0x7f62dc09c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:26.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e08ff700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc095240 0x7f62dc1a0c50 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f62cc009650 tx=0x7f62cc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:26.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62d1ffb700 1 -- 172.21.15.32:0/2466649206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62cc0093a0 con 0x7f62dc095240 2021-04-09T18:23:26.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62d1ffb700 1 -- 172.21.15.32:0/2466649206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc00bd50 con 0x7f62dc095240 2021-04-09T18:23:26.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc1a1670 con 0x7f62dc095240 2021-04-09T18:23:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62d1ffb700 1 -- 172.21.15.32:0/2466649206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62cc0052e0 con 0x7f62dc095240 2021-04-09T18:23:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.835 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc1a1670 con 0x7f62dc095240 2021-04-09T18:23:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.836 7f62d1ffb700 1 -- 172.21.15.32:0/2466649206 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f62cc006030 con 0x7f62dc095240 2021-04-09T18:23:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.836 7f62d1ffb700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62bc018930 0x7f62bc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.836 7f62d1ffb700 1 -- 172.21.15.32:0/2466649206 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f62cc0329e0 con 0x7f62dc095240 2021-04-09T18:23:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.836 7f62e1100700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62bc018930 0x7f62bc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.837 7f62e1100700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62bc018930 0x7f62bc01ae10 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:26.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.856 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc093d50 con 0x7f62dc095240 2021-04-09T18:23:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:26.859 7f62d1ffb700 1 -- 172.21.15.32:0/2466649206 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f62cc0009c0 con 0x7f62dc095240 2021-04-09T18:23:27.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.053 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f62dc10d480 con 0x7f62dc095240 2021-04-09T18:23:27.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.627 7f62d1ffb700 1 -- 172.21.15.32:0/2466649206 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v172) v1 ==== 105+0+0 (secure 0 0 0) 0x7f62cc006030 con 0x7f62dc095240 2021-04-09T18:23:27.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62bc018930 msgr2=0x7f62bc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:27.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62bc018930 0x7f62bc01ae10 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc095240 msgr2=0x7f62dc1a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:27.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc095240 0x7f62dc1a0c50 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f62cc009650 tx=0x7f62cc00b960).stop 2021-04-09T18:23:27.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 shutdown_connections 2021-04-09T18:23:27.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62bc018930 0x7f62bc01ae10 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62dc095240 0x7f62dc1a0c50 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62dc1a1270 0x7f62dc0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 --2- 172.21.15.32:0/2466649206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62dc0c3c00 0x7f62dc09c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 >> 172.21.15.32:0/2466649206 conn(0x7f62dc0975f0 msgr2=0x7f62dc096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:27.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 shutdown_connections 2021-04-09T18:23:27.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.671 7f62e32e4700 1 -- 172.21.15.32:0/2466649206 wait complete. 2021-04-09T18:23:27.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:27.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.836 7f597492e700 1 Processor -- start 2021-04-09T18:23:27.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.836 7f597492e700 1 -- start start 2021-04-09T18:23:27.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.836 7f597492e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f59700cffc0 0x7f59700d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.836 7f597492e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700d7820 0x7f59700d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.836 7f597492e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59700d72a0 con 0x7f59700d8240 2021-04-09T18:23:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.836 7f597492e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59700d9200 con 0x7f59700cffc0 2021-04-09T18:23:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.836 7f597492e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597010e000 con 0x7f59700d7820 2021-04-09T18:23:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596e61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f59700cffc0 0x7f59700d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596e61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f59700cffc0 0x7f59700d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50310/0 (socket says 172.21.15.32:50310) 2021-04-09T18:23:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596e61c700 1 -- 172.21.15.32:0/2835075280 learned_addr learned my addr 172.21.15.32:0/2835075280 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596de1b700 1 --2- 172.21.15.32:0/2835075280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700d7820 0x7f59700d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596e61c700 1 -- 172.21.15.32:0/2835075280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700d7820 msgr2=0x7f59700d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596e61c700 1 --2- 172.21.15.32:0/2835075280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700d7820 0x7f59700d7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596e61c700 1 -- 172.21.15.32:0/2835075280 >> v1:172.21.15.86:6789/0 conn(0x7f59700d8240 legacy=0x7f59700d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596e61c700 1 -- 172.21.15.32:0/2835075280 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597010e220 con 0x7f59700cffc0 2021-04-09T18:23:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596e61c700 1 --2- 172.21.15.32:0/2835075280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59700cffc0 0x7f59700d2460 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f5960006700 tx=0x7f5960009b00).ready entity=mon.0 client_cookie=44551041c7e9841f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596d61a700 1 -- 172.21.15.32:0/2835075280 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f596000c0c0 con 0x7f59700cffc0 2021-04-09T18:23:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596d61a700 1 -- 172.21.15.32:0/2835075280 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596000c400 con 0x7f59700cffc0 2021-04-09T18:23:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f596d61a700 1 -- 172.21.15.32:0/2835075280 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f596000c8e0 con 0x7f59700cffc0 2021-04-09T18:23:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f597492e700 1 -- 172.21.15.32:0/2835075280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59700cffc0 msgr2=0x7f59700d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.837 7f597492e700 1 --2- 172.21.15.32:0/2835075280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59700cffc0 0x7f59700d2460 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f5960006700 tx=0x7f5960009b00).stop 2021-04-09T18:23:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.838 7f597492e700 1 -- 172.21.15.32:0/2835075280 shutdown_connections 2021-04-09T18:23:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.838 7f597492e700 1 --2- 172.21.15.32:0/2835075280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700d7820 0x7f59700d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.838 7f597492e700 1 --2- 172.21.15.32:0/2835075280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59700cffc0 0x7f59700d2460 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.838 7f597492e700 1 -- 172.21.15.32:0/2835075280 >> 172.21.15.32:0/2835075280 conn(0x7f59700ce5e0 msgr2=0x7f59700cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.838 7f597492e700 1 -- 172.21.15.32:0/2835075280 shutdown_connections 2021-04-09T18:23:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.838 7f597492e700 1 -- 172.21.15.32:0/2835075280 wait complete. 2021-04-09T18:23:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.858 7f597492e700 1 Processor -- start 2021-04-09T18:23:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.858 7f597492e700 1 -- start start 2021-04-09T18:23:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f597492e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5970182e60 0x7f5970183260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:27.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f597492e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597009d0f0 0x7f597009d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:27.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f597492e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5970181330 0x7f5970181730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:27.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f597492e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597009cb70 con 0x7f5970182e60 2021-04-09T18:23:27.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f597492e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597009db10 con 0x7f5970181330 2021-04-09T18:23:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f597492e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5970181d50 con 0x7f597009d0f0 2021-04-09T18:23:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596e61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5970182e60 0x7f5970183260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596e61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5970182e60 0x7f5970183260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50316/0 (socket says 172.21.15.32:50316) 2021-04-09T18:23:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596e61c700 1 -- 172.21.15.32:0/123467877 learned_addr learned my addr 172.21.15.32:0/123467877 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596de1b700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597009d0f0 0x7f597009d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:27.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596e61c700 1 -- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597009d0f0 msgr2=0x7f597009d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:27.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596e61c700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597009d0f0 0x7f597009d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596e61c700 1 -- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5970181330 msgr2=0x7f5970181730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:27.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596ee1d700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5970181330 0x7f5970181730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596e61c700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5970181330 0x7f5970181730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596e61c700 1 -- 172.21.15.32:0/123467877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5970181f70 con 0x7f5970182e60 2021-04-09T18:23:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596ee1d700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5970181330 0x7f5970181730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.859 7f596e61c700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5970182e60 0x7f5970183260 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f596000b590 tx=0x7f596000b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:27.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.860 7f595effd700 1 -- 172.21.15.32:0/123467877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f596000bd20 con 0x7f5970182e60 2021-04-09T18:23:27.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.860 7f595effd700 1 -- 172.21.15.32:0/123467877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5960006390 con 0x7f5970182e60 2021-04-09T18:23:27.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.860 7f597492e700 1 -- 172.21.15.32:0/123467877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5970181f70 con 0x7f5970182e60 2021-04-09T18:23:27.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.860 7f595effd700 1 -- 172.21.15.32:0/123467877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59600052e0 con 0x7f5970182e60 2021-04-09T18:23:27.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.860 7f597492e700 1 -- 172.21.15.32:0/123467877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5970181f70 con 0x7f5970182e60 2021-04-09T18:23:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.860 7f595effd700 1 -- 172.21.15.32:0/123467877 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5960006030 con 0x7f5970182e60 2021-04-09T18:23:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.861 7f595effd700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5954018930 0x7f595401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.861 7f595effd700 1 -- 172.21.15.32:0/123467877 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5960032a50 con 0x7f5970182e60 2021-04-09T18:23:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.861 7f596de1b700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5954018930 0x7f595401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.861 7f596de1b700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5954018930 0x7f595401adc0 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.880 7f597492e700 1 -- 172.21.15.32:0/123467877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5970093df0 con 0x7f5970182e60 2021-04-09T18:23:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:27.884 7f595effd700 1 -- 172.21.15.32:0/123467877 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f596000bd20 con 0x7f5970182e60 2021-04-09T18:23:28.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.078 7f597492e700 1 -- 172.21.15.32:0/123467877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f59700d72a0 con 0x7f5970182e60 2021-04-09T18:23:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.635 7f595effd700 1 -- 172.21.15.32:0/123467877 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v173) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5960006030 con 0x7f5970182e60 2021-04-09T18:23:28.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 -- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5954018930 msgr2=0x7f595401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:28.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5954018930 0x7f595401adc0 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:28.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 -- 172.21.15.32:0/123467877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5970182e60 msgr2=0x7f5970183260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:28.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5970182e60 0x7f5970183260 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f596000b590 tx=0x7f596000b930).stop 2021-04-09T18:23:28.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 -- 172.21.15.32:0/123467877 shutdown_connections 2021-04-09T18:23:28.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5954018930 0x7f595401adc0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:28.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5970181330 0x7f5970181730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:28.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597009d0f0 0x7f597009d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:28.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 --2- 172.21.15.32:0/123467877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5970182e60 0x7f5970183260 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:28.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 -- 172.21.15.32:0/123467877 >> 172.21.15.32:0/123467877 conn(0x7f597009fa00 msgr2=0x7f59700b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:28.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 -- 172.21.15.32:0/123467877 shutdown_connections 2021-04-09T18:23:28.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.645 7f597492e700 1 -- 172.21.15.32:0/123467877 wait complete. 2021-04-09T18:23:28.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:28.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.812 7f459b11e700 1 Processor -- start 2021-04-09T18:23:28.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.812 7f459b11e700 1 -- start start 2021-04-09T18:23:28.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.812 7f459b11e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f459410da00 0x7f459410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:28.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.812 7f459b11e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459410d480 con 0x7f459410d010 2021-04-09T18:23:28.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.812 7f459b11e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45940d1f70 con 0x7f45940d7ca0 2021-04-09T18:23:28.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.812 7f459b11e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45940d8d30 con 0x7f459410da00 2021-04-09T18:23:28.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f4598f3a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f459410d010 0x7f45940d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59914/0 (socket says 172.21.15.32:59914) 2021-04-09T18:23:28.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f4598f3a700 1 -- 172.21.15.32:0/2442811136 learned_addr learned my addr 172.21.15.32:0/2442811136 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:28.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f459973b700 1 --2- 172.21.15.32:0/2442811136 >> v2:172.21.15.86:3300/0 conn(0x7f459410da00 0x7f459410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:28.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f45937fe700 1 -- 172.21.15.32:0/2442811136 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521455280 0 0) 0x7f4584000ce0 con 0x7f459410d010 2021-04-09T18:23:28.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f45937fe700 1 -- 172.21.15.32:0/2442811136 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c002fe0 con 0x7f459410d010 2021-04-09T18:23:28.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f45937fe700 1 -- 172.21.15.32:0/2442811136 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881598072 0 0) 0x7f4588000c60 con 0x7f45940d7ca0 2021-04-09T18:23:28.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f45937fe700 1 -- 172.21.15.32:0/2442811136 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c003210 con 0x7f45940d7ca0 2021-04-09T18:23:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f45937fe700 1 -- 172.21.15.32:0/2442811136 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 525005517 0 0) 0x7f4584000ed0 con 0x7f459410d010 2021-04-09T18:23:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f459973b700 1 -- 172.21.15.32:0/2442811136 >> v1:172.21.15.86:6789/0 conn(0x7f45940d7ca0 legacy=0x7f45940d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f459973b700 1 -- 172.21.15.32:0/2442811136 >> v1:172.21.15.32:6789/0 conn(0x7f459410d010 legacy=0x7f45940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f459973b700 1 -- 172.21.15.32:0/2442811136 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45940d9220 con 0x7f459410da00 2021-04-09T18:23:28.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.813 7f459973b700 1 --2- 172.21.15.32:0/2442811136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f459410da00 0x7f459410de00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f458c006e50 tx=0x7f458c009df0).ready entity=mon.1 client_cookie=61d2f70f3b69c3ec server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f45937fe700 1 -- 172.21.15.32:0/2442811136 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f458c00b510 con 0x7f459410da00 2021-04-09T18:23:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f45937fe700 1 -- 172.21.15.32:0/2442811136 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c003030 con 0x7f459410da00 2021-04-09T18:23:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f45937fe700 1 -- 172.21.15.32:0/2442811136 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f458c000f90 con 0x7f459410da00 2021-04-09T18:23:28.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f459b11e700 1 -- 172.21.15.32:0/2442811136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f459410da00 msgr2=0x7f459410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:28.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f459b11e700 1 --2- 172.21.15.32:0/2442811136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f459410da00 0x7f459410de00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f458c006e50 tx=0x7f458c009df0).stop 2021-04-09T18:23:28.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f459b11e700 1 -- 172.21.15.32:0/2442811136 shutdown_connections 2021-04-09T18:23:28.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f459b11e700 1 --2- 172.21.15.32:0/2442811136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f459410da00 0x7f459410de00 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:28.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f459b11e700 1 -- 172.21.15.32:0/2442811136 >> 172.21.15.32:0/2442811136 conn(0x7f45940ce5f0 msgr2=0x7f45940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f459b11e700 1 -- 172.21.15.32:0/2442811136 shutdown_connections 2021-04-09T18:23:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.815 7f459b11e700 1 -- 172.21.15.32:0/2442811136 wait complete. 2021-04-09T18:23:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.835 7f459b11e700 1 Processor -- start 2021-04-09T18:23:28.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f459b11e700 1 -- start start 2021-04-09T18:23:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f459b11e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45940c3c00 0x7f459409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f459b11e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4594095240 0x7f45941a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f459b11e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45941a1270 0x7f45940b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f459b11e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459409cda0 con 0x7f4594095240 2021-04-09T18:23:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f459b11e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45940c3680 con 0x7f45941a1270 2021-04-09T18:23:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f459b11e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4594095640 con 0x7f45940c3c00 2021-04-09T18:23:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f4593fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4594095240 0x7f45941a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f4593fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4594095240 0x7f45941a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50332/0 (socket says 172.21.15.32:50332) 2021-04-09T18:23:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.836 7f4593fff700 1 -- 172.21.15.32:0/2602920200 learned_addr learned my addr 172.21.15.32:0/2602920200 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f4598f3a700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45940c3c00 0x7f459409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f459973b700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45941a1270 0x7f45940b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f4593fff700 1 -- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45940c3c00 msgr2=0x7f459409c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f4593fff700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45940c3c00 0x7f459409c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f4593fff700 1 -- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45941a1270 msgr2=0x7f45940b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f4593fff700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45941a1270 0x7f45940b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:28.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f4593fff700 1 -- 172.21.15.32:0/2602920200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45941a1670 con 0x7f4594095240 2021-04-09T18:23:28.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f4598f3a700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45940c3c00 0x7f459409c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:28.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f459973b700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45941a1270 0x7f45940b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:28.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f4593fff700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4594095240 0x7f45941a0c50 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f4588006f10 tx=0x7f458800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:28.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f45917fa700 1 -- 172.21.15.32:0/2602920200 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f458800bd80 con 0x7f4594095240 2021-04-09T18:23:28.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f45917fa700 1 -- 172.21.15.32:0/2602920200 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4588000e20 con 0x7f4594095240 2021-04-09T18:23:28.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f459b11e700 1 -- 172.21.15.32:0/2602920200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45941a1670 con 0x7f4594095240 2021-04-09T18:23:28.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f45917fa700 1 -- 172.21.15.32:0/2602920200 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45880054f0 con 0x7f4594095240 2021-04-09T18:23:28.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.837 7f459b11e700 1 -- 172.21.15.32:0/2602920200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45941a1670 con 0x7f4594095240 2021-04-09T18:23:28.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.838 7f45917fa700 1 -- 172.21.15.32:0/2602920200 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4588009fe0 con 0x7f4594095240 2021-04-09T18:23:28.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.838 7f45917fa700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4574018930 0x7f457401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:28.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.838 7f45917fa700 1 -- 172.21.15.32:0/2602920200 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4588032a40 con 0x7f4594095240 2021-04-09T18:23:28.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.838 7f4598f3a700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4574018930 0x7f457401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:28.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.838 7f4598f3a700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4574018930 0x7f457401ae10 crc :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:28.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.857 7f459b11e700 1 -- 172.21.15.32:0/2602920200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4594093d50 con 0x7f4594095240 2021-04-09T18:23:28.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:28.861 7f45917fa700 1 -- 172.21.15.32:0/2602920200 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4588000e20 con 0x7f4594095240 2021-04-09T18:23:29.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.054 7f459b11e700 1 -- 172.21.15.32:0/2602920200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f459410d480 con 0x7f4594095240 2021-04-09T18:23:29.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.637 7f45917fa700 1 -- 172.21.15.32:0/2602920200 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v174) v1 ==== 105+0+0 (secure 0 0 0) 0x7f4588009fe0 con 0x7f4594095240 2021-04-09T18:23:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 -- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4574018930 msgr2=0x7f457401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4574018930 0x7f457401ae10 crc :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 -- 172.21.15.32:0/2602920200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4594095240 msgr2=0x7f45941a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4594095240 0x7f45941a0c50 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f4588006f10 tx=0x7f458800b990).stop 2021-04-09T18:23:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 -- 172.21.15.32:0/2602920200 shutdown_connections 2021-04-09T18:23:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4574018930 0x7f457401ae10 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4594095240 0x7f45941a0c50 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45941a1270 0x7f45940b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 --2- 172.21.15.32:0/2602920200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45940c3c00 0x7f459409c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 -- 172.21.15.32:0/2602920200 >> 172.21.15.32:0/2602920200 conn(0x7f45940975f0 msgr2=0x7f4594096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 -- 172.21.15.32:0/2602920200 shutdown_connections 2021-04-09T18:23:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.672 7f459b11e700 1 -- 172.21.15.32:0/2602920200 wait complete. 2021-04-09T18:23:29.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:29.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.835 7f20ec76b700 1 Processor -- start 2021-04-09T18:23:29.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.835 7f20ec76b700 1 -- start start 2021-04-09T18:23:29.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.836 7f20ec76b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f20e410d010 0x7f20e40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:29.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.836 7f20ec76b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e410d480 con 0x7f20e40d1da0 2021-04-09T18:23:29.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.836 7f20ec76b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e410e1f0 con 0x7f20e40d8490 2021-04-09T18:23:29.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.836 7f20ec76b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e40d8d30 con 0x7f20e410d010 2021-04-09T18:23:29.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.836 7f20ea587700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f20e410d010 0x7f20e40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:29.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.836 7f20e9d86700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f20e40d1da0 0x7f20e40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59932/0 (socket says 172.21.15.32:59932) 2021-04-09T18:23:29.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.836 7f20ea587700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f20e410d010 0x7f20e40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50338/0 (socket says 172.21.15.32:50338) 2021-04-09T18:23:29.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.836 7f20e9d86700 1 -- 172.21.15.32:0/2668573400 learned_addr learned my addr 172.21.15.32:0/2668573400 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:29.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20ea587700 1 -- 172.21.15.32:0/2668573400 >> v1:172.21.15.86:6789/0 conn(0x7f20e40d8490 legacy=0x7f20e410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:29.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20ea587700 1 -- 172.21.15.32:0/2668573400 >> v1:172.21.15.32:6789/0 conn(0x7f20e40d1da0 legacy=0x7f20e40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:29.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20e9585700 1 -- 172.21.15.32:0/2668573400 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239114754 0 0) 0x7f20d4000ce0 con 0x7f20e40d1da0 2021-04-09T18:23:29.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20ea587700 1 -- 172.21.15.32:0/2668573400 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e40d9220 con 0x7f20e410d010 2021-04-09T18:23:29.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20ea587700 1 --2- 172.21.15.32:0/2668573400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e410d010 0x7f20e40d1780 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7f20e00069d0 tx=0x7f20e0009dc0).ready entity=mon.0 client_cookie=ec0b53d0072d6e62 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:29.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20e9585700 1 -- 172.21.15.32:0/2668573400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20e000b510 con 0x7f20e410d010 2021-04-09T18:23:29.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20e9585700 1 -- 172.21.15.32:0/2668573400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e0006ac0 con 0x7f20e410d010 2021-04-09T18:23:29.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20e9585700 1 -- 172.21.15.32:0/2668573400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20e000ba80 con 0x7f20e410d010 2021-04-09T18:23:29.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20ec76b700 1 -- 172.21.15.32:0/2668573400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e410d010 msgr2=0x7f20e40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:29.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.837 7f20ec76b700 1 --2- 172.21.15.32:0/2668573400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e410d010 0x7f20e40d1780 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7f20e00069d0 tx=0x7f20e0009dc0).stop 2021-04-09T18:23:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.838 7f20ec76b700 1 -- 172.21.15.32:0/2668573400 shutdown_connections 2021-04-09T18:23:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.838 7f20ec76b700 1 --2- 172.21.15.32:0/2668573400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e410d010 0x7f20e40d1780 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.838 7f20ec76b700 1 -- 172.21.15.32:0/2668573400 >> 172.21.15.32:0/2668573400 conn(0x7f20e40ce5f0 msgr2=0x7f20e40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.838 7f20ec76b700 1 -- 172.21.15.32:0/2668573400 shutdown_connections 2021-04-09T18:23:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.838 7f20ec76b700 1 -- 172.21.15.32:0/2668573400 wait complete. 2021-04-09T18:23:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.858 7f20ec76b700 1 Processor -- start 2021-04-09T18:23:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.858 7f20ec76b700 1 -- start start 2021-04-09T18:23:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ec76b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20e40aece0 0x7f20e40ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ec76b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20e40acf30 0x7f20e40aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ec76b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e4196fa0 0x7f20e40ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ec76b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e4092a70 con 0x7f20e4196fa0 2021-04-09T18:23:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ec76b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e40abba0 con 0x7f20e40acf30 2021-04-09T18:23:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ec76b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e40aa8a0 con 0x7f20e40aece0 2021-04-09T18:23:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ead88700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e4196fa0 0x7f20e40ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ead88700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e4196fa0 0x7f20e40ab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50346/0 (socket says 172.21.15.32:50346) 2021-04-09T18:23:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ead88700 1 -- 172.21.15.32:0/2998825067 learned_addr learned my addr 172.21.15.32:0/2998825067 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:29.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ea587700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20e40aece0 0x7f20e40ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:29.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20e9d86700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20e40acf30 0x7f20e40aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:29.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ead88700 1 -- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20e40aece0 msgr2=0x7f20e40ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:29.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ead88700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20e40aece0 0x7f20e40ac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:29.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ead88700 1 -- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20e40acf30 msgr2=0x7f20e40aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:29.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ead88700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20e40acf30 0x7f20e40aae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:29.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ead88700 1 -- 172.21.15.32:0/2998825067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e40a9680 con 0x7f20e4196fa0 2021-04-09T18:23:29.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20ead88700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e4196fa0 0x7f20e40ab440 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7f20dc00b5b0 tx=0x7f20dc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:29.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.859 7f20daffd700 1 -- 172.21.15.32:0/2998825067 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20dc000960 con 0x7f20e4196fa0 2021-04-09T18:23:29.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.860 7f20daffd700 1 -- 172.21.15.32:0/2998825067 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20dc006240 con 0x7f20e4196fa0 2021-04-09T18:23:29.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.860 7f20daffd700 1 -- 172.21.15.32:0/2998825067 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20dc005690 con 0x7f20e4196fa0 2021-04-09T18:23:29.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.860 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e40a9680 con 0x7f20e4196fa0 2021-04-09T18:23:29.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.860 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e40a9680 con 0x7f20e4196fa0 2021-04-09T18:23:29.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.860 7f20daffd700 1 -- 172.21.15.32:0/2998825067 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f20dc0054a0 con 0x7f20e4196fa0 2021-04-09T18:23:29.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.860 7f20daffd700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20cc0188d0 0x7f20cc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:29.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.860 7f20daffd700 1 -- 172.21.15.32:0/2998825067 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f20dc032a40 con 0x7f20e4196fa0 2021-04-09T18:23:29.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.860 7f20ea587700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20cc0188d0 0x7f20cc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:29.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.861 7f20ea587700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20cc0188d0 0x7f20cc01add0 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:29.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.880 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e408dab0 con 0x7f20e4196fa0 2021-04-09T18:23:29.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:29.884 7f20daffd700 1 -- 172.21.15.32:0/2998825067 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f20dc006030 con 0x7f20e4196fa0 2021-04-09T18:23:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.076 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f20e410d480 con 0x7f20e4196fa0 2021-04-09T18:23:30.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.641 7f20daffd700 1 -- 172.21.15.32:0/2998825067 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v175) v1 ==== 105+0+0 (secure 0 0 0) 0x7f20dc0054a0 con 0x7f20e4196fa0 2021-04-09T18:23:30.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20cc0188d0 msgr2=0x7f20cc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:30.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20cc0188d0 0x7f20cc01add0 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e4196fa0 msgr2=0x7f20e40ab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:30.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e4196fa0 0x7f20e40ab440 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7f20dc00b5b0 tx=0x7f20dc00b950).stop 2021-04-09T18:23:30.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 shutdown_connections 2021-04-09T18:23:30.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20cc0188d0 0x7f20cc01add0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20e4196fa0 0x7f20e40ab440 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20e40acf30 0x7f20e40aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 --2- 172.21.15.32:0/2998825067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20e40aece0 0x7f20e40ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 >> 172.21.15.32:0/2998825067 conn(0x7f20e40bb410 msgr2=0x7f20e408dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:30.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 shutdown_connections 2021-04-09T18:23:30.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.694 7f20ec76b700 1 -- 172.21.15.32:0/2998825067 wait complete. 2021-04-09T18:23:30.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:30.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.863 7f3c71109700 1 Processor -- start 2021-04-09T18:23:30.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.863 7f3c71109700 1 -- start start 2021-04-09T18:23:30.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.864 7f3c71109700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c0cffc0 0x7f3c6c0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:30.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.864 7f3c71109700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c6c0d6ca0 0x7f3c6c10cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:30.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.864 7f3c71109700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6c0d72a0 con 0x7f3c6c0d7820 2021-04-09T18:23:30.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.864 7f3c71109700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6c0d70a0 con 0x7f3c6c0d6ca0 2021-04-09T18:23:30.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.864 7f3c71109700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0d85a0 con 0x7f3c6c0cffc0 2021-04-09T18:23:30.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.864 7f3c6b59e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c6c0d6ca0 0x7f3c6c10cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:30.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.864 7f3c6b59e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c6c0d6ca0 0x7f3c6c10cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50354/0 (socket says 172.21.15.32:50354) 2021-04-09T18:23:30.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.864 7f3c6b59e700 1 -- 172.21.15.32:0/2953662494 learned_addr learned my addr 172.21.15.32:0/2953662494 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:30.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c6ad9d700 1 --2- 172.21.15.32:0/2953662494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c0cffc0 0x7f3c6c0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:30.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c6b59e700 1 -- 172.21.15.32:0/2953662494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c0cffc0 msgr2=0x7f3c6c0d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:30.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c6b59e700 1 --2- 172.21.15.32:0/2953662494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c0cffc0 0x7f3c6c0d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c6b59e700 1 -- 172.21.15.32:0/2953662494 >> v1:172.21.15.86:6789/0 conn(0x7f3c6c0d7820 legacy=0x7f3c6c0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:30.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c6b59e700 1 -- 172.21.15.32:0/2953662494 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c0d9200 con 0x7f3c6c0d6ca0 2021-04-09T18:23:30.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c6ad9d700 1 --2- 172.21.15.32:0/2953662494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c0cffc0 0x7f3c6c0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:30.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c6b59e700 1 --2- 172.21.15.32:0/2953662494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c0d6ca0 0x7f3c6c10cff0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f3c64006b80 tx=0x7f3c64009b90).ready entity=mon.0 client_cookie=99369b10bb24ee9e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:30.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c69d9b700 1 -- 172.21.15.32:0/2953662494 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c640022e0 con 0x7f3c6c0d6ca0 2021-04-09T18:23:30.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c69d9b700 1 -- 172.21.15.32:0/2953662494 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c64002620 con 0x7f3c6c0d6ca0 2021-04-09T18:23:30.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c69d9b700 1 -- 172.21.15.32:0/2953662494 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c64002b00 con 0x7f3c6c0d6ca0 2021-04-09T18:23:30.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c71109700 1 -- 172.21.15.32:0/2953662494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c0d6ca0 msgr2=0x7f3c6c10cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:30.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.865 7f3c71109700 1 --2- 172.21.15.32:0/2953662494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c0d6ca0 0x7f3c6c10cff0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f3c64006b80 tx=0x7f3c64009b90).stop 2021-04-09T18:23:30.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.866 7f3c71109700 1 -- 172.21.15.32:0/2953662494 shutdown_connections 2021-04-09T18:23:30.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.866 7f3c71109700 1 --2- 172.21.15.32:0/2953662494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c0d6ca0 0x7f3c6c10cff0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.866 7f3c71109700 1 --2- 172.21.15.32:0/2953662494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c0cffc0 0x7f3c6c0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.866 7f3c71109700 1 -- 172.21.15.32:0/2953662494 >> 172.21.15.32:0/2953662494 conn(0x7f3c6c0ce5e0 msgr2=0x7f3c6c0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:30.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.866 7f3c71109700 1 -- 172.21.15.32:0/2953662494 shutdown_connections 2021-04-09T18:23:30.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.866 7f3c71109700 1 -- 172.21.15.32:0/2953662494 wait complete. 2021-04-09T18:23:30.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.886 7f3c71109700 1 Processor -- start 2021-04-09T18:23:30.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.886 7f3c71109700 1 -- start start 2021-04-09T18:23:30.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c71109700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c1b82c0 0x7f3c6c0b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:30.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c71109700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c1b86c0 0x7f3c6c0b9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:30.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c71109700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6c0b9f60 0x7f3c6c0c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c71109700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0b93c0 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c71109700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0ba360 con 0x7f3c6c0b9f60 2021-04-09T18:23:30.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c71109700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c1b8db0 con 0x7f3c6c1b82c0 2021-04-09T18:23:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c1b86c0 0x7f3c6c0b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6a59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c1b86c0 0x7f3c6c0b9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50358/0 (socket says 172.21.15.32:50358) 2021-04-09T18:23:30.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6a59c700 1 -- 172.21.15.32:0/3969935928 learned_addr learned my addr 172.21.15.32:0/3969935928 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6ad9d700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c1b82c0 0x7f3c6c0b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6b59e700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6c0b9f60 0x7f3c6c0c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6a59c700 1 -- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c1b82c0 msgr2=0x7f3c6c0b89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6a59c700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c1b82c0 0x7f3c6c0b89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6a59c700 1 -- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6c0b9f60 msgr2=0x7f3c6c0c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:30.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6a59c700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6c0b9f60 0x7f3c6c0c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:30.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6a59c700 1 -- 172.21.15.32:0/3969935928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c0b8fc0 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6ad9d700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c1b82c0 0x7f3c6c0b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:30.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c6a59c700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c1b86c0 0x7f3c6c0b9940 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f3c60009940 tx=0x7f3c6000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c5b7fe700 1 -- 172.21.15.32:0/3969935928 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c60003030 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c5b7fe700 1 -- 172.21.15.32:0/3969935928 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c600011a0 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c5b7fe700 1 -- 172.21.15.32:0/3969935928 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c60001680 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.887 7f3c71109700 1 -- 172.21.15.32:0/3969935928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c0b8fc0 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.888 7f3c71109700 1 -- 172.21.15.32:0/3969935928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c0b8fc0 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.888 7f3c5b7fe700 1 -- 172.21.15.32:0/3969935928 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3c6000a000 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.888 7f3c5b7fe700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c50018930 0x7f3c5001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:30.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.888 7f3c5b7fe700 1 -- 172.21.15.32:0/3969935928 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3c60032dd0 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.889 7f3c6ad9d700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c50018930 0x7f3c5001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:30.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.889 7f3c6ad9d700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c50018930 0x7f3c5001ae10 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:30.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.908 7f3c71109700 1 -- 172.21.15.32:0/3969935928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6c0c3e60 con 0x7f3c6c1b86c0 2021-04-09T18:23:30.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:30.912 7f3c5b7fe700 1 -- 172.21.15.32:0/3969935928 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3c60001180 con 0x7f3c6c1b86c0 2021-04-09T18:23:31.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.104 7f3c71109700 1 -- 172.21.15.32:0/3969935928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f3c6c0d72a0 con 0x7f3c6c1b86c0 2021-04-09T18:23:31.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.642 7f3c5b7fe700 1 -- 172.21.15.32:0/3969935928 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v176) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3c60001180 con 0x7f3c6c1b86c0 2021-04-09T18:23:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.671 7f3c71109700 1 -- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c50018930 msgr2=0x7f3c5001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.671 7f3c71109700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c50018930 0x7f3c5001ae10 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:31.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.671 7f3c71109700 1 -- 172.21.15.32:0/3969935928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c1b86c0 msgr2=0x7f3c6c0b9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:31.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.671 7f3c71109700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c1b86c0 0x7f3c6c0b9940 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f3c60009940 tx=0x7f3c6000bcf0).stop 2021-04-09T18:23:31.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.671 7f3c71109700 1 -- 172.21.15.32:0/3969935928 shutdown_connections 2021-04-09T18:23:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.671 7f3c71109700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c50018930 0x7f3c5001ae10 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.671 7f3c71109700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6c1b86c0 0x7f3c6c0b9940 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.671 7f3c71109700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6c0b9f60 0x7f3c6c0c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.671 7f3c71109700 1 --2- 172.21.15.32:0/3969935928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c6c1b82c0 0x7f3c6c0b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.672 7f3c71109700 1 -- 172.21.15.32:0/3969935928 >> 172.21.15.32:0/3969935928 conn(0x7f3c6c0a06a0 msgr2=0x7f3c6c08fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.672 7f3c71109700 1 -- 172.21.15.32:0/3969935928 shutdown_connections 2021-04-09T18:23:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.672 7f3c71109700 1 -- 172.21.15.32:0/3969935928 wait complete. 2021-04-09T18:23:31.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.836 7f34b6352700 1 Processor -- start 2021-04-09T18:23:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.836 7f34b6352700 1 -- start start 2021-04-09T18:23:31.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.836 7f34b6352700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34b010d010 0x7f34b00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:31.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.836 7f34b6352700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b010d480 con 0x7f34b00d1da0 2021-04-09T18:23:31.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.836 7f34b6352700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b010e1f0 con 0x7f34b00d8490 2021-04-09T18:23:31.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.836 7f34b6352700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b00d8d30 con 0x7f34b010d010 2021-04-09T18:23:31.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.836 7f34affff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34b010d010 0x7f34b00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:31.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.836 7f34a7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f34b00d1da0 0x7f34b00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59956/0 (socket says 172.21.15.32:59956) 2021-04-09T18:23:31.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.836 7f34a7fff700 1 -- 172.21.15.32:0/714028249 learned_addr learned my addr 172.21.15.32:0/714028249 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:31.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34affff700 1 -- 172.21.15.32:0/714028249 >> v1:172.21.15.86:6789/0 conn(0x7f34b00d8490 legacy=0x7f34b010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:31.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34affff700 1 -- 172.21.15.32:0/714028249 >> v1:172.21.15.32:6789/0 conn(0x7f34b00d1da0 legacy=0x7f34b00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:31.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34affff700 1 -- 172.21.15.32:0/714028249 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b00d9220 con 0x7f34b010d010 2021-04-09T18:23:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34af7fe700 1 -- 172.21.15.32:0/714028249 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410559651 0 0) 0x7f3498000ce0 con 0x7f34b00d1da0 2021-04-09T18:23:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34affff700 1 --2- 172.21.15.32:0/714028249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b010d010 0x7f34b00d1780 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7f34a0006650 tx=0x7f34a0009a00).ready entity=mon.0 client_cookie=b402b16ae993cfc4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34af7fe700 1 -- 172.21.15.32:0/714028249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34a0009df0 con 0x7f34b010d010 2021-04-09T18:23:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34af7fe700 1 -- 172.21.15.32:0/714028249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a00009b0 con 0x7f34b010d010 2021-04-09T18:23:31.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34af7fe700 1 -- 172.21.15.32:0/714028249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34a0009df0 con 0x7f34b010d010 2021-04-09T18:23:31.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34b6352700 1 -- 172.21.15.32:0/714028249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b010d010 msgr2=0x7f34b00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:31.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.837 7f34b6352700 1 --2- 172.21.15.32:0/714028249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b010d010 0x7f34b00d1780 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7f34a0006650 tx=0x7f34a0009a00).stop 2021-04-09T18:23:31.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.838 7f34b6352700 1 -- 172.21.15.32:0/714028249 shutdown_connections 2021-04-09T18:23:31.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.838 7f34b6352700 1 --2- 172.21.15.32:0/714028249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b010d010 0x7f34b00d1780 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:31.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.838 7f34b6352700 1 -- 172.21.15.32:0/714028249 >> 172.21.15.32:0/714028249 conn(0x7f34b00ce5f0 msgr2=0x7f34b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:31.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.838 7f34b6352700 1 -- 172.21.15.32:0/714028249 shutdown_connections 2021-04-09T18:23:31.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.838 7f34b6352700 1 -- 172.21.15.32:0/714028249 wait complete. 2021-04-09T18:23:31.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.858 7f34b6352700 1 Processor -- start 2021-04-09T18:23:31.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.858 7f34b6352700 1 -- start start 2021-04-09T18:23:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b6352700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b017d240 0x7f34b0095350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b6352700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34b017d640 0x7f34b0093c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b6352700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0095970 0x7f34b0094270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b6352700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b0095d70 con 0x7f34b017d240 2021-04-09T18:23:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b6352700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b0094dd0 con 0x7f34b017d640 2021-04-09T18:23:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b6352700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b0094890 con 0x7f34b0095970 2021-04-09T18:23:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b017d240 0x7f34b0095350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34affff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b017d240 0x7f34b0095350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50370/0 (socket says 172.21.15.32:50370) 2021-04-09T18:23:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34affff700 1 -- 172.21.15.32:0/1306461373 learned_addr learned my addr 172.21.15.32:0/1306461373 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34a7fff700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34b017d640 0x7f34b0093c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b496f700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0095970 0x7f34b0094270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34affff700 1 -- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0095970 msgr2=0x7f34b0094270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34affff700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0095970 0x7f34b0094270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34affff700 1 -- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34b017d640 msgr2=0x7f34b0093c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34affff700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34b017d640 0x7f34b0093c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34affff700 1 -- 172.21.15.32:0/1306461373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b0094ab0 con 0x7f34b017d240 2021-04-09T18:23:31.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b496f700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0095970 0x7f34b0094270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34affff700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b017d240 0x7f34b0095350 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f34a0006f40 tx=0x7f34a000dbc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34ad7fa700 1 -- 172.21.15.32:0/1306461373 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34a0006030 con 0x7f34b017d240 2021-04-09T18:23:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34ad7fa700 1 -- 172.21.15.32:0/1306461373 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a00063c0 con 0x7f34b017d240 2021-04-09T18:23:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b6352700 1 -- 172.21.15.32:0/1306461373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b0094ab0 con 0x7f34b017d240 2021-04-09T18:23:31.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34ad7fa700 1 -- 172.21.15.32:0/1306461373 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34a0002030 con 0x7f34b017d240 2021-04-09T18:23:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.859 7f34b6352700 1 -- 172.21.15.32:0/1306461373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b0094ab0 con 0x7f34b017d240 2021-04-09T18:23:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.860 7f34ad7fa700 1 -- 172.21.15.32:0/1306461373 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f34a0000c90 con 0x7f34b017d240 2021-04-09T18:23:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.860 7f34ad7fa700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34940188c0 0x7f349401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.860 7f34ad7fa700 1 -- 172.21.15.32:0/1306461373 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f34a0034c60 con 0x7f34b017d240 2021-04-09T18:23:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.860 7f34a7fff700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34940188c0 0x7f349401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:31.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.861 7f34a7fff700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34940188c0 0x7f349401adc0 crc :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:31.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.880 7f34b6352700 1 -- 172.21.15.32:0/1306461373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b008dac0 con 0x7f34b017d240 2021-04-09T18:23:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:31.884 7f34ad7fa700 1 -- 172.21.15.32:0/1306461373 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f34a00063c0 con 0x7f34b017d240 2021-04-09T18:23:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.078 7f34b6352700 1 -- 172.21.15.32:0/1306461373 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f34b010d480 con 0x7f34b017d240 2021-04-09T18:23:32.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.649 7f34ad7fa700 1 -- 172.21.15.32:0/1306461373 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v177) v1 ==== 105+0+0 (secure 0 0 0) 0x7f34a0000c90 con 0x7f34b017d240 2021-04-09T18:23:32.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 -- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34940188c0 msgr2=0x7f349401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34940188c0 0x7f349401adc0 crc :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 -- 172.21.15.32:0/1306461373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b017d240 msgr2=0x7f34b0095350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:32.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b017d240 0x7f34b0095350 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f34a0006f40 tx=0x7f34a000dbc0).stop 2021-04-09T18:23:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 -- 172.21.15.32:0/1306461373 shutdown_connections 2021-04-09T18:23:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34940188c0 0x7f349401adc0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34b0095970 0x7f34b0094270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34b017d640 0x7f34b0093c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 --2- 172.21.15.32:0/1306461373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34b017d240 0x7f34b0095350 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 -- 172.21.15.32:0/1306461373 >> 172.21.15.32:0/1306461373 conn(0x7f34b00bb410 msgr2=0x7f34b00a13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:32.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 -- 172.21.15.32:0/1306461373 shutdown_connections 2021-04-09T18:23:32.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.695 7f34b6352700 1 -- 172.21.15.32:0/1306461373 wait complete. 2021-04-09T18:23:32.764 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:32.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.860 7fd2f1b08700 1 Processor -- start 2021-04-09T18:23:32.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.860 7fd2f1b08700 1 -- start start 2021-04-09T18:23:32.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2f1b08700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec10d010 0x7fd2ec0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:32.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2f1b08700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2ec0d1da0 0x7fd2ec0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2f1b08700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec10d480 con 0x7fd2ec0d82c0 2021-04-09T18:23:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2f1b08700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec10e780 con 0x7fd2ec0d1da0 2021-04-09T18:23:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2f1b08700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0d8d30 con 0x7fd2ec10d010 2021-04-09T18:23:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2eaffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2ec0d1da0 0x7fd2ec0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2eaffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2ec0d1da0 0x7fd2ec0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50382/0 (socket says 172.21.15.32:50382) 2021-04-09T18:23:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2eaffd700 1 -- 172.21.15.32:0/1884440004 learned_addr learned my addr 172.21.15.32:0/1884440004 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2eb7fe700 1 --2- 172.21.15.32:0/1884440004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec10d010 0x7fd2ec0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2eaffd700 1 -- 172.21.15.32:0/1884440004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec10d010 msgr2=0x7fd2ec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2eaffd700 1 --2- 172.21.15.32:0/1884440004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec10d010 0x7fd2ec0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2eaffd700 1 -- 172.21.15.32:0/1884440004 >> v1:172.21.15.86:6789/0 conn(0x7fd2ec0d82c0 legacy=0x7fd2ec10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2eaffd700 1 -- 172.21.15.32:0/1884440004 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0d9220 con 0x7fd2ec0d1da0 2021-04-09T18:23:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2eaffd700 1 --2- 172.21.15.32:0/1884440004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0d1da0 0x7fd2ec0d7ca0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fd2d8006700 tx=0x7fd2d8009b00).ready entity=mon.0 client_cookie=e5d94a0821b53c37 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.861 7fd2ea7fc700 1 -- 172.21.15.32:0/1884440004 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2d800c0c0 con 0x7fd2ec0d1da0 2021-04-09T18:23:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2ea7fc700 1 -- 172.21.15.32:0/1884440004 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d800c400 con 0x7fd2ec0d1da0 2021-04-09T18:23:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2ea7fc700 1 -- 172.21.15.32:0/1884440004 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2d800c8e0 con 0x7fd2ec0d1da0 2021-04-09T18:23:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2f1b08700 1 -- 172.21.15.32:0/1884440004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0d1da0 msgr2=0x7fd2ec0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2f1b08700 1 --2- 172.21.15.32:0/1884440004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0d1da0 0x7fd2ec0d7ca0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fd2d8006700 tx=0x7fd2d8009b00).stop 2021-04-09T18:23:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2f1b08700 1 -- 172.21.15.32:0/1884440004 shutdown_connections 2021-04-09T18:23:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2f1b08700 1 --2- 172.21.15.32:0/1884440004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec0d1da0 0x7fd2ec0d7ca0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2f1b08700 1 --2- 172.21.15.32:0/1884440004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec10d010 0x7fd2ec0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2f1b08700 1 -- 172.21.15.32:0/1884440004 >> 172.21.15.32:0/1884440004 conn(0x7fd2ec0ce5f0 msgr2=0x7fd2ec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2f1b08700 1 -- 172.21.15.32:0/1884440004 shutdown_connections 2021-04-09T18:23:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.862 7fd2f1b08700 1 -- 172.21.15.32:0/1884440004 wait complete. 2021-04-09T18:23:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.882 7fd2f1b08700 1 Processor -- start 2021-04-09T18:23:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2f1b08700 1 -- start start 2021-04-09T18:23:32.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2f1b08700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec17bb20 0x7fd2ec0c1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:32.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2f1b08700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ec0c1890 0x7fd2ec0b9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:32.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2f1b08700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec0ba4b0 0x7fd2ec0ba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:32.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2f1b08700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec180680 con 0x7fd2ec17bb20 2021-04-09T18:23:32.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2f1b08700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0c0cf0 con 0x7fd2ec0c1890 2021-04-09T18:23:32.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2f1b08700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec17c400 con 0x7fd2ec0ba4b0 2021-04-09T18:23:32.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2eb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec17bb20 0x7fd2ec0c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:32.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2eb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec17bb20 0x7fd2ec0c1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50384/0 (socket says 172.21.15.32:50384) 2021-04-09T18:23:32.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.883 7fd2eb7fe700 1 -- 172.21.15.32:0/3916100680 learned_addr learned my addr 172.21.15.32:0/3916100680 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:32.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2eaffd700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ec0c1890 0x7fd2ec0b9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:32.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2ebfff700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec0ba4b0 0x7fd2ec0ba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:32.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2eb7fe700 1 -- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec0ba4b0 msgr2=0x7fd2ec0ba8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:32.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2eb7fe700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec0ba4b0 0x7fd2ec0ba8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2eb7fe700 1 -- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ec0c1890 msgr2=0x7fd2ec0b9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:32.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2eb7fe700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ec0c1890 0x7fd2ec0b9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2eb7fe700 1 -- 172.21.15.32:0/3916100680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0c1c90 con 0x7fd2ec17bb20 2021-04-09T18:23:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2eb7fe700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec17bb20 0x7fd2ec0c1270 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7fd2dc006f10 tx=0x7fd2dc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2e3fff700 1 -- 172.21.15.32:0/3916100680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2dc00bd80 con 0x7fd2ec17bb20 2021-04-09T18:23:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2e3fff700 1 -- 172.21.15.32:0/3916100680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc000c90 con 0x7fd2ec17bb20 2021-04-09T18:23:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec0c1c90 con 0x7fd2ec17bb20 2021-04-09T18:23:32.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2e3fff700 1 -- 172.21.15.32:0/3916100680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2dc0022b0 con 0x7fd2ec17bb20 2021-04-09T18:23:32.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.884 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec0c1c90 con 0x7fd2ec17bb20 2021-04-09T18:23:32.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.885 7fd2e3fff700 1 -- 172.21.15.32:0/3916100680 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd2dc0052e0 con 0x7fd2ec17bb20 2021-04-09T18:23:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.885 7fd2e3fff700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2c8018930 0x7fd2c801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.885 7fd2e3fff700 1 -- 172.21.15.32:0/3916100680 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd2dc0329d0 con 0x7fd2ec17bb20 2021-04-09T18:23:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.885 7fd2eaffd700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2c8018930 0x7fd2c801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.885 7fd2eaffd700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2c8018930 0x7fd2c801ae10 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.905 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec10d480 con 0x7fd2ec17bb20 2021-04-09T18:23:32.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:32.908 7fd2e3fff700 1 -- 172.21.15.32:0/3916100680 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd2dc006ba0 con 0x7fd2ec17bb20 2021-04-09T18:23:33.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.104 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fd2ec10d480 con 0x7fd2ec17bb20 2021-04-09T18:23:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.651 7fd2e3fff700 1 -- 172.21.15.32:0/3916100680 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v178) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd2dc0052e0 con 0x7fd2ec17bb20 2021-04-09T18:23:33.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.670 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2c8018930 msgr2=0x7fd2c801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.670 7fd2f1b08700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2c8018930 0x7fd2c801ae10 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.670 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec17bb20 msgr2=0x7fd2ec0c1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.670 7fd2f1b08700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec17bb20 0x7fd2ec0c1270 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7fd2dc006f10 tx=0x7fd2dc00b990).stop 2021-04-09T18:23:33.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.671 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 shutdown_connections 2021-04-09T18:23:33.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.671 7fd2f1b08700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd2c8018930 0x7fd2c801ae10 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.671 7fd2f1b08700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2ec0ba4b0 0x7fd2ec0ba8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.671 7fd2f1b08700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2ec0c1890 0x7fd2ec0b9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.671 7fd2f1b08700 1 --2- 172.21.15.32:0/3916100680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2ec17bb20 0x7fd2ec0c1270 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.671 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 >> 172.21.15.32:0/3916100680 conn(0x7fd2ec181390 msgr2=0x7fd2ec0adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.671 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 shutdown_connections 2021-04-09T18:23:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.671 7fd2f1b08700 1 -- 172.21.15.32:0/3916100680 wait complete. 2021-04-09T18:23:33.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:33.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.833 7fde9377b700 1 Processor -- start 2021-04-09T18:23:33.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.833 7fde9377b700 1 -- start start 2021-04-09T18:23:33.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.833 7fde9377b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde8c10d010 0x7fde8c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:33.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde9377b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c10d480 con 0x7fde8c0d1da0 2021-04-09T18:23:33.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde9377b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c10e1f0 con 0x7fde8c0d8490 2021-04-09T18:23:33.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde9377b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c10e780 con 0x7fde8c10d010 2021-04-09T18:23:33.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde90d96700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fde8c0d1da0 0x7fde8c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:59982/0 (socket says 172.21.15.32:59982) 2021-04-09T18:23:33.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde90d96700 1 -- 172.21.15.32:0/3955209856 learned_addr learned my addr 172.21.15.32:0/3955209856 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde91597700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde8c10d010 0x7fde8c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569407821 0 0) 0x7fde7c000ce0 con 0x7fde8c0d1da0 2021-04-09T18:23:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde78002fe0 con 0x7fde8c0d1da0 2021-04-09T18:23:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911993425 0 0) 0x7fde84001400 con 0x7fde8c0d8490 2021-04-09T18:23:33.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde78003210 con 0x7fde8c0d8490 2021-04-09T18:23:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3647824701 0 0) 0x7fde7c000ed0 con 0x7fde8c0d1da0 2021-04-09T18:23:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde8c10d010 msgr2=0x7fde8c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde8c10d010 0x7fde8c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 >> v1:172.21.15.86:6789/0 conn(0x7fde8c0d8490 legacy=0x7fde8c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0d8d30 con 0x7fde8c0d1da0 2021-04-09T18:23:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fde7c001600 con 0x7fde8c0d1da0 2021-04-09T18:23:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 >> v1:172.21.15.32:6789/0 conn(0x7fde8c0d1da0 legacy=0x7fde8c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde78003b10 0x7fde78005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:33.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde78006770 0x7fde78008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde780094f0 0x7fde7800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde780502d0 con 0x7fde780094f0 2021-04-09T18:23:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde780504b0 con 0x7fde78006770 2021-04-09T18:23:33.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde83fff700 1 -- 172.21.15.32:0/3955209856 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde780506d0 con 0x7fde78003b10 2021-04-09T18:23:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.834 7fde91d98700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde780094f0 0x7fde7800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde91597700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde78003b10 0x7fde78005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde90d96700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde78006770 0x7fde78008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde91d98700 1 -- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde78003b10 msgr2=0x7fde78005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:33.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde91d98700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde78003b10 0x7fde78005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde91d98700 1 -- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde78006770 msgr2=0x7fde78008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:33.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde91d98700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde78006770 0x7fde78008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde91d98700 1 -- 172.21.15.32:0/3955209856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde78050970 con 0x7fde780094f0 2021-04-09T18:23:33.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde91d98700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde780094f0 0x7fde7800ba10 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7fde840013c0 tx=0x7fde84009e60).ready entity=mon.0 client_cookie=cabb0dbbd21c4b23 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:33.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde83fff700 1 -- 172.21.15.32:0/3955209856 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde84001400 con 0x7fde780094f0 2021-04-09T18:23:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde83fff700 1 -- 172.21.15.32:0/3955209856 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde84006a70 con 0x7fde780094f0 2021-04-09T18:23:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde83fff700 1 -- 172.21.15.32:0/3955209856 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde84001930 con 0x7fde780094f0 2021-04-09T18:23:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde9377b700 1 -- 172.21.15.32:0/3955209856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde780094f0 msgr2=0x7fde7800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:33.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde9377b700 1 --2- 172.21.15.32:0/3955209856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde780094f0 0x7fde7800ba10 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7fde840013c0 tx=0x7fde84009e60).stop 2021-04-09T18:23:33.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.835 7fde91d98700 1 -- 172.21.15.32:0/3955209856 reap_dead start 2021-04-09T18:23:33.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.836 7fde9377b700 1 -- 172.21.15.32:0/3955209856 shutdown_connections 2021-04-09T18:23:33.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.836 7fde9377b700 1 -- 172.21.15.32:0/3955209856 >> 172.21.15.32:0/3955209856 conn(0x7fde8c0ce5f0 msgr2=0x7fde8c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:33.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.836 7fde9377b700 1 -- 172.21.15.32:0/3955209856 shutdown_connections 2021-04-09T18:23:33.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.836 7fde9377b700 1 -- 172.21.15.32:0/3955209856 wait complete. 2021-04-09T18:23:33.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.856 7fde9377b700 1 Processor -- start 2021-04-09T18:23:33.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.856 7fde9377b700 1 -- start start 2021-04-09T18:23:33.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde9377b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde8c1a0050 0x7fde8c1a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:33.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde9377b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8c0bdcd0 0x7fde8c0bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:33.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde9377b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde8c0bf080 0x7fde8c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:33.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde9377b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0bf500 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde9377b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0bb7b0 con 0x7fde8c1a0050 2021-04-09T18:23:33.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde9377b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0be0d0 con 0x7fde8c0bf080 2021-04-09T18:23:33.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde90d96700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8c0bdcd0 0x7fde8c0bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:33.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde90d96700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8c0bdcd0 0x7fde8c0bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50402/0 (socket says 172.21.15.32:50402) 2021-04-09T18:23:33.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde90d96700 1 -- 172.21.15.32:0/2845491025 learned_addr learned my addr 172.21.15.32:0/2845491025 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:33.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde91d98700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde8c0bf080 0x7fde8c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:33.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde91597700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde8c1a0050 0x7fde8c1a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:33.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde90d96700 1 -- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde8c0bf080 msgr2=0x7fde8c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:33.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde90d96700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde8c0bf080 0x7fde8c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde90d96700 1 -- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde8c1a0050 msgr2=0x7fde8c1a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:33.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde90d96700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde8c1a0050 0x7fde8c1a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:33.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.857 7fde90d96700 1 -- 172.21.15.32:0/2845491025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0b1310 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.858 7fde90d96700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8c0bdcd0 0x7fde8c0bea60 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7fde7c007680 tx=0x7fde7c00a960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:33.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.858 7fde81ffb700 1 -- 172.21.15.32:0/2845491025 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde7c0070d0 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.858 7fde81ffb700 1 -- 172.21.15.32:0/2845491025 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c001480 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.858 7fde81ffb700 1 -- 172.21.15.32:0/2845491025 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde7c0045b0 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.858 7fde9377b700 1 -- 172.21.15.32:0/2845491025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c0b1310 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.858 7fde9377b700 1 -- 172.21.15.32:0/2845491025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c0b1310 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.858 7fde81ffb700 1 -- 172.21.15.32:0/2845491025 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fde7c001a90 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.859 7fde81ffb700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fde6c0188c0 0x7fde6c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:33.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.859 7fde81ffb700 1 -- 172.21.15.32:0/2845491025 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fde7c0327a0 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.859 7fde91597700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fde6c0188c0 0x7fde6c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:33.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.859 7fde91597700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fde6c0188c0 0x7fde6c01adc0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:33.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.878 7fde9377b700 1 -- 172.21.15.32:0/2845491025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8c093d50 con 0x7fde8c0bdcd0 2021-04-09T18:23:33.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:33.881 7fde81ffb700 1 -- 172.21.15.32:0/2845491025 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fde7c000940 con 0x7fde8c0bdcd0 2021-04-09T18:23:34.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.074 7fde9377b700 1 -- 172.21.15.32:0/2845491025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fde8c10d480 con 0x7fde8c0bdcd0 2021-04-09T18:23:34.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.657 7fde81ffb700 1 -- 172.21.15.32:0/2845491025 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v179) v1 ==== 105+0+0 (secure 0 0 0) 0x7fde7c001a90 con 0x7fde8c0bdcd0 2021-04-09T18:23:34.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.691 7fde9377b700 1 -- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fde6c0188c0 msgr2=0x7fde6c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:34.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.691 7fde9377b700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fde6c0188c0 0x7fde6c01adc0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.691 7fde9377b700 1 -- 172.21.15.32:0/2845491025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8c0bdcd0 msgr2=0x7fde8c0bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:34.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.691 7fde9377b700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8c0bdcd0 0x7fde8c0bea60 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7fde7c007680 tx=0x7fde7c00a960).stop 2021-04-09T18:23:34.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.691 7fde9377b700 1 -- 172.21.15.32:0/2845491025 shutdown_connections 2021-04-09T18:23:34.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.691 7fde9377b700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fde6c0188c0 0x7fde6c01adc0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.691 7fde9377b700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde8c0bdcd0 0x7fde8c0bea60 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.691 7fde9377b700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde8c0bf080 0x7fde8c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.692 7fde9377b700 1 --2- 172.21.15.32:0/2845491025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde8c1a0050 0x7fde8c1a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.692 7fde9377b700 1 -- 172.21.15.32:0/2845491025 >> 172.21.15.32:0/2845491025 conn(0x7fde8c0975f0 msgr2=0x7fde8c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:34.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.692 7fde9377b700 1 -- 172.21.15.32:0/2845491025 shutdown_connections 2021-04-09T18:23:34.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.692 7fde9377b700 1 -- 172.21.15.32:0/2845491025 wait complete. 2021-04-09T18:23:34.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.855 7f3d4e838700 1 Processor -- start 2021-04-09T18:23:34.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.855 7f3d4e838700 1 -- start start 2021-04-09T18:23:34.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.855 7f3d4e838700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d480d7ca0 0x7f3d480d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:34.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.855 7f3d4e838700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4810d480 con 0x7f3d4810d010 2021-04-09T18:23:34.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.855 7f3d4e838700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d480d86c0 con 0x7f3d4810da00 2021-04-09T18:23:34.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.855 7f3d4e838700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d480d9220 con 0x7f3d480d7ca0 2021-04-09T18:23:34.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d47fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d4810d010 0x7f3d480d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60000/0 (socket says 172.21.15.32:60000) 2021-04-09T18:23:34.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d47fff700 1 -- 172.21.15.32:0/271044343 learned_addr learned my addr 172.21.15.32:0/271044343 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:34.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d3ffff700 1 --2- 172.21.15.32:0/271044343 >> v2:172.21.15.86:3300/0 conn(0x7f3d480d7ca0 0x7f3d480d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:34.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207447094 0 0) 0x7f3d38000ce0 con 0x7f3d4810d010 2021-04-09T18:23:34.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34002fe0 con 0x7f3d4810d010 2021-04-09T18:23:34.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693308653 0 0) 0x7f3d40001400 con 0x7f3d4810da00 2021-04-09T18:23:34.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34003210 con 0x7f3d4810da00 2021-04-09T18:23:34.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1133426252 0 0) 0x7f3d38000ed0 con 0x7f3d4810d010 2021-04-09T18:23:34.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 >> v2:172.21.15.86:3300/0 conn(0x7f3d480d7ca0 msgr2=0x7f3d480d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:34.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 --2- 172.21.15.32:0/271044343 >> v2:172.21.15.86:3300/0 conn(0x7f3d480d7ca0 0x7f3d480d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 >> v1:172.21.15.86:6789/0 conn(0x7f3d4810da00 legacy=0x7f3d4810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:34.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d480d1f70 con 0x7f3d4810d010 2021-04-09T18:23:34.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f3d38001600 con 0x7f3d4810d010 2021-04-09T18:23:34.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 >> v1:172.21.15.32:6789/0 conn(0x7f3d4810d010 legacy=0x7f3d480d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:34.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d34003b90 0x7f3d34005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:34.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d34006840 0x7f3d34008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:34.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d34009640 0x7f3d3400bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:34.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d47fff700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d34003b90 0x7f3d34005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:34.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340503c0 con 0x7f3d34003b90 2021-04-09T18:23:34.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340505a0 con 0x7f3d34006840 2021-04-09T18:23:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d477fe700 1 -- 172.21.15.32:0/271044343 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340507c0 con 0x7f3d34009640 2021-04-09T18:23:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d3ffff700 1 --2- 172.21.15.32:0/271044343 >> v2:172.21.15.86:3300/0 conn(0x7f3d480d7ca0 0x7f3d480d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.856 7f3d4ce55700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d34009640 0x7f3d3400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d3ffff700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d34006840 0x7f3d34008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d47fff700 1 -- 172.21.15.32:0/271044343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d34009640 msgr2=0x7f3d3400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d47fff700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d34009640 0x7f3d3400bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d47fff700 1 -- 172.21.15.32:0/271044343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d34006840 msgr2=0x7f3d34008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d47fff700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d34006840 0x7f3d34008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d4ce55700 1 -- 172.21.15.32:0/271044343 reap_dead start 2021-04-09T18:23:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d47fff700 1 -- 172.21.15.32:0/271044343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34050a60 con 0x7f3d34003b90 2021-04-09T18:23:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d3ffff700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d34006840 0x7f3d34008d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d47fff700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d34003b90 0x7f3d34005fa0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7f3d380014a0 tx=0x7f3d38007c70).ready entity=mon.0 client_cookie=102c4ce258267337 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d477fe700 1 -- 172.21.15.32:0/271044343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d380070c0 con 0x7f3d34003b90 2021-04-09T18:23:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d477fe700 1 -- 172.21.15.32:0/271044343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3800a520 con 0x7f3d34003b90 2021-04-09T18:23:34.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d477fe700 1 -- 172.21.15.32:0/271044343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d3800a970 con 0x7f3d34003b90 2021-04-09T18:23:34.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d4e838700 1 -- 172.21.15.32:0/271044343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d34003b90 msgr2=0x7f3d34005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:34.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.857 7f3d4e838700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d34003b90 0x7f3d34005fa0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7f3d380014a0 tx=0x7f3d38007c70).stop 2021-04-09T18:23:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.858 7f3d4e838700 1 -- 172.21.15.32:0/271044343 shutdown_connections 2021-04-09T18:23:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.858 7f3d4e838700 1 --2- 172.21.15.32:0/271044343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d34003b90 0x7f3d34005fa0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.858 7f3d4e838700 1 -- 172.21.15.32:0/271044343 >> 172.21.15.32:0/271044343 conn(0x7f3d480ce5f0 msgr2=0x7f3d480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.858 7f3d4e838700 1 -- 172.21.15.32:0/271044343 shutdown_connections 2021-04-09T18:23:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.858 7f3d4e838700 1 -- 172.21.15.32:0/271044343 wait complete. 2021-04-09T18:23:34.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.878 7f3d4e838700 1 Processor -- start 2021-04-09T18:23:34.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.878 7f3d4e838700 1 -- start start 2021-04-09T18:23:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d4e838700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d4809e530 0x7f3d4809b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d4e838700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d4809e930 0x7f3d4809f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d4e838700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d4809f990 0x7f3d48094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d4e838700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4809edf0 con 0x7f3d4809e930 2021-04-09T18:23:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d4e838700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4809fd90 con 0x7f3d4809f990 2021-04-09T18:23:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d4e838700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4809bd40 con 0x7f3d4809e530 2021-04-09T18:23:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d3ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d4809e930 0x7f3d4809f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d3ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d4809e930 0x7f3d4809f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50424/0 (socket says 172.21.15.32:50424) 2021-04-09T18:23:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d3ffff700 1 -- 172.21.15.32:0/3362106579 learned_addr learned my addr 172.21.15.32:0/3362106579 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d4ce55700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d4809f990 0x7f3d48094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d47fff700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d4809e530 0x7f3d4809b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d3ffff700 1 -- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d4809e530 msgr2=0x7f3d4809b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d3ffff700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d4809e530 0x7f3d4809b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d3ffff700 1 -- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d4809f990 msgr2=0x7f3d48094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d3ffff700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d4809f990 0x7f3d48094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d3ffff700 1 -- 172.21.15.32:0/3362106579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d48095290 con 0x7f3d4809e930 2021-04-09T18:23:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d3ffff700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d4809e930 0x7f3d4809f370 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f3d30006be0 tx=0x7f3d3000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d457fa700 1 -- 172.21.15.32:0/3362106579 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d30006c20 con 0x7f3d4809e930 2021-04-09T18:23:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d457fa700 1 -- 172.21.15.32:0/3362106579 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d30006240 con 0x7f3d4809e930 2021-04-09T18:23:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.879 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d48095290 con 0x7f3d4809e930 2021-04-09T18:23:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.880 7f3d457fa700 1 -- 172.21.15.32:0/3362106579 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d300054d0 con 0x7f3d4809e930 2021-04-09T18:23:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.880 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d48095290 con 0x7f3d4809e930 2021-04-09T18:23:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.880 7f3d457fa700 1 -- 172.21.15.32:0/3362106579 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3d300052e0 con 0x7f3d4809e930 2021-04-09T18:23:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.880 7f3d457fa700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d2c018930 0x7f3d2c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.880 7f3d457fa700 1 -- 172.21.15.32:0/3362106579 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3d30032a00 con 0x7f3d4809e930 2021-04-09T18:23:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.880 7f3d47fff700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d2c018930 0x7f3d2c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.881 7f3d47fff700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d2c018930 0x7f3d2c01ae10 crc :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.899 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d48095b10 con 0x7f3d4809e930 2021-04-09T18:23:34.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:34.903 7f3d457fa700 1 -- 172.21.15.32:0/3362106579 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3d30006030 con 0x7f3d4809e930 2021-04-09T18:23:35.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.097 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f3d48095b10 con 0x7f3d4809e930 2021-04-09T18:23:35.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.658 7f3d457fa700 1 -- 172.21.15.32:0/3362106579 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v180) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3d30006240 con 0x7f3d4809e930 2021-04-09T18:23:35.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.664 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d2c018930 msgr2=0x7f3d2c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:35.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.664 7f3d4e838700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d2c018930 0x7f3d2c01ae10 crc :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d4809e930 msgr2=0x7f3d4809f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:35.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d4809e930 0x7f3d4809f370 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f3d30006be0 tx=0x7f3d3000b990).stop 2021-04-09T18:23:35.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 shutdown_connections 2021-04-09T18:23:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d2c018930 0x7f3d2c01ae10 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d4809e930 0x7f3d4809f370 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d4809f990 0x7f3d48094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 --2- 172.21.15.32:0/3362106579 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d4809e530 0x7f3d4809b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 >> 172.21.15.32:0/3362106579 conn(0x7f3d480c4290 msgr2=0x7f3d480c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:35.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 shutdown_connections 2021-04-09T18:23:35.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.665 7f3d4e838700 1 -- 172.21.15.32:0/3362106579 wait complete. 2021-04-09T18:23:35.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:35.778 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:23:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.832 7f8329608700 1 Processor -- start 2021-04-09T18:23:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.832 7f8329608700 1 -- start start 2021-04-09T18:23:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f8329608700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f83240cffc0 0x7f83240d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f8329608700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83240d72a0 con 0x7f83240d6ca0 2021-04-09T18:23:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f8329608700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83240d8410 con 0x7f83240d7820 2021-04-09T18:23:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f8329608700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83240d70a0 con 0x7f83240cffc0 2021-04-09T18:23:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f8323ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f83240d6ca0 0x7f832410cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60034/0 (socket says 172.21.15.32:60034) 2021-04-09T18:23:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f8323ac0700 1 -- 172.21.15.32:0/1314663586 learned_addr learned my addr 172.21.15.32:0/1314663586 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f83232bf700 1 --2- 172.21.15.32:0/1314663586 >> v2:172.21.15.86:3300/0 conn(0x7f83240cffc0 0x7f83240d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f83222bd700 1 -- 172.21.15.32:0/1314663586 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423653813 0 0) 0x7f831c000d30 con 0x7f83240d6ca0 2021-04-09T18:23:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f83222bd700 1 -- 172.21.15.32:0/1314663586 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8310002fe0 con 0x7f83240d6ca0 2021-04-09T18:23:35.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f83222bd700 1 -- 172.21.15.32:0/1314663586 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622132234 0 0) 0x7f830c000c60 con 0x7f83240d7820 2021-04-09T18:23:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f83222bd700 1 -- 172.21.15.32:0/1314663586 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8310003210 con 0x7f83240d7820 2021-04-09T18:23:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.833 7f83222bd700 1 -- 172.21.15.32:0/1314663586 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3225865207 0 0) 0x7f831c001270 con 0x7f83240d6ca0 2021-04-09T18:23:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 >> v2:172.21.15.86:3300/0 conn(0x7f83240cffc0 msgr2=0x7f83240d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 --2- 172.21.15.32:0/1314663586 >> v2:172.21.15.86:3300/0 conn(0x7f83240cffc0 0x7f83240d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 >> v1:172.21.15.86:6789/0 conn(0x7f83240d7820 legacy=0x7f83240d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:35.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83240d9200 con 0x7f83240d6ca0 2021-04-09T18:23:35.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f831c0019a0 con 0x7f83240d6ca0 2021-04-09T18:23:35.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 >> v1:172.21.15.32:6789/0 conn(0x7f83240d6ca0 legacy=0x7f832410cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:35.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8310003b90 0x7f8310005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:35.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8310006840 0x7f8310008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:35.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8310009640 0x7f831000bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:35.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83100503c0 con 0x7f8310003b90 2021-04-09T18:23:35.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83100505a0 con 0x7f8310006840 2021-04-09T18:23:35.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83100507c0 con 0x7f8310009640 2021-04-09T18:23:35.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83232bf700 1 --2- 172.21.15.32:0/1314663586 >> v2:172.21.15.86:3300/0 conn(0x7f83240cffc0 0x7f83240d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:35.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8322abe700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8310006840 0x7f8310008d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:35.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8323ac0700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8310009640 0x7f831000bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:35.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83232bf700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8310003b90 0x7f8310005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:35.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8322abe700 1 -- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8310009640 msgr2=0x7f831000bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8322abe700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8310009640 0x7f831000bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8322abe700 1 -- 172.21.15.32:0/1314663586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8310003b90 msgr2=0x7f8310005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8322abe700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8310003b90 0x7f8310005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8322abe700 1 -- 172.21.15.32:0/1314663586 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8310050a60 con 0x7f8310006840 2021-04-09T18:23:35.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8323ac0700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8310009640 0x7f831000bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:35.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83232bf700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8310003b90 0x7f8310005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:35.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8323ac0700 1 -- 172.21.15.32:0/1314663586 reap_dead start 2021-04-09T18:23:35.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8322abe700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8310006840 0x7f8310008d60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f830c006670 tx=0x7f830c009b40).ready entity=mon.1 client_cookie=1702a026feb69987 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:35.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f830c0022e0 con 0x7f8310006840 2021-04-09T18:23:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830c006760 con 0x7f8310006840 2021-04-09T18:23:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f83222bd700 1 -- 172.21.15.32:0/1314663586 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f830c002620 con 0x7f8310006840 2021-04-09T18:23:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8329608700 1 -- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8310006840 msgr2=0x7f8310008d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.834 7f8329608700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8310006840 0x7f8310008d60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f830c006670 tx=0x7f830c009b40).stop 2021-04-09T18:23:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.835 7f8329608700 1 -- 172.21.15.32:0/1314663586 shutdown_connections 2021-04-09T18:23:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.835 7f8329608700 1 --2- 172.21.15.32:0/1314663586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8310006840 0x7f8310008d60 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.835 7f8329608700 1 -- 172.21.15.32:0/1314663586 >> 172.21.15.32:0/1314663586 conn(0x7f83240ce5e0 msgr2=0x7f83240cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.835 7f8329608700 1 -- 172.21.15.32:0/1314663586 shutdown_connections 2021-04-09T18:23:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.835 7f8329608700 1 -- 172.21.15.32:0/1314663586 wait complete. 2021-04-09T18:23:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.855 7f8329608700 1 Processor -- start 2021-04-09T18:23:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.855 7f8329608700 1 -- start start 2021-04-09T18:23:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8329608700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83241833a0 0x7f83240a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8329608700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83240a2550 0x7f83240a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8329608700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8324094d00 0x7f8324095100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8329608700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83240a1fd0 con 0x7f8324094d00 2021-04-09T18:23:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8329608700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83240a2f70 con 0x7f83240a2550 2021-04-09T18:23:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8329608700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8324095720 con 0x7f83241833a0 2021-04-09T18:23:35.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8323ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8324094d00 0x7f8324095100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:35.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8323ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8324094d00 0x7f8324095100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50458/0 (socket says 172.21.15.32:50458) 2021-04-09T18:23:35.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8323ac0700 1 -- 172.21.15.32:0/1930873981 learned_addr learned my addr 172.21.15.32:0/1930873981 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8322abe700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83240a2550 0x7f83240a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f83232bf700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83241833a0 0x7f83240a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8323ac0700 1 -- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83241833a0 msgr2=0x7f83240a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8323ac0700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83241833a0 0x7f83240a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8323ac0700 1 -- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83240a2550 msgr2=0x7f83240a2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8323ac0700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83240a2550 0x7f83240a2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.856 7f8323ac0700 1 -- 172.21.15.32:0/1930873981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8324095940 con 0x7f8324094d00 2021-04-09T18:23:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.857 7f8323ac0700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8324094d00 0x7f8324095100 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7f831c00a8e0 tx=0x7f831c00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.857 7f831bfff700 1 -- 172.21.15.32:0/1930873981 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f831c007020 con 0x7f8324094d00 2021-04-09T18:23:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.857 7f831bfff700 1 -- 172.21.15.32:0/1930873981 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c0073a0 con 0x7f8324094d00 2021-04-09T18:23:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.857 7f8329608700 1 -- 172.21.15.32:0/1930873981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8324095940 con 0x7f8324094d00 2021-04-09T18:23:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.857 7f831bfff700 1 -- 172.21.15.32:0/1930873981 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f831c004720 con 0x7f8324094d00 2021-04-09T18:23:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.857 7f8329608700 1 -- 172.21.15.32:0/1930873981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8324095940 con 0x7f8324094d00 2021-04-09T18:23:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.858 7f831bfff700 1 -- 172.21.15.32:0/1930873981 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f831c007580 con 0x7f8324094d00 2021-04-09T18:23:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.858 7f831bfff700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83080188d0 0x7f830801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.858 7f831bfff700 1 -- 172.21.15.32:0/1930873981 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f831c032d80 con 0x7f8324094d00 2021-04-09T18:23:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.858 7f83232bf700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83080188d0 0x7f830801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.858 7f83232bf700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83080188d0 0x7f830801add0 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:35.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.877 7f8329608700 1 -- 172.21.15.32:0/1930873981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8324095be0 con 0x7f8324094d00 2021-04-09T18:23:35.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:35.881 7f831bfff700 1 -- 172.21.15.32:0/1930873981 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f831c001cd0 con 0x7f8324094d00 2021-04-09T18:23:36.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.075 7f8329608700 1 -- 172.21.15.32:0/1930873981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f8324095be0 con 0x7f8324094d00 2021-04-09T18:23:36.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:23:36.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:23:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.665 7f831bfff700 1 -- 172.21.15.32:0/1930873981 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v181) v1 ==== 105+0+0 (secure 0 0 0) 0x7f831c0073a0 con 0x7f8324094d00 2021-04-09T18:23:36.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 -- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83080188d0 msgr2=0x7f830801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:36.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83080188d0 0x7f830801add0 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 -- 172.21.15.32:0/1930873981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8324094d00 msgr2=0x7f8324095100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:36.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8324094d00 0x7f8324095100 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7f831c00a8e0 tx=0x7f831c00ac80).stop 2021-04-09T18:23:36.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 -- 172.21.15.32:0/1930873981 shutdown_connections 2021-04-09T18:23:36.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f83080188d0 0x7f830801add0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8324094d00 0x7f8324095100 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83240a2550 0x7f83240a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 --2- 172.21.15.32:0/1930873981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83241833a0 0x7f83240a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 -- 172.21.15.32:0/1930873981 >> 172.21.15.32:0/1930873981 conn(0x7f832409fa00 msgr2=0x7f83240b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:36.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 -- 172.21.15.32:0/1930873981 shutdown_connections 2021-04-09T18:23:36.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.692 7f8329608700 1 -- 172.21.15.32:0/1930873981 wait complete. 2021-04-09T18:23:36.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.855 7fe71a60c700 1 Processor -- start 2021-04-09T18:23:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.855 7fe71a60c700 1 -- start start 2021-04-09T18:23:36.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe71a60c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe71410d010 0x7fe7140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe71a60c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe7140d1da0 0x7fe7140d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe71a60c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71410d480 con 0x7fe7140d82c0 2021-04-09T18:23:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe71a60c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7140d8d30 con 0x7fe71410d010 2021-04-09T18:23:36.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe71a60c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7140d9220 con 0x7fe7140d1da0 2021-04-09T18:23:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe713fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe71410d010 0x7fe7140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe718c29700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe7140d82c0 0x7fe71410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60060/0 (socket says 172.21.15.32:60060) 2021-04-09T18:23:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe718c29700 1 -- 172.21.15.32:0/1224769703 learned_addr learned my addr 172.21.15.32:0/1224769703 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:36.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe70bfff700 1 --2- 172.21.15.32:0/1224769703 >> v2:172.21.15.86:3300/0 conn(0x7fe7140d1da0 0x7fe7140d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe713fff700 1 -- 172.21.15.32:0/1224769703 >> v2:172.21.15.86:3300/0 conn(0x7fe7140d1da0 msgr2=0x7fe7140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe713fff700 1 --2- 172.21.15.32:0/1224769703 >> v2:172.21.15.86:3300/0 conn(0x7fe7140d1da0 0x7fe7140d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe713fff700 1 -- 172.21.15.32:0/1224769703 >> v1:172.21.15.32:6789/0 conn(0x7fe7140d82c0 legacy=0x7fe71410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe713fff700 1 -- 172.21.15.32:0/1224769703 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7140d86f0 con 0x7fe71410d010 2021-04-09T18:23:36.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe7137fe700 1 -- 172.21.15.32:0/1224769703 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816670226 0 0) 0x7fe70c000d30 con 0x7fe7140d82c0 2021-04-09T18:23:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.856 7fe713fff700 1 --2- 172.21.15.32:0/1224769703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71410d010 0x7fe7140d1780 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7fe704006650 tx=0x7fe704009a00).ready entity=mon.0 client_cookie=258a0e917f5d545d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe7137fe700 1 -- 172.21.15.32:0/1224769703 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe704009df0 con 0x7fe71410d010 2021-04-09T18:23:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe7137fe700 1 -- 172.21.15.32:0/1224769703 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7040009b0 con 0x7fe71410d010 2021-04-09T18:23:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe7137fe700 1 -- 172.21.15.32:0/1224769703 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe70400c2b0 con 0x7fe71410d010 2021-04-09T18:23:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe71a60c700 1 -- 172.21.15.32:0/1224769703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71410d010 msgr2=0x7fe7140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe71a60c700 1 --2- 172.21.15.32:0/1224769703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71410d010 0x7fe7140d1780 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7fe704006650 tx=0x7fe704009a00).stop 2021-04-09T18:23:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe71a60c700 1 -- 172.21.15.32:0/1224769703 shutdown_connections 2021-04-09T18:23:36.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe71a60c700 1 --2- 172.21.15.32:0/1224769703 >> v2:172.21.15.86:3300/0 conn(0x7fe7140d1da0 0x7fe7140d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe71a60c700 1 --2- 172.21.15.32:0/1224769703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71410d010 0x7fe7140d1780 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe71a60c700 1 -- 172.21.15.32:0/1224769703 >> 172.21.15.32:0/1224769703 conn(0x7fe7140ce5f0 msgr2=0x7fe7140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe71a60c700 1 -- 172.21.15.32:0/1224769703 shutdown_connections 2021-04-09T18:23:36.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.857 7fe71a60c700 1 -- 172.21.15.32:0/1224769703 wait complete. 2021-04-09T18:23:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe71a60c700 1 Processor -- start 2021-04-09T18:23:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe71a60c700 1 -- start start 2021-04-09T18:23:36.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe71a60c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71408a910 0x7fe71409e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:36.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe71a60c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71408ad10 0x7fe71409ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:36.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe71a60c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7140922e0 0x7fe7140926e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:36.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe71a60c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71418fc90 con 0x7fe71408a910 2021-04-09T18:23:36.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe71a60c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71409e090 con 0x7fe7140922e0 2021-04-09T18:23:36.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe71a60c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71408b400 con 0x7fe71408ad10 2021-04-09T18:23:36.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe713fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71408a910 0x7fe71409e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:36.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe713fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71408a910 0x7fe71409e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50474/0 (socket says 172.21.15.32:50474) 2021-04-09T18:23:36.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe713fff700 1 -- 172.21.15.32:0/2452264036 learned_addr learned my addr 172.21.15.32:0/2452264036 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:36.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe718c29700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7140922e0 0x7fe7140926e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:36.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe70bfff700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71408ad10 0x7fe71409ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:36.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe713fff700 1 -- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71408ad10 msgr2=0x7fe71409ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:36.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.878 7fe713fff700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71408ad10 0x7fe71409ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe713fff700 1 -- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7140922e0 msgr2=0x7fe7140926e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:36.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe713fff700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7140922e0 0x7fe7140926e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:36.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe713fff700 1 -- 172.21.15.32:0/2452264036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe714092d00 con 0x7fe71408a910 2021-04-09T18:23:36.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe70bfff700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71408ad10 0x7fe71409ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:36.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe718c29700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7140922e0 0x7fe7140926e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe713fff700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71408a910 0x7fe71409e610 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7fe7040096a0 tx=0x7fe70400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe7117fa700 1 -- 172.21.15.32:0/2452264036 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe70400bd80 con 0x7fe71408a910 2021-04-09T18:23:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe7117fa700 1 -- 172.21.15.32:0/2452264036 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe704006030 con 0x7fe71408a910 2021-04-09T18:23:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe714092d00 con 0x7fe71408a910 2021-04-09T18:23:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe7117fa700 1 -- 172.21.15.32:0/2452264036 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe704005560 con 0x7fe71408a910 2021-04-09T18:23:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe714092d00 con 0x7fe71408a910 2021-04-09T18:23:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.879 7fe7117fa700 1 -- 172.21.15.32:0/2452264036 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe7040052e0 con 0x7fe71408a910 2021-04-09T18:23:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.880 7fe7117fa700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6f8018930 0x7fe6f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:36.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.880 7fe7117fa700 1 -- 172.21.15.32:0/2452264036 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe704032a80 con 0x7fe71408a910 2021-04-09T18:23:36.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.880 7fe70bfff700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6f8018930 0x7fe6f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:36.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.880 7fe70bfff700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6f8018930 0x7fe6f801ae10 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:36.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.900 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71418fc90 con 0x7fe71408a910 2021-04-09T18:23:36.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:36.903 7fe7117fa700 1 -- 172.21.15.32:0/2452264036 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe7040009a0 con 0x7fe71408a910 2021-04-09T18:23:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.097 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fe71418fc90 con 0x7fe71408a910 2021-04-09T18:23:37.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.672 7fe7117fa700 1 -- 172.21.15.32:0/2452264036 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v183) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe7040052e0 con 0x7fe71408a910 2021-04-09T18:23:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6f8018930 msgr2=0x7fe6f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6f8018930 0x7fe6f801ae10 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71408a910 msgr2=0x7fe71409e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71408a910 0x7fe71409e610 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7fe7040096a0 tx=0x7fe70400b990).stop 2021-04-09T18:23:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 shutdown_connections 2021-04-09T18:23:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6f8018930 0x7fe6f801ae10 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7140922e0 0x7fe7140926e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71408ad10 0x7fe71409ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 --2- 172.21.15.32:0/2452264036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe71408a910 0x7fe71409e610 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 >> 172.21.15.32:0/2452264036 conn(0x7fe7140b2fa0 msgr2=0x7fe7140a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 shutdown_connections 2021-04-09T18:23:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.715 7fe71a60c700 1 -- 172.21.15.32:0/2452264036 wait complete. 2021-04-09T18:23:37.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:37.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.880 7f5af7da2700 1 Processor -- start 2021-04-09T18:23:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.880 7f5af7da2700 1 -- start start 2021-04-09T18:23:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af7da2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af010d010 0x7f5af00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:37.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af7da2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5af00d1da0 0x7f5af00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af7da2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af010d480 con 0x7f5af00d82c0 2021-04-09T18:23:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af7da2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af010e780 con 0x7f5af00d1da0 2021-04-09T18:23:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af7da2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00d8d30 con 0x7f5af010d010 2021-04-09T18:23:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af53bd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5af00d1da0 0x7f5af00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af53bd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5af00d1da0 0x7f5af00d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50490/0 (socket says 172.21.15.32:50490) 2021-04-09T18:23:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af53bd700 1 -- 172.21.15.32:0/1730145449 learned_addr learned my addr 172.21.15.32:0/1730145449 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af5bbe700 1 --2- 172.21.15.32:0/1730145449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af010d010 0x7f5af00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af53bd700 1 -- 172.21.15.32:0/1730145449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af010d010 msgr2=0x7f5af00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af53bd700 1 --2- 172.21.15.32:0/1730145449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af010d010 0x7f5af00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af53bd700 1 -- 172.21.15.32:0/1730145449 >> v1:172.21.15.86:6789/0 conn(0x7f5af00d82c0 legacy=0x7f5af010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af53bd700 1 -- 172.21.15.32:0/1730145449 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af00d9220 con 0x7f5af00d1da0 2021-04-09T18:23:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.881 7f5af5bbe700 1 --2- 172.21.15.32:0/1730145449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af010d010 0x7f5af00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.882 7f5af53bd700 1 --2- 172.21.15.32:0/1730145449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00d1da0 0x7f5af00d7ca0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f5ae0006700 tx=0x7f5ae0009b80).ready entity=mon.0 client_cookie=9efdb55249220ffa server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:37.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.882 7f5af4bbc700 1 -- 172.21.15.32:0/1730145449 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ae000c0c0 con 0x7f5af00d1da0 2021-04-09T18:23:37.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.882 7f5af4bbc700 1 -- 172.21.15.32:0/1730145449 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae000c440 con 0x7f5af00d1da0 2021-04-09T18:23:37.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.882 7f5af4bbc700 1 -- 172.21.15.32:0/1730145449 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ae000c920 con 0x7f5af00d1da0 2021-04-09T18:23:37.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.882 7f5af7da2700 1 -- 172.21.15.32:0/1730145449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00d1da0 msgr2=0x7f5af00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:37.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.882 7f5af7da2700 1 --2- 172.21.15.32:0/1730145449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00d1da0 0x7f5af00d7ca0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f5ae0006700 tx=0x7f5ae0009b80).stop 2021-04-09T18:23:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.883 7f5af7da2700 1 -- 172.21.15.32:0/1730145449 shutdown_connections 2021-04-09T18:23:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.883 7f5af7da2700 1 --2- 172.21.15.32:0/1730145449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00d1da0 0x7f5af00d7ca0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.883 7f5af7da2700 1 --2- 172.21.15.32:0/1730145449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af010d010 0x7f5af00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.883 7f5af7da2700 1 -- 172.21.15.32:0/1730145449 >> 172.21.15.32:0/1730145449 conn(0x7f5af00ce5f0 msgr2=0x7f5af00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.883 7f5af7da2700 1 -- 172.21.15.32:0/1730145449 shutdown_connections 2021-04-09T18:23:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.883 7f5af7da2700 1 -- 172.21.15.32:0/1730145449 wait complete. 2021-04-09T18:23:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.903 7f5af7da2700 1 Processor -- start 2021-04-09T18:23:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.903 7f5af7da2700 1 -- start start 2021-04-09T18:23:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.903 7f5af7da2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5af01a0050 0x7f5af01a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.903 7f5af7da2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00bdcd0 0x7f5af00bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.903 7f5af7da2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af00bf080 0x7f5af00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.903 7f5af7da2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00bf500 con 0x7f5af00bdcd0 2021-04-09T18:23:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.903 7f5af7da2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00bb7b0 con 0x7f5af01a0050 2021-04-09T18:23:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.903 7f5af7da2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00bb960 con 0x7f5af00bf080 2021-04-09T18:23:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af53bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00bdcd0 0x7f5af00bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af53bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00bdcd0 0x7f5af00bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50498/0 (socket says 172.21.15.32:50498) 2021-04-09T18:23:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af53bd700 1 -- 172.21.15.32:0/1460954838 learned_addr learned my addr 172.21.15.32:0/1460954838 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af5bbe700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5af01a0050 0x7f5af01a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af63bf700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af00bf080 0x7f5af00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af53bd700 1 -- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af00bf080 msgr2=0x7f5af00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af53bd700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af00bf080 0x7f5af00b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af53bd700 1 -- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5af01a0050 msgr2=0x7f5af01a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af53bd700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5af01a0050 0x7f5af01a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af53bd700 1 -- 172.21.15.32:0/1460954838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af00be0d0 con 0x7f5af00bdcd0 2021-04-09T18:23:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af53bd700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00bdcd0 0x7f5af00bea60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f5ae000b580 tx=0x7f5ae000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5ae67fc700 1 -- 172.21.15.32:0/1460954838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ae000bd10 con 0x7f5af00bdcd0 2021-04-09T18:23:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5ae67fc700 1 -- 172.21.15.32:0/1460954838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae0000c90 con 0x7f5af00bdcd0 2021-04-09T18:23:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af00be0d0 con 0x7f5af00bdcd0 2021-04-09T18:23:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5ae67fc700 1 -- 172.21.15.32:0/1460954838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ae0002220 con 0x7f5af00bdcd0 2021-04-09T18:23:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.904 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af00be0d0 con 0x7f5af00bdcd0 2021-04-09T18:23:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.905 7f5ae67fc700 1 -- 172.21.15.32:0/1460954838 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5ae0009f90 con 0x7f5af00bdcd0 2021-04-09T18:23:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.905 7f5ae67fc700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ad8018930 0x7f5ad801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.905 7f5ae67fc700 1 -- 172.21.15.32:0/1460954838 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5ae00329f0 con 0x7f5af00bdcd0 2021-04-09T18:23:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.905 7f5af5bbe700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ad8018930 0x7f5ad801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.905 7f5af5bbe700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ad8018930 0x7f5ad801ae10 crc :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.925 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af0093d50 con 0x7f5af00bdcd0 2021-04-09T18:23:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:37.928 7f5ae67fc700 1 -- 172.21.15.32:0/1460954838 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5ae00091e0 con 0x7f5af00bdcd0 2021-04-09T18:23:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.123 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f5af0093d50 con 0x7f5af00bdcd0 2021-04-09T18:23:38.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.675 7f5ae67fc700 1 -- 172.21.15.32:0/1460954838 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v184) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5ae0009f90 con 0x7f5af00bdcd0 2021-04-09T18:23:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ad8018930 msgr2=0x7f5ad801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ad8018930 0x7f5ad801ae10 crc :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00bdcd0 msgr2=0x7f5af00bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00bdcd0 0x7f5af00bea60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f5ae000b580 tx=0x7f5ae000b920).stop 2021-04-09T18:23:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 shutdown_connections 2021-04-09T18:23:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ad8018930 0x7f5ad801ae10 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af00bdcd0 0x7f5af00bea60 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af00bf080 0x7f5af00b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 --2- 172.21.15.32:0/1460954838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5af01a0050 0x7f5af01a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 >> 172.21.15.32:0/1460954838 conn(0x7f5af00975f0 msgr2=0x7f5af0096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 shutdown_connections 2021-04-09T18:23:38.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.691 7f5af7da2700 1 -- 172.21.15.32:0/1460954838 wait complete. 2021-04-09T18:23:38.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:38.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.856 7fb8cfdef700 1 Processor -- start 2021-04-09T18:23:38.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.856 7fb8cfdef700 1 -- start start 2021-04-09T18:23:38.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.857 7fb8cfdef700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c810d010 0x7fb8c80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:38.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.857 7fb8cfdef700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb8c80d1da0 0x7fb8c80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.857 7fb8cfdef700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c810d480 con 0x7fb8c80d82c0 2021-04-09T18:23:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.857 7fb8cfdef700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c810e780 con 0x7fb8c80d1da0 2021-04-09T18:23:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.857 7fb8cfdef700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80d8d30 con 0x7fb8c810d010 2021-04-09T18:23:38.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.857 7fb8cd40a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb8c80d1da0 0x7fb8c80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.857 7fb8cdc0b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c810d010 0x7fb8c80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.857 7fb8ce40c700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb8c80d82c0 0x7fb8c810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58020/0 (socket says 172.21.15.32:58020) 2021-04-09T18:23:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.857 7fb8ce40c700 1 -- 172.21.15.32:0/1088601485 learned_addr learned my addr 172.21.15.32:0/1088601485 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:38.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8cd40a700 1 -- 172.21.15.32:0/1088601485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c810d010 msgr2=0x7fb8c80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8cd40a700 1 --2- 172.21.15.32:0/1088601485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c810d010 0x7fb8c80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8cd40a700 1 -- 172.21.15.32:0/1088601485 >> v1:172.21.15.86:6789/0 conn(0x7fb8c80d82c0 legacy=0x7fb8c810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8cd40a700 1 -- 172.21.15.32:0/1088601485 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c80d9220 con 0x7fb8c80d1da0 2021-04-09T18:23:38.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8ccc09700 1 -- 172.21.15.32:0/1088601485 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672541767 0 0) 0x7fb8c0001440 con 0x7fb8c80d82c0 2021-04-09T18:23:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8cdc0b700 1 --2- 172.21.15.32:0/1088601485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c810d010 0x7fb8c80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8cd40a700 1 --2- 172.21.15.32:0/1088601485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80d1da0 0x7fb8c80d7ca0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7fb8b8006650 tx=0x7fb8b8009a80).ready entity=mon.1 client_cookie=73a2944e9ee4c90c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8ccc09700 1 -- 172.21.15.32:0/1088601485 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b80009b0 con 0x7fb8c80d1da0 2021-04-09T18:23:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8ccc09700 1 -- 172.21.15.32:0/1088601485 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b800c0c0 con 0x7fb8c80d1da0 2021-04-09T18:23:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8ccc09700 1 -- 172.21.15.32:0/1088601485 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b800c510 con 0x7fb8c80d1da0 2021-04-09T18:23:38.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8cfdef700 1 -- 172.21.15.32:0/1088601485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80d1da0 msgr2=0x7fb8c80d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:38.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.858 7fb8cfdef700 1 --2- 172.21.15.32:0/1088601485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80d1da0 0x7fb8c80d7ca0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7fb8b8006650 tx=0x7fb8b8009a80).stop 2021-04-09T18:23:38.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.859 7fb8cfdef700 1 -- 172.21.15.32:0/1088601485 shutdown_connections 2021-04-09T18:23:38.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.859 7fb8cfdef700 1 --2- 172.21.15.32:0/1088601485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80d1da0 0x7fb8c80d7ca0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.859 7fb8cfdef700 1 --2- 172.21.15.32:0/1088601485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c810d010 0x7fb8c80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.859 7fb8cfdef700 1 -- 172.21.15.32:0/1088601485 >> 172.21.15.32:0/1088601485 conn(0x7fb8c80ce5f0 msgr2=0x7fb8c80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.859 7fb8cfdef700 1 -- 172.21.15.32:0/1088601485 shutdown_connections 2021-04-09T18:23:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.859 7fb8cfdef700 1 -- 172.21.15.32:0/1088601485 wait complete. 2021-04-09T18:23:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.879 7fb8cfdef700 1 Processor -- start 2021-04-09T18:23:38.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.879 7fb8cfdef700 1 -- start start 2021-04-09T18:23:38.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cfdef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80b1680 0x7fb8c80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cfdef700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c809e330 0x7fb8c80b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cfdef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c80b2190 0x7fb8c809f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cfdef700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c809e730 con 0x7fb8c80b2190 2021-04-09T18:23:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cfdef700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c809f6c0 con 0x7fb8c80b1680 2021-04-09T18:23:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cfdef700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c809f8e0 con 0x7fb8c809e330 2021-04-09T18:23:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8ce40c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c80b2190 0x7fb8c809f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8ce40c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c80b2190 0x7fb8c809f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50510/0 (socket says 172.21.15.32:50510) 2021-04-09T18:23:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8ce40c700 1 -- 172.21.15.32:0/3229295915 learned_addr learned my addr 172.21.15.32:0/3229295915 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cd40a700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c809e330 0x7fb8c80b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:38.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cdc0b700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80b1680 0x7fb8c80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:38.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8ce40c700 1 -- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c809e330 msgr2=0x7fb8c80b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:38.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8ce40c700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c809e330 0x7fb8c80b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8ce40c700 1 -- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80b1680 msgr2=0x7fb8c80b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8ce40c700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80b1680 0x7fb8c80b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8ce40c700 1 -- 172.21.15.32:0/3229295915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c809fb00 con 0x7fb8c80b2190 2021-04-09T18:23:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cdc0b700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80b1680 0x7fb8c80b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8ce40c700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c80b2190 0x7fb8c809f0a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7fb8c000a8a0 tx=0x7fb8c000ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:38.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8be7fc700 1 -- 172.21.15.32:0/3229295915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8c00048b0 con 0x7fb8c80b2190 2021-04-09T18:23:38.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8be7fc700 1 -- 172.21.15.32:0/3229295915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c0004bb0 con 0x7fb8c80b2190 2021-04-09T18:23:38.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c809fb00 con 0x7fb8c80b2190 2021-04-09T18:23:38.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.880 7fb8be7fc700 1 -- 172.21.15.32:0/3229295915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8c0007020 con 0x7fb8c80b2190 2021-04-09T18:23:38.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.881 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c809fb00 con 0x7fb8c80b2190 2021-04-09T18:23:38.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.881 7fb8be7fc700 1 -- 172.21.15.32:0/3229295915 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb8c00048b0 con 0x7fb8c80b2190 2021-04-09T18:23:38.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.881 7fb8be7fc700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8b0018930 0x7fb8b001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:38.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.881 7fb8be7fc700 1 -- 172.21.15.32:0/3229295915 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb8c0032d70 con 0x7fb8c80b2190 2021-04-09T18:23:38.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.881 7fb8cdc0b700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8b0018930 0x7fb8b001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:38.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.882 7fb8cdc0b700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8b0018930 0x7fb8b001ae10 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:38.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.901 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c809eb70 con 0x7fb8c80b2190 2021-04-09T18:23:38.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:38.904 7fb8be7fc700 1 -- 172.21.15.32:0/3229295915 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb8c00046a0 con 0x7fb8c80b2190 2021-04-09T18:23:39.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.098 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fb8c809eb70 con 0x7fb8c80b2190 2021-04-09T18:23:39.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.680 7fb8be7fc700 1 -- 172.21.15.32:0/3229295915 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v186) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb8c00048b0 con 0x7fb8c80b2190 2021-04-09T18:23:39.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8b0018930 msgr2=0x7fb8b001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:39.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8b0018930 0x7fb8b001ae10 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:39.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c80b2190 msgr2=0x7fb8c809f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:39.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c80b2190 0x7fb8c809f0a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7fb8c000a8a0 tx=0x7fb8c000ac40).stop 2021-04-09T18:23:39.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 shutdown_connections 2021-04-09T18:23:39.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8b0018930 0x7fb8b001ae10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:39.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c80b2190 0x7fb8c809f0a0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:39.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c809e330 0x7fb8c80b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:39.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 --2- 172.21.15.32:0/3229295915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c80b1680 0x7fb8c80b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:39.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 >> 172.21.15.32:0/3229295915 conn(0x7fb8c80c4290 msgr2=0x7fb8c80c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:39.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 shutdown_connections 2021-04-09T18:23:39.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.717 7fb8cfdef700 1 -- 172.21.15.32:0/3229295915 wait complete. 2021-04-09T18:23:39.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:39.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.882 7f7e69d98700 1 Processor -- start 2021-04-09T18:23:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.882 7f7e69d98700 1 -- start start 2021-04-09T18:23:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.882 7f7e69d98700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6410da00 0x7f7e6410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.882 7f7e69d98700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e6410d480 con 0x7f7e6410d010 2021-04-09T18:23:39.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.882 7f7e69d98700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e640d1f70 con 0x7f7e640d7ca0 2021-04-09T18:23:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.882 7f7e69d98700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640d9220 con 0x7f7e6410da00 2021-04-09T18:23:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.882 7f7e637fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7e6410d010 0x7f7e640d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60106/0 (socket says 172.21.15.32:60106) 2021-04-09T18:23:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.882 7f7e637fe700 1 -- 172.21.15.32:0/4223071799 learned_addr learned my addr 172.21.15.32:0/4223071799 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.882 7f7e63fff700 1 --2- 172.21.15.32:0/4223071799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6410da00 0x7f7e6410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e627fc700 1 -- 172.21.15.32:0/4223071799 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682139293 0 0) 0x7f7e54000ce0 con 0x7f7e6410d010 2021-04-09T18:23:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e627fc700 1 -- 172.21.15.32:0/4223071799 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e4c002fe0 con 0x7f7e6410d010 2021-04-09T18:23:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e627fc700 1 -- 172.21.15.32:0/4223071799 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525838682 0 0) 0x7f7e58000c60 con 0x7f7e640d7ca0 2021-04-09T18:23:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e627fc700 1 -- 172.21.15.32:0/4223071799 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e4c0042c0 con 0x7f7e640d7ca0 2021-04-09T18:23:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e63fff700 1 -- 172.21.15.32:0/4223071799 >> v1:172.21.15.86:6789/0 conn(0x7f7e640d7ca0 legacy=0x7f7e640d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e63fff700 1 -- 172.21.15.32:0/4223071799 >> v1:172.21.15.32:6789/0 conn(0x7f7e6410d010 legacy=0x7f7e640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e63fff700 1 -- 172.21.15.32:0/4223071799 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6410e420 con 0x7f7e6410da00 2021-04-09T18:23:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e63fff700 1 --2- 172.21.15.32:0/4223071799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6410da00 0x7f7e6410de00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7e5c006e50 tx=0x7f7e5c009e70).ready entity=mon.2 client_cookie=ccb49c2e98245368 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e627fc700 1 -- 172.21.15.32:0/4223071799 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e5c006a20 con 0x7f7e6410da00 2021-04-09T18:23:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e627fc700 1 -- 172.21.15.32:0/4223071799 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c006790 con 0x7f7e6410da00 2021-04-09T18:23:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e627fc700 1 -- 172.21.15.32:0/4223071799 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e5c00b700 con 0x7f7e6410da00 2021-04-09T18:23:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e69d98700 1 -- 172.21.15.32:0/4223071799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6410da00 msgr2=0x7f7e6410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:39.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.883 7f7e69d98700 1 --2- 172.21.15.32:0/4223071799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6410da00 0x7f7e6410de00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7e5c006e50 tx=0x7f7e5c009e70).stop 2021-04-09T18:23:39.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.884 7f7e69d98700 1 -- 172.21.15.32:0/4223071799 shutdown_connections 2021-04-09T18:23:39.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.884 7f7e69d98700 1 --2- 172.21.15.32:0/4223071799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6410da00 0x7f7e6410de00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:39.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.884 7f7e69d98700 1 -- 172.21.15.32:0/4223071799 >> 172.21.15.32:0/4223071799 conn(0x7f7e640ce5f0 msgr2=0x7f7e640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:39.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.884 7f7e69d98700 1 -- 172.21.15.32:0/4223071799 shutdown_connections 2021-04-09T18:23:39.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.884 7f7e69d98700 1 -- 172.21.15.32:0/4223071799 wait complete. 2021-04-09T18:23:39.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.904 7f7e69d98700 1 Processor -- start 2021-04-09T18:23:39.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.904 7f7e69d98700 1 -- start start 2021-04-09T18:23:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e69d98700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e64191ee0 0x7f7e640a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e69d98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640a6c30 0x7f7e640b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e69d98700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e640b4330 0x7f7e641b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e69d98700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640b3790 con 0x7f7e640a6c30 2021-04-09T18:23:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e69d98700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640a7030 con 0x7f7e64191ee0 2021-04-09T18:23:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e69d98700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640b4730 con 0x7f7e640b4330 2021-04-09T18:23:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e62ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640a6c30 0x7f7e640b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e62ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640a6c30 0x7f7e640b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50522/0 (socket says 172.21.15.32:50522) 2021-04-09T18:23:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e62ffd700 1 -- 172.21.15.32:0/2624641163 learned_addr learned my addr 172.21.15.32:0/2624641163 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e637fe700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e64191ee0 0x7f7e640a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e63fff700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e640b4330 0x7f7e641b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e62ffd700 1 -- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e640b4330 msgr2=0x7f7e641b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e62ffd700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e640b4330 0x7f7e641b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e62ffd700 1 -- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e64191ee0 msgr2=0x7f7e640a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e62ffd700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e64191ee0 0x7f7e640a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e62ffd700 1 -- 172.21.15.32:0/2624641163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e641b8580 con 0x7f7e640a6c30 2021-04-09T18:23:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e62ffd700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640a6c30 0x7f7e640b3d10 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7f7e58006f10 tx=0x7f7e5800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.905 7f7e4bfff700 1 -- 172.21.15.32:0/2624641163 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e5800bd80 con 0x7f7e640a6c30 2021-04-09T18:23:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.906 7f7e4bfff700 1 -- 172.21.15.32:0/2624641163 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e58000e20 con 0x7f7e640a6c30 2021-04-09T18:23:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.906 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e641b8580 con 0x7f7e640a6c30 2021-04-09T18:23:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.906 7f7e4bfff700 1 -- 172.21.15.32:0/2624641163 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e580054f0 con 0x7f7e640a6c30 2021-04-09T18:23:39.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.906 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e641b8580 con 0x7f7e640a6c30 2021-04-09T18:23:39.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.906 7f7e4bfff700 1 -- 172.21.15.32:0/2624641163 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7e58009fe0 con 0x7f7e640a6c30 2021-04-09T18:23:39.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.906 7f7e4bfff700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e44018930 0x7f7e4401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:39.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.906 7f7e4bfff700 1 -- 172.21.15.32:0/2624641163 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7e58032a50 con 0x7f7e640a6c30 2021-04-09T18:23:39.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.906 7f7e637fe700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e44018930 0x7f7e4401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:39.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.907 7f7e637fe700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e44018930 0x7f7e4401ae10 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:39.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.926 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6410d480 con 0x7f7e640a6c30 2021-04-09T18:23:39.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:39.930 7f7e4bfff700 1 -- 172.21.15.32:0/2624641163 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7e58000e20 con 0x7f7e640a6c30 2021-04-09T18:23:40.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.122 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f7e6410d480 con 0x7f7e640a6c30 2021-04-09T18:23:40.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.682 7f7e4bfff700 1 -- 172.21.15.32:0/2624641163 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v187) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7e58009fe0 con 0x7f7e640a6c30 2021-04-09T18:23:40.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e44018930 msgr2=0x7f7e4401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:40.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e44018930 0x7f7e4401ae10 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640a6c30 msgr2=0x7f7e640b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640a6c30 0x7f7e640b3d10 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7f7e58006f10 tx=0x7f7e5800b990).stop 2021-04-09T18:23:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 shutdown_connections 2021-04-09T18:23:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7e44018930 0x7f7e4401ae10 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e640a6c30 0x7f7e640b3d10 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e640b4330 0x7f7e641b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 --2- 172.21.15.32:0/2624641163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e64191ee0 0x7f7e640a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 >> 172.21.15.32:0/2624641163 conn(0x7f7e640bbda0 msgr2=0x7f7e641b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:40.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 shutdown_connections 2021-04-09T18:23:40.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.689 7f7e69d98700 1 -- 172.21.15.32:0/2624641163 wait complete. 2021-04-09T18:23:40.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:40.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.852 7f5dac4d4700 1 Processor -- start 2021-04-09T18:23:40.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.852 7f5dac4d4700 1 -- start start 2021-04-09T18:23:40.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5dac4d4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5da40d6ca0 0x7f5da40d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:40.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5dac4d4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5da40d2460 0x7f5da410cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:40.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5dac4d4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40d72a0 con 0x7f5da40cffc0 2021-04-09T18:23:40.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5dac4d4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da40d70a0 con 0x7f5da40d2460 2021-04-09T18:23:40.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5dac4d4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da40d9200 con 0x7f5da40d6ca0 2021-04-09T18:23:40.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5daaaf1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5da40d2460 0x7f5da410cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:40.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5daa2f0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5da40cffc0 0x7f5da40d7820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60120/0 (socket says 172.21.15.32:60120) 2021-04-09T18:23:40.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5daa2f0700 1 -- 172.21.15.32:0/243338168 learned_addr learned my addr 172.21.15.32:0/243338168 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:40.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5da9aef700 1 --2- 172.21.15.32:0/243338168 >> v2:172.21.15.86:3300/0 conn(0x7f5da40d6ca0 0x7f5da40d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:40.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5da92ee700 1 -- 172.21.15.32:0/243338168 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473708615 0 0) 0x7f5d9c000ce0 con 0x7f5da40cffc0 2021-04-09T18:23:40.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5da92ee700 1 -- 172.21.15.32:0/243338168 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d90002fe0 con 0x7f5da40cffc0 2021-04-09T18:23:40.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5daaaf1700 1 -- 172.21.15.32:0/243338168 >> v2:172.21.15.86:3300/0 conn(0x7f5da40d6ca0 msgr2=0x7f5da40d8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:40.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5daaaf1700 1 --2- 172.21.15.32:0/243338168 >> v2:172.21.15.86:3300/0 conn(0x7f5da40d6ca0 0x7f5da40d8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5daaaf1700 1 -- 172.21.15.32:0/243338168 >> v1:172.21.15.32:6789/0 conn(0x7f5da40cffc0 legacy=0x7f5da40d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5daaaf1700 1 -- 172.21.15.32:0/243338168 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40d2860 con 0x7f5da40d2460 2021-04-09T18:23:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5daaaf1700 1 --2- 172.21.15.32:0/243338168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da40d2460 0x7f5da410cff0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f5da0006ab0 tx=0x7f5da0009b40).ready entity=mon.0 client_cookie=d8777908b57c0622 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5da92ee700 1 -- 172.21.15.32:0/243338168 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5da00022e0 con 0x7f5da40d2460 2021-04-09T18:23:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5da92ee700 1 -- 172.21.15.32:0/243338168 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da0002660 con 0x7f5da40d2460 2021-04-09T18:23:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.853 7f5da92ee700 1 -- 172.21.15.32:0/243338168 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5da0002b40 con 0x7f5da40d2460 2021-04-09T18:23:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.854 7f5dac4d4700 1 -- 172.21.15.32:0/243338168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da40d2460 msgr2=0x7f5da410cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.854 7f5dac4d4700 1 --2- 172.21.15.32:0/243338168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da40d2460 0x7f5da410cff0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f5da0006ab0 tx=0x7f5da0009b40).stop 2021-04-09T18:23:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.854 7f5dac4d4700 1 -- 172.21.15.32:0/243338168 shutdown_connections 2021-04-09T18:23:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.854 7f5dac4d4700 1 --2- 172.21.15.32:0/243338168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da40d2460 0x7f5da410cff0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.854 7f5dac4d4700 1 --2- 172.21.15.32:0/243338168 >> v2:172.21.15.86:3300/0 conn(0x7f5da40d6ca0 0x7f5da40d8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.854 7f5dac4d4700 1 -- 172.21.15.32:0/243338168 >> 172.21.15.32:0/243338168 conn(0x7f5da40ce5e0 msgr2=0x7f5da40cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.854 7f5dac4d4700 1 -- 172.21.15.32:0/243338168 shutdown_connections 2021-04-09T18:23:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.854 7f5dac4d4700 1 -- 172.21.15.32:0/243338168 wait complete. 2021-04-09T18:23:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.874 7f5dac4d4700 1 Processor -- start 2021-04-09T18:23:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5dac4d4700 1 -- start start 2021-04-09T18:23:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5dac4d4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da409ebc0 0x7f5da409adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5dac4d4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da4183050 0x7f5da419ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5dac4d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da4183450 0x7f5da40952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5dac4d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da409b660 con 0x7f5da4183450 2021-04-09T18:23:40.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5dac4d4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41a05c0 con 0x7f5da409ebc0 2021-04-09T18:23:40.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5dac4d4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4095900 con 0x7f5da4183050 2021-04-09T18:23:40.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5daaaf1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da4183450 0x7f5da40952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:40.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5daaaf1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da4183450 0x7f5da40952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50540/0 (socket says 172.21.15.32:50540) 2021-04-09T18:23:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5daaaf1700 1 -- 172.21.15.32:0/2315811590 learned_addr learned my addr 172.21.15.32:0/2315811590 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5daa2f0700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da409ebc0 0x7f5da409adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5da9aef700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da4183050 0x7f5da419ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5daaaf1700 1 -- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da4183050 msgr2=0x7f5da419ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:40.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5daaaf1700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da4183050 0x7f5da419ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5daaaf1700 1 -- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da409ebc0 msgr2=0x7f5da409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5daaaf1700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da409ebc0 0x7f5da409adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.875 7f5daaaf1700 1 -- 172.21.15.32:0/2315811590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da4095b20 con 0x7f5da4183450 2021-04-09T18:23:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.876 7f5daa2f0700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da409ebc0 0x7f5da409adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:40.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.876 7f5da9aef700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da4183050 0x7f5da419ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.876 7f5daaaf1700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da4183450 0x7f5da40952e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f5da000b5f0 tx=0x7f5da000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.876 7f5d96ffd700 1 -- 172.21.15.32:0/2315811590 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5da000bd80 con 0x7f5da4183450 2021-04-09T18:23:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.876 7f5d96ffd700 1 -- 172.21.15.32:0/2315811590 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da0006020 con 0x7f5da4183450 2021-04-09T18:23:40.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.876 7f5d96ffd700 1 -- 172.21.15.32:0/2315811590 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5da00054a0 con 0x7f5da4183450 2021-04-09T18:23:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.876 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da4095b20 con 0x7f5da4183450 2021-04-09T18:23:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.876 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da4095b20 con 0x7f5da4183450 2021-04-09T18:23:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.876 7f5d96ffd700 1 -- 172.21.15.32:0/2315811590 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5da00054a0 con 0x7f5da4183450 2021-04-09T18:23:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.877 7f5d96ffd700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d8c018930 0x7f5d8c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.877 7f5d96ffd700 1 -- 172.21.15.32:0/2315811590 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5da0032a10 con 0x7f5da4183450 2021-04-09T18:23:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.877 7f5daa2f0700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d8c018930 0x7f5d8c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:40.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.877 7f5daa2f0700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d8c018930 0x7f5d8c01adc0 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:40.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.897 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da4093df0 con 0x7f5da4183450 2021-04-09T18:23:40.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:40.900 7f5d96ffd700 1 -- 172.21.15.32:0/2315811590 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5da001f750 con 0x7f5da4183450 2021-04-09T18:23:41.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.095 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f5da41a0580 con 0x7f5da4183450 2021-04-09T18:23:41.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.730 7f5d96ffd700 1 -- 172.21.15.32:0/2315811590 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v188) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5da00054a0 con 0x7f5da4183450 2021-04-09T18:23:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.762 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d8c018930 msgr2=0x7f5d8c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.762 7f5dac4d4700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d8c018930 0x7f5d8c01adc0 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:41.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.762 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da4183450 msgr2=0x7f5da40952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:41.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.762 7f5dac4d4700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da4183450 0x7f5da40952e0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f5da000b5f0 tx=0x7f5da000b990).stop 2021-04-09T18:23:41.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.762 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 shutdown_connections 2021-04-09T18:23:41.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.762 7f5dac4d4700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d8c018930 0x7f5d8c01adc0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.763 7f5dac4d4700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da4183450 0x7f5da40952e0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.763 7f5dac4d4700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da4183050 0x7f5da419ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.763 7f5dac4d4700 1 --2- 172.21.15.32:0/2315811590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da409ebc0 0x7f5da409adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:41.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.763 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 >> 172.21.15.32:0/2315811590 conn(0x7f5da409fa00 msgr2=0x7f5da40b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:41.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.763 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 shutdown_connections 2021-04-09T18:23:41.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.763 7f5dac4d4700 1 -- 172.21.15.32:0/2315811590 wait complete. 2021-04-09T18:23:41.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.927 7f0a8ac5c700 1 Processor -- start 2021-04-09T18:23:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.927 7f0a8ac5c700 1 -- start start 2021-04-09T18:23:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a8ac5c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840d7ca0 0x7f0a840d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a8ac5c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a8410da00 0x7f0a8410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a8ac5c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a8410d480 con 0x7f0a8410d010 2021-04-09T18:23:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a8ac5c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a840d9220 con 0x7f0a8410da00 2021-04-09T18:23:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a8ac5c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a840d86c0 con 0x7f0a840d7ca0 2021-04-09T18:23:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a89279700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a8410da00 0x7f0a8410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a88a78700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0a8410d010 0x7f0a840d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60134/0 (socket says 172.21.15.32:60134) 2021-04-09T18:23:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a88a78700 1 -- 172.21.15.32:0/454065292 learned_addr learned my addr 172.21.15.32:0/454065292 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a89279700 1 --2- 172.21.15.32:0/454065292 >> v2:172.21.15.32:3300/0 conn(0x7f0a8410da00 0x7f0a8410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50548/0 (socket says 172.21.15.32:50548) 2021-04-09T18:23:42.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a83fff700 1 --2- 172.21.15.32:0/454065292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840d7ca0 0x7f0a840d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:42.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a89279700 1 -- 172.21.15.32:0/454065292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840d7ca0 msgr2=0x7f0a840d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a89279700 1 --2- 172.21.15.32:0/454065292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840d7ca0 0x7f0a840d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a89279700 1 -- 172.21.15.32:0/454065292 >> v1:172.21.15.32:6789/0 conn(0x7f0a8410d010 legacy=0x7f0a840d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a837fe700 1 -- 172.21.15.32:0/454065292 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916180227 0 0) 0x7f0a74001060 con 0x7f0a8410d010 2021-04-09T18:23:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a89279700 1 -- 172.21.15.32:0/454065292 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a840d1f70 con 0x7f0a8410da00 2021-04-09T18:23:42.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.928 7f0a89279700 1 --2- 172.21.15.32:0/454065292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8410da00 0x7f0a8410de00 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f0a7c006ab0 tx=0x7f0a7c009ac0).ready entity=mon.0 client_cookie=3002f6dae52e20da server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a837fe700 1 -- 172.21.15.32:0/454065292 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a7c0066d0 con 0x7f0a8410da00 2021-04-09T18:23:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a837fe700 1 -- 172.21.15.32:0/454065292 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7c0022e0 con 0x7f0a8410da00 2021-04-09T18:23:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a837fe700 1 -- 172.21.15.32:0/454065292 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a7c0027c0 con 0x7f0a8410da00 2021-04-09T18:23:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a8ac5c700 1 -- 172.21.15.32:0/454065292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8410da00 msgr2=0x7f0a8410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a8ac5c700 1 --2- 172.21.15.32:0/454065292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8410da00 0x7f0a8410de00 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f0a7c006ab0 tx=0x7f0a7c009ac0).stop 2021-04-09T18:23:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a8ac5c700 1 -- 172.21.15.32:0/454065292 shutdown_connections 2021-04-09T18:23:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a8ac5c700 1 --2- 172.21.15.32:0/454065292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8410da00 0x7f0a8410de00 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a8ac5c700 1 --2- 172.21.15.32:0/454065292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840d7ca0 0x7f0a840d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a8ac5c700 1 -- 172.21.15.32:0/454065292 >> 172.21.15.32:0/454065292 conn(0x7f0a840ce5f0 msgr2=0x7f0a840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.929 7f0a8ac5c700 1 -- 172.21.15.32:0/454065292 shutdown_connections 2021-04-09T18:23:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.930 7f0a8ac5c700 1 -- 172.21.15.32:0/454065292 wait complete. 2021-04-09T18:23:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.950 7f0a8ac5c700 1 Processor -- start 2021-04-09T18:23:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.950 7f0a8ac5c700 1 -- start start 2021-04-09T18:23:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.950 7f0a8ac5c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840b1680 0x7f0a840b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.950 7f0a8ac5c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8409e330 0x7f0a840b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.950 7f0a8ac5c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a840b2190 0x7f0a8409f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.950 7f0a8ac5c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8409e730 con 0x7f0a8409e330 2021-04-09T18:23:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.950 7f0a8ac5c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8409f6c0 con 0x7f0a840b2190 2021-04-09T18:23:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.950 7f0a8ac5c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8409f8e0 con 0x7f0a840b1680 2021-04-09T18:23:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a83fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8409e330 0x7f0a840b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a83fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8409e330 0x7f0a840b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50552/0 (socket says 172.21.15.32:50552) 2021-04-09T18:23:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a83fff700 1 -- 172.21.15.32:0/3180885500 learned_addr learned my addr 172.21.15.32:0/3180885500 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a88a78700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840b1680 0x7f0a840b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a89279700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a840b2190 0x7f0a8409f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a83fff700 1 -- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840b1680 msgr2=0x7f0a840b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a83fff700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840b1680 0x7f0a840b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a83fff700 1 -- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a840b2190 msgr2=0x7f0a8409f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a83fff700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a840b2190 0x7f0a8409f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a83fff700 1 -- 172.21.15.32:0/3180885500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8409fb00 con 0x7f0a8409e330 2021-04-09T18:23:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a83fff700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8409e330 0x7f0a840b1b70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f0a6c006be0 tx=0x7f0a6c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a817fa700 1 -- 172.21.15.32:0/3180885500 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a6c0064d0 con 0x7f0a8409e330 2021-04-09T18:23:42.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a817fa700 1 -- 172.21.15.32:0/3180885500 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c00bd80 con 0x7f0a8409e330 2021-04-09T18:23:42.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a817fa700 1 -- 172.21.15.32:0/3180885500 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a6c002030 con 0x7f0a8409e330 2021-04-09T18:23:42.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8409fb00 con 0x7f0a8409e330 2021-04-09T18:23:42.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.951 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8409fb00 con 0x7f0a8409e330 2021-04-09T18:23:42.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.952 7f0a817fa700 1 -- 172.21.15.32:0/3180885500 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0a6c006c90 con 0x7f0a8409e330 2021-04-09T18:23:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.952 7f0a817fa700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a68018930 0x7f0a6801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.952 7f0a817fa700 1 -- 172.21.15.32:0/3180885500 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0a6c032a10 con 0x7f0a8409e330 2021-04-09T18:23:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.952 7f0a88a78700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a68018930 0x7f0a6801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:42.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.953 7f0a88a78700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a68018930 0x7f0a6801ae10 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:42.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.972 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8410d480 con 0x7f0a8409e330 2021-04-09T18:23:42.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:41.975 7f0a817fa700 1 -- 172.21.15.32:0/3180885500 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0a6c006820 con 0x7f0a8409e330 2021-04-09T18:23:42.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.167 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f0a8410d480 con 0x7f0a8409e330 2021-04-09T18:23:42.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.737 7f0a817fa700 1 -- 172.21.15.32:0/3180885500 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v189) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0a6c006c90 con 0x7f0a8409e330 2021-04-09T18:23:42.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a68018930 msgr2=0x7f0a6801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:42.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a68018930 0x7f0a6801ae10 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8409e330 msgr2=0x7f0a840b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:42.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8409e330 0x7f0a840b1b70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f0a6c006be0 tx=0x7f0a6c00b990).stop 2021-04-09T18:23:42.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 shutdown_connections 2021-04-09T18:23:42.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a68018930 0x7f0a6801ae10 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a8409e330 0x7f0a840b1b70 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a840b2190 0x7f0a8409f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 --2- 172.21.15.32:0/3180885500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a840b1680 0x7f0a840b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:42.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 >> 172.21.15.32:0/3180885500 conn(0x7f0a840c4290 msgr2=0x7f0a840c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:42.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 shutdown_connections 2021-04-09T18:23:42.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.784 7f0a8ac5c700 1 -- 172.21.15.32:0/3180885500 wait complete. 2021-04-09T18:23:42.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:43.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.949 7fc3db393700 1 Processor -- start 2021-04-09T18:23:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.949 7fc3db393700 1 -- start start 2021-04-09T18:23:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3db393700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3d40d7ca0 0x7fc3d40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3db393700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d410d480 con 0x7fc3d410d010 2021-04-09T18:23:43.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3db393700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d40d86c0 con 0x7fc3d410da00 2021-04-09T18:23:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3db393700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d40d9220 con 0x7fc3d40d7ca0 2021-04-09T18:23:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3d91af700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc3d410d010 0x7fc3d40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60148/0 (socket says 172.21.15.32:60148) 2021-04-09T18:23:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3d91af700 1 -- 172.21.15.32:0/3826884605 learned_addr learned my addr 172.21.15.32:0/3826884605 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:43.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3d89ae700 1 --2- 172.21.15.32:0/3826884605 >> v2:172.21.15.86:3300/0 conn(0x7fc3d40d7ca0 0x7fc3d40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3cbfff700 1 -- 172.21.15.32:0/3826884605 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650422644 0 0) 0x7fc3d0001100 con 0x7fc3d410d010 2021-04-09T18:23:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3cbfff700 1 -- 172.21.15.32:0/3826884605 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c0002fe0 con 0x7fc3d410d010 2021-04-09T18:23:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3cbfff700 1 -- 172.21.15.32:0/3826884605 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496909725 0 0) 0x7fc3cc000d30 con 0x7fc3d410da00 2021-04-09T18:23:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3cbfff700 1 -- 172.21.15.32:0/3826884605 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c00042c0 con 0x7fc3d410da00 2021-04-09T18:23:43.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3d89ae700 1 -- 172.21.15.32:0/3826884605 >> v1:172.21.15.86:6789/0 conn(0x7fc3d410da00 legacy=0x7fc3d410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3d89ae700 1 -- 172.21.15.32:0/3826884605 >> v1:172.21.15.32:6789/0 conn(0x7fc3d410d010 legacy=0x7fc3d40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.950 7fc3d89ae700 1 -- 172.21.15.32:0/3826884605 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d40d1f70 con 0x7fc3d40d7ca0 2021-04-09T18:23:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.951 7fc3d89ae700 1 --2- 172.21.15.32:0/3826884605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d40d7ca0 0x7fc3d40d80a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fc3c4006650 tx=0x7fc3c4009a80).ready entity=mon.1 client_cookie=55296bd3d028053c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:43.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.951 7fc3cbfff700 1 -- 172.21.15.32:0/3826884605 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c40009b0 con 0x7fc3d40d7ca0 2021-04-09T18:23:43.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.951 7fc3cbfff700 1 -- 172.21.15.32:0/3826884605 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c400c0c0 con 0x7fc3d40d7ca0 2021-04-09T18:23:43.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.951 7fc3cbfff700 1 -- 172.21.15.32:0/3826884605 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c400c510 con 0x7fc3d40d7ca0 2021-04-09T18:23:43.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.951 7fc3db393700 1 -- 172.21.15.32:0/3826884605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d40d7ca0 msgr2=0x7fc3d40d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:43.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.951 7fc3db393700 1 --2- 172.21.15.32:0/3826884605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d40d7ca0 0x7fc3d40d80a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fc3c4006650 tx=0x7fc3c4009a80).stop 2021-04-09T18:23:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.952 7fc3db393700 1 -- 172.21.15.32:0/3826884605 shutdown_connections 2021-04-09T18:23:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.952 7fc3db393700 1 --2- 172.21.15.32:0/3826884605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d40d7ca0 0x7fc3d40d80a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:43.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.952 7fc3db393700 1 -- 172.21.15.32:0/3826884605 >> 172.21.15.32:0/3826884605 conn(0x7fc3d40ce5f0 msgr2=0x7fc3d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.952 7fc3db393700 1 -- 172.21.15.32:0/3826884605 shutdown_connections 2021-04-09T18:23:43.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.952 7fc3db393700 1 -- 172.21.15.32:0/3826884605 wait complete. 2021-04-09T18:23:43.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.972 7fc3db393700 1 Processor -- start 2021-04-09T18:23:43.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.972 7fc3db393700 1 -- start start 2021-04-09T18:23:43.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3db393700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d41b8550 0x7fc3d40bff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:43.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3db393700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d40c0540 0x7fc3d4194ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:43.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3db393700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d40c0940 0x7fc3d41950f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:43.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3db393700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41b8d80 con 0x7fc3d41b8550 2021-04-09T18:23:43.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3db393700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41b8950 con 0x7fc3d40c0940 2021-04-09T18:23:43.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3db393700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d4195710 con 0x7fc3d40c0540 2021-04-09T18:23:43.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d91af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d41b8550 0x7fc3d40bff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:43.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d91af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d41b8550 0x7fc3d40bff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50564/0 (socket says 172.21.15.32:50564) 2021-04-09T18:23:43.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d91af700 1 -- 172.21.15.32:0/4170018490 learned_addr learned my addr 172.21.15.32:0/4170018490 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:43.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d89ae700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d40c0540 0x7fc3d4194ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:43.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d99b0700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d40c0940 0x7fc3d41950f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:43.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d91af700 1 -- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d40c0540 msgr2=0x7fc3d4194ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:43.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d91af700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d40c0540 0x7fc3d4194ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:43.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d91af700 1 -- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d40c0940 msgr2=0x7fc3d41950f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:43.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d91af700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d40c0940 0x7fc3d41950f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:43.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d91af700 1 -- 172.21.15.32:0/4170018490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d40bb0f0 con 0x7fc3d41b8550 2021-04-09T18:23:43.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3d91af700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d41b8550 0x7fc3d40bff20 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7fc3d000b5c0 tx=0x7fc3d000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3c9ffb700 1 -- 172.21.15.32:0/4170018490 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3d0006020 con 0x7fc3d41b8550 2021-04-09T18:23:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3c9ffb700 1 -- 172.21.15.32:0/4170018490 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d0006320 con 0x7fc3d41b8550 2021-04-09T18:23:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3c9ffb700 1 -- 172.21.15.32:0/4170018490 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3d0004620 con 0x7fc3d41b8550 2021-04-09T18:23:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3db393700 1 -- 172.21.15.32:0/4170018490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d40bb0f0 con 0x7fc3d41b8550 2021-04-09T18:23:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.973 7fc3db393700 1 -- 172.21.15.32:0/4170018490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d40bb0f0 con 0x7fc3d41b8550 2021-04-09T18:23:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.974 7fc3c9ffb700 1 -- 172.21.15.32:0/4170018490 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc3d000a000 con 0x7fc3d41b8550 2021-04-09T18:23:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.974 7fc3c9ffb700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3b40188d0 0x7fc3b401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.974 7fc3c9ffb700 1 -- 172.21.15.32:0/4170018490 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc3d0032b40 con 0x7fc3d41b8550 2021-04-09T18:23:43.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.974 7fc3d89ae700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3b40188d0 0x7fc3b401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:43.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.975 7fc3d89ae700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3b40188d0 0x7fc3b401add0 crc :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:43.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.994 7fc3db393700 1 -- 172.21.15.32:0/4170018490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d410d480 con 0x7fc3d41b8550 2021-04-09T18:23:43.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:42.997 7fc3c9ffb700 1 -- 172.21.15.32:0/4170018490 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc3d0006b80 con 0x7fc3d41b8550 2021-04-09T18:23:43.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.192 7fc3db393700 1 -- 172.21.15.32:0/4170018490 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fc3d410d480 con 0x7fc3d41b8550 2021-04-09T18:23:43.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.745 7fc3c9ffb700 1 -- 172.21.15.32:0/4170018490 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v190) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc3d0006320 con 0x7fc3d41b8550 2021-04-09T18:23:43.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 -- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3b40188d0 msgr2=0x7fc3b401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:43.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3b40188d0 0x7fc3b401add0 crc :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:43.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 -- 172.21.15.32:0/4170018490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d41b8550 msgr2=0x7fc3d40bff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:43.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d41b8550 0x7fc3d40bff20 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7fc3d000b5c0 tx=0x7fc3d000b960).stop 2021-04-09T18:23:43.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 -- 172.21.15.32:0/4170018490 shutdown_connections 2021-04-09T18:23:43.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc3b40188d0 0x7fc3b401add0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:43.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d40c0940 0x7fc3d41950f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:43.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d40c0540 0x7fc3d4194ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:43.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 --2- 172.21.15.32:0/4170018490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3d41b8550 0x7fc3d40bff20 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:43.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 -- 172.21.15.32:0/4170018490 >> 172.21.15.32:0/4170018490 conn(0x7fc3d40bce00 msgr2=0x7fc3d40b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:43.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 -- 172.21.15.32:0/4170018490 shutdown_connections 2021-04-09T18:23:43.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.760 7fc3db393700 1 -- 172.21.15.32:0/4170018490 wait complete. 2021-04-09T18:23:43.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.929 7f576ad3b700 1 Processor -- start 2021-04-09T18:23:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.929 7f576ad3b700 1 -- start start 2021-04-09T18:23:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.929 7f576ad3b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f576410da00 0x7f576410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:44.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.929 7f576ad3b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576410d480 con 0x7f576410d010 2021-04-09T18:23:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.929 7f576ad3b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57640d1f70 con 0x7f57640d7ca0 2021-04-09T18:23:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.929 7f576ad3b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57640d8d30 con 0x7f576410da00 2021-04-09T18:23:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.929 7f5769358700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f576410da00 0x7f576410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:44.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.929 7f5768b57700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f576410d010 0x7f57640d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60162/0 (socket says 172.21.15.32:60162) 2021-04-09T18:23:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.929 7f5768b57700 1 -- 172.21.15.32:0/4242887473 learned_addr learned my addr 172.21.15.32:0/4242887473 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f5769358700 1 -- 172.21.15.32:0/4242887473 >> v1:172.21.15.86:6789/0 conn(0x7f57640d7ca0 legacy=0x7f57640d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f5769358700 1 -- 172.21.15.32:0/4242887473 >> v1:172.21.15.32:6789/0 conn(0x7f576410d010 legacy=0x7f57640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:44.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f57637fe700 1 -- 172.21.15.32:0/4242887473 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735410564 0 0) 0x7f574c000ce0 con 0x7f576410d010 2021-04-09T18:23:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f5769358700 1 -- 172.21.15.32:0/4242887473 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57640d9220 con 0x7f576410da00 2021-04-09T18:23:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f5769358700 1 --2- 172.21.15.32:0/4242887473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f576410da00 0x7f576410de00 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0x7f575c006e30 tx=0x7f575c009de0).ready entity=mon.0 client_cookie=818a17531bdc8769 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f57637fe700 1 -- 172.21.15.32:0/4242887473 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f575c00b300 con 0x7f576410da00 2021-04-09T18:23:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f57637fe700 1 -- 172.21.15.32:0/4242887473 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c00b680 con 0x7f576410da00 2021-04-09T18:23:44.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f57637fe700 1 -- 172.21.15.32:0/4242887473 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f575c00bb60 con 0x7f576410da00 2021-04-09T18:23:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f576ad3b700 1 -- 172.21.15.32:0/4242887473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f576410da00 msgr2=0x7f576410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.930 7f576ad3b700 1 --2- 172.21.15.32:0/4242887473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f576410da00 0x7f576410de00 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0x7f575c006e30 tx=0x7f575c009de0).stop 2021-04-09T18:23:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.931 7f576ad3b700 1 -- 172.21.15.32:0/4242887473 shutdown_connections 2021-04-09T18:23:44.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.931 7f576ad3b700 1 --2- 172.21.15.32:0/4242887473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f576410da00 0x7f576410de00 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.931 7f576ad3b700 1 -- 172.21.15.32:0/4242887473 >> 172.21.15.32:0/4242887473 conn(0x7f57640ce5f0 msgr2=0x7f57640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.931 7f576ad3b700 1 -- 172.21.15.32:0/4242887473 shutdown_connections 2021-04-09T18:23:44.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.931 7f576ad3b700 1 -- 172.21.15.32:0/4242887473 wait complete. 2021-04-09T18:23:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.951 7f576ad3b700 1 Processor -- start 2021-04-09T18:23:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.951 7f576ad3b700 1 -- start start 2021-04-09T18:23:44.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f576ad3b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57641852e0 0x7f576417b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f576ad3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57640bac00 0x7f576417b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f576ad3b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5764179610 0x7f5764179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f576ad3b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576417ac50 con 0x7f57640bac00 2021-04-09T18:23:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f576ad3b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5764184dd0 con 0x7f57641852e0 2021-04-09T18:23:44.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f576ad3b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576417a030 con 0x7f5764179610 2021-04-09T18:23:44.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5763fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57640bac00 0x7f576417b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:44.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5763fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57640bac00 0x7f576417b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50576/0 (socket says 172.21.15.32:50576) 2021-04-09T18:23:44.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5763fff700 1 -- 172.21.15.32:0/2960889952 learned_addr learned my addr 172.21.15.32:0/2960889952 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:44.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5768b57700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57641852e0 0x7f576417b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:44.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5769358700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5764179610 0x7f5764179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:44.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5763fff700 1 -- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5764179610 msgr2=0x7f5764179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:44.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5763fff700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5764179610 0x7f5764179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:44.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5763fff700 1 -- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57641852e0 msgr2=0x7f576417b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:44.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5763fff700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57641852e0 0x7f576417b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:44.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.952 7f5763fff700 1 -- 172.21.15.32:0/2960889952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576417a250 con 0x7f57640bac00 2021-04-09T18:23:44.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.953 7f5763fff700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57640bac00 0x7f576417b7f0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f575400b5b0 tx=0x7f575400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:44.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.953 7f57617fa700 1 -- 172.21.15.32:0/2960889952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f575400bd40 con 0x7f57640bac00 2021-04-09T18:23:44.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.953 7f57617fa700 1 -- 172.21.15.32:0/2960889952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5754006240 con 0x7f57640bac00 2021-04-09T18:23:44.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.953 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576417a250 con 0x7f57640bac00 2021-04-09T18:23:44.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.953 7f57617fa700 1 -- 172.21.15.32:0/2960889952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5754005690 con 0x7f57640bac00 2021-04-09T18:23:44.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.953 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576417a250 con 0x7f57640bac00 2021-04-09T18:23:44.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.953 7f57617fa700 1 -- 172.21.15.32:0/2960889952 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f57540054a0 con 0x7f57640bac00 2021-04-09T18:23:44.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.953 7f57617fa700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5748018930 0x7f574801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:44.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.953 7f57617fa700 1 -- 172.21.15.32:0/2960889952 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5754032a30 con 0x7f57640bac00 2021-04-09T18:23:44.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.954 7f5768b57700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5748018930 0x7f574801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:44.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.954 7f5768b57700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5748018930 0x7f574801ae10 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:44.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.973 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5764184db0 con 0x7f57640bac00 2021-04-09T18:23:44.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:43.977 7f57617fa700 1 -- 172.21.15.32:0/2960889952 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5754000c60 con 0x7f57640bac00 2021-04-09T18:23:44.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.168 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f5764184db0 con 0x7f57640bac00 2021-04-09T18:23:44.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.752 7f57617fa700 1 -- 172.21.15.32:0/2960889952 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v191) v1 ==== 105+0+0 (secure 0 0 0) 0x7f57540054a0 con 0x7f57640bac00 2021-04-09T18:23:44.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.785 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5748018930 msgr2=0x7f574801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:44.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.785 7f576ad3b700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5748018930 0x7f574801ae10 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:44.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.785 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57640bac00 msgr2=0x7f576417b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:44.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.785 7f576ad3b700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57640bac00 0x7f576417b7f0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f575400b5b0 tx=0x7f575400b950).stop 2021-04-09T18:23:44.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.786 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 shutdown_connections 2021-04-09T18:23:44.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.786 7f576ad3b700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5748018930 0x7f574801ae10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:44.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.786 7f576ad3b700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57640bac00 0x7f576417b7f0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:44.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.786 7f576ad3b700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5764179610 0x7f5764179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:44.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.786 7f576ad3b700 1 --2- 172.21.15.32:0/2960889952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57641852e0 0x7f576417b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:44.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.786 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 >> 172.21.15.32:0/2960889952 conn(0x7f57640b7520 msgr2=0x7f5764185740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:44.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.786 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 shutdown_connections 2021-04-09T18:23:44.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.786 7f576ad3b700 1 -- 172.21.15.32:0/2960889952 wait complete. 2021-04-09T18:23:44.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:45.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.953 7febbb22a700 1 Processor -- start 2021-04-09T18:23:45.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.953 7febbb22a700 1 -- start start 2021-04-09T18:23:45.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febbb22a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7febb40cffc0 0x7febb40d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:45.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febbb22a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7febb40d7820 0x7febb40d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:45.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febbb22a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb40d8240 0x7febb40d5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:45.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febbb22a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb40d9200 con 0x7febb40d7820 2021-04-09T18:23:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febbb22a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb40d72a0 con 0x7febb40cffc0 2021-04-09T18:23:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febbb22a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb40d62b0 con 0x7febb40d8240 2021-04-09T18:23:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb8845700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7febb40d7820 0x7febb40d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb8845700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7febb40d7820 0x7febb40d7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50584/0 (socket says 172.21.15.32:50584) 2021-04-09T18:23:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb8845700 1 -- 172.21.15.32:0/2292357756 learned_addr learned my addr 172.21.15.32:0/2292357756 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb9847700 1 --2- 172.21.15.32:0/2292357756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb40d8240 0x7febb40d5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb9046700 1 --2- 172.21.15.32:0/2292357756 >> v2:172.21.15.86:3300/0 conn(0x7febb40cffc0 0x7febb40d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb8845700 1 -- 172.21.15.32:0/2292357756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb40d8240 msgr2=0x7febb40d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb8845700 1 --2- 172.21.15.32:0/2292357756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb40d8240 0x7febb40d5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb8845700 1 -- 172.21.15.32:0/2292357756 >> v2:172.21.15.86:3300/0 conn(0x7febb40cffc0 msgr2=0x7febb40d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb8845700 1 --2- 172.21.15.32:0/2292357756 >> v2:172.21.15.86:3300/0 conn(0x7febb40cffc0 0x7febb40d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb8845700 1 -- 172.21.15.32:0/2292357756 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb40d64a0 con 0x7febb40d7820 2021-04-09T18:23:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb9847700 1 --2- 172.21.15.32:0/2292357756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb40d8240 0x7febb40d5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:45.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb9046700 1 --2- 172.21.15.32:0/2292357756 >> v2:172.21.15.86:3300/0 conn(0x7febb40cffc0 0x7febb40d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:45.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.954 7febb8845700 1 --2- 172.21.15.32:0/2292357756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb40d7820 0x7febb40d7c20 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7feba4006700 tx=0x7feba4009b80).ready entity=mon.0 client_cookie=fa01729302102a12 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:45.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febabfff700 1 -- 172.21.15.32:0/2292357756 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba400c0c0 con 0x7febb40d7820 2021-04-09T18:23:45.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febabfff700 1 -- 172.21.15.32:0/2292357756 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba400c440 con 0x7febb40d7820 2021-04-09T18:23:45.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febabfff700 1 -- 172.21.15.32:0/2292357756 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba400c920 con 0x7febb40d7820 2021-04-09T18:23:45.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febbb22a700 1 -- 172.21.15.32:0/2292357756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb40d7820 msgr2=0x7febb40d7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:45.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febbb22a700 1 --2- 172.21.15.32:0/2292357756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb40d7820 0x7febb40d7c20 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7feba4006700 tx=0x7feba4009b80).stop 2021-04-09T18:23:45.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febbb22a700 1 -- 172.21.15.32:0/2292357756 shutdown_connections 2021-04-09T18:23:45.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febbb22a700 1 --2- 172.21.15.32:0/2292357756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb40d8240 0x7febb40d5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febbb22a700 1 --2- 172.21.15.32:0/2292357756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb40d7820 0x7febb40d7c20 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febbb22a700 1 --2- 172.21.15.32:0/2292357756 >> v2:172.21.15.86:3300/0 conn(0x7febb40cffc0 0x7febb40d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.955 7febbb22a700 1 -- 172.21.15.32:0/2292357756 >> 172.21.15.32:0/2292357756 conn(0x7febb40ce5e0 msgr2=0x7febb40cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:45.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.956 7febbb22a700 1 -- 172.21.15.32:0/2292357756 shutdown_connections 2021-04-09T18:23:45.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.956 7febbb22a700 1 -- 172.21.15.32:0/2292357756 wait complete. 2021-04-09T18:23:45.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.976 7febbb22a700 1 Processor -- start 2021-04-09T18:23:45.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.976 7febbb22a700 1 -- start start 2021-04-09T18:23:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.976 7febbb22a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb4095e10 0x7febb4096210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:45.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.976 7febbb22a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb4096fe0 0x7febb40973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.976 7febbb22a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb417ffc0 0x7febb41803c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.976 7febbb22a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb4097a00 con 0x7febb4096fe0 2021-04-09T18:23:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.976 7febbb22a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb41855f0 con 0x7febb417ffc0 2021-04-09T18:23:45.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.976 7febbb22a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb41809e0 con 0x7febb4095e10 2021-04-09T18:23:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb8845700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb4096fe0 0x7febb40973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb8845700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb4096fe0 0x7febb40973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50588/0 (socket says 172.21.15.32:50588) 2021-04-09T18:23:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb8845700 1 -- 172.21.15.32:0/2786258507 learned_addr learned my addr 172.21.15.32:0/2786258507 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:45.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb9046700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb4095e10 0x7febb4096210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb9847700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb417ffc0 0x7febb41803c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb8845700 1 -- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb4095e10 msgr2=0x7febb4096210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb8845700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb4095e10 0x7febb4096210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb8845700 1 -- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb417ffc0 msgr2=0x7febb41803c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb8845700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb417ffc0 0x7febb41803c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb8845700 1 -- 172.21.15.32:0/2786258507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb4180c00 con 0x7febb4096fe0 2021-04-09T18:23:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb9046700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb4095e10 0x7febb4096210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:45.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb9847700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb417ffc0 0x7febb41803c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:23:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febb8845700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb4096fe0 0x7febb40973e0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7feba400b650 tx=0x7feba400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7feba9ffb700 1 -- 172.21.15.32:0/2786258507 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba400bde0 con 0x7febb4096fe0 2021-04-09T18:23:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7feba9ffb700 1 -- 172.21.15.32:0/2786258507 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba40091e0 con 0x7febb4096fe0 2021-04-09T18:23:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7feba9ffb700 1 -- 172.21.15.32:0/2786258507 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba4002220 con 0x7febb4096fe0 2021-04-09T18:23:45.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febbb22a700 1 -- 172.21.15.32:0/2786258507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb4180c00 con 0x7febb4096fe0 2021-04-09T18:23:45.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.977 7febbb22a700 1 -- 172.21.15.32:0/2786258507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb4180c00 con 0x7febb4096fe0 2021-04-09T18:23:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.978 7feba9ffb700 1 -- 172.21.15.32:0/2786258507 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7feba4009f90 con 0x7febb4096fe0 2021-04-09T18:23:45.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.978 7feba9ffb700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb94018930 0x7feb9401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:45.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.978 7feba9ffb700 1 -- 172.21.15.32:0/2786258507 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7feba40329f0 con 0x7febb4096fe0 2021-04-09T18:23:45.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.978 7febb9046700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb94018930 0x7feb9401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:45.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.979 7febb9046700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb94018930 0x7feb9401ae10 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:45.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:44.998 7febbb22a700 1 -- 172.21.15.32:0/2786258507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb40d72a0 con 0x7febb4096fe0 2021-04-09T18:23:45.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.001 7feba9ffb700 1 -- 172.21.15.32:0/2786258507 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7feba40091e0 con 0x7febb4096fe0 2021-04-09T18:23:45.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.195 7febbb22a700 1 -- 172.21.15.32:0/2786258507 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7febb40d72a0 con 0x7febb4096fe0 2021-04-09T18:23:45.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.753 7feba9ffb700 1 -- 172.21.15.32:0/2786258507 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v192) v1 ==== 105+0+0 (secure 0 0 0) 0x7feba4009f90 con 0x7febb4096fe0 2021-04-09T18:23:45.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 -- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb94018930 msgr2=0x7feb9401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb94018930 0x7feb9401ae10 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 -- 172.21.15.32:0/2786258507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb4096fe0 msgr2=0x7febb40973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb4096fe0 0x7febb40973e0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7feba400b650 tx=0x7feba400b9f0).stop 2021-04-09T18:23:45.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 -- 172.21.15.32:0/2786258507 shutdown_connections 2021-04-09T18:23:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb94018930 0x7feb9401ae10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb4096fe0 0x7febb40973e0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb417ffc0 0x7febb41803c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 --2- 172.21.15.32:0/2786258507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb4095e10 0x7febb4096210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:45.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 -- 172.21.15.32:0/2786258507 >> 172.21.15.32:0/2786258507 conn(0x7febb409bbc0 msgr2=0x7febb40ad9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:45.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 -- 172.21.15.32:0/2786258507 shutdown_connections 2021-04-09T18:23:45.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.762 7febbb22a700 1 -- 172.21.15.32:0/2786258507 wait complete. 2021-04-09T18:23:45.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:46.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.925 7f521bfd8700 1 Processor -- start 2021-04-09T18:23:46.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.926 7f521bfd8700 1 -- start start 2021-04-09T18:23:46.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.926 7f521bfd8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521410d010 0x7f52140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:46.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.926 7f521bfd8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521410d480 con 0x7f52140d1da0 2021-04-09T18:23:46.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.926 7f521bfd8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521410e1f0 con 0x7f52140d8490 2021-04-09T18:23:46.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.926 7f521bfd8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521410e780 con 0x7f521410d010 2021-04-09T18:23:46.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.926 7f52195f3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f52140d1da0 0x7f52140d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60190/0 (socket says 172.21.15.32:60190) 2021-04-09T18:23:46.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.926 7f52195f3700 1 -- 172.21.15.32:0/3800901042 learned_addr learned my addr 172.21.15.32:0/3800901042 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:46.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5219df4700 1 --2- 172.21.15.32:0/3800901042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521410d010 0x7f52140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:46.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5218df2700 1 -- 172.21.15.32:0/3800901042 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513771380 0 0) 0x7f5204000ce0 con 0x7f52140d1da0 2021-04-09T18:23:46.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5218df2700 1 -- 172.21.15.32:0/3800901042 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5200002fe0 con 0x7f52140d1da0 2021-04-09T18:23:46.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5218df2700 1 -- 172.21.15.32:0/3800901042 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523283109 0 0) 0x7f520c000d30 con 0x7f52140d8490 2021-04-09T18:23:46.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5218df2700 1 -- 172.21.15.32:0/3800901042 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5200003020 con 0x7f52140d8490 2021-04-09T18:23:46.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5219df4700 1 -- 172.21.15.32:0/3800901042 >> v1:172.21.15.86:6789/0 conn(0x7f52140d8490 legacy=0x7f521410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:46.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5219df4700 1 -- 172.21.15.32:0/3800901042 >> v1:172.21.15.32:6789/0 conn(0x7f52140d1da0 legacy=0x7f52140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:46.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5219df4700 1 -- 172.21.15.32:0/3800901042 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52140d8d30 con 0x7f521410d010 2021-04-09T18:23:46.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5219df4700 1 --2- 172.21.15.32:0/3800901042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521410d010 0x7f52140d1780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f5210006650 tx=0x7f5210009a80).ready entity=mon.2 client_cookie=ad1a8ab21ea7ec34 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:46.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5218df2700 1 -- 172.21.15.32:0/3800901042 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52100009b0 con 0x7f521410d010 2021-04-09T18:23:46.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5218df2700 1 -- 172.21.15.32:0/3800901042 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521000c0c0 con 0x7f521410d010 2021-04-09T18:23:46.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f5218df2700 1 -- 172.21.15.32:0/3800901042 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f521000c510 con 0x7f521410d010 2021-04-09T18:23:46.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f521bfd8700 1 -- 172.21.15.32:0/3800901042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521410d010 msgr2=0x7f52140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:46.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.927 7f521bfd8700 1 --2- 172.21.15.32:0/3800901042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521410d010 0x7f52140d1780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f5210006650 tx=0x7f5210009a80).stop 2021-04-09T18:23:46.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.928 7f521bfd8700 1 -- 172.21.15.32:0/3800901042 shutdown_connections 2021-04-09T18:23:46.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.928 7f521bfd8700 1 --2- 172.21.15.32:0/3800901042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521410d010 0x7f52140d1780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:46.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.928 7f521bfd8700 1 -- 172.21.15.32:0/3800901042 >> 172.21.15.32:0/3800901042 conn(0x7f52140ce5f0 msgr2=0x7f52140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:46.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.928 7f521bfd8700 1 -- 172.21.15.32:0/3800901042 shutdown_connections 2021-04-09T18:23:46.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.928 7f521bfd8700 1 -- 172.21.15.32:0/3800901042 wait complete. 2021-04-09T18:23:46.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.948 7f521bfd8700 1 Processor -- start 2021-04-09T18:23:46.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.948 7f521bfd8700 1 -- start start 2021-04-09T18:23:46.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f521bfd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5214092ad0 0x7f5214091d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:46.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f521bfd8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52140a2300 0x7f52140aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:46.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f521bfd8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52140befd0 0x7f52140bf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:46.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f521bfd8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52140a2700 con 0x7f5214092ad0 2021-04-09T18:23:46.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f521bfd8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5214092550 con 0x7f52140a2300 2021-04-09T18:23:46.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f521bfd8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52140aacb0 con 0x7f52140befd0 2021-04-09T18:23:46.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f5219df4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5214092ad0 0x7f5214091d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:46.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f5219df4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5214092ad0 0x7f5214091d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50604/0 (socket says 172.21.15.32:50604) 2021-04-09T18:23:46.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f5219df4700 1 -- 172.21.15.32:0/344003217 learned_addr learned my addr 172.21.15.32:0/344003217 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:46.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f521a5f5700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52140befd0 0x7f52140bf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:46.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f52195f3700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52140a2300 0x7f52140aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:46.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f5219df4700 1 -- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52140befd0 msgr2=0x7f52140bf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:46.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f5219df4700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52140befd0 0x7f52140bf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:46.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f5219df4700 1 -- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52140a2300 msgr2=0x7f52140aa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:46.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f5219df4700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52140a2300 0x7f52140aa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:46.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f5219df4700 1 -- 172.21.15.32:0/344003217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52140bf9f0 con 0x7f5214092ad0 2021-04-09T18:23:46.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f5219df4700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5214092ad0 0x7f5214091d50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f5210009650 tx=0x7f521000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:46.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.949 7f520a7fc700 1 -- 172.21.15.32:0/344003217 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52100093a0 con 0x7f5214092ad0 2021-04-09T18:23:46.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.950 7f520a7fc700 1 -- 172.21.15.32:0/344003217 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521000bd50 con 0x7f5214092ad0 2021-04-09T18:23:46.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.950 7f520a7fc700 1 -- 172.21.15.32:0/344003217 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52100052e0 con 0x7f5214092ad0 2021-04-09T18:23:46.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.950 7f521bfd8700 1 -- 172.21.15.32:0/344003217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52140bf9f0 con 0x7f5214092ad0 2021-04-09T18:23:46.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.950 7f521bfd8700 1 -- 172.21.15.32:0/344003217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52140bf9f0 con 0x7f5214092ad0 2021-04-09T18:23:46.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.950 7f520a7fc700 1 -- 172.21.15.32:0/344003217 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5210006030 con 0x7f5214092ad0 2021-04-09T18:23:46.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.950 7f520a7fc700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51fc018930 0x7f51fc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:46.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.950 7f520a7fc700 1 -- 172.21.15.32:0/344003217 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f52100329e0 con 0x7f5214092ad0 2021-04-09T18:23:46.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.951 7f52195f3700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51fc018930 0x7f51fc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:46.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.951 7f52195f3700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51fc018930 0x7f51fc01ae10 crc :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.970 7f521bfd8700 1 -- 172.21.15.32:0/344003217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52141b7050 con 0x7f5214092ad0 2021-04-09T18:23:46.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:45.974 7f520a7fc700 1 -- 172.21.15.32:0/344003217 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f52100009c0 con 0x7f5214092ad0 2021-04-09T18:23:46.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.166 7f521bfd8700 1 -- 172.21.15.32:0/344003217 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f521410d480 con 0x7f5214092ad0 2021-04-09T18:23:46.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.767 7f520a7fc700 1 -- 172.21.15.32:0/344003217 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v193) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5210006030 con 0x7f5214092ad0 2021-04-09T18:23:46.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 -- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51fc018930 msgr2=0x7f51fc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:46.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51fc018930 0x7f51fc01ae10 crc :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:46.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 -- 172.21.15.32:0/344003217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5214092ad0 msgr2=0x7f5214091d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:46.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5214092ad0 0x7f5214091d50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f5210009650 tx=0x7f521000b960).stop 2021-04-09T18:23:46.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 -- 172.21.15.32:0/344003217 shutdown_connections 2021-04-09T18:23:46.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51fc018930 0x7f51fc01ae10 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:46.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52140befd0 0x7f52140bf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:46.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52140a2300 0x7f52140aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:46.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 --2- 172.21.15.32:0/344003217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5214092ad0 0x7f5214091d50 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:46.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 -- 172.21.15.32:0/344003217 >> 172.21.15.32:0/344003217 conn(0x7f5214093070 msgr2=0x7f52140a3150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:46.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 -- 172.21.15.32:0/344003217 shutdown_connections 2021-04-09T18:23:46.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.784 7f521bfd8700 1 -- 172.21.15.32:0/344003217 wait complete. 2021-04-09T18:23:46.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:47.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.949 7f7d2b3f0700 1 Processor -- start 2021-04-09T18:23:47.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.949 7f7d2b3f0700 1 -- start start 2021-04-09T18:23:47.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d2b3f0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7d2410d010 0x7f7d240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:47.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d2b3f0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d240d8490 0x7f7d2410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:47.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d2b3f0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2410d480 con 0x7f7d240d1da0 2021-04-09T18:23:47.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d2b3f0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d240d8d30 con 0x7f7d240d8490 2021-04-09T18:23:47.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d2b3f0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d240d9220 con 0x7f7d2410d010 2021-04-09T18:23:47.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d29a0d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d240d8490 0x7f7d2410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:47.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d28a0b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7d240d1da0 0x7f7d240d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60202/0 (socket says 172.21.15.32:60202) 2021-04-09T18:23:47.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d28a0b700 1 -- 172.21.15.32:0/2177753443 learned_addr learned my addr 172.21.15.32:0/2177753443 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:47.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d2920c700 1 --2- 172.21.15.32:0/2177753443 >> v2:172.21.15.86:3300/0 conn(0x7f7d2410d010 0x7f7d240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:47.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d29a0d700 1 -- 172.21.15.32:0/2177753443 >> v2:172.21.15.86:3300/0 conn(0x7f7d2410d010 msgr2=0x7f7d240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:47.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d29a0d700 1 --2- 172.21.15.32:0/2177753443 >> v2:172.21.15.86:3300/0 conn(0x7f7d2410d010 0x7f7d240d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d29a0d700 1 -- 172.21.15.32:0/2177753443 >> v1:172.21.15.32:6789/0 conn(0x7f7d240d1da0 legacy=0x7f7d240d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:47.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d1bfff700 1 -- 172.21.15.32:0/2177753443 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042784733 0 0) 0x7f7d14000ce0 con 0x7f7d240d1da0 2021-04-09T18:23:47.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.950 7f7d29a0d700 1 -- 172.21.15.32:0/2177753443 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2410e020 con 0x7f7d240d8490 2021-04-09T18:23:47.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.951 7f7d29a0d700 1 --2- 172.21.15.32:0/2177753443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240d8490 0x7f7d2410da00 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7f7d1c006e30 tx=0x7f7d1c009e60).ready entity=mon.0 client_cookie=8e3b727e08c7efdc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:47.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.951 7f7d1bfff700 1 -- 172.21.15.32:0/2177753443 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d1c006a50 con 0x7f7d240d8490 2021-04-09T18:23:47.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.951 7f7d1bfff700 1 -- 172.21.15.32:0/2177753443 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1c00b300 con 0x7f7d240d8490 2021-04-09T18:23:47.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.951 7f7d1bfff700 1 -- 172.21.15.32:0/2177753443 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d1c00b750 con 0x7f7d240d8490 2021-04-09T18:23:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.951 7f7d2b3f0700 1 -- 172.21.15.32:0/2177753443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240d8490 msgr2=0x7f7d2410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.951 7f7d2b3f0700 1 --2- 172.21.15.32:0/2177753443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240d8490 0x7f7d2410da00 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7f7d1c006e30 tx=0x7f7d1c009e60).stop 2021-04-09T18:23:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.952 7f7d2b3f0700 1 -- 172.21.15.32:0/2177753443 shutdown_connections 2021-04-09T18:23:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.952 7f7d2b3f0700 1 --2- 172.21.15.32:0/2177753443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240d8490 0x7f7d2410da00 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.952 7f7d2b3f0700 1 --2- 172.21.15.32:0/2177753443 >> v2:172.21.15.86:3300/0 conn(0x7f7d2410d010 0x7f7d240d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.952 7f7d2b3f0700 1 -- 172.21.15.32:0/2177753443 >> 172.21.15.32:0/2177753443 conn(0x7f7d240ce5f0 msgr2=0x7f7d240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:47.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.952 7f7d2b3f0700 1 -- 172.21.15.32:0/2177753443 shutdown_connections 2021-04-09T18:23:47.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.952 7f7d2b3f0700 1 -- 172.21.15.32:0/2177753443 wait complete. 2021-04-09T18:23:47.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.972 7f7d2b3f0700 1 Processor -- start 2021-04-09T18:23:47.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.972 7f7d2b3f0700 1 -- start start 2021-04-09T18:23:47.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.972 7f7d2b3f0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240c3c00 0x7f7d2409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:47.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2b3f0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d24095240 0x7f7d241a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:47.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2b3f0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d241a1270 0x7f7d240b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:47.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2b3f0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2409cda0 con 0x7f7d240c3c00 2021-04-09T18:23:47.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2b3f0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d240c3680 con 0x7f7d24095240 2021-04-09T18:23:47.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2b3f0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d24095640 con 0x7f7d241a1270 2021-04-09T18:23:47.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2920c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240c3c00 0x7f7d2409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:47.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2920c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240c3c00 0x7f7d2409c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50616/0 (socket says 172.21.15.32:50616) 2021-04-09T18:23:47.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2920c700 1 -- 172.21.15.32:0/1390923845 learned_addr learned my addr 172.21.15.32:0/1390923845 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:47.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d29a0d700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d241a1270 0x7f7d240b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:47.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d28a0b700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d24095240 0x7f7d241a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:47.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2920c700 1 -- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d241a1270 msgr2=0x7f7d240b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:47.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2920c700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d241a1270 0x7f7d240b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2920c700 1 -- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d24095240 msgr2=0x7f7d241a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:47.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2920c700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d24095240 0x7f7d241a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2920c700 1 -- 172.21.15.32:0/1390923845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d241a1670 con 0x7f7d240c3c00 2021-04-09T18:23:47.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2920c700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240c3c00 0x7f7d2409c600 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7f7d20006be0 tx=0x7f7d2000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:47.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d19ffb700 1 -- 172.21.15.32:0/1390923845 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d200064d0 con 0x7f7d240c3c00 2021-04-09T18:23:47.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d19ffb700 1 -- 172.21.15.32:0/1390923845 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2000bd80 con 0x7f7d240c3c00 2021-04-09T18:23:47.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d19ffb700 1 -- 172.21.15.32:0/1390923845 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d20002030 con 0x7f7d240c3c00 2021-04-09T18:23:47.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d241a1670 con 0x7f7d240c3c00 2021-04-09T18:23:47.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.973 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d241a1670 con 0x7f7d240c3c00 2021-04-09T18:23:47.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.974 7f7d19ffb700 1 -- 172.21.15.32:0/1390923845 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7d20006c90 con 0x7f7d240c3c00 2021-04-09T18:23:47.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.974 7f7d19ffb700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d04018930 0x7f7d0401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:47.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.974 7f7d19ffb700 1 -- 172.21.15.32:0/1390923845 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7d20032a10 con 0x7f7d240c3c00 2021-04-09T18:23:47.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.974 7f7d28a0b700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d04018930 0x7f7d0401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:47.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.974 7f7d28a0b700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d04018930 0x7f7d0401ae10 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:47.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.994 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d24093d50 con 0x7f7d240c3c00 2021-04-09T18:23:47.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:46.997 7f7d19ffb700 1 -- 172.21.15.32:0/1390923845 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7d20006820 con 0x7f7d240c3c00 2021-04-09T18:23:47.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.189 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f7d2410d480 con 0x7f7d240c3c00 2021-04-09T18:23:47.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.774 7f7d19ffb700 1 -- 172.21.15.32:0/1390923845 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v194) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7d20006c90 con 0x7f7d240c3c00 2021-04-09T18:23:47.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.806 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d04018930 msgr2=0x7f7d0401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:47.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.806 7f7d2b3f0700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d04018930 0x7f7d0401ae10 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.806 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240c3c00 msgr2=0x7f7d2409c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:47.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.806 7f7d2b3f0700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240c3c00 0x7f7d2409c600 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7f7d20006be0 tx=0x7f7d2000b990).stop 2021-04-09T18:23:47.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.807 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 shutdown_connections 2021-04-09T18:23:47.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.807 7f7d2b3f0700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d04018930 0x7f7d0401ae10 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.807 7f7d2b3f0700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d241a1270 0x7f7d240b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.807 7f7d2b3f0700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d24095240 0x7f7d241a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.807 7f7d2b3f0700 1 --2- 172.21.15.32:0/1390923845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d240c3c00 0x7f7d2409c600 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:47.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.807 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 >> 172.21.15.32:0/1390923845 conn(0x7f7d240975f0 msgr2=0x7f7d24096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:47.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.807 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 shutdown_connections 2021-04-09T18:23:47.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.807 7f7d2b3f0700 1 -- 172.21.15.32:0/1390923845 wait complete. 2021-04-09T18:23:47.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:48.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.970 7f76f196c700 1 Processor -- start 2021-04-09T18:23:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.970 7f76f196c700 1 -- start start 2021-04-09T18:23:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76f196c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76ec0d7ca0 0x7f76ec0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76f196c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f76ec10da00 0x7f76ec10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76f196c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec10d480 con 0x7f76ec10d010 2021-04-09T18:23:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76f196c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ec0d9220 con 0x7f76ec0d7ca0 2021-04-09T18:23:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76f196c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ec112100 con 0x7f76ec10da00 2021-04-09T18:23:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76ead73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76ec0d7ca0 0x7f76ec0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76ead73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76ec0d7ca0 0x7f76ec0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50628/0 (socket says 172.21.15.32:50628) 2021-04-09T18:23:48.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76ead73700 1 -- 172.21.15.32:0/2127468263 learned_addr learned my addr 172.21.15.32:0/2127468263 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:48.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76ebd75700 1 --2- 172.21.15.32:0/2127468263 >> v2:172.21.15.86:3300/0 conn(0x7f76ec10da00 0x7f76ec10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:48.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76ead73700 1 -- 172.21.15.32:0/2127468263 >> v2:172.21.15.86:3300/0 conn(0x7f76ec10da00 msgr2=0x7f76ec10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:48.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76ead73700 1 --2- 172.21.15.32:0/2127468263 >> v2:172.21.15.86:3300/0 conn(0x7f76ec10da00 0x7f76ec10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.971 7f76ead73700 1 -- 172.21.15.32:0/2127468263 >> v1:172.21.15.86:6789/0 conn(0x7f76ec10d010 legacy=0x7f76ec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:48.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.972 7f76ead73700 1 -- 172.21.15.32:0/2127468263 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec0d86c0 con 0x7f76ec0d7ca0 2021-04-09T18:23:48.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.972 7f76ead73700 1 --2- 172.21.15.32:0/2127468263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0d7ca0 0x7f76ec0d80a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f76e0006700 tx=0x7f76e0009b80).ready entity=mon.0 client_cookie=9e5a429f76314e94 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:48.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.972 7f76ea572700 1 -- 172.21.15.32:0/2127468263 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76e000c0c0 con 0x7f76ec0d7ca0 2021-04-09T18:23:48.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.972 7f76ea572700 1 -- 172.21.15.32:0/2127468263 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e000c440 con 0x7f76ec0d7ca0 2021-04-09T18:23:48.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.972 7f76ea572700 1 -- 172.21.15.32:0/2127468263 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76e000c920 con 0x7f76ec0d7ca0 2021-04-09T18:23:48.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.972 7f76f196c700 1 -- 172.21.15.32:0/2127468263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0d7ca0 msgr2=0x7f76ec0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:48.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.972 7f76f196c700 1 --2- 172.21.15.32:0/2127468263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0d7ca0 0x7f76ec0d80a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f76e0006700 tx=0x7f76e0009b80).stop 2021-04-09T18:23:48.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.973 7f76f196c700 1 -- 172.21.15.32:0/2127468263 shutdown_connections 2021-04-09T18:23:48.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.973 7f76f196c700 1 --2- 172.21.15.32:0/2127468263 >> v2:172.21.15.86:3300/0 conn(0x7f76ec10da00 0x7f76ec10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.973 7f76f196c700 1 --2- 172.21.15.32:0/2127468263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0d7ca0 0x7f76ec0d80a0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.973 7f76f196c700 1 -- 172.21.15.32:0/2127468263 >> 172.21.15.32:0/2127468263 conn(0x7f76ec0ce5f0 msgr2=0x7f76ec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:48.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.973 7f76f196c700 1 -- 172.21.15.32:0/2127468263 shutdown_connections 2021-04-09T18:23:48.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.973 7f76f196c700 1 -- 172.21.15.32:0/2127468263 wait complete. 2021-04-09T18:23:48.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.993 7f76f196c700 1 Processor -- start 2021-04-09T18:23:48.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.993 7f76f196c700 1 -- start start 2021-04-09T18:23:48.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.993 7f76f196c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0c3c00 0x7f76ec09c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:48.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.993 7f76f196c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76ec095260 0x7f76ec1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:48.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.993 7f76f196c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76ec1a1270 0x7f76ec0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:48.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.993 7f76f196c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec09cda0 con 0x7f76ec0c3c00 2021-04-09T18:23:48.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.993 7f76f196c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec0c3680 con 0x7f76ec1a1270 2021-04-09T18:23:48.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.993 7f76f196c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec095660 con 0x7f76ec095260 2021-04-09T18:23:48.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.993 7f76eb574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0c3c00 0x7f76ec09c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:48.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76eb574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0c3c00 0x7f76ec09c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50630/0 (socket says 172.21.15.32:50630) 2021-04-09T18:23:48.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76eb574700 1 -- 172.21.15.32:0/26122304 learned_addr learned my addr 172.21.15.32:0/26122304 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:48.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76ead73700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76ec095260 0x7f76ec1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:48.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76ebd75700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76ec1a1270 0x7f76ec0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:48.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76eb574700 1 -- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76ec095260 msgr2=0x7f76ec1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:48.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76eb574700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76ec095260 0x7f76ec1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76eb574700 1 -- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76ec1a1270 msgr2=0x7f76ec0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:48.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76eb574700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76ec1a1270 0x7f76ec0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76eb574700 1 -- 172.21.15.32:0/26122304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec1a1670 con 0x7f76ec0c3c00 2021-04-09T18:23:48.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76eb574700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0c3c00 0x7f76ec09c600 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f76dc00b530 tx=0x7f76dc00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:48.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76d3fff700 1 -- 172.21.15.32:0/26122304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76dc00bcc0 con 0x7f76ec0c3c00 2021-04-09T18:23:48.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76d3fff700 1 -- 172.21.15.32:0/26122304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76dc006020 con 0x7f76ec0c3c00 2021-04-09T18:23:48.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76f196c700 1 -- 172.21.15.32:0/26122304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ec1a1670 con 0x7f76ec0c3c00 2021-04-09T18:23:48.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76d3fff700 1 -- 172.21.15.32:0/26122304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76dc0054a0 con 0x7f76ec0c3c00 2021-04-09T18:23:48.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.994 7f76f196c700 1 -- 172.21.15.32:0/26122304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ec1a1670 con 0x7f76ec0c3c00 2021-04-09T18:23:48.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.995 7f76d3fff700 1 -- 172.21.15.32:0/26122304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f76dc006200 con 0x7f76ec0c3c00 2021-04-09T18:23:48.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.995 7f76d3fff700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76cc018930 0x7f76cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:48.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.995 7f76d3fff700 1 -- 172.21.15.32:0/26122304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f76dc032b70 con 0x7f76ec0c3c00 2021-04-09T18:23:48.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.995 7f76ead73700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76cc018930 0x7f76cc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:48.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:47.995 7f76ead73700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76cc018930 0x7f76cc01adc0 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:48.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.015 7f76f196c700 1 -- 172.21.15.32:0/26122304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ec10d480 con 0x7f76ec0c3c00 2021-04-09T18:23:48.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.019 7f76d3fff700 1 -- 172.21.15.32:0/26122304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f76dc00bcc0 con 0x7f76ec0c3c00 2021-04-09T18:23:48.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.214 7f76f196c700 1 -- 172.21.15.32:0/26122304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f76ec10d480 con 0x7f76ec0c3c00 2021-04-09T18:23:48.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.779 7f76d3fff700 1 -- 172.21.15.32:0/26122304 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v195) v1 ==== 105+0+0 (secure 0 0 0) 0x7f76dc006020 con 0x7f76ec0c3c00 2021-04-09T18:23:48.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 -- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76cc018930 msgr2=0x7f76cc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:48.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76cc018930 0x7f76cc01adc0 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 -- 172.21.15.32:0/26122304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0c3c00 msgr2=0x7f76ec09c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:48.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0c3c00 0x7f76ec09c600 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f76dc00b530 tx=0x7f76dc00b8d0).stop 2021-04-09T18:23:48.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 -- 172.21.15.32:0/26122304 shutdown_connections 2021-04-09T18:23:48.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76cc018930 0x7f76cc01adc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76ec1a1270 0x7f76ec0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76ec095260 0x7f76ec1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 --2- 172.21.15.32:0/26122304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76ec0c3c00 0x7f76ec09c600 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:48.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 -- 172.21.15.32:0/26122304 >> 172.21.15.32:0/26122304 conn(0x7f76ec0975f0 msgr2=0x7f76ec096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.831 7f76f196c700 1 -- 172.21.15.32:0/26122304 shutdown_connections 2021-04-09T18:23:48.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.832 7f76f196c700 1 -- 172.21.15.32:0/26122304 wait complete. 2021-04-09T18:23:48.902 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:49.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.997 7f1a4f8be700 1 Processor -- start 2021-04-09T18:23:49.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.997 7f1a4f8be700 1 -- start start 2021-04-09T18:23:49.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.998 7f1a4f8be700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1a4810d010 0x7f1a480d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:49.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.998 7f1a4f8be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480d8490 0x7f1a4810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:49.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.998 7f1a4f8be700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4810d480 con 0x7f1a480d1da0 2021-04-09T18:23:49.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.998 7f1a4f8be700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a480d8d30 con 0x7f1a4810d010 2021-04-09T18:23:49.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.998 7f1a4f8be700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a480d9220 con 0x7f1a480d8490 2021-04-09T18:23:49.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.998 7f1a4d6da700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1a4810d010 0x7f1a480d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:49.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.998 7f1a4dedb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480d8490 0x7f1a4810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:49.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.998 7f1a4ced9700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1a480d1da0 0x7f1a480d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58158/0 (socket says 172.21.15.32:58158) 2021-04-09T18:23:49.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.998 7f1a4ced9700 1 -- 172.21.15.32:0/980505753 learned_addr learned my addr 172.21.15.32:0/980505753 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:49.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a4d6da700 1 -- 172.21.15.32:0/980505753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480d8490 msgr2=0x7f1a4810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:49.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a4d6da700 1 --2- 172.21.15.32:0/980505753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480d8490 0x7f1a4810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a4d6da700 1 -- 172.21.15.32:0/980505753 >> v1:172.21.15.86:6789/0 conn(0x7f1a480d1da0 legacy=0x7f1a480d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:49.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a3ffff700 1 -- 172.21.15.32:0/980505753 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874487149 0 0) 0x7f1a380010c0 con 0x7f1a480d1da0 2021-04-09T18:23:49.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a4d6da700 1 -- 172.21.15.32:0/980505753 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4810e020 con 0x7f1a4810d010 2021-04-09T18:23:49.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a4dedb700 1 --2- 172.21.15.32:0/980505753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480d8490 0x7f1a4810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:49.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a4d6da700 1 --2- 172.21.15.32:0/980505753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a4810d010 0x7f1a480d1780 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f1a440069d0 tx=0x7f1a44009e40).ready entity=mon.1 client_cookie=d43d2e52a867058a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:49.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a3ffff700 1 -- 172.21.15.32:0/980505753 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a4400b510 con 0x7f1a4810d010 2021-04-09T18:23:49.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a3ffff700 1 -- 172.21.15.32:0/980505753 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4400b890 con 0x7f1a4810d010 2021-04-09T18:23:49.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a3ffff700 1 -- 172.21.15.32:0/980505753 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a4400bd70 con 0x7f1a4810d010 2021-04-09T18:23:49.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a4f8be700 1 -- 172.21.15.32:0/980505753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a4810d010 msgr2=0x7f1a480d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:49.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:48.999 7f1a4f8be700 1 --2- 172.21.15.32:0/980505753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a4810d010 0x7f1a480d1780 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f1a440069d0 tx=0x7f1a44009e40).stop 2021-04-09T18:23:49.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.000 7f1a4f8be700 1 -- 172.21.15.32:0/980505753 shutdown_connections 2021-04-09T18:23:49.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.000 7f1a4f8be700 1 --2- 172.21.15.32:0/980505753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480d8490 0x7f1a4810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.000 7f1a4f8be700 1 --2- 172.21.15.32:0/980505753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a4810d010 0x7f1a480d1780 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.000 7f1a4f8be700 1 -- 172.21.15.32:0/980505753 >> 172.21.15.32:0/980505753 conn(0x7f1a480ce5f0 msgr2=0x7f1a480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:49.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.000 7f1a4f8be700 1 -- 172.21.15.32:0/980505753 shutdown_connections 2021-04-09T18:23:49.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.000 7f1a4f8be700 1 -- 172.21.15.32:0/980505753 wait complete. 2021-04-09T18:23:49.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.020 7f1a4f8be700 1 Processor -- start 2021-04-09T18:23:49.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.020 7f1a4f8be700 1 -- start start 2021-04-09T18:23:49.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4f8be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480c3c00 0x7f1a4809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:49.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4f8be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a48095240 0x7f1a481a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:49.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4f8be700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a481a1270 0x7f1a480b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:49.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4f8be700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4809cda0 con 0x7f1a48095240 2021-04-09T18:23:49.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4f8be700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a480c3680 con 0x7f1a481a1270 2021-04-09T18:23:49.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4f8be700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a48095640 con 0x7f1a480c3c00 2021-04-09T18:23:49.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4ced9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a48095240 0x7f1a481a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:49.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4ced9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a48095240 0x7f1a481a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50646/0 (socket says 172.21.15.32:50646) 2021-04-09T18:23:49.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4ced9700 1 -- 172.21.15.32:0/4028715269 learned_addr learned my addr 172.21.15.32:0/4028715269 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:49.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4dedb700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a481a1270 0x7f1a480b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:49.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4d6da700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480c3c00 0x7f1a4809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:49.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4ced9700 1 -- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480c3c00 msgr2=0x7f1a4809c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:49.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4ced9700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480c3c00 0x7f1a4809c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4ced9700 1 -- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a481a1270 msgr2=0x7f1a480b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:49.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4ced9700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a481a1270 0x7f1a480b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4ced9700 1 -- 172.21.15.32:0/4028715269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a481a1670 con 0x7f1a48095240 2021-04-09T18:23:49.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4dedb700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a481a1270 0x7f1a480b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:49.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4ced9700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a48095240 0x7f1a481a0c50 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7f1a3800a5d0 tx=0x7f1a3800a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:49.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a3dffb700 1 -- 172.21.15.32:0/4028715269 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a38000c10 con 0x7f1a48095240 2021-04-09T18:23:49.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a3dffb700 1 -- 172.21.15.32:0/4028715269 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a380070d0 con 0x7f1a48095240 2021-04-09T18:23:49.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a481a1670 con 0x7f1a48095240 2021-04-09T18:23:49.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a3dffb700 1 -- 172.21.15.32:0/4028715269 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a380047b0 con 0x7f1a48095240 2021-04-09T18:23:49.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.021 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a481a1670 con 0x7f1a48095240 2021-04-09T18:23:49.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.023 7f1a3dffb700 1 -- 172.21.15.32:0/4028715269 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1a38000c10 con 0x7f1a48095240 2021-04-09T18:23:49.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.023 7f1a3dffb700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a280188c0 0x7f1a2801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:49.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.023 7f1a3dffb700 1 -- 172.21.15.32:0/4028715269 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1a38032770 con 0x7f1a48095240 2021-04-09T18:23:49.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.023 7f1a4d6da700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a280188c0 0x7f1a2801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:49.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.023 7f1a4d6da700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a280188c0 0x7f1a2801adc0 crc :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:49.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.042 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a48093d50 con 0x7f1a48095240 2021-04-09T18:23:49.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.045 7f1a3dffb700 1 -- 172.21.15.32:0/4028715269 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1a38000c10 con 0x7f1a48095240 2021-04-09T18:23:49.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.241 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f1a4810d480 con 0x7f1a48095240 2021-04-09T18:23:49.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.783 7f1a3dffb700 1 -- 172.21.15.32:0/4028715269 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v196) v1 ==== 105+0+0 (secure 0 0 0) 0x7f1a38004680 con 0x7f1a48095240 2021-04-09T18:23:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.808 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a280188c0 msgr2=0x7f1a2801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.808 7f1a4f8be700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a280188c0 0x7f1a2801adc0 crc :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.808 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a48095240 msgr2=0x7f1a481a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.808 7f1a4f8be700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a48095240 0x7f1a481a0c50 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7f1a3800a5d0 tx=0x7f1a3800a970).stop 2021-04-09T18:23:49.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.809 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 shutdown_connections 2021-04-09T18:23:49.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.809 7f1a4f8be700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a280188c0 0x7f1a2801adc0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.809 7f1a4f8be700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a48095240 0x7f1a481a0c50 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.809 7f1a4f8be700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a481a1270 0x7f1a480b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.809 7f1a4f8be700 1 --2- 172.21.15.32:0/4028715269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a480c3c00 0x7f1a4809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:49.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.809 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 >> 172.21.15.32:0/4028715269 conn(0x7f1a480975f0 msgr2=0x7f1a48096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:49.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.809 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 shutdown_connections 2021-04-09T18:23:49.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.809 7f1a4f8be700 1 -- 172.21.15.32:0/4028715269 wait complete. 2021-04-09T18:23:49.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:50.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.977 7f9813c3e700 1 Processor -- start 2021-04-09T18:23:50.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.977 7f9813c3e700 1 -- start start 2021-04-09T18:23:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9813c3e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f980c10d010 0x7f980c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9813c3e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f980c0d1da0 0x7f980c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9813c3e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c10d480 con 0x7f980c0d82c0 2021-04-09T18:23:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9813c3e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c0d8d30 con 0x7f980c0d1da0 2021-04-09T18:23:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9813c3e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c0d9220 con 0x7f980c10d010 2021-04-09T18:23:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9811259700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f980c0d1da0 0x7f980c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f981225b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f980c0d82c0 0x7f980c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60244/0 (socket says 172.21.15.32:60244) 2021-04-09T18:23:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f981225b700 1 -- 172.21.15.32:0/3981309018 learned_addr learned my addr 172.21.15.32:0/3981309018 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9811a5a700 1 --2- 172.21.15.32:0/3981309018 >> v2:172.21.15.86:3300/0 conn(0x7f980c10d010 0x7f980c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9810a58700 1 -- 172.21.15.32:0/3981309018 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602598188 0 0) 0x7f9804000d30 con 0x7f980c0d82c0 2021-04-09T18:23:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9810a58700 1 -- 172.21.15.32:0/3981309018 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f8002fe0 con 0x7f980c0d82c0 2021-04-09T18:23:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9811259700 1 -- 172.21.15.32:0/3981309018 >> v2:172.21.15.86:3300/0 conn(0x7f980c10d010 msgr2=0x7f980c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:50.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9811259700 1 --2- 172.21.15.32:0/3981309018 >> v2:172.21.15.86:3300/0 conn(0x7f980c10d010 0x7f980c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9811259700 1 -- 172.21.15.32:0/3981309018 >> v1:172.21.15.32:6789/0 conn(0x7f980c0d82c0 legacy=0x7f980c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:50.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9811259700 1 -- 172.21.15.32:0/3981309018 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c0d86f0 con 0x7f980c0d1da0 2021-04-09T18:23:50.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9811a5a700 1 --2- 172.21.15.32:0/3981309018 >> v2:172.21.15.86:3300/0 conn(0x7f980c10d010 0x7f980c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:50.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9811259700 1 --2- 172.21.15.32:0/3981309018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0d1da0 0x7f980c0d7ca0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f97fc006650 tx=0x7f97fc009a80).ready entity=mon.0 client_cookie=bec7b1fc55d0795f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9810a58700 1 -- 172.21.15.32:0/3981309018 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97fc0009b0 con 0x7f980c0d1da0 2021-04-09T18:23:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9810a58700 1 -- 172.21.15.32:0/3981309018 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc00c0c0 con 0x7f980c0d1da0 2021-04-09T18:23:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.978 7f9810a58700 1 -- 172.21.15.32:0/3981309018 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97fc00c510 con 0x7f980c0d1da0 2021-04-09T18:23:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.979 7f9813c3e700 1 -- 172.21.15.32:0/3981309018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0d1da0 msgr2=0x7f980c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:50.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.979 7f9813c3e700 1 --2- 172.21.15.32:0/3981309018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0d1da0 0x7f980c0d7ca0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f97fc006650 tx=0x7f97fc009a80).stop 2021-04-09T18:23:50.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.979 7f9813c3e700 1 -- 172.21.15.32:0/3981309018 shutdown_connections 2021-04-09T18:23:50.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.979 7f9813c3e700 1 --2- 172.21.15.32:0/3981309018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0d1da0 0x7f980c0d7ca0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.979 7f9813c3e700 1 --2- 172.21.15.32:0/3981309018 >> v2:172.21.15.86:3300/0 conn(0x7f980c10d010 0x7f980c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.979 7f9813c3e700 1 -- 172.21.15.32:0/3981309018 >> 172.21.15.32:0/3981309018 conn(0x7f980c0ce5f0 msgr2=0x7f980c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:50.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.979 7f9813c3e700 1 -- 172.21.15.32:0/3981309018 shutdown_connections 2021-04-09T18:23:50.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:49.979 7f9813c3e700 1 -- 172.21.15.32:0/3981309018 wait complete. 2021-04-09T18:23:50.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9813c3e700 1 Processor -- start 2021-04-09T18:23:50.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9813c3e700 1 -- start start 2021-04-09T18:23:50.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9813c3e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0c3c00 0x7f980c09c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:50.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9813c3e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c095240 0x7f980c1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:50.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9813c3e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1a1270 0x7f980c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:50.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9813c3e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c09cda0 con 0x7f980c0c3c00 2021-04-09T18:23:50.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9813c3e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c0c3680 con 0x7f980c1a1270 2021-04-09T18:23:50.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9813c3e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c095640 con 0x7f980c095240 2021-04-09T18:23:50.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9811a5a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0c3c00 0x7f980c09c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:50.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.000 7f9811a5a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0c3c00 0x7f980c09c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50658/0 (socket says 172.21.15.32:50658) 2021-04-09T18:23:50.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9811a5a700 1 -- 172.21.15.32:0/3425030640 learned_addr learned my addr 172.21.15.32:0/3425030640 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:50.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f981225b700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1a1270 0x7f980c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:50.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9811259700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c095240 0x7f980c1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:50.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9811a5a700 1 -- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c095240 msgr2=0x7f980c1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:50.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9811a5a700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c095240 0x7f980c1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9811a5a700 1 -- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1a1270 msgr2=0x7f980c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:50.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9811a5a700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1a1270 0x7f980c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9811a5a700 1 -- 172.21.15.32:0/3425030640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c1a1670 con 0x7f980c0c3c00 2021-04-09T18:23:50.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9811a5a700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0c3c00 0x7f980c09c600 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f9808006f40 tx=0x7f980800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:50.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f98027fc700 1 -- 172.21.15.32:0/3425030640 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980800bd80 con 0x7f980c0c3c00 2021-04-09T18:23:50.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f98027fc700 1 -- 172.21.15.32:0/3425030640 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9808000c90 con 0x7f980c0c3c00 2021-04-09T18:23:50.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c1a1670 con 0x7f980c0c3c00 2021-04-09T18:23:50.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f98027fc700 1 -- 172.21.15.32:0/3425030640 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9808002220 con 0x7f980c0c3c00 2021-04-09T18:23:50.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.001 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c1a1670 con 0x7f980c0c3c00 2021-04-09T18:23:50.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.002 7f98027fc700 1 -- 172.21.15.32:0/3425030640 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f980800a000 con 0x7f980c0c3c00 2021-04-09T18:23:50.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.002 7f98027fc700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97f4018930 0x7f97f401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:50.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.002 7f98027fc700 1 -- 172.21.15.32:0/3425030640 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f98080329e0 con 0x7f980c0c3c00 2021-04-09T18:23:50.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.002 7f9811259700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97f4018930 0x7f97f401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:50.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.002 7f9811259700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97f4018930 0x7f97f401ae10 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:50.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.022 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c093d50 con 0x7f980c0c3c00 2021-04-09T18:23:50.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.025 7f98027fc700 1 -- 172.21.15.32:0/3425030640 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9808000c90 con 0x7f980c0c3c00 2021-04-09T18:23:50.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.220 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f980c10d480 con 0x7f980c0c3c00 2021-04-09T18:23:50.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.793 7f98027fc700 1 -- 172.21.15.32:0/3425030640 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v197) v1 ==== 105+0+0 (secure 0 0 0) 0x7f980800a000 con 0x7f980c0c3c00 2021-04-09T18:23:50.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97f4018930 msgr2=0x7f97f401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:50.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97f4018930 0x7f97f401ae10 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0c3c00 msgr2=0x7f980c09c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0c3c00 0x7f980c09c600 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f9808006f40 tx=0x7f980800b990).stop 2021-04-09T18:23:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 shutdown_connections 2021-04-09T18:23:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97f4018930 0x7f97f401ae10 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1a1270 0x7f980c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c095240 0x7f980c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 --2- 172.21.15.32:0/3425030640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c0c3c00 0x7f980c09c600 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:50.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 >> 172.21.15.32:0/3425030640 conn(0x7f980c0975f0 msgr2=0x7f980c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:50.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 shutdown_connections 2021-04-09T18:23:50.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:50.838 7f9813c3e700 1 -- 172.21.15.32:0/3425030640 wait complete. 2021-04-09T18:23:50.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:50.913 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:23:51.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.001 7fa12f363700 1 Processor -- start 2021-04-09T18:23:51.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.002 7fa12f363700 1 -- start start 2021-04-09T18:23:51.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.002 7fa12f363700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa12810da00 0x7fa12810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:51.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.002 7fa12f363700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12810d480 con 0x7fa12810d010 2021-04-09T18:23:51.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.002 7fa12f363700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1280d1f70 con 0x7fa1280d7ca0 2021-04-09T18:23:51.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.002 7fa12f363700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1280d8d30 con 0x7fa12810da00 2021-04-09T18:23:51.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.002 7fa12d980700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa12810da00 0x7fa12810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.002 7fa12d17f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa12810d010 0x7fa1280d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60268/0 (socket says 172.21.15.32:60268) 2021-04-09T18:23:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.002 7fa12d17f700 1 -- 172.21.15.32:0/2658923644 learned_addr learned my addr 172.21.15.32:0/2658923644 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa12d980700 1 -- 172.21.15.32:0/2658923644 >> v1:172.21.15.86:6789/0 conn(0x7fa1280d7ca0 legacy=0x7fa1280d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa12d980700 1 -- 172.21.15.32:0/2658923644 >> v1:172.21.15.32:6789/0 conn(0x7fa12810d010 legacy=0x7fa1280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa127fff700 1 -- 172.21.15.32:0/2658923644 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245268148 0 0) 0x7fa11c000ce0 con 0x7fa12810d010 2021-04-09T18:23:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa12d980700 1 -- 172.21.15.32:0/2658923644 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1280d9220 con 0x7fa12810da00 2021-04-09T18:23:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa12d980700 1 --2- 172.21.15.32:0/2658923644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa12810da00 0x7fa12810de00 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7fa120006e30 tx=0x7fa120009de0).ready entity=mon.0 client_cookie=d606aa01c6ef969b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa127fff700 1 -- 172.21.15.32:0/2658923644 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa12000b300 con 0x7fa12810da00 2021-04-09T18:23:51.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa127fff700 1 -- 172.21.15.32:0/2658923644 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12000b680 con 0x7fa12810da00 2021-04-09T18:23:51.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa127fff700 1 -- 172.21.15.32:0/2658923644 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa12000bb60 con 0x7fa12810da00 2021-04-09T18:23:51.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa12f363700 1 -- 172.21.15.32:0/2658923644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa12810da00 msgr2=0x7fa12810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.003 7fa12f363700 1 --2- 172.21.15.32:0/2658923644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa12810da00 0x7fa12810de00 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7fa120006e30 tx=0x7fa120009de0).stop 2021-04-09T18:23:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.004 7fa12f363700 1 -- 172.21.15.32:0/2658923644 shutdown_connections 2021-04-09T18:23:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.004 7fa12f363700 1 --2- 172.21.15.32:0/2658923644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa12810da00 0x7fa12810de00 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.004 7fa12f363700 1 -- 172.21.15.32:0/2658923644 >> 172.21.15.32:0/2658923644 conn(0x7fa1280ce5f0 msgr2=0x7fa1280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:51.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.004 7fa12f363700 1 -- 172.21.15.32:0/2658923644 shutdown_connections 2021-04-09T18:23:51.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.004 7fa12f363700 1 -- 172.21.15.32:0/2658923644 wait complete. 2021-04-09T18:23:51.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.024 7fa12f363700 1 Processor -- start 2021-04-09T18:23:51.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.024 7fa12f363700 1 -- start start 2021-04-09T18:23:51.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12f363700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1280aece0 0x7fa1280ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:51.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12f363700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1280acf30 0x7fa1280aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:51.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12f363700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa128196fa0 0x7fa1280ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:51.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12f363700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa128092a70 con 0x7fa1280acf30 2021-04-09T18:23:51.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12f363700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280abba0 con 0x7fa1280aece0 2021-04-09T18:23:51.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12f363700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280aa8a0 con 0x7fa128196fa0 2021-04-09T18:23:51.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12c97e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1280acf30 0x7fa1280aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:51.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12c97e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1280acf30 0x7fa1280aae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50686/0 (socket says 172.21.15.32:50686) 2021-04-09T18:23:51.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12c97e700 1 -- 172.21.15.32:0/952062195 learned_addr learned my addr 172.21.15.32:0/952062195 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:51.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12d980700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa128196fa0 0x7fa1280ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:51.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12d17f700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1280aece0 0x7fa1280ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:51.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12c97e700 1 -- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa128196fa0 msgr2=0x7fa1280ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:51.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12c97e700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa128196fa0 0x7fa1280ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:51.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12c97e700 1 -- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1280aece0 msgr2=0x7fa1280ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:51.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12c97e700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1280aece0 0x7fa1280ac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:51.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12c97e700 1 -- 172.21.15.32:0/952062195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12808e9c0 con 0x7fa1280acf30 2021-04-09T18:23:51.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.025 7fa12c97e700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1280acf30 0x7fa1280aae20 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7fa11800b5b0 tx=0x7fa11800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:51.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.026 7fa125ffb700 1 -- 172.21.15.32:0/952062195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa11800bd40 con 0x7fa1280acf30 2021-04-09T18:23:51.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.026 7fa125ffb700 1 -- 172.21.15.32:0/952062195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa118006240 con 0x7fa1280acf30 2021-04-09T18:23:51.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.026 7fa12f363700 1 -- 172.21.15.32:0/952062195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12808e9c0 con 0x7fa1280acf30 2021-04-09T18:23:51.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.026 7fa125ffb700 1 -- 172.21.15.32:0/952062195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa118005690 con 0x7fa1280acf30 2021-04-09T18:23:51.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.026 7fa12f363700 1 -- 172.21.15.32:0/952062195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12808e9c0 con 0x7fa1280acf30 2021-04-09T18:23:51.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.026 7fa125ffb700 1 -- 172.21.15.32:0/952062195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa1180054a0 con 0x7fa1280acf30 2021-04-09T18:23:51.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.026 7fa125ffb700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa108018930 0x7fa10801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:51.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.026 7fa125ffb700 1 -- 172.21.15.32:0/952062195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa118032a30 con 0x7fa1280acf30 2021-04-09T18:23:51.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.027 7fa12d17f700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa108018930 0x7fa10801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:51.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.027 7fa12d17f700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa108018930 0x7fa10801ae10 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:51.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.046 7fa12f363700 1 -- 172.21.15.32:0/952062195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12808dab0 con 0x7fa1280acf30 2021-04-09T18:23:51.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.050 7fa125ffb700 1 -- 172.21.15.32:0/952062195 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa118000c60 con 0x7fa1280acf30 2021-04-09T18:23:51.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.244 7fa12f363700 1 -- 172.21.15.32:0/952062195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fa12810d480 con 0x7fa1280acf30 2021-04-09T18:23:51.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.793 7fa125ffb700 1 -- 172.21.15.32:0/952062195 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v198) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa1180054a0 con 0x7fa1280acf30 2021-04-09T18:23:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.809 7fa12f363700 1 -- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa108018930 msgr2=0x7fa10801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.809 7fa12f363700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa108018930 0x7fa10801ae10 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.809 7fa12f363700 1 -- 172.21.15.32:0/952062195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1280acf30 msgr2=0x7fa1280aae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.809 7fa12f363700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1280acf30 0x7fa1280aae20 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7fa11800b5b0 tx=0x7fa11800b950).stop 2021-04-09T18:23:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.810 7fa12f363700 1 -- 172.21.15.32:0/952062195 shutdown_connections 2021-04-09T18:23:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.810 7fa12f363700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa108018930 0x7fa10801ae10 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.810 7fa12f363700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1280acf30 0x7fa1280aae20 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.810 7fa12f363700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa128196fa0 0x7fa1280ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:51.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.810 7fa12f363700 1 --2- 172.21.15.32:0/952062195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1280aece0 0x7fa1280ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:51.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.810 7fa12f363700 1 -- 172.21.15.32:0/952062195 >> 172.21.15.32:0/952062195 conn(0x7fa1280bb410 msgr2=0x7fa12808dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:51.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.810 7fa12f363700 1 -- 172.21.15.32:0/952062195 shutdown_connections 2021-04-09T18:23:51.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:51.810 7fa12f363700 1 -- 172.21.15.32:0/952062195 wait complete. 2021-04-09T18:23:51.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.012 7f2fcc2d1700 1 Processor -- start 2021-04-09T18:23:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.012 7f2fcc2d1700 1 -- start start 2021-04-09T18:23:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fcc2d1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fc40d7ca0 0x7f2fc40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fcc2d1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc410da00 0x7f2fc410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fcc2d1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc410d480 con 0x7f2fc410d010 2021-04-09T18:23:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fcc2d1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc40d9220 con 0x7f2fc40d7ca0 2021-04-09T18:23:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fcc2d1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40d86c0 con 0x7f2fc410da00 2021-04-09T18:23:52.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fc98ec700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fc40d7ca0 0x7f2fc40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fc98ec700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fc40d7ca0 0x7f2fc40d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50702/0 (socket says 172.21.15.32:50702) 2021-04-09T18:23:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fc98ec700 1 -- 172.21.15.32:0/35189599 learned_addr learned my addr 172.21.15.32:0/35189599 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fca8ee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc410da00 0x7f2fc410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:52.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.013 7f2fca0ed700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2fc410d010 0x7f2fc40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58214/0 (socket says 172.21.15.32:58214) 2021-04-09T18:23:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fc98ec700 1 -- 172.21.15.32:0/35189599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc410da00 msgr2=0x7f2fc410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fc98ec700 1 --2- 172.21.15.32:0/35189599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc410da00 0x7f2fc410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fc98ec700 1 -- 172.21.15.32:0/35189599 >> v1:172.21.15.86:6789/0 conn(0x7f2fc410d010 legacy=0x7f2fc40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:52.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fc90eb700 1 -- 172.21.15.32:0/35189599 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531427648 0 0) 0x7f2fc0000c60 con 0x7f2fc410d010 2021-04-09T18:23:52.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fc98ec700 1 -- 172.21.15.32:0/35189599 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40d1f70 con 0x7f2fc40d7ca0 2021-04-09T18:23:52.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fca8ee700 1 --2- 172.21.15.32:0/35189599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc410da00 0x7f2fc410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:52.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fc98ec700 1 --2- 172.21.15.32:0/35189599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc40d7ca0 0x7f2fc40d80a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f2fb4006700 tx=0x7f2fb4009b80).ready entity=mon.0 client_cookie=9fe411bfed04d10d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:52.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fc90eb700 1 -- 172.21.15.32:0/35189599 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fb400c0c0 con 0x7f2fc40d7ca0 2021-04-09T18:23:52.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fc90eb700 1 -- 172.21.15.32:0/35189599 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb400c440 con 0x7f2fc40d7ca0 2021-04-09T18:23:52.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fc90eb700 1 -- 172.21.15.32:0/35189599 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fb400c920 con 0x7f2fc40d7ca0 2021-04-09T18:23:52.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fcc2d1700 1 -- 172.21.15.32:0/35189599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc40d7ca0 msgr2=0x7f2fc40d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:52.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.014 7f2fcc2d1700 1 --2- 172.21.15.32:0/35189599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc40d7ca0 0x7f2fc40d80a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f2fb4006700 tx=0x7f2fb4009b80).stop 2021-04-09T18:23:52.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.015 7f2fcc2d1700 1 -- 172.21.15.32:0/35189599 shutdown_connections 2021-04-09T18:23:52.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.015 7f2fcc2d1700 1 --2- 172.21.15.32:0/35189599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc410da00 0x7f2fc410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.015 7f2fcc2d1700 1 --2- 172.21.15.32:0/35189599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc40d7ca0 0x7f2fc40d80a0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.015 7f2fcc2d1700 1 -- 172.21.15.32:0/35189599 >> 172.21.15.32:0/35189599 conn(0x7f2fc40ce5f0 msgr2=0x7f2fc40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:52.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.015 7f2fcc2d1700 1 -- 172.21.15.32:0/35189599 shutdown_connections 2021-04-09T18:23:52.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.015 7f2fcc2d1700 1 -- 172.21.15.32:0/35189599 wait complete. 2021-04-09T18:23:52.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.035 7f2fcc2d1700 1 Processor -- start 2021-04-09T18:23:52.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.035 7f2fcc2d1700 1 -- start start 2021-04-09T18:23:52.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fcc2d1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40a0a10 0x7f2fc40bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:52.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fcc2d1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc418ad00 0x7f2fc419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:52.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fcc2d1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc41a03f0 0x7f2fc4095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:52.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fcc2d1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc418b100 con 0x7f2fc41a03f0 2021-04-09T18:23:52.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fcc2d1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc4095860 con 0x7f2fc40a0a10 2021-04-09T18:23:52.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fcc2d1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc4095a80 con 0x7f2fc418ad00 2021-04-09T18:23:52.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca8ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc41a03f0 0x7f2fc4095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:52.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca8ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc41a03f0 0x7f2fc4095240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50708/0 (socket says 172.21.15.32:50708) 2021-04-09T18:23:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca8ee700 1 -- 172.21.15.32:0/1576802330 learned_addr learned my addr 172.21.15.32:0/1576802330 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca0ed700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40a0a10 0x7f2fc40bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fc98ec700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc418ad00 0x7f2fc419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca8ee700 1 -- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc418ad00 msgr2=0x7f2fc419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:52.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca8ee700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc418ad00 0x7f2fc419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca8ee700 1 -- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40a0a10 msgr2=0x7f2fc40bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:52.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca8ee700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40a0a10 0x7f2fc40bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca8ee700 1 -- 172.21.15.32:0/1576802330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4189250 con 0x7f2fc41a03f0 2021-04-09T18:23:52.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fca8ee700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc41a03f0 0x7f2fc4095240 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0x7f2fbc00b970 tx=0x7f2fbc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:52.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fbaffd700 1 -- 172.21.15.32:0/1576802330 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fbc000f90 con 0x7f2fc41a03f0 2021-04-09T18:23:52.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fbaffd700 1 -- 172.21.15.32:0/1576802330 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc006aa0 con 0x7f2fc41a03f0 2021-04-09T18:23:52.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fbaffd700 1 -- 172.21.15.32:0/1576802330 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fbc0015f0 con 0x7f2fc41a03f0 2021-04-09T18:23:52.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc4189250 con 0x7f2fc41a03f0 2021-04-09T18:23:52.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.036 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc4189250 con 0x7f2fc41a03f0 2021-04-09T18:23:52.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.037 7f2fbaffd700 1 -- 172.21.15.32:0/1576802330 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2fbc000e70 con 0x7f2fc41a03f0 2021-04-09T18:23:52.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.037 7f2fbaffd700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fac0188c0 0x7f2fac01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:52.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.037 7f2fbaffd700 1 -- 172.21.15.32:0/1576802330 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2fbc032b40 con 0x7f2fc41a03f0 2021-04-09T18:23:52.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.037 7f2fca0ed700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fac0188c0 0x7f2fac01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:52.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.038 7f2fca0ed700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fac0188c0 0x7f2fac01adc0 crc :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:52.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.057 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc4093d50 con 0x7f2fc41a03f0 2021-04-09T18:23:52.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.061 7f2fbaffd700 1 -- 172.21.15.32:0/1576802330 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2fbc0066f0 con 0x7f2fc41a03f0 2021-04-09T18:23:52.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.256 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f2fc410d480 con 0x7f2fc41a03f0 2021-04-09T18:23:52.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.807 7f2fbaffd700 1 -- 172.21.15.32:0/1576802330 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v200) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2fbc000e70 con 0x7f2fc41a03f0 2021-04-09T18:23:52.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fac0188c0 msgr2=0x7f2fac01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:52.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fac0188c0 0x7f2fac01adc0 crc :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc41a03f0 msgr2=0x7f2fc4095240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:52.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc41a03f0 0x7f2fc4095240 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0x7f2fbc00b970 tx=0x7f2fbc00bd10).stop 2021-04-09T18:23:52.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 shutdown_connections 2021-04-09T18:23:52.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fac0188c0 0x7f2fac01adc0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fc41a03f0 0x7f2fc4095240 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc418ad00 0x7f2fc419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 --2- 172.21.15.32:0/1576802330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40a0a10 0x7f2fc40bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:52.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.823 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 >> 172.21.15.32:0/1576802330 conn(0x7f2fc40975f0 msgr2=0x7f2fc4096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:52.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.824 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 shutdown_connections 2021-04-09T18:23:52.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.824 7f2fcc2d1700 1 -- 172.21.15.32:0/1576802330 wait complete. 2021-04-09T18:23:52.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.992 7f1c2eae9700 1 Processor -- start 2021-04-09T18:23:53.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.992 7f1c2eae9700 1 -- start start 2021-04-09T18:23:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2eae9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c2810d010 0x7f1c280d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2eae9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280d1da0 0x7f1c280d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2eae9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1c280d82c0 0x7f1c2810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:53.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2eae9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c280d8d30 con 0x7f1c280d82c0 2021-04-09T18:23:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2eae9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c280d9220 con 0x7f1c2810d010 2021-04-09T18:23:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2eae9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2810d480 con 0x7f1c280d1da0 2021-04-09T18:23:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2d106700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1c280d82c0 0x7f1c2810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:53.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2d106700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1c280d82c0 0x7f1c2810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50722/0 (socket says 172.21.15.32:50722) 2021-04-09T18:23:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2d106700 1 -- 172.21.15.32:0/3269266070 learned_addr learned my addr 172.21.15.32:0/3269266070 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c2c905700 1 --2- 172.21.15.32:0/3269266070 >> v2:172.21.15.86:3300/0 conn(0x7f1c2810d010 0x7f1c280d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.993 7f1c27fff700 1 --2- 172.21.15.32:0/3269266070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280d1da0 0x7f1c280d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c2d106700 1 -- 172.21.15.32:0/3269266070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280d1da0 msgr2=0x7f1c280d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:53.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c2d106700 1 --2- 172.21.15.32:0/3269266070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280d1da0 0x7f1c280d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c2d106700 1 -- 172.21.15.32:0/3269266070 >> v2:172.21.15.86:3300/0 conn(0x7f1c2810d010 msgr2=0x7f1c280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:53.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c2d106700 1 --2- 172.21.15.32:0/3269266070 >> v2:172.21.15.86:3300/0 conn(0x7f1c2810d010 0x7f1c280d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c2d106700 1 -- 172.21.15.32:0/3269266070 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280d86c0 con 0x7f1c280d82c0 2021-04-09T18:23:53.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c2c905700 1 --2- 172.21.15.32:0/3269266070 >> v2:172.21.15.86:3300/0 conn(0x7f1c2810d010 0x7f1c280d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:53.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c2d106700 1 --2- 172.21.15.32:0/3269266070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c280d82c0 0x7f1c2810da00 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f1c20006b80 tx=0x7f1c20009b10).ready entity=mon.0 client_cookie=54caaf75ba6177af server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:53.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c277fe700 1 -- 172.21.15.32:0/3269266070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c200022e0 con 0x7f1c280d82c0 2021-04-09T18:23:53.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c277fe700 1 -- 172.21.15.32:0/3269266070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c20002620 con 0x7f1c280d82c0 2021-04-09T18:23:53.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c277fe700 1 -- 172.21.15.32:0/3269266070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c20002b00 con 0x7f1c280d82c0 2021-04-09T18:23:53.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c2eae9700 1 -- 172.21.15.32:0/3269266070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c280d82c0 msgr2=0x7f1c2810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:53.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.994 7f1c2eae9700 1 --2- 172.21.15.32:0/3269266070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c280d82c0 0x7f1c2810da00 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f1c20006b80 tx=0x7f1c20009b10).stop 2021-04-09T18:23:53.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.995 7f1c2eae9700 1 -- 172.21.15.32:0/3269266070 shutdown_connections 2021-04-09T18:23:53.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.995 7f1c2eae9700 1 --2- 172.21.15.32:0/3269266070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c280d82c0 0x7f1c2810da00 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.995 7f1c2eae9700 1 --2- 172.21.15.32:0/3269266070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280d1da0 0x7f1c280d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.995 7f1c2eae9700 1 --2- 172.21.15.32:0/3269266070 >> v2:172.21.15.86:3300/0 conn(0x7f1c2810d010 0x7f1c280d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.995 7f1c2eae9700 1 -- 172.21.15.32:0/3269266070 >> 172.21.15.32:0/3269266070 conn(0x7f1c280ce5f0 msgr2=0x7f1c280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:53.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.995 7f1c2eae9700 1 -- 172.21.15.32:0/3269266070 shutdown_connections 2021-04-09T18:23:53.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:52.995 7f1c2eae9700 1 -- 172.21.15.32:0/3269266070 wait complete. 2021-04-09T18:23:53.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.015 7f1c2eae9700 1 Processor -- start 2021-04-09T18:23:53.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.015 7f1c2eae9700 1 -- start start 2021-04-09T18:23:53.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2eae9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c281a0050 0x7f1c281a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:53.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2eae9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c280bdcd0 0x7f1c280bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:53.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2eae9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280bf080 0x7f1c280b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:53.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2eae9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280bf500 con 0x7f1c281a0050 2021-04-09T18:23:53.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2eae9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280bb7b0 con 0x7f1c280bdcd0 2021-04-09T18:23:53.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2eae9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280bb960 con 0x7f1c280bf080 2021-04-09T18:23:53.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2c905700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c281a0050 0x7f1c281a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:53.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2c905700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c281a0050 0x7f1c281a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50724/0 (socket says 172.21.15.32:50724) 2021-04-09T18:23:53.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2c905700 1 -- 172.21.15.32:0/3928950656 learned_addr learned my addr 172.21.15.32:0/3928950656 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:53.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2d106700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280bf080 0x7f1c280b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:53.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2c905700 1 -- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280bf080 msgr2=0x7f1c280b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:53.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2c905700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280bf080 0x7f1c280b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c27fff700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c280bdcd0 0x7f1c280bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:53.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2c905700 1 -- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c280bdcd0 msgr2=0x7f1c280bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:53.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2c905700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c280bdcd0 0x7f1c280bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2c905700 1 -- 172.21.15.32:0/3928950656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280be0d0 con 0x7f1c281a0050 2021-04-09T18:23:53.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2c905700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c281a0050 0x7f1c281a0450 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f1c1800b910 tx=0x7f1c1800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:53.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c257fa700 1 -- 172.21.15.32:0/3928950656 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c1800c2f0 con 0x7f1c281a0050 2021-04-09T18:23:53.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c257fa700 1 -- 172.21.15.32:0/3928950656 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1800c5f0 con 0x7f1c281a0050 2021-04-09T18:23:53.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c280be0d0 con 0x7f1c281a0050 2021-04-09T18:23:53.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.016 7f1c257fa700 1 -- 172.21.15.32:0/3928950656 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c1800cad0 con 0x7f1c281a0050 2021-04-09T18:23:53.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.017 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c280be0d0 con 0x7f1c281a0050 2021-04-09T18:23:53.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.017 7f1c257fa700 1 -- 172.21.15.32:0/3928950656 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1c18001010 con 0x7f1c281a0050 2021-04-09T18:23:53.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.017 7f1c257fa700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c0c018930 0x7f1c0c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:53.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.017 7f1c257fa700 1 -- 172.21.15.32:0/3928950656 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1c18032d90 con 0x7f1c281a0050 2021-04-09T18:23:53.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.017 7f1c27fff700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c0c018930 0x7f1c0c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:53.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.018 7f1c27fff700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c0c018930 0x7f1c0c01ae10 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:53.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.037 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c28093d50 con 0x7f1c281a0050 2021-04-09T18:23:53.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.041 7f1c257fa700 1 -- 172.21.15.32:0/3928950656 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1c1800bdf0 con 0x7f1c281a0050 2021-04-09T18:23:53.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.234 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f1c28093d50 con 0x7f1c281a0050 2021-04-09T18:23:53.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.807 7f1c257fa700 1 -- 172.21.15.32:0/3928950656 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v201) v1 ==== 105+0+0 (secure 0 0 0) 0x7f1c18001010 con 0x7f1c281a0050 2021-04-09T18:23:53.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c0c018930 msgr2=0x7f1c0c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:53.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c0c018930 0x7f1c0c01ae10 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c281a0050 msgr2=0x7f1c281a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:53.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c281a0050 0x7f1c281a0450 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f1c1800b910 tx=0x7f1c1800bcb0).stop 2021-04-09T18:23:53.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 shutdown_connections 2021-04-09T18:23:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c0c018930 0x7f1c0c01ae10 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c280bf080 0x7f1c280b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c280bdcd0 0x7f1c280bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 --2- 172.21.15.32:0/3928950656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c281a0050 0x7f1c281a0450 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 >> 172.21.15.32:0/3928950656 conn(0x7f1c280975f0 msgr2=0x7f1c28096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 shutdown_connections 2021-04-09T18:23:53.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:53.852 7f1c2eae9700 1 -- 172.21.15.32:0/3928950656 wait complete. 2021-04-09T18:23:53.922 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:54.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.014 7f152757a700 1 Processor -- start 2021-04-09T18:23:54.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.014 7f152757a700 1 -- start start 2021-04-09T18:23:54.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f152757a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f152010d010 0x7f15200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:54.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f152757a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f15200d1da0 0x7f15200d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:54.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f152757a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152010d480 con 0x7f15200d82c0 2021-04-09T18:23:54.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f152757a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15200d8d30 con 0x7f152010d010 2021-04-09T18:23:54.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f152757a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15200d9220 con 0x7f15200d1da0 2021-04-09T18:23:54.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f1525396700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f152010d010 0x7f15200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:54.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f1525396700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f152010d010 0x7f15200d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50732/0 (socket says 172.21.15.32:50732) 2021-04-09T18:23:54.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f1525396700 1 -- 172.21.15.32:0/3901054838 learned_addr learned my addr 172.21.15.32:0/3901054838 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:54.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f1524b95700 1 --2- 172.21.15.32:0/3901054838 >> v2:172.21.15.86:3300/0 conn(0x7f15200d1da0 0x7f15200d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:54.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f1525396700 1 -- 172.21.15.32:0/3901054838 >> v2:172.21.15.86:3300/0 conn(0x7f15200d1da0 msgr2=0x7f15200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:54.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f1525396700 1 --2- 172.21.15.32:0/3901054838 >> v2:172.21.15.86:3300/0 conn(0x7f15200d1da0 0x7f15200d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f1525396700 1 -- 172.21.15.32:0/3901054838 >> v1:172.21.15.86:6789/0 conn(0x7f15200d82c0 legacy=0x7f152010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:54.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.015 7f1525396700 1 -- 172.21.15.32:0/3901054838 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15200d86f0 con 0x7f152010d010 2021-04-09T18:23:54.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f1525396700 1 --2- 172.21.15.32:0/3901054838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f152010d010 0x7f15200d1780 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f151c006700 tx=0x7f151c009b80).ready entity=mon.0 client_cookie=2b5dd28f0bdeba1e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:54.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f1517fff700 1 -- 172.21.15.32:0/3901054838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f151c00c0c0 con 0x7f152010d010 2021-04-09T18:23:54.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f1517fff700 1 -- 172.21.15.32:0/3901054838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151c00c440 con 0x7f152010d010 2021-04-09T18:23:54.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f1517fff700 1 -- 172.21.15.32:0/3901054838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f151c00c920 con 0x7f152010d010 2021-04-09T18:23:54.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f152757a700 1 -- 172.21.15.32:0/3901054838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f152010d010 msgr2=0x7f15200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:54.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f152757a700 1 --2- 172.21.15.32:0/3901054838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f152010d010 0x7f15200d1780 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f151c006700 tx=0x7f151c009b80).stop 2021-04-09T18:23:54.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f152757a700 1 -- 172.21.15.32:0/3901054838 shutdown_connections 2021-04-09T18:23:54.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f152757a700 1 --2- 172.21.15.32:0/3901054838 >> v2:172.21.15.86:3300/0 conn(0x7f15200d1da0 0x7f15200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f152757a700 1 --2- 172.21.15.32:0/3901054838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f152010d010 0x7f15200d1780 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.016 7f152757a700 1 -- 172.21.15.32:0/3901054838 >> 172.21.15.32:0/3901054838 conn(0x7f15200ce5f0 msgr2=0x7f15200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:54.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.017 7f152757a700 1 -- 172.21.15.32:0/3901054838 shutdown_connections 2021-04-09T18:23:54.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.017 7f152757a700 1 -- 172.21.15.32:0/3901054838 wait complete. 2021-04-09T18:23:54.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.037 7f152757a700 1 Processor -- start 2021-04-09T18:23:54.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.037 7f152757a700 1 -- start start 2021-04-09T18:23:54.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f152757a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15200c3c00 0x7f152009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:54.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f152757a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1520095260 0x7f15201a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:54.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f152757a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15201a1270 0x7f15200b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:54.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f152757a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152009cda0 con 0x7f15200c3c00 2021-04-09T18:23:54.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f152757a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15200c3680 con 0x7f1520095260 2021-04-09T18:23:54.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f152757a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1520095660 con 0x7f15201a1270 2021-04-09T18:23:54.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525396700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15200c3c00 0x7f152009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:54.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525396700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15200c3c00 0x7f152009c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50738/0 (socket says 172.21.15.32:50738) 2021-04-09T18:23:54.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525396700 1 -- 172.21.15.32:0/3524945144 learned_addr learned my addr 172.21.15.32:0/3524945144 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:54.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1524b95700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1520095260 0x7f15201a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:54.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525b97700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15201a1270 0x7f15200b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:54.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525396700 1 -- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15201a1270 msgr2=0x7f15200b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:54.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525396700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15201a1270 0x7f15200b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525396700 1 -- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1520095260 msgr2=0x7f15201a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:54.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525396700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1520095260 0x7f15201a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525396700 1 -- 172.21.15.32:0/3524945144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15201a1670 con 0x7f15200c3c00 2021-04-09T18:23:54.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1525396700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15200c3c00 0x7f152009c600 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f151c00b5e0 tx=0x7f151c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:54.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1515ffb700 1 -- 172.21.15.32:0/3524945144 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f151c00bd70 con 0x7f15200c3c00 2021-04-09T18:23:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1515ffb700 1 -- 172.21.15.32:0/3524945144 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151c002030 con 0x7f15200c3c00 2021-04-09T18:23:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f152757a700 1 -- 172.21.15.32:0/3524945144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15201a1670 con 0x7f15200c3c00 2021-04-09T18:23:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f1515ffb700 1 -- 172.21.15.32:0/3524945144 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f151c002550 con 0x7f15200c3c00 2021-04-09T18:23:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.038 7f152757a700 1 -- 172.21.15.32:0/3524945144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15201a1670 con 0x7f15200c3c00 2021-04-09T18:23:54.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.039 7f1515ffb700 1 -- 172.21.15.32:0/3524945144 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f151c000c90 con 0x7f15200c3c00 2021-04-09T18:23:54.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.039 7f1515ffb700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1500018930 0x7f150001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:54.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.039 7f1515ffb700 1 -- 172.21.15.32:0/3524945144 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f151c032a60 con 0x7f15200c3c00 2021-04-09T18:23:54.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.039 7f1524b95700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1500018930 0x7f150001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:54.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.040 7f1524b95700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1500018930 0x7f150001adc0 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:54.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.059 7f152757a700 1 -- 172.21.15.32:0/3524945144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152010d480 con 0x7f15200c3c00 2021-04-09T18:23:54.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.062 7f1515ffb700 1 -- 172.21.15.32:0/3524945144 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f151c01f7a0 con 0x7f15200c3c00 2021-04-09T18:23:54.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.255 7f152757a700 1 -- 172.21.15.32:0/3524945144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f152010d480 con 0x7f15200c3c00 2021-04-09T18:23:54.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.817 7f1515ffb700 1 -- 172.21.15.32:0/3524945144 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v203) v1 ==== 105+0+0 (secure 0 0 0) 0x7f151c01f7a0 con 0x7f15200c3c00 2021-04-09T18:23:54.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.821 7f152757a700 1 -- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1500018930 msgr2=0x7f150001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:54.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.821 7f152757a700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1500018930 0x7f150001adc0 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 -- 172.21.15.32:0/3524945144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15200c3c00 msgr2=0x7f152009c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:54.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15200c3c00 0x7f152009c600 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f151c00b5e0 tx=0x7f151c00b980).stop 2021-04-09T18:23:54.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 -- 172.21.15.32:0/3524945144 shutdown_connections 2021-04-09T18:23:54.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1500018930 0x7f150001adc0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15201a1270 0x7f15200b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1520095260 0x7f15201a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 --2- 172.21.15.32:0/3524945144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15200c3c00 0x7f152009c600 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:54.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 -- 172.21.15.32:0/3524945144 >> 172.21.15.32:0/3524945144 conn(0x7f15200975f0 msgr2=0x7f1520096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:54.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 -- 172.21.15.32:0/3524945144 shutdown_connections 2021-04-09T18:23:54.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.822 7f152757a700 1 -- 172.21.15.32:0/3524945144 wait complete. 2021-04-09T18:23:54.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:55.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.987 7fcea1dae700 1 Processor -- start 2021-04-09T18:23:55.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.987 7fcea1dae700 1 -- start start 2021-04-09T18:23:55.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fcea1dae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c0d7ca0 0x7fce9c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fcea1dae700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fce9c10da00 0x7fce9c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fcea1dae700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce9c10d480 con 0x7fce9c10d010 2021-04-09T18:23:55.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fcea1dae700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce9c0d9220 con 0x7fce9c10da00 2021-04-09T18:23:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fcea1dae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c0d86c0 con 0x7fce9c0d7ca0 2021-04-09T18:23:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9b7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fce9c10d010 0x7fce9c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60336/0 (socket says 172.21.15.32:60336) 2021-04-09T18:23:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9b7fe700 1 -- 172.21.15.32:0/368904872 learned_addr learned my addr 172.21.15.32:0/368904872 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:55.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9bfff700 1 --2- 172.21.15.32:0/368904872 >> v2:172.21.15.86:3300/0 conn(0x7fce9c10da00 0x7fce9c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9affd700 1 --2- 172.21.15.32:0/368904872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c0d7ca0 0x7fce9c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9a7fc700 1 -- 172.21.15.32:0/368904872 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989264860 0 0) 0x7fce8c000ce0 con 0x7fce9c10d010 2021-04-09T18:23:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9a7fc700 1 -- 172.21.15.32:0/368904872 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce84002fe0 con 0x7fce9c10d010 2021-04-09T18:23:55.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9bfff700 1 -- 172.21.15.32:0/368904872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c0d7ca0 msgr2=0x7fce9c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9bfff700 1 --2- 172.21.15.32:0/368904872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c0d7ca0 0x7fce9c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9bfff700 1 -- 172.21.15.32:0/368904872 >> v1:172.21.15.32:6789/0 conn(0x7fce9c10d010 legacy=0x7fce9c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9bfff700 1 -- 172.21.15.32:0/368904872 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c0d1f70 con 0x7fce9c10da00 2021-04-09T18:23:55.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.988 7fce9affd700 1 --2- 172.21.15.32:0/368904872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c0d7ca0 0x7fce9c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.989 7fce9bfff700 1 --2- 172.21.15.32:0/368904872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9c10da00 0x7fce9c10de00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7fce94006e50 tx=0x7fce94009e70).ready entity=mon.1 client_cookie=578ff3f6f9754ad0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.989 7fce9a7fc700 1 -- 172.21.15.32:0/368904872 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce94006a20 con 0x7fce9c10da00 2021-04-09T18:23:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.989 7fce9a7fc700 1 -- 172.21.15.32:0/368904872 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce94006790 con 0x7fce9c10da00 2021-04-09T18:23:55.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.989 7fce9a7fc700 1 -- 172.21.15.32:0/368904872 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce9400b700 con 0x7fce9c10da00 2021-04-09T18:23:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.989 7fcea1dae700 1 -- 172.21.15.32:0/368904872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9c10da00 msgr2=0x7fce9c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.989 7fcea1dae700 1 --2- 172.21.15.32:0/368904872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9c10da00 0x7fce9c10de00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7fce94006e50 tx=0x7fce94009e70).stop 2021-04-09T18:23:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.989 7fcea1dae700 1 -- 172.21.15.32:0/368904872 shutdown_connections 2021-04-09T18:23:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.989 7fcea1dae700 1 --2- 172.21.15.32:0/368904872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9c10da00 0x7fce9c10de00 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.989 7fcea1dae700 1 --2- 172.21.15.32:0/368904872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c0d7ca0 0x7fce9c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.990 7fcea1dae700 1 -- 172.21.15.32:0/368904872 >> 172.21.15.32:0/368904872 conn(0x7fce9c0ce5f0 msgr2=0x7fce9c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:55.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.990 7fcea1dae700 1 -- 172.21.15.32:0/368904872 shutdown_connections 2021-04-09T18:23:55.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:54.990 7fcea1dae700 1 -- 172.21.15.32:0/368904872 wait complete. 2021-04-09T18:23:55.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.010 7fcea1dae700 1 Processor -- start 2021-04-09T18:23:55.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.010 7fcea1dae700 1 -- start start 2021-04-09T18:23:55.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.010 7fcea1dae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9c08a910 0x7fce9c09e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.010 7fcea1dae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c08ad10 0x7fce9c09ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.010 7fcea1dae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce9c0922e0 0x7fce9c0926e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.010 7fcea1dae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c18fc90 con 0x7fce9c0922e0 2021-04-09T18:23:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.010 7fcea1dae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c09e090 con 0x7fce9c08a910 2021-04-09T18:23:55.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.010 7fcea1dae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c08b400 con 0x7fce9c08ad10 2021-04-09T18:23:55.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce9c0922e0 0x7fce9c0926e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:55.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce9c0922e0 0x7fce9c0926e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50756/0 (socket says 172.21.15.32:50756) 2021-04-09T18:23:55.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9bfff700 1 -- 172.21.15.32:0/2433626320 learned_addr learned my addr 172.21.15.32:0/2433626320 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:55.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9b7fe700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9c08a910 0x7fce9c09e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:55.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9affd700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c08ad10 0x7fce9c09ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:55.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9bfff700 1 -- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c08ad10 msgr2=0x7fce9c09ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:55.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9bfff700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c08ad10 0x7fce9c09ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9bfff700 1 -- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9c08a910 msgr2=0x7fce9c09e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:55.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9bfff700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9c08a910 0x7fce9c09e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9bfff700 1 -- 172.21.15.32:0/2433626320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c092d00 con 0x7fce9c0922e0 2021-04-09T18:23:55.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce9bfff700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce9c0922e0 0x7fce9c0926e0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7fce94006e60 tx=0x7fce9400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:55.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce83fff700 1 -- 172.21.15.32:0/2433626320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce94009130 con 0x7fce9c0922e0 2021-04-09T18:23:55.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce83fff700 1 -- 172.21.15.32:0/2433626320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce94009430 con 0x7fce9c0922e0 2021-04-09T18:23:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fce83fff700 1 -- 172.21.15.32:0/2433626320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce94001390 con 0x7fce9c0922e0 2021-04-09T18:23:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce9c092d00 con 0x7fce9c0922e0 2021-04-09T18:23:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.011 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce9c092d00 con 0x7fce9c0922e0 2021-04-09T18:23:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.012 7fce83fff700 1 -- 172.21.15.32:0/2433626320 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fce94009130 con 0x7fce9c0922e0 2021-04-09T18:23:55.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.012 7fce83fff700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce7c0188d0 0x7fce7c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.012 7fce83fff700 1 -- 172.21.15.32:0/2433626320 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fce94032d50 con 0x7fce9c0922e0 2021-04-09T18:23:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.012 7fce9b7fe700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce7c0188d0 0x7fce7c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:55.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.013 7fce9b7fe700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce7c0188d0 0x7fce7c01add0 crc :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:55.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.032 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9c18fc90 con 0x7fce9c0922e0 2021-04-09T18:23:55.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.035 7fce83fff700 1 -- 172.21.15.32:0/2433626320 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fce94009430 con 0x7fce9c0922e0 2021-04-09T18:23:55.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.228 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fce9c18fc90 con 0x7fce9c0922e0 2021-04-09T18:23:55.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.821 7fce83fff700 1 -- 172.21.15.32:0/2433626320 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v204) v1 ==== 105+0+0 (secure 0 0 0) 0x7fce94009430 con 0x7fce9c0922e0 2021-04-09T18:23:55.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce7c0188d0 msgr2=0x7fce7c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:55.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce7c0188d0 0x7fce7c01add0 crc :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce9c0922e0 msgr2=0x7fce9c0926e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:55.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce9c0922e0 0x7fce9c0926e0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7fce94006e60 tx=0x7fce9400bcf0).stop 2021-04-09T18:23:55.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 shutdown_connections 2021-04-09T18:23:55.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce7c0188d0 0x7fce7c01add0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce9c0922e0 0x7fce9c0926e0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9c08ad10 0x7fce9c09ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 --2- 172.21.15.32:0/2433626320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9c08a910 0x7fce9c09e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:55.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 >> 172.21.15.32:0/2433626320 conn(0x7fce9c0b2fa0 msgr2=0x7fce9c0a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:55.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 shutdown_connections 2021-04-09T18:23:55.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:55.845 7fcea1dae700 1 -- 172.21.15.32:0/2433626320 wait complete. 2021-04-09T18:23:55.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.008 7f911c1f6700 1 Processor -- start 2021-04-09T18:23:56.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.008 7f911c1f6700 1 -- start start 2021-04-09T18:23:56.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911c1f6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f911410d010 0x7f91140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:56.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911c1f6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f91140d1da0 0x7f91140d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:56.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911c1f6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911410d480 con 0x7f91140d82c0 2021-04-09T18:23:56.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911c1f6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f911410e780 con 0x7f91140d1da0 2021-04-09T18:23:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911c1f6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91140d8d30 con 0x7f911410d010 2021-04-09T18:23:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f9119811700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f91140d1da0 0x7f91140d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:56.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911a012700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f911410d010 0x7f91140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:56.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f9119811700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f91140d1da0 0x7f91140d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:42522/0 (socket says 172.21.15.32:42522) 2021-04-09T18:23:56.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f9119811700 1 -- 172.21.15.32:0/419068914 learned_addr learned my addr 172.21.15.32:0/419068914 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:56.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911a813700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f91140d82c0 0x7f911410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58280/0 (socket says 172.21.15.32:58280) 2021-04-09T18:23:56.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911a012700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f911410d010 0x7f91140d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:57988/0 (socket says 172.21.15.32:57988) 2021-04-09T18:23:56.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911a012700 1 -- 172.21.15.32:0/419068914 >> v2:172.21.15.86:3300/0 conn(0x7f91140d1da0 msgr2=0x7f91140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:56.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.009 7f911a012700 1 --2- 172.21.15.32:0/419068914 >> v2:172.21.15.86:3300/0 conn(0x7f91140d1da0 0x7f91140d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f911a012700 1 -- 172.21.15.32:0/419068914 >> v1:172.21.15.86:6789/0 conn(0x7f91140d82c0 legacy=0x7f911410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:56.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f9119010700 1 -- 172.21.15.32:0/419068914 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341738130 0 0) 0x7f910c000d30 con 0x7f91140d82c0 2021-04-09T18:23:56.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f911a012700 1 -- 172.21.15.32:0/419068914 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91140d9220 con 0x7f911410d010 2021-04-09T18:23:56.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f9119811700 1 --2- 172.21.15.32:0/419068914 >> v2:172.21.15.86:3300/0 conn(0x7f91140d1da0 0x7f91140d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:56.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f911a012700 1 --2- 172.21.15.32:0/419068914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f911410d010 0x7f91140d1780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9110006650 tx=0x7f9110009a80).ready entity=mon.2 client_cookie=c9504893465b927a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:56.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f9119010700 1 -- 172.21.15.32:0/419068914 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91100009b0 con 0x7f911410d010 2021-04-09T18:23:56.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f9119010700 1 -- 172.21.15.32:0/419068914 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911000c0c0 con 0x7f911410d010 2021-04-09T18:23:56.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f9119010700 1 -- 172.21.15.32:0/419068914 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f911000c510 con 0x7f911410d010 2021-04-09T18:23:56.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f911c1f6700 1 -- 172.21.15.32:0/419068914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f911410d010 msgr2=0x7f91140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:56.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.010 7f911c1f6700 1 --2- 172.21.15.32:0/419068914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f911410d010 0x7f91140d1780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9110006650 tx=0x7f9110009a80).stop 2021-04-09T18:23:56.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.011 7f911c1f6700 1 -- 172.21.15.32:0/419068914 shutdown_connections 2021-04-09T18:23:56.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.011 7f911c1f6700 1 --2- 172.21.15.32:0/419068914 >> v2:172.21.15.86:3300/0 conn(0x7f91140d1da0 0x7f91140d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.011 7f911c1f6700 1 --2- 172.21.15.32:0/419068914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f911410d010 0x7f91140d1780 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.011 7f911c1f6700 1 -- 172.21.15.32:0/419068914 >> 172.21.15.32:0/419068914 conn(0x7f91140ce5f0 msgr2=0x7f91140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:56.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.011 7f911c1f6700 1 -- 172.21.15.32:0/419068914 shutdown_connections 2021-04-09T18:23:56.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.011 7f911c1f6700 1 -- 172.21.15.32:0/419068914 wait complete. 2021-04-09T18:23:56.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.031 7f911c1f6700 1 Processor -- start 2021-04-09T18:23:56.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.031 7f911c1f6700 1 -- start start 2021-04-09T18:23:56.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911c1f6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91140b2140 0x7f911408b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:56.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911c1f6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f911409f550 0x7f911409f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:56.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911c1f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9114197c50 0x7f9114198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:56.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911c1f6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911409f010 con 0x7f9114197c50 2021-04-09T18:23:56.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911c1f6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911409ff70 con 0x7f911409f550 2021-04-09T18:23:56.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911c1f6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9114198670 con 0x7f91140b2140 2021-04-09T18:23:56.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a813700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9114197c50 0x7f9114198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:56.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a813700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9114197c50 0x7f9114198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50772/0 (socket says 172.21.15.32:50772) 2021-04-09T18:23:56.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a813700 1 -- 172.21.15.32:0/587727689 learned_addr learned my addr 172.21.15.32:0/587727689 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:56.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a012700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91140b2140 0x7f911408b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:56.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f9119811700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f911409f550 0x7f911409f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:56.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a813700 1 -- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91140b2140 msgr2=0x7f911408b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:56.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a813700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91140b2140 0x7f911408b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a813700 1 -- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f911409f550 msgr2=0x7f911409f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:56.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a813700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f911409f550 0x7f911409f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a813700 1 -- 172.21.15.32:0/587727689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9114198890 con 0x7f9114197c50 2021-04-09T18:23:56.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f911a813700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9114197c50 0x7f9114198050 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7f910c001b70 tx=0x7f910c00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:56.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f910affd700 1 -- 172.21.15.32:0/587727689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f910c0011f0 con 0x7f9114197c50 2021-04-09T18:23:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.032 7f910affd700 1 -- 172.21.15.32:0/587727689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c007150 con 0x7f9114197c50 2021-04-09T18:23:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.033 7f911c1f6700 1 -- 172.21.15.32:0/587727689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9114198890 con 0x7f9114197c50 2021-04-09T18:23:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.033 7f910affd700 1 -- 172.21.15.32:0/587727689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f910c007520 con 0x7f9114197c50 2021-04-09T18:23:56.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.033 7f911c1f6700 1 -- 172.21.15.32:0/587727689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9114198890 con 0x7f9114197c50 2021-04-09T18:23:56.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.033 7f910affd700 1 -- 172.21.15.32:0/587727689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f910c007330 con 0x7f9114197c50 2021-04-09T18:23:56.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.033 7f910affd700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90fc018930 0x7f90fc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:56.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.033 7f910affd700 1 -- 172.21.15.32:0/587727689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f910c032db0 con 0x7f9114197c50 2021-04-09T18:23:56.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.033 7f911a012700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90fc018930 0x7f90fc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:56.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.034 7f911a012700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90fc018930 0x7f90fc01ae10 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:56.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.053 7f911c1f6700 1 -- 172.21.15.32:0/587727689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911409f010 con 0x7f9114197c50 2021-04-09T18:23:56.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.057 7f910affd700 1 -- 172.21.15.32:0/587727689 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f910c000cc0 con 0x7f9114197c50 2021-04-09T18:23:56.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.250 7f911c1f6700 1 -- 172.21.15.32:0/587727689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f911409f010 con 0x7f9114197c50 2021-04-09T18:23:56.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.827 7f910affd700 1 -- 172.21.15.32:0/587727689 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v205) v1 ==== 105+0+0 (secure 0 0 0) 0x7f910c007150 con 0x7f9114197c50 2021-04-09T18:23:56.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 -- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90fc018930 msgr2=0x7f90fc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:56.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90fc018930 0x7f90fc01ae10 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 -- 172.21.15.32:0/587727689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9114197c50 msgr2=0x7f9114198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9114197c50 0x7f9114198050 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7f910c001b70 tx=0x7f910c00ac80).stop 2021-04-09T18:23:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 -- 172.21.15.32:0/587727689 shutdown_connections 2021-04-09T18:23:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90fc018930 0x7f90fc01ae10 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9114197c50 0x7f9114198050 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f911409f550 0x7f911409f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 --2- 172.21.15.32:0/587727689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91140b2140 0x7f911408b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:56.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 -- 172.21.15.32:0/587727689 >> 172.21.15.32:0/587727689 conn(0x7f91140b27e0 msgr2=0x7f91141b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:56.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.867 7f911c1f6700 1 -- 172.21.15.32:0/587727689 shutdown_connections 2021-04-09T18:23:56.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:56.868 7f911c1f6700 1 -- 172.21.15.32:0/587727689 wait complete. 2021-04-09T18:23:56.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:23:57.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.031 7fe3d1070700 1 Processor -- start 2021-04-09T18:23:57.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.031 7fe3d1070700 1 -- start start 2021-04-09T18:23:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3d1070700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc10d010 0x7fe3cc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3d1070700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe3cc0d1da0 0x7fe3cc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3d1070700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc10d480 con 0x7fe3cc0d82c0 2021-04-09T18:23:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3d1070700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc10e780 con 0x7fe3cc0d1da0 2021-04-09T18:23:57.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3d1070700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0d8d30 con 0x7fe3cc10d010 2021-04-09T18:23:57.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3ca59c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe3cc0d1da0 0x7fe3cc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:57.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3cb59e700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe3cc0d82c0 0x7fe3cc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58296/0 (socket says 172.21.15.32:58296) 2021-04-09T18:23:57.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3cb59e700 1 -- 172.21.15.32:0/3585056748 learned_addr learned my addr 172.21.15.32:0/3585056748 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:57.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3cad9d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc10d010 0x7fe3cc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:57.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.032 7fe3c9d9b700 1 -- 172.21.15.32:0/3585056748 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457699811 0 0) 0x7fe3c4001440 con 0x7fe3cc0d82c0 2021-04-09T18:23:57.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3c9d9b700 1 -- 172.21.15.32:0/3585056748 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b4002fe0 con 0x7fe3cc0d82c0 2021-04-09T18:23:57.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3ca59c700 1 -- 172.21.15.32:0/3585056748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc10d010 msgr2=0x7fe3cc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:57.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3ca59c700 1 --2- 172.21.15.32:0/3585056748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc10d010 0x7fe3cc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3ca59c700 1 -- 172.21.15.32:0/3585056748 >> v1:172.21.15.86:6789/0 conn(0x7fe3cc0d82c0 legacy=0x7fe3cc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:57.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3ca59c700 1 -- 172.21.15.32:0/3585056748 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc0d9220 con 0x7fe3cc0d1da0 2021-04-09T18:23:57.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3cad9d700 1 --2- 172.21.15.32:0/3585056748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc10d010 0x7fe3cc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:23:57.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3ca59c700 1 --2- 172.21.15.32:0/3585056748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3cc0d1da0 0x7fe3cc0d7ca0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7fe3c0006650 tx=0x7fe3c0009a80).ready entity=mon.1 client_cookie=a4a707a11133d79 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:57.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3c9d9b700 1 -- 172.21.15.32:0/3585056748 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3c00009b0 con 0x7fe3cc0d1da0 2021-04-09T18:23:57.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3c9d9b700 1 -- 172.21.15.32:0/3585056748 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c000c0c0 con 0x7fe3cc0d1da0 2021-04-09T18:23:57.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3c9d9b700 1 -- 172.21.15.32:0/3585056748 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3c000c510 con 0x7fe3cc0d1da0 2021-04-09T18:23:57.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3d1070700 1 -- 172.21.15.32:0/3585056748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3cc0d1da0 msgr2=0x7fe3cc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:57.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.033 7fe3d1070700 1 --2- 172.21.15.32:0/3585056748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3cc0d1da0 0x7fe3cc0d7ca0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7fe3c0006650 tx=0x7fe3c0009a80).stop 2021-04-09T18:23:57.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.034 7fe3d1070700 1 -- 172.21.15.32:0/3585056748 shutdown_connections 2021-04-09T18:23:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.034 7fe3d1070700 1 --2- 172.21.15.32:0/3585056748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3cc0d1da0 0x7fe3cc0d7ca0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.034 7fe3d1070700 1 --2- 172.21.15.32:0/3585056748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc10d010 0x7fe3cc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.034 7fe3d1070700 1 -- 172.21.15.32:0/3585056748 >> 172.21.15.32:0/3585056748 conn(0x7fe3cc0ce5f0 msgr2=0x7fe3cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:57.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.034 7fe3d1070700 1 -- 172.21.15.32:0/3585056748 shutdown_connections 2021-04-09T18:23:57.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.034 7fe3d1070700 1 -- 172.21.15.32:0/3585056748 wait complete. 2021-04-09T18:23:57.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.054 7fe3d1070700 1 Processor -- start 2021-04-09T18:23:57.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.054 7fe3d1070700 1 -- start start 2021-04-09T18:23:57.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3d1070700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3cc0b1e50 0x7fe3cc0b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3d1070700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc0b9fd0 0x7fe3cc0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3d1070700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3cc0b5510 0x7fe3cc0b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:57.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3d1070700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0b2250 con 0x7fe3cc0b5510 2021-04-09T18:23:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3d1070700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0a1220 con 0x7fe3cc0b1e50 2021-04-09T18:23:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3d1070700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0b6b30 con 0x7fe3cc0b9fd0 2021-04-09T18:23:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cb59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3cc0b5510 0x7fe3cc0b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:57.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cb59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3cc0b5510 0x7fe3cc0b5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50786/0 (socket says 172.21.15.32:50786) 2021-04-09T18:23:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cb59e700 1 -- 172.21.15.32:0/247337442 learned_addr learned my addr 172.21.15.32:0/247337442 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3ca59c700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc0b9fd0 0x7fe3cc0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cad9d700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3cc0b1e50 0x7fe3cc0b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cb59e700 1 -- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc0b9fd0 msgr2=0x7fe3cc0b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:57.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cb59e700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc0b9fd0 0x7fe3cc0b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cb59e700 1 -- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3cc0b1e50 msgr2=0x7fe3cc0b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:57.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cb59e700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3cc0b1e50 0x7fe3cc0b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cb59e700 1 -- 172.21.15.32:0/247337442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc0b5f30 con 0x7fe3cc0b5510 2021-04-09T18:23:57.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3cb59e700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3cc0b5510 0x7fe3cc0b5910 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7fe3c4001b90 tx=0x7fe3c400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:57.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3bb7fe700 1 -- 172.21.15.32:0/247337442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3c40070d0 con 0x7fe3cc0b5510 2021-04-09T18:23:57.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3bb7fe700 1 -- 172.21.15.32:0/247337442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c4007450 con 0x7fe3cc0b5510 2021-04-09T18:23:57.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3d1070700 1 -- 172.21.15.32:0/247337442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc0b5f30 con 0x7fe3cc0b5510 2021-04-09T18:23:57.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.055 7fe3bb7fe700 1 -- 172.21.15.32:0/247337442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3c40011f0 con 0x7fe3cc0b5510 2021-04-09T18:23:57.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.056 7fe3d1070700 1 -- 172.21.15.32:0/247337442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc0b5f30 con 0x7fe3cc0b5510 2021-04-09T18:23:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.056 7fe3bb7fe700 1 -- 172.21.15.32:0/247337442 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe3c400ad90 con 0x7fe3cc0b5510 2021-04-09T18:23:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.056 7fe3bb7fe700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3b0018930 0x7fe3b001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.056 7fe3bb7fe700 1 -- 172.21.15.32:0/247337442 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe3c4032d30 con 0x7fe3cc0b5510 2021-04-09T18:23:57.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.056 7fe3cad9d700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3b0018930 0x7fe3b001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:57.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.057 7fe3cad9d700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3b0018930 0x7fe3b001ae10 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:57.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.076 7fe3d1070700 1 -- 172.21.15.32:0/247337442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc1b7040 con 0x7fe3cc0b5510 2021-04-09T18:23:57.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.079 7fe3bb7fe700 1 -- 172.21.15.32:0/247337442 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe3c400ad90 con 0x7fe3cc0b5510 2021-04-09T18:23:57.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.272 7fe3d1070700 1 -- 172.21.15.32:0/247337442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fe3cc1b7040 con 0x7fe3cc0b5510 2021-04-09T18:23:57.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.830 7fe3bb7fe700 1 -- 172.21.15.32:0/247337442 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v206) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe3c4007450 con 0x7fe3cc0b5510 2021-04-09T18:23:57.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.839 7fe3d1070700 1 -- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3b0018930 msgr2=0x7fe3b001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:57.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.839 7fe3d1070700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3b0018930 0x7fe3b001ae10 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.839 7fe3d1070700 1 -- 172.21.15.32:0/247337442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3cc0b5510 msgr2=0x7fe3cc0b5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.839 7fe3d1070700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3cc0b5510 0x7fe3cc0b5910 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7fe3c4001b90 tx=0x7fe3c400ac50).stop 2021-04-09T18:23:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.839 7fe3d1070700 1 -- 172.21.15.32:0/247337442 shutdown_connections 2021-04-09T18:23:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.840 7fe3d1070700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3b0018930 0x7fe3b001ae10 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.840 7fe3d1070700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3cc0b5510 0x7fe3cc0b5910 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.840 7fe3d1070700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3cc0b9fd0 0x7fe3cc0b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.840 7fe3d1070700 1 --2- 172.21.15.32:0/247337442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3cc0b1e50 0x7fe3cc0b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.840 7fe3d1070700 1 -- 172.21.15.32:0/247337442 >> 172.21.15.32:0/247337442 conn(0x7fe3cc0b27e0 msgr2=0x7fe3cc1b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.840 7fe3d1070700 1 -- 172.21.15.32:0/247337442 shutdown_connections 2021-04-09T18:23:57.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:57.840 7fe3d1070700 1 -- 172.21.15.32:0/247337442 wait complete. 2021-04-09T18:23:57.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:23:58.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.003 7f17be130700 1 Processor -- start 2021-04-09T18:23:58.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.004 7f17be130700 1 -- start start 2021-04-09T18:23:58.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.004 7f17be130700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b810da00 0x7f17b810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:58.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.004 7f17be130700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b810d480 con 0x7f17b80d7ca0 2021-04-09T18:23:58.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.004 7f17be130700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b80d1f70 con 0x7f17b810d010 2021-04-09T18:23:58.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.004 7f17be130700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b80d9220 con 0x7f17b810da00 2021-04-09T18:23:58.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.004 7f17b6ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f17b80d7ca0 0x7f17b80d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60380/0 (socket says 172.21.15.32:60380) 2021-04-09T18:23:58.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.004 7f17b6ffd700 1 -- 172.21.15.32:0/2978326843 learned_addr learned my addr 172.21.15.32:0/2978326843 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:58.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b7fff700 1 --2- 172.21.15.32:0/2978326843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b810da00 0x7f17b810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:58.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b67fc700 1 -- 172.21.15.32:0/2978326843 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339745757 0 0) 0x7f17ac000ce0 con 0x7f17b80d7ca0 2021-04-09T18:23:58.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b67fc700 1 -- 172.21.15.32:0/2978326843 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a0002fe0 con 0x7f17b80d7ca0 2021-04-09T18:23:58.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b67fc700 1 -- 172.21.15.32:0/2978326843 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813386239 0 0) 0x7f17a8000c60 con 0x7f17b810d010 2021-04-09T18:23:58.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b67fc700 1 -- 172.21.15.32:0/2978326843 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a0003210 con 0x7f17b810d010 2021-04-09T18:23:58.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b67fc700 1 -- 172.21.15.32:0/2978326843 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3390629565 0 0) 0x7f17ac0010e0 con 0x7f17b80d7ca0 2021-04-09T18:23:58.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b7fff700 1 -- 172.21.15.32:0/2978326843 >> v1:172.21.15.86:6789/0 conn(0x7f17b810d010 legacy=0x7f17b80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:58.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b7fff700 1 -- 172.21.15.32:0/2978326843 >> v1:172.21.15.32:6789/0 conn(0x7f17b80d7ca0 legacy=0x7f17b80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:58.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b7fff700 1 -- 172.21.15.32:0/2978326843 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b810e420 con 0x7f17b810da00 2021-04-09T18:23:58.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b7fff700 1 --2- 172.21.15.32:0/2978326843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b810da00 0x7f17b810de00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f17b0006e50 tx=0x7f17b0009e70).ready entity=mon.2 client_cookie=626ed3c3aa67ee39 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:58.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b67fc700 1 -- 172.21.15.32:0/2978326843 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17b0006a20 con 0x7f17b810da00 2021-04-09T18:23:58.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b67fc700 1 -- 172.21.15.32:0/2978326843 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b0006790 con 0x7f17b810da00 2021-04-09T18:23:58.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17b67fc700 1 -- 172.21.15.32:0/2978326843 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17b000b700 con 0x7f17b810da00 2021-04-09T18:23:58.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17be130700 1 -- 172.21.15.32:0/2978326843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b810da00 msgr2=0x7f17b810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:58.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.005 7f17be130700 1 --2- 172.21.15.32:0/2978326843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b810da00 0x7f17b810de00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f17b0006e50 tx=0x7f17b0009e70).stop 2021-04-09T18:23:58.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.006 7f17be130700 1 -- 172.21.15.32:0/2978326843 shutdown_connections 2021-04-09T18:23:58.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.006 7f17be130700 1 --2- 172.21.15.32:0/2978326843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b810da00 0x7f17b810de00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:58.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.006 7f17be130700 1 -- 172.21.15.32:0/2978326843 >> 172.21.15.32:0/2978326843 conn(0x7f17b80ce5f0 msgr2=0x7f17b80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:58.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.006 7f17be130700 1 -- 172.21.15.32:0/2978326843 shutdown_connections 2021-04-09T18:23:58.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.006 7f17be130700 1 -- 172.21.15.32:0/2978326843 wait complete. 2021-04-09T18:23:58.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.026 7f17be130700 1 Processor -- start 2021-04-09T18:23:58.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.026 7f17be130700 1 -- start start 2021-04-09T18:23:58.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17be130700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b80a0a10 0x7f17b80bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:58.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17be130700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17b818ad00 0x7f17b819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:58.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17be130700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b81a03f0 0x7f17b8095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:58.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17be130700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b818b100 con 0x7f17b81a03f0 2021-04-09T18:23:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17be130700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b8095860 con 0x7f17b818ad00 2021-04-09T18:23:58.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17be130700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b8095a80 con 0x7f17b80a0a10 2021-04-09T18:23:58.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b81a03f0 0x7f17b8095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:58.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b81a03f0 0x7f17b8095240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50796/0 (socket says 172.21.15.32:50796) 2021-04-09T18:23:58.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b7fff700 1 -- 172.21.15.32:0/1064019304 learned_addr learned my addr 172.21.15.32:0/1064019304 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:58.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b6ffd700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17b818ad00 0x7f17b819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:58.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b77fe700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b80a0a10 0x7f17b80bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:58.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b7fff700 1 -- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b80a0a10 msgr2=0x7f17b80bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:58.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b7fff700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b80a0a10 0x7f17b80bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b7fff700 1 -- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17b818ad00 msgr2=0x7f17b819fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b7fff700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17b818ad00 0x7f17b819fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:58.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b7fff700 1 -- 172.21.15.32:0/1064019304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b8189250 con 0x7f17b81a03f0 2021-04-09T18:23:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b77fe700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b80a0a10 0x7f17b80bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:23:58.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.027 7f17b7fff700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b81a03f0 0x7f17b8095240 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f17b00099f0 tx=0x7f17b000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:58.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.028 7f179ffff700 1 -- 172.21.15.32:0/1064019304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17b00093d0 con 0x7f17b81a03f0 2021-04-09T18:23:58.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.028 7f179ffff700 1 -- 172.21.15.32:0/1064019304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b00090c0 con 0x7f17b81a03f0 2021-04-09T18:23:58.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.028 7f17be130700 1 -- 172.21.15.32:0/1064019304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b8189250 con 0x7f17b81a03f0 2021-04-09T18:23:58.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.028 7f179ffff700 1 -- 172.21.15.32:0/1064019304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17b0001420 con 0x7f17b81a03f0 2021-04-09T18:23:58.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.028 7f17be130700 1 -- 172.21.15.32:0/1064019304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b8189250 con 0x7f17b81a03f0 2021-04-09T18:23:58.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.028 7f179ffff700 1 -- 172.21.15.32:0/1064019304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f17b00011a0 con 0x7f17b81a03f0 2021-04-09T18:23:58.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.028 7f179ffff700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1798018930 0x7f179801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:58.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.029 7f179ffff700 1 -- 172.21.15.32:0/1064019304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f17b0032dd0 con 0x7f17b81a03f0 2021-04-09T18:23:58.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.029 7f17b77fe700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1798018930 0x7f179801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:58.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.029 7f17b77fe700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1798018930 0x7f179801adc0 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:58.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.048 7f17be130700 1 -- 172.21.15.32:0/1064019304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b8093d50 con 0x7f17b81a03f0 2021-04-09T18:23:58.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.051 7f179ffff700 1 -- 172.21.15.32:0/1064019304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f17b0000df0 con 0x7f17b81a03f0 2021-04-09T18:23:58.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.245 7f17be130700 1 -- 172.21.15.32:0/1064019304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f17b810d480 con 0x7f17b81a03f0 2021-04-09T18:23:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.836 7f179ffff700 1 -- 172.21.15.32:0/1064019304 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v207) v1 ==== 105+0+0 (secure 0 0 0) 0x7f17b00090a0 con 0x7f17b81a03f0 2021-04-09T18:23:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.862 7f17be130700 1 -- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1798018930 msgr2=0x7f179801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:58.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.862 7f17be130700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1798018930 0x7f179801adc0 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.862 7f17be130700 1 -- 172.21.15.32:0/1064019304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b81a03f0 msgr2=0x7f17b8095240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.862 7f17be130700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b81a03f0 0x7f17b8095240 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f17b00099f0 tx=0x7f17b000bd10).stop 2021-04-09T18:23:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.863 7f17be130700 1 -- 172.21.15.32:0/1064019304 shutdown_connections 2021-04-09T18:23:58.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.863 7f17be130700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1798018930 0x7f179801adc0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:58.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.863 7f17be130700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b81a03f0 0x7f17b8095240 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:58.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.863 7f17be130700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17b818ad00 0x7f17b819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:58.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.863 7f17be130700 1 --2- 172.21.15.32:0/1064019304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17b80a0a10 0x7f17b80bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:58.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.863 7f17be130700 1 -- 172.21.15.32:0/1064019304 >> 172.21.15.32:0/1064019304 conn(0x7f17b80975f0 msgr2=0x7f17b8096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.863 7f17be130700 1 -- 172.21.15.32:0/1064019304 shutdown_connections 2021-04-09T18:23:58.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:58.863 7f17be130700 1 -- 172.21.15.32:0/1064019304 wait complete. 2021-04-09T18:23:58.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:23:59.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.032 7f80de4e2700 1 Processor -- start 2021-04-09T18:23:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.032 7f80de4e2700 1 -- start start 2021-04-09T18:23:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.032 7f80de4e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d80cffc0 0x7f80d80d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:59.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.032 7f80de4e2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80d80d6ca0 0x7f80d810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:59.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.032 7f80de4e2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d80d72a0 con 0x7f80d80d7820 2021-04-09T18:23:59.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.032 7f80de4e2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d80d70a0 con 0x7f80d80d6ca0 2021-04-09T18:23:59.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.032 7f80de4e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d80d85a0 con 0x7f80d80cffc0 2021-04-09T18:23:59.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80dcaff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80d80d6ca0 0x7f80d810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:59.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80cffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f80d80d7820 0x7f80d80d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60390/0 (socket says 172.21.15.32:60390) 2021-04-09T18:23:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80cffff700 1 -- 172.21.15.32:0/4168678122 learned_addr learned my addr 172.21.15.32:0/4168678122 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:23:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80d7fff700 1 --2- 172.21.15.32:0/4168678122 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d80cffc0 0x7f80d80d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80dcaff700 1 -- 172.21.15.32:0/4168678122 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d80cffc0 msgr2=0x7f80d80d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:23:59.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80dcaff700 1 --2- 172.21.15.32:0/4168678122 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d80cffc0 0x7f80d80d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80dcaff700 1 -- 172.21.15.32:0/4168678122 >> v1:172.21.15.32:6789/0 conn(0x7f80d80d7820 legacy=0x7f80d80d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80dcaff700 1 -- 172.21.15.32:0/4168678122 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d80d9200 con 0x7f80d80d6ca0 2021-04-09T18:23:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80d77fe700 1 -- 172.21.15.32:0/4168678122 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282927185 0 0) 0x7f80c0000ce0 con 0x7f80d80d7820 2021-04-09T18:23:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80dcaff700 1 --2- 172.21.15.32:0/4168678122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d80d6ca0 0x7f80d810cff0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f80d0006e30 tx=0x7f80d0009de0).ready entity=mon.0 client_cookie=b24eb7c952d000d6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:59.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80d77fe700 1 -- 172.21.15.32:0/4168678122 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d000b300 con 0x7f80d80d6ca0 2021-04-09T18:23:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80d77fe700 1 -- 172.21.15.32:0/4168678122 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d000b680 con 0x7f80d80d6ca0 2021-04-09T18:23:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80d77fe700 1 -- 172.21.15.32:0/4168678122 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d000bb60 con 0x7f80d80d6ca0 2021-04-09T18:23:59.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80de4e2700 1 -- 172.21.15.32:0/4168678122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d80d6ca0 msgr2=0x7f80d810cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.033 7f80de4e2700 1 --2- 172.21.15.32:0/4168678122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d80d6ca0 0x7f80d810cff0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f80d0006e30 tx=0x7f80d0009de0).stop 2021-04-09T18:23:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.034 7f80de4e2700 1 -- 172.21.15.32:0/4168678122 shutdown_connections 2021-04-09T18:23:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.034 7f80de4e2700 1 --2- 172.21.15.32:0/4168678122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d80d6ca0 0x7f80d810cff0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.034 7f80de4e2700 1 --2- 172.21.15.32:0/4168678122 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d80cffc0 0x7f80d80d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.034 7f80de4e2700 1 -- 172.21.15.32:0/4168678122 >> 172.21.15.32:0/4168678122 conn(0x7f80d80ce5e0 msgr2=0x7f80d80cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:59.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.034 7f80de4e2700 1 -- 172.21.15.32:0/4168678122 shutdown_connections 2021-04-09T18:23:59.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.034 7f80de4e2700 1 -- 172.21.15.32:0/4168678122 wait complete. 2021-04-09T18:23:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.054 7f80de4e2700 1 Processor -- start 2021-04-09T18:23:59.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.054 7f80de4e2700 1 -- start start 2021-04-09T18:23:59.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80de4e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d819e290 0x7f80d809b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80de4e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d809bdd0 0x7f80d8094d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80de4e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80d80953a0 0x7f80d81890b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80de4e2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d809b230 con 0x7f80d809bdd0 2021-04-09T18:23:59.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80de4e2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d809c1d0 con 0x7f80d80953a0 2021-04-09T18:23:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80de4e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d80957a0 con 0x7f80d819e290 2021-04-09T18:23:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d809bdd0 0x7f80d8094d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d809bdd0 0x7f80d8094d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50808/0 (socket says 172.21.15.32:50808) 2021-04-09T18:23:59.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80cffff700 1 -- 172.21.15.32:0/1028160944 learned_addr learned my addr 172.21.15.32:0/1028160944 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:23:59.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80d7fff700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d819e290 0x7f80d809b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:59.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80dcaff700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80d80953a0 0x7f80d81890b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:59.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80cffff700 1 -- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d819e290 msgr2=0x7f80d809b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:59.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80cffff700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d819e290 0x7f80d809b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80cffff700 1 -- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80d80953a0 msgr2=0x7f80d81890b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80cffff700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80d80953a0 0x7f80d81890b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.055 7f80cffff700 1 -- 172.21.15.32:0/1028160944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d809ae30 con 0x7f80d809bdd0 2021-04-09T18:23:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.056 7f80cffff700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d809bdd0 0x7f80d8094d80 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7f80c0006f00 tx=0x7f80c000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:59.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.056 7f80d57fa700 1 -- 172.21.15.32:0/1028160944 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80c0006020 con 0x7f80d809bdd0 2021-04-09T18:23:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.056 7f80d57fa700 1 -- 172.21.15.32:0/1028160944 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c00063a0 con 0x7f80d809bdd0 2021-04-09T18:23:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.056 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d809ae30 con 0x7f80d809bdd0 2021-04-09T18:23:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.056 7f80d57fa700 1 -- 172.21.15.32:0/1028160944 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80c0004620 con 0x7f80d809bdd0 2021-04-09T18:23:59.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.056 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d809ae30 con 0x7f80d809bdd0 2021-04-09T18:23:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.056 7f80d57fa700 1 -- 172.21.15.32:0/1028160944 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f80c000a000 con 0x7f80d809bdd0 2021-04-09T18:23:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.057 7f80d57fa700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80bc0188d0 0x7f80bc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:23:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.057 7f80d57fa700 1 -- 172.21.15.32:0/1028160944 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f80c0032a10 con 0x7f80d809bdd0 2021-04-09T18:23:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.057 7f80d7fff700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80bc0188d0 0x7f80bc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:23:59.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.057 7f80d7fff700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80bc0188d0 0x7f80bc01add0 crc :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:23:59.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.076 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d8093e70 con 0x7f80d809bdd0 2021-04-09T18:23:59.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.080 7f80d57fa700 1 -- 172.21.15.32:0/1028160944 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f80c0006b90 con 0x7f80d809bdd0 2021-04-09T18:23:59.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.273 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f80d8093e70 con 0x7f80d809bdd0 2021-04-09T18:23:59.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.838 7f80d57fa700 1 -- 172.21.15.32:0/1028160944 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v208) v1 ==== 105+0+0 (secure 0 0 0) 0x7f80c000a000 con 0x7f80d809bdd0 2021-04-09T18:23:59.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80bc0188d0 msgr2=0x7f80bc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:59.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80bc0188d0 0x7f80bc01add0 crc :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d809bdd0 msgr2=0x7f80d8094d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:23:59.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d809bdd0 0x7f80d8094d80 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7f80c0006f00 tx=0x7f80c000b940).stop 2021-04-09T18:23:59.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 shutdown_connections 2021-04-09T18:23:59.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80bc0188d0 0x7f80bc01add0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80d809bdd0 0x7f80d8094d80 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80d80953a0 0x7f80d81890b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 --2- 172.21.15.32:0/1028160944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80d819e290 0x7f80d809b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:23:59.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 >> 172.21.15.32:0/1028160944 conn(0x7f80d809fa50 msgr2=0x7f80d80a2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:23:59.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 shutdown_connections 2021-04-09T18:23:59.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:23:59.841 7f80de4e2700 1 -- 172.21.15.32:0/1028160944 wait complete. 2021-04-09T18:23:59.913 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.013 7faaff972700 1 Processor -- start 2021-04-09T18:24:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.013 7faaff972700 1 -- start start 2021-04-09T18:24:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.013 7faaff972700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf810d010 0x7faaf80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faaff972700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faaf80d1da0 0x7faaf80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faaff972700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf810d480 con 0x7faaf80d82c0 2021-04-09T18:24:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faaff972700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf810e780 con 0x7faaf80d1da0 2021-04-09T18:24:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faaff972700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80d8d30 con 0x7faaf810d010 2021-04-09T18:24:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafd78e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf810d010 0x7faaf80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafcf8d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faaf80d1da0 0x7faaf80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafd78e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf810d010 0x7faaf80d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:58040/0 (socket says 172.21.15.32:58040) 2021-04-09T18:24:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafcf8d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faaf80d1da0 0x7faaf80d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:42572/0 (socket says 172.21.15.32:42572) 2021-04-09T18:24:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafd78e700 1 -- 172.21.15.32:0/2088107733 learned_addr learned my addr 172.21.15.32:0/2088107733 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafcf8d700 1 -- 172.21.15.32:0/2088107733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf810d010 msgr2=0x7faaf80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafcf8d700 1 --2- 172.21.15.32:0/2088107733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf810d010 0x7faaf80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafcf8d700 1 -- 172.21.15.32:0/2088107733 >> v1:172.21.15.86:6789/0 conn(0x7faaf80d82c0 legacy=0x7faaf810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faaeffff700 1 -- 172.21.15.32:0/2088107733 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833507439 0 0) 0x7faaf0000d30 con 0x7faaf80d82c0 2021-04-09T18:24:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafcf8d700 1 -- 172.21.15.32:0/2088107733 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf80d9220 con 0x7faaf80d1da0 2021-04-09T18:24:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.014 7faafd78e700 1 --2- 172.21.15.32:0/2088107733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf810d010 0x7faaf80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faafcf8d700 1 --2- 172.21.15.32:0/2088107733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf80d1da0 0x7faaf80d7ca0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7faaf4006650 tx=0x7faaf4009a80).ready entity=mon.1 client_cookie=d402657a82a26ede server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faaeffff700 1 -- 172.21.15.32:0/2088107733 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaf40009b0 con 0x7faaf80d1da0 2021-04-09T18:24:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faaeffff700 1 -- 172.21.15.32:0/2088107733 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf400c0c0 con 0x7faaf80d1da0 2021-04-09T18:24:00.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faaeffff700 1 -- 172.21.15.32:0/2088107733 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaf400c510 con 0x7faaf80d1da0 2021-04-09T18:24:00.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faaff972700 1 -- 172.21.15.32:0/2088107733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf80d1da0 msgr2=0x7faaf80d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:00.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faaff972700 1 --2- 172.21.15.32:0/2088107733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf80d1da0 0x7faaf80d7ca0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7faaf4006650 tx=0x7faaf4009a80).stop 2021-04-09T18:24:00.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faaff972700 1 -- 172.21.15.32:0/2088107733 shutdown_connections 2021-04-09T18:24:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faaff972700 1 --2- 172.21.15.32:0/2088107733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf80d1da0 0x7faaf80d7ca0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faaff972700 1 --2- 172.21.15.32:0/2088107733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf810d010 0x7faaf80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.015 7faaff972700 1 -- 172.21.15.32:0/2088107733 >> 172.21.15.32:0/2088107733 conn(0x7faaf80ce5f0 msgr2=0x7faaf80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.016 7faaff972700 1 -- 172.21.15.32:0/2088107733 shutdown_connections 2021-04-09T18:24:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.016 7faaff972700 1 -- 172.21.15.32:0/2088107733 wait complete. 2021-04-09T18:24:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.036 7faaff972700 1 Processor -- start 2021-04-09T18:24:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.036 7faaff972700 1 -- start start 2021-04-09T18:24:00.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.036 7faaff972700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf80c3c00 0x7faaf809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.036 7faaff972700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf8095240 0x7faaf81a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.036 7faaff972700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf81a1270 0x7faaf80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.036 7faaff972700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf809cda0 con 0x7faaf80c3c00 2021-04-09T18:24:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.036 7faaff972700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80c3680 con 0x7faaf81a1270 2021-04-09T18:24:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.036 7faaff972700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8095640 con 0x7faaf8095240 2021-04-09T18:24:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafd78e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf80c3c00 0x7faaf809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafd78e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf80c3c00 0x7faaf809c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50822/0 (socket says 172.21.15.32:50822) 2021-04-09T18:24:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafd78e700 1 -- 172.21.15.32:0/3946588504 learned_addr learned my addr 172.21.15.32:0/3946588504 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafcf8d700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf8095240 0x7faaf81a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafdf8f700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf81a1270 0x7faaf80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafd78e700 1 -- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf8095240 msgr2=0x7faaf81a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafd78e700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf8095240 0x7faaf81a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafd78e700 1 -- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf81a1270 msgr2=0x7faaf80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafd78e700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf81a1270 0x7faaf80b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafd78e700 1 -- 172.21.15.32:0/3946588504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf81a1670 con 0x7faaf80c3c00 2021-04-09T18:24:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faafd78e700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf80c3c00 0x7faaf809c600 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7faae8006ee0 tx=0x7faae800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faaedffb700 1 -- 172.21.15.32:0/3946588504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faae800bd50 con 0x7faaf80c3c00 2021-04-09T18:24:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faaedffb700 1 -- 172.21.15.32:0/3946588504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae80052e0 con 0x7faaf80c3c00 2021-04-09T18:24:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faaff972700 1 -- 172.21.15.32:0/3946588504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf81a1670 con 0x7faaf80c3c00 2021-04-09T18:24:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faaedffb700 1 -- 172.21.15.32:0/3946588504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faae8005840 con 0x7faaf80c3c00 2021-04-09T18:24:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.037 7faaff972700 1 -- 172.21.15.32:0/3946588504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf81a1670 con 0x7faaf80c3c00 2021-04-09T18:24:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.038 7faaedffb700 1 -- 172.21.15.32:0/3946588504 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faae80054c0 con 0x7faaf80c3c00 2021-04-09T18:24:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.038 7faaedffb700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faad8018930 0x7faad801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:00.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.038 7faaedffb700 1 -- 172.21.15.32:0/3946588504 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7faae8032a10 con 0x7faaf80c3c00 2021-04-09T18:24:00.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.038 7faafcf8d700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faad8018930 0x7faad801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:00.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.038 7faafcf8d700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faad8018930 0x7faad801ae10 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:00.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.058 7faaff972700 1 -- 172.21.15.32:0/3946588504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf8093d50 con 0x7faaf80c3c00 2021-04-09T18:24:00.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.061 7faaedffb700 1 -- 172.21.15.32:0/3946588504 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faae80054f0 con 0x7faaf80c3c00 2021-04-09T18:24:00.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.255 7faaff972700 1 -- 172.21.15.32:0/3946588504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7faaf810d480 con 0x7faaf80c3c00 2021-04-09T18:24:00.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.841 7faaedffb700 1 -- 172.21.15.32:0/3946588504 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v209) v1 ==== 105+0+0 (secure 0 0 0) 0x7faae80052e0 con 0x7faaf80c3c00 2021-04-09T18:24:00.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 -- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faad8018930 msgr2=0x7faad801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faad8018930 0x7faad801ae10 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 -- 172.21.15.32:0/3946588504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf80c3c00 msgr2=0x7faaf809c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf80c3c00 0x7faaf809c600 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7faae8006ee0 tx=0x7faae800b960).stop 2021-04-09T18:24:00.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 -- 172.21.15.32:0/3946588504 shutdown_connections 2021-04-09T18:24:00.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faad8018930 0x7faad801ae10 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faaf81a1270 0x7faaf80b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faaf8095240 0x7faaf81a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 --2- 172.21.15.32:0/3946588504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faaf80c3c00 0x7faaf809c600 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:00.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.872 7faaff972700 1 -- 172.21.15.32:0/3946588504 >> 172.21.15.32:0/3946588504 conn(0x7faaf80975f0 msgr2=0x7faaf8096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:00.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.873 7faaff972700 1 -- 172.21.15.32:0/3946588504 shutdown_connections 2021-04-09T18:24:00.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:00.873 7faaff972700 1 -- 172.21.15.32:0/3946588504 wait complete. 2021-04-09T18:24:00.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:01.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.036 7fee6290d700 1 Processor -- start 2021-04-09T18:24:01.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.036 7fee6290d700 1 -- start start 2021-04-09T18:24:01.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.037 7fee6290d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0d7ca0 0x7fee5c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.037 7fee6290d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee5c10da00 0x7fee5c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.037 7fee6290d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5c10d480 con 0x7fee5c10d010 2021-04-09T18:24:01.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.037 7fee6290d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5c0d9220 con 0x7fee5c10da00 2021-04-09T18:24:01.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.037 7fee6290d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0d86c0 con 0x7fee5c0d7ca0 2021-04-09T18:24:01.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.037 7fee5bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fee5c10d010 0x7fee5c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60416/0 (socket says 172.21.15.32:60416) 2021-04-09T18:24:01.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.037 7fee5bfff700 1 -- 172.21.15.32:0/957622549 learned_addr learned my addr 172.21.15.32:0/957622549 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:01.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.037 7fee53fff700 1 --2- 172.21.15.32:0/957622549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0d7ca0 0x7fee5c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:01.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.037 7fee60f2a700 1 --2- 172.21.15.32:0/957622549 >> v2:172.21.15.86:3300/0 conn(0x7fee5c10da00 0x7fee5c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:01.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee5b7fe700 1 -- 172.21.15.32:0/957622549 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623867453 0 0) 0x7fee4c000ce0 con 0x7fee5c10d010 2021-04-09T18:24:01.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee5b7fe700 1 -- 172.21.15.32:0/957622549 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee48002fe0 con 0x7fee5c10d010 2021-04-09T18:24:01.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee53fff700 1 -- 172.21.15.32:0/957622549 >> v2:172.21.15.86:3300/0 conn(0x7fee5c10da00 msgr2=0x7fee5c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:01.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee53fff700 1 --2- 172.21.15.32:0/957622549 >> v2:172.21.15.86:3300/0 conn(0x7fee5c10da00 0x7fee5c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee53fff700 1 -- 172.21.15.32:0/957622549 >> v1:172.21.15.32:6789/0 conn(0x7fee5c10d010 legacy=0x7fee5c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:01.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee5b7fe700 1 -- 172.21.15.32:0/957622549 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3077374284 0 0) 0x7fee4c0010e0 con 0x7fee5c10d010 2021-04-09T18:24:01.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee53fff700 1 -- 172.21.15.32:0/957622549 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c0d1f70 con 0x7fee5c0d7ca0 2021-04-09T18:24:01.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee60f2a700 1 --2- 172.21.15.32:0/957622549 >> v2:172.21.15.86:3300/0 conn(0x7fee5c10da00 0x7fee5c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee53fff700 1 --2- 172.21.15.32:0/957622549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0d7ca0 0x7fee5c0d80a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fee44006650 tx=0x7fee44009a80).ready entity=mon.2 client_cookie=7af2c019c27a9be0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee5b7fe700 1 -- 172.21.15.32:0/957622549 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee440009b0 con 0x7fee5c0d7ca0 2021-04-09T18:24:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee5b7fe700 1 -- 172.21.15.32:0/957622549 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4400c0c0 con 0x7fee5c0d7ca0 2021-04-09T18:24:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee5b7fe700 1 -- 172.21.15.32:0/957622549 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee4400c510 con 0x7fee5c0d7ca0 2021-04-09T18:24:01.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee6290d700 1 -- 172.21.15.32:0/957622549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0d7ca0 msgr2=0x7fee5c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:01.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.038 7fee6290d700 1 --2- 172.21.15.32:0/957622549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0d7ca0 0x7fee5c0d80a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fee44006650 tx=0x7fee44009a80).stop 2021-04-09T18:24:01.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.039 7fee6290d700 1 -- 172.21.15.32:0/957622549 shutdown_connections 2021-04-09T18:24:01.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.039 7fee6290d700 1 --2- 172.21.15.32:0/957622549 >> v2:172.21.15.86:3300/0 conn(0x7fee5c10da00 0x7fee5c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.039 7fee6290d700 1 --2- 172.21.15.32:0/957622549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0d7ca0 0x7fee5c0d80a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.039 7fee6290d700 1 -- 172.21.15.32:0/957622549 >> 172.21.15.32:0/957622549 conn(0x7fee5c0ce5f0 msgr2=0x7fee5c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:01.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.039 7fee6290d700 1 -- 172.21.15.32:0/957622549 shutdown_connections 2021-04-09T18:24:01.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.039 7fee6290d700 1 -- 172.21.15.32:0/957622549 wait complete. 2021-04-09T18:24:01.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.059 7fee6290d700 1 Processor -- start 2021-04-09T18:24:01.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.059 7fee6290d700 1 -- start start 2021-04-09T18:24:01.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee6290d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0a0cb0 0x7fee5c092550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:01.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee6290d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee5c0932e0 0x7fee5c08a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:01.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee6290d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee5c0936e0 0x7fee5c08ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:01.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee6290d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0b34f0 con 0x7fee5c0936e0 2021-04-09T18:24:01.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee6290d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c092d60 con 0x7fee5c0932e0 2021-04-09T18:24:01.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee6290d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c0900a0 con 0x7fee5c0a0cb0 2021-04-09T18:24:01.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee60f2a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee5c0936e0 0x7fee5c08ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:01.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee60f2a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee5c0936e0 0x7fee5c08ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50836/0 (socket says 172.21.15.32:50836) 2021-04-09T18:24:01.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee60f2a700 1 -- 172.21.15.32:0/3984666231 learned_addr learned my addr 172.21.15.32:0/3984666231 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:01.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee53fff700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee5c0932e0 0x7fee5c08a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:01.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee5bfff700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0a0cb0 0x7fee5c092550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:01.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee60f2a700 1 -- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0a0cb0 msgr2=0x7fee5c092550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:01.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee60f2a700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0a0cb0 0x7fee5c092550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee60f2a700 1 -- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee5c0932e0 msgr2=0x7fee5c08a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:01.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee60f2a700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee5c0932e0 0x7fee5c08a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee60f2a700 1 -- 172.21.15.32:0/3984666231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c0a10b0 con 0x7fee5c0936e0 2021-04-09T18:24:01.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee53fff700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee5c0932e0 0x7fee5c08a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:01.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.060 7fee60f2a700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee5c0936e0 0x7fee5c08ac40 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7fee5400b910 tx=0x7fee5400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:01.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.061 7fee597fa700 1 -- 172.21.15.32:0/3984666231 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee54006aa0 con 0x7fee5c0936e0 2021-04-09T18:24:01.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.061 7fee597fa700 1 -- 172.21.15.32:0/3984666231 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee54003030 con 0x7fee5c0936e0 2021-04-09T18:24:01.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.061 7fee6290d700 1 -- 172.21.15.32:0/3984666231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c0a10b0 con 0x7fee5c0936e0 2021-04-09T18:24:01.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.061 7fee597fa700 1 -- 172.21.15.32:0/3984666231 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee54001470 con 0x7fee5c0936e0 2021-04-09T18:24:01.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.061 7fee6290d700 1 -- 172.21.15.32:0/3984666231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5c0a10b0 con 0x7fee5c0936e0 2021-04-09T18:24:01.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.062 7fee597fa700 1 -- 172.21.15.32:0/3984666231 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fee54006aa0 con 0x7fee5c0936e0 2021-04-09T18:24:01.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.062 7fee597fa700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee400188c0 0x7fee4001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:01.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.062 7fee597fa700 1 -- 172.21.15.32:0/3984666231 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fee54032b00 con 0x7fee5c0936e0 2021-04-09T18:24:01.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.062 7fee5bfff700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee400188c0 0x7fee4001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:01.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.062 7fee5bfff700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee400188c0 0x7fee4001adc0 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:01.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.081 7fee6290d700 1 -- 172.21.15.32:0/3984666231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5c10d480 con 0x7fee5c0936e0 2021-04-09T18:24:01.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.084 7fee597fa700 1 -- 172.21.15.32:0/3984666231 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fee5400bdf0 con 0x7fee5c0936e0 2021-04-09T18:24:01.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.277 7fee6290d700 1 -- 172.21.15.32:0/3984666231 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fee5c10d480 con 0x7fee5c0936e0 2021-04-09T18:24:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.849 7fee597fa700 1 -- 172.21.15.32:0/3984666231 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v210) v1 ==== 105+0+0 (secure 0 0 0) 0x7fee54006aa0 con 0x7fee5c0936e0 2021-04-09T18:24:01.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 -- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee400188c0 msgr2=0x7fee4001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee400188c0 0x7fee4001adc0 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 -- 172.21.15.32:0/3984666231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee5c0936e0 msgr2=0x7fee5c08ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:01.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee5c0936e0 0x7fee5c08ac40 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7fee5400b910 tx=0x7fee5400bcb0).stop 2021-04-09T18:24:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 -- 172.21.15.32:0/3984666231 shutdown_connections 2021-04-09T18:24:01.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee400188c0 0x7fee4001adc0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee5c0936e0 0x7fee5c08ac40 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee5c0932e0 0x7fee5c08a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 --2- 172.21.15.32:0/3984666231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee5c0a0cb0 0x7fee5c092550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.894 7fee6290d700 1 -- 172.21.15.32:0/3984666231 >> 172.21.15.32:0/3984666231 conn(0x7fee5c0b2fa0 msgr2=0x7fee5c0a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:01.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.895 7fee6290d700 1 -- 172.21.15.32:0/3984666231 shutdown_connections 2021-04-09T18:24:01.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:01.895 7fee6290d700 1 -- 172.21.15.32:0/3984666231 wait complete. 2021-04-09T18:24:01.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:24:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.063 7f07f7eff700 1 Processor -- start 2021-04-09T18:24:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.063 7f07f7eff700 1 -- start start 2021-04-09T18:24:02.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f7eff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f010d010 0x7f07f00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:02.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f7eff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f07f00d1da0 0x7f07f00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:02.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f7eff700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f010d480 con 0x7f07f00d82c0 2021-04-09T18:24:02.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f7eff700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f010e780 con 0x7f07f00d1da0 2021-04-09T18:24:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f7eff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00d8d30 con 0x7f07f010d010 2021-04-09T18:24:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f651c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f07f00d82c0 0x7f07f010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60432/0 (socket says 172.21.15.32:60432) 2021-04-09T18:24:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f651c700 1 -- 172.21.15.32:0/3667983942 learned_addr learned my addr 172.21.15.32:0/3667983942 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f551a700 1 --2- 172.21.15.32:0/3667983942 >> v2:172.21.15.86:3300/0 conn(0x7f07f00d1da0 0x7f07f00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f5d1b700 1 --2- 172.21.15.32:0/3667983942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f010d010 0x7f07f00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f4d19700 1 -- 172.21.15.32:0/3667983942 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246770257 0 0) 0x7f07e8000d30 con 0x7f07f00d82c0 2021-04-09T18:24:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f4d19700 1 -- 172.21.15.32:0/3667983942 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07dc002fe0 con 0x7f07f00d82c0 2021-04-09T18:24:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f4d19700 1 -- 172.21.15.32:0/3667983942 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1437186777 0 0) 0x7f07e8001460 con 0x7f07f00d82c0 2021-04-09T18:24:02.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f5d1b700 1 -- 172.21.15.32:0/3667983942 >> v2:172.21.15.86:3300/0 conn(0x7f07f00d1da0 msgr2=0x7f07f00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f5d1b700 1 --2- 172.21.15.32:0/3667983942 >> v2:172.21.15.86:3300/0 conn(0x7f07f00d1da0 0x7f07f00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.064 7f07f5d1b700 1 -- 172.21.15.32:0/3667983942 >> v1:172.21.15.32:6789/0 conn(0x7f07f00d82c0 legacy=0x7f07f010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.065 7f07f5d1b700 1 -- 172.21.15.32:0/3667983942 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f00d9220 con 0x7f07f010d010 2021-04-09T18:24:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.065 7f07f551a700 1 --2- 172.21.15.32:0/3667983942 >> v2:172.21.15.86:3300/0 conn(0x7f07f00d1da0 0x7f07f00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:02.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.065 7f07f5d1b700 1 --2- 172.21.15.32:0/3667983942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f010d010 0x7f07f00d1780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f07ec006650 tx=0x7f07ec009a80).ready entity=mon.2 client_cookie=216b01deeae40b64 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:02.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.065 7f07f4d19700 1 -- 172.21.15.32:0/3667983942 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07ec0009b0 con 0x7f07f010d010 2021-04-09T18:24:02.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.065 7f07f4d19700 1 -- 172.21.15.32:0/3667983942 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ec00c0c0 con 0x7f07f010d010 2021-04-09T18:24:02.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.065 7f07f4d19700 1 -- 172.21.15.32:0/3667983942 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07ec00c510 con 0x7f07f010d010 2021-04-09T18:24:02.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.065 7f07f7eff700 1 -- 172.21.15.32:0/3667983942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f010d010 msgr2=0x7f07f00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:02.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.065 7f07f7eff700 1 --2- 172.21.15.32:0/3667983942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f010d010 0x7f07f00d1780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f07ec006650 tx=0x7f07ec009a80).stop 2021-04-09T18:24:02.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.066 7f07f7eff700 1 -- 172.21.15.32:0/3667983942 shutdown_connections 2021-04-09T18:24:02.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.066 7f07f7eff700 1 --2- 172.21.15.32:0/3667983942 >> v2:172.21.15.86:3300/0 conn(0x7f07f00d1da0 0x7f07f00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.066 7f07f7eff700 1 --2- 172.21.15.32:0/3667983942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f010d010 0x7f07f00d1780 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.066 7f07f7eff700 1 -- 172.21.15.32:0/3667983942 >> 172.21.15.32:0/3667983942 conn(0x7f07f00ce5f0 msgr2=0x7f07f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.066 7f07f7eff700 1 -- 172.21.15.32:0/3667983942 shutdown_connections 2021-04-09T18:24:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.066 7f07f7eff700 1 -- 172.21.15.32:0/3667983942 wait complete. 2021-04-09T18:24:02.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.086 7f07f7eff700 1 Processor -- start 2021-04-09T18:24:02.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.086 7f07f7eff700 1 -- start start 2021-04-09T18:24:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f7eff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07f01a0050 0x7f07f01a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:02.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f7eff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07f00bdcd0 0x7f07f00bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:02.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f7eff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f00bf080 0x7f07f00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:02.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f7eff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00bf500 con 0x7f07f01a0050 2021-04-09T18:24:02.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f7eff700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00bb7b0 con 0x7f07f00bdcd0 2021-04-09T18:24:02.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f7eff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00be0d0 con 0x7f07f00bf080 2021-04-09T18:24:02.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f5d1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07f01a0050 0x7f07f01a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:02.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f5d1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07f01a0050 0x7f07f01a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50846/0 (socket says 172.21.15.32:50846) 2021-04-09T18:24:02.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f5d1b700 1 -- 172.21.15.32:0/2601532381 learned_addr learned my addr 172.21.15.32:0/2601532381 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f551a700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07f00bdcd0 0x7f07f00bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f651c700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f00bf080 0x7f07f00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f5d1b700 1 -- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f00bf080 msgr2=0x7f07f00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:02.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f5d1b700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f00bf080 0x7f07f00b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f5d1b700 1 -- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07f00bdcd0 msgr2=0x7f07f00bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:02.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f5d1b700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07f00bdcd0 0x7f07f00bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f5d1b700 1 -- 172.21.15.32:0/2601532381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f00b1310 con 0x7f07f01a0050 2021-04-09T18:24:02.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f5d1b700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07f01a0050 0x7f07f01a0450 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7f07ec009630 tx=0x7f07ec00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:02.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07e67fc700 1 -- 172.21.15.32:0/2601532381 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07ec0093a0 con 0x7f07f01a0050 2021-04-09T18:24:02.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07e67fc700 1 -- 172.21.15.32:0/2601532381 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ec00bcf0 con 0x7f07f01a0050 2021-04-09T18:24:02.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f00b1310 con 0x7f07f01a0050 2021-04-09T18:24:02.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07e67fc700 1 -- 172.21.15.32:0/2601532381 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07ec0054d0 con 0x7f07f01a0050 2021-04-09T18:24:02.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.087 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f00b1310 con 0x7f07f01a0050 2021-04-09T18:24:02.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.088 7f07e67fc700 1 -- 172.21.15.32:0/2601532381 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f07ec0054f0 con 0x7f07f01a0050 2021-04-09T18:24:02.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.088 7f07e67fc700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07d8018930 0x7f07d801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:02.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.088 7f07e67fc700 1 -- 172.21.15.32:0/2601532381 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f07ec032a10 con 0x7f07f01a0050 2021-04-09T18:24:02.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.088 7f07f551a700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07d8018930 0x7f07d801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:02.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.088 7f07f551a700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07d8018930 0x7f07d801ae10 crc :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:02.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.108 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f0093d50 con 0x7f07f01a0050 2021-04-09T18:24:02.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.111 7f07e67fc700 1 -- 172.21.15.32:0/2601532381 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f07ec0052e0 con 0x7f07f01a0050 2021-04-09T18:24:02.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.304 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f07f0093d50 con 0x7f07f01a0050 2021-04-09T18:24:02.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.855 7f07e67fc700 1 -- 172.21.15.32:0/2601532381 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v211) v1 ==== 105+0+0 (secure 0 0 0) 0x7f07ec0054f0 con 0x7f07f01a0050 2021-04-09T18:24:02.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.870 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07d8018930 msgr2=0x7f07d801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:02.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07d8018930 0x7f07d801ae10 crc :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07f01a0050 msgr2=0x7f07f01a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:02.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07f01a0050 0x7f07f01a0450 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7f07ec009630 tx=0x7f07ec00b900).stop 2021-04-09T18:24:02.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 shutdown_connections 2021-04-09T18:24:02.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07d8018930 0x7f07d801ae10 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07f00bf080 0x7f07f00b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07f00bdcd0 0x7f07f00bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 --2- 172.21.15.32:0/2601532381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07f01a0050 0x7f07f01a0450 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:02.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 >> 172.21.15.32:0/2601532381 conn(0x7f07f00975f0 msgr2=0x7f07f0096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:02.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 shutdown_connections 2021-04-09T18:24:02.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:02.871 7f07f7eff700 1 -- 172.21.15.32:0/2601532381 wait complete. 2021-04-09T18:24:02.941 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.034 7f3ee382f700 1 Processor -- start 2021-04-09T18:24:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.034 7f3ee382f700 1 -- start start 2021-04-09T18:24:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.034 7f3ee382f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3edc0d7ca0 0x7f3edc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.034 7f3ee382f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc10da00 0x7f3edc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:03.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.034 7f3ee382f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc10d480 con 0x7f3edc10d010 2021-04-09T18:24:03.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.034 7f3ee382f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3edc0d9220 con 0x7f3edc0d7ca0 2021-04-09T18:24:03.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee382f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc0d86c0 con 0x7f3edc10da00 2021-04-09T18:24:03.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee0e4a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3edc0d7ca0 0x7f3edc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:03.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee164b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3edc10d010 0x7f3edc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60444/0 (socket says 172.21.15.32:60444) 2021-04-09T18:24:03.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee164b700 1 -- 172.21.15.32:0/2489370060 learned_addr learned my addr 172.21.15.32:0/2489370060 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:03.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee1e4c700 1 --2- 172.21.15.32:0/2489370060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc10da00 0x7f3edc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ed3fff700 1 -- 172.21.15.32:0/2489370060 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653056602 0 0) 0x7f3ed8000ce0 con 0x7f3edc10d010 2021-04-09T18:24:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ed3fff700 1 -- 172.21.15.32:0/2489370060 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec8002fe0 con 0x7f3edc10d010 2021-04-09T18:24:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee0e4a700 1 -- 172.21.15.32:0/2489370060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc10da00 msgr2=0x7f3edc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:03.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee0e4a700 1 --2- 172.21.15.32:0/2489370060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc10da00 0x7f3edc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee0e4a700 1 -- 172.21.15.32:0/2489370060 >> v1:172.21.15.32:6789/0 conn(0x7f3edc10d010 legacy=0x7f3edc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee0e4a700 1 -- 172.21.15.32:0/2489370060 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc0d1f70 con 0x7f3edc0d7ca0 2021-04-09T18:24:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee0e4a700 1 --2- 172.21.15.32:0/2489370060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0d7ca0 0x7f3edc0d80a0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7f3ecc006650 tx=0x7f3ecc009a80).ready entity=mon.0 client_cookie=c28fad5d43420bf1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:03.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ed3fff700 1 -- 172.21.15.32:0/2489370060 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ecc0009b0 con 0x7f3edc0d7ca0 2021-04-09T18:24:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ed3fff700 1 -- 172.21.15.32:0/2489370060 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ecc00c0c0 con 0x7f3edc0d7ca0 2021-04-09T18:24:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ed3fff700 1 -- 172.21.15.32:0/2489370060 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ecc00c510 con 0x7f3edc0d7ca0 2021-04-09T18:24:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee382f700 1 -- 172.21.15.32:0/2489370060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0d7ca0 msgr2=0x7f3edc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.035 7f3ee382f700 1 --2- 172.21.15.32:0/2489370060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0d7ca0 0x7f3edc0d80a0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7f3ecc006650 tx=0x7f3ecc009a80).stop 2021-04-09T18:24:03.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.036 7f3ee382f700 1 -- 172.21.15.32:0/2489370060 shutdown_connections 2021-04-09T18:24:03.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.036 7f3ee382f700 1 --2- 172.21.15.32:0/2489370060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc10da00 0x7f3edc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.036 7f3ee382f700 1 --2- 172.21.15.32:0/2489370060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0d7ca0 0x7f3edc0d80a0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.036 7f3ee382f700 1 -- 172.21.15.32:0/2489370060 >> 172.21.15.32:0/2489370060 conn(0x7f3edc0ce5f0 msgr2=0x7f3edc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:03.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.036 7f3ee382f700 1 -- 172.21.15.32:0/2489370060 shutdown_connections 2021-04-09T18:24:03.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.036 7f3ee382f700 1 -- 172.21.15.32:0/2489370060 wait complete. 2021-04-09T18:24:03.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.056 7f3ee382f700 1 Processor -- start 2021-04-09T18:24:03.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.056 7f3ee382f700 1 -- start start 2021-04-09T18:24:03.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee382f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3edc19e1e0 0x7f3edc0b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:03.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee382f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0bba10 0x7f3edc0bbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:03.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee382f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc0b6310 0x7f3edc0b6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:03.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee382f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc0bc5c0 con 0x7f3edc0bba10 2021-04-09T18:24:03.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee382f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc0b9aa0 con 0x7f3edc19e1e0 2021-04-09T18:24:03.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee382f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc0b6d30 con 0x7f3edc0b6310 2021-04-09T18:24:03.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee0e4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0bba10 0x7f3edc0bbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:03.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee0e4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0bba10 0x7f3edc0bbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50860/0 (socket says 172.21.15.32:50860) 2021-04-09T18:24:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee0e4a700 1 -- 172.21.15.32:0/2500667891 learned_addr learned my addr 172.21.15.32:0/2500667891 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee164b700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3edc19e1e0 0x7f3edc0b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee1e4c700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc0b6310 0x7f3edc0b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:03.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee0e4a700 1 -- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc0b6310 msgr2=0x7f3edc0b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:03.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee0e4a700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc0b6310 0x7f3edc0b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee0e4a700 1 -- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3edc19e1e0 msgr2=0x7f3edc0b7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:03.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee0e4a700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3edc19e1e0 0x7f3edc0b7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.057 7f3ee0e4a700 1 -- 172.21.15.32:0/2500667891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc0b7760 con 0x7f3edc0bba10 2021-04-09T18:24:03.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.058 7f3ee0e4a700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0bba10 0x7f3edc0bbe10 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f3ecc009630 tx=0x7f3ecc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:03.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.058 7f3ed1ffb700 1 -- 172.21.15.32:0/2500667891 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ecc0093a0 con 0x7f3edc0bba10 2021-04-09T18:24:03.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.058 7f3ed1ffb700 1 -- 172.21.15.32:0/2500667891 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ecc00bd70 con 0x7f3edc0bba10 2021-04-09T18:24:03.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.058 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc0b7760 con 0x7f3edc0bba10 2021-04-09T18:24:03.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.058 7f3ed1ffb700 1 -- 172.21.15.32:0/2500667891 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ecc005560 con 0x7f3edc0bba10 2021-04-09T18:24:03.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.058 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc0b7760 con 0x7f3edc0bba10 2021-04-09T18:24:03.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.059 7f3ed1ffb700 1 -- 172.21.15.32:0/2500667891 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3ecc0052e0 con 0x7f3edc0bba10 2021-04-09T18:24:03.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.059 7f3ed1ffb700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc018930 0x7f3ebc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:03.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.059 7f3ed1ffb700 1 -- 172.21.15.32:0/2500667891 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3ecc032a30 con 0x7f3edc0bba10 2021-04-09T18:24:03.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.059 7f3ee164b700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc018930 0x7f3ebc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:03.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.059 7f3ee164b700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc018930 0x7f3ebc01ae10 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:03.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.079 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc10d480 con 0x7f3edc0bba10 2021-04-09T18:24:03.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.082 7f3ed1ffb700 1 -- 172.21.15.32:0/2500667891 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3ecc01f770 con 0x7f3edc0bba10 2021-04-09T18:24:03.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.276 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f3edc10d480 con 0x7f3edc0bba10 2021-04-09T18:24:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.856 7f3ed1ffb700 1 -- 172.21.15.32:0/2500667891 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v212) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3ecc0052e0 con 0x7f3edc0bba10 2021-04-09T18:24:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc018930 msgr2=0x7f3ebc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc018930 0x7f3ebc01ae10 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0bba10 msgr2=0x7f3edc0bbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0bba10 0x7f3edc0bbe10 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f3ecc009630 tx=0x7f3ecc00b980).stop 2021-04-09T18:24:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 shutdown_connections 2021-04-09T18:24:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ebc018930 0x7f3ebc01ae10 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3edc0bba10 0x7f3edc0bbe10 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3edc0b6310 0x7f3edc0b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 --2- 172.21.15.32:0/2500667891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3edc19e1e0 0x7f3edc0b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:03.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 >> 172.21.15.32:0/2500667891 conn(0x7f3edc0a0ad0 msgr2=0x7f3edc0ba5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:03.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 shutdown_connections 2021-04-09T18:24:03.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:03.894 7f3ee382f700 1 -- 172.21.15.32:0/2500667891 wait complete. 2021-04-09T18:24:03.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:04.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.058 7f20c41ea700 1 Processor -- start 2021-04-09T18:24:04.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.058 7f20c41ea700 1 -- start start 2021-04-09T18:24:04.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.059 7f20c41ea700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f20bc0cffc0 0x7f20bc0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:04.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.059 7f20c41ea700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f20bc0d6ca0 0x7f20bc10cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:04.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.059 7f20c41ea700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc0d72a0 con 0x7f20bc0d7820 2021-04-09T18:24:04.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.059 7f20c41ea700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20bc0d70a0 con 0x7f20bc0cffc0 2021-04-09T18:24:04.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.059 7f20c41ea700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20bc0d9200 con 0x7f20bc0d6ca0 2021-04-09T18:24:04.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.059 7f20c2006700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f20bc0cffc0 0x7f20bc0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:04.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.059 7f20c1805700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f20bc0d7820 0x7f20bc0d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60462/0 (socket says 172.21.15.32:60462) 2021-04-09T18:24:04.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.059 7f20c1805700 1 -- 172.21.15.32:0/1984097337 learned_addr learned my addr 172.21.15.32:0/1984097337 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:04.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.059 7f20c2807700 1 --2- 172.21.15.32:0/1984097337 >> v2:172.21.15.86:3300/0 conn(0x7f20bc0d6ca0 0x7f20bc10cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:04.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c2006700 1 -- 172.21.15.32:0/1984097337 >> v2:172.21.15.86:3300/0 conn(0x7f20bc0d6ca0 msgr2=0x7f20bc10cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:04.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c2006700 1 --2- 172.21.15.32:0/1984097337 >> v2:172.21.15.86:3300/0 conn(0x7f20bc0d6ca0 0x7f20bc10cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c2006700 1 -- 172.21.15.32:0/1984097337 >> v1:172.21.15.32:6789/0 conn(0x7f20bc0d7820 legacy=0x7f20bc0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:04.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c2006700 1 -- 172.21.15.32:0/1984097337 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc0d8410 con 0x7f20bc0cffc0 2021-04-09T18:24:04.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c1004700 1 -- 172.21.15.32:0/1984097337 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788219520 0 0) 0x7f20b00009b0 con 0x7f20bc0d7820 2021-04-09T18:24:04.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c2006700 1 --2- 172.21.15.32:0/1984097337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc0cffc0 0x7f20bc0d2460 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f20b8006650 tx=0x7f20b8009a00).ready entity=mon.0 client_cookie=6055525901cf03c9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:04.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c1004700 1 -- 172.21.15.32:0/1984097337 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b8009df0 con 0x7f20bc0cffc0 2021-04-09T18:24:04.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c1004700 1 -- 172.21.15.32:0/1984097337 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b80009b0 con 0x7f20bc0cffc0 2021-04-09T18:24:04.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c1004700 1 -- 172.21.15.32:0/1984097337 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b800c2b0 con 0x7f20bc0cffc0 2021-04-09T18:24:04.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c41ea700 1 -- 172.21.15.32:0/1984097337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc0cffc0 msgr2=0x7f20bc0d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:04.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.060 7f20c41ea700 1 --2- 172.21.15.32:0/1984097337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc0cffc0 0x7f20bc0d2460 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f20b8006650 tx=0x7f20b8009a00).stop 2021-04-09T18:24:04.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.061 7f20c41ea700 1 -- 172.21.15.32:0/1984097337 shutdown_connections 2021-04-09T18:24:04.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.061 7f20c41ea700 1 --2- 172.21.15.32:0/1984097337 >> v2:172.21.15.86:3300/0 conn(0x7f20bc0d6ca0 0x7f20bc10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.061 7f20c41ea700 1 --2- 172.21.15.32:0/1984097337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc0cffc0 0x7f20bc0d2460 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.061 7f20c41ea700 1 -- 172.21.15.32:0/1984097337 >> 172.21.15.32:0/1984097337 conn(0x7f20bc0ce5e0 msgr2=0x7f20bc0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.061 7f20c41ea700 1 -- 172.21.15.32:0/1984097337 shutdown_connections 2021-04-09T18:24:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.061 7f20c41ea700 1 -- 172.21.15.32:0/1984097337 wait complete. 2021-04-09T18:24:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.081 7f20c41ea700 1 Processor -- start 2021-04-09T18:24:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.081 7f20c41ea700 1 -- start start 2021-04-09T18:24:04.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c41ea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20bc1833a0 0x7f20bc0a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:04.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c41ea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20bc0a2550 0x7f20bc0a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:04.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c41ea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc094d00 0x7f20bc095100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:04.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c41ea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc0a1fd0 con 0x7f20bc094d00 2021-04-09T18:24:04.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c41ea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc0a2f70 con 0x7f20bc1833a0 2021-04-09T18:24:04.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c41ea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc095720 con 0x7f20bc0a2550 2021-04-09T18:24:04.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2807700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc094d00 0x7f20bc095100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:04.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2807700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc094d00 0x7f20bc095100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50878/0 (socket says 172.21.15.32:50878) 2021-04-09T18:24:04.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2807700 1 -- 172.21.15.32:0/25164029 learned_addr learned my addr 172.21.15.32:0/25164029 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:04.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c1805700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20bc0a2550 0x7f20bc0a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:04.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2006700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20bc1833a0 0x7f20bc0a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:04.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2807700 1 -- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20bc0a2550 msgr2=0x7f20bc0a2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:04.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2807700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20bc0a2550 0x7f20bc0a2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2807700 1 -- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20bc1833a0 msgr2=0x7f20bc0a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:04.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2807700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20bc1833a0 0x7f20bc0a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2807700 1 -- 172.21.15.32:0/25164029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc095940 con 0x7f20bc094d00 2021-04-09T18:24:04.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2006700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20bc1833a0 0x7f20bc0a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:04.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c2807700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc094d00 0x7f20bc095100 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f20b400b910 tx=0x7f20b400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:04.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20aeffd700 1 -- 172.21.15.32:0/25164029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b4006aa0 con 0x7f20bc094d00 2021-04-09T18:24:04.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20aeffd700 1 -- 172.21.15.32:0/25164029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b4003030 con 0x7f20bc094d00 2021-04-09T18:24:04.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20c41ea700 1 -- 172.21.15.32:0/25164029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20bc095940 con 0x7f20bc094d00 2021-04-09T18:24:04.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.082 7f20aeffd700 1 -- 172.21.15.32:0/25164029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b4001470 con 0x7f20bc094d00 2021-04-09T18:24:04.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.083 7f20c41ea700 1 -- 172.21.15.32:0/25164029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20bc095940 con 0x7f20bc094d00 2021-04-09T18:24:04.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.083 7f20aeffd700 1 -- 172.21.15.32:0/25164029 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f20b4000ec0 con 0x7f20bc094d00 2021-04-09T18:24:04.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.083 7f20aeffd700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20a4018930 0x7f20a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:04.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.083 7f20aeffd700 1 -- 172.21.15.32:0/25164029 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f20b4032de0 con 0x7f20bc094d00 2021-04-09T18:24:04.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.083 7f20c2006700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20a4018930 0x7f20a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:04.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.084 7f20c2006700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20a4018930 0x7f20a401ae10 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:04.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.103 7f20c41ea700 1 -- 172.21.15.32:0/25164029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20bc095be0 con 0x7f20bc094d00 2021-04-09T18:24:04.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.107 7f20aeffd700 1 -- 172.21.15.32:0/25164029 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f20b400bdf0 con 0x7f20bc094d00 2021-04-09T18:24:04.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.300 7f20c41ea700 1 -- 172.21.15.32:0/25164029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f20bc095be0 con 0x7f20bc094d00 2021-04-09T18:24:04.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.863 7f20aeffd700 1 -- 172.21.15.32:0/25164029 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v213) v1 ==== 105+0+0 (secure 0 0 0) 0x7f20b4000ec0 con 0x7f20bc094d00 2021-04-09T18:24:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 -- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20a4018930 msgr2=0x7f20a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:04.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20a4018930 0x7f20a401ae10 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 -- 172.21.15.32:0/25164029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc094d00 msgr2=0x7f20bc095100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:04.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc094d00 0x7f20bc095100 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f20b400b910 tx=0x7f20b400bcb0).stop 2021-04-09T18:24:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 -- 172.21.15.32:0/25164029 shutdown_connections 2021-04-09T18:24:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f20a4018930 0x7f20a401ae10 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f20bc094d00 0x7f20bc095100 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20bc0a2550 0x7f20bc0a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 --2- 172.21.15.32:0/25164029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20bc1833a0 0x7f20bc0a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:04.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 -- 172.21.15.32:0/25164029 >> 172.21.15.32:0/25164029 conn(0x7f20bc09fa00 msgr2=0x7f20bc0b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:04.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 -- 172.21.15.32:0/25164029 shutdown_connections 2021-04-09T18:24:04.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:04.867 7f20c41ea700 1 -- 172.21.15.32:0/25164029 wait complete. 2021-04-09T18:24:04.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:24:05.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.033 7f12ce11e700 1 Processor -- start 2021-04-09T18:24:05.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.033 7f12ce11e700 1 -- start start 2021-04-09T18:24:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.034 7f12ce11e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c810da00 0x7f12c810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.034 7f12ce11e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c810d480 con 0x7f12c810d010 2021-04-09T18:24:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.034 7f12ce11e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c80d1f70 con 0x7f12c80d7ca0 2021-04-09T18:24:05.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.034 7f12ce11e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80d9220 con 0x7f12c810da00 2021-04-09T18:24:05.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.034 7f12c77fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f12c810d010 0x7f12c80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60472/0 (socket says 172.21.15.32:60472) 2021-04-09T18:24:05.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.034 7f12c77fe700 1 -- 172.21.15.32:0/4267186333 learned_addr learned my addr 172.21.15.32:0/4267186333 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:05.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.034 7f12c7fff700 1 --2- 172.21.15.32:0/4267186333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c810da00 0x7f12c810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:05.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c67fc700 1 -- 172.21.15.32:0/4267186333 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046417452 0 0) 0x7f12bc000c60 con 0x7f12c80d7ca0 2021-04-09T18:24:05.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c67fc700 1 -- 172.21.15.32:0/4267186333 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b0002fe0 con 0x7f12c80d7ca0 2021-04-09T18:24:05.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c67fc700 1 -- 172.21.15.32:0/4267186333 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236507169 0 0) 0x7f12b8000ce0 con 0x7f12c810d010 2021-04-09T18:24:05.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c67fc700 1 -- 172.21.15.32:0/4267186333 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b00042c0 con 0x7f12c810d010 2021-04-09T18:24:05.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c7fff700 1 -- 172.21.15.32:0/4267186333 >> v1:172.21.15.86:6789/0 conn(0x7f12c80d7ca0 legacy=0x7f12c80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:05.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c7fff700 1 -- 172.21.15.32:0/4267186333 >> v1:172.21.15.32:6789/0 conn(0x7f12c810d010 legacy=0x7f12c80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:05.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c7fff700 1 -- 172.21.15.32:0/4267186333 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c810e420 con 0x7f12c810da00 2021-04-09T18:24:05.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c7fff700 1 --2- 172.21.15.32:0/4267186333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c810da00 0x7f12c810de00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f12c0006e50 tx=0x7f12c0009e70).ready entity=mon.2 client_cookie=335763038eee2179 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:05.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c67fc700 1 -- 172.21.15.32:0/4267186333 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12c0006a20 con 0x7f12c810da00 2021-04-09T18:24:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c67fc700 1 -- 172.21.15.32:0/4267186333 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c0006790 con 0x7f12c810da00 2021-04-09T18:24:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12c67fc700 1 -- 172.21.15.32:0/4267186333 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12c000b700 con 0x7f12c810da00 2021-04-09T18:24:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12ce11e700 1 -- 172.21.15.32:0/4267186333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c810da00 msgr2=0x7f12c810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:05.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.035 7f12ce11e700 1 --2- 172.21.15.32:0/4267186333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c810da00 0x7f12c810de00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f12c0006e50 tx=0x7f12c0009e70).stop 2021-04-09T18:24:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.036 7f12ce11e700 1 -- 172.21.15.32:0/4267186333 shutdown_connections 2021-04-09T18:24:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.036 7f12ce11e700 1 --2- 172.21.15.32:0/4267186333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c810da00 0x7f12c810de00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.036 7f12ce11e700 1 -- 172.21.15.32:0/4267186333 >> 172.21.15.32:0/4267186333 conn(0x7f12c80ce5f0 msgr2=0x7f12c80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:05.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.036 7f12ce11e700 1 -- 172.21.15.32:0/4267186333 shutdown_connections 2021-04-09T18:24:05.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.036 7f12ce11e700 1 -- 172.21.15.32:0/4267186333 wait complete. 2021-04-09T18:24:05.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.056 7f12ce11e700 1 Processor -- start 2021-04-09T18:24:05.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.056 7f12ce11e700 1 -- start start 2021-04-09T18:24:05.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12ce11e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12c8190ee0 0x7f12c80a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:05.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12ce11e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c80a3570 0x7f12c80a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:05.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12ce11e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12c80a6150 0x7f12c80a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:05.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12ce11e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80a55b0 con 0x7f12c80a6150 2021-04-09T18:24:05.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12ce11e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80a3970 con 0x7f12c8190ee0 2021-04-09T18:24:05.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12ce11e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80a6550 con 0x7f12c80a3570 2021-04-09T18:24:05.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12c80a6150 0x7f12c80a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:05.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c77fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12c8190ee0 0x7f12c80a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:05.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12c80a6150 0x7f12c80a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50892/0 (socket says 172.21.15.32:50892) 2021-04-09T18:24:05.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c7fff700 1 -- 172.21.15.32:0/2126623132 learned_addr learned my addr 172.21.15.32:0/2126623132 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:05.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c6ffd700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c80a3570 0x7f12c80a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:05.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c7fff700 1 -- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c80a3570 msgr2=0x7f12c80a5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:05.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c7fff700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c80a3570 0x7f12c80a5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:05.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c7fff700 1 -- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12c8190ee0 msgr2=0x7f12c80a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:05.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c7fff700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12c8190ee0 0x7f12c80a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:05.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c7fff700 1 -- 172.21.15.32:0/2126623132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c80a3d80 con 0x7f12c80a6150 2021-04-09T18:24:05.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c77fe700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12c8190ee0 0x7f12c80a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:05.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12c7fff700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12c80a6150 0x7f12c80a4400 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7f12c00099f0 tx=0x7f12c000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:05.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12affff700 1 -- 172.21.15.32:0/2126623132 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12c00093d0 con 0x7f12c80a6150 2021-04-09T18:24:05.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12affff700 1 -- 172.21.15.32:0/2126623132 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c0003030 con 0x7f12c80a6150 2021-04-09T18:24:05.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c80a3d80 con 0x7f12c80a6150 2021-04-09T18:24:05.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12affff700 1 -- 172.21.15.32:0/2126623132 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12c0001420 con 0x7f12c80a6150 2021-04-09T18:24:05.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.057 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c80a3d80 con 0x7f12c80a6150 2021-04-09T18:24:05.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.058 7f12affff700 1 -- 172.21.15.32:0/2126623132 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f12c00011a0 con 0x7f12c80a6150 2021-04-09T18:24:05.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.058 7f12affff700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12a8018930 0x7f12a801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:05.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.058 7f12affff700 1 -- 172.21.15.32:0/2126623132 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f12c0032d70 con 0x7f12c80a6150 2021-04-09T18:24:05.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.058 7f12c77fe700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12a8018930 0x7f12a801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:05.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.059 7f12c77fe700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12a8018930 0x7f12a801ae10 crc :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:05.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.078 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c810d480 con 0x7f12c80a6150 2021-04-09T18:24:05.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.082 7f12affff700 1 -- 172.21.15.32:0/2126623132 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f12c0000df0 con 0x7f12c80a6150 2021-04-09T18:24:05.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.275 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f12c810d480 con 0x7f12c80a6150 2021-04-09T18:24:05.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.868 7f12affff700 1 -- 172.21.15.32:0/2126623132 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v214) v1 ==== 105+0+0 (secure 0 0 0) 0x7f12c001fab0 con 0x7f12c80a6150 2021-04-09T18:24:05.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.891 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12a8018930 msgr2=0x7f12a801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:05.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.891 7f12ce11e700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12a8018930 0x7f12a801ae10 crc :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:05.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12c80a6150 msgr2=0x7f12c80a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:05.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12c80a6150 0x7f12c80a4400 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7f12c00099f0 tx=0x7f12c000bd10).stop 2021-04-09T18:24:05.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 shutdown_connections 2021-04-09T18:24:05.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12a8018930 0x7f12a801ae10 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:05.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12c80a6150 0x7f12c80a4400 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:05.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12c80a3570 0x7f12c80a5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:05.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 --2- 172.21.15.32:0/2126623132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12c8190ee0 0x7f12c80a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:05.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 >> 172.21.15.32:0/2126623132 conn(0x7f12c80bbda0 msgr2=0x7f12c81b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:05.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 shutdown_connections 2021-04-09T18:24:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:05.892 7f12ce11e700 1 -- 172.21.15.32:0/2126623132 wait complete. 2021-04-09T18:24:05.965 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:05.987 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:24:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.071 7f5a4bc07700 1 Processor -- start 2021-04-09T18:24:06.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a4bc07700 1 -- start start 2021-04-09T18:24:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a4bc07700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5a440d7ca0 0x7f5a440d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:06.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a4bc07700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a4410da00 0x7f5a4410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a4bc07700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4410d480 con 0x7f5a4410d010 2021-04-09T18:24:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a4bc07700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a440d9220 con 0x7f5a4410da00 2021-04-09T18:24:06.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a4bc07700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a440d86c0 con 0x7f5a440d7ca0 2021-04-09T18:24:06.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a4a224700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5a4410da00 0x7f5a4410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:06.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a49a23700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5a4410d010 0x7f5a440d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60496/0 (socket says 172.21.15.32:60496) 2021-04-09T18:24:06.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a49a23700 1 -- 172.21.15.32:0/3616118339 learned_addr learned my addr 172.21.15.32:0/3616118339 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:06.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.072 7f5a49222700 1 --2- 172.21.15.32:0/3616118339 >> v2:172.21.15.86:3300/0 conn(0x7f5a440d7ca0 0x7f5a440d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:06.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a4a224700 1 -- 172.21.15.32:0/3616118339 >> v2:172.21.15.86:3300/0 conn(0x7f5a440d7ca0 msgr2=0x7f5a440d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:06.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a4a224700 1 --2- 172.21.15.32:0/3616118339 >> v2:172.21.15.86:3300/0 conn(0x7f5a440d7ca0 0x7f5a440d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:06.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a4a224700 1 -- 172.21.15.32:0/3616118339 >> v1:172.21.15.32:6789/0 conn(0x7f5a4410d010 legacy=0x7f5a440d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:06.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a4a224700 1 -- 172.21.15.32:0/3616118339 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a440d1f70 con 0x7f5a4410da00 2021-04-09T18:24:06.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a48a21700 1 -- 172.21.15.32:0/3616118339 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060260982 0 0) 0x7f5a34001060 con 0x7f5a4410d010 2021-04-09T18:24:06.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a4a224700 1 --2- 172.21.15.32:0/3616118339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4410da00 0x7f5a4410de00 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f5a3c006ab0 tx=0x7f5a3c009ac0).ready entity=mon.0 client_cookie=8adbb63891c3efcf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:06.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a48a21700 1 -- 172.21.15.32:0/3616118339 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a3c0066d0 con 0x7f5a4410da00 2021-04-09T18:24:06.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a48a21700 1 -- 172.21.15.32:0/3616118339 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c0022e0 con 0x7f5a4410da00 2021-04-09T18:24:06.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a48a21700 1 -- 172.21.15.32:0/3616118339 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a3c0027c0 con 0x7f5a4410da00 2021-04-09T18:24:06.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a4bc07700 1 -- 172.21.15.32:0/3616118339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4410da00 msgr2=0x7f5a4410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:06.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.073 7f5a4bc07700 1 --2- 172.21.15.32:0/3616118339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4410da00 0x7f5a4410de00 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f5a3c006ab0 tx=0x7f5a3c009ac0).stop 2021-04-09T18:24:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.074 7f5a4bc07700 1 -- 172.21.15.32:0/3616118339 shutdown_connections 2021-04-09T18:24:06.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.074 7f5a4bc07700 1 --2- 172.21.15.32:0/3616118339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4410da00 0x7f5a4410de00 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.074 7f5a4bc07700 1 --2- 172.21.15.32:0/3616118339 >> v2:172.21.15.86:3300/0 conn(0x7f5a440d7ca0 0x7f5a440d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:06.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.074 7f5a4bc07700 1 -- 172.21.15.32:0/3616118339 >> 172.21.15.32:0/3616118339 conn(0x7f5a440ce5f0 msgr2=0x7f5a440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.074 7f5a4bc07700 1 -- 172.21.15.32:0/3616118339 shutdown_connections 2021-04-09T18:24:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.074 7f5a4bc07700 1 -- 172.21.15.32:0/3616118339 wait complete. 2021-04-09T18:24:06.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.094 7f5a4bc07700 1 Processor -- start 2021-04-09T18:24:06.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.094 7f5a4bc07700 1 -- start start 2021-04-09T18:24:06.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a4bc07700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a4409e530 0x7f5a4409b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:06.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a4bc07700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4409e930 0x7f5a4409f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:06.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a4bc07700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a4409f990 0x7f5a44094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:06.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a4bc07700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4409edf0 con 0x7f5a4409e930 2021-04-09T18:24:06.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a4bc07700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4409fd90 con 0x7f5a4409f990 2021-04-09T18:24:06.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a4bc07700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4409bd40 con 0x7f5a4409e530 2021-04-09T18:24:06.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a49222700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4409e930 0x7f5a4409f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:06.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a49222700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4409e930 0x7f5a4409f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50910/0 (socket says 172.21.15.32:50910) 2021-04-09T18:24:06.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a49222700 1 -- 172.21.15.32:0/170532042 learned_addr learned my addr 172.21.15.32:0/170532042 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:06.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a49222700 1 -- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a4409e530 msgr2=0x7f5a4409b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:06.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a49222700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a4409e530 0x7f5a4409b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:24:06.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a49222700 1 -- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a4409f990 msgr2=0x7f5a44094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:06.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a49222700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a4409f990 0x7f5a44094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:24:06.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a49222700 1 -- 172.21.15.32:0/170532042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a44095290 con 0x7f5a4409e930 2021-04-09T18:24:06.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a49222700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4409e930 0x7f5a4409f370 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f5a38006f60 tx=0x7f5a3800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:06.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a427fc700 1 -- 172.21.15.32:0/170532042 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a3800bd60 con 0x7f5a4409e930 2021-04-09T18:24:06.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a427fc700 1 -- 172.21.15.32:0/170532042 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a38000c90 con 0x7f5a4409e930 2021-04-09T18:24:06.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a44095290 con 0x7f5a4409e930 2021-04-09T18:24:06.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a427fc700 1 -- 172.21.15.32:0/170532042 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a38002220 con 0x7f5a4409e930 2021-04-09T18:24:06.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.095 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a44095290 con 0x7f5a4409e930 2021-04-09T18:24:06.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.096 7f5a427fc700 1 -- 172.21.15.32:0/170532042 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5a380052e0 con 0x7f5a4409e930 2021-04-09T18:24:06.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.096 7f5a427fc700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a24018930 0x7f5a2401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:06.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.096 7f5a427fc700 1 -- 172.21.15.32:0/170532042 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5a38032a00 con 0x7f5a4409e930 2021-04-09T18:24:06.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.096 7f5a49a23700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a24018930 0x7f5a2401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:06.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.096 7f5a49a23700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a24018930 0x7f5a2401ae10 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:06.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.116 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4410d480 con 0x7f5a4409e930 2021-04-09T18:24:06.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.120 7f5a427fc700 1 -- 172.21.15.32:0/170532042 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5a38000c90 con 0x7f5a4409e930 2021-04-09T18:24:06.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.313 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f5a4410d480 con 0x7f5a4409e930 2021-04-09T18:24:06.679 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:24:06.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:24:06.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.872 7f5a427fc700 1 -- 172.21.15.32:0/170532042 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v215) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5a380052e0 con 0x7f5a4409e930 2021-04-09T18:24:06.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a24018930 msgr2=0x7f5a2401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:06.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a24018930 0x7f5a2401ae10 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:06.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4409e930 msgr2=0x7f5a4409f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:06.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4409e930 0x7f5a4409f370 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f5a38006f60 tx=0x7f5a3800b970).stop 2021-04-09T18:24:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 shutdown_connections 2021-04-09T18:24:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5a24018930 0x7f5a2401ae10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a4409e930 0x7f5a4409f370 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:06.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a4409f990 0x7f5a44094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:24:06.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 --2- 172.21.15.32:0/170532042 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a4409e530 0x7f5a4409b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:24:06.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 >> 172.21.15.32:0/170532042 conn(0x7f5a440c4290 msgr2=0x7f5a440c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:06.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 shutdown_connections 2021-04-09T18:24:06.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:06.880 7f5a4bc07700 1 -- 172.21.15.32:0/170532042 wait complete. 2021-04-09T18:24:06.952 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:07.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.051 7fd7b3e1a700 1 Processor -- start 2021-04-09T18:24:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.051 7fd7b3e1a700 1 -- start start 2021-04-09T18:24:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b3e1a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7ac0cffc0 0x7fd7ac0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:07.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b3e1a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac0d72a0 con 0x7fd7ac0d6ca0 2021-04-09T18:24:07.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b3e1a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac0d8410 con 0x7fd7ac0d7820 2021-04-09T18:24:07.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b3e1a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7ac0d70a0 con 0x7fd7ac0cffc0 2021-04-09T18:24:07.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b2437700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd7ac0d6ca0 0x7fd7ac10cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60512/0 (socket says 172.21.15.32:60512) 2021-04-09T18:24:07.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b2437700 1 -- 172.21.15.32:0/623466769 learned_addr learned my addr 172.21.15.32:0/623466769 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:07.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b1c36700 1 --2- 172.21.15.32:0/623466769 >> v2:172.21.15.86:3300/0 conn(0x7fd7ac0cffc0 0x7fd7ac0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:07.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b0c34700 1 -- 172.21.15.32:0/623466769 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152556590 0 0) 0x7fd7a4000d30 con 0x7fd7ac0d6ca0 2021-04-09T18:24:07.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b0c34700 1 -- 172.21.15.32:0/623466769 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd798002fe0 con 0x7fd7ac0d6ca0 2021-04-09T18:24:07.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b0c34700 1 -- 172.21.15.32:0/623466769 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942115563 0 0) 0x7fd79c000c60 con 0x7fd7ac0d7820 2021-04-09T18:24:07.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b1c36700 1 -- 172.21.15.32:0/623466769 >> v1:172.21.15.86:6789/0 conn(0x7fd7ac0d7820 legacy=0x7fd7ac0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:07.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b1c36700 1 -- 172.21.15.32:0/623466769 >> v1:172.21.15.32:6789/0 conn(0x7fd7ac0d6ca0 legacy=0x7fd7ac10cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:07.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.052 7fd7b1c36700 1 -- 172.21.15.32:0/623466769 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac0d9200 con 0x7fd7ac0cffc0 2021-04-09T18:24:07.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.053 7fd7b1c36700 1 --2- 172.21.15.32:0/623466769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac0cffc0 0x7fd7ac0d2460 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7fd7a8006650 tx=0x7fd7a8009a00).ready entity=mon.1 client_cookie=e1cd5ede5cecbfec server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:07.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.053 7fd7b0c34700 1 -- 172.21.15.32:0/623466769 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7a8009df0 con 0x7fd7ac0cffc0 2021-04-09T18:24:07.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.053 7fd7b0c34700 1 -- 172.21.15.32:0/623466769 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a80009b0 con 0x7fd7ac0cffc0 2021-04-09T18:24:07.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.053 7fd7b0c34700 1 -- 172.21.15.32:0/623466769 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7a800c2b0 con 0x7fd7ac0cffc0 2021-04-09T18:24:07.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.053 7fd7b3e1a700 1 -- 172.21.15.32:0/623466769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac0cffc0 msgr2=0x7fd7ac0d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.053 7fd7b3e1a700 1 --2- 172.21.15.32:0/623466769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac0cffc0 0x7fd7ac0d2460 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7fd7a8006650 tx=0x7fd7a8009a00).stop 2021-04-09T18:24:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.053 7fd7b3e1a700 1 -- 172.21.15.32:0/623466769 shutdown_connections 2021-04-09T18:24:07.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.053 7fd7b3e1a700 1 --2- 172.21.15.32:0/623466769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac0cffc0 0x7fd7ac0d2460 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:07.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.053 7fd7b3e1a700 1 -- 172.21.15.32:0/623466769 >> 172.21.15.32:0/623466769 conn(0x7fd7ac0ce5e0 msgr2=0x7fd7ac0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:07.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.054 7fd7b3e1a700 1 -- 172.21.15.32:0/623466769 shutdown_connections 2021-04-09T18:24:07.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.054 7fd7b3e1a700 1 -- 172.21.15.32:0/623466769 wait complete. 2021-04-09T18:24:07.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.074 7fd7b3e1a700 1 Processor -- start 2021-04-09T18:24:07.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.074 7fd7b3e1a700 1 -- start start 2021-04-09T18:24:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.074 7fd7b3e1a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac193e30 0x7fd7ac194230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.074 7fd7b3e1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac0a3a80 0x7fd7ac0ba580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:07.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.074 7fd7b3e1a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7ac0baba0 0x7fd7ac0b8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:07.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.074 7fd7b3e1a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0b0e70 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.074 7fd7b3e1a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0bafa0 con 0x7fd7ac193e30 2021-04-09T18:24:07.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.074 7fd7b3e1a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0a3e80 con 0x7fd7ac0baba0 2021-04-09T18:24:07.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.074 7fd7b1435700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac0a3a80 0x7fd7ac0ba580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1435700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac0a3a80 0x7fd7ac0ba580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50924/0 (socket says 172.21.15.32:50924) 2021-04-09T18:24:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1435700 1 -- 172.21.15.32:0/766502864 learned_addr learned my addr 172.21.15.32:0/766502864 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1c36700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac193e30 0x7fd7ac194230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:07.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b2437700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7ac0baba0 0x7fd7ac0b8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:07.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1435700 1 -- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7ac0baba0 msgr2=0x7fd7ac0b8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:07.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1435700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7ac0baba0 0x7fd7ac0b8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:07.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1435700 1 -- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac193e30 msgr2=0x7fd7ac194230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:07.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1435700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac193e30 0x7fd7ac194230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:07.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1435700 1 -- 172.21.15.32:0/766502864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac0b12d0 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1c36700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac193e30 0x7fd7ac194230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:07.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b1435700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac0a3a80 0x7fd7ac0ba580 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fd79c006f20 tx=0x7fd79c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:07.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7a27fc700 1 -- 172.21.15.32:0/766502864 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd79c00bd80 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7a27fc700 1 -- 172.21.15.32:0/766502864 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c001080 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7a27fc700 1 -- 172.21.15.32:0/766502864 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd79c001550 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac0b12d0 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.075 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ac0b12d0 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.076 7fd7a27fc700 1 -- 172.21.15.32:0/766502864 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd79c001260 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.076 7fd7a27fc700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd794018930 0x7fd79401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.076 7fd7a27fc700 1 -- 172.21.15.32:0/766502864 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd79c032a40 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.076 7fd7b1c36700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd794018930 0x7fd79401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.076 7fd7b1c36700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd794018930 0x7fd79401ae10 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.096 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ac0d72a0 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.099 7fd7a27fc700 1 -- 172.21.15.32:0/766502864 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd79c006bf0 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.293 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fd7ac0d72a0 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.878 7fd7a27fc700 1 -- 172.21.15.32:0/766502864 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v217) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd79c001080 con 0x7fd7ac0a3a80 2021-04-09T18:24:07.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd794018930 msgr2=0x7fd79401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:07.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd794018930 0x7fd79401ae10 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:07.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac0a3a80 msgr2=0x7fd7ac0ba580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:07.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac0a3a80 0x7fd7ac0ba580 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fd79c006f20 tx=0x7fd79c00b990).stop 2021-04-09T18:24:07.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 shutdown_connections 2021-04-09T18:24:07.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd794018930 0x7fd79401ae10 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:07.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac0a3a80 0x7fd7ac0ba580 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:07.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7ac0baba0 0x7fd7ac0b8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:07.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 --2- 172.21.15.32:0/766502864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac193e30 0x7fd7ac194230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:07.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 >> 172.21.15.32:0/766502864 conn(0x7fd7ac0a1640 msgr2=0x7fd7ac1b9120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:07.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.911 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 shutdown_connections 2021-04-09T18:24:07.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:07.912 7fd7b3e1a700 1 -- 172.21.15.32:0/766502864 wait complete. 2021-04-09T18:24:07.984 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:24:08.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.079 7f9e93aeb700 1 Processor -- start 2021-04-09T18:24:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.080 7f9e93aeb700 1 -- start start 2021-04-09T18:24:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.080 7f9e93aeb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9e8c10d010 0x7f9e8c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.080 7f9e93aeb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e8c0d1da0 0x7f9e8c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:08.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.080 7f9e93aeb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8c10d480 con 0x7f9e8c0d82c0 2021-04-09T18:24:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.080 7f9e93aeb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8c0d8d30 con 0x7f9e8c0d1da0 2021-04-09T18:24:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.080 7f9e93aeb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8c0d9220 con 0x7f9e8c10d010 2021-04-09T18:24:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.080 7f9e91106700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e8c0d1da0 0x7f9e8c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:08.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e92108700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9e8c0d82c0 0x7f9e8c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60524/0 (socket says 172.21.15.32:60524) 2021-04-09T18:24:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e92108700 1 -- 172.21.15.32:0/4113389320 learned_addr learned my addr 172.21.15.32:0/4113389320 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e91907700 1 --2- 172.21.15.32:0/4113389320 >> v2:172.21.15.86:3300/0 conn(0x7f9e8c10d010 0x7f9e8c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e91106700 1 -- 172.21.15.32:0/4113389320 >> v2:172.21.15.86:3300/0 conn(0x7f9e8c10d010 msgr2=0x7f9e8c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e91106700 1 --2- 172.21.15.32:0/4113389320 >> v2:172.21.15.86:3300/0 conn(0x7f9e8c10d010 0x7f9e8c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e91106700 1 -- 172.21.15.32:0/4113389320 >> v1:172.21.15.32:6789/0 conn(0x7f9e8c0d82c0 legacy=0x7f9e8c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e90905700 1 -- 172.21.15.32:0/4113389320 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611103118 0 0) 0x7f9e84000d30 con 0x7f9e8c0d82c0 2021-04-09T18:24:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e91106700 1 -- 172.21.15.32:0/4113389320 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c0d86f0 con 0x7f9e8c0d1da0 2021-04-09T18:24:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e91106700 1 --2- 172.21.15.32:0/4113389320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0d1da0 0x7f9e8c0d7ca0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f9e7c006650 tx=0x7f9e7c009a80).ready entity=mon.0 client_cookie=136002c4bb2bf179 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:08.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e90905700 1 -- 172.21.15.32:0/4113389320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e7c0009b0 con 0x7f9e8c0d1da0 2021-04-09T18:24:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e90905700 1 -- 172.21.15.32:0/4113389320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c00c0c0 con 0x7f9e8c0d1da0 2021-04-09T18:24:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e90905700 1 -- 172.21.15.32:0/4113389320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e7c00c510 con 0x7f9e8c0d1da0 2021-04-09T18:24:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e93aeb700 1 -- 172.21.15.32:0/4113389320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0d1da0 msgr2=0x7f9e8c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:08.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.081 7f9e93aeb700 1 --2- 172.21.15.32:0/4113389320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0d1da0 0x7f9e8c0d7ca0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f9e7c006650 tx=0x7f9e7c009a80).stop 2021-04-09T18:24:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.082 7f9e93aeb700 1 -- 172.21.15.32:0/4113389320 shutdown_connections 2021-04-09T18:24:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.082 7f9e93aeb700 1 --2- 172.21.15.32:0/4113389320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0d1da0 0x7f9e8c0d7ca0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.082 7f9e93aeb700 1 --2- 172.21.15.32:0/4113389320 >> v2:172.21.15.86:3300/0 conn(0x7f9e8c10d010 0x7f9e8c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.082 7f9e93aeb700 1 -- 172.21.15.32:0/4113389320 >> 172.21.15.32:0/4113389320 conn(0x7f9e8c0ce5f0 msgr2=0x7f9e8c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.082 7f9e93aeb700 1 -- 172.21.15.32:0/4113389320 shutdown_connections 2021-04-09T18:24:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.082 7f9e93aeb700 1 -- 172.21.15.32:0/4113389320 wait complete. 2021-04-09T18:24:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.102 7f9e93aeb700 1 Processor -- start 2021-04-09T18:24:08.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.102 7f9e93aeb700 1 -- start start 2021-04-09T18:24:08.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e93aeb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e8c1a0050 0x7f9e8c1a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e93aeb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e8c0bdcd0 0x7f9e8c0bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e93aeb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0bf080 0x7f9e8c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e93aeb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c0bf500 con 0x7f9e8c0bf080 2021-04-09T18:24:08.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e93aeb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c0bb7b0 con 0x7f9e8c1a0050 2021-04-09T18:24:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e93aeb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c0be0d0 con 0x7f9e8c0bdcd0 2021-04-09T18:24:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e92108700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0bf080 0x7f9e8c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e92108700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0bf080 0x7f9e8c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50942/0 (socket says 172.21.15.32:50942) 2021-04-09T18:24:08.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e92108700 1 -- 172.21.15.32:0/2401163327 learned_addr learned my addr 172.21.15.32:0/2401163327 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e91106700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e8c0bdcd0 0x7f9e8c0bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e91907700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e8c1a0050 0x7f9e8c1a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e92108700 1 -- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e8c0bdcd0 msgr2=0x7f9e8c0bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e92108700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e8c0bdcd0 0x7f9e8c0bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e92108700 1 -- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e8c1a0050 msgr2=0x7f9e8c1a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e92108700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e8c1a0050 0x7f9e8c1a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e92108700 1 -- 172.21.15.32:0/2401163327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c0b1310 con 0x7f9e8c0bf080 2021-04-09T18:24:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.103 7f9e92108700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0bf080 0x7f9e8c0b0cf0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f9e84007f90 tx=0x7f9e8400abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:08.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.104 7f9e827fc700 1 -- 172.21.15.32:0/2401163327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e84000cc0 con 0x7f9e8c0bf080 2021-04-09T18:24:08.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.104 7f9e827fc700 1 -- 172.21.15.32:0/2401163327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840070d0 con 0x7f9e8c0bf080 2021-04-09T18:24:08.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.104 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8c0b1310 con 0x7f9e8c0bf080 2021-04-09T18:24:08.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.104 7f9e827fc700 1 -- 172.21.15.32:0/2401163327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e84007520 con 0x7f9e8c0bf080 2021-04-09T18:24:08.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.104 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8c0b1310 con 0x7f9e8c0bf080 2021-04-09T18:24:08.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.104 7f9e827fc700 1 -- 172.21.15.32:0/2401163327 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9e840072b0 con 0x7f9e8c0bf080 2021-04-09T18:24:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.104 7f9e827fc700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e74018930 0x7f9e7401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.104 7f9e827fc700 1 -- 172.21.15.32:0/2401163327 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9e84032d30 con 0x7f9e8c0bf080 2021-04-09T18:24:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.105 7f9e91907700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e74018930 0x7f9e7401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:08.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.105 7f9e91907700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e74018930 0x7f9e7401ae10 crc :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:08.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.124 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8c093d50 con 0x7f9e8c0bf080 2021-04-09T18:24:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.128 7f9e827fc700 1 -- 172.21.15.32:0/2401163327 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9e84007d00 con 0x7f9e8c0bf080 2021-04-09T18:24:08.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.324 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f9e8c093d50 con 0x7f9e8c0bf080 2021-04-09T18:24:08.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.880 7f9e827fc700 1 -- 172.21.15.32:0/2401163327 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v218) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9e840070d0 con 0x7f9e8c0bf080 2021-04-09T18:24:08.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e74018930 msgr2=0x7f9e7401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:08.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e74018930 0x7f9e7401ae10 crc :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0bf080 msgr2=0x7f9e8c0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:08.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0bf080 0x7f9e8c0b0cf0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f9e84007f90 tx=0x7f9e8400abd0).stop 2021-04-09T18:24:08.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 shutdown_connections 2021-04-09T18:24:08.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e74018930 0x7f9e7401ae10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e8c0bf080 0x7f9e8c0b0cf0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e8c0bdcd0 0x7f9e8c0bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 --2- 172.21.15.32:0/2401163327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e8c1a0050 0x7f9e8c1a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:08.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 >> 172.21.15.32:0/2401163327 conn(0x7f9e8c0975f0 msgr2=0x7f9e8c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:08.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 shutdown_connections 2021-04-09T18:24:08.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:08.892 7f9e93aeb700 1 -- 172.21.15.32:0/2401163327 wait complete. 2021-04-09T18:24:08.964 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:09.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.059 7f0bf88a2700 1 Processor -- start 2021-04-09T18:24:09.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.059 7f0bf88a2700 1 -- start start 2021-04-09T18:24:09.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf88a2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0bf40d6ca0 0x7f0bf40d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:09.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf88a2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0bf40d2460 0x7f0bf410cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:09.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf88a2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf40d72a0 con 0x7f0bf40cffc0 2021-04-09T18:24:09.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf88a2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf40d70a0 con 0x7f0bf40d6ca0 2021-04-09T18:24:09.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf88a2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf40d9200 con 0x7f0bf40d2460 2021-04-09T18:24:09.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf1e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0bf40d6ca0 0x7f0bf40d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:09.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf1e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0bf40d6ca0 0x7f0bf40d8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50948/0 (socket says 172.21.15.32:50948) 2021-04-09T18:24:09.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf1e1b700 1 -- 172.21.15.32:0/2108392746 learned_addr learned my addr 172.21.15.32:0/2108392746 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:09.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf2e1d700 1 --2- 172.21.15.32:0/2108392746 >> v2:172.21.15.86:3300/0 conn(0x7f0bf40d2460 0x7f0bf410cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf1e1b700 1 -- 172.21.15.32:0/2108392746 >> v2:172.21.15.86:3300/0 conn(0x7f0bf40d2460 msgr2=0x7f0bf410cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf1e1b700 1 --2- 172.21.15.32:0/2108392746 >> v2:172.21.15.86:3300/0 conn(0x7f0bf40d2460 0x7f0bf410cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf1e1b700 1 -- 172.21.15.32:0/2108392746 >> v1:172.21.15.86:6789/0 conn(0x7f0bf40cffc0 legacy=0x7f0bf40d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf1e1b700 1 -- 172.21.15.32:0/2108392746 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf40d2860 con 0x7f0bf40d6ca0 2021-04-09T18:24:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf1e1b700 1 --2- 172.21.15.32:0/2108392746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf40d6ca0 0x7f0bf40d8010 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f0bdc006700 tx=0x7f0bdc009b80).ready entity=mon.0 client_cookie=4e951fd70667905a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.060 7f0bf161a700 1 -- 172.21.15.32:0/2108392746 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bdc00c0c0 con 0x7f0bf40d6ca0 2021-04-09T18:24:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf161a700 1 -- 172.21.15.32:0/2108392746 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc00c440 con 0x7f0bf40d6ca0 2021-04-09T18:24:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf161a700 1 -- 172.21.15.32:0/2108392746 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bdc00c920 con 0x7f0bf40d6ca0 2021-04-09T18:24:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf88a2700 1 -- 172.21.15.32:0/2108392746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf40d6ca0 msgr2=0x7f0bf40d8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf88a2700 1 --2- 172.21.15.32:0/2108392746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf40d6ca0 0x7f0bf40d8010 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f0bdc006700 tx=0x7f0bdc009b80).stop 2021-04-09T18:24:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf88a2700 1 -- 172.21.15.32:0/2108392746 shutdown_connections 2021-04-09T18:24:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf88a2700 1 --2- 172.21.15.32:0/2108392746 >> v2:172.21.15.86:3300/0 conn(0x7f0bf40d2460 0x7f0bf410cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf88a2700 1 --2- 172.21.15.32:0/2108392746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf40d6ca0 0x7f0bf40d8010 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf88a2700 1 -- 172.21.15.32:0/2108392746 >> 172.21.15.32:0/2108392746 conn(0x7f0bf40ce5e0 msgr2=0x7f0bf40cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:09.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf88a2700 1 -- 172.21.15.32:0/2108392746 shutdown_connections 2021-04-09T18:24:09.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.061 7f0bf88a2700 1 -- 172.21.15.32:0/2108392746 wait complete. 2021-04-09T18:24:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf88a2700 1 Processor -- start 2021-04-09T18:24:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf88a2700 1 -- start start 2021-04-09T18:24:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf88a2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bf4182e60 0x7f0bf4183260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf88a2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bf409d0f0 0x7f0bf409d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:09.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf88a2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf4181330 0x7f0bf4181730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:09.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf88a2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf409cb70 con 0x7f0bf4181330 2021-04-09T18:24:09.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf88a2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf409db10 con 0x7f0bf409d0f0 2021-04-09T18:24:09.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf88a2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf4181d50 con 0x7f0bf4182e60 2021-04-09T18:24:09.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf2e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf4181330 0x7f0bf4181730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:09.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf2e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf4181330 0x7f0bf4181730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50954/0 (socket says 172.21.15.32:50954) 2021-04-09T18:24:09.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf2e1d700 1 -- 172.21.15.32:0/1286230245 learned_addr learned my addr 172.21.15.32:0/1286230245 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:09.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf1e1b700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bf409d0f0 0x7f0bf409d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:09.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.082 7f0bf261c700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bf4182e60 0x7f0bf4183260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:09.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0bf2e1d700 1 -- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bf4182e60 msgr2=0x7f0bf4183260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:09.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0bf2e1d700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bf4182e60 0x7f0bf4183260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0bf2e1d700 1 -- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bf409d0f0 msgr2=0x7f0bf409d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:09.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0bf2e1d700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bf409d0f0 0x7f0bf409d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0bf2e1d700 1 -- 172.21.15.32:0/1286230245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf4181f70 con 0x7f0bf4181330 2021-04-09T18:24:09.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0bf2e1d700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf4181330 0x7f0bf4181730 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f0bec00b970 tx=0x7f0bec00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:09.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0beaffd700 1 -- 172.21.15.32:0/1286230245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bec000f90 con 0x7f0bf4181330 2021-04-09T18:24:09.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0beaffd700 1 -- 172.21.15.32:0/1286230245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec006aa0 con 0x7f0bf4181330 2021-04-09T18:24:09.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0beaffd700 1 -- 172.21.15.32:0/1286230245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bec0015f0 con 0x7f0bf4181330 2021-04-09T18:24:09.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf4181f70 con 0x7f0bf4181330 2021-04-09T18:24:09.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.083 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf4181f70 con 0x7f0bf4181330 2021-04-09T18:24:09.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.084 7f0beaffd700 1 -- 172.21.15.32:0/1286230245 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0bec000e70 con 0x7f0bf4181330 2021-04-09T18:24:09.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.084 7f0beaffd700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0bd8018930 0x7f0bd801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.084 7f0beaffd700 1 -- 172.21.15.32:0/1286230245 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0bec032d70 con 0x7f0bf4181330 2021-04-09T18:24:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.084 7f0bf261c700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0bd8018930 0x7f0bd801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.084 7f0bf261c700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0bd8018930 0x7f0bd801ae10 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:09.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.104 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf4093df0 con 0x7f0bf4181330 2021-04-09T18:24:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.107 7f0beaffd700 1 -- 172.21.15.32:0/1286230245 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0bec0066f0 con 0x7f0bf4181330 2021-04-09T18:24:09.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.301 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f0bf40d72a0 con 0x7f0bf4181330 2021-04-09T18:24:09.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.888 7f0beaffd700 1 -- 172.21.15.32:0/1286230245 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v220) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0bec006aa0 con 0x7f0bf4181330 2021-04-09T18:24:09.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0bd8018930 msgr2=0x7f0bd801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:09.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0bd8018930 0x7f0bd801ae10 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf4181330 msgr2=0x7f0bf4181730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:09.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf4181330 0x7f0bf4181730 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f0bec00b970 tx=0x7f0bec00bd10).stop 2021-04-09T18:24:09.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 shutdown_connections 2021-04-09T18:24:09.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0bd8018930 0x7f0bd801ae10 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0bf4181330 0x7f0bf4181730 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bf409d0f0 0x7f0bf409d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 --2- 172.21.15.32:0/1286230245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0bf4182e60 0x7f0bf4183260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:09.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.918 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 >> 172.21.15.32:0/1286230245 conn(0x7f0bf409fa00 msgr2=0x7f0bf40b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:09.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.919 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 shutdown_connections 2021-04-09T18:24:09.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:09.919 7f0bf88a2700 1 -- 172.21.15.32:0/1286230245 wait complete. 2021-04-09T18:24:09.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.085 7f65d45fa700 1 Processor -- start 2021-04-09T18:24:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.085 7f65d45fa700 1 -- start start 2021-04-09T18:24:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d45fa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65cc10da00 0x7f65cc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d45fa700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc10d480 con 0x7f65cc10d010 2021-04-09T18:24:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d45fa700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc0d1f70 con 0x7f65cc0d7ca0 2021-04-09T18:24:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d45fa700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc0d8d30 con 0x7f65cc10da00 2021-04-09T18:24:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d2c17700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65cc10da00 0x7f65cc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d2416700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f65cc10d010 0x7f65cc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60550/0 (socket says 172.21.15.32:60550) 2021-04-09T18:24:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d2416700 1 -- 172.21.15.32:0/2153831807 learned_addr learned my addr 172.21.15.32:0/2153831807 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d2c17700 1 -- 172.21.15.32:0/2153831807 >> v1:172.21.15.86:6789/0 conn(0x7f65cc0d7ca0 legacy=0x7f65cc0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d2c17700 1 -- 172.21.15.32:0/2153831807 >> v1:172.21.15.32:6789/0 conn(0x7f65cc10d010 legacy=0x7f65cc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d1414700 1 -- 172.21.15.32:0/2153831807 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652301547 0 0) 0x7f65c8000ce0 con 0x7f65cc10d010 2021-04-09T18:24:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.086 7f65d2c17700 1 -- 172.21.15.32:0/2153831807 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc0d9220 con 0x7f65cc10da00 2021-04-09T18:24:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.087 7f65d2c17700 1 --2- 172.21.15.32:0/2153831807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc10da00 0x7f65cc10de00 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f65c4006e30 tx=0x7f65c4009de0).ready entity=mon.0 client_cookie=387f1d1360a01f63 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.087 7f65d1414700 1 -- 172.21.15.32:0/2153831807 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65c400b300 con 0x7f65cc10da00 2021-04-09T18:24:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.087 7f65d1414700 1 -- 172.21.15.32:0/2153831807 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c400b680 con 0x7f65cc10da00 2021-04-09T18:24:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.087 7f65d1414700 1 -- 172.21.15.32:0/2153831807 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65c400bb60 con 0x7f65cc10da00 2021-04-09T18:24:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.087 7f65d45fa700 1 -- 172.21.15.32:0/2153831807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc10da00 msgr2=0x7f65cc10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.087 7f65d45fa700 1 --2- 172.21.15.32:0/2153831807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc10da00 0x7f65cc10de00 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f65c4006e30 tx=0x7f65c4009de0).stop 2021-04-09T18:24:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.088 7f65d45fa700 1 -- 172.21.15.32:0/2153831807 shutdown_connections 2021-04-09T18:24:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.088 7f65d45fa700 1 --2- 172.21.15.32:0/2153831807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc10da00 0x7f65cc10de00 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.088 7f65d45fa700 1 -- 172.21.15.32:0/2153831807 >> 172.21.15.32:0/2153831807 conn(0x7f65cc0ce5f0 msgr2=0x7f65cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.088 7f65d45fa700 1 -- 172.21.15.32:0/2153831807 shutdown_connections 2021-04-09T18:24:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.088 7f65d45fa700 1 -- 172.21.15.32:0/2153831807 wait complete. 2021-04-09T18:24:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.108 7f65d45fa700 1 Processor -- start 2021-04-09T18:24:10.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.108 7f65d45fa700 1 -- start start 2021-04-09T18:24:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.108 7f65d45fa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc17ff00 0x7f65cc090a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.108 7f65d45fa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc0acef0 0x7f65cc19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.108 7f65d45fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc19b0f0 0x7f65cc0ab890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.108 7f65d45fa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc17f980 con 0x7f65cc19b0f0 2021-04-09T18:24:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.108 7f65d45fa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0aee20 con 0x7f65cc17ff00 2021-04-09T18:24:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.108 7f65d45fa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc091080 con 0x7f65cc0acef0 2021-04-09T18:24:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.108 7f65d2c17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc19b0f0 0x7f65cc0ab890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d2c17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc19b0f0 0x7f65cc0ab890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50968/0 (socket says 172.21.15.32:50968) 2021-04-09T18:24:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d2c17700 1 -- 172.21.15.32:0/4232550638 learned_addr learned my addr 172.21.15.32:0/4232550638 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d2416700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc17ff00 0x7f65cc090a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d1c15700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc0acef0 0x7f65cc19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d2c17700 1 -- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc0acef0 msgr2=0x7f65cc19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d2c17700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc0acef0 0x7f65cc19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d2c17700 1 -- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc17ff00 msgr2=0x7f65cc090a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d2c17700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc17ff00 0x7f65cc090a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d2c17700 1 -- 172.21.15.32:0/4232550638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc19b4f0 con 0x7f65cc19b0f0 2021-04-09T18:24:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d1c15700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc0acef0 0x7f65cc19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d2c17700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc19b0f0 0x7f65cc0ab890 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f65c4009fc0 tx=0x7f65c400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65c2ffd700 1 -- 172.21.15.32:0/4232550638 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65c40093d0 con 0x7f65cc19b0f0 2021-04-09T18:24:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65c2ffd700 1 -- 172.21.15.32:0/4232550638 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c4005820 con 0x7f65cc19b0f0 2021-04-09T18:24:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65cc19b4f0 con 0x7f65cc19b0f0 2021-04-09T18:24:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65c2ffd700 1 -- 172.21.15.32:0/4232550638 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65c4005d80 con 0x7f65cc19b0f0 2021-04-09T18:24:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.109 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc19b4f0 con 0x7f65cc19b0f0 2021-04-09T18:24:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.110 7f65c2ffd700 1 -- 172.21.15.32:0/4232550638 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f65c4006bd0 con 0x7f65cc19b0f0 2021-04-09T18:24:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.110 7f65c2ffd700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b4018930 0x7f65b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.110 7f65c2ffd700 1 -- 172.21.15.32:0/4232550638 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f65c4032dd0 con 0x7f65cc19b0f0 2021-04-09T18:24:10.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.110 7f65d2416700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b4018930 0x7f65b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:10.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.110 7f65d2416700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b4018930 0x7f65b401ae10 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:10.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.130 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc091080 con 0x7f65cc19b0f0 2021-04-09T18:24:10.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.134 7f65c2ffd700 1 -- 172.21.15.32:0/4232550638 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f65c4009740 con 0x7f65cc19b0f0 2021-04-09T18:24:10.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.326 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f65cc091080 con 0x7f65cc19b0f0 2021-04-09T18:24:10.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.889 7f65c2ffd700 1 -- 172.21.15.32:0/4232550638 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v221) v1 ==== 105+0+0 (secure 0 0 0) 0x7f65c4006bd0 con 0x7f65cc19b0f0 2021-04-09T18:24:10.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b4018930 msgr2=0x7f65b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:10.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b4018930 0x7f65b401ae10 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:10.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc19b0f0 msgr2=0x7f65cc0ab890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:10.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc19b0f0 0x7f65cc0ab890 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f65c4009fc0 tx=0x7f65c400bd40).stop 2021-04-09T18:24:10.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 shutdown_connections 2021-04-09T18:24:10.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b4018930 0x7f65b401ae10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:10.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65cc19b0f0 0x7f65cc0ab890 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:10.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65cc0acef0 0x7f65cc19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:10.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 --2- 172.21.15.32:0/4232550638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65cc17ff00 0x7f65cc090a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:10.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 >> 172.21.15.32:0/4232550638 conn(0x7f65cc0bb410 msgr2=0x7f65cc08dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:10.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 shutdown_connections 2021-04-09T18:24:10.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:10.893 7f65d45fa700 1 -- 172.21.15.32:0/4232550638 wait complete. 2021-04-09T18:24:10.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:24:11.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.062 7f16fd2e9700 1 Processor -- start 2021-04-09T18:24:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.062 7f16fd2e9700 1 -- start start 2021-04-09T18:24:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.062 7f16fd2e9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f16f810d010 0x7f16f80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.062 7f16fd2e9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f810d480 con 0x7f16f80d1da0 2021-04-09T18:24:11.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.062 7f16fd2e9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f810e1f0 con 0x7f16f80d8490 2021-04-09T18:24:11.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.062 7f16fd2e9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f80d8d30 con 0x7f16f810d010 2021-04-09T18:24:11.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f707d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f16f810d010 0x7f16f80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:11.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f687c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f16f80d1da0 0x7f16f80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60570/0 (socket says 172.21.15.32:60570) 2021-04-09T18:24:11.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f687c700 1 -- 172.21.15.32:0/268372831 learned_addr learned my addr 172.21.15.32:0/268372831 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:11.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f607b700 1 -- 172.21.15.32:0/268372831 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806031759 0 0) 0x7f16ec000ce0 con 0x7f16f80d1da0 2021-04-09T18:24:11.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f607b700 1 -- 172.21.15.32:0/268372831 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e0002fe0 con 0x7f16f80d1da0 2021-04-09T18:24:11.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f707d700 1 -- 172.21.15.32:0/268372831 >> v1:172.21.15.86:6789/0 conn(0x7f16f80d8490 legacy=0x7f16f810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f707d700 1 -- 172.21.15.32:0/268372831 >> v1:172.21.15.32:6789/0 conn(0x7f16f80d1da0 legacy=0x7f16f80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f707d700 1 -- 172.21.15.32:0/268372831 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80d9220 con 0x7f16f810d010 2021-04-09T18:24:11.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f707d700 1 --2- 172.21.15.32:0/268372831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f810d010 0x7f16f80d1780 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f16e8006650 tx=0x7f16e8009a80).ready entity=mon.0 client_cookie=1e21569ec4f6c785 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:11.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f607b700 1 -- 172.21.15.32:0/268372831 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16e80009b0 con 0x7f16f810d010 2021-04-09T18:24:11.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f607b700 1 -- 172.21.15.32:0/268372831 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e800c0c0 con 0x7f16f810d010 2021-04-09T18:24:11.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16f607b700 1 -- 172.21.15.32:0/268372831 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16e800c520 con 0x7f16f810d010 2021-04-09T18:24:11.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16fd2e9700 1 -- 172.21.15.32:0/268372831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f810d010 msgr2=0x7f16f80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:11.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.063 7f16fd2e9700 1 --2- 172.21.15.32:0/268372831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f810d010 0x7f16f80d1780 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f16e8006650 tx=0x7f16e8009a80).stop 2021-04-09T18:24:11.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.064 7f16fd2e9700 1 -- 172.21.15.32:0/268372831 shutdown_connections 2021-04-09T18:24:11.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.064 7f16fd2e9700 1 --2- 172.21.15.32:0/268372831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f810d010 0x7f16f80d1780 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.064 7f16fd2e9700 1 -- 172.21.15.32:0/268372831 >> 172.21.15.32:0/268372831 conn(0x7f16f80ce5f0 msgr2=0x7f16f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.064 7f16fd2e9700 1 -- 172.21.15.32:0/268372831 shutdown_connections 2021-04-09T18:24:11.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.064 7f16fd2e9700 1 -- 172.21.15.32:0/268372831 wait complete. 2021-04-09T18:24:11.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.084 7f16fd2e9700 1 Processor -- start 2021-04-09T18:24:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.084 7f16fd2e9700 1 -- start start 2021-04-09T18:24:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16fd2e9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f80b7810 0x7f16f80b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16fd2e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80b0900 0x7f16f80b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16fd2e9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f80b1320 0x7f16f8179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:11.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16fd2e9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80b0090 con 0x7f16f80b0900 2021-04-09T18:24:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16fd2e9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8179c30 con 0x7f16f80b7810 2021-04-09T18:24:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16fd2e9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8179e50 con 0x7f16f80b1320 2021-04-09T18:24:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f687c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80b0900 0x7f16f80b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f687c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80b0900 0x7f16f80b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50984/0 (socket says 172.21.15.32:50984) 2021-04-09T18:24:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f687c700 1 -- 172.21.15.32:0/1365156009 learned_addr learned my addr 172.21.15.32:0/1365156009 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f787e700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f80b1320 0x7f16f8179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f707d700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f80b7810 0x7f16f80b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f687c700 1 -- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f80b1320 msgr2=0x7f16f8179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f687c700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f80b1320 0x7f16f8179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f687c700 1 -- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f80b7810 msgr2=0x7f16f80b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f687c700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f80b7810 0x7f16f80b7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f687c700 1 -- 172.21.15.32:0/1365156009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f817a070 con 0x7f16f80b0900 2021-04-09T18:24:11.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.085 7f16f687c700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80b0900 0x7f16f80b0d00 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f16ec00b530 tx=0x7f16ec00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:11.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.086 7f16dffff700 1 -- 172.21.15.32:0/1365156009 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16ec00bcc0 con 0x7f16f80b0900 2021-04-09T18:24:11.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.086 7f16dffff700 1 -- 172.21.15.32:0/1365156009 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ec006030 con 0x7f16f80b0900 2021-04-09T18:24:11.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.086 7f16dffff700 1 -- 172.21.15.32:0/1365156009 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16ec004620 con 0x7f16f80b0900 2021-04-09T18:24:11.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.086 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f817a070 con 0x7f16f80b0900 2021-04-09T18:24:11.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.086 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f817a070 con 0x7f16f80b0900 2021-04-09T18:24:11.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.086 7f16dffff700 1 -- 172.21.15.32:0/1365156009 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f16ec0043a0 con 0x7f16f80b0900 2021-04-09T18:24:11.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.086 7f16dffff700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d8018930 0x7f16d801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:11.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.086 7f16dffff700 1 -- 172.21.15.32:0/1365156009 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f16ec032b60 con 0x7f16f80b0900 2021-04-09T18:24:11.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.087 7f16f707d700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d8018930 0x7f16d801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:11.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.087 7f16f707d700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d8018930 0x7f16d801ae10 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:11.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.107 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f808dac0 con 0x7f16f80b0900 2021-04-09T18:24:11.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.110 7f16dffff700 1 -- 172.21.15.32:0/1365156009 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f16ec0009b0 con 0x7f16f80b0900 2021-04-09T18:24:11.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.304 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f16f810d480 con 0x7f16f80b0900 2021-04-09T18:24:12.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.968 7f16dffff700 1 -- 172.21.15.32:0/1365156009 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v222) v1 ==== 105+0+0 (secure 0 0 0) 0x7f16ec0043a0 con 0x7f16f80b0900 2021-04-09T18:24:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d8018930 msgr2=0x7f16d801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d8018930 0x7f16d801ae10 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80b0900 msgr2=0x7f16f80b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80b0900 0x7f16f80b0d00 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f16ec00b530 tx=0x7f16ec00b8d0).stop 2021-04-09T18:24:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 shutdown_connections 2021-04-09T18:24:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d8018930 0x7f16d801ae10 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80b0900 0x7f16f80b0d00 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f80b1320 0x7f16f8179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 --2- 172.21.15.32:0/1365156009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f80b7810 0x7f16f80b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 >> 172.21.15.32:0/1365156009 conn(0x7f16f80bb410 msgr2=0x7f16f80a13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:12.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 shutdown_connections 2021-04-09T18:24:12.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:11.972 7f16fd2e9700 1 -- 172.21.15.32:0/1365156009 wait complete. 2021-04-09T18:24:12.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.136 7f192f7df700 1 Processor -- start 2021-04-09T18:24:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.136 7f192f7df700 1 -- start start 2021-04-09T18:24:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192f7df700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19280d7ca0 0x7f19280d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192f7df700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f192810da00 0x7f192810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192f7df700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f192810d480 con 0x7f192810d010 2021-04-09T18:24:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192f7df700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19280d9220 con 0x7f19280d7ca0 2021-04-09T18:24:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192f7df700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19280d86c0 con 0x7f192810da00 2021-04-09T18:24:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192cdfa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19280d7ca0 0x7f19280d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:12.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192cdfa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19280d7ca0 0x7f19280d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50996/0 (socket says 172.21.15.32:50996) 2021-04-09T18:24:12.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192ddfc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f192810da00 0x7f192810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:12.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192cdfa700 1 -- 172.21.15.32:0/3722825129 learned_addr learned my addr 172.21.15.32:0/3722825129 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:12.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192cdfa700 1 -- 172.21.15.32:0/3722825129 >> v2:172.21.15.86:3300/0 conn(0x7f192810da00 msgr2=0x7f192810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192cdfa700 1 --2- 172.21.15.32:0/3722825129 >> v2:172.21.15.86:3300/0 conn(0x7f192810da00 0x7f192810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192cdfa700 1 -- 172.21.15.32:0/3722825129 >> v1:172.21.15.86:6789/0 conn(0x7f192810d010 legacy=0x7f19280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192cdfa700 1 -- 172.21.15.32:0/3722825129 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280d1f70 con 0x7f19280d7ca0 2021-04-09T18:24:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192cdfa700 1 --2- 172.21.15.32:0/3722825129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19280d7ca0 0x7f19280d80a0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f191c006700 tx=0x7f191c009b80).ready entity=mon.0 client_cookie=c5dace2fa8c543ab server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f1927fff700 1 -- 172.21.15.32:0/3722825129 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f191c00c0c0 con 0x7f19280d7ca0 2021-04-09T18:24:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f1927fff700 1 -- 172.21.15.32:0/3722825129 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191c00c440 con 0x7f19280d7ca0 2021-04-09T18:24:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f1927fff700 1 -- 172.21.15.32:0/3722825129 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f191c00c920 con 0x7f19280d7ca0 2021-04-09T18:24:12.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192f7df700 1 -- 172.21.15.32:0/3722825129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19280d7ca0 msgr2=0x7f19280d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:12.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.137 7f192f7df700 1 --2- 172.21.15.32:0/3722825129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19280d7ca0 0x7f19280d80a0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f191c006700 tx=0x7f191c009b80).stop 2021-04-09T18:24:12.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.138 7f192f7df700 1 -- 172.21.15.32:0/3722825129 shutdown_connections 2021-04-09T18:24:12.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.138 7f192f7df700 1 --2- 172.21.15.32:0/3722825129 >> v2:172.21.15.86:3300/0 conn(0x7f192810da00 0x7f192810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.138 7f192f7df700 1 --2- 172.21.15.32:0/3722825129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19280d7ca0 0x7f19280d80a0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.138 7f192f7df700 1 -- 172.21.15.32:0/3722825129 >> 172.21.15.32:0/3722825129 conn(0x7f19280ce5f0 msgr2=0x7f19280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.138 7f192f7df700 1 -- 172.21.15.32:0/3722825129 shutdown_connections 2021-04-09T18:24:12.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.138 7f192f7df700 1 -- 172.21.15.32:0/3722825129 wait complete. 2021-04-09T18:24:12.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.158 7f192f7df700 1 Processor -- start 2021-04-09T18:24:12.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192f7df700 1 -- start start 2021-04-09T18:24:12.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192f7df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19281a0050 0x7f19281a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:12.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192f7df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19280bdcd0 0x7f19280bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:12.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192f7df700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19280bf080 0x7f19280b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:12.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192f7df700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19280bf500 con 0x7f19281a0050 2021-04-09T18:24:12.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192f7df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19280bb7b0 con 0x7f19280bdcd0 2021-04-09T18:24:12.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192f7df700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19280be0d0 con 0x7f19280bf080 2021-04-09T18:24:12.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192d5fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19281a0050 0x7f19281a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192d5fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19281a0050 0x7f19281a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51002/0 (socket says 172.21.15.32:51002) 2021-04-09T18:24:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192d5fb700 1 -- 172.21.15.32:0/2209693169 learned_addr learned my addr 172.21.15.32:0/2209693169 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192cdfa700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19280bdcd0 0x7f19280bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.159 7f192ddfc700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19280bf080 0x7f19280b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:12.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192cdfa700 1 -- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19280bf080 msgr2=0x7f19280b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:12.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192cdfa700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19280bf080 0x7f19280b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192cdfa700 1 -- 172.21.15.32:0/2209693169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19281a0050 msgr2=0x7f19281a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192cdfa700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19281a0050 0x7f19281a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192cdfa700 1 -- 172.21.15.32:0/2209693169 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280b1310 con 0x7f19280bdcd0 2021-04-09T18:24:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192ddfc700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19280bf080 0x7f19280b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192d5fb700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19281a0050 0x7f19281a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192cdfa700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19280bdcd0 0x7f19280bea60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f191c006f20 tx=0x7f191c00b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f1925ffb700 1 -- 172.21.15.32:0/2209693169 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f191c0064e0 con 0x7f19280bdcd0 2021-04-09T18:24:12.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f1925ffb700 1 -- 172.21.15.32:0/2209693169 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191c00bd50 con 0x7f19280bdcd0 2021-04-09T18:24:12.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192f7df700 1 -- 172.21.15.32:0/2209693169 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19280b1310 con 0x7f19280bdcd0 2021-04-09T18:24:12.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f1925ffb700 1 -- 172.21.15.32:0/2209693169 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f191c0052e0 con 0x7f19280bdcd0 2021-04-09T18:24:12.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.160 7f192f7df700 1 -- 172.21.15.32:0/2209693169 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19280b1310 con 0x7f19280bdcd0 2021-04-09T18:24:12.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.161 7f1925ffb700 1 -- 172.21.15.32:0/2209693169 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f191c006030 con 0x7f19280bdcd0 2021-04-09T18:24:12.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.161 7f1925ffb700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1908018930 0x7f190801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:12.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.161 7f1925ffb700 1 -- 172.21.15.32:0/2209693169 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f191c032ac0 con 0x7f19280bdcd0 2021-04-09T18:24:12.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.161 7f192d5fb700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1908018930 0x7f190801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.161 7f192d5fb700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1908018930 0x7f190801adc0 crc :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:12.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.181 7f192f7df700 1 -- 172.21.15.32:0/2209693169 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19280bb7b0 con 0x7f19280bdcd0 2021-04-09T18:24:12.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.185 7f1925ffb700 1 -- 172.21.15.32:0/2209693169 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f191c0091d0 con 0x7f19280bdcd0 2021-04-09T18:24:12.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.378 7f192f7df700 1 -- 172.21.15.32:0/2209693169 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f19280bb7b0 con 0x7f19280bdcd0 2021-04-09T18:24:13.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.976 7f1925ffb700 1 -- 172.21.15.32:0/2209693169 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v223) v1 ==== 105+0+0 (secure 0 0 0) 0x7f191c006030 con 0x7f19280bdcd0 2021-04-09T18:24:13.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 -- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1908018930 msgr2=0x7f190801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1908018930 0x7f190801adc0 crc :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 -- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19280bdcd0 msgr2=0x7f19280bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19280bdcd0 0x7f19280bea60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f191c006f20 tx=0x7f191c00b8e0).stop 2021-04-09T18:24:13.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 -- 172.21.15.32:0/2209693169 shutdown_connections 2021-04-09T18:24:13.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1908018930 0x7f190801adc0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:13.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19280bf080 0x7f19280b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:13.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19280bdcd0 0x7f19280bea60 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:13.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 --2- 172.21.15.32:0/2209693169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19281a0050 0x7f19281a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:13.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 -- 172.21.15.32:0/2209693169 >> 172.21.15.32:0/2209693169 conn(0x7f19280975f0 msgr2=0x7f1928096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:13.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 -- 172.21.15.32:0/2209693169 shutdown_connections 2021-04-09T18:24:13.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:12.995 7f192f7df700 1 -- 172.21.15.32:0/2209693169 wait complete. 2021-04-09T18:24:13.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:13.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.167 7fb33906c700 1 Processor -- start 2021-04-09T18:24:13.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.167 7fb33906c700 1 -- start start 2021-04-09T18:24:13.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33906c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340d7ca0 0x7fb3340d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:13.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33906c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33410d480 con 0x7fb33410da00 2021-04-09T18:24:13.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33906c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3340d86c0 con 0x7fb33410d010 2021-04-09T18:24:13.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33906c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3340d9220 con 0x7fb3340d7ca0 2021-04-09T18:24:13.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33359e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb33410da00 0x7fb33410de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60598/0 (socket says 172.21.15.32:60598) 2021-04-09T18:24:13.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33359e700 1 -- 172.21.15.32:0/1521842566 learned_addr learned my addr 172.21.15.32:0/1521842566 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:13.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33259c700 1 --2- 172.21.15.32:0/1521842566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340d7ca0 0x7fb3340d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:13.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb331d9b700 1 -- 172.21.15.32:0/1521842566 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704723811 0 0) 0x7fb32c000d30 con 0x7fb33410da00 2021-04-09T18:24:13.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb331d9b700 1 -- 172.21.15.32:0/1521842566 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb31c002fe0 con 0x7fb33410da00 2021-04-09T18:24:13.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb331d9b700 1 -- 172.21.15.32:0/1521842566 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433807889 0 0) 0x7fb324001080 con 0x7fb33410d010 2021-04-09T18:24:13.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb331d9b700 1 -- 172.21.15.32:0/1521842566 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb31c003210 con 0x7fb33410d010 2021-04-09T18:24:13.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb331d9b700 1 -- 172.21.15.32:0/1521842566 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3718360480 0 0) 0x7fb32c001460 con 0x7fb33410da00 2021-04-09T18:24:13.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33259c700 1 -- 172.21.15.32:0/1521842566 >> v1:172.21.15.86:6789/0 conn(0x7fb33410d010 legacy=0x7fb3340d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:13.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33259c700 1 -- 172.21.15.32:0/1521842566 >> v1:172.21.15.32:6789/0 conn(0x7fb33410da00 legacy=0x7fb33410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:13.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.168 7fb33259c700 1 -- 172.21.15.32:0/1521842566 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334112100 con 0x7fb3340d7ca0 2021-04-09T18:24:13.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.169 7fb33259c700 1 --2- 172.21.15.32:0/1521842566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340d7ca0 0x7fb3340d80a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb328006650 tx=0x7fb328009a00).ready entity=mon.2 client_cookie=e3b3941d36b76315 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:13.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.169 7fb331d9b700 1 -- 172.21.15.32:0/1521842566 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb328009df0 con 0x7fb3340d7ca0 2021-04-09T18:24:13.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.169 7fb331d9b700 1 -- 172.21.15.32:0/1521842566 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3280009b0 con 0x7fb3340d7ca0 2021-04-09T18:24:13.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.169 7fb331d9b700 1 -- 172.21.15.32:0/1521842566 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb32800c2b0 con 0x7fb3340d7ca0 2021-04-09T18:24:13.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.169 7fb33906c700 1 -- 172.21.15.32:0/1521842566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340d7ca0 msgr2=0x7fb3340d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:13.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.169 7fb33906c700 1 --2- 172.21.15.32:0/1521842566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340d7ca0 0x7fb3340d80a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb328006650 tx=0x7fb328009a00).stop 2021-04-09T18:24:13.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.170 7fb33906c700 1 -- 172.21.15.32:0/1521842566 shutdown_connections 2021-04-09T18:24:13.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.170 7fb33906c700 1 --2- 172.21.15.32:0/1521842566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340d7ca0 0x7fb3340d80a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:13.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.170 7fb33906c700 1 -- 172.21.15.32:0/1521842566 >> 172.21.15.32:0/1521842566 conn(0x7fb3340ce5f0 msgr2=0x7fb3340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:13.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.170 7fb33906c700 1 -- 172.21.15.32:0/1521842566 shutdown_connections 2021-04-09T18:24:13.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.170 7fb33906c700 1 -- 172.21.15.32:0/1521842566 wait complete. 2021-04-09T18:24:13.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33906c700 1 Processor -- start 2021-04-09T18:24:13.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33906c700 1 -- start start 2021-04-09T18:24:13.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33906c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3340b1680 0x7fb3340b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:13.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33906c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb33409e330 0x7fb3340b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:13.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33906c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340b2190 0x7fb33409f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:13.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33906c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33409e730 con 0x7fb33409e330 2021-04-09T18:24:13.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33906c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33409f6c0 con 0x7fb3340b1680 2021-04-09T18:24:13.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33906c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33409f8e0 con 0x7fb3340b2190 2021-04-09T18:24:13.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb33409e330 0x7fb3340b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:13.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb33409e330 0x7fb3340b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51014/0 (socket says 172.21.15.32:51014) 2021-04-09T18:24:13.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33259c700 1 -- 172.21.15.32:0/312953132 learned_addr learned my addr 172.21.15.32:0/312953132 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:13.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb33359e700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340b2190 0x7fb33409f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:13.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.190 7fb332d9d700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3340b1680 0x7fb3340b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:13.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb33259c700 1 -- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340b2190 msgr2=0x7fb33409f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:13.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb33259c700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340b2190 0x7fb33409f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:13.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb33259c700 1 -- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3340b1680 msgr2=0x7fb3340b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:13.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb33259c700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3340b1680 0x7fb3340b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:13.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb33259c700 1 -- 172.21.15.32:0/312953132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33409fa90 con 0x7fb33409e330 2021-04-09T18:24:13.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb33259c700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb33409e330 0x7fb3340b1b70 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7fb3280096a0 tx=0x7fb32800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:13.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb3237fe700 1 -- 172.21.15.32:0/312953132 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb328000c90 con 0x7fb33409e330 2021-04-09T18:24:13.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb3237fe700 1 -- 172.21.15.32:0/312953132 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb328002030 con 0x7fb33409e330 2021-04-09T18:24:13.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb3237fe700 1 -- 172.21.15.32:0/312953132 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb328002550 con 0x7fb33409e330 2021-04-09T18:24:13.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb33906c700 1 -- 172.21.15.32:0/312953132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33409f8e0 con 0x7fb33409e330 2021-04-09T18:24:13.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.191 7fb33906c700 1 -- 172.21.15.32:0/312953132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33409f8e0 con 0x7fb33409e330 2021-04-09T18:24:13.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.192 7fb3237fe700 1 -- 172.21.15.32:0/312953132 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb328002210 con 0x7fb33409e330 2021-04-09T18:24:13.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.192 7fb3237fe700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3180188c0 0x7fb31801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:13.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.192 7fb3237fe700 1 -- 172.21.15.32:0/312953132 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb328032a70 con 0x7fb33409e330 2021-04-09T18:24:13.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.192 7fb332d9d700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3180188c0 0x7fb31801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:13.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.192 7fb332d9d700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3180188c0 0x7fb31801adc0 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:13.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.212 7fb33906c700 1 -- 172.21.15.32:0/312953132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33409eb70 con 0x7fb33409e330 2021-04-09T18:24:13.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.215 7fb3237fe700 1 -- 172.21.15.32:0/312953132 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb3280009a0 con 0x7fb33409e330 2021-04-09T18:24:13.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.411 7fb33906c700 1 -- 172.21.15.32:0/312953132 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7fb33409eb70 con 0x7fb33409e330 2021-04-09T18:24:14.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:13.980 7fb3237fe700 1 -- 172.21.15.32:0/312953132 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v224) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb328002010 con 0x7fb33409e330 2021-04-09T18:24:14.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 -- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3180188c0 msgr2=0x7fb31801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:14.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3180188c0 0x7fb31801adc0 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 -- 172.21.15.32:0/312953132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb33409e330 msgr2=0x7fb3340b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:14.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb33409e330 0x7fb3340b1b70 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7fb3280096a0 tx=0x7fb32800b990).stop 2021-04-09T18:24:14.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 -- 172.21.15.32:0/312953132 shutdown_connections 2021-04-09T18:24:14.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3180188c0 0x7fb31801adc0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb33409e330 0x7fb3340b1b70 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3340b2190 0x7fb33409f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 --2- 172.21.15.32:0/312953132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3340b1680 0x7fb3340b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 -- 172.21.15.32:0/312953132 >> 172.21.15.32:0/312953132 conn(0x7fb3340c4290 msgr2=0x7fb3340c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:14.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 -- 172.21.15.32:0/312953132 shutdown_connections 2021-04-09T18:24:14.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.028 7fb33906c700 1 -- 172.21.15.32:0/312953132 wait complete. 2021-04-09T18:24:14.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:24:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.191 7fa7fbcb7700 1 Processor -- start 2021-04-09T18:24:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.191 7fa7fbcb7700 1 -- start start 2021-04-09T18:24:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fbcb7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f410d010 0x7fa7f40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fbcb7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7f40d8490 0x7fa7f410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fbcb7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f410d480 con 0x7fa7f40d1da0 2021-04-09T18:24:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fbcb7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f410e780 con 0x7fa7f40d8490 2021-04-09T18:24:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fbcb7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f40d8d30 con 0x7fa7f410d010 2021-04-09T18:24:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fa2d4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa7f40d8490 0x7fa7f410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7f92d2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa7f40d1da0 0x7fa7f40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60614/0 (socket says 172.21.15.32:60614) 2021-04-09T18:24:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7f92d2700 1 -- 172.21.15.32:0/4213274502 learned_addr learned my addr 172.21.15.32:0/4213274502 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7f9ad3700 1 --2- 172.21.15.32:0/4213274502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f410d010 0x7fa7f40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fa2d4700 1 -- 172.21.15.32:0/4213274502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f410d010 msgr2=0x7fa7f40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fa2d4700 1 --2- 172.21.15.32:0/4213274502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f410d010 0x7fa7f40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fa2d4700 1 -- 172.21.15.32:0/4213274502 >> v1:172.21.15.32:6789/0 conn(0x7fa7f40d1da0 legacy=0x7fa7f40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7f8ad1700 1 -- 172.21.15.32:0/4213274502 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339079515 0 0) 0x7fa7e4000ce0 con 0x7fa7f40d1da0 2021-04-09T18:24:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.192 7fa7fa2d4700 1 -- 172.21.15.32:0/4213274502 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f40d9220 con 0x7fa7f40d8490 2021-04-09T18:24:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.193 7fa7fa2d4700 1 --2- 172.21.15.32:0/4213274502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f40d8490 0x7fa7f410da00 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7fa7ec006e30 tx=0x7fa7ec009de0).ready entity=mon.0 client_cookie=82e4a13d549b2470 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.193 7fa7f8ad1700 1 -- 172.21.15.32:0/4213274502 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7ec00b300 con 0x7fa7f40d8490 2021-04-09T18:24:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.193 7fa7f8ad1700 1 -- 172.21.15.32:0/4213274502 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ec00b680 con 0x7fa7f40d8490 2021-04-09T18:24:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.193 7fa7f8ad1700 1 -- 172.21.15.32:0/4213274502 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7ec00bb60 con 0x7fa7f40d8490 2021-04-09T18:24:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.193 7fa7fbcb7700 1 -- 172.21.15.32:0/4213274502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f40d8490 msgr2=0x7fa7f410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.193 7fa7fbcb7700 1 --2- 172.21.15.32:0/4213274502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f40d8490 0x7fa7f410da00 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7fa7ec006e30 tx=0x7fa7ec009de0).stop 2021-04-09T18:24:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.194 7fa7fbcb7700 1 -- 172.21.15.32:0/4213274502 shutdown_connections 2021-04-09T18:24:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.194 7fa7fbcb7700 1 --2- 172.21.15.32:0/4213274502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f40d8490 0x7fa7f410da00 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.194 7fa7fbcb7700 1 --2- 172.21.15.32:0/4213274502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f410d010 0x7fa7f40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.194 7fa7fbcb7700 1 -- 172.21.15.32:0/4213274502 >> 172.21.15.32:0/4213274502 conn(0x7fa7f40ce5f0 msgr2=0x7fa7f40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.194 7fa7fbcb7700 1 -- 172.21.15.32:0/4213274502 shutdown_connections 2021-04-09T18:24:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.194 7fa7fbcb7700 1 -- 172.21.15.32:0/4213274502 wait complete. 2021-04-09T18:24:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fbcb7700 1 Processor -- start 2021-04-09T18:24:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fbcb7700 1 -- start start 2021-04-09T18:24:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fbcb7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f40a0a10 0x7fa7f40bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fbcb7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7f418ad00 0x7fa7f419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fbcb7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f41a03f0 0x7fa7f4095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fbcb7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f418b100 con 0x7fa7f41a03f0 2021-04-09T18:24:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fbcb7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f4095860 con 0x7fa7f418ad00 2021-04-09T18:24:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fbcb7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f4095a80 con 0x7fa7f40a0a10 2021-04-09T18:24:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fa2d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f41a03f0 0x7fa7f4095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fa2d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f41a03f0 0x7fa7f4095240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51026/0 (socket says 172.21.15.32:51026) 2021-04-09T18:24:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.214 7fa7fa2d4700 1 -- 172.21.15.32:0/2819672245 learned_addr learned my addr 172.21.15.32:0/2819672245 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7f9ad3700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f40a0a10 0x7fa7f40bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7f92d2700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7f418ad00 0x7fa7f419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7fa2d4700 1 -- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f40a0a10 msgr2=0x7fa7f40bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7fa2d4700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f40a0a10 0x7fa7f40bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7fa2d4700 1 -- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7f418ad00 msgr2=0x7fa7f419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7fa2d4700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7f418ad00 0x7fa7f419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7fa2d4700 1 -- 172.21.15.32:0/2819672245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f4189250 con 0x7fa7f41a03f0 2021-04-09T18:24:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7f9ad3700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f40a0a10 0x7fa7f40bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7f92d2700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7f418ad00 0x7fa7f419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7fa2d4700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f41a03f0 0x7fa7f4095240 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7fa7ec009fc0 tx=0x7fa7ec00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7ea7fc700 1 -- 172.21.15.32:0/2819672245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7ec0093d0 con 0x7fa7f41a03f0 2021-04-09T18:24:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7ea7fc700 1 -- 172.21.15.32:0/2819672245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ec009690 con 0x7fa7f41a03f0 2021-04-09T18:24:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7ea7fc700 1 -- 172.21.15.32:0/2819672245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7ec001670 con 0x7fa7f41a03f0 2021-04-09T18:24:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f4189250 con 0x7fa7f41a03f0 2021-04-09T18:24:14.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.215 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f4189250 con 0x7fa7f41a03f0 2021-04-09T18:24:14.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.216 7fa7ea7fc700 1 -- 172.21.15.32:0/2819672245 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa7ec001270 con 0x7fa7f41a03f0 2021-04-09T18:24:14.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.216 7fa7ea7fc700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7dc018900 0x7fa7dc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:14.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.216 7fa7ea7fc700 1 -- 172.21.15.32:0/2819672245 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa7ec032d70 con 0x7fa7f41a03f0 2021-04-09T18:24:14.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.216 7fa7f9ad3700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7dc018900 0x7fa7dc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:14.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.216 7fa7f9ad3700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7dc018900 0x7fa7dc01adc0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:14.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.236 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f4093d50 con 0x7fa7f41a03f0 2021-04-09T18:24:14.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.240 7fa7ea7fc700 1 -- 172.21.15.32:0/2819672245 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa7ec0093b0 con 0x7fa7f41a03f0 2021-04-09T18:24:14.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.432 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7fa7f410d480 con 0x7fa7f41a03f0 2021-04-09T18:24:15.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.987 7fa7ea7fc700 1 -- 172.21.15.32:0/2819672245 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v225) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa7ec01fab0 con 0x7fa7f41a03f0 2021-04-09T18:24:15.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7dc018900 msgr2=0x7fa7dc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:15.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7dc018900 0x7fa7dc01adc0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f41a03f0 msgr2=0x7fa7f4095240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f41a03f0 0x7fa7f4095240 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7fa7ec009fc0 tx=0x7fa7ec00bd40).stop 2021-04-09T18:24:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 shutdown_connections 2021-04-09T18:24:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa7dc018900 0x7fa7dc01adc0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7f41a03f0 0x7fa7f4095240 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7f418ad00 0x7fa7f419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 --2- 172.21.15.32:0/2819672245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7f40a0a10 0x7fa7f40bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 >> 172.21.15.32:0/2819672245 conn(0x7fa7f40975f0 msgr2=0x7fa7f4096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 shutdown_connections 2021-04-09T18:24:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:14.998 7fa7fbcb7700 1 -- 172.21.15.32:0/2819672245 wait complete. 2021-04-09T18:24:15.070 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:15.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.162 7fd84dec1700 1 Processor -- start 2021-04-09T18:24:15.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.162 7fd84dec1700 1 -- start start 2021-04-09T18:24:15.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.162 7fd84dec1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd84810d010 0x7fd8480d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:15.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.162 7fd84dec1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8480d8490 0x7fd84810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:15.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.162 7fd84dec1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84810d480 con 0x7fd8480d1da0 2021-04-09T18:24:15.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.162 7fd84dec1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8480d8d30 con 0x7fd84810d010 2021-04-09T18:24:15.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.162 7fd84dec1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480d9220 con 0x7fd8480d8490 2021-04-09T18:24:15.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd846ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd8480d1da0 0x7fd8480d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60628/0 (socket says 172.21.15.32:60628) 2021-04-09T18:24:15.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd846ffd700 1 -- 172.21.15.32:0/1643080158 learned_addr learned my addr 172.21.15.32:0/1643080158 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:15.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd847fff700 1 --2- 172.21.15.32:0/1643080158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8480d8490 0x7fd84810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:15.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8477fe700 1 --2- 172.21.15.32:0/1643080158 >> v2:172.21.15.86:3300/0 conn(0x7fd84810d010 0x7fd8480d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:15.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8467fc700 1 -- 172.21.15.32:0/1643080158 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050365319 0 0) 0x7fd83c000ce0 con 0x7fd8480d1da0 2021-04-09T18:24:15.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8467fc700 1 -- 172.21.15.32:0/1643080158 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd830002fe0 con 0x7fd8480d1da0 2021-04-09T18:24:15.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8477fe700 1 -- 172.21.15.32:0/1643080158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8480d8490 msgr2=0x7fd84810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:15.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8477fe700 1 --2- 172.21.15.32:0/1643080158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8480d8490 0x7fd84810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8477fe700 1 -- 172.21.15.32:0/1643080158 >> v1:172.21.15.32:6789/0 conn(0x7fd8480d1da0 legacy=0x7fd8480d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:15.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8467fc700 1 -- 172.21.15.32:0/1643080158 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2562330757 0 0) 0x7fd83c0010e0 con 0x7fd8480d1da0 2021-04-09T18:24:15.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8477fe700 1 -- 172.21.15.32:0/1643080158 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84810e020 con 0x7fd84810d010 2021-04-09T18:24:15.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8477fe700 1 --2- 172.21.15.32:0/1643080158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84810d010 0x7fd8480d1780 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fd838006650 tx=0x7fd838009a00).ready entity=mon.1 client_cookie=ac3d8da549489b5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:15.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.163 7fd8467fc700 1 -- 172.21.15.32:0/1643080158 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd838009df0 con 0x7fd84810d010 2021-04-09T18:24:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd8467fc700 1 -- 172.21.15.32:0/1643080158 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8380009b0 con 0x7fd84810d010 2021-04-09T18:24:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd8467fc700 1 -- 172.21.15.32:0/1643080158 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd83800c2b0 con 0x7fd84810d010 2021-04-09T18:24:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd84dec1700 1 -- 172.21.15.32:0/1643080158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84810d010 msgr2=0x7fd8480d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:15.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd84dec1700 1 --2- 172.21.15.32:0/1643080158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84810d010 0x7fd8480d1780 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fd838006650 tx=0x7fd838009a00).stop 2021-04-09T18:24:15.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd84dec1700 1 -- 172.21.15.32:0/1643080158 shutdown_connections 2021-04-09T18:24:15.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd84dec1700 1 --2- 172.21.15.32:0/1643080158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8480d8490 0x7fd84810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd84dec1700 1 --2- 172.21.15.32:0/1643080158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84810d010 0x7fd8480d1780 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd84dec1700 1 -- 172.21.15.32:0/1643080158 >> 172.21.15.32:0/1643080158 conn(0x7fd8480ce5f0 msgr2=0x7fd8480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd84dec1700 1 -- 172.21.15.32:0/1643080158 shutdown_connections 2021-04-09T18:24:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.164 7fd84dec1700 1 -- 172.21.15.32:0/1643080158 wait complete. 2021-04-09T18:24:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.184 7fd84dec1700 1 Processor -- start 2021-04-09T18:24:15.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd84dec1700 1 -- start start 2021-04-09T18:24:15.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd84dec1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480a0a10 0x7fd8480bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:15.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd84dec1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd84818ad00 0x7fd84819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd84dec1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8481a03f0 0x7fd848095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd84dec1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84818b100 con 0x7fd8480a0a10 2021-04-09T18:24:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd84dec1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd848095860 con 0x7fd8481a03f0 2021-04-09T18:24:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd84dec1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd848095a80 con 0x7fd84818ad00 2021-04-09T18:24:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd8477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480a0a10 0x7fd8480bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd8477fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480a0a10 0x7fd8480bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51040/0 (socket says 172.21.15.32:51040) 2021-04-09T18:24:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd8477fe700 1 -- 172.21.15.32:0/1908294447 learned_addr learned my addr 172.21.15.32:0/1908294447 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd847fff700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8481a03f0 0x7fd848095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.185 7fd846ffd700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd84818ad00 0x7fd84819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:15.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd8477fe700 1 -- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd84818ad00 msgr2=0x7fd84819fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:15.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd8477fe700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd84818ad00 0x7fd84819fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd8477fe700 1 -- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8481a03f0 msgr2=0x7fd848095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:15.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd8477fe700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8481a03f0 0x7fd848095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd8477fe700 1 -- 172.21.15.32:0/1908294447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd848189250 con 0x7fd8480a0a10 2021-04-09T18:24:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd8477fe700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480a0a10 0x7fd8480bf870 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7fd8380096a0 tx=0x7fd83800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd82ffff700 1 -- 172.21.15.32:0/1908294447 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8380093a0 con 0x7fd8480a0a10 2021-04-09T18:24:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd82ffff700 1 -- 172.21.15.32:0/1908294447 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8380067f0 con 0x7fd8480a0a10 2021-04-09T18:24:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd848189250 con 0x7fd8480a0a10 2021-04-09T18:24:15.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd82ffff700 1 -- 172.21.15.32:0/1908294447 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8380022b0 con 0x7fd8480a0a10 2021-04-09T18:24:15.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.186 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd848189250 con 0x7fd8480a0a10 2021-04-09T18:24:15.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.187 7fd82ffff700 1 -- 172.21.15.32:0/1908294447 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd8380052e0 con 0x7fd8480a0a10 2021-04-09T18:24:15.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.187 7fd82ffff700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8280188d0 0x7fd82801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:15.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.187 7fd82ffff700 1 -- 172.21.15.32:0/1908294447 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd8380329f0 con 0x7fd8480a0a10 2021-04-09T18:24:15.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.187 7fd846ffd700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8280188d0 0x7fd82801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:15.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.187 7fd846ffd700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8280188d0 0x7fd82801add0 crc :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:15.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.206 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd848093d50 con 0x7fd8480a0a10 2021-04-09T18:24:15.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.210 7fd82ffff700 1 -- 172.21.15.32:0/1908294447 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd83801f730 con 0x7fd8480a0a10 2021-04-09T18:24:15.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.406 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fd84810d480 con 0x7fd8480a0a10 2021-04-09T18:24:16.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:15.989 7fd82ffff700 1 -- 172.21.15.32:0/1908294447 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v226) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd8380052e0 con 0x7fd8480a0a10 2021-04-09T18:24:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.023 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8280188d0 msgr2=0x7fd82801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.023 7fd84dec1700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8280188d0 0x7fd82801add0 crc :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.023 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480a0a10 msgr2=0x7fd8480bf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.023 7fd84dec1700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480a0a10 0x7fd8480bf870 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7fd8380096a0 tx=0x7fd83800b8e0).stop 2021-04-09T18:24:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.024 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 shutdown_connections 2021-04-09T18:24:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.024 7fd84dec1700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8280188d0 0x7fd82801add0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.024 7fd84dec1700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8481a03f0 0x7fd848095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.024 7fd84dec1700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd84818ad00 0x7fd84819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.024 7fd84dec1700 1 --2- 172.21.15.32:0/1908294447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8480a0a10 0x7fd8480bf870 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.024 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 >> 172.21.15.32:0/1908294447 conn(0x7fd8480975f0 msgr2=0x7fd848096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.024 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 shutdown_connections 2021-04-09T18:24:16.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.024 7fd84dec1700 1 -- 172.21.15.32:0/1908294447 wait complete. 2021-04-09T18:24:16.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.188 7fedaf11e700 1 Processor -- start 2021-04-09T18:24:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.188 7fedaf11e700 1 -- start start 2021-04-09T18:24:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.188 7fedaf11e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda810d010 0x7feda80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.188 7fedaf11e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feda80d1da0 0x7feda80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedaf11e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda810d480 con 0x7feda80d82c0 2021-04-09T18:24:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedaf11e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda810e780 con 0x7feda80d1da0 2021-04-09T18:24:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedaf11e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda80d8d30 con 0x7feda810d010 2021-04-09T18:24:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedacf3a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda810d010 0x7feda80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7feda7fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feda80d1da0 0x7feda80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedacf3a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda810d010 0x7feda80d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:58274/0 (socket says 172.21.15.32:58274) 2021-04-09T18:24:16.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7feda7fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feda80d1da0 0x7feda80d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:42806/0 (socket says 172.21.15.32:42806) 2021-04-09T18:24:16.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedacf3a700 1 -- 172.21.15.32:0/1768240055 learned_addr learned my addr 172.21.15.32:0/1768240055 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:16.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedad73b700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7feda80d82c0 0x7feda810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58564/0 (socket says 172.21.15.32:58564) 2021-04-09T18:24:16.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedacf3a700 1 -- 172.21.15.32:0/1768240055 >> v2:172.21.15.86:3300/0 conn(0x7feda80d1da0 msgr2=0x7feda80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:16.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedacf3a700 1 --2- 172.21.15.32:0/1768240055 >> v2:172.21.15.86:3300/0 conn(0x7feda80d1da0 0x7feda80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedacf3a700 1 -- 172.21.15.32:0/1768240055 >> v1:172.21.15.86:6789/0 conn(0x7feda80d82c0 legacy=0x7feda810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:16.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedacf3a700 1 -- 172.21.15.32:0/1768240055 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda80d9220 con 0x7feda810d010 2021-04-09T18:24:16.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7fedacf3a700 1 --2- 172.21.15.32:0/1768240055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda810d010 0x7feda80d1780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7feda0006a80 tx=0x7feda0009f00).ready entity=mon.2 client_cookie=73e45305567d8f36 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:16.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.189 7feda77fe700 1 -- 172.21.15.32:0/1768240055 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feda000b5c0 con 0x7feda810d010 2021-04-09T18:24:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7feda77fe700 1 -- 172.21.15.32:0/1768240055 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda000b8c0 con 0x7feda810d010 2021-04-09T18:24:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7feda77fe700 1 -- 172.21.15.32:0/1768240055 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feda000bda0 con 0x7feda810d010 2021-04-09T18:24:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7fedaf11e700 1 -- 172.21.15.32:0/1768240055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda810d010 msgr2=0x7feda80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7fedaf11e700 1 --2- 172.21.15.32:0/1768240055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda810d010 0x7feda80d1780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7feda0006a80 tx=0x7feda0009f00).stop 2021-04-09T18:24:16.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7fedaf11e700 1 -- 172.21.15.32:0/1768240055 shutdown_connections 2021-04-09T18:24:16.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7fedaf11e700 1 --2- 172.21.15.32:0/1768240055 >> v2:172.21.15.86:3300/0 conn(0x7feda80d1da0 0x7feda80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7fedaf11e700 1 --2- 172.21.15.32:0/1768240055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda810d010 0x7feda80d1780 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7fedaf11e700 1 -- 172.21.15.32:0/1768240055 >> 172.21.15.32:0/1768240055 conn(0x7feda80ce5f0 msgr2=0x7feda80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:16.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7fedaf11e700 1 -- 172.21.15.32:0/1768240055 shutdown_connections 2021-04-09T18:24:16.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.190 7fedaf11e700 1 -- 172.21.15.32:0/1768240055 wait complete. 2021-04-09T18:24:16.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.210 7fedaf11e700 1 Processor -- start 2021-04-09T18:24:16.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.210 7fedaf11e700 1 -- start start 2021-04-09T18:24:16.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.211 7fedaf11e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feda80b2140 0x7feda809d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:16.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.211 7fedaf11e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda809e010 0x7feda8194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:16.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.211 7fedaf11e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feda81950a0 0x7feda80a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:16.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.211 7fedaf11e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda809d470 con 0x7feda81950a0 2021-04-09T18:24:16.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.211 7fedaf11e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda809e410 con 0x7feda80b2140 2021-04-09T18:24:16.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.211 7fedaf11e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda81954a0 con 0x7feda809e010 2021-04-09T18:24:16.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.211 7fedad73b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feda81950a0 0x7feda80a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:16.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.211 7fedad73b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feda81950a0 0x7feda80a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51056/0 (socket says 172.21.15.32:51056) 2021-04-09T18:24:16.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.211 7fedad73b700 1 -- 172.21.15.32:0/4075846155 learned_addr learned my addr 172.21.15.32:0/4075846155 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:16.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7feda7fff700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda809e010 0x7feda8194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:16.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7fedacf3a700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feda80b2140 0x7feda809d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:16.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7fedad73b700 1 -- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda809e010 msgr2=0x7feda8194a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7fedad73b700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda809e010 0x7feda8194a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7fedad73b700 1 -- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feda80b2140 msgr2=0x7feda809d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7fedad73b700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feda80b2140 0x7feda809d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7fedad73b700 1 -- 172.21.15.32:0/4075846155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda81956c0 con 0x7feda81950a0 2021-04-09T18:24:16.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7fedad73b700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feda81950a0 0x7feda80a0460 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fed90001450 tx=0x7fed9000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7feda57fa700 1 -- 172.21.15.32:0/4075846155 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed9000ad20 con 0x7feda81950a0 2021-04-09T18:24:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7feda57fa700 1 -- 172.21.15.32:0/4075846155 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed900075a0 con 0x7feda81950a0 2021-04-09T18:24:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7feda57fa700 1 -- 172.21.15.32:0/4075846155 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed90004320 con 0x7feda81950a0 2021-04-09T18:24:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda81956c0 con 0x7feda81950a0 2021-04-09T18:24:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.212 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda81956c0 con 0x7feda81950a0 2021-04-09T18:24:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.213 7feda57fa700 1 -- 172.21.15.32:0/4075846155 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fed90004320 con 0x7feda81950a0 2021-04-09T18:24:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.213 7feda57fa700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed8c018930 0x7fed8c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.213 7feda57fa700 1 -- 172.21.15.32:0/4075846155 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fed90032a00 con 0x7feda81950a0 2021-04-09T18:24:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.213 7fedacf3a700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed8c018930 0x7fed8c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:16.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.213 7fedacf3a700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed8c018930 0x7fed8c01ae10 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:16.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.233 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda81b7040 con 0x7feda81950a0 2021-04-09T18:24:16.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.236 7feda57fa700 1 -- 172.21.15.32:0/4075846155 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fed90000960 con 0x7feda81950a0 2021-04-09T18:24:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.430 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7feda81b7040 con 0x7feda81950a0 2021-04-09T18:24:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:16.998 7feda57fa700 1 -- 172.21.15.32:0/4075846155 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v227) v1 ==== 105+0+0 (secure 0 0 0) 0x7fed90004320 con 0x7feda81950a0 2021-04-09T18:24:17.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed8c018930 msgr2=0x7fed8c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:17.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed8c018930 0x7fed8c01ae10 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feda81950a0 msgr2=0x7feda80a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:17.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feda81950a0 0x7feda80a0460 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fed90001450 tx=0x7fed9000a930).stop 2021-04-09T18:24:17.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 shutdown_connections 2021-04-09T18:24:17.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed8c018930 0x7fed8c01ae10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feda81950a0 0x7feda80a0460 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feda809e010 0x7feda8194a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 --2- 172.21.15.32:0/4075846155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feda80b2140 0x7feda809d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 >> 172.21.15.32:0/4075846155 conn(0x7feda80b27e0 msgr2=0x7feda81b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:17.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 shutdown_connections 2021-04-09T18:24:17.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.046 7fedaf11e700 1 -- 172.21.15.32:0/4075846155 wait complete. 2021-04-09T18:24:17.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:24:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.210 7f6eaf9ca700 1 Processor -- start 2021-04-09T18:24:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.210 7f6eaf9ca700 1 -- start start 2021-04-09T18:24:17.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6eaf9ca700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6ea810d010 0x7f6ea80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:17.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6eaf9ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea80d1da0 0x7f6ea80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:17.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6eaf9ca700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea810d480 con 0x7f6ea80d82c0 2021-04-09T18:24:17.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6eaf9ca700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea80d9220 con 0x7f6ea810d010 2021-04-09T18:24:17.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6eaf9ca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea80d86c0 con 0x7f6ea80d1da0 2021-04-09T18:24:17.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6ead7e6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6ea810d010 0x7f6ea80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:17.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6eacfe5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea80d1da0 0x7f6ea80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:17.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6eadfe7700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6ea80d82c0 0x7f6ea810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58578/0 (socket says 172.21.15.32:58578) 2021-04-09T18:24:17.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6eadfe7700 1 -- 172.21.15.32:0/1699357634 learned_addr learned my addr 172.21.15.32:0/1699357634 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:17.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6e9bfff700 1 -- 172.21.15.32:0/1699357634 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165734695 0 0) 0x7f6ea0001440 con 0x7f6ea80d82c0 2021-04-09T18:24:17.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6e9bfff700 1 -- 172.21.15.32:0/1699357634 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e94002fe0 con 0x7f6ea80d82c0 2021-04-09T18:24:17.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6ead7e6700 1 -- 172.21.15.32:0/1699357634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea80d1da0 msgr2=0x7f6ea80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:17.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6ead7e6700 1 --2- 172.21.15.32:0/1699357634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea80d1da0 0x7f6ea80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6ead7e6700 1 -- 172.21.15.32:0/1699357634 >> v1:172.21.15.86:6789/0 conn(0x7f6ea80d82c0 legacy=0x7f6ea810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6ead7e6700 1 -- 172.21.15.32:0/1699357634 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea810e1f0 con 0x7f6ea810d010 2021-04-09T18:24:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.211 7f6eacfe5700 1 --2- 172.21.15.32:0/1699357634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea80d1da0 0x7f6ea80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.212 7f6ead7e6700 1 --2- 172.21.15.32:0/1699357634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea810d010 0x7f6ea80d1780 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f6ea4006650 tx=0x7f6ea4009a80).ready entity=mon.1 client_cookie=53ac0be8b9d508c7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.212 7f6e9bfff700 1 -- 172.21.15.32:0/1699357634 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ea40009b0 con 0x7f6ea810d010 2021-04-09T18:24:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.212 7f6e9bfff700 1 -- 172.21.15.32:0/1699357634 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea400c0c0 con 0x7f6ea810d010 2021-04-09T18:24:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.212 7f6e9bfff700 1 -- 172.21.15.32:0/1699357634 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ea400c510 con 0x7f6ea810d010 2021-04-09T18:24:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.212 7f6eaf9ca700 1 -- 172.21.15.32:0/1699357634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea810d010 msgr2=0x7f6ea80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.212 7f6eaf9ca700 1 --2- 172.21.15.32:0/1699357634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea810d010 0x7f6ea80d1780 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f6ea4006650 tx=0x7f6ea4009a80).stop 2021-04-09T18:24:17.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.213 7f6eaf9ca700 1 -- 172.21.15.32:0/1699357634 shutdown_connections 2021-04-09T18:24:17.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.213 7f6eaf9ca700 1 --2- 172.21.15.32:0/1699357634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea80d1da0 0x7f6ea80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.213 7f6eaf9ca700 1 --2- 172.21.15.32:0/1699357634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea810d010 0x7f6ea80d1780 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.213 7f6eaf9ca700 1 -- 172.21.15.32:0/1699357634 >> 172.21.15.32:0/1699357634 conn(0x7f6ea80ce5f0 msgr2=0x7f6ea80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.213 7f6eaf9ca700 1 -- 172.21.15.32:0/1699357634 shutdown_connections 2021-04-09T18:24:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.213 7f6eaf9ca700 1 -- 172.21.15.32:0/1699357634 wait complete. 2021-04-09T18:24:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.233 7f6eaf9ca700 1 Processor -- start 2021-04-09T18:24:17.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.233 7f6eaf9ca700 1 -- start start 2021-04-09T18:24:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.233 7f6eaf9ca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea80b2140 0x7f6ea809d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.233 7f6eaf9ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ea809e010 0x7f6ea8194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:17.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.233 7f6eaf9ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea81950a0 0x7f6ea80a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.233 7f6eaf9ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea809d470 con 0x7f6ea809e010 2021-04-09T18:24:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.233 7f6eaf9ca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea809e410 con 0x7f6ea80b2140 2021-04-09T18:24:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.233 7f6eaf9ca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea81954a0 con 0x7f6ea81950a0 2021-04-09T18:24:17.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6eacfe5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ea809e010 0x7f6ea8194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6eacfe5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ea809e010 0x7f6ea8194a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51066/0 (socket says 172.21.15.32:51066) 2021-04-09T18:24:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6eacfe5700 1 -- 172.21.15.32:0/15260298 learned_addr learned my addr 172.21.15.32:0/15260298 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6ead7e6700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea80b2140 0x7f6ea809d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6eadfe7700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea81950a0 0x7f6ea80a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:17.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6ead7e6700 1 -- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea81950a0 msgr2=0x7f6ea80a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:17.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6ead7e6700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea81950a0 0x7f6ea80a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6ead7e6700 1 -- 172.21.15.32:0/15260298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ea809e010 msgr2=0x7f6ea8194a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:17.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6ead7e6700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ea809e010 0x7f6ea8194a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:17.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6ead7e6700 1 -- 172.21.15.32:0/15260298 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea81956c0 con 0x7f6ea80b2140 2021-04-09T18:24:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6eacfe5700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ea809e010 0x7f6ea8194a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6ead7e6700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea80b2140 0x7f6ea809d9f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f6ea400c4e0 tx=0x7f6ea400c880).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6e99ffb700 1 -- 172.21.15.32:0/15260298 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ea4000c90 con 0x7f6ea80b2140 2021-04-09T18:24:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6e99ffb700 1 -- 172.21.15.32:0/15260298 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea400cc70 con 0x7f6ea80b2140 2021-04-09T18:24:17.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6e99ffb700 1 -- 172.21.15.32:0/15260298 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ea4002220 con 0x7f6ea80b2140 2021-04-09T18:24:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea81956c0 con 0x7f6ea80b2140 2021-04-09T18:24:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.234 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea81956c0 con 0x7f6ea80b2140 2021-04-09T18:24:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.235 7f6e99ffb700 1 -- 172.21.15.32:0/15260298 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6ea400a020 con 0x7f6ea80b2140 2021-04-09T18:24:17.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.235 7f6e99ffb700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e880188d0 0x7f6e8801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:17.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.235 7f6e99ffb700 1 -- 172.21.15.32:0/15260298 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6ea4032930 con 0x7f6ea80b2140 2021-04-09T18:24:17.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.235 7f6eacfe5700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e880188d0 0x7f6e8801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:17.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.236 7f6eacfe5700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e880188d0 0x7f6e8801add0 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:17.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.255 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea81b7040 con 0x7f6ea80b2140 2021-04-09T18:24:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.258 7f6e99ffb700 1 -- 172.21.15.32:0/15260298 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6ea4006740 con 0x7f6ea80b2140 2021-04-09T18:24:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:17.452 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f6ea81b7040 con 0x7f6ea80b2140 2021-04-09T18:24:18.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.000 7f6e99ffb700 1 -- 172.21.15.32:0/15260298 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v228) v1 ==== 105+0+0 (secure 0 0 0) 0x7f6ea4009ef0 con 0x7f6ea80b2140 2021-04-09T18:24:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e880188d0 msgr2=0x7f6e8801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:18.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e880188d0 0x7f6e8801add0 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea80b2140 msgr2=0x7f6ea809d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:18.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea80b2140 0x7f6ea809d9f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f6ea400c4e0 tx=0x7f6ea400c880).stop 2021-04-09T18:24:18.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 shutdown_connections 2021-04-09T18:24:18.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e880188d0 0x7f6e8801add0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ea809e010 0x7f6ea8194a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea81950a0 0x7f6ea80a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 --2- 172.21.15.32:0/15260298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea80b2140 0x7f6ea809d9f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 >> 172.21.15.32:0/15260298 conn(0x7f6ea80b27e0 msgr2=0x7f6ea81b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 shutdown_connections 2021-04-09T18:24:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.019 7f6eaf9ca700 1 -- 172.21.15.32:0/15260298 wait complete. 2021-04-09T18:24:18.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:18.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.186 7f796bafb700 1 Processor -- start 2021-04-09T18:24:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f796bafb700 1 -- start start 2021-04-09T18:24:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f796bafb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796410d010 0x7f79640d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f796bafb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79640d1da0 0x7f79640d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f796bafb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f79640d82c0 0x7f796410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f796bafb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796410e780 con 0x7f79640d1da0 2021-04-09T18:24:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f796bafb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79640d8d30 con 0x7f79640d82c0 2021-04-09T18:24:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f796bafb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79640d9220 con 0x7f796410d010 2021-04-09T18:24:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f7969116700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79640d1da0 0x7f79640d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f7969116700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79640d1da0 0x7f79640d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51078/0 (socket says 172.21.15.32:51078) 2021-04-09T18:24:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.187 7f7969116700 1 -- 172.21.15.32:0/2388928689 learned_addr learned my addr 172.21.15.32:0/2388928689 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f796a118700 1 --2- 172.21.15.32:0/2388928689 >> v2:172.21.15.86:3300/0 conn(0x7f79640d82c0 0x7f796410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7969917700 1 --2- 172.21.15.32:0/2388928689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796410d010 0x7f79640d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7969116700 1 -- 172.21.15.32:0/2388928689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796410d010 msgr2=0x7f79640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:18.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7969116700 1 --2- 172.21.15.32:0/2388928689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796410d010 0x7f79640d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7969116700 1 -- 172.21.15.32:0/2388928689 >> v2:172.21.15.86:3300/0 conn(0x7f79640d82c0 msgr2=0x7f796410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7969116700 1 --2- 172.21.15.32:0/2388928689 >> v2:172.21.15.86:3300/0 conn(0x7f79640d82c0 0x7f796410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7969116700 1 -- 172.21.15.32:0/2388928689 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796410d480 con 0x7f79640d1da0 2021-04-09T18:24:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7969116700 1 --2- 172.21.15.32:0/2388928689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640d1da0 0x7f79640d7ca0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f7958006700 tx=0x7f7958009b00).ready entity=mon.0 client_cookie=2a2c8e89fdd28073 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7968915700 1 -- 172.21.15.32:0/2388928689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795800c0c0 con 0x7f79640d1da0 2021-04-09T18:24:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7968915700 1 -- 172.21.15.32:0/2388928689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795800c400 con 0x7f79640d1da0 2021-04-09T18:24:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f7968915700 1 -- 172.21.15.32:0/2388928689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795800c8e0 con 0x7f79640d1da0 2021-04-09T18:24:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f796bafb700 1 -- 172.21.15.32:0/2388928689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640d1da0 msgr2=0x7f79640d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.188 7f796bafb700 1 --2- 172.21.15.32:0/2388928689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640d1da0 0x7f79640d7ca0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f7958006700 tx=0x7f7958009b00).stop 2021-04-09T18:24:18.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.189 7f796bafb700 1 -- 172.21.15.32:0/2388928689 shutdown_connections 2021-04-09T18:24:18.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.189 7f796bafb700 1 --2- 172.21.15.32:0/2388928689 >> v2:172.21.15.86:3300/0 conn(0x7f79640d82c0 0x7f796410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.189 7f796bafb700 1 --2- 172.21.15.32:0/2388928689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640d1da0 0x7f79640d7ca0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.189 7f796bafb700 1 --2- 172.21.15.32:0/2388928689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796410d010 0x7f79640d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.189 7f796bafb700 1 -- 172.21.15.32:0/2388928689 >> 172.21.15.32:0/2388928689 conn(0x7f79640ce5f0 msgr2=0x7f79640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:18.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.189 7f796bafb700 1 -- 172.21.15.32:0/2388928689 shutdown_connections 2021-04-09T18:24:18.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.189 7f796bafb700 1 -- 172.21.15.32:0/2388928689 wait complete. 2021-04-09T18:24:18.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.209 7f796bafb700 1 Processor -- start 2021-04-09T18:24:18.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.209 7f796bafb700 1 -- start start 2021-04-09T18:24:18.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f796bafb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640a0a10 0x7f79640bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:18.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f796bafb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796418ad00 0x7f796419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:18.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f796bafb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79641a03f0 0x7f7964095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:18.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f796bafb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796418b100 con 0x7f79640a0a10 2021-04-09T18:24:18.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f796bafb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7964095860 con 0x7f796418ad00 2021-04-09T18:24:18.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f796bafb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7964095a80 con 0x7f79641a03f0 2021-04-09T18:24:18.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969917700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640a0a10 0x7f79640bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:18.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969917700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640a0a10 0x7f79640bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51080/0 (socket says 172.21.15.32:51080) 2021-04-09T18:24:18.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969917700 1 -- 172.21.15.32:0/2521851946 learned_addr learned my addr 172.21.15.32:0/2521851946 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:18.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f796a118700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79641a03f0 0x7f7964095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:18.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969116700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796418ad00 0x7f796419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:18.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969917700 1 -- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79641a03f0 msgr2=0x7f7964095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:18.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969917700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79641a03f0 0x7f7964095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969917700 1 -- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796418ad00 msgr2=0x7f796419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:18.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969917700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796418ad00 0x7f796419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:18.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969917700 1 -- 172.21.15.32:0/2521851946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7964189250 con 0x7f79640a0a10 2021-04-09T18:24:18.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f7969917700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640a0a10 0x7f79640bf870 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f7960006be0 tx=0x7f796000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:18.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.210 7f79567fc700 1 -- 172.21.15.32:0/2521851946 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79600064d0 con 0x7f79640a0a10 2021-04-09T18:24:18.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.211 7f79567fc700 1 -- 172.21.15.32:0/2521851946 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796000bd80 con 0x7f79640a0a10 2021-04-09T18:24:18.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.211 7f796bafb700 1 -- 172.21.15.32:0/2521851946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7964189250 con 0x7f79640a0a10 2021-04-09T18:24:18.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.211 7f79567fc700 1 -- 172.21.15.32:0/2521851946 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7960002030 con 0x7f79640a0a10 2021-04-09T18:24:18.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.211 7f796bafb700 1 -- 172.21.15.32:0/2521851946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7964189250 con 0x7f79640a0a10 2021-04-09T18:24:18.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.211 7f79567fc700 1 -- 172.21.15.32:0/2521851946 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7960006c90 con 0x7f79640a0a10 2021-04-09T18:24:18.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.211 7f79567fc700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f794c018930 0x7f794c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:18.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.211 7f79567fc700 1 -- 172.21.15.32:0/2521851946 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7960032a10 con 0x7f79640a0a10 2021-04-09T18:24:18.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.211 7f7969116700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f794c018930 0x7f794c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:18.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.212 7f7969116700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f794c018930 0x7f794c01ae10 crc :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:18.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.231 7f796bafb700 1 -- 172.21.15.32:0/2521851946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7964093d50 con 0x7f79640a0a10 2021-04-09T18:24:18.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.235 7f79567fc700 1 -- 172.21.15.32:0/2521851946 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7960006820 con 0x7f79640a0a10 2021-04-09T18:24:18.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:18.431 7f796bafb700 1 -- 172.21.15.32:0/2521851946 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f796410d480 con 0x7f79640a0a10 2021-04-09T18:24:19.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.005 7f79567fc700 1 -- 172.21.15.32:0/2521851946 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v229) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7960006c90 con 0x7f79640a0a10 2021-04-09T18:24:19.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 -- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f794c018930 msgr2=0x7f794c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:19.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f794c018930 0x7f794c01ae10 crc :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:19.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 -- 172.21.15.32:0/2521851946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640a0a10 msgr2=0x7f79640bf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:19.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640a0a10 0x7f79640bf870 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f7960006be0 tx=0x7f796000b990).stop 2021-04-09T18:24:19.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 -- 172.21.15.32:0/2521851946 shutdown_connections 2021-04-09T18:24:19.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f794c018930 0x7f794c01ae10 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:19.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79641a03f0 0x7f7964095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:19.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796418ad00 0x7f796419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:19.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 --2- 172.21.15.32:0/2521851946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79640a0a10 0x7f79640bf870 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:19.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 -- 172.21.15.32:0/2521851946 >> 172.21.15.32:0/2521851946 conn(0x7f79640975f0 msgr2=0x7f7964096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:19.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 -- 172.21.15.32:0/2521851946 shutdown_connections 2021-04-09T18:24:19.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.048 7f796bafb700 1 -- 172.21.15.32:0/2521851946 wait complete. 2021-04-09T18:24:19.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:19.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.217 7f14ca76e700 1 Processor -- start 2021-04-09T18:24:19.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.217 7f14ca76e700 1 -- start start 2021-04-09T18:24:19.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14ca76e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14c410d010 0x7f14c40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:19.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14ca76e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c410d480 con 0x7f14c40d1da0 2021-04-09T18:24:19.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14ca76e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c410e1f0 con 0x7f14c40d8490 2021-04-09T18:24:19.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14ca76e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c40d8d30 con 0x7f14c410d010 2021-04-09T18:24:19.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14c3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f14c410d010 0x7f14c40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:19.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14bbfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f14c40d1da0 0x7f14c40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60676/0 (socket says 172.21.15.32:60676) 2021-04-09T18:24:19.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14bbfff700 1 -- 172.21.15.32:0/3277505054 learned_addr learned my addr 172.21.15.32:0/3277505054 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:19.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14c3fff700 1 -- 172.21.15.32:0/3277505054 >> v1:172.21.15.86:6789/0 conn(0x7f14c40d8490 legacy=0x7f14c410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:19.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14c3fff700 1 -- 172.21.15.32:0/3277505054 >> v1:172.21.15.32:6789/0 conn(0x7f14c40d1da0 legacy=0x7f14c40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:19.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14c37fe700 1 -- 172.21.15.32:0/3277505054 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358087965 0 0) 0x7f14bc000f90 con 0x7f14c40d8490 2021-04-09T18:24:19.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.218 7f14c3fff700 1 -- 172.21.15.32:0/3277505054 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c40d9220 con 0x7f14c410d010 2021-04-09T18:24:19.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.219 7f14c3fff700 1 --2- 172.21.15.32:0/3277505054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c410d010 0x7f14c40d1780 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f14b4006650 tx=0x7f14b4009a80).ready entity=mon.0 client_cookie=783269730a8a17c7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:19.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.219 7f14c37fe700 1 -- 172.21.15.32:0/3277505054 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14b40009b0 con 0x7f14c410d010 2021-04-09T18:24:19.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.219 7f14c37fe700 1 -- 172.21.15.32:0/3277505054 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b400c0c0 con 0x7f14c410d010 2021-04-09T18:24:19.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.219 7f14c37fe700 1 -- 172.21.15.32:0/3277505054 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14b400c510 con 0x7f14c410d010 2021-04-09T18:24:19.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.219 7f14ca76e700 1 -- 172.21.15.32:0/3277505054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c410d010 msgr2=0x7f14c40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.219 7f14ca76e700 1 --2- 172.21.15.32:0/3277505054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c410d010 0x7f14c40d1780 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f14b4006650 tx=0x7f14b4009a80).stop 2021-04-09T18:24:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.220 7f14ca76e700 1 -- 172.21.15.32:0/3277505054 shutdown_connections 2021-04-09T18:24:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.220 7f14ca76e700 1 --2- 172.21.15.32:0/3277505054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c410d010 0x7f14c40d1780 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:19.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.220 7f14ca76e700 1 -- 172.21.15.32:0/3277505054 >> 172.21.15.32:0/3277505054 conn(0x7f14c40ce5f0 msgr2=0x7f14c40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.220 7f14ca76e700 1 -- 172.21.15.32:0/3277505054 shutdown_connections 2021-04-09T18:24:19.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.220 7f14ca76e700 1 -- 172.21.15.32:0/3277505054 wait complete. 2021-04-09T18:24:19.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14ca76e700 1 Processor -- start 2021-04-09T18:24:19.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14ca76e700 1 -- start start 2021-04-09T18:24:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14ca76e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40b7810 0x7f14c40b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14ca76e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c40b0900 0x7f14c40b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14ca76e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14c40b1320 0x7f14c4179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:19.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14ca76e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b0090 con 0x7f14c40b7810 2021-04-09T18:24:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14ca76e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c4179c30 con 0x7f14c40b0900 2021-04-09T18:24:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14ca76e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c4179e50 con 0x7f14c40b1320 2021-04-09T18:24:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14c3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40b7810 0x7f14c40b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:19.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14c3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40b7810 0x7f14c40b7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51092/0 (socket says 172.21.15.32:51092) 2021-04-09T18:24:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14c3fff700 1 -- 172.21.15.32:0/199573297 learned_addr learned my addr 172.21.15.32:0/199573297 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14bbfff700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c40b0900 0x7f14c40b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.240 7f14c8d8b700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14c40b1320 0x7f14c4179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14c3fff700 1 -- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14c40b1320 msgr2=0x7f14c4179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:19.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14c3fff700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14c40b1320 0x7f14c4179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:19.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14c3fff700 1 -- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c40b0900 msgr2=0x7f14c40b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:19.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14c3fff700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c40b0900 0x7f14c40b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:19.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14c3fff700 1 -- 172.21.15.32:0/199573297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c417a070 con 0x7f14c40b7810 2021-04-09T18:24:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14bbfff700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c40b0900 0x7f14c40b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14c3fff700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40b7810 0x7f14c40b7c10 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7f14b4009650 tx=0x7f14b400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14c17fa700 1 -- 172.21.15.32:0/199573297 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14b40093a0 con 0x7f14c40b7810 2021-04-09T18:24:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14c17fa700 1 -- 172.21.15.32:0/199573297 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b4006070 con 0x7f14c40b7810 2021-04-09T18:24:19.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14ca76e700 1 -- 172.21.15.32:0/199573297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c417a070 con 0x7f14c40b7810 2021-04-09T18:24:19.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14c17fa700 1 -- 172.21.15.32:0/199573297 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14b40052e0 con 0x7f14c40b7810 2021-04-09T18:24:19.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.241 7f14ca76e700 1 -- 172.21.15.32:0/199573297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c417a070 con 0x7f14c40b7810 2021-04-09T18:24:19.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.242 7f14c17fa700 1 -- 172.21.15.32:0/199573297 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f14b4009300 con 0x7f14c40b7810 2021-04-09T18:24:19.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.242 7f14c17fa700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14a80188c0 0x7f14a801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:19.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.242 7f14c17fa700 1 -- 172.21.15.32:0/199573297 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f14b4032a10 con 0x7f14c40b7810 2021-04-09T18:24:19.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.242 7f14bbfff700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14a80188c0 0x7f14a801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:19.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.242 7f14bbfff700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14a80188c0 0x7f14a801adc0 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:19.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.262 7f14ca76e700 1 -- 172.21.15.32:0/199573297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c408dac0 con 0x7f14c40b7810 2021-04-09T18:24:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.265 7f14c17fa700 1 -- 172.21.15.32:0/199573297 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f14b4006030 con 0x7f14c40b7810 2021-04-09T18:24:19.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:19.457 7f14ca76e700 1 -- 172.21.15.32:0/199573297 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f14c408dac0 con 0x7f14c40b7810 2021-04-09T18:24:20.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.007 7f14c17fa700 1 -- 172.21.15.32:0/199573297 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v230) v1 ==== 105+0+0 (secure 0 0 0) 0x7f14b4009300 con 0x7f14c40b7810 2021-04-09T18:24:20.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.023 7f14ca76e700 1 -- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14a80188c0 msgr2=0x7f14a801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:20.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.023 7f14ca76e700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14a80188c0 0x7f14a801adc0 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.023 7f14ca76e700 1 -- 172.21.15.32:0/199573297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40b7810 msgr2=0x7f14c40b7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:20.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.023 7f14ca76e700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40b7810 0x7f14c40b7c10 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7f14b4009650 tx=0x7f14b400b8e0).stop 2021-04-09T18:24:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.023 7f14ca76e700 1 -- 172.21.15.32:0/199573297 shutdown_connections 2021-04-09T18:24:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.023 7f14ca76e700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f14a80188c0 0x7f14a801adc0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.024 7f14ca76e700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14c40b1320 0x7f14c4179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.024 7f14ca76e700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c40b0900 0x7f14c40b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.024 7f14ca76e700 1 --2- 172.21.15.32:0/199573297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14c40b7810 0x7f14c40b7c10 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.024 7f14ca76e700 1 -- 172.21.15.32:0/199573297 >> 172.21.15.32:0/199573297 conn(0x7f14c40bb410 msgr2=0x7f14c40a13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.024 7f14ca76e700 1 -- 172.21.15.32:0/199573297 shutdown_connections 2021-04-09T18:24:20.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.024 7f14ca76e700 1 -- 172.21.15.32:0/199573297 wait complete. 2021-04-09T18:24:20.096 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:24:20.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.186 7f531c531700 1 Processor -- start 2021-04-09T18:24:20.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.186 7f531c531700 1 -- start start 2021-04-09T18:24:20.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f531c531700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f531410d010 0x7f53140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:20.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f531c531700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53140d8490 0x7f531410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:20.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f531c531700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531410d480 con 0x7f53140d1da0 2021-04-09T18:24:20.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f531c531700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53140d8d30 con 0x7f531410d010 2021-04-09T18:24:20.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f531c531700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53140d9220 con 0x7f53140d8490 2021-04-09T18:24:20.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f5319b4c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f53140d1da0 0x7f53140d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60692/0 (socket says 172.21.15.32:60692) 2021-04-09T18:24:20.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f5319b4c700 1 -- 172.21.15.32:0/811093405 learned_addr learned my addr 172.21.15.32:0/811093405 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:20.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f531ab4e700 1 --2- 172.21.15.32:0/811093405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53140d8490 0x7f531410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:20.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f531a34d700 1 --2- 172.21.15.32:0/811093405 >> v2:172.21.15.86:3300/0 conn(0x7f531410d010 0x7f53140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:20.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f531934b700 1 -- 172.21.15.32:0/811093405 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658828680 0 0) 0x7f5308000ce0 con 0x7f53140d1da0 2021-04-09T18:24:20.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.187 7f531934b700 1 -- 172.21.15.32:0/811093405 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5300002fe0 con 0x7f53140d1da0 2021-04-09T18:24:20.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531a34d700 1 -- 172.21.15.32:0/811093405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53140d8490 msgr2=0x7f531410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:20.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531a34d700 1 --2- 172.21.15.32:0/811093405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53140d8490 0x7f531410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531a34d700 1 -- 172.21.15.32:0/811093405 >> v1:172.21.15.32:6789/0 conn(0x7f53140d1da0 legacy=0x7f53140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:20.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531a34d700 1 -- 172.21.15.32:0/811093405 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531410e020 con 0x7f531410d010 2021-04-09T18:24:20.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531ab4e700 1 --2- 172.21.15.32:0/811093405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53140d8490 0x7f531410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:20.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531a34d700 1 --2- 172.21.15.32:0/811093405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f531410d010 0x7f53140d1780 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f5310006650 tx=0x7f5310009a80).ready entity=mon.1 client_cookie=1e3dbc5055360171 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:20.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531934b700 1 -- 172.21.15.32:0/811093405 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53100009b0 con 0x7f531410d010 2021-04-09T18:24:20.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531934b700 1 -- 172.21.15.32:0/811093405 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531000c0c0 con 0x7f531410d010 2021-04-09T18:24:20.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531934b700 1 -- 172.21.15.32:0/811093405 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f531000c510 con 0x7f531410d010 2021-04-09T18:24:20.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531c531700 1 -- 172.21.15.32:0/811093405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f531410d010 msgr2=0x7f53140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:20.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.188 7f531c531700 1 --2- 172.21.15.32:0/811093405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f531410d010 0x7f53140d1780 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f5310006650 tx=0x7f5310009a80).stop 2021-04-09T18:24:20.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.189 7f531c531700 1 -- 172.21.15.32:0/811093405 shutdown_connections 2021-04-09T18:24:20.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.189 7f531c531700 1 --2- 172.21.15.32:0/811093405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53140d8490 0x7f531410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.189 7f531c531700 1 --2- 172.21.15.32:0/811093405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f531410d010 0x7f53140d1780 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.189 7f531c531700 1 -- 172.21.15.32:0/811093405 >> 172.21.15.32:0/811093405 conn(0x7f53140ce5f0 msgr2=0x7f53140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:20.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.189 7f531c531700 1 -- 172.21.15.32:0/811093405 shutdown_connections 2021-04-09T18:24:20.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.189 7f531c531700 1 -- 172.21.15.32:0/811093405 wait complete. 2021-04-09T18:24:20.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.209 7f531c531700 1 Processor -- start 2021-04-09T18:24:20.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.209 7f531c531700 1 -- start start 2021-04-09T18:24:20.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531c531700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53140a0a10 0x7f53140bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:20.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531c531700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f531418ad00 0x7f531419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:20.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531c531700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53141a03f0 0x7f5314095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:20.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531c531700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531418b100 con 0x7f53141a03f0 2021-04-09T18:24:20.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531c531700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5314095860 con 0x7f53140a0a10 2021-04-09T18:24:20.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531c531700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5314095a80 con 0x7f531418ad00 2021-04-09T18:24:20.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531ab4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53141a03f0 0x7f5314095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:20.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531ab4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53141a03f0 0x7f5314095240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51110/0 (socket says 172.21.15.32:51110) 2021-04-09T18:24:20.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531ab4e700 1 -- 172.21.15.32:0/700196328 learned_addr learned my addr 172.21.15.32:0/700196328 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:20.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f5319b4c700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f531418ad00 0x7f531419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:20.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531a34d700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53140a0a10 0x7f53140bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:20.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531ab4e700 1 -- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f531418ad00 msgr2=0x7f531419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:20.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531ab4e700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f531418ad00 0x7f531419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531ab4e700 1 -- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53140a0a10 msgr2=0x7f53140bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:20.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531ab4e700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53140a0a10 0x7f53140bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:20.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531ab4e700 1 -- 172.21.15.32:0/700196328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5314189250 con 0x7f53141a03f0 2021-04-09T18:24:20.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531ab4e700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53141a03f0 0x7f5314095240 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f530c00b910 tx=0x7f530c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:20.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f5306ffd700 1 -- 172.21.15.32:0/700196328 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f530c006aa0 con 0x7f53141a03f0 2021-04-09T18:24:20.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f5306ffd700 1 -- 172.21.15.32:0/700196328 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530c0011f0 con 0x7f53141a03f0 2021-04-09T18:24:20.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f5306ffd700 1 -- 172.21.15.32:0/700196328 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f530c001640 con 0x7f53141a03f0 2021-04-09T18:24:20.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531c531700 1 -- 172.21.15.32:0/700196328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5314189250 con 0x7f53141a03f0 2021-04-09T18:24:20.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.210 7f531c531700 1 -- 172.21.15.32:0/700196328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5314189250 con 0x7f53141a03f0 2021-04-09T18:24:20.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.211 7f5306ffd700 1 -- 172.21.15.32:0/700196328 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f530c0056b0 con 0x7f53141a03f0 2021-04-09T18:24:20.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.211 7f5306ffd700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52fc0188c0 0x7f52fc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:20.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.211 7f5306ffd700 1 -- 172.21.15.32:0/700196328 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f530c032b80 con 0x7f53141a03f0 2021-04-09T18:24:20.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.211 7f531a34d700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52fc0188c0 0x7f52fc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:20.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.212 7f531a34d700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52fc0188c0 0x7f52fc01adc0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:20.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.231 7f531c531700 1 -- 172.21.15.32:0/700196328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5314093d50 con 0x7f53141a03f0 2021-04-09T18:24:20.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.234 7f5306ffd700 1 -- 172.21.15.32:0/700196328 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f530c00bdf0 con 0x7f53141a03f0 2021-04-09T18:24:20.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:20.428 7f531c531700 1 -- 172.21.15.32:0/700196328 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f531410d480 con 0x7f53141a03f0 2021-04-09T18:24:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.013 7f5306ffd700 1 -- 172.21.15.32:0/700196328 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v231) v1 ==== 105+0+0 (secure 0 0 0) 0x7f530c0056b0 con 0x7f53141a03f0 2021-04-09T18:24:21.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 -- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52fc0188c0 msgr2=0x7f52fc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:21.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52fc0188c0 0x7f52fc01adc0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 -- 172.21.15.32:0/700196328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53141a03f0 msgr2=0x7f5314095240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:21.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53141a03f0 0x7f5314095240 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f530c00b910 tx=0x7f530c00bcb0).stop 2021-04-09T18:24:21.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 -- 172.21.15.32:0/700196328 shutdown_connections 2021-04-09T18:24:21.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52fc0188c0 0x7f52fc01adc0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53141a03f0 0x7f5314095240 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f531418ad00 0x7f531419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 --2- 172.21.15.32:0/700196328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53140a0a10 0x7f53140bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 -- 172.21.15.32:0/700196328 >> 172.21.15.32:0/700196328 conn(0x7f53140975f0 msgr2=0x7f5314096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:21.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 -- 172.21.15.32:0/700196328 shutdown_connections 2021-04-09T18:24:21.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.045 7f531c531700 1 -- 172.21.15.32:0/700196328 wait complete. 2021-04-09T18:24:21.118 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:21.133 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:24:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.209 7fa23eb93700 1 Processor -- start 2021-04-09T18:24:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.209 7fa23eb93700 1 -- start start 2021-04-09T18:24:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa23eb93700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa23810d010 0x7fa2380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa23eb93700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa2380d1da0 0x7fa2380d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa23eb93700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380d82c0 0x7fa23810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa23eb93700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2380d8d30 con 0x7fa2380d1da0 2021-04-09T18:24:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa23eb93700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2380d9220 con 0x7fa23810d010 2021-04-09T18:24:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa23eb93700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23810d4b0 con 0x7fa2380d82c0 2021-04-09T18:24:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa237fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa2380d1da0 0x7fa2380d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa237fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa2380d1da0 0x7fa2380d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51116/0 (socket says 172.21.15.32:51116) 2021-04-09T18:24:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa237fff700 1 -- 172.21.15.32:0/350666215 learned_addr learned my addr 172.21.15.32:0/350666215 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa23c9af700 1 --2- 172.21.15.32:0/350666215 >> v2:172.21.15.86:3300/0 conn(0x7fa23810d010 0x7fa2380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.210 7fa23d1b0700 1 --2- 172.21.15.32:0/350666215 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380d82c0 0x7fa23810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa237fff700 1 -- 172.21.15.32:0/350666215 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380d82c0 msgr2=0x7fa23810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa237fff700 1 --2- 172.21.15.32:0/350666215 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380d82c0 0x7fa23810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa237fff700 1 -- 172.21.15.32:0/350666215 >> v2:172.21.15.86:3300/0 conn(0x7fa23810d010 msgr2=0x7fa2380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa237fff700 1 --2- 172.21.15.32:0/350666215 >> v2:172.21.15.86:3300/0 conn(0x7fa23810d010 0x7fa2380d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa237fff700 1 -- 172.21.15.32:0/350666215 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2380d86c0 con 0x7fa2380d1da0 2021-04-09T18:24:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa23d1b0700 1 --2- 172.21.15.32:0/350666215 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380d82c0 0x7fa23810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa23c9af700 1 --2- 172.21.15.32:0/350666215 >> v2:172.21.15.86:3300/0 conn(0x7fa23810d010 0x7fa2380d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa237fff700 1 --2- 172.21.15.32:0/350666215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2380d1da0 0x7fa2380d7ca0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7fa22c006700 tx=0x7fa22c009b80).ready entity=mon.0 client_cookie=b2dd31d1e471a15d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:21.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa2377fe700 1 -- 172.21.15.32:0/350666215 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa22c00c0c0 con 0x7fa2380d1da0 2021-04-09T18:24:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa2377fe700 1 -- 172.21.15.32:0/350666215 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22c00c440 con 0x7fa2380d1da0 2021-04-09T18:24:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa2377fe700 1 -- 172.21.15.32:0/350666215 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa22c00c920 con 0x7fa2380d1da0 2021-04-09T18:24:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa23eb93700 1 -- 172.21.15.32:0/350666215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2380d1da0 msgr2=0x7fa2380d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:21.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.211 7fa23eb93700 1 --2- 172.21.15.32:0/350666215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2380d1da0 0x7fa2380d7ca0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7fa22c006700 tx=0x7fa22c009b80).stop 2021-04-09T18:24:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.212 7fa23eb93700 1 -- 172.21.15.32:0/350666215 shutdown_connections 2021-04-09T18:24:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.212 7fa23eb93700 1 --2- 172.21.15.32:0/350666215 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380d82c0 0x7fa23810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.212 7fa23eb93700 1 --2- 172.21.15.32:0/350666215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2380d1da0 0x7fa2380d7ca0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.212 7fa23eb93700 1 --2- 172.21.15.32:0/350666215 >> v2:172.21.15.86:3300/0 conn(0x7fa23810d010 0x7fa2380d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.212 7fa23eb93700 1 -- 172.21.15.32:0/350666215 >> 172.21.15.32:0/350666215 conn(0x7fa2380ce5f0 msgr2=0x7fa2380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.212 7fa23eb93700 1 -- 172.21.15.32:0/350666215 shutdown_connections 2021-04-09T18:24:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.212 7fa23eb93700 1 -- 172.21.15.32:0/350666215 wait complete. 2021-04-09T18:24:21.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.232 7fa23eb93700 1 Processor -- start 2021-04-09T18:24:21.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.232 7fa23eb93700 1 -- start start 2021-04-09T18:24:21.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23eb93700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2381a0050 0x7fa2381a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:21.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23eb93700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380bdcd0 0x7fa2380bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:21.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23eb93700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2380bf080 0x7fa2380b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:21.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23eb93700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2380bf500 con 0x7fa2381a0050 2021-04-09T18:24:21.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23eb93700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2380bb7b0 con 0x7fa2380bf080 2021-04-09T18:24:21.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23eb93700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2380be0d0 con 0x7fa2380bdcd0 2021-04-09T18:24:21.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23c9af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2381a0050 0x7fa2381a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:21.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23c9af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2381a0050 0x7fa2381a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51120/0 (socket says 172.21.15.32:51120) 2021-04-09T18:24:21.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23c9af700 1 -- 172.21.15.32:0/1218782652 learned_addr learned my addr 172.21.15.32:0/1218782652 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:21.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa237fff700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380bdcd0 0x7fa2380bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:21.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23d1b0700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2380bf080 0x7fa2380b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:21.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23c9af700 1 -- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380bdcd0 msgr2=0x7fa2380bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:21.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23c9af700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380bdcd0 0x7fa2380bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23c9af700 1 -- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2380bf080 msgr2=0x7fa2380b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:21.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23c9af700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2380bf080 0x7fa2380b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:21.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23c9af700 1 -- 172.21.15.32:0/1218782652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2380b1310 con 0x7fa2381a0050 2021-04-09T18:24:21.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23c9af700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2381a0050 0x7fa2381a0450 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7fa22800b580 tx=0x7fa22800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:21.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa2357fa700 1 -- 172.21.15.32:0/1218782652 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa22800bd10 con 0x7fa2381a0050 2021-04-09T18:24:21.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa2357fa700 1 -- 172.21.15.32:0/1218782652 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa228000c90 con 0x7fa2381a0050 2021-04-09T18:24:21.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2380b1310 con 0x7fa2381a0050 2021-04-09T18:24:21.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa2357fa700 1 -- 172.21.15.32:0/1218782652 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa228002220 con 0x7fa2381a0050 2021-04-09T18:24:21.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.233 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2380b1310 con 0x7fa2381a0050 2021-04-09T18:24:21.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.234 7fa2357fa700 1 -- 172.21.15.32:0/1218782652 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa228006b00 con 0x7fa2381a0050 2021-04-09T18:24:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.234 7fa2357fa700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa218018930 0x7fa21801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.234 7fa2357fa700 1 -- 172.21.15.32:0/1218782652 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa2280329f0 con 0x7fa2381a0050 2021-04-09T18:24:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.234 7fa237fff700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa218018930 0x7fa21801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:21.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.235 7fa237fff700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa218018930 0x7fa21801ae10 crc :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.254 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa210002fd0 con 0x7fa2381a0050 2021-04-09T18:24:21.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.258 7fa2357fa700 1 -- 172.21.15.32:0/1218782652 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa22800bd10 con 0x7fa2381a0050 2021-04-09T18:24:21.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:21.452 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fa210002ef0 con 0x7fa2381a0050 2021-04-09T18:24:22.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.018 7fa2357fa700 1 -- 172.21.15.32:0/1218782652 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v232) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa228006b00 con 0x7fa2381a0050 2021-04-09T18:24:22.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.070 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa218018930 msgr2=0x7fa21801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.070 7fa23eb93700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa218018930 0x7fa21801ae10 crc :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.070 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2381a0050 msgr2=0x7fa2381a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:22.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.070 7fa23eb93700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2381a0050 0x7fa2381a0450 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7fa22800b580 tx=0x7fa22800b920).stop 2021-04-09T18:24:22.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.070 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 shutdown_connections 2021-04-09T18:24:22.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.070 7fa23eb93700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa218018930 0x7fa21801ae10 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.070 7fa23eb93700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2380bf080 0x7fa2380b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.070 7fa23eb93700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2380bdcd0 0x7fa2380bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.070 7fa23eb93700 1 --2- 172.21.15.32:0/1218782652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa2381a0050 0x7fa2381a0450 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.071 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 >> 172.21.15.32:0/1218782652 conn(0x7fa2380975f0 msgr2=0x7fa238096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.071 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 shutdown_connections 2021-04-09T18:24:22.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.071 7fa23eb93700 1 -- 172.21.15.32:0/1218782652 wait complete. 2021-04-09T18:24:22.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.240 7ff9a05a7700 1 Processor -- start 2021-04-09T18:24:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.240 7ff9a05a7700 1 -- start start 2021-04-09T18:24:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff9a05a7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff99810d010 0x7ff9980d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff9a05a7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9980d1da0 0x7ff9980d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff9a05a7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99810d480 con 0x7ff9980d82c0 2021-04-09T18:24:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff9a05a7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9980d8d30 con 0x7ff9980d1da0 2021-04-09T18:24:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff9a05a7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9980d9220 con 0x7ff99810d010 2021-04-09T18:24:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99dbc2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9980d1da0 0x7ff9980d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99dbc2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9980d1da0 0x7ff9980d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51138/0 (socket says 172.21.15.32:51138) 2021-04-09T18:24:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99dbc2700 1 -- 172.21.15.32:0/463490228 learned_addr learned my addr 172.21.15.32:0/463490228 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99e3c3700 1 --2- 172.21.15.32:0/463490228 >> v2:172.21.15.86:3300/0 conn(0x7ff99810d010 0x7ff9980d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99dbc2700 1 -- 172.21.15.32:0/463490228 >> v2:172.21.15.86:3300/0 conn(0x7ff99810d010 msgr2=0x7ff9980d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99dbc2700 1 --2- 172.21.15.32:0/463490228 >> v2:172.21.15.86:3300/0 conn(0x7ff99810d010 0x7ff9980d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99dbc2700 1 -- 172.21.15.32:0/463490228 >> v1:172.21.15.86:6789/0 conn(0x7ff9980d82c0 legacy=0x7ff99810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99dbc2700 1 -- 172.21.15.32:0/463490228 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9980d86f0 con 0x7ff9980d1da0 2021-04-09T18:24:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99dbc2700 1 --2- 172.21.15.32:0/463490228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9980d1da0 0x7ff9980d7ca0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7ff98c006700 tx=0x7ff98c009b80).ready entity=mon.0 client_cookie=7d3cbfd049b8ec7a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99d3c1700 1 -- 172.21.15.32:0/463490228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff98c00c0c0 con 0x7ff9980d1da0 2021-04-09T18:24:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99d3c1700 1 -- 172.21.15.32:0/463490228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c00c440 con 0x7ff9980d1da0 2021-04-09T18:24:22.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.241 7ff99d3c1700 1 -- 172.21.15.32:0/463490228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff98c00c920 con 0x7ff9980d1da0 2021-04-09T18:24:22.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.242 7ff9a05a7700 1 -- 172.21.15.32:0/463490228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9980d1da0 msgr2=0x7ff9980d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:22.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.242 7ff9a05a7700 1 --2- 172.21.15.32:0/463490228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9980d1da0 0x7ff9980d7ca0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7ff98c006700 tx=0x7ff98c009b80).stop 2021-04-09T18:24:22.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.242 7ff9a05a7700 1 -- 172.21.15.32:0/463490228 shutdown_connections 2021-04-09T18:24:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.242 7ff9a05a7700 1 --2- 172.21.15.32:0/463490228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9980d1da0 0x7ff9980d7ca0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.242 7ff9a05a7700 1 --2- 172.21.15.32:0/463490228 >> v2:172.21.15.86:3300/0 conn(0x7ff99810d010 0x7ff9980d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.242 7ff9a05a7700 1 -- 172.21.15.32:0/463490228 >> 172.21.15.32:0/463490228 conn(0x7ff9980ce5f0 msgr2=0x7ff9980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:22.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.242 7ff9a05a7700 1 -- 172.21.15.32:0/463490228 shutdown_connections 2021-04-09T18:24:22.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.242 7ff9a05a7700 1 -- 172.21.15.32:0/463490228 wait complete. 2021-04-09T18:24:22.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.262 7ff9a05a7700 1 Processor -- start 2021-04-09T18:24:22.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.262 7ff9a05a7700 1 -- start start 2021-04-09T18:24:22.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff9a05a7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9980a0a10 0x7ff9980bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:22.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff9a05a7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff99818ad00 0x7ff99819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:22.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff9a05a7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9981a03f0 0x7ff998095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:22.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff9a05a7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99818b100 con 0x7ff9981a03f0 2021-04-09T18:24:22.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff9a05a7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff998095860 con 0x7ff99818ad00 2021-04-09T18:24:22.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff9a05a7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff998095a80 con 0x7ff9980a0a10 2021-04-09T18:24:22.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99ebc4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9981a03f0 0x7ff998095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:22.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99ebc4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9981a03f0 0x7ff998095240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51144/0 (socket says 172.21.15.32:51144) 2021-04-09T18:24:22.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99ebc4700 1 -- 172.21.15.32:0/658384326 learned_addr learned my addr 172.21.15.32:0/658384326 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:22.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99dbc2700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff99818ad00 0x7ff99819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:22.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99e3c3700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9980a0a10 0x7ff9980bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:22.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99ebc4700 1 -- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9980a0a10 msgr2=0x7ff9980bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99ebc4700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9980a0a10 0x7ff9980bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99ebc4700 1 -- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff99818ad00 msgr2=0x7ff99819fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99ebc4700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff99818ad00 0x7ff99819fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.263 7ff99ebc4700 1 -- 172.21.15.32:0/658384326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff998189250 con 0x7ff9981a03f0 2021-04-09T18:24:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.264 7ff99ebc4700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9981a03f0 0x7ff998095240 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7ff99000b8e0 tx=0x7ff99000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:22.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.264 7ff98affd700 1 -- 172.21.15.32:0/658384326 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff990000f40 con 0x7ff9981a03f0 2021-04-09T18:24:22.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.264 7ff98affd700 1 -- 172.21.15.32:0/658384326 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99000c2f0 con 0x7ff9981a03f0 2021-04-09T18:24:22.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.264 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff998189250 con 0x7ff9981a03f0 2021-04-09T18:24:22.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.264 7ff98affd700 1 -- 172.21.15.32:0/658384326 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff99000c7d0 con 0x7ff9981a03f0 2021-04-09T18:24:22.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.264 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff998189250 con 0x7ff9981a03f0 2021-04-09T18:24:22.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.264 7ff98affd700 1 -- 172.21.15.32:0/658384326 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff990001220 con 0x7ff9981a03f0 2021-04-09T18:24:22.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.265 7ff98affd700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff980018930 0x7ff98001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.265 7ff98affd700 1 -- 172.21.15.32:0/658384326 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff990032d90 con 0x7ff9981a03f0 2021-04-09T18:24:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.265 7ff99e3c3700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff980018930 0x7ff98001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.265 7ff99e3c3700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff980018930 0x7ff98001ae10 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:22.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.285 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff998093d50 con 0x7ff9981a03f0 2021-04-09T18:24:22.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.288 7ff98affd700 1 -- 172.21.15.32:0/658384326 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff99000bdc0 con 0x7ff9981a03f0 2021-04-09T18:24:22.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:22.481 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7ff99810d480 con 0x7ff9981a03f0 2021-04-09T18:24:23.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.028 7ff98affd700 1 -- 172.21.15.32:0/658384326 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v234) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff990001220 con 0x7ff9981a03f0 2021-04-09T18:24:23.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff980018930 msgr2=0x7ff98001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff980018930 0x7ff98001ae10 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9981a03f0 msgr2=0x7ff998095240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9981a03f0 0x7ff998095240 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7ff99000b8e0 tx=0x7ff99000bc80).stop 2021-04-09T18:24:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 shutdown_connections 2021-04-09T18:24:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff980018930 0x7ff98001ae10 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9981a03f0 0x7ff998095240 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff99818ad00 0x7ff99819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 --2- 172.21.15.32:0/658384326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9980a0a10 0x7ff9980bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 >> 172.21.15.32:0/658384326 conn(0x7ff9980975f0 msgr2=0x7ff998096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:23.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 shutdown_connections 2021-04-09T18:24:23.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.048 7ff9a05a7700 1 -- 172.21.15.32:0/658384326 wait complete. 2021-04-09T18:24:23.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:24:23.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.213 7fec24d59700 1 Processor -- start 2021-04-09T18:24:23.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.213 7fec24d59700 1 -- start start 2021-04-09T18:24:23.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.213 7fec24d59700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec2010d010 0x7fec200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:23.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.213 7fec24d59700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec200d1da0 0x7fec200d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:23.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.213 7fec24d59700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2010d480 con 0x7fec200d82c0 2021-04-09T18:24:23.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.213 7fec24d59700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec200d9220 con 0x7fec2010d010 2021-04-09T18:24:23.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.213 7fec24d59700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec200d86c0 con 0x7fec200d1da0 2021-04-09T18:24:23.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec16312700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec200d1da0 0x7fec200d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:23.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec1eb13700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec2010d010 0x7fec200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:23.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec16312700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec200d1da0 0x7fec200d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:58384/0 (socket says 172.21.15.32:58384) 2021-04-09T18:24:23.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec1f314700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fec200d82c0 0x7fec2010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58672/0 (socket says 172.21.15.32:58672) 2021-04-09T18:24:23.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec16312700 1 -- 172.21.15.32:0/3666424526 learned_addr learned my addr 172.21.15.32:0/3666424526 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:23.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec1eb13700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec2010d010 0x7fec200d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:42912/0 (socket says 172.21.15.32:42912) 2021-04-09T18:24:23.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec16312700 1 -- 172.21.15.32:0/3666424526 >> v2:172.21.15.86:3300/0 conn(0x7fec2010d010 msgr2=0x7fec200d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:23.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec16312700 1 --2- 172.21.15.32:0/3666424526 >> v2:172.21.15.86:3300/0 conn(0x7fec2010d010 0x7fec200d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec16312700 1 -- 172.21.15.32:0/3666424526 >> v1:172.21.15.86:6789/0 conn(0x7fec200d82c0 legacy=0x7fec2010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:23.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec16312700 1 -- 172.21.15.32:0/3666424526 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2010e1f0 con 0x7fec200d1da0 2021-04-09T18:24:23.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec1e312700 1 -- 172.21.15.32:0/3666424526 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244274342 0 0) 0x7fec18000d30 con 0x7fec200d82c0 2021-04-09T18:24:23.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec1eb13700 1 --2- 172.21.15.32:0/3666424526 >> v2:172.21.15.86:3300/0 conn(0x7fec2010d010 0x7fec200d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:23.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec16312700 1 --2- 172.21.15.32:0/3666424526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec200d1da0 0x7fec200d7ca0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fec08006700 tx=0x7fec08009b80).ready entity=mon.2 client_cookie=5b056d12166362d0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:23.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec1e312700 1 -- 172.21.15.32:0/3666424526 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec0800c0c0 con 0x7fec200d1da0 2021-04-09T18:24:23.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec1e312700 1 -- 172.21.15.32:0/3666424526 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0800c440 con 0x7fec200d1da0 2021-04-09T18:24:23.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.214 7fec1e312700 1 -- 172.21.15.32:0/3666424526 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec0800c920 con 0x7fec200d1da0 2021-04-09T18:24:23.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.215 7fec24d59700 1 -- 172.21.15.32:0/3666424526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec200d1da0 msgr2=0x7fec200d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.215 7fec24d59700 1 --2- 172.21.15.32:0/3666424526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec200d1da0 0x7fec200d7ca0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fec08006700 tx=0x7fec08009b80).stop 2021-04-09T18:24:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.215 7fec24d59700 1 -- 172.21.15.32:0/3666424526 shutdown_connections 2021-04-09T18:24:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.215 7fec24d59700 1 --2- 172.21.15.32:0/3666424526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec200d1da0 0x7fec200d7ca0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.215 7fec24d59700 1 --2- 172.21.15.32:0/3666424526 >> v2:172.21.15.86:3300/0 conn(0x7fec2010d010 0x7fec200d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.215 7fec24d59700 1 -- 172.21.15.32:0/3666424526 >> 172.21.15.32:0/3666424526 conn(0x7fec200ce5f0 msgr2=0x7fec200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:23.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.215 7fec24d59700 1 -- 172.21.15.32:0/3666424526 shutdown_connections 2021-04-09T18:24:23.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.215 7fec24d59700 1 -- 172.21.15.32:0/3666424526 wait complete. 2021-04-09T18:24:23.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.235 7fec24d59700 1 Processor -- start 2021-04-09T18:24:23.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec24d59700 1 -- start start 2021-04-09T18:24:23.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec24d59700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec200b2140 0x7fec2008b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:23.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec24d59700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec2009f550 0x7fec2009f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:23.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec24d59700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec20197c50 0x7fec20198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:23.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec24d59700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2009f010 con 0x7fec200b2140 2021-04-09T18:24:23.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec24d59700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2009ff70 con 0x7fec20197c50 2021-04-09T18:24:23.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec24d59700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec20198670 con 0x7fec2009f550 2021-04-09T18:24:23.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec1eb13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec200b2140 0x7fec2008b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:23.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec1eb13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec200b2140 0x7fec2008b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51160/0 (socket says 172.21.15.32:51160) 2021-04-09T18:24:23.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.236 7fec1eb13700 1 -- 172.21.15.32:0/2635304751 learned_addr learned my addr 172.21.15.32:0/2635304751 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:23.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec16312700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec2009f550 0x7fec2009f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:23.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec1f314700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec20197c50 0x7fec20198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:23.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec1eb13700 1 -- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec2009f550 msgr2=0x7fec2009f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:23.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec1eb13700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec2009f550 0x7fec2009f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec1eb13700 1 -- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec20197c50 msgr2=0x7fec20198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:23.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec1eb13700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec20197c50 0x7fec20198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec1eb13700 1 -- 172.21.15.32:0/2635304751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec20198890 con 0x7fec200b2140 2021-04-09T18:24:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec1f314700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec20197c50 0x7fec20198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec1eb13700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec200b2140 0x7fec2008b930 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7fec10006f40 tx=0x7fec1000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec17fff700 1 -- 172.21.15.32:0/2635304751 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec1000bd80 con 0x7fec200b2140 2021-04-09T18:24:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec17fff700 1 -- 172.21.15.32:0/2635304751 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec10000c90 con 0x7fec200b2140 2021-04-09T18:24:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec24d59700 1 -- 172.21.15.32:0/2635304751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec20198890 con 0x7fec200b2140 2021-04-09T18:24:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec17fff700 1 -- 172.21.15.32:0/2635304751 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec10002220 con 0x7fec200b2140 2021-04-09T18:24:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.237 7fec24d59700 1 -- 172.21.15.32:0/2635304751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec20198890 con 0x7fec200b2140 2021-04-09T18:24:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.238 7fec17fff700 1 -- 172.21.15.32:0/2635304751 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fec1000a000 con 0x7fec200b2140 2021-04-09T18:24:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.238 7fec17fff700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec04018930 0x7fec0401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.238 7fec17fff700 1 -- 172.21.15.32:0/2635304751 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fec100329e0 con 0x7fec200b2140 2021-04-09T18:24:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.238 7fec16312700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec04018930 0x7fec0401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:23.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.238 7fec16312700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec04018930 0x7fec0401ae10 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:23.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.258 7fec24d59700 1 -- 172.21.15.32:0/2635304751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2009f010 con 0x7fec200b2140 2021-04-09T18:24:23.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.261 7fec17fff700 1 -- 172.21.15.32:0/2635304751 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fec10000c90 con 0x7fec200b2140 2021-04-09T18:24:23.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:23.453 7fec24d59700 1 -- 172.21.15.32:0/2635304751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7fec2009f010 con 0x7fec200b2140 2021-04-09T18:24:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.028 7fec17fff700 1 -- 172.21.15.32:0/2635304751 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v235) v1 ==== 105+0+0 (secure 0 0 0) 0x7fec1000a000 con 0x7fec200b2140 2021-04-09T18:24:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 -- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec04018930 msgr2=0x7fec0401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec04018930 0x7fec0401ae10 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 -- 172.21.15.32:0/2635304751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec200b2140 msgr2=0x7fec2008b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec200b2140 0x7fec2008b930 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7fec10006f40 tx=0x7fec1000b990).stop 2021-04-09T18:24:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 -- 172.21.15.32:0/2635304751 shutdown_connections 2021-04-09T18:24:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec04018930 0x7fec0401ae10 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec20197c50 0x7fec20198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec2009f550 0x7fec2009f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 --2- 172.21.15.32:0/2635304751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec200b2140 0x7fec2008b930 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 -- 172.21.15.32:0/2635304751 >> 172.21.15.32:0/2635304751 conn(0x7fec200b27e0 msgr2=0x7fec201b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 -- 172.21.15.32:0/2635304751 shutdown_connections 2021-04-09T18:24:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.071 7fec24d59700 1 -- 172.21.15.32:0/2635304751 wait complete. 2021-04-09T18:24:24.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:24.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.237 7fe70d203700 1 Processor -- start 2021-04-09T18:24:24.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.237 7fe70d203700 1 -- start start 2021-04-09T18:24:24.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.238 7fe70d203700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe70810d010 0x7fe7080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:24.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.238 7fe70d203700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70810d480 con 0x7fe7080d1da0 2021-04-09T18:24:24.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.238 7fe70d203700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70810e1f0 con 0x7fe7080d8490 2021-04-09T18:24:24.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.238 7fe70d203700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70810e780 con 0x7fe70810d010 2021-04-09T18:24:24.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.238 7fe70659c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe7080d1da0 0x7fe7080d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60760/0 (socket says 172.21.15.32:60760) 2021-04-09T18:24:24.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.238 7fe70659c700 1 -- 172.21.15.32:0/1383226897 learned_addr learned my addr 172.21.15.32:0/1383226897 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:24.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe706d9d700 1 --2- 172.21.15.32:0/1383226897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe70810d010 0x7fe7080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:24.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe705d9b700 1 -- 172.21.15.32:0/1383226897 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659266071 0 0) 0x7fe6fc000ce0 con 0x7fe7080d1da0 2021-04-09T18:24:24.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe705d9b700 1 -- 172.21.15.32:0/1383226897 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f0002fe0 con 0x7fe7080d1da0 2021-04-09T18:24:24.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe705d9b700 1 -- 172.21.15.32:0/1383226897 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219512165 0 0) 0x7fe700001400 con 0x7fe7080d8490 2021-04-09T18:24:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe705d9b700 1 -- 172.21.15.32:0/1383226897 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f0003210 con 0x7fe7080d8490 2021-04-09T18:24:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe705d9b700 1 -- 172.21.15.32:0/1383226897 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3913406699 0 0) 0x7fe6fc0010e0 con 0x7fe7080d1da0 2021-04-09T18:24:24.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe706d9d700 1 -- 172.21.15.32:0/1383226897 >> v1:172.21.15.86:6789/0 conn(0x7fe7080d8490 legacy=0x7fe70810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:24.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe706d9d700 1 -- 172.21.15.32:0/1383226897 >> v1:172.21.15.32:6789/0 conn(0x7fe7080d1da0 legacy=0x7fe7080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:24.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe706d9d700 1 -- 172.21.15.32:0/1383226897 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7080d8d30 con 0x7fe70810d010 2021-04-09T18:24:24.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe706d9d700 1 --2- 172.21.15.32:0/1383226897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe70810d010 0x7fe7080d1780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fe6f8006650 tx=0x7fe6f8009a80).ready entity=mon.2 client_cookie=5862fd79b4541487 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:24.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe705d9b700 1 -- 172.21.15.32:0/1383226897 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6f80009b0 con 0x7fe70810d010 2021-04-09T18:24:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe705d9b700 1 -- 172.21.15.32:0/1383226897 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f800c0c0 con 0x7fe70810d010 2021-04-09T18:24:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe705d9b700 1 -- 172.21.15.32:0/1383226897 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6f800c510 con 0x7fe70810d010 2021-04-09T18:24:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe70d203700 1 -- 172.21.15.32:0/1383226897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe70810d010 msgr2=0x7fe7080d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:24.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.239 7fe70d203700 1 --2- 172.21.15.32:0/1383226897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe70810d010 0x7fe7080d1780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fe6f8006650 tx=0x7fe6f8009a80).stop 2021-04-09T18:24:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.240 7fe70d203700 1 -- 172.21.15.32:0/1383226897 shutdown_connections 2021-04-09T18:24:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.240 7fe70d203700 1 --2- 172.21.15.32:0/1383226897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe70810d010 0x7fe7080d1780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.240 7fe70d203700 1 -- 172.21.15.32:0/1383226897 >> 172.21.15.32:0/1383226897 conn(0x7fe7080ce5f0 msgr2=0x7fe7080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.240 7fe70d203700 1 -- 172.21.15.32:0/1383226897 shutdown_connections 2021-04-09T18:24:24.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.240 7fe70d203700 1 -- 172.21.15.32:0/1383226897 wait complete. 2021-04-09T18:24:24.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.260 7fe70d203700 1 Processor -- start 2021-04-09T18:24:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.260 7fe70d203700 1 -- start start 2021-04-09T18:24:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70d203700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7080c3c00 0x7fe70809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70d203700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe708095240 0x7fe7081a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70d203700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7081a1270 0x7fe7080b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70d203700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70809cda0 con 0x7fe7081a1270 2021-04-09T18:24:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70d203700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7080c3680 con 0x7fe7080c3c00 2021-04-09T18:24:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70d203700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe708095640 con 0x7fe708095240 2021-04-09T18:24:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70759e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7081a1270 0x7fe7080b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70759e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7081a1270 0x7fe7080b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51176/0 (socket says 172.21.15.32:51176) 2021-04-09T18:24:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70759e700 1 -- 172.21.15.32:0/2213186840 learned_addr learned my addr 172.21.15.32:0/2213186840 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe706d9d700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7080c3c00 0x7fe70809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:24.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70659c700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe708095240 0x7fe7081a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:24.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70759e700 1 -- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe708095240 msgr2=0x7fe7081a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70759e700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe708095240 0x7fe7081a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70759e700 1 -- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7080c3c00 msgr2=0x7fe70809c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70759e700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7080c3c00 0x7fe70809c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70759e700 1 -- 172.21.15.32:0/2213186840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7081a1670 con 0x7fe7081a1270 2021-04-09T18:24:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe706d9d700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7080c3c00 0x7fe70809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.261 7fe70759e700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7081a1270 0x7fe7080b0cf0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7fe700009980 tx=0x7fe70000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.262 7fe6f77fe700 1 -- 172.21.15.32:0/2213186840 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe700003020 con 0x7fe7081a1270 2021-04-09T18:24:24.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.262 7fe6f77fe700 1 -- 172.21.15.32:0/2213186840 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe700005870 con 0x7fe7081a1270 2021-04-09T18:24:24.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.262 7fe6f77fe700 1 -- 172.21.15.32:0/2213186840 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe700005d50 con 0x7fe7081a1270 2021-04-09T18:24:24.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.262 7fe70d203700 1 -- 172.21.15.32:0/2213186840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7081a1670 con 0x7fe7081a1270 2021-04-09T18:24:24.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.262 7fe70d203700 1 -- 172.21.15.32:0/2213186840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7081a1670 con 0x7fe7081a1270 2021-04-09T18:24:24.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.262 7fe6f77fe700 1 -- 172.21.15.32:0/2213186840 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe700005a50 con 0x7fe7081a1270 2021-04-09T18:24:24.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.263 7fe6f77fe700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6ec0188c0 0x7fe6ec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:24.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.263 7fe6f77fe700 1 -- 172.21.15.32:0/2213186840 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe700032b30 con 0x7fe7081a1270 2021-04-09T18:24:24.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.263 7fe706d9d700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6ec0188c0 0x7fe6ec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:24.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.263 7fe706d9d700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6ec0188c0 0x7fe6ec01adc0 crc :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:24.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.282 7fe70d203700 1 -- 172.21.15.32:0/2213186840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe708093d50 con 0x7fe7081a1270 2021-04-09T18:24:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.286 7fe6f77fe700 1 -- 172.21.15.32:0/2213186840 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe700030f10 con 0x7fe7081a1270 2021-04-09T18:24:24.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:24.482 7fe70d203700 1 -- 172.21.15.32:0/2213186840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7fe70810d480 con 0x7fe7081a1270 2021-04-09T18:24:25.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.035 7fe6f77fe700 1 -- 172.21.15.32:0/2213186840 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v237) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe700030f10 con 0x7fe7081a1270 2021-04-09T18:24:25.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.049 7fe70d203700 1 -- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6ec0188c0 msgr2=0x7fe6ec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:25.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.049 7fe70d203700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6ec0188c0 0x7fe6ec01adc0 crc :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.049 7fe70d203700 1 -- 172.21.15.32:0/2213186840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7081a1270 msgr2=0x7fe7080b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.049 7fe70d203700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7081a1270 0x7fe7080b0cf0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7fe700009980 tx=0x7fe70000bcd0).stop 2021-04-09T18:24:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.049 7fe70d203700 1 -- 172.21.15.32:0/2213186840 shutdown_connections 2021-04-09T18:24:25.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.049 7fe70d203700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe6ec0188c0 0x7fe6ec01adc0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.049 7fe70d203700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7081a1270 0x7fe7080b0cf0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.049 7fe70d203700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe708095240 0x7fe7081a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.049 7fe70d203700 1 --2- 172.21.15.32:0/2213186840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7080c3c00 0x7fe70809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.050 7fe70d203700 1 -- 172.21.15.32:0/2213186840 >> 172.21.15.32:0/2213186840 conn(0x7fe7080975f0 msgr2=0x7fe708096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:25.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.050 7fe70d203700 1 -- 172.21.15.32:0/2213186840 shutdown_connections 2021-04-09T18:24:25.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.050 7fe70d203700 1 -- 172.21.15.32:0/2213186840 wait complete. 2021-04-09T18:24:25.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:25.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.215 7f27aca7b700 1 Processor -- start 2021-04-09T18:24:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.215 7f27aca7b700 1 -- start start 2021-04-09T18:24:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.216 7f27aca7b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f27a80d7ca0 0x7f27a80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.216 7f27aca7b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27a810da00 0x7f27a810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.216 7f27aca7b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a810d480 con 0x7f27a810d010 2021-04-09T18:24:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.216 7f27aca7b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a80d9220 con 0x7f27a810da00 2021-04-09T18:24:25.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.216 7f27aca7b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a80d86c0 con 0x7f27a80d7ca0 2021-04-09T18:24:25.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.216 7f27a705f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27a810da00 0x7f27a810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:25.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.216 7f27a685e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f27a810d010 0x7f27a80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60772/0 (socket says 172.21.15.32:60772) 2021-04-09T18:24:25.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.216 7f27a685e700 1 -- 172.21.15.32:0/1680493843 learned_addr learned my addr 172.21.15.32:0/1680493843 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:25.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.216 7f27a605d700 1 --2- 172.21.15.32:0/1680493843 >> v2:172.21.15.86:3300/0 conn(0x7f27a80d7ca0 0x7f27a80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:25.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27a705f700 1 -- 172.21.15.32:0/1680493843 >> v2:172.21.15.86:3300/0 conn(0x7f27a80d7ca0 msgr2=0x7f27a80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:25.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27a705f700 1 --2- 172.21.15.32:0/1680493843 >> v2:172.21.15.86:3300/0 conn(0x7f27a80d7ca0 0x7f27a80d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27a705f700 1 -- 172.21.15.32:0/1680493843 >> v1:172.21.15.32:6789/0 conn(0x7f27a810d010 legacy=0x7f27a80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:25.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27a705f700 1 -- 172.21.15.32:0/1680493843 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a80d1f70 con 0x7f27a810da00 2021-04-09T18:24:25.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27a585c700 1 -- 172.21.15.32:0/1680493843 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681041917 0 0) 0x7f2798000ce0 con 0x7f27a810d010 2021-04-09T18:24:25.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27a705f700 1 --2- 172.21.15.32:0/1680493843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a810da00 0x7f27a810de00 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7f27a0006e30 tx=0x7f27a0009de0).ready entity=mon.0 client_cookie=65f9b96644107a54 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:25.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27a585c700 1 -- 172.21.15.32:0/1680493843 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a000b300 con 0x7f27a810da00 2021-04-09T18:24:25.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27a585c700 1 -- 172.21.15.32:0/1680493843 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a000b680 con 0x7f27a810da00 2021-04-09T18:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27a585c700 1 -- 172.21.15.32:0/1680493843 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a000bb60 con 0x7f27a810da00 2021-04-09T18:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27aca7b700 1 -- 172.21.15.32:0/1680493843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a810da00 msgr2=0x7f27a810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.217 7f27aca7b700 1 --2- 172.21.15.32:0/1680493843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a810da00 0x7f27a810de00 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7f27a0006e30 tx=0x7f27a0009de0).stop 2021-04-09T18:24:25.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.218 7f27aca7b700 1 -- 172.21.15.32:0/1680493843 shutdown_connections 2021-04-09T18:24:25.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.218 7f27aca7b700 1 --2- 172.21.15.32:0/1680493843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a810da00 0x7f27a810de00 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.218 7f27aca7b700 1 --2- 172.21.15.32:0/1680493843 >> v2:172.21.15.86:3300/0 conn(0x7f27a80d7ca0 0x7f27a80d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.218 7f27aca7b700 1 -- 172.21.15.32:0/1680493843 >> 172.21.15.32:0/1680493843 conn(0x7f27a80ce5f0 msgr2=0x7f27a80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.218 7f27aca7b700 1 -- 172.21.15.32:0/1680493843 shutdown_connections 2021-04-09T18:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.218 7f27aca7b700 1 -- 172.21.15.32:0/1680493843 wait complete. 2021-04-09T18:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.246 7f27aca7b700 1 Processor -- start 2021-04-09T18:24:25.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.246 7f27aca7b700 1 -- start start 2021-04-09T18:24:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27aca7b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a808a910 0x7f27a809e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27aca7b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a808ad10 0x7f27a809ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:25.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27aca7b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a8092380 0x7f27a8092780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:25.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27aca7b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a808b110 con 0x7f27a808ad10 2021-04-09T18:24:25.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27aca7b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a809e0d0 con 0x7f27a808a910 2021-04-09T18:24:25.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27aca7b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a808b3f0 con 0x7f27a8092380 2021-04-09T18:24:25.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a605d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a808ad10 0x7f27a809ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:25.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a605d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a808ad10 0x7f27a809ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51192/0 (socket says 172.21.15.32:51192) 2021-04-09T18:24:25.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a605d700 1 -- 172.21.15.32:0/4070764109 learned_addr learned my addr 172.21.15.32:0/4070764109 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:25.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a685e700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a808a910 0x7f27a809e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:25.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a705f700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a8092380 0x7f27a8092780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:25.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a605d700 1 -- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a8092380 msgr2=0x7f27a8092780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:25.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a605d700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a8092380 0x7f27a8092780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a605d700 1 -- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a808a910 msgr2=0x7f27a809e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a605d700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a808a910 0x7f27a809e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.247 7f27a605d700 1 -- 172.21.15.32:0/4070764109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a8092da0 con 0x7f27a808ad10 2021-04-09T18:24:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.248 7f27a605d700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a808ad10 0x7f27a809ec70 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f279c00b560 tx=0x7f279c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:25.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.248 7f27977fe700 1 -- 172.21.15.32:0/4070764109 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f279c00bcf0 con 0x7f27a808ad10 2021-04-09T18:24:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.248 7f27977fe700 1 -- 172.21.15.32:0/4070764109 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279c000c90 con 0x7f27a808ad10 2021-04-09T18:24:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.248 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a8092da0 con 0x7f27a808ad10 2021-04-09T18:24:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.248 7f27977fe700 1 -- 172.21.15.32:0/4070764109 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f279c002220 con 0x7f27a808ad10 2021-04-09T18:24:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.248 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a8092da0 con 0x7f27a808ad10 2021-04-09T18:24:25.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.248 7f27977fe700 1 -- 172.21.15.32:0/4070764109 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f279c0052e0 con 0x7f27a808ad10 2021-04-09T18:24:25.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.248 7f27977fe700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:25.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.248 7f27977fe700 1 -- 172.21.15.32:0/4070764109 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f279c032a20 con 0x7f27a808ad10 2021-04-09T18:24:25.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.249 7f27a685e700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:25.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.249 7f27a685e700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 crc :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:25.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.260 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a818fc90 con 0x7f27a808ad10 2021-04-09T18:24:25.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.264 7f27977fe700 1 -- 172.21.15.32:0/4070764109 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f279c000c90 con 0x7f27a808ad10 2021-04-09T18:24:25.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:25.460 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f27a818fc90 con 0x7f27a808ad10 2021-04-09T18:24:26.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.037 7f27977fe700 1 -- 172.21.15.32:0/4070764109 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v238) v1 ==== 105+0+0 (secure 0 0 0) 0x7f279c0052e0 con 0x7f27a808ad10 2021-04-09T18:24:26.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.077 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 msgr2=0x7f278c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:26.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.077 7f27aca7b700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 crc :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:26.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.077 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a808ad10 msgr2=0x7f27a809ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:26.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.077 7f27aca7b700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a808ad10 0x7f27a809ec70 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f279c00b560 tx=0x7f279c00b900).stop 2021-04-09T18:24:26.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.078 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 shutdown_connections 2021-04-09T18:24:26.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.078 7f27aca7b700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.078 7f27aca7b700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27a808ad10 0x7f27a809ec70 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.078 7f27aca7b700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a8092380 0x7f27a8092780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.078 7f27aca7b700 1 --2- 172.21.15.32:0/4070764109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a808a910 0x7f27a809e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.078 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 >> 172.21.15.32:0/4070764109 conn(0x7f27a80b2fa0 msgr2=0x7f27a80a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.078 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 shutdown_connections 2021-04-09T18:24:26.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.078 7f27aca7b700 1 -- 172.21.15.32:0/4070764109 wait complete. 2021-04-09T18:24:26.150 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:24:26.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.242 7f79a0ad5700 1 Processor -- start 2021-04-09T18:24:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.242 7f79a0ad5700 1 -- start start 2021-04-09T18:24:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.243 7f79a0ad5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c10da00 0x7f799c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.243 7f79a0ad5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c10d480 con 0x7f799c0d7ca0 2021-04-09T18:24:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.243 7f79a0ad5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799c0d1f70 con 0x7f799c10d010 2021-04-09T18:24:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.243 7f79a0ad5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799c0d9220 con 0x7f799c10da00 2021-04-09T18:24:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.243 7f799a05d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f799c0d7ca0 0x7f799c0d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60788/0 (socket says 172.21.15.32:60788) 2021-04-09T18:24:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.243 7f799a05d700 1 -- 172.21.15.32:0/2040457806 learned_addr learned my addr 172.21.15.32:0/2040457806 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.243 7f799b05f700 1 --2- 172.21.15.32:0/2040457806 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c10da00 0x7f799c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799985c700 1 -- 172.21.15.32:0/2040457806 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9077429 0 0) 0x7f7990000ce0 con 0x7f799c0d7ca0 2021-04-09T18:24:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799985c700 1 -- 172.21.15.32:0/2040457806 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7984002fe0 con 0x7f799c0d7ca0 2021-04-09T18:24:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799985c700 1 -- 172.21.15.32:0/2040457806 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956983809 0 0) 0x7f798c000c60 con 0x7f799c10d010 2021-04-09T18:24:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799985c700 1 -- 172.21.15.32:0/2040457806 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7984003210 con 0x7f799c10d010 2021-04-09T18:24:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799b05f700 1 -- 172.21.15.32:0/2040457806 >> v1:172.21.15.86:6789/0 conn(0x7f799c10d010 legacy=0x7f799c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799b05f700 1 -- 172.21.15.32:0/2040457806 >> v1:172.21.15.32:6789/0 conn(0x7f799c0d7ca0 legacy=0x7f799c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799985c700 1 -- 172.21.15.32:0/2040457806 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2881519498 0 0) 0x7f79900010e0 con 0x7f799c0d7ca0 2021-04-09T18:24:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799b05f700 1 -- 172.21.15.32:0/2040457806 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c10e420 con 0x7f799c10da00 2021-04-09T18:24:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799b05f700 1 --2- 172.21.15.32:0/2040457806 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c10da00 0x7f799c10de00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f7994006e50 tx=0x7f7994009df0).ready entity=mon.2 client_cookie=e4190d0040e87e1b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799985c700 1 -- 172.21.15.32:0/2040457806 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f799400b510 con 0x7f799c10da00 2021-04-09T18:24:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799985c700 1 -- 172.21.15.32:0/2040457806 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7994003030 con 0x7f799c10da00 2021-04-09T18:24:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f799985c700 1 -- 172.21.15.32:0/2040457806 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f799400b7d0 con 0x7f799c10da00 2021-04-09T18:24:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f79a0ad5700 1 -- 172.21.15.32:0/2040457806 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c10da00 msgr2=0x7f799c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.244 7f79a0ad5700 1 --2- 172.21.15.32:0/2040457806 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c10da00 0x7f799c10de00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f7994006e50 tx=0x7f7994009df0).stop 2021-04-09T18:24:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.245 7f79a0ad5700 1 -- 172.21.15.32:0/2040457806 shutdown_connections 2021-04-09T18:24:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.245 7f79a0ad5700 1 --2- 172.21.15.32:0/2040457806 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c10da00 0x7f799c10de00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.245 7f79a0ad5700 1 -- 172.21.15.32:0/2040457806 >> 172.21.15.32:0/2040457806 conn(0x7f799c0ce5f0 msgr2=0x7f799c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:26.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.245 7f79a0ad5700 1 -- 172.21.15.32:0/2040457806 shutdown_connections 2021-04-09T18:24:26.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.245 7f79a0ad5700 1 -- 172.21.15.32:0/2040457806 wait complete. 2021-04-09T18:24:26.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.265 7f79a0ad5700 1 Processor -- start 2021-04-09T18:24:26.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.265 7f79a0ad5700 1 -- start start 2021-04-09T18:24:26.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f79a0ad5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799c0c3c00 0x7f799c09c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:26.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f79a0ad5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c095240 0x7f799c1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f79a0ad5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c1a1270 0x7f799c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f79a0ad5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c09cda0 con 0x7f799c0c3c00 2021-04-09T18:24:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f79a0ad5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c0c3680 con 0x7f799c1a1270 2021-04-09T18:24:26.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f79a0ad5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799c095640 con 0x7f799c095240 2021-04-09T18:24:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799c0c3c00 0x7f799c09c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799c0c3c00 0x7f799c09c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51202/0 (socket says 172.21.15.32:51202) 2021-04-09T18:24:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a85e700 1 -- 172.21.15.32:0/1932680750 learned_addr learned my addr 172.21.15.32:0/1932680750 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:26.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a05d700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c095240 0x7f799c1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:26.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799b05f700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c1a1270 0x7f799c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:26.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a85e700 1 -- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c095240 msgr2=0x7f799c1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:26.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a85e700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c095240 0x7f799c1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:26.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a85e700 1 -- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c1a1270 msgr2=0x7f799c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a85e700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c1a1270 0x7f799c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a85e700 1 -- 172.21.15.32:0/1932680750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c1a1670 con 0x7f799c0c3c00 2021-04-09T18:24:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f799a85e700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799c0c3c00 0x7f799c09c600 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f798c00b580 tx=0x7f798c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:26.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.266 7f798b7fe700 1 -- 172.21.15.32:0/1932680750 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f798c00bd10 con 0x7f799c0c3c00 2021-04-09T18:24:26.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.267 7f798b7fe700 1 -- 172.21.15.32:0/1932680750 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c000e20 con 0x7f799c0c3c00 2021-04-09T18:24:26.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.267 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799c1a1670 con 0x7f799c0c3c00 2021-04-09T18:24:26.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.267 7f798b7fe700 1 -- 172.21.15.32:0/1932680750 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f798c0054f0 con 0x7f799c0c3c00 2021-04-09T18:24:26.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.267 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c1a1670 con 0x7f799c0c3c00 2021-04-09T18:24:26.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.267 7f798b7fe700 1 -- 172.21.15.32:0/1932680750 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f798c009fe0 con 0x7f799c0c3c00 2021-04-09T18:24:26.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.267 7f798b7fe700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79800188d0 0x7f798001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:26.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.267 7f798b7fe700 1 -- 172.21.15.32:0/1932680750 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f798c032a50 con 0x7f799c0c3c00 2021-04-09T18:24:26.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.267 7f799a05d700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79800188d0 0x7f798001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:26.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.268 7f799a05d700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79800188d0 0x7f798001add0 crc :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:26.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.287 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799c093d50 con 0x7f799c0c3c00 2021-04-09T18:24:26.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.291 7f798b7fe700 1 -- 172.21.15.32:0/1932680750 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f798c000e20 con 0x7f799c0c3c00 2021-04-09T18:24:26.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:26.487 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f799c10d480 con 0x7f799c0c3c00 2021-04-09T18:24:27.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.037 7f798b7fe700 1 -- 172.21.15.32:0/1932680750 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v239) v1 ==== 105+0+0 (secure 0 0 0) 0x7f798c009fe0 con 0x7f799c0c3c00 2021-04-09T18:24:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79800188d0 msgr2=0x7f798001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79800188d0 0x7f798001add0 crc :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799c0c3c00 msgr2=0x7f799c09c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799c0c3c00 0x7f799c09c600 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f798c00b580 tx=0x7f798c00b920).stop 2021-04-09T18:24:27.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 shutdown_connections 2021-04-09T18:24:27.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79800188d0 0x7f798001add0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:27.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c1a1270 0x7f799c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:27.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c095240 0x7f799c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:27.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 --2- 172.21.15.32:0/1932680750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f799c0c3c00 0x7f799c09c600 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:27.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 >> 172.21.15.32:0/1932680750 conn(0x7f799c0975f0 msgr2=0x7f799c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:27.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 shutdown_connections 2021-04-09T18:24:27.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.054 7f79a0ad5700 1 -- 172.21.15.32:0/1932680750 wait complete. 2021-04-09T18:24:27.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2021-04-09T18:24:27.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.220 7f52f2ee0700 1 Processor -- start 2021-04-09T18:24:27.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.221 7f52f2ee0700 1 -- start start 2021-04-09T18:24:27.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.221 7f52f2ee0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f52ec0d7ca0 0x7f52ec0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.221 7f52f2ee0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec10d480 con 0x7f52ec10da00 2021-04-09T18:24:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.221 7f52f2ee0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0d86c0 con 0x7f52ec10d010 2021-04-09T18:24:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.221 7f52f2ee0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ec0d9220 con 0x7f52ec0d7ca0 2021-04-09T18:24:27.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.221 7f52f14fd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f52ec10da00 0x7f52ec10de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60800/0 (socket says 172.21.15.32:60800) 2021-04-09T18:24:27.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.221 7f52f14fd700 1 -- 172.21.15.32:0/3734203250 learned_addr learned my addr 172.21.15.32:0/3734203250 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:27.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52ebfff700 1 --2- 172.21.15.32:0/3734203250 >> v2:172.21.15.86:3300/0 conn(0x7f52ec0d7ca0 0x7f52ec0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:27.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52eb7fe700 1 -- 172.21.15.32:0/3734203250 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348750045 0 0) 0x7f52e4000d30 con 0x7f52ec10da00 2021-04-09T18:24:27.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52eb7fe700 1 -- 172.21.15.32:0/3734203250 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d8002fe0 con 0x7f52ec10da00 2021-04-09T18:24:27.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52eb7fe700 1 -- 172.21.15.32:0/3734203250 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2418194462 0 0) 0x7f52e4001460 con 0x7f52ec10da00 2021-04-09T18:24:27.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52ebfff700 1 -- 172.21.15.32:0/3734203250 >> v1:172.21.15.86:6789/0 conn(0x7f52ec10d010 legacy=0x7f52ec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52ebfff700 1 -- 172.21.15.32:0/3734203250 >> v1:172.21.15.32:6789/0 conn(0x7f52ec10da00 legacy=0x7f52ec10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52ebfff700 1 -- 172.21.15.32:0/3734203250 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0d1f70 con 0x7f52ec0d7ca0 2021-04-09T18:24:27.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52ebfff700 1 --2- 172.21.15.32:0/3734203250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52ec0d7ca0 0x7f52ec0d80a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f52dc006650 tx=0x7f52dc009a00).ready entity=mon.1 client_cookie=ae39f8fe072eb353 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52eb7fe700 1 -- 172.21.15.32:0/3734203250 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52dc009df0 con 0x7f52ec0d7ca0 2021-04-09T18:24:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52eb7fe700 1 -- 172.21.15.32:0/3734203250 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52dc0009b0 con 0x7f52ec0d7ca0 2021-04-09T18:24:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52eb7fe700 1 -- 172.21.15.32:0/3734203250 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52dc00c2b0 con 0x7f52ec0d7ca0 2021-04-09T18:24:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52f2ee0700 1 -- 172.21.15.32:0/3734203250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52ec0d7ca0 msgr2=0x7f52ec0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.222 7f52f2ee0700 1 --2- 172.21.15.32:0/3734203250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52ec0d7ca0 0x7f52ec0d80a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f52dc006650 tx=0x7f52dc009a00).stop 2021-04-09T18:24:27.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.223 7f52f2ee0700 1 -- 172.21.15.32:0/3734203250 shutdown_connections 2021-04-09T18:24:27.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.223 7f52f2ee0700 1 --2- 172.21.15.32:0/3734203250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52ec0d7ca0 0x7f52ec0d80a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:27.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.223 7f52f2ee0700 1 -- 172.21.15.32:0/3734203250 >> 172.21.15.32:0/3734203250 conn(0x7f52ec0ce5f0 msgr2=0x7f52ec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:27.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.223 7f52f2ee0700 1 -- 172.21.15.32:0/3734203250 shutdown_connections 2021-04-09T18:24:27.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.223 7f52f2ee0700 1 -- 172.21.15.32:0/3734203250 wait complete. 2021-04-09T18:24:27.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.243 7f52f2ee0700 1 Processor -- start 2021-04-09T18:24:27.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.243 7f52f2ee0700 1 -- start start 2021-04-09T18:24:27.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52f2ee0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52ec092ad0 0x7f52ec091d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:27.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52f2ee0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52ec0a2300 0x7f52ec0aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:27.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52f2ee0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52ec0befd0 0x7f52ec0bf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:27.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52f2ee0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0a2700 con 0x7f52ec0a2300 2021-04-09T18:24:27.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52f2ee0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec092550 con 0x7f52ec092ad0 2021-04-09T18:24:27.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52f2ee0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0aacb0 con 0x7f52ec0befd0 2021-04-09T18:24:27.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52ec0a2300 0x7f52ec0aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:27.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52ec0a2300 0x7f52ec0aa690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51216/0 (socket says 172.21.15.32:51216) 2021-04-09T18:24:27.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52ebfff700 1 -- 172.21.15.32:0/3320721970 learned_addr learned my addr 172.21.15.32:0/3320721970 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:27.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52f0cfc700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52ec092ad0 0x7f52ec091d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:27.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52f14fd700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52ec0befd0 0x7f52ec0bf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:27.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52ebfff700 1 -- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52ec0befd0 msgr2=0x7f52ec0bf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:27.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52ebfff700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52ec0befd0 0x7f52ec0bf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:27.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52ebfff700 1 -- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52ec092ad0 msgr2=0x7f52ec091d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:27.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52ebfff700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52ec092ad0 0x7f52ec091d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:27.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52ebfff700 1 -- 172.21.15.32:0/3320721970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0bf9f0 con 0x7f52ec0a2300 2021-04-09T18:24:27.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52ebfff700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52ec0a2300 0x7f52ec0aa690 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f52dc0096a0 tx=0x7f52dc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:27.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.244 7f52e97fa700 1 -- 172.21.15.32:0/3320721970 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52dc000c90 con 0x7f52ec0a2300 2021-04-09T18:24:27.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.245 7f52e97fa700 1 -- 172.21.15.32:0/3320721970 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52dc002030 con 0x7f52ec0a2300 2021-04-09T18:24:27.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.245 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec0bf9f0 con 0x7f52ec0a2300 2021-04-09T18:24:27.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.245 7f52e97fa700 1 -- 172.21.15.32:0/3320721970 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52dc002550 con 0x7f52ec0a2300 2021-04-09T18:24:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.245 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec0bf9f0 con 0x7f52ec0a2300 2021-04-09T18:24:27.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.245 7f52e97fa700 1 -- 172.21.15.32:0/3320721970 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f52dc002210 con 0x7f52ec0a2300 2021-04-09T18:24:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.245 7f52e97fa700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52d00188d0 0x7f52d001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.245 7f52e97fa700 1 -- 172.21.15.32:0/3320721970 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f52dc032bc0 con 0x7f52ec0a2300 2021-04-09T18:24:27.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.246 7f52f0cfc700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52d00188d0 0x7f52d001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:27.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.246 7f52f0cfc700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52d00188d0 0x7f52d001add0 crc :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.265 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec1b7050 con 0x7f52ec0a2300 2021-04-09T18:24:27.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.269 7f52e97fa700 1 -- 172.21.15.32:0/3320721970 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f52dc0009c0 con 0x7f52ec0a2300 2021-04-09T18:24:27.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:27.462 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7f52ec10d480 con 0x7f52ec0a2300 2021-04-09T18:24:28.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.043 7f52e97fa700 1 -- 172.21.15.32:0/3320721970 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v240) v1 ==== 105+0+0 (secure 0 0 0) 0x7f52dc002030 con 0x7f52ec0a2300 2021-04-09T18:24:28.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52d00188d0 msgr2=0x7f52d001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:28.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52d00188d0 0x7f52d001add0 crc :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52ec0a2300 msgr2=0x7f52ec0aa690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:28.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52ec0a2300 0x7f52ec0aa690 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f52dc0096a0 tx=0x7f52dc00b990).stop 2021-04-09T18:24:28.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 shutdown_connections 2021-04-09T18:24:28.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52d00188d0 0x7f52d001add0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52ec0a2300 0x7f52ec0aa690 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52ec0befd0 0x7f52ec0bf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 --2- 172.21.15.32:0/3320721970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52ec092ad0 0x7f52ec091d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.079 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 >> 172.21.15.32:0/3320721970 conn(0x7f52ec093070 msgr2=0x7f52ec0a3150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:28.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.080 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 shutdown_connections 2021-04-09T18:24:28.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.080 7f52f2ee0700 1 -- 172.21.15.32:0/3320721970 wait complete. 2021-04-09T18:24:28.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:24:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.243 7f1a8498c700 1 Processor -- start 2021-04-09T18:24:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.243 7f1a8498c700 1 -- start start 2021-04-09T18:24:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.244 7f1a8498c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1a8010d010 0x7f1a800d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.244 7f1a8498c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a800d1da0 0x7f1a800d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.244 7f1a8498c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8010d480 con 0x7f1a800d82c0 2021-04-09T18:24:28.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.244 7f1a8498c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a800d9220 con 0x7f1a8010d010 2021-04-09T18:24:28.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.244 7f1a8498c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a800d86c0 con 0x7f1a800d1da0 2021-04-09T18:24:28.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.244 7f1a7de1b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a800d1da0 0x7f1a800d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:28.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.244 7f1a7e61c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1a8010d010 0x7f1a800d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.244 7f1a7ee1d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1a800d82c0 0x7f1a8010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58740/0 (socket says 172.21.15.32:58740) 2021-04-09T18:24:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.244 7f1a7ee1d700 1 -- 172.21.15.32:0/598683842 learned_addr learned my addr 172.21.15.32:0/598683842 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7de1b700 1 -- 172.21.15.32:0/598683842 >> v2:172.21.15.86:3300/0 conn(0x7f1a8010d010 msgr2=0x7f1a800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7de1b700 1 --2- 172.21.15.32:0/598683842 >> v2:172.21.15.86:3300/0 conn(0x7f1a8010d010 0x7f1a800d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7de1b700 1 -- 172.21.15.32:0/598683842 >> v1:172.21.15.86:6789/0 conn(0x7f1a800d82c0 legacy=0x7f1a8010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7de1b700 1 -- 172.21.15.32:0/598683842 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8010e1f0 con 0x7f1a800d1da0 2021-04-09T18:24:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7e61c700 1 --2- 172.21.15.32:0/598683842 >> v2:172.21.15.86:3300/0 conn(0x7f1a8010d010 0x7f1a800d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7d61a700 1 -- 172.21.15.32:0/598683842 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023099156 0 0) 0x7f1a78001440 con 0x7f1a800d82c0 2021-04-09T18:24:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7de1b700 1 --2- 172.21.15.32:0/598683842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a800d1da0 0x7f1a800d7ca0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f1a74006650 tx=0x7f1a74009a80).ready entity=mon.2 client_cookie=aac00d0fa2357151 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7d61a700 1 -- 172.21.15.32:0/598683842 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a740009b0 con 0x7f1a800d1da0 2021-04-09T18:24:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7d61a700 1 -- 172.21.15.32:0/598683842 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7400c0c0 con 0x7f1a800d1da0 2021-04-09T18:24:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a7d61a700 1 -- 172.21.15.32:0/598683842 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a740009b0 con 0x7f1a800d1da0 2021-04-09T18:24:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a8498c700 1 -- 172.21.15.32:0/598683842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a800d1da0 msgr2=0x7f1a800d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:28.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.245 7f1a8498c700 1 --2- 172.21.15.32:0/598683842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a800d1da0 0x7f1a800d7ca0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f1a74006650 tx=0x7f1a74009a80).stop 2021-04-09T18:24:28.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.246 7f1a8498c700 1 -- 172.21.15.32:0/598683842 shutdown_connections 2021-04-09T18:24:28.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.246 7f1a8498c700 1 --2- 172.21.15.32:0/598683842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a800d1da0 0x7f1a800d7ca0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.246 7f1a8498c700 1 --2- 172.21.15.32:0/598683842 >> v2:172.21.15.86:3300/0 conn(0x7f1a8010d010 0x7f1a800d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.246 7f1a8498c700 1 -- 172.21.15.32:0/598683842 >> 172.21.15.32:0/598683842 conn(0x7f1a800ce5f0 msgr2=0x7f1a800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:28.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.246 7f1a8498c700 1 -- 172.21.15.32:0/598683842 shutdown_connections 2021-04-09T18:24:28.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.246 7f1a8498c700 1 -- 172.21.15.32:0/598683842 wait complete. 2021-04-09T18:24:28.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.266 7f1a8498c700 1 Processor -- start 2021-04-09T18:24:28.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.266 7f1a8498c700 1 -- start start 2021-04-09T18:24:28.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a8498c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a800a0a10 0x7f1a800bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:28.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a8498c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8018ad00 0x7f1a8019fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:28.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a8498c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a801a03f0 0x7f1a80095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:28.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a8498c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8018b100 con 0x7f1a800a0a10 2021-04-09T18:24:28.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a8498c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a80095860 con 0x7f1a8018ad00 2021-04-09T18:24:28.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a8498c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a80095a80 con 0x7f1a801a03f0 2021-04-09T18:24:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7e61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a800a0a10 0x7f1a800bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7e61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a800a0a10 0x7f1a800bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51228/0 (socket says 172.21.15.32:51228) 2021-04-09T18:24:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7e61c700 1 -- 172.21.15.32:0/1704540317 learned_addr learned my addr 172.21.15.32:0/1704540317 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7ee1d700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a801a03f0 0x7f1a80095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7de1b700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8018ad00 0x7f1a8019fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7e61c700 1 -- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a801a03f0 msgr2=0x7f1a80095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7e61c700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a801a03f0 0x7f1a80095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7e61c700 1 -- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8018ad00 msgr2=0x7f1a8019fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7e61c700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8018ad00 0x7f1a8019fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7e61c700 1 -- 172.21.15.32:0/1704540317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a80189250 con 0x7f1a800a0a10 2021-04-09T18:24:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a7e61c700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a800a0a10 0x7f1a800bf870 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f1a70006f40 tx=0x7f1a7000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a6effd700 1 -- 172.21.15.32:0/1704540317 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a7000bd80 con 0x7f1a800a0a10 2021-04-09T18:24:28.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a6effd700 1 -- 172.21.15.32:0/1704540317 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a70000c90 con 0x7f1a800a0a10 2021-04-09T18:24:28.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a80189250 con 0x7f1a800a0a10 2021-04-09T18:24:28.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a6effd700 1 -- 172.21.15.32:0/1704540317 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a70002220 con 0x7f1a800a0a10 2021-04-09T18:24:28.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.267 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a80189250 con 0x7f1a800a0a10 2021-04-09T18:24:28.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.268 7f1a6effd700 1 -- 172.21.15.32:0/1704540317 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1a7000a000 con 0x7f1a800a0a10 2021-04-09T18:24:28.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.268 7f1a6effd700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:28.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.268 7f1a6effd700 1 -- 172.21.15.32:0/1704540317 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1a700329e0 con 0x7f1a800a0a10 2021-04-09T18:24:28.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.268 7f1a7de1b700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:28.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.269 7f1a7de1b700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.288 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a80093d50 con 0x7f1a800a0a10 2021-04-09T18:24:28.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.291 7f1a6effd700 1 -- 172.21.15.32:0/1704540317 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1a70000c90 con 0x7f1a800a0a10 2021-04-09T18:24:28.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:28.485 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7f1a8010d480 con 0x7f1a800a0a10 2021-04-09T18:24:29.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.047 7f1a6effd700 1 -- 172.21.15.32:0/1704540317 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v241) v1 ==== 105+0+0 (secure 0 0 0) 0x7f1a7000a000 con 0x7f1a800a0a10 2021-04-09T18:24:29.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.052 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 msgr2=0x7f1a6401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:29.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.052 7f1a8498c700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.052 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a800a0a10 msgr2=0x7f1a800bf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:29.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.052 7f1a8498c700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a800a0a10 0x7f1a800bf870 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f1a70006f40 tx=0x7f1a7000b990).stop 2021-04-09T18:24:29.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.053 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 shutdown_connections 2021-04-09T18:24:29.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.053 7f1a8498c700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.053 7f1a8498c700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a801a03f0 0x7f1a80095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.053 7f1a8498c700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8018ad00 0x7f1a8019fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.053 7f1a8498c700 1 --2- 172.21.15.32:0/1704540317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a800a0a10 0x7f1a800bf870 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.053 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 >> 172.21.15.32:0/1704540317 conn(0x7f1a800975f0 msgr2=0x7f1a80096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.053 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 shutdown_connections 2021-04-09T18:24:29.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.053 7f1a8498c700 1 -- 172.21.15.32:0/1704540317 wait complete. 2021-04-09T18:24:29.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:29.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.219 7f8c9173d700 1 Processor -- start 2021-04-09T18:24:29.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.219 7f8c9173d700 1 -- start start 2021-04-09T18:24:29.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.219 7f8c9173d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8c8c10e7d0 0x7f8c8c0d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:29.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c9173d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8c0d7280 con 0x7f8c8c0cffa0 2021-04-09T18:24:29.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c9173d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8c0d2440 con 0x7f8c8c0d6c80 2021-04-09T18:24:29.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c9173d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c8c0d7080 con 0x7f8c8c10e7d0 2021-04-09T18:24:29.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8b549700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8c8c0cffa0 0x7f8c8c0d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60826/0 (socket says 172.21.15.32:60826) 2021-04-09T18:24:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8b549700 1 -- 172.21.15.32:0/2546665506 learned_addr learned my addr 172.21.15.32:0/2546665506 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8bd4a700 1 --2- 172.21.15.32:0/2546665506 >> v2:172.21.15.86:3300/0 conn(0x7f8c8c10e7d0 0x7f8c8c0d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852126266 0 0) 0x7f8c7c000ce0 con 0x7f8c8c0cffa0 2021-04-09T18:24:29.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c74002fe0 con 0x7f8c8c0cffa0 2021-04-09T18:24:29.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582780858 0 0) 0x7f8c80000fe0 con 0x7f8c8c0d6c80 2021-04-09T18:24:29.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c74003210 con 0x7f8c8c0d6c80 2021-04-09T18:24:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4267421392 0 0) 0x7f8c7c0010e0 con 0x7f8c8c0cffa0 2021-04-09T18:24:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 >> v2:172.21.15.86:3300/0 conn(0x7f8c8c10e7d0 msgr2=0x7f8c8c0d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 --2- 172.21.15.32:0/2546665506 >> v2:172.21.15.86:3300/0 conn(0x7f8c8c10e7d0 0x7f8c8c0d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 >> v1:172.21.15.86:6789/0 conn(0x7f8c8c0d6c80 legacy=0x7f8c8c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:29.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c0d91e0 con 0x7f8c8c0cffa0 2021-04-09T18:24:29.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f8c7c001600 con 0x7f8c8c0cffa0 2021-04-09T18:24:29.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 >> v1:172.21.15.32:6789/0 conn(0x7f8c8c0cffa0 legacy=0x7f8c8c0d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:29.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c74003b90 0x7f8c74005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:29.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c74006840 0x7f8c74008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:29.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c74009640 0x7f8c7400bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:29.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740503c0 con 0x7f8c74003b90 2021-04-09T18:24:29.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740505a0 con 0x7f8c74006840 2021-04-09T18:24:29.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740507c0 con 0x7f8c74009640 2021-04-09T18:24:29.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8b549700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c74003b90 0x7f8c74005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:29.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.220 7f8c8bd4a700 1 --2- 172.21.15.32:0/2546665506 >> v2:172.21.15.86:3300/0 conn(0x7f8c8c10e7d0 0x7f8c8c0d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:29.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8ad48700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c74006840 0x7f8c74008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:29.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8bd4a700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c74009640 0x7f8c7400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:29.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8b549700 1 -- 172.21.15.32:0/2546665506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c74009640 msgr2=0x7f8c7400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:29.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8b549700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c74009640 0x7f8c7400bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8b549700 1 -- 172.21.15.32:0/2546665506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c74006840 msgr2=0x7f8c74008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:29.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8b549700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c74006840 0x7f8c74008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8b549700 1 -- 172.21.15.32:0/2546665506 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74050a60 con 0x7f8c74003b90 2021-04-09T18:24:29.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8bd4a700 1 -- 172.21.15.32:0/2546665506 reap_dead start 2021-04-09T18:24:29.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8ad48700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c74006840 0x7f8c74008d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:29.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8b549700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c74003b90 0x7f8c74005fa0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f8c7c001900 tx=0x7f8c7c007ce0).ready entity=mon.0 client_cookie=c8e193a17fc3afba server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:29.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c7c0070c0 con 0x7f8c74003b90 2021-04-09T18:24:29.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c0043a0 con 0x7f8c74003b90 2021-04-09T18:24:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c8a547700 1 -- 172.21.15.32:0/2546665506 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c7c004830 con 0x7f8c74003b90 2021-04-09T18:24:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c9173d700 1 -- 172.21.15.32:0/2546665506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c74003b90 msgr2=0x7f8c74005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.221 7f8c9173d700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c74003b90 0x7f8c74005fa0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f8c7c001900 tx=0x7f8c7c007ce0).stop 2021-04-09T18:24:29.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.222 7f8c9173d700 1 -- 172.21.15.32:0/2546665506 shutdown_connections 2021-04-09T18:24:29.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.222 7f8c9173d700 1 --2- 172.21.15.32:0/2546665506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c74003b90 0x7f8c74005fa0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.222 7f8c9173d700 1 -- 172.21.15.32:0/2546665506 >> 172.21.15.32:0/2546665506 conn(0x7f8c8c0ce5c0 msgr2=0x7f8c8c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:29.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.222 7f8c9173d700 1 -- 172.21.15.32:0/2546665506 shutdown_connections 2021-04-09T18:24:29.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.222 7f8c9173d700 1 -- 172.21.15.32:0/2546665506 wait complete. 2021-04-09T18:24:29.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.242 7f8c9173d700 1 Processor -- start 2021-04-09T18:24:29.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.242 7f8c9173d700 1 -- start start 2021-04-09T18:24:29.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c9173d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c8c1acff0 0x7f8c8c1847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:29.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c9173d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c8c183fc0 0x7f8c8c1a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:29.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c9173d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c8c186ff0 0x7f8c8c1873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:29.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c9173d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c188c30 con 0x7f8c8c186ff0 2021-04-09T18:24:29.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c9173d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1843c0 con 0x7f8c8c1acff0 2021-04-09T18:24:29.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c9173d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c1a5c20 con 0x7f8c8c183fc0 2021-04-09T18:24:29.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8bd4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c8c186ff0 0x7f8c8c1873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:29.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8bd4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c8c186ff0 0x7f8c8c1873f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51252/0 (socket says 172.21.15.32:51252) 2021-04-09T18:24:29.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8bd4a700 1 -- 172.21.15.32:0/3496219572 learned_addr learned my addr 172.21.15.32:0/3496219572 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:29.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8ad48700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c8c183fc0 0x7f8c8c1a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8b549700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c8c1acff0 0x7f8c8c1847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8bd4a700 1 -- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c8c183fc0 msgr2=0x7f8c8c1a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8bd4a700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c8c183fc0 0x7f8c8c1a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8bd4a700 1 -- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c8c1acff0 msgr2=0x7f8c8c1847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8bd4a700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c8c1acff0 0x7f8c8c1847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8bd4a700 1 -- 172.21.15.32:0/3496219572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c187a10 con 0x7f8c8c186ff0 2021-04-09T18:24:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8b549700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c8c1acff0 0x7f8c8c1847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c8bd4a700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c8c186ff0 0x7f8c8c1873f0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f8c8400b590 tx=0x7f8c8400b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c73fff700 1 -- 172.21.15.32:0/3496219572 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c8400bd20 con 0x7f8c8c186ff0 2021-04-09T18:24:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c73fff700 1 -- 172.21.15.32:0/3496219572 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c84006b40 con 0x7f8c8c186ff0 2021-04-09T18:24:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.243 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8c187a10 con 0x7f8c8c186ff0 2021-04-09T18:24:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.244 7f8c73fff700 1 -- 172.21.15.32:0/3496219572 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c840020c0 con 0x7f8c8c186ff0 2021-04-09T18:24:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.244 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8c187a10 con 0x7f8c8c186ff0 2021-04-09T18:24:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.244 7f8c73fff700 1 -- 172.21.15.32:0/3496219572 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8c84005330 con 0x7f8c8c186ff0 2021-04-09T18:24:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.244 7f8c73fff700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c6c018930 0x7f8c6c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.244 7f8c73fff700 1 -- 172.21.15.32:0/3496219572 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8c840329e0 con 0x7f8c8c186ff0 2021-04-09T18:24:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.244 7f8c8b549700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c6c018930 0x7f8c6c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:29.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.245 7f8c8b549700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c6c018930 0x7f8c6c01ae10 crc :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:29.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.264 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8c182dc0 con 0x7f8c8c186ff0 2021-04-09T18:24:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.267 7f8c73fff700 1 -- 172.21.15.32:0/3496219572 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8c84006800 con 0x7f8c8c186ff0 2021-04-09T18:24:29.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.459 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c8c0d7280 con 0x7f8c8c186ff0 2021-04-09T18:24:29.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.459 7f8c73fff700 1 -- 172.21.15.32:0/3496219572 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+2801 (secure 0 0 0) 0x7f8c8400bd20 con 0x7f8c8c186ff0 2021-04-09T18:24:29.522 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 241 2021-04-09T18:24:29.523 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:29.523 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":241,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57074,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3708886434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3708886434},{"type":"v1","addr":"172.21.15.32:6834","nonce":3708886434}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":204},{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:24:29.043077","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7777,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":233,"state":"up:stopping","state_seq":6,"addr":"172.21.15.32:6835/1795575373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1795575373},{"type":"v1","addr":"172.21.15.32:6835","nonce":1795575373}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:24:29.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c6c018930 msgr2=0x7f8c6c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:29.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c6c018930 0x7f8c6c01ae10 crc :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c8c186ff0 msgr2=0x7f8c8c1873f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:29.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c8c186ff0 0x7f8c8c1873f0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f8c8400b590 tx=0x7f8c8400b930).stop 2021-04-09T18:24:29.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 shutdown_connections 2021-04-09T18:24:29.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c6c018930 0x7f8c6c01ae10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c8c186ff0 0x7f8c8c1873f0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c8c183fc0 0x7f8c8c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 --2- 172.21.15.32:0/3496219572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c8c1acff0 0x7f8c8c1847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:29.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 >> 172.21.15.32:0/3496219572 conn(0x7f8c8c09abb0 msgr2=0x7f8c8c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:29.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 shutdown_connections 2021-04-09T18:24:29.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:29.462 7f8c9173d700 1 -- 172.21.15.32:0/3496219572 wait complete. 2021-04-09T18:24:29.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 241, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:22:40.389392', 'modified': '2021-04-09 18:24:29.043077', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7777, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7777': {'gid': 7777, 'name': 'b', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/1938812020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1938812020}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1938812020}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 233, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.32:6835/1795575373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 1795575373}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1795575373}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:24:29.937 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2021-04-09T18:24:30.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.038 7f96586fd700 1 Processor -- start 2021-04-09T18:24:31.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.038 7f96586fd700 1 -- start start 2021-04-09T18:24:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f96586fd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f965010d010 0x7f96500d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f96586fd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500d1da0 0x7f96500d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f96586fd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965010d480 con 0x7f96500d82c0 2021-04-09T18:24:31.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f96586fd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96500d9220 con 0x7f965010d010 2021-04-09T18:24:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f96586fd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96500d86c0 con 0x7f96500d1da0 2021-04-09T18:24:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9656d1a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f96500d82c0 0x7f965010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60848/0 (socket says 172.21.15.32:60848) 2021-04-09T18:24:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9656d1a700 1 -- 172.21.15.32:0/568095019 learned_addr learned my addr 172.21.15.32:0/568095019 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9656519700 1 --2- 172.21.15.32:0/568095019 >> v2:172.21.15.86:3300/0 conn(0x7f965010d010 0x7f96500d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:31.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9655d18700 1 --2- 172.21.15.32:0/568095019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500d1da0 0x7f96500d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:31.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9655517700 1 -- 172.21.15.32:0/568095019 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259330858 0 0) 0x7f9648000d30 con 0x7f96500d82c0 2021-04-09T18:24:31.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9655517700 1 -- 172.21.15.32:0/568095019 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c002fe0 con 0x7f96500d82c0 2021-04-09T18:24:31.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9656519700 1 -- 172.21.15.32:0/568095019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500d1da0 msgr2=0x7f96500d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:31.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9656519700 1 --2- 172.21.15.32:0/568095019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500d1da0 0x7f96500d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9656519700 1 -- 172.21.15.32:0/568095019 >> v1:172.21.15.32:6789/0 conn(0x7f96500d82c0 legacy=0x7f965010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9656519700 1 -- 172.21.15.32:0/568095019 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9650112100 con 0x7f965010d010 2021-04-09T18:24:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9655d18700 1 --2- 172.21.15.32:0/568095019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500d1da0 0x7f96500d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.039 7f9656519700 1 --2- 172.21.15.32:0/568095019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965010d010 0x7f96500d1780 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f964c006650 tx=0x7f964c009a80).ready entity=mon.1 client_cookie=4be3233c2ab137c8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.040 7f9655517700 1 -- 172.21.15.32:0/568095019 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f964c0009b0 con 0x7f965010d010 2021-04-09T18:24:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.040 7f9655517700 1 -- 172.21.15.32:0/568095019 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964c00c0c0 con 0x7f965010d010 2021-04-09T18:24:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.040 7f9655517700 1 -- 172.21.15.32:0/568095019 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f964c00c510 con 0x7f965010d010 2021-04-09T18:24:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.040 7f96586fd700 1 -- 172.21.15.32:0/568095019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965010d010 msgr2=0x7f96500d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:31.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.040 7f96586fd700 1 --2- 172.21.15.32:0/568095019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965010d010 0x7f96500d1780 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f964c006650 tx=0x7f964c009a80).stop 2021-04-09T18:24:31.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.041 7f96586fd700 1 -- 172.21.15.32:0/568095019 shutdown_connections 2021-04-09T18:24:31.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.041 7f96586fd700 1 --2- 172.21.15.32:0/568095019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500d1da0 0x7f96500d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.041 7f96586fd700 1 --2- 172.21.15.32:0/568095019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f965010d010 0x7f96500d1780 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.041 7f96586fd700 1 -- 172.21.15.32:0/568095019 >> 172.21.15.32:0/568095019 conn(0x7f96500ce5f0 msgr2=0x7f96500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.041 7f96586fd700 1 -- 172.21.15.32:0/568095019 shutdown_connections 2021-04-09T18:24:31.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.041 7f96586fd700 1 -- 172.21.15.32:0/568095019 wait complete. 2021-04-09T18:24:31.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.061 7f96586fd700 1 Processor -- start 2021-04-09T18:24:31.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.061 7f96586fd700 1 -- start start 2021-04-09T18:24:31.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.061 7f96586fd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500a90c0 0x7f96501a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:31.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.061 7f96586fd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96501a76d0 0x7f96501a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:31.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.061 7f96586fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965017bce0 0x7f965017c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:31.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.061 7f96586fd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96501a7c60 con 0x7f965017bce0 2021-04-09T18:24:31.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.061 7f96586fd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96501a8e20 con 0x7f96501a76d0 2021-04-09T18:24:31.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.061 7f96586fd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965017c700 con 0x7f96500a90c0 2021-04-09T18:24:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656d1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965017bce0 0x7f965017c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656d1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965017bce0 0x7f965017c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51266/0 (socket says 172.21.15.32:51266) 2021-04-09T18:24:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656d1a700 1 -- 172.21.15.32:0/3440424618 learned_addr learned my addr 172.21.15.32:0/3440424618 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656519700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500a90c0 0x7f96501a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9655d18700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96501a76d0 0x7f96501a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656d1a700 1 -- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500a90c0 msgr2=0x7f96501a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656d1a700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500a90c0 0x7f96501a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656d1a700 1 -- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96501a76d0 msgr2=0x7f96501a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:31.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656d1a700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96501a76d0 0x7f96501a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656d1a700 1 -- 172.21.15.32:0/3440424618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965017c920 con 0x7f965017bce0 2021-04-09T18:24:31.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9655d18700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96501a76d0 0x7f96501a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:31.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656519700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500a90c0 0x7f96501a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:31.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9656d1a700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965017bce0 0x7f965017c0e0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f9648009960 tx=0x7f964800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9646ffd700 1 -- 172.21.15.32:0/3440424618 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9648006030 con 0x7f965017bce0 2021-04-09T18:24:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9646ffd700 1 -- 172.21.15.32:0/3440424618 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9648006370 con 0x7f965017bce0 2021-04-09T18:24:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f9646ffd700 1 -- 172.21.15.32:0/3440424618 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96480049a0 con 0x7f965017bce0 2021-04-09T18:24:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f96586fd700 1 -- 172.21.15.32:0/3440424618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965017c920 con 0x7f965017bce0 2021-04-09T18:24:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.062 7f96586fd700 1 -- 172.21.15.32:0/3440424618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965017c920 con 0x7f965017bce0 2021-04-09T18:24:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.063 7f9646ffd700 1 -- 172.21.15.32:0/3440424618 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9648000cc0 con 0x7f965017bce0 2021-04-09T18:24:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.063 7f9646ffd700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9638018930 0x7f963801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.063 7f9646ffd700 1 -- 172.21.15.32:0/3440424618 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9648032da0 con 0x7f965017bce0 2021-04-09T18:24:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.063 7f9656519700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9638018930 0x7f963801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:31.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.063 7f9656519700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9638018930 0x7f963801ae10 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:31.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.083 7f96586fd700 1 -- 172.21.15.32:0/3440424618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96501b7010 con 0x7f965017bce0 2021-04-09T18:24:31.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.086 7f9646ffd700 1 -- 172.21.15.32:0/3440424618 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f964800be30 con 0x7f965017bce0 2021-04-09T18:24:31.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.278 7f96586fd700 1 -- 172.21.15.32:0/3440424618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f965010d480 con 0x7f965017bce0 2021-04-09T18:24:31.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.278 7f9646ffd700 1 -- 172.21.15.32:0/3440424618 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+2801 (secure 0 0 0) 0x7f9648000cc0 con 0x7f965017bce0 2021-04-09T18:24:31.558 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 241 2021-04-09T18:24:31.559 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:31.559 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":241,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57074,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3708886434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3708886434},{"type":"v1","addr":"172.21.15.32:6834","nonce":3708886434}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":204},{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:24:29.043077","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7777,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":233,"state":"up:stopping","state_seq":6,"addr":"172.21.15.32:6835/1795575373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1795575373},{"type":"v1","addr":"172.21.15.32:6835","nonce":1795575373}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:24:31.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.280 7f96586fd700 1 -- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9638018930 msgr2=0x7f963801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:31.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.280 7f96586fd700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9638018930 0x7f963801ae10 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.280 7f96586fd700 1 -- 172.21.15.32:0/3440424618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965017bce0 msgr2=0x7f965017c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:31.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.280 7f96586fd700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965017bce0 0x7f965017c0e0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f9648009960 tx=0x7f964800bcf0).stop 2021-04-09T18:24:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.281 7f96586fd700 1 -- 172.21.15.32:0/3440424618 shutdown_connections 2021-04-09T18:24:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.281 7f96586fd700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9638018930 0x7f963801ae10 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.281 7f96586fd700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965017bce0 0x7f965017c0e0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.281 7f96586fd700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96501a76d0 0x7f96501a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.281 7f96586fd700 1 --2- 172.21.15.32:0/3440424618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96500a90c0 0x7f96501a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.281 7f96586fd700 1 -- 172.21.15.32:0/3440424618 >> 172.21.15.32:0/3440424618 conn(0x7f96501acae0 msgr2=0x7f965017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.281 7f96586fd700 1 -- 172.21.15.32:0/3440424618 shutdown_connections 2021-04-09T18:24:31.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:31.281 7f96586fd700 1 -- 172.21.15.32:0/3440424618 wait complete. 2021-04-09T18:24:31.966 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 241, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:22:40.389392', 'modified': '2021-04-09 18:24:29.043077', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7777, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7777': {'gid': 7777, 'name': 'b', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/1938812020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1938812020}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1938812020}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 233, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.32:6835/1795575373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 1795575373}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1795575373}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:24:31.967 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2021-04-09T18:24:32.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:33.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.057 7f440e9d6700 1 Processor -- start 2021-04-09T18:24:33.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.057 7f440e9d6700 1 -- start start 2021-04-09T18:24:33.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f440e9d6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f440810d010 0x7f44080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:33.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f440e9d6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d1da0 0x7f44080d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:33.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f440e9d6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f44080d82c0 0x7f440810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:33.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f440e9d6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44080d8d30 con 0x7f440810d010 2021-04-09T18:24:33.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f440e9d6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44080d9220 con 0x7f44080d82c0 2021-04-09T18:24:33.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f440e9d6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440810d480 con 0x7f44080d1da0 2021-04-09T18:24:33.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f4407fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f440810d010 0x7f44080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:33.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f4407fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f440810d010 0x7f44080d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51270/0 (socket says 172.21.15.32:51270) 2021-04-09T18:24:33.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f440cff3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f44080d82c0 0x7f440810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:33.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f4407fff700 1 -- 172.21.15.32:0/1303120364 learned_addr learned my addr 172.21.15.32:0/1303120364 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:33.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f43fffff700 1 --2- 172.21.15.32:0/1303120364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d1da0 0x7f44080d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:33.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f4407fff700 1 -- 172.21.15.32:0/1303120364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d1da0 msgr2=0x7f44080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:33.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f4407fff700 1 --2- 172.21.15.32:0/1303120364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d1da0 0x7f44080d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f4407fff700 1 -- 172.21.15.32:0/1303120364 >> v2:172.21.15.86:3300/0 conn(0x7f44080d82c0 msgr2=0x7f440810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:33.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f4407fff700 1 --2- 172.21.15.32:0/1303120364 >> v2:172.21.15.86:3300/0 conn(0x7f44080d82c0 0x7f440810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.058 7f4407fff700 1 -- 172.21.15.32:0/1303120364 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44080d86c0 con 0x7f440810d010 2021-04-09T18:24:33.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.059 7f4407fff700 1 --2- 172.21.15.32:0/1303120364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440810d010 0x7f44080d1780 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f43f8006700 tx=0x7f43f8009b80).ready entity=mon.0 client_cookie=564d19f7e694c8cf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:33.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.059 7f44077fe700 1 -- 172.21.15.32:0/1303120364 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f800c0c0 con 0x7f440810d010 2021-04-09T18:24:33.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.059 7f44077fe700 1 -- 172.21.15.32:0/1303120364 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f800c440 con 0x7f440810d010 2021-04-09T18:24:33.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.059 7f44077fe700 1 -- 172.21.15.32:0/1303120364 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f800c920 con 0x7f440810d010 2021-04-09T18:24:33.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.059 7f440e9d6700 1 -- 172.21.15.32:0/1303120364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440810d010 msgr2=0x7f44080d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:33.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.059 7f440e9d6700 1 --2- 172.21.15.32:0/1303120364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440810d010 0x7f44080d1780 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f43f8006700 tx=0x7f43f8009b80).stop 2021-04-09T18:24:33.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.060 7f440e9d6700 1 -- 172.21.15.32:0/1303120364 shutdown_connections 2021-04-09T18:24:33.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.060 7f440e9d6700 1 --2- 172.21.15.32:0/1303120364 >> v2:172.21.15.86:3300/0 conn(0x7f44080d82c0 0x7f440810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.060 7f440e9d6700 1 --2- 172.21.15.32:0/1303120364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d1da0 0x7f44080d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.060 7f440e9d6700 1 --2- 172.21.15.32:0/1303120364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440810d010 0x7f44080d1780 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.060 7f440e9d6700 1 -- 172.21.15.32:0/1303120364 >> 172.21.15.32:0/1303120364 conn(0x7f44080ce5f0 msgr2=0x7f44080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.060 7f440e9d6700 1 -- 172.21.15.32:0/1303120364 shutdown_connections 2021-04-09T18:24:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.060 7f440e9d6700 1 -- 172.21.15.32:0/1303120364 wait complete. 2021-04-09T18:24:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.080 7f440e9d6700 1 Processor -- start 2021-04-09T18:24:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.080 7f440e9d6700 1 -- start start 2021-04-09T18:24:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.080 7f440e9d6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44080b42d0 0x7f4408183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.080 7f440e9d6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44081ac090 0x7f44081ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.080 7f440e9d6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408184d30 0x7f4408185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.080 7f440e9d6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44080a1a90 con 0x7f44080b42d0 2021-04-09T18:24:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.080 7f440e9d6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44081acab0 con 0x7f4408184d30 2021-04-09T18:24:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.080 7f440e9d6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4408185a00 con 0x7f44081ac090 2021-04-09T18:24:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f4407fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44080b42d0 0x7f4408183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f4407fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44080b42d0 0x7f4408183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51276/0 (socket says 172.21.15.32:51276) 2021-04-09T18:24:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f4407fff700 1 -- 172.21.15.32:0/1977090427 learned_addr learned my addr 172.21.15.32:0/1977090427 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f440cff3700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408184d30 0x7f4408185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f43fffff700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44081ac090 0x7f44081ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f4407fff700 1 -- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44081ac090 msgr2=0x7f44081ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f4407fff700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44081ac090 0x7f44081ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f4407fff700 1 -- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408184d30 msgr2=0x7f4408185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:33.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f4407fff700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408184d30 0x7f4408185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f4407fff700 1 -- 172.21.15.32:0/1977090427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4408185750 con 0x7f44080b42d0 2021-04-09T18:24:33.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f440cff3700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408184d30 0x7f4408185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:33.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f4407fff700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44080b42d0 0x7f4408183520 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f43f800b5e0 tx=0x7f43f800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:33.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f44057fa700 1 -- 172.21.15.32:0/1977090427 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f800bd70 con 0x7f44080b42d0 2021-04-09T18:24:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f44057fa700 1 -- 172.21.15.32:0/1977090427 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f8002030 con 0x7f44080b42d0 2021-04-09T18:24:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4408185750 con 0x7f44080b42d0 2021-04-09T18:24:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f44057fa700 1 -- 172.21.15.32:0/1977090427 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f8002550 con 0x7f44080b42d0 2021-04-09T18:24:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.081 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4408185750 con 0x7f44080b42d0 2021-04-09T18:24:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.082 7f44057fa700 1 -- 172.21.15.32:0/1977090427 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f43f8002210 con 0x7f44080b42d0 2021-04-09T18:24:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.082 7f44057fa700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.082 7f44057fa700 1 -- 172.21.15.32:0/1977090427 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f43f80329d0 con 0x7f44080b42d0 2021-04-09T18:24:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.082 7f43fffff700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.083 7f43fffff700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01adc0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:33.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.101 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44080a1a90 con 0x7f44080b42d0 2021-04-09T18:24:33.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.105 7f44057fa700 1 -- 172.21.15.32:0/1977090427 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f43f8002240 con 0x7f44080b42d0 2021-04-09T18:24:33.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.296 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4408185750 con 0x7f44080b42d0 2021-04-09T18:24:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.297 7f44057fa700 1 -- 172.21.15.32:0/1977090427 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+2801 (secure 0 0 0) 0x7f43f8002030 con 0x7f44080b42d0 2021-04-09T18:24:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 241 2021-04-09T18:24:33.359 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:33.360 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":241,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57074,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3708886434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3708886434},{"type":"v1","addr":"172.21.15.32:6834","nonce":3708886434}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":204},{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:24:29.043077","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7777,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":233,"state":"up:stopping","state_seq":6,"addr":"172.21.15.32:6835/1795575373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1795575373},{"type":"v1","addr":"172.21.15.32:6835","nonce":1795575373}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:24:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 msgr2=0x7f43ec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01adc0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44080b42d0 msgr2=0x7f4408183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44080b42d0 0x7f4408183520 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f43f800b5e0 tx=0x7f43f800b980).stop 2021-04-09T18:24:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 shutdown_connections 2021-04-09T18:24:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01adc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408184d30 0x7f4408185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44081ac090 0x7f44081ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 --2- 172.21.15.32:0/1977090427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44080b42d0 0x7f4408183520 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 >> 172.21.15.32:0/1977090427 conn(0x7f440819b890 msgr2=0x7f44081a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 shutdown_connections 2021-04-09T18:24:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:33.299 7f440e9d6700 1 -- 172.21.15.32:0/1977090427 wait complete. 2021-04-09T18:24:33.773 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 241, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:22:40.389392', 'modified': '2021-04-09 18:24:29.043077', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7777, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7777': {'gid': 7777, 'name': 'b', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/1938812020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1938812020}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1938812020}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 233, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.32:6835/1795575373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 1795575373}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1795575373}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:24:33.774 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2021-04-09T18:24:34.775 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.869 7f2fefda6700 1 Processor -- start 2021-04-09T18:24:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.869 7f2fefda6700 1 -- start start 2021-04-09T18:24:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fefda6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2fe80cffa0 0x7f2fe80d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fefda6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fe80d7800 0x7f2fe80d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fefda6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe80d7280 con 0x7f2fe80d8220 2021-04-09T18:24:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fefda6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80d91e0 con 0x7f2fe80d7800 2021-04-09T18:24:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fefda6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe810dfe0 con 0x7f2fe80cffa0 2021-04-09T18:24:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fed3c1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fe80d7800 0x7f2fe80d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fed3c1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fe80d7800 0x7f2fe80d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51288/0 (socket says 172.21.15.32:51288) 2021-04-09T18:24:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fed3c1700 1 -- 172.21.15.32:0/1354675312 learned_addr learned my addr 172.21.15.32:0/1354675312 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fedbc2700 1 --2- 172.21.15.32:0/1354675312 >> v2:172.21.15.86:3300/0 conn(0x7f2fe80cffa0 0x7f2fe80d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fed3c1700 1 -- 172.21.15.32:0/1354675312 >> v2:172.21.15.86:3300/0 conn(0x7f2fe80cffa0 msgr2=0x7f2fe80d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fed3c1700 1 --2- 172.21.15.32:0/1354675312 >> v2:172.21.15.86:3300/0 conn(0x7f2fe80cffa0 0x7f2fe80d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fed3c1700 1 -- 172.21.15.32:0/1354675312 >> v1:172.21.15.86:6789/0 conn(0x7f2fe80d8220 legacy=0x7f2fe80d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.870 7f2fed3c1700 1 -- 172.21.15.32:0/1354675312 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe810e200 con 0x7f2fe80d7800 2021-04-09T18:24:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.871 7f2fed3c1700 1 --2- 172.21.15.32:0/1354675312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe80d7800 0x7f2fe80d7c00 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7f2fd8006700 tx=0x7f2fd8009b80).ready entity=mon.0 client_cookie=78d8e2f9ead94e78 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.871 7f2fecbc0700 1 -- 172.21.15.32:0/1354675312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fd800c0c0 con 0x7f2fe80d7800 2021-04-09T18:24:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.871 7f2fecbc0700 1 -- 172.21.15.32:0/1354675312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd800c440 con 0x7f2fe80d7800 2021-04-09T18:24:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.871 7f2fecbc0700 1 -- 172.21.15.32:0/1354675312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fd800c920 con 0x7f2fe80d7800 2021-04-09T18:24:34.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.871 7f2fefda6700 1 -- 172.21.15.32:0/1354675312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe80d7800 msgr2=0x7f2fe80d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.871 7f2fefda6700 1 --2- 172.21.15.32:0/1354675312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe80d7800 0x7f2fe80d7c00 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7f2fd8006700 tx=0x7f2fd8009b80).stop 2021-04-09T18:24:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.872 7f2fefda6700 1 -- 172.21.15.32:0/1354675312 shutdown_connections 2021-04-09T18:24:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.872 7f2fefda6700 1 --2- 172.21.15.32:0/1354675312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe80d7800 0x7f2fe80d7c00 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:34.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.872 7f2fefda6700 1 --2- 172.21.15.32:0/1354675312 >> v2:172.21.15.86:3300/0 conn(0x7f2fe80cffa0 0x7f2fe80d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.872 7f2fefda6700 1 -- 172.21.15.32:0/1354675312 >> 172.21.15.32:0/1354675312 conn(0x7f2fe80ce5c0 msgr2=0x7f2fe80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:34.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.872 7f2fefda6700 1 -- 172.21.15.32:0/1354675312 shutdown_connections 2021-04-09T18:24:34.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.872 7f2fefda6700 1 -- 172.21.15.32:0/1354675312 wait complete. 2021-04-09T18:24:34.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.892 7f2fefda6700 1 Processor -- start 2021-04-09T18:24:34.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.892 7f2fefda6700 1 -- start start 2021-04-09T18:24:34.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.892 7f2fefda6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fe81acff0 0x7f2fe81847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:34.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.892 7f2fefda6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe8183fc0 0x7f2fe81a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:34.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.892 7f2fefda6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8186ff0 0x7f2fe81873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:34.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.892 7f2fefda6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe8188c30 con 0x7f2fe8183fc0 2021-04-09T18:24:34.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.892 7f2fefda6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81843c0 con 0x7f2fe8186ff0 2021-04-09T18:24:34.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.892 7f2fefda6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81a5c20 con 0x7f2fe81acff0 2021-04-09T18:24:34.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fed3c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe8183fc0 0x7f2fe81a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:34.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fed3c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe8183fc0 0x7f2fe81a5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51292/0 (socket says 172.21.15.32:51292) 2021-04-09T18:24:34.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fed3c1700 1 -- 172.21.15.32:0/2704406327 learned_addr learned my addr 172.21.15.32:0/2704406327 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:34.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fedbc2700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fe81acff0 0x7f2fe81847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:34.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fee3c3700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8186ff0 0x7f2fe81873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:34.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fed3c1700 1 -- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fe81acff0 msgr2=0x7f2fe81847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:34.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fed3c1700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fe81acff0 0x7f2fe81847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:34.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fed3c1700 1 -- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8186ff0 msgr2=0x7f2fe81873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:34.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fed3c1700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8186ff0 0x7f2fe81873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:34.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fed3c1700 1 -- 172.21.15.32:0/2704406327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8187a10 con 0x7f2fe8183fc0 2021-04-09T18:24:34.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fee3c3700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8186ff0 0x7f2fe81873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:34.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fedbc2700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fe81acff0 0x7f2fe81847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:34.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fed3c1700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe8183fc0 0x7f2fe81a5600 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7f2fd800b650 tx=0x7f2fd800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:34.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fde7fc700 1 -- 172.21.15.32:0/2704406327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fd800bde0 con 0x7f2fe8183fc0 2021-04-09T18:24:34.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fde7fc700 1 -- 172.21.15.32:0/2704406327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd8002030 con 0x7f2fe8183fc0 2021-04-09T18:24:34.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe8187a10 con 0x7f2fe8183fc0 2021-04-09T18:24:34.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fde7fc700 1 -- 172.21.15.32:0/2704406327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fd8002510 con 0x7f2fe8183fc0 2021-04-09T18:24:34.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.893 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe8187a10 con 0x7f2fe8183fc0 2021-04-09T18:24:34.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.894 7f2fde7fc700 1 -- 172.21.15.32:0/2704406327 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2fd8000c90 con 0x7f2fe8183fc0 2021-04-09T18:24:34.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.894 7f2fde7fc700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd0018930 0x7f2fd001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:34.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.894 7f2fde7fc700 1 -- 172.21.15.32:0/2704406327 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2fd8032bf0 con 0x7f2fe8183fc0 2021-04-09T18:24:34.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.894 7f2fedbc2700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd0018930 0x7f2fd001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:34.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.895 7f2fedbc2700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd0018930 0x7f2fd001ae10 crc :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:34.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.914 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe8182dc0 con 0x7f2fe8183fc0 2021-04-09T18:24:34.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:34.917 7f2fde7fc700 1 -- 172.21.15.32:0/2704406327 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2fd801f7b0 con 0x7f2fe8183fc0 2021-04-09T18:24:35.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.110 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fe80d7280 con 0x7f2fe8183fc0 2021-04-09T18:24:35.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.110 7f2fde7fc700 1 -- 172.21.15.32:0/2704406327 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+2801 (secure 0 0 0) 0x7f2fd801f7b0 con 0x7f2fe8183fc0 2021-04-09T18:24:35.173 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 241 2021-04-09T18:24:35.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:35.174 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":241,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57074,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3708886434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3708886434},{"type":"v1","addr":"172.21.15.32:6834","nonce":3708886434}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":204},{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:24:29.043077","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7777,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":233,"state":"up:stopping","state_seq":6,"addr":"172.21.15.32:6835/1795575373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1795575373},{"type":"v1","addr":"172.21.15.32:6835","nonce":1795575373}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:24:35.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd0018930 msgr2=0x7f2fd001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:35.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd0018930 0x7f2fd001ae10 crc :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:35.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe8183fc0 msgr2=0x7f2fe81a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:35.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe8183fc0 0x7f2fe81a5600 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7f2fd800b650 tx=0x7f2fd800b9f0).stop 2021-04-09T18:24:35.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 shutdown_connections 2021-04-09T18:24:35.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd0018930 0x7f2fd001ae10 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:35.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe8183fc0 0x7f2fe81a5600 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fe8186ff0 0x7f2fe81873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 --2- 172.21.15.32:0/2704406327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fe81acff0 0x7f2fe81847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 >> 172.21.15.32:0/2704406327 conn(0x7f2fe809abb0 msgr2=0x7f2fe80a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:35.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 shutdown_connections 2021-04-09T18:24:35.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:35.113 7f2fefda6700 1 -- 172.21.15.32:0/2704406327 wait complete. 2021-04-09T18:24:35.588 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 241, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:22:40.389392', 'modified': '2021-04-09 18:24:29.043077', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7777, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7777': {'gid': 7777, 'name': 'b', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/1938812020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1938812020}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1938812020}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 233, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.32:6835/1795575373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 1795575373}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1795575373}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:24:35.589 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2021-04-09T18:24:36.173 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:24:36.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:36.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.694 7f131f2db700 1 Processor -- start 2021-04-09T18:24:36.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.694 7f131f2db700 1 -- start start 2021-04-09T18:24:36.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131f2db700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f13180d7ca0 0x7f13180d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:36.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131f2db700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f131810da00 0x7f131810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131f2db700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131810d480 con 0x7f131810d010 2021-04-09T18:24:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131f2db700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13180d9220 con 0x7f13180d7ca0 2021-04-09T18:24:36.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131f2db700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13180d86c0 con 0x7f131810da00 2021-04-09T18:24:36.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131c8f6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f13180d7ca0 0x7f13180d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:36.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131d0f7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f131810d010 0x7f13180d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60906/0 (socket says 172.21.15.32:60906) 2021-04-09T18:24:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131d0f7700 1 -- 172.21.15.32:0/761280002 learned_addr learned my addr 172.21.15.32:0/761280002 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131d8f8700 1 --2- 172.21.15.32:0/761280002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f131810da00 0x7f131810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131c8f6700 1 -- 172.21.15.32:0/761280002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f131810da00 msgr2=0x7f131810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131c8f6700 1 --2- 172.21.15.32:0/761280002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f131810da00 0x7f131810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131c8f6700 1 -- 172.21.15.32:0/761280002 >> v1:172.21.15.32:6789/0 conn(0x7f131810d010 legacy=0x7f13180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f130ffff700 1 -- 172.21.15.32:0/761280002 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381126728 0 0) 0x7f1314000ce0 con 0x7f131810d010 2021-04-09T18:24:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131c8f6700 1 -- 172.21.15.32:0/761280002 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13180d1f70 con 0x7f13180d7ca0 2021-04-09T18:24:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.695 7f131d8f8700 1 --2- 172.21.15.32:0/761280002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f131810da00 0x7f131810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.696 7f131c8f6700 1 --2- 172.21.15.32:0/761280002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13180d7ca0 0x7f13180d80a0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f1308006650 tx=0x7f1308009a80).ready entity=mon.0 client_cookie=f6486fb6421e7d04 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.696 7f130ffff700 1 -- 172.21.15.32:0/761280002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13080009b0 con 0x7f13180d7ca0 2021-04-09T18:24:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.696 7f130ffff700 1 -- 172.21.15.32:0/761280002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130800c0c0 con 0x7f13180d7ca0 2021-04-09T18:24:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.696 7f130ffff700 1 -- 172.21.15.32:0/761280002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f130800c510 con 0x7f13180d7ca0 2021-04-09T18:24:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.696 7f131f2db700 1 -- 172.21.15.32:0/761280002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13180d7ca0 msgr2=0x7f13180d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.696 7f131f2db700 1 --2- 172.21.15.32:0/761280002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13180d7ca0 0x7f13180d80a0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f1308006650 tx=0x7f1308009a80).stop 2021-04-09T18:24:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.697 7f131f2db700 1 -- 172.21.15.32:0/761280002 shutdown_connections 2021-04-09T18:24:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.697 7f131f2db700 1 --2- 172.21.15.32:0/761280002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f131810da00 0x7f131810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.697 7f131f2db700 1 --2- 172.21.15.32:0/761280002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13180d7ca0 0x7f13180d80a0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.697 7f131f2db700 1 -- 172.21.15.32:0/761280002 >> 172.21.15.32:0/761280002 conn(0x7f13180ce5f0 msgr2=0x7f13180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.697 7f131f2db700 1 -- 172.21.15.32:0/761280002 shutdown_connections 2021-04-09T18:24:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.697 7f131f2db700 1 -- 172.21.15.32:0/761280002 wait complete. 2021-04-09T18:24:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.717 7f131f2db700 1 Processor -- start 2021-04-09T18:24:36.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:24:36.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.717 7f131f2db700 1 -- start start 2021-04-09T18:24:36.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.717 7f131f2db700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13181a9360 0x7f13181918f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:36.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.717 7f131f2db700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f131818c440 0x7f131818c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:36.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.717 7f131f2db700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13181a80f0 0x7f13181a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:36.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.717 7f131f2db700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1318191f10 con 0x7f131818c440 2021-04-09T18:24:36.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.717 7f131f2db700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13181a9760 con 0x7f13181a80f0 2021-04-09T18:24:36.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.717 7f131f2db700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131818ce60 con 0x7f13181a9360 2021-04-09T18:24:36.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131c8f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f131818c440 0x7f131818c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:36.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131c8f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f131818c440 0x7f131818c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51326/0 (socket says 172.21.15.32:51326) 2021-04-09T18:24:36.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131c8f6700 1 -- 172.21.15.32:0/2652138635 learned_addr learned my addr 172.21.15.32:0/2652138635 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:36.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131d0f7700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13181a9360 0x7f13181918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:36.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131d8f8700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13181a80f0 0x7f13181a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:36.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131c8f6700 1 -- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13181a9360 msgr2=0x7f13181918f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:36.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131c8f6700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13181a9360 0x7f13181918f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:36.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131c8f6700 1 -- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13181a80f0 msgr2=0x7f13181a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:36.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131c8f6700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13181a80f0 0x7f13181a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:36.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131c8f6700 1 -- 172.21.15.32:0/2652138635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13181a8b10 con 0x7f131818c440 2021-04-09T18:24:36.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131d0f7700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13181a9360 0x7f13181918f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:36.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131c8f6700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f131818c440 0x7f131818c840 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7f1308009650 tx=0x7f130800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:36.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f130dffb700 1 -- 172.21.15.32:0/2652138635 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13080093a0 con 0x7f131818c440 2021-04-09T18:24:36.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f130dffb700 1 -- 172.21.15.32:0/2652138635 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130800bd50 con 0x7f131818c440 2021-04-09T18:24:36.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f130dffb700 1 -- 172.21.15.32:0/2652138635 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13080052e0 con 0x7f131818c440 2021-04-09T18:24:36.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131f2db700 1 -- 172.21.15.32:0/2652138635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13181a8b10 con 0x7f131818c440 2021-04-09T18:24:36.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.718 7f131f2db700 1 -- 172.21.15.32:0/2652138635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13181a8b10 con 0x7f131818c440 2021-04-09T18:24:36.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.719 7f130dffb700 1 -- 172.21.15.32:0/2652138635 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1308006030 con 0x7f131818c440 2021-04-09T18:24:36.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.719 7f130dffb700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12f80188c0 0x7f12f801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:36.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.719 7f130dffb700 1 -- 172.21.15.32:0/2652138635 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f13080327a0 con 0x7f131818c440 2021-04-09T18:24:36.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.719 7f131d0f7700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12f80188c0 0x7f12f801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:36.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.720 7f131d0f7700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12f80188c0 0x7f12f801adc0 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:36.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:24:36.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.739 7f131f2db700 1 -- 172.21.15.32:0/2652138635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13181a9760 con 0x7f131818c440 2021-04-09T18:24:36.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.742 7f130dffb700 1 -- 172.21.15.32:0/2652138635 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f13080009c0 con 0x7f131818c440 2021-04-09T18:24:36.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.936 7f131f2db700 1 -- 172.21.15.32:0/2652138635 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13181a9760 con 0x7f131818c440 2021-04-09T18:24:37.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.936 7f130dffb700 1 -- 172.21.15.32:0/2652138635 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+2465 (secure 0 0 0) 0x7f1308006030 con 0x7f131818c440 2021-04-09T18:24:37.000 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 243 2021-04-09T18:24:37.001 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:37.002 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":243,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:24:36.052053","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7777,"mds_1":57074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_57074":{"gid":57074,"name":"c","rank":1,"incarnation":243,"state":"up:starting","state_seq":2,"addr":"172.21.15.32:6834/3708886434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3708886434},{"type":"v1","addr":"172.21.15.32:6834","nonce":3708886434}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:24:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 -- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12f80188c0 msgr2=0x7f12f801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12f80188c0 0x7f12f801adc0 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 -- 172.21.15.32:0/2652138635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f131818c440 msgr2=0x7f131818c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f131818c440 0x7f131818c840 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7f1308009650 tx=0x7f130800b960).stop 2021-04-09T18:24:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 -- 172.21.15.32:0/2652138635 shutdown_connections 2021-04-09T18:24:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12f80188c0 0x7f12f801adc0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f131818c440 0x7f131818c840 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13181a80f0 0x7f13181a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 --2- 172.21.15.32:0/2652138635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13181a9360 0x7f13181918f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 -- 172.21.15.32:0/2652138635 >> 172.21.15.32:0/2652138635 conn(0x7f13181837d0 msgr2=0x7f1318195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 -- 172.21.15.32:0/2652138635 shutdown_connections 2021-04-09T18:24:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:36.939 7f131f2db700 1 -- 172.21.15.32:0/2652138635 wait complete. 2021-04-09T18:24:37.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 243, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:22:40.389392', 'modified': '2021-04-09 18:24:36.052053', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7777, 'mds_1': 57074}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7777': {'gid': 7777, 'name': 'b', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/1938812020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1938812020}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1938812020}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_57074': {'gid': 57074, 'name': 'c', 'rank': 1, 'incarnation': 243, 'state': 'up:starting', 'state_seq': 2, 'addr': '172.21.15.32:6834/3708886434', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3708886434}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 3708886434}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:24:37.415 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_57074:up:starting 2021-04-09T18:24:38.416 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.515 7fd8427ca700 1 Processor -- start 2021-04-09T18:24:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.515 7fd8427ca700 1 -- start start 2021-04-09T18:24:38.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.515 7fd8427ca700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd83c10d010 0x7fd83c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.515 7fd8427ca700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd83c0d8490 0x7fd83c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.515 7fd8427ca700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c10d480 con 0x7fd83c0d1da0 2021-04-09T18:24:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.515 7fd8427ca700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83c0d8d30 con 0x7fd83c10d010 2021-04-09T18:24:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.515 7fd8427ca700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83c0d9220 con 0x7fd83c0d8490 2021-04-09T18:24:38.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd83c10d010 0x7fd83c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd833fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd83c0d1da0 0x7fd83c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60926/0 (socket says 172.21.15.32:60926) 2021-04-09T18:24:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd833fff700 1 -- 172.21.15.32:0/4094895228 learned_addr learned my addr 172.21.15.32:0/4094895228 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd840de7700 1 --2- 172.21.15.32:0/4094895228 >> v2:172.21.15.86:3300/0 conn(0x7fd83c0d8490 0x7fd83c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:38.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83bfff700 1 -- 172.21.15.32:0/4094895228 >> v2:172.21.15.86:3300/0 conn(0x7fd83c0d8490 msgr2=0x7fd83c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83bfff700 1 --2- 172.21.15.32:0/4094895228 >> v2:172.21.15.86:3300/0 conn(0x7fd83c0d8490 0x7fd83c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83bfff700 1 -- 172.21.15.32:0/4094895228 >> v1:172.21.15.32:6789/0 conn(0x7fd83c0d1da0 legacy=0x7fd83c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83bfff700 1 -- 172.21.15.32:0/4094895228 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c10e020 con 0x7fd83c10d010 2021-04-09T18:24:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83b7fe700 1 -- 172.21.15.32:0/4094895228 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263582196 0 0) 0x7fd824000ce0 con 0x7fd83c0d1da0 2021-04-09T18:24:38.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd840de7700 1 --2- 172.21.15.32:0/4094895228 >> v2:172.21.15.86:3300/0 conn(0x7fd83c0d8490 0x7fd83c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83bfff700 1 --2- 172.21.15.32:0/4094895228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c10d010 0x7fd83c0d1780 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7fd82c006650 tx=0x7fd82c009a80).ready entity=mon.0 client_cookie=7f0fac823fd06fcd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83b7fe700 1 -- 172.21.15.32:0/4094895228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c0009b0 con 0x7fd83c10d010 2021-04-09T18:24:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83b7fe700 1 -- 172.21.15.32:0/4094895228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c00c0c0 con 0x7fd83c10d010 2021-04-09T18:24:38.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd83b7fe700 1 -- 172.21.15.32:0/4094895228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c00c510 con 0x7fd83c10d010 2021-04-09T18:24:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd8427ca700 1 -- 172.21.15.32:0/4094895228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c10d010 msgr2=0x7fd83c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.516 7fd8427ca700 1 --2- 172.21.15.32:0/4094895228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c10d010 0x7fd83c0d1780 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7fd82c006650 tx=0x7fd82c009a80).stop 2021-04-09T18:24:38.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.517 7fd8427ca700 1 -- 172.21.15.32:0/4094895228 shutdown_connections 2021-04-09T18:24:38.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.517 7fd8427ca700 1 --2- 172.21.15.32:0/4094895228 >> v2:172.21.15.86:3300/0 conn(0x7fd83c0d8490 0x7fd83c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.517 7fd8427ca700 1 --2- 172.21.15.32:0/4094895228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c10d010 0x7fd83c0d1780 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.517 7fd8427ca700 1 -- 172.21.15.32:0/4094895228 >> 172.21.15.32:0/4094895228 conn(0x7fd83c0ce5f0 msgr2=0x7fd83c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:38.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.517 7fd8427ca700 1 -- 172.21.15.32:0/4094895228 shutdown_connections 2021-04-09T18:24:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.517 7fd8427ca700 1 -- 172.21.15.32:0/4094895228 wait complete. 2021-04-09T18:24:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.537 7fd8427ca700 1 Processor -- start 2021-04-09T18:24:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.537 7fd8427ca700 1 -- start start 2021-04-09T18:24:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd8427ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c179fd0 0x7fd83c0976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd8427ca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd83c097cc0 0x7fd83c1a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd8427ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd83c1a4b10 0x7fd83c1a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd8427ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c098140 con 0x7fd83c179fd0 2021-04-09T18:24:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd8427ca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c17a3d0 con 0x7fd83c097cc0 2021-04-09T18:24:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd8427ca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c17c870 con 0x7fd83c1a4b10 2021-04-09T18:24:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd83bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c179fd0 0x7fd83c0976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd83bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c179fd0 0x7fd83c0976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51342/0 (socket says 172.21.15.32:51342) 2021-04-09T18:24:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd83bfff700 1 -- 172.21.15.32:0/38995929 learned_addr learned my addr 172.21.15.32:0/38995929 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd833fff700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd83c097cc0 0x7fd83c1a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd83bfff700 1 -- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd83c1a4b10 msgr2=0x7fd83c1a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd840de7700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd83c1a4b10 0x7fd83c1a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd83bfff700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd83c1a4b10 0x7fd83c1a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd83bfff700 1 -- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd83c097cc0 msgr2=0x7fd83c1a44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd83bfff700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd83c097cc0 0x7fd83c1a44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd83bfff700 1 -- 172.21.15.32:0/38995929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c179bf0 con 0x7fd83c179fd0 2021-04-09T18:24:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd833fff700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd83c097cc0 0x7fd83c1a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd840de7700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd83c1a4b10 0x7fd83c1a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.538 7fd83bfff700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c179fd0 0x7fd83c0976a0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7fd82c009650 tx=0x7fd82c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.539 7fd8397fa700 1 -- 172.21.15.32:0/38995929 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c0093a0 con 0x7fd83c179fd0 2021-04-09T18:24:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.539 7fd8397fa700 1 -- 172.21.15.32:0/38995929 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c00bd50 con 0x7fd83c179fd0 2021-04-09T18:24:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.539 7fd8427ca700 1 -- 172.21.15.32:0/38995929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83c179bf0 con 0x7fd83c179fd0 2021-04-09T18:24:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.539 7fd8397fa700 1 -- 172.21.15.32:0/38995929 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c0052e0 con 0x7fd83c179fd0 2021-04-09T18:24:38.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.539 7fd8427ca700 1 -- 172.21.15.32:0/38995929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83c179bf0 con 0x7fd83c179fd0 2021-04-09T18:24:38.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.539 7fd8397fa700 1 -- 172.21.15.32:0/38995929 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd82c006030 con 0x7fd83c179fd0 2021-04-09T18:24:38.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.539 7fd8397fa700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd820018930 0x7fd82001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:38.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.539 7fd8397fa700 1 -- 172.21.15.32:0/38995929 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd82c0329e0 con 0x7fd83c179fd0 2021-04-09T18:24:38.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.540 7fd833fff700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd820018930 0x7fd82001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:38.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.540 7fd833fff700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd820018930 0x7fd82001adc0 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.559 7fd8427ca700 1 -- 172.21.15.32:0/38995929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83c1b7050 con 0x7fd83c179fd0 2021-04-09T18:24:38.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.563 7fd8397fa700 1 -- 172.21.15.32:0/38995929 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd82c0009c0 con 0x7fd83c179fd0 2021-04-09T18:24:38.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.756 7fd8427ca700 1 -- 172.21.15.32:0/38995929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd83c10d480 con 0x7fd83c179fd0 2021-04-09T18:24:38.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.756 7fd8397fa700 1 -- 172.21.15.32:0/38995929 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+2464 (secure 0 0 0) 0x7fd82c00bd50 con 0x7fd83c179fd0 2021-04-09T18:24:38.821 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 244 2021-04-09T18:24:38.821 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:38.821 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":244,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:24:37.053503","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7777,"mds_1":57074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_57074":{"gid":57074,"name":"c","rank":1,"incarnation":243,"state":"up:active","state_seq":13,"addr":"172.21.15.32:6834/3708886434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3708886434},{"type":"v1","addr":"172.21.15.32:6834","nonce":3708886434}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:24:38.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.758 7fd8427ca700 1 -- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd820018930 msgr2=0x7fd82001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:38.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.758 7fd8427ca700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd820018930 0x7fd82001adc0 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.758 7fd8427ca700 1 -- 172.21.15.32:0/38995929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c179fd0 msgr2=0x7fd83c0976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:38.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.758 7fd8427ca700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c179fd0 0x7fd83c0976a0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7fd82c009650 tx=0x7fd82c00b960).stop 2021-04-09T18:24:38.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.758 7fd8427ca700 1 -- 172.21.15.32:0/38995929 shutdown_connections 2021-04-09T18:24:38.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.759 7fd8427ca700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd820018930 0x7fd82001adc0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.759 7fd8427ca700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd83c1a4b10 0x7fd83c1a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.759 7fd8427ca700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd83c097cc0 0x7fd83c1a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.759 7fd8427ca700 1 --2- 172.21.15.32:0/38995929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83c179fd0 0x7fd83c0976a0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:38.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.759 7fd8427ca700 1 -- 172.21.15.32:0/38995929 >> 172.21.15.32:0/38995929 conn(0x7fd83c199680 msgr2=0x7fd83c1a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:38.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.759 7fd8427ca700 1 -- 172.21.15.32:0/38995929 shutdown_connections 2021-04-09T18:24:38.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:38.759 7fd8427ca700 1 -- 172.21.15.32:0/38995929 wait complete. 2021-04-09T18:24:39.234 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 244, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:22:40.389392', 'modified': '2021-04-09 18:24:37.053503', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7777, 'mds_1': 57074}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7777': {'gid': 7777, 'name': 'b', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/1938812020', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1938812020}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1938812020}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_57074': {'gid': 57074, 'name': 'c', 'rank': 1, 'incarnation': 243, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.32:6834/3708886434', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3708886434}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 3708886434}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:24:39.235 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:24:39.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:24:39.372 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:24:39.373 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:24:39.373 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:24:39.373 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 7777, 2021-04-09T18:24:39.373 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:24:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:24:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 118.91967996299999, 2021-04-09T18:24:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 244, 2021-04-09T18:24:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 107, 2021-04-09T18:24:39.374 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 107, 2021-04-09T18:24:39.375 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 124.388454579 2021-04-09T18:24:39.375 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:24:39.378 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 7777, "want_state": "up:active", "state": "up:active", "rank_uptime": 118.91967996299999, "mdsmap_epoch": 244, "osdmap_epoch": 107, "osdmap_epoch_barrier": 107, "uptime": 124.388454579 } 2021-04-09T18:24:39.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:24:39.520 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:24:39.520 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:24:39.520 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:24:39.521 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 57074, 2021-04-09T18:24:39.521 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:24:39.521 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:24:39.521 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 3.4042321200000001, 2021-04-09T18:24:39.521 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 244, 2021-04-09T18:24:39.522 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 107, 2021-04-09T18:24:39.522 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 107, 2021-04-09T18:24:39.522 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 48.601540970000002 2021-04-09T18:24:39.522 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:24:39.526 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 57074, "want_state": "up:active", "state": "up:active", "rank_uptime": 3.4042321200000001, "mdsmap_epoch": 244, "osdmap_epoch": 107, "osdmap_epoch_barrier": 107, "uptime": 48.601540970000002 } 2021-04-09T18:24:39.527 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:24:39.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:24:39.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:24:39.642 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:24:39.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:24:39.643 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:24:39.644 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:24:39.688 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:24:39.688 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:24:39.689 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:24:39.689 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:24:39.689 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:24:39.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[63257]: fuse finished with error 0 and tester_r 0 2021-04-09T18:24:39.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:24:39.796 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:24:39.796 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:24:39.840 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:24:39.840 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:24:39.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:24:39.898 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:24:39.943 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:24:39.943 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:24:39.944 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:24:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:24:39.986 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:24:39.986 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:24:39.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:24:39.987 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:24:40.067 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:24:40.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.153 7f76c3ca6700 1 Processor -- start 2021-04-09T18:24:40.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.153 7f76c3ca6700 1 -- start start 2021-04-09T18:24:40.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c3ca6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76bc0cffa0 0x7f76bc0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c3ca6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc0d7800 0x7f76bc0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c3ca6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76bc0d7280 con 0x7f76bc0d8220 2021-04-09T18:24:40.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c3ca6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76bc0d91e0 con 0x7f76bc0cffa0 2021-04-09T18:24:40.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c3ca6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc10dfe0 con 0x7f76bc0d7800 2021-04-09T18:24:40.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c1ac2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76bc0cffa0 0x7f76bc0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c22c3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f76bc0d8220 0x7f76bc0d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60942/0 (socket says 172.21.15.32:60942) 2021-04-09T18:24:40.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c22c3700 1 -- 172.21.15.32:0/3409109200 learned_addr learned my addr 172.21.15.32:0/3409109200 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:40.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c12c1700 1 --2- 172.21.15.32:0/3409109200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc0d7800 0x7f76bc0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c1ac2700 1 -- 172.21.15.32:0/3409109200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc0d7800 msgr2=0x7f76bc0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:40.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c1ac2700 1 --2- 172.21.15.32:0/3409109200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc0d7800 0x7f76bc0d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c1ac2700 1 -- 172.21.15.32:0/3409109200 >> v1:172.21.15.32:6789/0 conn(0x7f76bc0d8220 legacy=0x7f76bc0d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c1ac2700 1 -- 172.21.15.32:0/3409109200 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc10e200 con 0x7f76bc0cffa0 2021-04-09T18:24:40.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c0ac0700 1 -- 172.21.15.32:0/3409109200 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239840986 0 0) 0x7f76ac000d30 con 0x7f76bc0d8220 2021-04-09T18:24:40.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.154 7f76c1ac2700 1 --2- 172.21.15.32:0/3409109200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc0cffa0 0x7f76bc0d2440 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f76b4006650 tx=0x7f76b4009a00).ready entity=mon.0 client_cookie=3306e0b0c4f83dce server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:40.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.155 7f76c0ac0700 1 -- 172.21.15.32:0/3409109200 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76b4009df0 con 0x7f76bc0cffa0 2021-04-09T18:24:40.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.155 7f76c0ac0700 1 -- 172.21.15.32:0/3409109200 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b40009b0 con 0x7f76bc0cffa0 2021-04-09T18:24:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.155 7f76c0ac0700 1 -- 172.21.15.32:0/3409109200 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76b400c2b0 con 0x7f76bc0cffa0 2021-04-09T18:24:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.155 7f76c3ca6700 1 -- 172.21.15.32:0/3409109200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc0cffa0 msgr2=0x7f76bc0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.155 7f76c3ca6700 1 --2- 172.21.15.32:0/3409109200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc0cffa0 0x7f76bc0d2440 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f76b4006650 tx=0x7f76b4009a00).stop 2021-04-09T18:24:40.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.155 7f76c3ca6700 1 -- 172.21.15.32:0/3409109200 shutdown_connections 2021-04-09T18:24:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.155 7f76c3ca6700 1 --2- 172.21.15.32:0/3409109200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc0d7800 0x7f76bc0d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.155 7f76c3ca6700 1 --2- 172.21.15.32:0/3409109200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc0cffa0 0x7f76bc0d2440 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.155 7f76c3ca6700 1 -- 172.21.15.32:0/3409109200 >> 172.21.15.32:0/3409109200 conn(0x7f76bc0ce5c0 msgr2=0x7f76bc0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.156 7f76c3ca6700 1 -- 172.21.15.32:0/3409109200 shutdown_connections 2021-04-09T18:24:40.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.156 7f76c3ca6700 1 -- 172.21.15.32:0/3409109200 wait complete. 2021-04-09T18:24:40.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.176 7f76c3ca6700 1 Processor -- start 2021-04-09T18:24:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.176 7f76c3ca6700 1 -- start start 2021-04-09T18:24:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c3ca6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76bc0a80c0 0x7f76bc17bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c3ca6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc17b1d0 0x7f76bc17b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c3ca6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc08dc50 0x7f76bc08e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c3ca6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc17bc70 con 0x7f76bc08dc50 2021-04-09T18:24:40.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c3ca6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc08e670 con 0x7f76bc0a80c0 2021-04-09T18:24:40.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c3ca6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc08e820 con 0x7f76bc17b1d0 2021-04-09T18:24:40.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c22c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc08dc50 0x7f76bc08e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c22c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc08dc50 0x7f76bc08e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51362/0 (socket says 172.21.15.32:51362) 2021-04-09T18:24:40.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c22c3700 1 -- 172.21.15.32:0/900245315 learned_addr learned my addr 172.21.15.32:0/900245315 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:40.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c12c1700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc17b1d0 0x7f76bc17b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c1ac2700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76bc0a80c0 0x7f76bc17bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c22c3700 1 -- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc17b1d0 msgr2=0x7f76bc17b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c22c3700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc17b1d0 0x7f76bc17b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c22c3700 1 -- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76bc0a80c0 msgr2=0x7f76bc17bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c22c3700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76bc0a80c0 0x7f76bc17bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c22c3700 1 -- 172.21.15.32:0/900245315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc08c660 con 0x7f76bc08dc50 2021-04-09T18:24:40.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.177 7f76c12c1700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc17b1d0 0x7f76bc17b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:40.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.178 7f76c22c3700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc08dc50 0x7f76bc08e050 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f76ac007f90 tx=0x7f76ac00ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:40.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.178 7f76ba7fc700 1 -- 172.21.15.32:0/900245315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76ac000cc0 con 0x7f76bc08dc50 2021-04-09T18:24:40.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.178 7f76ba7fc700 1 -- 172.21.15.32:0/900245315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ac007020 con 0x7f76bc08dc50 2021-04-09T18:24:40.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.178 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc08c660 con 0x7f76bc08dc50 2021-04-09T18:24:40.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.178 7f76ba7fc700 1 -- 172.21.15.32:0/900245315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76ac007500 con 0x7f76bc08dc50 2021-04-09T18:24:40.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.178 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc08c660 con 0x7f76bc08dc50 2021-04-09T18:24:40.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.178 7f76ba7fc700 1 -- 172.21.15.32:0/900245315 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f76ac007200 con 0x7f76bc08dc50 2021-04-09T18:24:40.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.178 7f76ba7fc700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f769c018930 0x7f769c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.178 7f76ba7fc700 1 -- 172.21.15.32:0/900245315 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f76ac032d50 con 0x7f76bc08dc50 2021-04-09T18:24:40.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.179 7f76c1ac2700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f769c018930 0x7f769c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.179 7f76c1ac2700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f769c018930 0x7f769c01ae10 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:40.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.198 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc1b70c0 con 0x7f76bc08dc50 2021-04-09T18:24:40.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.201 7f76ba7fc700 1 -- 172.21.15.32:0/900245315 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f76ac007d00 con 0x7f76bc08dc50 2021-04-09T18:24:40.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.391 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f76bc0d7280 con 0x7f76bc08dc50 2021-04-09T18:24:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.392 7f76ba7fc700 1 -- 172.21.15.32:0/900245315 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v107) v1 ==== 81+0+30373 (secure 0 0 0) 0x7f76ac006f00 con 0x7f76bc08dc50 2021-04-09T18:24:40.458 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:40.458 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:24:40.458 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 107, 2021-04-09T18:24:40.458 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:24:40.459 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:24:40.459 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:22:41.390553", 2021-04-09T18:24:40.459 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:24:40.459 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:24:40.460 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:24:40.460 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:24:40.460 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:24:40.460 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:24:40.461 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:24:40.461 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:24:40.461 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:24:40.461 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:24:40.462 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:24:40.462 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:24:40.462 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:24:40.462 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:24:40.462 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:24:40.463 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 19, 2021-04-09T18:24:40.463 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:24:40.463 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:24:40.463 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:24:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:24:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:24:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:24:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:24:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:24:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:24:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:24:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:24:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:24:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:24:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:24:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:24:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:24:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:24:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:24:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:24:40.468 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:24:40.468 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:24:40.468 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:24:40.468 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:24:40.468 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:24:40.469 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:24:40.469 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:24:40.469 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:24:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:24:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:24:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:24:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:24:40.471 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:24:40.471 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:24:40.471 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:24:40.471 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:24:40.472 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:24:40.472 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:24:40.472 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:24:40.472 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:24:40.472 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:24:40.473 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:24:40.473 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:24:40.473 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:24:40.473 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:24:40.473 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:24:40.474 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:24:40.474 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:24:40.474 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:24:40.474 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:24:40.474 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:24:40.475 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:24:40.475 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:24:40.475 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:24:40.475 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:24:40.476 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:24:40.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.476 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:24:40.476 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:24:40.476 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:24:40.477 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:24:40.477 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:24:40.477 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:24:40.477 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:24:40.478 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:24:40.478 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:24:40.478 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:24:40.478 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:24:40.478 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:24:40.479 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:24:40.479 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:24:40.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.480 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 18, 2021-04-09T18:24:40.480 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:24:40.480 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:22:37.739108", 2021-04-09T18:24:40.481 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:24:40.481 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:24:40.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:24:40.481 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:24:40.482 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:24:40.482 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:24:40.482 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:24:40.482 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:24:40.482 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:24:40.483 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:24:40.483 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:24:40.483 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:24:40.483 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:24:40.484 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:24:40.484 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:24:40.484 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:24:40.484 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:24:40.484 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:24:40.485 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:24:40.485 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:24:40.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.485 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "106", 2021-04-09T18:24:40.486 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:24:40.486 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:24:40.486 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:24:40.486 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:24:40.486 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:24:40.487 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:24:40.487 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:24:40.487 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:24:40.487 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:24:40.487 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:24:40.488 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:24:40.488 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:24:40.488 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:24:40.488 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:24:40.489 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:24:40.489 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:24:40.489 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:24:40.489 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:24:40.489 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:24:40.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:24:40.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:24:40.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:24:40.490 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:24:40.491 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:24:40.491 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:24:40.491 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:24:40.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.491 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:24:40.492 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:24:40.492 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:24:40.492 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:24:40.492 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:24:40.492 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:24:40.493 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:24:40.493 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:24:40.493 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:24:40.493 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:24:40.494 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:24:40.494 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:24:40.494 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:24:40.494 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:24:40.494 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:24:40.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.495 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:24:40.495 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:24:40.495 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:24:40.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.496 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.497 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 19, 2021-04-09T18:24:40.497 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:24:40.497 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:22:39.152543", 2021-04-09T18:24:40.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:24:40.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:24:40.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:24:40.498 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:24:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:24:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:24:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:24:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:24:40.499 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:24:40.500 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:24:40.500 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:24:40.500 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:24:40.500 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:24:40.501 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:24:40.501 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:24:40.501 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:24:40.501 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:24:40.502 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:24:40.502 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:24:40.502 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:24:40.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.502 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "107", 2021-04-09T18:24:40.503 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:24:40.503 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:24:40.503 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:24:40.503 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:24:40.504 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:24:40.504 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:24:40.504 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:24:40.504 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:24:40.504 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:24:40.505 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:24:40.505 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:24:40.505 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:24:40.505 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:24:40.505 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:24:40.506 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:24:40.506 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:24:40.506 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:24:40.506 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:24:40.507 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:24:40.507 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:24:40.507 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:24:40.507 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:24:40.507 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:24:40.508 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:24:40.508 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:24:40.508 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:24:40.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.509 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:24:40.509 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:24:40.509 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:24:40.509 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:24:40.509 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:24:40.510 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:24:40.510 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:24:40.510 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:24:40.510 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:24:40.511 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:24:40.511 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:24:40.511 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:24:40.511 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:24:40.511 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:24:40.512 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:24:40.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.513 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:24:40.513 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:24:40.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.513 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:24:40.513 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:24:40.514 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:40.514 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:40.514 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:40.514 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:40.515 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:40.515 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:40.515 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:40.515 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 100, 2021-04-09T18:24:40.516 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:40.516 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:40.516 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:40.516 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.516 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:24:40.517 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:40.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:24:40.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:40.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.519 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.519 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:40.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:24:40.520 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:40.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.521 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.521 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:24:40.521 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:40.522 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.522 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.522 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:40.522 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.523 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:24:40.523 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:40.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.524 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:24:40.524 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:40.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.525 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.525 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:40.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.526 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:24:40.526 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:40.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.527 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.527 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:24:40.528 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:40.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.528 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.528 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:24:40.529 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:24:40.529 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:24:40.529 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:24:40.529 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:40.529 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:40.530 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:40.530 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.530 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.531 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:24:40.531 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:24:40.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:40.531 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:40.532 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:40.532 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:40.532 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:40.532 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:40.533 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:40.533 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:40.533 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:40.533 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:40.533 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:40.534 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.534 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:24:40.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:40.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:24:40.536 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:40.536 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.536 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.536 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:40.537 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.537 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.537 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:24:40.538 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:40.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.538 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:24:40.539 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:40.539 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.539 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.540 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:40.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:24:40.541 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:40.541 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.541 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.541 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:24:40.542 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:40.542 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.542 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.543 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:40.543 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.543 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:24:40.544 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:40.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.544 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.544 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:24:40.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:40.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.545 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.546 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:24:40.546 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:24:40.546 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:24:40.546 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:24:40.546 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:40.547 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:40.547 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:40.547 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.548 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:24:40.548 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:24:40.548 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:40.548 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:40.549 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:40.549 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:40.549 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:40.549 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:40.549 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:40.550 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:40.550 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:40.550 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:40.550 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:40.550 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.551 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.551 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:24:40.551 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:40.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.552 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:24:40.553 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:40.553 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.553 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.553 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:40.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:24:40.554 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:40.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.555 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.555 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:24:40.556 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:40.556 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.556 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.556 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.558 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:24:40.558 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:40.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.558 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.559 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:24:40.559 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:40.559 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.559 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.560 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:40.560 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.561 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.561 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:24:40.561 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:40.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:24:40.562 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:40.562 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.562 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.563 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:24:40.563 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:24:40.563 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:24:40.564 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:24:40.565 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:40.565 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:40.565 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:40.565 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.566 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:24:40.566 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:24:40.566 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:40.567 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:40.567 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:40.567 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:40.567 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:40.567 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:40.568 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:40.568 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:40.568 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:40.568 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:40.568 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:40.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.569 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.569 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:24:40.570 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:40.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.570 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:24:40.571 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:40.571 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.571 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.571 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:40.572 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.572 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.572 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:24:40.573 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:40.573 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.573 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.573 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:24:40.574 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:40.574 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.574 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.575 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:40.575 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.575 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.575 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:24:40.576 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:40.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.576 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.577 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:24:40.577 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:40.577 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.577 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.578 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:40.578 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.578 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:24:40.579 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:40.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.579 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.580 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:24:40.580 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:40.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.580 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.581 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:24:40.581 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:24:40.581 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:24:40.581 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:24:40.582 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:40.582 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:40.582 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:40.582 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.583 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:24:40.583 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:24:40.583 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:40.584 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:40.584 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:40.584 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:40.584 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:40.584 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:40.585 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:40.585 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:40.585 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:40.585 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:40.585 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:40.586 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.586 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.586 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.586 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:24:40.587 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:40.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.587 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.588 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:24:40.588 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:40.588 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.588 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.589 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:40.589 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.589 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.590 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:24:40.590 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:40.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.590 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.590 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.591 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:24:40.591 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:40.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.591 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.592 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:40.592 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.592 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.593 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:24:40.593 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:40.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.593 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.594 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:24:40.594 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:40.594 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.594 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.595 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:40.595 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.595 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.596 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:24:40.596 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:40.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.596 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.597 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:24:40.597 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:40.597 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.597 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.597 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.598 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:24:40.598 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:24:40.598 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:24:40.598 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:24:40.599 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:40.599 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:40.599 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:40.599 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:24:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:24:40.600 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:40.601 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:40.601 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:40.601 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:40.601 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:40.601 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:40.602 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:40.602 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:40.602 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:40.602 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:40.602 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:40.603 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.603 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:24:40.604 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:40.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.604 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:24:40.605 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:40.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.605 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.606 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:40.606 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.606 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:24:40.607 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:40.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.607 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:24:40.608 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:40.608 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.608 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.609 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:40.609 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.609 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.610 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:24:40.610 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:40.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.611 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:24:40.611 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:40.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.611 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.612 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:40.612 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.613 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:24:40.613 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:40.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.614 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:24:40.614 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:40.614 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.614 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.615 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:24:40.615 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:24:40.615 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:24:40.615 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:24:40.616 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:40.616 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:40.616 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:40.616 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.617 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:24:40.617 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:24:40.617 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:40.618 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:40.618 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:40.618 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:40.618 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:40.618 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:40.619 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:40.619 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 90, 2021-04-09T18:24:40.619 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:40.619 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:40.620 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:40.620 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.620 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:24:40.621 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:40.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.621 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.621 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:24:40.622 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:40.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.622 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.623 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:40.623 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.623 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:24:40.624 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:40.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.624 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:24:40.625 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:40.625 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.625 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.626 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:40.626 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.626 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:24:40.627 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:40.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.627 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.628 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:24:40.628 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:40.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.628 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.629 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:40.629 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.630 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:24:40.630 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:40.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.631 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:24:40.631 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:40.631 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.631 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.632 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:24:40.632 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:24:40.632 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:24:40.632 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:24:40.633 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:40.633 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:40.633 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:40.633 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.634 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:24:40.634 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:24:40.634 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:40.634 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:40.635 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:40.635 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:40.635 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:40.635 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:40.636 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:40.636 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:40.636 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:40.636 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:40.636 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:40.637 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.637 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:24:40.637 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:40.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.638 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.638 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:24:40.639 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:40.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.639 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.639 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:40.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:24:40.641 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:40.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.641 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:24:40.642 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:40.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.642 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.643 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:40.643 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.643 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:24:40.644 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:40.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.644 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:24:40.645 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:40.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.645 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.646 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:40.646 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:40.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:40.646 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:24:40.647 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:40.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:40.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:24:40.648 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:40.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.648 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.649 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:24:40.649 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:24:40.649 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:24:40.649 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:24:40.649 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:40.650 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:40.650 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:40.650 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:40.650 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.650 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:24:40.651 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:24:40.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.651 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:24:40.651 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:40.652 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:40.652 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:40.652 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:40.652 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:40.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.653 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:24:40.653 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:40.653 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:40.653 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:40.654 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:40.654 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:40.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.655 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:24:40.655 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:40.655 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:40.655 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:40.655 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:40.656 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:40.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.656 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.656 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:24:40.656 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:40.657 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:40.657 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:40.657 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:40.657 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:40.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.658 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:24:40.658 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:40.658 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:40.659 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:40.659 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:40.659 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:40.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.660 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:24:40.660 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:40.660 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:40.660 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:40.661 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:40.661 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:40.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.661 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:24:40.662 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:40.662 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:40.662 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:40.662 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:40.663 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:40.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:40.663 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:24:40.663 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:40.664 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:40.664 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:40.664 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:40.664 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:40.664 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.665 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:24:40.665 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:24:40.665 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:24:40.665 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:24:40.666 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:24:40.666 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:24:40.666 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:24:40.666 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:24:40.666 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:24:40.667 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:24:40.667 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:24:40.667 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:24:40.667 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:24:40.668 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:24:40.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:40.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:40.668 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:24:40.668 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:24:40.669 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:24:40.669 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:24:40.669 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:24:40.669 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:24:40.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f769c018930 msgr2=0x7f769c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f769c018930 0x7f769c01ae10 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc08dc50 msgr2=0x7f76bc08e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc08dc50 0x7f76bc08e050 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f76ac007f90 tx=0x7f76ac00ac50).stop 2021-04-09T18:24:40.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 shutdown_connections 2021-04-09T18:24:40.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f769c018930 0x7f769c01ae10 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76bc08dc50 0x7f76bc08e050 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76bc17b1d0 0x7f76bc17b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 --2- 172.21.15.32:0/900245315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76bc0a80c0 0x7f76bc17bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 >> 172.21.15.32:0/900245315 conn(0x7f76bc189b00 msgr2=0x7f76bc1921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:40.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 shutdown_connections 2021-04-09T18:24:40.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.396 7f76c3ca6700 1 -- 172.21.15.32:0/900245315 wait complete. 2021-04-09T18:24:40.675 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:40.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.761 7ff365dcb700 1 Processor -- start 2021-04-09T18:24:40.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.761 7ff365dcb700 1 -- start start 2021-04-09T18:24:40.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.761 7ff365dcb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff3600cffa0 0x7ff3600d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.761 7ff365dcb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3600d7800 0x7ff3600d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.761 7ff365dcb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3600d7280 con 0x7ff3600d8220 2021-04-09T18:24:40.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.761 7ff365dcb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3600d91e0 con 0x7ff3600cffa0 2021-04-09T18:24:40.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.761 7ff365dcb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36010dfe0 con 0x7ff3600d7800 2021-04-09T18:24:40.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35effd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3600d7800 0x7ff3600d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35f7fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff3600cffa0 0x7ff3600d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35ffff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff3600d8220 0x7ff3600d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58884/0 (socket says 172.21.15.32:58884) 2021-04-09T18:24:40.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35ffff700 1 -- 172.21.15.32:0/1350657382 learned_addr learned my addr 172.21.15.32:0/1350657382 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:40.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35e7fc700 1 -- 172.21.15.32:0/1350657382 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5872801 0 0) 0x7ff358001440 con 0x7ff3600d8220 2021-04-09T18:24:40.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35e7fc700 1 -- 172.21.15.32:0/1350657382 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff348002fe0 con 0x7ff3600d8220 2021-04-09T18:24:40.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35f7fe700 1 -- 172.21.15.32:0/1350657382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3600d7800 msgr2=0x7ff3600d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:40.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35f7fe700 1 --2- 172.21.15.32:0/1350657382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3600d7800 0x7ff3600d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35f7fe700 1 -- 172.21.15.32:0/1350657382 >> v1:172.21.15.86:6789/0 conn(0x7ff3600d8220 legacy=0x7ff3600d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35f7fe700 1 -- 172.21.15.32:0/1350657382 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36010e200 con 0x7ff3600cffa0 2021-04-09T18:24:40.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35effd700 1 --2- 172.21.15.32:0/1350657382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3600d7800 0x7ff3600d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:40.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35f7fe700 1 --2- 172.21.15.32:0/1350657382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600cffa0 0x7ff3600d2440 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7ff350006650 tx=0x7ff350009a80).ready entity=mon.1 client_cookie=5e8f4dbc2a355595 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:40.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35e7fc700 1 -- 172.21.15.32:0/1350657382 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3500009b0 con 0x7ff3600cffa0 2021-04-09T18:24:40.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35e7fc700 1 -- 172.21.15.32:0/1350657382 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35000c0c0 con 0x7ff3600cffa0 2021-04-09T18:24:40.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.762 7ff35e7fc700 1 -- 172.21.15.32:0/1350657382 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff35000c510 con 0x7ff3600cffa0 2021-04-09T18:24:40.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.763 7ff365dcb700 1 -- 172.21.15.32:0/1350657382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600cffa0 msgr2=0x7ff3600d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.763 7ff365dcb700 1 --2- 172.21.15.32:0/1350657382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600cffa0 0x7ff3600d2440 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7ff350006650 tx=0x7ff350009a80).stop 2021-04-09T18:24:40.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.763 7ff365dcb700 1 -- 172.21.15.32:0/1350657382 shutdown_connections 2021-04-09T18:24:40.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.763 7ff365dcb700 1 --2- 172.21.15.32:0/1350657382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3600d7800 0x7ff3600d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.763 7ff365dcb700 1 --2- 172.21.15.32:0/1350657382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600cffa0 0x7ff3600d2440 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.763 7ff365dcb700 1 -- 172.21.15.32:0/1350657382 >> 172.21.15.32:0/1350657382 conn(0x7ff3600ce5c0 msgr2=0x7ff3600cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:40.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.763 7ff365dcb700 1 -- 172.21.15.32:0/1350657382 shutdown_connections 2021-04-09T18:24:40.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.763 7ff365dcb700 1 -- 172.21.15.32:0/1350657382 wait complete. 2021-04-09T18:24:40.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.783 7ff365dcb700 1 Processor -- start 2021-04-09T18:24:40.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.783 7ff365dcb700 1 -- start start 2021-04-09T18:24:40.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff365dcb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600a80c0 0x7ff36017bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff365dcb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff36017b1d0 0x7ff36017b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff365dcb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36008dc50 0x7ff36008e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff365dcb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36017bc70 con 0x7ff36008dc50 2021-04-09T18:24:40.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff365dcb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36008e670 con 0x7ff3600a80c0 2021-04-09T18:24:40.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff365dcb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36008e850 con 0x7ff36017b1d0 2021-04-09T18:24:40.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36008dc50 0x7ff36008e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36008dc50 0x7ff36008e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51372/0 (socket says 172.21.15.32:51372) 2021-04-09T18:24:40.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35ffff700 1 -- 172.21.15.32:0/1648361514 learned_addr learned my addr 172.21.15.32:0/1648361514 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:40.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35effd700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff36017b1d0 0x7ff36017b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35f7fe700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600a80c0 0x7ff36017bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35ffff700 1 -- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff36017b1d0 msgr2=0x7ff36017b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35ffff700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff36017b1d0 0x7ff36017b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35ffff700 1 -- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600a80c0 msgr2=0x7ff36017bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35ffff700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600a80c0 0x7ff36017bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35ffff700 1 -- 172.21.15.32:0/1648361514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36008c660 con 0x7ff36008dc50 2021-04-09T18:24:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.784 7ff35ffff700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36008dc50 0x7ff36008e050 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7ff35800a940 tx=0x7ff35800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:40.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.785 7ff347fff700 1 -- 172.21.15.32:0/1648361514 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3580070d0 con 0x7ff36008dc50 2021-04-09T18:24:40.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.785 7ff347fff700 1 -- 172.21.15.32:0/1648361514 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff358007410 con 0x7ff36008dc50 2021-04-09T18:24:40.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.785 7ff347fff700 1 -- 172.21.15.32:0/1648361514 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3580070d0 con 0x7ff36008dc50 2021-04-09T18:24:40.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.785 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36008c660 con 0x7ff36008dc50 2021-04-09T18:24:40.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.785 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36008c660 con 0x7ff36008dc50 2021-04-09T18:24:40.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.785 7ff347fff700 1 -- 172.21.15.32:0/1648361514 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff35800ae20 con 0x7ff36008dc50 2021-04-09T18:24:40.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.785 7ff347fff700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff340018900 0x7ff34001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:40.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.785 7ff347fff700 1 -- 172.21.15.32:0/1648361514 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff358032d70 con 0x7ff36008dc50 2021-04-09T18:24:40.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.786 7ff35f7fe700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff340018900 0x7ff34001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:40.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.786 7ff35f7fe700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff340018900 0x7ff34001adc0 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.805 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36008e850 con 0x7ff36008dc50 2021-04-09T18:24:40.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:40.809 7ff347fff700 1 -- 172.21.15.32:0/1648361514 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff358007db0 con 0x7ff36008dc50 2021-04-09T18:24:41.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.002 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff36008e850 con 0x7ff36008dc50 2021-04-09T18:24:41.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.003 7ff347fff700 1 -- 172.21.15.32:0/1648361514 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+2802 (secure 0 0 0) 0x7ff358007410 con 0x7ff36008dc50 2021-04-09T18:24:41.067 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 245 2021-04-09T18:24:41.067 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:41.067 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":245,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8859,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3459836132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3459836132},{"type":"v1","addr":"172.21.15.32:6835","nonce":3459836132}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":245},{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:24:37.053503","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7777,"mds_1":57074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7777":{"gid":7777,"name":"b","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/1938812020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1938812020},{"type":"v1","addr":"172.21.15.86:6837","nonce":1938812020}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_57074":{"gid":57074,"name":"c","rank":1,"incarnation":243,"state":"up:active","state_seq":13,"addr":"172.21.15.32:6834/3708886434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3708886434},{"type":"v1","addr":"172.21.15.32:6834","nonce":3708886434}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:24:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff340018900 msgr2=0x7ff34001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff340018900 0x7ff34001adc0 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36008dc50 msgr2=0x7ff36008e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36008dc50 0x7ff36008e050 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7ff35800a940 tx=0x7ff35800ace0).stop 2021-04-09T18:24:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 shutdown_connections 2021-04-09T18:24:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff340018900 0x7ff34001adc0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff36008dc50 0x7ff36008e050 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff36017b1d0 0x7ff36017b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 --2- 172.21.15.32:0/1648361514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600a80c0 0x7ff36017bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.005 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 >> 172.21.15.32:0/1648361514 conn(0x7ff360189b00 msgr2=0x7ff3601921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:41.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.006 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 shutdown_connections 2021-04-09T18:24:41.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.006 7ff365dcb700 1 -- 172.21.15.32:0/1648361514 wait complete. 2021-04-09T18:24:41.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:24:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.168 7f8dfad05700 1 Processor -- start 2021-04-09T18:24:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.168 7f8dfad05700 1 -- start start 2021-04-09T18:24:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8dfad05700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8df40d6c90 0x7f8df40d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8dfad05700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8df40d2450 0x7f8df410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8dfad05700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df40d7290 con 0x7f8df40cffb0 2021-04-09T18:24:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8dfad05700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df40d7090 con 0x7f8df40d6c90 2021-04-09T18:24:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8dfad05700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df40d91f0 con 0x7f8df40d2450 2021-04-09T18:24:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8df3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8df40d6c90 0x7f8df40d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8df9322700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8df40d2450 0x7f8df410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8df8b21700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8df40cffb0 0x7f8df40d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60970/0 (socket says 172.21.15.32:60970) 2021-04-09T18:24:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8df8b21700 1 -- 172.21.15.32:0/137361068 learned_addr learned my addr 172.21.15.32:0/137361068 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8df3fff700 1 -- 172.21.15.32:0/137361068 >> v2:172.21.15.86:3300/0 conn(0x7f8df40d2450 msgr2=0x7f8df410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8df3fff700 1 --2- 172.21.15.32:0/137361068 >> v2:172.21.15.86:3300/0 conn(0x7f8df40d2450 0x7f8df410cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8df3fff700 1 -- 172.21.15.32:0/137361068 >> v1:172.21.15.32:6789/0 conn(0x7f8df40cffb0 legacy=0x7f8df40d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8df3fff700 1 -- 172.21.15.32:0/137361068 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df40d2850 con 0x7f8df40d6c90 2021-04-09T18:24:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.169 7f8df9322700 1 --2- 172.21.15.32:0/137361068 >> v2:172.21.15.86:3300/0 conn(0x7f8df40d2450 0x7f8df410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.170 7f8df3fff700 1 --2- 172.21.15.32:0/137361068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40d6c90 0x7f8df40d8000 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f8de8006650 tx=0x7f8de8009a80).ready entity=mon.0 client_cookie=a95023ddf387b309 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.171 7f8df37fe700 1 -- 172.21.15.32:0/137361068 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8de80009b0 con 0x7f8df40d6c90 2021-04-09T18:24:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.171 7f8df37fe700 1 -- 172.21.15.32:0/137361068 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de800c0c0 con 0x7f8df40d6c90 2021-04-09T18:24:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.171 7f8df37fe700 1 -- 172.21.15.32:0/137361068 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8de800c510 con 0x7f8df40d6c90 2021-04-09T18:24:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.171 7f8dfad05700 1 -- 172.21.15.32:0/137361068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40d6c90 msgr2=0x7f8df40d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.171 7f8dfad05700 1 --2- 172.21.15.32:0/137361068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40d6c90 0x7f8df40d8000 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f8de8006650 tx=0x7f8de8009a80).stop 2021-04-09T18:24:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.172 7f8dfad05700 1 -- 172.21.15.32:0/137361068 shutdown_connections 2021-04-09T18:24:41.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.172 7f8dfad05700 1 --2- 172.21.15.32:0/137361068 >> v2:172.21.15.86:3300/0 conn(0x7f8df40d2450 0x7f8df410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.172 7f8dfad05700 1 --2- 172.21.15.32:0/137361068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40d6c90 0x7f8df40d8000 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.172 7f8dfad05700 1 -- 172.21.15.32:0/137361068 >> 172.21.15.32:0/137361068 conn(0x7f8df40ce5d0 msgr2=0x7f8df40cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:41.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.172 7f8dfad05700 1 -- 172.21.15.32:0/137361068 shutdown_connections 2021-04-09T18:24:41.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.172 7f8dfad05700 1 -- 172.21.15.32:0/137361068 wait complete. 2021-04-09T18:24:41.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.192 7f8dfad05700 1 Processor -- start 2021-04-09T18:24:41.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.192 7f8dfad05700 1 -- start start 2021-04-09T18:24:41.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.192 7f8dfad05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df408d210 0x7f8df408fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.192 7f8dfad05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40903f0 0x7f8df41a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.192 7f8dfad05700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df41a6080 0x7f8df41a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8dfad05700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df408d690 con 0x7f8df40903f0 2021-04-09T18:24:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8dfad05700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df40b0fd0 con 0x7f8df408d210 2021-04-09T18:24:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8dfad05700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df41a6480 con 0x7f8df41a6080 2021-04-09T18:24:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40903f0 0x7f8df41a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40903f0 0x7f8df41a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51386/0 (socket says 172.21.15.32:51386) 2021-04-09T18:24:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df3fff700 1 -- 172.21.15.32:0/3289908935 learned_addr learned my addr 172.21.15.32:0/3289908935 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df9322700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df41a6080 0x7f8df41a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df8b21700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df408d210 0x7f8df408fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df3fff700 1 -- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df41a6080 msgr2=0x7f8df41a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df3fff700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df41a6080 0x7f8df41a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df3fff700 1 -- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df408d210 msgr2=0x7f8df408fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df3fff700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df408d210 0x7f8df408fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df3fff700 1 -- 172.21.15.32:0/3289908935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df408d8b0 con 0x7f8df40903f0 2021-04-09T18:24:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df8b21700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df408d210 0x7f8df408fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df3fff700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40903f0 0x7f8df41a5a60 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f8de8009630 tx=0x7f8de800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df17fa700 1 -- 172.21.15.32:0/3289908935 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8de80093a0 con 0x7f8df40903f0 2021-04-09T18:24:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df17fa700 1 -- 172.21.15.32:0/3289908935 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de800bd70 con 0x7f8df40903f0 2021-04-09T18:24:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df408d8b0 con 0x7f8df40903f0 2021-04-09T18:24:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8df17fa700 1 -- 172.21.15.32:0/3289908935 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8de8005560 con 0x7f8df40903f0 2021-04-09T18:24:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.193 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df408d8b0 con 0x7f8df40903f0 2021-04-09T18:24:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.194 7f8df17fa700 1 -- 172.21.15.32:0/3289908935 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8de8005700 con 0x7f8df40903f0 2021-04-09T18:24:41.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.194 7f8df17fa700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8dd4018930 0x7f8dd401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:41.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.194 7f8df17fa700 1 -- 172.21.15.32:0/3289908935 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8de80329d0 con 0x7f8df40903f0 2021-04-09T18:24:41.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.194 7f8df8b21700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8dd4018930 0x7f8dd401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:41.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.194 7f8df8b21700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8dd4018930 0x7f8dd401ae10 crc :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:41.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.213 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df41a52b0 con 0x7f8df40903f0 2021-04-09T18:24:41.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.216 7f8df17fa700 1 -- 172.21.15.32:0/3289908935 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8de80054f0 con 0x7f8df40903f0 2021-04-09T18:24:41.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:41.400 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8df41a52b0 con 0x7f8df40903f0 2021-04-09T18:24:42.233 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:24:42.170 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:24:42.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.176 7f8df17fa700 1 -- 172.21.15.32:0/3289908935 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v246) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8de8005700 con 0x7f8df40903f0 2021-04-09T18:24:42.240 INFO:tasks.ceph.mds.b.smithi086.stderr: -4592> 2021-04-09 18:22:34.920 7fb9c5d9d700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2021-04-09T18:24:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:24:42.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8dd4018930 msgr2=0x7f8dd401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8dd4018930 0x7f8dd401ae10 crc :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40903f0 msgr2=0x7f8df41a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40903f0 0x7f8df41a5a60 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f8de8009630 tx=0x7f8de800b980).stop 2021-04-09T18:24:42.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 shutdown_connections 2021-04-09T18:24:42.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8dd4018930 0x7f8dd401ae10 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df40903f0 0x7f8df41a5a60 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df41a6080 0x7f8df41a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 --2- 172.21.15.32:0/3289908935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df408d210 0x7f8df408fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 >> 172.21.15.32:0/3289908935 conn(0x7f8df41a8f50 msgr2=0x7f8df4188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:42.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 shutdown_connections 2021-04-09T18:24:42.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.218 7f8dfad05700 1 -- 172.21.15.32:0/3289908935 wait complete. 2021-04-09T18:24:42.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:42.299 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:24:42.304 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:24:42.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.383 7f23d4eae700 1 Processor -- start 2021-04-09T18:24:42.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.383 7f23d4eae700 1 -- start start 2021-04-09T18:24:42.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.384 7f23d4eae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d00d7ca0 0x7f23d00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.384 7f23d4eae700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d010d480 con 0x7f23d010d010 2021-04-09T18:24:42.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.384 7f23d4eae700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d00d86c0 con 0x7f23d010da00 2021-04-09T18:24:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.384 7f23d4eae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d00d9220 con 0x7f23d00d7ca0 2021-04-09T18:24:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.384 7f23ceb13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f23d010d010 0x7f23d00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:60996/0 (socket says 172.21.15.32:60996) 2021-04-09T18:24:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.384 7f23ceb13700 1 -- 172.21.15.32:0/2716811605 learned_addr learned my addr 172.21.15.32:0/2716811605 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.384 7f23ce312700 1 --2- 172.21.15.32:0/2716811605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d00d7ca0 0x7f23d00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.384 7f23cdb11700 1 -- 172.21.15.32:0/2716811605 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881229391 0 0) 0x7f23c0000ce0 con 0x7f23d010d010 2021-04-09T18:24:42.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23cdb11700 1 -- 172.21.15.32:0/2716811605 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b8002fe0 con 0x7f23d010d010 2021-04-09T18:24:42.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23cdb11700 1 -- 172.21.15.32:0/2716811605 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334913184 0 0) 0x7f23c8000d30 con 0x7f23d010da00 2021-04-09T18:24:42.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23cdb11700 1 -- 172.21.15.32:0/2716811605 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b80042c0 con 0x7f23d010da00 2021-04-09T18:24:42.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23ce312700 1 -- 172.21.15.32:0/2716811605 >> v1:172.21.15.86:6789/0 conn(0x7f23d010da00 legacy=0x7f23d010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23ce312700 1 -- 172.21.15.32:0/2716811605 >> v1:172.21.15.32:6789/0 conn(0x7f23d010d010 legacy=0x7f23d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23ce312700 1 -- 172.21.15.32:0/2716811605 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d00d1f70 con 0x7f23d00d7ca0 2021-04-09T18:24:42.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23ce312700 1 --2- 172.21.15.32:0/2716811605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d00d7ca0 0x7f23d00d80a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f23c4006650 tx=0x7f23c4009a80).ready entity=mon.2 client_cookie=230c4bd0ec5a4304 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:42.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23cdb11700 1 -- 172.21.15.32:0/2716811605 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23c40009b0 con 0x7f23d00d7ca0 2021-04-09T18:24:42.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23cdb11700 1 -- 172.21.15.32:0/2716811605 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c400c0c0 con 0x7f23d00d7ca0 2021-04-09T18:24:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23cdb11700 1 -- 172.21.15.32:0/2716811605 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23c400c510 con 0x7f23d00d7ca0 2021-04-09T18:24:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23d4eae700 1 -- 172.21.15.32:0/2716811605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d00d7ca0 msgr2=0x7f23d00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.385 7f23d4eae700 1 --2- 172.21.15.32:0/2716811605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d00d7ca0 0x7f23d00d80a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f23c4006650 tx=0x7f23c4009a80).stop 2021-04-09T18:24:42.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.386 7f23d4eae700 1 -- 172.21.15.32:0/2716811605 shutdown_connections 2021-04-09T18:24:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.386 7f23d4eae700 1 --2- 172.21.15.32:0/2716811605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d00d7ca0 0x7f23d00d80a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.386 7f23d4eae700 1 -- 172.21.15.32:0/2716811605 >> 172.21.15.32:0/2716811605 conn(0x7f23d00ce5f0 msgr2=0x7f23d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.386 7f23d4eae700 1 -- 172.21.15.32:0/2716811605 shutdown_connections 2021-04-09T18:24:42.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.386 7f23d4eae700 1 -- 172.21.15.32:0/2716811605 wait complete. 2021-04-09T18:24:42.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.406 7f23d4eae700 1 Processor -- start 2021-04-09T18:24:42.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.406 7f23d4eae700 1 -- start start 2021-04-09T18:24:42.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23d4eae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23d0179610 0x7f23d0179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23d4eae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d017a6e0 0x7f23d017aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23d4eae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23d017b100 0x7f23d008db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23d4eae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d017d720 con 0x7f23d0179610 2021-04-09T18:24:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23d4eae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d0185ba0 con 0x7f23d017b100 2021-04-09T18:24:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23d4eae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d008e1b0 con 0x7f23d017a6e0 2021-04-09T18:24:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ceb13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23d0179610 0x7f23d0179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ceb13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23d0179610 0x7f23d0179a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51412/0 (socket says 172.21.15.32:51412) 2021-04-09T18:24:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ceb13700 1 -- 172.21.15.32:0/106397491 learned_addr learned my addr 172.21.15.32:0/106397491 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23cf314700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23d017b100 0x7f23d008db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ce312700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d017a6e0 0x7f23d017aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ceb13700 1 -- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d017a6e0 msgr2=0x7f23d017aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ceb13700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d017a6e0 0x7f23d017aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ceb13700 1 -- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23d017b100 msgr2=0x7f23d008db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ceb13700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23d017b100 0x7f23d008db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ceb13700 1 -- 172.21.15.32:0/106397491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d008e3d0 con 0x7f23d0179610 2021-04-09T18:24:42.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23ceb13700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23d0179610 0x7f23d0179a10 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7f23c000b530 tx=0x7f23c000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:42.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23bf7fe700 1 -- 172.21.15.32:0/106397491 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23c000bcc0 con 0x7f23d0179610 2021-04-09T18:24:42.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23bf7fe700 1 -- 172.21.15.32:0/106397491 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c0006030 con 0x7f23d0179610 2021-04-09T18:24:42.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23bf7fe700 1 -- 172.21.15.32:0/106397491 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23c0004620 con 0x7f23d0179610 2021-04-09T18:24:42.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.407 7f23d4eae700 1 -- 172.21.15.32:0/106397491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d008e3d0 con 0x7f23d0179610 2021-04-09T18:24:42.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.408 7f23d4eae700 1 -- 172.21.15.32:0/106397491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d008e3d0 con 0x7f23d0179610 2021-04-09T18:24:42.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.408 7f23bf7fe700 1 -- 172.21.15.32:0/106397491 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f23c00043a0 con 0x7f23d0179610 2021-04-09T18:24:42.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.408 7f23bf7fe700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f23b4018930 0x7f23b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.408 7f23bf7fe700 1 -- 172.21.15.32:0/106397491 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f23c0032a80 con 0x7f23d0179610 2021-04-09T18:24:42.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.408 7f23ce312700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f23b4018930 0x7f23b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.409 7f23ce312700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f23b4018930 0x7f23b401ae10 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:42.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.428 7f23d4eae700 1 -- 172.21.15.32:0/106397491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d017a1b0 con 0x7f23d0179610 2021-04-09T18:24:42.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.432 7f23bf7fe700 1 -- 172.21.15.32:0/106397491 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f23c0006b90 con 0x7f23d0179610 2021-04-09T18:24:42.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.624 7f23d4eae700 1 -- 172.21.15.32:0/106397491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23d017a1b0 con 0x7f23d0179610 2021-04-09T18:24:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.624 7f23bf7fe700 1 -- 172.21.15.32:0/106397491 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f23c00043a0 con 0x7f23d0179610 2021-04-09T18:24:42.688 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 246 2021-04-09T18:24:42.689 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:42.689 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":246,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8859,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3459836132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3459836132},{"type":"v1","addr":"172.21.15.32:6835","nonce":3459836132}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":245},{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":246,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:22:40.389392","modified":"2021-04-09 18:24:42.171208","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-04-09T18:24:42.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 -- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f23b4018930 msgr2=0x7f23b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f23b4018930 0x7f23b401ae10 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 -- 172.21.15.32:0/106397491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23d0179610 msgr2=0x7f23d0179a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23d0179610 0x7f23d0179a10 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7f23c000b530 tx=0x7f23c000b8d0).stop 2021-04-09T18:24:42.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 -- 172.21.15.32:0/106397491 shutdown_connections 2021-04-09T18:24:42.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f23b4018930 0x7f23b401ae10 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23d017b100 0x7f23d008db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23d017a6e0 0x7f23d017aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 --2- 172.21.15.32:0/106397491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23d0179610 0x7f23d0179a10 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 -- 172.21.15.32:0/106397491 >> 172.21.15.32:0/106397491 conn(0x7f23d0190550 msgr2=0x7f23d01b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 -- 172.21.15.32:0/106397491 shutdown_connections 2021-04-09T18:24:42.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.627 7f23d4eae700 1 -- 172.21.15.32:0/106397491 wait complete. 2021-04-09T18:24:42.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:24:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.829 7f60bbacd700 1 Processor -- start 2021-04-09T18:24:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.830 7f60bbacd700 1 -- start start 2021-04-09T18:24:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.830 7f60bbacd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b40cffb0 0x7f60b40d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.830 7f60bbacd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60b40d7810 0x7f60b40d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.830 7f60bbacd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f60b40d5c80 0x7f60b40d6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.830 7f60bbacd700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b40d8590 con 0x7f60b40d7810 2021-04-09T18:24:42.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.830 7f60bbacd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b40d91f0 con 0x7f60b40d5c80 2021-04-09T18:24:42.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.830 7f60bbacd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40d7290 con 0x7f60b40cffb0 2021-04-09T18:24:42.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b90e8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60b40d7810 0x7f60b40d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b90e8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60b40d7810 0x7f60b40d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51424/0 (socket says 172.21.15.32:51424) 2021-04-09T18:24:42.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b90e8700 1 -- 172.21.15.32:0/1195960125 learned_addr learned my addr 172.21.15.32:0/1195960125 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:42.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60ba0ea700 1 --2- 172.21.15.32:0/1195960125 >> v2:172.21.15.86:3300/0 conn(0x7f60b40d5c80 0x7f60b40d6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b98e9700 1 --2- 172.21.15.32:0/1195960125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b40cffb0 0x7f60b40d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b90e8700 1 -- 172.21.15.32:0/1195960125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b40cffb0 msgr2=0x7f60b40d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:42.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b90e8700 1 --2- 172.21.15.32:0/1195960125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b40cffb0 0x7f60b40d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b90e8700 1 -- 172.21.15.32:0/1195960125 >> v2:172.21.15.86:3300/0 conn(0x7f60b40d5c80 msgr2=0x7f60b40d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:42.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b90e8700 1 --2- 172.21.15.32:0/1195960125 >> v2:172.21.15.86:3300/0 conn(0x7f60b40d5c80 0x7f60b40d6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b90e8700 1 -- 172.21.15.32:0/1195960125 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40d8230 con 0x7f60b40d7810 2021-04-09T18:24:42.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b90e8700 1 --2- 172.21.15.32:0/1195960125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b40d7810 0x7f60b40d7c10 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7f60a4006700 tx=0x7f60a4009b00).ready entity=mon.0 client_cookie=a4eb0d0973a0579a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:42.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b88e7700 1 -- 172.21.15.32:0/1195960125 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a400c0c0 con 0x7f60b40d7810 2021-04-09T18:24:42.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b88e7700 1 -- 172.21.15.32:0/1195960125 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a400c400 con 0x7f60b40d7810 2021-04-09T18:24:42.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60b88e7700 1 -- 172.21.15.32:0/1195960125 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a400c8e0 con 0x7f60b40d7810 2021-04-09T18:24:42.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60bbacd700 1 -- 172.21.15.32:0/1195960125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b40d7810 msgr2=0x7f60b40d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.831 7f60bbacd700 1 --2- 172.21.15.32:0/1195960125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b40d7810 0x7f60b40d7c10 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7f60a4006700 tx=0x7f60a4009b00).stop 2021-04-09T18:24:42.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.832 7f60bbacd700 1 -- 172.21.15.32:0/1195960125 shutdown_connections 2021-04-09T18:24:42.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.832 7f60bbacd700 1 --2- 172.21.15.32:0/1195960125 >> v2:172.21.15.86:3300/0 conn(0x7f60b40d5c80 0x7f60b40d6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.832 7f60bbacd700 1 --2- 172.21.15.32:0/1195960125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b40d7810 0x7f60b40d7c10 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.832 7f60bbacd700 1 --2- 172.21.15.32:0/1195960125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b40cffb0 0x7f60b40d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.832 7f60bbacd700 1 -- 172.21.15.32:0/1195960125 >> 172.21.15.32:0/1195960125 conn(0x7f60b40ce5d0 msgr2=0x7f60b40cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:42.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.832 7f60bbacd700 1 -- 172.21.15.32:0/1195960125 shutdown_connections 2021-04-09T18:24:42.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.832 7f60bbacd700 1 -- 172.21.15.32:0/1195960125 wait complete. 2021-04-09T18:24:42.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.852 7f60bbacd700 1 Processor -- start 2021-04-09T18:24:42.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.852 7f60bbacd700 1 -- start start 2021-04-09T18:24:42.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60bbacd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b4184370 0x7f60b4184770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60bbacd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b41a7ea0 0x7f60b41a82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60bbacd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60b40b7e00 0x7f60b40b8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60bbacd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41a7920 con 0x7f60b41a7ea0 2021-04-09T18:24:42.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60bbacd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41a88c0 con 0x7f60b40b7e00 2021-04-09T18:24:42.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60bbacd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40b8820 con 0x7f60b4184370 2021-04-09T18:24:42.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b90e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b41a7ea0 0x7f60b41a82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b90e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b41a7ea0 0x7f60b41a82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51426/0 (socket says 172.21.15.32:51426) 2021-04-09T18:24:42.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b90e8700 1 -- 172.21.15.32:0/2682115552 learned_addr learned my addr 172.21.15.32:0/2682115552 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:42.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60ba0ea700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60b40b7e00 0x7f60b40b8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b98e9700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b4184370 0x7f60b4184770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b90e8700 1 -- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b4184370 msgr2=0x7f60b4184770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b90e8700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b4184370 0x7f60b4184770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b90e8700 1 -- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60b40b7e00 msgr2=0x7f60b40b8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:42.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b90e8700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60b40b7e00 0x7f60b40b8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:42.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b90e8700 1 -- 172.21.15.32:0/2682115552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40b8a40 con 0x7f60b41a7ea0 2021-04-09T18:24:42.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.853 7f60b90e8700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b41a7ea0 0x7f60b41a82a0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f60a4006f20 tx=0x7f60a400b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:42.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.854 7f60aa7fc700 1 -- 172.21.15.32:0/2682115552 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a40090d0 con 0x7f60b41a7ea0 2021-04-09T18:24:42.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.854 7f60aa7fc700 1 -- 172.21.15.32:0/2682115552 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a400bd00 con 0x7f60b41a7ea0 2021-04-09T18:24:42.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.854 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b40b8a40 con 0x7f60b41a7ea0 2021-04-09T18:24:42.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.854 7f60aa7fc700 1 -- 172.21.15.32:0/2682115552 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a4005560 con 0x7f60b41a7ea0 2021-04-09T18:24:42.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.854 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b40b8a40 con 0x7f60b41a7ea0 2021-04-09T18:24:42.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.854 7f60aa7fc700 1 -- 172.21.15.32:0/2682115552 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f60a4000c90 con 0x7f60b41a7ea0 2021-04-09T18:24:42.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.854 7f60aa7fc700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:42.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.854 7f60aa7fc700 1 -- 172.21.15.32:0/2682115552 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f60a4032a70 con 0x7f60b41a7ea0 2021-04-09T18:24:42.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.855 7f60b98e9700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:42.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.855 7f60b98e9700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:42.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.874 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b4182e80 con 0x7f60b41a7ea0 2021-04-09T18:24:42.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:42.878 7f60aa7fc700 1 -- 172.21.15.32:0/2682115552 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f60a40090d0 con 0x7f60b41a7ea0 2021-04-09T18:24:43.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.072 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f60b4182e80 con 0x7f60b41a7ea0 2021-04-09T18:24:43.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.178 7f60aa7fc700 1 -- 172.21.15.32:0/2682115552 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v247) v1 ==== 104+0+0 (secure 0 0 0) 0x7f60a4000c90 con 0x7f60b41a7ea0 2021-04-09T18:24:43.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 msgr2=0x7f609c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b41a7ea0 msgr2=0x7f60b41a82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b41a7ea0 0x7f60b41a82a0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f60a4006f20 tx=0x7f60a400b910).stop 2021-04-09T18:24:43.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 shutdown_connections 2021-04-09T18:24:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60b41a7ea0 0x7f60b41a82a0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60b40b7e00 0x7f60b40b8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 --2- 172.21.15.32:0/2682115552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60b4184370 0x7f60b4184770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 >> 172.21.15.32:0/2682115552 conn(0x7f60b41a8f50 msgr2=0x7f60b4188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.187 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 shutdown_connections 2021-04-09T18:24:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.188 7f60bbacd700 1 -- 172.21.15.32:0/2682115552 wait complete. 2021-04-09T18:24:43.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:24:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.358 7fa44a68f700 1 Processor -- start 2021-04-09T18:24:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.358 7fa44a68f700 1 -- start start 2021-04-09T18:24:43.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa44a68f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4440d7cc0 0x7fa4440d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa44a68f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa44410da80 0x7fa44410de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa44a68f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44410d500 con 0x7fa44410d050 2021-04-09T18:24:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa44a68f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4440d9260 con 0x7fa44410da80 2021-04-09T18:24:43.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa44a68f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4440d0330 con 0x7fa4440d7cc0 2021-04-09T18:24:43.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa448cac700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa44410da80 0x7fa44410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:43.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa43bfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4440d7cc0 0x7fa4440d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:43.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa448cac700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa44410da80 0x7fa44410de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51438/0 (socket says 172.21.15.32:51438) 2021-04-09T18:24:43.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa443fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa44410d050 0x7fa4440cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:58950/0 (socket says 172.21.15.32:58950) 2021-04-09T18:24:43.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.359 7fa448cac700 1 -- 172.21.15.32:0/3028129565 learned_addr learned my addr 172.21.15.32:0/3028129565 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:43.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa448cac700 1 -- 172.21.15.32:0/3028129565 >> v2:172.21.15.86:3300/0 conn(0x7fa4440d7cc0 msgr2=0x7fa4440d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:43.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa448cac700 1 --2- 172.21.15.32:0/3028129565 >> v2:172.21.15.86:3300/0 conn(0x7fa4440d7cc0 0x7fa4440d80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa448cac700 1 -- 172.21.15.32:0/3028129565 >> v1:172.21.15.86:6789/0 conn(0x7fa44410d050 legacy=0x7fa4440cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa448cac700 1 -- 172.21.15.32:0/3028129565 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4440d86e0 con 0x7fa44410da80 2021-04-09T18:24:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa448cac700 1 --2- 172.21.15.32:0/3028129565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44410da80 0x7fa44410de80 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7fa43c006f00 tx=0x7fa43c009e90).ready entity=mon.0 client_cookie=5e9f795ef5361a1e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa4437fe700 1 -- 172.21.15.32:0/3028129565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c00b300 con 0x7fa44410da80 2021-04-09T18:24:43.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa4437fe700 1 -- 172.21.15.32:0/3028129565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c00b640 con 0x7fa44410da80 2021-04-09T18:24:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa4437fe700 1 -- 172.21.15.32:0/3028129565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c00bb20 con 0x7fa44410da80 2021-04-09T18:24:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa44a68f700 1 -- 172.21.15.32:0/3028129565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44410da80 msgr2=0x7fa44410de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.360 7fa44a68f700 1 --2- 172.21.15.32:0/3028129565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44410da80 0x7fa44410de80 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7fa43c006f00 tx=0x7fa43c009e90).stop 2021-04-09T18:24:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.361 7fa44a68f700 1 -- 172.21.15.32:0/3028129565 shutdown_connections 2021-04-09T18:24:43.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.361 7fa44a68f700 1 --2- 172.21.15.32:0/3028129565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44410da80 0x7fa44410de80 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.361 7fa44a68f700 1 --2- 172.21.15.32:0/3028129565 >> v2:172.21.15.86:3300/0 conn(0x7fa4440d7cc0 0x7fa4440d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.361 7fa44a68f700 1 -- 172.21.15.32:0/3028129565 >> 172.21.15.32:0/3028129565 conn(0x7fa4440ce620 msgr2=0x7fa4440d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.361 7fa44a68f700 1 -- 172.21.15.32:0/3028129565 shutdown_connections 2021-04-09T18:24:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.361 7fa44a68f700 1 -- 172.21.15.32:0/3028129565 wait complete. 2021-04-09T18:24:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.381 7fa44a68f700 1 Processor -- start 2021-04-09T18:24:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.381 7fa44a68f700 1 -- start start 2021-04-09T18:24:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.381 7fa44a68f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44409b2f0 0x7fa44409b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.381 7fa44a68f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409a8b0 0x7fa44409e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.381 7fa44a68f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44409e9c0 0x7fa44409edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:43.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.381 7fa44a68f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440bde80 con 0x7fa44409e9c0 2021-04-09T18:24:43.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.381 7fa44a68f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44409ad70 con 0x7fa44409b2f0 2021-04-09T18:24:43.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.381 7fa44a68f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44409bd10 con 0x7fa44409a8b0 2021-04-09T18:24:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa448cac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44409e9c0 0x7fa44409edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa448cac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44409e9c0 0x7fa44409edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51444/0 (socket says 172.21.15.32:51444) 2021-04-09T18:24:43.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa448cac700 1 -- 172.21.15.32:0/689113666 learned_addr learned my addr 172.21.15.32:0/689113666 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:43.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa443fff700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44409b2f0 0x7fa44409b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:43.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa43bfff700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409a8b0 0x7fa44409e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa448cac700 1 -- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409a8b0 msgr2=0x7fa44409e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa448cac700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409a8b0 0x7fa44409e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa448cac700 1 -- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44409b2f0 msgr2=0x7fa44409b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa448cac700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44409b2f0 0x7fa44409b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa448cac700 1 -- 172.21.15.32:0/689113666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44409fbd0 con 0x7fa44409e9c0 2021-04-09T18:24:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa448cac700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44409e9c0 0x7fa44409edc0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7fa43c0099f0 tx=0x7fa43c00bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa4417fa700 1 -- 172.21.15.32:0/689113666 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c003030 con 0x7fa44409e9c0 2021-04-09T18:24:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa4417fa700 1 -- 172.21.15.32:0/689113666 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c000cc0 con 0x7fa44409e9c0 2021-04-09T18:24:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa4417fa700 1 -- 172.21.15.32:0/689113666 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c001530 con 0x7fa44409e9c0 2021-04-09T18:24:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa44a68f700 1 -- 172.21.15.32:0/689113666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44409fbd0 con 0x7fa44409e9c0 2021-04-09T18:24:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.382 7fa44a68f700 1 -- 172.21.15.32:0/689113666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44409fbd0 con 0x7fa44409e9c0 2021-04-09T18:24:43.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.383 7fa4417fa700 1 -- 172.21.15.32:0/689113666 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa43c001270 con 0x7fa44409e9c0 2021-04-09T18:24:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.383 7fa4417fa700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa428018930 0x7fa42801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.383 7fa4417fa700 1 -- 172.21.15.32:0/689113666 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fa43c032e80 con 0x7fa44409e9c0 2021-04-09T18:24:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.383 7fa443fff700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa428018930 0x7fa42801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:43.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.383 7fa443fff700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa428018930 0x7fa42801adc0 crc :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:43.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.403 7fa44a68f700 1 -- 172.21.15.32:0/689113666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4440bde80 con 0x7fa44409e9c0 2021-04-09T18:24:43.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.407 7fa4417fa700 1 -- 172.21.15.32:0/689113666 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa43c00be20 con 0x7fa44409e9c0 2021-04-09T18:24:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:43.592 7fa44a68f700 1 -- 172.21.15.32:0/689113666 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa4440bde80 con 0x7fa44409e9c0 2021-04-09T18:24:44.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.182 7fa4417fa700 1 -- 172.21.15.32:0/689113666 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v109) v1 ==== 185+0+0 (secure 0 0 0) 0x7fa43c01fb10 con 0x7fa44409e9c0 2021-04-09T18:24:44.270 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:24:44.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 -- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa428018930 msgr2=0x7fa42801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:44.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa428018930 0x7fa42801adc0 crc :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:44.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 -- 172.21.15.32:0/689113666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44409e9c0 msgr2=0x7fa44409edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:44.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44409e9c0 0x7fa44409edc0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7fa43c0099f0 tx=0x7fa43c00bce0).stop 2021-04-09T18:24:44.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 -- 172.21.15.32:0/689113666 shutdown_connections 2021-04-09T18:24:44.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa428018930 0x7fa42801adc0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:44.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa44409e9c0 0x7fa44409edc0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:44.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409a8b0 0x7fa44409e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:44.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 --2- 172.21.15.32:0/689113666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44409b2f0 0x7fa44409b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:44.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 -- 172.21.15.32:0/689113666 >> 172.21.15.32:0/689113666 conn(0x7fa4440ae6f0 msgr2=0x7fa444097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:44.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 -- 172.21.15.32:0/689113666 shutdown_connections 2021-04-09T18:24:44.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.210 7fa44a68f700 1 -- 172.21.15.32:0/689113666 wait complete. 2021-04-09T18:24:44.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:24:44.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.382 7f8e58ad1700 1 Processor -- start 2021-04-09T18:24:44.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.382 7f8e58ad1700 1 -- start start 2021-04-09T18:24:44.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.383 7f8e58ad1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540d7cc0 0x7f8e540d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:44.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.383 7f8e58ad1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5410d500 con 0x7f8e5410da80 2021-04-09T18:24:44.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.383 7f8e58ad1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e540d0330 con 0x7f8e5410d050 2021-04-09T18:24:44.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.383 7f8e58ad1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e540d9260 con 0x7f8e540d7cc0 2021-04-09T18:24:44.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.383 7f8e5305f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8e5410da80 0x7f8e5410de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:32808/0 (socket says 172.21.15.32:32808) 2021-04-09T18:24:44.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.383 7f8e5305f700 1 -- 172.21.15.32:0/3674286511 learned_addr learned my addr 172.21.15.32:0/3674286511 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:44.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.383 7f8e5205d700 1 --2- 172.21.15.32:0/3674286511 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540d7cc0 0x7f8e540d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:44.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.383 7f8e5185c700 1 -- 172.21.15.32:0/3674286511 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599135937 0 0) 0x7f8e4c000d30 con 0x7f8e5410da80 2021-04-09T18:24:44.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5185c700 1 -- 172.21.15.32:0/3674286511 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e3c002fe0 con 0x7f8e5410da80 2021-04-09T18:24:44.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5185c700 1 -- 172.21.15.32:0/3674286511 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437555656 0 0) 0x7f8e44000c60 con 0x7f8e5410d050 2021-04-09T18:24:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5185c700 1 -- 172.21.15.32:0/3674286511 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e3c003210 con 0x7f8e5410d050 2021-04-09T18:24:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5205d700 1 -- 172.21.15.32:0/3674286511 >> v1:172.21.15.86:6789/0 conn(0x7f8e5410d050 legacy=0x7f8e540cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5205d700 1 -- 172.21.15.32:0/3674286511 >> v1:172.21.15.32:6789/0 conn(0x7f8e5410da80 legacy=0x7f8e5410de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5185c700 1 -- 172.21.15.32:0/3674286511 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 284825967 0 0) 0x7f8e4c001460 con 0x7f8e5410da80 2021-04-09T18:24:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5205d700 1 -- 172.21.15.32:0/3674286511 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e54112140 con 0x7f8e540d7cc0 2021-04-09T18:24:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5205d700 1 --2- 172.21.15.32:0/3674286511 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540d7cc0 0x7f8e540d80c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8e48006650 tx=0x7f8e48009a00).ready entity=mon.2 client_cookie=b57a049be5a3f06f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5185c700 1 -- 172.21.15.32:0/3674286511 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e48009df0 con 0x7f8e540d7cc0 2021-04-09T18:24:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5185c700 1 -- 172.21.15.32:0/3674286511 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e480009b0 con 0x7f8e540d7cc0 2021-04-09T18:24:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e5185c700 1 -- 172.21.15.32:0/3674286511 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e4800c2b0 con 0x7f8e540d7cc0 2021-04-09T18:24:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e58ad1700 1 -- 172.21.15.32:0/3674286511 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540d7cc0 msgr2=0x7f8e540d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.384 7f8e58ad1700 1 --2- 172.21.15.32:0/3674286511 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540d7cc0 0x7f8e540d80c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8e48006650 tx=0x7f8e48009a00).stop 2021-04-09T18:24:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.385 7f8e58ad1700 1 -- 172.21.15.32:0/3674286511 shutdown_connections 2021-04-09T18:24:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.385 7f8e58ad1700 1 --2- 172.21.15.32:0/3674286511 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540d7cc0 0x7f8e540d80c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:44.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.385 7f8e58ad1700 1 -- 172.21.15.32:0/3674286511 >> 172.21.15.32:0/3674286511 conn(0x7f8e540ce620 msgr2=0x7f8e540d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:44.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.385 7f8e58ad1700 1 -- 172.21.15.32:0/3674286511 shutdown_connections 2021-04-09T18:24:44.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.385 7f8e58ad1700 1 -- 172.21.15.32:0/3674286511 wait complete. 2021-04-09T18:24:44.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.405 7f8e58ad1700 1 Processor -- start 2021-04-09T18:24:44.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.405 7f8e58ad1700 1 -- start start 2021-04-09T18:24:44.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e58ad1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540bd3f0 0x7f8e540bd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:44.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e58ad1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e540aac80 0x7f8e540ab080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:44.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e58ad1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e540ab6a0 0x7f8e540bdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:44.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e58ad1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e540aa780 con 0x7f8e540aac80 2021-04-09T18:24:44.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e58ad1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e540aa3b0 con 0x7f8e540ab6a0 2021-04-09T18:24:44.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e58ad1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e540be580 con 0x7f8e540bd3f0 2021-04-09T18:24:44.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5205d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e540aac80 0x7f8e540ab080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:44.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5205d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e540aac80 0x7f8e540ab080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51456/0 (socket says 172.21.15.32:51456) 2021-04-09T18:24:44.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5205d700 1 -- 172.21.15.32:0/864458783 learned_addr learned my addr 172.21.15.32:0/864458783 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:44.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5285e700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540bd3f0 0x7f8e540bd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:44.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5305f700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e540ab6a0 0x7f8e540bdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:44.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5205d700 1 -- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540bd3f0 msgr2=0x7f8e540bd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:44.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5205d700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540bd3f0 0x7f8e540bd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:44.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5205d700 1 -- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e540ab6a0 msgr2=0x7f8e540bdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:44.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5205d700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e540ab6a0 0x7f8e540bdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:44.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5205d700 1 -- 172.21.15.32:0/864458783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e540be7a0 con 0x7f8e540aac80 2021-04-09T18:24:44.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e5205d700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e540aac80 0x7f8e540ab080 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7f8e480096a0 tx=0x7f8e4800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:44.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e437fe700 1 -- 172.21.15.32:0/864458783 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e4800bd60 con 0x7f8e540aac80 2021-04-09T18:24:44.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e437fe700 1 -- 172.21.15.32:0/864458783 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e48006760 con 0x7f8e540aac80 2021-04-09T18:24:44.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e540be7a0 con 0x7f8e540aac80 2021-04-09T18:24:44.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.406 7f8e437fe700 1 -- 172.21.15.32:0/864458783 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e48000c90 con 0x7f8e540aac80 2021-04-09T18:24:44.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.407 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e540be7a0 con 0x7f8e540aac80 2021-04-09T18:24:44.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.407 7f8e437fe700 1 -- 172.21.15.32:0/864458783 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8e4800a000 con 0x7f8e540aac80 2021-04-09T18:24:44.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.407 7f8e437fe700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e380188c0 0x7f8e3801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:44.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.407 7f8e437fe700 1 -- 172.21.15.32:0/864458783 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5311+0+0 (secure 0 0 0) 0x7f8e480328d0 con 0x7f8e540aac80 2021-04-09T18:24:44.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.407 7f8e5285e700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e380188c0 0x7f8e3801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:44.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.408 7f8e5285e700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e380188c0 0x7f8e3801adc0 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:44.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.427 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e540b9040 con 0x7f8e540aac80 2021-04-09T18:24:44.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.430 7f8e437fe700 1 -- 172.21.15.32:0/864458783 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8e48006740 con 0x7f8e540aac80 2021-04-09T18:24:44.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:44.614 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8e5410d500 con 0x7f8e540aac80 2021-04-09T18:24:45.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.186 7f8e437fe700 1 -- 172.21.15.32:0/864458783 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v110) v1 ==== 173+0+0 (secure 0 0 0) 0x7f8e4801f720 con 0x7f8e540aac80 2021-04-09T18:24:45.292 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:24:45.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e380188c0 msgr2=0x7f8e3801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e380188c0 0x7f8e3801adc0 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e540aac80 msgr2=0x7f8e540ab080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e540aac80 0x7f8e540ab080 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7f8e480096a0 tx=0x7f8e4800b970).stop 2021-04-09T18:24:45.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 shutdown_connections 2021-04-09T18:24:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e380188c0 0x7f8e3801adc0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e540aac80 0x7f8e540ab080 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e540ab6a0 0x7f8e540bdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 --2- 172.21.15.32:0/864458783 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e540bd3f0 0x7f8e540bd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 >> 172.21.15.32:0/864458783 conn(0x7f8e540ae290 msgr2=0x7f8e54097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 shutdown_connections 2021-04-09T18:24:45.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.231 7f8e58ad1700 1 -- 172.21.15.32:0/864458783 wait complete. 2021-04-09T18:24:45.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:24:45.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.397 7f52d4e22700 1 Processor -- start 2021-04-09T18:24:45.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.397 7f52d4e22700 1 -- start start 2021-04-09T18:24:45.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.398 7f52d4e22700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52d00cffb0 0x7f52d00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.398 7f52d4e22700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f52d00d6c90 0x7f52d010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.398 7f52d4e22700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d00d7290 con 0x7f52d00d7810 2021-04-09T18:24:45.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.398 7f52d4e22700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d00d7090 con 0x7f52d00cffb0 2021-04-09T18:24:45.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.398 7f52d4e22700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d00d91f0 con 0x7f52d00d6c90 2021-04-09T18:24:45.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.398 7f52ceb13700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52d00cffb0 0x7f52d00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52ceb13700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52d00cffb0 0x7f52d00d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51462/0 (socket says 172.21.15.32:51462) 2021-04-09T18:24:45.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52ceb13700 1 -- 172.21.15.32:0/514187963 learned_addr learned my addr 172.21.15.32:0/514187963 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:45.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52cf314700 1 --2- 172.21.15.32:0/514187963 >> v2:172.21.15.86:3300/0 conn(0x7f52d00d6c90 0x7f52d010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52ceb13700 1 -- 172.21.15.32:0/514187963 >> v2:172.21.15.86:3300/0 conn(0x7f52d00d6c90 msgr2=0x7f52d010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:45.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52ceb13700 1 --2- 172.21.15.32:0/514187963 >> v2:172.21.15.86:3300/0 conn(0x7f52d00d6c90 0x7f52d010cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52ceb13700 1 -- 172.21.15.32:0/514187963 >> v1:172.21.15.86:6789/0 conn(0x7f52d00d7810 legacy=0x7f52d00d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52ceb13700 1 -- 172.21.15.32:0/514187963 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d00d8400 con 0x7f52d00cffb0 2021-04-09T18:24:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52ceb13700 1 --2- 172.21.15.32:0/514187963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d00cffb0 0x7f52d00d2450 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7f52c0006700 tx=0x7f52c0009b80).ready entity=mon.0 client_cookie=8ad6aca0fc66a37f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52cdb11700 1 -- 172.21.15.32:0/514187963 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52c000c0c0 con 0x7f52d00cffb0 2021-04-09T18:24:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52cdb11700 1 -- 172.21.15.32:0/514187963 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c000c440 con 0x7f52d00cffb0 2021-04-09T18:24:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52cdb11700 1 -- 172.21.15.32:0/514187963 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52c000c920 con 0x7f52d00cffb0 2021-04-09T18:24:45.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52d4e22700 1 -- 172.21.15.32:0/514187963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d00cffb0 msgr2=0x7f52d00d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.399 7f52d4e22700 1 --2- 172.21.15.32:0/514187963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d00cffb0 0x7f52d00d2450 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7f52c0006700 tx=0x7f52c0009b80).stop 2021-04-09T18:24:45.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.400 7f52d4e22700 1 -- 172.21.15.32:0/514187963 shutdown_connections 2021-04-09T18:24:45.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.400 7f52d4e22700 1 --2- 172.21.15.32:0/514187963 >> v2:172.21.15.86:3300/0 conn(0x7f52d00d6c90 0x7f52d010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.400 7f52d4e22700 1 --2- 172.21.15.32:0/514187963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d00cffb0 0x7f52d00d2450 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.400 7f52d4e22700 1 -- 172.21.15.32:0/514187963 >> 172.21.15.32:0/514187963 conn(0x7f52d00ce5d0 msgr2=0x7f52d00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:45.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.400 7f52d4e22700 1 -- 172.21.15.32:0/514187963 shutdown_connections 2021-04-09T18:24:45.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.400 7f52d4e22700 1 -- 172.21.15.32:0/514187963 wait complete. 2021-04-09T18:24:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.420 7f52d4e22700 1 Processor -- start 2021-04-09T18:24:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.420 7f52d4e22700 1 -- start start 2021-04-09T18:24:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52d4e22700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d00b8470 0x7f52d00b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52d4e22700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d018f050 0x7f52d018f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52d4e22700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0190180 0x7f52d0190580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52d4e22700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0193120 con 0x7f52d018f050 2021-04-09T18:24:45.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52d4e22700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d018fc00 con 0x7f52d0190180 2021-04-09T18:24:45.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52d4e22700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0190ba0 con 0x7f52d00b8470 2021-04-09T18:24:45.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ce312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d018f050 0x7f52d018f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ce312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d018f050 0x7f52d018f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51468/0 (socket says 172.21.15.32:51468) 2021-04-09T18:24:45.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ce312700 1 -- 172.21.15.32:0/2461489388 learned_addr learned my addr 172.21.15.32:0/2461489388 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:45.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52cf314700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0190180 0x7f52d0190580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ceb13700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d00b8470 0x7f52d00b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ce312700 1 -- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d00b8470 msgr2=0x7f52d00b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ce312700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d00b8470 0x7f52d00b8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ce312700 1 -- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0190180 msgr2=0x7f52d0190580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ce312700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0190180 0x7f52d0190580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ce312700 1 -- 172.21.15.32:0/2461489388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d01a9e60 con 0x7f52d018f050 2021-04-09T18:24:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.421 7f52ce312700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d018f050 0x7f52d018f450 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f52c4006ed0 tx=0x7f52c400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.422 7f52bf7fe700 1 -- 172.21.15.32:0/2461489388 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52c4006020 con 0x7f52d018f050 2021-04-09T18:24:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.422 7f52bf7fe700 1 -- 172.21.15.32:0/2461489388 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c4006360 con 0x7f52d018f050 2021-04-09T18:24:45.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.422 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d01a9e60 con 0x7f52d018f050 2021-04-09T18:24:45.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.422 7f52bf7fe700 1 -- 172.21.15.32:0/2461489388 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52c4005730 con 0x7f52d018f050 2021-04-09T18:24:45.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.422 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d01a9e60 con 0x7f52d018f050 2021-04-09T18:24:45.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.422 7f52bf7fe700 1 -- 172.21.15.32:0/2461489388 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f52c40054b0 con 0x7f52d018f050 2021-04-09T18:24:45.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.422 7f52bf7fe700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52b4018930 0x7f52b401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.422 7f52bf7fe700 1 -- 172.21.15.32:0/2461489388 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f52c40327e0 con 0x7f52d018f050 2021-04-09T18:24:45.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.423 7f52ceb13700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52b4018930 0x7f52b401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.423 7f52ceb13700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52b4018930 0x7f52b401adc0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:45.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.442 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d0182e10 con 0x7f52d018f050 2021-04-09T18:24:45.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.446 7f52bf7fe700 1 -- 172.21.15.32:0/2461489388 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f52c40054b0 con 0x7f52d018f050 2021-04-09T18:24:45.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.658 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f52d00d7290 con 0x7f52d018f050 2021-04-09T18:24:45.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.661 7f52bf7fe700 1 -- 172.21.15.32:0/2461489388 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f52c4005f20 con 0x7f52d018f050 2021-04-09T18:24:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52b4018930 msgr2=0x7f52b401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52b4018930 0x7f52b401adc0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d018f050 msgr2=0x7f52d018f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d018f050 0x7f52d018f450 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f52c4006ed0 tx=0x7f52c400b8e0).stop 2021-04-09T18:24:45.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 shutdown_connections 2021-04-09T18:24:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f52b4018930 0x7f52b401adc0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52d018f050 0x7f52d018f450 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0190180 0x7f52d0190580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 --2- 172.21.15.32:0/2461489388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d00b8470 0x7f52d00b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 >> 172.21.15.32:0/2461489388 conn(0x7f52d0183890 msgr2=0x7f52d019b900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:45.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 shutdown_connections 2021-04-09T18:24:45.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.664 7f52d4e22700 1 -- 172.21.15.32:0/2461489388 wait complete. 2021-04-09T18:24:45.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_thrash' 2021-04-09T18:24:45.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.822 7f8ca9fda700 1 Processor -- start 2021-04-09T18:24:45.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.822 7f8ca9fda700 1 -- start start 2021-04-09T18:24:45.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.823 7f8ca9fda700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8ca410da00 0x7f8ca410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.823 7f8ca9fda700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca410d480 con 0x7f8ca410d010 2021-04-09T18:24:45.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.823 7f8ca9fda700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca40d1f70 con 0x7f8ca40d7ca0 2021-04-09T18:24:45.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.823 7f8ca9fda700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca40d8d30 con 0x7f8ca410da00 2021-04-09T18:24:45.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.823 7f8ca3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8ca410da00 0x7f8ca410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.823 7f8ca37fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8ca410d010 0x7f8ca40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:32834/0 (socket says 172.21.15.32:32834) 2021-04-09T18:24:45.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.823 7f8ca37fe700 1 -- 172.21.15.32:0/2124873061 learned_addr learned my addr 172.21.15.32:0/2124873061 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:45.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.823 7f8ca3fff700 1 -- 172.21.15.32:0/2124873061 >> v1:172.21.15.86:6789/0 conn(0x7f8ca40d7ca0 legacy=0x7f8ca40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.824 7f8ca3fff700 1 -- 172.21.15.32:0/2124873061 >> v1:172.21.15.32:6789/0 conn(0x7f8ca410d010 legacy=0x7f8ca40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.824 7f8ca3fff700 1 -- 172.21.15.32:0/2124873061 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40d9220 con 0x7f8ca410da00 2021-04-09T18:24:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.824 7f8ca27fc700 1 -- 172.21.15.32:0/2124873061 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151109803 0 0) 0x7f8c94000ce0 con 0x7f8ca410d010 2021-04-09T18:24:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.824 7f8ca3fff700 1 --2- 172.21.15.32:0/2124873061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca410da00 0x7f8ca410de00 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f8c9c006e30 tx=0x7f8c9c009de0).ready entity=mon.0 client_cookie=690463749a2b1124 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:45.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.824 7f8ca27fc700 1 -- 172.21.15.32:0/2124873061 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c9c00b300 con 0x7f8ca410da00 2021-04-09T18:24:45.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.824 7f8ca27fc700 1 -- 172.21.15.32:0/2124873061 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c00b680 con 0x7f8ca410da00 2021-04-09T18:24:45.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.824 7f8ca27fc700 1 -- 172.21.15.32:0/2124873061 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c9c00bb60 con 0x7f8ca410da00 2021-04-09T18:24:45.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.824 7f8ca9fda700 1 -- 172.21.15.32:0/2124873061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca410da00 msgr2=0x7f8ca410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.824 7f8ca9fda700 1 --2- 172.21.15.32:0/2124873061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca410da00 0x7f8ca410de00 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f8c9c006e30 tx=0x7f8c9c009de0).stop 2021-04-09T18:24:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.825 7f8ca9fda700 1 -- 172.21.15.32:0/2124873061 shutdown_connections 2021-04-09T18:24:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.825 7f8ca9fda700 1 --2- 172.21.15.32:0/2124873061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca410da00 0x7f8ca410de00 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.825 7f8ca9fda700 1 -- 172.21.15.32:0/2124873061 >> 172.21.15.32:0/2124873061 conn(0x7f8ca40ce5f0 msgr2=0x7f8ca40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.825 7f8ca9fda700 1 -- 172.21.15.32:0/2124873061 shutdown_connections 2021-04-09T18:24:45.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.825 7f8ca9fda700 1 -- 172.21.15.32:0/2124873061 wait complete. 2021-04-09T18:24:45.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.845 7f8ca9fda700 1 Processor -- start 2021-04-09T18:24:45.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.845 7f8ca9fda700 1 -- start start 2021-04-09T18:24:45.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.845 7f8ca9fda700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ca41a9a50 0x7f8ca408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.845 7f8ca9fda700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ca409d7d0 0x7f8ca409dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.845 7f8ca9fda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4183c20 0x7f8ca4184020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.845 7f8ca9fda700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca409e1f0 con 0x7f8ca4183c20 2021-04-09T18:24:45.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.845 7f8ca9fda700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca408ea60 con 0x7f8ca41a9a50 2021-04-09T18:24:45.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.845 7f8ca9fda700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca408f540 con 0x7f8ca409d7d0 2021-04-09T18:24:45.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4183c20 0x7f8ca4184020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4183c20 0x7f8ca4184020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51484/0 (socket says 172.21.15.32:51484) 2021-04-09T18:24:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca3fff700 1 -- 172.21.15.32:0/792938693 learned_addr learned my addr 172.21.15.32:0/792938693 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca37fe700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ca41a9a50 0x7f8ca408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca2ffd700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ca409d7d0 0x7f8ca409dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca3fff700 1 -- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ca409d7d0 msgr2=0x7f8ca409dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca3fff700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ca409d7d0 0x7f8ca409dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca3fff700 1 -- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ca41a9a50 msgr2=0x7f8ca408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:45.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca3fff700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ca41a9a50 0x7f8ca408ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:45.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca3fff700 1 -- 172.21.15.32:0/792938693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca4184640 con 0x7f8ca4183c20 2021-04-09T18:24:45.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca2ffd700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ca409d7d0 0x7f8ca409dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:45.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca3fff700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4183c20 0x7f8ca4184020 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7f8c9c009fc0 tx=0x7f8c9c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:45.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8c8bfff700 1 -- 172.21.15.32:0/792938693 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c9c0093d0 con 0x7f8ca4183c20 2021-04-09T18:24:45.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8c8bfff700 1 -- 172.21.15.32:0/792938693 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c003030 con 0x7f8ca4183c20 2021-04-09T18:24:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8c8bfff700 1 -- 172.21.15.32:0/792938693 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c9c00c4e0 con 0x7f8ca4183c20 2021-04-09T18:24:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca4184640 con 0x7f8ca4183c20 2021-04-09T18:24:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.846 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca4184640 con 0x7f8ca4183c20 2021-04-09T18:24:45.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.847 7f8c8bfff700 1 -- 172.21.15.32:0/792938693 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8c9c001270 con 0x7f8ca4183c20 2021-04-09T18:24:45.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.847 7f8c8bfff700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c84018930 0x7f8c8401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:45.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.847 7f8c8bfff700 1 -- 172.21.15.32:0/792938693 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f8c9c032b80 con 0x7f8ca4183c20 2021-04-09T18:24:45.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.847 7f8ca37fe700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c84018930 0x7f8c8401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:45.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.847 7f8ca37fe700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c84018930 0x7f8c8401adc0 crc :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:45.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.867 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca410d480 con 0x7f8ca4183c20 2021-04-09T18:24:45.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:45.871 7f8c8bfff700 1 -- 172.21.15.32:0/792938693 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8c9c009720 con 0x7f8ca4183c20 2021-04-09T18:24:46.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.056 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_thrash"]} v 0) v1 -- 0x7f8ca410d480 con 0x7f8ca4183c20 2021-04-09T18:24:46.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.187 7f8c8bfff700 1 -- 172.21.15.32:0/792938693 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_thrash"]}]=0 v574) v1 ==== 135+0+0 (secure 0 0 0) 0x7f8c9c001270 con 0x7f8ca4183c20 2021-04-09T18:24:46.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c84018930 msgr2=0x7f8c8401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:46.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c84018930 0x7f8c8401adc0 crc :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4183c20 msgr2=0x7f8ca4184020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:46.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4183c20 0x7f8ca4184020 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7f8c9c009fc0 tx=0x7f8c9c00bd40).stop 2021-04-09T18:24:46.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 shutdown_connections 2021-04-09T18:24:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c84018930 0x7f8c8401adc0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4183c20 0x7f8ca4184020 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ca409d7d0 0x7f8ca409dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 --2- 172.21.15.32:0/792938693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ca41a9a50 0x7f8ca408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 >> 172.21.15.32:0/792938693 conn(0x7f8ca41965c0 msgr2=0x7f8ca41ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 shutdown_connections 2021-04-09T18:24:46.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.222 7f8ca9fda700 1 -- 172.21.15.32:0/792938693 wait complete. 2021-04-09T18:24:46.296 INFO:tasks.cephfs_test_runner:test_thrash (tasks.cephfs.test_failover.TestClusterResize) ... ok 2021-04-09T18:24:46.297 INFO:tasks.cephfs_test_runner:Starting test: test_up_less_than_max (tasks.cephfs.test_failover.TestClusterResize) 2021-04-09T18:24:46.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_up_less_than_max' 2021-04-09T18:24:46.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.382 7fc2eda42700 1 Processor -- start 2021-04-09T18:24:46.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.382 7fc2eda42700 1 -- start start 2021-04-09T18:24:46.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2eda42700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc2e80d7ca0 0x7fc2e80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2eda42700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e810da00 0x7fc2e810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2eda42700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e810d480 con 0x7fc2e810d010 2021-04-09T18:24:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2eda42700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e80d9220 con 0x7fc2e80d7ca0 2021-04-09T18:24:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2eda42700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80d86c0 con 0x7fc2e810da00 2021-04-09T18:24:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e6ffd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc2e80d7ca0 0x7fc2e80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:46.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e7fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e810da00 0x7fc2e810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:46.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e77fe700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc2e810d010 0x7fc2e80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:59006/0 (socket says 172.21.15.32:59006) 2021-04-09T18:24:46.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e77fe700 1 -- 172.21.15.32:0/2833522781 learned_addr learned my addr 172.21.15.32:0/2833522781 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:46.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e67fc700 1 -- 172.21.15.32:0/2833522781 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007637761 0 0) 0x7fc2d80010c0 con 0x7fc2e810d010 2021-04-09T18:24:46.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e7fff700 1 -- 172.21.15.32:0/2833522781 >> v2:172.21.15.86:3300/0 conn(0x7fc2e80d7ca0 msgr2=0x7fc2e80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:46.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e7fff700 1 --2- 172.21.15.32:0/2833522781 >> v2:172.21.15.86:3300/0 conn(0x7fc2e80d7ca0 0x7fc2e80d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e7fff700 1 -- 172.21.15.32:0/2833522781 >> v1:172.21.15.86:6789/0 conn(0x7fc2e810d010 legacy=0x7fc2e80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:46.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e7fff700 1 -- 172.21.15.32:0/2833522781 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80d1f70 con 0x7fc2e810da00 2021-04-09T18:24:46.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.383 7fc2e6ffd700 1 --2- 172.21.15.32:0/2833522781 >> v2:172.21.15.86:3300/0 conn(0x7fc2e80d7ca0 0x7fc2e80d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:46.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2e7fff700 1 --2- 172.21.15.32:0/2833522781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e810da00 0x7fc2e810de00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc2e0006ad0 tx=0x7fc2e0009ad0).ready entity=mon.2 client_cookie=f948327680b50b97 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:46.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2e67fc700 1 -- 172.21.15.32:0/2833522781 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e00066a0 con 0x7fc2e810da00 2021-04-09T18:24:46.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2e67fc700 1 -- 172.21.15.32:0/2833522781 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e0006410 con 0x7fc2e810da00 2021-04-09T18:24:46.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2e67fc700 1 -- 172.21.15.32:0/2833522781 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e00024d0 con 0x7fc2e810da00 2021-04-09T18:24:46.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2eda42700 1 -- 172.21.15.32:0/2833522781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e810da00 msgr2=0x7fc2e810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:46.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2eda42700 1 --2- 172.21.15.32:0/2833522781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e810da00 0x7fc2e810de00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc2e0006ad0 tx=0x7fc2e0009ad0).stop 2021-04-09T18:24:46.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2eda42700 1 -- 172.21.15.32:0/2833522781 shutdown_connections 2021-04-09T18:24:46.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2eda42700 1 --2- 172.21.15.32:0/2833522781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e810da00 0x7fc2e810de00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2eda42700 1 --2- 172.21.15.32:0/2833522781 >> v2:172.21.15.86:3300/0 conn(0x7fc2e80d7ca0 0x7fc2e80d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.384 7fc2eda42700 1 -- 172.21.15.32:0/2833522781 >> 172.21.15.32:0/2833522781 conn(0x7fc2e80ce5f0 msgr2=0x7fc2e80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:46.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.385 7fc2eda42700 1 -- 172.21.15.32:0/2833522781 shutdown_connections 2021-04-09T18:24:46.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.385 7fc2eda42700 1 -- 172.21.15.32:0/2833522781 wait complete. 2021-04-09T18:24:46.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.405 7fc2eda42700 1 Processor -- start 2021-04-09T18:24:46.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.405 7fc2eda42700 1 -- start start 2021-04-09T18:24:46.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2eda42700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e81aca20 0x7fc2e81ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:46.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2eda42700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e818bd40 0x7fc2e818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:46.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2eda42700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e808fba0 0x7fc2e808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:46.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2eda42700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e818c7e0 con 0x7fc2e808fba0 2021-04-09T18:24:46.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2eda42700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e818ca30 con 0x7fc2e818bd40 2021-04-09T18:24:46.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2eda42700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80905f0 con 0x7fc2e81aca20 2021-04-09T18:24:46.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e808fba0 0x7fc2e808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:46.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e808fba0 0x7fc2e808ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51500/0 (socket says 172.21.15.32:51500) 2021-04-09T18:24:46.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e7fff700 1 -- 172.21.15.32:0/2804607345 learned_addr learned my addr 172.21.15.32:0/2804607345 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:46.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e6ffd700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e818bd40 0x7fc2e818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:46.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e77fe700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e81aca20 0x7fc2e81ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:46.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e7fff700 1 -- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e81aca20 msgr2=0x7fc2e81ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:46.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e7fff700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e81aca20 0x7fc2e81ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e7fff700 1 -- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e818bd40 msgr2=0x7fc2e818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:46.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e7fff700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e818bd40 0x7fc2e818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:46.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e7fff700 1 -- 172.21.15.32:0/2804607345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80b4300 con 0x7fc2e808fba0 2021-04-09T18:24:46.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e77fe700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e81aca20 0x7fc2e81ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:46.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e6ffd700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e818bd40 0x7fc2e818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:46.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2e7fff700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e808fba0 0x7fc2e808ffa0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7fc2e0009670 tx=0x7fc2e000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:46.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2cffff700 1 -- 172.21.15.32:0/2804607345 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e0009020 con 0x7fc2e808fba0 2021-04-09T18:24:46.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2cffff700 1 -- 172.21.15.32:0/2804607345 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e000bd80 con 0x7fc2e808fba0 2021-04-09T18:24:46.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e80b4300 con 0x7fc2e808fba0 2021-04-09T18:24:46.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2cffff700 1 -- 172.21.15.32:0/2804607345 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e0002080 con 0x7fc2e808fba0 2021-04-09T18:24:46.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.406 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e80b4300 con 0x7fc2e808fba0 2021-04-09T18:24:46.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.408 7fc2cffff700 1 -- 172.21.15.32:0/2804607345 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc2e0002020 con 0x7fc2e808fba0 2021-04-09T18:24:46.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.408 7fc2cffff700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c80188c0 0x7fc2c801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:46.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.408 7fc2cffff700 1 -- 172.21.15.32:0/2804607345 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fc2e0032510 con 0x7fc2e808fba0 2021-04-09T18:24:46.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.408 7fc2e77fe700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c80188c0 0x7fc2c801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:46.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.408 7fc2e77fe700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c80188c0 0x7fc2c801adc0 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:46.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.427 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e8182d70 con 0x7fc2e808fba0 2021-04-09T18:24:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.430 7fc2cffff700 1 -- 172.21.15.32:0/2804607345 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc2e00093e0 con 0x7fc2e808fba0 2021-04-09T18:24:46.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:46.618 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_up_less_than_max"]} v 0) v1 -- 0x7fc2e818c760 con 0x7fc2e808fba0 2021-04-09T18:24:47.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.188 7fc2cffff700 1 -- 172.21.15.32:0/2804607345 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_up_less_than_max"]}]=0 v575) v1 ==== 148+0+0 (secure 0 0 0) 0x7fc2e0002020 con 0x7fc2e808fba0 2021-04-09T18:24:47.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.236 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c80188c0 msgr2=0x7fc2c801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:47.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.236 7fc2eda42700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c80188c0 0x7fc2c801adc0 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.236 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e808fba0 msgr2=0x7fc2e808ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:47.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.236 7fc2eda42700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e808fba0 0x7fc2e808ffa0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7fc2e0009670 tx=0x7fc2e000b990).stop 2021-04-09T18:24:47.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.237 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 shutdown_connections 2021-04-09T18:24:47.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.237 7fc2eda42700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc2c80188c0 0x7fc2c801adc0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.237 7fc2eda42700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2e808fba0 0x7fc2e808ffa0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.237 7fc2eda42700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2e818bd40 0x7fc2e818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.237 7fc2eda42700 1 --2- 172.21.15.32:0/2804607345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2e81aca20 0x7fc2e81ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.237 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 >> 172.21.15.32:0/2804607345 conn(0x7fc2e819b460 msgr2=0x7fc2e819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:47.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.237 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 shutdown_connections 2021-04-09T18:24:47.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.237 7fc2eda42700 1 -- 172.21.15.32:0/2804607345 wait complete. 2021-04-09T18:24:47.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:24:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.419 7f99f4e88700 1 Processor -- start 2021-04-09T18:24:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.419 7f99f4e88700 1 -- start start 2021-04-09T18:24:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.419 7f99f4e88700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f010d010 0x7f99f00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.419 7f99f4e88700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f010d480 con 0x7f99f00d1da0 2021-04-09T18:24:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.419 7f99f4e88700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f010e1f0 con 0x7f99f00d8490 2021-04-09T18:24:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.419 7f99f4e88700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f010e780 con 0x7f99f010d010 2021-04-09T18:24:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99ee312700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f99f00d1da0 0x7f99f00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:32868/0 (socket says 172.21.15.32:32868) 2021-04-09T18:24:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99ee312700 1 -- 172.21.15.32:0/1324244988 learned_addr learned my addr 172.21.15.32:0/1324244988 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99eeb13700 1 --2- 172.21.15.32:0/1324244988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f010d010 0x7f99f00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99edb11700 1 -- 172.21.15.32:0/1324244988 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910185108 0 0) 0x7f99e80009b0 con 0x7f99f00d8490 2021-04-09T18:24:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99edb11700 1 -- 172.21.15.32:0/1324244988 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d8002fe0 con 0x7f99f00d8490 2021-04-09T18:24:47.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99edb11700 1 -- 172.21.15.32:0/1324244988 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090856476 0 0) 0x7f99e4001060 con 0x7f99f00d1da0 2021-04-09T18:24:47.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99edb11700 1 -- 172.21.15.32:0/1324244988 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d8002fe0 con 0x7f99f00d1da0 2021-04-09T18:24:47.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99eeb13700 1 -- 172.21.15.32:0/1324244988 >> v1:172.21.15.86:6789/0 conn(0x7f99f00d8490 legacy=0x7f99f010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:47.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99eeb13700 1 -- 172.21.15.32:0/1324244988 >> v1:172.21.15.32:6789/0 conn(0x7f99f00d1da0 legacy=0x7f99f00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:47.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99eeb13700 1 -- 172.21.15.32:0/1324244988 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00d8d30 con 0x7f99f010d010 2021-04-09T18:24:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99eeb13700 1 --2- 172.21.15.32:0/1324244988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f010d010 0x7f99f00d1780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f99e0006650 tx=0x7f99e0009a80).ready entity=mon.2 client_cookie=4697c265905970d9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99edb11700 1 -- 172.21.15.32:0/1324244988 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99e00009b0 con 0x7f99f010d010 2021-04-09T18:24:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99edb11700 1 -- 172.21.15.32:0/1324244988 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e000c0c0 con 0x7f99f010d010 2021-04-09T18:24:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99edb11700 1 -- 172.21.15.32:0/1324244988 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99e000c510 con 0x7f99f010d010 2021-04-09T18:24:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99f4e88700 1 -- 172.21.15.32:0/1324244988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f010d010 msgr2=0x7f99f00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.420 7f99f4e88700 1 --2- 172.21.15.32:0/1324244988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f010d010 0x7f99f00d1780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f99e0006650 tx=0x7f99e0009a80).stop 2021-04-09T18:24:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.421 7f99f4e88700 1 -- 172.21.15.32:0/1324244988 shutdown_connections 2021-04-09T18:24:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.421 7f99f4e88700 1 --2- 172.21.15.32:0/1324244988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f010d010 0x7f99f00d1780 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.421 7f99f4e88700 1 -- 172.21.15.32:0/1324244988 >> 172.21.15.32:0/1324244988 conn(0x7f99f00ce5f0 msgr2=0x7f99f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:47.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.421 7f99f4e88700 1 -- 172.21.15.32:0/1324244988 shutdown_connections 2021-04-09T18:24:47.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.421 7f99f4e88700 1 -- 172.21.15.32:0/1324244988 wait complete. 2021-04-09T18:24:47.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.441 7f99f4e88700 1 Processor -- start 2021-04-09T18:24:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.441 7f99f4e88700 1 -- start start 2021-04-09T18:24:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99f4e88700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f0179610 0x7f99f0179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99f4e88700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f017a820 0x7f99f017ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99f4e88700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f01949f0 0x7f99f0194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99f4e88700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f01b8d50 con 0x7f99f017a820 2021-04-09T18:24:47.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99f4e88700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f017b240 con 0x7f99f0179610 2021-04-09T18:24:47.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99f4e88700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0195410 con 0x7f99f01949f0 2021-04-09T18:24:47.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ee312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f017a820 0x7f99f017ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:47.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ee312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f017a820 0x7f99f017ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51512/0 (socket says 172.21.15.32:51512) 2021-04-09T18:24:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ee312700 1 -- 172.21.15.32:0/1097878179 learned_addr learned my addr 172.21.15.32:0/1097878179 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99eeb13700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f0179610 0x7f99f0179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ef314700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f01949f0 0x7f99f0194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ee312700 1 -- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f01949f0 msgr2=0x7f99f0194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ee312700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f01949f0 0x7f99f0194df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ee312700 1 -- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f0179610 msgr2=0x7f99f0179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ee312700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f0179610 0x7f99f0179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ee312700 1 -- 172.21.15.32:0/1097878179 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f0195630 con 0x7f99f017a820 2021-04-09T18:24:47.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.442 7f99ee312700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f017a820 0x7f99f017ac20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7f99e4009960 tx=0x7f99e400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:47.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.443 7f99df7fe700 1 -- 172.21.15.32:0/1097878179 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99e40062b0 con 0x7f99f017a820 2021-04-09T18:24:47.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.443 7f99df7fe700 1 -- 172.21.15.32:0/1097878179 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e4004720 con 0x7f99f017a820 2021-04-09T18:24:47.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.443 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f0195630 con 0x7f99f017a820 2021-04-09T18:24:47.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.443 7f99df7fe700 1 -- 172.21.15.32:0/1097878179 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99e4004b70 con 0x7f99f017a820 2021-04-09T18:24:47.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.443 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f0195630 con 0x7f99f017a820 2021-04-09T18:24:47.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.443 7f99df7fe700 1 -- 172.21.15.32:0/1097878179 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f99e4006570 con 0x7f99f017a820 2021-04-09T18:24:47.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.443 7f99df7fe700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99d4018930 0x7f99d401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:47.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.443 7f99df7fe700 1 -- 172.21.15.32:0/1097878179 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f99e4032b20 con 0x7f99f017a820 2021-04-09T18:24:47.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.444 7f99eeb13700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99d4018930 0x7f99d401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:47.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.444 7f99eeb13700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99d4018930 0x7f99d401adc0 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:47.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.463 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f017a2f0 con 0x7f99f017a820 2021-04-09T18:24:47.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.467 7f99df7fe700 1 -- 172.21.15.32:0/1097878179 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f99e400be30 con 0x7f99f017a820 2021-04-09T18:24:47.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.681 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f99f017a2f0 con 0x7f99f017a820 2021-04-09T18:24:47.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.684 7f99df7fe700 1 -- 172.21.15.32:0/1097878179 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f99e4004720 con 0x7f99f017a820 2021-04-09T18:24:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99d4018930 msgr2=0x7f99d401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99d4018930 0x7f99d401adc0 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f017a820 msgr2=0x7f99f017ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f017a820 0x7f99f017ac20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7f99e4009960 tx=0x7f99e400bcf0).stop 2021-04-09T18:24:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 shutdown_connections 2021-04-09T18:24:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f99d4018930 0x7f99d401adc0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f017a820 0x7f99f017ac20 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99f01949f0 0x7f99f0194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 --2- 172.21.15.32:0/1097878179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99f0179610 0x7f99f0179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:47.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 >> 172.21.15.32:0/1097878179 conn(0x7f99f0092a00 msgr2=0x7f99f01b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:47.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 shutdown_connections 2021-04-09T18:24:47.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:47.691 7f99f4e88700 1 -- 172.21.15.32:0/1097878179 wait complete. 2021-04-09T18:24:47.765 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:24:47.765 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:24:47.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:24:47.881 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:24:47.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:24:47.882 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:24:47.882 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:24:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:24:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:24:47.926 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:24:47.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:24:47.927 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:24:47.928 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:24:47.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:24:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:24:47.971 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:24:47.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:24:48.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.057 7ff6ec242700 1 Processor -- start 2021-04-09T18:24:48.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.057 7ff6ec242700 1 -- start start 2021-04-09T18:24:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ec242700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff6e410d010 0x7ff6e40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ec242700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40d8490 0x7ff6e410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ec242700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e410d480 con 0x7ff6e40d1da0 2021-04-09T18:24:48.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ec242700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e40d8d30 con 0x7ff6e410d010 2021-04-09T18:24:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ec242700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40d9220 con 0x7ff6e40d8490 2021-04-09T18:24:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ea05e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff6e410d010 0x7ff6e40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6e985d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff6e40d1da0 0x7ff6e40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:32882/0 (socket says 172.21.15.32:32882) 2021-04-09T18:24:48.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6e985d700 1 -- 172.21.15.32:0/1001874286 learned_addr learned my addr 172.21.15.32:0/1001874286 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:48.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ea85f700 1 --2- 172.21.15.32:0/1001874286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40d8490 0x7ff6e410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6e905c700 1 -- 172.21.15.32:0/1001874286 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426522960 0 0) 0x7ff6d4000ce0 con 0x7ff6e40d1da0 2021-04-09T18:24:48.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ea05e700 1 -- 172.21.15.32:0/1001874286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40d8490 msgr2=0x7ff6e410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:48.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ea05e700 1 --2- 172.21.15.32:0/1001874286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40d8490 0x7ff6e410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ea05e700 1 -- 172.21.15.32:0/1001874286 >> v1:172.21.15.32:6789/0 conn(0x7ff6e40d1da0 legacy=0x7ff6e40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ea05e700 1 -- 172.21.15.32:0/1001874286 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e410e020 con 0x7ff6e410d010 2021-04-09T18:24:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ea05e700 1 --2- 172.21.15.32:0/1001874286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e410d010 0x7ff6e40d1780 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7ff6e0006650 tx=0x7ff6e0009a80).ready entity=mon.0 client_cookie=e888e780460d2ba8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6e905c700 1 -- 172.21.15.32:0/1001874286 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6e00009b0 con 0x7ff6e410d010 2021-04-09T18:24:48.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6e905c700 1 -- 172.21.15.32:0/1001874286 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e000c0c0 con 0x7ff6e410d010 2021-04-09T18:24:48.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6e905c700 1 -- 172.21.15.32:0/1001874286 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6e000c510 con 0x7ff6e410d010 2021-04-09T18:24:48.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ec242700 1 -- 172.21.15.32:0/1001874286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e410d010 msgr2=0x7ff6e40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.058 7ff6ec242700 1 --2- 172.21.15.32:0/1001874286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e410d010 0x7ff6e40d1780 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7ff6e0006650 tx=0x7ff6e0009a80).stop 2021-04-09T18:24:48.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.059 7ff6ec242700 1 -- 172.21.15.32:0/1001874286 shutdown_connections 2021-04-09T18:24:48.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.059 7ff6ec242700 1 --2- 172.21.15.32:0/1001874286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40d8490 0x7ff6e410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.059 7ff6ec242700 1 --2- 172.21.15.32:0/1001874286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e410d010 0x7ff6e40d1780 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.059 7ff6ec242700 1 -- 172.21.15.32:0/1001874286 >> 172.21.15.32:0/1001874286 conn(0x7ff6e40ce5f0 msgr2=0x7ff6e40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:48.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.059 7ff6ec242700 1 -- 172.21.15.32:0/1001874286 shutdown_connections 2021-04-09T18:24:48.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.059 7ff6ec242700 1 -- 172.21.15.32:0/1001874286 wait complete. 2021-04-09T18:24:48.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ec242700 1 Processor -- start 2021-04-09T18:24:48.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ec242700 1 -- start start 2021-04-09T18:24:48.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ec242700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e409a5e0 0x7ff6e41884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ec242700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e40a3350 0x7ff6e40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ec242700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40a15b0 0x7ff6e40a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ec242700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40a40d0 con 0x7ff6e409a5e0 2021-04-09T18:24:48.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ec242700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40a2dd0 con 0x7ff6e40a3350 2021-04-09T18:24:48.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ec242700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4188ad0 con 0x7ff6e40a15b0 2021-04-09T18:24:48.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ea05e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e409a5e0 0x7ff6e41884b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ea05e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e409a5e0 0x7ff6e41884b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51526/0 (socket says 172.21.15.32:51526) 2021-04-09T18:24:48.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.080 7ff6ea05e700 1 -- 172.21.15.32:0/821866934 learned_addr learned my addr 172.21.15.32:0/821866934 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:48.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6e985d700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e40a3350 0x7ff6e40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ea85f700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40a15b0 0x7ff6e40a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ea05e700 1 -- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40a15b0 msgr2=0x7ff6e40a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ea05e700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40a15b0 0x7ff6e40a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ea05e700 1 -- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e40a3350 msgr2=0x7ff6e40a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ea05e700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e40a3350 0x7ff6e40a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ea05e700 1 -- 172.21.15.32:0/821866934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e40a3d70 con 0x7ff6e409a5e0 2021-04-09T18:24:48.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ea85f700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40a15b0 0x7ff6e40a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:48.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ea05e700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e409a5e0 0x7ff6e41884b0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7ff6e0009650 tx=0x7ff6e000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:48.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6daffd700 1 -- 172.21.15.32:0/821866934 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6e00093a0 con 0x7ff6e409a5e0 2021-04-09T18:24:48.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6daffd700 1 -- 172.21.15.32:0/821866934 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e000bd50 con 0x7ff6e409a5e0 2021-04-09T18:24:48.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ec242700 1 -- 172.21.15.32:0/821866934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e40a3d70 con 0x7ff6e409a5e0 2021-04-09T18:24:48.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6daffd700 1 -- 172.21.15.32:0/821866934 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6e00052e0 con 0x7ff6e409a5e0 2021-04-09T18:24:48.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.081 7ff6ec242700 1 -- 172.21.15.32:0/821866934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e40a3d70 con 0x7ff6e409a5e0 2021-04-09T18:24:48.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.082 7ff6daffd700 1 -- 172.21.15.32:0/821866934 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff6e0006030 con 0x7ff6e409a5e0 2021-04-09T18:24:48.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.082 7ff6daffd700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6cc018930 0x7ff6cc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.082 7ff6daffd700 1 -- 172.21.15.32:0/821866934 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4929+0+0 (secure 0 0 0) 0x7ff6e0032760 con 0x7ff6e409a5e0 2021-04-09T18:24:48.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.082 7ff6e985d700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6cc018930 0x7ff6cc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.082 7ff6e985d700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6cc018930 0x7ff6cc01ae10 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:48.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.101 7ff6ec242700 1 -- 172.21.15.32:0/821866934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e410d480 con 0x7ff6e409a5e0 2021-04-09T18:24:48.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.105 7ff6daffd700 1 -- 172.21.15.32:0/821866934 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff6e00009c0 con 0x7ff6e409a5e0 2021-04-09T18:24:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.293 7ff6ec242700 1 -- 172.21.15.32:0/821866934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff6e410d480 con 0x7ff6e409a5e0 2021-04-09T18:24:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.294 7ff6daffd700 1 -- 172.21.15.32:0/821866934 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v110) v1 ==== 81+0+25689 (secure 0 0 0) 0x7ff6e0006030 con 0x7ff6e409a5e0 2021-04-09T18:24:48.358 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:48.359 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:24:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 110, 2021-04-09T18:24:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:24:48.359 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:24:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:24:45.181100", 2021-04-09T18:24:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:24:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:24:48.360 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:24:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:24:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:24:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:24:48.361 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:24:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:24:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:24:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:24:48.362 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:24:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:24:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:24:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:24:48.363 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:24:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 19, 2021-04-09T18:24:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:24:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:24:48.364 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:24:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:24:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:24:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.365 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:24:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:24:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:24:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:24:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:24:48.366 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:24:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:24:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:24:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:24:48.367 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:24:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:24:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:24:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:24:48.368 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:24:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:24:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:24:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:24:48.369 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:24:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:24:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:24:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:24:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:24:48.370 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:24:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:24:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:24:48.371 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:24:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:24:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:24:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:24:48.372 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:24:48.373 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:24:48.373 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:24:48.373 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:24:48.373 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:24:48.373 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:24:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:24:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:24:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:24:48.374 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:24:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:24:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:24:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:24:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:24:48.375 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:24:48.376 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:24:48.376 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:24:48.376 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:24:48.376 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:24:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:24:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:24:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.377 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:24:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:24:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:24:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:24:48.378 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:24:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:24:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:24:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:24:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:24:48.379 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:24:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:24:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:24:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:24:48.380 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:24:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:24:48.381 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:24:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:24:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:24:48.382 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:48.383 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 100, 2021-04-09T18:24:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:48.384 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.385 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:24:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:24:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.387 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:24:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:24:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:48.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:24:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:48.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:24:48.393 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:48.394 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:24:48.395 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:24:48.396 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:24:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.397 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:24:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:24:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:24:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:24:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:48.398 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:24:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:24:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:48.400 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:48.401 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:48.402 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:24:48.403 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:24:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.405 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.406 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:24:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.407 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:24:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:24:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:24:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.411 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.412 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:24:48.413 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:24:48.414 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:24:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.415 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:24:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:24:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:24:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:24:48.416 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:24:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:24:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:48.418 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:48.419 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:48.420 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.421 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:24:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.422 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:24:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.423 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:24:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:24:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.426 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:24:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:48.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:24:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:48.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:48.430 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:24:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:48.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:24:48.432 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:24:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:24:48.433 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:24:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:24:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:24:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:48.434 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:24:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:24:48.436 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:48.437 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:48.438 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.439 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:24:48.440 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:48.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.440 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.441 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:24:48.441 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:48.441 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.441 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.442 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:48.442 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.442 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:24:48.443 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:48.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.443 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.443 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.444 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:24:48.444 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:48.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.444 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.445 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:48.445 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.446 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.446 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:24:48.446 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:48.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.447 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:24:48.447 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:48.447 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.448 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.448 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:48.448 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.449 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.449 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:24:48.449 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:48.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.450 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.450 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:24:48.450 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:24:48.451 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.451 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.451 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:24:48.452 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:24:48.452 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:24:48.452 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:24:48.452 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:48.452 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:48.453 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:48.453 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.454 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:24:48.454 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:24:48.454 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:48.454 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:48.454 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:48.455 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:48.455 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:48.455 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:48.455 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:48.456 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:48.456 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:48.456 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:48.456 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:48.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:24:48.457 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:48.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.458 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.458 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:24:48.459 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:48.459 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.459 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.460 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:48.460 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.460 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.460 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:24:48.461 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:48.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.461 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.462 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:24:48.462 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:48.462 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.462 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.463 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.464 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:24:48.464 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:48.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.465 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:24:48.465 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:48.465 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.466 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.466 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:48.466 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.466 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.467 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.467 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:24:48.467 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:48.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.468 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:24:48.468 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:24:48.469 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.469 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.469 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:24:48.469 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:24:48.470 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:24:48.470 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:24:48.470 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:48.470 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:48.471 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:48.471 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.472 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:24:48.472 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:24:48.472 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:48.472 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:48.472 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:48.473 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:48.473 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:48.473 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:48.473 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:48.474 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:48.474 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:48.474 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:48.474 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:48.474 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.475 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:24:48.475 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:48.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.476 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.476 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:24:48.476 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:48.477 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.477 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.477 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:48.477 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.478 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.478 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:24:48.478 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:48.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.479 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:24:48.480 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:48.480 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.480 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.481 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:48.481 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.481 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:24:48.482 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:48.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.482 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.483 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:24:48.483 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:48.483 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.483 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.484 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:48.484 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.484 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.485 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:24:48.485 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:48.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.485 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.486 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:24:48.486 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:24:48.486 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.486 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.487 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:24:48.487 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:24:48.487 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:24:48.488 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:24:48.488 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:48.488 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:48.488 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:48.488 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.489 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:24:48.489 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:24:48.490 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:48.490 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:48.490 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:48.490 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:48.490 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:48.491 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:48.491 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:48.491 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 90, 2021-04-09T18:24:48.491 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:48.492 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:48.492 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:48.492 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.493 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:24:48.493 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:48.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.494 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.494 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:24:48.494 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:48.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.495 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.495 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:48.495 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.496 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.496 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:24:48.496 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:48.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.497 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:24:48.497 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:48.498 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.498 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.498 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:48.498 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.499 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:24:48.500 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:48.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.500 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.500 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:24:48.501 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:48.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.501 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.502 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:48.502 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:24:48.503 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:48.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.503 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.504 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:24:48.504 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:24:48.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.504 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.505 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:24:48.505 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:24:48.505 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:24:48.506 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:24:48.506 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:48.506 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:48.506 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:48.507 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.507 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:24:48.508 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:24:48.508 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:24:48.508 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:24:48.508 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:24:48.508 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:24:48.509 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:24:48.509 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:24:48.509 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:24:48.509 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:24:48.510 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:24:48.510 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:24:48.510 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:24:48.510 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.511 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:24:48.511 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:48.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.512 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:24:48.513 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:48.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.513 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.513 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:24:48.514 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.514 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.514 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.514 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:24:48.515 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:48.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.515 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.516 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:24:48.516 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:48.516 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.516 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.517 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:24:48.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:24:48.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:48.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.519 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:24:48.519 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:48.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.520 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.520 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:24:48.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:24:48.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.521 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:24:48.521 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:24:48.521 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:48.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.522 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:24:48.522 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:24:48.522 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:24:48.522 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.523 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.523 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:24:48.523 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:24:48.524 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:24:48.524 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:24:48.524 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:24:48.524 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:24:48.524 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:24:48.525 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:48.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.525 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:24:48.525 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:24:48.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.526 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:24:48.526 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:48.526 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:48.527 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:48.527 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:48.527 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:48.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.528 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:24:48.528 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:48.528 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:48.528 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:48.529 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:48.529 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:48.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.529 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:24:48.530 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:48.530 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:48.530 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:48.530 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:48.530 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:48.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.531 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:24:48.531 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:48.532 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:48.532 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:48.532 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:48.532 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:48.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.533 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:24:48.533 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:48.533 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:48.534 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:48.534 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:48.534 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:48.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.534 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.535 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:24:48.535 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:48.535 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:48.535 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:48.536 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:48.536 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:48.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.537 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:24:48.537 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:48.537 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:48.537 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:48.537 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:48.538 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:48.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:48.538 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:24:48.539 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:24:48.539 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:24:48.539 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:24:48.539 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:24:48.539 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:24:48.540 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.540 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:24:48.540 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:24:48.540 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:24:48.541 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:24:48.541 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:24:48.541 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:24:48.541 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/3708886434": "2021-04-10 18:24:41.401718", 2021-04-09T18:24:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6837/1938812020": "2021-04-10 18:24:41.401702", 2021-04-09T18:24:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/3708886434": "2021-04-10 18:24:41.401718", 2021-04-09T18:24:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/1938812020": "2021-04-10 18:24:41.401702" 2021-04-09T18:24:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.542 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:24:48.543 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:24:48.543 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:24:48.543 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:24:48.543 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:24:48.544 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:24:48.544 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:24:48.544 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:24:48.544 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:48.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:48.545 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:24:48.545 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:24:48.545 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:24:48.545 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:24:48.546 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:24:48.546 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:24:48.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 -- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6cc018930 msgr2=0x7ff6cc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6cc018930 0x7ff6cc01ae10 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 -- 172.21.15.32:0/821866934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e409a5e0 msgr2=0x7ff6e41884b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e409a5e0 0x7ff6e41884b0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7ff6e0009650 tx=0x7ff6e000b960).stop 2021-04-09T18:24:48.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 -- 172.21.15.32:0/821866934 shutdown_connections 2021-04-09T18:24:48.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6cc018930 0x7ff6cc01ae10 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e40a15b0 0x7ff6e40a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e40a3350 0x7ff6e40a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 --2- 172.21.15.32:0/821866934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e409a5e0 0x7ff6e41884b0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 -- 172.21.15.32:0/821866934 >> 172.21.15.32:0/821866934 conn(0x7ff6e41a0db0 msgr2=0x7ff6e41a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 -- 172.21.15.32:0/821866934 shutdown_connections 2021-04-09T18:24:48.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.296 7ff6ec242700 1 -- 172.21.15.32:0/821866934 wait complete. 2021-04-09T18:24:48.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:48.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.635 7f3be27b1700 1 Processor -- start 2021-04-09T18:24:48.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.635 7f3be27b1700 1 -- start start 2021-04-09T18:24:48.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3be27b1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3bdc10d010 0x7f3bdc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3be27b1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3bdc0d1da0 0x7f3bdc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3be27b1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bdc10d480 con 0x7f3bdc0d82c0 2021-04-09T18:24:48.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3be27b1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bdc0d8d30 con 0x7f3bdc0d1da0 2021-04-09T18:24:48.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3be27b1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bdc0d9220 con 0x7f3bdc10d010 2021-04-09T18:24:48.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3bd3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3bdc0d1da0 0x7f3bdc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3be0dce700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3bdc0d82c0 0x7f3bdc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:32894/0 (socket says 172.21.15.32:32894) 2021-04-09T18:24:48.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3be0dce700 1 -- 172.21.15.32:0/3986001462 learned_addr learned my addr 172.21.15.32:0/3986001462 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:48.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3bdbfff700 1 --2- 172.21.15.32:0/3986001462 >> v2:172.21.15.86:3300/0 conn(0x7f3bdc10d010 0x7f3bdc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3bdb7fe700 1 -- 172.21.15.32:0/3986001462 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800980368 0 0) 0x7f3bd4000f90 con 0x7f3bdc0d82c0 2021-04-09T18:24:48.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3bdb7fe700 1 -- 172.21.15.32:0/3986001462 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bbc002fe0 con 0x7f3bdc0d82c0 2021-04-09T18:24:48.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3bd3fff700 1 -- 172.21.15.32:0/3986001462 >> v2:172.21.15.86:3300/0 conn(0x7f3bdc10d010 msgr2=0x7f3bdc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:48.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3bd3fff700 1 --2- 172.21.15.32:0/3986001462 >> v2:172.21.15.86:3300/0 conn(0x7f3bdc10d010 0x7f3bdc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.636 7f3bd3fff700 1 -- 172.21.15.32:0/3986001462 >> v1:172.21.15.32:6789/0 conn(0x7f3bdc0d82c0 legacy=0x7f3bdc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.637 7f3bd3fff700 1 -- 172.21.15.32:0/3986001462 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc0d86f0 con 0x7f3bdc0d1da0 2021-04-09T18:24:48.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.637 7f3bd3fff700 1 --2- 172.21.15.32:0/3986001462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc0d1da0 0x7f3bdc0d7ca0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f3bc4006650 tx=0x7f3bc4009a80).ready entity=mon.0 client_cookie=b20fa1bdf84bbec2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:48.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.637 7f3bdb7fe700 1 -- 172.21.15.32:0/3986001462 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bc40009b0 con 0x7f3bdc0d1da0 2021-04-09T18:24:48.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.637 7f3bdb7fe700 1 -- 172.21.15.32:0/3986001462 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc400c0c0 con 0x7f3bdc0d1da0 2021-04-09T18:24:48.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.637 7f3bdb7fe700 1 -- 172.21.15.32:0/3986001462 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bc400c510 con 0x7f3bdc0d1da0 2021-04-09T18:24:48.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.637 7f3be27b1700 1 -- 172.21.15.32:0/3986001462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc0d1da0 msgr2=0x7f3bdc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.637 7f3be27b1700 1 --2- 172.21.15.32:0/3986001462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc0d1da0 0x7f3bdc0d7ca0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f3bc4006650 tx=0x7f3bc4009a80).stop 2021-04-09T18:24:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.638 7f3be27b1700 1 -- 172.21.15.32:0/3986001462 shutdown_connections 2021-04-09T18:24:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.638 7f3be27b1700 1 --2- 172.21.15.32:0/3986001462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc0d1da0 0x7f3bdc0d7ca0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.638 7f3be27b1700 1 --2- 172.21.15.32:0/3986001462 >> v2:172.21.15.86:3300/0 conn(0x7f3bdc10d010 0x7f3bdc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.638 7f3be27b1700 1 -- 172.21.15.32:0/3986001462 >> 172.21.15.32:0/3986001462 conn(0x7f3bdc0ce5f0 msgr2=0x7f3bdc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:48.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.638 7f3be27b1700 1 -- 172.21.15.32:0/3986001462 shutdown_connections 2021-04-09T18:24:48.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.638 7f3be27b1700 1 -- 172.21.15.32:0/3986001462 wait complete. 2021-04-09T18:24:48.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3be27b1700 1 Processor -- start 2021-04-09T18:24:48.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3be27b1700 1 -- start start 2021-04-09T18:24:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3be27b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc1abcb0 0x7f3bdc1ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3be27b1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bdc191140 0x7f3bdc191540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3be27b1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bdc191b60 0x7f3bdc1a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3be27b1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc191f60 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3be27b1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc1ac6d0 con 0x7f3bdc191b60 2021-04-09T18:24:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3be27b1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc1aa5c0 con 0x7f3bdc191140 2021-04-09T18:24:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3bdbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc1abcb0 0x7f3bdc1ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3bdbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc1abcb0 0x7f3bdc1ac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51538/0 (socket says 172.21.15.32:51538) 2021-04-09T18:24:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.658 7f3bdbfff700 1 -- 172.21.15.32:0/4122182732 learned_addr learned my addr 172.21.15.32:0/4122182732 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3be0dce700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bdc191b60 0x7f3bdc1a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bd3fff700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bdc191140 0x7f3bdc191540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bdbfff700 1 -- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bdc191140 msgr2=0x7f3bdc191540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bdbfff700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bdc191140 0x7f3bdc191540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bdbfff700 1 -- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bdc191b60 msgr2=0x7f3bdc1a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bdbfff700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bdc191b60 0x7f3bdc1a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bdbfff700 1 -- 172.21.15.32:0/4122182732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc1aa7e0 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bdbfff700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc1abcb0 0x7f3bdc1ac0b0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7f3bcc00b560 tx=0x7f3bcc00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bd97fa700 1 -- 172.21.15.32:0/4122182732 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bcc00bcf0 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bd97fa700 1 -- 172.21.15.32:0/4122182732 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bcc000c90 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bdc1aa7e0 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3bd97fa700 1 -- 172.21.15.32:0/4122182732 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bcc002220 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.659 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bdc1aa7e0 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.660 7f3bd97fa700 1 -- 172.21.15.32:0/4122182732 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3bcc0052e0 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.660 7f3bd97fa700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3bb40188c0 0x7f3bb401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:48.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.660 7f3bd97fa700 1 -- 172.21.15.32:0/4122182732 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f3bcc0327a0 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.660 7f3bd3fff700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3bb40188c0 0x7f3bb401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:48.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.660 7f3bd3fff700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3bb40188c0 0x7f3bb401adc0 crc :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:48.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.680 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bdc191f60 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.684 7f3bd97fa700 1 -- 172.21.15.32:0/4122182732 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3bcc000c90 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.877 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bdc191f60 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.877 7f3bd97fa700 1 -- 172.21.15.32:0/4122182732 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+1854 (secure 0 0 0) 0x7f3bcc006b00 con 0x7f3bdc1abcb0 2021-04-09T18:24:48.941 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 248 2021-04-09T18:24:48.942 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:48.942 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":248,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8647,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2324879882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2324879882},{"type":"v1","addr":"172.21.15.86:6837","nonce":2324879882}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":248},{"gid":8859,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3459836132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3459836132},{"type":"v1","addr":"172.21.15.32:6835","nonce":3459836132}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":245},{"gid":8910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/543985226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":543985226},{"type":"v1","addr":"172.21.15.32:6834","nonce":543985226}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":248},{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[]} 2021-04-09T18:24:48.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3bb40188c0 msgr2=0x7f3bb401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3bb40188c0 0x7f3bb401adc0 crc :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc1abcb0 msgr2=0x7f3bdc1ac0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:48.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc1abcb0 0x7f3bdc1ac0b0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7f3bcc00b560 tx=0x7f3bcc00b900).stop 2021-04-09T18:24:48.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 shutdown_connections 2021-04-09T18:24:48.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3bb40188c0 0x7f3bb401adc0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bdc191b60 0x7f3bdc1a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bdc191140 0x7f3bdc191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 --2- 172.21.15.32:0/4122182732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bdc1abcb0 0x7f3bdc1ac0b0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:48.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 >> 172.21.15.32:0/4122182732 conn(0x7f3bdc1837d0 msgr2=0x7f3bdc195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:48.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 shutdown_connections 2021-04-09T18:24:48.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:48.880 7f3be27b1700 1 -- 172.21.15.32:0/4122182732 wait complete. 2021-04-09T18:24:48.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:49.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.039 7f19148f8700 1 Processor -- start 2021-04-09T18:24:49.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.039 7f19148f8700 1 -- start start 2021-04-09T18:24:49.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.039 7f19148f8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191010da00 0x7f191010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.039 7f19148f8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191010d480 con 0x7f191010d010 2021-04-09T18:24:49.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.039 7f19148f8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19100d1f70 con 0x7f19100d7ca0 2021-04-09T18:24:49.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.039 7f19148f8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19100d9220 con 0x7f191010da00 2021-04-09T18:24:49.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190e61c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f191010d010 0x7f19100d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:32904/0 (socket says 172.21.15.32:32904) 2021-04-09T18:24:49.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190e61c700 1 -- 172.21.15.32:0/1611477525 learned_addr learned my addr 172.21.15.32:0/1611477525 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:49.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190ee1d700 1 --2- 172.21.15.32:0/1611477525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191010da00 0x7f191010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190d61a700 1 -- 172.21.15.32:0/1611477525 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37200200 0 0) 0x7f1900000ce0 con 0x7f191010d010 2021-04-09T18:24:49.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190d61a700 1 -- 172.21.15.32:0/1611477525 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f8002fe0 con 0x7f191010d010 2021-04-09T18:24:49.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190d61a700 1 -- 172.21.15.32:0/1611477525 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123317225 0 0) 0x7f1904001080 con 0x7f19100d7ca0 2021-04-09T18:24:49.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190d61a700 1 -- 172.21.15.32:0/1611477525 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f8003210 con 0x7f19100d7ca0 2021-04-09T18:24:49.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190d61a700 1 -- 172.21.15.32:0/1611477525 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1356124185 0 0) 0x7f19000010e0 con 0x7f191010d010 2021-04-09T18:24:49.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190ee1d700 1 -- 172.21.15.32:0/1611477525 >> v1:172.21.15.86:6789/0 conn(0x7f19100d7ca0 legacy=0x7f19100d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190ee1d700 1 -- 172.21.15.32:0/1611477525 >> v1:172.21.15.32:6789/0 conn(0x7f191010d010 legacy=0x7f19100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190ee1d700 1 -- 172.21.15.32:0/1611477525 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191010e420 con 0x7f191010da00 2021-04-09T18:24:49.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190ee1d700 1 --2- 172.21.15.32:0/1611477525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191010da00 0x7f191010de00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f1908006e50 tx=0x7f1908009df0).ready entity=mon.2 client_cookie=3c772ea9f7553c76 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:49.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.040 7f190d61a700 1 -- 172.21.15.32:0/1611477525 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f190800b510 con 0x7f191010da00 2021-04-09T18:24:49.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.041 7f190d61a700 1 -- 172.21.15.32:0/1611477525 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1908003030 con 0x7f191010da00 2021-04-09T18:24:49.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.041 7f190d61a700 1 -- 172.21.15.32:0/1611477525 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f190800b7d0 con 0x7f191010da00 2021-04-09T18:24:49.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.041 7f19148f8700 1 -- 172.21.15.32:0/1611477525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191010da00 msgr2=0x7f191010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.041 7f19148f8700 1 --2- 172.21.15.32:0/1611477525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191010da00 0x7f191010de00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f1908006e50 tx=0x7f1908009df0).stop 2021-04-09T18:24:49.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.041 7f19148f8700 1 -- 172.21.15.32:0/1611477525 shutdown_connections 2021-04-09T18:24:49.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.041 7f19148f8700 1 --2- 172.21.15.32:0/1611477525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191010da00 0x7f191010de00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.041 7f19148f8700 1 -- 172.21.15.32:0/1611477525 >> 172.21.15.32:0/1611477525 conn(0x7f19100ce5f0 msgr2=0x7f19100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:49.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.041 7f19148f8700 1 -- 172.21.15.32:0/1611477525 shutdown_connections 2021-04-09T18:24:49.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.041 7f19148f8700 1 -- 172.21.15.32:0/1611477525 wait complete. 2021-04-09T18:24:49.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.061 7f19148f8700 1 Processor -- start 2021-04-09T18:24:49.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f19148f8700 1 -- start start 2021-04-09T18:24:49.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f19148f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19101ac380 0x7f191018a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f19148f8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19101ac780 0x7f191018a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f19148f8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1910183520 0x7f1910183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f19148f8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19101ace10 con 0x7f19101ac380 2021-04-09T18:24:49.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f19148f8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1910189cf0 con 0x7f19101ac780 2021-04-09T18:24:49.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f19148f8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1910183f40 con 0x7f1910183520 2021-04-09T18:24:49.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f190e61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19101ac380 0x7f191018a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f190e61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19101ac380 0x7f191018a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51552/0 (socket says 172.21.15.32:51552) 2021-04-09T18:24:49.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f190e61c700 1 -- 172.21.15.32:0/3575153378 learned_addr learned my addr 172.21.15.32:0/3575153378 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:49.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f190ee1d700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1910183520 0x7f1910183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.062 7f190de1b700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19101ac780 0x7f191018a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f190e61c700 1 -- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1910183520 msgr2=0x7f1910183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f190e61c700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1910183520 0x7f1910183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f190e61c700 1 -- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19101ac780 msgr2=0x7f191018a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f190e61c700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19101ac780 0x7f191018a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f190e61c700 1 -- 172.21.15.32:0/3575153378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1910184160 con 0x7f19101ac380 2021-04-09T18:24:49.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f190de1b700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19101ac780 0x7f191018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:49.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f190e61c700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19101ac380 0x7f191018a230 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f1900000f90 tx=0x7f190000a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:49.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f18feffd700 1 -- 172.21.15.32:0/3575153378 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1900001a30 con 0x7f19101ac380 2021-04-09T18:24:49.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f18feffd700 1 -- 172.21.15.32:0/3575153378 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19000043a0 con 0x7f19101ac380 2021-04-09T18:24:49.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f19148f8700 1 -- 172.21.15.32:0/3575153378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1910184160 con 0x7f19101ac380 2021-04-09T18:24:49.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f18feffd700 1 -- 172.21.15.32:0/3575153378 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19000047f0 con 0x7f19101ac380 2021-04-09T18:24:49.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f19148f8700 1 -- 172.21.15.32:0/3575153378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1910184160 con 0x7f19101ac380 2021-04-09T18:24:49.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.063 7f18feffd700 1 -- 172.21.15.32:0/3575153378 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1900004580 con 0x7f19101ac380 2021-04-09T18:24:49.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.064 7f18feffd700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18f4018930 0x7f18f401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.064 7f18feffd700 1 -- 172.21.15.32:0/3575153378 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f1900032750 con 0x7f19101ac380 2021-04-09T18:24:49.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.064 7f190de1b700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18f4018930 0x7f18f401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.064 7f190de1b700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18f4018930 0x7f18f401adc0 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:49.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.083 7f19148f8700 1 -- 172.21.15.32:0/3575153378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1910182d10 con 0x7f19101ac380 2021-04-09T18:24:49.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.087 7f18feffd700 1 -- 172.21.15.32:0/3575153378 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1900001a10 con 0x7f19101ac380 2021-04-09T18:24:49.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.278 7f19148f8700 1 -- 172.21.15.32:0/3575153378 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1910182d10 con 0x7f19101ac380 2021-04-09T18:24:49.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.279 7f18feffd700 1 -- 172.21.15.32:0/3575153378 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+1854 (secure 0 0 0) 0x7f19000043a0 con 0x7f19101ac380 2021-04-09T18:24:49.343 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 248 2021-04-09T18:24:49.344 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:49.344 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":248,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8647,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2324879882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2324879882},{"type":"v1","addr":"172.21.15.86:6837","nonce":2324879882}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":248},{"gid":8859,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3459836132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3459836132},{"type":"v1","addr":"172.21.15.32:6835","nonce":3459836132}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":245},{"gid":8910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/543985226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":543985226},{"type":"v1","addr":"172.21.15.32:6834","nonce":543985226}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":248},{"gid":57224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3795682921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3795682921},{"type":"v1","addr":"172.21.15.86:6836","nonce":3795682921}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":238}],"filesystems":[]} 2021-04-09T18:24:49.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.281 7f19148f8700 1 -- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18f4018930 msgr2=0x7f18f401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.281 7f19148f8700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18f4018930 0x7f18f401adc0 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.281 7f19148f8700 1 -- 172.21.15.32:0/3575153378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19101ac380 msgr2=0x7f191018a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.281 7f19148f8700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19101ac380 0x7f191018a230 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 rx=0x7f1900000f90 tx=0x7f190000a940).stop 2021-04-09T18:24:49.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.282 7f19148f8700 1 -- 172.21.15.32:0/3575153378 shutdown_connections 2021-04-09T18:24:49.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.282 7f19148f8700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18f4018930 0x7f18f401adc0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.282 7f19148f8700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1910183520 0x7f1910183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.282 7f19148f8700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19101ac780 0x7f191018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.282 7f19148f8700 1 --2- 172.21.15.32:0/3575153378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19101ac380 0x7f191018a230 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.282 7f19148f8700 1 -- 172.21.15.32:0/3575153378 >> 172.21.15.32:0/3575153378 conn(0x7f191019b890 msgr2=0x7f19101a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:49.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.282 7f19148f8700 1 -- 172.21.15.32:0/3575153378 shutdown_connections 2021-04-09T18:24:49.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.282 7f19148f8700 1 -- 172.21.15.32:0/3575153378 wait complete. 2021-04-09T18:24:49.356 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:24:49.356 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:24:49.357 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:24:49.357 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:24:49.357 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:24:49.358 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:24:49.358 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:24:49.359 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:24:49.359 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:24:49.359 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:24:49.360 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:24:49.360 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:24:49.360 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:24:49.361 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:24:49.361 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:24:49.361 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:24:49.365 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:24:49.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:24:49.367 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:24:49.368 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:24:49.369 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:24:49.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:24:49.370 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:24:49.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:24:49.409 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:24:49.409 DEBUG:teuthology.parallel:result is None 2021-04-09T18:24:49.411 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:24:49.411 DEBUG:teuthology.parallel:result is None 2021-04-09T18:24:49.412 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:24:49.412 DEBUG:teuthology.parallel:result is None 2021-04-09T18:24:49.413 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:24:49.414 DEBUG:teuthology.parallel:result is None 2021-04-09T18:24:49.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:24:49.558 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:24:49.558 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:24:49.559 INFO:tasks.ceph.mds.a.smithi032.stderr:2021-04-09 18:24:49.495 7f09c81f2700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2021-04-09T18:24:49.568 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:24:49.569 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:24:49.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.504 7f3416cda700 1 Processor -- start 2021-04-09T18:24:49.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3416cda700 1 -- start start 2021-04-09T18:24:49.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3416cda700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34100cffb0 0x7f34100d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3416cda700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34100d7810 0x7f34100d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3416cda700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f34100d8230 0x7f34100d5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3416cda700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34100d91f0 con 0x7f34100cffb0 2021-04-09T18:24:49.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3416cda700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34100d7290 con 0x7f34100d8230 2021-04-09T18:24:49.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3416cda700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34100d62a0 con 0x7f34100d7810 2021-04-09T18:24:49.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3414af6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34100cffb0 0x7f34100d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3414af6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34100cffb0 0x7f34100d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51582/0 (socket says 172.21.15.32:51582) 2021-04-09T18:24:49.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f3414af6700 1 -- 172.21.15.32:0/727671206 learned_addr learned my addr 172.21.15.32:0/727671206 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:49.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f34152f7700 1 --2- 172.21.15.32:0/727671206 >> v2:172.21.15.86:3300/0 conn(0x7f34100d8230 0x7f34100d5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.505 7f340ffff700 1 --2- 172.21.15.32:0/727671206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34100d7810 0x7f34100d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f3414af6700 1 -- 172.21.15.32:0/727671206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34100d7810 msgr2=0x7f34100d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:49.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f3414af6700 1 --2- 172.21.15.32:0/727671206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34100d7810 0x7f34100d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f3414af6700 1 -- 172.21.15.32:0/727671206 >> v2:172.21.15.86:3300/0 conn(0x7f34100d8230 msgr2=0x7f34100d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:49.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f3414af6700 1 --2- 172.21.15.32:0/727671206 >> v2:172.21.15.86:3300/0 conn(0x7f34100d8230 0x7f34100d5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f3414af6700 1 -- 172.21.15.32:0/727671206 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100d64c0 con 0x7f34100cffb0 2021-04-09T18:24:49.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f3414af6700 1 --2- 172.21.15.32:0/727671206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34100cffb0 0x7f34100d2450 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7f33f8006700 tx=0x7f33f8009b00).ready entity=mon.0 client_cookie=9d0da5a161b5f1d3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:49.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f340f7fe700 1 -- 172.21.15.32:0/727671206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f800c0c0 con 0x7f34100cffb0 2021-04-09T18:24:49.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f340f7fe700 1 -- 172.21.15.32:0/727671206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f800c400 con 0x7f34100cffb0 2021-04-09T18:24:49.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f340f7fe700 1 -- 172.21.15.32:0/727671206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f800c8e0 con 0x7f34100cffb0 2021-04-09T18:24:49.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f3416cda700 1 -- 172.21.15.32:0/727671206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34100cffb0 msgr2=0x7f34100d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.506 7f3416cda700 1 --2- 172.21.15.32:0/727671206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34100cffb0 0x7f34100d2450 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7f33f8006700 tx=0x7f33f8009b00).stop 2021-04-09T18:24:49.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.507 7f3416cda700 1 -- 172.21.15.32:0/727671206 shutdown_connections 2021-04-09T18:24:49.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.507 7f3416cda700 1 --2- 172.21.15.32:0/727671206 >> v2:172.21.15.86:3300/0 conn(0x7f34100d8230 0x7f34100d5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.507 7f3416cda700 1 --2- 172.21.15.32:0/727671206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34100d7810 0x7f34100d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.507 7f3416cda700 1 --2- 172.21.15.32:0/727671206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34100cffb0 0x7f34100d2450 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.507 7f3416cda700 1 -- 172.21.15.32:0/727671206 >> 172.21.15.32:0/727671206 conn(0x7f34100ce5d0 msgr2=0x7f34100cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:49.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.507 7f3416cda700 1 -- 172.21.15.32:0/727671206 shutdown_connections 2021-04-09T18:24:49.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.507 7f3416cda700 1 -- 172.21.15.32:0/727671206 wait complete. 2021-04-09T18:24:49.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.527 7f3416cda700 1 Processor -- start 2021-04-09T18:24:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.527 7f3416cda700 1 -- start start 2021-04-09T18:24:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f3416cda700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34100a54b0 0x7f3410197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f3416cda700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34101a9ef0 0x7f34101aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f3416cda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34101aa2f0 0x7f34101ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f3416cda700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34101aa900 con 0x7f34101aa2f0 2021-04-09T18:24:49.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f3416cda700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100a58b0 con 0x7f34100a54b0 2021-04-09T18:24:49.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f3416cda700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34101984f0 con 0x7f34101a9ef0 2021-04-09T18:24:49.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f34152f7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34101aa2f0 0x7f34101ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f34152f7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34101aa2f0 0x7f34101ab4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51592/0 (socket says 172.21.15.32:51592) 2021-04-09T18:24:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f34152f7700 1 -- 172.21.15.32:0/3329604148 learned_addr learned my addr 172.21.15.32:0/3329604148 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f3414af6700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34100a54b0 0x7f3410197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f340ffff700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34101a9ef0 0x7f34101aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f34152f7700 1 -- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34101a9ef0 msgr2=0x7f34101aae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f34152f7700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34101a9ef0 0x7f34101aae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f34152f7700 1 -- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34100a54b0 msgr2=0x7f3410197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f34152f7700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34100a54b0 0x7f3410197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.528 7f34152f7700 1 -- 172.21.15.32:0/3329604148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100a5ce0 con 0x7f34101aa2f0 2021-04-09T18:24:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.529 7f34152f7700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34101aa2f0 0x7f34101ab4a0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7f340800a8a0 tx=0x7f340800ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.529 7f340d7fa700 1 -- 172.21.15.32:0/3329604148 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f340800c300 con 0x7f34101aa2f0 2021-04-09T18:24:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.529 7f340d7fa700 1 -- 172.21.15.32:0/3329604148 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3408003030 con 0x7f34101aa2f0 2021-04-09T18:24:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.529 7f3416cda700 1 -- 172.21.15.32:0/3329604148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34100a5ce0 con 0x7f34101aa2f0 2021-04-09T18:24:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.529 7f340d7fa700 1 -- 172.21.15.32:0/3329604148 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f340800c7f0 con 0x7f34101aa2f0 2021-04-09T18:24:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.529 7f3416cda700 1 -- 172.21.15.32:0/3329604148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34100a5ce0 con 0x7f34101aa2f0 2021-04-09T18:24:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.529 7f340d7fa700 1 -- 172.21.15.32:0/3329604148 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f34080066f0 con 0x7f34101aa2f0 2021-04-09T18:24:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.530 7f340d7fa700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33f4018930 0x7f33f401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.530 7f340d7fa700 1 -- 172.21.15.32:0/3329604148 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f3408034d40 con 0x7f34101aa2f0 2021-04-09T18:24:49.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.530 7f3414af6700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33f4018930 0x7f33f401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:49.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.530 7f3414af6700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33f4018930 0x7f33f401ae10 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:49.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.549 7f3416cda700 1 -- 172.21.15.32:0/3329604148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3410182e20 con 0x7f34101aa2f0 2021-04-09T18:24:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.552 7f340d7fa700 1 -- 172.21.15.32:0/3329604148 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f340800ad80 con 0x7f34101aa2f0 2021-04-09T18:24:49.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:49.754 7f3416cda700 1 -- 172.21.15.32:0/3329604148 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f34100d7290 con 0x7f34101aa2f0 2021-04-09T18:24:50.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.194 7f340d7fa700 1 -- 172.21.15.32:0/3329604148 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v111) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3408003010 con 0x7f34101aa2f0 2021-04-09T18:24:50.282 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:24:50.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 -- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33f4018930 msgr2=0x7f33f401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33f4018930 0x7f33f401ae10 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 -- 172.21.15.32:0/3329604148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34101aa2f0 msgr2=0x7f34101ab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34101aa2f0 0x7f34101ab4a0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7f340800a8a0 tx=0x7f340800ac40).stop 2021-04-09T18:24:50.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 -- 172.21.15.32:0/3329604148 shutdown_connections 2021-04-09T18:24:50.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33f4018930 0x7f33f401ae10 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34101aa2f0 0x7f34101ab4a0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34101a9ef0 0x7f34101aae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 --2- 172.21.15.32:0/3329604148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34100a54b0 0x7f3410197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 -- 172.21.15.32:0/3329604148 >> 172.21.15.32:0/3329604148 conn(0x7f34101838a0 msgr2=0x7f34101a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:50.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 -- 172.21.15.32:0/3329604148 shutdown_connections 2021-04-09T18:24:50.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.221 7f3416cda700 1 -- 172.21.15.32:0/3329604148 wait complete. 2021-04-09T18:24:50.296 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:24:50.296 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:24:50.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.381 7f4715e20700 1 Processor -- start 2021-04-09T18:24:50.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.381 7f4715e20700 1 -- start start 2021-04-09T18:24:50.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f4715e20700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f4715e20700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47100d1da0 0x7f47100d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f4715e20700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471010d480 con 0x7f47100d82c0 2021-04-09T18:24:50.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f4715e20700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471010e780 con 0x7f47100d1da0 2021-04-09T18:24:50.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f4715e20700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100d8d30 con 0x7f471010d010 2021-04-09T18:24:50.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f470effd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47100d1da0 0x7f47100d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f470effd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47100d1da0 0x7f47100d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51600/0 (socket says 172.21.15.32:51600) 2021-04-09T18:24:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f470effd700 1 -- 172.21.15.32:0/2826469922 learned_addr learned my addr 172.21.15.32:0/2826469922 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f470f7fe700 1 --2- 172.21.15.32:0/2826469922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f470effd700 1 -- 172.21.15.32:0/2826469922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 msgr2=0x7f47100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:50.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f470effd700 1 --2- 172.21.15.32:0/2826469922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f470effd700 1 -- 172.21.15.32:0/2826469922 >> v1:172.21.15.86:6789/0 conn(0x7f47100d82c0 legacy=0x7f471010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f470effd700 1 -- 172.21.15.32:0/2826469922 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100d9220 con 0x7f47100d1da0 2021-04-09T18:24:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.382 7f470effd700 1 --2- 172.21.15.32:0/2826469922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47100d1da0 0x7f47100d7ca0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7f4704006700 tx=0x7f4704009b80).ready entity=mon.0 client_cookie=8ab70facc5f22731 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f470e7fc700 1 -- 172.21.15.32:0/2826469922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470400c0c0 con 0x7f47100d1da0 2021-04-09T18:24:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f470e7fc700 1 -- 172.21.15.32:0/2826469922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470400c440 con 0x7f47100d1da0 2021-04-09T18:24:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f470e7fc700 1 -- 172.21.15.32:0/2826469922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470400c920 con 0x7f47100d1da0 2021-04-09T18:24:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f4715e20700 1 -- 172.21.15.32:0/2826469922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47100d1da0 msgr2=0x7f47100d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f4715e20700 1 --2- 172.21.15.32:0/2826469922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47100d1da0 0x7f47100d7ca0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7f4704006700 tx=0x7f4704009b80).stop 2021-04-09T18:24:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f4715e20700 1 -- 172.21.15.32:0/2826469922 shutdown_connections 2021-04-09T18:24:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f4715e20700 1 --2- 172.21.15.32:0/2826469922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47100d1da0 0x7f47100d7ca0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f4715e20700 1 --2- 172.21.15.32:0/2826469922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471010d010 0x7f47100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f4715e20700 1 -- 172.21.15.32:0/2826469922 >> 172.21.15.32:0/2826469922 conn(0x7f47100ce5f0 msgr2=0x7f47100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f4715e20700 1 -- 172.21.15.32:0/2826469922 shutdown_connections 2021-04-09T18:24:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.383 7f4715e20700 1 -- 172.21.15.32:0/2826469922 wait complete. 2021-04-09T18:24:50.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f4715e20700 1 Processor -- start 2021-04-09T18:24:50.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f4715e20700 1 -- start start 2021-04-09T18:24:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f4715e20700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a5210 0x7f4710185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f4715e20700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4710185a70 0x7f47100acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f4715e20700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4710185e70 0x7f47100ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f4715e20700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471018f7b0 con 0x7f4710185a70 2021-04-09T18:24:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f4715e20700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47100ad7f0 con 0x7f47101a5210 2021-04-09T18:24:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f4715e20700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100ada10 con 0x7f4710185e70 2021-04-09T18:24:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f470effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4710185a70 0x7f47100acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f470effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4710185a70 0x7f47100acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51602/0 (socket says 172.21.15.32:51602) 2021-04-09T18:24:50.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.404 7f470effd700 1 -- 172.21.15.32:0/4254209911 learned_addr learned my addr 172.21.15.32:0/4254209911 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:50.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f470f7fe700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a5210 0x7f4710185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f470ffff700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4710185e70 0x7f47100ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f470effd700 1 -- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4710185e70 msgr2=0x7f47100ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f470effd700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4710185e70 0x7f47100ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f470effd700 1 -- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a5210 msgr2=0x7f4710185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f470effd700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a5210 0x7f4710185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f470effd700 1 -- 172.21.15.32:0/4254209911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4710179610 con 0x7f4710185a70 2021-04-09T18:24:50.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f470effd700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4710185a70 0x7f47100acbb0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7f470400b580 tx=0x7f470400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:50.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f46f7fff700 1 -- 172.21.15.32:0/4254209911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470400bd10 con 0x7f4710185a70 2021-04-09T18:24:50.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f46f7fff700 1 -- 172.21.15.32:0/4254209911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4704000c90 con 0x7f4710185a70 2021-04-09T18:24:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f46f7fff700 1 -- 172.21.15.32:0/4254209911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4704002220 con 0x7f4710185a70 2021-04-09T18:24:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f4715e20700 1 -- 172.21.15.32:0/4254209911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4710179610 con 0x7f4710185a70 2021-04-09T18:24:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.405 7f4715e20700 1 -- 172.21.15.32:0/4254209911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4710179610 con 0x7f4710185a70 2021-04-09T18:24:50.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.406 7f46f7fff700 1 -- 172.21.15.32:0/4254209911 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4704009f90 con 0x7f4710185a70 2021-04-09T18:24:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.406 7f46f7fff700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f0018930 0x7f46f001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.406 7f46f7fff700 1 -- 172.21.15.32:0/4254209911 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f47040326c0 con 0x7f4710185a70 2021-04-09T18:24:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.406 7f470f7fe700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f0018930 0x7f46f001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.406 7f470f7fe700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f0018930 0x7f46f001ae10 crc :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:50.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.426 7f4715e20700 1 -- 172.21.15.32:0/4254209911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47101b7070 con 0x7f4710185a70 2021-04-09T18:24:50.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.429 7f46f7fff700 1 -- 172.21.15.32:0/4254209911 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f47040091e0 con 0x7f4710185a70 2021-04-09T18:24:50.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.616 7f4715e20700 1 -- 172.21.15.32:0/4254209911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f471010d480 con 0x7f4710185a70 2021-04-09T18:24:50.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.616 7f46f7fff700 1 -- 172.21.15.32:0/4254209911 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v29) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f4704000c90 con 0x7f4710185a70 2021-04-09T18:24:50.681 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:50.681 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:24:50.681 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:24:50.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.682 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:24:50.682 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:24:50.682 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.682 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:24:50.683 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.683 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.683 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:24:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:24:50.684 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.685 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:24:50.685 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.685 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.685 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.686 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.686 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:24:50.686 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:24:50.687 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.687 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:24:50.687 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.687 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.688 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.688 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.689 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:24:50.689 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:24:50.689 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.689 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:24:50.690 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.690 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.690 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:24:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:24:50.691 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.693 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:24:50.693 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:24:50.693 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.694 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.695 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:24:50.695 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:24:50.695 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.696 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.696 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.696 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.697 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:24:50.697 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:24:50.697 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.697 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.698 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.698 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.698 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.699 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:24:50.699 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:24:50.699 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.699 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.700 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.700 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.700 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.701 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:24:50.701 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:24:50.701 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.701 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.701 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.702 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.703 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:24:50.703 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:24:50.703 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.703 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.703 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.704 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.704 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.704 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:24:50.705 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:24:50.705 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.705 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.705 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.706 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.706 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:24:50.707 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:24:50.707 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:24:50.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:24:50.707 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:24:50.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.708 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:24:50.709 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:24:50.709 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.709 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:24:50.709 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:24:50.709 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:24:50.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.710 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:24:50.710 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:24:50.711 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.711 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:24:50.711 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:24:50.711 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:24:50.711 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.712 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.712 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:24:50.712 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:24:50.713 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.713 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:24:50.713 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:24:50.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.714 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:24:50.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:50.714 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:24:50.714 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:24:50.715 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:24:50.715 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:24:50.715 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:24:50.715 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:24:50.715 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.716 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:50.716 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:24:50.716 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:24:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 -- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f0018930 msgr2=0x7f46f001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f0018930 0x7f46f001ae10 crc :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 -- 172.21.15.32:0/4254209911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4710185a70 msgr2=0x7f47100acbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4710185a70 0x7f47100acbb0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7f470400b580 tx=0x7f470400b920).stop 2021-04-09T18:24:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 -- 172.21.15.32:0/4254209911 shutdown_connections 2021-04-09T18:24:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46f0018930 0x7f46f001ae10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4710185a70 0x7f47100acbb0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4710185e70 0x7f47100ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 --2- 172.21.15.32:0/4254209911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47101a5210 0x7f4710185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 -- 172.21.15.32:0/4254209911 >> 172.21.15.32:0/4254209911 conn(0x7f47101a99c0 msgr2=0x7f471017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 -- 172.21.15.32:0/4254209911 shutdown_connections 2021-04-09T18:24:50.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.619 7f4715e20700 1 -- 172.21.15.32:0/4254209911 wait complete. 2021-04-09T18:24:50.721 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:24:50.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.806 7fe430021700 1 Processor -- start 2021-04-09T18:24:50.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.806 7fe430021700 1 -- start start 2021-04-09T18:24:50.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.806 7fe430021700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe42810d010 0x7fe4280d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.806 7fe430021700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280d1da0 0x7fe4280d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.806 7fe430021700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42810d480 con 0x7fe4280d82c0 2021-04-09T18:24:50.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.806 7fe430021700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4280d9220 con 0x7fe42810d010 2021-04-09T18:24:50.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.806 7fe430021700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4280d86c0 con 0x7fe4280d1da0 2021-04-09T18:24:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42de3d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe42810d010 0x7fe4280d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42e63e700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe4280d82c0 0x7fe42810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:59128/0 (socket says 172.21.15.32:59128) 2021-04-09T18:24:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42e63e700 1 -- 172.21.15.32:0/3470744257 learned_addr learned my addr 172.21.15.32:0/3470744257 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:50.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42d63c700 1 --2- 172.21.15.32:0/3470744257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280d1da0 0x7fe4280d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42ce3b700 1 -- 172.21.15.32:0/3470744257 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664663657 0 0) 0x7fe420001440 con 0x7fe4280d82c0 2021-04-09T18:24:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42ce3b700 1 -- 172.21.15.32:0/3470744257 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe410002fe0 con 0x7fe4280d82c0 2021-04-09T18:24:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42de3d700 1 -- 172.21.15.32:0/3470744257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280d1da0 msgr2=0x7fe4280d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42de3d700 1 --2- 172.21.15.32:0/3470744257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280d1da0 0x7fe4280d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42de3d700 1 -- 172.21.15.32:0/3470744257 >> v1:172.21.15.86:6789/0 conn(0x7fe4280d82c0 legacy=0x7fe42810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42de3d700 1 -- 172.21.15.32:0/3470744257 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42810e1f0 con 0x7fe42810d010 2021-04-09T18:24:50.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42d63c700 1 --2- 172.21.15.32:0/3470744257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280d1da0 0x7fe4280d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:50.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42de3d700 1 --2- 172.21.15.32:0/3470744257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe42810d010 0x7fe4280d1780 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fe418006650 tx=0x7fe418009a80).ready entity=mon.1 client_cookie=cb2358c44ea75583 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:50.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42ce3b700 1 -- 172.21.15.32:0/3470744257 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4180009b0 con 0x7fe42810d010 2021-04-09T18:24:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42ce3b700 1 -- 172.21.15.32:0/3470744257 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41800c0c0 con 0x7fe42810d010 2021-04-09T18:24:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe42ce3b700 1 -- 172.21.15.32:0/3470744257 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41800c510 con 0x7fe42810d010 2021-04-09T18:24:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.807 7fe430021700 1 -- 172.21.15.32:0/3470744257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe42810d010 msgr2=0x7fe4280d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.808 7fe430021700 1 --2- 172.21.15.32:0/3470744257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe42810d010 0x7fe4280d1780 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fe418006650 tx=0x7fe418009a80).stop 2021-04-09T18:24:50.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.808 7fe430021700 1 -- 172.21.15.32:0/3470744257 shutdown_connections 2021-04-09T18:24:50.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.808 7fe430021700 1 --2- 172.21.15.32:0/3470744257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280d1da0 0x7fe4280d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.808 7fe430021700 1 --2- 172.21.15.32:0/3470744257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe42810d010 0x7fe4280d1780 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.808 7fe430021700 1 -- 172.21.15.32:0/3470744257 >> 172.21.15.32:0/3470744257 conn(0x7fe4280ce5f0 msgr2=0x7fe4280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:50.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.808 7fe430021700 1 -- 172.21.15.32:0/3470744257 shutdown_connections 2021-04-09T18:24:50.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.808 7fe430021700 1 -- 172.21.15.32:0/3470744257 wait complete. 2021-04-09T18:24:50.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.828 7fe430021700 1 Processor -- start 2021-04-09T18:24:50.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.828 7fe430021700 1 -- start start 2021-04-09T18:24:50.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe430021700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4281a5210 0x7fe4281a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe430021700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4281a6340 0x7fe4281a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe430021700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280abaa0 0x7fe4280abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe430021700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281a6d60 con 0x7fe4281a6340 2021-04-09T18:24:50.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe430021700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4280ac4c0 con 0x7fe4281a5210 2021-04-09T18:24:50.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe430021700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4280ac6e0 con 0x7fe4280abaa0 2021-04-09T18:24:50.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe42d63c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4281a6340 0x7fe4281a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe42d63c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4281a6340 0x7fe4281a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51616/0 (socket says 172.21.15.32:51616) 2021-04-09T18:24:50.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe42d63c700 1 -- 172.21.15.32:0/1796737734 learned_addr learned my addr 172.21.15.32:0/1796737734 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:50.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe42e63e700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280abaa0 0x7fe4280abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.829 7fe42de3d700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4281a5210 0x7fe4281a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe42d63c700 1 -- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280abaa0 msgr2=0x7fe4280abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe42d63c700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280abaa0 0x7fe4280abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe42d63c700 1 -- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4281a5210 msgr2=0x7fe4281a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:50.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe42d63c700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4281a5210 0x7fe4281a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe42d63c700 1 -- 172.21.15.32:0/1796737734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42817a720 con 0x7fe4281a6340 2021-04-09T18:24:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe42e63e700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280abaa0 0x7fe4280abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe42de3d700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4281a5210 0x7fe4281a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe42d63c700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4281a6340 0x7fe4281a6740 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 rx=0x7fe41c006f40 tx=0x7fe41c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe4267fc700 1 -- 172.21.15.32:0/1796737734 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c00bd80 con 0x7fe4281a6340 2021-04-09T18:24:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe4267fc700 1 -- 172.21.15.32:0/1796737734 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c000c90 con 0x7fe4281a6340 2021-04-09T18:24:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe430021700 1 -- 172.21.15.32:0/1796737734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42817a9c0 con 0x7fe4281a6340 2021-04-09T18:24:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe4267fc700 1 -- 172.21.15.32:0/1796737734 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c002220 con 0x7fe4281a6340 2021-04-09T18:24:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.830 7fe430021700 1 -- 172.21.15.32:0/1796737734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42817acb0 con 0x7fe4281a6340 2021-04-09T18:24:50.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.831 7fe4267fc700 1 -- 172.21.15.32:0/1796737734 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe41c00a000 con 0x7fe4281a6340 2021-04-09T18:24:50.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.831 7fe4267fc700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe408018930 0x7fe40801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:50.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.831 7fe4267fc700 1 -- 172.21.15.32:0/1796737734 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fe41c0326e0 con 0x7fe4281a6340 2021-04-09T18:24:50.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.831 7fe42de3d700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe408018930 0x7fe40801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.831 7fe42de3d700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe408018930 0x7fe40801ae10 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:50.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.850 7fe430021700 1 -- 172.21.15.32:0/1796737734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4281b7070 con 0x7fe4281a6340 2021-04-09T18:24:50.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:50.854 7fe4267fc700 1 -- 172.21.15.32:0/1796737734 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe41c006b00 con 0x7fe4281a6340 2021-04-09T18:24:51.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.044 7fe430021700 1 -- 172.21.15.32:0/1796737734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe42810d480 con 0x7fe4281a6340 2021-04-09T18:24:51.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.045 7fe4267fc700 1 -- 172.21.15.32:0/1796737734 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+11153 (secure 0 0 0) 0x7fe41c00a000 con 0x7fe4281a6340 2021-04-09T18:24:51.109 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":111,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:24:50.192503","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:24:51.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 -- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe408018930 msgr2=0x7fe40801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe408018930 0x7fe40801ae10 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 -- 172.21.15.32:0/1796737734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4281a6340 msgr2=0x7fe4281a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4281a6340 0x7fe4281a6740 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 rx=0x7fe41c006f40 tx=0x7fe41c00b990).stop 2021-04-09T18:24:51.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 -- 172.21.15.32:0/1796737734 shutdown_connections 2021-04-09T18:24:51.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe408018930 0x7fe40801ae10 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4281a6340 0x7fe4281a6740 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4280abaa0 0x7fe4280abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 --2- 172.21.15.32:0/1796737734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4281a5210 0x7fe4281a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 -- 172.21.15.32:0/1796737734 >> 172.21.15.32:0/1796737734 conn(0x7fe4281a99c0 msgr2=0x7fe42817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:51.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 -- 172.21.15.32:0/1796737734 shutdown_connections 2021-04-09T18:24:51.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.047 7fe430021700 1 -- 172.21.15.32:0/1796737734 wait complete. 2021-04-09T18:24:51.121 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:24:51.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:24:51.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.209 7f5970465700 1 Processor -- start 2021-04-09T18:24:51.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.210 7f5970465700 1 -- start start 2021-04-09T18:24:51.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.210 7f5970465700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f596810d010 0x7f59680d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.210 7f5970465700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59680d1da0 0x7f59680d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.210 7f5970465700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596810d480 con 0x7f59680d82c0 2021-04-09T18:24:51.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.210 7f5970465700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59680d9220 con 0x7f596810d010 2021-04-09T18:24:51.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.210 7f5970465700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59680d86c0 con 0x7f59680d1da0 2021-04-09T18:24:51.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.210 7f596e281700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f596810d010 0x7f59680d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596e281700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f596810d010 0x7f59680d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51624/0 (socket says 172.21.15.32:51624) 2021-04-09T18:24:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596e281700 1 -- 172.21.15.32:0/1842001030 learned_addr learned my addr 172.21.15.32:0/1842001030 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596da80700 1 --2- 172.21.15.32:0/1842001030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59680d1da0 0x7f59680d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596e281700 1 -- 172.21.15.32:0/1842001030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59680d1da0 msgr2=0x7f59680d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596e281700 1 --2- 172.21.15.32:0/1842001030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59680d1da0 0x7f59680d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596e281700 1 -- 172.21.15.32:0/1842001030 >> v1:172.21.15.86:6789/0 conn(0x7f59680d82c0 legacy=0x7f596810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596e281700 1 -- 172.21.15.32:0/1842001030 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596810e1f0 con 0x7f596810d010 2021-04-09T18:24:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596d27f700 1 -- 172.21.15.32:0/1842001030 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057404493 0 0) 0x7f5960000d30 con 0x7f59680d82c0 2021-04-09T18:24:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596e281700 1 --2- 172.21.15.32:0/1842001030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f596810d010 0x7f59680d1780 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7f5964006700 tx=0x7f5964009b00).ready entity=mon.0 client_cookie=f76772e573b676c0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596d27f700 1 -- 172.21.15.32:0/1842001030 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f596400c0c0 con 0x7f596810d010 2021-04-09T18:24:51.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596d27f700 1 -- 172.21.15.32:0/1842001030 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596400c400 con 0x7f596810d010 2021-04-09T18:24:51.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f596d27f700 1 -- 172.21.15.32:0/1842001030 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f596400c8e0 con 0x7f596810d010 2021-04-09T18:24:51.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f5970465700 1 -- 172.21.15.32:0/1842001030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f596810d010 msgr2=0x7f59680d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.211 7f5970465700 1 --2- 172.21.15.32:0/1842001030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f596810d010 0x7f59680d1780 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7f5964006700 tx=0x7f5964009b00).stop 2021-04-09T18:24:51.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.212 7f5970465700 1 -- 172.21.15.32:0/1842001030 shutdown_connections 2021-04-09T18:24:51.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.212 7f5970465700 1 --2- 172.21.15.32:0/1842001030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59680d1da0 0x7f59680d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.212 7f5970465700 1 --2- 172.21.15.32:0/1842001030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f596810d010 0x7f59680d1780 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.212 7f5970465700 1 -- 172.21.15.32:0/1842001030 >> 172.21.15.32:0/1842001030 conn(0x7f59680ce5f0 msgr2=0x7f59680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:51.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.212 7f5970465700 1 -- 172.21.15.32:0/1842001030 shutdown_connections 2021-04-09T18:24:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.212 7f5970465700 1 -- 172.21.15.32:0/1842001030 wait complete. 2021-04-09T18:24:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.232 7f5970465700 1 Processor -- start 2021-04-09T18:24:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.232 7f5970465700 1 -- start start 2021-04-09T18:24:51.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f5970465700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59680b1680 0x7f59680b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f5970465700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f596809e330 0x7f59680b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f5970465700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59680b2190 0x7f596809f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f5970465700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596809e730 con 0x7f59680b1680 2021-04-09T18:24:51.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f5970465700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596809f6c0 con 0x7f59680b2190 2021-04-09T18:24:51.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f5970465700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596809f8e0 con 0x7f596809e330 2021-04-09T18:24:51.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596e281700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59680b1680 0x7f59680b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596e281700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59680b1680 0x7f59680b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51630/0 (socket says 172.21.15.32:51630) 2021-04-09T18:24:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596e281700 1 -- 172.21.15.32:0/3199572617 learned_addr learned my addr 172.21.15.32:0/3199572617 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596ea82700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59680b2190 0x7f596809f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596da80700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f596809e330 0x7f59680b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596e281700 1 -- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f596809e330 msgr2=0x7f59680b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596e281700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f596809e330 0x7f59680b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596e281700 1 -- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59680b2190 msgr2=0x7f596809f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596e281700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59680b2190 0x7f596809f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596e281700 1 -- 172.21.15.32:0/3199572617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596809fb00 con 0x7f59680b1680 2021-04-09T18:24:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.233 7f596e281700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59680b1680 0x7f59680b0e30 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7f5964006f20 tx=0x7f596400b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.234 7f595effd700 1 -- 172.21.15.32:0/3199572617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59640090d0 con 0x7f59680b1680 2021-04-09T18:24:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.234 7f595effd700 1 -- 172.21.15.32:0/3199572617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596400bd00 con 0x7f59680b1680 2021-04-09T18:24:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.234 7f5970465700 1 -- 172.21.15.32:0/3199572617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596809fb00 con 0x7f59680b1680 2021-04-09T18:24:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.234 7f595effd700 1 -- 172.21.15.32:0/3199572617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5964005560 con 0x7f59680b1680 2021-04-09T18:24:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.234 7f5970465700 1 -- 172.21.15.32:0/3199572617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596809fb00 con 0x7f59680b1680 2021-04-09T18:24:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.235 7f595effd700 1 -- 172.21.15.32:0/3199572617 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f596400a000 con 0x7f59680b1680 2021-04-09T18:24:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.235 7f595effd700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59500188c0 0x7f595001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.235 7f595effd700 1 -- 172.21.15.32:0/3199572617 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f5964032450 con 0x7f59680b1680 2021-04-09T18:24:51.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.235 7f596da80700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59500188c0 0x7f595001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.235 7f596da80700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59500188c0 0x7f595001adc0 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:51.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.254 7f5970465700 1 -- 172.21.15.32:0/3199572617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596810d480 con 0x7f59680b1680 2021-04-09T18:24:51.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.258 7f595effd700 1 -- 172.21.15.32:0/3199572617 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f59640090d0 con 0x7f59680b1680 2021-04-09T18:24:51.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.439 7f5970465700 1 -- 172.21.15.32:0/3199572617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f596810d480 con 0x7f59680b1680 2021-04-09T18:24:51.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.440 7f595effd700 1 -- 172.21.15.32:0/3199572617 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v111) v1 ==== 92+0+10 (secure 0 0 0) 0x7f596400bd00 con 0x7f59680b1680 2021-04-09T18:24:51.504 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:24:51.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 -- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59500188c0 msgr2=0x7f595001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59500188c0 0x7f595001adc0 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 -- 172.21.15.32:0/3199572617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59680b1680 msgr2=0x7f59680b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59680b1680 0x7f59680b0e30 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7f5964006f20 tx=0x7f596400b910).stop 2021-04-09T18:24:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 -- 172.21.15.32:0/3199572617 shutdown_connections 2021-04-09T18:24:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59500188c0 0x7f595001adc0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59680b2190 0x7f596809f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f596809e330 0x7f59680b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 --2- 172.21.15.32:0/3199572617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59680b1680 0x7f59680b0e30 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 -- 172.21.15.32:0/3199572617 >> 172.21.15.32:0/3199572617 conn(0x7f59680c4290 msgr2=0x7f59680c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 -- 172.21.15.32:0/3199572617 shutdown_connections 2021-04-09T18:24:51.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.442 7f5970465700 1 -- 172.21.15.32:0/3199572617 wait complete. 2021-04-09T18:24:51.517 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:24:51.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.600 7f44c95c5700 1 Processor -- start 2021-04-09T18:24:51.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.600 7f44c95c5700 1 -- start start 2021-04-09T18:24:51.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c95c5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44c410d010 0x7f44c40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c95c5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f44c40d8490 0x7f44c410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c95c5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c410d480 con 0x7f44c40d1da0 2021-04-09T18:24:51.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c95c5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c40d8d30 con 0x7f44c410d010 2021-04-09T18:24:51.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c95c5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c40d9220 con 0x7f44c40d8490 2021-04-09T18:24:51.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c32bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44c410d010 0x7f44c40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c2abe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f44c40d1da0 0x7f44c40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33000/0 (socket says 172.21.15.32:33000) 2021-04-09T18:24:51.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c2abe700 1 -- 172.21.15.32:0/1762271540 learned_addr learned my addr 172.21.15.32:0/1762271540 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:51.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c3ac0700 1 --2- 172.21.15.32:0/1762271540 >> v2:172.21.15.86:3300/0 conn(0x7f44c40d8490 0x7f44c410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.601 7f44c22bd700 1 -- 172.21.15.32:0/1762271540 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438531569 0 0) 0x7f44b00009b0 con 0x7f44c40d1da0 2021-04-09T18:24:51.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c22bd700 1 -- 172.21.15.32:0/1762271540 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a8002fe0 con 0x7f44c40d1da0 2021-04-09T18:24:51.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c32bf700 1 -- 172.21.15.32:0/1762271540 >> v2:172.21.15.86:3300/0 conn(0x7f44c40d8490 msgr2=0x7f44c410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:51.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c32bf700 1 --2- 172.21.15.32:0/1762271540 >> v2:172.21.15.86:3300/0 conn(0x7f44c40d8490 0x7f44c410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c32bf700 1 -- 172.21.15.32:0/1762271540 >> v1:172.21.15.32:6789/0 conn(0x7f44c40d1da0 legacy=0x7f44c40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c32bf700 1 -- 172.21.15.32:0/1762271540 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c410e020 con 0x7f44c410d010 2021-04-09T18:24:51.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c3ac0700 1 --2- 172.21.15.32:0/1762271540 >> v2:172.21.15.86:3300/0 conn(0x7f44c40d8490 0x7f44c410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:51.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c32bf700 1 --2- 172.21.15.32:0/1762271540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c410d010 0x7f44c40d1780 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 rx=0x7f44b4006650 tx=0x7f44b4009a80).ready entity=mon.0 client_cookie=d651a1863943f0b5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:51.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c22bd700 1 -- 172.21.15.32:0/1762271540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b40009b0 con 0x7f44c410d010 2021-04-09T18:24:51.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c22bd700 1 -- 172.21.15.32:0/1762271540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b400c0c0 con 0x7f44c410d010 2021-04-09T18:24:51.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c22bd700 1 -- 172.21.15.32:0/1762271540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b40009b0 con 0x7f44c410d010 2021-04-09T18:24:51.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c95c5700 1 -- 172.21.15.32:0/1762271540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c410d010 msgr2=0x7f44c40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.602 7f44c95c5700 1 --2- 172.21.15.32:0/1762271540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c410d010 0x7f44c40d1780 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 rx=0x7f44b4006650 tx=0x7f44b4009a80).stop 2021-04-09T18:24:51.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.603 7f44c95c5700 1 -- 172.21.15.32:0/1762271540 shutdown_connections 2021-04-09T18:24:51.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.603 7f44c95c5700 1 --2- 172.21.15.32:0/1762271540 >> v2:172.21.15.86:3300/0 conn(0x7f44c40d8490 0x7f44c410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.603 7f44c95c5700 1 --2- 172.21.15.32:0/1762271540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c410d010 0x7f44c40d1780 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.603 7f44c95c5700 1 -- 172.21.15.32:0/1762271540 >> 172.21.15.32:0/1762271540 conn(0x7f44c40ce5f0 msgr2=0x7f44c40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:51.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.603 7f44c95c5700 1 -- 172.21.15.32:0/1762271540 shutdown_connections 2021-04-09T18:24:51.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.603 7f44c95c5700 1 -- 172.21.15.32:0/1762271540 wait complete. 2021-04-09T18:24:51.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c95c5700 1 Processor -- start 2021-04-09T18:24:51.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c95c5700 1 -- start start 2021-04-09T18:24:51.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c95c5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44c40a90c0 0x7f44c41a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c95c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c41a76d0 0x7f44c41a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c95c5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44c417bce0 0x7f44c417c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c95c5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c41a7c60 con 0x7f44c41a76d0 2021-04-09T18:24:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c95c5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c41a8e20 con 0x7f44c40a90c0 2021-04-09T18:24:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c95c5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c417c700 con 0x7f44c417bce0 2021-04-09T18:24:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c2abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c41a76d0 0x7f44c41a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c2abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c41a76d0 0x7f44c41a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51648/0 (socket says 172.21.15.32:51648) 2021-04-09T18:24:51.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.623 7f44c2abe700 1 -- 172.21.15.32:0/3146447614 learned_addr learned my addr 172.21.15.32:0/3146447614 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c32bf700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44c40a90c0 0x7f44c41a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c3ac0700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44c417bce0 0x7f44c417c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c2abe700 1 -- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44c417bce0 msgr2=0x7f44c417c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c2abe700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44c417bce0 0x7f44c417c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c2abe700 1 -- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44c40a90c0 msgr2=0x7f44c41a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c2abe700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44c40a90c0 0x7f44c41a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c2abe700 1 -- 172.21.15.32:0/3146447614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c417c920 con 0x7f44c41a76d0 2021-04-09T18:24:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c2abe700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c41a76d0 0x7f44c41a8800 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f44b000a4c0 tx=0x7f44b000a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:51.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44bbfff700 1 -- 172.21.15.32:0/3146447614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b000ac50 con 0x7f44c41a76d0 2021-04-09T18:24:51.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44bbfff700 1 -- 172.21.15.32:0/3146447614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b0004140 con 0x7f44c41a76d0 2021-04-09T18:24:51.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c417c920 con 0x7f44c41a76d0 2021-04-09T18:24:51.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44bbfff700 1 -- 172.21.15.32:0/3146447614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b0004620 con 0x7f44c41a76d0 2021-04-09T18:24:51.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.624 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c417c920 con 0x7f44c41a76d0 2021-04-09T18:24:51.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.625 7f44bbfff700 1 -- 172.21.15.32:0/3146447614 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f44b0004320 con 0x7f44c41a76d0 2021-04-09T18:24:51.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.625 7f44bbfff700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a00188c0 0x7f44a001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:51.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.625 7f44bbfff700 1 -- 172.21.15.32:0/3146447614 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f44b0032500 con 0x7f44c41a76d0 2021-04-09T18:24:51.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.625 7f44c32bf700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a00188c0 0x7f44a001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:51.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.626 7f44c32bf700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a00188c0 0x7f44a001adc0 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.645 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c41b7010 con 0x7f44c41a76d0 2021-04-09T18:24:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.649 7f44bbfff700 1 -- 172.21.15.32:0/3146447614 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f44b00009b0 con 0x7f44c41a76d0 2021-04-09T18:24:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.827 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f44c410d480 con 0x7f44c41a76d0 2021-04-09T18:24:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.827 7f44bbfff700 1 -- 172.21.15.32:0/3146447614 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v111) v1 ==== 84+0+64 (secure 0 0 0) 0x7f44b00042e0 con 0x7f44c41a76d0 2021-04-09T18:24:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:51.892 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:24:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:24:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:24:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:24:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:24:51.893 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:24:51.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a00188c0 msgr2=0x7f44a001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a00188c0 0x7f44a001adc0 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c41a76d0 msgr2=0x7f44c41a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:51.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c41a76d0 0x7f44c41a8800 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f44b000a4c0 tx=0x7f44b000a860).stop 2021-04-09T18:24:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 shutdown_connections 2021-04-09T18:24:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a00188c0 0x7f44a001adc0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44c41a76d0 0x7f44c41a8800 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44c417bce0 0x7f44c417c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 --2- 172.21.15.32:0/3146447614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44c40a90c0 0x7f44c41a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 >> 172.21.15.32:0/3146447614 conn(0x7f44c41acae0 msgr2=0x7f44c417ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 shutdown_connections 2021-04-09T18:24:51.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.830 7f44c95c5700 1 -- 172.21.15.32:0/3146447614 wait complete. 2021-04-09T18:24:51.904 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:24:51.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:24:52.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.993 7f4c3bf61700 1 Processor -- start 2021-04-09T18:24:52.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.993 7f4c3bf61700 1 -- start start 2021-04-09T18:24:52.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3bf61700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c340d7ca0 0x7f4c340d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:52.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3bf61700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4c3410da00 0x7f4c3410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:52.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3bf61700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3410d480 con 0x7f4c3410d010 2021-04-09T18:24:52.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3bf61700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c340d9220 con 0x7f4c3410da00 2021-04-09T18:24:52.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3bf61700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c34112100 con 0x7f4c340d7ca0 2021-04-09T18:24:52.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3a57e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4c3410da00 0x7f4c3410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:52.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3957c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c340d7ca0 0x7f4c340d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:52.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c39d7d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4c3410d010 0x7f4c340d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:59168/0 (socket says 172.21.15.32:59168) 2021-04-09T18:24:52.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c39d7d700 1 -- 172.21.15.32:0/2340190551 learned_addr learned my addr 172.21.15.32:0/2340190551 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:52.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3a57e700 1 -- 172.21.15.32:0/2340190551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c340d7ca0 msgr2=0x7f4c340d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:52.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3a57e700 1 --2- 172.21.15.32:0/2340190551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c340d7ca0 0x7f4c340d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:52.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3a57e700 1 -- 172.21.15.32:0/2340190551 >> v1:172.21.15.86:6789/0 conn(0x7f4c3410d010 legacy=0x7f4c340d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:52.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3a57e700 1 -- 172.21.15.32:0/2340190551 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c340d86c0 con 0x7f4c3410da00 2021-04-09T18:24:52.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c3957c700 1 --2- 172.21.15.32:0/2340190551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c340d7ca0 0x7f4c340d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:52.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.994 7f4c38d7b700 1 -- 172.21.15.32:0/2340190551 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487546309 0 0) 0x7f4c300010c0 con 0x7f4c3410d010 2021-04-09T18:24:52.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.995 7f4c3a57e700 1 --2- 172.21.15.32:0/2340190551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c3410da00 0x7f4c3410de00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f4c2c006e50 tx=0x7f4c2c009e70).ready entity=mon.1 client_cookie=5b0b1c7380c6b7df server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:52.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.995 7f4c38d7b700 1 -- 172.21.15.32:0/2340190551 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c2c006a20 con 0x7f4c3410da00 2021-04-09T18:24:52.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.995 7f4c38d7b700 1 -- 172.21.15.32:0/2340190551 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c006790 con 0x7f4c3410da00 2021-04-09T18:24:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.995 7f4c38d7b700 1 -- 172.21.15.32:0/2340190551 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c2c00b700 con 0x7f4c3410da00 2021-04-09T18:24:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.995 7f4c3bf61700 1 -- 172.21.15.32:0/2340190551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c3410da00 msgr2=0x7f4c3410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.995 7f4c3bf61700 1 --2- 172.21.15.32:0/2340190551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c3410da00 0x7f4c3410de00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f4c2c006e50 tx=0x7f4c2c009e70).stop 2021-04-09T18:24:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.996 7f4c3bf61700 1 -- 172.21.15.32:0/2340190551 shutdown_connections 2021-04-09T18:24:52.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.996 7f4c3bf61700 1 --2- 172.21.15.32:0/2340190551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c3410da00 0x7f4c3410de00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:52.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.996 7f4c3bf61700 1 --2- 172.21.15.32:0/2340190551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c340d7ca0 0x7f4c340d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:52.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.996 7f4c3bf61700 1 -- 172.21.15.32:0/2340190551 >> 172.21.15.32:0/2340190551 conn(0x7f4c340ce5f0 msgr2=0x7f4c340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:52.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.996 7f4c3bf61700 1 -- 172.21.15.32:0/2340190551 shutdown_connections 2021-04-09T18:24:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:51.996 7f4c3bf61700 1 -- 172.21.15.32:0/2340190551 wait complete. 2021-04-09T18:24:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.016 7f4c3bf61700 1 Processor -- start 2021-04-09T18:24:52.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.016 7f4c3bf61700 1 -- start start 2021-04-09T18:24:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.016 7f4c3bf61700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c340c3c00 0x7f4c3409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.016 7f4c3bf61700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c34095260 0x7f4c341a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.016 7f4c3bf61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c341a1270 0x7f4c340b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.016 7f4c3bf61700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3409cda0 con 0x7f4c341a1270 2021-04-09T18:24:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.016 7f4c3bf61700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c340c3680 con 0x7f4c340c3c00 2021-04-09T18:24:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.016 7f4c3bf61700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c34095660 con 0x7f4c34095260 2021-04-09T18:24:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3a57e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c341a1270 0x7f4c340b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3a57e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c341a1270 0x7f4c340b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51662/0 (socket says 172.21.15.32:51662) 2021-04-09T18:24:52.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3a57e700 1 -- 172.21.15.32:0/4288288807 learned_addr learned my addr 172.21.15.32:0/4288288807 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:52.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3957c700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c34095260 0x7f4c341a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:52.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c39d7d700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c340c3c00 0x7f4c3409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:52.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3a57e700 1 -- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c34095260 msgr2=0x7f4c341a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:52.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3a57e700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c34095260 0x7f4c341a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:52.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3a57e700 1 -- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c340c3c00 msgr2=0x7f4c3409c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:52.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3a57e700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c340c3c00 0x7f4c3409c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:52.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3a57e700 1 -- 172.21.15.32:0/4288288807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c341a1670 con 0x7f4c341a1270 2021-04-09T18:24:52.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3a57e700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c341a1270 0x7f4c340b0cf0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7f4c2c006e60 tx=0x7f4c2c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:52.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c267fc700 1 -- 172.21.15.32:0/4288288807 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c2c009130 con 0x7f4c341a1270 2021-04-09T18:24:52.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c267fc700 1 -- 172.21.15.32:0/4288288807 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c0093f0 con 0x7f4c341a1270 2021-04-09T18:24:52.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c341a1670 con 0x7f4c341a1270 2021-04-09T18:24:52.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c267fc700 1 -- 172.21.15.32:0/4288288807 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c2c001390 con 0x7f4c341a1270 2021-04-09T18:24:52.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.017 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c341a1670 con 0x7f4c341a1270 2021-04-09T18:24:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.018 7f4c267fc700 1 -- 172.21.15.32:0/4288288807 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4c2c0056b0 con 0x7f4c341a1270 2021-04-09T18:24:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.018 7f4c267fc700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c1c018930 0x7f4c1c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.018 7f4c267fc700 1 -- 172.21.15.32:0/4288288807 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f4c2c032aa0 con 0x7f4c341a1270 2021-04-09T18:24:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.018 7f4c39d7d700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c1c018930 0x7f4c1c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:52.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.018 7f4c39d7d700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c1c018930 0x7f4c1c01ae10 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:52.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.038 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3410d480 con 0x7f4c341a1270 2021-04-09T18:24:52.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.041 7f4c267fc700 1 -- 172.21.15.32:0/4288288807 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4c2c00be30 con 0x7f4c341a1270 2021-04-09T18:24:52.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:52.226 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4c3410d480 con 0x7f4c341a1270 2021-04-09T18:24:53.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.206 7f4c267fc700 1 -- 172.21.15.32:0/4288288807 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v112) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4c2c0056b0 con 0x7f4c341a1270 2021-04-09T18:24:53.306 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:24:53.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c1c018930 msgr2=0x7f4c1c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:53.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c1c018930 0x7f4c1c01ae10 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:53.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c341a1270 msgr2=0x7f4c340b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:53.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c341a1270 0x7f4c340b0cf0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7f4c2c006e60 tx=0x7f4c2c00bcf0).stop 2021-04-09T18:24:53.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 shutdown_connections 2021-04-09T18:24:53.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c1c018930 0x7f4c1c01ae10 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:53.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c341a1270 0x7f4c340b0cf0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:53.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c34095260 0x7f4c341a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:53.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 --2- 172.21.15.32:0/4288288807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c340c3c00 0x7f4c3409c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:53.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 >> 172.21.15.32:0/4288288807 conn(0x7f4c340975f0 msgr2=0x7f4c34096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:53.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 shutdown_connections 2021-04-09T18:24:53.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.245 7f4c3bf61700 1 -- 172.21.15.32:0/4288288807 wait complete. 2021-04-09T18:24:53.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:24:53.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.411 7f6deb947700 1 Processor -- start 2021-04-09T18:24:53.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.411 7f6deb947700 1 -- start start 2021-04-09T18:24:53.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.412 7f6deb947700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6de410d010 0x7f6de40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:53.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.412 7f6deb947700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de410d480 con 0x7f6de40d8490 2021-04-09T18:24:53.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.412 7f6deb947700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de410e1f0 con 0x7f6de40d1da0 2021-04-09T18:24:53.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.412 7f6deb947700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de40d8d30 con 0x7f6de410d010 2021-04-09T18:24:53.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.412 7f6de9763700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6de410d010 0x7f6de40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:53.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.412 7f6de9763700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6de410d010 0x7f6de40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51666/0 (socket says 172.21.15.32:51666) 2021-04-09T18:24:53.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.412 7f6de9f64700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6de40d8490 0x7f6de410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33026/0 (socket says 172.21.15.32:33026) 2021-04-09T18:24:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.412 7f6de9f64700 1 -- 172.21.15.32:0/3600776213 learned_addr learned my addr 172.21.15.32:0/3600776213 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.413 7f6de9763700 1 -- 172.21.15.32:0/3600776213 >> v1:172.21.15.86:6789/0 conn(0x7f6de40d1da0 legacy=0x7f6de40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.413 7f6de9763700 1 -- 172.21.15.32:0/3600776213 >> v1:172.21.15.32:6789/0 conn(0x7f6de40d8490 legacy=0x7f6de410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.413 7f6de9763700 1 -- 172.21.15.32:0/3600776213 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de40d9220 con 0x7f6de410d010 2021-04-09T18:24:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.413 7f6de9763700 1 --2- 172.21.15.32:0/3600776213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de410d010 0x7f6de40d1780 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 rx=0x7f6de0006650 tx=0x7f6de0009a80).ready entity=mon.0 client_cookie=e6b14e3c6a4d51c2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:53.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.413 7f6ddbfff700 1 -- 172.21.15.32:0/3600776213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6de00009b0 con 0x7f6de410d010 2021-04-09T18:24:53.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.413 7f6ddbfff700 1 -- 172.21.15.32:0/3600776213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de000c0c0 con 0x7f6de410d010 2021-04-09T18:24:53.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.413 7f6ddbfff700 1 -- 172.21.15.32:0/3600776213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6de000c510 con 0x7f6de410d010 2021-04-09T18:24:53.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.413 7f6deb947700 1 -- 172.21.15.32:0/3600776213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de410d010 msgr2=0x7f6de40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:53.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.413 7f6deb947700 1 --2- 172.21.15.32:0/3600776213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de410d010 0x7f6de40d1780 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 rx=0x7f6de0006650 tx=0x7f6de0009a80).stop 2021-04-09T18:24:53.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.414 7f6deb947700 1 -- 172.21.15.32:0/3600776213 shutdown_connections 2021-04-09T18:24:53.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.414 7f6deb947700 1 --2- 172.21.15.32:0/3600776213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de410d010 0x7f6de40d1780 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:53.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.414 7f6deb947700 1 -- 172.21.15.32:0/3600776213 >> 172.21.15.32:0/3600776213 conn(0x7f6de40ce5f0 msgr2=0x7f6de40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:53.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.414 7f6deb947700 1 -- 172.21.15.32:0/3600776213 shutdown_connections 2021-04-09T18:24:53.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.414 7f6deb947700 1 -- 172.21.15.32:0/3600776213 wait complete. 2021-04-09T18:24:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.434 7f6deb947700 1 Processor -- start 2021-04-09T18:24:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.434 7f6deb947700 1 -- start start 2021-04-09T18:24:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6deb947700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de41c2850 0x7f6de40bc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:53.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6deb947700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de408cc40 0x7f6de408d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:53.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6deb947700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a5680 0x7f6de41a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:53.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6deb947700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de40bc8a0 con 0x7f6de408cc40 2021-04-09T18:24:53.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6deb947700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de408d660 con 0x7f6de41c2850 2021-04-09T18:24:53.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6deb947700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de41a60a0 con 0x7f6de41a5680 2021-04-09T18:24:53.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de8f62700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de408cc40 0x7f6de408d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:53.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de8f62700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de408cc40 0x7f6de408d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51674/0 (socket says 172.21.15.32:51674) 2021-04-09T18:24:53.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de8f62700 1 -- 172.21.15.32:0/2811615796 learned_addr learned my addr 172.21.15.32:0/2811615796 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:53.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de9763700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de41c2850 0x7f6de40bc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:53.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de9f64700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a5680 0x7f6de41a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:53.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de8f62700 1 -- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a5680 msgr2=0x7f6de41a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:53.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de8f62700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a5680 0x7f6de41a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:53.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de8f62700 1 -- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de41c2850 msgr2=0x7f6de40bc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:53.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de8f62700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de41c2850 0x7f6de40bc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:53.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de8f62700 1 -- 172.21.15.32:0/2811615796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de41a6490 con 0x7f6de408cc40 2021-04-09T18:24:53.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6de8f62700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de408cc40 0x7f6de408d040 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7f6dd400b5b0 tx=0x7f6dd400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:53.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6dd9ffb700 1 -- 172.21.15.32:0/2811615796 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6dd400bd40 con 0x7f6de408cc40 2021-04-09T18:24:53.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6dd9ffb700 1 -- 172.21.15.32:0/2811615796 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd4006240 con 0x7f6de408cc40 2021-04-09T18:24:53.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.435 7f6deb947700 1 -- 172.21.15.32:0/2811615796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de41a6490 con 0x7f6de408cc40 2021-04-09T18:24:53.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.436 7f6dd9ffb700 1 -- 172.21.15.32:0/2811615796 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6dd4005690 con 0x7f6de408cc40 2021-04-09T18:24:53.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.436 7f6deb947700 1 -- 172.21.15.32:0/2811615796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de41a6490 con 0x7f6de408cc40 2021-04-09T18:24:53.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.436 7f6dd9ffb700 1 -- 172.21.15.32:0/2811615796 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6dd40054a0 con 0x7f6de408cc40 2021-04-09T18:24:53.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.436 7f6dd9ffb700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:53.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.436 7f6dd9ffb700 1 -- 172.21.15.32:0/2811615796 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5111+0+0 (secure 0 0 0) 0x7f6dd4032860 con 0x7f6de408cc40 2021-04-09T18:24:53.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.436 7f6de9763700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:53.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.437 7f6de9763700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401ae10 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:53.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.456 7f6deb947700 1 -- 172.21.15.32:0/2811615796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de41a4770 con 0x7f6de408cc40 2021-04-09T18:24:53.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.460 7f6dd9ffb700 1 -- 172.21.15.32:0/2811615796 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6dd4000c60 con 0x7f6de408cc40 2021-04-09T18:24:53.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:53.648 7f6deb947700 1 -- 172.21.15.32:0/2811615796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f6de4112070 con 0x7f6de408cc40 2021-04-09T18:24:54.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.518 7f6dd9ffb700 1 -- 172.21.15.32:0/2811615796 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v117) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6dd40054a0 con 0x7f6de408cc40 2021-04-09T18:24:54.628 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:24:54.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 -- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 msgr2=0x7f6dc401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401ae10 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:54.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 -- 172.21.15.32:0/2811615796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de408cc40 msgr2=0x7f6de408d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:54.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de408cc40 0x7f6de408d040 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7f6dd400b5b0 tx=0x7f6dd400b950).stop 2021-04-09T18:24:54.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 -- 172.21.15.32:0/2811615796 shutdown_connections 2021-04-09T18:24:54.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401ae10 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:54.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de408cc40 0x7f6de408d040 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:54.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a5680 0x7f6de41a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:54.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 --2- 172.21.15.32:0/2811615796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de41c2850 0x7f6de40bc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:54.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 -- 172.21.15.32:0/2811615796 >> 172.21.15.32:0/2811615796 conn(0x7f6de40bca90 msgr2=0x7f6de40b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:54.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 -- 172.21.15.32:0/2811615796 shutdown_connections 2021-04-09T18:24:54.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.567 7f6deb947700 1 -- 172.21.15.32:0/2811615796 wait complete. 2021-04-09T18:24:54.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:24:54.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.739 7fe765121700 1 Processor -- start 2021-04-09T18:24:54.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.739 7fe765121700 1 -- start start 2021-04-09T18:24:54.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.740 7fe765121700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe7600d7cc0 0x7fe7600d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:54.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.740 7fe765121700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76010d500 con 0x7fe76010da80 2021-04-09T18:24:54.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.740 7fe765121700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600d0330 con 0x7fe76010d050 2021-04-09T18:24:54.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.740 7fe765121700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7600d9260 con 0x7fe7600d7cc0 2021-04-09T18:24:54.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.740 7fe75f59e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe76010da80 0x7fe76010de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33044/0 (socket says 172.21.15.32:33044) 2021-04-09T18:24:54.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.740 7fe75f59e700 1 -- 172.21.15.32:0/1531606840 learned_addr learned my addr 172.21.15.32:0/1531606840 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:54.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75e59c700 1 --2- 172.21.15.32:0/1531606840 >> v2:172.21.15.86:3300/0 conn(0x7fe7600d7cc0 0x7fe7600d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:54.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75dd9b700 1 -- 172.21.15.32:0/1531606840 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742963264 0 0) 0x7fe758000d30 con 0x7fe76010da80 2021-04-09T18:24:54.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75dd9b700 1 -- 172.21.15.32:0/1531606840 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c002fe0 con 0x7fe76010da80 2021-04-09T18:24:54.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75dd9b700 1 -- 172.21.15.32:0/1531606840 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422224384 0 0) 0x7fe750000c60 con 0x7fe76010d050 2021-04-09T18:24:54.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75dd9b700 1 -- 172.21.15.32:0/1531606840 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c003210 con 0x7fe76010d050 2021-04-09T18:24:54.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75e59c700 1 -- 172.21.15.32:0/1531606840 >> v1:172.21.15.86:6789/0 conn(0x7fe76010d050 legacy=0x7fe7600cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:54.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75dd9b700 1 -- 172.21.15.32:0/1531606840 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 547406754 0 0) 0x7fe750001060 con 0x7fe76010d050 2021-04-09T18:24:54.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75e59c700 1 -- 172.21.15.32:0/1531606840 >> v1:172.21.15.32:6789/0 conn(0x7fe76010da80 legacy=0x7fe76010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:54.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75e59c700 1 -- 172.21.15.32:0/1531606840 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe760112140 con 0x7fe7600d7cc0 2021-04-09T18:24:54.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75e59c700 1 --2- 172.21.15.32:0/1531606840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7600d7cc0 0x7fe7600d80c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fe748006650 tx=0x7fe748009a00).ready entity=mon.1 client_cookie=4d63d48b74a028ce server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:54.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75dd9b700 1 -- 172.21.15.32:0/1531606840 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe748009df0 con 0x7fe7600d7cc0 2021-04-09T18:24:54.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75dd9b700 1 -- 172.21.15.32:0/1531606840 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7480009b0 con 0x7fe7600d7cc0 2021-04-09T18:24:54.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe75dd9b700 1 -- 172.21.15.32:0/1531606840 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe74800c2b0 con 0x7fe7600d7cc0 2021-04-09T18:24:54.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe765121700 1 -- 172.21.15.32:0/1531606840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7600d7cc0 msgr2=0x7fe7600d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:54.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.741 7fe765121700 1 --2- 172.21.15.32:0/1531606840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7600d7cc0 0x7fe7600d80c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fe748006650 tx=0x7fe748009a00).stop 2021-04-09T18:24:54.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.742 7fe765121700 1 -- 172.21.15.32:0/1531606840 shutdown_connections 2021-04-09T18:24:54.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.742 7fe765121700 1 --2- 172.21.15.32:0/1531606840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7600d7cc0 0x7fe7600d80c0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:54.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.742 7fe765121700 1 -- 172.21.15.32:0/1531606840 >> 172.21.15.32:0/1531606840 conn(0x7fe7600ce620 msgr2=0x7fe7600d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:54.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.742 7fe765121700 1 -- 172.21.15.32:0/1531606840 shutdown_connections 2021-04-09T18:24:54.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.742 7fe765121700 1 -- 172.21.15.32:0/1531606840 wait complete. 2021-04-09T18:24:54.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.762 7fe765121700 1 Processor -- start 2021-04-09T18:24:54.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.762 7fe765121700 1 -- start start 2021-04-09T18:24:54.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe765121700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76009b2f0 0x7fe76009b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:54.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe765121700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76009a8b0 0x7fe76009e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:54.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe765121700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76009e9c0 0x7fe76009edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:54.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe765121700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600bde80 con 0x7fe76009e9c0 2021-04-09T18:24:54.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe765121700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76009ad70 con 0x7fe76009a8b0 2021-04-09T18:24:54.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe765121700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76009bd10 con 0x7fe76009b2f0 2021-04-09T18:24:54.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75f59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76009e9c0 0x7fe76009edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:54.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75f59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76009e9c0 0x7fe76009edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51690/0 (socket says 172.21.15.32:51690) 2021-04-09T18:24:54.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75f59e700 1 -- 172.21.15.32:0/2273565388 learned_addr learned my addr 172.21.15.32:0/2273565388 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:54.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75e59c700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76009a8b0 0x7fe76009e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:54.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75ed9d700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76009b2f0 0x7fe76009b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:54.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75f59e700 1 -- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76009b2f0 msgr2=0x7fe76009b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:54.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75f59e700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76009b2f0 0x7fe76009b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:54.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75f59e700 1 -- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76009a8b0 msgr2=0x7fe76009e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:54.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75f59e700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76009a8b0 0x7fe76009e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:54.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75f59e700 1 -- 172.21.15.32:0/2273565388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76009fbd0 con 0x7fe76009e9c0 2021-04-09T18:24:54.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.763 7fe75ed9d700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76009b2f0 0x7fe76009b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:54.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.764 7fe75f59e700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76009e9c0 0x7fe76009edc0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 rx=0x7fe758009980 tx=0x7fe75800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:54.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.764 7fe7577fe700 1 -- 172.21.15.32:0/2273565388 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe75800be10 con 0x7fe76009e9c0 2021-04-09T18:24:54.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.764 7fe7577fe700 1 -- 172.21.15.32:0/2273565388 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758006080 con 0x7fe76009e9c0 2021-04-09T18:24:54.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.764 7fe765121700 1 -- 172.21.15.32:0/2273565388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76009fbd0 con 0x7fe76009e9c0 2021-04-09T18:24:54.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.764 7fe7577fe700 1 -- 172.21.15.32:0/2273565388 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe758004720 con 0x7fe76009e9c0 2021-04-09T18:24:54.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.764 7fe765121700 1 -- 172.21.15.32:0/2273565388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76009fbd0 con 0x7fe76009e9c0 2021-04-09T18:24:54.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.764 7fe7577fe700 1 -- 172.21.15.32:0/2273565388 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe758006260 con 0x7fe76009e9c0 2021-04-09T18:24:54.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.765 7fe7577fe700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7440188c0 0x7fe74401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:54.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.765 7fe7577fe700 1 -- 172.21.15.32:0/2273565388 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5461+0+0 (secure 0 0 0) 0x7fe758032a80 con 0x7fe76009e9c0 2021-04-09T18:24:54.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.765 7fe75ed9d700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7440188c0 0x7fe74401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:54.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.765 7fe75ed9d700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7440188c0 0x7fe74401adc0 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:54.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.784 7fe765121700 1 -- 172.21.15.32:0/2273565388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7600bde80 con 0x7fe76009e9c0 2021-04-09T18:24:54.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.788 7fe7577fe700 1 -- 172.21.15.32:0/2273565388 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe758000f90 con 0x7fe76009e9c0 2021-04-09T18:24:55.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.970 7fe765121700 1 -- 172.21.15.32:0/2273565388 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fe7600bde80 con 0x7fe76009e9c0 2021-04-09T18:24:55.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.974 7fe7577fe700 1 -- 172.21.15.32:0/2273565388 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 20 and data pool 21 v250) v1 ==== 189+0+0 (secure 0 0 0) 0x7fe758004720 con 0x7fe76009e9c0 2021-04-09T18:24:55.041 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 20 and data pool 21 2021-04-09T18:24:55.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.981 7fe765121700 1 -- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7440188c0 msgr2=0x7fe74401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.981 7fe765121700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7440188c0 0x7fe74401adc0 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.981 7fe765121700 1 -- 172.21.15.32:0/2273565388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76009e9c0 msgr2=0x7fe76009edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.981 7fe765121700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76009e9c0 0x7fe76009edc0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 rx=0x7fe758009980 tx=0x7fe75800bcd0).stop 2021-04-09T18:24:55.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.983 7fe765121700 1 -- 172.21.15.32:0/2273565388 shutdown_connections 2021-04-09T18:24:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.983 7fe765121700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7440188c0 0x7fe74401adc0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.983 7fe765121700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76009e9c0 0x7fe76009edc0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.983 7fe765121700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76009a8b0 0x7fe76009e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.983 7fe765121700 1 --2- 172.21.15.32:0/2273565388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76009b2f0 0x7fe76009b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.983 7fe765121700 1 -- 172.21.15.32:0/2273565388 >> 172.21.15.32:0/2273565388 conn(0x7fe7600ae6f0 msgr2=0x7fe760097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:55.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.984 7fe765121700 1 -- 172.21.15.32:0/2273565388 shutdown_connections 2021-04-09T18:24:55.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:54.985 7fe765121700 1 -- 172.21.15.32:0/2273565388 wait complete. 2021-04-09T18:24:55.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:24:55.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.144 7f4c622cf700 1 Processor -- start 2021-04-09T18:24:55.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.144 7f4c622cf700 1 -- start start 2021-04-09T18:24:55.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.144 7f4c622cf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c0d7ca0 0x7f4c5c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.144 7f4c622cf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c5c10da00 0x7f4c5c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.144 7f4c622cf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5c10d480 con 0x7f4c5c10d010 2021-04-09T18:24:55.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.144 7f4c622cf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c5c0d9220 con 0x7f4c5c10da00 2021-04-09T18:24:55.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.144 7f4c622cf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c0d86c0 con 0x7f4c5c0d7ca0 2021-04-09T18:24:55.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.144 7f4c608ec700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4c5c10da00 0x7f4c5c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.144 7f4c5bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4c5c10d010 0x7f4c5c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33064/0 (socket says 172.21.15.32:33064) 2021-04-09T18:24:55.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c5bfff700 1 -- 172.21.15.32:0/2652893306 learned_addr learned my addr 172.21.15.32:0/2652893306 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:55.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c53fff700 1 --2- 172.21.15.32:0/2652893306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c0d7ca0 0x7f4c5c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c608ec700 1 -- 172.21.15.32:0/2652893306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c0d7ca0 msgr2=0x7f4c5c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c608ec700 1 --2- 172.21.15.32:0/2652893306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c0d7ca0 0x7f4c5c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c608ec700 1 -- 172.21.15.32:0/2652893306 >> v1:172.21.15.32:6789/0 conn(0x7f4c5c10d010 legacy=0x7f4c5c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c5b7fe700 1 -- 172.21.15.32:0/2652893306 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548857463 0 0) 0x7f4c4c000ce0 con 0x7f4c5c10d010 2021-04-09T18:24:55.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c608ec700 1 -- 172.21.15.32:0/2652893306 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c0d1f70 con 0x7f4c5c10da00 2021-04-09T18:24:55.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c53fff700 1 --2- 172.21.15.32:0/2652893306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c0d7ca0 0x7f4c5c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:55.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c608ec700 1 --2- 172.21.15.32:0/2652893306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c10da00 0x7f4c5c10de00 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 rx=0x7f4c54006e30 tx=0x7f4c54009e60).ready entity=mon.0 client_cookie=f38a4de72bdb85b8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:55.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c5b7fe700 1 -- 172.21.15.32:0/2652893306 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c54006a50 con 0x7f4c5c10da00 2021-04-09T18:24:55.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c5b7fe700 1 -- 172.21.15.32:0/2652893306 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5400b300 con 0x7f4c5c10da00 2021-04-09T18:24:55.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c5b7fe700 1 -- 172.21.15.32:0/2652893306 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c5400b750 con 0x7f4c5c10da00 2021-04-09T18:24:55.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c622cf700 1 -- 172.21.15.32:0/2652893306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c10da00 msgr2=0x7f4c5c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.145 7f4c622cf700 1 --2- 172.21.15.32:0/2652893306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c10da00 0x7f4c5c10de00 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 rx=0x7f4c54006e30 tx=0x7f4c54009e60).stop 2021-04-09T18:24:55.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.146 7f4c622cf700 1 -- 172.21.15.32:0/2652893306 shutdown_connections 2021-04-09T18:24:55.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.146 7f4c622cf700 1 --2- 172.21.15.32:0/2652893306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c10da00 0x7f4c5c10de00 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.146 7f4c622cf700 1 --2- 172.21.15.32:0/2652893306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c0d7ca0 0x7f4c5c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.146 7f4c622cf700 1 -- 172.21.15.32:0/2652893306 >> 172.21.15.32:0/2652893306 conn(0x7f4c5c0ce5f0 msgr2=0x7f4c5c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:55.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.146 7f4c622cf700 1 -- 172.21.15.32:0/2652893306 shutdown_connections 2021-04-09T18:24:55.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.146 7f4c622cf700 1 -- 172.21.15.32:0/2652893306 wait complete. 2021-04-09T18:24:55.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.166 7f4c622cf700 1 Processor -- start 2021-04-09T18:24:55.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.166 7f4c622cf700 1 -- start start 2021-04-09T18:24:55.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c622cf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c5c1889c0 0x7f4c5c09a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c622cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c18e2c0 0x7f4c5c1aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c622cf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c1aaad0 0x7f4c5c0a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c622cf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c1aaed0 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c622cf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c1ab0f0 con 0x7f4c5c1889c0 2021-04-09T18:24:55.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c622cf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c0a3ed0 con 0x7f4c5c1aaad0 2021-04-09T18:24:55.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c53fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c18e2c0 0x7f4c5c1aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c53fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c18e2c0 0x7f4c5c1aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51716/0 (socket says 172.21.15.32:51716) 2021-04-09T18:24:55.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c53fff700 1 -- 172.21.15.32:0/3696510877 learned_addr learned my addr 172.21.15.32:0/3696510877 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c5bfff700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c5c1889c0 0x7f4c5c09a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c608ec700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c1aaad0 0x7f4c5c0a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c53fff700 1 -- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c1aaad0 msgr2=0x7f4c5c0a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c53fff700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c1aaad0 0x7f4c5c0a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c53fff700 1 -- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c5c1889c0 msgr2=0x7f4c5c09a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c53fff700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c5c1889c0 0x7f4c5c09a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c53fff700 1 -- 172.21.15.32:0/3696510877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c0a40f0 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c53fff700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c18e2c0 0x7f4c5c1aa4b0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 rx=0x7f4c4400b580 tx=0x7f4c4400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:55.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.167 7f4c597fa700 1 -- 172.21.15.32:0/3696510877 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c4400bd10 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.168 7f4c597fa700 1 -- 172.21.15.32:0/3696510877 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44000c90 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.168 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c0a40f0 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.168 7f4c597fa700 1 -- 172.21.15.32:0/3696510877 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c44002220 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.168 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5c0a40f0 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.168 7f4c597fa700 1 -- 172.21.15.32:0/3696510877 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4c4400a000 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.168 7f4c597fa700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c400188c0 0x7f4c4001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.168 7f4c597fa700 1 -- 172.21.15.32:0/3696510877 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4c440329e0 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.169 7f4c5bfff700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c400188c0 0x7f4c4001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.169 7f4c5bfff700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c400188c0 0x7f4c4001adc0 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:55.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.188 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5c10d480 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.192 7f4c597fa700 1 -- 172.21.15.32:0/3696510877 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4c44006bb0 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.379 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c5c10d480 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.380 7f4c597fa700 1 -- 172.21.15.32:0/3696510877 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f4c44000c90 con 0x7f4c5c18e2c0 2021-04-09T18:24:55.446 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:55.446 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":118,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:24:54.971049","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:24:52.261572","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-04-09 18:24:53.683747","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:24:55.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c400188c0 msgr2=0x7f4c4001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c400188c0 0x7f4c4001adc0 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c18e2c0 msgr2=0x7f4c5c1aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c18e2c0 0x7f4c5c1aa4b0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 rx=0x7f4c4400b580 tx=0x7f4c4400b920).stop 2021-04-09T18:24:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 shutdown_connections 2021-04-09T18:24:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4c400188c0 0x7f4c4001adc0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c5c18e2c0 0x7f4c5c1aa4b0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c5c1aaad0 0x7f4c5c0a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 --2- 172.21.15.32:0/3696510877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c5c1889c0 0x7f4c5c09a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 >> 172.21.15.32:0/3696510877 conn(0x7f4c5c1a0db0 msgr2=0x7f4c5c1a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 shutdown_connections 2021-04-09T18:24:55.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.382 7f4c622cf700 1 -- 172.21.15.32:0/3696510877 wait complete. 2021-04-09T18:24:55.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:24:55.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.542 7f867a7a1700 1 Processor -- start 2021-04-09T18:24:55.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.542 7f867a7a1700 1 -- start start 2021-04-09T18:24:55.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f867a7a1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867410d010 0x7f86740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f867a7a1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867410d480 con 0x7f86740d1da0 2021-04-09T18:24:55.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f867a7a1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867410e1f0 con 0x7f86740d8490 2021-04-09T18:24:55.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f867a7a1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867410e780 con 0x7f867410d010 2021-04-09T18:24:55.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f866bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f86740d1da0 0x7f86740d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33082/0 (socket says 172.21.15.32:33082) 2021-04-09T18:24:55.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f866bfff700 1 -- 172.21.15.32:0/673709491 learned_addr learned my addr 172.21.15.32:0/673709491 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:55.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f8673fff700 1 --2- 172.21.15.32:0/673709491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867410d010 0x7f86740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f86737fe700 1 -- 172.21.15.32:0/673709491 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170495859 0 0) 0x7f865c000ce0 con 0x7f86740d1da0 2021-04-09T18:24:55.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f86737fe700 1 -- 172.21.15.32:0/673709491 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8660002fe0 con 0x7f86740d1da0 2021-04-09T18:24:55.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f86737fe700 1 -- 172.21.15.32:0/673709491 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694326699 0 0) 0x7f866c000d30 con 0x7f86740d8490 2021-04-09T18:24:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f86737fe700 1 -- 172.21.15.32:0/673709491 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86600042c0 con 0x7f86740d8490 2021-04-09T18:24:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f8673fff700 1 -- 172.21.15.32:0/673709491 >> v1:172.21.15.86:6789/0 conn(0x7f86740d8490 legacy=0x7f867410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f8673fff700 1 -- 172.21.15.32:0/673709491 >> v1:172.21.15.32:6789/0 conn(0x7f86740d1da0 legacy=0x7f86740d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.543 7f8673fff700 1 -- 172.21.15.32:0/673709491 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86740d8d30 con 0x7f867410d010 2021-04-09T18:24:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.544 7f8673fff700 1 --2- 172.21.15.32:0/673709491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867410d010 0x7f86740d1780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f8664006650 tx=0x7f8664009a80).ready entity=mon.2 client_cookie=f6fa5301fcd9d40d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.544 7f86737fe700 1 -- 172.21.15.32:0/673709491 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86640009b0 con 0x7f867410d010 2021-04-09T18:24:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.544 7f86737fe700 1 -- 172.21.15.32:0/673709491 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866400c0c0 con 0x7f867410d010 2021-04-09T18:24:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.544 7f86737fe700 1 -- 172.21.15.32:0/673709491 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f866400c510 con 0x7f867410d010 2021-04-09T18:24:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.544 7f867a7a1700 1 -- 172.21.15.32:0/673709491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867410d010 msgr2=0x7f86740d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.544 7f867a7a1700 1 --2- 172.21.15.32:0/673709491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867410d010 0x7f86740d1780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f8664006650 tx=0x7f8664009a80).stop 2021-04-09T18:24:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.545 7f867a7a1700 1 -- 172.21.15.32:0/673709491 shutdown_connections 2021-04-09T18:24:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.545 7f867a7a1700 1 --2- 172.21.15.32:0/673709491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867410d010 0x7f86740d1780 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.545 7f867a7a1700 1 -- 172.21.15.32:0/673709491 >> 172.21.15.32:0/673709491 conn(0x7f86740ce5f0 msgr2=0x7f86740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.545 7f867a7a1700 1 -- 172.21.15.32:0/673709491 shutdown_connections 2021-04-09T18:24:55.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.545 7f867a7a1700 1 -- 172.21.15.32:0/673709491 wait complete. 2021-04-09T18:24:55.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f867a7a1700 1 Processor -- start 2021-04-09T18:24:55.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f867a7a1700 1 -- start start 2021-04-09T18:24:55.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f867a7a1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8674099920 0x7f8674179f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f867a7a1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867417a560 0x7f86741b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f867a7a1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86741b85a0 0x7f86741b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f867a7a1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867417a960 con 0x7f8674099920 2021-04-09T18:24:55.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f867a7a1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8674099d20 con 0x7f867417a560 2021-04-09T18:24:55.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f867a7a1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8674179610 con 0x7f86741b85a0 2021-04-09T18:24:55.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f8673fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8674099920 0x7f8674179f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f8673fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8674099920 0x7f8674179f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51726/0 (socket says 172.21.15.32:51726) 2021-04-09T18:24:55.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f8673fff700 1 -- 172.21.15.32:0/3653402895 learned_addr learned my addr 172.21.15.32:0/3653402895 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:55.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f866bfff700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867417a560 0x7f86741b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.565 7f8678dbe700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86741b85a0 0x7f86741b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f8673fff700 1 -- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86741b85a0 msgr2=0x7f86741b89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f8673fff700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86741b85a0 0x7f86741b89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f8673fff700 1 -- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867417a560 msgr2=0x7f86741b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f8673fff700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867417a560 0x7f86741b7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f8673fff700 1 -- 172.21.15.32:0/3653402895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674180be0 con 0x7f8674099920 2021-04-09T18:24:55.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f8673fff700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8674099920 0x7f8674179f40 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 rx=0x7f8664009650 tx=0x7f866400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:55.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f86717fa700 1 -- 172.21.15.32:0/3653402895 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86640093a0 con 0x7f8674099920 2021-04-09T18:24:55.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f86717fa700 1 -- 172.21.15.32:0/3653402895 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866400bd50 con 0x7f8674099920 2021-04-09T18:24:55.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f86717fa700 1 -- 172.21.15.32:0/3653402895 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86640052e0 con 0x7f8674099920 2021-04-09T18:24:55.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8674180be0 con 0x7f8674099920 2021-04-09T18:24:55.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.566 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8674180be0 con 0x7f8674099920 2021-04-09T18:24:55.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.567 7f86717fa700 1 -- 172.21.15.32:0/3653402895 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8664006030 con 0x7f8674099920 2021-04-09T18:24:55.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.567 7f86717fa700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86580188c0 0x7f865801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:55.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.567 7f86717fa700 1 -- 172.21.15.32:0/3653402895 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f86640327a0 con 0x7f8674099920 2021-04-09T18:24:55.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.567 7f866bfff700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86580188c0 0x7f865801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.568 7f866bfff700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86580188c0 0x7f865801adc0 crc :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:55.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.587 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86741b7070 con 0x7f8674099920 2021-04-09T18:24:55.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.590 7f86717fa700 1 -- 172.21.15.32:0/3653402895 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f86640009c0 con 0x7f8674099920 2021-04-09T18:24:55.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.778 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86741b7070 con 0x7f8674099920 2021-04-09T18:24:55.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.779 7f86717fa700 1 -- 172.21.15.32:0/3653402895 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f86640009c0 con 0x7f8674099920 2021-04-09T18:24:55.844 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:55.845 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":118,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:24:54.971049","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:24:52.261572","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-04-09 18:24:53.683747","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:24:55.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86580188c0 msgr2=0x7f865801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86580188c0 0x7f865801adc0 crc :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8674099920 msgr2=0x7f8674179f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:55.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8674099920 0x7f8674179f40 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 rx=0x7f8664009650 tx=0x7f866400b960).stop 2021-04-09T18:24:55.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 shutdown_connections 2021-04-09T18:24:55.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f86580188c0 0x7f865801adc0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86741b85a0 0x7f86741b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867417a560 0x7f86741b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 --2- 172.21.15.32:0/3653402895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8674099920 0x7f8674179f40 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:55.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 >> 172.21.15.32:0/3653402895 conn(0x7f86741aa2b0 msgr2=0x7f86741a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:55.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 shutdown_connections 2021-04-09T18:24:55.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.781 7f867a7a1700 1 -- 172.21.15.32:0/3653402895 wait complete. 2021-04-09T18:24:55.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:24:56.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.945 7fab89571700 1 Processor -- start 2021-04-09T18:24:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.945 7fab89571700 1 -- start start 2021-04-09T18:24:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.945 7fab89571700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fab8410d010 0x7fab840d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.945 7fab89571700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab840d8490 0x7fab8410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.945 7fab89571700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8410d480 con 0x7fab840d1da0 2021-04-09T18:24:56.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.945 7fab89571700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab840d8d30 con 0x7fab840d8490 2021-04-09T18:24:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.945 7fab89571700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab840d9220 con 0x7fab8410d010 2021-04-09T18:24:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab83ac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab840d8490 0x7fab8410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab82abe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fab840d1da0 0x7fab840d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33096/0 (socket says 172.21.15.32:33096) 2021-04-09T18:24:56.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab82abe700 1 -- 172.21.15.32:0/3494266110 learned_addr learned my addr 172.21.15.32:0/3494266110 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:56.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab832bf700 1 --2- 172.21.15.32:0/3494266110 >> v2:172.21.15.86:3300/0 conn(0x7fab8410d010 0x7fab840d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:56.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab83ac0700 1 -- 172.21.15.32:0/3494266110 >> v2:172.21.15.86:3300/0 conn(0x7fab8410d010 msgr2=0x7fab840d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:56.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab83ac0700 1 --2- 172.21.15.32:0/3494266110 >> v2:172.21.15.86:3300/0 conn(0x7fab8410d010 0x7fab840d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:56.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab83ac0700 1 -- 172.21.15.32:0/3494266110 >> v1:172.21.15.32:6789/0 conn(0x7fab840d1da0 legacy=0x7fab840d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab822bd700 1 -- 172.21.15.32:0/3494266110 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929603166 0 0) 0x7fab78000ce0 con 0x7fab840d1da0 2021-04-09T18:24:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab83ac0700 1 -- 172.21.15.32:0/3494266110 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8410e020 con 0x7fab840d8490 2021-04-09T18:24:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab832bf700 1 --2- 172.21.15.32:0/3494266110 >> v2:172.21.15.86:3300/0 conn(0x7fab8410d010 0x7fab840d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:24:56.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab83ac0700 1 --2- 172.21.15.32:0/3494266110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840d8490 0x7fab8410da00 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 rx=0x7fab7c006e30 tx=0x7fab7c009e60).ready entity=mon.0 client_cookie=9f0bba3879b12c84 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab822bd700 1 -- 172.21.15.32:0/3494266110 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab7c006a50 con 0x7fab840d8490 2021-04-09T18:24:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab822bd700 1 -- 172.21.15.32:0/3494266110 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7c00b300 con 0x7fab840d8490 2021-04-09T18:24:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab822bd700 1 -- 172.21.15.32:0/3494266110 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab7c000f90 con 0x7fab840d8490 2021-04-09T18:24:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab89571700 1 -- 172.21.15.32:0/3494266110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840d8490 msgr2=0x7fab8410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:56.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.946 7fab89571700 1 --2- 172.21.15.32:0/3494266110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840d8490 0x7fab8410da00 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 rx=0x7fab7c006e30 tx=0x7fab7c009e60).stop 2021-04-09T18:24:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.947 7fab89571700 1 -- 172.21.15.32:0/3494266110 shutdown_connections 2021-04-09T18:24:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.947 7fab89571700 1 --2- 172.21.15.32:0/3494266110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840d8490 0x7fab8410da00 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.947 7fab89571700 1 --2- 172.21.15.32:0/3494266110 >> v2:172.21.15.86:3300/0 conn(0x7fab8410d010 0x7fab840d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.947 7fab89571700 1 -- 172.21.15.32:0/3494266110 >> 172.21.15.32:0/3494266110 conn(0x7fab840ce5f0 msgr2=0x7fab840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:56.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.947 7fab89571700 1 -- 172.21.15.32:0/3494266110 shutdown_connections 2021-04-09T18:24:56.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.947 7fab89571700 1 -- 172.21.15.32:0/3494266110 wait complete. 2021-04-09T18:24:56.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.967 7fab89571700 1 Processor -- start 2021-04-09T18:24:56.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.967 7fab89571700 1 -- start start 2021-04-09T18:24:56.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab89571700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840c3c00 0x7fab8409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:56.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab89571700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab84095240 0x7fab841a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:56.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab89571700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab841a1270 0x7fab840b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:56.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab89571700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8409cda0 con 0x7fab840c3c00 2021-04-09T18:24:56.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab89571700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab840c3680 con 0x7fab84095240 2021-04-09T18:24:56.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab89571700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab84095640 con 0x7fab841a1270 2021-04-09T18:24:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab832bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840c3c00 0x7fab8409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab832bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840c3c00 0x7fab8409c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51740/0 (socket says 172.21.15.32:51740) 2021-04-09T18:24:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab832bf700 1 -- 172.21.15.32:0/1700914997 learned_addr learned my addr 172.21.15.32:0/1700914997 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab82abe700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab84095240 0x7fab841a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab832bf700 1 -- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab841a1270 msgr2=0x7fab840b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab83ac0700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab841a1270 0x7fab840b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab832bf700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab841a1270 0x7fab840b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab832bf700 1 -- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab84095240 msgr2=0x7fab841a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab832bf700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab84095240 0x7fab841a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab832bf700 1 -- 172.21.15.32:0/1700914997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab841a1670 con 0x7fab840c3c00 2021-04-09T18:24:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab82abe700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab84095240 0x7fab841a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:56.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab83ac0700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab841a1270 0x7fab840b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.968 7fab832bf700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840c3c00 0x7fab8409c600 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 rx=0x7fab74006f20 tx=0x7fab7400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.969 7fab6bfff700 1 -- 172.21.15.32:0/1700914997 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab7400bd80 con 0x7fab840c3c00 2021-04-09T18:24:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.969 7fab6bfff700 1 -- 172.21.15.32:0/1700914997 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab74000c90 con 0x7fab840c3c00 2021-04-09T18:24:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.969 7fab6bfff700 1 -- 172.21.15.32:0/1700914997 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab740022b0 con 0x7fab840c3c00 2021-04-09T18:24:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.969 7fab89571700 1 -- 172.21.15.32:0/1700914997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab841a1670 con 0x7fab840c3c00 2021-04-09T18:24:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.969 7fab89571700 1 -- 172.21.15.32:0/1700914997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab841a1670 con 0x7fab840c3c00 2021-04-09T18:24:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.969 7fab6bfff700 1 -- 172.21.15.32:0/1700914997 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fab7400a000 con 0x7fab840c3c00 2021-04-09T18:24:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.970 7fab6bfff700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab640188d0 0x7fab6401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.970 7fab6bfff700 1 -- 172.21.15.32:0/1700914997 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fab74032a30 con 0x7fab840c3c00 2021-04-09T18:24:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.970 7fab82abe700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab640188d0 0x7fab6401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.970 7fab82abe700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab640188d0 0x7fab6401add0 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.990 7fab89571700 1 -- 172.21.15.32:0/1700914997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab84093d50 con 0x7fab840c3c00 2021-04-09T18:24:56.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:55.993 7fab6bfff700 1 -- 172.21.15.32:0/1700914997 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fab74000f10 con 0x7fab840c3c00 2021-04-09T18:24:56.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:56.189 7fab89571700 1 -- 172.21.15.32:0/1700914997 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fab8410d480 con 0x7fab840c3c00 2021-04-09T18:24:57.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:56.977 7fab6bfff700 1 -- 172.21.15.32:0/1700914997 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v253) v1 ==== 118+0+0 (secure 0 0 0) 0x7fab7400a000 con 0x7fab840c3c00 2021-04-09T18:24:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 -- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab640188d0 msgr2=0x7fab6401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab640188d0 0x7fab6401add0 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 -- 172.21.15.32:0/1700914997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840c3c00 msgr2=0x7fab8409c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840c3c00 0x7fab8409c600 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 rx=0x7fab74006f20 tx=0x7fab7400b990).stop 2021-04-09T18:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 -- 172.21.15.32:0/1700914997 shutdown_connections 2021-04-09T18:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab640188d0 0x7fab6401add0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab841a1270 0x7fab840b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab84095240 0x7fab841a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 --2- 172.21.15.32:0/1700914997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab840c3c00 0x7fab8409c600 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 -- 172.21.15.32:0/1700914997 >> 172.21.15.32:0/1700914997 conn(0x7fab840975f0 msgr2=0x7fab84096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:57.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 -- 172.21.15.32:0/1700914997 shutdown_connections 2021-04-09T18:24:57.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.007 7fab89571700 1 -- 172.21.15.32:0/1700914997 wait complete. 2021-04-09T18:24:57.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:57.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.166 7f685b642700 1 Processor -- start 2021-04-09T18:24:57.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.166 7f685b642700 1 -- start start 2021-04-09T18:24:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.167 7f685b642700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f68540d7ca0 0x7f68540d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.167 7f685b642700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f685410d480 con 0x7f685410da00 2021-04-09T18:24:57.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.167 7f685b642700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68540d86c0 con 0x7f685410d010 2021-04-09T18:24:57.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.167 7f685b642700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68540d9220 con 0x7f68540d7ca0 2021-04-09T18:24:57.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.167 7f6859c5f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f685410da00 0x7f685410de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33108/0 (socket says 172.21.15.32:33108) 2021-04-09T18:24:57.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.167 7f6859c5f700 1 -- 172.21.15.32:0/4186260606 learned_addr learned my addr 172.21.15.32:0/4186260606 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:57.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.167 7f6858c5d700 1 --2- 172.21.15.32:0/4186260606 >> v2:172.21.15.86:3300/0 conn(0x7f68540d7ca0 0x7f68540d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.167 7f684bfff700 1 -- 172.21.15.32:0/4186260606 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503414715 0 0) 0x7f684c000d30 con 0x7f685410da00 2021-04-09T18:24:57.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f684bfff700 1 -- 172.21.15.32:0/4186260606 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6840002fe0 con 0x7f685410da00 2021-04-09T18:24:57.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f684bfff700 1 -- 172.21.15.32:0/4186260606 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156801221 0 0) 0x7f6850000c60 con 0x7f685410d010 2021-04-09T18:24:57.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f684bfff700 1 -- 172.21.15.32:0/4186260606 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6840003210 con 0x7f685410d010 2021-04-09T18:24:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f6858c5d700 1 -- 172.21.15.32:0/4186260606 >> v1:172.21.15.86:6789/0 conn(0x7f685410d010 legacy=0x7f68540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f684bfff700 1 -- 172.21.15.32:0/4186260606 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2598464480 0 0) 0x7f684c001270 con 0x7f685410da00 2021-04-09T18:24:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f6858c5d700 1 -- 172.21.15.32:0/4186260606 >> v1:172.21.15.32:6789/0 conn(0x7f685410da00 legacy=0x7f685410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f6858c5d700 1 -- 172.21.15.32:0/4186260606 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6854112100 con 0x7f68540d7ca0 2021-04-09T18:24:57.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f6858c5d700 1 --2- 172.21.15.32:0/4186260606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68540d7ca0 0x7f68540d80a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f6844006650 tx=0x7f6844009a80).ready entity=mon.1 client_cookie=78524dc565984f41 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:57.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f684bfff700 1 -- 172.21.15.32:0/4186260606 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68440009b0 con 0x7f68540d7ca0 2021-04-09T18:24:57.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f684bfff700 1 -- 172.21.15.32:0/4186260606 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684400c0c0 con 0x7f68540d7ca0 2021-04-09T18:24:57.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f684bfff700 1 -- 172.21.15.32:0/4186260606 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f684400c510 con 0x7f68540d7ca0 2021-04-09T18:24:57.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f685b642700 1 -- 172.21.15.32:0/4186260606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68540d7ca0 msgr2=0x7f68540d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.168 7f685b642700 1 --2- 172.21.15.32:0/4186260606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68540d7ca0 0x7f68540d80a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f6844006650 tx=0x7f6844009a80).stop 2021-04-09T18:24:57.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.169 7f685b642700 1 -- 172.21.15.32:0/4186260606 shutdown_connections 2021-04-09T18:24:57.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.169 7f685b642700 1 --2- 172.21.15.32:0/4186260606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68540d7ca0 0x7f68540d80a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.169 7f685b642700 1 -- 172.21.15.32:0/4186260606 >> 172.21.15.32:0/4186260606 conn(0x7f68540ce5f0 msgr2=0x7f68540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.169 7f685b642700 1 -- 172.21.15.32:0/4186260606 shutdown_connections 2021-04-09T18:24:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.169 7f685b642700 1 -- 172.21.15.32:0/4186260606 wait complete. 2021-04-09T18:24:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.189 7f685b642700 1 Processor -- start 2021-04-09T18:24:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.189 7f685b642700 1 -- start start 2021-04-09T18:24:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f685b642700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6854186fb0 0x7f6854184680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f685b642700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6854183da0 0x7f6854187b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f685b642700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6854188170 0x7f68541ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f685b642700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68541875d0 con 0x7f6854183da0 2021-04-09T18:24:57.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f685b642700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6854188570 con 0x7f6854186fb0 2021-04-09T18:24:57.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f685b642700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68541841a0 con 0x7f6854188170 2021-04-09T18:24:57.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f6858c5d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6854183da0 0x7f6854187b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f6858c5d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6854183da0 0x7f6854187b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51756/0 (socket says 172.21.15.32:51756) 2021-04-09T18:24:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f6858c5d700 1 -- 172.21.15.32:0/613265035 learned_addr learned my addr 172.21.15.32:0/613265035 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f685945e700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6854186fb0 0x7f6854184680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.190 7f6859c5f700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6854188170 0x7f68541ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f6858c5d700 1 -- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6854188170 msgr2=0x7f68541ac090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f6858c5d700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6854188170 0x7f68541ac090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f6858c5d700 1 -- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6854186fb0 msgr2=0x7f6854184680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f6858c5d700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6854186fb0 0x7f6854184680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f6858c5d700 1 -- 172.21.15.32:0/613265035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68541ac6b0 con 0x7f6854183da0 2021-04-09T18:24:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f6858c5d700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6854183da0 0x7f6854187b50 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f6844009650 tx=0x7f684400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f6849ffb700 1 -- 172.21.15.32:0/613265035 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68440093a0 con 0x7f6854183da0 2021-04-09T18:24:57.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f6849ffb700 1 -- 172.21.15.32:0/613265035 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684400bd50 con 0x7f6854183da0 2021-04-09T18:24:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f685b642700 1 -- 172.21.15.32:0/613265035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68541ac6b0 con 0x7f6854183da0 2021-04-09T18:24:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f6849ffb700 1 -- 172.21.15.32:0/613265035 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68440052e0 con 0x7f6854183da0 2021-04-09T18:24:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.191 7f685b642700 1 -- 172.21.15.32:0/613265035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68541ac6b0 con 0x7f6854183da0 2021-04-09T18:24:57.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.192 7f6849ffb700 1 -- 172.21.15.32:0/613265035 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6844006030 con 0x7f6854183da0 2021-04-09T18:24:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.192 7f6849ffb700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6834018930 0x7f683401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.192 7f6849ffb700 1 -- 172.21.15.32:0/613265035 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f68440329e0 con 0x7f6854183da0 2021-04-09T18:24:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.192 7f685945e700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6834018930 0x7f683401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.192 7f685945e700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6834018930 0x7f683401ae10 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:57.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.211 7f685b642700 1 -- 172.21.15.32:0/613265035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6854182d80 con 0x7f6854183da0 2021-04-09T18:24:57.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.215 7f6849ffb700 1 -- 172.21.15.32:0/613265035 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f68440009c0 con 0x7f6854183da0 2021-04-09T18:24:57.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.406 7f685b642700 1 -- 172.21.15.32:0/613265035 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f685410d480 con 0x7f6854183da0 2021-04-09T18:24:57.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.406 7f6849ffb700 1 -- 172.21.15.32:0/613265035 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f684400bd50 con 0x7f6854183da0 2021-04-09T18:24:57.471 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 253 2021-04-09T18:24:57.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:57.472 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":253,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":8752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:24:56.975524","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:24:57.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 -- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6834018930 msgr2=0x7f683401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6834018930 0x7f683401ae10 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 -- 172.21.15.32:0/613265035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6854183da0 msgr2=0x7f6854187b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6854183da0 0x7f6854187b50 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f6844009650 tx=0x7f684400b960).stop 2021-04-09T18:24:57.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 -- 172.21.15.32:0/613265035 shutdown_connections 2021-04-09T18:24:57.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6834018930 0x7f683401ae10 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6854183da0 0x7f6854187b50 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6854188170 0x7f68541ac090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 --2- 172.21.15.32:0/613265035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6854186fb0 0x7f6854184680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 -- 172.21.15.32:0/613265035 >> 172.21.15.32:0/613265035 conn(0x7f685419b890 msgr2=0x7f68541a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:57.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 -- 172.21.15.32:0/613265035 shutdown_connections 2021-04-09T18:24:57.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.409 7f685b642700 1 -- 172.21.15.32:0/613265035 wait complete. 2021-04-09T18:24:57.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:24:57.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.569 7f4342662700 1 Processor -- start 2021-04-09T18:24:57.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.569 7f4342662700 1 -- start start 2021-04-09T18:24:57.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.569 7f4342662700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f433c0d7ca0 0x7f433c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.569 7f4342662700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f433c10da00 0x7f433c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.569 7f4342662700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433c10d480 con 0x7f433c10d010 2021-04-09T18:24:57.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.569 7f4342662700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f433c0d9220 con 0x7f433c0d7ca0 2021-04-09T18:24:57.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.569 7f4342662700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f433c0d86c0 con 0x7f433c10da00 2021-04-09T18:24:57.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.569 7f4333fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f433c0d7ca0 0x7f433c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4333fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f433c0d7ca0 0x7f433c0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51766/0 (socket says 172.21.15.32:51766) 2021-04-09T18:24:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4333fff700 1 -- 172.21.15.32:0/2376752106 learned_addr learned my addr 172.21.15.32:0/2376752106 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4340c7f700 1 --2- 172.21.15.32:0/2376752106 >> v2:172.21.15.86:3300/0 conn(0x7f433c10da00 0x7f433c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4333fff700 1 -- 172.21.15.32:0/2376752106 >> v2:172.21.15.86:3300/0 conn(0x7f433c10da00 msgr2=0x7f433c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4333fff700 1 --2- 172.21.15.32:0/2376752106 >> v2:172.21.15.86:3300/0 conn(0x7f433c10da00 0x7f433c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4333fff700 1 -- 172.21.15.32:0/2376752106 >> v1:172.21.15.86:6789/0 conn(0x7f433c10d010 legacy=0x7f433c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4333fff700 1 -- 172.21.15.32:0/2376752106 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c0d1f70 con 0x7f433c0d7ca0 2021-04-09T18:24:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4333fff700 1 --2- 172.21.15.32:0/2376752106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c0d7ca0 0x7f433c0d80a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7f4324006700 tx=0x7f4324009b80).ready entity=mon.0 client_cookie=f38f30b1ad52b3c6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:57.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f433b7fe700 1 -- 172.21.15.32:0/2376752106 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432400c0c0 con 0x7f433c0d7ca0 2021-04-09T18:24:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f433b7fe700 1 -- 172.21.15.32:0/2376752106 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432400c440 con 0x7f433c0d7ca0 2021-04-09T18:24:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f433b7fe700 1 -- 172.21.15.32:0/2376752106 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432400c920 con 0x7f433c0d7ca0 2021-04-09T18:24:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4342662700 1 -- 172.21.15.32:0/2376752106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c0d7ca0 msgr2=0x7f433c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.570 7f4342662700 1 --2- 172.21.15.32:0/2376752106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c0d7ca0 0x7f433c0d80a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7f4324006700 tx=0x7f4324009b80).stop 2021-04-09T18:24:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.571 7f4342662700 1 -- 172.21.15.32:0/2376752106 shutdown_connections 2021-04-09T18:24:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.571 7f4342662700 1 --2- 172.21.15.32:0/2376752106 >> v2:172.21.15.86:3300/0 conn(0x7f433c10da00 0x7f433c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.571 7f4342662700 1 --2- 172.21.15.32:0/2376752106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c0d7ca0 0x7f433c0d80a0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.571 7f4342662700 1 -- 172.21.15.32:0/2376752106 >> 172.21.15.32:0/2376752106 conn(0x7f433c0ce5f0 msgr2=0x7f433c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.571 7f4342662700 1 -- 172.21.15.32:0/2376752106 shutdown_connections 2021-04-09T18:24:57.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.571 7f4342662700 1 -- 172.21.15.32:0/2376752106 wait complete. 2021-04-09T18:24:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.591 7f4342662700 1 Processor -- start 2021-04-09T18:24:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.591 7f4342662700 1 -- start start 2021-04-09T18:24:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4342662700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433c1a5210 0x7f433c08eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4342662700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f433c1a8ab0 0x7f433c1a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4342662700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c1a76d0 0x7f433c1a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4342662700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c17ea50 con 0x7f433c1a76d0 2021-04-09T18:24:57.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4342662700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c08f4f0 con 0x7f433c1a8ab0 2021-04-09T18:24:57.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4342662700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433c08ea60 con 0x7f433c1a5210 2021-04-09T18:24:57.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4340c7f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c1a76d0 0x7f433c1a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4340c7f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c1a76d0 0x7f433c1a7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51772/0 (socket says 172.21.15.32:51772) 2021-04-09T18:24:57.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4340c7f700 1 -- 172.21.15.32:0/2836059513 learned_addr learned my addr 172.21.15.32:0/2836059513 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4333fff700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f433c1a8ab0 0x7f433c1a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f433bfff700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433c1a5210 0x7f433c08eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4340c7f700 1 -- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433c1a5210 msgr2=0x7f433c08eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4340c7f700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433c1a5210 0x7f433c08eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4340c7f700 1 -- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f433c1a8ab0 msgr2=0x7f433c1a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4340c7f700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f433c1a8ab0 0x7f433c1a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4340c7f700 1 -- 172.21.15.32:0/2836059513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c1a80f0 con 0x7f433c1a76d0 2021-04-09T18:24:57.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4333fff700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f433c1a8ab0 0x7f433c1a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:57.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.592 7f4340c7f700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c1a76d0 0x7f433c1a7ad0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 rx=0x7f43340099a0 tx=0x7f433400bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:57.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.593 7f43397fa700 1 -- 172.21.15.32:0/2836059513 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f433400bdb0 con 0x7f433c1a76d0 2021-04-09T18:24:57.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.593 7f43397fa700 1 -- 172.21.15.32:0/2836059513 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4334003030 con 0x7f433c1a76d0 2021-04-09T18:24:57.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.593 7f4342662700 1 -- 172.21.15.32:0/2836059513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433c1a80f0 con 0x7f433c1a76d0 2021-04-09T18:24:57.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.593 7f43397fa700 1 -- 172.21.15.32:0/2836059513 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4334001470 con 0x7f433c1a76d0 2021-04-09T18:24:57.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.593 7f4342662700 1 -- 172.21.15.32:0/2836059513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433c1a80f0 con 0x7f433c1a76d0 2021-04-09T18:24:57.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.593 7f43397fa700 1 -- 172.21.15.32:0/2836059513 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f43340056b0 con 0x7f433c1a76d0 2021-04-09T18:24:57.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.593 7f43397fa700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43200188d0 0x7f432001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:57.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.593 7f43397fa700 1 -- 172.21.15.32:0/2836059513 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4334032da0 con 0x7f433c1a76d0 2021-04-09T18:24:57.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.594 7f433bfff700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43200188d0 0x7f432001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:57.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.594 7f433bfff700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43200188d0 0x7f432001add0 crc :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:57.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.613 7f4342662700 1 -- 172.21.15.32:0/2836059513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433c1b7070 con 0x7f433c1a76d0 2021-04-09T18:24:57.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.617 7f43397fa700 1 -- 172.21.15.32:0/2836059513 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f43340066f0 con 0x7f433c1a76d0 2021-04-09T18:24:57.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.805 7f4342662700 1 -- 172.21.15.32:0/2836059513 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f433c10d480 con 0x7f433c1a76d0 2021-04-09T18:24:57.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.806 7f43397fa700 1 -- 172.21.15.32:0/2836059513 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f433401fae0 con 0x7f433c1a76d0 2021-04-09T18:24:57.871 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:57.872 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":119,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:24:55.972582","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:24:52.261572","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-04-09 18:24:53.683747","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:24:57.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 -- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43200188d0 msgr2=0x7f432001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43200188d0 0x7f432001add0 crc :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 -- 172.21.15.32:0/2836059513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c1a76d0 msgr2=0x7f433c1a7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:57.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c1a76d0 0x7f433c1a7ad0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 rx=0x7f43340099a0 tx=0x7f433400bc70).stop 2021-04-09T18:24:57.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 -- 172.21.15.32:0/2836059513 shutdown_connections 2021-04-09T18:24:57.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43200188d0 0x7f432001add0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f433c1a76d0 0x7f433c1a7ad0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f433c1a8ab0 0x7f433c1a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 --2- 172.21.15.32:0/2836059513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f433c1a5210 0x7f433c08eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:57.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 -- 172.21.15.32:0/2836059513 >> 172.21.15.32:0/2836059513 conn(0x7f433c1a99c0 msgr2=0x7f433c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:57.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.808 7f4342662700 1 -- 172.21.15.32:0/2836059513 shutdown_connections 2021-04-09T18:24:57.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.809 7f4342662700 1 -- 172.21.15.32:0/2836059513 wait complete. 2021-04-09T18:24:57.884 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:24:58.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.980 7feb2565d700 1 Processor -- start 2021-04-09T18:24:58.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.980 7feb2565d700 1 -- start start 2021-04-09T18:24:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb2565d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feb200cffb0 0x7feb20087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb2565d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feb200d8050 0x7feb2010d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb2565d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb20088460 con 0x7feb200d7c50 2021-04-09T18:24:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb2565d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb201b0c80 con 0x7feb200cffb0 2021-04-09T18:24:58.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb2565d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200d91f0 con 0x7feb200d8050 2021-04-09T18:24:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1f2bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feb200cffb0 0x7feb20087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1eabe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feb200d7c50 0x7feb201b0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33134/0 (socket says 172.21.15.32:33134) 2021-04-09T18:24:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1eabe700 1 -- 172.21.15.32:0/1353466028 learned_addr learned my addr 172.21.15.32:0/1353466028 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:58.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1fac0700 1 --2- 172.21.15.32:0/1353466028 >> v2:172.21.15.86:3300/0 conn(0x7feb200d8050 0x7feb2010d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1f2bf700 1 -- 172.21.15.32:0/1353466028 >> v2:172.21.15.86:3300/0 conn(0x7feb200d8050 msgr2=0x7feb2010d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:58.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1f2bf700 1 --2- 172.21.15.32:0/1353466028 >> v2:172.21.15.86:3300/0 conn(0x7feb200d8050 0x7feb2010d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1f2bf700 1 -- 172.21.15.32:0/1353466028 >> v1:172.21.15.32:6789/0 conn(0x7feb200d7c50 legacy=0x7feb201b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1e2bd700 1 -- 172.21.15.32:0/1353466028 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898875929 0 0) 0x7feb08000ce0 con 0x7feb200d7c50 2021-04-09T18:24:58.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1f2bf700 1 -- 172.21.15.32:0/1353466028 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200d8620 con 0x7feb200cffb0 2021-04-09T18:24:58.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1f2bf700 1 --2- 172.21.15.32:0/1353466028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb200cffb0 0x7feb20087e40 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 rx=0x7feb18006650 tx=0x7feb18009a00).ready entity=mon.0 client_cookie=d08a0b1aeb996d4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:58.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1e2bd700 1 -- 172.21.15.32:0/1353466028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb18009df0 con 0x7feb200cffb0 2021-04-09T18:24:58.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1e2bd700 1 -- 172.21.15.32:0/1353466028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb180009b0 con 0x7feb200cffb0 2021-04-09T18:24:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb1e2bd700 1 -- 172.21.15.32:0/1353466028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb1800c2b0 con 0x7feb200cffb0 2021-04-09T18:24:58.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb2565d700 1 -- 172.21.15.32:0/1353466028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb200cffb0 msgr2=0x7feb20087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.981 7feb2565d700 1 --2- 172.21.15.32:0/1353466028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb200cffb0 0x7feb20087e40 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 rx=0x7feb18006650 tx=0x7feb18009a00).stop 2021-04-09T18:24:58.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.982 7feb2565d700 1 -- 172.21.15.32:0/1353466028 shutdown_connections 2021-04-09T18:24:58.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.982 7feb2565d700 1 --2- 172.21.15.32:0/1353466028 >> v2:172.21.15.86:3300/0 conn(0x7feb200d8050 0x7feb2010d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.982 7feb2565d700 1 --2- 172.21.15.32:0/1353466028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb200cffb0 0x7feb20087e40 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.982 7feb2565d700 1 -- 172.21.15.32:0/1353466028 >> 172.21.15.32:0/1353466028 conn(0x7feb200ce5b0 msgr2=0x7feb200cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.982 7feb2565d700 1 -- 172.21.15.32:0/1353466028 shutdown_connections 2021-04-09T18:24:58.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:57.982 7feb2565d700 1 -- 172.21.15.32:0/1353466028 wait complete. 2021-04-09T18:24:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.002 7feb2565d700 1 Processor -- start 2021-04-09T18:24:58.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb2565d700 1 -- start start 2021-04-09T18:24:58.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb2565d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb200b1950 0x7feb20183e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb2565d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb201795e0 0x7feb20094eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb2565d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb201799e0 0x7feb200954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb2565d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200b13d0 con 0x7feb201795e0 2021-04-09T18:24:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb2565d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200b1d50 con 0x7feb200b1950 2021-04-09T18:24:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb2565d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2019e4a0 con 0x7feb201799e0 2021-04-09T18:24:58.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb1eabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb201795e0 0x7feb20094eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb1eabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb201795e0 0x7feb20094eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51782/0 (socket says 172.21.15.32:51782) 2021-04-09T18:24:58.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb1eabe700 1 -- 172.21.15.32:0/3555720157 learned_addr learned my addr 172.21.15.32:0/3555720157 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:58.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb1fac0700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb201799e0 0x7feb200954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.003 7feb1f2bf700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb200b1950 0x7feb20183e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb1eabe700 1 -- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb201799e0 msgr2=0x7feb200954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb1eabe700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb201799e0 0x7feb200954d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb1eabe700 1 -- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb200b1950 msgr2=0x7feb20183e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb1eabe700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb200b1950 0x7feb20183e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb1eabe700 1 -- 172.21.15.32:0/3555720157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200b0180 con 0x7feb201795e0 2021-04-09T18:24:58.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb1eabe700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb201795e0 0x7feb20094eb0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7feb08006f00 tx=0x7feb0800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:58.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb17fff700 1 -- 172.21.15.32:0/3555720157 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb08006020 con 0x7feb201795e0 2021-04-09T18:24:58.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb17fff700 1 -- 172.21.15.32:0/3555720157 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb080063a0 con 0x7feb201795e0 2021-04-09T18:24:58.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb2565d700 1 -- 172.21.15.32:0/3555720157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb200b0180 con 0x7feb201795e0 2021-04-09T18:24:58.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb17fff700 1 -- 172.21.15.32:0/3555720157 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb08004620 con 0x7feb201795e0 2021-04-09T18:24:58.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb2565d700 1 -- 172.21.15.32:0/3555720157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb200b0180 con 0x7feb201795e0 2021-04-09T18:24:58.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.004 7feb17fff700 1 -- 172.21.15.32:0/3555720157 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7feb0800a000 con 0x7feb201795e0 2021-04-09T18:24:58.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.005 7feb17fff700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb04018930 0x7feb0401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.005 7feb17fff700 1 -- 172.21.15.32:0/3555720157 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7feb08032a10 con 0x7feb201795e0 2021-04-09T18:24:58.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.005 7feb1f2bf700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb04018930 0x7feb0401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.005 7feb1f2bf700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb04018930 0x7feb0401ae10 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:58.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.024 7feb2565d700 1 -- 172.21.15.32:0/3555720157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb20093fc0 con 0x7feb201795e0 2021-04-09T18:24:58.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.028 7feb17fff700 1 -- 172.21.15.32:0/3555720157 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7feb08006b90 con 0x7feb201795e0 2021-04-09T18:24:58.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.213 7feb2565d700 1 -- 172.21.15.32:0/3555720157 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7feb200d8620 con 0x7feb201795e0 2021-04-09T18:24:58.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.216 7feb17fff700 1 -- 172.21.15.32:0/3555720157 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v30) v1 ==== 184+0+0 (secure 0 0 0) 0x7feb0800a000 con 0x7feb201795e0 2021-04-09T18:24:58.285 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:24:58.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 -- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb04018930 msgr2=0x7feb0401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb04018930 0x7feb0401ae10 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 -- 172.21.15.32:0/3555720157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb201795e0 msgr2=0x7feb20094eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb201795e0 0x7feb20094eb0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7feb08006f00 tx=0x7feb0800b940).stop 2021-04-09T18:24:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 -- 172.21.15.32:0/3555720157 shutdown_connections 2021-04-09T18:24:58.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb04018930 0x7feb0401ae10 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb201795e0 0x7feb20094eb0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb201799e0 0x7feb200954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 --2- 172.21.15.32:0/3555720157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb200b1950 0x7feb20183e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 -- 172.21.15.32:0/3555720157 >> 172.21.15.32:0/3555720157 conn(0x7feb201889b0 msgr2=0x7feb20189210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:58.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 -- 172.21.15.32:0/3555720157 shutdown_connections 2021-04-09T18:24:58.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.223 7feb2565d700 1 -- 172.21.15.32:0/3555720157 wait complete. 2021-04-09T18:24:58.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:24:58.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.394 7fe7320d6700 1 Processor -- start 2021-04-09T18:24:58.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.394 7fe7320d6700 1 -- start start 2021-04-09T18:24:58.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.395 7fe7320d6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe72c0f28c0 0x7fe72c0cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.395 7fe7320d6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe72c1b04f0 0x7fe72c1b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.395 7fe7320d6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c0d8700 con 0x7fe72c0d2700 2021-04-09T18:24:58.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.395 7fe7320d6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72c0f2e80 con 0x7fe72c0f28c0 2021-04-09T18:24:58.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.395 7fe7320d6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72c0d91b0 con 0x7fe72c1b04f0 2021-04-09T18:24:58.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.395 7fe72affd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe72c0f28c0 0x7fe72c0cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.395 7fe72b7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe72c0d2700 0x7fe72c0d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33146/0 (socket says 172.21.15.32:33146) 2021-04-09T18:24:58.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.395 7fe72b7fe700 1 -- 172.21.15.32:0/4030219066 learned_addr learned my addr 172.21.15.32:0/4030219066 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:58.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.395 7fe72bfff700 1 --2- 172.21.15.32:0/4030219066 >> v2:172.21.15.86:3300/0 conn(0x7fe72c1b04f0 0x7fe72c1b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe72affd700 1 -- 172.21.15.32:0/4030219066 >> v2:172.21.15.86:3300/0 conn(0x7fe72c1b04f0 msgr2=0x7fe72c1b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:58.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe72affd700 1 --2- 172.21.15.32:0/4030219066 >> v2:172.21.15.86:3300/0 conn(0x7fe72c1b04f0 0x7fe72c1b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe72affd700 1 -- 172.21.15.32:0/4030219066 >> v1:172.21.15.32:6789/0 conn(0x7fe72c0d2700 legacy=0x7fe72c0d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe72affd700 1 -- 172.21.15.32:0/4030219066 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c0d8470 con 0x7fe72c0f28c0 2021-04-09T18:24:58.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe72a7fc700 1 -- 172.21.15.32:0/4030219066 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761650196 0 0) 0x7fe71c000ce0 con 0x7fe72c0d2700 2021-04-09T18:24:58.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe72affd700 1 --2- 172.21.15.32:0/4030219066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c0f28c0 0x7fe72c0cfeb0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 rx=0x7fe720006650 tx=0x7fe720009a00).ready entity=mon.0 client_cookie=5bcdae9249cad29d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:58.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe72a7fc700 1 -- 172.21.15.32:0/4030219066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe720009df0 con 0x7fe72c0f28c0 2021-04-09T18:24:58.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe72a7fc700 1 -- 172.21.15.32:0/4030219066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7200009b0 con 0x7fe72c0f28c0 2021-04-09T18:24:58.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe72a7fc700 1 -- 172.21.15.32:0/4030219066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe72000c2b0 con 0x7fe72c0f28c0 2021-04-09T18:24:58.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe7320d6700 1 -- 172.21.15.32:0/4030219066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c0f28c0 msgr2=0x7fe72c0cfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.396 7fe7320d6700 1 --2- 172.21.15.32:0/4030219066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c0f28c0 0x7fe72c0cfeb0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 rx=0x7fe720006650 tx=0x7fe720009a00).stop 2021-04-09T18:24:58.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.397 7fe7320d6700 1 -- 172.21.15.32:0/4030219066 shutdown_connections 2021-04-09T18:24:58.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.397 7fe7320d6700 1 --2- 172.21.15.32:0/4030219066 >> v2:172.21.15.86:3300/0 conn(0x7fe72c1b04f0 0x7fe72c1b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.397 7fe7320d6700 1 --2- 172.21.15.32:0/4030219066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c0f28c0 0x7fe72c0cfeb0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.397 7fe7320d6700 1 -- 172.21.15.32:0/4030219066 >> 172.21.15.32:0/4030219066 conn(0x7fe72c0ce590 msgr2=0x7fe72c0cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:58.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.397 7fe7320d6700 1 -- 172.21.15.32:0/4030219066 shutdown_connections 2021-04-09T18:24:58.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.397 7fe7320d6700 1 -- 172.21.15.32:0/4030219066 wait complete. 2021-04-09T18:24:58.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.417 7fe7320d6700 1 Processor -- start 2021-04-09T18:24:58.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.417 7fe7320d6700 1 -- start start 2021-04-09T18:24:58.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe7320d6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c09a720 0x7fe72c19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe7320d6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe72c09ab20 0x7fe72c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe7320d6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe72c19b440 0x7fe72c196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe7320d6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c19b840 con 0x7fe72c09a720 2021-04-09T18:24:58.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe7320d6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0a5450 con 0x7fe72c19b440 2021-04-09T18:24:58.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe7320d6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c19a8a0 con 0x7fe72c09ab20 2021-04-09T18:24:58.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c09a720 0x7fe72c19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c09a720 0x7fe72c19a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51794/0 (socket says 172.21.15.32:51794) 2021-04-09T18:24:58.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72b7fe700 1 -- 172.21.15.32:0/1748549942 learned_addr learned my addr 172.21.15.32:0/1748549942 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:58.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72bfff700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe72c19b440 0x7fe72c196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72affd700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe72c09ab20 0x7fe72c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72b7fe700 1 -- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe72c09ab20 msgr2=0x7fe72c19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72b7fe700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe72c09ab20 0x7fe72c19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72b7fe700 1 -- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe72c19b440 msgr2=0x7fe72c196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72b7fe700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe72c19b440 0x7fe72c196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72b7fe700 1 -- 172.21.15.32:0/1748549942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c197490 con 0x7fe72c09a720 2021-04-09T18:24:58.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72bfff700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe72c19b440 0x7fe72c196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:58.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72affd700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe72c09ab20 0x7fe72c19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:24:58.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe72b7fe700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c09a720 0x7fe72c19a090 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7fe71c006f00 tx=0x7fe71c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:58.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.418 7fe713fff700 1 -- 172.21.15.32:0/1748549942 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe71c00bd30 con 0x7fe72c09a720 2021-04-09T18:24:58.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.419 7fe713fff700 1 -- 172.21.15.32:0/1748549942 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c0060d0 con 0x7fe72c09a720 2021-04-09T18:24:58.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.419 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c197490 con 0x7fe72c09a720 2021-04-09T18:24:58.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.419 7fe713fff700 1 -- 172.21.15.32:0/1748549942 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe71c0043a0 con 0x7fe72c09a720 2021-04-09T18:24:58.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.419 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c197490 con 0x7fe72c09a720 2021-04-09T18:24:58.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.419 7fe713fff700 1 -- 172.21.15.32:0/1748549942 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe71c007000 con 0x7fe72c09a720 2021-04-09T18:24:58.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.419 7fe713fff700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe70c018930 0x7fe70c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.419 7fe713fff700 1 -- 172.21.15.32:0/1748549942 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe71c0329e0 con 0x7fe72c09a720 2021-04-09T18:24:58.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.419 7fe72affd700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe70c018930 0x7fe70c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.420 7fe72affd700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe70c018930 0x7fe70c01ae10 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:58.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.439 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c09af20 con 0x7fe72c09a720 2021-04-09T18:24:58.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.443 7fe713fff700 1 -- 172.21.15.32:0/1748549942 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe71c007ee0 con 0x7fe72c09a720 2021-04-09T18:24:58.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.627 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe72c09af20 con 0x7fe72c09a720 2021-04-09T18:24:58.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.630 7fe713fff700 1 -- 172.21.15.32:0/1748549942 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v31) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe71c007000 con 0x7fe72c09a720 2021-04-09T18:24:58.698 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:24:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe70c018930 msgr2=0x7fe70c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe70c018930 0x7fe70c01ae10 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c09a720 msgr2=0x7fe72c19a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c09a720 0x7fe72c19a090 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7fe71c006f00 tx=0x7fe71c00b940).stop 2021-04-09T18:24:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 shutdown_connections 2021-04-09T18:24:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe70c018930 0x7fe70c01ae10 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe72c19b440 0x7fe72c196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe72c09ab20 0x7fe72c19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 --2- 172.21.15.32:0/1748549942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe72c09a720 0x7fe72c19a090 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 >> 172.21.15.32:0/1748549942 conn(0x7fe72c1acb70 msgr2=0x7fe72c18c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 shutdown_connections 2021-04-09T18:24:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.636 7fe7320d6700 1 -- 172.21.15.32:0/1748549942 wait complete. 2021-04-09T18:24:58.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:24:58.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.796 7f04ff6f9700 1 Processor -- start 2021-04-09T18:24:58.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.796 7f04ff6f9700 1 -- start start 2021-04-09T18:24:58.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.796 7f04ff6f9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f80d7ca0 0x7f04f80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04ff6f9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f04f810da00 0x7f04f810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04ff6f9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f810d480 con 0x7f04f810d010 2021-04-09T18:24:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04ff6f9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f80d9220 con 0x7f04f810da00 2021-04-09T18:24:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04ff6f9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f80d86c0 con 0x7f04f80d7ca0 2021-04-09T18:24:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fd515700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f04f810d010 0x7f04f80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33160/0 (socket says 172.21.15.32:33160) 2021-04-09T18:24:58.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fd515700 1 -- 172.21.15.32:0/2815687190 learned_addr learned my addr 172.21.15.32:0/2815687190 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:58.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fdd16700 1 --2- 172.21.15.32:0/2815687190 >> v2:172.21.15.86:3300/0 conn(0x7f04f810da00 0x7f04f810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fcd14700 1 --2- 172.21.15.32:0/2815687190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f80d7ca0 0x7f04f80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04ebfff700 1 -- 172.21.15.32:0/2815687190 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201721992 0 0) 0x7f04f4000ce0 con 0x7f04f810d010 2021-04-09T18:24:58.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04ebfff700 1 -- 172.21.15.32:0/2815687190 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e4002fe0 con 0x7f04f810d010 2021-04-09T18:24:58.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fcd14700 1 -- 172.21.15.32:0/2815687190 >> v2:172.21.15.86:3300/0 conn(0x7f04f810da00 msgr2=0x7f04f810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:58.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fcd14700 1 --2- 172.21.15.32:0/2815687190 >> v2:172.21.15.86:3300/0 conn(0x7f04f810da00 0x7f04f810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fcd14700 1 -- 172.21.15.32:0/2815687190 >> v1:172.21.15.32:6789/0 conn(0x7f04f810d010 legacy=0x7f04f80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fcd14700 1 -- 172.21.15.32:0/2815687190 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f80d1f70 con 0x7f04f80d7ca0 2021-04-09T18:24:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fdd16700 1 --2- 172.21.15.32:0/2815687190 >> v2:172.21.15.86:3300/0 conn(0x7f04f810da00 0x7f04f810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04fcd14700 1 --2- 172.21.15.32:0/2815687190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f80d7ca0 0x7f04f80d80a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f04ec006650 tx=0x7f04ec009a80).ready entity=mon.2 client_cookie=860f59b99b5c312d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:58.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.797 7f04ebfff700 1 -- 172.21.15.32:0/2815687190 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ec0009b0 con 0x7f04f80d7ca0 2021-04-09T18:24:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ebfff700 1 -- 172.21.15.32:0/2815687190 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec00c0c0 con 0x7f04f80d7ca0 2021-04-09T18:24:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ebfff700 1 -- 172.21.15.32:0/2815687190 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ec00c510 con 0x7f04f80d7ca0 2021-04-09T18:24:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ff6f9700 1 -- 172.21.15.32:0/2815687190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f80d7ca0 msgr2=0x7f04f80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ff6f9700 1 --2- 172.21.15.32:0/2815687190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f80d7ca0 0x7f04f80d80a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f04ec006650 tx=0x7f04ec009a80).stop 2021-04-09T18:24:58.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ff6f9700 1 -- 172.21.15.32:0/2815687190 shutdown_connections 2021-04-09T18:24:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ff6f9700 1 --2- 172.21.15.32:0/2815687190 >> v2:172.21.15.86:3300/0 conn(0x7f04f810da00 0x7f04f810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ff6f9700 1 --2- 172.21.15.32:0/2815687190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f80d7ca0 0x7f04f80d80a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ff6f9700 1 -- 172.21.15.32:0/2815687190 >> 172.21.15.32:0/2815687190 conn(0x7f04f80ce5f0 msgr2=0x7f04f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ff6f9700 1 -- 172.21.15.32:0/2815687190 shutdown_connections 2021-04-09T18:24:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.798 7f04ff6f9700 1 -- 172.21.15.32:0/2815687190 wait complete. 2021-04-09T18:24:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.819 7f04ff6f9700 1 Processor -- start 2021-04-09T18:24:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.819 7f04ff6f9700 1 -- start start 2021-04-09T18:24:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.819 7f04ff6f9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80a90c0 0x7f04f81a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.819 7f04ff6f9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04f81a76d0 0x7f04f81a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.819 7f04ff6f9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f817bce0 0x7f04f817c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.819 7f04ff6f9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f81a7c60 con 0x7f04f81a76d0 2021-04-09T18:24:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.819 7f04ff6f9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f81a8e20 con 0x7f04f80a90c0 2021-04-09T18:24:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.819 7f04ff6f9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f817c700 con 0x7f04f817bce0 2021-04-09T18:24:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04fcd14700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04f81a76d0 0x7f04f81a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04fcd14700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04f81a76d0 0x7f04f81a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51808/0 (socket says 172.21.15.32:51808) 2021-04-09T18:24:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04fcd14700 1 -- 172.21.15.32:0/1861322460 learned_addr learned my addr 172.21.15.32:0/1861322460 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04fcd14700 1 -- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f817bce0 msgr2=0x7f04f817c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04fcd14700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f817bce0 0x7f04f817c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:24:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04fcd14700 1 -- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80a90c0 msgr2=0x7f04f81a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04fcd14700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80a90c0 0x7f04f81a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:24:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04fcd14700 1 -- 172.21.15.32:0/1861322460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f817c920 con 0x7f04f81a76d0 2021-04-09T18:24:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04fcd14700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04f81a76d0 0x7f04f81a8800 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 rx=0x7f04ec009650 tx=0x7f04ec00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:58.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04e9ffb700 1 -- 172.21.15.32:0/1861322460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ec0093a0 con 0x7f04f81a76d0 2021-04-09T18:24:58.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04e9ffb700 1 -- 172.21.15.32:0/1861322460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec00bd50 con 0x7f04f81a76d0 2021-04-09T18:24:58.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f817c920 con 0x7f04f81a76d0 2021-04-09T18:24:58.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04e9ffb700 1 -- 172.21.15.32:0/1861322460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ec0052e0 con 0x7f04f81a76d0 2021-04-09T18:24:58.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.820 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f817c920 con 0x7f04f81a76d0 2021-04-09T18:24:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.821 7f04e9ffb700 1 -- 172.21.15.32:0/1861322460 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f04ec006030 con 0x7f04f81a76d0 2021-04-09T18:24:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.821 7f04e9ffb700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04d8018930 0x7f04d801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.821 7f04e9ffb700 1 -- 172.21.15.32:0/1861322460 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f04ec032b20 con 0x7f04f81a76d0 2021-04-09T18:24:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.821 7f04fd515700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04d8018930 0x7f04d801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.821 7f04fd515700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04d8018930 0x7f04d801ae10 crc :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:58.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.841 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f81b7010 con 0x7f04f81a76d0 2021-04-09T18:24:58.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:58.844 7f04e9ffb700 1 -- 172.21.15.32:0/1861322460 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f04ec0009c0 con 0x7f04f81a76d0 2021-04-09T18:24:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.036 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04f810d480 con 0x7f04f81a76d0 2021-04-09T18:24:59.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.037 7f04e9ffb700 1 -- 172.21.15.32:0/1861322460 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f04ec00bd50 con 0x7f04f81a76d0 2021-04-09T18:24:59.102 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 253 2021-04-09T18:24:59.103 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:24:59.103 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":253,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":8752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:24:56.975524","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:24:59.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.039 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04d8018930 msgr2=0x7f04d801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:59.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.039 7f04ff6f9700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04d8018930 0x7f04d801ae10 crc :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:59.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.039 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04f81a76d0 msgr2=0x7f04f81a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:59.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.039 7f04ff6f9700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04f81a76d0 0x7f04f81a8800 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 rx=0x7f04ec009650 tx=0x7f04ec00b960).stop 2021-04-09T18:24:59.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.040 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 shutdown_connections 2021-04-09T18:24:59.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.040 7f04ff6f9700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04d8018930 0x7f04d801ae10 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:59.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.040 7f04ff6f9700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04f81a76d0 0x7f04f81a8800 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:59.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.040 7f04ff6f9700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04f817bce0 0x7f04f817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:24:59.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.040 7f04ff6f9700 1 --2- 172.21.15.32:0/1861322460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80a90c0 0x7f04f81a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-04-09T18:24:59.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.040 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 >> 172.21.15.32:0/1861322460 conn(0x7f04f81acae0 msgr2=0x7f04f817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:59.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.040 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 shutdown_connections 2021-04-09T18:24:59.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.040 7f04ff6f9700 1 -- 172.21.15.32:0/1861322460 wait complete. 2021-04-09T18:24:59.517 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 253, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:24:54.971184', 'modified': '2021-04-09 18:24:56.975524', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8740}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8740': {'gid': 8740, 'name': 'a', 'rank': 0, 'incarnation': 250, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2311632030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2311632030}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2311632030}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:24:59.518 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:24:59.518 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:24:59.658 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:24:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:24:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:24:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 8740, 2021-04-09T18:24:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:24:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:24:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 4.6216883969999998, 2021-04-09T18:24:59.660 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 253, 2021-04-09T18:24:59.660 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 119, 2021-04-09T18:24:59.660 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 119, 2021-04-09T18:24:59.660 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 10.097481873 2021-04-09T18:24:59.661 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:24:59.664 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 8740, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.6216883969999998, "mdsmap_epoch": 253, "osdmap_epoch": 119, "osdmap_epoch_barrier": 119, "uptime": 10.097481873 } 2021-04-09T18:24:59.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:24:59.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.748 7fe431b60700 1 Processor -- start 2021-04-09T18:24:59.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.748 7fe431b60700 1 -- start start 2021-04-09T18:24:59.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe431b60700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe42c10d010 0x7fe42c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:59.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe431b60700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe42c0d1da0 0x7fe42c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:59.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe431b60700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c10d480 con 0x7fe42c0d82c0 2021-04-09T18:24:59.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe431b60700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42c0d9220 con 0x7fe42c10d010 2021-04-09T18:24:59.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe431b60700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c0d86c0 con 0x7fe42c0d1da0 2021-04-09T18:24:59.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe42c0d82c0 0x7fe42c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33174/0 (socket says 172.21.15.32:33174) 2021-04-09T18:24:59.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42bfff700 1 -- 172.21.15.32:0/1611622992 learned_addr learned my addr 172.21.15.32:0/1611622992 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:24:59.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42b7fe700 1 --2- 172.21.15.32:0/1611622992 >> v2:172.21.15.86:3300/0 conn(0x7fe42c10d010 0x7fe42c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:59.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42affd700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe42c0d1da0 0x7fe42c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:59.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371981292 0 0) 0x7fe424000d30 con 0x7fe42c0d82c0 2021-04-09T18:24:59.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe414002fe0 con 0x7fe42c0d82c0 2021-04-09T18:24:59.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3465315079 0 0) 0x7fe424001270 con 0x7fe42c0d82c0 2021-04-09T18:24:59.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe42c0d1da0 msgr2=0x7fe42c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:59.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42a7fc700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe42c0d1da0 0x7fe42c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:59.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 >> v2:172.21.15.86:3300/0 conn(0x7fe42c10d010 msgr2=0x7fe42c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:59.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42a7fc700 1 --2- 172.21.15.32:0/1611622992 >> v2:172.21.15.86:3300/0 conn(0x7fe42c10d010 0x7fe42c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:59.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c10e1f0 con 0x7fe42c0d82c0 2021-04-09T18:24:59.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.749 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fe424001980 con 0x7fe42c0d82c0 2021-04-09T18:24:59.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 >> v1:172.21.15.32:6789/0 conn(0x7fe42c0d82c0 legacy=0x7fe42c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:59.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414003a60 0x7fe414005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:59.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41404a7a0 0x7fe41404cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:59.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41404d5a0 0x7fe41404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:59.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4140502e0 con 0x7fe414003a60 2021-04-09T18:24:59.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe414050500 con 0x7fe41404a7a0 2021-04-09T18:24:59.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe414050720 con 0x7fe41404d5a0 2021-04-09T18:24:59.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42b7fe700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414003a60 0x7fe414005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:59.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42affd700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41404a7a0 0x7fe41404cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:59.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42bfff700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41404d5a0 0x7fe41404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:59.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42affd700 1 -- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41404d5a0 msgr2=0x7fe41404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:59.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42affd700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41404d5a0 0x7fe41404fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:59.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42affd700 1 -- 172.21.15.32:0/1611622992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414003a60 msgr2=0x7fe414005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:24:59.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42affd700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414003a60 0x7fe414005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:59.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42affd700 1 -- 172.21.15.32:0/1611622992 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4140509c0 con 0x7fe41404a7a0 2021-04-09T18:24:59.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42b7fe700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414003a60 0x7fe414005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:24:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42affd700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41404a7a0 0x7fe41404cca0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7fe420006ac0 tx=0x7fe420009ab0).ready entity=mon.1 client_cookie=f18d5d64a9ab3675 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe42000c0c0 con 0x7fe41404a7a0 2021-04-09T18:24:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42000c400 con 0x7fe41404a7a0 2021-04-09T18:24:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42a7fc700 1 -- 172.21.15.32:0/1611622992 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe42000c8e0 con 0x7fe41404a7a0 2021-04-09T18:24:59.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe431b60700 1 -- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41404a7a0 msgr2=0x7fe41404cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe431b60700 1 --2- 172.21.15.32:0/1611622992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41404a7a0 0x7fe41404cca0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7fe420006ac0 tx=0x7fe420009ab0).stop 2021-04-09T18:24:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.750 7fe42bfff700 1 -- 172.21.15.32:0/1611622992 reap_dead start 2021-04-09T18:24:59.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.751 7fe431b60700 1 -- 172.21.15.32:0/1611622992 shutdown_connections 2021-04-09T18:24:59.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.751 7fe431b60700 1 -- 172.21.15.32:0/1611622992 >> 172.21.15.32:0/1611622992 conn(0x7fe42c0ce5f0 msgr2=0x7fe42c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:24:59.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.751 7fe431b60700 1 -- 172.21.15.32:0/1611622992 shutdown_connections 2021-04-09T18:24:59.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.751 7fe431b60700 1 -- 172.21.15.32:0/1611622992 wait complete. 2021-04-09T18:24:59.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.771 7fe431b60700 1 Processor -- start 2021-04-09T18:24:59.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.771 7fe431b60700 1 -- start start 2021-04-09T18:24:59.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe431b60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe42c0b5360 0x7fe42c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:59.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe431b60700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe42c18e140 0x7fe42c1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:59.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe431b60700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe42c1ad040 0x7fe42c08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe431b60700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c1a5360 con 0x7fe42c0b5360 2021-04-09T18:24:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe431b60700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c0902c0 con 0x7fe42c1ad040 2021-04-09T18:24:59.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe431b60700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c0904e0 con 0x7fe42c18e140 2021-04-09T18:24:59.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe42c0b5360 0x7fe42c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:59.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe42c0b5360 0x7fe42c1a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51830/0 (socket says 172.21.15.32:51830) 2021-04-09T18:24:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42b7fe700 1 -- 172.21.15.32:0/3850445976 learned_addr learned my addr 172.21.15.32:0/3850445976 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:24:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42affd700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe42c18e140 0x7fe42c1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:59.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42bfff700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe42c1ad040 0x7fe42c08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42b7fe700 1 -- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe42c18e140 msgr2=0x7fe42c1aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42b7fe700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe42c18e140 0x7fe42c1aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42b7fe700 1 -- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe42c1ad040 msgr2=0x7fe42c08fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:24:59.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42b7fe700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe42c1ad040 0x7fe42c08fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:24:59.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.772 7fe42b7fe700 1 -- 172.21.15.32:0/3850445976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c183c80 con 0x7fe42c0b5360 2021-04-09T18:24:59.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.773 7fe42b7fe700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe42c0b5360 0x7fe42c1a4d40 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 rx=0x7fe41c006f60 tx=0x7fe41c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:59.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.773 7fe413fff700 1 -- 172.21.15.32:0/3850445976 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c00bd60 con 0x7fe42c0b5360 2021-04-09T18:24:59.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.773 7fe413fff700 1 -- 172.21.15.32:0/3850445976 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c000c90 con 0x7fe42c0b5360 2021-04-09T18:24:59.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.773 7fe431b60700 1 -- 172.21.15.32:0/3850445976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42c183c80 con 0x7fe42c0b5360 2021-04-09T18:24:59.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.773 7fe413fff700 1 -- 172.21.15.32:0/3850445976 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c002220 con 0x7fe42c0b5360 2021-04-09T18:24:59.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.773 7fe431b60700 1 -- 172.21.15.32:0/3850445976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42c183c80 con 0x7fe42c0b5360 2021-04-09T18:24:59.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.773 7fe413fff700 1 -- 172.21.15.32:0/3850445976 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe41c0052e0 con 0x7fe42c0b5360 2021-04-09T18:24:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.773 7fe413fff700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe40c018930 0x7fe40c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:24:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.773 7fe413fff700 1 -- 172.21.15.32:0/3850445976 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe41c0329f0 con 0x7fe42c0b5360 2021-04-09T18:24:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.774 7fe42affd700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe40c018930 0x7fe40c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:24:59.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.774 7fe42affd700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe40c018930 0x7fe40c01ae10 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:24:59.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.793 7fe431b60700 1 -- 172.21.15.32:0/3850445976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42c182d70 con 0x7fe42c0b5360 2021-04-09T18:24:59.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.796 7fe413fff700 1 -- 172.21.15.32:0/3850445976 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe41c000c90 con 0x7fe42c0b5360 2021-04-09T18:25:00.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.986 7fe431b60700 1 -- 172.21.15.32:0/3850445976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe42c182d70 con 0x7fe42c0b5360 2021-04-09T18:25:00.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.987 7fe413fff700 1 -- 172.21.15.32:0/3850445976 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fe41c006b00 con 0x7fe42c0b5360 2021-04-09T18:25:00.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:00.053 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":119,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:24:55.972582","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:24:52.261572","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-04-09 18:24:53.683747","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:25:00.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 -- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe40c018930 msgr2=0x7fe40c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe40c018930 0x7fe40c01ae10 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 -- 172.21.15.32:0/3850445976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe42c0b5360 msgr2=0x7fe42c1a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe42c0b5360 0x7fe42c1a4d40 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 rx=0x7fe41c006f60 tx=0x7fe41c00b970).stop 2021-04-09T18:25:00.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 -- 172.21.15.32:0/3850445976 shutdown_connections 2021-04-09T18:25:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe40c018930 0x7fe40c01ae10 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe42c1ad040 0x7fe42c08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe42c18e140 0x7fe42c1aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 --2- 172.21.15.32:0/3850445976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe42c0b5360 0x7fe42c1a4d40 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 -- 172.21.15.32:0/3850445976 >> 172.21.15.32:0/3850445976 conn(0x7fe42c19b460 msgr2=0x7fe42c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 -- 172.21.15.32:0/3850445976 shutdown_connections 2021-04-09T18:25:00.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:24:59.990 7fe431b60700 1 -- 172.21.15.32:0/3850445976 wait complete. 2021-04-09T18:25:00.065 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:24:52.261572', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:24:53.683747', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:25:00.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:25:00.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.154 7f670ca68700 1 Processor -- start 2021-04-09T18:25:00.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.154 7f670ca68700 1 -- start start 2021-04-09T18:25:00.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670ca68700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f670810d010 0x7f67080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670ca68700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670810d480 con 0x7f67080d1da0 2021-04-09T18:25:00.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670ca68700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670810e1f0 con 0x7f67080d8490 2021-04-09T18:25:00.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670ca68700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67080d8d30 con 0x7f670810d010 2021-04-09T18:25:00.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670685e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f670810d010 0x7f67080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670605d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f67080d1da0 0x7f67080d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33196/0 (socket says 172.21.15.32:33196) 2021-04-09T18:25:00.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670605d700 1 -- 172.21.15.32:0/408525515 learned_addr learned my addr 172.21.15.32:0/408525515 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:00.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670685e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f670810d010 0x7f67080d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51834/0 (socket says 172.21.15.32:51834) 2021-04-09T18:25:00.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670685e700 1 -- 172.21.15.32:0/408525515 >> v1:172.21.15.86:6789/0 conn(0x7f67080d8490 legacy=0x7f670810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670685e700 1 -- 172.21.15.32:0/408525515 >> v1:172.21.15.32:6789/0 conn(0x7f67080d1da0 legacy=0x7f67080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670585c700 1 -- 172.21.15.32:0/408525515 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875148044 0 0) 0x7f6700000d30 con 0x7f67080d8490 2021-04-09T18:25:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670685e700 1 -- 172.21.15.32:0/408525515 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080d9220 con 0x7f670810d010 2021-04-09T18:25:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.155 7f670685e700 1 --2- 172.21.15.32:0/408525515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670810d010 0x7f67080d1780 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 rx=0x7f66f8006650 tx=0x7f66f8009a00).ready entity=mon.0 client_cookie=442c01fc39bdd35e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.156 7f670585c700 1 -- 172.21.15.32:0/408525515 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66f8009df0 con 0x7f670810d010 2021-04-09T18:25:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.156 7f670585c700 1 -- 172.21.15.32:0/408525515 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f80009b0 con 0x7f670810d010 2021-04-09T18:25:00.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.156 7f670585c700 1 -- 172.21.15.32:0/408525515 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66f8009df0 con 0x7f670810d010 2021-04-09T18:25:00.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.156 7f670ca68700 1 -- 172.21.15.32:0/408525515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670810d010 msgr2=0x7f67080d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.156 7f670ca68700 1 --2- 172.21.15.32:0/408525515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670810d010 0x7f67080d1780 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 rx=0x7f66f8006650 tx=0x7f66f8009a00).stop 2021-04-09T18:25:00.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.157 7f670ca68700 1 -- 172.21.15.32:0/408525515 shutdown_connections 2021-04-09T18:25:00.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.157 7f670ca68700 1 --2- 172.21.15.32:0/408525515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670810d010 0x7f67080d1780 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.157 7f670ca68700 1 -- 172.21.15.32:0/408525515 >> 172.21.15.32:0/408525515 conn(0x7f67080ce5f0 msgr2=0x7f67080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:00.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.157 7f670ca68700 1 -- 172.21.15.32:0/408525515 shutdown_connections 2021-04-09T18:25:00.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.157 7f670ca68700 1 -- 172.21.15.32:0/408525515 wait complete. 2021-04-09T18:25:00.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670ca68700 1 Processor -- start 2021-04-09T18:25:00.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670ca68700 1 -- start start 2021-04-09T18:25:00.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670ca68700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670817ff00 0x7f6708090a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670ca68700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67080acef0 0x7f670819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670ca68700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f670819b0f0 0x7f67080ab890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670ca68700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670817f980 con 0x7f670817ff00 2021-04-09T18:25:00.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670ca68700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67080aee20 con 0x7f67080acef0 2021-04-09T18:25:00.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670ca68700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6708091080 con 0x7f670819b0f0 2021-04-09T18:25:00.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670817ff00 0x7f6708090a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670817ff00 0x7f6708090a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51840/0 (socket says 172.21.15.32:51840) 2021-04-09T18:25:00.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670685e700 1 -- 172.21.15.32:0/3930783405 learned_addr learned my addr 172.21.15.32:0/3930783405 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:00.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670605d700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67080acef0 0x7f670819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.177 7f670705f700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f670819b0f0 0x7f67080ab890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f670685e700 1 -- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f670819b0f0 msgr2=0x7f67080ab890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f670685e700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f670819b0f0 0x7f67080ab890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f670685e700 1 -- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67080acef0 msgr2=0x7f670819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f670685e700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67080acef0 0x7f670819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f670685e700 1 -- 172.21.15.32:0/3930783405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670819b4f0 con 0x7f670817ff00 2021-04-09T18:25:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f670705f700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f670819b0f0 0x7f67080ab890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:00.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f670685e700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670817ff00 0x7f6708090a60 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7f66f80096a0 tx=0x7f66f800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:00.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f66f77fe700 1 -- 172.21.15.32:0/3930783405 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66f800bd80 con 0x7f670817ff00 2021-04-09T18:25:00.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f66f77fe700 1 -- 172.21.15.32:0/3930783405 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f8006030 con 0x7f670817ff00 2021-04-09T18:25:00.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f670ca68700 1 -- 172.21.15.32:0/3930783405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670819b4f0 con 0x7f670817ff00 2021-04-09T18:25:00.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f66f77fe700 1 -- 172.21.15.32:0/3930783405 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66f8005560 con 0x7f670817ff00 2021-04-09T18:25:00.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.178 7f670ca68700 1 -- 172.21.15.32:0/3930783405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670819b4f0 con 0x7f670817ff00 2021-04-09T18:25:00.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.179 7f66f77fe700 1 -- 172.21.15.32:0/3930783405 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f66f80052e0 con 0x7f670817ff00 2021-04-09T18:25:00.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.179 7f66f77fe700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66ec0188d0 0x7f66ec01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.179 7f66f77fe700 1 -- 172.21.15.32:0/3930783405 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f66f80329f0 con 0x7f670817ff00 2021-04-09T18:25:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.179 7f670605d700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66ec0188d0 0x7f66ec01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.179 7f670605d700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66ec0188d0 0x7f66ec01add0 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:00.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.199 7f670ca68700 1 -- 172.21.15.32:0/3930783405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6708091080 con 0x7f670817ff00 2021-04-09T18:25:00.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.202 7f66f77fe700 1 -- 172.21.15.32:0/3930783405 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f66f80009c0 con 0x7f670817ff00 2021-04-09T18:25:00.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.384 7f670ca68700 1 -- 172.21.15.32:0/3930783405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f6708091080 con 0x7f670817ff00 2021-04-09T18:25:00.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.384 7f66f77fe700 1 -- 172.21.15.32:0/3930783405 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v119) v1 ==== 92+0+10 (secure 0 0 0) 0x7f66f800bd80 con 0x7f670817ff00 2021-04-09T18:25:00.449 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:00.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 -- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66ec0188d0 msgr2=0x7f66ec01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66ec0188d0 0x7f66ec01add0 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 -- 172.21.15.32:0/3930783405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670817ff00 msgr2=0x7f6708090a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670817ff00 0x7f6708090a60 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7f66f80096a0 tx=0x7f66f800b990).stop 2021-04-09T18:25:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 -- 172.21.15.32:0/3930783405 shutdown_connections 2021-04-09T18:25:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66ec0188d0 0x7f66ec01add0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f670819b0f0 0x7f67080ab890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67080acef0 0x7f670819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 --2- 172.21.15.32:0/3930783405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f670817ff00 0x7f6708090a60 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 -- 172.21.15.32:0/3930783405 >> 172.21.15.32:0/3930783405 conn(0x7f67080bb410 msgr2=0x7f670808dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:00.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 -- 172.21.15.32:0/3930783405 shutdown_connections 2021-04-09T18:25:00.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.387 7f670ca68700 1 -- 172.21.15.32:0/3930783405 wait complete. 2021-04-09T18:25:00.462 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:25:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.549 7fe0029c9700 1 Processor -- start 2021-04-09T18:25:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.549 7fe0029c9700 1 -- start start 2021-04-09T18:25:00.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe0029c9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0cffc0 0x7fdffc0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe0029c9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdffc0d6ca0 0x7fdffc10cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe0029c9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc0d72a0 con 0x7fdffc0d7820 2021-04-09T18:25:00.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe0029c9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdffc0d70a0 con 0x7fdffc0d6ca0 2021-04-09T18:25:00.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe0029c9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc0d85a0 con 0x7fdffc0cffc0 2021-04-09T18:25:00.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe000fe6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdffc0d6ca0 0x7fdffc10cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe000fe6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdffc0d6ca0 0x7fdffc10cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51848/0 (socket says 172.21.15.32:51848) 2021-04-09T18:25:00.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe000fe6700 1 -- 172.21.15.32:0/931169921 learned_addr learned my addr 172.21.15.32:0/931169921 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:00.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fdffbfff700 1 --2- 172.21.15.32:0/931169921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0cffc0 0x7fdffc0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe000fe6700 1 -- 172.21.15.32:0/931169921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0cffc0 msgr2=0x7fdffc0d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe000fe6700 1 --2- 172.21.15.32:0/931169921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0cffc0 0x7fdffc0d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe000fe6700 1 -- 172.21.15.32:0/931169921 >> v1:172.21.15.86:6789/0 conn(0x7fdffc0d7820 legacy=0x7fdffc0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fdffb7fe700 1 -- 172.21.15.32:0/931169921 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748740286 0 0) 0x7fdfe4000ff0 con 0x7fdffc0d7820 2021-04-09T18:25:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe000fe6700 1 -- 172.21.15.32:0/931169921 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc0d9200 con 0x7fdffc0d6ca0 2021-04-09T18:25:00.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fdffbfff700 1 --2- 172.21.15.32:0/931169921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0cffc0 0x7fdffc0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:00.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fe000fe6700 1 --2- 172.21.15.32:0/931169921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc0d6ca0 0x7fdffc10cff0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 rx=0x7fdff4006f00 tx=0x7fdff4009f10).ready entity=mon.0 client_cookie=e6e381caa931e301 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:00.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fdffb7fe700 1 -- 172.21.15.32:0/931169921 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdff400b3b0 con 0x7fdffc0d6ca0 2021-04-09T18:25:00.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fdffb7fe700 1 -- 172.21.15.32:0/931169921 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff400b730 con 0x7fdffc0d6ca0 2021-04-09T18:25:00.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.550 7fdffb7fe700 1 -- 172.21.15.32:0/931169921 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdff400bc10 con 0x7fdffc0d6ca0 2021-04-09T18:25:00.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.551 7fe0029c9700 1 -- 172.21.15.32:0/931169921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc0d6ca0 msgr2=0x7fdffc10cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.551 7fe0029c9700 1 --2- 172.21.15.32:0/931169921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc0d6ca0 0x7fdffc10cff0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 rx=0x7fdff4006f00 tx=0x7fdff4009f10).stop 2021-04-09T18:25:00.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.551 7fe0029c9700 1 -- 172.21.15.32:0/931169921 shutdown_connections 2021-04-09T18:25:00.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.551 7fe0029c9700 1 --2- 172.21.15.32:0/931169921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc0d6ca0 0x7fdffc10cff0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.551 7fe0029c9700 1 --2- 172.21.15.32:0/931169921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0cffc0 0x7fdffc0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.551 7fe0029c9700 1 -- 172.21.15.32:0/931169921 >> 172.21.15.32:0/931169921 conn(0x7fdffc0ce5e0 msgr2=0x7fdffc0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:00.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.551 7fe0029c9700 1 -- 172.21.15.32:0/931169921 shutdown_connections 2021-04-09T18:25:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.551 7fe0029c9700 1 -- 172.21.15.32:0/931169921 wait complete. 2021-04-09T18:25:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.571 7fe0029c9700 1 Processor -- start 2021-04-09T18:25:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fe0029c9700 1 -- start start 2021-04-09T18:25:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fe0029c9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc1862c0 0x7fdffc09ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fe0029c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc19ffa0 0x7fdffc189690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fe0029c9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc189cb0 0x7fdffc09dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fe0029c9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc0b1330 con 0x7fdffc19ffa0 2021-04-09T18:25:00.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fe0029c9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc09d6f0 con 0x7fdffc1862c0 2021-04-09T18:25:00.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fe0029c9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc09d460 con 0x7fdffc189cb0 2021-04-09T18:25:00.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fdff3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc19ffa0 0x7fdffc189690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fdffbfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc1862c0 0x7fdffc09ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fdff3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc19ffa0 0x7fdffc189690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51854/0 (socket says 172.21.15.32:51854) 2021-04-09T18:25:00.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fdff3fff700 1 -- 172.21.15.32:0/757330400 learned_addr learned my addr 172.21.15.32:0/757330400 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:00.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.572 7fe000fe6700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc189cb0 0x7fdffc09dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdff3fff700 1 -- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc189cb0 msgr2=0x7fdffc09dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdff3fff700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc189cb0 0x7fdffc09dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdff3fff700 1 -- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc1862c0 msgr2=0x7fdffc09ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdff3fff700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc1862c0 0x7fdffc09ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdff3fff700 1 -- 172.21.15.32:0/757330400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc099e90 con 0x7fdffc19ffa0 2021-04-09T18:25:00.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdffbfff700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc1862c0 0x7fdffc09ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:00.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdff3fff700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc19ffa0 0x7fdffc189690 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7fdfe4006ed0 tx=0x7fdfe400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:00.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdff97fa700 1 -- 172.21.15.32:0/757330400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfe4006020 con 0x7fdffc19ffa0 2021-04-09T18:25:00.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdff97fa700 1 -- 172.21.15.32:0/757330400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe4006320 con 0x7fdffc19ffa0 2021-04-09T18:25:00.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fdff97fa700 1 -- 172.21.15.32:0/757330400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfe4005730 con 0x7fdffc19ffa0 2021-04-09T18:25:00.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fe0029c9700 1 -- 172.21.15.32:0/757330400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdffc099e90 con 0x7fdffc19ffa0 2021-04-09T18:25:00.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.573 7fe0029c9700 1 -- 172.21.15.32:0/757330400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdffc099e90 con 0x7fdffc19ffa0 2021-04-09T18:25:00.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.574 7fdff97fa700 1 -- 172.21.15.32:0/757330400 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdfe40054b0 con 0x7fdffc19ffa0 2021-04-09T18:25:00.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.574 7fdff97fa700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe00188c0 0x7fdfe001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:00.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.574 7fdff97fa700 1 -- 172.21.15.32:0/757330400 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdfe4032a10 con 0x7fdffc19ffa0 2021-04-09T18:25:00.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.574 7fdffbfff700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe00188c0 0x7fdfe001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:00.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.574 7fdffbfff700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe00188c0 0x7fdfe001adc0 crc :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:00.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.593 7fe0029c9700 1 -- 172.21.15.32:0/757330400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdffc093e70 con 0x7fdffc19ffa0 2021-04-09T18:25:00.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.597 7fdff97fa700 1 -- 172.21.15.32:0/757330400 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdfe4006300 con 0x7fdffc19ffa0 2021-04-09T18:25:00.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.777 7fe0029c9700 1 -- 172.21.15.32:0/757330400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fdffc093e70 con 0x7fdffc19ffa0 2021-04-09T18:25:00.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.778 7fdff97fa700 1 -- 172.21.15.32:0/757330400 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v119) v1 ==== 104+0+10 (secure 0 0 0) 0x7fdfe4006300 con 0x7fdffc19ffa0 2021-04-09T18:25:00.843 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:00.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 -- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe00188c0 msgr2=0x7fdfe001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe00188c0 0x7fdfe001adc0 crc :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 -- 172.21.15.32:0/757330400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc19ffa0 msgr2=0x7fdffc189690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc19ffa0 0x7fdffc189690 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7fdfe4006ed0 tx=0x7fdfe400b8e0).stop 2021-04-09T18:25:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 -- 172.21.15.32:0/757330400 shutdown_connections 2021-04-09T18:25:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdfe00188c0 0x7fdfe001adc0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdffc19ffa0 0x7fdffc189690 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc189cb0 0x7fdffc09dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 --2- 172.21.15.32:0/757330400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc1862c0 0x7fdffc09ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:00.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 -- 172.21.15.32:0/757330400 >> 172.21.15.32:0/757330400 conn(0x7fdffc09fa50 msgr2=0x7fdffc0a2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 -- 172.21.15.32:0/757330400 shutdown_connections 2021-04-09T18:25:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.780 7fe0029c9700 1 -- 172.21.15.32:0/757330400 wait complete. 2021-04-09T18:25:00.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:25:01.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.943 7f60c1bdc700 1 Processor -- start 2021-04-09T18:25:01.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.943 7f60c1bdc700 1 -- start start 2021-04-09T18:25:01.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.944 7f60c1bdc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60bc10e7f0 0x7f60bc0d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.944 7f60c1bdc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60bc0d72a0 con 0x7f60bc0d6ca0 2021-04-09T18:25:01.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.944 7f60c1bdc700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60bc0d2460 con 0x7f60bc0cffc0 2021-04-09T18:25:01.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.944 7f60c1bdc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60bc0d70a0 con 0x7f60bc10e7f0 2021-04-09T18:25:01.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.944 7f60bbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60bc10e7f0 0x7f60bc0d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.944 7f60baffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f60bc0d6ca0 0x7f60bc10cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33220/0 (socket says 172.21.15.32:33220) 2021-04-09T18:25:01.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.944 7f60baffd700 1 -- 172.21.15.32:0/86852972 learned_addr learned my addr 172.21.15.32:0/86852972 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60ba7fc700 1 -- 172.21.15.32:0/86852972 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537630424 0 0) 0x7f60ac000c60 con 0x7f60bc0cffc0 2021-04-09T18:25:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60bbfff700 1 -- 172.21.15.32:0/86852972 >> v1:172.21.15.86:6789/0 conn(0x7f60bc0cffc0 legacy=0x7f60bc0d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60bbfff700 1 -- 172.21.15.32:0/86852972 >> v1:172.21.15.32:6789/0 conn(0x7f60bc0d6ca0 legacy=0x7f60bc10cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60bbfff700 1 -- 172.21.15.32:0/86852972 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc0d9200 con 0x7f60bc10e7f0 2021-04-09T18:25:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60bbfff700 1 --2- 172.21.15.32:0/86852972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc10e7f0 0x7f60bc0d8010 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 rx=0x7f60b4006ab0 tx=0x7f60b4009b40).ready entity=mon.0 client_cookie=e68011c3aa1feb95 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:01.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60ba7fc700 1 -- 172.21.15.32:0/86852972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b40022e0 con 0x7f60bc10e7f0 2021-04-09T18:25:01.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60ba7fc700 1 -- 172.21.15.32:0/86852972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b4002660 con 0x7f60bc10e7f0 2021-04-09T18:25:01.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60ba7fc700 1 -- 172.21.15.32:0/86852972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b4002b40 con 0x7f60bc10e7f0 2021-04-09T18:25:01.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60c1bdc700 1 -- 172.21.15.32:0/86852972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc10e7f0 msgr2=0x7f60bc0d8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.945 7f60c1bdc700 1 --2- 172.21.15.32:0/86852972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc10e7f0 0x7f60bc0d8010 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 rx=0x7f60b4006ab0 tx=0x7f60b4009b40).stop 2021-04-09T18:25:01.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.946 7f60c1bdc700 1 -- 172.21.15.32:0/86852972 shutdown_connections 2021-04-09T18:25:01.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.946 7f60c1bdc700 1 --2- 172.21.15.32:0/86852972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc10e7f0 0x7f60bc0d8010 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.946 7f60c1bdc700 1 -- 172.21.15.32:0/86852972 >> 172.21.15.32:0/86852972 conn(0x7f60bc0ce5e0 msgr2=0x7f60bc0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:01.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.946 7f60c1bdc700 1 -- 172.21.15.32:0/86852972 shutdown_connections 2021-04-09T18:25:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.946 7f60c1bdc700 1 -- 172.21.15.32:0/86852972 wait complete. 2021-04-09T18:25:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.966 7f60c1bdc700 1 Processor -- start 2021-04-09T18:25:01.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.966 7f60c1bdc700 1 -- start start 2021-04-09T18:25:01.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60c1bdc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc0c0de0 0x7f60bc0c1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60c1bdc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60bc0c20d0 0x7f60bc0b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60c1bdc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60bc0b94d0 0x7f60bc0b98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60c1bdc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0c15b0 con 0x7f60bc0c0de0 2021-04-09T18:25:01.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60c1bdc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0c2510 con 0x7f60bc0b94d0 2021-04-09T18:25:01.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60c1bdc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0c11e0 con 0x7f60bc0c20d0 2021-04-09T18:25:01.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc0c0de0 0x7f60bc0c1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bb7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc0c0de0 0x7f60bc0c1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51866/0 (socket says 172.21.15.32:51866) 2021-04-09T18:25:01.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bb7fe700 1 -- 172.21.15.32:0/672552561 learned_addr learned my addr 172.21.15.32:0/672552561 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:01.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bbfff700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60bc0b94d0 0x7f60bc0b98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60baffd700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60bc0c20d0 0x7f60bc0b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bb7fe700 1 -- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60bc0c20d0 msgr2=0x7f60bc0b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bb7fe700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60bc0c20d0 0x7f60bc0b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bb7fe700 1 -- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60bc0b94d0 msgr2=0x7f60bc0b98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bb7fe700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60bc0b94d0 0x7f60bc0b98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bb7fe700 1 -- 172.21.15.32:0/672552561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc0934c0 con 0x7f60bc0c0de0 2021-04-09T18:25:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60baffd700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60bc0c20d0 0x7f60bc0b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:01.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bbfff700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60bc0b94d0 0x7f60bc0b98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:01.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60bb7fe700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc0c0de0 0x7f60bc0c1ab0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7f60ac006f20 tx=0x7f60ac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:01.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60a3fff700 1 -- 172.21.15.32:0/672552561 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60ac00bd80 con 0x7f60bc0c0de0 2021-04-09T18:25:01.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60a3fff700 1 -- 172.21.15.32:0/672552561 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac001080 con 0x7f60bc0c0de0 2021-04-09T18:25:01.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60a3fff700 1 -- 172.21.15.32:0/672552561 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60ac001550 con 0x7f60bc0c0de0 2021-04-09T18:25:01.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc0934c0 con 0x7f60bc0c0de0 2021-04-09T18:25:01.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.967 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc0934c0 con 0x7f60bc0c0de0 2021-04-09T18:25:01.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.968 7f60a3fff700 1 -- 172.21.15.32:0/672552561 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f60ac001260 con 0x7f60bc0c0de0 2021-04-09T18:25:01.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.968 7f60a3fff700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.968 7f60a3fff700 1 -- 172.21.15.32:0/672552561 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f60ac032a40 con 0x7f60bc0c0de0 2021-04-09T18:25:01.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.968 7f60baffd700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.969 7f60baffd700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:01.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.988 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc08dba0 con 0x7f60bc0c0de0 2021-04-09T18:25:01.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:00.992 7f60a3fff700 1 -- 172.21.15.32:0/672552561 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f60ac006bf0 con 0x7f60bc0c0de0 2021-04-09T18:25:01.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.171 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f60bc08dba0 con 0x7f60bc0c0de0 2021-04-09T18:25:01.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.171 7f60a3fff700 1 -- 172.21.15.32:0/672552561 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v119) v1 ==== 100+0+10 (secure 0 0 0) 0x7f60ac001080 con 0x7f60bc0c0de0 2021-04-09T18:25:01.237 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:01.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 msgr2=0x7f609c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc0c0de0 msgr2=0x7f60bc0c1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc0c0de0 0x7f60bc0c1ab0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7f60ac006f20 tx=0x7f60ac00b990).stop 2021-04-09T18:25:01.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 shutdown_connections 2021-04-09T18:25:01.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f609c018930 0x7f609c01ae10 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60bc0b94d0 0x7f60bc0b98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60bc0c20d0 0x7f60bc0b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 --2- 172.21.15.32:0/672552561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60bc0c0de0 0x7f60bc0c1ab0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 >> 172.21.15.32:0/672552561 conn(0x7f60bc0a3640 msgr2=0x7f60bc08ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:01.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 shutdown_connections 2021-04-09T18:25:01.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.174 7f60c1bdc700 1 -- 172.21.15.32:0/672552561 wait complete. 2021-04-09T18:25:01.249 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:25:01.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:01.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.334 7efbe5b47700 1 Processor -- start 2021-04-09T18:25:01.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.334 7efbe5b47700 1 -- start start 2021-04-09T18:25:01.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbe5b47700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efbe010d010 0x7efbe00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbe5b47700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbe010d480 con 0x7efbe00d8490 2021-04-09T18:25:01.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbe5b47700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbe010e1f0 con 0x7efbe00d1da0 2021-04-09T18:25:01.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbe5b47700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbe00d8d30 con 0x7efbe010d010 2021-04-09T18:25:01.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbdffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efbe00d8490 0x7efbe010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33236/0 (socket says 172.21.15.32:33236) 2021-04-09T18:25:01.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbdffff700 1 -- 172.21.15.32:0/2943179685 learned_addr learned my addr 172.21.15.32:0/2943179685 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:01.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbdf7fe700 1 --2- 172.21.15.32:0/2943179685 >> v2:172.21.15.86:3300/0 conn(0x7efbe010d010 0x7efbe00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642275338 0 0) 0x7efbd8000d30 con 0x7efbe00d8490 2021-04-09T18:25:01.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbc8002fe0 con 0x7efbe00d8490 2021-04-09T18:25:01.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640373734 0 0) 0x7efbd4000c60 con 0x7efbe00d1da0 2021-04-09T18:25:01.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbc8003210 con 0x7efbe00d1da0 2021-04-09T18:25:01.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2837834505 0 0) 0x7efbd8001270 con 0x7efbe00d8490 2021-04-09T18:25:01.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 >> v2:172.21.15.86:3300/0 conn(0x7efbe010d010 msgr2=0x7efbe00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:01.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 --2- 172.21.15.32:0/2943179685 >> v2:172.21.15.86:3300/0 conn(0x7efbe010d010 0x7efbe00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 >> v1:172.21.15.86:6789/0 conn(0x7efbe00d1da0 legacy=0x7efbe00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe00d9220 con 0x7efbe00d8490 2021-04-09T18:25:01.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7efbd8001980 con 0x7efbe00d8490 2021-04-09T18:25:01.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 >> v1:172.21.15.32:6789/0 conn(0x7efbe00d8490 legacy=0x7efbe010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbc8003b90 0x7efbc8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbc80067a0 0x7efbc8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbc80095a0 0x7efbc800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbdf7fe700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbc8003b90 0x7efbc8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbc8050320 con 0x7efbc8003b90 2021-04-09T18:25:01.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbc8050500 con 0x7efbc80095a0 2021-04-09T18:25:01.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.335 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbc8050720 con 0x7efbc80067a0 2021-04-09T18:25:01.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdeffd700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbc80067a0 0x7efbc8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdffff700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbc80095a0 0x7efbc800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdf7fe700 1 -- 172.21.15.32:0/2943179685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbc80067a0 msgr2=0x7efbc8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:01.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdf7fe700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbc80067a0 0x7efbc8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdf7fe700 1 -- 172.21.15.32:0/2943179685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbc80095a0 msgr2=0x7efbc800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:01.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdf7fe700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbc80095a0 0x7efbc800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdf7fe700 1 -- 172.21.15.32:0/2943179685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbc80509c0 con 0x7efbc8003b90 2021-04-09T18:25:01.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdffff700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbc80095a0 0x7efbc800bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdffff700 1 -- 172.21.15.32:0/2943179685 reap_dead start 2021-04-09T18:25:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbdf7fe700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbc8003b90 0x7efbc8005fa0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7efbd00066a0 tx=0x7efbd000cc00).ready entity=mon.0 client_cookie=78da8909c34d7f1a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:01.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd0006470 con 0x7efbc8003b90 2021-04-09T18:25:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbd00090d0 con 0x7efbc8003b90 2021-04-09T18:25:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbde7fc700 1 -- 172.21.15.32:0/2943179685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd0009520 con 0x7efbc8003b90 2021-04-09T18:25:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbe5b47700 1 -- 172.21.15.32:0/2943179685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbc8003b90 msgr2=0x7efbc8005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.336 7efbe5b47700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbc8003b90 0x7efbc8005fa0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7efbd00066a0 tx=0x7efbd000cc00).stop 2021-04-09T18:25:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.337 7efbe5b47700 1 -- 172.21.15.32:0/2943179685 shutdown_connections 2021-04-09T18:25:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.337 7efbe5b47700 1 --2- 172.21.15.32:0/2943179685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbc8003b90 0x7efbc8005fa0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.337 7efbe5b47700 1 -- 172.21.15.32:0/2943179685 >> 172.21.15.32:0/2943179685 conn(0x7efbe00ce5f0 msgr2=0x7efbe00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.337 7efbe5b47700 1 -- 172.21.15.32:0/2943179685 shutdown_connections 2021-04-09T18:25:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.337 7efbe5b47700 1 -- 172.21.15.32:0/2943179685 wait complete. 2021-04-09T18:25:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.357 7efbe5b47700 1 Processor -- start 2021-04-09T18:25:01.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.357 7efbe5b47700 1 -- start start 2021-04-09T18:25:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbe5b47700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbe01ac380 0x7efbe018a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbe5b47700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbe01ac780 0x7efbe018a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbe5b47700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbe0183520 0x7efbe0183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbe5b47700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbe01ace10 con 0x7efbe0183520 2021-04-09T18:25:01.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbe5b47700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbe0189cf0 con 0x7efbe01ac380 2021-04-09T18:25:01.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbe5b47700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbe0183f40 con 0x7efbe01ac780 2021-04-09T18:25:01.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbe0183520 0x7efbe0183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbe0183520 0x7efbe0183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51890/0 (socket says 172.21.15.32:51890) 2021-04-09T18:25:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdffff700 1 -- 172.21.15.32:0/1874890992 learned_addr learned my addr 172.21.15.32:0/1874890992 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdeffd700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbe01ac780 0x7efbe018a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdf7fe700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbe01ac380 0x7efbe018a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdffff700 1 -- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbe01ac780 msgr2=0x7efbe018a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdffff700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbe01ac780 0x7efbe018a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdffff700 1 -- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbe01ac380 msgr2=0x7efbe018a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdffff700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbe01ac380 0x7efbe018a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdffff700 1 -- 172.21.15.32:0/1874890992 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe0184160 con 0x7efbe0183520 2021-04-09T18:25:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdf7fe700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbe01ac380 0x7efbe018a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdeffd700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbe01ac780 0x7efbe018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:01.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbdffff700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbe0183520 0x7efbe0183920 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 rx=0x7efbd8001c10 tx=0x7efbd800ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbc7fff700 1 -- 172.21.15.32:0/1874890992 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd8004930 con 0x7efbe0183520 2021-04-09T18:25:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbc7fff700 1 -- 172.21.15.32:0/1874890992 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbd8007900 con 0x7efbe0183520 2021-04-09T18:25:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbe0184160 con 0x7efbe0183520 2021-04-09T18:25:01.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbc7fff700 1 -- 172.21.15.32:0/1874890992 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd80070d0 con 0x7efbe0183520 2021-04-09T18:25:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.358 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbe0184160 con 0x7efbe0183520 2021-04-09T18:25:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.359 7efbc7fff700 1 -- 172.21.15.32:0/1874890992 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efbd8004720 con 0x7efbe0183520 2021-04-09T18:25:01.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.359 7efbc7fff700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbc0018930 0x7efbc001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.359 7efbc7fff700 1 -- 172.21.15.32:0/1874890992 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efbd8032db0 con 0x7efbe0183520 2021-04-09T18:25:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.359 7efbdf7fe700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbc0018930 0x7efbc001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.360 7efbdf7fe700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbc0018930 0x7efbc001adc0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.379 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbe0189cf0 con 0x7efbe0183520 2021-04-09T18:25:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.382 7efbc7fff700 1 -- 172.21.15.32:0/1874890992 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efbd8004cb0 con 0x7efbe0183520 2021-04-09T18:25:01.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.572 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efbe0189cf0 con 0x7efbe0183520 2021-04-09T18:25:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.573 7efbc7fff700 1 -- 172.21.15.32:0/1874890992 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2786 (secure 0 0 0) 0x7efbd8004720 con 0x7efbe0183520 2021-04-09T18:25:01.638 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 253 2021-04-09T18:25:01.639 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":253,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":8752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:24:56.975524","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:01.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbc0018930 msgr2=0x7efbc001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbc0018930 0x7efbc001adc0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbe0183520 msgr2=0x7efbe0183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbe0183520 0x7efbe0183920 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 rx=0x7efbd8001c10 tx=0x7efbd800ac60).stop 2021-04-09T18:25:01.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 shutdown_connections 2021-04-09T18:25:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbc0018930 0x7efbc001adc0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbe0183520 0x7efbe0183920 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbe01ac780 0x7efbe018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 --2- 172.21.15.32:0/1874890992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbe01ac380 0x7efbe018a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:01.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 >> 172.21.15.32:0/1874890992 conn(0x7efbe019b890 msgr2=0x7efbe01a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:01.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 shutdown_connections 2021-04-09T18:25:01.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:01.575 7efbe5b47700 1 -- 172.21.15.32:0/1874890992 wait complete. 2021-04-09T18:25:02.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 253, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:24:54.971184', 'modified': '2021-04-09 18:24:56.975524', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8740}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8740': {'gid': 8740, 'name': 'a', 'rank': 0, 'incarnation': 250, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2311632030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2311632030}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2311632030}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:02.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:25:02.055 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:25:02.192 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:02.192 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:25:02.192 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:25:02.193 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 8740, 2021-04-09T18:25:02.193 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:25:02.193 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:25:02.193 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1554528949999998, 2021-04-09T18:25:02.194 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 253, 2021-04-09T18:25:02.194 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 119, 2021-04-09T18:25:02.194 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 119, 2021-04-09T18:25:02.194 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 12.631246389999999 2021-04-09T18:25:02.195 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:02.198 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 8740, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.1554528949999998, "mdsmap_epoch": 253, "osdmap_epoch": 119, "osdmap_epoch_barrier": 119, "uptime": 12.631246389999999 } 2021-04-09T18:25:02.199 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:25:02.199 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:25:02.199 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:25:02.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:02.200 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:02.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:25:02.304 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:25:02.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:25:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:25:02.639 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:25:02.641 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:25:02.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:02.644 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:25:02.696 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:25:02.697 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:02.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:25:02.806 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:25:02.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:25:02.871 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:25:02.871 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:25:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:25:02.874 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:02.874 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:25:02.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:25:02.827 7f5c881c7f80 -1 init, newargv = 0x5617635802b0 newargc=9 2021-04-09T18:25:02.893 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[72320]: starting ceph client 2021-04-09T18:25:02.897 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[72320]: starting fuse 2021-04-09T18:25:02.921 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:25:02.921 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:25:02.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:02.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:25:02.924 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:25:02.924 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:25:02.925 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:25:02.926 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:25:02.926 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:02.927 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:25:02.928 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:25:02.928 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:25:02.929 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:02.930 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:25:02.930 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:25:02.931 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:25:02.932 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:25:02.932 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:02.933 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:25:02.934 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:25:02.935 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:25:02.935 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:25:02.936 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:25:02.936 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:02.937 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:25:02.938 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:25:03.042 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.72320.asok 2021-04-09T18:25:03.046 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72320.asok 2021-04-09T18:25:03.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:03.048 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72320.asok status 2021-04-09T18:25:03.172 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:03.172 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:25:03.173 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:25:03.173 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:25:03.173 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:25:03.173 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:25:03.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:25:03.174 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "72320", 2021-04-09T18:25:03.174 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:25:03.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:03.174 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:25:03.175 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:25:03.175 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9162, 2021-04-09T18:25:03.175 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:25:03.175 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:25:03.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:25:03.176 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 9162 2021-04-09T18:25:03.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:03.176 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:25:03.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:03.177 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:25:03.177 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3003679964 2021-04-09T18:25:03.177 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:03.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:03.178 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:25:03.178 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:03.178 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:25:03.178 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3003679964 2021-04-09T18:25:03.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:03.179 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.9162 172.21.15.32:0/3003679964", 2021-04-09T18:25:03.179 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/3003679964", 2021-04-09T18:25:03.179 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:25:03.179 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 253, 2021-04-09T18:25:03.179 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 119, 2021-04-09T18:25:03.180 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:25:03.180 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:25:03.180 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:25:03.180 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:03.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:03.182 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:03.182 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:03.227 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:25:03.227 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:25:03.228 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:03.228 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:03.288 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:03.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.372 7fdc63e16700 1 Processor -- start 2021-04-09T18:25:03.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.372 7fdc63e16700 1 -- start start 2021-04-09T18:25:03.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.372 7fdc63e16700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c10d010 0x7fdc5c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.372 7fdc63e16700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdc5c0d8490 0x7fdc5c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc63e16700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc5c10d480 con 0x7fdc5c0d1da0 2021-04-09T18:25:03.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc63e16700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5c10e780 con 0x7fdc5c0d8490 2021-04-09T18:25:03.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc63e16700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c0d8d30 con 0x7fdc5c10d010 2021-04-09T18:25:03.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc61431700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdc5c0d1da0 0x7fdc5c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33270/0 (socket says 172.21.15.32:33270) 2021-04-09T18:25:03.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc61431700 1 -- 172.21.15.32:0/1807934723 learned_addr learned my addr 172.21.15.32:0/1807934723 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:03.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc61c32700 1 --2- 172.21.15.32:0/1807934723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c10d010 0x7fdc5c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc62433700 1 --2- 172.21.15.32:0/1807934723 >> v2:172.21.15.86:3300/0 conn(0x7fdc5c0d8490 0x7fdc5c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc60c30700 1 -- 172.21.15.32:0/1807934723 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196873814 0 0) 0x7fdc50000ce0 con 0x7fdc5c0d1da0 2021-04-09T18:25:03.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc60c30700 1 -- 172.21.15.32:0/1807934723 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc48002fe0 con 0x7fdc5c0d1da0 2021-04-09T18:25:03.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc62433700 1 -- 172.21.15.32:0/1807934723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c10d010 msgr2=0x7fdc5c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:03.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc62433700 1 --2- 172.21.15.32:0/1807934723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c10d010 0x7fdc5c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc60c30700 1 -- 172.21.15.32:0/1807934723 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1443126449 0 0) 0x7fdc500010e0 con 0x7fdc5c0d1da0 2021-04-09T18:25:03.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc62433700 1 -- 172.21.15.32:0/1807934723 >> v1:172.21.15.32:6789/0 conn(0x7fdc5c0d1da0 legacy=0x7fdc5c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc62433700 1 -- 172.21.15.32:0/1807934723 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c0d9220 con 0x7fdc5c0d8490 2021-04-09T18:25:03.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc61c32700 1 --2- 172.21.15.32:0/1807934723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c10d010 0x7fdc5c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:03.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.373 7fdc62433700 1 --2- 172.21.15.32:0/1807934723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc5c0d8490 0x7fdc5c10da00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fdc54006e50 tx=0x7fdc54009e70).ready entity=mon.1 client_cookie=a7452786bd39f925 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:03.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.374 7fdc60c30700 1 -- 172.21.15.32:0/1807934723 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc54006a20 con 0x7fdc5c0d8490 2021-04-09T18:25:03.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.374 7fdc60c30700 1 -- 172.21.15.32:0/1807934723 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc54006790 con 0x7fdc5c0d8490 2021-04-09T18:25:03.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.374 7fdc60c30700 1 -- 172.21.15.32:0/1807934723 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc5400b700 con 0x7fdc5c0d8490 2021-04-09T18:25:03.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.374 7fdc63e16700 1 -- 172.21.15.32:0/1807934723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc5c0d8490 msgr2=0x7fdc5c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.374 7fdc63e16700 1 --2- 172.21.15.32:0/1807934723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc5c0d8490 0x7fdc5c10da00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fdc54006e50 tx=0x7fdc54009e70).stop 2021-04-09T18:25:03.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.374 7fdc63e16700 1 -- 172.21.15.32:0/1807934723 shutdown_connections 2021-04-09T18:25:03.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.374 7fdc63e16700 1 --2- 172.21.15.32:0/1807934723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc5c0d8490 0x7fdc5c10da00 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.374 7fdc63e16700 1 --2- 172.21.15.32:0/1807934723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c10d010 0x7fdc5c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.374 7fdc63e16700 1 -- 172.21.15.32:0/1807934723 >> 172.21.15.32:0/1807934723 conn(0x7fdc5c0ce5f0 msgr2=0x7fdc5c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:03.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.375 7fdc63e16700 1 -- 172.21.15.32:0/1807934723 shutdown_connections 2021-04-09T18:25:03.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.375 7fdc63e16700 1 -- 172.21.15.32:0/1807934723 wait complete. 2021-04-09T18:25:03.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc63e16700 1 Processor -- start 2021-04-09T18:25:03.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc63e16700 1 -- start start 2021-04-09T18:25:03.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc63e16700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc5c0ac330 0x7fdc5c1ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc63e16700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c17f350 0x7fdc5c17f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc63e16700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc5c17da20 0x7fdc5c17de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc63e16700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c1a42f0 con 0x7fdc5c17da20 2021-04-09T18:25:03.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc63e16700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c17fe90 con 0x7fdc5c0ac330 2021-04-09T18:25:03.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc63e16700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c17e440 con 0x7fdc5c17f350 2021-04-09T18:25:03.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc62433700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc5c17da20 0x7fdc5c17de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc62433700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc5c17da20 0x7fdc5c17de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51916/0 (socket says 172.21.15.32:51916) 2021-04-09T18:25:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc62433700 1 -- 172.21.15.32:0/1268660110 learned_addr learned my addr 172.21.15.32:0/1268660110 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc61c32700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc5c0ac330 0x7fdc5c1ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.395 7fdc61431700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c17f350 0x7fdc5c17f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc62433700 1 -- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c17f350 msgr2=0x7fdc5c17f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc62433700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c17f350 0x7fdc5c17f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc62433700 1 -- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc5c0ac330 msgr2=0x7fdc5c1ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc62433700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc5c0ac330 0x7fdc5c1ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc62433700 1 -- 172.21.15.32:0/1268660110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c1ac8e0 con 0x7fdc5c17da20 2021-04-09T18:25:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc62433700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc5c17da20 0x7fdc5c17de20 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 rx=0x7fdc54006e60 tx=0x7fdc5400bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc4e7fc700 1 -- 172.21.15.32:0/1268660110 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc54000df0 con 0x7fdc5c17da20 2021-04-09T18:25:03.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc4e7fc700 1 -- 172.21.15.32:0/1268660110 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5400bdb0 con 0x7fdc5c17da20 2021-04-09T18:25:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc5c1ac8e0 con 0x7fdc5c17da20 2021-04-09T18:25:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc4e7fc700 1 -- 172.21.15.32:0/1268660110 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc540093d0 con 0x7fdc5c17da20 2021-04-09T18:25:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.396 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5c1ac8e0 con 0x7fdc5c17da20 2021-04-09T18:25:03.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.397 7fdc4e7fc700 1 -- 172.21.15.32:0/1268660110 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdc540056b0 con 0x7fdc5c17da20 2021-04-09T18:25:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.397 7fdc4e7fc700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdc44018930 0x7fdc4401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.397 7fdc4e7fc700 1 -- 172.21.15.32:0/1268660110 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdc54032d70 con 0x7fdc5c17da20 2021-04-09T18:25:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.397 7fdc61c32700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdc44018930 0x7fdc4401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.397 7fdc61c32700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdc44018930 0x7fdc4401ae10 crc :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:03.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.417 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5c1a42f0 con 0x7fdc5c17da20 2021-04-09T18:25:03.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.420 7fdc4e7fc700 1 -- 172.21.15.32:0/1268660110 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdc540093d0 con 0x7fdc5c17da20 2021-04-09T18:25:03.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.612 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc5c1a42f0 con 0x7fdc5c17da20 2021-04-09T18:25:03.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.612 7fdc4e7fc700 1 -- 172.21.15.32:0/1268660110 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fdc540056b0 con 0x7fdc5c17da20 2021-04-09T18:25:03.680 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 253 2021-04-09T18:25:03.681 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:03.681 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":253,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":8752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":253,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:24:56.975524","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdc44018930 msgr2=0x7fdc4401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdc44018930 0x7fdc4401ae10 crc :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc5c17da20 msgr2=0x7fdc5c17de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc5c17da20 0x7fdc5c17de20 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 rx=0x7fdc54006e60 tx=0x7fdc5400bc70).stop 2021-04-09T18:25:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 shutdown_connections 2021-04-09T18:25:03.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdc44018930 0x7fdc4401ae10 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdc5c17da20 0x7fdc5c17de20 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc5c17f350 0x7fdc5c17f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 --2- 172.21.15.32:0/1268660110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc5c0ac330 0x7fdc5c1ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 >> 172.21.15.32:0/1268660110 conn(0x7fdc5c08cec0 msgr2=0x7fdc5c08ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:03.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 shutdown_connections 2021-04-09T18:25:03.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.615 7fdc63e16700 1 -- 172.21.15.32:0/1268660110 wait complete. 2021-04-09T18:25:03.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2021-04-09T18:25:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.777 7f79edc06700 1 Processor -- start 2021-04-09T18:25:03.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.777 7f79edc06700 1 -- start start 2021-04-09T18:25:03.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79edc06700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e810d010 0x7f79e80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79edc06700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79e80d8490 0x7f79e810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79edc06700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e810d480 con 0x7f79e80d1da0 2021-04-09T18:25:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79edc06700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e810e780 con 0x7f79e80d8490 2021-04-09T18:25:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79edc06700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80d8d30 con 0x7f79e810d010 2021-04-09T18:25:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79e80d8490 0x7f79e810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e6ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f79e80d1da0 0x7f79e80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33284/0 (socket says 172.21.15.32:33284) 2021-04-09T18:25:03.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e6ffd700 1 -- 172.21.15.32:0/171124382 learned_addr learned my addr 172.21.15.32:0/171124382 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e77fe700 1 --2- 172.21.15.32:0/171124382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e810d010 0x7f79e80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e67fc700 1 -- 172.21.15.32:0/171124382 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703881251 0 0) 0x7f79dc0009b0 con 0x7f79e80d1da0 2021-04-09T18:25:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e67fc700 1 -- 172.21.15.32:0/171124382 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d0002fe0 con 0x7f79e80d1da0 2021-04-09T18:25:03.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e77fe700 1 -- 172.21.15.32:0/171124382 >> v2:172.21.15.32:3300/0 conn(0x7f79e80d8490 msgr2=0x7f79e810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e77fe700 1 --2- 172.21.15.32:0/171124382 >> v2:172.21.15.32:3300/0 conn(0x7f79e80d8490 0x7f79e810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e77fe700 1 -- 172.21.15.32:0/171124382 >> v1:172.21.15.32:6789/0 conn(0x7f79e80d1da0 legacy=0x7f79e80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e77fe700 1 -- 172.21.15.32:0/171124382 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e80d9220 con 0x7f79e810d010 2021-04-09T18:25:03.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e7fff700 1 --2- 172.21.15.32:0/171124382 >> v2:172.21.15.32:3300/0 conn(0x7f79e80d8490 0x7f79e810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:03.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.778 7f79e77fe700 1 --2- 172.21.15.32:0/171124382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e810d010 0x7f79e80d1780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f79d8006650 tx=0x7f79d8009a80).ready entity=mon.2 client_cookie=f051b954c979da37 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:03.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79e67fc700 1 -- 172.21.15.32:0/171124382 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79d80009b0 con 0x7f79e810d010 2021-04-09T18:25:03.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79e67fc700 1 -- 172.21.15.32:0/171124382 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d800c0c0 con 0x7f79e810d010 2021-04-09T18:25:03.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79e67fc700 1 -- 172.21.15.32:0/171124382 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79d800c510 con 0x7f79e810d010 2021-04-09T18:25:03.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79edc06700 1 -- 172.21.15.32:0/171124382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e810d010 msgr2=0x7f79e80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79edc06700 1 --2- 172.21.15.32:0/171124382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e810d010 0x7f79e80d1780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f79d8006650 tx=0x7f79d8009a80).stop 2021-04-09T18:25:03.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79edc06700 1 -- 172.21.15.32:0/171124382 shutdown_connections 2021-04-09T18:25:03.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79edc06700 1 --2- 172.21.15.32:0/171124382 >> v2:172.21.15.32:3300/0 conn(0x7f79e80d8490 0x7f79e810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79edc06700 1 --2- 172.21.15.32:0/171124382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e810d010 0x7f79e80d1780 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79edc06700 1 -- 172.21.15.32:0/171124382 >> 172.21.15.32:0/171124382 conn(0x7f79e80ce5f0 msgr2=0x7f79e80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.779 7f79edc06700 1 -- 172.21.15.32:0/171124382 shutdown_connections 2021-04-09T18:25:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.780 7f79edc06700 1 -- 172.21.15.32:0/171124382 wait complete. 2021-04-09T18:25:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.800 7f79edc06700 1 Processor -- start 2021-04-09T18:25:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.800 7f79edc06700 1 -- start start 2021-04-09T18:25:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.800 7f79edc06700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e80b2140 0x7f79e808b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.800 7f79edc06700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79e809f550 0x7f79e809f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.800 7f79edc06700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8197c50 0x7f79e8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.800 7f79edc06700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e809f010 con 0x7f79e809f550 2021-04-09T18:25:03.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.800 7f79edc06700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e809ff70 con 0x7f79e8197c50 2021-04-09T18:25:03.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.800 7f79edc06700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8198670 con 0x7f79e80b2140 2021-04-09T18:25:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.800 7f79e6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79e809f550 0x7f79e809f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79e809f550 0x7f79e809f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51930/0 (socket says 172.21.15.32:51930) 2021-04-09T18:25:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e6ffd700 1 -- 172.21.15.32:0/87897856 learned_addr learned my addr 172.21.15.32:0/87897856 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:03.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e77fe700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e80b2140 0x7f79e808b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e7fff700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8197c50 0x7f79e8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e6ffd700 1 -- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e80b2140 msgr2=0x7f79e808b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e6ffd700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e80b2140 0x7f79e808b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e6ffd700 1 -- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8197c50 msgr2=0x7f79e8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:03.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e6ffd700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8197c50 0x7f79e8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:03.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e6ffd700 1 -- 172.21.15.32:0/87897856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e8198890 con 0x7f79e809f550 2021-04-09T18:25:03.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79e6ffd700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79e809f550 0x7f79e809f950 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f79dc001c60 tx=0x7f79dc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:03.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79cffff700 1 -- 172.21.15.32:0/87897856 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79dc00acc0 con 0x7f79e809f550 2021-04-09T18:25:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79cffff700 1 -- 172.21.15.32:0/87897856 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc004140 con 0x7f79e809f550 2021-04-09T18:25:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79edc06700 1 -- 172.21.15.32:0/87897856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e8198890 con 0x7f79e809f550 2021-04-09T18:25:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79cffff700 1 -- 172.21.15.32:0/87897856 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79dc004620 con 0x7f79e809f550 2021-04-09T18:25:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.801 7f79edc06700 1 -- 172.21.15.32:0/87897856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e8198890 con 0x7f79e809f550 2021-04-09T18:25:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.802 7f79cffff700 1 -- 172.21.15.32:0/87897856 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f79dc0047c0 con 0x7f79e809f550 2021-04-09T18:25:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.802 7f79cffff700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79c8018930 0x7f79c801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.802 7f79cffff700 1 -- 172.21.15.32:0/87897856 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f79dc0329e0 con 0x7f79e809f550 2021-04-09T18:25:03.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.802 7f79e77fe700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79c8018930 0x7f79c801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:03.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.802 7f79e77fe700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79c8018930 0x7f79c801ae10 crc :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:03.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.822 7f79edc06700 1 -- 172.21.15.32:0/87897856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e809f010 con 0x7f79e809f550 2021-04-09T18:25:03.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:03.825 7f79cffff700 1 -- 172.21.15.32:0/87897856 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f79dc0045b0 con 0x7f79e809f550 2021-04-09T18:25:04.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:04.017 7f79edc06700 1 -- 172.21.15.32:0/87897856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "5"} v 0) v1 -- 0x7f79e809f010 con 0x7f79e809f550 2021-04-09T18:25:05.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:04.995 7f79cffff700 1 -- 172.21.15.32:0/87897856 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "5"}]=0 v254) v1 ==== 105+0+0 (secure 0 0 0) 0x7f79dc0047c0 con 0x7f79e809f550 2021-04-09T18:25:05.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.036 7f79edc06700 1 -- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79c8018930 msgr2=0x7f79c801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.036 7f79edc06700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79c8018930 0x7f79c801ae10 crc :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.036 7f79edc06700 1 -- 172.21.15.32:0/87897856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79e809f550 msgr2=0x7f79e809f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.036 7f79edc06700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79e809f550 0x7f79e809f950 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f79dc001c60 tx=0x7f79dc00a8d0).stop 2021-04-09T18:25:05.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.037 7f79edc06700 1 -- 172.21.15.32:0/87897856 shutdown_connections 2021-04-09T18:25:05.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.037 7f79edc06700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79c8018930 0x7f79c801ae10 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.037 7f79edc06700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79e809f550 0x7f79e809f950 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.037 7f79edc06700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8197c50 0x7f79e8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.037 7f79edc06700 1 --2- 172.21.15.32:0/87897856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e80b2140 0x7f79e808b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.037 7f79edc06700 1 -- 172.21.15.32:0/87897856 >> 172.21.15.32:0/87897856 conn(0x7f79e80b27e0 msgr2=0x7f79e81b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:05.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.037 7f79edc06700 1 -- 172.21.15.32:0/87897856 shutdown_connections 2021-04-09T18:25:05.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.037 7f79edc06700 1 -- 172.21.15.32:0/87897856 wait complete. 2021-04-09T18:25:05.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:25:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.198 7fd00a1f9700 1 Processor -- start 2021-04-09T18:25:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.198 7fd00a1f9700 1 -- start start 2021-04-09T18:25:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fd00a1f9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd00410e7d0 0x7fd0040d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fd00a1f9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0040d7280 con 0x7fd0040d6c80 2021-04-09T18:25:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fd00a1f9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0040d2440 con 0x7fd0040cffa0 2021-04-09T18:25:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fd00a1f9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0040d7080 con 0x7fd00410e7d0 2021-04-09T18:25:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fd008816700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd00410e7d0 0x7fd0040d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fcffbfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd0040d6c80 0x7fd00410cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33298/0 (socket says 172.21.15.32:33298) 2021-04-09T18:25:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fd008816700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd00410e7d0 0x7fd0040d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51938/0 (socket says 172.21.15.32:51938) 2021-04-09T18:25:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fcffbfff700 1 -- 172.21.15.32:0/3929358244 learned_addr learned my addr 172.21.15.32:0/3929358244 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fd0037fe700 1 -- 172.21.15.32:0/3929358244 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286261805 0 0) 0x7fcff4000c60 con 0x7fd0040cffa0 2021-04-09T18:25:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fd0037fe700 1 -- 172.21.15.32:0/3929358244 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff0002fe0 con 0x7fd0040cffa0 2021-04-09T18:25:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.199 7fd0037fe700 1 -- 172.21.15.32:0/3929358244 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534134215 0 0) 0x7fcfec000ce0 con 0x7fd0040d6c80 2021-04-09T18:25:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd0037fe700 1 -- 172.21.15.32:0/3929358244 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff0002fe0 con 0x7fd0040d6c80 2021-04-09T18:25:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd008816700 1 -- 172.21.15.32:0/3929358244 >> v1:172.21.15.86:6789/0 conn(0x7fd0040cffa0 legacy=0x7fd0040d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd008816700 1 -- 172.21.15.32:0/3929358244 >> v1:172.21.15.32:6789/0 conn(0x7fd0040d6c80 legacy=0x7fd00410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd008816700 1 -- 172.21.15.32:0/3929358244 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0040d91e0 con 0x7fd00410e7d0 2021-04-09T18:25:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd008816700 1 --2- 172.21.15.32:0/3929358244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00410e7d0 0x7fd0040d7ff0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7fcffc006e30 tx=0x7fcffc009de0).ready entity=mon.0 client_cookie=d4e098e6f72dd29f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd0037fe700 1 -- 172.21.15.32:0/3929358244 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcffc00b300 con 0x7fd00410e7d0 2021-04-09T18:25:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd0037fe700 1 -- 172.21.15.32:0/3929358244 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc00b680 con 0x7fd00410e7d0 2021-04-09T18:25:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd0037fe700 1 -- 172.21.15.32:0/3929358244 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcffc00bb60 con 0x7fd00410e7d0 2021-04-09T18:25:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd00a1f9700 1 -- 172.21.15.32:0/3929358244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00410e7d0 msgr2=0x7fd0040d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.200 7fd00a1f9700 1 --2- 172.21.15.32:0/3929358244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00410e7d0 0x7fd0040d7ff0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7fcffc006e30 tx=0x7fcffc009de0).stop 2021-04-09T18:25:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.201 7fd00a1f9700 1 -- 172.21.15.32:0/3929358244 shutdown_connections 2021-04-09T18:25:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.201 7fd00a1f9700 1 --2- 172.21.15.32:0/3929358244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00410e7d0 0x7fd0040d7ff0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.201 7fd00a1f9700 1 -- 172.21.15.32:0/3929358244 >> 172.21.15.32:0/3929358244 conn(0x7fd0040ce5c0 msgr2=0x7fd0040cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.201 7fd00a1f9700 1 -- 172.21.15.32:0/3929358244 shutdown_connections 2021-04-09T18:25:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.201 7fd00a1f9700 1 -- 172.21.15.32:0/3929358244 wait complete. 2021-04-09T18:25:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.221 7fd00a1f9700 1 Processor -- start 2021-04-09T18:25:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.221 7fd00a1f9700 1 -- start start 2021-04-09T18:25:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd00a1f9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0040b4250 0x7fd00409a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd00a1f9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00418e280 0x7fd004192370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd00a1f9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd004184d80 0x7fd004191250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd00a1f9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd004192a10 con 0x7fd00418e280 2021-04-09T18:25:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd00a1f9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd004191df0 con 0x7fd0040b4250 2021-04-09T18:25:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd00a1f9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd004191870 con 0x7fd004184d80 2021-04-09T18:25:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fcffbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00418e280 0x7fd004192370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fcffbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00418e280 0x7fd004192370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51942/0 (socket says 172.21.15.32:51942) 2021-04-09T18:25:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fcffbfff700 1 -- 172.21.15.32:0/2002501473 learned_addr learned my addr 172.21.15.32:0/2002501473 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd008816700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd004184d80 0x7fd004191250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd003fff700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0040b4250 0x7fd00409a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fcffbfff700 1 -- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd004184d80 msgr2=0x7fd004191250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fcffbfff700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd004184d80 0x7fd004191250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fcffbfff700 1 -- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0040b4250 msgr2=0x7fd00409a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fcffbfff700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0040b4250 0x7fd00409a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fcffbfff700 1 -- 172.21.15.32:0/2002501473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd004191a90 con 0x7fd00418e280 2021-04-09T18:25:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fcffbfff700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00418e280 0x7fd004192370 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7fcfec00b5c0 tx=0x7fcfec00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd0017fa700 1 -- 172.21.15.32:0/2002501473 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfec006030 con 0x7fd00418e280 2021-04-09T18:25:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd0017fa700 1 -- 172.21.15.32:0/2002501473 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec0043a0 con 0x7fd00418e280 2021-04-09T18:25:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd004191a90 con 0x7fd00418e280 2021-04-09T18:25:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd0017fa700 1 -- 172.21.15.32:0/2002501473 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfec0048c0 con 0x7fd00418e280 2021-04-09T18:25:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.222 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd004191a90 con 0x7fd00418e280 2021-04-09T18:25:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.223 7fd0017fa700 1 -- 172.21.15.32:0/2002501473 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcfec004580 con 0x7fd00418e280 2021-04-09T18:25:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.223 7fd0017fa700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfe8018930 0x7fcfe801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.223 7fd0017fa700 1 -- 172.21.15.32:0/2002501473 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcfec032a60 con 0x7fd00418e280 2021-04-09T18:25:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.223 7fd003fff700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfe8018930 0x7fcfe801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.224 7fd003fff700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfe8018930 0x7fcfe801adc0 crc :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:05.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.243 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd004182d60 con 0x7fd00418e280 2021-04-09T18:25:05.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.246 7fd0017fa700 1 -- 172.21.15.32:0/2002501473 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcfec006b80 con 0x7fd00418e280 2021-04-09T18:25:05.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.432 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd004182d60 con 0x7fd00418e280 2021-04-09T18:25:05.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.432 7fd0017fa700 1 -- 172.21.15.32:0/2002501473 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+160 (secure 0 0 0) 0x7fcfec00bd50 con 0x7fd00418e280 2021-04-09T18:25:05.497 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfe8018930 msgr2=0x7fcfe801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfe8018930 0x7fcfe801adc0 crc :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00418e280 msgr2=0x7fd004192370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00418e280 0x7fd004192370 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7fcfec00b5c0 tx=0x7fcfec00b960).stop 2021-04-09T18:25:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 shutdown_connections 2021-04-09T18:25:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfe8018930 0x7fcfe801adc0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd00418e280 0x7fd004192370 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd004184d80 0x7fd004191250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 --2- 172.21.15.32:0/2002501473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0040b4250 0x7fd00409a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 >> 172.21.15.32:0/2002501473 conn(0x7fd004090a90 msgr2=0x7fd0041832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 shutdown_connections 2021-04-09T18:25:05.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.435 7fd00a1f9700 1 -- 172.21.15.32:0/2002501473 wait complete. 2021-04-09T18:25:05.509 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds"}}},"status":"HEALTH_WARN"} 2021-04-09T18:25:05.509 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2021-04-09T18:25:05.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:25:05.636 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:05.637 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:25:05.637 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:05.642 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:25:05.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:05.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.727 7fcb5b0d8700 1 Processor -- start 2021-04-09T18:25:05.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb5b0d8700 1 -- start start 2021-04-09T18:25:05.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb5b0d8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcb540d6c80 0x7fcb540d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb5b0d8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb540d2440 0x7fcb5410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb5b0d8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb540d7280 con 0x7fcb540cffa0 2021-04-09T18:25:05.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb5b0d8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb540d7080 con 0x7fcb540d2440 2021-04-09T18:25:05.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb5b0d8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb540d91e0 con 0x7fcb540d6c80 2021-04-09T18:25:05.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb596f5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcb540d2440 0x7fcb5410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb58ef4700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcb540cffa0 0x7fcb540d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33308/0 (socket says 172.21.15.32:33308) 2021-04-09T18:25:05.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb58ef4700 1 -- 172.21.15.32:0/1467155078 learned_addr learned my addr 172.21.15.32:0/1467155078 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:05.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.728 7fcb53fff700 1 --2- 172.21.15.32:0/1467155078 >> v2:172.21.15.86:3300/0 conn(0x7fcb540d6c80 0x7fcb540d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb537fe700 1 -- 172.21.15.32:0/1467155078 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916924376 0 0) 0x7fcb44000ce0 con 0x7fcb540cffa0 2021-04-09T18:25:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb537fe700 1 -- 172.21.15.32:0/1467155078 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb3c002fe0 con 0x7fcb540cffa0 2021-04-09T18:25:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb596f5700 1 -- 172.21.15.32:0/1467155078 >> v2:172.21.15.86:3300/0 conn(0x7fcb540d6c80 msgr2=0x7fcb540d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb596f5700 1 --2- 172.21.15.32:0/1467155078 >> v2:172.21.15.86:3300/0 conn(0x7fcb540d6c80 0x7fcb540d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb596f5700 1 -- 172.21.15.32:0/1467155078 >> v1:172.21.15.32:6789/0 conn(0x7fcb540cffa0 legacy=0x7fcb540d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb596f5700 1 -- 172.21.15.32:0/1467155078 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb540d2840 con 0x7fcb540d2440 2021-04-09T18:25:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb596f5700 1 --2- 172.21.15.32:0/1467155078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb540d2440 0x7fcb5410cfd0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 rx=0x7fcb4c006e30 tx=0x7fcb4c009de0).ready entity=mon.0 client_cookie=4ad0beddf2041983 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb537fe700 1 -- 172.21.15.32:0/1467155078 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4c00b300 con 0x7fcb540d2440 2021-04-09T18:25:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb537fe700 1 -- 172.21.15.32:0/1467155078 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4c00b680 con 0x7fcb540d2440 2021-04-09T18:25:05.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb537fe700 1 -- 172.21.15.32:0/1467155078 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4c00bb60 con 0x7fcb540d2440 2021-04-09T18:25:05.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb5b0d8700 1 -- 172.21.15.32:0/1467155078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb540d2440 msgr2=0x7fcb5410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.729 7fcb5b0d8700 1 --2- 172.21.15.32:0/1467155078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb540d2440 0x7fcb5410cfd0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 rx=0x7fcb4c006e30 tx=0x7fcb4c009de0).stop 2021-04-09T18:25:05.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.730 7fcb5b0d8700 1 -- 172.21.15.32:0/1467155078 shutdown_connections 2021-04-09T18:25:05.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.730 7fcb5b0d8700 1 --2- 172.21.15.32:0/1467155078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb540d2440 0x7fcb5410cfd0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.730 7fcb5b0d8700 1 --2- 172.21.15.32:0/1467155078 >> v2:172.21.15.86:3300/0 conn(0x7fcb540d6c80 0x7fcb540d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.730 7fcb5b0d8700 1 -- 172.21.15.32:0/1467155078 >> 172.21.15.32:0/1467155078 conn(0x7fcb540ce5c0 msgr2=0x7fcb540cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:05.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.730 7fcb5b0d8700 1 -- 172.21.15.32:0/1467155078 shutdown_connections 2021-04-09T18:25:05.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.730 7fcb5b0d8700 1 -- 172.21.15.32:0/1467155078 wait complete. 2021-04-09T18:25:05.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.750 7fcb5b0d8700 1 Processor -- start 2021-04-09T18:25:05.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.750 7fcb5b0d8700 1 -- start start 2021-04-09T18:25:05.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb5b0d8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541acff0 0x7fcb5408fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb5b0d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb54090150 0x7fcb5408c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb5b0d8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb5408cca0 0x7fcb5408e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb5b0d8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5408c100 con 0x7fcb54090150 2021-04-09T18:25:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb5b0d8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5408d0a0 con 0x7fcb5408cca0 2021-04-09T18:25:05.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb5b0d8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5408e960 con 0x7fcb541acff0 2021-04-09T18:25:05.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb53fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb54090150 0x7fcb5408c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb53fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb54090150 0x7fcb5408c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51958/0 (socket says 172.21.15.32:51958) 2021-04-09T18:25:05.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb53fff700 1 -- 172.21.15.32:0/2497510476 learned_addr learned my addr 172.21.15.32:0/2497510476 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb596f5700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb5408cca0 0x7fcb5408e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb58ef4700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541acff0 0x7fcb5408fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb53fff700 1 -- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541acff0 msgr2=0x7fcb5408fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb53fff700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541acff0 0x7fcb5408fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb53fff700 1 -- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb5408cca0 msgr2=0x7fcb5408e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:05.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb53fff700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb5408cca0 0x7fcb5408e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:05.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb53fff700 1 -- 172.21.15.32:0/2497510476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5408eb80 con 0x7fcb54090150 2021-04-09T18:25:05.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb58ef4700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541acff0 0x7fcb5408fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb53fff700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb54090150 0x7fcb5408c680 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7fcb48006f60 tx=0x7fcb4800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.751 7fcb517fa700 1 -- 172.21.15.32:0/2497510476 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4800bd60 con 0x7fcb54090150 2021-04-09T18:25:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.752 7fcb517fa700 1 -- 172.21.15.32:0/2497510476 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb48000c90 con 0x7fcb54090150 2021-04-09T18:25:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.752 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5408eb80 con 0x7fcb54090150 2021-04-09T18:25:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.752 7fcb517fa700 1 -- 172.21.15.32:0/2497510476 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb48002220 con 0x7fcb54090150 2021-04-09T18:25:05.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.752 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5408eb80 con 0x7fcb54090150 2021-04-09T18:25:05.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.752 7fcb517fa700 1 -- 172.21.15.32:0/2497510476 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcb480052e0 con 0x7fcb54090150 2021-04-09T18:25:05.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.752 7fcb517fa700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb340188d0 0x7fcb3401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:05.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.752 7fcb517fa700 1 -- 172.21.15.32:0/2497510476 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcb480329f0 con 0x7fcb54090150 2021-04-09T18:25:05.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.752 7fcb58ef4700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb340188d0 0x7fcb3401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:05.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.753 7fcb58ef4700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb340188d0 0x7fcb3401add0 crc :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:05.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.772 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb54182dc0 con 0x7fcb54090150 2021-04-09T18:25:05.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.776 7fcb517fa700 1 -- 172.21.15.32:0/2497510476 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcb48000c90 con 0x7fcb54090150 2021-04-09T18:25:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.970 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb540d7280 con 0x7fcb54090150 2021-04-09T18:25:06.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.970 7fcb517fa700 1 -- 172.21.15.32:0/2497510476 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 255 v255) v1 ==== 95+0+2802 (secure 0 0 0) 0x7fcb48006b00 con 0x7fcb54090150 2021-04-09T18:25:06.036 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 255 2021-04-09T18:25:06.036 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:06.036 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":255,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":255,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:04.995870","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1],"up":{"mds_0":8740,"mds_1":8749},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8749":{"gid":8749,"name":"d","rank":1,"incarnation":255,"state":"up:creating","state_seq":2,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.972 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb340188d0 msgr2=0x7fcb3401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.972 7fcb5b0d8700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb340188d0 0x7fcb3401add0 crc :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.972 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb54090150 msgr2=0x7fcb5408c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.972 7fcb5b0d8700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb54090150 0x7fcb5408c680 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7fcb48006f60 tx=0x7fcb4800b970).stop 2021-04-09T18:25:06.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.973 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 shutdown_connections 2021-04-09T18:25:06.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.973 7fcb5b0d8700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb340188d0 0x7fcb3401add0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.973 7fcb5b0d8700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb54090150 0x7fcb5408c680 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.973 7fcb5b0d8700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb5408cca0 0x7fcb5408e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.973 7fcb5b0d8700 1 --2- 172.21.15.32:0/2497510476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541acff0 0x7fcb5408fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.973 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 >> 172.21.15.32:0/2497510476 conn(0x7fcb5409abb0 msgr2=0x7fcb540a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.973 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 shutdown_connections 2021-04-09T18:25:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:05.973 7fcb5b0d8700 1 -- 172.21.15.32:0/2497510476 wait complete. 2021-04-09T18:25:06.048 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 255, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 10, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-04-09 18:24:54.971184", "damaged": [], "data_pools": [ 21 ], "enabled": true, "epoch": 255, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8740": { "addr": "172.21.15.32:6835/2311632030", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6833", "nonce": 2311632030, "type": "v2" }, { "addr": "172.21.15.32:6835", "nonce": 2311632030, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 8740, "incarnation": 250, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 }, "gid_8749": { "addr": "172.21.15.86:6836/2147753572", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 2147753572, "type": "v2" }, { "addr": "172.21.15.86:6836", "nonce": 2147753572, "type": "v1" } ] }, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 8749, "incarnation": 255, "name": "d", "rank": 1, "state": "up:creating", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 5, "metadata_pool": 20, "min_compat_client": "-1 (unspecified)", "modified": "2021-04-09 18:25:04.995870", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8740, "mds_1": 8749 } } } ], "standbys": [ { "addr": "172.21.15.86:6837/1761380306", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6835", "nonce": 1761380306, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 1761380306, "type": "v1" } ] }, "epoch": 249, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 8752, "incarnation": 0, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 2 }, { "addr": "172.21.15.32:6834/4263974733", "addrs": { "addrvec": [ { "addr": "172.21.15.32:6832", "nonce": 4263974733, "type": "v2" }, { "addr": "172.21.15.32:6834", "nonce": 4263974733, "type": "v1" } ] }, "epoch": 249, "export_targets": [], "features": 4611087854035861503, "flags": 0, "gid": 57362, "incarnation": 0, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 2 } ] } 2021-04-09T18:25:06.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:06.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.135 7f02b4aaa700 1 Processor -- start 2021-04-09T18:25:06.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.135 7f02b4aaa700 1 -- start start 2021-04-09T18:25:06.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.135 7f02b4aaa700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f02b010d010 0x7f02b00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02b4aaa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02b00d1da0 0x7f02b00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02b4aaa700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b010d480 con 0x7f02b00d82c0 2021-04-09T18:25:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02b4aaa700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b00d9220 con 0x7f02b010d010 2021-04-09T18:25:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02b4aaa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b00d86c0 con 0x7f02b00d1da0 2021-04-09T18:25:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02af05f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f02b00d82c0 0x7f02b010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33326/0 (socket says 172.21.15.32:33326) 2021-04-09T18:25:06.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02af05f700 1 -- 172.21.15.32:0/3486271478 learned_addr learned my addr 172.21.15.32:0/3486271478 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ae85e700 1 --2- 172.21.15.32:0/3486271478 >> v2:172.21.15.86:3300/0 conn(0x7f02b010d010 0x7f02b00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ae05d700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02b00d1da0 0x7f02b00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756642406 0 0) 0x7f02a8000d30 con 0x7f02b00d82c0 2021-04-09T18:25:06.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029c002fe0 con 0x7f02b00d82c0 2021-04-09T18:25:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 513137441 0 0) 0x7f02a8001460 con 0x7f02b00d82c0 2021-04-09T18:25:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02b00d1da0 msgr2=0x7f02b00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02b00d1da0 0x7f02b00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 >> v2:172.21.15.86:3300/0 conn(0x7f02b010d010 msgr2=0x7f02b00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 --2- 172.21.15.32:0/3486271478 >> v2:172.21.15.86:3300/0 conn(0x7f02b010d010 0x7f02b00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b010e1f0 con 0x7f02b00d82c0 2021-04-09T18:25:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f02a8001980 con 0x7f02b00d82c0 2021-04-09T18:25:06.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ae85e700 1 --2- 172.21.15.32:0/3486271478 >> v2:172.21.15.86:3300/0 conn(0x7f02b010d010 0x7f02b00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 >> v1:172.21.15.32:6789/0 conn(0x7f02b00d82c0 legacy=0x7f02b010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029c047aa0 0x7f029c049f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f029c04a7b0 0x7f029c04cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f029c04d520 0x7f029c04fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0502f0 con 0x7f029c047aa0 2021-04-09T18:25:06.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c050510 con 0x7f029c04d520 2021-04-09T18:25:06.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.136 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c050730 con 0x7f029c04a7b0 2021-04-09T18:25:06.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ae85e700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029c047aa0 0x7f029c049f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ae05d700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f029c04a7b0 0x7f029c04cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02af05f700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f029c04d520 0x7f029c04fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ae85e700 1 -- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f029c04a7b0 msgr2=0x7f029c04cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:06.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ae85e700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f029c04a7b0 0x7f029c04cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ae85e700 1 -- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f029c04d520 msgr2=0x7f029c04fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ae85e700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f029c04d520 0x7f029c04fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ae85e700 1 -- 172.21.15.32:0/3486271478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0509d0 con 0x7f029c047aa0 2021-04-09T18:25:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ae05d700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f029c04a7b0 0x7f029c04cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ae85e700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029c047aa0 0x7f029c049f10 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f02a0006b00 tx=0x7f02a000cea0).ready entity=mon.0 client_cookie=bf3bcb5b605b6c4c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a00068d0 con 0x7f029c047aa0 2021-04-09T18:25:06.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a0009310 con 0x7f029c047aa0 2021-04-09T18:25:06.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02ad85c700 1 -- 172.21.15.32:0/3486271478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a0009760 con 0x7f029c047aa0 2021-04-09T18:25:06.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02b4aaa700 1 -- 172.21.15.32:0/3486271478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029c047aa0 msgr2=0x7f029c049f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02b4aaa700 1 --2- 172.21.15.32:0/3486271478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f029c047aa0 0x7f029c049f10 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f02a0006b00 tx=0x7f02a000cea0).stop 2021-04-09T18:25:06.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.137 7f02af05f700 1 -- 172.21.15.32:0/3486271478 reap_dead start 2021-04-09T18:25:06.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.138 7f02b4aaa700 1 -- 172.21.15.32:0/3486271478 shutdown_connections 2021-04-09T18:25:06.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.138 7f02b4aaa700 1 -- 172.21.15.32:0/3486271478 >> 172.21.15.32:0/3486271478 conn(0x7f02b00ce5f0 msgr2=0x7f02b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:06.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.138 7f02b4aaa700 1 -- 172.21.15.32:0/3486271478 shutdown_connections 2021-04-09T18:25:06.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.138 7f02b4aaa700 1 -- 172.21.15.32:0/3486271478 wait complete. 2021-04-09T18:25:06.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.158 7f02b4aaa700 1 Processor -- start 2021-04-09T18:25:06.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.158 7f02b4aaa700 1 -- start start 2021-04-09T18:25:06.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02b4aaa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02b01a91e0 0x7f02b00a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02b4aaa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b01a95e0 0x7f02b00a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02b4aaa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01ac850 0x7f02b01acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02b4aaa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01ad270 con 0x7f02b01ac850 2021-04-09T18:25:06.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02b4aaa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b00b7640 con 0x7f02b01a95e0 2021-04-09T18:25:06.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02b4aaa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0183c80 con 0x7f02b01a91e0 2021-04-09T18:25:06.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02af05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01ac850 0x7f02b01acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02af05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01ac850 0x7f02b01acc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51982/0 (socket says 172.21.15.32:51982) 2021-04-09T18:25:06.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02af05f700 1 -- 172.21.15.32:0/1894797952 learned_addr learned my addr 172.21.15.32:0/1894797952 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:06.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02ae85e700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02b01a91e0 0x7f02b00a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02ae05d700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b01a95e0 0x7f02b00a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02af05f700 1 -- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02b01a91e0 msgr2=0x7f02b00a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02af05f700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02b01a91e0 0x7f02b00a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02af05f700 1 -- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b01a95e0 msgr2=0x7f02b00a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02af05f700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b01a95e0 0x7f02b00a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.159 7f02af05f700 1 -- 172.21.15.32:0/1894797952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b0183ea0 con 0x7f02b01ac850 2021-04-09T18:25:06.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.160 7f02af05f700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01ac850 0x7f02b01acc50 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 rx=0x7f02a8007a00 tx=0x7f02a800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:06.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.160 7f02a77fe700 1 -- 172.21.15.32:0/1894797952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a8001d80 con 0x7f02b01ac850 2021-04-09T18:25:06.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.160 7f02a77fe700 1 -- 172.21.15.32:0/1894797952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a8001930 con 0x7f02b01ac850 2021-04-09T18:25:06.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.160 7f02a77fe700 1 -- 172.21.15.32:0/1894797952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a8007380 con 0x7f02b01ac850 2021-04-09T18:25:06.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.160 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b0183ea0 con 0x7f02b01ac850 2021-04-09T18:25:06.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.160 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b0183ea0 con 0x7f02b01ac850 2021-04-09T18:25:06.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.160 7f02a77fe700 1 -- 172.21.15.32:0/1894797952 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f02a80072e0 con 0x7f02b01ac850 2021-04-09T18:25:06.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.160 7f02a77fe700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0294018930 0x7f029401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.160 7f02a77fe700 1 -- 172.21.15.32:0/1894797952 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f02a8032f60 con 0x7f02b01ac850 2021-04-09T18:25:06.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.161 7f02ae85e700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0294018930 0x7f029401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.161 7f02ae85e700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0294018930 0x7f029401ae10 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:06.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.179 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b0182d70 con 0x7f02b01ac850 2021-04-09T18:25:06.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.183 7f02a77fe700 1 -- 172.21.15.32:0/1894797952 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f02a80070b0 con 0x7f02b01ac850 2021-04-09T18:25:06.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.374 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02b010d480 con 0x7f02b01ac850 2021-04-09T18:25:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.374 7f02a77fe700 1 -- 172.21.15.32:0/1894797952 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 257 v257) v1 ==== 95+0+2816 (secure 0 0 0) 0x7f02a80072c0 con 0x7f02b01ac850 2021-04-09T18:25:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 257 2021-04-09T18:25:06.440 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:06.441 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":257,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:06.001544","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2],"up":{"mds_0":8740,"mds_1":8749,"mds_2":8752},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8749":{"gid":8749,"name":"d","rank":1,"incarnation":255,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8752":{"gid":8752,"name":"b","rank":2,"incarnation":257,"state":"up:creating","state_seq":2,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:06.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0294018930 msgr2=0x7f029401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0294018930 0x7f029401ae10 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01ac850 msgr2=0x7f02b01acc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01ac850 0x7f02b01acc50 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 rx=0x7f02a8007a00 tx=0x7f02a800aca0).stop 2021-04-09T18:25:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 shutdown_connections 2021-04-09T18:25:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0294018930 0x7f029401ae10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02b01ac850 0x7f02b01acc50 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b01a95e0 0x7f02b00a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 --2- 172.21.15.32:0/1894797952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02b01a91e0 0x7f02b00a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 >> 172.21.15.32:0/1894797952 conn(0x7f02b019b460 msgr2=0x7f02b019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 shutdown_connections 2021-04-09T18:25:06.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.377 7f02b4aaa700 1 -- 172.21.15.32:0/1894797952 wait complete. 2021-04-09T18:25:06.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:25:06.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.540 7ffb1a8ba700 1 Processor -- start 2021-04-09T18:25:06.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.540 7ffb1a8ba700 1 -- start start 2021-04-09T18:25:06.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.540 7ffb1a8ba700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb140cffc0 0x7ffb140d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.540 7ffb1a8ba700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ffb140d7820 0x7ffb140d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.540 7ffb1a8ba700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140d72a0 con 0x7ffb140d8240 2021-04-09T18:25:06.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.540 7ffb1a8ba700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb140d9200 con 0x7ffb140cffc0 2021-04-09T18:25:06.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.540 7ffb1a8ba700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1410e000 con 0x7ffb140d7820 2021-04-09T18:25:06.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb13fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffb140cffc0 0x7ffb140d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb18ed7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffb140d8240 0x7ffb140d5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33346/0 (socket says 172.21.15.32:33346) 2021-04-09T18:25:06.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb18ed7700 1 -- 172.21.15.32:0/1884453755 learned_addr learned my addr 172.21.15.32:0/1884453755 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:06.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb0bfff700 1 --2- 172.21.15.32:0/1884453755 >> v2:172.21.15.86:3300/0 conn(0x7ffb140d7820 0x7ffb140d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb13fff700 1 -- 172.21.15.32:0/1884453755 >> v2:172.21.15.86:3300/0 conn(0x7ffb140d7820 msgr2=0x7ffb140d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:06.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb13fff700 1 --2- 172.21.15.32:0/1884453755 >> v2:172.21.15.86:3300/0 conn(0x7ffb140d7820 0x7ffb140d7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb13fff700 1 -- 172.21.15.32:0/1884453755 >> v1:172.21.15.32:6789/0 conn(0x7ffb140d8240 legacy=0x7ffb140d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb137fe700 1 -- 172.21.15.32:0/1884453755 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032267783 0 0) 0x7ffb0c000d30 con 0x7ffb140d8240 2021-04-09T18:25:06.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb13fff700 1 -- 172.21.15.32:0/1884453755 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1410e220 con 0x7ffb140cffc0 2021-04-09T18:25:06.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb13fff700 1 --2- 172.21.15.32:0/1884453755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb140cffc0 0x7ffb140d2460 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7ffb04006650 tx=0x7ffb04009a00).ready entity=mon.0 client_cookie=74a8ae728a08c481 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:06.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb137fe700 1 -- 172.21.15.32:0/1884453755 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb04009df0 con 0x7ffb140cffc0 2021-04-09T18:25:06.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb137fe700 1 -- 172.21.15.32:0/1884453755 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb040009b0 con 0x7ffb140cffc0 2021-04-09T18:25:06.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb137fe700 1 -- 172.21.15.32:0/1884453755 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb04009df0 con 0x7ffb140cffc0 2021-04-09T18:25:06.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb1a8ba700 1 -- 172.21.15.32:0/1884453755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb140cffc0 msgr2=0x7ffb140d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.541 7ffb1a8ba700 1 --2- 172.21.15.32:0/1884453755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb140cffc0 0x7ffb140d2460 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7ffb04006650 tx=0x7ffb04009a00).stop 2021-04-09T18:25:06.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.542 7ffb1a8ba700 1 -- 172.21.15.32:0/1884453755 shutdown_connections 2021-04-09T18:25:06.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.542 7ffb1a8ba700 1 --2- 172.21.15.32:0/1884453755 >> v2:172.21.15.86:3300/0 conn(0x7ffb140d7820 0x7ffb140d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.542 7ffb1a8ba700 1 --2- 172.21.15.32:0/1884453755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb140cffc0 0x7ffb140d2460 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.542 7ffb1a8ba700 1 -- 172.21.15.32:0/1884453755 >> 172.21.15.32:0/1884453755 conn(0x7ffb140ce5e0 msgr2=0x7ffb140cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:06.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.542 7ffb1a8ba700 1 -- 172.21.15.32:0/1884453755 shutdown_connections 2021-04-09T18:25:06.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.542 7ffb1a8ba700 1 -- 172.21.15.32:0/1884453755 wait complete. 2021-04-09T18:25:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.562 7ffb1a8ba700 1 Processor -- start 2021-04-09T18:25:06.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.562 7ffb1a8ba700 1 -- start start 2021-04-09T18:25:06.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb1a8ba700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb1409ebc0 0x7ffb1409adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb1a8ba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb14183050 0x7ffb1419ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb1a8ba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb14183450 0x7ffb140952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb1a8ba700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1409b660 con 0x7ffb14183450 2021-04-09T18:25:06.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb1a8ba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb141a05c0 con 0x7ffb14183050 2021-04-09T18:25:06.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb1a8ba700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb14095900 con 0x7ffb1409ebc0 2021-04-09T18:25:06.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb18ed7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb14183450 0x7ffb140952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb18ed7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb14183450 0x7ffb140952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51994/0 (socket says 172.21.15.32:51994) 2021-04-09T18:25:06.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb18ed7700 1 -- 172.21.15.32:0/2951362753 learned_addr learned my addr 172.21.15.32:0/2951362753 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:06.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb0bfff700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb14183050 0x7ffb1419ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb13fff700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb1409ebc0 0x7ffb1409adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb18ed7700 1 -- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb1409ebc0 msgr2=0x7ffb1409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb18ed7700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb1409ebc0 0x7ffb1409adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb18ed7700 1 -- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb14183050 msgr2=0x7ffb1419ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb18ed7700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb14183050 0x7ffb1419ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:06.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb18ed7700 1 -- 172.21.15.32:0/2951362753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb14095b20 con 0x7ffb14183450 2021-04-09T18:25:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb0bfff700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb14183050 0x7ffb1419ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.563 7ffb18ed7700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb14183450 0x7ffb140952e0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 rx=0x7ffb0c007f90 tx=0x7ffb0c00abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.564 7ffb117fa700 1 -- 172.21.15.32:0/2951362753 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0c000cc0 con 0x7ffb14183450 2021-04-09T18:25:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.564 7ffb117fa700 1 -- 172.21.15.32:0/2951362753 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c0070d0 con 0x7ffb14183450 2021-04-09T18:25:06.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.564 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb14095b20 con 0x7ffb14183450 2021-04-09T18:25:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.564 7ffb117fa700 1 -- 172.21.15.32:0/2951362753 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0c007520 con 0x7ffb14183450 2021-04-09T18:25:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.564 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb14095b20 con 0x7ffb14183450 2021-04-09T18:25:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.564 7ffb117fa700 1 -- 172.21.15.32:0/2951362753 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ffb0c0072b0 con 0x7ffb14183450 2021-04-09T18:25:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.564 7ffb117fa700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffaf8018930 0x7ffaf801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:06.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.564 7ffb117fa700 1 -- 172.21.15.32:0/2951362753 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ffb0c032d30 con 0x7ffb14183450 2021-04-09T18:25:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.565 7ffb13fff700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffaf8018930 0x7ffaf801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:06.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.565 7ffb13fff700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffaf8018930 0x7ffaf801adc0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:06.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.584 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb14093df0 con 0x7ffb14183450 2021-04-09T18:25:06.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.588 7ffb117fa700 1 -- 172.21.15.32:0/2951362753 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ffb0c007d00 con 0x7ffb14183450 2021-04-09T18:25:06.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:06.781 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7ffb141a0580 con 0x7ffb14183450 2021-04-09T18:25:06.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:25:06.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:25:07.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.006 7ffb117fa700 1 -- 172.21.15.32:0/2951362753 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v258) v1 ==== 105+0+0 (secure 0 0 0) 0x7ffb0c0070d0 con 0x7ffb14183450 2021-04-09T18:25:07.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffaf8018930 msgr2=0x7ffaf801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:07.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffaf8018930 0x7ffaf801adc0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb14183450 msgr2=0x7ffb140952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb14183450 0x7ffb140952e0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 rx=0x7ffb0c007f90 tx=0x7ffb0c00abd0).stop 2021-04-09T18:25:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 shutdown_connections 2021-04-09T18:25:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffaf8018930 0x7ffaf801adc0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb14183450 0x7ffb140952e0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb14183050 0x7ffb1419ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 --2- 172.21.15.32:0/2951362753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb1409ebc0 0x7ffb1409adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 >> 172.21.15.32:0/2951362753 conn(0x7ffb1409fa00 msgr2=0x7ffb140b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:07.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 shutdown_connections 2021-04-09T18:25:07.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.047 7ffb1a8ba700 1 -- 172.21.15.32:0/2951362753 wait complete. 2021-04-09T18:25:07.123 INFO:tasks.cephfs.test_failover:Waiting for cluster to shink. 2021-04-09T18:25:07.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:07.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.207 7f136ac47700 1 Processor -- start 2021-04-09T18:25:07.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.207 7f136ac47700 1 -- start start 2021-04-09T18:25:07.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f136ac47700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640d7ca0 0x7f13640d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f136ac47700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f136410da00 0x7f136410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f136ac47700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136410d480 con 0x7f136410d010 2021-04-09T18:25:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f136ac47700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13640d9220 con 0x7f136410da00 2021-04-09T18:25:07.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f136ac47700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640d86c0 con 0x7f13640d7ca0 2021-04-09T18:25:07.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f1368a63700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f136410d010 0x7f13640d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:59518/0 (socket says 172.21.15.32:59518) 2021-04-09T18:25:07.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f1368a63700 1 -- 172.21.15.32:0/572951250 learned_addr learned my addr 172.21.15.32:0/572951250 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:07.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f1369264700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f136410da00 0x7f136410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:07.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f1363fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640d7ca0 0x7f13640d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:07.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f13637fe700 1 -- 172.21.15.32:0/572951250 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711153989 0 0) 0x7f134c0010c0 con 0x7f136410d010 2021-04-09T18:25:07.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.208 7f13637fe700 1 -- 172.21.15.32:0/572951250 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1350002fe0 con 0x7f136410d010 2021-04-09T18:25:07.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f1363fff700 1 -- 172.21.15.32:0/572951250 >> v2:172.21.15.86:3300/0 conn(0x7f136410da00 msgr2=0x7f136410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:07.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f1363fff700 1 --2- 172.21.15.32:0/572951250 >> v2:172.21.15.86:3300/0 conn(0x7f136410da00 0x7f136410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f1363fff700 1 -- 172.21.15.32:0/572951250 >> v1:172.21.15.86:6789/0 conn(0x7f136410d010 legacy=0x7f13640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:07.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f1363fff700 1 -- 172.21.15.32:0/572951250 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640d1f70 con 0x7f13640d7ca0 2021-04-09T18:25:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f1369264700 1 --2- 172.21.15.32:0/572951250 >> v2:172.21.15.86:3300/0 conn(0x7f136410da00 0x7f136410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f1363fff700 1 --2- 172.21.15.32:0/572951250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640d7ca0 0x7f13640d80a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f13540069d0 tx=0x7f1354009e40).ready entity=mon.2 client_cookie=ed6859ca0620180c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:07.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f13637fe700 1 -- 172.21.15.32:0/572951250 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f135400b510 con 0x7f13640d7ca0 2021-04-09T18:25:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f13637fe700 1 -- 172.21.15.32:0/572951250 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135400b890 con 0x7f13640d7ca0 2021-04-09T18:25:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f13637fe700 1 -- 172.21.15.32:0/572951250 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f135400bd70 con 0x7f13640d7ca0 2021-04-09T18:25:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f136ac47700 1 -- 172.21.15.32:0/572951250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640d7ca0 msgr2=0x7f13640d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.209 7f136ac47700 1 --2- 172.21.15.32:0/572951250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640d7ca0 0x7f13640d80a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f13540069d0 tx=0x7f1354009e40).stop 2021-04-09T18:25:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.210 7f136ac47700 1 -- 172.21.15.32:0/572951250 shutdown_connections 2021-04-09T18:25:07.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.210 7f136ac47700 1 --2- 172.21.15.32:0/572951250 >> v2:172.21.15.86:3300/0 conn(0x7f136410da00 0x7f136410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.210 7f136ac47700 1 --2- 172.21.15.32:0/572951250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640d7ca0 0x7f13640d80a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.210 7f136ac47700 1 -- 172.21.15.32:0/572951250 >> 172.21.15.32:0/572951250 conn(0x7f13640ce5f0 msgr2=0x7f13640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:07.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.210 7f136ac47700 1 -- 172.21.15.32:0/572951250 shutdown_connections 2021-04-09T18:25:07.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.210 7f136ac47700 1 -- 172.21.15.32:0/572951250 wait complete. 2021-04-09T18:25:07.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.230 7f136ac47700 1 Processor -- start 2021-04-09T18:25:07.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.230 7f136ac47700 1 -- start start 2021-04-09T18:25:07.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f136ac47700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13641a5210 0x7f13641a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:07.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f136ac47700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13641a6340 0x7f13641a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:07.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f136ac47700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640abaa0 0x7f13640abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:07.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f136ac47700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13641a6d60 con 0x7f13641a5210 2021-04-09T18:25:07.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f136ac47700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13640ac4c0 con 0x7f13641a6340 2021-04-09T18:25:07.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f136ac47700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640ac6e0 con 0x7f13640abaa0 2021-04-09T18:25:07.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1368a63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13641a5210 0x7f13641a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1368a63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13641a5210 0x7f13641a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52008/0 (socket says 172.21.15.32:52008) 2021-04-09T18:25:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1368a63700 1 -- 172.21.15.32:0/2748397945 learned_addr learned my addr 172.21.15.32:0/2748397945 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1369264700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640abaa0 0x7f13640abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:07.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1363fff700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13641a6340 0x7f13641a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1368a63700 1 -- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640abaa0 msgr2=0x7f13640abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1368a63700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640abaa0 0x7f13640abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1368a63700 1 -- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13641a6340 msgr2=0x7f13641a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:07.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1368a63700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13641a6340 0x7f13641a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1368a63700 1 -- 172.21.15.32:0/2748397945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136417a720 con 0x7f13641a5210 2021-04-09T18:25:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.231 7f1368a63700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13641a5210 0x7f13641a56c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 rx=0x7f134c001390 tx=0x7f134c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.232 7f13617fa700 1 -- 172.21.15.32:0/2748397945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f134c000e70 con 0x7f13641a5210 2021-04-09T18:25:07.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.232 7f13617fa700 1 -- 172.21.15.32:0/2748397945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c00ad20 con 0x7f13641a5210 2021-04-09T18:25:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.232 7f136ac47700 1 -- 172.21.15.32:0/2748397945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136417a9c0 con 0x7f13641a5210 2021-04-09T18:25:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.232 7f13617fa700 1 -- 172.21.15.32:0/2748397945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f134c004320 con 0x7f13641a5210 2021-04-09T18:25:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.232 7f136ac47700 1 -- 172.21.15.32:0/2748397945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136417a9c0 con 0x7f13641a5210 2021-04-09T18:25:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.232 7f13617fa700 1 -- 172.21.15.32:0/2748397945 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f134c004320 con 0x7f13641a5210 2021-04-09T18:25:07.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.232 7f13617fa700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:07.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.233 7f13617fa700 1 -- 172.21.15.32:0/2748397945 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f134c0329c0 con 0x7f13641a5210 2021-04-09T18:25:07.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.233 7f1363fff700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:07.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.233 7f1363fff700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 crc :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:07.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.252 7f136ac47700 1 -- 172.21.15.32:0/2748397945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13641a6d60 con 0x7f13641a5210 2021-04-09T18:25:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.255 7f13617fa700 1 -- 172.21.15.32:0/2748397945 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f134c000c60 con 0x7f13641a5210 2021-04-09T18:25:07.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.445 7f136ac47700 1 -- 172.21.15.32:0/2748397945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13641a6d60 con 0x7f13641a5210 2021-04-09T18:25:07.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.446 7f13617fa700 1 -- 172.21.15.32:0/2748397945 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+2816 (secure 0 0 0) 0x7f134c00ad20 con 0x7f13641a5210 2021-04-09T18:25:07.512 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 259 2021-04-09T18:25:07.512 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:07.512 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":259,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:07.007490","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":8740,"mds_1":8749,"mds_2":8752},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8749":{"gid":8749,"name":"d","rank":1,"incarnation":255,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8752":{"gid":8752,"name":"b","rank":2,"incarnation":257,"state":"up:stopping","state_seq":6,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:07.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.448 7f136ac47700 1 -- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 msgr2=0x7f134801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:07.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.448 7f136ac47700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 crc :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.448 7f136ac47700 1 -- 172.21.15.32:0/2748397945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13641a5210 msgr2=0x7f13641a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:07.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.448 7f136ac47700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13641a5210 0x7f13641a56c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 rx=0x7f134c001390 tx=0x7f134c00a930).stop 2021-04-09T18:25:07.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.449 7f136ac47700 1 -- 172.21.15.32:0/2748397945 shutdown_connections 2021-04-09T18:25:07.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.449 7f136ac47700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1348018930 0x7f134801ae10 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.449 7f136ac47700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640abaa0 0x7f13640abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.449 7f136ac47700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13641a6340 0x7f13641a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.449 7f136ac47700 1 --2- 172.21.15.32:0/2748397945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13641a5210 0x7f13641a56c0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:07.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.449 7f136ac47700 1 -- 172.21.15.32:0/2748397945 >> 172.21.15.32:0/2748397945 conn(0x7f13641a99c0 msgr2=0x7f136417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:07.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.449 7f136ac47700 1 -- 172.21.15.32:0/2748397945 shutdown_connections 2021-04-09T18:25:07.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:07.449 7f136ac47700 1 -- 172.21.15.32:0/2748397945 wait complete. 2021-04-09T18:25:07.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 259, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:24:54.971184', 'modified': '2021-04-09 18:25:07.007490', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1, 2], 'up': {'mds_0': 8740, 'mds_1': 8749, 'mds_2': 8752}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8740': {'gid': 8740, 'name': 'a', 'rank': 0, 'incarnation': 250, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2311632030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2311632030}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2311632030}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8749': {'gid': 8749, 'name': 'd', 'rank': 1, 'incarnation': 255, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/2147753572', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2147753572}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 2147753572}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8752': {'gid': 8752, 'name': 'b', 'rank': 2, 'incarnation': 257, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.86:6837/1761380306', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1761380306}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1761380306}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:07.927 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8752:up:stopping 2021-04-09T18:25:08.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:09.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.012 7f5c5715c700 1 Processor -- start 2021-04-09T18:25:09.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.013 7f5c5715c700 1 -- start start 2021-04-09T18:25:09.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.013 7f5c5715c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5c5010da00 0x7f5c5010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:09.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.013 7f5c5715c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5010d480 con 0x7f5c500d7ca0 2021-04-09T18:25:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.013 7f5c5715c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c500d1f70 con 0x7f5c5010d010 2021-04-09T18:25:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.013 7f5c5715c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c500d8d30 con 0x7f5c5010da00 2021-04-09T18:25:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.013 7f5c55779700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5c5010da00 0x7f5c5010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.013 7f5c4ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5c500d7ca0 0x7f5c500d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33374/0 (socket says 172.21.15.32:33374) 2021-04-09T18:25:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.013 7f5c55779700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5c5010da00 0x7f5c5010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52016/0 (socket says 172.21.15.32:52016) 2021-04-09T18:25:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.013 7f5c4ffff700 1 -- 172.21.15.32:0/2578115478 learned_addr learned my addr 172.21.15.32:0/2578115478 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c55779700 1 -- 172.21.15.32:0/2578115478 >> v1:172.21.15.86:6789/0 conn(0x7f5c5010d010 legacy=0x7f5c500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c55779700 1 -- 172.21.15.32:0/2578115478 >> v1:172.21.15.32:6789/0 conn(0x7f5c500d7ca0 legacy=0x7f5c500d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c4f7fe700 1 -- 172.21.15.32:0/2578115478 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575753610 0 0) 0x7f5c40000ce0 con 0x7f5c500d7ca0 2021-04-09T18:25:09.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c55779700 1 -- 172.21.15.32:0/2578115478 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c500d9220 con 0x7f5c5010da00 2021-04-09T18:25:09.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c55779700 1 --2- 172.21.15.32:0/2578115478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c5010da00 0x7f5c5010de00 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f5c48006e30 tx=0x7f5c48009de0).ready entity=mon.0 client_cookie=a4acba1c7d2d17d8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:09.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c4f7fe700 1 -- 172.21.15.32:0/2578115478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c4800b300 con 0x7f5c5010da00 2021-04-09T18:25:09.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c4f7fe700 1 -- 172.21.15.32:0/2578115478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4800b680 con 0x7f5c5010da00 2021-04-09T18:25:09.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c4f7fe700 1 -- 172.21.15.32:0/2578115478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c4800bb60 con 0x7f5c5010da00 2021-04-09T18:25:09.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c5715c700 1 -- 172.21.15.32:0/2578115478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c5010da00 msgr2=0x7f5c5010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:09.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.014 7f5c5715c700 1 --2- 172.21.15.32:0/2578115478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c5010da00 0x7f5c5010de00 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f5c48006e30 tx=0x7f5c48009de0).stop 2021-04-09T18:25:09.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.015 7f5c5715c700 1 -- 172.21.15.32:0/2578115478 shutdown_connections 2021-04-09T18:25:09.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.015 7f5c5715c700 1 --2- 172.21.15.32:0/2578115478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c5010da00 0x7f5c5010de00 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:09.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.015 7f5c5715c700 1 -- 172.21.15.32:0/2578115478 >> 172.21.15.32:0/2578115478 conn(0x7f5c500ce5f0 msgr2=0x7f5c500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:09.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.015 7f5c5715c700 1 -- 172.21.15.32:0/2578115478 shutdown_connections 2021-04-09T18:25:09.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.015 7f5c5715c700 1 -- 172.21.15.32:0/2578115478 wait complete. 2021-04-09T18:25:09.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.035 7f5c5715c700 1 Processor -- start 2021-04-09T18:25:09.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.035 7f5c5715c700 1 -- start start 2021-04-09T18:25:09.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c5715c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c5019d870 0x7f5c50184260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:09.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c5715c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c50182850 0x7f5c50184880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:09.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c5715c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c500b0ac0 0x7f5c500b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:09.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c5715c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c50184ea0 con 0x7f5c50182850 2021-04-09T18:25:09.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c5715c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c50183ce0 con 0x7f5c5019d870 2021-04-09T18:25:09.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c5715c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c500b14e0 con 0x7f5c500b0ac0 2021-04-09T18:25:09.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c50182850 0x7f5c50184880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:09.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c50182850 0x7f5c50184880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52022/0 (socket says 172.21.15.32:52022) 2021-04-09T18:25:09.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4ffff700 1 -- 172.21.15.32:0/688283428 learned_addr learned my addr 172.21.15.32:0/688283428 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:09.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c54f78700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c5019d870 0x7f5c50184260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:09.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4ffff700 1 -- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c500b0ac0 msgr2=0x7f5c500b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:09.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c55779700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c500b0ac0 0x7f5c500b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:09.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4ffff700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c500b0ac0 0x7f5c500b0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:09.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4ffff700 1 -- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c5019d870 msgr2=0x7f5c50184260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4ffff700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c5019d870 0x7f5c50184260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4ffff700 1 -- 172.21.15.32:0/688283428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c500b16c0 con 0x7f5c50182850 2021-04-09T18:25:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c54f78700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c5019d870 0x7f5c50184260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c55779700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c500b0ac0 0x7f5c500b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4ffff700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c50182850 0x7f5c50184880 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 rx=0x7f5c40006f00 tx=0x7f5c4000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4d7fa700 1 -- 172.21.15.32:0/688283428 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c4000bd30 con 0x7f5c50182850 2021-04-09T18:25:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4d7fa700 1 -- 172.21.15.32:0/688283428 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c400060d0 con 0x7f5c50182850 2021-04-09T18:25:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c5715c700 1 -- 172.21.15.32:0/688283428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c500b16c0 con 0x7f5c50182850 2021-04-09T18:25:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.036 7f5c4d7fa700 1 -- 172.21.15.32:0/688283428 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c400043a0 con 0x7f5c50182850 2021-04-09T18:25:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.037 7f5c5715c700 1 -- 172.21.15.32:0/688283428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c500b16c0 con 0x7f5c50182850 2021-04-09T18:25:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.037 7f5c4d7fa700 1 -- 172.21.15.32:0/688283428 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5c4000a000 con 0x7f5c50182850 2021-04-09T18:25:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.037 7f5c4d7fa700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5c34018930 0x7f5c3401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.037 7f5c4d7fa700 1 -- 172.21.15.32:0/688283428 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5c400329a0 con 0x7f5c50182850 2021-04-09T18:25:09.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.037 7f5c54f78700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5c34018930 0x7f5c3401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:09.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.038 7f5c54f78700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5c34018930 0x7f5c3401ae10 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:09.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.057 7f5c5715c700 1 -- 172.21.15.32:0/688283428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5017cb20 con 0x7f5c50182850 2021-04-09T18:25:09.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.061 7f5c4d7fa700 1 -- 172.21.15.32:0/688283428 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5c400060d0 con 0x7f5c50182850 2021-04-09T18:25:09.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.252 7f5c5715c700 1 -- 172.21.15.32:0/688283428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c5010d480 con 0x7f5c50182850 2021-04-09T18:25:09.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.253 7f5c4d7fa700 1 -- 172.21.15.32:0/688283428 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+2816 (secure 0 0 0) 0x7f5c4000a000 con 0x7f5c50182850 2021-04-09T18:25:09.319 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 259 2021-04-09T18:25:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:09.320 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":259,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:07.007490","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":8740,"mds_1":8749,"mds_2":8752},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8749":{"gid":8749,"name":"d","rank":1,"incarnation":255,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8752":{"gid":8752,"name":"b","rank":2,"incarnation":257,"state":"up:stopping","state_seq":6,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:09.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.255 7f5c5715c700 1 -- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5c34018930 msgr2=0x7f5c3401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:09.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.255 7f5c5715c700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5c34018930 0x7f5c3401ae10 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:09.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.255 7f5c5715c700 1 -- 172.21.15.32:0/688283428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c50182850 msgr2=0x7f5c50184880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:09.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.255 7f5c5715c700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c50182850 0x7f5c50184880 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 rx=0x7f5c40006f00 tx=0x7f5c4000b940).stop 2021-04-09T18:25:09.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.256 7f5c5715c700 1 -- 172.21.15.32:0/688283428 shutdown_connections 2021-04-09T18:25:09.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.256 7f5c5715c700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5c34018930 0x7f5c3401ae10 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:09.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.256 7f5c5715c700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c50182850 0x7f5c50184880 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.256 7f5c5715c700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c500b0ac0 0x7f5c500b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.256 7f5c5715c700 1 --2- 172.21.15.32:0/688283428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c5019d870 0x7f5c50184260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.256 7f5c5715c700 1 -- 172.21.15.32:0/688283428 >> 172.21.15.32:0/688283428 conn(0x7f5c501965a0 msgr2=0x7f5c5019dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.256 7f5c5715c700 1 -- 172.21.15.32:0/688283428 shutdown_connections 2021-04-09T18:25:09.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:09.256 7f5c5715c700 1 -- 172.21.15.32:0/688283428 wait complete. 2021-04-09T18:25:09.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 259, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:24:54.971184', 'modified': '2021-04-09 18:25:07.007490', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1, 2], 'up': {'mds_0': 8740, 'mds_1': 8749, 'mds_2': 8752}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8740': {'gid': 8740, 'name': 'a', 'rank': 0, 'incarnation': 250, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2311632030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2311632030}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2311632030}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8749': {'gid': 8749, 'name': 'd', 'rank': 1, 'incarnation': 255, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/2147753572', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2147753572}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 2147753572}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8752': {'gid': 8752, 'name': 'b', 'rank': 2, 'incarnation': 257, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.86:6837/1761380306', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1761380306}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1761380306}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:09.734 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8752:up:stopping 2021-04-09T18:25:10.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:10.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.820 7fe5afa23700 1 Processor -- start 2021-04-09T18:25:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.820 7fe5afa23700 1 -- start start 2021-04-09T18:25:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.820 7fe5afa23700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe5a810d010 0x7fe5a80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.820 7fe5afa23700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a810d480 con 0x7fe5a80d8490 2021-04-09T18:25:10.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.820 7fe5afa23700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a810e1f0 con 0x7fe5a80d1da0 2021-04-09T18:25:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.820 7fe5afa23700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a80d8d30 con 0x7fe5a810d010 2021-04-09T18:25:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ae040700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe5a80d8490 0x7fe5a810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33386/0 (socket says 172.21.15.32:33386) 2021-04-09T18:25:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ae040700 1 -- 172.21.15.32:0/4048342337 learned_addr learned my addr 172.21.15.32:0/4048342337 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ad83f700 1 --2- 172.21.15.32:0/4048342337 >> v2:172.21.15.86:3300/0 conn(0x7fe5a810d010 0x7fe5a80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:10.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ac83d700 1 -- 172.21.15.32:0/4048342337 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822736192 0 0) 0x7fe5a0000d30 con 0x7fe5a80d8490 2021-04-09T18:25:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ac83d700 1 -- 172.21.15.32:0/4048342337 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe594002fe0 con 0x7fe5a80d8490 2021-04-09T18:25:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ac83d700 1 -- 172.21.15.32:0/4048342337 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614030058 0 0) 0x7fe598000c60 con 0x7fe5a80d1da0 2021-04-09T18:25:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ac83d700 1 -- 172.21.15.32:0/4048342337 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe594003210 con 0x7fe5a80d1da0 2021-04-09T18:25:10.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ad83f700 1 -- 172.21.15.32:0/4048342337 >> v1:172.21.15.86:6789/0 conn(0x7fe5a80d1da0 legacy=0x7fe5a80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ad83f700 1 -- 172.21.15.32:0/4048342337 >> v1:172.21.15.32:6789/0 conn(0x7fe5a80d8490 legacy=0x7fe5a810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ad83f700 1 -- 172.21.15.32:0/4048342337 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a80d9220 con 0x7fe5a810d010 2021-04-09T18:25:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ad83f700 1 --2- 172.21.15.32:0/4048342337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5a810d010 0x7fe5a80d1780 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7fe5a4006650 tx=0x7fe5a4009a80).ready entity=mon.1 client_cookie=8822e9d537c1edc1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ac83d700 1 -- 172.21.15.32:0/4048342337 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5a40009b0 con 0x7fe5a810d010 2021-04-09T18:25:10.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ac83d700 1 -- 172.21.15.32:0/4048342337 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a400c0c0 con 0x7fe5a810d010 2021-04-09T18:25:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5ac83d700 1 -- 172.21.15.32:0/4048342337 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5a400c510 con 0x7fe5a810d010 2021-04-09T18:25:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.821 7fe5afa23700 1 -- 172.21.15.32:0/4048342337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5a810d010 msgr2=0x7fe5a80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.822 7fe5afa23700 1 --2- 172.21.15.32:0/4048342337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5a810d010 0x7fe5a80d1780 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7fe5a4006650 tx=0x7fe5a4009a80).stop 2021-04-09T18:25:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.822 7fe5afa23700 1 -- 172.21.15.32:0/4048342337 shutdown_connections 2021-04-09T18:25:10.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.822 7fe5afa23700 1 --2- 172.21.15.32:0/4048342337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5a810d010 0x7fe5a80d1780 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:10.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.822 7fe5afa23700 1 -- 172.21.15.32:0/4048342337 >> 172.21.15.32:0/4048342337 conn(0x7fe5a80ce5f0 msgr2=0x7fe5a80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:10.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.822 7fe5afa23700 1 -- 172.21.15.32:0/4048342337 shutdown_connections 2021-04-09T18:25:10.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.822 7fe5afa23700 1 -- 172.21.15.32:0/4048342337 wait complete. 2021-04-09T18:25:10.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.842 7fe5afa23700 1 Processor -- start 2021-04-09T18:25:10.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.842 7fe5afa23700 1 -- start start 2021-04-09T18:25:10.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5afa23700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5a80a80d0 0x7fe5a81a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:10.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5afa23700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5a81a4ce0 0x7fe5a81a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:10.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5afa23700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5a81a5d70 0x7fe5a808ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:10.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5afa23700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a81a5250 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5afa23700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a81a61b0 con 0x7fe5a80a80d0 2021-04-09T18:25:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5afa23700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a818be80 con 0x7fe5a81a5d70 2021-04-09T18:25:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5ad03e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5a81a4ce0 0x7fe5a81a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5ad03e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5a81a4ce0 0x7fe5a81a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52034/0 (socket says 172.21.15.32:52034) 2021-04-09T18:25:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5ad03e700 1 -- 172.21.15.32:0/1225351929 learned_addr learned my addr 172.21.15.32:0/1225351929 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:10.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5ad83f700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5a80a80d0 0x7fe5a81a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:10.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5ae040700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5a81a5d70 0x7fe5a808ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:10.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5ad03e700 1 -- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5a81a5d70 msgr2=0x7fe5a808ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:10.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.843 7fe5ad03e700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5a81a5d70 0x7fe5a808ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:10.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe5ad03e700 1 -- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5a80a80d0 msgr2=0x7fe5a81a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:10.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe5ad03e700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5a80a80d0 0x7fe5a81a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:10.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe5ad03e700 1 -- 172.21.15.32:0/1225351929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a808b2b0 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe5ae040700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5a81a5d70 0x7fe5a808ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:10.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe5ad03e700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5a81a4ce0 0x7fe5a81a5750 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7fe598006f10 tx=0x7fe59800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:10.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe59e7fc700 1 -- 172.21.15.32:0/1225351929 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe59800bd80 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe59e7fc700 1 -- 172.21.15.32:0/1225351929 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe598000e20 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a808b2b0 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe59e7fc700 1 -- 172.21.15.32:0/1225351929 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5980054f0 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.844 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a808b2b0 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.845 7fe59e7fc700 1 -- 172.21.15.32:0/1225351929 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe5980059c0 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.845 7fe59e7fc700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe590018930 0x7fe59001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:10.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.845 7fe59e7fc700 1 -- 172.21.15.32:0/1225351929 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe598032a80 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.845 7fe5ad83f700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe590018930 0x7fe59001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:10.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.845 7fe5ad83f700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe590018930 0x7fe59001ae10 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:10.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.865 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a81b7050 con 0x7fe5a81a4ce0 2021-04-09T18:25:10.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:10.868 7fe59e7fc700 1 -- 172.21.15.32:0/1225351929 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe598000e20 con 0x7fe5a81a4ce0 2021-04-09T18:25:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.060 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5a81a6170 con 0x7fe5a81a4ce0 2021-04-09T18:25:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.061 7fe59e7fc700 1 -- 172.21.15.32:0/1225351929 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+2816 (secure 0 0 0) 0x7fe598005690 con 0x7fe5a81a4ce0 2021-04-09T18:25:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 259 2021-04-09T18:25:11.127 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:11.128 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":259,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:07.007490","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":8740,"mds_1":8749,"mds_2":8752},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8749":{"gid":8749,"name":"d","rank":1,"incarnation":255,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8752":{"gid":8752,"name":"b","rank":2,"incarnation":257,"state":"up:stopping","state_seq":6,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:11.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.063 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe590018930 msgr2=0x7fe59001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:11.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.063 7fe5afa23700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe590018930 0x7fe59001ae10 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:11.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.063 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5a81a4ce0 msgr2=0x7fe5a81a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:11.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.063 7fe5afa23700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5a81a4ce0 0x7fe5a81a5750 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7fe598006f10 tx=0x7fe59800b990).stop 2021-04-09T18:25:11.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.064 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 shutdown_connections 2021-04-09T18:25:11.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.064 7fe5afa23700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe590018930 0x7fe59001ae10 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:11.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.064 7fe5afa23700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5a81a4ce0 0x7fe5a81a5750 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:11.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.064 7fe5afa23700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5a81a5d70 0x7fe5a808ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:11.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.064 7fe5afa23700 1 --2- 172.21.15.32:0/1225351929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5a80a80d0 0x7fe5a81a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:11.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.064 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 >> 172.21.15.32:0/1225351929 conn(0x7fe5a8190570 msgr2=0x7fe5a818ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:11.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.064 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 shutdown_connections 2021-04-09T18:25:11.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:11.064 7fe5afa23700 1 -- 172.21.15.32:0/1225351929 wait complete. 2021-04-09T18:25:11.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 259, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:24:54.971184', 'modified': '2021-04-09 18:25:07.007490', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1, 2], 'up': {'mds_0': 8740, 'mds_1': 8749, 'mds_2': 8752}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8740': {'gid': 8740, 'name': 'a', 'rank': 0, 'incarnation': 250, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2311632030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2311632030}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2311632030}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8749': {'gid': 8749, 'name': 'd', 'rank': 1, 'incarnation': 255, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/2147753572', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2147753572}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 2147753572}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8752': {'gid': 8752, 'name': 'b', 'rank': 2, 'incarnation': 257, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.86:6837/1761380306', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1761380306}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1761380306}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:11.542 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8752:up:stopping 2021-04-09T18:25:12.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:12.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.634 7f12f9e7b700 1 Processor -- start 2021-04-09T18:25:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.634 7f12f9e7b700 1 -- start start 2021-04-09T18:25:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.634 7f12f9e7b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f12f410d010 0x7f12f40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.634 7f12f9e7b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f40d1da0 0x7f12f40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.634 7f12f9e7b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12f40d82c0 0x7f12f410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:12.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.634 7f12f9e7b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f40d8d30 con 0x7f12f40d82c0 2021-04-09T18:25:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.634 7f12f9e7b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f40d9220 con 0x7f12f410d010 2021-04-09T18:25:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f9e7b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f410d480 con 0x7f12f40d1da0 2021-04-09T18:25:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12f40d82c0 0x7f12f410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:12.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12f40d82c0 0x7f12f410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52046/0 (socket says 172.21.15.32:52046) 2021-04-09T18:25:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f3fff700 1 -- 172.21.15.32:0/2767020113 learned_addr learned my addr 172.21.15.32:0/2767020113 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f2ffd700 1 --2- 172.21.15.32:0/2767020113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f40d1da0 0x7f12f40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f37fe700 1 --2- 172.21.15.32:0/2767020113 >> v2:172.21.15.86:3300/0 conn(0x7f12f410d010 0x7f12f40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f3fff700 1 -- 172.21.15.32:0/2767020113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f40d1da0 msgr2=0x7f12f40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:12.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f3fff700 1 --2- 172.21.15.32:0/2767020113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f40d1da0 0x7f12f40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f3fff700 1 -- 172.21.15.32:0/2767020113 >> v2:172.21.15.86:3300/0 conn(0x7f12f410d010 msgr2=0x7f12f40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f3fff700 1 --2- 172.21.15.32:0/2767020113 >> v2:172.21.15.86:3300/0 conn(0x7f12f410d010 0x7f12f40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f3fff700 1 -- 172.21.15.32:0/2767020113 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f40d86c0 con 0x7f12f40d82c0 2021-04-09T18:25:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f3fff700 1 --2- 172.21.15.32:0/2767020113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f40d82c0 0x7f12f410da00 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 rx=0x7f12ec006f00 tx=0x7f12ec009e90).ready entity=mon.0 client_cookie=9200949880375ee3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f27fc700 1 -- 172.21.15.32:0/2767020113 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12ec00b300 con 0x7f12f40d82c0 2021-04-09T18:25:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f27fc700 1 -- 172.21.15.32:0/2767020113 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec00b640 con 0x7f12f40d82c0 2021-04-09T18:25:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f27fc700 1 -- 172.21.15.32:0/2767020113 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12ec00bb20 con 0x7f12f40d82c0 2021-04-09T18:25:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f9e7b700 1 -- 172.21.15.32:0/2767020113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f40d82c0 msgr2=0x7f12f410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:12.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.635 7f12f9e7b700 1 --2- 172.21.15.32:0/2767020113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f40d82c0 0x7f12f410da00 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 rx=0x7f12ec006f00 tx=0x7f12ec009e90).stop 2021-04-09T18:25:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.636 7f12f9e7b700 1 -- 172.21.15.32:0/2767020113 shutdown_connections 2021-04-09T18:25:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.636 7f12f9e7b700 1 --2- 172.21.15.32:0/2767020113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f40d82c0 0x7f12f410da00 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.636 7f12f9e7b700 1 --2- 172.21.15.32:0/2767020113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f40d1da0 0x7f12f40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.636 7f12f9e7b700 1 --2- 172.21.15.32:0/2767020113 >> v2:172.21.15.86:3300/0 conn(0x7f12f410d010 0x7f12f40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.636 7f12f9e7b700 1 -- 172.21.15.32:0/2767020113 >> 172.21.15.32:0/2767020113 conn(0x7f12f40ce5f0 msgr2=0x7f12f40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:12.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.636 7f12f9e7b700 1 -- 172.21.15.32:0/2767020113 shutdown_connections 2021-04-09T18:25:12.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.636 7f12f9e7b700 1 -- 172.21.15.32:0/2767020113 wait complete. 2021-04-09T18:25:12.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.656 7f12f9e7b700 1 Processor -- start 2021-04-09T18:25:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.656 7f12f9e7b700 1 -- start start 2021-04-09T18:25:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f9e7b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f41ac670 0x7f12f41a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f9e7b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f41a5dc0 0x7f12f40ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:12.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f9e7b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f41aca70 0x7f12f40ba800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f9e7b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f41a5220 con 0x7f12f41a5dc0 2021-04-09T18:25:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f9e7b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f418e100 con 0x7f12f41ac670 2021-04-09T18:25:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f9e7b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f419ccc0 con 0x7f12f41aca70 2021-04-09T18:25:12.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f41a5dc0 0x7f12f40ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f41a5dc0 0x7f12f40ba1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52050/0 (socket says 172.21.15.32:52050) 2021-04-09T18:25:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f2ffd700 1 -- 172.21.15.32:0/2587246220 learned_addr learned my addr 172.21.15.32:0/2587246220 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f37fe700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f41ac670 0x7f12f41a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f3fff700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f41aca70 0x7f12f40ba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:12.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f2ffd700 1 -- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f41aca70 msgr2=0x7f12f40ba800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f2ffd700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f41aca70 0x7f12f40ba800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f2ffd700 1 -- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f41ac670 msgr2=0x7f12f41a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f2ffd700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f41ac670 0x7f12f41a57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.657 7f12f2ffd700 1 -- 172.21.15.32:0/2587246220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f40bae20 con 0x7f12f41a5dc0 2021-04-09T18:25:12.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.658 7f12f2ffd700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f41a5dc0 0x7f12f40ba1e0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 rx=0x7f12e8006be0 tx=0x7f12e800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.658 7f12dbfff700 1 -- 172.21.15.32:0/2587246220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12e80064d0 con 0x7f12f41a5dc0 2021-04-09T18:25:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.658 7f12dbfff700 1 -- 172.21.15.32:0/2587246220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e800bd80 con 0x7f12f41a5dc0 2021-04-09T18:25:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.658 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f40bae20 con 0x7f12f41a5dc0 2021-04-09T18:25:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.658 7f12dbfff700 1 -- 172.21.15.32:0/2587246220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12e8002030 con 0x7f12f41a5dc0 2021-04-09T18:25:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.658 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f40bae20 con 0x7f12f41a5dc0 2021-04-09T18:25:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.658 7f12dbfff700 1 -- 172.21.15.32:0/2587246220 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f12e8006c90 con 0x7f12f41a5dc0 2021-04-09T18:25:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.659 7f12dbfff700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12d4018930 0x7f12d401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.659 7f12dbfff700 1 -- 172.21.15.32:0/2587246220 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f12e8032a10 con 0x7f12f41a5dc0 2021-04-09T18:25:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.659 7f12f37fe700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12d4018930 0x7f12d401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:12.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.659 7f12f37fe700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12d4018930 0x7f12d401ae10 crc :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:12.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.678 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f4182d10 con 0x7f12f41a5dc0 2021-04-09T18:25:12.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.682 7f12dbfff700 1 -- 172.21.15.32:0/2587246220 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f12e8006820 con 0x7f12f41a5dc0 2021-04-09T18:25:12.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.871 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12f4182d10 con 0x7f12f41a5dc0 2021-04-09T18:25:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.871 7f12dbfff700 1 -- 172.21.15.32:0/2587246220 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+2816 (secure 0 0 0) 0x7f12e800bd80 con 0x7f12f41a5dc0 2021-04-09T18:25:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 259 2021-04-09T18:25:12.939 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:12.940 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":259,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:07.007490","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":8740,"mds_1":8749,"mds_2":8752},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8749":{"gid":8749,"name":"d","rank":1,"incarnation":255,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8752":{"gid":8752,"name":"b","rank":2,"incarnation":257,"state":"up:stopping","state_seq":6,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12d4018930 msgr2=0x7f12d401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12d4018930 0x7f12d401ae10 crc :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f41a5dc0 msgr2=0x7f12f40ba1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f41a5dc0 0x7f12f40ba1e0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 rx=0x7f12e8006be0 tx=0x7f12e800b990).stop 2021-04-09T18:25:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 shutdown_connections 2021-04-09T18:25:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12d4018930 0x7f12d401ae10 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12f41a5dc0 0x7f12f40ba1e0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12f41aca70 0x7f12f40ba800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 --2- 172.21.15.32:0/2587246220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f41ac670 0x7f12f41a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 >> 172.21.15.32:0/2587246220 conn(0x7f12f419b890 msgr2=0x7f12f41a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 shutdown_connections 2021-04-09T18:25:12.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:12.874 7f12f9e7b700 1 -- 172.21.15.32:0/2587246220 wait complete. 2021-04-09T18:25:13.352 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 259, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:24:54.971184', 'modified': '2021-04-09 18:25:07.007490', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1, 2], 'up': {'mds_0': 8740, 'mds_1': 8749, 'mds_2': 8752}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8740': {'gid': 8740, 'name': 'a', 'rank': 0, 'incarnation': 250, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2311632030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2311632030}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2311632030}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8749': {'gid': 8749, 'name': 'd', 'rank': 1, 'incarnation': 255, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/2147753572', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2147753572}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 2147753572}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8752': {'gid': 8752, 'name': 'b', 'rank': 2, 'incarnation': 257, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.86:6837/1761380306', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1761380306}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1761380306}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:13.352 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8752:up:stopping 2021-04-09T18:25:14.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:14.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.449 7f7917e1e700 1 Processor -- start 2021-04-09T18:25:14.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.449 7f7917e1e700 1 -- start start 2021-04-09T18:25:14.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f7917e1e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f791010d010 0x7f79100d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:14.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f7917e1e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79100d8490 0x7f791010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:14.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f7917e1e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791010d480 con 0x7f79100d1da0 2021-04-09T18:25:14.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f7917e1e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f791010e780 con 0x7f79100d8490 2021-04-09T18:25:14.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f7917e1e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79100d8d30 con 0x7f791010d010 2021-04-09T18:25:14.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f791643b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79100d8490 0x7f791010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:14.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f791643b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f79100d8490 0x7f791010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52058/0 (socket says 172.21.15.32:52058) 2021-04-09T18:25:14.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f791643b700 1 -- 172.21.15.32:0/3578073830 learned_addr learned my addr 172.21.15.32:0/3578073830 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:14.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f7915c3a700 1 --2- 172.21.15.32:0/3578073830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f791010d010 0x7f79100d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:14.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f791643b700 1 -- 172.21.15.32:0/3578073830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f791010d010 msgr2=0x7f79100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:14.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f791643b700 1 --2- 172.21.15.32:0/3578073830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f791010d010 0x7f79100d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f791643b700 1 -- 172.21.15.32:0/3578073830 >> v1:172.21.15.86:6789/0 conn(0x7f79100d1da0 legacy=0x7f79100d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f791643b700 1 -- 172.21.15.32:0/3578073830 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79100d9220 con 0x7f79100d8490 2021-04-09T18:25:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.450 7f7915c3a700 1 --2- 172.21.15.32:0/3578073830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f791010d010 0x7f79100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:14.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.451 7f791643b700 1 --2- 172.21.15.32:0/3578073830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79100d8490 0x7f791010da00 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 rx=0x7f7908006f00 tx=0x7f7908009f10).ready entity=mon.0 client_cookie=d088e3bad41deb73 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.451 7f7914c38700 1 -- 172.21.15.32:0/3578073830 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f790800b3b0 con 0x7f79100d8490 2021-04-09T18:25:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.451 7f7914c38700 1 -- 172.21.15.32:0/3578073830 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790800b730 con 0x7f79100d8490 2021-04-09T18:25:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.451 7f7914c38700 1 -- 172.21.15.32:0/3578073830 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f790800bc10 con 0x7f79100d8490 2021-04-09T18:25:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.451 7f7917e1e700 1 -- 172.21.15.32:0/3578073830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79100d8490 msgr2=0x7f791010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:14.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.451 7f7917e1e700 1 --2- 172.21.15.32:0/3578073830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79100d8490 0x7f791010da00 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 rx=0x7f7908006f00 tx=0x7f7908009f10).stop 2021-04-09T18:25:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.451 7f7917e1e700 1 -- 172.21.15.32:0/3578073830 shutdown_connections 2021-04-09T18:25:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.451 7f7917e1e700 1 --2- 172.21.15.32:0/3578073830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79100d8490 0x7f791010da00 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.451 7f7917e1e700 1 --2- 172.21.15.32:0/3578073830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f791010d010 0x7f79100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.452 7f7917e1e700 1 -- 172.21.15.32:0/3578073830 >> 172.21.15.32:0/3578073830 conn(0x7f79100ce5f0 msgr2=0x7f79100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.452 7f7917e1e700 1 -- 172.21.15.32:0/3578073830 shutdown_connections 2021-04-09T18:25:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.452 7f7917e1e700 1 -- 172.21.15.32:0/3578073830 wait complete. 2021-04-09T18:25:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.472 7f7917e1e700 1 Processor -- start 2021-04-09T18:25:14.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.472 7f7917e1e700 1 -- start start 2021-04-09T18:25:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.472 7f7917e1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79101a5210 0x7f791008eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7917e1e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79101a8ab0 0x7f79101a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7917e1e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79101a76d0 0x7f79101a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:14.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7917e1e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791017ea50 con 0x7f79101a5210 2021-04-09T18:25:14.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7917e1e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791008f4f0 con 0x7f79101a8ab0 2021-04-09T18:25:14.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7917e1e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791008ea60 con 0x7f79101a76d0 2021-04-09T18:25:14.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915c3a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79101a5210 0x7f791008eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:14.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915c3a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79101a5210 0x7f791008eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52062/0 (socket says 172.21.15.32:52062) 2021-04-09T18:25:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915c3a700 1 -- 172.21.15.32:0/139761237 learned_addr learned my addr 172.21.15.32:0/139761237 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915439700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79101a8ab0 0x7f79101a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f791643b700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79101a76d0 0x7f79101a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915c3a700 1 -- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79101a76d0 msgr2=0x7f79101a7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:14.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915c3a700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79101a76d0 0x7f79101a7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915c3a700 1 -- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79101a8ab0 msgr2=0x7f79101a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915c3a700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79101a8ab0 0x7f79101a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915c3a700 1 -- 172.21.15.32:0/139761237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79101a80f0 con 0x7f79101a5210 2021-04-09T18:25:14.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7915c3a700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79101a5210 0x7f791008eed0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7f790c00b580 tx=0x7f790c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f79027fc700 1 -- 172.21.15.32:0/139761237 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f790c00bd10 con 0x7f79101a5210 2021-04-09T18:25:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f79027fc700 1 -- 172.21.15.32:0/139761237 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c000c90 con 0x7f79101a5210 2021-04-09T18:25:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f7917e1e700 1 -- 172.21.15.32:0/139761237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79101a80f0 con 0x7f79101a5210 2021-04-09T18:25:14.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.473 7f79027fc700 1 -- 172.21.15.32:0/139761237 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f790c002220 con 0x7f79101a5210 2021-04-09T18:25:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.474 7f7917e1e700 1 -- 172.21.15.32:0/139761237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79101a80f0 con 0x7f79101a5210 2021-04-09T18:25:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.474 7f79027fc700 1 -- 172.21.15.32:0/139761237 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f790c00a000 con 0x7f79101a5210 2021-04-09T18:25:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.474 7f79027fc700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78f8018930 0x7f78f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.474 7f79027fc700 1 -- 172.21.15.32:0/139761237 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f790c000940 con 0x7f79101a5210 2021-04-09T18:25:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.474 7f7915439700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78f8018930 0x7f78f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:14.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.475 7f7915439700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78f8018930 0x7f78f801ae10 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:14.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.494 7f7917e1e700 1 -- 172.21.15.32:0/139761237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79101b7070 con 0x7f79101a5210 2021-04-09T18:25:14.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.497 7f79027fc700 1 -- 172.21.15.32:0/139761237 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f790c00a000 con 0x7f79101a5210 2021-04-09T18:25:14.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.690 7f7917e1e700 1 -- 172.21.15.32:0/139761237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79101b7070 con 0x7f79101a5210 2021-04-09T18:25:14.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.690 7f79027fc700 1 -- 172.21.15.32:0/139761237 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+2816 (secure 0 0 0) 0x7f790c000940 con 0x7f79101a5210 2021-04-09T18:25:14.757 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 259 2021-04-09T18:25:14.757 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:14.757 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":259,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:07.007490","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":8740,"mds_1":8749,"mds_2":8752},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8749":{"gid":8749,"name":"d","rank":1,"incarnation":255,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8752":{"gid":8752,"name":"b","rank":2,"incarnation":257,"state":"up:stopping","state_seq":6,"addr":"172.21.15.86:6837/1761380306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1761380306},{"type":"v1","addr":"172.21.15.86:6837","nonce":1761380306}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:14.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 -- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78f8018930 msgr2=0x7f78f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:14.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78f8018930 0x7f78f801ae10 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 -- 172.21.15.32:0/139761237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79101a5210 msgr2=0x7f791008eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:14.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79101a5210 0x7f791008eed0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7f790c00b580 tx=0x7f790c00b920).stop 2021-04-09T18:25:14.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 -- 172.21.15.32:0/139761237 shutdown_connections 2021-04-09T18:25:14.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78f8018930 0x7f78f801ae10 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79101a76d0 0x7f79101a7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79101a8ab0 0x7f79101a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 --2- 172.21.15.32:0/139761237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79101a5210 0x7f791008eed0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:14.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 -- 172.21.15.32:0/139761237 >> 172.21.15.32:0/139761237 conn(0x7f79101a99c0 msgr2=0x7f791017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:14.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 -- 172.21.15.32:0/139761237 shutdown_connections 2021-04-09T18:25:14.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:14.693 7f7917e1e700 1 -- 172.21.15.32:0/139761237 wait complete. 2021-04-09T18:25:15.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 259, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:24:54.971184', 'modified': '2021-04-09 18:25:07.007490', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1, 2], 'up': {'mds_0': 8740, 'mds_1': 8749, 'mds_2': 8752}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8740': {'gid': 8740, 'name': 'a', 'rank': 0, 'incarnation': 250, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2311632030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2311632030}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2311632030}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8749': {'gid': 8749, 'name': 'd', 'rank': 1, 'incarnation': 255, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/2147753572', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2147753572}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 2147753572}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8752': {'gid': 8752, 'name': 'b', 'rank': 2, 'incarnation': 257, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.86:6837/1761380306', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1761380306}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1761380306}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:15.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8752:up:stopping 2021-04-09T18:25:15.581 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:25:16.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:16.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.264 7f0091603700 1 Processor -- start 2021-04-09T18:25:16.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.264 7f0091603700 1 -- start start 2021-04-09T18:25:16.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.264 7f0091603700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f008c0cffa0 0x7f008c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:16.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.264 7f0091603700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f008c0d6c80 0x7f008c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:16.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.264 7f0091603700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008c0d7280 con 0x7f008c0d7800 2021-04-09T18:25:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.264 7f0091603700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008c0d7080 con 0x7f008c0d6c80 2021-04-09T18:25:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.264 7f0091603700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008c0d91e0 con 0x7f008c0cffa0 2021-04-09T18:25:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.264 7f008bac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f008c0d6c80 0x7f008c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:16.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008aabe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f008c0d7800 0x7f008c0d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33432/0 (socket says 172.21.15.32:33432) 2021-04-09T18:25:16.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008aabe700 1 -- 172.21.15.32:0/3624502748 learned_addr learned my addr 172.21.15.32:0/3624502748 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:16.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008bac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f008c0d6c80 0x7f008c10cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52072/0 (socket says 172.21.15.32:52072) 2021-04-09T18:25:16.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008b2bf700 1 --2- 172.21.15.32:0/3624502748 >> v2:172.21.15.86:3300/0 conn(0x7f008c0cffa0 0x7f008c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:16.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008bac0700 1 -- 172.21.15.32:0/3624502748 >> v2:172.21.15.86:3300/0 conn(0x7f008c0cffa0 msgr2=0x7f008c0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:16.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008bac0700 1 --2- 172.21.15.32:0/3624502748 >> v2:172.21.15.86:3300/0 conn(0x7f008c0cffa0 0x7f008c0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008bac0700 1 -- 172.21.15.32:0/3624502748 >> v1:172.21.15.32:6789/0 conn(0x7f008c0d7800 legacy=0x7f008c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:16.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008bac0700 1 -- 172.21.15.32:0/3624502748 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c0d83f0 con 0x7f008c0d6c80 2021-04-09T18:25:16.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008a2bd700 1 -- 172.21.15.32:0/3624502748 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920583986 0 0) 0x7f00780009b0 con 0x7f008c0d7800 2021-04-09T18:25:16.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008b2bf700 1 --2- 172.21.15.32:0/3624502748 >> v2:172.21.15.86:3300/0 conn(0x7f008c0cffa0 0x7f008c0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:16.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008bac0700 1 --2- 172.21.15.32:0/3624502748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c0d6c80 0x7f008c10cfd0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7f0084006e30 tx=0x7f0084009e60).ready entity=mon.0 client_cookie=df32517fc443846f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:16.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008a2bd700 1 -- 172.21.15.32:0/3624502748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0084006a50 con 0x7f008c0d6c80 2021-04-09T18:25:16.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008a2bd700 1 -- 172.21.15.32:0/3624502748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008400b300 con 0x7f008c0d6c80 2021-04-09T18:25:16.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f008a2bd700 1 -- 172.21.15.32:0/3624502748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f008400b750 con 0x7f008c0d6c80 2021-04-09T18:25:16.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f0091603700 1 -- 172.21.15.32:0/3624502748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c0d6c80 msgr2=0x7f008c10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:16.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.265 7f0091603700 1 --2- 172.21.15.32:0/3624502748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c0d6c80 0x7f008c10cfd0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7f0084006e30 tx=0x7f0084009e60).stop 2021-04-09T18:25:16.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.266 7f0091603700 1 -- 172.21.15.32:0/3624502748 shutdown_connections 2021-04-09T18:25:16.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.266 7f0091603700 1 --2- 172.21.15.32:0/3624502748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c0d6c80 0x7f008c10cfd0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.266 7f0091603700 1 --2- 172.21.15.32:0/3624502748 >> v2:172.21.15.86:3300/0 conn(0x7f008c0cffa0 0x7f008c0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.266 7f0091603700 1 -- 172.21.15.32:0/3624502748 >> 172.21.15.32:0/3624502748 conn(0x7f008c0ce5c0 msgr2=0x7f008c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.266 7f0091603700 1 -- 172.21.15.32:0/3624502748 shutdown_connections 2021-04-09T18:25:16.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.266 7f0091603700 1 -- 172.21.15.32:0/3624502748 wait complete. 2021-04-09T18:25:16.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.286 7f0091603700 1 Processor -- start 2021-04-09T18:25:16.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.286 7f0091603700 1 -- start start 2021-04-09T18:25:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f0091603700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c1acff0 0x7f008c1847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f0091603700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c183fc0 0x7f008c1a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f0091603700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c186ff0 0x7f008c1873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f0091603700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c188c30 con 0x7f008c1acff0 2021-04-09T18:25:16.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f0091603700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c1843c0 con 0x7f008c183fc0 2021-04-09T18:25:16.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f0091603700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c1a5c20 con 0x7f008c186ff0 2021-04-09T18:25:16.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c1acff0 0x7f008c1847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:16.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c1acff0 0x7f008c1847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52076/0 (socket says 172.21.15.32:52076) 2021-04-09T18:25:16.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008b2bf700 1 -- 172.21.15.32:0/403968912 learned_addr learned my addr 172.21.15.32:0/403968912 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:16.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008bac0700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c186ff0 0x7f008c1873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:16.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008aabe700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c183fc0 0x7f008c1a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:16.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008b2bf700 1 -- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c186ff0 msgr2=0x7f008c1873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:16.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008b2bf700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c186ff0 0x7f008c1873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008b2bf700 1 -- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c183fc0 msgr2=0x7f008c1a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:16.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008b2bf700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c183fc0 0x7f008c1a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008b2bf700 1 -- 172.21.15.32:0/403968912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c187a10 con 0x7f008c1acff0 2021-04-09T18:25:16.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.287 7f008b2bf700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c1acff0 0x7f008c1847a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 rx=0x7f007c006f20 tx=0x7f007c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:16.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.288 7f0083fff700 1 -- 172.21.15.32:0/403968912 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f007c0009b0 con 0x7f008c1acff0 2021-04-09T18:25:16.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.288 7f0083fff700 1 -- 172.21.15.32:0/403968912 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007c00bd80 con 0x7f008c1acff0 2021-04-09T18:25:16.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.288 7f0091603700 1 -- 172.21.15.32:0/403968912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008c187a10 con 0x7f008c1acff0 2021-04-09T18:25:16.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.288 7f0083fff700 1 -- 172.21.15.32:0/403968912 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f007c002220 con 0x7f008c1acff0 2021-04-09T18:25:16.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.288 7f0091603700 1 -- 172.21.15.32:0/403968912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c187a10 con 0x7f008c1acff0 2021-04-09T18:25:16.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.288 7f0083fff700 1 -- 172.21.15.32:0/403968912 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f007c000c90 con 0x7f008c1acff0 2021-04-09T18:25:16.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.289 7f0083fff700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0068018930 0x7f006801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:16.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.289 7f0083fff700 1 -- 172.21.15.32:0/403968912 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f007c032a10 con 0x7f008c1acff0 2021-04-09T18:25:16.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.289 7f008aabe700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0068018930 0x7f006801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:16.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.289 7f008aabe700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0068018930 0x7f006801adc0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:16.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.309 7f0091603700 1 -- 172.21.15.32:0/403968912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008c182dc0 con 0x7f008c1acff0 2021-04-09T18:25:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.312 7f0083fff700 1 -- 172.21.15.32:0/403968912 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f007c006bb0 con 0x7f008c1acff0 2021-04-09T18:25:16.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.504 7f0091603700 1 -- 172.21.15.32:0/403968912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f008c0d7280 con 0x7f008c1acff0 2021-04-09T18:25:16.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.504 7f0083fff700 1 -- 172.21.15.32:0/403968912 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+2463 (secure 0 0 0) 0x7f007c00bd80 con 0x7f008c1acff0 2021-04-09T18:25:16.571 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 260 2021-04-09T18:25:16.572 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:16.572 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":260,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:15.457126","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8740,"mds_1":8749},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8749":{"gid":8749,"name":"d","rank":1,"incarnation":255,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:16.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 -- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0068018930 msgr2=0x7f006801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:16.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0068018930 0x7f006801adc0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 -- 172.21.15.32:0/403968912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c1acff0 msgr2=0x7f008c1847a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c1acff0 0x7f008c1847a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 rx=0x7f007c006f20 tx=0x7f007c00b990).stop 2021-04-09T18:25:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 -- 172.21.15.32:0/403968912 shutdown_connections 2021-04-09T18:25:16.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0068018930 0x7f006801adc0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c186ff0 0x7f008c1873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c183fc0 0x7f008c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 --2- 172.21.15.32:0/403968912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f008c1acff0 0x7f008c1847a0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 -- 172.21.15.32:0/403968912 >> 172.21.15.32:0/403968912 conn(0x7f008c09abb0 msgr2=0x7f008c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:16.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 -- 172.21.15.32:0/403968912 shutdown_connections 2021-04-09T18:25:16.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:16.507 7f0091603700 1 -- 172.21.15.32:0/403968912 wait complete. 2021-04-09T18:25:16.985 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 260, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:24:54.971184', 'modified': '2021-04-09 18:25:15.457126', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8740, 'mds_1': 8749}, 'failed': [], 'damaged': [], 'stopped': [2], 'info': {'gid_8740': {'gid': 8740, 'name': 'a', 'rank': 0, 'incarnation': 250, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2311632030', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2311632030}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2311632030}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_8749': {'gid': 8749, 'name': 'd', 'rank': 1, 'incarnation': 255, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/2147753572', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2147753572}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 2147753572}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:16.985 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:25:16.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:25:17.123 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:17.123 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:25:17.123 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:25:17.123 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 8740, 2021-04-09T18:25:17.124 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:25:17.124 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:25:17.124 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 22.085205194, 2021-04-09T18:25:17.124 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 260, 2021-04-09T18:25:17.125 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 119, 2021-04-09T18:25:17.125 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 119, 2021-04-09T18:25:17.125 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 27.560999942999999 2021-04-09T18:25:17.125 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:17.129 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 8740, "want_state": "up:active", "state": "up:active", "rank_uptime": 22.085205194, "mdsmap_epoch": 260, "osdmap_epoch": 119, "osdmap_epoch_barrier": 119, "uptime": 27.560999942999999 } 2021-04-09T18:25:17.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:25:17.256 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:25:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:25:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 1, 2021-04-09T18:25:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 8749, 2021-04-09T18:25:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:25:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:25:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 12.193520273000001, 2021-04-09T18:25:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 260, 2021-04-09T18:25:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 119, 2021-04-09T18:25:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 119, 2021-04-09T18:25:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 27.684005170999999 2021-04-09T18:25:17.259 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:25:17.262 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 8749, "want_state": "up:active", "state": "up:active", "rank_uptime": 12.193520273000001, "mdsmap_epoch": 260, "osdmap_epoch": 119, "osdmap_epoch_barrier": 119, "uptime": 27.684005170999999 } 2021-04-09T18:25:17.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:25:17.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.345 7f9795cc4700 1 Processor -- start 2021-04-09T18:25:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.346 7f9795cc4700 1 -- start start 2021-04-09T18:25:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.346 7f9795cc4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f97900d6c80 0x7f97900d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:17.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.346 7f9795cc4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97900d2440 0x7f979010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:17.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.346 7f9795cc4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97900d7280 con 0x7f97900cffa0 2021-04-09T18:25:17.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.346 7f9795cc4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97900d7080 con 0x7f97900d6c80 2021-04-09T18:25:17.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.346 7f9795cc4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97900d91e0 con 0x7f97900d2440 2021-04-09T18:25:17.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.346 7f978effd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f97900d6c80 0x7f97900d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:17.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.346 7f978f7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f97900cffa0 0x7f97900d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33442/0 (socket says 172.21.15.32:33442) 2021-04-09T18:25:17.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.346 7f978f7fe700 1 -- 172.21.15.32:0/683517255 learned_addr learned my addr 172.21.15.32:0/683517255 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:17.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978ffff700 1 --2- 172.21.15.32:0/683517255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97900d2440 0x7f979010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:17.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978effd700 1 -- 172.21.15.32:0/683517255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97900d2440 msgr2=0x7f979010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:17.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978effd700 1 --2- 172.21.15.32:0/683517255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97900d2440 0x7f979010cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978effd700 1 -- 172.21.15.32:0/683517255 >> v1:172.21.15.32:6789/0 conn(0x7f97900cffa0 legacy=0x7f97900d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:17.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978effd700 1 -- 172.21.15.32:0/683517255 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900d2840 con 0x7f97900d6c80 2021-04-09T18:25:17.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978ffff700 1 --2- 172.21.15.32:0/683517255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97900d2440 0x7f979010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:17.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978effd700 1 --2- 172.21.15.32:0/683517255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97900d6c80 0x7f97900d7ff0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 rx=0x7f9784006650 tx=0x7f9784009a80).ready entity=mon.0 client_cookie=eec2131853e2d5a0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:17.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978e7fc700 1 -- 172.21.15.32:0/683517255 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97840009b0 con 0x7f97900d6c80 2021-04-09T18:25:17.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978e7fc700 1 -- 172.21.15.32:0/683517255 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978400c0c0 con 0x7f97900d6c80 2021-04-09T18:25:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f978e7fc700 1 -- 172.21.15.32:0/683517255 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f978400c510 con 0x7f97900d6c80 2021-04-09T18:25:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f9795cc4700 1 -- 172.21.15.32:0/683517255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97900d6c80 msgr2=0x7f97900d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.347 7f9795cc4700 1 --2- 172.21.15.32:0/683517255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97900d6c80 0x7f97900d7ff0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 rx=0x7f9784006650 tx=0x7f9784009a80).stop 2021-04-09T18:25:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.348 7f9795cc4700 1 -- 172.21.15.32:0/683517255 shutdown_connections 2021-04-09T18:25:17.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.348 7f9795cc4700 1 --2- 172.21.15.32:0/683517255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97900d2440 0x7f979010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.348 7f9795cc4700 1 --2- 172.21.15.32:0/683517255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97900d6c80 0x7f97900d7ff0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.348 7f9795cc4700 1 -- 172.21.15.32:0/683517255 >> 172.21.15.32:0/683517255 conn(0x7f97900ce5c0 msgr2=0x7f97900cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:17.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.348 7f9795cc4700 1 -- 172.21.15.32:0/683517255 shutdown_connections 2021-04-09T18:25:17.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.348 7f9795cc4700 1 -- 172.21.15.32:0/683517255 wait complete. 2021-04-09T18:25:17.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.368 7f9795cc4700 1 Processor -- start 2021-04-09T18:25:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.368 7f9795cc4700 1 -- start start 2021-04-09T18:25:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9795cc4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9790194770 0x7f9790090bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9795cc4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97900911f0 0x7f979009aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:17.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9795cc4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979009b100 0x7f97901b7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:17.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9795cc4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9790091700 con 0x7f9790194770 2021-04-09T18:25:17.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9795cc4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979009a640 con 0x7f97900911f0 2021-04-09T18:25:17.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9795cc4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979009b500 con 0x7f979009b100 2021-04-09T18:25:17.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9790194770 0x7f9790090bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9790194770 0x7f9790090bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52090/0 (socket says 172.21.15.32:52090) 2021-04-09T18:25:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978f7fe700 1 -- 172.21.15.32:0/4009932870 learned_addr learned my addr 172.21.15.32:0/4009932870 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978effd700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97900911f0 0x7f979009aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:17.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978ffff700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979009b100 0x7f97901b7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:17.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978f7fe700 1 -- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979009b100 msgr2=0x7f97901b7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:17.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978f7fe700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979009b100 0x7f97901b7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978f7fe700 1 -- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97900911f0 msgr2=0x7f979009aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:17.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978f7fe700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97900911f0 0x7f979009aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978f7fe700 1 -- 172.21.15.32:0/4009932870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979009b720 con 0x7f9790194770 2021-04-09T18:25:17.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978effd700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97900911f0 0x7f979009aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:17.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978ffff700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979009b100 0x7f97901b7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:17.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f978f7fe700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9790194770 0x7f9790090bd0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 rx=0x7f9780001400 tx=0x7f978000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9777fff700 1 -- 172.21.15.32:0/4009932870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f978000acf0 con 0x7f9790194770 2021-04-09T18:25:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9777fff700 1 -- 172.21.15.32:0/4009932870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97800009b0 con 0x7f9790194770 2021-04-09T18:25:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979009b720 con 0x7f9790194770 2021-04-09T18:25:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9777fff700 1 -- 172.21.15.32:0/4009932870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9780004590 con 0x7f9790194770 2021-04-09T18:25:17.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.369 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979009b720 con 0x7f9790194770 2021-04-09T18:25:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.370 7f9777fff700 1 -- 172.21.15.32:0/4009932870 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f97800043a0 con 0x7f9790194770 2021-04-09T18:25:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.370 7f9777fff700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97700188c0 0x7f977001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.370 7f9777fff700 1 -- 172.21.15.32:0/4009932870 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f97800327c0 con 0x7f9790194770 2021-04-09T18:25:17.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.370 7f978effd700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97700188c0 0x7f977001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:17.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.371 7f978effd700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97700188c0 0x7f977001adc0 crc :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:17.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.390 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97901b8eb0 con 0x7f9790194770 2021-04-09T18:25:17.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.394 7f9777fff700 1 -- 172.21.15.32:0/4009932870 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9780030bd0 con 0x7f9790194770 2021-04-09T18:25:17.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.581 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f97901b8eb0 con 0x7f9790194770 2021-04-09T18:25:17.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.581 7f9777fff700 1 -- 172.21.15.32:0/4009932870 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7f9780030bd0 con 0x7f9790194770 2021-04-09T18:25:17.648 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97700188c0 msgr2=0x7f977001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97700188c0 0x7f977001adc0 crc :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9790194770 msgr2=0x7f9790090bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9790194770 0x7f9790090bd0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 rx=0x7f9780001400 tx=0x7f978000a900).stop 2021-04-09T18:25:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 shutdown_connections 2021-04-09T18:25:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97700188c0 0x7f977001adc0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979009b100 0x7f97901b7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97900911f0 0x7f979009aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 --2- 172.21.15.32:0/4009932870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9790194770 0x7f9790090bd0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 >> 172.21.15.32:0/4009932870 conn(0x7f97901b72e0 msgr2=0x7f97901b7780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 shutdown_connections 2021-04-09T18:25:17.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:17.584 7f9795cc4700 1 -- 172.21.15.32:0/4009932870 wait complete. 2021-04-09T18:25:17.660 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:25:17.660 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2021-04-09T18:25:17.661 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:25:17.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:25:17.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:25:17.787 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:25:17.792 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:17.792 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:17.793 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:17.849 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:25:17.850 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:25:17.850 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:25:17.851 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:25:17.852 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:17.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[72320]: fuse finished with error 0 and tester_r 0 2021-04-09T18:25:17.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:17.959 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:17.960 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:18.008 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:25:18.009 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:25:18.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:25:18.065 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:18.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:18.114 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:18.114 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:25:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:25:18.162 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:25:18.163 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:25:18.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:25:18.164 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:25:18.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:25:18.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.338 7f142e339700 1 Processor -- start 2021-04-09T18:25:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.338 7f142e339700 1 -- start start 2021-04-09T18:25:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142e339700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14280d7ca0 0x7f14280d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:18.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142e339700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f142810da00 0x7f142810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:18.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142e339700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142810d480 con 0x7f142810d010 2021-04-09T18:25:18.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142e339700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14280d9220 con 0x7f142810da00 2021-04-09T18:25:18.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142e339700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14280d86c0 con 0x7f14280d7ca0 2021-04-09T18:25:18.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142c956700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f142810da00 0x7f142810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f1427fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f142810d010 0x7f14280d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33456/0 (socket says 172.21.15.32:33456) 2021-04-09T18:25:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f1427fff700 1 -- 172.21.15.32:0/344953282 learned_addr learned my addr 172.21.15.32:0/344953282 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f141ffff700 1 --2- 172.21.15.32:0/344953282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14280d7ca0 0x7f14280d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142c956700 1 -- 172.21.15.32:0/344953282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14280d7ca0 msgr2=0x7f14280d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:18.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142c956700 1 --2- 172.21.15.32:0/344953282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14280d7ca0 0x7f14280d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142c956700 1 -- 172.21.15.32:0/344953282 >> v1:172.21.15.32:6789/0 conn(0x7f142810d010 legacy=0x7f14280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:18.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f14277fe700 1 -- 172.21.15.32:0/344953282 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953813150 0 0) 0x7f1418000ce0 con 0x7f142810d010 2021-04-09T18:25:18.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142c956700 1 -- 172.21.15.32:0/344953282 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14280d1f70 con 0x7f142810da00 2021-04-09T18:25:18.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.339 7f142c956700 1 --2- 172.21.15.32:0/344953282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f142810da00 0x7f142810de00 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 rx=0x7f1420006e30 tx=0x7f1420009e60).ready entity=mon.0 client_cookie=8f47de78b1a056d8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:18.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.340 7f14277fe700 1 -- 172.21.15.32:0/344953282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1420006a50 con 0x7f142810da00 2021-04-09T18:25:18.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.340 7f14277fe700 1 -- 172.21.15.32:0/344953282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142000b300 con 0x7f142810da00 2021-04-09T18:25:18.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.340 7f14277fe700 1 -- 172.21.15.32:0/344953282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f142000b750 con 0x7f142810da00 2021-04-09T18:25:18.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.340 7f142e339700 1 -- 172.21.15.32:0/344953282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f142810da00 msgr2=0x7f142810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:18.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.340 7f142e339700 1 --2- 172.21.15.32:0/344953282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f142810da00 0x7f142810de00 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 rx=0x7f1420006e30 tx=0x7f1420009e60).stop 2021-04-09T18:25:18.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.340 7f142e339700 1 -- 172.21.15.32:0/344953282 shutdown_connections 2021-04-09T18:25:18.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.340 7f142e339700 1 --2- 172.21.15.32:0/344953282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f142810da00 0x7f142810de00 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.340 7f142e339700 1 --2- 172.21.15.32:0/344953282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14280d7ca0 0x7f14280d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.340 7f142e339700 1 -- 172.21.15.32:0/344953282 >> 172.21.15.32:0/344953282 conn(0x7f14280ce5f0 msgr2=0x7f14280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.341 7f142e339700 1 -- 172.21.15.32:0/344953282 shutdown_connections 2021-04-09T18:25:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.341 7f142e339700 1 -- 172.21.15.32:0/344953282 wait complete. 2021-04-09T18:25:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f142e339700 1 Processor -- start 2021-04-09T18:25:18.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f142e339700 1 -- start start 2021-04-09T18:25:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f142e339700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280a90c0 0x7f14281a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:18.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f142e339700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14281a76d0 0x7f14281a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:18.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f142e339700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f142817bce0 0x7f142817c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:18.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f142e339700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14281a7c60 con 0x7f14280a90c0 2021-04-09T18:25:18.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f142e339700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14281a8e20 con 0x7f142817bce0 2021-04-09T18:25:18.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f142e339700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142817c700 con 0x7f14281a76d0 2021-04-09T18:25:18.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f1427fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280a90c0 0x7f14281a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:18.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f1427fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280a90c0 0x7f14281a81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52104/0 (socket says 172.21.15.32:52104) 2021-04-09T18:25:18.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.361 7f1427fff700 1 -- 172.21.15.32:0/4219751625 learned_addr learned my addr 172.21.15.32:0/4219751625 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:18.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f141ffff700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14281a76d0 0x7f14281a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f142c956700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f142817bce0 0x7f142817c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f1427fff700 1 -- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14281a76d0 msgr2=0x7f14281a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f1427fff700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14281a76d0 0x7f14281a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f1427fff700 1 -- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f142817bce0 msgr2=0x7f142817c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f1427fff700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f142817bce0 0x7f142817c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f1427fff700 1 -- 172.21.15.32:0/4219751625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142817c920 con 0x7f14280a90c0 2021-04-09T18:25:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f141ffff700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14281a76d0 0x7f14281a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f142c956700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f142817bce0 0x7f142817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f1427fff700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280a90c0 0x7f14281a81e0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7f1418006f00 tx=0x7f141800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f14257fa700 1 -- 172.21.15.32:0/4219751625 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1418006020 con 0x7f14280a90c0 2021-04-09T18:25:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f14257fa700 1 -- 172.21.15.32:0/4219751625 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14180063a0 con 0x7f14280a90c0 2021-04-09T18:25:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f14257fa700 1 -- 172.21.15.32:0/4219751625 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1418004620 con 0x7f14280a90c0 2021-04-09T18:25:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f142e339700 1 -- 172.21.15.32:0/4219751625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142817c920 con 0x7f14280a90c0 2021-04-09T18:25:18.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.362 7f142e339700 1 -- 172.21.15.32:0/4219751625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142817c920 con 0x7f14280a90c0 2021-04-09T18:25:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.363 7f14257fa700 1 -- 172.21.15.32:0/4219751625 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f141800a000 con 0x7f14280a90c0 2021-04-09T18:25:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.363 7f14257fa700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f140c0188c0 0x7f140c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.363 7f14257fa700 1 -- 172.21.15.32:0/4219751625 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f14180327a0 con 0x7f14280a90c0 2021-04-09T18:25:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.363 7f141ffff700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f140c0188c0 0x7f140c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:18.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.363 7f141ffff700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f140c0188c0 0x7f140c01adc0 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:18.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.383 7f142e339700 1 -- 172.21.15.32:0/4219751625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14281b7010 con 0x7f14280a90c0 2021-04-09T18:25:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.386 7f14257fa700 1 -- 172.21.15.32:0/4219751625 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f141802cfe0 con 0x7f14280a90c0 2021-04-09T18:25:18.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.575 7f142e339700 1 -- 172.21.15.32:0/4219751625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f142810d480 con 0x7f14280a90c0 2021-04-09T18:25:18.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.576 7f14257fa700 1 -- 172.21.15.32:0/4219751625 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v119) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f141802cfe0 con 0x7f14280a90c0 2021-04-09T18:25:18.644 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:18.645 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 119, 2021-04-09T18:25:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:25:18.645 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:25:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:24:55.972582", 2021-04-09T18:25:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:25:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:25:18.646 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:25:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:25:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:25:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:25:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:25:18.647 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:25:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:25:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:25:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:25:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:25:18.648 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:25:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:25:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:25:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 21, 2021-04-09T18:25:18.649 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:25:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:25:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:25:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:25:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:25:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:25:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:25:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:25:18.651 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:25:18.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:25:18.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:25:18.652 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:25:18.652 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:25:18.652 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:25:18.653 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:25:18.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:25:18.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:25:18.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:25:18.653 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:25:18.654 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:25:18.654 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:25:18.654 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:25:18.654 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:25:18.655 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:25:18.655 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:25:18.655 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:25:18.655 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:25:18.655 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:25:18.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.656 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:25:18.656 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:25:18.656 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:25:18.656 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:25:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:25:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:25:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:25:18.657 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:25:18.658 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:25:18.658 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:25:18.658 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:25:18.658 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:25:18.658 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:25:18.659 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:25:18.659 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:25:18.659 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:25:18.659 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:25:18.659 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:25:18.660 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:25:18.660 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:25:18.660 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:25:18.660 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:25:18.661 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:25:18.661 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:25:18.661 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:25:18.661 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:25:18.661 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:25:18.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.662 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:25:18.662 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:25:18.662 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:25:18.662 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:25:18.663 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:25:18.663 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:25:18.663 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:25:18.663 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:25:18.663 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:25:18.664 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:25:18.664 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:25:18.664 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:25:18.664 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:25:18.665 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:25:18.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.665 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 20, 2021-04-09T18:25:18.666 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:25:18.666 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:24:52.261572", 2021-04-09T18:25:18.666 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:25:18.666 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:25:18.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:25:18.667 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:25:18.667 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:25:18.667 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:25:18.667 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:25:18.667 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:25:18.668 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:25:18.668 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:25:18.668 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:25:18.668 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:25:18.669 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:25:18.669 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:25:18.669 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:25:18.669 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:25:18.669 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:25:18.670 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:25:18.670 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:25:18.670 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:25:18.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.670 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "118", 2021-04-09T18:25:18.671 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:25:18.671 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:25:18.671 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:25:18.671 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:25:18.672 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:25:18.672 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:25:18.672 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:25:18.672 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:25:18.672 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:25:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:25:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:25:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:25:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:25:18.673 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:25:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:25:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:25:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:25:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:25:18.674 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:25:18.675 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:25:18.675 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:25:18.675 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:25:18.675 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:25:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:25:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:25:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:25:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.676 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:25:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:25:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:25:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:25:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:25:18.677 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:25:18.678 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:25:18.678 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:25:18.678 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:25:18.678 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:25:18.678 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:25:18.679 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:25:18.679 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:25:18.679 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:25:18.679 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:25:18.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.680 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:25:18.680 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:25:18.680 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:25:18.680 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.681 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.681 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 21, 2021-04-09T18:25:18.681 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:25:18.682 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:24:53.683747", 2021-04-09T18:25:18.682 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:25:18.682 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:25:18.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:25:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:25:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:25:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:25:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:25:18.683 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:25:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:25:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:25:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:25:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:25:18.684 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:25:18.685 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:25:18.685 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:25:18.685 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:25:18.685 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:25:18.685 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:25:18.686 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:25:18.686 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:25:18.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.686 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "119", 2021-04-09T18:25:18.686 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:25:18.687 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:25:18.687 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:25:18.687 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:25:18.687 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:25:18.688 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:25:18.688 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:25:18.688 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:25:18.688 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:25:18.688 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:25:18.689 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:25:18.689 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:25:18.689 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:25:18.689 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:25:18.689 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:25:18.690 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:25:18.690 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:25:18.690 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:25:18.691 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:25:18.691 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:25:18.691 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:25:18.691 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:25:18.691 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:25:18.692 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:25:18.692 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:25:18.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:25:18.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.693 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:25:18.693 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:25:18.693 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:25:18.693 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:25:18.693 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:25:18.694 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:25:18.694 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:25:18.694 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:25:18.694 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:25:18.694 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:25:18.695 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:25:18.695 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:25:18.695 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:25:18.695 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:25:18.695 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:25:18.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.696 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:25:18.697 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:25:18.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.697 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:25:18.697 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:25:18.698 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:18.698 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:18.698 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:18.698 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:18.698 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:18.699 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:18.699 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:18.699 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 112, 2021-04-09T18:25:18.699 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:18.699 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:18.700 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:18.700 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.700 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.701 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:25:18.701 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:18.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.701 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.702 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:25:18.702 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:18.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.703 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:18.703 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.704 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:25:18.704 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:18.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.705 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:25:18.705 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:18.705 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.705 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.706 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:18.706 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.706 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.706 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:25:18.707 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:18.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.708 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:25:18.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:18.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.708 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.709 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:18.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:25:18.710 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:18.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.710 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:25:18.711 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:18.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.711 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.712 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:25:18.712 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:25:18.712 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:25:18.712 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:25:18.712 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:18.713 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:18.713 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:18.713 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:25:18.714 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:25:18.714 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:18.714 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:18.714 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:18.715 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:18.715 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:18.715 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:18.715 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:18.716 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:18.716 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:18.716 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:18.716 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:18.716 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.717 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.717 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:25:18.717 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:18.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.718 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:25:18.718 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:18.718 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.719 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.719 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:18.719 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.720 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.720 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:25:18.720 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:18.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.721 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:25:18.721 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:18.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.722 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.722 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:18.722 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.723 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:25:18.723 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:18.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.724 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.724 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:25:18.724 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:18.725 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.725 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.725 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:18.725 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.726 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.726 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:25:18.726 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:18.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:25:18.727 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:18.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:25:18.728 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:25:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:25:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:25:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:18.729 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:25:18.730 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:25:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:18.731 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:18.732 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.733 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:25:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:25:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:25:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:25:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.738 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.739 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:25:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:25:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.741 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:25:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:25:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.744 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:25:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:25:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:25:18.745 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:25:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:25:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:25:18.747 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:18.748 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:18.749 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:18.749 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:18.749 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:18.749 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:18.749 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:18.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.750 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:25:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.751 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:25:18.752 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.753 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:25:18.754 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.755 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:25:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.756 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:25:18.757 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:25:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.759 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:25:18.760 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:25:18.761 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.762 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:25:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:25:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:25:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:25:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:18.763 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:25:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:25:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:18.765 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:18.766 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:25:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:18.767 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:25:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:18.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:25:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:18.769 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:18.770 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:25:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:18.771 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:25:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:18.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:25:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:18.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:25:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:18.775 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:25:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:18.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:25:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:18.778 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:25:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:25:18.779 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:25:18.780 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:25:18.780 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:18.780 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:18.780 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:25:18.781 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:25:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:18.782 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:18.783 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.784 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:25:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.785 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:25:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.787 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:25:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.788 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:25:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.790 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:25:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.791 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:25:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.793 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:25:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.794 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:25:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:25:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:25:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:25:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:25:18.796 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:25:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:25:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:18.798 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:18.799 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:18.800 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:25:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:18.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:25:18.802 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:18.803 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:25:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:18.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:25:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:18.806 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:25:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:18.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:25:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:18.808 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:18.809 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:25:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:18.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:25:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:18.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:25:18.812 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:25:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:25:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:25:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:18.813 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:18.814 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:25:18.814 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:25:18.815 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:18.815 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:18.815 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:18.815 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:18.815 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:18.816 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:18.816 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:18.816 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:18.816 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:18.816 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:25:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:25:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.820 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:25:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:25:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.823 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:25:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:25:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:18.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:25:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:18.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:25:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.828 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:25:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:25:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:25:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:25:18.829 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.830 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:25:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:25:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:25:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:18.831 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:18.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:25:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:18.833 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.834 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:25:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:18.835 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:25:18.836 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:18.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:25:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:18.838 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:25:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:18.840 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:25:18.841 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:18.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:25:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:18.843 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.844 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:25:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:25:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:25:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:25:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:25:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:25:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:25:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:25:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:25:18.846 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:25:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:25:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:25:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:25:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:25:18.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:25:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:25:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:25:18.848 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:25:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:25:18.849 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 -- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f140c0188c0 msgr2=0x7f140c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f140c0188c0 0x7f140c01adc0 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 -- 172.21.15.32:0/4219751625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280a90c0 msgr2=0x7f14281a81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280a90c0 0x7f14281a81e0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7f1418006f00 tx=0x7f141800b940).stop 2021-04-09T18:25:18.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 -- 172.21.15.32:0/4219751625 shutdown_connections 2021-04-09T18:25:18.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f140c0188c0 0x7f140c01adc0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f142817bce0 0x7f142817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14281a76d0 0x7f14281a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 --2- 172.21.15.32:0/4219751625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14280a90c0 0x7f14281a81e0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 -- 172.21.15.32:0/4219751625 >> 172.21.15.32:0/4219751625 conn(0x7f14281acae0 msgr2=0x7f142817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 -- 172.21.15.32:0/4219751625 shutdown_connections 2021-04-09T18:25:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.581 7f142e339700 1 -- 172.21.15.32:0/4219751625 wait complete. 2021-04-09T18:25:18.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.939 7ff034a7a700 1 Processor -- start 2021-04-09T18:25:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.940 7ff034a7a700 1 -- start start 2021-04-09T18:25:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.940 7ff034a7a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff03010d010 0x7ff0300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.940 7ff034a7a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0300d1da0 0x7ff0300d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.940 7ff034a7a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03010d480 con 0x7ff0300d82c0 2021-04-09T18:25:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.940 7ff034a7a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0300d8d30 con 0x7ff0300d1da0 2021-04-09T18:25:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.940 7ff034a7a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0300d9220 con 0x7ff03010d010 2021-04-09T18:25:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02e05d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0300d1da0 0x7ff0300d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02f05f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0300d82c0 0x7ff03010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33472/0 (socket says 172.21.15.32:33472) 2021-04-09T18:25:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02f05f700 1 -- 172.21.15.32:0/73965125 learned_addr learned my addr 172.21.15.32:0/73965125 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02e85e700 1 --2- 172.21.15.32:0/73965125 >> v2:172.21.15.86:3300/0 conn(0x7ff03010d010 0x7ff0300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02e05d700 1 -- 172.21.15.32:0/73965125 >> v2:172.21.15.86:3300/0 conn(0x7ff03010d010 msgr2=0x7ff0300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02e05d700 1 --2- 172.21.15.32:0/73965125 >> v2:172.21.15.86:3300/0 conn(0x7ff03010d010 0x7ff0300d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02e05d700 1 -- 172.21.15.32:0/73965125 >> v1:172.21.15.32:6789/0 conn(0x7ff0300d82c0 legacy=0x7ff03010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02e05d700 1 -- 172.21.15.32:0/73965125 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0300d86f0 con 0x7ff0300d1da0 2021-04-09T18:25:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02d85c700 1 -- 172.21.15.32:0/73965125 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704114196 0 0) 0x7ff028000d30 con 0x7ff0300d82c0 2021-04-09T18:25:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02e05d700 1 --2- 172.21.15.32:0/73965125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300d1da0 0x7ff0300d7ca0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 rx=0x7ff024006650 tx=0x7ff024009a00).ready entity=mon.0 client_cookie=6f0e6c167b7c7c2a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:19.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02d85c700 1 -- 172.21.15.32:0/73965125 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff024009df0 con 0x7ff0300d1da0 2021-04-09T18:25:19.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02d85c700 1 -- 172.21.15.32:0/73965125 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0240009b0 con 0x7ff0300d1da0 2021-04-09T18:25:19.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff02d85c700 1 -- 172.21.15.32:0/73965125 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff02400c2b0 con 0x7ff0300d1da0 2021-04-09T18:25:19.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff034a7a700 1 -- 172.21.15.32:0/73965125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300d1da0 msgr2=0x7ff0300d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.941 7ff034a7a700 1 --2- 172.21.15.32:0/73965125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300d1da0 0x7ff0300d7ca0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 rx=0x7ff024006650 tx=0x7ff024009a00).stop 2021-04-09T18:25:19.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.942 7ff034a7a700 1 -- 172.21.15.32:0/73965125 shutdown_connections 2021-04-09T18:25:19.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.942 7ff034a7a700 1 --2- 172.21.15.32:0/73965125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300d1da0 0x7ff0300d7ca0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.942 7ff034a7a700 1 --2- 172.21.15.32:0/73965125 >> v2:172.21.15.86:3300/0 conn(0x7ff03010d010 0x7ff0300d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.942 7ff034a7a700 1 -- 172.21.15.32:0/73965125 >> 172.21.15.32:0/73965125 conn(0x7ff0300ce5f0 msgr2=0x7ff0300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:19.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.942 7ff034a7a700 1 -- 172.21.15.32:0/73965125 shutdown_connections 2021-04-09T18:25:19.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.942 7ff034a7a700 1 -- 172.21.15.32:0/73965125 wait complete. 2021-04-09T18:25:19.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.962 7ff034a7a700 1 Processor -- start 2021-04-09T18:25:19.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.962 7ff034a7a700 1 -- start start 2021-04-09T18:25:19.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff034a7a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0301a5210 0x7ff0301a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff034a7a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0301a6340 0x7ff0301a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff034a7a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300abaa0 0x7ff0300abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff034a7a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0301a6d60 con 0x7ff0300abaa0 2021-04-09T18:25:19.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff034a7a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300ac4c0 con 0x7ff0301a6340 2021-04-09T18:25:19.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff034a7a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300ac6e0 con 0x7ff0301a5210 2021-04-09T18:25:19.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02f05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300abaa0 0x7ff0300abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02f05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300abaa0 0x7ff0300abea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52122/0 (socket says 172.21.15.32:52122) 2021-04-09T18:25:19.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02f05f700 1 -- 172.21.15.32:0/1108510959 learned_addr learned my addr 172.21.15.32:0/1108510959 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:19.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02e05d700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0301a6340 0x7ff0301a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02e85e700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0301a5210 0x7ff0301a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02f05f700 1 -- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0301a5210 msgr2=0x7ff0301a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02f05f700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0301a5210 0x7ff0301a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02f05f700 1 -- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0301a6340 msgr2=0x7ff0301a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02f05f700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0301a6340 0x7ff0301a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.963 7ff02f05f700 1 -- 172.21.15.32:0/1108510959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03017a720 con 0x7ff0300abaa0 2021-04-09T18:25:19.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.964 7ff02f05f700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300abaa0 0x7ff0300abea0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7ff028007f90 tx=0x7ff02800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:19.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.964 7ff01f7fe700 1 -- 172.21.15.32:0/1108510959 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff028000cc0 con 0x7ff0300abaa0 2021-04-09T18:25:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.964 7ff01f7fe700 1 -- 172.21.15.32:0/1108510959 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff028007020 con 0x7ff0300abaa0 2021-04-09T18:25:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.964 7ff01f7fe700 1 -- 172.21.15.32:0/1108510959 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff028007500 con 0x7ff0300abaa0 2021-04-09T18:25:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.964 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03017a720 con 0x7ff0300abaa0 2021-04-09T18:25:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.964 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03017a720 con 0x7ff0300abaa0 2021-04-09T18:25:19.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.964 7ff01f7fe700 1 -- 172.21.15.32:0/1108510959 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff028007200 con 0x7ff0300abaa0 2021-04-09T18:25:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.965 7ff01f7fe700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0140188d0 0x7ff01401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.965 7ff01f7fe700 1 -- 172.21.15.32:0/1108510959 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff028032d70 con 0x7ff0300abaa0 2021-04-09T18:25:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.965 7ff02e85e700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0140188d0 0x7ff01401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.965 7ff02e85e700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0140188d0 0x7ff01401add0 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:19.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.984 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0301a6d60 con 0x7ff0300abaa0 2021-04-09T18:25:19.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:18.988 7ff01f7fe700 1 -- 172.21.15.32:0/1108510959 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff028007d00 con 0x7ff0300abaa0 2021-04-09T18:25:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.179 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0301a6d60 con 0x7ff0300abaa0 2021-04-09T18:25:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.179 7ff01f7fe700 1 -- 172.21.15.32:0/1108510959 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+2463 (secure 0 0 0) 0x7ff028007000 con 0x7ff0300abaa0 2021-04-09T18:25:19.246 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 260 2021-04-09T18:25:19.246 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:19.247 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":260,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:15.457126","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8740,"mds_1":8749},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8740":{"gid":8740,"name":"a","rank":0,"incarnation":250,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2311632030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2311632030},{"type":"v1","addr":"172.21.15.32:6835","nonce":2311632030}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_8749":{"gid":8749,"name":"d","rank":1,"incarnation":255,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2147753572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2147753572},{"type":"v1","addr":"172.21.15.86:6836","nonce":2147753572}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0140188d0 msgr2=0x7ff01401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0140188d0 0x7ff01401add0 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300abaa0 msgr2=0x7ff0300abea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300abaa0 0x7ff0300abea0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7ff028007f90 tx=0x7ff02800ac50).stop 2021-04-09T18:25:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 shutdown_connections 2021-04-09T18:25:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0140188d0 0x7ff01401add0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0300abaa0 0x7ff0300abea0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0301a6340 0x7ff0301a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 --2- 172.21.15.32:0/1108510959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0301a5210 0x7ff0301a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 >> 172.21.15.32:0/1108510959 conn(0x7ff0301a99c0 msgr2=0x7ff03017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 shutdown_connections 2021-04-09T18:25:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.182 7ff034a7a700 1 -- 172.21.15.32:0/1108510959 wait complete. 2021-04-09T18:25:19.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:25:19.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.345 7faa1f144700 1 Processor -- start 2021-04-09T18:25:19.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.345 7faa1f144700 1 -- start start 2021-04-09T18:25:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa1f144700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faa180cffb0 0x7faa180d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa1f144700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa180d7810 0x7faa180d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa1f144700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa180d7290 con 0x7faa180d8230 2021-04-09T18:25:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa1f144700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa180d91f0 con 0x7faa180cffb0 2021-04-09T18:25:19.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa1f144700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1810dff0 con 0x7faa180d7810 2021-04-09T18:25:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa1d761700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faa180d8230 0x7faa180d5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33484/0 (socket says 172.21.15.32:33484) 2021-04-09T18:25:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa1d761700 1 -- 172.21.15.32:0/1473591393 learned_addr learned my addr 172.21.15.32:0/1473591393 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa17fff700 1 --2- 172.21.15.32:0/1473591393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa180d7810 0x7faa180d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa1cf60700 1 --2- 172.21.15.32:0/1473591393 >> v2:172.21.15.86:3300/0 conn(0x7faa180cffb0 0x7faa180d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa177fe700 1 -- 172.21.15.32:0/1473591393 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905942412 0 0) 0x7faa100009b0 con 0x7faa180d8230 2021-04-09T18:25:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.346 7faa177fe700 1 -- 172.21.15.32:0/1473591393 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa04002fe0 con 0x7faa180d8230 2021-04-09T18:25:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa17fff700 1 -- 172.21.15.32:0/1473591393 >> v2:172.21.15.86:3300/0 conn(0x7faa180cffb0 msgr2=0x7faa180d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa17fff700 1 --2- 172.21.15.32:0/1473591393 >> v2:172.21.15.86:3300/0 conn(0x7faa180cffb0 0x7faa180d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa17fff700 1 -- 172.21.15.32:0/1473591393 >> v1:172.21.15.32:6789/0 conn(0x7faa180d8230 legacy=0x7faa180d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa17fff700 1 -- 172.21.15.32:0/1473591393 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1810e210 con 0x7faa180d7810 2021-04-09T18:25:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa1cf60700 1 --2- 172.21.15.32:0/1473591393 >> v2:172.21.15.86:3300/0 conn(0x7faa180cffb0 0x7faa180d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa17fff700 1 --2- 172.21.15.32:0/1473591393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa180d7810 0x7faa180d7c10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7faa080069d0 tx=0x7faa08009e40).ready entity=mon.2 client_cookie=3a3bfe840895dd30 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa177fe700 1 -- 172.21.15.32:0/1473591393 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa0800b510 con 0x7faa180d7810 2021-04-09T18:25:19.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa177fe700 1 -- 172.21.15.32:0/1473591393 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0800b890 con 0x7faa180d7810 2021-04-09T18:25:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa177fe700 1 -- 172.21.15.32:0/1473591393 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa0800bd70 con 0x7faa180d7810 2021-04-09T18:25:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa1f144700 1 -- 172.21.15.32:0/1473591393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa180d7810 msgr2=0x7faa180d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.347 7faa1f144700 1 --2- 172.21.15.32:0/1473591393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa180d7810 0x7faa180d7c10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7faa080069d0 tx=0x7faa08009e40).stop 2021-04-09T18:25:19.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.348 7faa1f144700 1 -- 172.21.15.32:0/1473591393 shutdown_connections 2021-04-09T18:25:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.348 7faa1f144700 1 --2- 172.21.15.32:0/1473591393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa180d7810 0x7faa180d7c10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.348 7faa1f144700 1 --2- 172.21.15.32:0/1473591393 >> v2:172.21.15.86:3300/0 conn(0x7faa180cffb0 0x7faa180d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.348 7faa1f144700 1 -- 172.21.15.32:0/1473591393 >> 172.21.15.32:0/1473591393 conn(0x7faa180ce5d0 msgr2=0x7faa180cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.348 7faa1f144700 1 -- 172.21.15.32:0/1473591393 shutdown_connections 2021-04-09T18:25:19.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.348 7faa1f144700 1 -- 172.21.15.32:0/1473591393 wait complete. 2021-04-09T18:25:19.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.368 7faa1f144700 1 Processor -- start 2021-04-09T18:25:19.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.368 7faa1f144700 1 -- start start 2021-04-09T18:25:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1f144700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1818e880 0x7faa181812c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1f144700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa18180340 0x7faa181b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1f144700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa181b8c90 0x7faa180adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1f144700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa18180d40 con 0x7faa181b8c90 2021-04-09T18:25:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1f144700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa181894f0 con 0x7faa18180340 2021-04-09T18:25:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1f144700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1818ec80 con 0x7faa1818e880 2021-04-09T18:25:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1d761700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa181b8c90 0x7faa180adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1d761700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa181b8c90 0x7faa180adf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52136/0 (socket says 172.21.15.32:52136) 2021-04-09T18:25:19.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1d761700 1 -- 172.21.15.32:0/3678893590 learned_addr learned my addr 172.21.15.32:0/3678893590 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa17fff700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa18180340 0x7faa181b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1cf60700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1818e880 0x7faa181812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1d761700 1 -- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1818e880 msgr2=0x7faa181812c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1d761700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1818e880 0x7faa181812c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1d761700 1 -- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa18180340 msgr2=0x7faa181b8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:19.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1d761700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa18180340 0x7faa181b8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:19.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1d761700 1 -- 172.21.15.32:0/3678893590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa181818e0 con 0x7faa181b8c90 2021-04-09T18:25:19.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1d761700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa181b8c90 0x7faa180adf80 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 rx=0x7faa1000a520 tx=0x7faa1000a8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:19.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa157fa700 1 -- 172.21.15.32:0/3678893590 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa1000acb0 con 0x7faa181b8c90 2021-04-09T18:25:19.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa157fa700 1 -- 172.21.15.32:0/3678893590 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa10007030 con 0x7faa181b8c90 2021-04-09T18:25:19.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1f144700 1 -- 172.21.15.32:0/3678893590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa181818e0 con 0x7faa181b8c90 2021-04-09T18:25:19.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa157fa700 1 -- 172.21.15.32:0/3678893590 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa10004590 con 0x7faa181b8c90 2021-04-09T18:25:19.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.369 7faa1f144700 1 -- 172.21.15.32:0/3678893590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa181818e0 con 0x7faa181b8c90 2021-04-09T18:25:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.370 7faa157fa700 1 -- 172.21.15.32:0/3678893590 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faa100043a0 con 0x7faa181b8c90 2021-04-09T18:25:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.370 7faa157fa700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa9fc018930 0x7fa9fc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.370 7faa157fa700 1 -- 172.21.15.32:0/3678893590 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5577+0+0 (secure 0 0 0) 0x7faa10032a70 con 0x7faa181b8c90 2021-04-09T18:25:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.370 7faa1cf60700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa9fc018930 0x7fa9fc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:19.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.371 7faa1cf60700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa9fc018930 0x7fa9fc01adc0 crc :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:19.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.390 7faa1f144700 1 -- 172.21.15.32:0/3678893590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1818ec80 con 0x7faa181b8c90 2021-04-09T18:25:19.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.393 7faa157fa700 1 -- 172.21.15.32:0/3678893590 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faa10000c10 con 0x7faa181b8c90 2021-04-09T18:25:19.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:19.577 7faa1f144700 1 -- 172.21.15.32:0/3678893590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7faa1818ec80 con 0x7faa181b8c90 2021-04-09T18:25:20.536 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:25:20.470 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:25:20.538 INFO:tasks.ceph.mds.a.smithi032.stderr: -1839> 2021-04-09 18:24:49.495 7f09c81f2700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2021-04-09T18:25:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.472 7faa157fa700 1 -- 172.21.15.32:0/3678893590 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v261) v1 ==== 161+0+0 (secure 0 0 0) 0x7faa100043a0 con 0x7faa181b8c90 2021-04-09T18:25:20.559 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:25:20.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.495 7faa1f144700 1 -- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa9fc018930 msgr2=0x7fa9fc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:20.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.495 7faa1f144700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa9fc018930 0x7fa9fc01adc0 crc :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.495 7faa1f144700 1 -- 172.21.15.32:0/3678893590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa181b8c90 msgr2=0x7faa180adf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:20.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.495 7faa1f144700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa181b8c90 0x7faa180adf80 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 rx=0x7faa1000a520 tx=0x7faa1000a8c0).stop 2021-04-09T18:25:20.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.496 7faa1f144700 1 -- 172.21.15.32:0/3678893590 shutdown_connections 2021-04-09T18:25:20.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.496 7faa1f144700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa9fc018930 0x7fa9fc01adc0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.496 7faa1f144700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa181b8c90 0x7faa180adf80 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.496 7faa1f144700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa18180340 0x7faa181b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.496 7faa1f144700 1 --2- 172.21.15.32:0/3678893590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1818e880 0x7faa181812c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.496 7faa1f144700 1 -- 172.21.15.32:0/3678893590 >> 172.21.15.32:0/3678893590 conn(0x7faa18189800 msgr2=0x7faa181b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:20.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.496 7faa1f144700 1 -- 172.21.15.32:0/3678893590 shutdown_connections 2021-04-09T18:25:20.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.496 7faa1f144700 1 -- 172.21.15.32:0/3678893590 wait complete. 2021-04-09T18:25:20.572 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:20.593 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:25:20.599 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:25:20.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.656 7feddd90a700 1 Processor -- start 2021-04-09T18:25:20.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.656 7feddd90a700 1 -- start start 2021-04-09T18:25:20.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.657 7feddd90a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedd80d7ca0 0x7fedd80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:20.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.657 7feddd90a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fedd810da00 0x7fedd810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:20.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.657 7feddd90a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd810d480 con 0x7fedd810d010 2021-04-09T18:25:20.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.657 7feddd90a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd80d9220 con 0x7fedd80d7ca0 2021-04-09T18:25:20.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.657 7feddd90a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd80d86c0 con 0x7fedd810da00 2021-04-09T18:25:20.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.657 7fedd6d73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fedd80d7ca0 0x7fedd80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:20.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.657 7fedd7574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fedd810d010 0x7fedd80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33510/0 (socket says 172.21.15.32:33510) 2021-04-09T18:25:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.657 7fedd7574700 1 -- 172.21.15.32:0/380791560 learned_addr learned my addr 172.21.15.32:0/380791560 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.657 7fedd7d75700 1 --2- 172.21.15.32:0/380791560 >> v2:172.21.15.86:3300/0 conn(0x7fedd810da00 0x7fedd810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd6d73700 1 -- 172.21.15.32:0/380791560 >> v2:172.21.15.86:3300/0 conn(0x7fedd810da00 msgr2=0x7fedd810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:20.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd6d73700 1 --2- 172.21.15.32:0/380791560 >> v2:172.21.15.86:3300/0 conn(0x7fedd810da00 0x7fedd810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd6d73700 1 -- 172.21.15.32:0/380791560 >> v1:172.21.15.32:6789/0 conn(0x7fedd810d010 legacy=0x7fedd80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd6d73700 1 -- 172.21.15.32:0/380791560 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80d1f70 con 0x7fedd80d7ca0 2021-04-09T18:25:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd7d75700 1 --2- 172.21.15.32:0/380791560 >> v2:172.21.15.86:3300/0 conn(0x7fedd810da00 0x7fedd810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd6572700 1 -- 172.21.15.32:0/380791560 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228947299 0 0) 0x7fedc8000ce0 con 0x7fedd810d010 2021-04-09T18:25:20.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd6d73700 1 --2- 172.21.15.32:0/380791560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd80d7ca0 0x7fedd80d80a0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 rx=0x7fedcc006650 tx=0x7fedcc009a80).ready entity=mon.0 client_cookie=c9d1724684087b62 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:20.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd6572700 1 -- 172.21.15.32:0/380791560 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedcc0009b0 con 0x7fedd80d7ca0 2021-04-09T18:25:20.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd6572700 1 -- 172.21.15.32:0/380791560 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc00c0c0 con 0x7fedd80d7ca0 2021-04-09T18:25:20.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7fedd6572700 1 -- 172.21.15.32:0/380791560 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedcc00c510 con 0x7fedd80d7ca0 2021-04-09T18:25:20.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7feddd90a700 1 -- 172.21.15.32:0/380791560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd80d7ca0 msgr2=0x7fedd80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:20.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.658 7feddd90a700 1 --2- 172.21.15.32:0/380791560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd80d7ca0 0x7fedd80d80a0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 rx=0x7fedcc006650 tx=0x7fedcc009a80).stop 2021-04-09T18:25:20.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.659 7feddd90a700 1 -- 172.21.15.32:0/380791560 shutdown_connections 2021-04-09T18:25:20.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.659 7feddd90a700 1 --2- 172.21.15.32:0/380791560 >> v2:172.21.15.86:3300/0 conn(0x7fedd810da00 0x7fedd810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.659 7feddd90a700 1 --2- 172.21.15.32:0/380791560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd80d7ca0 0x7fedd80d80a0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.659 7feddd90a700 1 -- 172.21.15.32:0/380791560 >> 172.21.15.32:0/380791560 conn(0x7fedd80ce5f0 msgr2=0x7fedd80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:20.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.659 7feddd90a700 1 -- 172.21.15.32:0/380791560 shutdown_connections 2021-04-09T18:25:20.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.659 7feddd90a700 1 -- 172.21.15.32:0/380791560 wait complete. 2021-04-09T18:25:20.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.679 7feddd90a700 1 Processor -- start 2021-04-09T18:25:20.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.679 7feddd90a700 1 -- start start 2021-04-09T18:25:20.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7feddd90a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd81ac380 0x7fedd818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:20.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7feddd90a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd81ac780 0x7fedd818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:20.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7feddd90a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8183520 0x7fedd8183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:20.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7feddd90a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81ace10 con 0x7fedd8183520 2021-04-09T18:25:20.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7feddd90a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8189cf0 con 0x7fedd81ac380 2021-04-09T18:25:20.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7feddd90a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8183f40 con 0x7fedd81ac780 2021-04-09T18:25:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7d75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8183520 0x7fedd8183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7d75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8183520 0x7fedd8183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52162/0 (socket says 172.21.15.32:52162) 2021-04-09T18:25:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7d75700 1 -- 172.21.15.32:0/1250912296 learned_addr learned my addr 172.21.15.32:0/1250912296 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7574700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd81ac380 0x7fedd818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd6d73700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd81ac780 0x7fedd818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:20.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7d75700 1 -- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd81ac780 msgr2=0x7fedd818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:20.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7d75700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd81ac780 0x7fedd818a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7d75700 1 -- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd81ac380 msgr2=0x7fedd818a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:20.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7d75700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd81ac380 0x7fedd818a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7d75700 1 -- 172.21.15.32:0/1250912296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd8184160 con 0x7fedd8183520 2021-04-09T18:25:20.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedd7d75700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8183520 0x7fedd8183920 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7fedd000b970 tx=0x7fedd000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedbffff700 1 -- 172.21.15.32:0/1250912296 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedd0000f90 con 0x7fedd8183520 2021-04-09T18:25:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedbffff700 1 -- 172.21.15.32:0/1250912296 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd0006aa0 con 0x7fedd8183520 2021-04-09T18:25:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7feddd90a700 1 -- 172.21.15.32:0/1250912296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd8184160 con 0x7fedd8183520 2021-04-09T18:25:20.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7fedbffff700 1 -- 172.21.15.32:0/1250912296 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedd00015f0 con 0x7fedd8183520 2021-04-09T18:25:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.680 7feddd90a700 1 -- 172.21.15.32:0/1250912296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd8184160 con 0x7fedd8183520 2021-04-09T18:25:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.681 7fedbffff700 1 -- 172.21.15.32:0/1250912296 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fedd0000e70 con 0x7fedd8183520 2021-04-09T18:25:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.681 7fedbffff700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedb8018930 0x7fedb801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.681 7fedbffff700 1 -- 172.21.15.32:0/1250912296 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fedd0032e20 con 0x7fedd8183520 2021-04-09T18:25:20.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.681 7fedd7574700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedb8018930 0x7fedb801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:20.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.682 7fedd7574700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedb8018930 0x7fedb801ae10 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:20.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.701 7feddd90a700 1 -- 172.21.15.32:0/1250912296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd8189cf0 con 0x7fedd8183520 2021-04-09T18:25:20.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.705 7fedbffff700 1 -- 172.21.15.32:0/1250912296 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fedd00066f0 con 0x7fedd8183520 2021-04-09T18:25:20.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.896 7feddd90a700 1 -- 172.21.15.32:0/1250912296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedd8189cf0 con 0x7fedd8183520 2021-04-09T18:25:20.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.896 7fedbffff700 1 -- 172.21.15.32:0/1250912296 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fedd0006aa0 con 0x7fedd8183520 2021-04-09T18:25:20.963 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 261 2021-04-09T18:25:20.964 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:20.964 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":261,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":57506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1430818598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1430818598},{"type":"v1","addr":"172.21.15.86:6837","nonce":1430818598}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":261}],"filesystems":[{"mdsmap":{"epoch":261,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:24:54.971184","modified":"2021-04-09 18:25:20.471084","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[2],"info":{},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-04-09T18:25:20.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 -- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedb8018930 msgr2=0x7fedb801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:20.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedb8018930 0x7fedb801ae10 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 -- 172.21.15.32:0/1250912296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8183520 msgr2=0x7fedd8183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:20.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8183520 0x7fedd8183920 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7fedd000b970 tx=0x7fedd000bd10).stop 2021-04-09T18:25:20.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 -- 172.21.15.32:0/1250912296 shutdown_connections 2021-04-09T18:25:20.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fedb8018930 0x7fedb801ae10 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedd8183520 0x7fedd8183920 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd81ac780 0x7fedd818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 --2- 172.21.15.32:0/1250912296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd81ac380 0x7fedd818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:20.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 -- 172.21.15.32:0/1250912296 >> 172.21.15.32:0/1250912296 conn(0x7fedd819b890 msgr2=0x7fedd81a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:20.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 -- 172.21.15.32:0/1250912296 shutdown_connections 2021-04-09T18:25:20.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:20.899 7feddd90a700 1 -- 172.21.15.32:0/1250912296 wait complete. 2021-04-09T18:25:20.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:25:21.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.101 7fc115fc1700 1 Processor -- start 2021-04-09T18:25:21.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.101 7fc115fc1700 1 -- start start 2021-04-09T18:25:21.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc115fc1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100cffb0 0x7fc1100d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc115fc1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc1100d6c90 0x7fc11010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc115fc1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1100d7290 con 0x7fc1100d7810 2021-04-09T18:25:21.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc115fc1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100d7090 con 0x7fc1100d6c90 2021-04-09T18:25:21.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc115fc1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1100d8590 con 0x7fc1100cffb0 2021-04-09T18:25:21.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10effd700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc1100d7810 0x7fc1100d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:59686/0 (socket says 172.21.15.32:59686) 2021-04-09T18:25:21.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10f7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100cffb0 0x7fc1100d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10effd700 1 -- 172.21.15.32:0/1377946238 learned_addr learned my addr 172.21.15.32:0/1377946238 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:21.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10ffff700 1 --2- 172.21.15.32:0/1377946238 >> v2:172.21.15.86:3300/0 conn(0x7fc1100d6c90 0x7fc11010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10e7fc700 1 -- 172.21.15.32:0/1377946238 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315058149 0 0) 0x7fc1040009b0 con 0x7fc1100d7810 2021-04-09T18:25:21.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10e7fc700 1 -- 172.21.15.32:0/1377946238 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f8002fe0 con 0x7fc1100d7810 2021-04-09T18:25:21.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10f7fe700 1 -- 172.21.15.32:0/1377946238 >> v2:172.21.15.86:3300/0 conn(0x7fc1100d6c90 msgr2=0x7fc11010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:21.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10f7fe700 1 --2- 172.21.15.32:0/1377946238 >> v2:172.21.15.86:3300/0 conn(0x7fc1100d6c90 0x7fc11010cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10f7fe700 1 -- 172.21.15.32:0/1377946238 >> v1:172.21.15.86:6789/0 conn(0x7fc1100d7810 legacy=0x7fc1100d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10f7fe700 1 -- 172.21.15.32:0/1377946238 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1100d91f0 con 0x7fc1100cffb0 2021-04-09T18:25:21.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.102 7fc10ffff700 1 --2- 172.21.15.32:0/1377946238 >> v2:172.21.15.86:3300/0 conn(0x7fc1100d6c90 0x7fc11010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:21.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.103 7fc10f7fe700 1 --2- 172.21.15.32:0/1377946238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100cffb0 0x7fc1100d2450 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc100006650 tx=0x7fc100009a80).ready entity=mon.2 client_cookie=6329b05cff4e3fd8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:21.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.103 7fc10e7fc700 1 -- 172.21.15.32:0/1377946238 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1000009b0 con 0x7fc1100cffb0 2021-04-09T18:25:21.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.103 7fc10e7fc700 1 -- 172.21.15.32:0/1377946238 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10000c0c0 con 0x7fc1100cffb0 2021-04-09T18:25:21.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.103 7fc115fc1700 1 -- 172.21.15.32:0/1377946238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100cffb0 msgr2=0x7fc1100d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.103 7fc10e7fc700 1 -- 172.21.15.32:0/1377946238 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc10000c510 con 0x7fc1100cffb0 2021-04-09T18:25:21.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.103 7fc115fc1700 1 --2- 172.21.15.32:0/1377946238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100cffb0 0x7fc1100d2450 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc100006650 tx=0x7fc100009a80).stop 2021-04-09T18:25:21.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.104 7fc115fc1700 1 -- 172.21.15.32:0/1377946238 shutdown_connections 2021-04-09T18:25:21.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.104 7fc115fc1700 1 --2- 172.21.15.32:0/1377946238 >> v2:172.21.15.86:3300/0 conn(0x7fc1100d6c90 0x7fc11010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.104 7fc115fc1700 1 --2- 172.21.15.32:0/1377946238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100cffb0 0x7fc1100d2450 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.104 7fc115fc1700 1 -- 172.21.15.32:0/1377946238 >> 172.21.15.32:0/1377946238 conn(0x7fc1100ce5d0 msgr2=0x7fc1100cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:21.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.104 7fc115fc1700 1 -- 172.21.15.32:0/1377946238 shutdown_connections 2021-04-09T18:25:21.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.104 7fc115fc1700 1 -- 172.21.15.32:0/1377946238 wait complete. 2021-04-09T18:25:21.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.124 7fc115fc1700 1 Processor -- start 2021-04-09T18:25:21.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.124 7fc115fc1700 1 -- start start 2021-04-09T18:25:21.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.124 7fc115fc1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100a80a0 0x7fc11017cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.124 7fc115fc1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc11017d0e0 0x7fc11008c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.124 7fc115fc1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc11008ce00 0x7fc11017afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.124 7fc115fc1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11008c2e0 con 0x7fc11008ce00 2021-04-09T18:25:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.124 7fc115fc1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11008d200 con 0x7fc11017d0e0 2021-04-09T18:25:21.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.124 7fc115fc1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11018f280 con 0x7fc1100a80a0 2021-04-09T18:25:21.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc11008ce00 0x7fc11017afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10effd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc11017d0e0 0x7fc11008c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc11008ce00 0x7fc11017afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52176/0 (socket says 172.21.15.32:52176) 2021-04-09T18:25:21.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10ffff700 1 -- 172.21.15.32:0/3956065779 learned_addr learned my addr 172.21.15.32:0/3956065779 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:21.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10f7fe700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100a80a0 0x7fc11017cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10ffff700 1 -- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100a80a0 msgr2=0x7fc11017cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10ffff700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100a80a0 0x7fc11017cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10ffff700 1 -- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc11017d0e0 msgr2=0x7fc11008c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10ffff700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc11017d0e0 0x7fc11008c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10ffff700 1 -- 172.21.15.32:0/3956065779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11017b600 con 0x7fc11008ce00 2021-04-09T18:25:21.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10effd700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc11017d0e0 0x7fc11008c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:21.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10f7fe700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100a80a0 0x7fc11017cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:21.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc10ffff700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc11008ce00 0x7fc11017afe0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 rx=0x7fc10800b970 tx=0x7fc10800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:21.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc0f7fff700 1 -- 172.21.15.32:0/3956065779 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc108000f90 con 0x7fc11008ce00 2021-04-09T18:25:21.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc0f7fff700 1 -- 172.21.15.32:0/3956065779 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc108003030 con 0x7fc11008ce00 2021-04-09T18:25:21.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11017b600 con 0x7fc11008ce00 2021-04-09T18:25:21.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc0f7fff700 1 -- 172.21.15.32:0/3956065779 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc108001400 con 0x7fc11008ce00 2021-04-09T18:25:21.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.125 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11017b600 con 0x7fc11008ce00 2021-04-09T18:25:21.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.126 7fc0f7fff700 1 -- 172.21.15.32:0/3956065779 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc108000e70 con 0x7fc11008ce00 2021-04-09T18:25:21.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.126 7fc0f7fff700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f0018930 0x7fc0f001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.126 7fc0f7fff700 1 -- 172.21.15.32:0/3956065779 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fc108032e10 con 0x7fc11008ce00 2021-04-09T18:25:21.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.126 7fc10f7fe700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f0018930 0x7fc0f001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.126 7fc10f7fe700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f0018930 0x7fc0f001ae10 crc :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:21.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.146 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11008d200 con 0x7fc11008ce00 2021-04-09T18:25:21.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.149 7fc0f7fff700 1 -- 172.21.15.32:0/3956065779 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc1080066f0 con 0x7fc11008ce00 2021-04-09T18:25:21.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.345 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc11008d200 con 0x7fc11008ce00 2021-04-09T18:25:21.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.585 7fc0f7fff700 1 -- 172.21.15.32:0/3956065779 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v262) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc108000e70 con 0x7fc11008ce00 2021-04-09T18:25:21.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f0018930 msgr2=0x7fc0f001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f0018930 0x7fc0f001ae10 crc :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc11008ce00 msgr2=0x7fc11017afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc11008ce00 0x7fc11017afe0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 rx=0x7fc10800b970 tx=0x7fc10800bd10).stop 2021-04-09T18:25:21.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 shutdown_connections 2021-04-09T18:25:21.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f0018930 0x7fc0f001ae10 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc11008ce00 0x7fc11017afe0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc11017d0e0 0x7fc11008c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 --2- 172.21.15.32:0/3956065779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100a80a0 0x7fc11017cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 >> 172.21.15.32:0/3956065779 conn(0x7fc1101905e0 msgr2=0x7fc1101b7350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:21.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 shutdown_connections 2021-04-09T18:25:21.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.612 7fc115fc1700 1 -- 172.21.15.32:0/3956065779 wait complete. 2021-04-09T18:25:21.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:25:21.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.783 7f8dcdb4a700 1 Processor -- start 2021-04-09T18:25:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.783 7f8dcdb4a700 1 -- start start 2021-04-09T18:25:21.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.784 7f8dcdb4a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8dc80d7cc0 0x7f8dc80d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.784 7f8dcdb4a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8dc810da80 0x7f8dc810de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.784 7f8dcdb4a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc810d500 con 0x7f8dc810d050 2021-04-09T18:25:21.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.784 7f8dcdb4a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc80d9260 con 0x7f8dc810da80 2021-04-09T18:25:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.784 7f8dcdb4a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc80d0330 con 0x7f8dc80d7cc0 2021-04-09T18:25:21.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.784 7f8dc7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8dc810da80 0x7f8dc810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.784 7f8dc7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8dc810da80 0x7f8dc810de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52180/0 (socket says 172.21.15.32:52180) 2021-04-09T18:25:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.784 7f8dc7fff700 1 -- 172.21.15.32:0/2702037722 learned_addr learned my addr 172.21.15.32:0/2702037722 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:21.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.784 7f8dc6ffd700 1 --2- 172.21.15.32:0/2702037722 >> v2:172.21.15.86:3300/0 conn(0x7f8dc80d7cc0 0x7f8dc80d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dc7fff700 1 -- 172.21.15.32:0/2702037722 >> v2:172.21.15.86:3300/0 conn(0x7f8dc80d7cc0 msgr2=0x7f8dc80d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dc7fff700 1 --2- 172.21.15.32:0/2702037722 >> v2:172.21.15.86:3300/0 conn(0x7f8dc80d7cc0 0x7f8dc80d80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dc7fff700 1 -- 172.21.15.32:0/2702037722 >> v1:172.21.15.86:6789/0 conn(0x7f8dc810d050 legacy=0x7f8dc80cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dc7fff700 1 -- 172.21.15.32:0/2702037722 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80d86e0 con 0x7f8dc810da80 2021-04-09T18:25:21.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dc7fff700 1 --2- 172.21.15.32:0/2702037722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc810da80 0x7f8dc810de80 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 rx=0x7f8dc0006f00 tx=0x7f8dc0009e90).ready entity=mon.0 client_cookie=b6d756dd0bb4fb9f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dc67fc700 1 -- 172.21.15.32:0/2702037722 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dc000b300 con 0x7f8dc810da80 2021-04-09T18:25:21.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dc67fc700 1 -- 172.21.15.32:0/2702037722 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc000b640 con 0x7f8dc810da80 2021-04-09T18:25:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dc67fc700 1 -- 172.21.15.32:0/2702037722 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dc000bb20 con 0x7f8dc810da80 2021-04-09T18:25:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dcdb4a700 1 -- 172.21.15.32:0/2702037722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc810da80 msgr2=0x7f8dc810de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.785 7f8dcdb4a700 1 --2- 172.21.15.32:0/2702037722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc810da80 0x7f8dc810de80 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 rx=0x7f8dc0006f00 tx=0x7f8dc0009e90).stop 2021-04-09T18:25:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.786 7f8dcdb4a700 1 -- 172.21.15.32:0/2702037722 shutdown_connections 2021-04-09T18:25:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.786 7f8dcdb4a700 1 --2- 172.21.15.32:0/2702037722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc810da80 0x7f8dc810de80 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.786 7f8dcdb4a700 1 --2- 172.21.15.32:0/2702037722 >> v2:172.21.15.86:3300/0 conn(0x7f8dc80d7cc0 0x7f8dc80d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.786 7f8dcdb4a700 1 -- 172.21.15.32:0/2702037722 >> 172.21.15.32:0/2702037722 conn(0x7f8dc80ce620 msgr2=0x7f8dc80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.786 7f8dcdb4a700 1 -- 172.21.15.32:0/2702037722 shutdown_connections 2021-04-09T18:25:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.786 7f8dcdb4a700 1 -- 172.21.15.32:0/2702037722 wait complete. 2021-04-09T18:25:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.806 7f8dcdb4a700 1 Processor -- start 2021-04-09T18:25:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.806 7f8dcdb4a700 1 -- start start 2021-04-09T18:25:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.806 7f8dcdb4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc80b84d0 0x7f8dc80c2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.806 7f8dcdb4a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dc80c2e20 0x7f8dc80a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.806 7f8dcdb4a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dc80a1b60 0x7f8dc80a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.806 7f8dcdb4a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc818e9f0 con 0x7f8dc80b84d0 2021-04-09T18:25:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.806 7f8dcdb4a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80a8420 con 0x7f8dc80a1b60 2021-04-09T18:25:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.806 7f8dcdb4a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80b3d90 con 0x7f8dc80c2e20 2021-04-09T18:25:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc80b84d0 0x7f8dc80c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc80b84d0 0x7f8dc80c2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52184/0 (socket says 172.21.15.32:52184) 2021-04-09T18:25:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc77fe700 1 -- 172.21.15.32:0/3874146912 learned_addr learned my addr 172.21.15.32:0/3874146912 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc7fff700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dc80a1b60 0x7f8dc80a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc6ffd700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dc80c2e20 0x7f8dc80a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc77fe700 1 -- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dc80c2e20 msgr2=0x7f8dc80a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc77fe700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dc80c2e20 0x7f8dc80a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc77fe700 1 -- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dc80a1b60 msgr2=0x7f8dc80a7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc77fe700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dc80a1b60 0x7f8dc80a7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc77fe700 1 -- 172.21.15.32:0/3874146912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80a7cb0 con 0x7f8dc80b84d0 2021-04-09T18:25:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dc77fe700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc80b84d0 0x7f8dc80c2800 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f8db800b530 tx=0x7f8db800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8daffff700 1 -- 172.21.15.32:0/3874146912 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8db800bcc0 con 0x7f8dc80b84d0 2021-04-09T18:25:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8daffff700 1 -- 172.21.15.32:0/3874146912 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db8006020 con 0x7f8dc80b84d0 2021-04-09T18:25:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8daffff700 1 -- 172.21.15.32:0/3874146912 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8db80054a0 con 0x7f8dc80b84d0 2021-04-09T18:25:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc80a7cb0 con 0x7f8dc80b84d0 2021-04-09T18:25:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.807 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc80a7cb0 con 0x7f8dc80b84d0 2021-04-09T18:25:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.808 7f8daffff700 1 -- 172.21.15.32:0/3874146912 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8db8006200 con 0x7f8dc80b84d0 2021-04-09T18:25:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.808 7f8daffff700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8da8018930 0x7f8da801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.808 7f8daffff700 1 -- 172.21.15.32:0/3874146912 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f8db8032ae0 con 0x7f8dc80b84d0 2021-04-09T18:25:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.808 7f8dc6ffd700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8da8018930 0x7f8da801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.808 7f8dc6ffd700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8da8018930 0x7f8da801adc0 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.828 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc81b7180 con 0x7f8dc80b84d0 2021-04-09T18:25:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:21.832 7f8daffff700 1 -- 172.21.15.32:0/3874146912 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8db800bcc0 con 0x7f8dc80b84d0 2021-04-09T18:25:22.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.014 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8dc810d500 con 0x7f8dc80b84d0 2021-04-09T18:25:22.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.589 7f8daffff700 1 -- 172.21.15.32:0/3874146912 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v121) v1 ==== 185+0+0 (secure 0 0 0) 0x7f8db80061e0 con 0x7f8dc80b84d0 2021-04-09T18:25:22.694 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:25:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.630 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8da8018930 msgr2=0x7f8da801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:22.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8da8018930 0x7f8da801adc0 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc80b84d0 msgr2=0x7f8dc80c2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:22.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc80b84d0 0x7f8dc80c2800 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f8db800b530 tx=0x7f8db800b8d0).stop 2021-04-09T18:25:22.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 shutdown_connections 2021-04-09T18:25:22.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8da8018930 0x7f8da801adc0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8dc80a1b60 0x7f8dc80a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8dc80c2e20 0x7f8dc80a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 --2- 172.21.15.32:0/3874146912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8dc80b84d0 0x7f8dc80c2800 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 >> 172.21.15.32:0/3874146912 conn(0x7f8dc80b6b70 msgr2=0x7f8dc80b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:22.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 shutdown_connections 2021-04-09T18:25:22.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.631 7f8dcdb4a700 1 -- 172.21.15.32:0/3874146912 wait complete. 2021-04-09T18:25:22.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:25:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.803 7feb5bfcf700 1 Processor -- start 2021-04-09T18:25:22.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.803 7feb5bfcf700 1 -- start start 2021-04-09T18:25:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb5bfcf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5410d020 0x7feb540d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:22.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb5bfcf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5410d490 con 0x7feb540d84a0 2021-04-09T18:25:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb5bfcf700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5410e200 con 0x7feb540d1db0 2021-04-09T18:25:22.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb5bfcf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5410e790 con 0x7feb5410d020 2021-04-09T18:25:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb5a5ec700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feb540d84a0 0x7feb5410da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33552/0 (socket says 172.21.15.32:33552) 2021-04-09T18:25:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb5a5ec700 1 -- 172.21.15.32:0/2510915047 learned_addr learned my addr 172.21.15.32:0/2510915047 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:22.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb59deb700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5410d020 0x7feb540d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:22.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb58de9700 1 -- 172.21.15.32:0/2510915047 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577564946 0 0) 0x7feb4c000d30 con 0x7feb540d84a0 2021-04-09T18:25:22.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb58de9700 1 -- 172.21.15.32:0/2510915047 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb40002fe0 con 0x7feb540d84a0 2021-04-09T18:25:22.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb58de9700 1 -- 172.21.15.32:0/2510915047 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 729955386 0 0) 0x7feb4c001270 con 0x7feb540d84a0 2021-04-09T18:25:22.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb58de9700 1 -- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5410d020 msgr2=0x7feb540d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb58de9700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5410d020 0x7feb540d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.804 7feb58de9700 1 -- 172.21.15.32:0/2510915047 >> v1:172.21.15.86:6789/0 conn(0x7feb540d1db0 legacy=0x7feb540d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 -- 172.21.15.32:0/2510915047 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb540d8d40 con 0x7feb540d84a0 2021-04-09T18:25:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 -- 172.21.15.32:0/2510915047 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7feb4c001980 con 0x7feb540d84a0 2021-04-09T18:25:22.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 -- 172.21.15.32:0/2510915047 >> v1:172.21.15.32:6789/0 conn(0x7feb540d84a0 legacy=0x7feb5410da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb40003a40 0x7feb40005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:22.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb40006730 0x7feb40008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb400094f0 0x7feb4000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:22.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 -- 172.21.15.32:0/2510915047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4000c2c0 con 0x7feb40006730 2021-04-09T18:25:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 -- 172.21.15.32:0/2510915047 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4000c4e0 con 0x7feb400094f0 2021-04-09T18:25:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 -- 172.21.15.32:0/2510915047 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb40050710 con 0x7feb40003a40 2021-04-09T18:25:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb595ea700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb40006730 0x7feb40008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:22.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb59deb700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb40003a40 0x7feb40005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:22.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb5a5ec700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb400094f0 0x7feb4000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:22.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb595ea700 1 -- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb40003a40 msgr2=0x7feb40005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:22.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb595ea700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb40003a40 0x7feb40005e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb595ea700 1 -- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb400094f0 msgr2=0x7feb4000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:22.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb595ea700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb400094f0 0x7feb4000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb595ea700 1 -- 172.21.15.32:0/2510915047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40050970 con 0x7feb40006730 2021-04-09T18:25:22.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb595ea700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb40006730 0x7feb40008c50 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7feb44006ad0 tx=0x7feb44009b40).ready entity=mon.0 client_cookie=dd8b52f5d5bd50d9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:22.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 -- 172.21.15.32:0/2510915047 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb4400c0c0 con 0x7feb40006730 2021-04-09T18:25:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 -- 172.21.15.32:0/2510915047 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4400c3c0 con 0x7feb40006730 2021-04-09T18:25:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb58de9700 1 -- 172.21.15.32:0/2510915047 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb4400c8a0 con 0x7feb40006730 2021-04-09T18:25:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb5bfcf700 1 -- 172.21.15.32:0/2510915047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb40006730 msgr2=0x7feb40008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb5bfcf700 1 --2- 172.21.15.32:0/2510915047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb40006730 0x7feb40008c50 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7feb44006ad0 tx=0x7feb44009b40).stop 2021-04-09T18:25:22.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.805 7feb5a5ec700 1 -- 172.21.15.32:0/2510915047 reap_dead start 2021-04-09T18:25:22.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.806 7feb5bfcf700 1 -- 172.21.15.32:0/2510915047 shutdown_connections 2021-04-09T18:25:22.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.806 7feb5bfcf700 1 -- 172.21.15.32:0/2510915047 >> 172.21.15.32:0/2510915047 conn(0x7feb540ce600 msgr2=0x7feb540d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:22.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.806 7feb5bfcf700 1 -- 172.21.15.32:0/2510915047 shutdown_connections 2021-04-09T18:25:22.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.806 7feb5bfcf700 1 -- 172.21.15.32:0/2510915047 wait complete. 2021-04-09T18:25:22.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.826 7feb5bfcf700 1 Processor -- start 2021-04-09T18:25:22.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.826 7feb5bfcf700 1 -- start start 2021-04-09T18:25:22.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5bfcf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb540ad730 0x7feb540ab320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:22.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5bfcf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb540a2990 0x7feb540a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:22.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5bfcf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb540b9f20 0x7feb540ba320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5bfcf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb540a2410 con 0x7feb540b9f20 2021-04-09T18:25:22.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5bfcf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb540bce20 con 0x7feb540ad730 2021-04-09T18:25:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5bfcf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb540a33b0 con 0x7feb540a2990 2021-04-09T18:25:22.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5a5ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb540b9f20 0x7feb540ba320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:22.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5a5ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb540b9f20 0x7feb540ba320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52208/0 (socket says 172.21.15.32:52208) 2021-04-09T18:25:22.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5a5ec700 1 -- 172.21.15.32:0/3677328887 learned_addr learned my addr 172.21.15.32:0/3677328887 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb59deb700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb540ad730 0x7feb540ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb595ea700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb540a2990 0x7feb540a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:22.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5a5ec700 1 -- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb540a2990 msgr2=0x7feb540a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:22.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5a5ec700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb540a2990 0x7feb540a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5a5ec700 1 -- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb540ad730 msgr2=0x7feb540ab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:22.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5a5ec700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb540ad730 0x7feb540ab320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:22.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5a5ec700 1 -- 172.21.15.32:0/3677328887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb540adb30 con 0x7feb540b9f20 2021-04-09T18:25:22.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb595ea700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb540a2990 0x7feb540a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:22.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.827 7feb5a5ec700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb540b9f20 0x7feb540ba320 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7feb4c00a900 tx=0x7feb4c00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:22.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.828 7feb4a7fc700 1 -- 172.21.15.32:0/3677328887 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb4c007bd0 con 0x7feb540b9f20 2021-04-09T18:25:22.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.828 7feb4a7fc700 1 -- 172.21.15.32:0/3677328887 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c004930 con 0x7feb540b9f20 2021-04-09T18:25:22.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.828 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb540adb30 con 0x7feb540b9f20 2021-04-09T18:25:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.828 7feb4a7fc700 1 -- 172.21.15.32:0/3677328887 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb4c004d80 con 0x7feb540b9f20 2021-04-09T18:25:22.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.828 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb540adb30 con 0x7feb540b9f20 2021-04-09T18:25:22.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.828 7feb4a7fc700 1 -- 172.21.15.32:0/3677328887 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7feb4c007950 con 0x7feb540b9f20 2021-04-09T18:25:22.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.829 7feb4a7fc700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb3c018930 0x7feb3c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:22.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.829 7feb4a7fc700 1 -- 172.21.15.32:0/3677328887 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5311+0+0 (secure 0 0 0) 0x7feb4c032c20 con 0x7feb540b9f20 2021-04-09T18:25:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.829 7feb59deb700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb3c018930 0x7feb3c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:22.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.829 7feb59deb700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb3c018930 0x7feb3c01ae10 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.848 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb540b9010 con 0x7feb540b9f20 2021-04-09T18:25:22.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:22.851 7feb4a7fc700 1 -- 172.21.15.32:0/3677328887 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7feb4c007950 con 0x7feb540b9f20 2021-04-09T18:25:23.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.034 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feb540b9010 con 0x7feb540b9f20 2021-04-09T18:25:23.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.593 7feb4a7fc700 1 -- 172.21.15.32:0/3677328887 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v122) v1 ==== 173+0+0 (secure 0 0 0) 0x7feb4c004720 con 0x7feb540b9f20 2021-04-09T18:25:23.664 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:25:23.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.600 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb3c018930 msgr2=0x7feb3c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:23.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.600 7feb5bfcf700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb3c018930 0x7feb3c01ae10 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb540b9f20 msgr2=0x7feb540ba320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:23.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb540b9f20 0x7feb540ba320 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7feb4c00a900 tx=0x7feb4c00aca0).stop 2021-04-09T18:25:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 shutdown_connections 2021-04-09T18:25:23.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb3c018930 0x7feb3c01ae10 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb540b9f20 0x7feb540ba320 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb540a2990 0x7feb540a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 --2- 172.21.15.32:0/3677328887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb540ad730 0x7feb540ab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 >> 172.21.15.32:0/3677328887 conn(0x7feb540b9a90 msgr2=0x7feb540c1370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:23.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 shutdown_connections 2021-04-09T18:25:23.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.601 7feb5bfcf700 1 -- 172.21.15.32:0/3677328887 wait complete. 2021-04-09T18:25:23.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:25:23.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.764 7f46e5e32700 1 Processor -- start 2021-04-09T18:25:23.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.764 7f46e5e32700 1 -- start start 2021-04-09T18:25:23.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.765 7f46e5e32700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46e00cffb0 0x7f46e00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:23.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.765 7f46e5e32700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f46e00d7810 0x7f46e00d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:23.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.765 7f46e5e32700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00d7290 con 0x7f46e00d8230 2021-04-09T18:25:23.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.765 7f46e5e32700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e00d91f0 con 0x7f46e00cffb0 2021-04-09T18:25:23.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.765 7f46e5e32700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e010dff0 con 0x7f46e00d7810 2021-04-09T18:25:23.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.765 7f46df7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46e00cffb0 0x7f46e00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:23.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.765 7f46df7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46e00cffb0 0x7f46e00d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52212/0 (socket says 172.21.15.32:52212) 2021-04-09T18:25:23.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.765 7f46deffd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f46e00d7810 0x7f46e00d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:23.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.765 7f46df7fe700 1 -- 172.21.15.32:0/2619863611 learned_addr learned my addr 172.21.15.32:0/2619863611 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:23.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46df7fe700 1 -- 172.21.15.32:0/2619863611 >> v2:172.21.15.86:3300/0 conn(0x7f46e00d7810 msgr2=0x7f46e00d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:23.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46df7fe700 1 --2- 172.21.15.32:0/2619863611 >> v2:172.21.15.86:3300/0 conn(0x7f46e00d7810 0x7f46e00d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46df7fe700 1 -- 172.21.15.32:0/2619863611 >> v1:172.21.15.86:6789/0 conn(0x7f46e00d8230 legacy=0x7f46e00d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:23.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46df7fe700 1 -- 172.21.15.32:0/2619863611 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e010e210 con 0x7f46e00cffb0 2021-04-09T18:25:23.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46deffd700 1 --2- 172.21.15.32:0/2619863611 >> v2:172.21.15.86:3300/0 conn(0x7f46e00d7810 0x7f46e00d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:23.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46df7fe700 1 --2- 172.21.15.32:0/2619863611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e00cffb0 0x7f46e00d2450 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 rx=0x7f46d0006700 tx=0x7f46d0009b80).ready entity=mon.0 client_cookie=5fd4cd7818a9773b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:23.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46de7fc700 1 -- 172.21.15.32:0/2619863611 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46d000c0c0 con 0x7f46e00cffb0 2021-04-09T18:25:23.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46de7fc700 1 -- 172.21.15.32:0/2619863611 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d000c440 con 0x7f46e00cffb0 2021-04-09T18:25:23.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46de7fc700 1 -- 172.21.15.32:0/2619863611 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46d000c920 con 0x7f46e00cffb0 2021-04-09T18:25:23.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46e5e32700 1 -- 172.21.15.32:0/2619863611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e00cffb0 msgr2=0x7f46e00d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:23.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.766 7f46e5e32700 1 --2- 172.21.15.32:0/2619863611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e00cffb0 0x7f46e00d2450 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 rx=0x7f46d0006700 tx=0x7f46d0009b80).stop 2021-04-09T18:25:23.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.767 7f46e5e32700 1 -- 172.21.15.32:0/2619863611 shutdown_connections 2021-04-09T18:25:23.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.767 7f46e5e32700 1 --2- 172.21.15.32:0/2619863611 >> v2:172.21.15.86:3300/0 conn(0x7f46e00d7810 0x7f46e00d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.767 7f46e5e32700 1 --2- 172.21.15.32:0/2619863611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e00cffb0 0x7f46e00d2450 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.767 7f46e5e32700 1 -- 172.21.15.32:0/2619863611 >> 172.21.15.32:0/2619863611 conn(0x7f46e00ce5d0 msgr2=0x7f46e00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:23.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.767 7f46e5e32700 1 -- 172.21.15.32:0/2619863611 shutdown_connections 2021-04-09T18:25:23.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.767 7f46e5e32700 1 -- 172.21.15.32:0/2619863611 wait complete. 2021-04-09T18:25:23.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.787 7f46e5e32700 1 Processor -- start 2021-04-09T18:25:23.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.787 7f46e5e32700 1 -- start start 2021-04-09T18:25:23.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46e5e32700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e008d210 0x7f46e008fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:23.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46e5e32700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46e00903f0 0x7f46e01a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:23.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46e5e32700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e01a6080 0x7f46e01a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46e5e32700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e008d690 con 0x7f46e008d210 2021-04-09T18:25:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46e5e32700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00b0fd0 con 0x7f46e01a6080 2021-04-09T18:25:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46e5e32700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e01a6480 con 0x7f46e00903f0 2021-04-09T18:25:23.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46df7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e008d210 0x7f46e008fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46df7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e008d210 0x7f46e008fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52218/0 (socket says 172.21.15.32:52218) 2021-04-09T18:25:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46df7fe700 1 -- 172.21.15.32:0/2927162518 learned_addr learned my addr 172.21.15.32:0/2927162518 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46dffff700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e01a6080 0x7f46e01a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46deffd700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46e00903f0 0x7f46e01a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:23.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46df7fe700 1 -- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46e00903f0 msgr2=0x7f46e01a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:23.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46df7fe700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46e00903f0 0x7f46e01a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46df7fe700 1 -- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e01a6080 msgr2=0x7f46e01a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:23.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46df7fe700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e01a6080 0x7f46e01a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:23.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46df7fe700 1 -- 172.21.15.32:0/2927162518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e008d8b0 con 0x7f46e008d210 2021-04-09T18:25:23.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46df7fe700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e008d210 0x7f46e008fdd0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f46d000b5e0 tx=0x7f46d000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:23.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46d7fff700 1 -- 172.21.15.32:0/2927162518 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46d000bd70 con 0x7f46e008d210 2021-04-09T18:25:23.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46d7fff700 1 -- 172.21.15.32:0/2927162518 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d0002030 con 0x7f46e008d210 2021-04-09T18:25:23.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e008d8b0 con 0x7f46e008d210 2021-04-09T18:25:23.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46d7fff700 1 -- 172.21.15.32:0/2927162518 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46d0002550 con 0x7f46e008d210 2021-04-09T18:25:23.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.788 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e008d8b0 con 0x7f46e008d210 2021-04-09T18:25:23.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.789 7f46d7fff700 1 -- 172.21.15.32:0/2927162518 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f46d0000c90 con 0x7f46e008d210 2021-04-09T18:25:23.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.789 7f46d7fff700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46c4018930 0x7f46c401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:23.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.789 7f46d7fff700 1 -- 172.21.15.32:0/2927162518 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f46d00327e0 con 0x7f46e008d210 2021-04-09T18:25:23.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.789 7f46deffd700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46c4018930 0x7f46c401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:23.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.790 7f46deffd700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46c4018930 0x7f46c401ae10 crc :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:23.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.809 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e01a52b0 con 0x7f46e008d210 2021-04-09T18:25:23.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:23.813 7f46d7fff700 1 -- 172.21.15.32:0/2927162518 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f46d001f7a0 con 0x7f46e008d210 2021-04-09T18:25:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.023 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f46e01a52b0 con 0x7f46e008d210 2021-04-09T18:25:24.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.027 7f46d7fff700 1 -- 172.21.15.32:0/2927162518 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f46d001f7a0 con 0x7f46e008d210 2021-04-09T18:25:24.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46c4018930 msgr2=0x7f46c401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46c4018930 0x7f46c401ae10 crc :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e008d210 msgr2=0x7f46e008fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e008d210 0x7f46e008fdd0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f46d000b5e0 tx=0x7f46d000b980).stop 2021-04-09T18:25:24.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 shutdown_connections 2021-04-09T18:25:24.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f46c4018930 0x7f46c401ae10 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e01a6080 0x7f46e01a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46e00903f0 0x7f46e01a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 --2- 172.21.15.32:0/2927162518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46e008d210 0x7f46e008fdd0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 >> 172.21.15.32:0/2927162518 conn(0x7f46e01a8f50 msgr2=0x7f46e0188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 shutdown_connections 2021-04-09T18:25:24.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.033 7f46e5e32700 1 -- 172.21.15.32:0/2927162518 wait complete. 2021-04-09T18:25:24.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_up_less_than_max' 2021-04-09T18:25:24.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.192 7f8f397b5700 1 Processor -- start 2021-04-09T18:25:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.192 7f8f397b5700 1 -- start start 2021-04-09T18:25:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.192 7f8f397b5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340cffa0 0x7f8f340d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.192 7f8f397b5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f340d6c80 0x7f8f3410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.192 7f8f397b5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f340d7280 con 0x7f8f340d7800 2021-04-09T18:25:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.192 7f8f397b5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f340d7080 con 0x7f8f340d6c80 2021-04-09T18:25:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.192 7f8f397b5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340d8580 con 0x7f8f340cffa0 2021-04-09T18:25:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f33d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f340d6c80 0x7f8f3410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f33d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f340d6c80 0x7f8f3410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52228/0 (socket says 172.21.15.32:52228) 2021-04-09T18:25:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f33d75700 1 -- 172.21.15.32:0/2195392433 learned_addr learned my addr 172.21.15.32:0/2195392433 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f33574700 1 --2- 172.21.15.32:0/2195392433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340cffa0 0x7f8f340d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f33d75700 1 -- 172.21.15.32:0/2195392433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340cffa0 msgr2=0x7f8f340d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f33d75700 1 --2- 172.21.15.32:0/2195392433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340cffa0 0x7f8f340d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f33d75700 1 -- 172.21.15.32:0/2195392433 >> v1:172.21.15.86:6789/0 conn(0x7f8f340d7800 legacy=0x7f8f340d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f32572700 1 -- 172.21.15.32:0/2195392433 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998356069 0 0) 0x7f8f28000ff0 con 0x7f8f340d7800 2021-04-09T18:25:24.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f33d75700 1 -- 172.21.15.32:0/2195392433 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f340d91e0 con 0x7f8f340d6c80 2021-04-09T18:25:24.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f33d75700 1 --2- 172.21.15.32:0/2195392433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340d6c80 0x7f8f3410cfd0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 rx=0x7f8f2c006f00 tx=0x7f8f2c009f10).ready entity=mon.0 client_cookie=f6a81a10d1f20b0f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:24.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f32572700 1 -- 172.21.15.32:0/2195392433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2c00b3b0 con 0x7f8f340d6c80 2021-04-09T18:25:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f32572700 1 -- 172.21.15.32:0/2195392433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c00b730 con 0x7f8f340d6c80 2021-04-09T18:25:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f32572700 1 -- 172.21.15.32:0/2195392433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2c00bc10 con 0x7f8f340d6c80 2021-04-09T18:25:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f397b5700 1 -- 172.21.15.32:0/2195392433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340d6c80 msgr2=0x7f8f3410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.193 7f8f397b5700 1 --2- 172.21.15.32:0/2195392433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340d6c80 0x7f8f3410cfd0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 rx=0x7f8f2c006f00 tx=0x7f8f2c009f10).stop 2021-04-09T18:25:24.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.194 7f8f397b5700 1 -- 172.21.15.32:0/2195392433 shutdown_connections 2021-04-09T18:25:24.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.194 7f8f397b5700 1 --2- 172.21.15.32:0/2195392433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340d6c80 0x7f8f3410cfd0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.194 7f8f397b5700 1 --2- 172.21.15.32:0/2195392433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340cffa0 0x7f8f340d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.194 7f8f397b5700 1 -- 172.21.15.32:0/2195392433 >> 172.21.15.32:0/2195392433 conn(0x7f8f340ce5c0 msgr2=0x7f8f340cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:24.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.194 7f8f397b5700 1 -- 172.21.15.32:0/2195392433 shutdown_connections 2021-04-09T18:25:24.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.194 7f8f397b5700 1 -- 172.21.15.32:0/2195392433 wait complete. 2021-04-09T18:25:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.215 7f8f397b5700 1 Processor -- start 2021-04-09T18:25:24.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.215 7f8f397b5700 1 -- start start 2021-04-09T18:25:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.215 7f8f397b5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3417bbf0 0x7f8f3417b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.215 7f8f397b5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f34193110 0x7f8f34193510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.215 7f8f397b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f3417c2f0 0x7f8f3417c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.215 7f8f397b5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34192c10 con 0x7f8f3417c2f0 2021-04-09T18:25:24.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.215 7f8f397b5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3417c030 con 0x7f8f3417bbf0 2021-04-09T18:25:24.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.215 7f8f397b5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34193b30 con 0x7f8f34193110 2021-04-09T18:25:24.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33574700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3417bbf0 0x7f8f3417b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33d75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f3417c2f0 0x7f8f3417c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33d75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f3417c2f0 0x7f8f3417c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52234/0 (socket says 172.21.15.32:52234) 2021-04-09T18:25:24.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33d75700 1 -- 172.21.15.32:0/2893569000 learned_addr learned my addr 172.21.15.32:0/2893569000 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:24.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33574700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3417bbf0 0x7f8f3417b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:43990/0 (socket says 172.21.15.32:43990) 2021-04-09T18:25:24.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f32d73700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f34193110 0x7f8f34193510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33d75700 1 -- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f34193110 msgr2=0x7f8f34193510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33d75700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f34193110 0x7f8f34193510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33d75700 1 -- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3417bbf0 msgr2=0x7f8f3417b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33d75700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3417bbf0 0x7f8f3417b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33d75700 1 -- 172.21.15.32:0/2893569000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3417cd10 con 0x7f8f3417c2f0 2021-04-09T18:25:24.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f32d73700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f34193110 0x7f8f34193510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33574700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3417bbf0 0x7f8f3417b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f33d75700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f3417c2f0 0x7f8f3417c6f0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 rx=0x7f8f2c00b9b0 tx=0x7f8f2c00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f1bfff700 1 -- 172.21.15.32:0/2893569000 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2c005820 con 0x7f8f3417c2f0 2021-04-09T18:25:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f1bfff700 1 -- 172.21.15.32:0/2893569000 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c005aa0 con 0x7f8f3417c2f0 2021-04-09T18:25:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3417cd10 con 0x7f8f3417c2f0 2021-04-09T18:25:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f1bfff700 1 -- 172.21.15.32:0/2893569000 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2c006030 con 0x7f8f3417c2f0 2021-04-09T18:25:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.216 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3417d2a0 con 0x7f8f3417c2f0 2021-04-09T18:25:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.217 7f8f1bfff700 1 -- 172.21.15.32:0/2893569000 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8f2c006030 con 0x7f8f3417c2f0 2021-04-09T18:25:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.217 7f8f1bfff700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f14018930 0x7f8f1401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.217 7f8f1bfff700 1 -- 172.21.15.32:0/2893569000 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f8f2c032b20 con 0x7f8f3417c2f0 2021-04-09T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.217 7f8f33574700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f14018930 0x7f8f1401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.218 7f8f33574700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f14018930 0x7f8f1401ae10 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:24.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.237 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f341b70f0 con 0x7f8f3417c2f0 2021-04-09T18:25:24.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.240 7f8f1bfff700 1 -- 172.21.15.32:0/2893569000 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8f2c000f90 con 0x7f8f3417c2f0 2021-04-09T18:25:24.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.429 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_up_less_than_max"]} v 0) v1 -- 0x7f8f341b70f0 con 0x7f8f3417c2f0 2021-04-09T18:25:24.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.594 7f8f1bfff700 1 -- 172.21.15.32:0/2893569000 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_up_less_than_max"]}]=0 v607) v1 ==== 145+0+0 (secure 0 0 0) 0x7f8f2c005a20 con 0x7f8f3417c2f0 2021-04-09T18:25:24.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f14018930 msgr2=0x7f8f1401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f14018930 0x7f8f1401ae10 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f3417c2f0 msgr2=0x7f8f3417c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f3417c2f0 0x7f8f3417c6f0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 rx=0x7f8f2c00b9b0 tx=0x7f8f2c00bd50).stop 2021-04-09T18:25:24.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 shutdown_connections 2021-04-09T18:25:24.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f14018930 0x7f8f1401ae10 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f3417c2f0 0x7f8f3417c6f0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f34193110 0x7f8f34193510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 --2- 172.21.15.32:0/2893569000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3417bbf0 0x7f8f3417b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 >> 172.21.15.32:0/2893569000 conn(0x7f8f341905d0 msgr2=0x7f8f341b7320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:24.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 shutdown_connections 2021-04-09T18:25:24.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.645 7f8f397b5700 1 -- 172.21.15.32:0/2893569000 wait complete. 2021-04-09T18:25:24.722 INFO:tasks.cephfs_test_runner:test_up_less_than_max (tasks.cephfs.test_failover.TestClusterResize) ... ok 2021-04-09T18:25:24.722 INFO:tasks.cephfs_test_runner:Starting test: test_client_abort (tasks.cephfs.test_failover.TestFailover) 2021-04-09T18:25:24.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestFailover.test_client_abort' 2021-04-09T18:25:24.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.805 7f96f1463700 1 Processor -- start 2021-04-09T18:25:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.805 7f96f1463700 1 -- start start 2021-04-09T18:25:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96f1463700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f96ec10d010 0x7f96ec0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96f1463700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec0d8490 0x7f96ec10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96f1463700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec10d480 con 0x7f96ec0d1da0 2021-04-09T18:25:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96f1463700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ec0d8d30 con 0x7f96ec10d010 2021-04-09T18:25:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96f1463700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec0d9220 con 0x7f96ec0d8490 2021-04-09T18:25:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96eb07d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f96ec10d010 0x7f96ec0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96ea87c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f96ec0d1da0 0x7f96ec0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33602/0 (socket says 172.21.15.32:33602) 2021-04-09T18:25:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96ea87c700 1 -- 172.21.15.32:0/2380046593 learned_addr learned my addr 172.21.15.32:0/2380046593 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:24.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96eb07d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f96ec10d010 0x7f96ec0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52240/0 (socket says 172.21.15.32:52240) 2021-04-09T18:25:24.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96eb87e700 1 --2- 172.21.15.32:0/2380046593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec0d8490 0x7f96ec10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96eb07d700 1 -- 172.21.15.32:0/2380046593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec0d8490 msgr2=0x7f96ec10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:24.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96eb07d700 1 --2- 172.21.15.32:0/2380046593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec0d8490 0x7f96ec10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96eb07d700 1 -- 172.21.15.32:0/2380046593 >> v1:172.21.15.32:6789/0 conn(0x7f96ec0d1da0 legacy=0x7f96ec0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96eb07d700 1 -- 172.21.15.32:0/2380046593 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec10e020 con 0x7f96ec10d010 2021-04-09T18:25:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96ea07b700 1 -- 172.21.15.32:0/2380046593 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064166249 0 0) 0x7f96e0000ce0 con 0x7f96ec0d1da0 2021-04-09T18:25:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96eb87e700 1 --2- 172.21.15.32:0/2380046593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec0d8490 0x7f96ec10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96eb07d700 1 --2- 172.21.15.32:0/2380046593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec10d010 0x7f96ec0d1780 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f96dc006650 tx=0x7f96dc009a80).ready entity=mon.0 client_cookie=1fc602aa029a6708 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.806 7f96ea07b700 1 -- 172.21.15.32:0/2380046593 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96dc0009b0 con 0x7f96ec10d010 2021-04-09T18:25:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96ea07b700 1 -- 172.21.15.32:0/2380046593 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96dc00c0c0 con 0x7f96ec10d010 2021-04-09T18:25:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96ea07b700 1 -- 172.21.15.32:0/2380046593 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96dc0009b0 con 0x7f96ec10d010 2021-04-09T18:25:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96f1463700 1 -- 172.21.15.32:0/2380046593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec10d010 msgr2=0x7f96ec0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96f1463700 1 --2- 172.21.15.32:0/2380046593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec10d010 0x7f96ec0d1780 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f96dc006650 tx=0x7f96dc009a80).stop 2021-04-09T18:25:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96f1463700 1 -- 172.21.15.32:0/2380046593 shutdown_connections 2021-04-09T18:25:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96f1463700 1 --2- 172.21.15.32:0/2380046593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec0d8490 0x7f96ec10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96f1463700 1 --2- 172.21.15.32:0/2380046593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec10d010 0x7f96ec0d1780 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96f1463700 1 -- 172.21.15.32:0/2380046593 >> 172.21.15.32:0/2380046593 conn(0x7f96ec0ce5f0 msgr2=0x7f96ec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96f1463700 1 -- 172.21.15.32:0/2380046593 shutdown_connections 2021-04-09T18:25:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.807 7f96f1463700 1 -- 172.21.15.32:0/2380046593 wait complete. 2021-04-09T18:25:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.827 7f96f1463700 1 Processor -- start 2021-04-09T18:25:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96f1463700 1 -- start start 2021-04-09T18:25:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96f1463700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec0b5360 0x7f96ec1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96f1463700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec18e140 0x7f96ec1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96f1463700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec1ad040 0x7f96ec08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96f1463700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec1a5360 con 0x7f96ec1ad040 2021-04-09T18:25:24.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96f1463700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec0902c0 con 0x7f96ec0b5360 2021-04-09T18:25:24.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96f1463700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec0904e0 con 0x7f96ec18e140 2021-04-09T18:25:24.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96eb87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec1ad040 0x7f96ec08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96eb87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec1ad040 0x7f96ec08fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52250/0 (socket says 172.21.15.32:52250) 2021-04-09T18:25:24.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96eb87e700 1 -- 172.21.15.32:0/353221563 learned_addr learned my addr 172.21.15.32:0/353221563 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:24.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96eb07d700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec0b5360 0x7f96ec1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.828 7f96ea87c700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec18e140 0x7f96ec1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96eb87e700 1 -- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec18e140 msgr2=0x7f96ec1aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96eb87e700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec18e140 0x7f96ec1aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96eb87e700 1 -- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec0b5360 msgr2=0x7f96ec1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:24.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96eb87e700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec0b5360 0x7f96ec1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:24.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96eb87e700 1 -- 172.21.15.32:0/353221563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec183c80 con 0x7f96ec1ad040 2021-04-09T18:25:24.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96eb07d700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec0b5360 0x7f96ec1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:24.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96eb87e700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec1ad040 0x7f96ec08fca0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f96e400b8c0 tx=0x7f96e400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:24.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96d3fff700 1 -- 172.21.15.32:0/353221563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96e4006030 con 0x7f96ec1ad040 2021-04-09T18:25:24.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96d3fff700 1 -- 172.21.15.32:0/353221563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e4006790 con 0x7f96ec1ad040 2021-04-09T18:25:24.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96f1463700 1 -- 172.21.15.32:0/353221563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec183c80 con 0x7f96ec1ad040 2021-04-09T18:25:24.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96d3fff700 1 -- 172.21.15.32:0/353221563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96e4006520 con 0x7f96ec1ad040 2021-04-09T18:25:24.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.829 7f96f1463700 1 -- 172.21.15.32:0/353221563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec183c80 con 0x7f96ec1ad040 2021-04-09T18:25:24.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.830 7f96d3fff700 1 -- 172.21.15.32:0/353221563 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f96e4006030 con 0x7f96ec1ad040 2021-04-09T18:25:24.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.830 7f96d3fff700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f96cc0188c0 0x7f96cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:24.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.830 7f96d3fff700 1 -- 172.21.15.32:0/353221563 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f96e40328f0 con 0x7f96ec1ad040 2021-04-09T18:25:24.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.830 7f96eb07d700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f96cc0188c0 0x7f96cc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:24.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.831 7f96eb07d700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f96cc0188c0 0x7f96cc01adc0 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:24.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.850 7f96f1463700 1 -- 172.21.15.32:0/353221563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec182d70 con 0x7f96ec1ad040 2021-04-09T18:25:24.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:24.853 7f96d3fff700 1 -- 172.21.15.32:0/353221563 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f96e400bda0 con 0x7f96ec1ad040 2021-04-09T18:25:25.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.043 7f96f1463700 1 -- 172.21.15.32:0/353221563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestFailover.test_client_abort"]} v 0) v1 -- 0x7f96ec182d70 con 0x7f96ec1ad040 2021-04-09T18:25:25.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.596 7f96d3fff700 1 -- 172.21.15.32:0/353221563 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestFailover.test_client_abort"]}]=0 v608) v1 ==== 139+0+0 (secure 0 0 0) 0x7f96e4006f20 con 0x7f96ec1ad040 2021-04-09T18:25:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.609 7f96f1463700 1 -- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f96cc0188c0 msgr2=0x7f96cc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.609 7f96f1463700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f96cc0188c0 0x7f96cc01adc0 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.609 7f96f1463700 1 -- 172.21.15.32:0/353221563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec1ad040 msgr2=0x7f96ec08fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.609 7f96f1463700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec1ad040 0x7f96ec08fca0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f96e400b8c0 tx=0x7f96e400bc60).stop 2021-04-09T18:25:25.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.610 7f96f1463700 1 -- 172.21.15.32:0/353221563 shutdown_connections 2021-04-09T18:25:25.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.610 7f96f1463700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f96cc0188c0 0x7f96cc01adc0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.610 7f96f1463700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96ec1ad040 0x7f96ec08fca0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.610 7f96f1463700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96ec18e140 0x7f96ec1aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.610 7f96f1463700 1 --2- 172.21.15.32:0/353221563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96ec0b5360 0x7f96ec1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.610 7f96f1463700 1 -- 172.21.15.32:0/353221563 >> 172.21.15.32:0/353221563 conn(0x7f96ec19b460 msgr2=0x7f96ec19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.610 7f96f1463700 1 -- 172.21.15.32:0/353221563 shutdown_connections 2021-04-09T18:25:25.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.610 7f96f1463700 1 -- 172.21.15.32:0/353221563 wait complete. 2021-04-09T18:25:25.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:25:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.775 7fdb4d595700 1 Processor -- start 2021-04-09T18:25:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.775 7fdb4d595700 1 -- start start 2021-04-09T18:25:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb4d595700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb4810d010 0x7fdb480d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb4d595700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdb480d8490 0x7fdb4810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb4d595700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4810d480 con 0x7fdb480d1da0 2021-04-09T18:25:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb4d595700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb480d8d30 con 0x7fdb4810d010 2021-04-09T18:25:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb4d595700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb480d9220 con 0x7fdb480d8490 2021-04-09T18:25:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb472bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb4810d010 0x7fdb480d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:25.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb46abe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdb480d1da0 0x7fdb480d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33618/0 (socket says 172.21.15.32:33618) 2021-04-09T18:25:25.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb46abe700 1 -- 172.21.15.32:0/208653177 learned_addr learned my addr 172.21.15.32:0/208653177 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:25.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb47ac0700 1 --2- 172.21.15.32:0/208653177 >> v2:172.21.15.86:3300/0 conn(0x7fdb480d8490 0x7fdb4810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:25.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb462bd700 1 -- 172.21.15.32:0/208653177 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877167294 0 0) 0x7fdb3c000d30 con 0x7fdb480d1da0 2021-04-09T18:25:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.776 7fdb462bd700 1 -- 172.21.15.32:0/208653177 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb30002fe0 con 0x7fdb480d1da0 2021-04-09T18:25:25.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb472bf700 1 -- 172.21.15.32:0/208653177 >> v2:172.21.15.86:3300/0 conn(0x7fdb480d8490 msgr2=0x7fdb4810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:25.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb472bf700 1 --2- 172.21.15.32:0/208653177 >> v2:172.21.15.86:3300/0 conn(0x7fdb480d8490 0x7fdb4810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb472bf700 1 -- 172.21.15.32:0/208653177 >> v1:172.21.15.32:6789/0 conn(0x7fdb480d1da0 legacy=0x7fdb480d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:25.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb472bf700 1 -- 172.21.15.32:0/208653177 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4810e020 con 0x7fdb4810d010 2021-04-09T18:25:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb472bf700 1 --2- 172.21.15.32:0/208653177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb4810d010 0x7fdb480d1780 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7fdb38006650 tx=0x7fdb38009a80).ready entity=mon.0 client_cookie=ec8af6a77c6851a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:25.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb462bd700 1 -- 172.21.15.32:0/208653177 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb380009b0 con 0x7fdb4810d010 2021-04-09T18:25:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb462bd700 1 -- 172.21.15.32:0/208653177 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3800c0c0 con 0x7fdb4810d010 2021-04-09T18:25:25.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb462bd700 1 -- 172.21.15.32:0/208653177 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb3800c510 con 0x7fdb4810d010 2021-04-09T18:25:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb4d595700 1 -- 172.21.15.32:0/208653177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb4810d010 msgr2=0x7fdb480d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.777 7fdb4d595700 1 --2- 172.21.15.32:0/208653177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb4810d010 0x7fdb480d1780 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7fdb38006650 tx=0x7fdb38009a80).stop 2021-04-09T18:25:25.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.779 7fdb4d595700 1 -- 172.21.15.32:0/208653177 shutdown_connections 2021-04-09T18:25:25.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.779 7fdb4d595700 1 --2- 172.21.15.32:0/208653177 >> v2:172.21.15.86:3300/0 conn(0x7fdb480d8490 0x7fdb4810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.779 7fdb4d595700 1 --2- 172.21.15.32:0/208653177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb4810d010 0x7fdb480d1780 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.779 7fdb4d595700 1 -- 172.21.15.32:0/208653177 >> 172.21.15.32:0/208653177 conn(0x7fdb480ce5f0 msgr2=0x7fdb480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:25.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.779 7fdb4d595700 1 -- 172.21.15.32:0/208653177 shutdown_connections 2021-04-09T18:25:25.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.779 7fdb4d595700 1 -- 172.21.15.32:0/208653177 wait complete. 2021-04-09T18:25:25.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.799 7fdb4d595700 1 Processor -- start 2021-04-09T18:25:25.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.799 7fdb4d595700 1 -- start start 2021-04-09T18:25:25.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.799 7fdb4d595700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb481a0050 0x7fdb481a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:25.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.799 7fdb4d595700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb480bdcd0 0x7fdb480bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:25.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb4d595700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb480bf080 0x7fdb480b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:25.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb4d595700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480bf500 con 0x7fdb480bf080 2021-04-09T18:25:25.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb4d595700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480bb7b0 con 0x7fdb480bdcd0 2021-04-09T18:25:25.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb4d595700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480be0d0 con 0x7fdb481a0050 2021-04-09T18:25:25.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb47ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb480bf080 0x7fdb480b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:25.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb47ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb480bf080 0x7fdb480b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52266/0 (socket says 172.21.15.32:52266) 2021-04-09T18:25:25.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb47ac0700 1 -- 172.21.15.32:0/491389452 learned_addr learned my addr 172.21.15.32:0/491389452 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb46abe700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb480bdcd0 0x7fdb480bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb472bf700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb481a0050 0x7fdb481a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb47ac0700 1 -- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb481a0050 msgr2=0x7fdb481a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:25.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb47ac0700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb481a0050 0x7fdb481a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb47ac0700 1 -- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb480bdcd0 msgr2=0x7fdb480bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:25.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb47ac0700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb480bdcd0 0x7fdb480bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:25.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb47ac0700 1 -- 172.21.15.32:0/491389452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb480b1310 con 0x7fdb480bf080 2021-04-09T18:25:25.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb47ac0700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb480bf080 0x7fdb480b0cf0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7fdb4000b5e0 tx=0x7fdb4000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb2ffff700 1 -- 172.21.15.32:0/491389452 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb40000c10 con 0x7fdb480bf080 2021-04-09T18:25:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb2ffff700 1 -- 172.21.15.32:0/491389452 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4000bd70 con 0x7fdb480bf080 2021-04-09T18:25:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb4d595700 1 -- 172.21.15.32:0/491389452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb480b1310 con 0x7fdb480bf080 2021-04-09T18:25:25.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb2ffff700 1 -- 172.21.15.32:0/491389452 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb40002560 con 0x7fdb480bf080 2021-04-09T18:25:25.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.800 7fdb4d595700 1 -- 172.21.15.32:0/491389452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb480b1310 con 0x7fdb480bf080 2021-04-09T18:25:25.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.801 7fdb2ffff700 1 -- 172.21.15.32:0/491389452 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdb40006720 con 0x7fdb480bf080 2021-04-09T18:25:25.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.801 7fdb2ffff700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb28018900 0x7fdb2801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:25.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.801 7fdb2ffff700 1 -- 172.21.15.32:0/491389452 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fdb40032810 con 0x7fdb480bf080 2021-04-09T18:25:25.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.801 7fdb472bf700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb28018900 0x7fdb2801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:25.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.802 7fdb472bf700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb28018900 0x7fdb2801adc0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:25.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.821 7fdb4d595700 1 -- 172.21.15.32:0/491389452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb480bb7b0 con 0x7fdb480bf080 2021-04-09T18:25:25.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:25.825 7fdb2ffff700 1 -- 172.21.15.32:0/491389452 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdb40006370 con 0x7fdb480bf080 2021-04-09T18:25:26.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.036 7fdb4d595700 1 -- 172.21.15.32:0/491389452 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdb480bb7b0 con 0x7fdb480bf080 2021-04-09T18:25:26.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.040 7fdb2ffff700 1 -- 172.21.15.32:0/491389452 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdb40006720 con 0x7fdb480bf080 2021-04-09T18:25:26.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 -- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb28018900 msgr2=0x7fdb2801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:26.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb28018900 0x7fdb2801adc0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 -- 172.21.15.32:0/491389452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb480bf080 msgr2=0x7fdb480b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:26.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb480bf080 0x7fdb480b0cf0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7fdb4000b5e0 tx=0x7fdb4000b980).stop 2021-04-09T18:25:26.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 -- 172.21.15.32:0/491389452 shutdown_connections 2021-04-09T18:25:26.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb28018900 0x7fdb2801adc0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb480bf080 0x7fdb480b0cf0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb480bdcd0 0x7fdb480bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 --2- 172.21.15.32:0/491389452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb481a0050 0x7fdb481a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 -- 172.21.15.32:0/491389452 >> 172.21.15.32:0/491389452 conn(0x7fdb480975f0 msgr2=0x7fdb48096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 -- 172.21.15.32:0/491389452 shutdown_connections 2021-04-09T18:25:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.046 7fdb4d595700 1 -- 172.21.15.32:0/491389452 wait complete. 2021-04-09T18:25:26.122 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:25:26.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:25:26.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:25:26.237 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:25:26.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:26.238 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:26.239 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:25:26.283 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:25:26.283 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:25:26.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:26.285 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:26.285 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:25:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:25:26.329 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:25:26.330 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:25:26.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:25:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.412 7f7c2878d700 1 Processor -- start 2021-04-09T18:25:26.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.413 7f7c2878d700 1 -- start start 2021-04-09T18:25:26.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.413 7f7c2878d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7c2010d010 0x7f7c200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:26.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.413 7f7c2878d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c200d1da0 0x7f7c200d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.413 7f7c2878d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2010d480 con 0x7f7c200d82c0 2021-04-09T18:25:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.413 7f7c2878d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c200d8d30 con 0x7f7c2010d010 2021-04-09T18:25:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.413 7f7c2878d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c200d9220 con 0x7f7c200d1da0 2021-04-09T18:25:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.413 7f7c265a9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7c2010d010 0x7f7c200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c265a9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7c2010d010 0x7f7c200d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52270/0 (socket says 172.21.15.32:52270) 2021-04-09T18:25:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c265a9700 1 -- 172.21.15.32:0/2613771142 learned_addr learned my addr 172.21.15.32:0/2613771142 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c25da8700 1 --2- 172.21.15.32:0/2613771142 >> v2:172.21.15.86:3300/0 conn(0x7f7c200d1da0 0x7f7c200d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c265a9700 1 -- 172.21.15.32:0/2613771142 >> v2:172.21.15.86:3300/0 conn(0x7f7c200d1da0 msgr2=0x7f7c200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c265a9700 1 --2- 172.21.15.32:0/2613771142 >> v2:172.21.15.86:3300/0 conn(0x7f7c200d1da0 0x7f7c200d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c265a9700 1 -- 172.21.15.32:0/2613771142 >> v1:172.21.15.86:6789/0 conn(0x7f7c200d82c0 legacy=0x7f7c2010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c265a9700 1 -- 172.21.15.32:0/2613771142 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c200d86f0 con 0x7f7c2010d010 2021-04-09T18:25:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c265a9700 1 --2- 172.21.15.32:0/2613771142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c2010d010 0x7f7c200d1780 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7f7c1c006700 tx=0x7f7c1c009b80).ready entity=mon.0 client_cookie=1cdc9b1c3b24caff server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c255a7700 1 -- 172.21.15.32:0/2613771142 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1c00c0c0 con 0x7f7c2010d010 2021-04-09T18:25:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c255a7700 1 -- 172.21.15.32:0/2613771142 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c00c440 con 0x7f7c2010d010 2021-04-09T18:25:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c255a7700 1 -- 172.21.15.32:0/2613771142 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1c00c920 con 0x7f7c2010d010 2021-04-09T18:25:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c2878d700 1 -- 172.21.15.32:0/2613771142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c2010d010 msgr2=0x7f7c200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.414 7f7c2878d700 1 --2- 172.21.15.32:0/2613771142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c2010d010 0x7f7c200d1780 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7f7c1c006700 tx=0x7f7c1c009b80).stop 2021-04-09T18:25:26.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.415 7f7c2878d700 1 -- 172.21.15.32:0/2613771142 shutdown_connections 2021-04-09T18:25:26.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.415 7f7c2878d700 1 --2- 172.21.15.32:0/2613771142 >> v2:172.21.15.86:3300/0 conn(0x7f7c200d1da0 0x7f7c200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.415 7f7c2878d700 1 --2- 172.21.15.32:0/2613771142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c2010d010 0x7f7c200d1780 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.415 7f7c2878d700 1 -- 172.21.15.32:0/2613771142 >> 172.21.15.32:0/2613771142 conn(0x7f7c200ce5f0 msgr2=0x7f7c200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:26.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.415 7f7c2878d700 1 -- 172.21.15.32:0/2613771142 shutdown_connections 2021-04-09T18:25:26.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.415 7f7c2878d700 1 -- 172.21.15.32:0/2613771142 wait complete. 2021-04-09T18:25:26.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.435 7f7c2878d700 1 Processor -- start 2021-04-09T18:25:26.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.435 7f7c2878d700 1 -- start start 2021-04-09T18:25:26.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c2878d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c2009a670 0x7f7c20093610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:26.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c2878d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c201a9fb0 0x7f7c201aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:26.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c2878d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200a3ab0 0x7f7c200a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:26.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c2878d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20094060 con 0x7f7c201a9fb0 2021-04-09T18:25:26.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c2878d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20094710 con 0x7f7c200a3ab0 2021-04-09T18:25:26.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c2878d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20093c30 con 0x7f7c2009a670 2021-04-09T18:25:26.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c25da8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c201a9fb0 0x7f7c201aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:26.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c25da8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c201a9fb0 0x7f7c201aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52278/0 (socket says 172.21.15.32:52278) 2021-04-09T18:25:26.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c25da8700 1 -- 172.21.15.32:0/3400178299 learned_addr learned my addr 172.21.15.32:0/3400178299 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:26.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c265a9700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c2009a670 0x7f7c20093610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:26.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c26daa700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200a3ab0 0x7f7c200a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:26.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c25da8700 1 -- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c2009a670 msgr2=0x7f7c20093610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:26.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c25da8700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c2009a670 0x7f7c20093610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c25da8700 1 -- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200a3ab0 msgr2=0x7f7c200a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:26.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c25da8700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200a3ab0 0x7f7c200a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.436 7f7c25da8700 1 -- 172.21.15.32:0/3400178299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c201aa9d0 con 0x7f7c201a9fb0 2021-04-09T18:25:26.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.437 7f7c25da8700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c201a9fb0 0x7f7c201aa3b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7f7c10006f60 tx=0x7f7c1000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:26.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.437 7f7c16ffd700 1 -- 172.21.15.32:0/3400178299 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1000bd60 con 0x7f7c201a9fb0 2021-04-09T18:25:26.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.437 7f7c16ffd700 1 -- 172.21.15.32:0/3400178299 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c10000c90 con 0x7f7c201a9fb0 2021-04-09T18:25:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.437 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c201aa9d0 con 0x7f7c201a9fb0 2021-04-09T18:25:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.437 7f7c16ffd700 1 -- 172.21.15.32:0/3400178299 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c10002220 con 0x7f7c201a9fb0 2021-04-09T18:25:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.437 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c201aa9d0 con 0x7f7c201a9fb0 2021-04-09T18:25:26.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.437 7f7c16ffd700 1 -- 172.21.15.32:0/3400178299 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7c100052e0 con 0x7f7c201a9fb0 2021-04-09T18:25:26.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.437 7f7c16ffd700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c08018930 0x7f7c0801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:26.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.438 7f7c16ffd700 1 -- 172.21.15.32:0/3400178299 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f7c10032780 con 0x7f7c201a9fb0 2021-04-09T18:25:26.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.438 7f7c265a9700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c08018930 0x7f7c0801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:26.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.438 7f7c265a9700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c08018930 0x7f7c0801ae10 crc :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:26.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.457 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2010d480 con 0x7f7c201a9fb0 2021-04-09T18:25:26.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.461 7f7c16ffd700 1 -- 172.21.15.32:0/3400178299 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7c10000c90 con 0x7f7c201a9fb0 2021-04-09T18:25:26.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.651 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f7c2010d480 con 0x7f7c201a9fb0 2021-04-09T18:25:26.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.652 7f7c16ffd700 1 -- 172.21.15.32:0/3400178299 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v122) v1 ==== 81+0+25690 (secure 0 0 0) 0x7f7c10006b00 con 0x7f7c201a9fb0 2021-04-09T18:25:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:26.718 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 122, 2021-04-09T18:25:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:25:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:25:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:25:23.589620", 2021-04-09T18:25:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:25:26.719 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:25:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:25:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:25:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:25:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:25:26.720 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:25:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:25:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:25:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:25:26.721 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:25:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:25:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:25:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:25:26.722 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:25:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 21, 2021-04-09T18:25:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:25:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:25:26.723 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:25:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:25:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:25:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.724 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:25:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:25:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:25:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:25:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:25:26.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:25:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:25:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:25:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:25:26.726 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:25:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:25:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:25:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:25:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:25:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:25:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:25:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:25:26.728 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:25:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:25:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:25:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:25:26.729 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:25:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:25:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:25:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:25:26.730 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:25:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:25:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:25:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:25:26.731 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:25:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:25:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:25:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:25:26.732 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:25:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:25:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:25:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:25:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:25:26.733 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:25:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:25:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:25:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:25:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:25:26.734 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:25:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:25:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:25:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:25:26.735 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:25:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:25:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:25:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:25:26.736 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:25:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:25:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:25:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:25:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:25:26.737 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:25:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:25:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:25:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:25:26.738 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:25:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:25:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:25:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:25:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.739 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:25:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:25:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.740 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:25:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:25:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:26.741 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 112, 2021-04-09T18:25:26.742 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:25:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:25:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:26.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.746 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:25:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:25:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.748 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.749 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:25:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:25:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.751 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:25:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.753 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:25:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:25:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.754 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:25:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:25:26.755 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:25:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:25:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:26.756 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:25:26.757 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:25:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:26.758 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:26.759 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:25:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:25:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.762 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:25:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:26.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:25:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:26.765 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:25:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:26.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:25:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:26.768 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.769 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.769 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:26.769 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.770 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.770 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:25:26.770 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:26.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.771 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.771 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:25:26.771 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:25:26.771 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.772 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.772 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:25:26.772 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:25:26.773 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:25:26.773 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:25:26.773 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:26.773 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:26.774 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:26.774 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.774 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.774 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:25:26.775 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:25:26.775 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:26.775 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:26.775 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:26.775 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:26.776 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:26.776 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:26.776 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:26.776 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:26.777 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:26.777 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:26.777 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:26.777 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.778 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.778 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:25:26.778 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:26.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.779 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:25:26.779 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:26.779 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.780 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.780 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:26.780 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.781 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.781 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:25:26.781 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:26.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.782 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:25:26.782 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:26.782 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.783 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.783 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:26.783 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.784 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.784 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:25:26.784 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:26.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.785 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.785 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:25:26.785 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:26.785 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.786 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.786 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:26.786 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.787 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.787 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:25:26.787 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:26.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.788 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.788 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:25:26.788 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:25:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:25:26.789 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:25:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:25:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:25:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:26.790 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.791 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:25:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:25:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:26.792 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:26.793 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:26.794 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:25:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:26.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:25:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:26.796 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.797 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:25:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:25:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:26.799 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:25:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:26.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.802 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.802 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:25:26.802 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:26.802 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.803 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.803 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:26.803 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.804 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.804 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:25:26.804 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:26.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.805 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:25:26.805 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:25:26.806 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.806 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.806 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:25:26.806 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:25:26.807 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:25:26.807 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:25:26.807 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:26.807 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:26.808 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:26.808 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.808 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:25:26.809 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:25:26.809 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:26.809 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:26.809 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:26.810 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:26.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:26.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:26.810 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:26.810 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 105, 2021-04-09T18:25:26.811 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:26.811 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:26.811 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:26.811 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.812 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.812 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:25:26.812 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:26.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.813 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:25:26.813 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:26.814 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.814 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.815 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:26.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:25:26.816 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:26.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.816 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.817 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:25:26.817 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:26.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.817 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.818 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:26.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:25:26.819 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:26.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.820 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:25:26.820 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:26.820 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.820 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.821 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:26.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:25:26.822 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:26.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.823 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:25:26.823 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:25:26.824 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.824 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.824 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:25:26.825 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:25:26.825 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:25:26.825 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:25:26.825 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:26.826 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:26.826 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:26.826 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.827 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:25:26.827 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:25:26.827 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:26.827 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:26.828 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:26.828 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:26.828 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:26.828 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:26.829 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:26.829 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:26.829 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:26.829 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:26.830 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:26.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:25:26.831 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:26.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.832 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:25:26.832 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:26.833 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.834 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.834 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:26.834 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.835 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.835 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:25:26.835 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:26.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.836 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:25:26.836 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:26.837 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.837 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.838 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:26.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.840 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:25:26.840 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:26.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:25:26.841 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:26.841 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.842 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.842 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:26.842 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.843 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:25:26.844 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:26.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.845 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:25:26.845 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:25:26.845 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.845 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.846 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:25:26.846 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:25:26.846 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:25:26.846 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:25:26.847 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:26.847 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:26.847 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:26.847 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.848 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:25:26.848 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:25:26.849 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:26.849 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:26.849 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:26.849 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:26.849 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:26.850 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:26.850 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:26.850 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:26.850 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:26.851 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:26.851 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:26.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:25:26.852 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:26.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.853 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:25:26.853 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:26.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.853 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.854 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:26.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:25:26.855 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:26.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.856 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:25:26.856 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:26.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.856 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.857 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:26.857 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.857 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.858 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:25:26.858 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:26.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.859 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.859 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:25:26.859 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:26.859 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.860 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.860 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:26.861 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.861 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.861 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:25:26.861 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:26.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.862 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:25:26.863 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:25:26.863 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.863 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.864 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:25:26.864 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:25:26.864 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:25:26.864 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:25:26.865 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:26.865 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:26.865 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:26.865 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.866 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:25:26.866 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:25:26.866 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:25:26.867 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:25:26.867 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:25:26.867 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:25:26.867 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:25:26.867 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:25:26.868 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:25:26.868 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:25:26.868 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:25:26.868 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:25:26.869 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:25:26.869 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.869 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.869 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:25:26.870 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:26.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.871 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:25:26.871 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:26.871 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.872 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.872 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:25:26.872 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.873 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.873 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:25:26.873 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:26.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.874 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.874 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:25:26.874 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:26.874 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.875 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.875 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:25:26.875 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.876 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.876 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:25:26.876 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:26.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.877 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.877 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:25:26.877 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:26.878 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.878 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.878 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:25:26.879 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:25:26.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.879 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:25:26.879 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:25:26.879 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:26.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:26.880 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:25:26.881 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:25:26.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.881 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.882 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:25:26.882 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:25:26.882 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:25:26.882 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:25:26.882 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:25:26.883 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:25:26.883 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:25:26.883 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:26.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.884 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:25:26.884 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:25:26.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.885 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:25:26.885 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:26.885 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:26.885 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:26.886 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:26.886 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:26.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.887 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:25:26.887 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:26.887 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:26.888 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:26.888 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:26.888 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:26.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.889 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:25:26.889 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:26.889 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:26.890 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:26.890 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:26.890 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:26.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.891 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:25:26.891 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:26.891 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:26.891 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:26.892 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:26.892 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:26.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.892 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:25:26.893 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:26.893 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:26.893 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:26.893 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:26.894 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:26.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.894 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:25:26.894 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:26.895 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:26.895 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:26.895 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:26.895 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:26.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.896 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:25:26.896 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:26.896 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:26.896 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:26.897 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:26.897 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:26.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:26.898 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:25:26.898 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:25:26.898 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:25:26.898 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:25:26.898 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:25:26.899 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:25:26.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.899 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:25:26.899 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:25:26.899 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:25:26.900 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:25:26.900 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:25:26.900 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:25:26.900 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/2311632030": "2021-04-10 18:25:19.579145", 2021-04-09T18:25:26.901 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/2311632030": "2021-04-10 18:25:19.579145", 2021-04-09T18:25:26.901 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/2147753572": "2021-04-10 18:25:19.579163", 2021-04-09T18:25:26.901 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/2147753572": "2021-04-10 18:25:19.579163" 2021-04-09T18:25:26.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.901 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:25:26.902 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:25:26.902 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:25:26.902 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:25:26.902 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:25:26.902 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:25:26.903 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:25:26.903 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:25:26.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:26.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:26.904 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:25:26.904 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:25:26.904 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:25:26.904 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:25:26.904 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:25:26.905 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c08018930 msgr2=0x7f7c0801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c08018930 0x7f7c0801ae10 crc :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c201a9fb0 msgr2=0x7f7c201aa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c201a9fb0 0x7f7c201aa3b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7f7c10006f60 tx=0x7f7c1000b970).stop 2021-04-09T18:25:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 shutdown_connections 2021-04-09T18:25:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c08018930 0x7f7c0801ae10 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c201a9fb0 0x7f7c201aa3b0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200a3ab0 0x7f7c200a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 --2- 172.21.15.32:0/3400178299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c2009a670 0x7f7c20093610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 >> 172.21.15.32:0/3400178299 conn(0x7f7c201a0db0 msgr2=0x7f7c201a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 shutdown_connections 2021-04-09T18:25:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.654 7f7c2878d700 1 -- 172.21.15.32:0/3400178299 wait complete. 2021-04-09T18:25:26.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:27.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.993 7f3d73de5700 1 Processor -- start 2021-04-09T18:25:27.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.993 7f3d73de5700 1 -- start start 2021-04-09T18:25:27.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d73de5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d6c0d7ca0 0x7f3d6c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d73de5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c10d480 con 0x7f3d6c10d010 2021-04-09T18:25:27.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d73de5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c0d86c0 con 0x7f3d6c10da00 2021-04-09T18:25:27.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d73de5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6c0d9220 con 0x7f3d6c0d7ca0 2021-04-09T18:25:27.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d71c01700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d6c10d010 0x7f3d6c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33642/0 (socket says 172.21.15.32:33642) 2021-04-09T18:25:27.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d71c01700 1 -- 172.21.15.32:0/1389004145 learned_addr learned my addr 172.21.15.32:0/1389004145 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:27.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d71400700 1 --2- 172.21.15.32:0/1389004145 >> v2:172.21.15.86:3300/0 conn(0x7f3d6c0d7ca0 0x7f3d6c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d70bff700 1 -- 172.21.15.32:0/1389004145 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196727438 0 0) 0x7f3d5c000ce0 con 0x7f3d6c10d010 2021-04-09T18:25:27.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d70bff700 1 -- 172.21.15.32:0/1389004145 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d58002fe0 con 0x7f3d6c10d010 2021-04-09T18:25:27.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d70bff700 1 -- 172.21.15.32:0/1389004145 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105244910 0 0) 0x7f3d64000d30 con 0x7f3d6c10da00 2021-04-09T18:25:27.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.994 7f3d70bff700 1 -- 172.21.15.32:0/1389004145 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d58003210 con 0x7f3d6c10da00 2021-04-09T18:25:27.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.995 7f3d71400700 1 -- 172.21.15.32:0/1389004145 >> v1:172.21.15.86:6789/0 conn(0x7f3d6c10da00 legacy=0x7f3d6c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.995 7f3d71400700 1 -- 172.21.15.32:0/1389004145 >> v1:172.21.15.32:6789/0 conn(0x7f3d6c10d010 legacy=0x7f3d6c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.995 7f3d71400700 1 -- 172.21.15.32:0/1389004145 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c0d1f70 con 0x7f3d6c0d7ca0 2021-04-09T18:25:27.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.995 7f3d71400700 1 --2- 172.21.15.32:0/1389004145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6c0d7ca0 0x7f3d6c0d80a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f3d68006650 tx=0x7f3d68009a80).ready entity=mon.1 client_cookie=51cd465530552e13 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:27.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.995 7f3d70bff700 1 -- 172.21.15.32:0/1389004145 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d680009b0 con 0x7f3d6c0d7ca0 2021-04-09T18:25:27.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.995 7f3d70bff700 1 -- 172.21.15.32:0/1389004145 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6800c0c0 con 0x7f3d6c0d7ca0 2021-04-09T18:25:27.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.995 7f3d70bff700 1 -- 172.21.15.32:0/1389004145 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d6800c510 con 0x7f3d6c0d7ca0 2021-04-09T18:25:27.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.995 7f3d73de5700 1 -- 172.21.15.32:0/1389004145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6c0d7ca0 msgr2=0x7f3d6c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.995 7f3d73de5700 1 --2- 172.21.15.32:0/1389004145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6c0d7ca0 0x7f3d6c0d80a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f3d68006650 tx=0x7f3d68009a80).stop 2021-04-09T18:25:27.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.996 7f3d73de5700 1 -- 172.21.15.32:0/1389004145 shutdown_connections 2021-04-09T18:25:27.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.996 7f3d73de5700 1 --2- 172.21.15.32:0/1389004145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6c0d7ca0 0x7f3d6c0d80a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.996 7f3d73de5700 1 -- 172.21.15.32:0/1389004145 >> 172.21.15.32:0/1389004145 conn(0x7f3d6c0ce5f0 msgr2=0x7f3d6c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:27.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.996 7f3d73de5700 1 -- 172.21.15.32:0/1389004145 shutdown_connections 2021-04-09T18:25:27.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:26.996 7f3d73de5700 1 -- 172.21.15.32:0/1389004145 wait complete. 2021-04-09T18:25:27.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.015 7f3d73de5700 1 Processor -- start 2021-04-09T18:25:27.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.015 7f3d73de5700 1 -- start start 2021-04-09T18:25:27.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d73de5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d6c179610 0x7f3d6c179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d73de5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6c17a6e0 0x7f3d6c17aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d73de5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6c17b100 0x7f3d6c08db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d73de5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c17d720 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d73de5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c185ba0 con 0x7f3d6c17b100 2021-04-09T18:25:27.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d73de5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c08e1b0 con 0x7f3d6c179610 2021-04-09T18:25:27.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6c17a6e0 0x7f3d6c17aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6c17a6e0 0x7f3d6c17aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52290/0 (socket says 172.21.15.32:52290) 2021-04-09T18:25:27.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71400700 1 -- 172.21.15.32:0/4056808573 learned_addr learned my addr 172.21.15.32:0/4056808573 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:27.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d72402700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6c17b100 0x7f3d6c08db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71c01700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d6c179610 0x7f3d6c179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71400700 1 -- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d6c179610 msgr2=0x7f3d6c179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71400700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d6c179610 0x7f3d6c179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71400700 1 -- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6c17b100 msgr2=0x7f3d6c08db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71400700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6c17b100 0x7f3d6c08db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71400700 1 -- 172.21.15.32:0/4056808573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c08e3d0 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d71400700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6c17a6e0 0x7f3d6c17aae0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7f3d68009650 tx=0x7f3d6800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:27.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.016 7f3d627fc700 1 -- 172.21.15.32:0/4056808573 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d680093a0 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.017 7f3d627fc700 1 -- 172.21.15.32:0/4056808573 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6800bd50 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.017 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6c08e3d0 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.017 7f3d627fc700 1 -- 172.21.15.32:0/4056808573 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d680052e0 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.017 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c08e3d0 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.017 7f3d627fc700 1 -- 172.21.15.32:0/4056808573 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3d68006030 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.017 7f3d627fc700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d54018930 0x7f3d5401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.017 7f3d627fc700 1 -- 172.21.15.32:0/4056808573 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f3d68032760 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.017 7f3d71c01700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d54018930 0x7f3d5401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.018 7f3d71c01700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d54018930 0x7f3d5401ae10 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:27.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.037 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6c17a1b0 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.040 7f3d627fc700 1 -- 172.21.15.32:0/4056808573 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3d680009c0 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.230 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d6c17a1b0 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.231 7f3d627fc700 1 -- 172.21.15.32:0/4056808573 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+1858 (secure 0 0 0) 0x7f3d6800bd50 con 0x7f3d6c17a6e0 2021-04-09T18:25:27.298 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 263 2021-04-09T18:25:27.299 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:27.299 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":263,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8980,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3174368076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3174368076},{"type":"v1","addr":"172.21.15.86:6836","nonce":3174368076}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":263},{"gid":9297,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2843252022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2843252022},{"type":"v1","addr":"172.21.15.32:6835","nonce":2843252022}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":263},{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":57506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1430818598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1430818598},{"type":"v1","addr":"172.21.15.86:6837","nonce":1430818598}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":261}],"filesystems":[]} 2021-04-09T18:25:27.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d54018930 msgr2=0x7f3d5401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d54018930 0x7f3d5401ae10 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6c17a6e0 msgr2=0x7f3d6c17aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6c17a6e0 0x7f3d6c17aae0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7f3d68009650 tx=0x7f3d6800b960).stop 2021-04-09T18:25:27.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 shutdown_connections 2021-04-09T18:25:27.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d54018930 0x7f3d5401ae10 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6c17a6e0 0x7f3d6c17aae0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6c17b100 0x7f3d6c08db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 --2- 172.21.15.32:0/4056808573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d6c179610 0x7f3d6c179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 >> 172.21.15.32:0/4056808573 conn(0x7f3d6c190550 msgr2=0x7f3d6c1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:27.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 shutdown_connections 2021-04-09T18:25:27.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.233 7f3d73de5700 1 -- 172.21.15.32:0/4056808573 wait complete. 2021-04-09T18:25:27.310 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:27.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.392 7fe770a3a700 1 Processor -- start 2021-04-09T18:25:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.392 7fe770a3a700 1 -- start start 2021-04-09T18:25:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.392 7fe770a3a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe76c10d010 0x7fe76c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.392 7fe770a3a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe76c0d8490 0x7fe76c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.392 7fe770a3a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c10d480 con 0x7fe76c0d1da0 2021-04-09T18:25:27.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.392 7fe770a3a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76c0d8d30 con 0x7fe76c10d010 2021-04-09T18:25:27.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.392 7fe770a3a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76c0d9220 con 0x7fe76c0d8490 2021-04-09T18:25:27.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76a61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe76c10d010 0x7fe76c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76a61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe76c10d010 0x7fe76c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52296/0 (socket says 172.21.15.32:52296) 2021-04-09T18:25:27.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76a61c700 1 -- 172.21.15.32:0/1642747353 learned_addr learned my addr 172.21.15.32:0/1642747353 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:27.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76ae1d700 1 --2- 172.21.15.32:0/1642747353 >> v2:172.21.15.86:3300/0 conn(0x7fe76c0d8490 0x7fe76c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76a61c700 1 -- 172.21.15.32:0/1642747353 >> v2:172.21.15.86:3300/0 conn(0x7fe76c0d8490 msgr2=0x7fe76c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:27.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76a61c700 1 --2- 172.21.15.32:0/1642747353 >> v2:172.21.15.86:3300/0 conn(0x7fe76c0d8490 0x7fe76c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76a61c700 1 -- 172.21.15.32:0/1642747353 >> v1:172.21.15.86:6789/0 conn(0x7fe76c0d1da0 legacy=0x7fe76c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76a61c700 1 -- 172.21.15.32:0/1642747353 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c10e020 con 0x7fe76c10d010 2021-04-09T18:25:27.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76ae1d700 1 --2- 172.21.15.32:0/1642747353 >> v2:172.21.15.86:3300/0 conn(0x7fe76c0d8490 0x7fe76c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:27.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76a61c700 1 --2- 172.21.15.32:0/1642747353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c10d010 0x7fe76c0d1780 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 rx=0x7fe75c006700 tx=0x7fe75c009b80).ready entity=mon.0 client_cookie=65982e083d8d16c4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:27.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76961a700 1 -- 172.21.15.32:0/1642747353 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe75c00c0c0 con 0x7fe76c10d010 2021-04-09T18:25:27.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76961a700 1 -- 172.21.15.32:0/1642747353 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c00c440 con 0x7fe76c10d010 2021-04-09T18:25:27.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe76961a700 1 -- 172.21.15.32:0/1642747353 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe75c00c920 con 0x7fe76c10d010 2021-04-09T18:25:27.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe770a3a700 1 -- 172.21.15.32:0/1642747353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c10d010 msgr2=0x7fe76c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.393 7fe770a3a700 1 --2- 172.21.15.32:0/1642747353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c10d010 0x7fe76c0d1780 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 rx=0x7fe75c006700 tx=0x7fe75c009b80).stop 2021-04-09T18:25:27.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.394 7fe770a3a700 1 -- 172.21.15.32:0/1642747353 shutdown_connections 2021-04-09T18:25:27.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.394 7fe770a3a700 1 --2- 172.21.15.32:0/1642747353 >> v2:172.21.15.86:3300/0 conn(0x7fe76c0d8490 0x7fe76c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.394 7fe770a3a700 1 --2- 172.21.15.32:0/1642747353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c10d010 0x7fe76c0d1780 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.394 7fe770a3a700 1 -- 172.21.15.32:0/1642747353 >> 172.21.15.32:0/1642747353 conn(0x7fe76c0ce5f0 msgr2=0x7fe76c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:27.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.394 7fe770a3a700 1 -- 172.21.15.32:0/1642747353 shutdown_connections 2021-04-09T18:25:27.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.394 7fe770a3a700 1 -- 172.21.15.32:0/1642747353 wait complete. 2021-04-09T18:25:27.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.414 7fe770a3a700 1 Processor -- start 2021-04-09T18:25:27.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.414 7fe770a3a700 1 -- start start 2021-04-09T18:25:27.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe770a3a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c1ac380 0x7fe76c18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe770a3a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76c1ac780 0x7fe76c18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe770a3a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76c183520 0x7fe76c183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe770a3a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1ace10 con 0x7fe76c1ac380 2021-04-09T18:25:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe770a3a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c189cf0 con 0x7fe76c183520 2021-04-09T18:25:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe770a3a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c183f40 con 0x7fe76c1ac780 2021-04-09T18:25:27.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76a61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c1ac380 0x7fe76c18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76a61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c1ac380 0x7fe76c18a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52302/0 (socket says 172.21.15.32:52302) 2021-04-09T18:25:27.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76a61c700 1 -- 172.21.15.32:0/2990477851 learned_addr learned my addr 172.21.15.32:0/2990477851 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:27.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76ae1d700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76c183520 0x7fe76c183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe769e1b700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76c1ac780 0x7fe76c18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76a61c700 1 -- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76c1ac780 msgr2=0x7fe76c18a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76a61c700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76c1ac780 0x7fe76c18a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76a61c700 1 -- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76c183520 msgr2=0x7fe76c183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76a61c700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76c183520 0x7fe76c183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76a61c700 1 -- 172.21.15.32:0/2990477851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c184160 con 0x7fe76c1ac380 2021-04-09T18:25:27.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe76a61c700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c1ac380 0x7fe76c18a230 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 rx=0x7fe75c00b5e0 tx=0x7fe75c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.415 7fe75affd700 1 -- 172.21.15.32:0/2990477851 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe75c00bd70 con 0x7fe76c1ac380 2021-04-09T18:25:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.416 7fe75affd700 1 -- 172.21.15.32:0/2990477851 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c002030 con 0x7fe76c1ac380 2021-04-09T18:25:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.416 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c184160 con 0x7fe76c1ac380 2021-04-09T18:25:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.416 7fe75affd700 1 -- 172.21.15.32:0/2990477851 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe75c002550 con 0x7fe76c1ac380 2021-04-09T18:25:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.416 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c184160 con 0x7fe76c1ac380 2021-04-09T18:25:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.416 7fe75affd700 1 -- 172.21.15.32:0/2990477851 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe75c000c90 con 0x7fe76c1ac380 2021-04-09T18:25:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.417 7fe75affd700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7500188c0 0x7fe75001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.417 7fe75affd700 1 -- 172.21.15.32:0/2990477851 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fe75c0325c0 con 0x7fe76c1ac380 2021-04-09T18:25:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.417 7fe769e1b700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7500188c0 0x7fe75001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.417 7fe769e1b700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7500188c0 0x7fe75001adc0 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.436 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c182d10 con 0x7fe76c1ac380 2021-04-09T18:25:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.440 7fe75affd700 1 -- 172.21.15.32:0/2990477851 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe75c009fb0 con 0x7fe76c1ac380 2021-04-09T18:25:27.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.630 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe76c182d10 con 0x7fe76c1ac380 2021-04-09T18:25:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.631 7fe75affd700 1 -- 172.21.15.32:0/2990477851 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+1858 (secure 0 0 0) 0x7fe75c009fb0 con 0x7fe76c1ac380 2021-04-09T18:25:27.698 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 263 2021-04-09T18:25:27.699 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:27.699 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":263,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8980,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3174368076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3174368076},{"type":"v1","addr":"172.21.15.86:6836","nonce":3174368076}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":263},{"gid":9297,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2843252022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2843252022},{"type":"v1","addr":"172.21.15.32:6835","nonce":2843252022}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":263},{"gid":57362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/4263974733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":4263974733},{"type":"v1","addr":"172.21.15.32:6834","nonce":4263974733}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":249},{"gid":57506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1430818598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1430818598},{"type":"v1","addr":"172.21.15.86:6837","nonce":1430818598}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":261}],"filesystems":[]} 2021-04-09T18:25:27.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7500188c0 msgr2=0x7fe75001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7500188c0 0x7fe75001adc0 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c1ac380 msgr2=0x7fe76c18a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c1ac380 0x7fe76c18a230 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 rx=0x7fe75c00b5e0 tx=0x7fe75c00b980).stop 2021-04-09T18:25:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 shutdown_connections 2021-04-09T18:25:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7500188c0 0x7fe75001adc0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe76c183520 0x7fe76c183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe76c1ac780 0x7fe76c18a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 --2- 172.21.15.32:0/2990477851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe76c1ac380 0x7fe76c18a230 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 >> 172.21.15.32:0/2990477851 conn(0x7fe76c19b890 msgr2=0x7fe76c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:27.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 shutdown_connections 2021-04-09T18:25:27.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.634 7fe770a3a700 1 -- 172.21.15.32:0/2990477851 wait complete. 2021-04-09T18:25:27.711 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:25:27.711 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:25:27.712 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:25:27.712 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:25:27.712 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:25:27.712 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:25:27.713 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:25:27.713 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:25:27.713 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:25:27.714 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:25:27.714 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:25:27.714 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:25:27.715 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:25:27.715 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:25:27.715 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:25:27.716 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:25:27.720 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:25:27.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:25:27.722 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:25:27.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:25:27.724 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:25:27.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:25:27.726 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:25:27.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:25:27.763 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:25:27.764 DEBUG:teuthology.parallel:result is None 2021-04-09T18:25:27.765 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:25:27.766 DEBUG:teuthology.parallel:result is None 2021-04-09T18:25:27.767 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:25:27.767 DEBUG:teuthology.parallel:result is None 2021-04-09T18:25:27.768 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:25:27.768 DEBUG:teuthology.parallel:result is None 2021-04-09T18:25:27.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:25:27.906 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:25:27.906 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:25:27.913 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:25:27.914 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:25:27.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbf6449700 1 Processor -- start 2021-04-09T18:25:27.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbf6449700 1 -- start start 2021-04-09T18:25:27.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbf6449700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf00cffb0 0x7efbf00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbf6449700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efbf00d6c90 0x7efbf010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbf6449700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf00d7290 con 0x7efbf00d7810 2021-04-09T18:25:27.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbf6449700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf00d7090 con 0x7efbf00d6c90 2021-04-09T18:25:27.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbf6449700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00d8590 con 0x7efbf00cffb0 2021-04-09T18:25:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbe7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efbf00d7810 0x7efbf00d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33694/0 (socket says 172.21.15.32:33694) 2021-04-09T18:25:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbe7fff700 1 -- 172.21.15.32:0/3091433220 learned_addr learned my addr 172.21.15.32:0/3091433220 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbf4a66700 1 --2- 172.21.15.32:0/3091433220 >> v2:172.21.15.86:3300/0 conn(0x7efbf00d6c90 0x7efbf010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.858 7efbeffff700 1 --2- 172.21.15.32:0/3091433220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf00cffb0 0x7efbf00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbef7fe700 1 -- 172.21.15.32:0/3091433220 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909479051 0 0) 0x7efbd8000ce0 con 0x7efbf00d7810 2021-04-09T18:25:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbef7fe700 1 -- 172.21.15.32:0/3091433220 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbdc002fe0 con 0x7efbf00d7810 2021-04-09T18:25:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbf4a66700 1 -- 172.21.15.32:0/3091433220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf00cffb0 msgr2=0x7efbf00d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbf4a66700 1 --2- 172.21.15.32:0/3091433220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf00cffb0 0x7efbf00d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbf4a66700 1 -- 172.21.15.32:0/3091433220 >> v1:172.21.15.32:6789/0 conn(0x7efbf00d7810 legacy=0x7efbf00d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbf4a66700 1 -- 172.21.15.32:0/3091433220 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf00d91f0 con 0x7efbf00d6c90 2021-04-09T18:25:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbeffff700 1 --2- 172.21.15.32:0/3091433220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf00cffb0 0x7efbf00d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbf4a66700 1 --2- 172.21.15.32:0/3091433220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbf00d6c90 0x7efbf010cfe0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7efbe8006e50 tx=0x7efbe8009e70).ready entity=mon.1 client_cookie=774e008466d1b66c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbef7fe700 1 -- 172.21.15.32:0/3091433220 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbe8006a20 con 0x7efbf00d6c90 2021-04-09T18:25:27.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbef7fe700 1 -- 172.21.15.32:0/3091433220 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe8006790 con 0x7efbf00d6c90 2021-04-09T18:25:27.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbef7fe700 1 -- 172.21.15.32:0/3091433220 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbe800b700 con 0x7efbf00d6c90 2021-04-09T18:25:27.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbf6449700 1 -- 172.21.15.32:0/3091433220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbf00d6c90 msgr2=0x7efbf010cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.859 7efbf6449700 1 --2- 172.21.15.32:0/3091433220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbf00d6c90 0x7efbf010cfe0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7efbe8006e50 tx=0x7efbe8009e70).stop 2021-04-09T18:25:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.860 7efbf6449700 1 -- 172.21.15.32:0/3091433220 shutdown_connections 2021-04-09T18:25:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.860 7efbf6449700 1 --2- 172.21.15.32:0/3091433220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbf00d6c90 0x7efbf010cfe0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.860 7efbf6449700 1 --2- 172.21.15.32:0/3091433220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf00cffb0 0x7efbf00d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.860 7efbf6449700 1 -- 172.21.15.32:0/3091433220 >> 172.21.15.32:0/3091433220 conn(0x7efbf00ce5d0 msgr2=0x7efbf00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.860 7efbf6449700 1 -- 172.21.15.32:0/3091433220 shutdown_connections 2021-04-09T18:25:27.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.860 7efbf6449700 1 -- 172.21.15.32:0/3091433220 wait complete. 2021-04-09T18:25:27.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf6449700 1 Processor -- start 2021-04-09T18:25:27.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf6449700 1 -- start start 2021-04-09T18:25:27.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf6449700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf01a9a90 0x7efbf01a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf6449700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbf008aaf0 0x7efbf008aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf6449700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbf008b510 0x7efbf00924e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf6449700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf01aa4b0 con 0x7efbf008b510 2021-04-09T18:25:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf6449700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf009b590 con 0x7efbf008aaf0 2021-04-09T18:25:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf6449700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf0190430 con 0x7efbf01a9a90 2021-04-09T18:25:27.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf4a66700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbf008b510 0x7efbf00924e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf4a66700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbf008b510 0x7efbf00924e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52340/0 (socket says 172.21.15.32:52340) 2021-04-09T18:25:27.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbf4a66700 1 -- 172.21.15.32:0/1570494303 learned_addr learned my addr 172.21.15.32:0/1570494303 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:27.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbe7fff700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbf008aaf0 0x7efbf008aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.881 7efbeffff700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf01a9a90 0x7efbf01a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbf4a66700 1 -- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf01a9a90 msgr2=0x7efbf01a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbf4a66700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf01a9a90 0x7efbf01a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbf4a66700 1 -- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbf008aaf0 msgr2=0x7efbf008aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbf4a66700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbf008aaf0 0x7efbf008aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbf4a66700 1 -- 172.21.15.32:0/1570494303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0092b00 con 0x7efbf008b510 2021-04-09T18:25:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbf4a66700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbf008b510 0x7efbf00924e0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 rx=0x7efbe80099f0 tx=0x7efbe800bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbed7fa700 1 -- 172.21.15.32:0/1570494303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbe80090c0 con 0x7efbf008b510 2021-04-09T18:25:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbed7fa700 1 -- 172.21.15.32:0/1570494303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe8000df0 con 0x7efbf008b510 2021-04-09T18:25:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbf6449700 1 -- 172.21.15.32:0/1570494303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf0092b00 con 0x7efbf008b510 2021-04-09T18:25:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbed7fa700 1 -- 172.21.15.32:0/1570494303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbe8001390 con 0x7efbf008b510 2021-04-09T18:25:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.882 7efbf6449700 1 -- 172.21.15.32:0/1570494303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf0092b00 con 0x7efbf008b510 2021-04-09T18:25:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.883 7efbed7fa700 1 -- 172.21.15.32:0/1570494303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efbe80056b0 con 0x7efbf008b510 2021-04-09T18:25:27.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.883 7efbed7fa700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbd4018930 0x7efbd401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:27.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.883 7efbed7fa700 1 -- 172.21.15.32:0/1570494303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4929+0+0 (secure 0 0 0) 0x7efbe8032af0 con 0x7efbf008b510 2021-04-09T18:25:27.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.883 7efbeffff700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbd4018930 0x7efbd401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:27.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.883 7efbeffff700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbd4018930 0x7efbd401ae10 crc :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.902 7efbf6449700 1 -- 172.21.15.32:0/1570494303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf01b7120 con 0x7efbf008b510 2021-04-09T18:25:27.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:27.905 7efbed7fa700 1 -- 172.21.15.32:0/1570494303 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efbe800bdd0 con 0x7efbf008b510 2021-04-09T18:25:28.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.107 7efbf6449700 1 -- 172.21.15.32:0/1570494303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7efbf01b7120 con 0x7efbf008b510 2021-04-09T18:25:28.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.603 7efbed7fa700 1 -- 172.21.15.32:0/1570494303 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v123) v1 ==== 97+0+0 (secure 0 0 0) 0x7efbe80056b0 con 0x7efbf008b510 2021-04-09T18:25:28.687 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:25:28.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 -- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbd4018930 msgr2=0x7efbd401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:28.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbd4018930 0x7efbd401ae10 crc :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:28.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 -- 172.21.15.32:0/1570494303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbf008b510 msgr2=0x7efbf00924e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:28.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbf008b510 0x7efbf00924e0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 rx=0x7efbe80099f0 tx=0x7efbe800bc90).stop 2021-04-09T18:25:28.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 -- 172.21.15.32:0/1570494303 shutdown_connections 2021-04-09T18:25:28.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbd4018930 0x7efbd401ae10 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:28.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efbf008b510 0x7efbf00924e0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:28.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efbf008aaf0 0x7efbf008aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 --2- 172.21.15.32:0/1570494303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efbf01a9a90 0x7efbf01a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.623 7efbf6449700 1 -- 172.21.15.32:0/1570494303 >> 172.21.15.32:0/1570494303 conn(0x7efbf01ad170 msgr2=0x7efbf01b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.624 7efbf6449700 1 -- 172.21.15.32:0/1570494303 shutdown_connections 2021-04-09T18:25:28.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.624 7efbf6449700 1 -- 172.21.15.32:0/1570494303 wait complete. 2021-04-09T18:25:28.701 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:25:28.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:25:28.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.786 7f0083678700 1 Processor -- start 2021-04-09T18:25:28.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.786 7f0083678700 1 -- start start 2021-04-09T18:25:28.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0083678700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f007c0d7ca0 0x7f007c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:28.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0083678700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007c10d480 con 0x7f007c10da00 2021-04-09T18:25:28.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0083678700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007c0d86c0 con 0x7f007c10d010 2021-04-09T18:25:28.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0083678700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f007c0d9220 con 0x7f007c0d7ca0 2021-04-09T18:25:28.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0080c93700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f007c0d7ca0 0x7f007c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:28.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0081c95700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f007c10da00 0x7f007c10de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33708/0 (socket says 172.21.15.32:33708) 2021-04-09T18:25:28.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0081c95700 1 -- 172.21.15.32:0/296212160 learned_addr learned my addr 172.21.15.32:0/296212160 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:28.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0080c93700 1 -- 172.21.15.32:0/296212160 >> v1:172.21.15.86:6789/0 conn(0x7f007c10d010 legacy=0x7f007c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:28.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0080c93700 1 -- 172.21.15.32:0/296212160 >> v1:172.21.15.32:6789/0 conn(0x7f007c10da00 legacy=0x7f007c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:28.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0080c93700 1 -- 172.21.15.32:0/296212160 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c112100 con 0x7f007c0d7ca0 2021-04-09T18:25:28.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f006ffff700 1 -- 172.21.15.32:0/296212160 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903775051 0 0) 0x7f0074000d30 con 0x7f007c10da00 2021-04-09T18:25:28.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.787 7f0080c93700 1 --2- 172.21.15.32:0/296212160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c0d7ca0 0x7f007c0d80a0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 rx=0x7f0070006650 tx=0x7f0070009a00).ready entity=mon.0 client_cookie=1b4af4febf09ac5c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:28.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f006ffff700 1 -- 172.21.15.32:0/296212160 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0070009df0 con 0x7f007c0d7ca0 2021-04-09T18:25:28.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f006ffff700 1 -- 172.21.15.32:0/296212160 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00700009b0 con 0x7f007c0d7ca0 2021-04-09T18:25:28.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f006ffff700 1 -- 172.21.15.32:0/296212160 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f007000c2b0 con 0x7f007c0d7ca0 2021-04-09T18:25:28.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f0083678700 1 -- 172.21.15.32:0/296212160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c0d7ca0 msgr2=0x7f007c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:28.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f0083678700 1 --2- 172.21.15.32:0/296212160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c0d7ca0 0x7f007c0d80a0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 rx=0x7f0070006650 tx=0x7f0070009a00).stop 2021-04-09T18:25:28.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f0083678700 1 -- 172.21.15.32:0/296212160 shutdown_connections 2021-04-09T18:25:28.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f0083678700 1 --2- 172.21.15.32:0/296212160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c0d7ca0 0x7f007c0d80a0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:28.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f0083678700 1 -- 172.21.15.32:0/296212160 >> 172.21.15.32:0/296212160 conn(0x7f007c0ce5f0 msgr2=0x7f007c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:28.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f0083678700 1 -- 172.21.15.32:0/296212160 shutdown_connections 2021-04-09T18:25:28.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.788 7f0083678700 1 -- 172.21.15.32:0/296212160 wait complete. 2021-04-09T18:25:28.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.809 7f0083678700 1 Processor -- start 2021-04-09T18:25:28.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.809 7f0083678700 1 -- start start 2021-04-09T18:25:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.809 7f0083678700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f007c1a9a50 0x7f007c08a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.809 7f0083678700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f007c08afb0 0x7f007c09d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.809 7f0083678700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c09ddf0 0x7f007c183b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:28.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.809 7f0083678700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c1a9e50 con 0x7f007c09ddf0 2021-04-09T18:25:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.809 7f0083678700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c09e410 con 0x7f007c08afb0 2021-04-09T18:25:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.809 7f0083678700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c1841a0 con 0x7f007c1a9a50 2021-04-09T18:25:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.809 7f0081c95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c09ddf0 0x7f007c183b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:28.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0080c93700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f007c08afb0 0x7f007c09d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:28.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0081c95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c09ddf0 0x7f007c183b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52358/0 (socket says 172.21.15.32:52358) 2021-04-09T18:25:28.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0081c95700 1 -- 172.21.15.32:0/2327176682 learned_addr learned my addr 172.21.15.32:0/2327176682 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:28.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0081494700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f007c1a9a50 0x7f007c08a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:28.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0081c95700 1 -- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f007c1a9a50 msgr2=0x7f007c08a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:28.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0081c95700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f007c1a9a50 0x7f007c08a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:28.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0081c95700 1 -- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f007c08afb0 msgr2=0x7f007c09d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:28.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0081c95700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f007c08afb0 0x7f007c09d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:28.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0081c95700 1 -- 172.21.15.32:0/2327176682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c184380 con 0x7f007c09ddf0 2021-04-09T18:25:28.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0080c93700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f007c08afb0 0x7f007c09d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:28.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0081c95700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c09ddf0 0x7f007c183b80 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 rx=0x7f0074007f90 tx=0x7f007400abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:28.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f006dffb700 1 -- 172.21.15.32:0/2327176682 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0074000cc0 con 0x7f007c09ddf0 2021-04-09T18:25:28.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f006dffb700 1 -- 172.21.15.32:0/2327176682 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00740070d0 con 0x7f007c09ddf0 2021-04-09T18:25:28.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0083678700 1 -- 172.21.15.32:0/2327176682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007c184380 con 0x7f007c09ddf0 2021-04-09T18:25:28.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f006dffb700 1 -- 172.21.15.32:0/2327176682 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0074007520 con 0x7f007c09ddf0 2021-04-09T18:25:28.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.810 7f0083678700 1 -- 172.21.15.32:0/2327176682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007c184380 con 0x7f007c09ddf0 2021-04-09T18:25:28.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.811 7f006dffb700 1 -- 172.21.15.32:0/2327176682 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f00740072b0 con 0x7f007c09ddf0 2021-04-09T18:25:28.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.811 7f006dffb700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f005c018930 0x7f005c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:28.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.811 7f006dffb700 1 -- 172.21.15.32:0/2327176682 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f0074032a00 con 0x7f007c09ddf0 2021-04-09T18:25:28.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.811 7f0081494700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f005c018930 0x7f005c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:28.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.811 7f0081494700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f005c018930 0x7f005c01ae10 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:28.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.831 7f0083678700 1 -- 172.21.15.32:0/2327176682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007c10d480 con 0x7f007c09ddf0 2021-04-09T18:25:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:28.835 7f006dffb700 1 -- 172.21.15.32:0/2327176682 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0074007d00 con 0x7f007c09ddf0 2021-04-09T18:25:29.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.020 7f0083678700 1 -- 172.21.15.32:0/2327176682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f007c10d480 con 0x7f007c09ddf0 2021-04-09T18:25:29.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.020 7f006dffb700 1 -- 172.21.15.32:0/2327176682 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v31) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f00740070d0 con 0x7f007c09ddf0 2021-04-09T18:25:29.088 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:29.088 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:29.088 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:25:29.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.089 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:25:29.089 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:25:29.090 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:25:29.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.091 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.092 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:25:29.092 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:25:29.092 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.093 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:25:29.093 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.093 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.094 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.094 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.095 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:25:29.095 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:25:29.095 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.096 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:25:29.096 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.096 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.096 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.097 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:25:29.098 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:25:29.098 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.098 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:25:29.098 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.099 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.099 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.099 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.100 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:25:29.100 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:25:29.100 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.100 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.101 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.101 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.101 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.102 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:25:29.102 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:25:29.102 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.103 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.103 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.103 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.105 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:25:29.105 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:25:29.105 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.106 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.106 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.107 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.107 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.108 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:25:29.108 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:25:29.109 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.109 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.109 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.109 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.110 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.110 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:25:29.111 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:25:29.111 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.111 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.111 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.112 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.112 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.113 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:25:29.113 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:25:29.113 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.113 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.113 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.114 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.115 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:25:29.115 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:25:29.115 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.116 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.116 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.116 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.116 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.117 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:25:29.117 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:25:29.117 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.118 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.118 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.118 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.118 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.119 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:25:29.119 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:25:29.120 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.120 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:25:29.120 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:25:29.120 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:25:29.121 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.121 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:25:29.122 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:25:29.122 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.122 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:25:29.122 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:25:29.123 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:25:29.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.123 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:25:29.124 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:25:29.124 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.124 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:25:29.125 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:25:29.125 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:25:29.126 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.126 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.127 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:25:29.128 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:25:29.128 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.128 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:25:29.129 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:25:29.129 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.129 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:29.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:29.130 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:25:29.131 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:25:29.131 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:25:29.131 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:25:29.131 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:25:29.132 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:25:29.132 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.132 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:29.132 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:25:29.133 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:29.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 -- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f005c018930 msgr2=0x7f005c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f005c018930 0x7f005c01ae10 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 -- 172.21.15.32:0/2327176682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c09ddf0 msgr2=0x7f007c183b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c09ddf0 0x7f007c183b80 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 rx=0x7f0074007f90 tx=0x7f007400abd0).stop 2021-04-09T18:25:29.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 -- 172.21.15.32:0/2327176682 shutdown_connections 2021-04-09T18:25:29.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f005c018930 0x7f005c01ae10 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f007c09ddf0 0x7f007c183b80 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f007c08afb0 0x7f007c09d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 --2- 172.21.15.32:0/2327176682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f007c1a9a50 0x7f007c08a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 -- 172.21.15.32:0/2327176682 >> 172.21.15.32:0/2327176682 conn(0x7f007c1965c0 msgr2=0x7f007c1ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:29.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 -- 172.21.15.32:0/2327176682 shutdown_connections 2021-04-09T18:25:29.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.023 7f0083678700 1 -- 172.21.15.32:0/2327176682 wait complete. 2021-04-09T18:25:29.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:25:29.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.220 7f80e6031700 1 Processor -- start 2021-04-09T18:25:29.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.220 7f80e6031700 1 -- start start 2021-04-09T18:25:29.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80e6031700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f80e010da00 0x7f80e010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80e6031700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e010d480 con 0x7f80e010d010 2021-04-09T18:25:29.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80e6031700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e00d1f70 con 0x7f80e00d7ca0 2021-04-09T18:25:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80e6031700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e00d8d30 con 0x7f80e010da00 2021-04-09T18:25:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80df7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f80e010d010 0x7f80e00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33720/0 (socket says 172.21.15.32:33720) 2021-04-09T18:25:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80df7fe700 1 -- 172.21.15.32:0/3277953493 learned_addr learned my addr 172.21.15.32:0/3277953493 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:29.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80dffff700 1 --2- 172.21.15.32:0/3277953493 >> v2:172.21.15.86:3300/0 conn(0x7f80e010da00 0x7f80e010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80de7fc700 1 -- 172.21.15.32:0/3277953493 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268403071 0 0) 0x7f80d0001100 con 0x7f80e010d010 2021-04-09T18:25:29.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80de7fc700 1 -- 172.21.15.32:0/3277953493 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80cc002fe0 con 0x7f80e010d010 2021-04-09T18:25:29.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80de7fc700 1 -- 172.21.15.32:0/3277953493 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597383030 0 0) 0x7f80c8000c60 con 0x7f80e00d7ca0 2021-04-09T18:25:29.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80dffff700 1 -- 172.21.15.32:0/3277953493 >> v1:172.21.15.86:6789/0 conn(0x7f80e00d7ca0 legacy=0x7f80e00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80dffff700 1 -- 172.21.15.32:0/3277953493 >> v1:172.21.15.32:6789/0 conn(0x7f80e010d010 legacy=0x7f80e00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.221 7f80dffff700 1 -- 172.21.15.32:0/3277953493 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e00d9220 con 0x7f80e010da00 2021-04-09T18:25:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.222 7f80dffff700 1 --2- 172.21.15.32:0/3277953493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e010da00 0x7f80e010de00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f80d8006e50 tx=0x7f80d8009e70).ready entity=mon.1 client_cookie=6337a76a28c51c4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:29.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.222 7f80de7fc700 1 -- 172.21.15.32:0/3277953493 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d8006a20 con 0x7f80e010da00 2021-04-09T18:25:29.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.222 7f80de7fc700 1 -- 172.21.15.32:0/3277953493 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d8006790 con 0x7f80e010da00 2021-04-09T18:25:29.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.222 7f80de7fc700 1 -- 172.21.15.32:0/3277953493 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d800b700 con 0x7f80e010da00 2021-04-09T18:25:29.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.222 7f80e6031700 1 -- 172.21.15.32:0/3277953493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e010da00 msgr2=0x7f80e010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.222 7f80e6031700 1 --2- 172.21.15.32:0/3277953493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e010da00 0x7f80e010de00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f80d8006e50 tx=0x7f80d8009e70).stop 2021-04-09T18:25:29.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.223 7f80e6031700 1 -- 172.21.15.32:0/3277953493 shutdown_connections 2021-04-09T18:25:29.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.223 7f80e6031700 1 --2- 172.21.15.32:0/3277953493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e010da00 0x7f80e010de00 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.223 7f80e6031700 1 -- 172.21.15.32:0/3277953493 >> 172.21.15.32:0/3277953493 conn(0x7f80e00ce5f0 msgr2=0x7f80e00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:29.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.223 7f80e6031700 1 -- 172.21.15.32:0/3277953493 shutdown_connections 2021-04-09T18:25:29.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.223 7f80e6031700 1 -- 172.21.15.32:0/3277953493 wait complete. 2021-04-09T18:25:29.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.243 7f80e6031700 1 Processor -- start 2021-04-09T18:25:29.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.243 7f80e6031700 1 -- start start 2021-04-09T18:25:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.243 7f80e6031700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80e00a91c0 0x7f80e01b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.243 7f80e6031700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e01b85a0 0x7f80e01ab860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.243 7f80e6031700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e01abe80 0x7f80e017be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.243 7f80e6031700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e01b89a0 con 0x7f80e01abe80 2021-04-09T18:25:29.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.243 7f80e6031700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e01ac280 con 0x7f80e01b85a0 2021-04-09T18:25:29.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.243 7f80e6031700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e01b8d60 con 0x7f80e00a91c0 2021-04-09T18:25:29.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80dffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e01abe80 0x7f80e017be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80dffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e01abe80 0x7f80e017be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52370/0 (socket says 172.21.15.32:52370) 2021-04-09T18:25:29.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80dffff700 1 -- 172.21.15.32:0/2043939778 learned_addr learned my addr 172.21.15.32:0/2043939778 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:29.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80df7fe700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80e00a91c0 0x7f80e01b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80deffd700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e01b85a0 0x7f80e01ab860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80dffff700 1 -- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80e00a91c0 msgr2=0x7f80e01b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80dffff700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80e00a91c0 0x7f80e01b7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80dffff700 1 -- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e01b85a0 msgr2=0x7f80e01ab860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80dffff700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e01b85a0 0x7f80e01ab860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80dffff700 1 -- 172.21.15.32:0/2043939778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e017c480 con 0x7f80e01abe80 2021-04-09T18:25:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80dffff700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e01abe80 0x7f80e017be60 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f80d80099f0 tx=0x7f80d800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80d7fff700 1 -- 172.21.15.32:0/2043939778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d80093d0 con 0x7f80e01abe80 2021-04-09T18:25:29.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80d7fff700 1 -- 172.21.15.32:0/2043939778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d80090c0 con 0x7f80e01abe80 2021-04-09T18:25:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80e6031700 1 -- 172.21.15.32:0/2043939778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e017c480 con 0x7f80e01abe80 2021-04-09T18:25:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80d7fff700 1 -- 172.21.15.32:0/2043939778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d8001420 con 0x7f80e01abe80 2021-04-09T18:25:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.244 7f80e6031700 1 -- 172.21.15.32:0/2043939778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e017c480 con 0x7f80e01abe80 2021-04-09T18:25:29.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.245 7f80d7fff700 1 -- 172.21.15.32:0/2043939778 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f80d80011a0 con 0x7f80e01abe80 2021-04-09T18:25:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.245 7f80d7fff700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80c4018930 0x7f80c401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.245 7f80d7fff700 1 -- 172.21.15.32:0/2043939778 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f80d8032aa0 con 0x7f80e01abe80 2021-04-09T18:25:29.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.245 7f80df7fe700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80c4018930 0x7f80c401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.245 7f80df7fe700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80c4018930 0x7f80c401ae10 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:29.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.265 7f80e6031700 1 -- 172.21.15.32:0/2043939778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e01ab330 con 0x7f80e01abe80 2021-04-09T18:25:29.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.268 7f80d7fff700 1 -- 172.21.15.32:0/2043939778 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f80d8000df0 con 0x7f80e01abe80 2021-04-09T18:25:29.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.457 7f80e6031700 1 -- 172.21.15.32:0/2043939778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80e01ab330 con 0x7f80e01abe80 2021-04-09T18:25:29.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.458 7f80d7fff700 1 -- 172.21.15.32:0/2043939778 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f80d80090a0 con 0x7f80e01abe80 2021-04-09T18:25:29.525 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:29.526 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":123,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:25:28.598837","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:25:29.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 -- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80c4018930 msgr2=0x7f80c401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80c4018930 0x7f80c401ae10 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 -- 172.21.15.32:0/2043939778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e01abe80 msgr2=0x7f80e017be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e01abe80 0x7f80e017be60 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f80d80099f0 tx=0x7f80d800bd10).stop 2021-04-09T18:25:29.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 -- 172.21.15.32:0/2043939778 shutdown_connections 2021-04-09T18:25:29.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f80c4018930 0x7f80c401ae10 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80e01abe80 0x7f80e017be60 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80e01b85a0 0x7f80e01ab860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 --2- 172.21.15.32:0/2043939778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80e00a91c0 0x7f80e01b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 -- 172.21.15.32:0/2043939778 >> 172.21.15.32:0/2043939778 conn(0x7f80e0190570 msgr2=0x7f80e018ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:29.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 -- 172.21.15.32:0/2043939778 shutdown_connections 2021-04-09T18:25:29.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.460 7f80e6031700 1 -- 172.21.15.32:0/2043939778 wait complete. 2021-04-09T18:25:29.537 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:25:29.538 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:25:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabee6b700 1 Processor -- start 2021-04-09T18:25:29.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabee6b700 1 -- start start 2021-04-09T18:25:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabee6b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feab810d010 0x7feab80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabee6b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab80d1da0 0x7feab80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabee6b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feab80d82c0 0x7feab810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabee6b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab80d8d30 con 0x7feab810d010 2021-04-09T18:25:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabee6b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab80d9220 con 0x7feab80d82c0 2021-04-09T18:25:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabee6b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab810d480 con 0x7feab80d1da0 2021-04-09T18:25:29.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabcc87700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feab810d010 0x7feab80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabcc87700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7feab810d010 0x7feab80d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52374/0 (socket says 172.21.15.32:52374) 2021-04-09T18:25:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabcc87700 1 -- 172.21.15.32:0/428696815 learned_addr learned my addr 172.21.15.32:0/428696815 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feab7fff700 1 --2- 172.21.15.32:0/428696815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab80d1da0 0x7feab80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.624 7feabd488700 1 --2- 172.21.15.32:0/428696815 >> v2:172.21.15.86:3300/0 conn(0x7feab80d82c0 0x7feab810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feabcc87700 1 -- 172.21.15.32:0/428696815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab80d1da0 msgr2=0x7feab80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feabcc87700 1 --2- 172.21.15.32:0/428696815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab80d1da0 0x7feab80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feabcc87700 1 -- 172.21.15.32:0/428696815 >> v2:172.21.15.86:3300/0 conn(0x7feab80d82c0 msgr2=0x7feab810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:29.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feabcc87700 1 --2- 172.21.15.32:0/428696815 >> v2:172.21.15.86:3300/0 conn(0x7feab80d82c0 0x7feab810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feabcc87700 1 -- 172.21.15.32:0/428696815 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80d86c0 con 0x7feab810d010 2021-04-09T18:25:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feab7fff700 1 --2- 172.21.15.32:0/428696815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab80d1da0 0x7feab80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:29.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feabd488700 1 --2- 172.21.15.32:0/428696815 >> v2:172.21.15.86:3300/0 conn(0x7feab80d82c0 0x7feab810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feabcc87700 1 --2- 172.21.15.32:0/428696815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab810d010 0x7feab80d1780 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 rx=0x7feab0006a80 tx=0x7feab0009f00).ready entity=mon.0 client_cookie=655bdacc4d85532e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feab77fe700 1 -- 172.21.15.32:0/428696815 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feab000b5c0 con 0x7feab810d010 2021-04-09T18:25:29.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feab77fe700 1 -- 172.21.15.32:0/428696815 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab000b8c0 con 0x7feab810d010 2021-04-09T18:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feab77fe700 1 -- 172.21.15.32:0/428696815 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feab000bda0 con 0x7feab810d010 2021-04-09T18:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feabee6b700 1 -- 172.21.15.32:0/428696815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab810d010 msgr2=0x7feab80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.625 7feabee6b700 1 --2- 172.21.15.32:0/428696815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab810d010 0x7feab80d1780 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 rx=0x7feab0006a80 tx=0x7feab0009f00).stop 2021-04-09T18:25:29.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.626 7feabee6b700 1 -- 172.21.15.32:0/428696815 shutdown_connections 2021-04-09T18:25:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.626 7feabee6b700 1 --2- 172.21.15.32:0/428696815 >> v2:172.21.15.86:3300/0 conn(0x7feab80d82c0 0x7feab810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.626 7feabee6b700 1 --2- 172.21.15.32:0/428696815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab80d1da0 0x7feab80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.626 7feabee6b700 1 --2- 172.21.15.32:0/428696815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab810d010 0x7feab80d1780 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.626 7feabee6b700 1 -- 172.21.15.32:0/428696815 >> 172.21.15.32:0/428696815 conn(0x7feab80ce5f0 msgr2=0x7feab80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.626 7feabee6b700 1 -- 172.21.15.32:0/428696815 shutdown_connections 2021-04-09T18:25:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.626 7feabee6b700 1 -- 172.21.15.32:0/428696815 wait complete. 2021-04-09T18:25:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.646 7feabee6b700 1 Processor -- start 2021-04-09T18:25:29.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.646 7feabee6b700 1 -- start start 2021-04-09T18:25:29.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feabee6b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feab80a0a10 0x7feab80bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feabee6b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab818ad00 0x7feab819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feabee6b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab81a03f0 0x7feab8095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feabee6b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab818b100 con 0x7feab818ad00 2021-04-09T18:25:29.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feabee6b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab8095860 con 0x7feab80a0a10 2021-04-09T18:25:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feabee6b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab8095a80 con 0x7feab81a03f0 2021-04-09T18:25:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feab7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab818ad00 0x7feab819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feab7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab818ad00 0x7feab819fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52382/0 (socket says 172.21.15.32:52382) 2021-04-09T18:25:29.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feab7fff700 1 -- 172.21.15.32:0/1097039688 learned_addr learned my addr 172.21.15.32:0/1097039688 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feabd488700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab81a03f0 0x7feab8095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feabcc87700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feab80a0a10 0x7feab80bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feab7fff700 1 -- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab81a03f0 msgr2=0x7feab8095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feab7fff700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab81a03f0 0x7feab8095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feab7fff700 1 -- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feab80a0a10 msgr2=0x7feab80bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feab7fff700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feab80a0a10 0x7feab80bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feab7fff700 1 -- 172.21.15.32:0/1097039688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab8189250 con 0x7feab818ad00 2021-04-09T18:25:29.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.647 7feab7fff700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab818ad00 0x7feab819fdd0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 rx=0x7feaac00b580 tx=0x7feaac00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.648 7feab57fa700 1 -- 172.21.15.32:0/1097039688 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaac00bd10 con 0x7feab818ad00 2021-04-09T18:25:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.648 7feab57fa700 1 -- 172.21.15.32:0/1097039688 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac000c90 con 0x7feab818ad00 2021-04-09T18:25:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.648 7feab57fa700 1 -- 172.21.15.32:0/1097039688 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaac002220 con 0x7feab818ad00 2021-04-09T18:25:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.648 7feabee6b700 1 -- 172.21.15.32:0/1097039688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab8189250 con 0x7feab818ad00 2021-04-09T18:25:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.648 7feabee6b700 1 -- 172.21.15.32:0/1097039688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab8189250 con 0x7feab818ad00 2021-04-09T18:25:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.648 7feab57fa700 1 -- 172.21.15.32:0/1097039688 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7feaac006b00 con 0x7feab818ad00 2021-04-09T18:25:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.648 7feab57fa700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea980188d0 0x7fea9801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.648 7feab57fa700 1 -- 172.21.15.32:0/1097039688 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4757+0+0 (secure 0 0 0) 0x7feaac0326c0 con 0x7feab818ad00 2021-04-09T18:25:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.649 7feabcc87700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea980188d0 0x7fea9801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.649 7feabcc87700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea980188d0 0x7fea9801add0 crc :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:29.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.668 7feabee6b700 1 -- 172.21.15.32:0/1097039688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab8093d50 con 0x7feab818ad00 2021-04-09T18:25:29.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.672 7feab57fa700 1 -- 172.21.15.32:0/1097039688 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7feaac00bd10 con 0x7feab818ad00 2021-04-09T18:25:29.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.854 7feabee6b700 1 -- 172.21.15.32:0/1097039688 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7feab810d480 con 0x7feab818ad00 2021-04-09T18:25:29.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.854 7feab57fa700 1 -- 172.21.15.32:0/1097039688 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v123) v1 ==== 92+0+10 (secure 0 0 0) 0x7feaac000c90 con 0x7feab818ad00 2021-04-09T18:25:29.922 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 -- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea980188d0 msgr2=0x7fea9801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea980188d0 0x7fea9801add0 crc :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 -- 172.21.15.32:0/1097039688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab818ad00 msgr2=0x7feab819fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab818ad00 0x7feab819fdd0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 rx=0x7feaac00b580 tx=0x7feaac00b920).stop 2021-04-09T18:25:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 -- 172.21.15.32:0/1097039688 shutdown_connections 2021-04-09T18:25:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fea980188d0 0x7fea9801add0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feab818ad00 0x7feab819fdd0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab81a03f0 0x7feab8095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 --2- 172.21.15.32:0/1097039688 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feab80a0a10 0x7feab80bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 -- 172.21.15.32:0/1097039688 >> 172.21.15.32:0/1097039688 conn(0x7feab80975f0 msgr2=0x7feab8096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 -- 172.21.15.32:0/1097039688 shutdown_connections 2021-04-09T18:25:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:29.857 7feabee6b700 1 -- 172.21.15.32:0/1097039688 wait complete. 2021-04-09T18:25:29.935 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:25:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.018 7f70296f0700 1 Processor -- start 2021-04-09T18:25:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.018 7f70296f0700 1 -- start start 2021-04-09T18:25:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.018 7f70296f0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f702410d010 0x7f70240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.018 7f70296f0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240d8490 0x7f702410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.018 7f70296f0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702410d480 con 0x7f70240d1da0 2021-04-09T18:25:30.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.018 7f70296f0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70240d8d30 con 0x7f702410d010 2021-04-09T18:25:30.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.018 7f70296f0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70240d9220 con 0x7f70240d8490 2021-04-09T18:25:30.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70232bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f702410d010 0x7f70240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f7022abe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f70240d1da0 0x7f70240d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33750/0 (socket says 172.21.15.32:33750) 2021-04-09T18:25:30.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f7022abe700 1 -- 172.21.15.32:0/4250691425 learned_addr learned my addr 172.21.15.32:0/4250691425 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:30.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f7023ac0700 1 --2- 172.21.15.32:0/4250691425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240d8490 0x7f702410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70232bf700 1 -- 172.21.15.32:0/4250691425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240d8490 msgr2=0x7f702410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:30.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70232bf700 1 --2- 172.21.15.32:0/4250691425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240d8490 0x7f702410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70232bf700 1 -- 172.21.15.32:0/4250691425 >> v1:172.21.15.32:6789/0 conn(0x7f70240d1da0 legacy=0x7f70240d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70222bd700 1 -- 172.21.15.32:0/4250691425 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273252493 0 0) 0x7f7018000ce0 con 0x7f70240d1da0 2021-04-09T18:25:30.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70232bf700 1 -- 172.21.15.32:0/4250691425 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702410e020 con 0x7f702410d010 2021-04-09T18:25:30.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70232bf700 1 --2- 172.21.15.32:0/4250691425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702410d010 0x7f70240d1780 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 rx=0x7f7014006650 tx=0x7f7014009a00).ready entity=mon.0 client_cookie=80b7865e1ed140bf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:30.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70222bd700 1 -- 172.21.15.32:0/4250691425 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7014009df0 con 0x7f702410d010 2021-04-09T18:25:30.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70222bd700 1 -- 172.21.15.32:0/4250691425 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70140009b0 con 0x7f702410d010 2021-04-09T18:25:30.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70222bd700 1 -- 172.21.15.32:0/4250691425 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7014009df0 con 0x7f702410d010 2021-04-09T18:25:30.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70296f0700 1 -- 172.21.15.32:0/4250691425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702410d010 msgr2=0x7f70240d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.019 7f70296f0700 1 --2- 172.21.15.32:0/4250691425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702410d010 0x7f70240d1780 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 rx=0x7f7014006650 tx=0x7f7014009a00).stop 2021-04-09T18:25:30.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.020 7f70296f0700 1 -- 172.21.15.32:0/4250691425 shutdown_connections 2021-04-09T18:25:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.020 7f70296f0700 1 --2- 172.21.15.32:0/4250691425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240d8490 0x7f702410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.020 7f70296f0700 1 --2- 172.21.15.32:0/4250691425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702410d010 0x7f70240d1780 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.020 7f70296f0700 1 -- 172.21.15.32:0/4250691425 >> 172.21.15.32:0/4250691425 conn(0x7f70240ce5f0 msgr2=0x7f70240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:30.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.020 7f70296f0700 1 -- 172.21.15.32:0/4250691425 shutdown_connections 2021-04-09T18:25:30.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.020 7f70296f0700 1 -- 172.21.15.32:0/4250691425 wait complete. 2021-04-09T18:25:30.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.040 7f70296f0700 1 Processor -- start 2021-04-09T18:25:30.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.040 7f70296f0700 1 -- start start 2021-04-09T18:25:30.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f70296f0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7024179bf0 0x7f70240abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f70296f0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240ac400 0x7f70241a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f70296f0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702417a110 0x7f70241a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f70296f0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702417a510 con 0x7f702417a110 2021-04-09T18:25:30.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f70296f0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70241a68b0 con 0x7f7024179bf0 2021-04-09T18:25:30.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f70296f0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7024182e30 con 0x7f70240ac400 2021-04-09T18:25:30.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7023ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702417a110 0x7f70241a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7023ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702417a110 0x7f70241a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52398/0 (socket says 172.21.15.32:52398) 2021-04-09T18:25:30.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7023ac0700 1 -- 172.21.15.32:0/917257652 learned_addr learned my addr 172.21.15.32:0/917257652 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:30.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7022abe700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240ac400 0x7f70241a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f70232bf700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7024179bf0 0x7f70240abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7023ac0700 1 -- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240ac400 msgr2=0x7f70241a6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7023ac0700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240ac400 0x7f70241a6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7023ac0700 1 -- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7024179bf0 msgr2=0x7f70240abde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7023ac0700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7024179bf0 0x7f70240abde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7023ac0700 1 -- 172.21.15.32:0/917257652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70241a56c0 con 0x7f702417a110 2021-04-09T18:25:30.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f70232bf700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7024179bf0 0x7f70240abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:30.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f7023ac0700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702417a110 0x7f70241a7450 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 rx=0x7f701c006f40 tx=0x7f701c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:30.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f700bfff700 1 -- 172.21.15.32:0/917257652 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f701c0011f0 con 0x7f702417a110 2021-04-09T18:25:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.041 7f700bfff700 1 -- 172.21.15.32:0/917257652 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701c003030 con 0x7f702417a110 2021-04-09T18:25:30.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.042 7f70296f0700 1 -- 172.21.15.32:0/917257652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70241a56c0 con 0x7f702417a110 2021-04-09T18:25:30.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.042 7f700bfff700 1 -- 172.21.15.32:0/917257652 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f701c0016a0 con 0x7f702417a110 2021-04-09T18:25:30.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.042 7f70296f0700 1 -- 172.21.15.32:0/917257652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70241a56c0 con 0x7f702417a110 2021-04-09T18:25:30.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.042 7f700bfff700 1 -- 172.21.15.32:0/917257652 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f701c001470 con 0x7f702417a110 2021-04-09T18:25:30.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.042 7f700bfff700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7004018900 0x7f700401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.042 7f700bfff700 1 -- 172.21.15.32:0/917257652 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f701c032aa0 con 0x7f702417a110 2021-04-09T18:25:30.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.042 7f70232bf700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7004018900 0x7f700401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.043 7f70232bf700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7004018900 0x7f700401adc0 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:30.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.062 7f70296f0700 1 -- 172.21.15.32:0/917257652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702417a510 con 0x7f702417a110 2021-04-09T18:25:30.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.066 7f700bfff700 1 -- 172.21.15.32:0/917257652 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f701c000f40 con 0x7f702417a110 2021-04-09T18:25:30.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.245 7f70296f0700 1 -- 172.21.15.32:0/917257652 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f702417a510 con 0x7f702417a110 2021-04-09T18:25:30.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.245 7f700bfff700 1 -- 172.21.15.32:0/917257652 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v123) v1 ==== 84+0+64 (secure 0 0 0) 0x7f701c0013f0 con 0x7f702417a110 2021-04-09T18:25:30.313 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:30.313 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:25:30.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:25:30.314 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:25:30.314 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:25:30.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:30.314 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:25:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 -- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7004018900 msgr2=0x7f700401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7004018900 0x7f700401adc0 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 -- 172.21.15.32:0/917257652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702417a110 msgr2=0x7f70241a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702417a110 0x7f70241a7450 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 rx=0x7f701c006f40 tx=0x7f701c00bd40).stop 2021-04-09T18:25:30.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 -- 172.21.15.32:0/917257652 shutdown_connections 2021-04-09T18:25:30.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7004018900 0x7f700401adc0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f702417a110 0x7f70241a7450 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70240ac400 0x7f70241a6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 --2- 172.21.15.32:0/917257652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7024179bf0 0x7f70240abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 -- 172.21.15.32:0/917257652 >> 172.21.15.32:0/917257652 conn(0x7f70241acae0 msgr2=0x7f702417ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:30.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 -- 172.21.15.32:0/917257652 shutdown_connections 2021-04-09T18:25:30.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.248 7f70296f0700 1 -- 172.21.15.32:0/917257652 wait complete. 2021-04-09T18:25:30.325 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:25:30.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:25:30.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.410 7f551dfc1700 1 Processor -- start 2021-04-09T18:25:30.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.410 7f551dfc1700 1 -- start start 2021-04-09T18:25:30.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f551dfc1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180d7ca0 0x7f55180d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f551dfc1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f551810da00 0x7f551810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f551dfc1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551810d480 con 0x7f551810d010 2021-04-09T18:25:30.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f551dfc1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55180d9220 con 0x7f551810da00 2021-04-09T18:25:30.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f551dfc1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5518112100 con 0x7f55180d7ca0 2021-04-09T18:25:30.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f5517fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f551810da00 0x7f551810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f55177fe700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f551810d010 0x7f55180d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:59918/0 (socket says 172.21.15.32:59918) 2021-04-09T18:25:30.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f55177fe700 1 -- 172.21.15.32:0/4216370676 learned_addr learned my addr 172.21.15.32:0/4216370676 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:30.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f5516ffd700 1 --2- 172.21.15.32:0/4216370676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180d7ca0 0x7f55180d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f5517fff700 1 -- 172.21.15.32:0/4216370676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180d7ca0 msgr2=0x7f55180d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:30.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.411 7f5517fff700 1 --2- 172.21.15.32:0/4216370676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180d7ca0 0x7f55180d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.412 7f5517fff700 1 -- 172.21.15.32:0/4216370676 >> v1:172.21.15.86:6789/0 conn(0x7f551810d010 legacy=0x7f55180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.412 7f55167fc700 1 -- 172.21.15.32:0/4216370676 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319909073 0 0) 0x7f55080010c0 con 0x7f551810d010 2021-04-09T18:25:30.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.412 7f5517fff700 1 -- 172.21.15.32:0/4216370676 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55180d86c0 con 0x7f551810da00 2021-04-09T18:25:30.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.412 7f5517fff700 1 --2- 172.21.15.32:0/4216370676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f551810da00 0x7f551810de00 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 rx=0x7f5510006e30 tx=0x7f5510009e60).ready entity=mon.0 client_cookie=215c123341ee0142 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:30.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.412 7f55167fc700 1 -- 172.21.15.32:0/4216370676 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5510006a50 con 0x7f551810da00 2021-04-09T18:25:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.412 7f55167fc700 1 -- 172.21.15.32:0/4216370676 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551000b300 con 0x7f551810da00 2021-04-09T18:25:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.412 7f55167fc700 1 -- 172.21.15.32:0/4216370676 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5510000f90 con 0x7f551810da00 2021-04-09T18:25:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.412 7f551dfc1700 1 -- 172.21.15.32:0/4216370676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f551810da00 msgr2=0x7f551810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.412 7f551dfc1700 1 --2- 172.21.15.32:0/4216370676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f551810da00 0x7f551810de00 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 rx=0x7f5510006e30 tx=0x7f5510009e60).stop 2021-04-09T18:25:30.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.413 7f551dfc1700 1 -- 172.21.15.32:0/4216370676 shutdown_connections 2021-04-09T18:25:30.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.413 7f551dfc1700 1 --2- 172.21.15.32:0/4216370676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f551810da00 0x7f551810de00 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.413 7f551dfc1700 1 --2- 172.21.15.32:0/4216370676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180d7ca0 0x7f55180d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.413 7f551dfc1700 1 -- 172.21.15.32:0/4216370676 >> 172.21.15.32:0/4216370676 conn(0x7f55180ce5f0 msgr2=0x7f55180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:30.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.413 7f551dfc1700 1 -- 172.21.15.32:0/4216370676 shutdown_connections 2021-04-09T18:25:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.413 7f551dfc1700 1 -- 172.21.15.32:0/4216370676 wait complete. 2021-04-09T18:25:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.433 7f551dfc1700 1 Processor -- start 2021-04-09T18:25:30.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.433 7f551dfc1700 1 -- start start 2021-04-09T18:25:30.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.433 7f551dfc1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180b2140 0x7f551808b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.433 7f551dfc1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f551809f550 0x7f551809f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.433 7f551dfc1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5518197c50 0x7f5518198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.433 7f551dfc1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551809f010 con 0x7f5518197c50 2021-04-09T18:25:30.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.433 7f551dfc1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551809ff70 con 0x7f551809f550 2021-04-09T18:25:30.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.433 7f551dfc1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5518198670 con 0x7f55180b2140 2021-04-09T18:25:30.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5517fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5518197c50 0x7f5518198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5517fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5518197c50 0x7f5518198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52408/0 (socket says 172.21.15.32:52408) 2021-04-09T18:25:30.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5517fff700 1 -- 172.21.15.32:0/236757422 learned_addr learned my addr 172.21.15.32:0/236757422 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:30.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f55177fe700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180b2140 0x7f551808b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5516ffd700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f551809f550 0x7f551809f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5517fff700 1 -- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180b2140 msgr2=0x7f551808b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5517fff700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180b2140 0x7f551808b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5517fff700 1 -- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f551809f550 msgr2=0x7f551809f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:30.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5517fff700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f551809f550 0x7f551809f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:30.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5517fff700 1 -- 172.21.15.32:0/236757422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5518198890 con 0x7f5518197c50 2021-04-09T18:25:30.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f55177fe700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180b2140 0x7f551808b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:30.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f5517fff700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5518197c50 0x7f5518198050 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7f5510009fc0 tx=0x7f551000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:30.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f54fffff700 1 -- 172.21.15.32:0/236757422 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5510003080 con 0x7f5518197c50 2021-04-09T18:25:30.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f54fffff700 1 -- 172.21.15.32:0/236757422 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5510000f60 con 0x7f5518197c50 2021-04-09T18:25:30.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f551dfc1700 1 -- 172.21.15.32:0/236757422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5518198890 con 0x7f5518197c50 2021-04-09T18:25:30.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f54fffff700 1 -- 172.21.15.32:0/236757422 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f551000c4e0 con 0x7f5518197c50 2021-04-09T18:25:30.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.434 7f551dfc1700 1 -- 172.21.15.32:0/236757422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5518198890 con 0x7f5518197c50 2021-04-09T18:25:30.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.435 7f54fffff700 1 -- 172.21.15.32:0/236757422 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5510001270 con 0x7f5518197c50 2021-04-09T18:25:30.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.435 7f54fffff700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54f8018930 0x7f54f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:30.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.435 7f54fffff700 1 -- 172.21.15.32:0/236757422 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f5510032bc0 con 0x7f5518197c50 2021-04-09T18:25:30.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.435 7f55177fe700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54f8018930 0x7f54f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:30.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.436 7f55177fe700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54f8018930 0x7f54f801ae10 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:30.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.455 7f551dfc1700 1 -- 172.21.15.32:0/236757422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551809f010 con 0x7f5518197c50 2021-04-09T18:25:30.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.459 7f54fffff700 1 -- 172.21.15.32:0/236757422 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5510000d30 con 0x7f5518197c50 2021-04-09T18:25:30.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:30.642 7f551dfc1700 1 -- 172.21.15.32:0/236757422 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f551809f010 con 0x7f5518197c50 2021-04-09T18:25:31.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.614 7f54fffff700 1 -- 172.21.15.32:0/236757422 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v124) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5510001270 con 0x7f5518197c50 2021-04-09T18:25:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:25:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 -- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54f8018930 msgr2=0x7f54f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54f8018930 0x7f54f801ae10 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 -- 172.21.15.32:0/236757422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5518197c50 msgr2=0x7f5518198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5518197c50 0x7f5518198050 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7f5510009fc0 tx=0x7f551000bd10).stop 2021-04-09T18:25:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 -- 172.21.15.32:0/236757422 shutdown_connections 2021-04-09T18:25:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54f8018930 0x7f54f801ae10 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5518197c50 0x7f5518198050 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f551809f550 0x7f551809f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 --2- 172.21.15.32:0/236757422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55180b2140 0x7f551808b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 -- 172.21.15.32:0/236757422 >> 172.21.15.32:0/236757422 conn(0x7f55180b27e0 msgr2=0x7f55181b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:31.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 -- 172.21.15.32:0/236757422 shutdown_connections 2021-04-09T18:25:31.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.661 7f551dfc1700 1 -- 172.21.15.32:0/236757422 wait complete. 2021-04-09T18:25:31.739 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:25:31.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.828 7f2bd7240700 1 Processor -- start 2021-04-09T18:25:31.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.828 7f2bd7240700 1 -- start start 2021-04-09T18:25:31.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bd7240700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2bd00d7ca0 0x7f2bd00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:31.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bd7240700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2bd010da00 0x7f2bd010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:31.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bd7240700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd010d480 con 0x7f2bd010d010 2021-04-09T18:25:31.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bd7240700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd00d9220 con 0x7f2bd00d7ca0 2021-04-09T18:25:31.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bd7240700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd00d86c0 con 0x7f2bd010da00 2021-04-09T18:25:31.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bd505c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2bd010d010 0x7f2bd00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33774/0 (socket says 172.21.15.32:33774) 2021-04-09T18:25:31.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bd485b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2bd00d7ca0 0x7f2bd00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:31.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bd505c700 1 -- 172.21.15.32:0/1520883720 learned_addr learned my addr 172.21.15.32:0/1520883720 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:31.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bd585d700 1 --2- 172.21.15.32:0/1520883720 >> v2:172.21.15.86:3300/0 conn(0x7f2bd010da00 0x7f2bd010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bc7fff700 1 -- 172.21.15.32:0/1520883720 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046612283 0 0) 0x7f2bcc000ce0 con 0x7f2bd010d010 2021-04-09T18:25:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.829 7f2bc7fff700 1 -- 172.21.15.32:0/1520883720 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bbc002fe0 con 0x7f2bd010d010 2021-04-09T18:25:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bd485b700 1 -- 172.21.15.32:0/1520883720 >> v2:172.21.15.86:3300/0 conn(0x7f2bd010da00 msgr2=0x7f2bd010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bd485b700 1 --2- 172.21.15.32:0/1520883720 >> v2:172.21.15.86:3300/0 conn(0x7f2bd010da00 0x7f2bd010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bd485b700 1 -- 172.21.15.32:0/1520883720 >> v1:172.21.15.32:6789/0 conn(0x7f2bd010d010 legacy=0x7f2bd00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bd485b700 1 -- 172.21.15.32:0/1520883720 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd00d1f70 con 0x7f2bd00d7ca0 2021-04-09T18:25:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bd485b700 1 --2- 172.21.15.32:0/1520883720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd00d7ca0 0x7f2bd00d80a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 rx=0x7f2bc0006650 tx=0x7f2bc0009a00).ready entity=mon.0 client_cookie=152c3c73c1b92ce7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bc7fff700 1 -- 172.21.15.32:0/1520883720 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc0009df0 con 0x7f2bd00d7ca0 2021-04-09T18:25:31.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bc7fff700 1 -- 172.21.15.32:0/1520883720 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc00009b0 con 0x7f2bd00d7ca0 2021-04-09T18:25:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bc7fff700 1 -- 172.21.15.32:0/1520883720 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc000c2b0 con 0x7f2bd00d7ca0 2021-04-09T18:25:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bd7240700 1 -- 172.21.15.32:0/1520883720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd00d7ca0 msgr2=0x7f2bd00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:31.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.830 7f2bd7240700 1 --2- 172.21.15.32:0/1520883720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd00d7ca0 0x7f2bd00d80a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 rx=0x7f2bc0006650 tx=0x7f2bc0009a00).stop 2021-04-09T18:25:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.831 7f2bd7240700 1 -- 172.21.15.32:0/1520883720 shutdown_connections 2021-04-09T18:25:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.831 7f2bd7240700 1 --2- 172.21.15.32:0/1520883720 >> v2:172.21.15.86:3300/0 conn(0x7f2bd010da00 0x7f2bd010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.831 7f2bd7240700 1 --2- 172.21.15.32:0/1520883720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd00d7ca0 0x7f2bd00d80a0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.831 7f2bd7240700 1 -- 172.21.15.32:0/1520883720 >> 172.21.15.32:0/1520883720 conn(0x7f2bd00ce5f0 msgr2=0x7f2bd00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.831 7f2bd7240700 1 -- 172.21.15.32:0/1520883720 shutdown_connections 2021-04-09T18:25:31.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.831 7f2bd7240700 1 -- 172.21.15.32:0/1520883720 wait complete. 2021-04-09T18:25:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.854 7f2bd7240700 1 Processor -- start 2021-04-09T18:25:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.854 7f2bd7240700 1 -- start start 2021-04-09T18:25:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.854 7f2bd7240700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bd00a0a10 0x7f2bd019fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:31.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.854 7f2bd7240700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bd01a0410 0x7f2bd0095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:31.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.854 7f2bd7240700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd0095860 0x7f2bd00bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:31.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.854 7f2bd7240700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00c0510 con 0x7f2bd0095860 2021-04-09T18:25:31.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.854 7f2bd7240700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00c3f30 con 0x7f2bd01a0410 2021-04-09T18:25:31.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.854 7f2bd7240700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00a0e10 con 0x7f2bd00a0a10 2021-04-09T18:25:31.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd585d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd0095860 0x7f2bd00bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:31.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd485b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bd01a0410 0x7f2bd0095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:31.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd505c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bd00a0a10 0x7f2bd019fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:31.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd585d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd0095860 0x7f2bd00bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52426/0 (socket says 172.21.15.32:52426) 2021-04-09T18:25:31.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd585d700 1 -- 172.21.15.32:0/1633311759 learned_addr learned my addr 172.21.15.32:0/1633311759 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:31.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd585d700 1 -- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bd00a0a10 msgr2=0x7f2bd019fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:31.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd585d700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bd00a0a10 0x7f2bd019fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd585d700 1 -- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bd01a0410 msgr2=0x7f2bd0095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd585d700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bd01a0410 0x7f2bd0095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd585d700 1 -- 172.21.15.32:0/1633311759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd00bf1a0 con 0x7f2bd0095860 2021-04-09T18:25:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.855 7f2bd585d700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd0095860 0x7f2bd00bf870 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7f2bc800b970 tx=0x7f2bc800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:31.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.857 7f2bc5ffb700 1 -- 172.21.15.32:0/1633311759 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc8000f90 con 0x7f2bd0095860 2021-04-09T18:25:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.857 7f2bc5ffb700 1 -- 172.21.15.32:0/1633311759 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc8006aa0 con 0x7f2bd0095860 2021-04-09T18:25:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.857 7f2bc5ffb700 1 -- 172.21.15.32:0/1633311759 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc8001680 con 0x7f2bd0095860 2021-04-09T18:25:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.857 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd00bf1a0 con 0x7f2bd0095860 2021-04-09T18:25:31.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.857 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd00bf1a0 con 0x7f2bd0095860 2021-04-09T18:25:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.858 7f2bc5ffb700 1 -- 172.21.15.32:0/1633311759 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2bc8000df0 con 0x7f2bd0095860 2021-04-09T18:25:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.858 7f2bc5ffb700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2bb00188d0 0x7f2bb001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.858 7f2bc5ffb700 1 -- 172.21.15.32:0/1633311759 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5111+0+0 (secure 0 0 0) 0x7f2bc8032ba0 con 0x7f2bd0095860 2021-04-09T18:25:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.859 7f2bd505c700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2bb00188d0 0x7f2bb001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:31.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.859 7f2bd505c700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2bb00188d0 0x7f2bb001add0 crc :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:31.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.873 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd00c3f30 con 0x7f2bd0095860 2021-04-09T18:25:31.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:31.877 7f2bc5ffb700 1 -- 172.21.15.32:0/1633311759 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2bc80066f0 con 0x7f2bd0095860 2021-04-09T18:25:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.059 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f2bd00c3f30 con 0x7f2bd0095860 2021-04-09T18:25:32.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.862 7f2bc5ffb700 1 -- 172.21.15.32:0/1633311759 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v129) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2bc8000df0 con 0x7f2bd0095860 2021-04-09T18:25:32.941 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:25:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2bb00188d0 msgr2=0x7f2bb001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2bb00188d0 0x7f2bb001add0 crc :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd0095860 msgr2=0x7f2bd00bf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd0095860 0x7f2bd00bf870 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7f2bc800b970 tx=0x7f2bc800bd10).stop 2021-04-09T18:25:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 shutdown_connections 2021-04-09T18:25:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2bb00188d0 0x7f2bb001add0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bd0095860 0x7f2bd00bf870 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bd01a0410 0x7f2bd0095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 --2- 172.21.15.32:0/1633311759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bd00a0a10 0x7f2bd019fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 >> 172.21.15.32:0/1633311759 conn(0x7f2bd00975f0 msgr2=0x7f2bd0096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 shutdown_connections 2021-04-09T18:25:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:32.877 7f2bd7240700 1 -- 172.21.15.32:0/1633311759 wait complete. 2021-04-09T18:25:32.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:25:33.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.057 7f1a707e5700 1 Processor -- start 2021-04-09T18:25:33.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.057 7f1a707e5700 1 -- start start 2021-04-09T18:25:33.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.058 7f1a707e5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a6810d020 0x7f1a680d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.058 7f1a707e5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a680d1db0 0x7f1a680d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.058 7f1a707e5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6810d490 con 0x7f1a680d82d0 2021-04-09T18:25:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.058 7f1a707e5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a680d9230 con 0x7f1a6810d020 2021-04-09T18:25:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.058 7f1a707e5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a680d86d0 con 0x7f1a680d1db0 2021-04-09T18:25:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.058 7f1a6de00700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a680d1db0 0x7f1a680d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.058 7f1a6de00700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a680d1db0 0x7f1a680d7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:59660/0 (socket says 172.21.15.32:59660) 2021-04-09T18:25:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.058 7f1a6de00700 1 -- 172.21.15.32:0/3665892753 learned_addr learned my addr 172.21.15.32:0/3665892753 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.058 7f1a6e601700 1 --2- 172.21.15.32:0/3665892753 >> v2:172.21.15.32:3300/0 conn(0x7f1a6810d020 0x7f1a680d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a6de00700 1 -- 172.21.15.32:0/3665892753 >> v2:172.21.15.32:3300/0 conn(0x7f1a6810d020 msgr2=0x7f1a680d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a6de00700 1 --2- 172.21.15.32:0/3665892753 >> v2:172.21.15.32:3300/0 conn(0x7f1a6810d020 0x7f1a680d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a6de00700 1 -- 172.21.15.32:0/3665892753 >> v1:172.21.15.86:6789/0 conn(0x7f1a680d82d0 legacy=0x7f1a6810da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a6de00700 1 -- 172.21.15.32:0/3665892753 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6810e200 con 0x7f1a680d1db0 2021-04-09T18:25:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a6d5ff700 1 -- 172.21.15.32:0/3665892753 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646418377 0 0) 0x7f1a60000d30 con 0x7f1a680d82d0 2021-04-09T18:25:33.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a6de00700 1 --2- 172.21.15.32:0/3665892753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a680d1db0 0x7f1a680d7cb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f1a58006700 tx=0x7f1a58009b00).ready entity=mon.2 client_cookie=d2a9bb74400035c1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a6d5ff700 1 -- 172.21.15.32:0/3665892753 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a5800c0c0 con 0x7f1a680d1db0 2021-04-09T18:25:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a6d5ff700 1 -- 172.21.15.32:0/3665892753 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5800c400 con 0x7f1a680d1db0 2021-04-09T18:25:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a6d5ff700 1 -- 172.21.15.32:0/3665892753 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a5800c8e0 con 0x7f1a680d1db0 2021-04-09T18:25:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a707e5700 1 -- 172.21.15.32:0/3665892753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a680d1db0 msgr2=0x7f1a680d7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.059 7f1a707e5700 1 --2- 172.21.15.32:0/3665892753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a680d1db0 0x7f1a680d7cb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f1a58006700 tx=0x7f1a58009b00).stop 2021-04-09T18:25:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.060 7f1a707e5700 1 -- 172.21.15.32:0/3665892753 shutdown_connections 2021-04-09T18:25:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.060 7f1a707e5700 1 --2- 172.21.15.32:0/3665892753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a680d1db0 0x7f1a680d7cb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.060 7f1a707e5700 1 --2- 172.21.15.32:0/3665892753 >> v2:172.21.15.32:3300/0 conn(0x7f1a6810d020 0x7f1a680d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.060 7f1a707e5700 1 -- 172.21.15.32:0/3665892753 >> 172.21.15.32:0/3665892753 conn(0x7f1a680ce600 msgr2=0x7f1a680d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.060 7f1a707e5700 1 -- 172.21.15.32:0/3665892753 shutdown_connections 2021-04-09T18:25:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.060 7f1a707e5700 1 -- 172.21.15.32:0/3665892753 wait complete. 2021-04-09T18:25:33.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.080 7f1a707e5700 1 Processor -- start 2021-04-09T18:25:33.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a707e5700 1 -- start start 2021-04-09T18:25:33.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a707e5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a68095190 0x7f1a680948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a707e5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a68092f30 0x7f1a6809a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a707e5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a68093330 0x7f1a6809acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a707e5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6809b360 con 0x7f1a68092f30 2021-04-09T18:25:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a707e5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6809b570 con 0x7f1a68095190 2021-04-09T18:25:33.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a707e5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a680c0bc0 con 0x7f1a68093330 2021-04-09T18:25:33.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a6de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a68092f30 0x7f1a6809a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a6de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a68092f30 0x7f1a6809a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52440/0 (socket says 172.21.15.32:52440) 2021-04-09T18:25:33.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a6de00700 1 -- 172.21.15.32:0/921104115 learned_addr learned my addr 172.21.15.32:0/921104115 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:33.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a6e601700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a68095190 0x7f1a680948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.081 7f1a6ee02700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a68093330 0x7f1a6809acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a6de00700 1 -- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a68093330 msgr2=0x7f1a6809acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a6de00700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a68093330 0x7f1a6809acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a6de00700 1 -- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a68095190 msgr2=0x7f1a680948b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a6de00700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a68095190 0x7f1a680948b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a6de00700 1 -- 172.21.15.32:0/921104115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6809bac0 con 0x7f1a68092f30 2021-04-09T18:25:33.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a6ee02700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a68093330 0x7f1a6809acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:33.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a6e601700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a68095190 0x7f1a680948b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:33.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a6de00700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a68092f30 0x7f1a6809a6a0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 rx=0x7f1a5800b660 tx=0x7f1a5800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:33.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a5effd700 1 -- 172.21.15.32:0/921104115 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a58009440 con 0x7f1a68092f30 2021-04-09T18:25:33.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a5effd700 1 -- 172.21.15.32:0/921104115 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a58006390 con 0x7f1a68092f30 2021-04-09T18:25:33.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a707e5700 1 -- 172.21.15.32:0/921104115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6809bac0 con 0x7f1a68092f30 2021-04-09T18:25:33.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a5effd700 1 -- 172.21.15.32:0/921104115 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a58002030 con 0x7f1a68092f30 2021-04-09T18:25:33.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.082 7f1a707e5700 1 -- 172.21.15.32:0/921104115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6809bac0 con 0x7f1a68092f30 2021-04-09T18:25:33.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.083 7f1a5effd700 1 -- 172.21.15.32:0/921104115 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1a580093e0 con 0x7f1a68092f30 2021-04-09T18:25:33.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.083 7f1a5effd700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a50018930 0x7f1a5001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.083 7f1a5effd700 1 -- 172.21.15.32:0/921104115 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f1a580329a0 con 0x7f1a68092f30 2021-04-09T18:25:33.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.083 7f1a6e601700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a50018930 0x7f1a5001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.083 7f1a6e601700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a50018930 0x7f1a5001ae10 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:33.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.100 7f1a707e5700 1 -- 172.21.15.32:0/921104115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a681b7120 con 0x7f1a68092f30 2021-04-09T18:25:33.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.104 7f1a5effd700 1 -- 172.21.15.32:0/921104115 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1a580090c0 con 0x7f1a68092f30 2021-04-09T18:25:33.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.286 7f1a707e5700 1 -- 172.21.15.32:0/921104115 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f1a6810d490 con 0x7f1a68092f30 2021-04-09T18:25:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.290 7f1a5effd700 1 -- 172.21.15.32:0/921104115 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 22 and data pool 23 v265) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1a580093e0 con 0x7f1a68092f30 2021-04-09T18:25:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 22 and data pool 23 2021-04-09T18:25:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.299 7f1a707e5700 1 -- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a50018930 msgr2=0x7f1a5001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.299 7f1a707e5700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a50018930 0x7f1a5001ae10 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.299 7f1a707e5700 1 -- 172.21.15.32:0/921104115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a68092f30 msgr2=0x7f1a6809a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.299 7f1a707e5700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a68092f30 0x7f1a6809a6a0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 rx=0x7f1a5800b660 tx=0x7f1a5800ba00).stop 2021-04-09T18:25:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.300 7f1a707e5700 1 -- 172.21.15.32:0/921104115 shutdown_connections 2021-04-09T18:25:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.300 7f1a707e5700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a50018930 0x7f1a5001ae10 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.300 7f1a707e5700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a68092f30 0x7f1a6809a6a0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.300 7f1a707e5700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a68093330 0x7f1a6809acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.300 7f1a707e5700 1 --2- 172.21.15.32:0/921104115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a68095190 0x7f1a680948b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.300 7f1a707e5700 1 -- 172.21.15.32:0/921104115 >> 172.21.15.32:0/921104115 conn(0x7f1a6808c120 msgr2=0x7f1a68099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.300 7f1a707e5700 1 -- 172.21.15.32:0/921104115 shutdown_connections 2021-04-09T18:25:33.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.300 7f1a707e5700 1 -- 172.21.15.32:0/921104115 wait complete. 2021-04-09T18:25:33.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:25:33.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.601 7f6659ddc700 1 Processor -- start 2021-04-09T18:25:33.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.601 7f6659ddc700 1 -- start start 2021-04-09T18:25:33.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f6659ddc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f665410da00 0x7f665410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f6659ddc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665410d480 con 0x7f665410d010 2021-04-09T18:25:33.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f6659ddc700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66540d1f70 con 0x7f66540d7ca0 2021-04-09T18:25:33.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f6659ddc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66540d9220 con 0x7f665410da00 2021-04-09T18:25:33.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f66537fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f665410d010 0x7f66540d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33806/0 (socket says 172.21.15.32:33806) 2021-04-09T18:25:33.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f66537fe700 1 -- 172.21.15.32:0/1622165864 learned_addr learned my addr 172.21.15.32:0/1622165864 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:33.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f6653fff700 1 --2- 172.21.15.32:0/1622165864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f665410da00 0x7f665410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f66527fc700 1 -- 172.21.15.32:0/1622165864 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474541334 0 0) 0x7f6644000ce0 con 0x7f665410d010 2021-04-09T18:25:33.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f66527fc700 1 -- 172.21.15.32:0/1622165864 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663c002fe0 con 0x7f665410d010 2021-04-09T18:25:33.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f66527fc700 1 -- 172.21.15.32:0/1622165864 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891966119 0 0) 0x7f6648000c60 con 0x7f66540d7ca0 2021-04-09T18:25:33.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f66527fc700 1 -- 172.21.15.32:0/1622165864 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663c003210 con 0x7f66540d7ca0 2021-04-09T18:25:33.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f66527fc700 1 -- 172.21.15.32:0/1622165864 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1959952384 0 0) 0x7f6644000ed0 con 0x7f665410d010 2021-04-09T18:25:33.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f6653fff700 1 -- 172.21.15.32:0/1622165864 >> v1:172.21.15.86:6789/0 conn(0x7f66540d7ca0 legacy=0x7f66540d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f6653fff700 1 -- 172.21.15.32:0/1622165864 >> v1:172.21.15.32:6789/0 conn(0x7f665410d010 legacy=0x7f66540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.602 7f6653fff700 1 -- 172.21.15.32:0/1622165864 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665410e420 con 0x7f665410da00 2021-04-09T18:25:33.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.603 7f6653fff700 1 --2- 172.21.15.32:0/1622165864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f665410da00 0x7f665410de00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f664c006e50 tx=0x7f664c009e70).ready entity=mon.2 client_cookie=5cf43c8a5995c57b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:33.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.603 7f66527fc700 1 -- 172.21.15.32:0/1622165864 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f664c006a20 con 0x7f665410da00 2021-04-09T18:25:33.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.603 7f66527fc700 1 -- 172.21.15.32:0/1622165864 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c006790 con 0x7f665410da00 2021-04-09T18:25:33.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.603 7f66527fc700 1 -- 172.21.15.32:0/1622165864 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f664c00b700 con 0x7f665410da00 2021-04-09T18:25:33.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.603 7f6659ddc700 1 -- 172.21.15.32:0/1622165864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f665410da00 msgr2=0x7f665410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.603 7f6659ddc700 1 --2- 172.21.15.32:0/1622165864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f665410da00 0x7f665410de00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f664c006e50 tx=0x7f664c009e70).stop 2021-04-09T18:25:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.604 7f6659ddc700 1 -- 172.21.15.32:0/1622165864 shutdown_connections 2021-04-09T18:25:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.604 7f6659ddc700 1 --2- 172.21.15.32:0/1622165864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f665410da00 0x7f665410de00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.604 7f6659ddc700 1 -- 172.21.15.32:0/1622165864 >> 172.21.15.32:0/1622165864 conn(0x7f66540ce5f0 msgr2=0x7f66540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:33.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.604 7f6659ddc700 1 -- 172.21.15.32:0/1622165864 shutdown_connections 2021-04-09T18:25:33.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.604 7f6659ddc700 1 -- 172.21.15.32:0/1622165864 wait complete. 2021-04-09T18:25:33.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.624 7f6659ddc700 1 Processor -- start 2021-04-09T18:25:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.624 7f6659ddc700 1 -- start start 2021-04-09T18:25:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6659ddc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66541ac380 0x7f665418a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6659ddc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66541ac780 0x7f665418a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6659ddc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6654183520 0x7f6654183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6659ddc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66541ace10 con 0x7f66541ac780 2021-04-09T18:25:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6659ddc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6654189cf0 con 0x7f6654183520 2021-04-09T18:25:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6659ddc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6654183f40 con 0x7f66541ac380 2021-04-09T18:25:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6652ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66541ac780 0x7f665418a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6652ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66541ac780 0x7f665418a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52454/0 (socket says 172.21.15.32:52454) 2021-04-09T18:25:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6652ffd700 1 -- 172.21.15.32:0/1417193691 learned_addr learned my addr 172.21.15.32:0/1417193691 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f66537fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66541ac380 0x7f665418a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6653fff700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6654183520 0x7f6654183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6652ffd700 1 -- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66541ac380 msgr2=0x7f665418a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6652ffd700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66541ac380 0x7f665418a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6652ffd700 1 -- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6654183520 msgr2=0x7f6654183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6652ffd700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6654183520 0x7f6654183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6652ffd700 1 -- 172.21.15.32:0/1417193691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6654184160 con 0x7f66541ac780 2021-04-09T18:25:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f6652ffd700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66541ac780 0x7f665418a850 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7f6648006f10 tx=0x7f664800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.625 7f663bfff700 1 -- 172.21.15.32:0/1417193691 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f664800bd80 con 0x7f66541ac780 2021-04-09T18:25:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.626 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6654184160 con 0x7f66541ac780 2021-04-09T18:25:33.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.626 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6654184160 con 0x7f66541ac780 2021-04-09T18:25:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.626 7f663bfff700 1 -- 172.21.15.32:0/1417193691 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6648000e20 con 0x7f66541ac780 2021-04-09T18:25:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.626 7f663bfff700 1 -- 172.21.15.32:0/1417193691 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f664800c070 con 0x7f66541ac780 2021-04-09T18:25:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.626 7f663bfff700 1 -- 172.21.15.32:0/1417193691 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f664800bd80 con 0x7f66541ac780 2021-04-09T18:25:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.626 7f663bfff700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6634018930 0x7f663401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.626 7f663bfff700 1 -- 172.21.15.32:0/1417193691 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f66480329e0 con 0x7f66541ac780 2021-04-09T18:25:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.626 7f66537fe700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6634018930 0x7f663401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.627 7f66537fe700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6634018930 0x7f663401ae10 crc :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:33.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.646 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6654189cf0 con 0x7f66541ac780 2021-04-09T18:25:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.649 7f663bfff700 1 -- 172.21.15.32:0/1417193691 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f664800bd80 con 0x7f66541ac780 2021-04-09T18:25:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.837 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6654189cf0 con 0x7f66541ac780 2021-04-09T18:25:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.838 7f663bfff700 1 -- 172.21.15.32:0/1417193691 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f6648000e20 con 0x7f66541ac780 2021-04-09T18:25:33.905 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:33.906 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":130,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:25:33.288355","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:25:30.677877","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-04-09 18:25:32.094578","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:25:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6634018930 msgr2=0x7f663401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6634018930 0x7f663401ae10 crc :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66541ac780 msgr2=0x7f665418a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:33.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66541ac780 0x7f665418a850 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7f6648006f10 tx=0x7f664800b990).stop 2021-04-09T18:25:33.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 shutdown_connections 2021-04-09T18:25:33.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6634018930 0x7f663401ae10 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66541ac780 0x7f665418a850 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6654183520 0x7f6654183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 --2- 172.21.15.32:0/1417193691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66541ac380 0x7f665418a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 >> 172.21.15.32:0/1417193691 conn(0x7f665419b890 msgr2=0x7f66541a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:33.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 shutdown_connections 2021-04-09T18:25:33.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:33.840 7f6659ddc700 1 -- 172.21.15.32:0/1417193691 wait complete. 2021-04-09T18:25:33.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:25:34.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.001 7f64a0c30700 1 Processor -- start 2021-04-09T18:25:34.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.001 7f64a0c30700 1 -- start start 2021-04-09T18:25:34.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.002 7f64a0c30700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f649c10da00 0x7f649c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.002 7f64a0c30700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c10d480 con 0x7f649c0d7ca0 2021-04-09T18:25:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.002 7f64a0c30700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0d1f70 con 0x7f649c10d010 2021-04-09T18:25:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.002 7f64a0c30700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f649c0d8d30 con 0x7f649c10da00 2021-04-09T18:25:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.002 7f649a05d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f649c0d7ca0 0x7f649c0d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33820/0 (socket says 172.21.15.32:33820) 2021-04-09T18:25:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.002 7f649a05d700 1 -- 172.21.15.32:0/554561160 learned_addr learned my addr 172.21.15.32:0/554561160 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:34.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649b05f700 1 --2- 172.21.15.32:0/554561160 >> v2:172.21.15.86:3300/0 conn(0x7f649c10da00 0x7f649c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649985c700 1 -- 172.21.15.32:0/554561160 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236269607 0 0) 0x7f6490000ce0 con 0x7f649c0d7ca0 2021-04-09T18:25:34.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649985c700 1 -- 172.21.15.32:0/554561160 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6484002fe0 con 0x7f649c0d7ca0 2021-04-09T18:25:34.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649985c700 1 -- 172.21.15.32:0/554561160 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084238549 0 0) 0x7f648c001080 con 0x7f649c10d010 2021-04-09T18:25:34.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649985c700 1 -- 172.21.15.32:0/554561160 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6484003210 con 0x7f649c10d010 2021-04-09T18:25:34.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649985c700 1 -- 172.21.15.32:0/554561160 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2640363516 0 0) 0x7f64900010e0 con 0x7f649c0d7ca0 2021-04-09T18:25:34.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649b05f700 1 -- 172.21.15.32:0/554561160 >> v1:172.21.15.86:6789/0 conn(0x7f649c10d010 legacy=0x7f649c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649b05f700 1 -- 172.21.15.32:0/554561160 >> v1:172.21.15.32:6789/0 conn(0x7f649c0d7ca0 legacy=0x7f649c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649b05f700 1 -- 172.21.15.32:0/554561160 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0d9220 con 0x7f649c10da00 2021-04-09T18:25:34.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649b05f700 1 --2- 172.21.15.32:0/554561160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f649c10da00 0x7f649c10de00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f6494006e50 tx=0x7f6494009e70).ready entity=mon.1 client_cookie=f99431d37a5973c7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:34.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649985c700 1 -- 172.21.15.32:0/554561160 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6494006a20 con 0x7f649c10da00 2021-04-09T18:25:34.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649985c700 1 -- 172.21.15.32:0/554561160 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6494006790 con 0x7f649c10da00 2021-04-09T18:25:34.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f649985c700 1 -- 172.21.15.32:0/554561160 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f649400b700 con 0x7f649c10da00 2021-04-09T18:25:34.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f64a0c30700 1 -- 172.21.15.32:0/554561160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f649c10da00 msgr2=0x7f649c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.003 7f64a0c30700 1 --2- 172.21.15.32:0/554561160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f649c10da00 0x7f649c10de00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f6494006e50 tx=0x7f6494009e70).stop 2021-04-09T18:25:34.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.004 7f64a0c30700 1 -- 172.21.15.32:0/554561160 shutdown_connections 2021-04-09T18:25:34.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.004 7f64a0c30700 1 --2- 172.21.15.32:0/554561160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f649c10da00 0x7f649c10de00 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.004 7f64a0c30700 1 -- 172.21.15.32:0/554561160 >> 172.21.15.32:0/554561160 conn(0x7f649c0ce5f0 msgr2=0x7f649c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:34.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.004 7f64a0c30700 1 -- 172.21.15.32:0/554561160 shutdown_connections 2021-04-09T18:25:34.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.004 7f64a0c30700 1 -- 172.21.15.32:0/554561160 wait complete. 2021-04-09T18:25:34.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.024 7f64a0c30700 1 Processor -- start 2021-04-09T18:25:34.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.024 7f64a0c30700 1 -- start start 2021-04-09T18:25:34.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f64a0c30700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f649c1a91e0 0x7f649c0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f64a0c30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f649c1a95e0 0x7f649c0a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f64a0c30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f649c1ac850 0x7f649c1acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f64a0c30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c1ad270 con 0x7f649c1a95e0 2021-04-09T18:25:34.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f64a0c30700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0b7640 con 0x7f649c1a91e0 2021-04-09T18:25:34.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f64a0c30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649c183c80 con 0x7f649c1ac850 2021-04-09T18:25:34.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649a05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f649c1a95e0 0x7f649c0a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649a05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f649c1a95e0 0x7f649c0a5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52466/0 (socket says 172.21.15.32:52466) 2021-04-09T18:25:34.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649a05d700 1 -- 172.21.15.32:0/2578322046 learned_addr learned my addr 172.21.15.32:0/2578322046 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:34.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649b05f700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f649c1ac850 0x7f649c1acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649a85e700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f649c1a91e0 0x7f649c0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649a05d700 1 -- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f649c1ac850 msgr2=0x7f649c1acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649a05d700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f649c1ac850 0x7f649c1acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649a05d700 1 -- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f649c1a91e0 msgr2=0x7f649c0a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649a05d700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f649c1a91e0 0x7f649c0a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.025 7f649a05d700 1 -- 172.21.15.32:0/2578322046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c183ea0 con 0x7f649c1a95e0 2021-04-09T18:25:34.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.026 7f649a05d700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f649c1a95e0 0x7f649c0a5080 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 rx=0x7f6490000f90 tx=0x7f649000a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:34.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.026 7f648b7fe700 1 -- 172.21.15.32:0/2578322046 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6490001a30 con 0x7f649c1a95e0 2021-04-09T18:25:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.026 7f648b7fe700 1 -- 172.21.15.32:0/2578322046 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64900043a0 con 0x7f649c1a95e0 2021-04-09T18:25:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.026 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c183ea0 con 0x7f649c1a95e0 2021-04-09T18:25:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.026 7f648b7fe700 1 -- 172.21.15.32:0/2578322046 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64900047f0 con 0x7f649c1a95e0 2021-04-09T18:25:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.026 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c183ea0 con 0x7f649c1a95e0 2021-04-09T18:25:34.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.027 7f648b7fe700 1 -- 172.21.15.32:0/2578322046 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6490004580 con 0x7f649c1a95e0 2021-04-09T18:25:34.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.027 7f648b7fe700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64800188c0 0x7f648001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.027 7f648b7fe700 1 -- 172.21.15.32:0/2578322046 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f64900327a0 con 0x7f649c1a95e0 2021-04-09T18:25:34.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.027 7f649a85e700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64800188c0 0x7f648001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.027 7f649a85e700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64800188c0 0x7f648001adc0 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:34.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.046 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c182d70 con 0x7f649c1a95e0 2021-04-09T18:25:34.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.050 7f648b7fe700 1 -- 172.21.15.32:0/2578322046 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6490001a30 con 0x7f649c1a95e0 2021-04-09T18:25:34.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.237 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f649c10d480 con 0x7f649c1a95e0 2021-04-09T18:25:34.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.238 7f648b7fe700 1 -- 172.21.15.32:0/2578322046 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f6490004560 con 0x7f649c1a95e0 2021-04-09T18:25:34.305 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:34.306 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":130,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:25:33.288355","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:25:30.677877","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-04-09 18:25:32.094578","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:25:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64800188c0 msgr2=0x7f648001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64800188c0 0x7f648001adc0 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f649c1a95e0 msgr2=0x7f649c0a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f649c1a95e0 0x7f649c0a5080 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 rx=0x7f6490000f90 tx=0x7f649000a940).stop 2021-04-09T18:25:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 shutdown_connections 2021-04-09T18:25:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64800188c0 0x7f648001adc0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f649c1a95e0 0x7f649c0a5080 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f649c1ac850 0x7f649c1acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 --2- 172.21.15.32:0/2578322046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f649c1a91e0 0x7f649c0a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 >> 172.21.15.32:0/2578322046 conn(0x7f649c19b460 msgr2=0x7f649c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:34.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 shutdown_connections 2021-04-09T18:25:34.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.240 7f64a0c30700 1 -- 172.21.15.32:0/2578322046 wait complete. 2021-04-09T18:25:34.318 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:25:34.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.404 7f967b23e700 1 Processor -- start 2021-04-09T18:25:34.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.404 7f967b23e700 1 -- start start 2021-04-09T18:25:34.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f967b23e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f96740cffc0 0x7f96740d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f967b23e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740d72a0 con 0x7f96740d7820 2021-04-09T18:25:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f967b23e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96740d8410 con 0x7f96740d6ca0 2021-04-09T18:25:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f967b23e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96740d70a0 con 0x7f96740cffc0 2021-04-09T18:25:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f9678859700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f96740d7820 0x7f96740d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33836/0 (socket says 172.21.15.32:33836) 2021-04-09T18:25:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f9678859700 1 -- 172.21.15.32:0/1009215559 learned_addr learned my addr 172.21.15.32:0/1009215559 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f967905a700 1 --2- 172.21.15.32:0/1009215559 >> v2:172.21.15.86:3300/0 conn(0x7f96740cffc0 0x7f96740d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f966bfff700 1 -- 172.21.15.32:0/1009215559 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213416618 0 0) 0x7f96640009b0 con 0x7f96740d7820 2021-04-09T18:25:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f966bfff700 1 -- 172.21.15.32:0/1009215559 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9660002fe0 con 0x7f96740d7820 2021-04-09T18:25:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f966bfff700 1 -- 172.21.15.32:0/1009215559 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201761684 0 0) 0x7f966c000d30 con 0x7f96740d6ca0 2021-04-09T18:25:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f966bfff700 1 -- 172.21.15.32:0/1009215559 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9660003020 con 0x7f96740d6ca0 2021-04-09T18:25:34.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f967905a700 1 -- 172.21.15.32:0/1009215559 >> v1:172.21.15.86:6789/0 conn(0x7f96740d6ca0 legacy=0x7f967410cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f967905a700 1 -- 172.21.15.32:0/1009215559 >> v1:172.21.15.32:6789/0 conn(0x7f96740d7820 legacy=0x7f96740d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f966bfff700 1 -- 172.21.15.32:0/1009215559 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1775886408 0 0) 0x7f9664001010 con 0x7f96740d7820 2021-04-09T18:25:34.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.405 7f967905a700 1 -- 172.21.15.32:0/1009215559 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96740d9200 con 0x7f96740cffc0 2021-04-09T18:25:34.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.406 7f967905a700 1 --2- 172.21.15.32:0/1009215559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740cffc0 0x7f96740d2460 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f9670006650 tx=0x7f9670009a80).ready entity=mon.1 client_cookie=819376f56a821246 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:34.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.406 7f966bfff700 1 -- 172.21.15.32:0/1009215559 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96700009b0 con 0x7f96740cffc0 2021-04-09T18:25:34.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.406 7f966bfff700 1 -- 172.21.15.32:0/1009215559 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967000c0c0 con 0x7f96740cffc0 2021-04-09T18:25:34.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.406 7f966bfff700 1 -- 172.21.15.32:0/1009215559 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96700009b0 con 0x7f96740cffc0 2021-04-09T18:25:34.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.406 7f967b23e700 1 -- 172.21.15.32:0/1009215559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740cffc0 msgr2=0x7f96740d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.406 7f967b23e700 1 --2- 172.21.15.32:0/1009215559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740cffc0 0x7f96740d2460 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f9670006650 tx=0x7f9670009a80).stop 2021-04-09T18:25:34.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.407 7f967b23e700 1 -- 172.21.15.32:0/1009215559 shutdown_connections 2021-04-09T18:25:34.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.407 7f967b23e700 1 --2- 172.21.15.32:0/1009215559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740cffc0 0x7f96740d2460 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.407 7f967b23e700 1 -- 172.21.15.32:0/1009215559 >> 172.21.15.32:0/1009215559 conn(0x7f96740ce5e0 msgr2=0x7f96740cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:34.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.407 7f967b23e700 1 -- 172.21.15.32:0/1009215559 shutdown_connections 2021-04-09T18:25:34.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.407 7f967b23e700 1 -- 172.21.15.32:0/1009215559 wait complete. 2021-04-09T18:25:34.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.427 7f967b23e700 1 Processor -- start 2021-04-09T18:25:34.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.427 7f967b23e700 1 -- start start 2021-04-09T18:25:34.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967b23e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96741833a0 0x7f96740a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967b23e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a2550 0x7f96740a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967b23e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9674094d00 0x7f9674095100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967b23e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96740a1fd0 con 0x7f96741833a0 2021-04-09T18:25:34.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967b23e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96740a2f70 con 0x7f96740a2550 2021-04-09T18:25:34.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967b23e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9674095720 con 0x7f9674094d00 2021-04-09T18:25:34.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967905a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96741833a0 0x7f96740a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967905a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96741833a0 0x7f96740a3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52480/0 (socket says 172.21.15.32:52480) 2021-04-09T18:25:34.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967905a700 1 -- 172.21.15.32:0/2338188006 learned_addr learned my addr 172.21.15.32:0/2338188006 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:34.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f9678859700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a2550 0x7f96740a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967985b700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9674094d00 0x7f9674095100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f9678859700 1 -- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9674094d00 msgr2=0x7f9674095100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f9678859700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9674094d00 0x7f9674095100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f9678859700 1 -- 172.21.15.32:0/2338188006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96741833a0 msgr2=0x7f96740a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:34.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f9678859700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96741833a0 0x7f96740a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:34.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f9678859700 1 -- 172.21.15.32:0/2338188006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674095940 con 0x7f96740a2550 2021-04-09T18:25:34.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f967905a700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96741833a0 0x7f96740a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:34.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.428 7f9678859700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a2550 0x7f96740a2950 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f9664001c60 tx=0x7f966400a870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:34.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.429 7f9669ffb700 1 -- 172.21.15.32:0/2338188006 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9664000d30 con 0x7f96740a2550 2021-04-09T18:25:34.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.429 7f9669ffb700 1 -- 172.21.15.32:0/2338188006 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9664001270 con 0x7f96740a2550 2021-04-09T18:25:34.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.429 7f967b23e700 1 -- 172.21.15.32:0/2338188006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9674095940 con 0x7f96740a2550 2021-04-09T18:25:34.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.429 7f9669ffb700 1 -- 172.21.15.32:0/2338188006 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9664004350 con 0x7f96740a2550 2021-04-09T18:25:34.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.429 7f967b23e700 1 -- 172.21.15.32:0/2338188006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9674095940 con 0x7f96740a2550 2021-04-09T18:25:34.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.429 7f9669ffb700 1 -- 172.21.15.32:0/2338188006 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9664004140 con 0x7f96740a2550 2021-04-09T18:25:34.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.430 7f9669ffb700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9654018930 0x7f965401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:34.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.430 7f9669ffb700 1 -- 172.21.15.32:0/2338188006 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f96640328e0 con 0x7f96740a2550 2021-04-09T18:25:34.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.430 7f967905a700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9654018930 0x7f965401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:34.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.430 7f967905a700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9654018930 0x7f965401ae10 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:34.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.449 7f967b23e700 1 -- 172.21.15.32:0/2338188006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9674095be0 con 0x7f96740a2550 2021-04-09T18:25:34.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.453 7f9669ffb700 1 -- 172.21.15.32:0/2338188006 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9664000cc0 con 0x7f96740a2550 2021-04-09T18:25:34.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:34.645 7f967b23e700 1 -- 172.21.15.32:0/2338188006 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f9674093df0 con 0x7f96740a2550 2021-04-09T18:25:35.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.623 7f9669ffb700 1 -- 172.21.15.32:0/2338188006 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v268) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9664004140 con 0x7f96740a2550 2021-04-09T18:25:35.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 -- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9654018930 msgr2=0x7f965401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:35.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9654018930 0x7f965401ae10 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 -- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a2550 msgr2=0x7f96740a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:35.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a2550 0x7f96740a2950 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f9664001c60 tx=0x7f966400a870).stop 2021-04-09T18:25:35.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 -- 172.21.15.32:0/2338188006 shutdown_connections 2021-04-09T18:25:35.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9654018930 0x7f965401ae10 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9674094d00 0x7f9674095100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96740a2550 0x7f96740a2950 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 --2- 172.21.15.32:0/2338188006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96741833a0 0x7f96740a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 -- 172.21.15.32:0/2338188006 >> 172.21.15.32:0/2338188006 conn(0x7f967409fa00 msgr2=0x7f96740b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:35.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 -- 172.21.15.32:0/2338188006 shutdown_connections 2021-04-09T18:25:35.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.664 7f967b23e700 1 -- 172.21.15.32:0/2338188006 wait complete. 2021-04-09T18:25:35.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:35.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.828 7fa4aae4d700 1 Processor -- start 2021-04-09T18:25:35.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.828 7fa4aae4d700 1 -- start start 2021-04-09T18:25:35.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4aae4d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4a40d7ca0 0x7fa4a40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:35.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4aae4d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4a410da00 0x7fa4a410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:35.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4aae4d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a410d480 con 0x7fa4a410d010 2021-04-09T18:25:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4aae4d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a40d9220 con 0x7fa4a410da00 2021-04-09T18:25:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4aae4d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a40d86c0 con 0x7fa4a40d7ca0 2021-04-09T18:25:35.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a946a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4a410da00 0x7fa4a410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:35.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a946a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4a410da00 0x7fa4a410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52488/0 (socket says 172.21.15.32:52488) 2021-04-09T18:25:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a946a700 1 -- 172.21.15.32:0/14384955 learned_addr learned my addr 172.21.15.32:0/14384955 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a3fff700 1 --2- 172.21.15.32:0/14384955 >> v2:172.21.15.86:3300/0 conn(0x7fa4a40d7ca0 0x7fa4a40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:35.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a946a700 1 -- 172.21.15.32:0/14384955 >> v2:172.21.15.86:3300/0 conn(0x7fa4a40d7ca0 msgr2=0x7fa4a40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a946a700 1 --2- 172.21.15.32:0/14384955 >> v2:172.21.15.86:3300/0 conn(0x7fa4a40d7ca0 0x7fa4a40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a946a700 1 -- 172.21.15.32:0/14384955 >> v1:172.21.15.86:6789/0 conn(0x7fa4a410d010 legacy=0x7fa4a40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:35.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a946a700 1 -- 172.21.15.32:0/14384955 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40d1f70 con 0x7fa4a410da00 2021-04-09T18:25:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a946a700 1 --2- 172.21.15.32:0/14384955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a410da00 0x7fa4a410de00 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 rx=0x7fa49c006b80 tx=0x7fa49c009b90).ready entity=mon.0 client_cookie=4800ccc6b9061555 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a37fe700 1 -- 172.21.15.32:0/14384955 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa49c0022e0 con 0x7fa4a410da00 2021-04-09T18:25:35.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a37fe700 1 -- 172.21.15.32:0/14384955 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c002620 con 0x7fa4a410da00 2021-04-09T18:25:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.829 7fa4a37fe700 1 -- 172.21.15.32:0/14384955 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa49c002b00 con 0x7fa4a410da00 2021-04-09T18:25:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.830 7fa4aae4d700 1 -- 172.21.15.32:0/14384955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a410da00 msgr2=0x7fa4a410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.830 7fa4aae4d700 1 --2- 172.21.15.32:0/14384955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a410da00 0x7fa4a410de00 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 rx=0x7fa49c006b80 tx=0x7fa49c009b90).stop 2021-04-09T18:25:35.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.830 7fa4aae4d700 1 -- 172.21.15.32:0/14384955 shutdown_connections 2021-04-09T18:25:35.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.830 7fa4aae4d700 1 --2- 172.21.15.32:0/14384955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a410da00 0x7fa4a410de00 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.830 7fa4aae4d700 1 --2- 172.21.15.32:0/14384955 >> v2:172.21.15.86:3300/0 conn(0x7fa4a40d7ca0 0x7fa4a40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.830 7fa4aae4d700 1 -- 172.21.15.32:0/14384955 >> 172.21.15.32:0/14384955 conn(0x7fa4a40ce5f0 msgr2=0x7fa4a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:35.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.830 7fa4aae4d700 1 -- 172.21.15.32:0/14384955 shutdown_connections 2021-04-09T18:25:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.830 7fa4aae4d700 1 -- 172.21.15.32:0/14384955 wait complete. 2021-04-09T18:25:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.851 7fa4aae4d700 1 Processor -- start 2021-04-09T18:25:35.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.851 7fa4aae4d700 1 -- start start 2021-04-09T18:25:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.851 7fa4aae4d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a41aca20 0x7fa4a41ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4aae4d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a418bd40 0x7fa4a418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4aae4d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a408fba0 0x7fa4a408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4aae4d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a418c7e0 con 0x7fa4a41aca20 2021-04-09T18:25:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4aae4d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a418ca30 con 0x7fa4a408fba0 2021-04-09T18:25:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4aae4d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40905f0 con 0x7fa4a418bd40 2021-04-09T18:25:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a8c69700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a41aca20 0x7fa4a41ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a946a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a408fba0 0x7fa4a408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a8c69700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a41aca20 0x7fa4a41ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52492/0 (socket says 172.21.15.32:52492) 2021-04-09T18:25:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a8c69700 1 -- 172.21.15.32:0/99483073 learned_addr learned my addr 172.21.15.32:0/99483073 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a3fff700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a418bd40 0x7fa4a418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a946a700 1 -- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a418bd40 msgr2=0x7fa4a418c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a946a700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a418bd40 0x7fa4a418c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a946a700 1 -- 172.21.15.32:0/99483073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a41aca20 msgr2=0x7fa4a41ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:35.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a946a700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a41aca20 0x7fa4a41ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:35.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a946a700 1 -- 172.21.15.32:0/99483073 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40b4300 con 0x7fa4a408fba0 2021-04-09T18:25:35.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a3fff700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a418bd40 0x7fa4a418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:35.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a8c69700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a41aca20 0x7fa4a41ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:35.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a946a700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a408fba0 0x7fa4a408ffa0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fa49c0095c0 tx=0x7fa49c00b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:35.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a17fa700 1 -- 172.21.15.32:0/99483073 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa49c00bcb0 con 0x7fa4a408fba0 2021-04-09T18:25:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a17fa700 1 -- 172.21.15.32:0/99483073 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c006a00 con 0x7fa4a408fba0 2021-04-09T18:25:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a40b4300 con 0x7fa4a408fba0 2021-04-09T18:25:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4a17fa700 1 -- 172.21.15.32:0/99483073 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa49c006450 con 0x7fa4a408fba0 2021-04-09T18:25:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.852 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a40b4300 con 0x7fa4a408fba0 2021-04-09T18:25:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.853 7fa4a17fa700 1 -- 172.21.15.32:0/99483073 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa49c006030 con 0x7fa4a408fba0 2021-04-09T18:25:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.853 7fa4a17fa700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa488018930 0x7fa48801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.853 7fa4a17fa700 1 -- 172.21.15.32:0/99483073 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa49c032950 con 0x7fa4a408fba0 2021-04-09T18:25:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.854 7fa4a8c69700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa488018930 0x7fa48801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.854 7fa4a8c69700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa488018930 0x7fa48801ae10 crc :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.873 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a4182d70 con 0x7fa4a408fba0 2021-04-09T18:25:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:35.876 7fa4a17fa700 1 -- 172.21.15.32:0/99483073 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa49c006a00 con 0x7fa4a408fba0 2021-04-09T18:25:36.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.067 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4a418c760 con 0x7fa4a408fba0 2021-04-09T18:25:36.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.067 7fa4a17fa700 1 -- 172.21.15.32:0/99483073 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2785 (secure 0 0 0) 0x7fa49c006a00 con 0x7fa4a408fba0 2021-04-09T18:25:36.135 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 268 2021-04-09T18:25:36.136 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:36.136 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":268,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:25:35.619821","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:25:36.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.069 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa488018930 msgr2=0x7fa48801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.069 7fa4aae4d700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa488018930 0x7fa48801ae10 crc :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.069 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a408fba0 msgr2=0x7fa4a408ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.069 7fa4aae4d700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a408fba0 0x7fa4a408ffa0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fa49c0095c0 tx=0x7fa49c00b8c0).stop 2021-04-09T18:25:36.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.070 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 shutdown_connections 2021-04-09T18:25:36.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.070 7fa4aae4d700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa488018930 0x7fa48801ae10 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.070 7fa4aae4d700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a408fba0 0x7fa4a408ffa0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.070 7fa4aae4d700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a418bd40 0x7fa4a418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.070 7fa4aae4d700 1 --2- 172.21.15.32:0/99483073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4a41aca20 0x7fa4a41ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.070 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 >> 172.21.15.32:0/99483073 conn(0x7fa4a419b460 msgr2=0x7fa4a419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:36.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.070 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 shutdown_connections 2021-04-09T18:25:36.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.070 7fa4aae4d700 1 -- 172.21.15.32:0/99483073 wait complete. 2021-04-09T18:25:36.147 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:25:36.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.229 7fe16e33e700 1 Processor -- start 2021-04-09T18:25:36.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.229 7fe16e33e700 1 -- start start 2021-04-09T18:25:36.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.229 7fe16e33e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1680d7ca0 0x7fe1680d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.229 7fe16e33e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe16810da00 0x7fe16810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.229 7fe16e33e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16810d480 con 0x7fe16810d010 2021-04-09T18:25:36.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe16e33e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1680d9220 con 0x7fe1680d7ca0 2021-04-09T18:25:36.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe16e33e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1680d86c0 con 0x7fe16810da00 2021-04-09T18:25:36.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe15ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1680d7ca0 0x7fe1680d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe167fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe16810d010 0x7fe1680d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33858/0 (socket says 172.21.15.32:33858) 2021-04-09T18:25:36.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe167fff700 1 -- 172.21.15.32:0/643779414 learned_addr learned my addr 172.21.15.32:0/643779414 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:36.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe16c95b700 1 --2- 172.21.15.32:0/643779414 >> v2:172.21.15.86:3300/0 conn(0x7fe16810da00 0x7fe16810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe15ffff700 1 -- 172.21.15.32:0/643779414 >> v2:172.21.15.86:3300/0 conn(0x7fe16810da00 msgr2=0x7fe16810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:36.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe15ffff700 1 --2- 172.21.15.32:0/643779414 >> v2:172.21.15.86:3300/0 conn(0x7fe16810da00 0x7fe16810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe15ffff700 1 -- 172.21.15.32:0/643779414 >> v1:172.21.15.32:6789/0 conn(0x7fe16810d010 legacy=0x7fe1680d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe15ffff700 1 -- 172.21.15.32:0/643779414 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680d1f70 con 0x7fe1680d7ca0 2021-04-09T18:25:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe1677fe700 1 -- 172.21.15.32:0/643779414 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481250176 0 0) 0x7fe158000ce0 con 0x7fe16810d010 2021-04-09T18:25:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe15ffff700 1 --2- 172.21.15.32:0/643779414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680d7ca0 0x7fe1680d80a0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 rx=0x7fe150006650 tx=0x7fe150009a00).ready entity=mon.0 client_cookie=47b447d87ae1de3b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe1677fe700 1 -- 172.21.15.32:0/643779414 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe150009df0 con 0x7fe1680d7ca0 2021-04-09T18:25:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe1677fe700 1 -- 172.21.15.32:0/643779414 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1500009b0 con 0x7fe1680d7ca0 2021-04-09T18:25:36.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe1677fe700 1 -- 172.21.15.32:0/643779414 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe150009df0 con 0x7fe1680d7ca0 2021-04-09T18:25:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe16e33e700 1 -- 172.21.15.32:0/643779414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680d7ca0 msgr2=0x7fe1680d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.230 7fe16e33e700 1 --2- 172.21.15.32:0/643779414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680d7ca0 0x7fe1680d80a0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 rx=0x7fe150006650 tx=0x7fe150009a00).stop 2021-04-09T18:25:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.231 7fe16e33e700 1 -- 172.21.15.32:0/643779414 shutdown_connections 2021-04-09T18:25:36.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.231 7fe16e33e700 1 --2- 172.21.15.32:0/643779414 >> v2:172.21.15.86:3300/0 conn(0x7fe16810da00 0x7fe16810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.231 7fe16e33e700 1 --2- 172.21.15.32:0/643779414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1680d7ca0 0x7fe1680d80a0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.231 7fe16e33e700 1 -- 172.21.15.32:0/643779414 >> 172.21.15.32:0/643779414 conn(0x7fe1680ce5f0 msgr2=0x7fe1680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.231 7fe16e33e700 1 -- 172.21.15.32:0/643779414 shutdown_connections 2021-04-09T18:25:36.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.231 7fe16e33e700 1 -- 172.21.15.32:0/643779414 wait complete. 2021-04-09T18:25:36.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.251 7fe16e33e700 1 Processor -- start 2021-04-09T18:25:36.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.251 7fe16e33e700 1 -- start start 2021-04-09T18:25:36.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16e33e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1681aca20 0x7fe1681ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16e33e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe16818bd40 0x7fe16818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16e33e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe16808fba0 0x7fe16808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16e33e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16818c7e0 con 0x7fe16808fba0 2021-04-09T18:25:36.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16e33e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16818ca30 con 0x7fe16818bd40 2021-04-09T18:25:36.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16e33e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680905f0 con 0x7fe1681aca20 2021-04-09T18:25:36.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16c95b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe16808fba0 0x7fe16808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16c95b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe16808fba0 0x7fe16808ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52510/0 (socket says 172.21.15.32:52510) 2021-04-09T18:25:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16c95b700 1 -- 172.21.15.32:0/3683214709 learned_addr learned my addr 172.21.15.32:0/3683214709 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe167fff700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1681aca20 0x7fe1681ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe15ffff700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe16818bd40 0x7fe16818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16c95b700 1 -- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1681aca20 msgr2=0x7fe1681ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16c95b700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1681aca20 0x7fe1681ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16c95b700 1 -- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe16818bd40 msgr2=0x7fe16818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16c95b700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe16818bd40 0x7fe16818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16c95b700 1 -- 172.21.15.32:0/3683214709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680b4300 con 0x7fe16808fba0 2021-04-09T18:25:36.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.252 7fe16c95b700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe16808fba0 0x7fe16808ffa0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 rx=0x7fe1600099a0 tx=0x7fe16000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:36.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.253 7fe1657fa700 1 -- 172.21.15.32:0/3683214709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1600066f0 con 0x7fe16808fba0 2021-04-09T18:25:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.253 7fe1657fa700 1 -- 172.21.15.32:0/3683214709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe160003030 con 0x7fe16808fba0 2021-04-09T18:25:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.253 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1680b4300 con 0x7fe16808fba0 2021-04-09T18:25:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.253 7fe1657fa700 1 -- 172.21.15.32:0/3683214709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe160001470 con 0x7fe16808fba0 2021-04-09T18:25:36.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.253 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1680b4300 con 0x7fe16808fba0 2021-04-09T18:25:36.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.253 7fe1657fa700 1 -- 172.21.15.32:0/3683214709 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe1600011f0 con 0x7fe16808fba0 2021-04-09T18:25:36.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.253 7fe1657fa700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe14c018930 0x7fe14c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.253 7fe1657fa700 1 -- 172.21.15.32:0/3683214709 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe160032da0 con 0x7fe16808fba0 2021-04-09T18:25:36.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.254 7fe167fff700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe14c018930 0x7fe14c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.254 7fe167fff700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe14c018930 0x7fe14c01adc0 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:36.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.273 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe168182d70 con 0x7fe16808fba0 2021-04-09T18:25:36.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.277 7fe1657fa700 1 -- 172.21.15.32:0/3683214709 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe16000be30 con 0x7fe16808fba0 2021-04-09T18:25:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.467 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe16818c760 con 0x7fe16808fba0 2021-04-09T18:25:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.468 7fe1657fa700 1 -- 172.21.15.32:0/3683214709 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fe16001fae0 con 0x7fe16808fba0 2021-04-09T18:25:36.535 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:36.535 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":131,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:25:34.289498","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:25:30.677877","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-04-09 18:25:32.094578","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:25:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe14c018930 msgr2=0x7fe14c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe14c018930 0x7fe14c01adc0 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe16808fba0 msgr2=0x7fe16808ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe16808fba0 0x7fe16808ffa0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 rx=0x7fe1600099a0 tx=0x7fe16000bcf0).stop 2021-04-09T18:25:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 shutdown_connections 2021-04-09T18:25:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe14c018930 0x7fe14c01adc0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe16808fba0 0x7fe16808ffa0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe16818bd40 0x7fe16818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 --2- 172.21.15.32:0/3683214709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1681aca20 0x7fe1681ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 >> 172.21.15.32:0/3683214709 conn(0x7fe16819b460 msgr2=0x7fe16819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:36.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 shutdown_connections 2021-04-09T18:25:36.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.470 7fe16e33e700 1 -- 172.21.15.32:0/3683214709 wait complete. 2021-04-09T18:25:36.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:25:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.641 7fee4172a700 1 Processor -- start 2021-04-09T18:25:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.641 7fee4172a700 1 -- start start 2021-04-09T18:25:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee4172a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c0cffb0 0x7fee3c087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee4172a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee3c0d8050 0x7fee3c10d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee4172a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c088460 con 0x7fee3c0d7c50 2021-04-09T18:25:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee4172a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3c1b0c80 con 0x7fee3c0d8050 2021-04-09T18:25:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee4172a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0d91f0 con 0x7fee3c0cffb0 2021-04-09T18:25:36.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee3bac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee3c0d8050 0x7fee3c10d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee3b2bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c0cffb0 0x7fee3c087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee3bac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee3c0d8050 0x7fee3c10d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52516/0 (socket says 172.21.15.32:52516) 2021-04-09T18:25:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee3bac0700 1 -- 172.21.15.32:0/3322292028 learned_addr learned my addr 172.21.15.32:0/3322292028 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.642 7fee3bac0700 1 -- 172.21.15.32:0/3322292028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c0cffb0 msgr2=0x7fee3c087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:36.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee3bac0700 1 --2- 172.21.15.32:0/3322292028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c0cffb0 0x7fee3c087e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee3bac0700 1 -- 172.21.15.32:0/3322292028 >> v1:172.21.15.86:6789/0 conn(0x7fee3c0d7c50 legacy=0x7fee3c1b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee3a2bd700 1 -- 172.21.15.32:0/3322292028 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336941811 0 0) 0x7fee28000ff0 con 0x7fee3c0d7c50 2021-04-09T18:25:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee3bac0700 1 -- 172.21.15.32:0/3322292028 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c0d8620 con 0x7fee3c0d8050 2021-04-09T18:25:36.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee3bac0700 1 --2- 172.21.15.32:0/3322292028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0d8050 0x7fee3c10d200 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7fee34006f00 tx=0x7fee34009f10).ready entity=mon.0 client_cookie=d98349b70eaf6380 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:36.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee3a2bd700 1 -- 172.21.15.32:0/3322292028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee3400b3b0 con 0x7fee3c0d8050 2021-04-09T18:25:36.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee3a2bd700 1 -- 172.21.15.32:0/3322292028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3400b730 con 0x7fee3c0d8050 2021-04-09T18:25:36.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee3a2bd700 1 -- 172.21.15.32:0/3322292028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee34000f90 con 0x7fee3c0d8050 2021-04-09T18:25:36.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee4172a700 1 -- 172.21.15.32:0/3322292028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0d8050 msgr2=0x7fee3c10d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.643 7fee4172a700 1 --2- 172.21.15.32:0/3322292028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0d8050 0x7fee3c10d200 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7fee34006f00 tx=0x7fee34009f10).stop 2021-04-09T18:25:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.644 7fee4172a700 1 -- 172.21.15.32:0/3322292028 shutdown_connections 2021-04-09T18:25:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.644 7fee4172a700 1 --2- 172.21.15.32:0/3322292028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0d8050 0x7fee3c10d200 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.644 7fee4172a700 1 --2- 172.21.15.32:0/3322292028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c0cffb0 0x7fee3c087e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.644 7fee4172a700 1 -- 172.21.15.32:0/3322292028 >> 172.21.15.32:0/3322292028 conn(0x7fee3c0ce5b0 msgr2=0x7fee3c0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:36.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.644 7fee4172a700 1 -- 172.21.15.32:0/3322292028 shutdown_connections 2021-04-09T18:25:36.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.644 7fee4172a700 1 -- 172.21.15.32:0/3322292028 wait complete. 2021-04-09T18:25:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.664 7fee4172a700 1 Processor -- start 2021-04-09T18:25:36.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.664 7fee4172a700 1 -- start start 2021-04-09T18:25:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee4172a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0c4080 0x7fee3c0958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee4172a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c094f20 0x7fee3c182540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee4172a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3c095320 0x7fee3c182b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee4172a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c1832c0 con 0x7fee3c0c4080 2021-04-09T18:25:36.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee4172a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c181fc0 con 0x7fee3c095320 2021-04-09T18:25:36.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee4172a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0c4480 con 0x7fee3c094f20 2021-04-09T18:25:36.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0c4080 0x7fee3c0958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0c4080 0x7fee3c0958d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52520/0 (socket says 172.21.15.32:52520) 2021-04-09T18:25:36.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3b2bf700 1 -- 172.21.15.32:0/425390265 learned_addr learned my addr 172.21.15.32:0/425390265 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3aabe700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c094f20 0x7fee3c182540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3bac0700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3c095320 0x7fee3c182b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3b2bf700 1 -- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c094f20 msgr2=0x7fee3c182540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3b2bf700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c094f20 0x7fee3c182540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3b2bf700 1 -- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3c095320 msgr2=0x7fee3c182b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3b2bf700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3c095320 0x7fee3c182b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3b2bf700 1 -- 172.21.15.32:0/425390265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c0bc4a0 con 0x7fee3c0c4080 2021-04-09T18:25:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3aabe700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c094f20 0x7fee3c182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:36.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3bac0700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3c095320 0x7fee3c182b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee3b2bf700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0c4080 0x7fee3c0958d0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 rx=0x7fee2c006f60 tx=0x7fee2c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee33fff700 1 -- 172.21.15.32:0/425390265 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee2c0009b0 con 0x7fee3c0c4080 2021-04-09T18:25:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee33fff700 1 -- 172.21.15.32:0/425390265 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2c00bd60 con 0x7fee3c0c4080 2021-04-09T18:25:36.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee4172a700 1 -- 172.21.15.32:0/425390265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3c0bc4a0 con 0x7fee3c0c4080 2021-04-09T18:25:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee33fff700 1 -- 172.21.15.32:0/425390265 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee2c002030 con 0x7fee3c0c4080 2021-04-09T18:25:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.665 7fee4172a700 1 -- 172.21.15.32:0/425390265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3c0bc4a0 con 0x7fee3c0c4080 2021-04-09T18:25:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.666 7fee33fff700 1 -- 172.21.15.32:0/425390265 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fee2c000c90 con 0x7fee3c0c4080 2021-04-09T18:25:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.666 7fee33fff700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee180188c0 0x7fee1801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:36.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.666 7fee33fff700 1 -- 172.21.15.32:0/425390265 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fee2c0327e0 con 0x7fee3c0c4080 2021-04-09T18:25:36.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.667 7fee3aabe700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee180188c0 0x7fee1801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:36.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.667 7fee3aabe700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee180188c0 0x7fee1801adc0 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:36.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.686 7fee4172a700 1 -- 172.21.15.32:0/425390265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3c093ff0 con 0x7fee3c0c4080 2021-04-09T18:25:36.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.690 7fee33fff700 1 -- 172.21.15.32:0/425390265 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fee2c006b00 con 0x7fee3c0c4080 2021-04-09T18:25:36.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.873 7fee4172a700 1 -- 172.21.15.32:0/425390265 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fee3c0d8620 con 0x7fee3c0c4080 2021-04-09T18:25:36.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.877 7fee33fff700 1 -- 172.21.15.32:0/425390265 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v32) v1 ==== 184+0+0 (secure 0 0 0) 0x7fee2c01f740 con 0x7fee3c0c4080 2021-04-09T18:25:36.947 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:25:36.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.882 7fee4172a700 1 -- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee180188c0 msgr2=0x7fee1801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.882 7fee4172a700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee180188c0 0x7fee1801adc0 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.882 7fee4172a700 1 -- 172.21.15.32:0/425390265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0c4080 msgr2=0x7fee3c0958d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.882 7fee4172a700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0c4080 0x7fee3c0958d0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 rx=0x7fee2c006f60 tx=0x7fee2c00b970).stop 2021-04-09T18:25:36.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.883 7fee4172a700 1 -- 172.21.15.32:0/425390265 shutdown_connections 2021-04-09T18:25:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.883 7fee4172a700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee180188c0 0x7fee1801adc0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.883 7fee4172a700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3c095320 0x7fee3c182b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.883 7fee4172a700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee3c094f20 0x7fee3c182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.883 7fee4172a700 1 --2- 172.21.15.32:0/425390265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee3c0c4080 0x7fee3c0958d0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:36.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.883 7fee4172a700 1 -- 172.21.15.32:0/425390265 >> 172.21.15.32:0/425390265 conn(0x7fee3c099610 msgr2=0x7fee3c0acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:36.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.883 7fee4172a700 1 -- 172.21.15.32:0/425390265 shutdown_connections 2021-04-09T18:25:36.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:36.883 7fee4172a700 1 -- 172.21.15.32:0/425390265 wait complete. 2021-04-09T18:25:36.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:25:36.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:25:37.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:25:37.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.055 7f001fe87700 1 Processor -- start 2021-04-09T18:25:37.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.055 7f001fe87700 1 -- start start 2021-04-09T18:25:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001fe87700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f00180d2700 0x7f00180cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001fe87700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00180f28c0 0x7f00180d7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001fe87700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00180d8700 con 0x7f00180d8270 2021-04-09T18:25:37.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001fe87700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00180d91b0 con 0x7f00180f28c0 2021-04-09T18:25:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001fe87700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00181b0ce0 con 0x7f00180d2700 2021-04-09T18:25:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001d4a2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f00180f28c0 0x7f00180d7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001e4a4700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f00180d8270 0x7f00181b04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33888/0 (socket says 172.21.15.32:33888) 2021-04-09T18:25:37.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001e4a4700 1 -- 172.21.15.32:0/2302574045 learned_addr learned my addr 172.21.15.32:0/2302574045 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001cca1700 1 -- 172.21.15.32:0/2302574045 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111557105 0 0) 0x7f00100009b0 con 0x7f00180d8270 2021-04-09T18:25:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001dca3700 1 --2- 172.21.15.32:0/2302574045 >> v2:172.21.15.86:3300/0 conn(0x7f00180d2700 0x7f00180cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001cca1700 1 -- 172.21.15.32:0/2302574045 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0004002fe0 con 0x7f00180d8270 2021-04-09T18:25:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001d4a2700 1 -- 172.21.15.32:0/2302574045 >> v2:172.21.15.86:3300/0 conn(0x7f00180d2700 msgr2=0x7f00180cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001d4a2700 1 --2- 172.21.15.32:0/2302574045 >> v2:172.21.15.86:3300/0 conn(0x7f00180d2700 0x7f00180cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001d4a2700 1 -- 172.21.15.32:0/2302574045 >> v1:172.21.15.32:6789/0 conn(0x7f00180d8270 legacy=0x7f00181b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001d4a2700 1 -- 172.21.15.32:0/2302574045 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0018112090 con 0x7f00180f28c0 2021-04-09T18:25:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.056 7f001d4a2700 1 --2- 172.21.15.32:0/2302574045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180f28c0 0x7f00180d7c50 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 rx=0x7f0008006650 tx=0x7f0008009a80).ready entity=mon.0 client_cookie=94c5da8440055c0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.057 7f001cca1700 1 -- 172.21.15.32:0/2302574045 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00080009b0 con 0x7f00180f28c0 2021-04-09T18:25:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.057 7f001cca1700 1 -- 172.21.15.32:0/2302574045 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000800c0c0 con 0x7f00180f28c0 2021-04-09T18:25:37.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.057 7f001cca1700 1 -- 172.21.15.32:0/2302574045 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f000800c510 con 0x7f00180f28c0 2021-04-09T18:25:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.057 7f001fe87700 1 -- 172.21.15.32:0/2302574045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180f28c0 msgr2=0x7f00180d7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.057 7f001fe87700 1 --2- 172.21.15.32:0/2302574045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180f28c0 0x7f00180d7c50 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 rx=0x7f0008006650 tx=0x7f0008009a80).stop 2021-04-09T18:25:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.057 7f001fe87700 1 -- 172.21.15.32:0/2302574045 shutdown_connections 2021-04-09T18:25:37.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.057 7f001fe87700 1 --2- 172.21.15.32:0/2302574045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180f28c0 0x7f00180d7c50 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.057 7f001fe87700 1 --2- 172.21.15.32:0/2302574045 >> v2:172.21.15.86:3300/0 conn(0x7f00180d2700 0x7f00180cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.057 7f001fe87700 1 -- 172.21.15.32:0/2302574045 >> 172.21.15.32:0/2302574045 conn(0x7f00180ce590 msgr2=0x7f00180cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.058 7f001fe87700 1 -- 172.21.15.32:0/2302574045 shutdown_connections 2021-04-09T18:25:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.058 7f001fe87700 1 -- 172.21.15.32:0/2302574045 wait complete. 2021-04-09T18:25:37.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001fe87700 1 Processor -- start 2021-04-09T18:25:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001fe87700 1 -- start start 2021-04-09T18:25:37.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001fe87700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0018104b80 0x7f0018105250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001fe87700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00180ae380 0x7f00180ab800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001fe87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180ac6b0 0x7f00180acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001fe87700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00180ac130 con 0x7f00180ac6b0 2021-04-09T18:25:37.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001fe87700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0018104f80 con 0x7f0018104b80 2021-04-09T18:25:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001fe87700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00180abe20 con 0x7f00180ae380 2021-04-09T18:25:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001e4a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180ac6b0 0x7f00180acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001e4a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180ac6b0 0x7f00180acab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52538/0 (socket says 172.21.15.32:52538) 2021-04-09T18:25:37.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.078 7f001e4a4700 1 -- 172.21.15.32:0/3948121627 learned_addr learned my addr 172.21.15.32:0/3948121627 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001dca3700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0018104b80 0x7f0018105250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001d4a2700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00180ae380 0x7f00180ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001e4a4700 1 -- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00180ae380 msgr2=0x7f00180ab800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001e4a4700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00180ae380 0x7f00180ab800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001e4a4700 1 -- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0018104b80 msgr2=0x7f0018105250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001e4a4700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0018104b80 0x7f0018105250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001e4a4700 1 -- 172.21.15.32:0/3948121627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180ad0d0 con 0x7f00180ac6b0 2021-04-09T18:25:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001dca3700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0018104b80 0x7f0018105250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:37.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001d4a2700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00180ae380 0x7f00180ab800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:25:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001e4a4700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180ac6b0 0x7f00180acab0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7f001000b5c0 tx=0x7f001000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f000e7fc700 1 -- 172.21.15.32:0/3948121627 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00100009b0 con 0x7f00180ac6b0 2021-04-09T18:25:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f000e7fc700 1 -- 172.21.15.32:0/3948121627 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0010006030 con 0x7f00180ac6b0 2021-04-09T18:25:37.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f000e7fc700 1 -- 172.21.15.32:0/3948121627 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0010004620 con 0x7f00180ac6b0 2021-04-09T18:25:37.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001fe87700 1 -- 172.21.15.32:0/3948121627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00180ad0d0 con 0x7f00180ac6b0 2021-04-09T18:25:37.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.079 7f001fe87700 1 -- 172.21.15.32:0/3948121627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00180ad0d0 con 0x7f00180ac6b0 2021-04-09T18:25:37.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.080 7f000e7fc700 1 -- 172.21.15.32:0/3948121627 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0010006210 con 0x7f00180ac6b0 2021-04-09T18:25:37.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.080 7f000e7fc700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0000018930 0x7f000001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.080 7f000e7fc700 1 -- 172.21.15.32:0/3948121627 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f00100329d0 con 0x7f00180ac6b0 2021-04-09T18:25:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.080 7f001dca3700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0000018930 0x7f000001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.080 7f001dca3700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0000018930 0x7f000001adc0 crc :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:37.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.100 7f001fe87700 1 -- 172.21.15.32:0/3948121627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0018103980 con 0x7f00180ac6b0 2021-04-09T18:25:37.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.103 7f000e7fc700 1 -- 172.21.15.32:0/3948121627 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0010006b80 con 0x7f00180ac6b0 2021-04-09T18:25:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.287 7f001fe87700 1 -- 172.21.15.32:0/3948121627 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0018103980 con 0x7f00180ac6b0 2021-04-09T18:25:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.290 7f000e7fc700 1 -- 172.21.15.32:0/3948121627 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v33) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0010006030 con 0x7f00180ac6b0 2021-04-09T18:25:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:25:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 -- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0000018930 msgr2=0x7f000001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0000018930 0x7f000001adc0 crc :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 -- 172.21.15.32:0/3948121627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180ac6b0 msgr2=0x7f00180acab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180ac6b0 0x7f00180acab0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7f001000b5c0 tx=0x7f001000b960).stop 2021-04-09T18:25:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 -- 172.21.15.32:0/3948121627 shutdown_connections 2021-04-09T18:25:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0000018930 0x7f000001adc0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00180ac6b0 0x7f00180acab0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00180ae380 0x7f00180ab800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 --2- 172.21.15.32:0/3948121627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0018104b80 0x7f0018105250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 -- 172.21.15.32:0/3948121627 >> 172.21.15.32:0/3948121627 conn(0x7f0018098e80 msgr2=0x7f00180a8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.296 7f001fe87700 1 -- 172.21.15.32:0/3948121627 shutdown_connections 2021-04-09T18:25:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.297 7f001fe87700 1 -- 172.21.15.32:0/3948121627 wait complete. 2021-04-09T18:25:37.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.455 7fed2f4b7700 1 Processor -- start 2021-04-09T18:25:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.455 7fed2f4b7700 1 -- start start 2021-04-09T18:25:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2f4b7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fed280d7ca0 0x7fed280d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2f4b7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fed2810da00 0x7fed2810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2f4b7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2810d480 con 0x7fed2810d010 2021-04-09T18:25:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2f4b7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed280d9220 con 0x7fed280d7ca0 2021-04-09T18:25:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2f4b7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed280d86c0 con 0x7fed2810da00 2021-04-09T18:25:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2cad2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fed280d7ca0 0x7fed280d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2d2d3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fed2810d010 0x7fed280d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33900/0 (socket says 172.21.15.32:33900) 2021-04-09T18:25:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2d2d3700 1 -- 172.21.15.32:0/1254737497 learned_addr learned my addr 172.21.15.32:0/1254737497 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2dad4700 1 --2- 172.21.15.32:0/1254737497 >> v2:172.21.15.86:3300/0 conn(0x7fed2810da00 0x7fed2810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed1bfff700 1 -- 172.21.15.32:0/1254737497 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915008203 0 0) 0x7fed24000ce0 con 0x7fed2810d010 2021-04-09T18:25:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed1bfff700 1 -- 172.21.15.32:0/1254737497 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed14002fe0 con 0x7fed2810d010 2021-04-09T18:25:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2cad2700 1 -- 172.21.15.32:0/1254737497 >> v2:172.21.15.86:3300/0 conn(0x7fed2810da00 msgr2=0x7fed2810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2cad2700 1 --2- 172.21.15.32:0/1254737497 >> v2:172.21.15.86:3300/0 conn(0x7fed2810da00 0x7fed2810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2cad2700 1 -- 172.21.15.32:0/1254737497 >> v1:172.21.15.32:6789/0 conn(0x7fed2810d010 legacy=0x7fed280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.456 7fed2cad2700 1 -- 172.21.15.32:0/1254737497 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed280d1f70 con 0x7fed280d7ca0 2021-04-09T18:25:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.457 7fed2cad2700 1 --2- 172.21.15.32:0/1254737497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed280d7ca0 0x7fed280d80a0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7fed1c006650 tx=0x7fed1c009a80).ready entity=mon.0 client_cookie=82e4e070f5050579 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.457 7fed1bfff700 1 -- 172.21.15.32:0/1254737497 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed1c0009b0 con 0x7fed280d7ca0 2021-04-09T18:25:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.457 7fed1bfff700 1 -- 172.21.15.32:0/1254737497 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c00c0c0 con 0x7fed280d7ca0 2021-04-09T18:25:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.457 7fed1bfff700 1 -- 172.21.15.32:0/1254737497 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed1c00c510 con 0x7fed280d7ca0 2021-04-09T18:25:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.457 7fed2f4b7700 1 -- 172.21.15.32:0/1254737497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed280d7ca0 msgr2=0x7fed280d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.457 7fed2f4b7700 1 --2- 172.21.15.32:0/1254737497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed280d7ca0 0x7fed280d80a0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7fed1c006650 tx=0x7fed1c009a80).stop 2021-04-09T18:25:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.458 7fed2f4b7700 1 -- 172.21.15.32:0/1254737497 shutdown_connections 2021-04-09T18:25:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.458 7fed2f4b7700 1 --2- 172.21.15.32:0/1254737497 >> v2:172.21.15.86:3300/0 conn(0x7fed2810da00 0x7fed2810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.458 7fed2f4b7700 1 --2- 172.21.15.32:0/1254737497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed280d7ca0 0x7fed280d80a0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.458 7fed2f4b7700 1 -- 172.21.15.32:0/1254737497 >> 172.21.15.32:0/1254737497 conn(0x7fed280ce5f0 msgr2=0x7fed280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.458 7fed2f4b7700 1 -- 172.21.15.32:0/1254737497 shutdown_connections 2021-04-09T18:25:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.458 7fed2f4b7700 1 -- 172.21.15.32:0/1254737497 wait complete. 2021-04-09T18:25:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.478 7fed2f4b7700 1 Processor -- start 2021-04-09T18:25:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.478 7fed2f4b7700 1 -- start start 2021-04-09T18:25:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.478 7fed2f4b7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed281ac380 0x7fed2818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.478 7fed2f4b7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed281ac780 0x7fed2818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.478 7fed2f4b7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed28183520 0x7fed28183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.478 7fed2f4b7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed281ace10 con 0x7fed28183520 2021-04-09T18:25:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.478 7fed2f4b7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed28189cf0 con 0x7fed281ac380 2021-04-09T18:25:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.478 7fed2f4b7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed28183f40 con 0x7fed281ac780 2021-04-09T18:25:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2dad4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed28183520 0x7fed28183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2dad4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed28183520 0x7fed28183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52550/0 (socket says 172.21.15.32:52550) 2021-04-09T18:25:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2dad4700 1 -- 172.21.15.32:0/3009993153 learned_addr learned my addr 172.21.15.32:0/3009993153 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2cad2700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed281ac780 0x7fed2818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2d2d3700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed281ac380 0x7fed2818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2dad4700 1 -- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed281ac780 msgr2=0x7fed2818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2dad4700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed281ac780 0x7fed2818a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2dad4700 1 -- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed281ac380 msgr2=0x7fed2818a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2dad4700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed281ac380 0x7fed2818a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2dad4700 1 -- 172.21.15.32:0/3009993153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed28184160 con 0x7fed28183520 2021-04-09T18:25:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2dad4700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed28183520 0x7fed28183920 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 rx=0x7fed2000b970 tx=0x7fed2000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed19ffb700 1 -- 172.21.15.32:0/3009993153 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed20000f90 con 0x7fed28183520 2021-04-09T18:25:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed19ffb700 1 -- 172.21.15.32:0/3009993153 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed20006aa0 con 0x7fed28183520 2021-04-09T18:25:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed28184160 con 0x7fed28183520 2021-04-09T18:25:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed19ffb700 1 -- 172.21.15.32:0/3009993153 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed200015f0 con 0x7fed28183520 2021-04-09T18:25:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.479 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed28184160 con 0x7fed28183520 2021-04-09T18:25:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.480 7fed19ffb700 1 -- 172.21.15.32:0/3009993153 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fed20000ed0 con 0x7fed28183520 2021-04-09T18:25:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.480 7fed19ffb700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed08018930 0x7fed0801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.480 7fed19ffb700 1 -- 172.21.15.32:0/3009993153 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fed20032da0 con 0x7fed28183520 2021-04-09T18:25:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.480 7fed2d2d3700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed08018930 0x7fed0801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.480 7fed2d2d3700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed08018930 0x7fed0801ae10 crc :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:37.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.500 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed28189cf0 con 0x7fed28183520 2021-04-09T18:25:37.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.503 7fed19ffb700 1 -- 172.21.15.32:0/3009993153 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fed200066f0 con 0x7fed28183520 2021-04-09T18:25:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.695 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed28189cf0 con 0x7fed28183520 2021-04-09T18:25:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.695 7fed19ffb700 1 -- 172.21.15.32:0/3009993153 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2785 (secure 0 0 0) 0x7fed20006aa0 con 0x7fed28183520 2021-04-09T18:25:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 268 2021-04-09T18:25:37.764 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:37.764 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":268,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:25:35.619821","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:25:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed08018930 msgr2=0x7fed0801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed08018930 0x7fed0801ae10 crc :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed28183520 msgr2=0x7fed28183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed28183520 0x7fed28183920 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 rx=0x7fed2000b970 tx=0x7fed2000bd10).stop 2021-04-09T18:25:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 shutdown_connections 2021-04-09T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed08018930 0x7fed0801ae10 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed28183520 0x7fed28183920 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed281ac780 0x7fed2818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 --2- 172.21.15.32:0/3009993153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed281ac380 0x7fed2818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 >> 172.21.15.32:0/3009993153 conn(0x7fed2819b890 msgr2=0x7fed281a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 shutdown_connections 2021-04-09T18:25:37.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:37.698 7fed2f4b7700 1 -- 172.21.15.32:0/3009993153 wait complete. 2021-04-09T18:25:38.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 268, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:25:33.288479', 'modified': '2021-04-09 18:25:35.619821', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9085': {'gid': 9085, 'name': 'd', 'rank': 0, 'incarnation': 265, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3538582472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3538582472}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3538582472}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:38.179 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:25:38.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:25:38.305 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:25:38.306 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:25:38.306 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:25:38.306 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 9085, 2021-04-09T18:25:38.306 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:25:38.307 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:25:38.307 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.9493254159999998, 2021-04-09T18:25:38.307 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 268, 2021-04-09T18:25:38.307 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 131, 2021-04-09T18:25:38.308 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 131, 2021-04-09T18:25:38.308 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10.397485122000001 2021-04-09T18:25:38.308 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:25:38.311 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 9085, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.9493254159999998, "mdsmap_epoch": 268, "osdmap_epoch": 131, "osdmap_epoch_barrier": 131, "uptime": 10.397485122000001 } 2021-04-09T18:25:38.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:25:38.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.393 7f19370bd700 1 Processor -- start 2021-04-09T18:25:38.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.393 7f19370bd700 1 -- start start 2021-04-09T18:25:38.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.393 7f19370bd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19300d7ca0 0x7f19300d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.393 7f19370bd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193010d480 con 0x7f193010da00 2021-04-09T18:25:38.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.393 7f19370bd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19300d86c0 con 0x7f193010d010 2021-04-09T18:25:38.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.393 7f19370bd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19300d9220 con 0x7f19300d7ca0 2021-04-09T18:25:38.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f19356da700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f193010da00 0x7f193010de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33912/0 (socket says 172.21.15.32:33912) 2021-04-09T18:25:38.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f19356da700 1 -- 172.21.15.32:0/3149481142 learned_addr learned my addr 172.21.15.32:0/3149481142 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:38.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192ffff700 1 --2- 172.21.15.32:0/3149481142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19300d7ca0 0x7f19300d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192f7fe700 1 -- 172.21.15.32:0/3149481142 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215338610 0 0) 0x7f19280009b0 con 0x7f193010da00 2021-04-09T18:25:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192f7fe700 1 -- 172.21.15.32:0/3149481142 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f191c002fe0 con 0x7f193010da00 2021-04-09T18:25:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192f7fe700 1 -- 172.21.15.32:0/3149481142 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682155794 0 0) 0x7f1918000c60 con 0x7f193010d010 2021-04-09T18:25:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192f7fe700 1 -- 172.21.15.32:0/3149481142 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f191c003210 con 0x7f193010d010 2021-04-09T18:25:38.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192f7fe700 1 -- 172.21.15.32:0/3149481142 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1403354120 0 0) 0x7f1928000ef0 con 0x7f193010da00 2021-04-09T18:25:38.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192ffff700 1 -- 172.21.15.32:0/3149481142 >> v1:172.21.15.86:6789/0 conn(0x7f193010d010 legacy=0x7f19300d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192ffff700 1 -- 172.21.15.32:0/3149481142 >> v1:172.21.15.32:6789/0 conn(0x7f193010da00 legacy=0x7f193010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192ffff700 1 -- 172.21.15.32:0/3149481142 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19300d1f70 con 0x7f19300d7ca0 2021-04-09T18:25:38.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192ffff700 1 --2- 172.21.15.32:0/3149481142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19300d7ca0 0x7f19300d80a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f19200069d0 tx=0x7f1920009e40).ready entity=mon.2 client_cookie=d1587ec94c7929f9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:38.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192f7fe700 1 -- 172.21.15.32:0/3149481142 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f192000b510 con 0x7f19300d7ca0 2021-04-09T18:25:38.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192f7fe700 1 -- 172.21.15.32:0/3149481142 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192000b890 con 0x7f19300d7ca0 2021-04-09T18:25:38.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.394 7f192f7fe700 1 -- 172.21.15.32:0/3149481142 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f192000bd70 con 0x7f19300d7ca0 2021-04-09T18:25:38.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.395 7f19370bd700 1 -- 172.21.15.32:0/3149481142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19300d7ca0 msgr2=0x7f19300d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.395 7f19370bd700 1 --2- 172.21.15.32:0/3149481142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19300d7ca0 0x7f19300d80a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f19200069d0 tx=0x7f1920009e40).stop 2021-04-09T18:25:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.395 7f19370bd700 1 -- 172.21.15.32:0/3149481142 shutdown_connections 2021-04-09T18:25:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.395 7f19370bd700 1 --2- 172.21.15.32:0/3149481142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19300d7ca0 0x7f19300d80a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.395 7f19370bd700 1 -- 172.21.15.32:0/3149481142 >> 172.21.15.32:0/3149481142 conn(0x7f19300ce5f0 msgr2=0x7f19300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:38.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.395 7f19370bd700 1 -- 172.21.15.32:0/3149481142 shutdown_connections 2021-04-09T18:25:38.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.395 7f19370bd700 1 -- 172.21.15.32:0/3149481142 wait complete. 2021-04-09T18:25:38.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.416 7f19370bd700 1 Processor -- start 2021-04-09T18:25:38.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.416 7f19370bd700 1 -- start start 2021-04-09T18:25:38.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.416 7f19370bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19301aca20 0x7f19301ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.416 7f19370bd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f193018bd40 0x7f193018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.416 7f19370bd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f193008fba0 0x7f193008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.416 7f19370bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193018c7e0 con 0x7f19301aca20 2021-04-09T18:25:38.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.416 7f19370bd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193018ca30 con 0x7f193018bd40 2021-04-09T18:25:38.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.416 7f19370bd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19300905f0 con 0x7f193008fba0 2021-04-09T18:25:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f1934ed9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19301aca20 0x7f19301ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f1934ed9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19301aca20 0x7f19301ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52558/0 (socket says 172.21.15.32:52558) 2021-04-09T18:25:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f1934ed9700 1 -- 172.21.15.32:0/2106310655 learned_addr learned my addr 172.21.15.32:0/2106310655 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:38.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f192ffff700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f193018bd40 0x7f193018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f19356da700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f193008fba0 0x7f193008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f1934ed9700 1 -- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f193008fba0 msgr2=0x7f193008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f1934ed9700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f193008fba0 0x7f193008ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f1934ed9700 1 -- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f193018bd40 msgr2=0x7f193018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f1934ed9700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f193018bd40 0x7f193018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f1934ed9700 1 -- 172.21.15.32:0/2106310655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19300b4300 con 0x7f19301aca20 2021-04-09T18:25:38.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f1934ed9700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19301aca20 0x7f19301ace20 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 rx=0x7f191800b580 tx=0x7f191800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:38.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f192d7fa700 1 -- 172.21.15.32:0/2106310655 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f191800bd10 con 0x7f19301aca20 2021-04-09T18:25:38.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f192d7fa700 1 -- 172.21.15.32:0/2106310655 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1918000e20 con 0x7f19301aca20 2021-04-09T18:25:38.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f19370bd700 1 -- 172.21.15.32:0/2106310655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19300b4300 con 0x7f19301aca20 2021-04-09T18:25:38.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f192d7fa700 1 -- 172.21.15.32:0/2106310655 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19180054f0 con 0x7f19301aca20 2021-04-09T18:25:38.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.417 7f19370bd700 1 -- 172.21.15.32:0/2106310655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19300b4300 con 0x7f19301aca20 2021-04-09T18:25:38.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.418 7f192d7fa700 1 -- 172.21.15.32:0/2106310655 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1918009fe0 con 0x7f19301aca20 2021-04-09T18:25:38.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.418 7f192d7fa700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1914018930 0x7f191401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.418 7f192d7fa700 1 -- 172.21.15.32:0/2106310655 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1918032a50 con 0x7f19301aca20 2021-04-09T18:25:38.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.418 7f192ffff700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1914018930 0x7f191401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.418 7f192ffff700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1914018930 0x7f191401ae10 crc :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:38.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.438 7f19370bd700 1 -- 172.21.15.32:0/2106310655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1930182d70 con 0x7f19301aca20 2021-04-09T18:25:38.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.441 7f192d7fa700 1 -- 172.21.15.32:0/2106310655 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1918000e20 con 0x7f19301aca20 2021-04-09T18:25:38.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.629 7f19370bd700 1 -- 172.21.15.32:0/2106310655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f193018c760 con 0x7f19301aca20 2021-04-09T18:25:38.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.629 7f192d7fa700 1 -- 172.21.15.32:0/2106310655 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f1918009fe0 con 0x7f19301aca20 2021-04-09T18:25:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:38.698 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":131,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:25:34.289498","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:25:30.677877","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-04-09 18:25:32.094578","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 -- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1914018930 msgr2=0x7f191401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1914018930 0x7f191401ae10 crc :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 -- 172.21.15.32:0/2106310655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19301aca20 msgr2=0x7f19301ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19301aca20 0x7f19301ace20 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 rx=0x7f191800b580 tx=0x7f191800b920).stop 2021-04-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 -- 172.21.15.32:0/2106310655 shutdown_connections 2021-04-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1914018930 0x7f191401ae10 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f193008fba0 0x7f193008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f193018bd40 0x7f193018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 --2- 172.21.15.32:0/2106310655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19301aca20 0x7f19301ace20 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 -- 172.21.15.32:0/2106310655 >> 172.21.15.32:0/2106310655 conn(0x7f193019b460 msgr2=0x7f193019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 -- 172.21.15.32:0/2106310655 shutdown_connections 2021-04-09T18:25:38.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.632 7f19370bd700 1 -- 172.21.15.32:0/2106310655 wait complete. 2021-04-09T18:25:38.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:25:30.677877', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:25:32.094578', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:25:38.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:25:38.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.796 7fba26c77700 1 Processor -- start 2021-04-09T18:25:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.796 7fba26c77700 1 -- start start 2021-04-09T18:25:38.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.796 7fba26c77700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2010d010 0x7fba200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba26c77700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2010d480 con 0x7fba200d8490 2021-04-09T18:25:38.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba26c77700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2010e1f0 con 0x7fba200d1da0 2021-04-09T18:25:38.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba26c77700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2010e780 con 0x7fba2010d010 2021-04-09T18:25:38.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba25294700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fba200d8490 0x7fba2010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33926/0 (socket says 172.21.15.32:33926) 2021-04-09T18:25:38.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba25294700 1 -- 172.21.15.32:0/3596006931 learned_addr learned my addr 172.21.15.32:0/3596006931 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:38.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba24a93700 1 --2- 172.21.15.32:0/3596006931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2010d010 0x7fba200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba1f7fe700 1 -- 172.21.15.32:0/3596006931 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030117884 0 0) 0x7fba18000d30 con 0x7fba200d8490 2021-04-09T18:25:38.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba1f7fe700 1 -- 172.21.15.32:0/3596006931 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba10002fe0 con 0x7fba200d8490 2021-04-09T18:25:38.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba1f7fe700 1 -- 172.21.15.32:0/3596006931 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243409718 0 0) 0x7fba0c000c60 con 0x7fba200d1da0 2021-04-09T18:25:38.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba1f7fe700 1 -- 172.21.15.32:0/3596006931 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba10003210 con 0x7fba200d1da0 2021-04-09T18:25:38.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba24a93700 1 -- 172.21.15.32:0/3596006931 >> v1:172.21.15.86:6789/0 conn(0x7fba200d1da0 legacy=0x7fba200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba24a93700 1 -- 172.21.15.32:0/3596006931 >> v1:172.21.15.32:6789/0 conn(0x7fba200d8490 legacy=0x7fba2010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba24a93700 1 -- 172.21.15.32:0/3596006931 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba200d8d30 con 0x7fba2010d010 2021-04-09T18:25:38.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba24a93700 1 --2- 172.21.15.32:0/3596006931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2010d010 0x7fba200d1780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fba14006650 tx=0x7fba14009a80).ready entity=mon.2 client_cookie=d739cc064a1fbaa3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba1f7fe700 1 -- 172.21.15.32:0/3596006931 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba140009b0 con 0x7fba2010d010 2021-04-09T18:25:38.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba1f7fe700 1 -- 172.21.15.32:0/3596006931 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1400c0c0 con 0x7fba2010d010 2021-04-09T18:25:38.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba1f7fe700 1 -- 172.21.15.32:0/3596006931 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba1400c510 con 0x7fba2010d010 2021-04-09T18:25:38.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba26c77700 1 -- 172.21.15.32:0/3596006931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2010d010 msgr2=0x7fba200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.797 7fba26c77700 1 --2- 172.21.15.32:0/3596006931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2010d010 0x7fba200d1780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fba14006650 tx=0x7fba14009a80).stop 2021-04-09T18:25:38.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.798 7fba26c77700 1 -- 172.21.15.32:0/3596006931 shutdown_connections 2021-04-09T18:25:38.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.798 7fba26c77700 1 --2- 172.21.15.32:0/3596006931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2010d010 0x7fba200d1780 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.798 7fba26c77700 1 -- 172.21.15.32:0/3596006931 >> 172.21.15.32:0/3596006931 conn(0x7fba200ce5f0 msgr2=0x7fba200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:38.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.798 7fba26c77700 1 -- 172.21.15.32:0/3596006931 shutdown_connections 2021-04-09T18:25:38.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.798 7fba26c77700 1 -- 172.21.15.32:0/3596006931 wait complete. 2021-04-09T18:25:38.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.818 7fba26c77700 1 Processor -- start 2021-04-09T18:25:38.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.818 7fba26c77700 1 -- start start 2021-04-09T18:25:38.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba26c77700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba200c3c00 0x7fba2009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba26c77700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba20095240 0x7fba201a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba26c77700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba201a1270 0x7fba200b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba26c77700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2009cda0 con 0x7fba201a1270 2021-04-09T18:25:38.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba26c77700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba200c3680 con 0x7fba200c3c00 2021-04-09T18:25:38.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba26c77700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba20095640 con 0x7fba20095240 2021-04-09T18:25:38.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba25294700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba201a1270 0x7fba200b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba25294700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba201a1270 0x7fba200b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52576/0 (socket says 172.21.15.32:52576) 2021-04-09T18:25:38.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba25294700 1 -- 172.21.15.32:0/1340335735 learned_addr learned my addr 172.21.15.32:0/1340335735 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:38.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba1ffff700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba20095240 0x7fba201a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.819 7fba24a93700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba200c3c00 0x7fba2009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba25294700 1 -- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba20095240 msgr2=0x7fba201a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba25294700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba20095240 0x7fba201a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba25294700 1 -- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba200c3c00 msgr2=0x7fba2009c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:38.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba25294700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba200c3c00 0x7fba2009c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:38.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba25294700 1 -- 172.21.15.32:0/1340335735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba201a1670 con 0x7fba201a1270 2021-04-09T18:25:38.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba25294700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba201a1270 0x7fba200b0cf0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7fba18009980 tx=0x7fba1800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:38.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba1d7fa700 1 -- 172.21.15.32:0/1340335735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba1800be10 con 0x7fba201a1270 2021-04-09T18:25:38.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba1d7fa700 1 -- 172.21.15.32:0/1340335735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba18006030 con 0x7fba201a1270 2021-04-09T18:25:38.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba1d7fa700 1 -- 172.21.15.32:0/1340335735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba180064c0 con 0x7fba201a1270 2021-04-09T18:25:38.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba26c77700 1 -- 172.21.15.32:0/1340335735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba201a1670 con 0x7fba201a1270 2021-04-09T18:25:38.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.820 7fba26c77700 1 -- 172.21.15.32:0/1340335735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba201a1670 con 0x7fba201a1270 2021-04-09T18:25:38.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.821 7fba1d7fa700 1 -- 172.21.15.32:0/1340335735 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fba18006210 con 0x7fba201a1270 2021-04-09T18:25:38.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.821 7fba1d7fa700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba00018930 0x7fba0001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:38.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.821 7fba1d7fa700 1 -- 172.21.15.32:0/1340335735 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fba18032d60 con 0x7fba201a1270 2021-04-09T18:25:38.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.821 7fba24a93700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba00018930 0x7fba0001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:38.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.821 7fba24a93700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba00018930 0x7fba0001ae10 crc :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:38.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.841 7fba26c77700 1 -- 172.21.15.32:0/1340335735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba20093d50 con 0x7fba201a1270 2021-04-09T18:25:38.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:38.844 7fba1d7fa700 1 -- 172.21.15.32:0/1340335735 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fba18000f90 con 0x7fba201a1270 2021-04-09T18:25:39.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.025 7fba26c77700 1 -- 172.21.15.32:0/1340335735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fba2010d480 con 0x7fba201a1270 2021-04-09T18:25:39.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.026 7fba1d7fa700 1 -- 172.21.15.32:0/1340335735 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v131) v1 ==== 92+0+10 (secure 0 0 0) 0x7fba18006030 con 0x7fba201a1270 2021-04-09T18:25:39.094 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:39.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 -- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba00018930 msgr2=0x7fba0001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba00018930 0x7fba0001ae10 crc :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 -- 172.21.15.32:0/1340335735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba201a1270 msgr2=0x7fba200b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba201a1270 0x7fba200b0cf0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7fba18009980 tx=0x7fba1800bcd0).stop 2021-04-09T18:25:39.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 -- 172.21.15.32:0/1340335735 shutdown_connections 2021-04-09T18:25:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba00018930 0x7fba0001ae10 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba201a1270 0x7fba200b0cf0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba20095240 0x7fba201a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 --2- 172.21.15.32:0/1340335735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba200c3c00 0x7fba2009c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.028 7fba26c77700 1 -- 172.21.15.32:0/1340335735 >> 172.21.15.32:0/1340335735 conn(0x7fba200975f0 msgr2=0x7fba20096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.029 7fba26c77700 1 -- 172.21.15.32:0/1340335735 shutdown_connections 2021-04-09T18:25:39.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.029 7fba26c77700 1 -- 172.21.15.32:0/1340335735 wait complete. 2021-04-09T18:25:39.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:25:39.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.191 7f02425e4700 1 Processor -- start 2021-04-09T18:25:39.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.191 7f02425e4700 1 -- start start 2021-04-09T18:25:39.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f02425e4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f023c0d7ca0 0x7f023c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f02425e4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023c10d480 con 0x7f023c10d010 2021-04-09T18:25:39.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f02425e4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023c0d86c0 con 0x7f023c10da00 2021-04-09T18:25:39.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f02425e4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f023c0d9220 con 0x7f023c0d7ca0 2021-04-09T18:25:39.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f023bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f023c10d010 0x7f023c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33938/0 (socket says 172.21.15.32:33938) 2021-04-09T18:25:39.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f023bfff700 1 -- 172.21.15.32:0/3078859901 learned_addr learned my addr 172.21.15.32:0/3078859901 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:39.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f02337fe700 1 --2- 172.21.15.32:0/3078859901 >> v2:172.21.15.86:3300/0 conn(0x7f023c0d7ca0 0x7f023c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f023b7fe700 1 -- 172.21.15.32:0/3078859901 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954523721 0 0) 0x7f022c000ce0 con 0x7f023c10d010 2021-04-09T18:25:39.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f023b7fe700 1 -- 172.21.15.32:0/3078859901 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0228002fe0 con 0x7f023c10d010 2021-04-09T18:25:39.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f023b7fe700 1 -- 172.21.15.32:0/3078859901 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128855220 0 0) 0x7f0234000d30 con 0x7f023c10da00 2021-04-09T18:25:39.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f023b7fe700 1 -- 172.21.15.32:0/3078859901 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0228003210 con 0x7f023c10da00 2021-04-09T18:25:39.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f023b7fe700 1 -- 172.21.15.32:0/3078859901 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 600281124 0 0) 0x7f022c000ed0 con 0x7f023c10d010 2021-04-09T18:25:39.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f02337fe700 1 -- 172.21.15.32:0/3078859901 >> v1:172.21.15.86:6789/0 conn(0x7f023c10da00 legacy=0x7f023c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f02337fe700 1 -- 172.21.15.32:0/3078859901 >> v1:172.21.15.32:6789/0 conn(0x7f023c10d010 legacy=0x7f023c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.192 7f02337fe700 1 -- 172.21.15.32:0/3078859901 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c0d1f70 con 0x7f023c0d7ca0 2021-04-09T18:25:39.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.193 7f02337fe700 1 --2- 172.21.15.32:0/3078859901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f023c0d7ca0 0x7f023c0d80a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f0224006650 tx=0x7f0224009a80).ready entity=mon.1 client_cookie=5b1b508fd0205dad server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:39.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.193 7f023b7fe700 1 -- 172.21.15.32:0/3078859901 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02240009b0 con 0x7f023c0d7ca0 2021-04-09T18:25:39.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.193 7f023b7fe700 1 -- 172.21.15.32:0/3078859901 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022400c0c0 con 0x7f023c0d7ca0 2021-04-09T18:25:39.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.193 7f023b7fe700 1 -- 172.21.15.32:0/3078859901 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f022400c510 con 0x7f023c0d7ca0 2021-04-09T18:25:39.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.193 7f02425e4700 1 -- 172.21.15.32:0/3078859901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f023c0d7ca0 msgr2=0x7f023c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.193 7f02425e4700 1 --2- 172.21.15.32:0/3078859901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f023c0d7ca0 0x7f023c0d80a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f0224006650 tx=0x7f0224009a80).stop 2021-04-09T18:25:39.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.194 7f02425e4700 1 -- 172.21.15.32:0/3078859901 shutdown_connections 2021-04-09T18:25:39.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.194 7f02425e4700 1 --2- 172.21.15.32:0/3078859901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f023c0d7ca0 0x7f023c0d80a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.194 7f02425e4700 1 -- 172.21.15.32:0/3078859901 >> 172.21.15.32:0/3078859901 conn(0x7f023c0ce5f0 msgr2=0x7f023c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:39.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.194 7f02425e4700 1 -- 172.21.15.32:0/3078859901 shutdown_connections 2021-04-09T18:25:39.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.194 7f02425e4700 1 -- 172.21.15.32:0/3078859901 wait complete. 2021-04-09T18:25:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.214 7f02425e4700 1 Processor -- start 2021-04-09T18:25:39.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.214 7f02425e4700 1 -- start start 2021-04-09T18:25:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.214 7f02425e4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f023c09e530 0x7f023c09b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.214 7f02425e4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f023c09e930 0x7f023c09f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.214 7f02425e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f023c09f990 0x7f023c094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.214 7f02425e4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c09edf0 con 0x7f023c09f990 2021-04-09T18:25:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.214 7f02425e4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c09fd90 con 0x7f023c09e930 2021-04-09T18:25:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.214 7f02425e4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023c09bd40 con 0x7f023c09e530 2021-04-09T18:25:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f0240c01700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f023c09f990 0x7f023c094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f0240c01700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f023c09f990 0x7f023c094c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52588/0 (socket says 172.21.15.32:52588) 2021-04-09T18:25:39.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f0240c01700 1 -- 172.21.15.32:0/2092583692 learned_addr learned my addr 172.21.15.32:0/2092583692 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f023bfff700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f023c09e530 0x7f023c09b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f02337fe700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f023c09e930 0x7f023c09f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f0240c01700 1 -- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f023c09e530 msgr2=0x7f023c09b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f0240c01700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f023c09e530 0x7f023c09b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f0240c01700 1 -- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f023c09e930 msgr2=0x7f023c09f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f0240c01700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f023c09e930 0x7f023c09f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f0240c01700 1 -- 172.21.15.32:0/2092583692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c095290 con 0x7f023c09f990 2021-04-09T18:25:39.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f0240c01700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f023c09f990 0x7f023c094c70 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 rx=0x7f0234009940 tx=0x7f023400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:39.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f02397fa700 1 -- 172.21.15.32:0/2092583692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0234000d30 con 0x7f023c09f990 2021-04-09T18:25:39.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f02397fa700 1 -- 172.21.15.32:0/2092583692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023400c2f0 con 0x7f023c09f990 2021-04-09T18:25:39.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f02425e4700 1 -- 172.21.15.32:0/2092583692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023c095290 con 0x7f023c09f990 2021-04-09T18:25:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f02397fa700 1 -- 172.21.15.32:0/2092583692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f023400c790 con 0x7f023c09f990 2021-04-09T18:25:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.215 7f02425e4700 1 -- 172.21.15.32:0/2092583692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023c095290 con 0x7f023c09f990 2021-04-09T18:25:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.216 7f02397fa700 1 -- 172.21.15.32:0/2092583692 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0234000cc0 con 0x7f023c09f990 2021-04-09T18:25:39.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.216 7f02397fa700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0220018930 0x7f022001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.216 7f02397fa700 1 -- 172.21.15.32:0/2092583692 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0234032e20 con 0x7f023c09f990 2021-04-09T18:25:39.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.216 7f023bfff700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0220018930 0x7f022001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.216 7f023bfff700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0220018930 0x7f022001ae10 crc :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:39.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.236 7f02425e4700 1 -- 172.21.15.32:0/2092583692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023c095b10 con 0x7f023c09f990 2021-04-09T18:25:39.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.239 7f02397fa700 1 -- 172.21.15.32:0/2092583692 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f023400be30 con 0x7f023c09f990 2021-04-09T18:25:39.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.418 7f02425e4700 1 -- 172.21.15.32:0/2092583692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f023c095b10 con 0x7f023c09f990 2021-04-09T18:25:39.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.419 7f02397fa700 1 -- 172.21.15.32:0/2092583692 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v131) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0234000cc0 con 0x7f023c09f990 2021-04-09T18:25:39.487 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:39.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.421 7f02425e4700 1 -- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0220018930 msgr2=0x7f022001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.421 7f02425e4700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0220018930 0x7f022001ae10 crc :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.421 7f02425e4700 1 -- 172.21.15.32:0/2092583692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f023c09f990 msgr2=0x7f023c094c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.421 7f02425e4700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f023c09f990 0x7f023c094c70 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 rx=0x7f0234009940 tx=0x7f023400bcf0).stop 2021-04-09T18:25:39.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.421 7f02425e4700 1 -- 172.21.15.32:0/2092583692 shutdown_connections 2021-04-09T18:25:39.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.421 7f02425e4700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0220018930 0x7f022001ae10 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.422 7f02425e4700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f023c09f990 0x7f023c094c70 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.422 7f02425e4700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f023c09e930 0x7f023c09f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.422 7f02425e4700 1 --2- 172.21.15.32:0/2092583692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f023c09e530 0x7f023c09b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.422 7f02425e4700 1 -- 172.21.15.32:0/2092583692 >> 172.21.15.32:0/2092583692 conn(0x7f023c0c4290 msgr2=0x7f023c0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:39.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.422 7f02425e4700 1 -- 172.21.15.32:0/2092583692 shutdown_connections 2021-04-09T18:25:39.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.422 7f02425e4700 1 -- 172.21.15.32:0/2092583692 wait complete. 2021-04-09T18:25:39.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:25:39.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.584 7f71689d0700 1 Processor -- start 2021-04-09T18:25:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.584 7f71689d0700 1 -- start start 2021-04-09T18:25:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f71689d0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71640d7ca0 0x7f71640d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f71689d0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f716410da00 0x7f716410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f71689d0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716410d480 con 0x7f716410d010 2021-04-09T18:25:39.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f71689d0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71640d9220 con 0x7f71640d7ca0 2021-04-09T18:25:39.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f71689d0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71640d86c0 con 0x7f716410da00 2021-04-09T18:25:39.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f7161e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71640d7ca0 0x7f71640d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f7161e1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f71640d7ca0 0x7f71640d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52598/0 (socket says 172.21.15.32:52598) 2021-04-09T18:25:39.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f7161e1b700 1 -- 172.21.15.32:0/2354287998 learned_addr learned my addr 172.21.15.32:0/2354287998 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:39.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f7162e1d700 1 --2- 172.21.15.32:0/2354287998 >> v2:172.21.15.86:3300/0 conn(0x7f716410da00 0x7f716410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f7161e1b700 1 -- 172.21.15.32:0/2354287998 >> v2:172.21.15.86:3300/0 conn(0x7f716410da00 msgr2=0x7f716410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:39.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f7161e1b700 1 --2- 172.21.15.32:0/2354287998 >> v2:172.21.15.86:3300/0 conn(0x7f716410da00 0x7f716410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f7161e1b700 1 -- 172.21.15.32:0/2354287998 >> v1:172.21.15.86:6789/0 conn(0x7f716410d010 legacy=0x7f71640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.585 7f7161e1b700 1 -- 172.21.15.32:0/2354287998 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71640d1f70 con 0x7f71640d7ca0 2021-04-09T18:25:39.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.586 7f7161e1b700 1 --2- 172.21.15.32:0/2354287998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640d7ca0 0x7f71640d80a0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7f7154006700 tx=0x7f7154009b00).ready entity=mon.0 client_cookie=4fea663f8c15656d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:39.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.586 7f716161a700 1 -- 172.21.15.32:0/2354287998 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f715400c0c0 con 0x7f71640d7ca0 2021-04-09T18:25:39.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.586 7f716161a700 1 -- 172.21.15.32:0/2354287998 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715400c400 con 0x7f71640d7ca0 2021-04-09T18:25:39.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.586 7f716161a700 1 -- 172.21.15.32:0/2354287998 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f715400c8e0 con 0x7f71640d7ca0 2021-04-09T18:25:39.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.586 7f71689d0700 1 -- 172.21.15.32:0/2354287998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640d7ca0 msgr2=0x7f71640d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.586 7f71689d0700 1 --2- 172.21.15.32:0/2354287998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640d7ca0 0x7f71640d80a0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7f7154006700 tx=0x7f7154009b00).stop 2021-04-09T18:25:39.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.587 7f71689d0700 1 -- 172.21.15.32:0/2354287998 shutdown_connections 2021-04-09T18:25:39.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.587 7f71689d0700 1 --2- 172.21.15.32:0/2354287998 >> v2:172.21.15.86:3300/0 conn(0x7f716410da00 0x7f716410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.587 7f71689d0700 1 --2- 172.21.15.32:0/2354287998 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640d7ca0 0x7f71640d80a0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.587 7f71689d0700 1 -- 172.21.15.32:0/2354287998 >> 172.21.15.32:0/2354287998 conn(0x7f71640ce5f0 msgr2=0x7f71640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:39.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.587 7f71689d0700 1 -- 172.21.15.32:0/2354287998 shutdown_connections 2021-04-09T18:25:39.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.587 7f71689d0700 1 -- 172.21.15.32:0/2354287998 wait complete. 2021-04-09T18:25:39.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.607 7f71689d0700 1 Processor -- start 2021-04-09T18:25:39.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.607 7f71689d0700 1 -- start start 2021-04-09T18:25:39.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.607 7f71689d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640b1e50 0x7f71640b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.607 7f71689d0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71640b9fd0 0x7f71640b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.607 7f71689d0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71640b5510 0x7f71640b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.607 7f71689d0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71640b2250 con 0x7f71640b1e50 2021-04-09T18:25:39.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.607 7f71689d0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71640a1220 con 0x7f71640b9fd0 2021-04-09T18:25:39.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.607 7f71689d0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71640b6b30 con 0x7f71640b5510 2021-04-09T18:25:39.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f716261c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640b1e50 0x7f71640b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f716261c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640b1e50 0x7f71640b99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52600/0 (socket says 172.21.15.32:52600) 2021-04-09T18:25:39.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f716261c700 1 -- 172.21.15.32:0/2228477960 learned_addr learned my addr 172.21.15.32:0/2228477960 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f7162e1d700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71640b5510 0x7f71640b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f7161e1b700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71640b9fd0 0x7f71640b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f716261c700 1 -- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71640b5510 msgr2=0x7f71640b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f716261c700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71640b5510 0x7f71640b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f716261c700 1 -- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71640b9fd0 msgr2=0x7f71640b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f716261c700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71640b9fd0 0x7f71640b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f716261c700 1 -- 172.21.15.32:0/2228477960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71640b5f30 con 0x7f71640b1e50 2021-04-09T18:25:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f716261c700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640b1e50 0x7f71640b99b0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 rx=0x7f714c00b580 tx=0x7f714c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f715affd700 1 -- 172.21.15.32:0/2228477960 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f714c007070 con 0x7f71640b1e50 2021-04-09T18:25:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f715affd700 1 -- 172.21.15.32:0/2228477960 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c007070 con 0x7f71640b1e50 2021-04-09T18:25:39.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f71689d0700 1 -- 172.21.15.32:0/2228477960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71640b5f30 con 0x7f71640b1e50 2021-04-09T18:25:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f715affd700 1 -- 172.21.15.32:0/2228477960 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f714c0073b0 con 0x7f71640b1e50 2021-04-09T18:25:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.608 7f71689d0700 1 -- 172.21.15.32:0/2228477960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71640b5f30 con 0x7f71640b1e50 2021-04-09T18:25:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.609 7f715affd700 1 -- 172.21.15.32:0/2228477960 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f714c00a000 con 0x7f71640b1e50 2021-04-09T18:25:39.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.609 7f715affd700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7148018930 0x7f714801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:39.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.609 7f715affd700 1 -- 172.21.15.32:0/2228477960 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f714c032a00 con 0x7f71640b1e50 2021-04-09T18:25:39.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.609 7f7161e1b700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7148018930 0x7f714801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:39.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.609 7f7161e1b700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7148018930 0x7f714801adc0 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:39.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.629 7f71689d0700 1 -- 172.21.15.32:0/2228477960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71641b7040 con 0x7f71640b1e50 2021-04-09T18:25:39.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.633 7f715affd700 1 -- 172.21.15.32:0/2228477960 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f714c006c00 con 0x7f71640b1e50 2021-04-09T18:25:39.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.814 7f71689d0700 1 -- 172.21.15.32:0/2228477960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f71641b7040 con 0x7f71640b1e50 2021-04-09T18:25:39.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.814 7f715affd700 1 -- 172.21.15.32:0/2228477960 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v131) v1 ==== 100+0+10 (secure 0 0 0) 0x7f714c00a000 con 0x7f71640b1e50 2021-04-09T18:25:39.883 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:39.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 -- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7148018930 msgr2=0x7f714801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7148018930 0x7f714801adc0 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 -- 172.21.15.32:0/2228477960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640b1e50 msgr2=0x7f71640b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:39.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640b1e50 0x7f71640b99b0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 rx=0x7f714c00b580 tx=0x7f714c00b920).stop 2021-04-09T18:25:39.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 -- 172.21.15.32:0/2228477960 shutdown_connections 2021-04-09T18:25:39.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7148018930 0x7f714801adc0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71640b5510 0x7f71640b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71640b9fd0 0x7f71640b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 --2- 172.21.15.32:0/2228477960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f71640b1e50 0x7f71640b99b0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 -- 172.21.15.32:0/2228477960 >> 172.21.15.32:0/2228477960 conn(0x7f71640b27e0 msgr2=0x7f71641b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 -- 172.21.15.32:0/2228477960 shutdown_connections 2021-04-09T18:25:39.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.817 7f71689d0700 1 -- 172.21.15.32:0/2228477960 wait complete. 2021-04-09T18:25:39.895 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:25:39.896 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.976 7f375d20f700 1 Processor -- start 2021-04-09T18:25:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.976 7f375d20f700 1 -- start start 2021-04-09T18:25:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f375d20f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37580cffa0 0x7f37580d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f375d20f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37580d6c80 0x7f375810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f375d20f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37580d7280 con 0x7f37580d7800 2021-04-09T18:25:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f375d20f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37580d7080 con 0x7f37580cffa0 2021-04-09T18:25:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f375d20f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37580d91e0 con 0x7f37580d6c80 2021-04-09T18:25:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f375659c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f37580d7800 0x7f37580d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33970/0 (socket says 172.21.15.32:33970) 2021-04-09T18:25:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f375659c700 1 -- 172.21.15.32:0/1029497126 learned_addr learned my addr 172.21.15.32:0/1029497126 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f375759e700 1 --2- 172.21.15.32:0/1029497126 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37580d6c80 0x7f375810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f3756d9d700 1 --2- 172.21.15.32:0/1029497126 >> v2:172.21.15.86:3300/0 conn(0x7f37580cffa0 0x7f37580d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:40.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f3755d9b700 1 -- 172.21.15.32:0/1029497126 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430628394 0 0) 0x7f37440009b0 con 0x7f37580d7800 2021-04-09T18:25:40.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.977 7f3755d9b700 1 -- 172.21.15.32:0/1029497126 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f373c002fe0 con 0x7f37580d7800 2021-04-09T18:25:40.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f3756d9d700 1 -- 172.21.15.32:0/1029497126 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37580d6c80 msgr2=0x7f375810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:40.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f3756d9d700 1 --2- 172.21.15.32:0/1029497126 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37580d6c80 0x7f375810cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f3756d9d700 1 -- 172.21.15.32:0/1029497126 >> v1:172.21.15.32:6789/0 conn(0x7f37580d7800 legacy=0x7f37580d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:40.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f3756d9d700 1 -- 172.21.15.32:0/1029497126 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580d83f0 con 0x7f37580cffa0 2021-04-09T18:25:40.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f375759e700 1 --2- 172.21.15.32:0/1029497126 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37580d6c80 0x7f375810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:40.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f3756d9d700 1 --2- 172.21.15.32:0/1029497126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37580cffa0 0x7f37580d2440 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f3748006650 tx=0x7f3748009a80).ready entity=mon.1 client_cookie=7b88f332ca0c750 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:40.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f3755d9b700 1 -- 172.21.15.32:0/1029497126 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37480009b0 con 0x7f37580cffa0 2021-04-09T18:25:40.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f3755d9b700 1 -- 172.21.15.32:0/1029497126 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374800c0c0 con 0x7f37580cffa0 2021-04-09T18:25:40.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f3755d9b700 1 -- 172.21.15.32:0/1029497126 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f374800c510 con 0x7f37580cffa0 2021-04-09T18:25:40.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f375d20f700 1 -- 172.21.15.32:0/1029497126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37580cffa0 msgr2=0x7f37580d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:40.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.978 7f375d20f700 1 --2- 172.21.15.32:0/1029497126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37580cffa0 0x7f37580d2440 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f3748006650 tx=0x7f3748009a80).stop 2021-04-09T18:25:40.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.979 7f375d20f700 1 -- 172.21.15.32:0/1029497126 shutdown_connections 2021-04-09T18:25:40.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.979 7f375d20f700 1 --2- 172.21.15.32:0/1029497126 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37580d6c80 0x7f375810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.979 7f375d20f700 1 --2- 172.21.15.32:0/1029497126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37580cffa0 0x7f37580d2440 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.979 7f375d20f700 1 -- 172.21.15.32:0/1029497126 >> 172.21.15.32:0/1029497126 conn(0x7f37580ce5c0 msgr2=0x7f37580cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:40.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.979 7f375d20f700 1 -- 172.21.15.32:0/1029497126 shutdown_connections 2021-04-09T18:25:40.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.979 7f375d20f700 1 -- 172.21.15.32:0/1029497126 wait complete. 2021-04-09T18:25:40.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.999 7f375d20f700 1 Processor -- start 2021-04-09T18:25:40.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.999 7f375d20f700 1 -- start start 2021-04-09T18:25:40.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:39.999 7f375d20f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f375819c910 0x7f3758093030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:40.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375d20f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3758093650 0x7f375819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:40.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375d20f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f375819cd10 0x7f375819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:40.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375d20f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3758093c20 con 0x7f3758093650 2021-04-09T18:25:40.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375d20f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37580941a0 con 0x7f375819cd10 2021-04-09T18:25:40.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375d20f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3758094350 con 0x7f375819c910 2021-04-09T18:25:40.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3758093650 0x7f375819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:40.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3758093650 0x7f375819f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52614/0 (socket says 172.21.15.32:52614) 2021-04-09T18:25:40.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375659c700 1 -- 172.21.15.32:0/24945618 learned_addr learned my addr 172.21.15.32:0/24945618 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:40.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375759e700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f375819cd10 0x7f375819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:40.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f3756d9d700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f375819c910 0x7f3758093030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:40.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375659c700 1 -- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f375819c910 msgr2=0x7f3758093030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:40.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375659c700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f375819c910 0x7f3758093030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375659c700 1 -- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f375819cd10 msgr2=0x7f375819fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:40.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375659c700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f375819cd10 0x7f375819fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375659c700 1 -- 172.21.15.32:0/24945618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3758094570 con 0x7f3758093650 2021-04-09T18:25:40.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375659c700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3758093650 0x7f375819f460 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7f3744001c60 tx=0x7f374400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:40.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f374f7fe700 1 -- 172.21.15.32:0/24945618 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f374400acc0 con 0x7f3758093650 2021-04-09T18:25:40.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f374f7fe700 1 -- 172.21.15.32:0/24945618 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3744004140 con 0x7f3758093650 2021-04-09T18:25:40.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f374f7fe700 1 -- 172.21.15.32:0/24945618 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3744004620 con 0x7f3758093650 2021-04-09T18:25:40.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375d20f700 1 -- 172.21.15.32:0/24945618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3758094570 con 0x7f3758093650 2021-04-09T18:25:40.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.000 7f375d20f700 1 -- 172.21.15.32:0/24945618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3758094570 con 0x7f3758093650 2021-04-09T18:25:40.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.001 7f374f7fe700 1 -- 172.21.15.32:0/24945618 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f37440047c0 con 0x7f3758093650 2021-04-09T18:25:40.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.001 7f374f7fe700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37340188d0 0x7f373401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:40.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.001 7f374f7fe700 1 -- 172.21.15.32:0/24945618 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f37440329e0 con 0x7f3758093650 2021-04-09T18:25:40.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.001 7f3756d9d700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37340188d0 0x7f373401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:40.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.002 7f3756d9d700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37340188d0 0x7f373401add0 crc :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:40.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.021 7f375d20f700 1 -- 172.21.15.32:0/24945618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37581b70c0 con 0x7f3758093650 2021-04-09T18:25:40.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.025 7f374f7fe700 1 -- 172.21.15.32:0/24945618 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f37440045b0 con 0x7f3758093650 2021-04-09T18:25:40.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.215 7f375d20f700 1 -- 172.21.15.32:0/24945618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37580d7280 con 0x7f3758093650 2021-04-09T18:25:40.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.215 7f374f7fe700 1 -- 172.21.15.32:0/24945618 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2785 (secure 0 0 0) 0x7f37440047c0 con 0x7f3758093650 2021-04-09T18:25:40.283 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 268 2021-04-09T18:25:40.284 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:40.284 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":268,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:25:35.619821","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:25:40.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 -- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37340188d0 msgr2=0x7f373401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:40.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37340188d0 0x7f373401add0 crc :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 -- 172.21.15.32:0/24945618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3758093650 msgr2=0x7f375819f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:40.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3758093650 0x7f375819f460 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7f3744001c60 tx=0x7f374400a8d0).stop 2021-04-09T18:25:40.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 -- 172.21.15.32:0/24945618 shutdown_connections 2021-04-09T18:25:40.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37340188d0 0x7f373401add0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3758093650 0x7f375819f460 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f375819cd10 0x7f375819fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 --2- 172.21.15.32:0/24945618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f375819c910 0x7f3758093030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:40.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 -- 172.21.15.32:0/24945618 >> 172.21.15.32:0/24945618 conn(0x7f37581864e0 msgr2=0x7f37581b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:40.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 -- 172.21.15.32:0/24945618 shutdown_connections 2021-04-09T18:25:40.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:40.218 7f375d20f700 1 -- 172.21.15.32:0/24945618 wait complete. 2021-04-09T18:25:40.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 268, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:25:33.288479', 'modified': '2021-04-09 18:25:35.619821', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9085': {'gid': 9085, 'name': 'd', 'rank': 0, 'incarnation': 265, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3538582472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3538582472}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3538582472}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:40.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:25:40.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:25:40.825 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:25:40.826 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:25:40.826 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:25:40.826 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 9085, 2021-04-09T18:25:40.826 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:25:40.827 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:25:40.827 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 7.469135402, 2021-04-09T18:25:40.827 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 268, 2021-04-09T18:25:40.827 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 131, 2021-04-09T18:25:40.827 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 131, 2021-04-09T18:25:40.828 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12.917294964 2021-04-09T18:25:40.828 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:25:40.831 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 9085, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.469135402, "mdsmap_epoch": 268, "osdmap_epoch": 131, "osdmap_epoch_barrier": 131, "uptime": 12.917294964 } 2021-04-09T18:25:40.832 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:25:40.832 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:25:40.832 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:25:40.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:40.833 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:40.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:25:40.939 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:25:40.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:25:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:25:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:25:40.999 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:25:40.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:41.000 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:25:41.043 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:25:41.044 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:41.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:25:41.145 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:25:41.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:25:41.204 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:25:41.204 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:25:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:25:41.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:41.206 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:25:41.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:25:41.167 7fbf9a90cf80 -1 init, newargv = 0x55c7e58e92c0 newargc=9 2021-04-09T18:25:41.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[75840]: starting ceph client 2021-04-09T18:25:41.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[75840]: starting fuse 2021-04-09T18:25:41.251 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:25:41.252 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:25:41.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:41.253 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:25:41.253 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:25:41.253 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:25:41.254 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:25:41.254 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:25:41.255 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:41.255 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:25:41.255 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:25:41.256 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:25:41.256 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:41.257 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:25:41.257 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:25:41.258 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:25:41.258 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:25:41.259 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:41.259 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:25:41.259 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:25:41.260 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:25:41.260 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:25:41.261 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:25:41.261 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:41.262 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:25:41.262 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:25:41.373 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.75840.asok 2021-04-09T18:25:41.378 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.75840.asok 2021-04-09T18:25:41.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:41.379 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.75840.asok status 2021-04-09T18:25:41.502 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:41.503 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:25:41.503 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:25:41.503 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:25:41.503 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:25:41.503 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:25:41.504 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:25:41.504 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "75840", 2021-04-09T18:25:41.504 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:25:41.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:41.505 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:25:41.505 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:25:41.505 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9540, 2021-04-09T18:25:41.505 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:25:41.505 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:25:41.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:25:41.506 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 9540 2021-04-09T18:25:41.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:41.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:25:41.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:41.507 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:25:41.507 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1654441886 2021-04-09T18:25:41.507 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:41.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:41.508 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:25:41.508 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:41.508 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:25:41.508 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1654441886 2021-04-09T18:25:41.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:41.509 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.9540 172.21.15.32:0/1654441886", 2021-04-09T18:25:41.509 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/1654441886", 2021-04-09T18:25:41.509 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:25:41.510 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 268, 2021-04-09T18:25:41.510 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 131, 2021-04-09T18:25:41.510 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:25:41.510 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:25:41.510 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:25:41.511 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:41.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:41.512 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:41.513 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:41.557 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:25:41.557 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:25:41.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:41.558 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:41.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get fuse_require_active_mds 2021-04-09T18:25:41.745 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:41.746 INFO:teuthology.orchestra.run.smithi032.stdout: "fuse_require_active_mds": "true" 2021-04-09T18:25:41.746 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:41.751 INFO:tasks.cephfs.filesystem:_json_asok output: { "fuse_require_active_mds": "true" } 2021-04-09T18:25:41.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:25:41.878 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:41.878 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:25:41.879 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:41.884 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:25:41.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:42.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.965 7f9902e0f700 1 Processor -- start 2021-04-09T18:25:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.965 7f9902e0f700 1 -- start start 2021-04-09T18:25:42.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f9902e0f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f98fc10d010 0x7f98fc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f9902e0f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc0d1da0 0x7f98fc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f9902e0f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f98fc0d82c0 0x7f98fc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f9902e0f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc0d8d30 con 0x7f98fc0d82c0 2021-04-09T18:25:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f9902e0f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc0d9220 con 0x7f98fc10d010 2021-04-09T18:25:42.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f9902e0f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc10d480 con 0x7f98fc0d1da0 2021-04-09T18:25:42.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f990142c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f98fc0d82c0 0x7f98fc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f990142c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f98fc0d82c0 0x7f98fc10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52640/0 (socket says 172.21.15.32:52640) 2021-04-09T18:25:42.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f990142c700 1 -- 172.21.15.32:0/869511971 learned_addr learned my addr 172.21.15.32:0/869511971 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f98fbfff700 1 --2- 172.21.15.32:0/869511971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc0d1da0 0x7f98fc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f9900c2b700 1 --2- 172.21.15.32:0/869511971 >> v2:172.21.15.86:3300/0 conn(0x7f98fc10d010 0x7f98fc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f990142c700 1 -- 172.21.15.32:0/869511971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc0d1da0 msgr2=0x7f98fc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f990142c700 1 --2- 172.21.15.32:0/869511971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc0d1da0 0x7f98fc0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f990142c700 1 -- 172.21.15.32:0/869511971 >> v2:172.21.15.86:3300/0 conn(0x7f98fc10d010 msgr2=0x7f98fc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:42.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f990142c700 1 --2- 172.21.15.32:0/869511971 >> v2:172.21.15.86:3300/0 conn(0x7f98fc10d010 0x7f98fc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f990142c700 1 -- 172.21.15.32:0/869511971 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0d86c0 con 0x7f98fc0d82c0 2021-04-09T18:25:42.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.966 7f98fbfff700 1 --2- 172.21.15.32:0/869511971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc0d1da0 0x7f98fc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:42.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f990142c700 1 --2- 172.21.15.32:0/869511971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc0d82c0 0x7f98fc10da00 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 rx=0x7f98f4006f00 tx=0x7f98f4009f10).ready entity=mon.0 client_cookie=8683e75261822939 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:42.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f98fb7fe700 1 -- 172.21.15.32:0/869511971 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98f400b3b0 con 0x7f98fc0d82c0 2021-04-09T18:25:42.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f98fb7fe700 1 -- 172.21.15.32:0/869511971 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f400b730 con 0x7f98fc0d82c0 2021-04-09T18:25:42.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f98fb7fe700 1 -- 172.21.15.32:0/869511971 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98f400bc10 con 0x7f98fc0d82c0 2021-04-09T18:25:42.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f9902e0f700 1 -- 172.21.15.32:0/869511971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc0d82c0 msgr2=0x7f98fc10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f9902e0f700 1 --2- 172.21.15.32:0/869511971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc0d82c0 0x7f98fc10da00 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 rx=0x7f98f4006f00 tx=0x7f98f4009f10).stop 2021-04-09T18:25:42.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f9902e0f700 1 -- 172.21.15.32:0/869511971 shutdown_connections 2021-04-09T18:25:42.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f9902e0f700 1 --2- 172.21.15.32:0/869511971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc0d82c0 0x7f98fc10da00 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f9902e0f700 1 --2- 172.21.15.32:0/869511971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc0d1da0 0x7f98fc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f9902e0f700 1 --2- 172.21.15.32:0/869511971 >> v2:172.21.15.86:3300/0 conn(0x7f98fc10d010 0x7f98fc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.967 7f9902e0f700 1 -- 172.21.15.32:0/869511971 >> 172.21.15.32:0/869511971 conn(0x7f98fc0ce5f0 msgr2=0x7f98fc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.968 7f9902e0f700 1 -- 172.21.15.32:0/869511971 shutdown_connections 2021-04-09T18:25:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.968 7f9902e0f700 1 -- 172.21.15.32:0/869511971 wait complete. 2021-04-09T18:25:42.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9902e0f700 1 Processor -- start 2021-04-09T18:25:42.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9902e0f700 1 -- start start 2021-04-09T18:25:42.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9902e0f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc183890 0x7f98fc0aaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9902e0f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc17f1d0 0x7f98fc192ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9902e0f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98fc1931c0 0x7f98fc179ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9902e0f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc08f760 con 0x7f98fc183890 2021-04-09T18:25:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9902e0f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc1935c0 con 0x7f98fc1931c0 2021-04-09T18:25:42.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9902e0f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc183e20 con 0x7f98fc17f1d0 2021-04-09T18:25:42.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9900c2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc183890 0x7f98fc0aaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9900c2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc183890 0x7f98fc0aaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52642/0 (socket says 172.21.15.32:52642) 2021-04-09T18:25:42.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.988 7f9900c2b700 1 -- 172.21.15.32:0/2575789586 learned_addr learned my addr 172.21.15.32:0/2575789586 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:42.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f98fbfff700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc17f1d0 0x7f98fc192ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f990142c700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98fc1931c0 0x7f98fc179ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f9900c2b700 1 -- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc17f1d0 msgr2=0x7f98fc192ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f9900c2b700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc17f1d0 0x7f98fc192ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f9900c2b700 1 -- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98fc1931c0 msgr2=0x7f98fc179ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f9900c2b700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98fc1931c0 0x7f98fc179ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f9900c2b700 1 -- 172.21.15.32:0/2575789586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0ab0c0 con 0x7f98fc183890 2021-04-09T18:25:42.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f990142c700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98fc1931c0 0x7f98fc179ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:42.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f9900c2b700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc183890 0x7f98fc0aaaa0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f98ec006f00 tx=0x7f98ec00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:42.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f98f97fa700 1 -- 172.21.15.32:0/2575789586 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98ec00bd30 con 0x7f98fc183890 2021-04-09T18:25:42.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f98f97fa700 1 -- 172.21.15.32:0/2575789586 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec002030 con 0x7f98fc183890 2021-04-09T18:25:42.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc0ab0c0 con 0x7f98fc183890 2021-04-09T18:25:42.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f98f97fa700 1 -- 172.21.15.32:0/2575789586 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98ec002590 con 0x7f98fc183890 2021-04-09T18:25:42.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.989 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc0ab0c0 con 0x7f98fc183890 2021-04-09T18:25:42.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.990 7f98f97fa700 1 -- 172.21.15.32:0/2575789586 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f98ec000c90 con 0x7f98fc183890 2021-04-09T18:25:42.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.990 7f98f97fa700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98dc018900 0x7f98dc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.990 7f98f97fa700 1 -- 172.21.15.32:0/2575789586 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f98ec032a40 con 0x7f98fc183890 2021-04-09T18:25:42.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.990 7f98fbfff700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98dc018900 0x7f98dc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:41.990 7f98fbfff700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98dc018900 0x7f98dc01adc0 crc :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:42.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.010 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc17ad90 con 0x7f98fc183890 2021-04-09T18:25:42.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.013 7f98f97fa700 1 -- 172.21.15.32:0/2575789586 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f98ec01f780 con 0x7f98fc183890 2021-04-09T18:25:42.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.205 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98fc17ad90 con 0x7f98fc183890 2021-04-09T18:25:42.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.205 7f98f97fa700 1 -- 172.21.15.32:0/2575789586 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2785 (secure 0 0 0) 0x7f98ec00bd30 con 0x7f98fc183890 2021-04-09T18:25:42.274 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 268 2021-04-09T18:25:42.274 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:42.275 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":268,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:25:35.619821","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:25:42.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98dc018900 msgr2=0x7f98dc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98dc018900 0x7f98dc01adc0 crc :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc183890 msgr2=0x7f98fc0aaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc183890 0x7f98fc0aaaa0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f98ec006f00 tx=0x7f98ec00b940).stop 2021-04-09T18:25:42.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 shutdown_connections 2021-04-09T18:25:42.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98dc018900 0x7f98dc01adc0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98fc1931c0 0x7f98fc179ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98fc17f1d0 0x7f98fc192ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 --2- 172.21.15.32:0/2575789586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98fc183890 0x7f98fc0aaaa0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 >> 172.21.15.32:0/2575789586 conn(0x7f98fc1925d0 msgr2=0x7f98fc1a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 shutdown_connections 2021-04-09T18:25:42.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.208 7f9902e0f700 1 -- 172.21.15.32:0/2575789586 wait complete. 2021-04-09T18:25:42.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:42.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.367 7fc8c567f700 1 Processor -- start 2021-04-09T18:25:42.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.367 7fc8c567f700 1 -- start start 2021-04-09T18:25:42.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.367 7fc8c567f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc8c010d010 0x7fc8c00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.367 7fc8c567f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c010d480 con 0x7fc8c00d1da0 2021-04-09T18:25:42.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.367 7fc8c567f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c010e1f0 con 0x7fc8c00d8490 2021-04-09T18:25:42.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.367 7fc8c567f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c00d8d30 con 0x7fc8c010d010 2021-04-09T18:25:42.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8beabe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc8c00d1da0 0x7fc8c00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34012/0 (socket says 172.21.15.32:34012) 2021-04-09T18:25:42.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8beabe700 1 -- 172.21.15.32:0/1574000264 learned_addr learned my addr 172.21.15.32:0/1574000264 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:42.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8bf2bf700 1 --2- 172.21.15.32:0/1574000264 >> v2:172.21.15.86:3300/0 conn(0x7fc8c010d010 0x7fc8c00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8be2bd700 1 -- 172.21.15.32:0/1574000264 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053713310 0 0) 0x7fc8b4000ce0 con 0x7fc8c00d1da0 2021-04-09T18:25:42.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8be2bd700 1 -- 172.21.15.32:0/1574000264 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a8002fe0 con 0x7fc8c00d1da0 2021-04-09T18:25:42.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8be2bd700 1 -- 172.21.15.32:0/1574000264 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053241138 0 0) 0x7fc8b80009b0 con 0x7fc8c00d8490 2021-04-09T18:25:42.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8be2bd700 1 -- 172.21.15.32:0/1574000264 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a8003210 con 0x7fc8c00d8490 2021-04-09T18:25:42.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8bf2bf700 1 -- 172.21.15.32:0/1574000264 >> v1:172.21.15.86:6789/0 conn(0x7fc8c00d8490 legacy=0x7fc8c010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8bf2bf700 1 -- 172.21.15.32:0/1574000264 >> v1:172.21.15.32:6789/0 conn(0x7fc8c00d1da0 legacy=0x7fc8c00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8bf2bf700 1 -- 172.21.15.32:0/1574000264 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c00d9220 con 0x7fc8c010d010 2021-04-09T18:25:42.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8bf2bf700 1 --2- 172.21.15.32:0/1574000264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c010d010 0x7fc8c00d1780 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fc8b00069d0 tx=0x7fc8b0009e40).ready entity=mon.1 client_cookie=8013334fd8d0d265 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:42.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8be2bd700 1 -- 172.21.15.32:0/1574000264 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b000b510 con 0x7fc8c010d010 2021-04-09T18:25:42.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8be2bd700 1 -- 172.21.15.32:0/1574000264 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b000b890 con 0x7fc8c010d010 2021-04-09T18:25:42.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8be2bd700 1 -- 172.21.15.32:0/1574000264 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b000bd70 con 0x7fc8c010d010 2021-04-09T18:25:42.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8c567f700 1 -- 172.21.15.32:0/1574000264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c010d010 msgr2=0x7fc8c00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.368 7fc8c567f700 1 --2- 172.21.15.32:0/1574000264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c010d010 0x7fc8c00d1780 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fc8b00069d0 tx=0x7fc8b0009e40).stop 2021-04-09T18:25:42.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.369 7fc8c567f700 1 -- 172.21.15.32:0/1574000264 shutdown_connections 2021-04-09T18:25:42.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.369 7fc8c567f700 1 --2- 172.21.15.32:0/1574000264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c010d010 0x7fc8c00d1780 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.369 7fc8c567f700 1 -- 172.21.15.32:0/1574000264 >> 172.21.15.32:0/1574000264 conn(0x7fc8c00ce5f0 msgr2=0x7fc8c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:42.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.369 7fc8c567f700 1 -- 172.21.15.32:0/1574000264 shutdown_connections 2021-04-09T18:25:42.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.369 7fc8c567f700 1 -- 172.21.15.32:0/1574000264 wait complete. 2021-04-09T18:25:42.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.389 7fc8c567f700 1 Processor -- start 2021-04-09T18:25:42.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.389 7fc8c567f700 1 -- start start 2021-04-09T18:25:42.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8c567f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8c00a8ae0 0x7fc8c00a8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8c567f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c01b8540 0x7fc8c01b8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8c567f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c01ab5a0 0x7fc8c01ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8c567f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c01abfc0 con 0x7fc8c01b8540 2021-04-09T18:25:42.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8c567f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c008cf70 con 0x7fc8c01ab5a0 2021-04-09T18:25:42.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8c567f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c008d190 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8beabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c01b8540 0x7fc8c01b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8beabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c01b8540 0x7fc8c01b8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52656/0 (socket says 172.21.15.32:52656) 2021-04-09T18:25:42.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8beabe700 1 -- 172.21.15.32:0/126130612 learned_addr learned my addr 172.21.15.32:0/126130612 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:42.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8bf2bf700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8c00a8ae0 0x7fc8c00a8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8bfac0700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c01ab5a0 0x7fc8c01ab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8bf2bf700 1 -- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c01ab5a0 msgr2=0x7fc8c01ab9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8bf2bf700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c01ab5a0 0x7fc8c01ab9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8bf2bf700 1 -- 172.21.15.32:0/126130612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c01b8540 msgr2=0x7fc8c01b8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8bf2bf700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c01b8540 0x7fc8c01b8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8bf2bf700 1 -- 172.21.15.32:0/126130612 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c008d3b0 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.390 7fc8beabe700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c01b8540 0x7fc8c01b8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:42.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.391 7fc8bf2bf700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8c00a8ae0 0x7fc8c00a8ee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fc8b0009310 tx=0x7fc8b000bc70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:42.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.391 7fc8a7fff700 1 -- 172.21.15.32:0/126130612 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b0006030 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.391 7fc8a7fff700 1 -- 172.21.15.32:0/126130612 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b0006370 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.391 7fc8c567f700 1 -- 172.21.15.32:0/126130612 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c008d3b0 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.391 7fc8a7fff700 1 -- 172.21.15.32:0/126130612 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b0005660 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.391 7fc8c567f700 1 -- 172.21.15.32:0/126130612 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c008d3b0 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.392 7fc8a7fff700 1 -- 172.21.15.32:0/126130612 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc8b0006550 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.392 7fc8a7fff700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8a00188d0 0x7fc8a001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:42.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.392 7fc8a7fff700 1 -- 172.21.15.32:0/126130612 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc8b0032cc0 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.392 7fc8beabe700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8a00188d0 0x7fc8a001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:42.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.392 7fc8beabe700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8a00188d0 0x7fc8a001add0 crc :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:42.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.412 7fc8c567f700 1 -- 172.21.15.32:0/126130612 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c01ab070 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.415 7fc8a7fff700 1 -- 172.21.15.32:0/126130612 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc8b000bdb0 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.607 7fc8c567f700 1 -- 172.21.15.32:0/126130612 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8c01ab070 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.607 7fc8a7fff700 1 -- 172.21.15.32:0/126130612 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2785 (secure 0 0 0) 0x7fc8b0006370 con 0x7fc8c00a8ae0 2021-04-09T18:25:42.676 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 268 2021-04-09T18:25:42.677 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:42.677 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":268,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:25:35.619821","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:25:42.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 -- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8a00188d0 msgr2=0x7fc8a001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8a00188d0 0x7fc8a001add0 crc :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 -- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8c00a8ae0 msgr2=0x7fc8c00a8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:42.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8c00a8ae0 0x7fc8c00a8ee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fc8b0009310 tx=0x7fc8b000bc70).stop 2021-04-09T18:25:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 -- 172.21.15.32:0/126130612 shutdown_connections 2021-04-09T18:25:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc8a00188d0 0x7fc8a001add0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc8c01b8540 0x7fc8c01b8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8c01ab5a0 0x7fc8c01ab9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 --2- 172.21.15.32:0/126130612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8c00a8ae0 0x7fc8c00a8ee0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 -- 172.21.15.32:0/126130612 >> 172.21.15.32:0/126130612 conn(0x7fc8c0190570 msgr2=0x7fc8c018ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:42.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 -- 172.21.15.32:0/126130612 shutdown_connections 2021-04-09T18:25:42.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:42.610 7fc8c567f700 1 -- 172.21.15.32:0/126130612 wait complete. 2021-04-09T18:25:42.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:42.688 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:42.689 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:42.733 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:25:42.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:25:42.735 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:25:42.736 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:25:42.737 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:42.835 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[75840]: fuse finished with error 0 and tester_r 0 2021-04-09T18:25:42.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:42.837 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:42.838 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:42.882 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:25:42.883 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:25:42.883 INFO:teuthology.orchestra.run:waiting for 900 2021-04-09T18:25:48.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:25:48.986 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:49.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:25:49.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.110 7fccb7338700 1 Processor -- start 2021-04-09T18:25:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.110 7fccb7338700 1 -- start start 2021-04-09T18:25:49.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.111 7fccb7338700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb010da00 0x7fccb010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.111 7fccb7338700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb010d480 con 0x7fccb00d7ca0 2021-04-09T18:25:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.111 7fccb7338700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb00d1f70 con 0x7fccb010d010 2021-04-09T18:25:49.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.111 7fccb7338700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb00d9220 con 0x7fccb010da00 2021-04-09T18:25:49.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.111 7fccb4953700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fccb00d7ca0 0x7fccb00d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34024/0 (socket says 172.21.15.32:34024) 2021-04-09T18:25:49.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.111 7fccb4953700 1 -- 172.21.15.32:0/3892636901 learned_addr learned my addr 172.21.15.32:0/3892636901 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:49.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.111 7fccb5955700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb010da00 0x7fccb010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.111 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565114837 0 0) 0x7fcca0000ce0 con 0x7fccb00d7ca0 2021-04-09T18:25:49.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.111 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc9c002fe0 con 0x7fccb00d7ca0 2021-04-09T18:25:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767350595 0 0) 0x7fccac000c60 con 0x7fccb010d010 2021-04-09T18:25:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc9c003210 con 0x7fccb010d010 2021-04-09T18:25:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2666968626 0 0) 0x7fcca0000ed0 con 0x7fccb00d7ca0 2021-04-09T18:25:49.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb010da00 msgr2=0x7fccb010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:49.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb010da00 0x7fccb010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 >> v1:172.21.15.86:6789/0 conn(0x7fccb010d010 legacy=0x7fccb00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb010e420 con 0x7fccb00d7ca0 2021-04-09T18:25:49.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fcca0001600 con 0x7fccb00d7ca0 2021-04-09T18:25:49.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb5955700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb010da00 0x7fccb010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:49.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 >> v1:172.21.15.32:6789/0 conn(0x7fccb00d7ca0 legacy=0x7fccb00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc9c003b90 0x7fcc9c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc9c0067a0 0x7fcc9c008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc9c009510 0x7fcc9c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c0502f0 con 0x7fcc9c009510 2021-04-09T18:25:49.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c050510 con 0x7fcc9c003b90 2021-04-09T18:25:49.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb5955700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc9c009510 0x7fcc9c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c050730 con 0x7fcc9c0067a0 2021-04-09T18:25:49.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb5154700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc9c003b90 0x7fcc9c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb4953700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc9c0067a0 0x7fcc9c008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb5955700 1 -- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc9c0067a0 msgr2=0x7fcc9c008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:49.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb5955700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc9c0067a0 0x7fcc9c008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb5955700 1 -- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc9c003b90 msgr2=0x7fcc9c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:49.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb5955700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc9c003b90 0x7fcc9c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb5955700 1 -- 172.21.15.32:0/3892636901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c0509d0 con 0x7fcc9c009510 2021-04-09T18:25:49.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fccb5955700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc9c009510 0x7fcc9c00ba30 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7fcca8009a40 tx=0x7fcca8009de0).ready entity=mon.0 client_cookie=83f6ce5ff0f51aa8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:49.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcca8006a90 con 0x7fcc9c009510 2021-04-09T18:25:49.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca8003030 con 0x7fcc9c009510 2021-04-09T18:25:49.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.112 7fcca7fff700 1 -- 172.21.15.32:0/3892636901 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcca800b790 con 0x7fcc9c009510 2021-04-09T18:25:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.113 7fccb7338700 1 -- 172.21.15.32:0/3892636901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc9c009510 msgr2=0x7fcc9c00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.113 7fccb7338700 1 --2- 172.21.15.32:0/3892636901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc9c009510 0x7fcc9c00ba30 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7fcca8009a40 tx=0x7fcca8009de0).stop 2021-04-09T18:25:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.113 7fccb5955700 1 -- 172.21.15.32:0/3892636901 reap_dead start 2021-04-09T18:25:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.113 7fccb7338700 1 -- 172.21.15.32:0/3892636901 shutdown_connections 2021-04-09T18:25:49.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.113 7fccb7338700 1 -- 172.21.15.32:0/3892636901 >> 172.21.15.32:0/3892636901 conn(0x7fccb00ce5f0 msgr2=0x7fccb00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:49.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.113 7fccb7338700 1 -- 172.21.15.32:0/3892636901 shutdown_connections 2021-04-09T18:25:49.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.113 7fccb7338700 1 -- 172.21.15.32:0/3892636901 wait complete. 2021-04-09T18:25:49.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.133 7fccb7338700 1 Processor -- start 2021-04-09T18:25:49.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb7338700 1 -- start start 2021-04-09T18:25:49.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb7338700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb0185060 0x7fccb0185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb7338700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccb0186c00 0x7fccb0187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb7338700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccb0191360 0x7fccb0191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb7338700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb01863a0 con 0x7fccb0186c00 2021-04-09T18:25:49.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb7338700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0185a60 con 0x7fccb0191360 2021-04-09T18:25:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb7338700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0187620 con 0x7fccb0185060 2021-04-09T18:25:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb4953700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccb0186c00 0x7fccb0187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb4953700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccb0186c00 0x7fccb0187000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52674/0 (socket says 172.21.15.32:52674) 2021-04-09T18:25:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb4953700 1 -- 172.21.15.32:0/3080137584 learned_addr learned my addr 172.21.15.32:0/3080137584 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:49.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.134 7fccb5955700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccb0191360 0x7fccb0191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fccb5154700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb0185060 0x7fccb0185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fccb4953700 1 -- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb0185060 msgr2=0x7fccb0185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fccb4953700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb0185060 0x7fccb0185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fccb4953700 1 -- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccb0191360 msgr2=0x7fccb0191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fccb4953700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccb0191360 0x7fccb0191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fccb4953700 1 -- 172.21.15.32:0/3080137584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0185460 con 0x7fccb0186c00 2021-04-09T18:25:49.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fccb4953700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccb0186c00 0x7fccb0187000 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7fcca0007680 tx=0x7fcca000a960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fcca5ffb700 1 -- 172.21.15.32:0/3080137584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcca00070d0 con 0x7fccb0186c00 2021-04-09T18:25:49.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fcca5ffb700 1 -- 172.21.15.32:0/3080137584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca0001480 con 0x7fccb0186c00 2021-04-09T18:25:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fccb7338700 1 -- 172.21.15.32:0/3080137584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb0185460 con 0x7fccb0186c00 2021-04-09T18:25:49.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fcca5ffb700 1 -- 172.21.15.32:0/3080137584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcca00045b0 con 0x7fccb0186c00 2021-04-09T18:25:49.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.135 7fccb7338700 1 -- 172.21.15.32:0/3080137584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb0185460 con 0x7fccb0186c00 2021-04-09T18:25:49.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.136 7fcca5ffb700 1 -- 172.21.15.32:0/3080137584 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcca0001a90 con 0x7fccb0186c00 2021-04-09T18:25:49.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.136 7fcca5ffb700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc90018900 0x7fcc9001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.136 7fcca5ffb700 1 -- 172.21.15.32:0/3080137584 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcca00329d0 con 0x7fccb0186c00 2021-04-09T18:25:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.136 7fccb5154700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc90018900 0x7fcc9001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.136 7fccb5154700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc90018900 0x7fcc9001adc0 crc :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.155 7fccb7338700 1 -- 172.21.15.32:0/3080137584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb01866d0 con 0x7fccb0186c00 2021-04-09T18:25:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.159 7fcca5ffb700 1 -- 172.21.15.32:0/3080137584 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcca0000940 con 0x7fccb0186c00 2021-04-09T18:25:49.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.347 7fccb7338700 1 -- 172.21.15.32:0/3080137584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb01866d0 con 0x7fccb0186c00 2021-04-09T18:25:49.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.348 7fcca5ffb700 1 -- 172.21.15.32:0/3080137584 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fcca0001a90 con 0x7fccb0186c00 2021-04-09T18:25:49.417 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:49.418 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":131,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:25:34.289498","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:25:30.677877","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-04-09 18:25:32.094578","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:25:49.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 -- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc90018900 msgr2=0x7fcc9001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc90018900 0x7fcc9001adc0 crc :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 -- 172.21.15.32:0/3080137584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccb0186c00 msgr2=0x7fccb0187000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccb0186c00 0x7fccb0187000 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7fcca0007680 tx=0x7fcca000a960).stop 2021-04-09T18:25:49.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 -- 172.21.15.32:0/3080137584 shutdown_connections 2021-04-09T18:25:49.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc90018900 0x7fcc9001adc0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fccb0186c00 0x7fccb0187000 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccb0191360 0x7fccb0191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 --2- 172.21.15.32:0/3080137584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccb0185060 0x7fccb0185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 -- 172.21.15.32:0/3080137584 >> 172.21.15.32:0/3080137584 conn(0x7fccb01837d0 msgr2=0x7fccb0195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:49.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 -- 172.21.15.32:0/3080137584 shutdown_connections 2021-04-09T18:25:49.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.350 7fccb7338700 1 -- 172.21.15.32:0/3080137584 wait complete. 2021-04-09T18:25:49.429 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:25:30.677877', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:25:32.094578', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:25:49.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:25:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.514 7f327e58a700 1 Processor -- start 2021-04-09T18:25:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.514 7f327e58a700 1 -- start start 2021-04-09T18:25:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.514 7f327e58a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f327810d010 0x7f32780d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.514 7f327e58a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f32780d8490 0x7f327810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.514 7f327e58a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327810d480 con 0x7f32780d1da0 2021-04-09T18:25:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.514 7f327e58a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32780d8d30 con 0x7f327810d010 2021-04-09T18:25:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.514 7f327e58a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32780d9220 con 0x7f32780d8490 2021-04-09T18:25:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.514 7f3277fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f327810d010 0x7f32780d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f3277fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f327810d010 0x7f32780d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52682/0 (socket says 172.21.15.32:52682) 2021-04-09T18:25:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f3277fff700 1 -- 172.21.15.32:0/3261963573 learned_addr learned my addr 172.21.15.32:0/3261963573 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f327cba7700 1 --2- 172.21.15.32:0/3261963573 >> v2:172.21.15.86:3300/0 conn(0x7f32780d8490 0x7f327810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f3277fff700 1 -- 172.21.15.32:0/3261963573 >> v2:172.21.15.86:3300/0 conn(0x7f32780d8490 msgr2=0x7f327810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f3277fff700 1 --2- 172.21.15.32:0/3261963573 >> v2:172.21.15.86:3300/0 conn(0x7f32780d8490 0x7f327810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f3277fff700 1 -- 172.21.15.32:0/3261963573 >> v1:172.21.15.86:6789/0 conn(0x7f32780d1da0 legacy=0x7f32780d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f3277fff700 1 -- 172.21.15.32:0/3261963573 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327810e020 con 0x7f327810d010 2021-04-09T18:25:49.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f3277fff700 1 --2- 172.21.15.32:0/3261963573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327810d010 0x7f32780d1780 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 rx=0x7f3268006700 tx=0x7f3268009b80).ready entity=mon.0 client_cookie=92478868e37d8af1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:49.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f32777fe700 1 -- 172.21.15.32:0/3261963573 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f326800c0c0 con 0x7f327810d010 2021-04-09T18:25:49.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f32777fe700 1 -- 172.21.15.32:0/3261963573 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326800c440 con 0x7f327810d010 2021-04-09T18:25:49.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f32777fe700 1 -- 172.21.15.32:0/3261963573 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f326800c920 con 0x7f327810d010 2021-04-09T18:25:49.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f327e58a700 1 -- 172.21.15.32:0/3261963573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327810d010 msgr2=0x7f32780d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.515 7f327e58a700 1 --2- 172.21.15.32:0/3261963573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327810d010 0x7f32780d1780 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 rx=0x7f3268006700 tx=0x7f3268009b80).stop 2021-04-09T18:25:49.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.516 7f327e58a700 1 -- 172.21.15.32:0/3261963573 shutdown_connections 2021-04-09T18:25:49.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.516 7f327e58a700 1 --2- 172.21.15.32:0/3261963573 >> v2:172.21.15.86:3300/0 conn(0x7f32780d8490 0x7f327810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.516 7f327e58a700 1 --2- 172.21.15.32:0/3261963573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327810d010 0x7f32780d1780 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.516 7f327e58a700 1 -- 172.21.15.32:0/3261963573 >> 172.21.15.32:0/3261963573 conn(0x7f32780ce5f0 msgr2=0x7f32780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:49.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.516 7f327e58a700 1 -- 172.21.15.32:0/3261963573 shutdown_connections 2021-04-09T18:25:49.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.516 7f327e58a700 1 -- 172.21.15.32:0/3261963573 wait complete. 2021-04-09T18:25:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.536 7f327e58a700 1 Processor -- start 2021-04-09T18:25:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.536 7f327e58a700 1 -- start start 2021-04-09T18:25:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f327e58a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327809e530 0x7f327809b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f327e58a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f327809e930 0x7f327809f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f327e58a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f327809f990 0x7f3278094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f327e58a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327809edf0 con 0x7f327809e530 2021-04-09T18:25:49.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f327e58a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327809fd90 con 0x7f327809f990 2021-04-09T18:25:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f327e58a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327809bd40 con 0x7f327809e930 2021-04-09T18:25:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f3277fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327809e530 0x7f327809b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f3277fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327809e530 0x7f327809b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52688/0 (socket says 172.21.15.32:52688) 2021-04-09T18:25:49.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f3277fff700 1 -- 172.21.15.32:0/496169566 learned_addr learned my addr 172.21.15.32:0/496169566 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f326ffff700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f327809e930 0x7f327809f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f327cba7700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f327809f990 0x7f3278094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f3277fff700 1 -- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f327809e930 msgr2=0x7f327809f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f3277fff700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f327809e930 0x7f327809f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f3277fff700 1 -- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f327809f990 msgr2=0x7f3278094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f3277fff700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f327809f990 0x7f3278094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f3277fff700 1 -- 172.21.15.32:0/496169566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3278095290 con 0x7f327809e530 2021-04-09T18:25:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f3277fff700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327809e530 0x7f327809b720 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 rx=0x7f326800b580 tx=0x7f326800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f32757fa700 1 -- 172.21.15.32:0/496169566 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f326800bd10 con 0x7f327809e530 2021-04-09T18:25:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f32757fa700 1 -- 172.21.15.32:0/496169566 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3268002030 con 0x7f327809e530 2021-04-09T18:25:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f327e58a700 1 -- 172.21.15.32:0/496169566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3278095290 con 0x7f327809e530 2021-04-09T18:25:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f32757fa700 1 -- 172.21.15.32:0/496169566 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3268002550 con 0x7f327809e530 2021-04-09T18:25:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.537 7f327e58a700 1 -- 172.21.15.32:0/496169566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3278095290 con 0x7f327809e530 2021-04-09T18:25:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.538 7f32757fa700 1 -- 172.21.15.32:0/496169566 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3268002210 con 0x7f327809e530 2021-04-09T18:25:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.538 7f32757fa700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f325c0188d0 0x7f325c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.538 7f32757fa700 1 -- 172.21.15.32:0/496169566 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3268032b70 con 0x7f327809e530 2021-04-09T18:25:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.538 7f326ffff700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f325c0188d0 0x7f325c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.539 7f326ffff700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f325c0188d0 0x7f325c01add0 crc :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:49.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.558 7f327e58a700 1 -- 172.21.15.32:0/496169566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3278095b10 con 0x7f327809e530 2021-04-09T18:25:49.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.562 7f32757fa700 1 -- 172.21.15.32:0/496169566 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3268002300 con 0x7f327809e530 2021-04-09T18:25:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.739 7f327e58a700 1 -- 172.21.15.32:0/496169566 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f3278095b10 con 0x7f327809e530 2021-04-09T18:25:49.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.739 7f32757fa700 1 -- 172.21.15.32:0/496169566 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v131) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3268002030 con 0x7f327809e530 2021-04-09T18:25:49.809 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:49.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 -- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f325c0188d0 msgr2=0x7f325c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f325c0188d0 0x7f325c01add0 crc :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 -- 172.21.15.32:0/496169566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327809e530 msgr2=0x7f327809b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:49.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327809e530 0x7f327809b720 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 rx=0x7f326800b580 tx=0x7f326800b920).stop 2021-04-09T18:25:49.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 -- 172.21.15.32:0/496169566 shutdown_connections 2021-04-09T18:25:49.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f325c0188d0 0x7f325c01add0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f327809f990 0x7f3278094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f327809e930 0x7f327809f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 --2- 172.21.15.32:0/496169566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f327809e530 0x7f327809b720 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 -- 172.21.15.32:0/496169566 >> 172.21.15.32:0/496169566 conn(0x7f32780c4290 msgr2=0x7f32780c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:49.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 -- 172.21.15.32:0/496169566 shutdown_connections 2021-04-09T18:25:49.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.742 7f327e58a700 1 -- 172.21.15.32:0/496169566 wait complete. 2021-04-09T18:25:49.821 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:25:49.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.904 7fbcf6157700 1 Processor -- start 2021-04-09T18:25:49.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.904 7fbcf6157700 1 -- start start 2021-04-09T18:25:49.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbcf6157700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbcf00cffc0 0x7fbcf00d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbcf6157700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbcf00d7820 0x7fbcf00d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbcf6157700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00d8240 0x7fbcf00d5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbcf6157700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf00d9200 con 0x7fbcf00d7820 2021-04-09T18:25:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbcf6157700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf00d72a0 con 0x7fbcf00cffc0 2021-04-09T18:25:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbcf6157700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00d62b0 con 0x7fbcf00d8240 2021-04-09T18:25:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbceeffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbcf00d7820 0x7fbcf00d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbceeffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbcf00d7820 0x7fbcf00d7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52700/0 (socket says 172.21.15.32:52700) 2021-04-09T18:25:49.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbceeffd700 1 -- 172.21.15.32:0/2213557637 learned_addr learned my addr 172.21.15.32:0/2213557637 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:49.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.905 7fbceffff700 1 --2- 172.21.15.32:0/2213557637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00d8240 0x7fbcf00d5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbcef7fe700 1 --2- 172.21.15.32:0/2213557637 >> v2:172.21.15.86:3300/0 conn(0x7fbcf00cffc0 0x7fbcf00d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:49.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbceeffd700 1 -- 172.21.15.32:0/2213557637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00d8240 msgr2=0x7fbcf00d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:49.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbceeffd700 1 --2- 172.21.15.32:0/2213557637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00d8240 0x7fbcf00d5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbceeffd700 1 -- 172.21.15.32:0/2213557637 >> v2:172.21.15.86:3300/0 conn(0x7fbcf00cffc0 msgr2=0x7fbcf00d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:49.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbceeffd700 1 --2- 172.21.15.32:0/2213557637 >> v2:172.21.15.86:3300/0 conn(0x7fbcf00cffc0 0x7fbcf00d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:49.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbceeffd700 1 -- 172.21.15.32:0/2213557637 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf00d64a0 con 0x7fbcf00d7820 2021-04-09T18:25:49.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbcef7fe700 1 --2- 172.21.15.32:0/2213557637 >> v2:172.21.15.86:3300/0 conn(0x7fbcf00cffc0 0x7fbcf00d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:25:50.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbceeffd700 1 --2- 172.21.15.32:0/2213557637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00d7820 0x7fbcf00d7c20 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7fbce4006700 tx=0x7fbce4009b00).ready entity=mon.0 client_cookie=7de73da0b962fcee server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:50.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbcee7fc700 1 -- 172.21.15.32:0/2213557637 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbce400c0c0 con 0x7fbcf00d7820 2021-04-09T18:25:50.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbcee7fc700 1 -- 172.21.15.32:0/2213557637 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce400c400 con 0x7fbcf00d7820 2021-04-09T18:25:50.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbcee7fc700 1 -- 172.21.15.32:0/2213557637 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbce400c8e0 con 0x7fbcf00d7820 2021-04-09T18:25:50.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbcf6157700 1 -- 172.21.15.32:0/2213557637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00d7820 msgr2=0x7fbcf00d7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.906 7fbcf6157700 1 --2- 172.21.15.32:0/2213557637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00d7820 0x7fbcf00d7c20 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7fbce4006700 tx=0x7fbce4009b00).stop 2021-04-09T18:25:50.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.907 7fbcf6157700 1 -- 172.21.15.32:0/2213557637 shutdown_connections 2021-04-09T18:25:50.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.907 7fbcf6157700 1 --2- 172.21.15.32:0/2213557637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00d8240 0x7fbcf00d5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.907 7fbcf6157700 1 --2- 172.21.15.32:0/2213557637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00d7820 0x7fbcf00d7c20 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.907 7fbcf6157700 1 --2- 172.21.15.32:0/2213557637 >> v2:172.21.15.86:3300/0 conn(0x7fbcf00cffc0 0x7fbcf00d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.907 7fbcf6157700 1 -- 172.21.15.32:0/2213557637 >> 172.21.15.32:0/2213557637 conn(0x7fbcf00ce5e0 msgr2=0x7fbcf00cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:50.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.907 7fbcf6157700 1 -- 172.21.15.32:0/2213557637 shutdown_connections 2021-04-09T18:25:50.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.907 7fbcf6157700 1 -- 172.21.15.32:0/2213557637 wait complete. 2021-04-09T18:25:50.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.927 7fbcf6157700 1 Processor -- start 2021-04-09T18:25:50.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.927 7fbcf6157700 1 -- start start 2021-04-09T18:25:50.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcf6157700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00b89a0 0x7fbcf00c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcf6157700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00b8da0 0x7fbcf00be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcf6157700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00b91a0 0x7fbcf00bead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcf6157700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00c39b0 con 0x7fbcf00b89a0 2021-04-09T18:25:50.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcf6157700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00c0d60 con 0x7fbcf00b91a0 2021-04-09T18:25:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcf6157700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00bf0f0 con 0x7fbcf00b8da0 2021-04-09T18:25:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00b89a0 0x7fbcf00c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00b89a0 0x7fbcf00c31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52702/0 (socket says 172.21.15.32:52702) 2021-04-09T18:25:50.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcef7fe700 1 -- 172.21.15.32:0/2080974947 learned_addr learned my addr 172.21.15.32:0/2080974947 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:50.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbceeffd700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00b8da0 0x7fbcf00be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbceffff700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00b91a0 0x7fbcf00bead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcef7fe700 1 -- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00b8da0 msgr2=0x7fbcf00be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcef7fe700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00b8da0 0x7fbcf00be4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcef7fe700 1 -- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00b91a0 msgr2=0x7fbcf00bead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcef7fe700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00b91a0 0x7fbcf00bead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcef7fe700 1 -- 172.21.15.32:0/2080974947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf00bf310 con 0x7fbcf00b89a0 2021-04-09T18:25:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcef7fe700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00b89a0 0x7fbcf00c31a0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 rx=0x7fbce0006f20 tx=0x7fbce000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:50.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcd7fff700 1 -- 172.21.15.32:0/2080974947 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbce000bd80 con 0x7fbcf00b89a0 2021-04-09T18:25:50.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcd7fff700 1 -- 172.21.15.32:0/2080974947 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce0000c90 con 0x7fbcf00b89a0 2021-04-09T18:25:50.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf00bf310 con 0x7fbcf00b89a0 2021-04-09T18:25:50.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.928 7fbcd7fff700 1 -- 172.21.15.32:0/2080974947 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbce00022b0 con 0x7fbcf00b89a0 2021-04-09T18:25:50.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.929 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf00bf310 con 0x7fbcf00b89a0 2021-04-09T18:25:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.929 7fbcd7fff700 1 -- 172.21.15.32:0/2080974947 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbce000a000 con 0x7fbcf00b89a0 2021-04-09T18:25:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.929 7fbcd7fff700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.929 7fbcd7fff700 1 -- 172.21.15.32:0/2080974947 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbce0032a30 con 0x7fbcf00b89a0 2021-04-09T18:25:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.929 7fbceeffd700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.930 7fbceeffd700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001adc0 crc :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:50.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.949 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf01b70c0 con 0x7fbcf00b89a0 2021-04-09T18:25:50.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:49.953 7fbcd7fff700 1 -- 172.21.15.32:0/2080974947 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbce0006bb0 con 0x7fbcf00b89a0 2021-04-09T18:25:50.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.133 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fbcf00d72a0 con 0x7fbcf00b89a0 2021-04-09T18:25:50.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.133 7fbcd7fff700 1 -- 172.21.15.32:0/2080974947 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v131) v1 ==== 104+0+10 (secure 0 0 0) 0x7fbce0006bb0 con 0x7fbcf00b89a0 2021-04-09T18:25:50.202 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:50.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 msgr2=0x7fbcd001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001adc0 crc :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00b89a0 msgr2=0x7fbcf00c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00b89a0 0x7fbcf00c31a0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 rx=0x7fbce0006f20 tx=0x7fbce000b990).stop 2021-04-09T18:25:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 shutdown_connections 2021-04-09T18:25:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcd0018930 0x7fbcd001adc0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00b91a0 0x7fbcf00bead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00b8da0 0x7fbcf00be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 --2- 172.21.15.32:0/2080974947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbcf00b89a0 0x7fbcf00c31a0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 >> 172.21.15.32:0/2080974947 conn(0x7fbcf00a06a0 msgr2=0x7fbcf008fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:50.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 shutdown_connections 2021-04-09T18:25:50.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.136 7fbcf6157700 1 -- 172.21.15.32:0/2080974947 wait complete. 2021-04-09T18:25:50.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:25:50.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.299 7f4d2871e700 1 Processor -- start 2021-04-09T18:25:50.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.299 7f4d2871e700 1 -- start start 2021-04-09T18:25:50.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2871e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4d2010d010 0x7f4d200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2871e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4d200d1da0 0x7f4d200d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2871e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2010d480 con 0x7f4d200d82c0 2021-04-09T18:25:50.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2871e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d200d8d30 con 0x7f4d2010d010 2021-04-09T18:25:50.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2871e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d200d9220 con 0x7f4d200d1da0 2021-04-09T18:25:50.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2653a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4d2010d010 0x7f4d200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2653a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4d2010d010 0x7f4d200d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52710/0 (socket says 172.21.15.32:52710) 2021-04-09T18:25:50.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2653a700 1 -- 172.21.15.32:0/619264232 learned_addr learned my addr 172.21.15.32:0/619264232 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:50.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d25d39700 1 --2- 172.21.15.32:0/619264232 >> v2:172.21.15.86:3300/0 conn(0x7f4d200d1da0 0x7f4d200d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2653a700 1 -- 172.21.15.32:0/619264232 >> v2:172.21.15.86:3300/0 conn(0x7f4d200d1da0 msgr2=0x7f4d200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:50.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2653a700 1 --2- 172.21.15.32:0/619264232 >> v2:172.21.15.86:3300/0 conn(0x7f4d200d1da0 0x7f4d200d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2653a700 1 -- 172.21.15.32:0/619264232 >> v1:172.21.15.86:6789/0 conn(0x7f4d200d82c0 legacy=0x7f4d2010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2653a700 1 -- 172.21.15.32:0/619264232 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d200d86f0 con 0x7f4d2010d010 2021-04-09T18:25:50.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d25d39700 1 --2- 172.21.15.32:0/619264232 >> v2:172.21.15.86:3300/0 conn(0x7f4d200d1da0 0x7f4d200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:50.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d25538700 1 -- 172.21.15.32:0/619264232 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915324051 0 0) 0x7f4d18000d30 con 0x7f4d200d82c0 2021-04-09T18:25:50.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.300 7f4d2653a700 1 --2- 172.21.15.32:0/619264232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2010d010 0x7f4d200d1780 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f4d1c006700 tx=0x7f4d1c009b80).ready entity=mon.0 client_cookie=828644ae6ff735b9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:50.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.301 7f4d25538700 1 -- 172.21.15.32:0/619264232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d1c00c0c0 con 0x7f4d2010d010 2021-04-09T18:25:50.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.301 7f4d25538700 1 -- 172.21.15.32:0/619264232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c00c440 con 0x7f4d2010d010 2021-04-09T18:25:50.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.301 7f4d25538700 1 -- 172.21.15.32:0/619264232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d1c00c920 con 0x7f4d2010d010 2021-04-09T18:25:50.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.301 7f4d2871e700 1 -- 172.21.15.32:0/619264232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2010d010 msgr2=0x7f4d200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.301 7f4d2871e700 1 --2- 172.21.15.32:0/619264232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2010d010 0x7f4d200d1780 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f4d1c006700 tx=0x7f4d1c009b80).stop 2021-04-09T18:25:50.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.301 7f4d2871e700 1 -- 172.21.15.32:0/619264232 shutdown_connections 2021-04-09T18:25:50.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.301 7f4d2871e700 1 --2- 172.21.15.32:0/619264232 >> v2:172.21.15.86:3300/0 conn(0x7f4d200d1da0 0x7f4d200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.301 7f4d2871e700 1 --2- 172.21.15.32:0/619264232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2010d010 0x7f4d200d1780 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.301 7f4d2871e700 1 -- 172.21.15.32:0/619264232 >> 172.21.15.32:0/619264232 conn(0x7f4d200ce5f0 msgr2=0x7f4d200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.302 7f4d2871e700 1 -- 172.21.15.32:0/619264232 shutdown_connections 2021-04-09T18:25:50.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.302 7f4d2871e700 1 -- 172.21.15.32:0/619264232 wait complete. 2021-04-09T18:25:50.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d2871e700 1 Processor -- start 2021-04-09T18:25:50.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d2871e700 1 -- start start 2021-04-09T18:25:50.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d2871e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d200b1680 0x7f4d200b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d2871e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2009e330 0x7f4d200b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d2871e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d200b2190 0x7f4d2009f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d2871e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2009e730 con 0x7f4d2009e330 2021-04-09T18:25:50.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d2871e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2009f6c0 con 0x7f4d200b2190 2021-04-09T18:25:50.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d2871e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2009f8e0 con 0x7f4d200b1680 2021-04-09T18:25:50.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d25d39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2009e330 0x7f4d200b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d25d39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2009e330 0x7f4d200b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52720/0 (socket says 172.21.15.32:52720) 2021-04-09T18:25:50.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d25d39700 1 -- 172.21.15.32:0/3792871015 learned_addr learned my addr 172.21.15.32:0/3792871015 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:50.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d2653a700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d200b1680 0x7f4d200b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.322 7f4d26d3b700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d200b2190 0x7f4d2009f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d25d39700 1 -- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d200b1680 msgr2=0x7f4d200b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d25d39700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d200b1680 0x7f4d200b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d25d39700 1 -- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d200b2190 msgr2=0x7f4d2009f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d25d39700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d200b2190 0x7f4d2009f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d25d39700 1 -- 172.21.15.32:0/3792871015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2009fb00 con 0x7f4d2009e330 2021-04-09T18:25:50.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d25d39700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2009e330 0x7f4d200b1b70 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 rx=0x7f4d10006f40 tx=0x7f4d1000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:50.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d16ffd700 1 -- 172.21.15.32:0/3792871015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d1000bd80 con 0x7f4d2009e330 2021-04-09T18:25:50.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d16ffd700 1 -- 172.21.15.32:0/3792871015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d10000c90 con 0x7f4d2009e330 2021-04-09T18:25:50.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2009fb00 con 0x7f4d2009e330 2021-04-09T18:25:50.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d16ffd700 1 -- 172.21.15.32:0/3792871015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d10002220 con 0x7f4d2009e330 2021-04-09T18:25:50.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.323 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2009fb00 con 0x7f4d2009e330 2021-04-09T18:25:50.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.324 7f4d16ffd700 1 -- 172.21.15.32:0/3792871015 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4d1000a000 con 0x7f4d2009e330 2021-04-09T18:25:50.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.324 7f4d16ffd700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d08018930 0x7f4d0801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.324 7f4d16ffd700 1 -- 172.21.15.32:0/3792871015 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4d100329e0 con 0x7f4d2009e330 2021-04-09T18:25:50.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.324 7f4d2653a700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d08018930 0x7f4d0801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.324 7f4d2653a700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d08018930 0x7f4d0801adc0 crc :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:50.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.344 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2010d480 con 0x7f4d2009e330 2021-04-09T18:25:50.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.347 7f4d16ffd700 1 -- 172.21.15.32:0/3792871015 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4d10000c90 con 0x7f4d2009e330 2021-04-09T18:25:50.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.526 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f4d2010d480 con 0x7f4d2009e330 2021-04-09T18:25:50.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.526 7f4d16ffd700 1 -- 172.21.15.32:0/3792871015 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v131) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4d10006b00 con 0x7f4d2009e330 2021-04-09T18:25:50.595 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:25:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d08018930 msgr2=0x7f4d0801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d08018930 0x7f4d0801adc0 crc :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2009e330 msgr2=0x7f4d200b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2009e330 0x7f4d200b1b70 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 rx=0x7f4d10006f40 tx=0x7f4d1000b990).stop 2021-04-09T18:25:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 shutdown_connections 2021-04-09T18:25:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d08018930 0x7f4d0801adc0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d2009e330 0x7f4d200b1b70 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d200b2190 0x7f4d2009f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 --2- 172.21.15.32:0/3792871015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d200b1680 0x7f4d200b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 >> 172.21.15.32:0/3792871015 conn(0x7f4d200c4290 msgr2=0x7f4d200c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 shutdown_connections 2021-04-09T18:25:50.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.529 7f4d2871e700 1 -- 172.21.15.32:0/3792871015 wait complete. 2021-04-09T18:25:50.608 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:25:50.609 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:50.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.690 7f63d0604700 1 Processor -- start 2021-04-09T18:25:50.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.690 7f63d0604700 1 -- start start 2021-04-09T18:25:50.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63d0604700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80d7ca0 0x7f63c80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63d0604700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c810d480 con 0x7f63c810da00 2021-04-09T18:25:50.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63d0604700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c80d86c0 con 0x7f63c810d010 2021-04-09T18:25:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63d0604700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80d9220 con 0x7f63c80d7ca0 2021-04-09T18:25:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63cec21700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f63c810da00 0x7f63c810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34082/0 (socket says 172.21.15.32:34082) 2021-04-09T18:25:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63cec21700 1 -- 172.21.15.32:0/3500308483 learned_addr learned my addr 172.21.15.32:0/3500308483 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:50.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63cdc1f700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80d7ca0 0x7f63c80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080244318 0 0) 0x7f63c0000d30 con 0x7f63c810da00 2021-04-09T18:25:50.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4002fe0 con 0x7f63c810da00 2021-04-09T18:25:50.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.691 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099096947 0 0) 0x7f63c4000c60 con 0x7f63c810d010 2021-04-09T18:25:50.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4003210 con 0x7f63c810d010 2021-04-09T18:25:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 258879280 0 0) 0x7f63c0001460 con 0x7f63c810da00 2021-04-09T18:25:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80d7ca0 msgr2=0x7f63c80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80d7ca0 0x7f63c80d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 >> v1:172.21.15.86:6789/0 conn(0x7f63c810d010 legacy=0x7f63c80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c80d1f70 con 0x7f63c810da00 2021-04-09T18:25:50.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f63c0001980 con 0x7f63c810da00 2021-04-09T18:25:50.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cdc1f700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80d7ca0 0x7f63c80d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:50.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 >> v1:172.21.15.32:6789/0 conn(0x7f63c810da00 legacy=0x7f63c810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63b4003b90 0x7f63b4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63b4006770 0x7f63b4008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b4009500 0x7f63b400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40502e0 con 0x7f63b4009500 2021-04-09T18:25:50.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b4050500 con 0x7f63b4006770 2021-04-09T18:25:50.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b4050720 con 0x7f63b4003b90 2021-04-09T18:25:50.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cec21700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b4009500 0x7f63b400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cdc1f700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63b4006770 0x7f63b4008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63ce420700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63b4003b90 0x7f63b4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cec21700 1 -- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63b4003b90 msgr2=0x7f63b4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:50.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cec21700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63b4003b90 0x7f63b4005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cec21700 1 -- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63b4006770 msgr2=0x7f63b4008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cec21700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63b4006770 0x7f63b4008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cec21700 1 -- 172.21.15.32:0/3500308483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40509c0 con 0x7f63b4009500 2021-04-09T18:25:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cec21700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b4009500 0x7f63b400ba20 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7f63c0007a10 tx=0x7f63c000a310).ready entity=mon.0 client_cookie=62b773ce694b910c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:50.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c00016e0 con 0x7f63b4009500 2021-04-09T18:25:50.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c00070c0 con 0x7f63b4009500 2021-04-09T18:25:50.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cd41e700 1 -- 172.21.15.32:0/3500308483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c0007550 con 0x7f63b4009500 2021-04-09T18:25:50.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63d0604700 1 -- 172.21.15.32:0/3500308483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b4009500 msgr2=0x7f63b400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63d0604700 1 --2- 172.21.15.32:0/3500308483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b4009500 0x7f63b400ba20 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7f63c0007a10 tx=0x7f63c000a310).stop 2021-04-09T18:25:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.692 7f63cec21700 1 -- 172.21.15.32:0/3500308483 reap_dead start 2021-04-09T18:25:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.693 7f63d0604700 1 -- 172.21.15.32:0/3500308483 shutdown_connections 2021-04-09T18:25:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.693 7f63d0604700 1 -- 172.21.15.32:0/3500308483 >> 172.21.15.32:0/3500308483 conn(0x7f63c80ce5f0 msgr2=0x7f63c80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:50.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.693 7f63d0604700 1 -- 172.21.15.32:0/3500308483 shutdown_connections 2021-04-09T18:25:50.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.693 7f63d0604700 1 -- 172.21.15.32:0/3500308483 wait complete. 2021-04-09T18:25:50.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.713 7f63d0604700 1 Processor -- start 2021-04-09T18:25:50.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.713 7f63d0604700 1 -- start start 2021-04-09T18:25:50.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63d0604700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c809a5e0 0x7f63c81884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63d0604700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c80a3350 0x7f63c80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63d0604700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80a15b0 0x7f63c80a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63d0604700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80a40d0 con 0x7f63c80a3350 2021-04-09T18:25:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63d0604700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80a2dd0 con 0x7f63c809a5e0 2021-04-09T18:25:50.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63d0604700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c8188ad0 con 0x7f63c80a15b0 2021-04-09T18:25:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cdc1f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c80a3350 0x7f63c80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cdc1f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c80a3350 0x7f63c80a3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52738/0 (socket says 172.21.15.32:52738) 2021-04-09T18:25:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cdc1f700 1 -- 172.21.15.32:0/1123498519 learned_addr learned my addr 172.21.15.32:0/1123498519 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63ce420700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c809a5e0 0x7f63c81884b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cec21700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80a15b0 0x7f63c80a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cdc1f700 1 -- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80a15b0 msgr2=0x7f63c80a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cdc1f700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80a15b0 0x7f63c80a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cdc1f700 1 -- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c809a5e0 msgr2=0x7f63c81884b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:50.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cdc1f700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c809a5e0 0x7f63c81884b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:50.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cdc1f700 1 -- 172.21.15.32:0/1123498519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c80a3d70 con 0x7f63c80a3350 2021-04-09T18:25:50.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.714 7f63cdc1f700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c80a3350 0x7f63c80a3750 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 rx=0x7f63b8006f40 tx=0x7f63b800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:50.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.715 7f63beffd700 1 -- 172.21.15.32:0/1123498519 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b800bd80 con 0x7f63c80a3350 2021-04-09T18:25:50.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.715 7f63beffd700 1 -- 172.21.15.32:0/1123498519 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8000c90 con 0x7f63c80a3350 2021-04-09T18:25:50.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.715 7f63d0604700 1 -- 172.21.15.32:0/1123498519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c80a3d70 con 0x7f63c80a3350 2021-04-09T18:25:50.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.715 7f63beffd700 1 -- 172.21.15.32:0/1123498519 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b80022b0 con 0x7f63c80a3350 2021-04-09T18:25:50.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.715 7f63d0604700 1 -- 172.21.15.32:0/1123498519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c80a3d70 con 0x7f63c80a3350 2021-04-09T18:25:50.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.715 7f63beffd700 1 -- 172.21.15.32:0/1123498519 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f63b800a000 con 0x7f63c80a3350 2021-04-09T18:25:50.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.715 7f63beffd700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b0018930 0x7f63b001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:50.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.715 7f63beffd700 1 -- 172.21.15.32:0/1123498519 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f63b8032a00 con 0x7f63c80a3350 2021-04-09T18:25:50.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.716 7f63ce420700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b0018930 0x7f63b001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:50.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.716 7f63ce420700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b0018930 0x7f63b001adc0 crc :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.735 7f63d0604700 1 -- 172.21.15.32:0/1123498519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c810d480 con 0x7f63c80a3350 2021-04-09T18:25:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.738 7f63beffd700 1 -- 172.21.15.32:0/1123498519 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f63b8000c90 con 0x7f63c80a3350 2021-04-09T18:25:50.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.929 7f63d0604700 1 -- 172.21.15.32:0/1123498519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63c810d480 con 0x7f63c80a3350 2021-04-09T18:25:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.930 7f63beffd700 1 -- 172.21.15.32:0/1123498519 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2785 (secure 0 0 0) 0x7f63b800bd80 con 0x7f63c80a3350 2021-04-09T18:25:50.999 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 268 2021-04-09T18:25:50.999 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:51.000 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":268,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:25:35.619821","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:25:51.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.932 7f63d0604700 1 -- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b0018930 msgr2=0x7f63b001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.932 7f63d0604700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b0018930 0x7f63b001adc0 crc :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.932 7f63d0604700 1 -- 172.21.15.32:0/1123498519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c80a3350 msgr2=0x7f63c80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.932 7f63d0604700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c80a3350 0x7f63c80a3750 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 rx=0x7f63b8006f40 tx=0x7f63b800b990).stop 2021-04-09T18:25:51.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.933 7f63d0604700 1 -- 172.21.15.32:0/1123498519 shutdown_connections 2021-04-09T18:25:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.933 7f63d0604700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63b0018930 0x7f63b001adc0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.933 7f63d0604700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63c80a3350 0x7f63c80a3750 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.933 7f63d0604700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63c80a15b0 0x7f63c80a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.933 7f63d0604700 1 --2- 172.21.15.32:0/1123498519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63c809a5e0 0x7f63c81884b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:51.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.933 7f63d0604700 1 -- 172.21.15.32:0/1123498519 >> 172.21.15.32:0/1123498519 conn(0x7f63c81a0db0 msgr2=0x7f63c81a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:51.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.933 7f63d0604700 1 -- 172.21.15.32:0/1123498519 shutdown_connections 2021-04-09T18:25:51.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:50.933 7f63d0604700 1 -- 172.21.15.32:0/1123498519 wait complete. 2021-04-09T18:25:51.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 268, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:25:33.288479', 'modified': '2021-04-09 18:25:35.619821', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9085': {'gid': 9085, 'name': 'd', 'rank': 0, 'incarnation': 265, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3538582472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3538582472}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3538582472}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:25:51.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:25:51.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:25:51.542 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:25:51.542 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:25:51.542 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:25:51.543 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 9085, 2021-04-09T18:25:51.543 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:25:51.543 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:25:51.543 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 18.184677876999999, 2021-04-09T18:25:51.543 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 268, 2021-04-09T18:25:51.544 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 131, 2021-04-09T18:25:51.544 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 131, 2021-04-09T18:25:51.544 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 23.632838494000001 2021-04-09T18:25:51.544 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:25:51.548 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 9085, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.184677876999999, "mdsmap_epoch": 268, "osdmap_epoch": 131, "osdmap_epoch_barrier": 131, "uptime": 23.632838494000001 } 2021-04-09T18:25:51.549 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:25:51.549 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:25:51.549 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:25:51.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:51.550 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:51.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:25:51.657 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:25:51.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:25:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:25:51.716 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:25:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:25:51.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:51.718 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:25:51.761 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:25:51.762 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:51.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:25:51.863 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:25:51.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:25:51.923 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:25:51.923 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:25:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:25:51.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:51.925 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:25:51.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:25:51.885 7fe470727f80 -1 init, newargv = 0x55a12195a290 newargc=9ceph-fuse[ 2021-04-09T18:25:51.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:76766]: starting ceph client 2021-04-09T18:25:51.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[76766]: starting fuse 2021-04-09T18:25:51.970 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:25:51.971 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:25:51.971 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:51.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:25:51.972 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:25:51.972 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:25:51.973 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:25:51.973 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:25:51.974 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:51.974 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:25:51.974 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:25:51.975 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:25:51.975 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:51.976 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:25:51.976 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:25:51.976 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:25:51.977 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:25:51.977 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:51.978 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:25:51.979 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:25:51.980 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:25:51.981 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:25:51.981 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:25:51.982 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:25:51.982 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:25:51.982 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:25:52.092 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.76766.asok 2021-04-09T18:25:52.096 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76766.asok 2021-04-09T18:25:52.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:52.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76766.asok status 2021-04-09T18:25:52.221 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:25:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:25:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:25:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:25:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:25:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:25:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:25:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "76766", 2021-04-09T18:25:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:25:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:25:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:25:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9588, 2021-04-09T18:25:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:25:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:25:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:25:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 9588 2021-04-09T18:25:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:25:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:25:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3756328610 2021-04-09T18:25:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:25:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:25:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:25:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:25:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3756328610 2021-04-09T18:25:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:25:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.9588 172.21.15.32:0/3756328610", 2021-04-09T18:25:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/3756328610", 2021-04-09T18:25:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:25:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 268, 2021-04-09T18:25:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 131, 2021-04-09T18:25:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:25:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:25:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:25:52.229 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:25:52.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:52.231 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:52.232 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:52.276 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:25:52.277 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:25:52.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:52.278 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:52.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:52.338 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:52.339 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:52.383 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:25:52.384 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:25:52.384 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:25:52.384 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:25:52.385 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:52.486 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[76766]: fuse finished with error 0 and tester_r 0 2021-04-09T18:25:52.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:25:52.488 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:25:52.489 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:52.533 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:25:52.533 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:25:52.534 INFO:teuthology.orchestra.run:waiting for 900 2021-04-09T18:25:52.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:25:52.535 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:25:52.616 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:25:52.617 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:25:52.617 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:25:52.618 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:25:52.618 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:25:52.618 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:25:52.619 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:25:52.619 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:25:52.625 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:25:52.625 DEBUG:teuthology.parallel:result is None 2021-04-09T18:25:52.626 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:25:52.627 DEBUG:teuthology.parallel:result is None 2021-04-09T18:25:52.628 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:25:52.628 DEBUG:teuthology.parallel:result is None 2021-04-09T18:25:52.629 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:25:52.629 DEBUG:teuthology.parallel:result is None 2021-04-09T18:25:52.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:52.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.744 7f06697b2700 1 Processor -- start 2021-04-09T18:25:52.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.744 7f06697b2700 1 -- start start 2021-04-09T18:25:52.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f06697b2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066410d010 0x7f06640d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:52.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f06697b2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f06640d1da0 0x7f06640d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:52.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f06697b2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066410d480 con 0x7f06640d82c0 2021-04-09T18:25:52.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f06697b2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f066410e780 con 0x7f06640d1da0 2021-04-09T18:25:52.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f06697b2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06640d8d30 con 0x7f066410d010 2021-04-09T18:25:52.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0662d73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f06640d1da0 0x7f06640d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:52.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0662d73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f06640d1da0 0x7f06640d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52762/0 (socket says 172.21.15.32:52762) 2021-04-09T18:25:52.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0662d73700 1 -- 172.21.15.32:0/248355692 learned_addr learned my addr 172.21.15.32:0/248355692 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:52.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0663574700 1 --2- 172.21.15.32:0/248355692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066410d010 0x7f06640d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:52.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0662d73700 1 -- 172.21.15.32:0/248355692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066410d010 msgr2=0x7f06640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:25:52.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0662d73700 1 --2- 172.21.15.32:0/248355692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066410d010 0x7f06640d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:52.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0662d73700 1 -- 172.21.15.32:0/248355692 >> v1:172.21.15.86:6789/0 conn(0x7f06640d82c0 legacy=0x7f066410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:52.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0662572700 1 -- 172.21.15.32:0/248355692 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680528935 0 0) 0x7f065c0009b0 con 0x7f06640d82c0 2021-04-09T18:25:52.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0662d73700 1 -- 172.21.15.32:0/248355692 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06640d9220 con 0x7f06640d1da0 2021-04-09T18:25:52.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0663574700 1 --2- 172.21.15.32:0/248355692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066410d010 0x7f06640d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:52.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.745 7f0662d73700 1 --2- 172.21.15.32:0/248355692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640d1da0 0x7f06640d7ca0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 rx=0x7f0658006a80 tx=0x7f0658009f00).ready entity=mon.0 client_cookie=8dde3a0c39f34888 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f0662572700 1 -- 172.21.15.32:0/248355692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065800b5c0 con 0x7f06640d1da0 2021-04-09T18:25:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f0662572700 1 -- 172.21.15.32:0/248355692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065800b8c0 con 0x7f06640d1da0 2021-04-09T18:25:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f0662572700 1 -- 172.21.15.32:0/248355692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065800bda0 con 0x7f06640d1da0 2021-04-09T18:25:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f06697b2700 1 -- 172.21.15.32:0/248355692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640d1da0 msgr2=0x7f06640d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:52.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f06697b2700 1 --2- 172.21.15.32:0/248355692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640d1da0 0x7f06640d7ca0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 rx=0x7f0658006a80 tx=0x7f0658009f00).stop 2021-04-09T18:25:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f06697b2700 1 -- 172.21.15.32:0/248355692 shutdown_connections 2021-04-09T18:25:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f06697b2700 1 --2- 172.21.15.32:0/248355692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06640d1da0 0x7f06640d7ca0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f06697b2700 1 --2- 172.21.15.32:0/248355692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066410d010 0x7f06640d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:52.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f06697b2700 1 -- 172.21.15.32:0/248355692 >> 172.21.15.32:0/248355692 conn(0x7f06640ce5f0 msgr2=0x7f06640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f06697b2700 1 -- 172.21.15.32:0/248355692 shutdown_connections 2021-04-09T18:25:52.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.746 7f06697b2700 1 -- 172.21.15.32:0/248355692 wait complete. 2021-04-09T18:25:52.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.766 7f06697b2700 1 Processor -- start 2021-04-09T18:25:52.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f06697b2700 1 -- start start 2021-04-09T18:25:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f06697b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06641aca20 0x7f06641ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f06697b2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066418bd40 0x7f066418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f06697b2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f066408fba0 0x7f066408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:52.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f06697b2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066418c7e0 con 0x7f06641aca20 2021-04-09T18:25:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f06697b2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066418ca30 con 0x7f066408fba0 2021-04-09T18:25:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f06697b2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06640905f0 con 0x7f066418bd40 2021-04-09T18:25:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f0663574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06641aca20 0x7f06641ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f0663574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06641aca20 0x7f06641ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52764/0 (socket says 172.21.15.32:52764) 2021-04-09T18:25:52.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f0663574700 1 -- 172.21.15.32:0/1277227072 learned_addr learned my addr 172.21.15.32:0/1277227072 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:52.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.767 7f0662d73700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066418bd40 0x7f066418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:52.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f0663d75700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f066408fba0 0x7f066408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:52.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f0663574700 1 -- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066418bd40 msgr2=0x7f066418c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:52.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f0663574700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066418bd40 0x7f066418c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f0663574700 1 -- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f066408fba0 msgr2=0x7f066408ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f0663574700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f066408fba0 0x7f066408ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f0663574700 1 -- 172.21.15.32:0/1277227072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06640b4300 con 0x7f06641aca20 2021-04-09T18:25:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f0662d73700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066418bd40 0x7f066418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:25:52.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f0663574700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06641aca20 0x7f06641ace20 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f0654006f40 tx=0x7f065400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f064bfff700 1 -- 172.21.15.32:0/1277227072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065400bd80 con 0x7f06641aca20 2021-04-09T18:25:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f064bfff700 1 -- 172.21.15.32:0/1277227072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0654000c90 con 0x7f06641aca20 2021-04-09T18:25:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f06697b2700 1 -- 172.21.15.32:0/1277227072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06640b4300 con 0x7f06641aca20 2021-04-09T18:25:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f064bfff700 1 -- 172.21.15.32:0/1277227072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0654002220 con 0x7f06641aca20 2021-04-09T18:25:52.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.768 7f06697b2700 1 -- 172.21.15.32:0/1277227072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06640b4300 con 0x7f06641aca20 2021-04-09T18:25:52.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.769 7f064bfff700 1 -- 172.21.15.32:0/1277227072 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f065400a000 con 0x7f06641aca20 2021-04-09T18:25:52.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.769 7f064bfff700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0644018930 0x7f064401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:52.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.769 7f064bfff700 1 -- 172.21.15.32:0/1277227072 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0654032a10 con 0x7f06641aca20 2021-04-09T18:25:52.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.769 7f0662d73700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0644018930 0x7f064401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:52.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.769 7f0662d73700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0644018930 0x7f064401adc0 crc :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:52.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.789 7f06697b2700 1 -- 172.21.15.32:0/1277227072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0664182d70 con 0x7f06641aca20 2021-04-09T18:25:52.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.792 7f064bfff700 1 -- 172.21.15.32:0/1277227072 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0654006b00 con 0x7f06641aca20 2021-04-09T18:25:53.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.985 7f06697b2700 1 -- 172.21.15.32:0/1277227072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f066418c760 con 0x7f06641aca20 2021-04-09T18:25:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.986 7f064bfff700 1 -- 172.21.15.32:0/1277227072 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2785 (secure 0 0 0) 0x7f065400a000 con 0x7f06641aca20 2021-04-09T18:25:53.055 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 268 2021-04-09T18:25:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:53.056 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":268,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:25:35.619821","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:25:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 -- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0644018930 msgr2=0x7f064401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:53.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0644018930 0x7f064401adc0 crc :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 -- 172.21.15.32:0/1277227072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06641aca20 msgr2=0x7f06641ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06641aca20 0x7f06641ace20 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f0654006f40 tx=0x7f065400b990).stop 2021-04-09T18:25:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 -- 172.21.15.32:0/1277227072 shutdown_connections 2021-04-09T18:25:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0644018930 0x7f064401adc0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:53.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f066408fba0 0x7f066408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f066418bd40 0x7f066418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 --2- 172.21.15.32:0/1277227072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06641aca20 0x7f06641ace20 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 -- 172.21.15.32:0/1277227072 >> 172.21.15.32:0/1277227072 conn(0x7f066419b460 msgr2=0x7f066419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:53.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 -- 172.21.15.32:0/1277227072 shutdown_connections 2021-04-09T18:25:53.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:52.988 7f06697b2700 1 -- 172.21.15.32:0/1277227072 wait complete. 2021-04-09T18:25:53.068 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:25:58.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:25:58.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.161 7fa8efc1c700 1 Processor -- start 2021-04-09T18:25:58.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.161 7fa8efc1c700 1 -- start start 2021-04-09T18:25:58.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.162 7fa8efc1c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8e80d7ca0 0x7fa8e80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:58.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.162 7fa8efc1c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e810d480 con 0x7fa8e810da00 2021-04-09T18:25:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.162 7fa8efc1c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e80d86c0 con 0x7fa8e810d010 2021-04-09T18:25:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.162 7fa8efc1c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e80d9220 con 0x7fa8e80d7ca0 2021-04-09T18:25:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.162 7fa8ee239700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa8e810da00 0x7fa8e810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34132/0 (socket says 172.21.15.32:34132) 2021-04-09T18:25:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.162 7fa8ee239700 1 -- 172.21.15.32:0/3735969506 learned_addr learned my addr 172.21.15.32:0/3735969506 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:25:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.162 7fa8ed237700 1 --2- 172.21.15.32:0/3735969506 >> v2:172.21.15.86:3300/0 conn(0x7fa8e80d7ca0 0x7fa8e80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.162 7fa8eca36700 1 -- 172.21.15.32:0/3735969506 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074613297 0 0) 0x7fa8e0000d30 con 0x7fa8e810da00 2021-04-09T18:25:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.162 7fa8eca36700 1 -- 172.21.15.32:0/3735969506 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d4002fe0 con 0x7fa8e810da00 2021-04-09T18:25:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8eca36700 1 -- 172.21.15.32:0/3735969506 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553385933 0 0) 0x7fa8e4000c60 con 0x7fa8e810d010 2021-04-09T18:25:58.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8eca36700 1 -- 172.21.15.32:0/3735969506 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d4003210 con 0x7fa8e810d010 2021-04-09T18:25:58.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8eca36700 1 -- 172.21.15.32:0/3735969506 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 225707865 0 0) 0x7fa8e0001460 con 0x7fa8e810da00 2021-04-09T18:25:58.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8ed237700 1 -- 172.21.15.32:0/3735969506 >> v1:172.21.15.86:6789/0 conn(0x7fa8e810d010 legacy=0x7fa8e80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:58.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8ed237700 1 -- 172.21.15.32:0/3735969506 >> v1:172.21.15.32:6789/0 conn(0x7fa8e810da00 legacy=0x7fa8e810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:58.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8ed237700 1 -- 172.21.15.32:0/3735969506 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e80d1f70 con 0x7fa8e80d7ca0 2021-04-09T18:25:58.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8ed237700 1 --2- 172.21.15.32:0/3735969506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80d7ca0 0x7fa8e80d80a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7fa8d8006650 tx=0x7fa8d8009a00).ready entity=mon.1 client_cookie=970bbdebab9710b3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8eca36700 1 -- 172.21.15.32:0/3735969506 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8d8009df0 con 0x7fa8e80d7ca0 2021-04-09T18:25:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8eca36700 1 -- 172.21.15.32:0/3735969506 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d80009b0 con 0x7fa8e80d7ca0 2021-04-09T18:25:58.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8eca36700 1 -- 172.21.15.32:0/3735969506 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8d800c2b0 con 0x7fa8e80d7ca0 2021-04-09T18:25:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8efc1c700 1 -- 172.21.15.32:0/3735969506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80d7ca0 msgr2=0x7fa8e80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.163 7fa8efc1c700 1 --2- 172.21.15.32:0/3735969506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80d7ca0 0x7fa8e80d80a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7fa8d8006650 tx=0x7fa8d8009a00).stop 2021-04-09T18:25:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.164 7fa8efc1c700 1 -- 172.21.15.32:0/3735969506 shutdown_connections 2021-04-09T18:25:58.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.164 7fa8efc1c700 1 --2- 172.21.15.32:0/3735969506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80d7ca0 0x7fa8e80d80a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.164 7fa8efc1c700 1 -- 172.21.15.32:0/3735969506 >> 172.21.15.32:0/3735969506 conn(0x7fa8e80ce5f0 msgr2=0x7fa8e80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.164 7fa8efc1c700 1 -- 172.21.15.32:0/3735969506 shutdown_connections 2021-04-09T18:25:58.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.164 7fa8efc1c700 1 -- 172.21.15.32:0/3735969506 wait complete. 2021-04-09T18:25:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.184 7fa8efc1c700 1 Processor -- start 2021-04-09T18:25:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.184 7fa8efc1c700 1 -- start start 2021-04-09T18:25:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8efc1c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81abcb0 0x7fa8e81ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:58.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8efc1c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8191140 0x7fa8e8191540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:58.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8efc1c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e8191b60 0x7fa8e81a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:58.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8efc1c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e8191f60 con 0x7fa8e81abcb0 2021-04-09T18:25:58.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8efc1c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e81ac6d0 con 0x7fa8e8191140 2021-04-09T18:25:58.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8efc1c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e81aa5c0 con 0x7fa8e8191b60 2021-04-09T18:25:58.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8eda38700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81abcb0 0x7fa8e81ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:58.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8eda38700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81abcb0 0x7fa8e81ac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52778/0 (socket says 172.21.15.32:52778) 2021-04-09T18:25:58.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8eda38700 1 -- 172.21.15.32:0/482507924 learned_addr learned my addr 172.21.15.32:0/482507924 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:25:58.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8ee239700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e8191b60 0x7fa8e81a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:58.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8ed237700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8191140 0x7fa8e8191540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:58.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8eda38700 1 -- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e8191b60 msgr2=0x7fa8e81a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:58.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8eda38700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e8191b60 0x7fa8e81a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:58.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8eda38700 1 -- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8191140 msgr2=0x7fa8e8191540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:58.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8eda38700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8191140 0x7fa8e8191540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:58.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8eda38700 1 -- 172.21.15.32:0/482507924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e81aa7e0 con 0x7fa8e81abcb0 2021-04-09T18:25:58.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8eda38700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81abcb0 0x7fa8e81ac0b0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 rx=0x7fa8e4006f10 tx=0x7fa8e400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:58.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8de7fc700 1 -- 172.21.15.32:0/482507924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8e400bd80 con 0x7fa8e81abcb0 2021-04-09T18:25:58.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8de7fc700 1 -- 172.21.15.32:0/482507924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e4000e20 con 0x7fa8e81abcb0 2021-04-09T18:25:58.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e81aa7e0 con 0x7fa8e81abcb0 2021-04-09T18:25:58.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.185 7fa8de7fc700 1 -- 172.21.15.32:0/482507924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8e40054f0 con 0x7fa8e81abcb0 2021-04-09T18:25:58.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.186 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e81aa7e0 con 0x7fa8e81abcb0 2021-04-09T18:25:58.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.186 7fa8de7fc700 1 -- 172.21.15.32:0/482507924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa8e4009fe0 con 0x7fa8e81abcb0 2021-04-09T18:25:58.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.186 7fa8de7fc700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8d0018930 0x7fa8d001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:25:58.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.186 7fa8de7fc700 1 -- 172.21.15.32:0/482507924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa8e4009fe0 con 0x7fa8e81abcb0 2021-04-09T18:25:58.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.186 7fa8ed237700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8d0018930 0x7fa8d001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:25:58.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.187 7fa8ed237700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8d0018930 0x7fa8d001ae10 crc :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:25:58.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.206 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e8182d50 con 0x7fa8e81abcb0 2021-04-09T18:25:58.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.209 7fa8de7fc700 1 -- 172.21.15.32:0/482507924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa8e4009fe0 con 0x7fa8e81abcb0 2021-04-09T18:25:58.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.402 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8e8182d50 con 0x7fa8e81abcb0 2021-04-09T18:25:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.403 7fa8de7fc700 1 -- 172.21.15.32:0/482507924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2785 (secure 0 0 0) 0x7fa8e4000bc0 con 0x7fa8e81abcb0 2021-04-09T18:25:58.471 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 268 2021-04-09T18:25:58.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:25:58.472 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":268,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:25:35.619821","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:25:58.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8d0018930 msgr2=0x7fa8d001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:58.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8d0018930 0x7fa8d001ae10 crc :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:58.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81abcb0 msgr2=0x7fa8e81ac0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:25:58.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81abcb0 0x7fa8e81ac0b0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 rx=0x7fa8e4006f10 tx=0x7fa8e400b990).stop 2021-04-09T18:25:58.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 shutdown_connections 2021-04-09T18:25:58.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa8d0018930 0x7fa8d001ae10 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:58.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e8191b60 0x7fa8e81a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:58.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8191140 0x7fa8e8191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:58.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 --2- 172.21.15.32:0/482507924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81abcb0 0x7fa8e81ac0b0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:25:58.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 >> 172.21.15.32:0/482507924 conn(0x7fa8e81837d0 msgr2=0x7fa8e8195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:25:58.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 shutdown_connections 2021-04-09T18:25:58.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:25:58.405 7fa8efc1c700 1 -- 172.21.15.32:0/482507924 wait complete. 2021-04-09T18:25:58.484 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:26:03.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:03.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.576 7fda72723700 1 Processor -- start 2021-04-09T18:26:03.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.576 7fda72723700 1 -- start start 2021-04-09T18:26:03.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda72723700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda6c10d010 0x7fda6c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:03.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda72723700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c0d1da0 0x7fda6c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:03.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda72723700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c10d480 con 0x7fda6c0d82c0 2021-04-09T18:26:03.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda72723700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6c0d9220 con 0x7fda6c10d010 2021-04-09T18:26:03.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda72723700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c0d86c0 con 0x7fda6c0d1da0 2021-04-09T18:26:03.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda6bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda6c10d010 0x7fda6c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:03.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda70d40700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fda6c0d82c0 0x7fda6c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34148/0 (socket says 172.21.15.32:34148) 2021-04-09T18:26:03.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda70d40700 1 -- 172.21.15.32:0/326107709 learned_addr learned my addr 172.21.15.32:0/326107709 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:03.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda63fff700 1 --2- 172.21.15.32:0/326107709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c0d1da0 0x7fda6c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:03.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda6bfff700 1 -- 172.21.15.32:0/326107709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c0d1da0 msgr2=0x7fda6c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:03.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda6bfff700 1 --2- 172.21.15.32:0/326107709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c0d1da0 0x7fda6c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda6bfff700 1 -- 172.21.15.32:0/326107709 >> v1:172.21.15.32:6789/0 conn(0x7fda6c0d82c0 legacy=0x7fda6c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:03.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda6b7fe700 1 -- 172.21.15.32:0/326107709 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768644393 0 0) 0x7fda64000f90 con 0x7fda6c0d82c0 2021-04-09T18:26:03.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda6bfff700 1 -- 172.21.15.32:0/326107709 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c10e1f0 con 0x7fda6c10d010 2021-04-09T18:26:03.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda6bfff700 1 --2- 172.21.15.32:0/326107709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c10d010 0x7fda6c0d1780 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7fda5c006650 tx=0x7fda5c009a00).ready entity=mon.0 client_cookie=89e9376534c50b33 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:03.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.577 7fda6b7fe700 1 -- 172.21.15.32:0/326107709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5c009df0 con 0x7fda6c10d010 2021-04-09T18:26:03.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda6b7fe700 1 -- 172.21.15.32:0/326107709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c0009b0 con 0x7fda6c10d010 2021-04-09T18:26:03.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda6b7fe700 1 -- 172.21.15.32:0/326107709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5c00c2b0 con 0x7fda6c10d010 2021-04-09T18:26:03.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda72723700 1 -- 172.21.15.32:0/326107709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c10d010 msgr2=0x7fda6c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:03.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda72723700 1 --2- 172.21.15.32:0/326107709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c10d010 0x7fda6c0d1780 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7fda5c006650 tx=0x7fda5c009a00).stop 2021-04-09T18:26:03.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda72723700 1 -- 172.21.15.32:0/326107709 shutdown_connections 2021-04-09T18:26:03.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda72723700 1 --2- 172.21.15.32:0/326107709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c0d1da0 0x7fda6c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda72723700 1 --2- 172.21.15.32:0/326107709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c10d010 0x7fda6c0d1780 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda72723700 1 -- 172.21.15.32:0/326107709 >> 172.21.15.32:0/326107709 conn(0x7fda6c0ce5f0 msgr2=0x7fda6c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:03.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda72723700 1 -- 172.21.15.32:0/326107709 shutdown_connections 2021-04-09T18:26:03.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.578 7fda72723700 1 -- 172.21.15.32:0/326107709 wait complete. 2021-04-09T18:26:03.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda72723700 1 Processor -- start 2021-04-09T18:26:03.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda72723700 1 -- start start 2021-04-09T18:26:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda72723700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c184a30 0x7fda6c1acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:03.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda72723700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c1a6b90 0x7fda6c1a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:03.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda72723700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1a5150 0x7fda6c1a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:03.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda72723700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c1a75b0 con 0x7fda6c184a30 2021-04-09T18:26:03.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda72723700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c184590 con 0x7fda6c1a5150 2021-04-09T18:26:03.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda72723700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c184740 con 0x7fda6c1a6b90 2021-04-09T18:26:03.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda6bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c184a30 0x7fda6c1acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:03.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda6bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c184a30 0x7fda6c1acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52792/0 (socket says 172.21.15.32:52792) 2021-04-09T18:26:03.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda6bfff700 1 -- 172.21.15.32:0/3664380092 learned_addr learned my addr 172.21.15.32:0/3664380092 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:03.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.599 7fda63fff700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c1a6b90 0x7fda6c1a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:03.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda70d40700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1a5150 0x7fda6c1a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:03.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda6bfff700 1 -- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c1a6b90 msgr2=0x7fda6c1a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:03.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda6bfff700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c1a6b90 0x7fda6c1a6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda6bfff700 1 -- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1a5150 msgr2=0x7fda6c1a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda6bfff700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1a5150 0x7fda6c1a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda6bfff700 1 -- 172.21.15.32:0/3664380092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c1a5b70 con 0x7fda6c184a30 2021-04-09T18:26:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda6bfff700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c184a30 0x7fda6c1acb80 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 rx=0x7fda5c0096a0 tx=0x7fda5c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:03.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda697fa700 1 -- 172.21.15.32:0/3664380092 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5c000c90 con 0x7fda6c184a30 2021-04-09T18:26:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda697fa700 1 -- 172.21.15.32:0/3664380092 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c002030 con 0x7fda6c184a30 2021-04-09T18:26:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda72723700 1 -- 172.21.15.32:0/3664380092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c1a5b70 con 0x7fda6c184a30 2021-04-09T18:26:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda697fa700 1 -- 172.21.15.32:0/3664380092 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5c002550 con 0x7fda6c184a30 2021-04-09T18:26:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.600 7fda72723700 1 -- 172.21.15.32:0/3664380092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c1a5b70 con 0x7fda6c184a30 2021-04-09T18:26:03.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.601 7fda697fa700 1 -- 172.21.15.32:0/3664380092 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fda5c002210 con 0x7fda6c184a30 2021-04-09T18:26:03.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.601 7fda697fa700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda50018930 0x7fda5001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:03.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.601 7fda697fa700 1 -- 172.21.15.32:0/3664380092 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fda5c032a70 con 0x7fda6c184a30 2021-04-09T18:26:03.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.601 7fda63fff700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda50018930 0x7fda5001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:03.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.601 7fda63fff700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda50018930 0x7fda5001ae10 crc :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:03.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.621 7fda72723700 1 -- 172.21.15.32:0/3664380092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c1b7080 con 0x7fda6c184a30 2021-04-09T18:26:03.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.624 7fda697fa700 1 -- 172.21.15.32:0/3664380092 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fda5c0009c0 con 0x7fda6c184a30 2021-04-09T18:26:03.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.815 7fda72723700 1 -- 172.21.15.32:0/3664380092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda6c10d480 con 0x7fda6c184a30 2021-04-09T18:26:03.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.816 7fda697fa700 1 -- 172.21.15.32:0/3664380092 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2785 (secure 0 0 0) 0x7fda5c00bd80 con 0x7fda6c184a30 2021-04-09T18:26:03.886 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 268 2021-04-09T18:26:03.886 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:03.886 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":268,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:25:35.619821","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:26:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 -- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda50018930 msgr2=0x7fda5001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:03.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda50018930 0x7fda5001ae10 crc :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 -- 172.21.15.32:0/3664380092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c184a30 msgr2=0x7fda6c1acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c184a30 0x7fda6c1acb80 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 rx=0x7fda5c0096a0 tx=0x7fda5c00b990).stop 2021-04-09T18:26:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 -- 172.21.15.32:0/3664380092 shutdown_connections 2021-04-09T18:26:03.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda50018930 0x7fda5001ae10 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1a5150 0x7fda6c1a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c1a6b90 0x7fda6c1a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 --2- 172.21.15.32:0/3664380092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c184a30 0x7fda6c1acb80 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:03.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 -- 172.21.15.32:0/3664380092 >> 172.21.15.32:0/3664380092 conn(0x7fda6c191560 msgr2=0x7fda6c1b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:03.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 -- 172.21.15.32:0/3664380092 shutdown_connections 2021-04-09T18:26:03.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:03.818 7fda72723700 1 -- 172.21.15.32:0/3664380092 wait complete. 2021-04-09T18:26:03.898 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:26:07.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:26:07.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:26:08.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:09.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.990 7f7c3c358700 1 Processor -- start 2021-04-09T18:26:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.990 7f7c3c358700 1 -- start start 2021-04-09T18:26:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.991 7f7c3c358700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c3410e7d0 0x7f7c340d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.991 7f7c3c358700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c340d7280 con 0x7f7c340cffa0 2021-04-09T18:26:09.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.991 7f7c3c358700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c340d2440 con 0x7f7c340d6c80 2021-04-09T18:26:09.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.991 7f7c3c358700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c340d7080 con 0x7f7c3410e7d0 2021-04-09T18:26:09.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.991 7f7c3a174700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7c340cffa0 0x7f7c340d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34158/0 (socket says 172.21.15.32:34158) 2021-04-09T18:26:09.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.991 7f7c3a174700 1 -- 172.21.15.32:0/326859818 learned_addr learned my addr 172.21.15.32:0/326859818 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:09.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c3a975700 1 --2- 172.21.15.32:0/326859818 >> v2:172.21.15.86:3300/0 conn(0x7f7c3410e7d0 0x7f7c340d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283392827 0 0) 0x7f7c30000ce0 con 0x7f7c340cffa0 2021-04-09T18:26:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c20002fe0 con 0x7f7c340cffa0 2021-04-09T18:26:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 872599116 0 0) 0x7f7c300010e0 con 0x7f7c340cffa0 2021-04-09T18:26:09.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 >> v2:172.21.15.86:3300/0 conn(0x7f7c3410e7d0 msgr2=0x7f7c340d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 --2- 172.21.15.32:0/326859818 >> v2:172.21.15.86:3300/0 conn(0x7f7c3410e7d0 0x7f7c340d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 >> v1:172.21.15.86:6789/0 conn(0x7f7c340d6c80 legacy=0x7f7c3410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c340d91e0 con 0x7f7c340cffa0 2021-04-09T18:26:09.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f7c30001600 con 0x7f7c340cffa0 2021-04-09T18:26:09.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 >> v1:172.21.15.32:6789/0 conn(0x7f7c340cffa0 legacy=0x7f7c340d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c20003af0 0x7f7c20005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c20006840 0x7f7c20008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c20009640 0x7f7c2000bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2000c3b0 con 0x7f7c20003af0 2021-04-09T18:26:09.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2000c5d0 con 0x7f7c20006840 2021-04-09T18:26:09.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39172700 1 -- 172.21.15.32:0/326859818 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20050800 con 0x7f7c20009640 2021-04-09T18:26:09.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c3a174700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c20003af0 0x7f7c20005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c3a975700 1 --2- 172.21.15.32:0/326859818 >> v2:172.21.15.86:3300/0 conn(0x7f7c3410e7d0 0x7f7c340d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:09.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c39973700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c20006840 0x7f7c20008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.992 7f7c3a975700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c20009640 0x7f7c2000bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c3a174700 1 -- 172.21.15.32:0/326859818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c20009640 msgr2=0x7f7c2000bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:09.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c3a174700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c20009640 0x7f7c2000bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c3a174700 1 -- 172.21.15.32:0/326859818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c20006840 msgr2=0x7f7c20008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:09.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c3a174700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c20006840 0x7f7c20008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c3a174700 1 -- 172.21.15.32:0/326859818 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c20050a60 con 0x7f7c20003af0 2021-04-09T18:26:09.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c3a975700 1 -- 172.21.15.32:0/326859818 reap_dead start 2021-04-09T18:26:09.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c39973700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c20006840 0x7f7c20008d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:09.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c3a174700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c20003af0 0x7f7c20005fa0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 rx=0x7f7c30007970 tx=0x7f7c30007d10).ready entity=mon.0 client_cookie=d8f9eb569baeb98e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:09.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c39172700 1 -- 172.21.15.32:0/326859818 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c30000c90 con 0x7f7c20003af0 2021-04-09T18:26:09.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c39172700 1 -- 172.21.15.32:0/326859818 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c300070c0 con 0x7f7c20003af0 2021-04-09T18:26:09.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c39172700 1 -- 172.21.15.32:0/326859818 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c3000a710 con 0x7f7c20003af0 2021-04-09T18:26:09.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c3c358700 1 -- 172.21.15.32:0/326859818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c20003af0 msgr2=0x7f7c20005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.993 7f7c3c358700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c20003af0 0x7f7c20005fa0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 rx=0x7f7c30007970 tx=0x7f7c30007d10).stop 2021-04-09T18:26:09.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.994 7f7c3c358700 1 -- 172.21.15.32:0/326859818 shutdown_connections 2021-04-09T18:26:09.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.994 7f7c3c358700 1 --2- 172.21.15.32:0/326859818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c20003af0 0x7f7c20005fa0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.994 7f7c3c358700 1 -- 172.21.15.32:0/326859818 >> 172.21.15.32:0/326859818 conn(0x7f7c340ce5c0 msgr2=0x7f7c340cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:09.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.994 7f7c3c358700 1 -- 172.21.15.32:0/326859818 shutdown_connections 2021-04-09T18:26:09.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:08.994 7f7c3c358700 1 -- 172.21.15.32:0/326859818 wait complete. 2021-04-09T18:26:09.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.014 7f7c3c358700 1 Processor -- start 2021-04-09T18:26:09.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.014 7f7c3c358700 1 -- start start 2021-04-09T18:26:09.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3c358700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c341acff0 0x7f7c3408fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3c358700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c34090150 0x7f7c3408c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3c358700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3408cca0 0x7f7c3408e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3c358700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3408c100 con 0x7f7c3408cca0 2021-04-09T18:26:09.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3c358700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3408d0a0 con 0x7f7c34090150 2021-04-09T18:26:09.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3c358700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3408e960 con 0x7f7c341acff0 2021-04-09T18:26:09.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a975700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3408cca0 0x7f7c3408e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a975700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3408cca0 0x7f7c3408e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52814/0 (socket says 172.21.15.32:52814) 2021-04-09T18:26:09.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a975700 1 -- 172.21.15.32:0/2726511247 learned_addr learned my addr 172.21.15.32:0/2726511247 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:09.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a174700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c341acff0 0x7f7c3408fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c39973700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c34090150 0x7f7c3408c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a975700 1 -- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c341acff0 msgr2=0x7f7c3408fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a975700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c341acff0 0x7f7c3408fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a975700 1 -- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c34090150 msgr2=0x7f7c3408c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a975700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c34090150 0x7f7c3408c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a975700 1 -- 172.21.15.32:0/2726511247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3408eb80 con 0x7f7c3408cca0 2021-04-09T18:26:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a174700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c341acff0 0x7f7c3408fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3a975700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3408cca0 0x7f7c3408e340 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 rx=0x7f7c2c0099a0 tx=0x7f7c2c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:09.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c2affd700 1 -- 172.21.15.32:0/2726511247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c2c0011f0 con 0x7f7c3408cca0 2021-04-09T18:26:09.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c2affd700 1 -- 172.21.15.32:0/2726511247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2c003030 con 0x7f7c3408cca0 2021-04-09T18:26:09.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3408eb80 con 0x7f7c3408cca0 2021-04-09T18:26:09.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c2affd700 1 -- 172.21.15.32:0/2726511247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c2c001530 con 0x7f7c3408cca0 2021-04-09T18:26:09.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.015 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3408eb80 con 0x7f7c3408cca0 2021-04-09T18:26:09.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.016 7f7c2affd700 1 -- 172.21.15.32:0/2726511247 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7c2c006bd0 con 0x7f7c3408cca0 2021-04-09T18:26:09.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.016 7f7c2affd700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c1c0188d0 0x7f7c1c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.016 7f7c2affd700 1 -- 172.21.15.32:0/2726511247 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7c2c032d60 con 0x7f7c3408cca0 2021-04-09T18:26:09.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.016 7f7c3a174700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c1c0188d0 0x7f7c1c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.017 7f7c3a174700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c1c0188d0 0x7f7c1c01add0 crc :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:09.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.035 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c34182dc0 con 0x7f7c3408cca0 2021-04-09T18:26:09.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.039 7f7c2affd700 1 -- 172.21.15.32:0/2726511247 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7c2c00be30 con 0x7f7c3408cca0 2021-04-09T18:26:09.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.230 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c340d7280 con 0x7f7c3408cca0 2021-04-09T18:26:09.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.231 7f7c2affd700 1 -- 172.21.15.32:0/2726511247 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+2957 (secure 0 0 0) 0x7f7c2c006bd0 con 0x7f7c3408cca0 2021-04-09T18:26:09.302 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 269 2021-04-09T18:26:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:09.303 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":269,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"laggy_since":"2021-04-09 18:26:07.577337","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"laggy_since":"2021-04-09 18:26:07.577350","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"laggy_since":"2021-04-09 18:26:07.577362","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269}],"filesystems":[{"mdsmap":{"epoch":269,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:26:07.577415","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"laggy_since":"2021-04-09 18:26:07.577322","export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:26:09.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c1c0188d0 msgr2=0x7f7c1c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c1c0188d0 0x7f7c1c01add0 crc :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3408cca0 msgr2=0x7f7c3408e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3408cca0 0x7f7c3408e340 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 rx=0x7f7c2c0099a0 tx=0x7f7c2c00bcf0).stop 2021-04-09T18:26:09.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 shutdown_connections 2021-04-09T18:26:09.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7c1c0188d0 0x7f7c1c01add0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c3408cca0 0x7f7c3408e340 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c34090150 0x7f7c3408c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 --2- 172.21.15.32:0/2726511247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c341acff0 0x7f7c3408fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.233 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 >> 172.21.15.32:0/2726511247 conn(0x7f7c3409abb0 msgr2=0x7f7c340a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:09.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.234 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 shutdown_connections 2021-04-09T18:26:09.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.234 7f7c3c358700 1 -- 172.21.15.32:0/2726511247 wait complete. 2021-04-09T18:26:09.314 DEBUG:tasks.ceph_test_case:wait_until_true: success in 15s 2021-04-09T18:26:09.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:26:09.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.393 7f0b9a520700 1 Processor -- start 2021-04-09T18:26:09.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.393 7f0b9a520700 1 -- start start 2021-04-09T18:26:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.393 7f0b9a520700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b9410d010 0x7f0b940d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.393 7f0b9a520700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b940d8490 0x7f0b9410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.393 7f0b9a520700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9410d480 con 0x7f0b940d1da0 2021-04-09T18:26:09.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.393 7f0b9a520700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9410e780 con 0x7f0b940d8490 2021-04-09T18:26:09.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.393 7f0b9a520700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b940d8d30 con 0x7f0b9410d010 2021-04-09T18:26:09.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b98b3d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b940d8490 0x7f0b9410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b93fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b9410d010 0x7f0b940d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b8bfff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0b940d1da0 0x7f0b940d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:60334/0 (socket says 172.21.15.32:60334) 2021-04-09T18:26:09.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b8bfff700 1 -- 172.21.15.32:0/1492732956 learned_addr learned my addr 172.21.15.32:0/1492732956 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:09.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b98b3d700 1 -- 172.21.15.32:0/1492732956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b9410d010 msgr2=0x7f0b940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:09.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b98b3d700 1 --2- 172.21.15.32:0/1492732956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b9410d010 0x7f0b940d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b98b3d700 1 -- 172.21.15.32:0/1492732956 >> v1:172.21.15.86:6789/0 conn(0x7f0b940d1da0 legacy=0x7f0b940d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b98b3d700 1 -- 172.21.15.32:0/1492732956 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b940d9220 con 0x7f0b940d8490 2021-04-09T18:26:09.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b937fe700 1 -- 172.21.15.32:0/1492732956 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182147828 0 0) 0x7f0b7c0010c0 con 0x7f0b940d1da0 2021-04-09T18:26:09.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b93fff700 1 --2- 172.21.15.32:0/1492732956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b9410d010 0x7f0b940d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:09.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b98b3d700 1 --2- 172.21.15.32:0/1492732956 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b940d8490 0x7f0b9410da00 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f0b8c006e50 tx=0x7f0b8c009e70).ready entity=mon.1 client_cookie=a75041ae6ed200c1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:09.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.394 7f0b937fe700 1 -- 172.21.15.32:0/1492732956 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8c006a20 con 0x7f0b940d8490 2021-04-09T18:26:09.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b937fe700 1 -- 172.21.15.32:0/1492732956 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c006790 con 0x7f0b940d8490 2021-04-09T18:26:09.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b937fe700 1 -- 172.21.15.32:0/1492732956 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8c00b700 con 0x7f0b940d8490 2021-04-09T18:26:09.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b9a520700 1 -- 172.21.15.32:0/1492732956 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b940d8490 msgr2=0x7f0b9410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b9a520700 1 --2- 172.21.15.32:0/1492732956 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b940d8490 0x7f0b9410da00 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f0b8c006e50 tx=0x7f0b8c009e70).stop 2021-04-09T18:26:09.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b9a520700 1 -- 172.21.15.32:0/1492732956 shutdown_connections 2021-04-09T18:26:09.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b9a520700 1 --2- 172.21.15.32:0/1492732956 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b940d8490 0x7f0b9410da00 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b9a520700 1 --2- 172.21.15.32:0/1492732956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b9410d010 0x7f0b940d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b9a520700 1 -- 172.21.15.32:0/1492732956 >> 172.21.15.32:0/1492732956 conn(0x7f0b940ce5f0 msgr2=0x7f0b940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:09.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b9a520700 1 -- 172.21.15.32:0/1492732956 shutdown_connections 2021-04-09T18:26:09.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.395 7f0b9a520700 1 -- 172.21.15.32:0/1492732956 wait complete. 2021-04-09T18:26:09.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.415 7f0b9a520700 1 Processor -- start 2021-04-09T18:26:09.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b9a520700 1 -- start start 2021-04-09T18:26:09.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b9a520700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b940b42d0 0x7f0b94183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b9a520700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b941ac090 0x7f0b941ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b9a520700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b94184d30 0x7f0b94185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b9a520700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b940a1a90 con 0x7f0b94184d30 2021-04-09T18:26:09.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b9a520700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b941acab0 con 0x7f0b941ac090 2021-04-09T18:26:09.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b9a520700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94185a00 con 0x7f0b940b42d0 2021-04-09T18:26:09.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b98b3d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b94184d30 0x7f0b94185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b98b3d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b94184d30 0x7f0b94185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52826/0 (socket says 172.21.15.32:52826) 2021-04-09T18:26:09.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b98b3d700 1 -- 172.21.15.32:0/1794103457 learned_addr learned my addr 172.21.15.32:0/1794103457 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b93fff700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b940b42d0 0x7f0b94183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.416 7f0b8bfff700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b941ac090 0x7f0b941ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b98b3d700 1 -- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b940b42d0 msgr2=0x7f0b94183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b98b3d700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b940b42d0 0x7f0b94183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b98b3d700 1 -- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b941ac090 msgr2=0x7f0b941ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b98b3d700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b941ac090 0x7f0b941ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b98b3d700 1 -- 172.21.15.32:0/1794103457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b94185750 con 0x7f0b94184d30 2021-04-09T18:26:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b98b3d700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b94184d30 0x7f0b94185130 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f0b8c0099f0 tx=0x7f0b8c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b917fa700 1 -- 172.21.15.32:0/1794103457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8c0093d0 con 0x7f0b94184d30 2021-04-09T18:26:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b917fa700 1 -- 172.21.15.32:0/1794103457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c0090c0 con 0x7f0b94184d30 2021-04-09T18:26:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b94185750 con 0x7f0b94184d30 2021-04-09T18:26:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b917fa700 1 -- 172.21.15.32:0/1794103457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8c001420 con 0x7f0b94184d30 2021-04-09T18:26:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b94185750 con 0x7f0b94184d30 2021-04-09T18:26:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.417 7f0b917fa700 1 -- 172.21.15.32:0/1794103457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0b8c0011a0 con 0x7f0b94184d30 2021-04-09T18:26:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.418 7f0b917fa700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b78018900 0x7f0b7801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.418 7f0b917fa700 1 -- 172.21.15.32:0/1794103457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0b8c032dd0 con 0x7f0b94184d30 2021-04-09T18:26:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.418 7f0b93fff700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b78018900 0x7f0b7801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.418 7f0b93fff700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b78018900 0x7f0b7801adc0 crc :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:09.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.438 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b94182d10 con 0x7f0b94184d30 2021-04-09T18:26:09.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.441 7f0b917fa700 1 -- 172.21.15.32:0/1794103457 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0b8c000df0 con 0x7f0b94184d30 2021-04-09T18:26:09.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.630 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b9410d480 con 0x7f0b94184d30 2021-04-09T18:26:09.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.630 7f0b917fa700 1 -- 172.21.15.32:0/1794103457 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f0b8c0090a0 con 0x7f0b94184d30 2021-04-09T18:26:09.701 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:09.701 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":131,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:25:34.289498","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:25:30.677877","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-04-09 18:25:32.094578","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"131","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:26:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b78018900 msgr2=0x7f0b7801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b78018900 0x7f0b7801adc0 crc :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b94184d30 msgr2=0x7f0b94185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b94184d30 0x7f0b94185130 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f0b8c0099f0 tx=0x7f0b8c00bd10).stop 2021-04-09T18:26:09.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 shutdown_connections 2021-04-09T18:26:09.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b78018900 0x7f0b7801adc0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b94184d30 0x7f0b94185130 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b941ac090 0x7f0b941ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 --2- 172.21.15.32:0/1794103457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b940b42d0 0x7f0b94183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 >> 172.21.15.32:0/1794103457 conn(0x7f0b9419b890 msgr2=0x7f0b941a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:09.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 shutdown_connections 2021-04-09T18:26:09.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.633 7f0b9a520700 1 -- 172.21.15.32:0/1794103457 wait complete. 2021-04-09T18:26:09.714 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:25:30.677877', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:25:32.094578', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:26:09.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:26:09.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.797 7f472b86c700 1 Processor -- start 2021-04-09T18:26:09.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.797 7f472b86c700 1 -- start start 2021-04-09T18:26:09.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f472b86c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f472410d010 0x7f47240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f472b86c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47240d1da0 0x7f47240d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f472b86c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47240d82c0 0x7f472410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f472b86c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472410e780 con 0x7f47240d82c0 2021-04-09T18:26:09.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f472b86c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47240d8d30 con 0x7f47240d1da0 2021-04-09T18:26:09.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f472b86c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47240d9220 con 0x7f472410d010 2021-04-09T18:26:09.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729e89700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47240d82c0 0x7f472410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729e89700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47240d82c0 0x7f472410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52836/0 (socket says 172.21.15.32:52836) 2021-04-09T18:26:09.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729e89700 1 -- 172.21.15.32:0/2630762914 learned_addr learned my addr 172.21.15.32:0/2630762914 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:09.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4728e87700 1 --2- 172.21.15.32:0/2630762914 >> v2:172.21.15.86:3300/0 conn(0x7f47240d1da0 0x7f47240d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729688700 1 --2- 172.21.15.32:0/2630762914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f472410d010 0x7f47240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729e89700 1 -- 172.21.15.32:0/2630762914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f472410d010 msgr2=0x7f47240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:09.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729e89700 1 --2- 172.21.15.32:0/2630762914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f472410d010 0x7f47240d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729e89700 1 -- 172.21.15.32:0/2630762914 >> v2:172.21.15.86:3300/0 conn(0x7f47240d1da0 msgr2=0x7f47240d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:09.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729e89700 1 --2- 172.21.15.32:0/2630762914 >> v2:172.21.15.86:3300/0 conn(0x7f47240d1da0 0x7f47240d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729e89700 1 -- 172.21.15.32:0/2630762914 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472410d480 con 0x7f47240d82c0 2021-04-09T18:26:09.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.798 7f4729e89700 1 --2- 172.21.15.32:0/2630762914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47240d82c0 0x7f472410da00 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 rx=0x7f471c006f00 tx=0x7f471c009e90).ready entity=mon.0 client_cookie=1b4ee77f45458705 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:09.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f4717fff700 1 -- 172.21.15.32:0/2630762914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f471c00b300 con 0x7f47240d82c0 2021-04-09T18:26:09.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f4717fff700 1 -- 172.21.15.32:0/2630762914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471c00b640 con 0x7f47240d82c0 2021-04-09T18:26:09.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f4717fff700 1 -- 172.21.15.32:0/2630762914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f471c00bb20 con 0x7f47240d82c0 2021-04-09T18:26:09.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f472b86c700 1 -- 172.21.15.32:0/2630762914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47240d82c0 msgr2=0x7f472410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f472b86c700 1 --2- 172.21.15.32:0/2630762914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47240d82c0 0x7f472410da00 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 rx=0x7f471c006f00 tx=0x7f471c009e90).stop 2021-04-09T18:26:09.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f472b86c700 1 -- 172.21.15.32:0/2630762914 shutdown_connections 2021-04-09T18:26:09.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f472b86c700 1 --2- 172.21.15.32:0/2630762914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47240d82c0 0x7f472410da00 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f472b86c700 1 --2- 172.21.15.32:0/2630762914 >> v2:172.21.15.86:3300/0 conn(0x7f47240d1da0 0x7f47240d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f472b86c700 1 --2- 172.21.15.32:0/2630762914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f472410d010 0x7f47240d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.799 7f472b86c700 1 -- 172.21.15.32:0/2630762914 >> 172.21.15.32:0/2630762914 conn(0x7f47240ce5f0 msgr2=0x7f47240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:09.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.800 7f472b86c700 1 -- 172.21.15.32:0/2630762914 shutdown_connections 2021-04-09T18:26:09.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.800 7f472b86c700 1 -- 172.21.15.32:0/2630762914 wait complete. 2021-04-09T18:26:09.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f472b86c700 1 Processor -- start 2021-04-09T18:26:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f472b86c700 1 -- start start 2021-04-09T18:26:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f472b86c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47241a0050 0x7f47241a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f472b86c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47240bdcd0 0x7f47240bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f472b86c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47240bf080 0x7f47240b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f472b86c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47240bf500 con 0x7f47241a0050 2021-04-09T18:26:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f472b86c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47240bb7b0 con 0x7f47240bf080 2021-04-09T18:26:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f472b86c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47240be0d0 con 0x7f47240bdcd0 2021-04-09T18:26:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f4729688700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47241a0050 0x7f47241a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f4729688700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47241a0050 0x7f47241a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52842/0 (socket says 172.21.15.32:52842) 2021-04-09T18:26:09.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f4729688700 1 -- 172.21.15.32:0/4118089080 learned_addr learned my addr 172.21.15.32:0/4118089080 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.820 7f4728e87700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47240bdcd0 0x7f47240bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4729e89700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47240bf080 0x7f47240b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4729688700 1 -- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47240bdcd0 msgr2=0x7f47240bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4729688700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47240bdcd0 0x7f47240bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4729688700 1 -- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47240bf080 msgr2=0x7f47240b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4729688700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47240bf080 0x7f47240b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4729688700 1 -- 172.21.15.32:0/4118089080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47240b1310 con 0x7f47241a0050 2021-04-09T18:26:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4729688700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47241a0050 0x7f47241a0450 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 rx=0x7f4720006d90 tx=0x7f472000b8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4715ffb700 1 -- 172.21.15.32:0/4118089080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f472000bcb0 con 0x7f47241a0050 2021-04-09T18:26:09.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4715ffb700 1 -- 172.21.15.32:0/4118089080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4720002030 con 0x7f47241a0050 2021-04-09T18:26:09.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f472b86c700 1 -- 172.21.15.32:0/4118089080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47240b1310 con 0x7f47241a0050 2021-04-09T18:26:09.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f4715ffb700 1 -- 172.21.15.32:0/4118089080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4720002510 con 0x7f47241a0050 2021-04-09T18:26:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.821 7f472b86c700 1 -- 172.21.15.32:0/4118089080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47240b1310 con 0x7f47241a0050 2021-04-09T18:26:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.822 7f4715ffb700 1 -- 172.21.15.32:0/4118089080 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4720000c90 con 0x7f47241a0050 2021-04-09T18:26:09.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.822 7f4715ffb700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47040188c0 0x7f470401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.822 7f4715ffb700 1 -- 172.21.15.32:0/4118089080 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f47200349a0 con 0x7f47241a0050 2021-04-09T18:26:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.822 7f4728e87700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47040188c0 0x7f470401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.823 7f4728e87700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47040188c0 0x7f470401adc0 crc :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.842 7f472b86c700 1 -- 172.21.15.32:0/4118089080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47240bb7b0 con 0x7f47241a0050 2021-04-09T18:26:09.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:09.845 7f4715ffb700 1 -- 172.21.15.32:0/4118089080 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f47200024a0 con 0x7f47241a0050 2021-04-09T18:26:10.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.025 7f472b86c700 1 -- 172.21.15.32:0/4118089080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f47240bb7b0 con 0x7f47241a0050 2021-04-09T18:26:10.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.025 7f4715ffb700 1 -- 172.21.15.32:0/4118089080 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v131) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4720000c90 con 0x7f47241a0050 2021-04-09T18:26:10.095 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:26:10.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.027 7f472b86c700 1 -- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47040188c0 msgr2=0x7f470401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.027 7f472b86c700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47040188c0 0x7f470401adc0 crc :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 -- 172.21.15.32:0/4118089080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47241a0050 msgr2=0x7f47241a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47241a0050 0x7f47241a0450 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 rx=0x7f4720006d90 tx=0x7f472000b8c0).stop 2021-04-09T18:26:10.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 -- 172.21.15.32:0/4118089080 shutdown_connections 2021-04-09T18:26:10.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47040188c0 0x7f470401adc0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47240bf080 0x7f47240b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47240bdcd0 0x7f47240bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 --2- 172.21.15.32:0/4118089080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47241a0050 0x7f47241a0450 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 -- 172.21.15.32:0/4118089080 >> 172.21.15.32:0/4118089080 conn(0x7f47240975f0 msgr2=0x7f4724096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:10.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 -- 172.21.15.32:0/4118089080 shutdown_connections 2021-04-09T18:26:10.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.028 7f472b86c700 1 -- 172.21.15.32:0/4118089080 wait complete. 2021-04-09T18:26:10.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:26:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.190 7fa0c9f4d700 1 Processor -- start 2021-04-09T18:26:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.190 7fa0c9f4d700 1 -- start start 2021-04-09T18:26:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c9f4d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa0c410d010 0x7fa0c40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c9f4d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa0c40d8490 0x7fa0c410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c9f4d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c410d480 con 0x7fa0c40d1da0 2021-04-09T18:26:10.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c9f4d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c40d8d30 con 0x7fa0c410d010 2021-04-09T18:26:10.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c9f4d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c40d9220 con 0x7fa0c40d8490 2021-04-09T18:26:10.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c37fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa0c410d010 0x7fa0c40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c2ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0c40d1da0 0x7fa0c40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34208/0 (socket says 172.21.15.32:34208) 2021-04-09T18:26:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c37fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa0c410d010 0x7fa0c40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52846/0 (socket says 172.21.15.32:52846) 2021-04-09T18:26:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c2ffd700 1 -- 172.21.15.32:0/1028749841 learned_addr learned my addr 172.21.15.32:0/1028749841 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c3fff700 1 --2- 172.21.15.32:0/1028749841 >> v2:172.21.15.86:3300/0 conn(0x7fa0c40d8490 0x7fa0c410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.191 7fa0c37fe700 1 -- 172.21.15.32:0/1028749841 >> v2:172.21.15.86:3300/0 conn(0x7fa0c40d8490 msgr2=0x7fa0c410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:10.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c37fe700 1 --2- 172.21.15.32:0/1028749841 >> v2:172.21.15.86:3300/0 conn(0x7fa0c40d8490 0x7fa0c410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c37fe700 1 -- 172.21.15.32:0/1028749841 >> v1:172.21.15.32:6789/0 conn(0x7fa0c40d1da0 legacy=0x7fa0c40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c37fe700 1 -- 172.21.15.32:0/1028749841 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c410e020 con 0x7fa0c410d010 2021-04-09T18:26:10.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c27fc700 1 -- 172.21.15.32:0/1028749841 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544701655 0 0) 0x7fa0b8000ce0 con 0x7fa0c40d1da0 2021-04-09T18:26:10.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c37fe700 1 --2- 172.21.15.32:0/1028749841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c410d010 0x7fa0c40d1780 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 rx=0x7fa0b4006650 tx=0x7fa0b4009a80).ready entity=mon.0 client_cookie=3e6216bb39fd47fd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:10.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c27fc700 1 -- 172.21.15.32:0/1028749841 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0b40009b0 con 0x7fa0c410d010 2021-04-09T18:26:10.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c27fc700 1 -- 172.21.15.32:0/1028749841 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b400c0c0 con 0x7fa0c410d010 2021-04-09T18:26:10.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c27fc700 1 -- 172.21.15.32:0/1028749841 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0b400c510 con 0x7fa0c410d010 2021-04-09T18:26:10.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c9f4d700 1 -- 172.21.15.32:0/1028749841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c410d010 msgr2=0x7fa0c40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.192 7fa0c9f4d700 1 --2- 172.21.15.32:0/1028749841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c410d010 0x7fa0c40d1780 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 rx=0x7fa0b4006650 tx=0x7fa0b4009a80).stop 2021-04-09T18:26:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.193 7fa0c9f4d700 1 -- 172.21.15.32:0/1028749841 shutdown_connections 2021-04-09T18:26:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.193 7fa0c9f4d700 1 --2- 172.21.15.32:0/1028749841 >> v2:172.21.15.86:3300/0 conn(0x7fa0c40d8490 0x7fa0c410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.193 7fa0c9f4d700 1 --2- 172.21.15.32:0/1028749841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c410d010 0x7fa0c40d1780 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.193 7fa0c9f4d700 1 -- 172.21.15.32:0/1028749841 >> 172.21.15.32:0/1028749841 conn(0x7fa0c40ce5f0 msgr2=0x7fa0c40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:10.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.193 7fa0c9f4d700 1 -- 172.21.15.32:0/1028749841 shutdown_connections 2021-04-09T18:26:10.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.193 7fa0c9f4d700 1 -- 172.21.15.32:0/1028749841 wait complete. 2021-04-09T18:26:10.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.213 7fa0c9f4d700 1 Processor -- start 2021-04-09T18:26:10.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.213 7fa0c9f4d700 1 -- start start 2021-04-09T18:26:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.213 7fa0c9f4d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c41a0050 0x7fa0c41a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.213 7fa0c9f4d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c40bdcd0 0x7fa0c40bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.213 7fa0c9f4d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c40bf080 0x7fa0c40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.213 7fa0c9f4d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40bf500 con 0x7fa0c40bf080 2021-04-09T18:26:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.213 7fa0c9f4d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40bb7b0 con 0x7fa0c41a0050 2021-04-09T18:26:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.213 7fa0c9f4d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40be0d0 con 0x7fa0c40bdcd0 2021-04-09T18:26:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c40bf080 0x7fa0c40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c40bf080 0x7fa0c40b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52854/0 (socket says 172.21.15.32:52854) 2021-04-09T18:26:10.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c3fff700 1 -- 172.21.15.32:0/1048873309 learned_addr learned my addr 172.21.15.32:0/1048873309 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c2ffd700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c40bdcd0 0x7fa0c40bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c37fe700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c41a0050 0x7fa0c41a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c3fff700 1 -- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c40bdcd0 msgr2=0x7fa0c40bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c3fff700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c40bdcd0 0x7fa0c40bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c3fff700 1 -- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c41a0050 msgr2=0x7fa0c41a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c3fff700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c41a0050 0x7fa0c41a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c3fff700 1 -- 172.21.15.32:0/1048873309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c40be280 con 0x7fa0c40bf080 2021-04-09T18:26:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c3fff700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c40bf080 0x7fa0c40b0cf0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 rx=0x7fa0bc00b970 tx=0x7fa0bc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:10.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0abfff700 1 -- 172.21.15.32:0/1048873309 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0bc000f90 con 0x7fa0c40bf080 2021-04-09T18:26:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0abfff700 1 -- 172.21.15.32:0/1048873309 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc006aa0 con 0x7fa0c40bf080 2021-04-09T18:26:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c40be0d0 con 0x7fa0c40bf080 2021-04-09T18:26:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0abfff700 1 -- 172.21.15.32:0/1048873309 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0bc0015f0 con 0x7fa0c40bf080 2021-04-09T18:26:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.214 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c40be0d0 con 0x7fa0c40bf080 2021-04-09T18:26:10.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.215 7fa0abfff700 1 -- 172.21.15.32:0/1048873309 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa0bc000ed0 con 0x7fa0c40bf080 2021-04-09T18:26:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.215 7fa0abfff700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a4018930 0x7fa0a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.215 7fa0abfff700 1 -- 172.21.15.32:0/1048873309 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa0bc032da0 con 0x7fa0c40bf080 2021-04-09T18:26:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.215 7fa0c37fe700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a4018930 0x7fa0a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.216 7fa0c37fe700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a4018930 0x7fa0a401ae10 crc :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:10.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.235 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c40bb7b0 con 0x7fa0c40bf080 2021-04-09T18:26:10.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.239 7fa0abfff700 1 -- 172.21.15.32:0/1048873309 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa0bc0066f0 con 0x7fa0c40bf080 2021-04-09T18:26:10.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.420 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa0c40bb7b0 con 0x7fa0c40bf080 2021-04-09T18:26:10.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.420 7fa0abfff700 1 -- 172.21.15.32:0/1048873309 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v131) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa0bc006aa0 con 0x7fa0c40bf080 2021-04-09T18:26:10.491 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:26:10.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a4018930 msgr2=0x7fa0a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a4018930 0x7fa0a401ae10 crc :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c40bf080 msgr2=0x7fa0c40b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c40bf080 0x7fa0c40b0cf0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 rx=0x7fa0bc00b970 tx=0x7fa0bc00bd10).stop 2021-04-09T18:26:10.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 shutdown_connections 2021-04-09T18:26:10.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa0a4018930 0x7fa0a401ae10 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0c40bf080 0x7fa0c40b0cf0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c40bdcd0 0x7fa0c40bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 --2- 172.21.15.32:0/1048873309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c41a0050 0x7fa0c41a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 >> 172.21.15.32:0/1048873309 conn(0x7fa0c40975f0 msgr2=0x7fa0c4096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:10.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 shutdown_connections 2021-04-09T18:26:10.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.423 7fa0c9f4d700 1 -- 172.21.15.32:0/1048873309 wait complete. 2021-04-09T18:26:10.503 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:26:10.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.587 7fdf83152700 1 Processor -- start 2021-04-09T18:26:10.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.587 7fdf83152700 1 -- start start 2021-04-09T18:26:10.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf83152700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c10d010 0x7fdf7c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf83152700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdf7c0d8490 0x7fdf7c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf83152700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c10d480 con 0x7fdf7c0d1da0 2021-04-09T18:26:10.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf83152700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7c10e780 con 0x7fdf7c0d8490 2021-04-09T18:26:10.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf83152700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0d8d30 con 0x7fdf7c10d010 2021-04-09T18:26:10.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf80f6e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c10d010 0x7fdf7c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf8176f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdf7c0d8490 0x7fdf7c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf7bfff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdf7c0d1da0 0x7fdf7c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:60376/0 (socket says 172.21.15.32:60376) 2021-04-09T18:26:10.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf80f6e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c10d010 0x7fdf7c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:60086/0 (socket says 172.21.15.32:60086) 2021-04-09T18:26:10.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf8176f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdf7c0d8490 0x7fdf7c10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:44622/0 (socket says 172.21.15.32:44622) 2021-04-09T18:26:10.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.588 7fdf7bfff700 1 -- 172.21.15.32:0/2763192456 learned_addr learned my addr 172.21.15.32:0/2763192456 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:10.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.589 7fdf8176f700 1 -- 172.21.15.32:0/2763192456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c10d010 msgr2=0x7fdf7c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:10.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.589 7fdf8176f700 1 --2- 172.21.15.32:0/2763192456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c10d010 0x7fdf7c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.589 7fdf8176f700 1 -- 172.21.15.32:0/2763192456 >> v1:172.21.15.86:6789/0 conn(0x7fdf7c0d1da0 legacy=0x7fdf7c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.589 7fdf8176f700 1 -- 172.21.15.32:0/2763192456 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0d9220 con 0x7fdf7c0d8490 2021-04-09T18:26:10.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.589 7fdf80f6e700 1 --2- 172.21.15.32:0/2763192456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c10d010 0x7fdf7c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:10.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.589 7fdf8176f700 1 --2- 172.21.15.32:0/2763192456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf7c0d8490 0x7fdf7c10da00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7fdf74006e50 tx=0x7fdf74009e70).ready entity=mon.1 client_cookie=1227bc0514b33cb8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:10.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.590 7fdf7b7fe700 1 -- 172.21.15.32:0/2763192456 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf74006a20 con 0x7fdf7c0d8490 2021-04-09T18:26:10.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.590 7fdf7b7fe700 1 -- 172.21.15.32:0/2763192456 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf74006790 con 0x7fdf7c0d8490 2021-04-09T18:26:10.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.590 7fdf7b7fe700 1 -- 172.21.15.32:0/2763192456 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf7400b700 con 0x7fdf7c0d8490 2021-04-09T18:26:10.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.590 7fdf83152700 1 -- 172.21.15.32:0/2763192456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf7c0d8490 msgr2=0x7fdf7c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.590 7fdf83152700 1 --2- 172.21.15.32:0/2763192456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf7c0d8490 0x7fdf7c10da00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7fdf74006e50 tx=0x7fdf74009e70).stop 2021-04-09T18:26:10.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.591 7fdf83152700 1 -- 172.21.15.32:0/2763192456 shutdown_connections 2021-04-09T18:26:10.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.591 7fdf83152700 1 --2- 172.21.15.32:0/2763192456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf7c0d8490 0x7fdf7c10da00 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.591 7fdf83152700 1 --2- 172.21.15.32:0/2763192456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c10d010 0x7fdf7c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.591 7fdf83152700 1 -- 172.21.15.32:0/2763192456 >> 172.21.15.32:0/2763192456 conn(0x7fdf7c0ce5f0 msgr2=0x7fdf7c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:10.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.591 7fdf83152700 1 -- 172.21.15.32:0/2763192456 shutdown_connections 2021-04-09T18:26:10.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.591 7fdf83152700 1 -- 172.21.15.32:0/2763192456 wait complete. 2021-04-09T18:26:10.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.611 7fdf83152700 1 Processor -- start 2021-04-09T18:26:10.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.611 7fdf83152700 1 -- start start 2021-04-09T18:26:10.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf83152700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf7c0b1680 0x7fdf7c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf83152700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7c09e330 0x7fdf7c0b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf83152700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c0b2190 0x7fdf7c09f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf83152700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c09e730 con 0x7fdf7c09e330 2021-04-09T18:26:10.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf83152700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c09f6c0 con 0x7fdf7c0b1680 2021-04-09T18:26:10.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf83152700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c09f8e0 con 0x7fdf7c0b2190 2021-04-09T18:26:10.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf7bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7c09e330 0x7fdf7c0b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf7bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7c09e330 0x7fdf7c0b1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52868/0 (socket says 172.21.15.32:52868) 2021-04-09T18:26:10.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf7bfff700 1 -- 172.21.15.32:0/3631623836 learned_addr learned my addr 172.21.15.32:0/3631623836 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:10.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf8176f700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c0b2190 0x7fdf7c09f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf80f6e700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf7c0b1680 0x7fdf7c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf7bfff700 1 -- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c0b2190 msgr2=0x7fdf7c09f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf7bfff700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c0b2190 0x7fdf7c09f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf7bfff700 1 -- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf7c0b1680 msgr2=0x7fdf7c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf7bfff700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf7c0b1680 0x7fdf7c0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf7bfff700 1 -- 172.21.15.32:0/3631623836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c09fb00 con 0x7fdf7c09e330 2021-04-09T18:26:10.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.612 7fdf7bfff700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7c09e330 0x7fdf7c0b1b70 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7fdf6400a540 tx=0x7fdf6400a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:10.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.613 7fdf797fa700 1 -- 172.21.15.32:0/3631623836 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf640070d0 con 0x7fdf7c09e330 2021-04-09T18:26:10.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.613 7fdf797fa700 1 -- 172.21.15.32:0/3631623836 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6400acd0 con 0x7fdf7c09e330 2021-04-09T18:26:10.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.613 7fdf83152700 1 -- 172.21.15.32:0/3631623836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c09fb00 con 0x7fdf7c09e330 2021-04-09T18:26:10.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.613 7fdf797fa700 1 -- 172.21.15.32:0/3631623836 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf640045a0 con 0x7fdf7c09e330 2021-04-09T18:26:10.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.613 7fdf83152700 1 -- 172.21.15.32:0/3631623836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c09fb00 con 0x7fdf7c09e330 2021-04-09T18:26:10.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.613 7fdf797fa700 1 -- 172.21.15.32:0/3631623836 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdf64004320 con 0x7fdf7c09e330 2021-04-09T18:26:10.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.613 7fdf797fa700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf60018930 0x7fdf6001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:10.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.613 7fdf797fa700 1 -- 172.21.15.32:0/3631623836 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdf64032a30 con 0x7fdf7c09e330 2021-04-09T18:26:10.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.614 7fdf80f6e700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf60018930 0x7fdf6001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:10.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.614 7fdf80f6e700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf60018930 0x7fdf6001ae10 crc :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:10.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.631 7fdf83152700 1 -- 172.21.15.32:0/3631623836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c10d480 con 0x7fdf7c09e330 2021-04-09T18:26:10.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.635 7fdf797fa700 1 -- 172.21.15.32:0/3631623836 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdf64000c10 con 0x7fdf7c09e330 2021-04-09T18:26:10.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.817 7fdf83152700 1 -- 172.21.15.32:0/3631623836 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fdf7c10d480 con 0x7fdf7c09e330 2021-04-09T18:26:10.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.817 7fdf797fa700 1 -- 172.21.15.32:0/3631623836 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v131) v1 ==== 100+0+10 (secure 0 0 0) 0x7fdf64004320 con 0x7fdf7c09e330 2021-04-09T18:26:10.888 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:26:10.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.819 7fdf83152700 1 -- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf60018930 msgr2=0x7fdf6001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.819 7fdf83152700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf60018930 0x7fdf6001ae10 crc :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 -- 172.21.15.32:0/3631623836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7c09e330 msgr2=0x7fdf7c0b1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7c09e330 0x7fdf7c0b1b70 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7fdf6400a540 tx=0x7fdf6400a8e0).stop 2021-04-09T18:26:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 -- 172.21.15.32:0/3631623836 shutdown_connections 2021-04-09T18:26:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf60018930 0x7fdf6001ae10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf7c09e330 0x7fdf7c0b1b70 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf7c0b2190 0x7fdf7c09f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 --2- 172.21.15.32:0/3631623836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf7c0b1680 0x7fdf7c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 -- 172.21.15.32:0/3631623836 >> 172.21.15.32:0/3631623836 conn(0x7fdf7c0c4290 msgr2=0x7fdf7c0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 -- 172.21.15.32:0/3631623836 shutdown_connections 2021-04-09T18:26:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.820 7fdf83152700 1 -- 172.21.15.32:0/3631623836 wait complete. 2021-04-09T18:26:10.900 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:26:10.901 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:11.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc1168b3700 1 Processor -- start 2021-04-09T18:26:11.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc1168b3700 1 -- start start 2021-04-09T18:26:11.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc1168b3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc11010d010 0x7fc1100d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:11.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc1168b3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100d8490 0x7fc11010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:11.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc1168b3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11010d480 con 0x7fc1100d1da0 2021-04-09T18:26:11.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc1168b3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100d8d30 con 0x7fc11010d010 2021-04-09T18:26:11.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc1168b3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1100d9220 con 0x7fc1100d8490 2021-04-09T18:26:11.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc114ed0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100d8490 0x7fc11010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:11.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc10ffff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc11010d010 0x7fc1100d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:11.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc107fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc1100d1da0 0x7fc1100d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:60392/0 (socket says 172.21.15.32:60392) 2021-04-09T18:26:11.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.979 7fc107fff700 1 -- 172.21.15.32:0/3510333437 learned_addr learned my addr 172.21.15.32:0/3510333437 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:11.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc10f7fe700 1 -- 172.21.15.32:0/3510333437 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468568508 0 0) 0x7fc0f80010c0 con 0x7fc1100d1da0 2021-04-09T18:26:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc10f7fe700 1 -- 172.21.15.32:0/3510333437 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0fc002fe0 con 0x7fc1100d1da0 2021-04-09T18:26:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc114ed0700 1 -- 172.21.15.32:0/3510333437 >> v2:172.21.15.86:3300/0 conn(0x7fc11010d010 msgr2=0x7fc1100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc114ed0700 1 --2- 172.21.15.32:0/3510333437 >> v2:172.21.15.86:3300/0 conn(0x7fc11010d010 0x7fc1100d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc114ed0700 1 -- 172.21.15.32:0/3510333437 >> v1:172.21.15.86:6789/0 conn(0x7fc1100d1da0 legacy=0x7fc1100d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc114ed0700 1 -- 172.21.15.32:0/3510333437 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11010e020 con 0x7fc1100d8490 2021-04-09T18:26:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc10ffff700 1 --2- 172.21.15.32:0/3510333437 >> v2:172.21.15.86:3300/0 conn(0x7fc11010d010 0x7fc1100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc114ed0700 1 --2- 172.21.15.32:0/3510333437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100d8490 0x7fc11010da00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fc108006e50 tx=0x7fc108009e70).ready entity=mon.2 client_cookie=23f689ed8dd90f57 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc10f7fe700 1 -- 172.21.15.32:0/3510333437 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc108006a20 con 0x7fc1100d8490 2021-04-09T18:26:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc10f7fe700 1 -- 172.21.15.32:0/3510333437 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc108006790 con 0x7fc1100d8490 2021-04-09T18:26:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc10f7fe700 1 -- 172.21.15.32:0/3510333437 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc10800b700 con 0x7fc1100d8490 2021-04-09T18:26:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc1168b3700 1 -- 172.21.15.32:0/3510333437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100d8490 msgr2=0x7fc11010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.980 7fc1168b3700 1 --2- 172.21.15.32:0/3510333437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100d8490 0x7fc11010da00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fc108006e50 tx=0x7fc108009e70).stop 2021-04-09T18:26:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.981 7fc1168b3700 1 -- 172.21.15.32:0/3510333437 shutdown_connections 2021-04-09T18:26:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.981 7fc1168b3700 1 --2- 172.21.15.32:0/3510333437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1100d8490 0x7fc11010da00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.981 7fc1168b3700 1 --2- 172.21.15.32:0/3510333437 >> v2:172.21.15.86:3300/0 conn(0x7fc11010d010 0x7fc1100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.981 7fc1168b3700 1 -- 172.21.15.32:0/3510333437 >> 172.21.15.32:0/3510333437 conn(0x7fc1100ce5f0 msgr2=0x7fc1100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.981 7fc1168b3700 1 -- 172.21.15.32:0/3510333437 shutdown_connections 2021-04-09T18:26:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:10.981 7fc1168b3700 1 -- 172.21.15.32:0/3510333437 wait complete. 2021-04-09T18:26:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.001 7fc1168b3700 1 Processor -- start 2021-04-09T18:26:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.001 7fc1168b3700 1 -- start start 2021-04-09T18:26:11.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc1168b3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1101a5210 0x7fc11008eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:11.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc1168b3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1101a8ab0 0x7fc1101a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc1168b3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1101a76d0 0x7fc1101a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc1168b3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11017ea50 con 0x7fc1101a8ab0 2021-04-09T18:26:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc1168b3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11008f4f0 con 0x7fc1101a76d0 2021-04-09T18:26:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc1168b3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11008ea60 con 0x7fc1101a5210 2021-04-09T18:26:11.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc107fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1101a8ab0 0x7fc1101a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:11.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc107fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1101a8ab0 0x7fc1101a8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52882/0 (socket says 172.21.15.32:52882) 2021-04-09T18:26:11.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc107fff700 1 -- 172.21.15.32:0/2506786960 learned_addr learned my addr 172.21.15.32:0/2506786960 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:11.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc10ffff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1101a5210 0x7fc11008eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc114ed0700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1101a76d0 0x7fc1101a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc10ffff700 1 -- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1101a76d0 msgr2=0x7fc1101a7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc10ffff700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1101a76d0 0x7fc1101a7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc10ffff700 1 -- 172.21.15.32:0/2506786960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1101a8ab0 msgr2=0x7fc1101a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc10ffff700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1101a8ab0 0x7fc1101a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc10ffff700 1 -- 172.21.15.32:0/2506786960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1101a80f0 con 0x7fc1101a5210 2021-04-09T18:26:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc107fff700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1101a8ab0 0x7fc1101a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.002 7fc10ffff700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1101a5210 0x7fc11008eed0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc100009540 tx=0x7fc10000b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.003 7fc10d7fa700 1 -- 172.21.15.32:0/2506786960 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc10000bcd0 con 0x7fc1101a5210 2021-04-09T18:26:11.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.003 7fc10d7fa700 1 -- 172.21.15.32:0/2506786960 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc100006740 con 0x7fc1101a5210 2021-04-09T18:26:11.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.003 7fc10d7fa700 1 -- 172.21.15.32:0/2506786960 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc100002220 con 0x7fc1101a5210 2021-04-09T18:26:11.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.003 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1101a80f0 con 0x7fc1101a5210 2021-04-09T18:26:11.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.003 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1101a80f0 con 0x7fc1101a5210 2021-04-09T18:26:11.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.003 7fc10d7fa700 1 -- 172.21.15.32:0/2506786960 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc100009fe0 con 0x7fc1101a5210 2021-04-09T18:26:11.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.003 7fc10d7fa700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f40188c0 0x7fc0f401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:11.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.003 7fc10d7fa700 1 -- 172.21.15.32:0/2506786960 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc100032720 con 0x7fc1101a5210 2021-04-09T18:26:11.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.003 7fc107fff700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f40188c0 0x7fc0f401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:11.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.004 7fc107fff700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f40188c0 0x7fc0f401adc0 crc :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:11.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.023 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1101b7070 con 0x7fc1101a5210 2021-04-09T18:26:11.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.027 7fc10d7fa700 1 -- 172.21.15.32:0/2506786960 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc100000c90 con 0x7fc1101a5210 2021-04-09T18:26:11.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.220 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc11010d480 con 0x7fc1101a5210 2021-04-09T18:26:11.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.220 7fc10d7fa700 1 -- 172.21.15.32:0/2506786960 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+2957 (secure 0 0 0) 0x7fc100009fe0 con 0x7fc1101a5210 2021-04-09T18:26:11.291 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 269 2021-04-09T18:26:11.291 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:11.291 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":269,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"laggy_since":"2021-04-09 18:26:07.577337","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"laggy_since":"2021-04-09 18:26:07.577350","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"laggy_since":"2021-04-09 18:26:07.577362","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269}],"filesystems":[{"mdsmap":{"epoch":269,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:26:07.577415","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"laggy_since":"2021-04-09 18:26:07.577322","export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:26:11.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.222 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f40188c0 msgr2=0x7fc0f401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:11.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.222 7fc1168b3700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f40188c0 0x7fc0f401adc0 crc :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.222 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1101a5210 msgr2=0x7fc11008eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:11.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.222 7fc1168b3700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1101a5210 0x7fc11008eed0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc100009540 tx=0x7fc10000b8e0).stop 2021-04-09T18:26:11.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.223 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 shutdown_connections 2021-04-09T18:26:11.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.223 7fc1168b3700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc0f40188c0 0x7fc0f401adc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.223 7fc1168b3700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1101a8ab0 0x7fc1101a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.223 7fc1168b3700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1101a76d0 0x7fc1101a7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.223 7fc1168b3700 1 --2- 172.21.15.32:0/2506786960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1101a5210 0x7fc11008eed0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:11.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.223 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 >> 172.21.15.32:0/2506786960 conn(0x7fc1101a99c0 msgr2=0x7fc11017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:11.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.223 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 shutdown_connections 2021-04-09T18:26:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:11.223 7fc1168b3700 1 -- 172.21.15.32:0/2506786960 wait complete. 2021-04-09T18:26:11.304 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 269, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:25:33.288479', 'modified': '2021-04-09 18:26:07.577415', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9085': {'gid': 9085, 'name': 'd', 'rank': 0, 'incarnation': 265, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3538582472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3538582472}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3538582472}]}, 'laggy_since': '2021-04-09 18:26:07.577322', 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:26:11.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:26:11.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:26:11.414 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2021-04-09T18:26:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-04-09T18:26:11.422 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:26:11.422 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:26:11.422 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:26:11.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:11.423 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:11.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:26:11.565 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:26:11.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:26:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:26:11.624 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:26:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:26:11.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:11.626 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:26:11.669 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:26:11.670 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:11.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:26:11.772 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:26:11.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:26:11.831 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:26:11.832 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:26:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:26:11.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:11.834 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:26:11.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:26:11.790 7fa6770f1f80 -1 init, newargv = 0x55b41d3ac2a0 newargc=9 2021-04-09T18:26:11.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[77768]: starting ceph client 2021-04-09T18:26:11.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[77768]: probably no MDS server is up? 2021-04-09T18:26:11.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[77768]: ceph mount failed with (65536) Unknown error 65536 2021-04-09T18:26:12.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:daemon-helper: command failed with exit status 1 2021-04-09T18:26:12.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:26:13.041 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:26:13.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:26:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:26:13.099 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:26:13.100 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:26:13.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:13.101 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:26:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:26:13.145 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:26:13.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:26:13.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:26:13.301 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:26:13.301 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:13.302 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:26:13.302 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:13.345 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:26:13.346 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:26:13.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:26:13.347 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:13.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:13.428 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:26:13.429 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:26:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:26:13.473 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:26:13.473 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:26:13.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:26:13.475 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:26:13.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:26:13.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.633 7fcb28009700 1 Processor -- start 2021-04-09T18:26:13.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.633 7fcb28009700 1 -- start start 2021-04-09T18:26:13.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.634 7fcb28009700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcb200d7ca0 0x7fcb200d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:13.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.634 7fcb28009700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2010d480 con 0x7fcb2010da00 2021-04-09T18:26:13.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.634 7fcb28009700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200d86c0 con 0x7fcb2010d010 2021-04-09T18:26:13.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.634 7fcb28009700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb200d9220 con 0x7fcb200d7ca0 2021-04-09T18:26:13.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.634 7fcb26626700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcb2010da00 0x7fcb2010de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34258/0 (socket says 172.21.15.32:34258) 2021-04-09T18:26:13.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.634 7fcb26626700 1 -- 172.21.15.32:0/3647228655 learned_addr learned my addr 172.21.15.32:0/3647228655 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:13.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.634 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039790596 0 0) 0x7fcb18000d30 con 0x7fcb2010da00 2021-04-09T18:26:13.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.634 7fcb25624700 1 --2- 172.21.15.32:0/3647228655 >> v2:172.21.15.86:3300/0 conn(0x7fcb200d7ca0 0x7fcb200d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:13.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.634 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb0c002fe0 con 0x7fcb2010da00 2021-04-09T18:26:13.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1416809389 0 0) 0x7fcb18001460 con 0x7fcb2010da00 2021-04-09T18:26:13.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 >> v2:172.21.15.86:3300/0 conn(0x7fcb200d7ca0 msgr2=0x7fcb200d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:13.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 --2- 172.21.15.32:0/3647228655 >> v2:172.21.15.86:3300/0 conn(0x7fcb200d7ca0 0x7fcb200d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:13.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 >> v1:172.21.15.86:6789/0 conn(0x7fcb2010d010 legacy=0x7fcb200d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:13.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200d1f70 con 0x7fcb2010da00 2021-04-09T18:26:13.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fcb18001980 con 0x7fcb2010da00 2021-04-09T18:26:13.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 >> v1:172.21.15.32:6789/0 conn(0x7fcb2010da00 legacy=0x7fcb2010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:13.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb0c003a40 0x7fcb0c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:13.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb0c006790 0x7fcb0c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:13.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb0c009590 0x7fcb0c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:13.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c00c300 con 0x7fcb0c006790 2021-04-09T18:26:13.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c00c520 con 0x7fcb0c003a40 2021-04-09T18:26:13.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c050750 con 0x7fcb0c009590 2021-04-09T18:26:13.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb25624700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb0c006790 0x7fcb0c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:13.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb25e25700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb0c003a40 0x7fcb0c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:13.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb26626700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb0c009590 0x7fcb0c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:13.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb25624700 1 -- 172.21.15.32:0/3647228655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb0c009590 msgr2=0x7fcb0c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:13.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb25624700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb0c009590 0x7fcb0c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:13.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb25624700 1 -- 172.21.15.32:0/3647228655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb0c003a40 msgr2=0x7fcb0c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:13.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb25624700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb0c003a40 0x7fcb0c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:13.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb25624700 1 -- 172.21.15.32:0/3647228655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c0509b0 con 0x7fcb0c006790 2021-04-09T18:26:13.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb26626700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb0c009590 0x7fcb0c00bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:13.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb26626700 1 -- 172.21.15.32:0/3647228655 reap_dead start 2021-04-09T18:26:13.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb25624700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb0c006790 0x7fcb0c008c90 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 rx=0x7fcb1000c8f0 tx=0x7fcb1000cc90).ready entity=mon.0 client_cookie=49a5e6482eccbaf4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:13.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb10009310 con 0x7fcb0c006790 2021-04-09T18:26:13.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10009610 con 0x7fcb0c006790 2021-04-09T18:26:13.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb24e23700 1 -- 172.21.15.32:0/3647228655 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb10009af0 con 0x7fcb0c006790 2021-04-09T18:26:13.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb28009700 1 -- 172.21.15.32:0/3647228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb0c006790 msgr2=0x7fcb0c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:13.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.635 7fcb28009700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb0c006790 0x7fcb0c008c90 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 rx=0x7fcb1000c8f0 tx=0x7fcb1000cc90).stop 2021-04-09T18:26:13.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.636 7fcb28009700 1 -- 172.21.15.32:0/3647228655 shutdown_connections 2021-04-09T18:26:13.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.636 7fcb28009700 1 --2- 172.21.15.32:0/3647228655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb0c006790 0x7fcb0c008c90 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:13.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.636 7fcb28009700 1 -- 172.21.15.32:0/3647228655 >> 172.21.15.32:0/3647228655 conn(0x7fcb200ce5f0 msgr2=0x7fcb200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:13.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.636 7fcb28009700 1 -- 172.21.15.32:0/3647228655 shutdown_connections 2021-04-09T18:26:13.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.636 7fcb28009700 1 -- 172.21.15.32:0/3647228655 wait complete. 2021-04-09T18:26:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.656 7fcb28009700 1 Processor -- start 2021-04-09T18:26:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb28009700 1 -- start start 2021-04-09T18:26:13.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb28009700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb201a9360 0x7fcb201918f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb28009700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2018c440 0x7fcb2018c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb28009700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb201a80f0 0x7fcb201a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:13.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb28009700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20191f10 con 0x7fcb2018c440 2021-04-09T18:26:13.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb28009700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb201a9760 con 0x7fcb201a9360 2021-04-09T18:26:13.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb28009700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2018ce60 con 0x7fcb201a80f0 2021-04-09T18:26:13.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb25624700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2018c440 0x7fcb2018c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:13.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb25624700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2018c440 0x7fcb2018c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52912/0 (socket says 172.21.15.32:52912) 2021-04-09T18:26:13.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb25624700 1 -- 172.21.15.32:0/3679534398 learned_addr learned my addr 172.21.15.32:0/3679534398 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:13.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb25e25700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb201a9360 0x7fcb201918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:13.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb26626700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb201a80f0 0x7fcb201a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:13.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb25624700 1 -- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb201a80f0 msgr2=0x7fcb201a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:13.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb25624700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb201a80f0 0x7fcb201a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:13.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb25624700 1 -- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb201a9360 msgr2=0x7fcb201918f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:13.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.657 7fcb25624700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb201a9360 0x7fcb201918f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:13.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.658 7fcb25624700 1 -- 172.21.15.32:0/3679534398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb201a8b10 con 0x7fcb2018c440 2021-04-09T18:26:13.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.658 7fcb26626700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb201a80f0 0x7fcb201a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:13.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.658 7fcb25624700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2018c440 0x7fcb2018c840 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 rx=0x7fcb10006fc0 tx=0x7fcb10009f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:13.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.658 7fcb167fc700 1 -- 172.21.15.32:0/3679534398 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb10006030 con 0x7fcb2018c440 2021-04-09T18:26:13.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.658 7fcb167fc700 1 -- 172.21.15.32:0/3679534398 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1000c770 con 0x7fcb2018c440 2021-04-09T18:26:13.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.658 7fcb167fc700 1 -- 172.21.15.32:0/3679534398 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb1000cc90 con 0x7fcb2018c440 2021-04-09T18:26:13.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.658 7fcb28009700 1 -- 172.21.15.32:0/3679534398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb201a8b10 con 0x7fcb2018c440 2021-04-09T18:26:13.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.658 7fcb28009700 1 -- 172.21.15.32:0/3679534398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb201a8b10 con 0x7fcb2018c440 2021-04-09T18:26:13.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.658 7fcb167fc700 1 -- 172.21.15.32:0/3679534398 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcb1000b660 con 0x7fcb2018c440 2021-04-09T18:26:13.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.659 7fcb167fc700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb08018930 0x7fcb0801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:13.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.659 7fcb167fc700 1 -- 172.21.15.32:0/3679534398 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcb100329f0 con 0x7fcb2018c440 2021-04-09T18:26:13.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.659 7fcb25e25700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb08018930 0x7fcb0801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:13.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.659 7fcb25e25700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb08018930 0x7fcb0801ae10 crc :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:13.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.678 7fcb28009700 1 -- 172.21.15.32:0/3679534398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb2018bf60 con 0x7fcb2018c440 2021-04-09T18:26:13.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.682 7fcb167fc700 1 -- 172.21.15.32:0/3679534398 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcb10006030 con 0x7fcb2018c440 2021-04-09T18:26:13.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.870 7fcb28009700 1 -- 172.21.15.32:0/3679534398 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fcb2018bf60 con 0x7fcb2018c440 2021-04-09T18:26:13.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.871 7fcb167fc700 1 -- 172.21.15.32:0/3679534398 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v131) v1 ==== 81+0+30374 (secure 0 0 0) 0x7fcb1000b660 con 0x7fcb2018c440 2021-04-09T18:26:13.943 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:13.944 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:26:13.944 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 131, 2021-04-09T18:26:13.944 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:26:13.944 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:26:13.945 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:25:34.289498", 2021-04-09T18:26:13.945 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:26:13.945 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:26:13.946 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:26:13.946 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:26:13.946 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:26:13.946 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:26:13.947 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:26:13.947 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:26:13.947 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:26:13.947 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:26:13.948 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:26:13.948 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:26:13.948 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:26:13.948 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:26:13.949 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:26:13.949 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 23, 2021-04-09T18:26:13.949 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:26:13.949 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:26:13.950 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:26:13.950 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:26:13.950 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:26:13.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:13.951 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:26:13.951 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:26:13.951 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:26:13.951 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:26:13.951 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:26:13.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:26:13.952 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:26:13.952 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:26:13.952 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:26:13.953 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:26:13.953 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:26:13.953 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:26:13.953 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:26:13.954 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:26:13.954 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:26:13.954 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:26:13.954 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:26:13.955 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:26:13.955 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:26:13.955 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:26:13.955 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:26:13.956 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:26:13.956 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:26:13.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:13.956 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:26:13.957 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:26:13.957 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:26:13.957 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:26:13.957 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:26:13.958 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:26:13.958 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:26:13.958 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:26:13.958 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:26:13.959 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:26:13.959 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:26:13.959 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:26:13.960 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:26:13.960 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:26:13.960 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:26:13.960 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:26:13.961 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:26:13.961 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:26:13.961 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:26:13.961 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:26:13.962 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:26:13.962 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:26:13.962 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:26:13.962 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:26:13.963 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:26:13.963 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:26:13.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:26:13.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:13.964 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:26:13.964 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:26:13.964 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:26:13.964 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:26:13.965 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:26:13.965 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:26:13.965 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:26:13.965 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:26:13.965 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:26:13.966 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:26:13.966 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:26:13.966 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:26:13.966 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:26:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:26:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 22, 2021-04-09T18:26:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:26:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:25:30.677877", 2021-04-09T18:26:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:26:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:26:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:26:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:26:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:26:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:26:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:26:13.970 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:26:13.970 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:26:13.970 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:26:13.970 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:26:13.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:26:13.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:26:13.971 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:26:13.971 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:26:13.971 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:26:13.972 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:26:13.972 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:26:13.972 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:26:13.972 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:26:13.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:13.973 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "130", 2021-04-09T18:26:13.973 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:26:13.973 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:26:13.973 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:26:13.974 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:26:13.974 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:26:13.974 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:26:13.974 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:26:13.975 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:26:13.975 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:26:13.975 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:26:13.975 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:26:13.975 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:26:13.976 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:26:13.976 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:26:13.976 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:26:13.976 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:26:13.977 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:26:13.977 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:26:13.977 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:26:13.977 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:26:13.978 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:26:13.978 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:26:13.978 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:26:13.978 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:26:13.978 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:26:13.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:26:13.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:13.979 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:26:13.979 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:26:13.980 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:26:13.980 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:26:13.980 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:26:13.981 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:26:13.981 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:26:13.981 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:26:13.981 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:26:13.982 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:26:13.982 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:26:13.982 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:26:13.982 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:26:13.983 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:26:13.983 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:26:13.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:13.983 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:26:13.984 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:26:13.984 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:26:13.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:13.984 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:13.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:13.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:13.985 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 23, 2021-04-09T18:26:13.985 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:26:13.986 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:25:32.094578", 2021-04-09T18:26:13.986 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:26:13.986 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:26:13.986 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:26:13.987 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:26:13.987 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:26:13.987 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:26:13.987 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:26:13.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:26:13.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:26:13.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:26:13.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:26:13.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:26:13.989 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:26:13.989 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:26:13.991 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:26:13.991 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:26:13.991 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:26:13.992 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:26:13.992 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:26:13.992 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:26:13.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:13.993 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "131", 2021-04-09T18:26:13.993 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:26:13.993 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:26:13.994 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:26:13.994 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:26:13.994 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:26:13.994 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:26:13.995 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:26:13.995 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:26:13.995 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:26:13.995 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:26:13.996 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:26:13.996 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:26:13.996 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:26:13.996 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:26:13.996 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:26:13.997 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:26:13.997 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:26:13.997 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:26:13.997 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:26:13.998 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:26:13.998 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:26:13.998 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:26:13.998 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:26:13.999 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:26:13.999 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:26:13.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:26:13.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.000 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:26:14.000 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:26:14.000 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:26:14.000 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:26:14.000 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:26:14.001 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:26:14.001 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:26:14.001 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:26:14.001 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:26:14.002 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:26:14.002 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:26:14.002 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:26:14.002 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:26:14.002 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:26:14.003 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:26:14.003 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.003 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.003 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.004 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:26:14.004 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:26:14.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.004 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:26:14.005 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:26:14.005 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:14.005 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:14.005 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:14.006 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:14.006 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:14.006 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:14.006 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:14.006 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:14.007 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:14.007 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:14.007 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:14.007 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.008 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.008 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.008 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:26:14.008 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:14.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.009 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.009 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:26:14.009 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:14.010 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.010 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.010 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:14.011 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.011 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.011 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:26:14.011 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:14.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.012 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.012 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:26:14.013 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:14.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.013 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.013 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:14.014 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.014 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.014 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:26:14.014 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:14.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.015 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.015 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:26:14.016 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:14.016 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.016 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.016 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:14.017 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.017 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.017 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.017 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:26:14.017 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:26:14.018 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:14.019 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.019 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.019 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:26:14.020 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:26:14.020 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:26:14.020 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:26:14.020 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:14.021 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:14.021 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:14.021 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.022 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:26:14.022 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:26:14.022 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:14.022 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:14.022 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:14.023 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:14.023 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:14.023 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:14.023 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:14.024 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 124, 2021-04-09T18:26:14.024 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:14.024 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:14.024 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:14.025 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.025 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.025 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:26:14.025 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:14.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.026 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.026 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:26:14.027 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:14.027 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.027 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.027 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:14.028 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.028 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.028 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:26:14.028 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:14.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.029 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:26:14.030 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:14.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.030 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.030 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:14.031 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.031 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.031 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:26:14.032 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:14.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.032 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:26:14.033 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:14.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.033 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.033 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.034 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:26:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.035 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:26:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.036 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:26:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:26:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:26:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:26:14.037 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:26:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:26:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:14.039 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:14.040 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:26:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:14.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:26:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:14.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:26:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:14.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:26:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:14.046 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.047 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.047 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:14.047 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.048 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:26:14.048 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:14.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.049 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.049 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:26:14.049 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:14.049 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.050 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.050 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:14.050 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.051 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.051 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:26:14.051 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:14.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.052 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:26:14.052 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:14.052 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.053 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.053 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:26:14.053 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:26:14.053 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:26:14.054 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:26:14.054 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:14.054 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:14.054 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:14.055 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.055 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:26:14.055 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:26:14.056 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:14.056 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:14.056 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:14.056 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:14.056 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:14.057 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:14.057 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:14.057 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:14.057 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:14.058 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:14.058 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:14.058 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.059 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:26:14.059 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:14.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.060 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.060 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:26:14.060 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:14.060 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.060 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.061 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:14.061 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.062 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:26:14.062 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:14.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.063 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:26:14.063 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:14.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.063 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.064 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:14.064 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.065 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:26:14.065 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:14.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.066 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.066 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:26:14.066 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:14.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.066 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.067 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:14.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.068 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:26:14.068 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:14.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.069 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.069 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:26:14.069 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:14.069 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.069 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.070 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:26:14.070 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:26:14.070 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:26:14.070 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:26:14.071 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:14.071 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:14.071 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:14.071 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.072 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:26:14.072 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:26:14.072 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:14.073 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:14.073 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:14.073 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:14.073 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:14.073 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:14.074 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:14.074 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:14.074 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:14.074 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:14.074 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:14.075 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.075 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.075 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:26:14.076 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:14.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.076 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.076 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:26:14.077 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:14.077 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.077 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.078 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:14.078 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.078 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:26:14.079 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:14.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.079 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.080 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:26:14.080 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:14.080 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.080 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.081 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:14.081 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.081 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:26:14.082 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:14.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.083 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:26:14.083 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:14.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.084 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.084 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:14.084 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.085 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:26:14.085 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:14.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:26:14.086 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:14.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.087 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.087 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:26:14.088 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:26:14.088 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:26:14.088 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:26:14.088 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:14.088 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:14.089 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:14.089 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.089 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:26:14.090 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:26:14.090 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:14.090 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:14.090 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:14.090 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:14.091 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:14.091 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:14.091 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:14.091 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:14.092 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:14.092 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:14.092 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:14.092 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.093 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.093 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:26:14.093 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:14.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.094 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.094 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:26:14.094 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:14.094 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.095 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.095 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:14.095 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.096 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.096 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:26:14.096 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:14.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.097 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:26:14.097 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:14.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.098 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.098 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:14.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.099 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:26:14.099 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:14.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.100 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.100 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:26:14.100 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:14.100 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.101 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.101 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:14.101 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.102 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:26:14.102 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:14.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.102 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.103 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:26:14.103 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:14.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.104 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.104 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:26:14.104 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:26:14.105 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:26:14.105 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:26:14.105 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:14.105 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:14.105 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:14.106 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.106 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:26:14.106 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:26:14.107 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:14.107 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:14.107 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:14.107 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:14.108 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:14.108 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:14.108 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:14.108 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 124, 2021-04-09T18:26:14.108 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:14.109 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:14.109 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:14.109 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.109 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:26:14.110 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:14.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.111 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:26:14.111 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:14.111 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.111 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.112 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:14.112 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.113 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:26:14.113 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:14.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.114 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:26:14.114 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:14.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.114 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.115 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:14.115 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.115 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.115 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.116 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:26:14.116 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:14.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:26:14.117 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:14.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.117 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.118 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:14.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.118 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:26:14.119 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:14.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.120 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:26:14.120 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:14.120 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.120 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.121 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:26:14.121 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:26:14.121 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:26:14.121 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:26:14.122 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:14.122 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:14.122 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:14.122 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.123 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:26:14.123 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:26:14.123 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:14.124 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:14.124 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:14.124 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:14.124 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:14.125 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:14.125 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:14.125 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:26:14.125 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:14.126 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:14.126 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:14.126 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.127 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:26:14.127 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:14.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.127 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:26:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:14.128 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.129 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.129 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:14.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:26:14.130 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:14.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.131 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:26:14.131 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:14.131 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.132 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.132 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:14.132 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.133 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:26:14.133 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:14.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.134 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.134 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:26:14.134 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:14.134 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.135 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.135 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:14.135 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:14.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:14.136 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:26:14.136 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:14.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:14.137 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:26:14.137 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:14.138 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.138 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.138 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:26:14.138 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:26:14.139 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:26:14.139 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:26:14.139 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:14.139 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:14.140 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:14.140 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:14.140 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.140 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:26:14.141 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:26:14.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.141 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:26:14.141 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:14.141 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:14.142 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:14.142 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:14.142 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:14.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.143 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:26:14.143 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:14.143 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:14.143 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:14.144 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:14.144 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:14.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.145 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:26:14.145 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:14.145 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:14.145 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:14.145 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:14.146 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:14.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.146 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:26:14.146 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:14.147 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:14.147 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:14.147 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:14.147 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:14.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.148 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:26:14.148 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:14.149 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:14.149 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:14.149 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:14.149 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:14.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.150 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:26:14.150 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:14.150 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:14.151 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:14.151 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:14.151 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:14.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.152 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:26:14.152 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:14.152 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:14.153 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:14.153 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:14.153 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:14.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:14.154 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:26:14.154 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:14.154 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:14.154 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:14.155 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:14.155 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:14.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.155 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:26:14.156 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:26:14.156 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:26:14.156 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:26:14.156 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:26:14.157 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:26:14.157 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:26:14.157 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:26:14.157 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:26:14.158 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:26:14.158 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:26:14.158 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:26:14.158 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:26:14.158 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:26:14.159 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:14.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:14.159 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:26:14.159 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:26:14.160 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:26:14.160 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:26:14.160 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:26:14.160 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:26:14.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 -- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb08018930 msgr2=0x7fcb0801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb08018930 0x7fcb0801ae10 crc :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 -- 172.21.15.32:0/3679534398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2018c440 msgr2=0x7fcb2018c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2018c440 0x7fcb2018c840 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 rx=0x7fcb10006fc0 tx=0x7fcb10009f70).stop 2021-04-09T18:26:14.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 -- 172.21.15.32:0/3679534398 shutdown_connections 2021-04-09T18:26:14.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb08018930 0x7fcb0801ae10 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb2018c440 0x7fcb2018c840 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb201a80f0 0x7fcb201a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 --2- 172.21.15.32:0/3679534398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb201a9360 0x7fcb201918f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 -- 172.21.15.32:0/3679534398 >> 172.21.15.32:0/3679534398 conn(0x7fcb201837d0 msgr2=0x7fcb20195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 -- 172.21.15.32:0/3679534398 shutdown_connections 2021-04-09T18:26:14.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:13.875 7fcb28009700 1 -- 172.21.15.32:0/3679534398 wait complete. 2021-04-09T18:26:14.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.244 7f78f5aee700 1 Processor -- start 2021-04-09T18:26:14.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.244 7f78f5aee700 1 -- start start 2021-04-09T18:26:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78f5aee700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f78f00d7ca0 0x7f78f00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78f5aee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f010da00 0x7f78f010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78f5aee700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f010d480 con 0x7f78f010d010 2021-04-09T18:26:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78f5aee700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f00d9220 con 0x7f78f00d7ca0 2021-04-09T18:26:14.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78f5aee700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f00d86c0 con 0x7f78f010da00 2021-04-09T18:26:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78ef7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f78f010d010 0x7f78f00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34276/0 (socket says 172.21.15.32:34276) 2021-04-09T18:26:14.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78ef7fe700 1 -- 172.21.15.32:0/3269330426 learned_addr learned my addr 172.21.15.32:0/3269330426 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:14.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78effff700 1 --2- 172.21.15.32:0/3269330426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f010da00 0x7f78f010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78eeffd700 1 --2- 172.21.15.32:0/3269330426 >> v2:172.21.15.86:3300/0 conn(0x7f78f00d7ca0 0x7f78f00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78ee7fc700 1 -- 172.21.15.32:0/3269330426 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334190560 0 0) 0x7f78e0000ce0 con 0x7f78f010d010 2021-04-09T18:26:14.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78ee7fc700 1 -- 172.21.15.32:0/3269330426 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d8002fe0 con 0x7f78f010d010 2021-04-09T18:26:14.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78effff700 1 -- 172.21.15.32:0/3269330426 >> v2:172.21.15.86:3300/0 conn(0x7f78f00d7ca0 msgr2=0x7f78f00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:14.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78effff700 1 --2- 172.21.15.32:0/3269330426 >> v2:172.21.15.86:3300/0 conn(0x7f78f00d7ca0 0x7f78f00d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78effff700 1 -- 172.21.15.32:0/3269330426 >> v1:172.21.15.32:6789/0 conn(0x7f78f010d010 legacy=0x7f78f00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78effff700 1 -- 172.21.15.32:0/3269330426 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f00d1f70 con 0x7f78f010da00 2021-04-09T18:26:14.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78eeffd700 1 --2- 172.21.15.32:0/3269330426 >> v2:172.21.15.86:3300/0 conn(0x7f78f00d7ca0 0x7f78f00d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:14.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.245 7f78effff700 1 --2- 172.21.15.32:0/3269330426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f010da00 0x7f78f010de00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f78e8006e50 tx=0x7f78e8009e70).ready entity=mon.2 client_cookie=ab36dea891cf5305 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:14.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78ee7fc700 1 -- 172.21.15.32:0/3269330426 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78e8006a20 con 0x7f78f010da00 2021-04-09T18:26:14.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78ee7fc700 1 -- 172.21.15.32:0/3269330426 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e8006790 con 0x7f78f010da00 2021-04-09T18:26:14.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78ee7fc700 1 -- 172.21.15.32:0/3269330426 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78e800b700 con 0x7f78f010da00 2021-04-09T18:26:14.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78f5aee700 1 -- 172.21.15.32:0/3269330426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f010da00 msgr2=0x7f78f010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78f5aee700 1 --2- 172.21.15.32:0/3269330426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f010da00 0x7f78f010de00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f78e8006e50 tx=0x7f78e8009e70).stop 2021-04-09T18:26:14.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78f5aee700 1 -- 172.21.15.32:0/3269330426 shutdown_connections 2021-04-09T18:26:14.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78f5aee700 1 --2- 172.21.15.32:0/3269330426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f010da00 0x7f78f010de00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78f5aee700 1 --2- 172.21.15.32:0/3269330426 >> v2:172.21.15.86:3300/0 conn(0x7f78f00d7ca0 0x7f78f00d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78f5aee700 1 -- 172.21.15.32:0/3269330426 >> 172.21.15.32:0/3269330426 conn(0x7f78f00ce5f0 msgr2=0x7f78f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:14.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.246 7f78f5aee700 1 -- 172.21.15.32:0/3269330426 shutdown_connections 2021-04-09T18:26:14.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.247 7f78f5aee700 1 -- 172.21.15.32:0/3269330426 wait complete. 2021-04-09T18:26:14.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78f5aee700 1 Processor -- start 2021-04-09T18:26:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78f5aee700 1 -- start start 2021-04-09T18:26:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78f5aee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78f0179bf0 0x7f78f00abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78f5aee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f00ac400 0x7f78f01a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78f5aee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f017a110 0x7f78f01a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78f5aee700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f017a510 con 0x7f78f0179bf0 2021-04-09T18:26:14.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78f5aee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01a68b0 con 0x7f78f00ac400 2021-04-09T18:26:14.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78f5aee700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0182e30 con 0x7f78f017a110 2021-04-09T18:26:14.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78ef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78f0179bf0 0x7f78f00abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78ef7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78f0179bf0 0x7f78f00abde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52924/0 (socket says 172.21.15.32:52924) 2021-04-09T18:26:14.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78ef7fe700 1 -- 172.21.15.32:0/2397668471 learned_addr learned my addr 172.21.15.32:0/2397668471 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:14.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.267 7f78effff700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f017a110 0x7f78f01a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78eeffd700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f00ac400 0x7f78f01a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78ef7fe700 1 -- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f017a110 msgr2=0x7f78f01a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78ef7fe700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f017a110 0x7f78f01a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78ef7fe700 1 -- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f00ac400 msgr2=0x7f78f01a6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78ef7fe700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f00ac400 0x7f78f01a6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78ef7fe700 1 -- 172.21.15.32:0/2397668471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f01a56c0 con 0x7f78f0179bf0 2021-04-09T18:26:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78ef7fe700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78f0179bf0 0x7f78f00abde0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 rx=0x7f78e0006f00 tx=0x7f78e000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78d7fff700 1 -- 172.21.15.32:0/2397668471 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78e0006020 con 0x7f78f0179bf0 2021-04-09T18:26:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78d7fff700 1 -- 172.21.15.32:0/2397668471 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e0006360 con 0x7f78f0179bf0 2021-04-09T18:26:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78d7fff700 1 -- 172.21.15.32:0/2397668471 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78e0004620 con 0x7f78f0179bf0 2021-04-09T18:26:14.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f01a56c0 con 0x7f78f0179bf0 2021-04-09T18:26:14.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.268 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f01a56c0 con 0x7f78f0179bf0 2021-04-09T18:26:14.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.269 7f78d7fff700 1 -- 172.21.15.32:0/2397668471 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f78e000a000 con 0x7f78f0179bf0 2021-04-09T18:26:14.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.269 7f78d7fff700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78d0018930 0x7f78d001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.269 7f78d7fff700 1 -- 172.21.15.32:0/2397668471 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f78e00329e0 con 0x7f78f0179bf0 2021-04-09T18:26:14.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.269 7f78eeffd700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78d0018930 0x7f78d001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.269 7f78eeffd700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78d0018930 0x7f78d001adc0 crc :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.289 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f017a510 con 0x7f78f0179bf0 2021-04-09T18:26:14.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.292 7f78d7fff700 1 -- 172.21.15.32:0/2397668471 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f78e00009b0 con 0x7f78f0179bf0 2021-04-09T18:26:14.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.483 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78f017a510 con 0x7f78f0179bf0 2021-04-09T18:26:14.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.484 7f78d7fff700 1 -- 172.21.15.32:0/2397668471 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+2957 (secure 0 0 0) 0x7f78e000a000 con 0x7f78f0179bf0 2021-04-09T18:26:14.554 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 269 2021-04-09T18:26:14.555 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:14.555 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":269,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"laggy_since":"2021-04-09 18:26:07.577337","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"laggy_since":"2021-04-09 18:26:07.577350","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"laggy_since":"2021-04-09 18:26:07.577362","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269}],"filesystems":[{"mdsmap":{"epoch":269,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:26:07.577415","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9085":{"gid":9085,"name":"d","rank":0,"incarnation":265,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3538582472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3538582472},{"type":"v1","addr":"172.21.15.86:6836","nonce":3538582472}]},"laggy_since":"2021-04-09 18:26:07.577322","export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:26:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78d0018930 msgr2=0x7f78d001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78d0018930 0x7f78d001adc0 crc :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78f0179bf0 msgr2=0x7f78f00abde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78f0179bf0 0x7f78f00abde0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 rx=0x7f78e0006f00 tx=0x7f78e000b940).stop 2021-04-09T18:26:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 shutdown_connections 2021-04-09T18:26:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78d0018930 0x7f78d001adc0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f017a110 0x7f78f01a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f00ac400 0x7f78f01a6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 --2- 172.21.15.32:0/2397668471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78f0179bf0 0x7f78f00abde0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 >> 172.21.15.32:0/2397668471 conn(0x7f78f01acae0 msgr2=0x7f78f017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:14.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 shutdown_connections 2021-04-09T18:26:14.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.486 7f78f5aee700 1 -- 172.21.15.32:0/2397668471 wait complete. 2021-04-09T18:26:14.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:26:14.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.649 7fe7c358a700 1 Processor -- start 2021-04-09T18:26:14.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.649 7fe7c358a700 1 -- start start 2021-04-09T18:26:14.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c358a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7bc0d6c90 0x7fe7bc0d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c358a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7bc0d7290 con 0x7fe7bc0d2450 2021-04-09T18:26:14.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c358a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7bc0d2850 con 0x7fe7bc0cffb0 2021-04-09T18:26:14.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c358a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7bc0d7090 con 0x7fe7bc0d6c90 2021-04-09T18:26:14.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c0ba5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7bc0d6c90 0x7fe7bc0d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c1ba7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe7bc0d2450 0x7fe7bc10cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34290/0 (socket says 172.21.15.32:34290) 2021-04-09T18:26:14.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c1ba7700 1 -- 172.21.15.32:0/3505812316 learned_addr learned my addr 172.21.15.32:0/3505812316 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:14.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c0ba5700 1 -- 172.21.15.32:0/3505812316 >> v1:172.21.15.86:6789/0 conn(0x7fe7bc0cffb0 legacy=0x7fe7bc0d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c0ba5700 1 -- 172.21.15.32:0/3505812316 >> v1:172.21.15.32:6789/0 conn(0x7fe7bc0d2450 legacy=0x7fe7bc10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7b3fff700 1 -- 172.21.15.32:0/3505812316 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645755241 0 0) 0x7fe7b4000d30 con 0x7fe7bc0d2450 2021-04-09T18:26:14.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.650 7fe7c0ba5700 1 -- 172.21.15.32:0/3505812316 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc0d91f0 con 0x7fe7bc0d6c90 2021-04-09T18:26:14.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.651 7fe7c0ba5700 1 --2- 172.21.15.32:0/3505812316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0d6c90 0x7fe7bc0d8000 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7fe7ac006650 tx=0x7fe7ac009a00).ready entity=mon.0 client_cookie=aa478bd8d540f1ef server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:14.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.651 7fe7b3fff700 1 -- 172.21.15.32:0/3505812316 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7ac009df0 con 0x7fe7bc0d6c90 2021-04-09T18:26:14.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.651 7fe7b3fff700 1 -- 172.21.15.32:0/3505812316 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ac0009b0 con 0x7fe7bc0d6c90 2021-04-09T18:26:14.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.651 7fe7b3fff700 1 -- 172.21.15.32:0/3505812316 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7ac009df0 con 0x7fe7bc0d6c90 2021-04-09T18:26:14.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.651 7fe7c358a700 1 -- 172.21.15.32:0/3505812316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0d6c90 msgr2=0x7fe7bc0d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.651 7fe7c358a700 1 --2- 172.21.15.32:0/3505812316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0d6c90 0x7fe7bc0d8000 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7fe7ac006650 tx=0x7fe7ac009a00).stop 2021-04-09T18:26:14.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.652 7fe7c358a700 1 -- 172.21.15.32:0/3505812316 shutdown_connections 2021-04-09T18:26:14.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.652 7fe7c358a700 1 --2- 172.21.15.32:0/3505812316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0d6c90 0x7fe7bc0d8000 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.652 7fe7c358a700 1 -- 172.21.15.32:0/3505812316 >> 172.21.15.32:0/3505812316 conn(0x7fe7bc0ce5d0 msgr2=0x7fe7bc0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:14.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.652 7fe7c358a700 1 -- 172.21.15.32:0/3505812316 shutdown_connections 2021-04-09T18:26:14.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.652 7fe7c358a700 1 -- 172.21.15.32:0/3505812316 wait complete. 2021-04-09T18:26:14.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.672 7fe7c358a700 1 Processor -- start 2021-04-09T18:26:14.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.672 7fe7c358a700 1 -- start start 2021-04-09T18:26:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.672 7fe7c358a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7bc0bc440 0x7fe7bc0bbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.672 7fe7c358a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7bc0a7500 0x7fe7bc0aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.672 7fe7c358a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0a7900 0x7fe7bc0af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.672 7fe7c358a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc08d020 con 0x7fe7bc0a7900 2021-04-09T18:26:14.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.672 7fe7c358a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc1a7dd0 con 0x7fe7bc0bc440 2021-04-09T18:26:14.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.672 7fe7c358a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc1a5620 con 0x7fe7bc0a7500 2021-04-09T18:26:14.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c1ba7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0a7900 0x7fe7bc0af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c1ba7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0a7900 0x7fe7bc0af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52940/0 (socket says 172.21.15.32:52940) 2021-04-09T18:26:14.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c1ba7700 1 -- 172.21.15.32:0/3214376055 learned_addr learned my addr 172.21.15.32:0/3214376055 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:14.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c13a6700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7bc0bc440 0x7fe7bc0bbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c0ba5700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7bc0a7500 0x7fe7bc0aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c13a6700 1 -- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7bc0a7500 msgr2=0x7fe7bc0aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c13a6700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7bc0a7500 0x7fe7bc0aeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c13a6700 1 -- 172.21.15.32:0/3214376055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0a7900 msgr2=0x7fe7bc0af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:14.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c13a6700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0a7900 0x7fe7bc0af120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:14.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c13a6700 1 -- 172.21.15.32:0/3214376055 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc1a5840 con 0x7fe7bc0bc440 2021-04-09T18:26:14.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c1ba7700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0a7900 0x7fe7bc0af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:14.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c13a6700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7bc0bc440 0x7fe7bc0bbd10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7fe7b8006e90 tx=0x7fe7b800b860).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:14.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7b1ffb700 1 -- 172.21.15.32:0/3214376055 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7b8000960 con 0x7fe7bc0bc440 2021-04-09T18:26:14.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7b1ffb700 1 -- 172.21.15.32:0/3214376055 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b800bc50 con 0x7fe7bc0bc440 2021-04-09T18:26:14.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7bc1a5840 con 0x7fe7bc0bc440 2021-04-09T18:26:14.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7b1ffb700 1 -- 172.21.15.32:0/3214376055 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7b80022b0 con 0x7fe7bc0bc440 2021-04-09T18:26:14.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.673 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7bc1a5840 con 0x7fe7bc0bc440 2021-04-09T18:26:14.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.674 7fe7b1ffb700 1 -- 172.21.15.32:0/3214376055 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe7b80052e0 con 0x7fe7bc0bc440 2021-04-09T18:26:14.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.674 7fe7b1ffb700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe79c018930 0x7fe79c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:14.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.674 7fe7b1ffb700 1 -- 172.21.15.32:0/3214376055 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe7b8032940 con 0x7fe7bc0bc440 2021-04-09T18:26:14.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.674 7fe7c0ba5700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe79c018930 0x7fe79c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:14.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.675 7fe7c0ba5700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe79c018930 0x7fe79c01adc0 crc :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:14.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.694 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7bc112040 con 0x7fe7bc0bc440 2021-04-09T18:26:14.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.698 7fe7b1ffb700 1 -- 172.21.15.32:0/3214376055 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe7b8000ca0 con 0x7fe7bc0bc440 2021-04-09T18:26:14.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.877 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe7bc112040 con 0x7fe7bc0bc440 2021-04-09T18:26:14.993 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:26:14.923 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:26:14.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.929 7fe7b1ffb700 1 -- 172.21.15.32:0/3214376055 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v270) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe7b80052e0 con 0x7fe7bc0bc440 2021-04-09T18:26:15.010 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:26:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe79c018930 msgr2=0x7fe79c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe79c018930 0x7fe79c01adc0 crc :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7bc0bc440 msgr2=0x7fe7bc0bbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7bc0bc440 0x7fe7bc0bbd10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7fe7b8006e90 tx=0x7fe7b800b860).stop 2021-04-09T18:26:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 shutdown_connections 2021-04-09T18:26:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe79c018930 0x7fe79c01adc0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7bc0a7900 0x7fe7bc0af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7bc0a7500 0x7fe7bc0aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 --2- 172.21.15.32:0/3214376055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7bc0bc440 0x7fe7bc0bbd10 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 >> 172.21.15.32:0/3214376055 conn(0x7fe7bc1a80a0 msgr2=0x7fe7bc1ab840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:15.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 shutdown_connections 2021-04-09T18:26:15.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:14.942 7fe7c358a700 1 -- 172.21.15.32:0/3214376055 wait complete. 2021-04-09T18:26:15.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:15.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.102 7fe88d19a700 1 Processor -- start 2021-04-09T18:26:15.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.102 7fe88d19a700 1 -- start start 2021-04-09T18:26:15.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.102 7fe88d19a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe88810d010 0x7fe8880d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.102 7fe88d19a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880d1da0 0x7fe8880d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.102 7fe88d19a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88810d480 con 0x7fe8880d82c0 2021-04-09T18:26:15.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.102 7fe88d19a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8880d9220 con 0x7fe88810d010 2021-04-09T18:26:15.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.102 7fe88d19a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8880d86c0 con 0x7fe8880d1da0 2021-04-09T18:26:15.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe886d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe88810d010 0x7fe8880d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe886d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe88810d010 0x7fe8880d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52944/0 (socket says 172.21.15.32:52944) 2021-04-09T18:26:15.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe886d9d700 1 -- 172.21.15.32:0/2092310569 learned_addr learned my addr 172.21.15.32:0/2092310569 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:15.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe88659c700 1 --2- 172.21.15.32:0/2092310569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880d1da0 0x7fe8880d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe886d9d700 1 -- 172.21.15.32:0/2092310569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880d1da0 msgr2=0x7fe8880d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:15.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe886d9d700 1 --2- 172.21.15.32:0/2092310569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880d1da0 0x7fe8880d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe886d9d700 1 -- 172.21.15.32:0/2092310569 >> v1:172.21.15.86:6789/0 conn(0x7fe8880d82c0 legacy=0x7fe88810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe885d9b700 1 -- 172.21.15.32:0/2092310569 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113435469 0 0) 0x7fe8800009b0 con 0x7fe8880d82c0 2021-04-09T18:26:15.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe886d9d700 1 -- 172.21.15.32:0/2092310569 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88810e1f0 con 0x7fe88810d010 2021-04-09T18:26:15.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe886d9d700 1 --2- 172.21.15.32:0/2092310569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe88810d010 0x7fe8880d1780 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7fe878006700 tx=0x7fe878009b80).ready entity=mon.0 client_cookie=f494fa06ef05a80a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:15.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe885d9b700 1 -- 172.21.15.32:0/2092310569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe87800c0c0 con 0x7fe88810d010 2021-04-09T18:26:15.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe885d9b700 1 -- 172.21.15.32:0/2092310569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87800c440 con 0x7fe88810d010 2021-04-09T18:26:15.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe885d9b700 1 -- 172.21.15.32:0/2092310569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe87800c920 con 0x7fe88810d010 2021-04-09T18:26:15.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe88d19a700 1 -- 172.21.15.32:0/2092310569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe88810d010 msgr2=0x7fe8880d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.103 7fe88d19a700 1 --2- 172.21.15.32:0/2092310569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe88810d010 0x7fe8880d1780 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 rx=0x7fe878006700 tx=0x7fe878009b80).stop 2021-04-09T18:26:15.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.104 7fe88d19a700 1 -- 172.21.15.32:0/2092310569 shutdown_connections 2021-04-09T18:26:15.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.104 7fe88d19a700 1 --2- 172.21.15.32:0/2092310569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880d1da0 0x7fe8880d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.104 7fe88d19a700 1 --2- 172.21.15.32:0/2092310569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe88810d010 0x7fe8880d1780 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.104 7fe88d19a700 1 -- 172.21.15.32:0/2092310569 >> 172.21.15.32:0/2092310569 conn(0x7fe8880ce5f0 msgr2=0x7fe8880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:15.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.104 7fe88d19a700 1 -- 172.21.15.32:0/2092310569 shutdown_connections 2021-04-09T18:26:15.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.104 7fe88d19a700 1 -- 172.21.15.32:0/2092310569 wait complete. 2021-04-09T18:26:15.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.125 7fe88d19a700 1 Processor -- start 2021-04-09T18:26:15.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.125 7fe88d19a700 1 -- start start 2021-04-09T18:26:15.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88d19a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe888179bf0 0x7fe8880abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88d19a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880ac400 0x7fe8881a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88d19a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe88817a110 0x7fe8881a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88d19a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88817a510 con 0x7fe888179bf0 2021-04-09T18:26:15.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88d19a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8881a68b0 con 0x7fe88817a110 2021-04-09T18:26:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88d19a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe888182e30 con 0x7fe8880ac400 2021-04-09T18:26:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe886d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe888179bf0 0x7fe8880abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe886d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe888179bf0 0x7fe8880abde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52950/0 (socket says 172.21.15.32:52950) 2021-04-09T18:26:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe886d9d700 1 -- 172.21.15.32:0/3130624103 learned_addr learned my addr 172.21.15.32:0/3130624103 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88659c700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880ac400 0x7fe8881a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88759e700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe88817a110 0x7fe8881a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe886d9d700 1 -- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880ac400 msgr2=0x7fe8881a6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe886d9d700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880ac400 0x7fe8881a6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe886d9d700 1 -- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe88817a110 msgr2=0x7fe8881a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe886d9d700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe88817a110 0x7fe8881a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe886d9d700 1 -- 172.21.15.32:0/3130624103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8881a56c0 con 0x7fe888179bf0 2021-04-09T18:26:15.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe886d9d700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe888179bf0 0x7fe8880abde0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 rx=0x7fe87800b650 tx=0x7fe87800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:15.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe8777fe700 1 -- 172.21.15.32:0/3130624103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe87800bde0 con 0x7fe888179bf0 2021-04-09T18:26:15.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe8777fe700 1 -- 172.21.15.32:0/3130624103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe878000c90 con 0x7fe888179bf0 2021-04-09T18:26:15.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8881a56c0 con 0x7fe888179bf0 2021-04-09T18:26:15.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe8777fe700 1 -- 172.21.15.32:0/3130624103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe878002220 con 0x7fe888179bf0 2021-04-09T18:26:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.126 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8881a56c0 con 0x7fe888179bf0 2021-04-09T18:26:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.127 7fe8777fe700 1 -- 172.21.15.32:0/3130624103 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe878002020 con 0x7fe888179bf0 2021-04-09T18:26:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.127 7fe8777fe700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe86c018930 0x7fe86c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.127 7fe8777fe700 1 -- 172.21.15.32:0/3130624103 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fe878032a40 con 0x7fe888179bf0 2021-04-09T18:26:15.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.127 7fe88659c700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe86c018930 0x7fe86c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.128 7fe88659c700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe86c018930 0x7fe86c01ae10 crc :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:15.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.147 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88817a510 con 0x7fe888179bf0 2021-04-09T18:26:15.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.150 7fe8777fe700 1 -- 172.21.15.32:0/3130624103 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe8780091e0 con 0x7fe888179bf0 2021-04-09T18:26:15.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.342 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe88817a510 con 0x7fe888179bf0 2021-04-09T18:26:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.343 7fe8777fe700 1 -- 172.21.15.32:0/3130624103 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 270 v270) v1 ==== 95+0+2569 (secure 0 0 0) 0x7fe87800bde0 con 0x7fe888179bf0 2021-04-09T18:26:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 270 2021-04-09T18:26:15.414 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:15.414 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":270,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"laggy_since":"2021-04-09 18:26:07.577337","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"laggy_since":"2021-04-09 18:26:07.577350","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"laggy_since":"2021-04-09 18:26:07.577362","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269}],"filesystems":[{"mdsmap":{"epoch":270,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:25:33.288479","modified":"2021-04-09 18:26:14.924547","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":132,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-04-09T18:26:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe86c018930 msgr2=0x7fe86c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe86c018930 0x7fe86c01ae10 crc :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe888179bf0 msgr2=0x7fe8880abde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe888179bf0 0x7fe8880abde0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 rx=0x7fe87800b650 tx=0x7fe87800b9f0).stop 2021-04-09T18:26:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 shutdown_connections 2021-04-09T18:26:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe86c018930 0x7fe86c01ae10 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe88817a110 0x7fe8881a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8880ac400 0x7fe8881a6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 --2- 172.21.15.32:0/3130624103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe888179bf0 0x7fe8880abde0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 >> 172.21.15.32:0/3130624103 conn(0x7fe8881acae0 msgr2=0x7fe88817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 shutdown_connections 2021-04-09T18:26:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.345 7fe88d19a700 1 -- 172.21.15.32:0/3130624103 wait complete. 2021-04-09T18:26:15.426 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:26:15.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.510 7fd7ce88b700 1 Processor -- start 2021-04-09T18:26:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.510 7fd7ce88b700 1 -- start start 2021-04-09T18:26:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.510 7fd7ce88b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd7c80d6c80 0x7fd7c80d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.510 7fd7ce88b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7c80d2440 0x7fd7c810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.510 7fd7ce88b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c80d7280 con 0x7fd7c80cffa0 2021-04-09T18:26:15.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.510 7fd7ce88b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c80d7080 con 0x7fd7c80d6c80 2021-04-09T18:26:15.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.510 7fd7ce88b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c80d91e0 con 0x7fd7c80d2440 2021-04-09T18:26:15.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7bffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd7c80d6c80 0x7fd7c80d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7ccea8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7c80d2440 0x7fd7c810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7c7fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd7c80cffa0 0x7fd7c80d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:60474/0 (socket says 172.21.15.32:60474) 2021-04-09T18:26:15.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7c7fff700 1 -- 172.21.15.32:0/3054526831 learned_addr learned my addr 172.21.15.32:0/3054526831 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:15.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7bffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd7c80d6c80 0x7fd7c80d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52960/0 (socket says 172.21.15.32:52960) 2021-04-09T18:26:15.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7ccea8700 1 -- 172.21.15.32:0/3054526831 >> v2:172.21.15.32:3300/0 conn(0x7fd7c80d6c80 msgr2=0x7fd7c80d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:15.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7ccea8700 1 --2- 172.21.15.32:0/3054526831 >> v2:172.21.15.32:3300/0 conn(0x7fd7c80d6c80 0x7fd7c80d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7ccea8700 1 -- 172.21.15.32:0/3054526831 >> v1:172.21.15.86:6789/0 conn(0x7fd7c80cffa0 legacy=0x7fd7c80d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7ccea8700 1 -- 172.21.15.32:0/3054526831 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c80d2840 con 0x7fd7c80d2440 2021-04-09T18:26:15.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7c77fe700 1 -- 172.21.15.32:0/3054526831 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128592336 0 0) 0x7fd7b80010c0 con 0x7fd7c80cffa0 2021-04-09T18:26:15.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7bffff700 1 --2- 172.21.15.32:0/3054526831 >> v2:172.21.15.32:3300/0 conn(0x7fd7c80d6c80 0x7fd7c80d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:15.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7ccea8700 1 --2- 172.21.15.32:0/3054526831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c80d2440 0x7fd7c810cfd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7fd7c0006e50 tx=0x7fd7c0009e70).ready entity=mon.1 client_cookie=3b8492640b3c3d41 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:15.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7c77fe700 1 -- 172.21.15.32:0/3054526831 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7c0006a20 con 0x7fd7c80d2440 2021-04-09T18:26:15.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7c77fe700 1 -- 172.21.15.32:0/3054526831 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c0006790 con 0x7fd7c80d2440 2021-04-09T18:26:15.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7c77fe700 1 -- 172.21.15.32:0/3054526831 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7c000b700 con 0x7fd7c80d2440 2021-04-09T18:26:15.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7ce88b700 1 -- 172.21.15.32:0/3054526831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c80d2440 msgr2=0x7fd7c810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.511 7fd7ce88b700 1 --2- 172.21.15.32:0/3054526831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c80d2440 0x7fd7c810cfd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7fd7c0006e50 tx=0x7fd7c0009e70).stop 2021-04-09T18:26:15.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.512 7fd7ce88b700 1 -- 172.21.15.32:0/3054526831 shutdown_connections 2021-04-09T18:26:15.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.512 7fd7ce88b700 1 --2- 172.21.15.32:0/3054526831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c80d2440 0x7fd7c810cfd0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.512 7fd7ce88b700 1 --2- 172.21.15.32:0/3054526831 >> v2:172.21.15.32:3300/0 conn(0x7fd7c80d6c80 0x7fd7c80d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.512 7fd7ce88b700 1 -- 172.21.15.32:0/3054526831 >> 172.21.15.32:0/3054526831 conn(0x7fd7c80ce5c0 msgr2=0x7fd7c80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:15.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.512 7fd7ce88b700 1 -- 172.21.15.32:0/3054526831 shutdown_connections 2021-04-09T18:26:15.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.512 7fd7ce88b700 1 -- 172.21.15.32:0/3054526831 wait complete. 2021-04-09T18:26:15.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.532 7fd7ce88b700 1 Processor -- start 2021-04-09T18:26:15.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.532 7fd7ce88b700 1 -- start start 2021-04-09T18:26:15.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7ce88b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c81a7db0 0x7fd7c81a81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7ce88b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c8184070 0x7fd7c8184470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7ce88b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c8184a90 0x7fd7c808f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7ce88b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c81a8b30 con 0x7fd7c8184070 2021-04-09T18:26:15.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7ce88b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c81a7860 con 0x7fd7c8184a90 2021-04-09T18:26:15.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7ce88b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c81a87d0 con 0x7fd7c81a7db0 2021-04-09T18:26:15.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c8184070 0x7fd7c8184470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c8184070 0x7fd7c8184470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52964/0 (socket says 172.21.15.32:52964) 2021-04-09T18:26:15.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7bffff700 1 -- 172.21.15.32:0/2103396379 learned_addr learned my addr 172.21.15.32:0/2103396379 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:15.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7ccea8700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c8184a90 0x7fd7c808f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7c7fff700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c81a7db0 0x7fd7c81a81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7bffff700 1 -- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c81a7db0 msgr2=0x7fd7c81a81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7bffff700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c81a7db0 0x7fd7c81a81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7bffff700 1 -- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c8184a90 msgr2=0x7fd7c808f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:15.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7bffff700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c8184a90 0x7fd7c808f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:15.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7bffff700 1 -- 172.21.15.32:0/2103396379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c808f810 con 0x7fd7c8184070 2021-04-09T18:26:15.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.533 7fd7bffff700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c8184070 0x7fd7c8184470 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7fd7b000b580 tx=0x7fd7b000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:15.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.534 7fd7c57fa700 1 -- 172.21.15.32:0/2103396379 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7b000bd10 con 0x7fd7c8184070 2021-04-09T18:26:15.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.534 7fd7c57fa700 1 -- 172.21.15.32:0/2103396379 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b0000c90 con 0x7fd7c8184070 2021-04-09T18:26:15.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.534 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c808f810 con 0x7fd7c8184070 2021-04-09T18:26:15.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.534 7fd7c57fa700 1 -- 172.21.15.32:0/2103396379 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7b0002220 con 0x7fd7c8184070 2021-04-09T18:26:15.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.534 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c808f810 con 0x7fd7c8184070 2021-04-09T18:26:15.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.534 7fd7c57fa700 1 -- 172.21.15.32:0/2103396379 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd7b000a000 con 0x7fd7c8184070 2021-04-09T18:26:15.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.534 7fd7c57fa700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7ac0188c0 0x7fd7ac01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:15.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.534 7fd7c57fa700 1 -- 172.21.15.32:0/2103396379 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fd7b0032a30 con 0x7fd7c8184070 2021-04-09T18:26:15.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.535 7fd7c7fff700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7ac0188c0 0x7fd7ac01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:15.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.535 7fd7c7fff700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7ac0188c0 0x7fd7ac01adc0 crc :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:15.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.554 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c8182e70 con 0x7fd7c8184070 2021-04-09T18:26:15.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.558 7fd7c57fa700 1 -- 172.21.15.32:0/2103396379 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd7b0000c90 con 0x7fd7c8184070 2021-04-09T18:26:15.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.752 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd7c80d7280 con 0x7fd7c8184070 2021-04-09T18:26:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.931 7fd7c57fa700 1 -- 172.21.15.32:0/2103396379 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v271) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd7b000a000 con 0x7fd7c8184070 2021-04-09T18:26:16.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7ac0188c0 msgr2=0x7fd7ac01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:16.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7ac0188c0 0x7fd7ac01adc0 crc :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c8184070 msgr2=0x7fd7c8184470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:16.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c8184070 0x7fd7c8184470 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7fd7b000b580 tx=0x7fd7b000b920).stop 2021-04-09T18:26:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 shutdown_connections 2021-04-09T18:26:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7ac0188c0 0x7fd7ac01adc0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c8184070 0x7fd7c8184470 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c8184a90 0x7fd7c808f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 --2- 172.21.15.32:0/2103396379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c81a7db0 0x7fd7c81a81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 >> 172.21.15.32:0/2103396379 conn(0x7fd7c81a8f40 msgr2=0x7fd7c8188050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 shutdown_connections 2021-04-09T18:26:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:15.968 7fd7ce88b700 1 -- 172.21.15.32:0/2103396379 wait complete. 2021-04-09T18:26:16.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:26:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.139 7f3e5ef9c700 1 Processor -- start 2021-04-09T18:26:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.139 7f3e5ef9c700 1 -- start start 2021-04-09T18:26:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e5ef9c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3e5810d050 0x7f3e580cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e5ef9c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580d0160 0x7f3e580d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e5ef9c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5810d500 con 0x7f3e580d82e0 2021-04-09T18:26:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e5ef9c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e580d9260 con 0x7f3e5810d050 2021-04-09T18:26:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e5ef9c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580d86e0 con 0x7f3e580d0160 2021-04-09T18:26:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e5d5b9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3e580d82e0 0x7f3e5810da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34330/0 (socket says 172.21.15.32:34330) 2021-04-09T18:26:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e5d5b9700 1 -- 172.21.15.32:0/2936047427 learned_addr learned my addr 172.21.15.32:0/2936047427 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e57fff700 1 --2- 172.21.15.32:0/2936047427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580d0160 0x7f3e580d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e5cdb8700 1 --2- 172.21.15.32:0/2936047427 >> v2:172.21.15.86:3300/0 conn(0x7f3e5810d050 0x7f3e580cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e577fe700 1 -- 172.21.15.32:0/2936047427 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866115356 0 0) 0x7f3e500009b0 con 0x7f3e580d82e0 2021-04-09T18:26:16.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.140 7f3e577fe700 1 -- 172.21.15.32:0/2936047427 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e44002fe0 con 0x7f3e580d82e0 2021-04-09T18:26:16.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e57fff700 1 -- 172.21.15.32:0/2936047427 >> v2:172.21.15.86:3300/0 conn(0x7f3e5810d050 msgr2=0x7f3e580cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e57fff700 1 --2- 172.21.15.32:0/2936047427 >> v2:172.21.15.86:3300/0 conn(0x7f3e5810d050 0x7f3e580cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e57fff700 1 -- 172.21.15.32:0/2936047427 >> v1:172.21.15.32:6789/0 conn(0x7f3e580d82e0 legacy=0x7f3e5810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e57fff700 1 -- 172.21.15.32:0/2936047427 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5810e270 con 0x7f3e580d0160 2021-04-09T18:26:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e5cdb8700 1 --2- 172.21.15.32:0/2936047427 >> v2:172.21.15.86:3300/0 conn(0x7f3e5810d050 0x7f3e580cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e57fff700 1 --2- 172.21.15.32:0/2936047427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580d0160 0x7f3e580d7cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3e480069d0 tx=0x7f3e48009e40).ready entity=mon.2 client_cookie=cb0176f66e4fe77a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e577fe700 1 -- 172.21.15.32:0/2936047427 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4800b510 con 0x7f3e580d0160 2021-04-09T18:26:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e577fe700 1 -- 172.21.15.32:0/2936047427 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4800b890 con 0x7f3e580d0160 2021-04-09T18:26:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e577fe700 1 -- 172.21.15.32:0/2936047427 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4800bd70 con 0x7f3e580d0160 2021-04-09T18:26:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e5ef9c700 1 -- 172.21.15.32:0/2936047427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580d0160 msgr2=0x7f3e580d7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.141 7f3e5ef9c700 1 --2- 172.21.15.32:0/2936047427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580d0160 0x7f3e580d7cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f3e480069d0 tx=0x7f3e48009e40).stop 2021-04-09T18:26:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.142 7f3e5ef9c700 1 -- 172.21.15.32:0/2936047427 shutdown_connections 2021-04-09T18:26:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.142 7f3e5ef9c700 1 --2- 172.21.15.32:0/2936047427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580d0160 0x7f3e580d7cc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.142 7f3e5ef9c700 1 --2- 172.21.15.32:0/2936047427 >> v2:172.21.15.86:3300/0 conn(0x7f3e5810d050 0x7f3e580cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.142 7f3e5ef9c700 1 -- 172.21.15.32:0/2936047427 >> 172.21.15.32:0/2936047427 conn(0x7f3e580ce620 msgr2=0x7f3e580d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.142 7f3e5ef9c700 1 -- 172.21.15.32:0/2936047427 shutdown_connections 2021-04-09T18:26:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.142 7f3e5ef9c700 1 -- 172.21.15.32:0/2936047427 wait complete. 2021-04-09T18:26:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.162 7f3e5ef9c700 1 Processor -- start 2021-04-09T18:26:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.162 7f3e5ef9c700 1 -- start start 2021-04-09T18:26:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e5ef9c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e580b0920 0x7f3e580b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e5ef9c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e580a7690 0x7f3e580a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e5ef9c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580c2510 0x7f3e580c2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e5ef9c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580b1340 con 0x7f3e580a7690 2021-04-09T18:26:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e5ef9c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580a82e0 con 0x7f3e580b0920 2021-04-09T18:26:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e5ef9c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580a8820 con 0x7f3e580c2510 2021-04-09T18:26:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e57fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e580a7690 0x7f3e580a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e57fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e580a7690 0x7f3e580a7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52978/0 (socket says 172.21.15.32:52978) 2021-04-09T18:26:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e57fff700 1 -- 172.21.15.32:0/2176240017 learned_addr learned my addr 172.21.15.32:0/2176240017 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e5d5b9700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580c2510 0x7f3e580c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e5cdb8700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e580b0920 0x7f3e580b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e57fff700 1 -- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580c2510 msgr2=0x7f3e580c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e57fff700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580c2510 0x7f3e580c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e57fff700 1 -- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e580b0920 msgr2=0x7f3e580b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:16.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e57fff700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e580b0920 0x7f3e580b0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:16.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e57fff700 1 -- 172.21.15.32:0/2176240017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5808fba0 con 0x7f3e580a7690 2021-04-09T18:26:16.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e57fff700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e580a7690 0x7f3e580a7a90 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7f3e4800c8e0 tx=0x7f3e4800cc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e557fa700 1 -- 172.21.15.32:0/2176240017 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4800cdc0 con 0x7f3e580a7690 2021-04-09T18:26:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e557fa700 1 -- 172.21.15.32:0/2176240017 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48006280 con 0x7f3e580a7690 2021-04-09T18:26:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.163 7f3e557fa700 1 -- 172.21.15.32:0/2176240017 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e48005660 con 0x7f3e580a7690 2021-04-09T18:26:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.164 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5808fba0 con 0x7f3e580a7690 2021-04-09T18:26:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.164 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5808fba0 con 0x7f3e580a7690 2021-04-09T18:26:16.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.164 7f3e557fa700 1 -- 172.21.15.32:0/2176240017 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3e48006460 con 0x7f3e580a7690 2021-04-09T18:26:16.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.164 7f3e557fa700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e3c018930 0x7f3e3c01adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:16.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.164 7f3e557fa700 1 -- 172.21.15.32:0/2176240017 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f3e48032df0 con 0x7f3e580a7690 2021-04-09T18:26:16.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.164 7f3e5cdb8700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e3c018930 0x7f3e3c01adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.165 7f3e5cdb8700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e3c018930 0x7f3e3c01adf0 crc :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.184 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e581b7180 con 0x7f3e580a7690 2021-04-09T18:26:16.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.187 7f3e557fa700 1 -- 172.21.15.32:0/2176240017 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3e48006490 con 0x7f3e580a7690 2021-04-09T18:26:16.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.369 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3e5810d500 con 0x7f3e580a7690 2021-04-09T18:26:17.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.936 7f3e557fa700 1 -- 172.21.15.32:0/2176240017 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v133) v1 ==== 185+0+0 (secure 0 0 0) 0x7f3e4801fae0 con 0x7f3e580a7690 2021-04-09T18:26:17.054 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:26:17.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e3c018930 msgr2=0x7f3e3c01adf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:17.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e3c018930 0x7f3e3c01adf0 crc :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e580a7690 msgr2=0x7f3e580a7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:17.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e580a7690 0x7f3e580a7a90 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7f3e4800c8e0 tx=0x7f3e4800cc80).stop 2021-04-09T18:26:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 shutdown_connections 2021-04-09T18:26:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e3c018930 0x7f3e3c01adf0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e580a7690 0x7f3e580a7a90 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e580c2510 0x7f3e580c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 --2- 172.21.15.32:0/2176240017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e580b0920 0x7f3e580b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 >> 172.21.15.32:0/2176240017 conn(0x7f3e580b6b70 msgr2=0x7f3e580b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 shutdown_connections 2021-04-09T18:26:17.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:16.986 7f3e5ef9c700 1 -- 172.21.15.32:0/2176240017 wait complete. 2021-04-09T18:26:17.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:26:17.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.158 7fee27ff4700 1 Processor -- start 2021-04-09T18:26:17.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.158 7fee27ff4700 1 -- start start 2021-04-09T18:26:17.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.159 7fee27ff4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee200d7cc0 0x7fee200d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:17.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.159 7fee27ff4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee2010da80 0x7fee2010de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:17.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.159 7fee27ff4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2010d500 con 0x7fee2010d050 2021-04-09T18:26:17.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.159 7fee27ff4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee200d9260 con 0x7fee2010da80 2021-04-09T18:26:17.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.159 7fee27ff4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee20112140 con 0x7fee200d7cc0 2021-04-09T18:26:17.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.159 7fee26611700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee2010da80 0x7fee2010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:17.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.159 7fee2560f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee200d7cc0 0x7fee200d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:17.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.159 7fee2560f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee200d7cc0 0x7fee200d80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:60212/0 (socket says 172.21.15.32:60212) 2021-04-09T18:26:17.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.159 7fee2560f700 1 -- 172.21.15.32:0/3955723815 learned_addr learned my addr 172.21.15.32:0/3955723815 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:17.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee2560f700 1 -- 172.21.15.32:0/3955723815 >> v2:172.21.15.86:3300/0 conn(0x7fee2010da80 msgr2=0x7fee2010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:17.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee2560f700 1 --2- 172.21.15.32:0/3955723815 >> v2:172.21.15.86:3300/0 conn(0x7fee2010da80 0x7fee2010de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee2560f700 1 -- 172.21.15.32:0/3955723815 >> v1:172.21.15.86:6789/0 conn(0x7fee2010d050 legacy=0x7fee200cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:17.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee2560f700 1 -- 172.21.15.32:0/3955723815 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200d0330 con 0x7fee200d7cc0 2021-04-09T18:26:17.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee26611700 1 --2- 172.21.15.32:0/3955723815 >> v2:172.21.15.86:3300/0 conn(0x7fee2010da80 0x7fee2010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:17.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee24e0e700 1 -- 172.21.15.32:0/3955723815 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710373735 0 0) 0x7fee1c000c60 con 0x7fee2010d050 2021-04-09T18:26:17.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee2560f700 1 --2- 172.21.15.32:0/3955723815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee200d7cc0 0x7fee200d80c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fee10006700 tx=0x7fee10009b80).ready entity=mon.2 client_cookie=2c0bc93b43620c88 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee24e0e700 1 -- 172.21.15.32:0/3955723815 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee1000c0c0 con 0x7fee200d7cc0 2021-04-09T18:26:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee24e0e700 1 -- 172.21.15.32:0/3955723815 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1000c440 con 0x7fee200d7cc0 2021-04-09T18:26:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee24e0e700 1 -- 172.21.15.32:0/3955723815 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee1000c920 con 0x7fee200d7cc0 2021-04-09T18:26:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee27ff4700 1 -- 172.21.15.32:0/3955723815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee200d7cc0 msgr2=0x7fee200d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.160 7fee27ff4700 1 --2- 172.21.15.32:0/3955723815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee200d7cc0 0x7fee200d80c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fee10006700 tx=0x7fee10009b80).stop 2021-04-09T18:26:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.161 7fee27ff4700 1 -- 172.21.15.32:0/3955723815 shutdown_connections 2021-04-09T18:26:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.161 7fee27ff4700 1 --2- 172.21.15.32:0/3955723815 >> v2:172.21.15.86:3300/0 conn(0x7fee2010da80 0x7fee2010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.161 7fee27ff4700 1 --2- 172.21.15.32:0/3955723815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee200d7cc0 0x7fee200d80c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.161 7fee27ff4700 1 -- 172.21.15.32:0/3955723815 >> 172.21.15.32:0/3955723815 conn(0x7fee200ce620 msgr2=0x7fee200d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.161 7fee27ff4700 1 -- 172.21.15.32:0/3955723815 shutdown_connections 2021-04-09T18:26:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.161 7fee27ff4700 1 -- 172.21.15.32:0/3955723815 wait complete. 2021-04-09T18:26:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.181 7fee27ff4700 1 Processor -- start 2021-04-09T18:26:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.181 7fee27ff4700 1 -- start start 2021-04-09T18:26:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee27ff4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee200a7930 0x7fee200c26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee27ff4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee20094560 0x7fee20094960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee27ff4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee20094f80 0x7fee20095940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee27ff4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee200a7db0 con 0x7fee200a7930 2021-04-09T18:26:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee27ff4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee200953c0 con 0x7fee20094560 2021-04-09T18:26:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee27ff4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee200c2cd0 con 0x7fee20094f80 2021-04-09T18:26:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee25e10700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee200a7930 0x7fee200c26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee25e10700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee200a7930 0x7fee200c26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52990/0 (socket says 172.21.15.32:52990) 2021-04-09T18:26:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee25e10700 1 -- 172.21.15.32:0/3608582451 learned_addr learned my addr 172.21.15.32:0/3608582451 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee26611700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee20094f80 0x7fee20095940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee2560f700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee20094560 0x7fee20094960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee25e10700 1 -- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee20094f80 msgr2=0x7fee20095940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee25e10700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee20094f80 0x7fee20095940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee25e10700 1 -- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee20094560 msgr2=0x7fee20094960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee25e10700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee20094560 0x7fee20094960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee25e10700 1 -- 172.21.15.32:0/3608582451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20190530 con 0x7fee200a7930 2021-04-09T18:26:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee2560f700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee20094560 0x7fee20094960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:26:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee25e10700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee200a7930 0x7fee200c26b0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 rx=0x7fee1c006f20 tx=0x7fee1c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.182 7fee167fc700 1 -- 172.21.15.32:0/3608582451 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee1c000e20 con 0x7fee200a7930 2021-04-09T18:26:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.183 7fee167fc700 1 -- 172.21.15.32:0/3608582451 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c00bd80 con 0x7fee200a7930 2021-04-09T18:26:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.183 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee20190530 con 0x7fee200a7930 2021-04-09T18:26:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.183 7fee167fc700 1 -- 172.21.15.32:0/3608582451 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee1c001270 con 0x7fee200a7930 2021-04-09T18:26:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.183 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee20190530 con 0x7fee200a7930 2021-04-09T18:26:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.183 7fee167fc700 1 -- 172.21.15.32:0/3608582451 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fee1c001080 con 0x7fee200a7930 2021-04-09T18:26:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.183 7fee167fc700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee08018930 0x7fee0801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.184 7fee167fc700 1 -- 172.21.15.32:0/3608582451 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5225+0+0 (secure 0 0 0) 0x7fee1c0328a0 con 0x7fee200a7930 2021-04-09T18:26:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.184 7fee2560f700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee08018930 0x7fee0801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.184 7fee2560f700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee08018930 0x7fee0801ae10 crc :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.203 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee200965c0 con 0x7fee200a7930 2021-04-09T18:26:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.207 7fee167fc700 1 -- 172.21.15.32:0/3608582451 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fee1c006bf0 con 0x7fee200a7930 2021-04-09T18:26:17.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.388 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fee200965c0 con 0x7fee200a7930 2021-04-09T18:26:18.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.944 7fee167fc700 1 -- 172.21.15.32:0/3608582451 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v134) v1 ==== 173+0+0 (secure 0 0 0) 0x7fee1c001080 con 0x7fee200a7930 2021-04-09T18:26:18.023 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:26:18.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee08018930 msgr2=0x7fee0801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee08018930 0x7fee0801ae10 crc :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee200a7930 msgr2=0x7fee200c26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee200a7930 0x7fee200c26b0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 rx=0x7fee1c006f20 tx=0x7fee1c00b990).stop 2021-04-09T18:26:18.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 shutdown_connections 2021-04-09T18:26:18.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee08018930 0x7fee0801ae10 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee20094f80 0x7fee20095940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee20094560 0x7fee20094960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 --2- 172.21.15.32:0/3608582451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee200a7930 0x7fee200c26b0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 >> 172.21.15.32:0/3608582451 conn(0x7fee20093fc0 msgr2=0x7fee200b92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 shutdown_connections 2021-04-09T18:26:18.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:17.956 7fee27ff4700 1 -- 172.21.15.32:0/3608582451 wait complete. 2021-04-09T18:26:18.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:26:18.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.125 7fcc54f63700 1 Processor -- start 2021-04-09T18:26:18.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.125 7fcc54f63700 1 -- start start 2021-04-09T18:26:18.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.126 7fcc54f63700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcc500cffb0 0x7fcc500d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.126 7fcc54f63700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcc500d6c90 0x7fcc5010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.126 7fcc54f63700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc500d7290 con 0x7fcc500d7810 2021-04-09T18:26:18.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.126 7fcc54f63700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc500d7090 con 0x7fcc500d6c90 2021-04-09T18:26:18.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.126 7fcc54f63700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc500d91f0 con 0x7fcc500cffb0 2021-04-09T18:26:18.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.126 7fcc4f314700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcc500d6c90 0x7fcc5010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.126 7fcc4e312700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcc500d7810 0x7fcc500d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34360/0 (socket says 172.21.15.32:34360) 2021-04-09T18:26:18.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.126 7fcc4e312700 1 -- 172.21.15.32:0/344756912 learned_addr learned my addr 172.21.15.32:0/344756912 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:18.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4eb13700 1 --2- 172.21.15.32:0/344756912 >> v2:172.21.15.86:3300/0 conn(0x7fcc500cffb0 0x7fcc500d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4f314700 1 -- 172.21.15.32:0/344756912 >> v2:172.21.15.86:3300/0 conn(0x7fcc500cffb0 msgr2=0x7fcc500d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:18.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4f314700 1 --2- 172.21.15.32:0/344756912 >> v2:172.21.15.86:3300/0 conn(0x7fcc500cffb0 0x7fcc500d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4f314700 1 -- 172.21.15.32:0/344756912 >> v1:172.21.15.32:6789/0 conn(0x7fcc500d7810 legacy=0x7fcc500d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4f314700 1 -- 172.21.15.32:0/344756912 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500d8400 con 0x7fcc500d6c90 2021-04-09T18:26:18.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4db11700 1 -- 172.21.15.32:0/344756912 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964454855 0 0) 0x7fcc440009b0 con 0x7fcc500d7810 2021-04-09T18:26:18.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4eb13700 1 --2- 172.21.15.32:0/344756912 >> v2:172.21.15.86:3300/0 conn(0x7fcc500cffb0 0x7fcc500d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:18.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4f314700 1 --2- 172.21.15.32:0/344756912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc500d6c90 0x7fcc5010cfe0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7fcc48006e30 tx=0x7fcc48009e60).ready entity=mon.0 client_cookie=6023cba21b8e6db0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:18.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4db11700 1 -- 172.21.15.32:0/344756912 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc48006a50 con 0x7fcc500d6c90 2021-04-09T18:26:18.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4db11700 1 -- 172.21.15.32:0/344756912 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4800b300 con 0x7fcc500d6c90 2021-04-09T18:26:18.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc4db11700 1 -- 172.21.15.32:0/344756912 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4800b750 con 0x7fcc500d6c90 2021-04-09T18:26:18.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc54f63700 1 -- 172.21.15.32:0/344756912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc500d6c90 msgr2=0x7fcc5010cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.127 7fcc54f63700 1 --2- 172.21.15.32:0/344756912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc500d6c90 0x7fcc5010cfe0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7fcc48006e30 tx=0x7fcc48009e60).stop 2021-04-09T18:26:18.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.128 7fcc54f63700 1 -- 172.21.15.32:0/344756912 shutdown_connections 2021-04-09T18:26:18.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.128 7fcc54f63700 1 --2- 172.21.15.32:0/344756912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc500d6c90 0x7fcc5010cfe0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.128 7fcc54f63700 1 --2- 172.21.15.32:0/344756912 >> v2:172.21.15.86:3300/0 conn(0x7fcc500cffb0 0x7fcc500d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.128 7fcc54f63700 1 -- 172.21.15.32:0/344756912 >> 172.21.15.32:0/344756912 conn(0x7fcc500ce5d0 msgr2=0x7fcc500cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:18.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.128 7fcc54f63700 1 -- 172.21.15.32:0/344756912 shutdown_connections 2021-04-09T18:26:18.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.128 7fcc54f63700 1 -- 172.21.15.32:0/344756912 wait complete. 2021-04-09T18:26:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.148 7fcc54f63700 1 Processor -- start 2021-04-09T18:26:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.148 7fcc54f63700 1 -- start start 2021-04-09T18:26:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc54f63700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc5008ac00 0x7fcc500b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc54f63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc5008b000 0x7fcc501ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc54f63700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0df0 0x7fcc501abc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc54f63700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc501ab130 con 0x7fcc5008b000 2021-04-09T18:26:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc54f63700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500b43e0 con 0x7fcc5008ac00 2021-04-09T18:26:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc54f63700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500b4600 con 0x7fcc500a0df0 2021-04-09T18:26:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4e312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc5008b000 0x7fcc501ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4e312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc5008b000 0x7fcc501ab630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53006/0 (socket says 172.21.15.32:53006) 2021-04-09T18:26:18.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4e312700 1 -- 172.21.15.32:0/3897939222 learned_addr learned my addr 172.21.15.32:0/3897939222 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:18.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4eb13700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc5008ac00 0x7fcc500b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4f314700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0df0 0x7fcc501abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4e312700 1 -- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0df0 msgr2=0x7fcc501abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4e312700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0df0 0x7fcc501abc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4e312700 1 -- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc5008ac00 msgr2=0x7fcc500b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4e312700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc5008ac00 0x7fcc500b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4e312700 1 -- 172.21.15.32:0/3897939222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc501a9e60 con 0x7fcc5008b000 2021-04-09T18:26:18.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc4e312700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc5008b000 0x7fcc501ab630 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 rx=0x7fcc44001c60 tx=0x7fcc4400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc3f7fe700 1 -- 172.21.15.32:0/3897939222 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4400acc0 con 0x7fcc5008b000 2021-04-09T18:26:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc3f7fe700 1 -- 172.21.15.32:0/3897939222 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc44004140 con 0x7fcc5008b000 2021-04-09T18:26:18.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc501a9e60 con 0x7fcc5008b000 2021-04-09T18:26:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc3f7fe700 1 -- 172.21.15.32:0/3897939222 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc44004620 con 0x7fcc5008b000 2021-04-09T18:26:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.149 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc501a9e60 con 0x7fcc5008b000 2021-04-09T18:26:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.151 7fcc3f7fe700 1 -- 172.21.15.32:0/3897939222 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcc44000dd0 con 0x7fcc5008b000 2021-04-09T18:26:18.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.151 7fcc3f7fe700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc340188c0 0x7fcc3401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.151 7fcc3f7fe700 1 -- 172.21.15.32:0/3897939222 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fcc440324e0 con 0x7fcc5008b000 2021-04-09T18:26:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.151 7fcc4eb13700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc340188c0 0x7fcc3401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.151 7fcc4eb13700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc340188c0 0x7fcc3401adc0 crc :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:18.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.170 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc50182e10 con 0x7fcc5008b000 2021-04-09T18:26:18.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.174 7fcc3f7fe700 1 -- 172.21.15.32:0/3897939222 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcc440009b0 con 0x7fcc5008b000 2021-04-09T18:26:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.383 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fcc50182e10 con 0x7fcc5008b000 2021-04-09T18:26:18.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.387 7fcc3f7fe700 1 -- 172.21.15.32:0/3897939222 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcc44000dd0 con 0x7fcc5008b000 2021-04-09T18:26:18.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc340188c0 msgr2=0x7fcc3401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc340188c0 0x7fcc3401adc0 crc :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc5008b000 msgr2=0x7fcc501ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc5008b000 0x7fcc501ab630 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 rx=0x7fcc44001c60 tx=0x7fcc4400a8d0).stop 2021-04-09T18:26:18.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 shutdown_connections 2021-04-09T18:26:18.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcc340188c0 0x7fcc3401adc0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc5008b000 0x7fcc501ab630 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0df0 0x7fcc501abc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 --2- 172.21.15.32:0/3897939222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc5008ac00 0x7fcc500b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 >> 172.21.15.32:0/3897939222 conn(0x7fcc50183890 msgr2=0x7fcc5019b900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:18.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 shutdown_connections 2021-04-09T18:26:18.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.393 7fcc54f63700 1 -- 172.21.15.32:0/3897939222 wait complete. 2021-04-09T18:26:18.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestFailover.test_client_abort' 2021-04-09T18:26:18.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.554 7f82a9cdb700 1 Processor -- start 2021-04-09T18:26:18.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.554 7f82a9cdb700 1 -- start start 2021-04-09T18:26:18.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a9cdb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82a40d7ca0 0x7f82a40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a9cdb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f82a410da00 0x7f82a410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a9cdb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a410d480 con 0x7f82a410d010 2021-04-09T18:26:18.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a9cdb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a40d9220 con 0x7f82a410da00 2021-04-09T18:26:18.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a9cdb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a40d86c0 con 0x7f82a40d7ca0 2021-04-09T18:26:18.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a37fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f82a410d010 0x7f82a40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34370/0 (socket says 172.21.15.32:34370) 2021-04-09T18:26:18.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a37fe700 1 -- 172.21.15.32:0/1846736863 learned_addr learned my addr 172.21.15.32:0/1846736863 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:18.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a3fff700 1 --2- 172.21.15.32:0/1846736863 >> v2:172.21.15.32:3300/0 conn(0x7f82a410da00 0x7f82a410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a2ffd700 1 --2- 172.21.15.32:0/1846736863 >> v2:172.21.15.86:3300/0 conn(0x7f82a40d7ca0 0x7f82a40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a27fc700 1 -- 172.21.15.32:0/1846736863 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169651288 0 0) 0x7f8294000ce0 con 0x7f82a410d010 2021-04-09T18:26:18.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a27fc700 1 -- 172.21.15.32:0/1846736863 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8288002fe0 con 0x7f82a410d010 2021-04-09T18:26:18.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a3fff700 1 -- 172.21.15.32:0/1846736863 >> v2:172.21.15.86:3300/0 conn(0x7f82a40d7ca0 msgr2=0x7f82a40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:18.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a3fff700 1 --2- 172.21.15.32:0/1846736863 >> v2:172.21.15.86:3300/0 conn(0x7f82a40d7ca0 0x7f82a40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a3fff700 1 -- 172.21.15.32:0/1846736863 >> v1:172.21.15.32:6789/0 conn(0x7f82a410d010 legacy=0x7f82a40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a3fff700 1 -- 172.21.15.32:0/1846736863 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a40d1f70 con 0x7f82a410da00 2021-04-09T18:26:18.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a2ffd700 1 --2- 172.21.15.32:0/1846736863 >> v2:172.21.15.86:3300/0 conn(0x7f82a40d7ca0 0x7f82a40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:26:18.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a3fff700 1 --2- 172.21.15.32:0/1846736863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a410da00 0x7f82a410de00 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7f829c006e30 tx=0x7f829c009e60).ready entity=mon.0 client_cookie=650f70b61d256684 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:18.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a27fc700 1 -- 172.21.15.32:0/1846736863 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f829c00b300 con 0x7f82a410da00 2021-04-09T18:26:18.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.555 7f82a27fc700 1 -- 172.21.15.32:0/1846736863 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c000d30 con 0x7f82a410da00 2021-04-09T18:26:18.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.556 7f82a27fc700 1 -- 172.21.15.32:0/1846736863 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f829c00b830 con 0x7f82a410da00 2021-04-09T18:26:18.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.556 7f82a9cdb700 1 -- 172.21.15.32:0/1846736863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a410da00 msgr2=0x7f82a410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.556 7f82a9cdb700 1 --2- 172.21.15.32:0/1846736863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a410da00 0x7f82a410de00 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7f829c006e30 tx=0x7f829c009e60).stop 2021-04-09T18:26:18.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.556 7f82a9cdb700 1 -- 172.21.15.32:0/1846736863 shutdown_connections 2021-04-09T18:26:18.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.556 7f82a9cdb700 1 --2- 172.21.15.32:0/1846736863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a410da00 0x7f82a410de00 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.556 7f82a9cdb700 1 --2- 172.21.15.32:0/1846736863 >> v2:172.21.15.86:3300/0 conn(0x7f82a40d7ca0 0x7f82a40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.556 7f82a9cdb700 1 -- 172.21.15.32:0/1846736863 >> 172.21.15.32:0/1846736863 conn(0x7f82a40ce5f0 msgr2=0x7f82a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:18.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.556 7f82a9cdb700 1 -- 172.21.15.32:0/1846736863 shutdown_connections 2021-04-09T18:26:18.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.556 7f82a9cdb700 1 -- 172.21.15.32:0/1846736863 wait complete. 2021-04-09T18:26:18.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.576 7f82a9cdb700 1 Processor -- start 2021-04-09T18:26:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a9cdb700 1 -- start start 2021-04-09T18:26:18.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a9cdb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41ac670 0x7f82a41a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a9cdb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a41a5dc0 0x7f82a40ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a9cdb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a41aca70 0x7f82a40ba800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a9cdb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a41a5220 con 0x7f82a41a5dc0 2021-04-09T18:26:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a9cdb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a418e100 con 0x7f82a41ac670 2021-04-09T18:26:18.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a9cdb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a419ccc0 con 0x7f82a41aca70 2021-04-09T18:26:18.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a41a5dc0 0x7f82a40ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a41a5dc0 0x7f82a40ba1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53020/0 (socket says 172.21.15.32:53020) 2021-04-09T18:26:18.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.577 7f82a2ffd700 1 -- 172.21.15.32:0/277935744 learned_addr learned my addr 172.21.15.32:0/277935744 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:18.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a3fff700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a41aca70 0x7f82a40ba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a37fe700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41ac670 0x7f82a41a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a2ffd700 1 -- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a41aca70 msgr2=0x7f82a40ba800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a2ffd700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a41aca70 0x7f82a40ba800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a2ffd700 1 -- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41ac670 msgr2=0x7f82a41a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:18.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a2ffd700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41ac670 0x7f82a41a57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:18.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a2ffd700 1 -- 172.21.15.32:0/277935744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a40bae20 con 0x7f82a41a5dc0 2021-04-09T18:26:18.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a2ffd700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a41a5dc0 0x7f82a40ba1e0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 rx=0x7f8290006f20 tx=0x7f829000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:18.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f829bfff700 1 -- 172.21.15.32:0/277935744 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f829000bd80 con 0x7f82a41a5dc0 2021-04-09T18:26:18.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f829bfff700 1 -- 172.21.15.32:0/277935744 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8290000c90 con 0x7f82a41a5dc0 2021-04-09T18:26:18.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f829bfff700 1 -- 172.21.15.32:0/277935744 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82900022b0 con 0x7f82a41a5dc0 2021-04-09T18:26:18.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a40bae20 con 0x7f82a41a5dc0 2021-04-09T18:26:18.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.578 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a40bae20 con 0x7f82a41a5dc0 2021-04-09T18:26:18.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.579 7f829bfff700 1 -- 172.21.15.32:0/277935744 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f829000a000 con 0x7f82a41a5dc0 2021-04-09T18:26:18.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.579 7f829bfff700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8280018930 0x7f828001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:18.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.579 7f829bfff700 1 -- 172.21.15.32:0/277935744 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f8290032750 con 0x7f82a41a5dc0 2021-04-09T18:26:18.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.579 7f82a37fe700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8280018930 0x7f828001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:18.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.579 7f82a37fe700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8280018930 0x7f828001adc0 crc :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:18.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.599 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a4182d10 con 0x7f82a41a5dc0 2021-04-09T18:26:18.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.602 7f829bfff700 1 -- 172.21.15.32:0/277935744 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8290006bb0 con 0x7f82a41a5dc0 2021-04-09T18:26:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.791 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestFailover.test_client_abort"]} v 0) v1 -- 0x7f82a4182d10 con 0x7f82a41a5dc0 2021-04-09T18:26:19.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.946 7f829bfff700 1 -- 172.21.15.32:0/277935744 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestFailover.test_client_abort"]}]=0 v644) v1 ==== 136+0+0 (secure 0 0 0) 0x7f8290006bb0 con 0x7f82a41a5dc0 2021-04-09T18:26:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8280018930 msgr2=0x7f828001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8280018930 0x7f828001adc0 crc :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a41a5dc0 msgr2=0x7f82a40ba1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:19.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a41a5dc0 0x7f82a40ba1e0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 rx=0x7f8290006f20 tx=0x7f829000b990).stop 2021-04-09T18:26:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 shutdown_connections 2021-04-09T18:26:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8280018930 0x7f828001adc0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f82a41a5dc0 0x7f82a40ba1e0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a41aca70 0x7f82a40ba800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 --2- 172.21.15.32:0/277935744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a41ac670 0x7f82a41a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 >> 172.21.15.32:0/277935744 conn(0x7f82a419b890 msgr2=0x7f82a41a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 shutdown_connections 2021-04-09T18:26:19.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:18.957 7f82a9cdb700 1 -- 172.21.15.32:0/277935744 wait complete. 2021-04-09T18:26:19.037 INFO:tasks.cephfs_test_runner:test_client_abort (tasks.cephfs.test_failover.TestFailover) ... ok 2021-04-09T18:26:19.037 INFO:tasks.cephfs_test_runner:Starting test: test_discontinuous_mdsmap (tasks.cephfs.test_failover.TestFailover) 2021-04-09T18:26:19.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestFailover.test_discontinuous_mdsmap' 2021-04-09T18:26:19.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.117 7f66ec6f3700 1 Processor -- start 2021-04-09T18:26:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.117 7f66ec6f3700 1 -- start start 2021-04-09T18:26:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ec6f3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66e40cffa0 0x7f66e40d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ec6f3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e40d6c80 0x7f66e410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ec6f3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e40d7280 con 0x7f66e40d7800 2021-04-09T18:26:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ec6f3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e40d7080 con 0x7f66e40cffa0 2021-04-09T18:26:19.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ec6f3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e40d91e0 con 0x7f66e40d6c80 2021-04-09T18:26:19.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ea50f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66e40cffa0 0x7f66e40d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:19.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ea50f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66e40cffa0 0x7f66e40d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53026/0 (socket says 172.21.15.32:53026) 2021-04-09T18:26:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ea50f700 1 -- 172.21.15.32:0/405354533 learned_addr learned my addr 172.21.15.32:0/405354533 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ead10700 1 --2- 172.21.15.32:0/405354533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e40d6c80 0x7f66e410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ea50f700 1 -- 172.21.15.32:0/405354533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e40d6c80 msgr2=0x7f66e410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ea50f700 1 --2- 172.21.15.32:0/405354533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e40d6c80 0x7f66e410cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ea50f700 1 -- 172.21.15.32:0/405354533 >> v1:172.21.15.86:6789/0 conn(0x7f66e40d7800 legacy=0x7f66e40d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ea50f700 1 -- 172.21.15.32:0/405354533 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e40d83f0 con 0x7f66e40cffa0 2021-04-09T18:26:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66ea50f700 1 --2- 172.21.15.32:0/405354533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e40cffa0 0x7f66e40d2440 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7f66e0006700 tx=0x7f66e0009b80).ready entity=mon.0 client_cookie=e2e3df4fb127f659 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.118 7f66e950d700 1 -- 172.21.15.32:0/405354533 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66e000c0c0 con 0x7f66e40cffa0 2021-04-09T18:26:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66e950d700 1 -- 172.21.15.32:0/405354533 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e000c440 con 0x7f66e40cffa0 2021-04-09T18:26:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66e950d700 1 -- 172.21.15.32:0/405354533 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66e000c920 con 0x7f66e40cffa0 2021-04-09T18:26:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66ec6f3700 1 -- 172.21.15.32:0/405354533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e40cffa0 msgr2=0x7f66e40d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66ec6f3700 1 --2- 172.21.15.32:0/405354533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e40cffa0 0x7f66e40d2440 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7f66e0006700 tx=0x7f66e0009b80).stop 2021-04-09T18:26:19.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66ec6f3700 1 -- 172.21.15.32:0/405354533 shutdown_connections 2021-04-09T18:26:19.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66ec6f3700 1 --2- 172.21.15.32:0/405354533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e40d6c80 0x7f66e410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66ec6f3700 1 --2- 172.21.15.32:0/405354533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e40cffa0 0x7f66e40d2440 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66ec6f3700 1 -- 172.21.15.32:0/405354533 >> 172.21.15.32:0/405354533 conn(0x7f66e40ce5c0 msgr2=0x7f66e40cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:19.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66ec6f3700 1 -- 172.21.15.32:0/405354533 shutdown_connections 2021-04-09T18:26:19.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.119 7f66ec6f3700 1 -- 172.21.15.32:0/405354533 wait complete. 2021-04-09T18:26:19.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.140 7f66ec6f3700 1 Processor -- start 2021-04-09T18:26:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.140 7f66ec6f3700 1 -- start start 2021-04-09T18:26:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ec6f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e41929f0 0x7f66e4192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ec6f3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66e408ddc0 0x7f66e408e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ec6f3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e408e7e0 0x7f66e408c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ec6f3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e4193410 con 0x7f66e41929f0 2021-04-09T18:26:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ec6f3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e40a9140 con 0x7f66e408ddc0 2021-04-09T18:26:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ec6f3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e408ce80 con 0x7f66e408e7e0 2021-04-09T18:26:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ea50f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e41929f0 0x7f66e4192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ea50f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e41929f0 0x7f66e4192df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53032/0 (socket says 172.21.15.32:53032) 2021-04-09T18:26:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ea50f700 1 -- 172.21.15.32:0/2146348893 learned_addr learned my addr 172.21.15.32:0/2146348893 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66e9d0e700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66e408ddc0 0x7f66e408e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ead10700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e408e7e0 0x7f66e408c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ea50f700 1 -- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e408e7e0 msgr2=0x7f66e408c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ea50f700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e408e7e0 0x7f66e408c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ea50f700 1 -- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66e408ddc0 msgr2=0x7f66e408e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ea50f700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66e408ddc0 0x7f66e408e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ea50f700 1 -- 172.21.15.32:0/2146348893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e408d0a0 con 0x7f66e41929f0 2021-04-09T18:26:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66ea50f700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e41929f0 0x7f66e4192df0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 rx=0x7f66e000b5e0 tx=0x7f66e000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.141 7f66d6ffd700 1 -- 172.21.15.32:0/2146348893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66e000bd70 con 0x7f66e41929f0 2021-04-09T18:26:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.142 7f66d6ffd700 1 -- 172.21.15.32:0/2146348893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e0002030 con 0x7f66e41929f0 2021-04-09T18:26:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.142 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e408d0a0 con 0x7f66e41929f0 2021-04-09T18:26:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.142 7f66d6ffd700 1 -- 172.21.15.32:0/2146348893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66e0002550 con 0x7f66e41929f0 2021-04-09T18:26:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.142 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e408d0a0 con 0x7f66e41929f0 2021-04-09T18:26:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.142 7f66d6ffd700 1 -- 172.21.15.32:0/2146348893 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f66e0000c90 con 0x7f66e41929f0 2021-04-09T18:26:19.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.142 7f66d6ffd700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66cc018930 0x7f66cc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:19.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.142 7f66d6ffd700 1 -- 172.21.15.32:0/2146348893 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f66e0032780 con 0x7f66e41929f0 2021-04-09T18:26:19.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.143 7f66e9d0e700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66cc018930 0x7f66cc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:19.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.143 7f66e9d0e700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66cc018930 0x7f66cc01ae10 crc :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:19.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.162 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e41b70c0 con 0x7f66e41929f0 2021-04-09T18:26:19.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.165 7f66d6ffd700 1 -- 172.21.15.32:0/2146348893 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f66e001f7a0 con 0x7f66e41929f0 2021-04-09T18:26:19.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.353 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestFailover.test_discontinuous_mdsmap"]} v 0) v1 -- 0x7f66e40d7280 con 0x7f66e41929f0 2021-04-09T18:26:20.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.947 7f66d6ffd700 1 -- 172.21.15.32:0/2146348893 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestFailover.test_discontinuous_mdsmap"]}]=0 v645) v1 ==== 147+0+0 (secure 0 0 0) 0x7f66e001f7a0 con 0x7f66e41929f0 2021-04-09T18:26:20.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66cc018930 msgr2=0x7f66cc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66cc018930 0x7f66cc01ae10 crc :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e41929f0 msgr2=0x7f66e4192df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e41929f0 0x7f66e4192df0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 rx=0x7f66e000b5e0 tx=0x7f66e000b980).stop 2021-04-09T18:26:20.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 shutdown_connections 2021-04-09T18:26:20.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66cc018930 0x7f66cc01ae10 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66e408e7e0 0x7f66e408c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66e408ddc0 0x7f66e408e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 --2- 172.21.15.32:0/2146348893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66e41929f0 0x7f66e4192df0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 >> 172.21.15.32:0/2146348893 conn(0x7f66e4189b00 msgr2=0x7f66e41921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:20.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 shutdown_connections 2021-04-09T18:26:20.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:19.970 7f66ec6f3700 1 -- 172.21.15.32:0/2146348893 wait complete. 2021-04-09T18:26:20.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:26:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.141 7f0656be1700 1 Processor -- start 2021-04-09T18:26:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.141 7f0656be1700 1 -- start start 2021-04-09T18:26:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.141 7f0656be1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f065010d010 0x7f06500d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.141 7f0656be1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065010d480 con 0x7f06500d1da0 2021-04-09T18:26:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.141 7f0656be1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065010e1f0 con 0x7f06500d8490 2021-04-09T18:26:20.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.141 7f0656be1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06500d8d30 con 0x7f065010d010 2021-04-09T18:26:20.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f06500d1da0 0x7f06500d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34402/0 (socket says 172.21.15.32:34402) 2021-04-09T18:26:20.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064ffff700 1 -- 172.21.15.32:0/1009505461 learned_addr learned my addr 172.21.15.32:0/1009505461 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:20.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f06549fd700 1 --2- 172.21.15.32:0/1009505461 >> v2:172.21.15.86:3300/0 conn(0x7f065010d010 0x7f06500d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401362924 0 0) 0x7f0644000ce0 con 0x7f06500d1da0 2021-04-09T18:26:20.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0638002fe0 con 0x7f06500d1da0 2021-04-09T18:26:20.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856413974 0 0) 0x7f0648000d30 con 0x7f06500d8490 2021-04-09T18:26:20.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0638003210 con 0x7f06500d8490 2021-04-09T18:26:20.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2666247919 0 0) 0x7f0644000ed0 con 0x7f06500d1da0 2021-04-09T18:26:20.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 >> v2:172.21.15.86:3300/0 conn(0x7f065010d010 msgr2=0x7f06500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:20.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 --2- 172.21.15.32:0/1009505461 >> v2:172.21.15.86:3300/0 conn(0x7f065010d010 0x7f06500d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 >> v1:172.21.15.86:6789/0 conn(0x7f06500d8490 legacy=0x7f065010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06500d9220 con 0x7f06500d1da0 2021-04-09T18:26:20.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f0644001600 con 0x7f06500d1da0 2021-04-09T18:26:20.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 >> v1:172.21.15.32:6789/0 conn(0x7f06500d1da0 legacy=0x7f06500d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06380038e0 0x7f0638003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06380067e0 0x7f0638008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0638009580 0x7f063800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0638050300 con 0x7f06380038e0 2021-04-09T18:26:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06380504e0 con 0x7f06380067e0 2021-04-09T18:26:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0638050700 con 0x7f0638009580 2021-04-09T18:26:20.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.142 7f06549fd700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06380038e0 0x7f0638003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f064ffff700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06380067e0 0x7f0638008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f06551fe700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0638009580 0x7f063800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f06549fd700 1 -- 172.21.15.32:0/1009505461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0638009580 msgr2=0x7f063800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:20.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f06549fd700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0638009580 0x7f063800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f06549fd700 1 -- 172.21.15.32:0/1009505461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06380067e0 msgr2=0x7f0638008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:20.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f06549fd700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06380067e0 0x7f0638008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f06549fd700 1 -- 172.21.15.32:0/1009505461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06380509a0 con 0x7f06380038e0 2021-04-09T18:26:20.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f06551fe700 1 -- 172.21.15.32:0/1009505461 reap_dead start 2021-04-09T18:26:20.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f06549fd700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06380038e0 0x7f0638003ce0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7f064000c8f0 tx=0x7f0640006c20).ready entity=mon.0 client_cookie=d3119ac3854c3001 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:20.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06400090d0 con 0x7f06380038e0 2021-04-09T18:26:20.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0640009450 con 0x7f06380038e0 2021-04-09T18:26:20.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f064f7fe700 1 -- 172.21.15.32:0/1009505461 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0640009930 con 0x7f06380038e0 2021-04-09T18:26:20.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f0656be1700 1 -- 172.21.15.32:0/1009505461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06380038e0 msgr2=0x7f0638003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.143 7f0656be1700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06380038e0 0x7f0638003ce0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7f064000c8f0 tx=0x7f0640006c20).stop 2021-04-09T18:26:20.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.144 7f0656be1700 1 -- 172.21.15.32:0/1009505461 shutdown_connections 2021-04-09T18:26:20.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.144 7f0656be1700 1 --2- 172.21.15.32:0/1009505461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06380038e0 0x7f0638003ce0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.144 7f0656be1700 1 -- 172.21.15.32:0/1009505461 >> 172.21.15.32:0/1009505461 conn(0x7f06500ce5f0 msgr2=0x7f06500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:20.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.144 7f0656be1700 1 -- 172.21.15.32:0/1009505461 shutdown_connections 2021-04-09T18:26:20.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.144 7f0656be1700 1 -- 172.21.15.32:0/1009505461 wait complete. 2021-04-09T18:26:20.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.164 7f0656be1700 1 Processor -- start 2021-04-09T18:26:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.164 7f0656be1700 1 -- start start 2021-04-09T18:26:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f0656be1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06501a0050 0x7f06501a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f0656be1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06500bdcd0 0x7f06500bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f0656be1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06500bf080 0x7f06500b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f0656be1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06500bf500 con 0x7f06500bdcd0 2021-04-09T18:26:20.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f0656be1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06500bb7b0 con 0x7f06500bf080 2021-04-09T18:26:20.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f0656be1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06500be0d0 con 0x7f06501a0050 2021-04-09T18:26:20.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06500bdcd0 0x7f06500bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06500bdcd0 0x7f06500bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53054/0 (socket says 172.21.15.32:53054) 2021-04-09T18:26:20.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064ffff700 1 -- 172.21.15.32:0/3474542170 learned_addr learned my addr 172.21.15.32:0/3474542170 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:20.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f06549fd700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06501a0050 0x7f06501a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f06551fe700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06500bf080 0x7f06500b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064ffff700 1 -- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06501a0050 msgr2=0x7f06501a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064ffff700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06501a0050 0x7f06501a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064ffff700 1 -- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06500bf080 msgr2=0x7f06500b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064ffff700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06500bf080 0x7f06500b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064ffff700 1 -- 172.21.15.32:0/3474542170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06500b1310 con 0x7f06500bdcd0 2021-04-09T18:26:20.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064ffff700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06500bdcd0 0x7f06500bea60 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7f0644007fc0 tx=0x7f064400a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:20.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064d7fa700 1 -- 172.21.15.32:0/3474542170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0644007280 con 0x7f06500bdcd0 2021-04-09T18:26:20.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064d7fa700 1 -- 172.21.15.32:0/3474542170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064400ad10 con 0x7f06500bdcd0 2021-04-09T18:26:20.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f0656be1700 1 -- 172.21.15.32:0/3474542170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06500b1310 con 0x7f06500bdcd0 2021-04-09T18:26:20.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f064d7fa700 1 -- 172.21.15.32:0/3474542170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06440043a0 con 0x7f06500bdcd0 2021-04-09T18:26:20.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.165 7f0656be1700 1 -- 172.21.15.32:0/3474542170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06500b1310 con 0x7f06500bdcd0 2021-04-09T18:26:20.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.166 7f064d7fa700 1 -- 172.21.15.32:0/3474542170 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0644007280 con 0x7f06500bdcd0 2021-04-09T18:26:20.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.166 7f064d7fa700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0630018930 0x7f063001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.166 7f064d7fa700 1 -- 172.21.15.32:0/3474542170 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f0644032710 con 0x7f06500bdcd0 2021-04-09T18:26:20.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.166 7f06549fd700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0630018930 0x7f063001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.167 7f06549fd700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0630018930 0x7f063001ae10 crc :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:20.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.186 7f0656be1700 1 -- 172.21.15.32:0/3474542170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0650093d50 con 0x7f06500bdcd0 2021-04-09T18:26:20.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.189 7f064d7fa700 1 -- 172.21.15.32:0/3474542170 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0644007070 con 0x7f06500bdcd0 2021-04-09T18:26:20.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.402 7f0656be1700 1 -- 172.21.15.32:0/3474542170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f065010d480 con 0x7f06500bdcd0 2021-04-09T18:26:20.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.405 7f064d7fa700 1 -- 172.21.15.32:0/3474542170 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0644007280 con 0x7f06500bdcd0 2021-04-09T18:26:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 -- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0630018930 msgr2=0x7f063001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0630018930 0x7f063001ae10 crc :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 -- 172.21.15.32:0/3474542170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06500bdcd0 msgr2=0x7f06500bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06500bdcd0 0x7f06500bea60 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7f0644007fc0 tx=0x7f064400a920).stop 2021-04-09T18:26:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 -- 172.21.15.32:0/3474542170 shutdown_connections 2021-04-09T18:26:20.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0630018930 0x7f063001ae10 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06500bdcd0 0x7f06500bea60 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06500bf080 0x7f06500b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 --2- 172.21.15.32:0/3474542170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06501a0050 0x7f06501a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 -- 172.21.15.32:0/3474542170 >> 172.21.15.32:0/3474542170 conn(0x7f06500975f0 msgr2=0x7f0650096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:20.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 -- 172.21.15.32:0/3474542170 shutdown_connections 2021-04-09T18:26:20.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.411 7f0656be1700 1 -- 172.21.15.32:0/3474542170 wait complete. 2021-04-09T18:26:20.492 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:26:20.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:26:20.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:26:20.607 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:26:20.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:20.608 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:26:20.608 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:26:20.654 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:26:20.654 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:26:20.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:20.656 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:26:20.657 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:26:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:26:20.699 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:26:20.699 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:26:20.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:26:20.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.776 7fbafbcfe700 1 Processor -- start 2021-04-09T18:26:20.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.776 7fbafbcfe700 1 -- start start 2021-04-09T18:26:20.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbafbcfe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbaf40cffa0 0x7fbaf40d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbafbcfe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf40d7800 0x7fbaf40d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbafbcfe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbaf40d8220 0x7fbaf40d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbafbcfe700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf40d91e0 con 0x7fbaf40cffa0 2021-04-09T18:26:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbafbcfe700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf40d7280 con 0x7fbaf40d8220 2021-04-09T18:26:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbafbcfe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf40d6290 con 0x7fbaf40d7800 2021-04-09T18:26:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9b1a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbaf40cffa0 0x7fbaf40d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9b1a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbaf40cffa0 0x7fbaf40d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53060/0 (socket says 172.21.15.32:53060) 2021-04-09T18:26:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9b1a700 1 -- 172.21.15.32:0/3815019096 learned_addr learned my addr 172.21.15.32:0/3815019096 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9319700 1 --2- 172.21.15.32:0/3815019096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf40d7800 0x7fbaf40d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbafa31b700 1 --2- 172.21.15.32:0/3815019096 >> v2:172.21.15.86:3300/0 conn(0x7fbaf40d8220 0x7fbaf40d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9b1a700 1 -- 172.21.15.32:0/3815019096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf40d7800 msgr2=0x7fbaf40d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9b1a700 1 --2- 172.21.15.32:0/3815019096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf40d7800 0x7fbaf40d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9b1a700 1 -- 172.21.15.32:0/3815019096 >> v2:172.21.15.86:3300/0 conn(0x7fbaf40d8220 msgr2=0x7fbaf40d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:20.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9b1a700 1 --2- 172.21.15.32:0/3815019096 >> v2:172.21.15.86:3300/0 conn(0x7fbaf40d8220 0x7fbaf40d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9b1a700 1 -- 172.21.15.32:0/3815019096 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf40d64b0 con 0x7fbaf40cffa0 2021-04-09T18:26:20.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf9b1a700 1 --2- 172.21.15.32:0/3815019096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf40cffa0 0x7fbaf40d2440 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 rx=0x7fbaf0006700 tx=0x7fbaf0009b00).ready entity=mon.0 client_cookie=ad825530d9477255 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:20.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.777 7fbaf8b18700 1 -- 172.21.15.32:0/3815019096 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaf000c0c0 con 0x7fbaf40cffa0 2021-04-09T18:26:20.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbaf8b18700 1 -- 172.21.15.32:0/3815019096 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf000c400 con 0x7fbaf40cffa0 2021-04-09T18:26:20.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbaf8b18700 1 -- 172.21.15.32:0/3815019096 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaf000c8e0 con 0x7fbaf40cffa0 2021-04-09T18:26:20.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbafbcfe700 1 -- 172.21.15.32:0/3815019096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf40cffa0 msgr2=0x7fbaf40d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbafbcfe700 1 --2- 172.21.15.32:0/3815019096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf40cffa0 0x7fbaf40d2440 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 rx=0x7fbaf0006700 tx=0x7fbaf0009b00).stop 2021-04-09T18:26:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbafbcfe700 1 -- 172.21.15.32:0/3815019096 shutdown_connections 2021-04-09T18:26:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbafbcfe700 1 --2- 172.21.15.32:0/3815019096 >> v2:172.21.15.86:3300/0 conn(0x7fbaf40d8220 0x7fbaf40d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbafbcfe700 1 --2- 172.21.15.32:0/3815019096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf40d7800 0x7fbaf40d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbafbcfe700 1 --2- 172.21.15.32:0/3815019096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf40cffa0 0x7fbaf40d2440 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbafbcfe700 1 -- 172.21.15.32:0/3815019096 >> 172.21.15.32:0/3815019096 conn(0x7fbaf40ce5c0 msgr2=0x7fbaf40cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:20.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbafbcfe700 1 -- 172.21.15.32:0/3815019096 shutdown_connections 2021-04-09T18:26:20.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.778 7fbafbcfe700 1 -- 172.21.15.32:0/3815019096 wait complete. 2021-04-09T18:26:20.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.798 7fbafbcfe700 1 Processor -- start 2021-04-09T18:26:20.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbafbcfe700 1 -- start start 2021-04-09T18:26:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbafbcfe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaf41842e0 0x7fbaf418dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbafbcfe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf419da40 0x7fbaf419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbafbcfe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf419ec20 0x7fbaf419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbafbcfe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf419e6a0 con 0x7fbaf419da40 2021-04-09T18:26:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbafbcfe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf419ca60 con 0x7fbaf41842e0 2021-04-09T18:26:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbafbcfe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf419f640 con 0x7fbaf419ec20 2021-04-09T18:26:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbaf9319700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf419da40 0x7fbaf419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbaf9319700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf419da40 0x7fbaf419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53068/0 (socket says 172.21.15.32:53068) 2021-04-09T18:26:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbaf9319700 1 -- 172.21.15.32:0/1989247190 learned_addr learned my addr 172.21.15.32:0/1989247190 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbaf9b1a700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaf41842e0 0x7fbaf418dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbafa31b700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf419ec20 0x7fbaf419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbaf9319700 1 -- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf419ec20 msgr2=0x7fbaf419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbaf9319700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf419ec20 0x7fbaf419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbaf9319700 1 -- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaf41842e0 msgr2=0x7fbaf418dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:20.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.799 7fbaf9319700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaf41842e0 0x7fbaf418dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:20.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.800 7fbaf9319700 1 -- 172.21.15.32:0/1989247190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf4184b10 con 0x7fbaf419da40 2021-04-09T18:26:20.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.800 7fbaf9319700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf419da40 0x7fbaf419de40 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7fbae400b580 tx=0x7fbae400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:20.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.800 7fbaea7fc700 1 -- 172.21.15.32:0/1989247190 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbae40064d0 con 0x7fbaf419da40 2021-04-09T18:26:20.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.800 7fbaea7fc700 1 -- 172.21.15.32:0/1989247190 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae400bd10 con 0x7fbaf419da40 2021-04-09T18:26:20.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.800 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf4184b10 con 0x7fbaf419da40 2021-04-09T18:26:20.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.800 7fbaea7fc700 1 -- 172.21.15.32:0/1989247190 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbae4002030 con 0x7fbaf419da40 2021-04-09T18:26:20.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.800 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf4184b10 con 0x7fbaf419da40 2021-04-09T18:26:20.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.801 7fbaea7fc700 1 -- 172.21.15.32:0/1989247190 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbae40052e0 con 0x7fbaf419da40 2021-04-09T18:26:20.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.801 7fbaea7fc700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc018930 0x7fbadc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:20.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.801 7fbaea7fc700 1 -- 172.21.15.32:0/1989247190 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fbae40326f0 con 0x7fbaf419da40 2021-04-09T18:26:20.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.801 7fbaf9b1a700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc018930 0x7fbadc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:20.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.801 7fbaf9b1a700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc018930 0x7fbadc01ae10 crc :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:20.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.821 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf4182e10 con 0x7fbaf419da40 2021-04-09T18:26:20.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:20.824 7fbaea7fc700 1 -- 172.21.15.32:0/1989247190 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbae4006820 con 0x7fbaf419da40 2021-04-09T18:26:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.014 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fbaf40d7280 con 0x7fbaf419da40 2021-04-09T18:26:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.015 7fbaea7fc700 1 -- 172.21.15.32:0/1989247190 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v134) v1 ==== 81+0+25550 (secure 0 0 0) 0x7fbae40052e0 con 0x7fbaf419da40 2021-04-09T18:26:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:21.087 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:26:21.087 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 134, 2021-04-09T18:26:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:26:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:26:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:26:17.936792", 2021-04-09T18:26:21.088 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:26:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:26:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:26:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:26:21.089 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:26:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:26:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:26:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:26:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:26:21.090 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:26:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:26:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:26:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:26:21.091 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:26:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:26:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 23, 2021-04-09T18:26:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:26:21.092 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:26:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:26:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:26:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:26:21.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:26:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:26:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:26:21.094 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:26:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:26:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:26:21.095 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:26:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:26:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:26:21.096 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:26:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:26:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:26:21.097 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:26:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:26:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:26:21.098 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:26:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:26:21.099 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:26:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:26:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:26:21.100 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:26:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:26:21.101 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:26:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:26:21.102 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:26:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:26:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:26:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:26:21.103 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:26:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:26:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:26:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:26:21.104 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:26:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:26:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:26:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:26:21.105 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:26:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:26:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:26:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:26:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:26:21.106 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:26:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:26:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:26:21.107 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:26:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:26:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:26:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:26:21.108 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:26:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:26:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:26:21.109 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:26:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:26:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:26:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:26:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:26:21.110 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:26:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:26:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:26:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:26:21.111 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:26:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:26:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:26:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:26:21.112 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:26:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:26:21.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:26:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:26:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:21.114 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:21.115 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:21.116 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:26:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.119 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:26:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:21.120 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.121 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:26:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:21.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:26:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:21.126 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:26:21.129 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:26:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.131 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.132 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:26:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:26:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:26:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:26:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:26:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:26:21.135 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:26:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:21.136 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:26:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:26:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:21.138 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:21.139 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 124, 2021-04-09T18:26:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:21.140 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.141 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:26:21.142 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.143 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:26:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:21.144 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:21.145 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:26:21.146 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.147 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:26:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:26:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:21.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:26:21.151 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:21.152 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.153 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:26:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.154 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:26:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:26:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.155 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.156 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:26:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:26:21.157 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:26:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:26:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:21.158 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.159 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:26:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:26:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:21.160 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:21.161 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:21.162 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.163 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:26:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.164 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:26:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:21.165 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:26:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:21.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.168 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:26:21.169 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:21.171 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.172 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:26:21.173 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:26:21.175 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.176 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.177 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.178 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:26:21.179 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:26:21.181 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:26:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.182 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.183 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:26:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:26:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:26:21.184 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:26:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:21.185 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:21.186 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:26:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:26:21.188 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:21.189 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:21.190 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:21.191 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:21.192 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:21.193 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.194 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:26:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:21.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.196 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:26:21.197 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:21.199 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.200 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:26:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:21.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:26:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:21.203 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:21.204 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:26:21.205 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:26:21.206 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.207 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.208 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:26:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.209 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:26:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:26:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:26:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:26:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:26:21.211 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:26:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:26:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:26:21.213 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:21.214 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.216 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:26:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:26:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:21.218 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.220 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:26:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:21.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:26:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:21.222 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:21.223 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:26:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:21.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.226 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:26:21.227 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.229 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:26:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.230 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:26:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:26:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.231 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:26:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:26:21.232 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:26:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:26:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:26:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:26:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:26:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:26:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:26:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:26:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:26:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:26:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:21.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:26:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:21.246 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:26:21.248 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:26:21.249 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:26:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.250 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:26:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:26:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:26:21.251 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:26:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:21.252 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.254 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:26:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:26:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:21.255 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:21.256 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:21.257 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 124, 2021-04-09T18:26:21.258 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:26:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:21.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:26:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:21.261 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:21.262 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:26:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:21.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:26:21.264 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.265 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.266 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:26:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:21.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:26:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:21.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:26:21.270 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:26:21.271 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:26:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.272 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:26:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:26:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:26:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:26:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:21.273 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:26:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:26:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:26:21.275 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:26:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:26:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:26:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:26:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:26:21.276 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:26:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 117, 2021-04-09T18:26:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:26:21.277 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:26:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:26:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:26:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:26:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.280 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:26:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:26:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.282 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:26:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:26:21.284 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:26:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:21.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:26:21.286 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:26:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:26:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.288 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:26:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:26:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:26:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:26:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.290 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:26:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:26:21.291 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:26:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:26:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:26:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:26:21.292 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:26:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:26:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:26:21.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:26:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:21.294 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:26:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:21.296 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.297 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:26:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:21.298 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:26:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:21.300 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.301 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:26:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:21.302 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.303 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:26:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:21.304 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.305 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:26:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:21.306 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:26:21.307 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:26:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:26:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:26:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:26:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:26:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:26:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:26:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:26:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:26:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:26:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:26:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/3538582472": "2021-04-10 18:26:14.878818", 2021-04-09T18:26:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/3538582472": "2021-04-10 18:26:14.878818" 2021-04-09T18:26:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:26:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:26:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:26:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:26:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:26:21.312 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:26:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:26:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:26:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:21.313 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:26:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:26:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:26:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:26:21.314 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:26:21.315 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:26:21.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc018930 msgr2=0x7fbadc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc018930 0x7fbadc01ae10 crc :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf419da40 msgr2=0x7fbaf419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf419da40 0x7fbaf419de40 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7fbae400b580 tx=0x7fbae400b920).stop 2021-04-09T18:26:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 shutdown_connections 2021-04-09T18:26:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc018930 0x7fbadc01ae10 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbaf419da40 0x7fbaf419de40 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaf419ec20 0x7fbaf419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 --2- 172.21.15.32:0/1989247190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaf41842e0 0x7fbaf418dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.017 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 >> 172.21.15.32:0/1989247190 conn(0x7fbaf4097550 msgr2=0x7fbaf418b920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:21.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.018 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 shutdown_connections 2021-04-09T18:26:21.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.018 7fbafbcfe700 1 -- 172.21.15.32:0/1989247190 wait complete. 2021-04-09T18:26:21.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:21.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.399 7fe639a39700 1 Processor -- start 2021-04-09T18:26:21.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.399 7fe639a39700 1 -- start start 2021-04-09T18:26:21.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe639a39700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe63410d010 0x7fe6340d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe639a39700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340d8490 0x7fe63410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe639a39700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63410d480 con 0x7fe6340d1da0 2021-04-09T18:26:21.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe639a39700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6340d8d30 con 0x7fe63410d010 2021-04-09T18:26:21.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe639a39700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6340d9220 con 0x7fe6340d8490 2021-04-09T18:26:21.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe6337fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe63410d010 0x7fe6340d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe632ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe6340d1da0 0x7fe6340d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34436/0 (socket says 172.21.15.32:34436) 2021-04-09T18:26:21.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe632ffd700 1 -- 172.21.15.32:0/4184917360 learned_addr learned my addr 172.21.15.32:0/4184917360 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:21.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe633fff700 1 --2- 172.21.15.32:0/4184917360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340d8490 0x7fe63410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe6337fe700 1 -- 172.21.15.32:0/4184917360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340d8490 msgr2=0x7fe63410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:21.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe6337fe700 1 --2- 172.21.15.32:0/4184917360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340d8490 0x7fe63410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe6337fe700 1 -- 172.21.15.32:0/4184917360 >> v1:172.21.15.32:6789/0 conn(0x7fe6340d1da0 legacy=0x7fe6340d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe6327fc700 1 -- 172.21.15.32:0/4184917360 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289133233 0 0) 0x7fe628000ce0 con 0x7fe6340d1da0 2021-04-09T18:26:21.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe6337fe700 1 -- 172.21.15.32:0/4184917360 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63410e020 con 0x7fe63410d010 2021-04-09T18:26:21.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.400 7fe633fff700 1 --2- 172.21.15.32:0/4184917360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340d8490 0x7fe63410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:21.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.401 7fe6337fe700 1 --2- 172.21.15.32:0/4184917360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63410d010 0x7fe6340d1780 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 rx=0x7fe624006650 tx=0x7fe624009a80).ready entity=mon.0 client_cookie=9c21dc5f7590fdcd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:21.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.401 7fe6327fc700 1 -- 172.21.15.32:0/4184917360 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe62400c0c0 con 0x7fe63410d010 2021-04-09T18:26:21.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.401 7fe6327fc700 1 -- 172.21.15.32:0/4184917360 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62400c3c0 con 0x7fe63410d010 2021-04-09T18:26:21.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.401 7fe6327fc700 1 -- 172.21.15.32:0/4184917360 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe62400c810 con 0x7fe63410d010 2021-04-09T18:26:21.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.401 7fe639a39700 1 -- 172.21.15.32:0/4184917360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63410d010 msgr2=0x7fe6340d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.401 7fe639a39700 1 --2- 172.21.15.32:0/4184917360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63410d010 0x7fe6340d1780 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 rx=0x7fe624006650 tx=0x7fe624009a80).stop 2021-04-09T18:26:21.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.401 7fe639a39700 1 -- 172.21.15.32:0/4184917360 shutdown_connections 2021-04-09T18:26:21.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.401 7fe639a39700 1 --2- 172.21.15.32:0/4184917360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340d8490 0x7fe63410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.401 7fe639a39700 1 --2- 172.21.15.32:0/4184917360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe63410d010 0x7fe6340d1780 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.402 7fe639a39700 1 -- 172.21.15.32:0/4184917360 >> 172.21.15.32:0/4184917360 conn(0x7fe6340ce5f0 msgr2=0x7fe6340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:21.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.402 7fe639a39700 1 -- 172.21.15.32:0/4184917360 shutdown_connections 2021-04-09T18:26:21.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.402 7fe639a39700 1 -- 172.21.15.32:0/4184917360 wait complete. 2021-04-09T18:26:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.422 7fe639a39700 1 Processor -- start 2021-04-09T18:26:21.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.422 7fe639a39700 1 -- start start 2021-04-09T18:26:21.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe639a39700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340b42d0 0x7fe634183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe639a39700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6341ac090 0x7fe6341ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe639a39700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe634184d30 0x7fe634185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe639a39700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6340a1a90 con 0x7fe634184d30 2021-04-09T18:26:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe639a39700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6341acab0 con 0x7fe6341ac090 2021-04-09T18:26:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe639a39700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe634185a00 con 0x7fe6340b42d0 2021-04-09T18:26:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe633fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe634184d30 0x7fe634185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe633fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe634184d30 0x7fe634185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53082/0 (socket says 172.21.15.32:53082) 2021-04-09T18:26:21.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe633fff700 1 -- 172.21.15.32:0/4022560304 learned_addr learned my addr 172.21.15.32:0/4022560304 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:21.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe6337fe700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340b42d0 0x7fe634183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe632ffd700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6341ac090 0x7fe6341ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe633fff700 1 -- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340b42d0 msgr2=0x7fe634183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe633fff700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340b42d0 0x7fe634183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe633fff700 1 -- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6341ac090 msgr2=0x7fe6341ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe633fff700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6341ac090 0x7fe6341ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe633fff700 1 -- 172.21.15.32:0/4022560304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe634185750 con 0x7fe634184d30 2021-04-09T18:26:21.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe633fff700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe634184d30 0x7fe634185130 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 rx=0x7fe62c00db80 tx=0x7fe62c009ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:21.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe61bfff700 1 -- 172.21.15.32:0/4022560304 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe62c00e3a0 con 0x7fe634184d30 2021-04-09T18:26:21.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe61bfff700 1 -- 172.21.15.32:0/4022560304 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62c006790 con 0x7fe634184d30 2021-04-09T18:26:21.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.423 7fe639a39700 1 -- 172.21.15.32:0/4022560304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe634185750 con 0x7fe634184d30 2021-04-09T18:26:21.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.424 7fe61bfff700 1 -- 172.21.15.32:0/4022560304 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe62c00e890 con 0x7fe634184d30 2021-04-09T18:26:21.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.424 7fe639a39700 1 -- 172.21.15.32:0/4022560304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe634185750 con 0x7fe634184d30 2021-04-09T18:26:21.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.424 7fe61bfff700 1 -- 172.21.15.32:0/4022560304 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe62c0056b0 con 0x7fe634184d30 2021-04-09T18:26:21.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.424 7fe61bfff700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe614018930 0x7fe61401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.424 7fe61bfff700 1 -- 172.21.15.32:0/4022560304 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fe62c034c90 con 0x7fe634184d30 2021-04-09T18:26:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.424 7fe6337fe700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe614018930 0x7fe61401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.425 7fe6337fe700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe614018930 0x7fe61401ae10 crc :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:21.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.444 7fe639a39700 1 -- 172.21.15.32:0/4022560304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe634182d10 con 0x7fe634184d30 2021-04-09T18:26:21.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.447 7fe61bfff700 1 -- 172.21.15.32:0/4022560304 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe62c0011f0 con 0x7fe634184d30 2021-04-09T18:26:21.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.638 7fe639a39700 1 -- 172.21.15.32:0/4022560304 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe63410d480 con 0x7fe634184d30 2021-04-09T18:26:21.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.638 7fe61bfff700 1 -- 172.21.15.32:0/4022560304 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 271 v271) v1 ==== 95+0+1647 (secure 0 0 0) 0x7fe62c0056b0 con 0x7fe634184d30 2021-04-09T18:26:21.710 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 271 2021-04-09T18:26:21.710 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:21.711 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":271,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"laggy_since":"2021-04-09 18:26:07.577337","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"laggy_since":"2021-04-09 18:26:07.577350","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"laggy_since":"2021-04-09 18:26:07.577362","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269}],"filesystems":[]} 2021-04-09T18:26:21.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 -- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe614018930 msgr2=0x7fe61401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe614018930 0x7fe61401ae10 crc :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 -- 172.21.15.32:0/4022560304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe634184d30 msgr2=0x7fe634185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe634184d30 0x7fe634185130 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 rx=0x7fe62c00db80 tx=0x7fe62c009ee0).stop 2021-04-09T18:26:21.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 -- 172.21.15.32:0/4022560304 shutdown_connections 2021-04-09T18:26:21.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe614018930 0x7fe61401ae10 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe634184d30 0x7fe634185130 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6341ac090 0x7fe6341ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 --2- 172.21.15.32:0/4022560304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6340b42d0 0x7fe634183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 -- 172.21.15.32:0/4022560304 >> 172.21.15.32:0/4022560304 conn(0x7fe63419b890 msgr2=0x7fe6341a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:21.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 -- 172.21.15.32:0/4022560304 shutdown_connections 2021-04-09T18:26:21.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.641 7fe639a39700 1 -- 172.21.15.32:0/4022560304 wait complete. 2021-04-09T18:26:21.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.800 7f1305439700 1 Processor -- start 2021-04-09T18:26:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.800 7f1305439700 1 -- start start 2021-04-09T18:26:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.800 7f1305439700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13000d7ca0 0x7f13000d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.800 7f1305439700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130010d480 con 0x7f130010da00 2021-04-09T18:26:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.800 7f1305439700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13000d86c0 con 0x7f130010d010 2021-04-09T18:26:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.800 7f1305439700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13000d9220 con 0x7f13000d7ca0 2021-04-09T18:26:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12ff87e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f130010da00 0x7f130010de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34444/0 (socket says 172.21.15.32:34444) 2021-04-09T18:26:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12ff87e700 1 -- 172.21.15.32:0/1152520160 learned_addr learned my addr 172.21.15.32:0/1152520160 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe87c700 1 --2- 172.21.15.32:0/1152520160 >> v2:172.21.15.86:3300/0 conn(0x7f13000d7ca0 0x7f13000d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe07b700 1 -- 172.21.15.32:0/1152520160 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606910482 0 0) 0x7f12f8000d30 con 0x7f130010da00 2021-04-09T18:26:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe07b700 1 -- 172.21.15.32:0/1152520160 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ec002fe0 con 0x7f130010da00 2021-04-09T18:26:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe07b700 1 -- 172.21.15.32:0/1152520160 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794621219 0 0) 0x7f12f0001080 con 0x7f130010d010 2021-04-09T18:26:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe07b700 1 -- 172.21.15.32:0/1152520160 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ec003210 con 0x7f130010d010 2021-04-09T18:26:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe87c700 1 -- 172.21.15.32:0/1152520160 >> v1:172.21.15.86:6789/0 conn(0x7f130010d010 legacy=0x7f13000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe87c700 1 -- 172.21.15.32:0/1152520160 >> v1:172.21.15.32:6789/0 conn(0x7f130010da00 legacy=0x7f130010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe87c700 1 -- 172.21.15.32:0/1152520160 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13000d1f70 con 0x7f13000d7ca0 2021-04-09T18:26:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe87c700 1 --2- 172.21.15.32:0/1152520160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13000d7ca0 0x7f13000d80a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f12e8006650 tx=0x7f12e8009a80).ready entity=mon.1 client_cookie=7a038a3894c2a33b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.801 7f12fe07b700 1 -- 172.21.15.32:0/1152520160 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12e80009b0 con 0x7f13000d7ca0 2021-04-09T18:26:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.802 7f12fe07b700 1 -- 172.21.15.32:0/1152520160 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e800c0c0 con 0x7f13000d7ca0 2021-04-09T18:26:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.802 7f12fe07b700 1 -- 172.21.15.32:0/1152520160 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12e800c510 con 0x7f13000d7ca0 2021-04-09T18:26:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.802 7f1305439700 1 -- 172.21.15.32:0/1152520160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13000d7ca0 msgr2=0x7f13000d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.802 7f1305439700 1 --2- 172.21.15.32:0/1152520160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13000d7ca0 0x7f13000d80a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f12e8006650 tx=0x7f12e8009a80).stop 2021-04-09T18:26:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.802 7f1305439700 1 -- 172.21.15.32:0/1152520160 shutdown_connections 2021-04-09T18:26:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.802 7f1305439700 1 --2- 172.21.15.32:0/1152520160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13000d7ca0 0x7f13000d80a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.802 7f1305439700 1 -- 172.21.15.32:0/1152520160 >> 172.21.15.32:0/1152520160 conn(0x7f13000ce5f0 msgr2=0x7f13000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.802 7f1305439700 1 -- 172.21.15.32:0/1152520160 shutdown_connections 2021-04-09T18:26:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.802 7f1305439700 1 -- 172.21.15.32:0/1152520160 wait complete. 2021-04-09T18:26:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.822 7f1305439700 1 Processor -- start 2021-04-09T18:26:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.822 7f1305439700 1 -- start start 2021-04-09T18:26:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f1305439700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13000a7f50 0x7f1300180be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f1305439700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1300181200 0x7f130008ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f1305439700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130008b3b0 0x7f13001a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f1305439700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130008a890 con 0x7f13000a7f50 2021-04-09T18:26:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f1305439700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1300181640 con 0x7f130008b3b0 2021-04-09T18:26:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f1305439700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13001819f0 con 0x7f1300181200 2021-04-09T18:26:21.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12ff07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13000a7f50 0x7f1300180be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12ff07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13000a7f50 0x7f1300180be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53090/0 (socket says 172.21.15.32:53090) 2021-04-09T18:26:21.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12ff07d700 1 -- 172.21.15.32:0/643041034 learned_addr learned my addr 172.21.15.32:0/643041034 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:21.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12ff87e700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130008b3b0 0x7f13001a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12fe87c700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1300181200 0x7f130008ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12ff07d700 1 -- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1300181200 msgr2=0x7f130008ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12ff07d700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1300181200 0x7f130008ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12ff07d700 1 -- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130008b3b0 msgr2=0x7f13001a32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:21.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12ff07d700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130008b3b0 0x7f13001a32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.823 7f12ff07d700 1 -- 172.21.15.32:0/643041034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130008b7b0 con 0x7f13000a7f50 2021-04-09T18:26:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.824 7f12ff07d700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13000a7f50 0x7f1300180be0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 rx=0x7f12f0006f00 tx=0x7f12f000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.824 7f12f7fff700 1 -- 172.21.15.32:0/643041034 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12f000bd30 con 0x7f13000a7f50 2021-04-09T18:26:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.824 7f12f7fff700 1 -- 172.21.15.32:0/643041034 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f0000e20 con 0x7f13000a7f50 2021-04-09T18:26:21.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.824 7f1305439700 1 -- 172.21.15.32:0/643041034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130008b7b0 con 0x7f13000a7f50 2021-04-09T18:26:21.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.824 7f12f7fff700 1 -- 172.21.15.32:0/643041034 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12f00054f0 con 0x7f13000a7f50 2021-04-09T18:26:21.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.824 7f1305439700 1 -- 172.21.15.32:0/643041034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130008b7b0 con 0x7f13000a7f50 2021-04-09T18:26:21.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.824 7f12f7fff700 1 -- 172.21.15.32:0/643041034 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f12f0006240 con 0x7f13000a7f50 2021-04-09T18:26:21.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.825 7f12f7fff700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e4018930 0x7f12e401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:21.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.825 7f12f7fff700 1 -- 172.21.15.32:0/643041034 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f12f0032750 con 0x7f13000a7f50 2021-04-09T18:26:21.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.825 7f12fe87c700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e4018930 0x7f12e401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:21.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.825 7f12fe87c700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e4018930 0x7f12e401ae10 crc :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:21.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.845 7f1305439700 1 -- 172.21.15.32:0/643041034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13001b7070 con 0x7f13000a7f50 2021-04-09T18:26:21.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:21.848 7f12f7fff700 1 -- 172.21.15.32:0/643041034 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f12f0006030 con 0x7f13000a7f50 2021-04-09T18:26:22.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.039 7f1305439700 1 -- 172.21.15.32:0/643041034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f130008a810 con 0x7f13000a7f50 2021-04-09T18:26:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.039 7f12f7fff700 1 -- 172.21.15.32:0/643041034 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 271 v271) v1 ==== 95+0+1647 (secure 0 0 0) 0x7f12f0000e20 con 0x7f13000a7f50 2021-04-09T18:26:22.111 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 271 2021-04-09T18:26:22.111 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:22.112 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":271,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1185588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1185588699},{"type":"v1","addr":"172.21.15.86:6837","nonce":1185588699}]},"laggy_since":"2021-04-09 18:26:07.577337","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1755736147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1755736147},{"type":"v1","addr":"172.21.15.32:6833","nonce":1755736147}]},"laggy_since":"2021-04-09 18:26:07.577350","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4049162379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4049162379},{"type":"v1","addr":"172.21.15.32:6835","nonce":4049162379}]},"laggy_since":"2021-04-09 18:26:07.577362","export_targets":[],"features":4611087854035861503,"flags":0,"epoch":269}],"filesystems":[]} 2021-04-09T18:26:22.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 -- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e4018930 msgr2=0x7f12e401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:22.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e4018930 0x7f12e401ae10 crc :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 -- 172.21.15.32:0/643041034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13000a7f50 msgr2=0x7f1300180be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:22.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13000a7f50 0x7f1300180be0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 rx=0x7f12f0006f00 tx=0x7f12f000b940).stop 2021-04-09T18:26:22.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 -- 172.21.15.32:0/643041034 shutdown_connections 2021-04-09T18:26:22.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e4018930 0x7f12e401ae10 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130008b3b0 0x7f13001a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1300181200 0x7f130008ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 --2- 172.21.15.32:0/643041034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13000a7f50 0x7f1300180be0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 -- 172.21.15.32:0/643041034 >> 172.21.15.32:0/643041034 conn(0x7f13001aa2b0 msgr2=0x7f13001a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:22.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 -- 172.21.15.32:0/643041034 shutdown_connections 2021-04-09T18:26:22.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.042 7f1305439700 1 -- 172.21.15.32:0/643041034 wait complete. 2021-04-09T18:26:22.124 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:26:22.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:26:22.125 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:26:22.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:26:22.126 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:26:22.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:26:22.126 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:26:22.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:26:22.129 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:26:22.130 DEBUG:teuthology.parallel:result is None 2021-04-09T18:26:22.131 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:26:22.131 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:26:22.132 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:26:22.132 DEBUG:teuthology.parallel:result is None 2021-04-09T18:26:22.133 DEBUG:teuthology.parallel:result is None 2021-04-09T18:26:22.133 DEBUG:teuthology.parallel:result is None 2021-04-09T18:26:22.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:26:22.270 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:26:22.271 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:26:22.282 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:26:22.283 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:26:22.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.219 7fd805184700 1 Processor -- start 2021-04-09T18:26:22.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.219 7fd805184700 1 -- start start 2021-04-09T18:26:22.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.219 7fd805184700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd8000d6c80 0x7fd8000d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:22.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.219 7fd805184700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8000d2440 0x7fd80010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:22.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.219 7fd805184700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8000d7280 con 0x7fd8000cffa0 2021-04-09T18:26:22.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.219 7fd805184700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8000d7080 con 0x7fd8000d2440 2021-04-09T18:26:22.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.219 7fd805184700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8000d91e0 con 0x7fd8000d6c80 2021-04-09T18:26:22.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7ff59e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8000d2440 0x7fd80010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:22.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7ff59e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8000d2440 0x7fd80010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53126/0 (socket says 172.21.15.32:53126) 2021-04-09T18:26:22.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7ff59e700 1 -- 172.21.15.32:0/1702634858 learned_addr learned my addr 172.21.15.32:0/1702634858 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:22.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7fe59c700 1 --2- 172.21.15.32:0/1702634858 >> v2:172.21.15.86:3300/0 conn(0x7fd8000d6c80 0x7fd8000d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:22.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7ff59e700 1 -- 172.21.15.32:0/1702634858 >> v2:172.21.15.86:3300/0 conn(0x7fd8000d6c80 msgr2=0x7fd8000d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:22.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7ff59e700 1 --2- 172.21.15.32:0/1702634858 >> v2:172.21.15.86:3300/0 conn(0x7fd8000d6c80 0x7fd8000d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7ff59e700 1 -- 172.21.15.32:0/1702634858 >> v1:172.21.15.86:6789/0 conn(0x7fd8000cffa0 legacy=0x7fd8000d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:22.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7ff59e700 1 -- 172.21.15.32:0/1702634858 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8001120c0 con 0x7fd8000d2440 2021-04-09T18:26:22.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7fe59c700 1 --2- 172.21.15.32:0/1702634858 >> v2:172.21.15.86:3300/0 conn(0x7fd8000d6c80 0x7fd8000d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7ff59e700 1 --2- 172.21.15.32:0/1702634858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8000d2440 0x7fd80010cfd0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 rx=0x7fd7f0006b80 tx=0x7fd7f0009b90).ready entity=mon.0 client_cookie=475ed00ea5e0df7f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7fdd9b700 1 -- 172.21.15.32:0/1702634858 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7f00022e0 con 0x7fd8000d2440 2021-04-09T18:26:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7fdd9b700 1 -- 172.21.15.32:0/1702634858 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f0002620 con 0x7fd8000d2440 2021-04-09T18:26:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd7fdd9b700 1 -- 172.21.15.32:0/1702634858 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7f0002a80 con 0x7fd8000d2440 2021-04-09T18:26:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd805184700 1 -- 172.21.15.32:0/1702634858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8000d2440 msgr2=0x7fd80010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.220 7fd805184700 1 --2- 172.21.15.32:0/1702634858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8000d2440 0x7fd80010cfd0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 rx=0x7fd7f0006b80 tx=0x7fd7f0009b90).stop 2021-04-09T18:26:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.221 7fd805184700 1 -- 172.21.15.32:0/1702634858 shutdown_connections 2021-04-09T18:26:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.221 7fd805184700 1 --2- 172.21.15.32:0/1702634858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8000d2440 0x7fd80010cfd0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.221 7fd805184700 1 --2- 172.21.15.32:0/1702634858 >> v2:172.21.15.86:3300/0 conn(0x7fd8000d6c80 0x7fd8000d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.221 7fd805184700 1 -- 172.21.15.32:0/1702634858 >> 172.21.15.32:0/1702634858 conn(0x7fd8000ce5c0 msgr2=0x7fd8000cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.221 7fd805184700 1 -- 172.21.15.32:0/1702634858 shutdown_connections 2021-04-09T18:26:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.221 7fd805184700 1 -- 172.21.15.32:0/1702634858 wait complete. 2021-04-09T18:26:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.241 7fd805184700 1 Processor -- start 2021-04-09T18:26:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.241 7fd805184700 1 -- start start 2021-04-09T18:26:22.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd805184700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd800184070 0x7fd8001955d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd805184700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd800195bf0 0x7fd80008ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd805184700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80008f550 0x7fd80008f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd805184700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800090570 con 0x7fd800184070 2021-04-09T18:26:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd805184700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800184470 con 0x7fd80008f550 2021-04-09T18:26:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd805184700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd800195ff0 con 0x7fd800195bf0 2021-04-09T18:26:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7fed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd800184070 0x7fd8001955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7fed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd800184070 0x7fd8001955d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53130/0 (socket says 172.21.15.32:53130) 2021-04-09T18:26:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7fed9d700 1 -- 172.21.15.32:0/1892169619 learned_addr learned my addr 172.21.15.32:0/1892169619 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7fe59c700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd800195bf0 0x7fd80008ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7ff59e700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80008f550 0x7fd80008f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7fed9d700 1 -- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd800195bf0 msgr2=0x7fd80008ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7fed9d700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd800195bf0 0x7fd80008ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7fed9d700 1 -- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80008f550 msgr2=0x7fd80008f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7fed9d700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80008f550 0x7fd80008f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.242 7fd7fed9d700 1 -- 172.21.15.32:0/1892169619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8001847a0 con 0x7fd800184070 2021-04-09T18:26:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.243 7fd7fed9d700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd800184070 0x7fd8001955d0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7fd7f8009940 tx=0x7fd7f800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.243 7fd7f77fe700 1 -- 172.21.15.32:0/1892169619 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7f8000ce0 con 0x7fd800184070 2021-04-09T18:26:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.243 7fd7f77fe700 1 -- 172.21.15.32:0/1892169619 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f800c2f0 con 0x7fd800184070 2021-04-09T18:26:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.243 7fd805184700 1 -- 172.21.15.32:0/1892169619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8001847a0 con 0x7fd800184070 2021-04-09T18:26:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.243 7fd7f77fe700 1 -- 172.21.15.32:0/1892169619 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7f800c7d0 con 0x7fd800184070 2021-04-09T18:26:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.243 7fd805184700 1 -- 172.21.15.32:0/1892169619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8001847a0 con 0x7fd800184070 2021-04-09T18:26:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.243 7fd7f77fe700 1 -- 172.21.15.32:0/1892169619 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd7f8001230 con 0x7fd800184070 2021-04-09T18:26:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.243 7fd7f77fe700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7e4018930 0x7fd7e401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.244 7fd7f77fe700 1 -- 172.21.15.32:0/1892169619 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fd7f8032b40 con 0x7fd800184070 2021-04-09T18:26:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.244 7fd7fe59c700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7e4018930 0x7fd7e401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.244 7fd7fe59c700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7e4018930 0x7fd7e401ae10 crc :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:22.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.263 7fd805184700 1 -- 172.21.15.32:0/1892169619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd800182e70 con 0x7fd800184070 2021-04-09T18:26:22.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.266 7fd7f77fe700 1 -- 172.21.15.32:0/1892169619 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd7f800be30 con 0x7fd800184070 2021-04-09T18:26:22.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.468 7fd805184700 1 -- 172.21.15.32:0/1892169619 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fd8000d7280 con 0x7fd800184070 2021-04-09T18:26:23.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.954 7fd7f77fe700 1 -- 172.21.15.32:0/1892169619 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v135) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd7f8001230 con 0x7fd800184070 2021-04-09T18:26:23.053 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:26:23.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 -- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7e4018930 msgr2=0x7fd7e401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7e4018930 0x7fd7e401ae10 crc :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 -- 172.21.15.32:0/1892169619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd800184070 msgr2=0x7fd8001955d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd800184070 0x7fd8001955d0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7fd7f8009940 tx=0x7fd7f800bcf0).stop 2021-04-09T18:26:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 -- 172.21.15.32:0/1892169619 shutdown_connections 2021-04-09T18:26:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7e4018930 0x7fd7e401ae10 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd80008f550 0x7fd80008f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd800195bf0 0x7fd80008ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 --2- 172.21.15.32:0/1892169619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd800184070 0x7fd8001955d0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 -- 172.21.15.32:0/1892169619 >> 172.21.15.32:0/1892169619 conn(0x7fd8001a8f40 msgr2=0x7fd800188050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 -- 172.21.15.32:0/1892169619 shutdown_connections 2021-04-09T18:26:23.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:22.985 7fd805184700 1 -- 172.21.15.32:0/1892169619 wait complete. 2021-04-09T18:26:23.067 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:26:23.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:26:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.148 7f7d11ec6700 1 Processor -- start 2021-04-09T18:26:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.148 7f7d11ec6700 1 -- start start 2021-04-09T18:26:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.148 7f7d11ec6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7d0c10d010 0x7f7d0c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.148 7f7d11ec6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d0c0d8490 0x7f7d0c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d11ec6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c10d480 con 0x7f7d0c0d1da0 2021-04-09T18:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d11ec6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d0c0d8d30 con 0x7f7d0c0d8490 2021-04-09T18:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d11ec6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d0c0d9220 con 0x7f7d0c10d010 2021-04-09T18:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d0c0d8490 0x7f7d0c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0affd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7d0c0d1da0 0x7f7d0c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34498/0 (socket says 172.21.15.32:34498) 2021-04-09T18:26:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0affd700 1 -- 172.21.15.32:0/3337398773 learned_addr learned my addr 172.21.15.32:0/3337398773 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0b7fe700 1 --2- 172.21.15.32:0/3337398773 >> v2:172.21.15.86:3300/0 conn(0x7f7d0c10d010 0x7f7d0c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0bfff700 1 -- 172.21.15.32:0/3337398773 >> v2:172.21.15.86:3300/0 conn(0x7f7d0c10d010 msgr2=0x7f7d0c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:23.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0bfff700 1 --2- 172.21.15.32:0/3337398773 >> v2:172.21.15.86:3300/0 conn(0x7f7d0c10d010 0x7f7d0c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0bfff700 1 -- 172.21.15.32:0/3337398773 >> v1:172.21.15.32:6789/0 conn(0x7f7d0c0d1da0 legacy=0x7f7d0c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0bfff700 1 -- 172.21.15.32:0/3337398773 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c10e020 con 0x7f7d0c0d8490 2021-04-09T18:26:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0a7fc700 1 -- 172.21.15.32:0/3337398773 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506039737 0 0) 0x7f7cf8000ce0 con 0x7f7d0c0d1da0 2021-04-09T18:26:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0b7fe700 1 --2- 172.21.15.32:0/3337398773 >> v2:172.21.15.86:3300/0 conn(0x7f7d0c10d010 0x7f7d0c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:23.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0bfff700 1 --2- 172.21.15.32:0/3337398773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c0d8490 0x7f7d0c10da00 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 rx=0x7f7d04006e30 tx=0x7f7d04009e60).ready entity=mon.0 client_cookie=486822a837d74d6e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:23.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0a7fc700 1 -- 172.21.15.32:0/3337398773 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d04006a50 con 0x7f7d0c0d8490 2021-04-09T18:26:23.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0a7fc700 1 -- 172.21.15.32:0/3337398773 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0400b300 con 0x7f7d0c0d8490 2021-04-09T18:26:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d0a7fc700 1 -- 172.21.15.32:0/3337398773 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d0400b750 con 0x7f7d0c0d8490 2021-04-09T18:26:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d11ec6700 1 -- 172.21.15.32:0/3337398773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c0d8490 msgr2=0x7f7d0c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.149 7f7d11ec6700 1 --2- 172.21.15.32:0/3337398773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c0d8490 0x7f7d0c10da00 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 rx=0x7f7d04006e30 tx=0x7f7d04009e60).stop 2021-04-09T18:26:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.150 7f7d11ec6700 1 -- 172.21.15.32:0/3337398773 shutdown_connections 2021-04-09T18:26:23.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.150 7f7d11ec6700 1 --2- 172.21.15.32:0/3337398773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c0d8490 0x7f7d0c10da00 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.150 7f7d11ec6700 1 --2- 172.21.15.32:0/3337398773 >> v2:172.21.15.86:3300/0 conn(0x7f7d0c10d010 0x7f7d0c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.150 7f7d11ec6700 1 -- 172.21.15.32:0/3337398773 >> 172.21.15.32:0/3337398773 conn(0x7f7d0c0ce5f0 msgr2=0x7f7d0c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.150 7f7d11ec6700 1 -- 172.21.15.32:0/3337398773 shutdown_connections 2021-04-09T18:26:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.150 7f7d11ec6700 1 -- 172.21.15.32:0/3337398773 wait complete. 2021-04-09T18:26:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.170 7f7d11ec6700 1 Processor -- start 2021-04-09T18:26:23.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.170 7f7d11ec6700 1 -- start start 2021-04-09T18:26:23.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d11ec6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d0c1aca20 0x7f7d0c1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d11ec6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c18bd40 0x7f7d0c18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d11ec6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d0c08fba0 0x7f7d0c08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d11ec6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c18c7e0 con 0x7f7d0c18bd40 2021-04-09T18:26:23.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d11ec6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c18ca30 con 0x7f7d0c08fba0 2021-04-09T18:26:23.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d11ec6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c0905f0 con 0x7f7d0c1aca20 2021-04-09T18:26:23.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c18bd40 0x7f7d0c18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c18bd40 0x7f7d0c18c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53142/0 (socket says 172.21.15.32:53142) 2021-04-09T18:26:23.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0affd700 1 -- 172.21.15.32:0/3553576315 learned_addr learned my addr 172.21.15.32:0/3553576315 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:23.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0bfff700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d0c08fba0 0x7f7d0c08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0b7fe700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d0c1aca20 0x7f7d0c1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0affd700 1 -- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d0c1aca20 msgr2=0x7f7d0c1ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0affd700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d0c1aca20 0x7f7d0c1ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0affd700 1 -- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d0c08fba0 msgr2=0x7f7d0c08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0affd700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d0c08fba0 0x7f7d0c08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0affd700 1 -- 172.21.15.32:0/3553576315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c0b4300 con 0x7f7d0c18bd40 2021-04-09T18:26:23.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.171 7f7d0bfff700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d0c08fba0 0x7f7d0c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:26:23.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.172 7f7d0affd700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c18bd40 0x7f7d0c18c140 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7f7cf8006f00 tx=0x7f7cf800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:23.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.172 7f7d03fff700 1 -- 172.21.15.32:0/3553576315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf8006020 con 0x7f7d0c18bd40 2021-04-09T18:26:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.172 7f7d03fff700 1 -- 172.21.15.32:0/3553576315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf80063a0 con 0x7f7d0c18bd40 2021-04-09T18:26:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.172 7f7d03fff700 1 -- 172.21.15.32:0/3553576315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf8004620 con 0x7f7d0c18bd40 2021-04-09T18:26:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.172 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c0b4300 con 0x7f7d0c18bd40 2021-04-09T18:26:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.172 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0c0b4300 con 0x7f7d0c18bd40 2021-04-09T18:26:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.172 7f7d03fff700 1 -- 172.21.15.32:0/3553576315 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7cf800a000 con 0x7f7d0c18bd40 2021-04-09T18:26:23.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.172 7f7d03fff700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ce80188d0 0x7f7ce801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.172 7f7d03fff700 1 -- 172.21.15.32:0/3553576315 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f7cf80326b0 con 0x7f7d0c18bd40 2021-04-09T18:26:23.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.173 7f7d0b7fe700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ce80188d0 0x7f7ce801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.173 7f7d0b7fe700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ce80188d0 0x7f7ce801add0 crc :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.193 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0c182d70 con 0x7f7d0c18bd40 2021-04-09T18:26:23.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.196 7f7d03fff700 1 -- 172.21.15.32:0/3553576315 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7cf8006b90 con 0x7f7d0c18bd40 2021-04-09T18:26:23.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.382 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7d0c18c760 con 0x7f7d0c18bd40 2021-04-09T18:26:23.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.382 7f7d03fff700 1 -- 172.21.15.32:0/3553576315 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v33) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f7cf800a000 con 0x7f7d0c18bd40 2021-04-09T18:26:23.454 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:23.455 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:26:23.455 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:26:23.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.456 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:26:23.456 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:26:23.456 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.457 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:26:23.457 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.457 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.458 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.459 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:26:23.459 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:26:23.460 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.460 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:26:23.460 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.461 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.461 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.461 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.462 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.462 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:26:23.463 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:26:23.463 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.463 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:26:23.464 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.464 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.464 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.465 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.466 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:26:23.466 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:26:23.466 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.467 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:26:23.467 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.467 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.468 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.468 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.469 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:26:23.469 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:26:23.470 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.470 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.470 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.471 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.472 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:26:23.472 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:26:23.473 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.473 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.473 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.474 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.474 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.475 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:26:23.476 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:26:23.476 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.476 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.477 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.477 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.477 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.478 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:26:23.479 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:26:23.479 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.479 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.480 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.480 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.480 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.482 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:26:23.482 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:26:23.483 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.483 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.483 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.484 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.484 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.485 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:26:23.485 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:26:23.486 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.486 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.486 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.487 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.488 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:26:23.488 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:26:23.489 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.489 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.489 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.489 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.490 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.490 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.490 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:26:23.491 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:26:23.491 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.491 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.491 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.492 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.492 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.493 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:26:23.493 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:26:23.493 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.494 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:26:23.494 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:26:23.494 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:26:23.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.495 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:26:23.495 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:26:23.496 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.496 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:26:23.496 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:26:23.497 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:26:23.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.498 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:26:23.498 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:26:23.498 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.498 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:26:23.499 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:26:23.499 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:26:23.499 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.499 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.500 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:26:23.500 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:26:23.501 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.501 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:26:23.501 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:26:23.501 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.502 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:23.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:23.502 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:26:23.503 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:26:23.503 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:26:23.503 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:26:23.503 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:26:23.504 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:26:23.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.504 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:23.504 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:26:23.504 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:26:23.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ce80188d0 msgr2=0x7f7ce801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ce80188d0 0x7f7ce801add0 crc :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c18bd40 msgr2=0x7f7d0c18c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c18bd40 0x7f7d0c18c140 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7f7cf8006f00 tx=0x7f7cf800b940).stop 2021-04-09T18:26:23.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 shutdown_connections 2021-04-09T18:26:23.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ce80188d0 0x7f7ce801add0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d0c18bd40 0x7f7d0c18c140 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d0c08fba0 0x7f7d0c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 --2- 172.21.15.32:0/3553576315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d0c1aca20 0x7f7d0c1ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 >> 172.21.15.32:0/3553576315 conn(0x7f7d0c19b460 msgr2=0x7f7d0c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:23.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 shutdown_connections 2021-04-09T18:26:23.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.385 7f7d11ec6700 1 -- 172.21.15.32:0/3553576315 wait complete. 2021-04-09T18:26:23.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:26:23.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.587 7fd7c9a40700 1 Processor -- start 2021-04-09T18:26:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.587 7fd7c9a40700 1 -- start start 2021-04-09T18:26:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c9a40700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c40d7ca0 0x7fd7c40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c9a40700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c410d480 con 0x7fd7c410da00 2021-04-09T18:26:23.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c9a40700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40d86c0 con 0x7fd7c410d010 2021-04-09T18:26:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c9a40700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40d9220 con 0x7fd7c40d7ca0 2021-04-09T18:26:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c3fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd7c410da00 0x7fd7c410de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34508/0 (socket says 172.21.15.32:34508) 2021-04-09T18:26:23.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c3fff700 1 -- 172.21.15.32:0/495088395 learned_addr learned my addr 172.21.15.32:0/495088395 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c2ffd700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c40d7ca0 0x7fd7c40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021001135 0 0) 0x7fd7bc000d30 con 0x7fd7c410da00 2021-04-09T18:26:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7ac002fe0 con 0x7fd7c410da00 2021-04-09T18:26:23.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010334666 0 0) 0x7fd7b4000c60 con 0x7fd7c410d010 2021-04-09T18:26:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7ac003210 con 0x7fd7c410d010 2021-04-09T18:26:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3557991898 0 0) 0x7fd7bc001270 con 0x7fd7c410da00 2021-04-09T18:26:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c40d7ca0 msgr2=0x7fd7c40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c27fc700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c40d7ca0 0x7fd7c40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.588 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 >> v1:172.21.15.86:6789/0 conn(0x7fd7c410d010 legacy=0x7fd7c40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40d1f70 con 0x7fd7c410da00 2021-04-09T18:26:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fd7bc001980 con 0x7fd7c410da00 2021-04-09T18:26:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 >> v1:172.21.15.32:6789/0 conn(0x7fd7c410da00 legacy=0x7fd7c410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac003b90 0x7fd7ac005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7ac0067a0 0x7fd7ac008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac009510 0x7fd7ac00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0502f0 con 0x7fd7ac003b90 2021-04-09T18:26:23.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0504d0 con 0x7fd7ac009510 2021-04-09T18:26:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c37fe700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac003b90 0x7fd7ac005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac0506f0 con 0x7fd7ac0067a0 2021-04-09T18:26:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c2ffd700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7ac0067a0 0x7fd7ac008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c3fff700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac009510 0x7fd7ac00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c37fe700 1 -- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7ac0067a0 msgr2=0x7fd7ac008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c37fe700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7ac0067a0 0x7fd7ac008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c37fe700 1 -- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac009510 msgr2=0x7fd7ac00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c37fe700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7ac009510 0x7fd7ac00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c37fe700 1 -- 172.21.15.32:0/495088395 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac050990 con 0x7fd7ac003b90 2021-04-09T18:26:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c37fe700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac003b90 0x7fd7ac005fa0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7fd7b4001040 tx=0x7fd7b4009aa0).ready entity=mon.0 client_cookie=e29057475a664a0e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7b40066e0 con 0x7fd7ac003b90 2021-04-09T18:26:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b400c0c0 con 0x7fd7ac003b90 2021-04-09T18:26:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c27fc700 1 -- 172.21.15.32:0/495088395 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7b400c560 con 0x7fd7ac003b90 2021-04-09T18:26:23.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c9a40700 1 -- 172.21.15.32:0/495088395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac003b90 msgr2=0x7fd7ac005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c9a40700 1 --2- 172.21.15.32:0/495088395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7ac003b90 0x7fd7ac005fa0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7fd7b4001040 tx=0x7fd7b4009aa0).stop 2021-04-09T18:26:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.589 7fd7c3fff700 1 -- 172.21.15.32:0/495088395 reap_dead start 2021-04-09T18:26:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.590 7fd7c9a40700 1 -- 172.21.15.32:0/495088395 shutdown_connections 2021-04-09T18:26:23.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.590 7fd7c9a40700 1 -- 172.21.15.32:0/495088395 >> 172.21.15.32:0/495088395 conn(0x7fd7c40ce5f0 msgr2=0x7fd7c40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.590 7fd7c9a40700 1 -- 172.21.15.32:0/495088395 shutdown_connections 2021-04-09T18:26:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.590 7fd7c9a40700 1 -- 172.21.15.32:0/495088395 wait complete. 2021-04-09T18:26:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.610 7fd7c9a40700 1 Processor -- start 2021-04-09T18:26:23.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c9a40700 1 -- start start 2021-04-09T18:26:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c9a40700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c4185060 0x7fd7c4185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c9a40700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c4186c00 0x7fd7c4187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c9a40700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4191360 0x7fd7c4191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c9a40700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c41863a0 con 0x7fd7c4185060 2021-04-09T18:26:23.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c9a40700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c4185a60 con 0x7fd7c4191360 2021-04-09T18:26:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c9a40700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c4187620 con 0x7fd7c4186c00 2021-04-09T18:26:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c4185060 0x7fd7c4185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c37fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c4185060 0x7fd7c4185d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53162/0 (socket says 172.21.15.32:53162) 2021-04-09T18:26:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c37fe700 1 -- 172.21.15.32:0/3142609442 learned_addr learned my addr 172.21.15.32:0/3142609442 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:23.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c2ffd700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c4186c00 0x7fd7c4187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.611 7fd7c3fff700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4191360 0x7fd7c4191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7c37fe700 1 -- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c4186c00 msgr2=0x7fd7c4187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7c37fe700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c4186c00 0x7fd7c4187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7c37fe700 1 -- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4191360 msgr2=0x7fd7c4191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7c37fe700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4191360 0x7fd7c4191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7c37fe700 1 -- 172.21.15.32:0/3142609442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c4185460 con 0x7fd7c4185060 2021-04-09T18:26:23.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7c37fe700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c4185060 0x7fd7c4185d80 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7fd7b4006e90 tx=0x7fd7b400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:23.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7abfff700 1 -- 172.21.15.32:0/3142609442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7b40091b0 con 0x7fd7c4185060 2021-04-09T18:26:23.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7abfff700 1 -- 172.21.15.32:0/3142609442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b4006030 con 0x7fd7c4185060 2021-04-09T18:26:23.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c4185460 con 0x7fd7c4185060 2021-04-09T18:26:23.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7abfff700 1 -- 172.21.15.32:0/3142609442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7b4005540 con 0x7fd7c4185060 2021-04-09T18:26:23.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.612 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c4185460 con 0x7fd7c4185060 2021-04-09T18:26:23.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.613 7fd7abfff700 1 -- 172.21.15.32:0/3142609442 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd7b4006210 con 0x7fd7c4185060 2021-04-09T18:26:23.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.613 7fd7abfff700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7a4018930 0x7fd7a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:23.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.613 7fd7abfff700 1 -- 172.21.15.32:0/3142609442 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fd7b4032800 con 0x7fd7c4185060 2021-04-09T18:26:23.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.613 7fd7c2ffd700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7a4018930 0x7fd7a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:23.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.613 7fd7c2ffd700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7a4018930 0x7fd7a401ae10 crc :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:23.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.632 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c41866d0 con 0x7fd7c4185060 2021-04-09T18:26:23.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.636 7fd7abfff700 1 -- 172.21.15.32:0/3142609442 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd7b4009430 con 0x7fd7c4185060 2021-04-09T18:26:23.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.824 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7c41866d0 con 0x7fd7c4185060 2021-04-09T18:26:23.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.825 7fd7abfff700 1 -- 172.21.15.32:0/3142609442 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+11154 (secure 0 0 0) 0x7fd7b400bcd0 con 0x7fd7c4185060 2021-04-09T18:26:23.896 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:23.897 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":135,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:26:22.949660","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:26:23.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7a4018930 msgr2=0x7fd7a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7a4018930 0x7fd7a401ae10 crc :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c4185060 msgr2=0x7fd7c4185d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:23.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c4185060 0x7fd7c4185d80 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7fd7b4006e90 tx=0x7fd7b400b8e0).stop 2021-04-09T18:26:23.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 shutdown_connections 2021-04-09T18:26:23.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd7a4018930 0x7fd7a401ae10 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4191360 0x7fd7c4191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c4186c00 0x7fd7c4187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 --2- 172.21.15.32:0/3142609442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7c4185060 0x7fd7c4185d80 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:23.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 >> 172.21.15.32:0/3142609442 conn(0x7fd7c41837d0 msgr2=0x7fd7c4195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:23.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 shutdown_connections 2021-04-09T18:26:23.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.827 7fd7c9a40700 1 -- 172.21.15.32:0/3142609442 wait complete. 2021-04-09T18:26:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:26:23.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:26:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.991 7fc1958de700 1 Processor -- start 2021-04-09T18:26:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.991 7fc1958de700 1 -- start start 2021-04-09T18:26:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.991 7fc1958de700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc1900d7ca0 0x7fc1900d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.991 7fc1958de700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19010da00 0x7fc19010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.991 7fc1958de700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19010d480 con 0x7fc19010d010 2021-04-09T18:26:24.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.991 7fc1958de700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1900d9220 con 0x7fc1900d7ca0 2021-04-09T18:26:24.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.991 7fc1958de700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900d86c0 con 0x7fc19010da00 2021-04-09T18:26:24.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18f574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc19010d010 0x7fc1900d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34528/0 (socket says 172.21.15.32:34528) 2021-04-09T18:26:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18f574700 1 -- 172.21.15.32:0/1304539773 learned_addr learned my addr 172.21.15.32:0/1304539773 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18fd75700 1 --2- 172.21.15.32:0/1304539773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19010da00 0x7fc19010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18ed73700 1 --2- 172.21.15.32:0/1304539773 >> v2:172.21.15.86:3300/0 conn(0x7fc1900d7ca0 0x7fc1900d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18e572700 1 -- 172.21.15.32:0/1304539773 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266752452 0 0) 0x7fc180001060 con 0x7fc19010d010 2021-04-09T18:26:24.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18e572700 1 -- 172.21.15.32:0/1304539773 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc178002fe0 con 0x7fc19010d010 2021-04-09T18:26:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18ed73700 1 -- 172.21.15.32:0/1304539773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19010da00 msgr2=0x7fc19010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18ed73700 1 --2- 172.21.15.32:0/1304539773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19010da00 0x7fc19010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18ed73700 1 -- 172.21.15.32:0/1304539773 >> v1:172.21.15.32:6789/0 conn(0x7fc19010d010 legacy=0x7fc1900d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18ed73700 1 -- 172.21.15.32:0/1304539773 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1900d1f70 con 0x7fc1900d7ca0 2021-04-09T18:26:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18fd75700 1 --2- 172.21.15.32:0/1304539773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19010da00 0x7fc19010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18ed73700 1 --2- 172.21.15.32:0/1304539773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1900d7ca0 0x7fc1900d80a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7fc184006650 tx=0x7fc184009a80).ready entity=mon.1 client_cookie=69c6c625cadc2c15 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18e572700 1 -- 172.21.15.32:0/1304539773 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1840009b0 con 0x7fc1900d7ca0 2021-04-09T18:26:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18e572700 1 -- 172.21.15.32:0/1304539773 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18400c0c0 con 0x7fc1900d7ca0 2021-04-09T18:26:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc18e572700 1 -- 172.21.15.32:0/1304539773 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc18400c510 con 0x7fc1900d7ca0 2021-04-09T18:26:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc1958de700 1 -- 172.21.15.32:0/1304539773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1900d7ca0 msgr2=0x7fc1900d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.992 7fc1958de700 1 --2- 172.21.15.32:0/1304539773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1900d7ca0 0x7fc1900d80a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7fc184006650 tx=0x7fc184009a80).stop 2021-04-09T18:26:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.993 7fc1958de700 1 -- 172.21.15.32:0/1304539773 shutdown_connections 2021-04-09T18:26:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.993 7fc1958de700 1 --2- 172.21.15.32:0/1304539773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19010da00 0x7fc19010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.993 7fc1958de700 1 --2- 172.21.15.32:0/1304539773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1900d7ca0 0x7fc1900d80a0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.993 7fc1958de700 1 -- 172.21.15.32:0/1304539773 >> 172.21.15.32:0/1304539773 conn(0x7fc1900ce5f0 msgr2=0x7fc1900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:24.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.993 7fc1958de700 1 -- 172.21.15.32:0/1304539773 shutdown_connections 2021-04-09T18:26:24.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:23.993 7fc1958de700 1 -- 172.21.15.32:0/1304539773 wait complete. 2021-04-09T18:26:24.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.013 7fc1958de700 1 Processor -- start 2021-04-09T18:26:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.013 7fc1958de700 1 -- start start 2021-04-09T18:26:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc1958de700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc19009c5b0 0x7fc19009d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc1958de700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19009c9b0 0x7fc19009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc1958de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc190090390 0x7fc190090790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc1958de700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19009e290 con 0x7fc190090390 2021-04-09T18:26:24.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc1958de700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19009cf90 con 0x7fc19009c5b0 2021-04-09T18:26:24.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc1958de700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc190090db0 con 0x7fc19009c9b0 2021-04-09T18:26:24.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc190090390 0x7fc190090790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc190090390 0x7fc190090790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53178/0 (socket says 172.21.15.32:53178) 2021-04-09T18:26:24.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18fd75700 1 -- 172.21.15.32:0/1348631626 learned_addr learned my addr 172.21.15.32:0/1348631626 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18f574700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc19009c5b0 0x7fc19009d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18ed73700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19009c9b0 0x7fc19009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18fd75700 1 -- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19009c9b0 msgr2=0x7fc19009db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18fd75700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19009c9b0 0x7fc19009db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18fd75700 1 -- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc19009c5b0 msgr2=0x7fc19009d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18fd75700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc19009c5b0 0x7fc19009d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18fd75700 1 -- 172.21.15.32:0/1348631626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc190090f90 con 0x7fc190090390 2021-04-09T18:26:24.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.014 7fc18fd75700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc190090390 0x7fc190090790 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 rx=0x7fc18800b5e0 tx=0x7fc18800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:24.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.015 7fc177fff700 1 -- 172.21.15.32:0/1348631626 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc188000c10 con 0x7fc190090390 2021-04-09T18:26:24.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.015 7fc177fff700 1 -- 172.21.15.32:0/1348631626 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18800bd70 con 0x7fc190090390 2021-04-09T18:26:24.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.015 7fc1958de700 1 -- 172.21.15.32:0/1348631626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc190090f90 con 0x7fc190090390 2021-04-09T18:26:24.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.015 7fc177fff700 1 -- 172.21.15.32:0/1348631626 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1880024d0 con 0x7fc190090390 2021-04-09T18:26:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.015 7fc1958de700 1 -- 172.21.15.32:0/1348631626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc190090f90 con 0x7fc190090390 2021-04-09T18:26:24.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.015 7fc177fff700 1 -- 172.21.15.32:0/1348631626 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc188006720 con 0x7fc190090390 2021-04-09T18:26:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.015 7fc177fff700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc170018900 0x7fc17001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.015 7fc177fff700 1 -- 172.21.15.32:0/1348631626 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fc188032700 con 0x7fc190090390 2021-04-09T18:26:24.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.016 7fc18f574700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc170018900 0x7fc17001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.016 7fc18f574700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc170018900 0x7fc17001adc0 crc :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:24.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.036 7fc1958de700 1 -- 172.21.15.32:0/1348631626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19010d480 con 0x7fc190090390 2021-04-09T18:26:24.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.039 7fc177fff700 1 -- 172.21.15.32:0/1348631626 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc188006370 con 0x7fc190090390 2021-04-09T18:26:24.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.220 7fc1958de700 1 -- 172.21.15.32:0/1348631626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc19010d480 con 0x7fc190090390 2021-04-09T18:26:24.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.221 7fc177fff700 1 -- 172.21.15.32:0/1348631626 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v135) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc18800bd70 con 0x7fc190090390 2021-04-09T18:26:24.292 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:26:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 -- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc170018900 msgr2=0x7fc17001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc170018900 0x7fc17001adc0 crc :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 -- 172.21.15.32:0/1348631626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc190090390 msgr2=0x7fc190090790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc190090390 0x7fc190090790 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 rx=0x7fc18800b5e0 tx=0x7fc18800b980).stop 2021-04-09T18:26:24.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 -- 172.21.15.32:0/1348631626 shutdown_connections 2021-04-09T18:26:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc170018900 0x7fc17001adc0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc190090390 0x7fc190090790 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc19009c9b0 0x7fc19009db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 --2- 172.21.15.32:0/1348631626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc19009c5b0 0x7fc19009d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 -- 172.21.15.32:0/1348631626 >> 172.21.15.32:0/1348631626 conn(0x7fc1900b2fa0 msgr2=0x7fc1900a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 -- 172.21.15.32:0/1348631626 shutdown_connections 2021-04-09T18:26:24.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.223 7fc1958de700 1 -- 172.21.15.32:0/1348631626 wait complete. 2021-04-09T18:26:24.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:26:24.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.382 7fb8cc38d700 1 Processor -- start 2021-04-09T18:26:24.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.382 7fb8cc38d700 1 -- start start 2021-04-09T18:26:24.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.382 7fb8cc38d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c40cffa0 0x7fb8c40d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.382 7fb8cc38d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb8c40d7800 0x7fb8c40d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.382 7fb8cc38d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb8c40d5c70 0x7fb8c40d6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.382 7fb8cc38d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c40d8580 con 0x7fb8c40d7800 2021-04-09T18:26:24.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.382 7fb8cc38d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c40d91e0 con 0x7fb8c40d5c70 2021-04-09T18:26:24.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.382 7fb8cc38d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c40d7280 con 0x7fb8c40cffa0 2021-04-09T18:26:24.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c99a8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb8c40d7800 0x7fb8c40d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c99a8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb8c40d7800 0x7fb8c40d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53188/0 (socket says 172.21.15.32:53188) 2021-04-09T18:26:24.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c99a8700 1 -- 172.21.15.32:0/1986464654 learned_addr learned my addr 172.21.15.32:0/1986464654 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:24.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8ca1a9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c40cffa0 0x7fb8c40d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8ca9aa700 1 --2- 172.21.15.32:0/1986464654 >> v2:172.21.15.86:3300/0 conn(0x7fb8c40d5c70 0x7fb8c40d6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c99a8700 1 -- 172.21.15.32:0/1986464654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c40cffa0 msgr2=0x7fb8c40d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:24.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c99a8700 1 --2- 172.21.15.32:0/1986464654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c40cffa0 0x7fb8c40d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c99a8700 1 -- 172.21.15.32:0/1986464654 >> v2:172.21.15.86:3300/0 conn(0x7fb8c40d5c70 msgr2=0x7fb8c40d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:24.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c99a8700 1 --2- 172.21.15.32:0/1986464654 >> v2:172.21.15.86:3300/0 conn(0x7fb8c40d5c70 0x7fb8c40d6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c99a8700 1 -- 172.21.15.32:0/1986464654 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c40d8220 con 0x7fb8c40d7800 2021-04-09T18:26:24.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8ca1a9700 1 --2- 172.21.15.32:0/1986464654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c40cffa0 0x7fb8c40d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:24.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8ca9aa700 1 --2- 172.21.15.32:0/1986464654 >> v2:172.21.15.86:3300/0 conn(0x7fb8c40d5c70 0x7fb8c40d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:24.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c99a8700 1 --2- 172.21.15.32:0/1986464654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c40d7800 0x7fb8c40d7c00 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7fb8b8006a80 tx=0x7fb8b8009f00).ready entity=mon.0 client_cookie=d40cec10688c1845 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:24.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c91a7700 1 -- 172.21.15.32:0/1986464654 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b800b5c0 con 0x7fb8c40d7800 2021-04-09T18:26:24.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c91a7700 1 -- 172.21.15.32:0/1986464654 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b800b8c0 con 0x7fb8c40d7800 2021-04-09T18:26:24.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8c91a7700 1 -- 172.21.15.32:0/1986464654 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b800bda0 con 0x7fb8c40d7800 2021-04-09T18:26:24.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8cc38d700 1 -- 172.21.15.32:0/1986464654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c40d7800 msgr2=0x7fb8c40d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.383 7fb8cc38d700 1 --2- 172.21.15.32:0/1986464654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c40d7800 0x7fb8c40d7c00 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7fb8b8006a80 tx=0x7fb8b8009f00).stop 2021-04-09T18:26:24.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.384 7fb8cc38d700 1 -- 172.21.15.32:0/1986464654 shutdown_connections 2021-04-09T18:26:24.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.384 7fb8cc38d700 1 --2- 172.21.15.32:0/1986464654 >> v2:172.21.15.86:3300/0 conn(0x7fb8c40d5c70 0x7fb8c40d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.384 7fb8cc38d700 1 --2- 172.21.15.32:0/1986464654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c40d7800 0x7fb8c40d7c00 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.384 7fb8cc38d700 1 --2- 172.21.15.32:0/1986464654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c40cffa0 0x7fb8c40d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.384 7fb8cc38d700 1 -- 172.21.15.32:0/1986464654 >> 172.21.15.32:0/1986464654 conn(0x7fb8c40ce5c0 msgr2=0x7fb8c40cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:24.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.384 7fb8cc38d700 1 -- 172.21.15.32:0/1986464654 shutdown_connections 2021-04-09T18:26:24.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.384 7fb8cc38d700 1 -- 172.21.15.32:0/1986464654 wait complete. 2021-04-09T18:26:24.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.404 7fb8cc38d700 1 Processor -- start 2021-04-09T18:26:24.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.404 7fb8cc38d700 1 -- start start 2021-04-09T18:26:24.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8cc38d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c41acff0 0x7fb8c408fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8cc38d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c4090150 0x7fb8c408c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8cc38d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c408cca0 0x7fb8c408e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8cc38d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c408c100 con 0x7fb8c41acff0 2021-04-09T18:26:24.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8cc38d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c408d0a0 con 0x7fb8c408cca0 2021-04-09T18:26:24.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8cc38d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4183cd0 con 0x7fb8c4090150 2021-04-09T18:26:24.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8ca1a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c41acff0 0x7fb8c408fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8ca1a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c41acff0 0x7fb8c408fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53190/0 (socket says 172.21.15.32:53190) 2021-04-09T18:26:24.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8ca1a9700 1 -- 172.21.15.32:0/2965853743 learned_addr learned my addr 172.21.15.32:0/2965853743 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:24.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8ca9aa700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c408cca0 0x7fb8c408e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8c99a8700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c4090150 0x7fb8c408c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8ca1a9700 1 -- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c4090150 msgr2=0x7fb8c408c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8ca1a9700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c4090150 0x7fb8c408c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8ca1a9700 1 -- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c408cca0 msgr2=0x7fb8c408e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8ca1a9700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c408cca0 0x7fb8c408e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.405 7fb8ca1a9700 1 -- 172.21.15.32:0/2965853743 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c4183ef0 con 0x7fb8c41acff0 2021-04-09T18:26:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.406 7fb8ca1a9700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c41acff0 0x7fb8c408fb30 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7fb8c000b580 tx=0x7fb8c000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.406 7fb8b6ffd700 1 -- 172.21.15.32:0/2965853743 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8c000bd10 con 0x7fb8c41acff0 2021-04-09T18:26:24.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.406 7fb8b6ffd700 1 -- 172.21.15.32:0/2965853743 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c0000c90 con 0x7fb8c41acff0 2021-04-09T18:26:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.406 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c4183ef0 con 0x7fb8c41acff0 2021-04-09T18:26:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.406 7fb8b6ffd700 1 -- 172.21.15.32:0/2965853743 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8c0002220 con 0x7fb8c41acff0 2021-04-09T18:26:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.406 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c4183ef0 con 0x7fb8c41acff0 2021-04-09T18:26:24.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.406 7fb8b6ffd700 1 -- 172.21.15.32:0/2965853743 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb8c0006b00 con 0x7fb8c41acff0 2021-04-09T18:26:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.406 7fb8b6ffd700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8ac018930 0x7fb8ac01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.406 7fb8b6ffd700 1 -- 172.21.15.32:0/2965853743 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fb8c00326c0 con 0x7fb8c41acff0 2021-04-09T18:26:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.407 7fb8c99a8700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8ac018930 0x7fb8ac01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.407 7fb8c99a8700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8ac018930 0x7fb8ac01adc0 crc :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:24.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.426 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c4182dc0 con 0x7fb8c41acff0 2021-04-09T18:26:24.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.430 7fb8b6ffd700 1 -- 172.21.15.32:0/2965853743 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb8c000bd10 con 0x7fb8c41acff0 2021-04-09T18:26:24.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.607 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb8c40d7280 con 0x7fb8c41acff0 2021-04-09T18:26:24.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.607 7fb8b6ffd700 1 -- 172.21.15.32:0/2965853743 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v135) v1 ==== 84+0+64 (secure 0 0 0) 0x7fb8c0000c90 con 0x7fb8c41acff0 2021-04-09T18:26:24.679 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:24.679 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:26:24.680 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:26:24.680 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:26:24.681 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:26:24.681 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:24.681 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:26:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8ac018930 msgr2=0x7fb8ac01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8ac018930 0x7fb8ac01adc0 crc :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c41acff0 msgr2=0x7fb8c408fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c41acff0 0x7fb8c408fb30 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7fb8c000b580 tx=0x7fb8c000b920).stop 2021-04-09T18:26:24.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 shutdown_connections 2021-04-09T18:26:24.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb8ac018930 0x7fb8ac01adc0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8c408cca0 0x7fb8c408e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8c4090150 0x7fb8c408c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 --2- 172.21.15.32:0/2965853743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8c41acff0 0x7fb8c408fb30 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 >> 172.21.15.32:0/2965853743 conn(0x7fb8c409abb0 msgr2=0x7fb8c40a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:24.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 shutdown_connections 2021-04-09T18:26:24.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.610 7fb8cc38d700 1 -- 172.21.15.32:0/2965853743 wait complete. 2021-04-09T18:26:24.692 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:26:24.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:26:24.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.774 7fa824459700 1 Processor -- start 2021-04-09T18:26:24.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.774 7fa824459700 1 -- start start 2021-04-09T18:26:24.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.775 7fa824459700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa81c10d010 0x7fa81c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.775 7fa824459700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c10d480 con 0x7fa81c0d1da0 2021-04-09T18:26:24.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.775 7fa824459700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c10e1f0 con 0x7fa81c0d8490 2021-04-09T18:26:24.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.775 7fa824459700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81c0d8d30 con 0x7fa81c10d010 2021-04-09T18:26:24.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.775 7fa822275700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa81c10d010 0x7fa81c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.775 7fa821a74700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa81c0d1da0 0x7fa81c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34560/0 (socket says 172.21.15.32:34560) 2021-04-09T18:26:24.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.775 7fa821a74700 1 -- 172.21.15.32:0/643278550 learned_addr learned my addr 172.21.15.32:0/643278550 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:24.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa821273700 1 -- 172.21.15.32:0/643278550 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986074515 0 0) 0x7fa80c000ce0 con 0x7fa81c0d1da0 2021-04-09T18:26:24.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa821273700 1 -- 172.21.15.32:0/643278550 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa808002fe0 con 0x7fa81c0d1da0 2021-04-09T18:26:24.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa821273700 1 -- 172.21.15.32:0/643278550 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762923770 0 0) 0x7fa814000d30 con 0x7fa81c0d8490 2021-04-09T18:26:24.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa822275700 1 -- 172.21.15.32:0/643278550 >> v1:172.21.15.86:6789/0 conn(0x7fa81c0d8490 legacy=0x7fa81c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa822275700 1 -- 172.21.15.32:0/643278550 >> v1:172.21.15.32:6789/0 conn(0x7fa81c0d1da0 legacy=0x7fa81c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa822275700 1 -- 172.21.15.32:0/643278550 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0d9220 con 0x7fa81c10d010 2021-04-09T18:26:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa822275700 1 --2- 172.21.15.32:0/643278550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c10d010 0x7fa81c0d1780 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 rx=0x7fa818006650 tx=0x7fa818009a80).ready entity=mon.0 client_cookie=e52a9db00d066214 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa821273700 1 -- 172.21.15.32:0/643278550 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8180009b0 con 0x7fa81c10d010 2021-04-09T18:26:24.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa821273700 1 -- 172.21.15.32:0/643278550 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81800c0c0 con 0x7fa81c10d010 2021-04-09T18:26:24.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa821273700 1 -- 172.21.15.32:0/643278550 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa81800c510 con 0x7fa81c10d010 2021-04-09T18:26:24.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa824459700 1 -- 172.21.15.32:0/643278550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c10d010 msgr2=0x7fa81c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.776 7fa824459700 1 --2- 172.21.15.32:0/643278550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c10d010 0x7fa81c0d1780 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 rx=0x7fa818006650 tx=0x7fa818009a80).stop 2021-04-09T18:26:24.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.777 7fa824459700 1 -- 172.21.15.32:0/643278550 shutdown_connections 2021-04-09T18:26:24.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.777 7fa824459700 1 --2- 172.21.15.32:0/643278550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c10d010 0x7fa81c0d1780 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.777 7fa824459700 1 -- 172.21.15.32:0/643278550 >> 172.21.15.32:0/643278550 conn(0x7fa81c0ce5f0 msgr2=0x7fa81c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:24.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.777 7fa824459700 1 -- 172.21.15.32:0/643278550 shutdown_connections 2021-04-09T18:26:24.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.777 7fa824459700 1 -- 172.21.15.32:0/643278550 wait complete. 2021-04-09T18:26:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.797 7fa824459700 1 Processor -- start 2021-04-09T18:26:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.797 7fa824459700 1 -- start start 2021-04-09T18:26:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa824459700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c090ed0 0x7fa81c0912d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa824459700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c08c920 0x7fa81c08cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa824459700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c08fab0 0x7fa81c08feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa824459700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c090950 con 0x7fa81c08fab0 2021-04-09T18:26:24.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa824459700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0918f0 con 0x7fa81c08c920 2021-04-09T18:26:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa824459700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c08d340 con 0x7fa81c090ed0 2021-04-09T18:26:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822a76700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c08fab0 0x7fa81c08feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822a76700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c08fab0 0x7fa81c08feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53206/0 (socket says 172.21.15.32:53206) 2021-04-09T18:26:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822a76700 1 -- 172.21.15.32:0/2764865904 learned_addr learned my addr 172.21.15.32:0/2764865904 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:24.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa821a74700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c08c920 0x7fa81c08cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822275700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c090ed0 0x7fa81c0912d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822a76700 1 -- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c090ed0 msgr2=0x7fa81c0912d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822a76700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c090ed0 0x7fa81c0912d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822a76700 1 -- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c08c920 msgr2=0x7fa81c08cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:24.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822a76700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c08c920 0x7fa81c08cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:24.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822a76700 1 -- 172.21.15.32:0/2764865904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0b1760 con 0x7fa81c08fab0 2021-04-09T18:26:24.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822275700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c090ed0 0x7fa81c0912d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:24.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.798 7fa822a76700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c08fab0 0x7fa81c08feb0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7fa814009980 tx=0x7fa81400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:24.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.799 7fa812ffd700 1 -- 172.21.15.32:0/2764865904 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa814005870 con 0x7fa81c08fab0 2021-04-09T18:26:24.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.799 7fa812ffd700 1 -- 172.21.15.32:0/2764865904 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa814005b70 con 0x7fa81c08fab0 2021-04-09T18:26:24.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.799 7fa824459700 1 -- 172.21.15.32:0/2764865904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c0b1760 con 0x7fa81c08fab0 2021-04-09T18:26:24.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.799 7fa812ffd700 1 -- 172.21.15.32:0/2764865904 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa814006030 con 0x7fa81c08fab0 2021-04-09T18:26:24.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.799 7fa824459700 1 -- 172.21.15.32:0/2764865904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c0b1760 con 0x7fa81c08fab0 2021-04-09T18:26:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.799 7fa812ffd700 1 -- 172.21.15.32:0/2764865904 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa814006030 con 0x7fa81c08fab0 2021-04-09T18:26:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.799 7fa812ffd700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.799 7fa812ffd700 1 -- 172.21.15.32:0/2764865904 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fa814032ae0 con 0x7fa81c08fab0 2021-04-09T18:26:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.800 7fa822275700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:24.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.800 7fa822275700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 crc :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.819 7fa824459700 1 -- 172.21.15.32:0/2764865904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c10d480 con 0x7fa81c08fab0 2021-04-09T18:26:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:24.823 7fa812ffd700 1 -- 172.21.15.32:0/2764865904 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa81400be10 con 0x7fa81c08fab0 2021-04-09T18:26:25.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.009 7fa824459700 1 -- 172.21.15.32:0/2764865904 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa81c10d480 con 0x7fa81c08fab0 2021-04-09T18:26:26.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.960 7fa812ffd700 1 -- 172.21.15.32:0/2764865904 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v136) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa81400be10 con 0x7fa81c08fab0 2021-04-09T18:26:26.046 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:26:26.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 -- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 msgr2=0x7fa80401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:26.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 crc :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 -- 172.21.15.32:0/2764865904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c08fab0 msgr2=0x7fa81c08feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:26.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c08fab0 0x7fa81c08feb0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7fa814009980 tx=0x7fa81400bcd0).stop 2021-04-09T18:26:26.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 -- 172.21.15.32:0/2764865904 shutdown_connections 2021-04-09T18:26:26.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c08fab0 0x7fa81c08feb0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c08c920 0x7fa81c08cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 --2- 172.21.15.32:0/2764865904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c090ed0 0x7fa81c0912d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 -- 172.21.15.32:0/2764865904 >> 172.21.15.32:0/2764865904 conn(0x7fa81c0b8980 msgr2=0x7fa81c1b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:26.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 -- 172.21.15.32:0/2764865904 shutdown_connections 2021-04-09T18:26:26.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:25.977 7fa824459700 1 -- 172.21.15.32:0/2764865904 wait complete. 2021-04-09T18:26:26.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:26:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.142 7fecfdd3b700 1 Processor -- start 2021-04-09T18:26:26.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.142 7fecfdd3b700 1 -- start start 2021-04-09T18:26:26.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecfdd3b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf810d010 0x7fecf80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:26.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecfdd3b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fecf80d8490 0x7fecf810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:26.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecfdd3b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf810d480 con 0x7fecf80d1da0 2021-04-09T18:26:26.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecfdd3b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf810e780 con 0x7fecf80d8490 2021-04-09T18:26:26.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecfdd3b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf80d8d30 con 0x7fecf810d010 2021-04-09T18:26:26.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecf6ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fecf80d1da0 0x7fecf80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34574/0 (socket says 172.21.15.32:34574) 2021-04-09T18:26:26.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecf6ffd700 1 -- 172.21.15.32:0/1525730050 learned_addr learned my addr 172.21.15.32:0/1525730050 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:26.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecf77fe700 1 --2- 172.21.15.32:0/1525730050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf810d010 0x7fecf80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:26.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecf7fff700 1 --2- 172.21.15.32:0/1525730050 >> v2:172.21.15.86:3300/0 conn(0x7fecf80d8490 0x7fecf810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:26.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecf67fc700 1 -- 172.21.15.32:0/1525730050 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144626338 0 0) 0x7fecec0009b0 con 0x7fecf80d1da0 2021-04-09T18:26:26.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecf67fc700 1 -- 172.21.15.32:0/1525730050 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece0002fe0 con 0x7fecf80d1da0 2021-04-09T18:26:26.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecf7fff700 1 -- 172.21.15.32:0/1525730050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf810d010 msgr2=0x7fecf80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:26.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecf7fff700 1 --2- 172.21.15.32:0/1525730050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf810d010 0x7fecf80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.143 7fecf7fff700 1 -- 172.21.15.32:0/1525730050 >> v1:172.21.15.32:6789/0 conn(0x7fecf80d1da0 legacy=0x7fecf80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:26.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.144 7fecf7fff700 1 -- 172.21.15.32:0/1525730050 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf80d9220 con 0x7fecf80d8490 2021-04-09T18:26:26.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.144 7fecf77fe700 1 --2- 172.21.15.32:0/1525730050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf810d010 0x7fecf80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:26.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.144 7fecf7fff700 1 --2- 172.21.15.32:0/1525730050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf80d8490 0x7fecf810da00 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7fecf0006e50 tx=0x7fecf0009e70).ready entity=mon.1 client_cookie=70fdaa03f754ed4d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:26.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.144 7fecf67fc700 1 -- 172.21.15.32:0/1525730050 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecf0006a20 con 0x7fecf80d8490 2021-04-09T18:26:26.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.144 7fecf67fc700 1 -- 172.21.15.32:0/1525730050 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf0006790 con 0x7fecf80d8490 2021-04-09T18:26:26.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.144 7fecf67fc700 1 -- 172.21.15.32:0/1525730050 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecf000b700 con 0x7fecf80d8490 2021-04-09T18:26:26.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.144 7fecfdd3b700 1 -- 172.21.15.32:0/1525730050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf80d8490 msgr2=0x7fecf810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:26.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.144 7fecfdd3b700 1 --2- 172.21.15.32:0/1525730050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf80d8490 0x7fecf810da00 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7fecf0006e50 tx=0x7fecf0009e70).stop 2021-04-09T18:26:26.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.145 7fecfdd3b700 1 -- 172.21.15.32:0/1525730050 shutdown_connections 2021-04-09T18:26:26.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.145 7fecfdd3b700 1 --2- 172.21.15.32:0/1525730050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf80d8490 0x7fecf810da00 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.145 7fecfdd3b700 1 --2- 172.21.15.32:0/1525730050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf810d010 0x7fecf80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.145 7fecfdd3b700 1 -- 172.21.15.32:0/1525730050 >> 172.21.15.32:0/1525730050 conn(0x7fecf80ce5f0 msgr2=0x7fecf80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:26.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.145 7fecfdd3b700 1 -- 172.21.15.32:0/1525730050 shutdown_connections 2021-04-09T18:26:26.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.145 7fecfdd3b700 1 -- 172.21.15.32:0/1525730050 wait complete. 2021-04-09T18:26:26.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.165 7fecfdd3b700 1 Processor -- start 2021-04-09T18:26:26.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.165 7fecfdd3b700 1 -- start start 2021-04-09T18:26:26.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.165 7fecfdd3b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf80b2140 0x7fecf809d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:26.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.165 7fecfdd3b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf809e010 0x7fecf8194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:26.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecfdd3b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf81950a0 0x7fecf80a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:26.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecfdd3b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf809d470 con 0x7fecf809e010 2021-04-09T18:26:26.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecfdd3b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf809e410 con 0x7fecf81950a0 2021-04-09T18:26:26.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecfdd3b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf81954a0 con 0x7fecf80b2140 2021-04-09T18:26:26.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf809e010 0x7fecf8194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:26.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf6ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf809e010 0x7fecf8194a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53220/0 (socket says 172.21.15.32:53220) 2021-04-09T18:26:26.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf6ffd700 1 -- 172.21.15.32:0/1029207799 learned_addr learned my addr 172.21.15.32:0/1029207799 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:26.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf77fe700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf80b2140 0x7fecf809d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:26.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf7fff700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf81950a0 0x7fecf80a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:26.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf6ffd700 1 -- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf80b2140 msgr2=0x7fecf809d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:26.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf6ffd700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf80b2140 0x7fecf809d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf6ffd700 1 -- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf81950a0 msgr2=0x7fecf80a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:26.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf6ffd700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf81950a0 0x7fecf80a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:26.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf6ffd700 1 -- 172.21.15.32:0/1029207799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf81956c0 con 0x7fecf809e010 2021-04-09T18:26:26.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecf6ffd700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf809e010 0x7fecf8194a80 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 rx=0x7fecec00a4c0 tx=0x7fecec00a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:26.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecdffff700 1 -- 172.21.15.32:0/1029207799 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecec00ac50 con 0x7fecf809e010 2021-04-09T18:26:26.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecdffff700 1 -- 172.21.15.32:0/1029207799 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec004140 con 0x7fecf809e010 2021-04-09T18:26:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecdffff700 1 -- 172.21.15.32:0/1029207799 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecec004620 con 0x7fecf809e010 2021-04-09T18:26:26.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf81956c0 con 0x7fecf809e010 2021-04-09T18:26:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.166 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf81956c0 con 0x7fecf809e010 2021-04-09T18:26:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.168 7fecdffff700 1 -- 172.21.15.32:0/1029207799 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fecec004140 con 0x7fecf809e010 2021-04-09T18:26:26.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.168 7fecdffff700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecd80188c0 0x7fecd801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.168 7fecdffff700 1 -- 172.21.15.32:0/1029207799 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5111+0+0 (secure 0 0 0) 0x7fecec0325b0 con 0x7fecf809e010 2021-04-09T18:26:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.168 7fecf77fe700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecd80188c0 0x7fecd801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:26.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.168 7fecf77fe700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecd80188c0 0x7fecd801adc0 crc :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:26.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.187 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf81b7040 con 0x7fecf809e010 2021-04-09T18:26:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.191 7fecdffff700 1 -- 172.21.15.32:0/1029207799 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fecec000e20 con 0x7fecf809e010 2021-04-09T18:26:26.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:26.378 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fecf81b7040 con 0x7fecf809e010 2021-04-09T18:26:27.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.223 7fecdffff700 1 -- 172.21.15.32:0/1029207799 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v140) v1 ==== 125+0+0 (secure 0 0 0) 0x7fecec004140 con 0x7fecf809e010 2021-04-09T18:26:27.314 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:26:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecd80188c0 msgr2=0x7fecd801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecd80188c0 0x7fecd801adc0 crc :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf809e010 msgr2=0x7fecf8194a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf809e010 0x7fecf8194a80 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 rx=0x7fecec00a4c0 tx=0x7fecec00a860).stop 2021-04-09T18:26:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 shutdown_connections 2021-04-09T18:26:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecd80188c0 0x7fecd801adc0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf809e010 0x7fecf8194a80 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf81950a0 0x7fecf80a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 --2- 172.21.15.32:0/1029207799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf80b2140 0x7fecf809d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 >> 172.21.15.32:0/1029207799 conn(0x7fecf80b27e0 msgr2=0x7fecf81b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:27.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 shutdown_connections 2021-04-09T18:26:27.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.246 7fecfdd3b700 1 -- 172.21.15.32:0/1029207799 wait complete. 2021-04-09T18:26:27.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:26:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.417 7febbf405700 1 Processor -- start 2021-04-09T18:26:27.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.418 7febbf405700 1 -- start start 2021-04-09T18:26:27.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.418 7febbf405700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80d7cc0 0x7febb80d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.418 7febbf405700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb810d500 con 0x7febb810da80 2021-04-09T18:26:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.418 7febbf405700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb80d0330 con 0x7febb810d050 2021-04-09T18:26:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.418 7febbf405700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb80d9260 con 0x7febb80d7cc0 2021-04-09T18:26:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.418 7febbda22700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7febb810da80 0x7febb810de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34590/0 (socket says 172.21.15.32:34590) 2021-04-09T18:26:27.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.418 7febbda22700 1 -- 172.21.15.32:0/1877522607 learned_addr learned my addr 172.21.15.32:0/1877522607 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:27.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febbca20700 1 --2- 172.21.15.32:0/1877522607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80d7cc0 0x7febb80d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febaffff700 1 -- 172.21.15.32:0/1877522607 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559867194 0 0) 0x7febb0000d30 con 0x7febb810da80 2021-04-09T18:26:27.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febaffff700 1 -- 172.21.15.32:0/1877522607 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba4002fe0 con 0x7febb810da80 2021-04-09T18:26:27.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febaffff700 1 -- 172.21.15.32:0/1877522607 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226562571 0 0) 0x7febb4000c60 con 0x7febb810d050 2021-04-09T18:26:27.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febaffff700 1 -- 172.21.15.32:0/1877522607 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba40042c0 con 0x7febb810d050 2021-04-09T18:26:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febaffff700 1 -- 172.21.15.32:0/1877522607 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1458805352 0 0) 0x7febb0001270 con 0x7febb810da80 2021-04-09T18:26:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febbca20700 1 -- 172.21.15.32:0/1877522607 >> v1:172.21.15.86:6789/0 conn(0x7febb810d050 legacy=0x7febb80cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febbca20700 1 -- 172.21.15.32:0/1877522607 >> v1:172.21.15.32:6789/0 conn(0x7febb810da80 legacy=0x7febb810de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febbca20700 1 -- 172.21.15.32:0/1877522607 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8112140 con 0x7febb80d7cc0 2021-04-09T18:26:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febbca20700 1 --2- 172.21.15.32:0/1877522607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80d7cc0 0x7febb80d80c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7feba8006650 tx=0x7feba8009a00).ready entity=mon.2 client_cookie=b9a8c17985cbe540 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febaffff700 1 -- 172.21.15.32:0/1877522607 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba8009df0 con 0x7febb80d7cc0 2021-04-09T18:26:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febaffff700 1 -- 172.21.15.32:0/1877522607 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba80009b0 con 0x7febb80d7cc0 2021-04-09T18:26:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febaffff700 1 -- 172.21.15.32:0/1877522607 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba800c2b0 con 0x7febb80d7cc0 2021-04-09T18:26:27.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febbf405700 1 -- 172.21.15.32:0/1877522607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80d7cc0 msgr2=0x7febb80d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.419 7febbf405700 1 --2- 172.21.15.32:0/1877522607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80d7cc0 0x7febb80d80c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7feba8006650 tx=0x7feba8009a00).stop 2021-04-09T18:26:27.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.420 7febbf405700 1 -- 172.21.15.32:0/1877522607 shutdown_connections 2021-04-09T18:26:27.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.420 7febbf405700 1 --2- 172.21.15.32:0/1877522607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80d7cc0 0x7febb80d80c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.420 7febbf405700 1 -- 172.21.15.32:0/1877522607 >> 172.21.15.32:0/1877522607 conn(0x7febb80ce620 msgr2=0x7febb80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.420 7febbf405700 1 -- 172.21.15.32:0/1877522607 shutdown_connections 2021-04-09T18:26:27.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.420 7febbf405700 1 -- 172.21.15.32:0/1877522607 wait complete. 2021-04-09T18:26:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.440 7febbf405700 1 Processor -- start 2021-04-09T18:26:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.440 7febbf405700 1 -- start start 2021-04-09T18:26:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.441 7febbf405700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80ad600 0x7febb809ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.441 7febbf405700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb809b1b0 0x7febb80ba220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.441 7febbf405700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb80ba840 0x7febb80a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.441 7febbf405700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb809fa80 con 0x7febb809b1b0 2021-04-09T18:26:27.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.441 7febbf405700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb809b5b0 con 0x7febb80ba840 2021-04-09T18:26:27.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.441 7febbf405700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb80bac40 con 0x7febb80ad600 2021-04-09T18:26:27.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.441 7febbca20700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb809b1b0 0x7febb80ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.441 7febbca20700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb809b1b0 0x7febb80ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53236/0 (socket says 172.21.15.32:53236) 2021-04-09T18:26:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.441 7febbca20700 1 -- 172.21.15.32:0/1888675587 learned_addr learned my addr 172.21.15.32:0/1888675587 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbda22700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb80ba840 0x7febb80a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbd221700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80ad600 0x7febb809ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbca20700 1 -- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80ad600 msgr2=0x7febb809ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbca20700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80ad600 0x7febb809ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbca20700 1 -- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb80ba840 msgr2=0x7febb80a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbca20700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb80ba840 0x7febb80a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbca20700 1 -- 172.21.15.32:0/1888675587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb80a97c0 con 0x7febb809b1b0 2021-04-09T18:26:27.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbda22700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb80ba840 0x7febb80a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:27.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbca20700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb809b1b0 0x7febb80ba220 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 rx=0x7feba80096a0 tx=0x7feba800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:27.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febadffb700 1 -- 172.21.15.32:0/1888675587 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba8000c90 con 0x7febb809b1b0 2021-04-09T18:26:27.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febadffb700 1 -- 172.21.15.32:0/1888675587 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba8002030 con 0x7febb809b1b0 2021-04-09T18:26:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbf405700 1 -- 172.21.15.32:0/1888675587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb80a97c0 con 0x7febb809b1b0 2021-04-09T18:26:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febadffb700 1 -- 172.21.15.32:0/1888675587 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feba8002550 con 0x7febb809b1b0 2021-04-09T18:26:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.442 7febbf405700 1 -- 172.21.15.32:0/1888675587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb80a97c0 con 0x7febb809b1b0 2021-04-09T18:26:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.443 7febadffb700 1 -- 172.21.15.32:0/1888675587 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7feba8002210 con 0x7febb809b1b0 2021-04-09T18:26:27.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.443 7febadffb700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb98018930 0x7feb9801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.443 7febadffb700 1 -- 172.21.15.32:0/1888675587 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5461+0+0 (secure 0 0 0) 0x7feba80329d0 con 0x7febb809b1b0 2021-04-09T18:26:27.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.443 7febbd221700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb98018930 0x7feb9801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.443 7febbd221700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb98018930 0x7feb9801ae10 crc :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:27.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.462 7febbf405700 1 -- 172.21.15.32:0/1888675587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb80aa330 con 0x7febb809b1b0 2021-04-09T18:26:27.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.466 7febadffb700 1 -- 172.21.15.32:0/1888675587 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7feba80009a0 con 0x7febb809b1b0 2021-04-09T18:26:27.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.649 7febbf405700 1 -- 172.21.15.32:0/1888675587 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7febb80aa330 con 0x7febb809b1b0 2021-04-09T18:26:27.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.653 7febadffb700 1 -- 172.21.15.32:0/1888675587 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 24 and data pool 25 v273) v1 ==== 189+0+0 (secure 0 0 0) 0x7feba8001f20 con 0x7febb809b1b0 2021-04-09T18:26:27.727 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 24 and data pool 25 2021-04-09T18:26:27.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.660 7febbf405700 1 -- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb98018930 msgr2=0x7feb9801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.660 7febbf405700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb98018930 0x7feb9801ae10 crc :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.660 7febbf405700 1 -- 172.21.15.32:0/1888675587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb809b1b0 msgr2=0x7febb80ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.660 7febbf405700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb809b1b0 0x7febb80ba220 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 rx=0x7feba80096a0 tx=0x7feba800b990).stop 2021-04-09T18:26:27.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.662 7febbf405700 1 -- 172.21.15.32:0/1888675587 shutdown_connections 2021-04-09T18:26:27.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.662 7febbf405700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feb98018930 0x7feb9801ae10 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.662 7febbf405700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febb809b1b0 0x7febb80ba220 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.662 7febbf405700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febb80ba840 0x7febb80a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.662 7febbf405700 1 --2- 172.21.15.32:0/1888675587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febb80ad600 0x7febb809ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.662 7febbf405700 1 -- 172.21.15.32:0/1888675587 >> 172.21.15.32:0/1888675587 conn(0x7febb80ae6f0 msgr2=0x7febb8097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:27.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.662 7febbf405700 1 -- 172.21.15.32:0/1888675587 shutdown_connections 2021-04-09T18:26:27.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.662 7febbf405700 1 -- 172.21.15.32:0/1888675587 wait complete. 2021-04-09T18:26:27.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:26:27.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.824 7f9065795700 1 Processor -- start 2021-04-09T18:26:27.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.824 7f9065795700 1 -- start start 2021-04-09T18:26:27.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f9065795700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f906010d010 0x7f90600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f9065795700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f90600d1da0 0x7f90600d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f9065795700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f90600d82c0 0x7f906010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f9065795700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906010e780 con 0x7f90600d82c0 2021-04-09T18:26:27.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f9065795700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90600d8d30 con 0x7f90600d1da0 2021-04-09T18:26:27.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f9065795700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90600d9220 con 0x7f906010d010 2021-04-09T18:26:27.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905fd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f90600d82c0 0x7f906010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905fd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f90600d82c0 0x7f906010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53246/0 (socket says 172.21.15.32:53246) 2021-04-09T18:26:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905fd75700 1 -- 172.21.15.32:0/2455408786 learned_addr learned my addr 172.21.15.32:0/2455408786 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905ed73700 1 --2- 172.21.15.32:0/2455408786 >> v2:172.21.15.86:3300/0 conn(0x7f90600d1da0 0x7f90600d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905f574700 1 --2- 172.21.15.32:0/2455408786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f906010d010 0x7f90600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905fd75700 1 -- 172.21.15.32:0/2455408786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f906010d010 msgr2=0x7f90600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905fd75700 1 --2- 172.21.15.32:0/2455408786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f906010d010 0x7f90600d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905fd75700 1 -- 172.21.15.32:0/2455408786 >> v2:172.21.15.86:3300/0 conn(0x7f90600d1da0 msgr2=0x7f90600d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905fd75700 1 --2- 172.21.15.32:0/2455408786 >> v2:172.21.15.86:3300/0 conn(0x7f90600d1da0 0x7f90600d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905fd75700 1 -- 172.21.15.32:0/2455408786 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906010d480 con 0x7f90600d82c0 2021-04-09T18:26:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905ed73700 1 --2- 172.21.15.32:0/2455408786 >> v2:172.21.15.86:3300/0 conn(0x7f90600d1da0 0x7f90600d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905fd75700 1 --2- 172.21.15.32:0/2455408786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90600d82c0 0x7f906010da00 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 rx=0x7f9058006b80 tx=0x7f9058009b90).ready entity=mon.0 client_cookie=b963a76f157efe70 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905e572700 1 -- 172.21.15.32:0/2455408786 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90580022e0 con 0x7f90600d82c0 2021-04-09T18:26:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905e572700 1 -- 172.21.15.32:0/2455408786 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9058002620 con 0x7f90600d82c0 2021-04-09T18:26:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f905e572700 1 -- 172.21.15.32:0/2455408786 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9058002b00 con 0x7f90600d82c0 2021-04-09T18:26:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.825 7f9065795700 1 -- 172.21.15.32:0/2455408786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90600d82c0 msgr2=0x7f906010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.826 7f9065795700 1 --2- 172.21.15.32:0/2455408786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90600d82c0 0x7f906010da00 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 rx=0x7f9058006b80 tx=0x7f9058009b90).stop 2021-04-09T18:26:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.826 7f9065795700 1 -- 172.21.15.32:0/2455408786 shutdown_connections 2021-04-09T18:26:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.826 7f9065795700 1 --2- 172.21.15.32:0/2455408786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90600d82c0 0x7f906010da00 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.826 7f9065795700 1 --2- 172.21.15.32:0/2455408786 >> v2:172.21.15.86:3300/0 conn(0x7f90600d1da0 0x7f90600d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.826 7f9065795700 1 --2- 172.21.15.32:0/2455408786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f906010d010 0x7f90600d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.826 7f9065795700 1 -- 172.21.15.32:0/2455408786 >> 172.21.15.32:0/2455408786 conn(0x7f90600ce5f0 msgr2=0x7f90600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.826 7f9065795700 1 -- 172.21.15.32:0/2455408786 shutdown_connections 2021-04-09T18:26:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.826 7f9065795700 1 -- 172.21.15.32:0/2455408786 wait complete. 2021-04-09T18:26:27.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.846 7f9065795700 1 Processor -- start 2021-04-09T18:26:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.846 7f9065795700 1 -- start start 2021-04-09T18:26:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f9065795700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9060185060 0x7f9060185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f9065795700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9060186c00 0x7f9060187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f9065795700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9060191360 0x7f9060191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f9065795700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90601863a0 con 0x7f9060191360 2021-04-09T18:26:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f9065795700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9060185a60 con 0x7f9060186c00 2021-04-09T18:26:27.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f9065795700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9060187620 con 0x7f9060185060 2021-04-09T18:26:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9060191360 0x7f9060191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9060191360 0x7f9060191760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53254/0 (socket says 172.21.15.32:53254) 2021-04-09T18:26:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905fd75700 1 -- 172.21.15.32:0/3369946733 learned_addr learned my addr 172.21.15.32:0/3369946733 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905ed73700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9060186c00 0x7f9060187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905f574700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9060185060 0x7f9060185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905fd75700 1 -- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9060185060 msgr2=0x7f9060185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905fd75700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9060185060 0x7f9060185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905fd75700 1 -- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9060186c00 msgr2=0x7f9060187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905fd75700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9060186c00 0x7f9060187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.847 7f905fd75700 1 -- 172.21.15.32:0/3369946733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9060185460 con 0x7f9060191360 2021-04-09T18:26:27.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.848 7f905ed73700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9060186c00 0x7f9060187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.848 7f905fd75700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9060191360 0x7f9060191760 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7f9058006f10 tx=0x7f905800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.848 7f9047fff700 1 -- 172.21.15.32:0/3369946733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9058000b90 con 0x7f9060191360 2021-04-09T18:26:27.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.848 7f9047fff700 1 -- 172.21.15.32:0/3369946733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905800bd50 con 0x7f9060191360 2021-04-09T18:26:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.848 7f9047fff700 1 -- 172.21.15.32:0/3369946733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9058002220 con 0x7f9060191360 2021-04-09T18:26:27.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.848 7f9065795700 1 -- 172.21.15.32:0/3369946733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9060185460 con 0x7f9060191360 2021-04-09T18:26:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.848 7f9065795700 1 -- 172.21.15.32:0/3369946733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9060185460 con 0x7f9060191360 2021-04-09T18:26:27.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.848 7f9047fff700 1 -- 172.21.15.32:0/3369946733 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9058009100 con 0x7f9060191360 2021-04-09T18:26:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.849 7f9047fff700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90400188c0 0x7f904001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:27.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.849 7f9047fff700 1 -- 172.21.15.32:0/3369946733 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f90580329e0 con 0x7f9060191360 2021-04-09T18:26:27.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.849 7f905f574700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90400188c0 0x7f904001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:27.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.849 7f905f574700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90400188c0 0x7f904001adc0 crc :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:27.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.869 7f9065795700 1 -- 172.21.15.32:0/3369946733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90601866d0 con 0x7f9060191360 2021-04-09T18:26:27.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:27.872 7f9047fff700 1 -- 172.21.15.32:0/3369946733 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9058000b90 con 0x7f9060191360 2021-04-09T18:26:28.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.060 7f9065795700 1 -- 172.21.15.32:0/3369946733 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90601866d0 con 0x7f9060191360 2021-04-09T18:26:28.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.061 7f9047fff700 1 -- 172.21.15.32:0/3369946733 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f9058000ef0 con 0x7f9060191360 2021-04-09T18:26:28.133 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:28.133 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":141,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:26:27.650915","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:26:25.044737","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-04-09 18:26:26.413903","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:26:28.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 -- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90400188c0 msgr2=0x7f904001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90400188c0 0x7f904001adc0 crc :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 -- 172.21.15.32:0/3369946733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9060191360 msgr2=0x7f9060191760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9060191360 0x7f9060191760 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7f9058006f10 tx=0x7f905800b960).stop 2021-04-09T18:26:28.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 -- 172.21.15.32:0/3369946733 shutdown_connections 2021-04-09T18:26:28.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f90400188c0 0x7f904001adc0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9060191360 0x7f9060191760 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9060186c00 0x7f9060187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 --2- 172.21.15.32:0/3369946733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9060185060 0x7f9060185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 -- 172.21.15.32:0/3369946733 >> 172.21.15.32:0/3369946733 conn(0x7f90601837d0 msgr2=0x7f9060195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:28.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 -- 172.21.15.32:0/3369946733 shutdown_connections 2021-04-09T18:26:28.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.063 7f9065795700 1 -- 172.21.15.32:0/3369946733 wait complete. 2021-04-09T18:26:28.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:26:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.221 7ff1ebec8700 1 Processor -- start 2021-04-09T18:26:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.222 7ff1ebec8700 1 -- start start 2021-04-09T18:26:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.222 7ff1ebec8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e40d7ca0 0x7ff1e40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.222 7ff1ebec8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff1e410da00 0x7ff1e410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.222 7ff1ebec8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e410d480 con 0x7ff1e410d010 2021-04-09T18:26:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.222 7ff1ebec8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e40d9220 con 0x7ff1e410da00 2021-04-09T18:26:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.222 7ff1ebec8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e4112100 con 0x7ff1e40d7ca0 2021-04-09T18:26:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.222 7ff1ea4e5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff1e410da00 0x7ff1e410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.222 7ff1ea4e5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff1e410da00 0x7ff1e410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53260/0 (socket says 172.21.15.32:53260) 2021-04-09T18:26:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.222 7ff1ea4e5700 1 -- 172.21.15.32:0/2062695464 learned_addr learned my addr 172.21.15.32:0/2062695464 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1e94e3700 1 --2- 172.21.15.32:0/2062695464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e40d7ca0 0x7ff1e40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1ea4e5700 1 -- 172.21.15.32:0/2062695464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e40d7ca0 msgr2=0x7ff1e40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1ea4e5700 1 --2- 172.21.15.32:0/2062695464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e40d7ca0 0x7ff1e40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1ea4e5700 1 -- 172.21.15.32:0/2062695464 >> v1:172.21.15.86:6789/0 conn(0x7ff1e410d010 legacy=0x7ff1e40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1ea4e5700 1 -- 172.21.15.32:0/2062695464 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e40d86c0 con 0x7ff1e410da00 2021-04-09T18:26:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1e94e3700 1 --2- 172.21.15.32:0/2062695464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e40d7ca0 0x7ff1e40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1ea4e5700 1 --2- 172.21.15.32:0/2062695464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e410da00 0x7ff1e410de00 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7ff1dc006f00 tx=0x7ff1dc009f10).ready entity=mon.0 client_cookie=e0b3f11e886a06f1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1e8ce2700 1 -- 172.21.15.32:0/2062695464 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1dc00b3b0 con 0x7ff1e410da00 2021-04-09T18:26:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1e8ce2700 1 -- 172.21.15.32:0/2062695464 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1dc00b730 con 0x7ff1e410da00 2021-04-09T18:26:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1e8ce2700 1 -- 172.21.15.32:0/2062695464 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1dc00bc10 con 0x7ff1e410da00 2021-04-09T18:26:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1ebec8700 1 -- 172.21.15.32:0/2062695464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e410da00 msgr2=0x7ff1e410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.223 7ff1ebec8700 1 --2- 172.21.15.32:0/2062695464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e410da00 0x7ff1e410de00 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7ff1dc006f00 tx=0x7ff1dc009f10).stop 2021-04-09T18:26:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.224 7ff1ebec8700 1 -- 172.21.15.32:0/2062695464 shutdown_connections 2021-04-09T18:26:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.224 7ff1ebec8700 1 --2- 172.21.15.32:0/2062695464 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e410da00 0x7ff1e410de00 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.224 7ff1ebec8700 1 --2- 172.21.15.32:0/2062695464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e40d7ca0 0x7ff1e40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.224 7ff1ebec8700 1 -- 172.21.15.32:0/2062695464 >> 172.21.15.32:0/2062695464 conn(0x7ff1e40ce5f0 msgr2=0x7ff1e40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.224 7ff1ebec8700 1 -- 172.21.15.32:0/2062695464 shutdown_connections 2021-04-09T18:26:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.224 7ff1ebec8700 1 -- 172.21.15.32:0/2062695464 wait complete. 2021-04-09T18:26:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.244 7ff1ebec8700 1 Processor -- start 2021-04-09T18:26:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.244 7ff1ebec8700 1 -- start start 2021-04-09T18:26:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1ebec8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e40b42d0 0x7ff1e4183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1ebec8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e41ac090 0x7ff1e41ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1ebec8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e4184d30 0x7ff1e4185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1ebec8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40a1a90 con 0x7ff1e40b42d0 2021-04-09T18:26:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1ebec8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e41acab0 con 0x7ff1e41ac090 2021-04-09T18:26:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1ebec8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e4185a00 con 0x7ff1e4184d30 2021-04-09T18:26:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e9ce4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e40b42d0 0x7ff1e4183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e9ce4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e40b42d0 0x7ff1e4183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53268/0 (socket says 172.21.15.32:53268) 2021-04-09T18:26:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e9ce4700 1 -- 172.21.15.32:0/2899981238 learned_addr learned my addr 172.21.15.32:0/2899981238 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1ea4e5700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e4184d30 0x7ff1e4185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e94e3700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e41ac090 0x7ff1e41ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e9ce4700 1 -- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e4184d30 msgr2=0x7ff1e4185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e9ce4700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e4184d30 0x7ff1e4185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e9ce4700 1 -- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e41ac090 msgr2=0x7ff1e41ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e9ce4700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e41ac090 0x7ff1e41ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e9ce4700 1 -- 172.21.15.32:0/2899981238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e4185750 con 0x7ff1e40b42d0 2021-04-09T18:26:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e94e3700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e41ac090 0x7ff1e41ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:28.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1e9ce4700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e40b42d0 0x7ff1e4183520 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7ff1e000b530 tx=0x7ff1e000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.245 7ff1d67fc700 1 -- 172.21.15.32:0/2899981238 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1e000bcc0 con 0x7ff1e40b42d0 2021-04-09T18:26:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.246 7ff1d67fc700 1 -- 172.21.15.32:0/2899981238 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e0006020 con 0x7ff1e40b42d0 2021-04-09T18:26:28.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.246 7ff1d67fc700 1 -- 172.21.15.32:0/2899981238 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1e00054a0 con 0x7ff1e40b42d0 2021-04-09T18:26:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.246 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e4185750 con 0x7ff1e40b42d0 2021-04-09T18:26:28.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.246 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e4185750 con 0x7ff1e40b42d0 2021-04-09T18:26:28.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.246 7ff1d67fc700 1 -- 172.21.15.32:0/2899981238 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff1e0006200 con 0x7ff1e40b42d0 2021-04-09T18:26:28.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.246 7ff1d67fc700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff1cc0188c0 0x7ff1cc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.246 7ff1d67fc700 1 -- 172.21.15.32:0/2899981238 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff1e0032810 con 0x7ff1e40b42d0 2021-04-09T18:26:28.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.247 7ff1e94e3700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff1cc0188c0 0x7ff1cc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.247 7ff1e94e3700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff1cc0188c0 0x7ff1cc01adc0 crc :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:28.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.266 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e4182d10 con 0x7ff1e40b42d0 2021-04-09T18:26:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.270 7ff1d67fc700 1 -- 172.21.15.32:0/2899981238 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff1e002e060 con 0x7ff1e40b42d0 2021-04-09T18:26:28.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.458 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1e410d480 con 0x7ff1e40b42d0 2021-04-09T18:26:28.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.459 7ff1d67fc700 1 -- 172.21.15.32:0/2899981238 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+14020 (secure 0 0 0) 0x7ff1e002e060 con 0x7ff1e40b42d0 2021-04-09T18:26:28.531 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:28.531 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":141,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:26:27.650915","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:26:25.044737","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-04-09 18:26:26.413903","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:26:28.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff1cc0188c0 msgr2=0x7ff1cc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff1cc0188c0 0x7ff1cc01adc0 crc :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e40b42d0 msgr2=0x7ff1e4183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e40b42d0 0x7ff1e4183520 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7ff1e000b530 tx=0x7ff1e000b8d0).stop 2021-04-09T18:26:28.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 shutdown_connections 2021-04-09T18:26:28.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff1cc0188c0 0x7ff1cc01adc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1e4184d30 0x7ff1e4185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e41ac090 0x7ff1e41ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 --2- 172.21.15.32:0/2899981238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1e40b42d0 0x7ff1e4183520 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.461 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 >> 172.21.15.32:0/2899981238 conn(0x7ff1e419b890 msgr2=0x7ff1e41a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:28.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.462 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 shutdown_connections 2021-04-09T18:26:28.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.462 7ff1ebec8700 1 -- 172.21.15.32:0/2899981238 wait complete. 2021-04-09T18:26:28.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:26:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.624 7fac38aba700 1 Processor -- start 2021-04-09T18:26:28.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.624 7fac38aba700 1 -- start start 2021-04-09T18:26:28.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac38aba700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340d7ca0 0x7fac340d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac38aba700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fac3410da00 0x7fac3410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac38aba700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3410d480 con 0x7fac3410d010 2021-04-09T18:26:28.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac38aba700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac340d9220 con 0x7fac3410da00 2021-04-09T18:26:28.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac38aba700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac34112100 con 0x7fac340d7ca0 2021-04-09T18:26:28.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3305f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fac3410da00 0x7fac3410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3305f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fac3410da00 0x7fac3410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53276/0 (socket says 172.21.15.32:53276) 2021-04-09T18:26:28.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3305f700 1 -- 172.21.15.32:0/3853249036 learned_addr learned my addr 172.21.15.32:0/3853249036 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:28.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3205d700 1 --2- 172.21.15.32:0/3853249036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340d7ca0 0x7fac340d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3305f700 1 -- 172.21.15.32:0/3853249036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340d7ca0 msgr2=0x7fac340d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:28.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3305f700 1 --2- 172.21.15.32:0/3853249036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340d7ca0 0x7fac340d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3305f700 1 -- 172.21.15.32:0/3853249036 >> v1:172.21.15.86:6789/0 conn(0x7fac3410d010 legacy=0x7fac340d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3305f700 1 -- 172.21.15.32:0/3853249036 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac340d86c0 con 0x7fac3410da00 2021-04-09T18:26:28.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3305f700 1 --2- 172.21.15.32:0/3853249036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac3410da00 0x7fac3410de00 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7fac2c006f00 tx=0x7fac2c009f10).ready entity=mon.0 client_cookie=20c4839188222178 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:28.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.625 7fac3185c700 1 -- 172.21.15.32:0/3853249036 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac2c00b3b0 con 0x7fac3410da00 2021-04-09T18:26:28.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac3185c700 1 -- 172.21.15.32:0/3853249036 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c00b730 con 0x7fac3410da00 2021-04-09T18:26:28.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac3185c700 1 -- 172.21.15.32:0/3853249036 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac2c00bc10 con 0x7fac3410da00 2021-04-09T18:26:28.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac38aba700 1 -- 172.21.15.32:0/3853249036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac3410da00 msgr2=0x7fac3410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac38aba700 1 --2- 172.21.15.32:0/3853249036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac3410da00 0x7fac3410de00 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7fac2c006f00 tx=0x7fac2c009f10).stop 2021-04-09T18:26:28.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac38aba700 1 -- 172.21.15.32:0/3853249036 shutdown_connections 2021-04-09T18:26:28.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac38aba700 1 --2- 172.21.15.32:0/3853249036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac3410da00 0x7fac3410de00 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac38aba700 1 --2- 172.21.15.32:0/3853249036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340d7ca0 0x7fac340d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac38aba700 1 -- 172.21.15.32:0/3853249036 >> 172.21.15.32:0/3853249036 conn(0x7fac340ce5f0 msgr2=0x7fac340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac38aba700 1 -- 172.21.15.32:0/3853249036 shutdown_connections 2021-04-09T18:26:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.626 7fac38aba700 1 -- 172.21.15.32:0/3853249036 wait complete. 2021-04-09T18:26:28.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac38aba700 1 Processor -- start 2021-04-09T18:26:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac38aba700 1 -- start start 2021-04-09T18:26:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac38aba700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac341a0050 0x7fac341a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac38aba700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340bdcd0 0x7fac340bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac38aba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac340bf080 0x7fac340b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac38aba700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac340bf500 con 0x7fac341a0050 2021-04-09T18:26:28.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac38aba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac340bb7b0 con 0x7fac340bf080 2021-04-09T18:26:28.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac38aba700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac340be0d0 con 0x7fac340bdcd0 2021-04-09T18:26:28.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac3285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac341a0050 0x7fac341a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac3285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac341a0050 0x7fac341a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53282/0 (socket says 172.21.15.32:53282) 2021-04-09T18:26:28.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac3285e700 1 -- 172.21.15.32:0/2742803311 learned_addr learned my addr 172.21.15.32:0/2742803311 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:28.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac3305f700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac340bf080 0x7fac340b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.647 7fac3205d700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340bdcd0 0x7fac340bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac3285e700 1 -- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340bdcd0 msgr2=0x7fac340bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac3285e700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340bdcd0 0x7fac340bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac3285e700 1 -- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac340bf080 msgr2=0x7fac340b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:28.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac3285e700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac340bf080 0x7fac340b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:28.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac3285e700 1 -- 172.21.15.32:0/2742803311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac340b1310 con 0x7fac341a0050 2021-04-09T18:26:28.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac3285e700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac341a0050 0x7fac341a0450 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 rx=0x7fac2400b530 tx=0x7fac2400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:28.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac2b7fe700 1 -- 172.21.15.32:0/2742803311 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac2400bcc0 con 0x7fac341a0050 2021-04-09T18:26:28.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac2b7fe700 1 -- 172.21.15.32:0/2742803311 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac24006020 con 0x7fac341a0050 2021-04-09T18:26:28.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac2b7fe700 1 -- 172.21.15.32:0/2742803311 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac240054a0 con 0x7fac341a0050 2021-04-09T18:26:28.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac38aba700 1 -- 172.21.15.32:0/2742803311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac340b1310 con 0x7fac341a0050 2021-04-09T18:26:28.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.648 7fac38aba700 1 -- 172.21.15.32:0/2742803311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac340b1310 con 0x7fac341a0050 2021-04-09T18:26:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.649 7fac2b7fe700 1 -- 172.21.15.32:0/2742803311 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fac24006200 con 0x7fac341a0050 2021-04-09T18:26:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.649 7fac2b7fe700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac180188c0 0x7fac1801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.649 7fac2b7fe700 1 -- 172.21.15.32:0/2742803311 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fac24032970 con 0x7fac341a0050 2021-04-09T18:26:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.649 7fac3205d700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac180188c0 0x7fac1801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:28.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.650 7fac3205d700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac180188c0 0x7fac1801adc0 crc :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:28.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.669 7fac38aba700 1 -- 172.21.15.32:0/2742803311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac340bb7b0 con 0x7fac341a0050 2021-04-09T18:26:28.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.672 7fac2b7fe700 1 -- 172.21.15.32:0/2742803311 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fac2400bcc0 con 0x7fac341a0050 2021-04-09T18:26:28.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:28.866 7fac38aba700 1 -- 172.21.15.32:0/2742803311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fac340bb7b0 con 0x7fac341a0050 2021-04-09T18:26:29.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.658 7fac2b7fe700 1 -- 172.21.15.32:0/2742803311 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v276) v1 ==== 118+0+0 (secure 0 0 0) 0x7fac24006050 con 0x7fac341a0050 2021-04-09T18:26:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 -- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac180188c0 msgr2=0x7fac1801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac180188c0 0x7fac1801adc0 crc :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 -- 172.21.15.32:0/2742803311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac341a0050 msgr2=0x7fac341a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:29.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac341a0050 0x7fac341a0450 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 rx=0x7fac2400b530 tx=0x7fac2400b8d0).stop 2021-04-09T18:26:29.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 -- 172.21.15.32:0/2742803311 shutdown_connections 2021-04-09T18:26:29.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fac180188c0 0x7fac1801adc0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac340bf080 0x7fac340b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac340bdcd0 0x7fac340bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 --2- 172.21.15.32:0/2742803311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac341a0050 0x7fac341a0450 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 -- 172.21.15.32:0/2742803311 >> 172.21.15.32:0/2742803311 conn(0x7fac340975f0 msgr2=0x7fac34096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:29.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 -- 172.21.15.32:0/2742803311 shutdown_connections 2021-04-09T18:26:29.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.685 7fac38aba700 1 -- 172.21.15.32:0/2742803311 wait complete. 2021-04-09T18:26:29.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:29.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.845 7f8505bb5700 1 Processor -- start 2021-04-09T18:26:29.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.845 7f8505bb5700 1 -- start start 2021-04-09T18:26:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f8505bb5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f850010d010 0x7f85000d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f8505bb5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f85000d1da0 0x7f85000d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f8505bb5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850010d480 con 0x7f85000d82c0 2021-04-09T18:26:29.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f8505bb5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85000d8d30 con 0x7f850010d010 2021-04-09T18:26:29.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f8505bb5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85000d9220 con 0x7f85000d1da0 2021-04-09T18:26:29.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84ff7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f850010d010 0x7f85000d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:29.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84fffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f85000d82c0 0x7f850010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34646/0 (socket says 172.21.15.32:34646) 2021-04-09T18:26:29.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84fffff700 1 -- 172.21.15.32:0/3097168031 learned_addr learned my addr 172.21.15.32:0/3097168031 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:29.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84feffd700 1 --2- 172.21.15.32:0/3097168031 >> v2:172.21.15.86:3300/0 conn(0x7f85000d1da0 0x7f85000d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:29.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84ff7fe700 1 -- 172.21.15.32:0/3097168031 >> v2:172.21.15.86:3300/0 conn(0x7f85000d1da0 msgr2=0x7f85000d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:29.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84ff7fe700 1 --2- 172.21.15.32:0/3097168031 >> v2:172.21.15.86:3300/0 conn(0x7f85000d1da0 0x7f85000d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84ff7fe700 1 -- 172.21.15.32:0/3097168031 >> v1:172.21.15.32:6789/0 conn(0x7f85000d82c0 legacy=0x7f850010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:29.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84ff7fe700 1 -- 172.21.15.32:0/3097168031 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85000d86f0 con 0x7f850010d010 2021-04-09T18:26:29.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84fe7fc700 1 -- 172.21.15.32:0/3097168031 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380365763 0 0) 0x7f84f8000d30 con 0x7f85000d82c0 2021-04-09T18:26:29.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.846 7f84ff7fe700 1 --2- 172.21.15.32:0/3097168031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850010d010 0x7f85000d1780 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 rx=0x7f84f0006650 tx=0x7f84f0009a00).ready entity=mon.0 client_cookie=211f24f702ebb9ac server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:29.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f84fe7fc700 1 -- 172.21.15.32:0/3097168031 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84f0009df0 con 0x7f850010d010 2021-04-09T18:26:29.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f84fe7fc700 1 -- 172.21.15.32:0/3097168031 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f00009b0 con 0x7f850010d010 2021-04-09T18:26:29.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f84fe7fc700 1 -- 172.21.15.32:0/3097168031 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84f000c2b0 con 0x7f850010d010 2021-04-09T18:26:29.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f8505bb5700 1 -- 172.21.15.32:0/3097168031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850010d010 msgr2=0x7f85000d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:29.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f8505bb5700 1 --2- 172.21.15.32:0/3097168031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850010d010 0x7f85000d1780 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 rx=0x7f84f0006650 tx=0x7f84f0009a00).stop 2021-04-09T18:26:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f8505bb5700 1 -- 172.21.15.32:0/3097168031 shutdown_connections 2021-04-09T18:26:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f8505bb5700 1 --2- 172.21.15.32:0/3097168031 >> v2:172.21.15.86:3300/0 conn(0x7f85000d1da0 0x7f85000d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f8505bb5700 1 --2- 172.21.15.32:0/3097168031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850010d010 0x7f85000d1780 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f8505bb5700 1 -- 172.21.15.32:0/3097168031 >> 172.21.15.32:0/3097168031 conn(0x7f85000ce5f0 msgr2=0x7f85000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:29.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f8505bb5700 1 -- 172.21.15.32:0/3097168031 shutdown_connections 2021-04-09T18:26:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.847 7f8505bb5700 1 -- 172.21.15.32:0/3097168031 wait complete. 2021-04-09T18:26:29.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f8505bb5700 1 Processor -- start 2021-04-09T18:26:29.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f8505bb5700 1 -- start start 2021-04-09T18:26:29.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f8505bb5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f850017baa0 0x7f850017bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:29.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f8505bb5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f850017c4c0 0x7f850018e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:29.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f8505bb5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850018ec80 0x7f850018f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:29.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f8505bb5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8500186840 con 0x7f850018ec80 2021-04-09T18:26:29.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f8505bb5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850018f6a0 con 0x7f850017baa0 2021-04-09T18:26:29.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f8505bb5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850018f8c0 con 0x7f850017c4c0 2021-04-09T18:26:29.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f84fffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850018ec80 0x7f850018f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:29.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f84feffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f850017c4c0 0x7f850018e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:29.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f84fffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850018ec80 0x7f850018f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53296/0 (socket says 172.21.15.32:53296) 2021-04-09T18:26:29.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.868 7f84fffff700 1 -- 172.21.15.32:0/3531006685 learned_addr learned my addr 172.21.15.32:0/3531006685 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:29.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84ff7fe700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f850017baa0 0x7f850017bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:29.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84fffff700 1 -- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f850017c4c0 msgr2=0x7f850018e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:29.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84fffff700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f850017c4c0 0x7f850018e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84fffff700 1 -- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f850017baa0 msgr2=0x7f850017bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:29.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84fffff700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f850017baa0 0x7f850017bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:29.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84fffff700 1 -- 172.21.15.32:0/3531006685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850018fae0 con 0x7f850018ec80 2021-04-09T18:26:29.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84feffd700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f850017c4c0 0x7f850018e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:29.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84ff7fe700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f850017baa0 0x7f850017bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:29.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84fffff700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850018ec80 0x7f850018f080 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7f84f8007f90 tx=0x7f84f800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:29.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84f7fff700 1 -- 172.21.15.32:0/3531006685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84f8000cc0 con 0x7f850018ec80 2021-04-09T18:26:29.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84f7fff700 1 -- 172.21.15.32:0/3531006685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f8007020 con 0x7f850018ec80 2021-04-09T18:26:29.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850018fae0 con 0x7f850018ec80 2021-04-09T18:26:29.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f84f7fff700 1 -- 172.21.15.32:0/3531006685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84f8007540 con 0x7f850018ec80 2021-04-09T18:26:29.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.869 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850018fae0 con 0x7f850018ec80 2021-04-09T18:26:29.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.870 7f84f7fff700 1 -- 172.21.15.32:0/3531006685 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f84f8007200 con 0x7f850018ec80 2021-04-09T18:26:29.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.870 7f84f7fff700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84dc018930 0x7f84dc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:29.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.870 7f84f7fff700 1 -- 172.21.15.32:0/3531006685 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f84f8032d40 con 0x7f850018ec80 2021-04-09T18:26:29.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.870 7f84ff7fe700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84dc018930 0x7f84dc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:29.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.870 7f84ff7fe700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84dc018930 0x7f84dc01ae10 crc :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:29.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.890 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85001b7080 con 0x7f850018ec80 2021-04-09T18:26:29.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:29.893 7f84f7fff700 1 -- 172.21.15.32:0/3531006685 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f84f8007d00 con 0x7f850018ec80 2021-04-09T18:26:30.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.085 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f850010d480 con 0x7f850018ec80 2021-04-09T18:26:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.086 7f84f7fff700 1 -- 172.21.15.32:0/3531006685 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 276 v276) v1 ==== 95+0+2780 (secure 0 0 0) 0x7f84f8006f20 con 0x7f850018ec80 2021-04-09T18:26:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 276 2021-04-09T18:26:30.158 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:30.158 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":276,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9762,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:29.656266","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:26:30.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84dc018930 msgr2=0x7f84dc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84dc018930 0x7f84dc01ae10 crc :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850018ec80 msgr2=0x7f850018f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850018ec80 0x7f850018f080 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7f84f8007f90 tx=0x7f84f800ac50).stop 2021-04-09T18:26:30.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 shutdown_connections 2021-04-09T18:26:30.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f84dc018930 0x7f84dc01ae10 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f850018ec80 0x7f850018f080 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f850017c4c0 0x7f850018e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 --2- 172.21.15.32:0/3531006685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f850017baa0 0x7f850017bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 >> 172.21.15.32:0/3531006685 conn(0x7f8500191560 msgr2=0x7f85001b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:30.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 shutdown_connections 2021-04-09T18:26:30.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.088 7f8505bb5700 1 -- 172.21.15.32:0/3531006685 wait complete. 2021-04-09T18:26:30.169 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:26:30.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.257 7f377acac700 1 Processor -- start 2021-04-09T18:26:30.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.257 7f377acac700 1 -- start start 2021-04-09T18:26:30.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f377acac700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f377410da00 0x7f377410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f377acac700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377410d480 con 0x7f377410d010 2021-04-09T18:26:30.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f377acac700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37740d1f70 con 0x7f37740d7ca0 2021-04-09T18:26:30.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f377acac700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37740d8d30 con 0x7f377410da00 2021-04-09T18:26:30.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f3778ac8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f377410d010 0x7f37740d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34658/0 (socket says 172.21.15.32:34658) 2021-04-09T18:26:30.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f3778ac8700 1 -- 172.21.15.32:0/2918075736 learned_addr learned my addr 172.21.15.32:0/2918075736 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:30.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f37792c9700 1 --2- 172.21.15.32:0/2918075736 >> v2:172.21.15.86:3300/0 conn(0x7f377410da00 0x7f377410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f37737fe700 1 -- 172.21.15.32:0/2918075736 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769192236 0 0) 0x7f375c000ce0 con 0x7f377410d010 2021-04-09T18:26:30.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f37737fe700 1 -- 172.21.15.32:0/2918075736 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3760002fe0 con 0x7f377410d010 2021-04-09T18:26:30.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f37737fe700 1 -- 172.21.15.32:0/2918075736 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418270674 0 0) 0x7f3764000c60 con 0x7f37740d7ca0 2021-04-09T18:26:30.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f37737fe700 1 -- 172.21.15.32:0/2918075736 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3760003210 con 0x7f37740d7ca0 2021-04-09T18:26:30.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f37792c9700 1 -- 172.21.15.32:0/2918075736 >> v1:172.21.15.86:6789/0 conn(0x7f37740d7ca0 legacy=0x7f37740d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f37792c9700 1 -- 172.21.15.32:0/2918075736 >> v1:172.21.15.32:6789/0 conn(0x7f377410d010 legacy=0x7f37740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.258 7f37792c9700 1 -- 172.21.15.32:0/2918075736 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37740d9220 con 0x7f377410da00 2021-04-09T18:26:30.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f37792c9700 1 --2- 172.21.15.32:0/2918075736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f377410da00 0x7f377410de00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f376c006e50 tx=0x7f376c009e70).ready entity=mon.1 client_cookie=22c46c9490580763 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:30.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f37737fe700 1 -- 172.21.15.32:0/2918075736 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f376c006a20 con 0x7f377410da00 2021-04-09T18:26:30.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f37737fe700 1 -- 172.21.15.32:0/2918075736 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376c006790 con 0x7f377410da00 2021-04-09T18:26:30.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f37737fe700 1 -- 172.21.15.32:0/2918075736 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f376c00b700 con 0x7f377410da00 2021-04-09T18:26:30.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f377acac700 1 -- 172.21.15.32:0/2918075736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f377410da00 msgr2=0x7f377410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f377acac700 1 --2- 172.21.15.32:0/2918075736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f377410da00 0x7f377410de00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f376c006e50 tx=0x7f376c009e70).stop 2021-04-09T18:26:30.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f377acac700 1 -- 172.21.15.32:0/2918075736 shutdown_connections 2021-04-09T18:26:30.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f377acac700 1 --2- 172.21.15.32:0/2918075736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f377410da00 0x7f377410de00 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f377acac700 1 -- 172.21.15.32:0/2918075736 >> 172.21.15.32:0/2918075736 conn(0x7f37740ce5f0 msgr2=0x7f37740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:30.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f377acac700 1 -- 172.21.15.32:0/2918075736 shutdown_connections 2021-04-09T18:26:30.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.259 7f377acac700 1 -- 172.21.15.32:0/2918075736 wait complete. 2021-04-09T18:26:30.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f377acac700 1 Processor -- start 2021-04-09T18:26:30.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f377acac700 1 -- start start 2021-04-09T18:26:30.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f377acac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3774185a00 0x7f3774097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f377acac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37740983b0 0x7f377417abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f377acac700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f377417b1d0 0x7f3774179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f377acac700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37740987b0 con 0x7f37740983b0 2021-04-09T18:26:30.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f377acac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377417b5d0 con 0x7f3774185a00 2021-04-09T18:26:30.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f377acac700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3774179c30 con 0x7f377417b1d0 2021-04-09T18:26:30.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f3773fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37740983b0 0x7f377417abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f3773fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37740983b0 0x7f377417abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53306/0 (socket says 172.21.15.32:53306) 2021-04-09T18:26:30.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f3773fff700 1 -- 172.21.15.32:0/812028382 learned_addr learned my addr 172.21.15.32:0/812028382 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:30.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f3778ac8700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3774185a00 0x7f3774097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.280 7f37792c9700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f377417b1d0 0x7f3774179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f3773fff700 1 -- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f377417b1d0 msgr2=0x7f3774179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f3773fff700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f377417b1d0 0x7f3774179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f3773fff700 1 -- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3774185a00 msgr2=0x7f3774097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f3773fff700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3774185a00 0x7f3774097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f3773fff700 1 -- 172.21.15.32:0/812028382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3774179e50 con 0x7f37740983b0 2021-04-09T18:26:30.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f3773fff700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37740983b0 0x7f377417abb0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7f376400b580 tx=0x7f376400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:30.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f37717fa700 1 -- 172.21.15.32:0/812028382 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f376400bd10 con 0x7f37740983b0 2021-04-09T18:26:30.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f37717fa700 1 -- 172.21.15.32:0/812028382 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3764000e20 con 0x7f37740983b0 2021-04-09T18:26:30.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f377acac700 1 -- 172.21.15.32:0/812028382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3774179e50 con 0x7f37740983b0 2021-04-09T18:26:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f37717fa700 1 -- 172.21.15.32:0/812028382 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37640054f0 con 0x7f37740983b0 2021-04-09T18:26:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.281 7f377acac700 1 -- 172.21.15.32:0/812028382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3774179e50 con 0x7f37740983b0 2021-04-09T18:26:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.282 7f37717fa700 1 -- 172.21.15.32:0/812028382 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3764009fe0 con 0x7f37740983b0 2021-04-09T18:26:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.282 7f37717fa700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3758018930 0x7f375801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.282 7f37717fa700 1 -- 172.21.15.32:0/812028382 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3764032a50 con 0x7f37740983b0 2021-04-09T18:26:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.282 7f3778ac8700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3758018930 0x7f375801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.282 7f3778ac8700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3758018930 0x7f375801ae10 crc :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:30.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.302 7f377acac700 1 -- 172.21.15.32:0/812028382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377417a3e0 con 0x7f37740983b0 2021-04-09T18:26:30.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.305 7f37717fa700 1 -- 172.21.15.32:0/812028382 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3764000e20 con 0x7f37740983b0 2021-04-09T18:26:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.495 7f377acac700 1 -- 172.21.15.32:0/812028382 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37740987b0 con 0x7f37740983b0 2021-04-09T18:26:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.496 7f37717fa700 1 -- 172.21.15.32:0/812028382 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f3764009fe0 con 0x7f37740983b0 2021-04-09T18:26:30.571 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:30.571 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":142,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:26:28.652397","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:26:25.044737","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-04-09 18:26:26.413903","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:26:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 -- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3758018930 msgr2=0x7f375801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3758018930 0x7f375801ae10 crc :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 -- 172.21.15.32:0/812028382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37740983b0 msgr2=0x7f377417abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37740983b0 0x7f377417abb0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7f376400b580 tx=0x7f376400b920).stop 2021-04-09T18:26:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 -- 172.21.15.32:0/812028382 shutdown_connections 2021-04-09T18:26:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3758018930 0x7f375801ae10 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37740983b0 0x7f377417abb0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f377417b1d0 0x7f3774179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 --2- 172.21.15.32:0/812028382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3774185a00 0x7f3774097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 -- 172.21.15.32:0/812028382 >> 172.21.15.32:0/812028382 conn(0x7f3774190550 msgr2=0x7f37741b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 -- 172.21.15.32:0/812028382 shutdown_connections 2021-04-09T18:26:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.498 7f377acac700 1 -- 172.21.15.32:0/812028382 wait complete. 2021-04-09T18:26:30.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:26:30.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.684 7fae6ad4b700 1 Processor -- start 2021-04-09T18:26:30.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.685 7fae6ad4b700 1 -- start start 2021-04-09T18:26:30.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.685 7fae6ad4b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae640d7c50 0x7fae641b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.685 7fae6ad4b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae640d8620 con 0x7fae640cffb0 2021-04-09T18:26:30.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.685 7fae6ad4b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae640d2800 con 0x7fae640d8050 2021-04-09T18:26:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.685 7fae6ad4b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae640d91f0 con 0x7fae640d7c50 2021-04-09T18:26:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.685 7fae68b67700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fae640cffb0 0x7fae64087e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34670/0 (socket says 172.21.15.32:34670) 2021-04-09T18:26:30.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.685 7fae68b67700 1 -- 172.21.15.32:0/223498482 learned_addr learned my addr 172.21.15.32:0/223498482 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:30.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.685 7fae63fff700 1 --2- 172.21.15.32:0/223498482 >> v2:172.21.15.86:3300/0 conn(0x7fae640d7c50 0x7fae641b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae637fe700 1 -- 172.21.15.32:0/223498482 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641809804 0 0) 0x7fae4c000ce0 con 0x7fae640cffb0 2021-04-09T18:26:30.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae637fe700 1 -- 172.21.15.32:0/223498482 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae50002fe0 con 0x7fae640cffb0 2021-04-09T18:26:30.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae63fff700 1 -- 172.21.15.32:0/223498482 >> v1:172.21.15.86:6789/0 conn(0x7fae640d8050 legacy=0x7fae6410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae637fe700 1 -- 172.21.15.32:0/223498482 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084710008 0 0) 0x7fae5c000d30 con 0x7fae640d8050 2021-04-09T18:26:30.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae63fff700 1 -- 172.21.15.32:0/223498482 >> v1:172.21.15.32:6789/0 conn(0x7fae640cffb0 legacy=0x7fae64087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae63fff700 1 -- 172.21.15.32:0/223498482 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae641b0ab0 con 0x7fae640d7c50 2021-04-09T18:26:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae63fff700 1 --2- 172.21.15.32:0/223498482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640d7c50 0x7fae641b0490 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fae54006650 tx=0x7fae54009a00).ready entity=mon.1 client_cookie=8fe4844b87c33f77 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae637fe700 1 -- 172.21.15.32:0/223498482 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae54009df0 con 0x7fae640d7c50 2021-04-09T18:26:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae637fe700 1 -- 172.21.15.32:0/223498482 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae540009b0 con 0x7fae640d7c50 2021-04-09T18:26:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae637fe700 1 -- 172.21.15.32:0/223498482 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae5400c2b0 con 0x7fae640d7c50 2021-04-09T18:26:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae6ad4b700 1 -- 172.21.15.32:0/223498482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640d7c50 msgr2=0x7fae641b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.686 7fae6ad4b700 1 --2- 172.21.15.32:0/223498482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640d7c50 0x7fae641b0490 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fae54006650 tx=0x7fae54009a00).stop 2021-04-09T18:26:30.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.687 7fae6ad4b700 1 -- 172.21.15.32:0/223498482 shutdown_connections 2021-04-09T18:26:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.687 7fae6ad4b700 1 --2- 172.21.15.32:0/223498482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640d7c50 0x7fae641b0490 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.687 7fae6ad4b700 1 -- 172.21.15.32:0/223498482 >> 172.21.15.32:0/223498482 conn(0x7fae640ce5b0 msgr2=0x7fae640cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:30.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.687 7fae6ad4b700 1 -- 172.21.15.32:0/223498482 shutdown_connections 2021-04-09T18:26:30.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.687 7fae6ad4b700 1 -- 172.21.15.32:0/223498482 wait complete. 2021-04-09T18:26:30.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.707 7fae6ad4b700 1 Processor -- start 2021-04-09T18:26:30.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.707 7fae6ad4b700 1 -- start start 2021-04-09T18:26:30.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae6ad4b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640c0f30 0x7fae640a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae6ad4b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640a9f20 0x7fae640a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae6ad4b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae640a1b50 0x7fae640a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae6ad4b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640a9380 con 0x7fae640c0f30 2021-04-09T18:26:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae6ad4b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640aa320 con 0x7fae640a9f20 2021-04-09T18:26:30.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae6ad4b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae640c1330 con 0x7fae640a1b50 2021-04-09T18:26:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae68b67700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640c0f30 0x7fae640a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae68b67700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640c0f30 0x7fae640a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53318/0 (socket says 172.21.15.32:53318) 2021-04-09T18:26:30.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae68b67700 1 -- 172.21.15.32:0/2484890357 learned_addr learned my addr 172.21.15.32:0/2484890357 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:30.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae63fff700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640a9f20 0x7fae640a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae69368700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae640a1b50 0x7fae640a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae63fff700 1 -- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae640a1b50 msgr2=0x7fae640a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae63fff700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae640a1b50 0x7fae640a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae63fff700 1 -- 172.21.15.32:0/2484890357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640c0f30 msgr2=0x7fae640a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae63fff700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640c0f30 0x7fae640a9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae63fff700 1 -- 172.21.15.32:0/2484890357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640a1f50 con 0x7fae640a9f20 2021-04-09T18:26:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae68b67700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640c0f30 0x7fae640a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:30.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.708 7fae63fff700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640a9f20 0x7fae640a1530 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fae5400b510 tx=0x7fae5400b8b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:30.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.709 7fae617fa700 1 -- 172.21.15.32:0/2484890357 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae54000940 con 0x7fae640a9f20 2021-04-09T18:26:30.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.709 7fae617fa700 1 -- 172.21.15.32:0/2484890357 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5400bda0 con 0x7fae640a9f20 2021-04-09T18:26:30.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.709 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae640a1f50 con 0x7fae640a9f20 2021-04-09T18:26:30.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.709 7fae617fa700 1 -- 172.21.15.32:0/2484890357 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae54002030 con 0x7fae640a9f20 2021-04-09T18:26:30.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.709 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae640a1f50 con 0x7fae640a9f20 2021-04-09T18:26:30.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.709 7fae617fa700 1 -- 172.21.15.32:0/2484890357 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fae54009280 con 0x7fae640a9f20 2021-04-09T18:26:30.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.709 7fae617fa700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae48018930 0x7fae4801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:30.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.709 7fae617fa700 1 -- 172.21.15.32:0/2484890357 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fae54032900 con 0x7fae640a9f20 2021-04-09T18:26:30.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.710 7fae68b67700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae48018930 0x7fae4801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:30.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.710 7fae68b67700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae48018930 0x7fae4801adc0 crc :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:30.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.729 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae64093ff0 con 0x7fae640a9f20 2021-04-09T18:26:30.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.733 7fae617fa700 1 -- 172.21.15.32:0/2484890357 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fae54000940 con 0x7fae640a9f20 2021-04-09T18:26:30.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.917 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fae640d8cd0 con 0x7fae640a9f20 2021-04-09T18:26:30.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.924 7fae617fa700 1 -- 172.21.15.32:0/2484890357 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v34) v1 ==== 184+0+0 (secure 0 0 0) 0x7fae54009280 con 0x7fae640a9f20 2021-04-09T18:26:30.996 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:26:30.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae48018930 msgr2=0x7fae4801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae48018930 0x7fae4801adc0 crc :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640a9f20 msgr2=0x7fae640a1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640a9f20 0x7fae640a1530 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fae5400b510 tx=0x7fae5400b8b0).stop 2021-04-09T18:26:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 shutdown_connections 2021-04-09T18:26:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fae48018930 0x7fae4801adc0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:30.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae640a1b50 0x7fae640a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae640a9f20 0x7fae640a1530 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 --2- 172.21.15.32:0/2484890357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae640c0f30 0x7fae640a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 >> 172.21.15.32:0/2484890357 conn(0x7fae640adb50 msgr2=0x7fae640ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:31.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 shutdown_connections 2021-04-09T18:26:31.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:30.926 7fae6ad4b700 1 -- 172.21.15.32:0/2484890357 wait complete. 2021-04-09T18:26:31.009 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:26:31.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.100 7f68f2261700 1 Processor -- start 2021-04-09T18:26:31.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68f2261700 1 -- start start 2021-04-09T18:26:31.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68f2261700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f68ec0d7c50 0x7f68ec1b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68f2261700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec0d8050 0x7f68ec10d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68f2261700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec0d8620 con 0x7f68ec0cffb0 2021-04-09T18:26:31.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68f2261700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ec0d91f0 con 0x7f68ec0d7c50 2021-04-09T18:26:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68f2261700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec0d2800 con 0x7f68ec0d8050 2021-04-09T18:26:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68e3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f68ec0d7c50 0x7f68ec1b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68e3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f68ec0d7c50 0x7f68ec1b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53328/0 (socket says 172.21.15.32:53328) 2021-04-09T18:26:31.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68e3fff700 1 -- 172.21.15.32:0/2072912680 learned_addr learned my addr 172.21.15.32:0/2072912680 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:31.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.101 7f68f087e700 1 --2- 172.21.15.32:0/2072912680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec0d8050 0x7f68ec10d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68e3fff700 1 -- 172.21.15.32:0/2072912680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec0d8050 msgr2=0x7f68ec10d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:31.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68e3fff700 1 --2- 172.21.15.32:0/2072912680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec0d8050 0x7f68ec10d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68e3fff700 1 -- 172.21.15.32:0/2072912680 >> v1:172.21.15.86:6789/0 conn(0x7f68ec0cffb0 legacy=0x7f68ec087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68e3fff700 1 -- 172.21.15.32:0/2072912680 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec1b0ab0 con 0x7f68ec0d7c50 2021-04-09T18:26:31.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68e3fff700 1 --2- 172.21.15.32:0/2072912680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec0d7c50 0x7f68ec1b0490 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7f68d4006700 tx=0x7f68d4009b80).ready entity=mon.0 client_cookie=de61a3d43f11c8a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:31.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68eb7fe700 1 -- 172.21.15.32:0/2072912680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68d400c0c0 con 0x7f68ec0d7c50 2021-04-09T18:26:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68eb7fe700 1 -- 172.21.15.32:0/2072912680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d400c440 con 0x7f68ec0d7c50 2021-04-09T18:26:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68eb7fe700 1 -- 172.21.15.32:0/2072912680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68d400c920 con 0x7f68ec0d7c50 2021-04-09T18:26:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68f2261700 1 -- 172.21.15.32:0/2072912680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec0d7c50 msgr2=0x7f68ec1b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.102 7f68f2261700 1 --2- 172.21.15.32:0/2072912680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec0d7c50 0x7f68ec1b0490 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7f68d4006700 tx=0x7f68d4009b80).stop 2021-04-09T18:26:31.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.103 7f68f2261700 1 -- 172.21.15.32:0/2072912680 shutdown_connections 2021-04-09T18:26:31.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.103 7f68f2261700 1 --2- 172.21.15.32:0/2072912680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec0d8050 0x7f68ec10d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.103 7f68f2261700 1 --2- 172.21.15.32:0/2072912680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec0d7c50 0x7f68ec1b0490 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.103 7f68f2261700 1 -- 172.21.15.32:0/2072912680 >> 172.21.15.32:0/2072912680 conn(0x7f68ec0ce5b0 msgr2=0x7f68ec0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.103 7f68f2261700 1 -- 172.21.15.32:0/2072912680 shutdown_connections 2021-04-09T18:26:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.103 7f68f2261700 1 -- 172.21.15.32:0/2072912680 wait complete. 2021-04-09T18:26:31.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.123 7f68f2261700 1 Processor -- start 2021-04-09T18:26:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.123 7f68f2261700 1 -- start start 2021-04-09T18:26:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68f2261700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec09d680 0x7f68ec09afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68f2261700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec09da80 0x7f68ec191dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68f2261700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ec1923f0 0x7f68ec09be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68f2261700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec09cbe0 con 0x7f68ec09da80 2021-04-09T18:26:31.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68f2261700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec09b8e0 con 0x7f68ec1923f0 2021-04-09T18:26:31.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68f2261700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec09b5c0 con 0x7f68ec09d680 2021-04-09T18:26:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec09da80 0x7f68ec191dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec09da80 0x7f68ec191dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53334/0 (socket says 172.21.15.32:53334) 2021-04-09T18:26:31.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e3fff700 1 -- 172.21.15.32:0/1879252854 learned_addr learned my addr 172.21.15.32:0/1879252854 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68ebfff700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec09d680 0x7f68ec09afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68f087e700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ec1923f0 0x7f68ec09be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e3fff700 1 -- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec09d680 msgr2=0x7f68ec09afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e3fff700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec09d680 0x7f68ec09afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e3fff700 1 -- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ec1923f0 msgr2=0x7f68ec09be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e3fff700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ec1923f0 0x7f68ec09be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e3fff700 1 -- 172.21.15.32:0/1879252854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec09c480 con 0x7f68ec09da80 2021-04-09T18:26:31.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e3fff700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec09da80 0x7f68ec191dd0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7f68d400b580 tx=0x7f68d400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:31.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e97fa700 1 -- 172.21.15.32:0/1879252854 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68d400bd10 con 0x7f68ec09da80 2021-04-09T18:26:31.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68e97fa700 1 -- 172.21.15.32:0/1879252854 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d4000c90 con 0x7f68ec09da80 2021-04-09T18:26:31.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.124 7f68f2261700 1 -- 172.21.15.32:0/1879252854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec09c480 con 0x7f68ec09da80 2021-04-09T18:26:31.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.125 7f68e97fa700 1 -- 172.21.15.32:0/1879252854 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68d4002220 con 0x7f68ec09da80 2021-04-09T18:26:31.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.125 7f68f2261700 1 -- 172.21.15.32:0/1879252854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec09c480 con 0x7f68ec09da80 2021-04-09T18:26:31.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.125 7f68e97fa700 1 -- 172.21.15.32:0/1879252854 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f68d4009f90 con 0x7f68ec09da80 2021-04-09T18:26:31.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.125 7f68e97fa700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68d0018930 0x7f68d001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.125 7f68e97fa700 1 -- 172.21.15.32:0/1879252854 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f68d40329e0 con 0x7f68ec09da80 2021-04-09T18:26:31.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.125 7f68ebfff700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68d0018930 0x7f68d001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.126 7f68ebfff700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68d0018930 0x7f68d001adc0 crc :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:31.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.145 7f68f2261700 1 -- 172.21.15.32:0/1879252854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec093fc0 con 0x7f68ec09da80 2021-04-09T18:26:31.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.149 7f68e97fa700 1 -- 172.21.15.32:0/1879252854 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f68d40091e0 con 0x7f68ec09da80 2021-04-09T18:26:31.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.333 7f68f2261700 1 -- 172.21.15.32:0/1879252854 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f68ec093fc0 con 0x7f68ec09da80 2021-04-09T18:26:31.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.339 7f68e97fa700 1 -- 172.21.15.32:0/1879252854 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v35) v1 ==== 184+0+0 (secure 0 0 0) 0x7f68d4000c90 con 0x7f68ec09da80 2021-04-09T18:26:31.412 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:26:31.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 -- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68d0018930 msgr2=0x7f68d001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68d0018930 0x7f68d001adc0 crc :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 -- 172.21.15.32:0/1879252854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec09da80 msgr2=0x7f68ec191dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec09da80 0x7f68ec191dd0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7f68d400b580 tx=0x7f68d400b920).stop 2021-04-09T18:26:31.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 -- 172.21.15.32:0/1879252854 shutdown_connections 2021-04-09T18:26:31.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68d0018930 0x7f68d001adc0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68ec09da80 0x7f68ec191dd0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68ec1923f0 0x7f68ec09be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 --2- 172.21.15.32:0/1879252854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68ec09d680 0x7f68ec09afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.342 7f68f2261700 1 -- 172.21.15.32:0/1879252854 >> 172.21.15.32:0/1879252854 conn(0x7f68ec0adaf0 msgr2=0x7f68ec0bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:31.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.343 7f68f2261700 1 -- 172.21.15.32:0/1879252854 shutdown_connections 2021-04-09T18:26:31.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.343 7f68f2261700 1 -- 172.21.15.32:0/1879252854 wait complete. 2021-04-09T18:26:31.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.500 7f8cbf770700 1 Processor -- start 2021-04-09T18:26:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.500 7f8cbf770700 1 -- start start 2021-04-09T18:26:31.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbf770700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8cb80cffa0 0x7f8cb80d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbf770700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80d7280 con 0x7f8cb80d7800 2021-04-09T18:26:31.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbf770700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80d83f0 con 0x7f8cb80d6c80 2021-04-09T18:26:31.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbf770700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb80d7080 con 0x7f8cb80cffa0 2021-04-09T18:26:31.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbd58c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8cb80cffa0 0x7f8cb80d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbcd8b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8cb80d7800 0x7f8cb80d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34700/0 (socket says 172.21.15.32:34700) 2021-04-09T18:26:31.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbcd8b700 1 -- 172.21.15.32:0/1100873480 learned_addr learned my addr 172.21.15.32:0/1100873480 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:31.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbd58c700 1 -- 172.21.15.32:0/1100873480 >> v1:172.21.15.86:6789/0 conn(0x7f8cb80d6c80 legacy=0x7f8cb810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbd58c700 1 -- 172.21.15.32:0/1100873480 >> v1:172.21.15.32:6789/0 conn(0x7f8cb80d7800 legacy=0x7f8cb80d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cb7fff700 1 -- 172.21.15.32:0/1100873480 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772539559 0 0) 0x7f8cb00009b0 con 0x7f8cb80d6c80 2021-04-09T18:26:31.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbd58c700 1 -- 172.21.15.32:0/1100873480 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80d91e0 con 0x7f8cb80cffa0 2021-04-09T18:26:31.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.501 7f8cbd58c700 1 --2- 172.21.15.32:0/1100873480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80cffa0 0x7f8cb80d2440 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 rx=0x7f8ca80069d0 tx=0x7f8ca8009e40).ready entity=mon.0 client_cookie=e70b958336deffff server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:31.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.502 7f8cb7fff700 1 -- 172.21.15.32:0/1100873480 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ca800b510 con 0x7f8cb80cffa0 2021-04-09T18:26:31.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.502 7f8cb7fff700 1 -- 172.21.15.32:0/1100873480 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca800b890 con 0x7f8cb80cffa0 2021-04-09T18:26:31.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.502 7f8cb7fff700 1 -- 172.21.15.32:0/1100873480 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ca800bd70 con 0x7f8cb80cffa0 2021-04-09T18:26:31.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.502 7f8cbf770700 1 -- 172.21.15.32:0/1100873480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80cffa0 msgr2=0x7f8cb80d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.502 7f8cbf770700 1 --2- 172.21.15.32:0/1100873480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80cffa0 0x7f8cb80d2440 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 rx=0x7f8ca80069d0 tx=0x7f8ca8009e40).stop 2021-04-09T18:26:31.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.502 7f8cbf770700 1 -- 172.21.15.32:0/1100873480 shutdown_connections 2021-04-09T18:26:31.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.502 7f8cbf770700 1 --2- 172.21.15.32:0/1100873480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80cffa0 0x7f8cb80d2440 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.502 7f8cbf770700 1 -- 172.21.15.32:0/1100873480 >> 172.21.15.32:0/1100873480 conn(0x7f8cb80ce5c0 msgr2=0x7f8cb80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:31.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.503 7f8cbf770700 1 -- 172.21.15.32:0/1100873480 shutdown_connections 2021-04-09T18:26:31.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.503 7f8cbf770700 1 -- 172.21.15.32:0/1100873480 wait complete. 2021-04-09T18:26:31.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.523 7f8cbf770700 1 Processor -- start 2021-04-09T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.523 7f8cbf770700 1 -- start start 2021-04-09T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.523 7f8cbf770700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cb81acc70 0x7f8cb819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.523 7f8cbf770700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80adac0 0x7f8cb80adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.523 7f8cbf770700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cb81a6250 0x7f8cb81a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.523 7f8cbf770700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb81a6f90 con 0x7f8cb80adac0 2021-04-09T18:26:31.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.523 7f8cbf770700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb81a6c70 con 0x7f8cb81a6250 2021-04-09T18:26:31.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbf770700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb81961e0 con 0x7f8cb81acc70 2021-04-09T18:26:31.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbcd8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80adac0 0x7f8cb80adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbcd8b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80adac0 0x7f8cb80adec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53344/0 (socket says 172.21.15.32:53344) 2021-04-09T18:26:31.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbcd8b700 1 -- 172.21.15.32:0/86236130 learned_addr learned my addr 172.21.15.32:0/86236130 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:31.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbdd8d700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cb81a6250 0x7f8cb81a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbd58c700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cb81acc70 0x7f8cb819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbcd8b700 1 -- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cb81acc70 msgr2=0x7f8cb819b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbcd8b700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cb81acc70 0x7f8cb819b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbcd8b700 1 -- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cb81a6250 msgr2=0x7f8cb81a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbcd8b700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cb81a6250 0x7f8cb81a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbcd8b700 1 -- 172.21.15.32:0/86236130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb81ac890 con 0x7f8cb80adac0 2021-04-09T18:26:31.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbcd8b700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80adac0 0x7f8cb80adec0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7f8cac001c60 tx=0x7f8cac00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cb5ffb700 1 -- 172.21.15.32:0/86236130 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cac00acc0 con 0x7f8cb80adac0 2021-04-09T18:26:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cb5ffb700 1 -- 172.21.15.32:0/86236130 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac004140 con 0x7f8cb80adac0 2021-04-09T18:26:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbf770700 1 -- 172.21.15.32:0/86236130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb81ac890 con 0x7f8cb80adac0 2021-04-09T18:26:31.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cb5ffb700 1 -- 172.21.15.32:0/86236130 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cac004620 con 0x7f8cb80adac0 2021-04-09T18:26:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.524 7f8cbf770700 1 -- 172.21.15.32:0/86236130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb81ac890 con 0x7f8cb80adac0 2021-04-09T18:26:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.525 7f8cb5ffb700 1 -- 172.21.15.32:0/86236130 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8cac000db0 con 0x7f8cb80adac0 2021-04-09T18:26:31.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.525 7f8cb5ffb700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c98018930 0x7f8c9801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:31.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.525 7f8cb5ffb700 1 -- 172.21.15.32:0/86236130 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8cac0329f0 con 0x7f8cb80adac0 2021-04-09T18:26:31.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.525 7f8cbd58c700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c98018930 0x7f8c9801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:31.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.525 7f8cbd58c700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c98018930 0x7f8c9801ae10 crc :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:31.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.545 7f8cbf770700 1 -- 172.21.15.32:0/86236130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb817cb90 con 0x7f8cb80adac0 2021-04-09T18:26:31.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.549 7f8cb5ffb700 1 -- 172.21.15.32:0/86236130 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8cac0009b0 con 0x7f8cb80adac0 2021-04-09T18:26:31.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.741 7f8cbf770700 1 -- 172.21.15.32:0/86236130 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cb817cb90 con 0x7f8cb80adac0 2021-04-09T18:26:31.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.742 7f8cb5ffb700 1 -- 172.21.15.32:0/86236130 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 276 v276) v1 ==== 95+0+2780 (secure 0 0 0) 0x7f8cac000db0 con 0x7f8cb80adac0 2021-04-09T18:26:31.814 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 276 2021-04-09T18:26:31.814 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:31.815 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":276,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9762,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:29.656266","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:26:31.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 -- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c98018930 msgr2=0x7f8c9801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c98018930 0x7f8c9801ae10 crc :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 -- 172.21.15.32:0/86236130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80adac0 msgr2=0x7f8cb80adec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:31.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80adac0 0x7f8cb80adec0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7f8cac001c60 tx=0x7f8cac00a8d0).stop 2021-04-09T18:26:31.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 -- 172.21.15.32:0/86236130 shutdown_connections 2021-04-09T18:26:31.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c98018930 0x7f8c9801ae10 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cb80adac0 0x7f8cb80adec0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cb81a6250 0x7f8cb81a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 --2- 172.21.15.32:0/86236130 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cb81acc70 0x7f8cb819b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:31.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 -- 172.21.15.32:0/86236130 >> 172.21.15.32:0/86236130 conn(0x7f8cb81a5900 msgr2=0x7f8cb818d340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:31.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 -- 172.21.15.32:0/86236130 shutdown_connections 2021-04-09T18:26:31.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:31.744 7f8cbf770700 1 -- 172.21.15.32:0/86236130 wait complete. 2021-04-09T18:26:32.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 276, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:26:27.651041', 'modified': '2021-04-09 18:26:29.656266', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9466': {'gid': 9466, 'name': 'd', 'rank': 0, 'incarnation': 273, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/630282836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 630282836}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 630282836}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:26:32.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:26:32.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:26:32.357 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:26:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:26:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:26:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 9466, 2021-04-09T18:26:32.359 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:26:32.359 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:26:32.360 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.6340192470000003, 2021-04-09T18:26:32.360 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 276, 2021-04-09T18:26:32.360 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 142, 2021-04-09T18:26:32.361 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 142, 2021-04-09T18:26:32.361 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10.084422007000001 2021-04-09T18:26:32.361 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:26:32.363 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 9466, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.6340192470000003, "mdsmap_epoch": 276, "osdmap_epoch": 142, "osdmap_epoch_barrier": 142, "uptime": 10.084422007000001 } 2021-04-09T18:26:32.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:26:32.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.441 7f387efc9700 1 Processor -- start 2021-04-09T18:26:32.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.442 7f387efc9700 1 -- start start 2021-04-09T18:26:32.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.442 7f387efc9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38780d7ca0 0x7f38780d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.442 7f387efc9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387810d480 con 0x7f387810d010 2021-04-09T18:26:32.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.442 7f387efc9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38780d86c0 con 0x7f387810da00 2021-04-09T18:26:32.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.442 7f387efc9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38780d9220 con 0x7f38780d7ca0 2021-04-09T18:26:32.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.442 7f3877fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38780d7ca0 0x7f38780d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.442 7f387cde5700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f387810d010 0x7f38780d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34710/0 (socket says 172.21.15.32:34710) 2021-04-09T18:26:32.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.442 7f387cde5700 1 -- 172.21.15.32:0/3288466702 learned_addr learned my addr 172.21.15.32:0/3288466702 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:32.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f3877fff700 1 -- 172.21.15.32:0/3288466702 >> v1:172.21.15.86:6789/0 conn(0x7f387810da00 legacy=0x7f387810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f3877fff700 1 -- 172.21.15.32:0/3288466702 >> v1:172.21.15.32:6789/0 conn(0x7f387810d010 legacy=0x7f38780d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f38777fe700 1 -- 172.21.15.32:0/3288466702 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915623873 0 0) 0x7f3860000ce0 con 0x7f387810d010 2021-04-09T18:26:32.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f3877fff700 1 -- 172.21.15.32:0/3288466702 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38780d1f70 con 0x7f38780d7ca0 2021-04-09T18:26:32.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f3877fff700 1 --2- 172.21.15.32:0/3288466702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38780d7ca0 0x7f38780d80a0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 rx=0x7f38680069d0 tx=0x7f3868009dc0).ready entity=mon.0 client_cookie=f9a49beee564324b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:32.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f38777fe700 1 -- 172.21.15.32:0/3288466702 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f386800b510 con 0x7f38780d7ca0 2021-04-09T18:26:32.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f38777fe700 1 -- 172.21.15.32:0/3288466702 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3868006ac0 con 0x7f38780d7ca0 2021-04-09T18:26:32.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f38777fe700 1 -- 172.21.15.32:0/3288466702 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3868000d30 con 0x7f38780d7ca0 2021-04-09T18:26:32.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f387efc9700 1 -- 172.21.15.32:0/3288466702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38780d7ca0 msgr2=0x7f38780d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.443 7f387efc9700 1 --2- 172.21.15.32:0/3288466702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38780d7ca0 0x7f38780d80a0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 rx=0x7f38680069d0 tx=0x7f3868009dc0).stop 2021-04-09T18:26:32.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.444 7f387efc9700 1 -- 172.21.15.32:0/3288466702 shutdown_connections 2021-04-09T18:26:32.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.444 7f387efc9700 1 --2- 172.21.15.32:0/3288466702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38780d7ca0 0x7f38780d80a0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.444 7f387efc9700 1 -- 172.21.15.32:0/3288466702 >> 172.21.15.32:0/3288466702 conn(0x7f38780ce5f0 msgr2=0x7f38780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:32.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.444 7f387efc9700 1 -- 172.21.15.32:0/3288466702 shutdown_connections 2021-04-09T18:26:32.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.444 7f387efc9700 1 -- 172.21.15.32:0/3288466702 wait complete. 2021-04-09T18:26:32.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.464 7f387efc9700 1 Processor -- start 2021-04-09T18:26:32.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.464 7f387efc9700 1 -- start start 2021-04-09T18:26:32.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.464 7f387efc9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38781a9b10 0x7f387817e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.464 7f387efc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3878182760 0x7f387817e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f387efc9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818ad70 0x7f387818b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f387efc9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387817edf0 con 0x7f3878182760 2021-04-09T18:26:32.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f387efc9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38781884b0 con 0x7f387818ad70 2021-04-09T18:26:32.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f387efc9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387818b790 con 0x7f38781a9b10 2021-04-09T18:26:32.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f3877fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3878182760 0x7f387817e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f3877fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3878182760 0x7f387817e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53360/0 (socket says 172.21.15.32:53360) 2021-04-09T18:26:32.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f3877fff700 1 -- 172.21.15.32:0/1868154074 learned_addr learned my addr 172.21.15.32:0/1868154074 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:32.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f387d5e6700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818ad70 0x7f387818b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f387cde5700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38781a9b10 0x7f387817e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f3877fff700 1 -- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38781a9b10 msgr2=0x7f387817e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f3877fff700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38781a9b10 0x7f387817e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f3877fff700 1 -- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818ad70 msgr2=0x7f387818b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f3877fff700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818ad70 0x7f387818b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f3877fff700 1 -- 172.21.15.32:0/1868154074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387818b9b0 con 0x7f3878182760 2021-04-09T18:26:32.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f387cde5700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38781a9b10 0x7f387817e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:32.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f3877fff700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3878182760 0x7f387817e7d0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7f38680099e0 tx=0x7f386800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:32.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f38757fa700 1 -- 172.21.15.32:0/1868154074 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3868006030 con 0x7f3878182760 2021-04-09T18:26:32.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f38757fa700 1 -- 172.21.15.32:0/1868154074 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38680062b0 con 0x7f3878182760 2021-04-09T18:26:32.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f387efc9700 1 -- 172.21.15.32:0/1868154074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387818b9b0 con 0x7f3878182760 2021-04-09T18:26:32.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f38757fa700 1 -- 172.21.15.32:0/1868154074 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3868005660 con 0x7f3878182760 2021-04-09T18:26:32.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.465 7f387efc9700 1 -- 172.21.15.32:0/1868154074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387818b9b0 con 0x7f3878182760 2021-04-09T18:26:32.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.466 7f38757fa700 1 -- 172.21.15.32:0/1868154074 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3868006490 con 0x7f3878182760 2021-04-09T18:26:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.466 7f38757fa700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f385c018930 0x7f385c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.466 7f38757fa700 1 -- 172.21.15.32:0/1868154074 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3868032e60 con 0x7f3878182760 2021-04-09T18:26:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.466 7f387cde5700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f385c018930 0x7f385c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.467 7f387cde5700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f385c018930 0x7f385c01adc0 crc :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:32.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.486 7f387efc9700 1 -- 172.21.15.32:0/1868154074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387810d480 con 0x7f3878182760 2021-04-09T18:26:32.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.490 7f38757fa700 1 -- 172.21.15.32:0/1868154074 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f386800be10 con 0x7f3878182760 2021-04-09T18:26:32.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.680 7f387efc9700 1 -- 172.21.15.32:0/1868154074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f387810d480 con 0x7f3878182760 2021-04-09T18:26:32.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.681 7f38757fa700 1 -- 172.21.15.32:0/1868154074 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f38680062b0 con 0x7f3878182760 2021-04-09T18:26:32.754 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:32.754 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":142,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:26:28.652397","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:26:25.044737","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-04-09 18:26:26.413903","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:26:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 -- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f385c018930 msgr2=0x7f385c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f385c018930 0x7f385c01adc0 crc :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 -- 172.21.15.32:0/1868154074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3878182760 msgr2=0x7f387817e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3878182760 0x7f387817e7d0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7f38680099e0 tx=0x7f386800bcd0).stop 2021-04-09T18:26:32.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 -- 172.21.15.32:0/1868154074 shutdown_connections 2021-04-09T18:26:32.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f385c018930 0x7f385c01adc0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3878182760 0x7f387817e7d0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f387818ad70 0x7f387818b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 --2- 172.21.15.32:0/1868154074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38781a9b10 0x7f387817e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 -- 172.21.15.32:0/1868154074 >> 172.21.15.32:0/1868154074 conn(0x7f38781aa1b0 msgr2=0x7f38781ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 -- 172.21.15.32:0/1868154074 shutdown_connections 2021-04-09T18:26:32.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.683 7f387efc9700 1 -- 172.21.15.32:0/1868154074 wait complete. 2021-04-09T18:26:32.766 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:26:25.044737', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:26:26.413903', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:26:32.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:26:32.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.847 7fdf0f289700 1 Processor -- start 2021-04-09T18:26:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.848 7fdf0f289700 1 -- start start 2021-04-09T18:26:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.848 7fdf0f289700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdf0810d010 0x7fdf080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.848 7fdf0f289700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080d8490 0x7fdf0810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.848 7fdf0f289700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0810d480 con 0x7fdf080d1da0 2021-04-09T18:26:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.848 7fdf0f289700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf080d8d30 con 0x7fdf0810d010 2021-04-09T18:26:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.848 7fdf0f289700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080d9220 con 0x7fdf080d8490 2021-04-09T18:26:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.848 7fdf0d0a5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdf0810d010 0x7fdf080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.848 7fdf0c8a4700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdf080d1da0 0x7fdf080d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34726/0 (socket says 172.21.15.32:34726) 2021-04-09T18:26:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.848 7fdf0c8a4700 1 -- 172.21.15.32:0/4206436160 learned_addr learned my addr 172.21.15.32:0/4206436160 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdf0d8a6700 1 --2- 172.21.15.32:0/4206436160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080d8490 0x7fdf0810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdf0d0a5700 1 -- 172.21.15.32:0/4206436160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080d8490 msgr2=0x7fdf0810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdf0d0a5700 1 --2- 172.21.15.32:0/4206436160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080d8490 0x7fdf0810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdf0d0a5700 1 -- 172.21.15.32:0/4206436160 >> v1:172.21.15.32:6789/0 conn(0x7fdf080d1da0 legacy=0x7fdf080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdefffff700 1 -- 172.21.15.32:0/4206436160 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895109084 0 0) 0x7fdef8000ce0 con 0x7fdf080d1da0 2021-04-09T18:26:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdf0d0a5700 1 -- 172.21.15.32:0/4206436160 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0810e020 con 0x7fdf0810d010 2021-04-09T18:26:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdf0d0a5700 1 --2- 172.21.15.32:0/4206436160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf0810d010 0x7fdf080d1780 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7fdf04006650 tx=0x7fdf04009a00).ready entity=mon.0 client_cookie=dfdee4f9459d52bf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdefffff700 1 -- 172.21.15.32:0/4206436160 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf04009df0 con 0x7fdf0810d010 2021-04-09T18:26:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdefffff700 1 -- 172.21.15.32:0/4206436160 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf040009b0 con 0x7fdf0810d010 2021-04-09T18:26:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdefffff700 1 -- 172.21.15.32:0/4206436160 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf0400c2b0 con 0x7fdf0810d010 2021-04-09T18:26:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdf0f289700 1 -- 172.21.15.32:0/4206436160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf0810d010 msgr2=0x7fdf080d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.849 7fdf0f289700 1 --2- 172.21.15.32:0/4206436160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf0810d010 0x7fdf080d1780 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7fdf04006650 tx=0x7fdf04009a00).stop 2021-04-09T18:26:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.850 7fdf0f289700 1 -- 172.21.15.32:0/4206436160 shutdown_connections 2021-04-09T18:26:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.850 7fdf0f289700 1 --2- 172.21.15.32:0/4206436160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080d8490 0x7fdf0810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.850 7fdf0f289700 1 --2- 172.21.15.32:0/4206436160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf0810d010 0x7fdf080d1780 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.850 7fdf0f289700 1 -- 172.21.15.32:0/4206436160 >> 172.21.15.32:0/4206436160 conn(0x7fdf080ce5f0 msgr2=0x7fdf080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.850 7fdf0f289700 1 -- 172.21.15.32:0/4206436160 shutdown_connections 2021-04-09T18:26:32.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.850 7fdf0f289700 1 -- 172.21.15.32:0/4206436160 wait complete. 2021-04-09T18:26:32.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.870 7fdf0f289700 1 Processor -- start 2021-04-09T18:26:32.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.870 7fdf0f289700 1 -- start start 2021-04-09T18:26:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0f289700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf081a0050 0x7fdf081a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0f289700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf080bdcd0 0x7fdf080bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0f289700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080bf080 0x7fdf080b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0f289700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080bf500 con 0x7fdf080bdcd0 2021-04-09T18:26:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0f289700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080bb7b0 con 0x7fdf081a0050 2021-04-09T18:26:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0f289700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080be0d0 con 0x7fdf080bf080 2021-04-09T18:26:32.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0c8a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf080bdcd0 0x7fdf080bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0c8a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf080bdcd0 0x7fdf080bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53372/0 (socket says 172.21.15.32:53372) 2021-04-09T18:26:32.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0c8a4700 1 -- 172.21.15.32:0/1985983782 learned_addr learned my addr 172.21.15.32:0/1985983782 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:32.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0d8a6700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080bf080 0x7fdf080b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0d0a5700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf081a0050 0x7fdf081a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.871 7fdf0c8a4700 1 -- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080bf080 msgr2=0x7fdf080b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdf0c8a4700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080bf080 0x7fdf080b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdf0c8a4700 1 -- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf081a0050 msgr2=0x7fdf081a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdf0c8a4700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf081a0050 0x7fdf081a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:32.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdf0c8a4700 1 -- 172.21.15.32:0/1985983782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf080b1310 con 0x7fdf080bdcd0 2021-04-09T18:26:32.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdf0c8a4700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf080bdcd0 0x7fdf080bea60 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 rx=0x7fdef8006f00 tx=0x7fdef800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:32.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdefdffb700 1 -- 172.21.15.32:0/1985983782 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdef8006020 con 0x7fdf080bdcd0 2021-04-09T18:26:32.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdefdffb700 1 -- 172.21.15.32:0/1985983782 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef80063a0 con 0x7fdf080bdcd0 2021-04-09T18:26:32.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdefdffb700 1 -- 172.21.15.32:0/1985983782 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdef8004620 con 0x7fdf080bdcd0 2021-04-09T18:26:32.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf080b1310 con 0x7fdf080bdcd0 2021-04-09T18:26:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.872 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf080b1310 con 0x7fdf080bdcd0 2021-04-09T18:26:32.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.873 7fdefdffb700 1 -- 172.21.15.32:0/1985983782 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdef800a000 con 0x7fdf080bdcd0 2021-04-09T18:26:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.873 7fdefdffb700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdee80188c0 0x7fdee801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:32.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.873 7fdefdffb700 1 -- 172.21.15.32:0/1985983782 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fdef80327d0 con 0x7fdf080bdcd0 2021-04-09T18:26:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.873 7fdf0d0a5700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdee80188c0 0x7fdee801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:32.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.873 7fdf0d0a5700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdee80188c0 0x7fdee801adc0 crc :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:32.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.892 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf08093d50 con 0x7fdf080bdcd0 2021-04-09T18:26:32.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:32.896 7fdefdffb700 1 -- 172.21.15.32:0/1985983782 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdef8025f20 con 0x7fdf080bdcd0 2021-04-09T18:26:33.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.074 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fdf0810d480 con 0x7fdf080bdcd0 2021-04-09T18:26:33.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.074 7fdefdffb700 1 -- 172.21.15.32:0/1985983782 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v142) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdef8025f20 con 0x7fdf080bdcd0 2021-04-09T18:26:33.147 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:26:33.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdee80188c0 msgr2=0x7fdee801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdee80188c0 0x7fdee801adc0 crc :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf080bdcd0 msgr2=0x7fdf080bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf080bdcd0 0x7fdf080bea60 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 rx=0x7fdef8006f00 tx=0x7fdef800b940).stop 2021-04-09T18:26:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 shutdown_connections 2021-04-09T18:26:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdee80188c0 0x7fdee801adc0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf080bdcd0 0x7fdf080bea60 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf080bf080 0x7fdf080b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 --2- 172.21.15.32:0/1985983782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf081a0050 0x7fdf081a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 >> 172.21.15.32:0/1985983782 conn(0x7fdf080975f0 msgr2=0x7fdf08096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 shutdown_connections 2021-04-09T18:26:33.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.077 7fdf0f289700 1 -- 172.21.15.32:0/1985983782 wait complete. 2021-04-09T18:26:33.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:26:33.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.239 7f0b8ffc7700 1 Processor -- start 2021-04-09T18:26:33.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.239 7f0b8ffc7700 1 -- start start 2021-04-09T18:26:33.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8ffc7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0b8810d010 0x7f0b880d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8ffc7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880d1da0 0x7f0b880d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8ffc7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b880d82c0 0x7f0b8810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8ffc7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b880d8d30 con 0x7f0b8810d010 2021-04-09T18:26:33.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8ffc7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b880d9220 con 0x7f0b880d82c0 2021-04-09T18:26:33.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8ffc7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8810d480 con 0x7f0b880d1da0 2021-04-09T18:26:33.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8dde3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0b8810d010 0x7f0b880d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8dde3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0b8810d010 0x7f0b880d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53380/0 (socket says 172.21.15.32:53380) 2021-04-09T18:26:33.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8e5e4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b880d82c0 0x7f0b8810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8dde3700 1 -- 172.21.15.32:0/1969102756 learned_addr learned my addr 172.21.15.32:0/1969102756 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:33.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.240 7f0b8d5e2700 1 --2- 172.21.15.32:0/1969102756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880d1da0 0x7f0b880d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8dde3700 1 -- 172.21.15.32:0/1969102756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880d1da0 msgr2=0x7f0b880d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:33.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8dde3700 1 --2- 172.21.15.32:0/1969102756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880d1da0 0x7f0b880d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8dde3700 1 -- 172.21.15.32:0/1969102756 >> v2:172.21.15.86:3300/0 conn(0x7f0b880d82c0 msgr2=0x7f0b8810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:33.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8dde3700 1 --2- 172.21.15.32:0/1969102756 >> v2:172.21.15.86:3300/0 conn(0x7f0b880d82c0 0x7f0b8810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8dde3700 1 -- 172.21.15.32:0/1969102756 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b880d86c0 con 0x7f0b8810d010 2021-04-09T18:26:33.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8e5e4700 1 --2- 172.21.15.32:0/1969102756 >> v2:172.21.15.86:3300/0 conn(0x7f0b880d82c0 0x7f0b8810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:33.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8d5e2700 1 --2- 172.21.15.32:0/1969102756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880d1da0 0x7f0b880d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:33.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8dde3700 1 --2- 172.21.15.32:0/1969102756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8810d010 0x7f0b880d1780 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 rx=0x7f0b84006700 tx=0x7f0b84009b80).ready entity=mon.0 client_cookie=a9dce050a0a4c432 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8cde1700 1 -- 172.21.15.32:0/1969102756 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8400c0c0 con 0x7f0b8810d010 2021-04-09T18:26:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8cde1700 1 -- 172.21.15.32:0/1969102756 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8400c440 con 0x7f0b8810d010 2021-04-09T18:26:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8cde1700 1 -- 172.21.15.32:0/1969102756 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8400c920 con 0x7f0b8810d010 2021-04-09T18:26:33.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8ffc7700 1 -- 172.21.15.32:0/1969102756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8810d010 msgr2=0x7f0b880d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.241 7f0b8ffc7700 1 --2- 172.21.15.32:0/1969102756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8810d010 0x7f0b880d1780 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 rx=0x7f0b84006700 tx=0x7f0b84009b80).stop 2021-04-09T18:26:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.242 7f0b8ffc7700 1 -- 172.21.15.32:0/1969102756 shutdown_connections 2021-04-09T18:26:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.242 7f0b8ffc7700 1 --2- 172.21.15.32:0/1969102756 >> v2:172.21.15.86:3300/0 conn(0x7f0b880d82c0 0x7f0b8810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.242 7f0b8ffc7700 1 --2- 172.21.15.32:0/1969102756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880d1da0 0x7f0b880d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.242 7f0b8ffc7700 1 --2- 172.21.15.32:0/1969102756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b8810d010 0x7f0b880d1780 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.242 7f0b8ffc7700 1 -- 172.21.15.32:0/1969102756 >> 172.21.15.32:0/1969102756 conn(0x7f0b880ce5f0 msgr2=0x7f0b880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:33.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.242 7f0b8ffc7700 1 -- 172.21.15.32:0/1969102756 shutdown_connections 2021-04-09T18:26:33.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.242 7f0b8ffc7700 1 -- 172.21.15.32:0/1969102756 wait complete. 2021-04-09T18:26:33.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.262 7f0b8ffc7700 1 Processor -- start 2021-04-09T18:26:33.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.262 7f0b8ffc7700 1 -- start start 2021-04-09T18:26:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8ffc7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b880b1680 0x7f0b880b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8ffc7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b8809e330 0x7f0b880b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8ffc7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880b2190 0x7f0b8809f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8ffc7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8809e730 con 0x7f0b880b1680 2021-04-09T18:26:33.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8ffc7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8809f6c0 con 0x7f0b8809e330 2021-04-09T18:26:33.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8ffc7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8809f8e0 con 0x7f0b880b2190 2021-04-09T18:26:33.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8dde3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b880b1680 0x7f0b880b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8dde3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b880b1680 0x7f0b880b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53388/0 (socket says 172.21.15.32:53388) 2021-04-09T18:26:33.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8dde3700 1 -- 172.21.15.32:0/1774797913 learned_addr learned my addr 172.21.15.32:0/1774797913 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:33.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8e5e4700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880b2190 0x7f0b8809f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8d5e2700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b8809e330 0x7f0b880b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8dde3700 1 -- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880b2190 msgr2=0x7f0b8809f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8dde3700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880b2190 0x7f0b8809f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8dde3700 1 -- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b8809e330 msgr2=0x7f0b880b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8dde3700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b8809e330 0x7f0b880b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8dde3700 1 -- 172.21.15.32:0/1774797913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8809fb00 con 0x7f0b880b1680 2021-04-09T18:26:33.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8dde3700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b880b1680 0x7f0b880b0e30 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7f0b8400b650 tx=0x7f0b8400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:33.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b7e7fc700 1 -- 172.21.15.32:0/1774797913 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8400bde0 con 0x7f0b880b1680 2021-04-09T18:26:33.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b7e7fc700 1 -- 172.21.15.32:0/1774797913 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b84000c90 con 0x7f0b880b1680 2021-04-09T18:26:33.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8809fb00 con 0x7f0b880b1680 2021-04-09T18:26:33.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.263 7f0b7e7fc700 1 -- 172.21.15.32:0/1774797913 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b84002220 con 0x7f0b880b1680 2021-04-09T18:26:33.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.264 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8809fb00 con 0x7f0b880b1680 2021-04-09T18:26:33.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.264 7f0b7e7fc700 1 -- 172.21.15.32:0/1774797913 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0b84009f90 con 0x7f0b880b1680 2021-04-09T18:26:33.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.264 7f0b7e7fc700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b70018930 0x7f0b7001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.264 7f0b7e7fc700 1 -- 172.21.15.32:0/1774797913 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0b840329d0 con 0x7f0b880b1680 2021-04-09T18:26:33.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.265 7f0b8d5e2700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b70018930 0x7f0b7001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.265 7f0b8d5e2700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b70018930 0x7f0b7001ae10 crc :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:33.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.284 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8809eb70 con 0x7f0b880b1680 2021-04-09T18:26:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.288 7f0b7e7fc700 1 -- 172.21.15.32:0/1774797913 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0b840091e0 con 0x7f0b880b1680 2021-04-09T18:26:33.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.468 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f0b8809eb70 con 0x7f0b880b1680 2021-04-09T18:26:33.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.468 7f0b7e7fc700 1 -- 172.21.15.32:0/1774797913 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v142) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0b8400bde0 con 0x7f0b880b1680 2021-04-09T18:26:33.541 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:26:33.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b70018930 msgr2=0x7f0b7001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b70018930 0x7f0b7001ae10 crc :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b880b1680 msgr2=0x7f0b880b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b880b1680 0x7f0b880b0e30 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7f0b8400b650 tx=0x7f0b8400b9f0).stop 2021-04-09T18:26:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 shutdown_connections 2021-04-09T18:26:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0b70018930 0x7f0b7001ae10 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b880b2190 0x7f0b8809f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b8809e330 0x7f0b880b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 --2- 172.21.15.32:0/1774797913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b880b1680 0x7f0b880b0e30 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 >> 172.21.15.32:0/1774797913 conn(0x7f0b880c4290 msgr2=0x7f0b880c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:33.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 shutdown_connections 2021-04-09T18:26:33.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.471 7f0b8ffc7700 1 -- 172.21.15.32:0/1774797913 wait complete. 2021-04-09T18:26:33.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:26:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.635 7f98c5f99700 1 Processor -- start 2021-04-09T18:26:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.635 7f98c5f99700 1 -- start start 2021-04-09T18:26:33.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.636 7f98c5f99700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c00cffc0 0x7f98c00d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.636 7f98c5f99700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f98c00d6ca0 0x7f98c010cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.636 7f98c5f99700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c00d72a0 con 0x7f98c00d7820 2021-04-09T18:26:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.636 7f98c5f99700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c00d70a0 con 0x7f98c00d6ca0 2021-04-09T18:26:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.636 7f98c5f99700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c00d85a0 con 0x7f98c00cffc0 2021-04-09T18:26:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.636 7f98bf7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c00cffc0 0x7f98c00d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98bffff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f98c00d6ca0 0x7f98c010cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98beffd700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f98c00d7820 0x7f98c00d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:60914/0 (socket says 172.21.15.32:60914) 2021-04-09T18:26:33.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98bf7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c00cffc0 0x7f98c00d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:60620/0 (socket says 172.21.15.32:60620) 2021-04-09T18:26:33.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98beffd700 1 -- 172.21.15.32:0/4098944660 learned_addr learned my addr 172.21.15.32:0/4098944660 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:33.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98bf7fe700 1 -- 172.21.15.32:0/4098944660 >> v2:172.21.15.86:3300/0 conn(0x7f98c00d6ca0 msgr2=0x7f98c010cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:33.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98bf7fe700 1 --2- 172.21.15.32:0/4098944660 >> v2:172.21.15.86:3300/0 conn(0x7f98c00d6ca0 0x7f98c010cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98bf7fe700 1 -- 172.21.15.32:0/4098944660 >> v1:172.21.15.86:6789/0 conn(0x7f98c00d7820 legacy=0x7f98c00d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98bf7fe700 1 -- 172.21.15.32:0/4098944660 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c00d9200 con 0x7f98c00cffc0 2021-04-09T18:26:33.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98bffff700 1 --2- 172.21.15.32:0/4098944660 >> v2:172.21.15.86:3300/0 conn(0x7f98c00d6ca0 0x7f98c010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:33.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98bf7fe700 1 --2- 172.21.15.32:0/4098944660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c00cffc0 0x7f98c00d2460 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f98b0006650 tx=0x7f98b0009a80).ready entity=mon.2 client_cookie=fdf0f199334b397d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:33.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98be7fc700 1 -- 172.21.15.32:0/4098944660 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98b00009b0 con 0x7f98c00cffc0 2021-04-09T18:26:33.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98be7fc700 1 -- 172.21.15.32:0/4098944660 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b000c0c0 con 0x7f98c00cffc0 2021-04-09T18:26:33.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98be7fc700 1 -- 172.21.15.32:0/4098944660 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98b000c510 con 0x7f98c00cffc0 2021-04-09T18:26:33.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98c5f99700 1 -- 172.21.15.32:0/4098944660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c00cffc0 msgr2=0x7f98c00d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.637 7f98c5f99700 1 --2- 172.21.15.32:0/4098944660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c00cffc0 0x7f98c00d2460 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f98b0006650 tx=0x7f98b0009a80).stop 2021-04-09T18:26:33.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.638 7f98c5f99700 1 -- 172.21.15.32:0/4098944660 shutdown_connections 2021-04-09T18:26:33.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.638 7f98c5f99700 1 --2- 172.21.15.32:0/4098944660 >> v2:172.21.15.86:3300/0 conn(0x7f98c00d6ca0 0x7f98c010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.638 7f98c5f99700 1 --2- 172.21.15.32:0/4098944660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c00cffc0 0x7f98c00d2460 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.638 7f98c5f99700 1 -- 172.21.15.32:0/4098944660 >> 172.21.15.32:0/4098944660 conn(0x7f98c00ce5e0 msgr2=0x7f98c00cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:33.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.638 7f98c5f99700 1 -- 172.21.15.32:0/4098944660 shutdown_connections 2021-04-09T18:26:33.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.638 7f98c5f99700 1 -- 172.21.15.32:0/4098944660 wait complete. 2021-04-09T18:26:33.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.658 7f98c5f99700 1 Processor -- start 2021-04-09T18:26:33.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.658 7f98c5f99700 1 -- start start 2021-04-09T18:26:33.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98c5f99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98c0180d70 0x7f98c00ba330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98c5f99700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98c00ba950 0x7f98c01795f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98c5f99700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c0179c10 0x7f98c01b8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98c5f99700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c00b1d40 con 0x7f98c0180d70 2021-04-09T18:26:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98c5f99700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c0181170 con 0x7f98c00ba950 2021-04-09T18:26:33.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98c5f99700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c00bad50 con 0x7f98c0179c10 2021-04-09T18:26:33.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98c0180d70 0x7f98c00ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98bf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98c0180d70 0x7f98c00ba330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53400/0 (socket says 172.21.15.32:53400) 2021-04-09T18:26:33.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98bf7fe700 1 -- 172.21.15.32:0/4177454840 learned_addr learned my addr 172.21.15.32:0/4177454840 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:33.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98beffd700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98c00ba950 0x7f98c01795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98bffff700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c0179c10 0x7f98c01b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98bf7fe700 1 -- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c0179c10 msgr2=0x7f98c01b8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98bf7fe700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c0179c10 0x7f98c01b8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98bf7fe700 1 -- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98c00ba950 msgr2=0x7f98c01795f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.659 7f98bf7fe700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98c00ba950 0x7f98c01795f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.660 7f98bf7fe700 1 -- 172.21.15.32:0/4177454840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c00b1880 con 0x7f98c0180d70 2021-04-09T18:26:33.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.660 7f98beffd700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98c00ba950 0x7f98c01795f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:33.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.660 7f98bf7fe700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98c0180d70 0x7f98c00ba330 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7f98b0009630 tx=0x7f98b000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:33.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.660 7f98b7fff700 1 -- 172.21.15.32:0/4177454840 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98b00093a0 con 0x7f98c0180d70 2021-04-09T18:26:33.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.660 7f98b7fff700 1 -- 172.21.15.32:0/4177454840 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b000bcf0 con 0x7f98c0180d70 2021-04-09T18:26:33.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.660 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c00b1880 con 0x7f98c0180d70 2021-04-09T18:26:33.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.660 7f98b7fff700 1 -- 172.21.15.32:0/4177454840 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98b00054d0 con 0x7f98c0180d70 2021-04-09T18:26:33.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.660 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c00b1880 con 0x7f98c0180d70 2021-04-09T18:26:33.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.660 7f98b7fff700 1 -- 172.21.15.32:0/4177454840 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f98b00052e0 con 0x7f98c0180d70 2021-04-09T18:26:33.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.661 7f98b7fff700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98a40188d0 0x7f98a401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:33.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.661 7f98b7fff700 1 -- 172.21.15.32:0/4177454840 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f98b0032a10 con 0x7f98c0180d70 2021-04-09T18:26:33.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.661 7f98beffd700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98a40188d0 0x7f98a401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:33.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.661 7f98beffd700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98a40188d0 0x7f98a401add0 crc :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:33.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.680 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c00b9e00 con 0x7f98c0180d70 2021-04-09T18:26:33.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.684 7f98b7fff700 1 -- 172.21.15.32:0/4177454840 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f98b0006030 con 0x7f98c0180d70 2021-04-09T18:26:33.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.865 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f98c00b9e00 con 0x7f98c0180d70 2021-04-09T18:26:33.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.865 7f98b7fff700 1 -- 172.21.15.32:0/4177454840 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v142) v1 ==== 100+0+10 (secure 0 0 0) 0x7f98b00052e0 con 0x7f98c0180d70 2021-04-09T18:26:33.937 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:26:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.867 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98a40188d0 msgr2=0x7f98a401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.867 7f98c5f99700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98a40188d0 0x7f98a401add0 crc :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.867 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98c0180d70 msgr2=0x7f98c00ba330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:33.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.867 7f98c5f99700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98c0180d70 0x7f98c00ba330 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7f98b0009630 tx=0x7f98b000b900).stop 2021-04-09T18:26:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.867 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 shutdown_connections 2021-04-09T18:26:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.867 7f98c5f99700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98a40188d0 0x7f98a401add0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.867 7f98c5f99700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98c0179c10 0x7f98c01b8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.867 7f98c5f99700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98c00ba950 0x7f98c01795f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.867 7f98c5f99700 1 --2- 172.21.15.32:0/4177454840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98c0180d70 0x7f98c00ba330 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.868 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 >> 172.21.15.32:0/4177454840 conn(0x7f98c009eb90 msgr2=0x7f98c00b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.868 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 shutdown_connections 2021-04-09T18:26:33.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:33.868 7f98c5f99700 1 -- 172.21.15.32:0/4177454840 wait complete. 2021-04-09T18:26:33.950 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:26:33.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:34.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.028 7f2f1e694700 1 Processor -- start 2021-04-09T18:26:34.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.028 7f2f1e694700 1 -- start start 2021-04-09T18:26:34.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1e694700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2f1810d010 0x7f2f180d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:34.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1e694700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2f180d8490 0x7f2f1810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:34.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1e694700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1810d480 con 0x7f2f180d1da0 2021-04-09T18:26:34.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1e694700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f180d8d30 con 0x7f2f180d8490 2021-04-09T18:26:34.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1e694700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f180d9220 con 0x7f2f1810d010 2021-04-09T18:26:34.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1ccb1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2f180d8490 0x7f2f1810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:34.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f0ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2f180d1da0 0x7f2f180d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34766/0 (socket says 172.21.15.32:34766) 2021-04-09T18:26:34.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f0ffff700 1 -- 172.21.15.32:0/3636547426 learned_addr learned my addr 172.21.15.32:0/3636547426 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:34.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f17fff700 1 --2- 172.21.15.32:0/3636547426 >> v2:172.21.15.86:3300/0 conn(0x7f2f1810d010 0x7f2f180d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:34.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f177fe700 1 -- 172.21.15.32:0/3636547426 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795999125 0 0) 0x7f2f00000ce0 con 0x7f2f180d1da0 2021-04-09T18:26:34.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f177fe700 1 -- 172.21.15.32:0/3636547426 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f04002fe0 con 0x7f2f180d1da0 2021-04-09T18:26:34.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1ccb1700 1 -- 172.21.15.32:0/3636547426 >> v2:172.21.15.86:3300/0 conn(0x7f2f1810d010 msgr2=0x7f2f180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:34.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1ccb1700 1 --2- 172.21.15.32:0/3636547426 >> v2:172.21.15.86:3300/0 conn(0x7f2f1810d010 0x7f2f180d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1ccb1700 1 -- 172.21.15.32:0/3636547426 >> v1:172.21.15.32:6789/0 conn(0x7f2f180d1da0 legacy=0x7f2f180d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:34.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1ccb1700 1 -- 172.21.15.32:0/3636547426 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1810e020 con 0x7f2f180d8490 2021-04-09T18:26:34.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f1ccb1700 1 --2- 172.21.15.32:0/3636547426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f180d8490 0x7f2f1810da00 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 rx=0x7f2f10006e30 tx=0x7f2f10009e60).ready entity=mon.0 client_cookie=a58212ed674b7677 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:34.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f177fe700 1 -- 172.21.15.32:0/3636547426 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f10006a50 con 0x7f2f180d8490 2021-04-09T18:26:34.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f177fe700 1 -- 172.21.15.32:0/3636547426 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1000b300 con 0x7f2f180d8490 2021-04-09T18:26:34.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.029 7f2f177fe700 1 -- 172.21.15.32:0/3636547426 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f1000b750 con 0x7f2f180d8490 2021-04-09T18:26:34.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.030 7f2f1e694700 1 -- 172.21.15.32:0/3636547426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f180d8490 msgr2=0x7f2f1810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:34.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.030 7f2f1e694700 1 --2- 172.21.15.32:0/3636547426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f180d8490 0x7f2f1810da00 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 rx=0x7f2f10006e30 tx=0x7f2f10009e60).stop 2021-04-09T18:26:34.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.030 7f2f1e694700 1 -- 172.21.15.32:0/3636547426 shutdown_connections 2021-04-09T18:26:34.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.030 7f2f1e694700 1 --2- 172.21.15.32:0/3636547426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f180d8490 0x7f2f1810da00 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.030 7f2f1e694700 1 --2- 172.21.15.32:0/3636547426 >> v2:172.21.15.86:3300/0 conn(0x7f2f1810d010 0x7f2f180d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.030 7f2f1e694700 1 -- 172.21.15.32:0/3636547426 >> 172.21.15.32:0/3636547426 conn(0x7f2f180ce5f0 msgr2=0x7f2f180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:34.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.030 7f2f1e694700 1 -- 172.21.15.32:0/3636547426 shutdown_connections 2021-04-09T18:26:34.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.030 7f2f1e694700 1 -- 172.21.15.32:0/3636547426 wait complete. 2021-04-09T18:26:34.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.050 7f2f1e694700 1 Processor -- start 2021-04-09T18:26:34.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.050 7f2f1e694700 1 -- start start 2021-04-09T18:26:34.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f1e694700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f181aca20 0x7f2f181ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:34.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f1e694700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f1818bd40 0x7f2f1818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:34.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f1e694700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f1808fba0 0x7f2f1808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:34.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f1e694700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1818c7e0 con 0x7f2f181aca20 2021-04-09T18:26:34.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f1e694700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1818ca30 con 0x7f2f1818bd40 2021-04-09T18:26:34.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f1e694700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f180905f0 con 0x7f2f1808fba0 2021-04-09T18:26:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f181aca20 0x7f2f181ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f181aca20 0x7f2f181ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53416/0 (socket says 172.21.15.32:53416) 2021-04-09T18:26:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f17fff700 1 -- 172.21.15.32:0/1979495124 learned_addr learned my addr 172.21.15.32:0/1979495124 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f0ffff700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f1818bd40 0x7f2f1818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:34.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f1ccb1700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f1808fba0 0x7f2f1808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f17fff700 1 -- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f1808fba0 msgr2=0x7f2f1808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f17fff700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f1808fba0 0x7f2f1808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f17fff700 1 -- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f1818bd40 msgr2=0x7f2f1818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:34.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f17fff700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f1818bd40 0x7f2f1818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.051 7f2f17fff700 1 -- 172.21.15.32:0/1979495124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f180b4300 con 0x7f2f181aca20 2021-04-09T18:26:34.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.052 7f2f17fff700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f181aca20 0x7f2f181ace20 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f2f08006f60 tx=0x7f2f0800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:34.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.052 7f2f157fa700 1 -- 172.21.15.32:0/1979495124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f0800bd60 con 0x7f2f181aca20 2021-04-09T18:26:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.052 7f2f157fa700 1 -- 172.21.15.32:0/1979495124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08000c90 con 0x7f2f181aca20 2021-04-09T18:26:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.052 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f180b4300 con 0x7f2f181aca20 2021-04-09T18:26:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.052 7f2f157fa700 1 -- 172.21.15.32:0/1979495124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f08002220 con 0x7f2f181aca20 2021-04-09T18:26:34.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.052 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f180b4300 con 0x7f2f181aca20 2021-04-09T18:26:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.052 7f2f157fa700 1 -- 172.21.15.32:0/1979495124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2f080052e0 con 0x7f2f181aca20 2021-04-09T18:26:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.053 7f2f157fa700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2efc0188c0 0x7f2efc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.053 7f2f157fa700 1 -- 172.21.15.32:0/1979495124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2f080329f0 con 0x7f2f181aca20 2021-04-09T18:26:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.053 7f2f0ffff700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2efc0188c0 0x7f2efc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:34.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.053 7f2f0ffff700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2efc0188c0 0x7f2efc01adc0 crc :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:34.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.072 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f18182d70 con 0x7f2f181aca20 2021-04-09T18:26:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.076 7f2f157fa700 1 -- 172.21.15.32:0/1979495124 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2f08000c90 con 0x7f2f181aca20 2021-04-09T18:26:34.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.268 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f1818c760 con 0x7f2f181aca20 2021-04-09T18:26:34.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.268 7f2f157fa700 1 -- 172.21.15.32:0/1979495124 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 276 v276) v1 ==== 95+0+2780 (secure 0 0 0) 0x7f2f08006b00 con 0x7f2f181aca20 2021-04-09T18:26:34.341 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 276 2021-04-09T18:26:34.341 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:34.341 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":276,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9762,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:29.656266","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:26:34.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2efc0188c0 msgr2=0x7f2efc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:34.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2efc0188c0 0x7f2efc01adc0 crc :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f181aca20 msgr2=0x7f2f181ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:34.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f181aca20 0x7f2f181ace20 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f2f08006f60 tx=0x7f2f0800b970).stop 2021-04-09T18:26:34.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 shutdown_connections 2021-04-09T18:26:34.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2efc0188c0 0x7f2efc01adc0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f1808fba0 0x7f2f1808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f1818bd40 0x7f2f1818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 --2- 172.21.15.32:0/1979495124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f181aca20 0x7f2f181ace20 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:34.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 >> 172.21.15.32:0/1979495124 conn(0x7f2f1819b460 msgr2=0x7f2f1819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:34.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 shutdown_connections 2021-04-09T18:26:34.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:34.271 7f2f1e694700 1 -- 172.21.15.32:0/1979495124 wait complete. 2021-04-09T18:26:34.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 276, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:26:27.651041', 'modified': '2021-04-09 18:26:29.656266', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9466': {'gid': 9466, 'name': 'd', 'rank': 0, 'incarnation': 273, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/630282836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 630282836}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 630282836}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:26:34.756 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:26:34.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:26:34.883 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:26:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:26:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:26:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 9466, 2021-04-09T18:26:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:26:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:26:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 7.1596678960000002, 2021-04-09T18:26:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 276, 2021-04-09T18:26:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 142, 2021-04-09T18:26:34.886 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 142, 2021-04-09T18:26:34.886 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12.610070627000001 2021-04-09T18:26:34.886 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:26:34.888 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 9466, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.1596678960000002, "mdsmap_epoch": 276, "osdmap_epoch": 142, "osdmap_epoch_barrier": 142, "uptime": 12.610070627000001 } 2021-04-09T18:26:34.889 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:26:34.889 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:26:34.890 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:26:34.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:34.891 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:34.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:26:35.034 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:26:35.035 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:26:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:26:35.094 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:26:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:26:35.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:35.096 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:26:35.140 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:26:35.141 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:35.183 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:26:35.245 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:26:35.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:26:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:26:35.306 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:26:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:26:35.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:35.308 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:26:35.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:26:35.259 7f81eb1e9f80 -1 init, newargv = 0x5638d06c5310 newargc=9ceph-fuse[ 2021-04-09T18:26:35.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:80314]: starting ceph client 2021-04-09T18:26:35.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[80314]: starting fuse 2021-04-09T18:26:35.353 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:26:35.353 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:26:35.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:35.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:26:35.355 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:26:35.355 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:26:35.356 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:26:35.356 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:26:35.356 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:26:35.357 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:26:35.357 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:26:35.358 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:26:35.358 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:26:35.358 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:26:35.359 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:26:35.359 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:26:35.360 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:26:35.360 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:26:35.360 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:26:35.361 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:26:35.361 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:26:35.362 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:26:35.362 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:26:35.362 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:26:35.363 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:26:35.363 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:26:35.475 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.80314.asok 2021-04-09T18:26:35.479 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.80314.asok 2021-04-09T18:26:35.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:35.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.80314.asok status 2021-04-09T18:26:35.604 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:26:35.604 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:26:35.604 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:26:35.605 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:26:35.605 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:26:35.605 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:26:35.605 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:26:35.606 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "80314", 2021-04-09T18:26:35.606 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:26:35.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:35.606 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:26:35.606 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:26:35.607 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9915, 2021-04-09T18:26:35.607 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:26:35.607 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:26:35.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:26:35.607 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 9915 2021-04-09T18:26:35.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:35.608 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:26:35.608 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:35.608 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:26:35.609 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 790894219 2021-04-09T18:26:35.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:26:35.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:35.609 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:26:35.609 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:26:35.610 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:26:35.610 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 790894219 2021-04-09T18:26:35.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:26:35.610 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.9915 172.21.15.32:0/790894219", 2021-04-09T18:26:35.610 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/790894219", 2021-04-09T18:26:35.611 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:26:35.611 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 276, 2021-04-09T18:26:35.611 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 142, 2021-04-09T18:26:35.611 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:26:35.612 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:26:35.612 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:26:35.612 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:26:35.613 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:35.614 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:26:35.614 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:35.658 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:26:35.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:26:35.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:35.660 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:35.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:26:35.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.800 7faee8d69700 1 Processor -- start 2021-04-09T18:26:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.801 7faee8d69700 1 -- start start 2021-04-09T18:26:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.801 7faee8d69700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faee40d7ca0 0x7faee40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:35.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.801 7faee8d69700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee410d480 con 0x7faee410d010 2021-04-09T18:26:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.801 7faee8d69700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee40d86c0 con 0x7faee410da00 2021-04-09T18:26:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.801 7faee8d69700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee40d9220 con 0x7faee40d7ca0 2021-04-09T18:26:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.801 7faee2312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faee40d7ca0 0x7faee40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:35.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee2b13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faee410d010 0x7faee40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34794/0 (socket says 172.21.15.32:34794) 2021-04-09T18:26:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee2b13700 1 -- 172.21.15.32:0/1883139748 learned_addr learned my addr 172.21.15.32:0/1883139748 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee2312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faee40d7ca0 0x7faee40d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53438/0 (socket says 172.21.15.32:53438) 2021-04-09T18:26:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee2312700 1 -- 172.21.15.32:0/1883139748 >> v1:172.21.15.86:6789/0 conn(0x7faee410da00 legacy=0x7faee410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:35.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee2312700 1 -- 172.21.15.32:0/1883139748 >> v1:172.21.15.32:6789/0 conn(0x7faee410d010 legacy=0x7faee40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee2312700 1 -- 172.21.15.32:0/1883139748 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee40d1f70 con 0x7faee40d7ca0 2021-04-09T18:26:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee1b11700 1 -- 172.21.15.32:0/1883139748 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940087356 0 0) 0x7faed4000ce0 con 0x7faee410d010 2021-04-09T18:26:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee2312700 1 --2- 172.21.15.32:0/1883139748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee40d7ca0 0x7faee40d80a0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 rx=0x7faed8006650 tx=0x7faed8009a80).ready entity=mon.0 client_cookie=573e8abc3913f014 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:35.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee1b11700 1 -- 172.21.15.32:0/1883139748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed80009b0 con 0x7faee40d7ca0 2021-04-09T18:26:35.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee1b11700 1 -- 172.21.15.32:0/1883139748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed800c0c0 con 0x7faee40d7ca0 2021-04-09T18:26:35.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee1b11700 1 -- 172.21.15.32:0/1883139748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed800c510 con 0x7faee40d7ca0 2021-04-09T18:26:35.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee8d69700 1 -- 172.21.15.32:0/1883139748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee40d7ca0 msgr2=0x7faee40d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:35.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.802 7faee8d69700 1 --2- 172.21.15.32:0/1883139748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee40d7ca0 0x7faee40d80a0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 rx=0x7faed8006650 tx=0x7faed8009a80).stop 2021-04-09T18:26:35.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.803 7faee8d69700 1 -- 172.21.15.32:0/1883139748 shutdown_connections 2021-04-09T18:26:35.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.803 7faee8d69700 1 --2- 172.21.15.32:0/1883139748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee40d7ca0 0x7faee40d80a0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:35.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.803 7faee8d69700 1 -- 172.21.15.32:0/1883139748 >> 172.21.15.32:0/1883139748 conn(0x7faee40ce5f0 msgr2=0x7faee40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:35.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.803 7faee8d69700 1 -- 172.21.15.32:0/1883139748 shutdown_connections 2021-04-09T18:26:35.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.803 7faee8d69700 1 -- 172.21.15.32:0/1883139748 wait complete. 2021-04-09T18:26:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.823 7faee8d69700 1 Processor -- start 2021-04-09T18:26:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.823 7faee8d69700 1 -- start start 2021-04-09T18:26:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee8d69700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee40afaf0 0x7faee417cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee8d69700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee417d4c0 0x7faee417b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee8d69700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee417baa0 0x7faee417bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee8d69700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee409c0f0 con 0x7faee417d4c0 2021-04-09T18:26:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee8d69700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee417c990 con 0x7faee417baa0 2021-04-09T18:26:35.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee8d69700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee40ad210 con 0x7faee40afaf0 2021-04-09T18:26:35.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee417d4c0 0x7faee417b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:35.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee417d4c0 0x7faee417b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53446/0 (socket says 172.21.15.32:53446) 2021-04-09T18:26:35.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2312700 1 -- 172.21.15.32:0/1629935183 learned_addr learned my addr 172.21.15.32:0/1629935183 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:35.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2b13700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee40afaf0 0x7faee417cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:35.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee3314700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee417baa0 0x7faee417bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:35.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2312700 1 -- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee40afaf0 msgr2=0x7faee417cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:35.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2312700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee40afaf0 0x7faee417cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:35.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2312700 1 -- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee417baa0 msgr2=0x7faee417bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:35.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2312700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee417baa0 0x7faee417bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:35.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2312700 1 -- 172.21.15.32:0/1629935183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee417d8c0 con 0x7faee417d4c0 2021-04-09T18:26:35.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee3314700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee417baa0 0x7faee417bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:35.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee2312700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee417d4c0 0x7faee417b480 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7faed8009650 tx=0x7faed800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:35.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faed37fe700 1 -- 172.21.15.32:0/1629935183 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed80093a0 con 0x7faee417d4c0 2021-04-09T18:26:35.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faed37fe700 1 -- 172.21.15.32:0/1629935183 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed8006070 con 0x7faee417d4c0 2021-04-09T18:26:35.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.824 7faee8d69700 1 -- 172.21.15.32:0/1629935183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee417d8c0 con 0x7faee417d4c0 2021-04-09T18:26:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.825 7faed37fe700 1 -- 172.21.15.32:0/1629935183 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed80052e0 con 0x7faee417d4c0 2021-04-09T18:26:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.825 7faee8d69700 1 -- 172.21.15.32:0/1629935183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee417d8c0 con 0x7faee417d4c0 2021-04-09T18:26:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.825 7faed37fe700 1 -- 172.21.15.32:0/1629935183 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faed8009300 con 0x7faee417d4c0 2021-04-09T18:26:35.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.825 7faed37fe700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faec8018930 0x7faec801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.825 7faed37fe700 1 -- 172.21.15.32:0/1629935183 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7faed8032b50 con 0x7faee417d4c0 2021-04-09T18:26:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.826 7faee2b13700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faec8018930 0x7faec801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:35.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.826 7faee2b13700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faec8018930 0x7faec801adc0 crc :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:35.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.845 7faee8d69700 1 -- 172.21.15.32:0/1629935183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee408daf0 con 0x7faee417d4c0 2021-04-09T18:26:35.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:35.849 7faed37fe700 1 -- 172.21.15.32:0/1629935183 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faed8006030 con 0x7faee417d4c0 2021-04-09T18:26:36.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.042 7faee8d69700 1 -- 172.21.15.32:0/1629935183 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7faee410d480 con 0x7faee417d4c0 2021-04-09T18:26:36.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.677 7faed37fe700 1 -- 172.21.15.32:0/1629935183 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v277) v1 ==== 105+0+0 (secure 0 0 0) 0x7faed8009300 con 0x7faee417d4c0 2021-04-09T18:26:36.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 -- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faec8018930 msgr2=0x7faec801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faec8018930 0x7faec801adc0 crc :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 -- 172.21.15.32:0/1629935183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee417d4c0 msgr2=0x7faee417b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee417d4c0 0x7faee417b480 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7faed8009650 tx=0x7faed800b8e0).stop 2021-04-09T18:26:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 -- 172.21.15.32:0/1629935183 shutdown_connections 2021-04-09T18:26:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faec8018930 0x7faec801adc0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faee417d4c0 0x7faee417b480 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee417baa0 0x7faee417bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 --2- 172.21.15.32:0/1629935183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee40afaf0 0x7faee417cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 -- 172.21.15.32:0/1629935183 >> 172.21.15.32:0/1629935183 conn(0x7faee40ba720 msgr2=0x7faee40b9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 -- 172.21.15.32:0/1629935183 shutdown_connections 2021-04-09T18:26:36.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.710 7faee8d69700 1 -- 172.21.15.32:0/1629935183 wait complete. 2021-04-09T18:26:36.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:36.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.869 7f2253cd5700 1 Processor -- start 2021-04-09T18:26:36.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.870 7f2253cd5700 1 -- start start 2021-04-09T18:26:36.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.870 7f2253cd5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f224c10d010 0x7f224c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:36.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.870 7f2253cd5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f224c0d1da0 0x7f224c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:36.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.870 7f2253cd5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c0d82c0 0x7f224c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:36.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.870 7f2253cd5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224c0d8d30 con 0x7f224c10d010 2021-04-09T18:26:36.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.870 7f2253cd5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224c0d9220 con 0x7f224c0d1da0 2021-04-09T18:26:36.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.870 7f2253cd5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c10d4b0 con 0x7f224c0d82c0 2021-04-09T18:26:36.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.870 7f2251af1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f224c10d010 0x7f224c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:36.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2251af1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f224c10d010 0x7f224c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53462/0 (socket says 172.21.15.32:53462) 2021-04-09T18:26:36.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2251af1700 1 -- 172.21.15.32:0/3439371653 learned_addr learned my addr 172.21.15.32:0/3439371653 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:36.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f22512f0700 1 --2- 172.21.15.32:0/3439371653 >> v2:172.21.15.86:3300/0 conn(0x7f224c0d1da0 0x7f224c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:36.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f22522f2700 1 --2- 172.21.15.32:0/3439371653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c0d82c0 0x7f224c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:36.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2251af1700 1 -- 172.21.15.32:0/3439371653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c0d82c0 msgr2=0x7f224c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:36.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2251af1700 1 --2- 172.21.15.32:0/3439371653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c0d82c0 0x7f224c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2251af1700 1 -- 172.21.15.32:0/3439371653 >> v2:172.21.15.86:3300/0 conn(0x7f224c0d1da0 msgr2=0x7f224c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:36.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2251af1700 1 --2- 172.21.15.32:0/3439371653 >> v2:172.21.15.86:3300/0 conn(0x7f224c0d1da0 0x7f224c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2251af1700 1 -- 172.21.15.32:0/3439371653 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c0d86c0 con 0x7f224c10d010 2021-04-09T18:26:36.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2251af1700 1 --2- 172.21.15.32:0/3439371653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c10d010 0x7f224c0d1780 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7f2248006a80 tx=0x7f2248009f00).ready entity=mon.0 client_cookie=afe72971a309d0ee server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:36.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2250aef700 1 -- 172.21.15.32:0/3439371653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f224800b5c0 con 0x7f224c10d010 2021-04-09T18:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2250aef700 1 -- 172.21.15.32:0/3439371653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224800b8c0 con 0x7f224c10d010 2021-04-09T18:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2250aef700 1 -- 172.21.15.32:0/3439371653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f224800bda0 con 0x7f224c10d010 2021-04-09T18:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2253cd5700 1 -- 172.21.15.32:0/3439371653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c10d010 msgr2=0x7f224c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.871 7f2253cd5700 1 --2- 172.21.15.32:0/3439371653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c10d010 0x7f224c0d1780 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7f2248006a80 tx=0x7f2248009f00).stop 2021-04-09T18:26:36.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.872 7f2253cd5700 1 -- 172.21.15.32:0/3439371653 shutdown_connections 2021-04-09T18:26:36.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.872 7f2253cd5700 1 --2- 172.21.15.32:0/3439371653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c0d82c0 0x7f224c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.872 7f2253cd5700 1 --2- 172.21.15.32:0/3439371653 >> v2:172.21.15.86:3300/0 conn(0x7f224c0d1da0 0x7f224c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.872 7f2253cd5700 1 --2- 172.21.15.32:0/3439371653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c10d010 0x7f224c0d1780 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.872 7f2253cd5700 1 -- 172.21.15.32:0/3439371653 >> 172.21.15.32:0/3439371653 conn(0x7f224c0ce5f0 msgr2=0x7f224c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:36.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.872 7f2253cd5700 1 -- 172.21.15.32:0/3439371653 shutdown_connections 2021-04-09T18:26:36.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.872 7f2253cd5700 1 -- 172.21.15.32:0/3439371653 wait complete. 2021-04-09T18:26:36.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.892 7f2253cd5700 1 Processor -- start 2021-04-09T18:26:36.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.892 7f2253cd5700 1 -- start start 2021-04-09T18:26:36.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f2253cd5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f224c1ac380 0x7f224c18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f2253cd5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c1ac780 0x7f224c18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f2253cd5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c183520 0x7f224c183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f2253cd5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c1ace10 con 0x7f224c183520 2021-04-09T18:26:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f2253cd5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c189cf0 con 0x7f224c1ac380 2021-04-09T18:26:36.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f2253cd5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c183f40 con 0x7f224c1ac780 2021-04-09T18:26:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22522f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c183520 0x7f224c183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22522f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c183520 0x7f224c183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53470/0 (socket says 172.21.15.32:53470) 2021-04-09T18:26:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22522f2700 1 -- 172.21.15.32:0/3606589678 learned_addr learned my addr 172.21.15.32:0/3606589678 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22512f0700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c1ac780 0x7f224c18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:36.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f2251af1700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f224c1ac380 0x7f224c18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:36.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22522f2700 1 -- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c1ac780 msgr2=0x7f224c18a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:36.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22522f2700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c1ac780 0x7f224c18a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22522f2700 1 -- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f224c1ac380 msgr2=0x7f224c18a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:36.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22522f2700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f224c1ac380 0x7f224c18a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:36.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22522f2700 1 -- 172.21.15.32:0/3606589678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c184160 con 0x7f224c183520 2021-04-09T18:26:36.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f22512f0700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c1ac780 0x7f224c18a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:36.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.893 7f2251af1700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f224c1ac380 0x7f224c18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.894 7f22522f2700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c183520 0x7f224c183920 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 rx=0x7f224400b5e0 tx=0x7f224400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.894 7f22427fc700 1 -- 172.21.15.32:0/3606589678 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2244006720 con 0x7f224c183520 2021-04-09T18:26:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.894 7f22427fc700 1 -- 172.21.15.32:0/3606589678 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224400bd70 con 0x7f224c183520 2021-04-09T18:26:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.894 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c184160 con 0x7f224c183520 2021-04-09T18:26:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.894 7f22427fc700 1 -- 172.21.15.32:0/3606589678 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2244002220 con 0x7f224c183520 2021-04-09T18:26:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.894 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c184160 con 0x7f224c183520 2021-04-09T18:26:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.895 7f22427fc700 1 -- 172.21.15.32:0/3606589678 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2244005330 con 0x7f224c183520 2021-04-09T18:26:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.895 7f22427fc700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2234018930 0x7f223401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.895 7f22427fc700 1 -- 172.21.15.32:0/3606589678 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2244032a20 con 0x7f224c183520 2021-04-09T18:26:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.895 7f2251af1700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2234018930 0x7f223401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.895 7f2251af1700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2234018930 0x7f223401adc0 crc :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:36.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.914 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c189cf0 con 0x7f224c183520 2021-04-09T18:26:36.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:36.918 7f22427fc700 1 -- 172.21.15.32:0/3606589678 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2244006370 con 0x7f224c183520 2021-04-09T18:26:37.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.109 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f224c189cf0 con 0x7f224c183520 2021-04-09T18:26:37.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.109 7f22427fc700 1 -- 172.21.15.32:0/3606589678 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2796 (secure 0 0 0) 0x7f224400bd70 con 0x7f224c183520 2021-04-09T18:26:37.182 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 278 2021-04-09T18:26:37.182 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:37.182 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":278,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":278,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:36.678988","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_9762":{"gid":9762,"name":"a","rank":1,"incarnation":278,"state":"up:creating","state_seq":2,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:26:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2234018930 msgr2=0x7f223401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2234018930 0x7f223401adc0 crc :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:37.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c183520 msgr2=0x7f224c183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:37.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c183520 0x7f224c183920 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 rx=0x7f224400b5e0 tx=0x7f224400b980).stop 2021-04-09T18:26:37.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 shutdown_connections 2021-04-09T18:26:37.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2234018930 0x7f223401adc0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:37.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f224c183520 0x7f224c183920 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:37.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f224c1ac780 0x7f224c18a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:37.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 --2- 172.21.15.32:0/3606589678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f224c1ac380 0x7f224c18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:37.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 >> 172.21.15.32:0/3606589678 conn(0x7f224c19b890 msgr2=0x7f224c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:37.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 shutdown_connections 2021-04-09T18:26:37.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:37.112 7f2253cd5700 1 -- 172.21.15.32:0/3606589678 wait complete. 2021-04-09T18:26:37.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:26:37.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:26:37.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 278, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:26:27.651041', 'modified': '2021-04-09 18:26:36.678988', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9466, 'mds_1': 9762}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9466': {'gid': 9466, 'name': 'd', 'rank': 0, 'incarnation': 273, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/630282836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 630282836}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 630282836}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_9762': {'gid': 9762, 'name': 'a', 'rank': 1, 'incarnation': 278, 'state': 'up:creating', 'state_seq': 2, 'addr': '172.21.15.32:6834/820321060', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 820321060}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 820321060}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:26:37.597 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9762:up:creating 2021-04-09T18:26:38.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:38.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.686 7fd1d6a9f700 1 Processor -- start 2021-04-09T18:26:38.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.686 7fd1d6a9f700 1 -- start start 2021-04-09T18:26:38.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.686 7fd1d6a9f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d010d010 0x7fd1d00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:38.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.686 7fd1d6a9f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1d00d1da0 0x7fd1d00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:38.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.686 7fd1d6a9f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1d00d82c0 0x7fd1d010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:38.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.686 7fd1d6a9f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d010e780 con 0x7fd1d00d1da0 2021-04-09T18:26:38.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.686 7fd1d6a9f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d00d8d30 con 0x7fd1d00d82c0 2021-04-09T18:26:38.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.686 7fd1d6a9f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d00d9220 con 0x7fd1d010d010 2021-04-09T18:26:38.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1d00d1da0 0x7fd1d00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:38.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd1d00d1da0 0x7fd1d00d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53478/0 (socket says 172.21.15.32:53478) 2021-04-09T18:26:38.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cffff700 1 -- 172.21.15.32:0/1100442199 learned_addr learned my addr 172.21.15.32:0/1100442199 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:38.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1d48bb700 1 --2- 172.21.15.32:0/1100442199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d010d010 0x7fd1d00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:38.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1d50bc700 1 --2- 172.21.15.32:0/1100442199 >> v2:172.21.15.86:3300/0 conn(0x7fd1d00d82c0 0x7fd1d010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:38.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cffff700 1 -- 172.21.15.32:0/1100442199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d010d010 msgr2=0x7fd1d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:38.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cffff700 1 --2- 172.21.15.32:0/1100442199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d010d010 0x7fd1d00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:38.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cffff700 1 -- 172.21.15.32:0/1100442199 >> v2:172.21.15.86:3300/0 conn(0x7fd1d00d82c0 msgr2=0x7fd1d010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:38.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cffff700 1 --2- 172.21.15.32:0/1100442199 >> v2:172.21.15.86:3300/0 conn(0x7fd1d00d82c0 0x7fd1d010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:38.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cffff700 1 -- 172.21.15.32:0/1100442199 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d010d480 con 0x7fd1d00d1da0 2021-04-09T18:26:38.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cffff700 1 --2- 172.21.15.32:0/1100442199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d00d1da0 0x7fd1d00d7ca0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7fd1c0006700 tx=0x7fd1c0009b00).ready entity=mon.0 client_cookie=d34efece813cbc6e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:38.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cf7fe700 1 -- 172.21.15.32:0/1100442199 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1c000c0c0 con 0x7fd1d00d1da0 2021-04-09T18:26:38.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cf7fe700 1 -- 172.21.15.32:0/1100442199 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c000c400 con 0x7fd1d00d1da0 2021-04-09T18:26:38.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1cf7fe700 1 -- 172.21.15.32:0/1100442199 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1c000c8e0 con 0x7fd1d00d1da0 2021-04-09T18:26:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1d6a9f700 1 -- 172.21.15.32:0/1100442199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d00d1da0 msgr2=0x7fd1d00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.687 7fd1d6a9f700 1 --2- 172.21.15.32:0/1100442199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d00d1da0 0x7fd1d00d7ca0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7fd1c0006700 tx=0x7fd1c0009b00).stop 2021-04-09T18:26:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.688 7fd1d6a9f700 1 -- 172.21.15.32:0/1100442199 shutdown_connections 2021-04-09T18:26:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.688 7fd1d6a9f700 1 --2- 172.21.15.32:0/1100442199 >> v2:172.21.15.86:3300/0 conn(0x7fd1d00d82c0 0x7fd1d010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.688 7fd1d6a9f700 1 --2- 172.21.15.32:0/1100442199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d00d1da0 0x7fd1d00d7ca0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.688 7fd1d6a9f700 1 --2- 172.21.15.32:0/1100442199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d010d010 0x7fd1d00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.688 7fd1d6a9f700 1 -- 172.21.15.32:0/1100442199 >> 172.21.15.32:0/1100442199 conn(0x7fd1d00ce5f0 msgr2=0x7fd1d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.688 7fd1d6a9f700 1 -- 172.21.15.32:0/1100442199 shutdown_connections 2021-04-09T18:26:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.688 7fd1d6a9f700 1 -- 172.21.15.32:0/1100442199 wait complete. 2021-04-09T18:26:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.708 7fd1d6a9f700 1 Processor -- start 2021-04-09T18:26:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.708 7fd1d6a9f700 1 -- start start 2021-04-09T18:26:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d6a9f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d00b5360 0x7fd1d01a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d6a9f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d018e140 0x7fd1d01aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d6a9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d01ad040 0x7fd1d008fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d6a9f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d01a5360 con 0x7fd1d01ad040 2021-04-09T18:26:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d6a9f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d00902c0 con 0x7fd1d018e140 2021-04-09T18:26:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d6a9f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d00904e0 con 0x7fd1d00b5360 2021-04-09T18:26:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d50bc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d01ad040 0x7fd1d008fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d50bc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d01ad040 0x7fd1d008fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53482/0 (socket says 172.21.15.32:53482) 2021-04-09T18:26:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d50bc700 1 -- 172.21.15.32:0/3766538930 learned_addr learned my addr 172.21.15.32:0/3766538930 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:38.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1cffff700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d018e140 0x7fd1d01aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:38.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d48bb700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d00b5360 0x7fd1d01a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:38.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.709 7fd1d50bc700 1 -- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d00b5360 msgr2=0x7fd1d01a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:38.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1d50bc700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d00b5360 0x7fd1d01a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1d50bc700 1 -- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d018e140 msgr2=0x7fd1d01aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1d50bc700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d018e140 0x7fd1d01aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1d50bc700 1 -- 172.21.15.32:0/3766538930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d0183c80 con 0x7fd1d01ad040 2021-04-09T18:26:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1d48bb700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d00b5360 0x7fd1d01a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:26:38.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1cffff700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d018e140 0x7fd1d01aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:38.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1d50bc700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d01ad040 0x7fd1d008fca0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 rx=0x7fd1c800a940 tx=0x7fd1c800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:38.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1cd7fa700 1 -- 172.21.15.32:0/3766538930 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1c8001400 con 0x7fd1d01ad040 2021-04-09T18:26:38.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1cd7fa700 1 -- 172.21.15.32:0/3766538930 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c800ae20 con 0x7fd1d01ad040 2021-04-09T18:26:38.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d0183c80 con 0x7fd1d01ad040 2021-04-09T18:26:38.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1cd7fa700 1 -- 172.21.15.32:0/3766538930 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1c8001930 con 0x7fd1d01ad040 2021-04-09T18:26:38.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.710 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d0183c80 con 0x7fd1d01ad040 2021-04-09T18:26:38.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.711 7fd1cd7fa700 1 -- 172.21.15.32:0/3766538930 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd1c80011f0 con 0x7fd1d01ad040 2021-04-09T18:26:38.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.711 7fd1cd7fa700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1b4018930 0x7fd1b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:38.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.711 7fd1cd7fa700 1 -- 172.21.15.32:0/3766538930 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd1c8032d70 con 0x7fd1d01ad040 2021-04-09T18:26:38.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.711 7fd1d48bb700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1b4018930 0x7fd1b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:38.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.711 7fd1d48bb700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1b4018930 0x7fd1b401ae10 crc :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:38.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.730 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d0182d70 con 0x7fd1d01ad040 2021-04-09T18:26:38.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.733 7fd1cd7fa700 1 -- 172.21.15.32:0/3766538930 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd1c80066f0 con 0x7fd1d01ad040 2021-04-09T18:26:39.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.931 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1d0182d70 con 0x7fd1d01ad040 2021-04-09T18:26:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.932 7fd1cd7fa700 1 -- 172.21.15.32:0/3766538930 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 279 v279) v1 ==== 95+0+2794 (secure 0 0 0) 0x7fd1c80011f0 con 0x7fd1d01ad040 2021-04-09T18:26:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 279 2021-04-09T18:26:39.006 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:39.006 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":279,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":279,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:37.680351","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_9762":{"gid":9762,"name":"a","rank":1,"incarnation":278,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:26:39.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1b4018930 msgr2=0x7fd1b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1b4018930 0x7fd1b401ae10 crc :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d01ad040 msgr2=0x7fd1d008fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d01ad040 0x7fd1d008fca0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 rx=0x7fd1c800a940 tx=0x7fd1c800ace0).stop 2021-04-09T18:26:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 shutdown_connections 2021-04-09T18:26:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1b4018930 0x7fd1b401ae10 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:39.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1d01ad040 0x7fd1d008fca0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:39.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d018e140 0x7fd1d01aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:39.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 --2- 172.21.15.32:0/3766538930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d00b5360 0x7fd1d01a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:39.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 >> 172.21.15.32:0/3766538930 conn(0x7fd1d019b460 msgr2=0x7fd1d019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:39.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 shutdown_connections 2021-04-09T18:26:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:38.934 7fd1d6a9f700 1 -- 172.21.15.32:0/3766538930 wait complete. 2021-04-09T18:26:39.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 279, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:26:27.651041', 'modified': '2021-04-09 18:26:37.680351', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9466, 'mds_1': 9762}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9466': {'gid': 9466, 'name': 'd', 'rank': 0, 'incarnation': 273, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/630282836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 630282836}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 630282836}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_9762': {'gid': 9762, 'name': 'a', 'rank': 1, 'incarnation': 278, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6834/820321060', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 820321060}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 820321060}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:26:39.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:26:39.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:26:39.554 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:26:39.554 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:26:39.555 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:26:39.555 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 9466, 2021-04-09T18:26:39.555 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:26:39.555 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:26:39.555 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 11.830786126, 2021-04-09T18:26:39.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 279, 2021-04-09T18:26:39.556 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 142, 2021-04-09T18:26:39.556 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 142, 2021-04-09T18:26:39.556 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 17.281190625000001 2021-04-09T18:26:39.557 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:26:39.560 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 9466, "want_state": "up:active", "state": "up:active", "rank_uptime": 11.830786126, "mdsmap_epoch": 279, "osdmap_epoch": 142, "osdmap_epoch_barrier": 142, "uptime": 17.281190625000001 } 2021-04-09T18:26:39.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:26:39.703 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:26:39.704 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:26:39.704 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:26:39.705 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9762, 2021-04-09T18:26:39.705 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:26:39.706 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:26:39.706 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 2.9516825029999998, 2021-04-09T18:26:39.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 279, 2021-04-09T18:26:39.707 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 142, 2021-04-09T18:26:39.707 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 142, 2021-04-09T18:26:39.708 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 17.417054060000002 2021-04-09T18:26:39.708 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:26:39.710 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 9762, "want_state": "up:active", "state": "up:active", "rank_uptime": 2.9516825029999998, "mdsmap_epoch": 279, "osdmap_epoch": 142, "osdmap_epoch_barrier": 142, "uptime": 17.417054060000002 } 2021-04-09T18:26:39.711 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:39.712 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:26:39.712 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:39.758 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:26:39.759 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:26:39.759 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:26:39.760 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:26:39.760 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:39.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[80314]: fuse finished with error 0 and tester_r 0 2021-04-09T18:26:39.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:26:39.827 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:26:39.828 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:39.871 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:26:39.872 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:26:39.872 INFO:teuthology.orchestra.run:waiting for 900 2021-04-09T18:26:39.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:26:39.889 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:26:39.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:26:40.098 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:26:40.099 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:26:40.099 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:26:40.104 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:26:40.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok config get mon_client_ping_timeout 2021-04-09T18:26:40.227 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:26:40.228 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_ping_timeout": "30.000000" 2021-04-09T18:26:40.228 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:26:40.234 INFO:tasks.cephfs.filesystem:_json_asok output: { "mon_client_ping_timeout": "30.000000" } 2021-04-09T18:26:40.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 12:0 true 2021-04-09T18:26:40.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.316 7f461faed700 1 Processor -- start 2021-04-09T18:26:40.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461faed700 1 -- start start 2021-04-09T18:26:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461faed700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f46180d6c80 0x7f46180d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461faed700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46180d2440 0x7f461810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461faed700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46180d7280 con 0x7f46180cffa0 2021-04-09T18:26:40.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461faed700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46180d7080 con 0x7f46180d2440 2021-04-09T18:26:40.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461faed700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46180d91e0 con 0x7f46180d6c80 2021-04-09T18:26:40.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461e10a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f46180d2440 0x7f461810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461d909700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f46180cffa0 0x7f46180d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34848/0 (socket says 172.21.15.32:34848) 2021-04-09T18:26:40.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461d909700 1 -- 172.21.15.32:0/186942539 learned_addr learned my addr 172.21.15.32:0/186942539 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.317 7f461d108700 1 --2- 172.21.15.32:0/186942539 >> v2:172.21.15.86:3300/0 conn(0x7f46180d6c80 0x7f46180d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461c907700 1 -- 172.21.15.32:0/186942539 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368128026 0 0) 0x7f4614000ce0 con 0x7f46180cffa0 2021-04-09T18:26:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461c907700 1 -- 172.21.15.32:0/186942539 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4604002fe0 con 0x7f46180cffa0 2021-04-09T18:26:40.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461e10a700 1 -- 172.21.15.32:0/186942539 >> v2:172.21.15.86:3300/0 conn(0x7f46180d6c80 msgr2=0x7f46180d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461e10a700 1 --2- 172.21.15.32:0/186942539 >> v2:172.21.15.86:3300/0 conn(0x7f46180d6c80 0x7f46180d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461e10a700 1 -- 172.21.15.32:0/186942539 >> v1:172.21.15.32:6789/0 conn(0x7f46180cffa0 legacy=0x7f46180d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461e10a700 1 -- 172.21.15.32:0/186942539 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46180d2840 con 0x7f46180d2440 2021-04-09T18:26:40.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461e10a700 1 --2- 172.21.15.32:0/186942539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46180d2440 0x7f461810cfd0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7f4610006e30 tx=0x7f4610009e60).ready entity=mon.0 client_cookie=57b4813cb850f332 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461c907700 1 -- 172.21.15.32:0/186942539 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4610006a50 con 0x7f46180d2440 2021-04-09T18:26:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461c907700 1 -- 172.21.15.32:0/186942539 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461000b300 con 0x7f46180d2440 2021-04-09T18:26:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461c907700 1 -- 172.21.15.32:0/186942539 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f461000b750 con 0x7f46180d2440 2021-04-09T18:26:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461faed700 1 -- 172.21.15.32:0/186942539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46180d2440 msgr2=0x7f461810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.318 7f461faed700 1 --2- 172.21.15.32:0/186942539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46180d2440 0x7f461810cfd0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7f4610006e30 tx=0x7f4610009e60).stop 2021-04-09T18:26:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.319 7f461faed700 1 -- 172.21.15.32:0/186942539 shutdown_connections 2021-04-09T18:26:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.319 7f461faed700 1 --2- 172.21.15.32:0/186942539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46180d2440 0x7f461810cfd0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.319 7f461faed700 1 --2- 172.21.15.32:0/186942539 >> v2:172.21.15.86:3300/0 conn(0x7f46180d6c80 0x7f46180d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.319 7f461faed700 1 -- 172.21.15.32:0/186942539 >> 172.21.15.32:0/186942539 conn(0x7f46180ce5c0 msgr2=0x7f46180cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.319 7f461faed700 1 -- 172.21.15.32:0/186942539 shutdown_connections 2021-04-09T18:26:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.319 7f461faed700 1 -- 172.21.15.32:0/186942539 wait complete. 2021-04-09T18:26:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.339 7f461faed700 1 Processor -- start 2021-04-09T18:26:40.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.339 7f461faed700 1 -- start start 2021-04-09T18:26:40.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461faed700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f461808d210 0x7f461808fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461faed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46180903f0 0x7f46181a5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461faed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46181a6070 0x7f46181a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461faed700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461808d690 con 0x7f46181a6070 2021-04-09T18:26:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461faed700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46180b0fd0 con 0x7f461808d210 2021-04-09T18:26:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461faed700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46181a6470 con 0x7f46180903f0 2021-04-09T18:26:40.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461e10a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46181a6070 0x7f46181a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461e10a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46181a6070 0x7f46181a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53500/0 (socket says 172.21.15.32:53500) 2021-04-09T18:26:40.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461e10a700 1 -- 172.21.15.32:0/846131147 learned_addr learned my addr 172.21.15.32:0/846131147 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:40.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461d909700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f461808d210 0x7f461808fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461d108700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46180903f0 0x7f46181a5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461e10a700 1 -- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46180903f0 msgr2=0x7f46181a5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461e10a700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46180903f0 0x7f46181a5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461e10a700 1 -- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f461808d210 msgr2=0x7f461808fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461e10a700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f461808d210 0x7f461808fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461e10a700 1 -- 172.21.15.32:0/846131147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461808d8b0 con 0x7f46181a6070 2021-04-09T18:26:40.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461e10a700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46181a6070 0x7f46181a46a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 rx=0x7f4610001070 tx=0x7f461000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:40.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f460a7fc700 1 -- 172.21.15.32:0/846131147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4610001270 con 0x7f46181a6070 2021-04-09T18:26:40.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f460a7fc700 1 -- 172.21.15.32:0/846131147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4610006df0 con 0x7f46181a6070 2021-04-09T18:26:40.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f461faed700 1 -- 172.21.15.32:0/846131147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461808d8b0 con 0x7f46181a6070 2021-04-09T18:26:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.340 7f460a7fc700 1 -- 172.21.15.32:0/846131147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4610001720 con 0x7f46181a6070 2021-04-09T18:26:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.341 7f461faed700 1 -- 172.21.15.32:0/846131147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461808d8b0 con 0x7f46181a6070 2021-04-09T18:26:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.341 7f460a7fc700 1 -- 172.21.15.32:0/846131147 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f46100014f0 con 0x7f46181a6070 2021-04-09T18:26:40.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.341 7f460a7fc700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4600018930 0x7f460001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.341 7f460a7fc700 1 -- 172.21.15.32:0/846131147 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4610032db0 con 0x7f46181a6070 2021-04-09T18:26:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.341 7f461d909700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4600018930 0x7f460001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.342 7f461d909700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4600018930 0x7f460001ae10 crc :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:40.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.361 7f461faed700 1 -- 172.21.15.32:0/846131147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4618182e70 con 0x7f46181a6070 2021-04-09T18:26:40.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.365 7f460a7fc700 1 -- 172.21.15.32:0/846131147 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4610006dd0 con 0x7f46181a6070 2021-04-09T18:26:40.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.557 7f461faed700 1 -- 172.21.15.32:0/846131147 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "12:0", "val": "true"} v 0) v1 -- 0x7f4618182e70 con 0x7f46181a6070 2021-04-09T18:26:40.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.690 7f460a7fc700 1 -- 172.21.15.32:0/846131147 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "12:0", "val": "true"}]=0 freezing mds.9466 v280) v1 ==== 113+0+0 (secure 0 0 0) 0x7f4610006dd0 con 0x7f46181a6070 2021-04-09T18:26:40.791 INFO:teuthology.orchestra.run.smithi032.stderr:freezing mds.9466 2021-04-09T18:26:40.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 -- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4600018930 msgr2=0x7f460001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4600018930 0x7f460001ae10 crc :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 -- 172.21.15.32:0/846131147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46181a6070 msgr2=0x7f46181a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46181a6070 0x7f46181a46a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 rx=0x7f4610001070 tx=0x7f461000bd40).stop 2021-04-09T18:26:40.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 -- 172.21.15.32:0/846131147 shutdown_connections 2021-04-09T18:26:40.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4600018930 0x7f460001ae10 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f46181a6070 0x7f46181a46a0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46180903f0 0x7f46181a5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 --2- 172.21.15.32:0/846131147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f461808d210 0x7f461808fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 -- 172.21.15.32:0/846131147 >> 172.21.15.32:0/846131147 conn(0x7f46181a8f40 msgr2=0x7f4618188050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:40.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 -- 172.21.15.32:0/846131147 shutdown_connections 2021-04-09T18:26:40.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.722 7f461faed700 1 -- 172.21.15.32:0/846131147 wait complete. 2021-04-09T18:26:40.804 INFO:tasks.ceph.mds.d:Sent signal 19 2021-04-09T18:26:40.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:40.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.882 7f4afc787700 1 Processor -- start 2021-04-09T18:26:40.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.882 7f4afc787700 1 -- start start 2021-04-09T18:26:40.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afc787700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4af40d7ca0 0x7f4af40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afc787700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4af410da00 0x7f4af410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afc787700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af410d480 con 0x7f4af410d010 2021-04-09T18:26:40.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afc787700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af40d9220 con 0x7f4af410da00 2021-04-09T18:26:40.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afc787700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af40d86c0 con 0x7f4af40d7ca0 2021-04-09T18:26:40.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afada4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4af410da00 0x7f4af410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afa5a3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4af410d010 0x7f4af40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34862/0 (socket says 172.21.15.32:34862) 2021-04-09T18:26:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afa5a3700 1 -- 172.21.15.32:0/589586906 learned_addr learned my addr 172.21.15.32:0/589586906 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4af9da2700 1 --2- 172.21.15.32:0/589586906 >> v2:172.21.15.86:3300/0 conn(0x7f4af40d7ca0 0x7f4af40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afada4700 1 -- 172.21.15.32:0/589586906 >> v2:172.21.15.86:3300/0 conn(0x7f4af40d7ca0 msgr2=0x7f4af40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afada4700 1 --2- 172.21.15.32:0/589586906 >> v2:172.21.15.86:3300/0 conn(0x7f4af40d7ca0 0x7f4af40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afada4700 1 -- 172.21.15.32:0/589586906 >> v1:172.21.15.32:6789/0 conn(0x7f4af410d010 legacy=0x7f4af40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4af95a1700 1 -- 172.21.15.32:0/589586906 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150694329 0 0) 0x7f4af0000ce0 con 0x7f4af410d010 2021-04-09T18:26:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afada4700 1 -- 172.21.15.32:0/589586906 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af40d1f70 con 0x7f4af410da00 2021-04-09T18:26:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4afada4700 1 --2- 172.21.15.32:0/589586906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af410da00 0x7f4af410de00 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 rx=0x7f4aec006e30 tx=0x7f4aec009e60).ready entity=mon.0 client_cookie=c1067ec19ec0feeb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:40.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4af95a1700 1 -- 172.21.15.32:0/589586906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aec006a50 con 0x7f4af410da00 2021-04-09T18:26:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4af95a1700 1 -- 172.21.15.32:0/589586906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aec00b300 con 0x7f4af410da00 2021-04-09T18:26:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.883 7f4af95a1700 1 -- 172.21.15.32:0/589586906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aec00b750 con 0x7f4af410da00 2021-04-09T18:26:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.884 7f4afc787700 1 -- 172.21.15.32:0/589586906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af410da00 msgr2=0x7f4af410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.884 7f4afc787700 1 --2- 172.21.15.32:0/589586906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af410da00 0x7f4af410de00 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 rx=0x7f4aec006e30 tx=0x7f4aec009e60).stop 2021-04-09T18:26:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.884 7f4afc787700 1 -- 172.21.15.32:0/589586906 shutdown_connections 2021-04-09T18:26:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.884 7f4afc787700 1 --2- 172.21.15.32:0/589586906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af410da00 0x7f4af410de00 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.884 7f4afc787700 1 --2- 172.21.15.32:0/589586906 >> v2:172.21.15.86:3300/0 conn(0x7f4af40d7ca0 0x7f4af40d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.884 7f4afc787700 1 -- 172.21.15.32:0/589586906 >> 172.21.15.32:0/589586906 conn(0x7f4af40ce5f0 msgr2=0x7f4af40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:40.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.884 7f4afc787700 1 -- 172.21.15.32:0/589586906 shutdown_connections 2021-04-09T18:26:40.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.884 7f4afc787700 1 -- 172.21.15.32:0/589586906 wait complete. 2021-04-09T18:26:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.904 7f4afc787700 1 Processor -- start 2021-04-09T18:26:40.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.904 7f4afc787700 1 -- start start 2021-04-09T18:26:40.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afc787700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af4184a30 0x7f4af41acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afc787700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4af41a6b90 0x7f4af41a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afc787700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4af41a5150 0x7f4af41a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afc787700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af41a75b0 con 0x7f4af4184a30 2021-04-09T18:26:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afc787700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af4184590 con 0x7f4af41a5150 2021-04-09T18:26:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afc787700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af4184740 con 0x7f4af41a6b90 2021-04-09T18:26:40.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afa5a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af4184a30 0x7f4af41acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afa5a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af4184a30 0x7f4af41acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53510/0 (socket says 172.21.15.32:53510) 2021-04-09T18:26:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afa5a3700 1 -- 172.21.15.32:0/2678413729 learned_addr learned my addr 172.21.15.32:0/2678413729 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4af9da2700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4af41a6b90 0x7f4af41a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afada4700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4af41a5150 0x7f4af41a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afa5a3700 1 -- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4af41a6b90 msgr2=0x7f4af41a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afa5a3700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4af41a6b90 0x7f4af41a6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afa5a3700 1 -- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4af41a5150 msgr2=0x7f4af41a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:40.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afa5a3700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4af41a5150 0x7f4af41a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.905 7f4afa5a3700 1 -- 172.21.15.32:0/2678413729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af41a5b70 con 0x7f4af4184a30 2021-04-09T18:26:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.906 7f4afa5a3700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af4184a30 0x7f4af41acb80 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 rx=0x7f4af000b530 tx=0x7f4af000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.906 7f4aeaffd700 1 -- 172.21.15.32:0/2678413729 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4af0006020 con 0x7f4af4184a30 2021-04-09T18:26:40.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.906 7f4aeaffd700 1 -- 172.21.15.32:0/2678413729 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af00063a0 con 0x7f4af4184a30 2021-04-09T18:26:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.906 7f4afc787700 1 -- 172.21.15.32:0/2678413729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af41a5b70 con 0x7f4af4184a30 2021-04-09T18:26:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.906 7f4aeaffd700 1 -- 172.21.15.32:0/2678413729 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4af0004590 con 0x7f4af4184a30 2021-04-09T18:26:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.906 7f4afc787700 1 -- 172.21.15.32:0/2678413729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af41a5b70 con 0x7f4af4184a30 2021-04-09T18:26:40.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.906 7f4aeaffd700 1 -- 172.21.15.32:0/2678413729 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4af000a000 con 0x7f4af4184a30 2021-04-09T18:26:40.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.907 7f4aeaffd700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4adc0188d0 0x7f4adc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:40.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.907 7f4aeaffd700 1 -- 172.21.15.32:0/2678413729 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4af0032a00 con 0x7f4af4184a30 2021-04-09T18:26:40.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.907 7f4af9da2700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4adc0188d0 0x7f4adc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:40.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.907 7f4af9da2700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4adc0188d0 0x7f4adc01add0 crc :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:40.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.927 7f4afc787700 1 -- 172.21.15.32:0/2678413729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af41b7080 con 0x7f4af4184a30 2021-04-09T18:26:41.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:40.930 7f4aeaffd700 1 -- 172.21.15.32:0/2678413729 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4af0006b90 con 0x7f4af4184a30 2021-04-09T18:26:41.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.120 7f4afc787700 1 -- 172.21.15.32:0/2678413729 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4af410d480 con 0x7f4af4184a30 2021-04-09T18:26:41.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.121 7f4aeaffd700 1 -- 172.21.15.32:0/2678413729 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 280 v280) v1 ==== 95+0+2794 (secure 0 0 0) 0x7f4af000a000 con 0x7f4af4184a30 2021-04-09T18:26:41.194 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 280 2021-04-09T18:26:41.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:41.194 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":280,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":280,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:40.686465","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":1},"gid_9762":{"gid":9762,"name":"a","rank":1,"incarnation":278,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:26:41.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 -- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4adc0188d0 msgr2=0x7f4adc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:41.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4adc0188d0 0x7f4adc01add0 crc :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:41.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 -- 172.21.15.32:0/2678413729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af4184a30 msgr2=0x7f4af41acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:41.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af4184a30 0x7f4af41acb80 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 rx=0x7f4af000b530 tx=0x7f4af000b8d0).stop 2021-04-09T18:26:41.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 -- 172.21.15.32:0/2678413729 shutdown_connections 2021-04-09T18:26:41.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4adc0188d0 0x7f4adc01add0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:41.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4af41a5150 0x7f4af41a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:41.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4af41a6b90 0x7f4af41a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:41.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 --2- 172.21.15.32:0/2678413729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4af4184a30 0x7f4af41acb80 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:41.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 -- 172.21.15.32:0/2678413729 >> 172.21.15.32:0/2678413729 conn(0x7f4af4191560 msgr2=0x7f4af41b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:41.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.123 7f4afc787700 1 -- 172.21.15.32:0/2678413729 shutdown_connections 2021-04-09T18:26:41.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:41.124 7f4afc787700 1 -- 172.21.15.32:0/2678413729 wait complete. 2021-04-09T18:26:41.206 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:26:46.207 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.287 7f12bf5b4700 1 Processor -- start 2021-04-09T18:26:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.287 7f12bf5b4700 1 -- start start 2021-04-09T18:26:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bf5b4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12b810da00 0x7f12b810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:46.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bf5b4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b810d480 con 0x7f12b80d7ca0 2021-04-09T18:26:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bf5b4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b80d1f70 con 0x7f12b810d010 2021-04-09T18:26:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bf5b4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b80d8d30 con 0x7f12b810da00 2021-04-09T18:26:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bdbd1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12b810da00 0x7f12b810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:46.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bcbcf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f12b80d7ca0 0x7f12b80d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34878/0 (socket says 172.21.15.32:34878) 2021-04-09T18:26:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bcbcf700 1 -- 172.21.15.32:0/292029403 learned_addr learned my addr 172.21.15.32:0/292029403 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12affff700 1 -- 172.21.15.32:0/292029403 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96126857 0 0) 0x7f12a8000ce0 con 0x7f12b80d7ca0 2021-04-09T18:26:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12affff700 1 -- 172.21.15.32:0/292029403 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a4002fe0 con 0x7f12b80d7ca0 2021-04-09T18:26:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bdbd1700 1 -- 172.21.15.32:0/292029403 >> v1:172.21.15.86:6789/0 conn(0x7f12b810d010 legacy=0x7f12b80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:46.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bdbd1700 1 -- 172.21.15.32:0/292029403 >> v1:172.21.15.32:6789/0 conn(0x7f12b80d7ca0 legacy=0x7f12b80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bdbd1700 1 -- 172.21.15.32:0/292029403 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b80d9220 con 0x7f12b810da00 2021-04-09T18:26:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.288 7f12bdbd1700 1 --2- 172.21.15.32:0/292029403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b810da00 0x7f12b810de00 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 rx=0x7f12b0006e30 tx=0x7f12b0009e60).ready entity=mon.0 client_cookie=771ce130b596ed97 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.289 7f12affff700 1 -- 172.21.15.32:0/292029403 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12b0006a50 con 0x7f12b810da00 2021-04-09T18:26:46.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.289 7f12affff700 1 -- 172.21.15.32:0/292029403 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b000b300 con 0x7f12b810da00 2021-04-09T18:26:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.289 7f12affff700 1 -- 172.21.15.32:0/292029403 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12b000b750 con 0x7f12b810da00 2021-04-09T18:26:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.289 7f12bf5b4700 1 -- 172.21.15.32:0/292029403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b810da00 msgr2=0x7f12b810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.289 7f12bf5b4700 1 --2- 172.21.15.32:0/292029403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b810da00 0x7f12b810de00 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 rx=0x7f12b0006e30 tx=0x7f12b0009e60).stop 2021-04-09T18:26:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.289 7f12bf5b4700 1 -- 172.21.15.32:0/292029403 shutdown_connections 2021-04-09T18:26:46.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.289 7f12bf5b4700 1 --2- 172.21.15.32:0/292029403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b810da00 0x7f12b810de00 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:46.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.289 7f12bf5b4700 1 -- 172.21.15.32:0/292029403 >> 172.21.15.32:0/292029403 conn(0x7f12b80ce5f0 msgr2=0x7f12b80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:46.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.290 7f12bf5b4700 1 -- 172.21.15.32:0/292029403 shutdown_connections 2021-04-09T18:26:46.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.290 7f12bf5b4700 1 -- 172.21.15.32:0/292029403 wait complete. 2021-04-09T18:26:46.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.310 7f12bf5b4700 1 Processor -- start 2021-04-09T18:26:46.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.310 7f12bf5b4700 1 -- start start 2021-04-09T18:26:46.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.310 7f12bf5b4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b8188d40 0x7f12b817e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:46.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.310 7f12bf5b4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12b8182410 0x7f12b81a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:46.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.310 7f12bf5b4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b8182810 0x7f12b81813a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:46.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.310 7f12bf5b4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b81a7000 con 0x7f12b8188d40 2021-04-09T18:26:46.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.310 7f12bf5b4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b8189140 con 0x7f12b8182810 2021-04-09T18:26:46.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.310 7f12bf5b4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b817e800 con 0x7f12b8182410 2021-04-09T18:26:46.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bd3d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b8188d40 0x7f12b817e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:46.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bd3d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b8188d40 0x7f12b817e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53524/0 (socket says 172.21.15.32:53524) 2021-04-09T18:26:46.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bd3d0700 1 -- 172.21.15.32:0/2982066409 learned_addr learned my addr 172.21.15.32:0/2982066409 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:46.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bdbd1700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b8182810 0x7f12b81813a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:46.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bcbcf700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12b8182410 0x7f12b81a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:46.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bd3d0700 1 -- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12b8182410 msgr2=0x7f12b81a69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:46.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bd3d0700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12b8182410 0x7f12b81a69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:46.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bd3d0700 1 -- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b8182810 msgr2=0x7f12b81813a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:46.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bd3d0700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b8182810 0x7f12b81813a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:46.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bd3d0700 1 -- 172.21.15.32:0/2982066409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b81819c0 con 0x7f12b8188d40 2021-04-09T18:26:46.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bdbd1700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b8182810 0x7f12b81813a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:46.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bd3d0700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b8188d40 0x7f12b817e010 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f12b400b5b0 tx=0x7f12b400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:46.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12adffb700 1 -- 172.21.15.32:0/2982066409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12b400bd40 con 0x7f12b8188d40 2021-04-09T18:26:46.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12adffb700 1 -- 172.21.15.32:0/2982066409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b4006240 con 0x7f12b8188d40 2021-04-09T18:26:46.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b81819c0 con 0x7f12b8188d40 2021-04-09T18:26:46.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12adffb700 1 -- 172.21.15.32:0/2982066409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12b4005690 con 0x7f12b8188d40 2021-04-09T18:26:46.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.311 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b81819c0 con 0x7f12b8188d40 2021-04-09T18:26:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.312 7f12adffb700 1 -- 172.21.15.32:0/2982066409 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f12b40054a0 con 0x7f12b8188d40 2021-04-09T18:26:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.312 7f12adffb700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1298018930 0x7f129801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.312 7f12adffb700 1 -- 172.21.15.32:0/2982066409 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f12b4032a30 con 0x7f12b8188d40 2021-04-09T18:26:46.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.312 7f12bcbcf700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1298018930 0x7f129801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:46.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.312 7f12bcbcf700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1298018930 0x7f129801ae10 crc :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:46.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.332 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b8181c60 con 0x7f12b8188d40 2021-04-09T18:26:46.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.335 7f12adffb700 1 -- 172.21.15.32:0/2982066409 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f12b4000c60 con 0x7f12b8188d40 2021-04-09T18:26:46.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.526 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12b8181c60 con 0x7f12b8188d40 2021-04-09T18:26:46.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.527 7f12adffb700 1 -- 172.21.15.32:0/2982066409 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 280 v280) v1 ==== 95+0+2794 (secure 0 0 0) 0x7f12b40054a0 con 0x7f12b8188d40 2021-04-09T18:26:46.600 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 280 2021-04-09T18:26:46.600 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:46.600 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":280,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":280,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:40.686465","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":1},"gid_9762":{"gid":9762,"name":"a","rank":1,"incarnation":278,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:26:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1298018930 msgr2=0x7f129801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:46.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1298018930 0x7f129801ae10 crc :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:46.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b8188d40 msgr2=0x7f12b817e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:46.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b8188d40 0x7f12b817e010 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f12b400b5b0 tx=0x7f12b400b950).stop 2021-04-09T18:26:46.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 shutdown_connections 2021-04-09T18:26:46.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1298018930 0x7f129801ae10 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:46.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12b8182810 0x7f12b81813a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:46.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12b8182410 0x7f12b81a69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:46.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 --2- 172.21.15.32:0/2982066409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12b8188d40 0x7f12b817e010 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:46.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 >> 172.21.15.32:0/2982066409 conn(0x7f12b81965a0 msgr2=0x7f12b819dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:46.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 shutdown_connections 2021-04-09T18:26:46.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:46.529 7f12bf5b4700 1 -- 172.21.15.32:0/2982066409 wait complete. 2021-04-09T18:26:46.612 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:26:51.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:51.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.696 7fb327a96700 1 Processor -- start 2021-04-09T18:26:51.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.696 7fb327a96700 1 -- start start 2021-04-09T18:26:51.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb327a96700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3200d7ca0 0x7fb3200d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:51.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb327a96700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb32010da00 0x7fb32010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:51.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb327a96700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32010d480 con 0x7fb32010d010 2021-04-09T18:26:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb327a96700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3200d9220 con 0x7fb32010da00 2021-04-09T18:26:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb327a96700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3200d86c0 con 0x7fb3200d7ca0 2021-04-09T18:26:51.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb3260b3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb32010da00 0x7fb32010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:51.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb3260b3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb32010da00 0x7fb32010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53530/0 (socket says 172.21.15.32:53530) 2021-04-09T18:26:51.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb3260b3700 1 -- 172.21.15.32:0/44458421 learned_addr learned my addr 172.21.15.32:0/44458421 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:51.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb3250b1700 1 --2- 172.21.15.32:0/44458421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3200d7ca0 0x7fb3200d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:51.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb3260b3700 1 -- 172.21.15.32:0/44458421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3200d7ca0 msgr2=0x7fb3200d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:51.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb3260b3700 1 --2- 172.21.15.32:0/44458421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3200d7ca0 0x7fb3200d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:51.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb3260b3700 1 -- 172.21.15.32:0/44458421 >> v1:172.21.15.86:6789/0 conn(0x7fb32010d010 legacy=0x7fb3200d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:51.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.697 7fb3260b3700 1 -- 172.21.15.32:0/44458421 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3200d1f70 con 0x7fb32010da00 2021-04-09T18:26:51.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb3260b3700 1 --2- 172.21.15.32:0/44458421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32010da00 0x7fb32010de00 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7fb318006f00 tx=0x7fb318009e90).ready entity=mon.0 client_cookie=37b60d2ac35d8be0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb3248b0700 1 -- 172.21.15.32:0/44458421 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb31800b300 con 0x7fb32010da00 2021-04-09T18:26:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb3248b0700 1 -- 172.21.15.32:0/44458421 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31800b640 con 0x7fb32010da00 2021-04-09T18:26:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb3248b0700 1 -- 172.21.15.32:0/44458421 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb31800bb20 con 0x7fb32010da00 2021-04-09T18:26:51.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb327a96700 1 -- 172.21.15.32:0/44458421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32010da00 msgr2=0x7fb32010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:51.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb327a96700 1 --2- 172.21.15.32:0/44458421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32010da00 0x7fb32010de00 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7fb318006f00 tx=0x7fb318009e90).stop 2021-04-09T18:26:51.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb327a96700 1 -- 172.21.15.32:0/44458421 shutdown_connections 2021-04-09T18:26:51.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb327a96700 1 --2- 172.21.15.32:0/44458421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb32010da00 0x7fb32010de00 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:51.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb327a96700 1 --2- 172.21.15.32:0/44458421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3200d7ca0 0x7fb3200d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.698 7fb327a96700 1 -- 172.21.15.32:0/44458421 >> 172.21.15.32:0/44458421 conn(0x7fb3200ce5f0 msgr2=0x7fb3200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.699 7fb327a96700 1 -- 172.21.15.32:0/44458421 shutdown_connections 2021-04-09T18:26:51.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.699 7fb327a96700 1 -- 172.21.15.32:0/44458421 wait complete. 2021-04-09T18:26:51.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb327a96700 1 Processor -- start 2021-04-09T18:26:51.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb327a96700 1 -- start start 2021-04-09T18:26:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb327a96700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3201a5210 0x7fb320185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb327a96700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb320185a70 0x7fb3200acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb327a96700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb320185e70 0x7fb3200ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:51.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb327a96700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32018f7b0 con 0x7fb320185a70 2021-04-09T18:26:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb327a96700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3200ad7f0 con 0x7fb3201a5210 2021-04-09T18:26:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb327a96700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3200ada10 con 0x7fb320185e70 2021-04-09T18:26:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb3250b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb320185a70 0x7fb3200acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:51.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb3250b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb320185a70 0x7fb3200acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53534/0 (socket says 172.21.15.32:53534) 2021-04-09T18:26:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.719 7fb3250b1700 1 -- 172.21.15.32:0/3797571669 learned_addr learned my addr 172.21.15.32:0/3797571669 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:51.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3260b3700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb320185e70 0x7fb3200ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:51.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3258b2700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3201a5210 0x7fb320185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:51.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3250b1700 1 -- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb320185e70 msgr2=0x7fb3200ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:51.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3250b1700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb320185e70 0x7fb3200ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3250b1700 1 -- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3201a5210 msgr2=0x7fb320185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3250b1700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3201a5210 0x7fb320185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3250b1700 1 -- 172.21.15.32:0/3797571669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb320179610 con 0x7fb320185a70 2021-04-09T18:26:51.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3258b2700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3201a5210 0x7fb320185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:26:51.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3250b1700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb320185a70 0x7fb3200acbb0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 rx=0x7fb310006f00 tx=0x7fb31000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:51.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3167fc700 1 -- 172.21.15.32:0/3797571669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb31000bd30 con 0x7fb320185a70 2021-04-09T18:26:51.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3167fc700 1 -- 172.21.15.32:0/3797571669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb310002030 con 0x7fb320185a70 2021-04-09T18:26:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb327a96700 1 -- 172.21.15.32:0/3797571669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb320179610 con 0x7fb320185a70 2021-04-09T18:26:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb3167fc700 1 -- 172.21.15.32:0/3797571669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb310002550 con 0x7fb320185a70 2021-04-09T18:26:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.720 7fb327a96700 1 -- 172.21.15.32:0/3797571669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb320179610 con 0x7fb320185a70 2021-04-09T18:26:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.721 7fb3167fc700 1 -- 172.21.15.32:0/3797571669 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb310002210 con 0x7fb320185a70 2021-04-09T18:26:51.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.721 7fb3167fc700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb308018930 0x7fb30801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.721 7fb3167fc700 1 -- 172.21.15.32:0/3797571669 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb310032a60 con 0x7fb320185a70 2021-04-09T18:26:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.721 7fb3258b2700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb308018930 0x7fb30801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:51.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.721 7fb3258b2700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb308018930 0x7fb30801ae10 crc :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:51.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.741 7fb327a96700 1 -- 172.21.15.32:0/3797571669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3201b7070 con 0x7fb320185a70 2021-04-09T18:26:51.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.744 7fb3167fc700 1 -- 172.21.15.32:0/3797571669 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb310000c90 con 0x7fb320185a70 2021-04-09T18:26:52.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.934 7fb327a96700 1 -- 172.21.15.32:0/3797571669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3201b7070 con 0x7fb320185a70 2021-04-09T18:26:52.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.935 7fb3167fc700 1 -- 172.21.15.32:0/3797571669 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 280 v280) v1 ==== 95+0+2794 (secure 0 0 0) 0x7fb310002010 con 0x7fb320185a70 2021-04-09T18:26:52.009 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 280 2021-04-09T18:26:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:52.011 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":280,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":280,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:40.686465","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":1},"gid_9762":{"gid":9762,"name":"a","rank":1,"incarnation":278,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:26:52.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 -- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb308018930 msgr2=0x7fb30801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:52.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb308018930 0x7fb30801ae10 crc :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:52.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 -- 172.21.15.32:0/3797571669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb320185a70 msgr2=0x7fb3200acbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:52.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb320185a70 0x7fb3200acbb0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 rx=0x7fb310006f00 tx=0x7fb31000b940).stop 2021-04-09T18:26:52.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 -- 172.21.15.32:0/3797571669 shutdown_connections 2021-04-09T18:26:52.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb308018930 0x7fb30801ae10 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb320185a70 0x7fb3200acbb0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb320185e70 0x7fb3200ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 --2- 172.21.15.32:0/3797571669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3201a5210 0x7fb320185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.937 7fb327a96700 1 -- 172.21.15.32:0/3797571669 >> 172.21.15.32:0/3797571669 conn(0x7fb3201a99c0 msgr2=0x7fb32017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:52.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.938 7fb327a96700 1 -- 172.21.15.32:0/3797571669 shutdown_connections 2021-04-09T18:26:52.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:51.938 7fb327a96700 1 -- 172.21.15.32:0/3797571669 wait complete. 2021-04-09T18:26:52.021 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:26:57.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:26:57.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.108 7fda43fc6700 1 Processor -- start 2021-04-09T18:26:57.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda43fc6700 1 -- start start 2021-04-09T18:26:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda43fc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c10d010 0x7fda3c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda43fc6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fda3c0d1da0 0x7fda3c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda43fc6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c10d480 con 0x7fda3c0d82c0 2021-04-09T18:26:57.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda43fc6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c10e780 con 0x7fda3c0d1da0 2021-04-09T18:26:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda43fc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0d8d30 con 0x7fda3c10d010 2021-04-09T18:26:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda41de2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c10d010 0x7fda3c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda415e1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fda3c0d1da0 0x7fda3c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:57.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda425e3700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fda3c0d82c0 0x7fda3c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:32828/0 (socket says 172.21.15.32:32828) 2021-04-09T18:26:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.109 7fda425e3700 1 -- 172.21.15.32:0/4117317211 learned_addr learned my addr 172.21.15.32:0/4117317211 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:26:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda415e1700 1 -- 172.21.15.32:0/4117317211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c10d010 msgr2=0x7fda3c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:26:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda415e1700 1 --2- 172.21.15.32:0/4117317211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c10d010 0x7fda3c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda415e1700 1 -- 172.21.15.32:0/4117317211 >> v1:172.21.15.86:6789/0 conn(0x7fda3c0d82c0 legacy=0x7fda3c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:57.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda40de0700 1 -- 172.21.15.32:0/4117317211 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829666375 0 0) 0x7fda34001440 con 0x7fda3c0d82c0 2021-04-09T18:26:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda415e1700 1 -- 172.21.15.32:0/4117317211 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0d9220 con 0x7fda3c0d1da0 2021-04-09T18:26:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda41de2700 1 --2- 172.21.15.32:0/4117317211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c10d010 0x7fda3c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:26:57.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda415e1700 1 --2- 172.21.15.32:0/4117317211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda3c0d1da0 0x7fda3c0d7ca0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7fda2c006650 tx=0x7fda2c009a80).ready entity=mon.1 client_cookie=f0c8deade407316b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda40de0700 1 -- 172.21.15.32:0/4117317211 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda2c0009b0 con 0x7fda3c0d1da0 2021-04-09T18:26:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda40de0700 1 -- 172.21.15.32:0/4117317211 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c00c0c0 con 0x7fda3c0d1da0 2021-04-09T18:26:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda40de0700 1 -- 172.21.15.32:0/4117317211 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda2c0009b0 con 0x7fda3c0d1da0 2021-04-09T18:26:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda43fc6700 1 -- 172.21.15.32:0/4117317211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda3c0d1da0 msgr2=0x7fda3c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:57.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.110 7fda43fc6700 1 --2- 172.21.15.32:0/4117317211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda3c0d1da0 0x7fda3c0d7ca0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7fda2c006650 tx=0x7fda2c009a80).stop 2021-04-09T18:26:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.111 7fda43fc6700 1 -- 172.21.15.32:0/4117317211 shutdown_connections 2021-04-09T18:26:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.111 7fda43fc6700 1 --2- 172.21.15.32:0/4117317211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda3c0d1da0 0x7fda3c0d7ca0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.111 7fda43fc6700 1 --2- 172.21.15.32:0/4117317211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c10d010 0x7fda3c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.111 7fda43fc6700 1 -- 172.21.15.32:0/4117317211 >> 172.21.15.32:0/4117317211 conn(0x7fda3c0ce5f0 msgr2=0x7fda3c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:57.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.111 7fda43fc6700 1 -- 172.21.15.32:0/4117317211 shutdown_connections 2021-04-09T18:26:57.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.111 7fda43fc6700 1 -- 172.21.15.32:0/4117317211 wait complete. 2021-04-09T18:26:57.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.131 7fda43fc6700 1 Processor -- start 2021-04-09T18:26:57.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.131 7fda43fc6700 1 -- start start 2021-04-09T18:26:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda43fc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda3c0b5360 0x7fda3c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda43fc6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda3c18e140 0x7fda3c1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda43fc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c1ad040 0x7fda3c08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:57.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda43fc6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a5360 con 0x7fda3c0b5360 2021-04-09T18:26:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda43fc6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0902c0 con 0x7fda3c18e140 2021-04-09T18:26:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda43fc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0904e0 con 0x7fda3c1ad040 2021-04-09T18:26:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda41de2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda3c0b5360 0x7fda3c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:57.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda41de2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda3c0b5360 0x7fda3c1a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53548/0 (socket says 172.21.15.32:53548) 2021-04-09T18:26:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda41de2700 1 -- 172.21.15.32:0/3810472056 learned_addr learned my addr 172.21.15.32:0/3810472056 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:26:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda415e1700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda3c18e140 0x7fda3c1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda425e3700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c1ad040 0x7fda3c08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda41de2700 1 -- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c1ad040 msgr2=0x7fda3c08fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:57.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda41de2700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c1ad040 0x7fda3c08fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda41de2700 1 -- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda3c18e140 msgr2=0x7fda3c1aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:57.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda41de2700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda3c18e140 0x7fda3c1aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda41de2700 1 -- 172.21.15.32:0/3810472056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c183c80 con 0x7fda3c0b5360 2021-04-09T18:26:57.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.132 7fda41de2700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda3c0b5360 0x7fda3c1a4d40 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 rx=0x7fda3800b580 tx=0x7fda3800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.133 7fda327fc700 1 -- 172.21.15.32:0/3810472056 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda3800bd10 con 0x7fda3c0b5360 2021-04-09T18:26:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.133 7fda327fc700 1 -- 172.21.15.32:0/3810472056 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda38000c90 con 0x7fda3c0b5360 2021-04-09T18:26:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.133 7fda327fc700 1 -- 172.21.15.32:0/3810472056 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda38002220 con 0x7fda3c0b5360 2021-04-09T18:26:57.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.133 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c183c80 con 0x7fda3c0b5360 2021-04-09T18:26:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.133 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c183c80 con 0x7fda3c0b5360 2021-04-09T18:26:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.133 7fda327fc700 1 -- 172.21.15.32:0/3810472056 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fda38006b00 con 0x7fda3c0b5360 2021-04-09T18:26:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.134 7fda327fc700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda240188c0 0x7fda2401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:26:57.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.134 7fda327fc700 1 -- 172.21.15.32:0/3810472056 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fda380327a0 con 0x7fda3c0b5360 2021-04-09T18:26:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.134 7fda415e1700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda240188c0 0x7fda2401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:26:57.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.134 7fda415e1700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda240188c0 0x7fda2401adc0 crc :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:26:57.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.153 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c182d70 con 0x7fda3c0b5360 2021-04-09T18:26:57.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.157 7fda327fc700 1 -- 172.21.15.32:0/3810472056 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fda3800bd10 con 0x7fda3c0b5360 2021-04-09T18:26:57.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.348 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda3c10d480 con 0x7fda3c0b5360 2021-04-09T18:26:57.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.348 7fda327fc700 1 -- 172.21.15.32:0/3810472056 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 280 v280) v1 ==== 95+0+2794 (secure 0 0 0) 0x7fda38006b00 con 0x7fda3c0b5360 2021-04-09T18:26:57.422 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 280 2021-04-09T18:26:57.423 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:26:57.423 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":280,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":280,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:40.686465","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":1},"gid_9762":{"gid":9762,"name":"a","rank":1,"incarnation":278,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:26:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda240188c0 msgr2=0x7fda2401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda240188c0 0x7fda2401adc0 crc :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda3c0b5360 msgr2=0x7fda3c1a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:26:57.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda3c0b5360 0x7fda3c1a4d40 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 rx=0x7fda3800b580 tx=0x7fda3800b920).stop 2021-04-09T18:26:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 shutdown_connections 2021-04-09T18:26:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda240188c0 0x7fda2401adc0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda3c1ad040 0x7fda3c08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda3c18e140 0x7fda3c1aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 --2- 172.21.15.32:0/3810472056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda3c0b5360 0x7fda3c1a4d40 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:26:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 >> 172.21.15.32:0/3810472056 conn(0x7fda3c19b460 msgr2=0x7fda3c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:26:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 shutdown_connections 2021-04-09T18:26:57.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:26:57.351 7fda43fc6700 1 -- 172.21.15.32:0/3810472056 wait complete. 2021-04-09T18:26:57.434 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:27:02.435 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:27:02.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.522 7f987215b700 1 Processor -- start 2021-04-09T18:27:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.522 7f987215b700 1 -- start start 2021-04-09T18:27:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.523 7f987215b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c0cffa0 0x7f986c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.523 7f987215b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0d7280 con 0x7f986c0d6c80 2021-04-09T18:27:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.523 7f987215b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c10e7d0 con 0x7f986c0d7800 2021-04-09T18:27:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.523 7f987215b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c0d7080 con 0x7f986c0cffa0 2021-04-09T18:27:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.523 7f986bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f986c0d6c80 0x7f986c10cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34916/0 (socket says 172.21.15.32:34916) 2021-04-09T18:27:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.523 7f986bfff700 1 -- 172.21.15.32:0/3192357622 learned_addr learned my addr 172.21.15.32:0/3192357622 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:27:02.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.523 7f986b7fe700 1 --2- 172.21.15.32:0/3192357622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c0cffa0 0x7f986c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:02.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.523 7f986a7fc700 1 -- 172.21.15.32:0/3192357622 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154861687 0 0) 0x7f9864000d30 con 0x7f986c0d6c80 2021-04-09T18:27:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986a7fc700 1 -- 172.21.15.32:0/3192357622 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9854002fe0 con 0x7f986c0d6c80 2021-04-09T18:27:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986a7fc700 1 -- 172.21.15.32:0/3192357622 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269988656 0 0) 0x7f9860000c60 con 0x7f986c0d7800 2021-04-09T18:27:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986a7fc700 1 -- 172.21.15.32:0/3192357622 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98540042c0 con 0x7f986c0d7800 2021-04-09T18:27:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986b7fe700 1 -- 172.21.15.32:0/3192357622 >> v1:172.21.15.86:6789/0 conn(0x7f986c0d7800 legacy=0x7f986c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986b7fe700 1 -- 172.21.15.32:0/3192357622 >> v1:172.21.15.32:6789/0 conn(0x7f986c0d6c80 legacy=0x7f986c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:02.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986b7fe700 1 -- 172.21.15.32:0/3192357622 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0d8580 con 0x7f986c0cffa0 2021-04-09T18:27:02.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986b7fe700 1 --2- 172.21.15.32:0/3192357622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c0cffa0 0x7f986c0d2440 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f985c006650 tx=0x7f985c009a80).ready entity=mon.2 client_cookie=8595ff5a5f0a7ae server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:02.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986a7fc700 1 -- 172.21.15.32:0/3192357622 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f985c006390 con 0x7f986c0cffa0 2021-04-09T18:27:02.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986a7fc700 1 -- 172.21.15.32:0/3192357622 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c0009b0 con 0x7f986c0cffa0 2021-04-09T18:27:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f986a7fc700 1 -- 172.21.15.32:0/3192357622 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f985c00c2b0 con 0x7f986c0cffa0 2021-04-09T18:27:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f987215b700 1 -- 172.21.15.32:0/3192357622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c0cffa0 msgr2=0x7f986c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:02.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.524 7f987215b700 1 --2- 172.21.15.32:0/3192357622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c0cffa0 0x7f986c0d2440 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f985c006650 tx=0x7f985c009a80).stop 2021-04-09T18:27:02.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.525 7f987215b700 1 -- 172.21.15.32:0/3192357622 shutdown_connections 2021-04-09T18:27:02.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.525 7f987215b700 1 --2- 172.21.15.32:0/3192357622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c0cffa0 0x7f986c0d2440 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:02.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.525 7f987215b700 1 -- 172.21.15.32:0/3192357622 >> 172.21.15.32:0/3192357622 conn(0x7f986c0ce5c0 msgr2=0x7f986c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:27:02.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.525 7f987215b700 1 -- 172.21.15.32:0/3192357622 shutdown_connections 2021-04-09T18:27:02.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.525 7f987215b700 1 -- 172.21.15.32:0/3192357622 wait complete. 2021-04-09T18:27:02.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.545 7f987215b700 1 Processor -- start 2021-04-09T18:27:02.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.545 7f987215b700 1 -- start start 2021-04-09T18:27:02.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f987215b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c18fe30 0x7f986c08d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:02.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f987215b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f986c190230 0x7f986c0aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f987215b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f986c0ab3d0 0x7f986c1b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f987215b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c08e3c0 con 0x7f986c190230 2021-04-09T18:27:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f987215b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c1b8bd0 con 0x7f986c0ab3d0 2021-04-09T18:27:02.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f987215b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c1b8df0 con 0x7f986c18fe30 2021-04-09T18:27:02.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f986c190230 0x7f986c0aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f986c190230 0x7f986c0aadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53562/0 (socket says 172.21.15.32:53562) 2021-04-09T18:27:02.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986affd700 1 -- 172.21.15.32:0/3195965744 learned_addr learned my addr 172.21.15.32:0/3195965744 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:27:02.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986bfff700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f986c0ab3d0 0x7f986c1b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:02.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986b7fe700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c18fe30 0x7f986c08d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:02.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986affd700 1 -- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c18fe30 msgr2=0x7f986c08d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:02.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986affd700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c18fe30 0x7f986c08d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:02.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986affd700 1 -- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f986c0ab3d0 msgr2=0x7f986c1b85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:02.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986affd700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f986c0ab3d0 0x7f986c1b85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:02.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986affd700 1 -- 172.21.15.32:0/3195965744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c098ac0 con 0x7f986c190230 2021-04-09T18:27:02.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986bfff700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f986c0ab3d0 0x7f986c1b85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:27:02.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.546 7f986affd700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f986c190230 0x7f986c0aadb0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 rx=0x7f9860006f30 tx=0x7f986000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:02.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.547 7f9853fff700 1 -- 172.21.15.32:0/3195965744 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f986000bd50 con 0x7f986c190230 2021-04-09T18:27:02.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.547 7f9853fff700 1 -- 172.21.15.32:0/3195965744 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9860007070 con 0x7f986c190230 2021-04-09T18:27:02.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.547 7f987215b700 1 -- 172.21.15.32:0/3195965744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986c098ac0 con 0x7f986c190230 2021-04-09T18:27:02.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.547 7f9853fff700 1 -- 172.21.15.32:0/3195965744 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9860007550 con 0x7f986c190230 2021-04-09T18:27:02.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.547 7f987215b700 1 -- 172.21.15.32:0/3195965744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c098ac0 con 0x7f986c190230 2021-04-09T18:27:02.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.547 7f9853fff700 1 -- 172.21.15.32:0/3195965744 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f986000a000 con 0x7f986c190230 2021-04-09T18:27:02.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.547 7f9853fff700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f984c0188d0 0x7f984c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:02.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.547 7f9853fff700 1 -- 172.21.15.32:0/3195965744 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9860032a20 con 0x7f986c190230 2021-04-09T18:27:02.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.548 7f986b7fe700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f984c0188d0 0x7f984c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:02.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.548 7f986b7fe700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f984c0188d0 0x7f984c01add0 crc :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:02.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.567 7f987215b700 1 -- 172.21.15.32:0/3195965744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c1b70c0 con 0x7f986c190230 2021-04-09T18:27:02.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.571 7f9853fff700 1 -- 172.21.15.32:0/3195965744 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9860006c20 con 0x7f986c190230 2021-04-09T18:27:02.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.762 7f987215b700 1 -- 172.21.15.32:0/3195965744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f986c1b70c0 con 0x7f986c190230 2021-04-09T18:27:02.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.763 7f9853fff700 1 -- 172.21.15.32:0/3195965744 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 281 v281) v1 ==== 95+0+2837 (secure 0 0 0) 0x7f986000a000 con 0x7f986c190230 2021-04-09T18:27:02.838 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 281 2021-04-09T18:27:02.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:27:02.839 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":281,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9765,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":281,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:26:57.582102","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9762},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"laggy_since":"2021-04-09 18:26:57.582075","export_targets":[],"features":4611087854035861503,"flags":1},"gid_9762":{"gid":9762,"name":"a","rank":1,"incarnation":278,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/820321060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":820321060},{"type":"v1","addr":"172.21.15.32:6834","nonce":820321060}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:27:02.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 -- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f984c0188d0 msgr2=0x7f984c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f984c0188d0 0x7f984c01add0 crc :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 -- 172.21.15.32:0/3195965744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f986c190230 msgr2=0x7f986c0aadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:02.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f986c190230 0x7f986c0aadb0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 rx=0x7f9860006f30 tx=0x7f986000b960).stop 2021-04-09T18:27:02.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 -- 172.21.15.32:0/3195965744 shutdown_connections 2021-04-09T18:27:02.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f984c0188d0 0x7f984c01add0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:02.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f986c190230 0x7f986c0aadb0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:02.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f986c0ab3d0 0x7f986c1b85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:02.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 --2- 172.21.15.32:0/3195965744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f986c18fe30 0x7f986c08d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 -- 172.21.15.32:0/3195965744 >> 172.21.15.32:0/3195965744 conn(0x7f986c1b72f0 msgr2=0x7f986c1b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:27:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 -- 172.21.15.32:0/3195965744 shutdown_connections 2021-04-09T18:27:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.765 7f987215b700 1 -- 172.21.15.32:0/3195965744 wait complete. 2021-04-09T18:27:02.850 DEBUG:tasks.ceph_test_case:wait_until_true: success in 20s 2021-04-09T18:27:02.850 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 12:1 2021-04-09T18:27:03.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.929 7f80a6c61700 1 Processor -- start 2021-04-09T18:27:03.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.929 7f80a6c61700 1 -- start start 2021-04-09T18:27:03.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a6c61700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a00d6c80 0x7f80a00d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a6c61700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80a00d2440 0x7f80a010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a6c61700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a00d7280 con 0x7f80a00cffa0 2021-04-09T18:27:03.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a6c61700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a00d7080 con 0x7f80a00d2440 2021-04-09T18:27:03.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a6c61700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00d91e0 con 0x7f80a00d6c80 2021-04-09T18:27:03.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a527e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f80a00d2440 0x7f80a010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a4a7d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f80a00cffa0 0x7f80a00d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34928/0 (socket says 172.21.15.32:34928) 2021-04-09T18:27:03.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a4a7d700 1 -- 172.21.15.32:0/418441558 learned_addr learned my addr 172.21.15.32:0/418441558 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:27:03.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f809ffff700 1 --2- 172.21.15.32:0/418441558 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a00d6c80 0x7f80a00d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a527e700 1 -- 172.21.15.32:0/418441558 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a00d6c80 msgr2=0x7f80a00d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:27:03.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a527e700 1 --2- 172.21.15.32:0/418441558 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a00d6c80 0x7f80a00d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a527e700 1 -- 172.21.15.32:0/418441558 >> v1:172.21.15.32:6789/0 conn(0x7f80a00cffa0 legacy=0x7f80a00d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f809f7fe700 1 -- 172.21.15.32:0/418441558 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348928893 0 0) 0x7f8088000ce0 con 0x7f80a00cffa0 2021-04-09T18:27:03.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a527e700 1 -- 172.21.15.32:0/418441558 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a00d2840 con 0x7f80a00d2440 2021-04-09T18:27:03.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f809ffff700 1 --2- 172.21.15.32:0/418441558 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a00d6c80 0x7f80a00d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:27:03.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f80a527e700 1 --2- 172.21.15.32:0/418441558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a00d2440 0x7f80a010cfd0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 rx=0x7f8098006e30 tx=0x7f8098009e60).ready entity=mon.0 client_cookie=1c13d142c1d024d2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f809f7fe700 1 -- 172.21.15.32:0/418441558 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8098006a50 con 0x7f80a00d2440 2021-04-09T18:27:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f809f7fe700 1 -- 172.21.15.32:0/418441558 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809800b300 con 0x7f80a00d2440 2021-04-09T18:27:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.930 7f809f7fe700 1 -- 172.21.15.32:0/418441558 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f809800b750 con 0x7f80a00d2440 2021-04-09T18:27:03.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.931 7f80a6c61700 1 -- 172.21.15.32:0/418441558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a00d2440 msgr2=0x7f80a010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.931 7f80a6c61700 1 --2- 172.21.15.32:0/418441558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a00d2440 0x7f80a010cfd0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 rx=0x7f8098006e30 tx=0x7f8098009e60).stop 2021-04-09T18:27:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.931 7f80a6c61700 1 -- 172.21.15.32:0/418441558 shutdown_connections 2021-04-09T18:27:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.931 7f80a6c61700 1 --2- 172.21.15.32:0/418441558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a00d2440 0x7f80a010cfd0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.931 7f80a6c61700 1 --2- 172.21.15.32:0/418441558 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a00d6c80 0x7f80a00d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.931 7f80a6c61700 1 -- 172.21.15.32:0/418441558 >> 172.21.15.32:0/418441558 conn(0x7f80a00ce5c0 msgr2=0x7f80a00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:27:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.931 7f80a6c61700 1 -- 172.21.15.32:0/418441558 shutdown_connections 2021-04-09T18:27:03.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.931 7f80a6c61700 1 -- 172.21.15.32:0/418441558 wait complete. 2021-04-09T18:27:03.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.951 7f80a6c61700 1 Processor -- start 2021-04-09T18:27:03.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.951 7f80a6c61700 1 -- start start 2021-04-09T18:27:03.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a6c61700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a01a7db0 0x7f80a01a81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a6c61700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a0184070 0x7f80a0184470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a6c61700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a0184a90 0x7f80a008f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a6c61700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a01a8b30 con 0x7f80a01a7db0 2021-04-09T18:27:03.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a6c61700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a01a7860 con 0x7f80a0184070 2021-04-09T18:27:03.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a6c61700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a01a87d0 con 0x7f80a0184a90 2021-04-09T18:27:03.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a4a7d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a01a7db0 0x7f80a01a81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a4a7d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a01a7db0 0x7f80a01a81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53576/0 (socket says 172.21.15.32:53576) 2021-04-09T18:27:03.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a4a7d700 1 -- 172.21.15.32:0/1066819121 learned_addr learned my addr 172.21.15.32:0/1066819121 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:27:03.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f80a527e700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a0184a90 0x7f80a008f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.952 7f809ffff700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a0184070 0x7f80a0184470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f80a4a7d700 1 -- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a0184a90 msgr2=0x7f80a008f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f80a4a7d700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a0184a90 0x7f80a008f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f80a4a7d700 1 -- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a0184070 msgr2=0x7f80a0184470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f80a4a7d700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a0184070 0x7f80a0184470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f80a4a7d700 1 -- 172.21.15.32:0/1066819121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a008f810 con 0x7f80a01a7db0 2021-04-09T18:27:03.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f809ffff700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a0184070 0x7f80a0184470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:27:03.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f80a527e700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a0184a90 0x7f80a008f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:27:03.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f80a4a7d700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a01a7db0 0x7f80a01a81b0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7f8088006f00 tx=0x7f808800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:03.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f809d7fa700 1 -- 172.21.15.32:0/1066819121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8088006020 con 0x7f80a01a7db0 2021-04-09T18:27:03.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f809d7fa700 1 -- 172.21.15.32:0/1066819121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80880063a0 con 0x7f80a01a7db0 2021-04-09T18:27:03.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a008f810 con 0x7f80a01a7db0 2021-04-09T18:27:03.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f809d7fa700 1 -- 172.21.15.32:0/1066819121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8088004620 con 0x7f80a01a7db0 2021-04-09T18:27:03.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.953 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a008f810 con 0x7f80a01a7db0 2021-04-09T18:27:03.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.954 7f809d7fa700 1 -- 172.21.15.32:0/1066819121 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f808800a000 con 0x7f80a01a7db0 2021-04-09T18:27:03.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.954 7f809d7fa700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8084018930 0x7f808401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.954 7f809d7fa700 1 -- 172.21.15.32:0/1066819121 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8088032a10 con 0x7f80a01a7db0 2021-04-09T18:27:03.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.954 7f809ffff700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8084018930 0x7f808401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.954 7f809ffff700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8084018930 0x7f808401adc0 crc :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:03.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.974 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a0182e70 con 0x7f80a01a7db0 2021-04-09T18:27:03.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:02.977 7f809d7fa700 1 -- 172.21.15.32:0/1066819121 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8088006b90 con 0x7f80a01a7db0 2021-04-09T18:27:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.169 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "12:1"} v 0) v1 -- 0x7f80a00d7280 con 0x7f80a01a7db0 2021-04-09T18:27:03.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.479 7f809d7fa700 1 -- 172.21.15.32:0/1066819121 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "12:1"}]=0 failed mds gid 9762 v282) v1 ==== 98+0+0 (secure 0 0 0) 0x7f808800a000 con 0x7f80a01a7db0 2021-04-09T18:27:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 9762 2021-04-09T18:27:03.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8084018930 msgr2=0x7f808401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8084018930 0x7f808401adc0 crc :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a01a7db0 msgr2=0x7f80a01a81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a01a7db0 0x7f80a01a81b0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7f8088006f00 tx=0x7f808800b940).stop 2021-04-09T18:27:03.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 shutdown_connections 2021-04-09T18:27:03.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8084018930 0x7f808401adc0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80a0184a90 0x7f80a008f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a0184070 0x7f80a0184470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 --2- 172.21.15.32:0/1066819121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80a01a7db0 0x7f80a01a81b0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 >> 172.21.15.32:0/1066819121 conn(0x7f80a01a8f40 msgr2=0x7f80a0188050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:27:03.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 shutdown_connections 2021-04-09T18:27:03.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.486 7f80a6c61700 1 -- 172.21.15.32:0/1066819121 wait complete. 2021-04-09T18:27:03.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:27:03.613 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:27:03.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.647 7fbd62570700 1 Processor -- start 2021-04-09T18:27:03.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.647 7fbd62570700 1 -- start start 2021-04-09T18:27:03.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd62570700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c10d010 0x7fbd5c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd62570700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd5c0d8490 0x7fbd5c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd62570700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5c10d480 con 0x7fbd5c0d1da0 2021-04-09T18:27:03.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd62570700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c10e780 con 0x7fbd5c0d8490 2021-04-09T18:27:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd62570700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0d8d30 con 0x7fbd5c10d010 2021-04-09T18:27:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd60b8d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd5c0d8490 0x7fbd5c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd53fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbd5c0d1da0 0x7fbd5c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34964/0 (socket says 172.21.15.32:34964) 2021-04-09T18:27:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd53fff700 1 -- 172.21.15.32:0/3921839213 learned_addr learned my addr 172.21.15.32:0/3921839213 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:27:03.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd5bfff700 1 --2- 172.21.15.32:0/3921839213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c10d010 0x7fbd5c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd60b8d700 1 -- 172.21.15.32:0/3921839213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c10d010 msgr2=0x7fbd5c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:27:03.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd60b8d700 1 --2- 172.21.15.32:0/3921839213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c10d010 0x7fbd5c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd60b8d700 1 -- 172.21.15.32:0/3921839213 >> v1:172.21.15.32:6789/0 conn(0x7fbd5c0d1da0 legacy=0x7fbd5c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd5b7fe700 1 -- 172.21.15.32:0/3921839213 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106091092 0 0) 0x7fbd44000ce0 con 0x7fbd5c0d1da0 2021-04-09T18:27:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.648 7fbd60b8d700 1 -- 172.21.15.32:0/3921839213 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c0d9220 con 0x7fbd5c0d8490 2021-04-09T18:27:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.649 7fbd60b8d700 1 --2- 172.21.15.32:0/3921839213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c0d8490 0x7fbd5c10da00 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7fbd54006e30 tx=0x7fbd54009de0).ready entity=mon.0 client_cookie=c0c0e3826c309fc8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.649 7fbd5b7fe700 1 -- 172.21.15.32:0/3921839213 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd5400b300 con 0x7fbd5c0d8490 2021-04-09T18:27:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.649 7fbd5b7fe700 1 -- 172.21.15.32:0/3921839213 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5400b680 con 0x7fbd5c0d8490 2021-04-09T18:27:03.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.649 7fbd5b7fe700 1 -- 172.21.15.32:0/3921839213 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd5400bb60 con 0x7fbd5c0d8490 2021-04-09T18:27:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.649 7fbd62570700 1 -- 172.21.15.32:0/3921839213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c0d8490 msgr2=0x7fbd5c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.649 7fbd62570700 1 --2- 172.21.15.32:0/3921839213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c0d8490 0x7fbd5c10da00 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7fbd54006e30 tx=0x7fbd54009de0).stop 2021-04-09T18:27:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.650 7fbd62570700 1 -- 172.21.15.32:0/3921839213 shutdown_connections 2021-04-09T18:27:03.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.650 7fbd62570700 1 --2- 172.21.15.32:0/3921839213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c0d8490 0x7fbd5c10da00 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.650 7fbd62570700 1 --2- 172.21.15.32:0/3921839213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c10d010 0x7fbd5c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.650 7fbd62570700 1 -- 172.21.15.32:0/3921839213 >> 172.21.15.32:0/3921839213 conn(0x7fbd5c0ce5f0 msgr2=0x7fbd5c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:27:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.650 7fbd62570700 1 -- 172.21.15.32:0/3921839213 shutdown_connections 2021-04-09T18:27:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.650 7fbd62570700 1 -- 172.21.15.32:0/3921839213 wait complete. 2021-04-09T18:27:03.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd62570700 1 Processor -- start 2021-04-09T18:27:03.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd62570700 1 -- start start 2021-04-09T18:27:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd62570700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c1a5210 0x7fbd5c08eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd62570700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c1a8ab0 0x7fbd5c1a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd62570700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd5c1a76d0 0x7fbd5c1a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd62570700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c17ea50 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd62570700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c08f4f0 con 0x7fbd5c1a76d0 2021-04-09T18:27:03.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd62570700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c08ea60 con 0x7fbd5c1a5210 2021-04-09T18:27:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd53fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c1a8ab0 0x7fbd5c1a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd53fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c1a8ab0 0x7fbd5c1a8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53610/0 (socket says 172.21.15.32:53610) 2021-04-09T18:27:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.670 7fbd53fff700 1 -- 172.21.15.32:0/1485025852 learned_addr learned my addr 172.21.15.32:0/1485025852 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:27:03.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd60b8d700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd5c1a76d0 0x7fbd5c1a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd5bfff700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c1a5210 0x7fbd5c08eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd53fff700 1 -- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c1a5210 msgr2=0x7fbd5c08eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd53fff700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c1a5210 0x7fbd5c08eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd53fff700 1 -- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd5c1a76d0 msgr2=0x7fbd5c1a7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd53fff700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd5c1a76d0 0x7fbd5c1a7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd53fff700 1 -- 172.21.15.32:0/1485025852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c1a80f0 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd60b8d700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd5c1a76d0 0x7fbd5c1a7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:27:03.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd53fff700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c1a8ab0 0x7fbd5c1a8eb0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 rx=0x7fbd4400b530 tx=0x7fbd4400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:03.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd597fa700 1 -- 172.21.15.32:0/1485025852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd44006020 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd597fa700 1 -- 172.21.15.32:0/1485025852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd440063a0 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd62570700 1 -- 172.21.15.32:0/1485025852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c1a80f0 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd597fa700 1 -- 172.21.15.32:0/1485025852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd44004590 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.671 7fbd62570700 1 -- 172.21.15.32:0/1485025852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c1a80f0 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.672 7fbd597fa700 1 -- 172.21.15.32:0/1485025852 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbd4400a000 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.672 7fbd597fa700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd400188c0 0x7fbd4001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.672 7fbd597fa700 1 -- 172.21.15.32:0/1485025852 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fbd440327c0 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.672 7fbd5bfff700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd400188c0 0x7fbd4001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.673 7fbd5bfff700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd400188c0 0x7fbd4001adc0 crc :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:03.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.692 7fbd62570700 1 -- 172.21.15.32:0/1485025852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c1b7070 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.695 7fbd597fa700 1 -- 172.21.15.32:0/1485025852 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbd44006b90 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.887 7fbd62570700 1 -- 172.21.15.32:0/1485025852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd5c10d480 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.887 7fbd597fa700 1 -- 172.21.15.32:0/1485025852 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+2504 (secure 0 0 0) 0x7fbd4400a000 con 0x7fbd5c1a8ab0 2021-04-09T18:27:03.962 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 283 2021-04-09T18:27:03.962 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:27:03.962 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":283,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:27:03.480239","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"laggy_since":"2021-04-09 18:26:57.582075","export_targets":[],"features":4611087854035861503,"flags":1},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:replay","state_seq":2,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:27:03.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.889 7fbd62570700 1 -- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd400188c0 msgr2=0x7fbd4001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.889 7fbd62570700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd400188c0 0x7fbd4001adc0 crc :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.889 7fbd62570700 1 -- 172.21.15.32:0/1485025852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c1a8ab0 msgr2=0x7fbd5c1a8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.890 7fbd62570700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c1a8ab0 0x7fbd5c1a8eb0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 rx=0x7fbd4400b530 tx=0x7fbd4400b8d0).stop 2021-04-09T18:27:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.890 7fbd62570700 1 -- 172.21.15.32:0/1485025852 shutdown_connections 2021-04-09T18:27:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.890 7fbd62570700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd400188c0 0x7fbd4001adc0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.890 7fbd62570700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd5c1a8ab0 0x7fbd5c1a8eb0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.890 7fbd62570700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd5c1a76d0 0x7fbd5c1a7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.890 7fbd62570700 1 --2- 172.21.15.32:0/1485025852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd5c1a5210 0x7fbd5c08eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:03.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.890 7fbd62570700 1 -- 172.21.15.32:0/1485025852 >> 172.21.15.32:0/1485025852 conn(0x7fbd5c1a99c0 msgr2=0x7fbd5c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:27:03.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.890 7fbd62570700 1 -- 172.21.15.32:0/1485025852 shutdown_connections 2021-04-09T18:27:03.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:03.890 7fbd62570700 1 -- 172.21.15.32:0/1485025852 wait complete. 2021-04-09T18:27:03.974 DEBUG:tasks.cephfs.filesystem:Looked up MDS state for mds.1: up:replay 2021-04-09T18:27:04.975 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:27:05.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.051 7f610858c700 1 Processor -- start 2021-04-09T18:27:05.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.051 7f610858c700 1 -- start start 2021-04-09T18:27:05.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f610858c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f610010e7d0 0x7f61000d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:05.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f610858c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61000d7280 con 0x7f61000d6c80 2021-04-09T18:27:05.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f610858c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61000d2440 con 0x7f61000cffa0 2021-04-09T18:27:05.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f610858c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61000d7080 con 0x7f610010e7d0 2021-04-09T18:27:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f6105ba7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f61000d6c80 0x7f610010cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34980/0 (socket says 172.21.15.32:34980) 2021-04-09T18:27:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f6105ba7700 1 -- 172.21.15.32:0/2537190266 learned_addr learned my addr 172.21.15.32:0/2537190266 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:27:05.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f6106ba9700 1 --2- 172.21.15.32:0/2537190266 >> v2:172.21.15.86:3300/0 conn(0x7f610010e7d0 0x7f61000d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f61053a6700 1 -- 172.21.15.32:0/2537190266 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520232243 0 0) 0x7f60f4000ce0 con 0x7f61000d6c80 2021-04-09T18:27:05.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f61053a6700 1 -- 172.21.15.32:0/2537190266 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60ec002fe0 con 0x7f61000d6c80 2021-04-09T18:27:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f61053a6700 1 -- 172.21.15.32:0/2537190266 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904602023 0 0) 0x7f60fc000c60 con 0x7f61000cffa0 2021-04-09T18:27:05.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.052 7f61053a6700 1 -- 172.21.15.32:0/2537190266 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60ec003210 con 0x7f61000cffa0 2021-04-09T18:27:05.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f6106ba9700 1 -- 172.21.15.32:0/2537190266 >> v1:172.21.15.86:6789/0 conn(0x7f61000cffa0 legacy=0x7f61000d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:05.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f6106ba9700 1 -- 172.21.15.32:0/2537190266 >> v1:172.21.15.32:6789/0 conn(0x7f61000d6c80 legacy=0x7f610010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:05.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f61053a6700 1 -- 172.21.15.32:0/2537190266 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1345096920 0 0) 0x7f60f40010e0 con 0x7f61000d6c80 2021-04-09T18:27:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f6106ba9700 1 -- 172.21.15.32:0/2537190266 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000d91e0 con 0x7f610010e7d0 2021-04-09T18:27:05.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f6106ba9700 1 --2- 172.21.15.32:0/2537190266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610010e7d0 0x7f61000d7ff0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f60f8006e50 tx=0x7f60f8009df0).ready entity=mon.1 client_cookie=95c5105422541973 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f61053a6700 1 -- 172.21.15.32:0/2537190266 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60f800b510 con 0x7f610010e7d0 2021-04-09T18:27:05.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f61053a6700 1 -- 172.21.15.32:0/2537190266 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f8003030 con 0x7f610010e7d0 2021-04-09T18:27:05.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f61053a6700 1 -- 172.21.15.32:0/2537190266 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60f800b7d0 con 0x7f610010e7d0 2021-04-09T18:27:05.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f610858c700 1 -- 172.21.15.32:0/2537190266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610010e7d0 msgr2=0x7f61000d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:05.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.053 7f610858c700 1 --2- 172.21.15.32:0/2537190266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610010e7d0 0x7f61000d7ff0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f60f8006e50 tx=0x7f60f8009df0).stop 2021-04-09T18:27:05.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.054 7f610858c700 1 -- 172.21.15.32:0/2537190266 shutdown_connections 2021-04-09T18:27:05.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.054 7f610858c700 1 --2- 172.21.15.32:0/2537190266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610010e7d0 0x7f61000d7ff0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:05.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.054 7f610858c700 1 -- 172.21.15.32:0/2537190266 >> 172.21.15.32:0/2537190266 conn(0x7f61000ce5c0 msgr2=0x7f61000cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:27:05.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.054 7f610858c700 1 -- 172.21.15.32:0/2537190266 shutdown_connections 2021-04-09T18:27:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.054 7f610858c700 1 -- 172.21.15.32:0/2537190266 wait complete. 2021-04-09T18:27:05.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.074 7f610858c700 1 Processor -- start 2021-04-09T18:27:05.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.074 7f610858c700 1 -- start start 2021-04-09T18:27:05.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f610858c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001842b0 0x7f61001acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:05.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f610858c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61001847e0 0x7f61001857d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f610858c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61000b6ff0 0x7f61000b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:05.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f610858c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6100184be0 con 0x7f61001842b0 2021-04-09T18:27:05.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f610858c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610018d0f0 con 0x7f61000b6ff0 2021-04-09T18:27:05.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f610858c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61001861e0 con 0x7f61001847e0 2021-04-09T18:27:05.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f61063a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001842b0 0x7f61001acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:05.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f61063a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001842b0 0x7f61001acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53626/0 (socket says 172.21.15.32:53626) 2021-04-09T18:27:05.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f61063a8700 1 -- 172.21.15.32:0/2457185346 learned_addr learned my addr 172.21.15.32:0/2457185346 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:27:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f6106ba9700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61000b6ff0 0x7f61000b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:05.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f6105ba7700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61001847e0 0x7f61001857d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f61063a8700 1 -- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61001847e0 msgr2=0x7f61001857d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:05.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f61063a8700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61001847e0 0x7f61001857d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:05.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f61063a8700 1 -- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61000b6ff0 msgr2=0x7f61000b73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:05.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f61063a8700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61000b6ff0 0x7f61000b73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:05.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f61063a8700 1 -- 172.21.15.32:0/2457185346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6100185df0 con 0x7f61001842b0 2021-04-09T18:27:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f61063a8700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001842b0 0x7f61001acd90 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 rx=0x7f60fc00b580 tx=0x7f60fc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:05.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f60f2ffd700 1 -- 172.21.15.32:0/2457185346 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60fc00bd10 con 0x7f61001842b0 2021-04-09T18:27:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f60f2ffd700 1 -- 172.21.15.32:0/2457185346 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60fc000e20 con 0x7f61001842b0 2021-04-09T18:27:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f610858c700 1 -- 172.21.15.32:0/2457185346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6100185df0 con 0x7f61001842b0 2021-04-09T18:27:05.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f60f2ffd700 1 -- 172.21.15.32:0/2457185346 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60fc0054f0 con 0x7f61001842b0 2021-04-09T18:27:05.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.075 7f610858c700 1 -- 172.21.15.32:0/2457185346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6100185df0 con 0x7f61001842b0 2021-04-09T18:27:05.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.076 7f60f2ffd700 1 -- 172.21.15.32:0/2457185346 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f60fc009fe0 con 0x7f61001842b0 2021-04-09T18:27:05.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.076 7f60f2ffd700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60e80188c0 0x7f60e801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:27:05.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.076 7f60f2ffd700 1 -- 172.21.15.32:0/2457185346 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f60fc0328c0 con 0x7f61001842b0 2021-04-09T18:27:05.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.076 7f6105ba7700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60e80188c0 0x7f60e801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:27:05.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.077 7f6105ba7700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60e80188c0 0x7f60e801adc0 crc :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:27:05.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.096 7f610858c700 1 -- 172.21.15.32:0/2457185346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6100184be0 con 0x7f61001842b0 2021-04-09T18:27:05.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.099 7f60f2ffd700 1 -- 172.21.15.32:0/2457185346 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f60fc01f790 con 0x7f61001842b0 2021-04-09T18:27:05.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.291 7f610858c700 1 -- 172.21.15.32:0/2457185346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6100184be0 con 0x7f61001842b0 2021-04-09T18:27:05.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.291 7f60f2ffd700 1 -- 172.21.15.32:0/2457185346 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+2506 (secure 0 0 0) 0x7f60fc009fe0 con 0x7f61001842b0 2021-04-09T18:27:05.366 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 284 2021-04-09T18:27:05.367 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:27:05.367 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":284,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":284,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:27:04.481684","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"laggy_since":"2021-04-09 18:26:57.582075","export_targets":[],"features":4611087854035861503,"flags":1},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:resolve","state_seq":12,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:27:05.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 -- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60e80188c0 msgr2=0x7f60e801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:05.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60e80188c0 0x7f60e801adc0 crc :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:05.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 -- 172.21.15.32:0/2457185346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001842b0 msgr2=0x7f61001acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:27:05.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001842b0 0x7f61001acd90 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 rx=0x7f60fc00b580 tx=0x7f60fc00b920).stop 2021-04-09T18:27:05.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 -- 172.21.15.32:0/2457185346 shutdown_connections 2021-04-09T18:27:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60e80188c0 0x7f60e801adc0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61000b6ff0 0x7f61000b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61001847e0 0x7f61001857d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:05.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 --2- 172.21.15.32:0/2457185346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001842b0 0x7f61001acd90 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:27:05.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 -- 172.21.15.32:0/2457185346 >> 172.21.15.32:0/2457185346 conn(0x7f6100183840 msgr2=0x7f610019b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:27:05.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 -- 172.21.15.32:0/2457185346 shutdown_connections 2021-04-09T18:27:05.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:27:05.294 7f610858c700 1 -- 172.21.15.32:0/2457185346 wait complete. 2021-04-09T18:27:05.378 DEBUG:tasks.cephfs.filesystem:Looked up MDS state for mds.1: up:resolve 2021-04-09T18:27:05.379 DEBUG:tasks.cephfs.filesystem:reached state 'up:resolve' in 1.8078036308288574s 2021-04-09T18:27:07.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:27:07.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:27:37.451 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:27:37.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:28:05.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:05.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.461 7f7b161c6700 1 Processor -- start 2021-04-09T18:28:05.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.461 7f7b161c6700 1 -- start start 2021-04-09T18:28:05.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.461 7f7b161c6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b100d7ca0 0x7f7b100d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.461 7f7b161c6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b1010da00 0x7f7b1010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b161c6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1010d480 con 0x7f7b1010d010 2021-04-09T18:28:05.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b161c6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b100d9220 con 0x7f7b1010da00 2021-04-09T18:28:05.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b161c6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b100d86c0 con 0x7f7b100d7ca0 2021-04-09T18:28:05.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b1010da00 0x7f7b1010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b1010da00 0x7f7b1010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53654/0 (socket says 172.21.15.32:53654) 2021-04-09T18:28:05.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0ffff700 1 -- 172.21.15.32:0/1692734214 learned_addr learned my addr 172.21.15.32:0/1692734214 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:05.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0effd700 1 --2- 172.21.15.32:0/1692734214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b100d7ca0 0x7f7b100d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0ffff700 1 -- 172.21.15.32:0/1692734214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b100d7ca0 msgr2=0x7f7b100d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:05.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0ffff700 1 --2- 172.21.15.32:0/1692734214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b100d7ca0 0x7f7b100d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0ffff700 1 -- 172.21.15.32:0/1692734214 >> v1:172.21.15.86:6789/0 conn(0x7f7b1010d010 legacy=0x7f7b100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0ffff700 1 -- 172.21.15.32:0/1692734214 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b100d1f70 con 0x7f7b1010da00 2021-04-09T18:28:05.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0ffff700 1 --2- 172.21.15.32:0/1692734214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1010da00 0x7f7b1010de00 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 rx=0x7f7b08006f00 tx=0x7f7b08009f10).ready entity=mon.0 client_cookie=27cf6168a0cb4dd7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:05.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0e7fc700 1 -- 172.21.15.32:0/1692734214 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b0800b3b0 con 0x7f7b1010da00 2021-04-09T18:28:05.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0e7fc700 1 -- 172.21.15.32:0/1692734214 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0800b730 con 0x7f7b1010da00 2021-04-09T18:28:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b0e7fc700 1 -- 172.21.15.32:0/1692734214 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b0800bc10 con 0x7f7b1010da00 2021-04-09T18:28:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b161c6700 1 -- 172.21.15.32:0/1692734214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1010da00 msgr2=0x7f7b1010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.462 7f7b161c6700 1 --2- 172.21.15.32:0/1692734214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1010da00 0x7f7b1010de00 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 rx=0x7f7b08006f00 tx=0x7f7b08009f10).stop 2021-04-09T18:28:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.463 7f7b161c6700 1 -- 172.21.15.32:0/1692734214 shutdown_connections 2021-04-09T18:28:05.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.463 7f7b161c6700 1 --2- 172.21.15.32:0/1692734214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b1010da00 0x7f7b1010de00 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.463 7f7b161c6700 1 --2- 172.21.15.32:0/1692734214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b100d7ca0 0x7f7b100d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.463 7f7b161c6700 1 -- 172.21.15.32:0/1692734214 >> 172.21.15.32:0/1692734214 conn(0x7f7b100ce5f0 msgr2=0x7f7b100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:05.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.463 7f7b161c6700 1 -- 172.21.15.32:0/1692734214 shutdown_connections 2021-04-09T18:28:05.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.463 7f7b161c6700 1 -- 172.21.15.32:0/1692734214 wait complete. 2021-04-09T18:28:05.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.483 7f7b161c6700 1 Processor -- start 2021-04-09T18:28:05.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b161c6700 1 -- start start 2021-04-09T18:28:05.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b161c6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b101a5210 0x7f7b10185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b161c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10185a70 0x7f7b100acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b161c6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b10185e70 0x7f7b100ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b161c6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1018f7b0 con 0x7f7b10185a70 2021-04-09T18:28:05.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b161c6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b100ad7f0 con 0x7f7b10185e70 2021-04-09T18:28:05.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b161c6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b100ada10 con 0x7f7b101a5210 2021-04-09T18:28:05.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b0effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10185a70 0x7f7b100acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b0effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10185a70 0x7f7b100acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53658/0 (socket says 172.21.15.32:53658) 2021-04-09T18:28:05.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b0effd700 1 -- 172.21.15.32:0/1393321072 learned_addr learned my addr 172.21.15.32:0/1393321072 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:05.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b0ffff700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b10185e70 0x7f7b100ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.484 7f7b0f7fe700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b101a5210 0x7f7b10185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7b0effd700 1 -- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b101a5210 msgr2=0x7f7b10185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7b0effd700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b101a5210 0x7f7b10185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7b0effd700 1 -- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b10185e70 msgr2=0x7f7b100ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7b0effd700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b10185e70 0x7f7b100ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7b0effd700 1 -- 172.21.15.32:0/1393321072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b10179610 con 0x7f7b10185a70 2021-04-09T18:28:05.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7b0effd700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10185a70 0x7f7b100acbb0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 rx=0x7f7b04006f00 tx=0x7f7b0400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:05.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7af7fff700 1 -- 172.21.15.32:0/1393321072 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b0400bd30 con 0x7f7b10185a70 2021-04-09T18:28:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7af7fff700 1 -- 172.21.15.32:0/1393321072 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b04002030 con 0x7f7b10185a70 2021-04-09T18:28:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b10179610 con 0x7f7b10185a70 2021-04-09T18:28:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7af7fff700 1 -- 172.21.15.32:0/1393321072 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b04002590 con 0x7f7b10185a70 2021-04-09T18:28:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.485 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b10179610 con 0x7f7b10185a70 2021-04-09T18:28:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.486 7f7af7fff700 1 -- 172.21.15.32:0/1393321072 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7b04002210 con 0x7f7b10185a70 2021-04-09T18:28:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.486 7f7af7fff700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7af0018930 0x7f7af001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.486 7f7af7fff700 1 -- 172.21.15.32:0/1393321072 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f7b04032a60 con 0x7f7b10185a70 2021-04-09T18:28:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.486 7f7b0f7fe700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7af0018930 0x7f7af001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.486 7f7b0f7fe700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7af0018930 0x7f7af001ae10 crc :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:05.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.505 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1018f7b0 con 0x7f7b10185a70 2021-04-09T18:28:05.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.509 7f7af7fff700 1 -- 172.21.15.32:0/1393321072 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7b04002240 con 0x7f7b10185a70 2021-04-09T18:28:05.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.701 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b1018f7b0 con 0x7f7b10185a70 2021-04-09T18:28:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.701 7f7af7fff700 1 -- 172.21.15.32:0/1393321072 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 285 v285) v1 ==== 95+0+2844 (secure 0 0 0) 0x7f7b04002010 con 0x7f7b10185a70 2021-04-09T18:28:05.780 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 285 2021-04-09T18:28:05.781 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:05.781 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":285,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":284,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:27:04.481684","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"laggy_since":"2021-04-09 18:26:57.582075","export_targets":[],"features":4611087854035861503,"flags":1},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:resolve","state_seq":12,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:28:05.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7af0018930 msgr2=0x7f7af001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7af0018930 0x7f7af001ae10 crc :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10185a70 msgr2=0x7f7b100acbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10185a70 0x7f7b100acbb0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 rx=0x7f7b04006f00 tx=0x7f7b0400b940).stop 2021-04-09T18:28:05.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 shutdown_connections 2021-04-09T18:28:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7af0018930 0x7f7af001ae10 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b10185a70 0x7f7b100acbb0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b10185e70 0x7f7b100ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 --2- 172.21.15.32:0/1393321072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b101a5210 0x7f7b10185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 >> 172.21.15.32:0/1393321072 conn(0x7f7b101a99c0 msgr2=0x7f7b1017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 shutdown_connections 2021-04-09T18:28:05.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.704 7f7b161c6700 1 -- 172.21.15.32:0/1393321072 wait complete. 2021-04-09T18:28:05.793 INFO:tasks.ceph.mds.d:Sent signal 18 2021-04-09T18:28:05.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.862 7fca02efe700 1 Processor -- start 2021-04-09T18:28:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.862 7fca02efe700 1 -- start start 2021-04-09T18:28:05.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.862 7fca02efe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc10da00 0x7fc9fc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca02efe700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc10d480 con 0x7fc9fc10d010 2021-04-09T18:28:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca02efe700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc0d1f70 con 0x7fc9fc0d7ca0 2021-04-09T18:28:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca02efe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc0d9220 con 0x7fc9fc10da00 2021-04-09T18:28:05.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca00d1a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc9fc10d010 0x7fc9fc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35024/0 (socket says 172.21.15.32:35024) 2021-04-09T18:28:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca00d1a700 1 -- 172.21.15.32:0/165151464 learned_addr learned my addr 172.21.15.32:0/165151464 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca0151b700 1 --2- 172.21.15.32:0/165151464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc10da00 0x7fc9fc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fc9fb7fe700 1 -- 172.21.15.32:0/165151464 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466208216 0 0) 0x7fc9e4000ce0 con 0x7fc9fc10d010 2021-04-09T18:28:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fc9fb7fe700 1 -- 172.21.15.32:0/165151464 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e8002fe0 con 0x7fc9fc10d010 2021-04-09T18:28:05.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fc9fb7fe700 1 -- 172.21.15.32:0/165151464 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232567225 0 0) 0x7fc9ec001400 con 0x7fc9fc0d7ca0 2021-04-09T18:28:05.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fc9fb7fe700 1 -- 172.21.15.32:0/165151464 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e8003210 con 0x7fc9fc0d7ca0 2021-04-09T18:28:05.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca0151b700 1 -- 172.21.15.32:0/165151464 >> v1:172.21.15.86:6789/0 conn(0x7fc9fc0d7ca0 legacy=0x7fc9fc0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca0151b700 1 -- 172.21.15.32:0/165151464 >> v1:172.21.15.32:6789/0 conn(0x7fc9fc10d010 legacy=0x7fc9fc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca0151b700 1 -- 172.21.15.32:0/165151464 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc10e420 con 0x7fc9fc10da00 2021-04-09T18:28:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.863 7fca0151b700 1 --2- 172.21.15.32:0/165151464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc10da00 0x7fc9fc10de00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fc9f4006ad0 tx=0x7fc9f4009ad0).ready entity=mon.2 client_cookie=c3cec4d6e88202d7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fc9fb7fe700 1 -- 172.21.15.32:0/165151464 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9f40022e0 con 0x7fc9fc10da00 2021-04-09T18:28:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fc9fb7fe700 1 -- 172.21.15.32:0/165151464 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f4006410 con 0x7fc9fc10da00 2021-04-09T18:28:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fc9fb7fe700 1 -- 172.21.15.32:0/165151464 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9f4002620 con 0x7fc9fc10da00 2021-04-09T18:28:05.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fca02efe700 1 -- 172.21.15.32:0/165151464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc10da00 msgr2=0x7fc9fc10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fca02efe700 1 --2- 172.21.15.32:0/165151464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc10da00 0x7fc9fc10de00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fc9f4006ad0 tx=0x7fc9f4009ad0).stop 2021-04-09T18:28:05.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fca02efe700 1 -- 172.21.15.32:0/165151464 shutdown_connections 2021-04-09T18:28:05.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fca02efe700 1 --2- 172.21.15.32:0/165151464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc10da00 0x7fc9fc10de00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fca02efe700 1 -- 172.21.15.32:0/165151464 >> 172.21.15.32:0/165151464 conn(0x7fc9fc0ce5f0 msgr2=0x7fc9fc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:05.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fca02efe700 1 -- 172.21.15.32:0/165151464 shutdown_connections 2021-04-09T18:28:05.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.864 7fca02efe700 1 -- 172.21.15.32:0/165151464 wait complete. 2021-04-09T18:28:05.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.884 7fca02efe700 1 Processor -- start 2021-04-09T18:28:05.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.884 7fca02efe700 1 -- start start 2021-04-09T18:28:05.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fca02efe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc185a00 0x7fc9fc097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fca02efe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9fc0983b0 0x7fc9fc17abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fca02efe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9fc17b1d0 0x7fc9fc179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fca02efe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc0987b0 con 0x7fc9fc0983b0 2021-04-09T18:28:05.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fca02efe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc17b5d0 con 0x7fc9fc17b1d0 2021-04-09T18:28:05.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fca02efe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc179c30 con 0x7fc9fc185a00 2021-04-09T18:28:05.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fc9fbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9fc0983b0 0x7fc9fc17abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fc9fbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9fc0983b0 0x7fc9fc17abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53670/0 (socket says 172.21.15.32:53670) 2021-04-09T18:28:05.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fc9fbfff700 1 -- 172.21.15.32:0/882822172 learned_addr learned my addr 172.21.15.32:0/882822172 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fca00d1a700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc185a00 0x7fc9fc097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.885 7fca0151b700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9fc17b1d0 0x7fc9fc179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fc9fbfff700 1 -- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc185a00 msgr2=0x7fc9fc097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fc9fbfff700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc185a00 0x7fc9fc097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fc9fbfff700 1 -- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9fc17b1d0 msgr2=0x7fc9fc179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fc9fbfff700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9fc17b1d0 0x7fc9fc179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fc9fbfff700 1 -- 172.21.15.32:0/882822172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc179e50 con 0x7fc9fc0983b0 2021-04-09T18:28:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fc9fbfff700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9fc0983b0 0x7fc9fc17abb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7fc9ec009980 tx=0x7fc9ec00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:05.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fc9f97fa700 1 -- 172.21.15.32:0/882822172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9ec003020 con 0x7fc9fc0983b0 2021-04-09T18:28:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fc9f97fa700 1 -- 172.21.15.32:0/882822172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ec005870 con 0x7fc9fc0983b0 2021-04-09T18:28:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fc9f97fa700 1 -- 172.21.15.32:0/882822172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9ec005d50 con 0x7fc9fc0983b0 2021-04-09T18:28:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fca02efe700 1 -- 172.21.15.32:0/882822172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9fc179e50 con 0x7fc9fc0983b0 2021-04-09T18:28:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.886 7fca02efe700 1 -- 172.21.15.32:0/882822172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9fc179e50 con 0x7fc9fc0983b0 2021-04-09T18:28:05.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.887 7fc9f97fa700 1 -- 172.21.15.32:0/882822172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc9ec005a50 con 0x7fc9fc0983b0 2021-04-09T18:28:05.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.887 7fc9f97fa700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9e0018930 0x7fc9e001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:05.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.887 7fc9f97fa700 1 -- 172.21.15.32:0/882822172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fc9ec032e50 con 0x7fc9fc0983b0 2021-04-09T18:28:05.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.887 7fca00d1a700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9e0018930 0x7fc9e001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:05.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.887 7fca00d1a700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9e0018930 0x7fc9e001adc0 crc :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:05.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.907 7fca02efe700 1 -- 172.21.15.32:0/882822172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9fc17a3e0 con 0x7fc9fc0983b0 2021-04-09T18:28:05.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:05.910 7fc9f97fa700 1 -- 172.21.15.32:0/882822172 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc9ec00be10 con 0x7fc9fc0983b0 2021-04-09T18:28:06.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.102 7fca02efe700 1 -- 172.21.15.32:0/882822172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9fc0987b0 con 0x7fc9fc0983b0 2021-04-09T18:28:06.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.102 7fc9f97fa700 1 -- 172.21.15.32:0/882822172 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 285 v285) v1 ==== 95+0+2844 (secure 0 0 0) 0x7fc9ec005870 con 0x7fc9fc0983b0 2021-04-09T18:28:06.182 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 285 2021-04-09T18:28:06.182 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:06.182 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":285,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":284,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:27:04.481684","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"laggy_since":"2021-04-09 18:26:57.582075","export_targets":[],"features":4611087854035861503,"flags":1},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:resolve","state_seq":12,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:28:06.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 -- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9e0018930 msgr2=0x7fc9e001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:06.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9e0018930 0x7fc9e001adc0 crc :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:06.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 -- 172.21.15.32:0/882822172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9fc0983b0 msgr2=0x7fc9fc17abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:06.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9fc0983b0 0x7fc9fc17abb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7fc9ec009980 tx=0x7fc9ec00bcd0).stop 2021-04-09T18:28:06.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 -- 172.21.15.32:0/882822172 shutdown_connections 2021-04-09T18:28:06.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc9e0018930 0x7fc9e001adc0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:06.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9fc0983b0 0x7fc9fc17abb0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:06.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9fc17b1d0 0x7fc9fc179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:06.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 --2- 172.21.15.32:0/882822172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9fc185a00 0x7fc9fc097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:06.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 -- 172.21.15.32:0/882822172 >> 172.21.15.32:0/882822172 conn(0x7fc9fc190550 msgr2=0x7fc9fc1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:06.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 -- 172.21.15.32:0/882822172 shutdown_connections 2021-04-09T18:28:06.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:06.105 7fca02efe700 1 -- 172.21.15.32:0/882822172 wait complete. 2021-04-09T18:28:06.193 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:28:07.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:28:07.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:28:11.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:11.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.276 7fe5de6c0700 1 Processor -- start 2021-04-09T18:28:11.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.276 7fe5de6c0700 1 -- start start 2021-04-09T18:28:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.277 7fe5de6c0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe5d810d010 0x7fe5d80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.277 7fe5de6c0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d80d8490 0x7fe5d810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.277 7fe5de6c0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d810d480 con 0x7fe5d80d1da0 2021-04-09T18:28:11.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.277 7fe5de6c0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d80d8d30 con 0x7fe5d810d010 2021-04-09T18:28:11.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.277 7fe5de6c0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80d9220 con 0x7fe5d80d8490 2021-04-09T18:28:11.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.277 7fe5cffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe5d80d1da0 0x7fe5d80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35040/0 (socket says 172.21.15.32:35040) 2021-04-09T18:28:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.277 7fe5cffff700 1 -- 172.21.15.32:0/3732182154 learned_addr learned my addr 172.21.15.32:0/3732182154 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:11.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.277 7fe5d7fff700 1 --2- 172.21.15.32:0/3732182154 >> v2:172.21.15.86:3300/0 conn(0x7fe5d810d010 0x7fe5d80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.277 7fe5dccdd700 1 --2- 172.21.15.32:0/3732182154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d80d8490 0x7fe5d810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d77fe700 1 -- 172.21.15.32:0/3732182154 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518705609 0 0) 0x7fe5c0000ce0 con 0x7fe5d80d1da0 2021-04-09T18:28:11.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d77fe700 1 -- 172.21.15.32:0/3732182154 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c4002fe0 con 0x7fe5d80d1da0 2021-04-09T18:28:11.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d7fff700 1 -- 172.21.15.32:0/3732182154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d80d8490 msgr2=0x7fe5d810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:11.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d7fff700 1 --2- 172.21.15.32:0/3732182154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d80d8490 0x7fe5d810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d7fff700 1 -- 172.21.15.32:0/3732182154 >> v1:172.21.15.32:6789/0 conn(0x7fe5d80d1da0 legacy=0x7fe5d80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d7fff700 1 -- 172.21.15.32:0/3732182154 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d810e020 con 0x7fe5d810d010 2021-04-09T18:28:11.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5dccdd700 1 --2- 172.21.15.32:0/3732182154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d80d8490 0x7fe5d810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:11.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d7fff700 1 --2- 172.21.15.32:0/3732182154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5d810d010 0x7fe5d80d1780 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fe5c8006650 tx=0x7fe5c8009a80).ready entity=mon.1 client_cookie=b09428b5dbaba8ed server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:11.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d77fe700 1 -- 172.21.15.32:0/3732182154 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5c80009b0 con 0x7fe5d810d010 2021-04-09T18:28:11.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d77fe700 1 -- 172.21.15.32:0/3732182154 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c800c0c0 con 0x7fe5d810d010 2021-04-09T18:28:11.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5d77fe700 1 -- 172.21.15.32:0/3732182154 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5c800c510 con 0x7fe5d810d010 2021-04-09T18:28:11.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5de6c0700 1 -- 172.21.15.32:0/3732182154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5d810d010 msgr2=0x7fe5d80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.278 7fe5de6c0700 1 --2- 172.21.15.32:0/3732182154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5d810d010 0x7fe5d80d1780 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fe5c8006650 tx=0x7fe5c8009a80).stop 2021-04-09T18:28:11.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.279 7fe5de6c0700 1 -- 172.21.15.32:0/3732182154 shutdown_connections 2021-04-09T18:28:11.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.279 7fe5de6c0700 1 --2- 172.21.15.32:0/3732182154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d80d8490 0x7fe5d810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.279 7fe5de6c0700 1 --2- 172.21.15.32:0/3732182154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5d810d010 0x7fe5d80d1780 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.279 7fe5de6c0700 1 -- 172.21.15.32:0/3732182154 >> 172.21.15.32:0/3732182154 conn(0x7fe5d80ce5f0 msgr2=0x7fe5d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:11.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.279 7fe5de6c0700 1 -- 172.21.15.32:0/3732182154 shutdown_connections 2021-04-09T18:28:11.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.279 7fe5de6c0700 1 -- 172.21.15.32:0/3732182154 wait complete. 2021-04-09T18:28:11.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.299 7fe5de6c0700 1 Processor -- start 2021-04-09T18:28:11.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.299 7fe5de6c0700 1 -- start start 2021-04-09T18:28:11.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5de6c0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5d81a5210 0x7fe5d808eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5de6c0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d81a8ab0 0x7fe5d81a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5de6c0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d81a76d0 0x7fe5d81a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5de6c0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d817ea50 con 0x7fe5d81a76d0 2021-04-09T18:28:11.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5de6c0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d808f4f0 con 0x7fe5d81a5210 2021-04-09T18:28:11.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5de6c0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d808ea60 con 0x7fe5d81a8ab0 2021-04-09T18:28:11.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5dccdd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d81a76d0 0x7fe5d81a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5dccdd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d81a76d0 0x7fe5d81a7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53686/0 (socket says 172.21.15.32:53686) 2021-04-09T18:28:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5dccdd700 1 -- 172.21.15.32:0/4120764592 learned_addr learned my addr 172.21.15.32:0/4120764592 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5cffff700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d81a8ab0 0x7fe5d81a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5d7fff700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5d81a5210 0x7fe5d808eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5dccdd700 1 -- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d81a8ab0 msgr2=0x7fe5d81a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5dccdd700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d81a8ab0 0x7fe5d81a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5dccdd700 1 -- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5d81a5210 msgr2=0x7fe5d808eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5dccdd700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5d81a5210 0x7fe5d808eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5dccdd700 1 -- 172.21.15.32:0/4120764592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d81a80f0 con 0x7fe5d81a76d0 2021-04-09T18:28:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.300 7fe5dccdd700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d81a76d0 0x7fe5d81a7ad0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 rx=0x7fe5d000b910 tx=0x7fe5d000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.301 7fe5d57fa700 1 -- 172.21.15.32:0/4120764592 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5d0006aa0 con 0x7fe5d81a76d0 2021-04-09T18:28:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.301 7fe5d57fa700 1 -- 172.21.15.32:0/4120764592 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d00011f0 con 0x7fe5d81a76d0 2021-04-09T18:28:11.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.301 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d81a80f0 con 0x7fe5d81a76d0 2021-04-09T18:28:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.301 7fe5d57fa700 1 -- 172.21.15.32:0/4120764592 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5d0001640 con 0x7fe5d81a76d0 2021-04-09T18:28:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.301 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d81a80f0 con 0x7fe5d81a76d0 2021-04-09T18:28:11.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.301 7fe5d57fa700 1 -- 172.21.15.32:0/4120764592 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe5d00013d0 con 0x7fe5d81a76d0 2021-04-09T18:28:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.301 7fe5d57fa700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5bc018930 0x7fe5bc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.302 7fe5d57fa700 1 -- 172.21.15.32:0/4120764592 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fe5d0032e20 con 0x7fe5d81a76d0 2021-04-09T18:28:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.302 7fe5d7fff700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5bc018930 0x7fe5bc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.302 7fe5d7fff700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5bc018930 0x7fe5bc01adc0 crc :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:11.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.321 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d81b7070 con 0x7fe5d81a76d0 2021-04-09T18:28:11.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.324 7fe5d57fa700 1 -- 172.21.15.32:0/4120764592 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe5d000bdf0 con 0x7fe5d81a76d0 2021-04-09T18:28:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.516 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5d810d480 con 0x7fe5d81a76d0 2021-04-09T18:28:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.516 7fe5d57fa700 1 -- 172.21.15.32:0/4120764592 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 287 v287) v1 ==== 95+0+2803 (secure 0 0 0) 0x7fe5d00011f0 con 0x7fe5d81a76d0 2021-04-09T18:28:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 287 2021-04-09T18:28:11.596 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:11.597 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":287,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:28:11.422642","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":7,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":1},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:28:11.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5bc018930 msgr2=0x7fe5bc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5bc018930 0x7fe5bc01adc0 crc :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d81a76d0 msgr2=0x7fe5d81a7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d81a76d0 0x7fe5d81a7ad0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 rx=0x7fe5d000b910 tx=0x7fe5d000bcb0).stop 2021-04-09T18:28:11.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 shutdown_connections 2021-04-09T18:28:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5bc018930 0x7fe5bc01adc0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5d81a76d0 0x7fe5d81a7ad0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5d81a8ab0 0x7fe5d81a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 --2- 172.21.15.32:0/4120764592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5d81a5210 0x7fe5d808eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 >> 172.21.15.32:0/4120764592 conn(0x7fe5d81a99c0 msgr2=0x7fe5d817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:11.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 shutdown_connections 2021-04-09T18:28:11.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.519 7fe5de6c0700 1 -- 172.21.15.32:0/4120764592 wait complete. 2021-04-09T18:28:11.609 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2021-04-09T18:28:11.610 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.679 7f5e6cee0700 1 Processor -- start 2021-04-09T18:28:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.679 7f5e6cee0700 1 -- start start 2021-04-09T18:28:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.680 7f5e6cee0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5e680d6c80 0x7f5e680d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.680 7f5e6cee0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e680d2440 0x7f5e6810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.680 7f5e6cee0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e680d7280 con 0x7f5e680cffa0 2021-04-09T18:28:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.680 7f5e6cee0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e680d7080 con 0x7f5e680d6c80 2021-04-09T18:28:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.680 7f5e6cee0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e680d91e0 con 0x7f5e680d2440 2021-04-09T18:28:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.680 7f5e67314700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e680d2440 0x7f5e6810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.680 7f5e67314700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e680d2440 0x7f5e6810cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:60918/0 (socket says 172.21.15.32:60918) 2021-04-09T18:28:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.680 7f5e66312700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5e680d6c80 0x7f5e680d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.680 7f5e67314700 1 -- 172.21.15.32:0/3724252108 learned_addr learned my addr 172.21.15.32:0/3724252108 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e67314700 1 -- 172.21.15.32:0/3724252108 >> v2:172.21.15.86:3300/0 conn(0x7f5e680d6c80 msgr2=0x7f5e680d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e67314700 1 --2- 172.21.15.32:0/3724252108 >> v2:172.21.15.86:3300/0 conn(0x7f5e680d6c80 0x7f5e680d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e65b11700 1 -- 172.21.15.32:0/3724252108 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303460071 0 0) 0x7f5e58000c60 con 0x7f5e680cffa0 2021-04-09T18:28:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e67314700 1 -- 172.21.15.32:0/3724252108 >> v1:172.21.15.86:6789/0 conn(0x7f5e680cffa0 legacy=0x7f5e680d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e67314700 1 -- 172.21.15.32:0/3724252108 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e680d2840 con 0x7f5e680d2440 2021-04-09T18:28:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e66312700 1 --2- 172.21.15.32:0/3724252108 >> v2:172.21.15.86:3300/0 conn(0x7f5e680d6c80 0x7f5e680d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e67314700 1 --2- 172.21.15.32:0/3724252108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e680d2440 0x7f5e6810cfd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f5e60006aa0 tx=0x7f5e60009f20).ready entity=mon.2 client_cookie=238eb7544f132c0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e65b11700 1 -- 172.21.15.32:0/3724252108 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e6000b5c0 con 0x7f5e680d2440 2021-04-09T18:28:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e65b11700 1 -- 172.21.15.32:0/3724252108 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e60003030 con 0x7f5e680d2440 2021-04-09T18:28:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e65b11700 1 -- 172.21.15.32:0/3724252108 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e6000bb40 con 0x7f5e680d2440 2021-04-09T18:28:11.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e6cee0700 1 -- 172.21.15.32:0/3724252108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e680d2440 msgr2=0x7f5e6810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.681 7f5e6cee0700 1 --2- 172.21.15.32:0/3724252108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e680d2440 0x7f5e6810cfd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f5e60006aa0 tx=0x7f5e60009f20).stop 2021-04-09T18:28:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.682 7f5e6cee0700 1 -- 172.21.15.32:0/3724252108 shutdown_connections 2021-04-09T18:28:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.682 7f5e6cee0700 1 --2- 172.21.15.32:0/3724252108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e680d2440 0x7f5e6810cfd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.682 7f5e6cee0700 1 --2- 172.21.15.32:0/3724252108 >> v2:172.21.15.86:3300/0 conn(0x7f5e680d6c80 0x7f5e680d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.682 7f5e6cee0700 1 -- 172.21.15.32:0/3724252108 >> 172.21.15.32:0/3724252108 conn(0x7f5e680ce5c0 msgr2=0x7f5e680cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.682 7f5e6cee0700 1 -- 172.21.15.32:0/3724252108 shutdown_connections 2021-04-09T18:28:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.682 7f5e6cee0700 1 -- 172.21.15.32:0/3724252108 wait complete. 2021-04-09T18:28:11.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.702 7f5e6cee0700 1 Processor -- start 2021-04-09T18:28:11.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.702 7f5e6cee0700 1 -- start start 2021-04-09T18:28:11.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e6cee0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e681acff0 0x7f5e681847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e6cee0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e68183fc0 0x7f5e681a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e6cee0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e68186ff0 0x7f5e681873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e6cee0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e68188c30 con 0x7f5e68186ff0 2021-04-09T18:28:11.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e6cee0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e681843c0 con 0x7f5e68183fc0 2021-04-09T18:28:11.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e6cee0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e681a5c20 con 0x7f5e681acff0 2021-04-09T18:28:11.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e67314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e68186ff0 0x7f5e681873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e67314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e68186ff0 0x7f5e681873f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53698/0 (socket says 172.21.15.32:53698) 2021-04-09T18:28:11.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e67314700 1 -- 172.21.15.32:0/1576372658 learned_addr learned my addr 172.21.15.32:0/1576372658 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:11.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e66312700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e68183fc0 0x7f5e681a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e66b13700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e681acff0 0x7f5e681847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e67314700 1 -- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e681acff0 msgr2=0x7f5e681847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e67314700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e681acff0 0x7f5e681847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e67314700 1 -- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e68183fc0 msgr2=0x7f5e681a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e67314700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e68183fc0 0x7f5e681a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e67314700 1 -- 172.21.15.32:0/1576372658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e68187a10 con 0x7f5e68186ff0 2021-04-09T18:28:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.703 7f5e67314700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e68186ff0 0x7f5e681873f0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 rx=0x7f5e60009f70 tx=0x7f5e6000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.704 7f5e5f7fe700 1 -- 172.21.15.32:0/1576372658 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e60006770 con 0x7f5e68186ff0 2021-04-09T18:28:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.704 7f5e5f7fe700 1 -- 172.21.15.32:0/1576372658 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e60001070 con 0x7f5e68186ff0 2021-04-09T18:28:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.704 7f5e5f7fe700 1 -- 172.21.15.32:0/1576372658 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e600014c0 con 0x7f5e68186ff0 2021-04-09T18:28:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.704 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e68187a10 con 0x7f5e68186ff0 2021-04-09T18:28:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.704 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e68187a10 con 0x7f5e68186ff0 2021-04-09T18:28:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.704 7f5e5f7fe700 1 -- 172.21.15.32:0/1576372658 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5e60006770 con 0x7f5e68186ff0 2021-04-09T18:28:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.704 7f5e5f7fe700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e440188d0 0x7f5e4401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.704 7f5e5f7fe700 1 -- 172.21.15.32:0/1576372658 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f5e60032f70 con 0x7f5e68186ff0 2021-04-09T18:28:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.704 7f5e66b13700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e440188d0 0x7f5e4401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.705 7f5e66b13700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e440188d0 0x7f5e4401add0 crc :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:11.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.724 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e68182dc0 con 0x7f5e68186ff0 2021-04-09T18:28:11.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.728 7f5e5f7fe700 1 -- 172.21.15.32:0/1576372658 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5e60009140 con 0x7f5e68186ff0 2021-04-09T18:28:12.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.921 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e680d7280 con 0x7f5e68186ff0 2021-04-09T18:28:12.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.922 7f5e5f7fe700 1 -- 172.21.15.32:0/1576372658 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 287 v287) v1 ==== 95+0+2803 (secure 0 0 0) 0x7f5e60006770 con 0x7f5e68186ff0 2021-04-09T18:28:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 287 2021-04-09T18:28:12.002 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:12.002 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":287,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:28:11.422642","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":7,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":1},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:28:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e440188d0 msgr2=0x7f5e4401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e440188d0 0x7f5e4401add0 crc :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e68186ff0 msgr2=0x7f5e681873f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e68186ff0 0x7f5e681873f0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 rx=0x7f5e60009f70 tx=0x7f5e6000bcf0).stop 2021-04-09T18:28:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 shutdown_connections 2021-04-09T18:28:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5e440188d0 0x7f5e4401add0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e68186ff0 0x7f5e681873f0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e68183fc0 0x7f5e681a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 --2- 172.21.15.32:0/1576372658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e681acff0 0x7f5e681847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 >> 172.21.15.32:0/1576372658 conn(0x7f5e6809abb0 msgr2=0x7f5e680a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 shutdown_connections 2021-04-09T18:28:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:11.925 7f5e6cee0700 1 -- 172.21.15.32:0/1576372658 wait complete. 2021-04-09T18:28:12.014 DEBUG:tasks.cephfs.filesystem:Looked up MDS state for mds.1: up:reconnect 2021-04-09T18:28:13.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:13.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.095 7f796d319700 1 Processor -- start 2021-04-09T18:28:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.095 7f796d319700 1 -- start start 2021-04-09T18:28:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.096 7f796d319700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f796810d010 0x7f79680d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.096 7f796d319700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f79680d8490 0x7f796810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.096 7f796d319700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796810d480 con 0x7f79680d1da0 2021-04-09T18:28:13.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.096 7f796d319700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79680d8d30 con 0x7f796810d010 2021-04-09T18:28:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.096 7f796d319700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79680d9220 con 0x7f79680d8490 2021-04-09T18:28:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.096 7f796707d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f796810d010 0x7f79680d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.096 7f796707d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f796810d010 0x7f79680d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53702/0 (socket says 172.21.15.32:53702) 2021-04-09T18:28:13.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.096 7f796707d700 1 -- 172.21.15.32:0/1156404834 learned_addr learned my addr 172.21.15.32:0/1156404834 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.096 7f796787e700 1 --2- 172.21.15.32:0/1156404834 >> v2:172.21.15.86:3300/0 conn(0x7f79680d8490 0x7f796810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796707d700 1 -- 172.21.15.32:0/1156404834 >> v2:172.21.15.86:3300/0 conn(0x7f79680d8490 msgr2=0x7f796810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796707d700 1 --2- 172.21.15.32:0/1156404834 >> v2:172.21.15.86:3300/0 conn(0x7f79680d8490 0x7f796810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796707d700 1 -- 172.21.15.32:0/1156404834 >> v1:172.21.15.86:6789/0 conn(0x7f79680d1da0 legacy=0x7f79680d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796707d700 1 -- 172.21.15.32:0/1156404834 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796810e020 con 0x7f796810d010 2021-04-09T18:28:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796707d700 1 --2- 172.21.15.32:0/1156404834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f796810d010 0x7f79680d1780 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f7958006700 tx=0x7f7958009b80).ready entity=mon.0 client_cookie=ad0cb1dc04b1bc4a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796607b700 1 -- 172.21.15.32:0/1156404834 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795800c0c0 con 0x7f796810d010 2021-04-09T18:28:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796607b700 1 -- 172.21.15.32:0/1156404834 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795800c440 con 0x7f796810d010 2021-04-09T18:28:13.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796607b700 1 -- 172.21.15.32:0/1156404834 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795800c920 con 0x7f796810d010 2021-04-09T18:28:13.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796d319700 1 -- 172.21.15.32:0/1156404834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f796810d010 msgr2=0x7f79680d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:13.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.097 7f796d319700 1 --2- 172.21.15.32:0/1156404834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f796810d010 0x7f79680d1780 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f7958006700 tx=0x7f7958009b80).stop 2021-04-09T18:28:13.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.098 7f796d319700 1 -- 172.21.15.32:0/1156404834 shutdown_connections 2021-04-09T18:28:13.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.098 7f796d319700 1 --2- 172.21.15.32:0/1156404834 >> v2:172.21.15.86:3300/0 conn(0x7f79680d8490 0x7f796810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.098 7f796d319700 1 --2- 172.21.15.32:0/1156404834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f796810d010 0x7f79680d1780 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.098 7f796d319700 1 -- 172.21.15.32:0/1156404834 >> 172.21.15.32:0/1156404834 conn(0x7f79680ce5f0 msgr2=0x7f79680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.098 7f796d319700 1 -- 172.21.15.32:0/1156404834 shutdown_connections 2021-04-09T18:28:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.098 7f796d319700 1 -- 172.21.15.32:0/1156404834 wait complete. 2021-04-09T18:28:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.118 7f796d319700 1 Processor -- start 2021-04-09T18:28:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.118 7f796d319700 1 -- start start 2021-04-09T18:28:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.118 7f796d319700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79680a90c0 0x7f79681a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796d319700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79681a76d0 0x7f79681a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796d319700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796817bce0 0x7f796817c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796d319700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79681a7c60 con 0x7f79680a90c0 2021-04-09T18:28:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796d319700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796817db80 con 0x7f79681a76d0 2021-04-09T18:28:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796d319700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79681a8e50 con 0x7f796817bce0 2021-04-09T18:28:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796707d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79680a90c0 0x7f79681a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796707d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79680a90c0 0x7f79681a81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53708/0 (socket says 172.21.15.32:53708) 2021-04-09T18:28:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796707d700 1 -- 172.21.15.32:0/868522498 learned_addr learned my addr 172.21.15.32:0/868522498 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796787e700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796817bce0 0x7f796817c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796687c700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79681a76d0 0x7f79681a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796707d700 1 -- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796817bce0 msgr2=0x7f796817c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796707d700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796817bce0 0x7f796817c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796707d700 1 -- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79681a76d0 msgr2=0x7f79681a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796707d700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79681a76d0 0x7f79681a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796707d700 1 -- 172.21.15.32:0/868522498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7968179610 con 0x7f79680a90c0 2021-04-09T18:28:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796707d700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79680a90c0 0x7f79681a81e0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 rx=0x7f795800b580 tx=0x7f795800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:13.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f795ffff700 1 -- 172.21.15.32:0/868522498 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795800bd10 con 0x7f79680a90c0 2021-04-09T18:28:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f795ffff700 1 -- 172.21.15.32:0/868522498 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7958000c90 con 0x7f79680a90c0 2021-04-09T18:28:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796d319700 1 -- 172.21.15.32:0/868522498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7968179610 con 0x7f79680a90c0 2021-04-09T18:28:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f795ffff700 1 -- 172.21.15.32:0/868522498 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7958002220 con 0x7f79680a90c0 2021-04-09T18:28:13.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.119 7f796d319700 1 -- 172.21.15.32:0/868522498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7968179610 con 0x7f79680a90c0 2021-04-09T18:28:13.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.120 7f795ffff700 1 -- 172.21.15.32:0/868522498 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7958009f90 con 0x7f79680a90c0 2021-04-09T18:28:13.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.120 7f795ffff700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79440188d0 0x7f794401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:13.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.120 7f795ffff700 1 -- 172.21.15.32:0/868522498 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f7958032a40 con 0x7f79680a90c0 2021-04-09T18:28:13.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.120 7f796687c700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79440188d0 0x7f794401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:13.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.121 7f796687c700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79440188d0 0x7f794401add0 crc :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.140 7f796d319700 1 -- 172.21.15.32:0/868522498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79681a8e50 con 0x7f79680a90c0 2021-04-09T18:28:13.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.144 7f795ffff700 1 -- 172.21.15.32:0/868522498 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f79580091e0 con 0x7f79680a90c0 2021-04-09T18:28:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.336 7f796d319700 1 -- 172.21.15.32:0/868522498 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79681a8e50 con 0x7f79680a90c0 2021-04-09T18:28:13.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.337 7f795ffff700 1 -- 172.21.15.32:0/868522498 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2800 (secure 0 0 0) 0x7f7958000c90 con 0x7f79680a90c0 2021-04-09T18:28:13.416 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:13.416 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":288,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":288,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:28:12.424497","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":7,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":1},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:28:13.417 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 288 2021-04-09T18:28:13.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 -- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79440188d0 msgr2=0x7f794401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:13.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79440188d0 0x7f794401add0 crc :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 -- 172.21.15.32:0/868522498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79680a90c0 msgr2=0x7f79681a81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:13.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79680a90c0 0x7f79681a81e0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 rx=0x7f795800b580 tx=0x7f795800b920).stop 2021-04-09T18:28:13.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 -- 172.21.15.32:0/868522498 shutdown_connections 2021-04-09T18:28:13.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f79440188d0 0x7f794401add0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796817bce0 0x7f796817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79681a76d0 0x7f79681a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 --2- 172.21.15.32:0/868522498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79680a90c0 0x7f79681a81e0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:13.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 -- 172.21.15.32:0/868522498 >> 172.21.15.32:0/868522498 conn(0x7f79681acae0 msgr2=0x7f796817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:13.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 -- 172.21.15.32:0/868522498 shutdown_connections 2021-04-09T18:28:13.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:13.339 7f796d319700 1 -- 172.21.15.32:0/868522498 wait complete. 2021-04-09T18:28:13.428 DEBUG:tasks.cephfs.filesystem:Looked up MDS state for mds.1: up:rejoin 2021-04-09T18:28:14.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.512 7fa1966fb700 1 Processor -- start 2021-04-09T18:28:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.512 7fa1966fb700 1 -- start start 2021-04-09T18:28:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa1966fb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa19010d010 0x7fa1900d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:14.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa1966fb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1900d1da0 0x7fa1900d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:14.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa1966fb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa1900d82c0 0x7fa19010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:14.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa1966fb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1900d8d30 con 0x7fa1900d82c0 2021-04-09T18:28:14.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa1966fb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1900d9220 con 0x7fa19010d010 2021-04-09T18:28:14.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa1966fb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19010d480 con 0x7fa1900d1da0 2021-04-09T18:28:14.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa194d18700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa1900d82c0 0x7fa19010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:14.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa194d18700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa1900d82c0 0x7fa19010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53722/0 (socket says 172.21.15.32:53722) 2021-04-09T18:28:14.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa194d18700 1 -- 172.21.15.32:0/662770012 learned_addr learned my addr 172.21.15.32:0/662770012 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:14.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa187fff700 1 --2- 172.21.15.32:0/662770012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1900d1da0 0x7fa1900d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:14.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa18ffff700 1 --2- 172.21.15.32:0/662770012 >> v2:172.21.15.86:3300/0 conn(0x7fa19010d010 0x7fa1900d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:14.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa194d18700 1 -- 172.21.15.32:0/662770012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1900d1da0 msgr2=0x7fa1900d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:14.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa194d18700 1 --2- 172.21.15.32:0/662770012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1900d1da0 0x7fa1900d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa194d18700 1 -- 172.21.15.32:0/662770012 >> v2:172.21.15.86:3300/0 conn(0x7fa19010d010 msgr2=0x7fa1900d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:14.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa194d18700 1 --2- 172.21.15.32:0/662770012 >> v2:172.21.15.86:3300/0 conn(0x7fa19010d010 0x7fa1900d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa194d18700 1 -- 172.21.15.32:0/662770012 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1900d86c0 con 0x7fa1900d82c0 2021-04-09T18:28:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.513 7fa194d18700 1 --2- 172.21.15.32:0/662770012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900d82c0 0x7fa19010da00 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 rx=0x7fa188006f00 tx=0x7fa188009e90).ready entity=mon.0 client_cookie=a9f492f1c7e6c9d4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:14.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa18f7fe700 1 -- 172.21.15.32:0/662770012 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa18800b300 con 0x7fa1900d82c0 2021-04-09T18:28:14.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa18f7fe700 1 -- 172.21.15.32:0/662770012 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18800b640 con 0x7fa1900d82c0 2021-04-09T18:28:14.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa18f7fe700 1 -- 172.21.15.32:0/662770012 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa18800baa0 con 0x7fa1900d82c0 2021-04-09T18:28:14.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa1966fb700 1 -- 172.21.15.32:0/662770012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900d82c0 msgr2=0x7fa19010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:14.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa1966fb700 1 --2- 172.21.15.32:0/662770012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900d82c0 0x7fa19010da00 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 rx=0x7fa188006f00 tx=0x7fa188009e90).stop 2021-04-09T18:28:14.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa1966fb700 1 -- 172.21.15.32:0/662770012 shutdown_connections 2021-04-09T18:28:14.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa1966fb700 1 --2- 172.21.15.32:0/662770012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900d82c0 0x7fa19010da00 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa1966fb700 1 --2- 172.21.15.32:0/662770012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1900d1da0 0x7fa1900d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa1966fb700 1 --2- 172.21.15.32:0/662770012 >> v2:172.21.15.86:3300/0 conn(0x7fa19010d010 0x7fa1900d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.514 7fa1966fb700 1 -- 172.21.15.32:0/662770012 >> 172.21.15.32:0/662770012 conn(0x7fa1900ce5f0 msgr2=0x7fa1900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:14.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.515 7fa1966fb700 1 -- 172.21.15.32:0/662770012 shutdown_connections 2021-04-09T18:28:14.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.515 7fa1966fb700 1 -- 172.21.15.32:0/662770012 wait complete. 2021-04-09T18:28:14.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa1966fb700 1 Processor -- start 2021-04-09T18:28:14.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa1966fb700 1 -- start start 2021-04-09T18:28:14.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa1966fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900b42d0 0x7fa190183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:14.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa1966fb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1901ac090 0x7fa1901ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa1966fb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa190184d30 0x7fa190185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa1966fb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1900a1a90 con 0x7fa1900b42d0 2021-04-09T18:28:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa1966fb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901acab0 con 0x7fa1901ac090 2021-04-09T18:28:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa1966fb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa190185a00 con 0x7fa190184d30 2021-04-09T18:28:14.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa18ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900b42d0 0x7fa190183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa18ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900b42d0 0x7fa190183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53726/0 (socket says 172.21.15.32:53726) 2021-04-09T18:28:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa18ffff700 1 -- 172.21.15.32:0/1937916768 learned_addr learned my addr 172.21.15.32:0/1937916768 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa194d18700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa190184d30 0x7fa190185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:14.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.535 7fa187fff700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1901ac090 0x7fa1901ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:14.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa18ffff700 1 -- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa190184d30 msgr2=0x7fa190185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:14.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa18ffff700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa190184d30 0x7fa190185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa18ffff700 1 -- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1901ac090 msgr2=0x7fa1901ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:14.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa18ffff700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1901ac090 0x7fa1901ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa18ffff700 1 -- 172.21.15.32:0/1937916768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa190185750 con 0x7fa1900b42d0 2021-04-09T18:28:14.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa187fff700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1901ac090 0x7fa1901ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:14.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa18ffff700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900b42d0 0x7fa190183520 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 rx=0x7fa180006f60 tx=0x7fa18000dba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:14.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa18d7fa700 1 -- 172.21.15.32:0/1937916768 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa18000b030 con 0x7fa1900b42d0 2021-04-09T18:28:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa18d7fa700 1 -- 172.21.15.32:0/1937916768 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa180000c90 con 0x7fa1900b42d0 2021-04-09T18:28:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa190185750 con 0x7fa1900b42d0 2021-04-09T18:28:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa18d7fa700 1 -- 172.21.15.32:0/1937916768 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa180002210 con 0x7fa1900b42d0 2021-04-09T18:28:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.536 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa190185750 con 0x7fa1900b42d0 2021-04-09T18:28:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.537 7fa18d7fa700 1 -- 172.21.15.32:0/1937916768 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa180009250 con 0x7fa1900b42d0 2021-04-09T18:28:14.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.537 7fa18d7fa700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1740188c0 0x7fa17401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:14.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.537 7fa18d7fa700 1 -- 172.21.15.32:0/1937916768 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fa180034c60 con 0x7fa1900b42d0 2021-04-09T18:28:14.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.537 7fa187fff700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1740188c0 0x7fa17401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:14.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.537 7fa187fff700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1740188c0 0x7fa17401adc0 crc :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:14.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.557 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa190182d10 con 0x7fa1900b42d0 2021-04-09T18:28:14.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.560 7fa18d7fa700 1 -- 172.21.15.32:0/1937916768 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa180000c90 con 0x7fa1900b42d0 2021-04-09T18:28:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.750 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa19010d480 con 0x7fa1900b42d0 2021-04-09T18:28:14.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.750 7fa18d7fa700 1 -- 172.21.15.32:0/1937916768 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 289 v289) v1 ==== 95+0+2800 (secure 0 0 0) 0x7fa180009250 con 0x7fa1900b42d0 2021-04-09T18:28:14.829 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 289 2021-04-09T18:28:14.830 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:14.830 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":289,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:28:13.428272","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":7,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":1},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:28:14.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1740188c0 msgr2=0x7fa17401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:14.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1740188c0 0x7fa17401adc0 crc :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900b42d0 msgr2=0x7fa190183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:14.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900b42d0 0x7fa190183520 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 rx=0x7fa180006f60 tx=0x7fa18000dba0).stop 2021-04-09T18:28:14.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 shutdown_connections 2021-04-09T18:28:14.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa1740188c0 0x7fa17401adc0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa190184d30 0x7fa190185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1901ac090 0x7fa1901ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 --2- 172.21.15.32:0/1937916768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1900b42d0 0x7fa190183520 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:14.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 >> 172.21.15.32:0/1937916768 conn(0x7fa19019b890 msgr2=0x7fa1901a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:14.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 shutdown_connections 2021-04-09T18:28:14.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.753 7fa1966fb700 1 -- 172.21.15.32:0/1937916768 wait complete. 2021-04-09T18:28:14.842 DEBUG:tasks.cephfs.filesystem:Looked up MDS state for mds.1: up:active 2021-04-09T18:28:14.842 DEBUG:tasks.cephfs.filesystem:reached state 'up:active' in 3.232353925704956s 2021-04-09T18:28:14.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.911 7f08dd797700 1 Processor -- start 2021-04-09T18:28:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08dd797700 1 -- start start 2021-04-09T18:28:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08dd797700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d80cffa0 0x7f08d80d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08dd797700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08d80d6c80 0x7f08d810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08dd797700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d80d7280 con 0x7f08d80d7800 2021-04-09T18:28:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08dd797700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d80d7080 con 0x7f08d80d6c80 2021-04-09T18:28:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08dd797700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d80d8580 con 0x7f08d80cffa0 2021-04-09T18:28:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08d7d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08d80d6c80 0x7f08d810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08d7d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08d80d6c80 0x7f08d810cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53734/0 (socket says 172.21.15.32:53734) 2021-04-09T18:28:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08d7d75700 1 -- 172.21.15.32:0/1403462746 learned_addr learned my addr 172.21.15.32:0/1403462746 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.912 7f08d7574700 1 --2- 172.21.15.32:0/1403462746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d80cffa0 0x7f08d80d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08d7d75700 1 -- 172.21.15.32:0/1403462746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d80cffa0 msgr2=0x7f08d80d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08d7d75700 1 --2- 172.21.15.32:0/1403462746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d80cffa0 0x7f08d80d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08d7d75700 1 -- 172.21.15.32:0/1403462746 >> v1:172.21.15.86:6789/0 conn(0x7f08d80d7800 legacy=0x7f08d80d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08d7d75700 1 -- 172.21.15.32:0/1403462746 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d80d91e0 con 0x7f08d80d6c80 2021-04-09T18:28:15.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08d7574700 1 --2- 172.21.15.32:0/1403462746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d80cffa0 0x7f08d80d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:15.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08d7d75700 1 --2- 172.21.15.32:0/1403462746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d80d6c80 0x7f08d810cfd0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7f08d0006f00 tx=0x7f08d0009f10).ready entity=mon.0 client_cookie=1e4cd0cd110de4a3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:15.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08d6572700 1 -- 172.21.15.32:0/1403462746 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08d000b3b0 con 0x7f08d80d6c80 2021-04-09T18:28:15.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08d6572700 1 -- 172.21.15.32:0/1403462746 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d000b730 con 0x7f08d80d6c80 2021-04-09T18:28:15.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08d6572700 1 -- 172.21.15.32:0/1403462746 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08d000bc10 con 0x7f08d80d6c80 2021-04-09T18:28:15.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08dd797700 1 -- 172.21.15.32:0/1403462746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d80d6c80 msgr2=0x7f08d810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:15.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.913 7f08dd797700 1 --2- 172.21.15.32:0/1403462746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d80d6c80 0x7f08d810cfd0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7f08d0006f00 tx=0x7f08d0009f10).stop 2021-04-09T18:28:15.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.914 7f08dd797700 1 -- 172.21.15.32:0/1403462746 shutdown_connections 2021-04-09T18:28:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.914 7f08dd797700 1 --2- 172.21.15.32:0/1403462746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d80d6c80 0x7f08d810cfd0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.914 7f08dd797700 1 --2- 172.21.15.32:0/1403462746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d80cffa0 0x7f08d80d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.914 7f08dd797700 1 -- 172.21.15.32:0/1403462746 >> 172.21.15.32:0/1403462746 conn(0x7f08d80ce5c0 msgr2=0x7f08d80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:15.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.914 7f08dd797700 1 -- 172.21.15.32:0/1403462746 shutdown_connections 2021-04-09T18:28:15.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.914 7f08dd797700 1 -- 172.21.15.32:0/1403462746 wait complete. 2021-04-09T18:28:15.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.934 7f08dd797700 1 Processor -- start 2021-04-09T18:28:15.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.934 7f08dd797700 1 -- start start 2021-04-09T18:28:15.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.934 7f08dd797700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d817bbf0 0x7f08d817b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.934 7f08dd797700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d8193110 0x7f08d8193510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.934 7f08dd797700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d817c2f0 0x7f08d817c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08dd797700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d8192c10 con 0x7f08d817bbf0 2021-04-09T18:28:15.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08dd797700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d817c030 con 0x7f08d817c2f0 2021-04-09T18:28:15.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08dd797700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d8193b30 con 0x7f08d8193110 2021-04-09T18:28:15.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d817bbf0 0x7f08d817b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d817bbf0 0x7f08d817b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53742/0 (socket says 172.21.15.32:53742) 2021-04-09T18:28:15.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7574700 1 -- 172.21.15.32:0/98733245 learned_addr learned my addr 172.21.15.32:0/98733245 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:15.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7d75700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d817c2f0 0x7f08d817c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d6d73700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d8193110 0x7f08d8193510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7574700 1 -- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d8193110 msgr2=0x7f08d8193510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:15.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7574700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d8193110 0x7f08d8193510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7574700 1 -- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d817c2f0 msgr2=0x7f08d817c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:15.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7574700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d817c2f0 0x7f08d817c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7574700 1 -- 172.21.15.32:0/98733245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d817cd10 con 0x7f08d817bbf0 2021-04-09T18:28:15.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08d7574700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d817bbf0 0x7f08d817b350 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7f08c800b580 tx=0x7f08c800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:15.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08bffff700 1 -- 172.21.15.32:0/98733245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08c800bd10 con 0x7f08d817bbf0 2021-04-09T18:28:15.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08bffff700 1 -- 172.21.15.32:0/98733245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c8000c90 con 0x7f08d817bbf0 2021-04-09T18:28:15.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08dd797700 1 -- 172.21.15.32:0/98733245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d817cd10 con 0x7f08d817bbf0 2021-04-09T18:28:15.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08bffff700 1 -- 172.21.15.32:0/98733245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08c8002220 con 0x7f08d817bbf0 2021-04-09T18:28:15.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.935 7f08dd797700 1 -- 172.21.15.32:0/98733245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d817cd10 con 0x7f08d817bbf0 2021-04-09T18:28:15.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.936 7f08bffff700 1 -- 172.21.15.32:0/98733245 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f08c800a000 con 0x7f08d817bbf0 2021-04-09T18:28:15.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.936 7f08bffff700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b8018930 0x7f08b801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.936 7f08bffff700 1 -- 172.21.15.32:0/98733245 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f08c8032a30 con 0x7f08d817bbf0 2021-04-09T18:28:15.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.936 7f08d6d73700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b8018930 0x7f08b801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.936 7f08d6d73700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b8018930 0x7f08b801adc0 crc :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:15.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.956 7f08dd797700 1 -- 172.21.15.32:0/98733245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d81b70f0 con 0x7f08d817bbf0 2021-04-09T18:28:15.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:14.960 7f08bffff700 1 -- 172.21.15.32:0/98733245 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f08c8006bb0 con 0x7f08d817bbf0 2021-04-09T18:28:15.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.151 7f08dd797700 1 -- 172.21.15.32:0/98733245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08d81b70f0 con 0x7f08d817bbf0 2021-04-09T18:28:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.151 7f08bffff700 1 -- 172.21.15.32:0/98733245 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 289 v289) v1 ==== 95+0+2800 (secure 0 0 0) 0x7f08c8000c90 con 0x7f08d817bbf0 2021-04-09T18:28:15.231 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 289 2021-04-09T18:28:15.231 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:15.232 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":289,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:28:13.428272","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":7,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":1},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:28:15.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.153 7f08dd797700 1 -- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b8018930 msgr2=0x7f08b801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.153 7f08dd797700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b8018930 0x7f08b801adc0 crc :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.153 7f08dd797700 1 -- 172.21.15.32:0/98733245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d817bbf0 msgr2=0x7f08d817b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.153 7f08dd797700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d817bbf0 0x7f08d817b350 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7f08c800b580 tx=0x7f08c800b920).stop 2021-04-09T18:28:15.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.154 7f08dd797700 1 -- 172.21.15.32:0/98733245 shutdown_connections 2021-04-09T18:28:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.154 7f08dd797700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b8018930 0x7f08b801adc0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.154 7f08dd797700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d817c2f0 0x7f08d817c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.154 7f08dd797700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d8193110 0x7f08d8193510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.154 7f08dd797700 1 --2- 172.21.15.32:0/98733245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d817bbf0 0x7f08d817b350 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.154 7f08dd797700 1 -- 172.21.15.32:0/98733245 >> 172.21.15.32:0/98733245 conn(0x7f08d81905d0 msgr2=0x7f08d81b7320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.154 7f08dd797700 1 -- 172.21.15.32:0/98733245 shutdown_connections 2021-04-09T18:28:15.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.154 7f08dd797700 1 -- 172.21.15.32:0/98733245 wait complete. 2021-04-09T18:28:15.245 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 12:0 false 2021-04-09T18:28:15.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51f2414700 1 Processor -- start 2021-04-09T18:28:15.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51f2414700 1 -- start start 2021-04-09T18:28:15.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51f2414700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f51ec0cffb0 0x7f51ec0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51f2414700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f51ec0d7810 0x7f51ec0d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51f2414700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec0d8230 0x7f51ec0d5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51f2414700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ec0d91f0 con 0x7f51ec0cffb0 2021-04-09T18:28:15.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51f2414700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ec0d7290 con 0x7f51ec0d7810 2021-04-09T18:28:15.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51f2414700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0d62a0 con 0x7f51ec0d8230 2021-04-09T18:28:15.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51ebfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f51ec0cffb0 0x7f51ec0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51ebfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f51ec0cffb0 0x7f51ec0d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53746/0 (socket says 172.21.15.32:53746) 2021-04-09T18:28:15.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.319 7f51ebfff700 1 -- 172.21.15.32:0/1989789481 learned_addr learned my addr 172.21.15.32:0/1989789481 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:15.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51e3fff700 1 --2- 172.21.15.32:0/1989789481 >> v2:172.21.15.86:3300/0 conn(0x7f51ec0d7810 0x7f51ec0d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51f0a31700 1 --2- 172.21.15.32:0/1989789481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec0d8230 0x7f51ec0d5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51ebfff700 1 -- 172.21.15.32:0/1989789481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec0d8230 msgr2=0x7f51ec0d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:15.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51ebfff700 1 --2- 172.21.15.32:0/1989789481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec0d8230 0x7f51ec0d5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51ebfff700 1 -- 172.21.15.32:0/1989789481 >> v2:172.21.15.86:3300/0 conn(0x7f51ec0d7810 msgr2=0x7f51ec0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:15.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51ebfff700 1 --2- 172.21.15.32:0/1989789481 >> v2:172.21.15.86:3300/0 conn(0x7f51ec0d7810 0x7f51ec0d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51ebfff700 1 -- 172.21.15.32:0/1989789481 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec0d6490 con 0x7f51ec0cffb0 2021-04-09T18:28:15.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51e3fff700 1 --2- 172.21.15.32:0/1989789481 >> v2:172.21.15.86:3300/0 conn(0x7f51ec0d7810 0x7f51ec0d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:15.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51ebfff700 1 --2- 172.21.15.32:0/1989789481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0cffb0 0x7f51ec0d2450 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 rx=0x7f51dc006700 tx=0x7f51dc009b00).ready entity=mon.0 client_cookie=887afa23618ed04d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:15.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51eb7fe700 1 -- 172.21.15.32:0/1989789481 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51dc00c0c0 con 0x7f51ec0cffb0 2021-04-09T18:28:15.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51eb7fe700 1 -- 172.21.15.32:0/1989789481 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc00c400 con 0x7f51ec0cffb0 2021-04-09T18:28:15.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51eb7fe700 1 -- 172.21.15.32:0/1989789481 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51dc00c8e0 con 0x7f51ec0cffb0 2021-04-09T18:28:15.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51f2414700 1 -- 172.21.15.32:0/1989789481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0cffb0 msgr2=0x7f51ec0d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:15.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.320 7f51f2414700 1 --2- 172.21.15.32:0/1989789481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0cffb0 0x7f51ec0d2450 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 rx=0x7f51dc006700 tx=0x7f51dc009b00).stop 2021-04-09T18:28:15.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.321 7f51f2414700 1 -- 172.21.15.32:0/1989789481 shutdown_connections 2021-04-09T18:28:15.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.321 7f51f2414700 1 --2- 172.21.15.32:0/1989789481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec0d8230 0x7f51ec0d5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.321 7f51f2414700 1 --2- 172.21.15.32:0/1989789481 >> v2:172.21.15.86:3300/0 conn(0x7f51ec0d7810 0x7f51ec0d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.321 7f51f2414700 1 --2- 172.21.15.32:0/1989789481 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0cffb0 0x7f51ec0d2450 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.321 7f51f2414700 1 -- 172.21.15.32:0/1989789481 >> 172.21.15.32:0/1989789481 conn(0x7f51ec0ce5d0 msgr2=0x7f51ec0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:15.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.321 7f51f2414700 1 -- 172.21.15.32:0/1989789481 shutdown_connections 2021-04-09T18:28:15.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.321 7f51f2414700 1 -- 172.21.15.32:0/1989789481 wait complete. 2021-04-09T18:28:15.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.341 7f51f2414700 1 Processor -- start 2021-04-09T18:28:15.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.341 7f51f2414700 1 -- start start 2021-04-09T18:28:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f2414700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec08ac00 0x7f51ec0b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f2414700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51ec08b000 0x7f51ec1ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f2414700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0a0df0 0x7f51ec1abc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f2414700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec1ab130 con 0x7f51ec0a0df0 2021-04-09T18:28:15.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f2414700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0b43e0 con 0x7f51ec08b000 2021-04-09T18:28:15.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f2414700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0b4600 con 0x7f51ec08ac00 2021-04-09T18:28:15.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f0a31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0a0df0 0x7f51ec1abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51e3fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51ec08b000 0x7f51ec1ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f0a31700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0a0df0 0x7f51ec1abc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53754/0 (socket says 172.21.15.32:53754) 2021-04-09T18:28:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f0a31700 1 -- 172.21.15.32:0/3392239052 learned_addr learned my addr 172.21.15.32:0/3392239052 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:15.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51ebfff700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec08ac00 0x7f51ec0b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f0a31700 1 -- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec08ac00 msgr2=0x7f51ec0b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f0a31700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec08ac00 0x7f51ec0b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f0a31700 1 -- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51ec08b000 msgr2=0x7f51ec1ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:15.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f0a31700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51ec08b000 0x7f51ec1ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:15.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f0a31700 1 -- 172.21.15.32:0/3392239052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec1a9e60 con 0x7f51ec0a0df0 2021-04-09T18:28:15.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51f0a31700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0a0df0 0x7f51ec1abc50 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7f51e400b970 tx=0x7f51e400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:15.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.342 7f51e97fa700 1 -- 172.21.15.32:0/3392239052 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51e4000f90 con 0x7f51ec0a0df0 2021-04-09T18:28:15.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.343 7f51e97fa700 1 -- 172.21.15.32:0/3392239052 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e4006aa0 con 0x7f51ec0a0df0 2021-04-09T18:28:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.343 7f51f2414700 1 -- 172.21.15.32:0/3392239052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ec1a9e60 con 0x7f51ec0a0df0 2021-04-09T18:28:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.343 7f51e97fa700 1 -- 172.21.15.32:0/3392239052 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51e4001470 con 0x7f51ec0a0df0 2021-04-09T18:28:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.343 7f51f2414700 1 -- 172.21.15.32:0/3392239052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ec1a9e60 con 0x7f51ec0a0df0 2021-04-09T18:28:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.343 7f51e97fa700 1 -- 172.21.15.32:0/3392239052 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f51e40011f0 con 0x7f51ec0a0df0 2021-04-09T18:28:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.343 7f51e97fa700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51d0018930 0x7f51d001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.343 7f51e97fa700 1 -- 172.21.15.32:0/3392239052 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f51e4032dc0 con 0x7f51ec0a0df0 2021-04-09T18:28:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.343 7f51ebfff700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51d0018930 0x7f51d001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:15.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.344 7f51ebfff700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51d0018930 0x7f51d001adc0 crc :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:15.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.363 7f51f2414700 1 -- 172.21.15.32:0/3392239052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ec0b4600 con 0x7f51ec0a0df0 2021-04-09T18:28:15.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.367 7f51e97fa700 1 -- 172.21.15.32:0/3392239052 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f51e40066f0 con 0x7f51ec0a0df0 2021-04-09T18:28:15.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:15.561 7f51f2414700 1 -- 172.21.15.32:0/3392239052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "12:0", "val": "false"} v 0) v1 -- 0x7f51ec0b4600 con 0x7f51ec0a0df0 2021-04-09T18:28:16.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.451 7f51e97fa700 1 -- 172.21.15.32:0/3392239052 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "12:0", "val": "false"}]=0 unfreezing mds.9466 v290) v1 ==== 116+0+0 (secure 0 0 0) 0x7f51e4002f20 con 0x7f51ec0a0df0 2021-04-09T18:28:16.556 INFO:teuthology.orchestra.run.smithi032.stderr:unfreezing mds.9466 2021-04-09T18:28:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 -- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51d0018930 msgr2=0x7f51d001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:16.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51d0018930 0x7f51d001adc0 crc :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:16.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 -- 172.21.15.32:0/3392239052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0a0df0 msgr2=0x7f51ec1abc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:16.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0a0df0 0x7f51ec1abc50 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7f51e400b970 tx=0x7f51e400bd10).stop 2021-04-09T18:28:16.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 -- 172.21.15.32:0/3392239052 shutdown_connections 2021-04-09T18:28:16.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51d0018930 0x7f51d001adc0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:16.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51ec0a0df0 0x7f51ec1abc50 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:16.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51ec08b000 0x7f51ec1ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:16.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 --2- 172.21.15.32:0/3392239052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51ec08ac00 0x7f51ec0b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:16.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 -- 172.21.15.32:0/3392239052 >> 172.21.15.32:0/3392239052 conn(0x7f51ec183890 msgr2=0x7f51ec19b900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:16.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 -- 172.21.15.32:0/3392239052 shutdown_connections 2021-04-09T18:28:16.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.479 7f51f2414700 1 -- 172.21.15.32:0/3392239052 wait complete. 2021-04-09T18:28:16.570 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:28:16.571 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:28:16.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:28:16.690 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:28:16.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:16.691 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:28:16.692 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:28:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:28:16.738 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:28:16.738 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:28:16.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:28:16.740 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:28:16.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:16.786 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:28:16.786 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:28:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:28:16.830 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:28:16.830 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:28:16.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:28:16.831 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:28:16.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:28:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.980 7fdb452a5700 1 Processor -- start 2021-04-09T18:28:17.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.980 7fdb452a5700 1 -- start start 2021-04-09T18:28:17.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb452a5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb400d7ca0 0x7fdb400d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb452a5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4010d480 con 0x7fdb4010d010 2021-04-09T18:28:17.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb452a5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb400d86c0 con 0x7fdb4010da00 2021-04-09T18:28:17.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb452a5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb400d9220 con 0x7fdb400d7ca0 2021-04-09T18:28:17.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e87c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fdb400d7ca0 0x7fdb400d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3f07d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdb4010d010 0x7fdb400d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35116/0 (socket says 172.21.15.32:35116) 2021-04-09T18:28:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3f07d700 1 -- 172.21.15.32:0/2601643786 learned_addr learned my addr 172.21.15.32:0/2601643786 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e07b700 1 -- 172.21.15.32:0/2601643786 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647156616 0 0) 0x7fdb38000ce0 con 0x7fdb4010d010 2021-04-09T18:28:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e07b700 1 -- 172.21.15.32:0/2601643786 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb2c002fe0 con 0x7fdb4010d010 2021-04-09T18:28:17.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e07b700 1 -- 172.21.15.32:0/2601643786 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600401255 0 0) 0x7fdb30000d30 con 0x7fdb4010da00 2021-04-09T18:28:17.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e87c700 1 -- 172.21.15.32:0/2601643786 >> v1:172.21.15.86:6789/0 conn(0x7fdb4010da00 legacy=0x7fdb4010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e87c700 1 -- 172.21.15.32:0/2601643786 >> v1:172.21.15.32:6789/0 conn(0x7fdb4010d010 legacy=0x7fdb400d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e87c700 1 -- 172.21.15.32:0/2601643786 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400d1f70 con 0x7fdb400d7ca0 2021-04-09T18:28:17.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e87c700 1 --2- 172.21.15.32:0/2601643786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400d7ca0 0x7fdb400d80a0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 rx=0x7fdb28006650 tx=0x7fdb28009a80).ready entity=mon.0 client_cookie=f00693f03f9e5bb7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:17.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e07b700 1 -- 172.21.15.32:0/2601643786 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb280009b0 con 0x7fdb400d7ca0 2021-04-09T18:28:17.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e07b700 1 -- 172.21.15.32:0/2601643786 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2800c0c0 con 0x7fdb400d7ca0 2021-04-09T18:28:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.981 7fdb3e07b700 1 -- 172.21.15.32:0/2601643786 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb2800c510 con 0x7fdb400d7ca0 2021-04-09T18:28:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.982 7fdb452a5700 1 -- 172.21.15.32:0/2601643786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400d7ca0 msgr2=0x7fdb400d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.982 7fdb452a5700 1 --2- 172.21.15.32:0/2601643786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400d7ca0 0x7fdb400d80a0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 rx=0x7fdb28006650 tx=0x7fdb28009a80).stop 2021-04-09T18:28:17.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.982 7fdb452a5700 1 -- 172.21.15.32:0/2601643786 shutdown_connections 2021-04-09T18:28:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.982 7fdb452a5700 1 --2- 172.21.15.32:0/2601643786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb400d7ca0 0x7fdb400d80a0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.982 7fdb452a5700 1 -- 172.21.15.32:0/2601643786 >> 172.21.15.32:0/2601643786 conn(0x7fdb400ce5f0 msgr2=0x7fdb400d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:17.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.982 7fdb452a5700 1 -- 172.21.15.32:0/2601643786 shutdown_connections 2021-04-09T18:28:17.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:16.982 7fdb452a5700 1 -- 172.21.15.32:0/2601643786 wait complete. 2021-04-09T18:28:17.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.002 7fdb452a5700 1 Processor -- start 2021-04-09T18:28:17.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.002 7fdb452a5700 1 -- start start 2021-04-09T18:28:17.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb452a5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400ab300 0x7fdb401a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb452a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb401a8ad0 0x7fdb401a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb452a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb40180db0 0x7fdb401811b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb452a5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb401817d0 con 0x7fdb40180db0 2021-04-09T18:28:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb452a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb401819f0 con 0x7fdb401a8ad0 2021-04-09T18:28:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb452a5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb401a1f80 con 0x7fdb400ab300 2021-04-09T18:28:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3f87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb40180db0 0x7fdb401811b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3f87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb40180db0 0x7fdb401811b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53768/0 (socket says 172.21.15.32:53768) 2021-04-09T18:28:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3f87e700 1 -- 172.21.15.32:0/193213862 learned_addr learned my addr 172.21.15.32:0/193213862 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3f07d700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400ab300 0x7fdb401a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3e87c700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb401a8ad0 0x7fdb401a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3f87e700 1 -- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400ab300 msgr2=0x7fdb401a12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3f87e700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400ab300 0x7fdb401a12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3f87e700 1 -- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb401a8ad0 msgr2=0x7fdb401a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3f87e700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb401a8ad0 0x7fdb401a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.003 7fdb3f87e700 1 -- 172.21.15.32:0/193213862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb401a21a0 con 0x7fdb40180db0 2021-04-09T18:28:17.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.004 7fdb3f87e700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb40180db0 0x7fdb401811b0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 rx=0x7fdb30009980 tx=0x7fdb3000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.004 7fdb37fff700 1 -- 172.21.15.32:0/193213862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb30005870 con 0x7fdb40180db0 2021-04-09T18:28:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.004 7fdb37fff700 1 -- 172.21.15.32:0/193213862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb30005b70 con 0x7fdb40180db0 2021-04-09T18:28:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.004 7fdb452a5700 1 -- 172.21.15.32:0/193213862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb401a21a0 con 0x7fdb40180db0 2021-04-09T18:28:17.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.004 7fdb37fff700 1 -- 172.21.15.32:0/193213862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb30006030 con 0x7fdb40180db0 2021-04-09T18:28:17.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.004 7fdb452a5700 1 -- 172.21.15.32:0/193213862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb401a21a0 con 0x7fdb40180db0 2021-04-09T18:28:17.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.004 7fdb37fff700 1 -- 172.21.15.32:0/193213862 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdb30006030 con 0x7fdb40180db0 2021-04-09T18:28:17.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.005 7fdb37fff700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb24018930 0x7fdb2401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.005 7fdb37fff700 1 -- 172.21.15.32:0/193213862 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fdb30032e60 con 0x7fdb40180db0 2021-04-09T18:28:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.005 7fdb3f07d700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb24018930 0x7fdb2401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.005 7fdb3f07d700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb24018930 0x7fdb2401ae10 crc :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:17.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.025 7fdb452a5700 1 -- 172.21.15.32:0/193213862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb401b7070 con 0x7fdb40180db0 2021-04-09T18:28:17.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.028 7fdb37fff700 1 -- 172.21.15.32:0/193213862 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdb3000be10 con 0x7fdb40180db0 2021-04-09T18:28:17.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.216 7fdb452a5700 1 -- 172.21.15.32:0/193213862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fdb4010d480 con 0x7fdb40180db0 2021-04-09T18:28:17.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.217 7fdb37fff700 1 -- 172.21.15.32:0/193213862 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v143) v1 ==== 81+0+30516 (secure 0 0 0) 0x7fdb3000be10 con 0x7fdb40180db0 2021-04-09T18:28:17.299 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:17.299 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:28:17.299 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 143, 2021-04-09T18:28:17.300 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:28:17.300 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:28:17.300 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:27:03.171047", 2021-04-09T18:28:17.300 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:28:17.300 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:28:17.301 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:28:17.301 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:28:17.301 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:28:17.302 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:28:17.302 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:28:17.302 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:28:17.302 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:28:17.302 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:28:17.303 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:28:17.303 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:28:17.303 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:28:17.303 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:28:17.303 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:28:17.304 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 25, 2021-04-09T18:28:17.304 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:28:17.304 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:28:17.304 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:28:17.305 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:28:17.305 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:28:17.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.305 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:28:17.305 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:28:17.306 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:28:17.306 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:28:17.306 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:28:17.306 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:28:17.306 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:28:17.307 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:28:17.307 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:28:17.307 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:28:17.307 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:28:17.308 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:28:17.308 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:28:17.308 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:28:17.308 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:28:17.309 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:28:17.309 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:28:17.309 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:28:17.309 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:28:17.309 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:28:17.310 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:28:17.310 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:28:17.310 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:28:17.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.310 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:28:17.311 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:28:17.311 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:28:17.311 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:28:17.311 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:28:17.311 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:28:17.312 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:28:17.312 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:28:17.312 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:28:17.312 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:28:17.313 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:28:17.313 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:28:17.313 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:28:17.313 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:28:17.313 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:28:17.314 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:28:17.314 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:28:17.314 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:28:17.314 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:28:17.315 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:28:17.315 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:28:17.315 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:28:17.315 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:28:17.315 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:28:17.316 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:28:17.316 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:28:17.316 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:28:17.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.316 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:28:17.317 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:28:17.317 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:28:17.317 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:28:17.317 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:28:17.317 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:28:17.318 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:28:17.318 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:28:17.318 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:28:17.318 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:28:17.319 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:28:17.319 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:28:17.319 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:28:17.319 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:28:17.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.320 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.320 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 24, 2021-04-09T18:28:17.320 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:28:17.320 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:26:25.044737", 2021-04-09T18:28:17.321 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:28:17.321 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:28:17.321 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:28:17.321 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:28:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:28:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:28:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:28:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:28:17.322 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:28:17.323 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:28:17.323 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:28:17.323 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:28:17.323 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:28:17.324 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:28:17.324 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:28:17.324 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:28:17.324 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:28:17.324 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:28:17.325 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:28:17.325 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:28:17.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.325 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "141", 2021-04-09T18:28:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:28:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:28:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:28:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:28:17.326 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:28:17.327 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:28:17.327 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:28:17.327 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:28:17.327 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:28:17.327 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:28:17.328 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:28:17.328 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:28:17.328 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:28:17.328 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:28:17.329 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:28:17.329 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:28:17.329 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:28:17.329 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:28:17.329 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:28:17.330 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:28:17.330 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:28:17.330 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:28:17.330 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:28:17.330 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:28:17.331 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:28:17.331 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:28:17.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.331 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:28:17.331 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:28:17.332 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:28:17.332 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:28:17.332 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:28:17.332 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:28:17.333 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:28:17.333 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:28:17.333 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:28:17.333 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:28:17.333 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:28:17.334 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:28:17.334 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:28:17.334 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:28:17.334 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:28:17.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.335 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:28:17.335 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:28:17.335 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:28:17.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.336 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.336 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 25, 2021-04-09T18:28:17.336 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:28:17.337 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:26:26.413903", 2021-04-09T18:28:17.337 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:28:17.337 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:28:17.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:28:17.337 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:28:17.338 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:28:17.338 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:28:17.338 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:28:17.338 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:28:17.338 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:28:17.339 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:28:17.339 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:28:17.339 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:28:17.339 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:28:17.340 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:28:17.340 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:28:17.340 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:28:17.340 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:28:17.340 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:28:17.341 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:28:17.341 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:28:17.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.341 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "142", 2021-04-09T18:28:17.342 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:28:17.342 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:28:17.342 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:28:17.342 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:28:17.343 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:28:17.343 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:28:17.343 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:28:17.343 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:28:17.344 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:28:17.344 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:28:17.344 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:28:17.344 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:28:17.344 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:28:17.345 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:28:17.345 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:28:17.345 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:28:17.345 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:28:17.346 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:28:17.346 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:28:17.346 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:28:17.346 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:28:17.346 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:28:17.347 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:28:17.347 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:28:17.347 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:28:17.347 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:28:17.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.348 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:28:17.348 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:28:17.348 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:28:17.348 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:28:17.348 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:28:17.349 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:28:17.349 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:28:17.349 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:28:17.349 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:28:17.350 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:28:17.350 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:28:17.350 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:28:17.350 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:28:17.350 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:28:17.351 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:28:17.351 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.351 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.351 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.351 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:28:17.352 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:28:17.352 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.352 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:28:17.352 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:28:17.353 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:17.353 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:17.353 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:17.353 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:17.353 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:17.354 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:17.354 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:17.354 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:17.354 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:17.354 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:17.355 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:17.355 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.355 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.355 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.355 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:28:17.356 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:17.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.356 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.357 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:28:17.357 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:17.357 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.357 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.358 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:17.358 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.358 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.358 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.358 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:28:17.359 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:17.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.359 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.359 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.360 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:28:17.360 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:17.360 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.360 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.361 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:17.361 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.361 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.361 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.361 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:28:17.362 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:17.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.362 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.362 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:28:17.363 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:17.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.363 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:28:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:28:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:28:17.367 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:28:17.367 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:28:17.367 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:28:17.367 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:17.367 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:17.368 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:17.368 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.369 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:28:17.369 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:28:17.369 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:17.369 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:17.369 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:17.370 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:17.370 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:17.370 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:17.370 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:17.370 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 136, 2021-04-09T18:28:17.371 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:17.371 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:17.371 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:17.371 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.372 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.372 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:28:17.372 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:17.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.373 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.373 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.373 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:28:17.373 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:17.373 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.374 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.374 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:17.374 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.375 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:28:17.375 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:17.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.376 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.376 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.376 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:28:17.376 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:17.376 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.377 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.377 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:17.377 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.378 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.378 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.378 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:28:17.378 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:17.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.379 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:28:17.379 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:17.379 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.380 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.380 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:17.380 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.381 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.381 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.381 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:28:17.381 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:17.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.382 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.382 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:28:17.382 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:17.382 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.383 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.383 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:28:17.383 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:28:17.383 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:28:17.384 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:28:17.384 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:17.384 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:17.384 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:17.385 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.385 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:28:17.385 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:28:17.386 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:17.386 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:17.386 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:17.386 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:17.386 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:17.387 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:17.387 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:17.387 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:17.387 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:17.387 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:17.388 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:17.388 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.388 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.389 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:28:17.389 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:17.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.389 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.390 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:28:17.390 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:17.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.390 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.391 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:17.391 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.391 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.391 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.391 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:28:17.392 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:17.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.393 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:28:17.393 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:17.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.393 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.393 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.394 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:17.394 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.394 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.394 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:28:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:28:17.396 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:17.396 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.396 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.396 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:17.397 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.397 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:28:17.397 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:17.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.398 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.398 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.398 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:28:17.399 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:17.399 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.399 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.399 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:28:17.400 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:28:17.400 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:28:17.400 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:28:17.400 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:17.400 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:17.401 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:17.401 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.401 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.402 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:28:17.402 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:28:17.402 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:17.402 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:17.402 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:17.403 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:17.403 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:17.403 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:17.403 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:17.404 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:17.404 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:17.404 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:17.404 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:17.404 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.405 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.405 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:28:17.405 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:17.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.406 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:28:17.406 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:17.406 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.407 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.407 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:17.407 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.407 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.408 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.408 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:28:17.408 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:17.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.409 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.409 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:28:17.409 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:17.409 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.410 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.410 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:17.410 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.411 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.411 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:28:17.411 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:17.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.411 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.412 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.412 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:28:17.412 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:17.412 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.412 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.413 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:17.413 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.414 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:28:17.414 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:17.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.415 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.415 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:28:17.415 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:17.415 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.415 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.416 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:28:17.416 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:28:17.416 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:28:17.416 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:28:17.417 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:17.417 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:17.417 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:17.417 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.418 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:28:17.418 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:28:17.418 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:17.419 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:17.419 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:17.419 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:17.419 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:17.419 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:17.420 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:17.420 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:28:17.420 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:17.420 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:17.421 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:17.421 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.421 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.421 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.421 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:28:17.422 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:17.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.422 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.422 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:28:17.423 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:17.423 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.423 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.424 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:17.424 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.424 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.424 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:28:17.425 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:17.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.425 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.425 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.425 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:28:17.426 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:17.426 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.426 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.427 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:17.427 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.427 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.427 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.427 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:28:17.428 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:17.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.428 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.428 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.428 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:28:17.429 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:17.429 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.429 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.430 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:17.430 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.430 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:28:17.431 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:17.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.431 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:28:17.432 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:17.432 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.432 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.432 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:28:17.433 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:28:17.433 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:28:17.433 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:28:17.433 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.435 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:28:17.435 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:28:17.435 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:17.435 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:17.435 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:17.436 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:17.436 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:17.436 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:17.436 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:17.437 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 136, 2021-04-09T18:28:17.437 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:17.437 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:17.437 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:17.437 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.438 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.438 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.438 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:28:17.438 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:17.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.439 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.439 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:28:17.439 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:17.440 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.440 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.440 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:17.440 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.441 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.441 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.441 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:28:17.441 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:17.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.442 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:28:17.442 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:17.442 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.443 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.443 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:17.443 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.444 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.444 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:28:17.444 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:17.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.445 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.445 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:28:17.445 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:17.445 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.446 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.446 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:17.446 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.447 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:28:17.447 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:17.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.448 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.448 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:28:17.448 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:17.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.449 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.449 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:28:17.449 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:28:17.449 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:28:17.450 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:28:17.450 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:17.450 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:17.450 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:17.450 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.451 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:28:17.451 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:28:17.451 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:17.452 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:17.452 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:17.452 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:17.452 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:17.453 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:17.453 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:17.453 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:17.453 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:17.453 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:17.454 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:17.454 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.454 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:28:17.455 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:17.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.455 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.455 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:28:17.456 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:17.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.456 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.457 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:17.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.457 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.457 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:28:17.458 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:17.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.458 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.458 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.458 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:28:17.459 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:17.459 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.459 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.460 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:17.460 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.460 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.460 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.460 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:28:17.461 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:17.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.461 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.462 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:28:17.462 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:17.462 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.462 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.464 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:28:17.464 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:17.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.464 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.465 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:28:17.465 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:17.465 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.465 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.466 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:28:17.466 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:28:17.466 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:28:17.466 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:28:17.466 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:17.467 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:17.467 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:17.467 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.468 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.468 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:28:17.468 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:28:17.468 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:17.468 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:17.469 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:17.469 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:17.469 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:17.469 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:17.469 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:17.470 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:17.470 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:17.470 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:17.470 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:17.471 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.471 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:28:17.471 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:28:17.472 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.473 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:28:17.474 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:28:17.475 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.476 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:28:17.477 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:28:17.478 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:17.479 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:28:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:17.480 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.481 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:17.481 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:28:17.481 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:17.482 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.482 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.482 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:28:17.482 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:28:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:28:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:28:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:17.483 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:28:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:28:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:28:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:28:17.486 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:17.487 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:28:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:17.488 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.489 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:28:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:17.490 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:28:17.491 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:17.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:28:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:17.493 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:28:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:17.495 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:28:17.496 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:17.497 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:28:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:28:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:28:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:28:17.498 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:28:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:28:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/820321060": "2021-04-10 18:27:03.171015", 2021-04-09T18:28:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/820321060": "2021-04-10 18:27:03.171015" 2021-04-09T18:28:17.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:28:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:28:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:28:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:28:17.500 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:28:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:28:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:28:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:28:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:17.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:28:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:28:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:28:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:28:17.502 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:28:17.503 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:28:17.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 -- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb24018930 msgr2=0x7fdb2401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb24018930 0x7fdb2401ae10 crc :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 -- 172.21.15.32:0/193213862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb40180db0 msgr2=0x7fdb401811b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb40180db0 0x7fdb401811b0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 rx=0x7fdb30009980 tx=0x7fdb3000bcd0).stop 2021-04-09T18:28:17.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 -- 172.21.15.32:0/193213862 shutdown_connections 2021-04-09T18:28:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb24018930 0x7fdb2401ae10 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb40180db0 0x7fdb401811b0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb401a8ad0 0x7fdb401a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 --2- 172.21.15.32:0/193213862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb400ab300 0x7fdb401a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.221 7fdb452a5700 1 -- 172.21.15.32:0/193213862 >> 172.21.15.32:0/193213862 conn(0x7fdb401aa2b0 msgr2=0x7fdb401a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.222 7fdb452a5700 1 -- 172.21.15.32:0/193213862 shutdown_connections 2021-04-09T18:28:17.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.222 7fdb452a5700 1 -- 172.21.15.32:0/193213862 wait complete. 2021-04-09T18:28:17.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:17.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.576 7fb4775bb700 1 Processor -- start 2021-04-09T18:28:17.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.576 7fb4775bb700 1 -- start start 2021-04-09T18:28:17.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb4775bb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb47010d010 0x7fb4700d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb4775bb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb4700d1da0 0x7fb4700d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb4775bb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb4700d82c0 0x7fb47010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb4775bb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47010e780 con 0x7fb4700d82c0 2021-04-09T18:28:17.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb4775bb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4700d8d30 con 0x7fb4700d1da0 2021-04-09T18:28:17.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb4775bb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4700d9220 con 0x7fb47010d010 2021-04-09T18:28:17.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb475bd8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb4700d82c0 0x7fb47010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb475bd8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb4700d82c0 0x7fb47010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53774/0 (socket says 172.21.15.32:53774) 2021-04-09T18:28:17.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb475bd8700 1 -- 172.21.15.32:0/725762129 learned_addr learned my addr 172.21.15.32:0/725762129 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:17.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb474bd6700 1 --2- 172.21.15.32:0/725762129 >> v2:172.21.15.86:3300/0 conn(0x7fb4700d1da0 0x7fb4700d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb4753d7700 1 --2- 172.21.15.32:0/725762129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb47010d010 0x7fb4700d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb475bd8700 1 -- 172.21.15.32:0/725762129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb47010d010 msgr2=0x7fb4700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:17.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb475bd8700 1 --2- 172.21.15.32:0/725762129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb47010d010 0x7fb4700d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb475bd8700 1 -- 172.21.15.32:0/725762129 >> v2:172.21.15.86:3300/0 conn(0x7fb4700d1da0 msgr2=0x7fb4700d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:17.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb475bd8700 1 --2- 172.21.15.32:0/725762129 >> v2:172.21.15.86:3300/0 conn(0x7fb4700d1da0 0x7fb4700d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb475bd8700 1 -- 172.21.15.32:0/725762129 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47010d480 con 0x7fb4700d82c0 2021-04-09T18:28:17.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb474bd6700 1 --2- 172.21.15.32:0/725762129 >> v2:172.21.15.86:3300/0 conn(0x7fb4700d1da0 0x7fb4700d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:17.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.577 7fb475bd8700 1 --2- 172.21.15.32:0/725762129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4700d82c0 0x7fb47010da00 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7fb468006b80 tx=0x7fb468009b90).ready entity=mon.0 client_cookie=865b7132215da3d0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:17.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb467fff700 1 -- 172.21.15.32:0/725762129 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4680022e0 con 0x7fb4700d82c0 2021-04-09T18:28:17.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb467fff700 1 -- 172.21.15.32:0/725762129 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb468002620 con 0x7fb4700d82c0 2021-04-09T18:28:17.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb467fff700 1 -- 172.21.15.32:0/725762129 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb468002b00 con 0x7fb4700d82c0 2021-04-09T18:28:17.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb4775bb700 1 -- 172.21.15.32:0/725762129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4700d82c0 msgr2=0x7fb47010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb4775bb700 1 --2- 172.21.15.32:0/725762129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4700d82c0 0x7fb47010da00 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7fb468006b80 tx=0x7fb468009b90).stop 2021-04-09T18:28:17.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb4775bb700 1 -- 172.21.15.32:0/725762129 shutdown_connections 2021-04-09T18:28:17.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb4775bb700 1 --2- 172.21.15.32:0/725762129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4700d82c0 0x7fb47010da00 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb4775bb700 1 --2- 172.21.15.32:0/725762129 >> v2:172.21.15.86:3300/0 conn(0x7fb4700d1da0 0x7fb4700d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb4775bb700 1 --2- 172.21.15.32:0/725762129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb47010d010 0x7fb4700d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.578 7fb4775bb700 1 -- 172.21.15.32:0/725762129 >> 172.21.15.32:0/725762129 conn(0x7fb4700ce5f0 msgr2=0x7fb4700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:17.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.579 7fb4775bb700 1 -- 172.21.15.32:0/725762129 shutdown_connections 2021-04-09T18:28:17.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.579 7fb4775bb700 1 -- 172.21.15.32:0/725762129 wait complete. 2021-04-09T18:28:17.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb4775bb700 1 Processor -- start 2021-04-09T18:28:17.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb4775bb700 1 -- start start 2021-04-09T18:28:17.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb4775bb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4701a2e80 0x7fb47008d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb4775bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb47008dfc0 0x7fb470090920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb4775bb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb470090f40 0x7fb47008c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb4775bb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb470096ce0 con 0x7fb47008dfc0 2021-04-09T18:28:17.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb4775bb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb470091340 con 0x7fb4701a2e80 2021-04-09T18:28:17.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb4775bb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb470091560 con 0x7fb470090f40 2021-04-09T18:28:17.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb474bd6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb47008dfc0 0x7fb470090920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb474bd6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb47008dfc0 0x7fb470090920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53782/0 (socket says 172.21.15.32:53782) 2021-04-09T18:28:17.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.599 7fb474bd6700 1 -- 172.21.15.32:0/2165393074 learned_addr learned my addr 172.21.15.32:0/2165393074 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb475bd8700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb470090f40 0x7fb47008c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb4753d7700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4701a2e80 0x7fb47008d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb475bd8700 1 -- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4701a2e80 msgr2=0x7fb47008d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb475bd8700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4701a2e80 0x7fb47008d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb475bd8700 1 -- 172.21.15.32:0/2165393074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb47008dfc0 msgr2=0x7fb470090920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb475bd8700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb47008dfc0 0x7fb470090920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb475bd8700 1 -- 172.21.15.32:0/2165393074 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47008ce80 con 0x7fb470090f40 2021-04-09T18:28:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb474bd6700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb47008dfc0 0x7fb470090920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb4753d7700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4701a2e80 0x7fb47008d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb475bd8700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb470090f40 0x7fb47008c860 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb468006f30 tx=0x7fb46800b890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb465ffb700 1 -- 172.21.15.32:0/2165393074 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb468000940 con 0x7fb470090f40 2021-04-09T18:28:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb465ffb700 1 -- 172.21.15.32:0/2165393074 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb468000c40 con 0x7fb470090f40 2021-04-09T18:28:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47008ce80 con 0x7fb470090f40 2021-04-09T18:28:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb465ffb700 1 -- 172.21.15.32:0/2165393074 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb468002470 con 0x7fb470090f40 2021-04-09T18:28:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.600 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47008ce80 con 0x7fb470090f40 2021-04-09T18:28:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.601 7fb465ffb700 1 -- 172.21.15.32:0/2165393074 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb468002020 con 0x7fb470090f40 2021-04-09T18:28:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.601 7fb465ffb700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb450018930 0x7fb45001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.601 7fb465ffb700 1 -- 172.21.15.32:0/2165393074 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fb468032980 con 0x7fb470090f40 2021-04-09T18:28:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.601 7fb4753d7700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb450018930 0x7fb45001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:17.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.601 7fb4753d7700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb450018930 0x7fb45001ae10 crc :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:17.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.621 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4700f1820 con 0x7fb470090f40 2021-04-09T18:28:17.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.624 7fb465ffb700 1 -- 172.21.15.32:0/2165393074 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb468000940 con 0x7fb470090f40 2021-04-09T18:28:17.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.815 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb47010d480 con 0x7fb470090f40 2021-04-09T18:28:17.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.816 7fb465ffb700 1 -- 172.21.15.32:0/2165393074 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) v1 ==== 95+0+2800 (secure 0 0 0) 0x7fb468000ef0 con 0x7fb470090f40 2021-04-09T18:28:17.896 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 290 2021-04-09T18:28:17.896 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:17.896 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":290,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":290,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:28:16.445159","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":143,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9466,"mds_1":9765},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9466":{"gid":9466,"name":"d","rank":0,"incarnation":273,"state":"up:active","state_seq":7,"addr":"172.21.15.86:6836/630282836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":630282836},{"type":"v1","addr":"172.21.15.86:6836","nonce":630282836}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_9765":{"gid":9765,"name":"c","rank":1,"incarnation":283,"state":"up:active","state_seq":31,"addr":"172.21.15.32:6835/1363525210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1363525210},{"type":"v1","addr":"172.21.15.32:6835","nonce":1363525210}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:28:17.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb450018930 msgr2=0x7fb45001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb450018930 0x7fb45001ae10 crc :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb470090f40 msgr2=0x7fb47008c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:17.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb470090f40 0x7fb47008c860 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb468006f30 tx=0x7fb46800b890).stop 2021-04-09T18:28:17.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 shutdown_connections 2021-04-09T18:28:17.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb450018930 0x7fb45001ae10 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb47008dfc0 0x7fb470090920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb470090f40 0x7fb47008c860 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 --2- 172.21.15.32:0/2165393074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4701a2e80 0x7fb47008d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:17.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 >> 172.21.15.32:0/2165393074 conn(0x7fb470192580 msgr2=0x7fb4700f1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:17.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 shutdown_connections 2021-04-09T18:28:17.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.818 7fb4775bb700 1 -- 172.21.15.32:0/2165393074 wait complete. 2021-04-09T18:28:17.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:28:18.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.980 7f8901c83700 1 Processor -- start 2021-04-09T18:28:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.980 7f8901c83700 1 -- start start 2021-04-09T18:28:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f8901c83700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88fc0cffb0 0x7f88fc0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f8901c83700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88fc0d7810 0x7f88fc0d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f8901c83700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc0d7290 con 0x7f88fc0d8230 2021-04-09T18:28:18.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f8901c83700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88fc0d91f0 con 0x7f88fc0cffb0 2021-04-09T18:28:18.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f8901c83700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88fc10dff0 con 0x7f88fc0d7810 2021-04-09T18:28:18.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fb7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88fc0cffb0 0x7f88fc0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fb7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88fc0cffb0 0x7f88fc0d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53786/0 (socket says 172.21.15.32:53786) 2021-04-09T18:28:18.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fb7fe700 1 -- 172.21.15.32:0/2691314368 learned_addr learned my addr 172.21.15.32:0/2691314368 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:18.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88faffd700 1 --2- 172.21.15.32:0/2691314368 >> v2:172.21.15.86:3300/0 conn(0x7f88fc0d7810 0x7f88fc0d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fb7fe700 1 -- 172.21.15.32:0/2691314368 >> v2:172.21.15.86:3300/0 conn(0x7f88fc0d7810 msgr2=0x7f88fc0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fb7fe700 1 --2- 172.21.15.32:0/2691314368 >> v2:172.21.15.86:3300/0 conn(0x7f88fc0d7810 0x7f88fc0d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fb7fe700 1 -- 172.21.15.32:0/2691314368 >> v1:172.21.15.86:6789/0 conn(0x7f88fc0d8230 legacy=0x7f88fc0d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fb7fe700 1 -- 172.21.15.32:0/2691314368 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc10e210 con 0x7f88fc0cffb0 2021-04-09T18:28:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fb7fe700 1 --2- 172.21.15.32:0/2691314368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc0cffb0 0x7f88fc0d2450 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 rx=0x7f88ec006700 tx=0x7f88ec009b80).ready entity=mon.0 client_cookie=63ac8f481d34f14c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fa7fc700 1 -- 172.21.15.32:0/2691314368 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88ec00c0c0 con 0x7f88fc0cffb0 2021-04-09T18:28:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fa7fc700 1 -- 172.21.15.32:0/2691314368 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec00c440 con 0x7f88fc0cffb0 2021-04-09T18:28:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.981 7f88fa7fc700 1 -- 172.21.15.32:0/2691314368 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88ec00c920 con 0x7f88fc0cffb0 2021-04-09T18:28:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.982 7f8901c83700 1 -- 172.21.15.32:0/2691314368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc0cffb0 msgr2=0x7f88fc0d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.982 7f8901c83700 1 --2- 172.21.15.32:0/2691314368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc0cffb0 0x7f88fc0d2450 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 rx=0x7f88ec006700 tx=0x7f88ec009b80).stop 2021-04-09T18:28:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.982 7f8901c83700 1 -- 172.21.15.32:0/2691314368 shutdown_connections 2021-04-09T18:28:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.982 7f8901c83700 1 --2- 172.21.15.32:0/2691314368 >> v2:172.21.15.86:3300/0 conn(0x7f88fc0d7810 0x7f88fc0d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.982 7f8901c83700 1 --2- 172.21.15.32:0/2691314368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc0cffb0 0x7f88fc0d2450 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.982 7f8901c83700 1 -- 172.21.15.32:0/2691314368 >> 172.21.15.32:0/2691314368 conn(0x7f88fc0ce5d0 msgr2=0x7f88fc0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.982 7f8901c83700 1 -- 172.21.15.32:0/2691314368 shutdown_connections 2021-04-09T18:28:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:17.982 7f8901c83700 1 -- 172.21.15.32:0/2691314368 wait complete. 2021-04-09T18:28:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.002 7f8901c83700 1 Processor -- start 2021-04-09T18:28:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f8901c83700 1 -- start start 2021-04-09T18:28:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f8901c83700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88fc0b7660 0x7f88fc0a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f8901c83700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc1ac3b0 0x7f88fc1ac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f8901c83700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88fc1acdd0 0x7f88fc09ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f8901c83700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc09f400 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f8901c83700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc09f620 con 0x7f88fc1acdd0 2021-04-09T18:28:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f8901c83700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc09f840 con 0x7f88fc0b7660 2021-04-09T18:28:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f88faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc1ac3b0 0x7f88fc1ac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f88faffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc1ac3b0 0x7f88fc1ac7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53792/0 (socket says 172.21.15.32:53792) 2021-04-09T18:28:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f88faffd700 1 -- 172.21.15.32:0/1115329542 learned_addr learned my addr 172.21.15.32:0/1115329542 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f88fb7fe700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88fc0b7660 0x7f88fc0a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.003 7f88fbfff700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88fc1acdd0 0x7f88fc09ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f88faffd700 1 -- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88fc0b7660 msgr2=0x7f88fc0a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f88faffd700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88fc0b7660 0x7f88fc0a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f88faffd700 1 -- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88fc1acdd0 msgr2=0x7f88fc09ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f88faffd700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88fc1acdd0 0x7f88fc09ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f88faffd700 1 -- 172.21.15.32:0/1115329542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc09fa60 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f88faffd700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc1ac3b0 0x7f88fc1ac7b0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 rx=0x7f88f0006f60 tx=0x7f88f000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f88e3fff700 1 -- 172.21.15.32:0/1115329542 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88f000bd60 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f88e3fff700 1 -- 172.21.15.32:0/1115329542 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0000c90 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f8901c83700 1 -- 172.21.15.32:0/1115329542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc09fa60 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f88e3fff700 1 -- 172.21.15.32:0/1115329542 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88f0002220 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.004 7f8901c83700 1 -- 172.21.15.32:0/1115329542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc09fa60 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.005 7f88e3fff700 1 -- 172.21.15.32:0/1115329542 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f88f00052e0 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.005 7f88e3fff700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88dc018930 0x7f88dc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.005 7f88e3fff700 1 -- 172.21.15.32:0/1115329542 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f88f0032a40 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.005 7f88fb7fe700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88dc018930 0x7f88dc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.005 7f88fb7fe700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88dc018930 0x7f88dc01ae10 crc :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:18.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.025 7f8901c83700 1 -- 172.21.15.32:0/1115329542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88fc182e10 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.028 7f88e3fff700 1 -- 172.21.15.32:0/1115329542 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f88f0000c90 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.209 7f8901c83700 1 -- 172.21.15.32:0/1115329542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f88fc182e10 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.531 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:28:18.451 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:28:18.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.458 7f88e3fff700 1 -- 172.21.15.32:0/1115329542 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v291) v1 ==== 161+0+0 (secure 0 0 0) 0x7f88f00052e0 con 0x7f88fc1ac3b0 2021-04-09T18:28:18.553 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:28:18.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 -- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88dc018930 msgr2=0x7f88dc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88dc018930 0x7f88dc01ae10 crc :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 -- 172.21.15.32:0/1115329542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc1ac3b0 msgr2=0x7f88fc1ac7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc1ac3b0 0x7f88fc1ac7b0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 rx=0x7f88f0006f60 tx=0x7f88f000b970).stop 2021-04-09T18:28:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 -- 172.21.15.32:0/1115329542 shutdown_connections 2021-04-09T18:28:18.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88dc018930 0x7f88dc01ae10 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88fc1ac3b0 0x7f88fc1ac7b0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88fc1acdd0 0x7f88fc09ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 --2- 172.21.15.32:0/1115329542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88fc0b7660 0x7f88fc0a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 -- 172.21.15.32:0/1115329542 >> 172.21.15.32:0/1115329542 conn(0x7f88fc183890 msgr2=0x7f88fc19b900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 -- 172.21.15.32:0/1115329542 shutdown_connections 2021-04-09T18:28:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.476 7f8901c83700 1 -- 172.21.15.32:0/1115329542 wait complete. 2021-04-09T18:28:18.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:18.595 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:28:18.597 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:28:18.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.637 7f9e7b9ab700 1 Processor -- start 2021-04-09T18:28:18.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.637 7f9e7b9ab700 1 -- start start 2021-04-09T18:28:18.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.637 7f9e7b9ab700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e7410da00 0x7f9e7410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.637 7f9e7b9ab700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7410d480 con 0x7f9e7410d010 2021-04-09T18:28:18.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.637 7f9e7b9ab700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740d1f70 con 0x7f9e740d7ca0 2021-04-09T18:28:18.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.637 7f9e7b9ab700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e740d8d30 con 0x7f9e7410da00 2021-04-09T18:28:18.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e79fc8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e7410da00 0x7f9e7410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e797c7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9e7410d010 0x7f9e740d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35170/0 (socket says 172.21.15.32:35170) 2021-04-09T18:28:18.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e79fc8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9e7410da00 0x7f9e7410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53816/0 (socket says 172.21.15.32:53816) 2021-04-09T18:28:18.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e797c7700 1 -- 172.21.15.32:0/4245658374 learned_addr learned my addr 172.21.15.32:0/4245658374 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:18.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e79fc8700 1 -- 172.21.15.32:0/4245658374 >> v1:172.21.15.86:6789/0 conn(0x7f9e740d7ca0 legacy=0x7f9e740d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e79fc8700 1 -- 172.21.15.32:0/4245658374 >> v1:172.21.15.32:6789/0 conn(0x7f9e7410d010 legacy=0x7f9e740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e6bfff700 1 -- 172.21.15.32:0/4245658374 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927106096 0 0) 0x7f9e70000ce0 con 0x7f9e7410d010 2021-04-09T18:28:18.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e79fc8700 1 -- 172.21.15.32:0/4245658374 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740d9220 con 0x7f9e7410da00 2021-04-09T18:28:18.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e79fc8700 1 --2- 172.21.15.32:0/4245658374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e7410da00 0x7f9e7410de00 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f9e6c006e30 tx=0x7f9e6c009de0).ready entity=mon.0 client_cookie=184672a89109b8c2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e6bfff700 1 -- 172.21.15.32:0/4245658374 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e6c00b300 con 0x7f9e7410da00 2021-04-09T18:28:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e6bfff700 1 -- 172.21.15.32:0/4245658374 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c00b680 con 0x7f9e7410da00 2021-04-09T18:28:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e6bfff700 1 -- 172.21.15.32:0/4245658374 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e6c00bb60 con 0x7f9e7410da00 2021-04-09T18:28:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e7b9ab700 1 -- 172.21.15.32:0/4245658374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e7410da00 msgr2=0x7f9e7410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.638 7f9e7b9ab700 1 --2- 172.21.15.32:0/4245658374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e7410da00 0x7f9e7410de00 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f9e6c006e30 tx=0x7f9e6c009de0).stop 2021-04-09T18:28:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.639 7f9e7b9ab700 1 -- 172.21.15.32:0/4245658374 shutdown_connections 2021-04-09T18:28:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.639 7f9e7b9ab700 1 --2- 172.21.15.32:0/4245658374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e7410da00 0x7f9e7410de00 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.639 7f9e7b9ab700 1 -- 172.21.15.32:0/4245658374 >> 172.21.15.32:0/4245658374 conn(0x7f9e740ce5f0 msgr2=0x7f9e740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:18.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.639 7f9e7b9ab700 1 -- 172.21.15.32:0/4245658374 shutdown_connections 2021-04-09T18:28:18.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.639 7f9e7b9ab700 1 -- 172.21.15.32:0/4245658374 wait complete. 2021-04-09T18:28:18.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.659 7f9e7b9ab700 1 Processor -- start 2021-04-09T18:28:18.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.659 7f9e7b9ab700 1 -- start start 2021-04-09T18:28:18.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e7b9ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e741a9a50 0x7f9e7408a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e7b9ab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7408afb0 0x7f9e7409d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e7b9ab700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7409ddf0 0x7f9e74183b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e7b9ab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e741a9e50 con 0x7f9e741a9a50 2021-04-09T18:28:18.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e7b9ab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7409e410 con 0x7f9e7408afb0 2021-04-09T18:28:18.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e7b9ab700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e741841a0 con 0x7f9e7409ddf0 2021-04-09T18:28:18.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e797c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e741a9a50 0x7f9e7408a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e797c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e741a9a50 0x7f9e7408a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53818/0 (socket says 172.21.15.32:53818) 2021-04-09T18:28:18.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e797c7700 1 -- 172.21.15.32:0/2849868924 learned_addr learned my addr 172.21.15.32:0/2849868924 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:18.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e79fc8700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7409ddf0 0x7f9e74183b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e78fc6700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7408afb0 0x7f9e7409d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e797c7700 1 -- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7409ddf0 msgr2=0x7f9e74183b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e797c7700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7409ddf0 0x7f9e74183b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e797c7700 1 -- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7408afb0 msgr2=0x7f9e7409d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e797c7700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7408afb0 0x7f9e7409d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e797c7700 1 -- 172.21.15.32:0/2849868924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e74184380 con 0x7f9e741a9a50 2021-04-09T18:28:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.660 7f9e797c7700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e741a9a50 0x7f9e7408a990 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 rx=0x7f9e7000b530 tx=0x7f9e7000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.661 7f9e69ffb700 1 -- 172.21.15.32:0/2849868924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e70006020 con 0x7f9e741a9a50 2021-04-09T18:28:18.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.661 7f9e69ffb700 1 -- 172.21.15.32:0/2849868924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e700063a0 con 0x7f9e741a9a50 2021-04-09T18:28:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.661 7f9e69ffb700 1 -- 172.21.15.32:0/2849868924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e70004590 con 0x7f9e741a9a50 2021-04-09T18:28:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.661 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e74184380 con 0x7f9e741a9a50 2021-04-09T18:28:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.661 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e74184380 con 0x7f9e741a9a50 2021-04-09T18:28:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.661 7f9e69ffb700 1 -- 172.21.15.32:0/2849868924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9e7000a000 con 0x7f9e741a9a50 2021-04-09T18:28:18.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.661 7f9e69ffb700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e54018930 0x7f9e5401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.662 7f9e69ffb700 1 -- 172.21.15.32:0/2849868924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f9e70032ad0 con 0x7f9e741a9a50 2021-04-09T18:28:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.662 7f9e78fc6700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e54018930 0x7f9e5401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:18.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.662 7f9e78fc6700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e54018930 0x7f9e5401adc0 crc :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:18.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.682 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7410d480 con 0x7f9e741a9a50 2021-04-09T18:28:18.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.685 7f9e69ffb700 1 -- 172.21.15.32:0/2849868924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9e70006b90 con 0x7f9e741a9a50 2021-04-09T18:28:18.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.875 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e7410d480 con 0x7f9e741a9a50 2021-04-09T18:28:18.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.876 7f9e69ffb700 1 -- 172.21.15.32:0/2849868924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 291 v291) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f9e7000a000 con 0x7f9e741a9a50 2021-04-09T18:28:18.956 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 291 2021-04-09T18:28:18.956 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:18.957 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":291,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272}],"filesystems":[{"mdsmap":{"epoch":291,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:26:27.651041","modified":"2021-04-09 18:28:18.452629","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":144,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-04-09T18:28:18.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e54018930 msgr2=0x7f9e5401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e54018930 0x7f9e5401adc0 crc :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e741a9a50 msgr2=0x7f9e7408a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:18.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e741a9a50 0x7f9e7408a990 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 rx=0x7f9e7000b530 tx=0x7f9e7000b8d0).stop 2021-04-09T18:28:18.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 shutdown_connections 2021-04-09T18:28:18.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9e54018930 0x7f9e5401adc0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7409ddf0 0x7f9e74183b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7408afb0 0x7f9e7409d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 --2- 172.21.15.32:0/2849868924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e741a9a50 0x7f9e7408a990 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:18.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 >> 172.21.15.32:0/2849868924 conn(0x7f9e741965c0 msgr2=0x7f9e741ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:18.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 shutdown_connections 2021-04-09T18:28:18.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:18.878 7f9e7b9ab700 1 -- 172.21.15.32:0/2849868924 wait complete. 2021-04-09T18:28:18.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:28:19.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.080 7f56b8db1700 1 Processor -- start 2021-04-09T18:28:19.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.080 7f56b8db1700 1 -- start start 2021-04-09T18:28:19.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.080 7f56b8db1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b40cffa0 0x7f56b40d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:19.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.080 7f56b8db1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f56b40d7800 0x7f56b40d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:19.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.080 7f56b8db1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b40d7280 con 0x7f56b40d5c70 2021-04-09T18:28:19.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.080 7f56b8db1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40d8580 con 0x7f56b40d7800 2021-04-09T18:28:19.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.080 7f56b8db1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40d91e0 con 0x7f56b40cffa0 2021-04-09T18:28:19.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b2b13700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b40cffa0 0x7f56b40d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:19.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b2312700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f56b40d7800 0x7f56b40d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:19.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b3314700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f56b40d5c70 0x7f56b40d6070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:33112/0 (socket says 172.21.15.32:33112) 2021-04-09T18:28:19.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b3314700 1 -- 172.21.15.32:0/2347418453 learned_addr learned my addr 172.21.15.32:0/2347418453 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:19.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b2312700 1 -- 172.21.15.32:0/2347418453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b40cffa0 msgr2=0x7f56b40d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:19.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b2312700 1 --2- 172.21.15.32:0/2347418453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b40cffa0 0x7f56b40d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:19.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b2312700 1 -- 172.21.15.32:0/2347418453 >> v1:172.21.15.86:6789/0 conn(0x7f56b40d5c70 legacy=0x7f56b40d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:19.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b2312700 1 -- 172.21.15.32:0/2347418453 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b40d8220 con 0x7f56b40d7800 2021-04-09T18:28:19.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b1b11700 1 -- 172.21.15.32:0/2347418453 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646726750 0 0) 0x7f56ac001440 con 0x7f56b40d5c70 2021-04-09T18:28:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b2312700 1 --2- 172.21.15.32:0/2347418453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56b40d7800 0x7f56b40d7c00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f56a8006650 tx=0x7f56a8009a80).ready entity=mon.1 client_cookie=61b70805e804ce90 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:19.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.081 7f56b1b11700 1 -- 172.21.15.32:0/2347418453 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a80009b0 con 0x7f56b40d7800 2021-04-09T18:28:19.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b1b11700 1 -- 172.21.15.32:0/2347418453 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a800c0c0 con 0x7f56b40d7800 2021-04-09T18:28:19.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b1b11700 1 -- 172.21.15.32:0/2347418453 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a800c510 con 0x7f56b40d7800 2021-04-09T18:28:19.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b8db1700 1 -- 172.21.15.32:0/2347418453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56b40d7800 msgr2=0x7f56b40d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:19.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b8db1700 1 --2- 172.21.15.32:0/2347418453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56b40d7800 0x7f56b40d7c00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f56a8006650 tx=0x7f56a8009a80).stop 2021-04-09T18:28:19.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b8db1700 1 -- 172.21.15.32:0/2347418453 shutdown_connections 2021-04-09T18:28:19.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b8db1700 1 --2- 172.21.15.32:0/2347418453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56b40d7800 0x7f56b40d7c00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:19.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b8db1700 1 --2- 172.21.15.32:0/2347418453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b40cffa0 0x7f56b40d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:19.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b8db1700 1 -- 172.21.15.32:0/2347418453 >> 172.21.15.32:0/2347418453 conn(0x7f56b40ce5c0 msgr2=0x7f56b40cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:19.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b8db1700 1 -- 172.21.15.32:0/2347418453 shutdown_connections 2021-04-09T18:28:19.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.082 7f56b8db1700 1 -- 172.21.15.32:0/2347418453 wait complete. 2021-04-09T18:28:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.102 7f56b8db1700 1 Processor -- start 2021-04-09T18:28:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.102 7f56b8db1700 1 -- start start 2021-04-09T18:28:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b8db1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56b418e870 0x7f56b41812b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b8db1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56b4180330 0x7f56b41b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:19.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b8db1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b41b8c80 0x7f56b40adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b8db1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4180d30 con 0x7f56b418e870 2021-04-09T18:28:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b8db1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b41894e0 con 0x7f56b4180330 2021-04-09T18:28:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b8db1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b418ec70 con 0x7f56b41b8c80 2021-04-09T18:28:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b2b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56b418e870 0x7f56b41812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b2b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56b418e870 0x7f56b41812b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53832/0 (socket says 172.21.15.32:53832) 2021-04-09T18:28:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b2b13700 1 -- 172.21.15.32:0/368404326 learned_addr learned my addr 172.21.15.32:0/368404326 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b3314700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b41b8c80 0x7f56b40adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.103 7f56b2312700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56b4180330 0x7f56b41b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56b2b13700 1 -- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b41b8c80 msgr2=0x7f56b40adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:19.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56b2b13700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b41b8c80 0x7f56b40adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:19.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56b2b13700 1 -- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56b4180330 msgr2=0x7f56b41b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:19.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56b2b13700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56b4180330 0x7f56b41b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:19.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56b2b13700 1 -- 172.21.15.32:0/368404326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b41818d0 con 0x7f56b418e870 2021-04-09T18:28:19.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56b2b13700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56b418e870 0x7f56b41812b0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 rx=0x7f56a4006f60 tx=0x7f56a400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:19.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56a37fe700 1 -- 172.21.15.32:0/368404326 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a400bd60 con 0x7f56b418e870 2021-04-09T18:28:19.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56a37fe700 1 -- 172.21.15.32:0/368404326 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a4000c90 con 0x7f56b418e870 2021-04-09T18:28:19.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56b8db1700 1 -- 172.21.15.32:0/368404326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b41818d0 con 0x7f56b418e870 2021-04-09T18:28:19.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56a37fe700 1 -- 172.21.15.32:0/368404326 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a4002220 con 0x7f56b418e870 2021-04-09T18:28:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56b8db1700 1 -- 172.21.15.32:0/368404326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b41818d0 con 0x7f56b418e870 2021-04-09T18:28:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.104 7f56a37fe700 1 -- 172.21.15.32:0/368404326 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f56a40052e0 con 0x7f56b418e870 2021-04-09T18:28:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.105 7f56a37fe700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5698018930 0x7f569801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.105 7f56a37fe700 1 -- 172.21.15.32:0/368404326 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f56a4032af0 con 0x7f56b418e870 2021-04-09T18:28:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.105 7f56b2312700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5698018930 0x7f569801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.105 7f56b2312700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5698018930 0x7f569801ae10 crc :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:19.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.124 7f56b8db1700 1 -- 172.21.15.32:0/368404326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b418ec70 con 0x7f56b418e870 2021-04-09T18:28:19.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.128 7f56a37fe700 1 -- 172.21.15.32:0/368404326 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f56a4000c90 con 0x7f56b418e870 2021-04-09T18:28:19.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:19.322 7f56b8db1700 1 -- 172.21.15.32:0/368404326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f56b418ec70 con 0x7f56b418e870 2021-04-09T18:28:20.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.215 7f56a37fe700 1 -- 172.21.15.32:0/368404326 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v292) v1 ==== 104+0+0 (secure 0 0 0) 0x7f56a40052e0 con 0x7f56b418e870 2021-04-09T18:28:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 -- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5698018930 msgr2=0x7f569801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5698018930 0x7f569801ae10 crc :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:20.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 -- 172.21.15.32:0/368404326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56b418e870 msgr2=0x7f56b41812b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:20.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56b418e870 0x7f56b41812b0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 rx=0x7f56a4006f60 tx=0x7f56a400b970).stop 2021-04-09T18:28:20.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 -- 172.21.15.32:0/368404326 shutdown_connections 2021-04-09T18:28:20.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5698018930 0x7f569801ae10 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:20.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56b41b8c80 0x7f56b40adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:20.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56b4180330 0x7f56b41b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:20.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 --2- 172.21.15.32:0/368404326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56b418e870 0x7f56b41812b0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:20.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 -- 172.21.15.32:0/368404326 >> 172.21.15.32:0/368404326 conn(0x7f56b41897f0 msgr2=0x7f56b41b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:20.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 -- 172.21.15.32:0/368404326 shutdown_connections 2021-04-09T18:28:20.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.241 7f56b8db1700 1 -- 172.21.15.32:0/368404326 wait complete. 2021-04-09T18:28:20.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:28:20.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.419 7f4d9c5bd700 1 Processor -- start 2021-04-09T18:28:20.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.419 7f4d9c5bd700 1 -- start start 2021-04-09T18:28:20.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.419 7f4d9c5bd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4d940d7cc0 0x7f4d940d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:20.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.419 7f4d9c5bd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9410d500 con 0x7f4d9410da80 2021-04-09T18:28:20.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.419 7f4d9c5bd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d940d0330 con 0x7f4d9410d050 2021-04-09T18:28:20.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.419 7f4d9c5bd700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d940d9260 con 0x7f4d940d7cc0 2021-04-09T18:28:20.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d99bd8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4d940d7cc0 0x7f4d940d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:20.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d9abda700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4d9410da80 0x7f4d9410de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35200/0 (socket says 172.21.15.32:35200) 2021-04-09T18:28:20.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d9abda700 1 -- 172.21.15.32:0/2026010507 learned_addr learned my addr 172.21.15.32:0/2026010507 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:20.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d993d7700 1 -- 172.21.15.32:0/2026010507 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292257265 0 0) 0x7f4d8c000d30 con 0x7f4d9410da80 2021-04-09T18:28:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d993d7700 1 -- 172.21.15.32:0/2026010507 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d80002fe0 con 0x7f4d9410da80 2021-04-09T18:28:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d993d7700 1 -- 172.21.15.32:0/2026010507 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690432790 0 0) 0x7f4d90000c60 con 0x7f4d9410d050 2021-04-09T18:28:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d99bd8700 1 -- 172.21.15.32:0/2026010507 >> v1:172.21.15.86:6789/0 conn(0x7f4d9410d050 legacy=0x7f4d940cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d99bd8700 1 -- 172.21.15.32:0/2026010507 >> v1:172.21.15.32:6789/0 conn(0x7f4d9410da80 legacy=0x7f4d9410de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:20.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d99bd8700 1 -- 172.21.15.32:0/2026010507 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d94112140 con 0x7f4d940d7cc0 2021-04-09T18:28:20.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d99bd8700 1 --2- 172.21.15.32:0/2026010507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d940d7cc0 0x7f4d940d80c0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7f4d84006650 tx=0x7f4d84009a00).ready entity=mon.0 client_cookie=2180d2e21b2740fe server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:20.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d993d7700 1 -- 172.21.15.32:0/2026010507 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d84009df0 con 0x7f4d940d7cc0 2021-04-09T18:28:20.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d993d7700 1 -- 172.21.15.32:0/2026010507 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d840009b0 con 0x7f4d940d7cc0 2021-04-09T18:28:20.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d993d7700 1 -- 172.21.15.32:0/2026010507 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d8400c2b0 con 0x7f4d940d7cc0 2021-04-09T18:28:20.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d9c5bd700 1 -- 172.21.15.32:0/2026010507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d940d7cc0 msgr2=0x7f4d940d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:20.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.420 7f4d9c5bd700 1 --2- 172.21.15.32:0/2026010507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d940d7cc0 0x7f4d940d80c0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7f4d84006650 tx=0x7f4d84009a00).stop 2021-04-09T18:28:20.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.421 7f4d9c5bd700 1 -- 172.21.15.32:0/2026010507 shutdown_connections 2021-04-09T18:28:20.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.421 7f4d9c5bd700 1 --2- 172.21.15.32:0/2026010507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d940d7cc0 0x7f4d940d80c0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:20.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.421 7f4d9c5bd700 1 -- 172.21.15.32:0/2026010507 >> 172.21.15.32:0/2026010507 conn(0x7f4d940ce620 msgr2=0x7f4d940d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:20.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.421 7f4d9c5bd700 1 -- 172.21.15.32:0/2026010507 shutdown_connections 2021-04-09T18:28:20.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.421 7f4d9c5bd700 1 -- 172.21.15.32:0/2026010507 wait complete. 2021-04-09T18:28:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.441 7f4d9c5bd700 1 Processor -- start 2021-04-09T18:28:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.441 7f4d9c5bd700 1 -- start start 2021-04-09T18:28:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9c5bd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d94190e80 0x7f4d9409dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:20.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9c5bd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d940937b0 0x7f4d94093bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:20.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9c5bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d9409d190 0x7f4d9409d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:20.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9c5bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940941d0 con 0x7f4d9409d190 2021-04-09T18:28:20.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9c5bd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9409dbb0 con 0x7f4d940937b0 2021-04-09T18:28:20.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9c5bd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9409e5c0 con 0x7f4d94190e80 2021-04-09T18:28:20.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9abda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d9409d190 0x7f4d9409d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:20.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9abda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d9409d190 0x7f4d9409d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53850/0 (socket says 172.21.15.32:53850) 2021-04-09T18:28:20.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9abda700 1 -- 172.21.15.32:0/1337926051 learned_addr learned my addr 172.21.15.32:0/1337926051 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:20.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d99bd8700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d940937b0 0x7f4d94093bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:20.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9a3d9700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d94190e80 0x7f4d9409dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:20.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9abda700 1 -- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d94190e80 msgr2=0x7f4d9409dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:20.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9abda700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d94190e80 0x7f4d9409dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:20.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9abda700 1 -- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d940937b0 msgr2=0x7f4d94093bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:20.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9abda700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d940937b0 0x7f4d94093bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:20.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.442 7f4d9abda700 1 -- 172.21.15.32:0/1337926051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d94094a10 con 0x7f4d9409d190 2021-04-09T18:28:20.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.443 7f4d9abda700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d9409d190 0x7f4d9409d590 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 rx=0x7f4d8c007f90 tx=0x7f4d8c00ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:20.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.443 7f4d8affd700 1 -- 172.21.15.32:0/1337926051 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d8c0070d0 con 0x7f4d9409d190 2021-04-09T18:28:20.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.443 7f4d8affd700 1 -- 172.21.15.32:0/1337926051 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8c007410 con 0x7f4d9409d190 2021-04-09T18:28:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.443 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d94094a10 con 0x7f4d9409d190 2021-04-09T18:28:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.443 7f4d8affd700 1 -- 172.21.15.32:0/1337926051 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d8c004910 con 0x7f4d9409d190 2021-04-09T18:28:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.443 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d94094a10 con 0x7f4d9409d190 2021-04-09T18:28:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.443 7f4d8affd700 1 -- 172.21.15.32:0/1337926051 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4d8c004720 con 0x7f4d9409d190 2021-04-09T18:28:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.443 7f4d8affd700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d7c018930 0x7f4d7c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.443 7f4d8affd700 1 -- 172.21.15.32:0/1337926051 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f4d8c032e80 con 0x7f4d9409d190 2021-04-09T18:28:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.444 7f4d9a3d9700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d7c018930 0x7f4d7c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.444 7f4d9a3d9700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d7c018930 0x7f4d7c01ae10 crc :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.464 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d94095580 con 0x7f4d9409d190 2021-04-09T18:28:20.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.467 7f4d8affd700 1 -- 172.21.15.32:0/1337926051 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4d8c007d00 con 0x7f4d9409d190 2021-04-09T18:28:20.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:20.651 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4d940944e0 con 0x7f4d9409d190 2021-04-09T18:28:21.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.466 7f4d8affd700 1 -- 172.21.15.32:0/1337926051 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v145) v1 ==== 185+0+0 (secure 0 0 0) 0x7f4d8c007310 con 0x7f4d9409d190 2021-04-09T18:28:21.546 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:28:21.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d7c018930 msgr2=0x7f4d7c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:21.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d7c018930 0x7f4d7c01ae10 crc :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d9409d190 msgr2=0x7f4d9409d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:21.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d9409d190 0x7f4d9409d590 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 rx=0x7f4d8c007f90 tx=0x7f4d8c00ac50).stop 2021-04-09T18:28:21.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 shutdown_connections 2021-04-09T18:28:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d7c018930 0x7f4d7c01ae10 crc :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d9409d190 0x7f4d9409d590 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d940937b0 0x7f4d94093bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 --2- 172.21.15.32:0/1337926051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d94190e80 0x7f4d9409dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 >> 172.21.15.32:0/1337926051 conn(0x7f4d940b8fb0 msgr2=0x7f4d940adef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:21.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 shutdown_connections 2021-04-09T18:28:21.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.471 7f4d9c5bd700 1 -- 172.21.15.32:0/1337926051 wait complete. 2021-04-09T18:28:21.562 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:28:21.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.649 7f5bcedc5700 1 Processor -- start 2021-04-09T18:28:21.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.649 7f5bcedc5700 1 -- start start 2021-04-09T18:28:21.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bcedc5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5bc80d7cc0 0x7f5bc80d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:21.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bcedc5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc810da80 0x7f5bc810de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:21.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bcedc5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc810d500 con 0x7f5bc810d050 2021-04-09T18:28:21.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bcedc5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc80d9260 con 0x7f5bc80d7cc0 2021-04-09T18:28:21.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bcedc5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc8112140 con 0x7f5bc810da80 2021-04-09T18:28:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bc7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5bc80d7cc0 0x7f5bc80d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bc7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5bc80d7cc0 0x7f5bc80d80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53858/0 (socket says 172.21.15.32:53858) 2021-04-09T18:28:21.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bc7fff700 1 -- 172.21.15.32:0/852405981 learned_addr learned my addr 172.21.15.32:0/852405981 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:21.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bccbe1700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5bc810d050 0x7f5bc80cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:33138/0 (socket says 172.21.15.32:33138) 2021-04-09T18:28:21.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bcd3e2700 1 --2- 172.21.15.32:0/852405981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc810da80 0x7f5bc810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:21.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bc77fe700 1 -- 172.21.15.32:0/852405981 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411226153 0 0) 0x7f5bb8000c60 con 0x7f5bc810d050 2021-04-09T18:28:21.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bc7fff700 1 -- 172.21.15.32:0/852405981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc810da80 msgr2=0x7f5bc810de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:21.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bc7fff700 1 --2- 172.21.15.32:0/852405981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc810da80 0x7f5bc810de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bc7fff700 1 -- 172.21.15.32:0/852405981 >> v1:172.21.15.86:6789/0 conn(0x7f5bc810d050 legacy=0x7f5bc80cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:21.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bc7fff700 1 -- 172.21.15.32:0/852405981 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc80d0330 con 0x7f5bc80d7cc0 2021-04-09T18:28:21.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.650 7f5bcd3e2700 1 --2- 172.21.15.32:0/852405981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc810da80 0x7f5bc810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:21.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bc7fff700 1 --2- 172.21.15.32:0/852405981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc80d7cc0 0x7f5bc80d80c0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7f5bbc006700 tx=0x7f5bbc009b80).ready entity=mon.0 client_cookie=c4fffca5704b12e7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:21.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bc77fe700 1 -- 172.21.15.32:0/852405981 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bbc00c0c0 con 0x7f5bc80d7cc0 2021-04-09T18:28:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bc77fe700 1 -- 172.21.15.32:0/852405981 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bbc00c400 con 0x7f5bc80d7cc0 2021-04-09T18:28:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bc77fe700 1 -- 172.21.15.32:0/852405981 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bbc00c8e0 con 0x7f5bc80d7cc0 2021-04-09T18:28:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bcedc5700 1 -- 172.21.15.32:0/852405981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc80d7cc0 msgr2=0x7f5bc80d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bcedc5700 1 --2- 172.21.15.32:0/852405981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc80d7cc0 0x7f5bc80d80c0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7f5bbc006700 tx=0x7f5bbc009b80).stop 2021-04-09T18:28:21.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bcedc5700 1 -- 172.21.15.32:0/852405981 shutdown_connections 2021-04-09T18:28:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bcedc5700 1 --2- 172.21.15.32:0/852405981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc810da80 0x7f5bc810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bcedc5700 1 --2- 172.21.15.32:0/852405981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc80d7cc0 0x7f5bc80d80c0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.651 7f5bcedc5700 1 -- 172.21.15.32:0/852405981 >> 172.21.15.32:0/852405981 conn(0x7f5bc80ce620 msgr2=0x7f5bc80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:21.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.652 7f5bcedc5700 1 -- 172.21.15.32:0/852405981 shutdown_connections 2021-04-09T18:28:21.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.652 7f5bcedc5700 1 -- 172.21.15.32:0/852405981 wait complete. 2021-04-09T18:28:21.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.672 7f5bcedc5700 1 Processor -- start 2021-04-09T18:28:21.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.672 7f5bcedc5700 1 -- start start 2021-04-09T18:28:21.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.672 7f5bcedc5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc80a7930 0x7f5bc80c26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:21.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.672 7f5bcedc5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc8094560 0x7f5bc8094960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:21.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.672 7f5bcedc5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bc8094f80 0x7f5bc8095940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:21.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.672 7f5bcedc5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80a7db0 con 0x7f5bc8094560 2021-04-09T18:28:21.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.672 7f5bcedc5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80953c0 con 0x7f5bc8094f80 2021-04-09T18:28:21.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.672 7f5bcedc5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80c2cd0 con 0x7f5bc80a7930 2021-04-09T18:28:21.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc8094560 0x7f5bc8094960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:21.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc8094560 0x7f5bc8094960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53860/0 (socket says 172.21.15.32:53860) 2021-04-09T18:28:21.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc7fff700 1 -- 172.21.15.32:0/3013494504 learned_addr learned my addr 172.21.15.32:0/3013494504 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:21.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bccbe1700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc80a7930 0x7f5bc80c26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:21.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bcd3e2700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bc8094f80 0x7f5bc8095940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:21.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc7fff700 1 -- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc80a7930 msgr2=0x7f5bc80c26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:21.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc7fff700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc80a7930 0x7f5bc80c26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc7fff700 1 -- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bc8094f80 msgr2=0x7f5bc8095940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:21.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc7fff700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bc8094f80 0x7f5bc8095940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:21.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc7fff700 1 -- 172.21.15.32:0/3013494504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8190530 con 0x7f5bc8094560 2021-04-09T18:28:21.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc7fff700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc8094560 0x7f5bc8094960 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7f5bbc00b5e0 tx=0x7f5bbc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:21.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc57fa700 1 -- 172.21.15.32:0/3013494504 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bbc00bd70 con 0x7f5bc8094560 2021-04-09T18:28:21.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc57fa700 1 -- 172.21.15.32:0/3013494504 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bbc002030 con 0x7f5bc8094560 2021-04-09T18:28:21.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc8190530 con 0x7f5bc8094560 2021-04-09T18:28:21.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bc57fa700 1 -- 172.21.15.32:0/3013494504 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bbc002510 con 0x7f5bc8094560 2021-04-09T18:28:21.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.673 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc8190530 con 0x7f5bc8094560 2021-04-09T18:28:21.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.674 7f5bc57fa700 1 -- 172.21.15.32:0/3013494504 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5bbc002210 con 0x7f5bc8094560 2021-04-09T18:28:21.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.674 7f5bc57fa700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ba8018930 0x7f5ba801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:21.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.674 7f5bc57fa700 1 -- 172.21.15.32:0/3013494504 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5397+0+0 (secure 0 0 0) 0x7f5bbc032990 con 0x7f5bc8094560 2021-04-09T18:28:21.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.674 7f5bccbe1700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ba8018930 0x7f5ba801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:21.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.675 7f5bccbe1700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ba8018930 0x7f5ba801ae10 crc :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:21.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.694 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc80c2cd0 con 0x7f5bc8094560 2021-04-09T18:28:21.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.697 7f5bc57fa700 1 -- 172.21.15.32:0/3013494504 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5bbc000c90 con 0x7f5bc8094560 2021-04-09T18:28:21.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:21.879 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5bc80c2cd0 con 0x7f5bc8094560 2021-04-09T18:28:22.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.469 7f5bc57fa700 1 -- 172.21.15.32:0/3013494504 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v146) v1 ==== 173+0+0 (secure 0 0 0) 0x7f5bbc001f10 con 0x7f5bc8094560 2021-04-09T18:28:22.573 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:28:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ba8018930 msgr2=0x7f5ba801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ba8018930 0x7f5ba801ae10 crc :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc8094560 msgr2=0x7f5bc8094960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc8094560 0x7f5bc8094960 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7f5bbc00b5e0 tx=0x7f5bbc00b980).stop 2021-04-09T18:28:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 shutdown_connections 2021-04-09T18:28:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ba8018930 0x7f5ba801ae10 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5bc8094560 0x7f5bc8094960 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5bc8094f80 0x7f5bc8095940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 --2- 172.21.15.32:0/3013494504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5bc80a7930 0x7f5bc80c26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 >> 172.21.15.32:0/3013494504 conn(0x7f5bc8093fc0 msgr2=0x7f5bc80b92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:22.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 shutdown_connections 2021-04-09T18:28:22.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.496 7f5bcedc5700 1 -- 172.21.15.32:0/3013494504 wait complete. 2021-04-09T18:28:22.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:28:22.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.660 7fcea7018700 1 Processor -- start 2021-04-09T18:28:22.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.660 7fcea7018700 1 -- start start 2021-04-09T18:28:22.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fcea7018700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcea00d6c80 0x7fcea00d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:22.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fcea7018700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcea00d2440 0x7fcea010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:22.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fcea7018700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea00d7280 con 0x7fcea00cffa0 2021-04-09T18:28:22.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fcea7018700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea00d7080 con 0x7fcea00d6c80 2021-04-09T18:28:22.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fcea7018700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea00d91e0 con 0x7fcea00d2440 2021-04-09T18:28:22.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fce9ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcea00d6c80 0x7fcea00d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:22.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fcea4e34700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcea00cffa0 0x7fcea00d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35226/0 (socket says 172.21.15.32:35226) 2021-04-09T18:28:22.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fcea4e34700 1 -- 172.21.15.32:0/354315151 learned_addr learned my addr 172.21.15.32:0/354315151 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:22.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fcea5635700 1 --2- 172.21.15.32:0/354315151 >> v2:172.21.15.86:3300/0 conn(0x7fcea00d2440 0x7fcea010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:22.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fce9ffff700 1 -- 172.21.15.32:0/354315151 >> v2:172.21.15.86:3300/0 conn(0x7fcea00d2440 msgr2=0x7fcea010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:22.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fce9ffff700 1 --2- 172.21.15.32:0/354315151 >> v2:172.21.15.86:3300/0 conn(0x7fcea00d2440 0x7fcea010cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fce9ffff700 1 -- 172.21.15.32:0/354315151 >> v1:172.21.15.32:6789/0 conn(0x7fcea00cffa0 legacy=0x7fcea00d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fce9f7fe700 1 -- 172.21.15.32:0/354315151 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138351612 0 0) 0x7fce98000ce0 con 0x7fcea00cffa0 2021-04-09T18:28:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.661 7fce9ffff700 1 -- 172.21.15.32:0/354315151 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea00d2840 con 0x7fcea00d6c80 2021-04-09T18:28:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.662 7fce9ffff700 1 --2- 172.21.15.32:0/354315151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea00d6c80 0x7fcea00d7ff0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7fce900069d0 tx=0x7fce90009dc0).ready entity=mon.0 client_cookie=1d31834dc6813e25 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.662 7fce9f7fe700 1 -- 172.21.15.32:0/354315151 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce9000b510 con 0x7fcea00d6c80 2021-04-09T18:28:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.662 7fce9f7fe700 1 -- 172.21.15.32:0/354315151 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce90006ac0 con 0x7fcea00d6c80 2021-04-09T18:28:22.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.662 7fce9f7fe700 1 -- 172.21.15.32:0/354315151 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce9000ba80 con 0x7fcea00d6c80 2021-04-09T18:28:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.662 7fcea7018700 1 -- 172.21.15.32:0/354315151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea00d6c80 msgr2=0x7fcea00d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.662 7fcea7018700 1 --2- 172.21.15.32:0/354315151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea00d6c80 0x7fcea00d7ff0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7fce900069d0 tx=0x7fce90009dc0).stop 2021-04-09T18:28:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.663 7fcea7018700 1 -- 172.21.15.32:0/354315151 shutdown_connections 2021-04-09T18:28:22.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.663 7fcea7018700 1 --2- 172.21.15.32:0/354315151 >> v2:172.21.15.86:3300/0 conn(0x7fcea00d2440 0x7fcea010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.663 7fcea7018700 1 --2- 172.21.15.32:0/354315151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea00d6c80 0x7fcea00d7ff0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.663 7fcea7018700 1 -- 172.21.15.32:0/354315151 >> 172.21.15.32:0/354315151 conn(0x7fcea00ce5c0 msgr2=0x7fcea00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:22.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.663 7fcea7018700 1 -- 172.21.15.32:0/354315151 shutdown_connections 2021-04-09T18:28:22.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.663 7fcea7018700 1 -- 172.21.15.32:0/354315151 wait complete. 2021-04-09T18:28:22.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.683 7fcea7018700 1 Processor -- start 2021-04-09T18:28:22.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.683 7fcea7018700 1 -- start start 2021-04-09T18:28:22.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea7018700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea01b8370 0x7fcea01b8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:22.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea7018700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea008c7b0 0x7fcea008cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:22.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea7018700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea008d1d0 0x7fcea01805a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:22.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea7018700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0181a20 con 0x7fcea01b8370 2021-04-09T18:28:22.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea7018700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea008c230 con 0x7fcea008c7b0 2021-04-09T18:28:22.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea7018700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea01b8d90 con 0x7fcea008d1d0 2021-04-09T18:28:22.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea4e34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea01b8370 0x7fcea01b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:22.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea4e34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea01b8370 0x7fcea01b8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53874/0 (socket says 172.21.15.32:53874) 2021-04-09T18:28:22.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea4e34700 1 -- 172.21.15.32:0/1371766074 learned_addr learned my addr 172.21.15.32:0/1371766074 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:22.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fce9ffff700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea008c7b0 0x7fcea008cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:22.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea5635700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea008d1d0 0x7fcea01805a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:22.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea4e34700 1 -- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea008d1d0 msgr2=0x7fcea01805a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:22.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea4e34700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea008d1d0 0x7fcea01805a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea4e34700 1 -- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea008c7b0 msgr2=0x7fcea008cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:22.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea4e34700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea008c7b0 0x7fcea008cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:22.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea4e34700 1 -- 172.21.15.32:0/1371766074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea01b8f80 con 0x7fcea01b8370 2021-04-09T18:28:22.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea4e34700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea01b8370 0x7fcea01b8770 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 rx=0x7fce98006f00 tx=0x7fce9800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:22.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fce9d7fa700 1 -- 172.21.15.32:0/1371766074 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce98006020 con 0x7fcea01b8370 2021-04-09T18:28:22.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fce9d7fa700 1 -- 172.21.15.32:0/1371766074 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce980063a0 con 0x7fcea01b8370 2021-04-09T18:28:22.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fce9d7fa700 1 -- 172.21.15.32:0/1371766074 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce98004620 con 0x7fcea01b8370 2021-04-09T18:28:22.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.684 7fcea7018700 1 -- 172.21.15.32:0/1371766074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea01b8f80 con 0x7fcea01b8370 2021-04-09T18:28:22.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.685 7fcea7018700 1 -- 172.21.15.32:0/1371766074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea01b8f80 con 0x7fcea01b8370 2021-04-09T18:28:22.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.685 7fce9d7fa700 1 -- 172.21.15.32:0/1371766074 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fce9800a000 con 0x7fcea01b8370 2021-04-09T18:28:22.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.685 7fce9d7fa700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce840188d0 0x7fce8401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:22.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.685 7fce9d7fa700 1 -- 172.21.15.32:0/1371766074 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5015+0+0 (secure 0 0 0) 0x7fce980327b0 con 0x7fcea01b8370 2021-04-09T18:28:22.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.686 7fce9ffff700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce840188d0 0x7fce8401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:22.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.686 7fce9ffff700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce840188d0 0x7fce8401add0 crc :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:22.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.705 7fcea7018700 1 -- 172.21.15.32:0/1371766074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea01b7170 con 0x7fcea01b8370 2021-04-09T18:28:22.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.708 7fce9d7fa700 1 -- 172.21.15.32:0/1371766074 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fce98006b90 con 0x7fcea01b8370 2021-04-09T18:28:23.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.920 7fcea7018700 1 -- 172.21.15.32:0/1371766074 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fcea008c230 con 0x7fcea01b8370 2021-04-09T18:28:23.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.923 7fce9d7fa700 1 -- 172.21.15.32:0/1371766074 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7fce9800a000 con 0x7fcea01b8370 2021-04-09T18:28:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 -- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce840188d0 msgr2=0x7fce8401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce840188d0 0x7fce8401add0 crc :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 -- 172.21.15.32:0/1371766074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea01b8370 msgr2=0x7fcea01b8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea01b8370 0x7fcea01b8770 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 rx=0x7fce98006f00 tx=0x7fce9800b940).stop 2021-04-09T18:28:23.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 -- 172.21.15.32:0/1371766074 shutdown_connections 2021-04-09T18:28:23.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce840188d0 0x7fce8401add0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcea008d1d0 0x7fcea01805a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea008c7b0 0x7fcea008cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 --2- 172.21.15.32:0/1371766074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcea01b8370 0x7fcea01b8770 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 -- 172.21.15.32:0/1371766074 >> 172.21.15.32:0/1371766074 conn(0x7fcea01897f0 msgr2=0x7fcea01b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 -- 172.21.15.32:0/1371766074 shutdown_connections 2021-04-09T18:28:23.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:22.929 7fcea7018700 1 -- 172.21.15.32:0/1371766074 wait complete. 2021-04-09T18:28:23.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestFailover.test_discontinuous_mdsmap' 2021-04-09T18:28:23.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.090 7fbc27686700 1 Processor -- start 2021-04-09T18:28:23.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.090 7fbc27686700 1 -- start start 2021-04-09T18:28:23.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc27686700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc200d7ca0 0x7fbc200d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc27686700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbc2010da00 0x7fbc2010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc27686700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2010d480 con 0x7fbc2010d010 2021-04-09T18:28:23.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc27686700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc200d9220 con 0x7fbc200d7ca0 2021-04-09T18:28:23.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc27686700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc200d86c0 con 0x7fbc2010da00 2021-04-09T18:28:23.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc24ca1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc200d7ca0 0x7fbc200d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc254a2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbc2010d010 0x7fbc200d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35240/0 (socket says 172.21.15.32:35240) 2021-04-09T18:28:23.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc254a2700 1 -- 172.21.15.32:0/3822393904 learned_addr learned my addr 172.21.15.32:0/3822393904 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:23.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc25ca3700 1 --2- 172.21.15.32:0/3822393904 >> v2:172.21.15.86:3300/0 conn(0x7fbc2010da00 0x7fbc2010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc24ca1700 1 -- 172.21.15.32:0/3822393904 >> v2:172.21.15.86:3300/0 conn(0x7fbc2010da00 msgr2=0x7fbc2010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:23.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc24ca1700 1 --2- 172.21.15.32:0/3822393904 >> v2:172.21.15.86:3300/0 conn(0x7fbc2010da00 0x7fbc2010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc24ca1700 1 -- 172.21.15.32:0/3822393904 >> v1:172.21.15.32:6789/0 conn(0x7fbc2010d010 legacy=0x7fbc200d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc24ca1700 1 -- 172.21.15.32:0/3822393904 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc200d1f70 con 0x7fbc200d7ca0 2021-04-09T18:28:23.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc17fff700 1 -- 172.21.15.32:0/3822393904 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150133614 0 0) 0x7fbc1c000ce0 con 0x7fbc2010d010 2021-04-09T18:28:23.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc25ca3700 1 --2- 172.21.15.32:0/3822393904 >> v2:172.21.15.86:3300/0 conn(0x7fbc2010da00 0x7fbc2010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:23.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc24ca1700 1 --2- 172.21.15.32:0/3822393904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc200d7ca0 0x7fbc200d80a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 rx=0x7fbc10006650 tx=0x7fbc10009a80).ready entity=mon.0 client_cookie=cfd5a25e65dab07e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:23.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc17fff700 1 -- 172.21.15.32:0/3822393904 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc100009b0 con 0x7fbc200d7ca0 2021-04-09T18:28:23.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc17fff700 1 -- 172.21.15.32:0/3822393904 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1000c0c0 con 0x7fbc200d7ca0 2021-04-09T18:28:23.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc17fff700 1 -- 172.21.15.32:0/3822393904 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc1000c510 con 0x7fbc200d7ca0 2021-04-09T18:28:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc27686700 1 -- 172.21.15.32:0/3822393904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc200d7ca0 msgr2=0x7fbc200d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.091 7fbc27686700 1 --2- 172.21.15.32:0/3822393904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc200d7ca0 0x7fbc200d80a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 rx=0x7fbc10006650 tx=0x7fbc10009a80).stop 2021-04-09T18:28:23.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.092 7fbc27686700 1 -- 172.21.15.32:0/3822393904 shutdown_connections 2021-04-09T18:28:23.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.092 7fbc27686700 1 --2- 172.21.15.32:0/3822393904 >> v2:172.21.15.86:3300/0 conn(0x7fbc2010da00 0x7fbc2010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.092 7fbc27686700 1 --2- 172.21.15.32:0/3822393904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc200d7ca0 0x7fbc200d80a0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.092 7fbc27686700 1 -- 172.21.15.32:0/3822393904 >> 172.21.15.32:0/3822393904 conn(0x7fbc200ce5f0 msgr2=0x7fbc200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:23.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.092 7fbc27686700 1 -- 172.21.15.32:0/3822393904 shutdown_connections 2021-04-09T18:28:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.092 7fbc27686700 1 -- 172.21.15.32:0/3822393904 wait complete. 2021-04-09T18:28:23.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.112 7fbc27686700 1 Processor -- start 2021-04-09T18:28:23.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.112 7fbc27686700 1 -- start start 2021-04-09T18:28:23.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc27686700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc20188780 0x7fbc20188b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc27686700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc201996c0 0x7fbc20199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc27686700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc2008de20 0x7fbc2008e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc27686700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2019a0e0 con 0x7fbc201996c0 2021-04-09T18:28:23.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc27686700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2008e840 con 0x7fbc2008de20 2021-04-09T18:28:23.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc27686700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2008ea60 con 0x7fbc20188780 2021-04-09T18:28:23.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc24ca1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc201996c0 0x7fbc20199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc24ca1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc201996c0 0x7fbc20199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53892/0 (socket says 172.21.15.32:53892) 2021-04-09T18:28:23.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc24ca1700 1 -- 172.21.15.32:0/1730497525 learned_addr learned my addr 172.21.15.32:0/1730497525 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:23.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc25ca3700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc2008de20 0x7fbc2008e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.113 7fbc254a2700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc20188780 0x7fbc20188b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc24ca1700 1 -- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc20188780 msgr2=0x7fbc20188b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc24ca1700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc20188780 0x7fbc20188b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc24ca1700 1 -- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc2008de20 msgr2=0x7fbc2008e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc24ca1700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc2008de20 0x7fbc2008e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc24ca1700 1 -- 172.21.15.32:0/1730497525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2008ec80 con 0x7fbc201996c0 2021-04-09T18:28:23.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc24ca1700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc201996c0 0x7fbc20199ac0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7fbc10009650 tx=0x7fbc1000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:23.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc15ffb700 1 -- 172.21.15.32:0/1730497525 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc100093a0 con 0x7fbc201996c0 2021-04-09T18:28:23.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc15ffb700 1 -- 172.21.15.32:0/1730497525 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc10006070 con 0x7fbc201996c0 2021-04-09T18:28:23.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc27686700 1 -- 172.21.15.32:0/1730497525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2008ec80 con 0x7fbc201996c0 2021-04-09T18:28:23.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc15ffb700 1 -- 172.21.15.32:0/1730497525 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc100052e0 con 0x7fbc201996c0 2021-04-09T18:28:23.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.114 7fbc27686700 1 -- 172.21.15.32:0/1730497525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2008ec80 con 0x7fbc201996c0 2021-04-09T18:28:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.115 7fbc15ffb700 1 -- 172.21.15.32:0/1730497525 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbc100052e0 con 0x7fbc201996c0 2021-04-09T18:28:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.115 7fbc15ffb700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc00018930 0x7fbc0001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.115 7fbc15ffb700 1 -- 172.21.15.32:0/1730497525 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5015+0+0 (secure 0 0 0) 0x7fbc100327e0 con 0x7fbc201996c0 2021-04-09T18:28:23.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.115 7fbc254a2700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc00018930 0x7fbc0001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.115 7fbc254a2700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc00018930 0x7fbc0001ae10 crc :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.135 7fbc27686700 1 -- 172.21.15.32:0/1730497525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc20182d10 con 0x7fbc201996c0 2021-04-09T18:28:23.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.138 7fbc15ffb700 1 -- 172.21.15.32:0/1730497525 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbc10006030 con 0x7fbc201996c0 2021-04-09T18:28:23.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.325 7fbc27686700 1 -- 172.21.15.32:0/1730497525 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestFailover.test_discontinuous_mdsmap"]} v 0) v1 -- 0x7fbc2010d480 con 0x7fbc201996c0 2021-04-09T18:28:23.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.471 7fbc15ffb700 1 -- 172.21.15.32:0/1730497525 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestFailover.test_discontinuous_mdsmap"]}]=0 v696) v1 ==== 144+0+0 (secure 0 0 0) 0x7fbc100052e0 con 0x7fbc201996c0 2021-04-09T18:28:23.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.490 7fbc27686700 1 -- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc00018930 msgr2=0x7fbc0001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc00018930 0x7fbc0001ae10 crc :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 -- 172.21.15.32:0/1730497525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc201996c0 msgr2=0x7fbc20199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc201996c0 0x7fbc20199ac0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7fbc10009650 tx=0x7fbc1000b8e0).stop 2021-04-09T18:28:23.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 -- 172.21.15.32:0/1730497525 shutdown_connections 2021-04-09T18:28:23.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc00018930 0x7fbc0001ae10 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc201996c0 0x7fbc20199ac0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc2008de20 0x7fbc2008e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 --2- 172.21.15.32:0/1730497525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc20188780 0x7fbc20188b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 -- 172.21.15.32:0/1730497525 >> 172.21.15.32:0/1730497525 conn(0x7fbc201a8e60 msgr2=0x7fbc201a0670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:23.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 -- 172.21.15.32:0/1730497525 shutdown_connections 2021-04-09T18:28:23.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.491 7fbc27686700 1 -- 172.21.15.32:0/1730497525 wait complete. 2021-04-09T18:28:23.583 INFO:tasks.cephfs_test_runner:test_discontinuous_mdsmap (tasks.cephfs.test_failover.TestFailover) ... ok 2021-04-09T18:28:23.584 INFO:tasks.cephfs_test_runner:Starting test: test_simple (tasks.cephfs.test_failover.TestFailover) 2021-04-09T18:28:23.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestFailover.test_simple' 2021-04-09T18:28:23.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.661 7fac1b005700 1 Processor -- start 2021-04-09T18:28:23.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.661 7fac1b005700 1 -- start start 2021-04-09T18:28:23.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac1b005700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac1410d010 0x7fac140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac1b005700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac140d8490 0x7fac1410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac1b005700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1410d480 con 0x7fac140d1da0 2021-04-09T18:28:23.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac1b005700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1410e780 con 0x7fac140d8490 2021-04-09T18:28:23.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac1b005700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac140d8d30 con 0x7fac1410d010 2021-04-09T18:28:23.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac18e21700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac1410d010 0x7fac140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac19622700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac140d8490 0x7fac1410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac13fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fac140d1da0 0x7fac140d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:33182/0 (socket says 172.21.15.32:33182) 2021-04-09T18:28:23.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac13fff700 1 -- 172.21.15.32:0/186347616 learned_addr learned my addr 172.21.15.32:0/186347616 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:23.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.662 7fac137fe700 1 -- 172.21.15.32:0/186347616 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564391207 0 0) 0x7fac04001440 con 0x7fac140d1da0 2021-04-09T18:28:23.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac137fe700 1 -- 172.21.15.32:0/186347616 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac00002fe0 con 0x7fac140d1da0 2021-04-09T18:28:23.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac18e21700 1 -- 172.21.15.32:0/186347616 >> v2:172.21.15.86:3300/0 conn(0x7fac140d8490 msgr2=0x7fac1410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:23.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac18e21700 1 --2- 172.21.15.32:0/186347616 >> v2:172.21.15.86:3300/0 conn(0x7fac140d8490 0x7fac1410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac18e21700 1 -- 172.21.15.32:0/186347616 >> v1:172.21.15.86:6789/0 conn(0x7fac140d1da0 legacy=0x7fac140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac18e21700 1 -- 172.21.15.32:0/186347616 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac140d9220 con 0x7fac1410d010 2021-04-09T18:28:23.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac19622700 1 --2- 172.21.15.32:0/186347616 >> v2:172.21.15.86:3300/0 conn(0x7fac140d8490 0x7fac1410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:23.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac18e21700 1 --2- 172.21.15.32:0/186347616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac1410d010 0x7fac140d1780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fabfc006650 tx=0x7fabfc009a80).ready entity=mon.2 client_cookie=ad0e47cc54e12551 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:23.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac137fe700 1 -- 172.21.15.32:0/186347616 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabfc0009b0 con 0x7fac1410d010 2021-04-09T18:28:23.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac137fe700 1 -- 172.21.15.32:0/186347616 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc00c0c0 con 0x7fac1410d010 2021-04-09T18:28:23.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac137fe700 1 -- 172.21.15.32:0/186347616 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabfc00c510 con 0x7fac1410d010 2021-04-09T18:28:23.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac1b005700 1 -- 172.21.15.32:0/186347616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac1410d010 msgr2=0x7fac140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.663 7fac1b005700 1 --2- 172.21.15.32:0/186347616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac1410d010 0x7fac140d1780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fabfc006650 tx=0x7fabfc009a80).stop 2021-04-09T18:28:23.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.664 7fac1b005700 1 -- 172.21.15.32:0/186347616 shutdown_connections 2021-04-09T18:28:23.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.664 7fac1b005700 1 --2- 172.21.15.32:0/186347616 >> v2:172.21.15.86:3300/0 conn(0x7fac140d8490 0x7fac1410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.664 7fac1b005700 1 --2- 172.21.15.32:0/186347616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac1410d010 0x7fac140d1780 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.664 7fac1b005700 1 -- 172.21.15.32:0/186347616 >> 172.21.15.32:0/186347616 conn(0x7fac140ce5f0 msgr2=0x7fac140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:23.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.664 7fac1b005700 1 -- 172.21.15.32:0/186347616 shutdown_connections 2021-04-09T18:28:23.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.664 7fac1b005700 1 -- 172.21.15.32:0/186347616 wait complete. 2021-04-09T18:28:23.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.684 7fac1b005700 1 Processor -- start 2021-04-09T18:28:23.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.684 7fac1b005700 1 -- start start 2021-04-09T18:28:23.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac1b005700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac141a5210 0x7fac141a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac1b005700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac141a6340 0x7fac141a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac1b005700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac140abaa0 0x7fac140abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac1b005700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac141a6d60 con 0x7fac141a5210 2021-04-09T18:28:23.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac1b005700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac140ac4c0 con 0x7fac141a6340 2021-04-09T18:28:23.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac1b005700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac140ac6e0 con 0x7fac140abaa0 2021-04-09T18:28:23.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac18e21700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac141a5210 0x7fac141a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac18e21700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac141a5210 0x7fac141a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53902/0 (socket says 172.21.15.32:53902) 2021-04-09T18:28:23.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac18e21700 1 -- 172.21.15.32:0/3591346454 learned_addr learned my addr 172.21.15.32:0/3591346454 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:23.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac13fff700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac141a6340 0x7fac141a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac19622700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac140abaa0 0x7fac140abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac18e21700 1 -- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac140abaa0 msgr2=0x7fac140abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac18e21700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac140abaa0 0x7fac140abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac18e21700 1 -- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac141a6340 msgr2=0x7fac141a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:23.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac18e21700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac141a6340 0x7fac141a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:23.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac18e21700 1 -- 172.21.15.32:0/3591346454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1417a720 con 0x7fac141a5210 2021-04-09T18:28:23.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac19622700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac140abaa0 0x7fac140abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:23.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac13fff700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac141a6340 0x7fac141a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:23.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac18e21700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac141a5210 0x7fac141a56c0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7fabfc00d7e0 tx=0x7fabfc00db80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:23.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac117fa700 1 -- 172.21.15.32:0/3591346454 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabfc000c90 con 0x7fac141a5210 2021-04-09T18:28:23.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac117fa700 1 -- 172.21.15.32:0/3591346454 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc000960 con 0x7fac141a5210 2021-04-09T18:28:23.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac117fa700 1 -- 172.21.15.32:0/3591346454 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabfc002030 con 0x7fac141a5210 2021-04-09T18:28:23.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.685 7fac1b005700 1 -- 172.21.15.32:0/3591346454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1417a9c0 con 0x7fac141a5210 2021-04-09T18:28:23.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.686 7fac1b005700 1 -- 172.21.15.32:0/3591346454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1417a9c0 con 0x7fac141a5210 2021-04-09T18:28:23.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.686 7fac117fa700 1 -- 172.21.15.32:0/3591346454 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fabfc0052e0 con 0x7fac141a5210 2021-04-09T18:28:23.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.686 7fac117fa700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabf80188d0 0x7fabf801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:23.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.686 7fac117fa700 1 -- 172.21.15.32:0/3591346454 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5015+0+0 (secure 0 0 0) 0x7fabfc0349b0 con 0x7fac141a5210 2021-04-09T18:28:23.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.686 7fac13fff700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabf80188d0 0x7fabf801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:23.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.687 7fac13fff700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabf80188d0 0x7fabf801add0 crc :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:23.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.706 7fac1b005700 1 -- 172.21.15.32:0/3591346454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac141a6d60 con 0x7fac141a5210 2021-04-09T18:28:23.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.709 7fac117fa700 1 -- 172.21.15.32:0/3591346454 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fabfc000940 con 0x7fac141a5210 2021-04-09T18:28:23.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:23.898 7fac1b005700 1 -- 172.21.15.32:0/3591346454 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestFailover.test_simple"]} v 0) v1 -- 0x7fac141a6d60 con 0x7fac141a5210 2021-04-09T18:28:24.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.473 7fac117fa700 1 -- 172.21.15.32:0/3591346454 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestFailover.test_simple"]}]=0 v697) v1 ==== 133+0+0 (secure 0 0 0) 0x7fabfc000940 con 0x7fac141a5210 2021-04-09T18:28:24.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 -- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabf80188d0 msgr2=0x7fabf801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:24.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabf80188d0 0x7fabf801add0 crc :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 -- 172.21.15.32:0/3591346454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac141a5210 msgr2=0x7fac141a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:24.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac141a5210 0x7fac141a56c0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7fabfc00d7e0 tx=0x7fabfc00db80).stop 2021-04-09T18:28:24.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 -- 172.21.15.32:0/3591346454 shutdown_connections 2021-04-09T18:28:24.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabf80188d0 0x7fabf801add0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac140abaa0 0x7fac140abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac141a6340 0x7fac141a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 --2- 172.21.15.32:0/3591346454 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac141a5210 0x7fac141a56c0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.517 7fac1b005700 1 -- 172.21.15.32:0/3591346454 >> 172.21.15.32:0/3591346454 conn(0x7fac141a99c0 msgr2=0x7fac1417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:24.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.518 7fac1b005700 1 -- 172.21.15.32:0/3591346454 shutdown_connections 2021-04-09T18:28:24.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.518 7fac1b005700 1 -- 172.21.15.32:0/3591346454 wait complete. 2021-04-09T18:28:24.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:28:24.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.683 7f7c13abc700 1 Processor -- start 2021-04-09T18:28:24.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.683 7f7c13abc700 1 -- start start 2021-04-09T18:28:24.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.684 7f7c13abc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c0c0cffc0 0x7f7c0c0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:24.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.684 7f7c13abc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c0d7820 0x7f7c0c0d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.684 7f7c13abc700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c0d72a0 con 0x7f7c0c0d8240 2021-04-09T18:28:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.684 7f7c13abc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c0d9200 con 0x7f7c0c0cffc0 2021-04-09T18:28:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.684 7f7c13abc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c10e000 con 0x7f7c0c0d7820 2021-04-09T18:28:24.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.684 7f7c110d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c0d7820 0x7f7c0c0d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.684 7f7c118d8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c0c0cffc0 0x7f7c0c0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.684 7f7c120d9700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7c0c0d8240 0x7f7c0c0d5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:33196/0 (socket says 172.21.15.32:33196) 2021-04-09T18:28:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.684 7f7c120d9700 1 -- 172.21.15.32:0/1514914904 learned_addr learned my addr 172.21.15.32:0/1514914904 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c108d6700 1 -- 172.21.15.32:0/1514914904 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111070930 0 0) 0x7f7c04001440 con 0x7f7c0c0d8240 2021-04-09T18:28:24.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c108d6700 1 -- 172.21.15.32:0/1514914904 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf8002fe0 con 0x7f7c0c0d8240 2021-04-09T18:28:24.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c110d7700 1 -- 172.21.15.32:0/1514914904 >> v2:172.21.15.86:3300/0 conn(0x7f7c0c0cffc0 msgr2=0x7f7c0c0d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:24.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c110d7700 1 --2- 172.21.15.32:0/1514914904 >> v2:172.21.15.86:3300/0 conn(0x7f7c0c0cffc0 0x7f7c0c0d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c110d7700 1 -- 172.21.15.32:0/1514914904 >> v1:172.21.15.86:6789/0 conn(0x7f7c0c0d8240 legacy=0x7f7c0c0d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:24.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c110d7700 1 -- 172.21.15.32:0/1514914904 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c10e220 con 0x7f7c0c0d7820 2021-04-09T18:28:24.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c118d8700 1 --2- 172.21.15.32:0/1514914904 >> v2:172.21.15.86:3300/0 conn(0x7f7c0c0cffc0 0x7f7c0c0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:24.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c110d7700 1 --2- 172.21.15.32:0/1514914904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c0d7820 0x7f7c0c0d7c20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f7bfc006650 tx=0x7f7bfc009a80).ready entity=mon.2 client_cookie=cb10f82de1eaa815 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:24.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c108d6700 1 -- 172.21.15.32:0/1514914904 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bfc0009b0 con 0x7f7c0c0d7820 2021-04-09T18:28:24.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c108d6700 1 -- 172.21.15.32:0/1514914904 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bfc00c0c0 con 0x7f7c0c0d7820 2021-04-09T18:28:24.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c108d6700 1 -- 172.21.15.32:0/1514914904 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bfc00c510 con 0x7f7c0c0d7820 2021-04-09T18:28:24.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c13abc700 1 -- 172.21.15.32:0/1514914904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c0d7820 msgr2=0x7f7c0c0d7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:24.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.685 7f7c13abc700 1 --2- 172.21.15.32:0/1514914904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c0d7820 0x7f7c0c0d7c20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f7bfc006650 tx=0x7f7bfc009a80).stop 2021-04-09T18:28:24.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.686 7f7c13abc700 1 -- 172.21.15.32:0/1514914904 shutdown_connections 2021-04-09T18:28:24.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.686 7f7c13abc700 1 --2- 172.21.15.32:0/1514914904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c0d7820 0x7f7c0c0d7c20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.686 7f7c13abc700 1 --2- 172.21.15.32:0/1514914904 >> v2:172.21.15.86:3300/0 conn(0x7f7c0c0cffc0 0x7f7c0c0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.686 7f7c13abc700 1 -- 172.21.15.32:0/1514914904 >> 172.21.15.32:0/1514914904 conn(0x7f7c0c0ce5e0 msgr2=0x7f7c0c0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:24.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.686 7f7c13abc700 1 -- 172.21.15.32:0/1514914904 shutdown_connections 2021-04-09T18:28:24.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.686 7f7c13abc700 1 -- 172.21.15.32:0/1514914904 wait complete. 2021-04-09T18:28:24.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.706 7f7c13abc700 1 Processor -- start 2021-04-09T18:28:24.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.706 7f7c13abc700 1 -- start start 2021-04-09T18:28:24.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c13abc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c0c1810f0 0x7f7c0c0b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:24.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c13abc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c0c0bb160 0x7f7c0c0bb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:24.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c13abc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c1b8300 0x7f7c0c1b8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:24.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c13abc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0b3940 con 0x7f7c0c1810f0 2021-04-09T18:28:24.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c13abc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0b36c0 con 0x7f7c0c0bb160 2021-04-09T18:28:24.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c13abc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0b1490 con 0x7f7c0c1b8300 2021-04-09T18:28:24.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c118d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c0c1810f0 0x7f7c0c0b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:24.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c118d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c0c1810f0 0x7f7c0c0b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53916/0 (socket says 172.21.15.32:53916) 2021-04-09T18:28:24.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c118d8700 1 -- 172.21.15.32:0/670248206 learned_addr learned my addr 172.21.15.32:0/670248206 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:24.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c120d9700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c1b8300 0x7f7c0c1b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:24.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c110d7700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c0c0bb160 0x7f7c0c0bb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:24.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c118d8700 1 -- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c1b8300 msgr2=0x7f7c0c1b8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:24.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c118d8700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c1b8300 0x7f7c0c1b8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c118d8700 1 -- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c0c0bb160 msgr2=0x7f7c0c0bb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c118d8700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c0c0bb160 0x7f7c0c0bb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c118d8700 1 -- 172.21.15.32:0/670248206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c0babe0 con 0x7f7c0c1810f0 2021-04-09T18:28:24.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.707 7f7c118d8700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c0c1810f0 0x7f7c0c0b0e70 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 rx=0x7f7c08006f40 tx=0x7f7c0800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.708 7f7c027fc700 1 -- 172.21.15.32:0/670248206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c0800bd80 con 0x7f7c0c1810f0 2021-04-09T18:28:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.708 7f7c027fc700 1 -- 172.21.15.32:0/670248206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c08000c90 con 0x7f7c0c1810f0 2021-04-09T18:28:24.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.708 7f7c13abc700 1 -- 172.21.15.32:0/670248206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c0babe0 con 0x7f7c0c1810f0 2021-04-09T18:28:24.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.708 7f7c027fc700 1 -- 172.21.15.32:0/670248206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c08002220 con 0x7f7c0c1810f0 2021-04-09T18:28:24.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.708 7f7c13abc700 1 -- 172.21.15.32:0/670248206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c0babe0 con 0x7f7c0c1810f0 2021-04-09T18:28:24.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.708 7f7c027fc700 1 -- 172.21.15.32:0/670248206 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7c0800a000 con 0x7f7c0c1810f0 2021-04-09T18:28:24.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.708 7f7c027fc700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bf4018930 0x7f7bf401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:24.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.708 7f7c027fc700 1 -- 172.21.15.32:0/670248206 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5015+0+0 (secure 0 0 0) 0x7f7c080327b0 con 0x7f7c0c1810f0 2021-04-09T18:28:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.708 7f7c110d7700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bf4018930 0x7f7bf401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:24.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.709 7f7c110d7700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bf4018930 0x7f7bf401ae10 crc :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:24.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.728 7f7c13abc700 1 -- 172.21.15.32:0/670248206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0c0d72a0 con 0x7f7c0c1810f0 2021-04-09T18:28:24.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.732 7f7c027fc700 1 -- 172.21.15.32:0/670248206 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7c08000c90 con 0x7f7c0c1810f0 2021-04-09T18:28:25.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.944 7f7c13abc700 1 -- 172.21.15.32:0/670248206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7c0c0babe0 con 0x7f7c0c1810f0 2021-04-09T18:28:25.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.947 7f7c027fc700 1 -- 172.21.15.32:0/670248206 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7c0800a000 con 0x7f7c0c1810f0 2021-04-09T18:28:25.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 -- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bf4018930 msgr2=0x7f7bf401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:25.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bf4018930 0x7f7bf401ae10 crc :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 -- 172.21.15.32:0/670248206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c0c1810f0 msgr2=0x7f7c0c0b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:25.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c0c1810f0 0x7f7c0c0b0e70 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 rx=0x7f7c08006f40 tx=0x7f7c0800b990).stop 2021-04-09T18:28:25.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 -- 172.21.15.32:0/670248206 shutdown_connections 2021-04-09T18:28:25.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bf4018930 0x7f7bf401ae10 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0c1b8300 0x7f7c0c1b8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c0c0bb160 0x7f7c0c0bb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 --2- 172.21.15.32:0/670248206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c0c1810f0 0x7f7c0c0b0e70 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 -- 172.21.15.32:0/670248206 >> 172.21.15.32:0/670248206 conn(0x7f7c0c0a1640 msgr2=0x7f7c0c1b9120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 -- 172.21.15.32:0/670248206 shutdown_connections 2021-04-09T18:28:25.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:24.949 7f7c13abc700 1 -- 172.21.15.32:0/670248206 wait complete. 2021-04-09T18:28:25.038 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:28:25.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:28:25.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:28:25.155 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:28:25.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:25.156 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:28:25.156 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:28:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:28:25.200 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:28:25.201 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:28:25.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:25.202 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:28:25.202 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:28:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:28:25.245 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:28:25.245 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:28:25.246 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:28:25.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.314 7f44963d7700 1 Processor -- start 2021-04-09T18:28:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.314 7f44963d7700 1 -- start start 2021-04-09T18:28:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f44963d7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f44900cffa0 0x7f44900d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:25.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f44963d7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44900d7800 0x7f44900d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:25.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f44963d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44900d8220 0x7f44900d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f44963d7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44900d91e0 con 0x7f44900d7800 2021-04-09T18:28:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f44963d7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44900d7280 con 0x7f44900cffa0 2021-04-09T18:28:25.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f44963d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44900d6290 con 0x7f44900d8220 2021-04-09T18:28:25.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f4487fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44900d7800 0x7f44900d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:25.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f4487fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44900d7800 0x7f44900d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53928/0 (socket says 172.21.15.32:53928) 2021-04-09T18:28:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f4487fff700 1 -- 172.21.15.32:0/3411940822 learned_addr learned my addr 172.21.15.32:0/3411940822 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:25.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f44949f4700 1 --2- 172.21.15.32:0/3411940822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44900d8220 0x7f44900d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:25.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.315 7f448ffff700 1 --2- 172.21.15.32:0/3411940822 >> v2:172.21.15.86:3300/0 conn(0x7f44900cffa0 0x7f44900d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:25.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f4487fff700 1 -- 172.21.15.32:0/3411940822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44900d8220 msgr2=0x7f44900d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f4487fff700 1 --2- 172.21.15.32:0/3411940822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44900d8220 0x7f44900d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f4487fff700 1 -- 172.21.15.32:0/3411940822 >> v2:172.21.15.86:3300/0 conn(0x7f44900cffa0 msgr2=0x7f44900d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:25.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f4487fff700 1 --2- 172.21.15.32:0/3411940822 >> v2:172.21.15.86:3300/0 conn(0x7f44900cffa0 0x7f44900d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f4487fff700 1 -- 172.21.15.32:0/3411940822 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44900d6480 con 0x7f44900d7800 2021-04-09T18:28:25.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f4487fff700 1 --2- 172.21.15.32:0/3411940822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44900d7800 0x7f44900d7c00 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 rx=0x7f4478006700 tx=0x7f4478009b00).ready entity=mon.0 client_cookie=f8d4d131d494dfd4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:25.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f448f7fe700 1 -- 172.21.15.32:0/3411940822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f447800c0c0 con 0x7f44900d7800 2021-04-09T18:28:25.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f448f7fe700 1 -- 172.21.15.32:0/3411940822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447800c400 con 0x7f44900d7800 2021-04-09T18:28:25.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f448f7fe700 1 -- 172.21.15.32:0/3411940822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f447800c8e0 con 0x7f44900d7800 2021-04-09T18:28:25.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f44963d7700 1 -- 172.21.15.32:0/3411940822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44900d7800 msgr2=0x7f44900d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:25.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.316 7f44963d7700 1 --2- 172.21.15.32:0/3411940822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44900d7800 0x7f44900d7c00 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 rx=0x7f4478006700 tx=0x7f4478009b00).stop 2021-04-09T18:28:25.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.317 7f44963d7700 1 -- 172.21.15.32:0/3411940822 shutdown_connections 2021-04-09T18:28:25.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.317 7f44963d7700 1 --2- 172.21.15.32:0/3411940822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44900d8220 0x7f44900d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.317 7f44963d7700 1 --2- 172.21.15.32:0/3411940822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44900d7800 0x7f44900d7c00 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.317 7f44963d7700 1 --2- 172.21.15.32:0/3411940822 >> v2:172.21.15.86:3300/0 conn(0x7f44900cffa0 0x7f44900d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.317 7f44963d7700 1 -- 172.21.15.32:0/3411940822 >> 172.21.15.32:0/3411940822 conn(0x7f44900ce5c0 msgr2=0x7f44900cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:25.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.317 7f44963d7700 1 -- 172.21.15.32:0/3411940822 shutdown_connections 2021-04-09T18:28:25.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.317 7f44963d7700 1 -- 172.21.15.32:0/3411940822 wait complete. 2021-04-09T18:28:25.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.337 7f44963d7700 1 Processor -- start 2021-04-09T18:28:25.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.337 7f44963d7700 1 -- start start 2021-04-09T18:28:25.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44963d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f449008ec80 0x7f449008f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44963d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f449008c950 0x7f449008cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44963d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f449008d560 0x7f449008d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44963d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449008e700 con 0x7f449008d560 2021-04-09T18:28:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44963d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449008f6a0 con 0x7f449008ec80 2021-04-09T18:28:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44963d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44901acff0 con 0x7f449008c950 2021-04-09T18:28:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44949f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f449008d560 0x7f449008d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44949f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f449008d560 0x7f449008d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53932/0 (socket says 172.21.15.32:53932) 2021-04-09T18:28:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44949f4700 1 -- 172.21.15.32:0/1827252070 learned_addr learned my addr 172.21.15.32:0/1827252070 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f448ffff700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f449008ec80 0x7f449008f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f4487fff700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f449008c950 0x7f449008cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44949f4700 1 -- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f449008c950 msgr2=0x7f449008cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44949f4700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f449008c950 0x7f449008cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44949f4700 1 -- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f449008ec80 msgr2=0x7f449008f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:25.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44949f4700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f449008ec80 0x7f449008f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44949f4700 1 -- 172.21.15.32:0/1827252070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449008df80 con 0x7f449008d560 2021-04-09T18:28:25.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f4487fff700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f449008c950 0x7f449008cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f448ffff700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f449008ec80 0x7f449008f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44949f4700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f449008d560 0x7f449008d960 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7f4488006f40 tx=0x7f448800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f448d7fa700 1 -- 172.21.15.32:0/1827252070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44880011f0 con 0x7f449008d560 2021-04-09T18:28:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f448d7fa700 1 -- 172.21.15.32:0/1827252070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4488003030 con 0x7f449008d560 2021-04-09T18:28:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f448d7fa700 1 -- 172.21.15.32:0/1827252070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4488001660 con 0x7f449008d560 2021-04-09T18:28:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44963d7700 1 -- 172.21.15.32:0/1827252070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449008df80 con 0x7f449008d560 2021-04-09T18:28:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.338 7f44963d7700 1 -- 172.21.15.32:0/1827252070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449008df80 con 0x7f449008d560 2021-04-09T18:28:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.339 7f448d7fa700 1 -- 172.21.15.32:0/1827252070 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f44880056b0 con 0x7f449008d560 2021-04-09T18:28:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.339 7f448d7fa700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44740188c0 0x7f447401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:25.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.339 7f448d7fa700 1 -- 172.21.15.32:0/1827252070 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5015+0+0 (secure 0 0 0) 0x7f4488032b10 con 0x7f449008d560 2021-04-09T18:28:25.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.339 7f448ffff700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44740188c0 0x7f447401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:25.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.340 7f448ffff700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44740188c0 0x7f447401adc0 crc :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:25.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.359 7f44963d7700 1 -- 172.21.15.32:0/1827252070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4490182dc0 con 0x7f449008d560 2021-04-09T18:28:25.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.362 7f448d7fa700 1 -- 172.21.15.32:0/1827252070 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4488000f40 con 0x7f449008d560 2021-04-09T18:28:25.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.550 7f44963d7700 1 -- 172.21.15.32:0/1827252070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f44900d7280 con 0x7f449008d560 2021-04-09T18:28:25.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.551 7f448d7fa700 1 -- 172.21.15.32:0/1827252070 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v146) v1 ==== 81+0+25826 (secure 0 0 0) 0x7f44880056b0 con 0x7f449008d560 2021-04-09T18:28:25.632 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:25.633 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:28:25.633 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 146, 2021-04-09T18:28:25.633 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:28:25.634 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:28:25.634 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:28:22.466099", 2021-04-09T18:28:25.634 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:28:25.635 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:28:25.635 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:28:25.637 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:28:25.637 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:28:25.637 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:28:25.638 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:28:25.638 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:28:25.638 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:28:25.638 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:28:25.639 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:28:25.639 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:28:25.639 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:28:25.639 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:28:25.640 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:28:25.640 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 25, 2021-04-09T18:28:25.640 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:28:25.640 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:28:25.641 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:28:25.641 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:28:25.641 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:28:25.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.642 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:28:25.642 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:28:25.642 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:28:25.642 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:28:25.643 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:28:25.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:28:25.643 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:28:25.643 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:28:25.644 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:28:25.644 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:28:25.644 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:28:25.644 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:28:25.645 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:28:25.645 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:28:25.645 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:28:25.645 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:28:25.645 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:28:25.646 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:28:25.646 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:28:25.646 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:28:25.646 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:28:25.647 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:28:25.647 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:28:25.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.647 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:28:25.648 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:28:25.648 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:28:25.648 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:28:25.648 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:28:25.649 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:28:25.649 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:28:25.649 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:28:25.649 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:28:25.650 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:28:25.650 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:28:25.650 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:28:25.650 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:28:25.651 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:28:25.651 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:28:25.651 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:28:25.651 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:28:25.652 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:28:25.652 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:28:25.652 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:28:25.652 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:28:25.653 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:28:25.653 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:28:25.653 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:28:25.653 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:28:25.654 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:28:25.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:28:25.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.654 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:28:25.654 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:28:25.655 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:28:25.655 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:28:25.655 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:28:25.656 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:28:25.656 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:28:25.656 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:28:25.656 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:28:25.656 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:28:25.657 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:28:25.657 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:28:25.657 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:28:25.657 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:28:25.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.658 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.658 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:28:25.658 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:28:25.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.659 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:28:25.659 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:28:25.659 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:25.660 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:25.660 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:25.660 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:25.660 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:25.661 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:25.661 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:25.661 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:25.661 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:25.662 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:25.662 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:25.662 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.663 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:28:25.663 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:25.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.664 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:28:25.664 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:25.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.665 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.665 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:25.665 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.666 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:28:25.666 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:25.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.667 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.667 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:28:25.668 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:25.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.668 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.669 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:25.669 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.670 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:28:25.670 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:25.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.671 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.671 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:28:25.671 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:25.671 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.671 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.672 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:25.672 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.673 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.673 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:28:25.673 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:25.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.674 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.674 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:28:25.674 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:28:25.675 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.675 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.675 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.675 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:28:25.676 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:28:25.676 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:28:25.676 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:28:25.676 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:25.677 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:25.677 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:25.677 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:28:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:28:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:25.679 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:25.679 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:25.679 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:25.679 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:25.680 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:25.680 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 136, 2021-04-09T18:28:25.680 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:25.680 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:25.681 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:25.681 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.681 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.682 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:28:25.682 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:25.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.683 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:28:25.683 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:25.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.684 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:25.684 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.685 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.685 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:28:25.685 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:25.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.686 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.686 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:28:25.686 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:25.686 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.687 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.687 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:25.687 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.688 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.688 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:28:25.688 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:25.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:28:25.690 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:25.690 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.690 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.691 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:25.691 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.691 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.692 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:28:25.692 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:25.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.693 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:28:25.693 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:28:25.693 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.693 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.694 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:28:25.694 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:28:25.694 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:28:25.695 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:28:25.695 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:25.695 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:25.695 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:25.696 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.696 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:28:25.697 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:28:25.697 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:25.697 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:25.698 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:25.698 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:25.698 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:25.698 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:25.699 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:25.699 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:25.699 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:25.699 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:25.699 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:25.700 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.700 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.700 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:28:25.701 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:25.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.701 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.701 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:28:25.702 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:25.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.703 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:25.703 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.703 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:28:25.704 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:25.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.704 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.705 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:28:25.705 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:25.705 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.705 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.706 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:25.706 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.706 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.706 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:28:25.707 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:25.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.708 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.708 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:28:25.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:25.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.709 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.709 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:25.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.710 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:28:25.710 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:25.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.711 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:28:25.711 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:28:25.712 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.712 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.712 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:28:25.712 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:28:25.713 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:28:25.713 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:28:25.713 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:25.713 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:25.713 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:25.714 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:28:25.715 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:28:25.715 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:25.715 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:25.715 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:25.715 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:25.716 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:25.716 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:25.716 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:25.717 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:25.717 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:25.717 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:25.717 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:25.717 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.718 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:28:25.718 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:25.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.719 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.719 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:28:25.719 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:25.720 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.720 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.720 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:25.720 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.721 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.721 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:28:25.721 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:25.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.722 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.722 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:28:25.722 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:25.723 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.723 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.723 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:25.723 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.724 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.724 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:28:25.724 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:25.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.725 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:28:25.725 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:25.726 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.726 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.726 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:25.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:28:25.727 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:25.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.728 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:28:25.729 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:28:25.729 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.729 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.729 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:28:25.730 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:28:25.730 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:28:25.730 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:28:25.730 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:25.730 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:25.731 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:25.731 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.732 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:28:25.732 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:28:25.732 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:25.732 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:25.733 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:25.733 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:25.733 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:25.733 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:25.734 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:25.734 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 129, 2021-04-09T18:28:25.734 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:25.734 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:25.734 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:25.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.735 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:28:25.736 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:25.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:28:25.737 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:25.737 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.737 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.738 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:25.738 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.738 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:28:25.739 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:25.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.739 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.739 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.739 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:28:25.740 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:25.740 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.740 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.741 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:25.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:28:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.742 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:28:25.743 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:25.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.743 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.744 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:25.744 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.745 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:28:25.745 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:25.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.746 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.746 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:28:25.746 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:28:25.746 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.746 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.747 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:28:25.747 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:28:25.747 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:28:25.748 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:28:25.748 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:25.748 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:25.748 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:25.748 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.749 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:28:25.749 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:28:25.749 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:25.750 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:25.750 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:25.750 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:25.750 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:25.750 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:25.751 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:25.751 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 136, 2021-04-09T18:28:25.751 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:25.751 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:25.752 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:25.752 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.752 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.752 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:28:25.754 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:25.754 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.755 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:28:25.755 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:25.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.756 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.756 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:25.757 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.757 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.757 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:28:25.757 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:25.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.759 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:28:25.759 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:25.759 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.760 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.760 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:25.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:28:25.761 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:25.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.762 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.762 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:28:25.762 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:25.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.763 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.763 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:25.763 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.764 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.764 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:28:25.764 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:25.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.765 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:28:25.765 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:28:25.766 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.766 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.766 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:28:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:28:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:28:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:28:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:25.767 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:25.768 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:25.768 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.769 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:28:25.769 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:28:25.769 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:25.769 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:25.769 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:25.770 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:25.771 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:25.771 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:25.771 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:25.771 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.772 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.772 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:28:25.773 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:25.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.773 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:28:25.774 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:25.774 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.774 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.775 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:25.775 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.775 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.775 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:28:25.776 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:25.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.776 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.776 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.777 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:28:25.777 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:25.777 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.777 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.778 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:25.778 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.778 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.779 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:28:25.779 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:25.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.779 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.780 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:28:25.780 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:25.780 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.780 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.781 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.782 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:28:25.782 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:25.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.783 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:28:25.783 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:28:25.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.783 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.784 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:28:25.784 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:28:25.784 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:28:25.784 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:28:25.785 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:25.785 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:25.785 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:25.785 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.786 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:28:25.786 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:28:25.786 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:28:25.787 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:28:25.787 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:28:25.787 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:28:25.787 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:28:25.788 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:28:25.788 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:28:25.788 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:28:25.788 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:28:25.788 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:28:25.789 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:28:25.789 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.790 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:28:25.790 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:25.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.790 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.790 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.791 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:28:25.791 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:25.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.791 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.792 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:28:25.792 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.792 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.793 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:28:25.793 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:25.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.793 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.794 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:28:25.794 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:25.794 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.794 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.795 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:28:25.795 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.795 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.796 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:28:25.796 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:25.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.797 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:28:25.797 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:25.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.797 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.798 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:28:25.798 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:28:25.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.798 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:28:25.799 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:28:25.799 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:25.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.800 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:25.800 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:28:25.800 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:28:25.800 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.800 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.801 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:28:25.801 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:28:25.801 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:28:25.801 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:28:25.802 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:28:25.802 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:28:25.802 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:28:25.802 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:25.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.803 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:28:25.803 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:28:25.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.804 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:28:25.804 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:25.804 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:25.804 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:25.804 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:25.805 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:25.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.805 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:28:25.805 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:25.806 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:25.806 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:25.806 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:25.806 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:25.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.807 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:28:25.807 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:25.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:28:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:25.809 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:25.810 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:25.810 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:25.810 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:25.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:28:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:28:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:25.814 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:25.814 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:25.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.814 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:28:25.815 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:25.815 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:25.815 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:25.815 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:25.816 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:25.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:25.816 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:28:25.816 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:28:25.817 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:28:25.817 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:28:25.817 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:28:25.817 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:28:25.818 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.818 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:28:25.818 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:28:25.818 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:28:25.818 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:28:25.819 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:28:25.819 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:28:25.819 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/1363525210": "2021-04-10 18:28:18.211237", 2021-04-09T18:28:25.819 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/1363525210": "2021-04-10 18:28:18.211237", 2021-04-09T18:28:25.819 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/630282836": "2021-04-10 18:28:18.211222", 2021-04-09T18:28:25.820 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/630282836": "2021-04-10 18:28:18.211222", 2021-04-09T18:28:25.820 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/820321060": "2021-04-10 18:27:03.171015", 2021-04-09T18:28:25.820 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/820321060": "2021-04-10 18:27:03.171015" 2021-04-09T18:28:25.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:28:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:28:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:28:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:28:25.821 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:28:25.822 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:28:25.822 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:28:25.822 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:28:25.822 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:25.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:25.823 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:28:25.823 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:28:25.823 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:28:25.823 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:28:25.824 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:28:25.824 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:28:25.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 -- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44740188c0 msgr2=0x7f447401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:25.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44740188c0 0x7f447401adc0 crc :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 -- 172.21.15.32:0/1827252070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f449008d560 msgr2=0x7f449008d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:25.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f449008d560 0x7f449008d960 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7f4488006f40 tx=0x7f448800bd40).stop 2021-04-09T18:28:25.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 -- 172.21.15.32:0/1827252070 shutdown_connections 2021-04-09T18:28:25.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44740188c0 0x7f447401adc0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f449008d560 0x7f449008d960 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f449008c950 0x7f449008cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 --2- 172.21.15.32:0/1827252070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f449008ec80 0x7f449008f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:25.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 -- 172.21.15.32:0/1827252070 >> 172.21.15.32:0/1827252070 conn(0x7f449009abb0 msgr2=0x7f44900a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:25.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 -- 172.21.15.32:0/1827252070 shutdown_connections 2021-04-09T18:28:25.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.553 7f44963d7700 1 -- 172.21.15.32:0/1827252070 wait complete. 2021-04-09T18:28:25.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:26.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.898 7fc32f45c700 1 Processor -- start 2021-04-09T18:28:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.898 7fc32f45c700 1 -- start start 2021-04-09T18:28:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.899 7fc32f45c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc32810d010 0x7fc3280d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.899 7fc32f45c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32810d480 con 0x7fc3280d1da0 2021-04-09T18:28:26.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.899 7fc32f45c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32810e1f0 con 0x7fc3280d8490 2021-04-09T18:28:26.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.899 7fc32f45c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3280d8d30 con 0x7fc32810d010 2021-04-09T18:28:26.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.899 7fc32ca77700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc3280d1da0 0x7fc3280d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35298/0 (socket says 172.21.15.32:35298) 2021-04-09T18:28:26.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.899 7fc32ca77700 1 -- 172.21.15.32:0/2794548158 learned_addr learned my addr 172.21.15.32:0/2794548158 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:26.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32d278700 1 --2- 172.21.15.32:0/2794548158 >> v2:172.21.15.86:3300/0 conn(0x7fc32810d010 0x7fc3280d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629224362 0 0) 0x7fc3180009b0 con 0x7fc3280d1da0 2021-04-09T18:28:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc314002fe0 con 0x7fc3280d1da0 2021-04-09T18:28:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027336301 0 0) 0x7fc320001400 con 0x7fc3280d8490 2021-04-09T18:28:26.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc314003210 con 0x7fc3280d8490 2021-04-09T18:28:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 279673493 0 0) 0x7fc318001010 con 0x7fc3280d1da0 2021-04-09T18:28:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 >> v2:172.21.15.86:3300/0 conn(0x7fc32810d010 msgr2=0x7fc3280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:26.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 --2- 172.21.15.32:0/2794548158 >> v2:172.21.15.86:3300/0 conn(0x7fc32810d010 0x7fc3280d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 >> v1:172.21.15.86:6789/0 conn(0x7fc3280d8490 legacy=0x7fc32810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3280d9220 con 0x7fc3280d1da0 2021-04-09T18:28:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fc318001590 con 0x7fc3280d1da0 2021-04-09T18:28:26.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 >> v1:172.21.15.32:6789/0 conn(0x7fc3280d1da0 legacy=0x7fc3280d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc314003b90 0x7fc314005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc314006840 0x7fc314008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc314009640 0x7fc31400bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140503c0 con 0x7fc314006840 2021-04-09T18:28:26.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140505a0 con 0x7fc314009640 2021-04-09T18:28:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32ca77700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc314006840 0x7fc314008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140507c0 con 0x7fc314003b90 2021-04-09T18:28:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32d278700 1 --2- 172.21.15.32:0/2794548158 >> v2:172.21.15.86:3300/0 conn(0x7fc32810d010 0x7fc3280d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:26.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32da79700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc314009640 0x7fc31400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32d278700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc314003b90 0x7fc314005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32ca77700 1 -- 172.21.15.32:0/2794548158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc314003b90 msgr2=0x7fc314005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32ca77700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc314003b90 0x7fc314005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32ca77700 1 -- 172.21.15.32:0/2794548158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc314009640 msgr2=0x7fc31400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:26.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32ca77700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc314009640 0x7fc31400bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32ca77700 1 -- 172.21.15.32:0/2794548158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc314050a60 con 0x7fc314006840 2021-04-09T18:28:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32da79700 1 -- 172.21.15.32:0/2794548158 reap_dead start 2021-04-09T18:28:26.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.900 7fc32d278700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc314003b90 0x7fc314005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.901 7fc32ca77700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc314006840 0x7fc314008d40 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7fc318001360 tx=0x7fc318007d40).ready entity=mon.0 client_cookie=e168a1b9b3046578 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.901 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc318001010 con 0x7fc314006840 2021-04-09T18:28:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.901 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318000940 con 0x7fc314006840 2021-04-09T18:28:26.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.901 7fc31ffff700 1 -- 172.21.15.32:0/2794548158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc31800a710 con 0x7fc314006840 2021-04-09T18:28:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.901 7fc32f45c700 1 -- 172.21.15.32:0/2794548158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc314006840 msgr2=0x7fc314008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.901 7fc32f45c700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc314006840 0x7fc314008d40 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7fc318001360 tx=0x7fc318007d40).stop 2021-04-09T18:28:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.901 7fc32f45c700 1 -- 172.21.15.32:0/2794548158 shutdown_connections 2021-04-09T18:28:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.901 7fc32f45c700 1 --2- 172.21.15.32:0/2794548158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc314006840 0x7fc314008d40 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.901 7fc32f45c700 1 -- 172.21.15.32:0/2794548158 >> 172.21.15.32:0/2794548158 conn(0x7fc3280ce5f0 msgr2=0x7fc3280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.902 7fc32f45c700 1 -- 172.21.15.32:0/2794548158 shutdown_connections 2021-04-09T18:28:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.902 7fc32f45c700 1 -- 172.21.15.32:0/2794548158 wait complete. 2021-04-09T18:28:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.922 7fc32f45c700 1 Processor -- start 2021-04-09T18:28:26.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.922 7fc32f45c700 1 -- start start 2021-04-09T18:28:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32f45c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3281aca20 0x7fc3281ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32f45c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32818bd40 0x7fc32818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32f45c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32808fba0 0x7fc32808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32f45c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32818c7e0 con 0x7fc32808fba0 2021-04-09T18:28:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32f45c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32818ca30 con 0x7fc32818bd40 2021-04-09T18:28:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32f45c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3280905f0 con 0x7fc3281aca20 2021-04-09T18:28:26.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32da79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32808fba0 0x7fc32808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32d278700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3281aca20 0x7fc3281ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32da79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32808fba0 0x7fc32808ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53954/0 (socket says 172.21.15.32:53954) 2021-04-09T18:28:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32da79700 1 -- 172.21.15.32:0/3825730725 learned_addr learned my addr 172.21.15.32:0/3825730725 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:26.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32ca77700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32818bd40 0x7fc32818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32da79700 1 -- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3281aca20 msgr2=0x7fc3281ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32da79700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3281aca20 0x7fc3281ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32da79700 1 -- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32818bd40 msgr2=0x7fc32818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32da79700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32818bd40 0x7fc32818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32da79700 1 -- 172.21.15.32:0/3825730725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3280b4300 con 0x7fc32808fba0 2021-04-09T18:28:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32d278700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3281aca20 0x7fc3281ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32da79700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32808fba0 0x7fc32808ffa0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 rx=0x7fc32000bbb0 tx=0x7fc32000def0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc31dffb700 1 -- 172.21.15.32:0/3825730725 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc32000b030 con 0x7fc32808fba0 2021-04-09T18:28:26.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc31dffb700 1 -- 172.21.15.32:0/3825730725 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32000e3a0 con 0x7fc32808fba0 2021-04-09T18:28:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3280b4300 con 0x7fc32808fba0 2021-04-09T18:28:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.923 7fc31dffb700 1 -- 172.21.15.32:0/3825730725 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc32000e900 con 0x7fc32808fba0 2021-04-09T18:28:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.924 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3280b4300 con 0x7fc32808fba0 2021-04-09T18:28:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.924 7fc31dffb700 1 -- 172.21.15.32:0/3825730725 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc3200011f0 con 0x7fc32808fba0 2021-04-09T18:28:26.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.924 7fc31dffb700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc308018930 0x7fc30801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.924 7fc31dffb700 1 -- 172.21.15.32:0/3825730725 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5015+0+0 (secure 0 0 0) 0x7fc320034d90 con 0x7fc32808fba0 2021-04-09T18:28:26.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.924 7fc32d278700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc308018930 0x7fc30801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.925 7fc32d278700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc308018930 0x7fc30801ae10 crc :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.943 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc328182d70 con 0x7fc32808fba0 2021-04-09T18:28:26.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:25.947 7fc31dffb700 1 -- 172.21.15.32:0/3825730725 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc3200338f0 con 0x7fc32808fba0 2021-04-09T18:28:26.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.138 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc32818c760 con 0x7fc32808fba0 2021-04-09T18:28:26.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.138 7fc31dffb700 1 -- 172.21.15.32:0/3825730725 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+1859 (secure 0 0 0) 0x7fc320021f10 con 0x7fc32808fba0 2021-04-09T18:28:26.220 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 293 2021-04-09T18:28:26.221 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:26.221 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":293,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":10086,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1476577937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1476577937},{"type":"v1","addr":"172.21.15.32:6835","nonce":1476577937}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":293},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":58070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2514441046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2514441046},{"type":"v1","addr":"172.21.15.86:6836","nonce":2514441046}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":293}],"filesystems":[]} 2021-04-09T18:28:26.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc308018930 msgr2=0x7fc30801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc308018930 0x7fc30801ae10 crc :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32808fba0 msgr2=0x7fc32808ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32808fba0 0x7fc32808ffa0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 rx=0x7fc32000bbb0 tx=0x7fc32000def0).stop 2021-04-09T18:28:26.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 shutdown_connections 2021-04-09T18:28:26.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc308018930 0x7fc30801ae10 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc32808fba0 0x7fc32808ffa0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32818bd40 0x7fc32818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 --2- 172.21.15.32:0/3825730725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3281aca20 0x7fc3281ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 >> 172.21.15.32:0/3825730725 conn(0x7fc32819b460 msgr2=0x7fc32819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:26.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 shutdown_connections 2021-04-09T18:28:26.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.141 7fc32f45c700 1 -- 172.21.15.32:0/3825730725 wait complete. 2021-04-09T18:28:26.232 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:26.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.300 7f2d5ddc6700 1 Processor -- start 2021-04-09T18:28:26.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.300 7f2d5ddc6700 1 -- start start 2021-04-09T18:28:26.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d5ddc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d5810da00 0x7f2d5810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d5ddc6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5810d480 con 0x7f2d5810d010 2021-04-09T18:28:26.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d5ddc6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d580d1f70 con 0x7f2d580d7ca0 2021-04-09T18:28:26.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d5ddc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d580d9220 con 0x7f2d5810da00 2021-04-09T18:28:26.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d577fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2d5810d010 0x7f2d580d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35316/0 (socket says 172.21.15.32:35316) 2021-04-09T18:28:26.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d577fe700 1 -- 172.21.15.32:0/1844263896 learned_addr learned my addr 172.21.15.32:0/1844263896 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:26.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195994194 0 0) 0x7f2d48000ce0 con 0x7f2d5810d010 2021-04-09T18:28:26.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d57fff700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d5810da00 0x7f2d5810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d3c002fe0 con 0x7f2d5810d010 2021-04-09T18:28:26.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708629409 0 0) 0x7f2d44000c60 con 0x7f2d580d7ca0 2021-04-09T18:28:26.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d3c003210 con 0x7f2d580d7ca0 2021-04-09T18:28:26.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4261224780 0 0) 0x7f2d480010e0 con 0x7f2d5810d010 2021-04-09T18:28:26.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d5810da00 msgr2=0x7f2d5810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:26.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d5810da00 0x7f2d5810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 >> v1:172.21.15.86:6789/0 conn(0x7f2d580d7ca0 legacy=0x7f2d580d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5810e420 con 0x7f2d5810d010 2021-04-09T18:28:26.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.301 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f2d48001600 con 0x7f2d5810d010 2021-04-09T18:28:26.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 >> v1:172.21.15.32:6789/0 conn(0x7f2d5810d010 legacy=0x7f2d580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d3c003b90 0x7f2d3c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d3c0067a0 0x7f2d3c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d3c0095a0 0x7f2d3c00ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c0502f0 con 0x7f2d3c0067a0 2021-04-09T18:28:26.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c0504d0 con 0x7f2d3c003b90 2021-04-09T18:28:26.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c0506f0 con 0x7f2d3c0095a0 2021-04-09T18:28:26.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d56ffd700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d3c0067a0 0x7f2d3c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d577fe700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d3c003b90 0x7f2d3c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d57fff700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d3c0095a0 0x7f2d3c00ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d56ffd700 1 -- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d3c0095a0 msgr2=0x7f2d3c00ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:26.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d56ffd700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d3c0095a0 0x7f2d3c00ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d56ffd700 1 -- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d3c003b90 msgr2=0x7f2d3c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:26.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d56ffd700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d3c003b90 0x7f2d3c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d56ffd700 1 -- 172.21.15.32:0/1844263896 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c050990 con 0x7f2d3c0067a0 2021-04-09T18:28:26.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d56ffd700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d3c0067a0 0x7f2d3c008ca0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 rx=0x7f2d44001040 tx=0x7f2d44009aa0).ready entity=mon.0 client_cookie=b7431674cde525fe server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:26.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d440066e0 con 0x7f2d3c0067a0 2021-04-09T18:28:26.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4400c0c0 con 0x7f2d3c0067a0 2021-04-09T18:28:26.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d567fc700 1 -- 172.21.15.32:0/1844263896 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d4400c560 con 0x7f2d3c0067a0 2021-04-09T18:28:26.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d5ddc6700 1 -- 172.21.15.32:0/1844263896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d3c0067a0 msgr2=0x7f2d3c008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d5ddc6700 1 --2- 172.21.15.32:0/1844263896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d3c0067a0 0x7f2d3c008ca0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 rx=0x7f2d44001040 tx=0x7f2d44009aa0).stop 2021-04-09T18:28:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.302 7f2d57fff700 1 -- 172.21.15.32:0/1844263896 reap_dead start 2021-04-09T18:28:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.303 7f2d5ddc6700 1 -- 172.21.15.32:0/1844263896 shutdown_connections 2021-04-09T18:28:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.303 7f2d5ddc6700 1 -- 172.21.15.32:0/1844263896 >> 172.21.15.32:0/1844263896 conn(0x7f2d580ce5f0 msgr2=0x7f2d580d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.303 7f2d5ddc6700 1 -- 172.21.15.32:0/1844263896 shutdown_connections 2021-04-09T18:28:26.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.303 7f2d5ddc6700 1 -- 172.21.15.32:0/1844263896 wait complete. 2021-04-09T18:28:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.323 7f2d5ddc6700 1 Processor -- start 2021-04-09T18:28:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.323 7f2d5ddc6700 1 -- start start 2021-04-09T18:28:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d5ddc6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d5809a670 0x7f2d58093610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d5ddc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d581a9fb0 0x7f2d581aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d5ddc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580a3ab0 0x7f2d580a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d5ddc6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d58094060 con 0x7f2d581a9fb0 2021-04-09T18:28:26.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d5ddc6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d58094710 con 0x7f2d5809a670 2021-04-09T18:28:26.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d5ddc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d58093c30 con 0x7f2d580a3ab0 2021-04-09T18:28:26.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d56ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d581a9fb0 0x7f2d581aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d56ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d581a9fb0 0x7f2d581aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53972/0 (socket says 172.21.15.32:53972) 2021-04-09T18:28:26.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d56ffd700 1 -- 172.21.15.32:0/1656295601 learned_addr learned my addr 172.21.15.32:0/1656295601 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:26.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d577fe700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d5809a670 0x7f2d58093610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d57fff700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580a3ab0 0x7f2d580a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d56ffd700 1 -- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580a3ab0 msgr2=0x7f2d580a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d56ffd700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580a3ab0 0x7f2d580a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d56ffd700 1 -- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d5809a670 msgr2=0x7f2d58093610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d56ffd700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d5809a670 0x7f2d58093610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d56ffd700 1 -- 172.21.15.32:0/1656295601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d581aa9d0 con 0x7f2d581a9fb0 2021-04-09T18:28:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d577fe700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d5809a670 0x7f2d58093610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.324 7f2d56ffd700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d581a9fb0 0x7f2d581aa3b0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7f2d44006e90 tx=0x7f2d4400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.325 7f2d4ffff700 1 -- 172.21.15.32:0/1656295601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d44009020 con 0x7f2d581a9fb0 2021-04-09T18:28:26.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.325 7f2d4ffff700 1 -- 172.21.15.32:0/1656295601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4400bd10 con 0x7f2d581a9fb0 2021-04-09T18:28:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.325 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d581aa9d0 con 0x7f2d581a9fb0 2021-04-09T18:28:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.325 7f2d4ffff700 1 -- 172.21.15.32:0/1656295601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d44006220 con 0x7f2d581a9fb0 2021-04-09T18:28:26.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.325 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d581aa9d0 con 0x7f2d581a9fb0 2021-04-09T18:28:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.326 7f2d4ffff700 1 -- 172.21.15.32:0/1656295601 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2d44006030 con 0x7f2d581a9fb0 2021-04-09T18:28:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.326 7f2d4ffff700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d340188c0 0x7f2d3401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.326 7f2d4ffff700 1 -- 172.21.15.32:0/1656295601 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5015+0+0 (secure 0 0 0) 0x7f2d44032550 con 0x7f2d581a9fb0 2021-04-09T18:28:26.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.326 7f2d577fe700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d340188c0 0x7f2d3401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.327 7f2d577fe700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d340188c0 0x7f2d3401adc0 crc :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:26.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.345 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5810d480 con 0x7f2d581a9fb0 2021-04-09T18:28:26.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.348 7f2d4ffff700 1 -- 172.21.15.32:0/1656295601 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2d44000be0 con 0x7f2d581a9fb0 2021-04-09T18:28:26.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.541 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d5810d480 con 0x7f2d581a9fb0 2021-04-09T18:28:26.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.541 7f2d4ffff700 1 -- 172.21.15.32:0/1656295601 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+1859 (secure 0 0 0) 0x7f2d44006030 con 0x7f2d581a9fb0 2021-04-09T18:28:26.622 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 293 2021-04-09T18:28:26.623 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:26.623 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":293,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2127406423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2127406423},{"type":"v1","addr":"172.21.15.32:6834","nonce":2127406423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":285},{"gid":10086,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1476577937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1476577937},{"type":"v1","addr":"172.21.15.32:6835","nonce":1476577937}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":293},{"gid":57845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3981849145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3981849145},{"type":"v1","addr":"172.21.15.86:6837","nonce":3981849145}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":272},{"gid":58070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2514441046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2514441046},{"type":"v1","addr":"172.21.15.86:6836","nonce":2514441046}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":293}],"filesystems":[]} 2021-04-09T18:28:26.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d340188c0 msgr2=0x7f2d3401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d340188c0 0x7f2d3401adc0 crc :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d581a9fb0 msgr2=0x7f2d581aa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d581a9fb0 0x7f2d581aa3b0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7f2d44006e90 tx=0x7f2d4400b8e0).stop 2021-04-09T18:28:26.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 shutdown_connections 2021-04-09T18:28:26.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d340188c0 0x7f2d3401adc0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d581a9fb0 0x7f2d581aa3b0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580a3ab0 0x7f2d580a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 --2- 172.21.15.32:0/1656295601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d5809a670 0x7f2d58093610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 >> 172.21.15.32:0/1656295601 conn(0x7f2d581a0db0 msgr2=0x7f2d581a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:26.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 shutdown_connections 2021-04-09T18:28:26.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.544 7f2d5ddc6700 1 -- 172.21.15.32:0/1656295601 wait complete. 2021-04-09T18:28:26.634 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:28:26.635 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:28:26.635 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:28:26.636 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:28:26.636 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:28:26.636 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:28:26.637 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:28:26.637 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:28:26.637 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:28:26.637 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:28:26.638 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:28:26.638 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:28:26.638 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:28:26.639 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:28:26.639 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:28:26.639 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:28:26.644 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:28:26.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:28:26.646 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:28:26.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:28:26.648 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:28:26.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:28:26.650 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:28:26.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:28:26.689 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:28:26.690 DEBUG:teuthology.parallel:result is None 2021-04-09T18:28:26.691 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:28:26.692 DEBUG:teuthology.parallel:result is None 2021-04-09T18:28:26.693 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:28:26.693 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:28:26.693 DEBUG:teuthology.parallel:result is None 2021-04-09T18:28:26.694 DEBUG:teuthology.parallel:result is None 2021-04-09T18:28:26.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:28:26.838 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:28:26.839 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:28:26.854 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:28:26.854 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:28:26.855 INFO:tasks.ceph.mds.d.smithi086.stderr:2021-04-09 18:28:26.846 7fa683222700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2021-04-09T18:28:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.769 7ff947175700 1 Processor -- start 2021-04-09T18:28:26.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.769 7ff947175700 1 -- start start 2021-04-09T18:28:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff947175700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9400d6c80 0x7ff9400d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff947175700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9400d2440 0x7ff94010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff947175700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9400d7280 con 0x7ff9400cffa0 2021-04-09T18:28:26.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff947175700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9400d7080 con 0x7ff9400d2440 2021-04-09T18:28:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff947175700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9400d91e0 con 0x7ff9400d6c80 2021-04-09T18:28:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff945792700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff9400d2440 0x7ff94010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff944f91700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff9400cffa0 0x7ff9400d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35360/0 (socket says 172.21.15.32:35360) 2021-04-09T18:28:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff944f91700 1 -- 172.21.15.32:0/846224889 learned_addr learned my addr 172.21.15.32:0/846224889 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:26.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff93ffff700 1 --2- 172.21.15.32:0/846224889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9400d6c80 0x7ff9400d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff945792700 1 -- 172.21.15.32:0/846224889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9400d6c80 msgr2=0x7ff9400d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:26.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff945792700 1 --2- 172.21.15.32:0/846224889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9400d6c80 0x7ff9400d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff945792700 1 -- 172.21.15.32:0/846224889 >> v1:172.21.15.32:6789/0 conn(0x7ff9400cffa0 legacy=0x7ff9400d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff945792700 1 -- 172.21.15.32:0/846224889 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9400d2840 con 0x7ff9400d2440 2021-04-09T18:28:26.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.770 7ff93f7fe700 1 -- 172.21.15.32:0/846224889 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336231357 0 0) 0x7ff928000ce0 con 0x7ff9400cffa0 2021-04-09T18:28:26.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.771 7ff945792700 1 --2- 172.21.15.32:0/846224889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9400d2440 0x7ff94010cfd0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7ff938006e30 tx=0x7ff938009de0).ready entity=mon.0 client_cookie=7d6ee76402799cd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:26.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.771 7ff93f7fe700 1 -- 172.21.15.32:0/846224889 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff93800b300 con 0x7ff9400d2440 2021-04-09T18:28:26.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.771 7ff93f7fe700 1 -- 172.21.15.32:0/846224889 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93800b680 con 0x7ff9400d2440 2021-04-09T18:28:26.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.771 7ff93f7fe700 1 -- 172.21.15.32:0/846224889 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff93800bb60 con 0x7ff9400d2440 2021-04-09T18:28:26.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.771 7ff947175700 1 -- 172.21.15.32:0/846224889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9400d2440 msgr2=0x7ff94010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.771 7ff947175700 1 --2- 172.21.15.32:0/846224889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9400d2440 0x7ff94010cfd0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7ff938006e30 tx=0x7ff938009de0).stop 2021-04-09T18:28:26.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.772 7ff947175700 1 -- 172.21.15.32:0/846224889 shutdown_connections 2021-04-09T18:28:26.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.772 7ff947175700 1 --2- 172.21.15.32:0/846224889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff9400d2440 0x7ff94010cfd0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.772 7ff947175700 1 --2- 172.21.15.32:0/846224889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9400d6c80 0x7ff9400d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.772 7ff947175700 1 -- 172.21.15.32:0/846224889 >> 172.21.15.32:0/846224889 conn(0x7ff9400ce5c0 msgr2=0x7ff9400cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:26.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.772 7ff947175700 1 -- 172.21.15.32:0/846224889 shutdown_connections 2021-04-09T18:28:26.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.772 7ff947175700 1 -- 172.21.15.32:0/846224889 wait complete. 2021-04-09T18:28:26.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.792 7ff947175700 1 Processor -- start 2021-04-09T18:28:26.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.792 7ff947175700 1 -- start start 2021-04-09T18:28:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.792 7ff947175700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9401b8660 0x7ff9401b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.792 7ff947175700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94008c720 0x7ff94008cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.792 7ff947175700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9400ab9c0 0x7ff9400abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.792 7ff947175700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94008c1a0 con 0x7ff94008c720 2021-04-09T18:28:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.792 7ff947175700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401819b0 con 0x7ff9400ab9c0 2021-04-09T18:28:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.792 7ff947175700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94008d140 con 0x7ff9401b8660 2021-04-09T18:28:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94008c720 0x7ff94008cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94008c720 0x7ff94008cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54008/0 (socket says 172.21.15.32:54008) 2021-04-09T18:28:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93ffff700 1 -- 172.21.15.32:0/1364694219 learned_addr learned my addr 172.21.15.32:0/1364694219 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff945792700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9400ab9c0 0x7ff9400abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff944f91700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9401b8660 0x7ff9401b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93ffff700 1 -- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9401b8660 msgr2=0x7ff9401b8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93ffff700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9401b8660 0x7ff9401b8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93ffff700 1 -- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9400ab9c0 msgr2=0x7ff9400abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93ffff700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9400ab9c0 0x7ff9400abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93ffff700 1 -- 172.21.15.32:0/1364694219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9400ac3e0 con 0x7ff94008c720 2021-04-09T18:28:26.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93ffff700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94008c720 0x7ff94008cb20 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 rx=0x7ff930006f60 tx=0x7ff93000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:26.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93d7fa700 1 -- 172.21.15.32:0/1364694219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff93000bd60 con 0x7ff94008c720 2021-04-09T18:28:26.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93d7fa700 1 -- 172.21.15.32:0/1364694219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff930000c90 con 0x7ff94008c720 2021-04-09T18:28:26.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff947175700 1 -- 172.21.15.32:0/1364694219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9400ac3e0 con 0x7ff94008c720 2021-04-09T18:28:26.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff93d7fa700 1 -- 172.21.15.32:0/1364694219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff930002220 con 0x7ff94008c720 2021-04-09T18:28:26.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.793 7ff947175700 1 -- 172.21.15.32:0/1364694219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9400ac3e0 con 0x7ff94008c720 2021-04-09T18:28:26.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.794 7ff93d7fa700 1 -- 172.21.15.32:0/1364694219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff9300052e0 con 0x7ff94008c720 2021-04-09T18:28:26.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.794 7ff93d7fa700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff924018930 0x7ff92401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:26.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.794 7ff93d7fa700 1 -- 172.21.15.32:0/1364694219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5015+0+0 (secure 0 0 0) 0x7ff930032900 con 0x7ff94008c720 2021-04-09T18:28:26.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.794 7ff944f91700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff924018930 0x7ff92401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:26.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.794 7ff944f91700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff924018930 0x7ff92401ae10 crc :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:26.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.813 7ff947175700 1 -- 172.21.15.32:0/1364694219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff940181940 con 0x7ff94008c720 2021-04-09T18:28:26.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:26.816 7ff93d7fa700 1 -- 172.21.15.32:0/1364694219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff930000c90 con 0x7ff94008c720 2021-04-09T18:28:27.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.018 7ff947175700 1 -- 172.21.15.32:0/1364694219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7ff940181940 con 0x7ff94008c720 2021-04-09T18:28:27.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.485 7ff93d7fa700 1 -- 172.21.15.32:0/1364694219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v147) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff9300052e0 con 0x7ff94008c720 2021-04-09T18:28:27.612 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:28:27.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.534 7ff947175700 1 -- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff924018930 msgr2=0x7ff92401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:27.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.534 7ff947175700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff924018930 0x7ff92401ae10 crc :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.534 7ff947175700 1 -- 172.21.15.32:0/1364694219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94008c720 msgr2=0x7ff94008cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:27.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.534 7ff947175700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94008c720 0x7ff94008cb20 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 rx=0x7ff930006f60 tx=0x7ff93000b970).stop 2021-04-09T18:28:27.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.535 7ff947175700 1 -- 172.21.15.32:0/1364694219 shutdown_connections 2021-04-09T18:28:27.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.535 7ff947175700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff924018930 0x7ff92401ae10 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.535 7ff947175700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff94008c720 0x7ff94008cb20 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.535 7ff947175700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9400ab9c0 0x7ff9400abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.535 7ff947175700 1 --2- 172.21.15.32:0/1364694219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9401b8660 0x7ff9401b8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.535 7ff947175700 1 -- 172.21.15.32:0/1364694219 >> 172.21.15.32:0/1364694219 conn(0x7ff9401897f0 msgr2=0x7ff9401b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:27.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.535 7ff947175700 1 -- 172.21.15.32:0/1364694219 shutdown_connections 2021-04-09T18:28:27.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.535 7ff947175700 1 -- 172.21.15.32:0/1364694219 wait complete. 2021-04-09T18:28:27.625 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:28:27.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:28:27.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.695 7f3a5de32700 1 Processor -- start 2021-04-09T18:28:27.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.695 7f3a5de32700 1 -- start start 2021-04-09T18:28:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.695 7f3a5de32700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a580cffa0 0x7f3a580d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a5de32700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a580d6c80 0x7f3a5810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a5de32700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a580d7280 con 0x7f3a580d7800 2021-04-09T18:28:27.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a5de32700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a580d7080 con 0x7f3a580d6c80 2021-04-09T18:28:27.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a5de32700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a580d8580 con 0x7f3a580cffa0 2021-04-09T18:28:27.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a57fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a580d6c80 0x7f3a5810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:27.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a57fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3a580d6c80 0x7f3a5810cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54018/0 (socket says 172.21.15.32:54018) 2021-04-09T18:28:27.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a57fff700 1 -- 172.21.15.32:0/2410616650 learned_addr learned my addr 172.21.15.32:0/2410616650 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:27.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a577fe700 1 --2- 172.21.15.32:0/2410616650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a580cffa0 0x7f3a580d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:27.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a57fff700 1 -- 172.21.15.32:0/2410616650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a580cffa0 msgr2=0x7f3a580d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:27.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a57fff700 1 --2- 172.21.15.32:0/2410616650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a580cffa0 0x7f3a580d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a57fff700 1 -- 172.21.15.32:0/2410616650 >> v1:172.21.15.86:6789/0 conn(0x7f3a580d7800 legacy=0x7f3a580d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:27.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a57fff700 1 -- 172.21.15.32:0/2410616650 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a580d91e0 con 0x7f3a580d6c80 2021-04-09T18:28:27.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a57fff700 1 --2- 172.21.15.32:0/2410616650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a580d6c80 0x7f3a5810cfd0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 rx=0x7f3a50006f00 tx=0x7f3a50009e90).ready entity=mon.0 client_cookie=2b504bc69bfd6c5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:27.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a567fc700 1 -- 172.21.15.32:0/2410616650 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a5000b300 con 0x7f3a580d6c80 2021-04-09T18:28:27.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a567fc700 1 -- 172.21.15.32:0/2410616650 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5000b640 con 0x7f3a580d6c80 2021-04-09T18:28:27.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a567fc700 1 -- 172.21.15.32:0/2410616650 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a5000bb20 con 0x7f3a580d6c80 2021-04-09T18:28:27.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a5de32700 1 -- 172.21.15.32:0/2410616650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a580d6c80 msgr2=0x7f3a5810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:27.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.696 7f3a5de32700 1 --2- 172.21.15.32:0/2410616650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a580d6c80 0x7f3a5810cfd0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 rx=0x7f3a50006f00 tx=0x7f3a50009e90).stop 2021-04-09T18:28:27.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.697 7f3a5de32700 1 -- 172.21.15.32:0/2410616650 shutdown_connections 2021-04-09T18:28:27.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.697 7f3a5de32700 1 --2- 172.21.15.32:0/2410616650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a580d6c80 0x7f3a5810cfd0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.697 7f3a5de32700 1 --2- 172.21.15.32:0/2410616650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a580cffa0 0x7f3a580d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.697 7f3a5de32700 1 -- 172.21.15.32:0/2410616650 >> 172.21.15.32:0/2410616650 conn(0x7f3a580ce5c0 msgr2=0x7f3a580cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:27.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.697 7f3a5de32700 1 -- 172.21.15.32:0/2410616650 shutdown_connections 2021-04-09T18:28:27.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.697 7f3a5de32700 1 -- 172.21.15.32:0/2410616650 wait complete. 2021-04-09T18:28:27.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.717 7f3a5de32700 1 Processor -- start 2021-04-09T18:28:27.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.717 7f3a5de32700 1 -- start start 2021-04-09T18:28:27.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a5de32700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a58198240 0x7f3a581acf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:27.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a5de32700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a58184e90 0x7f3a581abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:27.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a5de32700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a581ac510 0x7f3a58183ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:27.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a5de32700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a581aca20 con 0x7f3a581ac510 2021-04-09T18:28:27.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a5de32700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a58188e60 con 0x7f3a58198240 2021-04-09T18:28:27.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a5de32700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a58185290 con 0x7f3a58184e90 2021-04-09T18:28:27.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a57fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a581ac510 0x7f3a58183ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:27.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a57fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a581ac510 0x7f3a58183ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54026/0 (socket says 172.21.15.32:54026) 2021-04-09T18:28:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a57fff700 1 -- 172.21.15.32:0/478296990 learned_addr learned my addr 172.21.15.32:0/478296990 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a577fe700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a58198240 0x7f3a581acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:27.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a56ffd700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a58184e90 0x7f3a581abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:27.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a57fff700 1 -- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a58184e90 msgr2=0x7f3a581abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:27.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a57fff700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a58184e90 0x7f3a581abef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a57fff700 1 -- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a58198240 msgr2=0x7f3a581acf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a57fff700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a58198240 0x7f3a581acf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.718 7f3a57fff700 1 -- 172.21.15.32:0/478296990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a581854b0 con 0x7f3a581ac510 2021-04-09T18:28:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.719 7f3a577fe700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a58198240 0x7f3a581acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.719 7f3a57fff700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a581ac510 0x7f3a58183ff0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7f3a500099f0 tx=0x7f3a5000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.719 7f3a3ffff700 1 -- 172.21.15.32:0/478296990 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a50003030 con 0x7f3a581ac510 2021-04-09T18:28:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.719 7f3a3ffff700 1 -- 172.21.15.32:0/478296990 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a50000cc0 con 0x7f3a581ac510 2021-04-09T18:28:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.719 7f3a5de32700 1 -- 172.21.15.32:0/478296990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a581854b0 con 0x7f3a581ac510 2021-04-09T18:28:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.719 7f3a3ffff700 1 -- 172.21.15.32:0/478296990 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a50001530 con 0x7f3a581ac510 2021-04-09T18:28:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.719 7f3a5de32700 1 -- 172.21.15.32:0/478296990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a581854b0 con 0x7f3a581ac510 2021-04-09T18:28:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.719 7f3a3ffff700 1 -- 172.21.15.32:0/478296990 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3a50005820 con 0x7f3a581ac510 2021-04-09T18:28:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.719 7f3a3ffff700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a38018930 0x7f3a3801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.720 7f3a3ffff700 1 -- 172.21.15.32:0/478296990 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f3a50032a70 con 0x7f3a581ac510 2021-04-09T18:28:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.720 7f3a577fe700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a38018930 0x7f3a3801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.720 7f3a577fe700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a38018930 0x7f3a3801ae10 crc :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:27.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.740 7f3a5de32700 1 -- 172.21.15.32:0/478296990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a58184bf0 con 0x7f3a581ac510 2021-04-09T18:28:27.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.743 7f3a3ffff700 1 -- 172.21.15.32:0/478296990 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3a5000be20 con 0x7f3a581ac510 2021-04-09T18:28:28.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.927 7f3a5de32700 1 -- 172.21.15.32:0/478296990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3a58184bf0 con 0x7f3a581ac510 2021-04-09T18:28:28.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.927 7f3a3ffff700 1 -- 172.21.15.32:0/478296990 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f3a5001fae0 con 0x7f3a581ac510 2021-04-09T18:28:28.008 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:28.009 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:28:28.009 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:28:28.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.009 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:28:28.010 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:28:28.010 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.010 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:28:28.010 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.010 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.011 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.011 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.011 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.012 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:28:28.012 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:28:28.012 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.012 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:28:28.012 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.013 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.013 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.014 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:28:28.014 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:28:28.014 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.014 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:28:28.015 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.015 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.015 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.015 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.016 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:28:28.016 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:28:28.016 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.017 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:28:28.017 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.017 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.017 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.017 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.018 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:28:28.018 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:28:28.019 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.019 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.019 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.019 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.019 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.020 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:28:28.020 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:28:28.021 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.021 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.021 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.021 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.022 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:28:28.022 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:28:28.023 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.023 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.023 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.023 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.024 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.024 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.024 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:28:28.024 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:28:28.025 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.025 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.025 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.025 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.026 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.026 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:28:28.026 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:28:28.027 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.027 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.028 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:28:28.028 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:28:28.029 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.029 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.029 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.029 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.030 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:28:28.030 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:28:28.031 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.031 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.031 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.031 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.032 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.032 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:28:28.032 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:28:28.033 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.033 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.033 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.033 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.034 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:28:28.034 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:28:28.035 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.035 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:28:28.035 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:28:28.035 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:28:28.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.036 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:28:28.036 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:28:28.037 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.037 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:28:28.037 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:28:28.037 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:28:28.038 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.038 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:28:28.038 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:28:28.039 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.039 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:28:28.039 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:28:28.039 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:28:28.040 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.040 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:28:28.041 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:28:28.041 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.041 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:28:28.041 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:28:28.041 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.042 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:28.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:28.042 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:28:28.043 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:28:28.043 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:28:28.043 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:28:28.043 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:28:28.043 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:28:28.044 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.044 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:28.044 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:28:28.044 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:28:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.929 7f3a5de32700 1 -- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a38018930 msgr2=0x7f3a3801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.929 7f3a5de32700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a38018930 0x7f3a3801ae10 crc :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.929 7f3a5de32700 1 -- 172.21.15.32:0/478296990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a581ac510 msgr2=0x7f3a58183ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.930 7f3a5de32700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a581ac510 0x7f3a58183ff0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7f3a500099f0 tx=0x7f3a5000bce0).stop 2021-04-09T18:28:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.930 7f3a5de32700 1 -- 172.21.15.32:0/478296990 shutdown_connections 2021-04-09T18:28:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.930 7f3a5de32700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a38018930 0x7f3a3801ae10 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.930 7f3a5de32700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a581ac510 0x7f3a58183ff0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.930 7f3a5de32700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a58184e90 0x7f3a581abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.930 7f3a5de32700 1 --2- 172.21.15.32:0/478296990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a58198240 0x7f3a581acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.930 7f3a5de32700 1 -- 172.21.15.32:0/478296990 >> 172.21.15.32:0/478296990 conn(0x7f3a58183870 msgr2=0x7f3a581a9650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:28.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.930 7f3a5de32700 1 -- 172.21.15.32:0/478296990 shutdown_connections 2021-04-09T18:28:28.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:27.930 7f3a5de32700 1 -- 172.21.15.32:0/478296990 wait complete. 2021-04-09T18:28:28.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:28:28.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.117 7f35c4fd7700 1 Processor -- start 2021-04-09T18:28:28.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.117 7f35c4fd7700 1 -- start start 2021-04-09T18:28:28.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35c4fd7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f35c00cffa0 0x7f35c00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35c4fd7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f35c00d7800 0x7f35c00d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35c4fd7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c00d8220 0x7f35c00d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35c4fd7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c00d91e0 con 0x7f35c00cffa0 2021-04-09T18:28:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35c4fd7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c00d7280 con 0x7f35c00d7800 2021-04-09T18:28:28.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35c4fd7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c00d6290 con 0x7f35c00d8220 2021-04-09T18:28:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35bed9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f35c00cffa0 0x7f35c00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35bed9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f35c00cffa0 0x7f35c00d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54030/0 (socket says 172.21.15.32:54030) 2021-04-09T18:28:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35bed9d700 1 -- 172.21.15.32:0/428891131 learned_addr learned my addr 172.21.15.32:0/428891131 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:28.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35bf59e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c00d8220 0x7f35c00d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.118 7f35be59c700 1 --2- 172.21.15.32:0/428891131 >> v2:172.21.15.86:3300/0 conn(0x7f35c00d7800 0x7f35c00d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bf59e700 1 -- 172.21.15.32:0/428891131 >> v2:172.21.15.86:3300/0 conn(0x7f35c00d7800 msgr2=0x7f35c00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:28.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bf59e700 1 --2- 172.21.15.32:0/428891131 >> v2:172.21.15.86:3300/0 conn(0x7f35c00d7800 0x7f35c00d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bf59e700 1 -- 172.21.15.32:0/428891131 >> v2:172.21.15.32:3300/0 conn(0x7f35c00cffa0 msgr2=0x7f35c00d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bf59e700 1 --2- 172.21.15.32:0/428891131 >> v2:172.21.15.32:3300/0 conn(0x7f35c00cffa0 0x7f35c00d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bf59e700 1 -- 172.21.15.32:0/428891131 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c00d6480 con 0x7f35c00d8220 2021-04-09T18:28:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bed9d700 1 --2- 172.21.15.32:0/428891131 >> v2:172.21.15.32:3300/0 conn(0x7f35c00cffa0 0x7f35c00d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35be59c700 1 --2- 172.21.15.32:0/428891131 >> v2:172.21.15.86:3300/0 conn(0x7f35c00d7800 0x7f35c00d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bf59e700 1 --2- 172.21.15.32:0/428891131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c00d8220 0x7f35c00d5c70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f35b8006e50 tx=0x7f35b8009e70).ready entity=mon.2 client_cookie=b4d714e455dfbe3d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:28.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bdd9b700 1 -- 172.21.15.32:0/428891131 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b8006a20 con 0x7f35c00d8220 2021-04-09T18:28:28.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bdd9b700 1 -- 172.21.15.32:0/428891131 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b8006790 con 0x7f35c00d8220 2021-04-09T18:28:28.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35bdd9b700 1 -- 172.21.15.32:0/428891131 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b800b700 con 0x7f35c00d8220 2021-04-09T18:28:28.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35c4fd7700 1 -- 172.21.15.32:0/428891131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c00d8220 msgr2=0x7f35c00d5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.119 7f35c4fd7700 1 --2- 172.21.15.32:0/428891131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c00d8220 0x7f35c00d5c70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f35b8006e50 tx=0x7f35b8009e70).stop 2021-04-09T18:28:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.120 7f35c4fd7700 1 -- 172.21.15.32:0/428891131 shutdown_connections 2021-04-09T18:28:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.120 7f35c4fd7700 1 --2- 172.21.15.32:0/428891131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c00d8220 0x7f35c00d5c70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.120 7f35c4fd7700 1 --2- 172.21.15.32:0/428891131 >> v2:172.21.15.86:3300/0 conn(0x7f35c00d7800 0x7f35c00d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.120 7f35c4fd7700 1 --2- 172.21.15.32:0/428891131 >> v2:172.21.15.32:3300/0 conn(0x7f35c00cffa0 0x7f35c00d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.120 7f35c4fd7700 1 -- 172.21.15.32:0/428891131 >> 172.21.15.32:0/428891131 conn(0x7f35c00ce5c0 msgr2=0x7f35c00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:28.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.120 7f35c4fd7700 1 -- 172.21.15.32:0/428891131 shutdown_connections 2021-04-09T18:28:28.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.120 7f35c4fd7700 1 -- 172.21.15.32:0/428891131 wait complete. 2021-04-09T18:28:28.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.140 7f35c4fd7700 1 Processor -- start 2021-04-09T18:28:28.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.140 7f35c4fd7700 1 -- start start 2021-04-09T18:28:28.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35c4fd7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c01acff0 0x7f35c008fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35c4fd7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c0090150 0x7f35c008c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35c4fd7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c008cca0 0x7f35c008e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35c4fd7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c008c100 con 0x7f35c01acff0 2021-04-09T18:28:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35c4fd7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c008d0a0 con 0x7f35c008cca0 2021-04-09T18:28:28.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35c4fd7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c008e960 con 0x7f35c0090150 2021-04-09T18:28:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c01acff0 0x7f35c008fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bed9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c01acff0 0x7f35c008fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54036/0 (socket says 172.21.15.32:54036) 2021-04-09T18:28:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bed9d700 1 -- 172.21.15.32:0/1534670327 learned_addr learned my addr 172.21.15.32:0/1534670327 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35be59c700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c0090150 0x7f35c008c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bf59e700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c008cca0 0x7f35c008e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bed9d700 1 -- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c0090150 msgr2=0x7f35c008c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bed9d700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c0090150 0x7f35c008c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bed9d700 1 -- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c008cca0 msgr2=0x7f35c008e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bed9d700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c008cca0 0x7f35c008e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bed9d700 1 -- 172.21.15.32:0/1534670327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c008eb80 con 0x7f35c01acff0 2021-04-09T18:28:28.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bf59e700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c008cca0 0x7f35c008e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:28.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35bed9d700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c01acff0 0x7f35c008fb30 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 rx=0x7f35b0006ee0 tx=0x7f35b000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:28.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.141 7f35af7fe700 1 -- 172.21.15.32:0/1534670327 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b000bcd0 con 0x7f35c01acff0 2021-04-09T18:28:28.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.142 7f35af7fe700 1 -- 172.21.15.32:0/1534670327 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b00052e0 con 0x7f35c01acff0 2021-04-09T18:28:28.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.142 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c008eb80 con 0x7f35c01acff0 2021-04-09T18:28:28.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.142 7f35af7fe700 1 -- 172.21.15.32:0/1534670327 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b0005730 con 0x7f35c01acff0 2021-04-09T18:28:28.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.142 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c008eb80 con 0x7f35c01acff0 2021-04-09T18:28:28.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.142 7f35af7fe700 1 -- 172.21.15.32:0/1534670327 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f35b00054c0 con 0x7f35c01acff0 2021-04-09T18:28:28.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.142 7f35af7fe700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35a4018930 0x7f35a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.142 7f35af7fe700 1 -- 172.21.15.32:0/1534670327 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f35b00326c0 con 0x7f35c01acff0 2021-04-09T18:28:28.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.142 7f35be59c700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35a4018930 0x7f35a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.143 7f35be59c700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35a4018930 0x7f35a401ae10 crc :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:28.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.163 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c0182dc0 con 0x7f35c01acff0 2021-04-09T18:28:28.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.166 7f35af7fe700 1 -- 172.21.15.32:0/1534670327 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f35b001f730 con 0x7f35c01acff0 2021-04-09T18:28:28.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.353 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35c00d7280 con 0x7f35c01acff0 2021-04-09T18:28:28.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.354 7f35af7fe700 1 -- 172.21.15.32:0/1534670327 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f35b001f940 con 0x7f35c01acff0 2021-04-09T18:28:28.434 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:28.435 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":147,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:28:27.478092","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:28:28.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35a4018930 msgr2=0x7f35a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35a4018930 0x7f35a401ae10 crc :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c01acff0 msgr2=0x7f35c008fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c01acff0 0x7f35c008fb30 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 rx=0x7f35b0006ee0 tx=0x7f35b000b8e0).stop 2021-04-09T18:28:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 shutdown_connections 2021-04-09T18:28:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f35a4018930 0x7f35a401ae10 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c008cca0 0x7f35c008e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c0090150 0x7f35c008c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 --2- 172.21.15.32:0/1534670327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f35c01acff0 0x7f35c008fb30 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 >> 172.21.15.32:0/1534670327 conn(0x7f35c009abb0 msgr2=0x7f35c00a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:28.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 shutdown_connections 2021-04-09T18:28:28.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.356 7f35c4fd7700 1 -- 172.21.15.32:0/1534670327 wait complete. 2021-04-09T18:28:28.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:28:28.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:28:28.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.520 7f8f0bab4700 1 Processor -- start 2021-04-09T18:28:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.520 7f8f0bab4700 1 -- start start 2021-04-09T18:28:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f0bab4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f0410da00 0x7f8f0410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f0bab4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0410d480 con 0x7f8f040d7ca0 2021-04-09T18:28:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f0bab4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f040d1f70 con 0x7f8f0410d010 2021-04-09T18:28:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f0bab4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f040d8d30 con 0x7f8f0410da00 2021-04-09T18:28:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f0a0d1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f0410da00 0x7f8f0410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f090cf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8f040d7ca0 0x7f8f040d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35402/0 (socket says 172.21.15.32:35402) 2021-04-09T18:28:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f090cf700 1 -- 172.21.15.32:0/3737348232 learned_addr learned my addr 172.21.15.32:0/3737348232 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f0a0d1700 1 -- 172.21.15.32:0/3737348232 >> v1:172.21.15.86:6789/0 conn(0x7f8f0410d010 legacy=0x7f8f040d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f0a0d1700 1 -- 172.21.15.32:0/3737348232 >> v1:172.21.15.32:6789/0 conn(0x7f8f040d7ca0 legacy=0x7f8f040d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f088ce700 1 -- 172.21.15.32:0/3737348232 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939920988 0 0) 0x7f8efc000c60 con 0x7f8f0410d010 2021-04-09T18:28:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f0a0d1700 1 -- 172.21.15.32:0/3737348232 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f040d9220 con 0x7f8f0410da00 2021-04-09T18:28:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.521 7f8f0a0d1700 1 --2- 172.21.15.32:0/3737348232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0410da00 0x7f8f0410de00 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 rx=0x7f8ef4006e30 tx=0x7f8ef4009de0).ready entity=mon.0 client_cookie=42161bbd6a970604 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:28.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.522 7f8f088ce700 1 -- 172.21.15.32:0/3737348232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ef400b300 con 0x7f8f0410da00 2021-04-09T18:28:28.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.522 7f8f088ce700 1 -- 172.21.15.32:0/3737348232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef400b680 con 0x7f8f0410da00 2021-04-09T18:28:28.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.522 7f8f088ce700 1 -- 172.21.15.32:0/3737348232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ef400bb60 con 0x7f8f0410da00 2021-04-09T18:28:28.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.522 7f8f0bab4700 1 -- 172.21.15.32:0/3737348232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0410da00 msgr2=0x7f8f0410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.522 7f8f0bab4700 1 --2- 172.21.15.32:0/3737348232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0410da00 0x7f8f0410de00 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 rx=0x7f8ef4006e30 tx=0x7f8ef4009de0).stop 2021-04-09T18:28:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.522 7f8f0bab4700 1 -- 172.21.15.32:0/3737348232 shutdown_connections 2021-04-09T18:28:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.522 7f8f0bab4700 1 --2- 172.21.15.32:0/3737348232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0410da00 0x7f8f0410de00 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.522 7f8f0bab4700 1 -- 172.21.15.32:0/3737348232 >> 172.21.15.32:0/3737348232 conn(0x7f8f040ce5f0 msgr2=0x7f8f040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.523 7f8f0bab4700 1 -- 172.21.15.32:0/3737348232 shutdown_connections 2021-04-09T18:28:28.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.523 7f8f0bab4700 1 -- 172.21.15.32:0/3737348232 wait complete. 2021-04-09T18:28:28.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f0bab4700 1 Processor -- start 2021-04-09T18:28:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f0bab4700 1 -- start start 2021-04-09T18:28:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f0bab4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f040b7810 0x7f8f040b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f0bab4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f040b0900 0x7f8f040b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f0bab4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f040b1320 0x7f8f04179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f0bab4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040b0090 con 0x7f8f040b0900 2021-04-09T18:28:28.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f0bab4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04179c30 con 0x7f8f040b7810 2021-04-09T18:28:28.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f0bab4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04179e50 con 0x7f8f040b1320 2021-04-09T18:28:28.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f090cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f040b0900 0x7f8f040b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f090cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f040b0900 0x7f8f040b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54048/0 (socket says 172.21.15.32:54048) 2021-04-09T18:28:28.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f090cf700 1 -- 172.21.15.32:0/4188785426 learned_addr learned my addr 172.21.15.32:0/4188785426 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:28.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f098d0700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f040b7810 0x7f8f040b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.543 7f8f0a0d1700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f040b1320 0x7f8f04179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f090cf700 1 -- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f040b1320 msgr2=0x7f8f04179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f090cf700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f040b1320 0x7f8f04179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f090cf700 1 -- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f040b7810 msgr2=0x7f8f040b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f090cf700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f040b7810 0x7f8f040b7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f090cf700 1 -- 172.21.15.32:0/4188785426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0417a070 con 0x7f8f040b0900 2021-04-09T18:28:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f090cf700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f040b0900 0x7f8f040b0d00 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f8ef800a5a0 tx=0x7f8ef8001480).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:28.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f027fc700 1 -- 172.21.15.32:0/4188785426 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ef80070d0 con 0x7f8f040b0900 2021-04-09T18:28:28.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f027fc700 1 -- 172.21.15.32:0/4188785426 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef800aca0 con 0x7f8f040b0900 2021-04-09T18:28:28.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0417a070 con 0x7f8f040b0900 2021-04-09T18:28:28.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f027fc700 1 -- 172.21.15.32:0/4188785426 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ef8004590 con 0x7f8f040b0900 2021-04-09T18:28:28.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.544 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0417a070 con 0x7f8f040b0900 2021-04-09T18:28:28.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.545 7f8f027fc700 1 -- 172.21.15.32:0/4188785426 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8ef8008000 con 0x7f8f040b0900 2021-04-09T18:28:28.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.545 7f8f027fc700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8ee4018930 0x7f8ee401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:28.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.545 7f8f027fc700 1 -- 172.21.15.32:0/4188785426 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f8ef80326a0 con 0x7f8f040b0900 2021-04-09T18:28:28.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.545 7f8f098d0700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8ee4018930 0x7f8ee401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:28.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.545 7f8f098d0700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8ee4018930 0x7f8ee401ae10 crc :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:28.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.565 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0408dac0 con 0x7f8f040b0900 2021-04-09T18:28:28.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.568 7f8f027fc700 1 -- 172.21.15.32:0/4188785426 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8ef80009b0 con 0x7f8f040b0900 2021-04-09T18:28:28.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.747 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f8f0410d480 con 0x7f8f040b0900 2021-04-09T18:28:28.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.747 7f8f027fc700 1 -- 172.21.15.32:0/4188785426 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v147) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8ef8008000 con 0x7f8f040b0900 2021-04-09T18:28:28.828 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:28:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8ee4018930 msgr2=0x7f8ee401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8ee4018930 0x7f8ee401ae10 crc :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f040b0900 msgr2=0x7f8f040b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f040b0900 0x7f8f040b0d00 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f8ef800a5a0 tx=0x7f8ef8001480).stop 2021-04-09T18:28:28.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 shutdown_connections 2021-04-09T18:28:28.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8ee4018930 0x7f8ee401ae10 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f040b0900 0x7f8f040b0d00 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f040b1320 0x7f8f04179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 --2- 172.21.15.32:0/4188785426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f040b7810 0x7f8f040b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:28.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 >> 172.21.15.32:0/4188785426 conn(0x7f8f040bb410 msgr2=0x7f8f040a13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:28.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 shutdown_connections 2021-04-09T18:28:28.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.750 7f8f0bab4700 1 -- 172.21.15.32:0/4188785426 wait complete. 2021-04-09T18:28:28.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:28:29.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.909 7efcf3c56700 1 Processor -- start 2021-04-09T18:28:29.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.909 7efcf3c56700 1 -- start start 2021-04-09T18:28:29.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf3c56700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcec0d7ca0 0x7efcec0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf3c56700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec10da00 0x7efcec10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf3c56700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec10d480 con 0x7efcec10d010 2021-04-09T18:28:29.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf3c56700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec0d9220 con 0x7efcec0d7ca0 2021-04-09T18:28:29.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf3c56700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0d86c0 con 0x7efcec10da00 2021-04-09T18:28:29.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf2273700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec10da00 0x7efcec10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf1271700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcec0d7ca0 0x7efcec0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf1a72700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efcec10d010 0x7efcec0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:33340/0 (socket says 172.21.15.32:33340) 2021-04-09T18:28:29.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf1a72700 1 -- 172.21.15.32:0/2682457343 learned_addr learned my addr 172.21.15.32:0/2682457343 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:29.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf0a70700 1 -- 172.21.15.32:0/2682457343 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090631217 0 0) 0x7efce80010c0 con 0x7efcec10d010 2021-04-09T18:28:29.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf0a70700 1 -- 172.21.15.32:0/2682457343 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd8002fe0 con 0x7efcec10d010 2021-04-09T18:28:29.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf1271700 1 -- 172.21.15.32:0/2682457343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec10da00 msgr2=0x7efcec10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:29.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf1271700 1 --2- 172.21.15.32:0/2682457343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec10da00 0x7efcec10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf1271700 1 -- 172.21.15.32:0/2682457343 >> v1:172.21.15.86:6789/0 conn(0x7efcec10d010 legacy=0x7efcec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf1271700 1 -- 172.21.15.32:0/2682457343 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0d1f70 con 0x7efcec0d7ca0 2021-04-09T18:28:29.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf2273700 1 --2- 172.21.15.32:0/2682457343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec10da00 0x7efcec10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.910 7efcf1271700 1 --2- 172.21.15.32:0/2682457343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcec0d7ca0 0x7efcec0d80a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7efcdc006650 tx=0x7efcdc009a80).ready entity=mon.1 client_cookie=65cdc3d9b8b49a0a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.911 7efcf0a70700 1 -- 172.21.15.32:0/2682457343 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcdc0009b0 con 0x7efcec0d7ca0 2021-04-09T18:28:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.911 7efcf0a70700 1 -- 172.21.15.32:0/2682457343 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc00c0c0 con 0x7efcec0d7ca0 2021-04-09T18:28:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.911 7efcf0a70700 1 -- 172.21.15.32:0/2682457343 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcdc00c510 con 0x7efcec0d7ca0 2021-04-09T18:28:29.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.911 7efcf3c56700 1 -- 172.21.15.32:0/2682457343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcec0d7ca0 msgr2=0x7efcec0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.911 7efcf3c56700 1 --2- 172.21.15.32:0/2682457343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcec0d7ca0 0x7efcec0d80a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7efcdc006650 tx=0x7efcdc009a80).stop 2021-04-09T18:28:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.911 7efcf3c56700 1 -- 172.21.15.32:0/2682457343 shutdown_connections 2021-04-09T18:28:29.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.911 7efcf3c56700 1 --2- 172.21.15.32:0/2682457343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec10da00 0x7efcec10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.911 7efcf3c56700 1 --2- 172.21.15.32:0/2682457343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcec0d7ca0 0x7efcec0d80a0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.911 7efcf3c56700 1 -- 172.21.15.32:0/2682457343 >> 172.21.15.32:0/2682457343 conn(0x7efcec0ce5f0 msgr2=0x7efcec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:29.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.912 7efcf3c56700 1 -- 172.21.15.32:0/2682457343 shutdown_connections 2021-04-09T18:28:29.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.912 7efcf3c56700 1 -- 172.21.15.32:0/2682457343 wait complete. 2021-04-09T18:28:29.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.932 7efcf3c56700 1 Processor -- start 2021-04-09T18:28:29.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.932 7efcf3c56700 1 -- start start 2021-04-09T18:28:29.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.932 7efcf3c56700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec1a5210 0x7efcec1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.932 7efcf3c56700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcec1a6340 0x7efcec1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.932 7efcf3c56700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcec0abaa0 0x7efcec0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.932 7efcf3c56700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec1a6d60 con 0x7efcec1a6340 2021-04-09T18:28:29.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.932 7efcf3c56700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0ac4c0 con 0x7efcec0abaa0 2021-04-09T18:28:29.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.932 7efcf3c56700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0ac6e0 con 0x7efcec1a5210 2021-04-09T18:28:29.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.932 7efcf1271700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcec1a6340 0x7efcec1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf1271700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcec1a6340 0x7efcec1a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54066/0 (socket says 172.21.15.32:54066) 2021-04-09T18:28:29.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf1271700 1 -- 172.21.15.32:0/1332060959 learned_addr learned my addr 172.21.15.32:0/1332060959 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:29.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf1a72700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec1a5210 0x7efcec1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf2273700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcec0abaa0 0x7efcec0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf1271700 1 -- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec1a5210 msgr2=0x7efcec1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf1271700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec1a5210 0x7efcec1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf1271700 1 -- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcec0abaa0 msgr2=0x7efcec0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf1271700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcec0abaa0 0x7efcec0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf1271700 1 -- 172.21.15.32:0/1332060959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec17a720 con 0x7efcec1a6340 2021-04-09T18:28:29.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf1271700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcec1a6340 0x7efcec1a6740 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 rx=0x7efcdc009630 tx=0x7efcdc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:29.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efce27fc700 1 -- 172.21.15.32:0/1332060959 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcdc0093a0 con 0x7efcec1a6340 2021-04-09T18:28:29.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efce27fc700 1 -- 172.21.15.32:0/1332060959 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc00bd70 con 0x7efcec1a6340 2021-04-09T18:28:29.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efce27fc700 1 -- 172.21.15.32:0/1332060959 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcdc0054d0 con 0x7efcec1a6340 2021-04-09T18:28:29.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec17a720 con 0x7efcec1a6340 2021-04-09T18:28:29.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.933 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec17a720 con 0x7efcec1a6340 2021-04-09T18:28:29.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.934 7efce27fc700 1 -- 172.21.15.32:0/1332060959 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efcdc00a000 con 0x7efcec1a6340 2021-04-09T18:28:29.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.934 7efce27fc700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcd40188c0 0x7efcd401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.934 7efce27fc700 1 -- 172.21.15.32:0/1332060959 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4757+0+0 (secure 0 0 0) 0x7efcdc032890 con 0x7efcec1a6340 2021-04-09T18:28:29.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.934 7efcf1a72700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcd40188c0 0x7efcd401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.935 7efcf1a72700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcd40188c0 0x7efcd401adc0 crc :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:29.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.954 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec1b7070 con 0x7efcec1a6340 2021-04-09T18:28:29.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:28.957 7efce27fc700 1 -- 172.21.15.32:0/1332060959 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efcdc01f9c0 con 0x7efcec1a6340 2021-04-09T18:28:29.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.135 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efcec10d480 con 0x7efcec1a6340 2021-04-09T18:28:29.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.135 7efce27fc700 1 -- 172.21.15.32:0/1332060959 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v147) v1 ==== 84+0+64 (secure 0 0 0) 0x7efcdc01f780 con 0x7efcec1a6340 2021-04-09T18:28:29.216 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:29.217 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:28:29.217 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:28:29.217 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:28:29.217 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:28:29.218 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:29.218 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:28:29.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcd40188c0 msgr2=0x7efcd401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcd40188c0 0x7efcd401adc0 crc :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcec1a6340 msgr2=0x7efcec1a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcec1a6340 0x7efcec1a6740 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 rx=0x7efcdc009630 tx=0x7efcdc00b980).stop 2021-04-09T18:28:29.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 shutdown_connections 2021-04-09T18:28:29.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcd40188c0 0x7efcd401adc0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcec1a6340 0x7efcec1a6740 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcec0abaa0 0x7efcec0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 --2- 172.21.15.32:0/1332060959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcec1a5210 0x7efcec1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 >> 172.21.15.32:0/1332060959 conn(0x7efcec1a99c0 msgr2=0x7efcec17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:29.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 shutdown_connections 2021-04-09T18:28:29.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.138 7efcf3c56700 1 -- 172.21.15.32:0/1332060959 wait complete. 2021-04-09T18:28:29.228 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:28:29.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:28:29.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.300 7f27eace9700 1 Processor -- start 2021-04-09T18:28:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.300 7f27eace9700 1 -- start start 2021-04-09T18:28:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.301 7f27eace9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e410d010 0x7f27e40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.301 7f27eace9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e410d480 con 0x7f27e40d1da0 2021-04-09T18:28:29.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.301 7f27eace9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e410e1f0 con 0x7f27e40d8490 2021-04-09T18:28:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.301 7f27eace9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e410e780 con 0x7f27e410d010 2021-04-09T18:28:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.301 7f27e3fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f27e40d1da0 0x7f27e40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35432/0 (socket says 172.21.15.32:35432) 2021-04-09T18:28:29.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.301 7f27e3fff700 1 -- 172.21.15.32:0/4201882732 learned_addr learned my addr 172.21.15.32:0/4201882732 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:29.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.301 7f27e8b05700 1 --2- 172.21.15.32:0/4201882732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e410d010 0x7f27e40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e37fe700 1 -- 172.21.15.32:0/4201882732 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786258173 0 0) 0x7f27d8000ce0 con 0x7f27e40d1da0 2021-04-09T18:28:29.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e37fe700 1 -- 172.21.15.32:0/4201882732 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc002fe0 con 0x7f27e40d1da0 2021-04-09T18:28:29.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e37fe700 1 -- 172.21.15.32:0/4201882732 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681814297 0 0) 0x7f27dc000d30 con 0x7f27e40d8490 2021-04-09T18:28:29.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e37fe700 1 -- 172.21.15.32:0/4201882732 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc003210 con 0x7f27e40d8490 2021-04-09T18:28:29.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e8b05700 1 -- 172.21.15.32:0/4201882732 >> v1:172.21.15.86:6789/0 conn(0x7f27e40d8490 legacy=0x7f27e410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e37fe700 1 -- 172.21.15.32:0/4201882732 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3115545517 0 0) 0x7f27d80010e0 con 0x7f27e40d1da0 2021-04-09T18:28:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e8b05700 1 -- 172.21.15.32:0/4201882732 >> v1:172.21.15.32:6789/0 conn(0x7f27e40d1da0 legacy=0x7f27e40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e8b05700 1 -- 172.21.15.32:0/4201882732 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40d8d30 con 0x7f27e410d010 2021-04-09T18:28:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e8b05700 1 --2- 172.21.15.32:0/4201882732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e410d010 0x7f27e40d1780 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f27d4006650 tx=0x7f27d4009a80).ready entity=mon.2 client_cookie=88d83e9e3692a11 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:29.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e37fe700 1 -- 172.21.15.32:0/4201882732 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27d40009b0 con 0x7f27e410d010 2021-04-09T18:28:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e37fe700 1 -- 172.21.15.32:0/4201882732 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d400c0c0 con 0x7f27e410d010 2021-04-09T18:28:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27e37fe700 1 -- 172.21.15.32:0/4201882732 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27d400c510 con 0x7f27e410d010 2021-04-09T18:28:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27eace9700 1 -- 172.21.15.32:0/4201882732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e410d010 msgr2=0x7f27e40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.302 7f27eace9700 1 --2- 172.21.15.32:0/4201882732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e410d010 0x7f27e40d1780 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f27d4006650 tx=0x7f27d4009a80).stop 2021-04-09T18:28:29.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.303 7f27eace9700 1 -- 172.21.15.32:0/4201882732 shutdown_connections 2021-04-09T18:28:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.303 7f27eace9700 1 --2- 172.21.15.32:0/4201882732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e410d010 0x7f27e40d1780 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.303 7f27eace9700 1 -- 172.21.15.32:0/4201882732 >> 172.21.15.32:0/4201882732 conn(0x7f27e40ce5f0 msgr2=0x7f27e40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.303 7f27eace9700 1 -- 172.21.15.32:0/4201882732 shutdown_connections 2021-04-09T18:28:29.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.303 7f27eace9700 1 -- 172.21.15.32:0/4201882732 wait complete. 2021-04-09T18:28:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.323 7f27eace9700 1 Processor -- start 2021-04-09T18:28:29.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.323 7f27eace9700 1 -- start start 2021-04-09T18:28:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27eace9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40a0110 0x7f27e418c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27eace9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0510 0x7f27e40c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27eace9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40c36e0 0x7f27e40c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27eace9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e418cf60 con 0x7f27e40c36e0 2021-04-09T18:28:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27eace9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e4199ea0 con 0x7f27e40a0510 2021-04-09T18:28:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27eace9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e4187d70 con 0x7f27e40a0110 2021-04-09T18:28:29.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e9306700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40c36e0 0x7f27e40c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e8b05700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40a0110 0x7f27e418c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e3fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0510 0x7f27e40c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e9306700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40c36e0 0x7f27e40c3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54080/0 (socket says 172.21.15.32:54080) 2021-04-09T18:28:29.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e9306700 1 -- 172.21.15.32:0/2061867657 learned_addr learned my addr 172.21.15.32:0/2061867657 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e9306700 1 -- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40a0110 msgr2=0x7f27e418c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e9306700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40a0110 0x7f27e418c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e9306700 1 -- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0510 msgr2=0x7f27e40c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:29.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e9306700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0510 0x7f27e40c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e9306700 1 -- 172.21.15.32:0/2061867657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e418d4e0 con 0x7f27e40c36e0 2021-04-09T18:28:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.324 7f27e3fff700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0510 0x7f27e40c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.325 7f27e9306700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40c36e0 0x7f27e40c3ae0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 rx=0x7f27dc00bb90 tx=0x7f27dc00def0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:29.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.325 7f27e17fa700 1 -- 172.21.15.32:0/2061867657 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27dc0030d0 con 0x7f27e40c36e0 2021-04-09T18:28:29.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.325 7f27e17fa700 1 -- 172.21.15.32:0/2061867657 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27dc00e3a0 con 0x7f27e40c36e0 2021-04-09T18:28:29.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.325 7f27e17fa700 1 -- 172.21.15.32:0/2061867657 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27dc00e8c0 con 0x7f27e40c36e0 2021-04-09T18:28:29.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.325 7f27eace9700 1 -- 172.21.15.32:0/2061867657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e418d4e0 con 0x7f27e40c36e0 2021-04-09T18:28:29.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.325 7f27eace9700 1 -- 172.21.15.32:0/2061867657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e418d4e0 con 0x7f27e40c36e0 2021-04-09T18:28:29.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.326 7f27e17fa700 1 -- 172.21.15.32:0/2061867657 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f27dc00c000 con 0x7f27e40c36e0 2021-04-09T18:28:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.326 7f27e17fa700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27c40188c0 0x7f27c401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.326 7f27e17fa700 1 -- 172.21.15.32:0/2061867657 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f27dc034a20 con 0x7f27e40c36e0 2021-04-09T18:28:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.326 7f27e8b05700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27c40188c0 0x7f27c401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:29.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.326 7f27e8b05700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27c40188c0 0x7f27c401adc0 crc :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:29.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.345 7f27eace9700 1 -- 172.21.15.32:0/2061867657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e410d480 con 0x7f27e40c36e0 2021-04-09T18:28:29.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.349 7f27e17fa700 1 -- 172.21.15.32:0/2061867657 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f27dc02ef40 con 0x7f27e40c36e0 2021-04-09T18:28:29.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:29.533 7f27eace9700 1 -- 172.21.15.32:0/2061867657 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f27e410d480 con 0x7f27e40c36e0 2021-04-09T18:28:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.496 7f27e17fa700 1 -- 172.21.15.32:0/2061867657 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v148) v1 ==== 133+0+0 (secure 0 0 0) 0x7f27dc02ef40 con 0x7f27e40c36e0 2021-04-09T18:28:30.579 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:28:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 -- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27c40188c0 msgr2=0x7f27c401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27c40188c0 0x7f27c401adc0 crc :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 -- 172.21.15.32:0/2061867657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40c36e0 msgr2=0x7f27e40c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40c36e0 0x7f27e40c3ae0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 rx=0x7f27dc00bb90 tx=0x7f27dc00def0).stop 2021-04-09T18:28:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 -- 172.21.15.32:0/2061867657 shutdown_connections 2021-04-09T18:28:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27c40188c0 0x7f27c401adc0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40c36e0 0x7f27e40c3ae0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0510 0x7f27e40c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 --2- 172.21.15.32:0/2061867657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40a0110 0x7f27e418c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 -- 172.21.15.32:0/2061867657 >> 172.21.15.32:0/2061867657 conn(0x7f27e40b93a0 msgr2=0x7f27e40b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:30.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 -- 172.21.15.32:0/2061867657 shutdown_connections 2021-04-09T18:28:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.502 7f27eace9700 1 -- 172.21.15.32:0/2061867657 wait complete. 2021-04-09T18:28:30.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:28:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.672 7f03e4912700 1 Processor -- start 2021-04-09T18:28:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.672 7f03e4912700 1 -- start start 2021-04-09T18:28:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.673 7f03e4912700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00d7ca0 0x7f03e00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.673 7f03e4912700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f03e010da00 0x7f03e010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.673 7f03e4912700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e010d480 con 0x7f03e010d010 2021-04-09T18:28:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.673 7f03e4912700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e00d9220 con 0x7f03e010da00 2021-04-09T18:28:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.673 7f03e4912700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e00d86c0 con 0x7f03e00d7ca0 2021-04-09T18:28:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.673 7f03dee1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f03e010da00 0x7f03e010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.673 7f03de61c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f03e010d010 0x7f03e00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35442/0 (socket says 172.21.15.32:35442) 2021-04-09T18:28:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.673 7f03de61c700 1 -- 172.21.15.32:0/3637293932 learned_addr learned my addr 172.21.15.32:0/3637293932 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.673 7f03dde1b700 1 --2- 172.21.15.32:0/3637293932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00d7ca0 0x7f03e00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dd61a700 1 -- 172.21.15.32:0/3637293932 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628710398 0 0) 0x7f03d0000ce0 con 0x7f03e010d010 2021-04-09T18:28:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dd61a700 1 -- 172.21.15.32:0/3637293932 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c8002fe0 con 0x7f03e010d010 2021-04-09T18:28:30.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dee1d700 1 -- 172.21.15.32:0/3637293932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00d7ca0 msgr2=0x7f03e00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:30.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dee1d700 1 --2- 172.21.15.32:0/3637293932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00d7ca0 0x7f03e00d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dee1d700 1 -- 172.21.15.32:0/3637293932 >> v1:172.21.15.32:6789/0 conn(0x7f03e010d010 legacy=0x7f03e00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dee1d700 1 -- 172.21.15.32:0/3637293932 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e00d1f70 con 0x7f03e010da00 2021-04-09T18:28:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dee1d700 1 --2- 172.21.15.32:0/3637293932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e010da00 0x7f03e010de00 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 rx=0x7f03d8006ab0 tx=0x7f03d8009b40).ready entity=mon.0 client_cookie=c843c7d712de270f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dd61a700 1 -- 172.21.15.32:0/3637293932 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03d80022e0 con 0x7f03e010da00 2021-04-09T18:28:30.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dd61a700 1 -- 172.21.15.32:0/3637293932 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d8002660 con 0x7f03e010da00 2021-04-09T18:28:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03dd61a700 1 -- 172.21.15.32:0/3637293932 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03d8002b40 con 0x7f03e010da00 2021-04-09T18:28:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03e4912700 1 -- 172.21.15.32:0/3637293932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e010da00 msgr2=0x7f03e010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.674 7f03e4912700 1 --2- 172.21.15.32:0/3637293932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e010da00 0x7f03e010de00 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 rx=0x7f03d8006ab0 tx=0x7f03d8009b40).stop 2021-04-09T18:28:30.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.675 7f03e4912700 1 -- 172.21.15.32:0/3637293932 shutdown_connections 2021-04-09T18:28:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.675 7f03e4912700 1 --2- 172.21.15.32:0/3637293932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e010da00 0x7f03e010de00 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.675 7f03e4912700 1 --2- 172.21.15.32:0/3637293932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00d7ca0 0x7f03e00d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.675 7f03e4912700 1 -- 172.21.15.32:0/3637293932 >> 172.21.15.32:0/3637293932 conn(0x7f03e00ce5f0 msgr2=0x7f03e00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.675 7f03e4912700 1 -- 172.21.15.32:0/3637293932 shutdown_connections 2021-04-09T18:28:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.675 7f03e4912700 1 -- 172.21.15.32:0/3637293932 wait complete. 2021-04-09T18:28:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.695 7f03e4912700 1 Processor -- start 2021-04-09T18:28:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.695 7f03e4912700 1 -- start start 2021-04-09T18:28:30.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03e4912700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e00b1680 0x7f03e00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:30.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03e4912700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e009e330 0x7f03e00b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:30.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03e4912700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00b2190 0x7f03e009f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03e4912700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e009e730 con 0x7f03e00b1680 2021-04-09T18:28:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03e4912700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e009f6c0 con 0x7f03e009e330 2021-04-09T18:28:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03e4912700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e009f8e0 con 0x7f03e00b2190 2021-04-09T18:28:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03de61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e00b1680 0x7f03e00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03de61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e00b1680 0x7f03e00b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54090/0 (socket says 172.21.15.32:54090) 2021-04-09T18:28:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03de61c700 1 -- 172.21.15.32:0/2448310274 learned_addr learned my addr 172.21.15.32:0/2448310274 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03dee1d700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00b2190 0x7f03e009f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03dde1b700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e009e330 0x7f03e00b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:30.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03de61c700 1 -- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00b2190 msgr2=0x7f03e009f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:30.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03de61c700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00b2190 0x7f03e009f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03de61c700 1 -- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e009e330 msgr2=0x7f03e00b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:30.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03de61c700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e009e330 0x7f03e00b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:30.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03de61c700 1 -- 172.21.15.32:0/2448310274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e009fb00 con 0x7f03e00b1680 2021-04-09T18:28:30.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03de61c700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e00b1680 0x7f03e00b0e30 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 rx=0x7f03d000b530 tx=0x7f03d000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:30.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03ceffd700 1 -- 172.21.15.32:0/2448310274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03d000bcc0 con 0x7f03e00b1680 2021-04-09T18:28:30.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.696 7f03ceffd700 1 -- 172.21.15.32:0/2448310274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d0006030 con 0x7f03e00b1680 2021-04-09T18:28:30.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.697 7f03e4912700 1 -- 172.21.15.32:0/2448310274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e009fb00 con 0x7f03e00b1680 2021-04-09T18:28:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.697 7f03ceffd700 1 -- 172.21.15.32:0/2448310274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03d0004620 con 0x7f03e00b1680 2021-04-09T18:28:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.697 7f03e4912700 1 -- 172.21.15.32:0/2448310274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e009fb00 con 0x7f03e00b1680 2021-04-09T18:28:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.697 7f03ceffd700 1 -- 172.21.15.32:0/2448310274 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f03d00043a0 con 0x7f03e00b1680 2021-04-09T18:28:30.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.697 7f03ceffd700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03c40188c0 0x7f03c401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:30.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.697 7f03ceffd700 1 -- 172.21.15.32:0/2448310274 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5111+0+0 (secure 0 0 0) 0x7f03d0032800 con 0x7f03e00b1680 2021-04-09T18:28:30.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.698 7f03dde1b700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03c40188c0 0x7f03c401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:30.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.698 7f03dde1b700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03c40188c0 0x7f03c401adc0 crc :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:30.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.717 7f03e4912700 1 -- 172.21.15.32:0/2448310274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e010d480 con 0x7f03e00b1680 2021-04-09T18:28:30.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.721 7f03ceffd700 1 -- 172.21.15.32:0/2448310274 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f03d0006b90 con 0x7f03e00b1680 2021-04-09T18:28:30.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:30.911 7f03e4912700 1 -- 172.21.15.32:0/2448310274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f03e010d480 con 0x7f03e00b1680 2021-04-09T18:28:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.795 7f03ceffd700 1 -- 172.21.15.32:0/2448310274 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v153) v1 ==== 125+0+0 (secure 0 0 0) 0x7f03d00043a0 con 0x7f03e00b1680 2021-04-09T18:28:31.907 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:28:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 -- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03c40188c0 msgr2=0x7f03c401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03c40188c0 0x7f03c401adc0 crc :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 -- 172.21.15.32:0/2448310274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e00b1680 msgr2=0x7f03e00b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:31.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e00b1680 0x7f03e00b0e30 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 rx=0x7f03d000b530 tx=0x7f03d000b8d0).stop 2021-04-09T18:28:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 -- 172.21.15.32:0/2448310274 shutdown_connections 2021-04-09T18:28:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03c40188c0 0x7f03c401adc0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03e00b2190 0x7f03e009f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:31.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03e009e330 0x7f03e00b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 --2- 172.21.15.32:0/2448310274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03e00b1680 0x7f03e00b0e30 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 -- 172.21.15.32:0/2448310274 >> 172.21.15.32:0/2448310274 conn(0x7f03e00c4290 msgr2=0x7f03e00c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 -- 172.21.15.32:0/2448310274 shutdown_connections 2021-04-09T18:28:31.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:31.829 7f03e4912700 1 -- 172.21.15.32:0/2448310274 wait complete. 2021-04-09T18:28:31.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:28:32.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.000 7f27ebdcc700 1 Processor -- start 2021-04-09T18:28:32.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.000 7f27ebdcc700 1 -- start start 2021-04-09T18:28:32.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.000 7f27ebdcc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f27e410d020 0x7f27e40d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.000 7f27ebdcc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27e40d1db0 0x7f27e40d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.000 7f27ebdcc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e410d490 con 0x7f27e40d82d0 2021-04-09T18:28:32.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27ebdcc700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e40d8d40 con 0x7f27e40d1db0 2021-04-09T18:28:32.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27ebdcc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e40d9230 con 0x7f27e410d020 2021-04-09T18:28:32.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e93e7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27e40d1db0 0x7f27e40d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27ea3e9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f27e40d82d0 0x7f27e410da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35462/0 (socket says 172.21.15.32:35462) 2021-04-09T18:28:32.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27ea3e9700 1 -- 172.21.15.32:0/827826379 learned_addr learned my addr 172.21.15.32:0/827826379 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:32.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e9be8700 1 --2- 172.21.15.32:0/827826379 >> v2:172.21.15.86:3300/0 conn(0x7f27e410d020 0x7f27e40d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e93e7700 1 -- 172.21.15.32:0/827826379 >> v2:172.21.15.86:3300/0 conn(0x7f27e410d020 msgr2=0x7f27e40d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:32.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e93e7700 1 --2- 172.21.15.32:0/827826379 >> v2:172.21.15.86:3300/0 conn(0x7f27e410d020 0x7f27e40d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e93e7700 1 -- 172.21.15.32:0/827826379 >> v1:172.21.15.32:6789/0 conn(0x7f27e40d82d0 legacy=0x7f27e410da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e93e7700 1 -- 172.21.15.32:0/827826379 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40d8700 con 0x7f27e40d1db0 2021-04-09T18:28:32.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e8be6700 1 -- 172.21.15.32:0/827826379 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333795471 0 0) 0x7f27dc000d30 con 0x7f27e40d82d0 2021-04-09T18:28:32.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e93e7700 1 --2- 172.21.15.32:0/827826379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40d1db0 0x7f27e40d7cb0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f27d4006650 tx=0x7f27d4009a00).ready entity=mon.0 client_cookie=96aaf25d0771ca6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:32.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e8be6700 1 -- 172.21.15.32:0/827826379 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27d4009df0 con 0x7f27e40d1db0 2021-04-09T18:28:32.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e8be6700 1 -- 172.21.15.32:0/827826379 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d40009b0 con 0x7f27e40d1db0 2021-04-09T18:28:32.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27e8be6700 1 -- 172.21.15.32:0/827826379 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27d400c2b0 con 0x7f27e40d1db0 2021-04-09T18:28:32.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27ebdcc700 1 -- 172.21.15.32:0/827826379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40d1db0 msgr2=0x7f27e40d7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.001 7f27ebdcc700 1 --2- 172.21.15.32:0/827826379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40d1db0 0x7f27e40d7cb0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f27d4006650 tx=0x7f27d4009a00).stop 2021-04-09T18:28:32.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.002 7f27ebdcc700 1 -- 172.21.15.32:0/827826379 shutdown_connections 2021-04-09T18:28:32.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.002 7f27ebdcc700 1 --2- 172.21.15.32:0/827826379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40d1db0 0x7f27e40d7cb0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.002 7f27ebdcc700 1 --2- 172.21.15.32:0/827826379 >> v2:172.21.15.86:3300/0 conn(0x7f27e410d020 0x7f27e40d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.002 7f27ebdcc700 1 -- 172.21.15.32:0/827826379 >> 172.21.15.32:0/827826379 conn(0x7f27e40ce600 msgr2=0x7f27e40d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:32.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.002 7f27ebdcc700 1 -- 172.21.15.32:0/827826379 shutdown_connections 2021-04-09T18:28:32.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.002 7f27ebdcc700 1 -- 172.21.15.32:0/827826379 wait complete. 2021-04-09T18:28:32.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27ebdcc700 1 Processor -- start 2021-04-09T18:28:32.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27ebdcc700 1 -- start start 2021-04-09T18:28:32.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27ebdcc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40ad730 0x7f27e40ab120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27ebdcc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40adb30 0x7f27e40a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27ebdcc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0df0 0x7f27e40a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27ebdcc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40a11f0 con 0x7f27e40ad730 2021-04-09T18:28:32.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27ebdcc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40a1420 con 0x7f27e40a0df0 2021-04-09T18:28:32.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27ebdcc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40bce20 con 0x7f27e40adb30 2021-04-09T18:28:32.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27e9be8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40ad730 0x7f27e40ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27e9be8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40ad730 0x7f27e40ab120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54108/0 (socket says 172.21.15.32:54108) 2021-04-09T18:28:32.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27e9be8700 1 -- 172.21.15.32:0/627517832 learned_addr learned my addr 172.21.15.32:0/627517832 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:32.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27ea3e9700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0df0 0x7f27e40a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.023 7f27e93e7700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40adb30 0x7f27e40a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27e9be8700 1 -- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40adb30 msgr2=0x7f27e40a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27e9be8700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40adb30 0x7f27e40a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27e9be8700 1 -- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0df0 msgr2=0x7f27e40a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27e9be8700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0df0 0x7f27e40a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27e9be8700 1 -- 172.21.15.32:0/627517832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40ab740 con 0x7f27e40ad730 2021-04-09T18:28:32.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27ea3e9700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0df0 0x7f27e40a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:32.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27e9be8700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40ad730 0x7f27e40ab120 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f27e000b560 tx=0x7f27e000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:32.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27da7fc700 1 -- 172.21.15.32:0/627517832 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e000bcf0 con 0x7f27e40ad730 2021-04-09T18:28:32.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27da7fc700 1 -- 172.21.15.32:0/627517832 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e0000c90 con 0x7f27e40ad730 2021-04-09T18:28:32.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e40ab740 con 0x7f27e40ad730 2021-04-09T18:28:32.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27da7fc700 1 -- 172.21.15.32:0/627517832 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e00009b0 con 0x7f27e40ad730 2021-04-09T18:28:32.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.024 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e40ab740 con 0x7f27e40ad730 2021-04-09T18:28:32.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.025 7f27da7fc700 1 -- 172.21.15.32:0/627517832 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f27e0006b00 con 0x7f27e40ad730 2021-04-09T18:28:32.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.025 7f27da7fc700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27cc018930 0x7f27cc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.025 7f27da7fc700 1 -- 172.21.15.32:0/627517832 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f27e00329a0 con 0x7f27e40ad730 2021-04-09T18:28:32.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.025 7f27e93e7700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27cc018930 0x7f27cc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.025 7f27e93e7700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27cc018930 0x7f27cc01ae10 crc :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.045 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e40a11f0 con 0x7f27e40ad730 2021-04-09T18:28:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.048 7f27da7fc700 1 -- 172.21.15.32:0/627517832 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f27e0000960 con 0x7f27e40ad730 2021-04-09T18:28:32.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.230 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f27e40a11f0 con 0x7f27e40ad730 2021-04-09T18:28:32.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.234 7f27da7fc700 1 -- 172.21.15.32:0/627517832 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 26 and data pool 27 v295) v1 ==== 189+0+0 (secure 0 0 0) 0x7f27e0006b00 con 0x7f27e40ad730 2021-04-09T18:28:32.320 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 26 and data pool 27 2021-04-09T18:28:32.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27cc018930 msgr2=0x7f27cc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27cc018930 0x7f27cc01ae10 crc :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40ad730 msgr2=0x7f27e40ab120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40ad730 0x7f27e40ab120 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f27e000b560 tx=0x7f27e000b900).stop 2021-04-09T18:28:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 shutdown_connections 2021-04-09T18:28:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27cc018930 0x7f27cc01ae10 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40a0df0 0x7f27e40a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e40adb30 0x7f27e40a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 --2- 172.21.15.32:0/627517832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27e40ad730 0x7f27e40ab120 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.241 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 >> 172.21.15.32:0/627517832 conn(0x7f27e40b9a90 msgr2=0x7f27e40c1370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:32.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.242 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 shutdown_connections 2021-04-09T18:28:32.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.242 7f27ebdcc700 1 -- 172.21.15.32:0/627517832 wait complete. 2021-04-09T18:28:32.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:28:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.401 7f74da02e700 1 Processor -- start 2021-04-09T18:28:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.401 7f74da02e700 1 -- start start 2021-04-09T18:28:32.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74da02e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d40d7ca0 0x7f74d40d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74da02e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d410d480 con 0x7f74d410d010 2021-04-09T18:28:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74da02e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d40d86c0 con 0x7f74d410da00 2021-04-09T18:28:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74da02e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40d9220 con 0x7f74d40d7ca0 2021-04-09T18:28:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d37fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f74d410d010 0x7f74d40d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35482/0 (socket says 172.21.15.32:35482) 2021-04-09T18:28:32.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d37fe700 1 -- 172.21.15.32:0/1055252653 learned_addr learned my addr 172.21.15.32:0/1055252653 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d2ffd700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d40d7ca0 0x7f74d40d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984606869 0 0) 0x7f74cc001060 con 0x7f74d410d010 2021-04-09T18:28:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74bc002fe0 con 0x7f74d410d010 2021-04-09T18:28:32.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691244730 0 0) 0x7f74c40009b0 con 0x7f74d410da00 2021-04-09T18:28:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74bc003210 con 0x7f74d410da00 2021-04-09T18:28:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1397895752 0 0) 0x7f74cc001250 con 0x7f74d410d010 2021-04-09T18:28:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d40d7ca0 msgr2=0x7f74d40d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d40d7ca0 0x7f74d40d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 >> v1:172.21.15.86:6789/0 conn(0x7f74d410da00 legacy=0x7f74d410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d40d1f70 con 0x7f74d410d010 2021-04-09T18:28:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f74cc001980 con 0x7f74d410d010 2021-04-09T18:28:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 >> v1:172.21.15.32:6789/0 conn(0x7f74d410d010 legacy=0x7f74d40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74bc003ba0 0x7f74bc005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74bc0067b0 0x7f74bc008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.402 7f74d27fc700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74bc0095b0 0x7f74bc00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d2ffd700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74bc0067b0 0x7f74bc008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc050300 con 0x7f74bc0067b0 2021-04-09T18:28:32.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0504b0 con 0x7f74bc003ba0 2021-04-09T18:28:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0506d0 con 0x7f74bc0095b0 2021-04-09T18:28:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d37fe700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74bc003ba0 0x7f74bc005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d3fff700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74bc0095b0 0x7f74bc00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d2ffd700 1 -- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74bc0095b0 msgr2=0x7f74bc00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d2ffd700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74bc0095b0 0x7f74bc00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d2ffd700 1 -- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74bc003ba0 msgr2=0x7f74bc005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d2ffd700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74bc003ba0 0x7f74bc005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d2ffd700 1 -- 172.21.15.32:0/1055252653 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc050970 con 0x7f74bc0067b0 2021-04-09T18:28:32.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d2ffd700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74bc0067b0 0x7f74bc008cb0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f74c8006ac0 tx=0x7f74c8009ab0).ready entity=mon.0 client_cookie=8eb99cf23cfada3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:32.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74c800c0c0 con 0x7f74bc0067b0 2021-04-09T18:28:32.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c800c3c0 con 0x7f74bc0067b0 2021-04-09T18:28:32.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d27fc700 1 -- 172.21.15.32:0/1055252653 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74c800c8a0 con 0x7f74bc0067b0 2021-04-09T18:28:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74da02e700 1 -- 172.21.15.32:0/1055252653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74bc0067b0 msgr2=0x7f74bc008cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74da02e700 1 --2- 172.21.15.32:0/1055252653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74bc0067b0 0x7f74bc008cb0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f74c8006ac0 tx=0x7f74c8009ab0).stop 2021-04-09T18:28:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.403 7f74d3fff700 1 -- 172.21.15.32:0/1055252653 reap_dead start 2021-04-09T18:28:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.404 7f74da02e700 1 -- 172.21.15.32:0/1055252653 shutdown_connections 2021-04-09T18:28:32.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.404 7f74da02e700 1 -- 172.21.15.32:0/1055252653 >> 172.21.15.32:0/1055252653 conn(0x7f74d40ce5f0 msgr2=0x7f74d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.404 7f74da02e700 1 -- 172.21.15.32:0/1055252653 shutdown_connections 2021-04-09T18:28:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.404 7f74da02e700 1 -- 172.21.15.32:0/1055252653 wait complete. 2021-04-09T18:28:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.424 7f74da02e700 1 Processor -- start 2021-04-09T18:28:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.424 7f74da02e700 1 -- start start 2021-04-09T18:28:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74da02e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74d40aa2b0 0x7f74d4187620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74da02e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d4196f70 0x7f74d4187c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74da02e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74d40b7e90 0x7f74d40b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74da02e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40aa6f0 con 0x7f74d40b7e90 2021-04-09T18:28:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74da02e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d41870a0 con 0x7f74d40aa2b0 2021-04-09T18:28:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74da02e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40b88b0 con 0x7f74d4196f70 2021-04-09T18:28:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74d40b7e90 0x7f74d40b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74d40b7e90 0x7f74d40b8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54140/0 (socket says 172.21.15.32:54140) 2021-04-09T18:28:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d3fff700 1 -- 172.21.15.32:0/1193077604 learned_addr learned my addr 172.21.15.32:0/1193077604 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d37fe700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74d40aa2b0 0x7f74d4187620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d2ffd700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d4196f70 0x7f74d4187c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d37fe700 1 -- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d4196f70 msgr2=0x7f74d4187c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d37fe700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d4196f70 0x7f74d4187c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d37fe700 1 -- 172.21.15.32:0/1193077604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74d40b7e90 msgr2=0x7f74d40b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d37fe700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74d40b7e90 0x7f74d40b8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d37fe700 1 -- 172.21.15.32:0/1193077604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d40b8a90 con 0x7f74d40aa2b0 2021-04-09T18:28:32.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d2ffd700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d4196f70 0x7f74d4187c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:32.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d3fff700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74d40b7e90 0x7f74d40b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:32.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74d37fe700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74d40aa2b0 0x7f74d4187620 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f74cc00a850 tx=0x7f74cc00abf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.425 7f74bbfff700 1 -- 172.21.15.32:0/1193077604 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74cc001f10 con 0x7f74d40aa2b0 2021-04-09T18:28:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.426 7f74bbfff700 1 -- 172.21.15.32:0/1193077604 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc004930 con 0x7f74d40aa2b0 2021-04-09T18:28:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.426 7f74da02e700 1 -- 172.21.15.32:0/1193077604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d40b8a90 con 0x7f74d40aa2b0 2021-04-09T18:28:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.426 7f74bbfff700 1 -- 172.21.15.32:0/1193077604 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74cc004dc0 con 0x7f74d40aa2b0 2021-04-09T18:28:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.426 7f74da02e700 1 -- 172.21.15.32:0/1193077604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d40b8a90 con 0x7f74d40aa2b0 2021-04-09T18:28:32.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.426 7f74bbfff700 1 -- 172.21.15.32:0/1193077604 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f74cc004b10 con 0x7f74d40aa2b0 2021-04-09T18:28:32.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.426 7f74bbfff700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74b4018930 0x7f74b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.426 7f74bbfff700 1 -- 172.21.15.32:0/1193077604 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f74cc032dd0 con 0x7f74d40aa2b0 2021-04-09T18:28:32.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.426 7f74d2ffd700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74b4018930 0x7f74b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.427 7f74d2ffd700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74b4018930 0x7f74b401ae10 crc :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:32.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.446 7f74da02e700 1 -- 172.21.15.32:0/1193077604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d40aa6d0 con 0x7f74d40aa2b0 2021-04-09T18:28:32.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.450 7f74bbfff700 1 -- 172.21.15.32:0/1193077604 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f74cc004720 con 0x7f74d40aa2b0 2021-04-09T18:28:32.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.641 7f74da02e700 1 -- 172.21.15.32:0/1193077604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74d40aa6d0 con 0x7f74d40aa2b0 2021-04-09T18:28:32.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.642 7f74bbfff700 1 -- 172.21.15.32:0/1193077604 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f74cc004930 con 0x7f74d40aa2b0 2021-04-09T18:28:32.724 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:32.725 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":154,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:28:32.232002","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:28:29.569175","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-04-09 18:28:30.947070","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:28:32.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 -- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74b4018930 msgr2=0x7f74b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74b4018930 0x7f74b401ae10 crc :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 -- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74d40aa2b0 msgr2=0x7f74d4187620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74d40aa2b0 0x7f74d4187620 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f74cc00a850 tx=0x7f74cc00abf0).stop 2021-04-09T18:28:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 -- 172.21.15.32:0/1193077604 shutdown_connections 2021-04-09T18:28:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f74b4018930 0x7f74b401ae10 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74d40b7e90 0x7f74d40b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74d4196f70 0x7f74d4187c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 --2- 172.21.15.32:0/1193077604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74d40aa2b0 0x7f74d4187620 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 -- 172.21.15.32:0/1193077604 >> 172.21.15.32:0/1193077604 conn(0x7f74d41a13a0 msgr2=0x7f74d4094bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 -- 172.21.15.32:0/1193077604 shutdown_connections 2021-04-09T18:28:32.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.646 7f74da02e700 1 -- 172.21.15.32:0/1193077604 wait complete. 2021-04-09T18:28:32.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:28:32.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.805 7ff04c5aa700 1 Processor -- start 2021-04-09T18:28:32.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.805 7ff04c5aa700 1 -- start start 2021-04-09T18:28:32.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04c5aa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff04410d010 0x7ff0440d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04c5aa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0440d1da0 0x7ff0440d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04c5aa700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff0440d82c0 0x7ff04410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04c5aa700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0440d8d30 con 0x7ff04410d010 2021-04-09T18:28:32.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04c5aa700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0440d9220 con 0x7ff0440d82c0 2021-04-09T18:28:32.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04c5aa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04410d480 con 0x7ff0440d1da0 2021-04-09T18:28:32.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04a3c6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff04410d010 0x7ff0440d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04a3c6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff04410d010 0x7ff0440d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54144/0 (socket says 172.21.15.32:54144) 2021-04-09T18:28:32.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04a3c6700 1 -- 172.21.15.32:0/3875543685 learned_addr learned my addr 172.21.15.32:0/3875543685 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:32.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04abc7700 1 --2- 172.21.15.32:0/3875543685 >> v2:172.21.15.86:3300/0 conn(0x7ff0440d82c0 0x7ff04410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff049bc5700 1 --2- 172.21.15.32:0/3875543685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0440d1da0 0x7ff0440d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04a3c6700 1 -- 172.21.15.32:0/3875543685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0440d1da0 msgr2=0x7ff0440d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:32.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04a3c6700 1 --2- 172.21.15.32:0/3875543685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0440d1da0 0x7ff0440d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04a3c6700 1 -- 172.21.15.32:0/3875543685 >> v2:172.21.15.86:3300/0 conn(0x7ff0440d82c0 msgr2=0x7ff04410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:32.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04a3c6700 1 --2- 172.21.15.32:0/3875543685 >> v2:172.21.15.86:3300/0 conn(0x7ff0440d82c0 0x7ff04410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04a3c6700 1 -- 172.21.15.32:0/3875543685 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0440d86c0 con 0x7ff04410d010 2021-04-09T18:28:32.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04abc7700 1 --2- 172.21.15.32:0/3875543685 >> v2:172.21.15.86:3300/0 conn(0x7ff0440d82c0 0x7ff04410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:32.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff049bc5700 1 --2- 172.21.15.32:0/3875543685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0440d1da0 0x7ff0440d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:32.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff04a3c6700 1 --2- 172.21.15.32:0/3875543685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff04410d010 0x7ff0440d1780 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 rx=0x7ff040006700 tx=0x7ff040009b80).ready entity=mon.0 client_cookie=6e09198776b370c8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:32.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff0493c4700 1 -- 172.21.15.32:0/3875543685 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04000c0c0 con 0x7ff04410d010 2021-04-09T18:28:32.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.806 7ff0493c4700 1 -- 172.21.15.32:0/3875543685 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04000c440 con 0x7ff04410d010 2021-04-09T18:28:32.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff0493c4700 1 -- 172.21.15.32:0/3875543685 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04000c920 con 0x7ff04410d010 2021-04-09T18:28:32.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff04c5aa700 1 -- 172.21.15.32:0/3875543685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff04410d010 msgr2=0x7ff0440d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff04c5aa700 1 --2- 172.21.15.32:0/3875543685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff04410d010 0x7ff0440d1780 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 rx=0x7ff040006700 tx=0x7ff040009b80).stop 2021-04-09T18:28:32.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff04c5aa700 1 -- 172.21.15.32:0/3875543685 shutdown_connections 2021-04-09T18:28:32.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff04c5aa700 1 --2- 172.21.15.32:0/3875543685 >> v2:172.21.15.86:3300/0 conn(0x7ff0440d82c0 0x7ff04410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff04c5aa700 1 --2- 172.21.15.32:0/3875543685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0440d1da0 0x7ff0440d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff04c5aa700 1 --2- 172.21.15.32:0/3875543685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff04410d010 0x7ff0440d1780 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff04c5aa700 1 -- 172.21.15.32:0/3875543685 >> 172.21.15.32:0/3875543685 conn(0x7ff0440ce5f0 msgr2=0x7ff0440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:32.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff04c5aa700 1 -- 172.21.15.32:0/3875543685 shutdown_connections 2021-04-09T18:28:32.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.807 7ff04c5aa700 1 -- 172.21.15.32:0/3875543685 wait complete. 2021-04-09T18:28:32.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04c5aa700 1 Processor -- start 2021-04-09T18:28:32.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04c5aa700 1 -- start start 2021-04-09T18:28:32.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04c5aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0441a9580 0x7ff0441a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04c5aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff044187040 0x7ff044187440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04c5aa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0441aa260 0x7ff0441aa660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04c5aa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0441a90c0 con 0x7ff044187040 2021-04-09T18:28:32.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04c5aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0441a9980 con 0x7ff0441a9580 2021-04-09T18:28:32.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04c5aa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff044187a60 con 0x7ff0441aa260 2021-04-09T18:28:32.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff049bc5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff044187040 0x7ff044187440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff049bc5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff044187040 0x7ff044187440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54150/0 (socket says 172.21.15.32:54150) 2021-04-09T18:28:32.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff049bc5700 1 -- 172.21.15.32:0/1065396945 learned_addr learned my addr 172.21.15.32:0/1065396945 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:32.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04abc7700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0441aa260 0x7ff0441aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.828 7ff04a3c6700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0441a9580 0x7ff0441a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff049bc5700 1 -- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0441aa260 msgr2=0x7ff0441aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff049bc5700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0441aa260 0x7ff0441aa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff049bc5700 1 -- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0441a9580 msgr2=0x7ff0441a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:32.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff049bc5700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0441a9580 0x7ff0441a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff049bc5700 1 -- 172.21.15.32:0/1065396945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04417f450 con 0x7ff044187040 2021-04-09T18:28:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff04a3c6700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0441a9580 0x7ff0441a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff04abc7700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0441aa260 0x7ff0441aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:32.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff049bc5700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff044187040 0x7ff044187440 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7ff038006f20 tx=0x7ff03800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:32.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff036ffd700 1 -- 172.21.15.32:0/1065396945 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff038006bb0 con 0x7ff044187040 2021-04-09T18:28:32.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff036ffd700 1 -- 172.21.15.32:0/1065396945 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03800bd80 con 0x7ff044187040 2021-04-09T18:28:32.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04417f450 con 0x7ff044187040 2021-04-09T18:28:32.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff036ffd700 1 -- 172.21.15.32:0/1065396945 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff038002220 con 0x7ff044187040 2021-04-09T18:28:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.829 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04417f450 con 0x7ff044187040 2021-04-09T18:28:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.830 7ff036ffd700 1 -- 172.21.15.32:0/1065396945 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff03800a000 con 0x7ff044187040 2021-04-09T18:28:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.830 7ff036ffd700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff02c018930 0x7ff02c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:32.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.830 7ff036ffd700 1 -- 172.21.15.32:0/1065396945 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff038032a10 con 0x7ff044187040 2021-04-09T18:28:32.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.830 7ff04a3c6700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff02c018930 0x7ff02c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:32.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.830 7ff04a3c6700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff02c018930 0x7ff02c01ae10 crc :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:32.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.850 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0441aaf70 con 0x7ff044187040 2021-04-09T18:28:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:32.853 7ff036ffd700 1 -- 172.21.15.32:0/1065396945 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff038000c90 con 0x7ff044187040 2021-04-09T18:28:33.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.042 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0441aaf70 con 0x7ff044187040 2021-04-09T18:28:33.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.042 7ff036ffd700 1 -- 172.21.15.32:0/1065396945 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+14020 (secure 0 0 0) 0x7ff03800a000 con 0x7ff044187040 2021-04-09T18:28:33.124 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:33.124 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":154,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:28:32.232002","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:28:29.569175","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-04-09 18:28:30.947070","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:28:33.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff02c018930 msgr2=0x7ff02c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:33.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff02c018930 0x7ff02c01ae10 crc :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff044187040 msgr2=0x7ff044187440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:33.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff044187040 0x7ff044187440 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7ff038006f20 tx=0x7ff03800b990).stop 2021-04-09T18:28:33.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 shutdown_connections 2021-04-09T18:28:33.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff02c018930 0x7ff02c01ae10 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff044187040 0x7ff044187440 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0441aa260 0x7ff0441aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 --2- 172.21.15.32:0/1065396945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0441a9580 0x7ff0441a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 >> 172.21.15.32:0/1065396945 conn(0x7ff0441911a0 msgr2=0x7ff0441a52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:33.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 shutdown_connections 2021-04-09T18:28:33.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.045 7ff04c5aa700 1 -- 172.21.15.32:0/1065396945 wait complete. 2021-04-09T18:28:33.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:28:33.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.207 7f91d8619700 1 Processor -- start 2021-04-09T18:28:33.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.207 7f91d8619700 1 -- start start 2021-04-09T18:28:33.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.208 7f91d8619700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f91d010d010 0x7f91d00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:33.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.208 7f91d8619700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f91d00d1da0 0x7f91d00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:33.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.208 7f91d8619700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d010d480 con 0x7f91d00d82c0 2021-04-09T18:28:33.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.208 7f91d8619700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d00d8d30 con 0x7f91d010d010 2021-04-09T18:28:33.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.208 7f91d8619700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d00d9220 con 0x7f91d00d1da0 2021-04-09T18:28:33.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.208 7f91d6435700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f91d010d010 0x7f91d00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:33.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.208 7f91d6c36700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f91d00d82c0 0x7f91d010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35518/0 (socket says 172.21.15.32:35518) 2021-04-09T18:28:33.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.208 7f91d6c36700 1 -- 172.21.15.32:0/903223500 learned_addr learned my addr 172.21.15.32:0/903223500 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:33.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d5c34700 1 --2- 172.21.15.32:0/903223500 >> v2:172.21.15.86:3300/0 conn(0x7f91d00d1da0 0x7f91d00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:33.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d5433700 1 -- 172.21.15.32:0/903223500 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777832600 0 0) 0x7f91c8000d30 con 0x7f91d00d82c0 2021-04-09T18:28:33.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d5433700 1 -- 172.21.15.32:0/903223500 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91bc002fe0 con 0x7f91d00d82c0 2021-04-09T18:28:33.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d6435700 1 -- 172.21.15.32:0/903223500 >> v2:172.21.15.86:3300/0 conn(0x7f91d00d1da0 msgr2=0x7f91d00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:33.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d6435700 1 --2- 172.21.15.32:0/903223500 >> v2:172.21.15.86:3300/0 conn(0x7f91d00d1da0 0x7f91d00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d6435700 1 -- 172.21.15.32:0/903223500 >> v1:172.21.15.32:6789/0 conn(0x7f91d00d82c0 legacy=0x7f91d010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:33.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d6435700 1 -- 172.21.15.32:0/903223500 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d00d86f0 con 0x7f91d010d010 2021-04-09T18:28:33.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d6435700 1 --2- 172.21.15.32:0/903223500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d010d010 0x7f91d00d1780 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7f91cc006650 tx=0x7f91cc009a00).ready entity=mon.0 client_cookie=3dabf9bcd482aa31 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:33.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d5433700 1 -- 172.21.15.32:0/903223500 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91cc009df0 con 0x7f91d010d010 2021-04-09T18:28:33.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d5433700 1 -- 172.21.15.32:0/903223500 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc0009b0 con 0x7f91d010d010 2021-04-09T18:28:33.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d5433700 1 -- 172.21.15.32:0/903223500 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91cc00c2b0 con 0x7f91d010d010 2021-04-09T18:28:33.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d8619700 1 -- 172.21.15.32:0/903223500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d010d010 msgr2=0x7f91d00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:33.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.209 7f91d8619700 1 --2- 172.21.15.32:0/903223500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d010d010 0x7f91d00d1780 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7f91cc006650 tx=0x7f91cc009a00).stop 2021-04-09T18:28:33.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.210 7f91d8619700 1 -- 172.21.15.32:0/903223500 shutdown_connections 2021-04-09T18:28:33.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.210 7f91d8619700 1 --2- 172.21.15.32:0/903223500 >> v2:172.21.15.86:3300/0 conn(0x7f91d00d1da0 0x7f91d00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.210 7f91d8619700 1 --2- 172.21.15.32:0/903223500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d010d010 0x7f91d00d1780 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.210 7f91d8619700 1 -- 172.21.15.32:0/903223500 >> 172.21.15.32:0/903223500 conn(0x7f91d00ce5f0 msgr2=0x7f91d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:33.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.210 7f91d8619700 1 -- 172.21.15.32:0/903223500 shutdown_connections 2021-04-09T18:28:33.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.210 7f91d8619700 1 -- 172.21.15.32:0/903223500 wait complete. 2021-04-09T18:28:33.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.230 7f91d8619700 1 Processor -- start 2021-04-09T18:28:33.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.230 7f91d8619700 1 -- start start 2021-04-09T18:28:33.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d8619700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d0179bf0 0x7f91d0179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:33.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d8619700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d0094f60 0x7f91d0095360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:33.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d8619700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d0095980 0x7f91d009ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:33.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d8619700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d01796f0 con 0x7f91d0179bf0 2021-04-09T18:28:33.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d8619700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d017a610 con 0x7f91d0094f60 2021-04-09T18:28:33.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d8619700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d009b230 con 0x7f91d0095980 2021-04-09T18:28:33.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6435700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d0179bf0 0x7f91d0179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:33.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6435700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d0179bf0 0x7f91d0179ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54164/0 (socket says 172.21.15.32:54164) 2021-04-09T18:28:33.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6435700 1 -- 172.21.15.32:0/3001206683 learned_addr learned my addr 172.21.15.32:0/3001206683 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:33.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6c36700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d0095980 0x7f91d009ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:33.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d5c34700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d0094f60 0x7f91d0095360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:33.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6435700 1 -- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d0095980 msgr2=0x7f91d009ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:33.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6435700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d0095980 0x7f91d009ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6435700 1 -- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d0094f60 msgr2=0x7f91d0095360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:33.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6435700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d0094f60 0x7f91d0095360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:33.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6435700 1 -- 172.21.15.32:0/3001206683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d009b450 con 0x7f91d0179bf0 2021-04-09T18:28:33.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d6435700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d0179bf0 0x7f91d0179ff0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7f91cc0096a0 tx=0x7f91cc00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:33.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91c6ffd700 1 -- 172.21.15.32:0/3001206683 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91cc0093a0 con 0x7f91d0179bf0 2021-04-09T18:28:33.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91c6ffd700 1 -- 172.21.15.32:0/3001206683 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc0067f0 con 0x7f91d0179bf0 2021-04-09T18:28:33.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91d8619700 1 -- 172.21.15.32:0/3001206683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d009b450 con 0x7f91d0179bf0 2021-04-09T18:28:33.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.231 7f91c6ffd700 1 -- 172.21.15.32:0/3001206683 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91cc0022b0 con 0x7f91d0179bf0 2021-04-09T18:28:33.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.232 7f91d8619700 1 -- 172.21.15.32:0/3001206683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d009b450 con 0x7f91d0179bf0 2021-04-09T18:28:33.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.232 7f91c6ffd700 1 -- 172.21.15.32:0/3001206683 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f91cc000c90 con 0x7f91d0179bf0 2021-04-09T18:28:33.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.233 7f91c6ffd700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b80188c0 0x7f91b801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:33.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.233 7f91c6ffd700 1 -- 172.21.15.32:0/3001206683 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f91cc0327a0 con 0x7f91d0179bf0 2021-04-09T18:28:33.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.233 7f91d5c34700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b80188c0 0x7f91b801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:33.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.233 7f91d5c34700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b80188c0 0x7f91b801adc0 crc :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:33.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.252 7f91d8619700 1 -- 172.21.15.32:0/3001206683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d010d480 con 0x7f91d0179bf0 2021-04-09T18:28:33.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.256 7f91c6ffd700 1 -- 172.21.15.32:0/3001206683 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f91cc0052e0 con 0x7f91d0179bf0 2021-04-09T18:28:33.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:33.452 7f91d8619700 1 -- 172.21.15.32:0/3001206683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f91d010d480 con 0x7f91d0179bf0 2021-04-09T18:28:34.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.238 7f91c6ffd700 1 -- 172.21.15.32:0/3001206683 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v297) v1 ==== 118+0+0 (secure 0 0 0) 0x7f91cc000c90 con 0x7f91d0179bf0 2021-04-09T18:28:34.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 -- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b80188c0 msgr2=0x7f91b801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b80188c0 0x7f91b801adc0 crc :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 -- 172.21.15.32:0/3001206683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d0179bf0 msgr2=0x7f91d0179ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d0179bf0 0x7f91d0179ff0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7f91cc0096a0 tx=0x7f91cc00b8e0).stop 2021-04-09T18:28:34.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 -- 172.21.15.32:0/3001206683 shutdown_connections 2021-04-09T18:28:34.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f91b80188c0 0x7f91b801adc0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91d0095980 0x7f91d009ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91d0094f60 0x7f91d0095360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 --2- 172.21.15.32:0/3001206683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91d0179bf0 0x7f91d0179ff0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 -- 172.21.15.32:0/3001206683 >> 172.21.15.32:0/3001206683 conn(0x7f91d00c4290 msgr2=0x7f91d00c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:34.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 -- 172.21.15.32:0/3001206683 shutdown_connections 2021-04-09T18:28:34.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.270 7f91d8619700 1 -- 172.21.15.32:0/3001206683 wait complete. 2021-04-09T18:28:34.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:34.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.436 7fce873ec700 1 Processor -- start 2021-04-09T18:28:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.436 7fce873ec700 1 -- start start 2021-04-09T18:28:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.437 7fce873ec700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fce8010d010 0x7fce800d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.437 7fce873ec700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce800d8490 0x7fce8010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.437 7fce873ec700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8010d480 con 0x7fce800d1da0 2021-04-09T18:28:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.437 7fce873ec700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce800d8d30 con 0x7fce8010d010 2021-04-09T18:28:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.437 7fce873ec700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce800d9220 con 0x7fce800d8490 2021-04-09T18:28:34.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.437 7fce84a07700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fce800d1da0 0x7fce800d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35532/0 (socket says 172.21.15.32:35532) 2021-04-09T18:28:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.437 7fce85208700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fce8010d010 0x7fce800d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.437 7fce84a07700 1 -- 172.21.15.32:0/1837507073 learned_addr learned my addr 172.21.15.32:0/1837507073 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.437 7fce85a09700 1 --2- 172.21.15.32:0/1837507073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce800d8490 0x7fce8010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce77fff700 1 -- 172.21.15.32:0/1837507073 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517805344 0 0) 0x7fce70000ce0 con 0x7fce800d1da0 2021-04-09T18:28:34.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce77fff700 1 -- 172.21.15.32:0/1837507073 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce6c002fe0 con 0x7fce800d1da0 2021-04-09T18:28:34.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce85208700 1 -- 172.21.15.32:0/1837507073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce800d8490 msgr2=0x7fce8010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:34.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce85208700 1 --2- 172.21.15.32:0/1837507073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce800d8490 0x7fce8010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce85208700 1 -- 172.21.15.32:0/1837507073 >> v1:172.21.15.32:6789/0 conn(0x7fce800d1da0 legacy=0x7fce800d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce85208700 1 -- 172.21.15.32:0/1837507073 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8010e020 con 0x7fce8010d010 2021-04-09T18:28:34.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce85208700 1 --2- 172.21.15.32:0/1837507073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8010d010 0x7fce800d1780 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7fce7c006650 tx=0x7fce7c009a80).ready entity=mon.0 client_cookie=325ae3439091f2f6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:34.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce77fff700 1 -- 172.21.15.32:0/1837507073 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c0009b0 con 0x7fce8010d010 2021-04-09T18:28:34.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce77fff700 1 -- 172.21.15.32:0/1837507073 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c00c0c0 con 0x7fce8010d010 2021-04-09T18:28:34.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce77fff700 1 -- 172.21.15.32:0/1837507073 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c0009b0 con 0x7fce8010d010 2021-04-09T18:28:34.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce873ec700 1 -- 172.21.15.32:0/1837507073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8010d010 msgr2=0x7fce800d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.438 7fce873ec700 1 --2- 172.21.15.32:0/1837507073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8010d010 0x7fce800d1780 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7fce7c006650 tx=0x7fce7c009a80).stop 2021-04-09T18:28:34.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.439 7fce873ec700 1 -- 172.21.15.32:0/1837507073 shutdown_connections 2021-04-09T18:28:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.439 7fce873ec700 1 --2- 172.21.15.32:0/1837507073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce800d8490 0x7fce8010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.439 7fce873ec700 1 --2- 172.21.15.32:0/1837507073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8010d010 0x7fce800d1780 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.439 7fce873ec700 1 -- 172.21.15.32:0/1837507073 >> 172.21.15.32:0/1837507073 conn(0x7fce800ce5f0 msgr2=0x7fce800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:34.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.439 7fce873ec700 1 -- 172.21.15.32:0/1837507073 shutdown_connections 2021-04-09T18:28:34.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.439 7fce873ec700 1 -- 172.21.15.32:0/1837507073 wait complete. 2021-04-09T18:28:34.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.459 7fce873ec700 1 Processor -- start 2021-04-09T18:28:34.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.459 7fce873ec700 1 -- start start 2021-04-09T18:28:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce873ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce800b42d0 0x7fce80183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce873ec700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce801ac090 0x7fce801ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce873ec700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80184d30 0x7fce80185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce873ec700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce800a1a90 con 0x7fce800b42d0 2021-04-09T18:28:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce873ec700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce801acab0 con 0x7fce801ac090 2021-04-09T18:28:34.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce873ec700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce80185a00 con 0x7fce80184d30 2021-04-09T18:28:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85208700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce800b42d0 0x7fce80183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85208700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce800b42d0 0x7fce80183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54178/0 (socket says 172.21.15.32:54178) 2021-04-09T18:28:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85208700 1 -- 172.21.15.32:0/2998285553 learned_addr learned my addr 172.21.15.32:0/2998285553 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85a09700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80184d30 0x7fce80185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce84a07700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce801ac090 0x7fce801ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85208700 1 -- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80184d30 msgr2=0x7fce80185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85208700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80184d30 0x7fce80185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85208700 1 -- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce801ac090 msgr2=0x7fce801ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85208700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce801ac090 0x7fce801ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85208700 1 -- 172.21.15.32:0/2998285553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce80185750 con 0x7fce800b42d0 2021-04-09T18:28:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.460 7fce85208700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce800b42d0 0x7fce80183520 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7fce7c009650 tx=0x7fce7c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:34.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.461 7fce75ffb700 1 -- 172.21.15.32:0/2998285553 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c0093a0 con 0x7fce800b42d0 2021-04-09T18:28:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.461 7fce75ffb700 1 -- 172.21.15.32:0/2998285553 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c006070 con 0x7fce800b42d0 2021-04-09T18:28:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.461 7fce873ec700 1 -- 172.21.15.32:0/2998285553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce80185750 con 0x7fce800b42d0 2021-04-09T18:28:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.461 7fce75ffb700 1 -- 172.21.15.32:0/2998285553 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c0052e0 con 0x7fce800b42d0 2021-04-09T18:28:34.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.461 7fce873ec700 1 -- 172.21.15.32:0/2998285553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce80185750 con 0x7fce800b42d0 2021-04-09T18:28:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.461 7fce75ffb700 1 -- 172.21.15.32:0/2998285553 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fce7c009300 con 0x7fce800b42d0 2021-04-09T18:28:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.461 7fce75ffb700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce600188d0 0x7fce6001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.461 7fce75ffb700 1 -- 172.21.15.32:0/2998285553 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fce7c032a00 con 0x7fce800b42d0 2021-04-09T18:28:34.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.462 7fce84a07700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce600188d0 0x7fce6001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.462 7fce84a07700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce600188d0 0x7fce6001add0 crc :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:34.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.481 7fce873ec700 1 -- 172.21.15.32:0/2998285553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce80182d10 con 0x7fce800b42d0 2021-04-09T18:28:34.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.485 7fce75ffb700 1 -- 172.21.15.32:0/2998285553 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fce7c006030 con 0x7fce800b42d0 2021-04-09T18:28:34.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.678 7fce873ec700 1 -- 172.21.15.32:0/2998285553 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce8010d480 con 0x7fce800b42d0 2021-04-09T18:28:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.679 7fce75ffb700 1 -- 172.21.15.32:0/2998285553 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fce7c000940 con 0x7fce800b42d0 2021-04-09T18:28:34.759 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2021-04-09T18:28:34.760 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:34.760 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:34.236678","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9868":{"gid":9868,"name":"a","rank":0,"incarnation":295,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1939434763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1939434763},{"type":"v1","addr":"172.21.15.32:6835","nonce":1939434763}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:34.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 -- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce600188d0 msgr2=0x7fce6001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce600188d0 0x7fce6001add0 crc :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 -- 172.21.15.32:0/2998285553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce800b42d0 msgr2=0x7fce80183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce800b42d0 0x7fce80183520 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7fce7c009650 tx=0x7fce7c00b8e0).stop 2021-04-09T18:28:34.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 -- 172.21.15.32:0/2998285553 shutdown_connections 2021-04-09T18:28:34.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce600188d0 0x7fce6001add0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80184d30 0x7fce80185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce801ac090 0x7fce801ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 --2- 172.21.15.32:0/2998285553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce800b42d0 0x7fce80183520 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 -- 172.21.15.32:0/2998285553 >> 172.21.15.32:0/2998285553 conn(0x7fce8019b890 msgr2=0x7fce801a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.681 7fce873ec700 1 -- 172.21.15.32:0/2998285553 shutdown_connections 2021-04-09T18:28:34.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.682 7fce873ec700 1 -- 172.21.15.32:0/2998285553 wait complete. 2021-04-09T18:28:34.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:28:34.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.841 7f8ebb16f700 1 Processor -- start 2021-04-09T18:28:34.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.841 7f8ebb16f700 1 -- start start 2021-04-09T18:28:34.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.841 7f8ebb16f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8eb410d010 0x7f8eb40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8ebb16f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb410d480 con 0x7f8eb40d8490 2021-04-09T18:28:34.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8ebb16f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb410e1f0 con 0x7f8eb40d1da0 2021-04-09T18:28:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8ebb16f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb40d8d30 con 0x7f8eb410d010 2021-04-09T18:28:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb8f8b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8eb410d010 0x7f8eb40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb978c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8eb40d8490 0x7f8eb410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35544/0 (socket says 172.21.15.32:35544) 2021-04-09T18:28:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb978c700 1 -- 172.21.15.32:0/1253686938 learned_addr learned my addr 172.21.15.32:0/1253686938 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:34.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb8f8b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8eb410d010 0x7f8eb40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54186/0 (socket says 172.21.15.32:54186) 2021-04-09T18:28:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb8f8b700 1 -- 172.21.15.32:0/1253686938 >> v1:172.21.15.86:6789/0 conn(0x7f8eb40d1da0 legacy=0x7f8eb40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb8f8b700 1 -- 172.21.15.32:0/1253686938 >> v1:172.21.15.32:6789/0 conn(0x7f8eb40d8490 legacy=0x7f8eb410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb37fe700 1 -- 172.21.15.32:0/1253686938 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975118270 0 0) 0x7f8eac0009b0 con 0x7f8eb40d8490 2021-04-09T18:28:34.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb8f8b700 1 -- 172.21.15.32:0/1253686938 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb40d9220 con 0x7f8eb410d010 2021-04-09T18:28:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb8f8b700 1 --2- 172.21.15.32:0/1253686938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb410d010 0x7f8eb40d1780 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 rx=0x7f8e9c006650 tx=0x7f8e9c009a00).ready entity=mon.0 client_cookie=134ebcf85f761eca server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb37fe700 1 -- 172.21.15.32:0/1253686938 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e9c009df0 con 0x7f8eb410d010 2021-04-09T18:28:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb37fe700 1 -- 172.21.15.32:0/1253686938 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c0009b0 con 0x7f8eb410d010 2021-04-09T18:28:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8eb37fe700 1 -- 172.21.15.32:0/1253686938 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e9c00c2b0 con 0x7f8eb410d010 2021-04-09T18:28:34.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8ebb16f700 1 -- 172.21.15.32:0/1253686938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb410d010 msgr2=0x7f8eb40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.842 7f8ebb16f700 1 --2- 172.21.15.32:0/1253686938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb410d010 0x7f8eb40d1780 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 rx=0x7f8e9c006650 tx=0x7f8e9c009a00).stop 2021-04-09T18:28:34.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.843 7f8ebb16f700 1 -- 172.21.15.32:0/1253686938 shutdown_connections 2021-04-09T18:28:34.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.843 7f8ebb16f700 1 --2- 172.21.15.32:0/1253686938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb410d010 0x7f8eb40d1780 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.843 7f8ebb16f700 1 -- 172.21.15.32:0/1253686938 >> 172.21.15.32:0/1253686938 conn(0x7f8eb40ce5f0 msgr2=0x7f8eb40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.843 7f8ebb16f700 1 -- 172.21.15.32:0/1253686938 shutdown_connections 2021-04-09T18:28:34.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.843 7f8ebb16f700 1 -- 172.21.15.32:0/1253686938 wait complete. 2021-04-09T18:28:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.863 7f8ebb16f700 1 Processor -- start 2021-04-09T18:28:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.863 7f8ebb16f700 1 -- start start 2021-04-09T18:28:34.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8ebb16f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb4192a10 0x7f8eb41ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8ebb16f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41acaa0 0x7f8eb41aa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8ebb16f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb41aad30 0x7f8eb41ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8ebb16f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41abf00 con 0x7f8eb41acaa0 2021-04-09T18:28:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8ebb16f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb41acea0 con 0x7f8eb4192a10 2021-04-09T18:28:34.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8ebb16f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb4180860 con 0x7f8eb41aad30 2021-04-09T18:28:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41acaa0 0x7f8eb41aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41acaa0 0x7f8eb41aa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54192/0 (socket says 172.21.15.32:54192) 2021-04-09T18:28:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb3fff700 1 -- 172.21.15.32:0/4168626626 learned_addr learned my addr 172.21.15.32:0/4168626626 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:34.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb8f8b700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb4192a10 0x7f8eb41ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb978c700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb41aad30 0x7f8eb41ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb3fff700 1 -- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb41aad30 msgr2=0x7f8eb41ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb3fff700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb41aad30 0x7f8eb41ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb3fff700 1 -- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb4192a10 msgr2=0x7f8eb41ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:34.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb3fff700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb4192a10 0x7f8eb41ac480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.864 7f8eb3fff700 1 -- 172.21.15.32:0/4168626626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb41ab750 con 0x7f8eb41acaa0 2021-04-09T18:28:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.865 7f8eb3fff700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41acaa0 0x7f8eb41aa710 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f8ea4009940 tx=0x7f8ea400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.865 7f8eb17fa700 1 -- 172.21.15.32:0/4168626626 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ea4000fe0 con 0x7f8eb41acaa0 2021-04-09T18:28:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.865 7f8eb17fa700 1 -- 172.21.15.32:0/4168626626 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea400c2f0 con 0x7f8eb41acaa0 2021-04-09T18:28:34.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.865 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb41ab750 con 0x7f8eb41acaa0 2021-04-09T18:28:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.865 7f8eb17fa700 1 -- 172.21.15.32:0/4168626626 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ea400c7d0 con 0x7f8eb41acaa0 2021-04-09T18:28:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.865 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb41abce0 con 0x7f8eb41acaa0 2021-04-09T18:28:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.865 7f8eb17fa700 1 -- 172.21.15.32:0/4168626626 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8ea4001230 con 0x7f8eb41acaa0 2021-04-09T18:28:34.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.865 7f8eb17fa700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e98018930 0x7f8e9801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.865 7f8eb17fa700 1 -- 172.21.15.32:0/4168626626 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8ea4032d90 con 0x7f8eb41acaa0 2021-04-09T18:28:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.866 7f8eb8f8b700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e98018930 0x7f8e9801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:34.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.866 7f8eb8f8b700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e98018930 0x7f8e9801ae10 crc :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:34.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.886 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb417ca90 con 0x7f8eb41acaa0 2021-04-09T18:28:34.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:34.889 7f8eb17fa700 1 -- 172.21.15.32:0/4168626626 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8ea400be30 con 0x7f8eb41acaa0 2021-04-09T18:28:35.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.076 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8eb410d480 con 0x7f8eb41acaa0 2021-04-09T18:28:35.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.077 7f8eb17fa700 1 -- 172.21.15.32:0/4168626626 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f8ea4001230 con 0x7f8eb41acaa0 2021-04-09T18:28:35.157 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:35.158 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":155,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:28:33.233832","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:28:29.569175","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-04-09 18:28:30.947070","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:28:35.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e98018930 msgr2=0x7f8e9801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e98018930 0x7f8e9801ae10 crc :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41acaa0 msgr2=0x7f8eb41aa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41acaa0 0x7f8eb41aa710 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f8ea4009940 tx=0x7f8ea400bcf0).stop 2021-04-09T18:28:35.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 shutdown_connections 2021-04-09T18:28:35.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e98018930 0x7f8e9801ae10 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41acaa0 0x7f8eb41aa710 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb41aad30 0x7f8eb41ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 --2- 172.21.15.32:0/4168626626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb4192a10 0x7f8eb41ac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.079 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 >> 172.21.15.32:0/4168626626 conn(0x7f8eb4188510 msgr2=0x7f8eb41a7320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:35.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.080 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 shutdown_connections 2021-04-09T18:28:35.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.080 7f8ebb16f700 1 -- 172.21.15.32:0/4168626626 wait complete. 2021-04-09T18:28:35.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:28:35.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.250 7fa3a1491700 1 Processor -- start 2021-04-09T18:28:35.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.250 7fa3a1491700 1 -- start start 2021-04-09T18:28:35.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa3a1491700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa39c0cffb0 0x7fa39c087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa3a1491700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c0d8050 0x7fa39c10d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa3a1491700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa39c088460 con 0x7fa39c0d7c50 2021-04-09T18:28:35.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa3a1491700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c1b0c80 con 0x7fa39c0cffb0 2021-04-09T18:28:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa3a1491700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0d91f0 con 0x7fa39c0d8050 2021-04-09T18:28:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39287c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa39c0d7c50 0x7fa39c1b0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35560/0 (socket says 172.21.15.32:35560) 2021-04-09T18:28:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39287c700 1 -- 172.21.15.32:0/529831902 learned_addr learned my addr 172.21.15.32:0/529831902 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:35.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39b87e700 1 --2- 172.21.15.32:0/529831902 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c0d8050 0x7fa39c10d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39b07d700 1 --2- 172.21.15.32:0/529831902 >> v2:172.21.15.86:3300/0 conn(0x7fa39c0cffb0 0x7fa39c087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39a87c700 1 -- 172.21.15.32:0/529831902 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340666721 0 0) 0x7fa3840009b0 con 0x7fa39c0d7c50 2021-04-09T18:28:35.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39a87c700 1 -- 172.21.15.32:0/529831902 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa388002fe0 con 0x7fa39c0d7c50 2021-04-09T18:28:35.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39b07d700 1 -- 172.21.15.32:0/529831902 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c0d8050 msgr2=0x7fa39c10d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39b07d700 1 --2- 172.21.15.32:0/529831902 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c0d8050 0x7fa39c10d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39b07d700 1 -- 172.21.15.32:0/529831902 >> v1:172.21.15.32:6789/0 conn(0x7fa39c0d7c50 legacy=0x7fa39c1b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39b07d700 1 -- 172.21.15.32:0/529831902 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c0d8620 con 0x7fa39c0cffb0 2021-04-09T18:28:35.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39b87e700 1 --2- 172.21.15.32:0/529831902 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c0d8050 0x7fa39c10d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:35.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.251 7fa39b07d700 1 --2- 172.21.15.32:0/529831902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0cffb0 0x7fa39c087e40 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fa38c006650 tx=0x7fa38c009a80).ready entity=mon.1 client_cookie=53fcb94999df887e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:35.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.252 7fa39a87c700 1 -- 172.21.15.32:0/529831902 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa38c0009b0 con 0x7fa39c0cffb0 2021-04-09T18:28:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.252 7fa39a87c700 1 -- 172.21.15.32:0/529831902 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38c00c0c0 con 0x7fa39c0cffb0 2021-04-09T18:28:35.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.252 7fa39a87c700 1 -- 172.21.15.32:0/529831902 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa38c00c510 con 0x7fa39c0cffb0 2021-04-09T18:28:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.252 7fa3a1491700 1 -- 172.21.15.32:0/529831902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0cffb0 msgr2=0x7fa39c087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.252 7fa3a1491700 1 --2- 172.21.15.32:0/529831902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0cffb0 0x7fa39c087e40 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fa38c006650 tx=0x7fa38c009a80).stop 2021-04-09T18:28:35.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.252 7fa3a1491700 1 -- 172.21.15.32:0/529831902 shutdown_connections 2021-04-09T18:28:35.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.252 7fa3a1491700 1 --2- 172.21.15.32:0/529831902 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c0d8050 0x7fa39c10d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.252 7fa3a1491700 1 --2- 172.21.15.32:0/529831902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0cffb0 0x7fa39c087e40 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.252 7fa3a1491700 1 -- 172.21.15.32:0/529831902 >> 172.21.15.32:0/529831902 conn(0x7fa39c0ce5b0 msgr2=0x7fa39c0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:35.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.253 7fa3a1491700 1 -- 172.21.15.32:0/529831902 shutdown_connections 2021-04-09T18:28:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.253 7fa3a1491700 1 -- 172.21.15.32:0/529831902 wait complete. 2021-04-09T18:28:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.272 7fa3a1491700 1 Processor -- start 2021-04-09T18:28:35.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa3a1491700 1 -- start start 2021-04-09T18:28:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa3a1491700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa39c0b1950 0x7fa39c183e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa3a1491700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c1795e0 0x7fa39c094eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa3a1491700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c1799e0 0x7fa39c0954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa3a1491700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0b13d0 con 0x7fa39c0b1950 2021-04-09T18:28:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa3a1491700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0b1d50 con 0x7fa39c1799e0 2021-04-09T18:28:35.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa3a1491700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c19e4a0 con 0x7fa39c1795e0 2021-04-09T18:28:35.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa39b07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa39c0b1950 0x7fa39c183e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa39b07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa39c0b1950 0x7fa39c183e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54204/0 (socket says 172.21.15.32:54204) 2021-04-09T18:28:35.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa39b07d700 1 -- 172.21.15.32:0/3143624114 learned_addr learned my addr 172.21.15.32:0/3143624114 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:35.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa39b87e700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c1799e0 0x7fa39c0954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.273 7fa39287c700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c1795e0 0x7fa39c094eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa39b07d700 1 -- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c1795e0 msgr2=0x7fa39c094eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa39b07d700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c1795e0 0x7fa39c094eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa39b07d700 1 -- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c1799e0 msgr2=0x7fa39c0954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa39b07d700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c1799e0 0x7fa39c0954d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa39b07d700 1 -- 172.21.15.32:0/3143624114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c0b0180 con 0x7fa39c0b1950 2021-04-09T18:28:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa39b07d700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa39c0b1950 0x7fa39c183e60 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7fa38c009650 tx=0x7fa38c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa398878700 1 -- 172.21.15.32:0/3143624114 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa38c0093a0 con 0x7fa39c0b1950 2021-04-09T18:28:35.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa398878700 1 -- 172.21.15.32:0/3143624114 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38c00bd50 con 0x7fa39c0b1950 2021-04-09T18:28:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c0b0180 con 0x7fa39c0b1950 2021-04-09T18:28:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa398878700 1 -- 172.21.15.32:0/3143624114 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa38c0052e0 con 0x7fa39c0b1950 2021-04-09T18:28:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.274 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c0b0180 con 0x7fa39c0b1950 2021-04-09T18:28:35.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.275 7fa398878700 1 -- 172.21.15.32:0/3143624114 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa38c006030 con 0x7fa39c0b1950 2021-04-09T18:28:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.275 7fa398878700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa380018930 0x7fa38001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.275 7fa398878700 1 -- 172.21.15.32:0/3143624114 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa38c0329e0 con 0x7fa39c0b1950 2021-04-09T18:28:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.275 7fa39287c700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa380018930 0x7fa38001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.275 7fa39287c700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa380018930 0x7fa38001ae10 crc :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:35.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.295 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c093fc0 con 0x7fa39c0b1950 2021-04-09T18:28:35.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.298 7fa398878700 1 -- 172.21.15.32:0/3143624114 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa38c0009c0 con 0x7fa39c0b1950 2021-04-09T18:28:35.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.483 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa39c0d8620 con 0x7fa39c0b1950 2021-04-09T18:28:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.486 7fa398878700 1 -- 172.21.15.32:0/3143624114 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v36) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa38c006030 con 0x7fa39c0b1950 2021-04-09T18:28:35.566 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:28:35.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa380018930 msgr2=0x7fa38001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa380018930 0x7fa38001ae10 crc :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa39c0b1950 msgr2=0x7fa39c183e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa39c0b1950 0x7fa39c183e60 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7fa38c009650 tx=0x7fa38c00b960).stop 2021-04-09T18:28:35.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 shutdown_connections 2021-04-09T18:28:35.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa380018930 0x7fa38001ae10 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c1799e0 0x7fa39c0954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c1795e0 0x7fa39c094eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 --2- 172.21.15.32:0/3143624114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa39c0b1950 0x7fa39c183e60 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 >> 172.21.15.32:0/3143624114 conn(0x7fa39c1889b0 msgr2=0x7fa39c189210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:35.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 shutdown_connections 2021-04-09T18:28:35.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.488 7fa3a1491700 1 -- 172.21.15.32:0/3143624114 wait complete. 2021-04-09T18:28:35.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:28:35.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.660 7fb608ec2700 1 Processor -- start 2021-04-09T18:28:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.660 7fb608ec2700 1 -- start start 2021-04-09T18:28:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.661 7fb608ec2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6040cffb0 0x7fb604087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.661 7fb608ec2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb604088460 con 0x7fb6040d8050 2021-04-09T18:28:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.661 7fb608ec2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6040d8620 con 0x7fb6040d7c50 2021-04-09T18:28:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.661 7fb608ec2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6041b0c80 con 0x7fb6040cffb0 2021-04-09T18:28:35.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.661 7fb603314700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb6040d8050 0x7fb60410d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35570/0 (socket says 172.21.15.32:35570) 2021-04-09T18:28:35.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.661 7fb603314700 1 -- 172.21.15.32:0/1897728878 learned_addr learned my addr 172.21.15.32:0/1897728878 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:35.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb602b13700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6040cffb0 0x7fb604087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327509829 0 0) 0x7fb5fc000d30 con 0x7fb6040d8050 2021-04-09T18:28:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5f0002fe0 con 0x7fb6040d8050 2021-04-09T18:28:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3699652210 0 0) 0x7fb5fc001460 con 0x7fb6040d8050 2021-04-09T18:28:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6040cffb0 msgr2=0x7fb604087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6040cffb0 0x7fb604087e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 >> v1:172.21.15.86:6789/0 conn(0x7fb6040d7c50 legacy=0x7fb6041b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6040d91f0 con 0x7fb6040d8050 2021-04-09T18:28:35.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fb5fc001980 con 0x7fb6040d8050 2021-04-09T18:28:35.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 >> v1:172.21.15.32:6789/0 conn(0x7fb6040d8050 legacy=0x7fb60410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f0003a40 0x7fb5f0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5f0006790 0x7fb5f0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f0009590 0x7fb5f000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f000c2d0 con 0x7fb5f0003a40 2021-04-09T18:28:35.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f000c4f0 con 0x7fb5f0006790 2021-04-09T18:28:35.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb601b11700 1 -- 172.21.15.32:0/1897728878 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f0050720 con 0x7fb5f0009590 2021-04-09T18:28:35.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb602b13700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f0003a40 0x7fb5f0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb603314700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f0009590 0x7fb5f000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb602312700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5f0006790 0x7fb5f0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb602b13700 1 -- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f0009590 msgr2=0x7fb5f000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:35.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb602b13700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f0009590 0x7fb5f000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb602b13700 1 -- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5f0006790 msgr2=0x7fb5f0008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:35.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb602b13700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5f0006790 0x7fb5f0008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.662 7fb602b13700 1 -- 172.21.15.32:0/1897728878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f0050980 con 0x7fb5f0003a40 2021-04-09T18:28:35.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.663 7fb602b13700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f0003a40 0x7fb5f0005ef0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fb5f4006ac0 tx=0x7fb5f4009ab0).ready entity=mon.0 client_cookie=b29c89cc3bc63f04 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:35.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.663 7fb601b11700 1 -- 172.21.15.32:0/1897728878 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5f400c0c0 con 0x7fb5f0003a40 2021-04-09T18:28:35.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.663 7fb601b11700 1 -- 172.21.15.32:0/1897728878 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f400c3c0 con 0x7fb5f0003a40 2021-04-09T18:28:35.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.663 7fb601b11700 1 -- 172.21.15.32:0/1897728878 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5f400c8a0 con 0x7fb5f0003a40 2021-04-09T18:28:35.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.663 7fb608ec2700 1 -- 172.21.15.32:0/1897728878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f0003a40 msgr2=0x7fb5f0005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.663 7fb608ec2700 1 --2- 172.21.15.32:0/1897728878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5f0003a40 0x7fb5f0005ef0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fb5f4006ac0 tx=0x7fb5f4009ab0).stop 2021-04-09T18:28:35.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.663 7fb603314700 1 -- 172.21.15.32:0/1897728878 reap_dead start 2021-04-09T18:28:35.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.664 7fb608ec2700 1 -- 172.21.15.32:0/1897728878 shutdown_connections 2021-04-09T18:28:35.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.664 7fb608ec2700 1 -- 172.21.15.32:0/1897728878 >> 172.21.15.32:0/1897728878 conn(0x7fb6040ce5b0 msgr2=0x7fb6040cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:35.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.664 7fb608ec2700 1 -- 172.21.15.32:0/1897728878 shutdown_connections 2021-04-09T18:28:35.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.664 7fb608ec2700 1 -- 172.21.15.32:0/1897728878 wait complete. 2021-04-09T18:28:35.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb608ec2700 1 Processor -- start 2021-04-09T18:28:35.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb608ec2700 1 -- start start 2021-04-09T18:28:35.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb608ec2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6041886b0 0x7fb604188ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb608ec2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60418e130 0x7fb604197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb608ec2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb604197c50 0x7fb604183e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb608ec2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6041970e0 con 0x7fb6041886b0 2021-04-09T18:28:35.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb608ec2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb604198050 con 0x7fb60418e130 2021-04-09T18:28:35.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb608ec2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb604184460 con 0x7fb604197c50 2021-04-09T18:28:35.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb602b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6041886b0 0x7fb604188ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb602b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6041886b0 0x7fb604188ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54222/0 (socket says 172.21.15.32:54222) 2021-04-09T18:28:35.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.684 7fb602b13700 1 -- 172.21.15.32:0/2736347724 learned_addr learned my addr 172.21.15.32:0/2736347724 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:35.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb602312700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60418e130 0x7fb604197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb603314700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb604197c50 0x7fb604183e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb602b13700 1 -- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb604197c50 msgr2=0x7fb604183e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb602b13700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb604197c50 0x7fb604183e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb602b13700 1 -- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60418e130 msgr2=0x7fb604197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb602b13700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60418e130 0x7fb604197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb602b13700 1 -- 172.21.15.32:0/2736347724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb604184680 con 0x7fb6041886b0 2021-04-09T18:28:35.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb602312700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60418e130 0x7fb604197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:35.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb603314700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb604197c50 0x7fb604183e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:35.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb602b13700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6041886b0 0x7fb604188ab0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7fb5f40097a0 tx=0x7fb5f400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:35.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb5fb7fe700 1 -- 172.21.15.32:0/2736347724 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5f4009360 con 0x7fb6041886b0 2021-04-09T18:28:35.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb5fb7fe700 1 -- 172.21.15.32:0/2736347724 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f400bde0 con 0x7fb6041886b0 2021-04-09T18:28:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb604184680 con 0x7fb6041886b0 2021-04-09T18:28:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb5fb7fe700 1 -- 172.21.15.32:0/2736347724 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5f4002030 con 0x7fb6041886b0 2021-04-09T18:28:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.685 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb604184680 con 0x7fb6041886b0 2021-04-09T18:28:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.686 7fb5fb7fe700 1 -- 172.21.15.32:0/2736347724 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb5f4000c90 con 0x7fb6041886b0 2021-04-09T18:28:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.686 7fb5fb7fe700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5e8018930 0x7fb5e801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:35.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.686 7fb5fb7fe700 1 -- 172.21.15.32:0/2736347724 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb5f4032a40 con 0x7fb6041886b0 2021-04-09T18:28:35.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.686 7fb602312700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5e8018930 0x7fb5e801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:35.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.686 7fb602312700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5e8018930 0x7fb5e801adc0 crc :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.704 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb604182f30 con 0x7fb6041886b0 2021-04-09T18:28:35.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.708 7fb5fb7fe700 1 -- 172.21.15.32:0/2736347724 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb5f4009140 con 0x7fb6041886b0 2021-04-09T18:28:35.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.892 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb6040d8620 con 0x7fb6041886b0 2021-04-09T18:28:35.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.895 7fb5fb7fe700 1 -- 172.21.15.32:0/2736347724 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v37) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb5f401f780 con 0x7fb6041886b0 2021-04-09T18:28:35.980 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:28:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5e8018930 msgr2=0x7fb5e801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5e8018930 0x7fb5e801adc0 crc :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6041886b0 msgr2=0x7fb604188ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:35.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6041886b0 0x7fb604188ab0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7fb5f40097a0 tx=0x7fb5f400b970).stop 2021-04-09T18:28:35.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 shutdown_connections 2021-04-09T18:28:35.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb5e8018930 0x7fb5e801adc0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb604197c50 0x7fb604183e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60418e130 0x7fb604197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 --2- 172.21.15.32:0/2736347724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6041886b0 0x7fb604188ab0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:35.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.901 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 >> 172.21.15.32:0/2736347724 conn(0x7fb60409eb80 msgr2=0x7fb604183160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:35.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.902 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 shutdown_connections 2021-04-09T18:28:35.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:35.902 7fb608ec2700 1 -- 172.21.15.32:0/2736347724 wait complete. 2021-04-09T18:28:35.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:36.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.060 7f89e3c2b700 1 Processor -- start 2021-04-09T18:28:36.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.060 7f89e3c2b700 1 -- start start 2021-04-09T18:28:36.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e3c2b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f89dc10d010 0x7f89dc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:36.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e3c2b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89dc0d1da0 0x7f89dc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:36.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e3c2b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89dc10d480 con 0x7f89dc0d82c0 2021-04-09T18:28:36.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e3c2b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89dc0d8d30 con 0x7f89dc0d1da0 2021-04-09T18:28:36.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e3c2b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89dc0d9220 con 0x7f89dc10d010 2021-04-09T18:28:36.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e1246700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89dc0d1da0 0x7f89dc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:36.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e1246700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89dc0d1da0 0x7f89dc0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54232/0 (socket says 172.21.15.32:54232) 2021-04-09T18:28:36.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e1246700 1 -- 172.21.15.32:0/1907875268 learned_addr learned my addr 172.21.15.32:0/1907875268 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:36.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e1a47700 1 --2- 172.21.15.32:0/1907875268 >> v2:172.21.15.86:3300/0 conn(0x7f89dc10d010 0x7f89dc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:36.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e1246700 1 -- 172.21.15.32:0/1907875268 >> v2:172.21.15.86:3300/0 conn(0x7f89dc10d010 msgr2=0x7f89dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:36.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e1246700 1 --2- 172.21.15.32:0/1907875268 >> v2:172.21.15.86:3300/0 conn(0x7f89dc10d010 0x7f89dc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e1246700 1 -- 172.21.15.32:0/1907875268 >> v1:172.21.15.86:6789/0 conn(0x7f89dc0d82c0 legacy=0x7f89dc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:36.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e1246700 1 -- 172.21.15.32:0/1907875268 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc0d86f0 con 0x7f89dc0d1da0 2021-04-09T18:28:36.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e1246700 1 --2- 172.21.15.32:0/1907875268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc0d1da0 0x7f89dc0d7ca0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 rx=0x7f89d0006700 tx=0x7f89d0009b00).ready entity=mon.0 client_cookie=8f46454b5fc808d5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:36.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e0a45700 1 -- 172.21.15.32:0/1907875268 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89d000c0c0 con 0x7f89dc0d1da0 2021-04-09T18:28:36.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e0a45700 1 -- 172.21.15.32:0/1907875268 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d000c400 con 0x7f89dc0d1da0 2021-04-09T18:28:36.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.061 7f89e0a45700 1 -- 172.21.15.32:0/1907875268 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89d000c8e0 con 0x7f89dc0d1da0 2021-04-09T18:28:36.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.062 7f89e3c2b700 1 -- 172.21.15.32:0/1907875268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc0d1da0 msgr2=0x7f89dc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:36.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.062 7f89e3c2b700 1 --2- 172.21.15.32:0/1907875268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc0d1da0 0x7f89dc0d7ca0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 rx=0x7f89d0006700 tx=0x7f89d0009b00).stop 2021-04-09T18:28:36.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.062 7f89e3c2b700 1 -- 172.21.15.32:0/1907875268 shutdown_connections 2021-04-09T18:28:36.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.062 7f89e3c2b700 1 --2- 172.21.15.32:0/1907875268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc0d1da0 0x7f89dc0d7ca0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.062 7f89e3c2b700 1 --2- 172.21.15.32:0/1907875268 >> v2:172.21.15.86:3300/0 conn(0x7f89dc10d010 0x7f89dc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.062 7f89e3c2b700 1 -- 172.21.15.32:0/1907875268 >> 172.21.15.32:0/1907875268 conn(0x7f89dc0ce5f0 msgr2=0x7f89dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:36.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.062 7f89e3c2b700 1 -- 172.21.15.32:0/1907875268 shutdown_connections 2021-04-09T18:28:36.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.062 7f89e3c2b700 1 -- 172.21.15.32:0/1907875268 wait complete. 2021-04-09T18:28:36.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.083 7f89e3c2b700 1 Processor -- start 2021-04-09T18:28:36.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.083 7f89e3c2b700 1 -- start start 2021-04-09T18:28:36.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.083 7f89e3c2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc1aca20 0x7f89dc1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.083 7f89e3c2b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89dc18bd40 0x7f89dc18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.083 7f89e3c2b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89dc08fba0 0x7f89dc08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.083 7f89e3c2b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc18c7e0 con 0x7f89dc1aca20 2021-04-09T18:28:36.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.083 7f89e3c2b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc18ca30 con 0x7f89dc18bd40 2021-04-09T18:28:36.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.083 7f89e3c2b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc0905c0 con 0x7f89dc08fba0 2021-04-09T18:28:36.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1a47700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc1aca20 0x7f89dc1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:36.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1a47700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc1aca20 0x7f89dc1ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54236/0 (socket says 172.21.15.32:54236) 2021-04-09T18:28:36.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1a47700 1 -- 172.21.15.32:0/3928757015 learned_addr learned my addr 172.21.15.32:0/3928757015 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:36.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1246700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89dc18bd40 0x7f89dc18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:36.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e2248700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89dc08fba0 0x7f89dc08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:36.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1a47700 1 -- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89dc08fba0 msgr2=0x7f89dc08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:36.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1a47700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89dc08fba0 0x7f89dc08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1a47700 1 -- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89dc18bd40 msgr2=0x7f89dc18c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:36.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1a47700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89dc18bd40 0x7f89dc18c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1a47700 1 -- 172.21.15.32:0/3928757015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc0b4300 con 0x7f89dc1aca20 2021-04-09T18:28:36.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e1a47700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc1aca20 0x7f89dc1ace20 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f89d8006f60 tx=0x7f89d800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:36.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89ce7fc700 1 -- 172.21.15.32:0/3928757015 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89d800bd60 con 0x7f89dc1aca20 2021-04-09T18:28:36.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89ce7fc700 1 -- 172.21.15.32:0/3928757015 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d8000c90 con 0x7f89dc1aca20 2021-04-09T18:28:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89dc0b4300 con 0x7f89dc1aca20 2021-04-09T18:28:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89ce7fc700 1 -- 172.21.15.32:0/3928757015 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89d8002220 con 0x7f89dc1aca20 2021-04-09T18:28:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.084 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89dc0b4300 con 0x7f89dc1aca20 2021-04-09T18:28:36.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.085 7f89ce7fc700 1 -- 172.21.15.32:0/3928757015 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f89d80052e0 con 0x7f89dc1aca20 2021-04-09T18:28:36.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.085 7f89ce7fc700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89c4018930 0x7f89c401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:36.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.085 7f89ce7fc700 1 -- 172.21.15.32:0/3928757015 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f89d8032b10 con 0x7f89dc1aca20 2021-04-09T18:28:36.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.085 7f89e1246700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89c4018930 0x7f89c401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:36.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.085 7f89e1246700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89c4018930 0x7f89c401ae10 crc :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:36.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.105 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89dc182d70 con 0x7f89dc1aca20 2021-04-09T18:28:36.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.108 7f89ce7fc700 1 -- 172.21.15.32:0/3928757015 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f89d8000c90 con 0x7f89dc1aca20 2021-04-09T18:28:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.300 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89dc18c760 con 0x7f89dc1aca20 2021-04-09T18:28:36.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.301 7f89ce7fc700 1 -- 172.21.15.32:0/3928757015 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f89d8006b00 con 0x7f89dc1aca20 2021-04-09T18:28:36.383 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2021-04-09T18:28:36.384 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:36.384 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:34.236678","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9868":{"gid":9868,"name":"a","rank":0,"incarnation":295,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1939434763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1939434763},{"type":"v1","addr":"172.21.15.32:6835","nonce":1939434763}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:36.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.303 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89c4018930 msgr2=0x7f89c401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.303 7f89e3c2b700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89c4018930 0x7f89c401ae10 crc :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc1aca20 msgr2=0x7f89dc1ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:36.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc1aca20 0x7f89dc1ace20 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f89d8006f60 tx=0x7f89d800b970).stop 2021-04-09T18:28:36.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 shutdown_connections 2021-04-09T18:28:36.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89c4018930 0x7f89c401ae10 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89dc08fba0 0x7f89dc08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89dc18bd40 0x7f89dc18c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 --2- 172.21.15.32:0/3928757015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89dc1aca20 0x7f89dc1ace20 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 >> 172.21.15.32:0/3928757015 conn(0x7f89dc19b460 msgr2=0x7f89dc19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:36.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 shutdown_connections 2021-04-09T18:28:36.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:36.304 7f89e3c2b700 1 -- 172.21.15.32:0/3928757015 wait complete. 2021-04-09T18:28:36.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 297, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:28:32.232161', 'modified': '2021-04-09 18:28:34.236678', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9868}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9868': {'gid': 9868, 'name': 'a', 'rank': 0, 'incarnation': 295, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/1939434763', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 1939434763}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1939434763}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:28:36.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:28:36.799 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:28:36.936 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:28:36.936 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:28:36.936 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:28:36.937 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9868, 2021-04-09T18:28:36.937 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:28:36.937 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:28:36.937 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 4.6222131759999998, 2021-04-09T18:28:36.937 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 297, 2021-04-09T18:28:36.938 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 155, 2021-04-09T18:28:36.938 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 155, 2021-04-09T18:28:36.938 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 10.095014055 2021-04-09T18:28:36.938 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:28:36.942 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 9868, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.6222131759999998, "mdsmap_epoch": 297, "osdmap_epoch": 155, "osdmap_epoch_barrier": 155, "uptime": 10.095014055 } 2021-04-09T18:28:36.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:28:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.010 7f3cf10d5700 1 Processor -- start 2021-04-09T18:28:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.010 7f3cf10d5700 1 -- start start 2021-04-09T18:28:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3cf10d5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec0d7ca0 0x7f3cec0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3cf10d5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3cec10da00 0x7f3cec10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3cf10d5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cec10d480 con 0x7f3cec10d010 2021-04-09T18:28:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3cf10d5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cec0d9220 con 0x7f3cec10da00 2021-04-09T18:28:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3cf10d5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0d86c0 con 0x7f3cec0d7ca0 2021-04-09T18:28:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3ceb59e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3cec10da00 0x7f3cec10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3cead9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3cec10d010 0x7f3cec0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35602/0 (socket says 172.21.15.32:35602) 2021-04-09T18:28:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3cead9d700 1 -- 172.21.15.32:0/3934278174 learned_addr learned my addr 172.21.15.32:0/3934278174 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3cea59c700 1 --2- 172.21.15.32:0/3934278174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec0d7ca0 0x7f3cec0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3ceb59e700 1 -- 172.21.15.32:0/3934278174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec0d7ca0 msgr2=0x7f3cec0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3ceb59e700 1 --2- 172.21.15.32:0/3934278174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec0d7ca0 0x7f3cec0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3ceb59e700 1 -- 172.21.15.32:0/3934278174 >> v1:172.21.15.32:6789/0 conn(0x7f3cec10d010 legacy=0x7f3cec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3ceb59e700 1 -- 172.21.15.32:0/3934278174 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec0d1f70 con 0x7f3cec10da00 2021-04-09T18:28:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3ce9d9b700 1 -- 172.21.15.32:0/3934278174 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55122142 0 0) 0x7f3cdc000ce0 con 0x7f3cec10d010 2021-04-09T18:28:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.011 7f3ceb59e700 1 --2- 172.21.15.32:0/3934278174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec10da00 0x7f3cec10de00 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7f3ce4006e30 tx=0x7f3ce4009de0).ready entity=mon.0 client_cookie=f3a3beed380c8534 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.012 7f3ce9d9b700 1 -- 172.21.15.32:0/3934278174 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ce400b300 con 0x7f3cec10da00 2021-04-09T18:28:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.012 7f3ce9d9b700 1 -- 172.21.15.32:0/3934278174 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce400b680 con 0x7f3cec10da00 2021-04-09T18:28:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.012 7f3ce9d9b700 1 -- 172.21.15.32:0/3934278174 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ce400bb60 con 0x7f3cec10da00 2021-04-09T18:28:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.012 7f3cf10d5700 1 -- 172.21.15.32:0/3934278174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec10da00 msgr2=0x7f3cec10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.012 7f3cf10d5700 1 --2- 172.21.15.32:0/3934278174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec10da00 0x7f3cec10de00 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7f3ce4006e30 tx=0x7f3ce4009de0).stop 2021-04-09T18:28:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.013 7f3cf10d5700 1 -- 172.21.15.32:0/3934278174 shutdown_connections 2021-04-09T18:28:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.013 7f3cf10d5700 1 --2- 172.21.15.32:0/3934278174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec10da00 0x7f3cec10de00 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.013 7f3cf10d5700 1 --2- 172.21.15.32:0/3934278174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec0d7ca0 0x7f3cec0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.013 7f3cf10d5700 1 -- 172.21.15.32:0/3934278174 >> 172.21.15.32:0/3934278174 conn(0x7f3cec0ce5f0 msgr2=0x7f3cec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.013 7f3cf10d5700 1 -- 172.21.15.32:0/3934278174 shutdown_connections 2021-04-09T18:28:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.013 7f3cf10d5700 1 -- 172.21.15.32:0/3934278174 wait complete. 2021-04-09T18:28:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cf10d5700 1 Processor -- start 2021-04-09T18:28:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cf10d5700 1 -- start start 2021-04-09T18:28:37.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cf10d5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec186770 0x7f3cec181690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cf10d5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cec181cb0 0x7f3cec08eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cf10d5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec08f0d0 0x7f3cec1a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cf10d5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec1820b0 con 0x7f3cec186770 2021-04-09T18:28:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cf10d5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec186b70 con 0x7f3cec181cb0 2021-04-09T18:28:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cf10d5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec08f4d0 con 0x7f3cec08f0d0 2021-04-09T18:28:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cead9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec186770 0x7f3cec181690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cead9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec186770 0x7f3cec181690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54250/0 (socket says 172.21.15.32:54250) 2021-04-09T18:28:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.033 7f3cead9d700 1 -- 172.21.15.32:0/7174822 learned_addr learned my addr 172.21.15.32:0/7174822 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cea59c700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cec181cb0 0x7f3cec08eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3ceb59e700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec08f0d0 0x7f3cec1a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cead9d700 1 -- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec08f0d0 msgr2=0x7f3cec1a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cead9d700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec08f0d0 0x7f3cec1a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cead9d700 1 -- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cec181cb0 msgr2=0x7f3cec08eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cead9d700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cec181cb0 0x7f3cec08eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cead9d700 1 -- 172.21.15.32:0/7174822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec1a5d00 con 0x7f3cec186770 2021-04-09T18:28:37.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cead9d700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec186770 0x7f3cec181690 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 rx=0x7f3cdc006f00 tx=0x7f3cdc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cdb7fe700 1 -- 172.21.15.32:0/7174822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cdc00bd30 con 0x7f3cec186770 2021-04-09T18:28:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cdb7fe700 1 -- 172.21.15.32:0/7174822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc0060d0 con 0x7f3cec186770 2021-04-09T18:28:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cec1a5d00 con 0x7f3cec186770 2021-04-09T18:28:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cdb7fe700 1 -- 172.21.15.32:0/7174822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cdc004620 con 0x7f3cec186770 2021-04-09T18:28:37.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.034 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cec1a5d00 con 0x7f3cec186770 2021-04-09T18:28:37.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.035 7f3cdb7fe700 1 -- 172.21.15.32:0/7174822 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3cdc0062b0 con 0x7f3cec186770 2021-04-09T18:28:37.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.035 7f3cdb7fe700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3cd0018930 0x7f3cd001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.035 7f3cdb7fe700 1 -- 172.21.15.32:0/7174822 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3cdc032a20 con 0x7f3cec186770 2021-04-09T18:28:37.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.035 7f3cea59c700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3cd0018930 0x7f3cd001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.035 7f3cea59c700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3cd0018930 0x7f3cd001adc0 crc :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:37.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.055 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cec181160 con 0x7f3cec186770 2021-04-09T18:28:37.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.058 7f3cdb7fe700 1 -- 172.21.15.32:0/7174822 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3cdc0062e0 con 0x7f3cec186770 2021-04-09T18:28:37.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.247 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cec181160 con 0x7f3cec186770 2021-04-09T18:28:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.248 7f3cdb7fe700 1 -- 172.21.15.32:0/7174822 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f3cdc0060d0 con 0x7f3cec186770 2021-04-09T18:28:37.331 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:37.331 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":155,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:28:33.233832","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:28:29.569175","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-04-09 18:28:30.947070","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:28:37.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3cd0018930 msgr2=0x7f3cd001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3cd0018930 0x7f3cd001adc0 crc :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec186770 msgr2=0x7f3cec181690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec186770 0x7f3cec181690 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 rx=0x7f3cdc006f00 tx=0x7f3cdc00b940).stop 2021-04-09T18:28:37.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 shutdown_connections 2021-04-09T18:28:37.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3cd0018930 0x7f3cd001adc0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3cec08f0d0 0x7f3cec1a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3cec181cb0 0x7f3cec08eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 --2- 172.21.15.32:0/7174822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cec186770 0x7f3cec181690 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 >> 172.21.15.32:0/7174822 conn(0x7f3cec195e70 msgr2=0x7f3cec1a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:37.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 shutdown_connections 2021-04-09T18:28:37.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.252 7f3cf10d5700 1 -- 172.21.15.32:0/7174822 wait complete. 2021-04-09T18:28:37.343 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:28:29.569175', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '154', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:28:30.947070', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:28:37.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:28:37.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.416 7ff57428b700 1 Processor -- start 2021-04-09T18:28:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.416 7ff57428b700 1 -- start start 2021-04-09T18:28:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff57428b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c10d010 0x7ff56c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff57428b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c10d480 con 0x7ff56c0d8490 2021-04-09T18:28:37.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff57428b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c10e1f0 con 0x7ff56c0d1da0 2021-04-09T18:28:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff57428b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c10e780 con 0x7ff56c10d010 2021-04-09T18:28:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff5728a8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff56c0d8490 0x7ff56c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35618/0 (socket says 172.21.15.32:35618) 2021-04-09T18:28:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff5728a8700 1 -- 172.21.15.32:0/1453131526 learned_addr learned my addr 172.21.15.32:0/1453131526 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff5720a7700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c10d010 0x7ff56c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149291813 0 0) 0x7ff564000d30 con 0x7ff56c0d8490 2021-04-09T18:28:37.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff558002fe0 con 0x7ff56c0d8490 2021-04-09T18:28:37.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736087821 0 0) 0x7ff560001080 con 0x7ff56c0d1da0 2021-04-09T18:28:37.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.417 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff558003210 con 0x7ff56c0d1da0 2021-04-09T18:28:37.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2775063222 0 0) 0x7ff564001460 con 0x7ff56c0d8490 2021-04-09T18:28:37.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c10d010 msgr2=0x7ff56c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:37.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c10d010 0x7ff56c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 >> v1:172.21.15.86:6789/0 conn(0x7ff56c0d1da0 legacy=0x7ff56c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c0d8d30 con 0x7ff56c0d8490 2021-04-09T18:28:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7ff564001980 con 0x7ff56c0d8490 2021-04-09T18:28:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5720a7700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c10d010 0x7ff56c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:28:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 >> v1:172.21.15.32:6789/0 conn(0x7ff56c0d8490 legacy=0x7ff56c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff558003b90 0x7ff558005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5580067a0 0x7ff558008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5580095a0 0x7ff55800ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5720a7700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff558003b90 0x7ff558005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580502f0 con 0x7ff558003b90 2021-04-09T18:28:37.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff558050510 con 0x7ff5580067a0 2021-04-09T18:28:37.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff558050730 con 0x7ff5580095a0 2021-04-09T18:28:37.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5718a6700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5580067a0 0x7ff558008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5728a8700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5580095a0 0x7ff55800ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5720a7700 1 -- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5580095a0 msgr2=0x7ff55800ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:37.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5720a7700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5580095a0 0x7ff55800ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5720a7700 1 -- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5580067a0 msgr2=0x7ff558008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:37.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5720a7700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5580067a0 0x7ff558008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5720a7700 1 -- 172.21.15.32:0/1453131526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580509d0 con 0x7ff558003b90 2021-04-09T18:28:37.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5720a7700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff558003b90 0x7ff558005fa0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 rx=0x7ff568006ac0 tx=0x7ff568009a30).ready entity=mon.0 client_cookie=970ce194c12ada8e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:37.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff56800c0c0 con 0x7ff558003b90 2021-04-09T18:28:37.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff568009e20 con 0x7ff558003b90 2021-04-09T18:28:37.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5710a5700 1 -- 172.21.15.32:0/1453131526 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff56800c630 con 0x7ff558003b90 2021-04-09T18:28:37.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff57428b700 1 -- 172.21.15.32:0/1453131526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff558003b90 msgr2=0x7ff558005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff57428b700 1 --2- 172.21.15.32:0/1453131526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff558003b90 0x7ff558005fa0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 rx=0x7ff568006ac0 tx=0x7ff568009a30).stop 2021-04-09T18:28:37.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.418 7ff5728a8700 1 -- 172.21.15.32:0/1453131526 reap_dead start 2021-04-09T18:28:37.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.419 7ff57428b700 1 -- 172.21.15.32:0/1453131526 shutdown_connections 2021-04-09T18:28:37.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.419 7ff57428b700 1 -- 172.21.15.32:0/1453131526 >> 172.21.15.32:0/1453131526 conn(0x7ff56c0ce5f0 msgr2=0x7ff56c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:37.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.419 7ff57428b700 1 -- 172.21.15.32:0/1453131526 shutdown_connections 2021-04-09T18:28:37.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.419 7ff57428b700 1 -- 172.21.15.32:0/1453131526 wait complete. 2021-04-09T18:28:37.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.439 7ff57428b700 1 Processor -- start 2021-04-09T18:28:37.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.439 7ff57428b700 1 -- start start 2021-04-09T18:28:37.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff57428b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c0c3c00 0x7ff56c09c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff57428b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c095240 0x7ff56c1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff57428b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff56c1a1270 0x7ff56c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff57428b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c09cda0 con 0x7ff56c095240 2021-04-09T18:28:37.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff57428b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c0c3680 con 0x7ff56c1a1270 2021-04-09T18:28:37.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff57428b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c095640 con 0x7ff56c0c3c00 2021-04-09T18:28:37.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5718a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c095240 0x7ff56c1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5718a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c095240 0x7ff56c1a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54272/0 (socket says 172.21.15.32:54272) 2021-04-09T18:28:37.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5718a6700 1 -- 172.21.15.32:0/1736684457 learned_addr learned my addr 172.21.15.32:0/1736684457 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:37.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5728a8700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff56c1a1270 0x7ff56c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5720a7700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c0c3c00 0x7ff56c09c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5718a6700 1 -- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c0c3c00 msgr2=0x7ff56c09c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5718a6700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c0c3c00 0x7ff56c09c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5718a6700 1 -- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff56c1a1270 msgr2=0x7ff56c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5718a6700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff56c1a1270 0x7ff56c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.440 7ff5718a6700 1 -- 172.21.15.32:0/1736684457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c1a1670 con 0x7ff56c095240 2021-04-09T18:28:37.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.441 7ff5718a6700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c095240 0x7ff56c1a0c50 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 rx=0x7ff560006360 tx=0x7ff56000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:37.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.441 7ff55effd700 1 -- 172.21.15.32:0/1736684457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff56000ad20 con 0x7ff56c095240 2021-04-09T18:28:37.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.441 7ff55effd700 1 -- 172.21.15.32:0/1736684457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff560000e20 con 0x7ff56c095240 2021-04-09T18:28:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.441 7ff57428b700 1 -- 172.21.15.32:0/1736684457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c1a1670 con 0x7ff56c095240 2021-04-09T18:28:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.441 7ff55effd700 1 -- 172.21.15.32:0/1736684457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff56000b2f0 con 0x7ff56c095240 2021-04-09T18:28:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.441 7ff57428b700 1 -- 172.21.15.32:0/1736684457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c1a1670 con 0x7ff56c095240 2021-04-09T18:28:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.441 7ff55effd700 1 -- 172.21.15.32:0/1736684457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff560006740 con 0x7ff56c095240 2021-04-09T18:28:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.442 7ff55effd700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff554018930 0x7ff55401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.442 7ff55effd700 1 -- 172.21.15.32:0/1736684457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff5600329d0 con 0x7ff56c095240 2021-04-09T18:28:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.442 7ff5720a7700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff554018930 0x7ff55401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.442 7ff5720a7700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff554018930 0x7ff55401adc0 crc :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.461 7ff57428b700 1 -- 172.21.15.32:0/1736684457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c093d50 con 0x7ff56c095240 2021-04-09T18:28:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.464 7ff55effd700 1 -- 172.21.15.32:0/1736684457 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff560000c10 con 0x7ff56c095240 2021-04-09T18:28:37.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:28:37.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:28:37.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.647 7ff57428b700 1 -- 172.21.15.32:0/1736684457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff56c10d480 con 0x7ff56c095240 2021-04-09T18:28:37.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.648 7ff55effd700 1 -- 172.21.15.32:0/1736684457 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v155) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff56000ad20 con 0x7ff56c095240 2021-04-09T18:28:37.729 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:28:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 -- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff554018930 msgr2=0x7ff55401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff554018930 0x7ff55401adc0 crc :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 -- 172.21.15.32:0/1736684457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c095240 msgr2=0x7ff56c1a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c095240 0x7ff56c1a0c50 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 rx=0x7ff560006360 tx=0x7ff56000a930).stop 2021-04-09T18:28:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 -- 172.21.15.32:0/1736684457 shutdown_connections 2021-04-09T18:28:37.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff554018930 0x7ff55401adc0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c095240 0x7ff56c1a0c50 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff56c1a1270 0x7ff56c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 --2- 172.21.15.32:0/1736684457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff56c0c3c00 0x7ff56c09c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 -- 172.21.15.32:0/1736684457 >> 172.21.15.32:0/1736684457 conn(0x7ff56c0975f0 msgr2=0x7ff56c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:37.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 -- 172.21.15.32:0/1736684457 shutdown_connections 2021-04-09T18:28:37.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.651 7ff57428b700 1 -- 172.21.15.32:0/1736684457 wait complete. 2021-04-09T18:28:37.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:28:37.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.851 7f65ddc34700 1 Processor -- start 2021-04-09T18:28:37.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.851 7f65ddc34700 1 -- start start 2021-04-09T18:28:37.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65ddc34700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d80d7ca0 0x7f65d80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65ddc34700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65d810da00 0x7f65d810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65ddc34700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d810d480 con 0x7f65d810d010 2021-04-09T18:28:37.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65ddc34700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d80d9220 con 0x7f65d810da00 2021-04-09T18:28:37.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65ddc34700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d80d86c0 con 0x7f65d80d7ca0 2021-04-09T18:28:37.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65d7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65d810da00 0x7f65d810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65d7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65d810da00 0x7f65d810de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54278/0 (socket says 172.21.15.32:54278) 2021-04-09T18:28:37.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65d7fff700 1 -- 172.21.15.32:0/2269718100 learned_addr learned my addr 172.21.15.32:0/2269718100 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:37.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65d6ffd700 1 --2- 172.21.15.32:0/2269718100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d80d7ca0 0x7f65d80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65d7fff700 1 -- 172.21.15.32:0/2269718100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d80d7ca0 msgr2=0x7f65d80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:37.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65d7fff700 1 --2- 172.21.15.32:0/2269718100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d80d7ca0 0x7f65d80d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65d7fff700 1 -- 172.21.15.32:0/2269718100 >> v1:172.21.15.86:6789/0 conn(0x7f65d810d010 legacy=0x7f65d80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.852 7f65d7fff700 1 -- 172.21.15.32:0/2269718100 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d80d1f70 con 0x7f65d810da00 2021-04-09T18:28:37.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65d7fff700 1 --2- 172.21.15.32:0/2269718100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d810da00 0x7f65d810de00 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 rx=0x7f65d0006b80 tx=0x7f65d0009b90).ready entity=mon.0 client_cookie=302f75e3b70c32fc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:37.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65d67fc700 1 -- 172.21.15.32:0/2269718100 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d00022e0 con 0x7f65d810da00 2021-04-09T18:28:37.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65d67fc700 1 -- 172.21.15.32:0/2269718100 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0002620 con 0x7f65d810da00 2021-04-09T18:28:37.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65d67fc700 1 -- 172.21.15.32:0/2269718100 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d0002b00 con 0x7f65d810da00 2021-04-09T18:28:37.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65ddc34700 1 -- 172.21.15.32:0/2269718100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d810da00 msgr2=0x7f65d810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65ddc34700 1 --2- 172.21.15.32:0/2269718100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d810da00 0x7f65d810de00 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 rx=0x7f65d0006b80 tx=0x7f65d0009b90).stop 2021-04-09T18:28:37.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65ddc34700 1 -- 172.21.15.32:0/2269718100 shutdown_connections 2021-04-09T18:28:37.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65ddc34700 1 --2- 172.21.15.32:0/2269718100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d810da00 0x7f65d810de00 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65ddc34700 1 --2- 172.21.15.32:0/2269718100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d80d7ca0 0x7f65d80d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65ddc34700 1 -- 172.21.15.32:0/2269718100 >> 172.21.15.32:0/2269718100 conn(0x7f65d80ce5f0 msgr2=0x7f65d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.853 7f65ddc34700 1 -- 172.21.15.32:0/2269718100 shutdown_connections 2021-04-09T18:28:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.854 7f65ddc34700 1 -- 172.21.15.32:0/2269718100 wait complete. 2021-04-09T18:28:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.874 7f65ddc34700 1 Processor -- start 2021-04-09T18:28:37.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.874 7f65ddc34700 1 -- start start 2021-04-09T18:28:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.874 7f65ddc34700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d808a910 0x7f65d809e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.874 7f65ddc34700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d808ad10 0x7f65d809ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.874 7f65ddc34700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d8092350 0x7f65d8092750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.874 7f65ddc34700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d809e0d0 con 0x7f65d8092350 2021-04-09T18:28:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.874 7f65ddc34700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d809f290 con 0x7f65d808ad10 2021-04-09T18:28:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.874 7f65ddc34700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d808b430 con 0x7f65d808a910 2021-04-09T18:28:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.874 7f65d7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d8092350 0x7f65d8092750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d8092350 0x7f65d8092750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54286/0 (socket says 172.21.15.32:54286) 2021-04-09T18:28:37.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d7fff700 1 -- 172.21.15.32:0/3669548584 learned_addr learned my addr 172.21.15.32:0/3669548584 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:37.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d77fe700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d808a910 0x7f65d809e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d6ffd700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d808ad10 0x7f65d809ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d7fff700 1 -- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d808a910 msgr2=0x7f65d809e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d7fff700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d808a910 0x7f65d809e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d7fff700 1 -- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d808ad10 msgr2=0x7f65d809ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d7fff700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d808ad10 0x7f65d809ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:37.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d7fff700 1 -- 172.21.15.32:0/3669548584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d8092d70 con 0x7f65d8092350 2021-04-09T18:28:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65d7fff700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d8092350 0x7f65d8092750 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7f65d0006f10 tx=0x7f65d000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65bffff700 1 -- 172.21.15.32:0/3669548584 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d0000b90 con 0x7f65d8092350 2021-04-09T18:28:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65bffff700 1 -- 172.21.15.32:0/3669548584 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d000bcd0 con 0x7f65d8092350 2021-04-09T18:28:37.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d8092d70 con 0x7f65d8092350 2021-04-09T18:28:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65bffff700 1 -- 172.21.15.32:0/3669548584 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d0002220 con 0x7f65d8092350 2021-04-09T18:28:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.875 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d8092d70 con 0x7f65d8092350 2021-04-09T18:28:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.876 7f65bffff700 1 -- 172.21.15.32:0/3669548584 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f65d00054a0 con 0x7f65d8092350 2021-04-09T18:28:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.876 7f65bffff700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b8018930 0x7f65b801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.876 7f65bffff700 1 -- 172.21.15.32:0/3669548584 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f65d0032a20 con 0x7f65d8092350 2021-04-09T18:28:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.876 7f65d77fe700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b8018930 0x7f65b801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.876 7f65d77fe700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b8018930 0x7f65b801ae10 crc :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.896 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d809f290 con 0x7f65d8092350 2021-04-09T18:28:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:37.899 7f65bffff700 1 -- 172.21.15.32:0/3669548584 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f65d0009310 con 0x7f65d8092350 2021-04-09T18:28:38.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.079 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f65d809f290 con 0x7f65d8092350 2021-04-09T18:28:38.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.080 7f65bffff700 1 -- 172.21.15.32:0/3669548584 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v155) v1 ==== 104+0+10 (secure 0 0 0) 0x7f65d00054a0 con 0x7f65d8092350 2021-04-09T18:28:38.161 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:28:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b8018930 msgr2=0x7f65b801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b8018930 0x7f65b801ae10 crc :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d8092350 msgr2=0x7f65d8092750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d8092350 0x7f65d8092750 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7f65d0006f10 tx=0x7f65d000b8e0).stop 2021-04-09T18:28:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 shutdown_connections 2021-04-09T18:28:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65b8018930 0x7f65b801ae10 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65d8092350 0x7f65d8092750 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d808ad10 0x7f65d809ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 --2- 172.21.15.32:0/3669548584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65d808a910 0x7f65d809e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.082 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 >> 172.21.15.32:0/3669548584 conn(0x7f65d80b2fa0 msgr2=0x7f65d80a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.083 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 shutdown_connections 2021-04-09T18:28:38.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.083 7f65ddc34700 1 -- 172.21.15.32:0/3669548584 wait complete. 2021-04-09T18:28:38.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:28:38.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.245 7f8b68b4c700 1 Processor -- start 2021-04-09T18:28:38.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.245 7f8b68b4c700 1 -- start start 2021-04-09T18:28:38.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b68b4c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b6410d010 0x7f8b640d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b68b4c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b640d1da0 0x7f8b640d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b68b4c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6410d480 con 0x7f8b640d82c0 2021-04-09T18:28:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b68b4c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6410e780 con 0x7f8b640d1da0 2021-04-09T18:28:38.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b68b4c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b640d8d30 con 0x7f8b6410d010 2021-04-09T18:28:38.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b6205d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b640d1da0 0x7f8b640d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b6305f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8b640d82c0 0x7f8b6410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35650/0 (socket says 172.21.15.32:35650) 2021-04-09T18:28:38.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b6305f700 1 -- 172.21.15.32:0/2688115866 learned_addr learned my addr 172.21.15.32:0/2688115866 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:38.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b6285e700 1 --2- 172.21.15.32:0/2688115866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b6410d010 0x7f8b640d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b6205d700 1 -- 172.21.15.32:0/2688115866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b6410d010 msgr2=0x7f8b640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:38.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b6205d700 1 --2- 172.21.15.32:0/2688115866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b6410d010 0x7f8b640d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b6205d700 1 -- 172.21.15.32:0/2688115866 >> v1:172.21.15.32:6789/0 conn(0x7f8b640d82c0 legacy=0x7f8b6410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.247 7f8b6205d700 1 -- 172.21.15.32:0/2688115866 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b640d9220 con 0x7f8b640d1da0 2021-04-09T18:28:38.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.246 7f8b6185c700 1 -- 172.21.15.32:0/2688115866 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754252243 0 0) 0x7f8b5c000d30 con 0x7f8b640d82c0 2021-04-09T18:28:38.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.247 7f8b6205d700 1 --2- 172.21.15.32:0/2688115866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640d1da0 0x7f8b640d7ca0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 rx=0x7f8b58006650 tx=0x7f8b58009a00).ready entity=mon.0 client_cookie=337618904eb5c35b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:38.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.247 7f8b6185c700 1 -- 172.21.15.32:0/2688115866 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b58009df0 con 0x7f8b640d1da0 2021-04-09T18:28:38.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.247 7f8b6185c700 1 -- 172.21.15.32:0/2688115866 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b580009b0 con 0x7f8b640d1da0 2021-04-09T18:28:38.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.247 7f8b6185c700 1 -- 172.21.15.32:0/2688115866 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b5800c2b0 con 0x7f8b640d1da0 2021-04-09T18:28:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.247 7f8b68b4c700 1 -- 172.21.15.32:0/2688115866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640d1da0 msgr2=0x7f8b640d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.247 7f8b68b4c700 1 --2- 172.21.15.32:0/2688115866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640d1da0 0x7f8b640d7ca0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 rx=0x7f8b58006650 tx=0x7f8b58009a00).stop 2021-04-09T18:28:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.248 7f8b68b4c700 1 -- 172.21.15.32:0/2688115866 shutdown_connections 2021-04-09T18:28:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.248 7f8b68b4c700 1 --2- 172.21.15.32:0/2688115866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640d1da0 0x7f8b640d7ca0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.248 7f8b68b4c700 1 --2- 172.21.15.32:0/2688115866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b6410d010 0x7f8b640d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.248 7f8b68b4c700 1 -- 172.21.15.32:0/2688115866 >> 172.21.15.32:0/2688115866 conn(0x7f8b640ce5f0 msgr2=0x7f8b640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:38.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.248 7f8b68b4c700 1 -- 172.21.15.32:0/2688115866 shutdown_connections 2021-04-09T18:28:38.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.248 7f8b68b4c700 1 -- 172.21.15.32:0/2688115866 wait complete. 2021-04-09T18:28:38.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.268 7f8b68b4c700 1 Processor -- start 2021-04-09T18:28:38.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b68b4c700 1 -- start start 2021-04-09T18:28:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b68b4c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640c3c00 0x7f8b6409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b68b4c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b64095240 0x7f8b641a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b68b4c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b641a1270 0x7f8b640b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b68b4c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6409cda0 con 0x7f8b640c3c00 2021-04-09T18:28:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b68b4c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b640c3680 con 0x7f8b64095240 2021-04-09T18:28:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b68b4c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64095640 con 0x7f8b641a1270 2021-04-09T18:28:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b6285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640c3c00 0x7f8b6409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b6285e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640c3c00 0x7f8b6409c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54296/0 (socket says 172.21.15.32:54296) 2021-04-09T18:28:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b6285e700 1 -- 172.21.15.32:0/1106598582 learned_addr learned my addr 172.21.15.32:0/1106598582 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:38.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b6305f700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b641a1270 0x7f8b640b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.269 7f8b6205d700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b64095240 0x7f8b641a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b6285e700 1 -- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b641a1270 msgr2=0x7f8b640b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b6285e700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b641a1270 0x7f8b640b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b6285e700 1 -- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b64095240 msgr2=0x7f8b641a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b6285e700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b64095240 0x7f8b641a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b6285e700 1 -- 172.21.15.32:0/1106598582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b641a1670 con 0x7f8b640c3c00 2021-04-09T18:28:38.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b6285e700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640c3c00 0x7f8b6409c600 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f8b54006be0 tx=0x7f8b5400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:38.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b537fe700 1 -- 172.21.15.32:0/1106598582 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b540064d0 con 0x7f8b640c3c00 2021-04-09T18:28:38.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b537fe700 1 -- 172.21.15.32:0/1106598582 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5400bd80 con 0x7f8b640c3c00 2021-04-09T18:28:38.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b537fe700 1 -- 172.21.15.32:0/1106598582 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b54002030 con 0x7f8b640c3c00 2021-04-09T18:28:38.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b641a1670 con 0x7f8b640c3c00 2021-04-09T18:28:38.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b641a1670 con 0x7f8b640c3c00 2021-04-09T18:28:38.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.270 7f8b537fe700 1 -- 172.21.15.32:0/1106598582 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8b54006c90 con 0x7f8b640c3c00 2021-04-09T18:28:38.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.271 7f8b537fe700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b48018930 0x7f8b4801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.271 7f8b537fe700 1 -- 172.21.15.32:0/1106598582 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8b54032a10 con 0x7f8b640c3c00 2021-04-09T18:28:38.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.271 7f8b6205d700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b48018930 0x7f8b4801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.271 7f8b6205d700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b48018930 0x7f8b4801ae10 crc :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:38.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.290 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b64093d50 con 0x7f8b640c3c00 2021-04-09T18:28:38.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.293 7f8b537fe700 1 -- 172.21.15.32:0/1106598582 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8b54006820 con 0x7f8b640c3c00 2021-04-09T18:28:38.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.473 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8b6410d480 con 0x7f8b640c3c00 2021-04-09T18:28:38.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.473 7f8b537fe700 1 -- 172.21.15.32:0/1106598582 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v155) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8b5400bd80 con 0x7f8b640c3c00 2021-04-09T18:28:38.555 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:28:38.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b48018930 msgr2=0x7f8b4801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b48018930 0x7f8b4801ae10 crc :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640c3c00 msgr2=0x7f8b6409c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640c3c00 0x7f8b6409c600 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f8b54006be0 tx=0x7f8b5400b990).stop 2021-04-09T18:28:38.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 shutdown_connections 2021-04-09T18:28:38.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b48018930 0x7f8b4801ae10 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b641a1270 0x7f8b640b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b64095240 0x7f8b641a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 --2- 172.21.15.32:0/1106598582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b640c3c00 0x7f8b6409c600 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 >> 172.21.15.32:0/1106598582 conn(0x7f8b640975f0 msgr2=0x7f8b64096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:38.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 shutdown_connections 2021-04-09T18:28:38.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.476 7f8b68b4c700 1 -- 172.21.15.32:0/1106598582 wait complete. 2021-04-09T18:28:38.567 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:28:38.568 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:38.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.636 7fe41aa87700 1 Processor -- start 2021-04-09T18:28:38.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.636 7fe41aa87700 1 -- start start 2021-04-09T18:28:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe41aa87700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4140d7ca0 0x7fe4140d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe41aa87700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41410d480 con 0x7fe41410da00 2021-04-09T18:28:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe41aa87700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4140d86c0 con 0x7fe41410d010 2021-04-09T18:28:38.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe41aa87700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4140d9220 con 0x7fe4140d7ca0 2021-04-09T18:28:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe413fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe4140d7ca0 0x7fe4140d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe4190a4700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe41410da00 0x7fe41410de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35662/0 (socket says 172.21.15.32:35662) 2021-04-09T18:28:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe4190a4700 1 -- 172.21.15.32:0/4049638468 learned_addr learned my addr 172.21.15.32:0/4049638468 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe413fff700 1 -- 172.21.15.32:0/4049638468 >> v1:172.21.15.86:6789/0 conn(0x7fe41410d010 legacy=0x7fe4140d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe413fff700 1 -- 172.21.15.32:0/4049638468 >> v1:172.21.15.32:6789/0 conn(0x7fe41410da00 legacy=0x7fe41410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe413fff700 1 -- 172.21.15.32:0/4049638468 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4140d1f70 con 0x7fe4140d7ca0 2021-04-09T18:28:38.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe4137fe700 1 -- 172.21.15.32:0/4049638468 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681272451 0 0) 0x7fe40c0009b0 con 0x7fe41410da00 2021-04-09T18:28:38.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe413fff700 1 --2- 172.21.15.32:0/4049638468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4140d7ca0 0x7fe4140d80a0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7fe4040069d0 tx=0x7fe404009dc0).ready entity=mon.0 client_cookie=924e8c237c6d0e61 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:38.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe4137fe700 1 -- 172.21.15.32:0/4049638468 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe40400b510 con 0x7fe4140d7ca0 2021-04-09T18:28:38.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe4137fe700 1 -- 172.21.15.32:0/4049638468 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe404006ac0 con 0x7fe4140d7ca0 2021-04-09T18:28:38.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.637 7fe4137fe700 1 -- 172.21.15.32:0/4049638468 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe40400ba80 con 0x7fe4140d7ca0 2021-04-09T18:28:38.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.638 7fe41aa87700 1 -- 172.21.15.32:0/4049638468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4140d7ca0 msgr2=0x7fe4140d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.638 7fe41aa87700 1 --2- 172.21.15.32:0/4049638468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4140d7ca0 0x7fe4140d80a0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7fe4040069d0 tx=0x7fe404009dc0).stop 2021-04-09T18:28:38.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.638 7fe41aa87700 1 -- 172.21.15.32:0/4049638468 shutdown_connections 2021-04-09T18:28:38.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.638 7fe41aa87700 1 --2- 172.21.15.32:0/4049638468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4140d7ca0 0x7fe4140d80a0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.638 7fe41aa87700 1 -- 172.21.15.32:0/4049638468 >> 172.21.15.32:0/4049638468 conn(0x7fe4140ce5f0 msgr2=0x7fe4140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:38.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.638 7fe41aa87700 1 -- 172.21.15.32:0/4049638468 shutdown_connections 2021-04-09T18:28:38.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.638 7fe41aa87700 1 -- 172.21.15.32:0/4049638468 wait complete. 2021-04-09T18:28:38.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.658 7fe41aa87700 1 Processor -- start 2021-04-09T18:28:38.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.658 7fe41aa87700 1 -- start start 2021-04-09T18:28:38.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe41aa87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414188d40 0x7fe41417e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe41aa87700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe414182410 0x7fe4141a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe41aa87700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe414182810 0x7fe4141813a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe41aa87700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4141a7000 con 0x7fe414188d40 2021-04-09T18:28:38.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe41aa87700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe414189140 con 0x7fe414182410 2021-04-09T18:28:38.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe41aa87700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41417e800 con 0x7fe414182810 2021-04-09T18:28:38.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe4188a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414188d40 0x7fe41417e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe4188a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414188d40 0x7fe41417e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54308/0 (socket says 172.21.15.32:54308) 2021-04-09T18:28:38.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe4188a3700 1 -- 172.21.15.32:0/2603169306 learned_addr learned my addr 172.21.15.32:0/2603169306 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:38.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe4190a4700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe414182810 0x7fe4141813a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.659 7fe413fff700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe414182410 0x7fe4141a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4188a3700 1 -- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe414182810 msgr2=0x7fe4141813a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4188a3700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe414182810 0x7fe4141813a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4188a3700 1 -- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe414182410 msgr2=0x7fe4141a69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4188a3700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe414182410 0x7fe4141a69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4188a3700 1 -- 172.21.15.32:0/2603169306 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4141819c0 con 0x7fe414188d40 2021-04-09T18:28:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4188a3700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414188d40 0x7fe41417e010 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7fe3fc00b5b0 tx=0x7fe3fc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4117fa700 1 -- 172.21.15.32:0/2603169306 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3fc00bd40 con 0x7fe414188d40 2021-04-09T18:28:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4117fa700 1 -- 172.21.15.32:0/2603169306 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc006240 con 0x7fe414188d40 2021-04-09T18:28:38.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4141819c0 con 0x7fe414188d40 2021-04-09T18:28:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4117fa700 1 -- 172.21.15.32:0/2603169306 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3fc005690 con 0x7fe414188d40 2021-04-09T18:28:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4141819c0 con 0x7fe414188d40 2021-04-09T18:28:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.660 7fe4117fa700 1 -- 172.21.15.32:0/2603169306 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe3fc0054a0 con 0x7fe414188d40 2021-04-09T18:28:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.661 7fe4117fa700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f8018930 0x7fe3f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:38.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.661 7fe4117fa700 1 -- 172.21.15.32:0/2603169306 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe3fc032a30 con 0x7fe414188d40 2021-04-09T18:28:38.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.661 7fe413fff700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f8018930 0x7fe3f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:38.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.661 7fe413fff700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f8018930 0x7fe3f801ae10 crc :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:38.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.681 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe414181c60 con 0x7fe414188d40 2021-04-09T18:28:38.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.684 7fe4117fa700 1 -- 172.21.15.32:0/2603169306 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe3fc000c60 con 0x7fe414188d40 2021-04-09T18:28:38.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.875 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe414181c60 con 0x7fe414188d40 2021-04-09T18:28:38.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.876 7fe4117fa700 1 -- 172.21.15.32:0/2603169306 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fe3fc0054a0 con 0x7fe414188d40 2021-04-09T18:28:38.957 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2021-04-09T18:28:38.958 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:38.958 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:34.236678","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9868":{"gid":9868,"name":"a","rank":0,"incarnation":295,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1939434763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1939434763},{"type":"v1","addr":"172.21.15.32:6835","nonce":1939434763}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:38.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f8018930 msgr2=0x7fe3f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f8018930 0x7fe3f801ae10 crc :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414188d40 msgr2=0x7fe41417e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:38.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414188d40 0x7fe41417e010 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7fe3fc00b5b0 tx=0x7fe3fc00b950).stop 2021-04-09T18:28:38.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 shutdown_connections 2021-04-09T18:28:38.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f8018930 0x7fe3f801ae10 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe414182810 0x7fe4141813a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe414182410 0x7fe4141a69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 --2- 172.21.15.32:0/2603169306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe414188d40 0x7fe41417e010 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:38.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 >> 172.21.15.32:0/2603169306 conn(0x7fe4141965a0 msgr2=0x7fe41419dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:38.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 shutdown_connections 2021-04-09T18:28:38.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:38.878 7fe41aa87700 1 -- 172.21.15.32:0/2603169306 wait complete. 2021-04-09T18:28:39.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 297, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:28:32.232161', 'modified': '2021-04-09 18:28:34.236678', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9868}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9868': {'gid': 9868, 'name': 'a', 'rank': 0, 'incarnation': 295, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/1939434763', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 1939434763}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 1939434763}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:28:39.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:28:39.373 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:28:39.513 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:28:39.514 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:28:39.515 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:28:39.515 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 9868, 2021-04-09T18:28:39.516 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:28:39.516 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:28:39.517 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1994493019999997, 2021-04-09T18:28:39.517 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 297, 2021-04-09T18:28:39.518 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 155, 2021-04-09T18:28:39.518 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 155, 2021-04-09T18:28:39.519 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 12.672250818 2021-04-09T18:28:39.519 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:28:39.521 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 9868, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.1994493019999997, "mdsmap_epoch": 297, "osdmap_epoch": 155, "osdmap_epoch_barrier": 155, "uptime": 12.672250818 } 2021-04-09T18:28:39.522 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:28:39.523 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:28:39.524 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:28:39.524 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:39.525 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:28:39.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:28:39.669 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:28:39.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:28:39.729 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:28:39.730 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:28:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:28:39.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:39.733 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:28:39.778 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:28:39.779 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:28:39.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:28:39.881 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:28:39.882 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:28:39.941 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:28:39.942 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:28:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:28:39.943 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:39.944 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:28:39.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:28:39.888 7f39b8f42f80 -1 init, newargv = 0x55d150c1f270 newargc=9ceph-fuse[84477]: starting ceph client 2021-04-09T18:28:39.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr: 2021-04-09T18:28:39.974 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[84477]: starting fuse 2021-04-09T18:28:39.988 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:28:39.989 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:28:39.989 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:39.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:28:39.990 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:28:39.990 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:28:39.991 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:28:39.991 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:28:39.992 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:28:39.992 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:28:39.992 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:28:39.993 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:28:39.993 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:28:39.994 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:28:39.994 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:28:39.994 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:28:39.995 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:28:39.995 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:28:39.996 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:28:39.996 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:28:39.996 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:28:39.997 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:28:39.997 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:28:39.998 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:28:39.998 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:28:39.998 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:28:40.110 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.84477.asok 2021-04-09T18:28:40.114 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.84477.asok 2021-04-09T18:28:40.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:40.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.84477.asok status 2021-04-09T18:28:40.239 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:28:40.239 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:28:40.239 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:28:40.240 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:28:40.240 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:28:40.240 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:28:40.240 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:28:40.241 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "84477", 2021-04-09T18:28:40.241 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:28:40.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:40.241 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:28:40.241 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:28:40.242 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 10335, 2021-04-09T18:28:40.242 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:28:40.242 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:28:40.242 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:28:40.242 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 10335 2021-04-09T18:28:40.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:40.243 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:28:40.243 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:40.243 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:28:40.244 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2608544493 2021-04-09T18:28:40.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:28:40.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:40.244 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:28:40.244 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:28:40.245 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:28:40.245 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2608544493 2021-04-09T18:28:40.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:28:40.245 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.10335 172.21.15.32:0/2608544493", 2021-04-09T18:28:40.245 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/2608544493", 2021-04-09T18:28:40.246 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:28:40.246 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 297, 2021-04-09T18:28:40.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 155, 2021-04-09T18:28:40.246 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:28:40.247 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:28:40.247 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:28:40.247 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:28:40.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:40.249 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:28:40.249 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:28:40.293 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:28:40.294 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:28:40.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:28:40.295 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:28:40.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:40.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.421 7fce26109700 1 Processor -- start 2021-04-09T18:28:40.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.421 7fce26109700 1 -- start start 2021-04-09T18:28:40.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce26109700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fce2010d010 0x7fce200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce26109700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce200d1da0 0x7fce200d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce26109700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2010d480 con 0x7fce200d82c0 2021-04-09T18:28:40.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce26109700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce200d9220 con 0x7fce2010d010 2021-04-09T18:28:40.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce26109700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce200d86c0 con 0x7fce200d1da0 2021-04-09T18:28:40.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce1f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fce2010d010 0x7fce200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce1ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fce200d82c0 0x7fce2010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35690/0 (socket says 172.21.15.32:35690) 2021-04-09T18:28:40.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce1ffff700 1 -- 172.21.15.32:0/1269011343 learned_addr learned my addr 172.21.15.32:0/1269011343 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:40.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce1f7fe700 1 --2- 172.21.15.32:0/1269011343 >> v2:172.21.15.32:3300/0 conn(0x7fce2010d010 0x7fce200d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54330/0 (socket says 172.21.15.32:54330) 2021-04-09T18:28:40.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.422 7fce1effd700 1 --2- 172.21.15.32:0/1269011343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce200d1da0 0x7fce200d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1f7fe700 1 -- 172.21.15.32:0/1269011343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce200d1da0 msgr2=0x7fce200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:40.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1f7fe700 1 --2- 172.21.15.32:0/1269011343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce200d1da0 0x7fce200d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1f7fe700 1 -- 172.21.15.32:0/1269011343 >> v1:172.21.15.32:6789/0 conn(0x7fce200d82c0 legacy=0x7fce2010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:40.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1f7fe700 1 -- 172.21.15.32:0/1269011343 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2010e1f0 con 0x7fce2010d010 2021-04-09T18:28:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1e7fc700 1 -- 172.21.15.32:0/1269011343 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039037140 0 0) 0x7fce18000d30 con 0x7fce200d82c0 2021-04-09T18:28:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1effd700 1 --2- 172.21.15.32:0/1269011343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce200d1da0 0x7fce200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:40.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1f7fe700 1 --2- 172.21.15.32:0/1269011343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce2010d010 0x7fce200d1780 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7fce10006650 tx=0x7fce10009a80).ready entity=mon.0 client_cookie=f955c7f7782345ef server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:40.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1e7fc700 1 -- 172.21.15.32:0/1269011343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce100009b0 con 0x7fce2010d010 2021-04-09T18:28:40.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1e7fc700 1 -- 172.21.15.32:0/1269011343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1000c0c0 con 0x7fce2010d010 2021-04-09T18:28:40.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce1e7fc700 1 -- 172.21.15.32:0/1269011343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce1000c510 con 0x7fce2010d010 2021-04-09T18:28:40.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce26109700 1 -- 172.21.15.32:0/1269011343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce2010d010 msgr2=0x7fce200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:40.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.423 7fce26109700 1 --2- 172.21.15.32:0/1269011343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce2010d010 0x7fce200d1780 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7fce10006650 tx=0x7fce10009a80).stop 2021-04-09T18:28:40.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.424 7fce26109700 1 -- 172.21.15.32:0/1269011343 shutdown_connections 2021-04-09T18:28:40.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.424 7fce26109700 1 --2- 172.21.15.32:0/1269011343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce200d1da0 0x7fce200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.424 7fce26109700 1 --2- 172.21.15.32:0/1269011343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce2010d010 0x7fce200d1780 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.424 7fce26109700 1 -- 172.21.15.32:0/1269011343 >> 172.21.15.32:0/1269011343 conn(0x7fce200ce5f0 msgr2=0x7fce200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:40.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.424 7fce26109700 1 -- 172.21.15.32:0/1269011343 shutdown_connections 2021-04-09T18:28:40.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.424 7fce26109700 1 -- 172.21.15.32:0/1269011343 wait complete. 2021-04-09T18:28:40.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.444 7fce26109700 1 Processor -- start 2021-04-09T18:28:40.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.444 7fce26109700 1 -- start start 2021-04-09T18:28:40.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce26109700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce200a2140 0x7fce200a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce26109700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce201acc70 0x7fce2009f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce26109700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce201ad070 0x7fce2009fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce26109700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce201a91e0 con 0x7fce201ad070 2021-04-09T18:28:40.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce26109700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce200a13c0 con 0x7fce200a2140 2021-04-09T18:28:40.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce26109700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce200c4410 con 0x7fce201acc70 2021-04-09T18:28:40.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce201ad070 0x7fce2009fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce201ad070 0x7fce2009fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54340/0 (socket says 172.21.15.32:54340) 2021-04-09T18:28:40.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1ffff700 1 -- 172.21.15.32:0/923097691 learned_addr learned my addr 172.21.15.32:0/923097691 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:40.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1effd700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce201acc70 0x7fce2009f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1f7fe700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce200a2140 0x7fce200a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1ffff700 1 -- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce201acc70 msgr2=0x7fce2009f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:40.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1ffff700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce201acc70 0x7fce2009f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1ffff700 1 -- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce200a2140 msgr2=0x7fce200a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:40.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1ffff700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce200a2140 0x7fce200a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.445 7fce1ffff700 1 -- 172.21.15.32:0/923097691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce200a02e0 con 0x7fce201ad070 2021-04-09T18:28:40.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.446 7fce1ffff700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce201ad070 0x7fce2009fcc0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7fce18007f90 tx=0x7fce1800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:40.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.446 7fce17fff700 1 -- 172.21.15.32:0/923097691 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce18000cc0 con 0x7fce201ad070 2021-04-09T18:28:40.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.446 7fce17fff700 1 -- 172.21.15.32:0/923097691 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce18007020 con 0x7fce201ad070 2021-04-09T18:28:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.446 7fce26109700 1 -- 172.21.15.32:0/923097691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce200a02e0 con 0x7fce201ad070 2021-04-09T18:28:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.446 7fce17fff700 1 -- 172.21.15.32:0/923097691 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce18007500 con 0x7fce201ad070 2021-04-09T18:28:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.446 7fce26109700 1 -- 172.21.15.32:0/923097691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce200a02e0 con 0x7fce201ad070 2021-04-09T18:28:40.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.447 7fce17fff700 1 -- 172.21.15.32:0/923097691 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fce18007200 con 0x7fce201ad070 2021-04-09T18:28:40.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.447 7fce17fff700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdfc0188d0 0x7fcdfc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.447 7fce17fff700 1 -- 172.21.15.32:0/923097691 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fce18032d50 con 0x7fce201ad070 2021-04-09T18:28:40.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.447 7fce1f7fe700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdfc0188d0 0x7fcdfc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.447 7fce1f7fe700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdfc0188d0 0x7fcdfc01add0 crc :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:40.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.466 7fce26109700 1 -- 172.21.15.32:0/923097691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2010d480 con 0x7fce201ad070 2021-04-09T18:28:40.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.470 7fce17fff700 1 -- 172.21.15.32:0/923097691 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fce18007d00 con 0x7fce201ad070 2021-04-09T18:28:40.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.661 7fce26109700 1 -- 172.21.15.32:0/923097691 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce2010d480 con 0x7fce201ad070 2021-04-09T18:28:40.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.662 7fce17fff700 1 -- 172.21.15.32:0/923097691 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fce18006f00 con 0x7fce201ad070 2021-04-09T18:28:40.744 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2021-04-09T18:28:40.745 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:40.746 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:34.236678","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9868":{"gid":9868,"name":"a","rank":0,"incarnation":295,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1939434763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1939434763},{"type":"v1","addr":"172.21.15.32:6835","nonce":1939434763}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 -- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdfc0188d0 msgr2=0x7fcdfc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdfc0188d0 0x7fcdfc01add0 crc :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 -- 172.21.15.32:0/923097691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce201ad070 msgr2=0x7fce2009fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:40.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce201ad070 0x7fce2009fcc0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7fce18007f90 tx=0x7fce1800ac50).stop 2021-04-09T18:28:40.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 -- 172.21.15.32:0/923097691 shutdown_connections 2021-04-09T18:28:40.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcdfc0188d0 0x7fcdfc01add0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce201ad070 0x7fce2009fcc0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce201acc70 0x7fce2009f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 --2- 172.21.15.32:0/923097691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce200a2140 0x7fce200a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 -- 172.21.15.32:0/923097691 >> 172.21.15.32:0/923097691 conn(0x7fce201a9c60 msgr2=0x7fce200af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:40.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 -- 172.21.15.32:0/923097691 shutdown_connections 2021-04-09T18:28:40.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.664 7fce26109700 1 -- 172.21.15.32:0/923097691 wait complete. 2021-04-09T18:28:40.755 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2021-04-09T18:28:40.756 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:40.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.823 7f372f8bb700 1 Processor -- start 2021-04-09T18:28:40.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.823 7f372f8bb700 1 -- start start 2021-04-09T18:28:40.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.823 7f372f8bb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37280cffa0 0x7f37280d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.823 7f372f8bb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37280d7800 0x7f37280d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.823 7f372f8bb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37280d8220 0x7f37280d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.823 7f372f8bb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37280d91e0 con 0x7f37280cffa0 2021-04-09T18:28:40.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.823 7f372f8bb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37280d7280 con 0x7f37280d8220 2021-04-09T18:28:40.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.823 7f372f8bb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37280d6290 con 0x7f37280d7800 2021-04-09T18:28:40.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372d6d7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37280cffa0 0x7f37280d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372d6d7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f37280cffa0 0x7f37280d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54344/0 (socket says 172.21.15.32:54344) 2021-04-09T18:28:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372d6d7700 1 -- 172.21.15.32:0/2981698360 learned_addr learned my addr 172.21.15.32:0/2981698360 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372ded8700 1 --2- 172.21.15.32:0/2981698360 >> v2:172.21.15.86:3300/0 conn(0x7f37280d8220 0x7f37280d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372ced6700 1 --2- 172.21.15.32:0/2981698360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37280d7800 0x7f37280d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372d6d7700 1 -- 172.21.15.32:0/2981698360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37280d7800 msgr2=0x7f37280d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:40.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372d6d7700 1 --2- 172.21.15.32:0/2981698360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37280d7800 0x7f37280d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372d6d7700 1 -- 172.21.15.32:0/2981698360 >> v2:172.21.15.86:3300/0 conn(0x7f37280d8220 msgr2=0x7f37280d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:40.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372d6d7700 1 --2- 172.21.15.32:0/2981698360 >> v2:172.21.15.86:3300/0 conn(0x7f37280d8220 0x7f37280d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372d6d7700 1 -- 172.21.15.32:0/2981698360 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37280d64b0 con 0x7f37280cffa0 2021-04-09T18:28:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372d6d7700 1 --2- 172.21.15.32:0/2981698360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37280cffa0 0x7f37280d2440 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f3724006700 tx=0x7f3724009b00).ready entity=mon.0 client_cookie=b21c5ba0ff18a36d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f371ffff700 1 -- 172.21.15.32:0/2981698360 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372400c0c0 con 0x7f37280cffa0 2021-04-09T18:28:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f371ffff700 1 -- 172.21.15.32:0/2981698360 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372400c400 con 0x7f37280cffa0 2021-04-09T18:28:40.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f371ffff700 1 -- 172.21.15.32:0/2981698360 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372400c8e0 con 0x7f37280cffa0 2021-04-09T18:28:40.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372f8bb700 1 -- 172.21.15.32:0/2981698360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37280cffa0 msgr2=0x7f37280d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:40.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.824 7f372f8bb700 1 --2- 172.21.15.32:0/2981698360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37280cffa0 0x7f37280d2440 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f3724006700 tx=0x7f3724009b00).stop 2021-04-09T18:28:40.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.825 7f372f8bb700 1 -- 172.21.15.32:0/2981698360 shutdown_connections 2021-04-09T18:28:40.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.825 7f372f8bb700 1 --2- 172.21.15.32:0/2981698360 >> v2:172.21.15.86:3300/0 conn(0x7f37280d8220 0x7f37280d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.825 7f372f8bb700 1 --2- 172.21.15.32:0/2981698360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37280d7800 0x7f37280d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.825 7f372f8bb700 1 --2- 172.21.15.32:0/2981698360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37280cffa0 0x7f37280d2440 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.825 7f372f8bb700 1 -- 172.21.15.32:0/2981698360 >> 172.21.15.32:0/2981698360 conn(0x7f37280ce5c0 msgr2=0x7f37280cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:40.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.825 7f372f8bb700 1 -- 172.21.15.32:0/2981698360 shutdown_connections 2021-04-09T18:28:40.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.825 7f372f8bb700 1 -- 172.21.15.32:0/2981698360 wait complete. 2021-04-09T18:28:40.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.845 7f372f8bb700 1 Processor -- start 2021-04-09T18:28:40.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.845 7f372f8bb700 1 -- start start 2021-04-09T18:28:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.846 7f372f8bb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37281acff0 0x7f372808fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.846 7f372f8bb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3728090150 0x7f372808c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.846 7f372f8bb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372808cca0 0x7f372808e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.846 7f372f8bb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372808c100 con 0x7f3728090150 2021-04-09T18:28:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.846 7f372f8bb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372808d0a0 con 0x7f37281acff0 2021-04-09T18:28:40.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.846 7f372f8bb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372808e960 con 0x7f372808cca0 2021-04-09T18:28:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.846 7f372ced6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3728090150 0x7f372808c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.846 7f372ced6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3728090150 0x7f372808c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54352/0 (socket says 172.21.15.32:54352) 2021-04-09T18:28:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.846 7f372ced6700 1 -- 172.21.15.32:0/3125219852 learned_addr learned my addr 172.21.15.32:0/3125219852 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372ded8700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372808cca0 0x7f372808e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372d6d7700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37281acff0 0x7f372808fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372ced6700 1 -- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372808cca0 msgr2=0x7f372808e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372ced6700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372808cca0 0x7f372808e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372ced6700 1 -- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37281acff0 msgr2=0x7f372808fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:40.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372ced6700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37281acff0 0x7f372808fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:40.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372ced6700 1 -- 172.21.15.32:0/3125219852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372808eb80 con 0x7f3728090150 2021-04-09T18:28:40.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372ced6700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3728090150 0x7f372808c680 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7f371800b580 tx=0x7f371800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:40.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f371dffb700 1 -- 172.21.15.32:0/3125219852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3718006900 con 0x7f3728090150 2021-04-09T18:28:40.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f371dffb700 1 -- 172.21.15.32:0/3125219852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371800bd10 con 0x7f3728090150 2021-04-09T18:28:40.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372808eb80 con 0x7f3728090150 2021-04-09T18:28:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f371dffb700 1 -- 172.21.15.32:0/3125219852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3718002030 con 0x7f3728090150 2021-04-09T18:28:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.847 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372808eb80 con 0x7f3728090150 2021-04-09T18:28:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.848 7f371dffb700 1 -- 172.21.15.32:0/3125219852 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f37180068b0 con 0x7f3728090150 2021-04-09T18:28:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.848 7f371dffb700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3708018930 0x7f370801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:40.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.848 7f371dffb700 1 -- 172.21.15.32:0/3125219852 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3718032a60 con 0x7f3728090150 2021-04-09T18:28:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.848 7f372d6d7700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3708018930 0x7f370801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:40.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.848 7f372d6d7700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3708018930 0x7f370801ae10 crc :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:40.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.868 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3728182dc0 con 0x7f3728090150 2021-04-09T18:28:40.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:40.871 7f371dffb700 1 -- 172.21.15.32:0/3125219852 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3718000940 con 0x7f3728090150 2021-04-09T18:28:41.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.061 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37280d7280 con 0x7f3728090150 2021-04-09T18:28:41.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.062 7f371dffb700 1 -- 172.21.15.32:0/3125219852 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f37180068b0 con 0x7f3728090150 2021-04-09T18:28:41.144 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2021-04-09T18:28:41.144 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:41.144 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:34.236678","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9868":{"gid":9868,"name":"a","rank":0,"incarnation":295,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1939434763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1939434763},{"type":"v1","addr":"172.21.15.32:6835","nonce":1939434763}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:41.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.064 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3708018930 msgr2=0x7f370801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.064 7f372f8bb700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3708018930 0x7f370801ae10 crc :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.064 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3728090150 msgr2=0x7f372808c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.064 7f372f8bb700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3728090150 0x7f372808c680 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7f371800b580 tx=0x7f371800b920).stop 2021-04-09T18:28:41.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.065 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 shutdown_connections 2021-04-09T18:28:41.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.065 7f372f8bb700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3708018930 0x7f370801ae10 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.065 7f372f8bb700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3728090150 0x7f372808c680 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.065 7f372f8bb700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372808cca0 0x7f372808e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.065 7f372f8bb700 1 --2- 172.21.15.32:0/3125219852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37281acff0 0x7f372808fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.065 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 >> 172.21.15.32:0/3125219852 conn(0x7f372809abb0 msgr2=0x7f37280a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:41.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.065 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 shutdown_connections 2021-04-09T18:28:41.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.065 7f372f8bb700 1 -- 172.21.15.32:0/3125219852 wait complete. 2021-04-09T18:28:41.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:41.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.222 7fc85a9c6700 1 Processor -- start 2021-04-09T18:28:41.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.222 7fc85a9c6700 1 -- start start 2021-04-09T18:28:41.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc85a9c6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8540d7ca0 0x7fc8540d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc85a9c6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc85410da00 0x7fc85410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc85a9c6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85410d480 con 0x7fc85410d010 2021-04-09T18:28:41.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc85a9c6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8540d9220 con 0x7fc85410da00 2021-04-09T18:28:41.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc85a9c6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540d86c0 con 0x7fc8540d7ca0 2021-04-09T18:28:41.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc858fe3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc85410da00 0x7fc85410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc858fe3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc85410da00 0x7fc85410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54360/0 (socket says 172.21.15.32:54360) 2021-04-09T18:28:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc858fe3700 1 -- 172.21.15.32:0/4026934185 learned_addr learned my addr 172.21.15.32:0/4026934185 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc84bfff700 1 --2- 172.21.15.32:0/4026934185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8540d7ca0 0x7fc8540d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc858fe3700 1 -- 172.21.15.32:0/4026934185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8540d7ca0 msgr2=0x7fc8540d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:41.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.223 7fc858fe3700 1 --2- 172.21.15.32:0/4026934185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8540d7ca0 0x7fc8540d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.224 7fc858fe3700 1 -- 172.21.15.32:0/4026934185 >> v1:172.21.15.86:6789/0 conn(0x7fc85410d010 legacy=0x7fc8540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.224 7fc858fe3700 1 -- 172.21.15.32:0/4026934185 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8540d1f70 con 0x7fc85410da00 2021-04-09T18:28:41.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.224 7fc858fe3700 1 --2- 172.21.15.32:0/4026934185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85410da00 0x7fc85410de00 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7fc84c006f00 tx=0x7fc84c009e90).ready entity=mon.0 client_cookie=54780137d3515f6e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:41.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.224 7fc8537fe700 1 -- 172.21.15.32:0/4026934185 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc84c00b300 con 0x7fc85410da00 2021-04-09T18:28:41.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.224 7fc8537fe700 1 -- 172.21.15.32:0/4026934185 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c00b640 con 0x7fc85410da00 2021-04-09T18:28:41.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.224 7fc8537fe700 1 -- 172.21.15.32:0/4026934185 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc84c00bb20 con 0x7fc85410da00 2021-04-09T18:28:41.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.224 7fc85a9c6700 1 -- 172.21.15.32:0/4026934185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85410da00 msgr2=0x7fc85410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.224 7fc85a9c6700 1 --2- 172.21.15.32:0/4026934185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85410da00 0x7fc85410de00 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7fc84c006f00 tx=0x7fc84c009e90).stop 2021-04-09T18:28:41.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.225 7fc85a9c6700 1 -- 172.21.15.32:0/4026934185 shutdown_connections 2021-04-09T18:28:41.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.225 7fc85a9c6700 1 --2- 172.21.15.32:0/4026934185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85410da00 0x7fc85410de00 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.225 7fc85a9c6700 1 --2- 172.21.15.32:0/4026934185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8540d7ca0 0x7fc8540d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.225 7fc85a9c6700 1 -- 172.21.15.32:0/4026934185 >> 172.21.15.32:0/4026934185 conn(0x7fc8540ce5f0 msgr2=0x7fc8540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:41.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.225 7fc85a9c6700 1 -- 172.21.15.32:0/4026934185 shutdown_connections 2021-04-09T18:28:41.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.225 7fc85a9c6700 1 -- 172.21.15.32:0/4026934185 wait complete. 2021-04-09T18:28:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.245 7fc85a9c6700 1 Processor -- start 2021-04-09T18:28:41.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.245 7fc85a9c6700 1 -- start start 2021-04-09T18:28:41.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.245 7fc85a9c6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8541aca20 0x7fc8541ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.245 7fc85a9c6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc85418bd40 0x7fc85418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.245 7fc85a9c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85408fba0 0x7fc85408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.245 7fc85a9c6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85418c7e0 con 0x7fc85408fba0 2021-04-09T18:28:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.245 7fc85a9c6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85418ca30 con 0x7fc8541aca20 2021-04-09T18:28:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.245 7fc85a9c6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540905f0 con 0x7fc85418bd40 2021-04-09T18:28:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc858fe3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85408fba0 0x7fc85408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc858fe3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85408fba0 0x7fc85408ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54368/0 (socket says 172.21.15.32:54368) 2021-04-09T18:28:41.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc858fe3700 1 -- 172.21.15.32:0/580413219 learned_addr learned my addr 172.21.15.32:0/580413219 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:41.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc853fff700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8541aca20 0x7fc8541ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc84bfff700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc85418bd40 0x7fc85418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc858fe3700 1 -- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc85418bd40 msgr2=0x7fc85418c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc858fe3700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc85418bd40 0x7fc85418c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc858fe3700 1 -- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8541aca20 msgr2=0x7fc8541ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc858fe3700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8541aca20 0x7fc8541ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc858fe3700 1 -- 172.21.15.32:0/580413219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8540b4300 con 0x7fc85408fba0 2021-04-09T18:28:41.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc84bfff700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc85418bd40 0x7fc85418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:41.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc853fff700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8541aca20 0x7fc8541ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:28:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc858fe3700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85408fba0 0x7fc85408ffa0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7fc84c006fc0 tx=0x7fc84c00df10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc8517fa700 1 -- 172.21.15.32:0/580413219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc84c00b080 con 0x7fc85408fba0 2021-04-09T18:28:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc8517fa700 1 -- 172.21.15.32:0/580413219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c00b340 con 0x7fc85408fba0 2021-04-09T18:28:41.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8540b4300 con 0x7fc85408fba0 2021-04-09T18:28:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc8517fa700 1 -- 172.21.15.32:0/580413219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc84c0099c0 con 0x7fc85408fba0 2021-04-09T18:28:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.246 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8540b4300 con 0x7fc85408fba0 2021-04-09T18:28:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.247 7fc8517fa700 1 -- 172.21.15.32:0/580413219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc84c009720 con 0x7fc85408fba0 2021-04-09T18:28:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.247 7fc8517fa700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc838018900 0x7fc83801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.247 7fc8517fa700 1 -- 172.21.15.32:0/580413219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc84c034fa0 con 0x7fc85408fba0 2021-04-09T18:28:41.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.247 7fc853fff700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc838018900 0x7fc83801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.247 7fc853fff700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc838018900 0x7fc83801adc0 crc :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:41.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.267 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc854182d70 con 0x7fc85408fba0 2021-04-09T18:28:41.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.271 7fc8517fa700 1 -- 172.21.15.32:0/580413219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc84c000cc0 con 0x7fc85408fba0 2021-04-09T18:28:41.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.463 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc85418c760 con 0x7fc85408fba0 2021-04-09T18:28:41.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.464 7fc8517fa700 1 -- 172.21.15.32:0/580413219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fc84c00b340 con 0x7fc85408fba0 2021-04-09T18:28:41.546 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2021-04-09T18:28:41.546 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:41.547 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:34.236678","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9868":{"gid":9868,"name":"a","rank":0,"incarnation":295,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1939434763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1939434763},{"type":"v1","addr":"172.21.15.32:6835","nonce":1939434763}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:41.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.466 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc838018900 msgr2=0x7fc83801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.466 7fc85a9c6700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc838018900 0x7fc83801adc0 crc :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.466 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85408fba0 msgr2=0x7fc85408ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.466 7fc85a9c6700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85408fba0 0x7fc85408ffa0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7fc84c006fc0 tx=0x7fc84c00df10).stop 2021-04-09T18:28:41.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.466 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 shutdown_connections 2021-04-09T18:28:41.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.466 7fc85a9c6700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc838018900 0x7fc83801adc0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.466 7fc85a9c6700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc85408fba0 0x7fc85408ffa0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.467 7fc85a9c6700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc85418bd40 0x7fc85418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.467 7fc85a9c6700 1 --2- 172.21.15.32:0/580413219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8541aca20 0x7fc8541ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.467 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 >> 172.21.15.32:0/580413219 conn(0x7fc85419b460 msgr2=0x7fc85419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:41.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.467 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 shutdown_connections 2021-04-09T18:28:41.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.467 7fc85a9c6700 1 -- 172.21.15.32:0/580413219 wait complete. 2021-04-09T18:28:41.558 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:28:41.558 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:28:41.566 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:28:41.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:28:41.731 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:28:41.732 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:28:41.732 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:28:41.737 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:28:41.738 INFO:tasks.cephfs.test_failover:Waiting for promotion of one of the original standbys {'c', 'b', 'd'} 2021-04-09T18:28:41.738 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:41.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.805 7fd87e2ea700 1 Processor -- start 2021-04-09T18:28:41.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.805 7fd87e2ea700 1 -- start start 2021-04-09T18:28:41.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.806 7fd87e2ea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd87810d010 0x7fd8780d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.806 7fd87e2ea700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8780d8490 0x7fd87810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.806 7fd87e2ea700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87810d480 con 0x7fd8780d1da0 2021-04-09T18:28:41.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.806 7fd87e2ea700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87810e780 con 0x7fd8780d8490 2021-04-09T18:28:41.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.806 7fd87e2ea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780d8d30 con 0x7fd87810d010 2021-04-09T18:28:41.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.806 7fd87c907700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8780d8490 0x7fd87810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.806 7fd86ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd8780d1da0 0x7fd8780d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35738/0 (socket says 172.21.15.32:35738) 2021-04-09T18:28:41.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.806 7fd86ffff700 1 -- 172.21.15.32:0/805424535 learned_addr learned my addr 172.21.15.32:0/805424535 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:28:41.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.806 7fd877fff700 1 --2- 172.21.15.32:0/805424535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd87810d010 0x7fd8780d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd87c907700 1 -- 172.21.15.32:0/805424535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd87810d010 msgr2=0x7fd8780d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:41.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd87c907700 1 --2- 172.21.15.32:0/805424535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd87810d010 0x7fd8780d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd87c907700 1 -- 172.21.15.32:0/805424535 >> v1:172.21.15.32:6789/0 conn(0x7fd8780d1da0 legacy=0x7fd8780d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd87c907700 1 -- 172.21.15.32:0/805424535 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8780d9220 con 0x7fd8780d8490 2021-04-09T18:28:41.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd8777fe700 1 -- 172.21.15.32:0/805424535 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123179945 0 0) 0x7fd860000ce0 con 0x7fd8780d1da0 2021-04-09T18:28:41.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd877fff700 1 --2- 172.21.15.32:0/805424535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd87810d010 0x7fd8780d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:41.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd87c907700 1 --2- 172.21.15.32:0/805424535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8780d8490 0x7fd87810da00 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7fd870006e30 tx=0x7fd870009e60).ready entity=mon.0 client_cookie=51336ac64881cac5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:41.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd8777fe700 1 -- 172.21.15.32:0/805424535 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd870006a50 con 0x7fd8780d8490 2021-04-09T18:28:41.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd8777fe700 1 -- 172.21.15.32:0/805424535 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87000b300 con 0x7fd8780d8490 2021-04-09T18:28:41.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd8777fe700 1 -- 172.21.15.32:0/805424535 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd87000b750 con 0x7fd8780d8490 2021-04-09T18:28:41.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd87e2ea700 1 -- 172.21.15.32:0/805424535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8780d8490 msgr2=0x7fd87810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.807 7fd87e2ea700 1 --2- 172.21.15.32:0/805424535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8780d8490 0x7fd87810da00 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7fd870006e30 tx=0x7fd870009e60).stop 2021-04-09T18:28:41.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.808 7fd87e2ea700 1 -- 172.21.15.32:0/805424535 shutdown_connections 2021-04-09T18:28:41.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.808 7fd87e2ea700 1 --2- 172.21.15.32:0/805424535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8780d8490 0x7fd87810da00 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.808 7fd87e2ea700 1 --2- 172.21.15.32:0/805424535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd87810d010 0x7fd8780d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.808 7fd87e2ea700 1 -- 172.21.15.32:0/805424535 >> 172.21.15.32:0/805424535 conn(0x7fd8780ce5f0 msgr2=0x7fd8780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:41.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.808 7fd87e2ea700 1 -- 172.21.15.32:0/805424535 shutdown_connections 2021-04-09T18:28:41.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.808 7fd87e2ea700 1 -- 172.21.15.32:0/805424535 wait complete. 2021-04-09T18:28:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.828 7fd87e2ea700 1 Processor -- start 2021-04-09T18:28:41.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.828 7fd87e2ea700 1 -- start start 2021-04-09T18:28:41.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd87e2ea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8780b5360 0x7fd8781a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd87e2ea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd87818e140 0x7fd8781aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd87e2ea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8781ad040 0x7fd87808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd87e2ea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8781a5360 con 0x7fd87818e140 2021-04-09T18:28:41.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd87e2ea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780902c0 con 0x7fd8780b5360 2021-04-09T18:28:41.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd87e2ea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780904e0 con 0x7fd8781ad040 2021-04-09T18:28:41.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd86ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd87818e140 0x7fd8781aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd86ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd87818e140 0x7fd8781aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54382/0 (socket says 172.21.15.32:54382) 2021-04-09T18:28:41.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd86ffff700 1 -- 172.21.15.32:0/1061787237 learned_addr learned my addr 172.21.15.32:0/1061787237 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:41.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd877fff700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8780b5360 0x7fd8781a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd87c907700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8781ad040 0x7fd87808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd86ffff700 1 -- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8781ad040 msgr2=0x7fd87808fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd86ffff700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8781ad040 0x7fd87808fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd86ffff700 1 -- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8780b5360 msgr2=0x7fd8781a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:41.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd86ffff700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8780b5360 0x7fd8781a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:41.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd86ffff700 1 -- 172.21.15.32:0/1061787237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd878183c80 con 0x7fd87818e140 2021-04-09T18:28:41.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd877fff700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8780b5360 0x7fd8781a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:28:41.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd86ffff700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd87818e140 0x7fd8781aca20 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7fd860006f00 tx=0x7fd86000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:41.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd8757fa700 1 -- 172.21.15.32:0/1061787237 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd860006020 con 0x7fd87818e140 2021-04-09T18:28:41.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd8757fa700 1 -- 172.21.15.32:0/1061787237 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8600063a0 con 0x7fd87818e140 2021-04-09T18:28:41.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd878183c80 con 0x7fd87818e140 2021-04-09T18:28:41.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd8757fa700 1 -- 172.21.15.32:0/1061787237 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd860004620 con 0x7fd87818e140 2021-04-09T18:28:41.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.829 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd878183c80 con 0x7fd87818e140 2021-04-09T18:28:41.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.830 7fd8757fa700 1 -- 172.21.15.32:0/1061787237 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd86000a000 con 0x7fd87818e140 2021-04-09T18:28:41.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.830 7fd8757fa700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd85c018930 0x7fd85c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:41.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.830 7fd8757fa700 1 -- 172.21.15.32:0/1061787237 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd860032a10 con 0x7fd87818e140 2021-04-09T18:28:41.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.830 7fd877fff700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd85c018930 0x7fd85c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:41.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.831 7fd877fff700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd85c018930 0x7fd85c01ae10 crc :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:41.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.850 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd878182d70 con 0x7fd87818e140 2021-04-09T18:28:41.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:41.853 7fd8757fa700 1 -- 172.21.15.32:0/1061787237 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd860006b90 con 0x7fd87818e140 2021-04-09T18:28:42.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.044 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd87810d480 con 0x7fd87818e140 2021-04-09T18:28:42.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.045 7fd8757fa700 1 -- 172.21.15.32:0/1061787237 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fd86000a000 con 0x7fd87818e140 2021-04-09T18:28:42.126 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2021-04-09T18:28:42.127 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:42.127 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:34.236678","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9868":{"gid":9868,"name":"a","rank":0,"incarnation":295,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1939434763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1939434763},{"type":"v1","addr":"172.21.15.32:6835","nonce":1939434763}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd85c018930 msgr2=0x7fd85c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd85c018930 0x7fd85c01ae10 crc :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd87818e140 msgr2=0x7fd8781aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:42.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd87818e140 0x7fd8781aca20 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7fd860006f00 tx=0x7fd86000b940).stop 2021-04-09T18:28:42.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 shutdown_connections 2021-04-09T18:28:42.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd85c018930 0x7fd85c01ae10 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:42.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd87818e140 0x7fd8781aca20 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:42.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8781ad040 0x7fd87808fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:42.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 --2- 172.21.15.32:0/1061787237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8780b5360 0x7fd8781a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:42.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 >> 172.21.15.32:0/1061787237 conn(0x7fd87819b460 msgr2=0x7fd87819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:42.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 shutdown_connections 2021-04-09T18:28:42.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:42.047 7fd87e2ea700 1 -- 172.21.15.32:0/1061787237 wait complete. 2021-04-09T18:28:42.139 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:28:47.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:47.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.214 7f2ff3a02700 1 Processor -- start 2021-04-09T18:28:47.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.214 7f2ff3a02700 1 -- start start 2021-04-09T18:28:47.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff3a02700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec10d010 0x7f2fec0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:47.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff3a02700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fec0d1da0 0x7f2fec0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:47.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff3a02700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec10d480 con 0x7f2fec0d82c0 2021-04-09T18:28:47.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff3a02700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec10e780 con 0x7f2fec0d1da0 2021-04-09T18:28:47.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff3a02700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0d8d30 con 0x7f2fec10d010 2021-04-09T18:28:47.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff101d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fec0d1da0 0x7f2fec0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:47.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff101d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2fec0d1da0 0x7f2fec0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54398/0 (socket says 172.21.15.32:54398) 2021-04-09T18:28:47.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff101d700 1 -- 172.21.15.32:0/1951362195 learned_addr learned my addr 172.21.15.32:0/1951362195 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:47.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff181e700 1 --2- 172.21.15.32:0/1951362195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec10d010 0x7f2fec0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:47.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff101d700 1 -- 172.21.15.32:0/1951362195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec10d010 msgr2=0x7f2fec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:47.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff101d700 1 --2- 172.21.15.32:0/1951362195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec10d010 0x7f2fec0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff101d700 1 -- 172.21.15.32:0/1951362195 >> v1:172.21.15.86:6789/0 conn(0x7f2fec0d82c0 legacy=0x7f2fec10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:47.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff101d700 1 -- 172.21.15.32:0/1951362195 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec0d9220 con 0x7f2fec0d1da0 2021-04-09T18:28:47.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff101d700 1 --2- 172.21.15.32:0/1951362195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d1da0 0x7f2fec0d7ca0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7f2fdc006700 tx=0x7f2fdc009b00).ready entity=mon.0 client_cookie=b2b33350db120ce8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:47.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.215 7f2ff081c700 1 -- 172.21.15.32:0/1951362195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fdc00c0c0 con 0x7f2fec0d1da0 2021-04-09T18:28:47.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.216 7f2ff081c700 1 -- 172.21.15.32:0/1951362195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc00c400 con 0x7f2fec0d1da0 2021-04-09T18:28:47.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.216 7f2ff081c700 1 -- 172.21.15.32:0/1951362195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fdc00c8e0 con 0x7f2fec0d1da0 2021-04-09T18:28:47.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.216 7f2ff3a02700 1 -- 172.21.15.32:0/1951362195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d1da0 msgr2=0x7f2fec0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:47.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.216 7f2ff3a02700 1 --2- 172.21.15.32:0/1951362195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d1da0 0x7f2fec0d7ca0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7f2fdc006700 tx=0x7f2fdc009b00).stop 2021-04-09T18:28:47.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.216 7f2ff3a02700 1 -- 172.21.15.32:0/1951362195 shutdown_connections 2021-04-09T18:28:47.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.216 7f2ff3a02700 1 --2- 172.21.15.32:0/1951362195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec0d1da0 0x7f2fec0d7ca0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.216 7f2ff3a02700 1 --2- 172.21.15.32:0/1951362195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec10d010 0x7f2fec0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.216 7f2ff3a02700 1 -- 172.21.15.32:0/1951362195 >> 172.21.15.32:0/1951362195 conn(0x7f2fec0ce5f0 msgr2=0x7f2fec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:47.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.217 7f2ff3a02700 1 -- 172.21.15.32:0/1951362195 shutdown_connections 2021-04-09T18:28:47.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.217 7f2ff3a02700 1 -- 172.21.15.32:0/1951362195 wait complete. 2021-04-09T18:28:47.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.237 7f2ff3a02700 1 Processor -- start 2021-04-09T18:28:47.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.237 7f2ff3a02700 1 -- start start 2021-04-09T18:28:47.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.237 7f2ff3a02700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec0b5360 0x7f2fec1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:47.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.237 7f2ff3a02700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec18e140 0x7f2fec1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.237 7f2ff3a02700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fec1ad040 0x7f2fec08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.237 7f2ff3a02700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1a5360 con 0x7f2fec18e140 2021-04-09T18:28:47.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.237 7f2ff3a02700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0902c0 con 0x7f2fec1ad040 2021-04-09T18:28:47.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.237 7f2ff3a02700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0904e0 con 0x7f2fec0b5360 2021-04-09T18:28:47.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.237 7f2ff101d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec18e140 0x7f2fec1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:47.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff101d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec18e140 0x7f2fec1aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54402/0 (socket says 172.21.15.32:54402) 2021-04-09T18:28:47.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff101d700 1 -- 172.21.15.32:0/1633212241 learned_addr learned my addr 172.21.15.32:0/1633212241 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:47.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff181e700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec0b5360 0x7f2fec1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:47.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff201f700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fec1ad040 0x7f2fec08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:47.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff101d700 1 -- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec0b5360 msgr2=0x7f2fec1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:47.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff101d700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec0b5360 0x7f2fec1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff101d700 1 -- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fec1ad040 msgr2=0x7f2fec08fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:47.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff101d700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fec1ad040 0x7f2fec08fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff101d700 1 -- 172.21.15.32:0/1633212241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec183c80 con 0x7f2fec18e140 2021-04-09T18:28:47.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff101d700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec18e140 0x7f2fec1aca20 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f2fdc006f20 tx=0x7f2fdc00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2fe27fc700 1 -- 172.21.15.32:0/1633212241 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fdc00bdd0 con 0x7f2fec18e140 2021-04-09T18:28:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2fe27fc700 1 -- 172.21.15.32:0/1633212241 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc006440 con 0x7f2fec18e140 2021-04-09T18:28:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec183c80 con 0x7f2fec18e140 2021-04-09T18:28:47.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2fe27fc700 1 -- 172.21.15.32:0/1633212241 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fdc0052e0 con 0x7f2fec18e140 2021-04-09T18:28:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.238 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec183c80 con 0x7f2fec18e140 2021-04-09T18:28:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.239 7f2fe27fc700 1 -- 172.21.15.32:0/1633212241 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2fdc0052e0 con 0x7f2fec18e140 2021-04-09T18:28:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.239 7f2fe27fc700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd4018930 0x7f2fd401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.239 7f2fe27fc700 1 -- 172.21.15.32:0/1633212241 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2fdc032a00 con 0x7f2fec18e140 2021-04-09T18:28:47.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.239 7f2ff181e700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd4018930 0x7f2fd401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:47.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.239 7f2ff181e700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd4018930 0x7f2fd401ae10 crc :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:47.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.259 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec182d70 con 0x7f2fec18e140 2021-04-09T18:28:47.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.262 7f2fe27fc700 1 -- 172.21.15.32:0/1633212241 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2fdc006440 con 0x7f2fec18e140 2021-04-09T18:28:47.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.454 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fec10d480 con 0x7f2fec18e140 2021-04-09T18:28:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.454 7f2fe27fc700 1 -- 172.21.15.32:0/1633212241 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f2fdc00bdd0 con 0x7f2fec18e140 2021-04-09T18:28:47.537 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2021-04-09T18:28:47.537 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:47.538 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:34.236678","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9868":{"gid":9868,"name":"a","rank":0,"incarnation":295,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1939434763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1939434763},{"type":"v1","addr":"172.21.15.32:6835","nonce":1939434763}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:47.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd4018930 msgr2=0x7f2fd401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:47.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd4018930 0x7f2fd401ae10 crc :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec18e140 msgr2=0x7f2fec1aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec18e140 0x7f2fec1aca20 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f2fdc006f20 tx=0x7f2fdc00b9e0).stop 2021-04-09T18:28:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 shutdown_connections 2021-04-09T18:28:47.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2fd4018930 0x7f2fd401ae10 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fec18e140 0x7f2fec1aca20 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fec1ad040 0x7f2fec08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 --2- 172.21.15.32:0/1633212241 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fec0b5360 0x7f2fec1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:47.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 >> 172.21.15.32:0/1633212241 conn(0x7f2fec19b460 msgr2=0x7f2fec19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:47.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 shutdown_connections 2021-04-09T18:28:47.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:47.457 7f2ff3a02700 1 -- 172.21.15.32:0/1633212241 wait complete. 2021-04-09T18:28:47.549 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:28:52.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:52.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.617 7fd623099700 1 Processor -- start 2021-04-09T18:28:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.617 7fd623099700 1 -- start start 2021-04-09T18:28:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd623099700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd61c0d7ca0 0x7fd61c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:52.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd623099700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd61c10da00 0x7fd61c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:52.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd623099700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c10d480 con 0x7fd61c10d010 2021-04-09T18:28:52.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd623099700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd61c0d9220 con 0x7fd61c0d7ca0 2021-04-09T18:28:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd623099700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd61c0d86c0 con 0x7fd61c10da00 2021-04-09T18:28:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd61bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd61c0d7ca0 0x7fd61c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:52.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd61bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd61c0d7ca0 0x7fd61c0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54414/0 (socket says 172.21.15.32:54414) 2021-04-09T18:28:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd61bfff700 1 -- 172.21.15.32:0/371060757 learned_addr learned my addr 172.21.15.32:0/371060757 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:52.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd6216b6700 1 --2- 172.21.15.32:0/371060757 >> v2:172.21.15.86:3300/0 conn(0x7fd61c10da00 0x7fd61c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd61bfff700 1 -- 172.21.15.32:0/371060757 >> v2:172.21.15.86:3300/0 conn(0x7fd61c10da00 msgr2=0x7fd61c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:28:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd61bfff700 1 --2- 172.21.15.32:0/371060757 >> v2:172.21.15.86:3300/0 conn(0x7fd61c10da00 0x7fd61c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd61bfff700 1 -- 172.21.15.32:0/371060757 >> v1:172.21.15.86:6789/0 conn(0x7fd61c10d010 legacy=0x7fd61c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd61b7fe700 1 -- 172.21.15.32:0/371060757 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460788313 0 0) 0x7fd60c000c60 con 0x7fd61c10d010 2021-04-09T18:28:52.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.618 7fd61bfff700 1 -- 172.21.15.32:0/371060757 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c0d1f70 con 0x7fd61c0d7ca0 2021-04-09T18:28:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.619 7fd61bfff700 1 --2- 172.21.15.32:0/371060757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c0d7ca0 0x7fd61c0d80a0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7fd610006700 tx=0x7fd610009b00).ready entity=mon.0 client_cookie=134d67f430923dfb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.619 7fd61b7fe700 1 -- 172.21.15.32:0/371060757 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd61000c0c0 con 0x7fd61c0d7ca0 2021-04-09T18:28:52.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.619 7fd61b7fe700 1 -- 172.21.15.32:0/371060757 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61000c400 con 0x7fd61c0d7ca0 2021-04-09T18:28:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.619 7fd61b7fe700 1 -- 172.21.15.32:0/371060757 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd61000c8e0 con 0x7fd61c0d7ca0 2021-04-09T18:28:52.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.619 7fd623099700 1 -- 172.21.15.32:0/371060757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c0d7ca0 msgr2=0x7fd61c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.619 7fd623099700 1 --2- 172.21.15.32:0/371060757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c0d7ca0 0x7fd61c0d80a0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7fd610006700 tx=0x7fd610009b00).stop 2021-04-09T18:28:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.620 7fd623099700 1 -- 172.21.15.32:0/371060757 shutdown_connections 2021-04-09T18:28:52.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.620 7fd623099700 1 --2- 172.21.15.32:0/371060757 >> v2:172.21.15.86:3300/0 conn(0x7fd61c10da00 0x7fd61c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.620 7fd623099700 1 --2- 172.21.15.32:0/371060757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c0d7ca0 0x7fd61c0d80a0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.620 7fd623099700 1 -- 172.21.15.32:0/371060757 >> 172.21.15.32:0/371060757 conn(0x7fd61c0ce5f0 msgr2=0x7fd61c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.620 7fd623099700 1 -- 172.21.15.32:0/371060757 shutdown_connections 2021-04-09T18:28:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.620 7fd623099700 1 -- 172.21.15.32:0/371060757 wait complete. 2021-04-09T18:28:52.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.640 7fd623099700 1 Processor -- start 2021-04-09T18:28:52.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.640 7fd623099700 1 -- start start 2021-04-09T18:28:52.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.640 7fd623099700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd61c0b5360 0x7fd61c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:52.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.640 7fd623099700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c18e140 0x7fd61c1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:52.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.640 7fd623099700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c1ad040 0x7fd61c08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:52.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.640 7fd623099700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c1a5360 con 0x7fd61c18e140 2021-04-09T18:28:52.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.640 7fd623099700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c0902c0 con 0x7fd61c1ad040 2021-04-09T18:28:52.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.640 7fd623099700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c0904e0 con 0x7fd61c0b5360 2021-04-09T18:28:52.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd61bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c18e140 0x7fd61c1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:52.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd61bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c18e140 0x7fd61c1aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54420/0 (socket says 172.21.15.32:54420) 2021-04-09T18:28:52.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd61bfff700 1 -- 172.21.15.32:0/147942988 learned_addr learned my addr 172.21.15.32:0/147942988 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:52.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd6216b6700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c1ad040 0x7fd61c08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:52.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd620eb5700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd61c0b5360 0x7fd61c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:52.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd61bfff700 1 -- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd61c0b5360 msgr2=0x7fd61c1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:52.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd61bfff700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd61c0b5360 0x7fd61c1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd61bfff700 1 -- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c1ad040 msgr2=0x7fd61c08fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:52.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd61bfff700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c1ad040 0x7fd61c08fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd61bfff700 1 -- 172.21.15.32:0/147942988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c183c80 con 0x7fd61c18e140 2021-04-09T18:28:52.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd61bfff700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c18e140 0x7fd61c1aca20 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 rx=0x7fd610006f20 tx=0x7fd61000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:52.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd6197fa700 1 -- 172.21.15.32:0/147942988 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd61000bdd0 con 0x7fd61c18e140 2021-04-09T18:28:52.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd6197fa700 1 -- 172.21.15.32:0/147942988 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd610006440 con 0x7fd61c18e140 2021-04-09T18:28:52.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd623099700 1 -- 172.21.15.32:0/147942988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61c183c80 con 0x7fd61c18e140 2021-04-09T18:28:52.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd6197fa700 1 -- 172.21.15.32:0/147942988 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6100052e0 con 0x7fd61c18e140 2021-04-09T18:28:52.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.641 7fd623099700 1 -- 172.21.15.32:0/147942988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61c183c80 con 0x7fd61c18e140 2021-04-09T18:28:52.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.642 7fd6197fa700 1 -- 172.21.15.32:0/147942988 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd6100052e0 con 0x7fd61c18e140 2021-04-09T18:28:52.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.642 7fd6197fa700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5fc018930 0x7fd5fc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:52.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.642 7fd6197fa700 1 -- 172.21.15.32:0/147942988 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd610032a00 con 0x7fd61c18e140 2021-04-09T18:28:52.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.642 7fd620eb5700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5fc018930 0x7fd5fc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:52.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.642 7fd620eb5700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5fc018930 0x7fd5fc01ae10 crc :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:52.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.662 7fd623099700 1 -- 172.21.15.32:0/147942988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61c182d70 con 0x7fd61c18e140 2021-04-09T18:28:52.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.665 7fd6197fa700 1 -- 172.21.15.32:0/147942988 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd610006440 con 0x7fd61c18e140 2021-04-09T18:28:52.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.859 7fd623099700 1 -- 172.21.15.32:0/147942988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd61c10d480 con 0x7fd61c18e140 2021-04-09T18:28:52.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.860 7fd6197fa700 1 -- 172.21.15.32:0/147942988 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fd61000bdd0 con 0x7fd61c18e140 2021-04-09T18:28:52.942 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 297 2021-04-09T18:28:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:52.943 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":297,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:34.236678","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9868":{"gid":9868,"name":"a","rank":0,"incarnation":295,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/1939434763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1939434763},{"type":"v1","addr":"172.21.15.32:6835","nonce":1939434763}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:52.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 -- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5fc018930 msgr2=0x7fd5fc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:52.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5fc018930 0x7fd5fc01ae10 crc :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 -- 172.21.15.32:0/147942988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c18e140 msgr2=0x7fd61c1aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:52.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c18e140 0x7fd61c1aca20 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 rx=0x7fd610006f20 tx=0x7fd61000b9e0).stop 2021-04-09T18:28:52.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 -- 172.21.15.32:0/147942988 shutdown_connections 2021-04-09T18:28:52.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd5fc018930 0x7fd5fc01ae10 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd61c18e140 0x7fd61c1aca20 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c1ad040 0x7fd61c08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 --2- 172.21.15.32:0/147942988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd61c0b5360 0x7fd61c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:52.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 -- 172.21.15.32:0/147942988 >> 172.21.15.32:0/147942988 conn(0x7fd61c19b460 msgr2=0x7fd61c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:52.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 -- 172.21.15.32:0/147942988 shutdown_connections 2021-04-09T18:28:52.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:52.862 7fd623099700 1 -- 172.21.15.32:0/147942988 wait complete. 2021-04-09T18:28:52.955 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:28:57.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:28:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.033 7fd41a452700 1 Processor -- start 2021-04-09T18:28:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.033 7fd41a452700 1 -- start start 2021-04-09T18:28:58.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.033 7fd41a452700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd41410da00 0x7fd41410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:58.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.033 7fd41a452700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41410d480 con 0x7fd41410d010 2021-04-09T18:28:58.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd41a452700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4140d1f70 con 0x7fd4140d7ca0 2021-04-09T18:28:58.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd41a452700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4140d8d30 con 0x7fd41410da00 2021-04-09T18:28:58.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd418a6f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd41410da00 0x7fd41410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:58.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd418a6f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd41410da00 0x7fd41410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54432/0 (socket says 172.21.15.32:54432) 2021-04-09T18:28:58.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd418a6f700 1 -- 172.21.15.32:0/2199136249 learned_addr learned my addr 172.21.15.32:0/2199136249 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:58.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd413fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd41410d010 0x7fd4140d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35786/0 (socket says 172.21.15.32:35786) 2021-04-09T18:28:58.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd418a6f700 1 -- 172.21.15.32:0/2199136249 >> v1:172.21.15.86:6789/0 conn(0x7fd4140d7ca0 legacy=0x7fd4140d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd418a6f700 1 -- 172.21.15.32:0/2199136249 >> v1:172.21.15.32:6789/0 conn(0x7fd41410d010 legacy=0x7fd4140d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd418a6f700 1 -- 172.21.15.32:0/2199136249 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4140d9220 con 0x7fd41410da00 2021-04-09T18:28:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd4137fe700 1 -- 172.21.15.32:0/2199136249 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474144505 0 0) 0x7fd404000ce0 con 0x7fd41410d010 2021-04-09T18:28:58.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd418a6f700 1 --2- 172.21.15.32:0/2199136249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41410da00 0x7fd41410de00 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 rx=0x7fd40c006f00 tx=0x7fd40c009e90).ready entity=mon.0 client_cookie=1db6f5ed36bd05f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:58.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd4137fe700 1 -- 172.21.15.32:0/2199136249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd40c00b300 con 0x7fd41410da00 2021-04-09T18:28:58.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd4137fe700 1 -- 172.21.15.32:0/2199136249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40c00b640 con 0x7fd41410da00 2021-04-09T18:28:58.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd4137fe700 1 -- 172.21.15.32:0/2199136249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd40c00bb20 con 0x7fd41410da00 2021-04-09T18:28:58.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd41a452700 1 -- 172.21.15.32:0/2199136249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41410da00 msgr2=0x7fd41410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:58.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.034 7fd41a452700 1 --2- 172.21.15.32:0/2199136249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41410da00 0x7fd41410de00 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 rx=0x7fd40c006f00 tx=0x7fd40c009e90).stop 2021-04-09T18:28:58.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.035 7fd41a452700 1 -- 172.21.15.32:0/2199136249 shutdown_connections 2021-04-09T18:28:58.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.035 7fd41a452700 1 --2- 172.21.15.32:0/2199136249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41410da00 0x7fd41410de00 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:58.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.035 7fd41a452700 1 -- 172.21.15.32:0/2199136249 >> 172.21.15.32:0/2199136249 conn(0x7fd4140ce5f0 msgr2=0x7fd4140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:58.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.035 7fd41a452700 1 -- 172.21.15.32:0/2199136249 shutdown_connections 2021-04-09T18:28:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.035 7fd41a452700 1 -- 172.21.15.32:0/2199136249 wait complete. 2021-04-09T18:28:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.055 7fd41a452700 1 Processor -- start 2021-04-09T18:28:58.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.055 7fd41a452700 1 -- start start 2021-04-09T18:28:58.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd41a452700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4140a75d0 0x7fd4141bef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:58.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd41a452700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4140bc870 0x7fd4140b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:58.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd41a452700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4141bf520 0x7fd4140b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:58.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd41a452700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4141abd30 con 0x7fd4140a75d0 2021-04-09T18:28:58.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd41a452700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140b4290 con 0x7fd4140bc870 2021-04-09T18:28:58.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd41a452700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140a7b00 con 0x7fd4141bf520 2021-04-09T18:28:58.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd413fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4140a75d0 0x7fd4141bef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:58.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd413fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4140a75d0 0x7fd4141bef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54434/0 (socket says 172.21.15.32:54434) 2021-04-09T18:28:58.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd413fff700 1 -- 172.21.15.32:0/3002068991 learned_addr learned my addr 172.21.15.32:0/3002068991 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:28:58.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd418a6f700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4141bf520 0x7fd4140b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:58.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd40bfff700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4140bc870 0x7fd4140b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:58.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd413fff700 1 -- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4141bf520 msgr2=0x7fd4140b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd413fff700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4141bf520 0x7fd4140b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd413fff700 1 -- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4140bc870 msgr2=0x7fd4140b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.056 7fd413fff700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4140bc870 0x7fd4140b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:58.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.057 7fd413fff700 1 -- 172.21.15.32:0/3002068991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4140a7d20 con 0x7fd4140a75d0 2021-04-09T18:28:58.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.057 7fd413fff700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4140a75d0 0x7fd4141bef00 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 rx=0x7fd404006f00 tx=0x7fd40400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:58.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.057 7fd4117fa700 1 -- 172.21.15.32:0/3002068991 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd404006020 con 0x7fd4140a75d0 2021-04-09T18:28:58.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.057 7fd4117fa700 1 -- 172.21.15.32:0/3002068991 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd404006360 con 0x7fd4140a75d0 2021-04-09T18:28:58.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.057 7fd41a452700 1 -- 172.21.15.32:0/3002068991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4140a7d20 con 0x7fd4140a75d0 2021-04-09T18:28:58.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.057 7fd4117fa700 1 -- 172.21.15.32:0/3002068991 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd404004620 con 0x7fd4140a75d0 2021-04-09T18:28:58.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.057 7fd41a452700 1 -- 172.21.15.32:0/3002068991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4140a7d20 con 0x7fd4140a75d0 2021-04-09T18:28:58.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.057 7fd4117fa700 1 -- 172.21.15.32:0/3002068991 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd40400a030 con 0x7fd4140a75d0 2021-04-09T18:28:58.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.058 7fd4117fa700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd3f8018930 0x7fd3f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:28:58.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.058 7fd4117fa700 1 -- 172.21.15.32:0/3002068991 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fd404032a70 con 0x7fd4140a75d0 2021-04-09T18:28:58.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.058 7fd40bfff700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd3f8018930 0x7fd3f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:28:58.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.058 7fd40bfff700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd3f8018930 0x7fd3f801ae10 crc :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:28:58.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.078 7fd41a452700 1 -- 172.21.15.32:0/3002068991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4141abd30 con 0x7fd4140a75d0 2021-04-09T18:28:58.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.081 7fd4117fa700 1 -- 172.21.15.32:0/3002068991 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd404006b90 con 0x7fd4140a75d0 2021-04-09T18:28:58.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.272 7fd41a452700 1 -- 172.21.15.32:0/3002068991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4141abd30 con 0x7fd4140a75d0 2021-04-09T18:28:58.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.272 7fd4117fa700 1 -- 172.21.15.32:0/3002068991 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2450 (secure 0 0 0) 0x7fd40400a030 con 0x7fd4140a75d0 2021-04-09T18:28:58.356 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 298 2021-04-09T18:28:58.356 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:28:58.356 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":298,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":298,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:28:57.593620","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9871":{"gid":9871,"name":"d","rank":0,"incarnation":298,"state":"up:replay","state_seq":2,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:28:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 -- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd3f8018930 msgr2=0x7fd3f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd3f8018930 0x7fd3f801ae10 crc :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 -- 172.21.15.32:0/3002068991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4140a75d0 msgr2=0x7fd4141bef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:28:58.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4140a75d0 0x7fd4141bef00 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 rx=0x7fd404006f00 tx=0x7fd40400b940).stop 2021-04-09T18:28:58.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 -- 172.21.15.32:0/3002068991 shutdown_connections 2021-04-09T18:28:58.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd3f8018930 0x7fd3f801ae10 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:58.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4141bf520 0x7fd4140b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:58.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4140bc870 0x7fd4140b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 --2- 172.21.15.32:0/3002068991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4140a75d0 0x7fd4141bef00 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:28:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 -- 172.21.15.32:0/3002068991 >> 172.21.15.32:0/3002068991 conn(0x7fd4141acbe0 msgr2=0x7fd41408dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:28:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 -- 172.21.15.32:0/3002068991 shutdown_connections 2021-04-09T18:28:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:28:58.275 7fd41a452700 1 -- 172.21.15.32:0/3002068991 wait complete. 2021-04-09T18:28:58.368 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:29:03.370 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:03.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.437 7f6d4d73c700 1 Processor -- start 2021-04-09T18:29:03.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d4d73c700 1 -- start start 2021-04-09T18:29:03.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d4d73c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6d480cffa0 0x7f6d480d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d4d73c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d480d6c80 0x7f6d4810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d4d73c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d480d7280 con 0x7f6d480d7800 2021-04-09T18:29:03.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d4d73c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d480d7080 con 0x7f6d480cffa0 2021-04-09T18:29:03.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d4d73c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d480d91e0 con 0x7f6d480d6c80 2021-04-09T18:29:03.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d47549700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6d480cffa0 0x7f6d480d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d46d48700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6d480d7800 0x7f6d480d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35806/0 (socket says 172.21.15.32:35806) 2021-04-09T18:29:03.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d46d48700 1 -- 172.21.15.32:0/4213837056 learned_addr learned my addr 172.21.15.32:0/4213837056 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:03.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.438 7f6d47d4a700 1 --2- 172.21.15.32:0/4213837056 >> v2:172.21.15.86:3300/0 conn(0x7f6d480d6c80 0x7f6d4810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d47549700 1 -- 172.21.15.32:0/4213837056 >> v2:172.21.15.86:3300/0 conn(0x7f6d480d6c80 msgr2=0x7f6d4810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:03.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d47549700 1 --2- 172.21.15.32:0/4213837056 >> v2:172.21.15.86:3300/0 conn(0x7f6d480d6c80 0x7f6d4810cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d47549700 1 -- 172.21.15.32:0/4213837056 >> v1:172.21.15.32:6789/0 conn(0x7f6d480d7800 legacy=0x7f6d480d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d47549700 1 -- 172.21.15.32:0/4213837056 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d480d83f0 con 0x7f6d480cffa0 2021-04-09T18:29:03.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d46547700 1 -- 172.21.15.32:0/4213837056 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924867126 0 0) 0x7f6d340009b0 con 0x7f6d480d7800 2021-04-09T18:29:03.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d47549700 1 --2- 172.21.15.32:0/4213837056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d480cffa0 0x7f6d480d2440 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f6d38006650 tx=0x7f6d38009a00).ready entity=mon.0 client_cookie=fddb919b3cfe7ffb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:03.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d46547700 1 -- 172.21.15.32:0/4213837056 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d38009df0 con 0x7f6d480cffa0 2021-04-09T18:29:03.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d46547700 1 -- 172.21.15.32:0/4213837056 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d380009b0 con 0x7f6d480cffa0 2021-04-09T18:29:03.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d46547700 1 -- 172.21.15.32:0/4213837056 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d3800c2b0 con 0x7f6d480cffa0 2021-04-09T18:29:03.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d4d73c700 1 -- 172.21.15.32:0/4213837056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d480cffa0 msgr2=0x7f6d480d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.439 7f6d4d73c700 1 --2- 172.21.15.32:0/4213837056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d480cffa0 0x7f6d480d2440 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f6d38006650 tx=0x7f6d38009a00).stop 2021-04-09T18:29:03.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.440 7f6d4d73c700 1 -- 172.21.15.32:0/4213837056 shutdown_connections 2021-04-09T18:29:03.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.440 7f6d4d73c700 1 --2- 172.21.15.32:0/4213837056 >> v2:172.21.15.86:3300/0 conn(0x7f6d480d6c80 0x7f6d4810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.440 7f6d4d73c700 1 --2- 172.21.15.32:0/4213837056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d480cffa0 0x7f6d480d2440 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.440 7f6d4d73c700 1 -- 172.21.15.32:0/4213837056 >> 172.21.15.32:0/4213837056 conn(0x7f6d480ce5c0 msgr2=0x7f6d480cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:03.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.440 7f6d4d73c700 1 -- 172.21.15.32:0/4213837056 shutdown_connections 2021-04-09T18:29:03.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.440 7f6d4d73c700 1 -- 172.21.15.32:0/4213837056 wait complete. 2021-04-09T18:29:03.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.460 7f6d4d73c700 1 Processor -- start 2021-04-09T18:29:03.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.460 7f6d4d73c700 1 -- start start 2021-04-09T18:29:03.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d4d73c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d480a80c0 0x7f6d4817bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d4d73c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d4817b1d0 0x7f6d4817b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d4d73c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d4808dc50 0x7f6d4808e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d4d73c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4817bc70 con 0x7f6d4808dc50 2021-04-09T18:29:03.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d4d73c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4808e670 con 0x7f6d4817b1d0 2021-04-09T18:29:03.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d4d73c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4808e850 con 0x7f6d480a80c0 2021-04-09T18:29:03.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47d4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d4808dc50 0x7f6d4808e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47d4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d4808dc50 0x7f6d4808e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54454/0 (socket says 172.21.15.32:54454) 2021-04-09T18:29:03.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47d4a700 1 -- 172.21.15.32:0/2286193332 learned_addr learned my addr 172.21.15.32:0/2286193332 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:03.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d46d48700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d4817b1d0 0x7f6d4817b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47549700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d480a80c0 0x7f6d4817bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47d4a700 1 -- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d480a80c0 msgr2=0x7f6d4817bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47d4a700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d480a80c0 0x7f6d4817bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47d4a700 1 -- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d4817b1d0 msgr2=0x7f6d4817b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47d4a700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d4817b1d0 0x7f6d4817b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47d4a700 1 -- 172.21.15.32:0/2286193332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4808c660 con 0x7f6d4808dc50 2021-04-09T18:29:03.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d47d4a700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d4808dc50 0x7f6d4808e050 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f6d4000b970 tx=0x7f6d4000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:03.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d3ffff700 1 -- 172.21.15.32:0/2286193332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d40000f90 con 0x7f6d4808dc50 2021-04-09T18:29:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d3ffff700 1 -- 172.21.15.32:0/2286193332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d40006aa0 con 0x7f6d4808dc50 2021-04-09T18:29:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d3ffff700 1 -- 172.21.15.32:0/2286193332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d400015f0 con 0x7f6d4808dc50 2021-04-09T18:29:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4808c660 con 0x7f6d4808dc50 2021-04-09T18:29:03.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.461 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4808c660 con 0x7f6d4808dc50 2021-04-09T18:29:03.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.463 7f6d3ffff700 1 -- 172.21.15.32:0/2286193332 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6d40000ed0 con 0x7f6d4808dc50 2021-04-09T18:29:03.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.463 7f6d3ffff700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6d240188c0 0x7f6d2401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.463 7f6d3ffff700 1 -- 172.21.15.32:0/2286193332 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f6d40032b60 con 0x7f6d4808dc50 2021-04-09T18:29:03.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.463 7f6d47549700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6d240188c0 0x7f6d2401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.463 7f6d47549700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6d240188c0 0x7f6d2401adc0 crc :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:03.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.482 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4808e850 con 0x7f6d4808dc50 2021-04-09T18:29:03.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.486 7f6d3ffff700 1 -- 172.21.15.32:0/2286193332 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6d400066f0 con 0x7f6d4808dc50 2021-04-09T18:29:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.676 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d4808e850 con 0x7f6d4808dc50 2021-04-09T18:29:03.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.676 7f6d3ffff700 1 -- 172.21.15.32:0/2286193332 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 302 v302) v1 ==== 95+0+2451 (secure 0 0 0) 0x7f6d40000ed0 con 0x7f6d4808dc50 2021-04-09T18:29:03.760 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 302 2021-04-09T18:29:03.760 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:03.761 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":302,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:29:01.606816","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9871":{"gid":9871,"name":"d","rank":0,"incarnation":298,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:29:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6d240188c0 msgr2=0x7f6d2401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6d240188c0 0x7f6d2401adc0 crc :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d4808dc50 msgr2=0x7f6d4808e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d4808dc50 0x7f6d4808e050 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f6d4000b970 tx=0x7f6d4000bd10).stop 2021-04-09T18:29:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 shutdown_connections 2021-04-09T18:29:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6d240188c0 0x7f6d2401adc0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d4808dc50 0x7f6d4808e050 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d4817b1d0 0x7f6d4817b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 --2- 172.21.15.32:0/2286193332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d480a80c0 0x7f6d4817bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 >> 172.21.15.32:0/2286193332 conn(0x7f6d48189b00 msgr2=0x7f6d481921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:03.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 shutdown_connections 2021-04-09T18:29:03.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.679 7f6d4d73c700 1 -- 172.21.15.32:0/2286193332 wait complete. 2021-04-09T18:29:03.772 DEBUG:tasks.ceph_test_case:wait_until_true: success in 20s 2021-04-09T18:29:03.772 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:29:03.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:29:03.776 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:29:03.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:03.925 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:29:03.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.843 7f9610a88700 1 Processor -- start 2021-04-09T18:29:03.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.843 7f9610a88700 1 -- start start 2021-04-09T18:29:03.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f9610a88700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c10d010 0x7f960c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f9610a88700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f960c0d8490 0x7f960c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f9610a88700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960c10d480 con 0x7f960c0d1da0 2021-04-09T18:29:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f9610a88700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f960c10e780 con 0x7f960c0d8490 2021-04-09T18:29:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f9610a88700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960c0d8d30 con 0x7f960c10d010 2021-04-09T18:29:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960205d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f960c0d1da0 0x7f960c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35824/0 (socket says 172.21.15.32:35824) 2021-04-09T18:29:03.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960205d700 1 -- 172.21.15.32:0/4252621870 learned_addr learned my addr 172.21.15.32:0/4252621870 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960b05f700 1 --2- 172.21.15.32:0/4252621870 >> v2:172.21.15.86:3300/0 conn(0x7f960c0d8490 0x7f960c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960a85e700 1 --2- 172.21.15.32:0/4252621870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c10d010 0x7f960c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960a05d700 1 -- 172.21.15.32:0/4252621870 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206294208 0 0) 0x7f95f4000ce0 con 0x7f960c0d1da0 2021-04-09T18:29:03.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960a05d700 1 -- 172.21.15.32:0/4252621870 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f8002fe0 con 0x7f960c0d1da0 2021-04-09T18:29:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960b05f700 1 -- 172.21.15.32:0/4252621870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c10d010 msgr2=0x7f960c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960b05f700 1 --2- 172.21.15.32:0/4252621870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c10d010 0x7f960c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960b05f700 1 -- 172.21.15.32:0/4252621870 >> v1:172.21.15.32:6789/0 conn(0x7f960c0d1da0 legacy=0x7f960c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960b05f700 1 -- 172.21.15.32:0/4252621870 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c0d9220 con 0x7f960c0d8490 2021-04-09T18:29:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960a85e700 1 --2- 172.21.15.32:0/4252621870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c10d010 0x7f960c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.844 7f960b05f700 1 --2- 172.21.15.32:0/4252621870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f960c0d8490 0x7f960c10da00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7f9604006e50 tx=0x7f9604009e70).ready entity=mon.1 client_cookie=5a693b947be999c3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:03.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.845 7f960a05d700 1 -- 172.21.15.32:0/4252621870 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9604006a20 con 0x7f960c0d8490 2021-04-09T18:29:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.845 7f960a05d700 1 -- 172.21.15.32:0/4252621870 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9604006790 con 0x7f960c0d8490 2021-04-09T18:29:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.845 7f9610a88700 1 -- 172.21.15.32:0/4252621870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f960c0d8490 msgr2=0x7f960c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.845 7f960a05d700 1 -- 172.21.15.32:0/4252621870 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f960400b700 con 0x7f960c0d8490 2021-04-09T18:29:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.845 7f9610a88700 1 --2- 172.21.15.32:0/4252621870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f960c0d8490 0x7f960c10da00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7f9604006e50 tx=0x7f9604009e70).stop 2021-04-09T18:29:03.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.846 7f9610a88700 1 -- 172.21.15.32:0/4252621870 shutdown_connections 2021-04-09T18:29:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.846 7f9610a88700 1 --2- 172.21.15.32:0/4252621870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f960c0d8490 0x7f960c10da00 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.846 7f9610a88700 1 --2- 172.21.15.32:0/4252621870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c10d010 0x7f960c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.846 7f9610a88700 1 -- 172.21.15.32:0/4252621870 >> 172.21.15.32:0/4252621870 conn(0x7f960c0ce5f0 msgr2=0x7f960c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.846 7f9610a88700 1 -- 172.21.15.32:0/4252621870 shutdown_connections 2021-04-09T18:29:03.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.846 7f9610a88700 1 -- 172.21.15.32:0/4252621870 wait complete. 2021-04-09T18:29:03.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.866 7f9610a88700 1 Processor -- start 2021-04-09T18:29:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.866 7f9610a88700 1 -- start start 2021-04-09T18:29:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f9610a88700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c1a5210 0x7f960c1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f9610a88700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f960c1a6340 0x7f960c1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f9610a88700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c0abaa0 0x7f960c0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f9610a88700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c1a6d60 con 0x7f960c1a5210 2021-04-09T18:29:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f9610a88700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c0ac4c0 con 0x7f960c1a6340 2021-04-09T18:29:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f9610a88700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960c0ac6e0 con 0x7f960c0abaa0 2021-04-09T18:29:03.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c1a5210 0x7f960c1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c1a5210 0x7f960c1a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54476/0 (socket says 172.21.15.32:54476) 2021-04-09T18:29:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960a85e700 1 -- 172.21.15.32:0/1977649639 learned_addr learned my addr 172.21.15.32:0/1977649639 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960205d700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f960c1a6340 0x7f960c1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960b05f700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c0abaa0 0x7f960c0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960a85e700 1 -- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c0abaa0 msgr2=0x7f960c0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960a85e700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c0abaa0 0x7f960c0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960a85e700 1 -- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f960c1a6340 msgr2=0x7f960c1a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:03.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960a85e700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f960c1a6340 0x7f960c1a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:03.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960a85e700 1 -- 172.21.15.32:0/1977649639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c17a720 con 0x7f960c1a5210 2021-04-09T18:29:03.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f960a85e700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c1a5210 0x7f960c1a56c0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f95fc006f40 tx=0x7f95fc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:03.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f9603fff700 1 -- 172.21.15.32:0/1977649639 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95fc00bd80 con 0x7f960c1a5210 2021-04-09T18:29:03.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f9603fff700 1 -- 172.21.15.32:0/1977649639 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95fc000c90 con 0x7f960c1a5210 2021-04-09T18:29:03.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.867 7f9610a88700 1 -- 172.21.15.32:0/1977649639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960c17a720 con 0x7f960c1a5210 2021-04-09T18:29:03.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.868 7f9603fff700 1 -- 172.21.15.32:0/1977649639 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95fc002220 con 0x7f960c1a5210 2021-04-09T18:29:03.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.868 7f9610a88700 1 -- 172.21.15.32:0/1977649639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960c17a720 con 0x7f960c1a5210 2021-04-09T18:29:03.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.868 7f9603fff700 1 -- 172.21.15.32:0/1977649639 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f95fc00a000 con 0x7f960c1a5210 2021-04-09T18:29:03.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.868 7f9603fff700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f95f0018930 0x7f95f001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.868 7f9603fff700 1 -- 172.21.15.32:0/1977649639 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f95fc032a30 con 0x7f960c1a5210 2021-04-09T18:29:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.869 7f960205d700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f95f0018930 0x7f95f001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:03.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.869 7f960205d700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f95f0018930 0x7f95f001ae10 crc :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:03.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.888 7f9610a88700 1 -- 172.21.15.32:0/1977649639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960c1a6d60 con 0x7f960c1a5210 2021-04-09T18:29:03.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:03.892 7f9603fff700 1 -- 172.21.15.32:0/1977649639 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f95fc000c90 con 0x7f960c1a5210 2021-04-09T18:29:04.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.084 7f9610a88700 1 -- 172.21.15.32:0/1977649639 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f960c1a6d60 con 0x7f960c1a5210 2021-04-09T18:29:04.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.085 7f9603fff700 1 -- 172.21.15.32:0/1977649639 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 302 v302) v1 ==== 95+0+2451 (secure 0 0 0) 0x7f95fc006b00 con 0x7f960c1a5210 2021-04-09T18:29:04.169 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 302 2021-04-09T18:29:04.169 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:04.169 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":302,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:29:01.606816","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9871":{"gid":9871,"name":"d","rank":0,"incarnation":298,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:29:04.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.087 7f9610a88700 1 -- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f95f0018930 msgr2=0x7f95f001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:04.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.087 7f9610a88700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f95f0018930 0x7f95f001ae10 crc :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:04.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.087 7f9610a88700 1 -- 172.21.15.32:0/1977649639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c1a5210 msgr2=0x7f960c1a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:04.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.087 7f9610a88700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c1a5210 0x7f960c1a56c0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f95fc006f40 tx=0x7f95fc00b990).stop 2021-04-09T18:29:04.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.088 7f9610a88700 1 -- 172.21.15.32:0/1977649639 shutdown_connections 2021-04-09T18:29:04.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.088 7f9610a88700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f95f0018930 0x7f95f001ae10 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:04.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.088 7f9610a88700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f960c0abaa0 0x7f960c0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:04.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.088 7f9610a88700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f960c1a6340 0x7f960c1a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:04.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.088 7f9610a88700 1 --2- 172.21.15.32:0/1977649639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c1a5210 0x7f960c1a56c0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:04.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.088 7f9610a88700 1 -- 172.21.15.32:0/1977649639 >> 172.21.15.32:0/1977649639 conn(0x7f960c1a99c0 msgr2=0x7f960c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:04.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.088 7f9610a88700 1 -- 172.21.15.32:0/1977649639 shutdown_connections 2021-04-09T18:29:04.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:04.088 7f9610a88700 1 -- 172.21.15.32:0/1977649639 wait complete. 2021-04-09T18:29:04.181 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:29:07.737 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:29:07.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:29:09.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:09.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.259 7f196ea24700 1 Processor -- start 2021-04-09T18:29:09.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.259 7f196ea24700 1 -- start start 2021-04-09T18:29:09.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f196ea24700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196810d010 0x7f19680d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:09.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f196ea24700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196810d480 con 0x7f19680d8490 2021-04-09T18:29:09.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f196ea24700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196810e1f0 con 0x7f19680d1da0 2021-04-09T18:29:09.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f196ea24700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196810e780 con 0x7f196810d010 2021-04-09T18:29:09.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f196d041700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f19680d8490 0x7f196810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35846/0 (socket says 172.21.15.32:35846) 2021-04-09T18:29:09.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f196d041700 1 -- 172.21.15.32:0/75797368 learned_addr learned my addr 172.21.15.32:0/75797368 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:09.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f196c840700 1 --2- 172.21.15.32:0/75797368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196810d010 0x7f19680d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:09.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f19677fe700 1 -- 172.21.15.32:0/75797368 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737897766 0 0) 0x7f195c000c60 con 0x7f19680d1da0 2021-04-09T18:29:09.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f19677fe700 1 -- 172.21.15.32:0/75797368 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1950002fe0 con 0x7f19680d1da0 2021-04-09T18:29:09.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f19677fe700 1 -- 172.21.15.32:0/75797368 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652770602 0 0) 0x7f1960000d30 con 0x7f19680d8490 2021-04-09T18:29:09.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.260 7f19677fe700 1 -- 172.21.15.32:0/75797368 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1950003020 con 0x7f19680d8490 2021-04-09T18:29:09.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.261 7f196c840700 1 -- 172.21.15.32:0/75797368 >> v1:172.21.15.86:6789/0 conn(0x7f19680d1da0 legacy=0x7f19680d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:09.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.261 7f196c840700 1 -- 172.21.15.32:0/75797368 >> v1:172.21.15.32:6789/0 conn(0x7f19680d8490 legacy=0x7f196810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:09.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.261 7f196c840700 1 -- 172.21.15.32:0/75797368 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19680d8d30 con 0x7f196810d010 2021-04-09T18:29:09.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.261 7f196c840700 1 --2- 172.21.15.32:0/75797368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196810d010 0x7f19680d1780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1958006650 tx=0x7f1958009a80).ready entity=mon.2 client_cookie=9ea0e2ba2e8967ab server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:09.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.261 7f19677fe700 1 -- 172.21.15.32:0/75797368 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19580009b0 con 0x7f196810d010 2021-04-09T18:29:09.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.261 7f19677fe700 1 -- 172.21.15.32:0/75797368 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195800c0c0 con 0x7f196810d010 2021-04-09T18:29:09.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.261 7f19677fe700 1 -- 172.21.15.32:0/75797368 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195800c510 con 0x7f196810d010 2021-04-09T18:29:09.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.261 7f196ea24700 1 -- 172.21.15.32:0/75797368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196810d010 msgr2=0x7f19680d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:09.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.261 7f196ea24700 1 --2- 172.21.15.32:0/75797368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196810d010 0x7f19680d1780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1958006650 tx=0x7f1958009a80).stop 2021-04-09T18:29:09.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.262 7f196ea24700 1 -- 172.21.15.32:0/75797368 shutdown_connections 2021-04-09T18:29:09.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.262 7f196ea24700 1 --2- 172.21.15.32:0/75797368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196810d010 0x7f19680d1780 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:09.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.262 7f196ea24700 1 -- 172.21.15.32:0/75797368 >> 172.21.15.32:0/75797368 conn(0x7f19680ce5f0 msgr2=0x7f19680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:09.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.262 7f196ea24700 1 -- 172.21.15.32:0/75797368 shutdown_connections 2021-04-09T18:29:09.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.262 7f196ea24700 1 -- 172.21.15.32:0/75797368 wait complete. 2021-04-09T18:29:09.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.282 7f196ea24700 1 Processor -- start 2021-04-09T18:29:09.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.282 7f196ea24700 1 -- start start 2021-04-09T18:29:09.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196ea24700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f196808df30 0x7f19680a8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:09.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196ea24700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19681b8540 0x7f19681b8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:09.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196ea24700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196817fa80 0x7f196817fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:09.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196ea24700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196817f580 con 0x7f196808df30 2021-04-09T18:29:09.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196ea24700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19681804a0 con 0x7f196817fa80 2021-04-09T18:29:09.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196ea24700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19680a90b0 con 0x7f19681b8540 2021-04-09T18:29:09.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196c840700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f196808df30 0x7f19680a8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:09.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196c840700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f196808df30 0x7f19680a8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54494/0 (socket says 172.21.15.32:54494) 2021-04-09T18:29:09.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196c840700 1 -- 172.21.15.32:0/194778903 learned_addr learned my addr 172.21.15.32:0/194778903 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:09.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196d041700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196817fa80 0x7f196817fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:09.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f1967fff700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19681b8540 0x7f19681b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:09.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196c840700 1 -- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19681b8540 msgr2=0x7f19681b8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:09.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196c840700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19681b8540 0x7f19681b8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:09.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196c840700 1 -- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196817fa80 msgr2=0x7f196817fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:09.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196c840700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196817fa80 0x7f196817fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:09.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196c840700 1 -- 172.21.15.32:0/194778903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1968198ca0 con 0x7f196808df30 2021-04-09T18:29:09.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196c840700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f196808df30 0x7f19680a8a90 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 rx=0x7f1958009650 tx=0x7f195800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f19657fa700 1 -- 172.21.15.32:0/194778903 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19580093a0 con 0x7f196808df30 2021-04-09T18:29:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f19657fa700 1 -- 172.21.15.32:0/194778903 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195800bd50 con 0x7f196808df30 2021-04-09T18:29:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196ea24700 1 -- 172.21.15.32:0/194778903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1968198ca0 con 0x7f196808df30 2021-04-09T18:29:09.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f19657fa700 1 -- 172.21.15.32:0/194778903 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19580052e0 con 0x7f196808df30 2021-04-09T18:29:09.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.283 7f196ea24700 1 -- 172.21.15.32:0/194778903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1968198ca0 con 0x7f196808df30 2021-04-09T18:29:09.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.285 7f19657fa700 1 -- 172.21.15.32:0/194778903 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1958006030 con 0x7f196808df30 2021-04-09T18:29:09.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.285 7f19657fa700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19480188c0 0x7f194801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:09.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.285 7f19657fa700 1 -- 172.21.15.32:0/194778903 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f19580327f0 con 0x7f196808df30 2021-04-09T18:29:09.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.285 7f1967fff700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19480188c0 0x7f194801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:09.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.285 7f1967fff700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19480188c0 0x7f194801adc0 crc :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:09.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.304 7f196ea24700 1 -- 172.21.15.32:0/194778903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19681804a0 con 0x7f196808df30 2021-04-09T18:29:09.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.307 7f19657fa700 1 -- 172.21.15.32:0/194778903 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f19580009c0 con 0x7f196808df30 2021-04-09T18:29:09.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.501 7f196ea24700 1 -- 172.21.15.32:0/194778903 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19681804a0 con 0x7f196808df30 2021-04-09T18:29:09.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.501 7f19657fa700 1 -- 172.21.15.32:0/194778903 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2787 (secure 0 0 0) 0x7f19580009c0 con 0x7f196808df30 2021-04-09T18:29:09.585 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 303 2021-04-09T18:29:09.586 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:09.586 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":303,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/506460258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":506460258},{"type":"v1","addr":"172.21.15.32:6835","nonce":506460258}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":303},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:29:01.606816","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9871":{"gid":9871,"name":"d","rank":0,"incarnation":298,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:29:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 -- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19480188c0 msgr2=0x7f194801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19480188c0 0x7f194801adc0 crc :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 -- 172.21.15.32:0/194778903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f196808df30 msgr2=0x7f19680a8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f196808df30 0x7f19680a8a90 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 rx=0x7f1958009650 tx=0x7f195800b960).stop 2021-04-09T18:29:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 -- 172.21.15.32:0/194778903 shutdown_connections 2021-04-09T18:29:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19480188c0 0x7f194801adc0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196817fa80 0x7f196817fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19681b8540 0x7f19681b8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 --2- 172.21.15.32:0/194778903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f196808df30 0x7f19680a8a90 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 -- 172.21.15.32:0/194778903 >> 172.21.15.32:0/194778903 conn(0x7f1968190570 msgr2=0x7f196818ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 -- 172.21.15.32:0/194778903 shutdown_connections 2021-04-09T18:29:09.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:09.504 7f196ea24700 1 -- 172.21.15.32:0/194778903 wait complete. 2021-04-09T18:29:09.598 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2021-04-09T18:29:09.598 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:29:09.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:29:09.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:29:09.717 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:29:09.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:09.718 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:29:09.718 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:29:09.762 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:29:09.763 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:29:09.763 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:29:09.764 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:29:09.764 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:29:09.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[84477]: fuse finished with error 0 and tester_r 0 2021-04-09T18:29:09.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:09.833 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:29:09.833 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:29:09.877 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:29:09.877 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:29:09.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:29:09.965 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:29:10.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:10.014 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:29:10.015 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:29:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:29:10.066 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:29:10.067 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:29:10.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:29:10.068 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:29:10.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:29:10.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.242 7f85e0ea1700 1 Processor -- start 2021-04-09T18:29:10.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.242 7f85e0ea1700 1 -- start start 2021-04-09T18:29:10.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85e0ea1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85dc0d7ca0 0x7f85dc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85e0ea1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f85dc10da00 0x7f85dc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85e0ea1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85dc10d480 con 0x7f85dc10d010 2021-04-09T18:29:10.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85e0ea1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc0d9220 con 0x7f85dc0d7ca0 2021-04-09T18:29:10.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85e0ea1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc0d86c0 con 0x7f85dc10da00 2021-04-09T18:29:10.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85da312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85dc0d7ca0 0x7f85dc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85da312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85dc0d7ca0 0x7f85dc0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54504/0 (socket says 172.21.15.32:54504) 2021-04-09T18:29:10.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85da312700 1 -- 172.21.15.32:0/1709359973 learned_addr learned my addr 172.21.15.32:0/1709359973 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:10.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85db314700 1 --2- 172.21.15.32:0/1709359973 >> v2:172.21.15.86:3300/0 conn(0x7f85dc10da00 0x7f85dc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85da312700 1 -- 172.21.15.32:0/1709359973 >> v2:172.21.15.86:3300/0 conn(0x7f85dc10da00 msgr2=0x7f85dc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:10.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85da312700 1 --2- 172.21.15.32:0/1709359973 >> v2:172.21.15.86:3300/0 conn(0x7f85dc10da00 0x7f85dc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85da312700 1 -- 172.21.15.32:0/1709359973 >> v1:172.21.15.86:6789/0 conn(0x7f85dc10d010 legacy=0x7f85dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85da312700 1 -- 172.21.15.32:0/1709359973 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc0d1f70 con 0x7f85dc0d7ca0 2021-04-09T18:29:10.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.243 7f85da312700 1 --2- 172.21.15.32:0/1709359973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc0d7ca0 0x7f85dc0d80a0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f85c8006700 tx=0x7f85c8009b80).ready entity=mon.0 client_cookie=d2ce177165954408 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:10.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85d9b11700 1 -- 172.21.15.32:0/1709359973 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85c800c0c0 con 0x7f85dc0d7ca0 2021-04-09T18:29:10.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85d9b11700 1 -- 172.21.15.32:0/1709359973 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c800c440 con 0x7f85dc0d7ca0 2021-04-09T18:29:10.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85d9b11700 1 -- 172.21.15.32:0/1709359973 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85c800c920 con 0x7f85dc0d7ca0 2021-04-09T18:29:10.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85e0ea1700 1 -- 172.21.15.32:0/1709359973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc0d7ca0 msgr2=0x7f85dc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85e0ea1700 1 --2- 172.21.15.32:0/1709359973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc0d7ca0 0x7f85dc0d80a0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f85c8006700 tx=0x7f85c8009b80).stop 2021-04-09T18:29:10.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85e0ea1700 1 -- 172.21.15.32:0/1709359973 shutdown_connections 2021-04-09T18:29:10.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85e0ea1700 1 --2- 172.21.15.32:0/1709359973 >> v2:172.21.15.86:3300/0 conn(0x7f85dc10da00 0x7f85dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85e0ea1700 1 --2- 172.21.15.32:0/1709359973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc0d7ca0 0x7f85dc0d80a0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85e0ea1700 1 -- 172.21.15.32:0/1709359973 >> 172.21.15.32:0/1709359973 conn(0x7f85dc0ce5f0 msgr2=0x7f85dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:10.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85e0ea1700 1 -- 172.21.15.32:0/1709359973 shutdown_connections 2021-04-09T18:29:10.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.244 7f85e0ea1700 1 -- 172.21.15.32:0/1709359973 wait complete. 2021-04-09T18:29:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85e0ea1700 1 Processor -- start 2021-04-09T18:29:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85e0ea1700 1 -- start start 2021-04-09T18:29:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85e0ea1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc18db50 0x7f85dc19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85e0ea1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85dc18df50 0x7f85dc19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85e0ea1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc0b7df0 0x7f85dc0b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85e0ea1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc199b00 con 0x7f85dc18db50 2021-04-09T18:29:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85e0ea1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0b8810 con 0x7f85dc0b7df0 2021-04-09T18:29:10.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85e0ea1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0b8a30 con 0x7f85dc18df50 2021-04-09T18:29:10.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85dab13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc18db50 0x7f85dc19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85dab13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc18db50 0x7f85dc19a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54510/0 (socket says 172.21.15.32:54510) 2021-04-09T18:29:10.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85dab13700 1 -- 172.21.15.32:0/4068008312 learned_addr learned my addr 172.21.15.32:0/4068008312 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:10.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85db314700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc0b7df0 0x7f85dc0b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.265 7f85da312700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85dc18df50 0x7f85dc19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85db314700 1 -- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85dc18df50 msgr2=0x7f85dc19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85db314700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85dc18df50 0x7f85dc19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85db314700 1 -- 172.21.15.32:0/4068008312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc18db50 msgr2=0x7f85dc19a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85db314700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc18db50 0x7f85dc19a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85db314700 1 -- 172.21.15.32:0/4068008312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc198500 con 0x7f85dc0b7df0 2021-04-09T18:29:10.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85dab13700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc18db50 0x7f85dc19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:10.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85db314700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc0b7df0 0x7f85dc0b81f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f85d400b8a0 tx=0x7f85d400bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:10.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85d37fe700 1 -- 172.21.15.32:0/4068008312 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85d4000f90 con 0x7f85dc0b7df0 2021-04-09T18:29:10.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85d37fe700 1 -- 172.21.15.32:0/4068008312 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d400bd80 con 0x7f85dc0b7df0 2021-04-09T18:29:10.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85dc198500 con 0x7f85dc0b7df0 2021-04-09T18:29:10.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85d37fe700 1 -- 172.21.15.32:0/4068008312 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85d40013e0 con 0x7f85dc0b7df0 2021-04-09T18:29:10.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.266 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc198500 con 0x7f85dc0b7df0 2021-04-09T18:29:10.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.267 7f85d37fe700 1 -- 172.21.15.32:0/4068008312 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f85d40011f0 con 0x7f85dc0b7df0 2021-04-09T18:29:10.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.267 7f85d37fe700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85b80188c0 0x7f85b801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.267 7f85d37fe700 1 -- 172.21.15.32:0/4068008312 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f85d4032b00 con 0x7f85dc0b7df0 2021-04-09T18:29:10.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.267 7f85dab13700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85b80188c0 0x7f85b801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.267 7f85dab13700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85b80188c0 0x7f85b801adc0 crc :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:10.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.287 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85dc182d70 con 0x7f85dc0b7df0 2021-04-09T18:29:10.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.291 7f85d37fe700 1 -- 172.21.15.32:0/4068008312 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f85d40066f0 con 0x7f85dc0b7df0 2021-04-09T18:29:10.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.478 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f85dc10d480 con 0x7f85dc0b7df0 2021-04-09T18:29:10.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.479 7f85d37fe700 1 -- 172.21.15.32:0/4068008312 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v156) v1 ==== 81+0+30518 (secure 0 0 0) 0x7f85d40011f0 con 0x7f85dc0b7df0 2021-04-09T18:29:10.568 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:10.569 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:29:10.569 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 156, 2021-04-09T18:29:10.569 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:29:10.569 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:29:10.570 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:28:57.593255", 2021-04-09T18:29:10.570 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:29:10.570 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:29:10.570 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:29:10.571 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:29:10.571 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:29:10.571 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:29:10.571 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:29:10.571 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:29:10.572 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:29:10.572 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:29:10.572 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:29:10.572 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:29:10.573 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:29:10.573 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:29:10.573 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:29:10.573 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 27, 2021-04-09T18:29:10.573 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:29:10.574 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:29:10.574 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:29:10.574 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:29:10.574 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:29:10.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.575 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:29:10.575 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:29:10.575 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:29:10.575 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:29:10.576 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:29:10.576 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:29:10.576 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:29:10.576 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:29:10.576 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:29:10.577 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:29:10.577 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:29:10.577 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:29:10.577 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:29:10.577 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:29:10.578 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:29:10.578 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:29:10.578 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:29:10.578 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:29:10.579 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:29:10.579 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:29:10.579 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:29:10.579 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:29:10.579 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:29:10.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.580 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:29:10.580 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:29:10.580 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:29:10.580 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:29:10.581 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:29:10.581 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:29:10.581 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:29:10.581 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:29:10.582 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:29:10.582 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:29:10.582 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:29:10.582 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:29:10.582 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:29:10.583 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:29:10.583 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:29:10.583 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:29:10.583 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:29:10.584 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:29:10.584 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:29:10.584 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:29:10.584 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:29:10.584 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:29:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:29:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:29:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:29:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:29:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:29:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:29:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:29:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:29:10.586 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:29:10.587 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:29:10.587 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:29:10.587 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:29:10.587 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:29:10.588 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:29:10.588 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:29:10.588 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:29:10.588 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:29:10.588 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:29:10.589 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:29:10.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.589 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.589 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 26, 2021-04-09T18:29:10.590 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:29:10.590 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:28:29.569175", 2021-04-09T18:29:10.590 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:29:10.590 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:29:10.591 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:29:10.591 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:29:10.591 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:29:10.591 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:29:10.591 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:29:10.592 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:29:10.592 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:29:10.592 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:29:10.592 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:29:10.592 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:29:10.593 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:29:10.593 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:29:10.593 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:29:10.593 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:29:10.594 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:29:10.594 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:29:10.594 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:29:10.594 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:29:10.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.595 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "154", 2021-04-09T18:29:10.595 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:29:10.595 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:29:10.595 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:29:10.595 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:29:10.596 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:29:10.596 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:29:10.596 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:29:10.596 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:29:10.597 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:29:10.597 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:29:10.597 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:29:10.597 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:29:10.597 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:29:10.598 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:29:10.598 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:29:10.598 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:29:10.598 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:29:10.598 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:29:10.599 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:29:10.599 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:29:10.599 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:29:10.599 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:29:10.599 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:29:10.600 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:29:10.600 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:29:10.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:29:10.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.601 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:29:10.601 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:29:10.601 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:29:10.601 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:29:10.601 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:29:10.602 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:29:10.602 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:29:10.602 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:29:10.602 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:29:10.602 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:29:10.603 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:29:10.603 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:29:10.603 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:29:10.603 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:29:10.604 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:29:10.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.604 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:29:10.604 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:29:10.604 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:29:10.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.605 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.605 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 27, 2021-04-09T18:29:10.606 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:29:10.606 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:28:30.947070", 2021-04-09T18:29:10.606 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:29:10.606 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:29:10.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:29:10.607 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:29:10.607 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:29:10.607 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:29:10.607 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:29:10.608 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:29:10.608 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:29:10.608 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:29:10.608 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:29:10.609 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:29:10.609 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:29:10.609 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:29:10.609 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:29:10.609 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:29:10.610 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:29:10.610 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:29:10.610 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:29:10.610 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:29:10.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.611 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "155", 2021-04-09T18:29:10.611 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:29:10.611 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:29:10.611 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:29:10.611 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:29:10.612 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:29:10.612 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:29:10.612 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:29:10.612 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:29:10.613 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:29:10.613 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:29:10.613 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:29:10.613 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:29:10.613 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:29:10.614 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:29:10.614 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:29:10.614 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:29:10.614 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:29:10.614 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:29:10.615 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:29:10.615 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:29:10.615 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:29:10.615 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:29:10.616 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:29:10.616 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:29:10.616 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:29:10.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:29:10.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.617 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:29:10.617 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:29:10.617 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:29:10.617 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:29:10.617 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:29:10.618 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:29:10.618 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:29:10.618 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:29:10.618 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:29:10.618 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:29:10.619 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:29:10.619 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:29:10.619 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:29:10.619 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:29:10.620 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:29:10.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.620 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.620 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:29:10.621 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:29:10.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.621 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:29:10.621 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:29:10.621 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:10.622 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:10.622 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:10.622 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:10.622 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:10.623 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:10.623 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:10.623 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:10.623 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:10.623 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:10.624 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:10.624 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.624 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:29:10.625 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:10.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.625 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.626 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:29:10.626 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:10.626 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.626 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.627 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:10.627 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.627 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.627 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:29:10.628 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:10.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.629 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:29:10.629 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:10.629 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.629 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.630 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:10.630 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.630 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:29:10.631 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:10.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.631 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:29:10.632 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:10.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.633 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:10.633 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.633 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.633 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:29:10.634 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:10.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.634 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:29:10.635 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:10.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.636 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:29:10.636 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:29:10.636 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:29:10.636 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:29:10.636 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:10.637 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:10.637 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:10.637 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:29:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:29:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:10.638 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:10.639 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:10.639 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:10.639 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:10.639 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:10.640 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 148, 2021-04-09T18:29:10.640 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:10.640 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:10.640 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:10.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.641 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:29:10.641 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:10.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.642 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:29:10.642 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:10.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.643 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.643 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:10.643 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.644 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:29:10.644 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:10.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.645 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.645 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:29:10.645 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:10.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.646 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.646 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:10.646 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.646 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:29:10.647 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:10.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.648 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:29:10.648 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:10.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.649 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.649 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:10.649 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.649 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.650 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:29:10.650 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:10.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.651 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.651 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:29:10.651 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:10.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.652 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.652 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:29:10.652 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:29:10.652 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:29:10.653 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:29:10.653 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:10.653 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:10.653 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:10.654 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.654 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.654 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:29:10.654 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:29:10.655 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:10.655 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:10.655 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:10.655 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:10.655 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:10.656 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:10.656 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:10.656 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:10.656 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:10.656 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:10.657 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:10.657 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.657 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.658 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:29:10.658 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:10.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.658 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.659 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:29:10.659 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:10.659 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.659 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.660 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:10.660 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.660 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.661 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:29:10.661 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:10.661 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.661 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.662 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:29:10.662 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:10.662 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.662 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.663 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:10.663 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.663 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:29:10.664 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:10.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.665 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:29:10.665 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:10.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.665 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.666 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:10.666 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.666 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:29:10.667 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:10.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.667 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.668 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:29:10.668 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:10.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.668 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.669 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:29:10.669 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:29:10.669 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:29:10.669 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:29:10.669 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:10.670 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:10.670 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:10.670 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.670 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.671 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:29:10.671 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:29:10.671 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:10.671 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:10.672 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:10.672 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:10.672 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:10.672 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:10.672 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:10.673 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:29:10.673 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:10.673 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:10.673 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:10.673 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.674 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.674 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:29:10.674 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:10.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.675 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.675 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:29:10.675 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:10.676 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.677 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.677 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:10.677 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.677 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.678 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:29:10.678 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:10.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.679 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:29:10.679 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:10.679 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.680 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.680 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:10.680 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.681 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.681 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:29:10.681 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:10.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.682 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.682 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:29:10.682 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:10.682 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.683 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:10.683 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.684 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.684 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.684 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:29:10.684 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:10.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.685 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.685 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:29:10.685 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:10.685 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.686 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.686 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:29:10.686 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:29:10.686 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:29:10.687 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:29:10.687 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:10.687 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:10.687 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:10.688 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.688 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:29:10.688 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:29:10.689 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:10.689 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:10.689 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:10.689 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:10.689 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:10.690 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:10.690 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:10.690 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:10.690 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:10.691 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:10.691 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:10.691 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.691 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.691 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.692 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:29:10.692 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:10.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.693 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.693 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:29:10.693 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:10.693 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.693 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.694 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:10.694 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.694 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.695 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:29:10.695 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:10.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.695 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.696 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:29:10.696 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:10.696 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.696 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.697 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:10.697 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.697 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.697 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.698 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:29:10.698 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:10.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.698 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.699 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:29:10.699 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:10.699 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.699 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.700 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:10.700 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.700 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.700 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.701 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:29:10.701 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:10.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.701 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.701 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.702 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:29:10.702 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:10.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.703 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:29:10.703 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:29:10.703 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:29:10.703 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:29:10.704 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:10.704 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:10.704 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:10.704 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.705 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.705 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:29:10.705 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:29:10.705 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:10.705 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:10.706 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:10.706 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:10.706 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:10.706 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:10.707 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:10.707 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:10.707 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:10.707 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:10.707 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:10.708 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.708 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.708 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.708 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:29:10.708 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:10.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:29:10.710 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:10.710 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.710 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.710 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:10.711 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.711 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.711 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:29:10.711 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:10.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.712 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.712 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:29:10.713 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:10.713 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.713 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.713 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:10.714 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.714 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.714 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.714 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:29:10.714 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:10.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.715 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.715 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:29:10.715 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:10.716 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.716 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.716 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:10.717 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.717 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.717 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.717 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:29:10.717 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:29:10.718 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.719 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:29:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:29:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:29:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:29:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:10.720 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.721 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:29:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:29:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:10.722 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:10.723 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:29:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:29:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:10.726 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.727 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:29:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:29:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.729 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.730 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:29:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:29:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:10.732 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:29:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.734 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:29:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.735 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:29:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:29:10.736 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:29:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:29:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:10.737 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:29:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:29:10.738 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:10.739 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 148, 2021-04-09T18:29:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:10.740 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:29:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.742 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:29:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.744 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:29:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.745 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:29:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.747 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:29:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.748 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:29:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:10.750 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:29:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:10.751 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:29:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:29:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:29:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:29:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:29:10.753 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.754 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:29:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:29:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:29:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:10.755 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:10.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:29:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:10.757 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.758 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:29:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:10.759 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:29:10.760 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:29:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:29:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:29:10.765 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:10.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:29:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:10.767 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.768 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:29:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:29:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:29:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:29:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:29:10.769 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:29:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/1939434763": "2021-04-10 18:28:57.593214", 2021-04-09T18:29:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/1939434763": "2021-04-10 18:28:57.593214" 2021-04-09T18:29:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:29:10.770 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:29:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:29:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:29:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:29:10.771 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:29:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:29:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:29:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:29:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:29:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:29:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:29:10.773 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:29:10.773 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:29:10.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85b80188c0 msgr2=0x7f85b801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85b80188c0 0x7f85b801adc0 crc :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc0b7df0 msgr2=0x7f85dc0b81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc0b7df0 0x7f85dc0b81f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f85d400b8a0 tx=0x7f85d400bc40).stop 2021-04-09T18:29:10.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 shutdown_connections 2021-04-09T18:29:10.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85b80188c0 0x7f85b801adc0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc0b7df0 0x7f85dc0b81f0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85dc18df50 0x7f85dc19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 --2- 172.21.15.32:0/4068008312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85dc18db50 0x7f85dc19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 >> 172.21.15.32:0/4068008312 conn(0x7f85dc1a23e0 msgr2=0x7f85dc19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:10.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 shutdown_connections 2021-04-09T18:29:10.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.483 7f85e0ea1700 1 -- 172.21.15.32:0/4068008312 wait complete. 2021-04-09T18:29:10.779 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:10.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.849 7f4ed4cca700 1 Processor -- start 2021-04-09T18:29:10.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.849 7f4ed4cca700 1 -- start start 2021-04-09T18:29:10.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ed4cca700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4ed00d7ca0 0x7f4ed00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ed4cca700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed010d480 con 0x7f4ed010d010 2021-04-09T18:29:10.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ed4cca700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00d86c0 con 0x7f4ed010da00 2021-04-09T18:29:10.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ed4cca700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed00d9220 con 0x7f4ed00d7ca0 2021-04-09T18:29:10.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ece05d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4ed00d7ca0 0x7f4ed00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ece85e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4ed010d010 0x7f4ed00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35872/0 (socket says 172.21.15.32:35872) 2021-04-09T18:29:10.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ece85e700 1 -- 172.21.15.32:0/692486115 learned_addr learned my addr 172.21.15.32:0/692486115 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:10.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ece05d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4ed00d7ca0 0x7f4ed00d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54518/0 (socket says 172.21.15.32:54518) 2021-04-09T18:29:10.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ece05d700 1 -- 172.21.15.32:0/692486115 >> v1:172.21.15.86:6789/0 conn(0x7f4ed010da00 legacy=0x7f4ed010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ece05d700 1 -- 172.21.15.32:0/692486115 >> v1:172.21.15.32:6789/0 conn(0x7f4ed010d010 legacy=0x7f4ed00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ece05d700 1 -- 172.21.15.32:0/692486115 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00d1f70 con 0x7f4ed00d7ca0 2021-04-09T18:29:10.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ecd85c700 1 -- 172.21.15.32:0/692486115 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572261997 0 0) 0x7f4ec0000ce0 con 0x7f4ed010d010 2021-04-09T18:29:10.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ece05d700 1 --2- 172.21.15.32:0/692486115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed00d7ca0 0x7f4ed00d80a0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 rx=0x7f4ec4006650 tx=0x7f4ec4009a00).ready entity=mon.0 client_cookie=c25b8ced079a6f3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:10.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ecd85c700 1 -- 172.21.15.32:0/692486115 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ec4009df0 con 0x7f4ed00d7ca0 2021-04-09T18:29:10.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ecd85c700 1 -- 172.21.15.32:0/692486115 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec40009b0 con 0x7f4ed00d7ca0 2021-04-09T18:29:10.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ecd85c700 1 -- 172.21.15.32:0/692486115 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ec400c2b0 con 0x7f4ed00d7ca0 2021-04-09T18:29:10.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ed4cca700 1 -- 172.21.15.32:0/692486115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed00d7ca0 msgr2=0x7f4ed00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.850 7f4ed4cca700 1 --2- 172.21.15.32:0/692486115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed00d7ca0 0x7f4ed00d80a0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 rx=0x7f4ec4006650 tx=0x7f4ec4009a00).stop 2021-04-09T18:29:10.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.851 7f4ed4cca700 1 -- 172.21.15.32:0/692486115 shutdown_connections 2021-04-09T18:29:10.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.851 7f4ed4cca700 1 --2- 172.21.15.32:0/692486115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed00d7ca0 0x7f4ed00d80a0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.851 7f4ed4cca700 1 -- 172.21.15.32:0/692486115 >> 172.21.15.32:0/692486115 conn(0x7f4ed00ce5f0 msgr2=0x7f4ed00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:10.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.851 7f4ed4cca700 1 -- 172.21.15.32:0/692486115 shutdown_connections 2021-04-09T18:29:10.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.851 7f4ed4cca700 1 -- 172.21.15.32:0/692486115 wait complete. 2021-04-09T18:29:10.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.871 7f4ed4cca700 1 Processor -- start 2021-04-09T18:29:10.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.871 7f4ed4cca700 1 -- start start 2021-04-09T18:29:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ed4cca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed0192ff0 0x7f4ed01a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ed4cca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed01a9e00 0x7f4ed0185c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ed4cca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed0186240 0x7f4ed009d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ed4cca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed01934c0 con 0x7f4ed0186240 2021-04-09T18:29:10.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ed4cca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed01866b0 con 0x7f4ed01a9e00 2021-04-09T18:29:10.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ed4cca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed01868d0 con 0x7f4ed0192ff0 2021-04-09T18:29:10.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ecf05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed0186240 0x7f4ed009d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ecf05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed0186240 0x7f4ed009d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54524/0 (socket says 172.21.15.32:54524) 2021-04-09T18:29:10.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ecf05f700 1 -- 172.21.15.32:0/3521724106 learned_addr learned my addr 172.21.15.32:0/3521724106 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:10.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ece05d700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed01a9e00 0x7f4ed0185c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.872 7f4ece85e700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed0192ff0 0x7f4ed01a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ecf05f700 1 -- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed0192ff0 msgr2=0x7f4ed01a97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ecf05f700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed0192ff0 0x7f4ed01a97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ecf05f700 1 -- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed01a9e00 msgr2=0x7f4ed0185c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:10.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ecf05f700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed01a9e00 0x7f4ed0185c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:10.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ecf05f700 1 -- 172.21.15.32:0/3521724106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed009da80 con 0x7f4ed0186240 2021-04-09T18:29:10.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ece05d700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed01a9e00 0x7f4ed0185c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:10.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ecf05f700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed0186240 0x7f4ed009d460 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f4ec800b8e0 tx=0x7f4ec800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:10.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ebf7fe700 1 -- 172.21.15.32:0/3521724106 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ec8000f40 con 0x7f4ed0186240 2021-04-09T18:29:10.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ebf7fe700 1 -- 172.21.15.32:0/3521724106 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec800c2f0 con 0x7f4ed0186240 2021-04-09T18:29:10.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed009da80 con 0x7f4ed0186240 2021-04-09T18:29:10.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ebf7fe700 1 -- 172.21.15.32:0/3521724106 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ec800c7d0 con 0x7f4ed0186240 2021-04-09T18:29:10.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.873 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed009da80 con 0x7f4ed0186240 2021-04-09T18:29:10.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.874 7f4ebf7fe700 1 -- 172.21.15.32:0/3521724106 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4ec8001220 con 0x7f4ed0186240 2021-04-09T18:29:10.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.874 7f4ebf7fe700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4eb40188d0 0x7f4eb401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:10.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.874 7f4ebf7fe700 1 -- 172.21.15.32:0/3521724106 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f4ec8032e10 con 0x7f4ed0186240 2021-04-09T18:29:10.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.874 7f4ece85e700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4eb40188d0 0x7f4eb401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:10.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.874 7f4ece85e700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4eb40188d0 0x7f4eb401add0 crc :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:10.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.894 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed010d480 con 0x7f4ed0186240 2021-04-09T18:29:10.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:10.897 7f4ebf7fe700 1 -- 172.21.15.32:0/3521724106 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4ec800bdc0 con 0x7f4ed0186240 2021-04-09T18:29:11.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.090 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ed010d480 con 0x7f4ed0186240 2021-04-09T18:29:11.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.091 7f4ebf7fe700 1 -- 172.21.15.32:0/3521724106 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2787 (secure 0 0 0) 0x7f4ec8001220 con 0x7f4ed0186240 2021-04-09T18:29:11.175 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 303 2021-04-09T18:29:11.176 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:11.176 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":303,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/506460258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":506460258},{"type":"v1","addr":"172.21.15.32:6835","nonce":506460258}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":303},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:29:01.606816","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9871":{"gid":9871,"name":"d","rank":0,"incarnation":298,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6837/1399348025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1399348025},{"type":"v1","addr":"172.21.15.86:6837","nonce":1399348025}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:29:11.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4eb40188d0 msgr2=0x7f4eb401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:11.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4eb40188d0 0x7f4eb401add0 crc :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed0186240 msgr2=0x7f4ed009d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:11.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed0186240 0x7f4ed009d460 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f4ec800b8e0 tx=0x7f4ec800bc80).stop 2021-04-09T18:29:11.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 shutdown_connections 2021-04-09T18:29:11.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4eb40188d0 0x7f4eb401add0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ed0186240 0x7f4ed009d460 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed01a9e00 0x7f4ed0185c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 --2- 172.21.15.32:0/3521724106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed0192ff0 0x7f4ed01a97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.093 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 >> 172.21.15.32:0/3521724106 conn(0x7f4ed01aa480 msgr2=0x7f4ed01ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.094 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 shutdown_connections 2021-04-09T18:29:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.094 7f4ed4cca700 1 -- 172.21.15.32:0/3521724106 wait complete. 2021-04-09T18:29:11.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:29:11.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.257 7f3af1408700 1 Processor -- start 2021-04-09T18:29:11.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.257 7f3af1408700 1 -- start start 2021-04-09T18:29:11.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3af1408700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3aec0d6c80 0x7f3aec0d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:11.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3af1408700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3aec0d2440 0x7f3aec10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:11.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3af1408700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aec0d7280 con 0x7f3aec0cffa0 2021-04-09T18:29:11.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3af1408700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aec0d7080 con 0x7f3aec0d2440 2021-04-09T18:29:11.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3af1408700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aec0d91e0 con 0x7f3aec0d6c80 2021-04-09T18:29:11.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aeb87e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3aec0d2440 0x7f3aec10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:11.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aeb87e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3aec0d2440 0x7f3aec10cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54530/0 (socket says 172.21.15.32:54530) 2021-04-09T18:29:11.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aeb87e700 1 -- 172.21.15.32:0/96869144 learned_addr learned my addr 172.21.15.32:0/96869144 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:11.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aea87c700 1 --2- 172.21.15.32:0/96869144 >> v2:172.21.15.86:3300/0 conn(0x7f3aec0d6c80 0x7f3aec0d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:11.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aeb87e700 1 -- 172.21.15.32:0/96869144 >> v2:172.21.15.86:3300/0 conn(0x7f3aec0d6c80 msgr2=0x7f3aec0d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:11.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aeb87e700 1 --2- 172.21.15.32:0/96869144 >> v2:172.21.15.86:3300/0 conn(0x7f3aec0d6c80 0x7f3aec0d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aeb87e700 1 -- 172.21.15.32:0/96869144 >> v1:172.21.15.86:6789/0 conn(0x7f3aec0cffa0 legacy=0x7f3aec0d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:11.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aeb87e700 1 -- 172.21.15.32:0/96869144 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec0d2840 con 0x7f3aec0d2440 2021-04-09T18:29:11.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aeb87e700 1 --2- 172.21.15.32:0/96869144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec0d2440 0x7f3aec10cfd0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f3ae4006b80 tx=0x7f3ae4009b90).ready entity=mon.0 client_cookie=e4d84609525e9679 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:11.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aea07b700 1 -- 172.21.15.32:0/96869144 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae40022e0 con 0x7f3aec0d2440 2021-04-09T18:29:11.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aea07b700 1 -- 172.21.15.32:0/96869144 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4002620 con 0x7f3aec0d2440 2021-04-09T18:29:11.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3aea07b700 1 -- 172.21.15.32:0/96869144 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae4002b00 con 0x7f3aec0d2440 2021-04-09T18:29:11.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3af1408700 1 -- 172.21.15.32:0/96869144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec0d2440 msgr2=0x7f3aec10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:11.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.258 7f3af1408700 1 --2- 172.21.15.32:0/96869144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec0d2440 0x7f3aec10cfd0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f3ae4006b80 tx=0x7f3ae4009b90).stop 2021-04-09T18:29:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.259 7f3af1408700 1 -- 172.21.15.32:0/96869144 shutdown_connections 2021-04-09T18:29:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.259 7f3af1408700 1 --2- 172.21.15.32:0/96869144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec0d2440 0x7f3aec10cfd0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.259 7f3af1408700 1 --2- 172.21.15.32:0/96869144 >> v2:172.21.15.86:3300/0 conn(0x7f3aec0d6c80 0x7f3aec0d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.259 7f3af1408700 1 -- 172.21.15.32:0/96869144 >> 172.21.15.32:0/96869144 conn(0x7f3aec0ce5c0 msgr2=0x7f3aec0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:11.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.259 7f3af1408700 1 -- 172.21.15.32:0/96869144 shutdown_connections 2021-04-09T18:29:11.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.259 7f3af1408700 1 -- 172.21.15.32:0/96869144 wait complete. 2021-04-09T18:29:11.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.279 7f3af1408700 1 Processor -- start 2021-04-09T18:29:11.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.279 7f3af1408700 1 -- start start 2021-04-09T18:29:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3af1408700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec184070 0x7f3aec1955d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3af1408700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aec195bf0 0x7f3aec08ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3af1408700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aec08f550 0x7f3aec08f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:11.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3af1408700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec090570 con 0x7f3aec184070 2021-04-09T18:29:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3af1408700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec184470 con 0x7f3aec08f550 2021-04-09T18:29:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3af1408700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec195ff0 con 0x7f3aec195bf0 2021-04-09T18:29:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec184070 0x7f3aec1955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:11.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec184070 0x7f3aec1955d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54538/0 (socket says 172.21.15.32:54538) 2021-04-09T18:29:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb07d700 1 -- 172.21.15.32:0/2066800915 learned_addr learned my addr 172.21.15.32:0/2066800915 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb87e700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aec08f550 0x7f3aec08f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aea87c700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aec195bf0 0x7f3aec08ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb07d700 1 -- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aec195bf0 msgr2=0x7f3aec08ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:11.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb07d700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aec195bf0 0x7f3aec08ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb07d700 1 -- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aec08f550 msgr2=0x7f3aec08f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:11.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb07d700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aec08f550 0x7f3aec08f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:11.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb07d700 1 -- 172.21.15.32:0/2066800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec1847a0 con 0x7f3aec184070 2021-04-09T18:29:11.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.280 7f3aeb07d700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec184070 0x7f3aec1955d0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f3adc00b530 tx=0x7f3adc00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.281 7f3ad3fff700 1 -- 172.21.15.32:0/2066800915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3adc00bcc0 con 0x7f3aec184070 2021-04-09T18:29:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.281 7f3ad3fff700 1 -- 172.21.15.32:0/2066800915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc006020 con 0x7f3aec184070 2021-04-09T18:29:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.281 7f3af1408700 1 -- 172.21.15.32:0/2066800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec1847a0 con 0x7f3aec184070 2021-04-09T18:29:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.281 7f3ad3fff700 1 -- 172.21.15.32:0/2066800915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3adc0054a0 con 0x7f3aec184070 2021-04-09T18:29:11.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.281 7f3af1408700 1 -- 172.21.15.32:0/2066800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec1847a0 con 0x7f3aec184070 2021-04-09T18:29:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.281 7f3ad3fff700 1 -- 172.21.15.32:0/2066800915 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3adc006200 con 0x7f3aec184070 2021-04-09T18:29:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.282 7f3ad3fff700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3acc018900 0x7f3acc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.282 7f3ad3fff700 1 -- 172.21.15.32:0/2066800915 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f3adc032a80 con 0x7f3aec184070 2021-04-09T18:29:11.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.282 7f3aea87c700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3acc018900 0x7f3acc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:11.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.282 7f3aea87c700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3acc018900 0x7f3acc01adc0 crc :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:11.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.302 7f3af1408700 1 -- 172.21.15.32:0/2066800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aec182e70 con 0x7f3aec184070 2021-04-09T18:29:11.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.305 7f3ad3fff700 1 -- 172.21.15.32:0/2066800915 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3adc00bcc0 con 0x7f3aec184070 2021-04-09T18:29:11.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.485 7f3af1408700 1 -- 172.21.15.32:0/2066800915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3aec182e70 con 0x7f3aec184070 2021-04-09T18:29:11.997 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:29:11.914 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:29:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.919 7f3ad3fff700 1 -- 172.21.15.32:0/2066800915 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v304) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3adc0061d0 con 0x7f3aec184070 2021-04-09T18:29:12.003 INFO:tasks.ceph.mds.d.smithi086.stderr: -1685> 2021-04-09 18:28:26.846 7fa683222700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2021-04-09T18:29:12.033 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:29:12.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 -- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3acc018900 msgr2=0x7f3acc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3acc018900 0x7f3acc01adc0 crc :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 -- 172.21.15.32:0/2066800915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec184070 msgr2=0x7f3aec1955d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec184070 0x7f3aec1955d0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f3adc00b530 tx=0x7f3adc00b8d0).stop 2021-04-09T18:29:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 -- 172.21.15.32:0/2066800915 shutdown_connections 2021-04-09T18:29:12.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3acc018900 0x7f3acc01adc0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3aec08f550 0x7f3aec08f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3aec195bf0 0x7f3aec08ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 --2- 172.21.15.32:0/2066800915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3aec184070 0x7f3aec1955d0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.952 7f3af1408700 1 -- 172.21.15.32:0/2066800915 >> 172.21.15.32:0/2066800915 conn(0x7f3aec1a8f40 msgr2=0x7f3aec188050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:12.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.953 7f3af1408700 1 -- 172.21.15.32:0/2066800915 shutdown_connections 2021-04-09T18:29:12.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:11.953 7f3af1408700 1 -- 172.21.15.32:0/2066800915 wait complete. 2021-04-09T18:29:12.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:12.057 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:29:12.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.111 7f5da91b6700 1 Processor -- start 2021-04-09T18:29:12.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.111 7f5da91b6700 1 -- start start 2021-04-09T18:29:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da91b6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da410d010 0x7f5da40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da91b6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5da40d8490 0x7f5da410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da91b6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da410d480 con 0x7f5da40d1da0 2021-04-09T18:29:12.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da91b6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da410e780 con 0x7f5da40d8490 2021-04-09T18:29:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da91b6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40d8d30 con 0x7f5da410d010 2021-04-09T18:29:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da359e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5da40d8490 0x7f5da410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da359e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5da40d8490 0x7f5da410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54546/0 (socket says 172.21.15.32:54546) 2021-04-09T18:29:12.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da359e700 1 -- 172.21.15.32:0/4151132933 learned_addr learned my addr 172.21.15.32:0/4151132933 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da259c700 1 --1- 172.21.15.32:0/4151132933 >> v1:172.21.15.86:6789/0 conn(0x7f5da40d1da0 0x7f5da40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:33828/0 (socket says 172.21.15.32:33828) 2021-04-09T18:29:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da2d9d700 1 --2- 172.21.15.32:0/4151132933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da410d010 0x7f5da40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da359e700 1 -- 172.21.15.32:0/4151132933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da410d010 msgr2=0x7f5da40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:12.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da359e700 1 --2- 172.21.15.32:0/4151132933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da410d010 0x7f5da40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da359e700 1 -- 172.21.15.32:0/4151132933 >> v1:172.21.15.86:6789/0 conn(0x7f5da40d1da0 legacy=0x7f5da40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da1d9b700 1 -- 172.21.15.32:0/4151132933 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473896684 0 0) 0x7f5d8c000c60 con 0x7f5da40d1da0 2021-04-09T18:29:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da359e700 1 -- 172.21.15.32:0/4151132933 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40d9220 con 0x7f5da40d8490 2021-04-09T18:29:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da359e700 1 --2- 172.21.15.32:0/4151132933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da40d8490 0x7f5da410da00 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7f5d9c006f00 tx=0x7f5d9c009f10).ready entity=mon.0 client_cookie=cefd5097f57a78bf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:12.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da1d9b700 1 -- 172.21.15.32:0/4151132933 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d9c00b3b0 con 0x7f5da40d8490 2021-04-09T18:29:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da1d9b700 1 -- 172.21.15.32:0/4151132933 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c00b730 con 0x7f5da40d8490 2021-04-09T18:29:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.112 7f5da1d9b700 1 -- 172.21.15.32:0/4151132933 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d9c00bc10 con 0x7f5da40d8490 2021-04-09T18:29:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.113 7f5da91b6700 1 -- 172.21.15.32:0/4151132933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da40d8490 msgr2=0x7f5da410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.113 7f5da91b6700 1 --2- 172.21.15.32:0/4151132933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da40d8490 0x7f5da410da00 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7f5d9c006f00 tx=0x7f5d9c009f10).stop 2021-04-09T18:29:12.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.113 7f5da91b6700 1 -- 172.21.15.32:0/4151132933 shutdown_connections 2021-04-09T18:29:12.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.113 7f5da91b6700 1 --2- 172.21.15.32:0/4151132933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da40d8490 0x7f5da410da00 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.113 7f5da91b6700 1 --2- 172.21.15.32:0/4151132933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da410d010 0x7f5da40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.113 7f5da91b6700 1 -- 172.21.15.32:0/4151132933 >> 172.21.15.32:0/4151132933 conn(0x7f5da40ce5f0 msgr2=0x7f5da40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:12.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.113 7f5da91b6700 1 -- 172.21.15.32:0/4151132933 shutdown_connections 2021-04-09T18:29:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.113 7f5da91b6700 1 -- 172.21.15.32:0/4151132933 wait complete. 2021-04-09T18:29:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da91b6700 1 Processor -- start 2021-04-09T18:29:12.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da91b6700 1 -- start start 2021-04-09T18:29:12.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da91b6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da40b5360 0x7f5da41a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da91b6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da418e140 0x7f5da41aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da91b6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da41ad040 0x7f5da408fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da91b6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41a5360 con 0x7f5da41ad040 2021-04-09T18:29:12.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da91b6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40902c0 con 0x7f5da418e140 2021-04-09T18:29:12.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da91b6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40904e0 con 0x7f5da40b5360 2021-04-09T18:29:12.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da41ad040 0x7f5da408fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da41ad040 0x7f5da408fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54552/0 (socket says 172.21.15.32:54552) 2021-04-09T18:29:12.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da359e700 1 -- 172.21.15.32:0/970401697 learned_addr learned my addr 172.21.15.32:0/970401697 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:12.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da259c700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da418e140 0x7f5da41aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.134 7f5da2d9d700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da40b5360 0x7f5da41a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5da359e700 1 -- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da40b5360 msgr2=0x7f5da41a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5da359e700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da40b5360 0x7f5da41a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5da359e700 1 -- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da418e140 msgr2=0x7f5da41aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5da359e700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da418e140 0x7f5da41aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5da359e700 1 -- 172.21.15.32:0/970401697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da4183c80 con 0x7f5da41ad040 2021-04-09T18:29:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5da359e700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da41ad040 0x7f5da408fca0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f5d9c009390 tx=0x7f5d9c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5d9b7fe700 1 -- 172.21.15.32:0/970401697 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d9c003030 con 0x7f5da41ad040 2021-04-09T18:29:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5d9b7fe700 1 -- 172.21.15.32:0/970401697 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c0068e0 con 0x7f5da41ad040 2021-04-09T18:29:12.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5da91b6700 1 -- 172.21.15.32:0/970401697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da4183c80 con 0x7f5da41ad040 2021-04-09T18:29:12.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5d9b7fe700 1 -- 172.21.15.32:0/970401697 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d9c00c570 con 0x7f5da41ad040 2021-04-09T18:29:12.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.135 7f5da91b6700 1 -- 172.21.15.32:0/970401697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da4183c80 con 0x7f5da41ad040 2021-04-09T18:29:12.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.136 7f5d9b7fe700 1 -- 172.21.15.32:0/970401697 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5d9c005820 con 0x7f5da41ad040 2021-04-09T18:29:12.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.136 7f5d9b7fe700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d88018930 0x7f5d8801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.136 7f5d9b7fe700 1 -- 172.21.15.32:0/970401697 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f5d9c032e10 con 0x7f5da41ad040 2021-04-09T18:29:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.136 7f5da2d9d700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d88018930 0x7f5d8801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.136 7f5da2d9d700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d88018930 0x7f5d8801adc0 crc :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.156 7f5da91b6700 1 -- 172.21.15.32:0/970401697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da4182d70 con 0x7f5da41ad040 2021-04-09T18:29:12.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.159 7f5d9b7fe700 1 -- 172.21.15.32:0/970401697 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5d9c006c80 con 0x7f5da41ad040 2021-04-09T18:29:12.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.351 7f5da91b6700 1 -- 172.21.15.32:0/970401697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5da4182d70 con 0x7f5da41ad040 2021-04-09T18:29:12.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.351 7f5d9b7fe700 1 -- 172.21.15.32:0/970401697 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 304 v304) v1 ==== 95+0+2439 (secure 0 0 0) 0x7f5d9c0068e0 con 0x7f5da41ad040 2021-04-09T18:29:12.436 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 304 2021-04-09T18:29:12.436 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:12.436 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":304,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/506460258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":506460258},{"type":"v1","addr":"172.21.15.32:6835","nonce":506460258}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":303},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294}],"filesystems":[{"mdsmap":{"epoch":304,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:28:32.232161","modified":"2021-04-09 18:29:11.915289","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-04-09T18:29:12.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 -- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d88018930 msgr2=0x7f5d8801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d88018930 0x7f5d8801adc0 crc :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 -- 172.21.15.32:0/970401697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da41ad040 msgr2=0x7f5da408fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da41ad040 0x7f5da408fca0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f5d9c009390 tx=0x7f5d9c00bd40).stop 2021-04-09T18:29:12.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 -- 172.21.15.32:0/970401697 shutdown_connections 2021-04-09T18:29:12.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5d88018930 0x7f5d8801adc0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5da41ad040 0x7f5da408fca0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da418e140 0x7f5da41aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 --2- 172.21.15.32:0/970401697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5da40b5360 0x7f5da41a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 -- 172.21.15.32:0/970401697 >> 172.21.15.32:0/970401697 conn(0x7f5da419b460 msgr2=0x7f5da419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:12.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 -- 172.21.15.32:0/970401697 shutdown_connections 2021-04-09T18:29:12.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.354 7f5da91b6700 1 -- 172.21.15.32:0/970401697 wait complete. 2021-04-09T18:29:12.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:29:12.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.517 7f10c5d07700 1 Processor -- start 2021-04-09T18:29:12.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.517 7f10c5d07700 1 -- start start 2021-04-09T18:29:12.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10c5d07700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f10c00cffb0 0x7f10c00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10c5d07700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f10c00d7810 0x7f10c00d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10c5d07700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00d7290 con 0x7f10c00d8230 2021-04-09T18:29:12.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10c5d07700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c00d91f0 con 0x7f10c00cffb0 2021-04-09T18:29:12.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10c5d07700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c010dff0 con 0x7f10c00d7810 2021-04-09T18:29:12.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10bf7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f10c00cffb0 0x7f10c00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10bf7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f10c00cffb0 0x7f10c00d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54556/0 (socket says 172.21.15.32:54556) 2021-04-09T18:29:12.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10bf7fe700 1 -- 172.21.15.32:0/2470167121 learned_addr learned my addr 172.21.15.32:0/2470167121 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:12.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10beffd700 1 --2- 172.21.15.32:0/2470167121 >> v2:172.21.15.86:3300/0 conn(0x7f10c00d7810 0x7f10c00d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10bf7fe700 1 -- 172.21.15.32:0/2470167121 >> v2:172.21.15.86:3300/0 conn(0x7f10c00d7810 msgr2=0x7f10c00d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:12.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10bf7fe700 1 --2- 172.21.15.32:0/2470167121 >> v2:172.21.15.86:3300/0 conn(0x7f10c00d7810 0x7f10c00d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10bf7fe700 1 -- 172.21.15.32:0/2470167121 >> v1:172.21.15.86:6789/0 conn(0x7f10c00d8230 legacy=0x7f10c00d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10bf7fe700 1 -- 172.21.15.32:0/2470167121 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c010e210 con 0x7f10c00cffb0 2021-04-09T18:29:12.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10beffd700 1 --2- 172.21.15.32:0/2470167121 >> v2:172.21.15.86:3300/0 conn(0x7f10c00d7810 0x7f10c00d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:12.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10bf7fe700 1 --2- 172.21.15.32:0/2470167121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c00cffb0 0x7f10c00d2450 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7f10b0006700 tx=0x7f10b0009b80).ready entity=mon.0 client_cookie=98b72e0561828648 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:12.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10be7fc700 1 -- 172.21.15.32:0/2470167121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10b000c0c0 con 0x7f10c00cffb0 2021-04-09T18:29:12.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10be7fc700 1 -- 172.21.15.32:0/2470167121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b000c440 con 0x7f10c00cffb0 2021-04-09T18:29:12.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10be7fc700 1 -- 172.21.15.32:0/2470167121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10b000c920 con 0x7f10c00cffb0 2021-04-09T18:29:12.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10c5d07700 1 -- 172.21.15.32:0/2470167121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c00cffb0 msgr2=0x7f10c00d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.518 7f10c5d07700 1 --2- 172.21.15.32:0/2470167121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c00cffb0 0x7f10c00d2450 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7f10b0006700 tx=0x7f10b0009b80).stop 2021-04-09T18:29:12.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.519 7f10c5d07700 1 -- 172.21.15.32:0/2470167121 shutdown_connections 2021-04-09T18:29:12.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.519 7f10c5d07700 1 --2- 172.21.15.32:0/2470167121 >> v2:172.21.15.86:3300/0 conn(0x7f10c00d7810 0x7f10c00d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.519 7f10c5d07700 1 --2- 172.21.15.32:0/2470167121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c00cffb0 0x7f10c00d2450 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.519 7f10c5d07700 1 -- 172.21.15.32:0/2470167121 >> 172.21.15.32:0/2470167121 conn(0x7f10c00ce5d0 msgr2=0x7f10c00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:12.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.519 7f10c5d07700 1 -- 172.21.15.32:0/2470167121 shutdown_connections 2021-04-09T18:29:12.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.519 7f10c5d07700 1 -- 172.21.15.32:0/2470167121 wait complete. 2021-04-09T18:29:12.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.539 7f10c5d07700 1 Processor -- start 2021-04-09T18:29:12.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.539 7f10c5d07700 1 -- start start 2021-04-09T18:29:12.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10c5d07700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10c008a620 0x7f10c008b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10c5d07700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10c008b6d0 0x7f10c0180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10c5d07700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c0180c50 0x7f10c0181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10c5d07700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c008e660 con 0x7f10c0180c50 2021-04-09T18:29:12.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10c5d07700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c008ab30 con 0x7f10c008b6d0 2021-04-09T18:29:12.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10c5d07700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c008bad0 con 0x7f10c008a620 2021-04-09T18:29:12.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c0180c50 0x7f10c0181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c0180c50 0x7f10c0181050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54566/0 (socket says 172.21.15.32:54566) 2021-04-09T18:29:12.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10bffff700 1 -- 172.21.15.32:0/2205732187 learned_addr learned my addr 172.21.15.32:0/2205732187 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:12.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10beffd700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10c008b6d0 0x7f10c0180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10bf7fe700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10c008a620 0x7f10c008b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10bffff700 1 -- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10c008a620 msgr2=0x7f10c008b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10bffff700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10c008a620 0x7f10c008b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10bffff700 1 -- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10c008b6d0 msgr2=0x7f10c0180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:12.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10bffff700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10c008b6d0 0x7f10c0180630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:12.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.540 7f10bffff700 1 -- 172.21.15.32:0/2205732187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c008e1f0 con 0x7f10c0180c50 2021-04-09T18:29:12.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.541 7f10bffff700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c0180c50 0x7f10c0181050 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7f10b8009940 tx=0x7f10b800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:12.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.541 7f10b7fff700 1 -- 172.21.15.32:0/2205732187 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10b8000f40 con 0x7f10c0180c50 2021-04-09T18:29:12.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.541 7f10b7fff700 1 -- 172.21.15.32:0/2205732187 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b800c2f0 con 0x7f10c0180c50 2021-04-09T18:29:12.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.541 7f10b7fff700 1 -- 172.21.15.32:0/2205732187 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10b8000f40 con 0x7f10c0180c50 2021-04-09T18:29:12.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.541 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c008e1f0 con 0x7f10c0180c50 2021-04-09T18:29:12.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.541 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c008e1f0 con 0x7f10c0180c50 2021-04-09T18:29:12.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.541 7f10b7fff700 1 -- 172.21.15.32:0/2205732187 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f10b8001220 con 0x7f10c0180c50 2021-04-09T18:29:12.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.541 7f10b7fff700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f10a4018930 0x7f10a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:12.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.541 7f10b7fff700 1 -- 172.21.15.32:0/2205732187 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f10b8032e20 con 0x7f10c0180c50 2021-04-09T18:29:12.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.542 7f10bf7fe700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f10a4018930 0x7f10a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:12.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.542 7f10bf7fe700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f10a4018930 0x7f10a401ae10 crc :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:12.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.562 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c01b7180 con 0x7f10c0180c50 2021-04-09T18:29:12.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.565 7f10b7fff700 1 -- 172.21.15.32:0/2205732187 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f10b800be30 con 0x7f10c0180c50 2021-04-09T18:29:12.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.759 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f10c00d7290 con 0x7f10c0180c50 2021-04-09T18:29:13.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.921 7f10b7fff700 1 -- 172.21.15.32:0/2205732187 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v305) v1 ==== 104+0+0 (secure 0 0 0) 0x7f10b8001220 con 0x7f10c0180c50 2021-04-09T18:29:13.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f10a4018930 msgr2=0x7f10a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:13.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f10a4018930 0x7f10a401ae10 crc :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c0180c50 msgr2=0x7f10c0181050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:13.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c0180c50 0x7f10c0181050 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7f10b8009940 tx=0x7f10b800bcf0).stop 2021-04-09T18:29:13.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 shutdown_connections 2021-04-09T18:29:13.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f10a4018930 0x7f10a401ae10 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10c0180c50 0x7f10c0181050 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10c008b6d0 0x7f10c0180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 --2- 172.21.15.32:0/2205732187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10c008a620 0x7f10c008b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.925 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 >> 172.21.15.32:0/2205732187 conn(0x7f10c0189800 msgr2=0x7f10c01b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:13.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.926 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 shutdown_connections 2021-04-09T18:29:13.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:12.926 7f10c5d07700 1 -- 172.21.15.32:0/2205732187 wait complete. 2021-04-09T18:29:13.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:29:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.096 7f130af48700 1 Processor -- start 2021-04-09T18:29:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.096 7f130af48700 1 -- start start 2021-04-09T18:29:13.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f130af48700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f130410d050 0x7f13040cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:13.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f130af48700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f13040d84b0 0x7f130410da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:13.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f130af48700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130410d500 con 0x7f13040d0160 2021-04-09T18:29:13.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f130af48700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040d8d70 con 0x7f13040d84b0 2021-04-09T18:29:13.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f130af48700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040d9260 con 0x7f130410d050 2021-04-09T18:29:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f1309565700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f13040d84b0 0x7f130410da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f1309565700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f13040d84b0 0x7f130410da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54570/0 (socket says 172.21.15.32:54570) 2021-04-09T18:29:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f1309565700 1 -- 172.21.15.32:0/1635716264 learned_addr learned my addr 172.21.15.32:0/1635716264 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:13.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f1308d64700 1 --2- 172.21.15.32:0/1635716264 >> v2:172.21.15.86:3300/0 conn(0x7f130410d050 0x7f13040cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f1309565700 1 -- 172.21.15.32:0/1635716264 >> v2:172.21.15.86:3300/0 conn(0x7f130410d050 msgr2=0x7f13040cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f1309565700 1 --2- 172.21.15.32:0/1635716264 >> v2:172.21.15.86:3300/0 conn(0x7f130410d050 0x7f13040cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f1309565700 1 -- 172.21.15.32:0/1635716264 >> v1:172.21.15.86:6789/0 conn(0x7f13040d0160 legacy=0x7f13040d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:13.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f1309565700 1 -- 172.21.15.32:0/1635716264 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130410e0a0 con 0x7f13040d84b0 2021-04-09T18:29:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.097 7f1309565700 1 --2- 172.21.15.32:0/1635716264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13040d84b0 0x7f130410da80 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 rx=0x7f12fc006b80 tx=0x7f12fc009b90).ready entity=mon.0 client_cookie=e46cb1ad265a9f73 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f13037fe700 1 -- 172.21.15.32:0/1635716264 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12fc0022e0 con 0x7f13040d84b0 2021-04-09T18:29:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f13037fe700 1 -- 172.21.15.32:0/1635716264 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc002620 con 0x7f13040d84b0 2021-04-09T18:29:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f13037fe700 1 -- 172.21.15.32:0/1635716264 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12fc002b00 con 0x7f13040d84b0 2021-04-09T18:29:13.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f130af48700 1 -- 172.21.15.32:0/1635716264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13040d84b0 msgr2=0x7f130410da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f130af48700 1 --2- 172.21.15.32:0/1635716264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13040d84b0 0x7f130410da80 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 rx=0x7f12fc006b80 tx=0x7f12fc009b90).stop 2021-04-09T18:29:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f130af48700 1 -- 172.21.15.32:0/1635716264 shutdown_connections 2021-04-09T18:29:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f130af48700 1 --2- 172.21.15.32:0/1635716264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13040d84b0 0x7f130410da80 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f130af48700 1 --2- 172.21.15.32:0/1635716264 >> v2:172.21.15.86:3300/0 conn(0x7f130410d050 0x7f13040cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f130af48700 1 -- 172.21.15.32:0/1635716264 >> 172.21.15.32:0/1635716264 conn(0x7f13040ce620 msgr2=0x7f13040d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f130af48700 1 -- 172.21.15.32:0/1635716264 shutdown_connections 2021-04-09T18:29:13.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.098 7f130af48700 1 -- 172.21.15.32:0/1635716264 wait complete. 2021-04-09T18:29:13.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f130af48700 1 Processor -- start 2021-04-09T18:29:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f130af48700 1 -- start start 2021-04-09T18:29:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f130af48700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130409b2f0 0x7f130409b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f130af48700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130409a8b0 0x7f130409e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:13.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f130af48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130409e9c0 0x7f130409edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:13.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f130af48700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040bde80 con 0x7f130409e9c0 2021-04-09T18:29:13.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f130af48700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130409ad70 con 0x7f130409a8b0 2021-04-09T18:29:13.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f130af48700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130409bd10 con 0x7f130409b2f0 2021-04-09T18:29:13.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f1309565700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130409e9c0 0x7f130409edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:13.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f1309565700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130409e9c0 0x7f130409edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54578/0 (socket says 172.21.15.32:54578) 2021-04-09T18:29:13.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.119 7f1309565700 1 -- 172.21.15.32:0/741517938 learned_addr learned my addr 172.21.15.32:0/741517938 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:13.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f1303fff700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130409a8b0 0x7f130409e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:13.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f1308d64700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130409b2f0 0x7f130409b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:13.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f1309565700 1 -- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130409b2f0 msgr2=0x7f130409b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:13.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f1309565700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130409b2f0 0x7f130409b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f1309565700 1 -- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130409a8b0 msgr2=0x7f130409e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:13.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f1309565700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130409a8b0 0x7f130409e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f1309565700 1 -- 172.21.15.32:0/741517938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130409fbd0 con 0x7f130409e9c0 2021-04-09T18:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f1309565700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130409e9c0 0x7f130409edc0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f12fc009640 tx=0x7f12fc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f13017fa700 1 -- 172.21.15.32:0/741517938 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12fc009100 con 0x7f130409e9c0 2021-04-09T18:29:13.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f13017fa700 1 -- 172.21.15.32:0/741517938 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc00bd50 con 0x7f130409e9c0 2021-04-09T18:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f130af48700 1 -- 172.21.15.32:0/741517938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130409fbd0 con 0x7f130409e9c0 2021-04-09T18:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f13017fa700 1 -- 172.21.15.32:0/741517938 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12fc006430 con 0x7f130409e9c0 2021-04-09T18:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.120 7f130af48700 1 -- 172.21.15.32:0/741517938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130409fbd0 con 0x7f130409e9c0 2021-04-09T18:29:13.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.121 7f13017fa700 1 -- 172.21.15.32:0/741517938 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f12fc006240 con 0x7f130409e9c0 2021-04-09T18:29:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.121 7f13017fa700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e80188d0 0x7f12e801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.121 7f13017fa700 1 -- 172.21.15.32:0/741517938 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f12fc032a90 con 0x7f130409e9c0 2021-04-09T18:29:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.121 7f1308d64700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e80188d0 0x7f12e801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:13.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.121 7f1308d64700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e80188d0 0x7f12e801add0 crc :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:13.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.141 7f130af48700 1 -- 172.21.15.32:0/741517938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13040bde80 con 0x7f130409e9c0 2021-04-09T18:29:13.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.144 7f13017fa700 1 -- 172.21.15.32:0/741517938 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f12fc006030 con 0x7f130409e9c0 2021-04-09T18:29:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.328 7f130af48700 1 -- 172.21.15.32:0/741517938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f13040bde80 con 0x7f130409e9c0 2021-04-09T18:29:14.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.923 7f13017fa700 1 -- 172.21.15.32:0/741517938 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v158) v1 ==== 185+0+0 (secure 0 0 0) 0x7f12fc006240 con 0x7f130409e9c0 2021-04-09T18:29:14.026 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:29:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 -- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e80188d0 msgr2=0x7f12e801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e80188d0 0x7f12e801add0 crc :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 -- 172.21.15.32:0/741517938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130409e9c0 msgr2=0x7f130409edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:14.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130409e9c0 0x7f130409edc0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f12fc009640 tx=0x7f12fc00b960).stop 2021-04-09T18:29:14.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 -- 172.21.15.32:0/741517938 shutdown_connections 2021-04-09T18:29:14.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f12e80188d0 0x7f12e801add0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130409e9c0 0x7f130409edc0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130409a8b0 0x7f130409e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 --2- 172.21.15.32:0/741517938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130409b2f0 0x7f130409b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 -- 172.21.15.32:0/741517938 >> 172.21.15.32:0/741517938 conn(0x7f13040ae6f0 msgr2=0x7f1304097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:14.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 -- 172.21.15.32:0/741517938 shutdown_connections 2021-04-09T18:29:14.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:13.946 7f130af48700 1 -- 172.21.15.32:0/741517938 wait complete. 2021-04-09T18:29:14.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:29:14.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.126 7f01369db700 1 Processor -- start 2021-04-09T18:29:14.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.126 7f01369db700 1 -- start start 2021-04-09T18:29:14.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f01369db700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f013010d050 0x7f01300cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:14.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f01369db700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01300d0160 0x7f01300d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:14.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f01369db700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013010d500 con 0x7f01300d82e0 2021-04-09T18:29:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f01369db700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01300d9260 con 0x7f013010d050 2021-04-09T18:29:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f01369db700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01300d86e0 con 0x7f01300d0160 2021-04-09T18:29:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f012ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f013010d050 0x7f01300cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f012ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f013010d050 0x7f01300cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54582/0 (socket says 172.21.15.32:54582) 2021-04-09T18:29:14.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f012ffff700 1 -- 172.21.15.32:0/292181706 learned_addr learned my addr 172.21.15.32:0/292181706 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f0127fff700 1 --2- 172.21.15.32:0/292181706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01300d0160 0x7f01300d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f012ffff700 1 -- 172.21.15.32:0/292181706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01300d0160 msgr2=0x7f01300d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f012ffff700 1 --2- 172.21.15.32:0/292181706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01300d0160 0x7f01300d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f012ffff700 1 -- 172.21.15.32:0/292181706 >> v1:172.21.15.86:6789/0 conn(0x7f01300d82e0 legacy=0x7f013010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f012ffff700 1 -- 172.21.15.32:0/292181706 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013010e270 con 0x7f013010d050 2021-04-09T18:29:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f012ffff700 1 --2- 172.21.15.32:0/292181706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f013010d050 0x7f01300cfb40 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7f0120006700 tx=0x7f0120009b80).ready entity=mon.0 client_cookie=73d58de7c35a7c20 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.127 7f012f7fe700 1 -- 172.21.15.32:0/292181706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f012000c0c0 con 0x7f013010d050 2021-04-09T18:29:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f012f7fe700 1 -- 172.21.15.32:0/292181706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012000c440 con 0x7f013010d050 2021-04-09T18:29:14.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f012f7fe700 1 -- 172.21.15.32:0/292181706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f012000c920 con 0x7f013010d050 2021-04-09T18:29:14.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f01369db700 1 -- 172.21.15.32:0/292181706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f013010d050 msgr2=0x7f01300cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:14.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f01369db700 1 --2- 172.21.15.32:0/292181706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f013010d050 0x7f01300cfb40 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7f0120006700 tx=0x7f0120009b80).stop 2021-04-09T18:29:14.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f01369db700 1 -- 172.21.15.32:0/292181706 shutdown_connections 2021-04-09T18:29:14.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f01369db700 1 --2- 172.21.15.32:0/292181706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01300d0160 0x7f01300d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f01369db700 1 --2- 172.21.15.32:0/292181706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f013010d050 0x7f01300cfb40 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f01369db700 1 -- 172.21.15.32:0/292181706 >> 172.21.15.32:0/292181706 conn(0x7f01300ce620 msgr2=0x7f01300d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:14.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f01369db700 1 -- 172.21.15.32:0/292181706 shutdown_connections 2021-04-09T18:29:14.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.128 7f01369db700 1 -- 172.21.15.32:0/292181706 wait complete. 2021-04-09T18:29:14.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f01369db700 1 Processor -- start 2021-04-09T18:29:14.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f01369db700 1 -- start start 2021-04-09T18:29:14.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f01369db700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f013009a890 0x7f01300a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:14.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f01369db700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f013009ac90 0x7f01300a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:14.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f01369db700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01300c2220 0x7f01300c2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:14.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f01369db700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300b86e0 con 0x7f01300c2220 2021-04-09T18:29:14.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f01369db700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300a2920 con 0x7f013009a890 2021-04-09T18:29:14.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f01369db700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01300c2c40 con 0x7f013009ac90 2021-04-09T18:29:14.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f0134ff8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01300c2220 0x7f01300c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:14.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f0134ff8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01300c2220 0x7f01300c2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54592/0 (socket says 172.21.15.32:54592) 2021-04-09T18:29:14.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f0134ff8700 1 -- 172.21.15.32:0/1219754696 learned_addr learned my addr 172.21.15.32:0/1219754696 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:14.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f012ffff700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f013009a890 0x7f01300a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:14.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.149 7f0127fff700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f013009ac90 0x7f01300a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:14.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f0134ff8700 1 -- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f013009ac90 msgr2=0x7f01300a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:14.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f0134ff8700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f013009ac90 0x7f01300a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f0134ff8700 1 -- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f013009a890 msgr2=0x7f01300a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:14.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f0134ff8700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f013009a890 0x7f01300a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:14.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f0134ff8700 1 -- 172.21.15.32:0/1219754696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300c2e20 con 0x7f01300c2220 2021-04-09T18:29:14.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f0134ff8700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01300c2220 0x7f01300c2620 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7f012800b8e0 tx=0x7f012800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:14.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f012d7fa700 1 -- 172.21.15.32:0/1219754696 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0128000ce0 con 0x7f01300c2220 2021-04-09T18:29:14.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f012d7fa700 1 -- 172.21.15.32:0/1219754696 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012800c2f0 con 0x7f01300c2220 2021-04-09T18:29:14.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f01369db700 1 -- 172.21.15.32:0/1219754696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01300c30c0 con 0x7f01300c2220 2021-04-09T18:29:14.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f012d7fa700 1 -- 172.21.15.32:0/1219754696 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f012800c7d0 con 0x7f01300c2220 2021-04-09T18:29:14.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.150 7f01369db700 1 -- 172.21.15.32:0/1219754696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01300c2e20 con 0x7f01300c2220 2021-04-09T18:29:14.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.151 7f012d7fa700 1 -- 172.21.15.32:0/1219754696 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0128001220 con 0x7f01300c2220 2021-04-09T18:29:14.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.151 7f012d7fa700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01140188c0 0x7f011401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:14.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.151 7f012d7fa700 1 -- 172.21.15.32:0/1219754696 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 5311+0+0 (secure 0 0 0) 0x7f0128032c80 con 0x7f01300c2220 2021-04-09T18:29:14.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.151 7f012ffff700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01140188c0 0x7f011401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:14.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.151 7f012ffff700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01140188c0 0x7f011401adc0 crc :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:14.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.171 7f01369db700 1 -- 172.21.15.32:0/1219754696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01301b7180 con 0x7f01300c2220 2021-04-09T18:29:14.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.174 7f012d7fa700 1 -- 172.21.15.32:0/1219754696 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f012800bdc0 con 0x7f01300c2220 2021-04-09T18:29:14.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.360 7f01369db700 1 -- 172.21.15.32:0/1219754696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f013010d500 con 0x7f01300c2220 2021-04-09T18:29:15.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.928 7f012d7fa700 1 -- 172.21.15.32:0/1219754696 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v159) v1 ==== 173+0+0 (secure 0 0 0) 0x7f012801fad0 con 0x7f01300c2220 2021-04-09T18:29:15.058 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:29:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 -- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01140188c0 msgr2=0x7f011401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01140188c0 0x7f011401adc0 crc :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 -- 172.21.15.32:0/1219754696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01300c2220 msgr2=0x7f01300c2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01300c2220 0x7f01300c2620 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7f012800b8e0 tx=0x7f012800bc80).stop 2021-04-09T18:29:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 -- 172.21.15.32:0/1219754696 shutdown_connections 2021-04-09T18:29:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f01140188c0 0x7f011401adc0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01300c2220 0x7f01300c2620 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f013009ac90 0x7f01300a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 --2- 172.21.15.32:0/1219754696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f013009a890 0x7f01300a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 -- 172.21.15.32:0/1219754696 >> 172.21.15.32:0/1219754696 conn(0x7f01300b6b70 msgr2=0x7f01300b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:15.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 -- 172.21.15.32:0/1219754696 shutdown_connections 2021-04-09T18:29:15.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:14.977 7f01369db700 1 -- 172.21.15.32:0/1219754696 wait complete. 2021-04-09T18:29:15.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:29:15.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.138 7fbc48fef700 1 Processor -- start 2021-04-09T18:29:15.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.138 7fbc48fef700 1 -- start start 2021-04-09T18:29:15.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.139 7fbc48fef700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc440cffb0 0x7fbc440d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.139 7fbc48fef700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbc440d7810 0x7fbc440d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.139 7fbc48fef700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc440d8230 0x7fbc440d5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.139 7fbc48fef700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc440d91f0 con 0x7fbc440cffb0 2021-04-09T18:29:15.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.139 7fbc48fef700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc440d7290 con 0x7fbc440d7810 2021-04-09T18:29:15.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.139 7fbc48fef700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc440d62a0 con 0x7fbc440d8230 2021-04-09T18:29:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.139 7fbc42d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc440cffb0 0x7fbc440d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.139 7fbc42d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc440cffb0 0x7fbc440d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54596/0 (socket says 172.21.15.32:54596) 2021-04-09T18:29:15.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.139 7fbc42d9d700 1 -- 172.21.15.32:0/522472431 learned_addr learned my addr 172.21.15.32:0/522472431 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc4259c700 1 --2- 172.21.15.32:0/522472431 >> v2:172.21.15.86:3300/0 conn(0x7fbc440d7810 0x7fbc440d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc4359e700 1 --2- 172.21.15.32:0/522472431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc440d8230 0x7fbc440d5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc42d9d700 1 -- 172.21.15.32:0/522472431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc440d8230 msgr2=0x7fbc440d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:15.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc42d9d700 1 --2- 172.21.15.32:0/522472431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc440d8230 0x7fbc440d5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc42d9d700 1 -- 172.21.15.32:0/522472431 >> v2:172.21.15.86:3300/0 conn(0x7fbc440d7810 msgr2=0x7fbc440d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:15.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc42d9d700 1 --2- 172.21.15.32:0/522472431 >> v2:172.21.15.86:3300/0 conn(0x7fbc440d7810 0x7fbc440d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc42d9d700 1 -- 172.21.15.32:0/522472431 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440d6490 con 0x7fbc440cffb0 2021-04-09T18:29:15.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc4259c700 1 --2- 172.21.15.32:0/522472431 >> v2:172.21.15.86:3300/0 conn(0x7fbc440d7810 0x7fbc440d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:15.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc42d9d700 1 --2- 172.21.15.32:0/522472431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc440cffb0 0x7fbc440d2450 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fbc34006700 tx=0x7fbc34009b80).ready entity=mon.0 client_cookie=40372c84cbbbf444 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:15.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc41d9b700 1 -- 172.21.15.32:0/522472431 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc3400c0c0 con 0x7fbc440cffb0 2021-04-09T18:29:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc41d9b700 1 -- 172.21.15.32:0/522472431 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3400c440 con 0x7fbc440cffb0 2021-04-09T18:29:15.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc41d9b700 1 -- 172.21.15.32:0/522472431 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc3400c920 con 0x7fbc440cffb0 2021-04-09T18:29:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc48fef700 1 -- 172.21.15.32:0/522472431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc440cffb0 msgr2=0x7fbc440d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.140 7fbc48fef700 1 --2- 172.21.15.32:0/522472431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc440cffb0 0x7fbc440d2450 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fbc34006700 tx=0x7fbc34009b80).stop 2021-04-09T18:29:15.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.141 7fbc48fef700 1 -- 172.21.15.32:0/522472431 shutdown_connections 2021-04-09T18:29:15.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.141 7fbc48fef700 1 --2- 172.21.15.32:0/522472431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc440d8230 0x7fbc440d5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.141 7fbc48fef700 1 --2- 172.21.15.32:0/522472431 >> v2:172.21.15.86:3300/0 conn(0x7fbc440d7810 0x7fbc440d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.141 7fbc48fef700 1 --2- 172.21.15.32:0/522472431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc440cffb0 0x7fbc440d2450 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.141 7fbc48fef700 1 -- 172.21.15.32:0/522472431 >> 172.21.15.32:0/522472431 conn(0x7fbc440ce5d0 msgr2=0x7fbc440cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:15.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.141 7fbc48fef700 1 -- 172.21.15.32:0/522472431 shutdown_connections 2021-04-09T18:29:15.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.141 7fbc48fef700 1 -- 172.21.15.32:0/522472431 wait complete. 2021-04-09T18:29:15.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.161 7fbc48fef700 1 Processor -- start 2021-04-09T18:29:15.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.161 7fbc48fef700 1 -- start start 2021-04-09T18:29:15.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc48fef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc441b8380 0x7fbc441b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc48fef700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc4408c7b0 0x7fbc4408cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc48fef700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4408d1d0 0x7fbc441805b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc48fef700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc44181a30 con 0x7fbc4408d1d0 2021-04-09T18:29:15.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc48fef700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4408c230 con 0x7fbc441b8380 2021-04-09T18:29:15.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc48fef700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc441b8da0 con 0x7fbc4408c7b0 2021-04-09T18:29:15.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4408d1d0 0x7fbc441805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4408d1d0 0x7fbc441805b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54604/0 (socket says 172.21.15.32:54604) 2021-04-09T18:29:15.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4359e700 1 -- 172.21.15.32:0/2199139257 learned_addr learned my addr 172.21.15.32:0/2199139257 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:15.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc42d9d700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc441b8380 0x7fbc441b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4259c700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc4408c7b0 0x7fbc4408cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4359e700 1 -- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc4408c7b0 msgr2=0x7fbc4408cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4359e700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc4408c7b0 0x7fbc4408cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4359e700 1 -- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc441b8380 msgr2=0x7fbc441b8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4359e700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc441b8380 0x7fbc441b8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4359e700 1 -- 172.21.15.32:0/2199139257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc441b8f90 con 0x7fbc4408d1d0 2021-04-09T18:29:15.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4259c700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc4408c7b0 0x7fbc4408cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:15.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc4359e700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4408d1d0 0x7fbc441805b0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fbc3c0099a0 tx=0x7fbc3c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:15.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.162 7fbc3b7fe700 1 -- 172.21.15.32:0/2199139257 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc3c00be10 con 0x7fbc4408d1d0 2021-04-09T18:29:15.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.163 7fbc3b7fe700 1 -- 172.21.15.32:0/2199139257 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c003030 con 0x7fbc4408d1d0 2021-04-09T18:29:15.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.163 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc441b8f90 con 0x7fbc4408d1d0 2021-04-09T18:29:15.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.163 7fbc3b7fe700 1 -- 172.21.15.32:0/2199139257 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc3c0013e0 con 0x7fbc4408d1d0 2021-04-09T18:29:15.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.163 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc441b8f90 con 0x7fbc4408d1d0 2021-04-09T18:29:15.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.163 7fbc3b7fe700 1 -- 172.21.15.32:0/2199139257 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbc3c00a000 con 0x7fbc4408d1d0 2021-04-09T18:29:15.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.163 7fbc3b7fe700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc20018930 0x7fbc2001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.163 7fbc3b7fe700 1 -- 172.21.15.32:0/2199139257 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fbc3c032b20 con 0x7fbc4408d1d0 2021-04-09T18:29:15.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.163 7fbc42d9d700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc20018930 0x7fbc2001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.164 7fbc42d9d700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc20018930 0x7fbc2001adc0 crc :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:15.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.183 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc441b7180 con 0x7fbc4408d1d0 2021-04-09T18:29:15.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.187 7fbc3b7fe700 1 -- 172.21.15.32:0/2199139257 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbc3c0069c0 con 0x7fbc4408d1d0 2021-04-09T18:29:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.397 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbc4408c230 con 0x7fbc4408d1d0 2021-04-09T18:29:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.400 7fbc3b7fe700 1 -- 172.21.15.32:0/2199139257 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbc3c003010 con 0x7fbc4408d1d0 2021-04-09T18:29:15.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc20018930 msgr2=0x7fbc2001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc20018930 0x7fbc2001adc0 crc :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4408d1d0 msgr2=0x7fbc441805b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4408d1d0 0x7fbc441805b0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fbc3c0099a0 tx=0x7fbc3c00bcd0).stop 2021-04-09T18:29:15.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 shutdown_connections 2021-04-09T18:29:15.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc20018930 0x7fbc2001adc0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc4408d1d0 0x7fbc441805b0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc4408c7b0 0x7fbc4408cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 --2- 172.21.15.32:0/2199139257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc441b8380 0x7fbc441b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 >> 172.21.15.32:0/2199139257 conn(0x7fbc44189800 msgr2=0x7fbc441b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:15.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 shutdown_connections 2021-04-09T18:29:15.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.406 7fbc48fef700 1 -- 172.21.15.32:0/2199139257 wait complete. 2021-04-09T18:29:15.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestFailover.test_simple' 2021-04-09T18:29:15.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.565 7f9c5e231700 1 Processor -- start 2021-04-09T18:29:15.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.565 7f9c5e231700 1 -- start start 2021-04-09T18:29:15.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c5e231700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c580cffa0 0x7f9c580d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c5e231700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c580d7800 0x7f9c580d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c5e231700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9c580d8220 0x7f9c580d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c5e231700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c580d91e0 con 0x7f9c580cffa0 2021-04-09T18:29:15.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c5e231700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c580d7280 con 0x7f9c580d8220 2021-04-09T18:29:15.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c5e231700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c580d6290 con 0x7f9c580d7800 2021-04-09T18:29:15.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c57fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c580cffa0 0x7f9c580d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c57fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9c580cffa0 0x7f9c580d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54610/0 (socket says 172.21.15.32:54610) 2021-04-09T18:29:15.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c57fff700 1 -- 172.21.15.32:0/2630009852 learned_addr learned my addr 172.21.15.32:0/2630009852 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:15.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c4ffff700 1 --2- 172.21.15.32:0/2630009852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c580d7800 0x7f9c580d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c5c84e700 1 --2- 172.21.15.32:0/2630009852 >> v2:172.21.15.86:3300/0 conn(0x7f9c580d8220 0x7f9c580d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c57fff700 1 -- 172.21.15.32:0/2630009852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c580d7800 msgr2=0x7f9c580d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:15.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c57fff700 1 --2- 172.21.15.32:0/2630009852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c580d7800 0x7f9c580d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c57fff700 1 -- 172.21.15.32:0/2630009852 >> v2:172.21.15.86:3300/0 conn(0x7f9c580d8220 msgr2=0x7f9c580d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:15.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c57fff700 1 --2- 172.21.15.32:0/2630009852 >> v2:172.21.15.86:3300/0 conn(0x7f9c580d8220 0x7f9c580d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c57fff700 1 -- 172.21.15.32:0/2630009852 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c580d64b0 con 0x7f9c580cffa0 2021-04-09T18:29:15.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c57fff700 1 --2- 172.21.15.32:0/2630009852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c580cffa0 0x7f9c580d2440 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f9c48006700 tx=0x7f9c48009b00).ready entity=mon.0 client_cookie=ebda6a80f1a6905a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:15.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c577fe700 1 -- 172.21.15.32:0/2630009852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c4800c0c0 con 0x7f9c580cffa0 2021-04-09T18:29:15.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c577fe700 1 -- 172.21.15.32:0/2630009852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4800c400 con 0x7f9c580cffa0 2021-04-09T18:29:15.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c577fe700 1 -- 172.21.15.32:0/2630009852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c4800c8e0 con 0x7f9c580cffa0 2021-04-09T18:29:15.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c5e231700 1 -- 172.21.15.32:0/2630009852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c580cffa0 msgr2=0x7f9c580d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.566 7f9c5e231700 1 --2- 172.21.15.32:0/2630009852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c580cffa0 0x7f9c580d2440 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f9c48006700 tx=0x7f9c48009b00).stop 2021-04-09T18:29:15.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.567 7f9c5e231700 1 -- 172.21.15.32:0/2630009852 shutdown_connections 2021-04-09T18:29:15.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.567 7f9c5e231700 1 --2- 172.21.15.32:0/2630009852 >> v2:172.21.15.86:3300/0 conn(0x7f9c580d8220 0x7f9c580d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.567 7f9c5e231700 1 --2- 172.21.15.32:0/2630009852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c580d7800 0x7f9c580d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.567 7f9c5e231700 1 --2- 172.21.15.32:0/2630009852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c580cffa0 0x7f9c580d2440 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.567 7f9c5e231700 1 -- 172.21.15.32:0/2630009852 >> 172.21.15.32:0/2630009852 conn(0x7f9c580ce5c0 msgr2=0x7f9c580cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:15.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.567 7f9c5e231700 1 -- 172.21.15.32:0/2630009852 shutdown_connections 2021-04-09T18:29:15.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.567 7f9c5e231700 1 -- 172.21.15.32:0/2630009852 wait complete. 2021-04-09T18:29:15.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.587 7f9c5e231700 1 Processor -- start 2021-04-09T18:29:15.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.587 7f9c5e231700 1 -- start start 2021-04-09T18:29:15.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5e231700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c581acff0 0x7f9c581847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5e231700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c58183fc0 0x7f9c581a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5e231700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c58186ff0 0x7f9c581873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5e231700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c58188c30 con 0x7f9c58186ff0 2021-04-09T18:29:15.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5e231700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c581843c0 con 0x7f9c581acff0 2021-04-09T18:29:15.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5e231700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c581a5c20 con 0x7f9c58183fc0 2021-04-09T18:29:15.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5c84e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c58186ff0 0x7f9c581873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5c84e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c58186ff0 0x7f9c581873f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54620/0 (socket says 172.21.15.32:54620) 2021-04-09T18:29:15.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5c84e700 1 -- 172.21.15.32:0/135162025 learned_addr learned my addr 172.21.15.32:0/135162025 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:15.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c4ffff700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c58183fc0 0x7f9c581a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c57fff700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c581acff0 0x7f9c581847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5c84e700 1 -- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c58183fc0 msgr2=0x7f9c581a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5c84e700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c58183fc0 0x7f9c581a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5c84e700 1 -- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c581acff0 msgr2=0x7f9c581847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:15.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5c84e700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c581acff0 0x7f9c581847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:15.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.588 7f9c5c84e700 1 -- 172.21.15.32:0/135162025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c58187a10 con 0x7f9c58186ff0 2021-04-09T18:29:15.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.589 7f9c5c84e700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c58186ff0 0x7f9c581873f0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f9c5000a940 tx=0x7f9c5000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:15.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.589 7f9c557fa700 1 -- 172.21.15.32:0/135162025 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c50001400 con 0x7f9c58186ff0 2021-04-09T18:29:15.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.589 7f9c557fa700 1 -- 172.21.15.32:0/135162025 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5000ae20 con 0x7f9c58186ff0 2021-04-09T18:29:15.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.589 7f9c557fa700 1 -- 172.21.15.32:0/135162025 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c50001930 con 0x7f9c58186ff0 2021-04-09T18:29:15.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.589 7f9c5e231700 1 -- 172.21.15.32:0/135162025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c58187a10 con 0x7f9c58186ff0 2021-04-09T18:29:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.589 7f9c5e231700 1 -- 172.21.15.32:0/135162025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c58187a10 con 0x7f9c58186ff0 2021-04-09T18:29:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.589 7f9c557fa700 1 -- 172.21.15.32:0/135162025 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9c500011f0 con 0x7f9c58186ff0 2021-04-09T18:29:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.590 7f9c557fa700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c3c0188d0 0x7f9c3c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.590 7f9c557fa700 1 -- 172.21.15.32:0/135162025 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f9c50032af0 con 0x7f9c58186ff0 2021-04-09T18:29:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.590 7f9c57fff700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c3c0188d0 0x7f9c3c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.590 7f9c57fff700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c3c0188d0 0x7f9c3c01add0 crc :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:15.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.609 7f9c5e231700 1 -- 172.21.15.32:0/135162025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c58182dc0 con 0x7f9c58186ff0 2021-04-09T18:29:15.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.613 7f9c557fa700 1 -- 172.21.15.32:0/135162025 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9c500066f0 con 0x7f9c58186ff0 2021-04-09T18:29:15.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.801 7f9c5e231700 1 -- 172.21.15.32:0/135162025 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestFailover.test_simple"]} v 0) v1 -- 0x7f9c580d7280 con 0x7f9c58186ff0 2021-04-09T18:29:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.929 7f9c557fa700 1 -- 172.21.15.32:0/135162025 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestFailover.test_simple"]}]=0 v733) v1 ==== 130+0+0 (secure 0 0 0) 0x7f9c500011f0 con 0x7f9c58186ff0 2021-04-09T18:29:16.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 -- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c3c0188d0 msgr2=0x7f9c3c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:16.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c3c0188d0 0x7f9c3c01add0 crc :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:16.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 -- 172.21.15.32:0/135162025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c58186ff0 msgr2=0x7f9c581873f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:16.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c58186ff0 0x7f9c581873f0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f9c5000a940 tx=0x7f9c5000ace0).stop 2021-04-09T18:29:16.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 -- 172.21.15.32:0/135162025 shutdown_connections 2021-04-09T18:29:16.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9c3c0188d0 0x7f9c3c01add0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c58186ff0 0x7f9c581873f0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c58183fc0 0x7f9c581a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 --2- 172.21.15.32:0/135162025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c581acff0 0x7f9c581847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 -- 172.21.15.32:0/135162025 >> 172.21.15.32:0/135162025 conn(0x7f9c5809abb0 msgr2=0x7f9c580a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 -- 172.21.15.32:0/135162025 shutdown_connections 2021-04-09T18:29:16.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:15.967 7f9c5e231700 1 -- 172.21.15.32:0/135162025 wait complete. 2021-04-09T18:29:16.063 INFO:tasks.cephfs_test_runner:test_simple (tasks.cephfs.test_failover.TestFailover) ... ok 2021-04-09T18:29:16.063 INFO:tasks.cephfs_test_runner:Starting test: test_standby_count_wanted (tasks.cephfs.test_failover.TestFailover) 2021-04-09T18:29:16.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestFailover.test_standby_count_wanted' 2021-04-09T18:29:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.128 7fe029a00700 1 Processor -- start 2021-04-09T18:29:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.128 7fe029a00700 1 -- start start 2021-04-09T18:29:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.129 7fe029a00700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0240d7ca0 0x7fe0240d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.129 7fe029a00700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02410d480 con 0x7fe02410d010 2021-04-09T18:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.129 7fe029a00700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0240d86c0 con 0x7fe02410da00 2021-04-09T18:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.129 7fe029a00700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0240d9220 con 0x7fe0240d7ca0 2021-04-09T18:29:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.129 7fe022ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0240d7ca0 0x7fe0240d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.129 7fe0237fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe02410d010 0x7fe0240d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35982/0 (socket says 172.21.15.32:35982) 2021-04-09T18:29:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.129 7fe0237fe700 1 -- 172.21.15.32:0/1091666073 learned_addr learned my addr 172.21.15.32:0/1091666073 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:16.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.129 7fe022ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0240d7ca0 0x7fe0240d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54624/0 (socket says 172.21.15.32:54624) 2021-04-09T18:29:16.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe022ffd700 1 -- 172.21.15.32:0/1091666073 >> v1:172.21.15.86:6789/0 conn(0x7fe02410da00 legacy=0x7fe02410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:16.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe022ffd700 1 -- 172.21.15.32:0/1091666073 >> v1:172.21.15.32:6789/0 conn(0x7fe02410d010 legacy=0x7fe0240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe022ffd700 1 -- 172.21.15.32:0/1091666073 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0240d1f70 con 0x7fe0240d7ca0 2021-04-09T18:29:16.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe0227fc700 1 -- 172.21.15.32:0/1091666073 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398359699 0 0) 0x7fe014000ce0 con 0x7fe02410d010 2021-04-09T18:29:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe022ffd700 1 --2- 172.21.15.32:0/1091666073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0240d7ca0 0x7fe0240d80a0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7fe0180069d0 tx=0x7fe018009dc0).ready entity=mon.0 client_cookie=b582f9ac7da238bd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe0227fc700 1 -- 172.21.15.32:0/1091666073 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01800b510 con 0x7fe0240d7ca0 2021-04-09T18:29:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe0227fc700 1 -- 172.21.15.32:0/1091666073 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe018006ac0 con 0x7fe0240d7ca0 2021-04-09T18:29:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe0227fc700 1 -- 172.21.15.32:0/1091666073 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01800ba80 con 0x7fe0240d7ca0 2021-04-09T18:29:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe029a00700 1 -- 172.21.15.32:0/1091666073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0240d7ca0 msgr2=0x7fe0240d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.130 7fe029a00700 1 --2- 172.21.15.32:0/1091666073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0240d7ca0 0x7fe0240d80a0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7fe0180069d0 tx=0x7fe018009dc0).stop 2021-04-09T18:29:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.131 7fe029a00700 1 -- 172.21.15.32:0/1091666073 shutdown_connections 2021-04-09T18:29:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.131 7fe029a00700 1 --2- 172.21.15.32:0/1091666073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0240d7ca0 0x7fe0240d80a0 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.131 7fe029a00700 1 -- 172.21.15.32:0/1091666073 >> 172.21.15.32:0/1091666073 conn(0x7fe0240ce5f0 msgr2=0x7fe0240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.131 7fe029a00700 1 -- 172.21.15.32:0/1091666073 shutdown_connections 2021-04-09T18:29:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.131 7fe029a00700 1 -- 172.21.15.32:0/1091666073 wait complete. 2021-04-09T18:29:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.151 7fe029a00700 1 Processor -- start 2021-04-09T18:29:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.151 7fe029a00700 1 -- start start 2021-04-09T18:29:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe029a00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0241a69f0 0x7fe024188d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe029a00700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe024189a70 0x7fe024189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe029a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0241a9330 0x7fe0241a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe029a00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02418a510 con 0x7fe0241a9330 2021-04-09T18:29:16.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe029a00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0241a6f20 con 0x7fe0241a69f0 2021-04-09T18:29:16.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe029a00700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0241a9d50 con 0x7fe024189a70 2021-04-09T18:29:16.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe023fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0241a9330 0x7fe0241a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:16.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe023fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0241a9330 0x7fe0241a9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54632/0 (socket says 172.21.15.32:54632) 2021-04-09T18:29:16.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe023fff700 1 -- 172.21.15.32:0/3284177697 learned_addr learned my addr 172.21.15.32:0/3284177697 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:16.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe0237fe700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0241a69f0 0x7fe024188d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:16.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe022ffd700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe024189a70 0x7fe024189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:16.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe023fff700 1 -- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe024189a70 msgr2=0x7fe024189e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:16.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe023fff700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe024189a70 0x7fe024189e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:16.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe023fff700 1 -- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0241a69f0 msgr2=0x7fe024188d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe023fff700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0241a69f0 0x7fe024188d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe023fff700 1 -- 172.21.15.32:0/3284177697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0241884b0 con 0x7fe0241a9330 2021-04-09T18:29:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe0237fe700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0241a69f0 0x7fe024188d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.152 7fe023fff700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0241a9330 0x7fe0241a9730 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7fe01c006ed0 tx=0x7fe01c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:16.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.153 7fe00bfff700 1 -- 172.21.15.32:0/3284177697 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01c00bd50 con 0x7fe0241a9330 2021-04-09T18:29:16.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.153 7fe00bfff700 1 -- 172.21.15.32:0/3284177697 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01c006020 con 0x7fe0241a9330 2021-04-09T18:29:16.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.153 7fe00bfff700 1 -- 172.21.15.32:0/3284177697 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01c0054b0 con 0x7fe0241a9330 2021-04-09T18:29:16.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.153 7fe029a00700 1 -- 172.21.15.32:0/3284177697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0241884b0 con 0x7fe0241a9330 2021-04-09T18:29:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.153 7fe029a00700 1 -- 172.21.15.32:0/3284177697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0241884b0 con 0x7fe0241a9330 2021-04-09T18:29:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.153 7fe00bfff700 1 -- 172.21.15.32:0/3284177697 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe01c006200 con 0x7fe0241a9330 2021-04-09T18:29:16.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.153 7fe00bfff700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe004018930 0x7fe00401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:16.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.153 7fe00bfff700 1 -- 172.21.15.32:0/3284177697 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fe01c032790 con 0x7fe0241a9330 2021-04-09T18:29:16.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.155 7fe0237fe700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe004018930 0x7fe00401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:16.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.155 7fe0237fe700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe004018930 0x7fe00401ae10 crc :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:16.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.174 7fe029a00700 1 -- 172.21.15.32:0/3284177697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02410d480 con 0x7fe0241a9330 2021-04-09T18:29:16.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.178 7fe00bfff700 1 -- 172.21.15.32:0/3284177697 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe01c006230 con 0x7fe0241a9330 2021-04-09T18:29:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.366 7fe029a00700 1 -- 172.21.15.32:0/3284177697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestFailover.test_standby_count_wanted"]} v 0) v1 -- 0x7fe02410d480 con 0x7fe0241a9330 2021-04-09T18:29:17.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.932 7fe00bfff700 1 -- 172.21.15.32:0/3284177697 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestFailover.test_standby_count_wanted"]}]=0 v734) v1 ==== 147+0+0 (secure 0 0 0) 0x7fe01c006020 con 0x7fe0241a9330 2021-04-09T18:29:17.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 -- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe004018930 msgr2=0x7fe00401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe004018930 0x7fe00401ae10 crc :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 -- 172.21.15.32:0/3284177697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0241a9330 msgr2=0x7fe0241a9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0241a9330 0x7fe0241a9730 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7fe01c006ed0 tx=0x7fe01c00b960).stop 2021-04-09T18:29:17.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 -- 172.21.15.32:0/3284177697 shutdown_connections 2021-04-09T18:29:17.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe004018930 0x7fe00401ae10 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0241a9330 0x7fe0241a9730 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe024189a70 0x7fe024189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 --2- 172.21.15.32:0/3284177697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0241a69f0 0x7fe024188d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.983 7fe029a00700 1 -- 172.21.15.32:0/3284177697 >> 172.21.15.32:0/3284177697 conn(0x7fe0241aa1b0 msgr2=0x7fe0241ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:17.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.984 7fe029a00700 1 -- 172.21.15.32:0/3284177697 shutdown_connections 2021-04-09T18:29:17.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:16.984 7fe029a00700 1 -- 172.21.15.32:0/3284177697 wait complete. 2021-04-09T18:29:17.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:29:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.155 7f1182a28700 1 Processor -- start 2021-04-09T18:29:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1182a28700 1 -- start start 2021-04-09T18:29:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1182a28700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0cffc0 0x7f117c0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1182a28700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f117c0d7820 0x7f117c0d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1182a28700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c0d72a0 con 0x7f117c0d5c90 2021-04-09T18:29:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1182a28700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c0d85a0 con 0x7f117c0d7820 2021-04-09T18:29:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1182a28700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0d9200 con 0x7f117c0cffc0 2021-04-09T18:29:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1180844700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0cffc0 0x7f117c0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f117bfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f117c0d7820 0x7f117c0d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1181045700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f117c0d5c90 0x7f117c0d6090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:33920/0 (socket says 172.21.15.32:33920) 2021-04-09T18:29:17.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1180844700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0cffc0 0x7f117c0d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:33630/0 (socket says 172.21.15.32:33630) 2021-04-09T18:29:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.156 7f1181045700 1 -- 172.21.15.32:0/1271607521 learned_addr learned my addr 172.21.15.32:0/1271607521 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f1180844700 1 -- 172.21.15.32:0/1271607521 >> v2:172.21.15.86:3300/0 conn(0x7f117c0d7820 msgr2=0x7f117c0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f1180844700 1 --2- 172.21.15.32:0/1271607521 >> v2:172.21.15.86:3300/0 conn(0x7f117c0d7820 0x7f117c0d7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f1180844700 1 -- 172.21.15.32:0/1271607521 >> v1:172.21.15.86:6789/0 conn(0x7f117c0d5c90 legacy=0x7f117c0d6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f1180844700 1 -- 172.21.15.32:0/1271607521 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0d8240 con 0x7f117c0cffc0 2021-04-09T18:29:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f117bfff700 1 --2- 172.21.15.32:0/1271607521 >> v2:172.21.15.86:3300/0 conn(0x7f117c0d7820 0x7f117c0d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f1180844700 1 --2- 172.21.15.32:0/1271607521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0cffc0 0x7f117c0d2460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f1170006650 tx=0x7f1170009a80).ready entity=mon.2 client_cookie=2b5cda7d0ca6d59 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f117b7fe700 1 -- 172.21.15.32:0/1271607521 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11700009b0 con 0x7f117c0cffc0 2021-04-09T18:29:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f117b7fe700 1 -- 172.21.15.32:0/1271607521 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117000c0c0 con 0x7f117c0cffc0 2021-04-09T18:29:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f117b7fe700 1 -- 172.21.15.32:0/1271607521 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f117000c520 con 0x7f117c0cffc0 2021-04-09T18:29:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f1182a28700 1 -- 172.21.15.32:0/1271607521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0cffc0 msgr2=0x7f117c0d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.157 7f1182a28700 1 --2- 172.21.15.32:0/1271607521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0cffc0 0x7f117c0d2460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f1170006650 tx=0x7f1170009a80).stop 2021-04-09T18:29:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.158 7f1182a28700 1 -- 172.21.15.32:0/1271607521 shutdown_connections 2021-04-09T18:29:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.158 7f1182a28700 1 --2- 172.21.15.32:0/1271607521 >> v2:172.21.15.86:3300/0 conn(0x7f117c0d7820 0x7f117c0d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.158 7f1182a28700 1 --2- 172.21.15.32:0/1271607521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0cffc0 0x7f117c0d2460 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.158 7f1182a28700 1 -- 172.21.15.32:0/1271607521 >> 172.21.15.32:0/1271607521 conn(0x7f117c0ce5e0 msgr2=0x7f117c0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.158 7f1182a28700 1 -- 172.21.15.32:0/1271607521 shutdown_connections 2021-04-09T18:29:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.158 7f1182a28700 1 -- 172.21.15.32:0/1271607521 wait complete. 2021-04-09T18:29:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.178 7f1182a28700 1 Processor -- start 2021-04-09T18:29:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.178 7f1182a28700 1 -- start start 2021-04-09T18:29:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1182a28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f117c1b82c0 0x7f117c0b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1182a28700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c1b86c0 0x7f117c0b9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1182a28700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0b9f60 0x7f117c0c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1182a28700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0b93c0 con 0x7f117c1b82c0 2021-04-09T18:29:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1182a28700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0ba360 con 0x7f117c1b86c0 2021-04-09T18:29:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1182a28700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c1b8db0 con 0x7f117c0b9f60 2021-04-09T18:29:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1180844700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f117c1b82c0 0x7f117c0b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1180844700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f117c1b82c0 0x7f117c0b89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54640/0 (socket says 172.21.15.32:54640) 2021-04-09T18:29:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1180844700 1 -- 172.21.15.32:0/2789954786 learned_addr learned my addr 172.21.15.32:0/2789954786 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f117bfff700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c1b86c0 0x7f117c0b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1181045700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0b9f60 0x7f117c0c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1180844700 1 -- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0b9f60 msgr2=0x7f117c0c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1180844700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0b9f60 0x7f117c0c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1180844700 1 -- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c1b86c0 msgr2=0x7f117c0b9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1180844700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c1b86c0 0x7f117c0b9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1180844700 1 -- 172.21.15.32:0/2789954786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0b8fc0 con 0x7f117c1b82c0 2021-04-09T18:29:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1181045700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0b9f60 0x7f117c0c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1180844700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f117c1b82c0 0x7f117c0b89a0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f1170009630 tx=0x7f117000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f11797fa700 1 -- 172.21.15.32:0/2789954786 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11700093a0 con 0x7f117c1b82c0 2021-04-09T18:29:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f11797fa700 1 -- 172.21.15.32:0/2789954786 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117000bd70 con 0x7f117c1b82c0 2021-04-09T18:29:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f1182a28700 1 -- 172.21.15.32:0/2789954786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c0b8fc0 con 0x7f117c1b82c0 2021-04-09T18:29:17.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.179 7f11797fa700 1 -- 172.21.15.32:0/2789954786 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1170005560 con 0x7f117c1b82c0 2021-04-09T18:29:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.180 7f1182a28700 1 -- 172.21.15.32:0/2789954786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c0b8fc0 con 0x7f117c1b82c0 2021-04-09T18:29:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.180 7f11797fa700 1 -- 172.21.15.32:0/2789954786 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1170005700 con 0x7f117c1b82c0 2021-04-09T18:29:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.180 7f11797fa700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f115c018930 0x7f115c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.180 7f11797fa700 1 -- 172.21.15.32:0/2789954786 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f1170032750 con 0x7f117c1b82c0 2021-04-09T18:29:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.181 7f117bfff700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f115c018930 0x7f115c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.181 7f117bfff700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f115c018930 0x7f115c01ae10 crc :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:17.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.200 7f1182a28700 1 -- 172.21.15.32:0/2789954786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c0c3e60 con 0x7f117c1b82c0 2021-04-09T18:29:17.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.204 7f11797fa700 1 -- 172.21.15.32:0/2789954786 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f11700054f0 con 0x7f117c1b82c0 2021-04-09T18:29:17.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.417 7f1182a28700 1 -- 172.21.15.32:0/2789954786 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f117c0d72a0 con 0x7f117c1b82c0 2021-04-09T18:29:17.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.420 7f11797fa700 1 -- 172.21.15.32:0/2789954786 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1170005700 con 0x7f117c1b82c0 2021-04-09T18:29:17.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 -- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f115c018930 msgr2=0x7f115c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f115c018930 0x7f115c01ae10 crc :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 -- 172.21.15.32:0/2789954786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f117c1b82c0 msgr2=0x7f117c0b89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f117c1b82c0 0x7f117c0b89a0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f1170009630 tx=0x7f117000b980).stop 2021-04-09T18:29:17.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 -- 172.21.15.32:0/2789954786 shutdown_connections 2021-04-09T18:29:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f115c018930 0x7f115c01ae10 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c0b9f60 0x7f117c0c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c1b86c0 0x7f117c0b9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 --2- 172.21.15.32:0/2789954786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f117c1b82c0 0x7f117c0b89a0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 -- 172.21.15.32:0/2789954786 >> 172.21.15.32:0/2789954786 conn(0x7f117c0a06a0 msgr2=0x7f117c08fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 -- 172.21.15.32:0/2789954786 shutdown_connections 2021-04-09T18:29:17.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.426 7f1182a28700 1 -- 172.21.15.32:0/2789954786 wait complete. 2021-04-09T18:29:17.520 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:29:17.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:29:17.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:29:17.637 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:29:17.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:17.638 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:29:17.639 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:29:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:29:17.684 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:29:17.684 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:29:17.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:17.685 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:29:17.686 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:29:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:29:17.732 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:29:17.732 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:29:17.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:29:17.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.797 7f33685ac700 1 Processor -- start 2021-04-09T18:29:17.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.797 7f33685ac700 1 -- start start 2021-04-09T18:29:17.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f33685ac700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336010d010 0x7f33600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f33685ac700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33600d1da0 0x7f33600d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f33685ac700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336010d480 con 0x7f33600d82c0 2021-04-09T18:29:17.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f33685ac700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336010e780 con 0x7f33600d1da0 2021-04-09T18:29:17.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f33685ac700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33600d8d30 con 0x7f336010d010 2021-04-09T18:29:17.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f3366bc9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f33600d82c0 0x7f336010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36008/0 (socket says 172.21.15.32:36008) 2021-04-09T18:29:17.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f3366bc9700 1 -- 172.21.15.32:0/2609022908 learned_addr learned my addr 172.21.15.32:0/2609022908 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:17.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f3365bc7700 1 --2- 172.21.15.32:0/2609022908 >> v2:172.21.15.86:3300/0 conn(0x7f33600d1da0 0x7f33600d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f33663c8700 1 --2- 172.21.15.32:0/2609022908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336010d010 0x7f33600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f33653c6700 1 -- 172.21.15.32:0/2609022908 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882212524 0 0) 0x7f3358000d30 con 0x7f33600d82c0 2021-04-09T18:29:17.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.798 7f33653c6700 1 -- 172.21.15.32:0/2609022908 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f334c002fe0 con 0x7f33600d82c0 2021-04-09T18:29:17.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f33653c6700 1 -- 172.21.15.32:0/2609022908 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 940237548 0 0) 0x7f3358001270 con 0x7f33600d82c0 2021-04-09T18:29:17.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f3365bc7700 1 -- 172.21.15.32:0/2609022908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336010d010 msgr2=0x7f33600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:17.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f3365bc7700 1 --2- 172.21.15.32:0/2609022908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336010d010 0x7f33600d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f3365bc7700 1 -- 172.21.15.32:0/2609022908 >> v1:172.21.15.32:6789/0 conn(0x7f33600d82c0 legacy=0x7f336010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f3365bc7700 1 -- 172.21.15.32:0/2609022908 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33600d9220 con 0x7f33600d1da0 2021-04-09T18:29:17.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f33663c8700 1 --2- 172.21.15.32:0/2609022908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336010d010 0x7f33600d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:17.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f3365bc7700 1 --2- 172.21.15.32:0/2609022908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33600d1da0 0x7f33600d7ca0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7f3354006650 tx=0x7f3354009a80).ready entity=mon.1 client_cookie=fc27eaad50a47370 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:17.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f33653c6700 1 -- 172.21.15.32:0/2609022908 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33540009b0 con 0x7f33600d1da0 2021-04-09T18:29:17.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f33653c6700 1 -- 172.21.15.32:0/2609022908 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335400c0c0 con 0x7f33600d1da0 2021-04-09T18:29:17.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f33653c6700 1 -- 172.21.15.32:0/2609022908 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f335400c510 con 0x7f33600d1da0 2021-04-09T18:29:17.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f33685ac700 1 -- 172.21.15.32:0/2609022908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33600d1da0 msgr2=0x7f33600d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.799 7f33685ac700 1 --2- 172.21.15.32:0/2609022908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33600d1da0 0x7f33600d7ca0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7f3354006650 tx=0x7f3354009a80).stop 2021-04-09T18:29:17.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.800 7f33685ac700 1 -- 172.21.15.32:0/2609022908 shutdown_connections 2021-04-09T18:29:17.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.800 7f33685ac700 1 --2- 172.21.15.32:0/2609022908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33600d1da0 0x7f33600d7ca0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.800 7f33685ac700 1 --2- 172.21.15.32:0/2609022908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336010d010 0x7f33600d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.800 7f33685ac700 1 -- 172.21.15.32:0/2609022908 >> 172.21.15.32:0/2609022908 conn(0x7f33600ce5f0 msgr2=0x7f33600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:17.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.800 7f33685ac700 1 -- 172.21.15.32:0/2609022908 shutdown_connections 2021-04-09T18:29:17.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.800 7f33685ac700 1 -- 172.21.15.32:0/2609022908 wait complete. 2021-04-09T18:29:17.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.820 7f33685ac700 1 Processor -- start 2021-04-09T18:29:17.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.820 7f33685ac700 1 -- start start 2021-04-09T18:29:17.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33685ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33600b5360 0x7f33601a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33685ac700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336018e140 0x7f33601aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33685ac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33601ad040 0x7f336008fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33685ac700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33601a5360 con 0x7f33600b5360 2021-04-09T18:29:17.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33685ac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600902c0 con 0x7f33601ad040 2021-04-09T18:29:17.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33685ac700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33600904e0 con 0x7f336018e140 2021-04-09T18:29:17.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33663c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33600b5360 0x7f33601a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33663c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33600b5360 0x7f33601a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54654/0 (socket says 172.21.15.32:54654) 2021-04-09T18:29:17.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33663c8700 1 -- 172.21.15.32:0/1603626055 learned_addr learned my addr 172.21.15.32:0/1603626055 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:17.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f3366bc9700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33601ad040 0x7f336008fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f3365bc7700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336018e140 0x7f33601aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33663c8700 1 -- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336018e140 msgr2=0x7f33601aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33663c8700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336018e140 0x7f33601aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33663c8700 1 -- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33601ad040 msgr2=0x7f336008fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:17.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33663c8700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33601ad040 0x7f336008fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:17.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33663c8700 1 -- 172.21.15.32:0/1603626055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360183c80 con 0x7f33600b5360 2021-04-09T18:29:17.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33663c8700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33600b5360 0x7f33601a4d40 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7f335c006f40 tx=0x7f335c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:17.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f3352ffd700 1 -- 172.21.15.32:0/1603626055 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f335c00bd80 con 0x7f33600b5360 2021-04-09T18:29:17.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f3352ffd700 1 -- 172.21.15.32:0/1603626055 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c000c90 con 0x7f33600b5360 2021-04-09T18:29:17.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f33685ac700 1 -- 172.21.15.32:0/1603626055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3360183c80 con 0x7f33600b5360 2021-04-09T18:29:17.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.821 7f3352ffd700 1 -- 172.21.15.32:0/1603626055 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f335c002220 con 0x7f33600b5360 2021-04-09T18:29:17.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.822 7f33685ac700 1 -- 172.21.15.32:0/1603626055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3360183c80 con 0x7f33600b5360 2021-04-09T18:29:17.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.823 7f3352ffd700 1 -- 172.21.15.32:0/1603626055 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f335c00a000 con 0x7f33600b5360 2021-04-09T18:29:17.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.823 7f3352ffd700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33480188c0 0x7f334801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:17.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.823 7f3352ffd700 1 -- 172.21.15.32:0/1603626055 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f335c032510 con 0x7f33600b5360 2021-04-09T18:29:17.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.823 7f3365bc7700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33480188c0 0x7f334801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:17.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.823 7f3365bc7700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33480188c0 0x7f334801adc0 crc :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:17.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.842 7f33685ac700 1 -- 172.21.15.32:0/1603626055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3360182d70 con 0x7f33600b5360 2021-04-09T18:29:17.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:17.846 7f3352ffd700 1 -- 172.21.15.32:0/1603626055 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f335c006b00 con 0x7f33600b5360 2021-04-09T18:29:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.035 7f33685ac700 1 -- 172.21.15.32:0/1603626055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f3360182d70 con 0x7f33600b5360 2021-04-09T18:29:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.036 7f3352ffd700 1 -- 172.21.15.32:0/1603626055 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v159) v1 ==== 81+0+25690 (secure 0 0 0) 0x7f335c00a000 con 0x7f33600b5360 2021-04-09T18:29:18.123 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:18.123 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:29:18.123 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 159, 2021-04-09T18:29:18.124 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:29:18.124 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:29:18.124 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:29:14.923052", 2021-04-09T18:29:18.124 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:29:18.125 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:29:18.125 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:29:18.125 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:29:18.125 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:29:18.126 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:29:18.126 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:29:18.127 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:29:18.127 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:29:18.127 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:29:18.128 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:29:18.128 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:29:18.129 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:29:18.129 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:29:18.129 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:29:18.129 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 27, 2021-04-09T18:29:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:29:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:29:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:29:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:29:18.131 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:29:18.131 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.131 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:29:18.131 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:29:18.131 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:29:18.132 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:29:18.132 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:29:18.132 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:29:18.132 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:29:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:29:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:29:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:29:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:29:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:29:18.134 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:29:18.134 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:29:18.134 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:29:18.134 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:29:18.134 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:29:18.135 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:29:18.135 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:29:18.135 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:29:18.135 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:29:18.136 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:29:18.136 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:29:18.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.136 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:29:18.136 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:29:18.137 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:29:18.137 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:29:18.137 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:29:18.137 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:29:18.138 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:29:18.138 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:29:18.138 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:29:18.138 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:29:18.139 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:29:18.139 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:29:18.139 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:29:18.139 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:29:18.140 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:29:18.140 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:29:18.140 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:29:18.140 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:29:18.140 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:29:18.141 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:29:18.141 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:29:18.141 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:29:18.141 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:29:18.142 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:29:18.142 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:29:18.142 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:29:18.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:29:18.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.143 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:29:18.143 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:29:18.143 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:29:18.143 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:29:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:29:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:29:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:29:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:29:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:29:18.145 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:29:18.145 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:29:18.145 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:29:18.145 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:29:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:29:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:29:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:29:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:29:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:29:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:18.150 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:18.150 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.151 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:29:18.151 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:18.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:29:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.153 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:18.153 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:29:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.155 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:29:18.155 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:18.155 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.156 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.156 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:18.156 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.156 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.157 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:29:18.157 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:18.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.158 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.158 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:29:18.158 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:18.158 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.159 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.159 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:18.159 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.160 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.160 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:29:18.160 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:18.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.161 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.161 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:29:18.161 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:29:18.161 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.162 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.162 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:29:18.162 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:29:18.162 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:29:18.163 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:29:18.163 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:18.163 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:18.163 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:29:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:29:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 148, 2021-04-09T18:29:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.168 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:29:18.168 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:18.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.168 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.169 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:29:18.169 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:18.169 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.169 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.170 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:18.170 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.170 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.171 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:29:18.171 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:18.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:29:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.174 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:29:18.174 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:18.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.175 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:29:18.175 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:18.175 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.175 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.176 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:18.176 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.177 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:29:18.177 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:18.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.177 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.178 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:29:18.178 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:29:18.178 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.178 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.179 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:29:18.179 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:29:18.179 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:29:18.179 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:29:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:29:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:29:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:29:18.185 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:18.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.185 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.185 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:29:18.186 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:18.186 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.186 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.186 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:18.187 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.187 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.187 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:29:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:29:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:29:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:29:18.192 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:18.192 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.192 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.192 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:18.193 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.193 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.193 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:29:18.194 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:18.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.194 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:29:18.195 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:29:18.195 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.195 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.196 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:29:18.196 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:29:18.196 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:29:18.196 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:29:18.197 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:18.197 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:18.197 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:18.197 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.198 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:29:18.198 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:29:18.198 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:18.199 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:18.199 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:18.199 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:18.199 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:18.199 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:29:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:18.201 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.201 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.201 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:29:18.202 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:18.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.202 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:29:18.203 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:18.203 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.203 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.204 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:18.204 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.204 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:29:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.206 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:29:18.206 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:18.206 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.206 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.207 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:18.207 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.207 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.208 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:29:18.208 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:18.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.209 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:29:18.209 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:18.209 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.210 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.210 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:18.210 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.210 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.211 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.211 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:29:18.211 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:18.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.212 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.212 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:29:18.212 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:29:18.212 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.213 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.213 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:29:18.213 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:29:18.213 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:29:18.214 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:29:18.214 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:18.214 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:18.214 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:18.214 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.215 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.215 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:29:18.215 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:29:18.216 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:18.216 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:18.216 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:18.216 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:18.216 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:18.217 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:18.217 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:18.217 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:18.217 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:18.217 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:18.218 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:18.218 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.218 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.218 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.219 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:29:18.219 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:18.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:29:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.221 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:18.221 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.221 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.222 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:29:18.222 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:18.222 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.222 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.223 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:29:18.223 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:18.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.223 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.224 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:18.224 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.224 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.225 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:29:18.225 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:18.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.225 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.226 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:29:18.226 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:18.226 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.226 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.227 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:18.227 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.227 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.227 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.228 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:29:18.228 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:18.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.228 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.229 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:29:18.229 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:29:18.229 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.229 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.230 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:29:18.230 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:29:18.230 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:29:18.230 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:29:18.231 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:18.231 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:18.231 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:18.231 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.232 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:29:18.232 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:29:18.232 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:18.232 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:18.233 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:18.233 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:18.233 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:18.233 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:18.234 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:18.234 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:18.234 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:18.234 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:18.234 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:18.235 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.235 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.235 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.235 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:29:18.235 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:18.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.236 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.236 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:29:18.236 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:18.237 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.237 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.237 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:18.238 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.238 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:29:18.238 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:18.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.239 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:29:18.239 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:18.240 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.240 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.240 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:18.241 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.241 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:29:18.241 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:18.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.242 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.242 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:29:18.243 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:18.243 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.243 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.243 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:18.244 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.244 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.244 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:29:18.245 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:18.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.245 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.245 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:29:18.246 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:29:18.246 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.246 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.246 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:29:18.247 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:29:18.247 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:29:18.247 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:29:18.247 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:18.248 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:18.248 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:18.248 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.248 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.249 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:29:18.249 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:29:18.249 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:18.249 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:18.250 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:18.250 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:18.250 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:18.250 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:18.250 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:18.251 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:29:18.251 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:18.251 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:18.251 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:18.252 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.252 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.252 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.252 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:29:18.252 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:18.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.253 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:29:18.254 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:18.254 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.254 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.254 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:18.255 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.255 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.255 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.255 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:29:18.255 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:18.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.256 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.256 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:29:18.257 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:18.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.257 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.257 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:18.258 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.258 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.258 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:29:18.259 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:18.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.259 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.259 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:29:18.260 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:18.260 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.260 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.260 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:18.261 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.261 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.261 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.261 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:29:18.262 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:18.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.262 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.262 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:29:18.263 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:29:18.263 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.263 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.263 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:29:18.264 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:29:18.264 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:29:18.264 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:29:18.264 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:18.265 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:18.265 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:18.265 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.266 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:29:18.266 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:29:18.266 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:29:18.266 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:29:18.266 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:29:18.267 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:29:18.267 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:29:18.267 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:29:18.267 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:29:18.268 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 148, 2021-04-09T18:29:18.268 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:29:18.268 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:29:18.268 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:29:18.268 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.269 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.269 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.269 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:29:18.269 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:18.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.270 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:29:18.270 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:18.271 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.271 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.271 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:29:18.271 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.272 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.272 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:29:18.272 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:18.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.273 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.273 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:29:18.273 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:18.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.274 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.274 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:29:18.275 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.275 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.275 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:29:18.275 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:18.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.276 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:29:18.276 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:18.277 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.277 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.277 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:29:18.278 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:29:18.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:29:18.278 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:29:18.278 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:18.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.279 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:18.279 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:29:18.279 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:29:18.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.280 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.280 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:29:18.281 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:29:18.281 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:29:18.281 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:29:18.281 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:29:18.281 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:29:18.282 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:29:18.282 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:18.282 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.282 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:29:18.282 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:29:18.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.283 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:29:18.283 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:18.283 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:18.284 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:18.284 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:18.284 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:18.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.284 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.285 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:29:18.285 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:18.285 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:18.285 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:18.285 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:18.286 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:18.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.286 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:29:18.287 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:18.287 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:18.287 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:18.287 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:18.287 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:18.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.288 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.288 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:29:18.288 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:18.289 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:18.289 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:18.289 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:18.289 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:18.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.290 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:29:18.290 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:18.290 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:18.290 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:18.291 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:18.291 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:18.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.292 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:29:18.292 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:18.292 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:18.292 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:18.292 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:18.293 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:18.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.293 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:29:18.293 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:18.294 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:18.294 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:18.294 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:18.294 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:18.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:18.295 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:29:18.295 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:29:18.295 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:29:18.296 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:29:18.296 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:29:18.296 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:29:18.296 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.296 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:29:18.297 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:29:18.297 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:29:18.297 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:29:18.297 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:29:18.297 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:29:18.298 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/1399348025": "2021-04-10 18:29:11.487318", 2021-04-09T18:29:18.298 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/1939434763": "2021-04-10 18:28:57.593214", 2021-04-09T18:29:18.298 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6837/1399348025": "2021-04-10 18:29:11.487318", 2021-04-09T18:29:18.298 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/1939434763": "2021-04-10 18:28:57.593214" 2021-04-09T18:29:18.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.299 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:29:18.299 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:29:18.299 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:29:18.299 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:29:18.300 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:29:18.300 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:29:18.300 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:29:18.300 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:29:18.300 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:18.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:18.301 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:29:18.301 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:29:18.301 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:29:18.302 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:29:18.302 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:29:18.302 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:29:18.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 -- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33480188c0 msgr2=0x7f334801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33480188c0 0x7f334801adc0 crc :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 -- 172.21.15.32:0/1603626055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33600b5360 msgr2=0x7f33601a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33600b5360 0x7f33601a4d40 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7f335c006f40 tx=0x7f335c00b990).stop 2021-04-09T18:29:18.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 -- 172.21.15.32:0/1603626055 shutdown_connections 2021-04-09T18:29:18.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33480188c0 0x7f334801adc0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33601ad040 0x7f336008fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f336018e140 0x7f33601aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 --2- 172.21.15.32:0/1603626055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33600b5360 0x7f33601a4d40 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 -- 172.21.15.32:0/1603626055 >> 172.21.15.32:0/1603626055 conn(0x7f336019b460 msgr2=0x7f336019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:18.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 -- 172.21.15.32:0/1603626055 shutdown_connections 2021-04-09T18:29:18.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.041 7f33685ac700 1 -- 172.21.15.32:0/1603626055 wait complete. 2021-04-09T18:29:18.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.372 7fd052b2c700 1 Processor -- start 2021-04-09T18:29:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.372 7fd052b2c700 1 -- start start 2021-04-09T18:29:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd052b2c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd04c0d7ca0 0x7fd04c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd052b2c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd04c10da00 0x7fd04c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd052b2c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04c10d480 con 0x7fd04c10d010 2021-04-09T18:29:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd052b2c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c0d9220 con 0x7fd04c0d7ca0 2021-04-09T18:29:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd052b2c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c0d86c0 con 0x7fd04c10da00 2021-04-09T18:29:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd050948700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd04c10d010 0x7fd04c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36020/0 (socket says 172.21.15.32:36020) 2021-04-09T18:29:18.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd04bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd04c0d7ca0 0x7fd04c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd050948700 1 -- 172.21.15.32:0/3988796778 learned_addr learned my addr 172.21.15.32:0/3988796778 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:18.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd051149700 1 --2- 172.21.15.32:0/3988796778 >> v2:172.21.15.86:3300/0 conn(0x7fd04c10da00 0x7fd04c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd04b7fe700 1 -- 172.21.15.32:0/3988796778 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923564147 0 0) 0x7fd034000ce0 con 0x7fd04c10d010 2021-04-09T18:29:18.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd04b7fe700 1 -- 172.21.15.32:0/3988796778 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd038002fe0 con 0x7fd04c10d010 2021-04-09T18:29:18.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd04bfff700 1 -- 172.21.15.32:0/3988796778 >> v2:172.21.15.86:3300/0 conn(0x7fd04c10da00 msgr2=0x7fd04c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd04bfff700 1 --2- 172.21.15.32:0/3988796778 >> v2:172.21.15.86:3300/0 conn(0x7fd04c10da00 0x7fd04c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd04bfff700 1 -- 172.21.15.32:0/3988796778 >> v1:172.21.15.32:6789/0 conn(0x7fd04c10d010 legacy=0x7fd04c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.373 7fd04bfff700 1 -- 172.21.15.32:0/3988796778 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c0d1f70 con 0x7fd04c0d7ca0 2021-04-09T18:29:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd04bfff700 1 --2- 172.21.15.32:0/3988796778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c0d7ca0 0x7fd04c0d80a0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 rx=0x7fd03c006650 tx=0x7fd03c009a80).ready entity=mon.0 client_cookie=be346436ca6fed31 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd04b7fe700 1 -- 172.21.15.32:0/3988796778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd03c0009b0 con 0x7fd04c0d7ca0 2021-04-09T18:29:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd04b7fe700 1 -- 172.21.15.32:0/3988796778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c00c0c0 con 0x7fd04c0d7ca0 2021-04-09T18:29:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd04b7fe700 1 -- 172.21.15.32:0/3988796778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd03c00c510 con 0x7fd04c0d7ca0 2021-04-09T18:29:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd052b2c700 1 -- 172.21.15.32:0/3988796778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c0d7ca0 msgr2=0x7fd04c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd052b2c700 1 --2- 172.21.15.32:0/3988796778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c0d7ca0 0x7fd04c0d80a0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 rx=0x7fd03c006650 tx=0x7fd03c009a80).stop 2021-04-09T18:29:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd052b2c700 1 -- 172.21.15.32:0/3988796778 shutdown_connections 2021-04-09T18:29:18.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd052b2c700 1 --2- 172.21.15.32:0/3988796778 >> v2:172.21.15.86:3300/0 conn(0x7fd04c10da00 0x7fd04c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd052b2c700 1 --2- 172.21.15.32:0/3988796778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c0d7ca0 0x7fd04c0d80a0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.374 7fd052b2c700 1 -- 172.21.15.32:0/3988796778 >> 172.21.15.32:0/3988796778 conn(0x7fd04c0ce5f0 msgr2=0x7fd04c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:18.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.375 7fd052b2c700 1 -- 172.21.15.32:0/3988796778 shutdown_connections 2021-04-09T18:29:18.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.375 7fd052b2c700 1 -- 172.21.15.32:0/3988796778 wait complete. 2021-04-09T18:29:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.395 7fd052b2c700 1 Processor -- start 2021-04-09T18:29:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.395 7fd052b2c700 1 -- start start 2021-04-09T18:29:18.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.395 7fd052b2c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c1889c0 0x7fd04c09a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.395 7fd052b2c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd04c18e2c0 0x7fd04c1aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.395 7fd052b2c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd04c1aaad0 0x7fd04c0a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.395 7fd052b2c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c1aaed0 con 0x7fd04c1889c0 2021-04-09T18:29:18.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.395 7fd052b2c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c1ab0f0 con 0x7fd04c1aaad0 2021-04-09T18:29:18.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.395 7fd052b2c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0a3ed0 con 0x7fd04c18e2c0 2021-04-09T18:29:18.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd050948700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c1889c0 0x7fd04c09a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd050948700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c1889c0 0x7fd04c09a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54668/0 (socket says 172.21.15.32:54668) 2021-04-09T18:29:18.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd050948700 1 -- 172.21.15.32:0/3039376414 learned_addr learned my addr 172.21.15.32:0/3039376414 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:18.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd051149700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd04c1aaad0 0x7fd04c0a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd04bfff700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd04c18e2c0 0x7fd04c1aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd050948700 1 -- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd04c18e2c0 msgr2=0x7fd04c1aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd050948700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd04c18e2c0 0x7fd04c1aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd050948700 1 -- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd04c1aaad0 msgr2=0x7fd04c0a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd050948700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd04c1aaad0 0x7fd04c0a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd050948700 1 -- 172.21.15.32:0/3039376414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c0a40f0 con 0x7fd04c1889c0 2021-04-09T18:29:18.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd04bfff700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd04c18e2c0 0x7fd04c1aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:18.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd050948700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c1889c0 0x7fd04c09a3f0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7fd0340014b0 tx=0x7fd03400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd0497fa700 1 -- 172.21.15.32:0/3039376414 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd03400acc0 con 0x7fd04c1889c0 2021-04-09T18:29:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd0497fa700 1 -- 172.21.15.32:0/3039376414 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd034007030 con 0x7fd04c1889c0 2021-04-09T18:29:18.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c0a40f0 con 0x7fd04c1889c0 2021-04-09T18:29:18.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd0497fa700 1 -- 172.21.15.32:0/3039376414 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd034004620 con 0x7fd04c1889c0 2021-04-09T18:29:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.396 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c0a40f0 con 0x7fd04c1889c0 2021-04-09T18:29:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.397 7fd0497fa700 1 -- 172.21.15.32:0/3039376414 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd0340043a0 con 0x7fd04c1889c0 2021-04-09T18:29:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.397 7fd0497fa700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd030018930 0x7fd03001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.397 7fd0497fa700 1 -- 172.21.15.32:0/3039376414 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fd034032780 con 0x7fd04c1889c0 2021-04-09T18:29:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.397 7fd04bfff700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd030018930 0x7fd03001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.397 7fd04bfff700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd030018930 0x7fd03001ae10 crc :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:18.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.417 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c10d480 con 0x7fd04c1889c0 2021-04-09T18:29:18.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.420 7fd0497fa700 1 -- 172.21.15.32:0/3039376414 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd0340009b0 con 0x7fd04c1889c0 2021-04-09T18:29:18.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.610 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd04c10d480 con 0x7fd04c1889c0 2021-04-09T18:29:18.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.611 7fd0497fa700 1 -- 172.21.15.32:0/3039376414 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 306 v306) v1 ==== 95+0+1856 (secure 0 0 0) 0x7fd034007010 con 0x7fd04c1889c0 2021-04-09T18:29:18.695 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 306 2021-04-09T18:29:18.696 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:18.696 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":306,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/506460258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":506460258},{"type":"v1","addr":"172.21.15.32:6835","nonce":506460258}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":303},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3456810430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3456810430},{"type":"v1","addr":"172.21.15.86:6837","nonce":3456810430}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":306}],"filesystems":[]} 2021-04-09T18:29:18.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd030018930 msgr2=0x7fd03001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd030018930 0x7fd03001ae10 crc :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c1889c0 msgr2=0x7fd04c09a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c1889c0 0x7fd04c09a3f0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7fd0340014b0 tx=0x7fd03400a8d0).stop 2021-04-09T18:29:18.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 shutdown_connections 2021-04-09T18:29:18.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd030018930 0x7fd03001ae10 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd04c1aaad0 0x7fd04c0a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd04c18e2c0 0x7fd04c1aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 --2- 172.21.15.32:0/3039376414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd04c1889c0 0x7fd04c09a3f0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 >> 172.21.15.32:0/3039376414 conn(0x7fd04c1a0db0 msgr2=0x7fd04c1a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:18.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 shutdown_connections 2021-04-09T18:29:18.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.613 7fd052b2c700 1 -- 172.21.15.32:0/3039376414 wait complete. 2021-04-09T18:29:18.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.772 7fac0865b700 1 Processor -- start 2021-04-09T18:29:18.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.772 7fac0865b700 1 -- start start 2021-04-09T18:29:18.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac0865b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fac000d7ca0 0x7fac000d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac0865b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac0010da00 0x7fac0010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac0865b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0010d480 con 0x7fac0010d010 2021-04-09T18:29:18.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac0865b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac000d9220 con 0x7fac000d7ca0 2021-04-09T18:29:18.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac0865b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac00112100 con 0x7fac0010da00 2021-04-09T18:29:18.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac05c76700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fac000d7ca0 0x7fac000d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac05c76700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fac000d7ca0 0x7fac000d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54676/0 (socket says 172.21.15.32:54676) 2021-04-09T18:29:18.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac05c76700 1 -- 172.21.15.32:0/3299779918 learned_addr learned my addr 172.21.15.32:0/3299779918 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:18.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac06c78700 1 --2- 172.21.15.32:0/3299779918 >> v2:172.21.15.86:3300/0 conn(0x7fac0010da00 0x7fac0010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac05c76700 1 -- 172.21.15.32:0/3299779918 >> v2:172.21.15.86:3300/0 conn(0x7fac0010da00 msgr2=0x7fac0010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:18.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac05c76700 1 --2- 172.21.15.32:0/3299779918 >> v2:172.21.15.86:3300/0 conn(0x7fac0010da00 0x7fac0010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac05c76700 1 -- 172.21.15.32:0/3299779918 >> v1:172.21.15.86:6789/0 conn(0x7fac0010d010 legacy=0x7fac000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:18.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac05c76700 1 -- 172.21.15.32:0/3299779918 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac000d86c0 con 0x7fac000d7ca0 2021-04-09T18:29:18.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac05c76700 1 --2- 172.21.15.32:0/3299779918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac000d7ca0 0x7fac000d80a0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7fabf4006700 tx=0x7fabf4009b80).ready entity=mon.0 client_cookie=da94c35dcefd03df server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:18.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.773 7fac05475700 1 -- 172.21.15.32:0/3299779918 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabf400c0c0 con 0x7fac000d7ca0 2021-04-09T18:29:18.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac05475700 1 -- 172.21.15.32:0/3299779918 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf400c440 con 0x7fac000d7ca0 2021-04-09T18:29:18.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac05475700 1 -- 172.21.15.32:0/3299779918 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabf400c920 con 0x7fac000d7ca0 2021-04-09T18:29:18.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac0865b700 1 -- 172.21.15.32:0/3299779918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac000d7ca0 msgr2=0x7fac000d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac0865b700 1 --2- 172.21.15.32:0/3299779918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac000d7ca0 0x7fac000d80a0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7fabf4006700 tx=0x7fabf4009b80).stop 2021-04-09T18:29:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac0865b700 1 -- 172.21.15.32:0/3299779918 shutdown_connections 2021-04-09T18:29:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac0865b700 1 --2- 172.21.15.32:0/3299779918 >> v2:172.21.15.86:3300/0 conn(0x7fac0010da00 0x7fac0010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac0865b700 1 --2- 172.21.15.32:0/3299779918 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac000d7ca0 0x7fac000d80a0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac0865b700 1 -- 172.21.15.32:0/3299779918 >> 172.21.15.32:0/3299779918 conn(0x7fac000ce5f0 msgr2=0x7fac000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:18.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac0865b700 1 -- 172.21.15.32:0/3299779918 shutdown_connections 2021-04-09T18:29:18.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.774 7fac0865b700 1 -- 172.21.15.32:0/3299779918 wait complete. 2021-04-09T18:29:18.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac0865b700 1 Processor -- start 2021-04-09T18:29:18.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac0865b700 1 -- start start 2021-04-09T18:29:18.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac0865b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac00186480 0x7fac0008c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac0865b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac00186880 0x7fac0008c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac0865b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0008d680 0x7fac0008da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac0865b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0008bc60 con 0x7fac00186880 2021-04-09T18:29:18.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac0865b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0008e0a0 con 0x7fac0008d680 2021-04-09T18:29:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac0865b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac00179610 con 0x7fac00186480 2021-04-09T18:29:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac05c76700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac00186880 0x7fac0008c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac05c76700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac00186880 0x7fac0008c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54684/0 (socket says 172.21.15.32:54684) 2021-04-09T18:29:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.795 7fac05c76700 1 -- 172.21.15.32:0/1121572641 learned_addr learned my addr 172.21.15.32:0/1121572641 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:18.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac06c78700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0008d680 0x7fac0008da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac06477700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac00186480 0x7fac0008c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac05c76700 1 -- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac00186480 msgr2=0x7fac0008c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac05c76700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac00186480 0x7fac0008c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac05c76700 1 -- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0008d680 msgr2=0x7fac0008da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:18.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac05c76700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0008d680 0x7fac0008da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:18.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac05c76700 1 -- 172.21.15.32:0/1121572641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac00179830 con 0x7fac00186880 2021-04-09T18:29:18.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac06477700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac00186480 0x7fac0008c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:18.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac05c76700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac00186880 0x7fac0008c800 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7fabf400b650 tx=0x7fabf400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:18.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fabfeffd700 1 -- 172.21.15.32:0/1121572641 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabf40009b0 con 0x7fac00186880 2021-04-09T18:29:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fabfeffd700 1 -- 172.21.15.32:0/1121572641 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf4006570 con 0x7fac00186880 2021-04-09T18:29:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac0865b700 1 -- 172.21.15.32:0/1121572641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac00179830 con 0x7fac00186880 2021-04-09T18:29:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fabfeffd700 1 -- 172.21.15.32:0/1121572641 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabf4002030 con 0x7fac00186880 2021-04-09T18:29:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.796 7fac0865b700 1 -- 172.21.15.32:0/1121572641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac00179830 con 0x7fac00186880 2021-04-09T18:29:18.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.797 7fabfeffd700 1 -- 172.21.15.32:0/1121572641 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fabf4009f90 con 0x7fac00186880 2021-04-09T18:29:18.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.797 7fabfeffd700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe0018930 0x7fabe001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:18.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.797 7fabfeffd700 1 -- 172.21.15.32:0/1121572641 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fabf4032760 con 0x7fac00186880 2021-04-09T18:29:18.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.797 7fac06477700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe0018930 0x7fabe001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:18.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.797 7fac06477700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe0018930 0x7fabe001ae10 crc :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:18.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.817 7fac0865b700 1 -- 172.21.15.32:0/1121572641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0010d480 con 0x7fac00186880 2021-04-09T18:29:18.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:18.820 7fabfeffd700 1 -- 172.21.15.32:0/1121572641 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fabf4000c90 con 0x7fac00186880 2021-04-09T18:29:19.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.013 7fac0865b700 1 -- 172.21.15.32:0/1121572641 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac0010d480 con 0x7fac00186880 2021-04-09T18:29:19.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.013 7fabfeffd700 1 -- 172.21.15.32:0/1121572641 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 306 v306) v1 ==== 95+0+1856 (secure 0 0 0) 0x7fabf400bde0 con 0x7fac00186880 2021-04-09T18:29:19.098 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 306 2021-04-09T18:29:19.098 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:19.099 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":306,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9874,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2113468092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2113468092},{"type":"v1","addr":"172.21.15.86:6836","nonce":2113468092}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/506460258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":506460258},{"type":"v1","addr":"172.21.15.32:6835","nonce":506460258}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":303},{"gid":58133,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2714206945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2714206945},{"type":"v1","addr":"172.21.15.32:6834","nonce":2714206945}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":294},{"gid":58313,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3456810430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3456810430},{"type":"v1","addr":"172.21.15.86:6837","nonce":3456810430}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":306}],"filesystems":[]} 2021-04-09T18:29:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 -- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe0018930 msgr2=0x7fabe001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe0018930 0x7fabe001ae10 crc :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 -- 172.21.15.32:0/1121572641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac00186880 msgr2=0x7fac0008c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac00186880 0x7fac0008c800 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7fabf400b650 tx=0x7fabf400b9f0).stop 2021-04-09T18:29:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 -- 172.21.15.32:0/1121572641 shutdown_connections 2021-04-09T18:29:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe0018930 0x7fabe001ae10 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac00186880 0x7fac0008c800 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0008d680 0x7fac0008da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 --2- 172.21.15.32:0/1121572641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac00186480 0x7fac0008c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 -- 172.21.15.32:0/1121572641 >> 172.21.15.32:0/1121572641 conn(0x7fac00195e70 msgr2=0x7fac001a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:19.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 -- 172.21.15.32:0/1121572641 shutdown_connections 2021-04-09T18:29:19.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.016 7fac0865b700 1 -- 172.21.15.32:0/1121572641 wait complete. 2021-04-09T18:29:19.110 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:29:19.110 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:29:19.111 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:29:19.111 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:29:19.112 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:29:19.112 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:29:19.112 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:29:19.113 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:29:19.113 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:29:19.114 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:29:19.114 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:29:19.114 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:29:19.115 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:29:19.115 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:29:19.115 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:29:19.115 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:29:19.119 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:29:19.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:29:19.121 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:29:19.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:29:19.123 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:29:19.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:29:19.125 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:29:19.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:29:19.162 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:29:19.163 DEBUG:teuthology.parallel:result is None 2021-04-09T18:29:19.165 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:29:19.165 DEBUG:teuthology.parallel:result is None 2021-04-09T18:29:19.166 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:29:19.166 DEBUG:teuthology.parallel:result is None 2021-04-09T18:29:19.167 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:29:19.167 DEBUG:teuthology.parallel:result is None 2021-04-09T18:29:19.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:29:19.307 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:29:19.308 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:29:19.311 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:29:19.312 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:29:19.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.245 7f3fc8fc2700 1 Processor -- start 2021-04-09T18:29:19.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.246 7f3fc8fc2700 1 -- start start 2021-04-09T18:29:19.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.246 7f3fc8fc2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3fc410e7e0 0x7f3fc40d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.246 7f3fc8fc2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc40d7290 con 0x7f3fc40cffb0 2021-04-09T18:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.246 7f3fc8fc2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc40d2450 con 0x7f3fc40d6c90 2021-04-09T18:29:19.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.246 7f3fc8fc2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc40d7090 con 0x7f3fc410e7e0 2021-04-09T18:29:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.246 7f3fc2d9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3fc40cffb0 0x7f3fc40d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36070/0 (socket says 172.21.15.32:36070) 2021-04-09T18:29:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.246 7f3fc2d9d700 1 -- 172.21.15.32:0/2579647287 learned_addr learned my addr 172.21.15.32:0/2579647287 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:19.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc359e700 1 --2- 172.21.15.32:0/2579647287 >> v2:172.21.15.86:3300/0 conn(0x7f3fc410e7e0 0x7f3fc40d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc1d9b700 1 -- 172.21.15.32:0/2579647287 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047839123 0 0) 0x7f3fb4000ce0 con 0x7f3fc40cffb0 2021-04-09T18:29:19.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc1d9b700 1 -- 172.21.15.32:0/2579647287 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa8002fe0 con 0x7f3fc40cffb0 2021-04-09T18:29:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc1d9b700 1 -- 172.21.15.32:0/2579647287 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274064687 0 0) 0x7f3fb0000c60 con 0x7f3fc40d6c90 2021-04-09T18:29:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc1d9b700 1 -- 172.21.15.32:0/2579647287 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa8002fe0 con 0x7f3fc40d6c90 2021-04-09T18:29:19.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc359e700 1 -- 172.21.15.32:0/2579647287 >> v1:172.21.15.86:6789/0 conn(0x7f3fc40d6c90 legacy=0x7f3fc410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc359e700 1 -- 172.21.15.32:0/2579647287 >> v1:172.21.15.32:6789/0 conn(0x7f3fc40cffb0 legacy=0x7f3fc40d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc359e700 1 -- 172.21.15.32:0/2579647287 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc40d91f0 con 0x7f3fc410e7e0 2021-04-09T18:29:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc359e700 1 --2- 172.21.15.32:0/2579647287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc410e7e0 0x7f3fc40d8000 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f3fbc006e50 tx=0x7f3fbc009df0).ready entity=mon.1 client_cookie=3089d45d0f83aef8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:19.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc1d9b700 1 -- 172.21.15.32:0/2579647287 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fbc00b510 con 0x7f3fc410e7e0 2021-04-09T18:29:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc1d9b700 1 -- 172.21.15.32:0/2579647287 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fbc003030 con 0x7f3fc410e7e0 2021-04-09T18:29:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc1d9b700 1 -- 172.21.15.32:0/2579647287 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fbc00b7d0 con 0x7f3fc410e7e0 2021-04-09T18:29:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc8fc2700 1 -- 172.21.15.32:0/2579647287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc410e7e0 msgr2=0x7f3fc40d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:19.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.247 7f3fc8fc2700 1 --2- 172.21.15.32:0/2579647287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc410e7e0 0x7f3fc40d8000 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f3fbc006e50 tx=0x7f3fbc009df0).stop 2021-04-09T18:29:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.248 7f3fc8fc2700 1 -- 172.21.15.32:0/2579647287 shutdown_connections 2021-04-09T18:29:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.248 7f3fc8fc2700 1 --2- 172.21.15.32:0/2579647287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc410e7e0 0x7f3fc40d8000 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.248 7f3fc8fc2700 1 -- 172.21.15.32:0/2579647287 >> 172.21.15.32:0/2579647287 conn(0x7f3fc40ce5d0 msgr2=0x7f3fc40cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:19.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.248 7f3fc8fc2700 1 -- 172.21.15.32:0/2579647287 shutdown_connections 2021-04-09T18:29:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.248 7f3fc8fc2700 1 -- 172.21.15.32:0/2579647287 wait complete. 2021-04-09T18:29:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.268 7f3fc8fc2700 1 Processor -- start 2021-04-09T18:29:19.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.268 7f3fc8fc2700 1 -- start start 2021-04-09T18:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc8fc2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc41b8380 0x7f3fc41b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc8fc2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3fc408c7b0 0x7f3fc408cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc8fc2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc408d1d0 0x7f3fc41805b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:19.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc8fc2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc4181a30 con 0x7f3fc41b8380 2021-04-09T18:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc8fc2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc408c230 con 0x7f3fc408d1d0 2021-04-09T18:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc8fc2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc41b8da0 con 0x7f3fc408c7b0 2021-04-09T18:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc41b8380 0x7f3fc41b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:19.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc2d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc41b8380 0x7f3fc41b8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54718/0 (socket says 172.21.15.32:54718) 2021-04-09T18:29:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc2d9d700 1 -- 172.21.15.32:0/807270211 learned_addr learned my addr 172.21.15.32:0/807270211 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc259c700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3fc408c7b0 0x7f3fc408cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:19.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc359e700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc408d1d0 0x7f3fc41805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc2d9d700 1 -- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3fc408c7b0 msgr2=0x7f3fc408cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc2d9d700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3fc408c7b0 0x7f3fc408cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc2d9d700 1 -- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc408d1d0 msgr2=0x7f3fc41805b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:19.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.269 7f3fc2d9d700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc408d1d0 0x7f3fc41805b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:19.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.270 7f3fc2d9d700 1 -- 172.21.15.32:0/807270211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc41b8f90 con 0x7f3fc41b8380 2021-04-09T18:29:19.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.270 7f3fc2d9d700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc41b8380 0x7f3fc41b8780 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f3fb40014b0 tx=0x7f3fb400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:19.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.270 7f3fbb7fe700 1 -- 172.21.15.32:0/807270211 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fb400acc0 con 0x7f3fc41b8380 2021-04-09T18:29:19.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.270 7f3fbb7fe700 1 -- 172.21.15.32:0/807270211 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb4007030 con 0x7f3fc41b8380 2021-04-09T18:29:19.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.270 7f3fbb7fe700 1 -- 172.21.15.32:0/807270211 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fb4004620 con 0x7f3fc41b8380 2021-04-09T18:29:19.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.270 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc41b8f90 con 0x7f3fc41b8380 2021-04-09T18:29:19.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.270 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc41b8f90 con 0x7f3fc41b8380 2021-04-09T18:29:19.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.270 7f3fbb7fe700 1 -- 172.21.15.32:0/807270211 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3fb40043a0 con 0x7f3fc41b8380 2021-04-09T18:29:19.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.271 7f3fbb7fe700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3fa0018930 0x7f3fa001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:19.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.271 7f3fbb7fe700 1 -- 172.21.15.32:0/807270211 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f3fb4032780 con 0x7f3fc41b8380 2021-04-09T18:29:19.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.271 7f3fc259c700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3fa0018930 0x7f3fa001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:19.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.271 7f3fc259c700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3fa0018930 0x7f3fa001ae10 crc :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:19.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.290 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc41b7180 con 0x7f3fc41b8380 2021-04-09T18:29:19.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.293 7f3fbb7fe700 1 -- 172.21.15.32:0/807270211 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3fb40009b0 con 0x7f3fc41b8380 2021-04-09T18:29:19.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.493 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f3fc408c230 con 0x7f3fc41b8380 2021-04-09T18:29:20.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.942 7f3fbb7fe700 1 -- 172.21.15.32:0/807270211 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v160) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3fb40043a0 con 0x7f3fc41b8380 2021-04-09T18:29:20.042 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:29:20.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3fa0018930 msgr2=0x7f3fa001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3fa0018930 0x7f3fa001ae10 crc :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc41b8380 msgr2=0x7f3fc41b8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc41b8380 0x7f3fc41b8780 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f3fb40014b0 tx=0x7f3fb400a8d0).stop 2021-04-09T18:29:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 shutdown_connections 2021-04-09T18:29:20.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3fa0018930 0x7f3fa001ae10 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3fc408d1d0 0x7f3fc41805b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3fc408c7b0 0x7f3fc408cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 --2- 172.21.15.32:0/807270211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3fc41b8380 0x7f3fc41b8780 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 >> 172.21.15.32:0/807270211 conn(0x7f3fc4189800 msgr2=0x7f3fc41b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:20.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 shutdown_connections 2021-04-09T18:29:20.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:19.961 7f3fc8fc2700 1 -- 172.21.15.32:0/807270211 wait complete. 2021-04-09T18:29:20.055 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:29:20.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:29:20.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.121 7f29c8469700 1 Processor -- start 2021-04-09T18:29:20.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.121 7f29c8469700 1 -- start start 2021-04-09T18:29:20.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.121 7f29c8469700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c010d010 0x7f29c00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.121 7f29c8469700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f29c00d1da0 0x7f29c00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.121 7f29c8469700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f29c00d82c0 0x7f29c010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.121 7f29c8469700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c010e780 con 0x7f29c00d1da0 2021-04-09T18:29:20.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.121 7f29c8469700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c00d8d30 con 0x7f29c00d82c0 2021-04-09T18:29:20.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.121 7f29c8469700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c00d9220 con 0x7f29c010d010 2021-04-09T18:29:20.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5a84700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f29c00d1da0 0x7f29c00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5a84700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f29c00d1da0 0x7f29c00d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54730/0 (socket says 172.21.15.32:54730) 2021-04-09T18:29:20.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5a84700 1 -- 172.21.15.32:0/4117409050 learned_addr learned my addr 172.21.15.32:0/4117409050 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:20.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c6a86700 1 --2- 172.21.15.32:0/4117409050 >> v2:172.21.15.86:3300/0 conn(0x7f29c00d82c0 0x7f29c010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c6285700 1 --2- 172.21.15.32:0/4117409050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c010d010 0x7f29c00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5a84700 1 -- 172.21.15.32:0/4117409050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c010d010 msgr2=0x7f29c00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:20.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5a84700 1 --2- 172.21.15.32:0/4117409050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c010d010 0x7f29c00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5a84700 1 -- 172.21.15.32:0/4117409050 >> v2:172.21.15.86:3300/0 conn(0x7f29c00d82c0 msgr2=0x7f29c010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:20.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5a84700 1 --2- 172.21.15.32:0/4117409050 >> v2:172.21.15.86:3300/0 conn(0x7f29c00d82c0 0x7f29c010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5a84700 1 -- 172.21.15.32:0/4117409050 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c010d480 con 0x7f29c00d1da0 2021-04-09T18:29:20.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c6a86700 1 --2- 172.21.15.32:0/4117409050 >> v2:172.21.15.86:3300/0 conn(0x7f29c00d82c0 0x7f29c010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:20.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5a84700 1 --2- 172.21.15.32:0/4117409050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c00d1da0 0x7f29c00d7ca0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f29b0006700 tx=0x7f29b0009b80).ready entity=mon.0 client_cookie=2a37b853b76eb13d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:20.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5283700 1 -- 172.21.15.32:0/4117409050 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29b000c0c0 con 0x7f29c00d1da0 2021-04-09T18:29:20.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5283700 1 -- 172.21.15.32:0/4117409050 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b000c440 con 0x7f29c00d1da0 2021-04-09T18:29:20.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c5283700 1 -- 172.21.15.32:0/4117409050 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29b000c920 con 0x7f29c00d1da0 2021-04-09T18:29:20.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c8469700 1 -- 172.21.15.32:0/4117409050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c00d1da0 msgr2=0x7f29c00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.122 7f29c8469700 1 --2- 172.21.15.32:0/4117409050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c00d1da0 0x7f29c00d7ca0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f29b0006700 tx=0x7f29b0009b80).stop 2021-04-09T18:29:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.123 7f29c8469700 1 -- 172.21.15.32:0/4117409050 shutdown_connections 2021-04-09T18:29:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.123 7f29c8469700 1 --2- 172.21.15.32:0/4117409050 >> v2:172.21.15.86:3300/0 conn(0x7f29c00d82c0 0x7f29c010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.123 7f29c8469700 1 --2- 172.21.15.32:0/4117409050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c00d1da0 0x7f29c00d7ca0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.123 7f29c8469700 1 --2- 172.21.15.32:0/4117409050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c010d010 0x7f29c00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.123 7f29c8469700 1 -- 172.21.15.32:0/4117409050 >> 172.21.15.32:0/4117409050 conn(0x7f29c00ce5f0 msgr2=0x7f29c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.123 7f29c8469700 1 -- 172.21.15.32:0/4117409050 shutdown_connections 2021-04-09T18:29:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.123 7f29c8469700 1 -- 172.21.15.32:0/4117409050 wait complete. 2021-04-09T18:29:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.143 7f29c8469700 1 Processor -- start 2021-04-09T18:29:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.143 7f29c8469700 1 -- start start 2021-04-09T18:29:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c8469700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c00aa0f0 0x7f29c01832b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c8469700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c01838d0 0x7f29c017ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c8469700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c017f1d0 0x7f29c017cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c8469700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c017e630 con 0x7f29c01838d0 2021-04-09T18:29:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c8469700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0183cd0 con 0x7f29c00aa0f0 2021-04-09T18:29:20.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c8469700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c017f5d0 con 0x7f29c017f1d0 2021-04-09T18:29:20.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c5a84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c01838d0 0x7f29c017ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c5a84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c01838d0 0x7f29c017ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54732/0 (socket says 172.21.15.32:54732) 2021-04-09T18:29:20.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c5a84700 1 -- 172.21.15.32:0/1169859800 learned_addr learned my addr 172.21.15.32:0/1169859800 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:20.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c6a86700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c017f1d0 0x7f29c017cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c6285700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c00aa0f0 0x7f29c01832b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c5a84700 1 -- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c017f1d0 msgr2=0x7f29c017cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.144 7f29c5a84700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c017f1d0 0x7f29c017cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.145 7f29c5a84700 1 -- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c00aa0f0 msgr2=0x7f29c01832b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.145 7f29c5a84700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c00aa0f0 0x7f29c01832b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.145 7f29c5a84700 1 -- 172.21.15.32:0/1169859800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c017d5e0 con 0x7f29c01838d0 2021-04-09T18:29:20.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.145 7f29c5a84700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c01838d0 0x7f29c017ebb0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7f29b000b580 tx=0x7f29b000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:20.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.145 7f29b6ffd700 1 -- 172.21.15.32:0/1169859800 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29b000bd10 con 0x7f29c01838d0 2021-04-09T18:29:20.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.145 7f29b6ffd700 1 -- 172.21.15.32:0/1169859800 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b0002030 con 0x7f29c01838d0 2021-04-09T18:29:20.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.145 7f29c8469700 1 -- 172.21.15.32:0/1169859800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c017d5e0 con 0x7f29c01838d0 2021-04-09T18:29:20.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.145 7f29b6ffd700 1 -- 172.21.15.32:0/1169859800 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29b0002550 con 0x7f29c01838d0 2021-04-09T18:29:20.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.145 7f29c8469700 1 -- 172.21.15.32:0/1169859800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c017d5e0 con 0x7f29c01838d0 2021-04-09T18:29:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.146 7f29b6ffd700 1 -- 172.21.15.32:0/1169859800 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f29b0000c90 con 0x7f29c01838d0 2021-04-09T18:29:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.146 7f29b6ffd700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f29a8018930 0x7f29a801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.146 7f29b6ffd700 1 -- 172.21.15.32:0/1169859800 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f29b0032710 con 0x7f29c01838d0 2021-04-09T18:29:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.146 7f29c6285700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f29a8018930 0x7f29a801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.146 7f29c6285700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f29a8018930 0x7f29a801adc0 crc :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:20.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.165 7f29c8469700 1 -- 172.21.15.32:0/1169859800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c00f1850 con 0x7f29c01838d0 2021-04-09T18:29:20.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.169 7f29b6ffd700 1 -- 172.21.15.32:0/1169859800 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f29b001f780 con 0x7f29c01838d0 2021-04-09T18:29:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.356 7f29c8469700 1 -- 172.21.15.32:0/1169859800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f29c00f1850 con 0x7f29c01838d0 2021-04-09T18:29:20.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.356 7f29b6ffd700 1 -- 172.21.15.32:0/1169859800 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v37) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f29b001f780 con 0x7f29c01838d0 2021-04-09T18:29:20.441 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:20.441 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:29:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:29:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:29:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:29:20.442 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:29:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.443 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:29:20.444 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:29:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:29:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.445 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.446 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:29:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:29:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:29:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.447 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.448 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:29:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:29:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:29:20.449 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.450 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:29:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:29:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.451 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.452 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:29:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:29:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.453 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.454 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:29:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:29:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.455 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.456 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:29:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:29:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.457 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:29:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:29:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.459 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:29:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:29:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.462 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:29:20.463 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:29:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.464 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:29:20.465 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:29:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.466 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.467 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:29:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:29:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:29:20.468 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:29:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:29:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:29:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:29:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:29:20.470 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:29:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:29:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.471 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:29:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:29:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:29:20.472 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:29:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:29:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:29:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:29:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.474 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:29:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:29:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:29:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:29:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:29:20.476 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:29:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:29:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:29:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:20.477 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:29:20.478 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:29:20.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 -- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f29a8018930 msgr2=0x7f29a801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f29a8018930 0x7f29a801adc0 crc :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 -- 172.21.15.32:0/1169859800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c01838d0 msgr2=0x7f29c017ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c01838d0 0x7f29c017ebb0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7f29b000b580 tx=0x7f29b000b920).stop 2021-04-09T18:29:20.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 -- 172.21.15.32:0/1169859800 shutdown_connections 2021-04-09T18:29:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f29a8018930 0x7f29a801adc0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c01838d0 0x7f29c017ebb0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c017f1d0 0x7f29c017cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 --2- 172.21.15.32:0/1169859800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c00aa0f0 0x7f29c01832b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 -- 172.21.15.32:0/1169859800 >> 172.21.15.32:0/1169859800 conn(0x7f29c01925d0 msgr2=0x7f29c01a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 -- 172.21.15.32:0/1169859800 shutdown_connections 2021-04-09T18:29:20.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.359 7f29c8469700 1 -- 172.21.15.32:0/1169859800 wait complete. 2021-04-09T18:29:20.483 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:29:20.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.547 7f78a2400700 1 Processor -- start 2021-04-09T18:29:20.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.548 7f78a2400700 1 -- start start 2021-04-09T18:29:20.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.548 7f78a2400700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f789c0d7ca0 0x7f789c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.548 7f78a2400700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c10d480 con 0x7f789c10d010 2021-04-09T18:29:20.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.548 7f78a2400700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c0d86c0 con 0x7f789c10da00 2021-04-09T18:29:20.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.548 7f78a2400700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c0d9220 con 0x7f789c0d7ca0 2021-04-09T18:29:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.548 7f7893fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f789c0d7ca0 0x7f789c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.548 7f789bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f789c10d010 0x7f789c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36098/0 (socket says 172.21.15.32:36098) 2021-04-09T18:29:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.548 7f789bfff700 1 -- 172.21.15.32:0/1703779598 learned_addr learned my addr 172.21.15.32:0/1703779598 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f7893fff700 1 -- 172.21.15.32:0/1703779598 >> v1:172.21.15.86:6789/0 conn(0x7f789c10da00 legacy=0x7f789c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:20.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f7893fff700 1 -- 172.21.15.32:0/1703779598 >> v1:172.21.15.32:6789/0 conn(0x7f789c10d010 legacy=0x7f789c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f7893fff700 1 -- 172.21.15.32:0/1703779598 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c0d1f70 con 0x7f789c0d7ca0 2021-04-09T18:29:20.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f789b7fe700 1 -- 172.21.15.32:0/1703779598 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160712375 0 0) 0x7f788c000ce0 con 0x7f789c10d010 2021-04-09T18:29:20.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f7893fff700 1 --2- 172.21.15.32:0/1703779598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c0d7ca0 0x7f789c0d80a0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7f7884006650 tx=0x7f7884009a00).ready entity=mon.0 client_cookie=cf27c10d394ded9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:20.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f789b7fe700 1 -- 172.21.15.32:0/1703779598 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7884009df0 con 0x7f789c0d7ca0 2021-04-09T18:29:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f789b7fe700 1 -- 172.21.15.32:0/1703779598 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78840009b0 con 0x7f789c0d7ca0 2021-04-09T18:29:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f789b7fe700 1 -- 172.21.15.32:0/1703779598 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f788400c2b0 con 0x7f789c0d7ca0 2021-04-09T18:29:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f78a2400700 1 -- 172.21.15.32:0/1703779598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c0d7ca0 msgr2=0x7f789c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.549 7f78a2400700 1 --2- 172.21.15.32:0/1703779598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c0d7ca0 0x7f789c0d80a0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7f7884006650 tx=0x7f7884009a00).stop 2021-04-09T18:29:20.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.550 7f78a2400700 1 -- 172.21.15.32:0/1703779598 shutdown_connections 2021-04-09T18:29:20.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.550 7f78a2400700 1 --2- 172.21.15.32:0/1703779598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c0d7ca0 0x7f789c0d80a0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.550 7f78a2400700 1 -- 172.21.15.32:0/1703779598 >> 172.21.15.32:0/1703779598 conn(0x7f789c0ce5f0 msgr2=0x7f789c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:20.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.550 7f78a2400700 1 -- 172.21.15.32:0/1703779598 shutdown_connections 2021-04-09T18:29:20.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.550 7f78a2400700 1 -- 172.21.15.32:0/1703779598 wait complete. 2021-04-09T18:29:20.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.570 7f78a2400700 1 Processor -- start 2021-04-09T18:29:20.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.570 7f78a2400700 1 -- start start 2021-04-09T18:29:20.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a2400700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f789c0ae090 0x7f789c18a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a2400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c18a9f0 0x7f789c1a9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a2400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f789c1a9950 0x7f789c188d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a2400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c18adf0 con 0x7f789c18a9f0 2021-04-09T18:29:20.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a2400700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c1a9d50 con 0x7f789c1a9950 2021-04-09T18:29:20.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a2400700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c1884b0 con 0x7f789c0ae090 2021-04-09T18:29:20.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f7893fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c18a9f0 0x7f789c1a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f7893fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c18a9f0 0x7f789c1a9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54748/0 (socket says 172.21.15.32:54748) 2021-04-09T18:29:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f7893fff700 1 -- 172.21.15.32:0/1280570727 learned_addr learned my addr 172.21.15.32:0/1280570727 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:20.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f789bfff700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f789c0ae090 0x7f789c18a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a0a1d700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f789c1a9950 0x7f789c188d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f7893fff700 1 -- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f789c0ae090 msgr2=0x7f789c18a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f7893fff700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f789c0ae090 0x7f789c18a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f7893fff700 1 -- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f789c1a9950 msgr2=0x7f789c188d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f7893fff700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f789c1a9950 0x7f789c188d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f7893fff700 1 -- 172.21.15.32:0/1280570727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c08ef00 con 0x7f789c18a9f0 2021-04-09T18:29:20.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a0a1d700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f789c1a9950 0x7f789c188d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:20.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f7893fff700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c18a9f0 0x7f789c1a9330 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7f78840096a0 tx=0x7f788400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:20.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78997fa700 1 -- 172.21.15.32:0/1280570727 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7884000c90 con 0x7f789c18a9f0 2021-04-09T18:29:20.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78997fa700 1 -- 172.21.15.32:0/1280570727 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7884002030 con 0x7f789c18a9f0 2021-04-09T18:29:20.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a2400700 1 -- 172.21.15.32:0/1280570727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c08ef00 con 0x7f789c18a9f0 2021-04-09T18:29:20.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78997fa700 1 -- 172.21.15.32:0/1280570727 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7884002550 con 0x7f789c18a9f0 2021-04-09T18:29:20.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.571 7f78a2400700 1 -- 172.21.15.32:0/1280570727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c08ef00 con 0x7f789c18a9f0 2021-04-09T18:29:20.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.572 7f78997fa700 1 -- 172.21.15.32:0/1280570727 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7884002210 con 0x7f789c18a9f0 2021-04-09T18:29:20.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.572 7f78997fa700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78800188d0 0x7f788001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:20.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.572 7f78997fa700 1 -- 172.21.15.32:0/1280570727 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f7884032740 con 0x7f789c18a9f0 2021-04-09T18:29:20.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.572 7f789bfff700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78800188d0 0x7f788001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:20.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.573 7f789bfff700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78800188d0 0x7f788001add0 crc :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:20.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.592 7f78a2400700 1 -- 172.21.15.32:0/1280570727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c10d480 con 0x7f789c18a9f0 2021-04-09T18:29:20.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.596 7f78997fa700 1 -- 172.21.15.32:0/1280570727 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f78840009a0 con 0x7f789c18a9f0 2021-04-09T18:29:20.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.786 7f78a2400700 1 -- 172.21.15.32:0/1280570727 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f789c10d480 con 0x7f789c18a9f0 2021-04-09T18:29:20.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.787 7f78997fa700 1 -- 172.21.15.32:0/1280570727 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f7884002010 con 0x7f789c18a9f0 2021-04-09T18:29:20.872 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:20.872 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":160,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:29:19.935701","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:29:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 -- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78800188d0 msgr2=0x7f788001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78800188d0 0x7f788001add0 crc :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 -- 172.21.15.32:0/1280570727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c18a9f0 msgr2=0x7f789c1a9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:20.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c18a9f0 0x7f789c1a9330 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7f78840096a0 tx=0x7f788400b990).stop 2021-04-09T18:29:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 -- 172.21.15.32:0/1280570727 shutdown_connections 2021-04-09T18:29:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78800188d0 0x7f788001add0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f789c18a9f0 0x7f789c1a9330 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f789c1a9950 0x7f789c188d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 --2- 172.21.15.32:0/1280570727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f789c0ae090 0x7f789c18a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 -- 172.21.15.32:0/1280570727 >> 172.21.15.32:0/1280570727 conn(0x7f789c1aa1b0 msgr2=0x7f789c1ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 -- 172.21.15.32:0/1280570727 shutdown_connections 2021-04-09T18:29:20.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.789 7f78a2400700 1 -- 172.21.15.32:0/1280570727 wait complete. 2021-04-09T18:29:20.884 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:29:20.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:29:21.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.953 7fc5d8cc6700 1 Processor -- start 2021-04-09T18:29:21.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.953 7fc5d8cc6700 1 -- start start 2021-04-09T18:29:21.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d8cc6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc5d410d010 0x7fc5d40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d8cc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40d1da0 0x7fc5d40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d8cc6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d410d480 con 0x7fc5d40d82c0 2021-04-09T18:29:21.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d8cc6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d40d9220 con 0x7fc5d410d010 2021-04-09T18:29:21.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d8cc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40d86c0 con 0x7fc5d40d1da0 2021-04-09T18:29:21.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d305f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc5d40d82c0 0x7fc5d410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36116/0 (socket says 172.21.15.32:36116) 2021-04-09T18:29:21.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d305f700 1 -- 172.21.15.32:0/2080624247 learned_addr learned my addr 172.21.15.32:0/2080624247 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:21.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d205d700 1 --2- 172.21.15.32:0/2080624247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40d1da0 0x7fc5d40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d285e700 1 --2- 172.21.15.32:0/2080624247 >> v2:172.21.15.86:3300/0 conn(0x7fc5d410d010 0x7fc5d40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d185c700 1 -- 172.21.15.32:0/2080624247 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203930862 0 0) 0x7fc5cc000d30 con 0x7fc5d40d82c0 2021-04-09T18:29:21.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d185c700 1 -- 172.21.15.32:0/2080624247 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc002fe0 con 0x7fc5d40d82c0 2021-04-09T18:29:21.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d285e700 1 -- 172.21.15.32:0/2080624247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40d1da0 msgr2=0x7fc5d40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:21.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d285e700 1 --2- 172.21.15.32:0/2080624247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40d1da0 0x7fc5d40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d285e700 1 -- 172.21.15.32:0/2080624247 >> v1:172.21.15.32:6789/0 conn(0x7fc5d40d82c0 legacy=0x7fc5d410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d285e700 1 -- 172.21.15.32:0/2080624247 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d410e1f0 con 0x7fc5d410d010 2021-04-09T18:29:21.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.954 7fc5d205d700 1 --2- 172.21.15.32:0/2080624247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40d1da0 0x7fc5d40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.955 7fc5d285e700 1 --2- 172.21.15.32:0/2080624247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d410d010 0x7fc5d40d1780 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7fc5c4006650 tx=0x7fc5c4009a80).ready entity=mon.1 client_cookie=76a8a137973fc280 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.955 7fc5d185c700 1 -- 172.21.15.32:0/2080624247 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5c40009b0 con 0x7fc5d410d010 2021-04-09T18:29:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.955 7fc5d185c700 1 -- 172.21.15.32:0/2080624247 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c400c0c0 con 0x7fc5d410d010 2021-04-09T18:29:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.955 7fc5d185c700 1 -- 172.21.15.32:0/2080624247 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5c400c510 con 0x7fc5d410d010 2021-04-09T18:29:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.955 7fc5d8cc6700 1 -- 172.21.15.32:0/2080624247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d410d010 msgr2=0x7fc5d40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.955 7fc5d8cc6700 1 --2- 172.21.15.32:0/2080624247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d410d010 0x7fc5d40d1780 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7fc5c4006650 tx=0x7fc5c4009a80).stop 2021-04-09T18:29:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.956 7fc5d8cc6700 1 -- 172.21.15.32:0/2080624247 shutdown_connections 2021-04-09T18:29:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.956 7fc5d8cc6700 1 --2- 172.21.15.32:0/2080624247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40d1da0 0x7fc5d40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.956 7fc5d8cc6700 1 --2- 172.21.15.32:0/2080624247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d410d010 0x7fc5d40d1780 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.956 7fc5d8cc6700 1 -- 172.21.15.32:0/2080624247 >> 172.21.15.32:0/2080624247 conn(0x7fc5d40ce5f0 msgr2=0x7fc5d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.956 7fc5d8cc6700 1 -- 172.21.15.32:0/2080624247 shutdown_connections 2021-04-09T18:29:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.956 7fc5d8cc6700 1 -- 172.21.15.32:0/2080624247 wait complete. 2021-04-09T18:29:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.976 7fc5d8cc6700 1 Processor -- start 2021-04-09T18:29:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.976 7fc5d8cc6700 1 -- start start 2021-04-09T18:29:21.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.976 7fc5d8cc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40b1e50 0x7fc5d40b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.976 7fc5d8cc6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d40b9fd0 0x7fc5d40b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d8cc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d40b5510 0x7fc5d40b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d8cc6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40b2250 con 0x7fc5d40b5510 2021-04-09T18:29:21.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d8cc6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40a1220 con 0x7fc5d40b9fd0 2021-04-09T18:29:21.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d8cc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40b6b30 con 0x7fc5d40b1e50 2021-04-09T18:29:21.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d40b5510 0x7fc5d40b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d40b5510 0x7fc5d40b5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54764/0 (socket says 172.21.15.32:54764) 2021-04-09T18:29:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d285e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40b1e50 0x7fc5d40b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d305f700 1 -- 172.21.15.32:0/498207926 learned_addr learned my addr 172.21.15.32:0/498207926 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d205d700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d40b9fd0 0x7fc5d40b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d305f700 1 -- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40b1e50 msgr2=0x7fc5d40b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d305f700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40b1e50 0x7fc5d40b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d305f700 1 -- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d40b9fd0 msgr2=0x7fc5d40b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d305f700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d40b9fd0 0x7fc5d40b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d305f700 1 -- 172.21.15.32:0/498207926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d40b5f30 con 0x7fc5d40b5510 2021-04-09T18:29:21.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d285e700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40b1e50 0x7fc5d40b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:21.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d305f700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d40b5510 0x7fc5d40b5910 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7fc5cc009980 tx=0x7fc5cc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:21.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5c37fe700 1 -- 172.21.15.32:0/498207926 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5cc00be10 con 0x7fc5d40b5510 2021-04-09T18:29:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5c37fe700 1 -- 172.21.15.32:0/498207926 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc006080 con 0x7fc5d40b5510 2021-04-09T18:29:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5c37fe700 1 -- 172.21.15.32:0/498207926 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5cc004720 con 0x7fc5d40b5510 2021-04-09T18:29:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d40b5f30 con 0x7fc5d40b5510 2021-04-09T18:29:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.977 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d40b5f30 con 0x7fc5d40b5510 2021-04-09T18:29:21.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.978 7fc5c37fe700 1 -- 172.21.15.32:0/498207926 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc5cc006260 con 0x7fc5d40b5510 2021-04-09T18:29:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.978 7fc5c37fe700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc5b8018930 0x7fc5b801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.978 7fc5c37fe700 1 -- 172.21.15.32:0/498207926 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fc5cc032a40 con 0x7fc5d40b5510 2021-04-09T18:29:21.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.978 7fc5d285e700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc5b8018930 0x7fc5b801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.978 7fc5d285e700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc5b8018930 0x7fc5b801ae10 crc :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:21.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:20.997 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d41b7040 con 0x7fc5d40b5510 2021-04-09T18:29:21.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.001 7fc5c37fe700 1 -- 172.21.15.32:0/498207926 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc5cc000f90 con 0x7fc5d40b5510 2021-04-09T18:29:21.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.180 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc5d41b7040 con 0x7fc5d40b5510 2021-04-09T18:29:21.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.180 7fc5c37fe700 1 -- 172.21.15.32:0/498207926 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v160) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc5cc005f80 con 0x7fc5d40b5510 2021-04-09T18:29:21.265 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:29:21.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc5b8018930 msgr2=0x7fc5b801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc5b8018930 0x7fc5b801ae10 crc :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d40b5510 msgr2=0x7fc5d40b5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d40b5510 0x7fc5d40b5910 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7fc5cc009980 tx=0x7fc5cc00bcd0).stop 2021-04-09T18:29:21.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 shutdown_connections 2021-04-09T18:29:21.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc5b8018930 0x7fc5b801ae10 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5d40b5510 0x7fc5d40b5910 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d40b9fd0 0x7fc5d40b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 --2- 172.21.15.32:0/498207926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5d40b1e50 0x7fc5d40b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 >> 172.21.15.32:0/498207926 conn(0x7fc5d40b27e0 msgr2=0x7fc5d41b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:21.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 shutdown_connections 2021-04-09T18:29:21.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.183 7fc5d8cc6700 1 -- 172.21.15.32:0/498207926 wait complete. 2021-04-09T18:29:21.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:29:21.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.342 7f4a1538c700 1 Processor -- start 2021-04-09T18:29:21.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.342 7f4a1538c700 1 -- start start 2021-04-09T18:29:21.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a1538c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1010d010 0x7f4a100d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a1538c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a100d1da0 0x7f4a100d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a1538c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1010d480 con 0x7f4a100d82c0 2021-04-09T18:29:21.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a1538c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1010e780 con 0x7f4a100d1da0 2021-04-09T18:29:21.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a1538c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100d8d30 con 0x7f4a1010d010 2021-04-09T18:29:21.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a0e87c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a100d1da0 0x7f4a100d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a0f87e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4a100d82c0 0x7f4a1010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36130/0 (socket says 172.21.15.32:36130) 2021-04-09T18:29:21.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a0e87c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a100d1da0 0x7f4a100d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54768/0 (socket says 172.21.15.32:54768) 2021-04-09T18:29:21.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a0f87e700 1 -- 172.21.15.32:0/3806579980 learned_addr learned my addr 172.21.15.32:0/3806579980 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:21.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.343 7f4a0f07d700 1 --2- 172.21.15.32:0/3806579980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1010d010 0x7f4a100d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a0e87c700 1 -- 172.21.15.32:0/3806579980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1010d010 msgr2=0x7f4a100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a0e87c700 1 --2- 172.21.15.32:0/3806579980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1010d010 0x7f4a100d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a0e87c700 1 -- 172.21.15.32:0/3806579980 >> v1:172.21.15.32:6789/0 conn(0x7f4a100d82c0 legacy=0x7f4a1010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a0e87c700 1 -- 172.21.15.32:0/3806579980 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a100d9220 con 0x7f4a100d1da0 2021-04-09T18:29:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a0e07b700 1 -- 172.21.15.32:0/3806579980 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915668560 0 0) 0x7f4a08000d30 con 0x7f4a100d82c0 2021-04-09T18:29:21.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a0e87c700 1 --2- 172.21.15.32:0/3806579980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a100d1da0 0x7f4a100d7ca0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f49fc006650 tx=0x7f49fc009a00).ready entity=mon.0 client_cookie=57924c87ad7517bd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a0e07b700 1 -- 172.21.15.32:0/3806579980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49fc009df0 con 0x7f4a100d1da0 2021-04-09T18:29:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a0e07b700 1 -- 172.21.15.32:0/3806579980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49fc0009b0 con 0x7f4a100d1da0 2021-04-09T18:29:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a0e07b700 1 -- 172.21.15.32:0/3806579980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49fc00c2b0 con 0x7f4a100d1da0 2021-04-09T18:29:21.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a1538c700 1 -- 172.21.15.32:0/3806579980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a100d1da0 msgr2=0x7f4a100d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.344 7f4a1538c700 1 --2- 172.21.15.32:0/3806579980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a100d1da0 0x7f4a100d7ca0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f49fc006650 tx=0x7f49fc009a00).stop 2021-04-09T18:29:21.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.345 7f4a1538c700 1 -- 172.21.15.32:0/3806579980 shutdown_connections 2021-04-09T18:29:21.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.345 7f4a1538c700 1 --2- 172.21.15.32:0/3806579980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a100d1da0 0x7f4a100d7ca0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.345 7f4a1538c700 1 --2- 172.21.15.32:0/3806579980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1010d010 0x7f4a100d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.345 7f4a1538c700 1 -- 172.21.15.32:0/3806579980 >> 172.21.15.32:0/3806579980 conn(0x7f4a100ce5f0 msgr2=0x7f4a100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:21.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.345 7f4a1538c700 1 -- 172.21.15.32:0/3806579980 shutdown_connections 2021-04-09T18:29:21.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.345 7f4a1538c700 1 -- 172.21.15.32:0/3806579980 wait complete. 2021-04-09T18:29:21.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.365 7f4a1538c700 1 Processor -- start 2021-04-09T18:29:21.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.365 7f4a1538c700 1 -- start start 2021-04-09T18:29:21.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.365 7f4a1538c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a101a9360 0x7f4a101918f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.365 7f4a1538c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1018c440 0x7f4a1018c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a1538c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a101a80f0 0x7f4a101a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a1538c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a10191f10 con 0x7f4a101a9360 2021-04-09T18:29:21.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a1538c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a101a9760 con 0x7f4a101a80f0 2021-04-09T18:29:21.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a1538c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1018ce60 con 0x7f4a1018c440 2021-04-09T18:29:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a101a9360 0x7f4a101918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a101a9360 0x7f4a101918f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54774/0 (socket says 172.21.15.32:54774) 2021-04-09T18:29:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f07d700 1 -- 172.21.15.32:0/3683860468 learned_addr learned my addr 172.21.15.32:0/3683860468 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:21.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f87e700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a101a80f0 0x7f4a101a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0e87c700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1018c440 0x7f4a1018c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f07d700 1 -- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1018c440 msgr2=0x7f4a1018c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f07d700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1018c440 0x7f4a1018c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f07d700 1 -- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a101a80f0 msgr2=0x7f4a101a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f07d700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a101a80f0 0x7f4a101a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f07d700 1 -- 172.21.15.32:0/3683860468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a101a8b10 con 0x7f4a101a9360 2021-04-09T18:29:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a0f07d700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a101a9360 0x7f4a101918f0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7f4a00006f60 tx=0x7f4a0000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a07fff700 1 -- 172.21.15.32:0/3683860468 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a0000bd60 con 0x7f4a101a9360 2021-04-09T18:29:21.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a07fff700 1 -- 172.21.15.32:0/3683860468 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a00000c90 con 0x7f4a101a9360 2021-04-09T18:29:21.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a101a8b10 con 0x7f4a101a9360 2021-04-09T18:29:21.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a07fff700 1 -- 172.21.15.32:0/3683860468 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a00002220 con 0x7f4a101a9360 2021-04-09T18:29:21.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.366 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a101a8b10 con 0x7f4a101a9360 2021-04-09T18:29:21.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.367 7f4a07fff700 1 -- 172.21.15.32:0/3683860468 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4a000052e0 con 0x7f4a101a9360 2021-04-09T18:29:21.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.367 7f4a07fff700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49ec0188c0 0x7f49ec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.367 7f4a07fff700 1 -- 172.21.15.32:0/3683860468 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f4a00032460 con 0x7f4a101a9360 2021-04-09T18:29:21.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.367 7f4a0e87c700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49ec0188c0 0x7f49ec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.368 7f4a0e87c700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49ec0188c0 0x7f49ec01adc0 crc :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:21.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.388 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a101a9760 con 0x7f4a101a9360 2021-04-09T18:29:21.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.391 7f4a07fff700 1 -- 172.21.15.32:0/3683860468 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4a00006b00 con 0x7f4a101a9360 2021-04-09T18:29:21.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.569 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4a101a9760 con 0x7f4a101a9360 2021-04-09T18:29:21.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.570 7f4a07fff700 1 -- 172.21.15.32:0/3683860468 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v160) v1 ==== 84+0+64 (secure 0 0 0) 0x7f4a000052e0 con 0x7f4a101a9360 2021-04-09T18:29:21.654 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:21.655 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:29:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:29:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:29:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:29:21.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:21.656 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:29:21.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49ec0188c0 msgr2=0x7f49ec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49ec0188c0 0x7f49ec01adc0 crc :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a101a9360 msgr2=0x7f4a101918f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a101a9360 0x7f4a101918f0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7f4a00006f60 tx=0x7f4a0000b970).stop 2021-04-09T18:29:21.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 shutdown_connections 2021-04-09T18:29:21.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f49ec0188c0 0x7f49ec01adc0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a101a80f0 0x7f4a101a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a1018c440 0x7f4a1018c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 --2- 172.21.15.32:0/3683860468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a101a9360 0x7f4a101918f0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 >> 172.21.15.32:0/3683860468 conn(0x7f4a101837d0 msgr2=0x7f4a10195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:21.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 shutdown_connections 2021-04-09T18:29:21.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.572 7f4a1538c700 1 -- 172.21.15.32:0/3683860468 wait complete. 2021-04-09T18:29:21.667 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:29:21.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:29:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.736 7f59de267700 1 Processor -- start 2021-04-09T18:29:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.736 7f59de267700 1 -- start start 2021-04-09T18:29:21.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.736 7f59de267700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f59d810d010 0x7f59d80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59de267700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d80d8490 0x7f59d810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59de267700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d810d480 con 0x7f59d80d1da0 2021-04-09T18:29:21.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59de267700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d80d8d30 con 0x7f59d810d010 2021-04-09T18:29:21.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59de267700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80d9220 con 0x7f59d80d8490 2021-04-09T18:29:21.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59dc884700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d80d8490 0x7f59d810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59d7fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f59d810d010 0x7f59d80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59cffff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f59d80d1da0 0x7f59d80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:34070/0 (socket says 172.21.15.32:34070) 2021-04-09T18:29:21.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59cffff700 1 -- 172.21.15.32:0/262012653 learned_addr learned my addr 172.21.15.32:0/262012653 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:21.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59d7fff700 1 -- 172.21.15.32:0/262012653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d80d8490 msgr2=0x7f59d810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:21.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59d7fff700 1 --2- 172.21.15.32:0/262012653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d80d8490 0x7f59d810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59d7fff700 1 -- 172.21.15.32:0/262012653 >> v1:172.21.15.86:6789/0 conn(0x7f59d80d1da0 legacy=0x7f59d80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59d7fff700 1 -- 172.21.15.32:0/262012653 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d810e020 con 0x7f59d810d010 2021-04-09T18:29:21.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59d77fe700 1 -- 172.21.15.32:0/262012653 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105154675 0 0) 0x7f59c00010c0 con 0x7f59d80d1da0 2021-04-09T18:29:21.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59dc884700 1 --2- 172.21.15.32:0/262012653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d80d8490 0x7f59d810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:21.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.737 7f59d7fff700 1 --2- 172.21.15.32:0/262012653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d810d010 0x7f59d80d1780 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f59c8006650 tx=0x7f59c8009a80).ready entity=mon.1 client_cookie=1a7c7ffedd7cd988 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:21.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.738 7f59d77fe700 1 -- 172.21.15.32:0/262012653 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59c80009b0 con 0x7f59d810d010 2021-04-09T18:29:21.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.738 7f59d77fe700 1 -- 172.21.15.32:0/262012653 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c800c0c0 con 0x7f59d810d010 2021-04-09T18:29:21.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.738 7f59d77fe700 1 -- 172.21.15.32:0/262012653 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59c800c510 con 0x7f59d810d010 2021-04-09T18:29:21.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.738 7f59de267700 1 -- 172.21.15.32:0/262012653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d810d010 msgr2=0x7f59d80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.738 7f59de267700 1 --2- 172.21.15.32:0/262012653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d810d010 0x7f59d80d1780 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f59c8006650 tx=0x7f59c8009a80).stop 2021-04-09T18:29:21.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.738 7f59de267700 1 -- 172.21.15.32:0/262012653 shutdown_connections 2021-04-09T18:29:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.738 7f59de267700 1 --2- 172.21.15.32:0/262012653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d80d8490 0x7f59d810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.739 7f59de267700 1 --2- 172.21.15.32:0/262012653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d810d010 0x7f59d80d1780 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.739 7f59de267700 1 -- 172.21.15.32:0/262012653 >> 172.21.15.32:0/262012653 conn(0x7f59d80ce5f0 msgr2=0x7f59d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:21.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.739 7f59de267700 1 -- 172.21.15.32:0/262012653 shutdown_connections 2021-04-09T18:29:21.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.739 7f59de267700 1 -- 172.21.15.32:0/262012653 wait complete. 2021-04-09T18:29:21.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59de267700 1 Processor -- start 2021-04-09T18:29:21.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59de267700 1 -- start start 2021-04-09T18:29:21.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59de267700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d8184480 0x7f59d80befe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59de267700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d819dd60 0x7f59d819e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59de267700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d80c2ff0 0x7f59d80c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59de267700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80ada20 con 0x7f59d80c2ff0 2021-04-09T18:29:21.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59de267700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80c2a70 con 0x7f59d8184480 2021-04-09T18:29:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59de267700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d8184880 con 0x7f59d819dd60 2021-04-09T18:29:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59dc884700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d80c2ff0 0x7f59d80c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59dc884700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d80c2ff0 0x7f59d80c33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54792/0 (socket says 172.21.15.32:54792) 2021-04-09T18:29:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59dc884700 1 -- 172.21.15.32:0/1660168235 learned_addr learned my addr 172.21.15.32:0/1660168235 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59cffff700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d819dd60 0x7f59d819e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.759 7f59d7fff700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d8184480 0x7f59d80befe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59d7fff700 1 -- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d819dd60 msgr2=0x7f59d819e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59d7fff700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d819dd60 0x7f59d819e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59d7fff700 1 -- 172.21.15.32:0/1660168235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d80c2ff0 msgr2=0x7f59d80c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:21.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59d7fff700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d80c2ff0 0x7f59d80c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59d7fff700 1 -- 172.21.15.32:0/1660168235 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80c18c0 con 0x7f59d8184480 2021-04-09T18:29:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59cffff700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d819dd60 0x7f59d819e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:21.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59dc884700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d80c2ff0 0x7f59d80c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59d7fff700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d8184480 0x7f59d80befe0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f59c800b530 tx=0x7f59c800b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59d57fa700 1 -- 172.21.15.32:0/1660168235 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59c8009270 con 0x7f59d8184480 2021-04-09T18:29:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59d57fa700 1 -- 172.21.15.32:0/1660168235 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c8002030 con 0x7f59d8184480 2021-04-09T18:29:21.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59de267700 1 -- 172.21.15.32:0/1660168235 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d80c18c0 con 0x7f59d8184480 2021-04-09T18:29:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59d57fa700 1 -- 172.21.15.32:0/1660168235 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59c8002480 con 0x7f59d8184480 2021-04-09T18:29:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.760 7f59de267700 1 -- 172.21.15.32:0/1660168235 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d80c18c0 con 0x7f59d8184480 2021-04-09T18:29:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.761 7f59d57fa700 1 -- 172.21.15.32:0/1660168235 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f59c80091c0 con 0x7f59d8184480 2021-04-09T18:29:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.761 7f59d57fa700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59bc0188c0 0x7f59bc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:21.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.761 7f59d57fa700 1 -- 172.21.15.32:0/1660168235 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f59c8032410 con 0x7f59d8184480 2021-04-09T18:29:21.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.761 7f59cffff700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59bc0188c0 0x7f59bc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:21.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.761 7f59cffff700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59bc0188c0 0x7f59bc01adc0 crc :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:21.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.780 7f59de267700 1 -- 172.21.15.32:0/1660168235 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8093d50 con 0x7f59d8184480 2021-04-09T18:29:21.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.784 7f59d57fa700 1 -- 172.21.15.32:0/1660168235 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f59c8000c90 con 0x7f59d8184480 2021-04-09T18:29:22.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:21.969 7f59de267700 1 -- 172.21.15.32:0/1660168235 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f59d810d480 con 0x7f59d8184480 2021-04-09T18:29:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.948 7f59d57fa700 1 -- 172.21.15.32:0/1660168235 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v161) v1 ==== 133+0+0 (secure 0 0 0) 0x7f59c80091c0 con 0x7f59d8184480 2021-04-09T18:29:23.069 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:29:23.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.987 7f59de267700 1 -- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59bc0188c0 msgr2=0x7f59bc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.987 7f59de267700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59bc0188c0 0x7f59bc01adc0 crc :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.987 7f59de267700 1 -- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d8184480 msgr2=0x7f59d80befe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.987 7f59de267700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d8184480 0x7f59d80befe0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f59c800b530 tx=0x7f59c800b8d0).stop 2021-04-09T18:29:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.988 7f59de267700 1 -- 172.21.15.32:0/1660168235 shutdown_connections 2021-04-09T18:29:23.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.988 7f59de267700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59bc0188c0 0x7f59bc01adc0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:23.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.988 7f59de267700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59d80c2ff0 0x7f59d80c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:23.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.988 7f59de267700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d819dd60 0x7f59d819e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:23.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.988 7f59de267700 1 --2- 172.21.15.32:0/1660168235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d8184480 0x7f59d80befe0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:23.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.988 7f59de267700 1 -- 172.21.15.32:0/1660168235 >> 172.21.15.32:0/1660168235 conn(0x7f59d80a0b50 msgr2=0x7f59d80ac820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:23.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.988 7f59de267700 1 -- 172.21.15.32:0/1660168235 shutdown_connections 2021-04-09T18:29:23.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:22.988 7f59de267700 1 -- 172.21.15.32:0/1660168235 wait complete. 2021-04-09T18:29:23.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:29:23.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.155 7facddd6a700 1 Processor -- start 2021-04-09T18:29:23.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.155 7facddd6a700 1 -- start start 2021-04-09T18:29:23.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.156 7facddd6a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7facd80d7ca0 0x7facd80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:23.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.156 7facddd6a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd810d480 con 0x7facd810d010 2021-04-09T18:29:23.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.156 7facddd6a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd80d86c0 con 0x7facd810da00 2021-04-09T18:29:23.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.156 7facddd6a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd80d9220 con 0x7facd80d7ca0 2021-04-09T18:29:23.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.156 7facd6ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7facd80d7ca0 0x7facd80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:23.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.156 7facd6ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7facd80d7ca0 0x7facd80d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54798/0 (socket says 172.21.15.32:54798) 2021-04-09T18:29:23.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.156 7facd77fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7facd810d010 0x7facd80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36154/0 (socket says 172.21.15.32:36154) 2021-04-09T18:29:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.156 7facd6ffd700 1 -- 172.21.15.32:0/2462022238 learned_addr learned my addr 172.21.15.32:0/2462022238 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd67fc700 1 -- 172.21.15.32:0/2462022238 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054547843 0 0) 0x7facc80009b0 con 0x7facd810da00 2021-04-09T18:29:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd67fc700 1 -- 172.21.15.32:0/2462022238 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc4002fe0 con 0x7facd810da00 2021-04-09T18:29:23.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd67fc700 1 -- 172.21.15.32:0/2462022238 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663238738 0 0) 0x7facd0000ce0 con 0x7facd810d010 2021-04-09T18:29:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd6ffd700 1 -- 172.21.15.32:0/2462022238 >> v1:172.21.15.86:6789/0 conn(0x7facd810da00 legacy=0x7facd810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd6ffd700 1 -- 172.21.15.32:0/2462022238 >> v1:172.21.15.32:6789/0 conn(0x7facd810d010 legacy=0x7facd80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd6ffd700 1 -- 172.21.15.32:0/2462022238 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd80d1f70 con 0x7facd80d7ca0 2021-04-09T18:29:23.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd6ffd700 1 --2- 172.21.15.32:0/2462022238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd80d7ca0 0x7facd80d80a0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7facc0006a80 tx=0x7facc0009f00).ready entity=mon.0 client_cookie=a571cd1a8edc900e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:23.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd67fc700 1 -- 172.21.15.32:0/2462022238 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facc000b5c0 con 0x7facd80d7ca0 2021-04-09T18:29:23.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd67fc700 1 -- 172.21.15.32:0/2462022238 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc000b8c0 con 0x7facd80d7ca0 2021-04-09T18:29:23.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facd67fc700 1 -- 172.21.15.32:0/2462022238 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facc000bda0 con 0x7facd80d7ca0 2021-04-09T18:29:23.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facddd6a700 1 -- 172.21.15.32:0/2462022238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd80d7ca0 msgr2=0x7facd80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:23.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.157 7facddd6a700 1 --2- 172.21.15.32:0/2462022238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd80d7ca0 0x7facd80d80a0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7facc0006a80 tx=0x7facc0009f00).stop 2021-04-09T18:29:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.158 7facddd6a700 1 -- 172.21.15.32:0/2462022238 shutdown_connections 2021-04-09T18:29:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.158 7facddd6a700 1 --2- 172.21.15.32:0/2462022238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd80d7ca0 0x7facd80d80a0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.158 7facddd6a700 1 -- 172.21.15.32:0/2462022238 >> 172.21.15.32:0/2462022238 conn(0x7facd80ce5f0 msgr2=0x7facd80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:23.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.158 7facddd6a700 1 -- 172.21.15.32:0/2462022238 shutdown_connections 2021-04-09T18:29:23.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.158 7facddd6a700 1 -- 172.21.15.32:0/2462022238 wait complete. 2021-04-09T18:29:23.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.178 7facddd6a700 1 Processor -- start 2021-04-09T18:29:23.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.178 7facddd6a700 1 -- start start 2021-04-09T18:29:23.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facddd6a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd8179610 0x7facd8179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:23.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facddd6a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd817a820 0x7facd817ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facddd6a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd81949f0 0x7facd8194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facddd6a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd81b8d50 con 0x7facd8179610 2021-04-09T18:29:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facddd6a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd817b240 con 0x7facd81949f0 2021-04-09T18:29:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facddd6a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd8195410 con 0x7facd817a820 2021-04-09T18:29:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facd77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd8179610 0x7facd8179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facd77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd8179610 0x7facd8179a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54802/0 (socket says 172.21.15.32:54802) 2021-04-09T18:29:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facd77fe700 1 -- 172.21.15.32:0/1802326253 learned_addr learned my addr 172.21.15.32:0/1802326253 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facd7fff700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd81949f0 0x7facd8194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.179 7facd6ffd700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd817a820 0x7facd817ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7facd77fe700 1 -- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd817a820 msgr2=0x7facd817ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7facd77fe700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd817a820 0x7facd817ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7facd77fe700 1 -- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd81949f0 msgr2=0x7facd8194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7facd77fe700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd81949f0 0x7facd8194df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7facd77fe700 1 -- 172.21.15.32:0/1802326253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd8195630 con 0x7facd8179610 2021-04-09T18:29:23.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7facd77fe700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd8179610 0x7facd8179a10 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7facd000b530 tx=0x7facd000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:23.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7faccffff700 1 -- 172.21.15.32:0/1802326253 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd0006020 con 0x7facd8179610 2021-04-09T18:29:23.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7faccffff700 1 -- 172.21.15.32:0/1802326253 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd00063a0 con 0x7facd8179610 2021-04-09T18:29:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7facddd6a700 1 -- 172.21.15.32:0/1802326253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd8195630 con 0x7facd8179610 2021-04-09T18:29:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7faccffff700 1 -- 172.21.15.32:0/1802326253 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd0004590 con 0x7facd8179610 2021-04-09T18:29:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.180 7facddd6a700 1 -- 172.21.15.32:0/1802326253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd8195630 con 0x7facd8179610 2021-04-09T18:29:23.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.181 7faccffff700 1 -- 172.21.15.32:0/1802326253 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7facd000a030 con 0x7facd8179610 2021-04-09T18:29:23.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.181 7faccffff700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7facbc018930 0x7facbc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:23.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.181 7faccffff700 1 -- 172.21.15.32:0/1802326253 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5111+0+0 (secure 0 0 0) 0x7facd00327d0 con 0x7facd8179610 2021-04-09T18:29:23.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.181 7facd6ffd700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7facbc018930 0x7facbc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:23.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.181 7facd6ffd700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7facbc018930 0x7facbc01ae10 crc :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:23.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.200 7facddd6a700 1 -- 172.21.15.32:0/1802326253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd817a2f0 con 0x7facd8179610 2021-04-09T18:29:23.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.203 7faccffff700 1 -- 172.21.15.32:0/1802326253 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7facd0006380 con 0x7facd8179610 2021-04-09T18:29:23.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:23.393 7facddd6a700 1 -- 172.21.15.32:0/1802326253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7facd817a2f0 con 0x7facd8179610 2021-04-09T18:29:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.251 7faccffff700 1 -- 172.21.15.32:0/1802326253 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v166) v1 ==== 125+0+0 (secure 0 0 0) 0x7facd0006380 con 0x7facd8179610 2021-04-09T18:29:24.342 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:29:24.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 -- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7facbc018930 msgr2=0x7facbc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7facbc018930 0x7facbc01ae10 crc :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 -- 172.21.15.32:0/1802326253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd8179610 msgr2=0x7facd8179a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd8179610 0x7facd8179a10 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7facd000b530 tx=0x7facd000b8d0).stop 2021-04-09T18:29:24.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 -- 172.21.15.32:0/1802326253 shutdown_connections 2021-04-09T18:29:24.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7facbc018930 0x7facbc01ae10 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd81949f0 0x7facd8194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd817a820 0x7facd817ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 --2- 172.21.15.32:0/1802326253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7facd8179610 0x7facd8179a10 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 -- 172.21.15.32:0/1802326253 >> 172.21.15.32:0/1802326253 conn(0x7facd8092a00 msgr2=0x7facd81b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:24.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 -- 172.21.15.32:0/1802326253 shutdown_connections 2021-04-09T18:29:24.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.260 7facddd6a700 1 -- 172.21.15.32:0/1802326253 wait complete. 2021-04-09T18:29:24.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:29:24.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.433 7f1a8d582700 1 Processor -- start 2021-04-09T18:29:24.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.433 7f1a8d582700 1 -- start start 2021-04-09T18:29:24.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a8d582700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a8810d050 0x7f1a880cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a8d582700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1a880d0160 0x7f1a880d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a8d582700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8810d500 con 0x7f1a880d82e0 2021-04-09T18:29:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a8d582700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a880d9260 con 0x7f1a8810d050 2021-04-09T18:29:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a8d582700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a880d86e0 con 0x7f1a880d0160 2021-04-09T18:29:24.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a872bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a8810d050 0x7f1a880cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a87ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1a880d82e0 0x7f1a8810da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36176/0 (socket says 172.21.15.32:36176) 2021-04-09T18:29:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a87ac0700 1 -- 172.21.15.32:0/3018394444 learned_addr learned my addr 172.21.15.32:0/3018394444 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a86abe700 1 --2- 172.21.15.32:0/3018394444 >> v2:172.21.15.86:3300/0 conn(0x7f1a880d0160 0x7f1a880d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a872bf700 1 -- 172.21.15.32:0/3018394444 >> v2:172.21.15.86:3300/0 conn(0x7f1a880d0160 msgr2=0x7f1a880d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a872bf700 1 --2- 172.21.15.32:0/3018394444 >> v2:172.21.15.86:3300/0 conn(0x7f1a880d0160 0x7f1a880d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a872bf700 1 -- 172.21.15.32:0/3018394444 >> v1:172.21.15.32:6789/0 conn(0x7f1a880d82e0 legacy=0x7f1a8810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a862bd700 1 -- 172.21.15.32:0/3018394444 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612520926 0 0) 0x7f1a80000d30 con 0x7f1a880d82e0 2021-04-09T18:29:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a872bf700 1 -- 172.21.15.32:0/3018394444 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8810e270 con 0x7f1a8810d050 2021-04-09T18:29:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a872bf700 1 --2- 172.21.15.32:0/3018394444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8810d050 0x7f1a880cfb40 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 rx=0x7f1a78006650 tx=0x7f1a78009a00).ready entity=mon.0 client_cookie=5184fa6d1c9784b9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:24.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.434 7f1a862bd700 1 -- 172.21.15.32:0/3018394444 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a78009df0 con 0x7f1a8810d050 2021-04-09T18:29:24.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a862bd700 1 -- 172.21.15.32:0/3018394444 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a780009b0 con 0x7f1a8810d050 2021-04-09T18:29:24.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a862bd700 1 -- 172.21.15.32:0/3018394444 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a7800c2b0 con 0x7f1a8810d050 2021-04-09T18:29:24.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a8d582700 1 -- 172.21.15.32:0/3018394444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8810d050 msgr2=0x7f1a880cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a8d582700 1 --2- 172.21.15.32:0/3018394444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8810d050 0x7f1a880cfb40 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 rx=0x7f1a78006650 tx=0x7f1a78009a00).stop 2021-04-09T18:29:24.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a8d582700 1 -- 172.21.15.32:0/3018394444 shutdown_connections 2021-04-09T18:29:24.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a8d582700 1 --2- 172.21.15.32:0/3018394444 >> v2:172.21.15.86:3300/0 conn(0x7f1a880d0160 0x7f1a880d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a8d582700 1 --2- 172.21.15.32:0/3018394444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8810d050 0x7f1a880cfb40 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a8d582700 1 -- 172.21.15.32:0/3018394444 >> 172.21.15.32:0/3018394444 conn(0x7f1a880ce620 msgr2=0x7f1a880d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:24.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a8d582700 1 -- 172.21.15.32:0/3018394444 shutdown_connections 2021-04-09T18:29:24.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.435 7f1a8d582700 1 -- 172.21.15.32:0/3018394444 wait complete. 2021-04-09T18:29:24.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.455 7f1a8d582700 1 Processor -- start 2021-04-09T18:29:24.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a8d582700 1 -- start start 2021-04-09T18:29:24.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a8d582700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a880a1540 0x7f1a880a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a8d582700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a880a2840 0x7f1a88094550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a8d582700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a88094b70 0x7f1a88094f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a8d582700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880a1ca0 con 0x7f1a88094b70 2021-04-09T18:29:24.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a8d582700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a88095590 con 0x7f1a880a2840 2021-04-09T18:29:24.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a8d582700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880a6490 con 0x7f1a880a1540 2021-04-09T18:29:24.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a87ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a88094b70 0x7f1a88094f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a87ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a88094b70 0x7f1a88094f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54824/0 (socket says 172.21.15.32:54824) 2021-04-09T18:29:24.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a87ac0700 1 -- 172.21.15.32:0/2321603608 learned_addr learned my addr 172.21.15.32:0/2321603608 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:24.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a86abe700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a880a2840 0x7f1a88094550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.456 7f1a872bf700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a880a1540 0x7f1a880a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a87ac0700 1 -- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a880a1540 msgr2=0x7f1a880a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a87ac0700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a880a1540 0x7f1a880a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a87ac0700 1 -- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a880a2840 msgr2=0x7f1a88094550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a87ac0700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a880a2840 0x7f1a88094550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a87ac0700 1 -- 172.21.15.32:0/2321603608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a880a2c40 con 0x7f1a88094b70 2021-04-09T18:29:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a86abe700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a880a2840 0x7f1a88094550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a87ac0700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a88094b70 0x7f1a88094f70 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7f1a80007f90 tx=0x7f1a8000abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:24.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a6ffff700 1 -- 172.21.15.32:0/2321603608 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a80000cc0 con 0x7f1a88094b70 2021-04-09T18:29:24.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a6ffff700 1 -- 172.21.15.32:0/2321603608 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a800070d0 con 0x7f1a88094b70 2021-04-09T18:29:24.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a880a2c40 con 0x7f1a88094b70 2021-04-09T18:29:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a6ffff700 1 -- 172.21.15.32:0/2321603608 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a80007520 con 0x7f1a88094b70 2021-04-09T18:29:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.457 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a880a2c40 con 0x7f1a88094b70 2021-04-09T18:29:24.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.458 7f1a6ffff700 1 -- 172.21.15.32:0/2321603608 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1a800072b0 con 0x7f1a88094b70 2021-04-09T18:29:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.458 7f1a6ffff700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a680188d0 0x7f1a6801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.458 7f1a6ffff700 1 -- 172.21.15.32:0/2321603608 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f1a80032cc0 con 0x7f1a88094b70 2021-04-09T18:29:24.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.458 7f1a872bf700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a680188d0 0x7f1a6801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.458 7f1a872bf700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a680188d0 0x7f1a6801add0 crc :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:24.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.478 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a881b7180 con 0x7f1a88094b70 2021-04-09T18:29:24.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.482 7f1a6ffff700 1 -- 172.21.15.32:0/2321603608 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1a80007d00 con 0x7f1a88094b70 2021-04-09T18:29:24.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.666 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f1a8810d500 con 0x7f1a88094b70 2021-04-09T18:29:24.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.670 7f1a6ffff700 1 -- 172.21.15.32:0/2321603608 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 28 and data pool 29 v308) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1a80007290 con 0x7f1a88094b70 2021-04-09T18:29:24.757 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 28 and data pool 29 2021-04-09T18:29:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.676 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a680188d0 msgr2=0x7f1a6801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.676 7f1a8d582700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a680188d0 0x7f1a6801add0 crc :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.676 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a88094b70 msgr2=0x7f1a88094f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.676 7f1a8d582700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a88094b70 0x7f1a88094f70 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7f1a80007f90 tx=0x7f1a8000abd0).stop 2021-04-09T18:29:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.677 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 shutdown_connections 2021-04-09T18:29:24.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.677 7f1a8d582700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a680188d0 0x7f1a6801add0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.677 7f1a8d582700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a88094b70 0x7f1a88094f70 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.677 7f1a8d582700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a880a2840 0x7f1a88094550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.677 7f1a8d582700 1 --2- 172.21.15.32:0/2321603608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a880a1540 0x7f1a880a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.677 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 >> 172.21.15.32:0/2321603608 conn(0x7f1a880b92e0 msgr2=0x7f1a880af810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:24.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.677 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 shutdown_connections 2021-04-09T18:29:24.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.677 7f1a8d582700 1 -- 172.21.15.32:0/2321603608 wait complete. 2021-04-09T18:29:24.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:29:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.836 7fbfbcb57700 1 Processor -- start 2021-04-09T18:29:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.836 7fbfbcb57700 1 -- start start 2021-04-09T18:29:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfbcb57700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbfb810da00 0x7fbfb810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfbcb57700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb810d480 con 0x7fbfb810d010 2021-04-09T18:29:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfbcb57700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb80d1f70 con 0x7fbfb80d7ca0 2021-04-09T18:29:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfbcb57700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb80d8d30 con 0x7fbfb810da00 2021-04-09T18:29:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfb705f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbfb810da00 0x7fbfb810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfb685e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbfb810d010 0x7fbfb80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36196/0 (socket says 172.21.15.32:36196) 2021-04-09T18:29:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfb685e700 1 -- 172.21.15.32:0/906454690 learned_addr learned my addr 172.21.15.32:0/906454690 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfb705f700 1 -- 172.21.15.32:0/906454690 >> v1:172.21.15.86:6789/0 conn(0x7fbfb80d7ca0 legacy=0x7fbfb80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfb705f700 1 -- 172.21.15.32:0/906454690 >> v1:172.21.15.32:6789/0 conn(0x7fbfb810d010 legacy=0x7fbfb80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfb705f700 1 -- 172.21.15.32:0/906454690 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb80d9220 con 0x7fbfb810da00 2021-04-09T18:29:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfb585c700 1 -- 172.21.15.32:0/906454690 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267454084 0 0) 0x7fbfa8000ce0 con 0x7fbfb810d010 2021-04-09T18:29:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfb705f700 1 --2- 172.21.15.32:0/906454690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb810da00 0x7fbfb810de00 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7fbfb0006ab0 tx=0x7fbfb0009b40).ready entity=mon.0 client_cookie=aaf1a83992f2c9a8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:24.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.837 7fbfb585c700 1 -- 172.21.15.32:0/906454690 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfb00022e0 con 0x7fbfb810da00 2021-04-09T18:29:24.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.838 7fbfb585c700 1 -- 172.21.15.32:0/906454690 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb0002660 con 0x7fbfb810da00 2021-04-09T18:29:24.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.838 7fbfb585c700 1 -- 172.21.15.32:0/906454690 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfb0000c10 con 0x7fbfb810da00 2021-04-09T18:29:24.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.838 7fbfbcb57700 1 -- 172.21.15.32:0/906454690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb810da00 msgr2=0x7fbfb810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.838 7fbfbcb57700 1 --2- 172.21.15.32:0/906454690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb810da00 0x7fbfb810de00 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7fbfb0006ab0 tx=0x7fbfb0009b40).stop 2021-04-09T18:29:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.838 7fbfbcb57700 1 -- 172.21.15.32:0/906454690 shutdown_connections 2021-04-09T18:29:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.838 7fbfbcb57700 1 --2- 172.21.15.32:0/906454690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb810da00 0x7fbfb810de00 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.838 7fbfbcb57700 1 -- 172.21.15.32:0/906454690 >> 172.21.15.32:0/906454690 conn(0x7fbfb80ce5f0 msgr2=0x7fbfb80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.838 7fbfbcb57700 1 -- 172.21.15.32:0/906454690 shutdown_connections 2021-04-09T18:29:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.838 7fbfbcb57700 1 -- 172.21.15.32:0/906454690 wait complete. 2021-04-09T18:29:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfbcb57700 1 Processor -- start 2021-04-09T18:29:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfbcb57700 1 -- start start 2021-04-09T18:29:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfbcb57700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfb81a9270 0x7fbfb81a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfbcb57700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfb81923d0 0x7fbfb81927d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfbcb57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb8192df0 0x7fbfb8092ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfbcb57700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80932c0 con 0x7fbfb8192df0 2021-04-09T18:29:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfbcb57700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80934b0 con 0x7fbfb81923d0 2021-04-09T18:29:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfbcb57700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80936d0 con 0x7fbfb81a9270 2021-04-09T18:29:24.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfb705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb8192df0 0x7fbfb8092ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfb705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb8192df0 0x7fbfb8092ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54848/0 (socket says 172.21.15.32:54848) 2021-04-09T18:29:24.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfb705f700 1 -- 172.21.15.32:0/723795718 learned_addr learned my addr 172.21.15.32:0/723795718 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:24.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfb605d700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfb81923d0 0x7fbfb81927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.859 7fbfb685e700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfb81a9270 0x7fbfb81a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfb705f700 1 -- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfb81a9270 msgr2=0x7fbfb81a9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfb705f700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfb81a9270 0x7fbfb81a9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfb705f700 1 -- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfb81923d0 msgr2=0x7fbfb81927d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:24.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfb705f700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfb81923d0 0x7fbfb81927d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:24.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfb705f700 1 -- 172.21.15.32:0/723795718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb80938f0 con 0x7fbfb8192df0 2021-04-09T18:29:24.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfb685e700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfb81a9270 0x7fbfb81a9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:24.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfb705f700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb8192df0 0x7fbfb8092ca0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7fbfb000b5f0 tx=0x7fbfb000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:24.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfa77fe700 1 -- 172.21.15.32:0/723795718 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfb00093e0 con 0x7fbfb8192df0 2021-04-09T18:29:24.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfa77fe700 1 -- 172.21.15.32:0/723795718 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb000bd80 con 0x7fbfb8192df0 2021-04-09T18:29:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfa77fe700 1 -- 172.21.15.32:0/723795718 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfb00062a0 con 0x7fbfb8192df0 2021-04-09T18:29:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb80938f0 con 0x7fbfb8192df0 2021-04-09T18:29:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.860 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb80938f0 con 0x7fbfb8192df0 2021-04-09T18:29:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.861 7fbfa77fe700 1 -- 172.21.15.32:0/723795718 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbfb000a000 con 0x7fbfb8192df0 2021-04-09T18:29:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.861 7fbfa77fe700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf9c0188c0 0x7fbf9c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.861 7fbfa77fe700 1 -- 172.21.15.32:0/723795718 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbfb0032a10 con 0x7fbfb8192df0 2021-04-09T18:29:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.861 7fbfb685e700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf9c0188c0 0x7fbf9c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:24.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.861 7fbfb685e700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf9c0188c0 0x7fbf9c01adc0 crc :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:24.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.881 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb817cab0 con 0x7fbfb8192df0 2021-04-09T18:29:24.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:24.884 7fbfa77fe700 1 -- 172.21.15.32:0/723795718 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbfb0006820 con 0x7fbfb8192df0 2021-04-09T18:29:25.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.075 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfb817cab0 con 0x7fbfb8192df0 2021-04-09T18:29:25.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.076 7fbfa77fe700 1 -- 172.21.15.32:0/723795718 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fbfb000a000 con 0x7fbfb8192df0 2021-04-09T18:29:25.161 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:25.161 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":167,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:29:24.667716","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:29:22.005467","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-04-09 18:29:23.428510","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:29:25.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf9c0188c0 msgr2=0x7fbf9c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf9c0188c0 0x7fbf9c01adc0 crc :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb8192df0 msgr2=0x7fbfb8092ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb8192df0 0x7fbfb8092ca0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7fbfb000b5f0 tx=0x7fbfb000b990).stop 2021-04-09T18:29:25.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 shutdown_connections 2021-04-09T18:29:25.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf9c0188c0 0x7fbf9c01adc0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfb8192df0 0x7fbfb8092ca0 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfb81923d0 0x7fbfb81927d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 --2- 172.21.15.32:0/723795718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfb81a9270 0x7fbfb81a9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 >> 172.21.15.32:0/723795718 conn(0x7fbfb8094780 msgr2=0x7fbfb81ac930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:25.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 shutdown_connections 2021-04-09T18:29:25.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.078 7fbfbcb57700 1 -- 172.21.15.32:0/723795718 wait complete. 2021-04-09T18:29:25.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:29:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.236 7fb89f860700 1 Processor -- start 2021-04-09T18:29:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.236 7fb89f860700 1 -- start start 2021-04-09T18:29:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb89f860700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980d7ca0 0x7fb8980d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb89f860700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89810d480 con 0x7fb89810d010 2021-04-09T18:29:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb89f860700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8980d86c0 con 0x7fb89810da00 2021-04-09T18:29:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb89f860700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980d9220 con 0x7fb8980d7ca0 2021-04-09T18:29:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb89d67c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb89810d010 0x7fb8980d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36210/0 (socket says 172.21.15.32:36210) 2021-04-09T18:29:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb89d67c700 1 -- 172.21.15.32:0/30740911 learned_addr learned my addr 172.21.15.32:0/30740911 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb89ce7b700 1 --2- 172.21.15.32:0/30740911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980d7ca0 0x7fb8980d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb88ffff700 1 -- 172.21.15.32:0/30740911 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289107845 0 0) 0x7fb894000ce0 con 0x7fb89810d010 2021-04-09T18:29:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb88ffff700 1 -- 172.21.15.32:0/30740911 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb884002fe0 con 0x7fb89810d010 2021-04-09T18:29:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb88ffff700 1 -- 172.21.15.32:0/30740911 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821927596 0 0) 0x7fb890000d30 con 0x7fb89810da00 2021-04-09T18:29:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.237 7fb88ffff700 1 -- 172.21.15.32:0/30740911 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8840042c0 con 0x7fb89810da00 2021-04-09T18:29:25.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.238 7fb89ce7b700 1 -- 172.21.15.32:0/30740911 >> v1:172.21.15.86:6789/0 conn(0x7fb89810da00 legacy=0x7fb89810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.238 7fb89ce7b700 1 -- 172.21.15.32:0/30740911 >> v1:172.21.15.32:6789/0 conn(0x7fb89810d010 legacy=0x7fb8980d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.238 7fb89ce7b700 1 -- 172.21.15.32:0/30740911 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980d1f70 con 0x7fb8980d7ca0 2021-04-09T18:29:25.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.238 7fb89ce7b700 1 --2- 172.21.15.32:0/30740911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980d7ca0 0x7fb8980d80a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb888006650 tx=0x7fb888009a80).ready entity=mon.2 client_cookie=f3c1556be49cc163 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.238 7fb88ffff700 1 -- 172.21.15.32:0/30740911 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8880009b0 con 0x7fb8980d7ca0 2021-04-09T18:29:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.238 7fb88ffff700 1 -- 172.21.15.32:0/30740911 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88800c0c0 con 0x7fb8980d7ca0 2021-04-09T18:29:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.238 7fb88ffff700 1 -- 172.21.15.32:0/30740911 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb88800c510 con 0x7fb8980d7ca0 2021-04-09T18:29:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.238 7fb89f860700 1 -- 172.21.15.32:0/30740911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980d7ca0 msgr2=0x7fb8980d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.238 7fb89f860700 1 --2- 172.21.15.32:0/30740911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980d7ca0 0x7fb8980d80a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb888006650 tx=0x7fb888009a80).stop 2021-04-09T18:29:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.239 7fb89f860700 1 -- 172.21.15.32:0/30740911 shutdown_connections 2021-04-09T18:29:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.239 7fb89f860700 1 --2- 172.21.15.32:0/30740911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980d7ca0 0x7fb8980d80a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.239 7fb89f860700 1 -- 172.21.15.32:0/30740911 >> 172.21.15.32:0/30740911 conn(0x7fb8980ce5f0 msgr2=0x7fb8980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.239 7fb89f860700 1 -- 172.21.15.32:0/30740911 shutdown_connections 2021-04-09T18:29:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.239 7fb89f860700 1 -- 172.21.15.32:0/30740911 wait complete. 2021-04-09T18:29:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.259 7fb89f860700 1 Processor -- start 2021-04-09T18:29:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.259 7fb89f860700 1 -- start start 2021-04-09T18:29:25.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89f860700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980b42d0 0x7fb898183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89f860700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8981ac090 0x7fb8981ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89f860700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb898184d30 0x7fb898185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89f860700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980a1a90 con 0x7fb8981ac090 2021-04-09T18:29:25.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89f860700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981acab0 con 0x7fb898184d30 2021-04-09T18:29:25.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89f860700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb898185a00 con 0x7fb8980b42d0 2021-04-09T18:29:25.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89ce7b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8981ac090 0x7fb8981ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89de7d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb898184d30 0x7fb898185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89d67c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980b42d0 0x7fb898183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89ce7b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8981ac090 0x7fb8981ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54862/0 (socket says 172.21.15.32:54862) 2021-04-09T18:29:25.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.260 7fb89ce7b700 1 -- 172.21.15.32:0/820874201 learned_addr learned my addr 172.21.15.32:0/820874201 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:25.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89ce7b700 1 -- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980b42d0 msgr2=0x7fb898183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89ce7b700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980b42d0 0x7fb898183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89ce7b700 1 -- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb898184d30 msgr2=0x7fb898185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89ce7b700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb898184d30 0x7fb898185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89ce7b700 1 -- 172.21.15.32:0/820874201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb898185750 con 0x7fb8981ac090 2021-04-09T18:29:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89d67c700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980b42d0 0x7fb898183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:25.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89de7d700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb898184d30 0x7fb898185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:25.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89ce7b700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8981ac090 0x7fb8981ac490 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7fb888009650 tx=0x7fb88800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:25.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb88dffb700 1 -- 172.21.15.32:0/820874201 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8880093a0 con 0x7fb8981ac090 2021-04-09T18:29:25.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb88dffb700 1 -- 172.21.15.32:0/820874201 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88800bd50 con 0x7fb8981ac090 2021-04-09T18:29:25.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89f860700 1 -- 172.21.15.32:0/820874201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb898185750 con 0x7fb8981ac090 2021-04-09T18:29:25.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb88dffb700 1 -- 172.21.15.32:0/820874201 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8880052e0 con 0x7fb8981ac090 2021-04-09T18:29:25.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.261 7fb89f860700 1 -- 172.21.15.32:0/820874201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb898185750 con 0x7fb8981ac090 2021-04-09T18:29:25.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.262 7fb88dffb700 1 -- 172.21.15.32:0/820874201 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb888006030 con 0x7fb8981ac090 2021-04-09T18:29:25.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.262 7fb88dffb700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb878018930 0x7fb87801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.262 7fb88dffb700 1 -- 172.21.15.32:0/820874201 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb8880329e0 con 0x7fb8981ac090 2021-04-09T18:29:25.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.262 7fb89d67c700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb878018930 0x7fb87801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.262 7fb89d67c700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb878018930 0x7fb87801ae10 crc :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:25.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.281 7fb89f860700 1 -- 172.21.15.32:0/820874201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb898182d10 con 0x7fb8981ac090 2021-04-09T18:29:25.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.285 7fb88dffb700 1 -- 172.21.15.32:0/820874201 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb8880009c0 con 0x7fb8981ac090 2021-04-09T18:29:25.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.474 7fb89f860700 1 -- 172.21.15.32:0/820874201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8980a1a90 con 0x7fb8981ac090 2021-04-09T18:29:25.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.475 7fb88dffb700 1 -- 172.21.15.32:0/820874201 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fb888006030 con 0x7fb8981ac090 2021-04-09T18:29:25.560 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:25.561 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":167,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:29:24.667716","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:29:22.005467","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-04-09 18:29:23.428510","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:29:25.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 -- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb878018930 msgr2=0x7fb87801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb878018930 0x7fb87801ae10 crc :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 -- 172.21.15.32:0/820874201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8981ac090 msgr2=0x7fb8981ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8981ac090 0x7fb8981ac490 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7fb888009650 tx=0x7fb88800b960).stop 2021-04-09T18:29:25.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 -- 172.21.15.32:0/820874201 shutdown_connections 2021-04-09T18:29:25.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb878018930 0x7fb87801ae10 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8981ac090 0x7fb8981ac490 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb898184d30 0x7fb898185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 --2- 172.21.15.32:0/820874201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8980b42d0 0x7fb898183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 -- 172.21.15.32:0/820874201 >> 172.21.15.32:0/820874201 conn(0x7fb89819b890 msgr2=0x7fb8981a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:25.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 -- 172.21.15.32:0/820874201 shutdown_connections 2021-04-09T18:29:25.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.477 7fb89f860700 1 -- 172.21.15.32:0/820874201 wait complete. 2021-04-09T18:29:25.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:29:25.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.642 7f196e4c5700 1 Processor -- start 2021-04-09T18:29:25.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.642 7f196e4c5700 1 -- start start 2021-04-09T18:29:25.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.642 7f196e4c5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19680cffc0 0x7f19680d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.642 7f196e4c5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19680d6ca0 0x7f196810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.642 7f196e4c5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19680d72a0 con 0x7f19680d7820 2021-04-09T18:29:25.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.642 7f196e4c5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19680d70a0 con 0x7f19680cffc0 2021-04-09T18:29:25.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.642 7f196e4c5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19680d9200 con 0x7f19680d6ca0 2021-04-09T18:29:25.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.642 7f1967fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19680cffc0 0x7f19680d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f1967fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19680cffc0 0x7f19680d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54866/0 (socket says 172.21.15.32:54866) 2021-04-09T18:29:25.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f1967fff700 1 -- 172.21.15.32:0/2415296633 learned_addr learned my addr 172.21.15.32:0/2415296633 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:25.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f196cae2700 1 --2- 172.21.15.32:0/2415296633 >> v2:172.21.15.86:3300/0 conn(0x7f19680d6ca0 0x7f196810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f1967fff700 1 -- 172.21.15.32:0/2415296633 >> v2:172.21.15.86:3300/0 conn(0x7f19680d6ca0 msgr2=0x7f196810cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:25.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f1967fff700 1 --2- 172.21.15.32:0/2415296633 >> v2:172.21.15.86:3300/0 conn(0x7f19680d6ca0 0x7f196810cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f1967fff700 1 -- 172.21.15.32:0/2415296633 >> v1:172.21.15.86:6789/0 conn(0x7f19680d7820 legacy=0x7f19680d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f1967fff700 1 -- 172.21.15.32:0/2415296633 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19680d8410 con 0x7f19680cffc0 2021-04-09T18:29:25.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f1967fff700 1 --2- 172.21.15.32:0/2415296633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19680cffc0 0x7f19680d2460 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f1958006700 tx=0x7f1958009b80).ready entity=mon.0 client_cookie=5f5ff8a32dfaf231 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:25.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f19677fe700 1 -- 172.21.15.32:0/2415296633 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195800c0c0 con 0x7f19680cffc0 2021-04-09T18:29:25.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f19677fe700 1 -- 172.21.15.32:0/2415296633 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195800c440 con 0x7f19680cffc0 2021-04-09T18:29:25.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f19677fe700 1 -- 172.21.15.32:0/2415296633 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195800c920 con 0x7f19680cffc0 2021-04-09T18:29:25.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f196e4c5700 1 -- 172.21.15.32:0/2415296633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19680cffc0 msgr2=0x7f19680d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.643 7f196e4c5700 1 --2- 172.21.15.32:0/2415296633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19680cffc0 0x7f19680d2460 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f1958006700 tx=0x7f1958009b80).stop 2021-04-09T18:29:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.644 7f196e4c5700 1 -- 172.21.15.32:0/2415296633 shutdown_connections 2021-04-09T18:29:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.644 7f196e4c5700 1 --2- 172.21.15.32:0/2415296633 >> v2:172.21.15.86:3300/0 conn(0x7f19680d6ca0 0x7f196810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.644 7f196e4c5700 1 --2- 172.21.15.32:0/2415296633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19680cffc0 0x7f19680d2460 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.644 7f196e4c5700 1 -- 172.21.15.32:0/2415296633 >> 172.21.15.32:0/2415296633 conn(0x7f19680ce5e0 msgr2=0x7f19680cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:25.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.644 7f196e4c5700 1 -- 172.21.15.32:0/2415296633 shutdown_connections 2021-04-09T18:29:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.644 7f196e4c5700 1 -- 172.21.15.32:0/2415296633 wait complete. 2021-04-09T18:29:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.664 7f196e4c5700 1 Processor -- start 2021-04-09T18:29:25.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.664 7f196e4c5700 1 -- start start 2021-04-09T18:29:25.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196e4c5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196809ebc0 0x7f196809adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196e4c5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1968183050 0x7f196819ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196e4c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1968183450 0x7f19680952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196e4c5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196809b660 con 0x7f1968183450 2021-04-09T18:29:25.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196e4c5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19681a05c0 con 0x7f1968183050 2021-04-09T18:29:25.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196e4c5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1968095900 con 0x7f196809ebc0 2021-04-09T18:29:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196cae2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1968183450 0x7f19680952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196cae2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1968183450 0x7f19680952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54876/0 (socket says 172.21.15.32:54876) 2021-04-09T18:29:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196cae2700 1 -- 172.21.15.32:0/2760175065 learned_addr learned my addr 172.21.15.32:0/2760175065 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:25.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f1967fff700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196809ebc0 0x7f196809adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f195ffff700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1968183050 0x7f196819ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196cae2700 1 -- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196809ebc0 msgr2=0x7f196809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196cae2700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196809ebc0 0x7f196809adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196cae2700 1 -- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1968183050 msgr2=0x7f196819ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:25.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196cae2700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1968183050 0x7f196819ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:25.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196cae2700 1 -- 172.21.15.32:0/2760175065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1968095b20 con 0x7f1968183450 2021-04-09T18:29:25.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f196cae2700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1968183450 0x7f19680952e0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f196000b910 tx=0x7f196000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:25.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.665 7f19657fa700 1 -- 172.21.15.32:0/2760175065 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1960006aa0 con 0x7f1968183450 2021-04-09T18:29:25.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.666 7f19657fa700 1 -- 172.21.15.32:0/2760175065 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19600011f0 con 0x7f1968183450 2021-04-09T18:29:25.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.666 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1968095b20 con 0x7f1968183450 2021-04-09T18:29:25.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.666 7f19657fa700 1 -- 172.21.15.32:0/2760175065 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1960001640 con 0x7f1968183450 2021-04-09T18:29:25.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.666 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1968095b20 con 0x7f1968183450 2021-04-09T18:29:25.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.666 7f19657fa700 1 -- 172.21.15.32:0/2760175065 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f19600013d0 con 0x7f1968183450 2021-04-09T18:29:25.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.666 7f19657fa700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f194c0188d0 0x7f194c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:25.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.666 7f19657fa700 1 -- 172.21.15.32:0/2760175065 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1960032dd0 con 0x7f1968183450 2021-04-09T18:29:25.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.666 7f1967fff700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f194c0188d0 0x7f194c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:25.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.667 7f1967fff700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f194c0188d0 0x7f194c01add0 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:25.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.686 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1968093df0 con 0x7f1968183450 2021-04-09T18:29:25.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.690 7f19657fa700 1 -- 172.21.15.32:0/2760175065 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f196000bdf0 con 0x7f1968183450 2021-04-09T18:29:25.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:25.884 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f19681a0580 con 0x7f1968183450 2021-04-09T18:29:26.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.678 7f19657fa700 1 -- 172.21.15.32:0/2760175065 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v311) v1 ==== 118+0+0 (secure 0 0 0) 0x7f19600011f0 con 0x7f1968183450 2021-04-09T18:29:26.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f194c0188d0 msgr2=0x7f194c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:26.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f194c0188d0 0x7f194c01add0 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1968183450 msgr2=0x7f19680952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:26.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1968183450 0x7f19680952e0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f196000b910 tx=0x7f196000bcb0).stop 2021-04-09T18:29:26.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 shutdown_connections 2021-04-09T18:29:26.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f194c0188d0 0x7f194c01add0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1968183450 0x7f19680952e0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1968183050 0x7f196819ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 --2- 172.21.15.32:0/2760175065 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196809ebc0 0x7f196809adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 >> 172.21.15.32:0/2760175065 conn(0x7f196809fa00 msgr2=0x7f19680b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:26.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 shutdown_connections 2021-04-09T18:29:26.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.703 7f196e4c5700 1 -- 172.21.15.32:0/2760175065 wait complete. 2021-04-09T18:29:26.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:26.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.863 7ff57e2a4700 1 Processor -- start 2021-04-09T18:29:26.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.863 7ff57e2a4700 1 -- start start 2021-04-09T18:29:26.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.864 7ff57e2a4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57810d010 0x7ff5780d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:26.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.864 7ff57e2a4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff5780d1da0 0x7ff5780d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:26.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.864 7ff57e2a4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57810d480 con 0x7ff5780d82c0 2021-04-09T18:29:26.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.864 7ff57e2a4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57810e780 con 0x7ff5780d1da0 2021-04-09T18:29:26.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.864 7ff57e2a4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5780d8d30 con 0x7ff57810d010 2021-04-09T18:29:26.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.864 7ff577fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57810d010 0x7ff5780d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:26.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.864 7ff56ffff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff5780d1da0 0x7ff5780d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:26.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.864 7ff57c8c1700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff5780d82c0 0x7ff57810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:34166/0 (socket says 172.21.15.32:34166) 2021-04-09T18:29:26.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.864 7ff57c8c1700 1 -- 172.21.15.32:0/399031507 learned_addr learned my addr 172.21.15.32:0/399031507 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:26.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff56ffff700 1 -- 172.21.15.32:0/399031507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57810d010 msgr2=0x7ff5780d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:26.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff56ffff700 1 --2- 172.21.15.32:0/399031507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57810d010 0x7ff5780d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff56ffff700 1 -- 172.21.15.32:0/399031507 >> v1:172.21.15.86:6789/0 conn(0x7ff5780d82c0 legacy=0x7ff57810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:26.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff56ffff700 1 -- 172.21.15.32:0/399031507 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5780d9220 con 0x7ff5780d1da0 2021-04-09T18:29:26.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff577fff700 1 --2- 172.21.15.32:0/399031507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57810d010 0x7ff5780d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:26.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff5777fe700 1 -- 172.21.15.32:0/399031507 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238859198 0 0) 0x7ff570001440 con 0x7ff5780d82c0 2021-04-09T18:29:26.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff56ffff700 1 --2- 172.21.15.32:0/399031507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5780d1da0 0x7ff5780d7ca0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7ff560006650 tx=0x7ff560009a80).ready entity=mon.1 client_cookie=7917dce3b6f04a5d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:26.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff5777fe700 1 -- 172.21.15.32:0/399031507 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5600009b0 con 0x7ff5780d1da0 2021-04-09T18:29:26.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff5777fe700 1 -- 172.21.15.32:0/399031507 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56000c0c0 con 0x7ff5780d1da0 2021-04-09T18:29:26.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff5777fe700 1 -- 172.21.15.32:0/399031507 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff56000c510 con 0x7ff5780d1da0 2021-04-09T18:29:26.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff57e2a4700 1 -- 172.21.15.32:0/399031507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5780d1da0 msgr2=0x7ff5780d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:26.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.865 7ff57e2a4700 1 --2- 172.21.15.32:0/399031507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5780d1da0 0x7ff5780d7ca0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7ff560006650 tx=0x7ff560009a80).stop 2021-04-09T18:29:26.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.866 7ff57e2a4700 1 -- 172.21.15.32:0/399031507 shutdown_connections 2021-04-09T18:29:26.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.866 7ff57e2a4700 1 --2- 172.21.15.32:0/399031507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5780d1da0 0x7ff5780d7ca0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.866 7ff57e2a4700 1 --2- 172.21.15.32:0/399031507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57810d010 0x7ff5780d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.866 7ff57e2a4700 1 -- 172.21.15.32:0/399031507 >> 172.21.15.32:0/399031507 conn(0x7ff5780ce5f0 msgr2=0x7ff5780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.866 7ff57e2a4700 1 -- 172.21.15.32:0/399031507 shutdown_connections 2021-04-09T18:29:26.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.866 7ff57e2a4700 1 -- 172.21.15.32:0/399031507 wait complete. 2021-04-09T18:29:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.886 7ff57e2a4700 1 Processor -- start 2021-04-09T18:29:26.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.886 7ff57e2a4700 1 -- start start 2021-04-09T18:29:26.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff57e2a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5781a9360 0x7ff5781918f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:26.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff57e2a4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57818c440 0x7ff57818c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:26.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff57e2a4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5781a80f0 0x7ff5781a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:26.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff57e2a4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff578191f10 con 0x7ff5781a9360 2021-04-09T18:29:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff57e2a4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5781a9760 con 0x7ff5781a80f0 2021-04-09T18:29:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff57e2a4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57818ce60 con 0x7ff57818c440 2021-04-09T18:29:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff577fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5781a9360 0x7ff5781918f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:26.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff577fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5781a9360 0x7ff5781918f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54886/0 (socket says 172.21.15.32:54886) 2021-04-09T18:29:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff577fff700 1 -- 172.21.15.32:0/1362960731 learned_addr learned my addr 172.21.15.32:0/1362960731 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff56ffff700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57818c440 0x7ff57818c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff57c8c1700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5781a80f0 0x7ff5781a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:26.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff577fff700 1 -- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57818c440 msgr2=0x7ff57818c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff577fff700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57818c440 0x7ff57818c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff577fff700 1 -- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5781a80f0 msgr2=0x7ff5781a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff577fff700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5781a80f0 0x7ff5781a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:26.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff577fff700 1 -- 172.21.15.32:0/1362960731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5781a8b10 con 0x7ff5781a9360 2021-04-09T18:29:26.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff577fff700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5781a9360 0x7ff5781918f0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7ff56800b580 tx=0x7ff56800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:26.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff5757fa700 1 -- 172.21.15.32:0/1362960731 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff56800bd10 con 0x7ff5781a9360 2021-04-09T18:29:26.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff5757fa700 1 -- 172.21.15.32:0/1362960731 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff568000c90 con 0x7ff5781a9360 2021-04-09T18:29:26.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5781a8b10 con 0x7ff5781a9360 2021-04-09T18:29:26.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff5757fa700 1 -- 172.21.15.32:0/1362960731 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff568002220 con 0x7ff5781a9360 2021-04-09T18:29:26.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.887 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5781a8b10 con 0x7ff5781a9360 2021-04-09T18:29:26.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.888 7ff5757fa700 1 -- 172.21.15.32:0/1362960731 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff568006b00 con 0x7ff5781a9360 2021-04-09T18:29:26.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.888 7ff5757fa700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff55c018930 0x7ff55c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:26.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.888 7ff5757fa700 1 -- 172.21.15.32:0/1362960731 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff5680329f0 con 0x7ff5781a9360 2021-04-09T18:29:26.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.888 7ff56ffff700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff55c018930 0x7ff55c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:26.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.889 7ff56ffff700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff55c018930 0x7ff55c01adc0 crc :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:26.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.908 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5781a9760 con 0x7ff5781a9360 2021-04-09T18:29:26.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:26.911 7ff5757fa700 1 -- 172.21.15.32:0/1362960731 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff56800bd10 con 0x7ff5781a9360 2021-04-09T18:29:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.106 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5781a9760 con 0x7ff5781a9360 2021-04-09T18:29:27.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.106 7ff5757fa700 1 -- 172.21.15.32:0/1362960731 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 311 v311) v1 ==== 95+0+2788 (secure 0 0 0) 0x7ff568000c90 con 0x7ff5781a9360 2021-04-09T18:29:27.191 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 311 2021-04-09T18:29:27.192 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:27.192 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":311,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2532077375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2532077375},{"type":"v1","addr":"172.21.15.32:6834","nonce":2532077375}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":311,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:29:26.675661","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10177":{"gid":10177,"name":"a","rank":0,"incarnation":308,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2200487686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2200487686},{"type":"v1","addr":"172.21.15.32:6835","nonce":2200487686}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-04-09T18:29:27.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff55c018930 msgr2=0x7ff55c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff55c018930 0x7ff55c01adc0 crc :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5781a9360 msgr2=0x7ff5781918f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5781a9360 0x7ff5781918f0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7ff56800b580 tx=0x7ff56800b920).stop 2021-04-09T18:29:27.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 shutdown_connections 2021-04-09T18:29:27.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff55c018930 0x7ff55c01adc0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5781a80f0 0x7ff5781a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff57818c440 0x7ff57818c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 --2- 172.21.15.32:0/1362960731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5781a9360 0x7ff5781918f0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 >> 172.21.15.32:0/1362960731 conn(0x7ff5781837d0 msgr2=0x7ff578195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:27.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 shutdown_connections 2021-04-09T18:29:27.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.109 7ff57e2a4700 1 -- 172.21.15.32:0/1362960731 wait complete. 2021-04-09T18:29:27.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:29:27.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.269 7f520a466700 1 Processor -- start 2021-04-09T18:29:27.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.269 7f520a466700 1 -- start start 2021-04-09T18:29:27.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f520a466700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52040cffa0 0x7f52040d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f520a466700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52040d7800 0x7f52040d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f520a466700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f52040d5c70 0x7f52040d6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f520a466700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52040d8580 con 0x7f52040d7800 2021-04-09T18:29:27.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f520a466700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52040d91e0 con 0x7f52040d5c70 2021-04-09T18:29:27.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f520a466700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52040d7280 con 0x7f52040cffa0 2021-04-09T18:29:27.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f51fbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52040d7800 0x7f52040d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f51fbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f52040d7800 0x7f52040d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54898/0 (socket says 172.21.15.32:54898) 2021-04-09T18:29:27.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f51fbfff700 1 -- 172.21.15.32:0/2083047245 learned_addr learned my addr 172.21.15.32:0/2083047245 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:27.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f5203fff700 1 --2- 172.21.15.32:0/2083047245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52040cffa0 0x7f52040d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.270 7f5208a83700 1 --2- 172.21.15.32:0/2083047245 >> v2:172.21.15.86:3300/0 conn(0x7f52040d5c70 0x7f52040d6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f51fbfff700 1 -- 172.21.15.32:0/2083047245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52040cffa0 msgr2=0x7f52040d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f51fbfff700 1 --2- 172.21.15.32:0/2083047245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52040cffa0 0x7f52040d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f51fbfff700 1 -- 172.21.15.32:0/2083047245 >> v2:172.21.15.86:3300/0 conn(0x7f52040d5c70 msgr2=0x7f52040d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f51fbfff700 1 --2- 172.21.15.32:0/2083047245 >> v2:172.21.15.86:3300/0 conn(0x7f52040d5c70 0x7f52040d6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f51fbfff700 1 -- 172.21.15.32:0/2083047245 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52040d8220 con 0x7f52040d7800 2021-04-09T18:29:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f51fbfff700 1 --2- 172.21.15.32:0/2083047245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52040d7800 0x7f52040d7c00 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 rx=0x7f51ec006700 tx=0x7f51ec009b00).ready entity=mon.0 client_cookie=6137f99cbeeae8b4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f52037fe700 1 -- 172.21.15.32:0/2083047245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51ec00c0c0 con 0x7f52040d7800 2021-04-09T18:29:27.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f52037fe700 1 -- 172.21.15.32:0/2083047245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec00c400 con 0x7f52040d7800 2021-04-09T18:29:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f52037fe700 1 -- 172.21.15.32:0/2083047245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51ec00c8e0 con 0x7f52040d7800 2021-04-09T18:29:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f520a466700 1 -- 172.21.15.32:0/2083047245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52040d7800 msgr2=0x7f52040d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.271 7f520a466700 1 --2- 172.21.15.32:0/2083047245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52040d7800 0x7f52040d7c00 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 rx=0x7f51ec006700 tx=0x7f51ec009b00).stop 2021-04-09T18:29:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.272 7f520a466700 1 -- 172.21.15.32:0/2083047245 shutdown_connections 2021-04-09T18:29:27.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.272 7f520a466700 1 --2- 172.21.15.32:0/2083047245 >> v2:172.21.15.86:3300/0 conn(0x7f52040d5c70 0x7f52040d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.272 7f520a466700 1 --2- 172.21.15.32:0/2083047245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f52040d7800 0x7f52040d7c00 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.272 7f520a466700 1 --2- 172.21.15.32:0/2083047245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52040cffa0 0x7f52040d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.272 7f520a466700 1 -- 172.21.15.32:0/2083047245 >> 172.21.15.32:0/2083047245 conn(0x7f52040ce5c0 msgr2=0x7f52040cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:27.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.272 7f520a466700 1 -- 172.21.15.32:0/2083047245 shutdown_connections 2021-04-09T18:29:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.272 7f520a466700 1 -- 172.21.15.32:0/2083047245 wait complete. 2021-04-09T18:29:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.292 7f520a466700 1 Processor -- start 2021-04-09T18:29:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.292 7f520a466700 1 -- start start 2021-04-09T18:29:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f520a466700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52041acff0 0x7f520408fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f520a466700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5204090150 0x7f520408c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f520a466700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520408cca0 0x7f520408e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f520a466700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520408c100 con 0x7f5204090150 2021-04-09T18:29:27.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f520a466700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520408d0a0 con 0x7f520408cca0 2021-04-09T18:29:27.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f520a466700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520408e960 con 0x7f52041acff0 2021-04-09T18:29:27.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f51fbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5204090150 0x7f520408c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f51fbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5204090150 0x7f520408c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54900/0 (socket says 172.21.15.32:54900) 2021-04-09T18:29:27.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f51fbfff700 1 -- 172.21.15.32:0/798051060 learned_addr learned my addr 172.21.15.32:0/798051060 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:27.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f5203fff700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52041acff0 0x7f520408fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f5208a83700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520408cca0 0x7f520408e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f51fbfff700 1 -- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52041acff0 msgr2=0x7f520408fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f51fbfff700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52041acff0 0x7f520408fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f51fbfff700 1 -- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520408cca0 msgr2=0x7f520408e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f51fbfff700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520408cca0 0x7f520408e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f51fbfff700 1 -- 172.21.15.32:0/798051060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520408eb80 con 0x7f5204090150 2021-04-09T18:29:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f51fbfff700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5204090150 0x7f520408c680 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7f51ec006f20 tx=0x7f51ec00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f52017fa700 1 -- 172.21.15.32:0/798051060 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51ec00bdd0 con 0x7f5204090150 2021-04-09T18:29:27.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f52017fa700 1 -- 172.21.15.32:0/798051060 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec006440 con 0x7f5204090150 2021-04-09T18:29:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f520a466700 1 -- 172.21.15.32:0/798051060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520408eb80 con 0x7f5204090150 2021-04-09T18:29:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f52017fa700 1 -- 172.21.15.32:0/798051060 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51ec0052e0 con 0x7f5204090150 2021-04-09T18:29:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.293 7f520a466700 1 -- 172.21.15.32:0/798051060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520408eb80 con 0x7f5204090150 2021-04-09T18:29:27.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.294 7f52017fa700 1 -- 172.21.15.32:0/798051060 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f51ec0052e0 con 0x7f5204090150 2021-04-09T18:29:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.294 7f52017fa700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51e80188c0 0x7f51e801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.294 7f52017fa700 1 -- 172.21.15.32:0/798051060 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f51ec032a00 con 0x7f5204090150 2021-04-09T18:29:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.294 7f5203fff700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51e80188c0 0x7f51e801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.295 7f5203fff700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51e80188c0 0x7f51e801adc0 crc :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:27.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.314 7f520a466700 1 -- 172.21.15.32:0/798051060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5204182dc0 con 0x7f5204090150 2021-04-09T18:29:27.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.318 7f52017fa700 1 -- 172.21.15.32:0/798051060 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f51ec006440 con 0x7f5204090150 2021-04-09T18:29:27.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.506 7f520a466700 1 -- 172.21.15.32:0/798051060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52040d7280 con 0x7f5204090150 2021-04-09T18:29:27.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.507 7f52017fa700 1 -- 172.21.15.32:0/798051060 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f51ec006030 con 0x7f5204090150 2021-04-09T18:29:27.591 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:27.591 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":168,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:29:25.668831","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:29:22.005467","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-04-09 18:29:23.428510","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:29:27.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 -- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51e80188c0 msgr2=0x7f51e801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51e80188c0 0x7f51e801adc0 crc :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 -- 172.21.15.32:0/798051060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5204090150 msgr2=0x7f520408c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5204090150 0x7f520408c680 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7f51ec006f20 tx=0x7f51ec00b9e0).stop 2021-04-09T18:29:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 -- 172.21.15.32:0/798051060 shutdown_connections 2021-04-09T18:29:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f51e80188c0 0x7f51e801adc0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5204090150 0x7f520408c680 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520408cca0 0x7f520408e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 --2- 172.21.15.32:0/798051060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52041acff0 0x7f520408fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 -- 172.21.15.32:0/798051060 >> 172.21.15.32:0/798051060 conn(0x7f520409abb0 msgr2=0x7f52040a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:27.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 -- 172.21.15.32:0/798051060 shutdown_connections 2021-04-09T18:29:27.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.509 7f520a466700 1 -- 172.21.15.32:0/798051060 wait complete. 2021-04-09T18:29:27.604 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:29:27.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.682 7fcd506f5700 1 Processor -- start 2021-04-09T18:29:27.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.682 7fcd506f5700 1 -- start start 2021-04-09T18:29:27.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd506f5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcd480d7c50 0x7fcd481b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd506f5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd480d8620 con 0x7fcd480d8050 2021-04-09T18:29:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd506f5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd480d2800 con 0x7fcd480cffb0 2021-04-09T18:29:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd506f5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd480d91f0 con 0x7fcd480d7c50 2021-04-09T18:29:27.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd4dd10700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcd480d7c50 0x7fcd481b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd4ed12700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcd480d8050 0x7fcd4810d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36270/0 (socket says 172.21.15.32:36270) 2021-04-09T18:29:27.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd4ed12700 1 -- 172.21.15.32:0/1396173553 learned_addr learned my addr 172.21.15.32:0/1396173553 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:27.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd4dd10700 1 -- 172.21.15.32:0/1396173553 >> v1:172.21.15.86:6789/0 conn(0x7fcd480cffb0 legacy=0x7fcd48087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd4dd10700 1 -- 172.21.15.32:0/1396173553 >> v1:172.21.15.32:6789/0 conn(0x7fcd480d8050 legacy=0x7fcd4810d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd4d50f700 1 -- 172.21.15.32:0/1396173553 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177166439 0 0) 0x7fcd40000d30 con 0x7fcd480d8050 2021-04-09T18:29:27.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd4dd10700 1 -- 172.21.15.32:0/1396173553 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd481120d0 con 0x7fcd480d7c50 2021-04-09T18:29:27.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd4dd10700 1 --2- 172.21.15.32:0/1396173553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480d7c50 0x7fcd481b0490 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 rx=0x7fcd38006650 tx=0x7fcd38009a00).ready entity=mon.0 client_cookie=27f01ed76b31dd0f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:27.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.683 7fcd4d50f700 1 -- 172.21.15.32:0/1396173553 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd38009df0 con 0x7fcd480d7c50 2021-04-09T18:29:27.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.684 7fcd4d50f700 1 -- 172.21.15.32:0/1396173553 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd380009b0 con 0x7fcd480d7c50 2021-04-09T18:29:27.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.684 7fcd4d50f700 1 -- 172.21.15.32:0/1396173553 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd3800c2b0 con 0x7fcd480d7c50 2021-04-09T18:29:27.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.684 7fcd506f5700 1 -- 172.21.15.32:0/1396173553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480d7c50 msgr2=0x7fcd481b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.684 7fcd506f5700 1 --2- 172.21.15.32:0/1396173553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480d7c50 0x7fcd481b0490 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 rx=0x7fcd38006650 tx=0x7fcd38009a00).stop 2021-04-09T18:29:27.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.684 7fcd506f5700 1 -- 172.21.15.32:0/1396173553 shutdown_connections 2021-04-09T18:29:27.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.684 7fcd506f5700 1 --2- 172.21.15.32:0/1396173553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480d7c50 0x7fcd481b0490 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.684 7fcd506f5700 1 -- 172.21.15.32:0/1396173553 >> 172.21.15.32:0/1396173553 conn(0x7fcd480ce5b0 msgr2=0x7fcd480cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:27.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.685 7fcd506f5700 1 -- 172.21.15.32:0/1396173553 shutdown_connections 2021-04-09T18:29:27.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.685 7fcd506f5700 1 -- 172.21.15.32:0/1396173553 wait complete. 2021-04-09T18:29:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd506f5700 1 Processor -- start 2021-04-09T18:29:27.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd506f5700 1 -- start start 2021-04-09T18:29:27.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd506f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480c0f30 0x7fcd480a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd506f5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd480a9f20 0x7fcd480a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd506f5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd480a1b50 0x7fcd480a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd506f5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480a9380 con 0x7fcd480c0f30 2021-04-09T18:29:27.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd506f5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480aa320 con 0x7fcd480a9f20 2021-04-09T18:29:27.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd506f5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480c1330 con 0x7fcd480a1b50 2021-04-09T18:29:27.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd4e511700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480c0f30 0x7fcd480a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd4e511700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480c0f30 0x7fcd480a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54914/0 (socket says 172.21.15.32:54914) 2021-04-09T18:29:27.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.705 7fcd4e511700 1 -- 172.21.15.32:0/1811485471 learned_addr learned my addr 172.21.15.32:0/1811485471 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:27.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4e511700 1 -- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd480a1b50 msgr2=0x7fcd480a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4ed12700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd480a1b50 0x7fcd480a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4dd10700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd480a9f20 0x7fcd480a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4e511700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd480a1b50 0x7fcd480a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4e511700 1 -- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd480a9f20 msgr2=0x7fcd480a1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:27.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4e511700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd480a9f20 0x7fcd480a1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:27.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4e511700 1 -- 172.21.15.32:0/1811485471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd480a1f50 con 0x7fcd480c0f30 2021-04-09T18:29:27.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4dd10700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd480a9f20 0x7fcd480a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:27.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4ed12700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd480a1b50 0x7fcd480a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:27.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd4e511700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480c0f30 0x7fcd480a9900 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7fcd44006f20 tx=0x7fcd4400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:27.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd3effd700 1 -- 172.21.15.32:0/1811485471 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd4400bd80 con 0x7fcd480c0f30 2021-04-09T18:29:27.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd3effd700 1 -- 172.21.15.32:0/1811485471 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd44006240 con 0x7fcd480c0f30 2021-04-09T18:29:27.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd3effd700 1 -- 172.21.15.32:0/1811485471 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd44000dd0 con 0x7fcd480c0f30 2021-04-09T18:29:27.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd480a1f50 con 0x7fcd480c0f30 2021-04-09T18:29:27.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.706 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd480a1f50 con 0x7fcd480c0f30 2021-04-09T18:29:27.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.707 7fcd3effd700 1 -- 172.21.15.32:0/1811485471 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcd44005520 con 0x7fcd480c0f30 2021-04-09T18:29:27.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.707 7fcd3effd700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd30018930 0x7fcd3001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:27.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.707 7fcd3effd700 1 -- 172.21.15.32:0/1811485471 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcd44032a00 con 0x7fcd480c0f30 2021-04-09T18:29:27.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.707 7fcd4dd10700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd30018930 0x7fcd3001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.707 7fcd4dd10700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd30018930 0x7fcd3001ae10 crc :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:27.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.727 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd48093ff0 con 0x7fcd480c0f30 2021-04-09T18:29:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.730 7fcd3effd700 1 -- 172.21.15.32:0/1811485471 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcd44006030 con 0x7fcd480c0f30 2021-04-09T18:29:28.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.917 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fcd48093ff0 con 0x7fcd480c0f30 2021-04-09T18:29:28.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.921 7fcd3effd700 1 -- 172.21.15.32:0/1811485471 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v38) v1 ==== 184+0+0 (secure 0 0 0) 0x7fcd44005520 con 0x7fcd480c0f30 2021-04-09T18:29:28.009 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:29:28.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd30018930 msgr2=0x7fcd3001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd30018930 0x7fcd3001ae10 crc :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480c0f30 msgr2=0x7fcd480a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480c0f30 0x7fcd480a9900 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7fcd44006f20 tx=0x7fcd4400b990).stop 2021-04-09T18:29:28.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 shutdown_connections 2021-04-09T18:29:28.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd30018930 0x7fcd3001ae10 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd480a1b50 0x7fcd480a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd480a9f20 0x7fcd480a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 --2- 172.21.15.32:0/1811485471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd480c0f30 0x7fcd480a9900 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 >> 172.21.15.32:0/1811485471 conn(0x7fcd480adb50 msgr2=0x7fcd480ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:28.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 shutdown_connections 2021-04-09T18:29:28.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:27.927 7fcd506f5700 1 -- 172.21.15.32:0/1811485471 wait complete. 2021-04-09T18:29:28.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:29:28.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.099 7fbfc2893700 1 Processor -- start 2021-04-09T18:29:28.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.099 7fbfc2893700 1 -- start start 2021-04-09T18:29:28.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfc2893700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbfbc0d7c50 0x7fbfbc1b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfc2893700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbfbc0d8050 0x7fbfbc10d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfc2893700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfbc0d8620 con 0x7fbfbc0cffb0 2021-04-09T18:29:28.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfc2893700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfbc0d91f0 con 0x7fbfbc0d7c50 2021-04-09T18:29:28.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfc2893700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfbc0d2800 con 0x7fbfbc0d8050 2021-04-09T18:29:28.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfb3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbfbc0d7c50 0x7fbfbc1b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfbbfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbfbc0cffb0 0x7fbfbc087e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36280/0 (socket says 172.21.15.32:36280) 2021-04-09T18:29:28.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfbbfff700 1 -- 172.21.15.32:0/2486917993 learned_addr learned my addr 172.21.15.32:0/2486917993 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:28.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfc0eb0700 1 --2- 172.21.15.32:0/2486917993 >> v2:172.21.15.86:3300/0 conn(0x7fbfbc0d8050 0x7fbfbc10d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfb3fff700 1 -- 172.21.15.32:0/2486917993 >> v2:172.21.15.86:3300/0 conn(0x7fbfbc0d8050 msgr2=0x7fbfbc10d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:28.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfb3fff700 1 --2- 172.21.15.32:0/2486917993 >> v2:172.21.15.86:3300/0 conn(0x7fbfbc0d8050 0x7fbfbc10d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfb3fff700 1 -- 172.21.15.32:0/2486917993 >> v1:172.21.15.32:6789/0 conn(0x7fbfbc0cffb0 legacy=0x7fbfbc087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfbb7fe700 1 -- 172.21.15.32:0/2486917993 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066867205 0 0) 0x7fbfac000ce0 con 0x7fbfbc0cffb0 2021-04-09T18:29:28.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfb3fff700 1 -- 172.21.15.32:0/2486917993 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc1b0ab0 con 0x7fbfbc0d7c50 2021-04-09T18:29:28.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfc0eb0700 1 --2- 172.21.15.32:0/2486917993 >> v2:172.21.15.86:3300/0 conn(0x7fbfbc0d8050 0x7fbfbc10d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:28.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfb3fff700 1 --2- 172.21.15.32:0/2486917993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0d7c50 0x7fbfbc1b0490 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7fbfa4006650 tx=0x7fbfa4009a80).ready entity=mon.0 client_cookie=5c9d8edbe5a74531 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:28.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfbb7fe700 1 -- 172.21.15.32:0/2486917993 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfa40009b0 con 0x7fbfbc0d7c50 2021-04-09T18:29:28.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfbb7fe700 1 -- 172.21.15.32:0/2486917993 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa400c0c0 con 0x7fbfbc0d7c50 2021-04-09T18:29:28.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfbb7fe700 1 -- 172.21.15.32:0/2486917993 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfa400c510 con 0x7fbfbc0d7c50 2021-04-09T18:29:28.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfc2893700 1 -- 172.21.15.32:0/2486917993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0d7c50 msgr2=0x7fbfbc1b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.100 7fbfc2893700 1 --2- 172.21.15.32:0/2486917993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0d7c50 0x7fbfbc1b0490 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7fbfa4006650 tx=0x7fbfa4009a80).stop 2021-04-09T18:29:28.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.101 7fbfc2893700 1 -- 172.21.15.32:0/2486917993 shutdown_connections 2021-04-09T18:29:28.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.101 7fbfc2893700 1 --2- 172.21.15.32:0/2486917993 >> v2:172.21.15.86:3300/0 conn(0x7fbfbc0d8050 0x7fbfbc10d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.101 7fbfc2893700 1 --2- 172.21.15.32:0/2486917993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0d7c50 0x7fbfbc1b0490 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.101 7fbfc2893700 1 -- 172.21.15.32:0/2486917993 >> 172.21.15.32:0/2486917993 conn(0x7fbfbc0ce5b0 msgr2=0x7fbfbc0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:28.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.101 7fbfc2893700 1 -- 172.21.15.32:0/2486917993 shutdown_connections 2021-04-09T18:29:28.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.101 7fbfc2893700 1 -- 172.21.15.32:0/2486917993 wait complete. 2021-04-09T18:29:28.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.121 7fbfc2893700 1 Processor -- start 2021-04-09T18:29:28.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.121 7fbfc2893700 1 -- start start 2021-04-09T18:29:28.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc2893700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfbc180000 0x7fbfbc0c09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc2893700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfbc180ef0 0x7fbfbc1812f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc2893700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0a22b0 0x7fbfbc0a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc2893700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc180970 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc2893700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc0c1000 con 0x7fbfbc180000 2021-04-09T18:29:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc2893700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc180400 con 0x7fbfbc180ef0 2021-04-09T18:29:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc0eb0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0a22b0 0x7fbfbc0a26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc0eb0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0a22b0 0x7fbfbc0a26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54932/0 (socket says 172.21.15.32:54932) 2021-04-09T18:29:28.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc0eb0700 1 -- 172.21.15.32:0/3854552510 learned_addr learned my addr 172.21.15.32:0/3854552510 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:28.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfb3fff700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfbc180ef0 0x7fbfbc1812f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc0eb0700 1 -- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfbc180ef0 msgr2=0x7fbfbc1812f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfbbfff700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfbc180000 0x7fbfbc0c09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc0eb0700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfbc180ef0 0x7fbfbc1812f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc0eb0700 1 -- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfbc180000 msgr2=0x7fbfbc0c09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc0eb0700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfbc180000 0x7fbfbc0c09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfc0eb0700 1 -- 172.21.15.32:0/3854552510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc181910 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfb3fff700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfbc180ef0 0x7fbfbc1812f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:28.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.122 7fbfbbfff700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfbc180000 0x7fbfbc0c09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:28.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.123 7fbfc0eb0700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0a22b0 0x7fbfbc0a26b0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7fbfb4009980 tx=0x7fbfb400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:28.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.123 7fbfb97fa700 1 -- 172.21.15.32:0/3854552510 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfb400c2f0 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.123 7fbfb97fa700 1 -- 172.21.15.32:0/3854552510 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb4006790 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.123 7fbfb97fa700 1 -- 172.21.15.32:0/3854552510 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfb400c7e0 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.123 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfbc181910 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.123 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfbc181910 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.123 7fbfb97fa700 1 -- 172.21.15.32:0/3854552510 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbfb40011f0 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.124 7fbfb97fa700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf940188c0 0x7fbf9401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.124 7fbfb97fa700 1 -- 172.21.15.32:0/3854552510 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbfb4032b40 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.124 7fbfbbfff700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf940188c0 0x7fbf9401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.124 7fbfbbfff700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf940188c0 0x7fbf9401adc0 crc :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:28.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.143 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfbc093ff0 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.147 7fbfb97fa700 1 -- 172.21.15.32:0/3854552510 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbfb400be10 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.332 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fbfbc0d8620 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.335 7fbfb97fa700 1 -- 172.21.15.32:0/3854552510 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v39) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbfb40011f0 con 0x7fbfbc0a22b0 2021-04-09T18:29:28.424 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:29:28.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf940188c0 msgr2=0x7fbf9401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf940188c0 0x7fbf9401adc0 crc :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0a22b0 msgr2=0x7fbfbc0a26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0a22b0 0x7fbfbc0a26b0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7fbfb4009980 tx=0x7fbfb400bcd0).stop 2021-04-09T18:29:28.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 shutdown_connections 2021-04-09T18:29:28.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf940188c0 0x7fbf9401adc0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfbc0a22b0 0x7fbfbc0a26b0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfbc180ef0 0x7fbfbc1812f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 --2- 172.21.15.32:0/3854552510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfbc180000 0x7fbfbc0c09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 >> 172.21.15.32:0/3854552510 conn(0x7fbfbc0adaf0 msgr2=0x7fbfbc0bdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 shutdown_connections 2021-04-09T18:29:28.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.341 7fbfc2893700 1 -- 172.21.15.32:0/3854552510 wait complete. 2021-04-09T18:29:28.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:28.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.501 7f27f580b700 1 Processor -- start 2021-04-09T18:29:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.501 7f27f580b700 1 -- start start 2021-04-09T18:29:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27f580b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00cffa0 0x7f27f00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27f580b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27f00d6c80 0x7f27f010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27f580b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f00d7280 con 0x7f27f00d7800 2021-04-09T18:29:28.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27f580b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f00d7080 con 0x7f27f00d6c80 2021-04-09T18:29:28.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27f580b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00d8580 con 0x7f27f00cffa0 2021-04-09T18:29:28.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27efd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27f00d6c80 0x7f27f010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27efd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27f00d6c80 0x7f27f010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54936/0 (socket says 172.21.15.32:54936) 2021-04-09T18:29:28.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27efd75700 1 -- 172.21.15.32:0/1531313218 learned_addr learned my addr 172.21.15.32:0/1531313218 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:28.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27ef574700 1 --2- 172.21.15.32:0/1531313218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00cffa0 0x7f27f00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27efd75700 1 -- 172.21.15.32:0/1531313218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00cffa0 msgr2=0x7f27f00d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:28.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27efd75700 1 --2- 172.21.15.32:0/1531313218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00cffa0 0x7f27f00d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27efd75700 1 -- 172.21.15.32:0/1531313218 >> v1:172.21.15.86:6789/0 conn(0x7f27f00d7800 legacy=0x7f27f00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27efd75700 1 -- 172.21.15.32:0/1531313218 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f00d91e0 con 0x7f27f00d6c80 2021-04-09T18:29:28.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27efd75700 1 --2- 172.21.15.32:0/1531313218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00d6c80 0x7f27f010cfd0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f27e8006f00 tx=0x7f27e8009e90).ready entity=mon.0 client_cookie=a143302f7f16eed6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:28.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27ee572700 1 -- 172.21.15.32:0/1531313218 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e800b300 con 0x7f27f00d6c80 2021-04-09T18:29:28.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27ee572700 1 -- 172.21.15.32:0/1531313218 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e800b640 con 0x7f27f00d6c80 2021-04-09T18:29:28.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27ee572700 1 -- 172.21.15.32:0/1531313218 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e800bb20 con 0x7f27f00d6c80 2021-04-09T18:29:28.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27f580b700 1 -- 172.21.15.32:0/1531313218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00d6c80 msgr2=0x7f27f010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.502 7f27f580b700 1 --2- 172.21.15.32:0/1531313218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00d6c80 0x7f27f010cfd0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f27e8006f00 tx=0x7f27e8009e90).stop 2021-04-09T18:29:28.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.503 7f27f580b700 1 -- 172.21.15.32:0/1531313218 shutdown_connections 2021-04-09T18:29:28.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.503 7f27f580b700 1 --2- 172.21.15.32:0/1531313218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f00d6c80 0x7f27f010cfd0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.503 7f27f580b700 1 --2- 172.21.15.32:0/1531313218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f00cffa0 0x7f27f00d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.503 7f27f580b700 1 -- 172.21.15.32:0/1531313218 >> 172.21.15.32:0/1531313218 conn(0x7f27f00ce5c0 msgr2=0x7f27f00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:28.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.503 7f27f580b700 1 -- 172.21.15.32:0/1531313218 shutdown_connections 2021-04-09T18:29:28.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.503 7f27f580b700 1 -- 172.21.15.32:0/1531313218 wait complete. 2021-04-09T18:29:28.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.523 7f27f580b700 1 Processor -- start 2021-04-09T18:29:28.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.523 7f27f580b700 1 -- start start 2021-04-09T18:29:28.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27f580b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f0198230 0x7f27f0184e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27f580b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f01acee0 0x7f27f01abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27f580b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01ac510 0x7f27f0183ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27f580b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f01ac960 con 0x7f27f01acee0 2021-04-09T18:29:28.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27f580b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f01ad2e0 con 0x7f27f01ac510 2021-04-09T18:29:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27f580b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f0188e20 con 0x7f27f0198230 2021-04-09T18:29:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27eed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f01acee0 0x7f27f01abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27eed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f01acee0 0x7f27f01abef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54942/0 (socket says 172.21.15.32:54942) 2021-04-09T18:29:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27eed73700 1 -- 172.21.15.32:0/4058590869 learned_addr learned my addr 172.21.15.32:0/4058590869 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27efd75700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01ac510 0x7f27f0183ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27ef574700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f0198230 0x7f27f0184e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27eed73700 1 -- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f0198230 msgr2=0x7f27f0184e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27eed73700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f0198230 0x7f27f0184e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27eed73700 1 -- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01ac510 msgr2=0x7f27f0183ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27eed73700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01ac510 0x7f27f0183ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.524 7f27eed73700 1 -- 172.21.15.32:0/4058590869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f0184610 con 0x7f27f01acee0 2021-04-09T18:29:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.525 7f27eed73700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f01acee0 0x7f27f01abef0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 rx=0x7f27e400b5b0 tx=0x7f27e400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.525 7f27d7fff700 1 -- 172.21.15.32:0/4058590869 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e400bd40 con 0x7f27f01acee0 2021-04-09T18:29:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.525 7f27d7fff700 1 -- 172.21.15.32:0/4058590869 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e4006240 con 0x7f27f01acee0 2021-04-09T18:29:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.525 7f27f580b700 1 -- 172.21.15.32:0/4058590869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f0184610 con 0x7f27f01acee0 2021-04-09T18:29:28.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.525 7f27d7fff700 1 -- 172.21.15.32:0/4058590869 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27e4005690 con 0x7f27f01acee0 2021-04-09T18:29:28.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.525 7f27f580b700 1 -- 172.21.15.32:0/4058590869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f0184610 con 0x7f27f01acee0 2021-04-09T18:29:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.526 7f27d7fff700 1 -- 172.21.15.32:0/4058590869 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f27e40054a0 con 0x7f27f01acee0 2021-04-09T18:29:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.526 7f27d7fff700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d00188c0 0x7f27d001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.526 7f27d7fff700 1 -- 172.21.15.32:0/4058590869 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f27e40327f0 con 0x7f27f01acee0 2021-04-09T18:29:28.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.526 7f27ef574700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d00188c0 0x7f27d001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:28.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.526 7f27ef574700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d00188c0 0x7f27d001adc0 crc :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:28.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.546 7f27f580b700 1 -- 172.21.15.32:0/4058590869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f0182df0 con 0x7f27f01acee0 2021-04-09T18:29:28.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.549 7f27d7fff700 1 -- 172.21.15.32:0/4058590869 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f27e4000c60 con 0x7f27f01acee0 2021-04-09T18:29:28.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.741 7f27f580b700 1 -- 172.21.15.32:0/4058590869 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27f0182df0 con 0x7f27f01acee0 2021-04-09T18:29:28.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.741 7f27d7fff700 1 -- 172.21.15.32:0/4058590869 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 311 v311) v1 ==== 95+0+2788 (secure 0 0 0) 0x7f27e40054a0 con 0x7f27f01acee0 2021-04-09T18:29:28.827 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 311 2021-04-09T18:29:28.827 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:28.827 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":311,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2532077375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2532077375},{"type":"v1","addr":"172.21.15.32:6834","nonce":2532077375}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":311,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:29:26.675661","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10177":{"gid":10177,"name":"a","rank":0,"incarnation":308,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2200487686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2200487686},{"type":"v1","addr":"172.21.15.32:6835","nonce":2200487686}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-04-09T18:29:28.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 -- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d00188c0 msgr2=0x7f27d001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d00188c0 0x7f27d001adc0 crc :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 -- 172.21.15.32:0/4058590869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f01acee0 msgr2=0x7f27f01abef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f01acee0 0x7f27f01abef0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 rx=0x7f27e400b5b0 tx=0x7f27e400b950).stop 2021-04-09T18:29:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 -- 172.21.15.32:0/4058590869 shutdown_connections 2021-04-09T18:29:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27d00188c0 0x7f27d001adc0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27f01acee0 0x7f27f01abef0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01ac510 0x7f27f0183ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 --2- 172.21.15.32:0/4058590869 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f0198230 0x7f27f0184e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 -- 172.21.15.32:0/4058590869 >> 172.21.15.32:0/4058590869 conn(0x7f27f0183870 msgr2=0x7f27f01a9650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:28.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 -- 172.21.15.32:0/4058590869 shutdown_connections 2021-04-09T18:29:28.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:28.744 7f27f580b700 1 -- 172.21.15.32:0/4058590869 wait complete. 2021-04-09T18:29:29.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 311, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:29:24.667843', 'modified': '2021-04-09 18:29:26.675661', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10177': {'gid': 10177, 'name': 'a', 'rank': 0, 'incarnation': 308, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2200487686', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2200487686}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2200487686}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:29:29.243 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:29:29.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:29:29.382 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:29:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:29:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:29:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 10177, 2021-04-09T18:29:29.383 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:29:29.384 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:29:29.384 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 4.6293632660000004, 2021-04-09T18:29:29.384 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 311, 2021-04-09T18:29:29.385 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 168, 2021-04-09T18:29:29.385 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 168, 2021-04-09T18:29:29.385 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 10.068308672000001 2021-04-09T18:29:29.386 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:29:29.389 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 10177, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.6293632660000004, "mdsmap_epoch": 311, "osdmap_epoch": 168, "osdmap_epoch_barrier": 168, "uptime": 10.068308672000001 } 2021-04-09T18:29:29.389 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:29:29.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.454 7f2b4b8a0700 1 Processor -- start 2021-04-09T18:29:29.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.454 7f2b4b8a0700 1 -- start start 2021-04-09T18:29:29.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.454 7f2b4b8a0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b4410d010 0x7f2b440d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.454 7f2b4b8a0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2b440d1da0 0x7f2b440d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.454 7f2b4b8a0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b440d82c0 0x7f2b4410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.454 7f2b4b8a0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b440d8d30 con 0x7f2b4410d010 2021-04-09T18:29:29.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.454 7f2b4b8a0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b440d9220 con 0x7f2b440d1da0 2021-04-09T18:29:29.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.454 7f2b4b8a0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4410d4b0 con 0x7f2b440d82c0 2021-04-09T18:29:29.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b496bc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b4410d010 0x7f2b440d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b496bc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b4410d010 0x7f2b440d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54948/0 (socket says 172.21.15.32:54948) 2021-04-09T18:29:29.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b496bc700 1 -- 172.21.15.32:0/2064949401 learned_addr learned my addr 172.21.15.32:0/2064949401 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:29.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b48ebb700 1 --2- 172.21.15.32:0/2064949401 >> v2:172.21.15.86:3300/0 conn(0x7f2b440d1da0 0x7f2b440d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b49ebd700 1 --2- 172.21.15.32:0/2064949401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b440d82c0 0x7f2b4410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b496bc700 1 -- 172.21.15.32:0/2064949401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b440d82c0 msgr2=0x7f2b4410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:29.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b496bc700 1 --2- 172.21.15.32:0/2064949401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b440d82c0 0x7f2b4410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b496bc700 1 -- 172.21.15.32:0/2064949401 >> v2:172.21.15.86:3300/0 conn(0x7f2b440d1da0 msgr2=0x7f2b440d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:29.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b496bc700 1 --2- 172.21.15.32:0/2064949401 >> v2:172.21.15.86:3300/0 conn(0x7f2b440d1da0 0x7f2b440d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b496bc700 1 -- 172.21.15.32:0/2064949401 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b440d86c0 con 0x7f2b4410d010 2021-04-09T18:29:29.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b496bc700 1 --2- 172.21.15.32:0/2064949401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4410d010 0x7f2b440d1780 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7f2b40006700 tx=0x7f2b40009b00).ready entity=mon.0 client_cookie=446ff76593f7c0a3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b3bfff700 1 -- 172.21.15.32:0/2064949401 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b4000c0c0 con 0x7f2b4410d010 2021-04-09T18:29:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b3bfff700 1 -- 172.21.15.32:0/2064949401 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4000c400 con 0x7f2b4410d010 2021-04-09T18:29:29.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b3bfff700 1 -- 172.21.15.32:0/2064949401 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b4000c8e0 con 0x7f2b4410d010 2021-04-09T18:29:29.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b4b8a0700 1 -- 172.21.15.32:0/2064949401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4410d010 msgr2=0x7f2b440d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:29.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.455 7f2b4b8a0700 1 --2- 172.21.15.32:0/2064949401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4410d010 0x7f2b440d1780 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7f2b40006700 tx=0x7f2b40009b00).stop 2021-04-09T18:29:29.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.456 7f2b4b8a0700 1 -- 172.21.15.32:0/2064949401 shutdown_connections 2021-04-09T18:29:29.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.456 7f2b4b8a0700 1 --2- 172.21.15.32:0/2064949401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b440d82c0 0x7f2b4410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.456 7f2b4b8a0700 1 --2- 172.21.15.32:0/2064949401 >> v2:172.21.15.86:3300/0 conn(0x7f2b440d1da0 0x7f2b440d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.456 7f2b4b8a0700 1 --2- 172.21.15.32:0/2064949401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b4410d010 0x7f2b440d1780 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.456 7f2b4b8a0700 1 -- 172.21.15.32:0/2064949401 >> 172.21.15.32:0/2064949401 conn(0x7f2b440ce5f0 msgr2=0x7f2b440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:29.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.456 7f2b4b8a0700 1 -- 172.21.15.32:0/2064949401 shutdown_connections 2021-04-09T18:29:29.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.456 7f2b4b8a0700 1 -- 172.21.15.32:0/2064949401 wait complete. 2021-04-09T18:29:29.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.476 7f2b4b8a0700 1 Processor -- start 2021-04-09T18:29:29.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.476 7f2b4b8a0700 1 -- start start 2021-04-09T18:29:29.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b4b8a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b441acea0 0x7f2b441ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b4b8a0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b4419b2c0 0x7f2b44184da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b4b8a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b441853c0 0x7f2b44185fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b4b8a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b441857c0 con 0x7f2b441853c0 2021-04-09T18:29:29.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b4b8a0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b44185a40 con 0x7f2b441acea0 2021-04-09T18:29:29.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b4b8a0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b441865e0 con 0x7f2b4419b2c0 2021-04-09T18:29:29.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b49ebd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b441853c0 0x7f2b44185fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b49ebd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b441853c0 0x7f2b44185fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54958/0 (socket says 172.21.15.32:54958) 2021-04-09T18:29:29.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b49ebd700 1 -- 172.21.15.32:0/2560294227 learned_addr learned my addr 172.21.15.32:0/2560294227 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:29.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b496bc700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b441acea0 0x7f2b441ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.477 7f2b48ebb700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b4419b2c0 0x7f2b44184da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b49ebd700 1 -- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b4419b2c0 msgr2=0x7f2b44184da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:29.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b49ebd700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b4419b2c0 0x7f2b44184da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b49ebd700 1 -- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b441acea0 msgr2=0x7f2b441ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:29.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b49ebd700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b441acea0 0x7f2b441ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b49ebd700 1 -- 172.21.15.32:0/2560294227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b44186800 con 0x7f2b441853c0 2021-04-09T18:29:29.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b496bc700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b441acea0 0x7f2b441ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b49ebd700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b441853c0 0x7f2b44185fc0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7f2b3c00b970 tx=0x7f2b3c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:29.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b39ffb700 1 -- 172.21.15.32:0/2560294227 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b3c000f90 con 0x7f2b441853c0 2021-04-09T18:29:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b39ffb700 1 -- 172.21.15.32:0/2560294227 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c006aa0 con 0x7f2b441853c0 2021-04-09T18:29:29.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b39ffb700 1 -- 172.21.15.32:0/2560294227 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b3c001470 con 0x7f2b441853c0 2021-04-09T18:29:29.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b44186800 con 0x7f2b441853c0 2021-04-09T18:29:29.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b44186800 con 0x7f2b441853c0 2021-04-09T18:29:29.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.478 7f2b39ffb700 1 -- 172.21.15.32:0/2560294227 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2b3c001650 con 0x7f2b441853c0 2021-04-09T18:29:29.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.479 7f2b39ffb700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b24018930 0x7f2b2401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.479 7f2b39ffb700 1 -- 172.21.15.32:0/2560294227 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2b3c032da0 con 0x7f2b441853c0 2021-04-09T18:29:29.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.479 7f2b496bc700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b24018930 0x7f2b2401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.479 7f2b496bc700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b24018930 0x7f2b2401ae10 crc :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:29.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.499 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b44182d80 con 0x7f2b441853c0 2021-04-09T18:29:29.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.502 7f2b39ffb700 1 -- 172.21.15.32:0/2560294227 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2b3c001ef0 con 0x7f2b441853c0 2021-04-09T18:29:29.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.691 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b4410d480 con 0x7f2b441853c0 2021-04-09T18:29:29.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.692 7f2b39ffb700 1 -- 172.21.15.32:0/2560294227 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f2b3c001ef0 con 0x7f2b441853c0 2021-04-09T18:29:29.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:29.779 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":168,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:29:25.668831","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:29:22.005467","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-04-09 18:29:23.428510","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:29:29.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.694 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b24018930 msgr2=0x7f2b2401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:29.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.694 7f2b4b8a0700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b24018930 0x7f2b2401ae10 crc :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.694 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b441853c0 msgr2=0x7f2b44185fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:29.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.694 7f2b4b8a0700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b441853c0 0x7f2b44185fc0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7f2b3c00b970 tx=0x7f2b3c00bd10).stop 2021-04-09T18:29:29.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.695 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 shutdown_connections 2021-04-09T18:29:29.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.695 7f2b4b8a0700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b24018930 0x7f2b2401ae10 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.695 7f2b4b8a0700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b441853c0 0x7f2b44185fc0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.695 7f2b4b8a0700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b4419b2c0 0x7f2b44184da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.695 7f2b4b8a0700 1 --2- 172.21.15.32:0/2560294227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b441acea0 0x7f2b441ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.695 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 >> 172.21.15.32:0/2560294227 conn(0x7f2b4419b890 msgr2=0x7f2b441a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:29.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.695 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 shutdown_connections 2021-04-09T18:29:29.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.695 7f2b4b8a0700 1 -- 172.21.15.32:0/2560294227 wait complete. 2021-04-09T18:29:29.797 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 28, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:29:22.005467', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 29, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:29:23.428510', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '168', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:29:29.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:29:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.867 7f4b9cd37700 1 Processor -- start 2021-04-09T18:29:29.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.867 7f4b9cd37700 1 -- start start 2021-04-09T18:29:29.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b9cd37700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4b980d7ca0 0x7f4b980d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b9cd37700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4b9810da00 0x7f4b9810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b9cd37700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9810d480 con 0x7f4b9810d010 2021-04-09T18:29:29.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b9cd37700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b980d9220 con 0x7f4b9810da00 2021-04-09T18:29:29.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b9cd37700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b980d86c0 con 0x7f4b980d7ca0 2021-04-09T18:29:29.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b97314700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4b9810da00 0x7f4b9810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b96b13700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4b9810d010 0x7f4b980d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36320/0 (socket says 172.21.15.32:36320) 2021-04-09T18:29:29.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b96b13700 1 -- 172.21.15.32:0/2291196366 learned_addr learned my addr 172.21.15.32:0/2291196366 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:29.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b96312700 1 --2- 172.21.15.32:0/2291196366 >> v2:172.21.15.86:3300/0 conn(0x7f4b980d7ca0 0x7f4b980d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b97314700 1 -- 172.21.15.32:0/2291196366 >> v2:172.21.15.86:3300/0 conn(0x7f4b980d7ca0 msgr2=0x7f4b980d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:29.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b97314700 1 --2- 172.21.15.32:0/2291196366 >> v2:172.21.15.86:3300/0 conn(0x7f4b980d7ca0 0x7f4b980d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b97314700 1 -- 172.21.15.32:0/2291196366 >> v1:172.21.15.32:6789/0 conn(0x7f4b9810d010 legacy=0x7f4b980d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b95b11700 1 -- 172.21.15.32:0/2291196366 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041771814 0 0) 0x7f4b88000ce0 con 0x7f4b9810d010 2021-04-09T18:29:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b97314700 1 -- 172.21.15.32:0/2291196366 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980d1f70 con 0x7f4b9810da00 2021-04-09T18:29:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.868 7f4b96312700 1 --2- 172.21.15.32:0/2291196366 >> v2:172.21.15.86:3300/0 conn(0x7f4b980d7ca0 0x7f4b980d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:29.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.869 7f4b97314700 1 --2- 172.21.15.32:0/2291196366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b9810da00 0x7f4b9810de00 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f4b90006e30 tx=0x7f4b90009e60).ready entity=mon.0 client_cookie=a48f7345f976da39 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:29.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.869 7f4b95b11700 1 -- 172.21.15.32:0/2291196366 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b90006a50 con 0x7f4b9810da00 2021-04-09T18:29:29.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.869 7f4b95b11700 1 -- 172.21.15.32:0/2291196366 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9000b300 con 0x7f4b9810da00 2021-04-09T18:29:29.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.869 7f4b95b11700 1 -- 172.21.15.32:0/2291196366 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b9000b750 con 0x7f4b9810da00 2021-04-09T18:29:29.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.869 7f4b9cd37700 1 -- 172.21.15.32:0/2291196366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b9810da00 msgr2=0x7f4b9810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:29.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.869 7f4b9cd37700 1 --2- 172.21.15.32:0/2291196366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b9810da00 0x7f4b9810de00 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f4b90006e30 tx=0x7f4b90009e60).stop 2021-04-09T18:29:29.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.870 7f4b9cd37700 1 -- 172.21.15.32:0/2291196366 shutdown_connections 2021-04-09T18:29:29.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.870 7f4b9cd37700 1 --2- 172.21.15.32:0/2291196366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b9810da00 0x7f4b9810de00 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.870 7f4b9cd37700 1 --2- 172.21.15.32:0/2291196366 >> v2:172.21.15.86:3300/0 conn(0x7f4b980d7ca0 0x7f4b980d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.870 7f4b9cd37700 1 -- 172.21.15.32:0/2291196366 >> 172.21.15.32:0/2291196366 conn(0x7f4b980ce5f0 msgr2=0x7f4b980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:29.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.870 7f4b9cd37700 1 -- 172.21.15.32:0/2291196366 shutdown_connections 2021-04-09T18:29:29.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.870 7f4b9cd37700 1 -- 172.21.15.32:0/2291196366 wait complete. 2021-04-09T18:29:29.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.890 7f4b9cd37700 1 Processor -- start 2021-04-09T18:29:29.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.890 7f4b9cd37700 1 -- start start 2021-04-09T18:29:29.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.890 7f4b9cd37700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b980b1680 0x7f4b980b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.890 7f4b9cd37700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b9809e330 0x7f4b980b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b9cd37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b980b2190 0x7f4b9809f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b9cd37700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9809e730 con 0x7f4b980b2190 2021-04-09T18:29:29.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b9cd37700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9809f6c0 con 0x7f4b980b1680 2021-04-09T18:29:29.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b9cd37700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9809f8e0 con 0x7f4b9809e330 2021-04-09T18:29:29.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b97314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b980b2190 0x7f4b9809f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b97314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b980b2190 0x7f4b9809f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54970/0 (socket says 172.21.15.32:54970) 2021-04-09T18:29:29.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b97314700 1 -- 172.21.15.32:0/1739219829 learned_addr learned my addr 172.21.15.32:0/1739219829 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:29.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b96b13700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b980b1680 0x7f4b980b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b96312700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b9809e330 0x7f4b980b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b97314700 1 -- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b9809e330 msgr2=0x7f4b980b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:29.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b97314700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b9809e330 0x7f4b980b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b97314700 1 -- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b980b1680 msgr2=0x7f4b980b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b97314700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b980b1680 0x7f4b980b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b97314700 1 -- 172.21.15.32:0/1739219829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9809fb00 con 0x7f4b980b2190 2021-04-09T18:29:29.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b96312700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b9809e330 0x7f4b980b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b96b13700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b980b1680 0x7f4b980b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b97314700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b980b2190 0x7f4b9809f0a0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7f4b90001070 tx=0x7f4b9000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b877fe700 1 -- 172.21.15.32:0/1739219829 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b90001270 con 0x7f4b980b2190 2021-04-09T18:29:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b877fe700 1 -- 172.21.15.32:0/1739219829 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b90000d70 con 0x7f4b980b2190 2021-04-09T18:29:29.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b877fe700 1 -- 172.21.15.32:0/1739219829 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b90001270 con 0x7f4b980b2190 2021-04-09T18:29:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9809fb00 con 0x7f4b980b2190 2021-04-09T18:29:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.891 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9809fb00 con 0x7f4b980b2190 2021-04-09T18:29:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.892 7f4b877fe700 1 -- 172.21.15.32:0/1739219829 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4b900093d0 con 0x7f4b980b2190 2021-04-09T18:29:29.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.892 7f4b877fe700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b7c018930 0x7f4b7c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.892 7f4b877fe700 1 -- 172.21.15.32:0/1739219829 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4b90032dd0 con 0x7f4b980b2190 2021-04-09T18:29:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.892 7f4b96b13700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b7c018930 0x7f4b7c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:29.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.893 7f4b96b13700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b7c018930 0x7f4b7c01ae10 crc :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:29.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.912 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9810d480 con 0x7f4b980b2190 2021-04-09T18:29:30.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:29.915 7f4b877fe700 1 -- 172.21.15.32:0/1739219829 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4b90003020 con 0x7f4b980b2190 2021-04-09T18:29:30.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.099 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f4b9810d480 con 0x7f4b980b2190 2021-04-09T18:29:30.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.100 7f4b877fe700 1 -- 172.21.15.32:0/1739219829 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v168) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4b900093b0 con 0x7f4b980b2190 2021-04-09T18:29:30.186 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:29:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b7c018930 msgr2=0x7f4b7c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b7c018930 0x7f4b7c01ae10 crc :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b980b2190 msgr2=0x7f4b9809f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b980b2190 0x7f4b9809f0a0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7f4b90001070 tx=0x7f4b9000bd40).stop 2021-04-09T18:29:30.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 shutdown_connections 2021-04-09T18:29:30.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4b7c018930 0x7f4b7c01ae10 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b980b2190 0x7f4b9809f0a0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b9809e330 0x7f4b980b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 --2- 172.21.15.32:0/1739219829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b980b1680 0x7f4b980b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 >> 172.21.15.32:0/1739219829 conn(0x7f4b980c4290 msgr2=0x7f4b980c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:30.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 shutdown_connections 2021-04-09T18:29:30.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.102 7f4b9cd37700 1 -- 172.21.15.32:0/1739219829 wait complete. 2021-04-09T18:29:30.198 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:29:30.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.265 7f620f274700 1 Processor -- start 2021-04-09T18:29:30.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.266 7f620f274700 1 -- start start 2021-04-09T18:29:30.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.266 7f620f274700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f620810d010 0x7f62080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.266 7f620f274700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62080d1da0 0x7f62080d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.266 7f620f274700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f62080d82c0 0x7f620810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.266 7f620f274700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620810e780 con 0x7f62080d1da0 2021-04-09T18:29:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.266 7f620f274700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62080d8d30 con 0x7f62080d82c0 2021-04-09T18:29:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.266 7f620f274700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62080d9220 con 0x7f620810d010 2021-04-09T18:29:30.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620c88f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62080d1da0 0x7f62080d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620c88f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f62080d1da0 0x7f62080d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54978/0 (socket says 172.21.15.32:54978) 2021-04-09T18:29:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620c88f700 1 -- 172.21.15.32:0/3133490694 learned_addr learned my addr 172.21.15.32:0/3133490694 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620d090700 1 --2- 172.21.15.32:0/3133490694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f620810d010 0x7f62080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620d891700 1 --2- 172.21.15.32:0/3133490694 >> v2:172.21.15.86:3300/0 conn(0x7f62080d82c0 0x7f620810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620c88f700 1 -- 172.21.15.32:0/3133490694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f620810d010 msgr2=0x7f62080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620c88f700 1 --2- 172.21.15.32:0/3133490694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f620810d010 0x7f62080d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620c88f700 1 -- 172.21.15.32:0/3133490694 >> v2:172.21.15.86:3300/0 conn(0x7f62080d82c0 msgr2=0x7f620810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620c88f700 1 --2- 172.21.15.32:0/3133490694 >> v2:172.21.15.86:3300/0 conn(0x7f62080d82c0 0x7f620810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620c88f700 1 -- 172.21.15.32:0/3133490694 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620810d480 con 0x7f62080d1da0 2021-04-09T18:29:30.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620c88f700 1 --2- 172.21.15.32:0/3133490694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62080d1da0 0x7f62080d7ca0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7f61f8006700 tx=0x7f61f8009b00).ready entity=mon.0 client_cookie=bc31f25575749b72 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:30.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f61fffff700 1 -- 172.21.15.32:0/3133490694 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61f800c0c0 con 0x7f62080d1da0 2021-04-09T18:29:30.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f61fffff700 1 -- 172.21.15.32:0/3133490694 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f800c400 con 0x7f62080d1da0 2021-04-09T18:29:30.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f61fffff700 1 -- 172.21.15.32:0/3133490694 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61f800c8e0 con 0x7f62080d1da0 2021-04-09T18:29:30.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620f274700 1 -- 172.21.15.32:0/3133490694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62080d1da0 msgr2=0x7f62080d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.267 7f620f274700 1 --2- 172.21.15.32:0/3133490694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62080d1da0 0x7f62080d7ca0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7f61f8006700 tx=0x7f61f8009b00).stop 2021-04-09T18:29:30.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.268 7f620f274700 1 -- 172.21.15.32:0/3133490694 shutdown_connections 2021-04-09T18:29:30.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.268 7f620f274700 1 --2- 172.21.15.32:0/3133490694 >> v2:172.21.15.86:3300/0 conn(0x7f62080d82c0 0x7f620810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.268 7f620f274700 1 --2- 172.21.15.32:0/3133490694 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f62080d1da0 0x7f62080d7ca0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.268 7f620f274700 1 --2- 172.21.15.32:0/3133490694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f620810d010 0x7f62080d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.268 7f620f274700 1 -- 172.21.15.32:0/3133490694 >> 172.21.15.32:0/3133490694 conn(0x7f62080ce5f0 msgr2=0x7f62080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:30.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.268 7f620f274700 1 -- 172.21.15.32:0/3133490694 shutdown_connections 2021-04-09T18:29:30.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.268 7f620f274700 1 -- 172.21.15.32:0/3133490694 wait complete. 2021-04-09T18:29:30.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.288 7f620f274700 1 Processor -- start 2021-04-09T18:29:30.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.288 7f620f274700 1 -- start start 2021-04-09T18:29:30.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620f274700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62080c3c00 0x7f620809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620f274700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6208095240 0x7f62081a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620f274700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62081a1270 0x7f62080b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620f274700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620809cda0 con 0x7f6208095240 2021-04-09T18:29:30.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620f274700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62080c3680 con 0x7f62080c3c00 2021-04-09T18:29:30.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620f274700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6208095640 con 0x7f62081a1270 2021-04-09T18:29:30.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620c88f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6208095240 0x7f62081a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620c88f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6208095240 0x7f62081a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54982/0 (socket says 172.21.15.32:54982) 2021-04-09T18:29:30.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620c88f700 1 -- 172.21.15.32:0/2889059404 learned_addr learned my addr 172.21.15.32:0/2889059404 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:30.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620d090700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62080c3c00 0x7f620809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.289 7f620d891700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62081a1270 0x7f62080b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f620c88f700 1 -- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62081a1270 msgr2=0x7f62080b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f620c88f700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62081a1270 0x7f62080b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f620c88f700 1 -- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62080c3c00 msgr2=0x7f620809c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f620c88f700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62080c3c00 0x7f620809c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f620c88f700 1 -- 172.21.15.32:0/2889059404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62081a1670 con 0x7f6208095240 2021-04-09T18:29:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f620d090700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62080c3c00 0x7f620809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:30.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f620c88f700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6208095240 0x7f62081a0c50 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7f61f800b660 tx=0x7f61f800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:30.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f61fdffb700 1 -- 172.21.15.32:0/2889059404 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61f8009440 con 0x7f6208095240 2021-04-09T18:29:30.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f61fdffb700 1 -- 172.21.15.32:0/2889059404 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f8006390 con 0x7f6208095240 2021-04-09T18:29:30.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f620f274700 1 -- 172.21.15.32:0/2889059404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62081a1670 con 0x7f6208095240 2021-04-09T18:29:30.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f61fdffb700 1 -- 172.21.15.32:0/2889059404 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61f8002030 con 0x7f6208095240 2021-04-09T18:29:30.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.290 7f620f274700 1 -- 172.21.15.32:0/2889059404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62081a1670 con 0x7f6208095240 2021-04-09T18:29:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.291 7f61fdffb700 1 -- 172.21.15.32:0/2889059404 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f61f80093e0 con 0x7f6208095240 2021-04-09T18:29:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.291 7f61fdffb700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61e80188c0 0x7f61e801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.291 7f61fdffb700 1 -- 172.21.15.32:0/2889059404 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f61f80327d0 con 0x7f6208095240 2021-04-09T18:29:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.291 7f620d090700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61e80188c0 0x7f61e801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.291 7f620d090700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61e80188c0 0x7f61e801adc0 crc :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:30.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.310 7f620f274700 1 -- 172.21.15.32:0/2889059404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6208093d50 con 0x7f6208095240 2021-04-09T18:29:30.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.314 7f61fdffb700 1 -- 172.21.15.32:0/2889059404 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f61f80284e0 con 0x7f6208095240 2021-04-09T18:29:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.498 7f620f274700 1 -- 172.21.15.32:0/2889059404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f620810d480 con 0x7f6208095240 2021-04-09T18:29:30.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.498 7f61fdffb700 1 -- 172.21.15.32:0/2889059404 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v168) v1 ==== 104+0+10 (secure 0 0 0) 0x7f61f80284e0 con 0x7f6208095240 2021-04-09T18:29:30.584 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:29:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 -- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61e80188c0 msgr2=0x7f61e801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61e80188c0 0x7f61e801adc0 crc :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 -- 172.21.15.32:0/2889059404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6208095240 msgr2=0x7f62081a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6208095240 0x7f62081a0c50 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7f61f800b660 tx=0x7f61f800ba00).stop 2021-04-09T18:29:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 -- 172.21.15.32:0/2889059404 shutdown_connections 2021-04-09T18:29:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61e80188c0 0x7f61e801adc0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6208095240 0x7f62081a0c50 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62081a1270 0x7f62080b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 --2- 172.21.15.32:0/2889059404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62080c3c00 0x7f620809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 -- 172.21.15.32:0/2889059404 >> 172.21.15.32:0/2889059404 conn(0x7f62080975f0 msgr2=0x7f6208096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:30.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 -- 172.21.15.32:0/2889059404 shutdown_connections 2021-04-09T18:29:30.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.501 7f620f274700 1 -- 172.21.15.32:0/2889059404 wait complete. 2021-04-09T18:29:30.596 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:29:30.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.665 7f3a7535c700 1 Processor -- start 2021-04-09T18:29:30.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.665 7f3a7535c700 1 -- start start 2021-04-09T18:29:30.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a7535c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3a700d7ca0 0x7f3a700d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a7535c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7010d480 con 0x7f3a7010da00 2021-04-09T18:29:30.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a7535c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a700d86c0 con 0x7f3a7010d010 2021-04-09T18:29:30.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a7535c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a700d9220 con 0x7f3a700d7ca0 2021-04-09T18:29:30.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6f87e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3a7010da00 0x7f3a7010de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36350/0 (socket says 172.21.15.32:36350) 2021-04-09T18:29:30.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6f87e700 1 -- 172.21.15.32:0/3613023331 learned_addr learned my addr 172.21.15.32:0/3613023331 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:30.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6e87c700 1 --2- 172.21.15.32:0/3613023331 >> v2:172.21.15.86:3300/0 conn(0x7f3a700d7ca0 0x7f3a700d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6e07b700 1 -- 172.21.15.32:0/3613023331 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237682734 0 0) 0x7f3a68000d30 con 0x7f3a7010da00 2021-04-09T18:29:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6e07b700 1 -- 172.21.15.32:0/3613023331 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a58002fe0 con 0x7f3a7010da00 2021-04-09T18:29:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6e07b700 1 -- 172.21.15.32:0/3613023331 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440940910 0 0) 0x7f3a60000c60 con 0x7f3a7010d010 2021-04-09T18:29:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6e07b700 1 -- 172.21.15.32:0/3613023331 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a580042c0 con 0x7f3a7010d010 2021-04-09T18:29:30.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6e87c700 1 -- 172.21.15.32:0/3613023331 >> v1:172.21.15.86:6789/0 conn(0x7f3a7010d010 legacy=0x7f3a700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6e87c700 1 -- 172.21.15.32:0/3613023331 >> v1:172.21.15.32:6789/0 conn(0x7f3a7010da00 legacy=0x7f3a7010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6e87c700 1 -- 172.21.15.32:0/3613023331 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a700d1f70 con 0x7f3a700d7ca0 2021-04-09T18:29:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.666 7f3a6e87c700 1 --2- 172.21.15.32:0/3613023331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a700d7ca0 0x7f3a700d80a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7f3a64006650 tx=0x7f3a64009a00).ready entity=mon.1 client_cookie=d89274618be6f752 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:30.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a6e07b700 1 -- 172.21.15.32:0/3613023331 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a64009df0 con 0x7f3a700d7ca0 2021-04-09T18:29:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a6e07b700 1 -- 172.21.15.32:0/3613023331 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a640009b0 con 0x7f3a700d7ca0 2021-04-09T18:29:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a6e07b700 1 -- 172.21.15.32:0/3613023331 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a64009df0 con 0x7f3a700d7ca0 2021-04-09T18:29:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a7535c700 1 -- 172.21.15.32:0/3613023331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a700d7ca0 msgr2=0x7f3a700d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a7535c700 1 --2- 172.21.15.32:0/3613023331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a700d7ca0 0x7f3a700d80a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7f3a64006650 tx=0x7f3a64009a00).stop 2021-04-09T18:29:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a7535c700 1 -- 172.21.15.32:0/3613023331 shutdown_connections 2021-04-09T18:29:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a7535c700 1 --2- 172.21.15.32:0/3613023331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a700d7ca0 0x7f3a700d80a0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a7535c700 1 -- 172.21.15.32:0/3613023331 >> 172.21.15.32:0/3613023331 conn(0x7f3a700ce5f0 msgr2=0x7f3a700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a7535c700 1 -- 172.21.15.32:0/3613023331 shutdown_connections 2021-04-09T18:29:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.667 7f3a7535c700 1 -- 172.21.15.32:0/3613023331 wait complete. 2021-04-09T18:29:30.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.688 7f3a7535c700 1 Processor -- start 2021-04-09T18:29:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.688 7f3a7535c700 1 -- start start 2021-04-09T18:29:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.688 7f3a7535c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a70179610 0x7f3a70179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.688 7f3a7535c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7017a820 0x7f3a7017ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.688 7f3a7535c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a701949f0 0x7f3a70194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.688 7f3a7535c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a701b8d50 con 0x7f3a7017a820 2021-04-09T18:29:30.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.688 7f3a7535c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7017b240 con 0x7f3a70179610 2021-04-09T18:29:30.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.688 7f3a7535c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70195410 con 0x7f3a701949f0 2021-04-09T18:29:30.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.688 7f3a6e87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7017a820 0x7f3a7017ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6e87c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7017a820 0x7f3a7017ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54998/0 (socket says 172.21.15.32:54998) 2021-04-09T18:29:30.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6e87c700 1 -- 172.21.15.32:0/1780671823 learned_addr learned my addr 172.21.15.32:0/1780671823 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6f07d700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a70179610 0x7f3a70179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6f87e700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a701949f0 0x7f3a70194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6e87c700 1 -- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a701949f0 msgr2=0x7f3a70194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6e87c700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a701949f0 0x7f3a70194df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6e87c700 1 -- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a70179610 msgr2=0x7f3a70179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6e87c700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a70179610 0x7f3a70179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6e87c700 1 -- 172.21.15.32:0/1780671823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a70195630 con 0x7f3a7017a820 2021-04-09T18:29:30.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a6e87c700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7017a820 0x7f3a7017ac20 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f3a640096a0 tx=0x7f3a6400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a57fff700 1 -- 172.21.15.32:0/1780671823 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a64000c90 con 0x7f3a7017a820 2021-04-09T18:29:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a57fff700 1 -- 172.21.15.32:0/1780671823 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a64002030 con 0x7f3a7017a820 2021-04-09T18:29:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a57fff700 1 -- 172.21.15.32:0/1780671823 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a64002550 con 0x7f3a7017a820 2021-04-09T18:29:30.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a70195630 con 0x7f3a7017a820 2021-04-09T18:29:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.689 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a70195630 con 0x7f3a7017a820 2021-04-09T18:29:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.690 7f3a57fff700 1 -- 172.21.15.32:0/1780671823 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3a64002210 con 0x7f3a7017a820 2021-04-09T18:29:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.690 7f3a57fff700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a50018930 0x7f3a5001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:30.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.690 7f3a57fff700 1 -- 172.21.15.32:0/1780671823 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3a64032a70 con 0x7f3a7017a820 2021-04-09T18:29:30.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.690 7f3a6f07d700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a50018930 0x7f3a5001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:30.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.690 7f3a6f07d700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a50018930 0x7f3a5001ae10 crc :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:30.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.710 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7017a2f0 con 0x7f3a7017a820 2021-04-09T18:29:30.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.714 7f3a57fff700 1 -- 172.21.15.32:0/1780671823 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3a640009c0 con 0x7f3a7017a820 2021-04-09T18:29:30.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.896 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f3a7017a2f0 con 0x7f3a7017a820 2021-04-09T18:29:30.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.897 7f3a57fff700 1 -- 172.21.15.32:0/1780671823 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v168) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3a6400bd80 con 0x7f3a7017a820 2021-04-09T18:29:30.982 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:29:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a50018930 msgr2=0x7f3a5001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a50018930 0x7f3a5001ae10 crc :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7017a820 msgr2=0x7f3a7017ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7017a820 0x7f3a7017ac20 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f3a640096a0 tx=0x7f3a6400b990).stop 2021-04-09T18:29:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 shutdown_connections 2021-04-09T18:29:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3a50018930 0x7f3a5001ae10 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a7017a820 0x7f3a7017ac20 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a701949f0 0x7f3a70194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 --2- 172.21.15.32:0/1780671823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a70179610 0x7f3a70179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:30.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.899 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 >> 172.21.15.32:0/1780671823 conn(0x7f3a70092a00 msgr2=0x7f3a701b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:30.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.900 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 shutdown_connections 2021-04-09T18:29:30.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:30.900 7f3a7535c700 1 -- 172.21.15.32:0/1780671823 wait complete. 2021-04-09T18:29:30.995 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:29:30.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:31.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda5adf7700 1 Processor -- start 2021-04-09T18:29:31.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda5adf7700 1 -- start start 2021-04-09T18:29:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda5adf7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda5410d010 0x7fda540d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda5adf7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fda540d8490 0x7fda5410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda5adf7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5410d480 con 0x7fda540d1da0 2021-04-09T18:29:31.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda5adf7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda540d8d30 con 0x7fda5410d010 2021-04-09T18:29:31.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda5adf7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda540d9220 con 0x7fda540d8490 2021-04-09T18:29:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda58c13700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda5410d010 0x7fda540d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda53fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fda540d1da0 0x7fda540d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36362/0 (socket says 172.21.15.32:36362) 2021-04-09T18:29:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.059 7fda53fff700 1 -- 172.21.15.32:0/3617986463 learned_addr learned my addr 172.21.15.32:0/3617986463 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda59414700 1 --2- 172.21.15.32:0/3617986463 >> v2:172.21.15.86:3300/0 conn(0x7fda540d8490 0x7fda5410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:31.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda537fe700 1 -- 172.21.15.32:0/3617986463 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797325453 0 0) 0x7fda44001060 con 0x7fda540d1da0 2021-04-09T18:29:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda537fe700 1 -- 172.21.15.32:0/3617986463 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda40002fe0 con 0x7fda540d1da0 2021-04-09T18:29:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda58c13700 1 -- 172.21.15.32:0/3617986463 >> v2:172.21.15.86:3300/0 conn(0x7fda540d8490 msgr2=0x7fda5410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda58c13700 1 --2- 172.21.15.32:0/3617986463 >> v2:172.21.15.86:3300/0 conn(0x7fda540d8490 0x7fda5410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda58c13700 1 -- 172.21.15.32:0/3617986463 >> v1:172.21.15.32:6789/0 conn(0x7fda540d1da0 legacy=0x7fda540d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda58c13700 1 -- 172.21.15.32:0/3617986463 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5410e020 con 0x7fda5410d010 2021-04-09T18:29:31.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda58c13700 1 --2- 172.21.15.32:0/3617986463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5410d010 0x7fda540d1780 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7fda3c006650 tx=0x7fda3c009a80).ready entity=mon.0 client_cookie=5d955b5cfaee6679 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:31.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda537fe700 1 -- 172.21.15.32:0/3617986463 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda3c0009b0 con 0x7fda5410d010 2021-04-09T18:29:31.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda537fe700 1 -- 172.21.15.32:0/3617986463 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c00c0c0 con 0x7fda5410d010 2021-04-09T18:29:31.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda537fe700 1 -- 172.21.15.32:0/3617986463 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda3c00c510 con 0x7fda5410d010 2021-04-09T18:29:31.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda5adf7700 1 -- 172.21.15.32:0/3617986463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5410d010 msgr2=0x7fda540d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:31.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.060 7fda5adf7700 1 --2- 172.21.15.32:0/3617986463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5410d010 0x7fda540d1780 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7fda3c006650 tx=0x7fda3c009a80).stop 2021-04-09T18:29:31.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.061 7fda5adf7700 1 -- 172.21.15.32:0/3617986463 shutdown_connections 2021-04-09T18:29:31.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.061 7fda5adf7700 1 --2- 172.21.15.32:0/3617986463 >> v2:172.21.15.86:3300/0 conn(0x7fda540d8490 0x7fda5410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.061 7fda5adf7700 1 --2- 172.21.15.32:0/3617986463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5410d010 0x7fda540d1780 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.061 7fda5adf7700 1 -- 172.21.15.32:0/3617986463 >> 172.21.15.32:0/3617986463 conn(0x7fda540ce5f0 msgr2=0x7fda540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:31.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.061 7fda5adf7700 1 -- 172.21.15.32:0/3617986463 shutdown_connections 2021-04-09T18:29:31.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.061 7fda5adf7700 1 -- 172.21.15.32:0/3617986463 wait complete. 2021-04-09T18:29:31.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.081 7fda5adf7700 1 Processor -- start 2021-04-09T18:29:31.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.081 7fda5adf7700 1 -- start start 2021-04-09T18:29:31.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda5adf7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda540b8320 0x7fda540b8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:31.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda5adf7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda5418c540 0x7fda5418c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:31.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda5adf7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5418a350 0x7fda5418a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:31.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda5adf7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda54193080 con 0x7fda5418a350 2021-04-09T18:29:31.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda5adf7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5418bfc0 con 0x7fda5418c540 2021-04-09T18:29:31.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda5adf7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5418cf60 con 0x7fda540b8320 2021-04-09T18:29:31.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda59414700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5418a350 0x7fda5418a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:31.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda59414700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5418a350 0x7fda5418a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55008/0 (socket says 172.21.15.32:55008) 2021-04-09T18:29:31.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda59414700 1 -- 172.21.15.32:0/1864718582 learned_addr learned my addr 172.21.15.32:0/1864718582 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:31.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda58c13700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda540b8320 0x7fda540b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda53fff700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda5418c540 0x7fda5418c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda59414700 1 -- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda540b8320 msgr2=0x7fda540b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda59414700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda540b8320 0x7fda540b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda59414700 1 -- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda5418c540 msgr2=0x7fda5418c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda59414700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda5418c540 0x7fda5418c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda59414700 1 -- 172.21.15.32:0/1864718582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5408a650 con 0x7fda5418a350 2021-04-09T18:29:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda59414700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5418a350 0x7fda5418a750 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7fda4c00b580 tx=0x7fda4c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:31.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.082 7fda517fa700 1 -- 172.21.15.32:0/1864718582 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda4c00bd10 con 0x7fda5418a350 2021-04-09T18:29:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.083 7fda517fa700 1 -- 172.21.15.32:0/1864718582 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c006720 con 0x7fda5418a350 2021-04-09T18:29:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.083 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5408a650 con 0x7fda5418a350 2021-04-09T18:29:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.083 7fda517fa700 1 -- 172.21.15.32:0/1864718582 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda4c0024d0 con 0x7fda5418a350 2021-04-09T18:29:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.083 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5408a650 con 0x7fda5418a350 2021-04-09T18:29:31.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.083 7fda517fa700 1 -- 172.21.15.32:0/1864718582 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fda4c005330 con 0x7fda5418a350 2021-04-09T18:29:31.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.083 7fda517fa700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda38018930 0x7fda3801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:31.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.083 7fda517fa700 1 -- 172.21.15.32:0/1864718582 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fda4c032a00 con 0x7fda5418a350 2021-04-09T18:29:31.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.083 7fda58c13700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda38018930 0x7fda3801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:31.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.084 7fda58c13700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda38018930 0x7fda3801ae10 crc :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:31.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.103 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda54193080 con 0x7fda5418a350 2021-04-09T18:29:31.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.107 7fda517fa700 1 -- 172.21.15.32:0/1864718582 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fda4c006370 con 0x7fda5418a350 2021-04-09T18:29:31.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.298 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda54193080 con 0x7fda5418a350 2021-04-09T18:29:31.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.298 7fda517fa700 1 -- 172.21.15.32:0/1864718582 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 311 v311) v1 ==== 95+0+2788 (secure 0 0 0) 0x7fda4c006720 con 0x7fda5418a350 2021-04-09T18:29:31.384 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 311 2021-04-09T18:29:31.384 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:31.385 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":311,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2532077375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2532077375},{"type":"v1","addr":"172.21.15.32:6834","nonce":2532077375}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":311,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:29:26.675661","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10177":{"gid":10177,"name":"a","rank":0,"incarnation":308,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2200487686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2200487686},{"type":"v1","addr":"172.21.15.32:6835","nonce":2200487686}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-04-09T18:29:31.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda38018930 msgr2=0x7fda3801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:31.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda38018930 0x7fda3801ae10 crc :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5418a350 msgr2=0x7fda5418a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5418a350 0x7fda5418a750 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7fda4c00b580 tx=0x7fda4c00b920).stop 2021-04-09T18:29:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 shutdown_connections 2021-04-09T18:29:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda38018930 0x7fda3801ae10 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda5418a350 0x7fda5418a750 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda5418c540 0x7fda5418c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 --2- 172.21.15.32:0/1864718582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda540b8320 0x7fda540b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:31.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 >> 172.21.15.32:0/1864718582 conn(0x7fda54183b00 msgr2=0x7fda54193300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:31.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 shutdown_connections 2021-04-09T18:29:31.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:31.301 7fda5adf7700 1 -- 172.21.15.32:0/1864718582 wait complete. 2021-04-09T18:29:31.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 311, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:29:24.667843', 'modified': '2021-04-09 18:29:26.675661', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10177}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10177': {'gid': 10177, 'name': 'a', 'rank': 0, 'incarnation': 308, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2200487686', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2200487686}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2200487686}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:29:31.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:29:31.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:29:31.939 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:29:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:29:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:29:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 10177, 2021-04-09T18:29:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:29:31.940 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:29:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.1860273269999997, 2021-04-09T18:29:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 311, 2021-04-09T18:29:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 168, 2021-04-09T18:29:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 168, 2021-04-09T18:29:31.941 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 12.624973174999999 2021-04-09T18:29:31.942 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:29:31.945 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 10177, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.1860273269999997, "mdsmap_epoch": 311, "osdmap_epoch": 168, "osdmap_epoch_barrier": 168, "uptime": 12.624973174999999 } 2021-04-09T18:29:31.946 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:29:31.946 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:29:31.947 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:29:31.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:31.947 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:29:31.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:29:32.050 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:29:32.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:29:32.109 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:29:32.109 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:29:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:29:32.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:32.111 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:29:32.154 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:29:32.155 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:29:32.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:29:32.218 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:29:32.218 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:29:32.277 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:29:32.277 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:29:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:29:32.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:32.279 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:29:32.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:29:32.222 7f5c328a1f80 -1 init, newargv = 0x556affdd9270 newargc=9 2021-04-09T18:29:32.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[87871]: starting ceph client 2021-04-09T18:29:32.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[87871]: starting fuse 2021-04-09T18:29:32.324 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:29:32.325 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:29:32.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:32.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:29:32.328 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:29:32.328 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:29:32.329 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:29:32.330 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:29:32.331 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:29:32.332 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:29:32.332 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:29:32.333 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:29:32.334 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:29:32.335 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:29:32.336 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:29:32.336 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:29:32.337 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:29:32.338 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:29:32.339 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:29:32.339 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:29:32.340 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:29:32.341 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:29:32.342 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:29:32.342 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:29:32.343 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:29:32.344 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:29:32.447 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.87871.asok 2021-04-09T18:29:32.451 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.87871.asok 2021-04-09T18:29:32.451 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:32.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.87871.asok status 2021-04-09T18:29:32.576 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:29:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:29:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:29:32.577 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:29:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:29:32.578 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:29:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:29:32.579 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "87871", 2021-04-09T18:29:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:29:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:32.580 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:29:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:29:32.581 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 10689, 2021-04-09T18:29:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:29:32.582 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:29:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:29:32.583 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 10689 2021-04-09T18:29:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:29:32.584 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:29:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 732738277 2021-04-09T18:29:32.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:29:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:29:32.586 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:29:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:29:32.587 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 732738277 2021-04-09T18:29:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:29:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.10689 172.21.15.32:0/732738277", 2021-04-09T18:29:32.588 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/732738277", 2021-04-09T18:29:32.589 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:29:32.589 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 311, 2021-04-09T18:29:32.590 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 168, 2021-04-09T18:29:32.590 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:29:32.590 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:29:32.591 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:29:32.591 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:29:32.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:32.594 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:29:32.594 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:29:32.640 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:29:32.640 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:29:32.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:29:32.642 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:29:32.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:32.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.767 7f1946d07700 1 Processor -- start 2021-04-09T18:29:32.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.767 7f1946d07700 1 -- start start 2021-04-09T18:29:32.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.767 7f1946d07700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19400cffa0 0x7f19400d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:32.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.767 7f1946d07700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400d7280 con 0x7f19400d6c80 2021-04-09T18:29:32.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.767 7f1946d07700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400d83f0 con 0x7f19400d7800 2021-04-09T18:29:32.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.767 7f1946d07700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19400d7080 con 0x7f19400cffa0 2021-04-09T18:29:32.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f1944b23700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19400cffa0 0x7f19400d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:32.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f1945324700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f19400d6c80 0x7f194010cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36384/0 (socket says 172.21.15.32:36384) 2021-04-09T18:29:32.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f1945324700 1 -- 172.21.15.32:0/122638377 learned_addr learned my addr 172.21.15.32:0/122638377 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:32.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f193f7fe700 1 -- 172.21.15.32:0/122638377 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502529927 0 0) 0x7f19380009b0 con 0x7f19400d6c80 2021-04-09T18:29:32.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f193f7fe700 1 -- 172.21.15.32:0/122638377 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f192c002fe0 con 0x7f19400d6c80 2021-04-09T18:29:32.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f1944b23700 1 -- 172.21.15.32:0/122638377 >> v1:172.21.15.86:6789/0 conn(0x7f19400d7800 legacy=0x7f19400d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:32.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f1944b23700 1 -- 172.21.15.32:0/122638377 >> v1:172.21.15.32:6789/0 conn(0x7f19400d6c80 legacy=0x7f194010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:32.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f1944b23700 1 -- 172.21.15.32:0/122638377 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400d91e0 con 0x7f19400cffa0 2021-04-09T18:29:32.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f1944b23700 1 --2- 172.21.15.32:0/122638377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400cffa0 0x7f19400d2440 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f1928006650 tx=0x7f1928009a80).ready entity=mon.0 client_cookie=d391fba3dbfb12ea server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:32.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f193f7fe700 1 -- 172.21.15.32:0/122638377 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19280009b0 con 0x7f19400cffa0 2021-04-09T18:29:32.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f193f7fe700 1 -- 172.21.15.32:0/122638377 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192800c0c0 con 0x7f19400cffa0 2021-04-09T18:29:32.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f193f7fe700 1 -- 172.21.15.32:0/122638377 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f192800c510 con 0x7f19400cffa0 2021-04-09T18:29:32.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f1946d07700 1 -- 172.21.15.32:0/122638377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400cffa0 msgr2=0x7f19400d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:32.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.768 7f1946d07700 1 --2- 172.21.15.32:0/122638377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400cffa0 0x7f19400d2440 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f1928006650 tx=0x7f1928009a80).stop 2021-04-09T18:29:32.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.769 7f1946d07700 1 -- 172.21.15.32:0/122638377 shutdown_connections 2021-04-09T18:29:32.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.769 7f1946d07700 1 --2- 172.21.15.32:0/122638377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400cffa0 0x7f19400d2440 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:32.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.769 7f1946d07700 1 -- 172.21.15.32:0/122638377 >> 172.21.15.32:0/122638377 conn(0x7f19400ce5c0 msgr2=0x7f19400cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:32.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.769 7f1946d07700 1 -- 172.21.15.32:0/122638377 shutdown_connections 2021-04-09T18:29:32.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.769 7f1946d07700 1 -- 172.21.15.32:0/122638377 wait complete. 2021-04-09T18:29:32.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.789 7f1946d07700 1 Processor -- start 2021-04-09T18:29:32.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.789 7f1946d07700 1 -- start start 2021-04-09T18:29:32.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1946d07700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400ae100 0x7f19401acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:32.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1946d07700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401ac510 0x7f19401a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:32.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1946d07700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194019ac40 0x7f194019b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:32.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1946d07700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19401aca00 con 0x7f19400ae100 2021-04-09T18:29:32.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1946d07700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194019b660 con 0x7f19401ac510 2021-04-09T18:29:32.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1946d07700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194008a650 con 0x7f194019ac40 2021-04-09T18:29:32.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1944b23700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400ae100 0x7f19401acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:32.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1944b23700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400ae100 0x7f19401acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55034/0 (socket says 172.21.15.32:55034) 2021-04-09T18:29:32.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1944b23700 1 -- 172.21.15.32:0/3363202176 learned_addr learned my addr 172.21.15.32:0/3363202176 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:32.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f193ffff700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401ac510 0x7f19401a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:32.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1945324700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194019ac40 0x7f194019b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:32.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1944b23700 1 -- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194019ac40 msgr2=0x7f194019b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:32.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1944b23700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194019ac40 0x7f194019b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:32.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1944b23700 1 -- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401ac510 msgr2=0x7f19401a2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:32.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1944b23700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401ac510 0x7f19401a2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:32.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1944b23700 1 -- 172.21.15.32:0/3363202176 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194008a870 con 0x7f19400ae100 2021-04-09T18:29:32.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1945324700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194019ac40 0x7f194019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:32.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.790 7f1944b23700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400ae100 0x7f19401acf80 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f1928009650 tx=0x7f192800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.791 7f193d7fa700 1 -- 172.21.15.32:0/3363202176 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f192800bd50 con 0x7f19400ae100 2021-04-09T18:29:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.791 7f193d7fa700 1 -- 172.21.15.32:0/3363202176 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1928006030 con 0x7f19400ae100 2021-04-09T18:29:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.791 7f1946d07700 1 -- 172.21.15.32:0/3363202176 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194008a870 con 0x7f19400ae100 2021-04-09T18:29:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.791 7f193d7fa700 1 -- 172.21.15.32:0/3363202176 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19280054d0 con 0x7f19400ae100 2021-04-09T18:29:32.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.791 7f1946d07700 1 -- 172.21.15.32:0/3363202176 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194008a870 con 0x7f19400ae100 2021-04-09T18:29:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.791 7f193d7fa700 1 -- 172.21.15.32:0/3363202176 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f19280052e0 con 0x7f19400ae100 2021-04-09T18:29:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.791 7f193d7fa700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19240188d0 0x7f192401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.791 7f193d7fa700 1 -- 172.21.15.32:0/3363202176 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f19280329f0 con 0x7f19400ae100 2021-04-09T18:29:32.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.792 7f193ffff700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19240188d0 0x7f192401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:32.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.792 7f193ffff700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19240188d0 0x7f192401add0 crc :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:32.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.811 7f1946d07700 1 -- 172.21.15.32:0/3363202176 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194017caf0 con 0x7f19400ae100 2021-04-09T18:29:32.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:32.815 7f193d7fa700 1 -- 172.21.15.32:0/3363202176 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1928000940 con 0x7f19400ae100 2021-04-09T18:29:33.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.008 7f1946d07700 1 -- 172.21.15.32:0/3363202176 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f194017caf0 con 0x7f19400ae100 2021-04-09T18:29:33.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.008 7f193d7fa700 1 -- 172.21.15.32:0/3363202176 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 311 v311) v1 ==== 95+0+2788 (secure 0 0 0) 0x7f1928006030 con 0x7f19400ae100 2021-04-09T18:29:33.094 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 311 2021-04-09T18:29:33.095 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:33.095 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":311,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2532077375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2532077375},{"type":"v1","addr":"172.21.15.32:6834","nonce":2532077375}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":311,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:29:26.675661","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10177":{"gid":10177,"name":"a","rank":0,"incarnation":308,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2200487686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2200487686},{"type":"v1","addr":"172.21.15.32:6835","nonce":2200487686}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-04-09T18:29:33.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 -- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19240188d0 msgr2=0x7f192401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19240188d0 0x7f192401add0 crc :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 -- 172.21.15.32:0/3363202176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400ae100 msgr2=0x7f19401acf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400ae100 0x7f19401acf80 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f1928009650 tx=0x7f192800b960).stop 2021-04-09T18:29:33.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 -- 172.21.15.32:0/3363202176 shutdown_connections 2021-04-09T18:29:33.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19240188d0 0x7f192401add0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194019ac40 0x7f194019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401ac510 0x7f19401a2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 --2- 172.21.15.32:0/3363202176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400ae100 0x7f19401acf80 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 -- 172.21.15.32:0/3363202176 >> 172.21.15.32:0/3363202176 conn(0x7f1940099f20 msgr2=0x7f1940094920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:33.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 -- 172.21.15.32:0/3363202176 shutdown_connections 2021-04-09T18:29:33.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.011 7f1946d07700 1 -- 172.21.15.32:0/3363202176 wait complete. 2021-04-09T18:29:33.107 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2021-04-09T18:29:33.108 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mds_beacon_grace 2021-04-09T18:29:33.235 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:29:33.235 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_beacon_grace": "15.000000" 2021-04-09T18:29:33.235 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:29:33.241 INFO:tasks.cephfs.filesystem:_json_asok output: { "mds_beacon_grace": "15.000000" } 2021-04-09T18:29:33.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:33.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.304 7f1993a64700 1 Processor -- start 2021-04-09T18:29:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.305 7f1993a64700 1 -- start start 2021-04-09T18:29:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.305 7f1993a64700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f198c10d010 0x7f198c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.305 7f1993a64700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f198c0d1da0 0x7f198c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.305 7f1993a64700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f198c10d480 con 0x7f198c0d82c0 2021-04-09T18:29:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.305 7f1993a64700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198c0d8d30 con 0x7f198c10d010 2021-04-09T18:29:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.305 7f1993a64700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198c0d9220 con 0x7f198c0d1da0 2021-04-09T18:29:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.305 7f1991880700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f198c10d010 0x7f198c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.305 7f1992081700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f198c0d82c0 0x7f198c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36398/0 (socket says 172.21.15.32:36398) 2021-04-09T18:29:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.305 7f1992081700 1 -- 172.21.15.32:0/3113780356 learned_addr learned my addr 172.21.15.32:0/3113780356 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f199107f700 1 --2- 172.21.15.32:0/3113780356 >> v2:172.21.15.86:3300/0 conn(0x7f198c0d1da0 0x7f198c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f199087e700 1 -- 172.21.15.32:0/3113780356 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577230442 0 0) 0x7f1984000d30 con 0x7f198c0d82c0 2021-04-09T18:29:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f199087e700 1 -- 172.21.15.32:0/3113780356 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1978002fe0 con 0x7f198c0d82c0 2021-04-09T18:29:33.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f1991880700 1 -- 172.21.15.32:0/3113780356 >> v2:172.21.15.86:3300/0 conn(0x7f198c0d1da0 msgr2=0x7f198c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:33.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f1991880700 1 --2- 172.21.15.32:0/3113780356 >> v2:172.21.15.86:3300/0 conn(0x7f198c0d1da0 0x7f198c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f1991880700 1 -- 172.21.15.32:0/3113780356 >> v1:172.21.15.32:6789/0 conn(0x7f198c0d82c0 legacy=0x7f198c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f1991880700 1 -- 172.21.15.32:0/3113780356 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c0d86f0 con 0x7f198c10d010 2021-04-09T18:29:33.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f1991880700 1 --2- 172.21.15.32:0/3113780356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c10d010 0x7f198c0d1780 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f1988006650 tx=0x7f1988009a80).ready entity=mon.0 client_cookie=5525a5ed60cbdd17 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:33.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f199087e700 1 -- 172.21.15.32:0/3113780356 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19880009b0 con 0x7f198c10d010 2021-04-09T18:29:33.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f199087e700 1 -- 172.21.15.32:0/3113780356 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198800c0c0 con 0x7f198c10d010 2021-04-09T18:29:33.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f199087e700 1 -- 172.21.15.32:0/3113780356 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198800c510 con 0x7f198c10d010 2021-04-09T18:29:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f1993a64700 1 -- 172.21.15.32:0/3113780356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c10d010 msgr2=0x7f198c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.306 7f1993a64700 1 --2- 172.21.15.32:0/3113780356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c10d010 0x7f198c0d1780 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f1988006650 tx=0x7f1988009a80).stop 2021-04-09T18:29:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.307 7f1993a64700 1 -- 172.21.15.32:0/3113780356 shutdown_connections 2021-04-09T18:29:33.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.307 7f1993a64700 1 --2- 172.21.15.32:0/3113780356 >> v2:172.21.15.86:3300/0 conn(0x7f198c0d1da0 0x7f198c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.307 7f1993a64700 1 --2- 172.21.15.32:0/3113780356 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c10d010 0x7f198c0d1780 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.307 7f1993a64700 1 -- 172.21.15.32:0/3113780356 >> 172.21.15.32:0/3113780356 conn(0x7f198c0ce5f0 msgr2=0x7f198c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:33.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.307 7f1993a64700 1 -- 172.21.15.32:0/3113780356 shutdown_connections 2021-04-09T18:29:33.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.307 7f1993a64700 1 -- 172.21.15.32:0/3113780356 wait complete. 2021-04-09T18:29:33.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.327 7f1993a64700 1 Processor -- start 2021-04-09T18:29:33.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.327 7f1993a64700 1 -- start start 2021-04-09T18:29:33.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f1993a64700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198c184a30 0x7f198c1acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f1993a64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c1a6b90 0x7f198c1a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f1993a64700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198c1a5150 0x7f198c1a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f1993a64700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c1a75b0 con 0x7f198c1a6b90 2021-04-09T18:29:33.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f1993a64700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c184590 con 0x7f198c184a30 2021-04-09T18:29:33.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f1993a64700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198c1a5b70 con 0x7f198c1a5150 2021-04-09T18:29:33.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f199107f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c1a6b90 0x7f198c1a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f199107f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c1a6b90 0x7f198c1a6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55046/0 (socket says 172.21.15.32:55046) 2021-04-09T18:29:33.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f199107f700 1 -- 172.21.15.32:0/1205749274 learned_addr learned my addr 172.21.15.32:0/1205749274 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:33.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f1991880700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198c184a30 0x7f198c1acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f1992081700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198c1a5150 0x7f198c1a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f199107f700 1 -- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198c1a5150 msgr2=0x7f198c1a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f199107f700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198c1a5150 0x7f198c1a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f199107f700 1 -- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198c184a30 msgr2=0x7f198c1acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f199107f700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198c184a30 0x7f198c1acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f199107f700 1 -- 172.21.15.32:0/1205749274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c1a5d90 con 0x7f198c1a6b90 2021-04-09T18:29:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f199107f700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c1a6b90 0x7f198c1a6f90 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f198000b560 tx=0x7f198000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f197e7fc700 1 -- 172.21.15.32:0/1205749274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198000bcf0 con 0x7f198c1a6b90 2021-04-09T18:29:33.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f197e7fc700 1 -- 172.21.15.32:0/1205749274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1980000c90 con 0x7f198c1a6b90 2021-04-09T18:29:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f1993a64700 1 -- 172.21.15.32:0/1205749274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198c1a5d90 con 0x7f198c1a6b90 2021-04-09T18:29:33.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.328 7f197e7fc700 1 -- 172.21.15.32:0/1205749274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1980002220 con 0x7f198c1a6b90 2021-04-09T18:29:33.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.329 7f1993a64700 1 -- 172.21.15.32:0/1205749274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198c1a5d90 con 0x7f198c1a6b90 2021-04-09T18:29:33.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.329 7f197e7fc700 1 -- 172.21.15.32:0/1205749274 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f19800052e0 con 0x7f198c1a6b90 2021-04-09T18:29:33.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.329 7f197e7fc700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1974018930 0x7f197401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.329 7f197e7fc700 1 -- 172.21.15.32:0/1205749274 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1980032a20 con 0x7f198c1a6b90 2021-04-09T18:29:33.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.330 7f1991880700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1974018930 0x7f197401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.330 7f1991880700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1974018930 0x7f197401ae10 crc :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:33.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.349 7f1993a64700 1 -- 172.21.15.32:0/1205749274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198c1b7080 con 0x7f198c1a6b90 2021-04-09T18:29:33.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.353 7f197e7fc700 1 -- 172.21.15.32:0/1205749274 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1980000c90 con 0x7f198c1a6b90 2021-04-09T18:29:33.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.543 7f1993a64700 1 -- 172.21.15.32:0/1205749274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f198c10d480 con 0x7f198c1a6b90 2021-04-09T18:29:33.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.543 7f197e7fc700 1 -- 172.21.15.32:0/1205749274 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 311 v311) v1 ==== 95+0+2788 (secure 0 0 0) 0x7f1980006b00 con 0x7f198c1a6b90 2021-04-09T18:29:33.629 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 311 2021-04-09T18:29:33.629 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:33.630 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":311,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10530,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2532077375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2532077375},{"type":"v1","addr":"172.21.15.32:6834","nonce":2532077375}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":311,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:29:26.675661","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10177":{"gid":10177,"name":"a","rank":0,"incarnation":308,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2200487686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2200487686},{"type":"v1","addr":"172.21.15.32:6835","nonce":2200487686}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-04-09T18:29:33.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 -- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1974018930 msgr2=0x7f197401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1974018930 0x7f197401ae10 crc :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 -- 172.21.15.32:0/1205749274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c1a6b90 msgr2=0x7f198c1a6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c1a6b90 0x7f198c1a6f90 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f198000b560 tx=0x7f198000b900).stop 2021-04-09T18:29:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 -- 172.21.15.32:0/1205749274 shutdown_connections 2021-04-09T18:29:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1974018930 0x7f197401ae10 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f198c1a6b90 0x7f198c1a6f90 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f198c1a5150 0x7f198c1a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 --2- 172.21.15.32:0/1205749274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f198c184a30 0x7f198c1acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 -- 172.21.15.32:0/1205749274 >> 172.21.15.32:0/1205749274 conn(0x7f198c191560 msgr2=0x7f198c1b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:33.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 -- 172.21.15.32:0/1205749274 shutdown_connections 2021-04-09T18:29:33.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.546 7f1993a64700 1 -- 172.21.15.32:0/1205749274 wait complete. 2021-04-09T18:29:33.641 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 3 2021-04-09T18:29:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.709 7ff09682d700 1 Processor -- start 2021-04-09T18:29:33.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.709 7ff09682d700 1 -- start start 2021-04-09T18:29:33.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff09682d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff09010da00 0x7ff09010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff09682d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09010d480 con 0x7ff0900d7ca0 2021-04-09T18:29:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff09682d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0900d1f70 con 0x7ff09010d010 2021-04-09T18:29:33.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff09682d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900d9220 con 0x7ff09010da00 2021-04-09T18:29:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff087fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0900d7ca0 0x7ff0900d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36412/0 (socket says 172.21.15.32:36412) 2021-04-09T18:29:33.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff087fff700 1 -- 172.21.15.32:0/1371161736 learned_addr learned my addr 172.21.15.32:0/1371161736 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff094e4a700 1 --2- 172.21.15.32:0/1371161736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff09010da00 0x7ff09010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff08f7fe700 1 -- 172.21.15.32:0/1371161736 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53818889 0 0) 0x7ff078001100 con 0x7ff0900d7ca0 2021-04-09T18:29:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff08f7fe700 1 -- 172.21.15.32:0/1371161736 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff070002fe0 con 0x7ff0900d7ca0 2021-04-09T18:29:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff08f7fe700 1 -- 172.21.15.32:0/1371161736 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229455487 0 0) 0x7ff080000c60 con 0x7ff09010d010 2021-04-09T18:29:33.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff08f7fe700 1 -- 172.21.15.32:0/1371161736 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0700042c0 con 0x7ff09010d010 2021-04-09T18:29:33.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff094e4a700 1 -- 172.21.15.32:0/1371161736 >> v1:172.21.15.86:6789/0 conn(0x7ff09010d010 legacy=0x7ff0900d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff094e4a700 1 -- 172.21.15.32:0/1371161736 >> v1:172.21.15.32:6789/0 conn(0x7ff0900d7ca0 legacy=0x7ff0900d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff094e4a700 1 -- 172.21.15.32:0/1371161736 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff090112100 con 0x7ff09010da00 2021-04-09T18:29:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.710 7ff094e4a700 1 --2- 172.21.15.32:0/1371161736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff09010da00 0x7ff09010de00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7ff088006e50 tx=0x7ff088009e70).ready entity=mon.2 client_cookie=226c947e03600d08 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:33.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff08f7fe700 1 -- 172.21.15.32:0/1371161736 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff088006a20 con 0x7ff09010da00 2021-04-09T18:29:33.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff08f7fe700 1 -- 172.21.15.32:0/1371161736 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff088006790 con 0x7ff09010da00 2021-04-09T18:29:33.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff08f7fe700 1 -- 172.21.15.32:0/1371161736 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff08800b700 con 0x7ff09010da00 2021-04-09T18:29:33.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff09682d700 1 -- 172.21.15.32:0/1371161736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff09010da00 msgr2=0x7ff09010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff09682d700 1 --2- 172.21.15.32:0/1371161736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff09010da00 0x7ff09010de00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7ff088006e50 tx=0x7ff088009e70).stop 2021-04-09T18:29:33.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff09682d700 1 -- 172.21.15.32:0/1371161736 shutdown_connections 2021-04-09T18:29:33.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff09682d700 1 --2- 172.21.15.32:0/1371161736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff09010da00 0x7ff09010de00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff09682d700 1 -- 172.21.15.32:0/1371161736 >> 172.21.15.32:0/1371161736 conn(0x7ff0900ce5f0 msgr2=0x7ff0900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:33.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff09682d700 1 -- 172.21.15.32:0/1371161736 shutdown_connections 2021-04-09T18:29:33.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.711 7ff09682d700 1 -- 172.21.15.32:0/1371161736 wait complete. 2021-04-09T18:29:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.731 7ff09682d700 1 Processor -- start 2021-04-09T18:29:33.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff09682d700 1 -- start start 2021-04-09T18:29:33.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff09682d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff090190ee0 0x7ff0900a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff09682d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0900a3570 0x7ff0900a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff09682d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0900a6150 0x7ff0900a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff09682d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900a55b0 con 0x7ff0900a3570 2021-04-09T18:29:33.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff09682d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900a3970 con 0x7ff0900a6150 2021-04-09T18:29:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff09682d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900a6550 con 0x7ff090190ee0 2021-04-09T18:29:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff087fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0900a3570 0x7ff0900a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff087fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0900a3570 0x7ff0900a5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55058/0 (socket says 172.21.15.32:55058) 2021-04-09T18:29:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff087fff700 1 -- 172.21.15.32:0/3197413964 learned_addr learned my addr 172.21.15.32:0/3197413964 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:33.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff094e4a700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0900a6150 0x7ff0900a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff08ffff700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff090190ee0 0x7ff0900a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff087fff700 1 -- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff090190ee0 msgr2=0x7ff0900a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff087fff700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff090190ee0 0x7ff0900a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff087fff700 1 -- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0900a6150 msgr2=0x7ff0900a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:33.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.732 7ff087fff700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0900a6150 0x7ff0900a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:33.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.733 7ff087fff700 1 -- 172.21.15.32:0/3197413964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0900a3d80 con 0x7ff0900a3570 2021-04-09T18:29:33.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.733 7ff087fff700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0900a3570 0x7ff0900a5b30 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7ff07800b560 tx=0x7ff07800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:33.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.733 7ff08d7fa700 1 -- 172.21.15.32:0/3197413964 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff078006030 con 0x7ff0900a3570 2021-04-09T18:29:33.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.733 7ff08d7fa700 1 -- 172.21.15.32:0/3197413964 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078000960 con 0x7ff0900a3570 2021-04-09T18:29:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.733 7ff08d7fa700 1 -- 172.21.15.32:0/3197413964 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff078004620 con 0x7ff0900a3570 2021-04-09T18:29:33.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.733 7ff09682d700 1 -- 172.21.15.32:0/3197413964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0900a3d80 con 0x7ff0900a3570 2021-04-09T18:29:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.733 7ff09682d700 1 -- 172.21.15.32:0/3197413964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0900a3d80 con 0x7ff0900a3570 2021-04-09T18:29:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.733 7ff08d7fa700 1 -- 172.21.15.32:0/3197413964 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff078009fe0 con 0x7ff0900a3570 2021-04-09T18:29:33.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.734 7ff08d7fa700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff068018930 0x7ff06801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.734 7ff08d7fa700 1 -- 172.21.15.32:0/3197413964 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff0780329b0 con 0x7ff0900a3570 2021-04-09T18:29:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.734 7ff08ffff700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff068018930 0x7ff06801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:33.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.734 7ff08ffff700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff068018930 0x7ff06801adc0 crc :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:33.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.754 7ff09682d700 1 -- 172.21.15.32:0/3197413964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09010d480 con 0x7ff0900a3570 2021-04-09T18:29:33.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.757 7ff08d7fa700 1 -- 172.21.15.32:0/3197413964 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff078000940 con 0x7ff0900a3570 2021-04-09T18:29:34.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:33.949 7ff09682d700 1 -- 172.21.15.32:0/3197413964 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "3"} v 0) v1 -- 0x7ff09010d480 con 0x7ff0900a3570 2021-04-09T18:29:34.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.691 7ff08d7fa700 1 -- 172.21.15.32:0/3197413964 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "3"}]=0 v312) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff078000940 con 0x7ff0900a3570 2021-04-09T18:29:34.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 -- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff068018930 msgr2=0x7ff06801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:34.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff068018930 0x7ff06801adc0 crc :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:34.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 -- 172.21.15.32:0/3197413964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0900a3570 msgr2=0x7ff0900a5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:34.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0900a3570 0x7ff0900a5b30 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7ff07800b560 tx=0x7ff07800b900).stop 2021-04-09T18:29:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 -- 172.21.15.32:0/3197413964 shutdown_connections 2021-04-09T18:29:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff068018930 0x7ff06801adc0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0900a3570 0x7ff0900a5b30 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0900a6150 0x7ff0900a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:34.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 --2- 172.21.15.32:0/3197413964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff090190ee0 0x7ff0900a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:34.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 -- 172.21.15.32:0/3197413964 >> 172.21.15.32:0/3197413964 conn(0x7ff0900bbda0 msgr2=0x7ff0901b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:34.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 -- 172.21.15.32:0/3197413964 shutdown_connections 2021-04-09T18:29:34.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.717 7ff09682d700 1 -- 172.21.15.32:0/3197413964 wait complete. 2021-04-09T18:29:34.815 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:29:34.815 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:29:34.825 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:29:34.825 INFO:tasks.cephfs.test_failover:waiting for insufficient standby daemon warning 2021-04-09T18:29:34.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:29:35.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.934 7fe7afbe0700 1 Processor -- start 2021-04-09T18:29:35.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.934 7fe7afbe0700 1 -- start start 2021-04-09T18:29:35.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7afbe0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7a80d6c80 0x7fe7a80d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:35.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7afbe0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a80d7280 con 0x7fe7a80d2440 2021-04-09T18:29:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7afbe0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a80d2840 con 0x7fe7a80cffa0 2021-04-09T18:29:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7afbe0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a80d7080 con 0x7fe7a80d6c80 2021-04-09T18:29:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7ad1fb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7a80d6c80 0x7fe7a80d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:35.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7ad1fb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe7a80d6c80 0x7fe7a80d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55070/0 (socket says 172.21.15.32:55070) 2021-04-09T18:29:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7ae1fd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe7a80d2440 0x7fe7a810cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36426/0 (socket says 172.21.15.32:36426) 2021-04-09T18:29:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7ad1fb700 1 -- 172.21.15.32:0/3536426843 learned_addr learned my addr 172.21.15.32:0/3536426843 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7ad1fb700 1 -- 172.21.15.32:0/3536426843 >> v1:172.21.15.86:6789/0 conn(0x7fe7a80cffa0 legacy=0x7fe7a80d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7ad1fb700 1 -- 172.21.15.32:0/3536426843 >> v1:172.21.15.32:6789/0 conn(0x7fe7a80d2440 legacy=0x7fe7a810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:35.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7ac9fa700 1 -- 172.21.15.32:0/3536426843 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703384745 0 0) 0x7fe7a0000d30 con 0x7fe7a80d2440 2021-04-09T18:29:35.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.935 7fe7ad1fb700 1 -- 172.21.15.32:0/3536426843 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a80d91e0 con 0x7fe7a80d6c80 2021-04-09T18:29:35.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.936 7fe7ad1fb700 1 --2- 172.21.15.32:0/3536426843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a80d6c80 0x7fe7a80d7ff0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 rx=0x7fe798006700 tx=0x7fe798009b00).ready entity=mon.0 client_cookie=abcd30d629505d46 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:35.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.936 7fe7ac9fa700 1 -- 172.21.15.32:0/3536426843 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe79800c0c0 con 0x7fe7a80d6c80 2021-04-09T18:29:35.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.936 7fe7ac9fa700 1 -- 172.21.15.32:0/3536426843 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79800c400 con 0x7fe7a80d6c80 2021-04-09T18:29:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.936 7fe7ac9fa700 1 -- 172.21.15.32:0/3536426843 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe79800c8e0 con 0x7fe7a80d6c80 2021-04-09T18:29:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.936 7fe7afbe0700 1 -- 172.21.15.32:0/3536426843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a80d6c80 msgr2=0x7fe7a80d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.936 7fe7afbe0700 1 --2- 172.21.15.32:0/3536426843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a80d6c80 0x7fe7a80d7ff0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 rx=0x7fe798006700 tx=0x7fe798009b00).stop 2021-04-09T18:29:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.937 7fe7afbe0700 1 -- 172.21.15.32:0/3536426843 shutdown_connections 2021-04-09T18:29:35.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.937 7fe7afbe0700 1 --2- 172.21.15.32:0/3536426843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a80d6c80 0x7fe7a80d7ff0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:35.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.937 7fe7afbe0700 1 -- 172.21.15.32:0/3536426843 >> 172.21.15.32:0/3536426843 conn(0x7fe7a80ce5c0 msgr2=0x7fe7a80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:35.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.937 7fe7afbe0700 1 -- 172.21.15.32:0/3536426843 shutdown_connections 2021-04-09T18:29:35.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.937 7fe7afbe0700 1 -- 172.21.15.32:0/3536426843 wait complete. 2021-04-09T18:29:35.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.957 7fe7afbe0700 1 Processor -- start 2021-04-09T18:29:35.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.957 7fe7afbe0700 1 -- start start 2021-04-09T18:29:35.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.957 7fe7afbe0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7a81a6250 0x7fe7a81a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:35.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.957 7fe7afbe0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a81a7470 0x7fe7a81a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7afbe0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7a81aba60 0x7fe7a81abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7afbe0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a81a6c70 con 0x7fe7a81a7470 2021-04-09T18:29:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7afbe0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a81a6ef0 con 0x7fe7a81aba60 2021-04-09T18:29:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7afbe0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a81a7e90 con 0x7fe7a81a6250 2021-04-09T18:29:35.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad1fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a81a7470 0x7fe7a81a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:35.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad1fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a81a7470 0x7fe7a81a7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55072/0 (socket says 172.21.15.32:55072) 2021-04-09T18:29:35.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad1fb700 1 -- 172.21.15.32:0/3940507797 learned_addr learned my addr 172.21.15.32:0/3940507797 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:35.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ae1fd700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7a81aba60 0x7fe7a81abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:35.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad9fc700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7a81a6250 0x7fe7a81a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:35.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad1fb700 1 -- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7a81a6250 msgr2=0x7fe7a81a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:35.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad1fb700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7a81a6250 0x7fe7a81a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:35.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad1fb700 1 -- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7a81aba60 msgr2=0x7fe7a81abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:35.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad1fb700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7a81aba60 0x7fe7a81abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:35.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad1fb700 1 -- 172.21.15.32:0/3940507797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a8092900 con 0x7fe7a81a7470 2021-04-09T18:29:35.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7ad1fb700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a81a7470 0x7fe7a81a7870 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7fe79800b580 tx=0x7fe79800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:35.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe79e7fc700 1 -- 172.21.15.32:0/3940507797 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe798006470 con 0x7fe7a81a7470 2021-04-09T18:29:35.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe79e7fc700 1 -- 172.21.15.32:0/3940507797 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79800bd10 con 0x7fe7a81a7470 2021-04-09T18:29:35.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe79e7fc700 1 -- 172.21.15.32:0/3940507797 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe798005560 con 0x7fe7a81a7470 2021-04-09T18:29:35.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a8092900 con 0x7fe7a81a7470 2021-04-09T18:29:35.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.958 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a8092900 con 0x7fe7a81a7470 2021-04-09T18:29:35.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.959 7fe79e7fc700 1 -- 172.21.15.32:0/3940507797 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe798009420 con 0x7fe7a81a7470 2021-04-09T18:29:35.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.959 7fe79e7fc700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7900188d0 0x7fe79001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:35.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.959 7fe79e7fc700 1 -- 172.21.15.32:0/3940507797 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe798032a00 con 0x7fe7a81a7470 2021-04-09T18:29:35.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.959 7fe7ad9fc700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7900188d0 0x7fe79001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:35.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.959 7fe7ad9fc700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7900188d0 0x7fe79001add0 crc :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:35.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.979 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a81a6c70 con 0x7fe7a81a7470 2021-04-09T18:29:35.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:34.983 7fe79e7fc700 1 -- 172.21.15.32:0/3940507797 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe798006030 con 0x7fe7a81a7470 2021-04-09T18:29:35.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.170 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe7a81a6c70 con 0x7fe7a81a7470 2021-04-09T18:29:35.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.171 7fe79e7fc700 1 -- 172.21.15.32:0/3940507797 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7fe798009420 con 0x7fe7a81a7470 2021-04-09T18:29:35.257 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:35.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7900188d0 msgr2=0x7fe79001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:35.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7900188d0 0x7fe79001add0 crc :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:35.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a81a7470 msgr2=0x7fe7a81a7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:35.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a81a7470 0x7fe7a81a7870 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7fe79800b580 tx=0x7fe79800b920).stop 2021-04-09T18:29:35.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 shutdown_connections 2021-04-09T18:29:35.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe7900188d0 0x7fe79001add0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:35.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7a81a7470 0x7fe7a81a7870 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:35.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7a81aba60 0x7fe7a81abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:35.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 --2- 172.21.15.32:0/3940507797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7a81a6250 0x7fe7a81a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:35.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.173 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 >> 172.21.15.32:0/3940507797 conn(0x7fe7a818ce60 msgr2=0x7fe7a8094920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:35.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.174 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 shutdown_connections 2021-04-09T18:29:35.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:35.174 7fe7afbe0700 1 -- 172.21.15.32:0/3940507797 wait complete. 2021-04-09T18:29:35.269 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:29:35.270 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:29:35.271 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:29:37.833 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:29:37.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:29:40.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:29:40.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.345 7f70b9743700 1 Processor -- start 2021-04-09T18:29:40.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.345 7f70b9743700 1 -- start start 2021-04-09T18:29:40.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.345 7f70b9743700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70b40d6c80 0x7f70b40d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:40.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.345 7f70b9743700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b40d2440 0x7f70b410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:40.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.345 7f70b9743700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b40d7280 con 0x7f70b40cffa0 2021-04-09T18:29:40.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.345 7f70b9743700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b40d7080 con 0x7f70b40d6c80 2021-04-09T18:29:40.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.345 7f70b9743700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b40d91e0 con 0x7f70b40d2440 2021-04-09T18:29:40.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2d48700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70b40d6c80 0x7f70b40d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:40.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2d48700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f70b40d6c80 0x7f70b40d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55084/0 (socket says 172.21.15.32:55084) 2021-04-09T18:29:40.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2d48700 1 -- 172.21.15.32:0/1267250097 learned_addr learned my addr 172.21.15.32:0/1267250097 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b3d4a700 1 --2- 172.21.15.32:0/1267250097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b40d2440 0x7f70b410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2d48700 1 -- 172.21.15.32:0/1267250097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b40d2440 msgr2=0x7f70b410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2d48700 1 --2- 172.21.15.32:0/1267250097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b40d2440 0x7f70b410cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2d48700 1 -- 172.21.15.32:0/1267250097 >> v1:172.21.15.86:6789/0 conn(0x7f70b40cffa0 legacy=0x7f70b40d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:40.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2d48700 1 -- 172.21.15.32:0/1267250097 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b40d2840 con 0x7f70b40d6c80 2021-04-09T18:29:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2d48700 1 --2- 172.21.15.32:0/1267250097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b40d6c80 0x7f70b40d7ff0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7f70a8006700 tx=0x7f70a8009b80).ready entity=mon.0 client_cookie=dcc6a14a75f731fc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2547700 1 -- 172.21.15.32:0/1267250097 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70a800c0c0 con 0x7f70b40d6c80 2021-04-09T18:29:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2547700 1 -- 172.21.15.32:0/1267250097 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a800c440 con 0x7f70b40d6c80 2021-04-09T18:29:40.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b2547700 1 -- 172.21.15.32:0/1267250097 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70a800c920 con 0x7f70b40d6c80 2021-04-09T18:29:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b9743700 1 -- 172.21.15.32:0/1267250097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b40d6c80 msgr2=0x7f70b40d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.346 7f70b9743700 1 --2- 172.21.15.32:0/1267250097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b40d6c80 0x7f70b40d7ff0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7f70a8006700 tx=0x7f70a8009b80).stop 2021-04-09T18:29:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.347 7f70b9743700 1 -- 172.21.15.32:0/1267250097 shutdown_connections 2021-04-09T18:29:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.347 7f70b9743700 1 --2- 172.21.15.32:0/1267250097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b40d2440 0x7f70b410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.347 7f70b9743700 1 --2- 172.21.15.32:0/1267250097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b40d6c80 0x7f70b40d7ff0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.347 7f70b9743700 1 -- 172.21.15.32:0/1267250097 >> 172.21.15.32:0/1267250097 conn(0x7f70b40ce5c0 msgr2=0x7f70b40cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.347 7f70b9743700 1 -- 172.21.15.32:0/1267250097 shutdown_connections 2021-04-09T18:29:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.347 7f70b9743700 1 -- 172.21.15.32:0/1267250097 wait complete. 2021-04-09T18:29:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.367 7f70b9743700 1 Processor -- start 2021-04-09T18:29:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.367 7f70b9743700 1 -- start start 2021-04-09T18:29:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b9743700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b41a7580 0x7f70b41a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b9743700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b41b8260 0x7f70b41b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b9743700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b41b8c80 0x7f70b408a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b9743700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b408c800 con 0x7f70b41b8c80 2021-04-09T18:29:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b9743700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b408bbb0 con 0x7f70b41b8260 2021-04-09T18:29:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b9743700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b408c360 con 0x7f70b41a7580 2021-04-09T18:29:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3d4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b41b8c80 0x7f70b408a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3d4a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b41b8c80 0x7f70b408a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55086/0 (socket says 172.21.15.32:55086) 2021-04-09T18:29:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3d4a700 1 -- 172.21.15.32:0/621307586 learned_addr learned my addr 172.21.15.32:0/621307586 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b2d48700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b41b8260 0x7f70b41b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3549700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b41a7580 0x7f70b41a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:40.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3d4a700 1 -- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b41a7580 msgr2=0x7f70b41a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:40.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3d4a700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b41a7580 0x7f70b41a7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3d4a700 1 -- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b41b8260 msgr2=0x7f70b41b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:40.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3d4a700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b41b8260 0x7f70b41b8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3d4a700 1 -- 172.21.15.32:0/621307586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b408ad00 con 0x7f70b41b8c80 2021-04-09T18:29:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3549700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b41a7580 0x7f70b41a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b3d4a700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b41b8c80 0x7f70b408a6e0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 rx=0x7f70ac0099a0 tx=0x7f70ac00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f709bfff700 1 -- 172.21.15.32:0/621307586 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70ac0066f0 con 0x7f70b41b8c80 2021-04-09T18:29:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f709bfff700 1 -- 172.21.15.32:0/621307586 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ac003030 con 0x7f70b41b8c80 2021-04-09T18:29:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b9743700 1 -- 172.21.15.32:0/621307586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b408ad00 con 0x7f70b41b8c80 2021-04-09T18:29:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f709bfff700 1 -- 172.21.15.32:0/621307586 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70ac001470 con 0x7f70b41b8c80 2021-04-09T18:29:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.368 7f70b9743700 1 -- 172.21.15.32:0/621307586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b408ad00 con 0x7f70b41b8c80 2021-04-09T18:29:40.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.369 7f709bfff700 1 -- 172.21.15.32:0/621307586 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f70ac0011f0 con 0x7f70b41b8c80 2021-04-09T18:29:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.369 7f709bfff700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7094018930 0x7f709401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.369 7f709bfff700 1 -- 172.21.15.32:0/621307586 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f70ac032da0 con 0x7f70b41b8c80 2021-04-09T18:29:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.369 7f70b3549700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7094018930 0x7f709401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:40.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.370 7f70b3549700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7094018930 0x7f709401ae10 crc :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:40.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.389 7f70b9743700 1 -- 172.21.15.32:0/621307586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b41b7060 con 0x7f70b41b8c80 2021-04-09T18:29:40.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.393 7f709bfff700 1 -- 172.21.15.32:0/621307586 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f70ac00be30 con 0x7f70b41b8c80 2021-04-09T18:29:40.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.580 7f70b9743700 1 -- 172.21.15.32:0/621307586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f70b41b7060 con 0x7f70b41b8c80 2021-04-09T18:29:40.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.580 7f709bfff700 1 -- 172.21.15.32:0/621307586 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7f70ac01fae0 con 0x7f70b41b8c80 2021-04-09T18:29:40.666 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:40.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 -- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7094018930 msgr2=0x7f709401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7094018930 0x7f709401ae10 crc :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 -- 172.21.15.32:0/621307586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b41b8c80 msgr2=0x7f70b408a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b41b8c80 0x7f70b408a6e0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 rx=0x7f70ac0099a0 tx=0x7f70ac00bcf0).stop 2021-04-09T18:29:40.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 -- 172.21.15.32:0/621307586 shutdown_connections 2021-04-09T18:29:40.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7094018930 0x7f709401ae10 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b41b8c80 0x7f70b408a6e0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b41b8260 0x7f70b41b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 --2- 172.21.15.32:0/621307586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b41a7580 0x7f70b41a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:40.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.582 7f70b9743700 1 -- 172.21.15.32:0/621307586 >> 172.21.15.32:0/621307586 conn(0x7f70b41ad190 msgr2=0x7f70b41b7750 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:40.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.583 7f70b9743700 1 -- 172.21.15.32:0/621307586 shutdown_connections 2021-04-09T18:29:40.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:40.583 7f70b9743700 1 -- 172.21.15.32:0/621307586 wait complete. 2021-04-09T18:29:40.678 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:29:40.679 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:29:40.679 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:29:45.680 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:29:45.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.748 7f5b10bd3700 1 Processor -- start 2021-04-09T18:29:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.748 7f5b10bd3700 1 -- start start 2021-04-09T18:29:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.749 7f5b10bd3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5b0c0cffa0 0x7f5b0c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:45.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.749 7f5b10bd3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0d7800 0x7f5b0c0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.749 7f5b10bd3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0c0d7280 con 0x7f5b0c0d8220 2021-04-09T18:29:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.749 7f5b10bd3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0c0d91e0 con 0x7f5b0c0cffa0 2021-04-09T18:29:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.749 7f5b10bd3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c10dfe0 con 0x7f5b0c0d7800 2021-04-09T18:29:45.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.749 7f5b0b05f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5b0c0d8220 0x7f5b0c0d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36452/0 (socket says 172.21.15.32:36452) 2021-04-09T18:29:45.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.749 7f5b0b05f700 1 -- 172.21.15.32:0/2274275420 learned_addr learned my addr 172.21.15.32:0/2274275420 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:45.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190340355 0 0) 0x7f5b04000d30 con 0x7f5b0c0d8220 2021-04-09T18:29:45.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af8002fe0 con 0x7f5b0c0d8220 2021-04-09T18:29:45.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0a85e700 1 --2- 172.21.15.32:0/2274275420 >> v2:172.21.15.86:3300/0 conn(0x7f5b0c0cffa0 0x7f5b0c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0a05d700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0d7800 0x7f5b0c0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:45.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2869730662 0 0) 0x7f5b04001460 con 0x7f5b0c0d8220 2021-04-09T18:29:45.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0d7800 msgr2=0x7f5b0c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:45.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0d7800 0x7f5b0c0d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:45.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 >> v2:172.21.15.86:3300/0 conn(0x7f5b0c0cffa0 msgr2=0x7f5b0c0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:45.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 --2- 172.21.15.32:0/2274275420 >> v2:172.21.15.86:3300/0 conn(0x7f5b0c0cffa0 0x7f5b0c0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:45.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c10e200 con 0x7f5b0c0d8220 2021-04-09T18:29:45.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f5b04001980 con 0x7f5b0c0d8220 2021-04-09T18:29:45.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 >> v1:172.21.15.32:6789/0 conn(0x7f5b0c0d8220 legacy=0x7f5b0c0d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:45.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af8003a70 0x7f5af8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5af80067a0 0x7f5af8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af80095a0 0x7f5af800ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0a85e700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af8003a70 0x7f5af8005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80502f0 con 0x7f5af8003a70 2021-04-09T18:29:45.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80504d0 con 0x7f5af80067a0 2021-04-09T18:29:45.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80506f0 con 0x7f5af80095a0 2021-04-09T18:29:45.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0b05f700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af80095a0 0x7f5af800ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0a05d700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5af80067a0 0x7f5af8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0a85e700 1 -- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af80095a0 msgr2=0x7f5af800ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0a85e700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5af80095a0 0x7f5af800ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0a85e700 1 -- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5af80067a0 msgr2=0x7f5af8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0a85e700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5af80067a0 0x7f5af8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.750 7f5b0a85e700 1 -- 172.21.15.32:0/2274275420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af8050990 con 0x7f5af8003a70 2021-04-09T18:29:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.751 7f5b0a85e700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af8003a70 0x7f5af8005f00 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f5afc006360 tx=0x7f5afc009bb0).ready entity=mon.0 client_cookie=e57b3977e40a8091 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.751 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5afc006560 con 0x7f5af8003a70 2021-04-09T18:29:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.751 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc00c0c0 con 0x7f5af8003a70 2021-04-09T18:29:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.751 7f5b0985c700 1 -- 172.21.15.32:0/2274275420 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5afc00c550 con 0x7f5af8003a70 2021-04-09T18:29:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.751 7f5b10bd3700 1 -- 172.21.15.32:0/2274275420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af8003a70 msgr2=0x7f5af8005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.751 7f5b10bd3700 1 --2- 172.21.15.32:0/2274275420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5af8003a70 0x7f5af8005f00 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f5afc006360 tx=0x7f5afc009bb0).stop 2021-04-09T18:29:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.751 7f5b0b05f700 1 -- 172.21.15.32:0/2274275420 reap_dead start 2021-04-09T18:29:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.752 7f5b10bd3700 1 -- 172.21.15.32:0/2274275420 shutdown_connections 2021-04-09T18:29:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.752 7f5b10bd3700 1 -- 172.21.15.32:0/2274275420 >> 172.21.15.32:0/2274275420 conn(0x7f5b0c0ce5c0 msgr2=0x7f5b0c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.752 7f5b10bd3700 1 -- 172.21.15.32:0/2274275420 shutdown_connections 2021-04-09T18:29:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.752 7f5b10bd3700 1 -- 172.21.15.32:0/2274275420 wait complete. 2021-04-09T18:29:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b10bd3700 1 Processor -- start 2021-04-09T18:29:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b10bd3700 1 -- start start 2021-04-09T18:29:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b10bd3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b0c189d20 0x7f5b0c18a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b10bd3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0ba0d0 0x7f5b0c0ba4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b10bd3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b0c0baaf0 0x7f5b0c191250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b10bd3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c0baef0 con 0x7f5b0c189d20 2021-04-09T18:29:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b10bd3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c09f6e0 con 0x7f5b0c0baaf0 2021-04-09T18:29:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b10bd3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c192540 con 0x7f5b0c0ba0d0 2021-04-09T18:29:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b0a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b0c189d20 0x7f5b0c18a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b0a85e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b0c189d20 0x7f5b0c18a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55104/0 (socket says 172.21.15.32:55104) 2021-04-09T18:29:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.772 7f5b0a85e700 1 -- 172.21.15.32:0/2792004682 learned_addr learned my addr 172.21.15.32:0/2792004682 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:45.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0b05f700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b0c0baaf0 0x7f5b0c191250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0a05d700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0ba0d0 0x7f5b0c0ba4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0a85e700 1 -- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0ba0d0 msgr2=0x7f5b0c0ba4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0a85e700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0ba0d0 0x7f5b0c0ba4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:45.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0a85e700 1 -- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b0c0baaf0 msgr2=0x7f5b0c191250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0a85e700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b0c0baaf0 0x7f5b0c191250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0a85e700 1 -- 172.21.15.32:0/2792004682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c09a6f0 con 0x7f5b0c189d20 2021-04-09T18:29:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0b05f700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b0c0baaf0 0x7f5b0c191250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0a05d700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0ba0d0 0x7f5b0c0ba4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b0a85e700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b0c189d20 0x7f5b0c18a120 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f5afc006a00 tx=0x7f5afc00b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b037fe700 1 -- 172.21.15.32:0/2792004682 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5afc002030 con 0x7f5b0c189d20 2021-04-09T18:29:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b037fe700 1 -- 172.21.15.32:0/2792004682 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc0023b0 con 0x7f5b0c189d20 2021-04-09T18:29:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0c09a6f0 con 0x7f5b0c189d20 2021-04-09T18:29:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b037fe700 1 -- 172.21.15.32:0/2792004682 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5afc002890 con 0x7f5b0c189d20 2021-04-09T18:29:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.773 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0c09a6f0 con 0x7f5b0c189d20 2021-04-09T18:29:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.774 7f5b037fe700 1 -- 172.21.15.32:0/2792004682 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5afc000c90 con 0x7f5b0c189d20 2021-04-09T18:29:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.774 7f5b037fe700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5af00188c0 0x7f5af001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.774 7f5b037fe700 1 -- 172.21.15.32:0/2792004682 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5afc032790 con 0x7f5b0c189d20 2021-04-09T18:29:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.774 7f5b0a05d700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5af00188c0 0x7f5af001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.775 7f5b0a05d700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5af00188c0 0x7f5af001adc0 crc :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.793 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0c182d60 con 0x7f5b0c189d20 2021-04-09T18:29:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.797 7f5b037fe700 1 -- 172.21.15.32:0/2792004682 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5afc01f4d0 con 0x7f5b0c189d20 2021-04-09T18:29:46.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.983 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5b0c09f6e0 con 0x7f5b0c189d20 2021-04-09T18:29:46.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.984 7f5b037fe700 1 -- 172.21.15.32:0/2792004682 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7f5afc00bdb0 con 0x7f5b0c189d20 2021-04-09T18:29:46.070 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:46.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5af00188c0 msgr2=0x7f5af001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:46.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5af00188c0 0x7f5af001adc0 crc :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:46.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b0c189d20 msgr2=0x7f5b0c18a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:46.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b0c189d20 0x7f5b0c18a120 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 rx=0x7f5afc006a00 tx=0x7f5afc00b9c0).stop 2021-04-09T18:29:46.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 shutdown_connections 2021-04-09T18:29:46.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5af00188c0 0x7f5af001adc0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:46.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b0c0baaf0 0x7f5b0c191250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:46.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b0c0ba0d0 0x7f5b0c0ba4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:46.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 --2- 172.21.15.32:0/2792004682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b0c189d20 0x7f5b0c18a120 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:46.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 >> 172.21.15.32:0/2792004682 conn(0x7f5b0c090a90 msgr2=0x7f5b0c1832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:46.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 shutdown_connections 2021-04-09T18:29:46.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:45.986 7f5b10bd3700 1 -- 172.21.15.32:0/2792004682 wait complete. 2021-04-09T18:29:46.083 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:29:46.083 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2021-04-09T18:29:46.084 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:29:51.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:29:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.518 7f993757f700 1 Processor -- start 2021-04-09T18:29:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.519 7f993757f700 1 -- start start 2021-04-09T18:29:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.519 7f993757f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300cffa0 0x7f99300d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.519 7f993757f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f99300d6c80 0x7f993010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.519 7f993757f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99300d7280 con 0x7f99300d7800 2021-04-09T18:29:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.519 7f993757f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99300d7080 con 0x7f99300d6c80 2021-04-09T18:29:51.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.519 7f993757f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99300d8580 con 0x7f99300cffa0 2021-04-09T18:29:51.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f9935b9c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f99300d6c80 0x7f993010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:51.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f993539b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300cffa0 0x7f99300d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:51.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f9934b9a700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f99300d7800 0x7f99300d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:34400/0 (socket says 172.21.15.32:34400) 2021-04-09T18:29:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f993539b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300cffa0 0x7f99300d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:34106/0 (socket says 172.21.15.32:34106) 2021-04-09T18:29:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f9934b9a700 1 -- 172.21.15.32:0/2856617161 learned_addr learned my addr 172.21.15.32:0/2856617161 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f9935b9c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f99300d6c80 0x7f993010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:46872/0 (socket says 172.21.15.32:46872) 2021-04-09T18:29:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f993539b700 1 -- 172.21.15.32:0/2856617161 >> v2:172.21.15.86:3300/0 conn(0x7f99300d6c80 msgr2=0x7f993010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f993539b700 1 --2- 172.21.15.32:0/2856617161 >> v2:172.21.15.86:3300/0 conn(0x7f99300d6c80 0x7f993010cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f993539b700 1 -- 172.21.15.32:0/2856617161 >> v1:172.21.15.86:6789/0 conn(0x7f99300d7800 legacy=0x7f99300d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f993539b700 1 -- 172.21.15.32:0/2856617161 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99300d91e0 con 0x7f99300cffa0 2021-04-09T18:29:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f9935b9c700 1 --2- 172.21.15.32:0/2856617161 >> v2:172.21.15.86:3300/0 conn(0x7f99300d6c80 0x7f993010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f993539b700 1 --2- 172.21.15.32:0/2856617161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300cffa0 0x7f99300d2440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f992c006650 tx=0x7f992c009a80).ready entity=mon.2 client_cookie=94df7fc8e7012f5a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f9923fff700 1 -- 172.21.15.32:0/2856617161 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f992c0009b0 con 0x7f99300cffa0 2021-04-09T18:29:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f9923fff700 1 -- 172.21.15.32:0/2856617161 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992c00c0c0 con 0x7f99300cffa0 2021-04-09T18:29:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f9923fff700 1 -- 172.21.15.32:0/2856617161 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f992c00c510 con 0x7f99300cffa0 2021-04-09T18:29:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f993757f700 1 -- 172.21.15.32:0/2856617161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300cffa0 msgr2=0x7f99300d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:51.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.520 7f993757f700 1 --2- 172.21.15.32:0/2856617161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300cffa0 0x7f99300d2440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f992c006650 tx=0x7f992c009a80).stop 2021-04-09T18:29:51.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.521 7f993757f700 1 -- 172.21.15.32:0/2856617161 shutdown_connections 2021-04-09T18:29:51.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.521 7f993757f700 1 --2- 172.21.15.32:0/2856617161 >> v2:172.21.15.86:3300/0 conn(0x7f99300d6c80 0x7f993010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.521 7f993757f700 1 --2- 172.21.15.32:0/2856617161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300cffa0 0x7f99300d2440 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.521 7f993757f700 1 -- 172.21.15.32:0/2856617161 >> 172.21.15.32:0/2856617161 conn(0x7f99300ce5c0 msgr2=0x7f99300cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.521 7f993757f700 1 -- 172.21.15.32:0/2856617161 shutdown_connections 2021-04-09T18:29:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.521 7f993757f700 1 -- 172.21.15.32:0/2856617161 wait complete. 2021-04-09T18:29:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.541 7f993757f700 1 Processor -- start 2021-04-09T18:29:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.541 7f993757f700 1 -- start start 2021-04-09T18:29:51.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f993757f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300b4250 0x7f993009a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f993757f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993018e280 0x7f9930192370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f993757f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9930184d80 0x7f9930191250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f993757f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9930192a10 con 0x7f9930184d80 2021-04-09T18:29:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f993757f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9930191df0 con 0x7f993018e280 2021-04-09T18:29:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f993757f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9930191870 con 0x7f99300b4250 2021-04-09T18:29:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f9935b9c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9930184d80 0x7f9930191250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f9935b9c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9930184d80 0x7f9930191250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55120/0 (socket says 172.21.15.32:55120) 2021-04-09T18:29:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f9935b9c700 1 -- 172.21.15.32:0/554055599 learned_addr learned my addr 172.21.15.32:0/554055599 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f993539b700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300b4250 0x7f993009a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.542 7f9934b9a700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993018e280 0x7f9930192370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9935b9c700 1 -- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300b4250 msgr2=0x7f993009a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9935b9c700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300b4250 0x7f993009a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9935b9c700 1 -- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993018e280 msgr2=0x7f9930192370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:51.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9935b9c700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993018e280 0x7f9930192370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9935b9c700 1 -- 172.21.15.32:0/554055599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9930191a90 con 0x7f9930184d80 2021-04-09T18:29:51.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f993539b700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300b4250 0x7f993009a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:51.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9934b9a700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993018e280 0x7f9930192370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:29:51.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9935b9c700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9930184d80 0x7f9930191250 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f992800b970 tx=0x7f992800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:51.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9921ffb700 1 -- 172.21.15.32:0/554055599 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9928000f90 con 0x7f9930184d80 2021-04-09T18:29:51.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9921ffb700 1 -- 172.21.15.32:0/554055599 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9928006aa0 con 0x7f9930184d80 2021-04-09T18:29:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f9921ffb700 1 -- 172.21.15.32:0/554055599 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99280015f0 con 0x7f9930184d80 2021-04-09T18:29:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f993757f700 1 -- 172.21.15.32:0/554055599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9930191a90 con 0x7f9930184d80 2021-04-09T18:29:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.543 7f993757f700 1 -- 172.21.15.32:0/554055599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9930191a90 con 0x7f9930184d80 2021-04-09T18:29:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.544 7f9921ffb700 1 -- 172.21.15.32:0/554055599 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9928000e70 con 0x7f9930184d80 2021-04-09T18:29:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.544 7f9921ffb700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9910018930 0x7f991001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.544 7f9921ffb700 1 -- 172.21.15.32:0/554055599 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9928032d70 con 0x7f9930184d80 2021-04-09T18:29:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.544 7f993539b700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9910018930 0x7f991001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.544 7f993539b700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9910018930 0x7f991001adc0 crc :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.563 7f993757f700 1 -- 172.21.15.32:0/554055599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9930182d60 con 0x7f9930184d80 2021-04-09T18:29:51.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.567 7f9921ffb700 1 -- 172.21.15.32:0/554055599 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f99280066f0 con 0x7f9930184d80 2021-04-09T18:29:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.751 7f993757f700 1 -- 172.21.15.32:0/554055599 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9930182d60 con 0x7f9930184d80 2021-04-09T18:29:51.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.752 7f9921ffb700 1 -- 172.21.15.32:0/554055599 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+156 (secure 0 0 0) 0x7f9928006aa0 con 0x7f9930184d80 2021-04-09T18:29:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:51.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 -- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9910018930 msgr2=0x7f991001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:51.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9910018930 0x7f991001adc0 crc :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 -- 172.21.15.32:0/554055599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9930184d80 msgr2=0x7f9930191250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9930184d80 0x7f9930191250 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f992800b970 tx=0x7f992800bd10).stop 2021-04-09T18:29:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 -- 172.21.15.32:0/554055599 shutdown_connections 2021-04-09T18:29:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9910018930 0x7f991001adc0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9930184d80 0x7f9930191250 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993018e280 0x7f9930192370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 --2- 172.21.15.32:0/554055599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99300b4250 0x7f993009a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 -- 172.21.15.32:0/554055599 >> 172.21.15.32:0/554055599 conn(0x7f9930090a90 msgr2=0x7f99301832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 -- 172.21.15.32:0/554055599 shutdown_connections 2021-04-09T18:29:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.754 7f993757f700 1 -- 172.21.15.32:0/554055599 wait complete. 2021-04-09T18:29:51.851 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{"MDS_INSUFFICIENT_STANDBY":{"severity":"HEALTH_WARN","summary":{"message":"insufficient standby MDS daemons available"}}},"status":"HEALTH_WARN"} 2021-04-09T18:29:51.851 DEBUG:tasks.ceph_test_case:wait_until_true: success in 15s 2021-04-09T18:29:51.852 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:29:51.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:29:51.855 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:29:51.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:52.003 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:29:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.918 7f770d6a6700 1 Processor -- start 2021-04-09T18:29:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.919 7f770d6a6700 1 -- start start 2021-04-09T18:29:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.919 7f770d6a6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f770810d010 0x7f77080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f770d6a6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080d1da0 0x7f77080d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f770d6a6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770810d480 con 0x7f77080d82c0 2021-04-09T18:29:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f770d6a6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77080d9220 con 0x7f770810d010 2021-04-09T18:29:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f770d6a6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7708112100 con 0x7f77080d1da0 2021-04-09T18:29:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f7707ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f77080d82c0 0x7f770810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36500/0 (socket says 172.21.15.32:36500) 2021-04-09T18:29:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f7707ac0700 1 -- 172.21.15.32:0/4078111672 learned_addr learned my addr 172.21.15.32:0/4078111672 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f77072bf700 1 --2- 172.21.15.32:0/4078111672 >> v2:172.21.15.86:3300/0 conn(0x7f770810d010 0x7f77080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f76feabe700 1 --2- 172.21.15.32:0/4078111672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080d1da0 0x7f77080d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f7706abe700 1 -- 172.21.15.32:0/4078111672 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167994743 0 0) 0x7f7700001480 con 0x7f77080d82c0 2021-04-09T18:29:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f7706abe700 1 -- 172.21.15.32:0/4078111672 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f4002fe0 con 0x7f77080d82c0 2021-04-09T18:29:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f77072bf700 1 -- 172.21.15.32:0/4078111672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080d1da0 msgr2=0x7f77080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f77072bf700 1 --2- 172.21.15.32:0/4078111672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080d1da0 0x7f77080d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f77072bf700 1 -- 172.21.15.32:0/4078111672 >> v1:172.21.15.32:6789/0 conn(0x7f77080d82c0 legacy=0x7f770810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f77072bf700 1 -- 172.21.15.32:0/4078111672 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080d09c0 con 0x7f770810d010 2021-04-09T18:29:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f76feabe700 1 --2- 172.21.15.32:0/4078111672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080d1da0 0x7f77080d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:29:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f77072bf700 1 --2- 172.21.15.32:0/4078111672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770810d010 0x7f77080d1780 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7f76f8006650 tx=0x7f76f8009a80).ready entity=mon.1 client_cookie=840dcd6f3c1f8215 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f7706abe700 1 -- 172.21.15.32:0/4078111672 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76f80009b0 con 0x7f770810d010 2021-04-09T18:29:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f7706abe700 1 -- 172.21.15.32:0/4078111672 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f800c0c0 con 0x7f770810d010 2021-04-09T18:29:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.920 7f7706abe700 1 -- 172.21.15.32:0/4078111672 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76f800c510 con 0x7f770810d010 2021-04-09T18:29:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.921 7f770d6a6700 1 -- 172.21.15.32:0/4078111672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770810d010 msgr2=0x7f77080d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.921 7f770d6a6700 1 --2- 172.21.15.32:0/4078111672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770810d010 0x7f77080d1780 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7f76f8006650 tx=0x7f76f8009a80).stop 2021-04-09T18:29:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.921 7f770d6a6700 1 -- 172.21.15.32:0/4078111672 shutdown_connections 2021-04-09T18:29:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.921 7f770d6a6700 1 --2- 172.21.15.32:0/4078111672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080d1da0 0x7f77080d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.921 7f770d6a6700 1 --2- 172.21.15.32:0/4078111672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770810d010 0x7f77080d1780 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.921 7f770d6a6700 1 -- 172.21.15.32:0/4078111672 >> 172.21.15.32:0/4078111672 conn(0x7f77080ce5f0 msgr2=0x7f77080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:52.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.921 7f770d6a6700 1 -- 172.21.15.32:0/4078111672 shutdown_connections 2021-04-09T18:29:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.921 7f770d6a6700 1 -- 172.21.15.32:0/4078111672 wait complete. 2021-04-09T18:29:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.941 7f770d6a6700 1 Processor -- start 2021-04-09T18:29:52.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f770d6a6700 1 -- start start 2021-04-09T18:29:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f770d6a6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7708179bf0 0x7f7708179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:52.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f770d6a6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770817c540 0x7f770817c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:52.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f770d6a6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080aca60 0x7f77080ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:52.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f770d6a6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770817cf60 con 0x7f7708179bf0 2021-04-09T18:29:52.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f770d6a6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080ad480 con 0x7f770817c540 2021-04-09T18:29:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f770d6a6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77080ad6a0 con 0x7f77080aca60 2021-04-09T18:29:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f77072bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7708179bf0 0x7f7708179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f77072bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7708179bf0 0x7f7708179ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55144/0 (socket says 172.21.15.32:55144) 2021-04-09T18:29:52.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f77072bf700 1 -- 172.21.15.32:0/3965252055 learned_addr learned my addr 172.21.15.32:0/3965252055 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.942 7f7707ac0700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080aca60 0x7f77080ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f76feabe700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770817c540 0x7f770817c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f77072bf700 1 -- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080aca60 msgr2=0x7f77080ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:52.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f77072bf700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080aca60 0x7f77080ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f77072bf700 1 -- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770817c540 msgr2=0x7f770817c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:52.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f77072bf700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770817c540 0x7f770817c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f77072bf700 1 -- 172.21.15.32:0/3965252055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080ad8c0 con 0x7f7708179bf0 2021-04-09T18:29:52.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f7707ac0700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080aca60 0x7f77080ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:52.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f76feabe700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770817c540 0x7f770817c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:52.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f77072bf700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7708179bf0 0x7f7708179ff0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f76f8009650 tx=0x7f76f800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:52.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f7704aba700 1 -- 172.21.15.32:0/3965252055 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76f800bd50 con 0x7f7708179bf0 2021-04-09T18:29:52.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f7704aba700 1 -- 172.21.15.32:0/3965252055 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f8006030 con 0x7f7708179bf0 2021-04-09T18:29:52.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f7704aba700 1 -- 172.21.15.32:0/3965252055 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76f8005560 con 0x7f7708179bf0 2021-04-09T18:29:52.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77080ad8c0 con 0x7f7708179bf0 2021-04-09T18:29:52.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.943 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77080ad8c0 con 0x7f7708179bf0 2021-04-09T18:29:52.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.944 7f7704aba700 1 -- 172.21.15.32:0/3965252055 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f76f8009300 con 0x7f7708179bf0 2021-04-09T18:29:52.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.944 7f7704aba700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76ec018930 0x7f76ec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.944 7f7704aba700 1 -- 172.21.15.32:0/3965252055 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f76f80329d0 con 0x7f7708179bf0 2021-04-09T18:29:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.944 7f76feabe700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76ec018930 0x7f76ec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:52.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.944 7f76feabe700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76ec018930 0x7f76ec01ae10 crc :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:52.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.963 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770817cf60 con 0x7f7708179bf0 2021-04-09T18:29:52.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:51.967 7f7704aba700 1 -- 172.21.15.32:0/3965252055 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f76f8000940 con 0x7f7708179bf0 2021-04-09T18:29:52.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.157 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f770817cf60 con 0x7f7708179bf0 2021-04-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.158 7f7704aba700 1 -- 172.21.15.32:0/3965252055 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2449 (secure 0 0 0) 0x7f76f800bd50 con 0x7f7708179bf0 2021-04-09T18:29:52.245 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 313 2021-04-09T18:29:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:52.246 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":313,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:29:34.688038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10177":{"gid":10177,"name":"a","rank":0,"incarnation":308,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2200487686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2200487686},{"type":"v1","addr":"172.21.15.32:6835","nonce":2200487686}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":3},"id":14}]} 2021-04-09T18:29:52.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76ec018930 msgr2=0x7f76ec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:52.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76ec018930 0x7f76ec01ae10 crc :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7708179bf0 msgr2=0x7f7708179ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:52.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7708179bf0 0x7f7708179ff0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f76f8009650 tx=0x7f76f800b960).stop 2021-04-09T18:29:52.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 shutdown_connections 2021-04-09T18:29:52.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76ec018930 0x7f76ec01ae10 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77080aca60 0x7f77080ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f770817c540 0x7f770817c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 --2- 172.21.15.32:0/3965252055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7708179bf0 0x7f7708179ff0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:52.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.160 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 >> 172.21.15.32:0/3965252055 conn(0x7f77081acae0 msgr2=0x7f770817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:52.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.161 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 shutdown_connections 2021-04-09T18:29:52.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:52.161 7f770d6a6700 1 -- 172.21.15.32:0/3965252055 wait complete. 2021-04-09T18:29:52.257 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:29:57.259 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.328 7f53cf951700 1 Processor -- start 2021-04-09T18:29:57.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.328 7f53cf951700 1 -- start start 2021-04-09T18:29:57.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53cf951700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f53c80d6c80 0x7f53c80d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53cf951700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c80d7280 con 0x7f53c80cffa0 2021-04-09T18:29:57.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53cf951700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c80d2840 con 0x7f53c80d2440 2021-04-09T18:29:57.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53cf951700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c80d7080 con 0x7f53c80d6c80 2021-04-09T18:29:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53cd76d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f53c80cffa0 0x7f53c80d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36512/0 (socket says 172.21.15.32:36512) 2021-04-09T18:29:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53cd76d700 1 -- 172.21.15.32:0/2828606640 learned_addr learned my addr 172.21.15.32:0/2828606640 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53ccf6c700 1 --2- 172.21.15.32:0/2828606640 >> v2:172.21.15.86:3300/0 conn(0x7f53c80d6c80 0x7f53c80d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53bffff700 1 -- 172.21.15.32:0/2828606640 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423293467 0 0) 0x7f53c4000ce0 con 0x7f53c80cffa0 2021-04-09T18:29:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53bffff700 1 -- 172.21.15.32:0/2828606640 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b4002fe0 con 0x7f53c80cffa0 2021-04-09T18:29:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53bffff700 1 -- 172.21.15.32:0/2828606640 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 87601130 0 0) 0x7f53c4000ed0 con 0x7f53c80cffa0 2021-04-09T18:29:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53ccf6c700 1 -- 172.21.15.32:0/2828606640 >> v1:172.21.15.86:6789/0 conn(0x7f53c80d2440 legacy=0x7f53c810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53ccf6c700 1 -- 172.21.15.32:0/2828606640 >> v1:172.21.15.32:6789/0 conn(0x7f53c80cffa0 legacy=0x7f53c80d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.329 7f53ccf6c700 1 -- 172.21.15.32:0/2828606640 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c80d91e0 con 0x7f53c80d6c80 2021-04-09T18:29:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.330 7f53ccf6c700 1 --2- 172.21.15.32:0/2828606640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c80d6c80 0x7f53c80d7ff0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f53b8006650 tx=0x7f53b8009a00).ready entity=mon.1 client_cookie=b1865a1390f0c76c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:57.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.330 7f53bffff700 1 -- 172.21.15.32:0/2828606640 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53b8009df0 con 0x7f53c80d6c80 2021-04-09T18:29:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.330 7f53bffff700 1 -- 172.21.15.32:0/2828606640 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b80009b0 con 0x7f53c80d6c80 2021-04-09T18:29:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.330 7f53bffff700 1 -- 172.21.15.32:0/2828606640 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53b800c2b0 con 0x7f53c80d6c80 2021-04-09T18:29:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.330 7f53cf951700 1 -- 172.21.15.32:0/2828606640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c80d6c80 msgr2=0x7f53c80d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.330 7f53cf951700 1 --2- 172.21.15.32:0/2828606640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c80d6c80 0x7f53c80d7ff0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f53b8006650 tx=0x7f53b8009a00).stop 2021-04-09T18:29:57.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.330 7f53cf951700 1 -- 172.21.15.32:0/2828606640 shutdown_connections 2021-04-09T18:29:57.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.330 7f53cf951700 1 --2- 172.21.15.32:0/2828606640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c80d6c80 0x7f53c80d7ff0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.330 7f53cf951700 1 -- 172.21.15.32:0/2828606640 >> 172.21.15.32:0/2828606640 conn(0x7f53c80ce5c0 msgr2=0x7f53c80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:57.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.331 7f53cf951700 1 -- 172.21.15.32:0/2828606640 shutdown_connections 2021-04-09T18:29:57.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.331 7f53cf951700 1 -- 172.21.15.32:0/2828606640 wait complete. 2021-04-09T18:29:57.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.351 7f53cf951700 1 Processor -- start 2021-04-09T18:29:57.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.351 7f53cf951700 1 -- start start 2021-04-09T18:29:57.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.351 7f53cf951700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c81929f0 0x7f53c8192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.351 7f53cf951700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53c808ddc0 0x7f53c808e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.351 7f53cf951700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53c808e7e0 0x7f53c808c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.351 7f53cf951700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c8193410 con 0x7f53c808ddc0 2021-04-09T18:29:57.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.351 7f53cf951700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c80a9140 con 0x7f53c81929f0 2021-04-09T18:29:57.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.351 7f53cf951700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c808ce80 con 0x7f53c808e7e0 2021-04-09T18:29:57.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53ccf6c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53c808ddc0 0x7f53c808e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53ccf6c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53c808ddc0 0x7f53c808e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55162/0 (socket says 172.21.15.32:55162) 2021-04-09T18:29:57.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53ccf6c700 1 -- 172.21.15.32:0/576710638 learned_addr learned my addr 172.21.15.32:0/576710638 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:57.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53cd76d700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c81929f0 0x7f53c8192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53cdf6e700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53c808e7e0 0x7f53c808c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53ccf6c700 1 -- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53c808e7e0 msgr2=0x7f53c808c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53ccf6c700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53c808e7e0 0x7f53c808c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53ccf6c700 1 -- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c81929f0 msgr2=0x7f53c8192df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53ccf6c700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c81929f0 0x7f53c8192df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53ccf6c700 1 -- 172.21.15.32:0/576710638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c808d0a0 con 0x7f53c808ddc0 2021-04-09T18:29:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53ccf6c700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53c808ddc0 0x7f53c808e1c0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7f53b800b580 tx=0x7f53b800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53bdffb700 1 -- 172.21.15.32:0/576710638 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53b8000c90 con 0x7f53c808ddc0 2021-04-09T18:29:57.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53bdffb700 1 -- 172.21.15.32:0/576710638 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b8002030 con 0x7f53c808ddc0 2021-04-09T18:29:57.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53bdffb700 1 -- 172.21.15.32:0/576710638 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53b8002480 con 0x7f53c808ddc0 2021-04-09T18:29:57.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53cf951700 1 -- 172.21.15.32:0/576710638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c808d0a0 con 0x7f53c808ddc0 2021-04-09T18:29:57.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.352 7f53cf951700 1 -- 172.21.15.32:0/576710638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c808d0a0 con 0x7f53c808ddc0 2021-04-09T18:29:57.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.353 7f53bdffb700 1 -- 172.21.15.32:0/576710638 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f53b8002210 con 0x7f53c808ddc0 2021-04-09T18:29:57.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.353 7f53bdffb700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53a8018930 0x7f53a801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.353 7f53bdffb700 1 -- 172.21.15.32:0/576710638 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f53b8032a60 con 0x7f53c808ddc0 2021-04-09T18:29:57.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.353 7f53cd76d700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53a8018930 0x7f53a801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.354 7f53cd76d700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53a8018930 0x7f53a801adc0 crc :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:57.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.373 7f53cf951700 1 -- 172.21.15.32:0/576710638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c81b70c0 con 0x7f53c808ddc0 2021-04-09T18:29:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.376 7f53bdffb700 1 -- 172.21.15.32:0/576710638 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f53b8002240 con 0x7f53c808ddc0 2021-04-09T18:29:57.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.569 7f53cf951700 1 -- 172.21.15.32:0/576710638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53c80d7280 con 0x7f53c808ddc0 2021-04-09T18:29:57.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.570 7f53bdffb700 1 -- 172.21.15.32:0/576710638 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 314 v314) v1 ==== 95+0+2788 (secure 0 0 0) 0x7f53b800bd10 con 0x7f53c808ddc0 2021-04-09T18:29:57.658 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 314 2021-04-09T18:29:57.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:57.658 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":314,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3983419158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3983419158},{"type":"v1","addr":"172.21.15.32:6834","nonce":3983419158}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":314},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:29:34.688038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10177":{"gid":10177,"name":"a","rank":0,"incarnation":308,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2200487686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2200487686},{"type":"v1","addr":"172.21.15.32:6835","nonce":2200487686}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":3},"id":14}]} 2021-04-09T18:29:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 -- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53a8018930 msgr2=0x7f53a801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53a8018930 0x7f53a801adc0 crc :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 -- 172.21.15.32:0/576710638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53c808ddc0 msgr2=0x7f53c808e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53c808ddc0 0x7f53c808e1c0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7f53b800b580 tx=0x7f53b800b920).stop 2021-04-09T18:29:57.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 -- 172.21.15.32:0/576710638 shutdown_connections 2021-04-09T18:29:57.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53a8018930 0x7f53a801adc0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53c808ddc0 0x7f53c808e1c0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53c808e7e0 0x7f53c808c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 --2- 172.21.15.32:0/576710638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c81929f0 0x7f53c8192df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 -- 172.21.15.32:0/576710638 >> 172.21.15.32:0/576710638 conn(0x7f53c8189b00 msgr2=0x7f53c81921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:57.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 -- 172.21.15.32:0/576710638 shutdown_connections 2021-04-09T18:29:57.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.572 7f53cf951700 1 -- 172.21.15.32:0/576710638 wait complete. 2021-04-09T18:29:57.670 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2021-04-09T18:29:57.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:29:57.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.732 7fce94627700 1 Processor -- start 2021-04-09T18:29:57.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.732 7fce94627700 1 -- start start 2021-04-09T18:29:57.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.732 7fce94627700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fce8c0d6c80 0x7fce8c0d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce94627700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0d7280 con 0x7fce8c0d2440 2021-04-09T18:29:57.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce94627700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0d2840 con 0x7fce8c0cffa0 2021-04-09T18:29:57.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce94627700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8c0d7080 con 0x7fce8c0d6c80 2021-04-09T18:29:57.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce92c44700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fce8c0d2440 0x7fce8c10cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36528/0 (socket says 172.21.15.32:36528) 2021-04-09T18:29:57.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce92c44700 1 -- 172.21.15.32:0/3079446723 learned_addr learned my addr 172.21.15.32:0/3079446723 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:57.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce91c42700 1 --2- 172.21.15.32:0/3079446723 >> v2:172.21.15.86:3300/0 conn(0x7fce8c0d6c80 0x7fce8c0d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce91441700 1 -- 172.21.15.32:0/3079446723 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178951211 0 0) 0x7fce84000d30 con 0x7fce8c0d2440 2021-04-09T18:29:57.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce91441700 1 -- 172.21.15.32:0/3079446723 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce78002fe0 con 0x7fce8c0d2440 2021-04-09T18:29:57.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce91441700 1 -- 172.21.15.32:0/3079446723 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249713757 0 0) 0x7fce88001080 con 0x7fce8c0cffa0 2021-04-09T18:29:57.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce91c42700 1 -- 172.21.15.32:0/3079446723 >> v1:172.21.15.86:6789/0 conn(0x7fce8c0cffa0 legacy=0x7fce8c0d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce91c42700 1 -- 172.21.15.32:0/3079446723 >> v1:172.21.15.32:6789/0 conn(0x7fce8c0d2440 legacy=0x7fce8c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce91c42700 1 -- 172.21.15.32:0/3079446723 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c0d91e0 con 0x7fce8c0d6c80 2021-04-09T18:29:57.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.733 7fce91c42700 1 --2- 172.21.15.32:0/3079446723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0d6c80 0x7fce8c0d7ff0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7fce7c006650 tx=0x7fce7c009a00).ready entity=mon.1 client_cookie=a5e9bdb091a4dde5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:57.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.734 7fce91441700 1 -- 172.21.15.32:0/3079446723 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c009df0 con 0x7fce8c0d6c80 2021-04-09T18:29:57.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.734 7fce91441700 1 -- 172.21.15.32:0/3079446723 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c0009b0 con 0x7fce8c0d6c80 2021-04-09T18:29:57.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.734 7fce91441700 1 -- 172.21.15.32:0/3079446723 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c00c2b0 con 0x7fce8c0d6c80 2021-04-09T18:29:57.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.734 7fce94627700 1 -- 172.21.15.32:0/3079446723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0d6c80 msgr2=0x7fce8c0d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.734 7fce94627700 1 --2- 172.21.15.32:0/3079446723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0d6c80 0x7fce8c0d7ff0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7fce7c006650 tx=0x7fce7c009a00).stop 2021-04-09T18:29:57.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.734 7fce94627700 1 -- 172.21.15.32:0/3079446723 shutdown_connections 2021-04-09T18:29:57.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.734 7fce94627700 1 --2- 172.21.15.32:0/3079446723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0d6c80 0x7fce8c0d7ff0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.735 7fce94627700 1 -- 172.21.15.32:0/3079446723 >> 172.21.15.32:0/3079446723 conn(0x7fce8c0ce5c0 msgr2=0x7fce8c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:57.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.735 7fce94627700 1 -- 172.21.15.32:0/3079446723 shutdown_connections 2021-04-09T18:29:57.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.735 7fce94627700 1 -- 172.21.15.32:0/3079446723 wait complete. 2021-04-09T18:29:57.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce94627700 1 Processor -- start 2021-04-09T18:29:57.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce94627700 1 -- start start 2021-04-09T18:29:57.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce94627700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce8c1b8a70 0x7fce8c1b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce94627700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8c18b140 0x7fce8c18b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce94627700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c193eb0 0x7fce8c1942b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce94627700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c1b8e70 con 0x7fce8c18b140 2021-04-09T18:29:57.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce94627700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c18abc0 con 0x7fce8c193eb0 2021-04-09T18:29:57.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce94627700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c18a8e0 con 0x7fce8c1b8a70 2021-04-09T18:29:57.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce91c42700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8c18b140 0x7fce8c18b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce91c42700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8c18b140 0x7fce8c18b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55174/0 (socket says 172.21.15.32:55174) 2021-04-09T18:29:57.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.756 7fce91c42700 1 -- 172.21.15.32:0/2631030577 learned_addr learned my addr 172.21.15.32:0/2631030577 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:57.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce92c44700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c193eb0 0x7fce8c1942b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce92443700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce8c1b8a70 0x7fce8c1b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce91c42700 1 -- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce8c1b8a70 msgr2=0x7fce8c1b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce91c42700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce8c1b8a70 0x7fce8c1b8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce91c42700 1 -- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c193eb0 msgr2=0x7fce8c1942b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:57.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce91c42700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c193eb0 0x7fce8c1942b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:57.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce91c42700 1 -- 172.21.15.32:0/2631030577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c18bb60 con 0x7fce8c18b140 2021-04-09T18:29:57.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce91c42700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8c18b140 0x7fce8c18b540 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 rx=0x7fce7c00b580 tx=0x7fce7c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:57.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce82ffd700 1 -- 172.21.15.32:0/2631030577 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c006440 con 0x7fce8c18b140 2021-04-09T18:29:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce82ffd700 1 -- 172.21.15.32:0/2631030577 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c000cd0 con 0x7fce8c18b140 2021-04-09T18:29:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce82ffd700 1 -- 172.21.15.32:0/2631030577 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c0022b0 con 0x7fce8c18b140 2021-04-09T18:29:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce94627700 1 -- 172.21.15.32:0/2631030577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8c18bb60 con 0x7fce8c18b140 2021-04-09T18:29:57.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.757 7fce94627700 1 -- 172.21.15.32:0/2631030577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8c18bb60 con 0x7fce8c18b140 2021-04-09T18:29:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.758 7fce82ffd700 1 -- 172.21.15.32:0/2631030577 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fce7c00a030 con 0x7fce8c18b140 2021-04-09T18:29:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.758 7fce82ffd700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce74018930 0x7fce7401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.758 7fce82ffd700 1 -- 172.21.15.32:0/2631030577 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fce7c032a00 con 0x7fce8c18b140 2021-04-09T18:29:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.758 7fce92443700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce74018930 0x7fce7401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:57.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.759 7fce92443700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce74018930 0x7fce7401adc0 crc :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:57.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.778 7fce94627700 1 -- 172.21.15.32:0/2631030577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8c1b7090 con 0x7fce8c18b140 2021-04-09T18:29:57.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.781 7fce82ffd700 1 -- 172.21.15.32:0/2631030577 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fce7c000940 con 0x7fce8c18b140 2021-04-09T18:29:58.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.965 7fce94627700 1 -- 172.21.15.32:0/2631030577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce8c0d7280 con 0x7fce8c18b140 2021-04-09T18:29:58.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.966 7fce82ffd700 1 -- 172.21.15.32:0/2631030577 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7fce7c00a030 con 0x7fce8c18b140 2021-04-09T18:29:58.053 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:58.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 -- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce74018930 msgr2=0x7fce7401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce74018930 0x7fce7401adc0 crc :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 -- 172.21.15.32:0/2631030577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8c18b140 msgr2=0x7fce8c18b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8c18b140 0x7fce8c18b540 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 rx=0x7fce7c00b580 tx=0x7fce7c00b920).stop 2021-04-09T18:29:58.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 -- 172.21.15.32:0/2631030577 shutdown_connections 2021-04-09T18:29:58.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fce74018930 0x7fce7401adc0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce8c18b140 0x7fce8c18b540 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c193eb0 0x7fce8c1942b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 --2- 172.21.15.32:0/2631030577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce8c1b8a70 0x7fce8c1b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 -- 172.21.15.32:0/2631030577 >> 172.21.15.32:0/2631030577 conn(0x7fce8c08eef0 msgr2=0x7fce8c1b7760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:58.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 -- 172.21.15.32:0/2631030577 shutdown_connections 2021-04-09T18:29:58.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:57.969 7fce94627700 1 -- 172.21.15.32:0/2631030577 wait complete. 2021-04-09T18:29:58.064 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:29:58.065 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2021-04-09T18:29:58.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:29:58.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.128 7f48c636b700 1 Processor -- start 2021-04-09T18:29:58.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.128 7f48c636b700 1 -- start start 2021-04-09T18:29:58.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48c636b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48c010d010 0x7f48c00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48c636b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f48c00d1da0 0x7f48c00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48c636b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c00d82c0 0x7f48c010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48c636b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c00d8d30 con 0x7f48c010d010 2021-04-09T18:29:58.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48c636b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c00d9220 con 0x7f48c00d1da0 2021-04-09T18:29:58.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48c636b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c010d4b0 con 0x7f48c00d82c0 2021-04-09T18:29:58.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48c010d010 0x7f48c00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f48c010d010 0x7f48c00d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55182/0 (socket says 172.21.15.32:55182) 2021-04-09T18:29:58.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bffff700 1 -- 172.21.15.32:0/137036135 learned_addr learned my addr 172.21.15.32:0/137036135 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:58.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48c4988700 1 --2- 172.21.15.32:0/137036135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c00d82c0 0x7f48c010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48b7fff700 1 --2- 172.21.15.32:0/137036135 >> v2:172.21.15.86:3300/0 conn(0x7f48c00d1da0 0x7f48c00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bffff700 1 -- 172.21.15.32:0/137036135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c00d82c0 msgr2=0x7f48c010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:58.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bffff700 1 --2- 172.21.15.32:0/137036135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c00d82c0 0x7f48c010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bffff700 1 -- 172.21.15.32:0/137036135 >> v2:172.21.15.86:3300/0 conn(0x7f48c00d1da0 msgr2=0x7f48c00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:58.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bffff700 1 --2- 172.21.15.32:0/137036135 >> v2:172.21.15.86:3300/0 conn(0x7f48c00d1da0 0x7f48c00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bffff700 1 -- 172.21.15.32:0/137036135 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c00d86c0 con 0x7f48c010d010 2021-04-09T18:29:58.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bffff700 1 --2- 172.21.15.32:0/137036135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c010d010 0x7f48c00d1780 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7f48b0006700 tx=0x7f48b0009b00).ready entity=mon.0 client_cookie=a1c63e6ab5413d6a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:58.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bf7fe700 1 -- 172.21.15.32:0/137036135 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b000c0c0 con 0x7f48c010d010 2021-04-09T18:29:58.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.129 7f48bf7fe700 1 -- 172.21.15.32:0/137036135 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b000c400 con 0x7f48c010d010 2021-04-09T18:29:58.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48bf7fe700 1 -- 172.21.15.32:0/137036135 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b000c8e0 con 0x7f48c010d010 2021-04-09T18:29:58.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48c636b700 1 -- 172.21.15.32:0/137036135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c010d010 msgr2=0x7f48c00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48c636b700 1 --2- 172.21.15.32:0/137036135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c010d010 0x7f48c00d1780 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7f48b0006700 tx=0x7f48b0009b00).stop 2021-04-09T18:29:58.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48c636b700 1 -- 172.21.15.32:0/137036135 shutdown_connections 2021-04-09T18:29:58.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48c636b700 1 --2- 172.21.15.32:0/137036135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c00d82c0 0x7f48c010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48c636b700 1 --2- 172.21.15.32:0/137036135 >> v2:172.21.15.86:3300/0 conn(0x7f48c00d1da0 0x7f48c00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48c636b700 1 --2- 172.21.15.32:0/137036135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c010d010 0x7f48c00d1780 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48c636b700 1 -- 172.21.15.32:0/137036135 >> 172.21.15.32:0/137036135 conn(0x7f48c00ce5f0 msgr2=0x7f48c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:58.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48c636b700 1 -- 172.21.15.32:0/137036135 shutdown_connections 2021-04-09T18:29:58.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.130 7f48c636b700 1 -- 172.21.15.32:0/137036135 wait complete. 2021-04-09T18:29:58.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48c636b700 1 Processor -- start 2021-04-09T18:29:58.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48c636b700 1 -- start start 2021-04-09T18:29:58.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48c636b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c01ac380 0x7f48c018a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48c636b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c01ac780 0x7f48c018a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48c636b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c0183520 0x7f48c0183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48c636b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c01ace10 con 0x7f48c01ac380 2021-04-09T18:29:58.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48c636b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c0189cf0 con 0x7f48c01ac780 2021-04-09T18:29:58.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48c636b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c0183f40 con 0x7f48c0183520 2021-04-09T18:29:58.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c01ac380 0x7f48c018a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48bffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c01ac380 0x7f48c018a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55186/0 (socket says 172.21.15.32:55186) 2021-04-09T18:29:58.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48bffff700 1 -- 172.21.15.32:0/2493250984 learned_addr learned my addr 172.21.15.32:0/2493250984 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:58.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48c4988700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c0183520 0x7f48c0183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.151 7f48b7fff700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c01ac780 0x7f48c018a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48bffff700 1 -- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c0183520 msgr2=0x7f48c0183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48bffff700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c0183520 0x7f48c0183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48bffff700 1 -- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c01ac780 msgr2=0x7f48c018a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48bffff700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c01ac780 0x7f48c018a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48bffff700 1 -- 172.21.15.32:0/2493250984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0184160 con 0x7f48c01ac380 2021-04-09T18:29:58.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48b7fff700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c01ac780 0x7f48c018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:58.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48bffff700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c01ac380 0x7f48c018a230 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7f48b0006f20 tx=0x7f48b000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:58.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48bd7fa700 1 -- 172.21.15.32:0/2493250984 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b000bdd0 con 0x7f48c01ac380 2021-04-09T18:29:58.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48bd7fa700 1 -- 172.21.15.32:0/2493250984 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b0006440 con 0x7f48c01ac380 2021-04-09T18:29:58.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48c636b700 1 -- 172.21.15.32:0/2493250984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c0184160 con 0x7f48c01ac380 2021-04-09T18:29:58.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48bd7fa700 1 -- 172.21.15.32:0/2493250984 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b00052e0 con 0x7f48c01ac380 2021-04-09T18:29:58.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.152 7f48c636b700 1 -- 172.21.15.32:0/2493250984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c0184160 con 0x7f48c01ac380 2021-04-09T18:29:58.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.153 7f48bd7fa700 1 -- 172.21.15.32:0/2493250984 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f48b00052e0 con 0x7f48c01ac380 2021-04-09T18:29:58.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.153 7f48bd7fa700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48a40188c0 0x7f48a401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.153 7f48bd7fa700 1 -- 172.21.15.32:0/2493250984 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f48b0032790 con 0x7f48c01ac380 2021-04-09T18:29:58.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.153 7f48b7fff700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48a40188c0 0x7f48a401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.153 7f48b7fff700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48a40188c0 0x7f48a401adc0 crc :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:58.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.173 7f48c636b700 1 -- 172.21.15.32:0/2493250984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c0182d10 con 0x7f48c01ac380 2021-04-09T18:29:58.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.176 7f48bd7fa700 1 -- 172.21.15.32:0/2493250984 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f48b0006030 con 0x7f48c01ac380 2021-04-09T18:29:58.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.369 7f48c636b700 1 -- 172.21.15.32:0/2493250984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48c0182d10 con 0x7f48c01ac380 2021-04-09T18:29:58.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.370 7f48bd7fa700 1 -- 172.21.15.32:0/2493250984 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 314 v314) v1 ==== 95+0+2788 (secure 0 0 0) 0x7f48b000bdd0 con 0x7f48c01ac380 2021-04-09T18:29:58.457 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 314 2021-04-09T18:29:58.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:58.457 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":314,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3983419158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3983419158},{"type":"v1","addr":"172.21.15.32:6834","nonce":3983419158}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":314},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:29:34.688038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10177":{"gid":10177,"name":"a","rank":0,"incarnation":308,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2200487686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2200487686},{"type":"v1","addr":"172.21.15.32:6835","nonce":2200487686}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":3},"id":14}]} 2021-04-09T18:29:58.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 -- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48a40188c0 msgr2=0x7f48a401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48a40188c0 0x7f48a401adc0 crc :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 -- 172.21.15.32:0/2493250984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c01ac380 msgr2=0x7f48c018a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c01ac380 0x7f48c018a230 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7f48b0006f20 tx=0x7f48b000b9e0).stop 2021-04-09T18:29:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 -- 172.21.15.32:0/2493250984 shutdown_connections 2021-04-09T18:29:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f48a40188c0 0x7f48a401adc0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48c0183520 0x7f48c0183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48c01ac780 0x7f48c018a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 --2- 172.21.15.32:0/2493250984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48c01ac380 0x7f48c018a230 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 -- 172.21.15.32:0/2493250984 >> 172.21.15.32:0/2493250984 conn(0x7f48c019b890 msgr2=0x7f48c01a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:58.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 -- 172.21.15.32:0/2493250984 shutdown_connections 2021-04-09T18:29:58.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.372 7f48c636b700 1 -- 172.21.15.32:0/2493250984 wait complete. 2021-04-09T18:29:58.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 4 2021-04-09T18:29:58.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.534 7fb16f675700 1 Processor -- start 2021-04-09T18:29:58.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.534 7fb16f675700 1 -- start start 2021-04-09T18:29:58.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.535 7fb16f675700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb16810d010 0x7fb1680d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.535 7fb16f675700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16810d480 con 0x7fb1680d1da0 2021-04-09T18:29:58.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.535 7fb16f675700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16810e1f0 con 0x7fb1680d8490 2021-04-09T18:29:58.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.535 7fb16f675700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16810e780 con 0x7fb16810d010 2021-04-09T18:29:58.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.535 7fb16cc90700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb1680d1da0 0x7fb1680d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36556/0 (socket says 172.21.15.32:36556) 2021-04-09T18:29:58.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.535 7fb16cc90700 1 -- 172.21.15.32:0/2955809430 learned_addr learned my addr 172.21.15.32:0/2955809430 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:58.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.535 7fb16d491700 1 --2- 172.21.15.32:0/2955809430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb16810d010 0x7fb1680d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.535 7fb15bfff700 1 -- 172.21.15.32:0/2955809430 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825608518 0 0) 0x7fb15c0009b0 con 0x7fb1680d1da0 2021-04-09T18:29:58.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.535 7fb15bfff700 1 -- 172.21.15.32:0/2955809430 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb154002fe0 con 0x7fb1680d1da0 2021-04-09T18:29:58.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb15bfff700 1 -- 172.21.15.32:0/2955809430 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107278057 0 0) 0x7fb160000d30 con 0x7fb1680d8490 2021-04-09T18:29:58.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb15bfff700 1 -- 172.21.15.32:0/2955809430 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1540042c0 con 0x7fb1680d8490 2021-04-09T18:29:58.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb16d491700 1 -- 172.21.15.32:0/2955809430 >> v1:172.21.15.86:6789/0 conn(0x7fb1680d8490 legacy=0x7fb16810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb16d491700 1 -- 172.21.15.32:0/2955809430 >> v1:172.21.15.32:6789/0 conn(0x7fb1680d1da0 legacy=0x7fb1680d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb16d491700 1 -- 172.21.15.32:0/2955809430 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1680d8d30 con 0x7fb16810d010 2021-04-09T18:29:58.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb16d491700 1 --2- 172.21.15.32:0/2955809430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb16810d010 0x7fb1680d1780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fb164006650 tx=0x7fb164009a80).ready entity=mon.2 client_cookie=8629213153e20052 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:58.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb15bfff700 1 -- 172.21.15.32:0/2955809430 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1640009b0 con 0x7fb16810d010 2021-04-09T18:29:58.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb15bfff700 1 -- 172.21.15.32:0/2955809430 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16400c0c0 con 0x7fb16810d010 2021-04-09T18:29:58.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb15bfff700 1 -- 172.21.15.32:0/2955809430 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1640009b0 con 0x7fb16810d010 2021-04-09T18:29:58.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb16f675700 1 -- 172.21.15.32:0/2955809430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb16810d010 msgr2=0x7fb1680d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.536 7fb16f675700 1 --2- 172.21.15.32:0/2955809430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb16810d010 0x7fb1680d1780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fb164006650 tx=0x7fb164009a80).stop 2021-04-09T18:29:58.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.537 7fb16f675700 1 -- 172.21.15.32:0/2955809430 shutdown_connections 2021-04-09T18:29:58.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.537 7fb16f675700 1 --2- 172.21.15.32:0/2955809430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb16810d010 0x7fb1680d1780 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.537 7fb16f675700 1 -- 172.21.15.32:0/2955809430 >> 172.21.15.32:0/2955809430 conn(0x7fb1680ce5f0 msgr2=0x7fb1680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:58.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.537 7fb16f675700 1 -- 172.21.15.32:0/2955809430 shutdown_connections 2021-04-09T18:29:58.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.537 7fb16f675700 1 -- 172.21.15.32:0/2955809430 wait complete. 2021-04-09T18:29:58.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.557 7fb16f675700 1 Processor -- start 2021-04-09T18:29:58.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.557 7fb16f675700 1 -- start start 2021-04-09T18:29:58.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16f675700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb168179610 0x7fb168179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16f675700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16817a820 0x7fb16817ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16f675700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1681949f0 0x7fb168194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16f675700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1681b8d50 con 0x7fb16817a820 2021-04-09T18:29:58.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16f675700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16817b240 con 0x7fb1681949f0 2021-04-09T18:29:58.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16f675700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb168195410 con 0x7fb168179610 2021-04-09T18:29:58.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16cc90700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16817a820 0x7fb16817ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16cc90700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16817a820 0x7fb16817ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55200/0 (socket says 172.21.15.32:55200) 2021-04-09T18:29:58.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16cc90700 1 -- 172.21.15.32:0/1363659952 learned_addr learned my addr 172.21.15.32:0/1363659952 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:58.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16dc92700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1681949f0 0x7fb168194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16d491700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb168179610 0x7fb168179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16cc90700 1 -- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb168179610 msgr2=0x7fb168179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16cc90700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb168179610 0x7fb168179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16cc90700 1 -- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1681949f0 msgr2=0x7fb168194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:58.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16cc90700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1681949f0 0x7fb168194df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:58.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16cc90700 1 -- 172.21.15.32:0/1363659952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb168195630 con 0x7fb16817a820 2021-04-09T18:29:58.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb16cc90700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16817a820 0x7fb16817ac20 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7fb15c00a4c0 tx=0x7fb15c00a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:58.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.558 7fb159ffb700 1 -- 172.21.15.32:0/1363659952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb15c00ac50 con 0x7fb16817a820 2021-04-09T18:29:58.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.559 7fb159ffb700 1 -- 172.21.15.32:0/1363659952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c004140 con 0x7fb16817a820 2021-04-09T18:29:58.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.559 7fb16f675700 1 -- 172.21.15.32:0/1363659952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb168195630 con 0x7fb16817a820 2021-04-09T18:29:58.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.559 7fb159ffb700 1 -- 172.21.15.32:0/1363659952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb15c004620 con 0x7fb16817a820 2021-04-09T18:29:58.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.559 7fb16f675700 1 -- 172.21.15.32:0/1363659952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb168195630 con 0x7fb16817a820 2021-04-09T18:29:58.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.559 7fb159ffb700 1 -- 172.21.15.32:0/1363659952 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb15c004570 con 0x7fb16817a820 2021-04-09T18:29:58.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.559 7fb159ffb700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb148018930 0x7fb14801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:58.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.559 7fb159ffb700 1 -- 172.21.15.32:0/1363659952 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb15c032a00 con 0x7fb16817a820 2021-04-09T18:29:58.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.559 7fb16d491700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb148018930 0x7fb14801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:58.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.560 7fb16d491700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb148018930 0x7fb14801ae10 crc :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:58.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.579 7fb16f675700 1 -- 172.21.15.32:0/1363659952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16817a2f0 con 0x7fb16817a820 2021-04-09T18:29:58.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.583 7fb159ffb700 1 -- 172.21.15.32:0/1363659952 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb15c004360 con 0x7fb16817a820 2021-04-09T18:29:58.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.778 7fb16f675700 1 -- 172.21.15.32:0/1363659952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "4"} v 0) v1 -- 0x7fb16817a2f0 con 0x7fb16817a820 2021-04-09T18:29:59.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.932 7fb159ffb700 1 -- 172.21.15.32:0/1363659952 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "4"}]=0 v315) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb15c004570 con 0x7fb16817a820 2021-04-09T18:29:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 -- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb148018930 msgr2=0x7fb14801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb148018930 0x7fb14801ae10 crc :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 -- 172.21.15.32:0/1363659952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16817a820 msgr2=0x7fb16817ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16817a820 0x7fb16817ac20 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7fb15c00a4c0 tx=0x7fb15c00a860).stop 2021-04-09T18:29:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 -- 172.21.15.32:0/1363659952 shutdown_connections 2021-04-09T18:29:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb148018930 0x7fb14801ae10 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb16817a820 0x7fb16817ac20 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1681949f0 0x7fb168194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 --2- 172.21.15.32:0/1363659952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb168179610 0x7fb168179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 -- 172.21.15.32:0/1363659952 >> 172.21.15.32:0/1363659952 conn(0x7fb168092a00 msgr2=0x7fb1681b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 -- 172.21.15.32:0/1363659952 shutdown_connections 2021-04-09T18:29:59.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:58.943 7fb16f675700 1 -- 172.21.15.32:0/1363659952 wait complete. 2021-04-09T18:29:59.040 INFO:tasks.cephfs.test_failover:waiting for insufficient standby daemon warning 2021-04-09T18:29:59.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:29:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.101 7fa82410b700 1 Processor -- start 2021-04-09T18:29:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.101 7fa82410b700 1 -- start start 2021-04-09T18:29:59.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa82410b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c0cffb0 0x7fa81c087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa82410b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa81c0d7c50 0x7fa81c1b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa82410b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa81c0d8050 0x7fa81c10d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa82410b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81c0d91f0 con 0x7fa81c0d7c50 2021-04-09T18:29:59.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa82410b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81c088460 con 0x7fa81c0d8050 2021-04-09T18:29:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa82410b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0d8620 con 0x7fa81c0cffb0 2021-04-09T18:29:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821726700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa81c0d7c50 0x7fa81c1b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821726700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa81c0d7c50 0x7fa81c1b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55210/0 (socket says 172.21.15.32:55210) 2021-04-09T18:29:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821726700 1 -- 172.21.15.32:0/2125500542 learned_addr learned my addr 172.21.15.32:0/2125500542 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:59.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa822728700 1 --2- 172.21.15.32:0/2125500542 >> v2:172.21.15.86:3300/0 conn(0x7fa81c0d8050 0x7fa81c10d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821f27700 1 --2- 172.21.15.32:0/2125500542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c0cffb0 0x7fa81c087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821726700 1 -- 172.21.15.32:0/2125500542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c0cffb0 msgr2=0x7fa81c087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821726700 1 --2- 172.21.15.32:0/2125500542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c0cffb0 0x7fa81c087e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821726700 1 -- 172.21.15.32:0/2125500542 >> v2:172.21.15.86:3300/0 conn(0x7fa81c0d8050 msgr2=0x7fa81c10d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:29:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821726700 1 --2- 172.21.15.32:0/2125500542 >> v2:172.21.15.86:3300/0 conn(0x7fa81c0d8050 0x7fa81c10d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821726700 1 -- 172.21.15.32:0/2125500542 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0d2800 con 0x7fa81c0d7c50 2021-04-09T18:29:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa822728700 1 --2- 172.21.15.32:0/2125500542 >> v2:172.21.15.86:3300/0 conn(0x7fa81c0d8050 0x7fa81c10d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.102 7fa821f27700 1 --2- 172.21.15.32:0/2125500542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c0cffb0 0x7fa81c087e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:29:59.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.103 7fa821726700 1 --2- 172.21.15.32:0/2125500542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0d7c50 0x7fa81c1b0490 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7fa810006700 tx=0x7fa810009b80).ready entity=mon.0 client_cookie=641c6138f5f5f469 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.103 7fa820f25700 1 -- 172.21.15.32:0/2125500542 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa81000c0c0 con 0x7fa81c0d7c50 2021-04-09T18:29:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.103 7fa820f25700 1 -- 172.21.15.32:0/2125500542 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81000c440 con 0x7fa81c0d7c50 2021-04-09T18:29:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.103 7fa820f25700 1 -- 172.21.15.32:0/2125500542 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa81000c920 con 0x7fa81c0d7c50 2021-04-09T18:29:59.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.103 7fa82410b700 1 -- 172.21.15.32:0/2125500542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0d7c50 msgr2=0x7fa81c1b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.103 7fa82410b700 1 --2- 172.21.15.32:0/2125500542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0d7c50 0x7fa81c1b0490 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7fa810006700 tx=0x7fa810009b80).stop 2021-04-09T18:29:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.104 7fa82410b700 1 -- 172.21.15.32:0/2125500542 shutdown_connections 2021-04-09T18:29:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.104 7fa82410b700 1 --2- 172.21.15.32:0/2125500542 >> v2:172.21.15.86:3300/0 conn(0x7fa81c0d8050 0x7fa81c10d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.104 7fa82410b700 1 --2- 172.21.15.32:0/2125500542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0d7c50 0x7fa81c1b0490 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.104 7fa82410b700 1 --2- 172.21.15.32:0/2125500542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c0cffb0 0x7fa81c087e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.104 7fa82410b700 1 -- 172.21.15.32:0/2125500542 >> 172.21.15.32:0/2125500542 conn(0x7fa81c0ce5b0 msgr2=0x7fa81c0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.104 7fa82410b700 1 -- 172.21.15.32:0/2125500542 shutdown_connections 2021-04-09T18:29:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.104 7fa82410b700 1 -- 172.21.15.32:0/2125500542 wait complete. 2021-04-09T18:29:59.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa82410b700 1 Processor -- start 2021-04-09T18:29:59.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa82410b700 1 -- start start 2021-04-09T18:29:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa82410b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c180120 0x7fa81c09d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa82410b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c09db10 0x7fa81c0b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa82410b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0b94d0 0x7fa81c1a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa82410b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c09d050 con 0x7fa81c0b94d0 2021-04-09T18:29:59.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa82410b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0b98d0 con 0x7fa81c09db10 2021-04-09T18:29:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa82410b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0b9af0 con 0x7fa81c180120 2021-04-09T18:29:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa822728700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0b94d0 0x7fa81c1a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa822728700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0b94d0 0x7fa81c1a0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55218/0 (socket says 172.21.15.32:55218) 2021-04-09T18:29:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa822728700 1 -- 172.21.15.32:0/756809990 learned_addr learned my addr 172.21.15.32:0/756809990 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:59.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.124 7fa821726700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c09db10 0x7fa81c0b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa821f27700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c180120 0x7fa81c09d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa822728700 1 -- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c180120 msgr2=0x7fa81c09d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa822728700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c180120 0x7fa81c09d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa822728700 1 -- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c09db10 msgr2=0x7fa81c0b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa822728700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c09db10 0x7fa81c0b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa822728700 1 -- 172.21.15.32:0/756809990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c1a1550 con 0x7fa81c0b94d0 2021-04-09T18:29:59.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa822728700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0b94d0 0x7fa81c1a0f30 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 rx=0x7fa8140099a0 tx=0x7fa81400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa80e7fc700 1 -- 172.21.15.32:0/756809990 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8140011f0 con 0x7fa81c0b94d0 2021-04-09T18:29:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa80e7fc700 1 -- 172.21.15.32:0/756809990 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa814003030 con 0x7fa81c0b94d0 2021-04-09T18:29:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa82410b700 1 -- 172.21.15.32:0/756809990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c1a1550 con 0x7fa81c0b94d0 2021-04-09T18:29:59.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa80e7fc700 1 -- 172.21.15.32:0/756809990 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa814001570 con 0x7fa81c0b94d0 2021-04-09T18:29:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.125 7fa82410b700 1 -- 172.21.15.32:0/756809990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c1a1550 con 0x7fa81c0b94d0 2021-04-09T18:29:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.126 7fa80e7fc700 1 -- 172.21.15.32:0/756809990 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa814006950 con 0x7fa81c0b94d0 2021-04-09T18:29:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.126 7fa80e7fc700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.126 7fa80e7fc700 1 -- 172.21.15.32:0/756809990 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa814032d60 con 0x7fa81c0b94d0 2021-04-09T18:29:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.126 7fa821f27700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.126 7fa821f27700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 crc :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:59.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.146 7fa82410b700 1 -- 172.21.15.32:0/756809990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c0d8620 con 0x7fa81c0b94d0 2021-04-09T18:29:59.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.150 7fa80e7fc700 1 -- 172.21.15.32:0/756809990 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa81400be30 con 0x7fa81c0b94d0 2021-04-09T18:29:59.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.337 7fa82410b700 1 -- 172.21.15.32:0/756809990 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa81c0d8620 con 0x7fa81c0b94d0 2021-04-09T18:29:59.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.337 7fa80e7fc700 1 -- 172.21.15.32:0/756809990 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+156 (secure 0 0 0) 0x7fa814006950 con 0x7fa81c0b94d0 2021-04-09T18:29:59.425 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:29:59.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 -- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 msgr2=0x7fa80401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 crc :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 -- 172.21.15.32:0/756809990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0b94d0 msgr2=0x7fa81c1a0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0b94d0 0x7fa81c1a0f30 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 rx=0x7fa8140099a0 tx=0x7fa81400bcf0).stop 2021-04-09T18:29:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 -- 172.21.15.32:0/756809990 shutdown_connections 2021-04-09T18:29:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa804018930 0x7fa80401ae10 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa81c0b94d0 0x7fa81c1a0f30 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa81c09db10 0x7fa81c0b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 --2- 172.21.15.32:0/756809990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa81c180120 0x7fa81c09d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 -- 172.21.15.32:0/756809990 >> 172.21.15.32:0/756809990 conn(0x7fa81c0be8e0 msgr2=0x7fa81c0c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:59.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 -- 172.21.15.32:0/756809990 shutdown_connections 2021-04-09T18:29:59.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.340 7fa82410b700 1 -- 172.21.15.32:0/756809990 wait complete. 2021-04-09T18:29:59.437 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{"MDS_INSUFFICIENT_STANDBY":{"severity":"HEALTH_WARN","summary":{"message":"insufficient standby MDS daemons available"}}},"status":"HEALTH_WARN"} 2021-04-09T18:29:59.438 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2021-04-09T18:29:59.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:29:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.503 7f60a4aab700 1 Processor -- start 2021-04-09T18:29:59.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.503 7f60a4aab700 1 -- start start 2021-04-09T18:29:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.504 7f60a4aab700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f60a010d010 0x7f60a00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.504 7f60a4aab700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a010d480 con 0x7f60a00d8490 2021-04-09T18:29:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.504 7f60a4aab700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a010e1f0 con 0x7f60a00d1da0 2021-04-09T18:29:59.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.504 7f60a4aab700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a00d8d30 con 0x7f60a010d010 2021-04-09T18:29:59.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.504 7f609f05f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f60a00d8490 0x7f60a010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36582/0 (socket says 172.21.15.32:36582) 2021-04-09T18:29:59.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.504 7f609f05f700 1 -- 172.21.15.32:0/2767625168 learned_addr learned my addr 172.21.15.32:0/2767625168 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:29:59.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.504 7f609e85e700 1 --2- 172.21.15.32:0/2767625168 >> v2:172.21.15.86:3300/0 conn(0x7f60a010d010 0x7f60a00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.504 7f609d85c700 1 -- 172.21.15.32:0/2767625168 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083339284 0 0) 0x7f6098000d30 con 0x7f60a00d8490 2021-04-09T18:29:59.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f609d85c700 1 -- 172.21.15.32:0/2767625168 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6088002fe0 con 0x7f60a00d8490 2021-04-09T18:29:59.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f609d85c700 1 -- 172.21.15.32:0/2767625168 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4204004588 0 0) 0x7f6098001460 con 0x7f60a00d8490 2021-04-09T18:29:59.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f609e85e700 1 -- 172.21.15.32:0/2767625168 >> v1:172.21.15.86:6789/0 conn(0x7f60a00d1da0 legacy=0x7f60a00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f609e85e700 1 -- 172.21.15.32:0/2767625168 >> v1:172.21.15.32:6789/0 conn(0x7f60a00d8490 legacy=0x7f60a010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f609e85e700 1 -- 172.21.15.32:0/2767625168 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a00d9220 con 0x7f60a010d010 2021-04-09T18:29:59.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f609e85e700 1 --2- 172.21.15.32:0/2767625168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a010d010 0x7f60a00d1780 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f6090006650 tx=0x7f6090009a00).ready entity=mon.1 client_cookie=65aabac6c0ef4168 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:59.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f609d85c700 1 -- 172.21.15.32:0/2767625168 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6090009df0 con 0x7f60a010d010 2021-04-09T18:29:59.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f609d85c700 1 -- 172.21.15.32:0/2767625168 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60900009b0 con 0x7f60a010d010 2021-04-09T18:29:59.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f609d85c700 1 -- 172.21.15.32:0/2767625168 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f609000c2b0 con 0x7f60a010d010 2021-04-09T18:29:59.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f60a4aab700 1 -- 172.21.15.32:0/2767625168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a010d010 msgr2=0x7f60a00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.505 7f60a4aab700 1 --2- 172.21.15.32:0/2767625168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a010d010 0x7f60a00d1780 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f6090006650 tx=0x7f6090009a00).stop 2021-04-09T18:29:59.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.506 7f60a4aab700 1 -- 172.21.15.32:0/2767625168 shutdown_connections 2021-04-09T18:29:59.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.506 7f60a4aab700 1 --2- 172.21.15.32:0/2767625168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a010d010 0x7f60a00d1780 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.506 7f60a4aab700 1 -- 172.21.15.32:0/2767625168 >> 172.21.15.32:0/2767625168 conn(0x7f60a00ce5f0 msgr2=0x7f60a00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:29:59.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.506 7f60a4aab700 1 -- 172.21.15.32:0/2767625168 shutdown_connections 2021-04-09T18:29:59.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.506 7f60a4aab700 1 -- 172.21.15.32:0/2767625168 wait complete. 2021-04-09T18:29:59.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.526 7f60a4aab700 1 Processor -- start 2021-04-09T18:29:59.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.526 7f60a4aab700 1 -- start start 2021-04-09T18:29:59.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f60a4aab700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60a008fab0 0x7f60a008ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f60a4aab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a0090540 0x7f60a0090940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f60a4aab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a0194c00 0x7f60a0195000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f60a4aab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a0095da0 con 0x7f60a0090540 2021-04-09T18:29:59.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f60a4aab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a008ffc0 con 0x7f60a0194c00 2021-04-09T18:29:59.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f60a4aab700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a0090f60 con 0x7f60a008fab0 2021-04-09T18:29:59.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a0090540 0x7f60a0090940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a0090540 0x7f60a0090940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55230/0 (socket says 172.21.15.32:55230) 2021-04-09T18:29:59.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e05d700 1 -- 172.21.15.32:0/2728404437 learned_addr learned my addr 172.21.15.32:0/2728404437 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:29:59.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e85e700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60a008fab0 0x7f60a008ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609f05f700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a0194c00 0x7f60a0195000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e05d700 1 -- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60a008fab0 msgr2=0x7f60a008ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e05d700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60a008fab0 0x7f60a008ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e05d700 1 -- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a0194c00 msgr2=0x7f60a0195000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:29:59.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e05d700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a0194c00 0x7f60a0195000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:29:59.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e05d700 1 -- 172.21.15.32:0/2728404437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a0195620 con 0x7f60a0090540 2021-04-09T18:29:59.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f609e05d700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a0090540 0x7f60a0090940 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7f6094006f20 tx=0x7f609400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:59.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f608f7fe700 1 -- 172.21.15.32:0/2728404437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6094007070 con 0x7f60a0090540 2021-04-09T18:29:59.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f608f7fe700 1 -- 172.21.15.32:0/2728404437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60940073b0 con 0x7f60a0090540 2021-04-09T18:29:59.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a0195620 con 0x7f60a0090540 2021-04-09T18:29:59.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.527 7f608f7fe700 1 -- 172.21.15.32:0/2728404437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6094007800 con 0x7f60a0090540 2021-04-09T18:29:59.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.528 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a0195620 con 0x7f60a0090540 2021-04-09T18:29:59.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.529 7f608f7fe700 1 -- 172.21.15.32:0/2728404437 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6094007000 con 0x7f60a0090540 2021-04-09T18:29:59.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.529 7f608f7fe700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60840188c0 0x7f608401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:29:59.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.529 7f608f7fe700 1 -- 172.21.15.32:0/2728404437 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f60940327c0 con 0x7f60a0090540 2021-04-09T18:29:59.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.529 7f609e85e700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60840188c0 0x7f608401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:29:59.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.529 7f609e85e700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60840188c0 0x7f608401adc0 crc :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:29:59.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.548 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a0095da0 con 0x7f60a0090540 2021-04-09T18:29:59.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.551 7f608f7fe700 1 -- 172.21.15.32:0/2728404437 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6094006bf0 con 0x7f60a0090540 2021-04-09T18:29:59.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.743 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f60a0095da0 con 0x7f60a0090540 2021-04-09T18:30:00.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.932 7f608f7fe700 1 -- 172.21.15.32:0/2728404437 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v316) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6094007000 con 0x7f60a0090540 2021-04-09T18:30:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60840188c0 msgr2=0x7f608401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60840188c0 0x7f608401adc0 crc :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a0090540 msgr2=0x7f60a0090940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a0090540 0x7f60a0090940 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7f6094006f20 tx=0x7f609400b990).stop 2021-04-09T18:30:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 shutdown_connections 2021-04-09T18:30:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60840188c0 0x7f608401adc0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60a0090540 0x7f60a0090940 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60a0194c00 0x7f60a0195000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 --2- 172.21.15.32:0/2728404437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60a008fab0 0x7f60a008ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 >> 172.21.15.32:0/2728404437 conn(0x7f60a00b8980 msgr2=0x7f60a01b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:00.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 shutdown_connections 2021-04-09T18:30:00.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:29:59.959 7f60a4aab700 1 -- 172.21.15.32:0/2728404437 wait complete. 2021-04-09T18:30:00.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-04-09T18:30:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.118 7fe147035700 1 Processor -- start 2021-04-09T18:30:00.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.118 7fe147035700 1 -- start start 2021-04-09T18:30:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.119 7fe147035700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1400cffa0 0x7fe1400d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.119 7fe147035700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1400d7800 0x7fe1400d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.119 7fe147035700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1400d7280 con 0x7fe1400d8220 2021-04-09T18:30:00.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.119 7fe147035700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1400d91e0 con 0x7fe1400cffa0 2021-04-09T18:30:00.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.119 7fe147035700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14010dfe0 con 0x7fe1400d7800 2021-04-09T18:30:00.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.119 7fe144e51700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe1400cffa0 0x7fe1400d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:00.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.119 7fe145652700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe1400d8220 0x7fe1400d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36598/0 (socket says 172.21.15.32:36598) 2021-04-09T18:30:00.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.119 7fe145652700 1 -- 172.21.15.32:0/2834713160 learned_addr learned my addr 172.21.15.32:0/2834713160 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:00.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.119 7fe13ffff700 1 --2- 172.21.15.32:0/2834713160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1400d7800 0x7fe1400d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:00.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe144e51700 1 -- 172.21.15.32:0/2834713160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1400d7800 msgr2=0x7fe1400d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:00.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe144e51700 1 --2- 172.21.15.32:0/2834713160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1400d7800 0x7fe1400d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe144e51700 1 -- 172.21.15.32:0/2834713160 >> v1:172.21.15.32:6789/0 conn(0x7fe1400d8220 legacy=0x7fe1400d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:00.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe144e51700 1 -- 172.21.15.32:0/2834713160 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14010e200 con 0x7fe1400cffa0 2021-04-09T18:30:00.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe13f7fe700 1 -- 172.21.15.32:0/2834713160 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273305620 0 0) 0x7fe128000c10 con 0x7fe1400d8220 2021-04-09T18:30:00.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe144e51700 1 --2- 172.21.15.32:0/2834713160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1400cffa0 0x7fe1400d2440 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7fe138006650 tx=0x7fe138009a00).ready entity=mon.0 client_cookie=dab0c1ebaa574308 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:00.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe13f7fe700 1 -- 172.21.15.32:0/2834713160 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe138009df0 con 0x7fe1400cffa0 2021-04-09T18:30:00.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe13f7fe700 1 -- 172.21.15.32:0/2834713160 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1380009b0 con 0x7fe1400cffa0 2021-04-09T18:30:00.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe13f7fe700 1 -- 172.21.15.32:0/2834713160 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe13800c2b0 con 0x7fe1400cffa0 2021-04-09T18:30:00.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe147035700 1 -- 172.21.15.32:0/2834713160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1400cffa0 msgr2=0x7fe1400d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:00.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.120 7fe147035700 1 --2- 172.21.15.32:0/2834713160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1400cffa0 0x7fe1400d2440 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7fe138006650 tx=0x7fe138009a00).stop 2021-04-09T18:30:00.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.121 7fe147035700 1 -- 172.21.15.32:0/2834713160 shutdown_connections 2021-04-09T18:30:00.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.121 7fe147035700 1 --2- 172.21.15.32:0/2834713160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1400d7800 0x7fe1400d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.121 7fe147035700 1 --2- 172.21.15.32:0/2834713160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1400cffa0 0x7fe1400d2440 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.121 7fe147035700 1 -- 172.21.15.32:0/2834713160 >> 172.21.15.32:0/2834713160 conn(0x7fe1400ce5c0 msgr2=0x7fe1400cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:00.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.121 7fe147035700 1 -- 172.21.15.32:0/2834713160 shutdown_connections 2021-04-09T18:30:00.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.121 7fe147035700 1 -- 172.21.15.32:0/2834713160 wait complete. 2021-04-09T18:30:00.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.141 7fe147035700 1 Processor -- start 2021-04-09T18:30:00.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.141 7fe147035700 1 -- start start 2021-04-09T18:30:00.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe147035700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1400b4250 0x7fe14009a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:00.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe147035700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14018e280 0x7fe140192370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:00.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe147035700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe140184d80 0x7fe140191250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:00.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe147035700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe140192a10 con 0x7fe140184d80 2021-04-09T18:30:00.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe147035700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe140191df0 con 0x7fe1400b4250 2021-04-09T18:30:00.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe147035700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe140191870 con 0x7fe14018e280 2021-04-09T18:30:00.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe145652700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe140184d80 0x7fe140191250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:00.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe145652700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe140184d80 0x7fe140191250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55246/0 (socket says 172.21.15.32:55246) 2021-04-09T18:30:00.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe145652700 1 -- 172.21.15.32:0/3274991827 learned_addr learned my addr 172.21.15.32:0/3274991827 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:00.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe13ffff700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14018e280 0x7fe140192370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:00.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe144e51700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1400b4250 0x7fe14009a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:00.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe145652700 1 -- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14018e280 msgr2=0x7fe140192370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:00.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe145652700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14018e280 0x7fe140192370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe145652700 1 -- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1400b4250 msgr2=0x7fe14009a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:00.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe145652700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1400b4250 0x7fe14009a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe145652700 1 -- 172.21.15.32:0/3274991827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140191a90 con 0x7fe140184d80 2021-04-09T18:30:00.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe13ffff700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14018e280 0x7fe140192370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:30:00.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe144e51700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1400b4250 0x7fe14009a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:30:00.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe145652700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe140184d80 0x7fe140191250 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7fe12800a590 tx=0x7fe12800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:00.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.142 7fe13d7fa700 1 -- 172.21.15.32:0/3274991827 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe12800ad20 con 0x7fe140184d80 2021-04-09T18:30:00.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.143 7fe13d7fa700 1 -- 172.21.15.32:0/3274991827 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1280043a0 con 0x7fe140184d80 2021-04-09T18:30:00.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.143 7fe147035700 1 -- 172.21.15.32:0/3274991827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe140191a90 con 0x7fe140184d80 2021-04-09T18:30:00.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.143 7fe13d7fa700 1 -- 172.21.15.32:0/3274991827 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1280048c0 con 0x7fe140184d80 2021-04-09T18:30:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.143 7fe147035700 1 -- 172.21.15.32:0/3274991827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe140191a90 con 0x7fe140184d80 2021-04-09T18:30:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.143 7fe13d7fa700 1 -- 172.21.15.32:0/3274991827 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe128004580 con 0x7fe140184d80 2021-04-09T18:30:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.143 7fe13d7fa700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe124018930 0x7fe12401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.143 7fe13d7fa700 1 -- 172.21.15.32:0/3274991827 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe1280329e0 con 0x7fe140184d80 2021-04-09T18:30:00.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.144 7fe144e51700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe124018930 0x7fe12401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:00.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.144 7fe144e51700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe124018930 0x7fe12401ae10 crc :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:00.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.163 7fe147035700 1 -- 172.21.15.32:0/3274991827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe140182d60 con 0x7fe140184d80 2021-04-09T18:30:00.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.167 7fe13d7fa700 1 -- 172.21.15.32:0/3274991827 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe12800ad20 con 0x7fe140184d80 2021-04-09T18:30:00.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.353 7fe147035700 1 -- 172.21.15.32:0/3274991827 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe140182d60 con 0x7fe140184d80 2021-04-09T18:30:00.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.353 7fe13d7fa700 1 -- 172.21.15.32:0/3274991827 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+34 (secure 0 0 0) 0x7fe1280043a0 con 0x7fe140184d80 2021-04-09T18:30:00.441 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:00.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 -- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe124018930 msgr2=0x7fe12401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe124018930 0x7fe12401ae10 crc :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 -- 172.21.15.32:0/3274991827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe140184d80 msgr2=0x7fe140191250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe140184d80 0x7fe140191250 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7fe12800a590 tx=0x7fe12800a930).stop 2021-04-09T18:30:00.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 -- 172.21.15.32:0/3274991827 shutdown_connections 2021-04-09T18:30:00.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe124018930 0x7fe12401ae10 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe140184d80 0x7fe140191250 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14018e280 0x7fe140192370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 --2- 172.21.15.32:0/3274991827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1400b4250 0x7fe14009a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:00.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 -- 172.21.15.32:0/3274991827 >> 172.21.15.32:0/3274991827 conn(0x7fe140090a90 msgr2=0x7fe1401832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:00.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 -- 172.21.15.32:0/3274991827 shutdown_connections 2021-04-09T18:30:00.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:00.356 7fe147035700 1 -- 172.21.15.32:0/3274991827 wait complete. 2021-04-09T18:30:00.454 INFO:teuthology.orchestra.run.smithi032.stdout:{"checks":{},"status":"HEALTH_OK"} 2021-04-09T18:30:00.454 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2021-04-09T18:30:00.455 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:30:00.456 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:30:00.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:30:00.580 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:30:00.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:00.581 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:00.581 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:00.625 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:30:00.626 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:30:00.626 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:30:00.627 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:30:00.627 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:00.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[87871]: fuse finished with error 0 and tester_r 0 2021-04-09T18:30:00.733 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:00.734 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:00.734 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:00.779 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:30:00.779 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:30:00.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:30:00.890 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:00.973 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:00.974 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:00.974 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:30:01.024 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:30:01.024 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:30:01.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:30:01.025 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:01.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:30:01.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.135 7fd5700be700 1 Processor -- start 2021-04-09T18:30:01.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.136 7fd5700be700 1 -- start start 2021-04-09T18:30:01.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.136 7fd5700be700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd56810d010 0x7fd5680d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.136 7fd5700be700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5680d8490 0x7fd56810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.136 7fd5700be700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56810d480 con 0x7fd5680d1da0 2021-04-09T18:30:01.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.136 7fd5700be700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5680d8d30 con 0x7fd56810d010 2021-04-09T18:30:01.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.136 7fd5700be700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5680d9220 con 0x7fd5680d8490 2021-04-09T18:30:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.136 7fd56deda700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd56810d010 0x7fd5680d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56deda700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd56810d010 0x7fd5680d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55250/0 (socket says 172.21.15.32:55250) 2021-04-09T18:30:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56deda700 1 -- 172.21.15.32:0/3025401332 learned_addr learned my addr 172.21.15.32:0/3025401332 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56e6db700 1 --2- 172.21.15.32:0/3025401332 >> v2:172.21.15.86:3300/0 conn(0x7fd5680d8490 0x7fd56810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56deda700 1 -- 172.21.15.32:0/3025401332 >> v2:172.21.15.86:3300/0 conn(0x7fd5680d8490 msgr2=0x7fd56810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56deda700 1 --2- 172.21.15.32:0/3025401332 >> v2:172.21.15.86:3300/0 conn(0x7fd5680d8490 0x7fd56810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56deda700 1 -- 172.21.15.32:0/3025401332 >> v1:172.21.15.86:6789/0 conn(0x7fd5680d1da0 legacy=0x7fd5680d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56deda700 1 -- 172.21.15.32:0/3025401332 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56810e020 con 0x7fd56810d010 2021-04-09T18:30:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56deda700 1 --2- 172.21.15.32:0/3025401332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56810d010 0x7fd5680d1780 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7fd564006700 tx=0x7fd564009b00).ready entity=mon.0 client_cookie=1692a1268ed9a16f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56ced8700 1 -- 172.21.15.32:0/3025401332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd56400c0c0 con 0x7fd56810d010 2021-04-09T18:30:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56ced8700 1 -- 172.21.15.32:0/3025401332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56400c400 con 0x7fd56810d010 2021-04-09T18:30:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd56ced8700 1 -- 172.21.15.32:0/3025401332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd56400c8e0 con 0x7fd56810d010 2021-04-09T18:30:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd5700be700 1 -- 172.21.15.32:0/3025401332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56810d010 msgr2=0x7fd5680d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.137 7fd5700be700 1 --2- 172.21.15.32:0/3025401332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56810d010 0x7fd5680d1780 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7fd564006700 tx=0x7fd564009b00).stop 2021-04-09T18:30:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.138 7fd5700be700 1 -- 172.21.15.32:0/3025401332 shutdown_connections 2021-04-09T18:30:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.138 7fd5700be700 1 --2- 172.21.15.32:0/3025401332 >> v2:172.21.15.86:3300/0 conn(0x7fd5680d8490 0x7fd56810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.138 7fd5700be700 1 --2- 172.21.15.32:0/3025401332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd56810d010 0x7fd5680d1780 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.138 7fd5700be700 1 -- 172.21.15.32:0/3025401332 >> 172.21.15.32:0/3025401332 conn(0x7fd5680ce5f0 msgr2=0x7fd5680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.138 7fd5700be700 1 -- 172.21.15.32:0/3025401332 shutdown_connections 2021-04-09T18:30:01.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.138 7fd5700be700 1 -- 172.21.15.32:0/3025401332 wait complete. 2021-04-09T18:30:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.158 7fd5700be700 1 Processor -- start 2021-04-09T18:30:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.158 7fd5700be700 1 -- start start 2021-04-09T18:30:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd5700be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd568179bf0 0x7fd568179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd5700be700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56817c540 0x7fd56817c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd5700be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680aca60 0x7fd5680ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd5700be700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56817cf60 con 0x7fd568179bf0 2021-04-09T18:30:01.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd5700be700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680ad480 con 0x7fd56817c540 2021-04-09T18:30:01.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd5700be700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680ad6a0 con 0x7fd5680aca60 2021-04-09T18:30:01.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56deda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd568179bf0 0x7fd568179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56deda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd568179bf0 0x7fd568179ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55256/0 (socket says 172.21.15.32:55256) 2021-04-09T18:30:01.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56deda700 1 -- 172.21.15.32:0/2604299386 learned_addr learned my addr 172.21.15.32:0/2604299386 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:01.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56e6db700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680aca60 0x7fd5680ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56d6d9700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56817c540 0x7fd56817c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56deda700 1 -- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680aca60 msgr2=0x7fd5680ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56deda700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680aca60 0x7fd5680ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56deda700 1 -- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56817c540 msgr2=0x7fd56817c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:01.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56deda700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56817c540 0x7fd56817c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56deda700 1 -- 172.21.15.32:0/2604299386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680ad8c0 con 0x7fd568179bf0 2021-04-09T18:30:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56d6d9700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56817c540 0x7fd56817c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd56deda700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd568179bf0 0x7fd568179ff0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7fd56400b660 tx=0x7fd56400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:01.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd55a7fc700 1 -- 172.21.15.32:0/2604299386 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd564009440 con 0x7fd568179bf0 2021-04-09T18:30:01.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd55a7fc700 1 -- 172.21.15.32:0/2604299386 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd564006390 con 0x7fd568179bf0 2021-04-09T18:30:01.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd5700be700 1 -- 172.21.15.32:0/2604299386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5680ad8c0 con 0x7fd568179bf0 2021-04-09T18:30:01.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.159 7fd55a7fc700 1 -- 172.21.15.32:0/2604299386 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd564002030 con 0x7fd568179bf0 2021-04-09T18:30:01.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.160 7fd5700be700 1 -- 172.21.15.32:0/2604299386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5680ad8c0 con 0x7fd568179bf0 2021-04-09T18:30:01.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.160 7fd55a7fc700 1 -- 172.21.15.32:0/2604299386 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd5640093e0 con 0x7fd568179bf0 2021-04-09T18:30:01.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.160 7fd55a7fc700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd550018930 0x7fd55001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.160 7fd55a7fc700 1 -- 172.21.15.32:0/2604299386 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd5640329f0 con 0x7fd568179bf0 2021-04-09T18:30:01.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.160 7fd56d6d9700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd550018930 0x7fd55001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.161 7fd56d6d9700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd550018930 0x7fd55001ae10 crc :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:01.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.181 7fd5700be700 1 -- 172.21.15.32:0/2604299386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5681b7010 con 0x7fd568179bf0 2021-04-09T18:30:01.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.184 7fd55a7fc700 1 -- 172.21.15.32:0/2604299386 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd5640090c0 con 0x7fd568179bf0 2021-04-09T18:30:01.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.372 7fd5700be700 1 -- 172.21.15.32:0/2604299386 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd56810d480 con 0x7fd568179bf0 2021-04-09T18:30:01.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.373 7fd55a7fc700 1 -- 172.21.15.32:0/2604299386 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v168) v1 ==== 81+0+30374 (secure 0 0 0) 0x7fd564006390 con 0x7fd568179bf0 2021-04-09T18:30:01.462 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:01.462 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:30:01.463 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 168, 2021-04-09T18:30:01.463 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:30:01.463 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:30:01.464 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:29:25.668831", 2021-04-09T18:30:01.464 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:30:01.464 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:30:01.465 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:30:01.465 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:30:01.465 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:30:01.466 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:30:01.466 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:30:01.466 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:30:01.466 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:30:01.466 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:30:01.467 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:30:01.467 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:30:01.467 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:30:01.467 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:30:01.468 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:30:01.468 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 29, 2021-04-09T18:30:01.468 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:30:01.468 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:30:01.469 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:30:01.469 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:30:01.469 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:30:01.469 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.469 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:30:01.470 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:30:01.470 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:30:01.470 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:30:01.470 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:30:01.470 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:30:01.471 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:30:01.471 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:30:01.471 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:30:01.471 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:30:01.472 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:30:01.472 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:30:01.472 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:30:01.472 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:30:01.473 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:30:01.473 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:30:01.473 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:30:01.473 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:30:01.473 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:30:01.474 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:30:01.474 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:30:01.474 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:30:01.474 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:30:01.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.475 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:30:01.475 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:30:01.475 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:30:01.476 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:30:01.476 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:30:01.476 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:30:01.476 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:30:01.477 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:30:01.477 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:30:01.477 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:30:01.477 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:30:01.477 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:30:01.478 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:30:01.478 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:30:01.478 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:30:01.478 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:30:01.479 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:30:01.479 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:30:01.479 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:30:01.479 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:30:01.480 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:30:01.480 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:30:01.480 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:30:01.480 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:30:01.481 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:30:01.481 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:30:01.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:30:01.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.481 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:30:01.482 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:30:01.482 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:30:01.482 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:30:01.482 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:30:01.483 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:30:01.483 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:30:01.483 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:30:01.483 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:30:01.484 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:30:01.484 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:30:01.484 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:30:01.484 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:30:01.484 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:30:01.485 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.485 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.485 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 28, 2021-04-09T18:30:01.486 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:30:01.486 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:29:22.005467", 2021-04-09T18:30:01.486 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:30:01.486 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:30:01.486 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:30:01.487 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:30:01.487 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:30:01.487 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:30:01.488 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:30:01.488 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:30:01.488 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:30:01.488 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:30:01.489 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:30:01.489 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:30:01.489 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:30:01.489 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:30:01.490 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:30:01.490 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:30:01.490 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:30:01.491 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:30:01.491 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:30:01.491 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:30:01.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.492 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "167", 2021-04-09T18:30:01.493 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:30:01.493 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:30:01.494 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:30:01.494 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:30:01.494 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:30:01.495 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:30:01.495 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:30:01.496 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:30:01.496 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:30:01.496 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:30:01.497 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:30:01.497 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:30:01.498 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:30:01.498 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:30:01.498 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:30:01.499 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:30:01.499 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:30:01.500 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:30:01.500 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:30:01.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:30:01.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:30:01.501 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:30:01.502 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:30:01.502 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:30:01.504 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:30:01.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:30:01.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.505 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:30:01.505 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:30:01.505 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:30:01.505 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:30:01.506 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:30:01.506 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:30:01.506 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:30:01.506 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:30:01.506 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:30:01.507 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:30:01.507 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:30:01.507 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:30:01.507 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:30:01.508 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:30:01.508 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:30:01.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.508 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:30:01.509 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:30:01.509 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:30:01.509 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.509 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.510 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.510 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 29, 2021-04-09T18:30:01.510 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:30:01.510 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:29:23.428510", 2021-04-09T18:30:01.511 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:30:01.511 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:30:01.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:30:01.511 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:30:01.512 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:30:01.512 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:30:01.512 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:30:01.512 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:30:01.512 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:30:01.513 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:30:01.513 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:30:01.513 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:30:01.514 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:30:01.514 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:30:01.514 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:30:01.515 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:30:01.515 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:30:01.516 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:30:01.516 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:30:01.517 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:30:01.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.517 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "168", 2021-04-09T18:30:01.518 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:30:01.518 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:30:01.518 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:30:01.519 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:30:01.519 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:30:01.519 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:30:01.519 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:30:01.520 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:30:01.520 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:30:01.520 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:30:01.520 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:30:01.520 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:30:01.521 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:30:01.521 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:30:01.521 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:30:01.521 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:30:01.521 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:30:01.522 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:30:01.522 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:30:01.522 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:30:01.522 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:30:01.522 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:30:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:30:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:30:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:30:01.523 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:30:01.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.524 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:30:01.524 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:30:01.524 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:30:01.524 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:30:01.525 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:30:01.525 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:30:01.525 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:30:01.525 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:30:01.525 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:30:01.526 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:30:01.526 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:30:01.526 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:30:01.526 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:30:01.526 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:30:01.527 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:30:01.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.528 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:30:01.528 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:30:01.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.529 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:30:01.529 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:30:01.529 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:01.529 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:01.530 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:01.530 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:01.530 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:01.531 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:01.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:01.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 161, 2021-04-09T18:30:01.531 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:01.532 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:01.532 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:01.532 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.532 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.533 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:30:01.533 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:01.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:30:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:01.534 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.535 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.535 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:01.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.537 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.537 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:30:01.537 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:01.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.538 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:30:01.539 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:01.539 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.540 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.541 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:01.541 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.541 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.542 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.542 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:30:01.542 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:01.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:30:01.544 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:01.544 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.544 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.545 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:01.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.546 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:30:01.547 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:01.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.548 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.548 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:30:01.548 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:01.548 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.548 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.549 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:30:01.549 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:30:01.549 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:30:01.549 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:30:01.550 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:01.550 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:01.550 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:01.550 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.551 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.551 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:30:01.551 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:30:01.552 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:01.552 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:01.552 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:01.552 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:01.552 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:01.553 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:01.553 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:01.553 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 166, 2021-04-09T18:30:01.553 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:01.554 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:01.554 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:01.554 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.554 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.555 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:30:01.555 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:01.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.556 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:30:01.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:01.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.558 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.559 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:01.559 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.560 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:30:01.561 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:01.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:30:01.563 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:01.563 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.564 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.565 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:01.565 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.566 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.567 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:30:01.567 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:01.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.568 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.568 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.568 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:30:01.568 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:01.569 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.569 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.569 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:01.570 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.570 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:30:01.570 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:01.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.571 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.571 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.571 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:30:01.572 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:01.572 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.572 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.572 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:30:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:30:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:30:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:30:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:01.573 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.574 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:30:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:30:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:01.575 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:01.576 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:30:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:01.577 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.578 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:30:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.579 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:30:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.581 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:30:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.582 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:30:01.583 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:01.583 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.583 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.584 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:01.584 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.584 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.584 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:30:01.585 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:01.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.585 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.585 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.586 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:30:01.586 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:01.586 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.586 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.587 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:01.587 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.587 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.587 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:30:01.588 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:01.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.588 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.588 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.588 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:30:01.589 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:01.589 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.589 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.590 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:30:01.590 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:30:01.590 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:30:01.590 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:30:01.591 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:01.591 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:01.591 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:01.591 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.592 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:30:01.592 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:30:01.592 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:01.593 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:01.593 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:01.593 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:01.593 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:01.593 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:01.594 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:01.594 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:30:01.594 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:01.594 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:01.594 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:01.595 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.595 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.595 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:30:01.596 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:01.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.596 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.596 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.596 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:30:01.597 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:01.598 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.598 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.598 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:01.598 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.599 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.599 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.599 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:30:01.599 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:01.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.600 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:30:01.601 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:01.601 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.601 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.601 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:01.602 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.602 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:30:01.603 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:01.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.603 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:30:01.604 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:01.604 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.604 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.605 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:01.605 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.605 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.605 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.605 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:30:01.606 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:01.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.606 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:30:01.607 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:01.607 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.607 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.608 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:30:01.608 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:30:01.608 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:30:01.608 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:30:01.608 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:01.609 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:01.609 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:01.609 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.610 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:30:01.610 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:30:01.610 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:01.610 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:01.611 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:01.611 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:01.611 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:01.611 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:01.611 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:01.612 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 166, 2021-04-09T18:30:01.612 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:01.612 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:01.612 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:01.612 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.613 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:30:01.613 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:01.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.614 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:30:01.614 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:01.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.615 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.615 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:01.616 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.616 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:30:01.617 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:01.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.617 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:30:01.618 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:01.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.618 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.618 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:01.619 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.619 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.619 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:30:01.620 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:01.620 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.620 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:30:01.621 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:01.621 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.621 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.622 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:01.622 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.622 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.622 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:30:01.623 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:01.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.623 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:30:01.624 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:01.624 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.624 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.625 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:30:01.625 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:30:01.625 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:30:01.625 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:30:01.625 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:01.626 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:01.626 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:01.626 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.627 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:30:01.627 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:30:01.627 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:01.627 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:01.628 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:01.628 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:01.628 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:01.628 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:01.628 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:01.629 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 161, 2021-04-09T18:30:01.629 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:01.629 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:01.629 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:01.630 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.630 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.630 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:30:01.630 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:01.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.631 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:30:01.632 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:01.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.632 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:01.633 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.633 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.633 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.633 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:30:01.633 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:01.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.634 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:30:01.634 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:01.635 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.635 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.635 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:01.636 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.636 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.636 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:30:01.636 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:01.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.637 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:30:01.637 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:01.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.638 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:01.639 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.639 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:30:01.640 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:01.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.640 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.640 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:30:01.641 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:01.641 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.641 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.642 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:30:01.642 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:30:01.642 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:30:01.642 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:30:01.642 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:01.643 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:01.643 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:01.643 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.644 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:30:01.644 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:30:01.644 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:01.644 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:01.645 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:01.645 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:01.645 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:01.645 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:01.645 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:01.646 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 166, 2021-04-09T18:30:01.646 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:01.646 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:01.646 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:01.646 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:30:01.647 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:01.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.648 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:30:01.648 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:01.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.649 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.649 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:01.649 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.650 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:30:01.650 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:01.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.651 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.651 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:30:01.651 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:01.651 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.652 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.652 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:01.652 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.653 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.653 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:30:01.653 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:01.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.654 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:30:01.654 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:01.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.655 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.655 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:01.655 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.656 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:30:01.656 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:01.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.657 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.657 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:30:01.657 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:01.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.658 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.658 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:30:01.658 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:30:01.659 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:30:01.659 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:30:01.659 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:01.659 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:01.659 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:01.660 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.660 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.660 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:30:01.661 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:30:01.661 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:01.661 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:01.661 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:01.662 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:01.662 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:01.662 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:01.662 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:01.662 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 166, 2021-04-09T18:30:01.663 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:01.663 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:01.663 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:01.663 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.663 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.664 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:30:01.664 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:01.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.665 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.665 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:30:01.665 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:01.665 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.666 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.666 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:01.666 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.666 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.667 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.667 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:30:01.667 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:01.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.667 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.668 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.668 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:30:01.668 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:01.668 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.668 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.669 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:01.669 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.669 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.670 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.670 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:30:01.670 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:01.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.671 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.671 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.672 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:30:01.672 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:01.673 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.673 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.674 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:01.674 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:01.674 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.675 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:01.675 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:30:01.676 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:01.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:01.677 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:30:01.677 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:01.678 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.678 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:30:01.679 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:30:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:30:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:30:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:01.680 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:30:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:30:01.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:30:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:01.682 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.683 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:30:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:01.684 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:30:01.685 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:01.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:30:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:01.687 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.688 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:30:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:01.689 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.690 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:30:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:01.691 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:30:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:01.693 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:01.694 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:30:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:01.696 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:01.697 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:30:01.698 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:30:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:30:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:30:01.699 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:30:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:30:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:30:01.700 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:30:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:30:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:30:01.701 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:30:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:30:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:30:01.702 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:30:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:01.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:30:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:30:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:30:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:30:01.704 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:30:01.705 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:30:01.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 -- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd550018930 msgr2=0x7fd55001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:01.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd550018930 0x7fd55001ae10 crc :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 -- 172.21.15.32:0/2604299386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd568179bf0 msgr2=0x7fd568179ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:01.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd568179bf0 0x7fd568179ff0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7fd56400b660 tx=0x7fd56400ba00).stop 2021-04-09T18:30:01.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 -- 172.21.15.32:0/2604299386 shutdown_connections 2021-04-09T18:30:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd550018930 0x7fd55001ae10 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680aca60 0x7fd5680ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56817c540 0x7fd56817c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 --2- 172.21.15.32:0/2604299386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd568179bf0 0x7fd568179ff0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 -- 172.21.15.32:0/2604299386 >> 172.21.15.32:0/2604299386 conn(0x7fd5681acae0 msgr2=0x7fd56817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 -- 172.21.15.32:0/2604299386 shutdown_connections 2021-04-09T18:30:01.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.377 7fd5700be700 1 -- 172.21.15.32:0/2604299386 wait complete. 2021-04-09T18:30:01.710 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:01.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.782 7fd0f54cf700 1 Processor -- start 2021-04-09T18:30:01.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.782 7fd0f54cf700 1 -- start start 2021-04-09T18:30:01.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.782 7fd0f54cf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0f00d7ca0 0x7fd0f00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.782 7fd0f54cf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd0f010da00 0x7fd0f010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.782 7fd0f54cf700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f010d480 con 0x7fd0f010d010 2021-04-09T18:30:01.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.782 7fd0f54cf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f00d9220 con 0x7fd0f010da00 2021-04-09T18:30:01.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.782 7fd0f54cf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00d86c0 con 0x7fd0f00d7ca0 2021-04-09T18:30:01.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0efac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd0f010da00 0x7fd0f010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0efac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd0f010da00 0x7fd0f010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55264/0 (socket says 172.21.15.32:55264) 2021-04-09T18:30:01.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0efac0700 1 -- 172.21.15.32:0/1724751054 learned_addr learned my addr 172.21.15.32:0/1724751054 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:01.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0efac0700 1 -- 172.21.15.32:0/1724751054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0f00d7ca0 msgr2=0x7fd0f00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:01.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0efac0700 1 --2- 172.21.15.32:0/1724751054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0f00d7ca0 0x7fd0f00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-04-09T18:30:01.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0efac0700 1 -- 172.21.15.32:0/1724751054 >> v1:172.21.15.86:6789/0 conn(0x7fd0f010d010 legacy=0x7fd0f00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:01.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0efac0700 1 -- 172.21.15.32:0/1724751054 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f00d1f70 con 0x7fd0f010da00 2021-04-09T18:30:01.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0efac0700 1 --2- 172.21.15.32:0/1724751054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f010da00 0x7fd0f010de00 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7fd0e8006f00 tx=0x7fd0e8009f10).ready entity=mon.0 client_cookie=bb0031b9a7835ac6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:01.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0ee2bd700 1 -- 172.21.15.32:0/1724751054 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0e800b3b0 con 0x7fd0f010da00 2021-04-09T18:30:01.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0ee2bd700 1 -- 172.21.15.32:0/1724751054 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e800b670 con 0x7fd0f010da00 2021-04-09T18:30:01.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0ee2bd700 1 -- 172.21.15.32:0/1724751054 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0e800bb50 con 0x7fd0f010da00 2021-04-09T18:30:01.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0f54cf700 1 -- 172.21.15.32:0/1724751054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f010da00 msgr2=0x7fd0f010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:01.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.783 7fd0f54cf700 1 --2- 172.21.15.32:0/1724751054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f010da00 0x7fd0f010de00 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7fd0e8006f00 tx=0x7fd0e8009f10).stop 2021-04-09T18:30:01.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.784 7fd0f54cf700 1 -- 172.21.15.32:0/1724751054 shutdown_connections 2021-04-09T18:30:01.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.784 7fd0f54cf700 1 --2- 172.21.15.32:0/1724751054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f010da00 0x7fd0f010de00 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.784 7fd0f54cf700 1 --2- 172.21.15.32:0/1724751054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0f00d7ca0 0x7fd0f00d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-04-09T18:30:01.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.784 7fd0f54cf700 1 -- 172.21.15.32:0/1724751054 >> 172.21.15.32:0/1724751054 conn(0x7fd0f00ce5f0 msgr2=0x7fd0f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:01.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.784 7fd0f54cf700 1 -- 172.21.15.32:0/1724751054 shutdown_connections 2021-04-09T18:30:01.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.784 7fd0f54cf700 1 -- 172.21.15.32:0/1724751054 wait complete. 2021-04-09T18:30:01.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.804 7fd0f54cf700 1 Processor -- start 2021-04-09T18:30:01.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.804 7fd0f54cf700 1 -- start start 2021-04-09T18:30:01.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0f54cf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0f01a31a0 0x7fd0f01a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0f54cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f01a7b80 0x7fd0f01a7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0f54cf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0f01a85a0 0x7fd0f01a89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0f54cf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f0195420 con 0x7fd0f01a7b80 2021-04-09T18:30:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0f54cf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f01a9860 con 0x7fd0f01a31a0 2021-04-09T18:30:01.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0f54cf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f01a9db0 con 0x7fd0f01a85a0 2021-04-09T18:30:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0eeabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f01a7b80 0x7fd0f01a7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0eeabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f01a7b80 0x7fd0f01a7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55270/0 (socket says 172.21.15.32:55270) 2021-04-09T18:30:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0eeabe700 1 -- 172.21.15.32:0/612931876 learned_addr learned my addr 172.21.15.32:0/612931876 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0efac0700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0f01a85a0 0x7fd0f01a89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0ef2bf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0f01a31a0 0x7fd0f01a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0eeabe700 1 -- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0f01a85a0 msgr2=0x7fd0f01a89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0eeabe700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0f01a85a0 0x7fd0f01a89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0eeabe700 1 -- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0f01a31a0 msgr2=0x7fd0f01a35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:01.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0eeabe700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0f01a31a0 0x7fd0f01a35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0eeabe700 1 -- 172.21.15.32:0/612931876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f0187e80 con 0x7fd0f01a7b80 2021-04-09T18:30:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0ef2bf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0f01a31a0 0x7fd0f01a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.805 7fd0eeabe700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f01a7b80 0x7fd0f01a7f80 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7fd0e4006f00 tx=0x7fd0e400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:01.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.806 7fd0d7fff700 1 -- 172.21.15.32:0/612931876 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0e400bd30 con 0x7fd0f01a7b80 2021-04-09T18:30:01.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.806 7fd0d7fff700 1 -- 172.21.15.32:0/612931876 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e4000c90 con 0x7fd0f01a7b80 2021-04-09T18:30:01.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.806 7fd0d7fff700 1 -- 172.21.15.32:0/612931876 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0e40022a0 con 0x7fd0f01a7b80 2021-04-09T18:30:01.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.806 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f0187e80 con 0x7fd0f01a7b80 2021-04-09T18:30:01.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.806 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f0187e80 con 0x7fd0f01a7b80 2021-04-09T18:30:01.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.806 7fd0d7fff700 1 -- 172.21.15.32:0/612931876 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd0e4005310 con 0x7fd0f01a7b80 2021-04-09T18:30:01.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.806 7fd0d7fff700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0d00188c0 0x7fd0d001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:01.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.807 7fd0d7fff700 1 -- 172.21.15.32:0/612931876 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd0e40329b0 con 0x7fd0f01a7b80 2021-04-09T18:30:01.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.807 7fd0ef2bf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0d00188c0 0x7fd0d001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:01.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.807 7fd0ef2bf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0d00188c0 0x7fd0d001adc0 crc :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:01.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.827 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f017ca90 con 0x7fd0f01a7b80 2021-04-09T18:30:01.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:01.830 7fd0d7fff700 1 -- 172.21.15.32:0/612931876 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd0e4006b90 con 0x7fd0f01a7b80 2021-04-09T18:30:02.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.021 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0f010d480 con 0x7fd0f01a7b80 2021-04-09T18:30:02.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.021 7fd0d7fff700 1 -- 172.21.15.32:0/612931876 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 316 v316) v1 ==== 95+0+2788 (secure 0 0 0) 0x7fd0e400bd30 con 0x7fd0f01a7b80 2021-04-09T18:30:02.109 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 316 2021-04-09T18:30:02.110 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:02.110 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":316,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3983419158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3983419158},{"type":"v1","addr":"172.21.15.32:6834","nonce":3983419158}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":314},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":316,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:29:59.931279","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10177":{"gid":10177,"name":"a","rank":0,"incarnation":308,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2200487686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2200487686},{"type":"v1","addr":"172.21.15.32:6835","nonce":2200487686}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-04-09T18:30:02.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0d00188c0 msgr2=0x7fd0d001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:02.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0d00188c0 0x7fd0d001adc0 crc :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f01a7b80 msgr2=0x7fd0f01a7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f01a7b80 0x7fd0f01a7f80 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7fd0e4006f00 tx=0x7fd0e400b940).stop 2021-04-09T18:30:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 shutdown_connections 2021-04-09T18:30:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0d00188c0 0x7fd0d001adc0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0f01a7b80 0x7fd0f01a7f80 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0f01a85a0 0x7fd0f01a89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 --2- 172.21.15.32:0/612931876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0f01a31a0 0x7fd0f01a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 >> 172.21.15.32:0/612931876 conn(0x7fd0f01965c0 msgr2=0x7fd0f0091300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 shutdown_connections 2021-04-09T18:30:02.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.024 7fd0f54cf700 1 -- 172.21.15.32:0/612931876 wait complete. 2021-04-09T18:30:02.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:30:02.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.195 7f64343d0700 1 Processor -- start 2021-04-09T18:30:02.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.195 7f64343d0700 1 -- start start 2021-04-09T18:30:02.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64343d0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f642c0cffb0 0x7f642c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:02.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64343d0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c0d6c90 0x7f642c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:02.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64343d0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c0d7290 con 0x7f642c0d7810 2021-04-09T18:30:02.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64343d0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c0d7090 con 0x7f642c0cffb0 2021-04-09T18:30:02.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64343d0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0d91f0 con 0x7f642c0d6c90 2021-04-09T18:30:02.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64321ec700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f642c0cffb0 0x7f642c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:02.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64329ed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c0d6c90 0x7f642c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:02.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64319eb700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f642c0d7810 0x7f642c0d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:34564/0 (socket says 172.21.15.32:34564) 2021-04-09T18:30:02.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64319eb700 1 -- 172.21.15.32:0/3067901129 learned_addr learned my addr 172.21.15.32:0/3067901129 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:02.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64311ea700 1 -- 172.21.15.32:0/3067901129 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884474556 0 0) 0x7f641c0009b0 con 0x7f642c0d7810 2021-04-09T18:30:02.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64311ea700 1 -- 172.21.15.32:0/3067901129 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6418002fe0 con 0x7f642c0d7810 2021-04-09T18:30:02.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64329ed700 1 -- 172.21.15.32:0/3067901129 >> v2:172.21.15.86:3300/0 conn(0x7f642c0cffb0 msgr2=0x7f642c0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:02.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64329ed700 1 --2- 172.21.15.32:0/3067901129 >> v2:172.21.15.86:3300/0 conn(0x7f642c0cffb0 0x7f642c0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64329ed700 1 -- 172.21.15.32:0/3067901129 >> v1:172.21.15.86:6789/0 conn(0x7f642c0d7810 legacy=0x7f642c0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:02.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64329ed700 1 -- 172.21.15.32:0/3067901129 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c0d8400 con 0x7f642c0d6c90 2021-04-09T18:30:02.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64321ec700 1 --2- 172.21.15.32:0/3067901129 >> v2:172.21.15.86:3300/0 conn(0x7f642c0cffb0 0x7f642c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:02.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.196 7f64329ed700 1 --2- 172.21.15.32:0/3067901129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c0d6c90 0x7f642c10cfe0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f6424006e50 tx=0x7f6424009e70).ready entity=mon.2 client_cookie=baa32cf6f3303108 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:02.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.197 7f64311ea700 1 -- 172.21.15.32:0/3067901129 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6424006a20 con 0x7f642c0d6c90 2021-04-09T18:30:02.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.197 7f64311ea700 1 -- 172.21.15.32:0/3067901129 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424006790 con 0x7f642c0d6c90 2021-04-09T18:30:02.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.197 7f64311ea700 1 -- 172.21.15.32:0/3067901129 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f642400b700 con 0x7f642c0d6c90 2021-04-09T18:30:02.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.197 7f64343d0700 1 -- 172.21.15.32:0/3067901129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c0d6c90 msgr2=0x7f642c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:02.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.197 7f64343d0700 1 --2- 172.21.15.32:0/3067901129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c0d6c90 0x7f642c10cfe0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f6424006e50 tx=0x7f6424009e70).stop 2021-04-09T18:30:02.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.198 7f64343d0700 1 -- 172.21.15.32:0/3067901129 shutdown_connections 2021-04-09T18:30:02.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.198 7f64343d0700 1 --2- 172.21.15.32:0/3067901129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c0d6c90 0x7f642c10cfe0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.198 7f64343d0700 1 --2- 172.21.15.32:0/3067901129 >> v2:172.21.15.86:3300/0 conn(0x7f642c0cffb0 0x7f642c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.198 7f64343d0700 1 -- 172.21.15.32:0/3067901129 >> 172.21.15.32:0/3067901129 conn(0x7f642c0ce5d0 msgr2=0x7f642c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:02.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.198 7f64343d0700 1 -- 172.21.15.32:0/3067901129 shutdown_connections 2021-04-09T18:30:02.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.198 7f64343d0700 1 -- 172.21.15.32:0/3067901129 wait complete. 2021-04-09T18:30:02.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64343d0700 1 Processor -- start 2021-04-09T18:30:02.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64343d0700 1 -- start start 2021-04-09T18:30:02.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64343d0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1b8380 0x7f642c1b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:02.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64343d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f642c08c7b0 0x7f642c08cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:02.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64343d0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c08d1d0 0x7f642c1805b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:02.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64343d0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c181a30 con 0x7f642c08c7b0 2021-04-09T18:30:02.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64343d0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c08c230 con 0x7f642c08d1d0 2021-04-09T18:30:02.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64343d0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1b8da0 con 0x7f642c1b8380 2021-04-09T18:30:02.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64319eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f642c08c7b0 0x7f642c08cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:02.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64319eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f642c08c7b0 0x7f642c08cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55284/0 (socket says 172.21.15.32:55284) 2021-04-09T18:30:02.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64319eb700 1 -- 172.21.15.32:0/2138267986 learned_addr learned my addr 172.21.15.32:0/2138267986 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:02.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.218 7f64321ec700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1b8380 0x7f642c1b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:02.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64329ed700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c08d1d0 0x7f642c1805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:02.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64319eb700 1 -- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1b8380 msgr2=0x7f642c1b8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:02.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64319eb700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1b8380 0x7f642c1b8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64319eb700 1 -- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c08d1d0 msgr2=0x7f642c1805b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:02.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64319eb700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c08d1d0 0x7f642c1805b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:02.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64319eb700 1 -- 172.21.15.32:0/2138267986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c1b8f90 con 0x7f642c08c7b0 2021-04-09T18:30:02.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64321ec700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1b8380 0x7f642c1b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:02.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64319eb700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f642c08c7b0 0x7f642c08cbb0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7f641c001c60 tx=0x7f641c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:02.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f6422ffd700 1 -- 172.21.15.32:0/2138267986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f641c00acc0 con 0x7f642c08c7b0 2021-04-09T18:30:02.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f6422ffd700 1 -- 172.21.15.32:0/2138267986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641c004140 con 0x7f642c08c7b0 2021-04-09T18:30:02.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f6422ffd700 1 -- 172.21.15.32:0/2138267986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f641c004510 con 0x7f642c08c7b0 2021-04-09T18:30:02.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64343d0700 1 -- 172.21.15.32:0/2138267986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c1b8f90 con 0x7f642c08c7b0 2021-04-09T18:30:02.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.219 7f64343d0700 1 -- 172.21.15.32:0/2138267986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c1b8f90 con 0x7f642c08c7b0 2021-04-09T18:30:02.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.220 7f6422ffd700 1 -- 172.21.15.32:0/2138267986 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f641c009030 con 0x7f642c08c7b0 2021-04-09T18:30:02.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.220 7f6422ffd700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64140188c0 0x7f641401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:02.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.220 7f6422ffd700 1 -- 172.21.15.32:0/2138267986 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f641c032820 con 0x7f642c08c7b0 2021-04-09T18:30:02.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.220 7f64321ec700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64140188c0 0x7f641401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:02.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.221 7f64321ec700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64140188c0 0x7f641401adc0 crc :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:02.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.240 7f64343d0700 1 -- 172.21.15.32:0/2138267986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c1b7180 con 0x7f642c08c7b0 2021-04-09T18:30:02.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.243 7f6422ffd700 1 -- 172.21.15.32:0/2138267986 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f641c0009b0 con 0x7f642c08c7b0 2021-04-09T18:30:02.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.425 7f64343d0700 1 -- 172.21.15.32:0/2138267986 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f642c08c230 con 0x7f642c08c7b0 2021-04-09T18:30:03.027 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:30:02.940 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:30:03.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.944 7f6422ffd700 1 -- 172.21.15.32:0/2138267986 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v317) v1 ==== 161+0+0 (secure 0 0 0) 0x7f641c009030 con 0x7f642c08c7b0 2021-04-09T18:30:03.076 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:30:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 -- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64140188c0 msgr2=0x7f641401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64140188c0 0x7f641401adc0 crc :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 -- 172.21.15.32:0/2138267986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f642c08c7b0 msgr2=0x7f642c08cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f642c08c7b0 0x7f642c08cbb0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7f641c001c60 tx=0x7f641c00a8d0).stop 2021-04-09T18:30:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 -- 172.21.15.32:0/2138267986 shutdown_connections 2021-04-09T18:30:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64140188c0 0x7f641401adc0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f642c08c7b0 0x7f642c08cbb0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c08d1d0 0x7f642c1805b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 --2- 172.21.15.32:0/2138267986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1b8380 0x7f642c1b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 -- 172.21.15.32:0/2138267986 >> 172.21.15.32:0/2138267986 conn(0x7f642c189800 msgr2=0x7f642c1b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:03.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 -- 172.21.15.32:0/2138267986 shutdown_connections 2021-04-09T18:30:03.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:02.992 7f64343d0700 1 -- 172.21.15.32:0/2138267986 wait complete. 2021-04-09T18:30:03.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:03.096 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:30:03.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.153 7f6de9e5c700 1 Processor -- start 2021-04-09T18:30:03.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.153 7f6de9e5c700 1 -- start start 2021-04-09T18:30:03.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de9e5c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de410d010 0x7f6de40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de9e5c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6de40d8490 0x7f6de410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de9e5c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de410d480 con 0x7f6de40d1da0 2021-04-09T18:30:03.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de9e5c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de410e780 con 0x7f6de40d8490 2021-04-09T18:30:03.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de9e5c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de40d8d30 con 0x7f6de410d010 2021-04-09T18:30:03.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6de40d8490 0x7f6de410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6de40d8490 0x7f6de410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55304/0 (socket says 172.21.15.32:55304) 2021-04-09T18:30:03.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de3fff700 1 -- 172.21.15.32:0/3784925022 learned_addr learned my addr 172.21.15.32:0/3784925022 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:03.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de37fe700 1 --2- 172.21.15.32:0/3784925022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de410d010 0x7f6de40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de3fff700 1 -- 172.21.15.32:0/3784925022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de410d010 msgr2=0x7f6de40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:03.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de3fff700 1 --2- 172.21.15.32:0/3784925022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de410d010 0x7f6de40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de3fff700 1 -- 172.21.15.32:0/3784925022 >> v1:172.21.15.86:6789/0 conn(0x7f6de40d1da0 legacy=0x7f6de40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.154 7f6de3fff700 1 -- 172.21.15.32:0/3784925022 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de40d9220 con 0x7f6de40d8490 2021-04-09T18:30:03.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.155 7f6de3fff700 1 --2- 172.21.15.32:0/3784925022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de40d8490 0x7f6de410da00 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7f6ddc006f00 tx=0x7f6ddc009e90).ready entity=mon.0 client_cookie=eb424eae6f71cd65 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:03.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.155 7f6de27fc700 1 -- 172.21.15.32:0/3784925022 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ddc00b300 con 0x7f6de40d8490 2021-04-09T18:30:03.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.155 7f6de27fc700 1 -- 172.21.15.32:0/3784925022 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ddc00b640 con 0x7f6de40d8490 2021-04-09T18:30:03.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.155 7f6de27fc700 1 -- 172.21.15.32:0/3784925022 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ddc00bb20 con 0x7f6de40d8490 2021-04-09T18:30:03.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.155 7f6de9e5c700 1 -- 172.21.15.32:0/3784925022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de40d8490 msgr2=0x7f6de410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.155 7f6de9e5c700 1 --2- 172.21.15.32:0/3784925022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de40d8490 0x7f6de410da00 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7f6ddc006f00 tx=0x7f6ddc009e90).stop 2021-04-09T18:30:03.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.156 7f6de9e5c700 1 -- 172.21.15.32:0/3784925022 shutdown_connections 2021-04-09T18:30:03.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.156 7f6de9e5c700 1 --2- 172.21.15.32:0/3784925022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de40d8490 0x7f6de410da00 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.156 7f6de9e5c700 1 --2- 172.21.15.32:0/3784925022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de410d010 0x7f6de40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.156 7f6de9e5c700 1 -- 172.21.15.32:0/3784925022 >> 172.21.15.32:0/3784925022 conn(0x7f6de40ce5f0 msgr2=0x7f6de40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:03.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.156 7f6de9e5c700 1 -- 172.21.15.32:0/3784925022 shutdown_connections 2021-04-09T18:30:03.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.156 7f6de9e5c700 1 -- 172.21.15.32:0/3784925022 wait complete. 2021-04-09T18:30:03.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.176 7f6de9e5c700 1 Processor -- start 2021-04-09T18:30:03.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.176 7f6de9e5c700 1 -- start start 2021-04-09T18:30:03.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de9e5c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de40b8480 0x7f6de40b8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de9e5c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de418d800 0x7f6de418dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de9e5c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a9500 0x7f6de41a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de9e5c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de418d280 con 0x7f6de418d800 2021-04-09T18:30:03.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de9e5c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de418e220 con 0x7f6de40b8480 2021-04-09T18:30:03.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de9e5c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de41a9f20 con 0x7f6de41a9500 2021-04-09T18:30:03.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de418d800 0x7f6de418dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de2ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de418d800 0x7f6de418dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55308/0 (socket says 172.21.15.32:55308) 2021-04-09T18:30:03.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de2ffd700 1 -- 172.21.15.32:0/837361787 learned_addr learned my addr 172.21.15.32:0/837361787 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:03.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de37fe700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de40b8480 0x7f6de40b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de3fff700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a9500 0x7f6de41a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de2ffd700 1 -- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a9500 msgr2=0x7f6de41a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de2ffd700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a9500 0x7f6de41a9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de2ffd700 1 -- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de40b8480 msgr2=0x7f6de40b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de2ffd700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de40b8480 0x7f6de40b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de2ffd700 1 -- 172.21.15.32:0/837361787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de41aa140 con 0x7f6de418d800 2021-04-09T18:30:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de37fe700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de40b8480 0x7f6de40b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de2ffd700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de418d800 0x7f6de418dc00 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 rx=0x7f6dd800b5b0 tx=0x7f6dd800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:03.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6dcbfff700 1 -- 172.21.15.32:0/837361787 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6dd800bd40 con 0x7f6de418d800 2021-04-09T18:30:03.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6dcbfff700 1 -- 172.21.15.32:0/837361787 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd8006020 con 0x7f6de418d800 2021-04-09T18:30:03.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6dcbfff700 1 -- 172.21.15.32:0/837361787 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6dd80054a0 con 0x7f6de418d800 2021-04-09T18:30:03.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.177 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de41aa140 con 0x7f6de418d800 2021-04-09T18:30:03.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.178 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de41aa140 con 0x7f6de418d800 2021-04-09T18:30:03.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.178 7f6dcbfff700 1 -- 172.21.15.32:0/837361787 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6dd8006200 con 0x7f6de418d800 2021-04-09T18:30:03.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.178 7f6dcbfff700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.178 7f6dcbfff700 1 -- 172.21.15.32:0/837361787 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f6dd8032a50 con 0x7f6de418d800 2021-04-09T18:30:03.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.179 7f6de37fe700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.179 7f6de37fe700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401adc0 crc :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:03.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.198 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de4182d70 con 0x7f6de418d800 2021-04-09T18:30:03.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.202 7f6dcbfff700 1 -- 172.21.15.32:0/837361787 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6dd800bd40 con 0x7f6de418d800 2021-04-09T18:30:03.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.391 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6de410d480 con 0x7f6de418d800 2021-04-09T18:30:03.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.392 7f6dcbfff700 1 -- 172.21.15.32:0/837361787 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 317 v317) v1 ==== 95+0+2440 (secure 0 0 0) 0x7f6dd8006000 con 0x7f6de418d800 2021-04-09T18:30:03.480 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 317 2021-04-09T18:30:03.480 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:03.481 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":317,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3983419158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3983419158},{"type":"v1","addr":"172.21.15.32:6834","nonce":3983419158}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":314},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[{"mdsmap":{"epoch":317,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:29:24.667843","modified":"2021-04-09 18:30:02.941314","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":169,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-04-09T18:30:03.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 msgr2=0x7f6dc401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401adc0 crc :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de418d800 msgr2=0x7f6de418dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de418d800 0x7f6de418dc00 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 rx=0x7f6dd800b5b0 tx=0x7f6dd800b950).stop 2021-04-09T18:30:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 shutdown_connections 2021-04-09T18:30:03.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6dc4018930 0x7f6dc401adc0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6de418d800 0x7f6de418dc00 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6de41a9500 0x7f6de41a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 --2- 172.21.15.32:0/837361787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6de40b8480 0x7f6de40b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.394 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 >> 172.21.15.32:0/837361787 conn(0x7f6de408fb70 msgr2=0x7f6de419ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:03.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.395 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 shutdown_connections 2021-04-09T18:30:03.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.395 7f6de9e5c700 1 -- 172.21.15.32:0/837361787 wait complete. 2021-04-09T18:30:03.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:30:03.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.596 7fb2f4235700 1 Processor -- start 2021-04-09T18:30:03.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.596 7fb2f4235700 1 -- start start 2021-04-09T18:30:03.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f4235700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb2ec0cffb0 0x7fb2ec0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f4235700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec0d7290 con 0x7fb2ec0d6c90 2021-04-09T18:30:03.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f4235700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec0d8400 con 0x7fb2ec0d7810 2021-04-09T18:30:03.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f4235700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ec0d7090 con 0x7fb2ec0cffb0 2021-04-09T18:30:03.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f2051700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb2ec0cffb0 0x7fb2ec0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f2852700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb2ec0d6c90 0x7fb2ec10cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36674/0 (socket says 172.21.15.32:36674) 2021-04-09T18:30:03.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f2852700 1 -- 172.21.15.32:0/159584648 learned_addr learned my addr 172.21.15.32:0/159584648 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:03.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f2051700 1 -- 172.21.15.32:0/159584648 >> v1:172.21.15.86:6789/0 conn(0x7fb2ec0d7810 legacy=0x7fb2ec0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f2051700 1 -- 172.21.15.32:0/159584648 >> v1:172.21.15.32:6789/0 conn(0x7fb2ec0d6c90 legacy=0x7fb2ec10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f2051700 1 -- 172.21.15.32:0/159584648 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec0d91f0 con 0x7fb2ec0cffb0 2021-04-09T18:30:03.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f104f700 1 -- 172.21.15.32:0/159584648 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995862883 0 0) 0x7fb2e4000d30 con 0x7fb2ec0d6c90 2021-04-09T18:30:03.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.597 7fb2f2051700 1 --2- 172.21.15.32:0/159584648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec0cffb0 0x7fb2ec0d2450 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7fb2e8006650 tx=0x7fb2e8009a00).ready entity=mon.0 client_cookie=58b656a1730b10c7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:03.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.598 7fb2f104f700 1 -- 172.21.15.32:0/159584648 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2e8009df0 con 0x7fb2ec0cffb0 2021-04-09T18:30:03.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.598 7fb2f104f700 1 -- 172.21.15.32:0/159584648 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e80009b0 con 0x7fb2ec0cffb0 2021-04-09T18:30:03.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.598 7fb2f104f700 1 -- 172.21.15.32:0/159584648 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2e800c2b0 con 0x7fb2ec0cffb0 2021-04-09T18:30:03.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.598 7fb2f4235700 1 -- 172.21.15.32:0/159584648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec0cffb0 msgr2=0x7fb2ec0d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.598 7fb2f4235700 1 --2- 172.21.15.32:0/159584648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec0cffb0 0x7fb2ec0d2450 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7fb2e8006650 tx=0x7fb2e8009a00).stop 2021-04-09T18:30:03.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.598 7fb2f4235700 1 -- 172.21.15.32:0/159584648 shutdown_connections 2021-04-09T18:30:03.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.598 7fb2f4235700 1 --2- 172.21.15.32:0/159584648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec0cffb0 0x7fb2ec0d2450 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.598 7fb2f4235700 1 -- 172.21.15.32:0/159584648 >> 172.21.15.32:0/159584648 conn(0x7fb2ec0ce5d0 msgr2=0x7fb2ec0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:03.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.599 7fb2f4235700 1 -- 172.21.15.32:0/159584648 shutdown_connections 2021-04-09T18:30:03.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.599 7fb2f4235700 1 -- 172.21.15.32:0/159584648 wait complete. 2021-04-09T18:30:03.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f4235700 1 Processor -- start 2021-04-09T18:30:03.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f4235700 1 -- start start 2021-04-09T18:30:03.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f4235700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2ec1a38b0 0x7fb2ec1a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f4235700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2ec1a8390 0x7fb2ec1a8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f4235700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec1a8db0 0x7fb2ec1a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f4235700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1a4630 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f4235700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1a3360 con 0x7fb2ec1a38b0 2021-04-09T18:30:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f4235700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec1a42d0 con 0x7fb2ec1a8390 2021-04-09T18:30:03.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f2852700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec1a8db0 0x7fb2ec1a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f2852700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec1a8db0 0x7fb2ec1a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55326/0 (socket says 172.21.15.32:55326) 2021-04-09T18:30:03.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f2852700 1 -- 172.21.15.32:0/3369344003 learned_addr learned my addr 172.21.15.32:0/3369344003 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:03.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f2051700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2ec1a38b0 0x7fb2ec1a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.619 7fb2f1850700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2ec1a8390 0x7fb2ec1a8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2f2852700 1 -- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2ec1a8390 msgr2=0x7fb2ec1a8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2f2852700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2ec1a8390 0x7fb2ec1a8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2f2852700 1 -- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2ec1a38b0 msgr2=0x7fb2ec1a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:03.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2f2852700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2ec1a38b0 0x7fb2ec1a3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:03.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2f2852700 1 -- 172.21.15.32:0/3369344003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec1a91b0 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2f2852700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec1a8db0 0x7fb2ec1a1f90 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7fb2e4007f90 tx=0x7fb2e400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:03.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2e2ffd700 1 -- 172.21.15.32:0/3369344003 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2e4000cc0 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2e2ffd700 1 -- 172.21.15.32:0/3369344003 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e4007020 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec1a91b0 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2e2ffd700 1 -- 172.21.15.32:0/3369344003 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2e4007500 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.620 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec1a91b0 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.621 7fb2e2ffd700 1 -- 172.21.15.32:0/3369344003 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb2e4007200 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.621 7fb2e2ffd700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb2d4018930 0x7fb2d401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:03.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.621 7fb2e2ffd700 1 -- 172.21.15.32:0/3369344003 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fb2e4032d90 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.621 7fb2f2051700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb2d4018930 0x7fb2d401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:03.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.621 7fb2f2051700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb2d4018930 0x7fb2d401ae10 crc :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:03.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.641 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ec17cb40 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.644 7fb2e2ffd700 1 -- 172.21.15.32:0/3369344003 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb2e4007d00 con 0x7fb2ec1a8db0 2021-04-09T18:30:03.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.838 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb2ec17cb40 con 0x7fb2ec1a8db0 2021-04-09T18:30:04.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.948 7fb2e2ffd700 1 -- 172.21.15.32:0/3369344003 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v318) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb2e4007020 con 0x7fb2ec1a8db0 2021-04-09T18:30:04.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb2d4018930 msgr2=0x7fb2d401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:04.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb2d4018930 0x7fb2d401ae10 crc :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec1a8db0 msgr2=0x7fb2ec1a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:04.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec1a8db0 0x7fb2ec1a1f90 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7fb2e4007f90 tx=0x7fb2e400ac50).stop 2021-04-09T18:30:04.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 shutdown_connections 2021-04-09T18:30:04.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb2d4018930 0x7fb2d401ae10 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2ec1a8db0 0x7fb2ec1a1f90 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2ec1a8390 0x7fb2ec1a8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 --2- 172.21.15.32:0/3369344003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2ec1a38b0 0x7fb2ec1a3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 >> 172.21.15.32:0/3369344003 conn(0x7fb2ec1a5690 msgr2=0x7fb2ec181d20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:04.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 shutdown_connections 2021-04-09T18:30:04.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:03.954 7fb2f4235700 1 -- 172.21.15.32:0/3369344003 wait complete. 2021-04-09T18:30:04.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:30:04.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.127 7fa955af3700 1 Processor -- start 2021-04-09T18:30:04.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.127 7fa955af3700 1 -- start start 2021-04-09T18:30:04.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa955af3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa9500d7cc0 0x7fa9500d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:04.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa955af3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa95010da80 0x7fa95010de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:04.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa955af3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95010d500 con 0x7fa95010d050 2021-04-09T18:30:04.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa955af3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9500d9260 con 0x7fa9500d7cc0 2021-04-09T18:30:04.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa955af3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9500d0330 con 0x7fa95010da80 2021-04-09T18:30:04.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa94effd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa9500d7cc0 0x7fa9500d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:04.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa94f7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa95010d050 0x7fa9500cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36688/0 (socket says 172.21.15.32:36688) 2021-04-09T18:30:04.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa94f7fe700 1 -- 172.21.15.32:0/1771653820 learned_addr learned my addr 172.21.15.32:0/1771653820 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:04.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa94effd700 1 --2- 172.21.15.32:0/1771653820 >> v2:172.21.15.32:3300/0 conn(0x7fa9500d7cc0 0x7fa9500d80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55332/0 (socket says 172.21.15.32:55332) 2021-04-09T18:30:04.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa94ffff700 1 --2- 172.21.15.32:0/1771653820 >> v2:172.21.15.86:3300/0 conn(0x7fa95010da80 0x7fa95010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:04.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa94effd700 1 -- 172.21.15.32:0/1771653820 >> v2:172.21.15.86:3300/0 conn(0x7fa95010da80 msgr2=0x7fa95010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:04.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa94effd700 1 --2- 172.21.15.32:0/1771653820 >> v2:172.21.15.86:3300/0 conn(0x7fa95010da80 0x7fa95010de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa94effd700 1 -- 172.21.15.32:0/1771653820 >> v1:172.21.15.32:6789/0 conn(0x7fa95010d050 legacy=0x7fa9500cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:04.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.129 7fa94effd700 1 -- 172.21.15.32:0/1771653820 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9500d86e0 con 0x7fa9500d7cc0 2021-04-09T18:30:04.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.128 7fa94e7fc700 1 -- 172.21.15.32:0/1771653820 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727858485 0 0) 0x7fa940000ce0 con 0x7fa95010d050 2021-04-09T18:30:04.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.129 7fa94ffff700 1 --2- 172.21.15.32:0/1771653820 >> v2:172.21.15.86:3300/0 conn(0x7fa95010da80 0x7fa95010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.129 7fa94effd700 1 --2- 172.21.15.32:0/1771653820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9500d7cc0 0x7fa9500d80c0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7fa944006650 tx=0x7fa944009a80).ready entity=mon.0 client_cookie=cf4a98fc17dc91f6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.129 7fa94e7fc700 1 -- 172.21.15.32:0/1771653820 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9440009b0 con 0x7fa9500d7cc0 2021-04-09T18:30:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.129 7fa94e7fc700 1 -- 172.21.15.32:0/1771653820 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94400c0c0 con 0x7fa9500d7cc0 2021-04-09T18:30:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.129 7fa94e7fc700 1 -- 172.21.15.32:0/1771653820 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa94400c510 con 0x7fa9500d7cc0 2021-04-09T18:30:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.129 7fa955af3700 1 -- 172.21.15.32:0/1771653820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9500d7cc0 msgr2=0x7fa9500d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.129 7fa955af3700 1 --2- 172.21.15.32:0/1771653820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9500d7cc0 0x7fa9500d80c0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7fa944006650 tx=0x7fa944009a80).stop 2021-04-09T18:30:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.130 7fa955af3700 1 -- 172.21.15.32:0/1771653820 shutdown_connections 2021-04-09T18:30:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.130 7fa955af3700 1 --2- 172.21.15.32:0/1771653820 >> v2:172.21.15.86:3300/0 conn(0x7fa95010da80 0x7fa95010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.130 7fa955af3700 1 --2- 172.21.15.32:0/1771653820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9500d7cc0 0x7fa9500d80c0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.130 7fa955af3700 1 -- 172.21.15.32:0/1771653820 >> 172.21.15.32:0/1771653820 conn(0x7fa9500ce620 msgr2=0x7fa9500d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.130 7fa955af3700 1 -- 172.21.15.32:0/1771653820 shutdown_connections 2021-04-09T18:30:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.130 7fa955af3700 1 -- 172.21.15.32:0/1771653820 wait complete. 2021-04-09T18:30:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.150 7fa955af3700 1 Processor -- start 2021-04-09T18:30:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.150 7fa955af3700 1 -- start start 2021-04-09T18:30:04.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa955af3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95009b2f0 0x7fa95009b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:04.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa955af3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95009a8b0 0x7fa95009e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:04.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa955af3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa95009e9c0 0x7fa95009edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:04.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa955af3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9500bde80 con 0x7fa95009a8b0 2021-04-09T18:30:04.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa955af3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95009ad70 con 0x7fa95009b2f0 2021-04-09T18:30:04.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa955af3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95009bd10 con 0x7fa95009e9c0 2021-04-09T18:30:04.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95009a8b0 0x7fa95009e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:04.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95009a8b0 0x7fa95009e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55336/0 (socket says 172.21.15.32:55336) 2021-04-09T18:30:04.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94effd700 1 -- 172.21.15.32:0/4226142077 learned_addr learned my addr 172.21.15.32:0/4226142077 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:04.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94f7fe700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95009b2f0 0x7fa95009b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:04.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94ffff700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa95009e9c0 0x7fa95009edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:04.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94effd700 1 -- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa95009e9c0 msgr2=0x7fa95009edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:04.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94effd700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa95009e9c0 0x7fa95009edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94effd700 1 -- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95009b2f0 msgr2=0x7fa95009b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:04.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94effd700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95009b2f0 0x7fa95009b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:04.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94effd700 1 -- 172.21.15.32:0/4226142077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95009fbd0 con 0x7fa95009a8b0 2021-04-09T18:30:04.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa94effd700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95009a8b0 0x7fa95009e3a0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 rx=0x7fa944009650 tx=0x7fa94400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:04.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa937fff700 1 -- 172.21.15.32:0/4226142077 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9440093a0 con 0x7fa95009a8b0 2021-04-09T18:30:04.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa937fff700 1 -- 172.21.15.32:0/4226142077 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94400bd50 con 0x7fa95009a8b0 2021-04-09T18:30:04.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa937fff700 1 -- 172.21.15.32:0/4226142077 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9440052e0 con 0x7fa95009a8b0 2021-04-09T18:30:04.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa955af3700 1 -- 172.21.15.32:0/4226142077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95009fbd0 con 0x7fa95009a8b0 2021-04-09T18:30:04.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.151 7fa955af3700 1 -- 172.21.15.32:0/4226142077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95009fbd0 con 0x7fa95009a8b0 2021-04-09T18:30:04.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.152 7fa937fff700 1 -- 172.21.15.32:0/4226142077 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa944006030 con 0x7fa95009a8b0 2021-04-09T18:30:04.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.152 7fa937fff700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa930018930 0x7fa93001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.152 7fa937fff700 1 -- 172.21.15.32:0/4226142077 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fa944032a30 con 0x7fa95009a8b0 2021-04-09T18:30:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.152 7fa94f7fe700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa930018930 0x7fa93001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.153 7fa94f7fe700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa930018930 0x7fa93001adc0 crc :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:04.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.172 7fa955af3700 1 -- 172.21.15.32:0/4226142077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9500b9040 con 0x7fa95009a8b0 2021-04-09T18:30:04.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.175 7fa937fff700 1 -- 172.21.15.32:0/4226142077 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa9440009c0 con 0x7fa95009a8b0 2021-04-09T18:30:04.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.356 7fa955af3700 1 -- 172.21.15.32:0/4226142077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa95010d500 con 0x7fa95009a8b0 2021-04-09T18:30:05.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.956 7fa937fff700 1 -- 172.21.15.32:0/4226142077 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v170) v1 ==== 185+0+0 (secure 0 0 0) 0x7fa944006030 con 0x7fa95009a8b0 2021-04-09T18:30:05.058 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:30:05.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 -- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa930018930 msgr2=0x7fa93001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:05.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa930018930 0x7fa93001adc0 crc :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 -- 172.21.15.32:0/4226142077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95009a8b0 msgr2=0x7fa95009e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:05.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95009a8b0 0x7fa95009e3a0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 rx=0x7fa944009650 tx=0x7fa94400b960).stop 2021-04-09T18:30:05.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 -- 172.21.15.32:0/4226142077 shutdown_connections 2021-04-09T18:30:05.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa930018930 0x7fa93001adc0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa95009a8b0 0x7fa95009e3a0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa95009e9c0 0x7fa95009edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 --2- 172.21.15.32:0/4226142077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa95009b2f0 0x7fa95009b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 -- 172.21.15.32:0/4226142077 >> 172.21.15.32:0/4226142077 conn(0x7fa9500ae6f0 msgr2=0x7fa950097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:05.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 -- 172.21.15.32:0/4226142077 shutdown_connections 2021-04-09T18:30:05.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:04.974 7fa955af3700 1 -- 172.21.15.32:0/4226142077 wait complete. 2021-04-09T18:30:05.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:30:05.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.152 7fadf4935700 1 Processor -- start 2021-04-09T18:30:05.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.152 7fadf4935700 1 -- start start 2021-04-09T18:30:05.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.153 7fadf4935700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fadf010d050 0x7fadf00cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:05.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.153 7fadf4935700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fadf00d0160 0x7fadf00d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:05.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.153 7fadf4935700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf010d500 con 0x7fadf00d82e0 2021-04-09T18:30:05.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.153 7fadf4935700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf00d9260 con 0x7fadf010d050 2021-04-09T18:30:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.153 7fadf4935700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf00d86e0 con 0x7fadf00d0160 2021-04-09T18:30:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.153 7fadee61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fadf010d050 0x7fadf00cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.153 7fadee61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fadf010d050 0x7fadf00cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55344/0 (socket says 172.21.15.32:55344) 2021-04-09T18:30:05.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.153 7fadee61c700 1 -- 172.21.15.32:0/3899970497 learned_addr learned my addr 172.21.15.32:0/3899970497 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7fadede1b700 1 --2- 172.21.15.32:0/3899970497 >> v2:172.21.15.86:3300/0 conn(0x7fadf00d0160 0x7fadf00d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:05.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7fadee61c700 1 -- 172.21.15.32:0/3899970497 >> v2:172.21.15.86:3300/0 conn(0x7fadf00d0160 msgr2=0x7fadf00d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7fadee61c700 1 --2- 172.21.15.32:0/3899970497 >> v2:172.21.15.86:3300/0 conn(0x7fadf00d0160 0x7fadf00d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7fadee61c700 1 -- 172.21.15.32:0/3899970497 >> v1:172.21.15.86:6789/0 conn(0x7fadf00d82e0 legacy=0x7fadf010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:05.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7fadee61c700 1 -- 172.21.15.32:0/3899970497 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf010e270 con 0x7fadf010d050 2021-04-09T18:30:05.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7fadee61c700 1 --2- 172.21.15.32:0/3899970497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf010d050 0x7fadf00cfb40 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7fade8006a80 tx=0x7fade8009e80).ready entity=mon.0 client_cookie=210407e71b3622f0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:05.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7faded61a700 1 -- 172.21.15.32:0/3899970497 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fade800b510 con 0x7fadf010d050 2021-04-09T18:30:05.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7faded61a700 1 -- 172.21.15.32:0/3899970497 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade800b7d0 con 0x7fadf010d050 2021-04-09T18:30:05.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7faded61a700 1 -- 172.21.15.32:0/3899970497 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fade800bcb0 con 0x7fadf010d050 2021-04-09T18:30:05.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7fadf4935700 1 -- 172.21.15.32:0/3899970497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf010d050 msgr2=0x7fadf00cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:05.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.154 7fadf4935700 1 --2- 172.21.15.32:0/3899970497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf010d050 0x7fadf00cfb40 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7fade8006a80 tx=0x7fade8009e80).stop 2021-04-09T18:30:05.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.155 7fadf4935700 1 -- 172.21.15.32:0/3899970497 shutdown_connections 2021-04-09T18:30:05.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.155 7fadf4935700 1 --2- 172.21.15.32:0/3899970497 >> v2:172.21.15.86:3300/0 conn(0x7fadf00d0160 0x7fadf00d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.155 7fadf4935700 1 --2- 172.21.15.32:0/3899970497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf010d050 0x7fadf00cfb40 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.155 7fadf4935700 1 -- 172.21.15.32:0/3899970497 >> 172.21.15.32:0/3899970497 conn(0x7fadf00ce620 msgr2=0x7fadf00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:05.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.155 7fadf4935700 1 -- 172.21.15.32:0/3899970497 shutdown_connections 2021-04-09T18:30:05.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.155 7fadf4935700 1 -- 172.21.15.32:0/3899970497 wait complete. 2021-04-09T18:30:05.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.175 7fadf4935700 1 Processor -- start 2021-04-09T18:30:05.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.175 7fadf4935700 1 -- start start 2021-04-09T18:30:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadf4935700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf009a890 0x7fadf00a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadf4935700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf009ac90 0x7fadf00a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadf4935700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf00c2220 0x7fadf00c2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadf4935700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf00b86e0 con 0x7fadf00c2220 2021-04-09T18:30:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadf4935700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf00a2920 con 0x7fadf009a890 2021-04-09T18:30:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadf4935700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf00c2c40 con 0x7fadf009ac90 2021-04-09T18:30:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadeee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf00c2220 0x7fadf00c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadeee1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf00c2220 0x7fadf00c2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55354/0 (socket says 172.21.15.32:55354) 2021-04-09T18:30:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadeee1d700 1 -- 172.21.15.32:0/3586282453 learned_addr learned my addr 172.21.15.32:0/3586282453 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadede1b700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf009ac90 0x7fadf00a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadee61c700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf009a890 0x7fadf00a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadeee1d700 1 -- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf009ac90 msgr2=0x7fadf00a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadeee1d700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf009ac90 0x7fadf00a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadeee1d700 1 -- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf009a890 msgr2=0x7fadf00a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadeee1d700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf009a890 0x7fadf00a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadeee1d700 1 -- 172.21.15.32:0/3586282453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf00c2e20 con 0x7fadf00c2220 2021-04-09T18:30:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadeee1d700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf00c2220 0x7fadf00c2620 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7fade0006ed0 tx=0x7fade000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fade6ffd700 1 -- 172.21.15.32:0/3586282453 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fade000bd50 con 0x7fadf00c2220 2021-04-09T18:30:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fade6ffd700 1 -- 172.21.15.32:0/3586282453 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade0006020 con 0x7fadf00c2220 2021-04-09T18:30:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadf4935700 1 -- 172.21.15.32:0/3586282453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf00c30c0 con 0x7fadf00c2220 2021-04-09T18:30:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fade6ffd700 1 -- 172.21.15.32:0/3586282453 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fade00054b0 con 0x7fadf00c2220 2021-04-09T18:30:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.176 7fadf4935700 1 -- 172.21.15.32:0/3586282453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf00c2e20 con 0x7fadf00c2220 2021-04-09T18:30:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.177 7fade6ffd700 1 -- 172.21.15.32:0/3586282453 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fade0006200 con 0x7fadf00c2220 2021-04-09T18:30:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.177 7fade6ffd700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadd4018930 0x7fadd401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.177 7fade6ffd700 1 -- 172.21.15.32:0/3586282453 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5225+0+0 (secure 0 0 0) 0x7fade0032870 con 0x7fadf00c2220 2021-04-09T18:30:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.177 7fadee61c700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadd4018930 0x7fadd401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.178 7fadee61c700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadd4018930 0x7fadd401ae10 crc :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.197 7fadf4935700 1 -- 172.21.15.32:0/3586282453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf01b7180 con 0x7fadf00c2220 2021-04-09T18:30:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.201 7fade6ffd700 1 -- 172.21.15.32:0/3586282453 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fade0006230 con 0x7fadf00c2220 2021-04-09T18:30:05.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.385 7fadf4935700 1 -- 172.21.15.32:0/3586282453 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fadf010d500 con 0x7fadf00c2220 2021-04-09T18:30:06.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:05.961 7fade6ffd700 1 -- 172.21.15.32:0/3586282453 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v171) v1 ==== 173+0+0 (secure 0 0 0) 0x7fade0005f00 con 0x7fadf00c2220 2021-04-09T18:30:06.087 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:30:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 -- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadd4018930 msgr2=0x7fadd401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadd4018930 0x7fadd401ae10 crc :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 -- 172.21.15.32:0/3586282453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf00c2220 msgr2=0x7fadf00c2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf00c2220 0x7fadf00c2620 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7fade0006ed0 tx=0x7fade000b960).stop 2021-04-09T18:30:06.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 -- 172.21.15.32:0/3586282453 shutdown_connections 2021-04-09T18:30:06.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadd4018930 0x7fadd401ae10 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadf00c2220 0x7fadf00c2620 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf009ac90 0x7fadf00a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 --2- 172.21.15.32:0/3586282453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf009a890 0x7fadf00a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 -- 172.21.15.32:0/3586282453 >> 172.21.15.32:0/3586282453 conn(0x7fadf00b6b70 msgr2=0x7fadf00b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:06.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 -- 172.21.15.32:0/3586282453 shutdown_connections 2021-04-09T18:30:06.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.003 7fadf4935700 1 -- 172.21.15.32:0/3586282453 wait complete. 2021-04-09T18:30:06.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:30:06.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.167 7fba9dc83700 1 Processor -- start 2021-04-09T18:30:06.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.167 7fba9dc83700 1 -- start start 2021-04-09T18:30:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba9dc83700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fba980d6c90 0x7fba980d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba9dc83700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba980d7290 con 0x7fba980d2450 2021-04-09T18:30:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba9dc83700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba980d2850 con 0x7fba980cffb0 2021-04-09T18:30:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba9dc83700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba980d7090 con 0x7fba980d6c90 2021-04-09T18:30:06.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba97fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fba980d2450 0x7fba9810cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36718/0 (socket says 172.21.15.32:36718) 2021-04-09T18:30:06.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba97fff700 1 -- 172.21.15.32:0/2527491576 learned_addr learned my addr 172.21.15.32:0/2527491576 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:06.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba96ffd700 1 --2- 172.21.15.32:0/2527491576 >> v2:172.21.15.86:3300/0 conn(0x7fba980d6c90 0x7fba980d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba967fc700 1 -- 172.21.15.32:0/2527491576 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60193363 0 0) 0x7fba90000d30 con 0x7fba980d2450 2021-04-09T18:30:06.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba967fc700 1 -- 172.21.15.32:0/2527491576 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba80002fe0 con 0x7fba980d2450 2021-04-09T18:30:06.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba967fc700 1 -- 172.21.15.32:0/2527491576 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101635718 0 0) 0x7fba88000c60 con 0x7fba980cffb0 2021-04-09T18:30:06.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba967fc700 1 -- 172.21.15.32:0/2527491576 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba80003210 con 0x7fba980cffb0 2021-04-09T18:30:06.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba967fc700 1 -- 172.21.15.32:0/2527491576 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4095999057 0 0) 0x7fba90001270 con 0x7fba980d2450 2021-04-09T18:30:06.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.168 7fba96ffd700 1 -- 172.21.15.32:0/2527491576 >> v1:172.21.15.86:6789/0 conn(0x7fba980cffb0 legacy=0x7fba980d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.169 7fba96ffd700 1 -- 172.21.15.32:0/2527491576 >> v1:172.21.15.32:6789/0 conn(0x7fba980d2450 legacy=0x7fba9810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.169 7fba96ffd700 1 -- 172.21.15.32:0/2527491576 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba980d91f0 con 0x7fba980d6c90 2021-04-09T18:30:06.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.169 7fba96ffd700 1 --2- 172.21.15.32:0/2527491576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba980d6c90 0x7fba980d8000 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7fba8c006650 tx=0x7fba8c009a00).ready entity=mon.1 client_cookie=9312f598372a69fd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:06.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.169 7fba967fc700 1 -- 172.21.15.32:0/2527491576 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba8c009df0 con 0x7fba980d6c90 2021-04-09T18:30:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.169 7fba967fc700 1 -- 172.21.15.32:0/2527491576 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c0009b0 con 0x7fba980d6c90 2021-04-09T18:30:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.169 7fba967fc700 1 -- 172.21.15.32:0/2527491576 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba8c00c2b0 con 0x7fba980d6c90 2021-04-09T18:30:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.169 7fba9dc83700 1 -- 172.21.15.32:0/2527491576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba980d6c90 msgr2=0x7fba980d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.169 7fba9dc83700 1 --2- 172.21.15.32:0/2527491576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba980d6c90 0x7fba980d8000 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7fba8c006650 tx=0x7fba8c009a00).stop 2021-04-09T18:30:06.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.170 7fba9dc83700 1 -- 172.21.15.32:0/2527491576 shutdown_connections 2021-04-09T18:30:06.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.170 7fba9dc83700 1 --2- 172.21.15.32:0/2527491576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba980d6c90 0x7fba980d8000 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.170 7fba9dc83700 1 -- 172.21.15.32:0/2527491576 >> 172.21.15.32:0/2527491576 conn(0x7fba980ce5d0 msgr2=0x7fba980cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:06.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.170 7fba9dc83700 1 -- 172.21.15.32:0/2527491576 shutdown_connections 2021-04-09T18:30:06.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.170 7fba9dc83700 1 -- 172.21.15.32:0/2527491576 wait complete. 2021-04-09T18:30:06.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.190 7fba9dc83700 1 Processor -- start 2021-04-09T18:30:06.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.190 7fba9dc83700 1 -- start start 2021-04-09T18:30:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.190 7fba9dc83700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba981a7dc0 0x7fba981a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba9dc83700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba98184080 0x7fba98184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba9dc83700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98184aa0 0x7fba9808f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba9dc83700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba981a8b40 con 0x7fba98184aa0 2021-04-09T18:30:06.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba9dc83700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba981a7870 con 0x7fba98184080 2021-04-09T18:30:06.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba9dc83700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba981a87e0 con 0x7fba981a7dc0 2021-04-09T18:30:06.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba97fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98184aa0 0x7fba9808f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba97fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98184aa0 0x7fba9808f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55368/0 (socket says 172.21.15.32:55368) 2021-04-09T18:30:06.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba97fff700 1 -- 172.21.15.32:0/3401334359 learned_addr learned my addr 172.21.15.32:0/3401334359 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:06.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba96ffd700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba98184080 0x7fba98184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba977fe700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba981a7dc0 0x7fba981a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba97fff700 1 -- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba981a7dc0 msgr2=0x7fba981a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba97fff700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba981a7dc0 0x7fba981a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba97fff700 1 -- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba98184080 msgr2=0x7fba98184480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba97fff700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba98184080 0x7fba98184480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba97fff700 1 -- 172.21.15.32:0/3401334359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9808f810 con 0x7fba98184aa0 2021-04-09T18:30:06.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba97fff700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98184aa0 0x7fba9808f1f0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7fba9000a880 tx=0x7fba9000ac20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba7ffff700 1 -- 172.21.15.32:0/3401334359 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba90004930 con 0x7fba98184aa0 2021-04-09T18:30:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba7ffff700 1 -- 172.21.15.32:0/3401334359 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba90004c30 con 0x7fba98184aa0 2021-04-09T18:30:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9808f810 con 0x7fba98184aa0 2021-04-09T18:30:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba7ffff700 1 -- 172.21.15.32:0/3401334359 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba900072a0 con 0x7fba98184aa0 2021-04-09T18:30:06.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.191 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9808f810 con 0x7fba98184aa0 2021-04-09T18:30:06.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.192 7fba7ffff700 1 -- 172.21.15.32:0/3401334359 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fba9000ad60 con 0x7fba98184aa0 2021-04-09T18:30:06.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.192 7fba7ffff700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba78018930 0x7fba7801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.192 7fba7ffff700 1 -- 172.21.15.32:0/3401334359 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fba90032a70 con 0x7fba98184aa0 2021-04-09T18:30:06.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.192 7fba977fe700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba78018930 0x7fba7801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.193 7fba977fe700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba78018930 0x7fba7801ae10 crc :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:06.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.212 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba98182e80 con 0x7fba98184aa0 2021-04-09T18:30:06.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.215 7fba7ffff700 1 -- 172.21.15.32:0/3401334359 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fba90004c30 con 0x7fba98184aa0 2021-04-09T18:30:06.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.427 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fba980d7290 con 0x7fba98184aa0 2021-04-09T18:30:06.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.430 7fba7ffff700 1 -- 172.21.15.32:0/3401334359 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fba9000ad60 con 0x7fba98184aa0 2021-04-09T18:30:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.436 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba78018930 msgr2=0x7fba7801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.436 7fba9dc83700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba78018930 0x7fba7801ae10 crc :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.436 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98184aa0 msgr2=0x7fba9808f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.436 7fba9dc83700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98184aa0 0x7fba9808f1f0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7fba9000a880 tx=0x7fba9000ac20).stop 2021-04-09T18:30:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.437 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 shutdown_connections 2021-04-09T18:30:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.437 7fba9dc83700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fba78018930 0x7fba7801ae10 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.437 7fba9dc83700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba98184aa0 0x7fba9808f1f0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.437 7fba9dc83700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba98184080 0x7fba98184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.437 7fba9dc83700 1 --2- 172.21.15.32:0/3401334359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba981a7dc0 0x7fba981a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.437 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 >> 172.21.15.32:0/3401334359 conn(0x7fba981a8f50 msgr2=0x7fba98188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:06.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.437 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 shutdown_connections 2021-04-09T18:30:06.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.437 7fba9dc83700 1 -- 172.21.15.32:0/3401334359 wait complete. 2021-04-09T18:30:06.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestFailover.test_standby_count_wanted' 2021-04-09T18:30:06.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.597 7fcbb10c8700 1 Processor -- start 2021-04-09T18:30:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.597 7fcbb10c8700 1 -- start start 2021-04-09T18:30:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbb10c8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac10d010 0x7fcbac0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbb10c8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcbac0d1da0 0x7fcbac0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbb10c8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac10d480 con 0x7fcbac0d82c0 2021-04-09T18:30:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbb10c8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbac10e780 con 0x7fcbac0d1da0 2021-04-09T18:30:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbb10c8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0d8d30 con 0x7fcbac10d010 2021-04-09T18:30:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbaad9d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac10d010 0x7fcbac0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbaa59c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcbac0d1da0 0x7fcbac0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbab59e700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcbac0d82c0 0x7fcbac10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:34658/0 (socket says 172.21.15.32:34658) 2021-04-09T18:30:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbab59e700 1 -- 172.21.15.32:0/4173144934 learned_addr learned my addr 172.21.15.32:0/4173144934 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbaa59c700 1 -- 172.21.15.32:0/4173144934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac10d010 msgr2=0x7fcbac0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbaa59c700 1 --2- 172.21.15.32:0/4173144934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac10d010 0x7fcbac0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbaa59c700 1 -- 172.21.15.32:0/4173144934 >> v1:172.21.15.86:6789/0 conn(0x7fcbac0d82c0 legacy=0x7fcbac10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbaa59c700 1 -- 172.21.15.32:0/4173144934 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0d9220 con 0x7fcbac0d1da0 2021-04-09T18:30:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcba9d9b700 1 -- 172.21.15.32:0/4173144934 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412274798 0 0) 0x7fcba4001440 con 0x7fcbac0d82c0 2021-04-09T18:30:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbaad9d700 1 --2- 172.21.15.32:0/4173144934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac10d010 0x7fcbac0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:06.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.598 7fcbaa59c700 1 --2- 172.21.15.32:0/4173144934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbac0d1da0 0x7fcbac0d7ca0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7fcba0006650 tx=0x7fcba0009a80).ready entity=mon.1 client_cookie=1bdec7578e67c408 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.599 7fcba9d9b700 1 -- 172.21.15.32:0/4173144934 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcba00009b0 con 0x7fcbac0d1da0 2021-04-09T18:30:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.599 7fcba9d9b700 1 -- 172.21.15.32:0/4173144934 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba000c0c0 con 0x7fcbac0d1da0 2021-04-09T18:30:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.599 7fcba9d9b700 1 -- 172.21.15.32:0/4173144934 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcba000c510 con 0x7fcbac0d1da0 2021-04-09T18:30:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.599 7fcbb10c8700 1 -- 172.21.15.32:0/4173144934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbac0d1da0 msgr2=0x7fcbac0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.599 7fcbb10c8700 1 --2- 172.21.15.32:0/4173144934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbac0d1da0 0x7fcbac0d7ca0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7fcba0006650 tx=0x7fcba0009a80).stop 2021-04-09T18:30:06.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.599 7fcbb10c8700 1 -- 172.21.15.32:0/4173144934 shutdown_connections 2021-04-09T18:30:06.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.599 7fcbb10c8700 1 --2- 172.21.15.32:0/4173144934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbac0d1da0 0x7fcbac0d7ca0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.599 7fcbb10c8700 1 --2- 172.21.15.32:0/4173144934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac10d010 0x7fcbac0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.599 7fcbb10c8700 1 -- 172.21.15.32:0/4173144934 >> 172.21.15.32:0/4173144934 conn(0x7fcbac0ce5f0 msgr2=0x7fcbac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.600 7fcbb10c8700 1 -- 172.21.15.32:0/4173144934 shutdown_connections 2021-04-09T18:30:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.600 7fcbb10c8700 1 -- 172.21.15.32:0/4173144934 wait complete. 2021-04-09T18:30:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbb10c8700 1 Processor -- start 2021-04-09T18:30:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbb10c8700 1 -- start start 2021-04-09T18:30:06.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbb10c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbac1aca20 0x7fcbac1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbb10c8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbac18bd40 0x7fcbac18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbb10c8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac08fba0 0x7fcbac08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbb10c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac18c7e0 con 0x7fcbac1aca20 2021-04-09T18:30:06.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbb10c8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac18ca30 con 0x7fcbac18bd40 2021-04-09T18:30:06.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbb10c8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0905f0 con 0x7fcbac08fba0 2021-04-09T18:30:06.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbaad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbac1aca20 0x7fcbac1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbaad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbac1aca20 0x7fcbac1ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55378/0 (socket says 172.21.15.32:55378) 2021-04-09T18:30:06.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.620 7fcbaad9d700 1 -- 172.21.15.32:0/1752906497 learned_addr learned my addr 172.21.15.32:0/1752906497 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:06.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbaa59c700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbac18bd40 0x7fcbac18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbab59e700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac08fba0 0x7fcbac08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbaad9d700 1 -- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac08fba0 msgr2=0x7fcbac08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbaad9d700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac08fba0 0x7fcbac08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbaad9d700 1 -- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbac18bd40 msgr2=0x7fcbac18c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:06.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbaad9d700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbac18bd40 0x7fcbac18c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:06.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbaad9d700 1 -- 172.21.15.32:0/1752906497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0b4300 con 0x7fcbac1aca20 2021-04-09T18:30:06.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbab59e700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac08fba0 0x7fcbac08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:06.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbaad9d700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbac1aca20 0x7fcbac1ace20 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7fcb9c006f40 tx=0x7fcb9c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:06.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcb9b7fe700 1 -- 172.21.15.32:0/1752906497 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb9c00bd80 con 0x7fcbac1aca20 2021-04-09T18:30:06.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcb9b7fe700 1 -- 172.21.15.32:0/1752906497 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9c000c90 con 0x7fcbac1aca20 2021-04-09T18:30:06.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac0b4300 con 0x7fcbac1aca20 2021-04-09T18:30:06.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcb9b7fe700 1 -- 172.21.15.32:0/1752906497 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb9c002220 con 0x7fcbac1aca20 2021-04-09T18:30:06.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.621 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac0b4300 con 0x7fcbac1aca20 2021-04-09T18:30:06.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.622 7fcb9b7fe700 1 -- 172.21.15.32:0/1752906497 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcb9c00a000 con 0x7fcbac1aca20 2021-04-09T18:30:06.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.622 7fcb9b7fe700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb900188d0 0x7fcb9001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:06.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.622 7fcb9b7fe700 1 -- 172.21.15.32:0/1752906497 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fcb9c032730 con 0x7fcbac1aca20 2021-04-09T18:30:06.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.622 7fcbaa59c700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb900188d0 0x7fcb9001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:06.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.622 7fcbaa59c700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb900188d0 0x7fcb9001add0 crc :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:06.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.642 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbac182d70 con 0x7fcbac1aca20 2021-04-09T18:30:06.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.645 7fcb9b7fe700 1 -- 172.21.15.32:0/1752906497 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcb9c006b00 con 0x7fcbac1aca20 2021-04-09T18:30:06.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.833 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestFailover.test_standby_count_wanted"]} v 0) v1 -- 0x7fcbac18c760 con 0x7fcbac1aca20 2021-04-09T18:30:07.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.965 7fcb9b7fe700 1 -- 172.21.15.32:0/1752906497 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestFailover.test_standby_count_wanted"]}]=0 v769) v1 ==== 144+0+0 (secure 0 0 0) 0x7fcb9c00a000 con 0x7fcbac1aca20 2021-04-09T18:30:07.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb900188d0 msgr2=0x7fcb9001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:07.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb900188d0 0x7fcb9001add0 crc :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbac1aca20 msgr2=0x7fcbac1ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:07.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbac1aca20 0x7fcbac1ace20 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7fcb9c006f40 tx=0x7fcb9c00b990).stop 2021-04-09T18:30:07.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 shutdown_connections 2021-04-09T18:30:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcb900188d0 0x7fcb9001add0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbac08fba0 0x7fcbac08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbac18bd40 0x7fcbac18c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 --2- 172.21.15.32:0/1752906497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbac1aca20 0x7fcbac1ace20 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 >> 172.21.15.32:0/1752906497 conn(0x7fcbac19b460 msgr2=0x7fcbac19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:07.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 shutdown_connections 2021-04-09T18:30:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:06.999 7fcbb10c8700 1 -- 172.21.15.32:0/1752906497 wait complete. 2021-04-09T18:30:07.097 INFO:tasks.cephfs_test_runner:test_standby_count_wanted (tasks.cephfs.test_failover.TestFailover) ... ok 2021-04-09T18:30:07.097 INFO:tasks.cephfs_test_runner:Starting test: test_clients (tasks.cephfs.test_failover.TestMultiFilesystems) 2021-04-09T18:30:07.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestMultiFilesystems.test_clients' 2021-04-09T18:30:07.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.158 7f5ec71fb700 1 Processor -- start 2021-04-09T18:30:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.158 7f5ec71fb700 1 -- start start 2021-04-09T18:30:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.159 7f5ec71fb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5ec00cffa0 0x7f5ec00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.159 7f5ec71fb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ec00d6c80 0x7f5ec010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:07.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.159 7f5ec71fb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec00d7280 con 0x7f5ec00d7800 2021-04-09T18:30:07.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.159 7f5ec71fb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec00d7080 con 0x7f5ec00cffa0 2021-04-09T18:30:07.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.159 7f5ec71fb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec00d91e0 con 0x7f5ec00d6c80 2021-04-09T18:30:07.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.159 7f5ec5017700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5ec00cffa0 0x7f5ec00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:07.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.159 7f5ec5017700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5ec00cffa0 0x7f5ec00d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55386/0 (socket says 172.21.15.32:55386) 2021-04-09T18:30:07.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.159 7f5ec5017700 1 -- 172.21.15.32:0/1136784702 learned_addr learned my addr 172.21.15.32:0/1136784702 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:07.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.159 7f5ec5818700 1 --2- 172.21.15.32:0/1136784702 >> v2:172.21.15.86:3300/0 conn(0x7f5ec00d6c80 0x7f5ec010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5ec5017700 1 -- 172.21.15.32:0/1136784702 >> v2:172.21.15.86:3300/0 conn(0x7f5ec00d6c80 msgr2=0x7f5ec010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5ec5017700 1 --2- 172.21.15.32:0/1136784702 >> v2:172.21.15.86:3300/0 conn(0x7f5ec00d6c80 0x7f5ec010cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5ec5017700 1 -- 172.21.15.32:0/1136784702 >> v1:172.21.15.86:6789/0 conn(0x7f5ec00d7800 legacy=0x7f5ec00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:07.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5ec5017700 1 -- 172.21.15.32:0/1136784702 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec00d83f0 con 0x7f5ec00cffa0 2021-04-09T18:30:07.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5ec5017700 1 --2- 172.21.15.32:0/1136784702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec00cffa0 0x7f5ec00d2440 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7f5ebc006700 tx=0x7f5ebc009b80).ready entity=mon.0 client_cookie=43db2fd1c6d55342 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:07.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5eb7fff700 1 -- 172.21.15.32:0/1136784702 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ebc00c0c0 con 0x7f5ec00cffa0 2021-04-09T18:30:07.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5eb7fff700 1 -- 172.21.15.32:0/1136784702 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ebc00c440 con 0x7f5ec00cffa0 2021-04-09T18:30:07.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5eb7fff700 1 -- 172.21.15.32:0/1136784702 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ebc00c920 con 0x7f5ec00cffa0 2021-04-09T18:30:07.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5ec71fb700 1 -- 172.21.15.32:0/1136784702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec00cffa0 msgr2=0x7f5ec00d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:07.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.160 7f5ec71fb700 1 --2- 172.21.15.32:0/1136784702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec00cffa0 0x7f5ec00d2440 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7f5ebc006700 tx=0x7f5ebc009b80).stop 2021-04-09T18:30:07.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.161 7f5ec71fb700 1 -- 172.21.15.32:0/1136784702 shutdown_connections 2021-04-09T18:30:07.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.161 7f5ec71fb700 1 --2- 172.21.15.32:0/1136784702 >> v2:172.21.15.86:3300/0 conn(0x7f5ec00d6c80 0x7f5ec010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.161 7f5ec71fb700 1 --2- 172.21.15.32:0/1136784702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec00cffa0 0x7f5ec00d2440 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.161 7f5ec71fb700 1 -- 172.21.15.32:0/1136784702 >> 172.21.15.32:0/1136784702 conn(0x7f5ec00ce5c0 msgr2=0x7f5ec00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:07.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.161 7f5ec71fb700 1 -- 172.21.15.32:0/1136784702 shutdown_connections 2021-04-09T18:30:07.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.161 7f5ec71fb700 1 -- 172.21.15.32:0/1136784702 wait complete. 2021-04-09T18:30:07.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.181 7f5ec71fb700 1 Processor -- start 2021-04-09T18:30:07.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.181 7f5ec71fb700 1 -- start start 2021-04-09T18:30:07.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec71fb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ec01acff0 0x7f5ec01847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:07.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec71fb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ec0183fc0 0x7f5ec01a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec71fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0186ff0 0x7f5ec01873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec71fb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0188c30 con 0x7f5ec0186ff0 2021-04-09T18:30:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec71fb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec01843c0 con 0x7f5ec0183fc0 2021-04-09T18:30:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec71fb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec01a5c20 con 0x7f5ec01acff0 2021-04-09T18:30:07.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5818700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0186ff0 0x7f5ec01873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:07.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5818700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0186ff0 0x7f5ec01873f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55396/0 (socket says 172.21.15.32:55396) 2021-04-09T18:30:07.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5818700 1 -- 172.21.15.32:0/950519914 learned_addr learned my addr 172.21.15.32:0/950519914 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:07.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5017700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ec01acff0 0x7f5ec01847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:07.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec4816700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ec0183fc0 0x7f5ec01a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:07.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5818700 1 -- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ec01acff0 msgr2=0x7f5ec01847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:07.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5818700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ec01acff0 0x7f5ec01847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5818700 1 -- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ec0183fc0 msgr2=0x7f5ec01a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:07.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5818700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ec0183fc0 0x7f5ec01a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:07.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5818700 1 -- 172.21.15.32:0/950519914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec0187a10 con 0x7f5ec0186ff0 2021-04-09T18:30:07.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec5818700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0186ff0 0x7f5ec01873f0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f5eb800b970 tx=0x7f5eb800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:07.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5eb5ffb700 1 -- 172.21.15.32:0/950519914 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5eb8000f90 con 0x7f5ec0186ff0 2021-04-09T18:30:07.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5eb5ffb700 1 -- 172.21.15.32:0/950519914 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb8006aa0 con 0x7f5ec0186ff0 2021-04-09T18:30:07.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec0187a10 con 0x7f5ec0186ff0 2021-04-09T18:30:07.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5eb5ffb700 1 -- 172.21.15.32:0/950519914 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5eb80015f0 con 0x7f5ec0186ff0 2021-04-09T18:30:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.182 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec0187a10 con 0x7f5ec0186ff0 2021-04-09T18:30:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.183 7f5eb5ffb700 1 -- 172.21.15.32:0/950519914 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5eb8000ed0 con 0x7f5ec0186ff0 2021-04-09T18:30:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.183 7f5eb5ffb700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ea00188d0 0x7f5ea001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.183 7f5eb5ffb700 1 -- 172.21.15.32:0/950519914 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f5eb8032ac0 con 0x7f5ec0186ff0 2021-04-09T18:30:07.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.184 7f5ec5017700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ea00188d0 0x7f5ea001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:07.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.184 7f5ec5017700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ea00188d0 0x7f5ea001add0 crc :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:07.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.203 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec0182dc0 con 0x7f5ec0186ff0 2021-04-09T18:30:07.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.207 7f5eb5ffb700 1 -- 172.21.15.32:0/950519914 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5eb80066f0 con 0x7f5ec0186ff0 2021-04-09T18:30:07.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.394 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestMultiFilesystems.test_clients"]} v 0) v1 -- 0x7f5ec00d7280 con 0x7f5ec0186ff0 2021-04-09T18:30:07.920 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:30:07.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:30:08.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:07.969 7f5eb5ffb700 1 -- 172.21.15.32:0/950519914 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestMultiFilesystems.test_clients"]}]=0 v770) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5eb8000ed0 con 0x7f5ec0186ff0 2021-04-09T18:30:08.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ea00188d0 msgr2=0x7f5ea001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ea00188d0 0x7f5ea001add0 crc :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0186ff0 msgr2=0x7f5ec01873f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0186ff0 0x7f5ec01873f0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f5eb800b970 tx=0x7f5eb800bd10).stop 2021-04-09T18:30:08.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 shutdown_connections 2021-04-09T18:30:08.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5ea00188d0 0x7f5ea001add0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0186ff0 0x7f5ec01873f0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ec0183fc0 0x7f5ec01a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 --2- 172.21.15.32:0/950519914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ec01acff0 0x7f5ec01847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 >> 172.21.15.32:0/950519914 conn(0x7f5ec009abb0 msgr2=0x7f5ec00a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 shutdown_connections 2021-04-09T18:30:08.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.012 7f5ec71fb700 1 -- 172.21.15.32:0/950519914 wait complete. 2021-04-09T18:30:08.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:30:08.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.176 7f0db1be4700 1 Processor -- start 2021-04-09T18:30:08.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.176 7f0db1be4700 1 -- start start 2021-04-09T18:30:08.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.176 7f0db1be4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac10d010 0x7f0dac0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.176 7f0db1be4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0dac0d1da0 0x7f0dac0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0db1be4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dac10d480 con 0x7f0dac0d82c0 2021-04-09T18:30:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0db1be4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dac10e780 con 0x7f0dac0d1da0 2021-04-09T18:30:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0db1be4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0d8d30 con 0x7f0dac10d010 2021-04-09T18:30:08.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daaffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0dac0d1da0 0x7f0dac0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0dabfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0dac0d82c0 0x7f0dac10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36762/0 (socket says 172.21.15.32:36762) 2021-04-09T18:30:08.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0dabfff700 1 -- 172.21.15.32:0/1477753606 learned_addr learned my addr 172.21.15.32:0/1477753606 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:08.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0dab7fe700 1 --2- 172.21.15.32:0/1477753606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac10d010 0x7f0dac0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daaffd700 1 -- 172.21.15.32:0/1477753606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac10d010 msgr2=0x7f0dac0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:08.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daaffd700 1 --2- 172.21.15.32:0/1477753606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac10d010 0x7f0dac0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daaffd700 1 -- 172.21.15.32:0/1477753606 >> v1:172.21.15.32:6789/0 conn(0x7f0dac0d82c0 legacy=0x7f0dac10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daaffd700 1 -- 172.21.15.32:0/1477753606 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac0d9220 con 0x7f0dac0d1da0 2021-04-09T18:30:08.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0dab7fe700 1 --2- 172.21.15.32:0/1477753606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac10d010 0x7f0dac0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:08.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daa7fc700 1 -- 172.21.15.32:0/1477753606 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991841340 0 0) 0x7f0da40009b0 con 0x7f0dac0d82c0 2021-04-09T18:30:08.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daaffd700 1 --2- 172.21.15.32:0/1477753606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0d1da0 0x7f0dac0d7ca0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7f0da00069d0 tx=0x7f0da0009e40).ready entity=mon.0 client_cookie=79df262b6b0b1054 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:08.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daa7fc700 1 -- 172.21.15.32:0/1477753606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0da000b510 con 0x7f0dac0d1da0 2021-04-09T18:30:08.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daa7fc700 1 -- 172.21.15.32:0/1477753606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da000b890 con 0x7f0dac0d1da0 2021-04-09T18:30:08.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0daa7fc700 1 -- 172.21.15.32:0/1477753606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0da000bd70 con 0x7f0dac0d1da0 2021-04-09T18:30:08.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0db1be4700 1 -- 172.21.15.32:0/1477753606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0d1da0 msgr2=0x7f0dac0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.177 7f0db1be4700 1 --2- 172.21.15.32:0/1477753606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0d1da0 0x7f0dac0d7ca0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7f0da00069d0 tx=0x7f0da0009e40).stop 2021-04-09T18:30:08.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.178 7f0db1be4700 1 -- 172.21.15.32:0/1477753606 shutdown_connections 2021-04-09T18:30:08.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.178 7f0db1be4700 1 --2- 172.21.15.32:0/1477753606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0d1da0 0x7f0dac0d7ca0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.178 7f0db1be4700 1 --2- 172.21.15.32:0/1477753606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac10d010 0x7f0dac0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.178 7f0db1be4700 1 -- 172.21.15.32:0/1477753606 >> 172.21.15.32:0/1477753606 conn(0x7f0dac0ce5f0 msgr2=0x7f0dac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:08.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.178 7f0db1be4700 1 -- 172.21.15.32:0/1477753606 shutdown_connections 2021-04-09T18:30:08.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.178 7f0db1be4700 1 -- 172.21.15.32:0/1477753606 wait complete. 2021-04-09T18:30:08.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.198 7f0db1be4700 1 Processor -- start 2021-04-09T18:30:08.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.198 7f0db1be4700 1 -- start start 2021-04-09T18:30:08.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0db1be4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dac0a8b00 0x7f0dac18c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0db1be4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac0a8f00 0x7f0dac18cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0db1be4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0c2dd0 0x7f0dac0c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0db1be4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac18c300 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0db1be4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0c37f0 con 0x7f0dac0a8b00 2021-04-09T18:30:08.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0db1be4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0c39d0 con 0x7f0dac0a8f00 2021-04-09T18:30:08.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0dabfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0c2dd0 0x7f0dac0c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0dabfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0c2dd0 0x7f0dac0c31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55408/0 (socket says 172.21.15.32:55408) 2021-04-09T18:30:08.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0dabfff700 1 -- 172.21.15.32:0/65388668 learned_addr learned my addr 172.21.15.32:0/65388668 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:08.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0dab7fe700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dac0a8b00 0x7f0dac18c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0daaffd700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac0a8f00 0x7f0dac18cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0dabfff700 1 -- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac0a8f00 msgr2=0x7f0dac18cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0dabfff700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac0a8f00 0x7f0dac18cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0dabfff700 1 -- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dac0a8b00 msgr2=0x7f0dac18c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0dabfff700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dac0a8b00 0x7f0dac18c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.199 7f0dabfff700 1 -- 172.21.15.32:0/65388668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac0c3bf0 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.200 7f0dabfff700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0c2dd0 0x7f0dac0c31d0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7f0da400a590 tx=0x7f0da400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:08.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.200 7f0d93fff700 1 -- 172.21.15.32:0/65388668 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0da400ad20 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.200 7f0d93fff700 1 -- 172.21.15.32:0/65388668 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da40043a0 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.200 7f0d93fff700 1 -- 172.21.15.32:0/65388668 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0da4004880 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.200 7f0db1be4700 1 -- 172.21.15.32:0/65388668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dac0c3bf0 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.200 7f0db1be4700 1 -- 172.21.15.32:0/65388668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dac0c3bf0 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.200 7f0d93fff700 1 -- 172.21.15.32:0/65388668 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0da4004580 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.201 7f0d93fff700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d8c0188d0 0x7f0d8c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.201 7f0d93fff700 1 -- 172.21.15.32:0/65388668 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f0da4032720 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.201 7f0dab7fe700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d8c0188d0 0x7f0d8c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.201 7f0dab7fe700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d8c0188d0 0x7f0d8c01add0 crc :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:08.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.221 7f0db1be4700 1 -- 172.21.15.32:0/65388668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dac10d480 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.224 7f0d93fff700 1 -- 172.21.15.32:0/65388668 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0da40045b0 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.438 7f0db1be4700 1 -- 172.21.15.32:0/65388668 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0dac10d480 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.442 7f0d93fff700 1 -- 172.21.15.32:0/65388668 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0da40043a0 con 0x7f0dac0c2dd0 2021-04-09T18:30:08.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 -- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d8c0188d0 msgr2=0x7f0d8c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d8c0188d0 0x7f0d8c01add0 crc :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 -- 172.21.15.32:0/65388668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0c2dd0 msgr2=0x7f0dac0c31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0c2dd0 0x7f0dac0c31d0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7f0da400a590 tx=0x7f0da400a930).stop 2021-04-09T18:30:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 -- 172.21.15.32:0/65388668 shutdown_connections 2021-04-09T18:30:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0d8c0188d0 0x7f0d8c01add0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dac0c2dd0 0x7f0dac0c31d0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dac0a8f00 0x7f0dac18cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 --2- 172.21.15.32:0/65388668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dac0a8b00 0x7f0dac18c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 -- 172.21.15.32:0/65388668 >> 172.21.15.32:0/65388668 conn(0x7f0dac0b93a0 msgr2=0x7f0dac0b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:08.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 -- 172.21.15.32:0/65388668 shutdown_connections 2021-04-09T18:30:08.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.448 7f0db1be4700 1 -- 172.21.15.32:0/65388668 wait complete. 2021-04-09T18:30:08.545 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:30:08.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:30:08.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:30:08.661 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:30:08.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:08.662 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:08.663 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:30:08.709 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:30:08.709 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:30:08.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:08.710 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:08.711 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:08.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:30:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:30:08.754 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:30:08.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:30:08.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.817 7f1945c60700 1 Processor -- start 2021-04-09T18:30:08.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.817 7f1945c60700 1 -- start start 2021-04-09T18:30:08.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.817 7f1945c60700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19400d7ca0 0x7f19400d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f1945c60700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194010da00 0x7f194010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f1945c60700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194010d480 con 0x7f194010d010 2021-04-09T18:30:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f1945c60700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19400d9220 con 0x7f19400d7ca0 2021-04-09T18:30:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f1945c60700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19400d86c0 con 0x7f194010da00 2021-04-09T18:30:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193effd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f19400d7ca0 0x7f19400d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193f7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f194010d010 0x7f19400d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36774/0 (socket says 172.21.15.32:36774) 2021-04-09T18:30:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193f7fe700 1 -- 172.21.15.32:0/3624291337 learned_addr learned my addr 172.21.15.32:0/3624291337 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193ffff700 1 --2- 172.21.15.32:0/3624291337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194010da00 0x7f194010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193effd700 1 -- 172.21.15.32:0/3624291337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194010da00 msgr2=0x7f194010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193effd700 1 --2- 172.21.15.32:0/3624291337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194010da00 0x7f194010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193effd700 1 -- 172.21.15.32:0/3624291337 >> v1:172.21.15.32:6789/0 conn(0x7f194010d010 legacy=0x7f19400d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193effd700 1 -- 172.21.15.32:0/3624291337 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400d1f70 con 0x7f19400d7ca0 2021-04-09T18:30:08.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193e7fc700 1 -- 172.21.15.32:0/3624291337 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262214890 0 0) 0x7f1930000ce0 con 0x7f194010d010 2021-04-09T18:30:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193ffff700 1 --2- 172.21.15.32:0/3624291337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194010da00 0x7f194010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:30:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193effd700 1 --2- 172.21.15.32:0/3624291337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400d7ca0 0x7f19400d80a0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7f192c006650 tx=0x7f192c009a80).ready entity=mon.0 client_cookie=f13238eaeba87e79 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193e7fc700 1 -- 172.21.15.32:0/3624291337 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f192c0009b0 con 0x7f19400d7ca0 2021-04-09T18:30:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193e7fc700 1 -- 172.21.15.32:0/3624291337 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c00c0c0 con 0x7f19400d7ca0 2021-04-09T18:30:08.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f193e7fc700 1 -- 172.21.15.32:0/3624291337 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f192c00c510 con 0x7f19400d7ca0 2021-04-09T18:30:08.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f1945c60700 1 -- 172.21.15.32:0/3624291337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400d7ca0 msgr2=0x7f19400d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.818 7f1945c60700 1 --2- 172.21.15.32:0/3624291337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400d7ca0 0x7f19400d80a0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7f192c006650 tx=0x7f192c009a80).stop 2021-04-09T18:30:08.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.819 7f1945c60700 1 -- 172.21.15.32:0/3624291337 shutdown_connections 2021-04-09T18:30:08.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.819 7f1945c60700 1 --2- 172.21.15.32:0/3624291337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f194010da00 0x7f194010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.819 7f1945c60700 1 --2- 172.21.15.32:0/3624291337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19400d7ca0 0x7f19400d80a0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.819 7f1945c60700 1 -- 172.21.15.32:0/3624291337 >> 172.21.15.32:0/3624291337 conn(0x7f19400ce5f0 msgr2=0x7f19400d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:08.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.819 7f1945c60700 1 -- 172.21.15.32:0/3624291337 shutdown_connections 2021-04-09T18:30:08.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.819 7f1945c60700 1 -- 172.21.15.32:0/3624291337 wait complete. 2021-04-09T18:30:08.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.839 7f1945c60700 1 Processor -- start 2021-04-09T18:30:08.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.839 7f1945c60700 1 -- start start 2021-04-09T18:30:08.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f1945c60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1940184a30 0x7f19401acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f1945c60700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401a6b90 0x7f19401a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f1945c60700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19401a5150 0x7f19401a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f1945c60700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19401a75b0 con 0x7f1940184a30 2021-04-09T18:30:08.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f1945c60700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1940184590 con 0x7f19401a6b90 2021-04-09T18:30:08.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f1945c60700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19401a5b70 con 0x7f19401a5150 2021-04-09T18:30:08.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f193f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1940184a30 0x7f19401acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f193f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1940184a30 0x7f19401acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55422/0 (socket says 172.21.15.32:55422) 2021-04-09T18:30:08.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f193f7fe700 1 -- 172.21.15.32:0/2415516950 learned_addr learned my addr 172.21.15.32:0/2415516950 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:08.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f193ffff700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19401a5150 0x7f19401a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.840 7f193effd700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401a6b90 0x7f19401a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f193f7fe700 1 -- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19401a5150 msgr2=0x7f19401a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f193f7fe700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19401a5150 0x7f19401a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f193f7fe700 1 -- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401a6b90 msgr2=0x7f19401a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:08.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f193f7fe700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401a6b90 0x7f19401a6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:08.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f193f7fe700 1 -- 172.21.15.32:0/2415516950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19401a5d90 con 0x7f1940184a30 2021-04-09T18:30:08.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f193f7fe700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1940184a30 0x7f19401acb80 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f1930006f00 tx=0x7f193000dbc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:08.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f1937fff700 1 -- 172.21.15.32:0/2415516950 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1930006020 con 0x7f1940184a30 2021-04-09T18:30:08.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f1937fff700 1 -- 172.21.15.32:0/2415516950 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1930006360 con 0x7f1940184a30 2021-04-09T18:30:08.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f1945c60700 1 -- 172.21.15.32:0/2415516950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19401a5d90 con 0x7f1940184a30 2021-04-09T18:30:08.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f1937fff700 1 -- 172.21.15.32:0/2415516950 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1930004620 con 0x7f1940184a30 2021-04-09T18:30:08.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.841 7f1945c60700 1 -- 172.21.15.32:0/2415516950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19401a5d90 con 0x7f1940184a30 2021-04-09T18:30:08.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.842 7f1937fff700 1 -- 172.21.15.32:0/2415516950 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f193000c030 con 0x7f1940184a30 2021-04-09T18:30:08.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.842 7f1937fff700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f191c018930 0x7f191c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:08.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.842 7f1937fff700 1 -- 172.21.15.32:0/2415516950 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f1930034900 con 0x7f1940184a30 2021-04-09T18:30:08.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.842 7f193effd700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f191c018930 0x7f191c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:08.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.842 7f193effd700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f191c018930 0x7f191c01ae10 crc :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:08.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.862 7f1945c60700 1 -- 172.21.15.32:0/2415516950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19401b7080 con 0x7f1940184a30 2021-04-09T18:30:08.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:08.865 7f1937fff700 1 -- 172.21.15.32:0/2415516950 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1930006b90 con 0x7f1940184a30 2021-04-09T18:30:09.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.053 7f1945c60700 1 -- 172.21.15.32:0/2415516950 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f19401b7080 con 0x7f1940184a30 2021-04-09T18:30:09.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.054 7f1937fff700 1 -- 172.21.15.32:0/2415516950 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v171) v1 ==== 81+0+25550 (secure 0 0 0) 0x7f193000c030 con 0x7f1940184a30 2021-04-09T18:30:09.142 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:09.142 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:30:09.143 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 171, 2021-04-09T18:30:09.143 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:30:09.143 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:30:09.143 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:30:05.956985", 2021-04-09T18:30:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:30:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:30:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:30:09.144 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:30:09.145 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:30:09.145 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:30:09.145 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:30:09.145 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:30:09.145 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:30:09.146 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:30:09.146 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:30:09.146 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:30:09.146 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:30:09.146 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:30:09.147 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:30:09.147 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 29, 2021-04-09T18:30:09.147 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:30:09.147 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:30:09.148 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:30:09.148 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:30:09.148 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:30:09.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.148 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:30:09.149 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:30:09.149 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:30:09.149 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:30:09.149 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:30:09.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:30:09.150 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:30:09.150 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:30:09.150 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:30:09.150 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:30:09.151 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:30:09.151 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:30:09.151 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:30:09.151 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:30:09.151 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:30:09.152 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:30:09.152 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:30:09.152 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:30:09.152 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:30:09.152 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:30:09.153 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:30:09.153 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:30:09.153 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:30:09.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.153 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:30:09.154 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:30:09.154 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:30:09.154 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:30:09.154 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:30:09.155 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:30:09.155 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:30:09.155 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:30:09.155 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:30:09.155 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:30:09.156 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:30:09.156 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:30:09.156 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:30:09.157 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:30:09.157 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:30:09.157 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:30:09.157 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:30:09.157 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:30:09.158 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:30:09.158 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:30:09.159 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:30:09.159 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:30:09.159 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:30:09.159 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:30:09.160 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:30:09.160 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:30:09.160 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:30:09.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.161 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:30:09.161 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:30:09.161 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:30:09.161 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:30:09.162 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:30:09.162 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:30:09.162 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:30:09.162 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:30:09.162 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:30:09.163 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:30:09.163 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:30:09.163 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:30:09.163 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:30:09.164 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:30:09.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.164 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:30:09.164 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:30:09.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.165 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:30:09.165 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:30:09.165 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:09.165 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:09.166 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:09.166 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:09.166 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:09.166 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:09.167 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:09.167 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 161, 2021-04-09T18:30:09.167 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:09.167 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:09.168 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:09.168 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.168 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.168 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:30:09.169 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:09.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.169 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.170 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:30:09.170 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:09.170 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.170 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.171 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:09.171 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.171 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.172 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:30:09.172 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:09.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.172 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.173 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:30:09.173 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:09.173 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.173 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.174 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:09.174 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.174 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:30:09.175 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:09.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.175 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.176 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:30:09.176 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:09.176 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.176 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.177 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:09.177 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.177 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.178 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:30:09.178 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:09.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.178 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.179 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:30:09.179 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:30:09.179 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.179 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.180 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:30:09.180 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:30:09.180 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:30:09.180 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:30:09.181 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:09.181 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:09.181 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:09.181 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.182 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:30:09.182 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:30:09.182 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:09.183 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:09.183 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:09.183 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:09.183 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:09.183 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:09.184 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:09.184 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 166, 2021-04-09T18:30:09.184 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:09.184 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:09.185 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:09.185 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.185 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.186 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:30:09.186 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:09.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.186 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.187 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:30:09.187 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:09.187 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.187 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.188 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:09.188 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.188 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.188 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.189 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:30:09.189 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:09.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.190 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:30:09.190 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:09.190 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.190 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.191 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:09.191 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.191 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.192 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:30:09.192 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:09.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.192 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.193 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:30:09.193 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:09.193 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.193 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.194 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:09.194 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.195 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:30:09.195 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:09.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.195 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.196 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:30:09.196 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:30:09.196 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.196 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.197 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:30:09.197 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:30:09.197 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:30:09.198 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:30:09.198 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:09.198 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:09.198 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:09.198 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.199 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:30:09.199 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:30:09.200 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:09.200 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:09.200 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:09.200 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:09.200 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:09.201 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:09.201 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:09.201 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 153, 2021-04-09T18:30:09.201 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:09.201 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:09.202 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:09.202 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.202 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.202 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:30:09.203 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:09.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.203 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.204 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:30:09.204 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:09.204 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.204 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.205 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:09.205 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.205 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:30:09.206 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:09.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.206 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.206 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:30:09.207 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:09.207 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.207 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.208 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:09.208 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.208 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:30:09.209 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:09.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.209 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:30:09.210 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:09.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.210 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.211 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:09.211 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.211 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.212 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:30:09.212 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:09.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.212 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.213 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:30:09.213 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:30:09.213 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.213 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.214 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:30:09.214 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:30:09.214 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:30:09.215 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:30:09.215 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:09.215 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:09.215 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:09.215 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.216 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.216 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:30:09.216 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:30:09.216 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:09.217 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:09.217 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:09.217 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:09.217 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:09.217 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:09.218 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:09.218 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 140, 2021-04-09T18:30:09.218 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:09.218 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:09.218 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:09.219 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.219 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:30:09.220 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:09.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.220 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.221 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:30:09.221 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:09.221 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.221 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.222 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:09.222 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.222 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.222 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.223 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:30:09.223 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:09.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.223 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.223 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.224 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:30:09.224 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:09.224 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.224 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.225 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:09.225 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.225 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.225 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.225 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:30:09.226 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:09.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.226 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.226 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.226 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:30:09.227 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:09.227 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.227 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.228 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:09.228 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.228 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.228 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:30:09.229 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:09.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.229 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.229 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.230 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:30:09.230 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:30:09.230 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.230 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.231 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:30:09.231 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:30:09.231 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:30:09.231 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:30:09.231 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:09.232 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:09.232 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:09.232 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.232 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.233 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:30:09.233 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:30:09.233 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:09.233 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:09.234 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 166, 2021-04-09T18:30:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:09.235 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.236 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:30:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.237 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:30:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.238 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:30:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:30:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.241 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.242 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:30:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:30:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.244 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:30:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.246 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:30:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:30:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.247 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:30:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:30:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:30:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:30:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:09.249 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:30:09.250 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:30:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:09.251 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 161, 2021-04-09T18:30:09.252 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:30:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.254 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:30:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.256 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:30:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.257 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:30:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.258 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.259 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:30:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:30:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.262 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:30:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:30:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:30:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:30:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:30:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:30:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:30:09.265 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.266 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:30:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:30:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:09.267 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:09.268 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 166, 2021-04-09T18:30:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:09.269 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:30:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:09.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:30:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:09.271 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.272 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:30:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.273 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:30:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.274 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.275 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:30:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.276 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.277 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.277 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:30:09.277 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:09.277 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.277 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:30:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.279 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:30:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:30:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.280 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:30:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:30:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:30:09.281 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:30:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:09.282 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:30:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:30:09.283 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:30:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:30:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:30:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:30:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:30:09.284 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:30:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:30:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 166, 2021-04-09T18:30:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:30:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:30:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:30:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.286 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:30:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.287 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:30:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.288 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:30:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:30:09.289 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:30:09.290 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.291 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:30:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:30:09.292 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:30:09.293 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.294 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:30:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:30:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:30:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:30:09.295 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:09.296 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:30:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:30:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:30:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:30:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:30:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:30:09.298 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:30:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:30:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:30:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:09.299 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:30:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:30:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.300 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:30:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:09.301 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:30:09.302 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:09.303 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:30:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:09.304 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:09.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:30:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:09.306 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:09.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:30:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:09.308 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:30:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:09.310 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:30:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:09.312 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:09.313 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:30:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:30:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:30:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:30:09.314 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:30:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:30:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:30:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:30:09.315 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:30:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:30:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:30:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:30:09.316 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/2200487686": "2021-04-10 18:30:02.426463", 2021-04-09T18:30:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/2200487686": "2021-04-10 18:30:02.426463" 2021-04-09T18:30:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:30:09.317 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:30:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:30:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:30:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:30:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:30:09.318 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:30:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:30:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:09.319 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:30:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:30:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:30:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:30:09.320 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:30:09.320 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:30:09.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.056 7f1945c60700 1 -- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f191c018930 msgr2=0x7f191c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.056 7f1945c60700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f191c018930 0x7f191c01ae10 crc :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.056 7f1945c60700 1 -- 172.21.15.32:0/2415516950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1940184a30 msgr2=0x7f19401acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.056 7f1945c60700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1940184a30 0x7f19401acb80 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f1930006f00 tx=0x7f193000dbc0).stop 2021-04-09T18:30:09.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.057 7f1945c60700 1 -- 172.21.15.32:0/2415516950 shutdown_connections 2021-04-09T18:30:09.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.057 7f1945c60700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f191c018930 0x7f191c01ae10 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.057 7f1945c60700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19401a5150 0x7f19401a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.057 7f1945c60700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19401a6b90 0x7f19401a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.057 7f1945c60700 1 --2- 172.21.15.32:0/2415516950 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1940184a30 0x7f19401acb80 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.057 7f1945c60700 1 -- 172.21.15.32:0/2415516950 >> 172.21.15.32:0/2415516950 conn(0x7f1940191560 msgr2=0x7f19401b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:09.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.057 7f1945c60700 1 -- 172.21.15.32:0/2415516950 shutdown_connections 2021-04-09T18:30:09.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.057 7f1945c60700 1 -- 172.21.15.32:0/2415516950 wait complete. 2021-04-09T18:30:09.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:09.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.388 7f3b00b7f700 1 Processor -- start 2021-04-09T18:30:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.388 7f3b00b7f700 1 -- start start 2021-04-09T18:30:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3b00b7f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3afc0d7ca0 0x7f3afc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3b00b7f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3afc10d480 con 0x7f3afc10da00 2021-04-09T18:30:09.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3b00b7f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3afc0d86c0 con 0x7f3afc10d010 2021-04-09T18:30:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3b00b7f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3afc0d9220 con 0x7f3afc0d7ca0 2021-04-09T18:30:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3afb05f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3afc10da00 0x7f3afc10de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36790/0 (socket says 172.21.15.32:36790) 2021-04-09T18:30:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3afb05f700 1 -- 172.21.15.32:0/3047636891 learned_addr learned my addr 172.21.15.32:0/3047636891 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:09.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3afa05d700 1 --2- 172.21.15.32:0/3047636891 >> v2:172.21.15.86:3300/0 conn(0x7f3afc0d7ca0 0x7f3afc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3af985c700 1 -- 172.21.15.32:0/3047636891 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530514725 0 0) 0x7f3af4000d30 con 0x7f3afc10da00 2021-04-09T18:30:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3af985c700 1 -- 172.21.15.32:0/3047636891 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae4002fe0 con 0x7f3afc10da00 2021-04-09T18:30:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3af985c700 1 -- 172.21.15.32:0/3047636891 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103929193 0 0) 0x7f3aec000c60 con 0x7f3afc10d010 2021-04-09T18:30:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3af985c700 1 -- 172.21.15.32:0/3047636891 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae40042c0 con 0x7f3afc10d010 2021-04-09T18:30:09.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3afa05d700 1 -- 172.21.15.32:0/3047636891 >> v1:172.21.15.86:6789/0 conn(0x7f3afc10d010 legacy=0x7f3afc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3afa05d700 1 -- 172.21.15.32:0/3047636891 >> v1:172.21.15.32:6789/0 conn(0x7f3afc10da00 legacy=0x7f3afc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.389 7f3afa05d700 1 -- 172.21.15.32:0/3047636891 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc112100 con 0x7f3afc0d7ca0 2021-04-09T18:30:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.390 7f3afa05d700 1 --2- 172.21.15.32:0/3047636891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc0d7ca0 0x7f3afc0d80a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f3af0006650 tx=0x7f3af0009a80).ready entity=mon.1 client_cookie=fb3f0a0fe262ec7a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:09.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.390 7f3af985c700 1 -- 172.21.15.32:0/3047636891 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3af00009b0 con 0x7f3afc0d7ca0 2021-04-09T18:30:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.390 7f3af985c700 1 -- 172.21.15.32:0/3047636891 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af000c0c0 con 0x7f3afc0d7ca0 2021-04-09T18:30:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.390 7f3af985c700 1 -- 172.21.15.32:0/3047636891 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3af000c510 con 0x7f3afc0d7ca0 2021-04-09T18:30:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.390 7f3b00b7f700 1 -- 172.21.15.32:0/3047636891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc0d7ca0 msgr2=0x7f3afc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.390 7f3b00b7f700 1 --2- 172.21.15.32:0/3047636891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc0d7ca0 0x7f3afc0d80a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f3af0006650 tx=0x7f3af0009a80).stop 2021-04-09T18:30:09.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.391 7f3b00b7f700 1 -- 172.21.15.32:0/3047636891 shutdown_connections 2021-04-09T18:30:09.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.391 7f3b00b7f700 1 --2- 172.21.15.32:0/3047636891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc0d7ca0 0x7f3afc0d80a0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.391 7f3b00b7f700 1 -- 172.21.15.32:0/3047636891 >> 172.21.15.32:0/3047636891 conn(0x7f3afc0ce5f0 msgr2=0x7f3afc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:09.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.391 7f3b00b7f700 1 -- 172.21.15.32:0/3047636891 shutdown_connections 2021-04-09T18:30:09.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.391 7f3b00b7f700 1 -- 172.21.15.32:0/3047636891 wait complete. 2021-04-09T18:30:09.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3b00b7f700 1 Processor -- start 2021-04-09T18:30:09.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3b00b7f700 1 -- start start 2021-04-09T18:30:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3b00b7f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc179610 0x7f3afc179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3b00b7f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3afc17a6e0 0x7f3afc17aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3b00b7f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc17b100 0x7f3afc08db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3b00b7f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc17d720 con 0x7f3afc17a6e0 2021-04-09T18:30:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3b00b7f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc185ba0 con 0x7f3afc179610 2021-04-09T18:30:09.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3b00b7f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc08e1b0 con 0x7f3afc17b100 2021-04-09T18:30:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3afa05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3afc17a6e0 0x7f3afc17aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.411 7f3afa05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3afc17a6e0 0x7f3afc17aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55436/0 (socket says 172.21.15.32:55436) 2021-04-09T18:30:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3afa05d700 1 -- 172.21.15.32:0/1959969206 learned_addr learned my addr 172.21.15.32:0/1959969206 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:09.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3afa85e700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc179610 0x7f3afc179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3afb05f700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc17b100 0x7f3afc08db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3afa05d700 1 -- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc17b100 msgr2=0x7f3afc08db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3afa05d700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc17b100 0x7f3afc08db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3afa05d700 1 -- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc179610 msgr2=0x7f3afc179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3afa05d700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc179610 0x7f3afc179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3afa05d700 1 -- 172.21.15.32:0/1959969206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc08e360 con 0x7f3afc17a6e0 2021-04-09T18:30:09.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3afa05d700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3afc17a6e0 0x7f3afc17aae0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7f3af0009650 tx=0x7f3af000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:09.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3aeb7fe700 1 -- 172.21.15.32:0/1959969206 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3af00093a0 con 0x7f3afc17a6e0 2021-04-09T18:30:09.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3aeb7fe700 1 -- 172.21.15.32:0/1959969206 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af000bd50 con 0x7f3afc17a6e0 2021-04-09T18:30:09.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3aeb7fe700 1 -- 172.21.15.32:0/1959969206 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3af00052e0 con 0x7f3afc17a6e0 2021-04-09T18:30:09.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3afc08e1b0 con 0x7f3afc17a6e0 2021-04-09T18:30:09.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.412 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3afc08e8f0 con 0x7f3afc17a6e0 2021-04-09T18:30:09.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.413 7f3aeb7fe700 1 -- 172.21.15.32:0/1959969206 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3af0006030 con 0x7f3afc17a6e0 2021-04-09T18:30:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.413 7f3aeb7fe700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ae0018930 0x7f3ae001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.413 7f3aeb7fe700 1 -- 172.21.15.32:0/1959969206 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f3af0032700 con 0x7f3afc17a6e0 2021-04-09T18:30:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.413 7f3afa85e700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ae0018930 0x7f3ae001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.413 7f3afa85e700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ae0018930 0x7f3ae001adc0 crc :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:09.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.433 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3afc17a1b0 con 0x7f3afc17a6e0 2021-04-09T18:30:09.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.436 7f3aeb7fe700 1 -- 172.21.15.32:0/1959969206 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3af00009c0 con 0x7f3afc17a6e0 2021-04-09T18:30:09.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.629 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3afc17a1b0 con 0x7f3afc17a6e0 2021-04-09T18:30:09.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.629 7f3aeb7fe700 1 -- 172.21.15.32:0/1959969206 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 319 v319) v1 ==== 95+0+1857 (secure 0 0 0) 0x7f3af000bd50 con 0x7f3afc17a6e0 2021-04-09T18:30:09.718 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 319 2021-04-09T18:30:09.718 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:09.718 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":319,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3983419158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3983419158},{"type":"v1","addr":"172.21.15.32:6834","nonce":3983419158}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":314},{"gid":10821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1847221608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1847221608},{"type":"v1","addr":"172.21.15.32:6835","nonce":1847221608}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":319},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[]} 2021-04-09T18:30:09.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ae0018930 msgr2=0x7f3ae001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ae0018930 0x7f3ae001adc0 crc :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3afc17a6e0 msgr2=0x7f3afc17aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3afc17a6e0 0x7f3afc17aae0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7f3af0009650 tx=0x7f3af000b960).stop 2021-04-09T18:30:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 shutdown_connections 2021-04-09T18:30:09.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ae0018930 0x7f3ae001adc0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3afc17a6e0 0x7f3afc17aae0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc17b100 0x7f3afc08db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 --2- 172.21.15.32:0/1959969206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc179610 0x7f3afc179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 >> 172.21.15.32:0/1959969206 conn(0x7f3afc190550 msgr2=0x7f3afc1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:09.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 shutdown_connections 2021-04-09T18:30:09.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.632 7f3b00b7f700 1 -- 172.21.15.32:0/1959969206 wait complete. 2021-04-09T18:30:09.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:09.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.790 7f28b7754700 1 Processor -- start 2021-04-09T18:30:09.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.790 7f28b7754700 1 -- start start 2021-04-09T18:30:09.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b7754700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28b010d010 0x7f28b00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b7754700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f28b00d1da0 0x7f28b00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b7754700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b00d82c0 0x7f28b010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b7754700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b00d8d30 con 0x7f28b010d010 2021-04-09T18:30:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b7754700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b00d9220 con 0x7f28b00d1da0 2021-04-09T18:30:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b7754700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b010d4b0 con 0x7f28b00d82c0 2021-04-09T18:30:09.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5570700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28b010d010 0x7f28b00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5570700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28b010d010 0x7f28b00d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55444/0 (socket says 172.21.15.32:55444) 2021-04-09T18:30:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5570700 1 -- 172.21.15.32:0/3334488283 learned_addr learned my addr 172.21.15.32:0/3334488283 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b4d6f700 1 --2- 172.21.15.32:0/3334488283 >> v2:172.21.15.86:3300/0 conn(0x7f28b00d1da0 0x7f28b00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5d71700 1 --2- 172.21.15.32:0/3334488283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b00d82c0 0x7f28b010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5570700 1 -- 172.21.15.32:0/3334488283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b00d82c0 msgr2=0x7f28b010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:09.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5570700 1 --2- 172.21.15.32:0/3334488283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b00d82c0 0x7f28b010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5570700 1 -- 172.21.15.32:0/3334488283 >> v2:172.21.15.86:3300/0 conn(0x7f28b00d1da0 msgr2=0x7f28b00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:09.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5570700 1 --2- 172.21.15.32:0/3334488283 >> v2:172.21.15.86:3300/0 conn(0x7f28b00d1da0 0x7f28b00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5570700 1 -- 172.21.15.32:0/3334488283 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b00d86c0 con 0x7f28b010d010 2021-04-09T18:30:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.791 7f28b5570700 1 --2- 172.21.15.32:0/3334488283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b010d010 0x7f28b00d1780 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7f28ac006700 tx=0x7f28ac009b80).ready entity=mon.0 client_cookie=7e1e6abe056bde88 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28a7fff700 1 -- 172.21.15.32:0/3334488283 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ac00c0c0 con 0x7f28b010d010 2021-04-09T18:30:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28a7fff700 1 -- 172.21.15.32:0/3334488283 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac00c440 con 0x7f28b010d010 2021-04-09T18:30:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28a7fff700 1 -- 172.21.15.32:0/3334488283 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ac00c920 con 0x7f28b010d010 2021-04-09T18:30:09.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28b7754700 1 -- 172.21.15.32:0/3334488283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b010d010 msgr2=0x7f28b00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28b7754700 1 --2- 172.21.15.32:0/3334488283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b010d010 0x7f28b00d1780 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7f28ac006700 tx=0x7f28ac009b80).stop 2021-04-09T18:30:09.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28b7754700 1 -- 172.21.15.32:0/3334488283 shutdown_connections 2021-04-09T18:30:09.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28b7754700 1 --2- 172.21.15.32:0/3334488283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b00d82c0 0x7f28b010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28b7754700 1 --2- 172.21.15.32:0/3334488283 >> v2:172.21.15.86:3300/0 conn(0x7f28b00d1da0 0x7f28b00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28b7754700 1 --2- 172.21.15.32:0/3334488283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b010d010 0x7f28b00d1780 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.792 7f28b7754700 1 -- 172.21.15.32:0/3334488283 >> 172.21.15.32:0/3334488283 conn(0x7f28b00ce5f0 msgr2=0x7f28b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.793 7f28b7754700 1 -- 172.21.15.32:0/3334488283 shutdown_connections 2021-04-09T18:30:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.793 7f28b7754700 1 -- 172.21.15.32:0/3334488283 wait complete. 2021-04-09T18:30:09.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.813 7f28b7754700 1 Processor -- start 2021-04-09T18:30:09.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.813 7f28b7754700 1 -- start start 2021-04-09T18:30:09.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.813 7f28b7754700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28b01ac380 0x7f28b01ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.813 7f28b7754700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b0186c90 0x7f28b0187090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.813 7f28b7754700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b0185530 0x7f28b0185930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.813 7f28b7754700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b0186790 con 0x7f28b0185530 2021-04-09T18:30:09.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.813 7f28b7754700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b01acda0 con 0x7f28b01ac380 2021-04-09T18:30:09.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.813 7f28b7754700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b018e100 con 0x7f28b0186c90 2021-04-09T18:30:09.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5d71700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b0185530 0x7f28b0185930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5d71700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b0185530 0x7f28b0185930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55450/0 (socket says 172.21.15.32:55450) 2021-04-09T18:30:09.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5d71700 1 -- 172.21.15.32:0/4124387295 learned_addr learned my addr 172.21.15.32:0/4124387295 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:09.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5570700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28b01ac380 0x7f28b01ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b4d6f700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b0186c90 0x7f28b0187090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5d71700 1 -- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b0186c90 msgr2=0x7f28b0187090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5d71700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b0186c90 0x7f28b0187090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5d71700 1 -- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28b01ac380 msgr2=0x7f28b01ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:09.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5d71700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28b01ac380 0x7f28b01ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:09.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5d71700 1 -- 172.21.15.32:0/4124387295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b01876b0 con 0x7f28b0185530 2021-04-09T18:30:09.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b5d71700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b0185530 0x7f28b0185930 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f28a8005fd0 tx=0x7f28a800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:09.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28a5ffb700 1 -- 172.21.15.32:0/4124387295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28a800c300 con 0x7f28b0185530 2021-04-09T18:30:09.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28a5ffb700 1 -- 172.21.15.32:0/4124387295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a8003030 con 0x7f28b0185530 2021-04-09T18:30:09.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b7754700 1 -- 172.21.15.32:0/4124387295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b01876b0 con 0x7f28b0185530 2021-04-09T18:30:09.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28a5ffb700 1 -- 172.21.15.32:0/4124387295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28a800c7b0 con 0x7f28b0185530 2021-04-09T18:30:09.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.814 7f28b7754700 1 -- 172.21.15.32:0/4124387295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b01876b0 con 0x7f28b0185530 2021-04-09T18:30:09.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.815 7f28a5ffb700 1 -- 172.21.15.32:0/4124387295 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f28a80011f0 con 0x7f28b0185530 2021-04-09T18:30:09.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.815 7f28a5ffb700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2890018930 0x7f289001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:09.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.815 7f28a5ffb700 1 -- 172.21.15.32:0/4124387295 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f28a8032ac0 con 0x7f28b0185530 2021-04-09T18:30:09.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.815 7f28b5570700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2890018930 0x7f289001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:09.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.815 7f28b5570700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2890018930 0x7f289001ae10 crc :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:09.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.835 7f28b7754700 1 -- 172.21.15.32:0/4124387295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b01acda0 con 0x7f28b0185530 2021-04-09T18:30:09.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:09.838 7f28a5ffb700 1 -- 172.21.15.32:0/4124387295 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f28a800ad90 con 0x7f28b0185530 2021-04-09T18:30:10.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.030 7f28b7754700 1 -- 172.21.15.32:0/4124387295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28b01acda0 con 0x7f28b0185530 2021-04-09T18:30:10.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.031 7f28a5ffb700 1 -- 172.21.15.32:0/4124387295 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 319 v319) v1 ==== 95+0+1857 (secure 0 0 0) 0x7f28a80011f0 con 0x7f28b0185530 2021-04-09T18:30:10.119 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 319 2021-04-09T18:30:10.120 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:10.120 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":319,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3983419158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3983419158},{"type":"v1","addr":"172.21.15.32:6834","nonce":3983419158}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":314},{"gid":10821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1847221608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1847221608},{"type":"v1","addr":"172.21.15.32:6835","nonce":1847221608}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":319},{"gid":58367,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6835/463218992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":463218992},{"type":"v1","addr":"172.21.15.86:6835","nonce":463218992}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307},{"gid":58370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3971194084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3971194084},{"type":"v1","addr":"172.21.15.86:6837","nonce":3971194084}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":307}],"filesystems":[]} 2021-04-09T18:30:10.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 -- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2890018930 msgr2=0x7f289001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:10.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2890018930 0x7f289001ae10 crc :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:10.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 -- 172.21.15.32:0/4124387295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b0185530 msgr2=0x7f28b0185930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:10.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b0185530 0x7f28b0185930 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f28a8005fd0 tx=0x7f28a800ac50).stop 2021-04-09T18:30:10.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 -- 172.21.15.32:0/4124387295 shutdown_connections 2021-04-09T18:30:10.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2890018930 0x7f289001ae10 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:10.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b0185530 0x7f28b0185930 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:10.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28b0186c90 0x7f28b0187090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:10.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 --2- 172.21.15.32:0/4124387295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28b01ac380 0x7f28b01ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:10.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.033 7f28b7754700 1 -- 172.21.15.32:0/4124387295 >> 172.21.15.32:0/4124387295 conn(0x7f28b019b890 msgr2=0x7f28b01a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:10.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.034 7f28b7754700 1 -- 172.21.15.32:0/4124387295 shutdown_connections 2021-04-09T18:30:10.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.034 7f28b7754700 1 -- 172.21.15.32:0/4124387295 wait complete. 2021-04-09T18:30:10.132 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:30:10.132 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:30:10.132 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:30:10.133 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:30:10.133 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:30:10.133 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:30:10.134 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:30:10.134 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:30:10.134 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:30:10.135 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:30:10.135 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:30:10.135 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:30:10.136 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:30:10.136 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:30:10.136 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:30:10.136 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:30:10.141 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:30:10.141 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:30:10.142 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:30:10.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:30:10.144 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:30:10.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:30:10.145 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:30:10.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:30:10.183 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:30:10.184 DEBUG:teuthology.parallel:result is None 2021-04-09T18:30:10.185 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:30:10.185 DEBUG:teuthology.parallel:result is None 2021-04-09T18:30:10.186 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:30:10.187 DEBUG:teuthology.parallel:result is None 2021-04-09T18:30:10.187 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:30:10.188 DEBUG:teuthology.parallel:result is None 2021-04-09T18:30:10.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:30:10.336 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:30:10.337 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:30:10.338 INFO:tasks.ceph.mds.a.smithi032.stderr:2021-04-09 18:30:10.249 7f36d739a700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2021-04-09T18:30:10.346 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:30:10.347 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:30:10.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.256 7fcc053a1700 1 Processor -- start 2021-04-09T18:30:10.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.256 7fcc053a1700 1 -- start start 2021-04-09T18:30:10.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcc053a1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc000cffb0 0x7fcc000d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:10.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcc053a1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc000d7290 con 0x7fcc000d6c90 2021-04-09T18:30:10.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcc053a1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0010e7e0 con 0x7fcc000d7810 2021-04-09T18:30:10.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcc053a1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000d7090 con 0x7fcc000cffb0 2021-04-09T18:30:10.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbff87e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcc000d6c90 0x7fcc0010cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36838/0 (socket says 172.21.15.32:36838) 2021-04-09T18:30:10.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbff87e700 1 -- 172.21.15.32:0/1926681513 learned_addr learned my addr 172.21.15.32:0/1926681513 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:10.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbff07d700 1 --2- 172.21.15.32:0/1926681513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc000cffb0 0x7fcc000d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:10.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbfe07b700 1 -- 172.21.15.32:0/1926681513 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672026287 0 0) 0x7fcbf8000d30 con 0x7fcc000d6c90 2021-04-09T18:30:10.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbfe07b700 1 -- 172.21.15.32:0/1926681513 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe8002fe0 con 0x7fcc000d6c90 2021-04-09T18:30:10.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbfe07b700 1 -- 172.21.15.32:0/1926681513 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62444756 0 0) 0x7fcbf4000c60 con 0x7fcc000d7810 2021-04-09T18:30:10.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbfe07b700 1 -- 172.21.15.32:0/1926681513 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe80042c0 con 0x7fcc000d7810 2021-04-09T18:30:10.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbff07d700 1 -- 172.21.15.32:0/1926681513 >> v1:172.21.15.86:6789/0 conn(0x7fcc000d7810 legacy=0x7fcc000d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:10.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbff07d700 1 -- 172.21.15.32:0/1926681513 >> v1:172.21.15.32:6789/0 conn(0x7fcc000d6c90 legacy=0x7fcc0010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:10.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbff07d700 1 -- 172.21.15.32:0/1926681513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc000d8590 con 0x7fcc000cffb0 2021-04-09T18:30:10.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbff07d700 1 --2- 172.21.15.32:0/1926681513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc000cffb0 0x7fcc000d2450 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fcbf0006650 tx=0x7fcbf0009a80).ready entity=mon.2 client_cookie=8fa5672b299d3251 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:10.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbfe07b700 1 -- 172.21.15.32:0/1926681513 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbf00009b0 con 0x7fcc000cffb0 2021-04-09T18:30:10.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbfe07b700 1 -- 172.21.15.32:0/1926681513 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf000c0c0 con 0x7fcc000cffb0 2021-04-09T18:30:10.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcbfe07b700 1 -- 172.21.15.32:0/1926681513 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbf000c510 con 0x7fcc000cffb0 2021-04-09T18:30:10.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcc053a1700 1 -- 172.21.15.32:0/1926681513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc000cffb0 msgr2=0x7fcc000d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:10.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.257 7fcc053a1700 1 --2- 172.21.15.32:0/1926681513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc000cffb0 0x7fcc000d2450 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fcbf0006650 tx=0x7fcbf0009a80).stop 2021-04-09T18:30:10.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.258 7fcc053a1700 1 -- 172.21.15.32:0/1926681513 shutdown_connections 2021-04-09T18:30:10.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.258 7fcc053a1700 1 --2- 172.21.15.32:0/1926681513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc000cffb0 0x7fcc000d2450 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:10.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.258 7fcc053a1700 1 -- 172.21.15.32:0/1926681513 >> 172.21.15.32:0/1926681513 conn(0x7fcc000ce5d0 msgr2=0x7fcc000cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:10.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.258 7fcc053a1700 1 -- 172.21.15.32:0/1926681513 shutdown_connections 2021-04-09T18:30:10.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.258 7fcc053a1700 1 -- 172.21.15.32:0/1926681513 wait complete. 2021-04-09T18:30:10.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.278 7fcc053a1700 1 Processor -- start 2021-04-09T18:30:10.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcc053a1700 1 -- start start 2021-04-09T18:30:10.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcc053a1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc0008ac60 0x7fcc0008b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:10.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcc053a1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc0008b680 0x7fcc0019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:10.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcc053a1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc0019d2a0 0x7fcc0019d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:10.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcc053a1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0008a6e0 con 0x7fcc0019d2a0 2021-04-09T18:30:10.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcc053a1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0017c520 con 0x7fcc0008ac60 2021-04-09T18:30:10.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcc053a1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0017c060 con 0x7fcc0008b680 2021-04-09T18:30:10.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcbff87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc0019d2a0 0x7fcc0019d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:10.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcbff87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc0019d2a0 0x7fcc0019d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55482/0 (socket says 172.21.15.32:55482) 2021-04-09T18:30:10.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcbff87e700 1 -- 172.21.15.32:0/3954646711 learned_addr learned my addr 172.21.15.32:0/3954646711 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:10.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcbff07d700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc0008ac60 0x7fcc0008b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:10.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.279 7fcbfe87c700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc0008b680 0x7fcc0019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:10.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcbff87e700 1 -- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc0008b680 msgr2=0x7fcc0019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcbff87e700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc0008b680 0x7fcc0019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcbff87e700 1 -- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc0008ac60 msgr2=0x7fcc0008b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcbff87e700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc0008ac60 0x7fcc0008b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:10.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcbff87e700 1 -- 172.21.15.32:0/3954646711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0019dcc0 con 0x7fcc0019d2a0 2021-04-09T18:30:10.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcbff87e700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc0019d2a0 0x7fcc0019d6a0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 rx=0x7fcbf8009980 tx=0x7fcbf800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:10.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcbe7fff700 1 -- 172.21.15.32:0/3954646711 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbf800be10 con 0x7fcc0019d2a0 2021-04-09T18:30:10.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcbe7fff700 1 -- 172.21.15.32:0/3954646711 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf8006030 con 0x7fcc0019d2a0 2021-04-09T18:30:10.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcbe7fff700 1 -- 172.21.15.32:0/3954646711 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbf80064c0 con 0x7fcc0019d2a0 2021-04-09T18:30:10.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0019dcc0 con 0x7fcc0019d2a0 2021-04-09T18:30:10.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.280 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0019dcc0 con 0x7fcc0019d2a0 2021-04-09T18:30:10.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.281 7fcbe7fff700 1 -- 172.21.15.32:0/3954646711 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcbf8006210 con 0x7fcc0019d2a0 2021-04-09T18:30:10.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.281 7fcbe7fff700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbe0018930 0x7fcbe001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:10.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.281 7fcbe7fff700 1 -- 172.21.15.32:0/3954646711 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fcbf8032a90 con 0x7fcc0019d2a0 2021-04-09T18:30:10.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.281 7fcbff07d700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbe0018930 0x7fcbe001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:10.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.281 7fcbff07d700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbe0018930 0x7fcbe001adc0 crc :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:10.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.300 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc001b7130 con 0x7fcc0019d2a0 2021-04-09T18:30:10.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.304 7fcbe7fff700 1 -- 172.21.15.32:0/3954646711 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcbf8000f90 con 0x7fcc0019d2a0 2021-04-09T18:30:10.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.504 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fcc001b7130 con 0x7fcc0019d2a0 2021-04-09T18:30:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:10.977 7fcbe7fff700 1 -- 172.21.15.32:0/3954646711 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v172) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcbf8006030 con 0x7fcc0019d2a0 2021-04-09T18:30:11.106 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:30:11.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbe0018930 msgr2=0x7fcbe001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbe0018930 0x7fcbe001adc0 crc :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc0019d2a0 msgr2=0x7fcc0019d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc0019d2a0 0x7fcc0019d6a0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 rx=0x7fcbf8009980 tx=0x7fcbf800bcd0).stop 2021-04-09T18:30:11.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 shutdown_connections 2021-04-09T18:30:11.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbe0018930 0x7fcbe001adc0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc0019d2a0 0x7fcc0019d6a0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc0008b680 0x7fcc0019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 --2- 172.21.15.32:0/3954646711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc0008ac60 0x7fcc0008b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 >> 172.21.15.32:0/3954646711 conn(0x7fcc001b7360 msgr2=0x7fcc001b7800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:11.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 shutdown_connections 2021-04-09T18:30:11.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.021 7fcc053a1700 1 -- 172.21.15.32:0/3954646711 wait complete. 2021-04-09T18:30:11.120 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:30:11.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:30:11.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.182 7f120cc88700 1 Processor -- start 2021-04-09T18:30:11.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.182 7f120cc88700 1 -- start start 2021-04-09T18:30:11.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120cc88700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f12080d7ca0 0x7f12080d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120cc88700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f120810da00 0x7f120810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120cc88700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120810d480 con 0x7f120810d010 2021-04-09T18:30:11.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120cc88700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12080d9220 con 0x7f120810da00 2021-04-09T18:30:11.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120cc88700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12080d86c0 con 0x7f12080d7ca0 2021-04-09T18:30:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120705f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f120810da00 0x7f120810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120705f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f120810da00 0x7f120810de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55488/0 (socket says 172.21.15.32:55488) 2021-04-09T18:30:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120705f700 1 -- 172.21.15.32:0/2512081939 learned_addr learned my addr 172.21.15.32:0/2512081939 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:11.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120605d700 1 --2- 172.21.15.32:0/2512081939 >> v2:172.21.15.86:3300/0 conn(0x7f12080d7ca0 0x7f12080d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120705f700 1 -- 172.21.15.32:0/2512081939 >> v2:172.21.15.86:3300/0 conn(0x7f12080d7ca0 msgr2=0x7f12080d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:11.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120705f700 1 --2- 172.21.15.32:0/2512081939 >> v2:172.21.15.86:3300/0 conn(0x7f12080d7ca0 0x7f12080d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120705f700 1 -- 172.21.15.32:0/2512081939 >> v1:172.21.15.86:6789/0 conn(0x7f120810d010 legacy=0x7f12080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120705f700 1 -- 172.21.15.32:0/2512081939 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12080d1f70 con 0x7f120810da00 2021-04-09T18:30:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120705f700 1 --2- 172.21.15.32:0/2512081939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120810da00 0x7f120810de00 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f1200006f00 tx=0x7f1200009f10).ready entity=mon.0 client_cookie=b6739c53e66cf49f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120585c700 1 -- 172.21.15.32:0/2512081939 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f120000b3b0 con 0x7f120810da00 2021-04-09T18:30:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120585c700 1 -- 172.21.15.32:0/2512081939 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120000b730 con 0x7f120810da00 2021-04-09T18:30:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.183 7f120585c700 1 -- 172.21.15.32:0/2512081939 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f120000bc10 con 0x7f120810da00 2021-04-09T18:30:11.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.184 7f120cc88700 1 -- 172.21.15.32:0/2512081939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120810da00 msgr2=0x7f120810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.184 7f120cc88700 1 --2- 172.21.15.32:0/2512081939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120810da00 0x7f120810de00 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f1200006f00 tx=0x7f1200009f10).stop 2021-04-09T18:30:11.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.184 7f120cc88700 1 -- 172.21.15.32:0/2512081939 shutdown_connections 2021-04-09T18:30:11.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.184 7f120cc88700 1 --2- 172.21.15.32:0/2512081939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120810da00 0x7f120810de00 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.184 7f120cc88700 1 --2- 172.21.15.32:0/2512081939 >> v2:172.21.15.86:3300/0 conn(0x7f12080d7ca0 0x7f12080d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.184 7f120cc88700 1 -- 172.21.15.32:0/2512081939 >> 172.21.15.32:0/2512081939 conn(0x7f12080ce5f0 msgr2=0x7f12080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:11.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.184 7f120cc88700 1 -- 172.21.15.32:0/2512081939 shutdown_connections 2021-04-09T18:30:11.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.184 7f120cc88700 1 -- 172.21.15.32:0/2512081939 wait complete. 2021-04-09T18:30:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.204 7f120cc88700 1 Processor -- start 2021-04-09T18:30:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.204 7f120cc88700 1 -- start start 2021-04-09T18:30:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120cc88700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12081aca20 0x7f12081ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120cc88700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120818bd40 0x7f120818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120cc88700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f120808fba0 0x7f120808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120cc88700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120818c7e0 con 0x7f120818bd40 2021-04-09T18:30:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120cc88700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120818ca30 con 0x7f12081aca20 2021-04-09T18:30:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120cc88700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12080905f0 con 0x7f120808fba0 2021-04-09T18:30:11.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120605d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120818bd40 0x7f120818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120605d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120818bd40 0x7f120818c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55494/0 (socket says 172.21.15.32:55494) 2021-04-09T18:30:11.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120605d700 1 -- 172.21.15.32:0/1308207428 learned_addr learned my addr 172.21.15.32:0/1308207428 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:11.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120685e700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12081aca20 0x7f12081ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120705f700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f120808fba0 0x7f120808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120605d700 1 -- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f120808fba0 msgr2=0x7f120808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120605d700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f120808fba0 0x7f120808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120605d700 1 -- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12081aca20 msgr2=0x7f12081ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120605d700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12081aca20 0x7f12081ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.205 7f120605d700 1 -- 172.21.15.32:0/1308207428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12080b4300 con 0x7f120818bd40 2021-04-09T18:30:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.206 7f120605d700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120818bd40 0x7f120818c140 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f11f4006f60 tx=0x7f11f400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.206 7f11ff7fe700 1 -- 172.21.15.32:0/1308207428 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f400bd60 con 0x7f120818bd40 2021-04-09T18:30:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.206 7f11ff7fe700 1 -- 172.21.15.32:0/1308207428 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4000c90 con 0x7f120818bd40 2021-04-09T18:30:11.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.206 7f120cc88700 1 -- 172.21.15.32:0/1308207428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12080b4300 con 0x7f120818bd40 2021-04-09T18:30:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.206 7f11ff7fe700 1 -- 172.21.15.32:0/1308207428 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f4002220 con 0x7f120818bd40 2021-04-09T18:30:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.206 7f120cc88700 1 -- 172.21.15.32:0/1308207428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12080b4300 con 0x7f120818bd40 2021-04-09T18:30:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.207 7f11ff7fe700 1 -- 172.21.15.32:0/1308207428 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f11f40052e0 con 0x7f120818bd40 2021-04-09T18:30:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.207 7f11ff7fe700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e40188d0 0x7f11e401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.207 7f11ff7fe700 1 -- 172.21.15.32:0/1308207428 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f11f40326c0 con 0x7f120818bd40 2021-04-09T18:30:11.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.207 7f120685e700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e40188d0 0x7f11e401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.207 7f120685e700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e40188d0 0x7f11e401add0 crc :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:11.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.227 7f120cc88700 1 -- 172.21.15.32:0/1308207428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1208182d70 con 0x7f120818bd40 2021-04-09T18:30:11.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.230 7f11ff7fe700 1 -- 172.21.15.32:0/1308207428 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f11f4000c90 con 0x7f120818bd40 2021-04-09T18:30:11.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.414 7f120cc88700 1 -- 172.21.15.32:0/1308207428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f120818c760 con 0x7f120818bd40 2021-04-09T18:30:11.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.415 7f11ff7fe700 1 -- 172.21.15.32:0/1308207428 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v39) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f11f4006b00 con 0x7f120818bd40 2021-04-09T18:30:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:11.504 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:30:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:30:11.504 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:30:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:30:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:30:11.505 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.506 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:30:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:30:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.507 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:30:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.508 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:30:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:30:11.509 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:30:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.510 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:30:11.511 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:30:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:30:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.512 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.513 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:30:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:30:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.514 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.515 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:30:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:30:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.516 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:30:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:30:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.518 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.519 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:30:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:30:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.520 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.521 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:30:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:30:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.522 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.523 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:30:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:30:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.524 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:30:11.525 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:30:11.526 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.526 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.526 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.526 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.527 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:30:11.527 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:30:11.528 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.528 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.528 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.528 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.528 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.529 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.529 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:30:11.529 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:30:11.529 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.530 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:30:11.530 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:30:11.530 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:30:11.530 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.531 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:30:11.531 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:30:11.531 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.532 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:30:11.532 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:30:11.532 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:30:11.532 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.533 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:30:11.533 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:30:11.533 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.533 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:30:11.534 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:30:11.534 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:30:11.534 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.534 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.535 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.535 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:30:11.535 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:30:11.535 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.536 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:30:11.536 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:30:11.536 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.536 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:11.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:11.537 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:30:11.537 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:30:11.537 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:30:11.537 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:30:11.538 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:30:11.538 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:30:11.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.538 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:11.538 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:30:11.539 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:30:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 -- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e40188d0 msgr2=0x7f11e401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e40188d0 0x7f11e401add0 crc :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 -- 172.21.15.32:0/1308207428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120818bd40 msgr2=0x7f120818c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120818bd40 0x7f120818c140 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f11f4006f60 tx=0x7f11f400b970).stop 2021-04-09T18:30:11.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 -- 172.21.15.32:0/1308207428 shutdown_connections 2021-04-09T18:30:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e40188d0 0x7f11e401add0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f120818bd40 0x7f120818c140 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f120808fba0 0x7f120808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 --2- 172.21.15.32:0/1308207428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12081aca20 0x7f12081ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.417 7f120cc88700 1 -- 172.21.15.32:0/1308207428 >> 172.21.15.32:0/1308207428 conn(0x7f120819b460 msgr2=0x7f120819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:11.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.418 7f120cc88700 1 -- 172.21.15.32:0/1308207428 shutdown_connections 2021-04-09T18:30:11.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.418 7f120cc88700 1 -- 172.21.15.32:0/1308207428 wait complete. 2021-04-09T18:30:11.544 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-04-09T18:30:11.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.616 7fb5459be700 1 Processor -- start 2021-04-09T18:30:11.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.616 7fb5459be700 1 -- start start 2021-04-09T18:30:11.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb5459be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400d7cc0 0x7fb5400d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb5459be700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb54010da80 0x7fb54010de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb5459be700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54010d500 con 0x7fb54010d050 2021-04-09T18:30:11.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb5459be700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5400d9260 con 0x7fb54010da80 2021-04-09T18:30:11.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb5459be700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400d09f0 con 0x7fb5400d7cc0 2021-04-09T18:30:11.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb53f574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb54010d050 0x7fb5400cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36858/0 (socket says 172.21.15.32:36858) 2021-04-09T18:30:11.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb53f574700 1 -- 172.21.15.32:0/727699622 learned_addr learned my addr 172.21.15.32:0/727699622 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:11.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb53fd75700 1 --2- 172.21.15.32:0/727699622 >> v2:172.21.15.32:3300/0 conn(0x7fb54010da80 0x7fb54010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb53ed73700 1 --2- 172.21.15.32:0/727699622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400d7cc0 0x7fb5400d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.617 7fb53e572700 1 -- 172.21.15.32:0/727699622 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462436490 0 0) 0x7fb530000ce0 con 0x7fb54010d050 2021-04-09T18:30:11.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53e572700 1 -- 172.21.15.32:0/727699622 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528002fe0 con 0x7fb54010d050 2021-04-09T18:30:11.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53fd75700 1 -- 172.21.15.32:0/727699622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400d7cc0 msgr2=0x7fb5400d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:11.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53fd75700 1 --2- 172.21.15.32:0/727699622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400d7cc0 0x7fb5400d80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53fd75700 1 -- 172.21.15.32:0/727699622 >> v1:172.21.15.32:6789/0 conn(0x7fb54010d050 legacy=0x7fb5400cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53fd75700 1 -- 172.21.15.32:0/727699622 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5400d0330 con 0x7fb54010da80 2021-04-09T18:30:11.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53ed73700 1 --2- 172.21.15.32:0/727699622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400d7cc0 0x7fb5400d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:11.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53fd75700 1 --2- 172.21.15.32:0/727699622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54010da80 0x7fb54010de80 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7fb538006e30 tx=0x7fb538009e60).ready entity=mon.0 client_cookie=32ae16543bc06936 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:11.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53e572700 1 -- 172.21.15.32:0/727699622 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb538006a50 con 0x7fb54010da80 2021-04-09T18:30:11.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53e572700 1 -- 172.21.15.32:0/727699622 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53800b300 con 0x7fb54010da80 2021-04-09T18:30:11.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb53e572700 1 -- 172.21.15.32:0/727699622 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb53800b750 con 0x7fb54010da80 2021-04-09T18:30:11.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb5459be700 1 -- 172.21.15.32:0/727699622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54010da80 msgr2=0x7fb54010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.618 7fb5459be700 1 --2- 172.21.15.32:0/727699622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54010da80 0x7fb54010de80 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7fb538006e30 tx=0x7fb538009e60).stop 2021-04-09T18:30:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.619 7fb5459be700 1 -- 172.21.15.32:0/727699622 shutdown_connections 2021-04-09T18:30:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.619 7fb5459be700 1 --2- 172.21.15.32:0/727699622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb54010da80 0x7fb54010de80 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.619 7fb5459be700 1 --2- 172.21.15.32:0/727699622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400d7cc0 0x7fb5400d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.619 7fb5459be700 1 -- 172.21.15.32:0/727699622 >> 172.21.15.32:0/727699622 conn(0x7fb5400ce620 msgr2=0x7fb5400d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.619 7fb5459be700 1 -- 172.21.15.32:0/727699622 shutdown_connections 2021-04-09T18:30:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.619 7fb5459be700 1 -- 172.21.15.32:0/727699622 wait complete. 2021-04-09T18:30:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.639 7fb5459be700 1 Processor -- start 2021-04-09T18:30:11.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.639 7fb5459be700 1 -- start start 2021-04-09T18:30:11.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb5459be700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400a5f30 0x7fb540093bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb5459be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5400941f0 0x7fb540197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb5459be700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5401984c0 0x7fb540092250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb5459be700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400a6520 con 0x7fb5400941f0 2021-04-09T18:30:11.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb5459be700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb540090680 con 0x7fb5401984c0 2021-04-09T18:30:11.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb5459be700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb540093650 con 0x7fb5400a5f30 2021-04-09T18:30:11.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53ed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5400941f0 0x7fb540197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53ed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5400941f0 0x7fb540197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55508/0 (socket says 172.21.15.32:55508) 2021-04-09T18:30:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53ed73700 1 -- 172.21.15.32:0/2261894362 learned_addr learned my addr 172.21.15.32:0/2261894362 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53f574700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400a5f30 0x7fb540093bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53fd75700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5401984c0 0x7fb540092250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53ed73700 1 -- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400a5f30 msgr2=0x7fb540093bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53ed73700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400a5f30 0x7fb540093bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53ed73700 1 -- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5401984c0 msgr2=0x7fb540092250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:11.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53ed73700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5401984c0 0x7fb540092250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53ed73700 1 -- 172.21.15.32:0/2261894362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5400945f0 con 0x7fb5400941f0 2021-04-09T18:30:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb53ed73700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5400941f0 0x7fb540197ea0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 rx=0x7fb534006f40 tx=0x7fb53400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb527fff700 1 -- 172.21.15.32:0/2261894362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb53400bd80 con 0x7fb5400941f0 2021-04-09T18:30:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb527fff700 1 -- 172.21.15.32:0/2261894362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb534000c90 con 0x7fb5400941f0 2021-04-09T18:30:11.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb527fff700 1 -- 172.21.15.32:0/2261894362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb534002220 con 0x7fb5400941f0 2021-04-09T18:30:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb5459be700 1 -- 172.21.15.32:0/2261894362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5400945f0 con 0x7fb5400941f0 2021-04-09T18:30:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.640 7fb5459be700 1 -- 172.21.15.32:0/2261894362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5400945f0 con 0x7fb5400941f0 2021-04-09T18:30:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.641 7fb527fff700 1 -- 172.21.15.32:0/2261894362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb53400a000 con 0x7fb5400941f0 2021-04-09T18:30:11.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.641 7fb527fff700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb520018930 0x7fb52001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:11.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.641 7fb527fff700 1 -- 172.21.15.32:0/2261894362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fb5340326e0 con 0x7fb5400941f0 2021-04-09T18:30:11.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.641 7fb53f574700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb520018930 0x7fb52001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:11.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.642 7fb53f574700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb520018930 0x7fb52001adc0 crc :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:11.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.661 7fb5459be700 1 -- 172.21.15.32:0/2261894362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5400a8ea0 con 0x7fb5400941f0 2021-04-09T18:30:11.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.664 7fb527fff700 1 -- 172.21.15.32:0/2261894362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb534006b00 con 0x7fb5400941f0 2021-04-09T18:30:11.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:11.857 7fb5459be700 1 -- 172.21.15.32:0/2261894362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb5400a8ea0 con 0x7fb5400941f0 2021-04-09T18:30:12.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.046 7fb527fff700 1 -- 172.21.15.32:0/2261894362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v320) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb53400a000 con 0x7fb5400941f0 2021-04-09T18:30:12.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 -- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb520018930 msgr2=0x7fb52001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb520018930 0x7fb52001adc0 crc :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 -- 172.21.15.32:0/2261894362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5400941f0 msgr2=0x7fb540197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5400941f0 0x7fb540197ea0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 rx=0x7fb534006f40 tx=0x7fb53400b990).stop 2021-04-09T18:30:12.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 -- 172.21.15.32:0/2261894362 shutdown_connections 2021-04-09T18:30:12.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb520018930 0x7fb52001adc0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb5400941f0 0x7fb540197ea0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5401984c0 0x7fb540092250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 --2- 172.21.15.32:0/2261894362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5400a5f30 0x7fb540093bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 -- 172.21.15.32:0/2261894362 >> 172.21.15.32:0/2261894362 conn(0x7fb5400b92e0 msgr2=0x7fb5400af810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:12.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 -- 172.21.15.32:0/2261894362 shutdown_connections 2021-04-09T18:30:12.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.096 7fb5459be700 1 -- 172.21.15.32:0/2261894362 wait complete. 2021-04-09T18:30:12.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.254 7ff3b838e700 1 Processor -- start 2021-04-09T18:30:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.254 7ff3b838e700 1 -- start start 2021-04-09T18:30:12.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.255 7ff3b838e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff3b010da00 0x7ff3b010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.255 7ff3b838e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b010d480 con 0x7ff3b00d7ca0 2021-04-09T18:30:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.255 7ff3b838e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b00d1f70 con 0x7ff3b010d010 2021-04-09T18:30:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.255 7ff3b838e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b00d8d30 con 0x7ff3b010da00 2021-04-09T18:30:12.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.255 7ff3b59a9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff3b00d7ca0 0x7ff3b00d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36872/0 (socket says 172.21.15.32:36872) 2021-04-09T18:30:12.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.255 7ff3b59a9700 1 -- 172.21.15.32:0/1232492504 learned_addr learned my addr 172.21.15.32:0/1232492504 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:12.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b69ab700 1 --2- 172.21.15.32:0/1232492504 >> v2:172.21.15.86:3300/0 conn(0x7ff3b010da00 0x7ff3b010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b51a8700 1 -- 172.21.15.32:0/1232492504 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916527060 0 0) 0x7ff3a0000ce0 con 0x7ff3b00d7ca0 2021-04-09T18:30:12.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b51a8700 1 -- 172.21.15.32:0/1232492504 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c002fe0 con 0x7ff3b00d7ca0 2021-04-09T18:30:12.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b51a8700 1 -- 172.21.15.32:0/1232492504 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 135627389 0 0) 0x7ff3a00010e0 con 0x7ff3b00d7ca0 2021-04-09T18:30:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b69ab700 1 -- 172.21.15.32:0/1232492504 >> v1:172.21.15.86:6789/0 conn(0x7ff3b010d010 legacy=0x7ff3b00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b69ab700 1 -- 172.21.15.32:0/1232492504 >> v1:172.21.15.32:6789/0 conn(0x7ff3b00d7ca0 legacy=0x7ff3b00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b69ab700 1 -- 172.21.15.32:0/1232492504 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b00d9220 con 0x7ff3b010da00 2021-04-09T18:30:12.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b69ab700 1 --2- 172.21.15.32:0/1232492504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3b010da00 0x7ff3b010de00 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7ff3a8006e50 tx=0x7ff3a8009e70).ready entity=mon.1 client_cookie=66e75ef5a47ac543 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b51a8700 1 -- 172.21.15.32:0/1232492504 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3a8006a20 con 0x7ff3b010da00 2021-04-09T18:30:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b51a8700 1 -- 172.21.15.32:0/1232492504 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a8006790 con 0x7ff3b010da00 2021-04-09T18:30:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b51a8700 1 -- 172.21.15.32:0/1232492504 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3a800b700 con 0x7ff3b010da00 2021-04-09T18:30:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b838e700 1 -- 172.21.15.32:0/1232492504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3b010da00 msgr2=0x7ff3b010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.256 7ff3b838e700 1 --2- 172.21.15.32:0/1232492504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3b010da00 0x7ff3b010de00 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7ff3a8006e50 tx=0x7ff3a8009e70).stop 2021-04-09T18:30:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.257 7ff3b838e700 1 -- 172.21.15.32:0/1232492504 shutdown_connections 2021-04-09T18:30:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.257 7ff3b838e700 1 --2- 172.21.15.32:0/1232492504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3b010da00 0x7ff3b010de00 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.257 7ff3b838e700 1 -- 172.21.15.32:0/1232492504 >> 172.21.15.32:0/1232492504 conn(0x7ff3b00ce5f0 msgr2=0x7ff3b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.257 7ff3b838e700 1 -- 172.21.15.32:0/1232492504 shutdown_connections 2021-04-09T18:30:12.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.257 7ff3b838e700 1 -- 172.21.15.32:0/1232492504 wait complete. 2021-04-09T18:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.277 7ff3b838e700 1 Processor -- start 2021-04-09T18:30:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.277 7ff3b838e700 1 -- start start 2021-04-09T18:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b838e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3b00ab300 0x7ff3b01a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b838e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3b01a8ad0 0x7ff3b01a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b838e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3b0180db0 0x7ff3b01811b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b838e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01817d0 con 0x7ff3b0180db0 2021-04-09T18:30:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b838e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01819f0 con 0x7ff3b00ab300 2021-04-09T18:30:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b838e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01a1f80 con 0x7ff3b01a8ad0 2021-04-09T18:30:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b69ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3b0180db0 0x7ff3b01811b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b69ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3b0180db0 0x7ff3b01811b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55522/0 (socket says 172.21.15.32:55522) 2021-04-09T18:30:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b69ab700 1 -- 172.21.15.32:0/2836970460 learned_addr learned my addr 172.21.15.32:0/2836970460 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b59a9700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3b01a8ad0 0x7ff3b01a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b61aa700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3b00ab300 0x7ff3b01a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b69ab700 1 -- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3b01a8ad0 msgr2=0x7ff3b01a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b69ab700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3b01a8ad0 0x7ff3b01a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b69ab700 1 -- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3b00ab300 msgr2=0x7ff3b01a12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b69ab700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3b00ab300 0x7ff3b01a12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b69ab700 1 -- 172.21.15.32:0/2836970460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b01a21a0 con 0x7ff3b0180db0 2021-04-09T18:30:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.278 7ff3b69ab700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3b0180db0 0x7ff3b01811b0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7ff3a8006e60 tx=0x7ff3a800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.279 7ff3a6ffd700 1 -- 172.21.15.32:0/2836970460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3a8009130 con 0x7ff3b0180db0 2021-04-09T18:30:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.279 7ff3a6ffd700 1 -- 172.21.15.32:0/2836970460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a8009430 con 0x7ff3b0180db0 2021-04-09T18:30:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.279 7ff3a6ffd700 1 -- 172.21.15.32:0/2836970460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3a8001390 con 0x7ff3b0180db0 2021-04-09T18:30:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.279 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b01a21a0 con 0x7ff3b0180db0 2021-04-09T18:30:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.279 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b01a21a0 con 0x7ff3b0180db0 2021-04-09T18:30:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.279 7ff3a6ffd700 1 -- 172.21.15.32:0/2836970460 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff3a8009130 con 0x7ff3b0180db0 2021-04-09T18:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.279 7ff3a6ffd700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff398018930 0x7ff39801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.279 7ff3a6ffd700 1 -- 172.21.15.32:0/2836970460 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 4757+0+0 (secure 0 0 0) 0x7ff3a8032a20 con 0x7ff3b0180db0 2021-04-09T18:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.280 7ff3b61aa700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff398018930 0x7ff39801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.280 7ff3b61aa700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff398018930 0x7ff39801ae10 crc :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:12.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.300 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b01b7070 con 0x7ff3b0180db0 2021-04-09T18:30:12.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.303 7ff3a6ffd700 1 -- 172.21.15.32:0/2836970460 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff3a8009430 con 0x7ff3b0180db0 2021-04-09T18:30:12.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.493 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3b010d480 con 0x7ff3b0180db0 2021-04-09T18:30:12.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.493 7ff3a6ffd700 1 -- 172.21.15.32:0/2836970460 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+11154 (secure 0 0 0) 0x7ff3a8009430 con 0x7ff3b0180db0 2021-04-09T18:30:12.509 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:12.509 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":172,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:10.974069","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:12.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff398018930 msgr2=0x7ff39801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff398018930 0x7ff39801ae10 crc :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3b0180db0 msgr2=0x7ff3b01811b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3b0180db0 0x7ff3b01811b0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7ff3a8006e60 tx=0x7ff3a800bcf0).stop 2021-04-09T18:30:12.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 shutdown_connections 2021-04-09T18:30:12.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff398018930 0x7ff39801ae10 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3b0180db0 0x7ff3b01811b0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3b01a8ad0 0x7ff3b01a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 --2- 172.21.15.32:0/2836970460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3b00ab300 0x7ff3b01a12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 >> 172.21.15.32:0/2836970460 conn(0x7ff3b01aa2b0 msgr2=0x7ff3b01a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 shutdown_connections 2021-04-09T18:30:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.496 7ff3b838e700 1 -- 172.21.15.32:0/2836970460 wait complete. 2021-04-09T18:30:12.521 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:30:12.522 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:30:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.658 7f2267923700 1 Processor -- start 2021-04-09T18:30:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.659 7f2267923700 1 -- start start 2021-04-09T18:30:12.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.659 7f2267923700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226010d010 0x7f22600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.659 7f2267923700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22600d8490 0x7f226010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.659 7f2267923700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226010d480 con 0x7f22600d1da0 2021-04-09T18:30:12.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.659 7f2267923700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226010e780 con 0x7f22600d8490 2021-04-09T18:30:12.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.659 7f2267923700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22600d8d30 con 0x7f226010d010 2021-04-09T18:30:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.659 7f2265f40700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22600d8490 0x7f226010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.659 7f2265f40700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22600d8490 0x7f226010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55530/0 (socket says 172.21.15.32:55530) 2021-04-09T18:30:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.659 7f2265f40700 1 -- 172.21.15.32:0/1400587688 learned_addr learned my addr 172.21.15.32:0/1400587688 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f226573f700 1 --2- 172.21.15.32:0/1400587688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226010d010 0x7f22600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2265f40700 1 -- 172.21.15.32:0/1400587688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226010d010 msgr2=0x7f22600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2265f40700 1 --2- 172.21.15.32:0/1400587688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226010d010 0x7f22600d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2265f40700 1 -- 172.21.15.32:0/1400587688 >> v1:172.21.15.86:6789/0 conn(0x7f22600d1da0 legacy=0x7f22600d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2265f40700 1 -- 172.21.15.32:0/1400587688 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22600d9220 con 0x7f22600d8490 2021-04-09T18:30:12.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2265f40700 1 --2- 172.21.15.32:0/1400587688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600d8490 0x7f226010da00 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7f2258006f00 tx=0x7f2258009f10).ready entity=mon.0 client_cookie=42e6a1665b4324b4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2257fff700 1 -- 172.21.15.32:0/1400587688 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f225800b3b0 con 0x7f22600d8490 2021-04-09T18:30:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2257fff700 1 -- 172.21.15.32:0/1400587688 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225800b730 con 0x7f22600d8490 2021-04-09T18:30:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2257fff700 1 -- 172.21.15.32:0/1400587688 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f225800bc10 con 0x7f22600d8490 2021-04-09T18:30:12.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2267923700 1 -- 172.21.15.32:0/1400587688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600d8490 msgr2=0x7f226010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.660 7f2267923700 1 --2- 172.21.15.32:0/1400587688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600d8490 0x7f226010da00 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7f2258006f00 tx=0x7f2258009f10).stop 2021-04-09T18:30:12.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.661 7f2267923700 1 -- 172.21.15.32:0/1400587688 shutdown_connections 2021-04-09T18:30:12.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.661 7f2267923700 1 --2- 172.21.15.32:0/1400587688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600d8490 0x7f226010da00 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.661 7f2267923700 1 --2- 172.21.15.32:0/1400587688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226010d010 0x7f22600d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.661 7f2267923700 1 -- 172.21.15.32:0/1400587688 >> 172.21.15.32:0/1400587688 conn(0x7f22600ce5f0 msgr2=0x7f22600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:12.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.661 7f2267923700 1 -- 172.21.15.32:0/1400587688 shutdown_connections 2021-04-09T18:30:12.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.661 7f2267923700 1 -- 172.21.15.32:0/1400587688 wait complete. 2021-04-09T18:30:12.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.681 7f2267923700 1 Processor -- start 2021-04-09T18:30:12.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.681 7f2267923700 1 -- start start 2021-04-09T18:30:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2267923700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226017bb20 0x7f22600c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2267923700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600b7620 0x7f22600c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2267923700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226017bf20 0x7f226008a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2267923700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22600c1e00 con 0x7f22600b7620 2021-04-09T18:30:12.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2267923700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22600c0b00 con 0x7f226017bf20 2021-04-09T18:30:12.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2267923700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226008ac40 con 0x7f226017bb20 2021-04-09T18:30:12.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2264f3e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600b7620 0x7f22600c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2264f3e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600b7620 0x7f22600c16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55538/0 (socket says 172.21.15.32:55538) 2021-04-09T18:30:12.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2264f3e700 1 -- 172.21.15.32:0/3120699134 learned_addr learned my addr 172.21.15.32:0/3120699134 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:12.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f226573f700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226017bb20 0x7f22600c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2265f40700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226017bf20 0x7f226008a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2264f3e700 1 -- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226017bb20 msgr2=0x7f22600c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2264f3e700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226017bb20 0x7f22600c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2264f3e700 1 -- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226017bf20 msgr2=0x7f226008a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2264f3e700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226017bf20 0x7f226008a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2264f3e700 1 -- 172.21.15.32:0/3120699134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226008ae60 con 0x7f22600b7620 2021-04-09T18:30:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f226573f700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226017bb20 0x7f22600c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2264f3e700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600b7620 0x7f22600c16a0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7f225000b5b0 tx=0x7f225000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.682 7f2255ffb700 1 -- 172.21.15.32:0/3120699134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f225000bd40 con 0x7f22600b7620 2021-04-09T18:30:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.683 7f2255ffb700 1 -- 172.21.15.32:0/3120699134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2250006240 con 0x7f22600b7620 2021-04-09T18:30:12.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.683 7f2267923700 1 -- 172.21.15.32:0/3120699134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226008ae60 con 0x7f22600b7620 2021-04-09T18:30:12.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.683 7f2255ffb700 1 -- 172.21.15.32:0/3120699134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2250005690 con 0x7f22600b7620 2021-04-09T18:30:12.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.683 7f2267923700 1 -- 172.21.15.32:0/3120699134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226008ae60 con 0x7f22600b7620 2021-04-09T18:30:12.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.683 7f2255ffb700 1 -- 172.21.15.32:0/3120699134 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f22500054a0 con 0x7f22600b7620 2021-04-09T18:30:12.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.683 7f2255ffb700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2240018930 0x7f224001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.683 7f2255ffb700 1 -- 172.21.15.32:0/3120699134 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f22500326d0 con 0x7f22600b7620 2021-04-09T18:30:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.683 7f226573f700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2240018930 0x7f224001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.684 7f226573f700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2240018930 0x7f224001ae10 crc :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.703 7f2267923700 1 -- 172.21.15.32:0/3120699134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226010d480 con 0x7f22600b7620 2021-04-09T18:30:12.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.707 7f2255ffb700 1 -- 172.21.15.32:0/3120699134 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f22500009b0 con 0x7f22600b7620 2021-04-09T18:30:12.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.886 7f2267923700 1 -- 172.21.15.32:0/3120699134 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f226010d480 con 0x7f22600b7620 2021-04-09T18:30:12.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.886 7f2255ffb700 1 -- 172.21.15.32:0/3120699134 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v172) v1 ==== 92+0+10 (secure 0 0 0) 0x7f22500054a0 con 0x7f22600b7620 2021-04-09T18:30:12.902 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:12.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 -- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2240018930 msgr2=0x7f224001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2240018930 0x7f224001ae10 crc :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 -- 172.21.15.32:0/3120699134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600b7620 msgr2=0x7f22600c16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:12.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600b7620 0x7f22600c16a0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7f225000b5b0 tx=0x7f225000b950).stop 2021-04-09T18:30:12.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 -- 172.21.15.32:0/3120699134 shutdown_connections 2021-04-09T18:30:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2240018930 0x7f224001ae10 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600b7620 0x7f22600c16a0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226017bf20 0x7f226008a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 --2- 172.21.15.32:0/3120699134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f226017bb20 0x7f22600c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:12.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 -- 172.21.15.32:0/3120699134 >> 172.21.15.32:0/3120699134 conn(0x7f2260181390 msgr2=0x7f22600adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:12.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 -- 172.21.15.32:0/3120699134 shutdown_connections 2021-04-09T18:30:12.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:12.889 7f2267923700 1 -- 172.21.15.32:0/3120699134 wait complete. 2021-04-09T18:30:12.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:30:13.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.049 7fdb2e3df700 1 Processor -- start 2021-04-09T18:30:13.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.049 7fdb2e3df700 1 -- start start 2021-04-09T18:30:13.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb2e3df700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280d7ca0 0x7fdb280d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb2e3df700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2810d480 con 0x7fdb2810d010 2021-04-09T18:30:13.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb2e3df700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb280d86c0 con 0x7fdb2810da00 2021-04-09T18:30:13.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb27fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdb2810d010 0x7fdb280d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36900/0 (socket says 172.21.15.32:36900) 2021-04-09T18:30:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb2e3df700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb280d9220 con 0x7fdb280d7ca0 2021-04-09T18:30:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb27fff700 1 -- 172.21.15.32:0/3180529704 learned_addr learned my addr 172.21.15.32:0/3180529704 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb1ffff700 1 --2- 172.21.15.32:0/3180529704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280d7ca0 0x7fdb280d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb277fe700 1 -- 172.21.15.32:0/3180529704 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149569832 0 0) 0x7fdb18000ce0 con 0x7fdb2810d010 2021-04-09T18:30:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb277fe700 1 -- 172.21.15.32:0/3180529704 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb14002fe0 con 0x7fdb2810d010 2021-04-09T18:30:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb277fe700 1 -- 172.21.15.32:0/3180529704 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 166019663 0 0) 0x7fdb18000ed0 con 0x7fdb2810d010 2021-04-09T18:30:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb1ffff700 1 -- 172.21.15.32:0/3180529704 >> v1:172.21.15.86:6789/0 conn(0x7fdb2810da00 legacy=0x7fdb2810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb1ffff700 1 -- 172.21.15.32:0/3180529704 >> v1:172.21.15.32:6789/0 conn(0x7fdb2810d010 legacy=0x7fdb280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.050 7fdb1ffff700 1 -- 172.21.15.32:0/3180529704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280d09c0 con 0x7fdb280d7ca0 2021-04-09T18:30:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.051 7fdb1ffff700 1 --2- 172.21.15.32:0/3180529704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280d7ca0 0x7fdb280d80a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fdb10006650 tx=0x7fdb10009a00).ready entity=mon.2 client_cookie=8fdfc092963b00f8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.051 7fdb277fe700 1 -- 172.21.15.32:0/3180529704 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb10009df0 con 0x7fdb280d7ca0 2021-04-09T18:30:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.051 7fdb277fe700 1 -- 172.21.15.32:0/3180529704 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb100009b0 con 0x7fdb280d7ca0 2021-04-09T18:30:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.051 7fdb277fe700 1 -- 172.21.15.32:0/3180529704 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb1000c2b0 con 0x7fdb280d7ca0 2021-04-09T18:30:13.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.051 7fdb2e3df700 1 -- 172.21.15.32:0/3180529704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280d7ca0 msgr2=0x7fdb280d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.051 7fdb2e3df700 1 --2- 172.21.15.32:0/3180529704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280d7ca0 0x7fdb280d80a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fdb10006650 tx=0x7fdb10009a00).stop 2021-04-09T18:30:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.052 7fdb2e3df700 1 -- 172.21.15.32:0/3180529704 shutdown_connections 2021-04-09T18:30:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.052 7fdb2e3df700 1 --2- 172.21.15.32:0/3180529704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280d7ca0 0x7fdb280d80a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.052 7fdb2e3df700 1 -- 172.21.15.32:0/3180529704 >> 172.21.15.32:0/3180529704 conn(0x7fdb280ce5f0 msgr2=0x7fdb280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.052 7fdb2e3df700 1 -- 172.21.15.32:0/3180529704 shutdown_connections 2021-04-09T18:30:13.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.052 7fdb2e3df700 1 -- 172.21.15.32:0/3180529704 wait complete. 2021-04-09T18:30:13.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2e3df700 1 Processor -- start 2021-04-09T18:30:13.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2e3df700 1 -- start start 2021-04-09T18:30:13.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2e3df700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280a7f50 0x7fdb28180be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2e3df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb28181200 0x7fdb2808ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2e3df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2808b3b0 0x7fdb281a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2e3df700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2808a890 con 0x7fdb2808b3b0 2021-04-09T18:30:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2e3df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28181640 con 0x7fdb28181200 2021-04-09T18:30:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2e3df700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281819f0 con 0x7fdb280a7f50 2021-04-09T18:30:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2c9fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2808b3b0 0x7fdb281a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2c9fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2808b3b0 0x7fdb281a32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55550/0 (socket says 172.21.15.32:55550) 2021-04-09T18:30:13.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb2c9fc700 1 -- 172.21.15.32:0/3167490487 learned_addr learned my addr 172.21.15.32:0/3167490487 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:13.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb27fff700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280a7f50 0x7fdb28180be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.073 7fdb1ffff700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb28181200 0x7fdb2808ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb2c9fc700 1 -- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280a7f50 msgr2=0x7fdb28180be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb2c9fc700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280a7f50 0x7fdb28180be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb2c9fc700 1 -- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb28181200 msgr2=0x7fdb2808ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb2c9fc700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb28181200 0x7fdb2808ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb2c9fc700 1 -- 172.21.15.32:0/3167490487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2808b7b0 con 0x7fdb2808b3b0 2021-04-09T18:30:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb2c9fc700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2808b3b0 0x7fdb281a32e0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7fdb20009940 tx=0x7fdb2000bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb257fa700 1 -- 172.21.15.32:0/3167490487 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb20000d30 con 0x7fdb2808b3b0 2021-04-09T18:30:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb257fa700 1 -- 172.21.15.32:0/3167490487 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2000e500 con 0x7fdb2808b3b0 2021-04-09T18:30:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2808b7b0 con 0x7fdb2808b3b0 2021-04-09T18:30:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb257fa700 1 -- 172.21.15.32:0/3167490487 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb2000e9e0 con 0x7fdb2808b3b0 2021-04-09T18:30:13.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.074 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2808b7b0 con 0x7fdb2808b3b0 2021-04-09T18:30:13.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.075 7fdb257fa700 1 -- 172.21.15.32:0/3167490487 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdb20001220 con 0x7fdb2808b3b0 2021-04-09T18:30:13.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.075 7fdb257fa700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb0c018930 0x7fdb0c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.075 7fdb257fa700 1 -- 172.21.15.32:0/3167490487 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fdb20034c90 con 0x7fdb2808b3b0 2021-04-09T18:30:13.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.075 7fdb27fff700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb0c018930 0x7fdb0c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.075 7fdb27fff700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb0c018930 0x7fdb0c01ae10 crc :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.093 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb281b7070 con 0x7fdb2808b3b0 2021-04-09T18:30:13.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.097 7fdb257fa700 1 -- 172.21.15.32:0/3167490487 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdb2000bdb0 con 0x7fdb2808b3b0 2021-04-09T18:30:13.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.274 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdb2808a810 con 0x7fdb2808b3b0 2021-04-09T18:30:13.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.275 7fdb257fa700 1 -- 172.21.15.32:0/3167490487 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v172) v1 ==== 84+0+64 (secure 0 0 0) 0x7fdb20001220 con 0x7fdb2808b3b0 2021-04-09T18:30:13.290 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:13.291 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:30:13.291 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:13.291 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:30:13.292 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:30:13.292 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:13.293 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:30:13.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb0c018930 msgr2=0x7fdb0c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb0c018930 0x7fdb0c01ae10 crc :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2808b3b0 msgr2=0x7fdb281a32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2808b3b0 0x7fdb281a32e0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7fdb20009940 tx=0x7fdb2000bc70).stop 2021-04-09T18:30:13.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 shutdown_connections 2021-04-09T18:30:13.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdb0c018930 0x7fdb0c01ae10 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb2808b3b0 0x7fdb281a32e0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb28181200 0x7fdb2808ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 --2- 172.21.15.32:0/3167490487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb280a7f50 0x7fdb28180be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 >> 172.21.15.32:0/3167490487 conn(0x7fdb281aa2b0 msgr2=0x7fdb281a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 shutdown_connections 2021-04-09T18:30:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.277 7fdb2e3df700 1 -- 172.21.15.32:0/3167490487 wait complete. 2021-04-09T18:30:13.303 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'alpha' 2021-04-09T18:30:13.303 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create alpha_metadata 8 2021-04-09T18:30:13.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.442 7feea19d2700 1 Processor -- start 2021-04-09T18:30:13.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.442 7feea19d2700 1 -- start start 2021-04-09T18:30:13.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7feea19d2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee9c0cffc0 0x7fee9c0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7feea19d2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0d7820 0x7fee9c0d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7feea19d2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9c0d72a0 con 0x7fee9c0d8240 2021-04-09T18:30:13.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7feea19d2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c0d9200 con 0x7fee9c0cffc0 2021-04-09T18:30:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7feea19d2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c10e000 con 0x7fee9c0d7820 2021-04-09T18:30:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9b574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fee9c0cffc0 0x7fee9c0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9bd75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fee9c0d8240 0x7fee9c0d5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36916/0 (socket says 172.21.15.32:36916) 2021-04-09T18:30:13.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9bd75700 1 -- 172.21.15.32:0/523190166 learned_addr learned my addr 172.21.15.32:0/523190166 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9ad73700 1 --2- 172.21.15.32:0/523190166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0d7820 0x7fee9c0d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9b574700 1 -- 172.21.15.32:0/523190166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0d7820 msgr2=0x7fee9c0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9b574700 1 --2- 172.21.15.32:0/523190166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0d7820 0x7fee9c0d7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9b574700 1 -- 172.21.15.32:0/523190166 >> v1:172.21.15.32:6789/0 conn(0x7fee9c0d8240 legacy=0x7fee9c0d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9a572700 1 -- 172.21.15.32:0/523190166 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097077742 0 0) 0x7fee94000d30 con 0x7fee9c0d8240 2021-04-09T18:30:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9b574700 1 -- 172.21.15.32:0/523190166 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c10e220 con 0x7fee9c0cffc0 2021-04-09T18:30:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9b574700 1 --2- 172.21.15.32:0/523190166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0cffc0 0x7fee9c0d2460 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7fee8c006650 tx=0x7fee8c009a80).ready entity=mon.0 client_cookie=4d10a836cce61a8c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:13.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9a572700 1 -- 172.21.15.32:0/523190166 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee8c0009b0 con 0x7fee9c0cffc0 2021-04-09T18:30:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9a572700 1 -- 172.21.15.32:0/523190166 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8c00c0c0 con 0x7fee9c0cffc0 2021-04-09T18:30:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7fee9a572700 1 -- 172.21.15.32:0/523190166 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee8c00c510 con 0x7fee9c0cffc0 2021-04-09T18:30:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7feea19d2700 1 -- 172.21.15.32:0/523190166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0cffc0 msgr2=0x7fee9c0d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.443 7feea19d2700 1 --2- 172.21.15.32:0/523190166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0cffc0 0x7fee9c0d2460 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7fee8c006650 tx=0x7fee8c009a80).stop 2021-04-09T18:30:13.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.444 7feea19d2700 1 -- 172.21.15.32:0/523190166 shutdown_connections 2021-04-09T18:30:13.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.444 7feea19d2700 1 --2- 172.21.15.32:0/523190166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0d7820 0x7fee9c0d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.444 7feea19d2700 1 --2- 172.21.15.32:0/523190166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0cffc0 0x7fee9c0d2460 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.444 7feea19d2700 1 -- 172.21.15.32:0/523190166 >> 172.21.15.32:0/523190166 conn(0x7fee9c0ce5e0 msgr2=0x7fee9c0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:13.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.444 7feea19d2700 1 -- 172.21.15.32:0/523190166 shutdown_connections 2021-04-09T18:30:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.444 7feea19d2700 1 -- 172.21.15.32:0/523190166 wait complete. 2021-04-09T18:30:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.464 7feea19d2700 1 Processor -- start 2021-04-09T18:30:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.464 7feea19d2700 1 -- start start 2021-04-09T18:30:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7feea19d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0b89a0 0x7fee9c0c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7feea19d2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee9c0b8da0 0x7fee9c0be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7feea19d2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0b91a0 0x7fee9c0bead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7feea19d2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0c39b0 con 0x7fee9c0b89a0 2021-04-09T18:30:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7feea19d2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0c0d60 con 0x7fee9c0b8da0 2021-04-09T18:30:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7feea19d2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0bf0f0 con 0x7fee9c0b91a0 2021-04-09T18:30:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0b89a0 0x7fee9c0c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0b89a0 0x7fee9c0c31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55562/0 (socket says 172.21.15.32:55562) 2021-04-09T18:30:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9b574700 1 -- 172.21.15.32:0/1466488565 learned_addr learned my addr 172.21.15.32:0/1466488565 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9bd75700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0b91a0 0x7fee9c0bead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9ad73700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee9c0b8da0 0x7fee9c0be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9b574700 1 -- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0b91a0 msgr2=0x7fee9c0bead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9b574700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0b91a0 0x7fee9c0bead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9b574700 1 -- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee9c0b8da0 msgr2=0x7fee9c0be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:13.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9b574700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee9c0b8da0 0x7fee9c0be4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:13.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.465 7fee9b574700 1 -- 172.21.15.32:0/1466488565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c0bf310 con 0x7fee9c0b89a0 2021-04-09T18:30:13.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.466 7fee9b574700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0b89a0 0x7fee9c0c31a0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7fee8c009650 tx=0x7fee8c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:13.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.466 7fee93fff700 1 -- 172.21.15.32:0/1466488565 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee8c0093a0 con 0x7fee9c0b89a0 2021-04-09T18:30:13.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.466 7fee93fff700 1 -- 172.21.15.32:0/1466488565 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8c00bd50 con 0x7fee9c0b89a0 2021-04-09T18:30:13.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.466 7feea19d2700 1 -- 172.21.15.32:0/1466488565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9c0bf310 con 0x7fee9c0b89a0 2021-04-09T18:30:13.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.466 7fee93fff700 1 -- 172.21.15.32:0/1466488565 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee8c0052e0 con 0x7fee9c0b89a0 2021-04-09T18:30:13.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.466 7feea19d2700 1 -- 172.21.15.32:0/1466488565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c0bf310 con 0x7fee9c0b89a0 2021-04-09T18:30:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.466 7fee93fff700 1 -- 172.21.15.32:0/1466488565 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fee8c006030 con 0x7fee9c0b89a0 2021-04-09T18:30:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.466 7fee93fff700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee78018930 0x7fee7801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.466 7fee93fff700 1 -- 172.21.15.32:0/1466488565 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fee8c0326b0 con 0x7fee9c0b89a0 2021-04-09T18:30:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.467 7fee9ad73700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee78018930 0x7fee7801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:13.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.467 7fee9ad73700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee78018930 0x7fee7801ae10 crc :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:13.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.487 7feea19d2700 1 -- 172.21.15.32:0/1466488565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9c1b70c0 con 0x7fee9c0b89a0 2021-04-09T18:30:13.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.490 7fee93fff700 1 -- 172.21.15.32:0/1466488565 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fee8c0009c0 con 0x7fee9c0b89a0 2021-04-09T18:30:13.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:13.673 7feea19d2700 1 -- 172.21.15.32:0/1466488565 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_metadata"} v 0) v1 -- 0x7fee9c0d72a0 con 0x7fee9c0b89a0 2021-04-09T18:30:14.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.058 7fee93fff700 1 -- 172.21.15.32:0/1466488565 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_metadata"}]=0 pool 'alpha_metadata' created v173) v1 ==== 131+0+0 (secure 0 0 0) 0x7fee8c006030 con 0x7fee9c0b89a0 2021-04-09T18:30:14.102 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_metadata' created 2021-04-09T18:30:14.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 -- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee78018930 msgr2=0x7fee7801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:14.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee78018930 0x7fee7801ae10 crc :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:14.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 -- 172.21.15.32:0/1466488565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0b89a0 msgr2=0x7fee9c0c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:14.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0b89a0 0x7fee9c0c31a0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7fee8c009650 tx=0x7fee8c00b960).stop 2021-04-09T18:30:14.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 -- 172.21.15.32:0/1466488565 shutdown_connections 2021-04-09T18:30:14.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee78018930 0x7fee7801ae10 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:14.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee9c0b91a0 0x7fee9c0bead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:14.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee9c0b8da0 0x7fee9c0be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:14.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 --2- 172.21.15.32:0/1466488565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee9c0b89a0 0x7fee9c0c31a0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.089 7feea19d2700 1 -- 172.21.15.32:0/1466488565 >> 172.21.15.32:0/1466488565 conn(0x7fee9c0a06a0 msgr2=0x7fee9c08fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.090 7feea19d2700 1 -- 172.21.15.32:0/1466488565 shutdown_connections 2021-04-09T18:30:14.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.090 7feea19d2700 1 -- 172.21.15.32:0/1466488565 wait complete. 2021-04-09T18:30:14.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create alpha_data 8 2021-04-09T18:30:14.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.253 7f22686aa700 1 Processor -- start 2021-04-09T18:30:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.253 7f22686aa700 1 -- start start 2021-04-09T18:30:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.254 7f22686aa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22600d7ca0 0x7f22600d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.254 7f22686aa700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226010d480 con 0x7f226010d010 2021-04-09T18:30:14.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.254 7f22686aa700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22600d86c0 con 0x7f226010da00 2021-04-09T18:30:14.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.254 7f22686aa700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22600d9220 con 0x7f22600d7ca0 2021-04-09T18:30:14.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.254 7f2265cc5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22600d7ca0 0x7f22600d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:14.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.254 7f2265cc5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f22600d7ca0 0x7f22600d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55572/0 (socket says 172.21.15.32:55572) 2021-04-09T18:30:14.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.254 7f22664c6700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f226010d010 0x7f22600d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36928/0 (socket says 172.21.15.32:36928) 2021-04-09T18:30:14.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.254 7f2265cc5700 1 -- 172.21.15.32:0/4283493994 learned_addr learned my addr 172.21.15.32:0/4283493994 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:14.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.255 7f2265cc5700 1 -- 172.21.15.32:0/4283493994 >> v1:172.21.15.86:6789/0 conn(0x7f226010da00 legacy=0x7f226010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:14.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.255 7f2265cc5700 1 -- 172.21.15.32:0/4283493994 >> v1:172.21.15.32:6789/0 conn(0x7f226010d010 legacy=0x7f22600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:14.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.255 7f2265cc5700 1 -- 172.21.15.32:0/4283493994 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22600d1f70 con 0x7f22600d7ca0 2021-04-09T18:30:14.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.255 7f2265cc5700 1 --2- 172.21.15.32:0/4283493994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600d7ca0 0x7f22600d80a0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f2250006700 tx=0x7f2250009b80).ready entity=mon.0 client_cookie=f876ee092e337c3f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:14.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.256 7f22654c4700 1 -- 172.21.15.32:0/4283493994 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f225000c0c0 con 0x7f22600d7ca0 2021-04-09T18:30:14.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.256 7f22654c4700 1 -- 172.21.15.32:0/4283493994 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225000c440 con 0x7f22600d7ca0 2021-04-09T18:30:14.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.257 7f22654c4700 1 -- 172.21.15.32:0/4283493994 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f225000c920 con 0x7f22600d7ca0 2021-04-09T18:30:14.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.257 7f22686aa700 1 -- 172.21.15.32:0/4283493994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600d7ca0 msgr2=0x7f22600d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:14.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.257 7f22686aa700 1 --2- 172.21.15.32:0/4283493994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600d7ca0 0x7f22600d80a0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f2250006700 tx=0x7f2250009b80).stop 2021-04-09T18:30:14.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.258 7f22686aa700 1 -- 172.21.15.32:0/4283493994 shutdown_connections 2021-04-09T18:30:14.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.258 7f22686aa700 1 --2- 172.21.15.32:0/4283493994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22600d7ca0 0x7f22600d80a0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:14.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.258 7f22686aa700 1 -- 172.21.15.32:0/4283493994 >> 172.21.15.32:0/4283493994 conn(0x7f22600ce5f0 msgr2=0x7f22600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:14.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.258 7f22686aa700 1 -- 172.21.15.32:0/4283493994 shutdown_connections 2021-04-09T18:30:14.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.258 7f22686aa700 1 -- 172.21.15.32:0/4283493994 wait complete. 2021-04-09T18:30:14.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22686aa700 1 Processor -- start 2021-04-09T18:30:14.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22686aa700 1 -- start start 2021-04-09T18:30:14.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22686aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22601c2850 0x7f22600bc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:14.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22686aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226008cc40 0x7f226008d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:14.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22686aa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22601a5680 0x7f22601a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22686aa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22600bc8a0 con 0x7f22601c2850 2021-04-09T18:30:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22686aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226008d660 con 0x7f226008cc40 2021-04-09T18:30:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22686aa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22601a60a0 con 0x7f22601a5680 2021-04-09T18:30:14.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22664c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22601c2850 0x7f22600bc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22664c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22601c2850 0x7f22600bc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55576/0 (socket says 172.21.15.32:55576) 2021-04-09T18:30:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.279 7f22664c6700 1 -- 172.21.15.32:0/788501392 learned_addr learned my addr 172.21.15.32:0/788501392 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f2265cc5700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226008cc40 0x7f226008d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:14.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f2266cc7700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22601a5680 0x7f22601a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f22664c6700 1 -- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22601a5680 msgr2=0x7f22601a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f22664c6700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22601a5680 0x7f22601a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f22664c6700 1 -- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226008cc40 msgr2=0x7f226008d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:14.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f22664c6700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226008cc40 0x7f226008d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:14.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f22664c6700 1 -- 172.21.15.32:0/788501392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22601a6490 con 0x7f22601c2850 2021-04-09T18:30:14.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f22664c6700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22601c2850 0x7f22600bc280 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f225c0014d0 tx=0x7f225c00a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:14.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f2256ffd700 1 -- 172.21.15.32:0/788501392 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f225c000ef0 con 0x7f22601c2850 2021-04-09T18:30:14.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f2256ffd700 1 -- 172.21.15.32:0/788501392 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c00ac90 con 0x7f22601c2850 2021-04-09T18:30:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f22686aa700 1 -- 172.21.15.32:0/788501392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22601a6490 con 0x7f22601c2850 2021-04-09T18:30:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f2256ffd700 1 -- 172.21.15.32:0/788501392 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f225c004590 con 0x7f22601c2850 2021-04-09T18:30:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.280 7f22686aa700 1 -- 172.21.15.32:0/788501392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22601a6490 con 0x7f22601c2850 2021-04-09T18:30:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.281 7f2256ffd700 1 -- 172.21.15.32:0/788501392 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f225c008000 con 0x7f22601c2850 2021-04-09T18:30:14.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.281 7f2256ffd700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2248018930 0x7f224801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:14.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.281 7f2256ffd700 1 -- 172.21.15.32:0/788501392 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5110+0+0 (secure 0 0 0) 0x7f225c032880 con 0x7f22601c2850 2021-04-09T18:30:14.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.281 7f2265cc5700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2248018930 0x7f224801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:14.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.281 7f2265cc5700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2248018930 0x7f224801ae10 crc :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:14.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.298 7f22686aa700 1 -- 172.21.15.32:0/788501392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22601a4770 con 0x7f22601c2850 2021-04-09T18:30:14.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.302 7f2256ffd700 1 -- 172.21.15.32:0/788501392 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f225c0043a0 con 0x7f22601c2850 2021-04-09T18:30:14.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:14.487 7f22686aa700 1 -- 172.21.15.32:0/788501392 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_data"} v 0) v1 -- 0x7f2260112070 con 0x7f22601c2850 2021-04-09T18:30:15.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.349 7f2256ffd700 1 -- 172.21.15.32:0/788501392 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_data"}]=0 pool 'alpha_data' created v178) v1 ==== 123+0+0 (secure 0 0 0) 0x7f225c008000 con 0x7f22601c2850 2021-04-09T18:30:15.368 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_data' created 2021-04-09T18:30:15.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 -- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2248018930 msgr2=0x7f224801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2248018930 0x7f224801ae10 crc :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 -- 172.21.15.32:0/788501392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22601c2850 msgr2=0x7f22600bc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22601c2850 0x7f22600bc280 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f225c0014d0 tx=0x7f225c00a8a0).stop 2021-04-09T18:30:15.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 -- 172.21.15.32:0/788501392 shutdown_connections 2021-04-09T18:30:15.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2248018930 0x7f224801ae10 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22601a5680 0x7f22601a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226008cc40 0x7f226008d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 --2- 172.21.15.32:0/788501392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22601c2850 0x7f22600bc280 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 -- 172.21.15.32:0/788501392 >> 172.21.15.32:0/788501392 conn(0x7f22600bca90 msgr2=0x7f22600b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:15.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 -- 172.21.15.32:0/788501392 shutdown_connections 2021-04-09T18:30:15.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.356 7f22686aa700 1 -- 172.21.15.32:0/788501392 wait complete. 2021-04-09T18:30:15.382 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new alpha alpha_metadata alpha_data --force 2021-04-09T18:30:15.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.531 7f6f63143700 1 Processor -- start 2021-04-09T18:30:15.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.531 7f6f63143700 1 -- start start 2021-04-09T18:30:15.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f63143700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f5c10d050 0x7f6f5c0cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f63143700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0d0160 0x7f6f5c0d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f63143700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5c10d500 con 0x7f6f5c0d82e0 2021-04-09T18:30:15.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f63143700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f5c0d9260 con 0x7f6f5c10d050 2021-04-09T18:30:15.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f63143700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0d86e0 con 0x7f6f5c0d0160 2021-04-09T18:30:15.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f60f5f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f5c10d050 0x7f6f5c0cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f60f5f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f5c10d050 0x7f6f5c0cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55588/0 (socket says 172.21.15.32:55588) 2021-04-09T18:30:15.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f60f5f700 1 -- 172.21.15.32:0/2073402986 learned_addr learned my addr 172.21.15.32:0/2073402986 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:15.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f5bfff700 1 --2- 172.21.15.32:0/2073402986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0d0160 0x7f6f5c0d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f60f5f700 1 -- 172.21.15.32:0/2073402986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0d0160 msgr2=0x7f6f5c0d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:15.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f60f5f700 1 --2- 172.21.15.32:0/2073402986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0d0160 0x7f6f5c0d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f60f5f700 1 -- 172.21.15.32:0/2073402986 >> v1:172.21.15.86:6789/0 conn(0x7f6f5c0d82e0 legacy=0x7f6f5c10da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f60f5f700 1 -- 172.21.15.32:0/2073402986 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c10e270 con 0x7f6f5c10d050 2021-04-09T18:30:15.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.532 7f6f60f5f700 1 --2- 172.21.15.32:0/2073402986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c10d050 0x7f6f5c0cfb40 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f6f44006700 tx=0x7f6f44009b80).ready entity=mon.0 client_cookie=d29b3ae25004973 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:15.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f5b7fe700 1 -- 172.21.15.32:0/2073402986 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f4400c0c0 con 0x7f6f5c10d050 2021-04-09T18:30:15.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f5b7fe700 1 -- 172.21.15.32:0/2073402986 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4400c440 con 0x7f6f5c10d050 2021-04-09T18:30:15.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f5b7fe700 1 -- 172.21.15.32:0/2073402986 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f4400c920 con 0x7f6f5c10d050 2021-04-09T18:30:15.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f63143700 1 -- 172.21.15.32:0/2073402986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c10d050 msgr2=0x7f6f5c0cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f63143700 1 --2- 172.21.15.32:0/2073402986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c10d050 0x7f6f5c0cfb40 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f6f44006700 tx=0x7f6f44009b80).stop 2021-04-09T18:30:15.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f63143700 1 -- 172.21.15.32:0/2073402986 shutdown_connections 2021-04-09T18:30:15.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f63143700 1 --2- 172.21.15.32:0/2073402986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0d0160 0x7f6f5c0d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f63143700 1 --2- 172.21.15.32:0/2073402986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c10d050 0x7f6f5c0cfb40 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f63143700 1 -- 172.21.15.32:0/2073402986 >> 172.21.15.32:0/2073402986 conn(0x7f6f5c0ce620 msgr2=0x7f6f5c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:15.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f63143700 1 -- 172.21.15.32:0/2073402986 shutdown_connections 2021-04-09T18:30:15.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.533 7f6f63143700 1 -- 172.21.15.32:0/2073402986 wait complete. 2021-04-09T18:30:15.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.553 7f6f63143700 1 Processor -- start 2021-04-09T18:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f63143700 1 -- start start 2021-04-09T18:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f63143700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f5c0afb00 0x7f6f5c0aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f63143700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c0b3240 0x7f6f5c0ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f63143700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0ad6d0 0x7f6f5c0abca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f63143700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0acbb0 con 0x7f6f5c0b3240 2021-04-09T18:30:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f63143700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0adad0 con 0x7f6f5c0afb00 2021-04-09T18:30:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f63143700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0b3640 con 0x7f6f5c0ad6d0 2021-04-09T18:30:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f5bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c0b3240 0x7f6f5c0ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f5bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c0b3240 0x7f6f5c0ad0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55596/0 (socket says 172.21.15.32:55596) 2021-04-09T18:30:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f5bfff700 1 -- 172.21.15.32:0/2024205492 learned_addr learned my addr 172.21.15.32:0/2024205492 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f60f5f700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f5c0afb00 0x7f6f5c0aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.554 7f6f61760700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0ad6d0 0x7f6f5c0abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f5bfff700 1 -- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0ad6d0 msgr2=0x7f6f5c0abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f5bfff700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0ad6d0 0x7f6f5c0abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f5bfff700 1 -- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f5c0afb00 msgr2=0x7f6f5c0aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f5bfff700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f5c0afb00 0x7f6f5c0aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f5bfff700 1 -- 172.21.15.32:0/2024205492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c0ac2c0 con 0x7f6f5c0b3240 2021-04-09T18:30:15.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f5bfff700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c0b3240 0x7f6f5c0ad0b0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7f6f4c00b530 tx=0x7f6f4c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:15.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f597fa700 1 -- 172.21.15.32:0/2024205492 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f4c00bcc0 con 0x7f6f5c0b3240 2021-04-09T18:30:15.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f597fa700 1 -- 172.21.15.32:0/2024205492 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4c002030 con 0x7f6f5c0b3240 2021-04-09T18:30:15.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f63143700 1 -- 172.21.15.32:0/2024205492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5c0ac2c0 con 0x7f6f5c0b3240 2021-04-09T18:30:15.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f597fa700 1 -- 172.21.15.32:0/2024205492 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f4c002480 con 0x7f6f5c0b3240 2021-04-09T18:30:15.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.555 7f6f63143700 1 -- 172.21.15.32:0/2024205492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5c0ac2c0 con 0x7f6f5c0b3240 2021-04-09T18:30:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.556 7f6f597fa700 1 -- 172.21.15.32:0/2024205492 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6f4c002210 con 0x7f6f5c0b3240 2021-04-09T18:30:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.556 7f6f597fa700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.556 7f6f597fa700 1 -- 172.21.15.32:0/2024205492 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5459+0+0 (secure 0 0 0) 0x7f6f4c032980 con 0x7f6f5c0b3240 2021-04-09T18:30:15.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.556 7f6f60f5f700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.556 7f6f60f5f700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 crc :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:15.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.576 7f6f63143700 1 -- 172.21.15.32:0/2024205492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c1b7130 con 0x7f6f5c0b3240 2021-04-09T18:30:15.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.580 7f6f597fa700 1 -- 172.21.15.32:0/2024205492 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6f4c002240 con 0x7f6f5c0b3240 2021-04-09T18:30:15.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.762 7f6f63143700 1 -- 172.21.15.32:0/2024205492 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "alpha_data", "prefix": "fs new", "force": true, "fs_name": "alpha", "metadata": "alpha_metadata"} v 0) v1 -- 0x7f6f5c0f3ae0 con 0x7f6f5c0b3240 2021-04-09T18:30:15.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.766 7f6f597fa700 1 -- 172.21.15.32:0/2024205492 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "alpha_data", "prefix": "fs new", "force": true, "fs_name": "alpha", "metadata": "alpha_metadata"}]=0 new fs with metadata pool 30 and data pool 31 v322) v1 ==== 186+0+0 (secure 0 0 0) 0x7f6f4c000c90 con 0x7f6f5c0b3240 2021-04-09T18:30:15.787 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 30 and data pool 31 2021-04-09T18:30:15.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.773 7f6f63143700 1 -- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 msgr2=0x7f6f4001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.773 7f6f63143700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 crc :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.773 7f6f63143700 1 -- 172.21.15.32:0/2024205492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c0b3240 msgr2=0x7f6f5c0ad0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.773 7f6f63143700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c0b3240 0x7f6f5c0ad0b0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7f6f4c00b530 tx=0x7f6f4c00b8d0).stop 2021-04-09T18:30:15.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.774 7f6f63143700 1 -- 172.21.15.32:0/2024205492 shutdown_connections 2021-04-09T18:30:15.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.774 7f6f63143700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f40018930 0x7f6f4001ae10 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.774 7f6f63143700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f5c0b3240 0x7f6f5c0ad0b0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.774 7f6f63143700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f5c0ad6d0 0x7f6f5c0abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.774 7f6f63143700 1 --2- 172.21.15.32:0/2024205492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f5c0afb00 0x7f6f5c0aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.774 7f6f63143700 1 -- 172.21.15.32:0/2024205492 >> 172.21.15.32:0/2024205492 conn(0x7f6f5c1b7d20 msgr2=0x7f6f5c1b81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:15.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.775 7f6f63143700 1 -- 172.21.15.32:0/2024205492 shutdown_connections 2021-04-09T18:30:15.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.775 7f6f63143700 1 -- 172.21.15.32:0/2024205492 wait complete. 2021-04-09T18:30:15.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:15.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.936 7fad3c89a700 1 Processor -- start 2021-04-09T18:30:15.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.936 7fad3c89a700 1 -- start start 2021-04-09T18:30:15.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad3c89a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad380d7ca0 0x7fad380d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad3c89a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fad3810da00 0x7fad3810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad3c89a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3810d480 con 0x7fad3810d010 2021-04-09T18:30:15.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad3c89a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad380d9220 con 0x7fad3810da00 2021-04-09T18:30:15.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad3c89a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380d86c0 con 0x7fad380d7ca0 2021-04-09T18:30:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad3661c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fad3810d010 0x7fad380d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36972/0 (socket says 172.21.15.32:36972) 2021-04-09T18:30:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad3661c700 1 -- 172.21.15.32:0/1190253260 learned_addr learned my addr 172.21.15.32:0/1190253260 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad36e1d700 1 --2- 172.21.15.32:0/1190253260 >> v2:172.21.15.86:3300/0 conn(0x7fad3810da00 0x7fad3810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad35e1b700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad380d7ca0 0x7fad380d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad3561a700 1 -- 172.21.15.32:0/1190253260 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228894558 0 0) 0x7fad28000ce0 con 0x7fad3810d010 2021-04-09T18:30:15.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.937 7fad3561a700 1 -- 172.21.15.32:0/1190253260 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad20002fe0 con 0x7fad3810d010 2021-04-09T18:30:15.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 796578401 0 0) 0x7fad280010e0 con 0x7fad3810d010 2021-04-09T18:30:15.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad380d7ca0 msgr2=0x7fad380d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:15.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad380d7ca0 0x7fad380d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 >> v2:172.21.15.86:3300/0 conn(0x7fad3810da00 msgr2=0x7fad3810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:15.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 --2- 172.21.15.32:0/1190253260 >> v2:172.21.15.86:3300/0 conn(0x7fad3810da00 0x7fad3810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad380d1f70 con 0x7fad3810d010 2021-04-09T18:30:15.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fad28001600 con 0x7fad3810d010 2021-04-09T18:30:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 >> v1:172.21.15.32:6789/0 conn(0x7fad3810d010 legacy=0x7fad380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad20003a60 0x7fad20005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2004a780 0x7fad2004cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad2004d5a0 0x7fad2004fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad200502e0 con 0x7fad2004a780 2021-04-09T18:30:15.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad20050500 con 0x7fad20003a60 2021-04-09T18:30:15.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad20050720 con 0x7fad2004d5a0 2021-04-09T18:30:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad35e1b700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2004a780 0x7fad2004cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3661c700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad20003a60 0x7fad20005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad36e1d700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad2004d5a0 0x7fad2004fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad35e1b700 1 -- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad2004d5a0 msgr2=0x7fad2004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad35e1b700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad2004d5a0 0x7fad2004fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad35e1b700 1 -- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad20003a60 msgr2=0x7fad20005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:15.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad35e1b700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad20003a60 0x7fad20005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad35e1b700 1 -- 172.21.15.32:0/1190253260 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad200509c0 con 0x7fad2004a780 2021-04-09T18:30:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3661c700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad20003a60 0x7fad20005ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:15.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad35e1b700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2004a780 0x7fad2004cca0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7fad2c006ac0 tx=0x7fad2c009ab0).ready entity=mon.0 client_cookie=78cd5a8980b24c67 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:15.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad2c00c0c0 con 0x7fad2004a780 2021-04-09T18:30:15.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c00c3c0 con 0x7fad2004a780 2021-04-09T18:30:15.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3561a700 1 -- 172.21.15.32:0/1190253260 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad2c00c8a0 con 0x7fad2004a780 2021-04-09T18:30:15.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3c89a700 1 -- 172.21.15.32:0/1190253260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2004a780 msgr2=0x7fad2004cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.938 7fad3c89a700 1 --2- 172.21.15.32:0/1190253260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2004a780 0x7fad2004cca0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7fad2c006ac0 tx=0x7fad2c009ab0).stop 2021-04-09T18:30:15.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.939 7fad36e1d700 1 -- 172.21.15.32:0/1190253260 reap_dead start 2021-04-09T18:30:15.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.939 7fad3c89a700 1 -- 172.21.15.32:0/1190253260 shutdown_connections 2021-04-09T18:30:15.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.939 7fad3c89a700 1 -- 172.21.15.32:0/1190253260 >> 172.21.15.32:0/1190253260 conn(0x7fad380ce5f0 msgr2=0x7fad380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:15.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.939 7fad3c89a700 1 -- 172.21.15.32:0/1190253260 shutdown_connections 2021-04-09T18:30:15.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.939 7fad3c89a700 1 -- 172.21.15.32:0/1190253260 wait complete. 2021-04-09T18:30:15.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.959 7fad3c89a700 1 Processor -- start 2021-04-09T18:30:15.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.959 7fad3c89a700 1 -- start start 2021-04-09T18:30:15.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad3c89a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad381889c0 0x7fad3809a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad3c89a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad3818e2c0 0x7fad381aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad3c89a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad381aaad0 0x7fad380a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:15.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad3c89a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381aaed0 con 0x7fad381aaad0 2021-04-09T18:30:15.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad3c89a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381ab0f0 con 0x7fad381889c0 2021-04-09T18:30:15.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad3c89a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380a3ed0 con 0x7fad3818e2c0 2021-04-09T18:30:15.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad36e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad381aaad0 0x7fad380a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad36e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad381aaad0 0x7fad380a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55630/0 (socket says 172.21.15.32:55630) 2021-04-09T18:30:15.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad36e1d700 1 -- 172.21.15.32:0/987703826 learned_addr learned my addr 172.21.15.32:0/987703826 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:15.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad35e1b700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad3818e2c0 0x7fad381aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad3661c700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad381889c0 0x7fad3809a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:15.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad36e1d700 1 -- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad3818e2c0 msgr2=0x7fad381aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad36e1d700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad3818e2c0 0x7fad381aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad36e1d700 1 -- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad381889c0 msgr2=0x7fad3809a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:15.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad36e1d700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad381889c0 0x7fad3809a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:15.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad36e1d700 1 -- 172.21.15.32:0/987703826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad380a40f0 con 0x7fad381aaad0 2021-04-09T18:30:15.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad35e1b700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad3818e2c0 0x7fad381aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:30:15.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.960 7fad3661c700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad381889c0 0x7fad3809a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:30:15.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.961 7fad36e1d700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad381aaad0 0x7fad380a38b0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7fad300099e0 tx=0x7fad3000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:15.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.961 7fad26ffd700 1 -- 172.21.15.32:0/987703826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad30001400 con 0x7fad381aaad0 2021-04-09T18:30:16.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.961 7fad26ffd700 1 -- 172.21.15.32:0/987703826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3000be60 con 0x7fad381aaad0 2021-04-09T18:30:16.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.961 7fad3c89a700 1 -- 172.21.15.32:0/987703826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad380a40f0 con 0x7fad381aaad0 2021-04-09T18:30:16.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.961 7fad26ffd700 1 -- 172.21.15.32:0/987703826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad30001700 con 0x7fad381aaad0 2021-04-09T18:30:16.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.961 7fad3c89a700 1 -- 172.21.15.32:0/987703826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad380a40f0 con 0x7fad381aaad0 2021-04-09T18:30:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.961 7fad26ffd700 1 -- 172.21.15.32:0/987703826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fad300011f0 con 0x7fad381aaad0 2021-04-09T18:30:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.961 7fad26ffd700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad1c018930 0x7fad1c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.961 7fad26ffd700 1 -- 172.21.15.32:0/987703826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad30032e30 con 0x7fad381aaad0 2021-04-09T18:30:16.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.962 7fad3661c700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad1c018930 0x7fad1c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.962 7fad3661c700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad1c018930 0x7fad1c01ae10 crc :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:16.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.981 7fad3c89a700 1 -- 172.21.15.32:0/987703826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3810d480 con 0x7fad381aaad0 2021-04-09T18:30:16.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:15.985 7fad26ffd700 1 -- 172.21.15.32:0/987703826 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fad30001660 con 0x7fad381aaad0 2021-04-09T18:30:16.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.174 7fad3c89a700 1 -- 172.21.15.32:0/987703826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad3810d480 con 0x7fad381aaad0 2021-04-09T18:30:16.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.175 7fad26ffd700 1 -- 172.21.15.32:0/987703826 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+14016 (secure 0 0 0) 0x7fad300011f0 con 0x7fad381aaad0 2021-04-09T18:30:16.191 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:16.191 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":179,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:15.764180","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:30:13.709244","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":31,"pool_name":"alpha_data","create_time":"2021-04-09 18:30:14.523869","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:16.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 -- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad1c018930 msgr2=0x7fad1c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad1c018930 0x7fad1c01ae10 crc :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 -- 172.21.15.32:0/987703826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad381aaad0 msgr2=0x7fad380a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad381aaad0 0x7fad380a38b0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7fad300099e0 tx=0x7fad3000bd20).stop 2021-04-09T18:30:16.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 -- 172.21.15.32:0/987703826 shutdown_connections 2021-04-09T18:30:16.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad1c018930 0x7fad1c01ae10 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad381aaad0 0x7fad380a38b0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad3818e2c0 0x7fad381aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 --2- 172.21.15.32:0/987703826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad381889c0 0x7fad3809a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.177 7fad3c89a700 1 -- 172.21.15.32:0/987703826 >> 172.21.15.32:0/987703826 conn(0x7fad381a0db0 msgr2=0x7fad381a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:16.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.178 7fad3c89a700 1 -- 172.21.15.32:0/987703826 shutdown_connections 2021-04-09T18:30:16.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.178 7fad3c89a700 1 -- 172.21.15.32:0/987703826 wait complete. 2021-04-09T18:30:16.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.339 7fadcd84a700 1 Processor -- start 2021-04-09T18:30:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.339 7fadcd84a700 1 -- start start 2021-04-09T18:30:16.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadcd84a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc810d010 0x7fadc80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadcd84a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fadc80d1da0 0x7fadc80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadcd84a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc810d480 con 0x7fadc80d82c0 2021-04-09T18:30:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadcd84a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc810e780 con 0x7fadc80d1da0 2021-04-09T18:30:16.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadcd84a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80d8d30 con 0x7fadc810d010 2021-04-09T18:30:16.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc6d73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fadc80d1da0 0x7fadc80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc7d75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fadc80d82c0 0x7fadc810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36994/0 (socket says 172.21.15.32:36994) 2021-04-09T18:30:16.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc7d75700 1 -- 172.21.15.32:0/2586391674 learned_addr learned my addr 172.21.15.32:0/2586391674 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:16.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc7574700 1 --2- 172.21.15.32:0/2586391674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc810d010 0x7fadc80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc6d73700 1 -- 172.21.15.32:0/2586391674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc810d010 msgr2=0x7fadc80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc6d73700 1 --2- 172.21.15.32:0/2586391674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc810d010 0x7fadc80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc6d73700 1 -- 172.21.15.32:0/2586391674 >> v1:172.21.15.32:6789/0 conn(0x7fadc80d82c0 legacy=0x7fadc810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc6d73700 1 -- 172.21.15.32:0/2586391674 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc80d9220 con 0x7fadc80d1da0 2021-04-09T18:30:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc6572700 1 -- 172.21.15.32:0/2586391674 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464648758 0 0) 0x7fadc0000d30 con 0x7fadc80d82c0 2021-04-09T18:30:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.340 7fadc6d73700 1 --2- 172.21.15.32:0/2586391674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc80d1da0 0x7fadc80d7ca0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7fadbc006650 tx=0x7fadbc009a00).ready entity=mon.0 client_cookie=d5f49f1386206217 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadc6572700 1 -- 172.21.15.32:0/2586391674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadbc009df0 con 0x7fadc80d1da0 2021-04-09T18:30:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadc6572700 1 -- 172.21.15.32:0/2586391674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc0009b0 con 0x7fadc80d1da0 2021-04-09T18:30:16.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadc6572700 1 -- 172.21.15.32:0/2586391674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadbc00c2b0 con 0x7fadc80d1da0 2021-04-09T18:30:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadcd84a700 1 -- 172.21.15.32:0/2586391674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc80d1da0 msgr2=0x7fadc80d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadcd84a700 1 --2- 172.21.15.32:0/2586391674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc80d1da0 0x7fadc80d7ca0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7fadbc006650 tx=0x7fadbc009a00).stop 2021-04-09T18:30:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadcd84a700 1 -- 172.21.15.32:0/2586391674 shutdown_connections 2021-04-09T18:30:16.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadcd84a700 1 --2- 172.21.15.32:0/2586391674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc80d1da0 0x7fadc80d7ca0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadcd84a700 1 --2- 172.21.15.32:0/2586391674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc810d010 0x7fadc80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadcd84a700 1 -- 172.21.15.32:0/2586391674 >> 172.21.15.32:0/2586391674 conn(0x7fadc80ce5f0 msgr2=0x7fadc80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadcd84a700 1 -- 172.21.15.32:0/2586391674 shutdown_connections 2021-04-09T18:30:16.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.341 7fadcd84a700 1 -- 172.21.15.32:0/2586391674 wait complete. 2021-04-09T18:30:16.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadcd84a700 1 Processor -- start 2021-04-09T18:30:16.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadcd84a700 1 -- start start 2021-04-09T18:30:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadcd84a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc81a5210 0x7fadc8185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadcd84a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadc8185a70 0x7fadc80acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadcd84a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc8185e70 0x7fadc80ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadcd84a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc818f7b0 con 0x7fadc81a5210 2021-04-09T18:30:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadcd84a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80ad7f0 con 0x7fadc8185a70 2021-04-09T18:30:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadcd84a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80ada10 con 0x7fadc8185e70 2021-04-09T18:30:16.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadc7574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc81a5210 0x7fadc8185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadc7574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc81a5210 0x7fadc8185450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55640/0 (socket says 172.21.15.32:55640) 2021-04-09T18:30:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.362 7fadc7574700 1 -- 172.21.15.32:0/140952646 learned_addr learned my addr 172.21.15.32:0/140952646 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadc6d73700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadc8185a70 0x7fadc80acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadc7d75700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc8185e70 0x7fadc80ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadc7574700 1 -- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc8185e70 msgr2=0x7fadc80ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadc7574700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc8185e70 0x7fadc80ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadc7574700 1 -- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadc8185a70 msgr2=0x7fadc80acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadc7574700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadc8185a70 0x7fadc80acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadc7574700 1 -- 172.21.15.32:0/140952646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc8179610 con 0x7fadc81a5210 2021-04-09T18:30:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadc6d73700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadc8185a70 0x7fadc80acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadc7574700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc81a5210 0x7fadc8185450 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7fadb8006f60 tx=0x7fadb800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadaffff700 1 -- 172.21.15.32:0/140952646 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadb800bd60 con 0x7fadc81a5210 2021-04-09T18:30:16.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadaffff700 1 -- 172.21.15.32:0/140952646 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb8000c90 con 0x7fadc81a5210 2021-04-09T18:30:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadcd84a700 1 -- 172.21.15.32:0/140952646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc8179610 con 0x7fadc81a5210 2021-04-09T18:30:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadaffff700 1 -- 172.21.15.32:0/140952646 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadb8002220 con 0x7fadc81a5210 2021-04-09T18:30:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.363 7fadcd84a700 1 -- 172.21.15.32:0/140952646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc8179610 con 0x7fadc81a5210 2021-04-09T18:30:16.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.364 7fadaffff700 1 -- 172.21.15.32:0/140952646 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fadb80052e0 con 0x7fadc81a5210 2021-04-09T18:30:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.364 7fadaffff700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fada8018930 0x7fada801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.364 7fadaffff700 1 -- 172.21.15.32:0/140952646 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fadb80329f0 con 0x7fadc81a5210 2021-04-09T18:30:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.364 7fadc6d73700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fada8018930 0x7fada801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.364 7fadc6d73700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fada8018930 0x7fada801ae10 crc :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:16.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.384 7fadcd84a700 1 -- 172.21.15.32:0/140952646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc818f7b0 con 0x7fadc81a5210 2021-04-09T18:30:16.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.387 7fadaffff700 1 -- 172.21.15.32:0/140952646 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fadb8000c90 con 0x7fadc81a5210 2021-04-09T18:30:16.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.579 7fadcd84a700 1 -- 172.21.15.32:0/140952646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadc818f7b0 con 0x7fadc81a5210 2021-04-09T18:30:16.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.579 7fadaffff700 1 -- 172.21.15.32:0/140952646 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+14016 (secure 0 0 0) 0x7fadb8006b00 con 0x7fadc81a5210 2021-04-09T18:30:16.594 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:16.594 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":179,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:15.764180","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:30:13.709244","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":31,"pool_name":"alpha_data","create_time":"2021-04-09 18:30:14.523869","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:16.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 -- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fada8018930 msgr2=0x7fada801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fada8018930 0x7fada801ae10 crc :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 -- 172.21.15.32:0/140952646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc81a5210 msgr2=0x7fadc8185450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc81a5210 0x7fadc8185450 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7fadb8006f60 tx=0x7fadb800b970).stop 2021-04-09T18:30:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 -- 172.21.15.32:0/140952646 shutdown_connections 2021-04-09T18:30:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fada8018930 0x7fada801ae10 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadc8185e70 0x7fadc80ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadc8185a70 0x7fadc80acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 --2- 172.21.15.32:0/140952646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fadc81a5210 0x7fadc8185450 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 -- 172.21.15.32:0/140952646 >> 172.21.15.32:0/140952646 conn(0x7fadc81a99c0 msgr2=0x7fadc817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 -- 172.21.15.32:0/140952646 shutdown_connections 2021-04-09T18:30:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.582 7fadcd84a700 1 -- 172.21.15.32:0/140952646 wait complete. 2021-04-09T18:30:16.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set alpha standby_count_wanted 0 2021-04-09T18:30:16.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.745 7efc301aa700 1 Processor -- start 2021-04-09T18:30:16.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.745 7efc301aa700 1 -- start start 2021-04-09T18:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc301aa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc2810d010 0x7efc280d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc301aa700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efc280d1da0 0x7efc280d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc301aa700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2810d480 con 0x7efc280d82c0 2021-04-09T18:30:16.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc301aa700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc280d8d30 con 0x7efc2810d010 2021-04-09T18:30:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc301aa700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc280d9220 con 0x7efc280d1da0 2021-04-09T18:30:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2dfc6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc2810d010 0x7efc280d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2e7c7700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efc280d82c0 0x7efc2810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37008/0 (socket says 172.21.15.32:37008) 2021-04-09T18:30:16.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2dfc6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc2810d010 0x7efc280d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55648/0 (socket says 172.21.15.32:55648) 2021-04-09T18:30:16.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2e7c7700 1 -- 172.21.15.32:0/342233260 learned_addr learned my addr 172.21.15.32:0/342233260 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:16.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2d7c5700 1 --2- 172.21.15.32:0/342233260 >> v2:172.21.15.86:3300/0 conn(0x7efc280d1da0 0x7efc280d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2dfc6700 1 -- 172.21.15.32:0/342233260 >> v2:172.21.15.86:3300/0 conn(0x7efc280d1da0 msgr2=0x7efc280d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2dfc6700 1 --2- 172.21.15.32:0/342233260 >> v2:172.21.15.86:3300/0 conn(0x7efc280d1da0 0x7efc280d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2dfc6700 1 -- 172.21.15.32:0/342233260 >> v1:172.21.15.32:6789/0 conn(0x7efc280d82c0 legacy=0x7efc2810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2dfc6700 1 -- 172.21.15.32:0/342233260 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc280d86f0 con 0x7efc2810d010 2021-04-09T18:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2cfc4700 1 -- 172.21.15.32:0/342233260 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980204953 0 0) 0x7efc20000d30 con 0x7efc280d82c0 2021-04-09T18:30:16.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2dfc6700 1 --2- 172.21.15.32:0/342233260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc2810d010 0x7efc280d1780 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 rx=0x7efc24006650 tx=0x7efc24009a00).ready entity=mon.0 client_cookie=e0db4273b26608ab server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.746 7efc2cfc4700 1 -- 172.21.15.32:0/342233260 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc24009df0 con 0x7efc2810d010 2021-04-09T18:30:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc2cfc4700 1 -- 172.21.15.32:0/342233260 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc240009b0 con 0x7efc2810d010 2021-04-09T18:30:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc2cfc4700 1 -- 172.21.15.32:0/342233260 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc2400c2b0 con 0x7efc2810d010 2021-04-09T18:30:16.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc301aa700 1 -- 172.21.15.32:0/342233260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc2810d010 msgr2=0x7efc280d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc301aa700 1 --2- 172.21.15.32:0/342233260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc2810d010 0x7efc280d1780 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 rx=0x7efc24006650 tx=0x7efc24009a00).stop 2021-04-09T18:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc301aa700 1 -- 172.21.15.32:0/342233260 shutdown_connections 2021-04-09T18:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc301aa700 1 --2- 172.21.15.32:0/342233260 >> v2:172.21.15.86:3300/0 conn(0x7efc280d1da0 0x7efc280d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc301aa700 1 --2- 172.21.15.32:0/342233260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc2810d010 0x7efc280d1780 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc301aa700 1 -- 172.21.15.32:0/342233260 >> 172.21.15.32:0/342233260 conn(0x7efc280ce5f0 msgr2=0x7efc280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc301aa700 1 -- 172.21.15.32:0/342233260 shutdown_connections 2021-04-09T18:30:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.747 7efc301aa700 1 -- 172.21.15.32:0/342233260 wait complete. 2021-04-09T18:30:16.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.768 7efc301aa700 1 Processor -- start 2021-04-09T18:30:16.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.768 7efc301aa700 1 -- start start 2021-04-09T18:30:16.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc301aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc280a0a10 0x7efc280bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc301aa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc2818ad00 0x7efc2819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc301aa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc281a03f0 0x7efc28095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc301aa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2818b100 con 0x7efc281a03f0 2021-04-09T18:30:16.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc301aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc28095860 con 0x7efc280a0a10 2021-04-09T18:30:16.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc301aa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc28095a80 con 0x7efc2818ad00 2021-04-09T18:30:16.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2e7c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc281a03f0 0x7efc28095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2e7c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc281a03f0 0x7efc28095240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55656/0 (socket says 172.21.15.32:55656) 2021-04-09T18:30:16.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2e7c7700 1 -- 172.21.15.32:0/4053823530 learned_addr learned my addr 172.21.15.32:0/4053823530 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:16.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2d7c5700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc2818ad00 0x7efc2819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2dfc6700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc280a0a10 0x7efc280bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2e7c7700 1 -- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc2818ad00 msgr2=0x7efc2819fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2e7c7700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc2818ad00 0x7efc2819fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2e7c7700 1 -- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc280a0a10 msgr2=0x7efc280bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:16.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2e7c7700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc280a0a10 0x7efc280bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:16.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2e7c7700 1 -- 172.21.15.32:0/4053823530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc28189250 con 0x7efc281a03f0 2021-04-09T18:30:16.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2d7c5700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc2818ad00 0x7efc2819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:16.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc2e7c7700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc281a03f0 0x7efc28095240 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7efc20007f90 tx=0x7efc2000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:16.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc1e7fc700 1 -- 172.21.15.32:0/4053823530 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc20000cc0 con 0x7efc281a03f0 2021-04-09T18:30:16.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc1e7fc700 1 -- 172.21.15.32:0/4053823530 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc20007020 con 0x7efc281a03f0 2021-04-09T18:30:16.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc301aa700 1 -- 172.21.15.32:0/4053823530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc28189250 con 0x7efc281a03f0 2021-04-09T18:30:16.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.769 7efc1e7fc700 1 -- 172.21.15.32:0/4053823530 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc20007540 con 0x7efc281a03f0 2021-04-09T18:30:16.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.770 7efc301aa700 1 -- 172.21.15.32:0/4053823530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc28189250 con 0x7efc281a03f0 2021-04-09T18:30:16.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.770 7efc1e7fc700 1 -- 172.21.15.32:0/4053823530 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efc20007200 con 0x7efc281a03f0 2021-04-09T18:30:16.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.770 7efc1e7fc700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efc10018930 0x7efc1001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:16.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.770 7efc1e7fc700 1 -- 172.21.15.32:0/4053823530 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efc20032d40 con 0x7efc281a03f0 2021-04-09T18:30:16.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.770 7efc2dfc6700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efc10018930 0x7efc1001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:16.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.771 7efc2dfc6700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efc10018930 0x7efc1001ae10 crc :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:16.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.790 7efc301aa700 1 -- 172.21.15.32:0/4053823530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc28093d50 con 0x7efc281a03f0 2021-04-09T18:30:16.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.793 7efc1e7fc700 1 -- 172.21.15.32:0/4053823530 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efc20007d00 con 0x7efc281a03f0 2021-04-09T18:30:17.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:16.987 7efc301aa700 1 -- 172.21.15.32:0/4053823530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "alpha", "val": "0"} v 0) v1 -- 0x7efc2810d480 con 0x7efc281a03f0 2021-04-09T18:30:17.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.774 7efc1e7fc700 1 -- 172.21.15.32:0/4053823530 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "alpha", "val": "0"}]=0 v325) v1 ==== 117+0+0 (secure 0 0 0) 0x7efc20006f20 con 0x7efc281a03f0 2021-04-09T18:30:17.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 -- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efc10018930 msgr2=0x7efc1001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:17.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efc10018930 0x7efc1001ae10 crc :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:17.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 -- 172.21.15.32:0/4053823530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc281a03f0 msgr2=0x7efc28095240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:17.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc281a03f0 0x7efc28095240 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7efc20007f90 tx=0x7efc2000ac50).stop 2021-04-09T18:30:17.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 -- 172.21.15.32:0/4053823530 shutdown_connections 2021-04-09T18:30:17.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efc10018930 0x7efc1001ae10 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:17.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc281a03f0 0x7efc28095240 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:17.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc2818ad00 0x7efc2819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:17.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 --2- 172.21.15.32:0/4053823530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc280a0a10 0x7efc280bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:17.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.805 7efc301aa700 1 -- 172.21.15.32:0/4053823530 >> 172.21.15.32:0/4053823530 conn(0x7efc280975f0 msgr2=0x7efc28096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:17.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.806 7efc301aa700 1 -- 172.21.15.32:0/4053823530 shutdown_connections 2021-04-09T18:30:17.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.806 7efc301aa700 1 -- 172.21.15.32:0/4053823530 wait complete. 2021-04-09T18:30:17.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.967 7f57936ad700 1 Processor -- start 2021-04-09T18:30:18.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.967 7f57936ad700 1 -- start start 2021-04-09T18:30:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f57936ad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c10d010 0x7f578c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f57936ad700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f578c0d8490 0x7f578c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f57936ad700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578c10d480 con 0x7f578c0d1da0 2021-04-09T18:30:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f57936ad700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f578c10e780 con 0x7f578c0d8490 2021-04-09T18:30:18.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f57936ad700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c0d8d30 con 0x7f578c10d010 2021-04-09T18:30:18.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f5790cc8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f578c0d1da0 0x7f578c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37022/0 (socket says 172.21.15.32:37022) 2021-04-09T18:30:18.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f5790cc8700 1 -- 172.21.15.32:0/100423469 learned_addr learned my addr 172.21.15.32:0/100423469 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:18.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f5791cca700 1 --2- 172.21.15.32:0/100423469 >> v2:172.21.15.86:3300/0 conn(0x7f578c0d8490 0x7f578c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f57914c9700 1 --2- 172.21.15.32:0/100423469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c10d010 0x7f578c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f5783fff700 1 -- 172.21.15.32:0/100423469 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331680541 0 0) 0x7f577c000ce0 con 0x7f578c0d1da0 2021-04-09T18:30:18.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.968 7f5783fff700 1 -- 172.21.15.32:0/100423469 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5778002fe0 con 0x7f578c0d1da0 2021-04-09T18:30:18.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f5783fff700 1 -- 172.21.15.32:0/100423469 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1273139678 0 0) 0x7f577c0010e0 con 0x7f578c0d1da0 2021-04-09T18:30:18.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f5791cca700 1 -- 172.21.15.32:0/100423469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c10d010 msgr2=0x7f578c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:18.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f5791cca700 1 --2- 172.21.15.32:0/100423469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c10d010 0x7f578c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f5791cca700 1 -- 172.21.15.32:0/100423469 >> v1:172.21.15.32:6789/0 conn(0x7f578c0d1da0 legacy=0x7f578c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f5791cca700 1 -- 172.21.15.32:0/100423469 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c0d9220 con 0x7f578c0d8490 2021-04-09T18:30:18.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f57914c9700 1 --2- 172.21.15.32:0/100423469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c10d010 0x7f578c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:18.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f5791cca700 1 --2- 172.21.15.32:0/100423469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f578c0d8490 0x7f578c10da00 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f5784006e50 tx=0x7f5784009e70).ready entity=mon.1 client_cookie=8b776c9f6d1d5f35 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:18.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f5783fff700 1 -- 172.21.15.32:0/100423469 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5784006a20 con 0x7f578c0d8490 2021-04-09T18:30:18.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f5783fff700 1 -- 172.21.15.32:0/100423469 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5784006790 con 0x7f578c0d8490 2021-04-09T18:30:18.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f5783fff700 1 -- 172.21.15.32:0/100423469 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5784000f90 con 0x7f578c0d8490 2021-04-09T18:30:18.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f57936ad700 1 -- 172.21.15.32:0/100423469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f578c0d8490 msgr2=0x7f578c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.969 7f57936ad700 1 --2- 172.21.15.32:0/100423469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f578c0d8490 0x7f578c10da00 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f5784006e50 tx=0x7f5784009e70).stop 2021-04-09T18:30:18.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.970 7f57936ad700 1 -- 172.21.15.32:0/100423469 shutdown_connections 2021-04-09T18:30:18.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.970 7f57936ad700 1 --2- 172.21.15.32:0/100423469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f578c0d8490 0x7f578c10da00 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.970 7f57936ad700 1 --2- 172.21.15.32:0/100423469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c10d010 0x7f578c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.970 7f57936ad700 1 -- 172.21.15.32:0/100423469 >> 172.21.15.32:0/100423469 conn(0x7f578c0ce5f0 msgr2=0x7f578c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:18.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.970 7f57936ad700 1 -- 172.21.15.32:0/100423469 shutdown_connections 2021-04-09T18:30:18.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.970 7f57936ad700 1 -- 172.21.15.32:0/100423469 wait complete. 2021-04-09T18:30:18.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.990 7f57936ad700 1 Processor -- start 2021-04-09T18:30:18.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.990 7f57936ad700 1 -- start start 2021-04-09T18:30:18.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f57936ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f578c1a91e0 0x7f578c0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f57936ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f578c1a95e0 0x7f578c0a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f57936ad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c1ac850 0x7f578c1acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f57936ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c1ad270 con 0x7f578c1a95e0 2021-04-09T18:30:18.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f57936ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c0b7640 con 0x7f578c1a91e0 2021-04-09T18:30:18.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f57936ad700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c183c80 con 0x7f578c1ac850 2021-04-09T18:30:18.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5790cc8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f578c1a95e0 0x7f578c0a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5790cc8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f578c1a95e0 0x7f578c0a5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55668/0 (socket says 172.21.15.32:55668) 2021-04-09T18:30:18.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5790cc8700 1 -- 172.21.15.32:0/152929669 learned_addr learned my addr 172.21.15.32:0/152929669 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:18.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f57914c9700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f578c1a91e0 0x7f578c0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5791cca700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c1ac850 0x7f578c1acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5790cc8700 1 -- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c1ac850 msgr2=0x7f578c1acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5790cc8700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c1ac850 0x7f578c1acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5790cc8700 1 -- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f578c1a91e0 msgr2=0x7f578c0a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5790cc8700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f578c1a91e0 0x7f578c0a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5790cc8700 1 -- 172.21.15.32:0/152929669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c183ea0 con 0x7f578c1a95e0 2021-04-09T18:30:18.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5790cc8700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f578c1a95e0 0x7f578c0a5080 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f577c000f90 tx=0x7f577c00a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:18.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5781ffb700 1 -- 172.21.15.32:0/152929669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f577c001a30 con 0x7f578c1a95e0 2021-04-09T18:30:18.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.991 7f5781ffb700 1 -- 172.21.15.32:0/152929669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c0043a0 con 0x7f578c1a95e0 2021-04-09T18:30:18.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.992 7f57936ad700 1 -- 172.21.15.32:0/152929669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578c183ea0 con 0x7f578c1a95e0 2021-04-09T18:30:18.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.992 7f5781ffb700 1 -- 172.21.15.32:0/152929669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f577c0047f0 con 0x7f578c1a95e0 2021-04-09T18:30:18.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.992 7f57936ad700 1 -- 172.21.15.32:0/152929669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578c183ea0 con 0x7f578c1a95e0 2021-04-09T18:30:18.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.992 7f5781ffb700 1 -- 172.21.15.32:0/152929669 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f577c004580 con 0x7f578c1a95e0 2021-04-09T18:30:18.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.992 7f5781ffb700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f576c018930 0x7f576c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.992 7f5781ffb700 1 -- 172.21.15.32:0/152929669 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f577c0329e0 con 0x7f578c1a95e0 2021-04-09T18:30:18.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.992 7f57914c9700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f576c018930 0x7f576c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:17.993 7f57914c9700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f576c018930 0x7f576c01ae10 crc :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:18.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.012 7f57936ad700 1 -- 172.21.15.32:0/152929669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578c182d70 con 0x7f578c1a95e0 2021-04-09T18:30:18.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.016 7f5781ffb700 1 -- 172.21.15.32:0/152929669 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f577c001a30 con 0x7f578c1a95e0 2021-04-09T18:30:18.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.208 7f57936ad700 1 -- 172.21.15.32:0/152929669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f578c10d480 con 0x7f578c1a95e0 2021-04-09T18:30:18.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.208 7f5781ffb700 1 -- 172.21.15.32:0/152929669 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 325 v325) v1 ==== 95+0+2788 (secure 0 0 0) 0x7f577c00ad30 con 0x7f578c1a95e0 2021-04-09T18:30:18.223 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 325 2021-04-09T18:30:18.224 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:18.224 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":325,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10914,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3158551863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3158551863},{"type":"v1","addr":"172.21.15.32:6835","nonce":3158551863}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":321},{"gid":10920,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2893945823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2893945823},{"type":"v1","addr":"172.21.15.86:6837","nonce":2893945823}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":321},{"gid":58607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1628176610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1628176610},{"type":"v1","addr":"172.21.15.86:6836","nonce":1628176610}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":321}],"filesystems":[{"mdsmap":{"epoch":325,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:17.772722","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10911":{"gid":10911,"name":"c","rank":0,"incarnation":322,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6834/3729993394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3729993394},{"type":"v1","addr":"172.21.15.32:6834","nonce":3729993394}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15}]} 2021-04-09T18:30:18.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 -- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f576c018930 msgr2=0x7f576c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f576c018930 0x7f576c01ae10 crc :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 -- 172.21.15.32:0/152929669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f578c1a95e0 msgr2=0x7f578c0a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f578c1a95e0 0x7f578c0a5080 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f577c000f90 tx=0x7f577c00a940).stop 2021-04-09T18:30:18.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 -- 172.21.15.32:0/152929669 shutdown_connections 2021-04-09T18:30:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f576c018930 0x7f576c01ae10 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f578c1a95e0 0x7f578c0a5080 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f578c1ac850 0x7f578c1acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 --2- 172.21.15.32:0/152929669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f578c1a91e0 0x7f578c0a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 -- 172.21.15.32:0/152929669 >> 172.21.15.32:0/152929669 conn(0x7f578c19b460 msgr2=0x7f578c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 -- 172.21.15.32:0/152929669 shutdown_connections 2021-04-09T18:30:18.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.211 7f57936ad700 1 -- 172.21.15.32:0/152929669 wait complete. 2021-04-09T18:30:18.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:18.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.370 7f1cfb1bd700 1 Processor -- start 2021-04-09T18:30:18.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.370 7f1cfb1bd700 1 -- start start 2021-04-09T18:30:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.370 7f1cfb1bd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1cf410d010 0x7f1cf40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.370 7f1cfb1bd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf40d1da0 0x7f1cf40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.370 7f1cfb1bd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1cf40d82c0 0x7f1cf410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.370 7f1cfb1bd700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf40d8d30 con 0x7f1cf410d010 2021-04-09T18:30:18.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.370 7f1cfb1bd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf40d9220 con 0x7f1cf40d82c0 2021-04-09T18:30:18.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.370 7f1cfb1bd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf410d480 con 0x7f1cf40d1da0 2021-04-09T18:30:18.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.370 7f1cf8fd9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1cf410d010 0x7f1cf40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf8fd9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1cf410d010 0x7f1cf40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55676/0 (socket says 172.21.15.32:55676) 2021-04-09T18:30:18.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf8fd9700 1 -- 172.21.15.32:0/3167944006 learned_addr learned my addr 172.21.15.32:0/3167944006 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:18.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf97da700 1 --2- 172.21.15.32:0/3167944006 >> v2:172.21.15.86:3300/0 conn(0x7f1cf40d82c0 0x7f1cf410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf3fff700 1 --2- 172.21.15.32:0/3167944006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf40d1da0 0x7f1cf40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf8fd9700 1 -- 172.21.15.32:0/3167944006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf40d1da0 msgr2=0x7f1cf40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:18.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf8fd9700 1 --2- 172.21.15.32:0/3167944006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf40d1da0 0x7f1cf40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf8fd9700 1 -- 172.21.15.32:0/3167944006 >> v2:172.21.15.86:3300/0 conn(0x7f1cf40d82c0 msgr2=0x7f1cf410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf8fd9700 1 --2- 172.21.15.32:0/3167944006 >> v2:172.21.15.86:3300/0 conn(0x7f1cf40d82c0 0x7f1cf410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf8fd9700 1 -- 172.21.15.32:0/3167944006 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf40d86c0 con 0x7f1cf410d010 2021-04-09T18:30:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf8fd9700 1 --2- 172.21.15.32:0/3167944006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf410d010 0x7f1cf40d1780 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f1cdc006700 tx=0x7f1cdc009b00).ready entity=mon.0 client_cookie=4abe94f221503738 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf37fe700 1 -- 172.21.15.32:0/3167944006 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cdc00c0c0 con 0x7f1cf410d010 2021-04-09T18:30:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf37fe700 1 -- 172.21.15.32:0/3167944006 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc00c400 con 0x7f1cf410d010 2021-04-09T18:30:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cf37fe700 1 -- 172.21.15.32:0/3167944006 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cdc00c8e0 con 0x7f1cf410d010 2021-04-09T18:30:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cfb1bd700 1 -- 172.21.15.32:0/3167944006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf410d010 msgr2=0x7f1cf40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.371 7f1cfb1bd700 1 --2- 172.21.15.32:0/3167944006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf410d010 0x7f1cf40d1780 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f1cdc006700 tx=0x7f1cdc009b00).stop 2021-04-09T18:30:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.372 7f1cfb1bd700 1 -- 172.21.15.32:0/3167944006 shutdown_connections 2021-04-09T18:30:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.372 7f1cfb1bd700 1 --2- 172.21.15.32:0/3167944006 >> v2:172.21.15.86:3300/0 conn(0x7f1cf40d82c0 0x7f1cf410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.372 7f1cfb1bd700 1 --2- 172.21.15.32:0/3167944006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf40d1da0 0x7f1cf40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.372 7f1cfb1bd700 1 --2- 172.21.15.32:0/3167944006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf410d010 0x7f1cf40d1780 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.372 7f1cfb1bd700 1 -- 172.21.15.32:0/3167944006 >> 172.21.15.32:0/3167944006 conn(0x7f1cf40ce5f0 msgr2=0x7f1cf40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.372 7f1cfb1bd700 1 -- 172.21.15.32:0/3167944006 shutdown_connections 2021-04-09T18:30:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.372 7f1cfb1bd700 1 -- 172.21.15.32:0/3167944006 wait complete. 2021-04-09T18:30:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.392 7f1cfb1bd700 1 Processor -- start 2021-04-09T18:30:18.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.392 7f1cfb1bd700 1 -- start start 2021-04-09T18:30:18.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cfb1bd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cf41ac380 0x7f1cf41ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cfb1bd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf4186c90 0x7f1cf4187090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cfb1bd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf4185530 0x7f1cf4185930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cfb1bd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4186790 con 0x7f1cf4185530 2021-04-09T18:30:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cfb1bd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf41acda0 con 0x7f1cf41ac380 2021-04-09T18:30:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cfb1bd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf418e100 con 0x7f1cf4186c90 2021-04-09T18:30:18.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf97da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf4185530 0x7f1cf4185930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf97da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf4185530 0x7f1cf4185930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55684/0 (socket says 172.21.15.32:55684) 2021-04-09T18:30:18.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf3fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf4186c90 0x7f1cf4187090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf97da700 1 -- 172.21.15.32:0/2276798715 learned_addr learned my addr 172.21.15.32:0/2276798715 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:18.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf8fd9700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cf41ac380 0x7f1cf41ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf97da700 1 -- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf4186c90 msgr2=0x7f1cf4187090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf97da700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf4186c90 0x7f1cf4187090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf97da700 1 -- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cf41ac380 msgr2=0x7f1cf41ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf97da700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cf41ac380 0x7f1cf41ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf97da700 1 -- 172.21.15.32:0/2276798715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf41876b0 con 0x7f1cf4185530 2021-04-09T18:30:18.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf97da700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf4185530 0x7f1cf4185930 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7f1cec00b5e0 tx=0x7f1cec00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:18.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf17fa700 1 -- 172.21.15.32:0/2276798715 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cec000c10 con 0x7f1cf4185530 2021-04-09T18:30:18.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.393 7f1cf17fa700 1 -- 172.21.15.32:0/2276798715 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec00bd70 con 0x7f1cf4185530 2021-04-09T18:30:18.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.394 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf41876b0 con 0x7f1cf4185530 2021-04-09T18:30:18.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.394 7f1cf17fa700 1 -- 172.21.15.32:0/2276798715 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cec002560 con 0x7f1cf4185530 2021-04-09T18:30:18.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.394 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf41876b0 con 0x7f1cf4185530 2021-04-09T18:30:18.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.394 7f1cf17fa700 1 -- 172.21.15.32:0/2276798715 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1cec006720 con 0x7f1cf4185530 2021-04-09T18:30:18.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.394 7f1cf17fa700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cd8018930 0x7f1cd801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.394 7f1cf17fa700 1 -- 172.21.15.32:0/2276798715 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1cec032c10 con 0x7f1cf4185530 2021-04-09T18:30:18.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.395 7f1cf8fd9700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cd8018930 0x7f1cd801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.395 7f1cf8fd9700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cd8018930 0x7f1cd801ae10 crc :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:18.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.414 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf41acda0 con 0x7f1cf4185530 2021-04-09T18:30:18.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.418 7f1cf17fa700 1 -- 172.21.15.32:0/2276798715 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1cec006370 con 0x7f1cf4185530 2021-04-09T18:30:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.607 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cf41acda0 con 0x7f1cf4185530 2021-04-09T18:30:18.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.607 7f1cf17fa700 1 -- 172.21.15.32:0/2276798715 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14003 (secure 0 0 0) 0x7f1cec006720 con 0x7f1cf4185530 2021-04-09T18:30:18.623 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:18.623 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":180,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:16.765825","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:30:13.709244","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":31,"pool_name":"alpha_data","create_time":"2021-04-09 18:30:14.523869","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cd8018930 msgr2=0x7f1cd801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cd8018930 0x7f1cd801ae10 crc :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf4185530 msgr2=0x7f1cf4185930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf4185530 0x7f1cf4185930 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7f1cec00b5e0 tx=0x7f1cec00b980).stop 2021-04-09T18:30:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 shutdown_connections 2021-04-09T18:30:18.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cd8018930 0x7f1cd801ae10 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cf4185530 0x7f1cf4185930 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cf4186c90 0x7f1cf4187090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 --2- 172.21.15.32:0/2276798715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cf41ac380 0x7f1cf41ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 >> 172.21.15.32:0/2276798715 conn(0x7f1cf419b890 msgr2=0x7f1cf41a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 shutdown_connections 2021-04-09T18:30:18.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.610 7f1cfb1bd700 1 -- 172.21.15.32:0/2276798715 wait complete. 2021-04-09T18:30:18.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:18.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.770 7f5223eb2700 1 Processor -- start 2021-04-09T18:30:18.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.770 7f5223eb2700 1 -- start start 2021-04-09T18:30:18.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5223eb2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f521c10d010 0x7f521c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5223eb2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c0d1da0 0x7f521c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5223eb2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c10d480 con 0x7f521c0d82c0 2021-04-09T18:30:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5223eb2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c0d9220 con 0x7f521c10d010 2021-04-09T18:30:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5223eb2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0d86c0 con 0x7f521c0d1da0 2021-04-09T18:30:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5221cce700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f521c10d010 0x7f521c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f52224cf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f521c0d82c0 0x7f521c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37048/0 (socket says 172.21.15.32:37048) 2021-04-09T18:30:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f52224cf700 1 -- 172.21.15.32:0/4103067445 learned_addr learned my addr 172.21.15.32:0/4103067445 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f52214cd700 1 --2- 172.21.15.32:0/4103067445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c0d1da0 0x7f521c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5221cce700 1 -- 172.21.15.32:0/4103067445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c0d1da0 msgr2=0x7f521c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5221cce700 1 --2- 172.21.15.32:0/4103067445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c0d1da0 0x7f521c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5221cce700 1 -- 172.21.15.32:0/4103067445 >> v1:172.21.15.32:6789/0 conn(0x7f521c0d82c0 legacy=0x7f521c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5220ccc700 1 -- 172.21.15.32:0/4103067445 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673631223 0 0) 0x7f5214000d30 con 0x7f521c0d82c0 2021-04-09T18:30:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f5221cce700 1 -- 172.21.15.32:0/4103067445 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c10e1f0 con 0x7f521c10d010 2021-04-09T18:30:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.771 7f52214cd700 1 --2- 172.21.15.32:0/4103067445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c0d1da0 0x7f521c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5221cce700 1 --2- 172.21.15.32:0/4103067445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c10d010 0x7f521c0d1780 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7f5218006650 tx=0x7f5218009a80).ready entity=mon.0 client_cookie=d1a1324c1558d57c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5220ccc700 1 -- 172.21.15.32:0/4103067445 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52180009b0 con 0x7f521c10d010 2021-04-09T18:30:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5220ccc700 1 -- 172.21.15.32:0/4103067445 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521800c0c0 con 0x7f521c10d010 2021-04-09T18:30:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5220ccc700 1 -- 172.21.15.32:0/4103067445 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f521800c510 con 0x7f521c10d010 2021-04-09T18:30:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5223eb2700 1 -- 172.21.15.32:0/4103067445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c10d010 msgr2=0x7f521c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5223eb2700 1 --2- 172.21.15.32:0/4103067445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c10d010 0x7f521c0d1780 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7f5218006650 tx=0x7f5218009a80).stop 2021-04-09T18:30:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5223eb2700 1 -- 172.21.15.32:0/4103067445 shutdown_connections 2021-04-09T18:30:18.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5223eb2700 1 --2- 172.21.15.32:0/4103067445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c0d1da0 0x7f521c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5223eb2700 1 --2- 172.21.15.32:0/4103067445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c10d010 0x7f521c0d1780 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.772 7f5223eb2700 1 -- 172.21.15.32:0/4103067445 >> 172.21.15.32:0/4103067445 conn(0x7f521c0ce5f0 msgr2=0x7f521c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:18.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.773 7f5223eb2700 1 -- 172.21.15.32:0/4103067445 shutdown_connections 2021-04-09T18:30:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.773 7f5223eb2700 1 -- 172.21.15.32:0/4103067445 wait complete. 2021-04-09T18:30:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.793 7f5223eb2700 1 Processor -- start 2021-04-09T18:30:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.793 7f5223eb2700 1 -- start start 2021-04-09T18:30:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f5223eb2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c186fb0 0x7f521c184680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f5223eb2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f521c183da0 0x7f521c187b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f5223eb2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c188170 0x7f521c1ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f5223eb2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c1875d0 con 0x7f521c188170 2021-04-09T18:30:18.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f5223eb2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c188570 con 0x7f521c183da0 2021-04-09T18:30:18.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f5223eb2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c1841a0 con 0x7f521c186fb0 2021-04-09T18:30:18.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52224cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c188170 0x7f521c1ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52224cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c188170 0x7f521c1ac090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55698/0 (socket says 172.21.15.32:55698) 2021-04-09T18:30:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52224cf700 1 -- 172.21.15.32:0/3698684532 learned_addr learned my addr 172.21.15.32:0/3698684532 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52214cd700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f521c183da0 0x7f521c187b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f5221cce700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c186fb0 0x7f521c184680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52224cf700 1 -- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c186fb0 msgr2=0x7f521c184680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52224cf700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c186fb0 0x7f521c184680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52224cf700 1 -- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f521c183da0 msgr2=0x7f521c187b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:18.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52224cf700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f521c183da0 0x7f521c187b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:18.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52224cf700 1 -- 172.21.15.32:0/3698684532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c1ac6b0 con 0x7f521c188170 2021-04-09T18:30:18.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52224cf700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c188170 0x7f521c1ac090 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7f5214007f90 tx=0x7f521400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:18.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.794 7f52127fc700 1 -- 172.21.15.32:0/3698684532 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5214000cc0 con 0x7f521c188170 2021-04-09T18:30:18.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.795 7f52127fc700 1 -- 172.21.15.32:0/3698684532 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5214007020 con 0x7f521c188170 2021-04-09T18:30:18.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.795 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c1ac6b0 con 0x7f521c188170 2021-04-09T18:30:18.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.795 7f52127fc700 1 -- 172.21.15.32:0/3698684532 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5214007500 con 0x7f521c188170 2021-04-09T18:30:18.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.795 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c1ac6b0 con 0x7f521c188170 2021-04-09T18:30:18.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.795 7f52127fc700 1 -- 172.21.15.32:0/3698684532 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5214007200 con 0x7f521c188170 2021-04-09T18:30:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.795 7f52127fc700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5204018930 0x7f520401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.796 7f52127fc700 1 -- 172.21.15.32:0/3698684532 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5214032d50 con 0x7f521c188170 2021-04-09T18:30:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.796 7f5221cce700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5204018930 0x7f520401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.796 7f5221cce700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5204018930 0x7f520401ae10 crc :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:18.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.815 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c182d80 con 0x7f521c188170 2021-04-09T18:30:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:18.818 7f52127fc700 1 -- 172.21.15.32:0/3698684532 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5214007d00 con 0x7f521c188170 2021-04-09T18:30:19.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.006 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f521c10d480 con 0x7f521c188170 2021-04-09T18:30:19.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.007 7f52127fc700 1 -- 172.21.15.32:0/3698684532 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14003 (secure 0 0 0) 0x7f5214006f00 con 0x7f521c188170 2021-04-09T18:30:19.021 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:19.022 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":180,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:16.765825","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:30:13.709244","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":31,"pool_name":"alpha_data","create_time":"2021-04-09 18:30:14.523869","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:19.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5204018930 msgr2=0x7f520401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5204018930 0x7f520401ae10 crc :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c188170 msgr2=0x7f521c1ac090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c188170 0x7f521c1ac090 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7f5214007f90 tx=0x7f521400ac50).stop 2021-04-09T18:30:19.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 shutdown_connections 2021-04-09T18:30:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5204018930 0x7f520401ae10 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f521c188170 0x7f521c1ac090 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f521c183da0 0x7f521c187b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 --2- 172.21.15.32:0/3698684532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f521c186fb0 0x7f521c184680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 >> 172.21.15.32:0/3698684532 conn(0x7f521c19b890 msgr2=0x7f521c1a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 shutdown_connections 2021-04-09T18:30:19.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.009 7f5223eb2700 1 -- 172.21.15.32:0/3698684532 wait complete. 2021-04-09T18:30:19.033 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 30, 'pool_name': 'alpha_metadata', 'create_time': '2021-04-09 18:30:13.709244', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'alpha'}}}, {'pool': 31, 'pool_name': 'alpha_data', 'create_time': '2021-04-09 18:30:14.523869', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'alpha'}}}] 2021-04-09T18:30:19.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:30:19.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.172 7fe415528700 1 Processor -- start 2021-04-09T18:30:19.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.172 7fe415528700 1 -- start start 2021-04-09T18:30:19.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.173 7fe415528700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe41010d010 0x7fe4100d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.173 7fe415528700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41010d480 con 0x7fe4100d8490 2021-04-09T18:30:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.173 7fe415528700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41010e1f0 con 0x7fe4100d1da0 2021-04-09T18:30:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.173 7fe415528700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4100d8d30 con 0x7fe41010d010 2021-04-09T18:30:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.173 7fe40fac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe4100d8490 0x7fe41010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37062/0 (socket says 172.21.15.32:37062) 2021-04-09T18:30:19.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.173 7fe40fac0700 1 -- 172.21.15.32:0/3965508970 learned_addr learned my addr 172.21.15.32:0/3965508970 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:19.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.173 7fe40f2bf700 1 --2- 172.21.15.32:0/3965508970 >> v2:172.21.15.86:3300/0 conn(0x7fe41010d010 0x7fe4100d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40e2bd700 1 -- 172.21.15.32:0/3965508970 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670969393 0 0) 0x7fe4000009b0 con 0x7fe4100d8490 2021-04-09T18:30:19.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40e2bd700 1 -- 172.21.15.32:0/3965508970 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3fc002fe0 con 0x7fe4100d8490 2021-04-09T18:30:19.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40e2bd700 1 -- 172.21.15.32:0/3965508970 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209702814 0 0) 0x7fe3f8000c60 con 0x7fe4100d1da0 2021-04-09T18:30:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40e2bd700 1 -- 172.21.15.32:0/3965508970 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3fc003210 con 0x7fe4100d1da0 2021-04-09T18:30:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40e2bd700 1 -- 172.21.15.32:0/3965508970 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3505890521 0 0) 0x7fe4000010e0 con 0x7fe4100d8490 2021-04-09T18:30:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40f2bf700 1 -- 172.21.15.32:0/3965508970 >> v1:172.21.15.86:6789/0 conn(0x7fe4100d1da0 legacy=0x7fe4100d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40f2bf700 1 -- 172.21.15.32:0/3965508970 >> v1:172.21.15.32:6789/0 conn(0x7fe4100d8490 legacy=0x7fe41010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40f2bf700 1 -- 172.21.15.32:0/3965508970 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4100d9220 con 0x7fe41010d010 2021-04-09T18:30:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40f2bf700 1 --2- 172.21.15.32:0/3965508970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41010d010 0x7fe4100d1780 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7fe4080069d0 tx=0x7fe408009dc0).ready entity=mon.1 client_cookie=71d7eebae7967013 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40e2bd700 1 -- 172.21.15.32:0/3965508970 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe40800b510 con 0x7fe41010d010 2021-04-09T18:30:19.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40e2bd700 1 -- 172.21.15.32:0/3965508970 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe408006ac0 con 0x7fe41010d010 2021-04-09T18:30:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe40e2bd700 1 -- 172.21.15.32:0/3965508970 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe40800ba80 con 0x7fe41010d010 2021-04-09T18:30:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe415528700 1 -- 172.21.15.32:0/3965508970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41010d010 msgr2=0x7fe4100d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.174 7fe415528700 1 --2- 172.21.15.32:0/3965508970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41010d010 0x7fe4100d1780 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7fe4080069d0 tx=0x7fe408009dc0).stop 2021-04-09T18:30:19.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.175 7fe415528700 1 -- 172.21.15.32:0/3965508970 shutdown_connections 2021-04-09T18:30:19.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.175 7fe415528700 1 --2- 172.21.15.32:0/3965508970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe41010d010 0x7fe4100d1780 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.175 7fe415528700 1 -- 172.21.15.32:0/3965508970 >> 172.21.15.32:0/3965508970 conn(0x7fe4100ce5f0 msgr2=0x7fe4100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:19.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.175 7fe415528700 1 -- 172.21.15.32:0/3965508970 shutdown_connections 2021-04-09T18:30:19.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.175 7fe415528700 1 -- 172.21.15.32:0/3965508970 wait complete. 2021-04-09T18:30:19.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.195 7fe415528700 1 Processor -- start 2021-04-09T18:30:19.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.195 7fe415528700 1 -- start start 2021-04-09T18:30:19.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe415528700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe410179610 0x7fe410179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe415528700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41017a820 0x7fe41017ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe415528700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4101949f0 0x7fe410194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe415528700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4101b8d50 con 0x7fe410179610 2021-04-09T18:30:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe415528700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41017b240 con 0x7fe4101949f0 2021-04-09T18:30:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe415528700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe410195410 con 0x7fe41017a820 2021-04-09T18:30:19.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe410179610 0x7fe410179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe410179610 0x7fe410179a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55710/0 (socket says 172.21.15.32:55710) 2021-04-09T18:30:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40f2bf700 1 -- 172.21.15.32:0/1979385220 learned_addr learned my addr 172.21.15.32:0/1979385220 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40eabe700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41017a820 0x7fe41017ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40fac0700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4101949f0 0x7fe410194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40f2bf700 1 -- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41017a820 msgr2=0x7fe41017ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40f2bf700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41017a820 0x7fe41017ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40f2bf700 1 -- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4101949f0 msgr2=0x7fe410194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40f2bf700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4101949f0 0x7fe410194df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40f2bf700 1 -- 172.21.15.32:0/1979385220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe410195630 con 0x7fe410179610 2021-04-09T18:30:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe40f2bf700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe410179610 0x7fe410179a10 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 rx=0x7fe4080099e0 tx=0x7fe40800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe407fff700 1 -- 172.21.15.32:0/1979385220 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe408006030 con 0x7fe410179610 2021-04-09T18:30:19.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.196 7fe407fff700 1 -- 172.21.15.32:0/1979385220 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4080062f0 con 0x7fe410179610 2021-04-09T18:30:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.197 7fe415528700 1 -- 172.21.15.32:0/1979385220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe410195630 con 0x7fe410179610 2021-04-09T18:30:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.197 7fe407fff700 1 -- 172.21.15.32:0/1979385220 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe408005660 con 0x7fe410179610 2021-04-09T18:30:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.197 7fe415528700 1 -- 172.21.15.32:0/1979385220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe410195630 con 0x7fe410179610 2021-04-09T18:30:19.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.197 7fe407fff700 1 -- 172.21.15.32:0/1979385220 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe4080064d0 con 0x7fe410179610 2021-04-09T18:30:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.197 7fe407fff700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f4018930 0x7fe3f401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.197 7fe407fff700 1 -- 172.21.15.32:0/1979385220 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe408032da0 con 0x7fe410179610 2021-04-09T18:30:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.197 7fe40eabe700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f4018930 0x7fe3f401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.198 7fe40eabe700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f4018930 0x7fe3f401adc0 crc :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:19.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.217 7fe415528700 1 -- 172.21.15.32:0/1979385220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41017a2f0 con 0x7fe410179610 2021-04-09T18:30:19.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.221 7fe407fff700 1 -- 172.21.15.32:0/1979385220 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe40800be10 con 0x7fe410179610 2021-04-09T18:30:19.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.400 7fe415528700 1 -- 172.21.15.32:0/1979385220 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe41017a2f0 con 0x7fe410179610 2021-04-09T18:30:19.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.400 7fe407fff700 1 -- 172.21.15.32:0/1979385220 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v180) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe4080062f0 con 0x7fe410179610 2021-04-09T18:30:19.415 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:19.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.402 7fe415528700 1 -- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f4018930 msgr2=0x7fe3f401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.402 7fe415528700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f4018930 0x7fe3f401adc0 crc :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.402 7fe415528700 1 -- 172.21.15.32:0/1979385220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe410179610 msgr2=0x7fe410179a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.402 7fe415528700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe410179610 0x7fe410179a10 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 rx=0x7fe4080099e0 tx=0x7fe40800bcd0).stop 2021-04-09T18:30:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.402 7fe415528700 1 -- 172.21.15.32:0/1979385220 shutdown_connections 2021-04-09T18:30:19.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.402 7fe415528700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe3f4018930 0x7fe3f401adc0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.402 7fe415528700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4101949f0 0x7fe410194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.402 7fe415528700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe41017a820 0x7fe41017ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.402 7fe415528700 1 --2- 172.21.15.32:0/1979385220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe410179610 0x7fe410179a10 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.403 7fe415528700 1 -- 172.21.15.32:0/1979385220 >> 172.21.15.32:0/1979385220 conn(0x7fe410092a00 msgr2=0x7fe4101b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:19.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.403 7fe415528700 1 -- 172.21.15.32:0/1979385220 shutdown_connections 2021-04-09T18:30:19.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.403 7fe415528700 1 -- 172.21.15.32:0/1979385220 wait complete. 2021-04-09T18:30:19.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_metadata pg_num 2021-04-09T18:30:19.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.566 7fc2aca63700 1 Processor -- start 2021-04-09T18:30:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.566 7fc2aca63700 1 -- start start 2021-04-09T18:30:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2aca63700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc2a80cffc0 0x7fc2a80d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2aca63700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc2a80d6ca0 0x7fc2a810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2aca63700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a80d72a0 con 0x7fc2a80d7820 2021-04-09T18:30:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2aca63700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a80d70a0 con 0x7fc2a80cffc0 2021-04-09T18:30:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2aca63700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a80d9200 con 0x7fc2a80d6ca0 2021-04-09T18:30:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2a685e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc2a80cffc0 0x7fc2a80d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2a685e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc2a80cffc0 0x7fc2a80d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55714/0 (socket says 172.21.15.32:55714) 2021-04-09T18:30:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2a685e700 1 -- 172.21.15.32:0/1972685093 learned_addr learned my addr 172.21.15.32:0/1972685093 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2a705f700 1 --2- 172.21.15.32:0/1972685093 >> v2:172.21.15.86:3300/0 conn(0x7fc2a80d6ca0 0x7fc2a810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2a685e700 1 -- 172.21.15.32:0/1972685093 >> v2:172.21.15.86:3300/0 conn(0x7fc2a80d6ca0 msgr2=0x7fc2a810cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:19.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2a685e700 1 --2- 172.21.15.32:0/1972685093 >> v2:172.21.15.86:3300/0 conn(0x7fc2a80d6ca0 0x7fc2a810cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2a685e700 1 -- 172.21.15.32:0/1972685093 >> v1:172.21.15.86:6789/0 conn(0x7fc2a80d7820 legacy=0x7fc2a80d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2a585c700 1 -- 172.21.15.32:0/1972685093 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028963757 0 0) 0x7fc29c001370 con 0x7fc2a80d7820 2021-04-09T18:30:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.567 7fc2a685e700 1 -- 172.21.15.32:0/1972685093 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80d8410 con 0x7fc2a80cffc0 2021-04-09T18:30:19.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.568 7fc2a685e700 1 --2- 172.21.15.32:0/1972685093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a80cffc0 0x7fc2a80d2460 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7fc298006700 tx=0x7fc298009b00).ready entity=mon.0 client_cookie=75aaa245d45e7b2d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.568 7fc2a585c700 1 -- 172.21.15.32:0/1972685093 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc29800c0c0 con 0x7fc2a80cffc0 2021-04-09T18:30:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.568 7fc2a585c700 1 -- 172.21.15.32:0/1972685093 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29800c400 con 0x7fc2a80cffc0 2021-04-09T18:30:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.568 7fc2a585c700 1 -- 172.21.15.32:0/1972685093 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc29800c8e0 con 0x7fc2a80cffc0 2021-04-09T18:30:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.568 7fc2aca63700 1 -- 172.21.15.32:0/1972685093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a80cffc0 msgr2=0x7fc2a80d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.568 7fc2aca63700 1 --2- 172.21.15.32:0/1972685093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a80cffc0 0x7fc2a80d2460 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7fc298006700 tx=0x7fc298009b00).stop 2021-04-09T18:30:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.569 7fc2aca63700 1 -- 172.21.15.32:0/1972685093 shutdown_connections 2021-04-09T18:30:19.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.569 7fc2aca63700 1 --2- 172.21.15.32:0/1972685093 >> v2:172.21.15.86:3300/0 conn(0x7fc2a80d6ca0 0x7fc2a810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.569 7fc2aca63700 1 --2- 172.21.15.32:0/1972685093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a80cffc0 0x7fc2a80d2460 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.569 7fc2aca63700 1 -- 172.21.15.32:0/1972685093 >> 172.21.15.32:0/1972685093 conn(0x7fc2a80ce5e0 msgr2=0x7fc2a80cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.569 7fc2aca63700 1 -- 172.21.15.32:0/1972685093 shutdown_connections 2021-04-09T18:30:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.569 7fc2aca63700 1 -- 172.21.15.32:0/1972685093 wait complete. 2021-04-09T18:30:19.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.589 7fc2aca63700 1 Processor -- start 2021-04-09T18:30:19.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.589 7fc2aca63700 1 -- start start 2021-04-09T18:30:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.589 7fc2aca63700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2a8180d70 0x7fc2a80ba330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.589 7fc2aca63700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2a80ba950 0x7fc2a81795f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.589 7fc2aca63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a8179c10 0x7fc2a81b8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.589 7fc2aca63700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80b1d40 con 0x7fc2a8179c10 2021-04-09T18:30:19.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.589 7fc2aca63700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8181170 con 0x7fc2a80ba950 2021-04-09T18:30:19.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.589 7fc2aca63700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80bad50 con 0x7fc2a8180d70 2021-04-09T18:30:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a8179c10 0x7fc2a81b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a8179c10 0x7fc2a81b8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55724/0 (socket says 172.21.15.32:55724) 2021-04-09T18:30:19.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a705f700 1 -- 172.21.15.32:0/1518549200 learned_addr learned my addr 172.21.15.32:0/1518549200 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a685e700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2a8180d70 0x7fc2a80ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a605d700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2a80ba950 0x7fc2a81795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a705f700 1 -- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2a8180d70 msgr2=0x7fc2a80ba330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a705f700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2a8180d70 0x7fc2a80ba330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a705f700 1 -- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2a80ba950 msgr2=0x7fc2a81795f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a705f700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2a80ba950 0x7fc2a81795f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a705f700 1 -- 172.21.15.32:0/1518549200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80b1880 con 0x7fc2a8179c10 2021-04-09T18:30:19.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2a705f700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a8179c10 0x7fc2a81b8040 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7fc2a000b5e0 tx=0x7fc2a000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2977fe700 1 -- 172.21.15.32:0/1518549200 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2a000bd70 con 0x7fc2a8179c10 2021-04-09T18:30:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2977fe700 1 -- 172.21.15.32:0/1518549200 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a0006720 con 0x7fc2a8179c10 2021-04-09T18:30:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2977fe700 1 -- 172.21.15.32:0/1518549200 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2a0000c10 con 0x7fc2a8179c10 2021-04-09T18:30:19.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a80b1880 con 0x7fc2a8179c10 2021-04-09T18:30:19.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.590 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a80b1880 con 0x7fc2a8179c10 2021-04-09T18:30:19.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.591 7fc2977fe700 1 -- 172.21.15.32:0/1518549200 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc2a0005330 con 0x7fc2a8179c10 2021-04-09T18:30:19.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.591 7fc2977fe700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc28c0188d0 0x7fc28c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:19.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.591 7fc2977fe700 1 -- 172.21.15.32:0/1518549200 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc2a0032a60 con 0x7fc2a8179c10 2021-04-09T18:30:19.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.591 7fc2a685e700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc28c0188d0 0x7fc28c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:19.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.591 7fc2a685e700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc28c0188d0 0x7fc28c01add0 crc :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:19.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.611 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a80d72a0 con 0x7fc2a8179c10 2021-04-09T18:30:19.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.615 7fc2977fe700 1 -- 172.21.15.32:0/1518549200 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc2a0006370 con 0x7fc2a8179c10 2021-04-09T18:30:19.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.796 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"} v 0) v1 -- 0x7fc2a80d72a0 con 0x7fc2a8179c10 2021-04-09T18:30:19.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.796 7fc2977fe700 1 -- 172.21.15.32:0/1518549200 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"}]=0 v180) v1 ==== 103+0+10 (secure 0 0 0) 0x7fc2a0005330 con 0x7fc2a8179c10 2021-04-09T18:30:19.811 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:19.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc28c0188d0 msgr2=0x7fc28c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc28c0188d0 0x7fc28c01add0 crc :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a8179c10 msgr2=0x7fc2a81b8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:19.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a8179c10 0x7fc2a81b8040 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7fc2a000b5e0 tx=0x7fc2a000b980).stop 2021-04-09T18:30:19.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 shutdown_connections 2021-04-09T18:30:19.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc28c0188d0 0x7fc28c01add0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2a8179c10 0x7fc2a81b8040 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2a80ba950 0x7fc2a81795f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 --2- 172.21.15.32:0/1518549200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2a8180d70 0x7fc2a80ba330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:19.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 >> 172.21.15.32:0/1518549200 conn(0x7fc2a809eb90 msgr2=0x7fc2a80b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:19.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 shutdown_connections 2021-04-09T18:30:19.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.799 7fc2aca63700 1 -- 172.21.15.32:0/1518549200 wait complete. 2021-04-09T18:30:19.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_data pg_num 2021-04-09T18:30:20.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.961 7fa559255700 1 Processor -- start 2021-04-09T18:30:20.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.961 7fa559255700 1 -- start start 2021-04-09T18:30:20.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.962 7fa559255700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa55410d010 0x7fa5540d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.962 7fa559255700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa55410d480 con 0x7fa5540d8490 2021-04-09T18:30:20.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.962 7fa559255700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa55410e1f0 con 0x7fa5540d1da0 2021-04-09T18:30:20.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.962 7fa559255700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55410e780 con 0x7fa55410d010 2021-04-09T18:30:20.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.962 7fa55359e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa5540d8490 0x7fa55410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37088/0 (socket says 172.21.15.32:37088) 2021-04-09T18:30:20.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.962 7fa55359e700 1 -- 172.21.15.32:0/373501121 learned_addr learned my addr 172.21.15.32:0/373501121 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:20.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa552d9d700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa55410d010 0x7fa5540d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656204540 0 0) 0x7fa54c000d30 con 0x7fa5540d8490 2021-04-09T18:30:20.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa540002fe0 con 0x7fa5540d8490 2021-04-09T18:30:20.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900025334 0 0) 0x7fa53c000c60 con 0x7fa5540d1da0 2021-04-09T18:30:20.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa540003210 con 0x7fa5540d1da0 2021-04-09T18:30:20.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2940154369 0 0) 0x7fa54c001460 con 0x7fa5540d8490 2021-04-09T18:30:20.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa55410d010 msgr2=0x7fa5540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa55410d010 0x7fa5540d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 >> v1:172.21.15.86:6789/0 conn(0x7fa5540d1da0 legacy=0x7fa5540d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5540d8d30 con 0x7fa5540d8490 2021-04-09T18:30:20.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fa54c001980 con 0x7fa5540d8490 2021-04-09T18:30:20.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 >> v1:172.21.15.32:6789/0 conn(0x7fa5540d8490 legacy=0x7fa55410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa540003ba0 0x7fa540005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5400067b0 0x7fa540008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa540009520 0x7fa54000ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa540050300 con 0x7fa540003ba0 2021-04-09T18:30:20.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400504b0 con 0x7fa540009520 2021-04-09T18:30:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa552d9d700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa540003ba0 0x7fa540005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa551d9b700 1 -- 172.21.15.32:0/373501121 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400506d0 con 0x7fa5400067b0 2021-04-09T18:30:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa55259c700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5400067b0 0x7fa540008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa55359e700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa540009520 0x7fa54000ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa552d9d700 1 -- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5400067b0 msgr2=0x7fa540008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:20.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa552d9d700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5400067b0 0x7fa540008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa552d9d700 1 -- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa540009520 msgr2=0x7fa54000ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:20.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa552d9d700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa540009520 0x7fa54000ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa552d9d700 1 -- 172.21.15.32:0/373501121 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540050970 con 0x7fa540003ba0 2021-04-09T18:30:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.963 7fa552d9d700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa540003ba0 0x7fa540005fb0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7fa544006ac0 tx=0x7fa544009a30).ready entity=mon.0 client_cookie=f777376989dd8a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:20.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.964 7fa551d9b700 1 -- 172.21.15.32:0/373501121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa54400c0c0 con 0x7fa540003ba0 2021-04-09T18:30:20.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.964 7fa551d9b700 1 -- 172.21.15.32:0/373501121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa544009e20 con 0x7fa540003ba0 2021-04-09T18:30:20.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.964 7fa551d9b700 1 -- 172.21.15.32:0/373501121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa54400c6c0 con 0x7fa540003ba0 2021-04-09T18:30:20.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.964 7fa559255700 1 -- 172.21.15.32:0/373501121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa540003ba0 msgr2=0x7fa540005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.964 7fa559255700 1 --2- 172.21.15.32:0/373501121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa540003ba0 0x7fa540005fb0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7fa544006ac0 tx=0x7fa544009a30).stop 2021-04-09T18:30:20.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.964 7fa55359e700 1 -- 172.21.15.32:0/373501121 reap_dead start 2021-04-09T18:30:20.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.965 7fa559255700 1 -- 172.21.15.32:0/373501121 shutdown_connections 2021-04-09T18:30:20.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.965 7fa559255700 1 -- 172.21.15.32:0/373501121 >> 172.21.15.32:0/373501121 conn(0x7fa5540ce5f0 msgr2=0x7fa5540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:20.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.965 7fa559255700 1 -- 172.21.15.32:0/373501121 shutdown_connections 2021-04-09T18:30:20.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.965 7fa559255700 1 -- 172.21.15.32:0/373501121 wait complete. 2021-04-09T18:30:20.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa559255700 1 Processor -- start 2021-04-09T18:30:20.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa559255700 1 -- start start 2021-04-09T18:30:20.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa559255700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5540a0a10 0x7fa5540bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa559255700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa55418ad00 0x7fa55419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa559255700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5541a03f0 0x7fa554095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa559255700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55418b100 con 0x7fa55418ad00 2021-04-09T18:30:20.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa559255700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa554095860 con 0x7fa5541a03f0 2021-04-09T18:30:20.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa559255700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa554095a80 con 0x7fa5540a0a10 2021-04-09T18:30:20.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa55259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa55418ad00 0x7fa55419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa55259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa55418ad00 0x7fa55419fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55740/0 (socket says 172.21.15.32:55740) 2021-04-09T18:30:20.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa55259c700 1 -- 172.21.15.32:0/2780855145 learned_addr learned my addr 172.21.15.32:0/2780855145 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:20.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.985 7fa552d9d700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5540a0a10 0x7fa5540bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa55359e700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5541a03f0 0x7fa554095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa55259c700 1 -- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5540a0a10 msgr2=0x7fa5540bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa55259c700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5540a0a10 0x7fa5540bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa55259c700 1 -- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5541a03f0 msgr2=0x7fa554095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa55259c700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5541a03f0 0x7fa554095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa55259c700 1 -- 172.21.15.32:0/2780855145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa554189250 con 0x7fa55418ad00 2021-04-09T18:30:20.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa552d9d700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5540a0a10 0x7fa5540bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:20.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa55259c700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa55418ad00 0x7fa55419fdd0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7fa53c0059f0 tx=0x7fa53c00a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:20.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa54b7fe700 1 -- 172.21.15.32:0/2780855145 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa53c00ad10 con 0x7fa55418ad00 2021-04-09T18:30:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa54b7fe700 1 -- 172.21.15.32:0/2780855145 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53c000c10 con 0x7fa55418ad00 2021-04-09T18:30:20.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa54b7fe700 1 -- 172.21.15.32:0/2780855145 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa53c002220 con 0x7fa55418ad00 2021-04-09T18:30:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa559255700 1 -- 172.21.15.32:0/2780855145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa554189250 con 0x7fa55418ad00 2021-04-09T18:30:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.986 7fa559255700 1 -- 172.21.15.32:0/2780855145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa554189250 con 0x7fa55418ad00 2021-04-09T18:30:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.987 7fa54b7fe700 1 -- 172.21.15.32:0/2780855145 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa53c006760 con 0x7fa55418ad00 2021-04-09T18:30:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.987 7fa54b7fe700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa538018930 0x7fa53801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.987 7fa54b7fe700 1 -- 172.21.15.32:0/2780855145 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa53c032a00 con 0x7fa55418ad00 2021-04-09T18:30:20.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.987 7fa552d9d700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa538018930 0x7fa53801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:19.987 7fa552d9d700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa538018930 0x7fa53801adc0 crc :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:20.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.006 7fa559255700 1 -- 172.21.15.32:0/2780855145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa554093d50 con 0x7fa55418ad00 2021-04-09T18:30:20.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.010 7fa54b7fe700 1 -- 172.21.15.32:0/2780855145 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa53c000c10 con 0x7fa55418ad00 2021-04-09T18:30:20.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.189 7fa559255700 1 -- 172.21.15.32:0/2780855145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"} v 0) v1 -- 0x7fa55410d480 con 0x7fa55418ad00 2021-04-09T18:30:20.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.189 7fa54b7fe700 1 -- 172.21.15.32:0/2780855145 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"}]=0 v180) v1 ==== 99+0+10 (secure 0 0 0) 0x7fa53c000e20 con 0x7fa55418ad00 2021-04-09T18:30:20.205 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 -- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa538018930 msgr2=0x7fa53801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa538018930 0x7fa53801adc0 crc :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 -- 172.21.15.32:0/2780855145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa55418ad00 msgr2=0x7fa55419fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa55418ad00 0x7fa55419fdd0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7fa53c0059f0 tx=0x7fa53c00a920).stop 2021-04-09T18:30:20.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 -- 172.21.15.32:0/2780855145 shutdown_connections 2021-04-09T18:30:20.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa538018930 0x7fa53801adc0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa55418ad00 0x7fa55419fdd0 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5541a03f0 0x7fa554095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 --2- 172.21.15.32:0/2780855145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5540a0a10 0x7fa5540bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 -- 172.21.15.32:0/2780855145 >> 172.21.15.32:0/2780855145 conn(0x7fa5540975f0 msgr2=0x7fa554096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 -- 172.21.15.32:0/2780855145 shutdown_connections 2021-04-09T18:30:20.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.192 7fa559255700 1 -- 172.21.15.32:0/2780855145 wait complete. 2021-04-09T18:30:20.217 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:30:20.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.352 7f852b690700 1 Processor -- start 2021-04-09T18:30:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.352 7f852b690700 1 -- start start 2021-04-09T18:30:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f852b690700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852410d010 0x7f85240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f852b690700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85240d1da0 0x7f85240d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f852b690700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852410d480 con 0x7f85240d82c0 2021-04-09T18:30:20.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f852b690700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852410e780 con 0x7f85240d1da0 2021-04-09T18:30:20.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f852b690700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85240d8d30 con 0x7f852410d010 2021-04-09T18:30:20.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f8528cab700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85240d1da0 0x7f85240d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f8528cab700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f85240d1da0 0x7f85240d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55746/0 (socket says 172.21.15.32:55746) 2021-04-09T18:30:20.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f8528cab700 1 -- 172.21.15.32:0/2197734615 learned_addr learned my addr 172.21.15.32:0/2197734615 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f85294ac700 1 --2- 172.21.15.32:0/2197734615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852410d010 0x7f85240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f8528cab700 1 -- 172.21.15.32:0/2197734615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852410d010 msgr2=0x7f85240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f8528cab700 1 --2- 172.21.15.32:0/2197734615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852410d010 0x7f85240d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f8528cab700 1 -- 172.21.15.32:0/2197734615 >> v1:172.21.15.86:6789/0 conn(0x7f85240d82c0 legacy=0x7f852410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f8528cab700 1 -- 172.21.15.32:0/2197734615 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85240d9220 con 0x7f85240d1da0 2021-04-09T18:30:20.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f8528cab700 1 --2- 172.21.15.32:0/2197734615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85240d1da0 0x7f85240d7ca0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f8514006700 tx=0x7f8514009b80).ready entity=mon.0 client_cookie=5438a3b8542819d2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:20.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f851bfff700 1 -- 172.21.15.32:0/2197734615 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f851400c0c0 con 0x7f85240d1da0 2021-04-09T18:30:20.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f851bfff700 1 -- 172.21.15.32:0/2197734615 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851400c440 con 0x7f85240d1da0 2021-04-09T18:30:20.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.353 7f851bfff700 1 -- 172.21.15.32:0/2197734615 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f851400c920 con 0x7f85240d1da0 2021-04-09T18:30:20.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.354 7f852b690700 1 -- 172.21.15.32:0/2197734615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85240d1da0 msgr2=0x7f85240d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.354 7f852b690700 1 --2- 172.21.15.32:0/2197734615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85240d1da0 0x7f85240d7ca0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f8514006700 tx=0x7f8514009b80).stop 2021-04-09T18:30:20.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.354 7f852b690700 1 -- 172.21.15.32:0/2197734615 shutdown_connections 2021-04-09T18:30:20.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.354 7f852b690700 1 --2- 172.21.15.32:0/2197734615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85240d1da0 0x7f85240d7ca0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.354 7f852b690700 1 --2- 172.21.15.32:0/2197734615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852410d010 0x7f85240d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.354 7f852b690700 1 -- 172.21.15.32:0/2197734615 >> 172.21.15.32:0/2197734615 conn(0x7f85240ce5f0 msgr2=0x7f85240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:20.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.354 7f852b690700 1 -- 172.21.15.32:0/2197734615 shutdown_connections 2021-04-09T18:30:20.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.354 7f852b690700 1 -- 172.21.15.32:0/2197734615 wait complete. 2021-04-09T18:30:20.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f852b690700 1 Processor -- start 2021-04-09T18:30:20.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f852b690700 1 -- start start 2021-04-09T18:30:20.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f852b690700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85241aca20 0x7f85241ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f852b690700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852418bd40 0x7f852418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f852b690700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f852408fba0 0x7f852408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f852b690700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852418c7e0 con 0x7f852408fba0 2021-04-09T18:30:20.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f852b690700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852418ca30 con 0x7f85241aca20 2021-04-09T18:30:20.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f852b690700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85240905f0 con 0x7f852418bd40 2021-04-09T18:30:20.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f8529cad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f852408fba0 0x7f852408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f8529cad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f852408fba0 0x7f852408ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55752/0 (socket says 172.21.15.32:55752) 2021-04-09T18:30:20.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.375 7f8529cad700 1 -- 172.21.15.32:0/3338464689 learned_addr learned my addr 172.21.15.32:0/3338464689 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:20.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f85294ac700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85241aca20 0x7f85241ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8528cab700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852418bd40 0x7f852418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8529cad700 1 -- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852418bd40 msgr2=0x7f852418c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8529cad700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852418bd40 0x7f852418c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8529cad700 1 -- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85241aca20 msgr2=0x7f85241ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8529cad700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85241aca20 0x7f85241ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8529cad700 1 -- 172.21.15.32:0/3338464689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85240b4300 con 0x7f852408fba0 2021-04-09T18:30:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8529cad700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f852408fba0 0x7f852408ffa0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f851c0098f0 tx=0x7f851c00bbc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8519ffb700 1 -- 172.21.15.32:0/3338464689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f851c005830 con 0x7f852408fba0 2021-04-09T18:30:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8519ffb700 1 -- 172.21.15.32:0/3338464689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c005b70 con 0x7f852408fba0 2021-04-09T18:30:20.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f8519ffb700 1 -- 172.21.15.32:0/3338464689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f851c006030 con 0x7f852408fba0 2021-04-09T18:30:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f852b690700 1 -- 172.21.15.32:0/3338464689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85240b4300 con 0x7f852408fba0 2021-04-09T18:30:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.376 7f852b690700 1 -- 172.21.15.32:0/3338464689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85240b4300 con 0x7f852408fba0 2021-04-09T18:30:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.377 7f8519ffb700 1 -- 172.21.15.32:0/3338464689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f851c006030 con 0x7f852408fba0 2021-04-09T18:30:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.377 7f8519ffb700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8504018930 0x7f850401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.377 7f8519ffb700 1 -- 172.21.15.32:0/3338464689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f851c0371d0 con 0x7f852408fba0 2021-04-09T18:30:20.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.377 7f85294ac700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8504018930 0x7f850401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.377 7f85294ac700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8504018930 0x7f850401ae10 crc :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:20.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.397 7f852b690700 1 -- 172.21.15.32:0/3338464689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8524182d70 con 0x7f852408fba0 2021-04-09T18:30:20.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.400 7f8519ffb700 1 -- 172.21.15.32:0/3338464689 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f851c005830 con 0x7f852408fba0 2021-04-09T18:30:20.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.577 7f852b690700 1 -- 172.21.15.32:0/3338464689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f852418c760 con 0x7f852408fba0 2021-04-09T18:30:20.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.577 7f8519ffb700 1 -- 172.21.15.32:0/3338464689 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v180) v1 ==== 84+0+206 (secure 0 0 0) 0x7f851c005b70 con 0x7f852408fba0 2021-04-09T18:30:20.592 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:20.592 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:30:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:30:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:30:20.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 30, 2021-04-09T18:30:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "alpha_metadata" 2021-04-09T18:30:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:20.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:30:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 31, 2021-04-09T18:30:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "alpha_data" 2021-04-09T18:30:20.595 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:20.595 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:30:20.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 -- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8504018930 msgr2=0x7f850401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8504018930 0x7f850401ae10 crc :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 -- 172.21.15.32:0/3338464689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f852408fba0 msgr2=0x7f852408ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f852408fba0 0x7f852408ffa0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7f851c0098f0 tx=0x7f851c00bbc0).stop 2021-04-09T18:30:20.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 -- 172.21.15.32:0/3338464689 shutdown_connections 2021-04-09T18:30:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8504018930 0x7f850401ae10 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f852408fba0 0x7f852408ffa0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f852418bd40 0x7f852418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 --2- 172.21.15.32:0/3338464689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85241aca20 0x7f85241ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 -- 172.21.15.32:0/3338464689 >> 172.21.15.32:0/3338464689 conn(0x7f852419b460 msgr2=0x7f852419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 -- 172.21.15.32:0/3338464689 shutdown_connections 2021-04-09T18:30:20.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.580 7f852b690700 1 -- 172.21.15.32:0/3338464689 wait complete. 2021-04-09T18:30:20.604 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'bravo' 2021-04-09T18:30:20.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create bravo_metadata 8 2021-04-09T18:30:20.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c7ed79700 1 Processor -- start 2021-04-09T18:30:20.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c7ed79700 1 -- start start 2021-04-09T18:30:20.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c7ed79700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c7810d010 0x7f3c780d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c7ed79700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3c780d8490 0x7f3c7810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c7ed79700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7810d480 con 0x7f3c780d1da0 2021-04-09T18:30:20.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c7ed79700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780d8d30 con 0x7f3c7810d010 2021-04-09T18:30:20.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c7ed79700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780d9220 con 0x7f3c780d8490 2021-04-09T18:30:20.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c7cb95700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c7810d010 0x7f3c780d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c77fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3c780d1da0 0x7f3c780d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37120/0 (socket says 172.21.15.32:37120) 2021-04-09T18:30:20.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.743 7f3c77fff700 1 -- 172.21.15.32:0/1142066548 learned_addr learned my addr 172.21.15.32:0/1142066548 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:20.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c7d396700 1 --2- 172.21.15.32:0/1142066548 >> v2:172.21.15.86:3300/0 conn(0x7f3c780d8490 0x7f3c7810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c7cb95700 1 -- 172.21.15.32:0/1142066548 >> v2:172.21.15.86:3300/0 conn(0x7f3c780d8490 msgr2=0x7f3c7810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:20.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c7cb95700 1 --2- 172.21.15.32:0/1142066548 >> v2:172.21.15.86:3300/0 conn(0x7f3c780d8490 0x7f3c7810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c7cb95700 1 -- 172.21.15.32:0/1142066548 >> v1:172.21.15.32:6789/0 conn(0x7f3c780d1da0 legacy=0x7f3c780d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c7cb95700 1 -- 172.21.15.32:0/1142066548 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7810e020 con 0x7f3c7810d010 2021-04-09T18:30:20.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c777fe700 1 -- 172.21.15.32:0/1142066548 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217783036 0 0) 0x7f3c6c000ce0 con 0x7f3c780d1da0 2021-04-09T18:30:20.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c7cb95700 1 --2- 172.21.15.32:0/1142066548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c7810d010 0x7f3c780d1780 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f3c68006650 tx=0x7f3c68009a00).ready entity=mon.0 client_cookie=95fc9a231de6dc82 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:20.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c777fe700 1 -- 172.21.15.32:0/1142066548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c68009df0 con 0x7f3c7810d010 2021-04-09T18:30:20.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c777fe700 1 -- 172.21.15.32:0/1142066548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c680009b0 con 0x7f3c7810d010 2021-04-09T18:30:20.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c777fe700 1 -- 172.21.15.32:0/1142066548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c6800c2b0 con 0x7f3c7810d010 2021-04-09T18:30:20.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c7ed79700 1 -- 172.21.15.32:0/1142066548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c7810d010 msgr2=0x7f3c780d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.744 7f3c7ed79700 1 --2- 172.21.15.32:0/1142066548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c7810d010 0x7f3c780d1780 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f3c68006650 tx=0x7f3c68009a00).stop 2021-04-09T18:30:20.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.745 7f3c7ed79700 1 -- 172.21.15.32:0/1142066548 shutdown_connections 2021-04-09T18:30:20.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.745 7f3c7ed79700 1 --2- 172.21.15.32:0/1142066548 >> v2:172.21.15.86:3300/0 conn(0x7f3c780d8490 0x7f3c7810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.745 7f3c7ed79700 1 --2- 172.21.15.32:0/1142066548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c7810d010 0x7f3c780d1780 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.745 7f3c7ed79700 1 -- 172.21.15.32:0/1142066548 >> 172.21.15.32:0/1142066548 conn(0x7f3c780ce5f0 msgr2=0x7f3c780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:20.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.745 7f3c7ed79700 1 -- 172.21.15.32:0/1142066548 shutdown_connections 2021-04-09T18:30:20.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.745 7f3c7ed79700 1 -- 172.21.15.32:0/1142066548 wait complete. 2021-04-09T18:30:20.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.765 7f3c7ed79700 1 Processor -- start 2021-04-09T18:30:20.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.765 7f3c7ed79700 1 -- start start 2021-04-09T18:30:20.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7ed79700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c78195110 0x7f3c7808c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7ed79700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c7808cc50 0x7f3c780a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7ed79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780a0fe0 0x7f3c78091080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7ed79700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7808e510 con 0x7f3c780a0fe0 2021-04-09T18:30:20.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7ed79700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7808d210 con 0x7f3c78195110 2021-04-09T18:30:20.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7ed79700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780916a0 con 0x7f3c7808cc50 2021-04-09T18:30:20.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7d396700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780a0fe0 0x7f3c78091080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7d396700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780a0fe0 0x7f3c78091080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55766/0 (socket says 172.21.15.32:55766) 2021-04-09T18:30:20.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7d396700 1 -- 172.21.15.32:0/1461302767 learned_addr learned my addr 172.21.15.32:0/1461302767 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:20.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7cb95700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c78195110 0x7f3c7808c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c77fff700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c7808cc50 0x7f3c780a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7d396700 1 -- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c7808cc50 msgr2=0x7f3c780a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7d396700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c7808cc50 0x7f3c780a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7d396700 1 -- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c78195110 msgr2=0x7f3c7808c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:20.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7d396700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c78195110 0x7f3c7808c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:20.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7d396700 1 -- 172.21.15.32:0/1461302767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c78091880 con 0x7f3c780a0fe0 2021-04-09T18:30:20.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.766 7f3c7d396700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780a0fe0 0x7f3c78091080 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f3c7000b910 tx=0x7f3c7000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.767 7f3c757fa700 1 -- 172.21.15.32:0/1461302767 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c70006aa0 con 0x7f3c780a0fe0 2021-04-09T18:30:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.767 7f3c757fa700 1 -- 172.21.15.32:0/1461302767 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c700011f0 con 0x7f3c780a0fe0 2021-04-09T18:30:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.767 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c78091880 con 0x7f3c780a0fe0 2021-04-09T18:30:20.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.767 7f3c757fa700 1 -- 172.21.15.32:0/1461302767 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c70001640 con 0x7f3c780a0fe0 2021-04-09T18:30:20.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.767 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c78091880 con 0x7f3c780a0fe0 2021-04-09T18:30:20.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.767 7f3c757fa700 1 -- 172.21.15.32:0/1461302767 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3c700056b0 con 0x7f3c780a0fe0 2021-04-09T18:30:20.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.767 7f3c757fa700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c58018930 0x7f3c5801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:20.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.768 7f3c757fa700 1 -- 172.21.15.32:0/1461302767 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3c70032da0 con 0x7f3c780a0fe0 2021-04-09T18:30:20.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.768 7f3c7cb95700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c58018930 0x7f3c5801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:20.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.768 7f3c7cb95700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c58018930 0x7f3c5801ae10 crc :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:20.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.787 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7808d210 con 0x7f3c780a0fe0 2021-04-09T18:30:20.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.791 7f3c757fa700 1 -- 172.21.15.32:0/1461302767 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3c7000bdf0 con 0x7f3c780a0fe0 2021-04-09T18:30:20.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:20.972 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_metadata"} v 0) v1 -- 0x7f3c7808d210 con 0x7f3c780a0fe0 2021-04-09T18:30:21.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.785 7f3c757fa700 1 -- 172.21.15.32:0/1461302767 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_metadata"}]=0 pool 'bravo_metadata' created v181) v1 ==== 131+0+0 (secure 0 0 0) 0x7f3c700056b0 con 0x7f3c780a0fe0 2021-04-09T18:30:21.801 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_metadata' created 2021-04-09T18:30:21.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c58018930 msgr2=0x7f3c5801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:21.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c58018930 0x7f3c5801ae10 crc :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:21.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780a0fe0 msgr2=0x7f3c78091080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:21.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780a0fe0 0x7f3c78091080 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f3c7000b910 tx=0x7f3c7000bcb0).stop 2021-04-09T18:30:21.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 shutdown_connections 2021-04-09T18:30:21.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c58018930 0x7f3c5801ae10 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:21.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c780a0fe0 0x7f3c78091080 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:21.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c7808cc50 0x7f3c780a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:21.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 --2- 172.21.15.32:0/1461302767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c78195110 0x7f3c7808c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 >> 172.21.15.32:0/1461302767 conn(0x7f3c780b2fa0 msgr2=0x7f3c780a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 shutdown_connections 2021-04-09T18:30:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.790 7f3c7ed79700 1 -- 172.21.15.32:0/1461302767 wait complete. 2021-04-09T18:30:21.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create bravo_data 8 2021-04-09T18:30:21.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.953 7f8e50675700 1 Processor -- start 2021-04-09T18:30:21.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.953 7f8e50675700 1 -- start start 2021-04-09T18:30:21.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e50675700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8e4810d010 0x7f8e480d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:21.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e50675700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8e480d1da0 0x7f8e480d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:21.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e50675700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4810d480 con 0x7f8e480d82c0 2021-04-09T18:30:21.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e50675700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e480d8d30 con 0x7f8e480d1da0 2021-04-09T18:30:21.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e50675700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e480d9220 con 0x7f8e4810d010 2021-04-09T18:30:21.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e4dc90700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8e480d1da0 0x7f8e480d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:21.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e4dc90700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8e480d1da0 0x7f8e480d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55776/0 (socket says 172.21.15.32:55776) 2021-04-09T18:30:21.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e4dc90700 1 -- 172.21.15.32:0/1462705735 learned_addr learned my addr 172.21.15.32:0/1462705735 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:21.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e4e491700 1 --2- 172.21.15.32:0/1462705735 >> v2:172.21.15.86:3300/0 conn(0x7f8e4810d010 0x7f8e480d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:21.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e4dc90700 1 -- 172.21.15.32:0/1462705735 >> v2:172.21.15.86:3300/0 conn(0x7f8e4810d010 msgr2=0x7f8e480d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:21.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e4dc90700 1 --2- 172.21.15.32:0/1462705735 >> v2:172.21.15.86:3300/0 conn(0x7f8e4810d010 0x7f8e480d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:21.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e4dc90700 1 -- 172.21.15.32:0/1462705735 >> v1:172.21.15.86:6789/0 conn(0x7f8e480d82c0 legacy=0x7f8e4810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:21.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e4dc90700 1 -- 172.21.15.32:0/1462705735 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e480d86f0 con 0x7f8e480d1da0 2021-04-09T18:30:21.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.954 7f8e4dc90700 1 --2- 172.21.15.32:0/1462705735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480d1da0 0x7f8e480d7ca0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7f8e3c006700 tx=0x7f8e3c009b80).ready entity=mon.0 client_cookie=c18ff7e79f7c7674 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:21.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.955 7f8e4d48f700 1 -- 172.21.15.32:0/1462705735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e3c00c0c0 con 0x7f8e480d1da0 2021-04-09T18:30:21.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.955 7f8e4d48f700 1 -- 172.21.15.32:0/1462705735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c00c440 con 0x7f8e480d1da0 2021-04-09T18:30:21.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.955 7f8e4d48f700 1 -- 172.21.15.32:0/1462705735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e3c00c920 con 0x7f8e480d1da0 2021-04-09T18:30:21.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.955 7f8e50675700 1 -- 172.21.15.32:0/1462705735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480d1da0 msgr2=0x7f8e480d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:21.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.955 7f8e50675700 1 --2- 172.21.15.32:0/1462705735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480d1da0 0x7f8e480d7ca0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7f8e3c006700 tx=0x7f8e3c009b80).stop 2021-04-09T18:30:21.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.956 7f8e50675700 1 -- 172.21.15.32:0/1462705735 shutdown_connections 2021-04-09T18:30:21.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.956 7f8e50675700 1 --2- 172.21.15.32:0/1462705735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480d1da0 0x7f8e480d7ca0 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:21.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.956 7f8e50675700 1 --2- 172.21.15.32:0/1462705735 >> v2:172.21.15.86:3300/0 conn(0x7f8e4810d010 0x7f8e480d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:22.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.956 7f8e50675700 1 -- 172.21.15.32:0/1462705735 >> 172.21.15.32:0/1462705735 conn(0x7f8e480ce5f0 msgr2=0x7f8e480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:22.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.956 7f8e50675700 1 -- 172.21.15.32:0/1462705735 shutdown_connections 2021-04-09T18:30:22.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.956 7f8e50675700 1 -- 172.21.15.32:0/1462705735 wait complete. 2021-04-09T18:30:22.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e50675700 1 Processor -- start 2021-04-09T18:30:22.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e50675700 1 -- start start 2021-04-09T18:30:22.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e50675700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e480b2140 0x7f8e480a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:22.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e50675700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480b2ee0 0x7f8e480bf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:22.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e50675700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e480b32e0 0x7f8e480bf830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:22.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e50675700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e48197c50 con 0x7f8e480b2ee0 2021-04-09T18:30:22.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e50675700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e48197e70 con 0x7f8e480b32e0 2021-04-09T18:30:22.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e50675700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e48198090 con 0x7f8e480b2140 2021-04-09T18:30:22.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e4dc90700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480b2ee0 0x7f8e480bf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:22.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e4dc90700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480b2ee0 0x7f8e480bf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55780/0 (socket says 172.21.15.32:55780) 2021-04-09T18:30:22.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.976 7f8e4dc90700 1 -- 172.21.15.32:0/3420347419 learned_addr learned my addr 172.21.15.32:0/3420347419 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:22.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e4ec92700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e480b32e0 0x7f8e480bf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:22.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e4e491700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e480b2140 0x7f8e480a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:22.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e4dc90700 1 -- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e480b2140 msgr2=0x7f8e480a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:22.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e4dc90700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e480b2140 0x7f8e480a0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:22.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e4dc90700 1 -- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e480b32e0 msgr2=0x7f8e480bf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:22.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e4dc90700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e480b32e0 0x7f8e480bf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:22.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e4dc90700 1 -- 172.21.15.32:0/3420347419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e481982b0 con 0x7f8e480b2ee0 2021-04-09T18:30:22.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e4dc90700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480b2ee0 0x7f8e480bf210 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f8e3c00b650 tx=0x7f8e3c00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:22.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e46ffd700 1 -- 172.21.15.32:0/3420347419 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e3c00bde0 con 0x7f8e480b2ee0 2021-04-09T18:30:22.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e46ffd700 1 -- 172.21.15.32:0/3420347419 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c000c90 con 0x7f8e480b2ee0 2021-04-09T18:30:22.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e50675700 1 -- 172.21.15.32:0/3420347419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e481982b0 con 0x7f8e480b2ee0 2021-04-09T18:30:22.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e46ffd700 1 -- 172.21.15.32:0/3420347419 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e3c002220 con 0x7f8e480b2ee0 2021-04-09T18:30:22.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.977 7f8e50675700 1 -- 172.21.15.32:0/3420347419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e481982b0 con 0x7f8e480b2ee0 2021-04-09T18:30:22.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.978 7f8e46ffd700 1 -- 172.21.15.32:0/3420347419 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8e3c009f90 con 0x7f8e480b2ee0 2021-04-09T18:30:22.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.978 7f8e46ffd700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e28018930 0x7f8e2801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:22.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.978 7f8e46ffd700 1 -- 172.21.15.32:0/3420347419 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5926+0+0 (secure 0 0 0) 0x7f8e3c032b50 con 0x7f8e480b2ee0 2021-04-09T18:30:22.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.978 7f8e4e491700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e28018930 0x7f8e2801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:22.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.978 7f8e4e491700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e28018930 0x7f8e2801adc0 crc :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:22.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:21.997 7f8e50675700 1 -- 172.21.15.32:0/3420347419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e481b7040 con 0x7f8e480b2ee0 2021-04-09T18:30:22.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.001 7f8e46ffd700 1 -- 172.21.15.32:0/3420347419 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8e3c0091e0 con 0x7f8e480b2ee0 2021-04-09T18:30:22.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.185 7f8e50675700 1 -- 172.21.15.32:0/3420347419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_data"} v 0) v1 -- 0x7f8e4810d480 con 0x7f8e480b2ee0 2021-04-09T18:30:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.787 7f8e46ffd700 1 -- 172.21.15.32:0/3420347419 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_data"}]=0 pool 'bravo_data' created v182) v1 ==== 123+0+0 (secure 0 0 0) 0x7f8e3c009f90 con 0x7f8e480b2ee0 2021-04-09T18:30:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_data' created 2021-04-09T18:30:22.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 -- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e28018930 msgr2=0x7f8e2801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:22.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e28018930 0x7f8e2801adc0 crc :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:22.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 -- 172.21.15.32:0/3420347419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480b2ee0 msgr2=0x7f8e480bf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:22.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480b2ee0 0x7f8e480bf210 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f8e3c00b650 tx=0x7f8e3c00b9f0).stop 2021-04-09T18:30:22.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 -- 172.21.15.32:0/3420347419 shutdown_connections 2021-04-09T18:30:22.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8e28018930 0x7f8e2801adc0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:22.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e480b2ee0 0x7f8e480bf210 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:22.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e480b32e0 0x7f8e480bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:22.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 --2- 172.21.15.32:0/3420347419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e480b2140 0x7f8e480a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:22.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 -- 172.21.15.32:0/3420347419 >> 172.21.15.32:0/3420347419 conn(0x7f8e480b27e0 msgr2=0x7f8e481b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:22.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 -- 172.21.15.32:0/3420347419 shutdown_connections 2021-04-09T18:30:22.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.803 7f8e50675700 1 -- 172.21.15.32:0/3420347419 wait complete. 2021-04-09T18:30:22.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new bravo bravo_metadata bravo_data --force 2021-04-09T18:30:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.976 7fd48d03c700 1 Processor -- start 2021-04-09T18:30:23.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.976 7fd48d03c700 1 -- start start 2021-04-09T18:30:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48d03c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd4880d7cc0 0x7fd4880d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48d03c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48810d500 con 0x7fd48810da80 2021-04-09T18:30:23.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48d03c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880d0330 con 0x7fd48810d050 2021-04-09T18:30:23.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48d03c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4880d9260 con 0x7fd4880d7cc0 2021-04-09T18:30:23.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48659c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd4880d7cc0 0x7fd4880d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48759e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd48810da80 0x7fd48810de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37146/0 (socket says 172.21.15.32:37146) 2021-04-09T18:30:23.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48759e700 1 -- 172.21.15.32:0/844190417 learned_addr learned my addr 172.21.15.32:0/844190417 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:23.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48659c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd4880d7cc0 0x7fd4880d80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55790/0 (socket says 172.21.15.32:55790) 2021-04-09T18:30:23.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48659c700 1 -- 172.21.15.32:0/844190417 >> v1:172.21.15.86:6789/0 conn(0x7fd48810d050 legacy=0x7fd4880cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48659c700 1 -- 172.21.15.32:0/844190417 >> v1:172.21.15.32:6789/0 conn(0x7fd48810da80 legacy=0x7fd48810de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48659c700 1 -- 172.21.15.32:0/844190417 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4880d86e0 con 0x7fd4880d7cc0 2021-04-09T18:30:23.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd485d9b700 1 -- 172.21.15.32:0/844190417 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4487946 0 0) 0x7fd480000d30 con 0x7fd48810da80 2021-04-09T18:30:23.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.977 7fd48659c700 1 --2- 172.21.15.32:0/844190417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880d7cc0 0x7fd4880d80c0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7fd47c006650 tx=0x7fd47c009a00).ready entity=mon.0 client_cookie=f394c3ae09266649 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:23.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.978 7fd485d9b700 1 -- 172.21.15.32:0/844190417 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd47c009df0 con 0x7fd4880d7cc0 2021-04-09T18:30:23.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.978 7fd485d9b700 1 -- 172.21.15.32:0/844190417 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c0009b0 con 0x7fd4880d7cc0 2021-04-09T18:30:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.978 7fd485d9b700 1 -- 172.21.15.32:0/844190417 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd47c009df0 con 0x7fd4880d7cc0 2021-04-09T18:30:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.978 7fd48d03c700 1 -- 172.21.15.32:0/844190417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880d7cc0 msgr2=0x7fd4880d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.978 7fd48d03c700 1 --2- 172.21.15.32:0/844190417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880d7cc0 0x7fd4880d80c0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7fd47c006650 tx=0x7fd47c009a00).stop 2021-04-09T18:30:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.979 7fd48d03c700 1 -- 172.21.15.32:0/844190417 shutdown_connections 2021-04-09T18:30:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.979 7fd48d03c700 1 --2- 172.21.15.32:0/844190417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880d7cc0 0x7fd4880d80c0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.979 7fd48d03c700 1 -- 172.21.15.32:0/844190417 >> 172.21.15.32:0/844190417 conn(0x7fd4880ce620 msgr2=0x7fd4880d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.979 7fd48d03c700 1 -- 172.21.15.32:0/844190417 shutdown_connections 2021-04-09T18:30:23.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.979 7fd48d03c700 1 -- 172.21.15.32:0/844190417 wait complete. 2021-04-09T18:30:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48d03c700 1 Processor -- start 2021-04-09T18:30:23.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48d03c700 1 -- start start 2021-04-09T18:30:23.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48d03c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4880a42b0 0x7fd4880ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48d03c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880b5e00 0x7fd4880ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48d03c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4881998f0 0x7fd488199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48d03c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4880b62d0 con 0x7fd4880b5e00 2021-04-09T18:30:23.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48d03c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4880aec80 con 0x7fd4881998f0 2021-04-09T18:30:23.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48d03c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48819a310 con 0x7fd4880a42b0 2021-04-09T18:30:23.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880b5e00 0x7fd4880ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880b5e00 0x7fd4880ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55792/0 (socket says 172.21.15.32:55792) 2021-04-09T18:30:23.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48659c700 1 -- 172.21.15.32:0/3692782631 learned_addr learned my addr 172.21.15.32:0/3692782631 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:23.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd48759e700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4881998f0 0x7fd488199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:22.999 7fd486d9d700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4880a42b0 0x7fd4880ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd48659c700 1 -- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4880a42b0 msgr2=0x7fd4880ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd48659c700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4880a42b0 0x7fd4880ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd48659c700 1 -- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4881998f0 msgr2=0x7fd488199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd48659c700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4881998f0 0x7fd488199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd48659c700 1 -- 172.21.15.32:0/3692782631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48819a530 con 0x7fd4880b5e00 2021-04-09T18:30:23.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd48659c700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880b5e00 0x7fd4880ae660 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7fd47c0096a0 tx=0x7fd47c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd4777fe700 1 -- 172.21.15.32:0/3692782631 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd47c000c90 con 0x7fd4880b5e00 2021-04-09T18:30:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd4777fe700 1 -- 172.21.15.32:0/3692782631 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c002030 con 0x7fd4880b5e00 2021-04-09T18:30:23.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd4777fe700 1 -- 172.21.15.32:0/3692782631 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd47c002550 con 0x7fd4880b5e00 2021-04-09T18:30:23.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48819a530 con 0x7fd4880b5e00 2021-04-09T18:30:23.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.000 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48819a530 con 0x7fd4880b5e00 2021-04-09T18:30:23.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.001 7fd4777fe700 1 -- 172.21.15.32:0/3692782631 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd47c002210 con 0x7fd4880b5e00 2021-04-09T18:30:23.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.001 7fd4777fe700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd46c018930 0x7fd46c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.001 7fd4777fe700 1 -- 172.21.15.32:0/3692782631 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6275+0+0 (secure 0 0 0) 0x7fd47c032d30 con 0x7fd4880b5e00 2021-04-09T18:30:23.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.001 7fd486d9d700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd46c018930 0x7fd46c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.001 7fd486d9d700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd46c018930 0x7fd46c01adc0 crc :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:23.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.021 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4880b2d70 con 0x7fd4880b5e00 2021-04-09T18:30:23.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.025 7fd4777fe700 1 -- 172.21.15.32:0/3692782631 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd47c0009c0 con 0x7fd4880b5e00 2021-04-09T18:30:23.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.207 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "bravo_data", "prefix": "fs new", "force": true, "fs_name": "bravo", "metadata": "bravo_metadata"} v 0) v1 -- 0x7fd48810d500 con 0x7fd4880b5e00 2021-04-09T18:30:23.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.211 7fd4777fe700 1 -- 172.21.15.32:0/3692782631 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "bravo_data", "prefix": "fs new", "force": true, "fs_name": "bravo", "metadata": "bravo_metadata"}]=0 new fs with metadata pool 32 and data pool 33 v326) v1 ==== 186+0+0 (secure 0 0 0) 0x7fd47c002010 con 0x7fd4880b5e00 2021-04-09T18:30:23.231 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 32 and data pool 33 2021-04-09T18:30:23.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd46c018930 msgr2=0x7fd46c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd46c018930 0x7fd46c01adc0 crc :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880b5e00 msgr2=0x7fd4880ae660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880b5e00 0x7fd4880ae660 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7fd47c0096a0 tx=0x7fd47c00b990).stop 2021-04-09T18:30:23.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 shutdown_connections 2021-04-09T18:30:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd46c018930 0x7fd46c01adc0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4880b5e00 0x7fd4880ae660 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4881998f0 0x7fd488199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 --2- 172.21.15.32:0/3692782631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4880a42b0 0x7fd4880ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.219 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 >> 172.21.15.32:0/3692782631 conn(0x7fd488097a20 msgr2=0x7fd4880c0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:23.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.221 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 shutdown_connections 2021-04-09T18:30:23.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.222 7fd48d03c700 1 -- 172.21.15.32:0/3692782631 wait complete. 2021-04-09T18:30:23.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:23.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.383 7f53d67b3700 1 Processor -- start 2021-04-09T18:30:23.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.383 7f53d67b3700 1 -- start start 2021-04-09T18:30:23.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d67b3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f53d010da00 0x7f53d010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d67b3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d010d480 con 0x7f53d00d7ca0 2021-04-09T18:30:23.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d67b3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d00d1f70 con 0x7f53d010d010 2021-04-09T18:30:23.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d67b3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d00d8d30 con 0x7f53d010da00 2021-04-09T18:30:23.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d4dd0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f53d010da00 0x7f53d010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53c7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f53d00d7ca0 0x7f53d00d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37170/0 (socket says 172.21.15.32:37170) 2021-04-09T18:30:23.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53c7fff700 1 -- 172.21.15.32:0/195299228 learned_addr learned my addr 172.21.15.32:0/195299228 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:23.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d4dd0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f53d010da00 0x7f53d010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55812/0 (socket says 172.21.15.32:55812) 2021-04-09T18:30:23.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d4dd0700 1 -- 172.21.15.32:0/195299228 >> v1:172.21.15.86:6789/0 conn(0x7f53d010d010 legacy=0x7f53d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53cf7fe700 1 -- 172.21.15.32:0/195299228 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709495548 0 0) 0x7f53b8000ce0 con 0x7f53d00d7ca0 2021-04-09T18:30:23.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d4dd0700 1 -- 172.21.15.32:0/195299228 >> v1:172.21.15.32:6789/0 conn(0x7f53d00d7ca0 legacy=0x7f53d00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d4dd0700 1 -- 172.21.15.32:0/195299228 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d00d9220 con 0x7f53d010da00 2021-04-09T18:30:23.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.384 7f53d4dd0700 1 --2- 172.21.15.32:0/195299228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d010da00 0x7f53d010de00 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7f53c8006e30 tx=0x7f53c8009de0).ready entity=mon.0 client_cookie=4ad959f1dbdc2954 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:23.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53cf7fe700 1 -- 172.21.15.32:0/195299228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53c800b300 con 0x7f53d010da00 2021-04-09T18:30:23.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53cf7fe700 1 -- 172.21.15.32:0/195299228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c800b680 con 0x7f53d010da00 2021-04-09T18:30:23.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53cf7fe700 1 -- 172.21.15.32:0/195299228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53c800bb60 con 0x7f53d010da00 2021-04-09T18:30:23.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53d67b3700 1 -- 172.21.15.32:0/195299228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d010da00 msgr2=0x7f53d010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53d67b3700 1 --2- 172.21.15.32:0/195299228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d010da00 0x7f53d010de00 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7f53c8006e30 tx=0x7f53c8009de0).stop 2021-04-09T18:30:23.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53d67b3700 1 -- 172.21.15.32:0/195299228 shutdown_connections 2021-04-09T18:30:23.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53d67b3700 1 --2- 172.21.15.32:0/195299228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d010da00 0x7f53d010de00 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53d67b3700 1 -- 172.21.15.32:0/195299228 >> 172.21.15.32:0/195299228 conn(0x7f53d00ce5f0 msgr2=0x7f53d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:23.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53d67b3700 1 -- 172.21.15.32:0/195299228 shutdown_connections 2021-04-09T18:30:23.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.385 7f53d67b3700 1 -- 172.21.15.32:0/195299228 wait complete. 2021-04-09T18:30:23.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53d67b3700 1 Processor -- start 2021-04-09T18:30:23.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53d67b3700 1 -- start start 2021-04-09T18:30:23.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53d67b3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d01acea0 0x7f53d01ac750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53d67b3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53d0197ca0 0x7f53d01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53d67b3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53d0198eb0 0x7f53d01992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53d67b3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d01986c0 con 0x7f53d01acea0 2021-04-09T18:30:23.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53d67b3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d0198930 con 0x7f53d0198eb0 2021-04-09T18:30:23.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53d67b3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d01ad2a0 con 0x7f53d0197ca0 2021-04-09T18:30:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d01acea0 0x7f53d01ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53cffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d01acea0 0x7f53d01ac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55816/0 (socket says 172.21.15.32:55816) 2021-04-09T18:30:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53cffff700 1 -- 172.21.15.32:0/2833292526 learned_addr learned my addr 172.21.15.32:0/2833292526 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:23.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53d4dd0700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53d0198eb0 0x7f53d01992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.406 7f53c7fff700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53d0197ca0 0x7f53d01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cffff700 1 -- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53d0197ca0 msgr2=0x7f53d01980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cffff700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53d0197ca0 0x7f53d01980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cffff700 1 -- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53d0198eb0 msgr2=0x7f53d01992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cffff700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53d0198eb0 0x7f53d01992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cffff700 1 -- 172.21.15.32:0/2833292526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d01998d0 con 0x7f53d01acea0 2021-04-09T18:30:23.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cffff700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d01acea0 0x7f53d01ac750 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7f53c0006f20 tx=0x7f53c000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:23.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cd7fa700 1 -- 172.21.15.32:0/2833292526 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53c000bd80 con 0x7f53d01acea0 2021-04-09T18:30:23.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cd7fa700 1 -- 172.21.15.32:0/2833292526 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c0006030 con 0x7f53d01acea0 2021-04-09T18:30:23.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d01998d0 con 0x7f53d01acea0 2021-04-09T18:30:23.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cd7fa700 1 -- 172.21.15.32:0/2833292526 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53c0005520 con 0x7f53d01acea0 2021-04-09T18:30:23.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d01998d0 con 0x7f53d01acea0 2021-04-09T18:30:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.407 7f53cd7fa700 1 -- 172.21.15.32:0/2833292526 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f53c0006210 con 0x7f53d01acea0 2021-04-09T18:30:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.408 7f53cd7fa700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53b40188c0 0x7f53b401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.408 7f53cd7fa700 1 -- 172.21.15.32:0/2833292526 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f53c0032ba0 con 0x7f53d01acea0 2021-04-09T18:30:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.408 7f53c7fff700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53b40188c0 0x7f53b401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.408 7f53c7fff700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53b40188c0 0x7f53b401adc0 crc :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:23.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.427 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d0184b30 con 0x7f53d01acea0 2021-04-09T18:30:23.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.431 7f53cd7fa700 1 -- 172.21.15.32:0/2833292526 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f53c0006bb0 con 0x7f53d01acea0 2021-04-09T18:30:23.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.619 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53d0184b30 con 0x7f53d01acea0 2021-04-09T18:30:23.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.620 7f53cd7fa700 1 -- 172.21.15.32:0/2833292526 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+16865 (secure 0 0 0) 0x7f53c0006bb0 con 0x7f53d01acea0 2021-04-09T18:30:23.636 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:23.637 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":183,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:23.208745","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:30:13.709244","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":31,"pool_name":"alpha_data","create_time":"2021-04-09 18:30:14.523869","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":32,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:30:21.008022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":33,"pool_name":"bravo_data","create_time":"2021-04-09 18:30:22.220789","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:23.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53b40188c0 msgr2=0x7f53b401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53b40188c0 0x7f53b401adc0 crc :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d01acea0 msgr2=0x7f53d01ac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d01acea0 0x7f53d01ac750 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7f53c0006f20 tx=0x7f53c000b990).stop 2021-04-09T18:30:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 shutdown_connections 2021-04-09T18:30:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f53b40188c0 0x7f53b401adc0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53d0198eb0 0x7f53d01992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f53d0197ca0 0x7f53d01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 --2- 172.21.15.32:0/2833292526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53d01acea0 0x7f53d01ac750 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 >> 172.21.15.32:0/2833292526 conn(0x7f53d019c7a0 msgr2=0x7f53d0196a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 shutdown_connections 2021-04-09T18:30:23.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.624 7f53d67b3700 1 -- 172.21.15.32:0/2833292526 wait complete. 2021-04-09T18:30:23.649 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:23.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.783 7f32fcace700 1 Processor -- start 2021-04-09T18:30:23.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.783 7f32fcace700 1 -- start start 2021-04-09T18:30:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32fcace700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f32f80d6c80 0x7f32f80d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32fcace700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f80d2440 0x7f32f810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32fcace700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80d7280 con 0x7f32f80cffa0 2021-04-09T18:30:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32fcace700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f80d7080 con 0x7f32f80d6c80 2021-04-09T18:30:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32fcace700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80d91e0 con 0x7f32f80d2440 2021-04-09T18:30:23.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32f685e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f32f80cffa0 0x7f32f80d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37182/0 (socket says 172.21.15.32:37182) 2021-04-09T18:30:23.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32f685e700 1 -- 172.21.15.32:0/3246510416 learned_addr learned my addr 172.21.15.32:0/3246510416 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:23.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32f605d700 1 --2- 172.21.15.32:0/3246510416 >> v2:172.21.15.32:3300/0 conn(0x7f32f80d6c80 0x7f32f80d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32f705f700 1 --2- 172.21.15.32:0/3246510416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f80d2440 0x7f32f810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.784 7f32f585c700 1 -- 172.21.15.32:0/3246510416 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846881724 0 0) 0x7f32e8000ce0 con 0x7f32f80cffa0 2021-04-09T18:30:23.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f585c700 1 -- 172.21.15.32:0/3246510416 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e0002fe0 con 0x7f32f80cffa0 2021-04-09T18:30:23.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f605d700 1 -- 172.21.15.32:0/3246510416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f80d2440 msgr2=0x7f32f810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:23.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f605d700 1 --2- 172.21.15.32:0/3246510416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f80d2440 0x7f32f810cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f605d700 1 -- 172.21.15.32:0/3246510416 >> v1:172.21.15.32:6789/0 conn(0x7f32f80cffa0 legacy=0x7f32f80d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f605d700 1 -- 172.21.15.32:0/3246510416 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80d2840 con 0x7f32f80d6c80 2021-04-09T18:30:23.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f705f700 1 --2- 172.21.15.32:0/3246510416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f80d2440 0x7f32f810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:23.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f605d700 1 --2- 172.21.15.32:0/3246510416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f80d6c80 0x7f32f80d7ff0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f32ec006650 tx=0x7f32ec009a80).ready entity=mon.0 client_cookie=6382bbc6a9e28932 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f585c700 1 -- 172.21.15.32:0/3246510416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32ec0009b0 con 0x7f32f80d6c80 2021-04-09T18:30:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f585c700 1 -- 172.21.15.32:0/3246510416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ec00c0c0 con 0x7f32f80d6c80 2021-04-09T18:30:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32f585c700 1 -- 172.21.15.32:0/3246510416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32ec00c510 con 0x7f32f80d6c80 2021-04-09T18:30:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32fcace700 1 -- 172.21.15.32:0/3246510416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f80d6c80 msgr2=0x7f32f80d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.785 7f32fcace700 1 --2- 172.21.15.32:0/3246510416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f80d6c80 0x7f32f80d7ff0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f32ec006650 tx=0x7f32ec009a80).stop 2021-04-09T18:30:23.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.786 7f32fcace700 1 -- 172.21.15.32:0/3246510416 shutdown_connections 2021-04-09T18:30:23.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.786 7f32fcace700 1 --2- 172.21.15.32:0/3246510416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f80d2440 0x7f32f810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.786 7f32fcace700 1 --2- 172.21.15.32:0/3246510416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f80d6c80 0x7f32f80d7ff0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.786 7f32fcace700 1 -- 172.21.15.32:0/3246510416 >> 172.21.15.32:0/3246510416 conn(0x7f32f80ce5c0 msgr2=0x7f32f80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:23.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.786 7f32fcace700 1 -- 172.21.15.32:0/3246510416 shutdown_connections 2021-04-09T18:30:23.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.786 7f32fcace700 1 -- 172.21.15.32:0/3246510416 wait complete. 2021-04-09T18:30:23.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.806 7f32fcace700 1 Processor -- start 2021-04-09T18:30:23.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.806 7f32fcace700 1 -- start start 2021-04-09T18:30:23.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32fcace700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32f81929f0 0x7f32f8192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32fcace700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f808ddc0 0x7f32f808e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32fcace700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f808e7e0 0x7f32f808c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32fcace700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f8193410 con 0x7f32f808e7e0 2021-04-09T18:30:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32fcace700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80a9140 con 0x7f32f81929f0 2021-04-09T18:30:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32fcace700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f808ce80 con 0x7f32f808ddc0 2021-04-09T18:30:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f808e7e0 0x7f32f808c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f705f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f808e7e0 0x7f32f808c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55832/0 (socket says 172.21.15.32:55832) 2021-04-09T18:30:23.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f705f700 1 -- 172.21.15.32:0/1340878893 learned_addr learned my addr 172.21.15.32:0/1340878893 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:23.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f685e700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32f81929f0 0x7f32f8192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f605d700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f808ddc0 0x7f32f808e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f705f700 1 -- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f808ddc0 msgr2=0x7f32f808e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f705f700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f808ddc0 0x7f32f808e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f705f700 1 -- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32f81929f0 msgr2=0x7f32f8192df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f705f700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32f81929f0 0x7f32f8192df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f705f700 1 -- 172.21.15.32:0/1340878893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f808d0a0 con 0x7f32f808e7e0 2021-04-09T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.807 7f32f705f700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f808e7e0 0x7f32f808c860 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 rx=0x7f32f000b8c0 tx=0x7f32f000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.808 7f32e77fe700 1 -- 172.21.15.32:0/1340878893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32f000c2f0 con 0x7f32f808e7e0 2021-04-09T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.808 7f32e77fe700 1 -- 172.21.15.32:0/1340878893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f0006790 con 0x7f32f808e7e0 2021-04-09T18:30:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.808 7f32fcace700 1 -- 172.21.15.32:0/1340878893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f808d0a0 con 0x7f32f808e7e0 2021-04-09T18:30:23.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.808 7f32e77fe700 1 -- 172.21.15.32:0/1340878893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32f000c7e0 con 0x7f32f808e7e0 2021-04-09T18:30:23.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.808 7f32fcace700 1 -- 172.21.15.32:0/1340878893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f808d0a0 con 0x7f32f808e7e0 2021-04-09T18:30:23.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.808 7f32e77fe700 1 -- 172.21.15.32:0/1340878893 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f32f00011f0 con 0x7f32f808e7e0 2021-04-09T18:30:23.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.808 7f32e77fe700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32dc018930 0x7f32dc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.808 7f32e77fe700 1 -- 172.21.15.32:0/1340878893 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f32f00330a0 con 0x7f32f808e7e0 2021-04-09T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.809 7f32f685e700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32dc018930 0x7f32dc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:23.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.809 7f32f685e700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32dc018930 0x7f32dc01ae10 crc :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:23.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.828 7f32fcace700 1 -- 172.21.15.32:0/1340878893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f81b70c0 con 0x7f32f808e7e0 2021-04-09T18:30:23.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:23.831 7f32e77fe700 1 -- 172.21.15.32:0/1340878893 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f32f000bda0 con 0x7f32f808e7e0 2021-04-09T18:30:24.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.019 7f32fcace700 1 -- 172.21.15.32:0/1340878893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32f80d7280 con 0x7f32f808e7e0 2021-04-09T18:30:24.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.020 7f32e77fe700 1 -- 172.21.15.32:0/1340878893 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+16865 (secure 0 0 0) 0x7f32f00011f0 con 0x7f32f808e7e0 2021-04-09T18:30:24.035 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:24.035 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":183,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:23.208745","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:30:13.709244","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":31,"pool_name":"alpha_data","create_time":"2021-04-09 18:30:14.523869","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":32,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:30:21.008022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":33,"pool_name":"bravo_data","create_time":"2021-04-09 18:30:22.220789","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:24.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 -- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32dc018930 msgr2=0x7f32dc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:24.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32dc018930 0x7f32dc01ae10 crc :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 -- 172.21.15.32:0/1340878893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f808e7e0 msgr2=0x7f32f808c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f808e7e0 0x7f32f808c860 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 rx=0x7f32f000b8c0 tx=0x7f32f000bc60).stop 2021-04-09T18:30:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 -- 172.21.15.32:0/1340878893 shutdown_connections 2021-04-09T18:30:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f32dc018930 0x7f32dc01ae10 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:24.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f808e7e0 0x7f32f808c860 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:24.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32f808ddc0 0x7f32f808e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:24.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 --2- 172.21.15.32:0/1340878893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32f81929f0 0x7f32f8192df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:24.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 -- 172.21.15.32:0/1340878893 >> 172.21.15.32:0/1340878893 conn(0x7f32f8189b00 msgr2=0x7f32f81921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:24.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.022 7f32fcace700 1 -- 172.21.15.32:0/1340878893 shutdown_connections 2021-04-09T18:30:24.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.023 7f32fcace700 1 -- 172.21.15.32:0/1340878893 wait complete. 2021-04-09T18:30:24.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set bravo standby_count_wanted 0 2021-04-09T18:30:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.186 7f6fbed98700 1 Processor -- start 2021-04-09T18:30:24.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.186 7f6fbed98700 1 -- start start 2021-04-09T18:30:24.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbed98700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6fb80cffc0 0x7f6fb80d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:24.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbed98700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80d72a0 con 0x7f6fb80d6ca0 2021-04-09T18:30:24.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbed98700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80d8410 con 0x7f6fb80d7820 2021-04-09T18:30:24.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbed98700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb80d70a0 con 0x7f6fb80cffc0 2021-04-09T18:30:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbcbb4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6fb80cffc0 0x7f6fb80d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbd3b5700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6fb80d6ca0 0x7f6fb810cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37198/0 (socket says 172.21.15.32:37198) 2021-04-09T18:30:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbd3b5700 1 -- 172.21.15.32:0/3933302233 learned_addr learned my addr 172.21.15.32:0/3933302233 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbcbb4700 1 -- 172.21.15.32:0/3933302233 >> v1:172.21.15.86:6789/0 conn(0x7f6fb80d7820 legacy=0x7f6fb80d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbcbb4700 1 -- 172.21.15.32:0/3933302233 >> v1:172.21.15.32:6789/0 conn(0x7f6fb80d6ca0 legacy=0x7f6fb810cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbcbb4700 1 -- 172.21.15.32:0/3933302233 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80d9200 con 0x7f6fb80cffc0 2021-04-09T18:30:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fb77fe700 1 -- 172.21.15.32:0/3933302233 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065431088 0 0) 0x7f6fb0000d30 con 0x7f6fb80d6ca0 2021-04-09T18:30:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fbcbb4700 1 --2- 172.21.15.32:0/3933302233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb80cffc0 0x7f6fb80d2460 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7f6fa8006650 tx=0x7f6fa8009a00).ready entity=mon.0 client_cookie=d1f47b37cbbcae0a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.187 7f6fb77fe700 1 -- 172.21.15.32:0/3933302233 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fa8009df0 con 0x7f6fb80cffc0 2021-04-09T18:30:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.188 7f6fb77fe700 1 -- 172.21.15.32:0/3933302233 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa80009b0 con 0x7f6fb80cffc0 2021-04-09T18:30:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.188 7f6fb77fe700 1 -- 172.21.15.32:0/3933302233 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fa800c2b0 con 0x7f6fb80cffc0 2021-04-09T18:30:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.188 7f6fbed98700 1 -- 172.21.15.32:0/3933302233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb80cffc0 msgr2=0x7f6fb80d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.188 7f6fbed98700 1 --2- 172.21.15.32:0/3933302233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb80cffc0 0x7f6fb80d2460 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7f6fa8006650 tx=0x7f6fa8009a00).stop 2021-04-09T18:30:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.188 7f6fbed98700 1 -- 172.21.15.32:0/3933302233 shutdown_connections 2021-04-09T18:30:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.188 7f6fbed98700 1 --2- 172.21.15.32:0/3933302233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb80cffc0 0x7f6fb80d2460 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.188 7f6fbed98700 1 -- 172.21.15.32:0/3933302233 >> 172.21.15.32:0/3933302233 conn(0x7f6fb80ce5e0 msgr2=0x7f6fb80cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.188 7f6fbed98700 1 -- 172.21.15.32:0/3933302233 shutdown_connections 2021-04-09T18:30:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.188 7f6fbed98700 1 -- 172.21.15.32:0/3933302233 wait complete. 2021-04-09T18:30:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.208 7f6fbed98700 1 Processor -- start 2021-04-09T18:30:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.208 7f6fbed98700 1 -- start start 2021-04-09T18:30:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbed98700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6fb80968d0 0x7f6fb80a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbed98700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fb8096cd0 0x7f6fb808ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbed98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb8095dc0 0x7f6fb808f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:24.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbed98700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb8096350 con 0x7f6fb8095dc0 2021-04-09T18:30:24.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbed98700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80a6fe0 con 0x7f6fb8096cd0 2021-04-09T18:30:24.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbed98700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80b63e0 con 0x7f6fb80968d0 2021-04-09T18:30:24.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbd3b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb8095dc0 0x7f6fb808f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:24.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbd3b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb8095dc0 0x7f6fb808f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55848/0 (socket says 172.21.15.32:55848) 2021-04-09T18:30:24.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbd3b5700 1 -- 172.21.15.32:0/3794971684 learned_addr learned my addr 172.21.15.32:0/3794971684 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:24.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fb7fff700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fb8096cd0 0x7f6fb808ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:24.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.209 7f6fbcbb4700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6fb80968d0 0x7f6fb80a69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:24.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fbd3b5700 1 -- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6fb80968d0 msgr2=0x7f6fb80a69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:24.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fbd3b5700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6fb80968d0 0x7f6fb80a69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:24.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fbd3b5700 1 -- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fb8096cd0 msgr2=0x7f6fb808ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:24.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fbd3b5700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fb8096cd0 0x7f6fb808ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:24.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fbd3b5700 1 -- 172.21.15.32:0/3794971684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb808f6c0 con 0x7f6fb8095dc0 2021-04-09T18:30:24.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fbcbb4700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6fb80968d0 0x7f6fb80a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fbd3b5700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb8095dc0 0x7f6fb808f0a0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f6fb0007f90 tx=0x7f6fb000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fb57fa700 1 -- 172.21.15.32:0/3794971684 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb0000cc0 con 0x7f6fb8095dc0 2021-04-09T18:30:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fb57fa700 1 -- 172.21.15.32:0/3794971684 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb0007020 con 0x7f6fb8095dc0 2021-04-09T18:30:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fb57fa700 1 -- 172.21.15.32:0/3794971684 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb0007500 con 0x7f6fb8095dc0 2021-04-09T18:30:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb808f6c0 con 0x7f6fb8095dc0 2021-04-09T18:30:24.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.210 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb819f4f0 con 0x7f6fb8095dc0 2021-04-09T18:30:24.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.213 7f6fb57fa700 1 -- 172.21.15.32:0/3794971684 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6fb000ad90 con 0x7f6fb8095dc0 2021-04-09T18:30:24.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.213 7f6fb57fa700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f980188c0 0x7f6f9801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:24.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.213 7f6fb57fa700 1 -- 172.21.15.32:0/3794971684 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f6fb0032de0 con 0x7f6fb8095dc0 2021-04-09T18:30:24.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.213 7f6fbcbb4700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f980188c0 0x7f6f9801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:24.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.213 7f6fbcbb4700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f980188c0 0x7f6f9801adc0 crc :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:24.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.231 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb808f960 con 0x7f6fb8095dc0 2021-04-09T18:30:24.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.235 7f6fb57fa700 1 -- 172.21.15.32:0/3794971684 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6fb0001270 con 0x7f6fb8095dc0 2021-04-09T18:30:24.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:24.427 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "bravo", "val": "0"} v 0) v1 -- 0x7f6fb808f960 con 0x7f6fb8095dc0 2021-04-09T18:30:25.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.218 7f6fb57fa700 1 -- 172.21.15.32:0/3794971684 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "bravo", "val": "0"}]=0 v329) v1 ==== 117+0+0 (secure 0 0 0) 0x7f6fb000ad90 con 0x7f6fb8095dc0 2021-04-09T18:30:25.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.244 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f980188c0 msgr2=0x7f6f9801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.244 7f6fbed98700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f980188c0 0x7f6f9801adc0 crc :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.244 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb8095dc0 msgr2=0x7f6fb808f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.244 7f6fbed98700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb8095dc0 0x7f6fb808f0a0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f6fb0007f90 tx=0x7f6fb000ac50).stop 2021-04-09T18:30:25.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.245 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 shutdown_connections 2021-04-09T18:30:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.245 7f6fbed98700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f980188c0 0x7f6f9801adc0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.245 7f6fbed98700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fb8095dc0 0x7f6fb808f0a0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.245 7f6fbed98700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fb8096cd0 0x7f6fb808ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.245 7f6fbed98700 1 --2- 172.21.15.32:0/3794971684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6fb80968d0 0x7f6fb80a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.245 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 >> 172.21.15.32:0/3794971684 conn(0x7f6fb8099110 msgr2=0x7f6fb80af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:25.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.245 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 shutdown_connections 2021-04-09T18:30:25.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.245 7f6fbed98700 1 -- 172.21.15.32:0/3794971684 wait complete. 2021-04-09T18:30:25.269 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:25.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.403 7f761909b700 1 Processor -- start 2021-04-09T18:30:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.403 7f761909b700 1 -- start start 2021-04-09T18:30:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f761909b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f761410d010 0x7f76140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f761909b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76140d1da0 0x7f76140d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f761909b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761410d480 con 0x7f76140d82c0 2021-04-09T18:30:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f761909b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76140d9220 con 0x7f761410d010 2021-04-09T18:30:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f761909b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76140d86c0 con 0x7f76140d1da0 2021-04-09T18:30:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f7612d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f761410d010 0x7f76140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.441 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f7612d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f761410d010 0x7f76140d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55852/0 (socket says 172.21.15.32:55852) 2021-04-09T18:30:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f7612d9d700 1 -- 172.21.15.32:0/2421194619 learned_addr learned my addr 172.21.15.32:0/2421194619 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f7612d9d700 1 -- 172.21.15.32:0/2421194619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76140d1da0 msgr2=0x7f76140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:25.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f7612d9d700 1 --2- 172.21.15.32:0/2421194619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76140d1da0 0x7f76140d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-04-09T18:30:25.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f7612d9d700 1 -- 172.21.15.32:0/2421194619 >> v1:172.21.15.86:6789/0 conn(0x7f76140d82c0 legacy=0x7f761410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:25.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f7612d9d700 1 -- 172.21.15.32:0/2421194619 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761410e1f0 con 0x7f761410d010 2021-04-09T18:30:25.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.404 7f7612d9d700 1 --2- 172.21.15.32:0/2421194619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761410d010 0x7f76140d1780 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7f7604006700 tx=0x7f7604009b00).ready entity=mon.0 client_cookie=7d4ed49c0f975385 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:25.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.405 7f7611d9b700 1 -- 172.21.15.32:0/2421194619 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f760400c0c0 con 0x7f761410d010 2021-04-09T18:30:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.405 7f7611d9b700 1 -- 172.21.15.32:0/2421194619 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760400c400 con 0x7f761410d010 2021-04-09T18:30:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.405 7f7611d9b700 1 -- 172.21.15.32:0/2421194619 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f760400c8e0 con 0x7f761410d010 2021-04-09T18:30:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.405 7f761909b700 1 -- 172.21.15.32:0/2421194619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761410d010 msgr2=0x7f76140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.405 7f761909b700 1 --2- 172.21.15.32:0/2421194619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761410d010 0x7f76140d1780 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7f7604006700 tx=0x7f7604009b00).stop 2021-04-09T18:30:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.406 7f761909b700 1 -- 172.21.15.32:0/2421194619 shutdown_connections 2021-04-09T18:30:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.406 7f761909b700 1 --2- 172.21.15.32:0/2421194619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76140d1da0 0x7f76140d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-04-09T18:30:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.406 7f761909b700 1 --2- 172.21.15.32:0/2421194619 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f761410d010 0x7f76140d1780 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.406 7f761909b700 1 -- 172.21.15.32:0/2421194619 >> 172.21.15.32:0/2421194619 conn(0x7f76140ce5f0 msgr2=0x7f76140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:25.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.406 7f761909b700 1 -- 172.21.15.32:0/2421194619 shutdown_connections 2021-04-09T18:30:25.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.406 7f761909b700 1 -- 172.21.15.32:0/2421194619 wait complete. 2021-04-09T18:30:25.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761909b700 1 Processor -- start 2021-04-09T18:30:25.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761909b700 1 -- start start 2021-04-09T18:30:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761909b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76141acea0 0x7f76141ac750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761909b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7614197ca0 0x7f76141980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761909b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7614198eb0 0x7f76141992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761909b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76141986c0 con 0x7f7614197ca0 2021-04-09T18:30:25.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761909b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7614198930 con 0x7f7614198eb0 2021-04-09T18:30:25.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761909b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76141ad2a0 con 0x7f76141acea0 2021-04-09T18:30:25.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7614197ca0 0x7f76141980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7614197ca0 0x7f76141980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55860/0 (socket says 172.21.15.32:55860) 2021-04-09T18:30:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.426 7f761259c700 1 -- 172.21.15.32:0/4122223180 learned_addr learned my addr 172.21.15.32:0/4122223180 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f7612d9d700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76141acea0 0x7f76141ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f761359e700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7614198eb0 0x7f76141992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f761259c700 1 -- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76141acea0 msgr2=0x7f76141ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f761259c700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76141acea0 0x7f76141ac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f761259c700 1 -- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7614198eb0 msgr2=0x7f76141992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f761259c700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7614198eb0 0x7f76141992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f761259c700 1 -- 172.21.15.32:0/4122223180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76141998d0 con 0x7f7614197ca0 2021-04-09T18:30:25.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f761259c700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7614197ca0 0x7f76141980a0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f7600006f00 tx=0x7f760000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:25.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f760b7fe700 1 -- 172.21.15.32:0/4122223180 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f760000bd30 con 0x7f7614197ca0 2021-04-09T18:30:25.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f760b7fe700 1 -- 172.21.15.32:0/4122223180 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7600000c90 con 0x7f7614197ca0 2021-04-09T18:30:25.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f761909b700 1 -- 172.21.15.32:0/4122223180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76141998d0 con 0x7f7614197ca0 2021-04-09T18:30:25.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f760b7fe700 1 -- 172.21.15.32:0/4122223180 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76000022a0 con 0x7f7614197ca0 2021-04-09T18:30:25.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.427 7f761909b700 1 -- 172.21.15.32:0/4122223180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76141998d0 con 0x7f7614197ca0 2021-04-09T18:30:25.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.428 7f760b7fe700 1 -- 172.21.15.32:0/4122223180 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7600005310 con 0x7f7614197ca0 2021-04-09T18:30:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.428 7f760b7fe700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75f0018930 0x7f75f001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.428 7f760b7fe700 1 -- 172.21.15.32:0/4122223180 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f7600032ce0 con 0x7f7614197ca0 2021-04-09T18:30:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.428 7f7612d9d700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75f0018930 0x7f75f001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.428 7f7612d9d700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75f0018930 0x7f75f001adc0 crc :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:25.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.448 7f761909b700 1 -- 172.21.15.32:0/4122223180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7614184b30 con 0x7f7614197ca0 2021-04-09T18:30:25.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.451 7f760b7fe700 1 -- 172.21.15.32:0/4122223180 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7600006b90 con 0x7f7614197ca0 2021-04-09T18:30:25.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.642 7f761909b700 1 -- 172.21.15.32:0/4122223180 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7614184b30 con 0x7f7614197ca0 2021-04-09T18:30:25.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.643 7f760b7fe700 1 -- 172.21.15.32:0/4122223180 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 329 v329) v1 ==== 95+0+3719 (secure 0 0 0) 0x7f760000bd30 con 0x7f7614197ca0 2021-04-09T18:30:25.657 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 329 2021-04-09T18:30:25.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:25.658 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":329,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10920,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2893945823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2893945823},{"type":"v1","addr":"172.21.15.86:6837","nonce":2893945823}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":321},{"gid":58607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1628176610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1628176610},{"type":"v1","addr":"172.21.15.86:6836","nonce":1628176610}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":321}],"filesystems":[{"mdsmap":{"epoch":325,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:17.772722","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10911":{"gid":10911,"name":"c","rank":0,"incarnation":322,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6834/3729993394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3729993394},{"type":"v1","addr":"172.21.15.32:6834","nonce":3729993394}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":329,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:25.213901","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10914},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10914":{"gid":10914,"name":"a","rank":0,"incarnation":326,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3158551863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3158551863},{"type":"v1","addr":"172.21.15.32:6835","nonce":3158551863}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:25.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 -- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75f0018930 msgr2=0x7f75f001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75f0018930 0x7f75f001adc0 crc :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 -- 172.21.15.32:0/4122223180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7614197ca0 msgr2=0x7f76141980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7614197ca0 0x7f76141980a0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f7600006f00 tx=0x7f760000b940).stop 2021-04-09T18:30:25.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 -- 172.21.15.32:0/4122223180 shutdown_connections 2021-04-09T18:30:25.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75f0018930 0x7f75f001adc0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7614197ca0 0x7f76141980a0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7614198eb0 0x7f76141992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 --2- 172.21.15.32:0/4122223180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76141acea0 0x7f76141ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 -- 172.21.15.32:0/4122223180 >> 172.21.15.32:0/4122223180 conn(0x7f761419c7a0 msgr2=0x7f7614196a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:25.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 -- 172.21.15.32:0/4122223180 shutdown_connections 2021-04-09T18:30:25.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.645 7f761909b700 1 -- 172.21.15.32:0/4122223180 wait complete. 2021-04-09T18:30:25.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.804 7f34637a3700 1 Processor -- start 2021-04-09T18:30:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.804 7f34637a3700 1 -- start start 2021-04-09T18:30:25.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f34637a3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f345c10d010 0x7f345c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f34637a3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0d8490 0x7f345c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f34637a3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c10d480 con 0x7f345c0d1da0 2021-04-09T18:30:25.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f34637a3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345c0d8d30 con 0x7f345c10d010 2021-04-09T18:30:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f34637a3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0d9220 con 0x7f345c0d8490 2021-04-09T18:30:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f34615bf700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f345c10d010 0x7f345c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f3461dc0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0d8490 0x7f345c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f3460dbe700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f345c0d1da0 0x7f345c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:35154/0 (socket says 172.21.15.32:35154) 2021-04-09T18:30:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f3460dbe700 1 -- 172.21.15.32:0/1167864785 learned_addr learned my addr 172.21.15.32:0/1167864785 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f3461dc0700 1 -- 172.21.15.32:0/1167864785 >> v2:172.21.15.86:3300/0 conn(0x7f345c10d010 msgr2=0x7f345c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f3461dc0700 1 --2- 172.21.15.32:0/1167864785 >> v2:172.21.15.86:3300/0 conn(0x7f345c10d010 0x7f345c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f3461dc0700 1 -- 172.21.15.32:0/1167864785 >> v1:172.21.15.86:6789/0 conn(0x7f345c0d1da0 legacy=0x7f345c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f3453fff700 1 -- 172.21.15.32:0/1167864785 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400178591 0 0) 0x7f344c0010c0 con 0x7f345c0d1da0 2021-04-09T18:30:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f3461dc0700 1 -- 172.21.15.32:0/1167864785 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c10e020 con 0x7f345c0d8490 2021-04-09T18:30:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.805 7f34615bf700 1 --2- 172.21.15.32:0/1167864785 >> v2:172.21.15.86:3300/0 conn(0x7f345c10d010 0x7f345c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.806 7f3461dc0700 1 --2- 172.21.15.32:0/1167864785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0d8490 0x7f345c10da00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3454006e50 tx=0x7f3454009e70).ready entity=mon.2 client_cookie=bf2b7c528332f8a1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.806 7f3453fff700 1 -- 172.21.15.32:0/1167864785 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3454006a20 con 0x7f345c0d8490 2021-04-09T18:30:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.806 7f3453fff700 1 -- 172.21.15.32:0/1167864785 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3454006790 con 0x7f345c0d8490 2021-04-09T18:30:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.806 7f3453fff700 1 -- 172.21.15.32:0/1167864785 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f345400b700 con 0x7f345c0d8490 2021-04-09T18:30:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.806 7f34637a3700 1 -- 172.21.15.32:0/1167864785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0d8490 msgr2=0x7f345c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.806 7f34637a3700 1 --2- 172.21.15.32:0/1167864785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0d8490 0x7f345c10da00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3454006e50 tx=0x7f3454009e70).stop 2021-04-09T18:30:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.807 7f34637a3700 1 -- 172.21.15.32:0/1167864785 shutdown_connections 2021-04-09T18:30:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.807 7f34637a3700 1 --2- 172.21.15.32:0/1167864785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0d8490 0x7f345c10da00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.807 7f34637a3700 1 --2- 172.21.15.32:0/1167864785 >> v2:172.21.15.86:3300/0 conn(0x7f345c10d010 0x7f345c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.807 7f34637a3700 1 -- 172.21.15.32:0/1167864785 >> 172.21.15.32:0/1167864785 conn(0x7f345c0ce5f0 msgr2=0x7f345c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.807 7f34637a3700 1 -- 172.21.15.32:0/1167864785 shutdown_connections 2021-04-09T18:30:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.807 7f34637a3700 1 -- 172.21.15.32:0/1167864785 wait complete. 2021-04-09T18:30:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.827 7f34637a3700 1 Processor -- start 2021-04-09T18:30:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.827 7f34637a3700 1 -- start start 2021-04-09T18:30:25.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f34637a3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0b5360 0x7f345c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f34637a3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c18e140 0x7f345c1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f34637a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f345c1ad040 0x7f345c08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f34637a3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c1a5360 con 0x7f345c1ad040 2021-04-09T18:30:25.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f34637a3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0902c0 con 0x7f345c18e140 2021-04-09T18:30:25.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f34637a3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0904e0 con 0x7f345c0b5360 2021-04-09T18:30:25.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3461dc0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f345c1ad040 0x7f345c08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3461dc0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f345c1ad040 0x7f345c08fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55876/0 (socket says 172.21.15.32:55876) 2021-04-09T18:30:25.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3461dc0700 1 -- 172.21.15.32:0/2946630178 learned_addr learned my addr 172.21.15.32:0/2946630178 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:25.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f34615bf700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0b5360 0x7f345c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3460dbe700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c18e140 0x7f345c1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3461dc0700 1 -- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0b5360 msgr2=0x7f345c1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3461dc0700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0b5360 0x7f345c1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3461dc0700 1 -- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c18e140 msgr2=0x7f345c1aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:25.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3461dc0700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c18e140 0x7f345c1aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3461dc0700 1 -- 172.21.15.32:0/2946630178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c183c80 con 0x7f345c1ad040 2021-04-09T18:30:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3461dc0700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f345c1ad040 0x7f345c08fca0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f3454006e60 tx=0x7f345400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:25.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3451ffb700 1 -- 172.21.15.32:0/2946630178 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3454009130 con 0x7f345c1ad040 2021-04-09T18:30:25.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3451ffb700 1 -- 172.21.15.32:0/2946630178 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34540011a0 con 0x7f345c1ad040 2021-04-09T18:30:25.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f34637a3700 1 -- 172.21.15.32:0/2946630178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345c183c80 con 0x7f345c1ad040 2021-04-09T18:30:25.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f3451ffb700 1 -- 172.21.15.32:0/2946630178 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34540016a0 con 0x7f345c1ad040 2021-04-09T18:30:25.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.828 7f34637a3700 1 -- 172.21.15.32:0/2946630178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345c183c80 con 0x7f345c1ad040 2021-04-09T18:30:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.829 7f3451ffb700 1 -- 172.21.15.32:0/2946630178 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3454009130 con 0x7f345c1ad040 2021-04-09T18:30:25.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.829 7f3451ffb700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f343c018930 0x7f343c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.829 7f3451ffb700 1 -- 172.21.15.32:0/2946630178 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f3454033060 con 0x7f345c1ad040 2021-04-09T18:30:25.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.829 7f34615bf700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f343c018930 0x7f343c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:25.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.830 7f34615bf700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f343c018930 0x7f343c01adc0 crc :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:25.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.849 7f34637a3700 1 -- 172.21.15.32:0/2946630178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345c182d70 con 0x7f345c1ad040 2021-04-09T18:30:25.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:25.852 7f3451ffb700 1 -- 172.21.15.32:0/2946630178 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f345400be30 con 0x7f345c1ad040 2021-04-09T18:30:26.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.039 7f34637a3700 1 -- 172.21.15.32:0/2946630178 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f345c182d70 con 0x7f345c1ad040 2021-04-09T18:30:26.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.040 7f3451ffb700 1 -- 172.21.15.32:0/2946630178 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f34540011a0 con 0x7f345c1ad040 2021-04-09T18:30:26.055 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:26.055 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":184,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:24.210342","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:30:13.709244","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":31,"pool_name":"alpha_data","create_time":"2021-04-09 18:30:14.523869","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":32,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:30:21.008022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":33,"pool_name":"bravo_data","create_time":"2021-04-09 18:30:22.220789","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:26.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 -- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f343c018930 msgr2=0x7f343c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:26.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f343c018930 0x7f343c01adc0 crc :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 -- 172.21.15.32:0/2946630178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f345c1ad040 msgr2=0x7f345c08fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:26.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f345c1ad040 0x7f345c08fca0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f3454006e60 tx=0x7f345400bcf0).stop 2021-04-09T18:30:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 -- 172.21.15.32:0/2946630178 shutdown_connections 2021-04-09T18:30:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f343c018930 0x7f343c01adc0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f345c1ad040 0x7f345c08fca0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c18e140 0x7f345c1aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 --2- 172.21.15.32:0/2946630178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0b5360 0x7f345c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 -- 172.21.15.32:0/2946630178 >> 172.21.15.32:0/2946630178 conn(0x7f345c19b460 msgr2=0x7f345c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:26.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 -- 172.21.15.32:0/2946630178 shutdown_connections 2021-04-09T18:30:26.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.042 7f34637a3700 1 -- 172.21.15.32:0/2946630178 wait complete. 2021-04-09T18:30:26.067 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:30:26.068 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:30:26.068 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:30:26.068 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:30:26.069 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:30:26.069 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:30:26.070 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:30:26.070 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:30:26.070 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:30:26.070 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:30:26.071 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:30:26.071 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:30:26.071 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:30:26.072 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:30:26.072 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:30:26.072 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:30:26.077 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:30:26.077 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:30:26.079 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:30:26.079 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:30:26.081 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:30:26.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:30:26.090 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:30:26.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:30:26.120 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:30:26.121 DEBUG:teuthology.parallel:result is None 2021-04-09T18:30:26.122 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:30:26.123 DEBUG:teuthology.parallel:result is None 2021-04-09T18:30:26.132 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:30:26.133 DEBUG:teuthology.parallel:result is None 2021-04-09T18:30:26.134 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:30:26.135 DEBUG:teuthology.parallel:result is None 2021-04-09T18:30:26.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:26.270 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:30:26.270 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:30:26.287 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:30:26.288 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:30:26.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.274 7fab2fdd7700 1 Processor -- start 2021-04-09T18:30:26.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.274 7fab2fdd7700 1 -- start start 2021-04-09T18:30:26.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2fdd7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab2810d010 0x7fab280d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:26.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2fdd7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fab280d8490 0x7fab2810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:26.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2fdd7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2810d480 con 0x7fab280d1da0 2021-04-09T18:30:26.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2fdd7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab280d8d30 con 0x7fab2810d010 2021-04-09T18:30:26.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2fdd7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab280d9220 con 0x7fab280d8490 2021-04-09T18:30:26.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2dbf3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab2810d010 0x7fab280d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:26.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2d3f2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fab280d1da0 0x7fab280d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37264/0 (socket says 172.21.15.32:37264) 2021-04-09T18:30:26.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2d3f2700 1 -- 172.21.15.32:0/3053050808 learned_addr learned my addr 172.21.15.32:0/3053050808 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2e3f4700 1 --2- 172.21.15.32:0/3053050808 >> v2:172.21.15.86:3300/0 conn(0x7fab280d8490 0x7fab2810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2dbf3700 1 -- 172.21.15.32:0/3053050808 >> v2:172.21.15.86:3300/0 conn(0x7fab280d8490 msgr2=0x7fab2810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2dbf3700 1 --2- 172.21.15.32:0/3053050808 >> v2:172.21.15.86:3300/0 conn(0x7fab280d8490 0x7fab2810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2dbf3700 1 -- 172.21.15.32:0/3053050808 >> v1:172.21.15.32:6789/0 conn(0x7fab280d1da0 legacy=0x7fab280d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:26.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2cbf1700 1 -- 172.21.15.32:0/3053050808 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662578431 0 0) 0x7fab18000ce0 con 0x7fab280d1da0 2021-04-09T18:30:26.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2dbf3700 1 -- 172.21.15.32:0/3053050808 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2810e020 con 0x7fab2810d010 2021-04-09T18:30:26.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2dbf3700 1 --2- 172.21.15.32:0/3053050808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab2810d010 0x7fab280d1780 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fab24006650 tx=0x7fab24009a80).ready entity=mon.0 client_cookie=fee133246a484da1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:26.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2cbf1700 1 -- 172.21.15.32:0/3053050808 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab240009b0 con 0x7fab2810d010 2021-04-09T18:30:26.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2cbf1700 1 -- 172.21.15.32:0/3053050808 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2400c0c0 con 0x7fab2810d010 2021-04-09T18:30:26.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2cbf1700 1 -- 172.21.15.32:0/3053050808 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab2400c510 con 0x7fab2810d010 2021-04-09T18:30:26.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.275 7fab2fdd7700 1 -- 172.21.15.32:0/3053050808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab2810d010 msgr2=0x7fab280d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:26.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.276 7fab2fdd7700 1 --2- 172.21.15.32:0/3053050808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab2810d010 0x7fab280d1780 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fab24006650 tx=0x7fab24009a80).stop 2021-04-09T18:30:26.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.277 7fab2fdd7700 1 -- 172.21.15.32:0/3053050808 shutdown_connections 2021-04-09T18:30:26.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.277 7fab2fdd7700 1 --2- 172.21.15.32:0/3053050808 >> v2:172.21.15.86:3300/0 conn(0x7fab280d8490 0x7fab2810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.277 7fab2fdd7700 1 --2- 172.21.15.32:0/3053050808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab2810d010 0x7fab280d1780 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.277 7fab2fdd7700 1 -- 172.21.15.32:0/3053050808 >> 172.21.15.32:0/3053050808 conn(0x7fab280ce5f0 msgr2=0x7fab280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:26.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.277 7fab2fdd7700 1 -- 172.21.15.32:0/3053050808 shutdown_connections 2021-04-09T18:30:26.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.277 7fab2fdd7700 1 -- 172.21.15.32:0/3053050808 wait complete. 2021-04-09T18:30:26.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.297 7fab2fdd7700 1 Processor -- start 2021-04-09T18:30:26.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.297 7fab2fdd7700 1 -- start start 2021-04-09T18:30:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.297 7fab2fdd7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab280a90c0 0x7fab281a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.297 7fab2fdd7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab281a76d0 0x7fab281a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.297 7fab2fdd7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab2817bce0 0x7fab2817c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.297 7fab2fdd7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281a7c60 con 0x7fab281a76d0 2021-04-09T18:30:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.297 7fab2fdd7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281a8e20 con 0x7fab2817bce0 2021-04-09T18:30:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.297 7fab2fdd7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2817c700 con 0x7fab280a90c0 2021-04-09T18:30:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.297 7fab2d3f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab281a76d0 0x7fab281a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2d3f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab281a76d0 0x7fab281a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55912/0 (socket says 172.21.15.32:55912) 2021-04-09T18:30:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2d3f2700 1 -- 172.21.15.32:0/4081493047 learned_addr learned my addr 172.21.15.32:0/4081493047 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2e3f4700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab2817bce0 0x7fab2817c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2dbf3700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab280a90c0 0x7fab281a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2d3f2700 1 -- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab280a90c0 msgr2=0x7fab281a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2d3f2700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab280a90c0 0x7fab281a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2d3f2700 1 -- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab2817bce0 msgr2=0x7fab2817c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2d3f2700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab2817bce0 0x7fab2817c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2d3f2700 1 -- 172.21.15.32:0/4081493047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2817c920 con 0x7fab281a76d0 2021-04-09T18:30:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2e3f4700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab2817bce0 0x7fab2817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2d3f2700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab281a76d0 0x7fab281a8800 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7fab18006f00 tx=0x7fab1800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab1e7fc700 1 -- 172.21.15.32:0/4081493047 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab18006020 con 0x7fab281a76d0 2021-04-09T18:30:26.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab1e7fc700 1 -- 172.21.15.32:0/4081493047 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab180063a0 con 0x7fab281a76d0 2021-04-09T18:30:26.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2817c920 con 0x7fab281a76d0 2021-04-09T18:30:26.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab1e7fc700 1 -- 172.21.15.32:0/4081493047 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab18004620 con 0x7fab281a76d0 2021-04-09T18:30:26.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.298 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2817c920 con 0x7fab281a76d0 2021-04-09T18:30:26.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.299 7fab1e7fc700 1 -- 172.21.15.32:0/4081493047 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fab1800a000 con 0x7fab281a76d0 2021-04-09T18:30:26.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.299 7fab1e7fc700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab10018930 0x7fab1001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:26.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.299 7fab1e7fc700 1 -- 172.21.15.32:0/4081493047 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fab18032d40 con 0x7fab281a76d0 2021-04-09T18:30:26.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.299 7fab2dbf3700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab10018930 0x7fab1001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.299 7fab2dbf3700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab10018930 0x7fab1001ae10 crc :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.317 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab281b7010 con 0x7fab281a76d0 2021-04-09T18:30:26.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.321 7fab1e7fc700 1 -- 172.21.15.32:0/4081493047 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fab18006b90 con 0x7fab281a76d0 2021-04-09T18:30:26.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.514 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab2810d480 con 0x7fab281a76d0 2021-04-09T18:30:26.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.514 7fab1e7fc700 1 -- 172.21.15.32:0/4081493047 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 329 v329) v1 ==== 95+0+3719 (secure 0 0 0) 0x7fab1800a000 con 0x7fab281a76d0 2021-04-09T18:30:26.529 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 329 2021-04-09T18:30:26.530 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:26.530 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":329,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10920,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2893945823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2893945823},{"type":"v1","addr":"172.21.15.86:6837","nonce":2893945823}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":321},{"gid":58607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1628176610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1628176610},{"type":"v1","addr":"172.21.15.86:6836","nonce":1628176610}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":321}],"filesystems":[{"mdsmap":{"epoch":325,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:17.772722","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10911":{"gid":10911,"name":"c","rank":0,"incarnation":322,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6834/3729993394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3729993394},{"type":"v1","addr":"172.21.15.32:6834","nonce":3729993394}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":329,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:25.213901","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10914},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10914":{"gid":10914,"name":"a","rank":0,"incarnation":326,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3158551863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3158551863},{"type":"v1","addr":"172.21.15.32:6835","nonce":3158551863}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:26.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab10018930 msgr2=0x7fab1001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:26.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab10018930 0x7fab1001ae10 crc :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab281a76d0 msgr2=0x7fab281a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:26.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab281a76d0 0x7fab281a8800 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7fab18006f00 tx=0x7fab1800b940).stop 2021-04-09T18:30:26.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 shutdown_connections 2021-04-09T18:30:26.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fab10018930 0x7fab1001ae10 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab281a76d0 0x7fab281a8800 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab2817bce0 0x7fab2817c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 --2- 172.21.15.32:0/4081493047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab280a90c0 0x7fab281a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 >> 172.21.15.32:0/4081493047 conn(0x7fab281acae0 msgr2=0x7fab2817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 shutdown_connections 2021-04-09T18:30:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:26.517 7fab2fdd7700 1 -- 172.21.15.32:0/4081493047 wait complete. 2021-04-09T18:30:26.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 325, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:15.764321', 'modified': '2021-04-09 18:30:17.772722', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10911}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10911': {'gid': 10911, 'name': 'c', 'rank': 0, 'incarnation': 322, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6834/3729993394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3729993394}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 3729993394}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:26.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:30:26.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:30:27.081 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:30:27.082 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:30:27.082 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": -1, 2021-04-09T18:30:27.082 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 58739, 2021-04-09T18:30:27.082 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:boot", 2021-04-09T18:30:27.083 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "null", 2021-04-09T18:30:27.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 329, 2021-04-09T18:30:27.083 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 0, 2021-04-09T18:30:27.083 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 0, 2021-04-09T18:30:27.084 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 0.80969656199999995 2021-04-09T18:30:27.084 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:30:27.088 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": -1, "id": 58739, "want_state": "up:boot", "state": "null", "mdsmap_epoch": 329, "osdmap_epoch": 0, "osdmap_epoch_barrier": 0, "uptime": 0.80969656199999995 } 2021-04-09T18:30:28.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:28.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.236 7fc18a48a700 1 Processor -- start 2021-04-09T18:30:28.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.236 7fc18a48a700 1 -- start start 2021-04-09T18:30:28.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc18a48a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc18410da00 0x7fc18410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:28.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc18a48a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18410d480 con 0x7fc1840d7ca0 2021-04-09T18:30:28.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc18a48a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1840d1f70 con 0x7fc18410d010 2021-04-09T18:30:28.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc18a48a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1840d8d30 con 0x7fc18410da00 2021-04-09T18:30:28.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc188aa7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc18410da00 0x7fc18410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:28.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc17bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc1840d7ca0 0x7fc1840d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37278/0 (socket says 172.21.15.32:37278) 2021-04-09T18:30:28.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc17bfff700 1 -- 172.21.15.32:0/103955684 learned_addr learned my addr 172.21.15.32:0/103955684 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:28.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc188aa7700 1 -- 172.21.15.32:0/103955684 >> v1:172.21.15.86:6789/0 conn(0x7fc18410d010 legacy=0x7fc1840d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:28.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc188aa7700 1 -- 172.21.15.32:0/103955684 >> v1:172.21.15.32:6789/0 conn(0x7fc1840d7ca0 legacy=0x7fc1840d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:28.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc1837fe700 1 -- 172.21.15.32:0/103955684 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714960320 0 0) 0x7fc16c000ce0 con 0x7fc1840d7ca0 2021-04-09T18:30:28.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc188aa7700 1 -- 172.21.15.32:0/103955684 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840d9220 con 0x7fc18410da00 2021-04-09T18:30:28.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc188aa7700 1 --2- 172.21.15.32:0/103955684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18410da00 0x7fc18410de00 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7fc17c006e30 tx=0x7fc17c009de0).ready entity=mon.0 client_cookie=3713457e48228ddc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:28.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc1837fe700 1 -- 172.21.15.32:0/103955684 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc17c00b300 con 0x7fc18410da00 2021-04-09T18:30:28.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc1837fe700 1 -- 172.21.15.32:0/103955684 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c00b680 con 0x7fc18410da00 2021-04-09T18:30:28.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc1837fe700 1 -- 172.21.15.32:0/103955684 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc17c00bb60 con 0x7fc18410da00 2021-04-09T18:30:28.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc18a48a700 1 -- 172.21.15.32:0/103955684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18410da00 msgr2=0x7fc18410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:28.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.237 7fc18a48a700 1 --2- 172.21.15.32:0/103955684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18410da00 0x7fc18410de00 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7fc17c006e30 tx=0x7fc17c009de0).stop 2021-04-09T18:30:28.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.238 7fc18a48a700 1 -- 172.21.15.32:0/103955684 shutdown_connections 2021-04-09T18:30:28.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.238 7fc18a48a700 1 --2- 172.21.15.32:0/103955684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18410da00 0x7fc18410de00 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:28.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.238 7fc18a48a700 1 -- 172.21.15.32:0/103955684 >> 172.21.15.32:0/103955684 conn(0x7fc1840ce5f0 msgr2=0x7fc1840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:28.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.238 7fc18a48a700 1 -- 172.21.15.32:0/103955684 shutdown_connections 2021-04-09T18:30:28.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.238 7fc18a48a700 1 -- 172.21.15.32:0/103955684 wait complete. 2021-04-09T18:30:28.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.258 7fc18a48a700 1 Processor -- start 2021-04-09T18:30:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc18a48a700 1 -- start start 2021-04-09T18:30:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc18a48a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1840b1280 0x7fc18408b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc18a48a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18408b930 0x7fc184091b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc18a48a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc184092130 0x7fc1840981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc18a48a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc184092530 con 0x7fc18408b930 2021-04-09T18:30:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc18a48a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc184180940 con 0x7fc1840b1280 2021-04-09T18:30:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc18a48a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc184180b60 con 0x7fc184092130 2021-04-09T18:30:28.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc17bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18408b930 0x7fc184091b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:28.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc17bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18408b930 0x7fc184091b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55924/0 (socket says 172.21.15.32:55924) 2021-04-09T18:30:28.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc17bfff700 1 -- 172.21.15.32:0/1222332936 learned_addr learned my addr 172.21.15.32:0/1222332936 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:28.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc188aa7700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc184092130 0x7fc1840981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:28.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.259 7fc183fff700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1840b1280 0x7fc18408b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:28.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc17bfff700 1 -- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc184092130 msgr2=0x7fc1840981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:28.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc17bfff700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc184092130 0x7fc1840981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:28.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc17bfff700 1 -- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1840b1280 msgr2=0x7fc18408b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc17bfff700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1840b1280 0x7fc18408b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc17bfff700 1 -- 172.21.15.32:0/1222332936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840987e0 con 0x7fc18408b930 2021-04-09T18:30:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc183fff700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1840b1280 0x7fc18408b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc17bfff700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18408b930 0x7fc184091b10 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7fc16c00b530 tx=0x7fc16c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc1817fa700 1 -- 172.21.15.32:0/1222332936 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc16c006020 con 0x7fc18408b930 2021-04-09T18:30:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc1817fa700 1 -- 172.21.15.32:0/1222332936 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16c0063a0 con 0x7fc18408b930 2021-04-09T18:30:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1840987e0 con 0x7fc18408b930 2021-04-09T18:30:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc1817fa700 1 -- 172.21.15.32:0/1222332936 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc16c004590 con 0x7fc18408b930 2021-04-09T18:30:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.260 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1840987e0 con 0x7fc18408b930 2021-04-09T18:30:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.261 7fc1817fa700 1 -- 172.21.15.32:0/1222332936 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc16c00a000 con 0x7fc18408b930 2021-04-09T18:30:28.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.261 7fc1817fa700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1680188c0 0x7fc16801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:28.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.261 7fc1817fa700 1 -- 172.21.15.32:0/1222332936 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fc16c032d30 con 0x7fc18408b930 2021-04-09T18:30:28.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.261 7fc183fff700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1680188c0 0x7fc16801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:28.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.261 7fc183fff700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1680188c0 0x7fc16801adc0 crc :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.280 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18417cae0 con 0x7fc18408b930 2021-04-09T18:30:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.284 7fc1817fa700 1 -- 172.21.15.32:0/1222332936 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc16c006b90 con 0x7fc18408b930 2021-04-09T18:30:28.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.473 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc18410d480 con 0x7fc18408b930 2021-04-09T18:30:28.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.474 7fc1817fa700 1 -- 172.21.15.32:0/1222332936 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 329 v329) v1 ==== 95+0+3719 (secure 0 0 0) 0x7fc16c00a000 con 0x7fc18408b930 2021-04-09T18:30:28.489 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 329 2021-04-09T18:30:28.489 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:28.489 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":329,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10920,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2893945823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2893945823},{"type":"v1","addr":"172.21.15.86:6837","nonce":2893945823}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":321},{"gid":58607,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1628176610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1628176610},{"type":"v1","addr":"172.21.15.86:6836","nonce":1628176610}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":321}],"filesystems":[{"mdsmap":{"epoch":325,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:17.772722","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10911":{"gid":10911,"name":"c","rank":0,"incarnation":322,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6834/3729993394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3729993394},{"type":"v1","addr":"172.21.15.32:6834","nonce":3729993394}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":329,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:25.213901","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10914},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10914":{"gid":10914,"name":"a","rank":0,"incarnation":326,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3158551863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3158551863},{"type":"v1","addr":"172.21.15.32:6835","nonce":3158551863}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:28.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.476 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1680188c0 msgr2=0x7fc16801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.476 7fc18a48a700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1680188c0 0x7fc16801adc0 crc :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.476 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18408b930 msgr2=0x7fc184091b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.476 7fc18a48a700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18408b930 0x7fc184091b10 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7fc16c00b530 tx=0x7fc16c00b8d0).stop 2021-04-09T18:30:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.477 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 shutdown_connections 2021-04-09T18:30:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.477 7fc18a48a700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1680188c0 0x7fc16801adc0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.477 7fc18a48a700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc18408b930 0x7fc184091b10 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.477 7fc18a48a700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc184092130 0x7fc1840981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.477 7fc18a48a700 1 --2- 172.21.15.32:0/1222332936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1840b1280 0x7fc18408b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.477 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 >> 172.21.15.32:0/1222332936 conn(0x7fc18417e6e0 msgr2=0x7fc184094dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.477 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 shutdown_connections 2021-04-09T18:30:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:28.477 7fc18a48a700 1 -- 172.21.15.32:0/1222332936 wait complete. 2021-04-09T18:30:28.903 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 325, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:15.764321', 'modified': '2021-04-09 18:30:17.772722', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10911}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10911': {'gid': 10911, 'name': 'c', 'rank': 0, 'incarnation': 322, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6834/3729993394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 3729993394}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 3729993394}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:28.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:30:28.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:30:29.042 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:30:29.043 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:30:29.043 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": -1, 2021-04-09T18:30:29.043 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 58739, 2021-04-09T18:30:29.044 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:boot", 2021-04-09T18:30:29.044 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "null", 2021-04-09T18:30:29.044 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 329, 2021-04-09T18:30:29.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 0, 2021-04-09T18:30:29.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 0, 2021-04-09T18:30:29.045 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 2.7706530250000001 2021-04-09T18:30:29.045 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:30:29.049 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": -1, "id": 58739, "want_state": "up:boot", "state": "null", "mdsmap_epoch": 329, "osdmap_epoch": 0, "osdmap_epoch_barrier": 0, "uptime": 2.7706530250000001 } 2021-04-09T18:30:30.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:30.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.196 7f8c9ac41700 1 Processor -- start 2021-04-09T18:30:30.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.196 7f8c9ac41700 1 -- start start 2021-04-09T18:30:30.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.197 7f8c9ac41700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8c940d7ca0 0x7f8c940d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:30.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.197 7f8c9ac41700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9410d480 con 0x7f8c9410d010 2021-04-09T18:30:30.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.197 7f8c9ac41700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c940d86c0 con 0x7f8c9410da00 2021-04-09T18:30:30.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.197 7f8c9ac41700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c940d9220 con 0x7f8c940d7ca0 2021-04-09T18:30:30.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.197 7f8c98a5d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8c9410d010 0x7f8c940d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37290/0 (socket says 172.21.15.32:37290) 2021-04-09T18:30:30.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.197 7f8c98a5d700 1 -- 172.21.15.32:0/2497103330 learned_addr learned my addr 172.21.15.32:0/2497103330 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:30.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c93fff700 1 --2- 172.21.15.32:0/2497103330 >> v2:172.21.15.86:3300/0 conn(0x7f8c940d7ca0 0x7f8c940d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:30.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870657287 0 0) 0x7f8c7c000ce0 con 0x7f8c9410d010 2021-04-09T18:30:30.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c80002fe0 con 0x7f8c9410d010 2021-04-09T18:30:30.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2614857780 0 0) 0x7f8c7c0010e0 con 0x7f8c9410d010 2021-04-09T18:30:30.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 >> v2:172.21.15.86:3300/0 conn(0x7f8c940d7ca0 msgr2=0x7f8c940d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:30.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 --2- 172.21.15.32:0/2497103330 >> v2:172.21.15.86:3300/0 conn(0x7f8c940d7ca0 0x7f8c940d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 >> v1:172.21.15.86:6789/0 conn(0x7f8c9410da00 legacy=0x7f8c9410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:30.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c940d1f70 con 0x7f8c9410d010 2021-04-09T18:30:30.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f8c7c001600 con 0x7f8c9410d010 2021-04-09T18:30:30.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 >> v1:172.21.15.32:6789/0 conn(0x7f8c9410d010 legacy=0x7f8c940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:30.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c80003a40 0x7f8c80005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:30.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c80006790 0x7f8c80008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c80009590 0x7f8c8000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8000c300 con 0x7f8c80009590 2021-04-09T18:30:30.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8000c520 con 0x7f8c80006790 2021-04-09T18:30:30.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8000c740 con 0x7f8c80003a40 2021-04-09T18:30:30.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c9925e700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c80009590 0x7f8c8000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:30.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c98a5d700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c80003a40 0x7f8c80005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:30.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c93fff700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c80006790 0x7f8c80008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c9925e700 1 -- 172.21.15.32:0/2497103330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c80003a40 msgr2=0x7f8c80005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c9925e700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c80003a40 0x7f8c80005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c9925e700 1 -- 172.21.15.32:0/2497103330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c80006790 msgr2=0x7f8c80008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:30.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c9925e700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c80006790 0x7f8c80008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c9925e700 1 -- 172.21.15.32:0/2497103330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c800509b0 con 0x7f8c80009590 2021-04-09T18:30:30.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c98a5d700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c80003a40 0x7f8c80005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:30.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.198 7f8c9925e700 1 -- 172.21.15.32:0/2497103330 reap_dead start 2021-04-09T18:30:30.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c9925e700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c80009590 0x7f8c8000bab0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f8c8c006ac0 tx=0x7f8c8c009af0).ready entity=mon.0 client_cookie=57f4de4c3a8ac86f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:30.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c8c00c0c0 con 0x7f8c80009590 2021-04-09T18:30:30.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8c00c440 con 0x7f8c80009590 2021-04-09T18:30:30.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c937fe700 1 -- 172.21.15.32:0/2497103330 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c8c00c920 con 0x7f8c80009590 2021-04-09T18:30:30.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c9ac41700 1 -- 172.21.15.32:0/2497103330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c80009590 msgr2=0x7f8c8000bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:30.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c9ac41700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c80009590 0x7f8c8000bab0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f8c8c006ac0 tx=0x7f8c8c009af0).stop 2021-04-09T18:30:30.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c9ac41700 1 -- 172.21.15.32:0/2497103330 shutdown_connections 2021-04-09T18:30:30.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c9ac41700 1 --2- 172.21.15.32:0/2497103330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c80009590 0x7f8c8000bab0 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c9ac41700 1 -- 172.21.15.32:0/2497103330 >> 172.21.15.32:0/2497103330 conn(0x7f8c940ce5f0 msgr2=0x7f8c940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:30.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c9ac41700 1 -- 172.21.15.32:0/2497103330 shutdown_connections 2021-04-09T18:30:30.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.199 7f8c9ac41700 1 -- 172.21.15.32:0/2497103330 wait complete. 2021-04-09T18:30:30.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.220 7f8c9ac41700 1 Processor -- start 2021-04-09T18:30:30.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.220 7f8c9ac41700 1 -- start start 2021-04-09T18:30:30.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c9ac41700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c940b42d0 0x7f8c94183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:30.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c9ac41700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c941ac090 0x7f8c941ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:30.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c9ac41700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c94184d30 0x7f8c94185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:30.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c9ac41700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940a1a90 con 0x7f8c941ac090 2021-04-09T18:30:30.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c9ac41700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c941acab0 con 0x7f8c940b42d0 2021-04-09T18:30:30.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c9ac41700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c94185a00 con 0x7f8c94184d30 2021-04-09T18:30:30.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c93fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c941ac090 0x7f8c941ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:30.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c93fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c941ac090 0x7f8c941ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55944/0 (socket says 172.21.15.32:55944) 2021-04-09T18:30:30.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c93fff700 1 -- 172.21.15.32:0/4192155273 learned_addr learned my addr 172.21.15.32:0/4192155273 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:30.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c9925e700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c94184d30 0x7f8c94185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:30.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c98a5d700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c940b42d0 0x7f8c94183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:30.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c93fff700 1 -- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c94184d30 msgr2=0x7f8c94185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:30.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c93fff700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c94184d30 0x7f8c94185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c93fff700 1 -- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c940b42d0 msgr2=0x7f8c94183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:30.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c93fff700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c940b42d0 0x7f8c94183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c93fff700 1 -- 172.21.15.32:0/4192155273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c94185750 con 0x7f8c941ac090 2021-04-09T18:30:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c93fff700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c941ac090 0x7f8c941ac490 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7f8c840099e0 tx=0x7f8c8400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c917fa700 1 -- 172.21.15.32:0/4192155273 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c8400c2f0 con 0x7f8c941ac090 2021-04-09T18:30:30.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c917fa700 1 -- 172.21.15.32:0/4192155273 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8400c630 con 0x7f8c941ac090 2021-04-09T18:30:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c94185750 con 0x7f8c941ac090 2021-04-09T18:30:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c917fa700 1 -- 172.21.15.32:0/4192155273 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c8400cb10 con 0x7f8c941ac090 2021-04-09T18:30:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.221 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c94185750 con 0x7f8c941ac090 2021-04-09T18:30:30.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.222 7f8c917fa700 1 -- 172.21.15.32:0/4192155273 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8c84001010 con 0x7f8c941ac090 2021-04-09T18:30:30.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.222 7f8c917fa700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c78018930 0x7f8c7801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:30.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.222 7f8c917fa700 1 -- 172.21.15.32:0/4192155273 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f8c840330d0 con 0x7f8c941ac090 2021-04-09T18:30:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.222 7f8c98a5d700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c78018930 0x7f8c7801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.223 7f8c98a5d700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c78018930 0x7f8c7801ae10 crc :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.241 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c94182d10 con 0x7f8c941ac090 2021-04-09T18:30:30.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.245 7f8c917fa700 1 -- 172.21.15.32:0/4192155273 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8c8400be30 con 0x7f8c941ac090 2021-04-09T18:30:30.325 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:30:30.311 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 2 filesystems are offline (MDS_ALL_DOWN) 2021-04-09T18:30:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.445 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c9410d480 con 0x7f8c941ac090 2021-04-09T18:30:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.446 7f8c917fa700 1 -- 172.21.15.32:0/4192155273 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 331 v331) v1 ==== 95+0+3717 (secure 0 0 0) 0x7f8c84001010 con 0x7f8c941ac090 2021-04-09T18:30:30.461 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 331 2021-04-09T18:30:30.461 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:30.461 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":331,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58739,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330},{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":331,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:30.314243","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:replay","state_seq":2,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":331,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:30.314243","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":186,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58736":{"gid":58736,"name":"a","rank":0,"incarnation":331,"state":"up:replay","state_seq":2,"addr":"172.21.15.32:6834/558658679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":558658679},{"type":"v1","addr":"172.21.15.32:6834","nonce":558658679}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c78018930 msgr2=0x7f8c7801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:30.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c78018930 0x7f8c7801ae10 crc :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c941ac090 msgr2=0x7f8c941ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:30.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c941ac090 0x7f8c941ac490 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7f8c840099e0 tx=0x7f8c8400bcf0).stop 2021-04-09T18:30:30.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 shutdown_connections 2021-04-09T18:30:30.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c78018930 0x7f8c7801ae10 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c941ac090 0x7f8c941ac490 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c94184d30 0x7f8c94185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 --2- 172.21.15.32:0/4192155273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c940b42d0 0x7f8c94183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:30.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.448 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 >> 172.21.15.32:0/4192155273 conn(0x7f8c9419b890 msgr2=0x7f8c941a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:30.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.449 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 shutdown_connections 2021-04-09T18:30:30.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:30.449 7f8c9ac41700 1 -- 172.21.15.32:0/4192155273 wait complete. 2021-04-09T18:30:30.875 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 331, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:15.764321', 'modified': '2021-04-09 18:30:30.314243', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 185, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11136}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11136': {'gid': 11136, 'name': 'd', 'rank': 0, 'incarnation': 331, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.86:6836/264493238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 264493238}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 264493238}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:30.876 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_11136:up:replay 2021-04-09T18:30:31.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:32.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.012 7f9f5a335700 1 Processor -- start 2021-04-09T18:30:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.012 7f9f5a335700 1 -- start start 2021-04-09T18:30:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.013 7f9f5a335700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540d7ca0 0x7f9f540d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.013 7f9f5a335700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9f5410da00 0x7f9f5410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.013 7f9f5a335700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5410d480 con 0x7f9f5410d010 2021-04-09T18:30:32.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.013 7f9f5a335700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f540d9220 con 0x7f9f5410da00 2021-04-09T18:30:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.013 7f9f5a335700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540d86c0 con 0x7f9f540d7ca0 2021-04-09T18:30:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.013 7f9f58952700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9f5410da00 0x7f9f5410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.013 7f9f4bfff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540d7ca0 0x7f9f540d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:32.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.013 7f9f53fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9f5410d010 0x7f9f540d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:35248/0 (socket says 172.21.15.32:35248) 2021-04-09T18:30:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.013 7f9f53fff700 1 -- 172.21.15.32:0/2531843243 learned_addr learned my addr 172.21.15.32:0/2531843243 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f58952700 1 -- 172.21.15.32:0/2531843243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540d7ca0 msgr2=0x7f9f540d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f58952700 1 --2- 172.21.15.32:0/2531843243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540d7ca0 0x7f9f540d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f58952700 1 -- 172.21.15.32:0/2531843243 >> v1:172.21.15.86:6789/0 conn(0x7f9f5410d010 legacy=0x7f9f540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:32.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f537fe700 1 -- 172.21.15.32:0/2531843243 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226483341 0 0) 0x7f9f440010c0 con 0x7f9f5410d010 2021-04-09T18:30:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f58952700 1 -- 172.21.15.32:0/2531843243 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f540d1f70 con 0x7f9f5410da00 2021-04-09T18:30:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f4bfff700 1 --2- 172.21.15.32:0/2531843243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540d7ca0 0x7f9f540d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f58952700 1 --2- 172.21.15.32:0/2531843243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f5410da00 0x7f9f5410de00 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f9f4c006e50 tx=0x7f9f4c009e70).ready entity=mon.1 client_cookie=c04a27ee07e1aa87 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:32.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f537fe700 1 -- 172.21.15.32:0/2531843243 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f4c006a20 con 0x7f9f5410da00 2021-04-09T18:30:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f537fe700 1 -- 172.21.15.32:0/2531843243 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c006790 con 0x7f9f5410da00 2021-04-09T18:30:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f537fe700 1 -- 172.21.15.32:0/2531843243 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f4c00b700 con 0x7f9f5410da00 2021-04-09T18:30:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f5a335700 1 -- 172.21.15.32:0/2531843243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f5410da00 msgr2=0x7f9f5410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:32.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.014 7f9f5a335700 1 --2- 172.21.15.32:0/2531843243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f5410da00 0x7f9f5410de00 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f9f4c006e50 tx=0x7f9f4c009e70).stop 2021-04-09T18:30:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.015 7f9f5a335700 1 -- 172.21.15.32:0/2531843243 shutdown_connections 2021-04-09T18:30:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.015 7f9f5a335700 1 --2- 172.21.15.32:0/2531843243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f5410da00 0x7f9f5410de00 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.015 7f9f5a335700 1 --2- 172.21.15.32:0/2531843243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540d7ca0 0x7f9f540d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.015 7f9f5a335700 1 -- 172.21.15.32:0/2531843243 >> 172.21.15.32:0/2531843243 conn(0x7f9f540ce5f0 msgr2=0x7f9f540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.015 7f9f5a335700 1 -- 172.21.15.32:0/2531843243 shutdown_connections 2021-04-09T18:30:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.015 7f9f5a335700 1 -- 172.21.15.32:0/2531843243 wait complete. 2021-04-09T18:30:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.035 7f9f5a335700 1 Processor -- start 2021-04-09T18:30:32.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.035 7f9f5a335700 1 -- start start 2021-04-09T18:30:32.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f5a335700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540b5360 0x7f9f541a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:32.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f5a335700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f5418e140 0x7f9f541aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:32.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f5a335700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f541ad040 0x7f9f5408fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:32.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f5a335700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f541a5360 con 0x7f9f5418e140 2021-04-09T18:30:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f5a335700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540902c0 con 0x7f9f541ad040 2021-04-09T18:30:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f5a335700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540904e0 con 0x7f9f540b5360 2021-04-09T18:30:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f4bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f5418e140 0x7f9f541aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f4bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f5418e140 0x7f9f541aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55970/0 (socket says 172.21.15.32:55970) 2021-04-09T18:30:32.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f4bfff700 1 -- 172.21.15.32:0/2282075611 learned_addr learned my addr 172.21.15.32:0/2282075611 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f58952700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f541ad040 0x7f9f5408fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f53fff700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540b5360 0x7f9f541a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f4bfff700 1 -- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540b5360 msgr2=0x7f9f541a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:32.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f4bfff700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540b5360 0x7f9f541a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f4bfff700 1 -- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f541ad040 msgr2=0x7f9f5408fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:32.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f4bfff700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f541ad040 0x7f9f5408fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f4bfff700 1 -- 172.21.15.32:0/2282075611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f54183c80 con 0x7f9f5418e140 2021-04-09T18:30:32.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f4bfff700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f5418e140 0x7f9f541aca20 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7f9f3c006f40 tx=0x7f9f3c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:32.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f517fa700 1 -- 172.21.15.32:0/2282075611 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f3c00bd80 con 0x7f9f5418e140 2021-04-09T18:30:32.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f517fa700 1 -- 172.21.15.32:0/2282075611 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c000c90 con 0x7f9f5418e140 2021-04-09T18:30:32.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f54183c80 con 0x7f9f5418e140 2021-04-09T18:30:32.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.036 7f9f517fa700 1 -- 172.21.15.32:0/2282075611 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f3c002220 con 0x7f9f5418e140 2021-04-09T18:30:32.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.037 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f54183c80 con 0x7f9f5418e140 2021-04-09T18:30:32.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.037 7f9f517fa700 1 -- 172.21.15.32:0/2282075611 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9f3c00a000 con 0x7f9f5418e140 2021-04-09T18:30:32.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.037 7f9f517fa700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f380188c0 0x7f9f3801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:32.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.037 7f9f517fa700 1 -- 172.21.15.32:0/2282075611 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f9f3c032b80 con 0x7f9f5418e140 2021-04-09T18:30:32.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.037 7f9f53fff700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f380188c0 0x7f9f3801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:32.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.038 7f9f53fff700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f380188c0 0x7f9f3801adc0 crc :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:32.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.057 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f54182d70 con 0x7f9f5418e140 2021-04-09T18:30:32.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.060 7f9f517fa700 1 -- 172.21.15.32:0/2282075611 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9f3c006b00 con 0x7f9f5418e140 2021-04-09T18:30:32.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.252 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f5410d480 con 0x7f9f5418e140 2021-04-09T18:30:32.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.253 7f9f517fa700 1 -- 172.21.15.32:0/2282075611 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 332 v332) v1 ==== 95+0+3723 (secure 0 0 0) 0x7f9f3c00a000 con 0x7f9f5418e140 2021-04-09T18:30:32.269 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 332 2021-04-09T18:30:32.269 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:32.270 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":332,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58739,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330},{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:31.316038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:31.316038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":186,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58736":{"gid":58736,"name":"a","rank":0,"incarnation":331,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.32:6834/558658679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":558658679},{"type":"v1","addr":"172.21.15.32:6834","nonce":558658679}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:32.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f380188c0 msgr2=0x7f9f3801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:32.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f380188c0 0x7f9f3801adc0 crc :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f5418e140 msgr2=0x7f9f541aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f5418e140 0x7f9f541aca20 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7f9f3c006f40 tx=0x7f9f3c00b990).stop 2021-04-09T18:30:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 shutdown_connections 2021-04-09T18:30:32.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f380188c0 0x7f9f3801adc0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f5418e140 0x7f9f541aca20 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f541ad040 0x7f9f5408fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 --2- 172.21.15.32:0/2282075611 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f540b5360 0x7f9f541a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:32.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 >> 172.21.15.32:0/2282075611 conn(0x7f9f5419b460 msgr2=0x7f9f5419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 shutdown_connections 2021-04-09T18:30:32.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:32.255 7f9f5a335700 1 -- 172.21.15.32:0/2282075611 wait complete. 2021-04-09T18:30:32.683 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 332, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:15.764321', 'modified': '2021-04-09 18:30:31.316038', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 185, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11136}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11136': {'gid': 11136, 'name': 'd', 'rank': 0, 'incarnation': 331, 'state': 'up:reconnect', 'state_seq': 3, 'addr': '172.21.15.86:6836/264493238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 264493238}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 264493238}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:32.684 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_11136:up:reconnect 2021-04-09T18:30:33.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:33.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.830 7f8b8d414700 1 Processor -- start 2021-04-09T18:30:33.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.830 7f8b8d414700 1 -- start start 2021-04-09T18:30:33.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.831 7f8b8d414700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8b8810d010 0x7f8b880d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:33.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.831 7f8b8d414700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b880d8490 0x7f8b8810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:33.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.831 7f8b8d414700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8810d480 con 0x7f8b880d1da0 2021-04-09T18:30:33.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.831 7f8b8d414700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b880d8d30 con 0x7f8b880d8490 2021-04-09T18:30:33.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.831 7f8b8d414700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b880d9220 con 0x7f8b8810d010 2021-04-09T18:30:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.831 7f8b8787e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b880d8490 0x7f8b8810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.831 7f8b8687c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8b880d1da0 0x7f8b880d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37340/0 (socket says 172.21.15.32:37340) 2021-04-09T18:30:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.831 7f8b8687c700 1 -- 172.21.15.32:0/1657425299 learned_addr learned my addr 172.21.15.32:0/1657425299 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:33.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8707d700 1 --2- 172.21.15.32:0/1657425299 >> v2:172.21.15.86:3300/0 conn(0x7f8b8810d010 0x7f8b880d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:33.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8787e700 1 -- 172.21.15.32:0/1657425299 >> v2:172.21.15.86:3300/0 conn(0x7f8b8810d010 msgr2=0x7f8b880d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:33.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8787e700 1 --2- 172.21.15.32:0/1657425299 >> v2:172.21.15.86:3300/0 conn(0x7f8b8810d010 0x7f8b880d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:33.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8787e700 1 -- 172.21.15.32:0/1657425299 >> v1:172.21.15.32:6789/0 conn(0x7f8b880d1da0 legacy=0x7f8b880d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:33.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8607b700 1 -- 172.21.15.32:0/1657425299 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473927471 0 0) 0x7f8b740009b0 con 0x7f8b880d1da0 2021-04-09T18:30:33.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8787e700 1 -- 172.21.15.32:0/1657425299 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8810e020 con 0x7f8b880d8490 2021-04-09T18:30:33.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8787e700 1 --2- 172.21.15.32:0/1657425299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b880d8490 0x7f8b8810da00 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 rx=0x7f8b80006e30 tx=0x7f8b80009de0).ready entity=mon.0 client_cookie=3a0834c8f962dbed server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:33.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8607b700 1 -- 172.21.15.32:0/1657425299 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8000b300 con 0x7f8b880d8490 2021-04-09T18:30:33.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8607b700 1 -- 172.21.15.32:0/1657425299 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8000b680 con 0x7f8b880d8490 2021-04-09T18:30:33.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8607b700 1 -- 172.21.15.32:0/1657425299 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8000bb60 con 0x7f8b880d8490 2021-04-09T18:30:33.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8d414700 1 -- 172.21.15.32:0/1657425299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b880d8490 msgr2=0x7f8b8810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:33.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.832 7f8b8d414700 1 --2- 172.21.15.32:0/1657425299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b880d8490 0x7f8b8810da00 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 rx=0x7f8b80006e30 tx=0x7f8b80009de0).stop 2021-04-09T18:30:33.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.833 7f8b8d414700 1 -- 172.21.15.32:0/1657425299 shutdown_connections 2021-04-09T18:30:33.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.833 7f8b8d414700 1 --2- 172.21.15.32:0/1657425299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b880d8490 0x7f8b8810da00 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:33.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.833 7f8b8d414700 1 --2- 172.21.15.32:0/1657425299 >> v2:172.21.15.86:3300/0 conn(0x7f8b8810d010 0x7f8b880d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:33.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.833 7f8b8d414700 1 -- 172.21.15.32:0/1657425299 >> 172.21.15.32:0/1657425299 conn(0x7f8b880ce5f0 msgr2=0x7f8b880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:33.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.833 7f8b8d414700 1 -- 172.21.15.32:0/1657425299 shutdown_connections 2021-04-09T18:30:33.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.833 7f8b8d414700 1 -- 172.21.15.32:0/1657425299 wait complete. 2021-04-09T18:30:33.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.853 7f8b8d414700 1 Processor -- start 2021-04-09T18:30:33.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.853 7f8b8d414700 1 -- start start 2021-04-09T18:30:33.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8d414700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b8818c5c0 0x7f8b881a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:33.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8d414700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b88191880 0x7f8b8818cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:33.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8d414700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b88191c80 0x7f8b8818d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:33.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8d414700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8818c040 con 0x7f8b8818c5c0 2021-04-09T18:30:33.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8d414700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b881aa5c0 con 0x7f8b88191c80 2021-04-09T18:30:33.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8d414700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8818bd20 con 0x7f8b88191880 2021-04-09T18:30:33.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8707d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b8818c5c0 0x7f8b881a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:33.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8707d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b8818c5c0 0x7f8b881a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55984/0 (socket says 172.21.15.32:55984) 2021-04-09T18:30:33.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8707d700 1 -- 172.21.15.32:0/1714289637 learned_addr learned my addr 172.21.15.32:0/1714289637 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:33.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8787e700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b88191c80 0x7f8b8818d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:33.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8687c700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b88191880 0x7f8b8818cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:33.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8707d700 1 -- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b88191880 msgr2=0x7f8b8818cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:33.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8707d700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b88191880 0x7f8b8818cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:33.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8707d700 1 -- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b88191c80 msgr2=0x7f8b8818d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:33.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8707d700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b88191c80 0x7f8b8818d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:33.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8707d700 1 -- 172.21.15.32:0/1714289637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8818da30 con 0x7f8b8818c5c0 2021-04-09T18:30:33.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8787e700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b88191c80 0x7f8b8818d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:33.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8707d700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b8818c5c0 0x7f8b881a9fa0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f8b7800b530 tx=0x7f8b7800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:33.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b7ffff700 1 -- 172.21.15.32:0/1714289637 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b7800bcc0 con 0x7f8b8818c5c0 2021-04-09T18:30:33.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b7ffff700 1 -- 172.21.15.32:0/1714289637 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b78002030 con 0x7f8b8818c5c0 2021-04-09T18:30:33.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8818da30 con 0x7f8b8818c5c0 2021-04-09T18:30:33.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b7ffff700 1 -- 172.21.15.32:0/1714289637 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b78002480 con 0x7f8b8818c5c0 2021-04-09T18:30:33.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.854 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8818da30 con 0x7f8b8818c5c0 2021-04-09T18:30:33.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.855 7f8b7ffff700 1 -- 172.21.15.32:0/1714289637 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8b78006020 con 0x7f8b8818c5c0 2021-04-09T18:30:33.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.855 7f8b7ffff700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b64018930 0x7f8b6401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:33.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.855 7f8b7ffff700 1 -- 172.21.15.32:0/1714289637 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f8b78032e00 con 0x7f8b8818c5c0 2021-04-09T18:30:33.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.855 7f8b8687c700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b64018930 0x7f8b6401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:33.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.856 7f8b8687c700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b64018930 0x7f8b6401adc0 crc :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.875 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8818dfc0 con 0x7f8b8818c5c0 2021-04-09T18:30:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:33.879 7f8b7ffff700 1 -- 172.21.15.32:0/1714289637 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8b78000c90 con 0x7f8b8818c5c0 2021-04-09T18:30:34.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.072 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b8818dfc0 con 0x7f8b8818c5c0 2021-04-09T18:30:34.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.073 7f8b7ffff700 1 -- 172.21.15.32:0/1714289637 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 334 v334) v1 ==== 95+0+3717 (secure 0 0 0) 0x7f8b78006000 con 0x7f8b8818c5c0 2021-04-09T18:30:34.089 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 334 2021-04-09T18:30:34.090 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:34.090 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":334,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58739,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330},{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":186,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58736":{"gid":58736,"name":"a","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/558658679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":558658679},{"type":"v1","addr":"172.21.15.32:6834","nonce":558658679}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:34.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b64018930 msgr2=0x7f8b6401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:34.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b64018930 0x7f8b6401adc0 crc :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b8818c5c0 msgr2=0x7f8b881a9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:34.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b8818c5c0 0x7f8b881a9fa0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f8b7800b530 tx=0x7f8b7800b8d0).stop 2021-04-09T18:30:34.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 shutdown_connections 2021-04-09T18:30:34.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b64018930 0x7f8b6401adc0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b88191c80 0x7f8b8818d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b88191880 0x7f8b8818cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 --2- 172.21.15.32:0/1714289637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b8818c5c0 0x7f8b881a9fa0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.075 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 >> 172.21.15.32:0/1714289637 conn(0x7f8b881837d0 msgr2=0x7f8b88195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:34.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.076 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 shutdown_connections 2021-04-09T18:30:34.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.076 7f8b8d414700 1 -- 172.21.15.32:0/1714289637 wait complete. 2021-04-09T18:30:34.503 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 334, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:15.764321', 'modified': '2021-04-09 18:30:33.326504', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 185, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11136}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11136': {'gid': 11136, 'name': 'd', 'rank': 0, 'incarnation': 331, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/264493238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 264493238}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 264493238}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:34.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:30:34.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:30:34.631 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:30:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:30:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:30:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 11136, 2021-04-09T18:30:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:30:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:30:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.3010161629999999, 2021-04-09T18:30:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 334, 2021-04-09T18:30:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 188, 2021-04-09T18:30:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 188, 2021-04-09T18:30:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 8.3419680599999992 2021-04-09T18:30:34.634 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:30:34.637 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 11136, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.3010161629999999, "mdsmap_epoch": 334, "osdmap_epoch": 188, "osdmap_epoch_barrier": 188, "uptime": 8.3419680599999992 } 2021-04-09T18:30:34.637 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:34.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.772 7f4db3e5e700 1 Processor -- start 2021-04-09T18:30:34.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.772 7f4db3e5e700 1 -- start start 2021-04-09T18:30:34.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db3e5e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac10d010 0x7f4dac0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:34.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db3e5e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4dac0d1da0 0x7f4dac0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:34.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db3e5e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac10d480 con 0x7f4dac0d82c0 2021-04-09T18:30:34.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db3e5e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dac10e780 con 0x7f4dac0d1da0 2021-04-09T18:30:34.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db3e5e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0d8d30 con 0x7f4dac10d010 2021-04-09T18:30:34.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db247b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4dac0d82c0 0x7f4dac10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37352/0 (socket says 172.21.15.32:37352) 2021-04-09T18:30:34.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db247b700 1 -- 172.21.15.32:0/3882762658 learned_addr learned my addr 172.21.15.32:0/3882762658 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:34.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db1479700 1 --2- 172.21.15.32:0/3882762658 >> v2:172.21.15.86:3300/0 conn(0x7f4dac0d1da0 0x7f4dac0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:34.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db1c7a700 1 --2- 172.21.15.32:0/3882762658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac10d010 0x7f4dac0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:34.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db0c78700 1 -- 172.21.15.32:0/3882762658 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743554852 0 0) 0x7f4da4000d30 con 0x7f4dac0d82c0 2021-04-09T18:30:34.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.773 7f4db0c78700 1 -- 172.21.15.32:0/3882762658 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d98002fe0 con 0x7f4dac0d82c0 2021-04-09T18:30:34.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db1479700 1 -- 172.21.15.32:0/3882762658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac10d010 msgr2=0x7f4dac0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:34.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db1479700 1 --2- 172.21.15.32:0/3882762658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac10d010 0x7f4dac0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db1479700 1 -- 172.21.15.32:0/3882762658 >> v1:172.21.15.32:6789/0 conn(0x7f4dac0d82c0 legacy=0x7f4dac10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:34.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db0c78700 1 -- 172.21.15.32:0/3882762658 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2881599935 0 0) 0x7f4da4001270 con 0x7f4dac0d82c0 2021-04-09T18:30:34.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db1479700 1 -- 172.21.15.32:0/3882762658 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0d9220 con 0x7f4dac0d1da0 2021-04-09T18:30:34.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db1479700 1 --2- 172.21.15.32:0/3882762658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dac0d1da0 0x7f4dac0d7ca0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f4d9c006650 tx=0x7f4d9c009a80).ready entity=mon.1 client_cookie=8b7e4bb87343a776 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:34.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db0c78700 1 -- 172.21.15.32:0/3882762658 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d9c0009b0 con 0x7f4dac0d1da0 2021-04-09T18:30:34.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db0c78700 1 -- 172.21.15.32:0/3882762658 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c00c0c0 con 0x7f4dac0d1da0 2021-04-09T18:30:34.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db0c78700 1 -- 172.21.15.32:0/3882762658 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d9c00c510 con 0x7f4dac0d1da0 2021-04-09T18:30:34.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db3e5e700 1 -- 172.21.15.32:0/3882762658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dac0d1da0 msgr2=0x7f4dac0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:34.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.774 7f4db3e5e700 1 --2- 172.21.15.32:0/3882762658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dac0d1da0 0x7f4dac0d7ca0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f4d9c006650 tx=0x7f4d9c009a80).stop 2021-04-09T18:30:34.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.775 7f4db3e5e700 1 -- 172.21.15.32:0/3882762658 shutdown_connections 2021-04-09T18:30:34.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.775 7f4db3e5e700 1 --2- 172.21.15.32:0/3882762658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dac0d1da0 0x7f4dac0d7ca0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.775 7f4db3e5e700 1 --2- 172.21.15.32:0/3882762658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac10d010 0x7f4dac0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.775 7f4db3e5e700 1 -- 172.21.15.32:0/3882762658 >> 172.21.15.32:0/3882762658 conn(0x7f4dac0ce5f0 msgr2=0x7f4dac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:34.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.775 7f4db3e5e700 1 -- 172.21.15.32:0/3882762658 shutdown_connections 2021-04-09T18:30:34.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.775 7f4db3e5e700 1 -- 172.21.15.32:0/3882762658 wait complete. 2021-04-09T18:30:34.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.795 7f4db3e5e700 1 Processor -- start 2021-04-09T18:30:34.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.795 7f4db3e5e700 1 -- start start 2021-04-09T18:30:34.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db3e5e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac1ac380 0x7f4dac18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:34.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db3e5e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dac1ac780 0x7f4dac18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:34.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db3e5e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dac183520 0x7f4dac183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:34.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db3e5e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac1ace10 con 0x7f4dac183520 2021-04-09T18:30:34.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db3e5e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac189cf0 con 0x7f4dac1ac780 2021-04-09T18:30:34.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db3e5e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac183f40 con 0x7f4dac1ac380 2021-04-09T18:30:34.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db247b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dac183520 0x7f4dac183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:34.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db247b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dac183520 0x7f4dac183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56002/0 (socket says 172.21.15.32:56002) 2021-04-09T18:30:34.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db247b700 1 -- 172.21.15.32:0/1966442944 learned_addr learned my addr 172.21.15.32:0/1966442944 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:34.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db1479700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dac1ac780 0x7f4dac18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:34.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db1c7a700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac1ac380 0x7f4dac18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:34.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db247b700 1 -- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac1ac380 msgr2=0x7f4dac18a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db247b700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac1ac380 0x7f4dac18a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db247b700 1 -- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dac1ac780 msgr2=0x7f4dac18a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db247b700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dac1ac780 0x7f4dac18a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:34.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db247b700 1 -- 172.21.15.32:0/1966442944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac184160 con 0x7f4dac183520 2021-04-09T18:30:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db247b700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dac183520 0x7f4dac183920 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f4da400a880 tx=0x7f4da400ac20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:34.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4da27fc700 1 -- 172.21.15.32:0/1966442944 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da4004930 con 0x7f4dac183520 2021-04-09T18:30:34.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4da27fc700 1 -- 172.21.15.32:0/1966442944 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da4004c30 con 0x7f4dac183520 2021-04-09T18:30:34.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac184160 con 0x7f4dac183520 2021-04-09T18:30:34.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.796 7f4da27fc700 1 -- 172.21.15.32:0/1966442944 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da40072a0 con 0x7f4dac183520 2021-04-09T18:30:34.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.797 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac184160 con 0x7f4dac183520 2021-04-09T18:30:34.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.797 7f4da27fc700 1 -- 172.21.15.32:0/1966442944 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4da400ad60 con 0x7f4dac183520 2021-04-09T18:30:34.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.797 7f4da27fc700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d94018930 0x7f4d9401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:34.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.797 7f4da27fc700 1 -- 172.21.15.32:0/1966442944 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f4da4033130 con 0x7f4dac183520 2021-04-09T18:30:34.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.798 7f4db1c7a700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d94018930 0x7f4d9401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:34.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.798 7f4db1c7a700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d94018930 0x7f4d9401ae10 crc :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:34.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.817 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac189cf0 con 0x7f4dac183520 2021-04-09T18:30:34.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:34.821 7f4da27fc700 1 -- 172.21.15.32:0/1966442944 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4da4004c30 con 0x7f4dac183520 2021-04-09T18:30:35.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.012 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dac189cf0 con 0x7f4dac183520 2021-04-09T18:30:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.013 7f4da27fc700 1 -- 172.21.15.32:0/1966442944 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 334 v334) v1 ==== 95+0+3717 (secure 0 0 0) 0x7f4da400ad60 con 0x7f4dac183520 2021-04-09T18:30:35.029 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 334 2021-04-09T18:30:35.029 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:35.030 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":334,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58739,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330},{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":186,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58736":{"gid":58736,"name":"a","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/558658679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":558658679},{"type":"v1","addr":"172.21.15.32:6834","nonce":558658679}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:35.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.015 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d94018930 msgr2=0x7f4d9401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:35.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.015 7f4db3e5e700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d94018930 0x7f4d9401ae10 crc :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.015 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dac183520 msgr2=0x7f4dac183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.015 7f4db3e5e700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dac183520 0x7f4dac183920 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f4da400a880 tx=0x7f4da400ac20).stop 2021-04-09T18:30:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.015 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 shutdown_connections 2021-04-09T18:30:35.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.015 7f4db3e5e700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4d94018930 0x7f4d9401ae10 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.015 7f4db3e5e700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dac183520 0x7f4dac183920 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.015 7f4db3e5e700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dac1ac780 0x7f4dac18a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.015 7f4db3e5e700 1 --2- 172.21.15.32:0/1966442944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dac1ac380 0x7f4dac18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.016 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 >> 172.21.15.32:0/1966442944 conn(0x7f4dac19b890 msgr2=0x7f4dac1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:35.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.016 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 shutdown_connections 2021-04-09T18:30:35.034 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.016 7f4db3e5e700 1 -- 172.21.15.32:0/1966442944 wait complete. 2021-04-09T18:30:35.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 334, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:23.208891', 'modified': '2021-04-09 18:30:33.326504', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 186, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 58736}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_58736': {'gid': 58736, 'name': 'a', 'rank': 0, 'incarnation': 331, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6834/558658679', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 558658679}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 558658679}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'bravo', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:35.444 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:30:35.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:30:35.585 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:30:35.586 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:30:35.586 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:30:35.586 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 58736, 2021-04-09T18:30:35.586 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:30:35.587 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:30:35.587 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 5.2546975270000003, 2021-04-09T18:30:35.587 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 334, 2021-04-09T18:30:35.588 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 188, 2021-04-09T18:30:35.588 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 188, 2021-04-09T18:30:35.588 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 9.3138930430000002 2021-04-09T18:30:35.588 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:30:35.592 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 58736, "want_state": "up:active", "state": "up:active", "rank_uptime": 5.2546975270000003, "mdsmap_epoch": 334, "osdmap_epoch": 188, "osdmap_epoch_barrier": 188, "uptime": 9.3138930430000002 } 2021-04-09T18:30:35.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=alpha_data, allow rw pool=bravo_data' 2021-04-09T18:30:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.738 7ff2a41c2700 1 Processor -- start 2021-04-09T18:30:35.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.738 7ff2a41c2700 1 -- start start 2021-04-09T18:30:35.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a41c2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c0d2700 0x7ff29c0cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:35.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a41c2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff29c1b04f0 0x7ff29c1b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:35.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a41c2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c0d8700 con 0x7ff29c0f28c0 2021-04-09T18:30:35.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a41c2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29c0d91b0 con 0x7ff29c1b04f0 2021-04-09T18:30:35.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a41c2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c0d8440 con 0x7ff29c0d2700 2021-04-09T18:30:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a17dd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff29c0f28c0 0x7ff29c0d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37368/0 (socket says 172.21.15.32:37368) 2021-04-09T18:30:35.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a17dd700 1 -- 172.21.15.32:0/4029129883 learned_addr learned my addr 172.21.15.32:0/4029129883 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a1fde700 1 --2- 172.21.15.32:0/4029129883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c0d2700 0x7ff29c0cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a27df700 1 --2- 172.21.15.32:0/4029129883 >> v2:172.21.15.86:3300/0 conn(0x7ff29c1b04f0 0x7ff29c1b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:35.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a0fdc700 1 -- 172.21.15.32:0/4029129883 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859860326 0 0) 0x7ff28c000ce0 con 0x7ff29c0f28c0 2021-04-09T18:30:35.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.739 7ff2a0fdc700 1 -- 172.21.15.32:0/4029129883 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff288002fe0 con 0x7ff29c0f28c0 2021-04-09T18:30:35.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a0fdc700 1 -- 172.21.15.32:0/4029129883 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3627229122 0 0) 0x7ff28c0010e0 con 0x7ff29c0f28c0 2021-04-09T18:30:35.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a1fde700 1 -- 172.21.15.32:0/4029129883 >> v2:172.21.15.86:3300/0 conn(0x7ff29c1b04f0 msgr2=0x7ff29c1b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:35.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a1fde700 1 --2- 172.21.15.32:0/4029129883 >> v2:172.21.15.86:3300/0 conn(0x7ff29c1b04f0 0x7ff29c1b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a1fde700 1 -- 172.21.15.32:0/4029129883 >> v1:172.21.15.32:6789/0 conn(0x7ff29c0f28c0 legacy=0x7ff29c0d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:35.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a1fde700 1 -- 172.21.15.32:0/4029129883 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c108dc0 con 0x7ff29c0d2700 2021-04-09T18:30:35.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a27df700 1 --2- 172.21.15.32:0/4029129883 >> v2:172.21.15.86:3300/0 conn(0x7ff29c1b04f0 0x7ff29c1b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:35.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a1fde700 1 --2- 172.21.15.32:0/4029129883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c0d2700 0x7ff29c0cfeb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7ff298006650 tx=0x7ff298009a80).ready entity=mon.2 client_cookie=87c17f0f41f5be21 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:35.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a0fdc700 1 -- 172.21.15.32:0/4029129883 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2980009b0 con 0x7ff29c0d2700 2021-04-09T18:30:35.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a0fdc700 1 -- 172.21.15.32:0/4029129883 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29800c0c0 con 0x7ff29c0d2700 2021-04-09T18:30:35.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a0fdc700 1 -- 172.21.15.32:0/4029129883 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29800c510 con 0x7ff29c0d2700 2021-04-09T18:30:35.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a41c2700 1 -- 172.21.15.32:0/4029129883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c0d2700 msgr2=0x7ff29c0cfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:35.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.740 7ff2a41c2700 1 --2- 172.21.15.32:0/4029129883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c0d2700 0x7ff29c0cfeb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7ff298006650 tx=0x7ff298009a80).stop 2021-04-09T18:30:35.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.741 7ff2a41c2700 1 -- 172.21.15.32:0/4029129883 shutdown_connections 2021-04-09T18:30:35.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.741 7ff2a41c2700 1 --2- 172.21.15.32:0/4029129883 >> v2:172.21.15.86:3300/0 conn(0x7ff29c1b04f0 0x7ff29c1b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.741 7ff2a41c2700 1 --2- 172.21.15.32:0/4029129883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c0d2700 0x7ff29c0cfeb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.741 7ff2a41c2700 1 -- 172.21.15.32:0/4029129883 >> 172.21.15.32:0/4029129883 conn(0x7ff29c0ce590 msgr2=0x7ff29c0cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:35.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.741 7ff2a41c2700 1 -- 172.21.15.32:0/4029129883 shutdown_connections 2021-04-09T18:30:35.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.741 7ff2a41c2700 1 -- 172.21.15.32:0/4029129883 wait complete. 2021-04-09T18:30:35.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.761 7ff2a41c2700 1 Processor -- start 2021-04-09T18:30:35.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.761 7ff2a41c2700 1 -- start start 2021-04-09T18:30:35.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a41c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c09a540 0x7ff29c0c3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:35.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a41c2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c09b2e0 0x7ff29c09b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:35.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a41c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff29c104880 0x7ff29c104c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:35.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a41c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c09ad60 con 0x7ff29c104880 2021-04-09T18:30:35.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a41c2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c19b1d0 con 0x7ff29c09a540 2021-04-09T18:30:35.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a41c2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c09bd00 con 0x7ff29c09b2e0 2021-04-09T18:30:35.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a27df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff29c104880 0x7ff29c104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:35.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a27df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff29c104880 0x7ff29c104c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56016/0 (socket says 172.21.15.32:56016) 2021-04-09T18:30:35.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a27df700 1 -- 172.21.15.32:0/3886210210 learned_addr learned my addr 172.21.15.32:0/3886210210 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:35.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a17dd700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c09b2e0 0x7ff29c09b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:35.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a1fde700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c09a540 0x7ff29c0c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:35.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a27df700 1 -- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c09b2e0 msgr2=0x7ff29c09b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:35.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a27df700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c09b2e0 0x7ff29c09b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a27df700 1 -- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c09a540 msgr2=0x7ff29c0c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:35.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a27df700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c09a540 0x7ff29c0c3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a27df700 1 -- 172.21.15.32:0/3886210210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c0c3b70 con 0x7ff29c104880 2021-04-09T18:30:35.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2a27df700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff29c104880 0x7ff29c104c80 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7ff29400b970 tx=0x7ff29400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:35.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.762 7ff2927fc700 1 -- 172.21.15.32:0/3886210210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff294000f90 con 0x7ff29c104880 2021-04-09T18:30:35.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.763 7ff2927fc700 1 -- 172.21.15.32:0/3886210210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff294006aa0 con 0x7ff29c104880 2021-04-09T18:30:35.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.763 7ff2927fc700 1 -- 172.21.15.32:0/3886210210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2940015f0 con 0x7ff29c104880 2021-04-09T18:30:35.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.763 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29c0c3b70 con 0x7ff29c104880 2021-04-09T18:30:35.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.763 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29c0c3b70 con 0x7ff29c104880 2021-04-09T18:30:35.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.763 7ff2927fc700 1 -- 172.21.15.32:0/3886210210 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff294000e70 con 0x7ff29c104880 2021-04-09T18:30:35.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.764 7ff2927fc700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2840188c0 0x7ff28401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:35.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.764 7ff2927fc700 1 -- 172.21.15.32:0/3886210210 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7ff294032f20 con 0x7ff29c104880 2021-04-09T18:30:35.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.764 7ff2a1fde700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2840188c0 0x7ff28401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:35.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.764 7ff2a1fde700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2840188c0 0x7ff28401adc0 crc :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:35.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.783 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29c103970 con 0x7ff29c104880 2021-04-09T18:30:35.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.787 7ff2927fc700 1 -- 172.21.15.32:0/3886210210 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff2940066f0 con 0x7ff29c104880 2021-04-09T18:30:35.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.971 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]} v 0) v1 -- 0x7ff29c103970 con 0x7ff29c104880 2021-04-09T18:30:35.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.975 7ff2927fc700 1 -- 172.21.15.32:0/3886210210 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]}]=0 updated caps for client.0 v40) v1 ==== 209+0+0 (secure 0 0 0) 0x7ff294006b70 con 0x7ff29c104880 2021-04-09T18:30:35.994 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:30:35.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2840188c0 msgr2=0x7ff28401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:35.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2840188c0 0x7ff28401adc0 crc :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff29c104880 msgr2=0x7ff29c104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:35.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff29c104880 0x7ff29c104c80 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7ff29400b970 tx=0x7ff29400bd10).stop 2021-04-09T18:30:35.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 shutdown_connections 2021-04-09T18:30:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff2840188c0 0x7ff28401adc0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff29c104880 0x7ff29c104c80 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff29c09b2e0 0x7ff29c09b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 --2- 172.21.15.32:0/3886210210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c09a540 0x7ff29c0c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 >> 172.21.15.32:0/3886210210 conn(0x7ff29c0c42e0 msgr2=0x7ff29c08fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 shutdown_connections 2021-04-09T18:30:35.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:35.981 7ff2a41c2700 1 -- 172.21.15.32:0/3886210210 wait complete. 2021-04-09T18:30:36.006 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=alpha_data, allow rw pool=bravo_data' 2021-04-09T18:30:36.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.151 7f771d85d700 1 Processor -- start 2021-04-09T18:30:36.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.151 7f771d85d700 1 -- start start 2021-04-09T18:30:36.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f771d85d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f77180cffb0 0x7f7718087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f771d85d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77180d7c50 0x7f77181b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f771d85d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7718088460 con 0x7f77180d8050 2021-04-09T18:30:36.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f771d85d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77180d91f0 con 0x7f77180cffb0 2021-04-09T18:30:36.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f771d85d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77180d8620 con 0x7f77180d7c50 2021-04-09T18:30:36.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f7717574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f77180cffb0 0x7f7718087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f7717574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f77180cffb0 0x7f7718087e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56020/0 (socket says 172.21.15.32:56020) 2021-04-09T18:30:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f7717574700 1 -- 172.21.15.32:0/2405626248 learned_addr learned my addr 172.21.15.32:0/2405626248 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f7716d73700 1 --2- 172.21.15.32:0/2405626248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77180d7c50 0x7f77181b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f7717574700 1 -- 172.21.15.32:0/2405626248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77180d7c50 msgr2=0x7f77181b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f7717574700 1 --2- 172.21.15.32:0/2405626248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77180d7c50 0x7f77181b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f7717574700 1 -- 172.21.15.32:0/2405626248 >> v1:172.21.15.86:6789/0 conn(0x7f77180d8050 legacy=0x7f771810d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.152 7f7717574700 1 -- 172.21.15.32:0/2405626248 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77180d2800 con 0x7f77180cffb0 2021-04-09T18:30:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.153 7f7717574700 1 --2- 172.21.15.32:0/2405626248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180cffb0 0x7f7718087e40 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f7708006700 tx=0x7f7708009b80).ready entity=mon.0 client_cookie=e0853d7294f91309 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.153 7f7716572700 1 -- 172.21.15.32:0/2405626248 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f770800c0c0 con 0x7f77180cffb0 2021-04-09T18:30:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.153 7f7716572700 1 -- 172.21.15.32:0/2405626248 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770800c440 con 0x7f77180cffb0 2021-04-09T18:30:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.153 7f7716572700 1 -- 172.21.15.32:0/2405626248 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f770800c920 con 0x7f77180cffb0 2021-04-09T18:30:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.153 7f771d85d700 1 -- 172.21.15.32:0/2405626248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180cffb0 msgr2=0x7f7718087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.153 7f771d85d700 1 --2- 172.21.15.32:0/2405626248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180cffb0 0x7f7718087e40 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f7708006700 tx=0x7f7708009b80).stop 2021-04-09T18:30:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.153 7f771d85d700 1 -- 172.21.15.32:0/2405626248 shutdown_connections 2021-04-09T18:30:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.153 7f771d85d700 1 --2- 172.21.15.32:0/2405626248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77180d7c50 0x7f77181b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.153 7f771d85d700 1 --2- 172.21.15.32:0/2405626248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180cffb0 0x7f7718087e40 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.154 7f771d85d700 1 -- 172.21.15.32:0/2405626248 >> 172.21.15.32:0/2405626248 conn(0x7f77180ce5b0 msgr2=0x7f77180cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.154 7f771d85d700 1 -- 172.21.15.32:0/2405626248 shutdown_connections 2021-04-09T18:30:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.154 7f771d85d700 1 -- 172.21.15.32:0/2405626248 wait complete. 2021-04-09T18:30:36.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.174 7f771d85d700 1 Processor -- start 2021-04-09T18:30:36.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.174 7f771d85d700 1 -- start start 2021-04-09T18:30:36.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.174 7f771d85d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7718099d00 0x7f7718182fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.174 7f771d85d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180a6c10 0x7f77180a7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.174 7f771d85d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7718191ae0 0x7f7718191ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.174 7f771d85d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77180a6690 con 0x7f77180a6c10 2021-04-09T18:30:36.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.174 7f771d85d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77180bf0e0 con 0x7f7718191ae0 2021-04-09T18:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.174 7f771d85d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77180a7630 con 0x7f7718099d00 2021-04-09T18:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7716d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180a6c10 0x7f77180a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7716d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180a6c10 0x7f77180a7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56028/0 (socket says 172.21.15.32:56028) 2021-04-09T18:30:36.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7716d73700 1 -- 172.21.15.32:0/2569589708 learned_addr learned my addr 172.21.15.32:0/2569589708 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:36.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7717574700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7718099d00 0x7f7718182fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7717d75700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7718191ae0 0x7f7718191ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7716d73700 1 -- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7718099d00 msgr2=0x7f7718182fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7716d73700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7718099d00 0x7f7718182fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7716d73700 1 -- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7718191ae0 msgr2=0x7f7718191ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7716d73700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7718191ae0 0x7f7718191ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7716d73700 1 -- 172.21.15.32:0/2569589708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7718192500 con 0x7f77180a6c10 2021-04-09T18:30:36.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7717574700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7718099d00 0x7f7718182fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:36.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f7716d73700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180a6c10 0x7f77180a7010 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7f7704006f60 tx=0x7f770400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:36.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f770ffff700 1 -- 172.21.15.32:0/2569589708 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f770400bd60 con 0x7f77180a6c10 2021-04-09T18:30:36.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f770ffff700 1 -- 172.21.15.32:0/2569589708 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7704000c90 con 0x7f77180a6c10 2021-04-09T18:30:36.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f770ffff700 1 -- 172.21.15.32:0/2569589708 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7704002220 con 0x7f77180a6c10 2021-04-09T18:30:36.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f771d85d700 1 -- 172.21.15.32:0/2569589708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7718192500 con 0x7f77180a6c10 2021-04-09T18:30:36.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.175 7f771d85d700 1 -- 172.21.15.32:0/2569589708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7718192500 con 0x7f77180a6c10 2021-04-09T18:30:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.176 7f770ffff700 1 -- 172.21.15.32:0/2569589708 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f77040052e0 con 0x7f77180a6c10 2021-04-09T18:30:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.176 7f770ffff700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76f4018930 0x7f76f401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.176 7f770ffff700 1 -- 172.21.15.32:0/2569589708 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f7704032da0 con 0x7f77180a6c10 2021-04-09T18:30:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.176 7f7717574700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76f4018930 0x7f76f401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.176 7f7717574700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76f4018930 0x7f76f401ae10 crc :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:36.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.196 7f771d85d700 1 -- 172.21.15.32:0/2569589708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77181835d0 con 0x7f77180a6c10 2021-04-09T18:30:36.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.200 7f770ffff700 1 -- 172.21.15.32:0/2569589708 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7704006bb0 con 0x7f77180a6c10 2021-04-09T18:30:36.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.387 7f771d85d700 1 -- 172.21.15.32:0/2569589708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]} v 0) v1 -- 0x7f77181835d0 con 0x7f77180a6c10 2021-04-09T18:30:36.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.389 7f770ffff700 1 -- 172.21.15.32:0/2569589708 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]}]=0 updated caps for client.1 v41) v1 ==== 209+0+0 (secure 0 0 0) 0x7f7704000c90 con 0x7f77180a6c10 2021-04-09T18:30:36.405 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:30:36.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 -- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76f4018930 msgr2=0x7f76f401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76f4018930 0x7f76f401ae10 crc :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 -- 172.21.15.32:0/2569589708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180a6c10 msgr2=0x7f77180a7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180a6c10 0x7f77180a7010 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7f7704006f60 tx=0x7f770400b970).stop 2021-04-09T18:30:36.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 -- 172.21.15.32:0/2569589708 shutdown_connections 2021-04-09T18:30:36.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76f4018930 0x7f76f401ae10 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77180a6c10 0x7f77180a7010 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7718191ae0 0x7f7718191ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 --2- 172.21.15.32:0/2569589708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7718099d00 0x7f7718182fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 -- 172.21.15.32:0/2569589708 >> 172.21.15.32:0/2569589708 conn(0x7f77180adaf0 msgr2=0x7f77180bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:36.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 -- 172.21.15.32:0/2569589708 shutdown_connections 2021-04-09T18:30:36.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.392 7f771d85d700 1 -- 172.21.15.32:0/2569589708 wait complete. 2021-04-09T18:30:36.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:36.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.551 7fe91ee60700 1 Processor -- start 2021-04-09T18:30:36.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.551 7fe91ee60700 1 -- start start 2021-04-09T18:30:36.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91ee60700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe91810d010 0x7fe9180d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91ee60700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe9180d8490 0x7fe91810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91ee60700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91810d480 con 0x7fe9180d1da0 2021-04-09T18:30:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91ee60700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe91810e780 con 0x7fe9180d8490 2021-04-09T18:30:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91ee60700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9180d8d30 con 0x7fe91810d010 2021-04-09T18:30:36.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91d47d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe9180d8490 0x7fe91810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe917fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe9180d1da0 0x7fe9180d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37392/0 (socket says 172.21.15.32:37392) 2021-04-09T18:30:36.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe917fff700 1 -- 172.21.15.32:0/2067147957 learned_addr learned my addr 172.21.15.32:0/2067147957 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:36.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91cc7c700 1 --2- 172.21.15.32:0/2067147957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe91810d010 0x7fe9180d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91d47d700 1 -- 172.21.15.32:0/2067147957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe91810d010 msgr2=0x7fe9180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:36.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91d47d700 1 --2- 172.21.15.32:0/2067147957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe91810d010 0x7fe9180d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91d47d700 1 -- 172.21.15.32:0/2067147957 >> v1:172.21.15.32:6789/0 conn(0x7fe9180d1da0 legacy=0x7fe9180d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91d47d700 1 -- 172.21.15.32:0/2067147957 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9180d9220 con 0x7fe9180d8490 2021-04-09T18:30:36.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe9177fe700 1 -- 172.21.15.32:0/2067147957 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335541213 0 0) 0x7fe904000ce0 con 0x7fe9180d1da0 2021-04-09T18:30:36.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe91d47d700 1 --2- 172.21.15.32:0/2067147957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9180d8490 0x7fe91810da00 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7fe910006e30 tx=0x7fe910009de0).ready entity=mon.0 client_cookie=a518519faaceed24 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:36.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe9177fe700 1 -- 172.21.15.32:0/2067147957 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe91000b300 con 0x7fe9180d8490 2021-04-09T18:30:36.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe9177fe700 1 -- 172.21.15.32:0/2067147957 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91000b680 con 0x7fe9180d8490 2021-04-09T18:30:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.552 7fe9177fe700 1 -- 172.21.15.32:0/2067147957 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe91000bb60 con 0x7fe9180d8490 2021-04-09T18:30:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.553 7fe91ee60700 1 -- 172.21.15.32:0/2067147957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9180d8490 msgr2=0x7fe91810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.553 7fe91ee60700 1 --2- 172.21.15.32:0/2067147957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9180d8490 0x7fe91810da00 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7fe910006e30 tx=0x7fe910009de0).stop 2021-04-09T18:30:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.553 7fe91ee60700 1 -- 172.21.15.32:0/2067147957 shutdown_connections 2021-04-09T18:30:36.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.553 7fe91ee60700 1 --2- 172.21.15.32:0/2067147957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9180d8490 0x7fe91810da00 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.553 7fe91ee60700 1 --2- 172.21.15.32:0/2067147957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe91810d010 0x7fe9180d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.553 7fe91ee60700 1 -- 172.21.15.32:0/2067147957 >> 172.21.15.32:0/2067147957 conn(0x7fe9180ce5f0 msgr2=0x7fe9180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.553 7fe91ee60700 1 -- 172.21.15.32:0/2067147957 shutdown_connections 2021-04-09T18:30:36.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.553 7fe91ee60700 1 -- 172.21.15.32:0/2067147957 wait complete. 2021-04-09T18:30:36.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91ee60700 1 Processor -- start 2021-04-09T18:30:36.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91ee60700 1 -- start start 2021-04-09T18:30:36.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91ee60700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe918185060 0x7fe918185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91ee60700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe918186c00 0x7fe918187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91ee60700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe918191360 0x7fe918191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91ee60700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9181863a0 con 0x7fe918185060 2021-04-09T18:30:36.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91ee60700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe918185a60 con 0x7fe918186c00 2021-04-09T18:30:36.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91ee60700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe918187620 con 0x7fe918191360 2021-04-09T18:30:36.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91cc7c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe918185060 0x7fe918185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91cc7c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe918185060 0x7fe918185d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56038/0 (socket says 172.21.15.32:56038) 2021-04-09T18:30:36.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.574 7fe91cc7c700 1 -- 172.21.15.32:0/3248412535 learned_addr learned my addr 172.21.15.32:0/3248412535 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:36.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe917fff700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe918186c00 0x7fe918187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe91d47d700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe918191360 0x7fe918191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe91cc7c700 1 -- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe918191360 msgr2=0x7fe918191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe91cc7c700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe918191360 0x7fe918191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe91cc7c700 1 -- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe918186c00 msgr2=0x7fe918187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe91cc7c700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe918186c00 0x7fe918187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe91cc7c700 1 -- 172.21.15.32:0/3248412535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918185460 con 0x7fe918185060 2021-04-09T18:30:36.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe917fff700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe918186c00 0x7fe918187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:36.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe91cc7c700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe918185060 0x7fe918185d80 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7fe90c00b580 tx=0x7fe90c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:36.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe9157fa700 1 -- 172.21.15.32:0/3248412535 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe90c0064d0 con 0x7fe918185060 2021-04-09T18:30:36.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe9157fa700 1 -- 172.21.15.32:0/3248412535 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c00bd10 con 0x7fe918185060 2021-04-09T18:30:36.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe9157fa700 1 -- 172.21.15.32:0/3248412535 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe90c002030 con 0x7fe918185060 2021-04-09T18:30:36.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe918185460 con 0x7fe918185060 2021-04-09T18:30:36.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.575 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe918185460 con 0x7fe918185060 2021-04-09T18:30:36.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.576 7fe9157fa700 1 -- 172.21.15.32:0/3248412535 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe90c006c90 con 0x7fe918185060 2021-04-09T18:30:36.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.576 7fe9157fa700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8f8018930 0x7fe8f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.576 7fe9157fa700 1 -- 172.21.15.32:0/3248412535 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fe90c032db0 con 0x7fe918185060 2021-04-09T18:30:36.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.576 7fe917fff700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8f8018930 0x7fe8f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:36.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.576 7fe917fff700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8f8018930 0x7fe8f801ae10 crc :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:36.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.596 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9181866d0 con 0x7fe918185060 2021-04-09T18:30:36.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.599 7fe9157fa700 1 -- 172.21.15.32:0/3248412535 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe90c006820 con 0x7fe918185060 2021-04-09T18:30:36.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.787 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9181866d0 con 0x7fe918185060 2021-04-09T18:30:36.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.788 7fe9157fa700 1 -- 172.21.15.32:0/3248412535 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+17091 (secure 0 0 0) 0x7fe90c006c90 con 0x7fe918185060 2021-04-09T18:30:36.804 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:36.805 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":188,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:30.280596","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:30:13.709244","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":31,"pool_name":"alpha_data","create_time":"2021-04-09 18:30:14.523869","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":32,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:30:21.008022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":33,"pool_name":"bravo_data","create_time":"2021-04-09 18:30:22.220789","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.32:6835/3158551863":"2021-04-10 18:30:30.264043","172.21.15.32:6833/3158551863":"2021-04-10 18:30:30.264043","172.21.15.32:6834/3729993394":"2021-04-10 18:30:30.261199","172.21.15.32:6832/3729993394":"2021-04-10 18:30:30.261199"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:36.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8f8018930 msgr2=0x7fe8f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8f8018930 0x7fe8f801ae10 crc :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe918185060 msgr2=0x7fe918185d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:36.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe918185060 0x7fe918185d80 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7fe90c00b580 tx=0x7fe90c00b920).stop 2021-04-09T18:30:36.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 shutdown_connections 2021-04-09T18:30:36.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8f8018930 0x7fe8f801ae10 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe918191360 0x7fe918191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe918186c00 0x7fe918187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 --2- 172.21.15.32:0/3248412535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe918185060 0x7fe918185d80 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:36.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 >> 172.21.15.32:0/3248412535 conn(0x7fe9181837d0 msgr2=0x7fe918195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:36.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 shutdown_connections 2021-04-09T18:30:36.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.790 7fe91ee60700 1 -- 172.21.15.32:0/3248412535 wait complete. 2021-04-09T18:30:36.816 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 30, 'pool_name': 'alpha_metadata', 'create_time': '2021-04-09 18:30:13.709244', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'alpha'}}}, {'pool': 31, 'pool_name': 'alpha_data', 'create_time': '2021-04-09 18:30:14.523869', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'alpha'}}}, {'pool': 32, 'pool_name': 'bravo_metadata', 'create_time': '2021-04-09 18:30:21.008022', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '183', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bravo'}}}, {'pool': 33, 'pool_name': 'bravo_data', 'create_time': '2021-04-09 18:30:22.220789', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bravo'}}}] 2021-04-09T18:30:36.817 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:30:36.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.955 7fe1510ab700 1 Processor -- start 2021-04-09T18:30:36.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.955 7fe1510ab700 1 -- start start 2021-04-09T18:30:36.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.955 7fe1510ab700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c10d010 0x7fe14c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.955 7fe1510ab700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe14c0d1da0 0x7fe14c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.955 7fe1510ab700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe14c0d82c0 0x7fe14c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:36.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.955 7fe1510ab700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe14c10e780 con 0x7fe14c0d1da0 2021-04-09T18:30:37.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.955 7fe1510ab700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe14c0d8d30 con 0x7fe14c0d82c0 2021-04-09T18:30:37.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.955 7fe1510ab700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c0d9220 con 0x7fe14c10d010 2021-04-09T18:30:37.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14a59c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe14c0d1da0 0x7fe14c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14a59c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe14c0d1da0 0x7fe14c0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56048/0 (socket says 172.21.15.32:56048) 2021-04-09T18:30:37.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14a59c700 1 -- 172.21.15.32:0/809307979 learned_addr learned my addr 172.21.15.32:0/809307979 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:37.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14ad9d700 1 --2- 172.21.15.32:0/809307979 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c10d010 0x7fe14c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14b59e700 1 --2- 172.21.15.32:0/809307979 >> v2:172.21.15.86:3300/0 conn(0x7fe14c0d82c0 0x7fe14c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14a59c700 1 -- 172.21.15.32:0/809307979 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c10d010 msgr2=0x7fe14c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:37.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14a59c700 1 --2- 172.21.15.32:0/809307979 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c10d010 0x7fe14c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14a59c700 1 -- 172.21.15.32:0/809307979 >> v2:172.21.15.86:3300/0 conn(0x7fe14c0d82c0 msgr2=0x7fe14c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:37.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14a59c700 1 --2- 172.21.15.32:0/809307979 >> v2:172.21.15.86:3300/0 conn(0x7fe14c0d82c0 0x7fe14c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14a59c700 1 -- 172.21.15.32:0/809307979 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c10d480 con 0x7fe14c0d1da0 2021-04-09T18:30:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe14a59c700 1 --2- 172.21.15.32:0/809307979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0d1da0 0x7fe14c0d7ca0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 rx=0x7fe140006700 tx=0x7fe140009b00).ready entity=mon.0 client_cookie=98fac9758d97c4ed server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe149d9b700 1 -- 172.21.15.32:0/809307979 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe14000c0c0 con 0x7fe14c0d1da0 2021-04-09T18:30:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe149d9b700 1 -- 172.21.15.32:0/809307979 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14000c400 con 0x7fe14c0d1da0 2021-04-09T18:30:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe149d9b700 1 -- 172.21.15.32:0/809307979 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe14000c8e0 con 0x7fe14c0d1da0 2021-04-09T18:30:37.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe1510ab700 1 -- 172.21.15.32:0/809307979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0d1da0 msgr2=0x7fe14c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.956 7fe1510ab700 1 --2- 172.21.15.32:0/809307979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0d1da0 0x7fe14c0d7ca0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 rx=0x7fe140006700 tx=0x7fe140009b00).stop 2021-04-09T18:30:37.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.957 7fe1510ab700 1 -- 172.21.15.32:0/809307979 shutdown_connections 2021-04-09T18:30:37.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.957 7fe1510ab700 1 --2- 172.21.15.32:0/809307979 >> v2:172.21.15.86:3300/0 conn(0x7fe14c0d82c0 0x7fe14c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.957 7fe1510ab700 1 --2- 172.21.15.32:0/809307979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0d1da0 0x7fe14c0d7ca0 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.957 7fe1510ab700 1 --2- 172.21.15.32:0/809307979 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c10d010 0x7fe14c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.957 7fe1510ab700 1 -- 172.21.15.32:0/809307979 >> 172.21.15.32:0/809307979 conn(0x7fe14c0ce5f0 msgr2=0x7fe14c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:37.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.957 7fe1510ab700 1 -- 172.21.15.32:0/809307979 shutdown_connections 2021-04-09T18:30:37.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.957 7fe1510ab700 1 -- 172.21.15.32:0/809307979 wait complete. 2021-04-09T18:30:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.977 7fe1510ab700 1 Processor -- start 2021-04-09T18:30:37.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe1510ab700 1 -- start start 2021-04-09T18:30:37.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe1510ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0c3c00 0x7fe14c09c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe1510ab700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c095240 0x7fe14c1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe1510ab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe14c1a1270 0x7fe14c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe1510ab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c09cda0 con 0x7fe14c0c3c00 2021-04-09T18:30:37.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe1510ab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c0c3680 con 0x7fe14c1a1270 2021-04-09T18:30:37.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe1510ab700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c095640 con 0x7fe14c095240 2021-04-09T18:30:37.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe14ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0c3c00 0x7fe14c09c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe14ad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0c3c00 0x7fe14c09c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56054/0 (socket says 172.21.15.32:56054) 2021-04-09T18:30:37.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe14ad9d700 1 -- 172.21.15.32:0/916671189 learned_addr learned my addr 172.21.15.32:0/916671189 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:37.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe14a59c700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c095240 0x7fe14c1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.978 7fe14b59e700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe14c1a1270 0x7fe14c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe14ad9d700 1 -- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c095240 msgr2=0x7fe14c1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe14ad9d700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c095240 0x7fe14c1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe14ad9d700 1 -- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe14c1a1270 msgr2=0x7fe14c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe14ad9d700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe14c1a1270 0x7fe14c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe14ad9d700 1 -- 172.21.15.32:0/916671189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c1a1670 con 0x7fe14c0c3c00 2021-04-09T18:30:37.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe14a59c700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c095240 0x7fe14c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:37.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe14ad9d700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0c3c00 0x7fe14c09c600 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7fe13c006be0 tx=0x7fe13c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:37.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe13b7fe700 1 -- 172.21.15.32:0/916671189 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe13c0064d0 con 0x7fe14c0c3c00 2021-04-09T18:30:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe13b7fe700 1 -- 172.21.15.32:0/916671189 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13c00bd80 con 0x7fe14c0c3c00 2021-04-09T18:30:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe1510ab700 1 -- 172.21.15.32:0/916671189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14c1a1670 con 0x7fe14c0c3c00 2021-04-09T18:30:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe13b7fe700 1 -- 172.21.15.32:0/916671189 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe13c002030 con 0x7fe14c0c3c00 2021-04-09T18:30:37.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.979 7fe1510ab700 1 -- 172.21.15.32:0/916671189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14c1a1670 con 0x7fe14c0c3c00 2021-04-09T18:30:37.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.980 7fe13b7fe700 1 -- 172.21.15.32:0/916671189 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe13c006c90 con 0x7fe14c0c3c00 2021-04-09T18:30:37.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.980 7fe13b7fe700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe130018930 0x7fe13001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.980 7fe13b7fe700 1 -- 172.21.15.32:0/916671189 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fe13c032df0 con 0x7fe14c0c3c00 2021-04-09T18:30:37.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.980 7fe14a59c700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe130018930 0x7fe13001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.980 7fe14a59c700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe130018930 0x7fe13001ae10 crc :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:36.999 7fe1510ab700 1 -- 172.21.15.32:0/916671189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14c093d50 con 0x7fe14c0c3c00 2021-04-09T18:30:37.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.003 7fe13b7fe700 1 -- 172.21.15.32:0/916671189 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe13c006820 con 0x7fe14c0c3c00 2021-04-09T18:30:37.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.182 7fe1510ab700 1 -- 172.21.15.32:0/916671189 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe14c10d480 con 0x7fe14c0c3c00 2021-04-09T18:30:37.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.182 7fe13b7fe700 1 -- 172.21.15.32:0/916671189 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v188) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe13c00bd80 con 0x7fe14c0c3c00 2021-04-09T18:30:37.198 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:37.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 -- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe130018930 msgr2=0x7fe13001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe130018930 0x7fe13001ae10 crc :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 -- 172.21.15.32:0/916671189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0c3c00 msgr2=0x7fe14c09c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0c3c00 0x7fe14c09c600 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7fe13c006be0 tx=0x7fe13c00b990).stop 2021-04-09T18:30:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 -- 172.21.15.32:0/916671189 shutdown_connections 2021-04-09T18:30:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe130018930 0x7fe13001ae10 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe14c1a1270 0x7fe14c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe14c095240 0x7fe14c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 --2- 172.21.15.32:0/916671189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe14c0c3c00 0x7fe14c09c600 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 -- 172.21.15.32:0/916671189 >> 172.21.15.32:0/916671189 conn(0x7fe14c0975f0 msgr2=0x7fe14c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:37.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 -- 172.21.15.32:0/916671189 shutdown_connections 2021-04-09T18:30:37.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.185 7fe1510ab700 1 -- 172.21.15.32:0/916671189 wait complete. 2021-04-09T18:30:37.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_metadata pg_num 2021-04-09T18:30:37.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.348 7f57f32cd700 1 Processor -- start 2021-04-09T18:30:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.348 7f57f32cd700 1 -- start start 2021-04-09T18:30:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.348 7f57f32cd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f57ec10d010 0x7f57ec0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.348 7f57f32cd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ec10d480 con 0x7f57ec0d1da0 2021-04-09T18:30:37.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.348 7f57f32cd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ec10e1f0 con 0x7f57ec0d8490 2021-04-09T18:30:37.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.348 7f57f32cd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec0d8d30 con 0x7f57ec10d010 2021-04-09T18:30:37.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57f08e8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f57ec0d1da0 0x7f57ec0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37418/0 (socket says 172.21.15.32:37418) 2021-04-09T18:30:37.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57f08e8700 1 -- 172.21.15.32:0/4267953188 learned_addr learned my addr 172.21.15.32:0/4267953188 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57f10e9700 1 --2- 172.21.15.32:0/4267953188 >> v2:172.21.15.86:3300/0 conn(0x7f57ec10d010 0x7f57ec0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57e3fff700 1 -- 172.21.15.32:0/4267953188 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649130090 0 0) 0x7f57e8001060 con 0x7f57ec0d1da0 2021-04-09T18:30:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57e3fff700 1 -- 172.21.15.32:0/4267953188 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d8002fe0 con 0x7f57ec0d1da0 2021-04-09T18:30:37.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57e3fff700 1 -- 172.21.15.32:0/4267953188 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253576101 0 0) 0x7f57e40009b0 con 0x7f57ec0d8490 2021-04-09T18:30:37.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57e3fff700 1 -- 172.21.15.32:0/4267953188 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d80042c0 con 0x7f57ec0d8490 2021-04-09T18:30:37.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57f10e9700 1 -- 172.21.15.32:0/4267953188 >> v1:172.21.15.86:6789/0 conn(0x7f57ec0d8490 legacy=0x7f57ec10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57f10e9700 1 -- 172.21.15.32:0/4267953188 >> v1:172.21.15.32:6789/0 conn(0x7f57ec0d1da0 legacy=0x7f57ec0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57f10e9700 1 -- 172.21.15.32:0/4267953188 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec0d9220 con 0x7f57ec10d010 2021-04-09T18:30:37.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57f10e9700 1 --2- 172.21.15.32:0/4267953188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec10d010 0x7f57ec0d1780 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f57dc006650 tx=0x7f57dc009a80).ready entity=mon.1 client_cookie=56de208846dca0a9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:37.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57e3fff700 1 -- 172.21.15.32:0/4267953188 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57dc0009b0 con 0x7f57ec10d010 2021-04-09T18:30:37.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57e3fff700 1 -- 172.21.15.32:0/4267953188 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc00c0c0 con 0x7f57ec10d010 2021-04-09T18:30:37.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57e3fff700 1 -- 172.21.15.32:0/4267953188 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57dc00c510 con 0x7f57ec10d010 2021-04-09T18:30:37.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57f32cd700 1 -- 172.21.15.32:0/4267953188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec10d010 msgr2=0x7f57ec0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.349 7f57f32cd700 1 --2- 172.21.15.32:0/4267953188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec10d010 0x7f57ec0d1780 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f57dc006650 tx=0x7f57dc009a80).stop 2021-04-09T18:30:37.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.350 7f57f32cd700 1 -- 172.21.15.32:0/4267953188 shutdown_connections 2021-04-09T18:30:37.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.350 7f57f32cd700 1 --2- 172.21.15.32:0/4267953188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec10d010 0x7f57ec0d1780 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.350 7f57f32cd700 1 -- 172.21.15.32:0/4267953188 >> 172.21.15.32:0/4267953188 conn(0x7f57ec0ce5f0 msgr2=0x7f57ec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:37.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.350 7f57f32cd700 1 -- 172.21.15.32:0/4267953188 shutdown_connections 2021-04-09T18:30:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.350 7f57f32cd700 1 -- 172.21.15.32:0/4267953188 wait complete. 2021-04-09T18:30:37.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.371 7f57f32cd700 1 Processor -- start 2021-04-09T18:30:37.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.371 7f57f32cd700 1 -- start start 2021-04-09T18:30:37.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.371 7f57f32cd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57ec191ee0 0x7f57ec0a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.371 7f57f32cd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec0a6c30 0x7f57ec0b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.371 7f57f32cd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec0b4330 0x7f57ec1b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.371 7f57f32cd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec0b3790 con 0x7f57ec0a6c30 2021-04-09T18:30:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.371 7f57f32cd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec0a7030 con 0x7f57ec0b4330 2021-04-09T18:30:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.371 7f57f32cd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec0b4730 con 0x7f57ec191ee0 2021-04-09T18:30:37.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f08e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec0a6c30 0x7f57ec0b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f08e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec0a6c30 0x7f57ec0b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56064/0 (socket says 172.21.15.32:56064) 2021-04-09T18:30:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f08e8700 1 -- 172.21.15.32:0/950952929 learned_addr learned my addr 172.21.15.32:0/950952929 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f10e9700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57ec191ee0 0x7f57ec0a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f18ea700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec0b4330 0x7f57ec1b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f08e8700 1 -- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57ec191ee0 msgr2=0x7f57ec0a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f08e8700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57ec191ee0 0x7f57ec0a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f08e8700 1 -- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec0b4330 msgr2=0x7f57ec1b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f08e8700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec0b4330 0x7f57ec1b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f08e8700 1 -- 172.21.15.32:0/950952929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec1b8580 con 0x7f57ec0a6c30 2021-04-09T18:30:37.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f10e9700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57ec191ee0 0x7f57ec0a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f18ea700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec0b4330 0x7f57ec1b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f08e8700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec0a6c30 0x7f57ec0b3d10 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7f57e8009980 tx=0x7f57e800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57e1ffb700 1 -- 172.21.15.32:0/950952929 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57e8006080 con 0x7f57ec0a6c30 2021-04-09T18:30:37.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57e1ffb700 1 -- 172.21.15.32:0/950952929 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e8006380 con 0x7f57ec0a6c30 2021-04-09T18:30:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f32cd700 1 -- 172.21.15.32:0/950952929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ec1b8580 con 0x7f57ec0a6c30 2021-04-09T18:30:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57e1ffb700 1 -- 172.21.15.32:0/950952929 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57e80049a0 con 0x7f57ec0a6c30 2021-04-09T18:30:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.372 7f57f32cd700 1 -- 172.21.15.32:0/950952929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ec1b8580 con 0x7f57ec0a6c30 2021-04-09T18:30:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.373 7f57e1ffb700 1 -- 172.21.15.32:0/950952929 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f57e8006560 con 0x7f57ec0a6c30 2021-04-09T18:30:37.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.373 7f57e1ffb700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57cc018930 0x7f57cc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.373 7f57e1ffb700 1 -- 172.21.15.32:0/950952929 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f57e8033190 con 0x7f57ec0a6c30 2021-04-09T18:30:37.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.373 7f57f10e9700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57cc018930 0x7f57cc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.373 7f57f10e9700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57cc018930 0x7f57cc01ae10 crc :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:37.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.392 7f57f32cd700 1 -- 172.21.15.32:0/950952929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57ec10d480 con 0x7f57ec0a6c30 2021-04-09T18:30:37.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.396 7f57e1ffb700 1 -- 172.21.15.32:0/950952929 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f57e8000d30 con 0x7f57ec0a6c30 2021-04-09T18:30:37.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.576 7f57f32cd700 1 -- 172.21.15.32:0/950952929 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"} v 0) v1 -- 0x7f57ec10d480 con 0x7f57ec0a6c30 2021-04-09T18:30:37.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.577 7f57e1ffb700 1 -- 172.21.15.32:0/950952929 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"}]=0 v188) v1 ==== 103+0+10 (secure 0 0 0) 0x7f57e8006380 con 0x7f57ec0a6c30 2021-04-09T18:30:37.593 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:37.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 -- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57cc018930 msgr2=0x7f57cc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57cc018930 0x7f57cc01ae10 crc :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 -- 172.21.15.32:0/950952929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec0a6c30 msgr2=0x7f57ec0b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec0a6c30 0x7f57ec0b3d10 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7f57e8009980 tx=0x7f57e800bcd0).stop 2021-04-09T18:30:37.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 -- 172.21.15.32:0/950952929 shutdown_connections 2021-04-09T18:30:37.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57cc018930 0x7f57cc01ae10 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ec0a6c30 0x7f57ec0b3d10 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57ec0b4330 0x7f57ec1b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 --2- 172.21.15.32:0/950952929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57ec191ee0 0x7f57ec0a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 -- 172.21.15.32:0/950952929 >> 172.21.15.32:0/950952929 conn(0x7f57ec0bbda0 msgr2=0x7f57ec1b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 -- 172.21.15.32:0/950952929 shutdown_connections 2021-04-09T18:30:37.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.579 7f57f32cd700 1 -- 172.21.15.32:0/950952929 wait complete. 2021-04-09T18:30:37.605 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_data pg_num 2021-04-09T18:30:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.743 7fe453c81700 1 Processor -- start 2021-04-09T18:30:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.743 7fe453c81700 1 -- start start 2021-04-09T18:30:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.743 7fe453c81700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe44c10d010 0x7fe44c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.743 7fe453c81700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe44c0d8490 0x7fe44c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.743 7fe453c81700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44c10d480 con 0x7fe44c0d1da0 2021-04-09T18:30:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.743 7fe453c81700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe44c0d8d30 con 0x7fe44c10d010 2021-04-09T18:30:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.743 7fe453c81700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe44c0d9220 con 0x7fe44c0d8490 2021-04-09T18:30:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe451a9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe44c10d010 0x7fe44c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe45129c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe44c0d1da0 0x7fe44c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37434/0 (socket says 172.21.15.32:37434) 2021-04-09T18:30:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe45129c700 1 -- 172.21.15.32:0/247188313 learned_addr learned my addr 172.21.15.32:0/247188313 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe45229e700 1 --2- 172.21.15.32:0/247188313 >> v2:172.21.15.86:3300/0 conn(0x7fe44c0d8490 0x7fe44c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe450a9b700 1 -- 172.21.15.32:0/247188313 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808864669 0 0) 0x7fe440000ce0 con 0x7fe44c0d1da0 2021-04-09T18:30:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe450a9b700 1 -- 172.21.15.32:0/247188313 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe438002fe0 con 0x7fe44c0d1da0 2021-04-09T18:30:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe451a9d700 1 -- 172.21.15.32:0/247188313 >> v2:172.21.15.86:3300/0 conn(0x7fe44c0d8490 msgr2=0x7fe44c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe451a9d700 1 --2- 172.21.15.32:0/247188313 >> v2:172.21.15.86:3300/0 conn(0x7fe44c0d8490 0x7fe44c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe451a9d700 1 -- 172.21.15.32:0/247188313 >> v1:172.21.15.32:6789/0 conn(0x7fe44c0d1da0 legacy=0x7fe44c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe451a9d700 1 -- 172.21.15.32:0/247188313 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c10e020 con 0x7fe44c10d010 2021-04-09T18:30:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe451a9d700 1 --2- 172.21.15.32:0/247188313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c10d010 0x7fe44c0d1780 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7fe448006650 tx=0x7fe448009a80).ready entity=mon.0 client_cookie=e6bc930eb1c6ac81 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe450a9b700 1 -- 172.21.15.32:0/247188313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4480009b0 con 0x7fe44c10d010 2021-04-09T18:30:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe450a9b700 1 -- 172.21.15.32:0/247188313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44800c0c0 con 0x7fe44c10d010 2021-04-09T18:30:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe450a9b700 1 -- 172.21.15.32:0/247188313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe44800c510 con 0x7fe44c10d010 2021-04-09T18:30:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe453c81700 1 -- 172.21.15.32:0/247188313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c10d010 msgr2=0x7fe44c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.744 7fe453c81700 1 --2- 172.21.15.32:0/247188313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c10d010 0x7fe44c0d1780 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7fe448006650 tx=0x7fe448009a80).stop 2021-04-09T18:30:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.745 7fe453c81700 1 -- 172.21.15.32:0/247188313 shutdown_connections 2021-04-09T18:30:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.745 7fe453c81700 1 --2- 172.21.15.32:0/247188313 >> v2:172.21.15.86:3300/0 conn(0x7fe44c0d8490 0x7fe44c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.745 7fe453c81700 1 --2- 172.21.15.32:0/247188313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c10d010 0x7fe44c0d1780 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.745 7fe453c81700 1 -- 172.21.15.32:0/247188313 >> 172.21.15.32:0/247188313 conn(0x7fe44c0ce5f0 msgr2=0x7fe44c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.745 7fe453c81700 1 -- 172.21.15.32:0/247188313 shutdown_connections 2021-04-09T18:30:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.745 7fe453c81700 1 -- 172.21.15.32:0/247188313 wait complete. 2021-04-09T18:30:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.765 7fe453c81700 1 Processor -- start 2021-04-09T18:30:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.765 7fe453c81700 1 -- start start 2021-04-09T18:30:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe453c81700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe44c09e530 0x7fe44c09b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe453c81700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe44c09e930 0x7fe44c09f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe453c81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c09f990 0x7fe44c094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe453c81700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c09edf0 con 0x7fe44c09f990 2021-04-09T18:30:37.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe453c81700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c09fd90 con 0x7fe44c09e930 2021-04-09T18:30:37.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe453c81700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe44c09bd40 con 0x7fe44c09e530 2021-04-09T18:30:37.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45229e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c09f990 0x7fe44c094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45229e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c09f990 0x7fe44c094c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56078/0 (socket says 172.21.15.32:56078) 2021-04-09T18:30:37.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45229e700 1 -- 172.21.15.32:0/2810491766 learned_addr learned my addr 172.21.15.32:0/2810491766 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:37.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45129c700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe44c09e930 0x7fe44c09f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe451a9d700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe44c09e530 0x7fe44c09b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45229e700 1 -- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe44c09e530 msgr2=0x7fe44c09b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45229e700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe44c09e530 0x7fe44c09b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45229e700 1 -- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe44c09e930 msgr2=0x7fe44c09f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45229e700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe44c09e930 0x7fe44c09f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45229e700 1 -- 172.21.15.32:0/2810491766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c095290 con 0x7fe44c09f990 2021-04-09T18:30:37.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe451a9d700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe44c09e530 0x7fe44c09b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:37.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.766 7fe45229e700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c09f990 0x7fe44c094c70 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7fe44400b970 tx=0x7fe44400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:37.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.767 7fe43e7fc700 1 -- 172.21.15.32:0/2810491766 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe444000f90 con 0x7fe44c09f990 2021-04-09T18:30:37.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.767 7fe43e7fc700 1 -- 172.21.15.32:0/2810491766 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe444006aa0 con 0x7fe44c09f990 2021-04-09T18:30:37.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.767 7fe453c81700 1 -- 172.21.15.32:0/2810491766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44c095290 con 0x7fe44c09f990 2021-04-09T18:30:37.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.767 7fe43e7fc700 1 -- 172.21.15.32:0/2810491766 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4440015f0 con 0x7fe44c09f990 2021-04-09T18:30:37.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.767 7fe453c81700 1 -- 172.21.15.32:0/2810491766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44c095290 con 0x7fe44c09f990 2021-04-09T18:30:37.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.767 7fe43e7fc700 1 -- 172.21.15.32:0/2810491766 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe444000ed0 con 0x7fe44c09f990 2021-04-09T18:30:37.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.768 7fe43e7fc700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe434018930 0x7fe43401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:37.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.768 7fe43e7fc700 1 -- 172.21.15.32:0/2810491766 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fe444033300 con 0x7fe44c09f990 2021-04-09T18:30:37.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.768 7fe451a9d700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe434018930 0x7fe43401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:37.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.768 7fe451a9d700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe434018930 0x7fe43401ae10 crc :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:37.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.787 7fe453c81700 1 -- 172.21.15.32:0/2810491766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44c10d480 con 0x7fe44c09f990 2021-04-09T18:30:37.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.791 7fe43e7fc700 1 -- 172.21.15.32:0/2810491766 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe4440066f0 con 0x7fe44c09f990 2021-04-09T18:30:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.970 7fe453c81700 1 -- 172.21.15.32:0/2810491766 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"} v 0) v1 -- 0x7fe44c10d480 con 0x7fe44c09f990 2021-04-09T18:30:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.971 7fe43e7fc700 1 -- 172.21.15.32:0/2810491766 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"}]=0 v188) v1 ==== 99+0+10 (secure 0 0 0) 0x7fe444006aa0 con 0x7fe44c09f990 2021-04-09T18:30:37.987 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 -- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe434018930 msgr2=0x7fe43401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe434018930 0x7fe43401ae10 crc :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 -- 172.21.15.32:0/2810491766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c09f990 msgr2=0x7fe44c094c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c09f990 0x7fe44c094c70 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7fe44400b970 tx=0x7fe44400bd10).stop 2021-04-09T18:30:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 -- 172.21.15.32:0/2810491766 shutdown_connections 2021-04-09T18:30:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe434018930 0x7fe43401ae10 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe44c09f990 0x7fe44c094c70 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe44c09e930 0x7fe44c09f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 --2- 172.21.15.32:0/2810491766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe44c09e530 0x7fe44c09b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.973 7fe453c81700 1 -- 172.21.15.32:0/2810491766 >> 172.21.15.32:0/2810491766 conn(0x7fe44c0c4290 msgr2=0x7fe44c0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.974 7fe453c81700 1 -- 172.21.15.32:0/2810491766 shutdown_connections 2021-04-09T18:30:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:37.974 7fe453c81700 1 -- 172.21.15.32:0/2810491766 wait complete. 2021-04-09T18:30:38.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get bravo_metadata pg_num 2021-04-09T18:30:38.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:30:38.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:30:38.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.137 7f45bc065700 1 Processor -- start 2021-04-09T18:30:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.137 7f45bc065700 1 -- start start 2021-04-09T18:30:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.138 7f45bc065700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f45b410d010 0x7f45b40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.138 7f45bc065700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40d1da0 0x7f45b40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.138 7f45bc065700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b410d480 con 0x7f45b40d82c0 2021-04-09T18:30:38.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.138 7f45bc065700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b40d9220 con 0x7f45b410d010 2021-04-09T18:30:38.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.138 7f45bc065700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b40d86c0 con 0x7f45b40d1da0 2021-04-09T18:30:38.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.138 7f45b9e81700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f45b410d010 0x7f45b40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.138 7f45ba682700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f45b40d82c0 0x7f45b410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37448/0 (socket says 172.21.15.32:37448) 2021-04-09T18:30:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.138 7f45ba682700 1 -- 172.21.15.32:0/864841490 learned_addr learned my addr 172.21.15.32:0/864841490 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.138 7f45b9680700 1 --2- 172.21.15.32:0/864841490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40d1da0 0x7f45b40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45b9e81700 1 -- 172.21.15.32:0/864841490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40d1da0 msgr2=0x7f45b40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45b9e81700 1 --2- 172.21.15.32:0/864841490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40d1da0 0x7f45b40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45b9e81700 1 -- 172.21.15.32:0/864841490 >> v1:172.21.15.32:6789/0 conn(0x7f45b40d82c0 legacy=0x7f45b410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45b9e81700 1 -- 172.21.15.32:0/864841490 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b40d09c0 con 0x7f45b410d010 2021-04-09T18:30:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45b8e7f700 1 -- 172.21.15.32:0/864841490 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536518248 0 0) 0x7f45ac000d30 con 0x7f45b40d82c0 2021-04-09T18:30:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45b9e81700 1 --2- 172.21.15.32:0/864841490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b410d010 0x7f45b40d1780 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f45b0006650 tx=0x7f45b0009a00).ready entity=mon.0 client_cookie=93792caa22e449f4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:38.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45b8e7f700 1 -- 172.21.15.32:0/864841490 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45b0009df0 con 0x7f45b410d010 2021-04-09T18:30:38.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45b8e7f700 1 -- 172.21.15.32:0/864841490 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b00009b0 con 0x7f45b410d010 2021-04-09T18:30:38.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45b8e7f700 1 -- 172.21.15.32:0/864841490 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45b000c2b0 con 0x7f45b410d010 2021-04-09T18:30:38.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45bc065700 1 -- 172.21.15.32:0/864841490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b410d010 msgr2=0x7f45b40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.139 7f45bc065700 1 --2- 172.21.15.32:0/864841490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b410d010 0x7f45b40d1780 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f45b0006650 tx=0x7f45b0009a00).stop 2021-04-09T18:30:38.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.140 7f45bc065700 1 -- 172.21.15.32:0/864841490 shutdown_connections 2021-04-09T18:30:38.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.140 7f45bc065700 1 --2- 172.21.15.32:0/864841490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40d1da0 0x7f45b40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.140 7f45bc065700 1 --2- 172.21.15.32:0/864841490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b410d010 0x7f45b40d1780 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.140 7f45bc065700 1 -- 172.21.15.32:0/864841490 >> 172.21.15.32:0/864841490 conn(0x7f45b40ce5f0 msgr2=0x7f45b40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.140 7f45bc065700 1 -- 172.21.15.32:0/864841490 shutdown_connections 2021-04-09T18:30:38.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.140 7f45bc065700 1 -- 172.21.15.32:0/864841490 wait complete. 2021-04-09T18:30:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.160 7f45bc065700 1 Processor -- start 2021-04-09T18:30:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.160 7f45bc065700 1 -- start start 2021-04-09T18:30:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45bc065700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b41a0050 0x7f45b41a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45bc065700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b40bdcd0 0x7f45b40bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45bc065700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40bf080 0x7f45b40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45bc065700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b40bf500 con 0x7f45b40bdcd0 2021-04-09T18:30:38.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45bc065700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b40bb7b0 con 0x7f45b41a0050 2021-04-09T18:30:38.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45bc065700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b40be0d0 con 0x7f45b40bf080 2021-04-09T18:30:38.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9680700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b40bdcd0 0x7f45b40bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9680700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b40bdcd0 0x7f45b40bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56094/0 (socket says 172.21.15.32:56094) 2021-04-09T18:30:38.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9680700 1 -- 172.21.15.32:0/4029137646 learned_addr learned my addr 172.21.15.32:0/4029137646 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:38.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45ba682700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40bf080 0x7f45b40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9e81700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b41a0050 0x7f45b41a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9680700 1 -- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40bf080 msgr2=0x7f45b40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9680700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40bf080 0x7f45b40b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9680700 1 -- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b41a0050 msgr2=0x7f45b41a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9680700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b41a0050 0x7f45b41a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9680700 1 -- 172.21.15.32:0/4029137646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b40b1310 con 0x7f45b40bdcd0 2021-04-09T18:30:38.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9e81700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b41a0050 0x7f45b41a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:30:38.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45ba682700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40bf080 0x7f45b40b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:30:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.161 7f45b9680700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b40bdcd0 0x7f45b40bea60 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7f45a8006f00 tx=0x7f45a800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.162 7f45a67fc700 1 -- 172.21.15.32:0/4029137646 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45a800bd30 con 0x7f45b40bdcd0 2021-04-09T18:30:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.162 7f45a67fc700 1 -- 172.21.15.32:0/4029137646 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a800bd30 con 0x7f45b40bdcd0 2021-04-09T18:30:38.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.162 7f45bc065700 1 -- 172.21.15.32:0/4029137646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b40b1310 con 0x7f45b40bdcd0 2021-04-09T18:30:38.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.162 7f45a67fc700 1 -- 172.21.15.32:0/4029137646 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45a8002030 con 0x7f45b40bdcd0 2021-04-09T18:30:38.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.162 7f45bc065700 1 -- 172.21.15.32:0/4029137646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b40b1310 con 0x7f45b40bdcd0 2021-04-09T18:30:38.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.162 7f45a67fc700 1 -- 172.21.15.32:0/4029137646 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f45a8009fb0 con 0x7f45b40bdcd0 2021-04-09T18:30:38.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.162 7f45a67fc700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f459c018930 0x7f459c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.162 7f45a67fc700 1 -- 172.21.15.32:0/4029137646 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f45a8032e20 con 0x7f45b40bdcd0 2021-04-09T18:30:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.163 7f45b9e81700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f459c018930 0x7f459c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.163 7f45b9e81700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f459c018930 0x7f459c01ae10 crc :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:38.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.182 7f45bc065700 1 -- 172.21.15.32:0/4029137646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b4093d50 con 0x7f45b40bdcd0 2021-04-09T18:30:38.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.186 7f45a67fc700 1 -- 172.21.15.32:0/4029137646 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f45a8000940 con 0x7f45b40bdcd0 2021-04-09T18:30:38.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.364 7f45bc065700 1 -- 172.21.15.32:0/4029137646 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "bravo_metadata"} v 0) v1 -- 0x7f45b410d480 con 0x7f45b40bdcd0 2021-04-09T18:30:38.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.365 7f45a67fc700 1 -- 172.21.15.32:0/4029137646 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "bravo_metadata"}]=0 v188) v1 ==== 103+0+10 (secure 0 0 0) 0x7f45a8009fb0 con 0x7f45b40bdcd0 2021-04-09T18:30:38.381 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:38.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 -- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f459c018930 msgr2=0x7f459c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f459c018930 0x7f459c01ae10 crc :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 -- 172.21.15.32:0/4029137646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b40bdcd0 msgr2=0x7f45b40bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b40bdcd0 0x7f45b40bea60 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7f45a8006f00 tx=0x7f45a800b940).stop 2021-04-09T18:30:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 -- 172.21.15.32:0/4029137646 shutdown_connections 2021-04-09T18:30:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f459c018930 0x7f459c01ae10 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45b40bdcd0 0x7f45b40bea60 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b40bf080 0x7f45b40b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 --2- 172.21.15.32:0/4029137646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b41a0050 0x7f45b41a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.367 7f45bc065700 1 -- 172.21.15.32:0/4029137646 >> 172.21.15.32:0/4029137646 conn(0x7f45b40975f0 msgr2=0x7f45b4096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:38.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.368 7f45bc065700 1 -- 172.21.15.32:0/4029137646 shutdown_connections 2021-04-09T18:30:38.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.368 7f45bc065700 1 -- 172.21.15.32:0/4029137646 wait complete. 2021-04-09T18:30:38.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get bravo_data pg_num 2021-04-09T18:30:38.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.569 7f9e0462b700 1 Processor -- start 2021-04-09T18:30:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.569 7f9e0462b700 1 -- start start 2021-04-09T18:30:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e0462b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0cffc0 0x7f9dfc0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e0462b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9dfc0d7820 0x7f9dfc0d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e0462b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc0d72a0 con 0x7f9dfc0d5c90 2021-04-09T18:30:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e0462b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dfc0d85a0 con 0x7f9dfc0d7820 2021-04-09T18:30:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e0462b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc0d9200 con 0x7f9dfc0cffc0 2021-04-09T18:30:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e02c48700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9dfc0d5c90 0x7f9dfc0d6090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37460/0 (socket says 172.21.15.32:37460) 2021-04-09T18:30:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e02c48700 1 -- 172.21.15.32:0/2005563631 learned_addr learned my addr 172.21.15.32:0/2005563631 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:38.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e02447700 1 --2- 172.21.15.32:0/2005563631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0cffc0 0x7f9dfc0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e01c46700 1 --2- 172.21.15.32:0/2005563631 >> v2:172.21.15.86:3300/0 conn(0x7f9dfc0d7820 0x7f9dfc0d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e01445700 1 -- 172.21.15.32:0/2005563631 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165488474 0 0) 0x7f9df40009b0 con 0x7f9dfc0d5c90 2021-04-09T18:30:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.570 7f9e01445700 1 -- 172.21.15.32:0/2005563631 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de8002fe0 con 0x7f9dfc0d5c90 2021-04-09T18:30:38.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e02447700 1 -- 172.21.15.32:0/2005563631 >> v2:172.21.15.86:3300/0 conn(0x7f9dfc0d7820 msgr2=0x7f9dfc0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e02447700 1 --2- 172.21.15.32:0/2005563631 >> v2:172.21.15.86:3300/0 conn(0x7f9dfc0d7820 0x7f9dfc0d7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e02447700 1 -- 172.21.15.32:0/2005563631 >> v1:172.21.15.32:6789/0 conn(0x7f9dfc0d5c90 legacy=0x7f9dfc0d6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e01445700 1 -- 172.21.15.32:0/2005563631 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 465250267 0 0) 0x7f9df4000ef0 con 0x7f9dfc0d5c90 2021-04-09T18:30:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e02447700 1 -- 172.21.15.32:0/2005563631 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc0d8240 con 0x7f9dfc0cffc0 2021-04-09T18:30:38.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e01c46700 1 --2- 172.21.15.32:0/2005563631 >> v2:172.21.15.86:3300/0 conn(0x7f9dfc0d7820 0x7f9dfc0d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e02447700 1 --2- 172.21.15.32:0/2005563631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0cffc0 0x7f9dfc0d2460 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9df80069d0 tx=0x7f9df8009e40).ready entity=mon.2 client_cookie=164ddb8549d71d9f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e01445700 1 -- 172.21.15.32:0/2005563631 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df800b510 con 0x7f9dfc0cffc0 2021-04-09T18:30:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e01445700 1 -- 172.21.15.32:0/2005563631 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df800b890 con 0x7f9dfc0cffc0 2021-04-09T18:30:38.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e01445700 1 -- 172.21.15.32:0/2005563631 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df800bd70 con 0x7f9dfc0cffc0 2021-04-09T18:30:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e0462b700 1 -- 172.21.15.32:0/2005563631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0cffc0 msgr2=0x7f9dfc0d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.571 7f9e0462b700 1 --2- 172.21.15.32:0/2005563631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0cffc0 0x7f9dfc0d2460 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9df80069d0 tx=0x7f9df8009e40).stop 2021-04-09T18:30:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.572 7f9e0462b700 1 -- 172.21.15.32:0/2005563631 shutdown_connections 2021-04-09T18:30:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.572 7f9e0462b700 1 --2- 172.21.15.32:0/2005563631 >> v2:172.21.15.86:3300/0 conn(0x7f9dfc0d7820 0x7f9dfc0d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.572 7f9e0462b700 1 --2- 172.21.15.32:0/2005563631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0cffc0 0x7f9dfc0d2460 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.572 7f9e0462b700 1 -- 172.21.15.32:0/2005563631 >> 172.21.15.32:0/2005563631 conn(0x7f9dfc0ce5e0 msgr2=0x7f9dfc0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.572 7f9e0462b700 1 -- 172.21.15.32:0/2005563631 shutdown_connections 2021-04-09T18:30:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.572 7f9e0462b700 1 -- 172.21.15.32:0/2005563631 wait complete. 2021-04-09T18:30:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.592 7f9e0462b700 1 Processor -- start 2021-04-09T18:30:38.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.592 7f9e0462b700 1 -- start start 2021-04-09T18:30:38.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e0462b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0b89a0 0x7f9dfc0c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e0462b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dfc0b8da0 0x7f9dfc0be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e0462b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dfc0b91a0 0x7f9dfc0bead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e0462b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc0c39b0 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e0462b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc0c0d60 con 0x7f9dfc0b8da0 2021-04-09T18:30:38.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e0462b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc0bf0f0 con 0x7f9dfc0b89a0 2021-04-09T18:30:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02c48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dfc0b91a0 0x7f9dfc0bead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02c48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dfc0b91a0 0x7f9dfc0bead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56110/0 (socket says 172.21.15.32:56110) 2021-04-09T18:30:38.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02c48700 1 -- 172.21.15.32:0/2685032956 learned_addr learned my addr 172.21.15.32:0/2685032956 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e01c46700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dfc0b8da0 0x7f9dfc0be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02447700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0b89a0 0x7f9dfc0c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02c48700 1 -- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0b89a0 msgr2=0x7f9dfc0c31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02c48700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0b89a0 0x7f9dfc0c31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02c48700 1 -- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dfc0b8da0 msgr2=0x7f9dfc0be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02c48700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dfc0b8da0 0x7f9dfc0be4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02c48700 1 -- 172.21.15.32:0/2685032956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc0bf310 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e02c48700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dfc0b91a0 0x7f9dfc0bead0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7f9df4007fa0 tx=0x7f9df400a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:38.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9df2ffd700 1 -- 172.21.15.32:0/2685032956 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df4001200 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9df2ffd700 1 -- 172.21.15.32:0/2685032956 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df400ac90 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc0bf310 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9df2ffd700 1 -- 172.21.15.32:0/2685032956 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df40047a0 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.593 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc0bf310 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.594 7f9df2ffd700 1 -- 172.21.15.32:0/2685032956 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9df4007fe0 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.594 7f9df2ffd700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.594 7f9df2ffd700 1 -- 172.21.15.32:0/2685032956 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f9df4032f40 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.594 7f9e02447700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:38.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.595 7f9e02447700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 crc :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:38.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.614 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc1b70c0 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.618 7f9df2ffd700 1 -- 172.21.15.32:0/2685032956 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9df4001200 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.797 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "bravo_data"} v 0) v1 -- 0x7f9dfc0d72a0 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.797 7f9df2ffd700 1 -- 172.21.15.32:0/2685032956 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "bravo_data"}]=0 v188) v1 ==== 99+0+10 (secure 0 0 0) 0x7f9df4007fe0 con 0x7f9dfc0b91a0 2021-04-09T18:30:38.813 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 msgr2=0x7f9de401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 crc :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dfc0b91a0 msgr2=0x7f9dfc0bead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:38.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dfc0b91a0 0x7f9dfc0bead0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7f9df4007fa0 tx=0x7f9df400a8a0).stop 2021-04-09T18:30:38.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 shutdown_connections 2021-04-09T18:30:38.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9dfc0b91a0 0x7f9dfc0bead0 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dfc0b8da0 0x7f9dfc0be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 --2- 172.21.15.32:0/2685032956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dfc0b89a0 0x7f9dfc0c31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 >> 172.21.15.32:0/2685032956 conn(0x7f9dfc0a06a0 msgr2=0x7f9dfc08fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 shutdown_connections 2021-04-09T18:30:38.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.800 7f9e0462b700 1 -- 172.21.15.32:0/2685032956 wait complete. 2021-04-09T18:30:38.826 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:30:38.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:39.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.961 7f7f023ca700 1 Processor -- start 2021-04-09T18:30:39.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.961 7f7f023ca700 1 -- start start 2021-04-09T18:30:39.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.961 7f7f023ca700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7efc0d7ca0 0x7f7efc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:39.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.961 7f7f023ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc10da00 0x7f7efc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:39.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.961 7f7f023ca700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc10d480 con 0x7f7efc10d010 2021-04-09T18:30:39.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.961 7f7f023ca700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7efc0d9220 con 0x7f7efc0d7ca0 2021-04-09T18:30:39.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.961 7f7f023ca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0d86c0 con 0x7f7efc10da00 2021-04-09T18:30:39.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7efbfff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7efc10d010 0x7f7efc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:35398/0 (socket says 172.21.15.32:35398) 2021-04-09T18:30:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.961 7f7f009e7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc10da00 0x7f7efc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7ef3fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7efc0d7ca0 0x7f7efc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7efbfff700 1 -- 172.21.15.32:0/1426979117 learned_addr learned my addr 172.21.15.32:0/1426979117 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:39.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7efb7fe700 1 -- 172.21.15.32:0/1426979117 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174019400 0 0) 0x7f7eec0010c0 con 0x7f7efc10d010 2021-04-09T18:30:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7efb7fe700 1 -- 172.21.15.32:0/1426979117 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee8002fe0 con 0x7f7efc10d010 2021-04-09T18:30:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7ef3fff700 1 -- 172.21.15.32:0/1426979117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc10da00 msgr2=0x7f7efc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7ef3fff700 1 --2- 172.21.15.32:0/1426979117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc10da00 0x7f7efc10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7ef3fff700 1 -- 172.21.15.32:0/1426979117 >> v1:172.21.15.86:6789/0 conn(0x7f7efc10d010 legacy=0x7f7efc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:39.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7ef3fff700 1 -- 172.21.15.32:0/1426979117 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc0d1f70 con 0x7f7efc0d7ca0 2021-04-09T18:30:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7f009e7700 1 --2- 172.21.15.32:0/1426979117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc10da00 0x7f7efc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7ef3fff700 1 --2- 172.21.15.32:0/1426979117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7efc0d7ca0 0x7f7efc0d80a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f7ee4006650 tx=0x7f7ee4009a80).ready entity=mon.1 client_cookie=64fded3aa1cd323f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7efb7fe700 1 -- 172.21.15.32:0/1426979117 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ee40009b0 con 0x7f7efc0d7ca0 2021-04-09T18:30:39.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7efb7fe700 1 -- 172.21.15.32:0/1426979117 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee400c0c0 con 0x7f7efc0d7ca0 2021-04-09T18:30:39.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7efb7fe700 1 -- 172.21.15.32:0/1426979117 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ee400c510 con 0x7f7efc0d7ca0 2021-04-09T18:30:39.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7f023ca700 1 -- 172.21.15.32:0/1426979117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7efc0d7ca0 msgr2=0x7f7efc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:39.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.962 7f7f023ca700 1 --2- 172.21.15.32:0/1426979117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7efc0d7ca0 0x7f7efc0d80a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f7ee4006650 tx=0x7f7ee4009a80).stop 2021-04-09T18:30:39.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.963 7f7f023ca700 1 -- 172.21.15.32:0/1426979117 shutdown_connections 2021-04-09T18:30:39.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.963 7f7f023ca700 1 --2- 172.21.15.32:0/1426979117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc10da00 0x7f7efc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.963 7f7f023ca700 1 --2- 172.21.15.32:0/1426979117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7efc0d7ca0 0x7f7efc0d80a0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.963 7f7f023ca700 1 -- 172.21.15.32:0/1426979117 >> 172.21.15.32:0/1426979117 conn(0x7f7efc0ce5f0 msgr2=0x7f7efc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:39.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.963 7f7f023ca700 1 -- 172.21.15.32:0/1426979117 shutdown_connections 2021-04-09T18:30:39.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.963 7f7f023ca700 1 -- 172.21.15.32:0/1426979117 wait complete. 2021-04-09T18:30:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.983 7f7f023ca700 1 Processor -- start 2021-04-09T18:30:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.983 7f7f023ca700 1 -- start start 2021-04-09T18:30:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f023ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc1a5210 0x7f7efc1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:39.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f023ca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7efc1a6340 0x7f7efc1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f023ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7efc0abaa0 0x7f7efc0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f023ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc1a6d60 con 0x7f7efc0abaa0 2021-04-09T18:30:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f023ca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0ac4c0 con 0x7f7efc1a6340 2021-04-09T18:30:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f023ca700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0ac6e0 con 0x7f7efc1a5210 2021-04-09T18:30:39.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f009e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7efc0abaa0 0x7f7efc0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:39.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f009e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7efc0abaa0 0x7f7efc0abea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56122/0 (socket says 172.21.15.32:56122) 2021-04-09T18:30:39.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f009e7700 1 -- 172.21.15.32:0/4198670647 learned_addr learned my addr 172.21.15.32:0/4198670647 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:39.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7ef3fff700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7efc1a6340 0x7f7efc1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:39.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7efbfff700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc1a5210 0x7f7efc1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:39.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f009e7700 1 -- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc1a5210 msgr2=0x7f7efc1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:39.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f009e7700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc1a5210 0x7f7efc1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f009e7700 1 -- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7efc1a6340 msgr2=0x7f7efc1a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:39.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f009e7700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7efc1a6340 0x7f7efc1a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f009e7700 1 -- 172.21.15.32:0/4198670647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc17a720 con 0x7f7efc0abaa0 2021-04-09T18:30:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.984 7f7f009e7700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7efc0abaa0 0x7f7efc0abea0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7f7ef400b910 tx=0x7f7ef400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.985 7f7ef97fa700 1 -- 172.21.15.32:0/4198670647 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ef4006aa0 con 0x7f7efc0abaa0 2021-04-09T18:30:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.985 7f7ef97fa700 1 -- 172.21.15.32:0/4198670647 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef40011f0 con 0x7f7efc0abaa0 2021-04-09T18:30:39.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.985 7f7ef97fa700 1 -- 172.21.15.32:0/4198670647 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ef4001640 con 0x7f7efc0abaa0 2021-04-09T18:30:39.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.985 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7efc17a720 con 0x7f7efc0abaa0 2021-04-09T18:30:39.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.985 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7efc17a720 con 0x7f7efc0abaa0 2021-04-09T18:30:39.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.985 7f7ef97fa700 1 -- 172.21.15.32:0/4198670647 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7ef40013d0 con 0x7f7efc0abaa0 2021-04-09T18:30:39.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.985 7f7ef97fa700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ee00188c0 0x7f7ee001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:39.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.985 7f7ef97fa700 1 -- 172.21.15.32:0/4198670647 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f7ef4032f40 con 0x7f7efc0abaa0 2021-04-09T18:30:39.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.986 7f7efbfff700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ee00188c0 0x7f7ee001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:39.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:38.986 7f7efbfff700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ee00188c0 0x7f7ee001adc0 crc :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:39.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.005 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7efc1a6d60 con 0x7f7efc0abaa0 2021-04-09T18:30:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.009 7f7ef97fa700 1 -- 172.21.15.32:0/4198670647 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7ef400bdf0 con 0x7f7efc0abaa0 2021-04-09T18:30:39.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.201 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7efc1a6d60 con 0x7f7efc0abaa0 2021-04-09T18:30:39.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.202 7f7ef97fa700 1 -- 172.21.15.32:0/4198670647 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 334 v334) v1 ==== 95+0+3717 (secure 0 0 0) 0x7f7ef40013d0 con 0x7f7efc0abaa0 2021-04-09T18:30:39.218 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 334 2021-04-09T18:30:39.218 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:39.218 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":334,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58739,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330},{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":186,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58736":{"gid":58736,"name":"a","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/558658679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":558658679},{"type":"v1","addr":"172.21.15.32:6834","nonce":558658679}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:39.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ee00188c0 msgr2=0x7f7ee001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:39.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ee00188c0 0x7f7ee001adc0 crc :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7efc0abaa0 msgr2=0x7f7efc0abea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7efc0abaa0 0x7f7efc0abea0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7f7ef400b910 tx=0x7f7ef400bcb0).stop 2021-04-09T18:30:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 shutdown_connections 2021-04-09T18:30:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ee00188c0 0x7f7ee001adc0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7efc0abaa0 0x7f7efc0abea0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7efc1a6340 0x7f7efc1a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 --2- 172.21.15.32:0/4198670647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7efc1a5210 0x7f7efc1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:39.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 >> 172.21.15.32:0/4198670647 conn(0x7f7efc1a99c0 msgr2=0x7f7efc17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:39.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 shutdown_connections 2021-04-09T18:30:39.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:39.204 7f7f023ca700 1 -- 172.21.15.32:0/4198670647 wait complete. 2021-04-09T18:30:39.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 334, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:15.764321', 'modified': '2021-04-09 18:30:33.326504', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 185, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11136}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11136': {'gid': 11136, 'name': 'd', 'rank': 0, 'incarnation': 331, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/264493238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 264493238}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 264493238}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:39.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:30:39.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:30:39.761 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:30:39.762 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:30:39.762 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:30:39.763 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 11136, 2021-04-09T18:30:39.763 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:30:39.763 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:30:39.764 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 9.4311725630000005, 2021-04-09T18:30:39.764 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 334, 2021-04-09T18:30:39.765 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 188, 2021-04-09T18:30:39.765 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 188, 2021-04-09T18:30:39.766 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 13.472126121000001 2021-04-09T18:30:39.766 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:30:39.768 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 11136, "want_state": "up:active", "state": "up:active", "rank_uptime": 9.4311725630000005, "mdsmap_epoch": 334, "osdmap_epoch": 188, "osdmap_epoch_barrier": 188, "uptime": 13.472126121000001 } 2021-04-09T18:30:39.769 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:30:39.770 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:30:39.771 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:30:39.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:39.772 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:39.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:30:39.888 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:30:39.889 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:30:39.949 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:30:39.949 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:30:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:30:39.951 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:39.951 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:30:39.996 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:30:39.997 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mds_namespace=alpha --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:40.038 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:30:40.099 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:30:40.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:30:40.159 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:30:40.160 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:30:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:30:40.162 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:40.162 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:30:40.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:30:40.173 7f541914af80 -1 init, newargv = 0x55d7e2bbb330 newargc=9 2021-04-09T18:30:40.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[92974]: starting ceph client 2021-04-09T18:30:40.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[92974]: starting fuse 2021-04-09T18:30:40.208 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:30:40.208 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:30:40.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:40.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:30:40.211 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:30:40.211 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:30:40.212 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:30:40.213 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:30:40.214 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:30:40.214 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:30:40.215 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:30:40.216 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:30:40.217 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:30:40.217 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:30:40.218 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:30:40.219 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:30:40.220 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:30:40.221 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:30:40.221 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:30:40.222 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:30:40.223 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:30:40.224 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:30:40.224 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:30:40.225 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:30:40.226 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:30:40.227 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:30:40.331 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.92974.asok 2021-04-09T18:30:40.336 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.92974.asok 2021-04-09T18:30:40.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:40.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.92974.asok status 2021-04-09T18:30:40.462 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:30:40.462 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:30:40.463 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:30:40.463 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:30:40.463 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:30:40.463 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:30:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:30:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "92974", 2021-04-09T18:30:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:30:40.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:30:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:30:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 11244, 2021-04-09T18:30:40.465 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:30:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:30:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:30:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 11244 2021-04-09T18:30:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:40.466 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:30:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:30:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2548047912 2021-04-09T18:30:40.467 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:30:40.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:40.468 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:30:40.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:30:40.468 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:30:40.469 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2548047912 2021-04-09T18:30:40.469 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:30:40.469 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.11244 172.21.15.32:0/2548047912", 2021-04-09T18:30:40.469 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/2548047912", 2021-04-09T18:30:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:30:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 334, 2021-04-09T18:30:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 188, 2021-04-09T18:30:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:30:40.470 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:30:40.471 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "alpha" 2021-04-09T18:30:40.471 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:30:40.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:40.473 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:40.473 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:40.518 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:30:40.518 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:30:40.519 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=pad.bin bs=1M conv=fdatasync count=1 seek=0 2021-04-09T18:30:40.599 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2021-04-09T18:30:40.600 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2021-04-09T18:30:40.600 INFO:teuthology.orchestra.run.smithi032.stderr:1048576 bytes (1.0 MB) copied, 0.0156464 s, 67.0 MB/s 2021-04-09T18:30:40.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:40.601 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:40.602 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:40.665 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:30:40.666 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:30:40.667 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=test.bin bs=1M conv=fdatasync count=2 seek=0 2021-04-09T18:30:40.800 INFO:teuthology.orchestra.run.smithi032.stderr:2+0 records in 2021-04-09T18:30:40.801 INFO:teuthology.orchestra.run.smithi032.stderr:2+0 records out 2021-04-09T18:30:40.801 INFO:teuthology.orchestra.run.smithi032.stderr:2097152 bytes (2.1 MB) copied, 0.0198081 s, 106 MB/s 2021-04-09T18:30:40.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2021-04-09T18:30:40.803 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:30:40.804 DEBUG:teuthology.orchestra.run.smithi032:> import stat 2021-04-09T18:30:40.804 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:30:40.805 DEBUG:teuthology.orchestra.run.smithi032:> print(os.stat("/home/ubuntu/cephtest/mnt.0/test.bin").st_ino) 2021-04-09T18:30:40.805 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:30:40.919 INFO:teuthology.orchestra.run.smithi032.stdout:1099511627777 2021-04-09T18:30:41.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:41.117 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:41.118 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:41.201 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:30:41.202 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:30:41.202 INFO:tasks.cephfs.mount:Creating file a 2021-04-09T18:30:41.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch /home/ubuntu/cephtest/mnt.0/a 2021-04-09T18:30:41.266 INFO:tasks.cephfs.mount:Creating file b 2021-04-09T18:30:41.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch /home/ubuntu/cephtest/mnt.0/b 2021-04-09T18:30:41.328 INFO:tasks.cephfs.mount:Creating file c 2021-04-09T18:30:41.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch /home/ubuntu/cephtest/mnt.0/c 2021-04-09T18:30:41.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:30:41.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.525 7fbc770af700 1 Processor -- start 2021-04-09T18:30:41.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.525 7fbc770af700 1 -- start start 2021-04-09T18:30:41.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc770af700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc7010d010 0x7fbc700d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc770af700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc700d1da0 0x7fbc700d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc770af700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7010d480 con 0x7fbc700d82c0 2021-04-09T18:30:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc770af700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc700d9220 con 0x7fbc7010d010 2021-04-09T18:30:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc770af700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700d86c0 con 0x7fbc700d1da0 2021-04-09T18:30:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc756cc700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbc700d82c0 0x7fbc7010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37508/0 (socket says 172.21.15.32:37508) 2021-04-09T18:30:41.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc756cc700 1 -- 172.21.15.32:0/498516144 learned_addr learned my addr 172.21.15.32:0/498516144 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc74ecb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbc7010d010 0x7fbc700d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc6ffff700 1 --2- 172.21.15.32:0/498516144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc700d1da0 0x7fbc700d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc6f7fe700 1 -- 172.21.15.32:0/498516144 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560688215 0 0) 0x7fbc680009b0 con 0x7fbc700d82c0 2021-04-09T18:30:41.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc6f7fe700 1 -- 172.21.15.32:0/498516144 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc5c002fe0 con 0x7fbc700d82c0 2021-04-09T18:30:41.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc74ecb700 1 -- 172.21.15.32:0/498516144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc700d1da0 msgr2=0x7fbc700d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:41.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc74ecb700 1 --2- 172.21.15.32:0/498516144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc700d1da0 0x7fbc700d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc74ecb700 1 -- 172.21.15.32:0/498516144 >> v1:172.21.15.32:6789/0 conn(0x7fbc700d82c0 legacy=0x7fbc7010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.526 7fbc74ecb700 1 -- 172.21.15.32:0/498516144 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7010e1f0 con 0x7fbc7010d010 2021-04-09T18:30:41.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc74ecb700 1 --2- 172.21.15.32:0/498516144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc7010d010 0x7fbc700d1780 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7fbc58006650 tx=0x7fbc58009a00).ready entity=mon.0 client_cookie=15e1151300d92896 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:41.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc6f7fe700 1 -- 172.21.15.32:0/498516144 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc58009df0 con 0x7fbc7010d010 2021-04-09T18:30:41.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc6f7fe700 1 -- 172.21.15.32:0/498516144 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc580009b0 con 0x7fbc7010d010 2021-04-09T18:30:41.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc6f7fe700 1 -- 172.21.15.32:0/498516144 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc58009df0 con 0x7fbc7010d010 2021-04-09T18:30:41.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc770af700 1 -- 172.21.15.32:0/498516144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc7010d010 msgr2=0x7fbc700d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc770af700 1 --2- 172.21.15.32:0/498516144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc7010d010 0x7fbc700d1780 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7fbc58006650 tx=0x7fbc58009a00).stop 2021-04-09T18:30:41.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc770af700 1 -- 172.21.15.32:0/498516144 shutdown_connections 2021-04-09T18:30:41.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc770af700 1 --2- 172.21.15.32:0/498516144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc700d1da0 0x7fbc700d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc770af700 1 --2- 172.21.15.32:0/498516144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc7010d010 0x7fbc700d1780 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc770af700 1 -- 172.21.15.32:0/498516144 >> 172.21.15.32:0/498516144 conn(0x7fbc700ce5f0 msgr2=0x7fbc700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:41.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.527 7fbc770af700 1 -- 172.21.15.32:0/498516144 shutdown_connections 2021-04-09T18:30:41.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.528 7fbc770af700 1 -- 172.21.15.32:0/498516144 wait complete. 2021-04-09T18:30:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.548 7fbc770af700 1 Processor -- start 2021-04-09T18:30:41.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.548 7fbc770af700 1 -- start start 2021-04-09T18:30:41.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.548 7fbc770af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc700b42d0 0x7fbc70183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.548 7fbc770af700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc701ac090 0x7fbc701ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.548 7fbc770af700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc70184d30 0x7fbc70185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.548 7fbc770af700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700a1a90 con 0x7fbc700b42d0 2021-04-09T18:30:41.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.548 7fbc770af700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc701acab0 con 0x7fbc70184d30 2021-04-09T18:30:41.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.548 7fbc770af700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc70185a00 con 0x7fbc701ac090 2021-04-09T18:30:41.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.548 7fbc74ecb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc700b42d0 0x7fbc70183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc74ecb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc700b42d0 0x7fbc70183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56156/0 (socket says 172.21.15.32:56156) 2021-04-09T18:30:41.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc74ecb700 1 -- 172.21.15.32:0/1827380295 learned_addr learned my addr 172.21.15.32:0/1827380295 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:41.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc6ffff700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc701ac090 0x7fbc701ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc756cc700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc70184d30 0x7fbc70185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc74ecb700 1 -- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc701ac090 msgr2=0x7fbc701ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc74ecb700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc701ac090 0x7fbc701ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc74ecb700 1 -- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc70184d30 msgr2=0x7fbc70185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc74ecb700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc70184d30 0x7fbc70185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc74ecb700 1 -- 172.21.15.32:0/1827380295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc70185750 con 0x7fbc700b42d0 2021-04-09T18:30:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc74ecb700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc700b42d0 0x7fbc70183520 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7fbc580096a0 tx=0x7fbc5800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc6d7fa700 1 -- 172.21.15.32:0/1827380295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc580093a0 con 0x7fbc700b42d0 2021-04-09T18:30:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc6d7fa700 1 -- 172.21.15.32:0/1827380295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc580067f0 con 0x7fbc700b42d0 2021-04-09T18:30:41.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc770af700 1 -- 172.21.15.32:0/1827380295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc70185750 con 0x7fbc700b42d0 2021-04-09T18:30:41.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc6d7fa700 1 -- 172.21.15.32:0/1827380295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc58002220 con 0x7fbc700b42d0 2021-04-09T18:30:41.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.549 7fbc770af700 1 -- 172.21.15.32:0/1827380295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc70185750 con 0x7fbc700b42d0 2021-04-09T18:30:41.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.550 7fbc6d7fa700 1 -- 172.21.15.32:0/1827380295 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbc580052e0 con 0x7fbc700b42d0 2021-04-09T18:30:41.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.551 7fbc6d7fa700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc540188c0 0x7fbc5401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.551 7fbc6d7fa700 1 -- 172.21.15.32:0/1827380295 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fbc58032bf0 con 0x7fbc700b42d0 2021-04-09T18:30:41.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.551 7fbc6ffff700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc540188c0 0x7fbc5401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.551 7fbc6ffff700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc540188c0 0x7fbc5401adc0 crc :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:41.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.570 7fbc770af700 1 -- 172.21.15.32:0/1827380295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc70182d10 con 0x7fbc700b42d0 2021-04-09T18:30:41.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.573 7fbc6d7fa700 1 -- 172.21.15.32:0/1827380295 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbc58000c90 con 0x7fbc700b42d0 2021-04-09T18:30:41.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.762 7fbc770af700 1 -- 172.21.15.32:0/1827380295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc7010d480 con 0x7fbc700b42d0 2021-04-09T18:30:41.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.763 7fbc6d7fa700 1 -- 172.21.15.32:0/1827380295 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+17091 (secure 0 0 0) 0x7fbc580052e0 con 0x7fbc700b42d0 2021-04-09T18:30:41.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:41.780 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":188,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:30:30.280596","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:30:13.709244","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":31,"pool_name":"alpha_data","create_time":"2021-04-09 18:30:14.523869","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":32,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:30:21.008022","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":33,"pool_name":"bravo_data","create_time":"2021-04-09 18:30:22.220789","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.32:6835/3158551863":"2021-04-10 18:30:30.264043","172.21.15.32:6833/3158551863":"2021-04-10 18:30:30.264043","172.21.15.32:6834/3729993394":"2021-04-10 18:30:30.261199","172.21.15.32:6832/3729993394":"2021-04-10 18:30:30.261199"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:30:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 -- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc540188c0 msgr2=0x7fbc5401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc540188c0 0x7fbc5401adc0 crc :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 -- 172.21.15.32:0/1827380295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc700b42d0 msgr2=0x7fbc70183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc700b42d0 0x7fbc70183520 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7fbc580096a0 tx=0x7fbc5800b9b0).stop 2021-04-09T18:30:41.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 -- 172.21.15.32:0/1827380295 shutdown_connections 2021-04-09T18:30:41.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbc540188c0 0x7fbc5401adc0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc70184d30 0x7fbc70185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc701ac090 0x7fbc701ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 --2- 172.21.15.32:0/1827380295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc700b42d0 0x7fbc70183520 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 -- 172.21.15.32:0/1827380295 >> 172.21.15.32:0/1827380295 conn(0x7fbc7019b890 msgr2=0x7fbc701a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 -- 172.21.15.32:0/1827380295 shutdown_connections 2021-04-09T18:30:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.765 7fbc770af700 1 -- 172.21.15.32:0/1827380295 wait complete. 2021-04-09T18:30:41.791 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 30, 'pool_name': 'alpha_metadata', 'create_time': '2021-04-09 18:30:13.709244', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'alpha'}}}, {'pool': 31, 'pool_name': 'alpha_data', 'create_time': '2021-04-09 18:30:14.523869', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'alpha'}}}, {'pool': 32, 'pool_name': 'bravo_metadata', 'create_time': '2021-04-09 18:30:21.008022', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '183', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bravo'}}}, {'pool': 33, 'pool_name': 'bravo_data', 'create_time': '2021-04-09 18:30:22.220789', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bravo'}}}] 2021-04-09T18:30:41.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:30:41.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.929 7f148bed1700 1 Processor -- start 2021-04-09T18:30:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.929 7f148bed1700 1 -- start start 2021-04-09T18:30:41.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f148bed1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f148410d010 0x7f14840d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f148bed1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840d8490 0x7f148410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f148bed1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148410d480 con 0x7f14840d1da0 2021-04-09T18:30:41.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f148bed1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14840d8d30 con 0x7f148410d010 2021-04-09T18:30:41.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f148bed1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14840d9220 con 0x7f14840d8490 2021-04-09T18:30:41.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f1489ced700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f148410d010 0x7f14840d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f148a4ee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840d8490 0x7f148410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f1489ced700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f148410d010 0x7f14840d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:47918/0 (socket says 172.21.15.32:47918) 2021-04-09T18:30:41.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f148a4ee700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840d8490 0x7f148410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:35156/0 (socket says 172.21.15.32:35156) 2021-04-09T18:30:41.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f14894ec700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f14840d1da0 0x7f14840d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:35448/0 (socket says 172.21.15.32:35448) 2021-04-09T18:30:41.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f1489ced700 1 -- 172.21.15.32:0/1473454154 learned_addr learned my addr 172.21.15.32:0/1473454154 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:41.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f1489ced700 1 -- 172.21.15.32:0/1473454154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840d8490 msgr2=0x7f148410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:41.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f1489ced700 1 --2- 172.21.15.32:0/1473454154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840d8490 0x7f148410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f1489ced700 1 -- 172.21.15.32:0/1473454154 >> v1:172.21.15.86:6789/0 conn(0x7f14840d1da0 legacy=0x7f14840d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f1489ced700 1 -- 172.21.15.32:0/1473454154 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148410e020 con 0x7f148410d010 2021-04-09T18:30:41.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f1488ceb700 1 -- 172.21.15.32:0/1473454154 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103549390 0 0) 0x7f1474000c60 con 0x7f14840d1da0 2021-04-09T18:30:41.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.930 7f148a4ee700 1 --2- 172.21.15.32:0/1473454154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840d8490 0x7f148410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:41.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.931 7f1489ced700 1 --2- 172.21.15.32:0/1473454154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f148410d010 0x7f14840d1780 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f1480006700 tx=0x7f1480009b80).ready entity=mon.1 client_cookie=27d3f6627a5a4c20 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:41.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.931 7f1488ceb700 1 -- 172.21.15.32:0/1473454154 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f148000c0c0 con 0x7f148410d010 2021-04-09T18:30:41.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.931 7f1488ceb700 1 -- 172.21.15.32:0/1473454154 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148000c440 con 0x7f148410d010 2021-04-09T18:30:41.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.931 7f1488ceb700 1 -- 172.21.15.32:0/1473454154 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f148000c920 con 0x7f148410d010 2021-04-09T18:30:41.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.931 7f148bed1700 1 -- 172.21.15.32:0/1473454154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f148410d010 msgr2=0x7f14840d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.931 7f148bed1700 1 --2- 172.21.15.32:0/1473454154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f148410d010 0x7f14840d1780 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f1480006700 tx=0x7f1480009b80).stop 2021-04-09T18:30:41.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.932 7f148bed1700 1 -- 172.21.15.32:0/1473454154 shutdown_connections 2021-04-09T18:30:41.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.932 7f148bed1700 1 --2- 172.21.15.32:0/1473454154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840d8490 0x7f148410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.932 7f148bed1700 1 --2- 172.21.15.32:0/1473454154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f148410d010 0x7f14840d1780 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.932 7f148bed1700 1 -- 172.21.15.32:0/1473454154 >> 172.21.15.32:0/1473454154 conn(0x7f14840ce5f0 msgr2=0x7f14840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:41.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.932 7f148bed1700 1 -- 172.21.15.32:0/1473454154 shutdown_connections 2021-04-09T18:30:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.932 7f148bed1700 1 -- 172.21.15.32:0/1473454154 wait complete. 2021-04-09T18:30:41.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.952 7f148bed1700 1 Processor -- start 2021-04-09T18:30:41.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.952 7f148bed1700 1 -- start start 2021-04-09T18:30:41.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.952 7f148bed1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14840a3b10 0x7f14840a32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.952 7f148bed1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840b33b0 0x7f14840b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.952 7f148bed1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484195b80 0x7f1484195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.952 7f148bed1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14840b4130 con 0x7f1484195b80 2021-04-09T18:30:41.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.952 7f148bed1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14840a3f10 con 0x7f14840a3b10 2021-04-09T18:30:41.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.952 7f148bed1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14840b3dd0 con 0x7f14840b33b0 2021-04-09T18:30:41.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148a4ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484195b80 0x7f1484195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148a4ee700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484195b80 0x7f1484195f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56170/0 (socket says 172.21.15.32:56170) 2021-04-09T18:30:41.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148a4ee700 1 -- 172.21.15.32:0/1129459822 learned_addr learned my addr 172.21.15.32:0/1129459822 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:41.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f1489ced700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14840a3b10 0x7f14840a32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f14894ec700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840b33b0 0x7f14840b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148a4ee700 1 -- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840b33b0 msgr2=0x7f14840b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148a4ee700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840b33b0 0x7f14840b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148a4ee700 1 -- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14840a3b10 msgr2=0x7f14840a32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148a4ee700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14840a3b10 0x7f14840a32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:41.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148a4ee700 1 -- 172.21.15.32:0/1129459822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14841965a0 con 0x7f1484195b80 2021-04-09T18:30:41.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148a4ee700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484195b80 0x7f1484195f80 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7f147c00b970 tx=0x7f147c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:41.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f147a7fc700 1 -- 172.21.15.32:0/1129459822 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f147c000f90 con 0x7f1484195b80 2021-04-09T18:30:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f147a7fc700 1 -- 172.21.15.32:0/1129459822 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147c00be50 con 0x7f1484195b80 2021-04-09T18:30:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148bed1700 1 -- 172.21.15.32:0/1129459822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14841965a0 con 0x7f1484195b80 2021-04-09T18:30:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f147a7fc700 1 -- 172.21.15.32:0/1129459822 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f147c006aa0 con 0x7f1484195b80 2021-04-09T18:30:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.953 7f148bed1700 1 -- 172.21.15.32:0/1129459822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14841965a0 con 0x7f1484195b80 2021-04-09T18:30:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.954 7f147a7fc700 1 -- 172.21.15.32:0/1129459822 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f147c006aa0 con 0x7f1484195b80 2021-04-09T18:30:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.954 7f147a7fc700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f146c018930 0x7f146c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.954 7f147a7fc700 1 -- 172.21.15.32:0/1129459822 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f147c033180 con 0x7f1484195b80 2021-04-09T18:30:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.954 7f1489ced700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f146c018930 0x7f146c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.954 7f1489ced700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f146c018930 0x7f146c01ae10 crc :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:41.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.974 7f148bed1700 1 -- 172.21.15.32:0/1129459822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14840a3f10 con 0x7f1484195b80 2021-04-09T18:30:41.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:41.977 7f147a7fc700 1 -- 172.21.15.32:0/1129459822 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f147c01fae0 con 0x7f1484195b80 2021-04-09T18:30:42.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.160 7f148bed1700 1 -- 172.21.15.32:0/1129459822 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f14840a3f10 con 0x7f1484195b80 2021-04-09T18:30:42.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.160 7f147a7fc700 1 -- 172.21.15.32:0/1129459822 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v188) v1 ==== 92+0+10 (secure 0 0 0) 0x7f147c006aa0 con 0x7f1484195b80 2021-04-09T18:30:42.176 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 -- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f146c018930 msgr2=0x7f146c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f146c018930 0x7f146c01ae10 crc :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 -- 172.21.15.32:0/1129459822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484195b80 msgr2=0x7f1484195f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484195b80 0x7f1484195f80 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7f147c00b970 tx=0x7f147c00bd10).stop 2021-04-09T18:30:42.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 -- 172.21.15.32:0/1129459822 shutdown_connections 2021-04-09T18:30:42.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f146c018930 0x7f146c01ae10 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1484195b80 0x7f1484195f80 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14840b33b0 0x7f14840b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 --2- 172.21.15.32:0/1129459822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14840a3b10 0x7f14840a32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 -- 172.21.15.32:0/1129459822 >> 172.21.15.32:0/1129459822 conn(0x7f1484094140 msgr2=0x7f14841b7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 -- 172.21.15.32:0/1129459822 shutdown_connections 2021-04-09T18:30:42.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.163 7f148bed1700 1 -- 172.21.15.32:0/1129459822 wait complete. 2021-04-09T18:30:42.188 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_metadata pg_num 2021-04-09T18:30:42.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.325 7f2e03989700 1 Processor -- start 2021-04-09T18:30:42.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e03989700 1 -- start start 2021-04-09T18:30:42.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e03989700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0cffc0 0x7f2dfc0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e03989700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2dfc0d6ca0 0x7f2dfc10cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e03989700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dfc0d72a0 con 0x7f2dfc0d7820 2021-04-09T18:30:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e03989700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dfc0d70a0 con 0x7f2dfc0d6ca0 2021-04-09T18:30:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e03989700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dfc0d9200 con 0x7f2dfc0cffc0 2021-04-09T18:30:42.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e01fa6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2dfc0d6ca0 0x7f2dfc10cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e01fa6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2dfc0d6ca0 0x7f2dfc10cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56176/0 (socket says 172.21.15.32:56176) 2021-04-09T18:30:42.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e01fa6700 1 -- 172.21.15.32:0/3138775542 learned_addr learned my addr 172.21.15.32:0/3138775542 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e00fa4700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2dfc0d7820 0x7f2dfc0d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37536/0 (socket says 172.21.15.32:37536) 2021-04-09T18:30:42.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.326 7f2e017a5700 1 --2- 172.21.15.32:0/3138775542 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0cffc0 0x7f2dfc0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2e01fa6700 1 -- 172.21.15.32:0/3138775542 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0cffc0 msgr2=0x7f2dfc0d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:42.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2e01fa6700 1 --2- 172.21.15.32:0/3138775542 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0cffc0 0x7f2dfc0d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2e01fa6700 1 -- 172.21.15.32:0/3138775542 >> v1:172.21.15.32:6789/0 conn(0x7f2dfc0d7820 legacy=0x7f2dfc0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2e01fa6700 1 -- 172.21.15.32:0/3138775542 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc0d8410 con 0x7f2dfc0d6ca0 2021-04-09T18:30:42.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2df3fff700 1 -- 172.21.15.32:0/3138775542 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026362165 0 0) 0x7f2dec0009b0 con 0x7f2dfc0d7820 2021-04-09T18:30:42.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2e01fa6700 1 --2- 172.21.15.32:0/3138775542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc0d6ca0 0x7f2dfc10cff0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7f2df4006f00 tx=0x7f2df4009e90).ready entity=mon.0 client_cookie=53b3dbc9bb7742d9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:42.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2df3fff700 1 -- 172.21.15.32:0/3138775542 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df400b300 con 0x7f2dfc0d6ca0 2021-04-09T18:30:42.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2df3fff700 1 -- 172.21.15.32:0/3138775542 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df400b640 con 0x7f2dfc0d6ca0 2021-04-09T18:30:42.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2df3fff700 1 -- 172.21.15.32:0/3138775542 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df400bb20 con 0x7f2dfc0d6ca0 2021-04-09T18:30:42.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2e03989700 1 -- 172.21.15.32:0/3138775542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc0d6ca0 msgr2=0x7f2dfc10cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.327 7f2e03989700 1 --2- 172.21.15.32:0/3138775542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc0d6ca0 0x7f2dfc10cff0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7f2df4006f00 tx=0x7f2df4009e90).stop 2021-04-09T18:30:42.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.328 7f2e03989700 1 -- 172.21.15.32:0/3138775542 shutdown_connections 2021-04-09T18:30:42.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.328 7f2e03989700 1 --2- 172.21.15.32:0/3138775542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc0d6ca0 0x7f2dfc10cff0 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.328 7f2e03989700 1 --2- 172.21.15.32:0/3138775542 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0cffc0 0x7f2dfc0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.328 7f2e03989700 1 -- 172.21.15.32:0/3138775542 >> 172.21.15.32:0/3138775542 conn(0x7f2dfc0ce5e0 msgr2=0x7f2dfc0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:42.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.328 7f2e03989700 1 -- 172.21.15.32:0/3138775542 shutdown_connections 2021-04-09T18:30:42.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.328 7f2e03989700 1 -- 172.21.15.32:0/3138775542 wait complete. 2021-04-09T18:30:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.348 7f2e03989700 1 Processor -- start 2021-04-09T18:30:42.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.348 7f2e03989700 1 -- start start 2021-04-09T18:30:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e03989700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc190280 0x7f2dfc187b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e03989700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc190680 0x7f2dfc188130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e03989700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc099800 0x7f2dfc099c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e03989700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc187590 con 0x7f2dfc190280 2021-04-09T18:30:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e03989700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc09a220 con 0x7f2dfc190680 2021-04-09T18:30:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e03989700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc098680 con 0x7f2dfc099800 2021-04-09T18:30:42.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e017a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc190280 0x7f2dfc187b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e017a5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc190280 0x7f2dfc187b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56180/0 (socket says 172.21.15.32:56180) 2021-04-09T18:30:42.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e017a5700 1 -- 172.21.15.32:0/2375302084 learned_addr learned my addr 172.21.15.32:0/2375302084 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:42.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e01fa6700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc099800 0x7f2dfc099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e00fa4700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc190680 0x7f2dfc188130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e017a5700 1 -- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc099800 msgr2=0x7f2dfc099c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e017a5700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc099800 0x7f2dfc099c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e017a5700 1 -- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc190680 msgr2=0x7f2dfc188130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e017a5700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc190680 0x7f2dfc188130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e017a5700 1 -- 172.21.15.32:0/2375302084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc0988a0 con 0x7f2dfc190280 2021-04-09T18:30:42.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e00fa4700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc190680 0x7f2dfc188130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:42.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2e017a5700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc190280 0x7f2dfc187b10 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 rx=0x7f2df8006f60 tx=0x7f2df800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:42.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.349 7f2df1ffb700 1 -- 172.21.15.32:0/2375302084 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df800bd60 con 0x7f2dfc190280 2021-04-09T18:30:42.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.350 7f2df1ffb700 1 -- 172.21.15.32:0/2375302084 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df8000c90 con 0x7f2dfc190280 2021-04-09T18:30:42.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.350 7f2e03989700 1 -- 172.21.15.32:0/2375302084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc0988a0 con 0x7f2dfc190280 2021-04-09T18:30:42.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.350 7f2df1ffb700 1 -- 172.21.15.32:0/2375302084 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df8002220 con 0x7f2dfc190280 2021-04-09T18:30:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.350 7f2e03989700 1 -- 172.21.15.32:0/2375302084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc0988a0 con 0x7f2dfc190280 2021-04-09T18:30:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.350 7f2df1ffb700 1 -- 172.21.15.32:0/2375302084 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2df80052e0 con 0x7f2dfc190280 2021-04-09T18:30:42.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.350 7f2df1ffb700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2ddc018930 0x7f2ddc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.350 7f2df1ffb700 1 -- 172.21.15.32:0/2375302084 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f2df8032db0 con 0x7f2dfc190280 2021-04-09T18:30:42.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.351 7f2e00fa4700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2ddc018930 0x7f2ddc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.351 7f2e00fa4700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2ddc018930 0x7f2ddc01ae10 crc :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:42.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.370 7f2e03989700 1 -- 172.21.15.32:0/2375302084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc0d72a0 con 0x7f2dfc190280 2021-04-09T18:30:42.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.374 7f2df1ffb700 1 -- 172.21.15.32:0/2375302084 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2df8000c90 con 0x7f2dfc190280 2021-04-09T18:30:42.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.554 7f2e03989700 1 -- 172.21.15.32:0/2375302084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"} v 0) v1 -- 0x7f2dfc0d72a0 con 0x7f2dfc190280 2021-04-09T18:30:42.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.554 7f2df1ffb700 1 -- 172.21.15.32:0/2375302084 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"}]=0 v188) v1 ==== 103+0+10 (secure 0 0 0) 0x7f2df8006b00 con 0x7f2dfc190280 2021-04-09T18:30:42.571 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 -- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2ddc018930 msgr2=0x7f2ddc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2ddc018930 0x7f2ddc01ae10 crc :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 -- 172.21.15.32:0/2375302084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc190280 msgr2=0x7f2dfc187b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc190280 0x7f2dfc187b10 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 rx=0x7f2df8006f60 tx=0x7f2df800b970).stop 2021-04-09T18:30:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 -- 172.21.15.32:0/2375302084 shutdown_connections 2021-04-09T18:30:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2ddc018930 0x7f2ddc01ae10 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc099800 0x7f2dfc099c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc190680 0x7f2dfc188130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 --2- 172.21.15.32:0/2375302084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2dfc190280 0x7f2dfc187b10 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 -- 172.21.15.32:0/2375302084 >> 172.21.15.32:0/2375302084 conn(0x7f2dfc09bbc0 msgr2=0x7f2dfc0ad9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:42.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 -- 172.21.15.32:0/2375302084 shutdown_connections 2021-04-09T18:30:42.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.557 7f2e03989700 1 -- 172.21.15.32:0/2375302084 wait complete. 2021-04-09T18:30:42.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_data pg_num 2021-04-09T18:30:42.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.724 7f9f010df700 1 Processor -- start 2021-04-09T18:30:42.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.724 7f9f010df700 1 -- start start 2021-04-09T18:30:42.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9f010df700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9efc10d010 0x7f9efc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9f010df700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9efc0d1da0 0x7f9efc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9f010df700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc10d480 con 0x7f9efc0d82c0 2021-04-09T18:30:42.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9f010df700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9efc0d8d30 con 0x7f9efc0d1da0 2021-04-09T18:30:42.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9f010df700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9efc0d9220 con 0x7f9efc10d010 2021-04-09T18:30:42.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efa59c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9efc0d1da0 0x7f9efc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efad9d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9efc10d010 0x7f9efc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efa59c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9efc0d1da0 0x7f9efc0d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56192/0 (socket says 172.21.15.32:56192) 2021-04-09T18:30:42.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efb59e700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9efc0d82c0 0x7f9efc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:35474/0 (socket says 172.21.15.32:35474) 2021-04-09T18:30:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efa59c700 1 -- 172.21.15.32:0/4032806483 learned_addr learned my addr 172.21.15.32:0/4032806483 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efad9d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9efc10d010 0x7f9efc0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:47946/0 (socket says 172.21.15.32:47946) 2021-04-09T18:30:42.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efa59c700 1 -- 172.21.15.32:0/4032806483 >> v2:172.21.15.86:3300/0 conn(0x7f9efc10d010 msgr2=0x7f9efc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:42.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efa59c700 1 --2- 172.21.15.32:0/4032806483 >> v2:172.21.15.86:3300/0 conn(0x7f9efc10d010 0x7f9efc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efa59c700 1 -- 172.21.15.32:0/4032806483 >> v1:172.21.15.86:6789/0 conn(0x7f9efc0d82c0 legacy=0x7f9efc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efa59c700 1 -- 172.21.15.32:0/4032806483 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0d86f0 con 0x7f9efc0d1da0 2021-04-09T18:30:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efad9d700 1 --2- 172.21.15.32:0/4032806483 >> v2:172.21.15.86:3300/0 conn(0x7f9efc10d010 0x7f9efc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:42.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9efa59c700 1 --2- 172.21.15.32:0/4032806483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0d1da0 0x7f9efc0d7ca0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 rx=0x7f9ef0006700 tx=0x7f9ef0009b80).ready entity=mon.0 client_cookie=5e6db1226d5b42ee server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:42.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9ef9d9b700 1 -- 172.21.15.32:0/4032806483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ef000c0c0 con 0x7f9efc0d1da0 2021-04-09T18:30:42.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9ef9d9b700 1 -- 172.21.15.32:0/4032806483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef000c440 con 0x7f9efc0d1da0 2021-04-09T18:30:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.725 7f9ef9d9b700 1 -- 172.21.15.32:0/4032806483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ef000c920 con 0x7f9efc0d1da0 2021-04-09T18:30:42.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.726 7f9f010df700 1 -- 172.21.15.32:0/4032806483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0d1da0 msgr2=0x7f9efc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.726 7f9f010df700 1 --2- 172.21.15.32:0/4032806483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0d1da0 0x7f9efc0d7ca0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 rx=0x7f9ef0006700 tx=0x7f9ef0009b80).stop 2021-04-09T18:30:42.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.726 7f9f010df700 1 -- 172.21.15.32:0/4032806483 shutdown_connections 2021-04-09T18:30:42.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.726 7f9f010df700 1 --2- 172.21.15.32:0/4032806483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0d1da0 0x7f9efc0d7ca0 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.726 7f9f010df700 1 --2- 172.21.15.32:0/4032806483 >> v2:172.21.15.86:3300/0 conn(0x7f9efc10d010 0x7f9efc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.726 7f9f010df700 1 -- 172.21.15.32:0/4032806483 >> 172.21.15.32:0/4032806483 conn(0x7f9efc0ce5f0 msgr2=0x7f9efc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.726 7f9f010df700 1 -- 172.21.15.32:0/4032806483 shutdown_connections 2021-04-09T18:30:42.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.726 7f9f010df700 1 -- 172.21.15.32:0/4032806483 wait complete. 2021-04-09T18:30:42.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.746 7f9f010df700 1 Processor -- start 2021-04-09T18:30:42.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.746 7f9f010df700 1 -- start start 2021-04-09T18:30:42.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9f010df700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9efc0a0110 0x7f9efc18c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9f010df700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0a0510 0x7f9efc0c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9f010df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9efc0c36e0 0x7f9efc0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9f010df700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc18cf60 con 0x7f9efc0a0510 2021-04-09T18:30:42.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9f010df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc199ea0 con 0x7f9efc0c36e0 2021-04-09T18:30:42.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9f010df700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc187d70 con 0x7f9efc0a0110 2021-04-09T18:30:42.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9efa59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0a0510 0x7f9efc0c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9efa59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0a0510 0x7f9efc0c30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56194/0 (socket says 172.21.15.32:56194) 2021-04-09T18:30:42.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9efa59c700 1 -- 172.21.15.32:0/3062790883 learned_addr learned my addr 172.21.15.32:0/3062790883 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:42.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9efad9d700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9efc0a0110 0x7f9efc18c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.747 7f9efb59e700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9efc0c36e0 0x7f9efc0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9efa59c700 1 -- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9efc0a0110 msgr2=0x7f9efc18c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9efa59c700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9efc0a0110 0x7f9efc18c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9efa59c700 1 -- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9efc0c36e0 msgr2=0x7f9efc0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9efa59c700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9efc0c36e0 0x7f9efc0c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9efa59c700 1 -- 172.21.15.32:0/3062790883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc18d4e0 con 0x7f9efc0a0510 2021-04-09T18:30:42.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9efad9d700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9efc0a0110 0x7f9efc18c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:42.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9efa59c700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0a0510 0x7f9efc0c30c0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7f9ef000b650 tx=0x7f9ef000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:42.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9eeb7fe700 1 -- 172.21.15.32:0/3062790883 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ef000bde0 con 0x7f9efc0a0510 2021-04-09T18:30:42.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9eeb7fe700 1 -- 172.21.15.32:0/3062790883 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef0002030 con 0x7f9efc0a0510 2021-04-09T18:30:42.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9f010df700 1 -- 172.21.15.32:0/3062790883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc18d4e0 con 0x7f9efc0a0510 2021-04-09T18:30:42.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9eeb7fe700 1 -- 172.21.15.32:0/3062790883 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ef0002590 con 0x7f9efc0a0510 2021-04-09T18:30:42.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.748 7f9f010df700 1 -- 172.21.15.32:0/3062790883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc18d4e0 con 0x7f9efc0a0510 2021-04-09T18:30:42.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.749 7f9eeb7fe700 1 -- 172.21.15.32:0/3062790883 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9ef0000c90 con 0x7f9efc0a0510 2021-04-09T18:30:42.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.749 7f9eeb7fe700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ee00188d0 0x7f9ee001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:42.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.749 7f9eeb7fe700 1 -- 172.21.15.32:0/3062790883 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f9ef0032e90 con 0x7f9efc0a0510 2021-04-09T18:30:42.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.749 7f9efad9d700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ee00188d0 0x7f9ee001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:42.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.749 7f9efad9d700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ee00188d0 0x7f9ee001add0 crc :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:42.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.769 7f9f010df700 1 -- 172.21.15.32:0/3062790883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc10d480 con 0x7f9efc0a0510 2021-04-09T18:30:42.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.772 7f9eeb7fe700 1 -- 172.21.15.32:0/3062790883 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9ef001f7f0 con 0x7f9efc0a0510 2021-04-09T18:30:42.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.953 7f9f010df700 1 -- 172.21.15.32:0/3062790883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"} v 0) v1 -- 0x7f9efc10d480 con 0x7f9efc0a0510 2021-04-09T18:30:42.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.954 7f9eeb7fe700 1 -- 172.21.15.32:0/3062790883 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"}]=0 v188) v1 ==== 99+0+10 (secure 0 0 0) 0x7f9ef000bde0 con 0x7f9efc0a0510 2021-04-09T18:30:42.970 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:42.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 -- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ee00188d0 msgr2=0x7f9ee001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ee00188d0 0x7f9ee001add0 crc :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 -- 172.21.15.32:0/3062790883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0a0510 msgr2=0x7f9efc0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:42.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0a0510 0x7f9efc0c30c0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7f9ef000b650 tx=0x7f9ef000b9f0).stop 2021-04-09T18:30:42.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 -- 172.21.15.32:0/3062790883 shutdown_connections 2021-04-09T18:30:42.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ee00188d0 0x7f9ee001add0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9efc0a0510 0x7f9efc0c30c0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9efc0c36e0 0x7f9efc0c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 --2- 172.21.15.32:0/3062790883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9efc0a0110 0x7f9efc18c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:42.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 -- 172.21.15.32:0/3062790883 >> 172.21.15.32:0/3062790883 conn(0x7f9efc0b93a0 msgr2=0x7f9efc0b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:42.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.956 7f9f010df700 1 -- 172.21.15.32:0/3062790883 shutdown_connections 2021-04-09T18:30:42.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:42.957 7f9f010df700 1 -- 172.21.15.32:0/3062790883 wait complete. 2021-04-09T18:30:42.982 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get bravo_metadata pg_num 2021-04-09T18:30:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.119 7f5a1cb01700 1 Processor -- start 2021-04-09T18:30:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.119 7f5a1cb01700 1 -- start start 2021-04-09T18:30:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.120 7f5a1cb01700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5a1810e7f0 0x7f5a180d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.120 7f5a1cb01700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180d72a0 con 0x7f5a180d6ca0 2021-04-09T18:30:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.120 7f5a1cb01700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180d2460 con 0x7f5a180cffc0 2021-04-09T18:30:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.120 7f5a1cb01700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a180d70a0 con 0x7f5a1810e7f0 2021-04-09T18:30:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.120 7f5a1605d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5a180d6ca0 0x7f5a1810cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37562/0 (socket says 172.21.15.32:37562) 2021-04-09T18:30:43.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.120 7f5a1605d700 1 -- 172.21.15.32:0/2069983343 learned_addr learned my addr 172.21.15.32:0/2069983343 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:43.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.120 7f5a1705f700 1 --2- 172.21.15.32:0/2069983343 >> v2:172.21.15.86:3300/0 conn(0x7f5a1810e7f0 0x7f5a180d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.120 7f5a1585c700 1 -- 172.21.15.32:0/2069983343 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145744730 0 0) 0x7f5a0c000ce0 con 0x7f5a180d6ca0 2021-04-09T18:30:43.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1585c700 1 -- 172.21.15.32:0/2069983343 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00002fe0 con 0x7f5a180d6ca0 2021-04-09T18:30:43.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1585c700 1 -- 172.21.15.32:0/2069983343 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338941210 0 0) 0x7f5a08001080 con 0x7f5a180cffc0 2021-04-09T18:30:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1585c700 1 -- 172.21.15.32:0/2069983343 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00003210 con 0x7f5a180cffc0 2021-04-09T18:30:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1705f700 1 -- 172.21.15.32:0/2069983343 >> v1:172.21.15.86:6789/0 conn(0x7f5a180cffc0 legacy=0x7f5a180d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1585c700 1 -- 172.21.15.32:0/2069983343 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3092147714 0 0) 0x7f5a0c0010e0 con 0x7f5a180d6ca0 2021-04-09T18:30:43.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1705f700 1 -- 172.21.15.32:0/2069983343 >> v1:172.21.15.32:6789/0 conn(0x7f5a180d6ca0 legacy=0x7f5a1810cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1705f700 1 -- 172.21.15.32:0/2069983343 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180d9200 con 0x7f5a1810e7f0 2021-04-09T18:30:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1705f700 1 --2- 172.21.15.32:0/2069983343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a1810e7f0 0x7f5a180d8010 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7f5a10006e50 tx=0x7f5a10009df0).ready entity=mon.1 client_cookie=aed4663139b35daf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1585c700 1 -- 172.21.15.32:0/2069983343 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a1000b510 con 0x7f5a1810e7f0 2021-04-09T18:30:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1585c700 1 -- 172.21.15.32:0/2069983343 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a10003030 con 0x7f5a1810e7f0 2021-04-09T18:30:43.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1585c700 1 -- 172.21.15.32:0/2069983343 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a1000b7d0 con 0x7f5a1810e7f0 2021-04-09T18:30:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1cb01700 1 -- 172.21.15.32:0/2069983343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a1810e7f0 msgr2=0x7f5a180d8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.121 7f5a1cb01700 1 --2- 172.21.15.32:0/2069983343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a1810e7f0 0x7f5a180d8010 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7f5a10006e50 tx=0x7f5a10009df0).stop 2021-04-09T18:30:43.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.122 7f5a1cb01700 1 -- 172.21.15.32:0/2069983343 shutdown_connections 2021-04-09T18:30:43.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.122 7f5a1cb01700 1 --2- 172.21.15.32:0/2069983343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a1810e7f0 0x7f5a180d8010 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.122 7f5a1cb01700 1 -- 172.21.15.32:0/2069983343 >> 172.21.15.32:0/2069983343 conn(0x7f5a180ce5e0 msgr2=0x7f5a180cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:43.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.122 7f5a1cb01700 1 -- 172.21.15.32:0/2069983343 shutdown_connections 2021-04-09T18:30:43.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.122 7f5a1cb01700 1 -- 172.21.15.32:0/2069983343 wait complete. 2021-04-09T18:30:43.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.142 7f5a1cb01700 1 Processor -- start 2021-04-09T18:30:43.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.142 7f5a1cb01700 1 -- start start 2021-04-09T18:30:43.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1cb01700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a18182e60 0x7f5a18183260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1cb01700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1809d0f0 0x7f5a1809d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1cb01700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a18181330 0x7f5a18181730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1cb01700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1809cb70 con 0x7f5a1809d0f0 2021-04-09T18:30:43.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1cb01700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1809db10 con 0x7f5a18181330 2021-04-09T18:30:43.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1cb01700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18181d50 con 0x7f5a18182e60 2021-04-09T18:30:43.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1605d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1809d0f0 0x7f5a1809d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1605d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1809d0f0 0x7f5a1809d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56208/0 (socket says 172.21.15.32:56208) 2021-04-09T18:30:43.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1605d700 1 -- 172.21.15.32:0/880352445 learned_addr learned my addr 172.21.15.32:0/880352445 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:43.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1705f700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a18181330 0x7f5a18181730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1685e700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a18182e60 0x7f5a18183260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1605d700 1 -- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a18182e60 msgr2=0x7f5a18183260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1605d700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a18182e60 0x7f5a18183260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1605d700 1 -- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a18181330 msgr2=0x7f5a18181730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1605d700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a18181330 0x7f5a18181730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1605d700 1 -- 172.21.15.32:0/880352445 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a18181f70 con 0x7f5a1809d0f0 2021-04-09T18:30:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a1605d700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1809d0f0 0x7f5a1809d4f0 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 rx=0x7f5a0c007f90 tx=0x7f5a0c00c8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:43.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.143 7f5a077fe700 1 -- 172.21.15.32:0/880352445 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a0c00ccb0 con 0x7f5a1809d0f0 2021-04-09T18:30:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.144 7f5a077fe700 1 -- 172.21.15.32:0/880352445 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c007020 con 0x7f5a1809d0f0 2021-04-09T18:30:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.144 7f5a077fe700 1 -- 172.21.15.32:0/880352445 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a0c0043a0 con 0x7f5a1809d0f0 2021-04-09T18:30:43.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.144 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a18181f70 con 0x7f5a1809d0f0 2021-04-09T18:30:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.144 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a18181f70 con 0x7f5a1809d0f0 2021-04-09T18:30:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.144 7f5a077fe700 1 -- 172.21.15.32:0/880352445 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5a0c007200 con 0x7f5a1809d0f0 2021-04-09T18:30:43.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.144 7f5a077fe700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59fc018930 0x7f59fc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.144 7f5a077fe700 1 -- 172.21.15.32:0/880352445 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f5a0c035020 con 0x7f5a1809d0f0 2021-04-09T18:30:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.144 7f5a1685e700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59fc018930 0x7f59fc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.145 7f5a1685e700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59fc018930 0x7f59fc01adc0 crc :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:43.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.164 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18093df0 con 0x7f5a1809d0f0 2021-04-09T18:30:43.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.167 7f5a077fe700 1 -- 172.21.15.32:0/880352445 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5a0c007b50 con 0x7f5a1809d0f0 2021-04-09T18:30:43.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.347 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "bravo_metadata"} v 0) v1 -- 0x7f5a180d72a0 con 0x7f5a1809d0f0 2021-04-09T18:30:43.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.348 7f5a077fe700 1 -- 172.21.15.32:0/880352445 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "bravo_metadata"}]=0 v188) v1 ==== 103+0+10 (secure 0 0 0) 0x7f5a0c007000 con 0x7f5a1809d0f0 2021-04-09T18:30:43.364 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:43.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59fc018930 msgr2=0x7f59fc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59fc018930 0x7f59fc01adc0 crc :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1809d0f0 msgr2=0x7f5a1809d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1809d0f0 0x7f5a1809d4f0 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 rx=0x7f5a0c007f90 tx=0x7f5a0c00c8c0).stop 2021-04-09T18:30:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 shutdown_connections 2021-04-09T18:30:43.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f59fc018930 0x7f59fc01adc0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a1809d0f0 0x7f5a1809d4f0 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a18181330 0x7f5a18181730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 --2- 172.21.15.32:0/880352445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a18182e60 0x7f5a18183260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.350 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 >> 172.21.15.32:0/880352445 conn(0x7f5a1809fa00 msgr2=0x7f5a180b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:43.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.351 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 shutdown_connections 2021-04-09T18:30:43.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.351 7f5a1cb01700 1 -- 172.21.15.32:0/880352445 wait complete. 2021-04-09T18:30:43.376 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get bravo_data pg_num 2021-04-09T18:30:43.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.512 7f60715a8700 1 Processor -- start 2021-04-09T18:30:43.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.512 7f60715a8700 1 -- start start 2021-04-09T18:30:43.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f60715a8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f606c10da00 0x7f606c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f60715a8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c10d480 con 0x7f606c0d7ca0 2021-04-09T18:30:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f60715a8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0d1f70 con 0x7f606c10d010 2021-04-09T18:30:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f60715a8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c0d8d30 con 0x7f606c10da00 2021-04-09T18:30:43.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f606aabe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f606c0d7ca0 0x7f606c0d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37576/0 (socket says 172.21.15.32:37576) 2021-04-09T18:30:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f606aabe700 1 -- 172.21.15.32:0/1951075916 learned_addr learned my addr 172.21.15.32:0/1951075916 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f606bac0700 1 --2- 172.21.15.32:0/1951075916 >> v2:172.21.15.86:3300/0 conn(0x7f606c10da00 0x7f606c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f606a2bd700 1 -- 172.21.15.32:0/1951075916 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551458883 0 0) 0x7f6060000ce0 con 0x7f606c0d7ca0 2021-04-09T18:30:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f606a2bd700 1 -- 172.21.15.32:0/1951075916 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6054002fe0 con 0x7f606c0d7ca0 2021-04-09T18:30:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f606a2bd700 1 -- 172.21.15.32:0/1951075916 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028939846 0 0) 0x7f605c000c60 con 0x7f606c10d010 2021-04-09T18:30:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.513 7f606a2bd700 1 -- 172.21.15.32:0/1951075916 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6054003210 con 0x7f606c10d010 2021-04-09T18:30:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.514 7f606bac0700 1 -- 172.21.15.32:0/1951075916 >> v1:172.21.15.86:6789/0 conn(0x7f606c10d010 legacy=0x7f606c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.514 7f606bac0700 1 -- 172.21.15.32:0/1951075916 >> v1:172.21.15.32:6789/0 conn(0x7f606c0d7ca0 legacy=0x7f606c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.514 7f606bac0700 1 -- 172.21.15.32:0/1951075916 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c0d9220 con 0x7f606c10da00 2021-04-09T18:30:43.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.514 7f606bac0700 1 --2- 172.21.15.32:0/1951075916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606c10da00 0x7f606c10de00 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f6064006e50 tx=0x7f6064009e70).ready entity=mon.1 client_cookie=5f6ec92046f32d62 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:43.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.514 7f606a2bd700 1 -- 172.21.15.32:0/1951075916 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6064006a20 con 0x7f606c10da00 2021-04-09T18:30:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.514 7f606a2bd700 1 -- 172.21.15.32:0/1951075916 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6064006790 con 0x7f606c10da00 2021-04-09T18:30:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.514 7f606a2bd700 1 -- 172.21.15.32:0/1951075916 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f606400b700 con 0x7f606c10da00 2021-04-09T18:30:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.514 7f60715a8700 1 -- 172.21.15.32:0/1951075916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606c10da00 msgr2=0x7f606c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.514 7f60715a8700 1 --2- 172.21.15.32:0/1951075916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606c10da00 0x7f606c10de00 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f6064006e50 tx=0x7f6064009e70).stop 2021-04-09T18:30:43.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.515 7f60715a8700 1 -- 172.21.15.32:0/1951075916 shutdown_connections 2021-04-09T18:30:43.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.515 7f60715a8700 1 --2- 172.21.15.32:0/1951075916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606c10da00 0x7f606c10de00 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.515 7f60715a8700 1 -- 172.21.15.32:0/1951075916 >> 172.21.15.32:0/1951075916 conn(0x7f606c0ce5f0 msgr2=0x7f606c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:43.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.515 7f60715a8700 1 -- 172.21.15.32:0/1951075916 shutdown_connections 2021-04-09T18:30:43.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.515 7f60715a8700 1 -- 172.21.15.32:0/1951075916 wait complete. 2021-04-09T18:30:43.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.535 7f60715a8700 1 Processor -- start 2021-04-09T18:30:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.535 7f60715a8700 1 -- start start 2021-04-09T18:30:43.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.535 7f60715a8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f606c193c00 0x7f606c179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.535 7f60715a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606c179c30 0x7f606c1b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.535 7f60715a8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606c1b8b60 0x7f606c17a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.535 7f60715a8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c17a060 con 0x7f606c179c30 2021-04-09T18:30:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.535 7f60715a8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c0c0de0 con 0x7f606c1b8b60 2021-04-09T18:30:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.535 7f60715a8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c194000 con 0x7f606c193c00 2021-04-09T18:30:43.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606aabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606c179c30 0x7f606c1b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606aabe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606c179c30 0x7f606c1b8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56222/0 (socket says 172.21.15.32:56222) 2021-04-09T18:30:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606aabe700 1 -- 172.21.15.32:0/1770581032 learned_addr learned my addr 172.21.15.32:0/1770581032 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606b2bf700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f606c193c00 0x7f606c179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606bac0700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606c1b8b60 0x7f606c17a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606aabe700 1 -- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f606c193c00 msgr2=0x7f606c179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606aabe700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f606c193c00 0x7f606c179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606aabe700 1 -- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606c1b8b60 msgr2=0x7f606c17a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606aabe700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606c1b8b60 0x7f606c17a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606aabe700 1 -- 172.21.15.32:0/1770581032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c194220 con 0x7f606c179c30 2021-04-09T18:30:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f606aabe700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606c179c30 0x7f606c1b8540 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f6060001400 tx=0x7f606000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f6053fff700 1 -- 172.21.15.32:0/1770581032 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6060000ef0 con 0x7f606c179c30 2021-04-09T18:30:43.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f6053fff700 1 -- 172.21.15.32:0/1770581032 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606000acf0 con 0x7f606c179c30 2021-04-09T18:30:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f60715a8700 1 -- 172.21.15.32:0/1770581032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c194220 con 0x7f606c179c30 2021-04-09T18:30:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f6053fff700 1 -- 172.21.15.32:0/1770581032 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6060004590 con 0x7f606c179c30 2021-04-09T18:30:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.536 7f60715a8700 1 -- 172.21.15.32:0/1770581032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c194220 con 0x7f606c179c30 2021-04-09T18:30:43.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.537 7f6053fff700 1 -- 172.21.15.32:0/1770581032 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f60600043a0 con 0x7f606c179c30 2021-04-09T18:30:43.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.537 7f6053fff700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f604c0188c0 0x7f604c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.537 7f6053fff700 1 -- 172.21.15.32:0/1770581032 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f6060032b40 con 0x7f606c179c30 2021-04-09T18:30:43.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.538 7f606b2bf700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f604c0188c0 0x7f604c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.538 7f606b2bf700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f604c0188c0 0x7f604c01adc0 crc :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:43.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.557 7f60715a8700 1 -- 172.21.15.32:0/1770581032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c10d480 con 0x7f606c179c30 2021-04-09T18:30:43.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.561 7f6053fff700 1 -- 172.21.15.32:0/1770581032 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f60600009b0 con 0x7f606c179c30 2021-04-09T18:30:43.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.739 7f60715a8700 1 -- 172.21.15.32:0/1770581032 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "bravo_data"} v 0) v1 -- 0x7f606c10d480 con 0x7f606c179c30 2021-04-09T18:30:43.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.739 7f6053fff700 1 -- 172.21.15.32:0/1770581032 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "bravo_data"}]=0 v188) v1 ==== 99+0+10 (secure 0 0 0) 0x7f606000acf0 con 0x7f606c179c30 2021-04-09T18:30:43.756 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:30:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 -- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f604c0188c0 msgr2=0x7f604c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f604c0188c0 0x7f604c01adc0 crc :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 -- 172.21.15.32:0/1770581032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606c179c30 msgr2=0x7f606c1b8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606c179c30 0x7f606c1b8540 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f6060001400 tx=0x7f606000a900).stop 2021-04-09T18:30:43.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 -- 172.21.15.32:0/1770581032 shutdown_connections 2021-04-09T18:30:43.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f604c0188c0 0x7f604c01adc0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f606c179c30 0x7f606c1b8540 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606c1b8b60 0x7f606c17a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 --2- 172.21.15.32:0/1770581032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f606c193c00 0x7f606c179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.741 7f60715a8700 1 -- 172.21.15.32:0/1770581032 >> 172.21.15.32:0/1770581032 conn(0x7f606c0bbda0 msgr2=0x7f606c1b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:43.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.742 7f60715a8700 1 -- 172.21.15.32:0/1770581032 shutdown_connections 2021-04-09T18:30:43.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.742 7f60715a8700 1 -- 172.21.15.32:0/1770581032 wait complete. 2021-04-09T18:30:43.767 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:30:43.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:43.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.900 7f991831d700 1 Processor -- start 2021-04-09T18:30:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.900 7f991831d700 1 -- start start 2021-04-09T18:30:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991831d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f99100d7ca0 0x7f99100d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991831d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f991010da00 0x7f991010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991831d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991010d480 con 0x7f991010d010 2021-04-09T18:30:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991831d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99100d9220 con 0x7f99100d7ca0 2021-04-09T18:30:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991831d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99100d86c0 con 0x7f991010da00 2021-04-09T18:30:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f9916139700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f991010d010 0x7f99100d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37586/0 (socket says 172.21.15.32:37586) 2021-04-09T18:30:43.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f9916139700 1 -- 172.21.15.32:0/3675908760 learned_addr learned my addr 172.21.15.32:0/3675908760 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991693a700 1 --2- 172.21.15.32:0/3675908760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f991010da00 0x7f991010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f9915938700 1 --2- 172.21.15.32:0/3675908760 >> v2:172.21.15.86:3300/0 conn(0x7f99100d7ca0 0x7f99100d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f9915137700 1 -- 172.21.15.32:0/3675908760 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187633450 0 0) 0x7f990c001060 con 0x7f991010d010 2021-04-09T18:30:43.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f9915137700 1 -- 172.21.15.32:0/3675908760 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98fc002fe0 con 0x7f991010d010 2021-04-09T18:30:43.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991693a700 1 -- 172.21.15.32:0/3675908760 >> v2:172.21.15.86:3300/0 conn(0x7f99100d7ca0 msgr2=0x7f99100d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:43.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991693a700 1 --2- 172.21.15.32:0/3675908760 >> v2:172.21.15.86:3300/0 conn(0x7f99100d7ca0 0x7f99100d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991693a700 1 -- 172.21.15.32:0/3675908760 >> v1:172.21.15.32:6789/0 conn(0x7f991010d010 legacy=0x7f99100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f991693a700 1 -- 172.21.15.32:0/3675908760 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99100d1f70 con 0x7f991010da00 2021-04-09T18:30:43.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.901 7f9915938700 1 --2- 172.21.15.32:0/3675908760 >> v2:172.21.15.86:3300/0 conn(0x7f99100d7ca0 0x7f99100d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:43.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.902 7f991693a700 1 --2- 172.21.15.32:0/3675908760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f991010da00 0x7f991010de00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9908006ad0 tx=0x7f9908009ad0).ready entity=mon.2 client_cookie=5547c3b45fe612c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:43.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.902 7f9915137700 1 -- 172.21.15.32:0/3675908760 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99080022e0 con 0x7f991010da00 2021-04-09T18:30:43.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.902 7f9915137700 1 -- 172.21.15.32:0/3675908760 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9908006410 con 0x7f991010da00 2021-04-09T18:30:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.902 7f9915137700 1 -- 172.21.15.32:0/3675908760 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9908000c10 con 0x7f991010da00 2021-04-09T18:30:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.902 7f991831d700 1 -- 172.21.15.32:0/3675908760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f991010da00 msgr2=0x7f991010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.902 7f991831d700 1 --2- 172.21.15.32:0/3675908760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f991010da00 0x7f991010de00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9908006ad0 tx=0x7f9908009ad0).stop 2021-04-09T18:30:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.903 7f991831d700 1 -- 172.21.15.32:0/3675908760 shutdown_connections 2021-04-09T18:30:43.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.903 7f991831d700 1 --2- 172.21.15.32:0/3675908760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f991010da00 0x7f991010de00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.903 7f991831d700 1 --2- 172.21.15.32:0/3675908760 >> v2:172.21.15.86:3300/0 conn(0x7f99100d7ca0 0x7f99100d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.903 7f991831d700 1 -- 172.21.15.32:0/3675908760 >> 172.21.15.32:0/3675908760 conn(0x7f99100ce5f0 msgr2=0x7f99100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.903 7f991831d700 1 -- 172.21.15.32:0/3675908760 shutdown_connections 2021-04-09T18:30:43.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.903 7f991831d700 1 -- 172.21.15.32:0/3675908760 wait complete. 2021-04-09T18:30:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.923 7f991831d700 1 Processor -- start 2021-04-09T18:30:43.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.923 7f991831d700 1 -- start start 2021-04-09T18:30:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.923 7f991831d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9910184a30 0x7f99101acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.923 7f991831d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99101a6b90 0x7f99101a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f991831d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99101a5150 0x7f99101a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f991831d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99101a75b0 con 0x7f9910184a30 2021-04-09T18:30:43.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f991831d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9910184590 con 0x7f99101a6b90 2021-04-09T18:30:43.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f991831d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99101a5b70 con 0x7f99101a5150 2021-04-09T18:30:43.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9916139700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9910184a30 0x7f99101acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9916139700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9910184a30 0x7f99101acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56234/0 (socket says 172.21.15.32:56234) 2021-04-09T18:30:43.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9916139700 1 -- 172.21.15.32:0/2823988051 learned_addr learned my addr 172.21.15.32:0/2823988051 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:43.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9915938700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99101a6b90 0x7f99101a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f991693a700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99101a5150 0x7f99101a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9916139700 1 -- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99101a5150 msgr2=0x7f99101a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9916139700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99101a5150 0x7f99101a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9916139700 1 -- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99101a6b90 msgr2=0x7f99101a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9916139700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99101a6b90 0x7f99101a6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9916139700 1 -- 172.21.15.32:0/2823988051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99101a5d90 con 0x7f9910184a30 2021-04-09T18:30:43.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9915938700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99101a6b90 0x7f99101a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9916139700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9910184a30 0x7f99101acb80 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 rx=0x7f990c00a880 tx=0x7f990c00ac20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9906ffd700 1 -- 172.21.15.32:0/2823988051 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f990c001270 con 0x7f9910184a30 2021-04-09T18:30:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9906ffd700 1 -- 172.21.15.32:0/2823988051 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c0070d0 con 0x7f9910184a30 2021-04-09T18:30:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f991831d700 1 -- 172.21.15.32:0/2823988051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99101a5d90 con 0x7f9910184a30 2021-04-09T18:30:43.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f9906ffd700 1 -- 172.21.15.32:0/2823988051 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f990c004720 con 0x7f9910184a30 2021-04-09T18:30:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.924 7f991831d700 1 -- 172.21.15.32:0/2823988051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99101a5d90 con 0x7f9910184a30 2021-04-09T18:30:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.925 7f9906ffd700 1 -- 172.21.15.32:0/2823988051 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f990c0072b0 con 0x7f9910184a30 2021-04-09T18:30:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.925 7f9906ffd700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98f8018930 0x7f98f801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:43.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.925 7f9906ffd700 1 -- 172.21.15.32:0/2823988051 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f990c033130 con 0x7f9910184a30 2021-04-09T18:30:43.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.925 7f9915938700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98f8018930 0x7f98f801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:43.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.925 7f9915938700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98f8018930 0x7f98f801ae10 crc :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:43.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.945 7f991831d700 1 -- 172.21.15.32:0/2823988051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99101b7080 con 0x7f9910184a30 2021-04-09T18:30:43.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:43.948 7f9906ffd700 1 -- 172.21.15.32:0/2823988051 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f990c007cb0 con 0x7f9910184a30 2021-04-09T18:30:44.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.140 7f991831d700 1 -- 172.21.15.32:0/2823988051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99101b7080 con 0x7f9910184a30 2021-04-09T18:30:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.140 7f9906ffd700 1 -- 172.21.15.32:0/2823988051 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 334 v334) v1 ==== 95+0+3717 (secure 0 0 0) 0x7f990c0070d0 con 0x7f9910184a30 2021-04-09T18:30:44.156 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 334 2021-04-09T18:30:44.157 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:44.157 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":334,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58739,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330},{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":186,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58736":{"gid":58736,"name":"a","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/558658679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":558658679},{"type":"v1","addr":"172.21.15.32:6834","nonce":558658679}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:44.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 -- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98f8018930 msgr2=0x7f98f801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:44.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98f8018930 0x7f98f801ae10 crc :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:44.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 -- 172.21.15.32:0/2823988051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9910184a30 msgr2=0x7f99101acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:44.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9910184a30 0x7f99101acb80 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 rx=0x7f990c00a880 tx=0x7f990c00ac20).stop 2021-04-09T18:30:44.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 -- 172.21.15.32:0/2823988051 shutdown_connections 2021-04-09T18:30:44.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f98f8018930 0x7f98f801ae10 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:44.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99101a5150 0x7f99101a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:44.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99101a6b90 0x7f99101a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:44.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 --2- 172.21.15.32:0/2823988051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9910184a30 0x7f99101acb80 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:44.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 -- 172.21.15.32:0/2823988051 >> 172.21.15.32:0/2823988051 conn(0x7f9910191560 msgr2=0x7f99101b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:44.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 -- 172.21.15.32:0/2823988051 shutdown_connections 2021-04-09T18:30:44.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:44.143 7f991831d700 1 -- 172.21.15.32:0/2823988051 wait complete. 2021-04-09T18:30:44.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 334, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:15.764321', 'modified': '2021-04-09 18:30:33.326504', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 185, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11136}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11136': {'gid': 11136, 'name': 'd', 'rank': 0, 'incarnation': 331, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/264493238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 264493238}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 264493238}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:44.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:30:44.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:30:44.702 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:30:44.703 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:30:44.703 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:30:44.704 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 11136, 2021-04-09T18:30:44.704 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:30:44.705 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:30:44.705 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 14.372243535999999, 2021-04-09T18:30:44.706 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 334, 2021-04-09T18:30:44.706 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 188, 2021-04-09T18:30:44.706 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 188, 2021-04-09T18:30:44.707 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 18.413199095 2021-04-09T18:30:44.707 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:30:44.709 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 11136, "want_state": "up:active", "state": "up:active", "rank_uptime": 14.372243535999999, "mdsmap_epoch": 334, "osdmap_epoch": 188, "osdmap_epoch_barrier": 188, "uptime": 18.413199095 } 2021-04-09T18:30:44.710 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:30:44.711 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-04-09T18:30:44.711 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi086.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-04-09T18:30:44.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:44.713 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:44.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo modprobe fuse 2021-04-09T18:30:44.905 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:30:44.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:30:44.974 INFO:teuthology.orchestra.run.smithi086.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:30:44.974 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:30:44.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:44.976 DEBUG:teuthology.orchestra.run.smithi086:> ls /sys/fs/fuse/connections 2021-04-09T18:30:45.019 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:30:45.019 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mds_namespace=bravo --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:45.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo modprobe fuse 2021-04-09T18:30:45.115 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:30:45.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:30:45.169 INFO:teuthology.orchestra.run.smithi086.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:30:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:30:45.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:45.173 DEBUG:teuthology.orchestra.run.smithi086:> ls /sys/fs/fuse/connections 2021-04-09T18:30:45.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi086.stderr:2021-04-09 18:30:45.193 7febd23a7f80 -1 init, newargv = 0x5566d1058390 newargc=9 2021-04-09T18:30:45.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi086.stderr:ceph-fuse[31844]: starting ceph client 2021-04-09T18:30:45.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi086.stderr:ceph-fuse[31844]: starting fuse 2021-04-09T18:30:45.217 INFO:teuthology.orchestra.run.smithi086.stdout:42 2021-04-09T18:30:45.217 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:30:45.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:45.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo python3 -c ' 2021-04-09T18:30:45.220 DEBUG:teuthology.orchestra.run.smithi086:> import glob 2021-04-09T18:30:45.220 DEBUG:teuthology.orchestra.run.smithi086:> import re 2021-04-09T18:30:45.221 DEBUG:teuthology.orchestra.run.smithi086:> import os 2021-04-09T18:30:45.221 DEBUG:teuthology.orchestra.run.smithi086:> import subprocess 2021-04-09T18:30:45.222 DEBUG:teuthology.orchestra.run.smithi086:> 2021-04-09T18:30:45.223 DEBUG:teuthology.orchestra.run.smithi086:> def find_socket(client_name): 2021-04-09T18:30:45.224 DEBUG:teuthology.orchestra.run.smithi086:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-04-09T18:30:45.225 DEBUG:teuthology.orchestra.run.smithi086:> files = glob.glob(asok_path) 2021-04-09T18:30:45.225 DEBUG:teuthology.orchestra.run.smithi086:> 2021-04-09T18:30:45.226 DEBUG:teuthology.orchestra.run.smithi086:> # Given a non-glob path, it better be there 2021-04-09T18:30:45.227 DEBUG:teuthology.orchestra.run.smithi086:> if "*" not in asok_path: 2021-04-09T18:30:45.227 DEBUG:teuthology.orchestra.run.smithi086:> assert(len(files) == 1) 2021-04-09T18:30:45.228 DEBUG:teuthology.orchestra.run.smithi086:> return files[0] 2021-04-09T18:30:45.229 DEBUG:teuthology.orchestra.run.smithi086:> 2021-04-09T18:30:45.229 DEBUG:teuthology.orchestra.run.smithi086:> for f in files: 2021-04-09T18:30:45.230 DEBUG:teuthology.orchestra.run.smithi086:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:30:45.231 DEBUG:teuthology.orchestra.run.smithi086:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:30:45.231 DEBUG:teuthology.orchestra.run.smithi086:> return f 2021-04-09T18:30:45.232 DEBUG:teuthology.orchestra.run.smithi086:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:30:45.233 DEBUG:teuthology.orchestra.run.smithi086:> 2021-04-09T18:30:45.233 DEBUG:teuthology.orchestra.run.smithi086:> print(find_socket("client.1")) 2021-04-09T18:30:45.234 DEBUG:teuthology.orchestra.run.smithi086:> ' 2021-04-09T18:30:45.342 INFO:teuthology.orchestra.run.smithi086.stdout:/var/run/ceph/ceph-client.1.31844.asok 2021-04-09T18:30:45.346 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.31844.asok 2021-04-09T18:30:45.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:45.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.31844.asok status 2021-04-09T18:30:45.462 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:30:45.462 INFO:teuthology.orchestra.run.smithi086.stdout: "metadata": { 2021-04-09T18:30:45.463 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:30:45.463 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:30:45.463 INFO:teuthology.orchestra.run.smithi086.stdout: "entity_id": "1", 2021-04-09T18:30:45.464 INFO:teuthology.orchestra.run.smithi086.stdout: "hostname": "smithi086", 2021-04-09T18:30:45.464 INFO:teuthology.orchestra.run.smithi086.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-04-09T18:30:45.464 INFO:teuthology.orchestra.run.smithi086.stdout: "pid": "31844", 2021-04-09T18:30:45.464 INFO:teuthology.orchestra.run.smithi086.stdout: "root": "/" 2021-04-09T18:30:45.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2021-04-09T18:30:45.465 INFO:teuthology.orchestra.run.smithi086.stdout: "dentry_count": 0, 2021-04-09T18:30:45.465 INFO:teuthology.orchestra.run.smithi086.stdout: "dentry_pinned_count": 0, 2021-04-09T18:30:45.465 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 58853, 2021-04-09T18:30:45.465 INFO:teuthology.orchestra.run.smithi086.stdout: "inst": { 2021-04-09T18:30:45.466 INFO:teuthology.orchestra.run.smithi086.stdout: "name": { 2021-04-09T18:30:45.466 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "client", 2021-04-09T18:30:45.466 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 58853 2021-04-09T18:30:45.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2021-04-09T18:30:45.466 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": { 2021-04-09T18:30:45.467 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2021-04-09T18:30:45.467 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:0", 2021-04-09T18:30:45.467 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 438438429 2021-04-09T18:30:45.467 INFO:teuthology.orchestra.run.smithi086.stdout: } 2021-04-09T18:30:45.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2021-04-09T18:30:45.468 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": { 2021-04-09T18:30:45.468 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2021-04-09T18:30:45.468 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:0", 2021-04-09T18:30:45.468 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 438438429 2021-04-09T18:30:45.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2021-04-09T18:30:45.469 INFO:teuthology.orchestra.run.smithi086.stdout: "inst_str": "client.58853 172.21.15.86:0/438438429", 2021-04-09T18:30:45.469 INFO:teuthology.orchestra.run.smithi086.stdout: "addr_str": "172.21.15.86:0/438438429", 2021-04-09T18:30:45.469 INFO:teuthology.orchestra.run.smithi086.stdout: "inode_count": 1, 2021-04-09T18:30:45.469 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_epoch": 334, 2021-04-09T18:30:45.470 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_epoch": 188, 2021-04-09T18:30:45.470 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:30:45.470 INFO:teuthology.orchestra.run.smithi086.stdout: "blacklisted": false, 2021-04-09T18:30:45.470 INFO:teuthology.orchestra.run.smithi086.stdout: "fs_name": "bravo" 2021-04-09T18:30:45.470 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:30:45.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:45.472 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:45.473 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:45.515 INFO:teuthology.orchestra.run.smithi086.stdout:fuseblk 2021-04-09T18:30:45.515 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-04-09T18:30:45.516 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=test.bin bs=1M conv=fdatasync count=1 seek=0 2021-04-09T18:30:45.630 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2021-04-09T18:30:45.631 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2021-04-09T18:30:45.631 INFO:teuthology.orchestra.run.smithi086.stderr:1048576 bytes (1.0 MB) copied, 0.0198398 s, 52.9 MB/s 2021-04-09T18:30:45.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2021-04-09T18:30:45.633 DEBUG:teuthology.orchestra.run.smithi086:> import os 2021-04-09T18:30:45.633 DEBUG:teuthology.orchestra.run.smithi086:> import stat 2021-04-09T18:30:45.633 DEBUG:teuthology.orchestra.run.smithi086:> 2021-04-09T18:30:45.634 DEBUG:teuthology.orchestra.run.smithi086:> print(os.stat("/home/ubuntu/cephtest/mnt.1/test.bin").st_ino) 2021-04-09T18:30:45.634 DEBUG:teuthology.orchestra.run.smithi086:> ' 2021-04-09T18:30:45.732 INFO:teuthology.orchestra.run.smithi086.stdout:1099511627776 2021-04-09T18:30:45.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:45.966 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:45.966 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:46.028 INFO:teuthology.orchestra.run.smithi086.stdout:fuseblk 2021-04-09T18:30:46.028 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-04-09T18:30:46.029 INFO:tasks.cephfs.mount:Creating file a 2021-04-09T18:30:46.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch /home/ubuntu/cephtest/mnt.1/a 2021-04-09T18:30:46.130 INFO:tasks.cephfs.mount:Creating file b 2021-04-09T18:30:46.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch /home/ubuntu/cephtest/mnt.1/b 2021-04-09T18:30:46.199 INFO:tasks.cephfs.mount:Creating file c 2021-04-09T18:30:46.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch /home/ubuntu/cephtest/mnt.1/c 2021-04-09T18:30:46.262 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:46.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.410 7f5b42200700 1 Processor -- start 2021-04-09T18:30:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.410 7f5b42200700 1 -- start start 2021-04-09T18:30:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b42200700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c0d7ca0 0x7f5b3c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b42200700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5b3c10da00 0x7f5b3c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b42200700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3c10d480 con 0x7f5b3c10d010 2021-04-09T18:30:46.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b42200700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b3c0d9220 con 0x7f5b3c10da00 2021-04-09T18:30:46.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b42200700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c0d86c0 con 0x7f5b3c0d7ca0 2021-04-09T18:30:46.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b3bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5b3c10d010 0x7f5b3c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37600/0 (socket says 172.21.15.32:37600) 2021-04-09T18:30:46.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b3bfff700 1 -- 172.21.15.32:0/2219894753 learned_addr learned my addr 172.21.15.32:0/2219894753 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:46.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b33fff700 1 --2- 172.21.15.32:0/2219894753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c0d7ca0 0x7f5b3c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b4081d700 1 --2- 172.21.15.32:0/2219894753 >> v2:172.21.15.86:3300/0 conn(0x7f5b3c10da00 0x7f5b3c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b3b7fe700 1 -- 172.21.15.32:0/2219894753 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132583548 0 0) 0x7f5b2c000ce0 con 0x7f5b3c10d010 2021-04-09T18:30:46.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b3b7fe700 1 -- 172.21.15.32:0/2219894753 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b28002fe0 con 0x7f5b3c10d010 2021-04-09T18:30:46.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b3b7fe700 1 -- 172.21.15.32:0/2219894753 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1032250322 0 0) 0x7f5b2c000ed0 con 0x7f5b3c10d010 2021-04-09T18:30:46.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b33fff700 1 -- 172.21.15.32:0/2219894753 >> v2:172.21.15.86:3300/0 conn(0x7f5b3c10da00 msgr2=0x7f5b3c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:46.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b33fff700 1 --2- 172.21.15.32:0/2219894753 >> v2:172.21.15.86:3300/0 conn(0x7f5b3c10da00 0x7f5b3c10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b33fff700 1 -- 172.21.15.32:0/2219894753 >> v1:172.21.15.32:6789/0 conn(0x7f5b3c10d010 legacy=0x7f5b3c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b33fff700 1 -- 172.21.15.32:0/2219894753 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3c0d1f70 con 0x7f5b3c0d7ca0 2021-04-09T18:30:46.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.411 7f5b4081d700 1 --2- 172.21.15.32:0/2219894753 >> v2:172.21.15.86:3300/0 conn(0x7f5b3c10da00 0x7f5b3c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:46.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.412 7f5b33fff700 1 --2- 172.21.15.32:0/2219894753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c0d7ca0 0x7f5b3c0d80a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f5b24006650 tx=0x7f5b24009a80).ready entity=mon.2 client_cookie=e870dbf7f1333e40 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:46.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.412 7f5b3b7fe700 1 -- 172.21.15.32:0/2219894753 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b240009b0 con 0x7f5b3c0d7ca0 2021-04-09T18:30:46.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.412 7f5b3b7fe700 1 -- 172.21.15.32:0/2219894753 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2400c0c0 con 0x7f5b3c0d7ca0 2021-04-09T18:30:46.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.412 7f5b3b7fe700 1 -- 172.21.15.32:0/2219894753 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b2400c510 con 0x7f5b3c0d7ca0 2021-04-09T18:30:46.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.412 7f5b42200700 1 -- 172.21.15.32:0/2219894753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c0d7ca0 msgr2=0x7f5b3c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.412 7f5b42200700 1 --2- 172.21.15.32:0/2219894753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c0d7ca0 0x7f5b3c0d80a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f5b24006650 tx=0x7f5b24009a80).stop 2021-04-09T18:30:46.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.413 7f5b42200700 1 -- 172.21.15.32:0/2219894753 shutdown_connections 2021-04-09T18:30:46.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.413 7f5b42200700 1 --2- 172.21.15.32:0/2219894753 >> v2:172.21.15.86:3300/0 conn(0x7f5b3c10da00 0x7f5b3c10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.413 7f5b42200700 1 --2- 172.21.15.32:0/2219894753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c0d7ca0 0x7f5b3c0d80a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.413 7f5b42200700 1 -- 172.21.15.32:0/2219894753 >> 172.21.15.32:0/2219894753 conn(0x7f5b3c0ce5f0 msgr2=0x7f5b3c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:46.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.413 7f5b42200700 1 -- 172.21.15.32:0/2219894753 shutdown_connections 2021-04-09T18:30:46.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.413 7f5b42200700 1 -- 172.21.15.32:0/2219894753 wait complete. 2021-04-09T18:30:46.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.433 7f5b42200700 1 Processor -- start 2021-04-09T18:30:46.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.433 7f5b42200700 1 -- start start 2021-04-09T18:30:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.433 7f5b42200700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c179bf0 0x7f5b3c179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.433 7f5b42200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3c17c540 0x7f5b3c17c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.433 7f5b42200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3c0aca60 0x7f5b3c0ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.433 7f5b42200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c17cf60 con 0x7f5b3c17c540 2021-04-09T18:30:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.433 7f5b42200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c0ad480 con 0x7f5b3c0aca60 2021-04-09T18:30:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.433 7f5b42200700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c0ad6a0 con 0x7f5b3c179bf0 2021-04-09T18:30:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b33fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3c17c540 0x7f5b3c17c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b33fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3c17c540 0x7f5b3c17c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56250/0 (socket says 172.21.15.32:56250) 2021-04-09T18:30:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b33fff700 1 -- 172.21.15.32:0/1172519149 learned_addr learned my addr 172.21.15.32:0/1172519149 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b3bfff700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c179bf0 0x7f5b3c179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b4081d700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3c0aca60 0x7f5b3c0ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b33fff700 1 -- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c179bf0 msgr2=0x7f5b3c179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b33fff700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c179bf0 0x7f5b3c179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b33fff700 1 -- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3c0aca60 msgr2=0x7f5b3c0ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b33fff700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3c0aca60 0x7f5b3c0ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b33fff700 1 -- 172.21.15.32:0/1172519149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3c0ad8c0 con 0x7f5b3c17c540 2021-04-09T18:30:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b3bfff700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c179bf0 0x7f5b3c179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b4081d700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3c0aca60 0x7f5b3c0ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:46.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b33fff700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3c17c540 0x7f5b3c17c940 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7f5b24009650 tx=0x7f5b2400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:46.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b397fa700 1 -- 172.21.15.32:0/1172519149 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b240093a0 con 0x7f5b3c17c540 2021-04-09T18:30:46.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b397fa700 1 -- 172.21.15.32:0/1172519149 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2400bd50 con 0x7f5b3c17c540 2021-04-09T18:30:46.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b42200700 1 -- 172.21.15.32:0/1172519149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3c0ad8c0 con 0x7f5b3c17c540 2021-04-09T18:30:46.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b397fa700 1 -- 172.21.15.32:0/1172519149 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b240052e0 con 0x7f5b3c17c540 2021-04-09T18:30:46.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.434 7f5b42200700 1 -- 172.21.15.32:0/1172519149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3c0ad8c0 con 0x7f5b3c17c540 2021-04-09T18:30:46.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.435 7f5b397fa700 1 -- 172.21.15.32:0/1172519149 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5b24006030 con 0x7f5b3c17c540 2021-04-09T18:30:46.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.435 7f5b397fa700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b200188c0 0x7f5b2001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.435 7f5b397fa700 1 -- 172.21.15.32:0/1172519149 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f5b24032b60 con 0x7f5b3c17c540 2021-04-09T18:30:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.435 7f5b3bfff700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b200188c0 0x7f5b2001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.436 7f5b3bfff700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b200188c0 0x7f5b2001adc0 crc :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:46.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.455 7f5b42200700 1 -- 172.21.15.32:0/1172519149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b3c17cf60 con 0x7f5b3c17c540 2021-04-09T18:30:46.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.458 7f5b397fa700 1 -- 172.21.15.32:0/1172519149 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5b240009c0 con 0x7f5b3c17c540 2021-04-09T18:30:46.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.649 7f5b42200700 1 -- 172.21.15.32:0/1172519149 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b3c17cf60 con 0x7f5b3c17c540 2021-04-09T18:30:46.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.650 7f5b397fa700 1 -- 172.21.15.32:0/1172519149 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 334 v334) v1 ==== 95+0+3717 (secure 0 0 0) 0x7f5b24006030 con 0x7f5b3c17c540 2021-04-09T18:30:46.666 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 334 2021-04-09T18:30:46.667 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:46.667 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":334,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58739,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330},{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":186,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58736":{"gid":58736,"name":"a","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/558658679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":558658679},{"type":"v1","addr":"172.21.15.32:6834","nonce":558658679}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:46.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 -- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b200188c0 msgr2=0x7f5b2001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b200188c0 0x7f5b2001adc0 crc :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 -- 172.21.15.32:0/1172519149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3c17c540 msgr2=0x7f5b3c17c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3c17c540 0x7f5b3c17c940 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7f5b24009650 tx=0x7f5b2400b960).stop 2021-04-09T18:30:46.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 -- 172.21.15.32:0/1172519149 shutdown_connections 2021-04-09T18:30:46.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5b200188c0 0x7f5b2001adc0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b3c17c540 0x7f5b3c17c940 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b3c0aca60 0x7f5b3c0ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 --2- 172.21.15.32:0/1172519149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b3c179bf0 0x7f5b3c179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 -- 172.21.15.32:0/1172519149 >> 172.21.15.32:0/1172519149 conn(0x7f5b3c1acae0 msgr2=0x7f5b3c17ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:46.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 -- 172.21.15.32:0/1172519149 shutdown_connections 2021-04-09T18:30:46.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.652 7f5b42200700 1 -- 172.21.15.32:0/1172519149 wait complete. 2021-04-09T18:30:46.678 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:30:46.679 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:30:46.686 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:30:46.687 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2021-04-09T18:30:46.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.866 7f15bf664700 1 Processor -- start 2021-04-09T18:30:46.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.866 7f15bf664700 1 -- start start 2021-04-09T18:30:46.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bf664700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f15b80cffb0 0x7f15b80d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bf664700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f15b80d7810 0x7f15b80d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bf664700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b80d7290 con 0x7f15b80d8230 2021-04-09T18:30:46.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bf664700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b80d91f0 con 0x7f15b80d7810 2021-04-09T18:30:46.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bf664700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b810dff0 con 0x7f15b80cffb0 2021-04-09T18:30:46.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bcc7f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f15b80d7810 0x7f15b80d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bcc7f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f15b80d7810 0x7f15b80d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56260/0 (socket says 172.21.15.32:56260) 2021-04-09T18:30:46.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bcc7f700 1 -- 172.21.15.32:0/3148706543 learned_addr learned my addr 172.21.15.32:0/3148706543 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:46.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bd480700 1 --2- 172.21.15.32:0/3148706543 >> v2:172.21.15.86:3300/0 conn(0x7f15b80cffb0 0x7f15b80d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bcc7f700 1 -- 172.21.15.32:0/3148706543 >> v2:172.21.15.86:3300/0 conn(0x7f15b80cffb0 msgr2=0x7f15b80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:46.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bcc7f700 1 --2- 172.21.15.32:0/3148706543 >> v2:172.21.15.86:3300/0 conn(0x7f15b80cffb0 0x7f15b80d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bcc7f700 1 -- 172.21.15.32:0/3148706543 >> v1:172.21.15.86:6789/0 conn(0x7f15b80d8230 legacy=0x7f15b80d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bcc7f700 1 -- 172.21.15.32:0/3148706543 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b810e210 con 0x7f15b80d7810 2021-04-09T18:30:46.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15bcc7f700 1 --2- 172.21.15.32:0/3148706543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80d7810 0x7f15b80d7c10 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7f15a8006700 tx=0x7f15a8009b80).ready entity=mon.0 client_cookie=cd2e9a93a8958c17 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:46.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.867 7f15affff700 1 -- 172.21.15.32:0/3148706543 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a800c0c0 con 0x7f15b80d7810 2021-04-09T18:30:46.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.868 7f15affff700 1 -- 172.21.15.32:0/3148706543 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a800c440 con 0x7f15b80d7810 2021-04-09T18:30:46.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.868 7f15affff700 1 -- 172.21.15.32:0/3148706543 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a800c920 con 0x7f15b80d7810 2021-04-09T18:30:46.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.868 7f15bf664700 1 -- 172.21.15.32:0/3148706543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80d7810 msgr2=0x7f15b80d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.868 7f15bf664700 1 --2- 172.21.15.32:0/3148706543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80d7810 0x7f15b80d7c10 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7f15a8006700 tx=0x7f15a8009b80).stop 2021-04-09T18:30:46.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.868 7f15bf664700 1 -- 172.21.15.32:0/3148706543 shutdown_connections 2021-04-09T18:30:46.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.868 7f15bf664700 1 --2- 172.21.15.32:0/3148706543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b80d7810 0x7f15b80d7c10 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.868 7f15bf664700 1 --2- 172.21.15.32:0/3148706543 >> v2:172.21.15.86:3300/0 conn(0x7f15b80cffb0 0x7f15b80d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.868 7f15bf664700 1 -- 172.21.15.32:0/3148706543 >> 172.21.15.32:0/3148706543 conn(0x7f15b80ce5d0 msgr2=0x7f15b80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.869 7f15bf664700 1 -- 172.21.15.32:0/3148706543 shutdown_connections 2021-04-09T18:30:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.869 7f15bf664700 1 -- 172.21.15.32:0/3148706543 wait complete. 2021-04-09T18:30:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.889 7f15bf664700 1 Processor -- start 2021-04-09T18:30:46.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.889 7f15bf664700 1 -- start start 2021-04-09T18:30:46.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.889 7f15bf664700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15b80b8470 0x7f15b80b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.889 7f15bf664700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15b818f050 0x7f15b818f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.889 7f15bf664700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b8190150 0x7f15b8190550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.889 7f15bf664700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8193110 con 0x7f15b8190150 2021-04-09T18:30:46.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.889 7f15bf664700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b818fbd0 con 0x7f15b818f050 2021-04-09T18:30:46.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.889 7f15bf664700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8190b70 con 0x7f15b80b8470 2021-04-09T18:30:46.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.889 7f15bdc81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b8190150 0x7f15b8190550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bdc81700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b8190150 0x7f15b8190550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56264/0 (socket says 172.21.15.32:56264) 2021-04-09T18:30:46.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bdc81700 1 -- 172.21.15.32:0/3556737434 learned_addr learned my addr 172.21.15.32:0/3556737434 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:46.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bcc7f700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15b818f050 0x7f15b818f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bd480700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15b80b8470 0x7f15b80b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bdc81700 1 -- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15b80b8470 msgr2=0x7f15b80b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bdc81700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15b80b8470 0x7f15b80b8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bdc81700 1 -- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15b818f050 msgr2=0x7f15b818f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:46.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bdc81700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15b818f050 0x7f15b818f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:46.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bdc81700 1 -- 172.21.15.32:0/3556737434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b81a9e60 con 0x7f15b8190150 2021-04-09T18:30:46.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bdc81700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b8190150 0x7f15b8190550 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7f15b0009940 tx=0x7f15b000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15adffb700 1 -- 172.21.15.32:0/3556737434 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15b0000f40 con 0x7f15b8190150 2021-04-09T18:30:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15adffb700 1 -- 172.21.15.32:0/3556737434 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b000c2f0 con 0x7f15b8190150 2021-04-09T18:30:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15adffb700 1 -- 172.21.15.32:0/3556737434 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15b000c7d0 con 0x7f15b8190150 2021-04-09T18:30:46.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bf664700 1 -- 172.21.15.32:0/3556737434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b81a9e60 con 0x7f15b8190150 2021-04-09T18:30:46.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.890 7f15bf664700 1 -- 172.21.15.32:0/3556737434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b81a9e60 con 0x7f15b8190150 2021-04-09T18:30:46.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.891 7f15adffb700 1 -- 172.21.15.32:0/3556737434 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f15b0001220 con 0x7f15b8190150 2021-04-09T18:30:46.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.891 7f15adffb700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f15980188d0 0x7f159801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:46.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.891 7f15adffb700 1 -- 172.21.15.32:0/3556737434 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f15b0033160 con 0x7f15b8190150 2021-04-09T18:30:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.891 7f15bd480700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f15980188d0 0x7f159801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:46.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.891 7f15bd480700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f15980188d0 0x7f159801add0 crc :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:46.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.911 7f15bf664700 1 -- 172.21.15.32:0/3556737434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b8182e10 con 0x7f15b8190150 2021-04-09T18:30:46.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:46.915 7f15adffb700 1 -- 172.21.15.32:0/3556737434 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f15b000be30 con 0x7f15b8190150 2021-04-09T18:30:47.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.107 7f15bf664700 1 -- 172.21.15.32:0/3556737434 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f15b80d7290 con 0x7f15b8190150 2021-04-09T18:30:47.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.616 7f15adffb700 1 -- 172.21.15.32:0/3556737434 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 58736 v335) v1 ==== 96+0+0 (secure 0 0 0) 0x7f15b0001220 con 0x7f15b8190150 2021-04-09T18:30:47.637 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 58736 2021-04-09T18:30:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 -- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f15980188d0 msgr2=0x7f159801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f15980188d0 0x7f159801add0 crc :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 -- 172.21.15.32:0/3556737434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b8190150 msgr2=0x7f15b8190550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b8190150 0x7f15b8190550 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7f15b0009940 tx=0x7f15b000bcf0).stop 2021-04-09T18:30:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 -- 172.21.15.32:0/3556737434 shutdown_connections 2021-04-09T18:30:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f15980188d0 0x7f159801add0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f15b8190150 0x7f15b8190550 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15b818f050 0x7f15b818f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 --2- 172.21.15.32:0/3556737434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15b80b8470 0x7f15b80b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 -- 172.21.15.32:0/3556737434 >> 172.21.15.32:0/3556737434 conn(0x7f15b8183890 msgr2=0x7f15b819b900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 -- 172.21.15.32:0/3556737434 shutdown_connections 2021-04-09T18:30:47.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.625 7f15bf664700 1 -- 172.21.15.32:0/3556737434 wait complete. 2021-04-09T18:30:47.652 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:47.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.788 7f2d917de700 1 Processor -- start 2021-04-09T18:30:47.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.788 7f2d917de700 1 -- start start 2021-04-09T18:30:47.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d917de700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0d7ca0 0x7f2d8c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:47.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d917de700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2d8c10da00 0x7f2d8c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:47.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d917de700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c10d480 con 0x7f2d8c10d010 2021-04-09T18:30:47.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d917de700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8c0d9220 con 0x7f2d8c10da00 2021-04-09T18:30:47.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d917de700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0d09c0 con 0x7f2d8c0d7ca0 2021-04-09T18:30:47.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8bd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2d8c10da00 0x7f2d8c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:47.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8b574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2d8c10d010 0x7f2d8c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37636/0 (socket says 172.21.15.32:37636) 2021-04-09T18:30:47.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8b574700 1 -- 172.21.15.32:0/4281495933 learned_addr learned my addr 172.21.15.32:0/4281495933 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:47.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8ad73700 1 --2- 172.21.15.32:0/4281495933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0d7ca0 0x7f2d8c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:47.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8bd75700 1 -- 172.21.15.32:0/4281495933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0d7ca0 msgr2=0x7f2d8c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:47.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8bd75700 1 --2- 172.21.15.32:0/4281495933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0d7ca0 0x7f2d8c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8bd75700 1 -- 172.21.15.32:0/4281495933 >> v1:172.21.15.32:6789/0 conn(0x7f2d8c10d010 legacy=0x7f2d8c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:47.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8a572700 1 -- 172.21.15.32:0/4281495933 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132247791 0 0) 0x7f2d7c000ce0 con 0x7f2d8c10d010 2021-04-09T18:30:47.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8bd75700 1 -- 172.21.15.32:0/4281495933 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0d86c0 con 0x7f2d8c10da00 2021-04-09T18:30:47.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8bd75700 1 --2- 172.21.15.32:0/4281495933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c10da00 0x7f2d8c10de00 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 rx=0x7f2d84006e30 tx=0x7f2d84009de0).ready entity=mon.0 client_cookie=631e2eae989e77db server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:47.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8a572700 1 -- 172.21.15.32:0/4281495933 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d8400b300 con 0x7f2d8c10da00 2021-04-09T18:30:47.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8a572700 1 -- 172.21.15.32:0/4281495933 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8400b680 con 0x7f2d8c10da00 2021-04-09T18:30:47.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d8a572700 1 -- 172.21.15.32:0/4281495933 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d8400bb60 con 0x7f2d8c10da00 2021-04-09T18:30:47.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.789 7f2d917de700 1 -- 172.21.15.32:0/4281495933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c10da00 msgr2=0x7f2d8c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:47.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.790 7f2d917de700 1 --2- 172.21.15.32:0/4281495933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c10da00 0x7f2d8c10de00 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 rx=0x7f2d84006e30 tx=0x7f2d84009de0).stop 2021-04-09T18:30:47.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.790 7f2d917de700 1 -- 172.21.15.32:0/4281495933 shutdown_connections 2021-04-09T18:30:47.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.790 7f2d917de700 1 --2- 172.21.15.32:0/4281495933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c10da00 0x7f2d8c10de00 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.790 7f2d917de700 1 --2- 172.21.15.32:0/4281495933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0d7ca0 0x7f2d8c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.790 7f2d917de700 1 -- 172.21.15.32:0/4281495933 >> 172.21.15.32:0/4281495933 conn(0x7f2d8c0ce5f0 msgr2=0x7f2d8c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:47.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.790 7f2d917de700 1 -- 172.21.15.32:0/4281495933 shutdown_connections 2021-04-09T18:30:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.790 7f2d917de700 1 -- 172.21.15.32:0/4281495933 wait complete. 2021-04-09T18:30:47.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.810 7f2d917de700 1 Processor -- start 2021-04-09T18:30:47.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.810 7f2d917de700 1 -- start start 2021-04-09T18:30:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d917de700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d8c1a5210 0x7f2d8c1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d917de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c1a6340 0x7f2d8c1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d917de700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0abaa0 0x7f2d8c0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:47.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d917de700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c1a6d60 con 0x7f2d8c1a6340 2021-04-09T18:30:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d917de700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0ac4c0 con 0x7f2d8c1a5210 2021-04-09T18:30:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d917de700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0ac6e0 con 0x7f2d8c0abaa0 2021-04-09T18:30:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8ad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c1a6340 0x7f2d8c1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:47.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8ad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c1a6340 0x7f2d8c1a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56286/0 (socket says 172.21.15.32:56286) 2021-04-09T18:30:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8ad73700 1 -- 172.21.15.32:0/1903415307 learned_addr learned my addr 172.21.15.32:0/1903415307 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8bd75700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0abaa0 0x7f2d8c0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8b574700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d8c1a5210 0x7f2d8c1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:47.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8ad73700 1 -- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0abaa0 msgr2=0x7f2d8c0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8ad73700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0abaa0 0x7f2d8c0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8ad73700 1 -- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d8c1a5210 msgr2=0x7f2d8c1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8ad73700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d8c1a5210 0x7f2d8c1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8ad73700 1 -- 172.21.15.32:0/1903415307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0ac890 con 0x7f2d8c1a6340 2021-04-09T18:30:47.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.811 7f2d8ad73700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c1a6340 0x7f2d8c1a6740 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7f2d80006be0 tx=0x7f2d8000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.812 7f2d73fff700 1 -- 172.21.15.32:0/1903415307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d800064d0 con 0x7f2d8c1a6340 2021-04-09T18:30:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.812 7f2d73fff700 1 -- 172.21.15.32:0/1903415307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8000bd80 con 0x7f2d8c1a6340 2021-04-09T18:30:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.812 7f2d917de700 1 -- 172.21.15.32:0/1903415307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c0ac6e0 con 0x7f2d8c1a6340 2021-04-09T18:30:47.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.812 7f2d73fff700 1 -- 172.21.15.32:0/1903415307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d80002030 con 0x7f2d8c1a6340 2021-04-09T18:30:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.812 7f2d917de700 1 -- 172.21.15.32:0/1903415307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c0ac6e0 con 0x7f2d8c1a6340 2021-04-09T18:30:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.812 7f2d73fff700 1 -- 172.21.15.32:0/1903415307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2d80006c90 con 0x7f2d8c1a6340 2021-04-09T18:30:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.812 7f2d73fff700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d6c018900 0x7f2d6c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.812 7f2d73fff700 1 -- 172.21.15.32:0/1903415307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f2d80032e40 con 0x7f2d8c1a6340 2021-04-09T18:30:47.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.813 7f2d8b574700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d6c018900 0x7f2d6c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:47.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.813 7f2d8b574700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d6c018900 0x7f2d6c01adc0 crc :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:47.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.833 7f2d917de700 1 -- 172.21.15.32:0/1903415307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c1b7070 con 0x7f2d8c1a6340 2021-04-09T18:30:47.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:47.837 7f2d73fff700 1 -- 172.21.15.32:0/1903415307 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2d80006820 con 0x7f2d8c1a6340 2021-04-09T18:30:48.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.031 7f2d917de700 1 -- 172.21.15.32:0/1903415307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d8c10d480 con 0x7f2d8c1a6340 2021-04-09T18:30:48.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.033 7f2d73fff700 1 -- 172.21.15.32:0/1903415307 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 335 v335) v1 ==== 95+0+3381 (secure 0 0 0) 0x7f2d8000bd80 con 0x7f2d8c1a6340 2021-04-09T18:30:48.049 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 335 2021-04-09T18:30:48.049 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:48.050 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":335,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":335,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:47.615425","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":189,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58739":{"gid":58739,"name":"c","rank":0,"incarnation":335,"state":"up:replay","state_seq":2,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 -- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d6c018900 msgr2=0x7f2d6c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:48.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d6c018900 0x7f2d6c01adc0 crc :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 -- 172.21.15.32:0/1903415307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c1a6340 msgr2=0x7f2d8c1a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:48.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c1a6340 0x7f2d8c1a6740 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7f2d80006be0 tx=0x7f2d8000b990).stop 2021-04-09T18:30:48.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 -- 172.21.15.32:0/1903415307 shutdown_connections 2021-04-09T18:30:48.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2d6c018900 0x7f2d6c01adc0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:48.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d8c1a6340 0x7f2d8c1a6740 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:48.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d8c0abaa0 0x7f2d8c0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:48.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 --2- 172.21.15.32:0/1903415307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d8c1a5210 0x7f2d8c1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:48.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 -- 172.21.15.32:0/1903415307 >> 172.21.15.32:0/1903415307 conn(0x7f2d8c1a99c0 msgr2=0x7f2d8c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:48.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 -- 172.21.15.32:0/1903415307 shutdown_connections 2021-04-09T18:30:48.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:48.035 7f2d917de700 1 -- 172.21.15.32:0/1903415307 wait complete. 2021-04-09T18:30:48.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 335, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:23.208891', 'modified': '2021-04-09 18:30:47.615425', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 189, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 58739}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_58739': {'gid': 58739, 'name': 'c', 'rank': 0, 'incarnation': 335, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.32:6835/2933940031', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2933940031}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2933940031}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'bravo', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:48.363 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_58739:up:replay 2021-04-09T18:30:49.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:49.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.511 7ffbba325700 1 Processor -- start 2021-04-09T18:30:49.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.511 7ffbba325700 1 -- start start 2021-04-09T18:30:49.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbba325700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbb410d010 0x7ffbb40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbba325700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb410d480 con 0x7ffbb40d1da0 2021-04-09T18:30:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbba325700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb410e1f0 con 0x7ffbb40d8490 2021-04-09T18:30:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbba325700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb410e780 con 0x7ffbb410d010 2021-04-09T18:30:49.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbabfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffbb40d1da0 0x7ffbb40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37654/0 (socket says 172.21.15.32:37654) 2021-04-09T18:30:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbabfff700 1 -- 172.21.15.32:0/583616315 learned_addr learned my addr 172.21.15.32:0/583616315 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbb3fff700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbb410d010 0x7ffbb40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542390856 0 0) 0x7ffb9c000ce0 con 0x7ffbb40d1da0 2021-04-09T18:30:49.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba0002fe0 con 0x7ffbb40d1da0 2021-04-09T18:30:49.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615504533 0 0) 0x7ffbac001400 con 0x7ffbb40d8490 2021-04-09T18:30:49.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba0003210 con 0x7ffbb40d8490 2021-04-09T18:30:49.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.512 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2116575739 0 0) 0x7ffb9c0010e0 con 0x7ffbb40d1da0 2021-04-09T18:30:49.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbb410d010 msgr2=0x7ffbb40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:49.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbb410d010 0x7ffbb40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 >> v1:172.21.15.86:6789/0 conn(0x7ffbb40d8490 legacy=0x7ffbb410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:49.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb40d8d30 con 0x7ffbb40d1da0 2021-04-09T18:30:49.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7ffb9c001600 con 0x7ffbb40d1da0 2021-04-09T18:30:49.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 >> v1:172.21.15.32:6789/0 conn(0x7ffbb40d1da0 legacy=0x7ffbb40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:49.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffba0003b90 0x7ffba0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:49.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba0006770 0x7ffba0008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:49.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffba0009510 0x7ffba000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:49.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba00502f0 con 0x7ffba0009510 2021-04-09T18:30:49.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba00504d0 con 0x7ffba0006770 2021-04-09T18:30:49.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba00506f0 con 0x7ffba0003b90 2021-04-09T18:30:49.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb8942700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffba0009510 0x7ffba000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:49.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbabfff700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba0006770 0x7ffba0008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:49.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb3fff700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffba0003b90 0x7ffba0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb8942700 1 -- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffba0003b90 msgr2=0x7ffba0005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb8942700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffba0003b90 0x7ffba0005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb8942700 1 -- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba0006770 msgr2=0x7ffba0008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:49.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb8942700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba0006770 0x7ffba0008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb8942700 1 -- 172.21.15.32:0/583616315 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba0050990 con 0x7ffba0009510 2021-04-09T18:30:49.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb8942700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffba0009510 0x7ffba000ba30 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7ffbac0013c0 tx=0x7ffbac009de0).ready entity=mon.0 client_cookie=c322b24e2948cd32 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:49.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbac006a70 con 0x7ffba0009510 2021-04-09T18:30:49.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac001400 con 0x7ffba0009510 2021-04-09T18:30:49.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb37fe700 1 -- 172.21.15.32:0/583616315 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbac001850 con 0x7ffba0009510 2021-04-09T18:30:49.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbba325700 1 -- 172.21.15.32:0/583616315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffba0009510 msgr2=0x7ffba000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:49.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbba325700 1 --2- 172.21.15.32:0/583616315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffba0009510 0x7ffba000ba30 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7ffbac0013c0 tx=0x7ffbac009de0).stop 2021-04-09T18:30:49.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.513 7ffbb8942700 1 -- 172.21.15.32:0/583616315 reap_dead start 2021-04-09T18:30:49.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.514 7ffbba325700 1 -- 172.21.15.32:0/583616315 shutdown_connections 2021-04-09T18:30:49.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.514 7ffbba325700 1 -- 172.21.15.32:0/583616315 >> 172.21.15.32:0/583616315 conn(0x7ffbb40ce5f0 msgr2=0x7ffbb40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:49.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.514 7ffbba325700 1 -- 172.21.15.32:0/583616315 shutdown_connections 2021-04-09T18:30:49.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.514 7ffbba325700 1 -- 172.21.15.32:0/583616315 wait complete. 2021-04-09T18:30:49.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.534 7ffbba325700 1 Processor -- start 2021-04-09T18:30:49.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbba325700 1 -- start start 2021-04-09T18:30:49.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbba325700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb41abcb0 0x7ffbb41ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbba325700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbb4191140 0x7ffbb4191540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbba325700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb4191b60 0x7ffbb41a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbba325700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb4191f60 con 0x7ffbb41abcb0 2021-04-09T18:30:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbba325700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb41ac6d0 con 0x7ffbb4191b60 2021-04-09T18:30:49.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbba325700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb41aa5c0 con 0x7ffbb4191140 2021-04-09T18:30:49.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbb3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb41abcb0 0x7ffbb41ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:49.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbb3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb41abcb0 0x7ffbb41ac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56304/0 (socket says 172.21.15.32:56304) 2021-04-09T18:30:49.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbb3fff700 1 -- 172.21.15.32:0/2577455124 learned_addr learned my addr 172.21.15.32:0/2577455124 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:49.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbabfff700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbb4191140 0x7ffbb4191540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:49.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbb8942700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb4191b60 0x7ffbb41a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:49.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbb3fff700 1 -- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbb4191140 msgr2=0x7ffbb4191540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:49.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbb3fff700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbb4191140 0x7ffbb4191540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbb3fff700 1 -- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb4191b60 msgr2=0x7ffbb41a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.535 7ffbb3fff700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb4191b60 0x7ffbb41a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.536 7ffbb3fff700 1 -- 172.21.15.32:0/2577455124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb41aa7e0 con 0x7ffbb41abcb0 2021-04-09T18:30:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.536 7ffbb8942700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb4191b60 0x7ffbb41a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:30:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.536 7ffbb3fff700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb41abcb0 0x7ffbb41ac0b0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 rx=0x7ffba400b560 tx=0x7ffba400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.536 7ffbb17fa700 1 -- 172.21.15.32:0/2577455124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba400bcf0 con 0x7ffbb41abcb0 2021-04-09T18:30:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.536 7ffbb17fa700 1 -- 172.21.15.32:0/2577455124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba4000c90 con 0x7ffbb41abcb0 2021-04-09T18:30:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.536 7ffbba325700 1 -- 172.21.15.32:0/2577455124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb41aa7e0 con 0x7ffbb41abcb0 2021-04-09T18:30:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.536 7ffbb17fa700 1 -- 172.21.15.32:0/2577455124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba4002220 con 0x7ffbb41abcb0 2021-04-09T18:30:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.536 7ffbba325700 1 -- 172.21.15.32:0/2577455124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb41aa7e0 con 0x7ffbb41abcb0 2021-04-09T18:30:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.536 7ffbb17fa700 1 -- 172.21.15.32:0/2577455124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ffba400a000 con 0x7ffbb41abcb0 2021-04-09T18:30:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.537 7ffbb17fa700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffb98018930 0x7ffb9801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:49.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.537 7ffbb17fa700 1 -- 172.21.15.32:0/2577455124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6647+0+0 (secure 0 0 0) 0x7ffba4032e50 con 0x7ffbb41abcb0 2021-04-09T18:30:49.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.537 7ffbabfff700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffb98018930 0x7ffb9801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:49.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.537 7ffbabfff700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffb98018930 0x7ffb9801adc0 crc :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:49.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.556 7ffbba325700 1 -- 172.21.15.32:0/2577455124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb4191f60 con 0x7ffbb41abcb0 2021-04-09T18:30:49.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.559 7ffbb17fa700 1 -- 172.21.15.32:0/2577455124 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ffba400a000 con 0x7ffbb41abcb0 2021-04-09T18:30:49.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.752 7ffbba325700 1 -- 172.21.15.32:0/2577455124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbb4191f60 con 0x7ffbb41abcb0 2021-04-09T18:30:49.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.752 7ffbb17fa700 1 -- 172.21.15.32:0/2577455124 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 337 v337) v1 ==== 95+0+3381 (secure 0 0 0) 0x7ffba4000c90 con 0x7ffbb41abcb0 2021-04-09T18:30:49.768 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 337 2021-04-09T18:30:49.768 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:49.768 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":337,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:49.618237","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":189,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58739":{"gid":58739,"name":"c","rank":0,"incarnation":335,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:49.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 -- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffb98018930 msgr2=0x7ffb9801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:49.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffb98018930 0x7ffb9801adc0 crc :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 -- 172.21.15.32:0/2577455124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb41abcb0 msgr2=0x7ffbb41ac0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:49.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb41abcb0 0x7ffbb41ac0b0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 rx=0x7ffba400b560 tx=0x7ffba400b900).stop 2021-04-09T18:30:49.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 -- 172.21.15.32:0/2577455124 shutdown_connections 2021-04-09T18:30:49.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffb98018930 0x7ffb9801adc0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbb4191b60 0x7ffbb41a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbb4191140 0x7ffbb4191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 --2- 172.21.15.32:0/2577455124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbb41abcb0 0x7ffbb41ac0b0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 -- 172.21.15.32:0/2577455124 >> 172.21.15.32:0/2577455124 conn(0x7ffbb41837d0 msgr2=0x7ffbb4195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 -- 172.21.15.32:0/2577455124 shutdown_connections 2021-04-09T18:30:49.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:49.755 7ffbba325700 1 -- 172.21.15.32:0/2577455124 wait complete. 2021-04-09T18:30:50.082 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 337, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:23.208891', 'modified': '2021-04-09 18:30:49.618237', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 189, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 58739}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_58739': {'gid': 58739, 'name': 'c', 'rank': 0, 'incarnation': 335, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.32:6835/2933940031', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2933940031}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2933940031}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'bravo', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:50.082 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_58739:up:rejoin 2021-04-09T18:30:51.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:30:51.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.222 7f88cc9b4700 1 Processor -- start 2021-04-09T18:30:51.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.222 7f88cc9b4700 1 -- start start 2021-04-09T18:30:51.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.222 7f88cc9b4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c80d7ca0 0x7f88c80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:51.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.222 7f88cc9b4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88c810da00 0x7f88c810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:51.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.222 7f88cc9b4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c810d480 con 0x7f88c810d010 2021-04-09T18:30:51.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.222 7f88cc9b4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c80d9220 con 0x7f88c810da00 2021-04-09T18:30:51.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.222 7f88cc9b4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80d86c0 con 0x7f88c80d7ca0 2021-04-09T18:30:51.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c5e1b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c80d7ca0 0x7f88c80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:51.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c6e1d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88c810da00 0x7f88c810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:51.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c661c700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f88c810d010 0x7f88c80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:35596/0 (socket says 172.21.15.32:35596) 2021-04-09T18:30:51.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c661c700 1 -- 172.21.15.32:0/2983797672 learned_addr learned my addr 172.21.15.32:0/2983797672 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:30:51.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c5e1b700 1 --2- 172.21.15.32:0/2983797672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c80d7ca0 0x7f88c80d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:35310/0 (socket says 172.21.15.32:35310) 2021-04-09T18:30:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c5e1b700 1 -- 172.21.15.32:0/2983797672 >> v2:172.21.15.86:3300/0 conn(0x7f88c810da00 msgr2=0x7f88c810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:30:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c5e1b700 1 --2- 172.21.15.32:0/2983797672 >> v2:172.21.15.86:3300/0 conn(0x7f88c810da00 0x7f88c810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c5e1b700 1 -- 172.21.15.32:0/2983797672 >> v1:172.21.15.86:6789/0 conn(0x7f88c810d010 legacy=0x7f88c80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c5e1b700 1 -- 172.21.15.32:0/2983797672 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c8112100 con 0x7f88c80d7ca0 2021-04-09T18:30:51.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c561a700 1 -- 172.21.15.32:0/2983797672 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799405650 0 0) 0x7f88b80010c0 con 0x7f88c810d010 2021-04-09T18:30:51.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c6e1d700 1 --2- 172.21.15.32:0/2983797672 >> v2:172.21.15.86:3300/0 conn(0x7f88c810da00 0x7f88c810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:51.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c5e1b700 1 --2- 172.21.15.32:0/2983797672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c80d7ca0 0x7f88c80d80a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f88b0006650 tx=0x7f88b0009a80).ready entity=mon.2 client_cookie=14da72cacfe366b9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:51.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.223 7f88c561a700 1 -- 172.21.15.32:0/2983797672 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88b00009b0 con 0x7f88c80d7ca0 2021-04-09T18:30:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88c561a700 1 -- 172.21.15.32:0/2983797672 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b000c0c0 con 0x7f88c80d7ca0 2021-04-09T18:30:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88c561a700 1 -- 172.21.15.32:0/2983797672 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88b000c510 con 0x7f88c80d7ca0 2021-04-09T18:30:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88cc9b4700 1 -- 172.21.15.32:0/2983797672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c80d7ca0 msgr2=0x7f88c80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88cc9b4700 1 --2- 172.21.15.32:0/2983797672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c80d7ca0 0x7f88c80d80a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f88b0006650 tx=0x7f88b0009a80).stop 2021-04-09T18:30:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88cc9b4700 1 -- 172.21.15.32:0/2983797672 shutdown_connections 2021-04-09T18:30:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88cc9b4700 1 --2- 172.21.15.32:0/2983797672 >> v2:172.21.15.86:3300/0 conn(0x7f88c810da00 0x7f88c810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88cc9b4700 1 --2- 172.21.15.32:0/2983797672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c80d7ca0 0x7f88c80d80a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88cc9b4700 1 -- 172.21.15.32:0/2983797672 >> 172.21.15.32:0/2983797672 conn(0x7f88c80ce5f0 msgr2=0x7f88c80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88cc9b4700 1 -- 172.21.15.32:0/2983797672 shutdown_connections 2021-04-09T18:30:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.224 7f88cc9b4700 1 -- 172.21.15.32:0/2983797672 wait complete. 2021-04-09T18:30:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.244 7f88cc9b4700 1 Processor -- start 2021-04-09T18:30:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88cc9b4700 1 -- start start 2021-04-09T18:30:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88cc9b4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88c81acea0 0x7f88c81ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88cc9b4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88c819b2c0 0x7f88c8184da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88cc9b4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c81853c0 0x7f88c8185fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88cc9b4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c81857c0 con 0x7f88c819b2c0 2021-04-09T18:30:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88cc9b4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c8185a40 con 0x7f88c81acea0 2021-04-09T18:30:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88cc9b4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c81865e0 con 0x7f88c81853c0 2021-04-09T18:30:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88c5e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88c819b2c0 0x7f88c8184da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88c6e1d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c81853c0 0x7f88c8185fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:51.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88c661c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88c81acea0 0x7f88c81ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:51.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88c5e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88c819b2c0 0x7f88c8184da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56318/0 (socket says 172.21.15.32:56318) 2021-04-09T18:30:51.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.245 7f88c5e1b700 1 -- 172.21.15.32:0/58224339 learned_addr learned my addr 172.21.15.32:0/58224339 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:30:51.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88c6e1d700 1 -- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88c81acea0 msgr2=0x7f88c81ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:51.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88c6e1d700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88c81acea0 0x7f88c81ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88c6e1d700 1 -- 172.21.15.32:0/58224339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88c819b2c0 msgr2=0x7f88c8184da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88c6e1d700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88c819b2c0 0x7f88c8184da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88c6e1d700 1 -- 172.21.15.32:0/58224339 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c8186800 con 0x7f88c81853c0 2021-04-09T18:30:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88c661c700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88c81acea0 0x7f88c81ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88c5e1b700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88c819b2c0 0x7f88c8184da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:30:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88c6e1d700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c81853c0 0x7f88c8185fc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f88c000b8a0 tx=0x7f88c000bc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88beffd700 1 -- 172.21.15.32:0/58224339 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88c0006790 con 0x7f88c81853c0 2021-04-09T18:30:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88beffd700 1 -- 172.21.15.32:0/58224339 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c0000f90 con 0x7f88c81853c0 2021-04-09T18:30:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c8186800 con 0x7f88c81853c0 2021-04-09T18:30:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88beffd700 1 -- 172.21.15.32:0/58224339 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88c00013e0 con 0x7f88c81853c0 2021-04-09T18:30:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.246 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c8186800 con 0x7f88c81853c0 2021-04-09T18:30:51.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.247 7f88beffd700 1 -- 172.21.15.32:0/58224339 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f88c00011f0 con 0x7f88c81853c0 2021-04-09T18:30:51.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.247 7f88beffd700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88ac0188d0 0x7f88ac01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:30:51.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.247 7f88beffd700 1 -- 172.21.15.32:0/58224339 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f88c00330d0 con 0x7f88c81853c0 2021-04-09T18:30:51.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.247 7f88c661c700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88ac0188d0 0x7f88ac01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:30:51.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.247 7f88c661c700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88ac0188d0 0x7f88ac01add0 crc :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:30:51.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.266 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c8182d80 con 0x7f88c81853c0 2021-04-09T18:30:51.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.270 7f88beffd700 1 -- 172.21.15.32:0/58224339 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f88c000bd80 con 0x7f88c81853c0 2021-04-09T18:30:51.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.462 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88c810d480 con 0x7f88c81853c0 2021-04-09T18:30:51.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.463 7f88beffd700 1 -- 172.21.15.32:0/58224339 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+3381 (secure 0 0 0) 0x7f88c0000e80 con 0x7f88c81853c0 2021-04-09T18:30:51.478 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 338 2021-04-09T18:30:51.479 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:30:51.479 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":338,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":338,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:50.619386","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":189,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58739":{"gid":58739,"name":"c","rank":0,"incarnation":335,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:30:51.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.465 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88ac0188d0 msgr2=0x7f88ac01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:51.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88ac0188d0 0x7f88ac01add0 crc :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c81853c0 msgr2=0x7f88c8185fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:30:51.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c81853c0 0x7f88c8185fc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f88c000b8a0 tx=0x7f88c000bc40).stop 2021-04-09T18:30:51.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 shutdown_connections 2021-04-09T18:30:51.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88ac0188d0 0x7f88ac01add0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88c819b2c0 0x7f88c8184da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88c81853c0 0x7f88c8185fc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 --2- 172.21.15.32:0/58224339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88c81acea0 0x7f88c81ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:30:51.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 >> 172.21.15.32:0/58224339 conn(0x7f88c819b890 msgr2=0x7f88c81a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:30:51.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 shutdown_connections 2021-04-09T18:30:51.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:30:51.466 7f88cc9b4700 1 -- 172.21.15.32:0/58224339 wait complete. 2021-04-09T18:30:51.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 338, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:30:23.208891', 'modified': '2021-04-09 18:30:50.619386', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 189, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 58739}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_58739': {'gid': 58739, 'name': 'c', 'rank': 0, 'incarnation': 335, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.32:6835/2933940031', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2933940031}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2933940031}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'bravo', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:30:51.793 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:30:51.793 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:30:51.930 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:30:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:30:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:30:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 58739, 2021-04-09T18:30:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:30:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:30:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 4.2992093249999996, 2021-04-09T18:30:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 338, 2021-04-09T18:30:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 189, 2021-04-09T18:30:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 189, 2021-04-09T18:30:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 25.658192197000002 2021-04-09T18:30:51.936 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:30:51.937 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 58739, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.2992093249999996, "mdsmap_epoch": 338, "osdmap_epoch": 189, "osdmap_epoch_barrier": 189, "uptime": 25.658192197000002 } 2021-04-09T18:30:51.939 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:51.940 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:51.940 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:51.990 INFO:teuthology.orchestra.run.smithi086.stdout:fuseblk 2021-04-09T18:30:51.990 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-04-09T18:30:51.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2021-04-09T18:30:51.991 DEBUG:teuthology.orchestra.run.smithi086:> import os 2021-04-09T18:30:51.992 DEBUG:teuthology.orchestra.run.smithi086:> import time 2021-04-09T18:30:51.992 DEBUG:teuthology.orchestra.run.smithi086:> 2021-04-09T18:30:51.992 DEBUG:teuthology.orchestra.run.smithi086:> fd = os.open("/home/ubuntu/cephtest/mnt.1/background_file", os.O_RDWR | os.O_CREAT, 0o644) 2021-04-09T18:30:51.993 DEBUG:teuthology.orchestra.run.smithi086:> try: 2021-04-09T18:30:51.993 DEBUG:teuthology.orchestra.run.smithi086:> while True: 2021-04-09T18:30:51.993 DEBUG:teuthology.orchestra.run.smithi086:> os.write(fd, b'"'"'content'"'"') 2021-04-09T18:30:51.994 DEBUG:teuthology.orchestra.run.smithi086:> time.sleep(1) 2021-04-09T18:30:51.994 DEBUG:teuthology.orchestra.run.smithi086:> if not False: 2021-04-09T18:30:51.995 DEBUG:teuthology.orchestra.run.smithi086:> break 2021-04-09T18:30:51.995 DEBUG:teuthology.orchestra.run.smithi086:> except IOError as e: 2021-04-09T18:30:51.995 DEBUG:teuthology.orchestra.run.smithi086:> pass 2021-04-09T18:30:51.996 DEBUG:teuthology.orchestra.run.smithi086:> os.close(fd) 2021-04-09T18:30:51.996 DEBUG:teuthology.orchestra.run.smithi086:> ' 2021-04-09T18:30:52.100 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:30:57.101 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2021-04-09T18:30:57.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:57.102 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:57.103 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:57.154 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:30:57.155 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:30:57.155 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:30:57.156 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:30:57.156 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:57.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[92974]: fuse finished with error 0 and tester_r 0 2021-04-09T18:30:57.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:57.228 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:57.228 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:57.274 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:30:57.274 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:30:57.274 INFO:teuthology.orchestra.run:waiting for 900 2021-04-09T18:30:57.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:30:57.364 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:30:57.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:57.408 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:57.409 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:57.452 INFO:teuthology.orchestra.run.smithi086.stdout:fuseblk 2021-04-09T18:30:57.453 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-04-09T18:30:57.453 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi086.front.sepia.ceph.com... 2021-04-09T18:30:57.454 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:30:57.454 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:57.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi086.stderr:ceph-fuse[31844]: fuse finished with error 0 and tester_r 0 2021-04-09T18:30:57.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:30:57.565 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:30:57.565 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:30:57.607 INFO:teuthology.orchestra.run.smithi086.stdout:ext2/ext3 2021-04-09T18:30:57.608 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:30:57.608 INFO:teuthology.orchestra.run:waiting for 900 2021-04-09T18:31:03.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:31:03.711 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:31:03.779 DEBUG:teuthology.orchestra.run.smithi032:> rados -p alpha_data ls 2021-04-09T18:31:03.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff68b0549c0 1 Processor -- start 2021-04-09T18:31:03.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff68b0549c0 1 -- start start 2021-04-09T18:31:03.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff68b0549c0 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3635bb870 0x55f3635a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff68b0549c0 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634bdf00 0x55f3634b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff68b0549c0 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3634bb1a0 0x55f3634b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff68b0549c0 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3634b54c0 con 0x55f3635bb870 2021-04-09T18:31:03.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff68b0549c0 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3634b7170 con 0x55f3634bdf00 2021-04-09T18:31:03.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff68b0549c0 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3634b5e70 con 0x55f3634bb1a0 2021-04-09T18:31:03.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff67a945700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3635bb870 0x55f3635a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff67a945700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3635bb870 0x55f3635a4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56330/0 (socket says 172.21.15.32:56330) 2021-04-09T18:31:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff67a945700 1 -- 172.21.15.32:0/2601333873 learned_addr learned my addr 172.21.15.32:0/2601333873 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:03.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff67a144700 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634bdf00 0x55f3634b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.871 7ff67b146700 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3634bb1a0 0x55f3634b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff67a945700 1 -- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3634bb1a0 msgr2=0x55f3634b63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff67a945700 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3634bb1a0 0x55f3634b63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff67a945700 1 -- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634bdf00 msgr2=0x55f3634b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff67a945700 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634bdf00 0x55f3634b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff67a945700 1 -- 172.21.15.32:0/2601333873 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3634b50d0 con 0x55f3635bb870 2021-04-09T18:31:03.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff67a144700 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634bdf00 0x55f3634b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:03.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff67b146700 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3634bb1a0 0x55f3634b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:03.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff67a945700 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3635bb870 0x55f3635a4ed0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 rx=0x7ff66c0055d0 tx=0x7ff66c008870).ready entity=mon.0 client_cookie=a1cc654edcef2758 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:03.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff678941700 1 -- 172.21.15.32:0/2601333873 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff66c0009b0 con 0x55f3635bb870 2021-04-09T18:31:03.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff678941700 1 -- 172.21.15.32:0/2601333873 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66c008c60 con 0x55f3635bb870 2021-04-09T18:31:03.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff678941700 1 -- 172.21.15.32:0/2601333873 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff66c00a340 con 0x55f3635bb870 2021-04-09T18:31:03.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff68b0549c0 1 -- 172.21.15.32:0/2601333873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3635bb870 msgr2=0x55f3635a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:03.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff68b0549c0 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3635bb870 0x55f3635a4ed0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 rx=0x7ff66c0055d0 tx=0x7ff66c008870).stop 2021-04-09T18:31:03.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.872 7ff68b0549c0 1 -- 172.21.15.32:0/2601333873 shutdown_connections 2021-04-09T18:31:03.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.873 7ff68b0549c0 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3634bb1a0 0x55f3634b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.873 7ff68b0549c0 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634bdf00 0x55f3634b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.873 7ff68b0549c0 1 --2- 172.21.15.32:0/2601333873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3635bb870 0x55f3635a4ed0 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.873 7ff68b0549c0 1 -- 172.21.15.32:0/2601333873 >> 172.21.15.32:0/2601333873 conn(0x55f3634b89c0 msgr2=0x55f3634b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:03.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.873 7ff68b0549c0 1 -- 172.21.15.32:0/2601333873 shutdown_connections 2021-04-09T18:31:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.873 7ff68b0549c0 1 -- 172.21.15.32:0/2601333873 wait complete. 2021-04-09T18:31:03.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.873 7ff68b0549c0 1 Processor -- start 2021-04-09T18:31:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.873 7ff68b0549c0 1 -- start start 2021-04-09T18:31:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff68b0549c0 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634df3d0 0x55f3634b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff68b0549c0 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3634b7d10 0x55f3635bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff68b0549c0 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3635c0020 0x55f3634b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff68b0549c0 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3635c0550 con 0x55f3634b7d10 2021-04-09T18:31:03.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff68b0549c0 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3634b50f0 con 0x55f3634df3d0 2021-04-09T18:31:03.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff68b0549c0 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3634b48f0 con 0x55f3635c0020 2021-04-09T18:31:03.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a144700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3634b7d10 0x55f3635bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a144700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3634b7d10 0x55f3635bfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56338/0 (socket says 172.21.15.32:56338) 2021-04-09T18:31:03.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a144700 1 -- 172.21.15.32:0/381725484 learned_addr learned my addr 172.21.15.32:0/381725484 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:03.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a945700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634df3d0 0x55f3634b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67b146700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3635c0020 0x55f3634b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a144700 1 -- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3635c0020 msgr2=0x55f3634b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:03.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a144700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3635c0020 0x55f3634b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a144700 1 -- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634df3d0 msgr2=0x55f3634b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a144700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634df3d0 0x55f3634b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a144700 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3634b4b10 con 0x55f3634b7d10 2021-04-09T18:31:03.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a945700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f3634df3d0 0x55f3634b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67b146700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f3635c0020 0x55f3634b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:03.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff67a144700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3634b7d10 0x55f3635bfa00 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 rx=0x7ff670008fc0 tx=0x7ff67000a640).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:03.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff66bfff700 1 -- 172.21.15.32:0/381725484 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6700054d0 con 0x55f3634b7d10 2021-04-09T18:31:03.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff66bfff700 1 -- 172.21.15.32:0/381725484 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff670000cd0 con 0x55f3634b7d10 2021-04-09T18:31:03.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f3634b4b10 con 0x55f3634b7d10 2021-04-09T18:31:03.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff66bfff700 1 -- 172.21.15.32:0/381725484 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff67000acb0 con 0x55f3634b7d10 2021-04-09T18:31:03.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.874 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3634b4b10 con 0x55f3634b7d10 2021-04-09T18:31:03.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.875 7ff66bfff700 1 -- 172.21.15.32:0/381725484 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff670009000 con 0x55f3634b7d10 2021-04-09T18:31:03.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.875 7ff66bfff700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff65c017620 0x7ff65c019b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.875 7ff66bfff700 1 -- 172.21.15.32:0/381725484 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6647+0+0 (secure 0 0 0) 0x7ff670031890 con 0x55f3634b7d10 2021-04-09T18:31:03.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.875 7ff67a945700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff65c017620 0x7ff65c019b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.876 7ff68b0549c0 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x55f3634b48f0 0x55f3635ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.876 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] -- osd_op(unknown.0.0:1 31.0 31:00000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x55f3635cec70 con 0x55f3634b48f0 2021-04-09T18:31:03.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.876 7ff67b146700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x55f3634b48f0 0x55f3635ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.876 7ff67a945700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff65c017620 0x7ff65c019b00 crc :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:03.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.876 7ff67b146700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x55f3634b48f0 0x55f3635ce650 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:03.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.876 7ff67b146700 1 -- 172.21.15.32:0/381725484 <== osd.4 v2:172.21.15.86:6802/25232 1 ==== osd_op_reply(1 [pgnls start_epoch 189] v188'2 uv2 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ff674000f40 con 0x55f3634b48f0 2021-04-09T18:31:03.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.876 7ff67b146700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7ff674009bb0 0x7ff674008050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.876 7ff67b146700 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] -- osd_op(unknown.0.0:2 31.4 31:20000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7ff674008750 con 0x7ff674009bb0 2021-04-09T18:31:03.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.877 7ff67a945700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7ff674009bb0 0x7ff674008050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.877 7ff67a945700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7ff674009bb0 0x7ff674008050 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:03.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.877 7ff67a945700 1 -- 172.21.15.32:0/381725484 <== osd.0 v2:172.21.15.32:6800/24886 1 ==== osd_op_reply(2 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff66c000c90 con 0x7ff674009bb0 2021-04-09T18:31:03.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.877 7ff67a945700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7ff66c002560 0x7ff66c0083b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.877 7ff67a945700 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] -- osd_op(unknown.0.0:3 31.2 31:40000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7ff66c002960 con 0x7ff66c002560 2021-04-09T18:31:03.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.877 7ff67a144700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7ff66c002560 0x7ff66c0083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.878 7ff67a144700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7ff66c002560 0x7ff66c0083b0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:03.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.878 7ff67a144700 1 -- 172.21.15.32:0/381725484 <== osd.6 v2:172.21.15.86:6818/25238 1 ==== osd_op_reply(3 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff67002fd90 con 0x7ff66c002560 2021-04-09T18:31:03.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.878 7ff67a144700 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] -- osd_op(unknown.0.0:4 31.6 31:60000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7ff67001e740 con 0x7ff66c002560 2021-04-09T18:31:03.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.878 7ff67a144700 1 -- 172.21.15.32:0/381725484 <== osd.6 v2:172.21.15.86:6818/25238 2 ==== osd_op_reply(4 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff67002fd90 con 0x7ff66c002560 2021-04-09T18:31:03.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.878 7ff67a144700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7ff670009260 0x7ff670031020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:03.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.878 7ff67a144700 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] -- osd_op(unknown.0.0:5 31.1 31:80000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7ff670031640 con 0x7ff670009260 2021-04-09T18:31:03.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.878 7ff67b146700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7ff670009260 0x7ff670031020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:03.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.879 7ff67b146700 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7ff670009260 0x7ff670031020 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:03.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.879 7ff67b146700 1 -- 172.21.15.32:0/381725484 <== osd.1 v2:172.21.15.32:6808/24901 1 ==== osd_op_reply(5 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff674008ba0 con 0x7ff670009260 2021-04-09T18:31:03.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.879 7ff67b146700 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] -- osd_op(unknown.0.0:6 31.5 31:a0000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7ff6740100c0 con 0x7ff674009bb0 2021-04-09T18:31:03.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.879 7ff67a945700 1 -- 172.21.15.32:0/381725484 <== osd.0 v2:172.21.15.32:6800/24886 2 ==== osd_op_reply(6 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff66c000c90 con 0x7ff674009bb0 2021-04-09T18:31:03.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.879 7ff67a945700 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] -- osd_op(unknown.0.0:7 31.3 31:c0000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7ff66c0089d0 con 0x55f3634b48f0 2021-04-09T18:31:03.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff67b146700 1 -- 172.21.15.32:0/381725484 <== osd.4 v2:172.21.15.86:6802/25232 2 ==== osd_op_reply(7 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff674008750 con 0x55f3634b48f0 2021-04-09T18:31:03.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff67b146700 1 -- 172.21.15.32:0/381725484 --> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] -- osd_op(unknown.0.0:8 31.7 31:e0000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7ff6740100c0 con 0x7ff670009260 2021-04-09T18:31:03.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff67b146700 1 -- 172.21.15.32:0/381725484 <== osd.1 v2:172.21.15.32:6808/24901 2 ==== osd_op_reply(8 [pgnls start_epoch 189] v188'1 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ff674008e90 con 0x7ff670009260 2021-04-09T18:31:03.938 INFO:teuthology.orchestra.run.smithi032.stdout:10000000000.00000000 2021-04-09T18:31:03.939 INFO:teuthology.orchestra.run.smithi032.stdout:10000000001.00000000 2021-04-09T18:31:03.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7ff674009bb0 msgr2=0x7ff674008050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:03.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6800/24886,v1:172.21.15.32:6801/24886] conn(0x7ff674009bb0 0x7ff674008050 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7ff670009260 msgr2=0x7ff670031020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:03.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:6808/24901,v1:172.21.15.32:6809/24901] conn(0x7ff670009260 0x7ff670031020 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x55f3634b48f0 msgr2=0x55f3635ce650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:03.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6802/25232,v1:172.21.15.86:6803/25232] conn(0x55f3634b48f0 0x55f3635ce650 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7ff66c002560 msgr2=0x7ff66c0083b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6818/25238,v1:172.21.15.86:6819/25238] conn(0x7ff66c002560 0x7ff66c0083b0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff67b146700 1 -- 172.21.15.32:0/381725484 reap_dead start 2021-04-09T18:31:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff67b146700 1 -- 172.21.15.32:0/381725484 reap_dead start 2021-04-09T18:31:03.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff65c017620 msgr2=0x7ff65c019b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff65c017620 0x7ff65c019b00 crc :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3634b7d10 msgr2=0x55f3635bfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3634b7d10 0x55f3635bfa00 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 rx=0x7ff670008fc0 tx=0x7ff67000a640).stop 2021-04-09T18:31:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 shutdown_connections 2021-04-09T18:31:03.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f3634b7d10 0x55f3635bfa00 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 --2- 172.21.15.32:0/381725484 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff65c017620 0x7ff65c019b00 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:03.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 >> 172.21.15.32:0/381725484 conn(0x55f3634c3950 msgr2=0x55f3635b9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:03.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 shutdown_connections 2021-04-09T18:31:03.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:03.880 7ff68b0549c0 1 -- 172.21.15.32:0/381725484 wait complete. 2021-04-09T18:31:03.946 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 1048576 found 2021-04-09T18:31:03.946 DEBUG:teuthology.orchestra.run.smithi032:> rados -p bravo_data ls 2021-04-09T18:31:04.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.032 7f8d2fed99c0 1 Processor -- start 2021-04-09T18:31:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.032 7f8d2fed99c0 1 -- start start 2021-04-09T18:31:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.032 7f8d2fed99c0 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0d97c50 0x55a2e0d90a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.032 7f8d2fed99c0 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0e79010 0x55a2e0db55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.032 7f8d2fed99c0 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0db5c00 0x55a2e0d8c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.032 7f8d2fed99c0 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e0d8d510 con 0x55a2e0db5c00 2021-04-09T18:31:04.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.032 7f8d2fed99c0 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e0db0de0 con 0x55a2e0d97c50 2021-04-09T18:31:04.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.032 7f8d2fed99c0 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2e0d976d0 con 0x55a2e0e79010 2021-04-09T18:31:04.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.032 7f8d1ffcb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0db5c00 0x55a2e0d8c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1ffcb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0db5c00 0x55a2e0d8c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56362/0 (socket says 172.21.15.32:56362) 2021-04-09T18:31:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1ffcb700 1 -- 172.21.15.32:0/921878586 learned_addr learned my addr 172.21.15.32:0/921878586 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:04.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1ffcb700 1 -- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0e79010 msgr2=0x55a2e0db55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:04.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1efc9700 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0e79010 0x55a2e0db55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1f7ca700 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0d97c50 0x55a2e0d90a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1ffcb700 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0e79010 0x55a2e0db55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1ffcb700 1 -- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0d97c50 msgr2=0x55a2e0d90a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:04.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1ffcb700 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0d97c50 0x55a2e0d90a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1ffcb700 1 -- 172.21.15.32:0/921878586 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2e0e96b00 con 0x55a2e0db5c00 2021-04-09T18:31:04.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1efc9700 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0e79010 0x55a2e0db55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:04.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1f7ca700 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0d97c50 0x55a2e0d90a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:04.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1ffcb700 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0db5c00 0x55a2e0d8c930 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 rx=0x7f8d180086b0 tx=0x7f8d18005bd0).ready entity=mon.0 client_cookie=ebe88e5bb215bd11 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:04.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1d7c6700 1 -- 172.21.15.32:0/921878586 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d180080c0 con 0x55a2e0db5c00 2021-04-09T18:31:04.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1d7c6700 1 -- 172.21.15.32:0/921878586 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d18003030 con 0x55a2e0db5c00 2021-04-09T18:31:04.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d1d7c6700 1 -- 172.21.15.32:0/921878586 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d1800a4f0 con 0x55a2e0db5c00 2021-04-09T18:31:04.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d2fed99c0 1 -- 172.21.15.32:0/921878586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0db5c00 msgr2=0x55a2e0d8c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.033 7f8d2fed99c0 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0db5c00 0x55a2e0d8c930 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 rx=0x7f8d180086b0 tx=0x7f8d18005bd0).stop 2021-04-09T18:31:04.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.034 7f8d2fed99c0 1 -- 172.21.15.32:0/921878586 shutdown_connections 2021-04-09T18:31:04.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.034 7f8d2fed99c0 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0db5c00 0x55a2e0d8c930 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.034 7f8d2fed99c0 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0e79010 0x55a2e0db55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.034 7f8d2fed99c0 1 --2- 172.21.15.32:0/921878586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0d97c50 0x55a2e0d90a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.034 7f8d2fed99c0 1 -- 172.21.15.32:0/921878586 >> 172.21.15.32:0/921878586 conn(0x55a2e0e87250 msgr2=0x55a2e0db05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.034 7f8d2fed99c0 1 -- 172.21.15.32:0/921878586 shutdown_connections 2021-04-09T18:31:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.034 7f8d2fed99c0 1 -- 172.21.15.32:0/921878586 wait complete. 2021-04-09T18:31:04.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.034 7f8d2fed99c0 1 Processor -- start 2021-04-09T18:31:04.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.034 7f8d2fed99c0 1 -- start start 2021-04-09T18:31:04.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d2fed99c0 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0d8c110 0x55a2e0e77a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d2fed99c0 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0d8ccf0 0x55a2e0d8d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d2fed99c0 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0e78120 0x55a2e0e78520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d2fed99c0 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e0d8c770 con 0x55a2e0d8ccf0 2021-04-09T18:31:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d2fed99c0 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2e0e8ecf0 con 0x55a2e0e78120 2021-04-09T18:31:04.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d2fed99c0 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a2e0d8d710 con 0x55a2e0d8c110 2021-04-09T18:31:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1efc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0d8ccf0 0x55a2e0d8d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1efc9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0d8ccf0 0x55a2e0d8d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56366/0 (socket says 172.21.15.32:56366) 2021-04-09T18:31:04.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1efc9700 1 -- 172.21.15.32:0/3099202080 learned_addr learned my addr 172.21.15.32:0/3099202080 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:04.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1f7ca700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0d8c110 0x55a2e0e77a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1ffcb700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0e78120 0x55a2e0e78520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1efc9700 1 -- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0d8c110 msgr2=0x55a2e0e77a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1efc9700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0d8c110 0x55a2e0e77a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1efc9700 1 -- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0e78120 msgr2=0x55a2e0e78520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1efc9700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a2e0e78120 0x55a2e0e78520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1efc9700 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2e0dada30 con 0x55a2e0d8ccf0 2021-04-09T18:31:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1f7ca700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a2e0d8c110 0x55a2e0e77a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:04.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1efc9700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0d8ccf0 0x55a2e0d8d0f0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 rx=0x7f8d1000a290 tx=0x7f8d1000a630).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.035 7f8d1cfc5700 1 -- 172.21.15.32:0/3099202080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d1000aa20 con 0x55a2e0d8ccf0 2021-04-09T18:31:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.036 7f8d1cfc5700 1 -- 172.21.15.32:0/3099202080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1000ad60 con 0x55a2e0d8ccf0 2021-04-09T18:31:04.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.036 7f8d1cfc5700 1 -- 172.21.15.32:0/3099202080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d100022b0 con 0x55a2e0d8ccf0 2021-04-09T18:31:04.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.036 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a2e0dada30 con 0x55a2e0d8ccf0 2021-04-09T18:31:04.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.036 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2e0dada30 con 0x55a2e0d8ccf0 2021-04-09T18:31:04.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.036 7f8d1cfc5700 1 -- 172.21.15.32:0/3099202080 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8d10000c90 con 0x55a2e0d8ccf0 2021-04-09T18:31:04.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.036 7f8d1cfc5700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d0c017600 0x7f8d0c019b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.036 7f8d1cfc5700 1 -- 172.21.15.32:0/3099202080 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f8d10031970 con 0x55a2e0d8ccf0 2021-04-09T18:31:04.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.037 7f8d1f7ca700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d0c017600 0x7f8d0c019b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.037 7f8d2fed99c0 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x55a2e0ea6620 0x55a2e0ea8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.037 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] -- osd_op(unknown.0.0:1 33.0 33:00000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x55a2e0ea9050 con 0x55a2e0ea6620 2021-04-09T18:31:04.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.037 7f8d1ffcb700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x55a2e0ea6620 0x55a2e0ea8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.037 7f8d1f7ca700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d0c017600 0x7f8d0c019b00 crc :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:04.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.037 7f8d1ffcb700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x55a2e0ea6620 0x55a2e0ea8a30 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:04.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.037 7f8d1ffcb700 1 -- 172.21.15.32:0/3099202080 <== osd.2 v2:172.21.15.32:6816/24934 1 ==== osd_op_reply(1 [pgnls start_epoch 189] v188'2 uv2 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f8d18005c60 con 0x55a2e0ea6620 2021-04-09T18:31:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.037 7f8d1ffcb700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f8d18004a10 0x7f8d18008060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.037 7f8d1ffcb700 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] -- osd_op(unknown.0.0:2 33.4 33:20000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7f8d18008680 con 0x7f8d18004a10 2021-04-09T18:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.038 7f8d1f7ca700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f8d18004a10 0x7f8d18008060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.038 7f8d1f7ca700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f8d18004a10 0x7f8d18008060 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:04.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.038 7f8d1f7ca700 1 -- 172.21.15.32:0/3099202080 <== osd.5 v2:172.21.15.86:6810/25237 1 ==== osd_op_reply(2 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d140018f0 con 0x7f8d18004a10 2021-04-09T18:31:04.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.038 7f8d1f7ca700 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] -- osd_op(unknown.0.0:3 33.2 33:40000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7f8d14009410 con 0x55a2e0ea6620 2021-04-09T18:31:04.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.039 7f8d1ffcb700 1 -- 172.21.15.32:0/3099202080 <== osd.2 v2:172.21.15.32:6816/24934 2 ==== osd_op_reply(3 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d18005a50 con 0x55a2e0ea6620 2021-04-09T18:31:04.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.039 7f8d1ffcb700 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] -- osd_op(unknown.0.0:4 33.6 33:60000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7f8d180089b0 con 0x55a2e0ea6620 2021-04-09T18:31:04.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.039 7f8d1ffcb700 1 -- 172.21.15.32:0/3099202080 <== osd.2 v2:172.21.15.32:6816/24934 3 ==== osd_op_reply(4 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d18004eb0 con 0x55a2e0ea6620 2021-04-09T18:31:04.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.039 7f8d1ffcb700 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] -- osd_op(unknown.0.0:5 33.1 33:80000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7f8d18008970 con 0x7f8d18004a10 2021-04-09T18:31:04.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.039 7f8d1f7ca700 1 -- 172.21.15.32:0/3099202080 <== osd.5 v2:172.21.15.86:6810/25237 2 ==== osd_op_reply(5 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d140017f0 con 0x7f8d18004a10 2021-04-09T18:31:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.039 7f8d1f7ca700 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] -- osd_op(unknown.0.0:6 33.5 33:a0000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7f8d14009700 con 0x7f8d18004a10 2021-04-09T18:31:04.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.040 7f8d1f7ca700 1 -- 172.21.15.32:0/3099202080 <== osd.5 v2:172.21.15.86:6810/25237 3 ==== osd_op_reply(6 [pgnls start_epoch 189] v189'1 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f8d140090e0 con 0x7f8d18004a10 2021-04-09T18:31:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.040 7f8d1f7ca700 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] -- osd_op(unknown.0.0:7 33.3 33:c0000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7f8d140018d0 con 0x55a2e0ea6620 2021-04-09T18:31:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.040 7f8d1ffcb700 1 -- 172.21.15.32:0/3099202080 <== osd.2 v2:172.21.15.32:6816/24934 4 ==== osd_op_reply(7 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d18004eb0 con 0x55a2e0ea6620 2021-04-09T18:31:04.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.040 7f8d1ffcb700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f8d18008ae0 0x7f8d1800a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.040 7f8d1ffcb700 1 -- 172.21.15.32:0/3099202080 --> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] -- osd_op(unknown.0.0:8 33.7 33:e0000000::::head [pgnls start_epoch 189] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e189) v8 -- 0x7f8d1800ab80 con 0x7f8d18008ae0 2021-04-09T18:31:04.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.040 7f8d1efc9700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f8d18008ae0 0x7f8d1800a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.040 7f8d1efc9700 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f8d18008ae0 0x7f8d1800a560 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:04.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.040 7f8d1efc9700 1 -- 172.21.15.32:0/3099202080 <== osd.3 v2:172.21.15.32:6824/24942 1 ==== osd_op_reply(8 [pgnls start_epoch 189] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d1001e4a0 con 0x7f8d18008ae0 2021-04-09T18:31:04.099 INFO:teuthology.orchestra.run.smithi032.stdout:10000000000.00000000 2021-04-09T18:31:04.100 INFO:teuthology.orchestra.run.smithi032.stdout:10000000004.00000000 2021-04-09T18:31:04.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x55a2e0ea6620 msgr2=0x55a2e0ea8a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6816/24934,v1:172.21.15.32:6817/24934] conn(0x55a2e0ea6620 0x55a2e0ea8a30 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f8d18008ae0 msgr2=0x7f8d1800a560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:6824/24942,v1:172.21.15.32:6825/24942] conn(0x7f8d18008ae0 0x7f8d1800a560 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f8d18004a10 msgr2=0x7f8d18008060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6810/25237,v1:172.21.15.86:6811/25237] conn(0x7f8d18004a10 0x7f8d18008060 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d1ffcb700 1 -- 172.21.15.32:0/3099202080 reap_dead start 2021-04-09T18:31:04.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d0c017600 msgr2=0x7f8d0c019b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d0c017600 0x7f8d0c019b00 crc :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0d8ccf0 msgr2=0x55a2e0d8d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0d8ccf0 0x55a2e0d8d0f0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 rx=0x7f8d1000a290 tx=0x7f8d1000a630).stop 2021-04-09T18:31:04.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 shutdown_connections 2021-04-09T18:31:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a2e0d8ccf0 0x55a2e0d8d0f0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 --2- 172.21.15.32:0/3099202080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d0c017600 0x7f8d0c019b00 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 >> 172.21.15.32:0/3099202080 conn(0x55a2e0db0d60 msgr2=0x55a2e0dbcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:04.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 shutdown_connections 2021-04-09T18:31:04.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.041 7f8d2fed99c0 1 -- 172.21.15.32:0/3099202080 wait complete. 2021-04-09T18:31:04.110 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627776 size 1048576 found 2021-04-09T18:31:04.110 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:31:04.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:31:04.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:31:04.234 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:31:04.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:31:04.235 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:31:04.236 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:31:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:31:04.282 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:31:04.283 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:31:04.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:31:04.285 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:31:04.365 INFO:tasks.cephfs.mount:Terminating background process 2021-04-09T18:31:04.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:31:04.367 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:31:04.368 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:31:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:31:04.421 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:31:04.421 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:31:04.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:31:04.423 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:31:04.468 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:31:04.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.601 7f2af37dd700 1 Processor -- start 2021-04-09T18:31:04.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.601 7f2af37dd700 1 -- start start 2021-04-09T18:31:04.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.602 7f2af37dd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2aec10d010 0x7f2aec0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.602 7f2af37dd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec0d1da0 0x7f2aec0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.602 7f2af37dd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec10d480 con 0x7f2aec0d82c0 2021-04-09T18:31:04.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.602 7f2af37dd700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aec0d9220 con 0x7f2aec10d010 2021-04-09T18:31:04.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.602 7f2af37dd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0d86c0 con 0x7f2aec0d1da0 2021-04-09T18:31:04.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.602 7f2af15f9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2aec10d010 0x7f2aec0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.602 7f2af1dfa700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2aec0d82c0 0x7f2aec10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37738/0 (socket says 172.21.15.32:37738) 2021-04-09T18:31:04.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.602 7f2af1dfa700 1 -- 172.21.15.32:0/847965059 learned_addr learned my addr 172.21.15.32:0/847965059 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:04.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.602 7f2af0df8700 1 --2- 172.21.15.32:0/847965059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec0d1da0 0x7f2aec0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2af15f9700 1 -- 172.21.15.32:0/847965059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec0d1da0 msgr2=0x7f2aec0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:04.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2af15f9700 1 --2- 172.21.15.32:0/847965059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec0d1da0 0x7f2aec0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2af15f9700 1 -- 172.21.15.32:0/847965059 >> v1:172.21.15.32:6789/0 conn(0x7f2aec0d82c0 legacy=0x7f2aec10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2ae3fff700 1 -- 172.21.15.32:0/847965059 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625661410 0 0) 0x7f2ae4000d30 con 0x7f2aec0d82c0 2021-04-09T18:31:04.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2af15f9700 1 -- 172.21.15.32:0/847965059 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec10e1f0 con 0x7f2aec10d010 2021-04-09T18:31:04.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2af15f9700 1 --2- 172.21.15.32:0/847965059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec10d010 0x7f2aec0d1780 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f2ae8006650 tx=0x7f2ae8009a00).ready entity=mon.0 client_cookie=b9bf7e362ef32afa server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2ae3fff700 1 -- 172.21.15.32:0/847965059 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ae8009df0 con 0x7f2aec10d010 2021-04-09T18:31:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2ae3fff700 1 -- 172.21.15.32:0/847965059 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae80009b0 con 0x7f2aec10d010 2021-04-09T18:31:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2ae3fff700 1 -- 172.21.15.32:0/847965059 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ae800c2b0 con 0x7f2aec10d010 2021-04-09T18:31:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2af37dd700 1 -- 172.21.15.32:0/847965059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec10d010 msgr2=0x7f2aec0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.603 7f2af37dd700 1 --2- 172.21.15.32:0/847965059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec10d010 0x7f2aec0d1780 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f2ae8006650 tx=0x7f2ae8009a00).stop 2021-04-09T18:31:04.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.604 7f2af37dd700 1 -- 172.21.15.32:0/847965059 shutdown_connections 2021-04-09T18:31:04.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.604 7f2af37dd700 1 --2- 172.21.15.32:0/847965059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec0d1da0 0x7f2aec0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.604 7f2af37dd700 1 --2- 172.21.15.32:0/847965059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec10d010 0x7f2aec0d1780 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.604 7f2af37dd700 1 -- 172.21.15.32:0/847965059 >> 172.21.15.32:0/847965059 conn(0x7f2aec0ce5f0 msgr2=0x7f2aec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:04.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.604 7f2af37dd700 1 -- 172.21.15.32:0/847965059 shutdown_connections 2021-04-09T18:31:04.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.604 7f2af37dd700 1 -- 172.21.15.32:0/847965059 wait complete. 2021-04-09T18:31:04.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.624 7f2af37dd700 1 Processor -- start 2021-04-09T18:31:04.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.624 7f2af37dd700 1 -- start start 2021-04-09T18:31:04.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af37dd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec179bf0 0x7f2aec0abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af37dd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec0ac400 0x7f2aec1a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af37dd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec17a110 0x7f2aec1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af37dd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec17a510 con 0x7f2aec0ac400 2021-04-09T18:31:04.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af37dd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec1a68b0 con 0x7f2aec179bf0 2021-04-09T18:31:04.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af37dd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec182e30 con 0x7f2aec17a110 2021-04-09T18:31:04.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af0df8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec0ac400 0x7f2aec1a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af0df8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec0ac400 0x7f2aec1a6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56386/0 (socket says 172.21.15.32:56386) 2021-04-09T18:31:04.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af0df8700 1 -- 172.21.15.32:0/2275843342 learned_addr learned my addr 172.21.15.32:0/2275843342 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:04.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af1dfa700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec17a110 0x7f2aec1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af15f9700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec179bf0 0x7f2aec0abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af0df8700 1 -- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec17a110 msgr2=0x7f2aec1a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af0df8700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec17a110 0x7f2aec1a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af0df8700 1 -- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec179bf0 msgr2=0x7f2aec0abde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:04.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af0df8700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec179bf0 0x7f2aec0abde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:04.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af0df8700 1 -- 172.21.15.32:0/2275843342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec1a56c0 con 0x7f2aec0ac400 2021-04-09T18:31:04.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af0df8700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec0ac400 0x7f2aec1a6e30 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7f2adc006f60 tx=0x7f2adc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:04.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2ae1ffb700 1 -- 172.21.15.32:0/2275843342 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2adc00bd60 con 0x7f2aec0ac400 2021-04-09T18:31:04.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2ae1ffb700 1 -- 172.21.15.32:0/2275843342 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc000c90 con 0x7f2aec0ac400 2021-04-09T18:31:04.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2ae1ffb700 1 -- 172.21.15.32:0/2275843342 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2adc002220 con 0x7f2aec0ac400 2021-04-09T18:31:04.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec1a56c0 con 0x7f2aec0ac400 2021-04-09T18:31:04.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.625 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec1a56c0 con 0x7f2aec0ac400 2021-04-09T18:31:04.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.626 7f2ae1ffb700 1 -- 172.21.15.32:0/2275843342 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2adc0052e0 con 0x7f2aec0ac400 2021-04-09T18:31:04.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.626 7f2ae1ffb700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2acc018930 0x7f2acc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:04.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.626 7f2ae1ffb700 1 -- 172.21.15.32:0/2275843342 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f2adc032e20 con 0x7f2aec0ac400 2021-04-09T18:31:04.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.626 7f2af15f9700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2acc018930 0x7f2acc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:04.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.627 7f2af15f9700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2acc018930 0x7f2acc01ae10 crc :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:04.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.646 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aec17a510 con 0x7f2aec0ac400 2021-04-09T18:31:04.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.650 7f2ae1ffb700 1 -- 172.21.15.32:0/2275843342 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2adc000c90 con 0x7f2aec0ac400 2021-04-09T18:31:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.840 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2aec17a510 con 0x7f2aec0ac400 2021-04-09T18:31:04.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.841 7f2ae1ffb700 1 -- 172.21.15.32:0/2275843342 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v189) v1 ==== 81+0+35756 (secure 0 0 0) 0x7f2adc006b00 con 0x7f2aec0ac400 2021-04-09T18:31:04.859 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:04.859 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:31:04.860 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 189, 2021-04-09T18:31:04.860 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:31:04.861 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:31:04.861 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:30:47.109600", 2021-04-09T18:31:04.862 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:31:04.862 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:31:04.862 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:31:04.863 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:31:04.863 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:31:04.864 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:31:04.864 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:31:04.864 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:31:04.865 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:31:04.865 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:31:04.865 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:31:04.866 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:31:04.866 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:31:04.867 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:31:04.867 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:31:04.867 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 33, 2021-04-09T18:31:04.868 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:31:04.868 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:31:04.869 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:31:04.869 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:31:04.869 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:31:04.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.870 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:31:04.870 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:31:04.871 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:31:04.871 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:31:04.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:31:04.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:31:04.872 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:31:04.873 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:31:04.873 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:31:04.873 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:31:04.874 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:31:04.874 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:31:04.875 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:31:04.875 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:31:04.876 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:31:04.876 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:31:04.877 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:31:04.877 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:31:04.877 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:31:04.878 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:31:04.878 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:31:04.879 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:31:04.879 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:31:04.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.880 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:31:04.881 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:31:04.881 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:31:04.881 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:31:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:31:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:31:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:31:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:31:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:31:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:31:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:31:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:31:04.885 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:31:04.885 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:31:04.885 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:31:04.886 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:31:04.886 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:31:04.887 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:31:04.887 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:31:04.887 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:31:04.888 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:31:04.888 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:31:04.889 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:31:04.889 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:31:04.889 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:31:04.890 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:31:04.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:31:04.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.891 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:31:04.892 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:31:04.892 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:31:04.892 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:31:04.893 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:31:04.893 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:31:04.894 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:31:04.894 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:31:04.894 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:31:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:31:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:31:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:31:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:31:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:31:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 30, 2021-04-09T18:31:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "alpha_metadata", 2021-04-09T18:31:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:30:13.709244", 2021-04-09T18:31:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:31:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:31:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:31:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:31:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:31:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:31:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:31:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:31:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:31:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:31:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:31:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:31:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:31:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:31:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:31:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:31:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:31:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:31:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:31:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:31:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "179", 2021-04-09T18:31:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:31:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:31:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:31:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:31:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:31:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:31:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:31:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:31:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:31:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:31:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:31:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:31:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:31:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:31:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:31:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:31:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:31:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:31:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:31:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:31:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:31:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:31:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:31:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:31:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:31:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:31:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:31:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:31:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:31:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:31:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:31:04.917 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:31:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:31:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:31:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:31:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:31:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:31:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:31:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:31:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:31:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:31:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:31:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:31:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "alpha" 2021-04-09T18:31:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 31, 2021-04-09T18:31:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "alpha_data", 2021-04-09T18:31:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:30:14.523869", 2021-04-09T18:31:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:31:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:31:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:31:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:31:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:31:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:31:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:31:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:31:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:31:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:31:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:31:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:31:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:31:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:31:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:31:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:31:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:31:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:31:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:31:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:31:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "180", 2021-04-09T18:31:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:31:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:31:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:31:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:31:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:31:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:31:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:31:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:31:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:31:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:31:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:31:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:31:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:31:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:31:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:31:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:31:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:31:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:31:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:31:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:31:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:31:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:31:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:31:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:31:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:31:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:31:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:31:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:31:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:31:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:31:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:31:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:31:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:31:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:31:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:31:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:31:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:31:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:31:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:31:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:31:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "alpha" 2021-04-09T18:31:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 32, 2021-04-09T18:31:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "bravo_metadata", 2021-04-09T18:31:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:30:21.008022", 2021-04-09T18:31:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:31:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:31:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:31:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:31:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:31:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:31:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:31:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:31:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:31:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:31:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:31:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:31:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:31:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:31:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:31:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:31:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:31:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:31:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:31:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:31:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "183", 2021-04-09T18:31:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:31:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:31:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:31:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:31:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:31:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:31:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:31:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:31:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:31:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:31:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:31:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:31:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:31:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:31:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:31:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:31:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:31:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:31:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:31:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:31:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:31:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:31:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:31:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:31:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:31:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:31:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:31:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:31:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:31:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:31:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:31:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:31:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:31:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:31:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:31:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:31:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:31:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:31:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:31:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:31:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:31:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:31:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:31:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "bravo" 2021-04-09T18:31:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 33, 2021-04-09T18:31:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "bravo_data", 2021-04-09T18:31:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:30:22.220789", 2021-04-09T18:31:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:31:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:31:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:31:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:31:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:31:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:31:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:31:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:31:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:31:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:31:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:31:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:31:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:31:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:31:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:31:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:31:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:31:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:31:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:31:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:31:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "184", 2021-04-09T18:31:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:31:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:31:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:31:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:31:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:31:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:31:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:31:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:31:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:31:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:31:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:31:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:31:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:31:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:31:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:31:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:31:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:31:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:31:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:31:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:31:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:31:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:31:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:31:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:31:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:31:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:31:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:31:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:31:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:31:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:31:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:31:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:31:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:31:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:31:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:31:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:31:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:31:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:31:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:31:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:31:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "bravo" 2021-04-09T18:31:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:31:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:31:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:31:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:31:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 181, 2021-04-09T18:31:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:31:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:31:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:04.991 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:31:04.991 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:04.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:04.993 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:31:04.993 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:04.993 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.994 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:04.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.994 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:04.995 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:04.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.995 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:04.996 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:31:04.996 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:04.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:04.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:04.997 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:04.998 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:31:04.998 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:04.999 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:04.999 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:04.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.000 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:05.000 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.001 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:31:05.001 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:05.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.003 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.003 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:31:05.003 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:05.003 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.004 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.005 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:31:05.005 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:31:05.005 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:31:05.006 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:31:05.006 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:05.006 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:05.006 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:05.007 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.008 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:31:05.008 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:31:05.009 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:05.009 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:05.009 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:05.011 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 178, 2021-04-09T18:31:05.011 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:05.011 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:05.012 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:05.012 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.013 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:31:05.014 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:05.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.015 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.015 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:31:05.016 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:05.017 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.017 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.018 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:05.018 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.019 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.020 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:31:05.020 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:05.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.020 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:31:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.022 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.022 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:05.023 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.023 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.024 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:31:05.024 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:05.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.025 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.026 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:31:05.026 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:05.026 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.026 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.027 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:05.027 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:31:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:05.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.029 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:31:05.030 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:05.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.030 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.031 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:31:05.031 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:31:05.031 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:31:05.032 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:31:05.032 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:05.032 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:05.032 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:05.033 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.033 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:31:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:31:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:05.035 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:05.035 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:05.035 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:05.035 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 182, 2021-04-09T18:31:05.035 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:05.036 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:05.036 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:05.036 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:31:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.038 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:31:05.038 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:05.038 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:31:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:31:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.043 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.043 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:31:05.043 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:05.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.044 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.044 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:31:05.045 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:05.045 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.046 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.046 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:05.047 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.048 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:31:05.049 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:05.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.050 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.050 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:31:05.051 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:05.051 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.052 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.053 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:31:05.053 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:31:05.053 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:31:05.054 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:31:05.054 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:05.055 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:05.055 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:05.055 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.057 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:31:05.057 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:31:05.057 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:05.058 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:05.058 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:05.059 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:05.059 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:05.060 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:05.060 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:05.060 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 182, 2021-04-09T18:31:05.061 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:05.061 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:05.062 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:05.062 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.063 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:31:05.064 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:05.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.065 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:31:05.066 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:05.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.066 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.067 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:05.068 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.069 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.069 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:31:05.069 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:05.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.071 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.071 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:31:05.071 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:05.072 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.072 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.073 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:05.073 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.074 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.075 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:31:05.075 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:05.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.076 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.077 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:31:05.077 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:05.077 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.078 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.079 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:05.079 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.080 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.080 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:31:05.081 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:05.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.082 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:31:05.083 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:05.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.084 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.084 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:31:05.085 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:31:05.085 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:31:05.086 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:31:05.086 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:05.086 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:05.087 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:05.087 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.088 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:31:05.089 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:31:05.089 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:05.090 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:05.090 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:05.091 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:05.091 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:05.092 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:05.092 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:05.092 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 178, 2021-04-09T18:31:05.093 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:05.093 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:05.094 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:05.094 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.095 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.095 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:31:05.096 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:05.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:31:05.098 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:05.098 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.099 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.100 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:05.100 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.101 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:31:05.102 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:05.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:31:05.104 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:05.105 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.105 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.106 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:05.107 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:31:05.108 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:05.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.110 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:31:05.110 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:05.110 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.111 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.111 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:05.111 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.112 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:31:05.112 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:05.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.113 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:31:05.113 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:05.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.114 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.114 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:31:05.114 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:31:05.115 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:31:05.115 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:31:05.115 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:05.115 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:05.116 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:05.116 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.116 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:31:05.117 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:31:05.117 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:05.117 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:05.117 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:05.117 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:05.118 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:05.118 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:05.118 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:05.118 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 182, 2021-04-09T18:31:05.119 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:05.119 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:05.119 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:05.119 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.119 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.120 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.120 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:31:05.120 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:05.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.121 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:31:05.121 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:05.121 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.122 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.122 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:05.122 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:31:05.123 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:05.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.124 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:31:05.124 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:05.124 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.125 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.125 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:05.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.125 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.126 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:31:05.126 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:05.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.127 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.127 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:31:05.127 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:05.127 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.128 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.128 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:05.128 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.129 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:31:05.129 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:05.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:31:05.130 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:05.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.131 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.131 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:31:05.132 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:31:05.132 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:31:05.132 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:31:05.133 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:05.133 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:05.133 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:05.134 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.134 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.135 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:31:05.135 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:31:05.136 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:05.136 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:05.136 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:05.136 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:05.137 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:05.137 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:05.137 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:05.137 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 181, 2021-04-09T18:31:05.138 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:05.138 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:05.138 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:05.138 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.139 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:31:05.139 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:05.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.140 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.140 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:31:05.141 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:05.141 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.141 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.142 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:05.142 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.143 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.143 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:31:05.143 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:05.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.145 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.145 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:31:05.146 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:05.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.146 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.147 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:05.147 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.148 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.148 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:31:05.149 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:05.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.150 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:31:05.150 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:05.151 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.151 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.152 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:05.152 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.153 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.154 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:31:05.154 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:05.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.156 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:31:05.156 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:05.156 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.157 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.158 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:31:05.158 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:31:05.159 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:31:05.159 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:31:05.159 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:05.159 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:05.160 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:05.160 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.161 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:31:05.161 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:31:05.161 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:05.161 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:05.162 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:05.162 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:05.162 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:05.162 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:05.162 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:05.163 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 176, 2021-04-09T18:31:05.163 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:05.163 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:05.164 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:05.164 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.165 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.165 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:31:05.165 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:05.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.167 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.167 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:31:05.168 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:05.168 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.168 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.169 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:05.170 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.171 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.171 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:31:05.171 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:05.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.173 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.173 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:31:05.174 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:05.174 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.174 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.175 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:05.176 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.177 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.177 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:31:05.177 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:05.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.178 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.179 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:31:05.179 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:05.180 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.180 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.181 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:05.182 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:05.182 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.183 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:05.183 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:31:05.184 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:05.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.184 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.185 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:05.185 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:31:05.186 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:05.186 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.187 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.188 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:31:05.188 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:31:05.188 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:31:05.189 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:31:05.189 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:05.189 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:05.189 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:05.190 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:05.190 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.190 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:31:05.190 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:31:05.191 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.191 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:31:05.191 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:05.191 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:05.192 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:05.192 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:05.192 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:05.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.193 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.193 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:31:05.193 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:05.193 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:05.194 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:05.194 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:05.194 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:05.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.195 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.195 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:31:05.195 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:05.196 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:05.196 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:05.196 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:05.196 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:05.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.197 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:31:05.197 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:05.198 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:05.198 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:05.199 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:05.199 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:05.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.200 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.201 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:31:05.201 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:05.202 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:05.202 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:05.203 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:05.203 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:05.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.205 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:31:05.205 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:05.206 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:05.206 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:05.206 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:05.207 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:05.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.208 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.208 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:31:05.208 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:05.209 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:05.209 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:05.210 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:05.210 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:05.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:05.212 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:31:05.212 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:05.212 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:05.213 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:05.213 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:05.214 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:05.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.215 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:31:05.215 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:31:05.216 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:31:05.216 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:31:05.217 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:31:05.217 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:31:05.217 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/558658679": "2021-04-10 18:30:47.109571", 2021-04-09T18:31:05.218 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/3158551863": "2021-04-10 18:30:30.264043", 2021-04-09T18:31:05.218 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/558658679": "2021-04-10 18:30:47.109571", 2021-04-09T18:31:05.219 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/3158551863": "2021-04-10 18:30:30.264043", 2021-04-09T18:31:05.219 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/3729993394": "2021-04-10 18:30:30.261199", 2021-04-09T18:31:05.219 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/3729993394": "2021-04-10 18:30:30.261199" 2021-04-09T18:31:05.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.220 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:31:05.220 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:31:05.221 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:31:05.221 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:31:05.221 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:31:05.222 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:31:05.222 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:31:05.223 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:31:05.223 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:05.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:05.224 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:31:05.224 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:31:05.225 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:31:05.225 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:31:05.226 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:31:05.226 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:31:05.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2acc018930 msgr2=0x7f2acc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2acc018930 0x7f2acc01ae10 crc :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec0ac400 msgr2=0x7f2aec1a6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec0ac400 0x7f2aec1a6e30 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7f2adc006f60 tx=0x7f2adc00b970).stop 2021-04-09T18:31:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 shutdown_connections 2021-04-09T18:31:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2acc018930 0x7f2acc01ae10 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2aec0ac400 0x7f2aec1a6e30 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aec17a110 0x7f2aec1a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 --2- 172.21.15.32:0/2275843342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aec179bf0 0x7f2aec0abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 >> 172.21.15.32:0/2275843342 conn(0x7f2aec1acae0 msgr2=0x7f2aec17ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:05.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 shutdown_connections 2021-04-09T18:31:05.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:04.845 7f2af37dd700 1 -- 172.21.15.32:0/2275843342 wait complete. 2021-04-09T18:31:05.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:05.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.373 7f25170c2700 1 Processor -- start 2021-04-09T18:31:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.373 7f25170c2700 1 -- start start 2021-04-09T18:31:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25170c2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f25100d7ca0 0x7f25100d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25170c2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f251010da00 0x7f251010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25170c2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251010d480 con 0x7f251010d010 2021-04-09T18:31:05.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25170c2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25100d9220 con 0x7f251010da00 2021-04-09T18:31:05.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25170c2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25100d86c0 con 0x7f25100d7ca0 2021-04-09T18:31:05.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25156df700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f251010da00 0x7f251010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f2514ede700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f251010d010 0x7f25100d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37750/0 (socket says 172.21.15.32:37750) 2021-04-09T18:31:05.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f2514ede700 1 -- 172.21.15.32:0/1411532332 learned_addr learned my addr 172.21.15.32:0/1411532332 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:05.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f2507fff700 1 --2- 172.21.15.32:0/1411532332 >> v2:172.21.15.86:3300/0 conn(0x7f25100d7ca0 0x7f25100d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25156df700 1 -- 172.21.15.32:0/1411532332 >> v2:172.21.15.86:3300/0 conn(0x7f25100d7ca0 msgr2=0x7f25100d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:05.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25156df700 1 --2- 172.21.15.32:0/1411532332 >> v2:172.21.15.86:3300/0 conn(0x7f25100d7ca0 0x7f25100d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25156df700 1 -- 172.21.15.32:0/1411532332 >> v1:172.21.15.32:6789/0 conn(0x7f251010d010 legacy=0x7f25100d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25077fe700 1 -- 172.21.15.32:0/1411532332 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827974499 0 0) 0x7f250c000ce0 con 0x7f251010d010 2021-04-09T18:31:05.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25156df700 1 -- 172.21.15.32:0/1411532332 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25100d1f70 con 0x7f251010da00 2021-04-09T18:31:05.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25156df700 1 --2- 172.21.15.32:0/1411532332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f251010da00 0x7f251010de00 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7f2508006e30 tx=0x7f2508009de0).ready entity=mon.0 client_cookie=2e0bd29d5c322c8f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:05.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25077fe700 1 -- 172.21.15.32:0/1411532332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250800b300 con 0x7f251010da00 2021-04-09T18:31:05.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25077fe700 1 -- 172.21.15.32:0/1411532332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250800b680 con 0x7f251010da00 2021-04-09T18:31:05.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25077fe700 1 -- 172.21.15.32:0/1411532332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250800bb60 con 0x7f251010da00 2021-04-09T18:31:05.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25170c2700 1 -- 172.21.15.32:0/1411532332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f251010da00 msgr2=0x7f251010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.374 7f25170c2700 1 --2- 172.21.15.32:0/1411532332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f251010da00 0x7f251010de00 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7f2508006e30 tx=0x7f2508009de0).stop 2021-04-09T18:31:05.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.375 7f25170c2700 1 -- 172.21.15.32:0/1411532332 shutdown_connections 2021-04-09T18:31:05.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.375 7f25170c2700 1 --2- 172.21.15.32:0/1411532332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f251010da00 0x7f251010de00 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.375 7f25170c2700 1 --2- 172.21.15.32:0/1411532332 >> v2:172.21.15.86:3300/0 conn(0x7f25100d7ca0 0x7f25100d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.375 7f25170c2700 1 -- 172.21.15.32:0/1411532332 >> 172.21.15.32:0/1411532332 conn(0x7f25100ce5f0 msgr2=0x7f25100d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:05.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.375 7f25170c2700 1 -- 172.21.15.32:0/1411532332 shutdown_connections 2021-04-09T18:31:05.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.375 7f25170c2700 1 -- 172.21.15.32:0/1411532332 wait complete. 2021-04-09T18:31:05.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.395 7f25170c2700 1 Processor -- start 2021-04-09T18:31:05.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.395 7f25170c2700 1 -- start start 2021-04-09T18:31:05.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f25170c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2510179bf0 0x7f2510179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f25170c2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f251017c540 0x7f251017c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f25170c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25100aca60 0x7f25100ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f25170c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251017cf60 con 0x7f2510179bf0 2021-04-09T18:31:05.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f25170c2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25100ad480 con 0x7f25100aca60 2021-04-09T18:31:05.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f25170c2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25100ad6a0 con 0x7f251017c540 2021-04-09T18:31:05.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f2514ede700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2510179bf0 0x7f2510179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f2514ede700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2510179bf0 0x7f2510179ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56398/0 (socket says 172.21.15.32:56398) 2021-04-09T18:31:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f2514ede700 1 -- 172.21.15.32:0/2143953125 learned_addr learned my addr 172.21.15.32:0/2143953125 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:05.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f25156df700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25100aca60 0x7f25100ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f2507fff700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f251017c540 0x7f251017c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f2514ede700 1 -- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f251017c540 msgr2=0x7f251017c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f2514ede700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f251017c540 0x7f251017c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f2514ede700 1 -- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25100aca60 msgr2=0x7f25100ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f2514ede700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25100aca60 0x7f25100ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f2514ede700 1 -- 172.21.15.32:0/2143953125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25100ad8c0 con 0x7f2510179bf0 2021-04-09T18:31:05.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.396 7f25156df700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25100aca60 0x7f25100ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:05.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.397 7f2514ede700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2510179bf0 0x7f2510179ff0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f250c00b530 tx=0x7f250c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.397 7f25057fa700 1 -- 172.21.15.32:0/2143953125 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250c006020 con 0x7f2510179bf0 2021-04-09T18:31:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.397 7f25057fa700 1 -- 172.21.15.32:0/2143953125 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c0063a0 con 0x7f2510179bf0 2021-04-09T18:31:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.397 7f25170c2700 1 -- 172.21.15.32:0/2143953125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25100ad8c0 con 0x7f2510179bf0 2021-04-09T18:31:05.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.397 7f25057fa700 1 -- 172.21.15.32:0/2143953125 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250c004590 con 0x7f2510179bf0 2021-04-09T18:31:05.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.397 7f25170c2700 1 -- 172.21.15.32:0/2143953125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25100ad8c0 con 0x7f2510179bf0 2021-04-09T18:31:05.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.397 7f25057fa700 1 -- 172.21.15.32:0/2143953125 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f250c00a000 con 0x7f2510179bf0 2021-04-09T18:31:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.398 7f25057fa700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24f0018930 0x7f24f001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.398 7f25057fa700 1 -- 172.21.15.32:0/2143953125 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f250c032e30 con 0x7f2510179bf0 2021-04-09T18:31:05.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.398 7f2507fff700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24f0018930 0x7f24f001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.398 7f2507fff700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24f0018930 0x7f24f001adc0 crc :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:05.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.417 7f25170c2700 1 -- 172.21.15.32:0/2143953125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251017cf60 con 0x7f2510179bf0 2021-04-09T18:31:05.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.421 7f25057fa700 1 -- 172.21.15.32:0/2143953125 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f250c006b90 con 0x7f2510179bf0 2021-04-09T18:31:05.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.613 7f25170c2700 1 -- 172.21.15.32:0/2143953125 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f251017cf60 con 0x7f2510179bf0 2021-04-09T18:31:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.614 7f25057fa700 1 -- 172.21.15.32:0/2143953125 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+3381 (secure 0 0 0) 0x7f250c00a000 con 0x7f2510179bf0 2021-04-09T18:31:05.630 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 338 2021-04-09T18:31:05.630 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:05.631 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":338,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:30:33.326504","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":185,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11136":{"gid":11136,"name":"d","rank":0,"incarnation":331,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/264493238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":264493238},{"type":"v1","addr":"172.21.15.86:6836","nonce":264493238}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":338,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:30:50.619386","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":189,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":58739},"failed":[],"damaged":[],"stopped":[],"info":{"gid_58739":{"gid":58739,"name":"c","rank":0,"incarnation":335,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6835/2933940031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2933940031},{"type":"v1","addr":"172.21.15.32:6835","nonce":2933940031}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:31:05.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 -- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24f0018930 msgr2=0x7f24f001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24f0018930 0x7f24f001adc0 crc :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 -- 172.21.15.32:0/2143953125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2510179bf0 msgr2=0x7f2510179ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2510179bf0 0x7f2510179ff0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f250c00b530 tx=0x7f250c00b8d0).stop 2021-04-09T18:31:05.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 -- 172.21.15.32:0/2143953125 shutdown_connections 2021-04-09T18:31:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24f0018930 0x7f24f001adc0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25100aca60 0x7f25100ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f251017c540 0x7f251017c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 --2- 172.21.15.32:0/2143953125 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2510179bf0 0x7f2510179ff0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 -- 172.21.15.32:0/2143953125 >> 172.21.15.32:0/2143953125 conn(0x7f25101acae0 msgr2=0x7f251017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 -- 172.21.15.32:0/2143953125 shutdown_connections 2021-04-09T18:31:05.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.616 7f25170c2700 1 -- 172.21.15.32:0/2143953125 wait complete. 2021-04-09T18:31:05.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail alpha 2021-04-09T18:31:05.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.780 7f38d9b15700 1 Processor -- start 2021-04-09T18:31:05.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.780 7f38d9b15700 1 -- start start 2021-04-09T18:31:05.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d9b15700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38d40cffa0 0x7f38d40d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d9b15700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d40d7800 0x7f38d40d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d9b15700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f38d40d8220 0x7f38d40d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d9b15700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d40d91e0 con 0x7f38d40cffa0 2021-04-09T18:31:05.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d9b15700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d40d7280 con 0x7f38d40d8220 2021-04-09T18:31:05.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d9b15700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40d6290 con 0x7f38d40d7800 2021-04-09T18:31:05.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d37fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38d40cffa0 0x7f38d40d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d37fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f38d40cffa0 0x7f38d40d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56406/0 (socket says 172.21.15.32:56406) 2021-04-09T18:31:05.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d37fe700 1 -- 172.21.15.32:0/164840536 learned_addr learned my addr 172.21.15.32:0/164840536 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d3fff700 1 --2- 172.21.15.32:0/164840536 >> v2:172.21.15.86:3300/0 conn(0x7f38d40d8220 0x7f38d40d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d2ffd700 1 --2- 172.21.15.32:0/164840536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d40d7800 0x7f38d40d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d37fe700 1 -- 172.21.15.32:0/164840536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d40d7800 msgr2=0x7f38d40d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:05.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d37fe700 1 --2- 172.21.15.32:0/164840536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d40d7800 0x7f38d40d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d37fe700 1 -- 172.21.15.32:0/164840536 >> v2:172.21.15.86:3300/0 conn(0x7f38d40d8220 msgr2=0x7f38d40d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:05.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d37fe700 1 --2- 172.21.15.32:0/164840536 >> v2:172.21.15.86:3300/0 conn(0x7f38d40d8220 0x7f38d40d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d37fe700 1 -- 172.21.15.32:0/164840536 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d40d64b0 con 0x7f38d40cffa0 2021-04-09T18:31:05.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d3fff700 1 --2- 172.21.15.32:0/164840536 >> v2:172.21.15.86:3300/0 conn(0x7f38d40d8220 0x7f38d40d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.781 7f38d37fe700 1 --2- 172.21.15.32:0/164840536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d40cffa0 0x7f38d40d2440 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 rx=0x7f38c4006700 tx=0x7f38c4009b00).ready entity=mon.0 client_cookie=74ba55da17cd1efe server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d27fc700 1 -- 172.21.15.32:0/164840536 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38c400c0c0 con 0x7f38d40cffa0 2021-04-09T18:31:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d27fc700 1 -- 172.21.15.32:0/164840536 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c400c400 con 0x7f38d40cffa0 2021-04-09T18:31:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d27fc700 1 -- 172.21.15.32:0/164840536 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38c400c8e0 con 0x7f38d40cffa0 2021-04-09T18:31:05.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d9b15700 1 -- 172.21.15.32:0/164840536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d40cffa0 msgr2=0x7f38d40d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d9b15700 1 --2- 172.21.15.32:0/164840536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d40cffa0 0x7f38d40d2440 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 rx=0x7f38c4006700 tx=0x7f38c4009b00).stop 2021-04-09T18:31:05.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d9b15700 1 -- 172.21.15.32:0/164840536 shutdown_connections 2021-04-09T18:31:05.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d9b15700 1 --2- 172.21.15.32:0/164840536 >> v2:172.21.15.86:3300/0 conn(0x7f38d40d8220 0x7f38d40d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d9b15700 1 --2- 172.21.15.32:0/164840536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d40d7800 0x7f38d40d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d9b15700 1 --2- 172.21.15.32:0/164840536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d40cffa0 0x7f38d40d2440 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d9b15700 1 -- 172.21.15.32:0/164840536 >> 172.21.15.32:0/164840536 conn(0x7f38d40ce5c0 msgr2=0x7f38d40cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:05.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d9b15700 1 -- 172.21.15.32:0/164840536 shutdown_connections 2021-04-09T18:31:05.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.782 7f38d9b15700 1 -- 172.21.15.32:0/164840536 wait complete. 2021-04-09T18:31:05.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.802 7f38d9b15700 1 Processor -- start 2021-04-09T18:31:05.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d9b15700 1 -- start start 2021-04-09T18:31:05.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d9b15700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d408d210 0x7f38d408fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d9b15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d40903f0 0x7f38d41a5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d9b15700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d41a6070 0x7f38d41a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d9b15700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d408d690 con 0x7f38d41a6070 2021-04-09T18:31:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d9b15700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40b0fd0 con 0x7f38d40903f0 2021-04-09T18:31:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d9b15700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d41a6470 con 0x7f38d408d210 2021-04-09T18:31:05.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d41a6070 0x7f38d41a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d41a6070 0x7f38d41a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56416/0 (socket says 172.21.15.32:56416) 2021-04-09T18:31:05.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d3fff700 1 -- 172.21.15.32:0/1630001847 learned_addr learned my addr 172.21.15.32:0/1630001847 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:05.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d37fe700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d408d210 0x7f38d408fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.803 7f38d2ffd700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d40903f0 0x7f38d41a5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38d3fff700 1 -- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d408d210 msgr2=0x7f38d408fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38d3fff700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d408d210 0x7f38d408fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38d3fff700 1 -- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d40903f0 msgr2=0x7f38d41a5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:05.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38d3fff700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d40903f0 0x7f38d41a5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:05.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38d3fff700 1 -- 172.21.15.32:0/1630001847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d408d8b0 con 0x7f38d41a6070 2021-04-09T18:31:05.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38d3fff700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d41a6070 0x7f38d41a46a0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7f38cc00b930 tx=0x7f38cc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:05.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38bbfff700 1 -- 172.21.15.32:0/1630001847 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38cc006030 con 0x7f38d41a6070 2021-04-09T18:31:05.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38bbfff700 1 -- 172.21.15.32:0/1630001847 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38cc006370 con 0x7f38d41a6070 2021-04-09T18:31:05.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d408d8b0 con 0x7f38d41a6070 2021-04-09T18:31:05.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38bbfff700 1 -- 172.21.15.32:0/1630001847 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38cc005930 con 0x7f38d41a6070 2021-04-09T18:31:05.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.804 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d408d8b0 con 0x7f38d41a6070 2021-04-09T18:31:05.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.805 7f38bbfff700 1 -- 172.21.15.32:0/1630001847 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f38cc006550 con 0x7f38d41a6070 2021-04-09T18:31:05.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.805 7f38bbfff700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38b4018930 0x7f38b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:05.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.805 7f38bbfff700 1 -- 172.21.15.32:0/1630001847 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f38cc033260 con 0x7f38d41a6070 2021-04-09T18:31:05.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.805 7f38d37fe700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38b4018930 0x7f38b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:05.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.805 7f38d37fe700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38b4018930 0x7f38b401ae10 crc :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:05.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.825 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d41a52a0 con 0x7f38d41a6070 2021-04-09T18:31:05.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:05.828 7f38bbfff700 1 -- 172.21.15.32:0/1630001847 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f38cc00be10 con 0x7f38d41a6070 2021-04-09T18:31:06.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.010 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "alpha"} v 0) v1 -- 0x7f38d41a52a0 con 0x7f38d41a6070 2021-04-09T18:31:06.537 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:31:06.522 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:31:06.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.524 7f38bbfff700 1 -- 172.21.15.32:0/1630001847 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "alpha"}]=0 alpha marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v339) v1 ==== 159+0+0 (secure 0 0 0) 0x7f38cc006410 con 0x7f38d41a6070 2021-04-09T18:31:06.590 INFO:teuthology.orchestra.run.smithi032.stderr:alpha marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:31:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.576 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38b4018930 msgr2=0x7f38b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.576 7f38d9b15700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38b4018930 0x7f38b401ae10 crc :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.576 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d41a6070 msgr2=0x7f38d41a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.576 7f38d9b15700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d41a6070 0x7f38d41a46a0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7f38cc00b930 tx=0x7f38cc00bcd0).stop 2021-04-09T18:31:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.577 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 shutdown_connections 2021-04-09T18:31:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.577 7f38d9b15700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f38b4018930 0x7f38b401ae10 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.577 7f38d9b15700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38d41a6070 0x7f38d41a46a0 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.577 7f38d9b15700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d40903f0 0x7f38d41a5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.577 7f38d9b15700 1 --2- 172.21.15.32:0/1630001847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38d408d210 0x7f38d408fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.577 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 >> 172.21.15.32:0/1630001847 conn(0x7f38d41a8f40 msgr2=0x7f38d4188050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.577 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 shutdown_connections 2021-04-09T18:31:06.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.577 7f38d9b15700 1 -- 172.21.15.32:0/1630001847 wait complete. 2021-04-09T18:31:06.598 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:31:06.603 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail bravo 2021-04-09T18:31:06.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.739 7fc482343700 1 Processor -- start 2021-04-09T18:31:06.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.739 7fc482343700 1 -- start start 2021-04-09T18:31:06.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc482343700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc47c0d6c90 0x7fc47c0d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:06.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc482343700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0d2450 0x7fc47c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:06.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc482343700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47c0d7290 con 0x7fc47c0cffb0 2021-04-09T18:31:06.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc482343700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47c0d7090 con 0x7fc47c0d6c90 2021-04-09T18:31:06.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc482343700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c0d91f0 con 0x7fc47c0d2450 2021-04-09T18:31:06.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc480960700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0d2450 0x7fc47c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:06.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc473fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc47c0d6c90 0x7fc47c0d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:06.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc480960700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0d2450 0x7fc47c10cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:35416/0 (socket says 172.21.15.32:35416) 2021-04-09T18:31:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc47bfff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc47c0cffb0 0x7fc47c0d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:35704/0 (socket says 172.21.15.32:35704) 2021-04-09T18:31:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.740 7fc480960700 1 -- 172.21.15.32:0/821447058 learned_addr learned my addr 172.21.15.32:0/821447058 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc480960700 1 -- 172.21.15.32:0/821447058 >> v2:172.21.15.86:3300/0 conn(0x7fc47c0d6c90 msgr2=0x7fc47c0d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc480960700 1 --2- 172.21.15.32:0/821447058 >> v2:172.21.15.86:3300/0 conn(0x7fc47c0d6c90 0x7fc47c0d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc480960700 1 -- 172.21.15.32:0/821447058 >> v1:172.21.15.86:6789/0 conn(0x7fc47c0cffb0 legacy=0x7fc47c0d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:06.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc480960700 1 -- 172.21.15.32:0/821447058 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c1120d0 con 0x7fc47c0d2450 2021-04-09T18:31:06.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc480960700 1 --2- 172.21.15.32:0/821447058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0d2450 0x7fc47c10cfe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fc474006aa0 tx=0x7fc474009f20).ready entity=mon.2 client_cookie=e5862355fcdef66e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:06.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc47b7fe700 1 -- 172.21.15.32:0/821447058 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc47400b5c0 con 0x7fc47c0d2450 2021-04-09T18:31:06.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc47b7fe700 1 -- 172.21.15.32:0/821447058 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc474003030 con 0x7fc47c0d2450 2021-04-09T18:31:06.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc47b7fe700 1 -- 172.21.15.32:0/821447058 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc47400bb40 con 0x7fc47c0d2450 2021-04-09T18:31:06.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc482343700 1 -- 172.21.15.32:0/821447058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0d2450 msgr2=0x7fc47c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:06.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.741 7fc482343700 1 --2- 172.21.15.32:0/821447058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0d2450 0x7fc47c10cfe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fc474006aa0 tx=0x7fc474009f20).stop 2021-04-09T18:31:06.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.742 7fc482343700 1 -- 172.21.15.32:0/821447058 shutdown_connections 2021-04-09T18:31:06.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.742 7fc482343700 1 --2- 172.21.15.32:0/821447058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0d2450 0x7fc47c10cfe0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.742 7fc482343700 1 --2- 172.21.15.32:0/821447058 >> v2:172.21.15.86:3300/0 conn(0x7fc47c0d6c90 0x7fc47c0d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.742 7fc482343700 1 -- 172.21.15.32:0/821447058 >> 172.21.15.32:0/821447058 conn(0x7fc47c0ce5d0 msgr2=0x7fc47c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:06.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.742 7fc482343700 1 -- 172.21.15.32:0/821447058 shutdown_connections 2021-04-09T18:31:06.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.742 7fc482343700 1 -- 172.21.15.32:0/821447058 wait complete. 2021-04-09T18:31:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.762 7fc482343700 1 Processor -- start 2021-04-09T18:31:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.762 7fc482343700 1 -- start start 2021-04-09T18:31:06.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc482343700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc47c08d210 0x7fc47c08fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:06.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc482343700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0903f0 0x7fc47c1a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:06.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc482343700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47c1a6080 0x7fc47c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:06.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc482343700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c08d690 con 0x7fc47c1a6080 2021-04-09T18:31:06.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc482343700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c0b0fd0 con 0x7fc47c08d210 2021-04-09T18:31:06.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc482343700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c1a6480 con 0x7fc47c0903f0 2021-04-09T18:31:06.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc480960700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47c1a6080 0x7fc47c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:06.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc47bfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc47c08d210 0x7fc47c08fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:06.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc480960700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47c1a6080 0x7fc47c1a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56430/0 (socket says 172.21.15.32:56430) 2021-04-09T18:31:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc480960700 1 -- 172.21.15.32:0/2894330436 learned_addr learned my addr 172.21.15.32:0/2894330436 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc473fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0903f0 0x7fc47c1a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc47bfff700 1 -- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0903f0 msgr2=0x7fc47c1a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:06.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc47bfff700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0903f0 0x7fc47c1a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc47bfff700 1 -- 172.21.15.32:0/2894330436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47c1a6080 msgr2=0x7fc47c1a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc47bfff700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47c1a6080 0x7fc47c1a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc47bfff700 1 -- 172.21.15.32:0/2894330436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c08d8b0 con 0x7fc47c08d210 2021-04-09T18:31:06.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc480960700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47c1a6080 0x7fc47c1a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc473fff700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0903f0 0x7fc47c1a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc47bfff700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc47c08d210 0x7fc47c08fdd0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7fc46c00a4f0 tx=0x7fc46c00a890).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc4797fa700 1 -- 172.21.15.32:0/2894330436 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc46c0070d0 con 0x7fc47c08d210 2021-04-09T18:31:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.763 7fc4797fa700 1 -- 172.21.15.32:0/2894330436 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c00ac80 con 0x7fc47c08d210 2021-04-09T18:31:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.764 7fc482343700 1 -- 172.21.15.32:0/2894330436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47c08d8b0 con 0x7fc47c08d210 2021-04-09T18:31:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.764 7fc4797fa700 1 -- 172.21.15.32:0/2894330436 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc46c0045a0 con 0x7fc47c08d210 2021-04-09T18:31:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.764 7fc482343700 1 -- 172.21.15.32:0/2894330436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47c08d8b0 con 0x7fc47c08d210 2021-04-09T18:31:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.764 7fc4797fa700 1 -- 172.21.15.32:0/2894330436 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc46c004320 con 0x7fc47c08d210 2021-04-09T18:31:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.764 7fc4797fa700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4600188d0 0x7fc46001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.764 7fc4797fa700 1 -- 172.21.15.32:0/2894330436 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6733+0+0 (secure 0 0 0) 0x7fc46c032f40 con 0x7fc47c08d210 2021-04-09T18:31:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.764 7fc473fff700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4600188d0 0x7fc46001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.765 7fc473fff700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4600188d0 0x7fc46001add0 crc :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:06.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.784 7fc482343700 1 -- 172.21.15.32:0/2894330436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47c1a52b0 con 0x7fc47c08d210 2021-04-09T18:31:06.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.788 7fc4797fa700 1 -- 172.21.15.32:0/2894330436 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc46c000c10 con 0x7fc47c08d210 2021-04-09T18:31:06.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:06.970 7fc482343700 1 -- 172.21.15.32:0/2894330436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "bravo"} v 0) v1 -- 0x7fc47c1a52b0 con 0x7fc47c08d210 2021-04-09T18:31:07.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.540 7fc4797fa700 1 -- 172.21.15.32:0/2894330436 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "bravo"}]=0 bravo marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v340) v1 ==== 159+0+0 (secure 0 0 0) 0x7fc46c01f660 con 0x7fc47c08d210 2021-04-09T18:31:07.600 INFO:teuthology.orchestra.run.smithi032.stderr:bravo marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:31:07.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 -- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4600188d0 msgr2=0x7fc46001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:07.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4600188d0 0x7fc46001add0 crc :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:07.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 -- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc47c08d210 msgr2=0x7fc47c08fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:07.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc47c08d210 0x7fc47c08fdd0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7fc46c00a4f0 tx=0x7fc46c00a890).stop 2021-04-09T18:31:07.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 -- 172.21.15.32:0/2894330436 shutdown_connections 2021-04-09T18:31:07.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4600188d0 0x7fc46001add0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:07.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc47c1a6080 0x7fc47c1a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:07.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc47c0903f0 0x7fc47c1a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:07.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 --2- 172.21.15.32:0/2894330436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc47c08d210 0x7fc47c08fdd0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:07.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 -- 172.21.15.32:0/2894330436 >> 172.21.15.32:0/2894330436 conn(0x7fc47c1a8f50 msgr2=0x7fc47c188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:07.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 -- 172.21.15.32:0/2894330436 shutdown_connections 2021-04-09T18:31:07.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.587 7fc482343700 1 -- 172.21.15.32:0/2894330436 wait complete. 2021-04-09T18:31:07.613 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:07.615 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:31:07.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.746 7f3dee1ff700 1 Processor -- start 2021-04-09T18:31:07.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.746 7f3dee1ff700 1 -- start start 2021-04-09T18:31:07.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3dee1ff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3de810da00 0x7f3de810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:07.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3dee1ff700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de810d480 con 0x7f3de810d010 2021-04-09T18:31:07.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3dee1ff700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de80d1f70 con 0x7f3de80d7ca0 2021-04-09T18:31:07.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3dee1ff700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de80d8d30 con 0x7f3de810da00 2021-04-09T18:31:07.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3de7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3de810d010 0x7f3de80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37804/0 (socket says 172.21.15.32:37804) 2021-04-09T18:31:07.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3de7fff700 1 -- 172.21.15.32:0/3680968783 learned_addr learned my addr 172.21.15.32:0/3680968783 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:07.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3dec81c700 1 --2- 172.21.15.32:0/3680968783 >> v2:172.21.15.86:3300/0 conn(0x7f3de810da00 0x7f3de810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:07.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3de77fe700 1 -- 172.21.15.32:0/3680968783 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946490278 0 0) 0x7f3dd8000ce0 con 0x7f3de810d010 2021-04-09T18:31:07.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3de77fe700 1 -- 172.21.15.32:0/3680968783 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd4002fe0 con 0x7f3de810d010 2021-04-09T18:31:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3de77fe700 1 -- 172.21.15.32:0/3680968783 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813944319 0 0) 0x7f3dd0000c60 con 0x7f3de80d7ca0 2021-04-09T18:31:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.747 7f3de77fe700 1 -- 172.21.15.32:0/3680968783 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd4003020 con 0x7f3de80d7ca0 2021-04-09T18:31:07.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.748 7f3dec81c700 1 -- 172.21.15.32:0/3680968783 >> v1:172.21.15.86:6789/0 conn(0x7f3de80d7ca0 legacy=0x7f3de80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:07.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.748 7f3dec81c700 1 -- 172.21.15.32:0/3680968783 >> v1:172.21.15.32:6789/0 conn(0x7f3de810d010 legacy=0x7f3de80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:07.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.748 7f3dec81c700 1 -- 172.21.15.32:0/3680968783 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de80d9220 con 0x7f3de810da00 2021-04-09T18:31:07.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.748 7f3dec81c700 1 --2- 172.21.15.32:0/3680968783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de810da00 0x7f3de810de00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f3de0006e50 tx=0x7f3de0009e70).ready entity=mon.1 client_cookie=8318a806031df5c0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:07.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.748 7f3de77fe700 1 -- 172.21.15.32:0/3680968783 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3de0006a20 con 0x7f3de810da00 2021-04-09T18:31:07.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.748 7f3de77fe700 1 -- 172.21.15.32:0/3680968783 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de0006790 con 0x7f3de810da00 2021-04-09T18:31:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.748 7f3de77fe700 1 -- 172.21.15.32:0/3680968783 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3de000b700 con 0x7f3de810da00 2021-04-09T18:31:07.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.748 7f3dee1ff700 1 -- 172.21.15.32:0/3680968783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de810da00 msgr2=0x7f3de810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:07.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.748 7f3dee1ff700 1 --2- 172.21.15.32:0/3680968783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de810da00 0x7f3de810de00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f3de0006e50 tx=0x7f3de0009e70).stop 2021-04-09T18:31:07.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.749 7f3dee1ff700 1 -- 172.21.15.32:0/3680968783 shutdown_connections 2021-04-09T18:31:07.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.749 7f3dee1ff700 1 --2- 172.21.15.32:0/3680968783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de810da00 0x7f3de810de00 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:07.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.749 7f3dee1ff700 1 -- 172.21.15.32:0/3680968783 >> 172.21.15.32:0/3680968783 conn(0x7f3de80ce5f0 msgr2=0x7f3de80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:07.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.749 7f3dee1ff700 1 -- 172.21.15.32:0/3680968783 shutdown_connections 2021-04-09T18:31:07.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.749 7f3dee1ff700 1 -- 172.21.15.32:0/3680968783 wait complete. 2021-04-09T18:31:07.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.769 7f3dee1ff700 1 Processor -- start 2021-04-09T18:31:07.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.769 7f3dee1ff700 1 -- start start 2021-04-09T18:31:07.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.769 7f3dee1ff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de8179610 0x7f3de8179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:07.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.769 7f3dee1ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3de817a6e0 0x7f3de817aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:07.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.769 7f3dee1ff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de817b100 0x7f3de808db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:07.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.769 7f3dee1ff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de817d720 con 0x7f3de817a6e0 2021-04-09T18:31:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.769 7f3dee1ff700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de8185ba0 con 0x7f3de8179610 2021-04-09T18:31:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.769 7f3dee1ff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de8185d50 con 0x7f3de817b100 2021-04-09T18:31:07.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.769 7f3ddffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3de817a6e0 0x7f3de817aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:07.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3ddffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3de817a6e0 0x7f3de817aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56454/0 (socket says 172.21.15.32:56454) 2021-04-09T18:31:07.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3ddffff700 1 -- 172.21.15.32:0/4154295346 learned_addr learned my addr 172.21.15.32:0/4154295346 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:07.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3de7fff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de8179610 0x7f3de8179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:07.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3dec81c700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de817b100 0x7f3de808db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:07.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3ddffff700 1 -- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de817b100 msgr2=0x7f3de808db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:07.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3ddffff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de817b100 0x7f3de808db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:07.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3ddffff700 1 -- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de8179610 msgr2=0x7f3de8179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:07.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3ddffff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de8179610 0x7f3de8179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:07.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3ddffff700 1 -- 172.21.15.32:0/4154295346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de808e1b0 con 0x7f3de817a6e0 2021-04-09T18:31:07.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3dec81c700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de817b100 0x7f3de808db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:07.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3de7fff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de8179610 0x7f3de8179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:07.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3ddffff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3de817a6e0 0x7f3de817aae0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 rx=0x7f3dd0006f10 tx=0x7f3dd000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:07.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3de57fa700 1 -- 172.21.15.32:0/4154295346 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3dd000bd80 con 0x7f3de817a6e0 2021-04-09T18:31:07.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3de57fa700 1 -- 172.21.15.32:0/4154295346 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd0000e20 con 0x7f3de817a6e0 2021-04-09T18:31:07.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de808e1b0 con 0x7f3de817a6e0 2021-04-09T18:31:07.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3de57fa700 1 -- 172.21.15.32:0/4154295346 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3dd00054f0 con 0x7f3de817a6e0 2021-04-09T18:31:07.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.770 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de808e1b0 con 0x7f3de817a6e0 2021-04-09T18:31:07.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.771 7f3de57fa700 1 -- 172.21.15.32:0/4154295346 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3dd0009fe0 con 0x7f3de817a6e0 2021-04-09T18:31:07.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.771 7f3de57fa700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3dcc0188d0 0x7f3dcc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:07.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.771 7f3de57fa700 1 -- 172.21.15.32:0/4154295346 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f3dd0032ee0 con 0x7f3de817a6e0 2021-04-09T18:31:07.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.771 7f3de7fff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3dcc0188d0 0x7f3dcc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:07.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.771 7f3de7fff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3dcc0188d0 0x7f3dcc01add0 crc :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:07.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.791 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de817a1b0 con 0x7f3de817a6e0 2021-04-09T18:31:07.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.794 7f3de57fa700 1 -- 172.21.15.32:0/4154295346 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3dd0000b80 con 0x7f3de817a6e0 2021-04-09T18:31:08.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.987 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3de817a1b0 con 0x7f3de817a6e0 2021-04-09T18:31:08.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.987 7f3de57fa700 1 -- 172.21.15.32:0/4154295346 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 340 v340) v1 ==== 95+0+2684 (secure 0 0 0) 0x7f3dd0000e20 con 0x7f3de817a6e0 2021-04-09T18:31:08.004 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 340 2021-04-09T18:31:08.004 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:08.004 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":340,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330}],"filesystems":[{"mdsmap":{"epoch":339,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:15.764321","modified":"2021-04-09 18:31:06.523579","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":340,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:30:23.208891","modified":"2021-04-09 18:31:07.533682","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":191,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":16}]} 2021-04-09T18:31:08.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3dcc0188d0 msgr2=0x7f3dcc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3dcc0188d0 0x7f3dcc01add0 crc :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3de817a6e0 msgr2=0x7f3de817aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3de817a6e0 0x7f3de817aae0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 rx=0x7f3dd0006f10 tx=0x7f3dd000b990).stop 2021-04-09T18:31:08.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 shutdown_connections 2021-04-09T18:31:08.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3dcc0188d0 0x7f3dcc01add0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3de817a6e0 0x7f3de817aae0 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de817b100 0x7f3de808db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 --2- 172.21.15.32:0/4154295346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de8179610 0x7f3de8179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 >> 172.21.15.32:0/4154295346 conn(0x7f3de8190550 msgr2=0x7f3de81b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:08.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 shutdown_connections 2021-04-09T18:31:08.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:07.990 7f3dee1ff700 1 -- 172.21.15.32:0/4154295346 wait complete. 2021-04-09T18:31:08.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm alpha --yes-i-really-mean-it 2021-04-09T18:31:08.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:31:08.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:31:08.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.152 7fa5c3b5c700 1 Processor -- start 2021-04-09T18:31:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.152 7fa5c3b5c700 1 -- start start 2021-04-09T18:31:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c3b5c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5bc0cffb0 0x7fa5bc0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c3b5c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0d6c90 0x7fa5bc10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c3b5c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0d7290 con 0x7fa5bc0d7810 2021-04-09T18:31:08.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c3b5c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5bc0d7090 con 0x7fa5bc0cffb0 2021-04-09T18:31:08.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c3b5c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0d91f0 con 0x7fa5bc0d6c90 2021-04-09T18:31:08.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c2179700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0d6c90 0x7fa5bc10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c1978700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5bc0cffb0 0x7fa5bc0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c1177700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa5bc0d7810 0x7fa5bc0d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:35748/0 (socket says 172.21.15.32:35748) 2021-04-09T18:31:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c1177700 1 -- 172.21.15.32:0/3313502740 learned_addr learned my addr 172.21.15.32:0/3313502740 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c0976700 1 -- 172.21.15.32:0/3313502740 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326155659 0 0) 0x7fa5ac0009b0 con 0x7fa5bc0d7810 2021-04-09T18:31:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c2179700 1 -- 172.21.15.32:0/3313502740 >> v2:172.21.15.86:3300/0 conn(0x7fa5bc0cffb0 msgr2=0x7fa5bc0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c2179700 1 --2- 172.21.15.32:0/3313502740 >> v2:172.21.15.86:3300/0 conn(0x7fa5bc0cffb0 0x7fa5bc0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c2179700 1 -- 172.21.15.32:0/3313502740 >> v1:172.21.15.86:6789/0 conn(0x7fa5bc0d7810 legacy=0x7fa5bc0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.153 7fa5c2179700 1 -- 172.21.15.32:0/3313502740 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0d8400 con 0x7fa5bc0d6c90 2021-04-09T18:31:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.154 7fa5c1978700 1 --2- 172.21.15.32:0/3313502740 >> v2:172.21.15.86:3300/0 conn(0x7fa5bc0cffb0 0x7fa5bc0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:08.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.154 7fa5c2179700 1 --2- 172.21.15.32:0/3313502740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0d6c90 0x7fa5bc10cfe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fa5b4006e50 tx=0x7fa5b4009e70).ready entity=mon.2 client_cookie=9774c9fc0b20bdbe server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.154 7fa5c0976700 1 -- 172.21.15.32:0/3313502740 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b4006a20 con 0x7fa5bc0d6c90 2021-04-09T18:31:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.154 7fa5c0976700 1 -- 172.21.15.32:0/3313502740 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b4006790 con 0x7fa5bc0d6c90 2021-04-09T18:31:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.154 7fa5c0976700 1 -- 172.21.15.32:0/3313502740 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b400b700 con 0x7fa5bc0d6c90 2021-04-09T18:31:08.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.154 7fa5c3b5c700 1 -- 172.21.15.32:0/3313502740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0d6c90 msgr2=0x7fa5bc10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.154 7fa5c3b5c700 1 --2- 172.21.15.32:0/3313502740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0d6c90 0x7fa5bc10cfe0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fa5b4006e50 tx=0x7fa5b4009e70).stop 2021-04-09T18:31:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.155 7fa5c3b5c700 1 -- 172.21.15.32:0/3313502740 shutdown_connections 2021-04-09T18:31:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.155 7fa5c3b5c700 1 --2- 172.21.15.32:0/3313502740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0d6c90 0x7fa5bc10cfe0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.155 7fa5c3b5c700 1 --2- 172.21.15.32:0/3313502740 >> v2:172.21.15.86:3300/0 conn(0x7fa5bc0cffb0 0x7fa5bc0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.155 7fa5c3b5c700 1 -- 172.21.15.32:0/3313502740 >> 172.21.15.32:0/3313502740 conn(0x7fa5bc0ce5d0 msgr2=0x7fa5bc0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.155 7fa5c3b5c700 1 -- 172.21.15.32:0/3313502740 shutdown_connections 2021-04-09T18:31:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.155 7fa5c3b5c700 1 -- 172.21.15.32:0/3313502740 wait complete. 2021-04-09T18:31:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.175 7fa5c3b5c700 1 Processor -- start 2021-04-09T18:31:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.175 7fa5c3b5c700 1 -- start start 2021-04-09T18:31:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.175 7fa5c3b5c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0b7610 0x7fa5bc1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.175 7fa5c3b5c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5bc198fc0 0x7fa5bc186050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.175 7fa5c3b5c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5bc186670 0x7fa5bc186a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.175 7fa5c3b5c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc1a8c20 con 0x7fa5bc186670 2021-04-09T18:31:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.175 7fa5c3b5c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc1993c0 con 0x7fa5bc198fc0 2021-04-09T18:31:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.175 7fa5c3b5c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc187e90 con 0x7fa5bc0b7610 2021-04-09T18:31:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c2179700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5bc186670 0x7fa5bc186a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c2179700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5bc186670 0x7fa5bc186a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56468/0 (socket says 172.21.15.32:56468) 2021-04-09T18:31:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c2179700 1 -- 172.21.15.32:0/3069621984 learned_addr learned my addr 172.21.15.32:0/3069621984 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c1978700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0b7610 0x7fa5bc1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c1177700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5bc198fc0 0x7fa5bc186050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c2179700 1 -- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0b7610 msgr2=0x7fa5bc1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c2179700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0b7610 0x7fa5bc1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c2179700 1 -- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5bc198fc0 msgr2=0x7fa5bc186050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c2179700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5bc198fc0 0x7fa5bc186050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c2179700 1 -- 172.21.15.32:0/3069621984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc187090 con 0x7fa5bc186670 2021-04-09T18:31:08.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c2179700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5bc186670 0x7fa5bc186a70 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 rx=0x7fa5b40099f0 tx=0x7fa5b400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5b27fc700 1 -- 172.21.15.32:0/3069621984 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b40093d0 con 0x7fa5bc186670 2021-04-09T18:31:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5b27fc700 1 -- 172.21.15.32:0/3069621984 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b40090c0 con 0x7fa5bc186670 2021-04-09T18:31:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5b27fc700 1 -- 172.21.15.32:0/3069621984 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b4001420 con 0x7fa5bc186670 2021-04-09T18:31:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc187090 con 0x7fa5bc186670 2021-04-09T18:31:08.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.176 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc187090 con 0x7fa5bc186670 2021-04-09T18:31:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.178 7fa5b27fc700 1 -- 172.21.15.32:0/3069621984 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa5b40056b0 con 0x7fa5bc186670 2021-04-09T18:31:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.178 7fa5b27fc700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa5a40188c0 0x7fa5a401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.178 7fa5b27fc700 1 -- 172.21.15.32:0/3069621984 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6819+0+0 (secure 0 0 0) 0x7fa5b4033020 con 0x7fa5bc186670 2021-04-09T18:31:08.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.178 7fa5c1978700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa5a40188c0 0x7fa5a401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.178 7fa5c1978700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa5a40188c0 0x7fa5a401adc0 crc :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:08.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.197 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc182e80 con 0x7fa5bc186670 2021-04-09T18:31:08.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.200 7fa5b27fc700 1 -- 172.21.15.32:0/3069621984 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa5b4000df0 con 0x7fa5bc186670 2021-04-09T18:31:08.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.394 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "alpha", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa5bc182e80 con 0x7fa5bc186670 2021-04-09T18:31:08.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.546 7fa5b27fc700 1 -- 172.21.15.32:0/3069621984 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "alpha", "yes_i_really_mean_it": true}]=0 v341) v1 ==== 103+0+0 (secure 0 0 0) 0x7fa5b40056b0 con 0x7fa5bc186670 2021-04-09T18:31:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa5a40188c0 msgr2=0x7fa5a401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa5a40188c0 0x7fa5a401adc0 crc :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5bc186670 msgr2=0x7fa5bc186a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5bc186670 0x7fa5bc186a70 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 rx=0x7fa5b40099f0 tx=0x7fa5b400bd10).stop 2021-04-09T18:31:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 shutdown_connections 2021-04-09T18:31:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa5a40188c0 0x7fa5a401adc0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5bc186670 0x7fa5bc186a70 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5bc198fc0 0x7fa5bc186050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 --2- 172.21.15.32:0/3069621984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5bc0b7610 0x7fa5bc1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 >> 172.21.15.32:0/3069621984 conn(0x7fa5bc09db40 msgr2=0x7fa5bc0a7910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 shutdown_connections 2021-04-09T18:31:08.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.560 7fa5c3b5c700 1 -- 172.21.15.32:0/3069621984 wait complete. 2021-04-09T18:31:08.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete alpha_metadata alpha_metadata --yes-i-really-really-mean-it 2021-04-09T18:31:08.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.731 7f3e56b35700 1 Processor -- start 2021-04-09T18:31:08.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.731 7f3e56b35700 1 -- start start 2021-04-09T18:31:08.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e56b35700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e500d7cc0 0x7f3e500d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e56b35700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3e5010da80 0x7f3e5010de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e56b35700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5010d500 con 0x7f3e5010d050 2021-04-09T18:31:08.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e56b35700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e500d9260 con 0x7f3e5010da80 2021-04-09T18:31:08.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e56b35700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500d0330 con 0x7f3e500d7cc0 2021-04-09T18:31:08.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e55152700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3e5010da80 0x7f3e5010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e55152700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3e5010da80 0x7f3e5010de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56476/0 (socket says 172.21.15.32:56476) 2021-04-09T18:31:08.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e55152700 1 -- 172.21.15.32:0/1440990569 learned_addr learned my addr 172.21.15.32:0/1440990569 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:08.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e4ffff700 1 --2- 172.21.15.32:0/1440990569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e500d7cc0 0x7f3e500d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e55152700 1 -- 172.21.15.32:0/1440990569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e500d7cc0 msgr2=0x7f3e500d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:08.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e55152700 1 --2- 172.21.15.32:0/1440990569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e500d7cc0 0x7f3e500d80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e55152700 1 -- 172.21.15.32:0/1440990569 >> v1:172.21.15.86:6789/0 conn(0x7f3e5010d050 legacy=0x7f3e500cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e55152700 1 -- 172.21.15.32:0/1440990569 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e50112140 con 0x7f3e5010da80 2021-04-09T18:31:08.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e4f7fe700 1 -- 172.21.15.32:0/1440990569 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150181091 0 0) 0x7f3e40000c60 con 0x7f3e5010d050 2021-04-09T18:31:08.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e55152700 1 --2- 172.21.15.32:0/1440990569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5010da80 0x7f3e5010de80 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 rx=0x7f3e48006f00 tx=0x7f3e48009e90).ready entity=mon.0 client_cookie=ecfefe537dde03df server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:08.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.732 7f3e4f7fe700 1 -- 172.21.15.32:0/1440990569 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4800b300 con 0x7f3e5010da80 2021-04-09T18:31:08.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e4f7fe700 1 -- 172.21.15.32:0/1440990569 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4800b640 con 0x7f3e5010da80 2021-04-09T18:31:08.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e4f7fe700 1 -- 172.21.15.32:0/1440990569 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4800bb20 con 0x7f3e5010da80 2021-04-09T18:31:08.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e56b35700 1 -- 172.21.15.32:0/1440990569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5010da80 msgr2=0x7f3e5010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e56b35700 1 --2- 172.21.15.32:0/1440990569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5010da80 0x7f3e5010de80 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 rx=0x7f3e48006f00 tx=0x7f3e48009e90).stop 2021-04-09T18:31:08.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e56b35700 1 -- 172.21.15.32:0/1440990569 shutdown_connections 2021-04-09T18:31:08.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e56b35700 1 --2- 172.21.15.32:0/1440990569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e5010da80 0x7f3e5010de80 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e56b35700 1 --2- 172.21.15.32:0/1440990569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e500d7cc0 0x7f3e500d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e56b35700 1 -- 172.21.15.32:0/1440990569 >> 172.21.15.32:0/1440990569 conn(0x7f3e500ce620 msgr2=0x7f3e500d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e56b35700 1 -- 172.21.15.32:0/1440990569 shutdown_connections 2021-04-09T18:31:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.733 7f3e56b35700 1 -- 172.21.15.32:0/1440990569 wait complete. 2021-04-09T18:31:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.753 7f3e56b35700 1 Processor -- start 2021-04-09T18:31:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.753 7f3e56b35700 1 -- start start 2021-04-09T18:31:08.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.754 7f3e56b35700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5009a890 0x7f3e500a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.754 7f3e56b35700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e5009ac90 0x7f3e500a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.754 7f3e56b35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e500c2220 0x7f3e500c2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.754 7f3e56b35700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500b86e0 con 0x7f3e500c2220 2021-04-09T18:31:08.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.754 7f3e56b35700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500a2920 con 0x7f3e5009a890 2021-04-09T18:31:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.754 7f3e56b35700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e500c2c40 con 0x7f3e5009ac90 2021-04-09T18:31:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.754 7f3e55152700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e500c2220 0x7f3e500c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.754 7f3e55152700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e500c2220 0x7f3e500c2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56482/0 (socket says 172.21.15.32:56482) 2021-04-09T18:31:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.754 7f3e55152700 1 -- 172.21.15.32:0/548569199 learned_addr learned my addr 172.21.15.32:0/548569199 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:08.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e4ffff700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e5009ac90 0x7f3e500a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e54951700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5009a890 0x7f3e500a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e55152700 1 -- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e5009ac90 msgr2=0x7f3e500a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e55152700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e5009ac90 0x7f3e500a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e55152700 1 -- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5009a890 msgr2=0x7f3e500a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e55152700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5009a890 0x7f3e500a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e55152700 1 -- 172.21.15.32:0/548569199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e500c2e20 con 0x7f3e500c2220 2021-04-09T18:31:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e4ffff700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e5009ac90 0x7f3e500a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:08.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e55152700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e500c2220 0x7f3e500c2620 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7f3e4800b910 tx=0x7f3e4800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e4d7fa700 1 -- 172.21.15.32:0/548569199 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4800bdf0 con 0x7f3e500c2220 2021-04-09T18:31:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e4d7fa700 1 -- 172.21.15.32:0/548569199 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48005820 con 0x7f3e500c2220 2021-04-09T18:31:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e56b35700 1 -- 172.21.15.32:0/548569199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e500c30c0 con 0x7f3e500c2220 2021-04-09T18:31:08.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e4d7fa700 1 -- 172.21.15.32:0/548569199 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e48005c70 con 0x7f3e500c2220 2021-04-09T18:31:08.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.755 7f3e56b35700 1 -- 172.21.15.32:0/548569199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e500c2e20 con 0x7f3e500c2220 2021-04-09T18:31:08.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.756 7f3e4d7fa700 1 -- 172.21.15.32:0/548569199 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3e48005a00 con 0x7f3e500c2220 2021-04-09T18:31:08.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.756 7f3e4d7fa700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e300188c0 0x7f3e3001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.756 7f3e4d7fa700 1 -- 172.21.15.32:0/548569199 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f3e480332b0 con 0x7f3e500c2220 2021-04-09T18:31:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.756 7f3e54951700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e300188c0 0x7f3e3001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:08.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.756 7f3e54951700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e300188c0 0x7f3e3001adc0 crc :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:08.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.776 7f3e56b35700 1 -- 172.21.15.32:0/548569199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e501b7180 con 0x7f3e500c2220 2021-04-09T18:31:08.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.779 7f3e4d7fa700 1 -- 172.21.15.32:0/548569199 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3e4800bdf0 con 0x7f3e500c2220 2021-04-09T18:31:08.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:08.963 7f3e56b35700 1 -- 172.21.15.32:0/548569199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "alpha_metadata", "prefix": "osd pool delete", "pool": "alpha_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3e5010d500 con 0x7f3e500c2220 2021-04-09T18:31:09.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.548 7f3e4d7fa700 1 -- 172.21.15.32:0/548569199 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "alpha_metadata", "prefix": "osd pool delete", "pool": "alpha_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'alpha_metadata' removed v192) v1 ==== 182+0+0 (secure 0 0 0) 0x7f3e48005820 con 0x7f3e500c2220 2021-04-09T18:31:09.594 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_metadata' removed 2021-04-09T18:31:09.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 -- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e300188c0 msgr2=0x7f3e3001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:09.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e300188c0 0x7f3e3001adc0 crc :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 -- 172.21.15.32:0/548569199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e500c2220 msgr2=0x7f3e500c2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:09.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e500c2220 0x7f3e500c2620 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7f3e4800b910 tx=0x7f3e4800bcb0).stop 2021-04-09T18:31:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 -- 172.21.15.32:0/548569199 shutdown_connections 2021-04-09T18:31:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3e300188c0 0x7f3e3001adc0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e500c2220 0x7f3e500c2620 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e5009ac90 0x7f3e500a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 --2- 172.21.15.32:0/548569199 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5009a890 0x7f3e500a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 -- 172.21.15.32:0/548569199 >> 172.21.15.32:0/548569199 conn(0x7f3e500b6b70 msgr2=0x7f3e500b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:09.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 -- 172.21.15.32:0/548569199 shutdown_connections 2021-04-09T18:31:09.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.581 7f3e56b35700 1 -- 172.21.15.32:0/548569199 wait complete. 2021-04-09T18:31:09.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete alpha_data alpha_data --yes-i-really-really-mean-it 2021-04-09T18:31:09.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.755 7f37ee8f8700 1 Processor -- start 2021-04-09T18:31:09.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.755 7f37ee8f8700 1 -- start start 2021-04-09T18:31:09.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.755 7f37ee8f8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e80d7cc0 0x7f37e80d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:09.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.755 7f37ee8f8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37e810da80 0x7f37e810de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:09.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.755 7f37ee8f8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e810d500 con 0x7f37e810d050 2021-04-09T18:31:09.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.755 7f37ee8f8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e80d9260 con 0x7f37e810da80 2021-04-09T18:31:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.755 7f37ee8f8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e80d0330 con 0x7f37e80d7cc0 2021-04-09T18:31:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37dffff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e80d7cc0 0x7f37e80d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37ecf15700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37e810da80 0x7f37e810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37dffff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e80d7cc0 0x7f37e80d80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:35486/0 (socket says 172.21.15.32:35486) 2021-04-09T18:31:09.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37dffff700 1 -- 172.21.15.32:0/973360424 learned_addr learned my addr 172.21.15.32:0/973360424 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:09.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37e77fe700 1 -- 172.21.15.32:0/973360424 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986076534 0 0) 0x7f37d8000c60 con 0x7f37e810d050 2021-04-09T18:31:09.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37e77fe700 1 -- 172.21.15.32:0/973360424 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d4002fe0 con 0x7f37e810d050 2021-04-09T18:31:09.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37dffff700 1 -- 172.21.15.32:0/973360424 >> v2:172.21.15.86:3300/0 conn(0x7f37e810da80 msgr2=0x7f37e810de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:09.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37dffff700 1 --2- 172.21.15.32:0/973360424 >> v2:172.21.15.86:3300/0 conn(0x7f37e810da80 0x7f37e810de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37dffff700 1 -- 172.21.15.32:0/973360424 >> v1:172.21.15.86:6789/0 conn(0x7f37e810d050 legacy=0x7f37e80cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:09.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37dffff700 1 -- 172.21.15.32:0/973360424 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e80d86e0 con 0x7f37e80d7cc0 2021-04-09T18:31:09.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37ecf15700 1 --2- 172.21.15.32:0/973360424 >> v2:172.21.15.86:3300/0 conn(0x7f37e810da80 0x7f37e810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:09.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37dffff700 1 --2- 172.21.15.32:0/973360424 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e80d7cc0 0x7f37e80d80c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f37d0006700 tx=0x7f37d0009b80).ready entity=mon.2 client_cookie=4a527616ae44f8d6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:09.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37e77fe700 1 -- 172.21.15.32:0/973360424 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37d000c0c0 con 0x7f37e80d7cc0 2021-04-09T18:31:09.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37e77fe700 1 -- 172.21.15.32:0/973360424 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d000c440 con 0x7f37e80d7cc0 2021-04-09T18:31:09.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.756 7f37e77fe700 1 -- 172.21.15.32:0/973360424 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37d000c920 con 0x7f37e80d7cc0 2021-04-09T18:31:09.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.757 7f37ee8f8700 1 -- 172.21.15.32:0/973360424 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e80d7cc0 msgr2=0x7f37e80d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:09.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.757 7f37ee8f8700 1 --2- 172.21.15.32:0/973360424 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e80d7cc0 0x7f37e80d80c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f37d0006700 tx=0x7f37d0009b80).stop 2021-04-09T18:31:09.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.757 7f37ee8f8700 1 -- 172.21.15.32:0/973360424 shutdown_connections 2021-04-09T18:31:09.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.757 7f37ee8f8700 1 --2- 172.21.15.32:0/973360424 >> v2:172.21.15.86:3300/0 conn(0x7f37e810da80 0x7f37e810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.757 7f37ee8f8700 1 --2- 172.21.15.32:0/973360424 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e80d7cc0 0x7f37e80d80c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.757 7f37ee8f8700 1 -- 172.21.15.32:0/973360424 >> 172.21.15.32:0/973360424 conn(0x7f37e80ce620 msgr2=0x7f37e80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:09.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.757 7f37ee8f8700 1 -- 172.21.15.32:0/973360424 shutdown_connections 2021-04-09T18:31:09.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.757 7f37ee8f8700 1 -- 172.21.15.32:0/973360424 wait complete. 2021-04-09T18:31:09.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.777 7f37ee8f8700 1 Processor -- start 2021-04-09T18:31:09.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.777 7f37ee8f8700 1 -- start start 2021-04-09T18:31:09.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37ee8f8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e81799a0 0x7f37e80b9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:09.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37ee8f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37e80ba500 0x7f37e80b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:09.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37ee8f8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e80b93c0 0x7f37e80b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:09.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37ee8f8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e80b7130 con 0x7f37e80ba500 2021-04-09T18:31:09.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37ee8f8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e80b9960 con 0x7f37e80b93c0 2021-04-09T18:31:09.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37ee8f8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e80b6c90 con 0x7f37e81799a0 2021-04-09T18:31:09.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37dffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37e80ba500 0x7f37e80b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:09.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37dffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37e80ba500 0x7f37e80b5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56494/0 (socket says 172.21.15.32:56494) 2021-04-09T18:31:09.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37dffff700 1 -- 172.21.15.32:0/797676433 learned_addr learned my addr 172.21.15.32:0/797676433 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:09.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37ecf15700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e80b93c0 0x7f37e80b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:09.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.778 7f37e7fff700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e81799a0 0x7f37e80b9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:09.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37dffff700 1 -- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e81799a0 msgr2=0x7f37e80b9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:09.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37dffff700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e81799a0 0x7f37e80b9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37dffff700 1 -- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e80b93c0 msgr2=0x7f37e80b5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:09.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37dffff700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e80b93c0 0x7f37e80b5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:09.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37dffff700 1 -- 172.21.15.32:0/797676433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e80b6350 con 0x7f37e80ba500 2021-04-09T18:31:09.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37dffff700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37e80ba500 0x7f37e80b5710 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 rx=0x7f37d000b580 tx=0x7f37d000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:09.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37e57fa700 1 -- 172.21.15.32:0/797676433 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37d000bd10 con 0x7f37e80ba500 2021-04-09T18:31:09.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37e57fa700 1 -- 172.21.15.32:0/797676433 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d0002030 con 0x7f37e80ba500 2021-04-09T18:31:09.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e80b6350 con 0x7f37e80ba500 2021-04-09T18:31:09.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37e57fa700 1 -- 172.21.15.32:0/797676433 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37d0002550 con 0x7f37e80ba500 2021-04-09T18:31:09.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e80b6350 con 0x7f37e80ba500 2021-04-09T18:31:09.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.779 7f37e57fa700 1 -- 172.21.15.32:0/797676433 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f37d0000c90 con 0x7f37e80ba500 2021-04-09T18:31:09.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.780 7f37e57fa700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37cc0188d0 0x7f37cc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:09.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.780 7f37e57fa700 1 -- 172.21.15.32:0/797676433 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f37d0032d60 con 0x7f37e80ba500 2021-04-09T18:31:09.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.780 7f37e7fff700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37cc0188d0 0x7f37cc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:09.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.780 7f37e7fff700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37cc0188d0 0x7f37cc01add0 crc :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:09.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.799 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e80b6c90 con 0x7f37e80ba500 2021-04-09T18:31:09.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.803 7f37e57fa700 1 -- 172.21.15.32:0/797676433 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f37d001f780 con 0x7f37e80ba500 2021-04-09T18:31:10.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:09.988 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "alpha_data", "prefix": "osd pool delete", "pool": "alpha_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f37e80b6c90 con 0x7f37e80ba500 2021-04-09T18:31:10.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.558 7f37e57fa700 1 -- 172.21.15.32:0/797676433 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "alpha_data", "prefix": "osd pool delete", "pool": "alpha_data", "yes_i_really_really_mean_it": true}]=0 pool 'alpha_data' removed v193) v1 ==== 170+0+0 (secure 0 0 0) 0x7f37d001f780 con 0x7f37e80ba500 2021-04-09T18:31:10.618 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_data' removed 2021-04-09T18:31:10.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37cc0188d0 msgr2=0x7f37cc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:10.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37cc0188d0 0x7f37cc01add0 crc :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37e80ba500 msgr2=0x7f37e80b5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:10.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37e80ba500 0x7f37e80b5710 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 rx=0x7f37d000b580 tx=0x7f37d000b920).stop 2021-04-09T18:31:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 shutdown_connections 2021-04-09T18:31:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f37cc0188d0 0x7f37cc01add0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37e80ba500 0x7f37e80b5710 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e80b93c0 0x7f37e80b5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 --2- 172.21.15.32:0/797676433 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e81799a0 0x7f37e80b9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 >> 172.21.15.32:0/797676433 conn(0x7f37e80c3960 msgr2=0x7f37e80c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 shutdown_connections 2021-04-09T18:31:10.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.605 7f37ee8f8700 1 -- 172.21.15.32:0/797676433 wait complete. 2021-04-09T18:31:10.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm bravo --yes-i-really-mean-it 2021-04-09T18:31:10.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.771 7f1a8a9fa700 1 Processor -- start 2021-04-09T18:31:10.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.771 7f1a8a9fa700 1 -- start start 2021-04-09T18:31:10.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.771 7f1a8a9fa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a840cffb0 0x7f1a840d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:10.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.771 7f1a8a9fa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a840d7810 0x7f1a840d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:10.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.771 7f1a8a9fa700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a840d7290 con 0x7f1a840d8230 2021-04-09T18:31:10.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.771 7f1a8a9fa700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a840d91f0 con 0x7f1a840cffb0 2021-04-09T18:31:10.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.771 7f1a8a9fa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8410dff0 con 0x7f1a840d7810 2021-04-09T18:31:10.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a88816700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a840cffb0 0x7f1a840d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:10.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a89017700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1a840d8230 0x7f1a840d5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37862/0 (socket says 172.21.15.32:37862) 2021-04-09T18:31:10.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a89017700 1 -- 172.21.15.32:0/2424161954 learned_addr learned my addr 172.21.15.32:0/2424161954 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:10.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a83fff700 1 --2- 172.21.15.32:0/2424161954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a840d7810 0x7f1a840d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:10.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a88816700 1 -- 172.21.15.32:0/2424161954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a840d7810 msgr2=0x7f1a840d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:10.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a88816700 1 --2- 172.21.15.32:0/2424161954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a840d7810 0x7f1a840d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a88816700 1 -- 172.21.15.32:0/2424161954 >> v1:172.21.15.32:6789/0 conn(0x7f1a840d8230 legacy=0x7f1a840d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:10.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a88816700 1 -- 172.21.15.32:0/2424161954 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8410e210 con 0x7f1a840cffb0 2021-04-09T18:31:10.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a837fe700 1 -- 172.21.15.32:0/2424161954 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762286282 0 0) 0x7f1a7c000d30 con 0x7f1a840d8230 2021-04-09T18:31:10.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a88816700 1 --2- 172.21.15.32:0/2424161954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a840cffb0 0x7f1a840d2450 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f1a74006650 tx=0x7f1a74009a00).ready entity=mon.0 client_cookie=3b2df2de9e911177 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:10.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a837fe700 1 -- 172.21.15.32:0/2424161954 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a74009df0 con 0x7f1a840cffb0 2021-04-09T18:31:10.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a837fe700 1 -- 172.21.15.32:0/2424161954 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a740009b0 con 0x7f1a840cffb0 2021-04-09T18:31:10.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a837fe700 1 -- 172.21.15.32:0/2424161954 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a7400c2b0 con 0x7f1a840cffb0 2021-04-09T18:31:10.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a8a9fa700 1 -- 172.21.15.32:0/2424161954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a840cffb0 msgr2=0x7f1a840d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:10.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.772 7f1a8a9fa700 1 --2- 172.21.15.32:0/2424161954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a840cffb0 0x7f1a840d2450 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f1a74006650 tx=0x7f1a74009a00).stop 2021-04-09T18:31:10.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.773 7f1a8a9fa700 1 -- 172.21.15.32:0/2424161954 shutdown_connections 2021-04-09T18:31:10.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.773 7f1a8a9fa700 1 --2- 172.21.15.32:0/2424161954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a840d7810 0x7f1a840d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.773 7f1a8a9fa700 1 --2- 172.21.15.32:0/2424161954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a840cffb0 0x7f1a840d2450 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.773 7f1a8a9fa700 1 -- 172.21.15.32:0/2424161954 >> 172.21.15.32:0/2424161954 conn(0x7f1a840ce5d0 msgr2=0x7f1a840cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:10.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.773 7f1a8a9fa700 1 -- 172.21.15.32:0/2424161954 shutdown_connections 2021-04-09T18:31:10.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.773 7f1a8a9fa700 1 -- 172.21.15.32:0/2424161954 wait complete. 2021-04-09T18:31:10.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.793 7f1a8a9fa700 1 Processor -- start 2021-04-09T18:31:10.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.793 7f1a8a9fa700 1 -- start start 2021-04-09T18:31:10.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a8a9fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8418f620 0x7f1a84185100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:10.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a8a9fa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a84185720 0x7f1a841abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:10.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a8a9fa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a841ac320 0x7f1a841ac720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:10.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a8a9fa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8418fa20 con 0x7f1a8418f620 2021-04-09T18:31:10.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a8a9fa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84185b20 con 0x7f1a841ac320 2021-04-09T18:31:10.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a8a9fa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84195e70 con 0x7f1a84185720 2021-04-09T18:31:10.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a88816700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8418f620 0x7f1a84185100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:10.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a88816700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8418f620 0x7f1a84185100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56508/0 (socket says 172.21.15.32:56508) 2021-04-09T18:31:10.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a88816700 1 -- 172.21.15.32:0/3852372362 learned_addr learned my addr 172.21.15.32:0/3852372362 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:10.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a89017700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a841ac320 0x7f1a841ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:10.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a83fff700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a84185720 0x7f1a841abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:10.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a88816700 1 -- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a84185720 msgr2=0x7f1a841abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:10.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a88816700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a84185720 0x7f1a841abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a88816700 1 -- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a841ac320 msgr2=0x7f1a841ac720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:10.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a88816700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a841ac320 0x7f1a841ac720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:10.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.794 7f1a88816700 1 -- 172.21.15.32:0/3852372362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a841acd40 con 0x7f1a8418f620 2021-04-09T18:31:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.795 7f1a88816700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8418f620 0x7f1a84185100 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 rx=0x7f1a7400b580 tx=0x7f1a7400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.795 7f1a817fa700 1 -- 172.21.15.32:0/3852372362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a74000c90 con 0x7f1a8418f620 2021-04-09T18:31:10.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.795 7f1a817fa700 1 -- 172.21.15.32:0/3852372362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74002030 con 0x7f1a8418f620 2021-04-09T18:31:10.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.795 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a841acd40 con 0x7f1a8418f620 2021-04-09T18:31:10.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.795 7f1a817fa700 1 -- 172.21.15.32:0/3852372362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a74002480 con 0x7f1a8418f620 2021-04-09T18:31:10.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.795 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a841acd40 con 0x7f1a8418f620 2021-04-09T18:31:10.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.795 7f1a817fa700 1 -- 172.21.15.32:0/3852372362 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1a74002210 con 0x7f1a8418f620 2021-04-09T18:31:10.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.796 7f1a817fa700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.796 7f1a817fa700 1 -- 172.21.15.32:0/3852372362 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f1a74032bf0 con 0x7f1a8418f620 2021-04-09T18:31:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.796 7f1a83fff700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:10.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.796 7f1a83fff700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 crc :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.816 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a84099370 con 0x7f1a8418f620 2021-04-09T18:31:10.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:10.819 7f1a817fa700 1 -- 172.21.15.32:0/3852372362 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1a740009c0 con 0x7f1a8418f620 2021-04-09T18:31:11.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.014 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "bravo", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1a84099370 con 0x7f1a8418f620 2021-04-09T18:31:11.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.559 7f1a817fa700 1 -- 172.21.15.32:0/3852372362 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "bravo", "yes_i_really_mean_it": true}]=0 v342) v1 ==== 103+0+0 (secure 0 0 0) 0x7f1a74002030 con 0x7f1a8418f620 2021-04-09T18:31:11.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 msgr2=0x7f1a6401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:11.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 crc :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8418f620 msgr2=0x7f1a84185100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:11.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8418f620 0x7f1a84185100 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 rx=0x7f1a7400b580 tx=0x7f1a7400b920).stop 2021-04-09T18:31:11.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 shutdown_connections 2021-04-09T18:31:11.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a64018930 0x7f1a6401ae10 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a841ac320 0x7f1a841ac720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a84185720 0x7f1a841abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 --2- 172.21.15.32:0/3852372362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a8418f620 0x7f1a84185100 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 >> 172.21.15.32:0/3852372362 conn(0x7f1a84184a80 msgr2=0x7f1a841a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:11.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 shutdown_connections 2021-04-09T18:31:11.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.581 7f1a8a9fa700 1 -- 172.21.15.32:0/3852372362 wait complete. 2021-04-09T18:31:11.608 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete bravo_metadata bravo_metadata --yes-i-really-really-mean-it 2021-04-09T18:31:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.756 7f6855697700 1 Processor -- start 2021-04-09T18:31:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.756 7f6855697700 1 -- start start 2021-04-09T18:31:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f6855697700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500d7cc0 0x7f68500d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f6855697700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f685010da80 0x7f685010de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f6855697700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f685010d500 con 0x7f685010d050 2021-04-09T18:31:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f6855697700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68500d9260 con 0x7f685010da80 2021-04-09T18:31:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f6855697700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6850112140 con 0x7f68500d7cc0 2021-04-09T18:31:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684f2bf700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f685010d050 0x7f68500cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:35800/0 (socket says 172.21.15.32:35800) 2021-04-09T18:31:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684fac0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f685010da80 0x7f685010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684f2bf700 1 -- 172.21.15.32:0/1987059661 learned_addr learned my addr 172.21.15.32:0/1987059661 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684eabe700 1 --2- 172.21.15.32:0/1987059661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500d7cc0 0x7f68500d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684fac0700 1 -- 172.21.15.32:0/1987059661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500d7cc0 msgr2=0x7f68500d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684fac0700 1 --2- 172.21.15.32:0/1987059661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500d7cc0 0x7f68500d80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684fac0700 1 -- 172.21.15.32:0/1987059661 >> v1:172.21.15.86:6789/0 conn(0x7f685010d050 legacy=0x7f68500cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684fac0700 1 -- 172.21.15.32:0/1987059661 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68500d09f0 con 0x7f685010da80 2021-04-09T18:31:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684e2bd700 1 -- 172.21.15.32:0/1987059661 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061225269 0 0) 0x7f68400010c0 con 0x7f685010d050 2021-04-09T18:31:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.757 7f684fac0700 1 --2- 172.21.15.32:0/1987059661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f685010da80 0x7f685010de80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f6848006e50 tx=0x7f6848009df0).ready entity=mon.1 client_cookie=4fd9d01c8a76163e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f684e2bd700 1 -- 172.21.15.32:0/1987059661 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f684800b510 con 0x7f685010da80 2021-04-09T18:31:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f684e2bd700 1 -- 172.21.15.32:0/1987059661 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6848003030 con 0x7f685010da80 2021-04-09T18:31:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f684e2bd700 1 -- 172.21.15.32:0/1987059661 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f684800b7d0 con 0x7f685010da80 2021-04-09T18:31:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f6855697700 1 -- 172.21.15.32:0/1987059661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f685010da80 msgr2=0x7f685010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f6855697700 1 --2- 172.21.15.32:0/1987059661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f685010da80 0x7f685010de80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f6848006e50 tx=0x7f6848009df0).stop 2021-04-09T18:31:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f6855697700 1 -- 172.21.15.32:0/1987059661 shutdown_connections 2021-04-09T18:31:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f6855697700 1 --2- 172.21.15.32:0/1987059661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f685010da80 0x7f685010de80 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f6855697700 1 --2- 172.21.15.32:0/1987059661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500d7cc0 0x7f68500d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f6855697700 1 -- 172.21.15.32:0/1987059661 >> 172.21.15.32:0/1987059661 conn(0x7f68500ce620 msgr2=0x7f68500d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:11.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f6855697700 1 -- 172.21.15.32:0/1987059661 shutdown_connections 2021-04-09T18:31:11.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.758 7f6855697700 1 -- 172.21.15.32:0/1987059661 wait complete. 2021-04-09T18:31:11.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.778 7f6855697700 1 Processor -- start 2021-04-09T18:31:11.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.778 7f6855697700 1 -- start start 2021-04-09T18:31:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f6855697700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68500b84d0 0x7f68500c2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f6855697700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68500c2e20 0x7f68500a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f6855697700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500a1b60 0x7f68500a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f6855697700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685018e9f0 con 0x7f68500b84d0 2021-04-09T18:31:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f6855697700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68500a8420 con 0x7f68500c2e20 2021-04-09T18:31:11.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f6855697700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68500b3d90 con 0x7f68500a1b60 2021-04-09T18:31:11.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f684f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68500b84d0 0x7f68500c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:11.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f684f2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68500b84d0 0x7f68500c2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56522/0 (socket says 172.21.15.32:56522) 2021-04-09T18:31:11.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f684f2bf700 1 -- 172.21.15.32:0/1845650219 learned_addr learned my addr 172.21.15.32:0/1845650219 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:11.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f684fac0700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500a1b60 0x7f68500a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:11.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f684eabe700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68500c2e20 0x7f68500a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f684f2bf700 1 -- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500a1b60 msgr2=0x7f68500a7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.779 7f684f2bf700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500a1b60 0x7f68500a7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f684f2bf700 1 -- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68500c2e20 msgr2=0x7f68500a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f684f2bf700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68500c2e20 0x7f68500a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:11.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f684f2bf700 1 -- 172.21.15.32:0/1845650219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68500a7cb0 con 0x7f68500b84d0 2021-04-09T18:31:11.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f684eabe700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68500c2e20 0x7f68500a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:11.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f684fac0700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500a1b60 0x7f68500a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:11.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f684f2bf700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68500b84d0 0x7f68500c2800 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f684000a5d0 tx=0x7f684000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:11.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f6837fff700 1 -- 172.21.15.32:0/1845650219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6840000c10 con 0x7f68500b84d0 2021-04-09T18:31:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f6837fff700 1 -- 172.21.15.32:0/1845650219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68400070d0 con 0x7f68500b84d0 2021-04-09T18:31:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f6855697700 1 -- 172.21.15.32:0/1845650219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68500a7cb0 con 0x7f68500b84d0 2021-04-09T18:31:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f6837fff700 1 -- 172.21.15.32:0/1845650219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68400047b0 con 0x7f68500b84d0 2021-04-09T18:31:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.780 7f6855697700 1 -- 172.21.15.32:0/1845650219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68500a7cb0 con 0x7f68500b84d0 2021-04-09T18:31:11.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.781 7f6837fff700 1 -- 172.21.15.32:0/1845650219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6840000c10 con 0x7f68500b84d0 2021-04-09T18:31:11.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.781 7f6837fff700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68300188c0 0x7f683001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:11.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.781 7f6837fff700 1 -- 172.21.15.32:0/1845650219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6003+0+0 (secure 0 0 0) 0x7f6840032920 con 0x7f68500b84d0 2021-04-09T18:31:11.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.781 7f684eabe700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68300188c0 0x7f683001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:11.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.782 7f684eabe700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68300188c0 0x7f683001adc0 crc :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:11.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.801 7f6855697700 1 -- 172.21.15.32:0/1845650219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68501b7180 con 0x7f68500b84d0 2021-04-09T18:31:11.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.805 7f6837fff700 1 -- 172.21.15.32:0/1845650219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6840000c10 con 0x7f68500b84d0 2021-04-09T18:31:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:11.988 7f6855697700 1 -- 172.21.15.32:0/1845650219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "bravo_metadata", "prefix": "osd pool delete", "pool": "bravo_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f685010d500 con 0x7f68500b84d0 2021-04-09T18:31:12.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.612 7f6837fff700 1 -- 172.21.15.32:0/1845650219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "bravo_metadata", "prefix": "osd pool delete", "pool": "bravo_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'bravo_metadata' removed v194) v1 ==== 182+0+0 (secure 0 0 0) 0x7f68400047f0 con 0x7f68500b84d0 2021-04-09T18:31:12.668 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_metadata' removed 2021-04-09T18:31:12.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 -- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68300188c0 msgr2=0x7f683001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:12.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68300188c0 0x7f683001adc0 crc :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 -- 172.21.15.32:0/1845650219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68500b84d0 msgr2=0x7f68500c2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:12.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68500b84d0 0x7f68500c2800 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f684000a5d0 tx=0x7f684000a970).stop 2021-04-09T18:31:12.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 -- 172.21.15.32:0/1845650219 shutdown_connections 2021-04-09T18:31:12.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f68300188c0 0x7f683001adc0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f68500a1b60 0x7f68500a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68500c2e20 0x7f68500a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 --2- 172.21.15.32:0/1845650219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68500b84d0 0x7f68500c2800 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.655 7f6855697700 1 -- 172.21.15.32:0/1845650219 >> 172.21.15.32:0/1845650219 conn(0x7f68500b6b70 msgr2=0x7f68500b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:12.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.656 7f6855697700 1 -- 172.21.15.32:0/1845650219 shutdown_connections 2021-04-09T18:31:12.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.656 7f6855697700 1 -- 172.21.15.32:0/1845650219 wait complete. 2021-04-09T18:31:12.683 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete bravo_data bravo_data --yes-i-really-really-mean-it 2021-04-09T18:31:12.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.831 7fe60031d700 1 Processor -- start 2021-04-09T18:31:12.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.831 7fe60031d700 1 -- start start 2021-04-09T18:31:12.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe60031d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5f810d050 0x7fe5f80cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:12.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe60031d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f80d0160 0x7fe5f80d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:12.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe60031d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f810d500 con 0x7fe5f80d82e0 2021-04-09T18:31:12.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe60031d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f80d9260 con 0x7fe5f810d050 2021-04-09T18:31:12.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe60031d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80d86e0 con 0x7fe5f80d0160 2021-04-09T18:31:12.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe5fe139700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5f810d050 0x7fe5f80cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:12.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe5fe93a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe5f80d82e0 0x7fe5f810da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37892/0 (socket says 172.21.15.32:37892) 2021-04-09T18:31:12.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe5fe93a700 1 -- 172.21.15.32:0/3364491283 learned_addr learned my addr 172.21.15.32:0/3364491283 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:12.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe5fd938700 1 --2- 172.21.15.32:0/3364491283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f80d0160 0x7fe5f80d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:12.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe5fe139700 1 -- 172.21.15.32:0/3364491283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f80d0160 msgr2=0x7fe5f80d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:12.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe5fe139700 1 --2- 172.21.15.32:0/3364491283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f80d0160 0x7fe5f80d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe5fe139700 1 -- 172.21.15.32:0/3364491283 >> v1:172.21.15.32:6789/0 conn(0x7fe5f80d82e0 legacy=0x7fe5f810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:12.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.832 7fe5fd137700 1 -- 172.21.15.32:0/3364491283 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166535457 0 0) 0x7fe5f0000d30 con 0x7fe5f80d82e0 2021-04-09T18:31:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.833 7fe5fe139700 1 -- 172.21.15.32:0/3364491283 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f810e270 con 0x7fe5f810d050 2021-04-09T18:31:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.833 7fe5fe139700 1 --2- 172.21.15.32:0/3364491283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f810d050 0x7fe5f80cfb40 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7fe5f4006650 tx=0x7fe5f4009a00).ready entity=mon.0 client_cookie=e9a1132a75620b91 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.833 7fe5fd137700 1 -- 172.21.15.32:0/3364491283 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5f4009df0 con 0x7fe5f810d050 2021-04-09T18:31:12.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.833 7fe5fd137700 1 -- 172.21.15.32:0/3364491283 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f40009b0 con 0x7fe5f810d050 2021-04-09T18:31:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.833 7fe5fd137700 1 -- 172.21.15.32:0/3364491283 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5f400c2b0 con 0x7fe5f810d050 2021-04-09T18:31:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.833 7fe60031d700 1 -- 172.21.15.32:0/3364491283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f810d050 msgr2=0x7fe5f80cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.833 7fe60031d700 1 --2- 172.21.15.32:0/3364491283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f810d050 0x7fe5f80cfb40 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7fe5f4006650 tx=0x7fe5f4009a00).stop 2021-04-09T18:31:12.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.834 7fe60031d700 1 -- 172.21.15.32:0/3364491283 shutdown_connections 2021-04-09T18:31:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.834 7fe60031d700 1 --2- 172.21.15.32:0/3364491283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f80d0160 0x7fe5f80d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.834 7fe60031d700 1 --2- 172.21.15.32:0/3364491283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f810d050 0x7fe5f80cfb40 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.834 7fe60031d700 1 -- 172.21.15.32:0/3364491283 >> 172.21.15.32:0/3364491283 conn(0x7fe5f80ce620 msgr2=0x7fe5f80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:12.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.834 7fe60031d700 1 -- 172.21.15.32:0/3364491283 shutdown_connections 2021-04-09T18:31:12.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.834 7fe60031d700 1 -- 172.21.15.32:0/3364491283 wait complete. 2021-04-09T18:31:12.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.854 7fe60031d700 1 Processor -- start 2021-04-09T18:31:12.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.854 7fe60031d700 1 -- start start 2021-04-09T18:31:12.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.854 7fe60031d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f809a890 0x7fe5f80a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:12.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.854 7fe60031d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f809ac90 0x7fe5f80a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:12.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.854 7fe60031d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f80c2220 0x7fe5f80c2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:12.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.854 7fe60031d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80b86e0 con 0x7fe5f809a890 2021-04-09T18:31:12.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.854 7fe60031d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80a2920 con 0x7fe5f80c2220 2021-04-09T18:31:12.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.854 7fe60031d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80c2c40 con 0x7fe5f809ac90 2021-04-09T18:31:12.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe139700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f809a890 0x7fe5f80a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:12.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe139700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f809a890 0x7fe5f80a2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56538/0 (socket says 172.21.15.32:56538) 2021-04-09T18:31:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe139700 1 -- 172.21.15.32:0/1244500898 learned_addr learned my addr 172.21.15.32:0/1244500898 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fd938700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f809ac90 0x7fe5f80a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe93a700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f80c2220 0x7fe5f80c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:12.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe139700 1 -- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f809ac90 msgr2=0x7fe5f80a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:12.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe139700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f809ac90 0x7fe5f80a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe139700 1 -- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f80c2220 msgr2=0x7fe5f80c2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:12.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe139700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f80c2220 0x7fe5f80c2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:12.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe139700 1 -- 172.21.15.32:0/1244500898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f80c2e20 con 0x7fe5f809a890 2021-04-09T18:31:12.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5fe139700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f809a890 0x7fe5f80a2ea0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 rx=0x7fe5f40096a0 tx=0x7fe5f400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:12.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5eeffd700 1 -- 172.21.15.32:0/1244500898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5f40093a0 con 0x7fe5f809a890 2021-04-09T18:31:12.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5eeffd700 1 -- 172.21.15.32:0/1244500898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f40067f0 con 0x7fe5f809a890 2021-04-09T18:31:12.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe60031d700 1 -- 172.21.15.32:0/1244500898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f80c30c0 con 0x7fe5f809a890 2021-04-09T18:31:12.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe5eeffd700 1 -- 172.21.15.32:0/1244500898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5f40022b0 con 0x7fe5f809a890 2021-04-09T18:31:12.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.855 7fe60031d700 1 -- 172.21.15.32:0/1244500898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f80c2e20 con 0x7fe5f809a890 2021-04-09T18:31:12.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.856 7fe5eeffd700 1 -- 172.21.15.32:0/1244500898 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe5f40052e0 con 0x7fe5f809a890 2021-04-09T18:31:12.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.856 7fe5eeffd700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5e00188c0 0x7fe5e001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:12.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.856 7fe5eeffd700 1 -- 172.21.15.32:0/1244500898 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 5567+0+0 (secure 0 0 0) 0x7fe5f40327c0 con 0x7fe5f809a890 2021-04-09T18:31:12.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.856 7fe5fd938700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5e00188c0 0x7fe5e001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:12.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.857 7fe5fd938700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5e00188c0 0x7fe5e001adc0 crc :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:12.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.876 7fe60031d700 1 -- 172.21.15.32:0/1244500898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f81b7180 con 0x7fe5f809a890 2021-04-09T18:31:12.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:12.880 7fe5eeffd700 1 -- 172.21.15.32:0/1244500898 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe5f4005610 con 0x7fe5f809a890 2021-04-09T18:31:13.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.063 7fe60031d700 1 -- 172.21.15.32:0/1244500898 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "bravo_data", "prefix": "osd pool delete", "pool": "bravo_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe5f81b7180 con 0x7fe5f809a890 2021-04-09T18:31:13.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.613 7fe5eeffd700 1 -- 172.21.15.32:0/1244500898 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "bravo_data", "prefix": "osd pool delete", "pool": "bravo_data", "yes_i_really_really_mean_it": true}]=0 pool 'bravo_data' removed v195) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe5f4005820 con 0x7fe5f809a890 2021-04-09T18:31:13.643 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_data' removed 2021-04-09T18:31:13.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 -- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5e00188c0 msgr2=0x7fe5e001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:13.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5e00188c0 0x7fe5e001adc0 crc :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 -- 172.21.15.32:0/1244500898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f809a890 msgr2=0x7fe5f80a2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:13.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f809a890 0x7fe5f80a2ea0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 rx=0x7fe5f40096a0 tx=0x7fe5f400b8e0).stop 2021-04-09T18:31:13.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 -- 172.21.15.32:0/1244500898 shutdown_connections 2021-04-09T18:31:13.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5e00188c0 0x7fe5e001adc0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5f80c2220 0x7fe5f80c2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5f809ac90 0x7fe5f80a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 --2- 172.21.15.32:0/1244500898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f809a890 0x7fe5f80a2ea0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 -- 172.21.15.32:0/1244500898 >> 172.21.15.32:0/1244500898 conn(0x7fe5f80b6b70 msgr2=0x7fe5f80b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:13.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 -- 172.21.15.32:0/1244500898 shutdown_connections 2021-04-09T18:31:13.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.630 7fe60031d700 1 -- 172.21.15.32:0/1244500898 wait complete. 2021-04-09T18:31:13.658 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:31:13.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.797 7fc05f7fe700 1 Processor -- start 2021-04-09T18:31:13.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.797 7fc05f7fe700 1 -- start start 2021-04-09T18:31:13.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.797 7fc05f7fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc0580cffb0 0x7fc0580d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:13.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.797 7fc05f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0580d6c90 0x7fc05810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:13.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.797 7fc05f7fe700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0580d7290 con 0x7fc0580d7810 2021-04-09T18:31:13.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.797 7fc05f7fe700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0580d7090 con 0x7fc0580d6c90 2021-04-09T18:31:13.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.797 7fc05f7fe700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0580d91f0 con 0x7fc0580cffb0 2021-04-09T18:31:13.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05de1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0580d6c90 0x7fc05810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:13.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05ce19700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc0580d7810 0x7fc0580d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37908/0 (socket says 172.21.15.32:37908) 2021-04-09T18:31:13.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05ce19700 1 -- 172.21.15.32:0/3683849954 learned_addr learned my addr 172.21.15.32:0/3683849954 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:13.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05d61a700 1 --2- 172.21.15.32:0/3683849954 >> v2:172.21.15.86:3300/0 conn(0x7fc0580cffb0 0x7fc0580d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:13.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05de1b700 1 -- 172.21.15.32:0/3683849954 >> v2:172.21.15.86:3300/0 conn(0x7fc0580cffb0 msgr2=0x7fc0580d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:13.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05de1b700 1 --2- 172.21.15.32:0/3683849954 >> v2:172.21.15.86:3300/0 conn(0x7fc0580cffb0 0x7fc0580d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05de1b700 1 -- 172.21.15.32:0/3683849954 >> v1:172.21.15.32:6789/0 conn(0x7fc0580d7810 legacy=0x7fc0580d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:13.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc04ffff700 1 -- 172.21.15.32:0/3683849954 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269089228 0 0) 0x7fc0480009b0 con 0x7fc0580d7810 2021-04-09T18:31:13.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05de1b700 1 -- 172.21.15.32:0/3683849954 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0580d8400 con 0x7fc0580d6c90 2021-04-09T18:31:13.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05de1b700 1 --2- 172.21.15.32:0/3683849954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580d6c90 0x7fc05810cfe0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 rx=0x7fc050006e30 tx=0x7fc050009e60).ready entity=mon.0 client_cookie=b17825d7c81b3bab server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:13.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc04ffff700 1 -- 172.21.15.32:0/3683849954 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc050006a50 con 0x7fc0580d6c90 2021-04-09T18:31:13.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc04ffff700 1 -- 172.21.15.32:0/3683849954 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05000b300 con 0x7fc0580d6c90 2021-04-09T18:31:13.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc04ffff700 1 -- 172.21.15.32:0/3683849954 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc05000b750 con 0x7fc0580d6c90 2021-04-09T18:31:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05f7fe700 1 -- 172.21.15.32:0/3683849954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580d6c90 msgr2=0x7fc05810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.798 7fc05f7fe700 1 --2- 172.21.15.32:0/3683849954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580d6c90 0x7fc05810cfe0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 rx=0x7fc050006e30 tx=0x7fc050009e60).stop 2021-04-09T18:31:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.799 7fc05f7fe700 1 -- 172.21.15.32:0/3683849954 shutdown_connections 2021-04-09T18:31:13.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.799 7fc05f7fe700 1 --2- 172.21.15.32:0/3683849954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0580d6c90 0x7fc05810cfe0 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.799 7fc05f7fe700 1 --2- 172.21.15.32:0/3683849954 >> v2:172.21.15.86:3300/0 conn(0x7fc0580cffb0 0x7fc0580d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.799 7fc05f7fe700 1 -- 172.21.15.32:0/3683849954 >> 172.21.15.32:0/3683849954 conn(0x7fc0580ce5d0 msgr2=0x7fc0580cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:13.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.799 7fc05f7fe700 1 -- 172.21.15.32:0/3683849954 shutdown_connections 2021-04-09T18:31:13.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.799 7fc05f7fe700 1 -- 172.21.15.32:0/3683849954 wait complete. 2021-04-09T18:31:13.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.819 7fc05f7fe700 1 Processor -- start 2021-04-09T18:31:13.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.819 7fc05f7fe700 1 -- start start 2021-04-09T18:31:13.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05f7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05818e880 0x7fc0581812c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:13.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc058180340 0x7fc0581b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:13.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05f7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581b8c90 0x7fc0580adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:13.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05f7fe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc058180d40 con 0x7fc05818e880 2021-04-09T18:31:13.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05f7fe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0581894f0 con 0x7fc058180340 2021-04-09T18:31:13.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05f7fe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05818ec80 con 0x7fc0581b8c90 2021-04-09T18:31:13.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05d61a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05818e880 0x7fc0581812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:13.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05d61a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05818e880 0x7fc0581812c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56552/0 (socket says 172.21.15.32:56552) 2021-04-09T18:31:13.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05d61a700 1 -- 172.21.15.32:0/308190474 learned_addr learned my addr 172.21.15.32:0/308190474 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:13.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05de1b700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581b8c90 0x7fc0580adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:13.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05ce19700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc058180340 0x7fc0581b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:13.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05d61a700 1 -- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581b8c90 msgr2=0x7fc0580adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:13.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05d61a700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581b8c90 0x7fc0580adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05d61a700 1 -- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc058180340 msgr2=0x7fc0581b8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:13.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05d61a700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc058180340 0x7fc0581b8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:13.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.820 7fc05d61a700 1 -- 172.21.15.32:0/308190474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0581818e0 con 0x7fc05818e880 2021-04-09T18:31:13.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.821 7fc05d61a700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05818e880 0x7fc0581812c0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7fc054006f60 tx=0x7fc05400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:13.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.821 7fc04dffb700 1 -- 172.21.15.32:0/308190474 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc05400bd60 con 0x7fc05818e880 2021-04-09T18:31:13.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.821 7fc04dffb700 1 -- 172.21.15.32:0/308190474 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc054000c90 con 0x7fc05818e880 2021-04-09T18:31:13.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.821 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0581818e0 con 0x7fc05818e880 2021-04-09T18:31:13.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.821 7fc04dffb700 1 -- 172.21.15.32:0/308190474 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc054002220 con 0x7fc05818e880 2021-04-09T18:31:13.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.821 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0581818e0 con 0x7fc05818e880 2021-04-09T18:31:13.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.821 7fc04dffb700 1 -- 172.21.15.32:0/308190474 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc0540052e0 con 0x7fc05818e880 2021-04-09T18:31:13.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.821 7fc04dffb700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc038018900 0x7fc03801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:13.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.822 7fc04dffb700 1 -- 172.21.15.32:0/308190474 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5187+0+0 (secure 0 0 0) 0x7fc054032870 con 0x7fc05818e880 2021-04-09T18:31:13.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.822 7fc05ce19700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc038018900 0x7fc03801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:13.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.822 7fc05ce19700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc038018900 0x7fc03801adc0 crc :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:13.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.842 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05818ec80 con 0x7fc05818e880 2021-04-09T18:31:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:13.845 7fc04dffb700 1 -- 172.21.15.32:0/308190474 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc054000c90 con 0x7fc05818e880 2021-04-09T18:31:14.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.056 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc05818ec80 con 0x7fc05818e880 2021-04-09T18:31:14.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.060 7fc04dffb700 1 -- 172.21.15.32:0/308190474 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc0540052e0 con 0x7fc05818e880 2021-04-09T18:31:14.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc038018900 msgr2=0x7fc03801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc038018900 0x7fc03801adc0 crc :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05818e880 msgr2=0x7fc0581812c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05818e880 0x7fc0581812c0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7fc054006f60 tx=0x7fc05400b970).stop 2021-04-09T18:31:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 shutdown_connections 2021-04-09T18:31:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc038018900 0x7fc03801adc0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0581b8c90 0x7fc0580adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc058180340 0x7fc0581b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 --2- 172.21.15.32:0/308190474 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc05818e880 0x7fc0581812c0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 >> 172.21.15.32:0/308190474 conn(0x7fc058189800 msgr2=0x7fc0581b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:14.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 shutdown_connections 2021-04-09T18:31:14.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.066 7fc05f7fe700 1 -- 172.21.15.32:0/308190474 wait complete. 2021-04-09T18:31:14.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestMultiFilesystems.test_clients' 2021-04-09T18:31:14.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.225 7fb4b524e700 1 Processor -- start 2021-04-09T18:31:14.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.225 7fb4b524e700 1 -- start start 2021-04-09T18:31:14.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.225 7fb4b524e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b010d010 0x7fb4b00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.225 7fb4b524e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b010d480 con 0x7fb4b00d1da0 2021-04-09T18:31:14.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.225 7fb4b524e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b010e1f0 con 0x7fb4b00d8490 2021-04-09T18:31:14.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.225 7fb4b524e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b010e780 con 0x7fb4b010d010 2021-04-09T18:31:14.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4ae59c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb4b00d1da0 0x7fb4b00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37922/0 (socket says 172.21.15.32:37922) 2021-04-09T18:31:14.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4ae59c700 1 -- 172.21.15.32:0/870687051 learned_addr learned my addr 172.21.15.32:0/870687051 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:14.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4aed9d700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b010d010 0x7fb4b00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303330532 0 0) 0x7fb49c000ce0 con 0x7fb4b00d1da0 2021-04-09T18:31:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb494002fe0 con 0x7fb4b00d1da0 2021-04-09T18:31:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336064178 0 0) 0x7fb4a8000d30 con 0x7fb4b00d8490 2021-04-09T18:31:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb494003210 con 0x7fb4b00d8490 2021-04-09T18:31:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 486791724 0 0) 0x7fb49c000ed0 con 0x7fb4b00d1da0 2021-04-09T18:31:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b010d010 msgr2=0x7fb4b00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b010d010 0x7fb4b00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 >> v1:172.21.15.86:6789/0 conn(0x7fb4b00d8490 legacy=0x7fb4b010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b00d8d30 con 0x7fb4b00d1da0 2021-04-09T18:31:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fb49c001600 con 0x7fb4b00d1da0 2021-04-09T18:31:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4aed9d700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b010d010 0x7fb4b00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 >> v1:172.21.15.32:6789/0 conn(0x7fb4b00d1da0 legacy=0x7fb4b00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb494003ba0 0x7fb494005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4940067b0 0x7fb494008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb494009520 0x7fb49400ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb494050300 con 0x7fb494003ba0 2021-04-09T18:31:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb494050520 con 0x7fb494009520 2021-04-09T18:31:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4add9b700 1 -- 172.21.15.32:0/870687051 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb494050740 con 0x7fb4940067b0 2021-04-09T18:31:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4aed9d700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb494003ba0 0x7fb494005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4ae59c700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4940067b0 0x7fb494008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.226 7fb4af59e700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb494009520 0x7fb49400ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4aed9d700 1 -- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4940067b0 msgr2=0x7fb494008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4aed9d700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4940067b0 0x7fb494008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4aed9d700 1 -- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb494009520 msgr2=0x7fb49400ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4aed9d700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb494009520 0x7fb49400ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4aed9d700 1 -- 172.21.15.32:0/870687051 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4940509e0 con 0x7fb494003ba0 2021-04-09T18:31:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4aed9d700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb494003ba0 0x7fb494005fb0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7fb4a0006ac0 tx=0x7fb4a0009ab0).ready entity=mon.0 client_cookie=d68b90b96f6a2eae server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4add9b700 1 -- 172.21.15.32:0/870687051 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4a000c0c0 con 0x7fb494003ba0 2021-04-09T18:31:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4add9b700 1 -- 172.21.15.32:0/870687051 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a000c3c0 con 0x7fb494003ba0 2021-04-09T18:31:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4add9b700 1 -- 172.21.15.32:0/870687051 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4a000c8a0 con 0x7fb494003ba0 2021-04-09T18:31:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4b524e700 1 -- 172.21.15.32:0/870687051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb494003ba0 msgr2=0x7fb494005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4b524e700 1 --2- 172.21.15.32:0/870687051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb494003ba0 0x7fb494005fb0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7fb4a0006ac0 tx=0x7fb4a0009ab0).stop 2021-04-09T18:31:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.227 7fb4af59e700 1 -- 172.21.15.32:0/870687051 reap_dead start 2021-04-09T18:31:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.228 7fb4b524e700 1 -- 172.21.15.32:0/870687051 shutdown_connections 2021-04-09T18:31:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.228 7fb4b524e700 1 -- 172.21.15.32:0/870687051 >> 172.21.15.32:0/870687051 conn(0x7fb4b00ce5f0 msgr2=0x7fb4b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.228 7fb4b524e700 1 -- 172.21.15.32:0/870687051 shutdown_connections 2021-04-09T18:31:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.228 7fb4b524e700 1 -- 172.21.15.32:0/870687051 wait complete. 2021-04-09T18:31:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.248 7fb4b524e700 1 Processor -- start 2021-04-09T18:31:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.248 7fb4b524e700 1 -- start start 2021-04-09T18:31:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4b524e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b00b5360 0x7fb4b01a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4b524e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4b018e140 0x7fb4b01aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4b524e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4b01ad040 0x7fb4b008fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4b524e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b01a5360 con 0x7fb4b018e140 2021-04-09T18:31:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4b524e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b00902c0 con 0x7fb4b01ad040 2021-04-09T18:31:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4b524e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b00904e0 con 0x7fb4b00b5360 2021-04-09T18:31:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4ae59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4b018e140 0x7fb4b01aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4ae59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4b018e140 0x7fb4b01aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56574/0 (socket says 172.21.15.32:56574) 2021-04-09T18:31:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4ae59c700 1 -- 172.21.15.32:0/361349331 learned_addr learned my addr 172.21.15.32:0/361349331 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4aed9d700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b00b5360 0x7fb4b01a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4af59e700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4b01ad040 0x7fb4b008fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4ae59c700 1 -- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b00b5360 msgr2=0x7fb4b01a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4ae59c700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b00b5360 0x7fb4b01a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4ae59c700 1 -- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4b01ad040 msgr2=0x7fb4b008fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4ae59c700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4b01ad040 0x7fb4b008fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4ae59c700 1 -- 172.21.15.32:0/361349331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b0183c80 con 0x7fb4b018e140 2021-04-09T18:31:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4aed9d700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b00b5360 0x7fb4b01a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4af59e700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4b01ad040 0x7fb4b008fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4ae59c700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4b018e140 0x7fb4b01aca20 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 rx=0x7fb49c001590 tx=0x7fb49c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4a77fe700 1 -- 172.21.15.32:0/361349331 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb49c00ad20 con 0x7fb4b018e140 2021-04-09T18:31:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4a77fe700 1 -- 172.21.15.32:0/361349331 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c000ce0 con 0x7fb4b018e140 2021-04-09T18:31:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4b524e700 1 -- 172.21.15.32:0/361349331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b0183c80 con 0x7fb4b018e140 2021-04-09T18:31:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4a77fe700 1 -- 172.21.15.32:0/361349331 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb49c004590 con 0x7fb4b018e140 2021-04-09T18:31:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.249 7fb4b524e700 1 -- 172.21.15.32:0/361349331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b0183c80 con 0x7fb4b018e140 2021-04-09T18:31:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.250 7fb4a77fe700 1 -- 172.21.15.32:0/361349331 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb49c0045b0 con 0x7fb4b018e140 2021-04-09T18:31:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.250 7fb4a77fe700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb48c018930 0x7fb48c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.250 7fb4a77fe700 1 -- 172.21.15.32:0/361349331 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5187+0+0 (secure 0 0 0) 0x7fb49c032890 con 0x7fb4b018e140 2021-04-09T18:31:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.250 7fb4aed9d700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb48c018930 0x7fb48c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.251 7fb4aed9d700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb48c018930 0x7fb48c01ae10 crc :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.270 7fb4b524e700 1 -- 172.21.15.32:0/361349331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b0182d70 con 0x7fb4b018e140 2021-04-09T18:31:14.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.273 7fb4a77fe700 1 -- 172.21.15.32:0/361349331 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb49c000cc0 con 0x7fb4b018e140 2021-04-09T18:31:14.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.462 7fb4b524e700 1 -- 172.21.15.32:0/361349331 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestMultiFilesystems.test_clients"]} v 0) v1 -- 0x7fb4b010d480 con 0x7fb4b018e140 2021-04-09T18:31:14.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.615 7fb4a77fe700 1 -- 172.21.15.32:0/361349331 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestMultiFilesystems.test_clients"]}]=0 v821) v1 ==== 139+0+0 (secure 0 0 0) 0x7fb49c000cc0 con 0x7fb4b018e140 2021-04-09T18:31:14.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 -- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb48c018930 msgr2=0x7fb48c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb48c018930 0x7fb48c01ae10 crc :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 -- 172.21.15.32:0/361349331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4b018e140 msgr2=0x7fb4b01aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4b018e140 0x7fb4b01aca20 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 rx=0x7fb49c001590 tx=0x7fb49c00a930).stop 2021-04-09T18:31:14.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 -- 172.21.15.32:0/361349331 shutdown_connections 2021-04-09T18:31:14.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb48c018930 0x7fb48c01ae10 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4b018e140 0x7fb4b01aca20 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4b01ad040 0x7fb4b008fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 --2- 172.21.15.32:0/361349331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4b00b5360 0x7fb4b01a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 -- 172.21.15.32:0/361349331 >> 172.21.15.32:0/361349331 conn(0x7fb4b019b460 msgr2=0x7fb4b019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:14.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 -- 172.21.15.32:0/361349331 shutdown_connections 2021-04-09T18:31:14.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.628 7fb4b524e700 1 -- 172.21.15.32:0/361349331 wait complete. 2021-04-09T18:31:14.654 INFO:tasks.cephfs_test_runner:test_clients (tasks.cephfs.test_failover.TestMultiFilesystems) ... ok 2021-04-09T18:31:14.655 INFO:tasks.cephfs_test_runner:Starting test: test_grow_shrink (tasks.cephfs.test_failover.TestMultiFilesystems) 2021-04-09T18:31:14.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestMultiFilesystems.test_grow_shrink' 2021-04-09T18:31:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.788 7f4a98158700 1 Processor -- start 2021-04-09T18:31:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.788 7f4a98158700 1 -- start start 2021-04-09T18:31:14.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a98158700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a9010d010 0x7f4a900d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a98158700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9010d480 con 0x7f4a900d8490 2021-04-09T18:31:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a98158700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9010e1f0 con 0x7f4a900d1da0 2021-04-09T18:31:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a98158700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a900d8d30 con 0x7f4a9010d010 2021-04-09T18:31:14.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a95f74700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4a9010d010 0x7f4a900d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a96775700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4a900d8490 0x7f4a9010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37940/0 (socket says 172.21.15.32:37940) 2021-04-09T18:31:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a96775700 1 -- 172.21.15.32:0/2000790900 learned_addr learned my addr 172.21.15.32:0/2000790900 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a95f74700 1 -- 172.21.15.32:0/2000790900 >> v1:172.21.15.86:6789/0 conn(0x7f4a900d1da0 legacy=0x7f4a900d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:14.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a95f74700 1 -- 172.21.15.32:0/2000790900 >> v1:172.21.15.32:6789/0 conn(0x7f4a900d8490 legacy=0x7f4a9010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a95f74700 1 -- 172.21.15.32:0/2000790900 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a900d9220 con 0x7f4a9010d010 2021-04-09T18:31:14.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a94f72700 1 -- 172.21.15.32:0/2000790900 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153591477 0 0) 0x7f4a80000d30 con 0x7f4a900d8490 2021-04-09T18:31:14.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.789 7f4a95f74700 1 --2- 172.21.15.32:0/2000790900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9010d010 0x7f4a900d1780 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 rx=0x7f4a88006650 tx=0x7f4a88009a00).ready entity=mon.0 client_cookie=1253eb2cbd9a5676 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:14.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.790 7f4a94f72700 1 -- 172.21.15.32:0/2000790900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a88009df0 con 0x7f4a9010d010 2021-04-09T18:31:14.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.790 7f4a94f72700 1 -- 172.21.15.32:0/2000790900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88006740 con 0x7f4a9010d010 2021-04-09T18:31:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.790 7f4a94f72700 1 -- 172.21.15.32:0/2000790900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a88009df0 con 0x7f4a9010d010 2021-04-09T18:31:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.790 7f4a98158700 1 -- 172.21.15.32:0/2000790900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9010d010 msgr2=0x7f4a900d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.790 7f4a98158700 1 --2- 172.21.15.32:0/2000790900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9010d010 0x7f4a900d1780 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 rx=0x7f4a88006650 tx=0x7f4a88009a00).stop 2021-04-09T18:31:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.790 7f4a98158700 1 -- 172.21.15.32:0/2000790900 shutdown_connections 2021-04-09T18:31:14.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.790 7f4a98158700 1 --2- 172.21.15.32:0/2000790900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9010d010 0x7f4a900d1780 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.790 7f4a98158700 1 -- 172.21.15.32:0/2000790900 >> 172.21.15.32:0/2000790900 conn(0x7f4a900ce5f0 msgr2=0x7f4a900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:14.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.791 7f4a98158700 1 -- 172.21.15.32:0/2000790900 shutdown_connections 2021-04-09T18:31:14.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.791 7f4a98158700 1 -- 172.21.15.32:0/2000790900 wait complete. 2021-04-09T18:31:14.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a98158700 1 Processor -- start 2021-04-09T18:31:14.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a98158700 1 -- start start 2021-04-09T18:31:14.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a98158700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a901a90a0 0x7f4a901a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a98158700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a901a2940 0x7f4a901a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a98158700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a901ac260 0x7f4a901ac660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a98158700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a901ad200 con 0x7f4a901a2940 2021-04-09T18:31:14.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a98158700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a901a3360 con 0x7f4a901a90a0 2021-04-09T18:31:14.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a98158700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a901a94a0 con 0x7f4a901ac260 2021-04-09T18:31:14.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a95773700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a901a2940 0x7f4a901a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a95773700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a901a2940 0x7f4a901a2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56588/0 (socket says 172.21.15.32:56588) 2021-04-09T18:31:14.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.811 7f4a95773700 1 -- 172.21.15.32:0/2851282683 learned_addr learned my addr 172.21.15.32:0/2851282683 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:14.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a95f74700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a901a90a0 0x7f4a901a85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a96775700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a901ac260 0x7f4a901ac660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a95773700 1 -- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a901ac260 msgr2=0x7f4a901ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a95773700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a901ac260 0x7f4a901ac660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a95773700 1 -- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a901a90a0 msgr2=0x7f4a901a85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:14.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a95773700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a901a90a0 0x7f4a901a85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:14.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a95773700 1 -- 172.21.15.32:0/2851282683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a901a8bd0 con 0x7f4a901a2940 2021-04-09T18:31:14.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a95f74700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a901a90a0 0x7f4a901a85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:14.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a95773700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a901a2940 0x7f4a901a2d40 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 rx=0x7f4a8400b5b0 tx=0x7f4a8400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:14.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a8e7fc700 1 -- 172.21.15.32:0/2851282683 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a8400bd40 con 0x7f4a901a2940 2021-04-09T18:31:14.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a8e7fc700 1 -- 172.21.15.32:0/2851282683 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a84006020 con 0x7f4a901a2940 2021-04-09T18:31:14.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a98158700 1 -- 172.21.15.32:0/2851282683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a901a8bd0 con 0x7f4a901a2940 2021-04-09T18:31:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a8e7fc700 1 -- 172.21.15.32:0/2851282683 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a840054a0 con 0x7f4a901a2940 2021-04-09T18:31:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.812 7f4a98158700 1 -- 172.21.15.32:0/2851282683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a901a8bd0 con 0x7f4a901a2940 2021-04-09T18:31:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.813 7f4a8e7fc700 1 -- 172.21.15.32:0/2851282683 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4a84006200 con 0x7f4a901a2940 2021-04-09T18:31:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.813 7f4a8e7fc700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4a70018930 0x7f4a7001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.813 7f4a8e7fc700 1 -- 172.21.15.32:0/2851282683 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5187+0+0 (secure 0 0 0) 0x7f4a84032880 con 0x7f4a901a2940 2021-04-09T18:31:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.813 7f4a95f74700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4a70018930 0x7f4a7001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.813 7f4a95f74700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4a70018930 0x7f4a7001ae10 crc :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:14.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.833 7f4a98158700 1 -- 172.21.15.32:0/2851282683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9017cb20 con 0x7f4a901a2940 2021-04-09T18:31:14.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:14.836 7f4a8e7fc700 1 -- 172.21.15.32:0/2851282683 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4a8400bd40 con 0x7f4a901a2940 2021-04-09T18:31:15.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.025 7f4a98158700 1 -- 172.21.15.32:0/2851282683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestMultiFilesystems.test_grow_shrink"]} v 0) v1 -- 0x7f4a9010d480 con 0x7f4a901a2940 2021-04-09T18:31:15.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.616 7f4a8e7fc700 1 -- 172.21.15.32:0/2851282683 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestMultiFilesystems.test_grow_shrink"]}]=0 v822) v1 ==== 146+0+0 (secure 0 0 0) 0x7f4a84006110 con 0x7f4a901a2940 2021-04-09T18:31:15.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.641 7f4a98158700 1 -- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4a70018930 msgr2=0x7f4a7001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:15.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.641 7f4a98158700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4a70018930 0x7f4a7001ae10 crc :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.641 7f4a98158700 1 -- 172.21.15.32:0/2851282683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a901a2940 msgr2=0x7f4a901a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:15.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.641 7f4a98158700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a901a2940 0x7f4a901a2d40 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 rx=0x7f4a8400b5b0 tx=0x7f4a8400b950).stop 2021-04-09T18:31:15.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.642 7f4a98158700 1 -- 172.21.15.32:0/2851282683 shutdown_connections 2021-04-09T18:31:15.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.642 7f4a98158700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4a70018930 0x7f4a7001ae10 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.642 7f4a98158700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a901a2940 0x7f4a901a2d40 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.642 7f4a98158700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a901ac260 0x7f4a901ac660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.642 7f4a98158700 1 --2- 172.21.15.32:0/2851282683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a901a90a0 0x7f4a901a85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.642 7f4a98158700 1 -- 172.21.15.32:0/2851282683 >> 172.21.15.32:0/2851282683 conn(0x7f4a9019c7a0 msgr2=0x7f4a90196a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:15.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.642 7f4a98158700 1 -- 172.21.15.32:0/2851282683 shutdown_connections 2021-04-09T18:31:15.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.642 7f4a98158700 1 -- 172.21.15.32:0/2851282683 wait complete. 2021-04-09T18:31:15.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:31:15.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.807 7fb048b3f700 1 Processor -- start 2021-04-09T18:31:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.807 7fb048b3f700 1 -- start start 2021-04-09T18:31:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.808 7fb048b3f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb04410d010 0x7fb0440d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:15.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.808 7fb048b3f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0440d1da0 0x7fb0440d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:15.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.808 7fb048b3f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04410d480 con 0x7fb0440d82c0 2021-04-09T18:31:15.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.808 7fb048b3f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0440d8d30 con 0x7fb0440d1da0 2021-04-09T18:31:15.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.808 7fb048b3f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0440d9220 con 0x7fb04410d010 2021-04-09T18:31:15.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.808 7fb04205d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0440d1da0 0x7fb0440d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:15.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.808 7fb04205d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0440d1da0 0x7fb0440d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56598/0 (socket says 172.21.15.32:56598) 2021-04-09T18:31:15.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.808 7fb04205d700 1 -- 172.21.15.32:0/540192601 learned_addr learned my addr 172.21.15.32:0/540192601 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:15.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.808 7fb04285e700 1 --2- 172.21.15.32:0/540192601 >> v2:172.21.15.86:3300/0 conn(0x7fb04410d010 0x7fb0440d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:15.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb04205d700 1 -- 172.21.15.32:0/540192601 >> v2:172.21.15.86:3300/0 conn(0x7fb04410d010 msgr2=0x7fb0440d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:15.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb04205d700 1 --2- 172.21.15.32:0/540192601 >> v2:172.21.15.86:3300/0 conn(0x7fb04410d010 0x7fb0440d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb04205d700 1 -- 172.21.15.32:0/540192601 >> v1:172.21.15.86:6789/0 conn(0x7fb0440d82c0 legacy=0x7fb04410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:15.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb04205d700 1 -- 172.21.15.32:0/540192601 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0440d86f0 con 0x7fb0440d1da0 2021-04-09T18:31:15.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb04185c700 1 -- 172.21.15.32:0/540192601 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212700237 0 0) 0x7fb03c000d30 con 0x7fb0440d82c0 2021-04-09T18:31:15.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb04205d700 1 --2- 172.21.15.32:0/540192601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440d1da0 0x7fb0440d7ca0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 rx=0x7fb038006700 tx=0x7fb038009b00).ready entity=mon.0 client_cookie=480faa29d89bbd02 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb04185c700 1 -- 172.21.15.32:0/540192601 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb03800c0c0 con 0x7fb0440d1da0 2021-04-09T18:31:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb04185c700 1 -- 172.21.15.32:0/540192601 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03800c400 con 0x7fb0440d1da0 2021-04-09T18:31:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb04185c700 1 -- 172.21.15.32:0/540192601 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb03800c8e0 con 0x7fb0440d1da0 2021-04-09T18:31:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb048b3f700 1 -- 172.21.15.32:0/540192601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440d1da0 msgr2=0x7fb0440d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:15.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.809 7fb048b3f700 1 --2- 172.21.15.32:0/540192601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440d1da0 0x7fb0440d7ca0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 rx=0x7fb038006700 tx=0x7fb038009b00).stop 2021-04-09T18:31:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.810 7fb048b3f700 1 -- 172.21.15.32:0/540192601 shutdown_connections 2021-04-09T18:31:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.810 7fb048b3f700 1 --2- 172.21.15.32:0/540192601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440d1da0 0x7fb0440d7ca0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.810 7fb048b3f700 1 --2- 172.21.15.32:0/540192601 >> v2:172.21.15.86:3300/0 conn(0x7fb04410d010 0x7fb0440d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.810 7fb048b3f700 1 -- 172.21.15.32:0/540192601 >> 172.21.15.32:0/540192601 conn(0x7fb0440ce5f0 msgr2=0x7fb0440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:15.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.810 7fb048b3f700 1 -- 172.21.15.32:0/540192601 shutdown_connections 2021-04-09T18:31:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.810 7fb048b3f700 1 -- 172.21.15.32:0/540192601 wait complete. 2021-04-09T18:31:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.830 7fb048b3f700 1 Processor -- start 2021-04-09T18:31:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.830 7fb048b3f700 1 -- start start 2021-04-09T18:31:15.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb048b3f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04417bb20 0x7fb0440c1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb048b3f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0440c1890 0x7fb0440b9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb048b3f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440ba4b0 0x7fb0440ba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb048b3f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb044180680 con 0x7fb0440ba4b0 2021-04-09T18:31:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb048b3f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440c0cf0 con 0x7fb04417bb20 2021-04-09T18:31:15.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb048b3f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04417c450 con 0x7fb0440c1890 2021-04-09T18:31:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440ba4b0 0x7fb0440ba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04285e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04417bb20 0x7fb0440c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440ba4b0 0x7fb0440ba8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56604/0 (socket says 172.21.15.32:56604) 2021-04-09T18:31:15.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04305f700 1 -- 172.21.15.32:0/229700203 learned_addr learned my addr 172.21.15.32:0/229700203 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04205d700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0440c1890 0x7fb0440b9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04305f700 1 -- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0440c1890 msgr2=0x7fb0440b9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04305f700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0440c1890 0x7fb0440b9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04305f700 1 -- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04417bb20 msgr2=0x7fb0440c1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04305f700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04417bb20 0x7fb0440c1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04305f700 1 -- 172.21.15.32:0/229700203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0440c1c90 con 0x7fb0440ba4b0 2021-04-09T18:31:15.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04285e700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04417bb20 0x7fb0440c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:15.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb04305f700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440ba4b0 0x7fb0440ba8b0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7fb03c009980 tx=0x7fb03c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:15.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb0337fe700 1 -- 172.21.15.32:0/229700203 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb03c005870 con 0x7fb0440ba4b0 2021-04-09T18:31:15.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb0337fe700 1 -- 172.21.15.32:0/229700203 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c005b70 con 0x7fb0440ba4b0 2021-04-09T18:31:15.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb048b3f700 1 -- 172.21.15.32:0/229700203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0440c1c90 con 0x7fb0440ba4b0 2021-04-09T18:31:15.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb0337fe700 1 -- 172.21.15.32:0/229700203 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb03c006030 con 0x7fb0440ba4b0 2021-04-09T18:31:15.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.831 7fb048b3f700 1 -- 172.21.15.32:0/229700203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0440c1c90 con 0x7fb0440ba4b0 2021-04-09T18:31:15.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.832 7fb0337fe700 1 -- 172.21.15.32:0/229700203 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb03c006030 con 0x7fb0440ba4b0 2021-04-09T18:31:15.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.832 7fb0337fe700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb028018930 0x7fb02801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:15.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.832 7fb0337fe700 1 -- 172.21.15.32:0/229700203 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5187+0+0 (secure 0 0 0) 0x7fb03c032c90 con 0x7fb0440ba4b0 2021-04-09T18:31:15.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.832 7fb04285e700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb028018930 0x7fb02801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:15.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.833 7fb04285e700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb028018930 0x7fb02801ae10 crc :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:15.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.852 7fb048b3f700 1 -- 172.21.15.32:0/229700203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04418ec20 con 0x7fb0440ba4b0 2021-04-09T18:31:15.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:15.856 7fb0337fe700 1 -- 172.21.15.32:0/229700203 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb03c00be10 con 0x7fb0440ba4b0 2021-04-09T18:31:16.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.070 7fb048b3f700 1 -- 172.21.15.32:0/229700203 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb04418ec20 con 0x7fb0440ba4b0 2021-04-09T18:31:16.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.074 7fb0337fe700 1 -- 172.21.15.32:0/229700203 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb03c00be10 con 0x7fb0440ba4b0 2021-04-09T18:31:16.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 -- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb028018930 msgr2=0x7fb02801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:16.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb028018930 0x7fb02801ae10 crc :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:16.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 -- 172.21.15.32:0/229700203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440ba4b0 msgr2=0x7fb0440ba8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:16.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440ba4b0 0x7fb0440ba8b0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7fb03c009980 tx=0x7fb03c00bcd0).stop 2021-04-09T18:31:16.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 -- 172.21.15.32:0/229700203 shutdown_connections 2021-04-09T18:31:16.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb028018930 0x7fb02801ae10 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:16.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0440ba4b0 0x7fb0440ba8b0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:16.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0440c1890 0x7fb0440b9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:16.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 --2- 172.21.15.32:0/229700203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04417bb20 0x7fb0440c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:16.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 -- 172.21.15.32:0/229700203 >> 172.21.15.32:0/229700203 conn(0x7fb044181390 msgr2=0x7fb0440adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:16.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 -- 172.21.15.32:0/229700203 shutdown_connections 2021-04-09T18:31:16.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.080 7fb048b3f700 1 -- 172.21.15.32:0/229700203 wait complete. 2021-04-09T18:31:16.106 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:31:16.106 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:31:16.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:31:16.222 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:31:16.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:31:16.223 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:31:16.223 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:31:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:31:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:31:16.268 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:31:16.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:31:16.269 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:31:16.269 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:31:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:31:16.312 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:31:16.312 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:31:16.312 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:31:16.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.445 7f7457c2e700 1 Processor -- start 2021-04-09T18:31:16.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.445 7f7457c2e700 1 -- start start 2021-04-09T18:31:16.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7457c2e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74500d7ca0 0x7f74500d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7457c2e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745010d480 con 0x7f745010d010 2021-04-09T18:31:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7457c2e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74500d86c0 con 0x7f745010da00 2021-04-09T18:31:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7457c2e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74500d9220 con 0x7f74500d7ca0 2021-04-09T18:31:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7455249700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f74500d7ca0 0x7f74500d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:16.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7455a4a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f745010d010 0x7f74500d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37966/0 (socket says 172.21.15.32:37966) 2021-04-09T18:31:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7455a4a700 1 -- 172.21.15.32:0/2847499771 learned_addr learned my addr 172.21.15.32:0/2847499771 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7455249700 1 -- 172.21.15.32:0/2847499771 >> v1:172.21.15.86:6789/0 conn(0x7f745010da00 legacy=0x7f745010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7455249700 1 -- 172.21.15.32:0/2847499771 >> v1:172.21.15.32:6789/0 conn(0x7f745010d010 legacy=0x7f74500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:16.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7455249700 1 -- 172.21.15.32:0/2847499771 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74500d1f70 con 0x7f74500d7ca0 2021-04-09T18:31:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7454a48700 1 -- 172.21.15.32:0/2847499771 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437027058 0 0) 0x7f744c000ce0 con 0x7f745010d010 2021-04-09T18:31:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7455249700 1 --2- 172.21.15.32:0/2847499771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500d7ca0 0x7f74500d80a0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 rx=0x7f7440006650 tx=0x7f7440009a80).ready entity=mon.0 client_cookie=1b909a0a76a04476 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.446 7f7454a48700 1 -- 172.21.15.32:0/2847499771 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74400009b0 con 0x7f74500d7ca0 2021-04-09T18:31:16.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.447 7f7454a48700 1 -- 172.21.15.32:0/2847499771 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744000c0c0 con 0x7f74500d7ca0 2021-04-09T18:31:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.447 7f7454a48700 1 -- 172.21.15.32:0/2847499771 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f744000c510 con 0x7f74500d7ca0 2021-04-09T18:31:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.447 7f7457c2e700 1 -- 172.21.15.32:0/2847499771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500d7ca0 msgr2=0x7f74500d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.447 7f7457c2e700 1 --2- 172.21.15.32:0/2847499771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500d7ca0 0x7f74500d80a0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 rx=0x7f7440006650 tx=0x7f7440009a80).stop 2021-04-09T18:31:16.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.447 7f7457c2e700 1 -- 172.21.15.32:0/2847499771 shutdown_connections 2021-04-09T18:31:16.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.447 7f7457c2e700 1 --2- 172.21.15.32:0/2847499771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500d7ca0 0x7f74500d80a0 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:16.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.447 7f7457c2e700 1 -- 172.21.15.32:0/2847499771 >> 172.21.15.32:0/2847499771 conn(0x7f74500ce5f0 msgr2=0x7f74500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:16.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.447 7f7457c2e700 1 -- 172.21.15.32:0/2847499771 shutdown_connections 2021-04-09T18:31:16.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.447 7f7457c2e700 1 -- 172.21.15.32:0/2847499771 wait complete. 2021-04-09T18:31:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.467 7f7457c2e700 1 Processor -- start 2021-04-09T18:31:16.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7457c2e700 1 -- start start 2021-04-09T18:31:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7457c2e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74500a75d0 0x7f74501bef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:16.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7457c2e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500bc870 0x7f74500b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:16.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7457c2e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74501bf520 0x7f74500b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:16.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7457c2e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74501abd30 con 0x7f74500bc870 2021-04-09T18:31:16.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7457c2e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74500b4290 con 0x7f74500a75d0 2021-04-09T18:31:16.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7457c2e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74500a7b00 con 0x7f74501bf520 2021-04-09T18:31:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7455249700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500bc870 0x7f74500b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7455249700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500bc870 0x7f74500b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56618/0 (socket says 172.21.15.32:56618) 2021-04-09T18:31:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7455249700 1 -- 172.21.15.32:0/3398733247 learned_addr learned my addr 172.21.15.32:0/3398733247 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f7455a4a700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74500a75d0 0x7f74501bef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:16.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.468 7f745624b700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74501bf520 0x7f74500b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:16.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f7455249700 1 -- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74501bf520 msgr2=0x7f74500b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:16.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f7455249700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74501bf520 0x7f74500b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:16.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f7455249700 1 -- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74500a75d0 msgr2=0x7f74501bef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:16.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f7455249700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74500a75d0 0x7f74501bef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:16.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f7455249700 1 -- 172.21.15.32:0/3398733247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74500a7d20 con 0x7f74500bc870 2021-04-09T18:31:16.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f7455249700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500bc870 0x7f74500b3560 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7f7440009650 tx=0x7f744000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f74467fc700 1 -- 172.21.15.32:0/3398733247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74400093a0 con 0x7f74500bc870 2021-04-09T18:31:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f74467fc700 1 -- 172.21.15.32:0/3398733247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7440006070 con 0x7f74500bc870 2021-04-09T18:31:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74500a7d20 con 0x7f74500bc870 2021-04-09T18:31:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f74467fc700 1 -- 172.21.15.32:0/3398733247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74400052e0 con 0x7f74500bc870 2021-04-09T18:31:16.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.469 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74500a7d20 con 0x7f74500bc870 2021-04-09T18:31:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.470 7f74467fc700 1 -- 172.21.15.32:0/3398733247 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7440009300 con 0x7f74500bc870 2021-04-09T18:31:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.470 7f74467fc700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.470 7f74467fc700 1 -- 172.21.15.32:0/3398733247 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5187+0+0 (secure 0 0 0) 0x7f7440032890 con 0x7f74500bc870 2021-04-09T18:31:16.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.470 7f7455a4a700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:16.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.470 7f7455a4a700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 crc :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:16.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.490 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74501a4770 con 0x7f74500bc870 2021-04-09T18:31:16.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.493 7f74467fc700 1 -- 172.21.15.32:0/3398733247 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7440006030 con 0x7f74500bc870 2021-04-09T18:31:16.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.683 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f7450112070 con 0x7f74500bc870 2021-04-09T18:31:16.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.684 7f74467fc700 1 -- 172.21.15.32:0/3398733247 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v195) v1 ==== 81+0+26106 (secure 0 0 0) 0x7f7440000940 con 0x7f74500bc870 2021-04-09T18:31:16.702 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:16.702 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:31:16.703 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 195, 2021-04-09T18:31:16.703 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:31:16.703 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:31:16.703 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:31:13.612535", 2021-04-09T18:31:16.704 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:31:16.704 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:31:16.704 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:31:16.705 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:31:16.705 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:31:16.705 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:31:16.706 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:31:16.706 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:31:16.706 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:31:16.706 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:31:16.707 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:31:16.707 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:31:16.707 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:31:16.707 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:31:16.708 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:31:16.708 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 33, 2021-04-09T18:31:16.708 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:31:16.709 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:31:16.709 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:31:16.709 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:31:16.709 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:31:16.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.710 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:31:16.710 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:31:16.710 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:31:16.710 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:31:16.711 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:31:16.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:31:16.711 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:31:16.711 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:31:16.712 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:31:16.712 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:31:16.712 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:31:16.712 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:31:16.713 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:31:16.713 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:31:16.713 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:31:16.713 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:31:16.713 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:31:16.714 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:31:16.714 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:31:16.714 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:31:16.714 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:31:16.715 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:31:16.715 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:31:16.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.715 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:31:16.715 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:31:16.716 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:31:16.716 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:31:16.716 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:31:16.716 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:31:16.717 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:31:16.717 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:31:16.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:31:16.718 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:31:16.718 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:31:16.718 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:31:16.719 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:31:16.719 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:31:16.719 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:31:16.720 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:31:16.720 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:31:16.720 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:31:16.720 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:31:16.720 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:31:16.721 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:31:16.721 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:31:16.722 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:31:16.722 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:31:16.722 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:31:16.723 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:31:16.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:31:16.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.724 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:31:16.724 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:31:16.724 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:31:16.724 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:31:16.724 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:31:16.725 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:31:16.725 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:31:16.725 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:31:16.725 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:31:16.725 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:31:16.726 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:31:16.726 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:31:16.726 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:31:16.727 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:31:16.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.727 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:31:16.728 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:31:16.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.728 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:31:16.728 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:31:16.729 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:16.729 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:16.729 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:16.729 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:16.730 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:16.730 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:16.730 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:16.731 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 181, 2021-04-09T18:31:16.731 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:16.731 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:16.731 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:16.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:31:16.732 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:16.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.733 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.734 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.735 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.736 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:31:16.737 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:16.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.737 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.737 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.738 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:31:16.738 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:16.738 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.738 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.739 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:16.739 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.740 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.740 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:31:16.740 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:16.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.741 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:31:16.741 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:16.742 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.742 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.742 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:31:16.743 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.744 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.745 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:31:16.746 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:16.747 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.748 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.749 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:31:16.749 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:31:16.750 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:16.750 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:16.751 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:16.751 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:16.752 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:16.752 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:16.752 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:16.753 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 178, 2021-04-09T18:31:16.753 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:16.754 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:16.754 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:16.754 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.756 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:31:16.756 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:16.757 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.757 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.757 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.758 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:31:16.758 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:16.758 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.758 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.759 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:16.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.760 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:31:16.761 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:16.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.763 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:31:16.763 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:16.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.764 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.764 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:16.764 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.765 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:31:16.765 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:16.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.766 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.766 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.766 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:31:16.766 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:16.767 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.767 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.767 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:16.767 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.768 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:31:16.768 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:16.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.769 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.769 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.769 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:31:16.769 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:31:16.770 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.770 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.770 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:31:16.770 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:31:16.771 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:31:16.771 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:31:16.771 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:16.771 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:16.772 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:16.772 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.772 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.772 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:31:16.773 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:31:16.773 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:16.773 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:16.773 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:16.774 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:16.774 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:16.774 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:16.774 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:16.774 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 182, 2021-04-09T18:31:16.775 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:16.775 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:16.775 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:16.775 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.776 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.776 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:31:16.776 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:16.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.777 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.777 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.777 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:31:16.777 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:16.777 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:31:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:31:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:31:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:31:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:31:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:31:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:31:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:31:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:31:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:31:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:31:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:16.789 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.789 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:31:16.789 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:31:16.790 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:16.790 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:16.790 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:16.790 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:16.791 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:16.791 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:16.791 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:16.791 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 182, 2021-04-09T18:31:16.791 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:16.792 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:16.792 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:16.792 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.792 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.793 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.793 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:31:16.793 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:16.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.794 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.794 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:31:16.794 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:16.794 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.795 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.795 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:16.795 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.795 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.796 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.796 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:31:16.796 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:16.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.797 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:31:16.797 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:16.797 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.798 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.798 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:16.798 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.798 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.799 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.799 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:31:16.799 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:16.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.800 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.800 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.800 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:31:16.800 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:16.800 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.801 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.801 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:16.801 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.802 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.802 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:31:16.802 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:16.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.803 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:31:16.803 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:31:16.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.804 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.804 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:31:16.804 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:31:16.804 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:31:16.805 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:31:16.805 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:16.805 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:16.805 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:16.806 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.806 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:31:16.806 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:31:16.807 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:16.807 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:16.807 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:16.807 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:16.807 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:16.808 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:16.808 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:16.808 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 178, 2021-04-09T18:31:16.808 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:16.809 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:16.809 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:16.809 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.810 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:31:16.810 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:16.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.810 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.811 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:31:16.811 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:16.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.811 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.812 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:16.812 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.812 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.813 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:31:16.813 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:16.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.813 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.814 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:31:16.814 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:16.814 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.814 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.815 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:16.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.816 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:31:16.816 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:16.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.816 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.817 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:31:16.817 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:16.817 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.817 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.818 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:16.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.819 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:31:16.819 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:16.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.820 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.820 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:31:16.820 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:31:16.820 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.820 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.821 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:31:16.821 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:31:16.821 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:31:16.821 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:31:16.822 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:16.822 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:16.822 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:16.822 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.823 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:31:16.823 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:31:16.823 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:16.824 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:16.824 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:16.824 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:16.824 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:16.824 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:16.825 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:16.825 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 182, 2021-04-09T18:31:16.825 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:16.825 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:16.826 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:16.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:31:16.827 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:16.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:31:16.828 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:16.974 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.975 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.975 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:16.976 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.977 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:31:16.977 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:16.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.979 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:31:16.979 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:16.979 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.980 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.980 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:16.980 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:31:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:16.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.983 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:31:16.983 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:16.983 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.983 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.984 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:16.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.985 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:31:16.985 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:16.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.988 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:31:16.988 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:31:16.989 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.989 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.989 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:31:16.990 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:31:16.990 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:31:16.990 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:31:16.991 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:16.991 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:16.991 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:16.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.992 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.992 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:31:16.992 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:31:16.992 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:16.993 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:16.993 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:16.993 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:16.994 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:16.994 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:16.994 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:16.995 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 181, 2021-04-09T18:31:16.995 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:16.995 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:16.995 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:16.995 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:16.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.996 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:16.996 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:31:16.996 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:16.997 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:16.998 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:31:16.998 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:16.998 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:16.999 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:16.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:16.999 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:16.999 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:17.000 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:17.000 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:31:17.000 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:17.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:17.002 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:31:17.002 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:17.002 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.002 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:17.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.005 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:17.006 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:17.007 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.008 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:17.009 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:31:17.010 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:17.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.012 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:17.013 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:31:17.014 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:17.015 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.016 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:17.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.017 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:17.017 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:17.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.018 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:17.019 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:31:17.020 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:17.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:17.021 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:31:17.021 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:31:17.022 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.022 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:17.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.023 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:31:17.023 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:31:17.023 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:31:17.023 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:31:17.024 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:17.024 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:17.024 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:17.024 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:17.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.025 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:31:17.025 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:31:17.026 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:31:17.026 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:31:17.026 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:31:17.027 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:31:17.027 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:31:17.027 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:31:17.027 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:31:17.028 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 176, 2021-04-09T18:31:17.028 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:31:17.028 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:31:17.028 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:31:17.028 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:17.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:17.029 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:31:17.029 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:17.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.030 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:17.030 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:31:17.031 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:17.031 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.031 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:17.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.032 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:31:17.032 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:17.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:17.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:31:17.033 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:17.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:17.034 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:31:17.034 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:17.034 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.034 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:17.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.035 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:31:17.035 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:17.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:17.036 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:31:17.036 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:17.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.037 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:17.037 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:31:17.037 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:17.037 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.037 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:17.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.038 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:31:17.038 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:31:17.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.039 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:31:17.039 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:31:17.039 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:17.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:31:17.040 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:31:17.040 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:31:17.041 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.041 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:17.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.041 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:31:17.042 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:31:17.042 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:31:17.042 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:31:17.042 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:31:17.042 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:31:17.043 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:31:17.043 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:17.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.043 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:31:17.043 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:31:17.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:31:17.044 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:17.044 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:17.045 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:17.045 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:17.045 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:17.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.046 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:31:17.046 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:17.046 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:17.046 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:17.047 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:17.047 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:17.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:31:17.048 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:17.048 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:17.048 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:17.048 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:17.049 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:17.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.049 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:31:17.049 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:17.050 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:17.050 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:17.050 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:17.050 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:17.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.051 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:31:17.051 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:17.051 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:17.052 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:17.052 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:17.052 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:17.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.053 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:31:17.053 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:17.053 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:17.053 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:17.054 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:17.054 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:17.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.054 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:31:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:17.055 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:31:17.056 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:31:17.057 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:31:17.057 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:31:17.057 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:31:17.057 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:31:17.058 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.058 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:31:17.058 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:31:17.058 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:31:17.058 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:31:17.059 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:31:17.059 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:31:17.059 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/2933940031": "2021-04-10 18:31:06.973211", 2021-04-09T18:31:17.059 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/264493238": "2021-04-10 18:31:06.012116", 2021-04-09T18:31:17.060 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/3729993394": "2021-04-10 18:30:30.261199", 2021-04-09T18:31:17.060 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/3729993394": "2021-04-10 18:30:30.261199", 2021-04-09T18:31:17.060 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/264493238": "2021-04-10 18:31:06.012116", 2021-04-09T18:31:17.060 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/3158551863": "2021-04-10 18:30:30.264043", 2021-04-09T18:31:17.060 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/558658679": "2021-04-10 18:30:47.109571", 2021-04-09T18:31:17.061 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/2933940031": "2021-04-10 18:31:06.973211", 2021-04-09T18:31:17.061 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/3158551863": "2021-04-10 18:30:30.264043", 2021-04-09T18:31:17.061 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/558658679": "2021-04-10 18:30:47.109571" 2021-04-09T18:31:17.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.062 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:31:17.062 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:31:17.062 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:31:17.062 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:31:17.062 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:31:17.063 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:31:17.063 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:31:17.063 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:31:17.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:17.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:17.064 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:31:17.064 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:31:17.064 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:31:17.064 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:31:17.065 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:31:17.065 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:31:17.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 msgr2=0x7f743801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 crc :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500bc870 msgr2=0x7f74500b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500bc870 0x7f74500b3560 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7f7440009650 tx=0x7f744000b8e0).stop 2021-04-09T18:31:17.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 shutdown_connections 2021-04-09T18:31:17.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7438018930 0x7f743801ae10 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74500bc870 0x7f74500b3560 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74501bf520 0x7f74500b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 --2- 172.21.15.32:0/3398733247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74500a75d0 0x7f74501bef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 >> 172.21.15.32:0/3398733247 conn(0x7f74501acbe0 msgr2=0x7f745008dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:17.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.688 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 shutdown_connections 2021-04-09T18:31:17.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:16.689 7f7457c2e700 1 -- 172.21.15.32:0/3398733247 wait complete. 2021-04-09T18:31:17.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:17.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.206 7fc707e1c700 1 Processor -- start 2021-04-09T18:31:17.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.206 7fc707e1c700 1 -- start start 2021-04-09T18:31:17.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.206 7fc707e1c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc70010d010 0x7fc7000d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.206 7fc707e1c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7000d1da0 0x7fc7000d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.206 7fc707e1c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70010d480 con 0x7fc7000d82c0 2021-04-09T18:31:17.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.206 7fc707e1c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7000d8d30 con 0x7fc7000d1da0 2021-04-09T18:31:17.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.206 7fc707e1c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7000d9220 con 0x7fc70010d010 2021-04-09T18:31:17.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc705437700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc7000d1da0 0x7fc7000d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc706439700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc7000d82c0 0x7fc70010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37982/0 (socket says 172.21.15.32:37982) 2021-04-09T18:31:17.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc706439700 1 -- 172.21.15.32:0/3343766418 learned_addr learned my addr 172.21.15.32:0/3343766418 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:17.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc705c38700 1 --2- 172.21.15.32:0/3343766418 >> v2:172.21.15.86:3300/0 conn(0x7fc70010d010 0x7fc7000d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc705437700 1 -- 172.21.15.32:0/3343766418 >> v2:172.21.15.86:3300/0 conn(0x7fc70010d010 msgr2=0x7fc7000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:17.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc705437700 1 --2- 172.21.15.32:0/3343766418 >> v2:172.21.15.86:3300/0 conn(0x7fc70010d010 0x7fc7000d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc705437700 1 -- 172.21.15.32:0/3343766418 >> v1:172.21.15.32:6789/0 conn(0x7fc7000d82c0 legacy=0x7fc70010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc704c36700 1 -- 172.21.15.32:0/3343766418 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477007994 0 0) 0x7fc6f8000d30 con 0x7fc7000d82c0 2021-04-09T18:31:17.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc705437700 1 -- 172.21.15.32:0/3343766418 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000d86f0 con 0x7fc7000d1da0 2021-04-09T18:31:17.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc705437700 1 --2- 172.21.15.32:0/3343766418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7000d1da0 0x7fc7000d7ca0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 rx=0x7fc6f0006650 tx=0x7fc6f0009a00).ready entity=mon.0 client_cookie=e34ccb3e803776db server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:17.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc704c36700 1 -- 172.21.15.32:0/3343766418 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f0009df0 con 0x7fc7000d1da0 2021-04-09T18:31:17.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc704c36700 1 -- 172.21.15.32:0/3343766418 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f00009b0 con 0x7fc7000d1da0 2021-04-09T18:31:17.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc704c36700 1 -- 172.21.15.32:0/3343766418 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f000c2b0 con 0x7fc7000d1da0 2021-04-09T18:31:17.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc707e1c700 1 -- 172.21.15.32:0/3343766418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7000d1da0 msgr2=0x7fc7000d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.207 7fc707e1c700 1 --2- 172.21.15.32:0/3343766418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7000d1da0 0x7fc7000d7ca0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 rx=0x7fc6f0006650 tx=0x7fc6f0009a00).stop 2021-04-09T18:31:17.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.208 7fc707e1c700 1 -- 172.21.15.32:0/3343766418 shutdown_connections 2021-04-09T18:31:17.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.208 7fc707e1c700 1 --2- 172.21.15.32:0/3343766418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7000d1da0 0x7fc7000d7ca0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.208 7fc707e1c700 1 --2- 172.21.15.32:0/3343766418 >> v2:172.21.15.86:3300/0 conn(0x7fc70010d010 0x7fc7000d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.208 7fc707e1c700 1 -- 172.21.15.32:0/3343766418 >> 172.21.15.32:0/3343766418 conn(0x7fc7000ce5f0 msgr2=0x7fc7000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:17.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.208 7fc707e1c700 1 -- 172.21.15.32:0/3343766418 shutdown_connections 2021-04-09T18:31:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.208 7fc707e1c700 1 -- 172.21.15.32:0/3343766418 wait complete. 2021-04-09T18:31:17.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.228 7fc707e1c700 1 Processor -- start 2021-04-09T18:31:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.228 7fc707e1c700 1 -- start start 2021-04-09T18:31:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc707e1c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7000b5360 0x7fc7001a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc707e1c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc70018e140 0x7fc7001aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc707e1c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7001ad040 0x7fc70008fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc707e1c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7001a5360 con 0x7fc7001ad040 2021-04-09T18:31:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc707e1c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000902c0 con 0x7fc7000b5360 2021-04-09T18:31:17.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc707e1c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000904e0 con 0x7fc70018e140 2021-04-09T18:31:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc706439700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7001ad040 0x7fc70008fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc706439700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7001ad040 0x7fc70008fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56630/0 (socket says 172.21.15.32:56630) 2021-04-09T18:31:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc706439700 1 -- 172.21.15.32:0/753780056 learned_addr learned my addr 172.21.15.32:0/753780056 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:17.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc705c38700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7000b5360 0x7fc7001a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc705437700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc70018e140 0x7fc7001aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc706439700 1 -- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc70018e140 msgr2=0x7fc7001aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc706439700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc70018e140 0x7fc7001aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc706439700 1 -- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7000b5360 msgr2=0x7fc7001a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc706439700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7000b5360 0x7fc7001a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc706439700 1 -- 172.21.15.32:0/753780056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc700183c80 con 0x7fc7001ad040 2021-04-09T18:31:17.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.229 7fc706439700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7001ad040 0x7fc70008fca0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 rx=0x7fc6f8007f90 tx=0x7fc6f800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.230 7fc6f67fc700 1 -- 172.21.15.32:0/753780056 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f8000cc0 con 0x7fc7001ad040 2021-04-09T18:31:17.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.230 7fc6f67fc700 1 -- 172.21.15.32:0/753780056 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f8007020 con 0x7fc7001ad040 2021-04-09T18:31:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.230 7fc707e1c700 1 -- 172.21.15.32:0/753780056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc700183c80 con 0x7fc7001ad040 2021-04-09T18:31:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.230 7fc6f67fc700 1 -- 172.21.15.32:0/753780056 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6f8007500 con 0x7fc7001ad040 2021-04-09T18:31:17.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.230 7fc707e1c700 1 -- 172.21.15.32:0/753780056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc700183c80 con 0x7fc7001ad040 2021-04-09T18:31:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.230 7fc6f67fc700 1 -- 172.21.15.32:0/753780056 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc6f8007200 con 0x7fc7001ad040 2021-04-09T18:31:17.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.230 7fc6f67fc700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6e8018930 0x7fc6e801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.231 7fc6f67fc700 1 -- 172.21.15.32:0/753780056 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5187+0+0 (secure 0 0 0) 0x7fc6f8032f50 con 0x7fc7001ad040 2021-04-09T18:31:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.231 7fc705c38700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6e8018930 0x7fc6e801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.231 7fc705c38700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6e8018930 0x7fc6e801ae10 crc :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.250 7fc707e1c700 1 -- 172.21.15.32:0/753780056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc700182d70 con 0x7fc7001ad040 2021-04-09T18:31:17.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.254 7fc6f67fc700 1 -- 172.21.15.32:0/753780056 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc6f8001270 con 0x7fc7001ad040 2021-04-09T18:31:17.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.445 7fc707e1c700 1 -- 172.21.15.32:0/753780056 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc70010d480 con 0x7fc7001ad040 2021-04-09T18:31:17.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.446 7fc6f67fc700 1 -- 172.21.15.32:0/753780056 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+1519 (secure 0 0 0) 0x7fc6f8007020 con 0x7fc7001ad040 2021-04-09T18:31:17.463 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 343 2021-04-09T18:31:17.463 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:17.463 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":343,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11382,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1341967017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1341967017},{"type":"v1","addr":"172.21.15.32:6833","nonce":1341967017}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":343},{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330},{"gid":58907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2312178739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2312178739},{"type":"v1","addr":"172.21.15.86:6836","nonce":2312178739}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":342}],"filesystems":[]} 2021-04-09T18:31:17.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 -- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6e8018930 msgr2=0x7fc6e801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6e8018930 0x7fc6e801ae10 crc :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 -- 172.21.15.32:0/753780056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7001ad040 msgr2=0x7fc70008fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7001ad040 0x7fc70008fca0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 rx=0x7fc6f8007f90 tx=0x7fc6f800ac50).stop 2021-04-09T18:31:17.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 -- 172.21.15.32:0/753780056 shutdown_connections 2021-04-09T18:31:17.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc6e8018930 0x7fc6e801ae10 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc7001ad040 0x7fc70008fca0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc70018e140 0x7fc7001aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 --2- 172.21.15.32:0/753780056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7000b5360 0x7fc7001a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 -- 172.21.15.32:0/753780056 >> 172.21.15.32:0/753780056 conn(0x7fc70019b460 msgr2=0x7fc70019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:17.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 -- 172.21.15.32:0/753780056 shutdown_connections 2021-04-09T18:31:17.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.448 7fc707e1c700 1 -- 172.21.15.32:0/753780056 wait complete. 2021-04-09T18:31:17.474 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:17.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.607 7fb345ac3700 1 Processor -- start 2021-04-09T18:31:17.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.607 7fb345ac3700 1 -- start start 2021-04-09T18:31:17.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.607 7fb345ac3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3400cffa0 0x7fb3400d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.607 7fb345ac3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3400d7800 0x7fb3400d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.607 7fb345ac3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3400d8220 0x7fb3400d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.607 7fb345ac3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3400d91e0 con 0x7fb3400d8220 2021-04-09T18:31:17.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.607 7fb345ac3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3400d7280 con 0x7fb3400cffa0 2021-04-09T18:31:17.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.607 7fb345ac3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3400d6290 con 0x7fb3400d7800 2021-04-09T18:31:17.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3400d8220 0x7fb3400d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33f7fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3400cffa0 0x7fb3400d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33effd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3400d7800 0x7fb3400d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3400d8220 0x7fb3400d5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56636/0 (socket says 172.21.15.32:56636) 2021-04-09T18:31:17.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33f7fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3400cffa0 0x7fb3400d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:48392/0 (socket says 172.21.15.32:48392) 2021-04-09T18:31:17.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33ffff700 1 -- 172.21.15.32:0/472049508 learned_addr learned my addr 172.21.15.32:0/472049508 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:17.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33ffff700 1 -- 172.21.15.32:0/472049508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3400d7800 msgr2=0x7fb3400d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:17.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33ffff700 1 --2- 172.21.15.32:0/472049508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3400d7800 0x7fb3400d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33ffff700 1 -- 172.21.15.32:0/472049508 >> v2:172.21.15.86:3300/0 conn(0x7fb3400cffa0 msgr2=0x7fb3400d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:17.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33ffff700 1 --2- 172.21.15.32:0/472049508 >> v2:172.21.15.86:3300/0 conn(0x7fb3400cffa0 0x7fb3400d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33ffff700 1 -- 172.21.15.32:0/472049508 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3400d64b0 con 0x7fb3400d8220 2021-04-09T18:31:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33f7fe700 1 --2- 172.21.15.32:0/472049508 >> v2:172.21.15.86:3300/0 conn(0x7fb3400cffa0 0x7fb3400d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33effd700 1 --2- 172.21.15.32:0/472049508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3400d7800 0x7fb3400d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33ffff700 1 --2- 172.21.15.32:0/472049508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3400d8220 0x7fb3400d5c70 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 rx=0x7fb338006f00 tx=0x7fb338009f10).ready entity=mon.0 client_cookie=5706995e886510ae server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:17.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33e7fc700 1 -- 172.21.15.32:0/472049508 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb33800b3b0 con 0x7fb3400d8220 2021-04-09T18:31:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33e7fc700 1 -- 172.21.15.32:0/472049508 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33800b730 con 0x7fb3400d8220 2021-04-09T18:31:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb33e7fc700 1 -- 172.21.15.32:0/472049508 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb33800bc10 con 0x7fb3400d8220 2021-04-09T18:31:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb345ac3700 1 -- 172.21.15.32:0/472049508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3400d8220 msgr2=0x7fb3400d5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.608 7fb345ac3700 1 --2- 172.21.15.32:0/472049508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3400d8220 0x7fb3400d5c70 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 rx=0x7fb338006f00 tx=0x7fb338009f10).stop 2021-04-09T18:31:17.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.609 7fb345ac3700 1 -- 172.21.15.32:0/472049508 shutdown_connections 2021-04-09T18:31:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.609 7fb345ac3700 1 --2- 172.21.15.32:0/472049508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3400d8220 0x7fb3400d5c70 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.609 7fb345ac3700 1 --2- 172.21.15.32:0/472049508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3400d7800 0x7fb3400d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.609 7fb345ac3700 1 --2- 172.21.15.32:0/472049508 >> v2:172.21.15.86:3300/0 conn(0x7fb3400cffa0 0x7fb3400d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.609 7fb345ac3700 1 -- 172.21.15.32:0/472049508 >> 172.21.15.32:0/472049508 conn(0x7fb3400ce5c0 msgr2=0x7fb3400cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:17.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.609 7fb345ac3700 1 -- 172.21.15.32:0/472049508 shutdown_connections 2021-04-09T18:31:17.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.609 7fb345ac3700 1 -- 172.21.15.32:0/472049508 wait complete. 2021-04-09T18:31:17.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb345ac3700 1 Processor -- start 2021-04-09T18:31:17.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb345ac3700 1 -- start start 2021-04-09T18:31:17.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb345ac3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3400b72f0 0x7fb3400b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb345ac3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34019c860 0x7fb34019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb345ac3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34019fdd0 0x7fb34019da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb345ac3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34019f230 con 0x7fb34019fdd0 2021-04-09T18:31:17.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb345ac3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34008a6a0 con 0x7fb3400b72f0 2021-04-09T18:31:17.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb345ac3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34018dce0 con 0x7fb34019c860 2021-04-09T18:31:17.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb33ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34019fdd0 0x7fb34019da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.630 7fb33ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34019fdd0 0x7fb34019da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56640/0 (socket says 172.21.15.32:56640) 2021-04-09T18:31:17.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb33ffff700 1 -- 172.21.15.32:0/3927192672 learned_addr learned my addr 172.21.15.32:0/3927192672 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:17.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb33f7fe700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3400b72f0 0x7fb3400b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb33effd700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34019c860 0x7fb34019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb33ffff700 1 -- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34019c860 msgr2=0x7fb34019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb33ffff700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34019c860 0x7fb34019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb33ffff700 1 -- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3400b72f0 msgr2=0x7fb3400b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb33ffff700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3400b72f0 0x7fb3400b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb33ffff700 1 -- 172.21.15.32:0/3927192672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34018e150 con 0x7fb34019fdd0 2021-04-09T18:31:17.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb33ffff700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34019fdd0 0x7fb34019da40 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 rx=0x7fb33800b8c0 tx=0x7fb33800bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:17.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb327fff700 1 -- 172.21.15.32:0/3927192672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb338000f90 con 0x7fb34019fdd0 2021-04-09T18:31:17.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb327fff700 1 -- 172.21.15.32:0/3927192672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33800bda0 con 0x7fb34019fdd0 2021-04-09T18:31:17.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34018e150 con 0x7fb34019fdd0 2021-04-09T18:31:17.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb327fff700 1 -- 172.21.15.32:0/3927192672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb338001460 con 0x7fb34019fdd0 2021-04-09T18:31:17.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.631 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34018e150 con 0x7fb34019fdd0 2021-04-09T18:31:17.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.632 7fb327fff700 1 -- 172.21.15.32:0/3927192672 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb338005820 con 0x7fb34019fdd0 2021-04-09T18:31:17.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.632 7fb327fff700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb320018930 0x7fb32001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:17.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.632 7fb327fff700 1 -- 172.21.15.32:0/3927192672 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5187+0+0 (secure 0 0 0) 0x7fb338032c20 con 0x7fb34019fdd0 2021-04-09T18:31:17.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.632 7fb33f7fe700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb320018930 0x7fb32001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:17.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.633 7fb33f7fe700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb320018930 0x7fb32001ae10 crc :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:17.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.651 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb340182e10 con 0x7fb34019fdd0 2021-04-09T18:31:17.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.655 7fb327fff700 1 -- 172.21.15.32:0/3927192672 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb338000cc0 con 0x7fb34019fdd0 2021-04-09T18:31:17.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.846 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb340182e10 con 0x7fb34019fdd0 2021-04-09T18:31:17.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.847 7fb327fff700 1 -- 172.21.15.32:0/3927192672 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+1519 (secure 0 0 0) 0x7fb33800bda0 con 0x7fb34019fdd0 2021-04-09T18:31:17.863 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 343 2021-04-09T18:31:17.863 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:17.864 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":343,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11382,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1341967017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1341967017},{"type":"v1","addr":"172.21.15.32:6833","nonce":1341967017}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":343},{"gid":58745,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/3922875743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3922875743},{"type":"v1","addr":"172.21.15.86:6837","nonce":3922875743}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":330},{"gid":58907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/2312178739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2312178739},{"type":"v1","addr":"172.21.15.86:6836","nonce":2312178739}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":342}],"filesystems":[]} 2021-04-09T18:31:17.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb320018930 msgr2=0x7fb32001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb320018930 0x7fb32001ae10 crc :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34019fdd0 msgr2=0x7fb34019da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:17.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34019fdd0 0x7fb34019da40 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 rx=0x7fb33800b8c0 tx=0x7fb33800bc60).stop 2021-04-09T18:31:17.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 shutdown_connections 2021-04-09T18:31:17.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb320018930 0x7fb32001ae10 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34019fdd0 0x7fb34019da40 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34019c860 0x7fb34019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 --2- 172.21.15.32:0/3927192672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3400b72f0 0x7fb3400b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:17.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 >> 172.21.15.32:0/3927192672 conn(0x7fb340097550 msgr2=0x7fb34018b920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:17.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 shutdown_connections 2021-04-09T18:31:17.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:17.849 7fb345ac3700 1 -- 172.21.15.32:0/3927192672 wait complete. 2021-04-09T18:31:17.876 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:31:17.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:31:17.877 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:31:17.878 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:31:17.878 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:31:17.878 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:31:17.879 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:31:17.879 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:31:17.880 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:31:17.880 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:31:17.881 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:31:17.881 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:31:17.882 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:31:17.883 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:31:17.886 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:31:17.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:31:17.888 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:31:17.888 DEBUG:teuthology.parallel:result is None 2021-04-09T18:31:17.889 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:31:17.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:31:17.891 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:31:17.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:31:17.929 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:31:17.929 DEBUG:teuthology.parallel:result is None 2021-04-09T18:31:17.932 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:31:17.932 DEBUG:teuthology.parallel:result is None 2021-04-09T18:31:17.934 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:31:17.934 DEBUG:teuthology.parallel:result is None 2021-04-09T18:31:17.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:31:18.038 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:31:18.073 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:31:18.074 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:31:18.077 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:31:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.074 7fbddf8a4700 1 Processor -- start 2021-04-09T18:31:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.075 7fbddf8a4700 1 -- start start 2021-04-09T18:31:18.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.075 7fbddf8a4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80cffb0 0x7fbdd80d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.075 7fbddf8a4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbdd80d7810 0x7fbdd80d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.075 7fbddf8a4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbdd80d5c80 0x7fbdd80d6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.075 7fbddf8a4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd80d8590 con 0x7fbdd80d7810 2021-04-09T18:31:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.075 7fbddf8a4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd80d91f0 con 0x7fbdd80d5c80 2021-04-09T18:31:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.075 7fbddf8a4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd80d7290 con 0x7fbdd80cffb0 2021-04-09T18:31:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.075 7fbddcebf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbdd80d7810 0x7fbdd80d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.075 7fbddcebf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbdd80d7810 0x7fbdd80d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56672/0 (socket says 172.21.15.32:56672) 2021-04-09T18:31:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddcebf700 1 -- 172.21.15.32:0/3006978282 learned_addr learned my addr 172.21.15.32:0/3006978282 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddd6c0700 1 --2- 172.21.15.32:0/3006978282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80cffb0 0x7fbdd80d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbdddec1700 1 --2- 172.21.15.32:0/3006978282 >> v2:172.21.15.86:3300/0 conn(0x7fbdd80d5c80 0x7fbdd80d6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddcebf700 1 -- 172.21.15.32:0/3006978282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80cffb0 msgr2=0x7fbdd80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddcebf700 1 --2- 172.21.15.32:0/3006978282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80cffb0 0x7fbdd80d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddcebf700 1 -- 172.21.15.32:0/3006978282 >> v2:172.21.15.86:3300/0 conn(0x7fbdd80d5c80 msgr2=0x7fbdd80d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddcebf700 1 --2- 172.21.15.32:0/3006978282 >> v2:172.21.15.86:3300/0 conn(0x7fbdd80d5c80 0x7fbdd80d6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddcebf700 1 -- 172.21.15.32:0/3006978282 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd80d8230 con 0x7fbdd80d7810 2021-04-09T18:31:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbdddec1700 1 --2- 172.21.15.32:0/3006978282 >> v2:172.21.15.86:3300/0 conn(0x7fbdd80d5c80 0x7fbdd80d6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddcebf700 1 --2- 172.21.15.32:0/3006978282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd80d7810 0x7fbdd80d7c10 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 rx=0x7fbdd4006a80 tx=0x7fbdd4009e80).ready entity=mon.0 client_cookie=ed7848d7320ed8de server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbdcffff700 1 -- 172.21.15.32:0/3006978282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdd400b510 con 0x7fbdd80d7810 2021-04-09T18:31:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbdcffff700 1 -- 172.21.15.32:0/3006978282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd400b7d0 con 0x7fbdd80d7810 2021-04-09T18:31:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbdcffff700 1 -- 172.21.15.32:0/3006978282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdd400bcb0 con 0x7fbdd80d7810 2021-04-09T18:31:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddf8a4700 1 -- 172.21.15.32:0/3006978282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd80d7810 msgr2=0x7fbdd80d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.076 7fbddf8a4700 1 --2- 172.21.15.32:0/3006978282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd80d7810 0x7fbdd80d7c10 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 rx=0x7fbdd4006a80 tx=0x7fbdd4009e80).stop 2021-04-09T18:31:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.077 7fbddf8a4700 1 -- 172.21.15.32:0/3006978282 shutdown_connections 2021-04-09T18:31:18.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.077 7fbddf8a4700 1 --2- 172.21.15.32:0/3006978282 >> v2:172.21.15.86:3300/0 conn(0x7fbdd80d5c80 0x7fbdd80d6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.077 7fbddf8a4700 1 --2- 172.21.15.32:0/3006978282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd80d7810 0x7fbdd80d7c10 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.077 7fbddf8a4700 1 --2- 172.21.15.32:0/3006978282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80cffb0 0x7fbdd80d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.077 7fbddf8a4700 1 -- 172.21.15.32:0/3006978282 >> 172.21.15.32:0/3006978282 conn(0x7fbdd80ce5d0 msgr2=0x7fbdd80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:18.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.077 7fbddf8a4700 1 -- 172.21.15.32:0/3006978282 shutdown_connections 2021-04-09T18:31:18.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.077 7fbddf8a4700 1 -- 172.21.15.32:0/3006978282 wait complete. 2021-04-09T18:31:18.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.097 7fbddf8a4700 1 Processor -- start 2021-04-09T18:31:18.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.097 7fbddf8a4700 1 -- start start 2021-04-09T18:31:18.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddf8a4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd81b8670 0x7fbdd81b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddf8a4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd808c720 0x7fbdd808cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddf8a4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80ab9c0 0x7fbdd80abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddf8a4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd808c1a0 con 0x7fbdd808c720 2021-04-09T18:31:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddf8a4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd81819c0 con 0x7fbdd81b8670 2021-04-09T18:31:18.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddf8a4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd808d140 con 0x7fbdd80ab9c0 2021-04-09T18:31:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddcebf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd808c720 0x7fbdd808cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddcebf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd808c720 0x7fbdd808cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56680/0 (socket says 172.21.15.32:56680) 2021-04-09T18:31:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddcebf700 1 -- 172.21.15.32:0/199430029 learned_addr learned my addr 172.21.15.32:0/199430029 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbdddec1700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80ab9c0 0x7fbdd80abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.098 7fbddd6c0700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd81b8670 0x7fbdd81b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbddcebf700 1 -- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80ab9c0 msgr2=0x7fbdd80abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbddcebf700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80ab9c0 0x7fbdd80abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbddcebf700 1 -- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd81b8670 msgr2=0x7fbdd81b8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:18.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbddcebf700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd81b8670 0x7fbdd81b8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbddcebf700 1 -- 172.21.15.32:0/199430029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd80ac3e0 con 0x7fbdd808c720 2021-04-09T18:31:18.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbddcebf700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd808c720 0x7fbdd808cb20 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 rx=0x7fbdd40099b0 tx=0x7fbdd400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:18.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbdcdffb700 1 -- 172.21.15.32:0/199430029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdd4006030 con 0x7fbdd808c720 2021-04-09T18:31:18.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbdcdffb700 1 -- 172.21.15.32:0/199430029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd4006330 con 0x7fbdd808c720 2021-04-09T18:31:18.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd80ac3e0 con 0x7fbdd808c720 2021-04-09T18:31:18.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbdcdffb700 1 -- 172.21.15.32:0/199430029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdd4005660 con 0x7fbdd808c720 2021-04-09T18:31:18.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.099 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd80ac3e0 con 0x7fbdd808c720 2021-04-09T18:31:18.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.100 7fbdcdffb700 1 -- 172.21.15.32:0/199430029 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbdd4006030 con 0x7fbdd808c720 2021-04-09T18:31:18.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.100 7fbdcdffb700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb8018930 0x7fbdb801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.100 7fbdcdffb700 1 -- 172.21.15.32:0/199430029 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5187+0+0 (secure 0 0 0) 0x7fbdd4032bf0 con 0x7fbdd808c720 2021-04-09T18:31:18.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.100 7fbddd6c0700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb8018930 0x7fbdb801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.100 7fbddd6c0700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb8018930 0x7fbdb801adc0 crc :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:18.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.119 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd8181950 con 0x7fbdd808c720 2021-04-09T18:31:18.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.122 7fbdcdffb700 1 -- 172.21.15.32:0/199430029 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbdd400be10 con 0x7fbdd808c720 2021-04-09T18:31:18.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.325 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fbdd8181950 con 0x7fbdd808c720 2021-04-09T18:31:18.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.623 7fbdcdffb700 1 -- 172.21.15.32:0/199430029 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v196) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbdd4006030 con 0x7fbdd808c720 2021-04-09T18:31:18.655 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:31:18.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.641 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb8018930 msgr2=0x7fbdb801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:18.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.641 7fbddf8a4700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb8018930 0x7fbdb801adc0 crc :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.641 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd808c720 msgr2=0x7fbdd808cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:18.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.641 7fbddf8a4700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd808c720 0x7fbdd808cb20 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 rx=0x7fbdd40099b0 tx=0x7fbdd400bcd0).stop 2021-04-09T18:31:18.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.642 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 shutdown_connections 2021-04-09T18:31:18.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.642 7fbddf8a4700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbdb8018930 0x7fbdb801adc0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.642 7fbddf8a4700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbdd808c720 0x7fbdd808cb20 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.642 7fbddf8a4700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80ab9c0 0x7fbdd80abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.642 7fbddf8a4700 1 --2- 172.21.15.32:0/199430029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd81b8670 0x7fbdd81b8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.642 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 >> 172.21.15.32:0/199430029 conn(0x7fbdd8189800 msgr2=0x7fbdd81b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:18.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.642 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 shutdown_connections 2021-04-09T18:31:18.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.642 7fbddf8a4700 1 -- 172.21.15.32:0/199430029 wait complete. 2021-04-09T18:31:18.668 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:31:18.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:31:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.801 7f892ea6d700 1 Processor -- start 2021-04-09T18:31:18.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.801 7f892ea6d700 1 -- start start 2021-04-09T18:31:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.802 7f892ea6d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f89280cffa0 0x7f89280d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.802 7f892ea6d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89280d7800 0x7f89280d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.802 7f892ea6d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89280d7280 con 0x7f89280d8220 2021-04-09T18:31:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.802 7f892ea6d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280d91e0 con 0x7f89280cffa0 2021-04-09T18:31:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.802 7f892ea6d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892810dfe0 con 0x7f89280d7800 2021-04-09T18:31:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.802 7f892d08a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f89280d8220 0x7f89280d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38044/0 (socket says 172.21.15.32:38044) 2021-04-09T18:31:18.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.802 7f892d08a700 1 -- 172.21.15.32:0/3236475709 learned_addr learned my addr 172.21.15.32:0/3236475709 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:18.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.802 7f892c889700 1 --2- 172.21.15.32:0/3236475709 >> v2:172.21.15.86:3300/0 conn(0x7f89280cffa0 0x7f89280d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.802 7f8927fff700 1 --2- 172.21.15.32:0/3236475709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89280d7800 0x7f89280d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f89277fe700 1 -- 172.21.15.32:0/3236475709 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755007998 0 0) 0x7f8920000d30 con 0x7f89280d8220 2021-04-09T18:31:18.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f89277fe700 1 -- 172.21.15.32:0/3236475709 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910002fe0 con 0x7f89280d8220 2021-04-09T18:31:18.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f8927fff700 1 -- 172.21.15.32:0/3236475709 >> v2:172.21.15.86:3300/0 conn(0x7f89280cffa0 msgr2=0x7f89280d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f8927fff700 1 --2- 172.21.15.32:0/3236475709 >> v2:172.21.15.86:3300/0 conn(0x7f89280cffa0 0x7f89280d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f8927fff700 1 -- 172.21.15.32:0/3236475709 >> v1:172.21.15.32:6789/0 conn(0x7f89280d8220 legacy=0x7f89280d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:18.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f8927fff700 1 -- 172.21.15.32:0/3236475709 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892810e200 con 0x7f89280d7800 2021-04-09T18:31:18.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f892c889700 1 --2- 172.21.15.32:0/3236475709 >> v2:172.21.15.86:3300/0 conn(0x7f89280cffa0 0x7f89280d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:18.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f8927fff700 1 --2- 172.21.15.32:0/3236475709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89280d7800 0x7f89280d7c00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f891c006650 tx=0x7f891c009a80).ready entity=mon.2 client_cookie=d864a1920bc85c0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f89277fe700 1 -- 172.21.15.32:0/3236475709 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f891c0009b0 con 0x7f89280d7800 2021-04-09T18:31:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f89277fe700 1 -- 172.21.15.32:0/3236475709 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c00c0c0 con 0x7f89280d7800 2021-04-09T18:31:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f89277fe700 1 -- 172.21.15.32:0/3236475709 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f891c00c510 con 0x7f89280d7800 2021-04-09T18:31:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f892ea6d700 1 -- 172.21.15.32:0/3236475709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89280d7800 msgr2=0x7f89280d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.803 7f892ea6d700 1 --2- 172.21.15.32:0/3236475709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89280d7800 0x7f89280d7c00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f891c006650 tx=0x7f891c009a80).stop 2021-04-09T18:31:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.804 7f892ea6d700 1 -- 172.21.15.32:0/3236475709 shutdown_connections 2021-04-09T18:31:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.804 7f892ea6d700 1 --2- 172.21.15.32:0/3236475709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89280d7800 0x7f89280d7c00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.804 7f892ea6d700 1 --2- 172.21.15.32:0/3236475709 >> v2:172.21.15.86:3300/0 conn(0x7f89280cffa0 0x7f89280d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.804 7f892ea6d700 1 -- 172.21.15.32:0/3236475709 >> 172.21.15.32:0/3236475709 conn(0x7f89280ce5c0 msgr2=0x7f89280cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.804 7f892ea6d700 1 -- 172.21.15.32:0/3236475709 shutdown_connections 2021-04-09T18:31:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.804 7f892ea6d700 1 -- 172.21.15.32:0/3236475709 wait complete. 2021-04-09T18:31:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.824 7f892ea6d700 1 Processor -- start 2021-04-09T18:31:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.824 7f892ea6d700 1 -- start start 2021-04-09T18:31:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892ea6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8928190320 0x7f892818fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892ea6d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892808ab30 0x7f892808af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892ea6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89280aa9b0 0x7f89280aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892ea6d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892808a650 con 0x7f89280aa9b0 2021-04-09T18:31:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892ea6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892808b550 con 0x7f8928190320 2021-04-09T18:31:18.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892ea6d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89280ab3d0 con 0x7f892808ab30 2021-04-09T18:31:18.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892d08a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89280aa9b0 0x7f89280aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892d08a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89280aa9b0 0x7f89280aadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56694/0 (socket says 172.21.15.32:56694) 2021-04-09T18:31:18.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892d08a700 1 -- 172.21.15.32:0/3834672313 learned_addr learned my addr 172.21.15.32:0/3834672313 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f8927fff700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892808ab30 0x7f892808af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892c889700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8928190320 0x7f892818fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892d08a700 1 -- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892808ab30 msgr2=0x7f892808af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:18.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892d08a700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892808ab30 0x7f892808af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892d08a700 1 -- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8928190320 msgr2=0x7f892818fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:18.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892d08a700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8928190320 0x7f892818fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:18.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892d08a700 1 -- 172.21.15.32:0/3834672313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89280ab5f0 con 0x7f89280aa9b0 2021-04-09T18:31:18.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892c889700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8928190320 0x7f892818fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:18.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892d08a700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89280aa9b0 0x7f89280aadb0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 rx=0x7f8920009980 tx=0x7f892000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:18.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f89257fa700 1 -- 172.21.15.32:0/3834672313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892000be10 con 0x7f89280aa9b0 2021-04-09T18:31:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f89257fa700 1 -- 172.21.15.32:0/3834672313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8920006080 con 0x7f89280aa9b0 2021-04-09T18:31:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89280ab5f0 con 0x7f89280aa9b0 2021-04-09T18:31:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f89257fa700 1 -- 172.21.15.32:0/3834672313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892000be10 con 0x7f89280aa9b0 2021-04-09T18:31:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.825 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89280ab5f0 con 0x7f89280aa9b0 2021-04-09T18:31:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.826 7f89257fa700 1 -- 172.21.15.32:0/3834672313 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8920006260 con 0x7f89280aa9b0 2021-04-09T18:31:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.826 7f89257fa700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8908018930 0x7f890801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.826 7f89257fa700 1 -- 172.21.15.32:0/3834672313 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f8920032a30 con 0x7f89280aa9b0 2021-04-09T18:31:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.826 7f892c889700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8908018930 0x7f890801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.827 7f892c889700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8908018930 0x7f890801ae10 crc :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.846 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892808a650 con 0x7f89280aa9b0 2021-04-09T18:31:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:18.850 7f89257fa700 1 -- 172.21.15.32:0/3834672313 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8920000f90 con 0x7f89280aa9b0 2021-04-09T18:31:19.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.035 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f892808a650 con 0x7f89280aa9b0 2021-04-09T18:31:19.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.036 7f89257fa700 1 -- 172.21.15.32:0/3834672313 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+4589 (secure 0 0 0) 0x7f8920006080 con 0x7f89280aa9b0 2021-04-09T18:31:19.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:19.052 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:31:19.053 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:31:19.053 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.053 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:31:19.053 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:31:19.054 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.054 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:31:19.054 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.054 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.054 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.055 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.055 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:31:19.056 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:31:19.056 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.056 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:31:19.056 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.056 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.057 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.057 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.057 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:31:19.058 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:31:19.058 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.058 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:31:19.058 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.059 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.059 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.059 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.060 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:31:19.060 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:31:19.060 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.060 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:31:19.060 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.061 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.061 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.061 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.062 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:31:19.062 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:31:19.062 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.062 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.063 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.063 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.064 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:31:19.064 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:31:19.064 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.064 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.064 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.065 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.065 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.066 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:31:19.066 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:31:19.066 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.066 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.066 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.067 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.067 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.067 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:31:19.068 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:31:19.068 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.068 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.068 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.069 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.069 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.069 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:31:19.070 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:31:19.070 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.070 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.070 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.070 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.071 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.071 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:31:19.071 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:31:19.072 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.072 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.072 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.073 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.073 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:31:19.073 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:31:19.074 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.074 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.074 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.074 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.074 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.075 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.075 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:31:19.075 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:31:19.075 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.076 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.076 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.076 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.076 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.077 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:31:19.077 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:31:19.078 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.078 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:31:19.078 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:31:19.078 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=alpha_data, allow rw pool=bravo_data" 2021-04-09T18:31:19.078 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.079 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:31:19.079 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:31:19.079 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.080 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:31:19.080 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:31:19.080 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=alpha_data, allow rw pool=bravo_data" 2021-04-09T18:31:19.080 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:31:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:31:19.081 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.082 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:31:19.082 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:31:19.082 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:31:19.082 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.083 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:31:19.083 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:31:19.083 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.084 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:31:19.084 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:31:19.084 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.084 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:19.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:19.085 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:31:19.085 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:31:19.085 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:31:19.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:31:19.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:31:19.086 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:31:19.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:19.087 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:31:19.087 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:31:19.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8908018930 msgr2=0x7f890801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8908018930 0x7f890801ae10 crc :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89280aa9b0 msgr2=0x7f89280aadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89280aa9b0 0x7f89280aadb0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 rx=0x7f8920009980 tx=0x7f892000bcd0).stop 2021-04-09T18:31:19.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 shutdown_connections 2021-04-09T18:31:19.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8908018930 0x7f890801ae10 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89280aa9b0 0x7f89280aadb0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f892808ab30 0x7f892808af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 --2- 172.21.15.32:0/3834672313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8928190320 0x7f892818fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.038 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 >> 172.21.15.32:0/3834672313 conn(0x7f89281941b0 msgr2=0x7f8928183120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:19.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.039 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 shutdown_connections 2021-04-09T18:31:19.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.039 7f892ea6d700 1 -- 172.21.15.32:0/3834672313 wait complete. 2021-04-09T18:31:19.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-04-09T18:31:19.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.235 7f5801ee8700 1 Processor -- start 2021-04-09T18:31:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.236 7f5801ee8700 1 -- start start 2021-04-09T18:31:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.236 7f5801ee8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc10d050 0x7f57fc0cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.236 7f5801ee8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57fc0d84b0 0x7f57fc10da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.236 7f5801ee8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57fc10d500 con 0x7f57fc0d0160 2021-04-09T18:31:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.236 7f5801ee8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57fc10e800 con 0x7f57fc0d84b0 2021-04-09T18:31:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.236 7f5801ee8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0d8d70 con 0x7f57fc10d050 2021-04-09T18:31:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.236 7f57fbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57fc0d84b0 0x7f57fc10da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.236 7f57fbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57fc0d84b0 0x7f57fc10da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56700/0 (socket says 172.21.15.32:56700) 2021-04-09T18:31:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.236 7f57fbfff700 1 -- 172.21.15.32:0/3534921567 learned_addr learned my addr 172.21.15.32:0/3534921567 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f57fb7fe700 1 --2- 172.21.15.32:0/3534921567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc10d050 0x7f57fc0cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f57fbfff700 1 -- 172.21.15.32:0/3534921567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc10d050 msgr2=0x7f57fc0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f57fbfff700 1 --2- 172.21.15.32:0/3534921567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc10d050 0x7f57fc0cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f57fbfff700 1 -- 172.21.15.32:0/3534921567 >> v1:172.21.15.86:6789/0 conn(0x7f57fc0d0160 legacy=0x7f57fc0d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f57fbfff700 1 -- 172.21.15.32:0/3534921567 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc0d9260 con 0x7f57fc0d84b0 2021-04-09T18:31:19.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f57fbfff700 1 --2- 172.21.15.32:0/3534921567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0d84b0 0x7f57fc10da80 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 rx=0x7f57f4006f00 tx=0x7f57f4009e90).ready entity=mon.0 client_cookie=42d04c726755ba15 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f57fa7fc700 1 -- 172.21.15.32:0/3534921567 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57f400b300 con 0x7f57fc0d84b0 2021-04-09T18:31:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f57fa7fc700 1 -- 172.21.15.32:0/3534921567 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f400b640 con 0x7f57fc0d84b0 2021-04-09T18:31:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f57fa7fc700 1 -- 172.21.15.32:0/3534921567 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57f400bb20 con 0x7f57fc0d84b0 2021-04-09T18:31:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f5801ee8700 1 -- 172.21.15.32:0/3534921567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0d84b0 msgr2=0x7f57fc10da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.237 7f5801ee8700 1 --2- 172.21.15.32:0/3534921567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0d84b0 0x7f57fc10da80 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 rx=0x7f57f4006f00 tx=0x7f57f4009e90).stop 2021-04-09T18:31:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.238 7f5801ee8700 1 -- 172.21.15.32:0/3534921567 shutdown_connections 2021-04-09T18:31:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.238 7f5801ee8700 1 --2- 172.21.15.32:0/3534921567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0d84b0 0x7f57fc10da80 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.238 7f5801ee8700 1 --2- 172.21.15.32:0/3534921567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc10d050 0x7f57fc0cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.238 7f5801ee8700 1 -- 172.21.15.32:0/3534921567 >> 172.21.15.32:0/3534921567 conn(0x7f57fc0ce620 msgr2=0x7f57fc0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.238 7f5801ee8700 1 -- 172.21.15.32:0/3534921567 shutdown_connections 2021-04-09T18:31:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.238 7f5801ee8700 1 -- 172.21.15.32:0/3534921567 wait complete. 2021-04-09T18:31:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.258 7f5801ee8700 1 Processor -- start 2021-04-09T18:31:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.258 7f5801ee8700 1 -- start start 2021-04-09T18:31:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f5801ee8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57fc09a890 0x7f57fc0a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f5801ee8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc09ac90 0x7f57fc0a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f5801ee8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0c2220 0x7f57fc0c2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f5801ee8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0b86e0 con 0x7f57fc0c2220 2021-04-09T18:31:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f5801ee8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0a2920 con 0x7f57fc09a890 2021-04-09T18:31:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f5801ee8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0c2c40 con 0x7f57fc09ac90 2021-04-09T18:31:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0c2220 0x7f57fc0c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0c2220 0x7f57fc0c2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56706/0 (socket says 172.21.15.32:56706) 2021-04-09T18:31:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fbfff700 1 -- 172.21.15.32:0/548785563 learned_addr learned my addr 172.21.15.32:0/548785563 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fb7fe700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57fc09a890 0x7f57fc0a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57faffd700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc09ac90 0x7f57fc0a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fbfff700 1 -- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc09ac90 msgr2=0x7f57fc0a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fbfff700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc09ac90 0x7f57fc0a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fbfff700 1 -- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57fc09a890 msgr2=0x7f57fc0a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fbfff700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57fc09a890 0x7f57fc0a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fbfff700 1 -- 172.21.15.32:0/548785563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc0c2e20 con 0x7f57fc0c2220 2021-04-09T18:31:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57fbfff700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0c2220 0x7f57fc0c2620 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 rx=0x7f57f400b890 tx=0x7f57f400bc30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:19.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57e3fff700 1 -- 172.21.15.32:0/548785563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57f400bd70 con 0x7f57fc0c2220 2021-04-09T18:31:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57e3fff700 1 -- 172.21.15.32:0/548785563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f4005820 con 0x7f57fc0c2220 2021-04-09T18:31:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f5801ee8700 1 -- 172.21.15.32:0/548785563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc0c30c0 con 0x7f57fc0c2220 2021-04-09T18:31:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f57e3fff700 1 -- 172.21.15.32:0/548785563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57f4005c70 con 0x7f57fc0c2220 2021-04-09T18:31:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.259 7f5801ee8700 1 -- 172.21.15.32:0/548785563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc0c2e20 con 0x7f57fc0c2220 2021-04-09T18:31:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.260 7f57e3fff700 1 -- 172.21.15.32:0/548785563 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f57f4005a00 con 0x7f57fc0c2220 2021-04-09T18:31:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.260 7f57e3fff700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57dc018930 0x7f57dc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.260 7f57e3fff700 1 -- 172.21.15.32:0/548785563 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f57f4032a70 con 0x7f57fc0c2220 2021-04-09T18:31:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.260 7f57fb7fe700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57dc018930 0x7f57dc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.261 7f57fb7fe700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57dc018930 0x7f57dc01ae10 crc :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:19.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.280 7f5801ee8700 1 -- 172.21.15.32:0/548785563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57fc1b7180 con 0x7f57fc0c2220 2021-04-09T18:31:19.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.284 7f57e3fff700 1 -- 172.21.15.32:0/548785563 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f57f400bd70 con 0x7f57fc0c2220 2021-04-09T18:31:19.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.478 7f5801ee8700 1 -- 172.21.15.32:0/548785563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f57fc10d500 con 0x7f57fc0c2220 2021-04-09T18:31:19.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.626 7f57e3fff700 1 -- 172.21.15.32:0/548785563 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v344) v1 ==== 136+0+0 (secure 0 0 0) 0x7f57f4005820 con 0x7f57fc0c2220 2021-04-09T18:31:19.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 -- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57dc018930 msgr2=0x7f57dc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57dc018930 0x7f57dc01ae10 crc :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 -- 172.21.15.32:0/548785563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0c2220 msgr2=0x7f57fc0c2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0c2220 0x7f57fc0c2620 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 rx=0x7f57f400b890 tx=0x7f57f400bc30).stop 2021-04-09T18:31:19.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 -- 172.21.15.32:0/548785563 shutdown_connections 2021-04-09T18:31:19.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57dc018930 0x7f57dc01ae10 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57fc0c2220 0x7f57fc0c2620 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57fc09ac90 0x7f57fc0a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 --2- 172.21.15.32:0/548785563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57fc09a890 0x7f57fc0a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.644 7f5801ee8700 1 -- 172.21.15.32:0/548785563 >> 172.21.15.32:0/548785563 conn(0x7f57fc0b6b70 msgr2=0x7f57fc0b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.645 7f5801ee8700 1 -- 172.21.15.32:0/548785563 shutdown_connections 2021-04-09T18:31:19.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.645 7f5801ee8700 1 -- 172.21.15.32:0/548785563 wait complete. 2021-04-09T18:31:19.671 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:31:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.802 7f195582b700 1 Processor -- start 2021-04-09T18:31:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.802 7f195582b700 1 -- start start 2021-04-09T18:31:19.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f195582b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19500d7ca0 0x7f19500d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f195582b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f195010da00 0x7f195010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f195582b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195010d480 con 0x7f195010d010 2021-04-09T18:31:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f195582b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19500d9220 con 0x7f195010da00 2021-04-09T18:31:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f195582b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19500d86c0 con 0x7f19500d7ca0 2021-04-09T18:31:19.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194fd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f195010da00 0x7f195010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194f574700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f195010d010 0x7f19500d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38070/0 (socket says 172.21.15.32:38070) 2021-04-09T18:31:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194f574700 1 -- 172.21.15.32:0/388834026 learned_addr learned my addr 172.21.15.32:0/388834026 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194fd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f195010da00 0x7f195010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56716/0 (socket says 172.21.15.32:56716) 2021-04-09T18:31:19.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194ed73700 1 --2- 172.21.15.32:0/388834026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19500d7ca0 0x7f19500d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194fd75700 1 -- 172.21.15.32:0/388834026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19500d7ca0 msgr2=0x7f19500d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194fd75700 1 --2- 172.21.15.32:0/388834026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19500d7ca0 0x7f19500d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194fd75700 1 -- 172.21.15.32:0/388834026 >> v1:172.21.15.32:6789/0 conn(0x7f195010d010 legacy=0x7f19500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194fd75700 1 -- 172.21.15.32:0/388834026 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500d1f70 con 0x7f195010da00 2021-04-09T18:31:19.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194e572700 1 -- 172.21.15.32:0/388834026 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146668398 0 0) 0x7f1940000ce0 con 0x7f195010d010 2021-04-09T18:31:19.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.803 7f194fd75700 1 --2- 172.21.15.32:0/388834026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f195010da00 0x7f195010de00 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 rx=0x7f1948006e30 tx=0x7f1948009de0).ready entity=mon.0 client_cookie=bc683524c9541a25 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:19.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f194e572700 1 -- 172.21.15.32:0/388834026 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f194800b300 con 0x7f195010da00 2021-04-09T18:31:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f194e572700 1 -- 172.21.15.32:0/388834026 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194800b680 con 0x7f195010da00 2021-04-09T18:31:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f194e572700 1 -- 172.21.15.32:0/388834026 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f194800bb60 con 0x7f195010da00 2021-04-09T18:31:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f195582b700 1 -- 172.21.15.32:0/388834026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f195010da00 msgr2=0x7f195010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f195582b700 1 --2- 172.21.15.32:0/388834026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f195010da00 0x7f195010de00 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 rx=0x7f1948006e30 tx=0x7f1948009de0).stop 2021-04-09T18:31:19.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f195582b700 1 -- 172.21.15.32:0/388834026 shutdown_connections 2021-04-09T18:31:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f195582b700 1 --2- 172.21.15.32:0/388834026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f195010da00 0x7f195010de00 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f195582b700 1 --2- 172.21.15.32:0/388834026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19500d7ca0 0x7f19500d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f195582b700 1 -- 172.21.15.32:0/388834026 >> 172.21.15.32:0/388834026 conn(0x7f19500ce5f0 msgr2=0x7f19500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:19.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f195582b700 1 -- 172.21.15.32:0/388834026 shutdown_connections 2021-04-09T18:31:19.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.804 7f195582b700 1 -- 172.21.15.32:0/388834026 wait complete. 2021-04-09T18:31:19.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.825 7f195582b700 1 Processor -- start 2021-04-09T18:31:19.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f195582b700 1 -- start start 2021-04-09T18:31:19.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f195582b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1950186fb0 0x7f1950184680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f195582b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1950183da0 0x7f1950187b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f195582b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1950188170 0x7f19501ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f195582b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19501875d0 con 0x7f1950188170 2021-04-09T18:31:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f195582b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1950188570 con 0x7f1950186fb0 2021-04-09T18:31:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f195582b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19501841a0 con 0x7f1950183da0 2021-04-09T18:31:19.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f194fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1950188170 0x7f19501ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f194fd75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1950188170 0x7f19501ac090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56722/0 (socket says 172.21.15.32:56722) 2021-04-09T18:31:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.826 7f194fd75700 1 -- 172.21.15.32:0/4166265829 learned_addr learned my addr 172.21.15.32:0/4166265829 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194ed73700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1950183da0 0x7f1950187b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194f574700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1950186fb0 0x7f1950184680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194fd75700 1 -- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1950183da0 msgr2=0x7f1950187b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194fd75700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1950183da0 0x7f1950187b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194fd75700 1 -- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1950186fb0 msgr2=0x7f1950184680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:19.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194fd75700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1950186fb0 0x7f1950184680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:19.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194fd75700 1 -- 172.21.15.32:0/4166265829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19501ac6b0 con 0x7f1950188170 2021-04-09T18:31:19.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194f574700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1950186fb0 0x7f1950184680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:19.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194ed73700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1950183da0 0x7f1950187b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:19.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f194fd75700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1950188170 0x7f19501ac090 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 rx=0x7f1948009fc0 tx=0x7f194800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:19.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f1947fff700 1 -- 172.21.15.32:0/4166265829 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19480093d0 con 0x7f1950188170 2021-04-09T18:31:19.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f1947fff700 1 -- 172.21.15.32:0/4166265829 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1948009690 con 0x7f1950188170 2021-04-09T18:31:19.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f1947fff700 1 -- 172.21.15.32:0/4166265829 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1948005aa0 con 0x7f1950188170 2021-04-09T18:31:19.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f195582b700 1 -- 172.21.15.32:0/4166265829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19501ac6b0 con 0x7f1950188170 2021-04-09T18:31:19.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.827 7f195582b700 1 -- 172.21.15.32:0/4166265829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19501ac6b0 con 0x7f1950188170 2021-04-09T18:31:19.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.828 7f1947fff700 1 -- 172.21.15.32:0/4166265829 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1948005820 con 0x7f1950188170 2021-04-09T18:31:19.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.828 7f1947fff700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1934018930 0x7f193401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:19.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.828 7f1947fff700 1 -- 172.21.15.32:0/4166265829 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f1948032ad0 con 0x7f1950188170 2021-04-09T18:31:19.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.828 7f194f574700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1934018930 0x7f193401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:19.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.828 7f194f574700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1934018930 0x7f193401ae10 crc :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:19.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.848 7f195582b700 1 -- 172.21.15.32:0/4166265829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1950182d80 con 0x7f1950188170 2021-04-09T18:31:19.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:19.851 7f1947fff700 1 -- 172.21.15.32:0/4166265829 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1948006bd0 con 0x7f1950188170 2021-04-09T18:31:20.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.039 7f195582b700 1 -- 172.21.15.32:0/4166265829 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f195010d480 con 0x7f1950188170 2021-04-09T18:31:20.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.040 7f1947fff700 1 -- 172.21.15.32:0/4166265829 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f1948009690 con 0x7f1950188170 2021-04-09T18:31:20.055 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:20.056 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":196,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:31:18.619091","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":182,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:31:20.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 -- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1934018930 msgr2=0x7f193401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1934018930 0x7f193401ae10 crc :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 -- 172.21.15.32:0/4166265829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1950188170 msgr2=0x7f19501ac090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1950188170 0x7f19501ac090 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 rx=0x7f1948009fc0 tx=0x7f194800bd40).stop 2021-04-09T18:31:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 -- 172.21.15.32:0/4166265829 shutdown_connections 2021-04-09T18:31:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1934018930 0x7f193401ae10 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1950188170 0x7f19501ac090 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1950183da0 0x7f1950187b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 --2- 172.21.15.32:0/4166265829 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1950186fb0 0x7f1950184680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 -- 172.21.15.32:0/4166265829 >> 172.21.15.32:0/4166265829 conn(0x7f195019b890 msgr2=0x7f19501a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:20.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 -- 172.21.15.32:0/4166265829 shutdown_connections 2021-04-09T18:31:20.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.042 7f195582b700 1 -- 172.21.15.32:0/4166265829 wait complete. 2021-04-09T18:31:20.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:31:20.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:31:20.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.206 7f66dd3fe700 1 Processor -- start 2021-04-09T18:31:20.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.206 7f66dd3fe700 1 -- start start 2021-04-09T18:31:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.206 7f66dd3fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66d810d010 0x7f66d80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.206 7f66dd3fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f66d80d1da0 0x7f66d80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66dd3fe700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d810d480 con 0x7f66d80d82c0 2021-04-09T18:31:20.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66dd3fe700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d80d8d30 con 0x7f66d810d010 2021-04-09T18:31:20.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66dd3fe700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d80d9220 con 0x7f66d80d1da0 2021-04-09T18:31:20.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d707d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66d810d010 0x7f66d80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d707d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66d810d010 0x7f66d80d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56726/0 (socket says 172.21.15.32:56726) 2021-04-09T18:31:20.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d707d700 1 -- 172.21.15.32:0/2962636409 learned_addr learned my addr 172.21.15.32:0/2962636409 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:20.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d687c700 1 --2- 172.21.15.32:0/2962636409 >> v2:172.21.15.86:3300/0 conn(0x7f66d80d1da0 0x7f66d80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d707d700 1 -- 172.21.15.32:0/2962636409 >> v2:172.21.15.86:3300/0 conn(0x7f66d80d1da0 msgr2=0x7f66d80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:20.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d707d700 1 --2- 172.21.15.32:0/2962636409 >> v2:172.21.15.86:3300/0 conn(0x7f66d80d1da0 0x7f66d80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d707d700 1 -- 172.21.15.32:0/2962636409 >> v1:172.21.15.86:6789/0 conn(0x7f66d80d82c0 legacy=0x7f66d810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d707d700 1 -- 172.21.15.32:0/2962636409 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d80d86f0 con 0x7f66d810d010 2021-04-09T18:31:20.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d707d700 1 --2- 172.21.15.32:0/2962636409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d810d010 0x7f66d80d1780 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 rx=0x7f66c8006700 tx=0x7f66c8009b80).ready entity=mon.0 client_cookie=d3bcd3e6d509ecb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:20.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d607b700 1 -- 172.21.15.32:0/2962636409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66c800c0c0 con 0x7f66d810d010 2021-04-09T18:31:20.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d607b700 1 -- 172.21.15.32:0/2962636409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c800c440 con 0x7f66d810d010 2021-04-09T18:31:20.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66d607b700 1 -- 172.21.15.32:0/2962636409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66c800c920 con 0x7f66d810d010 2021-04-09T18:31:20.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66dd3fe700 1 -- 172.21.15.32:0/2962636409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d810d010 msgr2=0x7f66d80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.207 7f66dd3fe700 1 --2- 172.21.15.32:0/2962636409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d810d010 0x7f66d80d1780 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 rx=0x7f66c8006700 tx=0x7f66c8009b80).stop 2021-04-09T18:31:20.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.208 7f66dd3fe700 1 -- 172.21.15.32:0/2962636409 shutdown_connections 2021-04-09T18:31:20.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.208 7f66dd3fe700 1 --2- 172.21.15.32:0/2962636409 >> v2:172.21.15.86:3300/0 conn(0x7f66d80d1da0 0x7f66d80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.208 7f66dd3fe700 1 --2- 172.21.15.32:0/2962636409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d810d010 0x7f66d80d1780 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.208 7f66dd3fe700 1 -- 172.21.15.32:0/2962636409 >> 172.21.15.32:0/2962636409 conn(0x7f66d80ce5f0 msgr2=0x7f66d80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:20.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.208 7f66dd3fe700 1 -- 172.21.15.32:0/2962636409 shutdown_connections 2021-04-09T18:31:20.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.208 7f66dd3fe700 1 -- 172.21.15.32:0/2962636409 wait complete. 2021-04-09T18:31:20.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.228 7f66dd3fe700 1 Processor -- start 2021-04-09T18:31:20.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.228 7f66dd3fe700 1 -- start start 2021-04-09T18:31:20.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66dd3fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d808a910 0x7f66d809e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66dd3fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d808ad10 0x7f66d809ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66dd3fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d80922e0 0x7f66d80926e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66dd3fe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d818fc90 con 0x7f66d808ad10 2021-04-09T18:31:20.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66dd3fe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d809e090 con 0x7f66d808a910 2021-04-09T18:31:20.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66dd3fe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d808b400 con 0x7f66d80922e0 2021-04-09T18:31:20.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d687c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d808ad10 0x7f66d809ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d687c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d808ad10 0x7f66d809ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56734/0 (socket says 172.21.15.32:56734) 2021-04-09T18:31:20.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d687c700 1 -- 172.21.15.32:0/219827759 learned_addr learned my addr 172.21.15.32:0/219827759 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:20.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d787e700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d80922e0 0x7f66d80926e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d707d700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d808a910 0x7f66d809e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d687c700 1 -- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d80922e0 msgr2=0x7f66d80926e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d687c700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d80922e0 0x7f66d80926e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d687c700 1 -- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d808a910 msgr2=0x7f66d809e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d687c700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d808a910 0x7f66d809e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d687c700 1 -- 172.21.15.32:0/219827759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d8092d00 con 0x7f66d808ad10 2021-04-09T18:31:20.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.229 7f66d787e700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d80922e0 0x7f66d80926e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:20.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.230 7f66d687c700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d808ad10 0x7f66d809ec30 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 rx=0x7f66cc006f60 tx=0x7f66cc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:20.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.230 7f66bffff700 1 -- 172.21.15.32:0/219827759 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66cc00bd60 con 0x7f66d808ad10 2021-04-09T18:31:20.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.230 7f66bffff700 1 -- 172.21.15.32:0/219827759 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66cc000c90 con 0x7f66d808ad10 2021-04-09T18:31:20.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.230 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d8092d00 con 0x7f66d808ad10 2021-04-09T18:31:20.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.230 7f66bffff700 1 -- 172.21.15.32:0/219827759 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66cc002220 con 0x7f66d808ad10 2021-04-09T18:31:20.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.230 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d8092d00 con 0x7f66d808ad10 2021-04-09T18:31:20.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.231 7f66bffff700 1 -- 172.21.15.32:0/219827759 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f66cc0052e0 con 0x7f66d808ad10 2021-04-09T18:31:20.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.231 7f66bffff700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66b8018930 0x7f66b801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.231 7f66bffff700 1 -- 172.21.15.32:0/219827759 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f66cc0326c0 con 0x7f66d808ad10 2021-04-09T18:31:20.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.231 7f66d707d700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66b8018930 0x7f66b801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.231 7f66d707d700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66b8018930 0x7f66b801adc0 crc :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:20.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.251 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d818fc90 con 0x7f66d808ad10 2021-04-09T18:31:20.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.254 7f66bffff700 1 -- 172.21.15.32:0/219827759 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f66cc000c90 con 0x7f66d808ad10 2021-04-09T18:31:20.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.435 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f66d818fc90 con 0x7f66d808ad10 2021-04-09T18:31:20.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.435 7f66bffff700 1 -- 172.21.15.32:0/219827759 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v196) v1 ==== 92+0+10 (secure 0 0 0) 0x7f66cc006b00 con 0x7f66d808ad10 2021-04-09T18:31:20.450 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:31:20.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66b8018930 msgr2=0x7f66b801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66b8018930 0x7f66b801adc0 crc :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d808ad10 msgr2=0x7f66d809ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d808ad10 0x7f66d809ec30 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 rx=0x7f66cc006f60 tx=0x7f66cc00b970).stop 2021-04-09T18:31:20.453 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 shutdown_connections 2021-04-09T18:31:20.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f66b8018930 0x7f66b801adc0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66d808ad10 0x7f66d809ec30 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d80922e0 0x7f66d80926e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 --2- 172.21.15.32:0/219827759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d808a910 0x7f66d809e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.437 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 >> 172.21.15.32:0/219827759 conn(0x7f66d80b2fa0 msgr2=0x7f66d80a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:20.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.438 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 shutdown_connections 2021-04-09T18:31:20.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.438 7f66dd3fe700 1 -- 172.21.15.32:0/219827759 wait complete. 2021-04-09T18:31:20.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:31:20.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.596 7f4f2f0da700 1 Processor -- start 2021-04-09T18:31:20.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.596 7f4f2f0da700 1 -- start start 2021-04-09T18:31:20.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2f0da700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4f2810da00 0x7f4f2810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2f0da700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2810d480 con 0x7f4f2810d010 2021-04-09T18:31:20.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2f0da700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f280d1f70 con 0x7f4f280d7ca0 2021-04-09T18:31:20.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2f0da700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f280d8d30 con 0x7f4f2810da00 2021-04-09T18:31:20.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2d6f7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f4f2810da00 0x7f4f2810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2cef6700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4f2810d010 0x7f4f280d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38098/0 (socket says 172.21.15.32:38098) 2021-04-09T18:31:20.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2cef6700 1 -- 172.21.15.32:0/4036314245 learned_addr learned my addr 172.21.15.32:0/4036314245 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:20.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f277fe700 1 -- 172.21.15.32:0/4036314245 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789540791 0 0) 0x7f4f10000ce0 con 0x7f4f2810d010 2021-04-09T18:31:20.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f277fe700 1 -- 172.21.15.32:0/4036314245 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14002fe0 con 0x7f4f2810d010 2021-04-09T18:31:20.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2d6f7700 1 -- 172.21.15.32:0/4036314245 >> v1:172.21.15.86:6789/0 conn(0x7f4f280d7ca0 legacy=0x7f4f280d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2d6f7700 1 -- 172.21.15.32:0/4036314245 >> v1:172.21.15.32:6789/0 conn(0x7f4f2810d010 legacy=0x7f4f280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.597 7f4f2d6f7700 1 -- 172.21.15.32:0/4036314245 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f280d9220 con 0x7f4f2810da00 2021-04-09T18:31:20.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f2d6f7700 1 --2- 172.21.15.32:0/4036314245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f2810da00 0x7f4f2810de00 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 rx=0x7f4f20006e30 tx=0x7f4f20009e60).ready entity=mon.0 client_cookie=f815ffb7f83ae3cc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:20.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f277fe700 1 -- 172.21.15.32:0/4036314245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f20006a50 con 0x7f4f2810da00 2021-04-09T18:31:20.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f277fe700 1 -- 172.21.15.32:0/4036314245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2000b300 con 0x7f4f2810da00 2021-04-09T18:31:20.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f277fe700 1 -- 172.21.15.32:0/4036314245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f2000b750 con 0x7f4f2810da00 2021-04-09T18:31:20.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f2f0da700 1 -- 172.21.15.32:0/4036314245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f2810da00 msgr2=0x7f4f2810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f2f0da700 1 --2- 172.21.15.32:0/4036314245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f2810da00 0x7f4f2810de00 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 rx=0x7f4f20006e30 tx=0x7f4f20009e60).stop 2021-04-09T18:31:20.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f2f0da700 1 -- 172.21.15.32:0/4036314245 shutdown_connections 2021-04-09T18:31:20.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f2f0da700 1 --2- 172.21.15.32:0/4036314245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f2810da00 0x7f4f2810de00 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f2f0da700 1 -- 172.21.15.32:0/4036314245 >> 172.21.15.32:0/4036314245 conn(0x7f4f280ce5f0 msgr2=0x7f4f280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:20.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.598 7f4f2f0da700 1 -- 172.21.15.32:0/4036314245 shutdown_connections 2021-04-09T18:31:20.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.599 7f4f2f0da700 1 -- 172.21.15.32:0/4036314245 wait complete. 2021-04-09T18:31:20.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.619 7f4f2f0da700 1 Processor -- start 2021-04-09T18:31:20.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.619 7f4f2f0da700 1 -- start start 2021-04-09T18:31:20.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.619 7f4f2f0da700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2819d870 0x7f4f28184260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.619 7f4f2f0da700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f28182850 0x7f4f28184880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.619 7f4f2f0da700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f280b0ac0 0x7f4f280b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.619 7f4f2f0da700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28184ea0 con 0x7f4f280b0ac0 2021-04-09T18:31:20.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.619 7f4f2f0da700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28183ce0 con 0x7f4f28182850 2021-04-09T18:31:20.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.619 7f4f2f0da700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f280b14e0 con 0x7f4f2819d870 2021-04-09T18:31:20.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.619 7f4f2d6f7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f280b0ac0 0x7f4f280b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2d6f7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f280b0ac0 0x7f4f280b0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56748/0 (socket says 172.21.15.32:56748) 2021-04-09T18:31:20.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2d6f7700 1 -- 172.21.15.32:0/802939129 learned_addr learned my addr 172.21.15.32:0/802939129 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:20.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2cef6700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2819d870 0x7f4f28184260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f27fff700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f28182850 0x7f4f28184880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2d6f7700 1 -- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2819d870 msgr2=0x7f4f28184260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2d6f7700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2819d870 0x7f4f28184260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2d6f7700 1 -- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f28182850 msgr2=0x7f4f28184880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2d6f7700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f28182850 0x7f4f28184880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2d6f7700 1 -- 172.21.15.32:0/802939129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f280b16c0 con 0x7f4f280b0ac0 2021-04-09T18:31:20.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2cef6700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2819d870 0x7f4f28184260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2d6f7700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f280b0ac0 0x7f4f280b0ec0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 rx=0x7f4f20001070 tx=0x7f4f2000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f257fa700 1 -- 172.21.15.32:0/802939129 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f20001270 con 0x7f4f280b0ac0 2021-04-09T18:31:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f257fa700 1 -- 172.21.15.32:0/802939129 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f20006df0 con 0x7f4f280b0ac0 2021-04-09T18:31:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f257fa700 1 -- 172.21.15.32:0/802939129 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f20001720 con 0x7f4f280b0ac0 2021-04-09T18:31:20.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f280b16c0 con 0x7f4f280b0ac0 2021-04-09T18:31:20.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.620 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f280b16c0 con 0x7f4f280b0ac0 2021-04-09T18:31:20.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.621 7f4f257fa700 1 -- 172.21.15.32:0/802939129 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4f200014f0 con 0x7f4f280b0ac0 2021-04-09T18:31:20.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.621 7f4f257fa700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f0c0188c0 0x7f4f0c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:20.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.621 7f4f257fa700 1 -- 172.21.15.32:0/802939129 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f4f20032820 con 0x7f4f280b0ac0 2021-04-09T18:31:20.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.621 7f4f2cef6700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f0c0188c0 0x7f4f0c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:20.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.622 7f4f2cef6700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f0c0188c0 0x7f4f0c01adc0 crc :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:20.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.641 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2817cb20 con 0x7f4f280b0ac0 2021-04-09T18:31:20.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.645 7f4f257fa700 1 -- 172.21.15.32:0/802939129 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4f20026790 con 0x7f4f280b0ac0 2021-04-09T18:31:20.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.821 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4f2817cb20 con 0x7f4f280b0ac0 2021-04-09T18:31:20.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.821 7f4f257fa700 1 -- 172.21.15.32:0/802939129 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v196) v1 ==== 84+0+64 (secure 0 0 0) 0x7f4f20026790 con 0x7f4f280b0ac0 2021-04-09T18:31:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:20.837 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:31:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:20.837 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:31:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:31:20.838 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:20.838 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:31:20.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f0c0188c0 msgr2=0x7f4f0c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f0c0188c0 0x7f4f0c01adc0 crc :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f280b0ac0 msgr2=0x7f4f280b0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:20.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f280b0ac0 0x7f4f280b0ec0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 rx=0x7f4f20001070 tx=0x7f4f2000bd40).stop 2021-04-09T18:31:20.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 shutdown_connections 2021-04-09T18:31:20.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f0c0188c0 0x7f4f0c01adc0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f280b0ac0 0x7f4f280b0ec0 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f28182850 0x7f4f28184880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 --2- 172.21.15.32:0/802939129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2819d870 0x7f4f28184260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:20.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 >> 172.21.15.32:0/802939129 conn(0x7f4f281965a0 msgr2=0x7f4f2819dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:20.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 shutdown_connections 2021-04-09T18:31:20.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.824 7f4f2f0da700 1 -- 172.21.15.32:0/802939129 wait complete. 2021-04-09T18:31:20.849 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'alpha' 2021-04-09T18:31:20.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create alpha_metadata 8 2021-04-09T18:31:21.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.987 7f27c6537700 1 Processor -- start 2021-04-09T18:31:21.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.987 7f27c6537700 1 -- start start 2021-04-09T18:31:21.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.988 7f27c6537700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00d7ca0 0x7f27c00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.988 7f27c6537700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c010d480 con 0x7f27c010da00 2021-04-09T18:31:21.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.988 7f27c6537700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c00d86c0 con 0x7f27c010d010 2021-04-09T18:31:21.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.988 7f27c6537700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c00d9220 con 0x7f27c00d7ca0 2021-04-09T18:31:21.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.988 7f27c4b54700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f27c010da00 0x7f27c010de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38114/0 (socket says 172.21.15.32:38114) 2021-04-09T18:31:21.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.988 7f27c4b54700 1 -- 172.21.15.32:0/3182643166 learned_addr learned my addr 172.21.15.32:0/3182643166 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:21.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.988 7f27b7fff700 1 --2- 172.21.15.32:0/3182643166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00d7ca0 0x7f27c00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.988 7f27bf7fe700 1 -- 172.21.15.32:0/3182643166 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585415437 0 0) 0x7f27b8000d30 con 0x7f27c010da00 2021-04-09T18:31:21.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27bf7fe700 1 -- 172.21.15.32:0/3182643166 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a0002fe0 con 0x7f27c010da00 2021-04-09T18:31:21.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27bf7fe700 1 -- 172.21.15.32:0/3182643166 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481081412 0 0) 0x7f27b00009b0 con 0x7f27c010d010 2021-04-09T18:31:21.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27bf7fe700 1 -- 172.21.15.32:0/3182643166 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a0003210 con 0x7f27c010d010 2021-04-09T18:31:21.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27b7fff700 1 -- 172.21.15.32:0/3182643166 >> v1:172.21.15.86:6789/0 conn(0x7f27c010d010 legacy=0x7f27c00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27b7fff700 1 -- 172.21.15.32:0/3182643166 >> v1:172.21.15.32:6789/0 conn(0x7f27c010da00 legacy=0x7f27c010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27b7fff700 1 -- 172.21.15.32:0/3182643166 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c0112100 con 0x7f27c00d7ca0 2021-04-09T18:31:21.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27b7fff700 1 --2- 172.21.15.32:0/3182643166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00d7ca0 0x7f27c00d80a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f27a8006650 tx=0x7f27a8009a80).ready entity=mon.2 client_cookie=bde186dd4b37247d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:21.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27bf7fe700 1 -- 172.21.15.32:0/3182643166 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a80009b0 con 0x7f27c00d7ca0 2021-04-09T18:31:21.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27bf7fe700 1 -- 172.21.15.32:0/3182643166 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a800c0c0 con 0x7f27c00d7ca0 2021-04-09T18:31:21.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27bf7fe700 1 -- 172.21.15.32:0/3182643166 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a800c510 con 0x7f27c00d7ca0 2021-04-09T18:31:21.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27c6537700 1 -- 172.21.15.32:0/3182643166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00d7ca0 msgr2=0x7f27c00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.989 7f27c6537700 1 --2- 172.21.15.32:0/3182643166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00d7ca0 0x7f27c00d80a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f27a8006650 tx=0x7f27a8009a80).stop 2021-04-09T18:31:21.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.990 7f27c6537700 1 -- 172.21.15.32:0/3182643166 shutdown_connections 2021-04-09T18:31:21.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.990 7f27c6537700 1 --2- 172.21.15.32:0/3182643166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00d7ca0 0x7f27c00d80a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.990 7f27c6537700 1 -- 172.21.15.32:0/3182643166 >> 172.21.15.32:0/3182643166 conn(0x7f27c00ce5f0 msgr2=0x7f27c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:21.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.990 7f27c6537700 1 -- 172.21.15.32:0/3182643166 shutdown_connections 2021-04-09T18:31:21.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:20.990 7f27c6537700 1 -- 172.21.15.32:0/3182643166 wait complete. 2021-04-09T18:31:21.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.010 7f27c6537700 1 Processor -- start 2021-04-09T18:31:21.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.010 7f27c6537700 1 -- start start 2021-04-09T18:31:21.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.010 7f27c6537700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c0190ee0 0x7f27c00a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27c6537700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c00a3570 0x7f27c00a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27c6537700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00a6150 0x7f27c00a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27c6537700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c00a55b0 con 0x7f27c00a3570 2021-04-09T18:31:21.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27c6537700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c00a3970 con 0x7f27c0190ee0 2021-04-09T18:31:21.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27c6537700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c00a6550 con 0x7f27c00a6150 2021-04-09T18:31:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c00a3570 0x7f27c00a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27b7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c00a3570 0x7f27c00a5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56760/0 (socket says 172.21.15.32:56760) 2021-04-09T18:31:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27b7fff700 1 -- 172.21.15.32:0/2933548523 learned_addr learned my addr 172.21.15.32:0/2933548523 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:21.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27c4b54700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00a6150 0x7f27c00a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27bffff700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c0190ee0 0x7f27c00a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27b7fff700 1 -- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00a6150 msgr2=0x7f27c00a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27b7fff700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00a6150 0x7f27c00a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27b7fff700 1 -- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c0190ee0 msgr2=0x7f27c00a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27b7fff700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c0190ee0 0x7f27c00a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27b7fff700 1 -- 172.21.15.32:0/2933548523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c00a3d80 con 0x7f27c00a3570 2021-04-09T18:31:21.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27bffff700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c0190ee0 0x7f27c00a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:21.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27b7fff700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c00a3570 0x7f27c00a5b30 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 rx=0x7f27a8009650 tx=0x7f27a800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:21.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27bd7fa700 1 -- 172.21.15.32:0/2933548523 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a8006030 con 0x7f27c00a3570 2021-04-09T18:31:21.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27bd7fa700 1 -- 172.21.15.32:0/2933548523 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a8006820 con 0x7f27c00a3570 2021-04-09T18:31:21.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27c6537700 1 -- 172.21.15.32:0/2933548523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c00a3d80 con 0x7f27c00a3570 2021-04-09T18:31:21.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27bd7fa700 1 -- 172.21.15.32:0/2933548523 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a80052e0 con 0x7f27c00a3570 2021-04-09T18:31:21.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.011 7f27c6537700 1 -- 172.21.15.32:0/2933548523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c00a3d80 con 0x7f27c00a3570 2021-04-09T18:31:21.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.012 7f27bd7fa700 1 -- 172.21.15.32:0/2933548523 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f27a8009300 con 0x7f27c00a3570 2021-04-09T18:31:21.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.012 7f27bd7fa700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27980188c0 0x7f279801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.012 7f27bd7fa700 1 -- 172.21.15.32:0/2933548523 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f27a80324a0 con 0x7f27c00a3570 2021-04-09T18:31:21.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.012 7f27bffff700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27980188c0 0x7f279801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.012 7f27bffff700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27980188c0 0x7f279801adc0 crc :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:21.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.032 7f27c6537700 1 -- 172.21.15.32:0/2933548523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c010d480 con 0x7f27c00a3570 2021-04-09T18:31:21.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.035 7f27bd7fa700 1 -- 172.21.15.32:0/2933548523 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f27a802efe0 con 0x7f27c00a3570 2021-04-09T18:31:21.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.219 7f27c6537700 1 -- 172.21.15.32:0/2933548523 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_metadata"} v 0) v1 -- 0x7f27c010d480 con 0x7f27c00a3570 2021-04-09T18:31:21.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.633 7f27bd7fa700 1 -- 172.21.15.32:0/2933548523 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_metadata"}]=0 pool 'alpha_metadata' created v197) v1 ==== 131+0+0 (secure 0 0 0) 0x7f27a802efe0 con 0x7f27c00a3570 2021-04-09T18:31:21.698 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_metadata' created 2021-04-09T18:31:21.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 -- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27980188c0 msgr2=0x7f279801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27980188c0 0x7f279801adc0 crc :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 -- 172.21.15.32:0/2933548523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c00a3570 msgr2=0x7f27c00a5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c00a3570 0x7f27c00a5b30 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 rx=0x7f27a8009650 tx=0x7f27a800b8e0).stop 2021-04-09T18:31:21.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 -- 172.21.15.32:0/2933548523 shutdown_connections 2021-04-09T18:31:21.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27980188c0 0x7f279801adc0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27c00a3570 0x7f27c00a5b30 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27c00a6150 0x7f27c00a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 --2- 172.21.15.32:0/2933548523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c0190ee0 0x7f27c00a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 -- 172.21.15.32:0/2933548523 >> 172.21.15.32:0/2933548523 conn(0x7f27c00bbda0 msgr2=0x7f27c01b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:21.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.685 7f27c6537700 1 -- 172.21.15.32:0/2933548523 shutdown_connections 2021-04-09T18:31:21.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.686 7f27c6537700 1 -- 172.21.15.32:0/2933548523 wait complete. 2021-04-09T18:31:21.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create alpha_data 8 2021-04-09T18:31:21.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.849 7fc22bfd6700 1 Processor -- start 2021-04-09T18:31:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.849 7fc22bfd6700 1 -- start start 2021-04-09T18:31:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.849 7fc22bfd6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc22410d010 0x7fc2240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.849 7fc22bfd6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240d8490 0x7fc22410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.849 7fc22bfd6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22410d480 con 0x7fc2240d1da0 2021-04-09T18:31:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.849 7fc22bfd6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2240d8d30 con 0x7fc22410d010 2021-04-09T18:31:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.849 7fc22bfd6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240d9220 con 0x7fc2240d8490 2021-04-09T18:31:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc2295f1700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc2240d1da0 0x7fc2240d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38128/0 (socket says 172.21.15.32:38128) 2021-04-09T18:31:21.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc2295f1700 1 -- 172.21.15.32:0/2474459571 learned_addr learned my addr 172.21.15.32:0/2474459571 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc22a5f3700 1 --2- 172.21.15.32:0/2474459571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240d8490 0x7fc22410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc229df2700 1 --2- 172.21.15.32:0/2474459571 >> v2:172.21.15.86:3300/0 conn(0x7fc22410d010 0x7fc2240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc228df0700 1 -- 172.21.15.32:0/2474459571 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273472922 0 0) 0x7fc214000ce0 con 0x7fc2240d1da0 2021-04-09T18:31:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc228df0700 1 -- 172.21.15.32:0/2474459571 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc210002fe0 con 0x7fc2240d1da0 2021-04-09T18:31:21.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc229df2700 1 -- 172.21.15.32:0/2474459571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240d8490 msgr2=0x7fc22410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc229df2700 1 --2- 172.21.15.32:0/2474459571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240d8490 0x7fc22410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc229df2700 1 -- 172.21.15.32:0/2474459571 >> v1:172.21.15.32:6789/0 conn(0x7fc2240d1da0 legacy=0x7fc2240d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc229df2700 1 -- 172.21.15.32:0/2474459571 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22410e020 con 0x7fc22410d010 2021-04-09T18:31:21.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc22a5f3700 1 --2- 172.21.15.32:0/2474459571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240d8490 0x7fc22410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc229df2700 1 --2- 172.21.15.32:0/2474459571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc22410d010 0x7fc2240d1780 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7fc220006650 tx=0x7fc220009a80).ready entity=mon.1 client_cookie=1f6a83f0164e96e6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc228df0700 1 -- 172.21.15.32:0/2474459571 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2200009b0 con 0x7fc22410d010 2021-04-09T18:31:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.850 7fc228df0700 1 -- 172.21.15.32:0/2474459571 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22000c0c0 con 0x7fc22410d010 2021-04-09T18:31:21.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.851 7fc228df0700 1 -- 172.21.15.32:0/2474459571 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc22000c510 con 0x7fc22410d010 2021-04-09T18:31:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.851 7fc22bfd6700 1 -- 172.21.15.32:0/2474459571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc22410d010 msgr2=0x7fc2240d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.851 7fc22bfd6700 1 --2- 172.21.15.32:0/2474459571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc22410d010 0x7fc2240d1780 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7fc220006650 tx=0x7fc220009a80).stop 2021-04-09T18:31:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.851 7fc22bfd6700 1 -- 172.21.15.32:0/2474459571 shutdown_connections 2021-04-09T18:31:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.851 7fc22bfd6700 1 --2- 172.21.15.32:0/2474459571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240d8490 0x7fc22410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.851 7fc22bfd6700 1 --2- 172.21.15.32:0/2474459571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc22410d010 0x7fc2240d1780 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.851 7fc22bfd6700 1 -- 172.21.15.32:0/2474459571 >> 172.21.15.32:0/2474459571 conn(0x7fc2240ce5f0 msgr2=0x7fc2240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.851 7fc22bfd6700 1 -- 172.21.15.32:0/2474459571 shutdown_connections 2021-04-09T18:31:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.852 7fc22bfd6700 1 -- 172.21.15.32:0/2474459571 wait complete. 2021-04-09T18:31:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc22bfd6700 1 Processor -- start 2021-04-09T18:31:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc22bfd6700 1 -- start start 2021-04-09T18:31:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc22bfd6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240b1e50 0x7fc2240b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc22bfd6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2240b9fd0 0x7fc2240b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc22bfd6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2240b5510 0x7fc2240b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc22bfd6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240b2250 con 0x7fc2240b9fd0 2021-04-09T18:31:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc22bfd6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240a1220 con 0x7fc2240b5510 2021-04-09T18:31:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc22bfd6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240b6b30 con 0x7fc2240b1e50 2021-04-09T18:31:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc2295f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2240b9fd0 0x7fc2240b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc2295f1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2240b9fd0 0x7fc2240b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56772/0 (socket says 172.21.15.32:56772) 2021-04-09T18:31:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.872 7fc2295f1700 1 -- 172.21.15.32:0/1821034947 learned_addr learned my addr 172.21.15.32:0/1821034947 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc22a5f3700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2240b5510 0x7fc2240b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc229df2700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240b1e50 0x7fc2240b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc2295f1700 1 -- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240b1e50 msgr2=0x7fc2240b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc2295f1700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240b1e50 0x7fc2240b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc2295f1700 1 -- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2240b5510 msgr2=0x7fc2240b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc2295f1700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2240b5510 0x7fc2240b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc2295f1700 1 -- 172.21.15.32:0/1821034947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2240b5f30 con 0x7fc2240b9fd0 2021-04-09T18:31:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc229df2700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240b1e50 0x7fc2240b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:21.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc22a5f3700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2240b5510 0x7fc2240b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc2295f1700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2240b9fd0 0x7fc2240b4ef0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 rx=0x7fc214006f00 tx=0x7fc21400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc21a7fc700 1 -- 172.21.15.32:0/1821034947 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc21400bd30 con 0x7fc2240b9fd0 2021-04-09T18:31:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc21a7fc700 1 -- 172.21.15.32:0/1821034947 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc214006030 con 0x7fc2240b9fd0 2021-04-09T18:31:21.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc21a7fc700 1 -- 172.21.15.32:0/1821034947 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc214004590 con 0x7fc2240b9fd0 2021-04-09T18:31:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2240b5f30 con 0x7fc2240b9fd0 2021-04-09T18:31:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.873 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2240b5f30 con 0x7fc2240b9fd0 2021-04-09T18:31:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.874 7fc21a7fc700 1 -- 172.21.15.32:0/1821034947 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc2140043a0 con 0x7fc2240b9fd0 2021-04-09T18:31:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.874 7fc21a7fc700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc20c018930 0x7fc20c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.874 7fc21a7fc700 1 -- 172.21.15.32:0/1821034947 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5110+0+0 (secure 0 0 0) 0x7fc2140327f0 con 0x7fc2240b9fd0 2021-04-09T18:31:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.874 7fc229df2700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc20c018930 0x7fc20c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.874 7fc229df2700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc20c018930 0x7fc20c01adc0 crc :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.893 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2241b7040 con 0x7fc2240b9fd0 2021-04-09T18:31:21.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:21.897 7fc21a7fc700 1 -- 172.21.15.32:0/1821034947 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc2140009b0 con 0x7fc2240b9fd0 2021-04-09T18:31:22.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:22.084 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_data"} v 0) v1 -- 0x7fc2241b7040 con 0x7fc2240b9fd0 2021-04-09T18:31:23.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.082 7fc21a7fc700 1 -- 172.21.15.32:0/1821034947 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_data"}]=0 pool 'alpha_data' created v201) v1 ==== 123+0+0 (secure 0 0 0) 0x7fc2140043a0 con 0x7fc2240b9fd0 2021-04-09T18:31:23.115 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_data' created 2021-04-09T18:31:23.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc20c018930 msgr2=0x7fc20c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc20c018930 0x7fc20c01adc0 crc :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2240b9fd0 msgr2=0x7fc2240b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2240b9fd0 0x7fc2240b4ef0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 rx=0x7fc214006f00 tx=0x7fc21400b940).stop 2021-04-09T18:31:23.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 shutdown_connections 2021-04-09T18:31:23.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc20c018930 0x7fc20c01adc0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2240b9fd0 0x7fc2240b4ef0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2240b5510 0x7fc2240b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 --2- 172.21.15.32:0/1821034947 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2240b1e50 0x7fc2240b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 >> 172.21.15.32:0/1821034947 conn(0x7fc2240b27e0 msgr2=0x7fc2241b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:23.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 shutdown_connections 2021-04-09T18:31:23.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.103 7fc22bfd6700 1 -- 172.21.15.32:0/1821034947 wait complete. 2021-04-09T18:31:23.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new alpha alpha_metadata alpha_data --force 2021-04-09T18:31:23.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.282 7f658ac9f700 1 Processor -- start 2021-04-09T18:31:23.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.282 7f658ac9f700 1 -- start start 2021-04-09T18:31:23.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f658ac9f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65840d7cc0 0x7f65840d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f658ac9f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f658410da80 0x7f658410de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f658ac9f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658410d500 con 0x7f658410d050 2021-04-09T18:31:23.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f658ac9f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65840d9260 con 0x7f65840d7cc0 2021-04-09T18:31:23.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f658ac9f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65840d0330 con 0x7f658410da80 2021-04-09T18:31:23.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f6583fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65840d7cc0 0x7f65840d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f6583fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65840d7cc0 0x7f65840d80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56784/0 (socket says 172.21.15.32:56784) 2021-04-09T18:31:23.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f6583fff700 1 -- 172.21.15.32:0/349273766 learned_addr learned my addr 172.21.15.32:0/349273766 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:23.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f65892bc700 1 --2- 172.21.15.32:0/349273766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f658410da80 0x7f658410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f6583fff700 1 -- 172.21.15.32:0/349273766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f658410da80 msgr2=0x7f658410de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:23.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f6583fff700 1 --2- 172.21.15.32:0/349273766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f658410da80 0x7f658410de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f6583fff700 1 -- 172.21.15.32:0/349273766 >> v1:172.21.15.86:6789/0 conn(0x7f658410d050 legacy=0x7f65840cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f65837fe700 1 -- 172.21.15.32:0/349273766 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209841530 0 0) 0x7f656c000c60 con 0x7f658410d050 2021-04-09T18:31:23.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f6583fff700 1 -- 172.21.15.32:0/349273766 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65840d86e0 con 0x7f65840d7cc0 2021-04-09T18:31:23.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f6583fff700 1 --2- 172.21.15.32:0/349273766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840d7cc0 0x7f65840d80c0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7f6574006a80 tx=0x7f6574009f00).ready entity=mon.0 client_cookie=90802d755c52a43b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:23.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f65837fe700 1 -- 172.21.15.32:0/349273766 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f657400b5c0 con 0x7f65840d7cc0 2021-04-09T18:31:23.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f65837fe700 1 -- 172.21.15.32:0/349273766 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657400b8c0 con 0x7f65840d7cc0 2021-04-09T18:31:23.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f65837fe700 1 -- 172.21.15.32:0/349273766 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f657400bda0 con 0x7f65840d7cc0 2021-04-09T18:31:23.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f658ac9f700 1 -- 172.21.15.32:0/349273766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840d7cc0 msgr2=0x7f65840d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.283 7f658ac9f700 1 --2- 172.21.15.32:0/349273766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840d7cc0 0x7f65840d80c0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7f6574006a80 tx=0x7f6574009f00).stop 2021-04-09T18:31:23.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.284 7f658ac9f700 1 -- 172.21.15.32:0/349273766 shutdown_connections 2021-04-09T18:31:23.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.284 7f658ac9f700 1 --2- 172.21.15.32:0/349273766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f658410da80 0x7f658410de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.284 7f658ac9f700 1 --2- 172.21.15.32:0/349273766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840d7cc0 0x7f65840d80c0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.284 7f658ac9f700 1 -- 172.21.15.32:0/349273766 >> 172.21.15.32:0/349273766 conn(0x7f65840ce620 msgr2=0x7f65840d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:23.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.284 7f658ac9f700 1 -- 172.21.15.32:0/349273766 shutdown_connections 2021-04-09T18:31:23.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.284 7f658ac9f700 1 -- 172.21.15.32:0/349273766 wait complete. 2021-04-09T18:31:23.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.304 7f658ac9f700 1 Processor -- start 2021-04-09T18:31:23.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.304 7f658ac9f700 1 -- start start 2021-04-09T18:31:23.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f658ac9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840b0920 0x7f65840b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f658ac9f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65840a7690 0x7f65840a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f658ac9f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65840c2510 0x7f65840c2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f658ac9f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65840b1340 con 0x7f65840b0920 2021-04-09T18:31:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f658ac9f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65840a82e0 con 0x7f65840c2510 2021-04-09T18:31:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f658ac9f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65840a8820 con 0x7f65840a7690 2021-04-09T18:31:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f6588abb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840b0920 0x7f65840b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f6588abb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840b0920 0x7f65840b0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56788/0 (socket says 172.21.15.32:56788) 2021-04-09T18:31:23.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f6588abb700 1 -- 172.21.15.32:0/2011439319 learned_addr learned my addr 172.21.15.32:0/2011439319 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:23.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f6583fff700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65840a7690 0x7f65840a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.305 7f65892bc700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65840c2510 0x7f65840c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f6588abb700 1 -- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65840a7690 msgr2=0x7f65840a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f6588abb700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65840a7690 0x7f65840a7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f6588abb700 1 -- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65840c2510 msgr2=0x7f65840c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f6588abb700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65840c2510 0x7f65840c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f6588abb700 1 -- 172.21.15.32:0/2011439319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658408fba0 con 0x7f65840b0920 2021-04-09T18:31:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f6588abb700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840b0920 0x7f65840b0d20 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 rx=0x7f656c006f20 tx=0x7f656c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:23.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f65817fa700 1 -- 172.21.15.32:0/2011439319 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f656c00bd80 con 0x7f65840b0920 2021-04-09T18:31:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f65817fa700 1 -- 172.21.15.32:0/2011439319 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c001080 con 0x7f65840b0920 2021-04-09T18:31:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f65817fa700 1 -- 172.21.15.32:0/2011439319 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f656c001550 con 0x7f65840b0920 2021-04-09T18:31:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658408fba0 con 0x7f65840b0920 2021-04-09T18:31:23.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.306 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658408fba0 con 0x7f65840b0920 2021-04-09T18:31:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.307 7f65817fa700 1 -- 172.21.15.32:0/2011439319 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f656c001260 con 0x7f65840b0920 2021-04-09T18:31:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.307 7f65817fa700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65680188d0 0x7f656801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.307 7f65817fa700 1 -- 172.21.15.32:0/2011439319 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5459+0+0 (secure 0 0 0) 0x7f656c0329d0 con 0x7f65840b0920 2021-04-09T18:31:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.307 7f6583fff700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65680188d0 0x7f656801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.307 7f6583fff700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65680188d0 0x7f656801add0 crc :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:23.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.326 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65841b7180 con 0x7f65840b0920 2021-04-09T18:31:23.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.330 7f65817fa700 1 -- 172.21.15.32:0/2011439319 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f656c006bf0 con 0x7f65840b0920 2021-04-09T18:31:23.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.512 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "alpha_data", "prefix": "fs new", "force": true, "fs_name": "alpha", "metadata": "alpha_metadata"} v 0) v1 -- 0x7f658410d500 con 0x7f65840b0920 2021-04-09T18:31:23.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.516 7f65817fa700 1 -- 172.21.15.32:0/2011439319 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "alpha_data", "prefix": "fs new", "force": true, "fs_name": "alpha", "metadata": "alpha_metadata"}]=0 new fs with metadata pool 34 and data pool 35 v346) v1 ==== 186+0+0 (secure 0 0 0) 0x7f656c001240 con 0x7f65840b0920 2021-04-09T18:31:23.534 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 34 and data pool 35 2021-04-09T18:31:23.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65680188d0 msgr2=0x7f656801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65680188d0 0x7f656801add0 crc :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840b0920 msgr2=0x7f65840b0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840b0920 0x7f65840b0d20 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 rx=0x7f656c006f20 tx=0x7f656c00b990).stop 2021-04-09T18:31:23.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 shutdown_connections 2021-04-09T18:31:23.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65680188d0 0x7f656801add0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65840c2510 0x7f65840c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65840a7690 0x7f65840a7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 --2- 172.21.15.32:0/2011439319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65840b0920 0x7f65840b0d20 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 >> 172.21.15.32:0/2011439319 conn(0x7f65840b6b70 msgr2=0x7f65840b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:23.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 shutdown_connections 2021-04-09T18:31:23.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.522 7f658ac9f700 1 -- 172.21.15.32:0/2011439319 wait complete. 2021-04-09T18:31:23.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:31:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.681 7f89f448f700 1 Processor -- start 2021-04-09T18:31:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.681 7f89f448f700 1 -- start start 2021-04-09T18:31:23.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.681 7f89f448f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89ec0d7ca0 0x7f89ec0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.681 7f89f448f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec10da00 0x7f89ec10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f448f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ec10d480 con 0x7f89ec10d010 2021-04-09T18:31:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f448f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ec0d9220 con 0x7f89ec0d7ca0 2021-04-09T18:31:23.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f448f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec0d86c0 con 0x7f89ec10da00 2021-04-09T18:31:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f1aaa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f89ec0d7ca0 0x7f89ec0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f22ab700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f89ec10d010 0x7f89ec0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38154/0 (socket says 172.21.15.32:38154) 2021-04-09T18:31:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f22ab700 1 -- 172.21.15.32:0/4014815121 learned_addr learned my addr 172.21.15.32:0/4014815121 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f2aac700 1 --2- 172.21.15.32:0/4014815121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec10da00 0x7f89ec10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f1aaa700 1 -- 172.21.15.32:0/4014815121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec10da00 msgr2=0x7f89ec10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:23.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f1aaa700 1 --2- 172.21.15.32:0/4014815121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec10da00 0x7f89ec10de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f1aaa700 1 -- 172.21.15.32:0/4014815121 >> v1:172.21.15.32:6789/0 conn(0x7f89ec10d010 legacy=0x7f89ec0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f1aaa700 1 -- 172.21.15.32:0/4014815121 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec0d1f70 con 0x7f89ec0d7ca0 2021-04-09T18:31:23.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f12a9700 1 -- 172.21.15.32:0/4014815121 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363946548 0 0) 0x7f89e8000ce0 con 0x7f89ec10d010 2021-04-09T18:31:23.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f1aaa700 1 --2- 172.21.15.32:0/4014815121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec0d7ca0 0x7f89ec0d80a0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 rx=0x7f89dc006650 tx=0x7f89dc009a00).ready entity=mon.0 client_cookie=f8bce3ec0f3f777f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:23.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f12a9700 1 -- 172.21.15.32:0/4014815121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89dc009df0 con 0x7f89ec0d7ca0 2021-04-09T18:31:23.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f12a9700 1 -- 172.21.15.32:0/4014815121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc0009b0 con 0x7f89ec0d7ca0 2021-04-09T18:31:23.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f12a9700 1 -- 172.21.15.32:0/4014815121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89dc00c2b0 con 0x7f89ec0d7ca0 2021-04-09T18:31:23.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f448f700 1 -- 172.21.15.32:0/4014815121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec0d7ca0 msgr2=0x7f89ec0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.682 7f89f448f700 1 --2- 172.21.15.32:0/4014815121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec0d7ca0 0x7f89ec0d80a0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 rx=0x7f89dc006650 tx=0x7f89dc009a00).stop 2021-04-09T18:31:23.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.683 7f89f448f700 1 -- 172.21.15.32:0/4014815121 shutdown_connections 2021-04-09T18:31:23.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.683 7f89f448f700 1 --2- 172.21.15.32:0/4014815121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec10da00 0x7f89ec10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.683 7f89f448f700 1 --2- 172.21.15.32:0/4014815121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec0d7ca0 0x7f89ec0d80a0 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.683 7f89f448f700 1 -- 172.21.15.32:0/4014815121 >> 172.21.15.32:0/4014815121 conn(0x7f89ec0ce5f0 msgr2=0x7f89ec0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:23.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.683 7f89f448f700 1 -- 172.21.15.32:0/4014815121 shutdown_connections 2021-04-09T18:31:23.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.683 7f89f448f700 1 -- 172.21.15.32:0/4014815121 wait complete. 2021-04-09T18:31:23.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.703 7f89f448f700 1 Processor -- start 2021-04-09T18:31:23.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.703 7f89f448f700 1 -- start start 2021-04-09T18:31:23.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f448f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89ec1acea0 0x7f89ec09e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f448f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec1ac380 0x7f89ec09d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f448f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec09df40 0x7f89ec09f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f448f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec09edb0 con 0x7f89ec09df40 2021-04-09T18:31:23.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f448f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec09ea10 con 0x7f89ec1acea0 2021-04-09T18:31:23.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f448f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec1ac780 con 0x7f89ec1ac380 2021-04-09T18:31:23.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f2aac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec09df40 0x7f89ec09f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f2aac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec09df40 0x7f89ec09f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56804/0 (socket says 172.21.15.32:56804) 2021-04-09T18:31:23.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f2aac700 1 -- 172.21.15.32:0/3843738782 learned_addr learned my addr 172.21.15.32:0/3843738782 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:23.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f1aaa700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec1ac380 0x7f89ec09d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f22ab700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89ec1acea0 0x7f89ec09e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f2aac700 1 -- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec1ac380 msgr2=0x7f89ec09d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f2aac700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec1ac380 0x7f89ec09d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f2aac700 1 -- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89ec1acea0 msgr2=0x7f89ec09e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f2aac700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89ec1acea0 0x7f89ec09e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f2aac700 1 -- 172.21.15.32:0/3843738782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec1ac9a0 con 0x7f89ec09df40 2021-04-09T18:31:23.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89f2aac700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec09df40 0x7f89ec09f2b0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 rx=0x7f89e400b970 tx=0x7f89e400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:23.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.704 7f89e2ffd700 1 -- 172.21.15.32:0/3843738782 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89e4000f90 con 0x7f89ec09df40 2021-04-09T18:31:23.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.705 7f89e2ffd700 1 -- 172.21.15.32:0/3843738782 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e4006aa0 con 0x7f89ec09df40 2021-04-09T18:31:23.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.705 7f89f448f700 1 -- 172.21.15.32:0/3843738782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ec1ac9a0 con 0x7f89ec09df40 2021-04-09T18:31:23.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.705 7f89e2ffd700 1 -- 172.21.15.32:0/3843738782 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89e4001470 con 0x7f89ec09df40 2021-04-09T18:31:23.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.705 7f89f448f700 1 -- 172.21.15.32:0/3843738782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ec1ac9a0 con 0x7f89ec09df40 2021-04-09T18:31:23.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.705 7f89e2ffd700 1 -- 172.21.15.32:0/3843738782 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f89e4001650 con 0x7f89ec09df40 2021-04-09T18:31:23.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.705 7f89e2ffd700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89d4018930 0x7f89d401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:23.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.705 7f89e2ffd700 1 -- 172.21.15.32:0/3843738782 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f89e4032da0 con 0x7f89ec09df40 2021-04-09T18:31:23.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.705 7f89f22ab700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89d4018930 0x7f89d401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:23.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.706 7f89f22ab700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89d4018930 0x7f89d401adc0 crc :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:23.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.726 7f89f448f700 1 -- 172.21.15.32:0/3843738782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ec182d80 con 0x7f89ec09df40 2021-04-09T18:31:23.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.729 7f89e2ffd700 1 -- 172.21.15.32:0/3843738782 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f89e4001ef0 con 0x7f89ec09df40 2021-04-09T18:31:23.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.918 7f89f448f700 1 -- 172.21.15.32:0/3843738782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89ec10d480 con 0x7f89ec09df40 2021-04-09T18:31:23.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.919 7f89e2ffd700 1 -- 172.21.15.32:0/3843738782 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+14016 (secure 0 0 0) 0x7f89e4001ef0 con 0x7f89ec09df40 2021-04-09T18:31:23.935 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:23.935 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":202,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:31:23.513384","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:31:21.254184","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":35,"pool_name":"alpha_data","create_time":"2021-04-09 18:31:22.120246","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:31:23.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 -- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89d4018930 msgr2=0x7f89d401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89d4018930 0x7f89d401adc0 crc :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 -- 172.21.15.32:0/3843738782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec09df40 msgr2=0x7f89ec09f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:23.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec09df40 0x7f89ec09f2b0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 rx=0x7f89e400b970 tx=0x7f89e400bd10).stop 2021-04-09T18:31:23.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 -- 172.21.15.32:0/3843738782 shutdown_connections 2021-04-09T18:31:23.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f89d4018930 0x7f89d401adc0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89ec09df40 0x7f89ec09f2b0 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89ec1ac380 0x7f89ec09d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 --2- 172.21.15.32:0/3843738782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89ec1acea0 0x7f89ec09e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:23.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.921 7f89f448f700 1 -- 172.21.15.32:0/3843738782 >> 172.21.15.32:0/3843738782 conn(0x7f89ec19b890 msgr2=0x7f89ec1a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:23.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.922 7f89f448f700 1 -- 172.21.15.32:0/3843738782 shutdown_connections 2021-04-09T18:31:23.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:23.922 7f89f448f700 1 -- 172.21.15.32:0/3843738782 wait complete. 2021-04-09T18:31:23.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:31:24.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.082 7f3d0b00b700 1 Processor -- start 2021-04-09T18:31:24.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.082 7f3d0b00b700 1 -- start start 2021-04-09T18:31:24.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.082 7f3d0b00b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d040d6c80 0x7f3d040d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.082 7f3d0b00b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d040d7280 con 0x7f3d040cffa0 2021-04-09T18:31:24.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.082 7f3d0b00b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d040d2840 con 0x7f3d040d2440 2021-04-09T18:31:24.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.082 7f3d0b00b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d040d7080 con 0x7f3d040d6c80 2021-04-09T18:31:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d03fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3d040d6c80 0x7f3d040d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d08e27700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d040cffa0 0x7f3d040d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38168/0 (socket says 172.21.15.32:38168) 2021-04-09T18:31:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d08e27700 1 -- 172.21.15.32:0/2803830033 learned_addr learned my addr 172.21.15.32:0/2803830033 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d03fff700 1 -- 172.21.15.32:0/2803830033 >> v1:172.21.15.86:6789/0 conn(0x7f3d040d2440 legacy=0x7f3d0410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d03fff700 1 -- 172.21.15.32:0/2803830033 >> v1:172.21.15.32:6789/0 conn(0x7f3d040cffa0 legacy=0x7f3d040d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d037fe700 1 -- 172.21.15.32:0/2803830033 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767696375 0 0) 0x7f3cfc000f90 con 0x7f3d040d2440 2021-04-09T18:31:24.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d03fff700 1 -- 172.21.15.32:0/2803830033 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d040d91e0 con 0x7f3d040d6c80 2021-04-09T18:31:24.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d03fff700 1 --2- 172.21.15.32:0/2803830033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d040d6c80 0x7f3d040d7ff0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 rx=0x7f3cf0006650 tx=0x7f3cf0009a80).ready entity=mon.0 client_cookie=f2dc2d155ebad3f7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:24.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d037fe700 1 -- 172.21.15.32:0/2803830033 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cf00009b0 con 0x7f3d040d6c80 2021-04-09T18:31:24.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d037fe700 1 -- 172.21.15.32:0/2803830033 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf000c0c0 con 0x7f3d040d6c80 2021-04-09T18:31:24.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d037fe700 1 -- 172.21.15.32:0/2803830033 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cf000c510 con 0x7f3d040d6c80 2021-04-09T18:31:24.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d0b00b700 1 -- 172.21.15.32:0/2803830033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d040d6c80 msgr2=0x7f3d040d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.083 7f3d0b00b700 1 --2- 172.21.15.32:0/2803830033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d040d6c80 0x7f3d040d7ff0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 rx=0x7f3cf0006650 tx=0x7f3cf0009a80).stop 2021-04-09T18:31:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.084 7f3d0b00b700 1 -- 172.21.15.32:0/2803830033 shutdown_connections 2021-04-09T18:31:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.084 7f3d0b00b700 1 --2- 172.21.15.32:0/2803830033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d040d6c80 0x7f3d040d7ff0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.084 7f3d0b00b700 1 -- 172.21.15.32:0/2803830033 >> 172.21.15.32:0/2803830033 conn(0x7f3d040ce5c0 msgr2=0x7f3d040cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:24.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.084 7f3d0b00b700 1 -- 172.21.15.32:0/2803830033 shutdown_connections 2021-04-09T18:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.084 7f3d0b00b700 1 -- 172.21.15.32:0/2803830033 wait complete. 2021-04-09T18:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.104 7f3d0b00b700 1 Processor -- start 2021-04-09T18:31:24.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.104 7f3d0b00b700 1 -- start start 2021-04-09T18:31:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d0b00b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d040ae0b0 0x7f3d04181e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d0b00b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d041a8240 0x7f3d041a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d0b00b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d04189e70 0x7f3d0418a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d0b00b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d041a8fa0 con 0x7f3d041a8240 2021-04-09T18:31:24.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d0b00b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d041a8ca0 con 0x7f3d040ae0b0 2021-04-09T18:31:24.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d0b00b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0418a890 con 0x7f3d04189e70 2021-04-09T18:31:24.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d03fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d041a8240 0x7f3d041a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d03fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d041a8240 0x7f3d041a8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56818/0 (socket says 172.21.15.32:56818) 2021-04-09T18:31:24.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d03fff700 1 -- 172.21.15.32:0/2048409495 learned_addr learned my addr 172.21.15.32:0/2048409495 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d08e27700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d040ae0b0 0x7f3d04181e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d09628700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d04189e70 0x7f3d0418a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d03fff700 1 -- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d04189e70 msgr2=0x7f3d0418a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d03fff700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d04189e70 0x7f3d0418a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d03fff700 1 -- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d040ae0b0 msgr2=0x7f3d04181e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d03fff700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d040ae0b0 0x7f3d04181e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d03fff700 1 -- 172.21.15.32:0/2048409495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d04182430 con 0x7f3d041a8240 2021-04-09T18:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d09628700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d04189e70 0x7f3d0418a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d08e27700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d040ae0b0 0x7f3d04181e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d03fff700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d041a8240 0x7f3d041a8640 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 rx=0x7f3cf0009650 tx=0x7f3cf000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:24.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d017fa700 1 -- 172.21.15.32:0/2048409495 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cf000bd50 con 0x7f3d041a8240 2021-04-09T18:31:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.105 7f3d017fa700 1 -- 172.21.15.32:0/2048409495 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf0006030 con 0x7f3d041a8240 2021-04-09T18:31:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.106 7f3d017fa700 1 -- 172.21.15.32:0/2048409495 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cf0005560 con 0x7f3d041a8240 2021-04-09T18:31:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.106 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d04182430 con 0x7f3d041a8240 2021-04-09T18:31:24.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.106 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d04182430 con 0x7f3d041a8240 2021-04-09T18:31:24.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.106 7f3d017fa700 1 -- 172.21.15.32:0/2048409495 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3cf0009300 con 0x7f3d041a8240 2021-04-09T18:31:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.106 7f3d017fa700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ce40188c0 0x7f3ce401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.106 7f3d017fa700 1 -- 172.21.15.32:0/2048409495 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3cf0032a00 con 0x7f3d041a8240 2021-04-09T18:31:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.106 7f3d08e27700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ce40188c0 0x7f3ce401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.107 7f3d08e27700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ce40188c0 0x7f3ce401adc0 crc :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:24.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.126 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0417cb50 con 0x7f3d041a8240 2021-04-09T18:31:24.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.130 7f3d017fa700 1 -- 172.21.15.32:0/2048409495 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3cf0000940 con 0x7f3d041a8240 2021-04-09T18:31:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.318 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d040d7280 con 0x7f3d041a8240 2021-04-09T18:31:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.319 7f3d017fa700 1 -- 172.21.15.32:0/2048409495 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+14016 (secure 0 0 0) 0x7f3cf0009300 con 0x7f3d041a8240 2021-04-09T18:31:24.334 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:24.334 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":202,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:31:23.513384","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:31:21.254184","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":35,"pool_name":"alpha_data","create_time":"2021-04-09 18:31:22.120246","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:31:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ce40188c0 msgr2=0x7f3ce401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ce40188c0 0x7f3ce401adc0 crc :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d041a8240 msgr2=0x7f3d041a8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d041a8240 0x7f3d041a8640 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 rx=0x7f3cf0009650 tx=0x7f3cf000b960).stop 2021-04-09T18:31:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 shutdown_connections 2021-04-09T18:31:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ce40188c0 0x7f3ce401adc0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d041a8240 0x7f3d041a8640 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d04189e70 0x7f3d0418a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 --2- 172.21.15.32:0/2048409495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d040ae0b0 0x7f3d04181e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 >> 172.21.15.32:0/2048409495 conn(0x7f3d041a9bd0 msgr2=0x7f3d0418d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 shutdown_connections 2021-04-09T18:31:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.321 7f3d0b00b700 1 -- 172.21.15.32:0/2048409495 wait complete. 2021-04-09T18:31:24.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set alpha standby_count_wanted 0 2021-04-09T18:31:24.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.484 7f0372142700 1 Processor -- start 2021-04-09T18:31:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.484 7f0372142700 1 -- start start 2021-04-09T18:31:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f0372142700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c0cffc0 0x7f036c0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f0372142700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f036c0d6ca0 0x7f036c10cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f0372142700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036c0d72a0 con 0x7f036c0d7820 2021-04-09T18:31:24.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f0372142700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036c0d70a0 con 0x7f036c0d6ca0 2021-04-09T18:31:24.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f0372142700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c0d85a0 con 0x7f036c0cffc0 2021-04-09T18:31:24.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f036b7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c0cffc0 0x7f036c0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f036bfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f036c0d6ca0 0x7f036c10cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f036affd700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f036c0d7820 0x7f036c0d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:36112/0 (socket says 172.21.15.32:36112) 2021-04-09T18:31:24.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f036affd700 1 -- 172.21.15.32:0/935654508 learned_addr learned my addr 172.21.15.32:0/935654508 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:24.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f036a7fc700 1 -- 172.21.15.32:0/935654508 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114713320 0 0) 0x7f03600009b0 con 0x7f036c0d7820 2021-04-09T18:31:24.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f036a7fc700 1 -- 172.21.15.32:0/935654508 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0354002fe0 con 0x7f036c0d7820 2021-04-09T18:31:24.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f036bfff700 1 -- 172.21.15.32:0/935654508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c0cffc0 msgr2=0x7f036c0d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:24.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f036bfff700 1 --2- 172.21.15.32:0/935654508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c0cffc0 0x7f036c0d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.485 7f036bfff700 1 -- 172.21.15.32:0/935654508 >> v1:172.21.15.86:6789/0 conn(0x7f036c0d7820 legacy=0x7f036c0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.486 7f036bfff700 1 -- 172.21.15.32:0/935654508 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c0d9200 con 0x7f036c0d6ca0 2021-04-09T18:31:24.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.486 7f036b7fe700 1 --2- 172.21.15.32:0/935654508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c0cffc0 0x7f036c0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.486 7f036bfff700 1 --2- 172.21.15.32:0/935654508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f036c0d6ca0 0x7f036c10cff0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f0364006e50 tx=0x7f0364009e70).ready entity=mon.1 client_cookie=cf01627b8bcf0f06 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.486 7f036a7fc700 1 -- 172.21.15.32:0/935654508 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0364006a20 con 0x7f036c0d6ca0 2021-04-09T18:31:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.486 7f036a7fc700 1 -- 172.21.15.32:0/935654508 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0364006790 con 0x7f036c0d6ca0 2021-04-09T18:31:24.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.486 7f036a7fc700 1 -- 172.21.15.32:0/935654508 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f036400b700 con 0x7f036c0d6ca0 2021-04-09T18:31:24.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.486 7f0372142700 1 -- 172.21.15.32:0/935654508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f036c0d6ca0 msgr2=0x7f036c10cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.486 7f0372142700 1 --2- 172.21.15.32:0/935654508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f036c0d6ca0 0x7f036c10cff0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f0364006e50 tx=0x7f0364009e70).stop 2021-04-09T18:31:24.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.487 7f0372142700 1 -- 172.21.15.32:0/935654508 shutdown_connections 2021-04-09T18:31:24.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.487 7f0372142700 1 --2- 172.21.15.32:0/935654508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f036c0d6ca0 0x7f036c10cff0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.487 7f0372142700 1 --2- 172.21.15.32:0/935654508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c0cffc0 0x7f036c0d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.487 7f0372142700 1 -- 172.21.15.32:0/935654508 >> 172.21.15.32:0/935654508 conn(0x7f036c0ce5e0 msgr2=0x7f036c0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:24.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.487 7f0372142700 1 -- 172.21.15.32:0/935654508 shutdown_connections 2021-04-09T18:31:24.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.487 7f0372142700 1 -- 172.21.15.32:0/935654508 wait complete. 2021-04-09T18:31:24.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.507 7f0372142700 1 Processor -- start 2021-04-09T18:31:24.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.507 7f0372142700 1 -- start start 2021-04-09T18:31:24.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.507 7f0372142700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c19e290 0x7f036c09b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0372142700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f036c09bdd0 0x7f036c094d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0372142700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f036c0953a0 0x7f036c1890b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0372142700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c09b230 con 0x7f036c0953a0 2021-04-09T18:31:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0372142700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c09c1d0 con 0x7f036c09bdd0 2021-04-09T18:31:24.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0372142700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c0957a0 con 0x7f036c19e290 2021-04-09T18:31:24.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f036c0953a0 0x7f036c1890b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f036c0953a0 0x7f036c1890b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56834/0 (socket says 172.21.15.32:56834) 2021-04-09T18:31:24.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036bfff700 1 -- 172.21.15.32:0/1388338438 learned_addr learned my addr 172.21.15.32:0/1388338438 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:24.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036affd700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f036c09bdd0 0x7f036c094d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036b7fe700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c19e290 0x7f036c09b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036bfff700 1 -- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c19e290 msgr2=0x7f036c09b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036bfff700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c19e290 0x7f036c09b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036bfff700 1 -- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f036c09bdd0 msgr2=0x7f036c094d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:24.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036bfff700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f036c09bdd0 0x7f036c094d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:24.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036bfff700 1 -- 172.21.15.32:0/1388338438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c09ae30 con 0x7f036c0953a0 2021-04-09T18:31:24.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036b7fe700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c19e290 0x7f036c09b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:24.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f036bfff700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f036c0953a0 0x7f036c1890b0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f0364006e60 tx=0x7f036400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:24.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0353fff700 1 -- 172.21.15.32:0/1388338438 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0364009130 con 0x7f036c0953a0 2021-04-09T18:31:24.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0353fff700 1 -- 172.21.15.32:0/1388338438 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03640093f0 con 0x7f036c0953a0 2021-04-09T18:31:24.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0353fff700 1 -- 172.21.15.32:0/1388338438 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0364001390 con 0x7f036c0953a0 2021-04-09T18:31:24.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0372142700 1 -- 172.21.15.32:0/1388338438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036c09ae30 con 0x7f036c0953a0 2021-04-09T18:31:24.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.508 7f0372142700 1 -- 172.21.15.32:0/1388338438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c09ae30 con 0x7f036c0953a0 2021-04-09T18:31:24.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.509 7f0353fff700 1 -- 172.21.15.32:0/1388338438 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f03640056b0 con 0x7f036c0953a0 2021-04-09T18:31:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.509 7f0353fff700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f034c0188c0 0x7f034c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.509 7f0353fff700 1 -- 172.21.15.32:0/1388338438 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0364032b20 con 0x7f036c0953a0 2021-04-09T18:31:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.509 7f036b7fe700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f034c0188c0 0x7f034c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:24.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.510 7f036b7fe700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f034c0188c0 0x7f034c01adc0 crc :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:24.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.529 7f0372142700 1 -- 172.21.15.32:0/1388338438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036c093e70 con 0x7f036c0953a0 2021-04-09T18:31:24.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.532 7f0353fff700 1 -- 172.21.15.32:0/1388338438 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0364030f00 con 0x7f036c0953a0 2021-04-09T18:31:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:24.727 7f0372142700 1 -- 172.21.15.32:0/1388338438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "alpha", "val": "0"} v 0) v1 -- 0x7f036c093e70 con 0x7f036c0953a0 2021-04-09T18:31:25.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.655 7f0353fff700 1 -- 172.21.15.32:0/1388338438 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "alpha", "val": "0"}]=0 v349) v1 ==== 117+0+0 (secure 0 0 0) 0x7f0364030f00 con 0x7f036c0953a0 2021-04-09T18:31:25.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.695 7f0372142700 1 -- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f034c0188c0 msgr2=0x7f034c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:25.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f034c0188c0 0x7f034c01adc0 crc :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 -- 172.21.15.32:0/1388338438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f036c0953a0 msgr2=0x7f036c1890b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f036c0953a0 0x7f036c1890b0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7f0364006e60 tx=0x7f036400bcf0).stop 2021-04-09T18:31:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 -- 172.21.15.32:0/1388338438 shutdown_connections 2021-04-09T18:31:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f034c0188c0 0x7f034c01adc0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f036c0953a0 0x7f036c1890b0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f036c09bdd0 0x7f036c094d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 --2- 172.21.15.32:0/1388338438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f036c19e290 0x7f036c09b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 -- 172.21.15.32:0/1388338438 >> 172.21.15.32:0/1388338438 conn(0x7f036c09fa50 msgr2=0x7f036c0a2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 -- 172.21.15.32:0/1388338438 shutdown_connections 2021-04-09T18:31:25.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.696 7f0372142700 1 -- 172.21.15.32:0/1388338438 wait complete. 2021-04-09T18:31:25.722 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:25.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.860 7f57f670b700 1 Processor -- start 2021-04-09T18:31:25.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.860 7f57f670b700 1 -- start start 2021-04-09T18:31:25.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57f670b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f57f010d010 0x7f57f00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:25.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57f670b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57f00d1da0 0x7f57f00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:25.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57f670b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f010d480 con 0x7f57f00d82c0 2021-04-09T18:31:25.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57f670b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f00d8d30 con 0x7f57f00d1da0 2021-04-09T18:31:25.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57f670b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f00d9220 con 0x7f57f010d010 2021-04-09T18:31:25.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57e7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57f00d1da0 0x7f57f00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:25.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57f4d28700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f57f00d82c0 0x7f57f010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38198/0 (socket says 172.21.15.32:38198) 2021-04-09T18:31:25.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57f4d28700 1 -- 172.21.15.32:0/564179121 learned_addr learned my addr 172.21.15.32:0/564179121 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:25.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57effff700 1 --2- 172.21.15.32:0/564179121 >> v2:172.21.15.86:3300/0 conn(0x7f57f010d010 0x7f57f00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:25.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57e7fff700 1 -- 172.21.15.32:0/564179121 >> v2:172.21.15.86:3300/0 conn(0x7f57f010d010 msgr2=0x7f57f00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:25.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57e7fff700 1 --2- 172.21.15.32:0/564179121 >> v2:172.21.15.86:3300/0 conn(0x7f57f010d010 0x7f57f00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57e7fff700 1 -- 172.21.15.32:0/564179121 >> v1:172.21.15.32:6789/0 conn(0x7f57f00d82c0 legacy=0x7f57f010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57e7fff700 1 -- 172.21.15.32:0/564179121 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f00d86f0 con 0x7f57f00d1da0 2021-04-09T18:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57ef7fe700 1 -- 172.21.15.32:0/564179121 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477625990 0 0) 0x7f57e8000d30 con 0x7f57f00d82c0 2021-04-09T18:31:25.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.861 7f57e7fff700 1 --2- 172.21.15.32:0/564179121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f00d1da0 0x7f57f00d7ca0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 rx=0x7f57d8006650 tx=0x7f57d8009a00).ready entity=mon.0 client_cookie=c70ef1762fe48ac8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57ef7fe700 1 -- 172.21.15.32:0/564179121 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57d8009df0 con 0x7f57f00d1da0 2021-04-09T18:31:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57ef7fe700 1 -- 172.21.15.32:0/564179121 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d80009b0 con 0x7f57f00d1da0 2021-04-09T18:31:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57ef7fe700 1 -- 172.21.15.32:0/564179121 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57d800c2b0 con 0x7f57f00d1da0 2021-04-09T18:31:25.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57f670b700 1 -- 172.21.15.32:0/564179121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f00d1da0 msgr2=0x7f57f00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57f670b700 1 --2- 172.21.15.32:0/564179121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f00d1da0 0x7f57f00d7ca0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 rx=0x7f57d8006650 tx=0x7f57d8009a00).stop 2021-04-09T18:31:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57f670b700 1 -- 172.21.15.32:0/564179121 shutdown_connections 2021-04-09T18:31:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57f670b700 1 --2- 172.21.15.32:0/564179121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f00d1da0 0x7f57f00d7ca0 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57f670b700 1 --2- 172.21.15.32:0/564179121 >> v2:172.21.15.86:3300/0 conn(0x7f57f010d010 0x7f57f00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57f670b700 1 -- 172.21.15.32:0/564179121 >> 172.21.15.32:0/564179121 conn(0x7f57f00ce5f0 msgr2=0x7f57f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57f670b700 1 -- 172.21.15.32:0/564179121 shutdown_connections 2021-04-09T18:31:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.862 7f57f670b700 1 -- 172.21.15.32:0/564179121 wait complete. 2021-04-09T18:31:25.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57f670b700 1 Processor -- start 2021-04-09T18:31:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57f670b700 1 -- start start 2021-04-09T18:31:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57f670b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57f0185060 0x7f57f0185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57f670b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f0186c00 0x7f57f0187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:25.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57f670b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0191360 0x7f57f0191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57f670b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f01863a0 con 0x7f57f0186c00 2021-04-09T18:31:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57f670b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f0185a60 con 0x7f57f0191360 2021-04-09T18:31:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57f670b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f0187620 con 0x7f57f0185060 2021-04-09T18:31:25.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57e7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f0186c00 0x7f57f0187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57e7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f0186c00 0x7f57f0187000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56848/0 (socket says 172.21.15.32:56848) 2021-04-09T18:31:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57e7fff700 1 -- 172.21.15.32:0/360838906 learned_addr learned my addr 172.21.15.32:0/360838906 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57f4d28700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0191360 0x7f57f0191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:25.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.883 7f57effff700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57f0185060 0x7f57f0185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57e7fff700 1 -- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57f0185060 msgr2=0x7f57f0185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57e7fff700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57f0185060 0x7f57f0185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57e7fff700 1 -- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0191360 msgr2=0x7f57f0191760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57e7fff700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0191360 0x7f57f0191760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:25.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57e7fff700 1 -- 172.21.15.32:0/360838906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f0185460 con 0x7f57f0186c00 2021-04-09T18:31:25.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57e7fff700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f0186c00 0x7f57f0187000 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 rx=0x7f57d800b580 tx=0x7f57d800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:25.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57ed7fa700 1 -- 172.21.15.32:0/360838906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57d8000c90 con 0x7f57f0186c00 2021-04-09T18:31:25.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57ed7fa700 1 -- 172.21.15.32:0/360838906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d8002030 con 0x7f57f0186c00 2021-04-09T18:31:25.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57ed7fa700 1 -- 172.21.15.32:0/360838906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57d8002480 con 0x7f57f0186c00 2021-04-09T18:31:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57f670b700 1 -- 172.21.15.32:0/360838906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f0185460 con 0x7f57f0186c00 2021-04-09T18:31:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.884 7f57f670b700 1 -- 172.21.15.32:0/360838906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f0185460 con 0x7f57f0186c00 2021-04-09T18:31:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.885 7f57ed7fa700 1 -- 172.21.15.32:0/360838906 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f57d8002210 con 0x7f57f0186c00 2021-04-09T18:31:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.885 7f57ed7fa700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57d40188d0 0x7f57d401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.885 7f57ed7fa700 1 -- 172.21.15.32:0/360838906 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f57d8032a40 con 0x7f57f0186c00 2021-04-09T18:31:25.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.885 7f57effff700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57d40188d0 0x7f57d401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:25.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.885 7f57effff700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57d40188d0 0x7f57d401add0 crc :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.905 7f57f670b700 1 -- 172.21.15.32:0/360838906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f01866d0 con 0x7f57f0186c00 2021-04-09T18:31:25.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:25.908 7f57ed7fa700 1 -- 172.21.15.32:0/360838906 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f57d80009c0 con 0x7f57f0186c00 2021-04-09T18:31:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.099 7f57f670b700 1 -- 172.21.15.32:0/360838906 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57f01866d0 con 0x7f57f0186c00 2021-04-09T18:31:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.099 7f57ed7fa700 1 -- 172.21.15.32:0/360838906 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 349 v349) v1 ==== 95+0+2788 (secure 0 0 0) 0x7f57d800bd10 con 0x7f57f0186c00 2021-04-09T18:31:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 349 2021-04-09T18:31:26.116 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:26.116 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":349,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/4017196651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":4017196651},{"type":"v1","addr":"172.21.15.86:6837","nonce":4017196651}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":345},{"gid":11481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2104377376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2104377376},{"type":"v1","addr":"172.21.15.32:6833","nonce":2104377376}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":345},{"gid":11502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4153029254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4153029254},{"type":"v1","addr":"172.21.15.32:6835","nonce":4153029254}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":345}],"filesystems":[{"mdsmap":{"epoch":349,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:25.650475","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11020},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11020":{"gid":11020,"name":"d","rank":0,"incarnation":346,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/1129605860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1129605860},{"type":"v1","addr":"172.21.15.86:6836","nonce":1129605860}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17}]} 2021-04-09T18:31:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 -- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57d40188d0 msgr2=0x7f57d401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57d40188d0 0x7f57d401add0 crc :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 -- 172.21.15.32:0/360838906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f0186c00 msgr2=0x7f57f0187000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f0186c00 0x7f57f0187000 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 rx=0x7f57d800b580 tx=0x7f57d800b920).stop 2021-04-09T18:31:26.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 -- 172.21.15.32:0/360838906 shutdown_connections 2021-04-09T18:31:26.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57d40188d0 0x7f57d401add0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57f0186c00 0x7f57f0187000 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0191360 0x7f57f0191760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 --2- 172.21.15.32:0/360838906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57f0185060 0x7f57f0185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 -- 172.21.15.32:0/360838906 >> 172.21.15.32:0/360838906 conn(0x7f57f01837d0 msgr2=0x7f57f0195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:26.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 -- 172.21.15.32:0/360838906 shutdown_connections 2021-04-09T18:31:26.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.102 7f57f670b700 1 -- 172.21.15.32:0/360838906 wait complete. 2021-04-09T18:31:26.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:31:26.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.261 7f3400c75700 1 Processor -- start 2021-04-09T18:31:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.261 7f3400c75700 1 -- start start 2021-04-09T18:31:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f3400c75700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33fc0d7ca0 0x7f33fc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f3400c75700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc10d480 con 0x7f33fc10da00 2021-04-09T18:31:26.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f3400c75700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc0d86c0 con 0x7f33fc10d010 2021-04-09T18:31:26.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f3400c75700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33fc0d9220 con 0x7f33fc0d7ca0 2021-04-09T18:31:26.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33fb05f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f33fc10da00 0x7f33fc10de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38212/0 (socket says 172.21.15.32:38212) 2021-04-09T18:31:26.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33fb05f700 1 -- 172.21.15.32:0/975195839 learned_addr learned my addr 172.21.15.32:0/975195839 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:26.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33fa05d700 1 --2- 172.21.15.32:0/975195839 >> v2:172.21.15.86:3300/0 conn(0x7f33fc0d7ca0 0x7f33fc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33f985c700 1 -- 172.21.15.32:0/975195839 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707318064 0 0) 0x7f33f4000d30 con 0x7f33fc10da00 2021-04-09T18:31:26.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33f985c700 1 -- 172.21.15.32:0/975195839 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e8002fe0 con 0x7f33fc10da00 2021-04-09T18:31:26.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33f985c700 1 -- 172.21.15.32:0/975195839 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803346859 0 0) 0x7f33ec000c60 con 0x7f33fc10d010 2021-04-09T18:31:26.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33f985c700 1 -- 172.21.15.32:0/975195839 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e8003210 con 0x7f33fc10d010 2021-04-09T18:31:26.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33fa05d700 1 -- 172.21.15.32:0/975195839 >> v1:172.21.15.86:6789/0 conn(0x7f33fc10d010 legacy=0x7f33fc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33fa05d700 1 -- 172.21.15.32:0/975195839 >> v1:172.21.15.32:6789/0 conn(0x7f33fc10da00 legacy=0x7f33fc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.262 7f33fa05d700 1 -- 172.21.15.32:0/975195839 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc112100 con 0x7f33fc0d7ca0 2021-04-09T18:31:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.263 7f33fa05d700 1 --2- 172.21.15.32:0/975195839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33fc0d7ca0 0x7f33fc0d80a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f33e4006650 tx=0x7f33e4009a80).ready entity=mon.1 client_cookie=639256872a95c667 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.263 7f33f985c700 1 -- 172.21.15.32:0/975195839 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33e40009b0 con 0x7f33fc0d7ca0 2021-04-09T18:31:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.263 7f33f985c700 1 -- 172.21.15.32:0/975195839 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e400c0c0 con 0x7f33fc0d7ca0 2021-04-09T18:31:26.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.263 7f33f985c700 1 -- 172.21.15.32:0/975195839 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33e400c510 con 0x7f33fc0d7ca0 2021-04-09T18:31:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.263 7f3400c75700 1 -- 172.21.15.32:0/975195839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33fc0d7ca0 msgr2=0x7f33fc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.263 7f3400c75700 1 --2- 172.21.15.32:0/975195839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33fc0d7ca0 0x7f33fc0d80a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f33e4006650 tx=0x7f33e4009a80).stop 2021-04-09T18:31:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.264 7f3400c75700 1 -- 172.21.15.32:0/975195839 shutdown_connections 2021-04-09T18:31:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.264 7f3400c75700 1 --2- 172.21.15.32:0/975195839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33fc0d7ca0 0x7f33fc0d80a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.264 7f3400c75700 1 -- 172.21.15.32:0/975195839 >> 172.21.15.32:0/975195839 conn(0x7f33fc0ce5f0 msgr2=0x7f33fc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.264 7f3400c75700 1 -- 172.21.15.32:0/975195839 shutdown_connections 2021-04-09T18:31:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.264 7f3400c75700 1 -- 172.21.15.32:0/975195839 wait complete. 2021-04-09T18:31:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f3400c75700 1 Processor -- start 2021-04-09T18:31:26.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f3400c75700 1 -- start start 2021-04-09T18:31:26.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f3400c75700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33fc179610 0x7f33fc179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f3400c75700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33fc17a6e0 0x7f33fc17aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f3400c75700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33fc17b100 0x7f33fc08db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f3400c75700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc17d720 con 0x7f33fc17b100 2021-04-09T18:31:26.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f3400c75700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc185ba0 con 0x7f33fc179610 2021-04-09T18:31:26.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f3400c75700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc08e1b0 con 0x7f33fc17a6e0 2021-04-09T18:31:26.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f33fb05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33fc17b100 0x7f33fc08db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f33fb05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33fc17b100 0x7f33fc08db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56862/0 (socket says 172.21.15.32:56862) 2021-04-09T18:31:26.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.284 7f33fb05f700 1 -- 172.21.15.32:0/1855500574 learned_addr learned my addr 172.21.15.32:0/1855500574 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:26.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33fa05d700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33fc17a6e0 0x7f33fc17aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33fa85e700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33fc179610 0x7f33fc179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33fb05f700 1 -- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33fc17a6e0 msgr2=0x7f33fc17aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33fb05f700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33fc17a6e0 0x7f33fc17aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33fb05f700 1 -- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33fc179610 msgr2=0x7f33fc179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33fb05f700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33fc179610 0x7f33fc179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33fb05f700 1 -- 172.21.15.32:0/1855500574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc08e3d0 con 0x7f33fc17b100 2021-04-09T18:31:26.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33fb05f700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33fc17b100 0x7f33fc08db90 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 rx=0x7f33f4009980 tx=0x7f33f400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:26.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33f37fe700 1 -- 172.21.15.32:0/1855500574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f400be10 con 0x7f33fc17b100 2021-04-09T18:31:26.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33f37fe700 1 -- 172.21.15.32:0/1855500574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f4006030 con 0x7f33fc17b100 2021-04-09T18:31:26.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f3400c75700 1 -- 172.21.15.32:0/1855500574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc08e3d0 con 0x7f33fc17b100 2021-04-09T18:31:26.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f33f37fe700 1 -- 172.21.15.32:0/1855500574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f40064c0 con 0x7f33fc17b100 2021-04-09T18:31:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.285 7f3400c75700 1 -- 172.21.15.32:0/1855500574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc08e3d0 con 0x7f33fc17b100 2021-04-09T18:31:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.286 7f33f37fe700 1 -- 172.21.15.32:0/1855500574 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f33f4006210 con 0x7f33fc17b100 2021-04-09T18:31:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.286 7f33f37fe700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33e00188d0 0x7f33e001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.286 7f33f37fe700 1 -- 172.21.15.32:0/1855500574 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f33f4032d70 con 0x7f33fc17b100 2021-04-09T18:31:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.286 7f33fa85e700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33e00188d0 0x7f33e001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.286 7f33fa85e700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33e00188d0 0x7f33e001add0 crc :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.306 7f3400c75700 1 -- 172.21.15.32:0/1855500574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33fc17a1b0 con 0x7f33fc17b100 2021-04-09T18:31:26.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.309 7f33f37fe700 1 -- 172.21.15.32:0/1855500574 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f33f4000f90 con 0x7f33fc17b100 2021-04-09T18:31:26.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.500 7f3400c75700 1 -- 172.21.15.32:0/1855500574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33fc17a1b0 con 0x7f33fc17b100 2021-04-09T18:31:26.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.501 7f33f37fe700 1 -- 172.21.15.32:0/1855500574 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v203) v1 ==== 74+0+14003 (secure 0 0 0) 0x7f33f4006030 con 0x7f33fc17b100 2021-04-09T18:31:26.517 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:26.517 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":203,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:31:24.514994","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:31:21.254184","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":35,"pool_name":"alpha_data","create_time":"2021-04-09 18:31:22.120246","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:31:26.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 -- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33e00188d0 msgr2=0x7f33e001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33e00188d0 0x7f33e001add0 crc :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 -- 172.21.15.32:0/1855500574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33fc17b100 msgr2=0x7f33fc08db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33fc17b100 0x7f33fc08db90 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 rx=0x7f33f4009980 tx=0x7f33f400bcd0).stop 2021-04-09T18:31:26.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 -- 172.21.15.32:0/1855500574 shutdown_connections 2021-04-09T18:31:26.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33e00188d0 0x7f33e001add0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33fc17b100 0x7f33fc08db90 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33fc17a6e0 0x7f33fc17aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 --2- 172.21.15.32:0/1855500574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33fc179610 0x7f33fc179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 -- 172.21.15.32:0/1855500574 >> 172.21.15.32:0/1855500574 conn(0x7f33fc190550 msgr2=0x7f33fc1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:26.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 -- 172.21.15.32:0/1855500574 shutdown_connections 2021-04-09T18:31:26.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.503 7f3400c75700 1 -- 172.21.15.32:0/1855500574 wait complete. 2021-04-09T18:31:26.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:31:26.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.662 7ff3f5204700 1 Processor -- start 2021-04-09T18:31:26.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.662 7ff3f5204700 1 -- start start 2021-04-09T18:31:26.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.663 7ff3f5204700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff3f010da00 0x7ff3f010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.663 7ff3f5204700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f010d480 con 0x7ff3f010d010 2021-04-09T18:31:26.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.663 7ff3f5204700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f00d1f70 con 0x7ff3f00d7ca0 2021-04-09T18:31:26.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.663 7ff3f5204700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f00d8d30 con 0x7ff3f010da00 2021-04-09T18:31:26.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.663 7ff3eed9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff3f010d010 0x7ff3f00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38224/0 (socket says 172.21.15.32:38224) 2021-04-09T18:31:26.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.663 7ff3eed9d700 1 -- 172.21.15.32:0/2544447174 learned_addr learned my addr 172.21.15.32:0/2544447174 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:26.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.663 7ff3ef59e700 1 --2- 172.21.15.32:0/2544447174 >> v2:172.21.15.86:3300/0 conn(0x7ff3f010da00 0x7ff3f010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.663 7ff3edd9b700 1 -- 172.21.15.32:0/2544447174 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704547251 0 0) 0x7ff3e0000ce0 con 0x7ff3f010d010 2021-04-09T18:31:26.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.663 7ff3edd9b700 1 -- 172.21.15.32:0/2544447174 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8002fe0 con 0x7ff3f010d010 2021-04-09T18:31:26.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.664 7ff3ef59e700 1 -- 172.21.15.32:0/2544447174 >> v1:172.21.15.86:6789/0 conn(0x7ff3f00d7ca0 legacy=0x7ff3f00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.664 7ff3ef59e700 1 -- 172.21.15.32:0/2544447174 >> v1:172.21.15.32:6789/0 conn(0x7ff3f010d010 legacy=0x7ff3f00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.664 7ff3ef59e700 1 -- 172.21.15.32:0/2544447174 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f00d9220 con 0x7ff3f010da00 2021-04-09T18:31:26.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.664 7ff3ef59e700 1 --2- 172.21.15.32:0/2544447174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f010da00 0x7ff3f010de00 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7ff3e8006e50 tx=0x7ff3e8009e70).ready entity=mon.1 client_cookie=5f4d5e5823a8c9a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:26.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.666 7ff3edd9b700 1 -- 172.21.15.32:0/2544447174 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e8006a20 con 0x7ff3f010da00 2021-04-09T18:31:26.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.666 7ff3edd9b700 1 -- 172.21.15.32:0/2544447174 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e8006790 con 0x7ff3f010da00 2021-04-09T18:31:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.666 7ff3edd9b700 1 -- 172.21.15.32:0/2544447174 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e800b700 con 0x7ff3f010da00 2021-04-09T18:31:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.666 7ff3f5204700 1 -- 172.21.15.32:0/2544447174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f010da00 msgr2=0x7ff3f010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.666 7ff3f5204700 1 --2- 172.21.15.32:0/2544447174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f010da00 0x7ff3f010de00 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7ff3e8006e50 tx=0x7ff3e8009e70).stop 2021-04-09T18:31:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.667 7ff3f5204700 1 -- 172.21.15.32:0/2544447174 shutdown_connections 2021-04-09T18:31:26.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.667 7ff3f5204700 1 --2- 172.21.15.32:0/2544447174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f010da00 0x7ff3f010de00 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.667 7ff3f5204700 1 -- 172.21.15.32:0/2544447174 >> 172.21.15.32:0/2544447174 conn(0x7ff3f00ce5f0 msgr2=0x7ff3f00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.667 7ff3f5204700 1 -- 172.21.15.32:0/2544447174 shutdown_connections 2021-04-09T18:31:26.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.667 7ff3f5204700 1 -- 172.21.15.32:0/2544447174 wait complete. 2021-04-09T18:31:26.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.701 7ff3f5204700 1 Processor -- start 2021-04-09T18:31:26.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.701 7ff3f5204700 1 -- start start 2021-04-09T18:31:26.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3f5204700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f00af100 0x7ff3f0090b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3f5204700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3f01868f0 0x7ff3f0185ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3f5204700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f01a9470 0x7ff3f01a9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3f5204700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f01860d0 con 0x7ff3f01868f0 2021-04-09T18:31:26.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3f5204700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0186cf0 con 0x7ff3f00af100 2021-04-09T18:31:26.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3f5204700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0195280 con 0x7ff3f01a9470 2021-04-09T18:31:26.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3ee59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3f01868f0 0x7ff3f0185ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3ee59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3f01868f0 0x7ff3f0185ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56872/0 (socket says 172.21.15.32:56872) 2021-04-09T18:31:26.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3ee59c700 1 -- 172.21.15.32:0/970009361 learned_addr learned my addr 172.21.15.32:0/970009361 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:26.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3eed9d700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f00af100 0x7ff3f0090b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3ef59e700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f01a9470 0x7ff3f01a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3ee59c700 1 -- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f01a9470 msgr2=0x7ff3f01a9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3ee59c700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f01a9470 0x7ff3f01a9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3ee59c700 1 -- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f00af100 msgr2=0x7ff3f0090b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3ee59c700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f00af100 0x7ff3f0090b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.702 7ff3ee59c700 1 -- 172.21.15.32:0/970009361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f009d9d0 con 0x7ff3f01868f0 2021-04-09T18:31:26.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.703 7ff3ee59c700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3f01868f0 0x7ff3f0185ab0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 rx=0x7ff3e400b530 tx=0x7ff3e400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:26.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.703 7ff3df7fe700 1 -- 172.21.15.32:0/970009361 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e400bcc0 con 0x7ff3f01868f0 2021-04-09T18:31:26.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.703 7ff3df7fe700 1 -- 172.21.15.32:0/970009361 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e4006020 con 0x7ff3f01868f0 2021-04-09T18:31:26.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.703 7ff3df7fe700 1 -- 172.21.15.32:0/970009361 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e40054a0 con 0x7ff3f01868f0 2021-04-09T18:31:26.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.703 7ff3f5204700 1 -- 172.21.15.32:0/970009361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f009d9d0 con 0x7ff3f01868f0 2021-04-09T18:31:26.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.703 7ff3f5204700 1 -- 172.21.15.32:0/970009361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f009d9d0 con 0x7ff3f01868f0 2021-04-09T18:31:26.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.703 7ff3df7fe700 1 -- 172.21.15.32:0/970009361 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff3e4006200 con 0x7ff3f01868f0 2021-04-09T18:31:26.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.704 7ff3df7fe700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3d4018930 0x7ff3d401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:26.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.704 7ff3df7fe700 1 -- 172.21.15.32:0/970009361 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff3e4032a30 con 0x7ff3f01868f0 2021-04-09T18:31:26.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.704 7ff3eed9d700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3d4018930 0x7ff3d401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:26.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.704 7ff3eed9d700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3d4018930 0x7ff3d401ae10 crc :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:26.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.707 7ff3f5204700 1 -- 172.21.15.32:0/970009361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f010d480 con 0x7ff3f01868f0 2021-04-09T18:31:26.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.711 7ff3df7fe700 1 -- 172.21.15.32:0/970009361 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff3e400bcc0 con 0x7ff3f01868f0 2021-04-09T18:31:26.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.900 7ff3f5204700 1 -- 172.21.15.32:0/970009361 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3f010d480 con 0x7ff3f01868f0 2021-04-09T18:31:26.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.901 7ff3df7fe700 1 -- 172.21.15.32:0/970009361 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v203) v1 ==== 74+0+14003 (secure 0 0 0) 0x7ff3e4006020 con 0x7ff3f01868f0 2021-04-09T18:31:26.916 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:26.916 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":203,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:31:24.514994","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:31:21.254184","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":35,"pool_name":"alpha_data","create_time":"2021-04-09 18:31:22.120246","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:31:26.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 -- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3d4018930 msgr2=0x7ff3d401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3d4018930 0x7ff3d401ae10 crc :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 -- 172.21.15.32:0/970009361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3f01868f0 msgr2=0x7ff3f0185ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:26.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3f01868f0 0x7ff3f0185ab0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 rx=0x7ff3e400b530 tx=0x7ff3e400b8d0).stop 2021-04-09T18:31:26.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 -- 172.21.15.32:0/970009361 shutdown_connections 2021-04-09T18:31:26.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3d4018930 0x7ff3d401ae10 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3f01868f0 0x7ff3f0185ab0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f01a9470 0x7ff3f01a9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 --2- 172.21.15.32:0/970009361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f00af100 0x7ff3f0090b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:26.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 -- 172.21.15.32:0/970009361 >> 172.21.15.32:0/970009361 conn(0x7ff3f01965c0 msgr2=0x7ff3f01ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:26.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 -- 172.21.15.32:0/970009361 shutdown_connections 2021-04-09T18:31:26.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:26.903 7ff3f5204700 1 -- 172.21.15.32:0/970009361 wait complete. 2021-04-09T18:31:26.928 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 34, 'pool_name': 'alpha_metadata', 'create_time': '2021-04-09 18:31:21.254184', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '202', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'alpha'}}}, {'pool': 35, 'pool_name': 'alpha_data', 'create_time': '2021-04-09 18:31:22.120246', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '203', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'alpha'}}}] 2021-04-09T18:31:26.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:31:27.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.065 7fb1e3a3c700 1 Processor -- start 2021-04-09T18:31:27.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.065 7fb1e3a3c700 1 -- start start 2021-04-09T18:31:27.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e3a3c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb1dc10d010 0x7fb1dc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e3a3c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc0d1da0 0x7fb1dc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e3a3c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc10d480 con 0x7fb1dc0d82c0 2021-04-09T18:31:27.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e3a3c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1dc0d9220 con 0x7fb1dc10d010 2021-04-09T18:31:27.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e3a3c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc0d86c0 con 0x7fb1dc0d1da0 2021-04-09T18:31:27.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e1858700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb1dc10d010 0x7fb1dc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e2059700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb1dc0d82c0 0x7fb1dc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38238/0 (socket says 172.21.15.32:38238) 2021-04-09T18:31:27.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e2059700 1 -- 172.21.15.32:0/712072755 learned_addr learned my addr 172.21.15.32:0/712072755 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:27.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e1057700 1 --2- 172.21.15.32:0/712072755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc0d1da0 0x7fb1dc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e1858700 1 -- 172.21.15.32:0/712072755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc0d1da0 msgr2=0x7fb1dc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:27.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e1858700 1 --2- 172.21.15.32:0/712072755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc0d1da0 0x7fb1dc0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e1858700 1 -- 172.21.15.32:0/712072755 >> v1:172.21.15.32:6789/0 conn(0x7fb1dc0d82c0 legacy=0x7fb1dc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e0856700 1 -- 172.21.15.32:0/712072755 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653059130 0 0) 0x7fb1d4000d30 con 0x7fb1dc0d82c0 2021-04-09T18:31:27.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.066 7fb1e1858700 1 -- 172.21.15.32:0/712072755 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc10e1f0 con 0x7fb1dc10d010 2021-04-09T18:31:27.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.067 7fb1e1858700 1 --2- 172.21.15.32:0/712072755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc10d010 0x7fb1dc0d1780 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7fb1d8006650 tx=0x7fb1d8009a00).ready entity=mon.0 client_cookie=5b9e290e6abbae44 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:27.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.067 7fb1e0856700 1 -- 172.21.15.32:0/712072755 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1d8009df0 con 0x7fb1dc10d010 2021-04-09T18:31:27.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.067 7fb1e0856700 1 -- 172.21.15.32:0/712072755 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d80009b0 con 0x7fb1dc10d010 2021-04-09T18:31:27.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.067 7fb1e0856700 1 -- 172.21.15.32:0/712072755 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1d800c2b0 con 0x7fb1dc10d010 2021-04-09T18:31:27.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.067 7fb1e3a3c700 1 -- 172.21.15.32:0/712072755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc10d010 msgr2=0x7fb1dc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.067 7fb1e3a3c700 1 --2- 172.21.15.32:0/712072755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc10d010 0x7fb1dc0d1780 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7fb1d8006650 tx=0x7fb1d8009a00).stop 2021-04-09T18:31:27.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.067 7fb1e3a3c700 1 -- 172.21.15.32:0/712072755 shutdown_connections 2021-04-09T18:31:27.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.067 7fb1e3a3c700 1 --2- 172.21.15.32:0/712072755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc0d1da0 0x7fb1dc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.068 7fb1e3a3c700 1 --2- 172.21.15.32:0/712072755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc10d010 0x7fb1dc0d1780 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.068 7fb1e3a3c700 1 -- 172.21.15.32:0/712072755 >> 172.21.15.32:0/712072755 conn(0x7fb1dc0ce5f0 msgr2=0x7fb1dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:27.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.068 7fb1e3a3c700 1 -- 172.21.15.32:0/712072755 shutdown_connections 2021-04-09T18:31:27.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.068 7fb1e3a3c700 1 -- 172.21.15.32:0/712072755 wait complete. 2021-04-09T18:31:27.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.088 7fb1e3a3c700 1 Processor -- start 2021-04-09T18:31:27.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.088 7fb1e3a3c700 1 -- start start 2021-04-09T18:31:27.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.088 7fb1e3a3c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc09c5b0 0x7fb1dc09d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.088 7fb1e3a3c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1dc09c9b0 0x7fb1dc09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.088 7fb1e3a3c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc090390 0x7fb1dc090790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e3a3c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc09e290 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e3a3c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc09cf90 con 0x7fb1dc09c9b0 2021-04-09T18:31:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e3a3c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc090db0 con 0x7fb1dc090390 2021-04-09T18:31:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1858700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc09c5b0 0x7fb1dc09d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1858700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc09c5b0 0x7fb1dc09d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56884/0 (socket says 172.21.15.32:56884) 2021-04-09T18:31:27.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1858700 1 -- 172.21.15.32:0/2483396080 learned_addr learned my addr 172.21.15.32:0/2483396080 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:27.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e2059700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc090390 0x7fb1dc090790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1057700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1dc09c9b0 0x7fb1dc09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1858700 1 -- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc090390 msgr2=0x7fb1dc090790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1858700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc090390 0x7fb1dc090790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1858700 1 -- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1dc09c9b0 msgr2=0x7fb1dc09db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1858700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1dc09c9b0 0x7fb1dc09db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1858700 1 -- 172.21.15.32:0/2483396080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc090f90 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e1858700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc09c5b0 0x7fb1dc09d510 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 rx=0x7fb1d80096a0 tx=0x7fb1d800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:27.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1d27fc700 1 -- 172.21.15.32:0/2483396080 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1d80093a0 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1d27fc700 1 -- 172.21.15.32:0/2483396080 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d80067f0 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc090f90 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.089 7fb1d27fc700 1 -- 172.21.15.32:0/2483396080 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1d80022b0 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.090 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc090f90 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.090 7fb1d27fc700 1 -- 172.21.15.32:0/2483396080 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb1d80052e0 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.090 7fb1d27fc700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1c4018930 0x7fb1c401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.090 7fb1d27fc700 1 -- 172.21.15.32:0/2483396080 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb1d80329f0 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.090 7fb1e1057700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1c4018930 0x7fb1c401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.091 7fb1e1057700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1c4018930 0x7fb1c401ae10 crc :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:27.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.110 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc091b80 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.113 7fb1d27fc700 1 -- 172.21.15.32:0/2483396080 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb1d801f730 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.292 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fb1dc10d480 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.292 7fb1d27fc700 1 -- 172.21.15.32:0/2483396080 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v203) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb1d80052e0 con 0x7fb1dc09c5b0 2021-04-09T18:31:27.308 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:31:27.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1c4018930 msgr2=0x7fb1c401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1c4018930 0x7fb1c401ae10 crc :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc09c5b0 msgr2=0x7fb1dc09d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc09c5b0 0x7fb1dc09d510 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 rx=0x7fb1d80096a0 tx=0x7fb1d800b8e0).stop 2021-04-09T18:31:27.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 shutdown_connections 2021-04-09T18:31:27.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb1c4018930 0x7fb1c401ae10 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1dc090390 0x7fb1dc090790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1dc09c9b0 0x7fb1dc09db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 --2- 172.21.15.32:0/2483396080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1dc09c5b0 0x7fb1dc09d510 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 >> 172.21.15.32:0/2483396080 conn(0x7fb1dc0b2fa0 msgr2=0x7fb1dc0a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:27.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 shutdown_connections 2021-04-09T18:31:27.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.295 7fb1e3a3c700 1 -- 172.21.15.32:0/2483396080 wait complete. 2021-04-09T18:31:27.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_metadata pg_num 2021-04-09T18:31:27.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.458 7fab13169700 1 Processor -- start 2021-04-09T18:31:27.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.458 7fab13169700 1 -- start start 2021-04-09T18:31:27.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab13169700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fab0c10d010 0x7fab0c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab13169700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0d1da0 0x7fab0c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab13169700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab0c10d480 con 0x7fab0c0d82c0 2021-04-09T18:31:27.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab13169700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0c0d9220 con 0x7fab0c10d010 2021-04-09T18:31:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab13169700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c0d86c0 con 0x7fab0c0d1da0 2021-04-09T18:31:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab11786700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fab0c0d82c0 0x7fab0c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38250/0 (socket says 172.21.15.32:38250) 2021-04-09T18:31:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab11786700 1 -- 172.21.15.32:0/3051913369 learned_addr learned my addr 172.21.15.32:0/3051913369 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab0bfff700 1 --2- 172.21.15.32:0/3051913369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0d1da0 0x7fab0c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab10f85700 1 --2- 172.21.15.32:0/3051913369 >> v2:172.21.15.86:3300/0 conn(0x7fab0c10d010 0x7fab0c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab0b7fe700 1 -- 172.21.15.32:0/3051913369 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136618964 0 0) 0x7fab040009b0 con 0x7fab0c0d82c0 2021-04-09T18:31:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab0b7fe700 1 -- 172.21.15.32:0/3051913369 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf8002fe0 con 0x7fab0c0d82c0 2021-04-09T18:31:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab0bfff700 1 -- 172.21.15.32:0/3051913369 >> v2:172.21.15.86:3300/0 conn(0x7fab0c10d010 msgr2=0x7fab0c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab0bfff700 1 --2- 172.21.15.32:0/3051913369 >> v2:172.21.15.86:3300/0 conn(0x7fab0c10d010 0x7fab0c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab0bfff700 1 -- 172.21.15.32:0/3051913369 >> v1:172.21.15.32:6789/0 conn(0x7fab0c0d82c0 legacy=0x7fab0c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab0b7fe700 1 -- 172.21.15.32:0/3051913369 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 986801943 0 0) 0x7fab04000ef0 con 0x7fab0c0d82c0 2021-04-09T18:31:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.459 7fab0bfff700 1 -- 172.21.15.32:0/3051913369 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c10e1f0 con 0x7fab0c0d1da0 2021-04-09T18:31:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab0bfff700 1 --2- 172.21.15.32:0/3051913369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0d1da0 0x7fab0c0d7ca0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7faafc0069d0 tx=0x7faafc009dc0).ready entity=mon.2 client_cookie=d1a621529d1640e2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:27.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab0b7fe700 1 -- 172.21.15.32:0/3051913369 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faafc00b510 con 0x7fab0c0d1da0 2021-04-09T18:31:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab0b7fe700 1 -- 172.21.15.32:0/3051913369 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faafc006ac0 con 0x7fab0c0d1da0 2021-04-09T18:31:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab0b7fe700 1 -- 172.21.15.32:0/3051913369 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faafc00ba80 con 0x7fab0c0d1da0 2021-04-09T18:31:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab13169700 1 -- 172.21.15.32:0/3051913369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0d1da0 msgr2=0x7fab0c0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab13169700 1 --2- 172.21.15.32:0/3051913369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0d1da0 0x7fab0c0d7ca0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7faafc0069d0 tx=0x7faafc009dc0).stop 2021-04-09T18:31:27.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab13169700 1 -- 172.21.15.32:0/3051913369 shutdown_connections 2021-04-09T18:31:27.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab13169700 1 --2- 172.21.15.32:0/3051913369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0d1da0 0x7fab0c0d7ca0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab13169700 1 --2- 172.21.15.32:0/3051913369 >> v2:172.21.15.86:3300/0 conn(0x7fab0c10d010 0x7fab0c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.460 7fab13169700 1 -- 172.21.15.32:0/3051913369 >> 172.21.15.32:0/3051913369 conn(0x7fab0c0ce5f0 msgr2=0x7fab0c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:27.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.461 7fab13169700 1 -- 172.21.15.32:0/3051913369 shutdown_connections 2021-04-09T18:31:27.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.461 7fab13169700 1 -- 172.21.15.32:0/3051913369 wait complete. 2021-04-09T18:31:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.481 7fab13169700 1 Processor -- start 2021-04-09T18:31:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.481 7fab13169700 1 -- start start 2021-04-09T18:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.481 7fab13169700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c0a0110 0x7fab0c18c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.481 7fab13169700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0a0510 0x7fab0c0c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.481 7fab13169700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0c36e0 0x7fab0c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.481 7fab13169700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c18cf60 con 0x7fab0c0a0110 2021-04-09T18:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.481 7fab13169700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c199ea0 con 0x7fab0c0a0510 2021-04-09T18:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.481 7fab13169700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c187d70 con 0x7fab0c0c36e0 2021-04-09T18:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab10f85700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c0a0110 0x7fab0c18c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab10f85700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c0a0110 0x7fab0c18c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56896/0 (socket says 172.21.15.32:56896) 2021-04-09T18:31:27.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab10f85700 1 -- 172.21.15.32:0/2063608101 learned_addr learned my addr 172.21.15.32:0/2063608101 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab11786700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0c36e0 0x7fab0c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab0bfff700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0a0510 0x7fab0c0c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab10f85700 1 -- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0c36e0 msgr2=0x7fab0c0c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab10f85700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0c36e0 0x7fab0c0c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab10f85700 1 -- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0a0510 msgr2=0x7fab0c0c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab10f85700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0a0510 0x7fab0c0c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab10f85700 1 -- 172.21.15.32:0/2063608101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c18d4e0 con 0x7fab0c0a0110 2021-04-09T18:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab10f85700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c0a0110 0x7fab0c18c750 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7faaf4006f60 tx=0x7faaf400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:27.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab097fa700 1 -- 172.21.15.32:0/2063608101 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaf400bd60 con 0x7fab0c0a0110 2021-04-09T18:31:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab097fa700 1 -- 172.21.15.32:0/2063608101 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf4000c90 con 0x7fab0c0a0110 2021-04-09T18:31:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab13169700 1 -- 172.21.15.32:0/2063608101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0c18d4e0 con 0x7fab0c0a0110 2021-04-09T18:31:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab097fa700 1 -- 172.21.15.32:0/2063608101 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaf4002220 con 0x7fab0c0a0110 2021-04-09T18:31:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.482 7fab13169700 1 -- 172.21.15.32:0/2063608101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0c18d4e0 con 0x7fab0c0a0110 2021-04-09T18:31:27.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.483 7fab097fa700 1 -- 172.21.15.32:0/2063608101 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faaf40052e0 con 0x7fab0c0a0110 2021-04-09T18:31:27.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.483 7fab097fa700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faaf0018930 0x7faaf001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.483 7fab097fa700 1 -- 172.21.15.32:0/2063608101 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faaf40329f0 con 0x7fab0c0a0110 2021-04-09T18:31:27.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.483 7fab0bfff700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faaf0018930 0x7faaf001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.483 7fab0bfff700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faaf0018930 0x7faaf001ae10 crc :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:27.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.503 7fab13169700 1 -- 172.21.15.32:0/2063608101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0c10d480 con 0x7fab0c0a0110 2021-04-09T18:31:27.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.506 7fab097fa700 1 -- 172.21.15.32:0/2063608101 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faaf4000c90 con 0x7fab0c0a0110 2021-04-09T18:31:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.687 7fab13169700 1 -- 172.21.15.32:0/2063608101 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"} v 0) v1 -- 0x7fab0c10d480 con 0x7fab0c0a0110 2021-04-09T18:31:27.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.687 7fab097fa700 1 -- 172.21.15.32:0/2063608101 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"}]=0 v203) v1 ==== 103+0+10 (secure 0 0 0) 0x7faaf4006b00 con 0x7fab0c0a0110 2021-04-09T18:31:27.703 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:31:27.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 -- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faaf0018930 msgr2=0x7faaf001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faaf0018930 0x7faaf001ae10 crc :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 -- 172.21.15.32:0/2063608101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c0a0110 msgr2=0x7fab0c18c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c0a0110 0x7fab0c18c750 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7faaf4006f60 tx=0x7faaf400b970).stop 2021-04-09T18:31:27.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 -- 172.21.15.32:0/2063608101 shutdown_connections 2021-04-09T18:31:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faaf0018930 0x7faaf001ae10 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab0c0c36e0 0x7fab0c0c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0a0510 0x7fab0c0c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 --2- 172.21.15.32:0/2063608101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab0c0a0110 0x7fab0c18c750 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 -- 172.21.15.32:0/2063608101 >> 172.21.15.32:0/2063608101 conn(0x7fab0c0b93a0 msgr2=0x7fab0c0b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 -- 172.21.15.32:0/2063608101 shutdown_connections 2021-04-09T18:31:27.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.690 7fab13169700 1 -- 172.21.15.32:0/2063608101 wait complete. 2021-04-09T18:31:27.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_data pg_num 2021-04-09T18:31:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.853 7f0489140700 1 Processor -- start 2021-04-09T18:31:27.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.853 7f0489140700 1 -- start start 2021-04-09T18:31:27.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.854 7f0489140700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f048410d010 0x7f04840d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.854 7f0489140700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f04840d1da0 0x7f04840d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.854 7f0489140700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048410d480 con 0x7f04840d82c0 2021-04-09T18:31:27.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.854 7f0489140700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04840d8d30 con 0x7f048410d010 2021-04-09T18:31:27.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.854 7f0489140700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04840d9220 con 0x7f04840d1da0 2021-04-09T18:31:27.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.854 7f0482d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f048410d010 0x7f04840d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.854 7f0482d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f048410d010 0x7f04840d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56904/0 (socket says 172.21.15.32:56904) 2021-04-09T18:31:27.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.854 7f0482d9d700 1 -- 172.21.15.32:0/1194602408 learned_addr learned my addr 172.21.15.32:0/1194602408 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:27.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.854 7f048259c700 1 --2- 172.21.15.32:0/1194602408 >> v2:172.21.15.86:3300/0 conn(0x7f04840d1da0 0x7f04840d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0482d9d700 1 -- 172.21.15.32:0/1194602408 >> v2:172.21.15.86:3300/0 conn(0x7f04840d1da0 msgr2=0x7f04840d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:27.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0482d9d700 1 --2- 172.21.15.32:0/1194602408 >> v2:172.21.15.86:3300/0 conn(0x7f04840d1da0 0x7f04840d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0482d9d700 1 -- 172.21.15.32:0/1194602408 >> v1:172.21.15.86:6789/0 conn(0x7f04840d82c0 legacy=0x7f048410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0482d9d700 1 -- 172.21.15.32:0/1194602408 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04840d86f0 con 0x7f048410d010 2021-04-09T18:31:27.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0482d9d700 1 --2- 172.21.15.32:0/1194602408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048410d010 0x7f04840d1780 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 rx=0x7f0474006700 tx=0x7f0474009b80).ready entity=mon.0 client_cookie=102beb3926107bb7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:27.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0481d9b700 1 -- 172.21.15.32:0/1194602408 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f047400c0c0 con 0x7f048410d010 2021-04-09T18:31:27.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0481d9b700 1 -- 172.21.15.32:0/1194602408 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047400c440 con 0x7f048410d010 2021-04-09T18:31:27.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0481d9b700 1 -- 172.21.15.32:0/1194602408 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f047400c920 con 0x7f048410d010 2021-04-09T18:31:27.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0489140700 1 -- 172.21.15.32:0/1194602408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048410d010 msgr2=0x7f04840d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.855 7f0489140700 1 --2- 172.21.15.32:0/1194602408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048410d010 0x7f04840d1780 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 rx=0x7f0474006700 tx=0x7f0474009b80).stop 2021-04-09T18:31:27.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.856 7f0489140700 1 -- 172.21.15.32:0/1194602408 shutdown_connections 2021-04-09T18:31:27.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.856 7f0489140700 1 --2- 172.21.15.32:0/1194602408 >> v2:172.21.15.86:3300/0 conn(0x7f04840d1da0 0x7f04840d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.856 7f0489140700 1 --2- 172.21.15.32:0/1194602408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048410d010 0x7f04840d1780 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.856 7f0489140700 1 -- 172.21.15.32:0/1194602408 >> 172.21.15.32:0/1194602408 conn(0x7f04840ce5f0 msgr2=0x7f04840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:27.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.856 7f0489140700 1 -- 172.21.15.32:0/1194602408 shutdown_connections 2021-04-09T18:31:27.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.856 7f0489140700 1 -- 172.21.15.32:0/1194602408 wait complete. 2021-04-09T18:31:27.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.876 7f0489140700 1 Processor -- start 2021-04-09T18:31:27.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.876 7f0489140700 1 -- start start 2021-04-09T18:31:27.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.876 7f0489140700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048409e530 0x7f048409b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.876 7f0489140700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048409e930 0x7f048409f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f0489140700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048409f990 0x7f0484094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f0489140700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048409edf0 con 0x7f048409f990 2021-04-09T18:31:27.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f0489140700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048409fd90 con 0x7f048409e530 2021-04-09T18:31:27.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f0489140700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048409bd40 con 0x7f048409e930 2021-04-09T18:31:27.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048409f990 0x7f0484094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048359e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048409f990 0x7f0484094c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56914/0 (socket says 172.21.15.32:56914) 2021-04-09T18:31:27.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048359e700 1 -- 172.21.15.32:0/1039271830 learned_addr learned my addr 172.21.15.32:0/1039271830 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:27.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048259c700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048409e930 0x7f048409f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f0482d9d700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048409e530 0x7f048409b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048359e700 1 -- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048409e930 msgr2=0x7f048409f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048359e700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048409e930 0x7f048409f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048359e700 1 -- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048409e530 msgr2=0x7f048409b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:27.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048359e700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048409e530 0x7f048409b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048359e700 1 -- 172.21.15.32:0/1039271830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0484095290 con 0x7f048409f990 2021-04-09T18:31:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048259c700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048409e930 0x7f048409f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f0482d9d700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048409e530 0x7f048409b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:27.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f048359e700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048409f990 0x7f0484094c70 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 rx=0x7f047c009940 tx=0x7f047c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:27.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f04737fe700 1 -- 172.21.15.32:0/1039271830 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f047c000f40 con 0x7f048409f990 2021-04-09T18:31:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f04737fe700 1 -- 172.21.15.32:0/1039271830 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c00c2f0 con 0x7f048409f990 2021-04-09T18:31:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f04737fe700 1 -- 172.21.15.32:0/1039271830 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f047c00c7d0 con 0x7f048409f990 2021-04-09T18:31:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f0489140700 1 -- 172.21.15.32:0/1039271830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0484095290 con 0x7f048409f990 2021-04-09T18:31:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.877 7f0489140700 1 -- 172.21.15.32:0/1039271830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0484095290 con 0x7f048409f990 2021-04-09T18:31:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.878 7f04737fe700 1 -- 172.21.15.32:0/1039271830 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f047c001220 con 0x7f048409f990 2021-04-09T18:31:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.878 7f04737fe700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04680188d0 0x7f046801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.878 7f04737fe700 1 -- 172.21.15.32:0/1039271830 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f047c032d70 con 0x7f048409f990 2021-04-09T18:31:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.878 7f0482d9d700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04680188d0 0x7f046801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.879 7f0482d9d700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04680188d0 0x7f046801add0 crc :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:27.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.898 7f0489140700 1 -- 172.21.15.32:0/1039271830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0484095b10 con 0x7f048409f990 2021-04-09T18:31:27.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:27.902 7f04737fe700 1 -- 172.21.15.32:0/1039271830 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f047c00be30 con 0x7f048409f990 2021-04-09T18:31:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.082 7f0489140700 1 -- 172.21.15.32:0/1039271830 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"} v 0) v1 -- 0x7f0484095b10 con 0x7f048409f990 2021-04-09T18:31:28.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.082 7f04737fe700 1 -- 172.21.15.32:0/1039271830 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"}]=0 v203) v1 ==== 99+0+10 (secure 0 0 0) 0x7f047c001220 con 0x7f048409f990 2021-04-09T18:31:28.098 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:31:28.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 -- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04680188d0 msgr2=0x7f046801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04680188d0 0x7f046801add0 crc :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 -- 172.21.15.32:0/1039271830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048409f990 msgr2=0x7f0484094c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048409f990 0x7f0484094c70 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 rx=0x7f047c009940 tx=0x7f047c00bcf0).stop 2021-04-09T18:31:28.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 -- 172.21.15.32:0/1039271830 shutdown_connections 2021-04-09T18:31:28.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f04680188d0 0x7f046801add0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f048409f990 0x7f0484094c70 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f048409e930 0x7f048409f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 --2- 172.21.15.32:0/1039271830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f048409e530 0x7f048409b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 -- 172.21.15.32:0/1039271830 >> 172.21.15.32:0/1039271830 conn(0x7f04840c4290 msgr2=0x7f04840c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:28.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 -- 172.21.15.32:0/1039271830 shutdown_connections 2021-04-09T18:31:28.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.085 7f0489140700 1 -- 172.21.15.32:0/1039271830 wait complete. 2021-04-09T18:31:28.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:31:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.245 7f18658b6700 1 Processor -- start 2021-04-09T18:31:28.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.245 7f18658b6700 1 -- start start 2021-04-09T18:31:28.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.245 7f18658b6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f18600cffa0 0x7f18600d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.245 7f18658b6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18600d7800 0x7f18600d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.245 7f18658b6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18600d7280 con 0x7f18600d8220 2021-04-09T18:31:28.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.245 7f18658b6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18600d91e0 con 0x7f18600d7800 2021-04-09T18:31:28.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.245 7f18658b6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186010dfe0 con 0x7f18600cffa0 2021-04-09T18:31:28.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185ed73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f18600d7800 0x7f18600d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185fd75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f18600d8220 0x7f18600d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38278/0 (socket says 172.21.15.32:38278) 2021-04-09T18:31:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185fd75700 1 -- 172.21.15.32:0/517974053 learned_addr learned my addr 172.21.15.32:0/517974053 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185f574700 1 --2- 172.21.15.32:0/517974053 >> v2:172.21.15.86:3300/0 conn(0x7f18600cffa0 0x7f18600d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185ed73700 1 -- 172.21.15.32:0/517974053 >> v2:172.21.15.86:3300/0 conn(0x7f18600cffa0 msgr2=0x7f18600d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185ed73700 1 --2- 172.21.15.32:0/517974053 >> v2:172.21.15.86:3300/0 conn(0x7f18600cffa0 0x7f18600d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185ed73700 1 -- 172.21.15.32:0/517974053 >> v1:172.21.15.32:6789/0 conn(0x7f18600d8220 legacy=0x7f18600d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185ed73700 1 -- 172.21.15.32:0/517974053 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186010e200 con 0x7f18600d7800 2021-04-09T18:31:28.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185e572700 1 -- 172.21.15.32:0/517974053 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424136238 0 0) 0x7f1858000d30 con 0x7f18600d8220 2021-04-09T18:31:28.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185ed73700 1 --2- 172.21.15.32:0/517974053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18600d7800 0x7f18600d7c00 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 rx=0x7f1854006650 tx=0x7f1854009a00).ready entity=mon.0 client_cookie=284fa09c663677b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:28.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185e572700 1 -- 172.21.15.32:0/517974053 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1854009df0 con 0x7f18600d7800 2021-04-09T18:31:28.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185e572700 1 -- 172.21.15.32:0/517974053 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18540009b0 con 0x7f18600d7800 2021-04-09T18:31:28.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f185e572700 1 -- 172.21.15.32:0/517974053 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f185400c2b0 con 0x7f18600d7800 2021-04-09T18:31:28.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f18658b6700 1 -- 172.21.15.32:0/517974053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18600d7800 msgr2=0x7f18600d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.246 7f18658b6700 1 --2- 172.21.15.32:0/517974053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18600d7800 0x7f18600d7c00 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 rx=0x7f1854006650 tx=0x7f1854009a00).stop 2021-04-09T18:31:28.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.247 7f18658b6700 1 -- 172.21.15.32:0/517974053 shutdown_connections 2021-04-09T18:31:28.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.247 7f18658b6700 1 --2- 172.21.15.32:0/517974053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18600d7800 0x7f18600d7c00 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.247 7f18658b6700 1 --2- 172.21.15.32:0/517974053 >> v2:172.21.15.86:3300/0 conn(0x7f18600cffa0 0x7f18600d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.247 7f18658b6700 1 -- 172.21.15.32:0/517974053 >> 172.21.15.32:0/517974053 conn(0x7f18600ce5c0 msgr2=0x7f18600cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:28.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.247 7f18658b6700 1 -- 172.21.15.32:0/517974053 shutdown_connections 2021-04-09T18:31:28.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.247 7f18658b6700 1 -- 172.21.15.32:0/517974053 wait complete. 2021-04-09T18:31:28.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.267 7f18658b6700 1 Processor -- start 2021-04-09T18:31:28.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.267 7f18658b6700 1 -- start start 2021-04-09T18:31:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f18658b6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18601acff0 0x7f18601847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f18658b6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1860183fc0 0x7f18601a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f18658b6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1860186ff0 0x7f18601873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f18658b6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1860188c30 con 0x7f1860183fc0 2021-04-09T18:31:28.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f18658b6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18601843c0 con 0x7f1860186ff0 2021-04-09T18:31:28.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f18658b6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18601a5c20 con 0x7f18601acff0 2021-04-09T18:31:28.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185ed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1860183fc0 0x7f18601a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185ed73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1860183fc0 0x7f18601a5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56928/0 (socket says 172.21.15.32:56928) 2021-04-09T18:31:28.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185ed73700 1 -- 172.21.15.32:0/3590808019 learned_addr learned my addr 172.21.15.32:0/3590808019 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185fd75700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1860186ff0 0x7f18601873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185f574700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18601acff0 0x7f18601847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185ed73700 1 -- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18601acff0 msgr2=0x7f18601847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185ed73700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18601acff0 0x7f18601847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185ed73700 1 -- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1860186ff0 msgr2=0x7f18601873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185ed73700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1860186ff0 0x7f18601873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185ed73700 1 -- 172.21.15.32:0/3590808019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1860187a10 con 0x7f1860183fc0 2021-04-09T18:31:28.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185fd75700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1860186ff0 0x7f18601873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:28.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f185ed73700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1860183fc0 0x7f18601a5600 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 rx=0x7f18540096a0 tx=0x7f185400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:28.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f1847fff700 1 -- 172.21.15.32:0/3590808019 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1854000c90 con 0x7f1860183fc0 2021-04-09T18:31:28.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.268 7f1847fff700 1 -- 172.21.15.32:0/3590808019 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1854002030 con 0x7f1860183fc0 2021-04-09T18:31:28.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.269 7f18658b6700 1 -- 172.21.15.32:0/3590808019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1860187a10 con 0x7f1860183fc0 2021-04-09T18:31:28.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.269 7f1847fff700 1 -- 172.21.15.32:0/3590808019 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1854002550 con 0x7f1860183fc0 2021-04-09T18:31:28.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.269 7f18658b6700 1 -- 172.21.15.32:0/3590808019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1860187a10 con 0x7f1860183fc0 2021-04-09T18:31:28.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.269 7f1847fff700 1 -- 172.21.15.32:0/3590808019 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1854002210 con 0x7f1860183fc0 2021-04-09T18:31:28.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.269 7f1847fff700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18400188d0 0x7f184001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.269 7f1847fff700 1 -- 172.21.15.32:0/3590808019 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1854032a40 con 0x7f1860183fc0 2021-04-09T18:31:28.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.269 7f185f574700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18400188d0 0x7f184001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.270 7f185f574700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18400188d0 0x7f184001add0 crc :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:28.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.289 7f18658b6700 1 -- 172.21.15.32:0/3590808019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1860182dc0 con 0x7f1860183fc0 2021-04-09T18:31:28.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.293 7f1847fff700 1 -- 172.21.15.32:0/3590808019 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f18540009c0 con 0x7f1860183fc0 2021-04-09T18:31:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.470 7f18658b6700 1 -- 172.21.15.32:0/3590808019 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f18600d7280 con 0x7f1860183fc0 2021-04-09T18:31:28.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.470 7f1847fff700 1 -- 172.21.15.32:0/3590808019 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v203) v1 ==== 84+0+206 (secure 0 0 0) 0x7f185400bd80 con 0x7f1860183fc0 2021-04-09T18:31:28.486 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:28.486 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:31:28.486 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:28.487 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:31:28.487 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:31:28.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:28.487 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:28.488 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 34, 2021-04-09T18:31:28.488 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "alpha_metadata" 2021-04-09T18:31:28.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:31:28.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:31:28.489 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 35, 2021-04-09T18:31:28.489 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "alpha_data" 2021-04-09T18:31:28.489 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:31:28.489 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:31:28.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.472 7f18658b6700 1 -- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18400188d0 msgr2=0x7f184001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18400188d0 0x7f184001add0 crc :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 -- 172.21.15.32:0/3590808019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1860183fc0 msgr2=0x7f18601a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1860183fc0 0x7f18601a5600 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 rx=0x7f18540096a0 tx=0x7f185400b990).stop 2021-04-09T18:31:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 -- 172.21.15.32:0/3590808019 shutdown_connections 2021-04-09T18:31:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18400188d0 0x7f184001add0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1860183fc0 0x7f18601a5600 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1860186ff0 0x7f18601873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 --2- 172.21.15.32:0/3590808019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18601acff0 0x7f18601847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 -- 172.21.15.32:0/3590808019 >> 172.21.15.32:0/3590808019 conn(0x7f186009abb0 msgr2=0x7f18600a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 -- 172.21.15.32:0/3590808019 shutdown_connections 2021-04-09T18:31:28.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.473 7f18658b6700 1 -- 172.21.15.32:0/3590808019 wait complete. 2021-04-09T18:31:28.498 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'bravo' 2021-04-09T18:31:28.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create bravo_metadata 8 2021-04-09T18:31:28.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.636 7f647498b700 1 Processor -- start 2021-04-09T18:31:28.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.636 7f647498b700 1 -- start start 2021-04-09T18:31:28.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.637 7f647498b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f64700d7ca0 0x7f64700d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.637 7f647498b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f647010da00 0x7f647010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.637 7f647498b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647010d480 con 0x7f647010d010 2021-04-09T18:31:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.637 7f647498b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64700d9220 con 0x7f64700d7ca0 2021-04-09T18:31:28.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.637 7f647498b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64700d86c0 con 0x7f647010da00 2021-04-09T18:31:28.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.637 7f646e61c700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f647010d010 0x7f64700d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:36216/0 (socket says 172.21.15.32:36216) 2021-04-09T18:31:28.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.637 7f646e61c700 1 -- 172.21.15.32:0/629611456 learned_addr learned my addr 172.21.15.32:0/629611456 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:28.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.637 7f646ee1d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f647010da00 0x7f647010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.637 7f646de1b700 1 --2- 172.21.15.32:0/629611456 >> v2:172.21.15.86:3300/0 conn(0x7f64700d7ca0 0x7f64700d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646d61a700 1 -- 172.21.15.32:0/629611456 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486207977 0 0) 0x7f64600010c0 con 0x7f647010d010 2021-04-09T18:31:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646d61a700 1 -- 172.21.15.32:0/629611456 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6458002fe0 con 0x7f647010d010 2021-04-09T18:31:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646de1b700 1 -- 172.21.15.32:0/629611456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f647010da00 msgr2=0x7f647010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646de1b700 1 --2- 172.21.15.32:0/629611456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f647010da00 0x7f647010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646de1b700 1 -- 172.21.15.32:0/629611456 >> v1:172.21.15.86:6789/0 conn(0x7f647010d010 legacy=0x7f64700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646de1b700 1 -- 172.21.15.32:0/629611456 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700d1f70 con 0x7f64700d7ca0 2021-04-09T18:31:28.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646ee1d700 1 --2- 172.21.15.32:0/629611456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f647010da00 0x7f647010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:28.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646de1b700 1 --2- 172.21.15.32:0/629611456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64700d7ca0 0x7f64700d80a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7f6464006650 tx=0x7f6464009a80).ready entity=mon.1 client_cookie=2a0cc02b2e063bdc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:28.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646d61a700 1 -- 172.21.15.32:0/629611456 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64640009b0 con 0x7f64700d7ca0 2021-04-09T18:31:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646d61a700 1 -- 172.21.15.32:0/629611456 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646400c0c0 con 0x7f64700d7ca0 2021-04-09T18:31:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f646d61a700 1 -- 172.21.15.32:0/629611456 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f646400c510 con 0x7f64700d7ca0 2021-04-09T18:31:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f647498b700 1 -- 172.21.15.32:0/629611456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64700d7ca0 msgr2=0x7f64700d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.638 7f647498b700 1 --2- 172.21.15.32:0/629611456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64700d7ca0 0x7f64700d80a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7f6464006650 tx=0x7f6464009a80).stop 2021-04-09T18:31:28.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.639 7f647498b700 1 -- 172.21.15.32:0/629611456 shutdown_connections 2021-04-09T18:31:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.639 7f647498b700 1 --2- 172.21.15.32:0/629611456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f647010da00 0x7f647010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.639 7f647498b700 1 --2- 172.21.15.32:0/629611456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64700d7ca0 0x7f64700d80a0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.639 7f647498b700 1 -- 172.21.15.32:0/629611456 >> 172.21.15.32:0/629611456 conn(0x7f64700ce5f0 msgr2=0x7f64700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:28.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.639 7f647498b700 1 -- 172.21.15.32:0/629611456 shutdown_connections 2021-04-09T18:31:28.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.639 7f647498b700 1 -- 172.21.15.32:0/629611456 wait complete. 2021-04-09T18:31:28.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.659 7f647498b700 1 Processor -- start 2021-04-09T18:31:28.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.659 7f647498b700 1 -- start start 2021-04-09T18:31:28.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.659 7f647498b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f647017bb20 0x7f64700c1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f647498b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64700c1890 0x7f64700b9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f647498b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64700ba4b0 0x7f64700ba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f647498b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6470180680 con 0x7f64700c1890 2021-04-09T18:31:28.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f647498b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64700c0cf0 con 0x7f647017bb20 2021-04-09T18:31:28.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f647498b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647017c450 con 0x7f64700ba4b0 2021-04-09T18:31:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64700c1890 0x7f64700b9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64700c1890 0x7f64700b9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56940/0 (socket says 172.21.15.32:56940) 2021-04-09T18:31:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646de1b700 1 -- 172.21.15.32:0/1845912761 learned_addr learned my addr 172.21.15.32:0/1845912761 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:28.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646ee1d700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64700ba4b0 0x7f64700ba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646e61c700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f647017bb20 0x7f64700c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646ee1d700 1 -- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f647017bb20 msgr2=0x7f64700c1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646ee1d700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f647017bb20 0x7f64700c1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646ee1d700 1 -- 172.21.15.32:0/1845912761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64700c1890 msgr2=0x7f64700b9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:28.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646ee1d700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64700c1890 0x7f64700b9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:28.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646ee1d700 1 -- 172.21.15.32:0/1845912761 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700c1c90 con 0x7f64700ba4b0 2021-04-09T18:31:28.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646de1b700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64700c1890 0x7f64700b9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:28.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646e61c700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f647017bb20 0x7f64700c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:28.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f646ee1d700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64700ba4b0 0x7f64700ba8b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f646800b8a0 tx=0x7f646800bc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:28.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f645effd700 1 -- 172.21.15.32:0/1845912761 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6468006790 con 0x7f64700ba4b0 2021-04-09T18:31:28.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f645effd700 1 -- 172.21.15.32:0/1845912761 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468000f90 con 0x7f64700ba4b0 2021-04-09T18:31:28.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f647498b700 1 -- 172.21.15.32:0/1845912761 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64700c1c90 con 0x7f64700ba4b0 2021-04-09T18:31:28.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f645effd700 1 -- 172.21.15.32:0/1845912761 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f646800c570 con 0x7f64700ba4b0 2021-04-09T18:31:28.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.660 7f647498b700 1 -- 172.21.15.32:0/1845912761 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64700c1c90 con 0x7f64700ba4b0 2021-04-09T18:31:28.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.661 7f645effd700 1 -- 172.21.15.32:0/1845912761 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f64680011f0 con 0x7f64700ba4b0 2021-04-09T18:31:28.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.661 7f645effd700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64540188d0 0x7f645401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:28.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.661 7f645effd700 1 -- 172.21.15.32:0/1845912761 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6468032d40 con 0x7f64700ba4b0 2021-04-09T18:31:28.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.661 7f646e61c700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64540188d0 0x7f645401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:28.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.662 7f646e61c700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64540188d0 0x7f645401add0 crc :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:28.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.681 7f647498b700 1 -- 172.21.15.32:0/1845912761 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647018ec20 con 0x7f64700ba4b0 2021-04-09T18:31:28.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.685 7f645effd700 1 -- 172.21.15.32:0/1845912761 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6468000f70 con 0x7f64700ba4b0 2021-04-09T18:31:28.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:28.868 7f647498b700 1 -- 172.21.15.32:0/1845912761 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_metadata"} v 0) v1 -- 0x7f647018ec20 con 0x7f64700ba4b0 2021-04-09T18:31:29.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.674 7f645effd700 1 -- 172.21.15.32:0/1845912761 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_metadata"}]=0 pool 'bravo_metadata' created v204) v1 ==== 131+0+0 (secure 0 0 0) 0x7f6468006f20 con 0x7f64700ba4b0 2021-04-09T18:31:29.698 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_metadata' created 2021-04-09T18:31:29.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 -- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64540188d0 msgr2=0x7f645401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:29.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64540188d0 0x7f645401add0 crc :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:29.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 -- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64700ba4b0 msgr2=0x7f64700ba8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:29.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64700ba4b0 0x7f64700ba8b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f646800b8a0 tx=0x7f646800bc40).stop 2021-04-09T18:31:29.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 -- 172.21.15.32:0/1845912761 shutdown_connections 2021-04-09T18:31:29.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64540188d0 0x7f645401add0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64700c1890 0x7f64700b9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:29.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64700ba4b0 0x7f64700ba8b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 --2- 172.21.15.32:0/1845912761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f647017bb20 0x7f64700c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:29.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 -- 172.21.15.32:0/1845912761 >> 172.21.15.32:0/1845912761 conn(0x7f6470181390 msgr2=0x7f64700adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 -- 172.21.15.32:0/1845912761 shutdown_connections 2021-04-09T18:31:29.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.687 7f647498b700 1 -- 172.21.15.32:0/1845912761 wait complete. 2021-04-09T18:31:29.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create bravo_data 8 2021-04-09T18:31:29.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.856 7f7ff9513700 1 Processor -- start 2021-04-09T18:31:29.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.856 7f7ff9513700 1 -- start start 2021-04-09T18:31:29.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff9513700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff410d010 0x7f7ff40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:29.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff9513700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff410d480 con 0x7f7ff40d1da0 2021-04-09T18:31:29.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff9513700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff410e1f0 con 0x7f7ff40d8490 2021-04-09T18:31:29.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff9513700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff410e780 con 0x7f7ff410d010 2021-04-09T18:31:29.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff2abe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7ff40d1da0 0x7f7ff40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38308/0 (socket says 172.21.15.32:38308) 2021-04-09T18:31:29.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff2abe700 1 -- 172.21.15.32:0/2605293075 learned_addr learned my addr 172.21.15.32:0/2605293075 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:29.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff32bf700 1 --2- 172.21.15.32:0/2605293075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff410d010 0x7f7ff40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:29.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff22bd700 1 -- 172.21.15.32:0/2605293075 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377514438 0 0) 0x7f7fe8000ce0 con 0x7f7ff40d1da0 2021-04-09T18:31:29.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff22bd700 1 -- 172.21.15.32:0/2605293075 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fdc002fe0 con 0x7f7ff40d1da0 2021-04-09T18:31:29.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff22bd700 1 -- 172.21.15.32:0/2605293075 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633048220 0 0) 0x7f7fec000d30 con 0x7f7ff40d8490 2021-04-09T18:31:29.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff22bd700 1 -- 172.21.15.32:0/2605293075 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fdc0042c0 con 0x7f7ff40d8490 2021-04-09T18:31:29.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff32bf700 1 -- 172.21.15.32:0/2605293075 >> v1:172.21.15.86:6789/0 conn(0x7f7ff40d8490 legacy=0x7f7ff410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:29.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff32bf700 1 -- 172.21.15.32:0/2605293075 >> v1:172.21.15.32:6789/0 conn(0x7f7ff40d1da0 legacy=0x7f7ff40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:29.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.857 7f7ff32bf700 1 -- 172.21.15.32:0/2605293075 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff40d8d30 con 0x7f7ff410d010 2021-04-09T18:31:29.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.858 7f7ff32bf700 1 --2- 172.21.15.32:0/2605293075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff410d010 0x7f7ff40d1780 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7fe4006650 tx=0x7f7fe4009a80).ready entity=mon.2 client_cookie=4d201fa13b79dd3f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:29.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.858 7f7ff22bd700 1 -- 172.21.15.32:0/2605293075 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fe40009b0 con 0x7f7ff410d010 2021-04-09T18:31:29.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.858 7f7ff22bd700 1 -- 172.21.15.32:0/2605293075 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe400c0c0 con 0x7f7ff410d010 2021-04-09T18:31:29.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.858 7f7ff22bd700 1 -- 172.21.15.32:0/2605293075 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fe400c510 con 0x7f7ff410d010 2021-04-09T18:31:29.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.858 7f7ff9513700 1 -- 172.21.15.32:0/2605293075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff410d010 msgr2=0x7f7ff40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:29.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.858 7f7ff9513700 1 --2- 172.21.15.32:0/2605293075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff410d010 0x7f7ff40d1780 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7fe4006650 tx=0x7f7fe4009a80).stop 2021-04-09T18:31:29.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.859 7f7ff9513700 1 -- 172.21.15.32:0/2605293075 shutdown_connections 2021-04-09T18:31:29.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.859 7f7ff9513700 1 --2- 172.21.15.32:0/2605293075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff410d010 0x7f7ff40d1780 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:29.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.859 7f7ff9513700 1 -- 172.21.15.32:0/2605293075 >> 172.21.15.32:0/2605293075 conn(0x7f7ff40ce5f0 msgr2=0x7f7ff40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:29.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.859 7f7ff9513700 1 -- 172.21.15.32:0/2605293075 shutdown_connections 2021-04-09T18:31:29.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.859 7f7ff9513700 1 -- 172.21.15.32:0/2605293075 wait complete. 2021-04-09T18:31:29.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.879 7f7ff9513700 1 Processor -- start 2021-04-09T18:31:29.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.879 7f7ff9513700 1 -- start start 2021-04-09T18:31:29.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff9513700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff40b8de0 0x7f7ff40c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:29.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff9513700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40c0640 0x7f7ff41936d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:29.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff9513700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff4193cf0 0x7f7ff41940f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:29.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff9513700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff40c0a40 con 0x7f7ff4193cf0 2021-04-09T18:31:29.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff9513700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff40b89a0 con 0x7f7ff40b8de0 2021-04-09T18:31:29.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff9513700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff41b7b70 con 0x7f7ff40c0640 2021-04-09T18:31:29.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff3ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff4193cf0 0x7f7ff41940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:29.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff3ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff4193cf0 0x7f7ff41940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56956/0 (socket says 172.21.15.32:56956) 2021-04-09T18:31:29.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff3ac0700 1 -- 172.21.15.32:0/2273383341 learned_addr learned my addr 172.21.15.32:0/2273383341 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:29.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff2abe700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40c0640 0x7f7ff41936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:29.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.880 7f7ff32bf700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff40b8de0 0x7f7ff40c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:29.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7ff3ac0700 1 -- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40c0640 msgr2=0x7f7ff41936d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:29.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7ff3ac0700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40c0640 0x7f7ff41936d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:29.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7ff3ac0700 1 -- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff40b8de0 msgr2=0x7f7ff40c0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:29.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7ff3ac0700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff40b8de0 0x7f7ff40c0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:29.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7ff3ac0700 1 -- 172.21.15.32:0/2273383341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff40bb2d0 con 0x7f7ff4193cf0 2021-04-09T18:31:29.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7ff3ac0700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff4193cf0 0x7f7ff41940f0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 rx=0x7f7fec009980 tx=0x7f7fec00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:29.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7fdbfff700 1 -- 172.21.15.32:0/2273383341 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fec00be10 con 0x7f7ff4193cf0 2021-04-09T18:31:29.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7fdbfff700 1 -- 172.21.15.32:0/2273383341 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec005870 con 0x7f7ff4193cf0 2021-04-09T18:31:29.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff40bb2d0 con 0x7f7ff4193cf0 2021-04-09T18:31:29.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7fdbfff700 1 -- 172.21.15.32:0/2273383341 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fec005d50 con 0x7f7ff4193cf0 2021-04-09T18:31:29.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.881 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff40bb2d0 con 0x7f7ff4193cf0 2021-04-09T18:31:29.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.882 7f7fdbfff700 1 -- 172.21.15.32:0/2273383341 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7fec005870 con 0x7f7ff4193cf0 2021-04-09T18:31:29.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.882 7f7fdbfff700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd40188c0 0x7f7fd401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:29.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.883 7f7fdbfff700 1 -- 172.21.15.32:0/2273383341 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5926+0+0 (secure 0 0 0) 0x7f7fec032c50 con 0x7f7ff4193cf0 2021-04-09T18:31:29.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.883 7f7ff32bf700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd40188c0 0x7f7fd401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:29.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.883 7f7ff32bf700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd40188c0 0x7f7fd401adc0 crc :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:29.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.901 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff40b89a0 con 0x7f7ff4193cf0 2021-04-09T18:31:29.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:29.905 7f7fdbfff700 1 -- 172.21.15.32:0/2273383341 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7fec00be10 con 0x7f7ff4193cf0 2021-04-09T18:31:30.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.088 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_data"} v 0) v1 -- 0x7f7ff40b89a0 con 0x7f7ff4193cf0 2021-04-09T18:31:30.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.681 7f7fdbfff700 1 -- 172.21.15.32:0/2273383341 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_data"}]=0 pool 'bravo_data' created v205) v1 ==== 123+0+0 (secure 0 0 0) 0x7f7fec005870 con 0x7f7ff4193cf0 2021-04-09T18:31:30.718 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_data' created 2021-04-09T18:31:30.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd40188c0 msgr2=0x7f7fd401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:30.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd40188c0 0x7f7fd401adc0 crc :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff4193cf0 msgr2=0x7f7ff41940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:30.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff4193cf0 0x7f7ff41940f0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 rx=0x7f7fec009980 tx=0x7f7fec00bcd0).stop 2021-04-09T18:31:30.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 shutdown_connections 2021-04-09T18:31:30.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fd40188c0 0x7f7fd401adc0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ff4193cf0 0x7f7ff41940f0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ff40c0640 0x7f7ff41936d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 --2- 172.21.15.32:0/2273383341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ff40b8de0 0x7f7ff40c0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.705 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 >> 172.21.15.32:0/2273383341 conn(0x7f7ff4092a00 msgr2=0x7f7ff41b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:30.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.706 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 shutdown_connections 2021-04-09T18:31:30.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.706 7f7ff9513700 1 -- 172.21.15.32:0/2273383341 wait complete. 2021-04-09T18:31:30.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new bravo bravo_metadata bravo_data --force 2021-04-09T18:31:30.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24d260f700 1 Processor -- start 2021-04-09T18:31:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24d260f700 1 -- start start 2021-04-09T18:31:30.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24d260f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc10d050 0x7f24cc0cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24d260f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f24cc0d84b0 0x7f24cc10da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24d260f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24cc10d500 con 0x7f24cc0d0160 2021-04-09T18:31:30.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24d260f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24cc10e800 con 0x7f24cc0d84b0 2021-04-09T18:31:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24d260f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0d8d70 con 0x7f24cc10d050 2021-04-09T18:31:30.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24d0c2c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f24cc0d84b0 0x7f24cc10da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24c3fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f24cc0d0160 0x7f24cc0d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38318/0 (socket says 172.21.15.32:38318) 2021-04-09T18:31:30.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.881 7f24c3fff700 1 -- 172.21.15.32:0/1558073202 learned_addr learned my addr 172.21.15.32:0/1558073202 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24cbfff700 1 --2- 172.21.15.32:0/1558073202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc10d050 0x7f24cc0cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:30.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24d0c2c700 1 -- 172.21.15.32:0/1558073202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc10d050 msgr2=0x7f24cc0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24d0c2c700 1 --2- 172.21.15.32:0/1558073202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc10d050 0x7f24cc0cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24d0c2c700 1 -- 172.21.15.32:0/1558073202 >> v1:172.21.15.32:6789/0 conn(0x7f24cc0d0160 legacy=0x7f24cc0d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24cb7fe700 1 -- 172.21.15.32:0/1558073202 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913115297 0 0) 0x7f24b4000ce0 con 0x7f24cc0d0160 2021-04-09T18:31:30.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24d0c2c700 1 -- 172.21.15.32:0/1558073202 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc0d9260 con 0x7f24cc0d84b0 2021-04-09T18:31:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24cbfff700 1 --2- 172.21.15.32:0/1558073202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc10d050 0x7f24cc0cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24d0c2c700 1 --2- 172.21.15.32:0/1558073202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc0d84b0 0x7f24cc10da80 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 rx=0x7f24c4006e30 tx=0x7f24c4009e60).ready entity=mon.0 client_cookie=2447cbb139fd1cad server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:30.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24cb7fe700 1 -- 172.21.15.32:0/1558073202 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24c4006a50 con 0x7f24cc0d84b0 2021-04-09T18:31:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24cb7fe700 1 -- 172.21.15.32:0/1558073202 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c400b300 con 0x7f24cc0d84b0 2021-04-09T18:31:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24cb7fe700 1 -- 172.21.15.32:0/1558073202 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24c400b750 con 0x7f24cc0d84b0 2021-04-09T18:31:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24d260f700 1 -- 172.21.15.32:0/1558073202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc0d84b0 msgr2=0x7f24cc10da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:30.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.882 7f24d260f700 1 --2- 172.21.15.32:0/1558073202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc0d84b0 0x7f24cc10da80 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 rx=0x7f24c4006e30 tx=0x7f24c4009e60).stop 2021-04-09T18:31:30.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.883 7f24d260f700 1 -- 172.21.15.32:0/1558073202 shutdown_connections 2021-04-09T18:31:30.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.883 7f24d260f700 1 --2- 172.21.15.32:0/1558073202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc0d84b0 0x7f24cc10da80 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.883 7f24d260f700 1 --2- 172.21.15.32:0/1558073202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc10d050 0x7f24cc0cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.883 7f24d260f700 1 -- 172.21.15.32:0/1558073202 >> 172.21.15.32:0/1558073202 conn(0x7f24cc0ce620 msgr2=0x7f24cc0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:30.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.883 7f24d260f700 1 -- 172.21.15.32:0/1558073202 shutdown_connections 2021-04-09T18:31:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.883 7f24d260f700 1 -- 172.21.15.32:0/1558073202 wait complete. 2021-04-09T18:31:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.903 7f24d260f700 1 Processor -- start 2021-04-09T18:31:30.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.903 7f24d260f700 1 -- start start 2021-04-09T18:31:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24d260f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc1818c0 0x7f24cc1822c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:30.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24d260f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24cc0a6040 0x7f24cc1828e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24d260f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc0bebf0 0x7f24cc0beff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24d260f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc183040 con 0x7f24cc1818c0 2021-04-09T18:31:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24d260f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc0bd070 con 0x7f24cc0a6040 2021-04-09T18:31:30.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24d260f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc181d40 con 0x7f24cc0bebf0 2021-04-09T18:31:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc1818c0 0x7f24cc1822c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24cbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc1818c0 0x7f24cc1822c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56968/0 (socket says 172.21.15.32:56968) 2021-04-09T18:31:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24cbfff700 1 -- 172.21.15.32:0/1098265730 learned_addr learned my addr 172.21.15.32:0/1098265730 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:30.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24c3fff700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24cc0a6040 0x7f24cc1828e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:30.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24d0c2c700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc0bebf0 0x7f24cc0beff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:30.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24cbfff700 1 -- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc0bebf0 msgr2=0x7f24cc0beff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:30.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24cbfff700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc0bebf0 0x7f24cc0beff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24cbfff700 1 -- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24cc0a6040 msgr2=0x7f24cc1828e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24cbfff700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24cc0a6040 0x7f24cc1828e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24cbfff700 1 -- 172.21.15.32:0/1098265730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc08f3d0 con 0x7f24cc1818c0 2021-04-09T18:31:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.904 7f24cbfff700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc1818c0 0x7f24cc1822c0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 rx=0x7f24bc006f20 tx=0x7f24bc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:30.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.905 7f24c97fa700 1 -- 172.21.15.32:0/1098265730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24bc00bd80 con 0x7f24cc1818c0 2021-04-09T18:31:30.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.905 7f24c97fa700 1 -- 172.21.15.32:0/1098265730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc000c90 con 0x7f24cc1818c0 2021-04-09T18:31:30.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.905 7f24d260f700 1 -- 172.21.15.32:0/1098265730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24cc08f3d0 con 0x7f24cc1818c0 2021-04-09T18:31:30.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.905 7f24c97fa700 1 -- 172.21.15.32:0/1098265730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24bc0022b0 con 0x7f24cc1818c0 2021-04-09T18:31:30.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.905 7f24d260f700 1 -- 172.21.15.32:0/1098265730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24cc08f3d0 con 0x7f24cc1818c0 2021-04-09T18:31:30.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.905 7f24c97fa700 1 -- 172.21.15.32:0/1098265730 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f24bc00a000 con 0x7f24cc1818c0 2021-04-09T18:31:30.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.906 7f24c97fa700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24b00188c0 0x7f24b001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:30.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.906 7f24c97fa700 1 -- 172.21.15.32:0/1098265730 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6275+0+0 (secure 0 0 0) 0x7f24bc032ac0 con 0x7f24cc1818c0 2021-04-09T18:31:30.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.906 7f24c3fff700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24b00188c0 0x7f24b001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:30.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.906 7f24c3fff700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24b00188c0 0x7f24b001adc0 crc :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:30.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.925 7f24d260f700 1 -- 172.21.15.32:0/1098265730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc0bd070 con 0x7f24cc1818c0 2021-04-09T18:31:30.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:30.929 7f24c97fa700 1 -- 172.21.15.32:0/1098265730 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f24bc006bb0 con 0x7f24cc1818c0 2021-04-09T18:31:31.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.112 7f24d260f700 1 -- 172.21.15.32:0/1098265730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "bravo_data", "prefix": "fs new", "force": true, "fs_name": "bravo", "metadata": "bravo_metadata"} v 0) v1 -- 0x7f24cc0bd070 con 0x7f24cc1818c0 2021-04-09T18:31:31.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.116 7f24c97fa700 1 -- 172.21.15.32:0/1098265730 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "bravo_data", "prefix": "fs new", "force": true, "fs_name": "bravo", "metadata": "bravo_metadata"}]=0 new fs with metadata pool 36 and data pool 37 v350) v1 ==== 186+0+0 (secure 0 0 0) 0x7f24bc00a000 con 0x7f24cc1818c0 2021-04-09T18:31:31.134 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 36 and data pool 37 2021-04-09T18:31:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.123 7f24d260f700 1 -- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24b00188c0 msgr2=0x7f24b001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.123 7f24d260f700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24b00188c0 0x7f24b001adc0 crc :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.123 7f24d260f700 1 -- 172.21.15.32:0/1098265730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc1818c0 msgr2=0x7f24cc1822c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.123 7f24d260f700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc1818c0 0x7f24cc1822c0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 rx=0x7f24bc006f20 tx=0x7f24bc00b990).stop 2021-04-09T18:31:31.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.125 7f24d260f700 1 -- 172.21.15.32:0/1098265730 shutdown_connections 2021-04-09T18:31:31.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.125 7f24d260f700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f24b00188c0 0x7f24b001adc0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.125 7f24d260f700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24cc0bebf0 0x7f24cc0beff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.125 7f24d260f700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24cc0a6040 0x7f24cc1828e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.125 7f24d260f700 1 --2- 172.21.15.32:0/1098265730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f24cc1818c0 0x7f24cc1822c0 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.125 7f24d260f700 1 -- 172.21.15.32:0/1098265730 >> 172.21.15.32:0/1098265730 conn(0x7f24cc08a230 msgr2=0x7f24cc191400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:31.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.125 7f24d260f700 1 -- 172.21.15.32:0/1098265730 shutdown_connections 2021-04-09T18:31:31.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.126 7f24d260f700 1 -- 172.21.15.32:0/1098265730 wait complete. 2021-04-09T18:31:31.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:31:31.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.286 7fbd0bc51700 1 Processor -- start 2021-04-09T18:31:31.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.286 7fbd0bc51700 1 -- start start 2021-04-09T18:31:31.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd0bc51700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd0410d010 0x7fbd040d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd0bc51700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd040d1da0 0x7fbd040d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd0bc51700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0410d480 con 0x7fbd040d82c0 2021-04-09T18:31:31.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd0bc51700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd040d8d30 con 0x7fbd0410d010 2021-04-09T18:31:31.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd0bc51700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd040d9220 con 0x7fbd040d1da0 2021-04-09T18:31:31.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd09a6d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd0410d010 0x7fbd040d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd09a6d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd0410d010 0x7fbd040d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56974/0 (socket says 172.21.15.32:56974) 2021-04-09T18:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd09a6d700 1 -- 172.21.15.32:0/4072759037 learned_addr learned my addr 172.21.15.32:0/4072759037 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd0926c700 1 --2- 172.21.15.32:0/4072759037 >> v2:172.21.15.86:3300/0 conn(0x7fbd040d1da0 0x7fbd040d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd09a6d700 1 -- 172.21.15.32:0/4072759037 >> v2:172.21.15.86:3300/0 conn(0x7fbd040d1da0 msgr2=0x7fbd040d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd09a6d700 1 --2- 172.21.15.32:0/4072759037 >> v2:172.21.15.86:3300/0 conn(0x7fbd040d1da0 0x7fbd040d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd09a6d700 1 -- 172.21.15.32:0/4072759037 >> v1:172.21.15.86:6789/0 conn(0x7fbd040d82c0 legacy=0x7fbd0410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd09a6d700 1 -- 172.21.15.32:0/4072759037 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd040d86f0 con 0x7fbd0410d010 2021-04-09T18:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd09a6d700 1 --2- 172.21.15.32:0/4072759037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0410d010 0x7fbd040d1780 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7fbd00006700 tx=0x7fbd00009b80).ready entity=mon.0 client_cookie=92202501303c96b8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.287 7fbd08a6b700 1 -- 172.21.15.32:0/4072759037 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd0000c0c0 con 0x7fbd0410d010 2021-04-09T18:31:31.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.288 7fbd08a6b700 1 -- 172.21.15.32:0/4072759037 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0000c0c0 con 0x7fbd0410d010 2021-04-09T18:31:31.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.288 7fbd08a6b700 1 -- 172.21.15.32:0/4072759037 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd0000c440 con 0x7fbd0410d010 2021-04-09T18:31:31.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.288 7fbd0bc51700 1 -- 172.21.15.32:0/4072759037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0410d010 msgr2=0x7fbd040d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.288 7fbd0bc51700 1 --2- 172.21.15.32:0/4072759037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0410d010 0x7fbd040d1780 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7fbd00006700 tx=0x7fbd00009b80).stop 2021-04-09T18:31:31.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.288 7fbd0bc51700 1 -- 172.21.15.32:0/4072759037 shutdown_connections 2021-04-09T18:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.288 7fbd0bc51700 1 --2- 172.21.15.32:0/4072759037 >> v2:172.21.15.86:3300/0 conn(0x7fbd040d1da0 0x7fbd040d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.288 7fbd0bc51700 1 --2- 172.21.15.32:0/4072759037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd0410d010 0x7fbd040d1780 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.288 7fbd0bc51700 1 -- 172.21.15.32:0/4072759037 >> 172.21.15.32:0/4072759037 conn(0x7fbd040ce5f0 msgr2=0x7fbd040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.288 7fbd0bc51700 1 -- 172.21.15.32:0/4072759037 shutdown_connections 2021-04-09T18:31:31.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.289 7fbd0bc51700 1 -- 172.21.15.32:0/4072759037 wait complete. 2021-04-09T18:31:31.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd0bc51700 1 Processor -- start 2021-04-09T18:31:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd0bc51700 1 -- start start 2021-04-09T18:31:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd0bc51700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd041acea0 0x7fbd041ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd0bc51700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0419b2c0 0x7fbd04184da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd0bc51700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd041853c0 0x7fbd04185fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd0bc51700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041857c0 con 0x7fbd041acea0 2021-04-09T18:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd0bc51700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd04185a40 con 0x7fbd0419b2c0 2021-04-09T18:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd0bc51700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041865e0 con 0x7fbd041853c0 2021-04-09T18:31:31.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd09a6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd041acea0 0x7fbd041ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.309 7fbd09a6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd041acea0 0x7fbd041ac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56980/0 (socket says 172.21.15.32:56980) 2021-04-09T18:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd09a6d700 1 -- 172.21.15.32:0/136278692 learned_addr learned my addr 172.21.15.32:0/136278692 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd0926c700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0419b2c0 0x7fbd04184da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd0a26e700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd041853c0 0x7fbd04185fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd09a6d700 1 -- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd041853c0 msgr2=0x7fbd04185fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd09a6d700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd041853c0 0x7fbd04185fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd09a6d700 1 -- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0419b2c0 msgr2=0x7fbd04184da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd09a6d700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0419b2c0 0x7fbd04184da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd09a6d700 1 -- 172.21.15.32:0/136278692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04186800 con 0x7fbd041acea0 2021-04-09T18:31:31.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd0926c700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0419b2c0 0x7fbd04184da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:31.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd09a6d700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd041acea0 0x7fbd041ac670 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 rx=0x7fbd0000b580 tx=0x7fbd0000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:31.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbcf67fc700 1 -- 172.21.15.32:0/136278692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd0000bd10 con 0x7fbd041acea0 2021-04-09T18:31:31.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbcf67fc700 1 -- 172.21.15.32:0/136278692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd00002030 con 0x7fbd041acea0 2021-04-09T18:31:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd04186800 con 0x7fbd041acea0 2021-04-09T18:31:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbcf67fc700 1 -- 172.21.15.32:0/136278692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd00002550 con 0x7fbd041acea0 2021-04-09T18:31:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.310 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd04186800 con 0x7fbd041acea0 2021-04-09T18:31:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.311 7fbcf67fc700 1 -- 172.21.15.32:0/136278692 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbd00000c90 con 0x7fbd041acea0 2021-04-09T18:31:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.311 7fbcf67fc700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcec018930 0x7fbcec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.311 7fbcf67fc700 1 -- 172.21.15.32:0/136278692 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fbd00032d70 con 0x7fbd041acea0 2021-04-09T18:31:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.311 7fbd0926c700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcec018930 0x7fbcec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.311 7fbd0926c700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcec018930 0x7fbcec01ae10 crc :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:31.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.331 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd04182d80 con 0x7fbd041acea0 2021-04-09T18:31:31.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.334 7fbcf67fc700 1 -- 172.21.15.32:0/136278692 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbd0001f780 con 0x7fbd041acea0 2021-04-09T18:31:31.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.521 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd04182d80 con 0x7fbd041acea0 2021-04-09T18:31:31.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.522 7fbcf67fc700 1 -- 172.21.15.32:0/136278692 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v206) v1 ==== 74+0+16865 (secure 0 0 0) 0x7fbd00000c90 con 0x7fbd041acea0 2021-04-09T18:31:31.539 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:31.540 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":206,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:31:31.113314","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:31:21.254184","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":35,"pool_name":"alpha_data","create_time":"2021-04-09 18:31:22.120246","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":36,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:31:28.904296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":37,"pool_name":"bravo_data","create_time":"2021-04-09 18:31:30.123936","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:31:31.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcec018930 msgr2=0x7fbcec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcec018930 0x7fbcec01ae10 crc :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd041acea0 msgr2=0x7fbd041ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd041acea0 0x7fbd041ac670 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 rx=0x7fbd0000b580 tx=0x7fbd0000b920).stop 2021-04-09T18:31:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 shutdown_connections 2021-04-09T18:31:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbcec018930 0x7fbcec01ae10 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd041853c0 0x7fbd04185fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0419b2c0 0x7fbd04184da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 --2- 172.21.15.32:0/136278692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd041acea0 0x7fbd041ac670 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 >> 172.21.15.32:0/136278692 conn(0x7fbd0419b890 msgr2=0x7fbd041a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:31.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 shutdown_connections 2021-04-09T18:31:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.526 7fbd0bc51700 1 -- 172.21.15.32:0/136278692 wait complete. 2021-04-09T18:31:31.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:31:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.686 7f7b262a0700 1 Processor -- start 2021-04-09T18:31:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.686 7f7b262a0700 1 -- start start 2021-04-09T18:31:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b262a0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b2010d010 0x7f7b200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b262a0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b200d8490 0x7f7b2010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b262a0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2010d480 con 0x7f7b200d1da0 2021-04-09T18:31:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b262a0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b200d8d30 con 0x7f7b2010d010 2021-04-09T18:31:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b262a0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b200d9220 con 0x7f7b200d8490 2021-04-09T18:31:31.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b1ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b2010d010 0x7f7b200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b1ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b2010d010 0x7f7b200d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56988/0 (socket says 172.21.15.32:56988) 2021-04-09T18:31:31.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b1ffff700 1 -- 172.21.15.32:0/1894172460 learned_addr learned my addr 172.21.15.32:0/1894172460 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:31.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b248bd700 1 --2- 172.21.15.32:0/1894172460 >> v2:172.21.15.86:3300/0 conn(0x7f7b200d8490 0x7f7b2010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b1ffff700 1 -- 172.21.15.32:0/1894172460 >> v2:172.21.15.86:3300/0 conn(0x7f7b200d8490 msgr2=0x7f7b2010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b1ffff700 1 --2- 172.21.15.32:0/1894172460 >> v2:172.21.15.86:3300/0 conn(0x7f7b200d8490 0x7f7b2010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b1ffff700 1 -- 172.21.15.32:0/1894172460 >> v1:172.21.15.86:6789/0 conn(0x7f7b200d1da0 legacy=0x7f7b200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.687 7f7b1ffff700 1 -- 172.21.15.32:0/1894172460 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2010e020 con 0x7f7b2010d010 2021-04-09T18:31:31.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.688 7f7b1ffff700 1 --2- 172.21.15.32:0/1894172460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2010d010 0x7f7b200d1780 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 rx=0x7f7b10006700 tx=0x7f7b10009b80).ready entity=mon.0 client_cookie=b4707f233996b10a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.688 7f7b1f7fe700 1 -- 172.21.15.32:0/1894172460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b1000c0c0 con 0x7f7b2010d010 2021-04-09T18:31:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.688 7f7b1f7fe700 1 -- 172.21.15.32:0/1894172460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1000c440 con 0x7f7b2010d010 2021-04-09T18:31:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.688 7f7b1f7fe700 1 -- 172.21.15.32:0/1894172460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b1000c920 con 0x7f7b2010d010 2021-04-09T18:31:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.688 7f7b262a0700 1 -- 172.21.15.32:0/1894172460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2010d010 msgr2=0x7f7b200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.688 7f7b262a0700 1 --2- 172.21.15.32:0/1894172460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2010d010 0x7f7b200d1780 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 rx=0x7f7b10006700 tx=0x7f7b10009b80).stop 2021-04-09T18:31:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.689 7f7b262a0700 1 -- 172.21.15.32:0/1894172460 shutdown_connections 2021-04-09T18:31:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.689 7f7b262a0700 1 --2- 172.21.15.32:0/1894172460 >> v2:172.21.15.86:3300/0 conn(0x7f7b200d8490 0x7f7b2010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.689 7f7b262a0700 1 --2- 172.21.15.32:0/1894172460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b2010d010 0x7f7b200d1780 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.689 7f7b262a0700 1 -- 172.21.15.32:0/1894172460 >> 172.21.15.32:0/1894172460 conn(0x7f7b200ce5f0 msgr2=0x7f7b200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.689 7f7b262a0700 1 -- 172.21.15.32:0/1894172460 shutdown_connections 2021-04-09T18:31:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.689 7f7b262a0700 1 -- 172.21.15.32:0/1894172460 wait complete. 2021-04-09T18:31:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.709 7f7b262a0700 1 Processor -- start 2021-04-09T18:31:31.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.709 7f7b262a0700 1 -- start start 2021-04-09T18:31:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.709 7f7b262a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2009a5e0 0x7f7b201884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.709 7f7b262a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b200a3350 0x7f7b200a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.709 7f7b262a0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b200a15b0 0x7f7b200a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.709 7f7b262a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200a40d0 con 0x7f7b200a3350 2021-04-09T18:31:31.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.709 7f7b262a0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b200a2dd0 con 0x7f7b2009a5e0 2021-04-09T18:31:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.709 7f7b262a0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b20188ad0 con 0x7f7b200a15b0 2021-04-09T18:31:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b200a3350 0x7f7b200a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b200a3350 0x7f7b200a3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56994/0 (socket says 172.21.15.32:56994) 2021-04-09T18:31:31.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b1ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2009a5e0 0x7f7b201884b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b17fff700 1 -- 172.21.15.32:0/2076706765 learned_addr learned my addr 172.21.15.32:0/2076706765 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:31.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b248bd700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b200a15b0 0x7f7b200a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b17fff700 1 -- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b200a15b0 msgr2=0x7f7b200a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b17fff700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b200a15b0 0x7f7b200a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b17fff700 1 -- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2009a5e0 msgr2=0x7f7b201884b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b17fff700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2009a5e0 0x7f7b201884b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b17fff700 1 -- 172.21.15.32:0/2076706765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b200a3d70 con 0x7f7b200a3350 2021-04-09T18:31:31.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b1ffff700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2009a5e0 0x7f7b201884b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:31.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b17fff700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b200a3350 0x7f7b200a3750 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 rx=0x7f7b0800b5b0 tx=0x7f7b0800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:31.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b1d7fa700 1 -- 172.21.15.32:0/2076706765 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b0800bd40 con 0x7f7b200a3350 2021-04-09T18:31:31.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b200a3d70 con 0x7f7b200a3350 2021-04-09T18:31:31.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b1d7fa700 1 -- 172.21.15.32:0/2076706765 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0800bd40 con 0x7f7b200a3350 2021-04-09T18:31:31.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b1d7fa700 1 -- 172.21.15.32:0/2076706765 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b08006020 con 0x7f7b200a3350 2021-04-09T18:31:31.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.710 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b200a22c0 con 0x7f7b200a3350 2021-04-09T18:31:31.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.711 7f7b1d7fa700 1 -- 172.21.15.32:0/2076706765 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7b080054a0 con 0x7f7b200a3350 2021-04-09T18:31:31.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.711 7f7b1d7fa700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b040188d0 0x7f7b0401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:31.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.711 7f7b1d7fa700 1 -- 172.21.15.32:0/2076706765 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f7b08032ef0 con 0x7f7b200a3350 2021-04-09T18:31:31.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.711 7f7b1ffff700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b040188d0 0x7f7b0401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:31.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.712 7f7b1ffff700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b040188d0 0x7f7b0401add0 crc :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:31.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.730 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2010d480 con 0x7f7b200a3350 2021-04-09T18:31:31.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.734 7f7b1d7fa700 1 -- 172.21.15.32:0/2076706765 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7b08006220 con 0x7f7b200a3350 2021-04-09T18:31:31.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.922 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b2010d480 con 0x7f7b200a3350 2021-04-09T18:31:31.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.923 7f7b1d7fa700 1 -- 172.21.15.32:0/2076706765 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v206) v1 ==== 74+0+16865 (secure 0 0 0) 0x7f7b080054a0 con 0x7f7b200a3350 2021-04-09T18:31:31.939 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:31.939 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":206,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:31:31.113314","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:31:21.254184","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":35,"pool_name":"alpha_data","create_time":"2021-04-09 18:31:22.120246","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":36,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:31:28.904296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":37,"pool_name":"bravo_data","create_time":"2021-04-09 18:31:30.123936","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:31:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b040188d0 msgr2=0x7f7b0401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b040188d0 0x7f7b0401add0 crc :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b200a3350 msgr2=0x7f7b200a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b200a3350 0x7f7b200a3750 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 rx=0x7f7b0800b5b0 tx=0x7f7b0800b950).stop 2021-04-09T18:31:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 shutdown_connections 2021-04-09T18:31:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7b040188d0 0x7f7b0401add0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b200a3350 0x7f7b200a3750 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b200a15b0 0x7f7b200a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 --2- 172.21.15.32:0/2076706765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2009a5e0 0x7f7b201884b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:31.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.925 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 >> 172.21.15.32:0/2076706765 conn(0x7f7b201a0db0 msgr2=0x7f7b201a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:31.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.926 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 shutdown_connections 2021-04-09T18:31:31.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:31.926 7f7b262a0700 1 -- 172.21.15.32:0/2076706765 wait complete. 2021-04-09T18:31:31.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set bravo standby_count_wanted 0 2021-04-09T18:31:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.089 7fbd85f49700 1 Processor -- start 2021-04-09T18:31:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.089 7fbd85f49700 1 -- start start 2021-04-09T18:31:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd85f49700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8010da00 0x7fbd8010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd85f49700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8010d480 con 0x7fbd800d7ca0 2021-04-09T18:31:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd85f49700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd800d1f70 con 0x7fbd8010d010 2021-04-09T18:31:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd85f49700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800d9220 con 0x7fbd8010da00 2021-04-09T18:31:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7effd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbd800d7ca0 0x7fbd800d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38360/0 (socket says 172.21.15.32:38360) 2021-04-09T18:31:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7effd700 1 -- 172.21.15.32:0/1791162206 learned_addr learned my addr 172.21.15.32:0/1791162206 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7ffff700 1 --2- 172.21.15.32:0/1791162206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8010da00 0x7fbd8010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7e7fc700 1 -- 172.21.15.32:0/1791162206 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275458970 0 0) 0x7fbd74000ce0 con 0x7fbd800d7ca0 2021-04-09T18:31:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7e7fc700 1 -- 172.21.15.32:0/1791162206 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd68002fe0 con 0x7fbd800d7ca0 2021-04-09T18:31:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7e7fc700 1 -- 172.21.15.32:0/1791162206 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103908715 0 0) 0x7fbd70000c60 con 0x7fbd8010d010 2021-04-09T18:31:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7e7fc700 1 -- 172.21.15.32:0/1791162206 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd68002fe0 con 0x7fbd8010d010 2021-04-09T18:31:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7ffff700 1 -- 172.21.15.32:0/1791162206 >> v1:172.21.15.86:6789/0 conn(0x7fbd8010d010 legacy=0x7fbd800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7ffff700 1 -- 172.21.15.32:0/1791162206 >> v1:172.21.15.32:6789/0 conn(0x7fbd800d7ca0 legacy=0x7fbd800d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7ffff700 1 -- 172.21.15.32:0/1791162206 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80112100 con 0x7fbd8010da00 2021-04-09T18:31:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.090 7fbd7ffff700 1 --2- 172.21.15.32:0/1791162206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8010da00 0x7fbd8010de00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fbd78006e50 tx=0x7fbd78009e70).ready entity=mon.2 client_cookie=90cce60fb4f6cf0d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd7e7fc700 1 -- 172.21.15.32:0/1791162206 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd78006a20 con 0x7fbd8010da00 2021-04-09T18:31:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd7e7fc700 1 -- 172.21.15.32:0/1791162206 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd78006790 con 0x7fbd8010da00 2021-04-09T18:31:32.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd7e7fc700 1 -- 172.21.15.32:0/1791162206 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd7800b700 con 0x7fbd8010da00 2021-04-09T18:31:32.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd85f49700 1 -- 172.21.15.32:0/1791162206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8010da00 msgr2=0x7fbd8010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:32.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd85f49700 1 --2- 172.21.15.32:0/1791162206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8010da00 0x7fbd8010de00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fbd78006e50 tx=0x7fbd78009e70).stop 2021-04-09T18:31:32.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd85f49700 1 -- 172.21.15.32:0/1791162206 shutdown_connections 2021-04-09T18:31:32.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd85f49700 1 --2- 172.21.15.32:0/1791162206 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8010da00 0x7fbd8010de00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:32.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd85f49700 1 -- 172.21.15.32:0/1791162206 >> 172.21.15.32:0/1791162206 conn(0x7fbd800ce5f0 msgr2=0x7fbd800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:32.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd85f49700 1 -- 172.21.15.32:0/1791162206 shutdown_connections 2021-04-09T18:31:32.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.091 7fbd85f49700 1 -- 172.21.15.32:0/1791162206 wait complete. 2021-04-09T18:31:32.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.111 7fbd85f49700 1 Processor -- start 2021-04-09T18:31:32.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd85f49700 1 -- start start 2021-04-09T18:31:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd85f49700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd80193c00 0x7fbd80179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd85f49700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd80179c30 0x7fbd801b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd85f49700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd801b8b60 0x7fbd8017a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:32.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd85f49700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8017a060 con 0x7fbd80179c30 2021-04-09T18:31:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd85f49700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800c0de0 con 0x7fbd801b8b60 2021-04-09T18:31:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd85f49700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd80194000 con 0x7fbd80193c00 2021-04-09T18:31:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd7effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd80179c30 0x7fbd801b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd7effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd80179c30 0x7fbd801b8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57008/0 (socket says 172.21.15.32:57008) 2021-04-09T18:31:32.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd7effd700 1 -- 172.21.15.32:0/109743575 learned_addr learned my addr 172.21.15.32:0/109743575 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:32.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd7f7fe700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd80193c00 0x7fbd80179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:32.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.112 7fbd7ffff700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd801b8b60 0x7fbd8017a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:32.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd7effd700 1 -- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd80193c00 msgr2=0x7fbd80179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:32.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd7effd700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd80193c00 0x7fbd80179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:32.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd7effd700 1 -- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd801b8b60 msgr2=0x7fbd8017a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:32.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd7effd700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd801b8b60 0x7fbd8017a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:32.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd7effd700 1 -- 172.21.15.32:0/109743575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80194220 con 0x7fbd80179c30 2021-04-09T18:31:32.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd7effd700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd80179c30 0x7fbd801b8540 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 rx=0x7fbd74006f00 tx=0x7fbd7400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:32.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd67fff700 1 -- 172.21.15.32:0/109743575 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd7400bd30 con 0x7fbd80179c30 2021-04-09T18:31:32.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd67fff700 1 -- 172.21.15.32:0/109743575 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd74006030 con 0x7fbd80179c30 2021-04-09T18:31:32.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd85f49700 1 -- 172.21.15.32:0/109743575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd80194220 con 0x7fbd80179c30 2021-04-09T18:31:32.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd67fff700 1 -- 172.21.15.32:0/109743575 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd74004590 con 0x7fbd80179c30 2021-04-09T18:31:32.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.113 7fbd85f49700 1 -- 172.21.15.32:0/109743575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd80194220 con 0x7fbd80179c30 2021-04-09T18:31:32.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.114 7fbd67fff700 1 -- 172.21.15.32:0/109743575 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbd740043a0 con 0x7fbd80179c30 2021-04-09T18:31:32.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.114 7fbd67fff700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd60018930 0x7fbd6001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:32.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.114 7fbd67fff700 1 -- 172.21.15.32:0/109743575 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fbd74032cf0 con 0x7fbd80179c30 2021-04-09T18:31:32.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.114 7fbd7f7fe700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd60018930 0x7fbd6001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:32.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.114 7fbd7f7fe700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd60018930 0x7fbd6001adc0 crc :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:32.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.134 7fbd85f49700 1 -- 172.21.15.32:0/109743575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8010d480 con 0x7fbd80179c30 2021-04-09T18:31:32.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.137 7fbd67fff700 1 -- 172.21.15.32:0/109743575 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbd740009b0 con 0x7fbd80179c30 2021-04-09T18:31:32.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:32.330 7fbd85f49700 1 -- 172.21.15.32:0/109743575 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "bravo", "val": "0"} v 0) v1 -- 0x7fbd8010d480 con 0x7fbd80179c30 2021-04-09T18:31:33.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.121 7fbd67fff700 1 -- 172.21.15.32:0/109743575 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "bravo", "val": "0"}]=0 v353) v1 ==== 117+0+0 (secure 0 0 0) 0x7fbd740043a0 con 0x7fbd80179c30 2021-04-09T18:31:33.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 -- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd60018930 msgr2=0x7fbd6001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd60018930 0x7fbd6001adc0 crc :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 -- 172.21.15.32:0/109743575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd80179c30 msgr2=0x7fbd801b8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd80179c30 0x7fbd801b8540 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 rx=0x7fbd74006f00 tx=0x7fbd7400b940).stop 2021-04-09T18:31:33.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 -- 172.21.15.32:0/109743575 shutdown_connections 2021-04-09T18:31:33.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd60018930 0x7fbd6001adc0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd80179c30 0x7fbd801b8540 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd801b8b60 0x7fbd8017a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 --2- 172.21.15.32:0/109743575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd80193c00 0x7fbd80179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.148 7fbd85f49700 1 -- 172.21.15.32:0/109743575 >> 172.21.15.32:0/109743575 conn(0x7fbd800bbda0 msgr2=0x7fbd801b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:33.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.149 7fbd85f49700 1 -- 172.21.15.32:0/109743575 shutdown_connections 2021-04-09T18:31:33.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.149 7fbd85f49700 1 -- 172.21.15.32:0/109743575 wait complete. 2021-04-09T18:31:33.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:33.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.316 7f8f8ef95700 1 Processor -- start 2021-04-09T18:31:33.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.316 7f8f8ef95700 1 -- start start 2021-04-09T18:31:33.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.317 7f8f8ef95700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8f8810e7d0 0x7f8f880d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.317 7f8f8ef95700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f880d7280 con 0x7f8f880d6c80 2021-04-09T18:31:33.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.317 7f8f8ef95700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f880d2440 con 0x7f8f880cffa0 2021-04-09T18:31:33.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.317 7f8f8ef95700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f880d7080 con 0x7f8f8810e7d0 2021-04-09T18:31:33.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.317 7f8f87fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8f880d6c80 0x7f8f8810cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38372/0 (socket says 172.21.15.32:38372) 2021-04-09T18:31:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.317 7f8f87fff700 1 -- 172.21.15.32:0/3784017730 learned_addr learned my addr 172.21.15.32:0/3784017730 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.317 7f8f8d5b2700 1 --2- 172.21.15.32:0/3784017730 >> v2:172.21.15.86:3300/0 conn(0x7f8f8810e7d0 0x7f8f880d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.317 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549278037 0 0) 0x7f8f70000ce0 con 0x7f8f880d6c80 2021-04-09T18:31:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.317 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f74002fe0 con 0x7f8f880d6c80 2021-04-09T18:31:33.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590509511 0 0) 0x7f8f78000fe0 con 0x7f8f880cffa0 2021-04-09T18:31:33.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f74003210 con 0x7f8f880cffa0 2021-04-09T18:31:33.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1538494499 0 0) 0x7f8f700010e0 con 0x7f8f880d6c80 2021-04-09T18:31:33.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 >> v2:172.21.15.86:3300/0 conn(0x7f8f8810e7d0 msgr2=0x7f8f880d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:33.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 --2- 172.21.15.32:0/3784017730 >> v2:172.21.15.86:3300/0 conn(0x7f8f8810e7d0 0x7f8f880d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 >> v1:172.21.15.86:6789/0 conn(0x7f8f880cffa0 legacy=0x7f8f880d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f880d91e0 con 0x7f8f880d6c80 2021-04-09T18:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f8f70001600 con 0x7f8f880d6c80 2021-04-09T18:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 >> v1:172.21.15.32:6789/0 conn(0x7f8f880d6c80 legacy=0x7f8f8810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f74003b10 0x7f8f74005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f74006780 0x7f8f74008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f74009580 0x7f8f7400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f74050300 con 0x7f8f74003b10 2021-04-09T18:31:33.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740504e0 con 0x7f8f74009580 2021-04-09T18:31:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f74050700 con 0x7f8f74006780 2021-04-09T18:31:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f8cdb1700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f74003b10 0x7f8f74005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f87fff700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f74006780 0x7f8f74008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f8d5b2700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f74009580 0x7f8f7400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f8cdb1700 1 -- 172.21.15.32:0/3784017730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f74006780 msgr2=0x7f8f74008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f8cdb1700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f74006780 0x7f8f74008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f8cdb1700 1 -- 172.21.15.32:0/3784017730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f74009580 msgr2=0x7f8f7400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f8cdb1700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f74009580 0x7f8f7400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f8cdb1700 1 -- 172.21.15.32:0/3784017730 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f740509a0 con 0x7f8f74003b10 2021-04-09T18:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f8d5b2700 1 -- 172.21.15.32:0/3784017730 reap_dead start 2021-04-09T18:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f8cdb1700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f74003b10 0x7f8f74005f60 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 rx=0x7f8f780013c0 tx=0x7f8f78009da0).ready entity=mon.0 client_cookie=b5d1470925daeb4a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f78006a60 con 0x7f8f74003b10 2021-04-09T18:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f78001400 con 0x7f8f74003b10 2021-04-09T18:31:33.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.318 7f8f877fe700 1 -- 172.21.15.32:0/3784017730 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f780019a0 con 0x7f8f74003b10 2021-04-09T18:31:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.319 7f8f8ef95700 1 -- 172.21.15.32:0/3784017730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f74003b10 msgr2=0x7f8f74005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.319 7f8f8ef95700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f74003b10 0x7f8f74005f60 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 rx=0x7f8f780013c0 tx=0x7f8f78009da0).stop 2021-04-09T18:31:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.319 7f8f8ef95700 1 -- 172.21.15.32:0/3784017730 shutdown_connections 2021-04-09T18:31:33.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.319 7f8f8ef95700 1 --2- 172.21.15.32:0/3784017730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f74003b10 0x7f8f74005f60 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.319 7f8f8ef95700 1 -- 172.21.15.32:0/3784017730 >> 172.21.15.32:0/3784017730 conn(0x7f8f880ce5c0 msgr2=0x7f8f880cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.319 7f8f8ef95700 1 -- 172.21.15.32:0/3784017730 shutdown_connections 2021-04-09T18:31:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.319 7f8f8ef95700 1 -- 172.21.15.32:0/3784017730 wait complete. 2021-04-09T18:31:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.339 7f8f8ef95700 1 Processor -- start 2021-04-09T18:31:33.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.339 7f8f8ef95700 1 -- start start 2021-04-09T18:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8ef95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f881acff0 0x7f8f881847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8ef95700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f88183fc0 0x7f8f881a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8ef95700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f88186ff0 0x7f8f881873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8ef95700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f88188c30 con 0x7f8f881acff0 2021-04-09T18:31:33.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8ef95700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f881843c0 con 0x7f8f88186ff0 2021-04-09T18:31:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8ef95700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f881a5c20 con 0x7f8f88183fc0 2021-04-09T18:31:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8cdb1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f881acff0 0x7f8f881847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8cdb1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f881acff0 0x7f8f881847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57024/0 (socket says 172.21.15.32:57024) 2021-04-09T18:31:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8cdb1700 1 -- 172.21.15.32:0/1148337508 learned_addr learned my addr 172.21.15.32:0/1148337508 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f87fff700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f88183fc0 0x7f8f881a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.340 7f8f8d5b2700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f88186ff0 0x7f8f881873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f8cdb1700 1 -- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f88183fc0 msgr2=0x7f8f881a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f8cdb1700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f88183fc0 0x7f8f881a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f8cdb1700 1 -- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f88186ff0 msgr2=0x7f8f881873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f8cdb1700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f88186ff0 0x7f8f881873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f8cdb1700 1 -- 172.21.15.32:0/1148337508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f88187a10 con 0x7f8f881acff0 2021-04-09T18:31:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f8cdb1700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f881acff0 0x7f8f881847a0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 rx=0x7f8f780091e0 tx=0x7f8f7800bd30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f857fa700 1 -- 172.21.15.32:0/1148337508 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f780093d0 con 0x7f8f881acff0 2021-04-09T18:31:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f857fa700 1 -- 172.21.15.32:0/1148337508 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f780011d0 con 0x7f8f881acff0 2021-04-09T18:31:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f88187a10 con 0x7f8f881acff0 2021-04-09T18:31:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f857fa700 1 -- 172.21.15.32:0/1148337508 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f78001660 con 0x7f8f881acff0 2021-04-09T18:31:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.341 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f88187a10 con 0x7f8f881acff0 2021-04-09T18:31:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.342 7f8f857fa700 1 -- 172.21.15.32:0/1148337508 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8f780093d0 con 0x7f8f881acff0 2021-04-09T18:31:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.342 7f8f857fa700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f6c018930 0x7f8f6c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.342 7f8f857fa700 1 -- 172.21.15.32:0/1148337508 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f8f780330a0 con 0x7f8f881acff0 2021-04-09T18:31:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.342 7f8f87fff700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f6c018930 0x7f8f6c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.342 7f8f87fff700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f6c018930 0x7f8f6c01ae10 crc :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:33.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.361 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f88182dc0 con 0x7f8f881acff0 2021-04-09T18:31:33.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.365 7f8f857fa700 1 -- 172.21.15.32:0/1148337508 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8f78000ed0 con 0x7f8f881acff0 2021-04-09T18:31:33.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.558 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f880d7280 con 0x7f8f881acff0 2021-04-09T18:31:33.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.558 7f8f857fa700 1 -- 172.21.15.32:0/1148337508 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 353 v353) v1 ==== 95+0+3719 (secure 0 0 0) 0x7f8f780093d0 con 0x7f8f881acff0 2021-04-09T18:31:33.574 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 353 2021-04-09T18:31:33.575 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:33.575 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":353,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2104377376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2104377376},{"type":"v1","addr":"172.21.15.32:6833","nonce":2104377376}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":345},{"gid":11502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4153029254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4153029254},{"type":"v1","addr":"172.21.15.32:6835","nonce":4153029254}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":345}],"filesystems":[{"mdsmap":{"epoch":349,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:25.650475","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11020},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11020":{"gid":11020,"name":"d","rank":0,"incarnation":346,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/1129605860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1129605860},{"type":"v1","addr":"172.21.15.86:6836","nonce":1129605860}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:33.117979","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11023},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11023":{"gid":11023,"name":"b","rank":0,"incarnation":350,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/4017196651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":4017196651},{"type":"v1","addr":"172.21.15.86:6837","nonce":4017196651}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:33.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f6c018930 msgr2=0x7f8f6c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f6c018930 0x7f8f6c01ae10 crc :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f881acff0 msgr2=0x7f8f881847a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f881acff0 0x7f8f881847a0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 rx=0x7f8f780091e0 tx=0x7f8f7800bd30).stop 2021-04-09T18:31:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 shutdown_connections 2021-04-09T18:31:33.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f6c018930 0x7f8f6c01ae10 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f88186ff0 0x7f8f881873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f88183fc0 0x7f8f881a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 --2- 172.21.15.32:0/1148337508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f881acff0 0x7f8f881847a0 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 >> 172.21.15.32:0/1148337508 conn(0x7f8f8809abb0 msgr2=0x7f8f880a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 shutdown_connections 2021-04-09T18:31:33.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.561 7f8f8ef95700 1 -- 172.21.15.32:0/1148337508 wait complete. 2021-04-09T18:31:33.586 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:31:33.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.720 7f041897c700 1 Processor -- start 2021-04-09T18:31:33.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.720 7f041897c700 1 -- start start 2021-04-09T18:31:33.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041897c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04140d7ca0 0x7f04140d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041897c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041410d480 con 0x7f041410d010 2021-04-09T18:31:33.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041897c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04140d86c0 con 0x7f041410da00 2021-04-09T18:31:33.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041897c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04140d9220 con 0x7f04140d7ca0 2021-04-09T18:31:33.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041261c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f041410d010 0x7f04140d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38390/0 (socket says 172.21.15.32:38390) 2021-04-09T18:31:33.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041261c700 1 -- 172.21.15.32:0/4139907956 learned_addr learned my addr 172.21.15.32:0/4139907956 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:33.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f0411e1b700 1 --2- 172.21.15.32:0/4139907956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04140d7ca0 0x7f04140d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041161a700 1 -- 172.21.15.32:0/4139907956 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217754892 0 0) 0x7f0404000ce0 con 0x7f041410d010 2021-04-09T18:31:33.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041161a700 1 -- 172.21.15.32:0/4139907956 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03fc002fe0 con 0x7f041410d010 2021-04-09T18:31:33.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041161a700 1 -- 172.21.15.32:0/4139907956 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393267769 0 0) 0x7f040c0009b0 con 0x7f041410da00 2021-04-09T18:31:33.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.721 7f041161a700 1 -- 172.21.15.32:0/4139907956 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03fc0042c0 con 0x7f041410da00 2021-04-09T18:31:33.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.722 7f0411e1b700 1 -- 172.21.15.32:0/4139907956 >> v1:172.21.15.86:6789/0 conn(0x7f041410da00 legacy=0x7f041410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.722 7f0411e1b700 1 -- 172.21.15.32:0/4139907956 >> v1:172.21.15.32:6789/0 conn(0x7f041410d010 legacy=0x7f04140d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.722 7f0411e1b700 1 -- 172.21.15.32:0/4139907956 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04140d1f70 con 0x7f04140d7ca0 2021-04-09T18:31:33.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.722 7f0411e1b700 1 --2- 172.21.15.32:0/4139907956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04140d7ca0 0x7f04140d80a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f04080069d0 tx=0x7f0408009e40).ready entity=mon.2 client_cookie=7646a0aed4a5c127 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:33.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.722 7f041161a700 1 -- 172.21.15.32:0/4139907956 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f040800b510 con 0x7f04140d7ca0 2021-04-09T18:31:33.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.722 7f041161a700 1 -- 172.21.15.32:0/4139907956 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040800b890 con 0x7f04140d7ca0 2021-04-09T18:31:33.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.722 7f041161a700 1 -- 172.21.15.32:0/4139907956 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f040800bd70 con 0x7f04140d7ca0 2021-04-09T18:31:33.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.722 7f041897c700 1 -- 172.21.15.32:0/4139907956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04140d7ca0 msgr2=0x7f04140d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.722 7f041897c700 1 --2- 172.21.15.32:0/4139907956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04140d7ca0 0x7f04140d80a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f04080069d0 tx=0x7f0408009e40).stop 2021-04-09T18:31:33.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.723 7f041897c700 1 -- 172.21.15.32:0/4139907956 shutdown_connections 2021-04-09T18:31:33.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.723 7f041897c700 1 --2- 172.21.15.32:0/4139907956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04140d7ca0 0x7f04140d80a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.723 7f041897c700 1 -- 172.21.15.32:0/4139907956 >> 172.21.15.32:0/4139907956 conn(0x7f04140ce5f0 msgr2=0x7f04140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:33.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.723 7f041897c700 1 -- 172.21.15.32:0/4139907956 shutdown_connections 2021-04-09T18:31:33.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.723 7f041897c700 1 -- 172.21.15.32:0/4139907956 wait complete. 2021-04-09T18:31:33.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.743 7f041897c700 1 Processor -- start 2021-04-09T18:31:33.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.743 7f041897c700 1 -- start start 2021-04-09T18:31:33.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f041897c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0414185a00 0x7f0414097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f041897c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04140983b0 0x7f041417abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f041897c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041417b1d0 0x7f0414179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f041897c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04140987b0 con 0x7f04140983b0 2021-04-09T18:31:33.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f041897c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041417b5d0 con 0x7f041417b1d0 2021-04-09T18:31:33.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f041897c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0414179c30 con 0x7f0414185a00 2021-04-09T18:31:33.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0411e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04140983b0 0x7f041417abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0411e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04140983b0 0x7f041417abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57042/0 (socket says 172.21.15.32:57042) 2021-04-09T18:31:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0411e1b700 1 -- 172.21.15.32:0/3472656320 learned_addr learned my addr 172.21.15.32:0/3472656320 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:33.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f041261c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0414185a00 0x7f0414097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0412e1d700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041417b1d0 0x7f0414179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0411e1b700 1 -- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0414185a00 msgr2=0x7f0414097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0411e1b700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0414185a00 0x7f0414097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0411e1b700 1 -- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041417b1d0 msgr2=0x7f0414179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0411e1b700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041417b1d0 0x7f0414179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0411e1b700 1 -- 172.21.15.32:0/3472656320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414179e50 con 0x7f04140983b0 2021-04-09T18:31:33.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f041261c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0414185a00 0x7f0414097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:33.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0411e1b700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04140983b0 0x7f041417abb0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 rx=0x7f04080099e0 tx=0x7f040800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:33.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0402ffd700 1 -- 172.21.15.32:0/3472656320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0408006240 con 0x7f04140983b0 2021-04-09T18:31:33.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0402ffd700 1 -- 172.21.15.32:0/3472656320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040800be60 con 0x7f04140983b0 2021-04-09T18:31:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f041897c700 1 -- 172.21.15.32:0/3472656320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0414179e50 con 0x7f04140983b0 2021-04-09T18:31:33.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.744 7f0402ffd700 1 -- 172.21.15.32:0/3472656320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0408005660 con 0x7f04140983b0 2021-04-09T18:31:33.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.745 7f041897c700 1 -- 172.21.15.32:0/3472656320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0414179e50 con 0x7f04140983b0 2021-04-09T18:31:33.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.746 7f0402ffd700 1 -- 172.21.15.32:0/3472656320 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0408000cc0 con 0x7f04140983b0 2021-04-09T18:31:33.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.746 7f0402ffd700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03f80188c0 0x7f03f801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:33.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.746 7f0402ffd700 1 -- 172.21.15.32:0/3472656320 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f0408032e40 con 0x7f04140983b0 2021-04-09T18:31:33.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.746 7f041261c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03f80188c0 0x7f03f801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:33.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.746 7f041261c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03f80188c0 0x7f03f801adc0 crc :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:33.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.765 7f041897c700 1 -- 172.21.15.32:0/3472656320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041417a3e0 con 0x7f04140983b0 2021-04-09T18:31:33.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.769 7f0402ffd700 1 -- 172.21.15.32:0/3472656320 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0408008f10 con 0x7f04140983b0 2021-04-09T18:31:33.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.960 7f041897c700 1 -- 172.21.15.32:0/3472656320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04140987b0 con 0x7f04140983b0 2021-04-09T18:31:33.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.960 7f0402ffd700 1 -- 172.21.15.32:0/3472656320 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f0408000cc0 con 0x7f04140983b0 2021-04-09T18:31:33.976 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:33.976 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":207,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:31:32.114805","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:31:21.254184","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":35,"pool_name":"alpha_data","create_time":"2021-04-09 18:31:22.120246","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":36,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:31:28.904296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"206","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":37,"pool_name":"bravo_data","create_time":"2021-04-09 18:31:30.123936","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:31:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 -- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03f80188c0 msgr2=0x7f03f801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03f80188c0 0x7f03f801adc0 crc :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 -- 172.21.15.32:0/3472656320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04140983b0 msgr2=0x7f041417abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04140983b0 0x7f041417abb0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 rx=0x7f04080099e0 tx=0x7f040800bd20).stop 2021-04-09T18:31:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 -- 172.21.15.32:0/3472656320 shutdown_connections 2021-04-09T18:31:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f03f80188c0 0x7f03f801adc0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04140983b0 0x7f041417abb0 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041417b1d0 0x7f0414179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 --2- 172.21.15.32:0/3472656320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0414185a00 0x7f0414097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 -- 172.21.15.32:0/3472656320 >> 172.21.15.32:0/3472656320 conn(0x7f0414190550 msgr2=0x7f04141b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:33.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 -- 172.21.15.32:0/3472656320 shutdown_connections 2021-04-09T18:31:33.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:33.963 7f041897c700 1 -- 172.21.15.32:0/3472656320 wait complete. 2021-04-09T18:31:33.989 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:31:33.989 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:31:33.989 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:31:33.990 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:31:33.990 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:31:33.990 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:31:33.991 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:31:33.991 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:31:33.991 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:31:33.991 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:31:33.992 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:31:33.992 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:31:33.992 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:31:33.993 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:31:33.993 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:31:33.994 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:31:33.998 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:31:33.998 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:31:34.000 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:31:34.000 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:31:34.001 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:31:34.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:31:34.009 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:31:34.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:31:34.041 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:31:34.042 DEBUG:teuthology.parallel:result is None 2021-04-09T18:31:34.043 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:31:34.043 DEBUG:teuthology.parallel:result is None 2021-04-09T18:31:34.050 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:31:34.051 DEBUG:teuthology.parallel:result is None 2021-04-09T18:31:34.052 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:31:34.052 DEBUG:teuthology.parallel:result is None 2021-04-09T18:31:34.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:34.191 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:31:34.191 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:31:34.193 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:31:34.194 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:31:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.189 7fa4d4205700 1 Processor -- start 2021-04-09T18:31:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.189 7fa4d4205700 1 -- start start 2021-04-09T18:31:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d4205700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc10d010 0x7fa4cc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d4205700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4cc0d8490 0x7fa4cc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:34.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d4205700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4cc10d480 con 0x7fa4cc0d1da0 2021-04-09T18:31:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d4205700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4cc10e780 con 0x7fa4cc0d8490 2021-04-09T18:31:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d4205700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc0d8d30 con 0x7fa4cc10d010 2021-04-09T18:31:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d2822700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4cc0d8490 0x7fa4cc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:34.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d2822700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4cc0d8490 0x7fa4cc10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57072/0 (socket says 172.21.15.32:57072) 2021-04-09T18:31:34.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d2822700 1 -- 172.21.15.32:0/655187194 learned_addr learned my addr 172.21.15.32:0/655187194 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:34.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d2021700 1 --2- 172.21.15.32:0/655187194 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc10d010 0x7fa4cc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:34.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d2822700 1 -- 172.21.15.32:0/655187194 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc10d010 msgr2=0x7fa4cc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:34.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d2822700 1 --2- 172.21.15.32:0/655187194 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc10d010 0x7fa4cc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d2822700 1 -- 172.21.15.32:0/655187194 >> v1:172.21.15.86:6789/0 conn(0x7fa4cc0d1da0 legacy=0x7fa4cc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:34.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d2822700 1 -- 172.21.15.32:0/655187194 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc0d9220 con 0x7fa4cc0d8490 2021-04-09T18:31:34.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d2822700 1 --2- 172.21.15.32:0/655187194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc0d8490 0x7fa4cc10da00 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7fa4c4006f00 tx=0x7fa4c4009f10).ready entity=mon.0 client_cookie=fe7e9b7bda2d3d07 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:34.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.190 7fa4d101f700 1 -- 172.21.15.32:0/655187194 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4c400b3b0 con 0x7fa4cc0d8490 2021-04-09T18:31:34.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.191 7fa4d101f700 1 -- 172.21.15.32:0/655187194 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c400b730 con 0x7fa4cc0d8490 2021-04-09T18:31:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.191 7fa4d101f700 1 -- 172.21.15.32:0/655187194 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4c400bc10 con 0x7fa4cc0d8490 2021-04-09T18:31:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.191 7fa4d4205700 1 -- 172.21.15.32:0/655187194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc0d8490 msgr2=0x7fa4cc10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.191 7fa4d4205700 1 --2- 172.21.15.32:0/655187194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc0d8490 0x7fa4cc10da00 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7fa4c4006f00 tx=0x7fa4c4009f10).stop 2021-04-09T18:31:34.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.192 7fa4d4205700 1 -- 172.21.15.32:0/655187194 shutdown_connections 2021-04-09T18:31:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.192 7fa4d4205700 1 --2- 172.21.15.32:0/655187194 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc0d8490 0x7fa4cc10da00 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.192 7fa4d4205700 1 --2- 172.21.15.32:0/655187194 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc10d010 0x7fa4cc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.192 7fa4d4205700 1 -- 172.21.15.32:0/655187194 >> 172.21.15.32:0/655187194 conn(0x7fa4cc0ce5f0 msgr2=0x7fa4cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.192 7fa4d4205700 1 -- 172.21.15.32:0/655187194 shutdown_connections 2021-04-09T18:31:34.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.192 7fa4d4205700 1 -- 172.21.15.32:0/655187194 wait complete. 2021-04-09T18:31:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.212 7fa4d4205700 1 Processor -- start 2021-04-09T18:31:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.212 7fa4d4205700 1 -- start start 2021-04-09T18:31:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.212 7fa4d4205700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4cc18d9e0 0x7fa4cc187420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.212 7fa4d4205700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc18dde0 0x7fa4cc187a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.212 7fa4d4205700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc1858c0 0x7fa4cc185cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.212 7fa4d4205700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc18e1e0 con 0x7fa4cc1858c0 2021-04-09T18:31:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.212 7fa4d4205700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc186ea0 con 0x7fa4cc18d9e0 2021-04-09T18:31:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.212 7fa4d4205700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc1862e0 con 0x7fa4cc18dde0 2021-04-09T18:31:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2822700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc1858c0 0x7fa4cc185cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:34.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2822700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc1858c0 0x7fa4cc185cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57078/0 (socket says 172.21.15.32:57078) 2021-04-09T18:31:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2822700 1 -- 172.21.15.32:0/2142244983 learned_addr learned my addr 172.21.15.32:0/2142244983 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2021700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4cc18d9e0 0x7fa4cc187420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d1820700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc18dde0 0x7fa4cc187a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:34.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2822700 1 -- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc18dde0 msgr2=0x7fa4cc187a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2822700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc18dde0 0x7fa4cc187a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2822700 1 -- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4cc18d9e0 msgr2=0x7fa4cc187420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2822700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4cc18d9e0 0x7fa4cc187420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2822700 1 -- 172.21.15.32:0/2142244983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc1864c0 con 0x7fa4cc1858c0 2021-04-09T18:31:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d2822700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc1858c0 0x7fa4cc185cc0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 rx=0x7fa4c4009390 tx=0x7fa4c400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4beffd700 1 -- 172.21.15.32:0/2142244983 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4c4003030 con 0x7fa4cc1858c0 2021-04-09T18:31:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4beffd700 1 -- 172.21.15.32:0/2142244983 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c40068e0 con 0x7fa4cc1858c0 2021-04-09T18:31:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4cc1864c0 con 0x7fa4cc1858c0 2021-04-09T18:31:34.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4beffd700 1 -- 172.21.15.32:0/2142244983 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4c400c570 con 0x7fa4cc1858c0 2021-04-09T18:31:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.213 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4cc1864c0 con 0x7fa4cc1858c0 2021-04-09T18:31:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.214 7fa4beffd700 1 -- 172.21.15.32:0/2142244983 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa4c4005820 con 0x7fa4cc1858c0 2021-04-09T18:31:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.214 7fa4beffd700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4b4018930 0x7fa4b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.214 7fa4beffd700 1 -- 172.21.15.32:0/2142244983 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fa4c40330b0 con 0x7fa4cc1858c0 2021-04-09T18:31:34.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.214 7fa4d2021700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4b4018930 0x7fa4b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:34.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.214 7fa4d2021700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4b4018930 0x7fa4b401ae10 crc :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:34.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.233 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4cc18e1e0 con 0x7fa4cc1858c0 2021-04-09T18:31:34.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.236 7fa4beffd700 1 -- 172.21.15.32:0/2142244983 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa4c4006c80 con 0x7fa4cc1858c0 2021-04-09T18:31:34.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.428 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4cc18e1e0 con 0x7fa4cc1858c0 2021-04-09T18:31:34.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.428 7fa4beffd700 1 -- 172.21.15.32:0/2142244983 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 353 v353) v1 ==== 95+0+3719 (secure 0 0 0) 0x7fa4c40068e0 con 0x7fa4cc1858c0 2021-04-09T18:31:34.444 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 353 2021-04-09T18:31:34.444 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:34.445 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":353,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2104377376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2104377376},{"type":"v1","addr":"172.21.15.32:6833","nonce":2104377376}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":345},{"gid":11502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4153029254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4153029254},{"type":"v1","addr":"172.21.15.32:6835","nonce":4153029254}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":345}],"filesystems":[{"mdsmap":{"epoch":349,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:25.650475","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11020},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11020":{"gid":11020,"name":"d","rank":0,"incarnation":346,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/1129605860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1129605860},{"type":"v1","addr":"172.21.15.86:6836","nonce":1129605860}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:33.117979","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11023},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11023":{"gid":11023,"name":"b","rank":0,"incarnation":350,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/4017196651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":4017196651},{"type":"v1","addr":"172.21.15.86:6837","nonce":4017196651}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:34.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4b4018930 msgr2=0x7fa4b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:34.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4b4018930 0x7fa4b401ae10 crc :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc1858c0 msgr2=0x7fa4cc185cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:34.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc1858c0 0x7fa4cc185cc0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 rx=0x7fa4c4009390 tx=0x7fa4c400bd40).stop 2021-04-09T18:31:34.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 shutdown_connections 2021-04-09T18:31:34.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4b4018930 0x7fa4b401ae10 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4cc1858c0 0x7fa4cc185cc0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4cc18dde0 0x7fa4cc187a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 --2- 172.21.15.32:0/2142244983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4cc18d9e0 0x7fa4cc187420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:34.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 >> 172.21.15.32:0/2142244983 conn(0x7fa4cc09abb0 msgr2=0x7fa4cc19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:34.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 shutdown_connections 2021-04-09T18:31:34.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:34.431 7fa4d4205700 1 -- 172.21.15.32:0/2142244983 wait complete. 2021-04-09T18:31:34.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 349, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:31:23.513548', 'modified': '2021-04-09 18:31:25.650475', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11020}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11020': {'gid': 11020, 'name': 'd', 'rank': 0, 'incarnation': 346, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/1129605860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 1129605860}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 1129605860}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:31:34.859 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:31:34.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:31:34.989 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:31:34.990 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:31:34.990 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": -1, 2021-04-09T18:31:34.990 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 59144, 2021-04-09T18:31:34.990 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:boot", 2021-04-09T18:31:34.991 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "null", 2021-04-09T18:31:34.991 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 353, 2021-04-09T18:31:34.991 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 0, 2021-04-09T18:31:34.991 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 0, 2021-04-09T18:31:34.992 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 0.79466932800000001 2021-04-09T18:31:34.992 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:31:34.995 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": -1, "id": 59144, "want_state": "up:boot", "state": "null", "mdsmap_epoch": 353, "osdmap_epoch": 0, "osdmap_epoch_barrier": 0, "uptime": 0.79466932800000001 } 2021-04-09T18:31:35.997 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:36.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.142 7fd0ea5d7700 1 Processor -- start 2021-04-09T18:31:36.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.142 7fd0ea5d7700 1 -- start start 2021-04-09T18:31:36.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.142 7fd0ea5d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e410d010 0x7fd0e40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:36.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.142 7fd0ea5d7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd0e40d1da0 0x7fd0e40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:36.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0ea5d7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e410d480 con 0x7fd0e40d82c0 2021-04-09T18:31:36.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0ea5d7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e410e780 con 0x7fd0e40d1da0 2021-04-09T18:31:36.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0ea5d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e40d8d30 con 0x7fd0e410d010 2021-04-09T18:31:36.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0dbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd0e40d1da0 0x7fd0e40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:36.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0e8bf4700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd0e40d82c0 0x7fd0e410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38444/0 (socket says 172.21.15.32:38444) 2021-04-09T18:31:36.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0e8bf4700 1 -- 172.21.15.32:0/1091497503 learned_addr learned my addr 172.21.15.32:0/1091497503 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:36.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0e3fff700 1 --2- 172.21.15.32:0/1091497503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e410d010 0x7fd0e40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:36.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0dbfff700 1 -- 172.21.15.32:0/1091497503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e410d010 msgr2=0x7fd0e40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:36.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0dbfff700 1 --2- 172.21.15.32:0/1091497503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e410d010 0x7fd0e40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0dbfff700 1 -- 172.21.15.32:0/1091497503 >> v1:172.21.15.32:6789/0 conn(0x7fd0e40d82c0 legacy=0x7fd0e410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:36.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0dbfff700 1 -- 172.21.15.32:0/1091497503 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e40d9220 con 0x7fd0e40d1da0 2021-04-09T18:31:36.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0e37fe700 1 -- 172.21.15.32:0/1091497503 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645426077 0 0) 0x7fd0dc000d30 con 0x7fd0e40d82c0 2021-04-09T18:31:36.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0dbfff700 1 --2- 172.21.15.32:0/1091497503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e40d1da0 0x7fd0e40d7ca0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 rx=0x7fd0cc006650 tx=0x7fd0cc009a00).ready entity=mon.0 client_cookie=3392425340389ed6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:36.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0e37fe700 1 -- 172.21.15.32:0/1091497503 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0cc009df0 con 0x7fd0e40d1da0 2021-04-09T18:31:36.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0e37fe700 1 -- 172.21.15.32:0/1091497503 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0cc0009b0 con 0x7fd0e40d1da0 2021-04-09T18:31:36.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0e37fe700 1 -- 172.21.15.32:0/1091497503 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0cc00c2b0 con 0x7fd0e40d1da0 2021-04-09T18:31:36.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0ea5d7700 1 -- 172.21.15.32:0/1091497503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e40d1da0 msgr2=0x7fd0e40d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:36.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.143 7fd0ea5d7700 1 --2- 172.21.15.32:0/1091497503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e40d1da0 0x7fd0e40d7ca0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 rx=0x7fd0cc006650 tx=0x7fd0cc009a00).stop 2021-04-09T18:31:36.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.144 7fd0ea5d7700 1 -- 172.21.15.32:0/1091497503 shutdown_connections 2021-04-09T18:31:36.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.144 7fd0ea5d7700 1 --2- 172.21.15.32:0/1091497503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e40d1da0 0x7fd0e40d7ca0 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.144 7fd0ea5d7700 1 --2- 172.21.15.32:0/1091497503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e410d010 0x7fd0e40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.144 7fd0ea5d7700 1 -- 172.21.15.32:0/1091497503 >> 172.21.15.32:0/1091497503 conn(0x7fd0e40ce5f0 msgr2=0x7fd0e40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.144 7fd0ea5d7700 1 -- 172.21.15.32:0/1091497503 shutdown_connections 2021-04-09T18:31:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.144 7fd0ea5d7700 1 -- 172.21.15.32:0/1091497503 wait complete. 2021-04-09T18:31:36.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0ea5d7700 1 Processor -- start 2021-04-09T18:31:36.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0ea5d7700 1 -- start start 2021-04-09T18:31:36.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0ea5d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e4179fd0 0x7fd0e40976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0ea5d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4097cc0 0x7fd0e41a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0ea5d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e41a4b10 0x7fd0e41a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:36.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0ea5d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e4098140 con 0x7fd0e41a4b10 2021-04-09T18:31:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0ea5d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e417a3d0 con 0x7fd0e4097cc0 2021-04-09T18:31:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0ea5d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e417c870 con 0x7fd0e4179fd0 2021-04-09T18:31:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0e8bf4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e41a4b10 0x7fd0e41a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:36.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0e8bf4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e41a4b10 0x7fd0e41a4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57094/0 (socket says 172.21.15.32:57094) 2021-04-09T18:31:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0e8bf4700 1 -- 172.21.15.32:0/2403007285 learned_addr learned my addr 172.21.15.32:0/2403007285 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.165 7fd0dbfff700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4097cc0 0x7fd0e41a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e3fff700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e4179fd0 0x7fd0e40976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:36.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e8bf4700 1 -- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e4179fd0 msgr2=0x7fd0e40976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e8bf4700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e4179fd0 0x7fd0e40976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e8bf4700 1 -- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4097cc0 msgr2=0x7fd0e41a44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e8bf4700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4097cc0 0x7fd0e41a44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e8bf4700 1 -- 172.21.15.32:0/2403007285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e4179bf0 con 0x7fd0e41a4b10 2021-04-09T18:31:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0dbfff700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4097cc0 0x7fd0e41a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e8bf4700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e41a4b10 0x7fd0e41a4f10 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 rx=0x7fd0dc007f90 tx=0x7fd0dc00ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e17fa700 1 -- 172.21.15.32:0/2403007285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0dc000cc0 con 0x7fd0e41a4b10 2021-04-09T18:31:36.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e17fa700 1 -- 172.21.15.32:0/2403007285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc007020 con 0x7fd0e41a4b10 2021-04-09T18:31:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e17fa700 1 -- 172.21.15.32:0/2403007285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0dc007500 con 0x7fd0e41a4b10 2021-04-09T18:31:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e4179bf0 con 0x7fd0e41a4b10 2021-04-09T18:31:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e4179bf0 con 0x7fd0e41a4b10 2021-04-09T18:31:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.166 7fd0e17fa700 1 -- 172.21.15.32:0/2403007285 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd0dc007200 con 0x7fd0e41a4b10 2021-04-09T18:31:36.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.167 7fd0e17fa700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0c8018930 0x7fd0c801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:36.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.167 7fd0e17fa700 1 -- 172.21.15.32:0/2403007285 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fd0dc0330a0 con 0x7fd0e41a4b10 2021-04-09T18:31:36.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.167 7fd0e3fff700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0c8018930 0x7fd0c801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:36.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.167 7fd0e3fff700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0c8018930 0x7fd0c801adc0 crc :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:36.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.186 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e4098140 con 0x7fd0e41a4b10 2021-04-09T18:31:36.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.190 7fd0e17fa700 1 -- 172.21.15.32:0/2403007285 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd0dc007d00 con 0x7fd0e41a4b10 2021-04-09T18:31:36.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.381 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0e4098140 con 0x7fd0e41a4b10 2021-04-09T18:31:36.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.382 7fd0e17fa700 1 -- 172.21.15.32:0/2403007285 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 353 v353) v1 ==== 95+0+3719 (secure 0 0 0) 0x7fd0dc007000 con 0x7fd0e41a4b10 2021-04-09T18:31:36.399 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 353 2021-04-09T18:31:36.399 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:36.399 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":353,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11481,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2104377376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2104377376},{"type":"v1","addr":"172.21.15.32:6833","nonce":2104377376}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":345},{"gid":11502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4153029254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":4153029254},{"type":"v1","addr":"172.21.15.32:6835","nonce":4153029254}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":345}],"filesystems":[{"mdsmap":{"epoch":349,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:25.650475","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11020},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11020":{"gid":11020,"name":"d","rank":0,"incarnation":346,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/1129605860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1129605860},{"type":"v1","addr":"172.21.15.86:6836","nonce":1129605860}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:33.117979","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11023},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11023":{"gid":11023,"name":"b","rank":0,"incarnation":350,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/4017196651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":4017196651},{"type":"v1","addr":"172.21.15.86:6837","nonce":4017196651}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0c8018930 msgr2=0x7fd0c801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0c8018930 0x7fd0c801adc0 crc :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e41a4b10 msgr2=0x7fd0e41a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e41a4b10 0x7fd0e41a4f10 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 rx=0x7fd0dc007f90 tx=0x7fd0dc00ac50).stop 2021-04-09T18:31:36.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 shutdown_connections 2021-04-09T18:31:36.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0c8018930 0x7fd0c801adc0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0e41a4b10 0x7fd0e41a4f10 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0e4097cc0 0x7fd0e41a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 --2- 172.21.15.32:0/2403007285 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0e4179fd0 0x7fd0e40976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:36.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 >> 172.21.15.32:0/2403007285 conn(0x7fd0e4199680 msgr2=0x7fd0e41a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:36.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 shutdown_connections 2021-04-09T18:31:36.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:36.384 7fd0ea5d7700 1 -- 172.21.15.32:0/2403007285 wait complete. 2021-04-09T18:31:36.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 349, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:31:23.513548', 'modified': '2021-04-09 18:31:25.650475', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11020}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11020': {'gid': 11020, 'name': 'd', 'rank': 0, 'incarnation': 346, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/1129605860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 1129605860}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 1129605860}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:31:36.814 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:31:36.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:31:36.941 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:31:36.942 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:31:36.942 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": -1, 2021-04-09T18:31:36.943 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 59144, 2021-04-09T18:31:36.943 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:boot", 2021-04-09T18:31:36.943 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "null", 2021-04-09T18:31:36.944 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 353, 2021-04-09T18:31:36.944 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 0, 2021-04-09T18:31:36.945 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 0, 2021-04-09T18:31:36.945 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 2.74634176 2021-04-09T18:31:36.945 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:31:36.947 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": -1, "id": 59144, "want_state": "up:boot", "state": "null", "mdsmap_epoch": 353, "osdmap_epoch": 0, "osdmap_epoch_barrier": 0, "uptime": 2.74634176 } 2021-04-09T18:31:37.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.094 7f65b8f59700 1 Processor -- start 2021-04-09T18:31:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.094 7f65b8f59700 1 -- start start 2021-04-09T18:31:38.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.094 7f65b8f59700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65b410d010 0x7f65b40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b8f59700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b40d8490 0x7f65b410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b8f59700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b410d480 con 0x7f65b40d1da0 2021-04-09T18:31:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b8f59700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b40d8d30 con 0x7f65b410d010 2021-04-09T18:31:38.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b8f59700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40d9220 con 0x7f65b40d8490 2021-04-09T18:31:38.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b2b13700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65b410d010 0x7f65b40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:38.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b2b13700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f65b410d010 0x7f65b40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57098/0 (socket says 172.21.15.32:57098) 2021-04-09T18:31:38.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b2b13700 1 -- 172.21.15.32:0/1137849921 learned_addr learned my addr 172.21.15.32:0/1137849921 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:38.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b3314700 1 --2- 172.21.15.32:0/1137849921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b40d8490 0x7f65b410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b2b13700 1 -- 172.21.15.32:0/1137849921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b40d8490 msgr2=0x7f65b410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b2b13700 1 --2- 172.21.15.32:0/1137849921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b40d8490 0x7f65b410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b2b13700 1 -- 172.21.15.32:0/1137849921 >> v1:172.21.15.86:6789/0 conn(0x7f65b40d1da0 legacy=0x7f65b40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b2b13700 1 -- 172.21.15.32:0/1137849921 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b410e020 con 0x7f65b410d010 2021-04-09T18:31:38.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b3314700 1 --2- 172.21.15.32:0/1137849921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b40d8490 0x7f65b410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b2b13700 1 --2- 172.21.15.32:0/1137849921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b410d010 0x7f65b40d1780 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 rx=0x7f65a4006700 tx=0x7f65a4009b80).ready entity=mon.0 client_cookie=60e19ad0f3711d63 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b1b11700 1 -- 172.21.15.32:0/1137849921 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65a400c0c0 con 0x7f65b410d010 2021-04-09T18:31:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b1b11700 1 -- 172.21.15.32:0/1137849921 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a400c440 con 0x7f65b410d010 2021-04-09T18:31:38.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b1b11700 1 -- 172.21.15.32:0/1137849921 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65a400c920 con 0x7f65b410d010 2021-04-09T18:31:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b8f59700 1 -- 172.21.15.32:0/1137849921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b410d010 msgr2=0x7f65b40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.095 7f65b8f59700 1 --2- 172.21.15.32:0/1137849921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b410d010 0x7f65b40d1780 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 rx=0x7f65a4006700 tx=0x7f65a4009b80).stop 2021-04-09T18:31:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.096 7f65b8f59700 1 -- 172.21.15.32:0/1137849921 shutdown_connections 2021-04-09T18:31:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.096 7f65b8f59700 1 --2- 172.21.15.32:0/1137849921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b40d8490 0x7f65b410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.096 7f65b8f59700 1 --2- 172.21.15.32:0/1137849921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b410d010 0x7f65b40d1780 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.096 7f65b8f59700 1 -- 172.21.15.32:0/1137849921 >> 172.21.15.32:0/1137849921 conn(0x7f65b40ce5f0 msgr2=0x7f65b40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.096 7f65b8f59700 1 -- 172.21.15.32:0/1137849921 shutdown_connections 2021-04-09T18:31:38.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.096 7f65b8f59700 1 -- 172.21.15.32:0/1137849921 wait complete. 2021-04-09T18:31:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.116 7f65b8f59700 1 Processor -- start 2021-04-09T18:31:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.116 7f65b8f59700 1 -- start start 2021-04-09T18:31:38.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b8f59700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b41ac380 0x7f65b418a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b8f59700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b41ac780 0x7f65b418a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b8f59700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65b4183520 0x7f65b4183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b8f59700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b41ace10 con 0x7f65b41ac380 2021-04-09T18:31:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b8f59700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4189cf0 con 0x7f65b4183520 2021-04-09T18:31:38.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b8f59700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4183f40 con 0x7f65b41ac780 2021-04-09T18:31:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b41ac380 0x7f65b418a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2b13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b41ac380 0x7f65b418a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57104/0 (socket says 172.21.15.32:57104) 2021-04-09T18:31:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2b13700 1 -- 172.21.15.32:0/3555347010 learned_addr learned my addr 172.21.15.32:0/3555347010 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:38.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b3314700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65b4183520 0x7f65b4183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2312700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b41ac780 0x7f65b418a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2b13700 1 -- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b41ac780 msgr2=0x7f65b418a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2b13700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b41ac780 0x7f65b418a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2b13700 1 -- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65b4183520 msgr2=0x7f65b4183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:38.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2b13700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65b4183520 0x7f65b4183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2b13700 1 -- 172.21.15.32:0/3555347010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b4184160 con 0x7f65b41ac380 2021-04-09T18:31:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b3314700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65b4183520 0x7f65b4183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2312700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b41ac780 0x7f65b418a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:38.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.117 7f65b2b13700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b41ac380 0x7f65b418a230 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 rx=0x7f65a400b5e0 tx=0x7f65a400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.118 7f65a37fe700 1 -- 172.21.15.32:0/3555347010 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65a400bd70 con 0x7f65b41ac380 2021-04-09T18:31:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.118 7f65a37fe700 1 -- 172.21.15.32:0/3555347010 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a4002030 con 0x7f65b41ac380 2021-04-09T18:31:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.118 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b4184160 con 0x7f65b41ac380 2021-04-09T18:31:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.118 7f65a37fe700 1 -- 172.21.15.32:0/3555347010 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65a40009b0 con 0x7f65b41ac380 2021-04-09T18:31:38.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.118 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b4184160 con 0x7f65b41ac380 2021-04-09T18:31:38.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.118 7f65a37fe700 1 -- 172.21.15.32:0/3555347010 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f65a4002210 con 0x7f65b41ac380 2021-04-09T18:31:38.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.118 7f65a37fe700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65980188c0 0x7f659801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:38.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.118 7f65a37fe700 1 -- 172.21.15.32:0/3555347010 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f65a4032d10 con 0x7f65b41ac380 2021-04-09T18:31:38.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.119 7f65b2312700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65980188c0 0x7f659801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:38.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.119 7f65b2312700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65980188c0 0x7f659801adc0 crc :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:38.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.139 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b4182d10 con 0x7f65b41ac380 2021-04-09T18:31:38.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.142 7f65a37fe700 1 -- 172.21.15.32:0/3555347010 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f65a40009b0 con 0x7f65b41ac380 2021-04-09T18:31:38.246 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:31:38.230 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 2 filesystems are offline (MDS_ALL_DOWN) 2021-04-09T18:31:38.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:31:38.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:31:38.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.343 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65b4182d10 con 0x7f65b41ac380 2021-04-09T18:31:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.343 7f65a37fe700 1 -- 172.21.15.32:0/3555347010 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 355 v355) v1 ==== 95+0+3720 (secure 0 0 0) 0x7f65a400bd70 con 0x7f65b41ac380 2021-04-09T18:31:38.360 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 355 2021-04-09T18:31:38.360 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:38.361 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":355,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354},{"gid":59147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":355,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:38.234494","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:replay","state_seq":2,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":355,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:38.234494","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:replay","state_seq":2,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:38.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65980188c0 msgr2=0x7f659801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:38.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65980188c0 0x7f659801adc0 crc :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b41ac380 msgr2=0x7f65b418a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:38.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b41ac380 0x7f65b418a230 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 rx=0x7f65a400b5e0 tx=0x7f65a400b980).stop 2021-04-09T18:31:38.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 shutdown_connections 2021-04-09T18:31:38.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f65980188c0 0x7f659801adc0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65b4183520 0x7f65b4183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65b41ac780 0x7f65b418a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 --2- 172.21.15.32:0/3555347010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65b41ac380 0x7f65b418a230 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:38.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 >> 172.21.15.32:0/3555347010 conn(0x7f65b419b890 msgr2=0x7f65b41a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:38.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 shutdown_connections 2021-04-09T18:31:38.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:38.346 7f65b8f59700 1 -- 172.21.15.32:0/3555347010 wait complete. 2021-04-09T18:31:38.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 355, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:31:23.513548', 'modified': '2021-04-09 18:31:38.234494', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 211, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11697}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11697': {'gid': 11697, 'name': 'a', 'rank': 0, 'incarnation': 355, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/358744459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 358744459}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 358744459}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:31:38.776 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_11697:up:replay 2021-04-09T18:31:39.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.923 7f3130594700 1 Processor -- start 2021-04-09T18:31:39.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.923 7f3130594700 1 -- start start 2021-04-09T18:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f3130594700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31280d7ca0 0x7f31280d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:39.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f3130594700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f312810da00 0x7f312810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f3130594700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312810d480 con 0x7f312810d010 2021-04-09T18:31:39.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f3130594700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31280d9220 con 0x7f31280d7ca0 2021-04-09T18:31:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f3130594700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31280d86c0 con 0x7f312810da00 2021-04-09T18:31:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312dbaf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31280d7ca0 0x7f31280d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:39.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312dbaf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f31280d7ca0 0x7f31280d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57136/0 (socket says 172.21.15.32:57136) 2021-04-09T18:31:39.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312dbaf700 1 -- 172.21.15.32:0/163387028 learned_addr learned my addr 172.21.15.32:0/163387028 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:39.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312ebb1700 1 --2- 172.21.15.32:0/163387028 >> v2:172.21.15.86:3300/0 conn(0x7f312810da00 0x7f312810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:39.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312dbaf700 1 -- 172.21.15.32:0/163387028 >> v2:172.21.15.86:3300/0 conn(0x7f312810da00 msgr2=0x7f312810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:39.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312dbaf700 1 --2- 172.21.15.32:0/163387028 >> v2:172.21.15.86:3300/0 conn(0x7f312810da00 0x7f312810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:39.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312dbaf700 1 -- 172.21.15.32:0/163387028 >> v1:172.21.15.86:6789/0 conn(0x7f312810d010 legacy=0x7f31280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:39.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312dbaf700 1 -- 172.21.15.32:0/163387028 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31280d1f70 con 0x7f31280d7ca0 2021-04-09T18:31:39.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312d3ae700 1 -- 172.21.15.32:0/163387028 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063136217 0 0) 0x7f3124000fe0 con 0x7f312810d010 2021-04-09T18:31:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312dbaf700 1 --2- 172.21.15.32:0/163387028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31280d7ca0 0x7f31280d80a0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 rx=0x7f3118006700 tx=0x7f3118009b80).ready entity=mon.0 client_cookie=ed133a545e6bdf3e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:39.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312d3ae700 1 -- 172.21.15.32:0/163387028 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f311800c0c0 con 0x7f31280d7ca0 2021-04-09T18:31:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.924 7f312d3ae700 1 -- 172.21.15.32:0/163387028 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311800c440 con 0x7f31280d7ca0 2021-04-09T18:31:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.925 7f312d3ae700 1 -- 172.21.15.32:0/163387028 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f311800c920 con 0x7f31280d7ca0 2021-04-09T18:31:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.925 7f3130594700 1 -- 172.21.15.32:0/163387028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31280d7ca0 msgr2=0x7f31280d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.925 7f3130594700 1 --2- 172.21.15.32:0/163387028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31280d7ca0 0x7f31280d80a0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 rx=0x7f3118006700 tx=0x7f3118009b80).stop 2021-04-09T18:31:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.925 7f3130594700 1 -- 172.21.15.32:0/163387028 shutdown_connections 2021-04-09T18:31:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.925 7f3130594700 1 --2- 172.21.15.32:0/163387028 >> v2:172.21.15.86:3300/0 conn(0x7f312810da00 0x7f312810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.925 7f3130594700 1 --2- 172.21.15.32:0/163387028 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31280d7ca0 0x7f31280d80a0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.925 7f3130594700 1 -- 172.21.15.32:0/163387028 >> 172.21.15.32:0/163387028 conn(0x7f31280ce5f0 msgr2=0x7f31280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.926 7f3130594700 1 -- 172.21.15.32:0/163387028 shutdown_connections 2021-04-09T18:31:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.926 7f3130594700 1 -- 172.21.15.32:0/163387028 wait complete. 2021-04-09T18:31:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f3130594700 1 Processor -- start 2021-04-09T18:31:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f3130594700 1 -- start start 2021-04-09T18:31:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f3130594700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31281a5210 0x7f31281a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f3130594700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31281a6340 0x7f31281a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f3130594700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31280abaa0 0x7f31280abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f3130594700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281a6d60 con 0x7f31281a5210 2021-04-09T18:31:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f3130594700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31280ac4c0 con 0x7f31280abaa0 2021-04-09T18:31:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f3130594700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31280ac6e0 con 0x7f31281a6340 2021-04-09T18:31:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f312e3b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31281a5210 0x7f31281a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f312e3b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31281a5210 0x7f31281a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57138/0 (socket says 172.21.15.32:57138) 2021-04-09T18:31:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.946 7f312e3b0700 1 -- 172.21.15.32:0/2911889924 learned_addr learned my addr 172.21.15.32:0/2911889924 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f312ebb1700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31280abaa0 0x7f31280abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f312dbaf700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31281a6340 0x7f31281a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:39.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f312e3b0700 1 -- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31281a6340 msgr2=0x7f31281a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:39.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f312e3b0700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31281a6340 0x7f31281a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:39.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f312e3b0700 1 -- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31280abaa0 msgr2=0x7f31280abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:39.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f312e3b0700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31280abaa0 0x7f31280abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:39.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f312e3b0700 1 -- 172.21.15.32:0/2911889924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312817a720 con 0x7f31281a5210 2021-04-09T18:31:39.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f312e3b0700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31281a5210 0x7f31281a56c0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 rx=0x7f3124009980 tx=0x7f312400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:39.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f311effd700 1 -- 172.21.15.32:0/2911889924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3124003030 con 0x7f31281a5210 2021-04-09T18:31:39.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f311effd700 1 -- 172.21.15.32:0/2911889924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3124001400 con 0x7f31281a5210 2021-04-09T18:31:39.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f3130594700 1 -- 172.21.15.32:0/2911889924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312817a9c0 con 0x7f31281a5210 2021-04-09T18:31:39.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f311effd700 1 -- 172.21.15.32:0/2911889924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3124001850 con 0x7f31281a5210 2021-04-09T18:31:39.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.947 7f3130594700 1 -- 172.21.15.32:0/2911889924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312817a9c0 con 0x7f31281a5210 2021-04-09T18:31:39.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.948 7f311effd700 1 -- 172.21.15.32:0/2911889924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f31240015e0 con 0x7f31281a5210 2021-04-09T18:31:39.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.948 7f311effd700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3110018930 0x7f311001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:39.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.948 7f311effd700 1 -- 172.21.15.32:0/2911889924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f3124033180 con 0x7f31281a5210 2021-04-09T18:31:39.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.948 7f312dbaf700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3110018930 0x7f311001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:39.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.948 7f312dbaf700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3110018930 0x7f311001ae10 crc :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:39.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.968 7f3130594700 1 -- 172.21.15.32:0/2911889924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31281a6d60 con 0x7f31281a5210 2021-04-09T18:31:39.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:39.971 7f311effd700 1 -- 172.21.15.32:0/2911889924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f312400be10 con 0x7f31281a5210 2021-04-09T18:31:40.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.163 7f3130594700 1 -- 172.21.15.32:0/2911889924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31281a6d60 con 0x7f31281a5210 2021-04-09T18:31:40.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.163 7f311effd700 1 -- 172.21.15.32:0/2911889924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 356 v356) v1 ==== 95+0+3726 (secure 0 0 0) 0x7f3124001400 con 0x7f31281a5210 2021-04-09T18:31:40.181 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 356 2021-04-09T18:31:40.181 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:40.182 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":356,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354},{"gid":59147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":356,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:39.236451","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":356,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:39.236451","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:40.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 -- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3110018930 msgr2=0x7f311001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:40.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3110018930 0x7f311001ae10 crc :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:40.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 -- 172.21.15.32:0/2911889924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31281a5210 msgr2=0x7f31281a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:40.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31281a5210 0x7f31281a56c0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 rx=0x7f3124009980 tx=0x7f312400bcd0).stop 2021-04-09T18:31:40.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 -- 172.21.15.32:0/2911889924 shutdown_connections 2021-04-09T18:31:40.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3110018930 0x7f311001ae10 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:40.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31280abaa0 0x7f31280abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:40.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31281a6340 0x7f31281a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:40.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 --2- 172.21.15.32:0/2911889924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f31281a5210 0x7f31281a56c0 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:40.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 -- 172.21.15.32:0/2911889924 >> 172.21.15.32:0/2911889924 conn(0x7f31281a99c0 msgr2=0x7f312817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 -- 172.21.15.32:0/2911889924 shutdown_connections 2021-04-09T18:31:40.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:40.166 7f3130594700 1 -- 172.21.15.32:0/2911889924 wait complete. 2021-04-09T18:31:40.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 356, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:31:23.513548', 'modified': '2021-04-09 18:31:39.236451', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 211, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11697}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11697': {'gid': 11697, 'name': 'a', 'rank': 0, 'incarnation': 355, 'state': 'up:reconnect', 'state_seq': 3, 'addr': '172.21.15.32:6834/358744459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 358744459}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 358744459}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:31:40.596 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_11697:up:reconnect 2021-04-09T18:31:41.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:41.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.735 7f75efdab700 1 Processor -- start 2021-04-09T18:31:41.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75efdab700 1 -- start start 2021-04-09T18:31:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75efdab700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80cffa0 0x7f75e80d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75efdab700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f75e80d7800 0x7f75e80d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:41.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75efdab700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75e80d5c70 0x7f75e80d6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75efdab700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e80d8580 con 0x7f75e80d5c70 2021-04-09T18:31:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75efdab700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e80d91e0 con 0x7f75e80d7800 2021-04-09T18:31:41.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75efdab700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e80d7280 con 0x7f75e80cffa0 2021-04-09T18:31:41.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75ee3c8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75e80d5c70 0x7f75e80d6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:41.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75ee3c8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75e80d5c70 0x7f75e80d6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57150/0 (socket says 172.21.15.32:57150) 2021-04-09T18:31:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75ee3c8700 1 -- 172.21.15.32:0/1527416743 learned_addr learned my addr 172.21.15.32:0/1527416743 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75ed3c6700 1 --2- 172.21.15.32:0/1527416743 >> v2:172.21.15.86:3300/0 conn(0x7f75e80d7800 0x7f75e80d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:41.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.736 7f75edbc7700 1 --2- 172.21.15.32:0/1527416743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80cffa0 0x7f75e80d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75ee3c8700 1 -- 172.21.15.32:0/1527416743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80cffa0 msgr2=0x7f75e80d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:41.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75ee3c8700 1 --2- 172.21.15.32:0/1527416743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80cffa0 0x7f75e80d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75ee3c8700 1 -- 172.21.15.32:0/1527416743 >> v2:172.21.15.86:3300/0 conn(0x7f75e80d7800 msgr2=0x7f75e80d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:41.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75ee3c8700 1 --2- 172.21.15.32:0/1527416743 >> v2:172.21.15.86:3300/0 conn(0x7f75e80d7800 0x7f75e80d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75ee3c8700 1 -- 172.21.15.32:0/1527416743 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e80d8220 con 0x7f75e80d5c70 2021-04-09T18:31:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75ee3c8700 1 --2- 172.21.15.32:0/1527416743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e80d5c70 0x7f75e80d6070 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 rx=0x7f75e0006f00 tx=0x7f75e0009e90).ready entity=mon.0 client_cookie=621608f020f3123b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:41.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75ecbc5700 1 -- 172.21.15.32:0/1527416743 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75e000b300 con 0x7f75e80d5c70 2021-04-09T18:31:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75ecbc5700 1 -- 172.21.15.32:0/1527416743 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e000b640 con 0x7f75e80d5c70 2021-04-09T18:31:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75ecbc5700 1 -- 172.21.15.32:0/1527416743 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75e000bb20 con 0x7f75e80d5c70 2021-04-09T18:31:41.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75efdab700 1 -- 172.21.15.32:0/1527416743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e80d5c70 msgr2=0x7f75e80d6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.737 7f75efdab700 1 --2- 172.21.15.32:0/1527416743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e80d5c70 0x7f75e80d6070 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 rx=0x7f75e0006f00 tx=0x7f75e0009e90).stop 2021-04-09T18:31:41.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.738 7f75efdab700 1 -- 172.21.15.32:0/1527416743 shutdown_connections 2021-04-09T18:31:41.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.738 7f75efdab700 1 --2- 172.21.15.32:0/1527416743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e80d5c70 0x7f75e80d6070 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.738 7f75efdab700 1 --2- 172.21.15.32:0/1527416743 >> v2:172.21.15.86:3300/0 conn(0x7f75e80d7800 0x7f75e80d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.738 7f75efdab700 1 --2- 172.21.15.32:0/1527416743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80cffa0 0x7f75e80d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.738 7f75efdab700 1 -- 172.21.15.32:0/1527416743 >> 172.21.15.32:0/1527416743 conn(0x7f75e80ce5c0 msgr2=0x7f75e80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:41.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.738 7f75efdab700 1 -- 172.21.15.32:0/1527416743 shutdown_connections 2021-04-09T18:31:41.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.738 7f75efdab700 1 -- 172.21.15.32:0/1527416743 wait complete. 2021-04-09T18:31:41.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.758 7f75efdab700 1 Processor -- start 2021-04-09T18:31:41.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.758 7f75efdab700 1 -- start start 2021-04-09T18:31:41.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75efdab700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81acff0 0x7f75e81847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:41.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75efdab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e8183fc0 0x7f75e81a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:41.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75efdab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8186ff0 0x7f75e81873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:41.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75efdab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e8188c30 con 0x7f75e8186ff0 2021-04-09T18:31:41.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75efdab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e81843c0 con 0x7f75e8183fc0 2021-04-09T18:31:41.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75efdab700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e81a5c20 con 0x7f75e81acff0 2021-04-09T18:31:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ee3c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8186ff0 0x7f75e81873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:41.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ee3c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8186ff0 0x7f75e81873f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57158/0 (socket says 172.21.15.32:57158) 2021-04-09T18:31:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ee3c8700 1 -- 172.21.15.32:0/1470518746 learned_addr learned my addr 172.21.15.32:0/1470518746 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ed3c6700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e8183fc0 0x7f75e81a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:41.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75edbc7700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81acff0 0x7f75e81847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ee3c8700 1 -- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81acff0 msgr2=0x7f75e81847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:41.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ee3c8700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81acff0 0x7f75e81847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ee3c8700 1 -- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e8183fc0 msgr2=0x7f75e81a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:41.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ee3c8700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e8183fc0 0x7f75e81a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ee3c8700 1 -- 172.21.15.32:0/1470518746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e8187a10 con 0x7f75e8186ff0 2021-04-09T18:31:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ed3c6700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e8183fc0 0x7f75e81a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:41.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75ee3c8700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8186ff0 0x7f75e81873f0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 rx=0x7f75e000b9b0 tx=0x7f75e000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:41.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75de7fc700 1 -- 172.21.15.32:0/1470518746 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75e0005820 con 0x7f75e8186ff0 2021-04-09T18:31:41.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75de7fc700 1 -- 172.21.15.32:0/1470518746 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e0005ae0 con 0x7f75e8186ff0 2021-04-09T18:31:41.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75efdab700 1 -- 172.21.15.32:0/1470518746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e8187a10 con 0x7f75e8186ff0 2021-04-09T18:31:41.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75de7fc700 1 -- 172.21.15.32:0/1470518746 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75e0006030 con 0x7f75e8186ff0 2021-04-09T18:31:41.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.759 7f75efdab700 1 -- 172.21.15.32:0/1470518746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e8187a10 con 0x7f75e8186ff0 2021-04-09T18:31:41.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.760 7f75de7fc700 1 -- 172.21.15.32:0/1470518746 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f75e0006030 con 0x7f75e8186ff0 2021-04-09T18:31:41.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.760 7f75de7fc700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d0018930 0x7f75d001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:41.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.760 7f75de7fc700 1 -- 172.21.15.32:0/1470518746 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f75e0033180 con 0x7f75e8186ff0 2021-04-09T18:31:41.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.760 7f75edbc7700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d0018930 0x7f75d001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:41.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.761 7f75edbc7700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d0018930 0x7f75d001adc0 crc :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:41.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.780 7f75efdab700 1 -- 172.21.15.32:0/1470518746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e8182dc0 con 0x7f75e8186ff0 2021-04-09T18:31:41.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.784 7f75de7fc700 1 -- 172.21.15.32:0/1470518746 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f75e0005ae0 con 0x7f75e8186ff0 2021-04-09T18:31:41.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.975 7f75efdab700 1 -- 172.21.15.32:0/1470518746 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75e80d7280 con 0x7f75e8186ff0 2021-04-09T18:31:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.975 7f75de7fc700 1 -- 172.21.15.32:0/1470518746 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 358 v358) v1 ==== 95+0+3720 (secure 0 0 0) 0x7f75e0006030 con 0x7f75e8186ff0 2021-04-09T18:31:41.992 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 358 2021-04-09T18:31:41.992 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:41.992 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":358,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354},{"gid":59147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":358,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:41.244086","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":358,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:41.244086","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 -- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d0018930 msgr2=0x7f75d001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d0018930 0x7f75d001adc0 crc :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 -- 172.21.15.32:0/1470518746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8186ff0 msgr2=0x7f75e81873f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8186ff0 0x7f75e81873f0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 rx=0x7f75e000b9b0 tx=0x7f75e000bd50).stop 2021-04-09T18:31:41.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 -- 172.21.15.32:0/1470518746 shutdown_connections 2021-04-09T18:31:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d0018930 0x7f75d001adc0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8186ff0 0x7f75e81873f0 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e8183fc0 0x7f75e81a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 --2- 172.21.15.32:0/1470518746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81acff0 0x7f75e81847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 -- 172.21.15.32:0/1470518746 >> 172.21.15.32:0/1470518746 conn(0x7f75e809abb0 msgr2=0x7f75e80a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 -- 172.21.15.32:0/1470518746 shutdown_connections 2021-04-09T18:31:41.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:41.978 7f75efdab700 1 -- 172.21.15.32:0/1470518746 wait complete. 2021-04-09T18:31:42.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 358, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:31:23.513548', 'modified': '2021-04-09 18:31:41.244086', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 211, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11697}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11697': {'gid': 11697, 'name': 'a', 'rank': 0, 'incarnation': 355, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6834/358744459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 358744459}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 358744459}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:31:42.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:31:42.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:31:42.546 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:31:42.547 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:31:42.547 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:31:42.547 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 11697, 2021-04-09T18:31:42.548 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:31:42.548 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:31:42.548 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 4.2950927920000002, 2021-04-09T18:31:42.548 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 358, 2021-04-09T18:31:42.549 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 211, 2021-04-09T18:31:42.549 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 211, 2021-04-09T18:31:42.549 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 8.3541173799999999 2021-04-09T18:31:42.549 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:31:42.553 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 11697, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.2950927920000002, "mdsmap_epoch": 358, "osdmap_epoch": 211, "osdmap_epoch_barrier": 211, "uptime": 8.3541173799999999 } 2021-04-09T18:31:42.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:42.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.686 7f1f4188d700 1 Processor -- start 2021-04-09T18:31:42.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.686 7f1f4188d700 1 -- start start 2021-04-09T18:31:42.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.686 7f1f4188d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1f3c10d010 0x7f1f3c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.686 7f1f4188d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c0d1da0 0x7f1f3c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.686 7f1f4188d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c10d480 con 0x7f1f3c0d82c0 2021-04-09T18:31:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f4188d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c0d9220 con 0x7f1f3c10d010 2021-04-09T18:31:42.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f4188d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0d86c0 con 0x7f1f3c0d1da0 2021-04-09T18:31:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3b574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1f3c10d010 0x7f1f3c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3b574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1f3c10d010 0x7f1f3c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57162/0 (socket says 172.21.15.32:57162) 2021-04-09T18:31:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3b574700 1 -- 172.21.15.32:0/3263843836 learned_addr learned my addr 172.21.15.32:0/3263843836 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:42.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3ad73700 1 --2- 172.21.15.32:0/3263843836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c0d1da0 0x7f1f3c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:42.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3b574700 1 -- 172.21.15.32:0/3263843836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c0d1da0 msgr2=0x7f1f3c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:42.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3b574700 1 --2- 172.21.15.32:0/3263843836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c0d1da0 0x7f1f3c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3b574700 1 -- 172.21.15.32:0/3263843836 >> v1:172.21.15.86:6789/0 conn(0x7f1f3c0d82c0 legacy=0x7f1f3c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3b574700 1 -- 172.21.15.32:0/3263843836 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c10e1f0 con 0x7f1f3c10d010 2021-04-09T18:31:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3b574700 1 --2- 172.21.15.32:0/3263843836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c10d010 0x7f1f3c0d1780 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 rx=0x7f1f34006a80 tx=0x7f1f34009e80).ready entity=mon.0 client_cookie=4b4919092af22b6c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3a572700 1 -- 172.21.15.32:0/3263843836 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f3400b510 con 0x7f1f3c10d010 2021-04-09T18:31:42.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3a572700 1 -- 172.21.15.32:0/3263843836 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3400b7d0 con 0x7f1f3c10d010 2021-04-09T18:31:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f3a572700 1 -- 172.21.15.32:0/3263843836 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f3400bcb0 con 0x7f1f3c10d010 2021-04-09T18:31:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f4188d700 1 -- 172.21.15.32:0/3263843836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c10d010 msgr2=0x7f1f3c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:42.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.687 7f1f4188d700 1 --2- 172.21.15.32:0/3263843836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c10d010 0x7f1f3c0d1780 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 rx=0x7f1f34006a80 tx=0x7f1f34009e80).stop 2021-04-09T18:31:42.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.688 7f1f4188d700 1 -- 172.21.15.32:0/3263843836 shutdown_connections 2021-04-09T18:31:42.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.688 7f1f4188d700 1 --2- 172.21.15.32:0/3263843836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c0d1da0 0x7f1f3c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.688 7f1f4188d700 1 --2- 172.21.15.32:0/3263843836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c10d010 0x7f1f3c0d1780 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.688 7f1f4188d700 1 -- 172.21.15.32:0/3263843836 >> 172.21.15.32:0/3263843836 conn(0x7f1f3c0ce5f0 msgr2=0x7f1f3c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:42.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.688 7f1f4188d700 1 -- 172.21.15.32:0/3263843836 shutdown_connections 2021-04-09T18:31:42.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.688 7f1f4188d700 1 -- 172.21.15.32:0/3263843836 wait complete. 2021-04-09T18:31:42.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.708 7f1f4188d700 1 Processor -- start 2021-04-09T18:31:42.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.708 7f1f4188d700 1 -- start start 2021-04-09T18:31:42.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f4188d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c0b42d0 0x7f1f3c183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:42.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f4188d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f3c1ac090 0x7f1f3c1ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:42.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f4188d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c184d30 0x7f1f3c185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:42.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f4188d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0a1a90 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f4188d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c1acab0 con 0x7f1f3c1ac090 2021-04-09T18:31:42.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f4188d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c185a00 con 0x7f1f3c184d30 2021-04-09T18:31:42.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c0b42d0 0x7f1f3c183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:42.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c0b42d0 0x7f1f3c183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57168/0 (socket says 172.21.15.32:57168) 2021-04-09T18:31:42.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3b574700 1 -- 172.21.15.32:0/3734474002 learned_addr learned my addr 172.21.15.32:0/3734474002 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:42.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3ad73700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f3c1ac090 0x7f1f3c1ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:42.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3bd75700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c184d30 0x7f1f3c185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:42.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3b574700 1 -- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c184d30 msgr2=0x7f1f3c185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:42.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3b574700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c184d30 0x7f1f3c185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3b574700 1 -- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f3c1ac090 msgr2=0x7f1f3c1ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:42.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3b574700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f3c1ac090 0x7f1f3c1ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.709 7f1f3b574700 1 -- 172.21.15.32:0/3734474002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c185750 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.710 7f1f3b574700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c0b42d0 0x7f1f3c183520 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 rx=0x7f1f340099b0 tx=0x7f1f3400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:42.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.710 7f1f33fff700 1 -- 172.21.15.32:0/3734474002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f34006030 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.710 7f1f33fff700 1 -- 172.21.15.32:0/3734474002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f34006330 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.710 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c185750 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.710 7f1f33fff700 1 -- 172.21.15.32:0/3734474002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f34005660 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.710 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c185750 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.710 7f1f33fff700 1 -- 172.21.15.32:0/3734474002 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1f34006030 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.710 7f1f33fff700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f20018930 0x7f1f2001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:42.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.710 7f1f33fff700 1 -- 172.21.15.32:0/3734474002 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f1f34033140 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.711 7f1f3ad73700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f20018930 0x7f1f2001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:42.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.711 7f1f3ad73700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f20018930 0x7f1f2001ae10 crc :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:42.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.731 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c182d10 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.734 7f1f33fff700 1 -- 172.21.15.32:0/3734474002 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1f3400be10 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.926 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f3c10d480 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.927 7f1f33fff700 1 -- 172.21.15.32:0/3734474002 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 358 v358) v1 ==== 95+0+3720 (secure 0 0 0) 0x7f1f34006030 con 0x7f1f3c0b42d0 2021-04-09T18:31:42.944 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 358 2021-04-09T18:31:42.944 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:42.944 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":358,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354},{"gid":59147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":358,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:41.244086","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":358,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:41.244086","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f20018930 msgr2=0x7f1f2001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f20018930 0x7f1f2001ae10 crc :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c0b42d0 msgr2=0x7f1f3c183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c0b42d0 0x7f1f3c183520 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 rx=0x7f1f340099b0 tx=0x7f1f3400bcd0).stop 2021-04-09T18:31:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 shutdown_connections 2021-04-09T18:31:42.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f20018930 0x7f1f2001ae10 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f3c184d30 0x7f1f3c185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f3c1ac090 0x7f1f3c1ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 --2- 172.21.15.32:0/3734474002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f3c0b42d0 0x7f1f3c183520 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 >> 172.21.15.32:0/3734474002 conn(0x7f1f3c19b890 msgr2=0x7f1f3c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 shutdown_connections 2021-04-09T18:31:42.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:42.929 7f1f4188d700 1 -- 172.21.15.32:0/3734474002 wait complete. 2021-04-09T18:31:43.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 358, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:31:31.113459', 'modified': '2021-04-09 18:31:41.244086', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 210, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11700}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11700': {'gid': 11700, 'name': 'c', 'rank': 0, 'incarnation': 355, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6835/3816325339', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 3816325339}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3816325339}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'bravo', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:31:43.359 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:31:43.359 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok status 2021-04-09T18:31:43.497 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:31:43.497 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:31:43.498 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:31:43.498 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 11700, 2021-04-09T18:31:43.498 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:31:43.498 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:31:43.499 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 5.2459180099999996, 2021-04-09T18:31:43.499 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 358, 2021-04-09T18:31:43.499 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 211, 2021-04-09T18:31:43.500 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 211, 2021-04-09T18:31:43.500 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 9.3044589349999995 2021-04-09T18:31:43.500 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:31:43.503 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 11700, "want_state": "up:active", "state": "up:active", "rank_uptime": 5.2459180099999996, "mdsmap_epoch": 358, "osdmap_epoch": 211, "osdmap_epoch_barrier": 211, "uptime": 9.3044589349999995 } 2021-04-09T18:31:43.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=alpha_data, allow rw pool=bravo_data' 2021-04-09T18:31:43.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.647 7ff6046ea700 1 Processor -- start 2021-04-09T18:31:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.648 7ff6046ea700 1 -- start start 2021-04-09T18:31:43.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.648 7ff6046ea700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff5fc0d2700 0x7ff5fc0cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.648 7ff6046ea700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5fc0f28c0 0x7ff5fc0d7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.648 7ff6046ea700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0d8700 con 0x7ff5fc0d8270 2021-04-09T18:31:43.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.648 7ff6046ea700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc0d91b0 con 0x7ff5fc0f28c0 2021-04-09T18:31:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.648 7ff6046ea700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc1b0ce0 con 0x7ff5fc0d2700 2021-04-09T18:31:43.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.648 7ff601d05700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5fc0f28c0 0x7ff5fc0d7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.648 7ff602d07700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff5fc0d8270 0x7ff5fc1b04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38534/0 (socket says 172.21.15.32:38534) 2021-04-09T18:31:43.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.648 7ff602d07700 1 -- 172.21.15.32:0/4093992360 learned_addr learned my addr 172.21.15.32:0/4093992360 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff602506700 1 --2- 172.21.15.32:0/4093992360 >> v2:172.21.15.86:3300/0 conn(0x7ff5fc0d2700 0x7ff5fc0cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:43.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601504700 1 -- 172.21.15.32:0/4093992360 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423996383 0 0) 0x7ff5f4000d30 con 0x7ff5fc0d8270 2021-04-09T18:31:43.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601504700 1 -- 172.21.15.32:0/4093992360 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e8002fe0 con 0x7ff5fc0d8270 2021-04-09T18:31:43.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601d05700 1 -- 172.21.15.32:0/4093992360 >> v2:172.21.15.86:3300/0 conn(0x7ff5fc0d2700 msgr2=0x7ff5fc0cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:43.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601d05700 1 --2- 172.21.15.32:0/4093992360 >> v2:172.21.15.86:3300/0 conn(0x7ff5fc0d2700 0x7ff5fc0cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601d05700 1 -- 172.21.15.32:0/4093992360 >> v1:172.21.15.32:6789/0 conn(0x7ff5fc0d8270 legacy=0x7ff5fc1b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:43.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601d05700 1 -- 172.21.15.32:0/4093992360 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc108dc0 con 0x7ff5fc0f28c0 2021-04-09T18:31:43.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601d05700 1 --2- 172.21.15.32:0/4093992360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc0f28c0 0x7ff5fc0d7c50 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 rx=0x7ff5ec006650 tx=0x7ff5ec009a00).ready entity=mon.0 client_cookie=729c6c822cbfa162 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:43.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601504700 1 -- 172.21.15.32:0/4093992360 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5ec009df0 con 0x7ff5fc0f28c0 2021-04-09T18:31:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601504700 1 -- 172.21.15.32:0/4093992360 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec0009b0 con 0x7ff5fc0f28c0 2021-04-09T18:31:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff601504700 1 -- 172.21.15.32:0/4093992360 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5ec00c2b0 con 0x7ff5fc0f28c0 2021-04-09T18:31:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff6046ea700 1 -- 172.21.15.32:0/4093992360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc0f28c0 msgr2=0x7ff5fc0d7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:43.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.649 7ff6046ea700 1 --2- 172.21.15.32:0/4093992360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc0f28c0 0x7ff5fc0d7c50 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 rx=0x7ff5ec006650 tx=0x7ff5ec009a00).stop 2021-04-09T18:31:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.650 7ff6046ea700 1 -- 172.21.15.32:0/4093992360 shutdown_connections 2021-04-09T18:31:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.650 7ff6046ea700 1 --2- 172.21.15.32:0/4093992360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc0f28c0 0x7ff5fc0d7c50 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.650 7ff6046ea700 1 --2- 172.21.15.32:0/4093992360 >> v2:172.21.15.86:3300/0 conn(0x7ff5fc0d2700 0x7ff5fc0cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.650 7ff6046ea700 1 -- 172.21.15.32:0/4093992360 >> 172.21.15.32:0/4093992360 conn(0x7ff5fc0ce590 msgr2=0x7ff5fc0cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.650 7ff6046ea700 1 -- 172.21.15.32:0/4093992360 shutdown_connections 2021-04-09T18:31:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.650 7ff6046ea700 1 -- 172.21.15.32:0/4093992360 wait complete. 2021-04-09T18:31:43.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.670 7ff6046ea700 1 Processor -- start 2021-04-09T18:31:43.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.670 7ff6046ea700 1 -- start start 2021-04-09T18:31:43.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff6046ea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5fc0a9860 0x7ff5fc19b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:43.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff6046ea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc09a430 0x7ff5fc09a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff6046ea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5fc19d350 0x7ff5fc19d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff6046ea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc09ae50 con 0x7ff5fc09a430 2021-04-09T18:31:43.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff6046ea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc19dd70 con 0x7ff5fc19d350 2021-04-09T18:31:43.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff6046ea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc104890 con 0x7ff5fc0a9860 2021-04-09T18:31:43.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff601d05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc09a430 0x7ff5fc09a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:43.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff601d05700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc09a430 0x7ff5fc09a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57180/0 (socket says 172.21.15.32:57180) 2021-04-09T18:31:43.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff601d05700 1 -- 172.21.15.32:0/4193106161 learned_addr learned my addr 172.21.15.32:0/4193106161 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:43.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff602506700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5fc0a9860 0x7ff5fc19b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:43.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff602d07700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5fc19d350 0x7ff5fc19d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:43.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff601d05700 1 -- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5fc0a9860 msgr2=0x7ff5fc19b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:43.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff601d05700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5fc0a9860 0x7ff5fc19b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff601d05700 1 -- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5fc19d350 msgr2=0x7ff5fc19d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:43.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff601d05700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5fc19d350 0x7ff5fc19d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.671 7ff601d05700 1 -- 172.21.15.32:0/4193106161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc104ab0 con 0x7ff5fc09a430 2021-04-09T18:31:43.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.672 7ff601d05700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc09a430 0x7ff5fc09a830 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 rx=0x7ff5ec00b580 tx=0x7ff5ec00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:43.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.672 7ff5f2ffd700 1 -- 172.21.15.32:0/4193106161 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5ec000c90 con 0x7ff5fc09a430 2021-04-09T18:31:43.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.672 7ff5f2ffd700 1 -- 172.21.15.32:0/4193106161 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec002030 con 0x7ff5fc09a430 2021-04-09T18:31:43.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.672 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc104ab0 con 0x7ff5fc09a430 2021-04-09T18:31:43.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.672 7ff5f2ffd700 1 -- 172.21.15.32:0/4193106161 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5ec002480 con 0x7ff5fc09a430 2021-04-09T18:31:43.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.672 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc104ab0 con 0x7ff5fc09a430 2021-04-09T18:31:43.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.672 7ff5f2ffd700 1 -- 172.21.15.32:0/4193106161 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff5ec002210 con 0x7ff5fc09a430 2021-04-09T18:31:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.672 7ff5f2ffd700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff5e4018930 0x7ff5e401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.673 7ff5f2ffd700 1 -- 172.21.15.32:0/4193106161 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7ff5ec032e40 con 0x7ff5fc09a430 2021-04-09T18:31:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.673 7ff602506700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff5e4018930 0x7ff5e401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:43.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.673 7ff602506700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff5e4018930 0x7ff5e401ae10 crc :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:43.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.692 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc09ae50 con 0x7ff5fc09a430 2021-04-09T18:31:43.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.696 7ff5f2ffd700 1 -- 172.21.15.32:0/4193106161 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff5ec002240 con 0x7ff5fc09a430 2021-04-09T18:31:43.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.882 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]} v 0) v1 -- 0x7ff5fc09ae50 con 0x7ff5fc09a430 2021-04-09T18:31:43.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.885 7ff5f2ffd700 1 -- 172.21.15.32:0/4193106161 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]}]=0 updated caps for client.0 v43) v1 ==== 209+0+0 (secure 0 0 0) 0x7ff5ec002230 con 0x7ff5fc09a430 2021-04-09T18:31:43.905 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:31:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.891 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff5e4018930 msgr2=0x7ff5e401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:43.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.891 7ff6046ea700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff5e4018930 0x7ff5e401ae10 crc :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc09a430 msgr2=0x7ff5fc09a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc09a430 0x7ff5fc09a830 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 rx=0x7ff5ec00b580 tx=0x7ff5ec00b920).stop 2021-04-09T18:31:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 shutdown_connections 2021-04-09T18:31:43.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff5e4018930 0x7ff5e401ae10 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5fc09a430 0x7ff5fc09a830 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5fc19d350 0x7ff5fc19d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 --2- 172.21.15.32:0/4193106161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5fc0a9860 0x7ff5fc19b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 >> 172.21.15.32:0/4193106161 conn(0x7ff5fc1acb70 msgr2=0x7ff5fc18c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:43.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 shutdown_connections 2021-04-09T18:31:43.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:43.892 7ff6046ea700 1 -- 172.21.15.32:0/4193106161 wait complete. 2021-04-09T18:31:43.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=alpha_data, allow rw pool=bravo_data' 2021-04-09T18:31:44.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.063 7f3697c19700 1 Processor -- start 2021-04-09T18:31:44.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.063 7f3697c19700 1 -- start start 2021-04-09T18:31:44.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3697c19700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f36900cffb0 0x7f3690087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3697c19700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36900d7c50 0x7f36901b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3697c19700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3690088460 con 0x7f36900d8050 2021-04-09T18:31:44.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3697c19700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36900d91f0 con 0x7f36900cffb0 2021-04-09T18:31:44.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3697c19700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36900d8620 con 0x7f36900d7c50 2021-04-09T18:31:44.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3696236700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f36900d8050 0x7f369010d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38548/0 (socket says 172.21.15.32:38548) 2021-04-09T18:31:44.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3696236700 1 -- 172.21.15.32:0/1974467449 learned_addr learned my addr 172.21.15.32:0/1974467449 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:44.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3695234700 1 --2- 172.21.15.32:0/1974467449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36900d7c50 0x7f36901b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3695a35700 1 --2- 172.21.15.32:0/1974467449 >> v2:172.21.15.86:3300/0 conn(0x7f36900cffb0 0x7f3690087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3694a33700 1 -- 172.21.15.32:0/1974467449 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416691345 0 0) 0x7f3688000d30 con 0x7f36900d8050 2021-04-09T18:31:44.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3694a33700 1 -- 172.21.15.32:0/1974467449 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f367c002fe0 con 0x7f36900d8050 2021-04-09T18:31:44.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3695a35700 1 -- 172.21.15.32:0/1974467449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36900d7c50 msgr2=0x7f36901b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:44.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3695a35700 1 --2- 172.21.15.32:0/1974467449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36900d7c50 0x7f36901b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3695a35700 1 -- 172.21.15.32:0/1974467449 >> v1:172.21.15.32:6789/0 conn(0x7f36900d8050 legacy=0x7f369010d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3695a35700 1 -- 172.21.15.32:0/1974467449 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36900d2800 con 0x7f36900cffb0 2021-04-09T18:31:44.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3695a35700 1 --2- 172.21.15.32:0/1974467449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36900cffb0 0x7f3690087e40 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7f368c006650 tx=0x7f368c009a80).ready entity=mon.1 client_cookie=752d0630e9309d9a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:44.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.064 7f3694a33700 1 -- 172.21.15.32:0/1974467449 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f368c0009b0 con 0x7f36900cffb0 2021-04-09T18:31:44.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.065 7f3694a33700 1 -- 172.21.15.32:0/1974467449 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368c00c0c0 con 0x7f36900cffb0 2021-04-09T18:31:44.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.065 7f3694a33700 1 -- 172.21.15.32:0/1974467449 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f368c00c510 con 0x7f36900cffb0 2021-04-09T18:31:44.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.065 7f3697c19700 1 -- 172.21.15.32:0/1974467449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36900cffb0 msgr2=0x7f3690087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.065 7f3697c19700 1 --2- 172.21.15.32:0/1974467449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36900cffb0 0x7f3690087e40 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7f368c006650 tx=0x7f368c009a80).stop 2021-04-09T18:31:44.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.065 7f3697c19700 1 -- 172.21.15.32:0/1974467449 shutdown_connections 2021-04-09T18:31:44.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.065 7f3697c19700 1 --2- 172.21.15.32:0/1974467449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36900d7c50 0x7f36901b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.065 7f3697c19700 1 --2- 172.21.15.32:0/1974467449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36900cffb0 0x7f3690087e40 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.065 7f3697c19700 1 -- 172.21.15.32:0/1974467449 >> 172.21.15.32:0/1974467449 conn(0x7f36900ce5b0 msgr2=0x7f36900cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:44.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.066 7f3697c19700 1 -- 172.21.15.32:0/1974467449 shutdown_connections 2021-04-09T18:31:44.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.066 7f3697c19700 1 -- 172.21.15.32:0/1974467449 wait complete. 2021-04-09T18:31:44.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3697c19700 1 Processor -- start 2021-04-09T18:31:44.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3697c19700 1 -- start start 2021-04-09T18:31:44.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3697c19700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3690099290 0x7f36900a91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3697c19700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36900a9800 0x7f36901a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3697c19700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36901a14b0 0x7f369009d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3697c19700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3690099690 con 0x7f3690099290 2021-04-09T18:31:44.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3697c19700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36900a9c00 con 0x7f36900a9800 2021-04-09T18:31:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3697c19700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36900ac400 con 0x7f36901a14b0 2021-04-09T18:31:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3695a35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3690099290 0x7f36900a91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3695a35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3690099290 0x7f36900a91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57194/0 (socket says 172.21.15.32:57194) 2021-04-09T18:31:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3695a35700 1 -- 172.21.15.32:0/3026227763 learned_addr learned my addr 172.21.15.32:0/3026227763 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:44.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3696236700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36901a14b0 0x7f369009d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.086 7f3695234700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36900a9800 0x7f36901a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f3695a35700 1 -- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36901a14b0 msgr2=0x7f369009d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f3695a35700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36901a14b0 0x7f369009d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f3695a35700 1 -- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36900a9800 msgr2=0x7f36901a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f3695a35700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36900a9800 0x7f36901a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f3695a35700 1 -- 172.21.15.32:0/3026227763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36901a18b0 con 0x7f3690099290 2021-04-09T18:31:44.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f3695a35700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3690099290 0x7f36900a91e0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 rx=0x7f368c009630 tx=0x7f368c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:44.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f36827fc700 1 -- 172.21.15.32:0/3026227763 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f368c0093a0 con 0x7f3690099290 2021-04-09T18:31:44.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f36827fc700 1 -- 172.21.15.32:0/3026227763 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368c00bd70 con 0x7f3690099290 2021-04-09T18:31:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f3697c19700 1 -- 172.21.15.32:0/3026227763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36901a18b0 con 0x7f3690099290 2021-04-09T18:31:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f36827fc700 1 -- 172.21.15.32:0/3026227763 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f368c0054d0 con 0x7f3690099290 2021-04-09T18:31:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.087 7f3697c19700 1 -- 172.21.15.32:0/3026227763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36901a18b0 con 0x7f3690099290 2021-04-09T18:31:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.088 7f36827fc700 1 -- 172.21.15.32:0/3026227763 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f368c0054f0 con 0x7f3690099290 2021-04-09T18:31:44.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.088 7f36827fc700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3678018930 0x7f367801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.088 7f36827fc700 1 -- 172.21.15.32:0/3026227763 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f368c032e10 con 0x7f3690099290 2021-04-09T18:31:44.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.088 7f3695234700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3678018930 0x7f367801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.088 7f3695234700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3678018930 0x7f367801ae10 crc :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:44.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.108 7f3697c19700 1 -- 172.21.15.32:0/3026227763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36900ac8f0 con 0x7f3690099290 2021-04-09T18:31:44.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.111 7f36827fc700 1 -- 172.21.15.32:0/3026227763 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f368c0052e0 con 0x7f3690099290 2021-04-09T18:31:44.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.299 7f3697c19700 1 -- 172.21.15.32:0/3026227763 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]} v 0) v1 -- 0x7f36900ac8f0 con 0x7f3690099290 2021-04-09T18:31:44.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.301 7f36827fc700 1 -- 172.21.15.32:0/3026227763 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]}]=0 updated caps for client.1 v44) v1 ==== 209+0+0 (secure 0 0 0) 0x7f368c0054f0 con 0x7f3690099290 2021-04-09T18:31:44.318 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:31:44.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 -- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3678018930 msgr2=0x7f367801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3678018930 0x7f367801ae10 crc :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 -- 172.21.15.32:0/3026227763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3690099290 msgr2=0x7f36900a91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3690099290 0x7f36900a91e0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 rx=0x7f368c009630 tx=0x7f368c00b980).stop 2021-04-09T18:31:44.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 -- 172.21.15.32:0/3026227763 shutdown_connections 2021-04-09T18:31:44.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3678018930 0x7f367801ae10 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36901a14b0 0x7f369009d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36900a9800 0x7f36901a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 --2- 172.21.15.32:0/3026227763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3690099290 0x7f36900a91e0 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 -- 172.21.15.32:0/3026227763 >> 172.21.15.32:0/3026227763 conn(0x7f369009e710 msgr2=0x7f36900acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:44.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 -- 172.21.15.32:0/3026227763 shutdown_connections 2021-04-09T18:31:44.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.304 7f3697c19700 1 -- 172.21.15.32:0/3026227763 wait complete. 2021-04-09T18:31:44.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set bravo max_mds 2 2021-04-09T18:31:44.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.468 7fd0406e4700 1 Processor -- start 2021-04-09T18:31:44.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.468 7fd0406e4700 1 -- start start 2021-04-09T18:31:44.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd0406e4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd03810d010 0x7fd0380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd0406e4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0380d1da0 0x7fd0380d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd0406e4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03810d480 con 0x7fd0380d82c0 2021-04-09T18:31:44.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd0406e4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0380d9220 con 0x7fd03810d010 2021-04-09T18:31:44.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd0406e4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380d86c0 con 0x7fd0380d1da0 2021-04-09T18:31:44.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd03ed01700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd0380d82c0 0x7fd03810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38562/0 (socket says 172.21.15.32:38562) 2021-04-09T18:31:44.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd03ed01700 1 -- 172.21.15.32:0/3260875232 learned_addr learned my addr 172.21.15.32:0/3260875232 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:44.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd03dcff700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0380d1da0 0x7fd0380d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd03e500700 1 --2- 172.21.15.32:0/3260875232 >> v2:172.21.15.86:3300/0 conn(0x7fd03810d010 0x7fd0380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258559895 0 0) 0x7fd030000d30 con 0x7fd0380d82c0 2021-04-09T18:31:44.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.469 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd024002fe0 con 0x7fd0380d82c0 2021-04-09T18:31:44.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3747900301 0 0) 0x7fd030001460 con 0x7fd0380d82c0 2021-04-09T18:31:44.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0380d1da0 msgr2=0x7fd0380d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:44.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0380d1da0 0x7fd0380d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 >> v2:172.21.15.86:3300/0 conn(0x7fd03810d010 msgr2=0x7fd0380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:44.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 --2- 172.21.15.32:0/3260875232 >> v2:172.21.15.86:3300/0 conn(0x7fd03810d010 0x7fd0380d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03810e1f0 con 0x7fd0380d82c0 2021-04-09T18:31:44.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fd030001980 con 0x7fd0380d82c0 2021-04-09T18:31:44.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 >> v1:172.21.15.32:6789/0 conn(0x7fd0380d82c0 legacy=0x7fd03810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd024003a40 0x7fd024005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd02404a760 0x7fd02404cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd02404d500 0x7fd02404fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0240502d0 con 0x7fd02404a760 2021-04-09T18:31:44.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0240504f0 con 0x7fd02404d500 2021-04-09T18:31:44.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd024050710 con 0x7fd024003a40 2021-04-09T18:31:44.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03dcff700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd02404a760 0x7fd02404cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03e500700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd024003a40 0x7fd024005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03ed01700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd02404d500 0x7fd02404fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03dcff700 1 -- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd024003a40 msgr2=0x7fd024005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03dcff700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd024003a40 0x7fd024005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03dcff700 1 -- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd02404d500 msgr2=0x7fd02404fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:44.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03dcff700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd02404d500 0x7fd02404fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03dcff700 1 -- 172.21.15.32:0/3260875232 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0240509b0 con 0x7fd02404a760 2021-04-09T18:31:44.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03ed01700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd02404d500 0x7fd02404fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:44.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03dcff700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd02404a760 0x7fd02404cc60 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 rx=0x7fd028009310 tx=0x7fd028009a90).ready entity=mon.0 client_cookie=e72dc0e7bcf3f8af server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:44.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd02800c0c0 con 0x7fd02404a760 2021-04-09T18:31:44.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd028006400 con 0x7fd02404a760 2021-04-09T18:31:44.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03d4fe700 1 -- 172.21.15.32:0/3260875232 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd02800c400 con 0x7fd02404a760 2021-04-09T18:31:44.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd0406e4700 1 -- 172.21.15.32:0/3260875232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd02404a760 msgr2=0x7fd02404cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd0406e4700 1 --2- 172.21.15.32:0/3260875232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd02404a760 0x7fd02404cc60 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 rx=0x7fd028009310 tx=0x7fd028009a90).stop 2021-04-09T18:31:44.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.470 7fd03ed01700 1 -- 172.21.15.32:0/3260875232 reap_dead start 2021-04-09T18:31:44.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.471 7fd0406e4700 1 -- 172.21.15.32:0/3260875232 shutdown_connections 2021-04-09T18:31:44.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.471 7fd0406e4700 1 -- 172.21.15.32:0/3260875232 >> 172.21.15.32:0/3260875232 conn(0x7fd0380ce5f0 msgr2=0x7fd0380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:44.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.471 7fd0406e4700 1 -- 172.21.15.32:0/3260875232 shutdown_connections 2021-04-09T18:31:44.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.471 7fd0406e4700 1 -- 172.21.15.32:0/3260875232 wait complete. 2021-04-09T18:31:44.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.492 7fd0406e4700 1 Processor -- start 2021-04-09T18:31:44.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.492 7fd0406e4700 1 -- start start 2021-04-09T18:31:44.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.492 7fd0406e4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0381a0050 0x7fd0381a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.492 7fd0406e4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0380bdcd0 0x7fd0380bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd0406e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0380bf080 0x7fd0380b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd0406e4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380bf500 con 0x7fd0380bf080 2021-04-09T18:31:44.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd0406e4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380bb7b0 con 0x7fd0381a0050 2021-04-09T18:31:44.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd0406e4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380bb960 con 0x7fd0380bdcd0 2021-04-09T18:31:44.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03ed01700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0380bf080 0x7fd0380b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03ed01700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0380bf080 0x7fd0380b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57218/0 (socket says 172.21.15.32:57218) 2021-04-09T18:31:44.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03ed01700 1 -- 172.21.15.32:0/1625882431 learned_addr learned my addr 172.21.15.32:0/1625882431 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:44.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03e500700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0381a0050 0x7fd0381a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03dcff700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0380bdcd0 0x7fd0380bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03ed01700 1 -- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0380bdcd0 msgr2=0x7fd0380bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03ed01700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0380bdcd0 0x7fd0380bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03ed01700 1 -- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0381a0050 msgr2=0x7fd0381a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:44.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03ed01700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0381a0050 0x7fd0381a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:44.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03ed01700 1 -- 172.21.15.32:0/1625882431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0380be0d0 con 0x7fd0380bf080 2021-04-09T18:31:44.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd03ed01700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0380bf080 0x7fd0380b0cf0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 rx=0x7fd03000a8f0 tx=0x7fd03000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:44.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd02effd700 1 -- 172.21.15.32:0/1625882431 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd030004930 con 0x7fd0380bf080 2021-04-09T18:31:44.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd02effd700 1 -- 172.21.15.32:0/1625882431 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd030004c70 con 0x7fd0380bf080 2021-04-09T18:31:44.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0380be0d0 con 0x7fd0380bf080 2021-04-09T18:31:44.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd02effd700 1 -- 172.21.15.32:0/1625882431 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0300072a0 con 0x7fd0380bf080 2021-04-09T18:31:44.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.493 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0380be0d0 con 0x7fd0380bf080 2021-04-09T18:31:44.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.494 7fd02effd700 1 -- 172.21.15.32:0/1625882431 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd03000add0 con 0x7fd0380bf080 2021-04-09T18:31:44.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.495 7fd02effd700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0200188c0 0x7fd02001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:44.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.495 7fd02effd700 1 -- 172.21.15.32:0/1625882431 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fd030032f20 con 0x7fd0380bf080 2021-04-09T18:31:44.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.495 7fd03e500700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0200188c0 0x7fd02001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:44.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.495 7fd03e500700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0200188c0 0x7fd02001adc0 crc :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:44.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.513 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd038093d50 con 0x7fd0380bf080 2021-04-09T18:31:44.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.517 7fd02effd700 1 -- 172.21.15.32:0/1625882431 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd030004720 con 0x7fd0380bf080 2021-04-09T18:31:44.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:44.709 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "bravo", "val": "2"} v 0) v1 -- 0x7fd038093d50 con 0x7fd0380bf080 2021-04-09T18:31:45.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.305 7fd02effd700 1 -- 172.21.15.32:0/1625882431 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "bravo", "val": "2"}]=0 v359) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd03000add0 con 0x7fd0380bf080 2021-04-09T18:31:45.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.326 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0200188c0 msgr2=0x7fd02001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.326 7fd0406e4700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0200188c0 0x7fd02001adc0 crc :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0380bf080 msgr2=0x7fd0380b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0380bf080 0x7fd0380b0cf0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 rx=0x7fd03000a8f0 tx=0x7fd03000ac90).stop 2021-04-09T18:31:45.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 shutdown_connections 2021-04-09T18:31:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd0200188c0 0x7fd02001adc0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0380bf080 0x7fd0380b0cf0 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0380bdcd0 0x7fd0380bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 --2- 172.21.15.32:0/1625882431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0381a0050 0x7fd0381a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 >> 172.21.15.32:0/1625882431 conn(0x7fd0380975f0 msgr2=0x7fd038096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:45.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 shutdown_connections 2021-04-09T18:31:45.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.327 7fd0406e4700 1 -- 172.21.15.32:0/1625882431 wait complete. 2021-04-09T18:31:45.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:45.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.491 7f664f87b700 1 Processor -- start 2021-04-09T18:31:45.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.491 7f664f87b700 1 -- start start 2021-04-09T18:31:45.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.491 7f664f87b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f66480d7ca0 0x7f66480d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:45.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664f87b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664810da00 0x7f664810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:45.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664f87b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664810d480 con 0x7f664810d010 2021-04-09T18:31:45.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664f87b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66480d9220 con 0x7f66480d7ca0 2021-04-09T18:31:45.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664f87b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66480d86c0 con 0x7f664810da00 2021-04-09T18:31:45.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664de98700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664810da00 0x7f664810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:45.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664d697700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f664810d010 0x7f66480d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:36506/0 (socket says 172.21.15.32:36506) 2021-04-09T18:31:45.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664ce96700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f66480d7ca0 0x7f66480d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:45.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664d697700 1 -- 172.21.15.32:0/524372709 learned_addr learned my addr 172.21.15.32:0/524372709 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:45.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f663ffff700 1 -- 172.21.15.32:0/524372709 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782315025 0 0) 0x7f66440010c0 con 0x7f664810d010 2021-04-09T18:31:45.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f663ffff700 1 -- 172.21.15.32:0/524372709 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634002fe0 con 0x7f664810d010 2021-04-09T18:31:45.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664ce96700 1 -- 172.21.15.32:0/524372709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664810da00 msgr2=0x7f664810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:45.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664ce96700 1 --2- 172.21.15.32:0/524372709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664810da00 0x7f664810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664ce96700 1 -- 172.21.15.32:0/524372709 >> v1:172.21.15.86:6789/0 conn(0x7f664810d010 legacy=0x7f66480d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:45.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664ce96700 1 -- 172.21.15.32:0/524372709 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66480d1f70 con 0x7f66480d7ca0 2021-04-09T18:31:45.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664de98700 1 --2- 172.21.15.32:0/524372709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664810da00 0x7f664810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:45.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f664ce96700 1 --2- 172.21.15.32:0/524372709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66480d7ca0 0x7f66480d80a0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f6638006650 tx=0x7f6638009a80).ready entity=mon.1 client_cookie=5544b5e4cef22142 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:45.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.492 7f663ffff700 1 -- 172.21.15.32:0/524372709 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66380009b0 con 0x7f66480d7ca0 2021-04-09T18:31:45.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f663ffff700 1 -- 172.21.15.32:0/524372709 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663800c0c0 con 0x7f66480d7ca0 2021-04-09T18:31:45.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f663ffff700 1 -- 172.21.15.32:0/524372709 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f663800c510 con 0x7f66480d7ca0 2021-04-09T18:31:45.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f664f87b700 1 -- 172.21.15.32:0/524372709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66480d7ca0 msgr2=0x7f66480d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:45.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f664f87b700 1 --2- 172.21.15.32:0/524372709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66480d7ca0 0x7f66480d80a0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f6638006650 tx=0x7f6638009a80).stop 2021-04-09T18:31:45.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f664f87b700 1 -- 172.21.15.32:0/524372709 shutdown_connections 2021-04-09T18:31:45.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f664f87b700 1 --2- 172.21.15.32:0/524372709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664810da00 0x7f664810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f664f87b700 1 --2- 172.21.15.32:0/524372709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66480d7ca0 0x7f66480d80a0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f664f87b700 1 -- 172.21.15.32:0/524372709 >> 172.21.15.32:0/524372709 conn(0x7f66480ce5f0 msgr2=0x7f66480d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:45.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f664f87b700 1 -- 172.21.15.32:0/524372709 shutdown_connections 2021-04-09T18:31:45.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.493 7f664f87b700 1 -- 172.21.15.32:0/524372709 wait complete. 2021-04-09T18:31:45.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.514 7f664f87b700 1 Processor -- start 2021-04-09T18:31:45.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.514 7f664f87b700 1 -- start start 2021-04-09T18:31:45.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.514 7f664f87b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f664808f040 0x7f66481a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:45.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.514 7f664f87b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664808f440 0x7f66481a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:45.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.514 7f664f87b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66481a22e0 0x7f66481a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.514 7f664f87b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66481a3100 con 0x7f66481a22e0 2021-04-09T18:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.514 7f664f87b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66481a2d00 con 0x7f664808f040 2021-04-09T18:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.514 7f664f87b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664817c7e0 con 0x7f664808f440 2021-04-09T18:31:45.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664de98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66481a22e0 0x7f66481a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664de98700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66481a22e0 0x7f66481a26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57230/0 (socket says 172.21.15.32:57230) 2021-04-09T18:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664de98700 1 -- 172.21.15.32:0/2101574910 learned_addr learned my addr 172.21.15.32:0/2101574910 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664ce96700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664808f440 0x7f66481a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664d697700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f664808f040 0x7f66481a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:45.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664de98700 1 -- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664808f440 msgr2=0x7f66481a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664de98700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664808f440 0x7f66481a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664de98700 1 -- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f664808f040 msgr2=0x7f66481a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664de98700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f664808f040 0x7f66481a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664de98700 1 -- 172.21.15.32:0/2101574910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66480aee10 con 0x7f66481a22e0 2021-04-09T18:31:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664de98700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66481a22e0 0x7f66481a26e0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 rx=0x7f664000b970 tx=0x7f664000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f663dffb700 1 -- 172.21.15.32:0/2101574910 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6640000f90 con 0x7f66481a22e0 2021-04-09T18:31:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f663dffb700 1 -- 172.21.15.32:0/2101574910 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6640006aa0 con 0x7f66481a22e0 2021-04-09T18:31:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664f87b700 1 -- 172.21.15.32:0/2101574910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66480aee10 con 0x7f66481a22e0 2021-04-09T18:31:45.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f663dffb700 1 -- 172.21.15.32:0/2101574910 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66400015f0 con 0x7f66481a22e0 2021-04-09T18:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.515 7f664f87b700 1 -- 172.21.15.32:0/2101574910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66480aee10 con 0x7f66481a22e0 2021-04-09T18:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.516 7f663dffb700 1 -- 172.21.15.32:0/2101574910 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6640000e70 con 0x7f66481a22e0 2021-04-09T18:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.516 7f663dffb700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6628018930 0x7f662801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.516 7f663dffb700 1 -- 172.21.15.32:0/2101574910 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f6640033150 con 0x7f66481a22e0 2021-04-09T18:31:45.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.516 7f664d697700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6628018930 0x7f662801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:45.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.516 7f664d697700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6628018930 0x7f662801ae10 crc :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:45.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.536 7f664f87b700 1 -- 172.21.15.32:0/2101574910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66481b7050 con 0x7f66481a22e0 2021-04-09T18:31:45.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.539 7f663dffb700 1 -- 172.21.15.32:0/2101574910 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f66400066f0 con 0x7f66481a22e0 2021-04-09T18:31:45.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.730 7f664f87b700 1 -- 172.21.15.32:0/2101574910 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f664810d480 con 0x7f66481a22e0 2021-04-09T18:31:45.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.730 7f663dffb700 1 -- 172.21.15.32:0/2101574910 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+3738 (secure 0 0 0) 0x7f6640006aa0 con 0x7f66481a22e0 2021-04-09T18:31:45.747 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 360 2021-04-09T18:31:45.747 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:45.748 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":360,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":358,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:41.244086","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":360,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:45.307050","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":11700,"mds_1":59144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59144":{"gid":59144,"name":"d","rank":1,"incarnation":360,"state":"up:creating","state_seq":2,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 -- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6628018930 msgr2=0x7f662801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:45.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6628018930 0x7f662801ae10 crc :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 -- 172.21.15.32:0/2101574910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66481a22e0 msgr2=0x7f66481a26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:45.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66481a22e0 0x7f66481a26e0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 rx=0x7f664000b970 tx=0x7f664000bd10).stop 2021-04-09T18:31:45.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 -- 172.21.15.32:0/2101574910 shutdown_connections 2021-04-09T18:31:45.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6628018930 0x7f662801ae10 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66481a22e0 0x7f66481a26e0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f664808f440 0x7f66481a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 --2- 172.21.15.32:0/2101574910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f664808f040 0x7f66481a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:45.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 -- 172.21.15.32:0/2101574910 >> 172.21.15.32:0/2101574910 conn(0x7f6648199680 msgr2=0x7f66481a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 -- 172.21.15.32:0/2101574910 shutdown_connections 2021-04-09T18:31:45.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:45.733 7f664f87b700 1 -- 172.21.15.32:0/2101574910 wait complete. 2021-04-09T18:31:45.759 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting... 2021-04-09T18:31:50.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:50.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.905 7f86bd684700 1 Processor -- start 2021-04-09T18:31:50.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.905 7f86bd684700 1 -- start start 2021-04-09T18:31:50.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.906 7f86bd684700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b810d010 0x7f86b80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.906 7f86bd684700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86b80d8490 0x7f86b810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.906 7f86bd684700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b810d480 con 0x7f86b80d1da0 2021-04-09T18:31:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.906 7f86bd684700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b810e780 con 0x7f86b80d8490 2021-04-09T18:31:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.906 7f86bd684700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b80d8d30 con 0x7f86b810d010 2021-04-09T18:31:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.906 7f86b6abe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f86b80d1da0 0x7f86b80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38600/0 (socket says 172.21.15.32:38600) 2021-04-09T18:31:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.906 7f86b6abe700 1 -- 172.21.15.32:0/3883934392 learned_addr learned my addr 172.21.15.32:0/3883934392 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.906 7f86b72bf700 1 --2- 172.21.15.32:0/3883934392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b810d010 0x7f86b80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.906 7f86b7ac0700 1 --2- 172.21.15.32:0/3883934392 >> v2:172.21.15.86:3300/0 conn(0x7f86b80d8490 0x7f86b810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b62bd700 1 -- 172.21.15.32:0/3883934392 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065750842 0 0) 0x7f86ac000ce0 con 0x7f86b80d1da0 2021-04-09T18:31:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b62bd700 1 -- 172.21.15.32:0/3883934392 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a0002fe0 con 0x7f86b80d1da0 2021-04-09T18:31:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b62bd700 1 -- 172.21.15.32:0/3883934392 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1302759011 0 0) 0x7f86ac000ed0 con 0x7f86b80d1da0 2021-04-09T18:31:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b72bf700 1 -- 172.21.15.32:0/3883934392 >> v2:172.21.15.86:3300/0 conn(0x7f86b80d8490 msgr2=0x7f86b810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b72bf700 1 --2- 172.21.15.32:0/3883934392 >> v2:172.21.15.86:3300/0 conn(0x7f86b80d8490 0x7f86b810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:50.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b72bf700 1 -- 172.21.15.32:0/3883934392 >> v1:172.21.15.32:6789/0 conn(0x7f86b80d1da0 legacy=0x7f86b80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:50.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b72bf700 1 -- 172.21.15.32:0/3883934392 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b80d9220 con 0x7f86b810d010 2021-04-09T18:31:50.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b7ac0700 1 --2- 172.21.15.32:0/3883934392 >> v2:172.21.15.86:3300/0 conn(0x7f86b80d8490 0x7f86b810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:31:50.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b72bf700 1 --2- 172.21.15.32:0/3883934392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b810d010 0x7f86b80d1780 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f86a8006650 tx=0x7f86a8009a80).ready entity=mon.2 client_cookie=e989e343912afdb3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b62bd700 1 -- 172.21.15.32:0/3883934392 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86a800c0c0 con 0x7f86b810d010 2021-04-09T18:31:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b62bd700 1 -- 172.21.15.32:0/3883934392 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a800c3c0 con 0x7f86b810d010 2021-04-09T18:31:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86b62bd700 1 -- 172.21.15.32:0/3883934392 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86a80009b0 con 0x7f86b810d010 2021-04-09T18:31:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86bd684700 1 -- 172.21.15.32:0/3883934392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b810d010 msgr2=0x7f86b80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.907 7f86bd684700 1 --2- 172.21.15.32:0/3883934392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b810d010 0x7f86b80d1780 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f86a8006650 tx=0x7f86a8009a80).stop 2021-04-09T18:31:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.908 7f86bd684700 1 -- 172.21.15.32:0/3883934392 shutdown_connections 2021-04-09T18:31:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.908 7f86bd684700 1 --2- 172.21.15.32:0/3883934392 >> v2:172.21.15.86:3300/0 conn(0x7f86b80d8490 0x7f86b810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.908 7f86bd684700 1 --2- 172.21.15.32:0/3883934392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b810d010 0x7f86b80d1780 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.908 7f86bd684700 1 -- 172.21.15.32:0/3883934392 >> 172.21.15.32:0/3883934392 conn(0x7f86b80ce5f0 msgr2=0x7f86b80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:50.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.908 7f86bd684700 1 -- 172.21.15.32:0/3883934392 shutdown_connections 2021-04-09T18:31:50.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.908 7f86bd684700 1 -- 172.21.15.32:0/3883934392 wait complete. 2021-04-09T18:31:50.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.928 7f86bd684700 1 Processor -- start 2021-04-09T18:31:50.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.928 7f86bd684700 1 -- start start 2021-04-09T18:31:50.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86bd684700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b80b5360 0x7f86b81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:50.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86bd684700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b818e140 0x7f86b81aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86bd684700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86b81ad040 0x7f86b808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86bd684700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b81a5360 con 0x7f86b80b5360 2021-04-09T18:31:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86bd684700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b80902c0 con 0x7f86b81ad040 2021-04-09T18:31:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86bd684700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b80904e0 con 0x7f86b818e140 2021-04-09T18:31:50.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b72bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b80b5360 0x7f86b81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:50.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b72bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b80b5360 0x7f86b81a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57244/0 (socket says 172.21.15.32:57244) 2021-04-09T18:31:50.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b72bf700 1 -- 172.21.15.32:0/2877353091 learned_addr learned my addr 172.21.15.32:0/2877353091 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:50.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b6abe700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b818e140 0x7f86b81aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:50.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b7ac0700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86b81ad040 0x7f86b808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b72bf700 1 -- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b818e140 msgr2=0x7f86b81aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b72bf700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b818e140 0x7f86b81aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b72bf700 1 -- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86b81ad040 msgr2=0x7f86b808fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b72bf700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86b81ad040 0x7f86b808fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.929 7f86b72bf700 1 -- 172.21.15.32:0/2877353091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b8183c80 con 0x7f86b80b5360 2021-04-09T18:31:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.930 7f86b72bf700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b80b5360 0x7f86b81a4d40 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 rx=0x7f86a80093a0 tx=0x7f86a8002670).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.930 7f869ffff700 1 -- 172.21.15.32:0/2877353091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86a8006070 con 0x7f86b80b5360 2021-04-09T18:31:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.930 7f869ffff700 1 -- 172.21.15.32:0/2877353091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a8006ae0 con 0x7f86b80b5360 2021-04-09T18:31:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.930 7f86bd684700 1 -- 172.21.15.32:0/2877353091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b8183c80 con 0x7f86b80b5360 2021-04-09T18:31:50.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.930 7f869ffff700 1 -- 172.21.15.32:0/2877353091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86a8007030 con 0x7f86b80b5360 2021-04-09T18:31:50.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.930 7f86bd684700 1 -- 172.21.15.32:0/2877353091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b8183c80 con 0x7f86b80b5360 2021-04-09T18:31:50.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.930 7f869ffff700 1 -- 172.21.15.32:0/2877353091 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f86a8007000 con 0x7f86b80b5360 2021-04-09T18:31:50.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.930 7f869ffff700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8698018930 0x7f869801adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:50.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.930 7f869ffff700 1 -- 172.21.15.32:0/2877353091 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f86a8032e00 con 0x7f86b80b5360 2021-04-09T18:31:50.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.931 7f86b6abe700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8698018930 0x7f869801adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:50.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.931 7f86b6abe700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8698018930 0x7f869801adf0 crc :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:50.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.950 7f86bd684700 1 -- 172.21.15.32:0/2877353091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b8182d70 con 0x7f86b80b5360 2021-04-09T18:31:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:50.954 7f869ffff700 1 -- 172.21.15.32:0/2877353091 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f86a8002a60 con 0x7f86b80b5360 2021-04-09T18:31:51.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.144 7f86bd684700 1 -- 172.21.15.32:0/2877353091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86b8182d70 con 0x7f86b80b5360 2021-04-09T18:31:51.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.145 7f869ffff700 1 -- 172.21.15.32:0/2877353091 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 361 v361) v1 ==== 95+0+3736 (secure 0 0 0) 0x7f86a8007000 con 0x7f86b80b5360 2021-04-09T18:31:51.162 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 361 2021-04-09T18:31:51.162 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":361,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":358,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:41.244086","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":361,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:46.307978","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":11700,"mds_1":59144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59144":{"gid":59144,"name":"d","rank":1,"incarnation":360,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:51.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 -- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8698018930 msgr2=0x7f869801adf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:51.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8698018930 0x7f869801adf0 crc :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 -- 172.21.15.32:0/2877353091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b80b5360 msgr2=0x7f86b81a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:51.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b80b5360 0x7f86b81a4d40 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 rx=0x7f86a80093a0 tx=0x7f86a8002670).stop 2021-04-09T18:31:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 -- 172.21.15.32:0/2877353091 shutdown_connections 2021-04-09T18:31:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8698018930 0x7f869801adf0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86b81ad040 0x7f86b808fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86b818e140 0x7f86b81aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 --2- 172.21.15.32:0/2877353091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b80b5360 0x7f86b81a4d40 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 -- 172.21.15.32:0/2877353091 >> 172.21.15.32:0/2877353091 conn(0x7f86b819b460 msgr2=0x7f86b819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:51.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 -- 172.21.15.32:0/2877353091 shutdown_connections 2021-04-09T18:31:51.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.147 7f86bd684700 1 -- 172.21.15.32:0/2877353091 wait complete. 2021-04-09T18:31:51.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set alpha max_mds 2 2021-04-09T18:31:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.310 7fad6b861700 1 Processor -- start 2021-04-09T18:31:51.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.311 7fad6b861700 1 -- start start 2021-04-09T18:31:51.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.311 7fad6b861700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640d7ca0 0x7fad640d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:51.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.311 7fad6b861700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fad6410da00 0x7fad6410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.311 7fad6b861700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6410d480 con 0x7fad6410d010 2021-04-09T18:31:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.311 7fad6b861700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad640d9220 con 0x7fad6410da00 2021-04-09T18:31:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.311 7fad6b861700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad640d86c0 con 0x7fad640d7ca0 2021-04-09T18:31:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad69e7e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fad6410da00 0x7fad6410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad68e7c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640d7ca0 0x7fad640d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad69e7e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fad6410da00 0x7fad6410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:49010/0 (socket says 172.21.15.32:49010) 2021-04-09T18:31:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad68e7c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640d7ca0 0x7fad640d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:36246/0 (socket says 172.21.15.32:36246) 2021-04-09T18:31:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad6967d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fad6410d010 0x7fad640d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:36536/0 (socket says 172.21.15.32:36536) 2021-04-09T18:31:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad69e7e700 1 -- 172.21.15.32:0/2018426952 learned_addr learned my addr 172.21.15.32:0/2018426952 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:51.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad69e7e700 1 -- 172.21.15.32:0/2018426952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640d7ca0 msgr2=0x7fad640d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:51.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad69e7e700 1 --2- 172.21.15.32:0/2018426952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640d7ca0 0x7fad640d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad69e7e700 1 -- 172.21.15.32:0/2018426952 >> v1:172.21.15.86:6789/0 conn(0x7fad6410d010 legacy=0x7fad640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:51.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad69e7e700 1 -- 172.21.15.32:0/2018426952 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad640d1f70 con 0x7fad6410da00 2021-04-09T18:31:51.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad5bfff700 1 -- 172.21.15.32:0/2018426952 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586076047 0 0) 0x7fad600010c0 con 0x7fad6410d010 2021-04-09T18:31:51.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad69e7e700 1 --2- 172.21.15.32:0/2018426952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad6410da00 0x7fad6410de00 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7fad5c006aa0 tx=0x7fad5c009ea0).ready entity=mon.1 client_cookie=1595ff1f7ee93566 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:51.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad5bfff700 1 -- 172.21.15.32:0/2018426952 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad5c00b510 con 0x7fad6410da00 2021-04-09T18:31:51.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad5bfff700 1 -- 172.21.15.32:0/2018426952 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5c003030 con 0x7fad6410da00 2021-04-09T18:31:51.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad5bfff700 1 -- 172.21.15.32:0/2018426952 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad5c00bad0 con 0x7fad6410da00 2021-04-09T18:31:51.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad6b861700 1 -- 172.21.15.32:0/2018426952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad6410da00 msgr2=0x7fad6410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:51.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.312 7fad6b861700 1 --2- 172.21.15.32:0/2018426952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad6410da00 0x7fad6410de00 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7fad5c006aa0 tx=0x7fad5c009ea0).stop 2021-04-09T18:31:51.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.313 7fad6b861700 1 -- 172.21.15.32:0/2018426952 shutdown_connections 2021-04-09T18:31:51.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.313 7fad6b861700 1 --2- 172.21.15.32:0/2018426952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad6410da00 0x7fad6410de00 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.313 7fad6b861700 1 --2- 172.21.15.32:0/2018426952 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640d7ca0 0x7fad640d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.313 7fad6b861700 1 -- 172.21.15.32:0/2018426952 >> 172.21.15.32:0/2018426952 conn(0x7fad640ce5f0 msgr2=0x7fad640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:51.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.313 7fad6b861700 1 -- 172.21.15.32:0/2018426952 shutdown_connections 2021-04-09T18:31:51.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.313 7fad6b861700 1 -- 172.21.15.32:0/2018426952 wait complete. 2021-04-09T18:31:51.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.333 7fad6b861700 1 Processor -- start 2021-04-09T18:31:51.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.333 7fad6b861700 1 -- start start 2021-04-09T18:31:51.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad6b861700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad641a0050 0x7fad641a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:51.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad6b861700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad640bdcd0 0x7fad640bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:51.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad6b861700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640bf080 0x7fad640b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:51.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad6b861700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad640bf500 con 0x7fad640bdcd0 2021-04-09T18:31:51.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad6b861700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad640bb7b0 con 0x7fad641a0050 2021-04-09T18:31:51.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad6b861700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad640be0d0 con 0x7fad640bf080 2021-04-09T18:31:51.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad68e7c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad640bdcd0 0x7fad640bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:51.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad68e7c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad640bdcd0 0x7fad640bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57256/0 (socket says 172.21.15.32:57256) 2021-04-09T18:31:51.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad68e7c700 1 -- 172.21.15.32:0/126742415 learned_addr learned my addr 172.21.15.32:0/126742415 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:51.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad69e7e700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640bf080 0x7fad640b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:51.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.334 7fad6967d700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad641a0050 0x7fad641a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:51.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad68e7c700 1 -- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640bf080 msgr2=0x7fad640b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:51.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad68e7c700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640bf080 0x7fad640b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad68e7c700 1 -- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad641a0050 msgr2=0x7fad641a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:51.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad68e7c700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad641a0050 0x7fad641a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:51.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad68e7c700 1 -- 172.21.15.32:0/126742415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad640b1310 con 0x7fad640bdcd0 2021-04-09T18:31:51.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad68e7c700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad640bdcd0 0x7fad640bea60 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 rx=0x7fad54006fa0 tx=0x7fad5400b8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:51.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad59ffb700 1 -- 172.21.15.32:0/126742415 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad54000c90 con 0x7fad640bdcd0 2021-04-09T18:31:51.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad59ffb700 1 -- 172.21.15.32:0/126742415 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5400bcb0 con 0x7fad640bdcd0 2021-04-09T18:31:51.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad6b861700 1 -- 172.21.15.32:0/126742415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad640b1310 con 0x7fad640bdcd0 2021-04-09T18:31:51.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad59ffb700 1 -- 172.21.15.32:0/126742415 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad54002220 con 0x7fad640bdcd0 2021-04-09T18:31:51.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.335 7fad6b861700 1 -- 172.21.15.32:0/126742415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad640b1310 con 0x7fad640bdcd0 2021-04-09T18:31:51.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.336 7fad59ffb700 1 -- 172.21.15.32:0/126742415 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fad54006b00 con 0x7fad640bdcd0 2021-04-09T18:31:51.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.336 7fad59ffb700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad44018930 0x7fad4401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:51.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.336 7fad59ffb700 1 -- 172.21.15.32:0/126742415 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fad54035020 con 0x7fad640bdcd0 2021-04-09T18:31:51.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.336 7fad6967d700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad44018930 0x7fad4401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:51.366 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.336 7fad6967d700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad44018930 0x7fad4401adc0 crc :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:51.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.355 7fad6b861700 1 -- 172.21.15.32:0/126742415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad64093d50 con 0x7fad640bdcd0 2021-04-09T18:31:51.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.359 7fad59ffb700 1 -- 172.21.15.32:0/126742415 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fad54000c90 con 0x7fad640bdcd0 2021-04-09T18:31:51.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:51.553 7fad6b861700 1 -- 172.21.15.32:0/126742415 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "alpha", "val": "2"} v 0) v1 -- 0x7fad6410d480 con 0x7fad640bdcd0 2021-04-09T18:31:52.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.537 7fad59ffb700 1 -- 172.21.15.32:0/126742415 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "alpha", "val": "2"}]=0 v362) v1 ==== 104+0+0 (secure 0 0 0) 0x7fad54006b00 con 0x7fad640bdcd0 2021-04-09T18:31:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 -- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad44018930 msgr2=0x7fad4401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:52.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad44018930 0x7fad4401adc0 crc :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 -- 172.21.15.32:0/126742415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad640bdcd0 msgr2=0x7fad640bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad640bdcd0 0x7fad640bea60 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 rx=0x7fad54006fa0 tx=0x7fad5400b8c0).stop 2021-04-09T18:31:52.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 -- 172.21.15.32:0/126742415 shutdown_connections 2021-04-09T18:31:52.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fad44018930 0x7fad4401adc0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad640bdcd0 0x7fad640bea60 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad640bf080 0x7fad640b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 --2- 172.21.15.32:0/126742415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad641a0050 0x7fad641a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 -- 172.21.15.32:0/126742415 >> 172.21.15.32:0/126742415 conn(0x7fad640975f0 msgr2=0x7fad64096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:52.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 -- 172.21.15.32:0/126742415 shutdown_connections 2021-04-09T18:31:52.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.572 7fad6b861700 1 -- 172.21.15.32:0/126742415 wait complete. 2021-04-09T18:31:52.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.733 7f5fa25e0700 1 Processor -- start 2021-04-09T18:31:52.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.733 7f5fa25e0700 1 -- start start 2021-04-09T18:31:52.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5fa25e0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f9c0d7ca0 0x7f5f9c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:52.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5fa25e0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c10d480 con 0x7f5f9c10d010 2021-04-09T18:31:52.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5fa25e0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c0d86c0 con 0x7f5f9c10da00 2021-04-09T18:31:52.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5fa25e0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9c0d9220 con 0x7f5f9c0d7ca0 2021-04-09T18:31:52.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5f93fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5f9c0d7ca0 0x7f5f9c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5f9bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5f9c10d010 0x7f5f9c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38622/0 (socket says 172.21.15.32:38622) 2021-04-09T18:31:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5f9bfff700 1 -- 172.21.15.32:0/3931564622 learned_addr learned my addr 172.21.15.32:0/3931564622 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:31:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5f93fff700 1 -- 172.21.15.32:0/3931564622 >> v1:172.21.15.86:6789/0 conn(0x7f5f9c10da00 legacy=0x7f5f9c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5f93fff700 1 -- 172.21.15.32:0/3931564622 >> v1:172.21.15.32:6789/0 conn(0x7f5f9c10d010 legacy=0x7f5f9c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5f9b7fe700 1 -- 172.21.15.32:0/3931564622 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254746592 0 0) 0x7f5f8c000ce0 con 0x7f5f9c10d010 2021-04-09T18:31:52.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.734 7f5f93fff700 1 -- 172.21.15.32:0/3931564622 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c0d1f70 con 0x7f5f9c0d7ca0 2021-04-09T18:31:52.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.735 7f5f93fff700 1 --2- 172.21.15.32:0/3931564622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0d7ca0 0x7f5f9c0d80a0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 rx=0x7f5f84006650 tx=0x7f5f84009a00).ready entity=mon.0 client_cookie=28d4858d0dc66ef0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.735 7f5f9b7fe700 1 -- 172.21.15.32:0/3931564622 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f84009df0 con 0x7f5f9c0d7ca0 2021-04-09T18:31:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.735 7f5f9b7fe700 1 -- 172.21.15.32:0/3931564622 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f840009b0 con 0x7f5f9c0d7ca0 2021-04-09T18:31:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.735 7f5f9b7fe700 1 -- 172.21.15.32:0/3931564622 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f8400c2b0 con 0x7f5f9c0d7ca0 2021-04-09T18:31:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.735 7f5fa25e0700 1 -- 172.21.15.32:0/3931564622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0d7ca0 msgr2=0x7f5f9c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.735 7f5fa25e0700 1 --2- 172.21.15.32:0/3931564622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0d7ca0 0x7f5f9c0d80a0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 rx=0x7f5f84006650 tx=0x7f5f84009a00).stop 2021-04-09T18:31:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.735 7f5fa25e0700 1 -- 172.21.15.32:0/3931564622 shutdown_connections 2021-04-09T18:31:52.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.735 7f5fa25e0700 1 --2- 172.21.15.32:0/3931564622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0d7ca0 0x7f5f9c0d80a0 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.735 7f5fa25e0700 1 -- 172.21.15.32:0/3931564622 >> 172.21.15.32:0/3931564622 conn(0x7f5f9c0ce5f0 msgr2=0x7f5f9c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:52.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.736 7f5fa25e0700 1 -- 172.21.15.32:0/3931564622 shutdown_connections 2021-04-09T18:31:52.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.736 7f5fa25e0700 1 -- 172.21.15.32:0/3931564622 wait complete. 2021-04-09T18:31:52.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5fa25e0700 1 Processor -- start 2021-04-09T18:31:52.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5fa25e0700 1 -- start start 2021-04-09T18:31:52.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5fa25e0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0af100 0x7f5f9c090b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:52.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5fa25e0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f9c1868f0 0x7f5f9c185ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:52.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5fa25e0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f9c1a9470 0x7f5f9c1a9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:52.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5fa25e0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c1860d0 con 0x7f5f9c0af100 2021-04-09T18:31:52.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5fa25e0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c186cf0 con 0x7f5f9c1868f0 2021-04-09T18:31:52.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5fa25e0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c195280 con 0x7f5f9c1a9470 2021-04-09T18:31:52.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5f9bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0af100 0x7f5f9c090b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:52.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.756 7f5f9bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0af100 0x7f5f9c090b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57270/0 (socket says 172.21.15.32:57270) 2021-04-09T18:31:52.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f9bfff700 1 -- 172.21.15.32:0/3966872313 learned_addr learned my addr 172.21.15.32:0/3966872313 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:52.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f93fff700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f9c1868f0 0x7f5f9c185ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:52.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f9bfff700 1 -- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f9c1a9470 msgr2=0x7f5f9c1a9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:52.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5fa0bfd700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f9c1a9470 0x7f5f9c1a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:52.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f9bfff700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f9c1a9470 0x7f5f9c1a9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f9bfff700 1 -- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f9c1868f0 msgr2=0x7f5f9c185ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:52.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f9bfff700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f9c1868f0 0x7f5f9c185ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f9bfff700 1 -- 172.21.15.32:0/3966872313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c09d9d0 con 0x7f5f9c0af100 2021-04-09T18:31:52.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f93fff700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f9c1868f0 0x7f5f9c185ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:31:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f9bfff700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0af100 0x7f5f9c090b80 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 rx=0x7f5f8c006f00 tx=0x7f5f8c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f997fa700 1 -- 172.21.15.32:0/3966872313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f8c006020 con 0x7f5f9c0af100 2021-04-09T18:31:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f997fa700 1 -- 172.21.15.32:0/3966872313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c0063a0 con 0x7f5f9c0af100 2021-04-09T18:31:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c09d9d0 con 0x7f5f9c0af100 2021-04-09T18:31:52.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5f997fa700 1 -- 172.21.15.32:0/3966872313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f8c004620 con 0x7f5f9c0af100 2021-04-09T18:31:52.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.757 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c09d9d0 con 0x7f5f9c0af100 2021-04-09T18:31:52.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.758 7f5f997fa700 1 -- 172.21.15.32:0/3966872313 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5f8c00a000 con 0x7f5f9c0af100 2021-04-09T18:31:52.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.758 7f5f997fa700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f800188c0 0x7f5f8001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:52.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.758 7f5f997fa700 1 -- 172.21.15.32:0/3966872313 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f5f8c032b60 con 0x7f5f9c0af100 2021-04-09T18:31:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.759 7f5f93fff700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f800188c0 0x7f5f8001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.759 7f5f93fff700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f800188c0 0x7f5f8001adc0 crc :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:52.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.778 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c10d480 con 0x7f5f9c0af100 2021-04-09T18:31:52.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.781 7f5f997fa700 1 -- 172.21.15.32:0/3966872313 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5f8c006b90 con 0x7f5f9c0af100 2021-04-09T18:31:52.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.973 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f9c10d480 con 0x7f5f9c0af100 2021-04-09T18:31:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.973 7f5f997fa700 1 -- 172.21.15.32:0/3966872313 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 363 v363) v1 ==== 95+0+3755 (secure 0 0 0) 0x7f5f8c00a000 con 0x7f5f9c0af100 2021-04-09T18:31:52.989 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 363 2021-04-09T18:31:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:52.990 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":363,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":363,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:52.538383","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":11697,"mds_1":59147},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59147":{"gid":59147,"name":"b","rank":1,"incarnation":363,"state":"up:creating","state_seq":2,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":361,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:46.307978","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":11700,"mds_1":59144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59144":{"gid":59144,"name":"d","rank":1,"incarnation":360,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f800188c0 msgr2=0x7f5f8001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f800188c0 0x7f5f8001adc0 crc :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0af100 msgr2=0x7f5f9c090b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:52.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0af100 0x7f5f9c090b80 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 rx=0x7f5f8c006f00 tx=0x7f5f8c00b940).stop 2021-04-09T18:31:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 shutdown_connections 2021-04-09T18:31:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5f800188c0 0x7f5f8001adc0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f9c1a9470 0x7f5f9c1a9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f9c1868f0 0x7f5f9c185ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 --2- 172.21.15.32:0/3966872313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f9c0af100 0x7f5f9c090b80 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:52.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 >> 172.21.15.32:0/3966872313 conn(0x7f5f9c1965c0 msgr2=0x7f5f9c1ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 shutdown_connections 2021-04-09T18:31:52.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:52.976 7f5fa25e0700 1 -- 172.21.15.32:0/3966872313 wait complete. 2021-04-09T18:31:53.002 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting... 2021-04-09T18:31:58.004 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:58.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.149 7f503b2e0700 1 Processor -- start 2021-04-09T18:31:58.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.149 7f503b2e0700 1 -- start start 2021-04-09T18:31:58.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.149 7f503b2e0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f50340d7ca0 0x7f50340d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.149 7f503b2e0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503410da00 0x7f503410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f503b2e0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503410d480 con 0x7f503410d010 2021-04-09T18:31:58.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f503b2e0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50340d9220 con 0x7f50340d7ca0 2021-04-09T18:31:58.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f503b2e0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50340d86c0 con 0x7f503410da00 2021-04-09T18:31:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50388fb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f50340d7ca0 0x7f50340d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50388fb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f50340d7ca0 0x7f50340d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57282/0 (socket says 172.21.15.32:57282) 2021-04-09T18:31:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50388fb700 1 -- 172.21.15.32:0/2325892739 learned_addr learned my addr 172.21.15.32:0/2325892739 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50398fd700 1 --2- 172.21.15.32:0/2325892739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503410da00 0x7f503410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50388fb700 1 -- 172.21.15.32:0/2325892739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503410da00 msgr2=0x7f503410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:58.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50388fb700 1 --2- 172.21.15.32:0/2325892739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503410da00 0x7f503410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50388fb700 1 -- 172.21.15.32:0/2325892739 >> v1:172.21.15.86:6789/0 conn(0x7f503410d010 legacy=0x7f50340d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50388fb700 1 -- 172.21.15.32:0/2325892739 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50340d1f70 con 0x7f50340d7ca0 2021-04-09T18:31:58.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50398fd700 1 --2- 172.21.15.32:0/2325892739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503410da00 0x7f503410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f50388fb700 1 --2- 172.21.15.32:0/2325892739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50340d7ca0 0x7f50340d80a0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7f5028006700 tx=0x7f5028009b80).ready entity=mon.0 client_cookie=6be60032e297deaa server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f5027fff700 1 -- 172.21.15.32:0/2325892739 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f502800c0c0 con 0x7f50340d7ca0 2021-04-09T18:31:58.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f5027fff700 1 -- 172.21.15.32:0/2325892739 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502800c440 con 0x7f50340d7ca0 2021-04-09T18:31:58.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f5027fff700 1 -- 172.21.15.32:0/2325892739 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f502800c920 con 0x7f50340d7ca0 2021-04-09T18:31:58.201 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f503b2e0700 1 -- 172.21.15.32:0/2325892739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50340d7ca0 msgr2=0x7f50340d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.150 7f503b2e0700 1 --2- 172.21.15.32:0/2325892739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50340d7ca0 0x7f50340d80a0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7f5028006700 tx=0x7f5028009b80).stop 2021-04-09T18:31:58.202 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.151 7f503b2e0700 1 -- 172.21.15.32:0/2325892739 shutdown_connections 2021-04-09T18:31:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.151 7f503b2e0700 1 --2- 172.21.15.32:0/2325892739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503410da00 0x7f503410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.151 7f503b2e0700 1 --2- 172.21.15.32:0/2325892739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50340d7ca0 0x7f50340d80a0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.203 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.151 7f503b2e0700 1 -- 172.21.15.32:0/2325892739 >> 172.21.15.32:0/2325892739 conn(0x7f50340ce5f0 msgr2=0x7f50340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.151 7f503b2e0700 1 -- 172.21.15.32:0/2325892739 shutdown_connections 2021-04-09T18:31:58.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.151 7f503b2e0700 1 -- 172.21.15.32:0/2325892739 wait complete. 2021-04-09T18:31:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.171 7f503b2e0700 1 Processor -- start 2021-04-09T18:31:58.205 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f503b2e0700 1 -- start start 2021-04-09T18:31:58.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f503b2e0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50341aca20 0x7f50341ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f503b2e0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503418bd40 0x7f503418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.206 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f503b2e0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503408fba0 0x7f503408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f503b2e0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503418c7e0 con 0x7f503408fba0 2021-04-09T18:31:58.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f503b2e0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503418ca30 con 0x7f50341aca20 2021-04-09T18:31:58.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f503b2e0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50340905f0 con 0x7f503418bd40 2021-04-09T18:31:58.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f50398fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503408fba0 0x7f503408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f50398fd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503408fba0 0x7f503408ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57290/0 (socket says 172.21.15.32:57290) 2021-04-09T18:31:58.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.172 7f50398fd700 1 -- 172.21.15.32:0/2478080324 learned_addr learned my addr 172.21.15.32:0/2478080324 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:58.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50388fb700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503418bd40 0x7f503418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50390fc700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50341aca20 0x7f50341ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50398fd700 1 -- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503418bd40 msgr2=0x7f503418c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50398fd700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503418bd40 0x7f503418c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50398fd700 1 -- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50341aca20 msgr2=0x7f50341ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50398fd700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50341aca20 0x7f50341ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50398fd700 1 -- 172.21.15.32:0/2478080324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50340b4300 con 0x7f503408fba0 2021-04-09T18:31:58.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50388fb700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503418bd40 0x7f503418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:58.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50390fc700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50341aca20 0x7f50341ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f50398fd700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503408fba0 0x7f503408ffa0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 rx=0x7f502c009980 tx=0x7f502c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f5025ffb700 1 -- 172.21.15.32:0/2478080324 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f502c00c2f0 con 0x7f503408fba0 2021-04-09T18:31:58.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f5025ffb700 1 -- 172.21.15.32:0/2478080324 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502c006790 con 0x7f503408fba0 2021-04-09T18:31:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f5025ffb700 1 -- 172.21.15.32:0/2478080324 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f502c00c7e0 con 0x7f503408fba0 2021-04-09T18:31:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50340b4300 con 0x7f503408fba0 2021-04-09T18:31:58.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.173 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50340b4300 con 0x7f503408fba0 2021-04-09T18:31:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.174 7f5025ffb700 1 -- 172.21.15.32:0/2478080324 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f502c0011f0 con 0x7f503408fba0 2021-04-09T18:31:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.174 7f5025ffb700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50140188c0 0x7f501401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.174 7f5025ffb700 1 -- 172.21.15.32:0/2478080324 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f502c032f50 con 0x7f503408fba0 2021-04-09T18:31:58.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.175 7f50390fc700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50140188c0 0x7f501401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.175 7f50390fc700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50140188c0 0x7f501401adc0 crc :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.194 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5034182d70 con 0x7f503408fba0 2021-04-09T18:31:58.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.197 7f5025ffb700 1 -- 172.21.15.32:0/2478080324 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f502c00be10 con 0x7f503408fba0 2021-04-09T18:31:58.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.389 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f503418c760 con 0x7f503408fba0 2021-04-09T18:31:58.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.389 7f5025ffb700 1 -- 172.21.15.32:0/2478080324 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 364 v364) v1 ==== 95+0+3753 (secure 0 0 0) 0x7f502c0011f0 con 0x7f503408fba0 2021-04-09T18:31:58.405 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 364 2021-04-09T18:31:58.406 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:58.406 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":364,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":364,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:53.539936","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":11697,"mds_1":59147},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59147":{"gid":59147,"name":"b","rank":1,"incarnation":363,"state":"up:active","state_seq":6,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":361,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:46.307978","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":11700,"mds_1":59144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59144":{"gid":59144,"name":"d","rank":1,"incarnation":360,"state":"up:active","state_seq":4,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:58.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50140188c0 msgr2=0x7f501401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50140188c0 0x7f501401adc0 crc :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503408fba0 msgr2=0x7f503408ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503408fba0 0x7f503408ffa0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 rx=0x7f502c009980 tx=0x7f502c00bcd0).stop 2021-04-09T18:31:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 shutdown_connections 2021-04-09T18:31:58.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f50140188c0 0x7f501401adc0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f503408fba0 0x7f503408ffa0 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f503418bd40 0x7f503418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 --2- 172.21.15.32:0/2478080324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50341aca20 0x7f50341ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 >> 172.21.15.32:0/2478080324 conn(0x7f503419b460 msgr2=0x7f503419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:58.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 shutdown_connections 2021-04-09T18:31:58.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.392 7f503b2e0700 1 -- 172.21.15.32:0/2478080324 wait complete. 2021-04-09T18:31:58.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set bravo max_mds 1 2021-04-09T18:31:58.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.555 7f2b3d018700 1 Processor -- start 2021-04-09T18:31:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.555 7f2b3d018700 1 -- start start 2021-04-09T18:31:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b3d018700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b3810d010 0x7f2b380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b3d018700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2b380d1da0 0x7f2b380d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b3d018700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3810d480 con 0x7f2b380d82c0 2021-04-09T18:31:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b3d018700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b380d8d30 con 0x7f2b3810d010 2021-04-09T18:31:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b3d018700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b380d9220 con 0x7f2b380d1da0 2021-04-09T18:31:58.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b36d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b3810d010 0x7f2b380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b36d9d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b3810d010 0x7f2b380d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57294/0 (socket says 172.21.15.32:57294) 2021-04-09T18:31:58.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b36d9d700 1 -- 172.21.15.32:0/2553854273 learned_addr learned my addr 172.21.15.32:0/2553854273 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:58.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b3659c700 1 --2- 172.21.15.32:0/2553854273 >> v2:172.21.15.86:3300/0 conn(0x7f2b380d1da0 0x7f2b380d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b36d9d700 1 -- 172.21.15.32:0/2553854273 >> v2:172.21.15.86:3300/0 conn(0x7f2b380d1da0 msgr2=0x7f2b380d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:58.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.556 7f2b36d9d700 1 --2- 172.21.15.32:0/2553854273 >> v2:172.21.15.86:3300/0 conn(0x7f2b380d1da0 0x7f2b380d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.557 7f2b36d9d700 1 -- 172.21.15.32:0/2553854273 >> v1:172.21.15.86:6789/0 conn(0x7f2b380d82c0 legacy=0x7f2b3810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.557 7f2b36d9d700 1 -- 172.21.15.32:0/2553854273 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b380d86f0 con 0x7f2b3810d010 2021-04-09T18:31:58.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.557 7f2b36d9d700 1 --2- 172.21.15.32:0/2553854273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3810d010 0x7f2b380d1780 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 rx=0x7f2b28006700 tx=0x7f2b28009b00).ready entity=mon.0 client_cookie=c1890a0411c0482a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:58.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.557 7f2b35d9b700 1 -- 172.21.15.32:0/2553854273 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b2800c0c0 con 0x7f2b3810d010 2021-04-09T18:31:58.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.557 7f2b35d9b700 1 -- 172.21.15.32:0/2553854273 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2800c400 con 0x7f2b3810d010 2021-04-09T18:31:58.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.557 7f2b35d9b700 1 -- 172.21.15.32:0/2553854273 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b2800c8e0 con 0x7f2b3810d010 2021-04-09T18:31:58.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.557 7f2b3d018700 1 -- 172.21.15.32:0/2553854273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3810d010 msgr2=0x7f2b380d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.557 7f2b3d018700 1 --2- 172.21.15.32:0/2553854273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3810d010 0x7f2b380d1780 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 rx=0x7f2b28006700 tx=0x7f2b28009b00).stop 2021-04-09T18:31:58.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.558 7f2b3d018700 1 -- 172.21.15.32:0/2553854273 shutdown_connections 2021-04-09T18:31:58.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.558 7f2b3d018700 1 --2- 172.21.15.32:0/2553854273 >> v2:172.21.15.86:3300/0 conn(0x7f2b380d1da0 0x7f2b380d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.558 7f2b3d018700 1 --2- 172.21.15.32:0/2553854273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3810d010 0x7f2b380d1780 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.558 7f2b3d018700 1 -- 172.21.15.32:0/2553854273 >> 172.21.15.32:0/2553854273 conn(0x7f2b380ce5f0 msgr2=0x7f2b380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:58.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.558 7f2b3d018700 1 -- 172.21.15.32:0/2553854273 shutdown_connections 2021-04-09T18:31:58.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.558 7f2b3d018700 1 -- 172.21.15.32:0/2553854273 wait complete. 2021-04-09T18:31:58.601 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.578 7f2b3d018700 1 Processor -- start 2021-04-09T18:31:58.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.578 7f2b3d018700 1 -- start start 2021-04-09T18:31:58.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.578 7f2b3d018700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3809e530 0x7f2b3809b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.578 7f2b3d018700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b3809e930 0x7f2b3809f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.602 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.578 7f2b3d018700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b3809f990 0x7f2b38094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.578 7f2b3d018700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3809edf0 con 0x7f2b3809e530 2021-04-09T18:31:58.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.578 7f2b3d018700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3809fd90 con 0x7f2b3809f990 2021-04-09T18:31:58.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.578 7f2b3d018700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3809bd40 con 0x7f2b3809e930 2021-04-09T18:31:58.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.578 7f2b36d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3809e530 0x7f2b3809b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b36d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3809e530 0x7f2b3809b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57300/0 (socket says 172.21.15.32:57300) 2021-04-09T18:31:58.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b36d9d700 1 -- 172.21.15.32:0/3269568303 learned_addr learned my addr 172.21.15.32:0/3269568303 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:58.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b3659c700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b3809e930 0x7f2b3809f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b3759e700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b3809f990 0x7f2b38094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b36d9d700 1 -- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b3809e930 msgr2=0x7f2b3809f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b36d9d700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b3809e930 0x7f2b3809f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b36d9d700 1 -- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b3809f990 msgr2=0x7f2b38094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:58.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b36d9d700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b3809f990 0x7f2b38094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:58.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b36d9d700 1 -- 172.21.15.32:0/3269568303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b38095290 con 0x7f2b3809e530 2021-04-09T18:31:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b3759e700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b3809f990 0x7f2b38094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:31:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b36d9d700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3809e530 0x7f2b3809b720 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 rx=0x7f2b2800b660 tx=0x7f2b2800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b277fe700 1 -- 172.21.15.32:0/3269568303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b28009440 con 0x7f2b3809e530 2021-04-09T18:31:58.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b277fe700 1 -- 172.21.15.32:0/3269568303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b28006390 con 0x7f2b3809e530 2021-04-09T18:31:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b38095290 con 0x7f2b3809e530 2021-04-09T18:31:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b277fe700 1 -- 172.21.15.32:0/3269568303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b28002030 con 0x7f2b3809e530 2021-04-09T18:31:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.579 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b38095290 con 0x7f2b3809e530 2021-04-09T18:31:58.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.580 7f2b277fe700 1 -- 172.21.15.32:0/3269568303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2b280093e0 con 0x7f2b3809e530 2021-04-09T18:31:58.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.580 7f2b277fe700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b1c0188c0 0x7f2b1c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:58.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.580 7f2b277fe700 1 -- 172.21.15.32:0/3269568303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f2b28032bb0 con 0x7f2b3809e530 2021-04-09T18:31:58.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.581 7f2b3659c700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b1c0188c0 0x7f2b1c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:58.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.581 7f2b3659c700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b1c0188c0 0x7f2b1c01adc0 crc :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:58.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.600 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b38095b10 con 0x7f2b3809e530 2021-04-09T18:31:58.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.603 7f2b277fe700 1 -- 172.21.15.32:0/3269568303 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2b280090c0 con 0x7f2b3809e530 2021-04-09T18:31:58.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:58.799 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "bravo", "val": "1"} v 0) v1 -- 0x7f2b38095b10 con 0x7f2b3809e530 2021-04-09T18:31:59.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.447 7f2b277fe700 1 -- 172.21.15.32:0/3269568303 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "bravo", "val": "1"}]=0 v365) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2b280093e0 con 0x7f2b3809e530 2021-04-09T18:31:59.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b1c0188c0 msgr2=0x7f2b1c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:59.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b1c0188c0 0x7f2b1c01adc0 crc :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3809e530 msgr2=0x7f2b3809b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:59.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3809e530 0x7f2b3809b720 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 rx=0x7f2b2800b660 tx=0x7f2b2800ba00).stop 2021-04-09T18:31:59.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 shutdown_connections 2021-04-09T18:31:59.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2b1c0188c0 0x7f2b1c01adc0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b3809f990 0x7f2b38094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b3809e930 0x7f2b3809f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 --2- 172.21.15.32:0/3269568303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b3809e530 0x7f2b3809b720 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.467 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 >> 172.21.15.32:0/3269568303 conn(0x7f2b380c4290 msgr2=0x7f2b380c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:59.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.468 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 shutdown_connections 2021-04-09T18:31:59.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.468 7f2b3d018700 1 -- 172.21.15.32:0/3269568303 wait complete. 2021-04-09T18:31:59.496 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:31:59.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.636 7ff597322700 1 Processor -- start 2021-04-09T18:31:59.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.636 7ff597322700 1 -- start start 2021-04-09T18:31:59.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.637 7ff597322700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5900d7ca0 0x7ff5900d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:59.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.637 7ff597322700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff59010da00 0x7ff59010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:59.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.637 7ff597322700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59010d480 con 0x7ff59010d010 2021-04-09T18:31:59.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.637 7ff597322700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5900d9220 con 0x7ff5900d7ca0 2021-04-09T18:31:59.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.637 7ff597322700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5900d86c0 con 0x7ff59010da00 2021-04-09T18:31:59.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.637 7ff59493d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5900d7ca0 0x7ff5900d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:59.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.637 7ff59493d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff5900d7ca0 0x7ff5900d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57312/0 (socket says 172.21.15.32:57312) 2021-04-09T18:31:59.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.637 7ff59493d700 1 -- 172.21.15.32:0/1793948550 learned_addr learned my addr 172.21.15.32:0/1793948550 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:59.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.637 7ff59593f700 1 --2- 172.21.15.32:0/1793948550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff59010da00 0x7ff59010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:59.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff587fff700 1 -- 172.21.15.32:0/1793948550 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920538651 0 0) 0x7ff58c000c60 con 0x7ff59010d010 2021-04-09T18:31:59.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff59493d700 1 -- 172.21.15.32:0/1793948550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff59010da00 msgr2=0x7ff59010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:31:59.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff59493d700 1 --2- 172.21.15.32:0/1793948550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff59010da00 0x7ff59010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff59493d700 1 -- 172.21.15.32:0/1793948550 >> v1:172.21.15.86:6789/0 conn(0x7ff59010d010 legacy=0x7ff5900d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:59.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff59493d700 1 -- 172.21.15.32:0/1793948550 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590112100 con 0x7ff5900d7ca0 2021-04-09T18:31:59.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff59493d700 1 --2- 172.21.15.32:0/1793948550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5900d7ca0 0x7ff5900d80a0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7ff580006700 tx=0x7ff580009b80).ready entity=mon.0 client_cookie=8078b25f662c7bc8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:59.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff587fff700 1 -- 172.21.15.32:0/1793948550 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff58000c0c0 con 0x7ff5900d7ca0 2021-04-09T18:31:59.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff587fff700 1 -- 172.21.15.32:0/1793948550 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58000c440 con 0x7ff5900d7ca0 2021-04-09T18:31:59.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff587fff700 1 -- 172.21.15.32:0/1793948550 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff58000c920 con 0x7ff5900d7ca0 2021-04-09T18:31:59.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff597322700 1 -- 172.21.15.32:0/1793948550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5900d7ca0 msgr2=0x7ff5900d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:59.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.638 7ff597322700 1 --2- 172.21.15.32:0/1793948550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5900d7ca0 0x7ff5900d80a0 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7ff580006700 tx=0x7ff580009b80).stop 2021-04-09T18:31:59.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.639 7ff597322700 1 -- 172.21.15.32:0/1793948550 shutdown_connections 2021-04-09T18:31:59.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.639 7ff597322700 1 --2- 172.21.15.32:0/1793948550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff59010da00 0x7ff59010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.639 7ff597322700 1 --2- 172.21.15.32:0/1793948550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5900d7ca0 0x7ff5900d80a0 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.639 7ff597322700 1 -- 172.21.15.32:0/1793948550 >> 172.21.15.32:0/1793948550 conn(0x7ff5900ce5f0 msgr2=0x7ff5900d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.639 7ff597322700 1 -- 172.21.15.32:0/1793948550 shutdown_connections 2021-04-09T18:31:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.639 7ff597322700 1 -- 172.21.15.32:0/1793948550 wait complete. 2021-04-09T18:31:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.659 7ff597322700 1 Processor -- start 2021-04-09T18:31:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.659 7ff597322700 1 -- start start 2021-04-09T18:31:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff597322700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900a90c0 0x7ff5901a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff597322700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5901a76d0 0x7ff5901a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:59.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff597322700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff59017bce0 0x7ff59017c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:59.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff597322700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5901a7c60 con 0x7ff59017bce0 2021-04-09T18:31:59.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff597322700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5901a8e20 con 0x7ff5900a90c0 2021-04-09T18:31:59.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff597322700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59017db80 con 0x7ff5901a76d0 2021-04-09T18:31:59.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59593f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff59017bce0 0x7ff59017c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:59.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59593f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff59017bce0 0x7ff59017c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57314/0 (socket says 172.21.15.32:57314) 2021-04-09T18:31:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59593f700 1 -- 172.21.15.32:0/3319125319 learned_addr learned my addr 172.21.15.32:0/3319125319 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:31:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59493d700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5901a76d0 0x7ff5901a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:59.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59513e700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900a90c0 0x7ff5901a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59593f700 1 -- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5901a76d0 msgr2=0x7ff5901a8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59593f700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5901a76d0 0x7ff5901a8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59593f700 1 -- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900a90c0 msgr2=0x7ff5901a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59593f700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900a90c0 0x7ff5901a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59593f700 1 -- 172.21.15.32:0/3319125319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590179610 con 0x7ff59017bce0 2021-04-09T18:31:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff59593f700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff59017bce0 0x7ff59017c0e0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 rx=0x7ff58800b910 tx=0x7ff58800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:59.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff585ffb700 1 -- 172.21.15.32:0/3319125319 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff588006aa0 con 0x7ff59017bce0 2021-04-09T18:31:59.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff585ffb700 1 -- 172.21.15.32:0/3319125319 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5880011f0 con 0x7ff59017bce0 2021-04-09T18:31:59.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.660 7ff597322700 1 -- 172.21.15.32:0/3319125319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff590179610 con 0x7ff59017bce0 2021-04-09T18:31:59.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.661 7ff585ffb700 1 -- 172.21.15.32:0/3319125319 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff588001640 con 0x7ff59017bce0 2021-04-09T18:31:59.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.661 7ff597322700 1 -- 172.21.15.32:0/3319125319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff590179610 con 0x7ff59017bce0 2021-04-09T18:31:59.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.661 7ff585ffb700 1 -- 172.21.15.32:0/3319125319 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff5880056b0 con 0x7ff59017bce0 2021-04-09T18:31:59.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.661 7ff585ffb700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff570018930 0x7ff57001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:31:59.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.661 7ff585ffb700 1 -- 172.21.15.32:0/3319125319 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7ff588033180 con 0x7ff59017bce0 2021-04-09T18:31:59.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.661 7ff59513e700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff570018930 0x7ff57001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:31:59.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.662 7ff59513e700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff570018930 0x7ff57001ae10 crc :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:31:59.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.681 7ff597322700 1 -- 172.21.15.32:0/3319125319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5901b7010 con 0x7ff59017bce0 2021-04-09T18:31:59.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.685 7ff585ffb700 1 -- 172.21.15.32:0/3319125319 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff58800bdf0 con 0x7ff59017bce0 2021-04-09T18:31:59.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.876 7ff597322700 1 -- 172.21.15.32:0/3319125319 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff59010d480 con 0x7ff59017bce0 2021-04-09T18:31:59.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.876 7ff585ffb700 1 -- 172.21.15.32:0/3319125319 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 366 v366) v1 ==== 95+0+3755 (secure 0 0 0) 0x7ff5880011f0 con 0x7ff59017bce0 2021-04-09T18:31:59.893 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 366 2021-04-09T18:31:59.894 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:31:59.894 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":366,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":364,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:31:53.539936","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":11697,"mds_1":59147},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59147":{"gid":59147,"name":"b","rank":1,"incarnation":363,"state":"up:active","state_seq":6,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":366,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:59.447586","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":11700,"mds_1":59144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59144":{"gid":59144,"name":"d","rank":1,"incarnation":360,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:31:59.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.878 7ff597322700 1 -- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff570018930 msgr2=0x7ff57001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:59.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.878 7ff597322700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff570018930 0x7ff57001ae10 crc :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.878 7ff597322700 1 -- 172.21.15.32:0/3319125319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff59017bce0 msgr2=0x7ff59017c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:31:59.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.878 7ff597322700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff59017bce0 0x7ff59017c0e0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 rx=0x7ff58800b910 tx=0x7ff58800bcb0).stop 2021-04-09T18:31:59.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.879 7ff597322700 1 -- 172.21.15.32:0/3319125319 shutdown_connections 2021-04-09T18:31:59.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.879 7ff597322700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff570018930 0x7ff57001ae10 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.879 7ff597322700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff59017bce0 0x7ff59017c0e0 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.879 7ff597322700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5901a76d0 0x7ff5901a8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.879 7ff597322700 1 --2- 172.21.15.32:0/3319125319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900a90c0 0x7ff5901a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:31:59.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.879 7ff597322700 1 -- 172.21.15.32:0/3319125319 >> 172.21.15.32:0/3319125319 conn(0x7ff5901acae0 msgr2=0x7ff59017ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:31:59.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.879 7ff597322700 1 -- 172.21.15.32:0/3319125319 shutdown_connections 2021-04-09T18:31:59.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:31:59.879 7ff597322700 1 -- 172.21.15.32:0/3319125319 wait complete. 2021-04-09T18:31:59.905 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set alpha max_mds 3 2021-04-09T18:32:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.043 7f8474612700 1 Processor -- start 2021-04-09T18:32:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.043 7f8474612700 1 -- start start 2021-04-09T18:32:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8474612700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c10d010 0x7f846c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8474612700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f846c0d8490 0x7f846c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8474612700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846c10d480 con 0x7f846c0d1da0 2021-04-09T18:32:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8474612700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f846c10e780 con 0x7f846c0d8490 2021-04-09T18:32:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8474612700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846c0d8d30 con 0x7f846c10d010 2021-04-09T18:32:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8472c2f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f846c0d8490 0x7f846c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8472c2f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f846c0d8490 0x7f846c10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57324/0 (socket says 172.21.15.32:57324) 2021-04-09T18:32:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8472c2f700 1 -- 172.21.15.32:0/2806619172 learned_addr learned my addr 172.21.15.32:0/2806619172 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f847242e700 1 --2- 172.21.15.32:0/2806619172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c10d010 0x7f846c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8472c2f700 1 -- 172.21.15.32:0/2806619172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c10d010 msgr2=0x7f846c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8472c2f700 1 --2- 172.21.15.32:0/2806619172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c10d010 0x7f846c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8472c2f700 1 -- 172.21.15.32:0/2806619172 >> v1:172.21.15.86:6789/0 conn(0x7f846c0d1da0 legacy=0x7f846c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8472c2f700 1 -- 172.21.15.32:0/2806619172 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c0d9220 con 0x7f846c0d8490 2021-04-09T18:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f8472c2f700 1 --2- 172.21.15.32:0/2806619172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0d8490 0x7f846c10da00 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 rx=0x7f8468006b80 tx=0x7f8468009b90).ready entity=mon.0 client_cookie=e36db52f61712801 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.044 7f847142c700 1 -- 172.21.15.32:0/2806619172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84680022e0 con 0x7f846c0d8490 2021-04-09T18:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f847142c700 1 -- 172.21.15.32:0/2806619172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8468002620 con 0x7f846c0d8490 2021-04-09T18:32:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f847142c700 1 -- 172.21.15.32:0/2806619172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8468002b00 con 0x7f846c0d8490 2021-04-09T18:32:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f8474612700 1 -- 172.21.15.32:0/2806619172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0d8490 msgr2=0x7f846c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f8474612700 1 --2- 172.21.15.32:0/2806619172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0d8490 0x7f846c10da00 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 rx=0x7f8468006b80 tx=0x7f8468009b90).stop 2021-04-09T18:32:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f8474612700 1 -- 172.21.15.32:0/2806619172 shutdown_connections 2021-04-09T18:32:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f8474612700 1 --2- 172.21.15.32:0/2806619172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0d8490 0x7f846c10da00 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f8474612700 1 --2- 172.21.15.32:0/2806619172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c10d010 0x7f846c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f8474612700 1 -- 172.21.15.32:0/2806619172 >> 172.21.15.32:0/2806619172 conn(0x7f846c0ce5f0 msgr2=0x7f846c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f8474612700 1 -- 172.21.15.32:0/2806619172 shutdown_connections 2021-04-09T18:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.045 7f8474612700 1 -- 172.21.15.32:0/2806619172 wait complete. 2021-04-09T18:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.065 7f8474612700 1 Processor -- start 2021-04-09T18:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8474612700 1 -- start start 2021-04-09T18:32:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8474612700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f846c1a0050 0x7f846c1a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8474612700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c0bdcd0 0x7f846c0bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8474612700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0bf080 0x7f846c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8474612700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c0bf500 con 0x7f846c0bf080 2021-04-09T18:32:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8474612700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c0bb7b0 con 0x7f846c1a0050 2021-04-09T18:32:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8474612700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846c0be0d0 con 0x7f846c0bdcd0 2021-04-09T18:32:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8472c2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0bf080 0x7f846c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8472c2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0bf080 0x7f846c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57330/0 (socket says 172.21.15.32:57330) 2021-04-09T18:32:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8472c2f700 1 -- 172.21.15.32:0/342186209 learned_addr learned my addr 172.21.15.32:0/342186209 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:00.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f847242e700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f846c1a0050 0x7f846c1a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.066 7f8471c2d700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c0bdcd0 0x7f846c0bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8472c2f700 1 -- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c0bdcd0 msgr2=0x7f846c0bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8472c2f700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c0bdcd0 0x7f846c0bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8472c2f700 1 -- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f846c1a0050 msgr2=0x7f846c1a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8472c2f700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f846c1a0050 0x7f846c1a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8472c2f700 1 -- 172.21.15.32:0/342186209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c0b1310 con 0x7f846c0bf080 2021-04-09T18:32:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f847242e700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f846c1a0050 0x7f846c1a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8471c2d700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c0bdcd0 0x7f846c0bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8472c2f700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0bf080 0x7f846c0b0cf0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 rx=0x7f8468006f10 tx=0x7f846800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8462ffd700 1 -- 172.21.15.32:0/342186209 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8468000b90 con 0x7f846c0bf080 2021-04-09T18:32:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8462ffd700 1 -- 172.21.15.32:0/342186209 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846800bd50 con 0x7f846c0bf080 2021-04-09T18:32:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8474612700 1 -- 172.21.15.32:0/342186209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c0b1310 con 0x7f846c0bf080 2021-04-09T18:32:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8462ffd700 1 -- 172.21.15.32:0/342186209 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84680022b0 con 0x7f846c0bf080 2021-04-09T18:32:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.067 7f8474612700 1 -- 172.21.15.32:0/342186209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846c0b1310 con 0x7f846c0bf080 2021-04-09T18:32:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.068 7f8462ffd700 1 -- 172.21.15.32:0/342186209 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8468009100 con 0x7f846c0bf080 2021-04-09T18:32:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.068 7f8462ffd700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8454018930 0x7f845401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.068 7f8462ffd700 1 -- 172.21.15.32:0/342186209 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f8468032e20 con 0x7f846c0bf080 2021-04-09T18:32:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.068 7f847242e700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8454018930 0x7f845401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.068 7f847242e700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8454018930 0x7f845401ae10 crc :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.088 7f8474612700 1 -- 172.21.15.32:0/342186209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846c0bb7b0 con 0x7f846c0bf080 2021-04-09T18:32:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.091 7f8462ffd700 1 -- 172.21.15.32:0/342186209 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8468000b90 con 0x7f846c0bf080 2021-04-09T18:32:00.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.283 7f8474612700 1 -- 172.21.15.32:0/342186209 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "alpha", "val": "3"} v 0) v1 -- 0x7f846c0bb7b0 con 0x7f846c0bf080 2021-04-09T18:32:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.451 7f8462ffd700 1 -- 172.21.15.32:0/342186209 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "alpha", "val": "3"}]=0 v367) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8468009100 con 0x7f846c0bf080 2021-04-09T18:32:00.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 -- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8454018930 msgr2=0x7f845401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8454018930 0x7f845401ae10 crc :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 -- 172.21.15.32:0/342186209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0bf080 msgr2=0x7f846c0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0bf080 0x7f846c0b0cf0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 rx=0x7f8468006f10 tx=0x7f846800b960).stop 2021-04-09T18:32:00.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 -- 172.21.15.32:0/342186209 shutdown_connections 2021-04-09T18:32:00.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8454018930 0x7f845401ae10 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f846c0bf080 0x7f846c0b0cf0 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f846c0bdcd0 0x7f846c0bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 --2- 172.21.15.32:0/342186209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f846c1a0050 0x7f846c1a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 -- 172.21.15.32:0/342186209 >> 172.21.15.32:0/342186209 conn(0x7f846c0975f0 msgr2=0x7f846c096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:00.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 -- 172.21.15.32:0/342186209 shutdown_connections 2021-04-09T18:32:00.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.499 7f8474612700 1 -- 172.21.15.32:0/342186209 wait complete. 2021-04-09T18:32:00.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:00.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.660 7f9cc7a37700 1 Processor -- start 2021-04-09T18:32:00.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.660 7f9cc7a37700 1 -- start start 2021-04-09T18:32:00.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc7a37700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00cffa0 0x7f9cc00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc7a37700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9cc00d6c80 0x7f9cc010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc7a37700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc00d7280 con 0x7f9cc00d7800 2021-04-09T18:32:00.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc7a37700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc00d7080 con 0x7f9cc00d6c80 2021-04-09T18:32:00.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc7a37700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00d8580 con 0x7f9cc00cffa0 2021-04-09T18:32:00.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc6054700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9cc00d6c80 0x7f9cc010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc5052700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9cc00d7800 0x7f9cc00d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38694/0 (socket says 172.21.15.32:38694) 2021-04-09T18:32:00.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc5052700 1 -- 172.21.15.32:0/2677607733 learned_addr learned my addr 172.21.15.32:0/2677607733 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:00.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc5853700 1 --2- 172.21.15.32:0/2677607733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00cffa0 0x7f9cc00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc6054700 1 -- 172.21.15.32:0/2677607733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00cffa0 msgr2=0x7f9cc00d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc6054700 1 --2- 172.21.15.32:0/2677607733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00cffa0 0x7f9cc00d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc6054700 1 -- 172.21.15.32:0/2677607733 >> v1:172.21.15.32:6789/0 conn(0x7f9cc00d7800 legacy=0x7f9cc00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc4851700 1 -- 172.21.15.32:0/2677607733 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514996632 0 0) 0x7f9cb0000ce0 con 0x7f9cc00d7800 2021-04-09T18:32:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc6054700 1 -- 172.21.15.32:0/2677607733 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc00d91e0 con 0x7f9cc00d6c80 2021-04-09T18:32:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc6054700 1 --2- 172.21.15.32:0/2677607733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc00d6c80 0x7f9cc010cfd0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 rx=0x7f9cb8006e30 tx=0x7f9cb8009de0).ready entity=mon.0 client_cookie=8e841c92ccd1d939 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc4851700 1 -- 172.21.15.32:0/2677607733 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cb800b300 con 0x7f9cc00d6c80 2021-04-09T18:32:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc4851700 1 -- 172.21.15.32:0/2677607733 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb800b680 con 0x7f9cc00d6c80 2021-04-09T18:32:00.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc4851700 1 -- 172.21.15.32:0/2677607733 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cb800bb60 con 0x7f9cc00d6c80 2021-04-09T18:32:00.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.661 7f9cc7a37700 1 -- 172.21.15.32:0/2677607733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc00d6c80 msgr2=0x7f9cc010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.662 7f9cc7a37700 1 --2- 172.21.15.32:0/2677607733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc00d6c80 0x7f9cc010cfd0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 rx=0x7f9cb8006e30 tx=0x7f9cb8009de0).stop 2021-04-09T18:32:00.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.662 7f9cc7a37700 1 -- 172.21.15.32:0/2677607733 shutdown_connections 2021-04-09T18:32:00.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.662 7f9cc7a37700 1 --2- 172.21.15.32:0/2677607733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc00d6c80 0x7f9cc010cfd0 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.662 7f9cc7a37700 1 --2- 172.21.15.32:0/2677607733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00cffa0 0x7f9cc00d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.662 7f9cc7a37700 1 -- 172.21.15.32:0/2677607733 >> 172.21.15.32:0/2677607733 conn(0x7f9cc00ce5c0 msgr2=0x7f9cc00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.662 7f9cc7a37700 1 -- 172.21.15.32:0/2677607733 shutdown_connections 2021-04-09T18:32:00.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.662 7f9cc7a37700 1 -- 172.21.15.32:0/2677607733 wait complete. 2021-04-09T18:32:00.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc7a37700 1 Processor -- start 2021-04-09T18:32:00.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc7a37700 1 -- start start 2021-04-09T18:32:00.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc7a37700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc008ff00 0x7f9cc0090300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc7a37700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cc00be590 0x7f9cc00be990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc7a37700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00befb0 0x7f9cc00c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc7a37700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc008fa00 con 0x7f9cc008ff00 2021-04-09T18:32:00.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc7a37700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc018f050 con 0x7f9cc00be590 2021-04-09T18:32:00.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc7a37700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc008f5d0 con 0x7f9cc00befb0 2021-04-09T18:32:00.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc5853700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc008ff00 0x7f9cc0090300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc5853700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc008ff00 0x7f9cc0090300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57340/0 (socket says 172.21.15.32:57340) 2021-04-09T18:32:00.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.683 7f9cc5853700 1 -- 172.21.15.32:0/631718529 learned_addr learned my addr 172.21.15.32:0/631718529 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:00.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc6054700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00befb0 0x7f9cc00c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc5052700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cc00be590 0x7f9cc00be990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc5853700 1 -- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00befb0 msgr2=0x7f9cc00c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc5853700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00befb0 0x7f9cc00c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc5853700 1 -- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cc00be590 msgr2=0x7f9cc00be990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc5853700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cc00be590 0x7f9cc00be990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc5853700 1 -- 172.21.15.32:0/631718529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc009aca0 con 0x7f9cc008ff00 2021-04-09T18:32:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc5853700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc008ff00 0x7f9cc0090300 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 rx=0x7f9cbc006f00 tx=0x7f9cbc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cb67fc700 1 -- 172.21.15.32:0/631718529 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cbc00bd30 con 0x7f9cc008ff00 2021-04-09T18:32:00.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cb67fc700 1 -- 172.21.15.32:0/631718529 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cbc002030 con 0x7f9cc008ff00 2021-04-09T18:32:00.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc009aca0 con 0x7f9cc008ff00 2021-04-09T18:32:00.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cb67fc700 1 -- 172.21.15.32:0/631718529 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cbc002590 con 0x7f9cc008ff00 2021-04-09T18:32:00.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.684 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc009aca0 con 0x7f9cc008ff00 2021-04-09T18:32:00.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.685 7f9cb67fc700 1 -- 172.21.15.32:0/631718529 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9cbc000c90 con 0x7f9cc008ff00 2021-04-09T18:32:00.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.685 7f9cb67fc700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ca8018930 0x7f9ca801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:00.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.685 7f9cb67fc700 1 -- 172.21.15.32:0/631718529 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f9cbc032e50 con 0x7f9cc008ff00 2021-04-09T18:32:00.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.685 7f9cc5052700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ca8018930 0x7f9ca801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:00.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.686 7f9cc5052700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ca8018930 0x7f9ca801ae10 crc :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:00.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.704 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc0182df0 con 0x7f9cc008ff00 2021-04-09T18:32:00.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.708 7f9cb67fc700 1 -- 172.21.15.32:0/631718529 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9cbc00bd30 con 0x7f9cc008ff00 2021-04-09T18:32:00.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.899 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cc0182df0 con 0x7f9cc008ff00 2021-04-09T18:32:00.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.900 7f9cb67fc700 1 -- 172.21.15.32:0/631718529 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+3755 (secure 0 0 0) 0x7f9cbc000c90 con 0x7f9cc008ff00 2021-04-09T18:32:00.916 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 367 2021-04-09T18:32:00.916 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:00.916 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":367,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:32:00.449180","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1],"up":{"mds_0":11697,"mds_1":59147},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59147":{"gid":59147,"name":"b","rank":1,"incarnation":363,"state":"up:active","state_seq":6,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":366,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:59.447586","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":11700,"mds_1":59144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59144":{"gid":59144,"name":"d","rank":1,"incarnation":360,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:32:00.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ca8018930 msgr2=0x7f9ca801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ca8018930 0x7f9ca801ae10 crc :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc008ff00 msgr2=0x7f9cc0090300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:00.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc008ff00 0x7f9cc0090300 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 rx=0x7f9cbc006f00 tx=0x7f9cbc00b940).stop 2021-04-09T18:32:00.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 shutdown_connections 2021-04-09T18:32:00.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9ca8018930 0x7f9ca801ae10 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cc00befb0 0x7f9cc00c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cc00be590 0x7f9cc00be990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 --2- 172.21.15.32:0/631718529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc008ff00 0x7f9cc0090300 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:00.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 >> 172.21.15.32:0/631718529 conn(0x7f9cc009b330 msgr2=0x7f9cc01a9650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:00.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 shutdown_connections 2021-04-09T18:32:00.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:00.902 7f9cc7a37700 1 -- 172.21.15.32:0/631718529 wait complete. 2021-04-09T18:32:00.928 DEBUG:tasks.ceph_test_case:wait_until_equal: 2 != 3, waiting... 2021-04-09T18:32:05.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:06.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.075 7fb68e4b6700 1 Processor -- start 2021-04-09T18:32:06.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.075 7fb68e4b6700 1 -- start start 2021-04-09T18:32:06.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb68e4b6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb68810d010 0x7fb6880d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:06.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb68e4b6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb6880d8490 0x7fb68810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:06.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb68e4b6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68810d480 con 0x7fb6880d1da0 2021-04-09T18:32:06.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb68e4b6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6880d8d30 con 0x7fb68810d010 2021-04-09T18:32:06.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb68e4b6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6880d9220 con 0x7fb6880d8490 2021-04-09T18:32:06.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb687fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb68810d010 0x7fb6880d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:06.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb67ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb6880d1da0 0x7fb6880d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38710/0 (socket says 172.21.15.32:38710) 2021-04-09T18:32:06.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb67ffff700 1 -- 172.21.15.32:0/4199425632 learned_addr learned my addr 172.21.15.32:0/4199425632 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:06.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb68cad3700 1 --2- 172.21.15.32:0/4199425632 >> v2:172.21.15.86:3300/0 conn(0x7fb6880d8490 0x7fb68810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:06.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb6877fe700 1 -- 172.21.15.32:0/4199425632 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276282521 0 0) 0x7fb670000ce0 con 0x7fb6880d1da0 2021-04-09T18:32:06.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.076 7fb6877fe700 1 -- 172.21.15.32:0/4199425632 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb674002fe0 con 0x7fb6880d1da0 2021-04-09T18:32:06.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb687fff700 1 -- 172.21.15.32:0/4199425632 >> v2:172.21.15.86:3300/0 conn(0x7fb6880d8490 msgr2=0x7fb68810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:06.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb687fff700 1 --2- 172.21.15.32:0/4199425632 >> v2:172.21.15.86:3300/0 conn(0x7fb6880d8490 0x7fb68810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb687fff700 1 -- 172.21.15.32:0/4199425632 >> v1:172.21.15.32:6789/0 conn(0x7fb6880d1da0 legacy=0x7fb6880d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:06.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb687fff700 1 -- 172.21.15.32:0/4199425632 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68810e020 con 0x7fb68810d010 2021-04-09T18:32:06.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb68cad3700 1 --2- 172.21.15.32:0/4199425632 >> v2:172.21.15.86:3300/0 conn(0x7fb6880d8490 0x7fb68810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:06.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb687fff700 1 --2- 172.21.15.32:0/4199425632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68810d010 0x7fb6880d1780 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 rx=0x7fb678006650 tx=0x7fb678009a80).ready entity=mon.0 client_cookie=10f78e949585877b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:06.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb6877fe700 1 -- 172.21.15.32:0/4199425632 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6780009b0 con 0x7fb68810d010 2021-04-09T18:32:06.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb6877fe700 1 -- 172.21.15.32:0/4199425632 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67800c0c0 con 0x7fb68810d010 2021-04-09T18:32:06.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb6877fe700 1 -- 172.21.15.32:0/4199425632 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb67800c520 con 0x7fb68810d010 2021-04-09T18:32:06.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb68e4b6700 1 -- 172.21.15.32:0/4199425632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68810d010 msgr2=0x7fb6880d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:06.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.077 7fb68e4b6700 1 --2- 172.21.15.32:0/4199425632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68810d010 0x7fb6880d1780 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 rx=0x7fb678006650 tx=0x7fb678009a80).stop 2021-04-09T18:32:06.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.078 7fb68e4b6700 1 -- 172.21.15.32:0/4199425632 shutdown_connections 2021-04-09T18:32:06.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.078 7fb68e4b6700 1 --2- 172.21.15.32:0/4199425632 >> v2:172.21.15.86:3300/0 conn(0x7fb6880d8490 0x7fb68810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.078 7fb68e4b6700 1 --2- 172.21.15.32:0/4199425632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb68810d010 0x7fb6880d1780 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.078 7fb68e4b6700 1 -- 172.21.15.32:0/4199425632 >> 172.21.15.32:0/4199425632 conn(0x7fb6880ce5f0 msgr2=0x7fb6880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:06.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.078 7fb68e4b6700 1 -- 172.21.15.32:0/4199425632 shutdown_connections 2021-04-09T18:32:06.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.078 7fb68e4b6700 1 -- 172.21.15.32:0/4199425632 wait complete. 2021-04-09T18:32:06.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.098 7fb68e4b6700 1 Processor -- start 2021-04-09T18:32:06.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.098 7fb68e4b6700 1 -- start start 2021-04-09T18:32:06.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68e4b6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb688179fd0 0x7fb6880976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:06.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68e4b6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb688097cc0 0x7fb6881a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:06.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68e4b6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6881a4b10 0x7fb6881a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:06.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68e4b6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb688098140 con 0x7fb688097cc0 2021-04-09T18:32:06.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68e4b6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68817a3d0 con 0x7fb688179fd0 2021-04-09T18:32:06.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68e4b6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68817c870 con 0x7fb6881a4b10 2021-04-09T18:32:06.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb67ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb688097cc0 0x7fb6881a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:06.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb67ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb688097cc0 0x7fb6881a44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57358/0 (socket says 172.21.15.32:57358) 2021-04-09T18:32:06.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb67ffff700 1 -- 172.21.15.32:0/2984535188 learned_addr learned my addr 172.21.15.32:0/2984535188 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:06.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68cad3700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6881a4b10 0x7fb6881a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:06.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb687fff700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb688179fd0 0x7fb6880976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:06.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb67ffff700 1 -- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6881a4b10 msgr2=0x7fb6881a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb67ffff700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6881a4b10 0x7fb6881a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb67ffff700 1 -- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb688179fd0 msgr2=0x7fb6880976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb67ffff700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb688179fd0 0x7fb6880976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb67ffff700 1 -- 172.21.15.32:0/2984535188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb688179bf0 con 0x7fb688097cc0 2021-04-09T18:32:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb687fff700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb688179fd0 0x7fb6880976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68cad3700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6881a4b10 0x7fb6881a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb67ffff700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb688097cc0 0x7fb6881a44f0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 rx=0x7fb67000a5d0 tx=0x7fb67000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb6857fa700 1 -- 172.21.15.32:0/2984535188 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb670000ea0 con 0x7fb688097cc0 2021-04-09T18:32:06.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb6857fa700 1 -- 172.21.15.32:0/2984535188 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67000ad60 con 0x7fb688097cc0 2021-04-09T18:32:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb688179bf0 con 0x7fb688097cc0 2021-04-09T18:32:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb6857fa700 1 -- 172.21.15.32:0/2984535188 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb670004590 con 0x7fb688097cc0 2021-04-09T18:32:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.099 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb688179bf0 con 0x7fb688097cc0 2021-04-09T18:32:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.100 7fb6857fa700 1 -- 172.21.15.32:0/2984535188 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb670008000 con 0x7fb688097cc0 2021-04-09T18:32:06.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.100 7fb6857fa700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb66c0188c0 0x7fb66c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.100 7fb6857fa700 1 -- 172.21.15.32:0/2984535188 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fb670032dd0 con 0x7fb688097cc0 2021-04-09T18:32:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.101 7fb687fff700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb66c0188c0 0x7fb66c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:06.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.101 7fb687fff700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb66c0188c0 0x7fb66c01adc0 crc :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:06.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.120 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6881b7050 con 0x7fb688097cc0 2021-04-09T18:32:06.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.123 7fb6857fa700 1 -- 172.21.15.32:0/2984535188 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb6700075e0 con 0x7fb688097cc0 2021-04-09T18:32:06.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.315 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb68810d480 con 0x7fb688097cc0 2021-04-09T18:32:06.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.315 7fb6857fa700 1 -- 172.21.15.32:0/2984535188 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+3755 (secure 0 0 0) 0x7fb67000ad60 con 0x7fb688097cc0 2021-04-09T18:32:06.332 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 367 2021-04-09T18:32:06.332 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:06.332 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":367,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:32:00.449180","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1],"up":{"mds_0":11697,"mds_1":59147},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59147":{"gid":59147,"name":"b","rank":1,"incarnation":363,"state":"up:active","state_seq":6,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":366,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:31:59.447586","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":11700,"mds_1":59144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59144":{"gid":59144,"name":"d","rank":1,"incarnation":360,"state":"up:stopping","state_seq":4,"addr":"172.21.15.86:6836/1479556423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1479556423},{"type":"v1","addr":"172.21.15.86:6836","nonce":1479556423}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:32:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb66c0188c0 msgr2=0x7fb66c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb66c0188c0 0x7fb66c01adc0 crc :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb688097cc0 msgr2=0x7fb6881a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:06.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb688097cc0 0x7fb6881a44f0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 rx=0x7fb67000a5d0 tx=0x7fb67000a970).stop 2021-04-09T18:32:06.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 shutdown_connections 2021-04-09T18:32:06.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb66c0188c0 0x7fb66c01adc0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb688097cc0 0x7fb6881a44f0 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6881a4b10 0x7fb6881a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 --2- 172.21.15.32:0/2984535188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb688179fd0 0x7fb6880976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 >> 172.21.15.32:0/2984535188 conn(0x7fb688199680 msgr2=0x7fb6881a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 shutdown_connections 2021-04-09T18:32:06.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:06.318 7fb68e4b6700 1 -- 172.21.15.32:0/2984535188 wait complete. 2021-04-09T18:32:06.344 DEBUG:tasks.ceph_test_case:wait_until_equal: 2 != 3, waiting... 2021-04-09T18:32:08.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:32:08.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:32:09.614 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:32:11.345 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:11.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.490 7f99476f7700 1 Processor -- start 2021-04-09T18:32:11.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.490 7f99476f7700 1 -- start start 2021-04-09T18:32:11.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f99476f7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f994010d010 0x7f99400d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:11.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f99476f7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99400d1da0 0x7f99400d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:11.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f99476f7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994010d480 con 0x7f99400d82c0 2021-04-09T18:32:11.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f99476f7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994010e780 con 0x7f99400d1da0 2021-04-09T18:32:11.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f99476f7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99400d8d30 con 0x7f994010d010 2021-04-09T18:32:11.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f9944d12700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99400d1da0 0x7f99400d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:11.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f9944d12700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99400d1da0 0x7f99400d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57362/0 (socket says 172.21.15.32:57362) 2021-04-09T18:32:11.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f9944d12700 1 -- 172.21.15.32:0/559268793 learned_addr learned my addr 172.21.15.32:0/559268793 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:11.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f9945513700 1 --2- 172.21.15.32:0/559268793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f994010d010 0x7f99400d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:11.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f9944d12700 1 -- 172.21.15.32:0/559268793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f994010d010 msgr2=0x7f99400d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:11.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f9944d12700 1 --2- 172.21.15.32:0/559268793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f994010d010 0x7f99400d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f9944d12700 1 -- 172.21.15.32:0/559268793 >> v1:172.21.15.86:6789/0 conn(0x7f99400d82c0 legacy=0x7f994010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:11.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f9944d12700 1 -- 172.21.15.32:0/559268793 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400d9220 con 0x7f99400d1da0 2021-04-09T18:32:11.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.491 7f9944d12700 1 --2- 172.21.15.32:0/559268793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99400d1da0 0x7f99400d7ca0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 rx=0x7f9930006700 tx=0x7f9930009b80).ready entity=mon.0 client_cookie=276039841f3fd7d8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:11.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.492 7f9937fff700 1 -- 172.21.15.32:0/559268793 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f993000c0c0 con 0x7f99400d1da0 2021-04-09T18:32:11.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.492 7f9937fff700 1 -- 172.21.15.32:0/559268793 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993000c440 con 0x7f99400d1da0 2021-04-09T18:32:11.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.492 7f9937fff700 1 -- 172.21.15.32:0/559268793 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f993000c920 con 0x7f99400d1da0 2021-04-09T18:32:11.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.492 7f99476f7700 1 -- 172.21.15.32:0/559268793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99400d1da0 msgr2=0x7f99400d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:11.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.492 7f99476f7700 1 --2- 172.21.15.32:0/559268793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99400d1da0 0x7f99400d7ca0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 rx=0x7f9930006700 tx=0x7f9930009b80).stop 2021-04-09T18:32:11.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.492 7f99476f7700 1 -- 172.21.15.32:0/559268793 shutdown_connections 2021-04-09T18:32:11.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.492 7f99476f7700 1 --2- 172.21.15.32:0/559268793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99400d1da0 0x7f99400d7ca0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.492 7f99476f7700 1 --2- 172.21.15.32:0/559268793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f994010d010 0x7f99400d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.492 7f99476f7700 1 -- 172.21.15.32:0/559268793 >> 172.21.15.32:0/559268793 conn(0x7f99400ce5f0 msgr2=0x7f99400d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:11.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.493 7f99476f7700 1 -- 172.21.15.32:0/559268793 shutdown_connections 2021-04-09T18:32:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.493 7f99476f7700 1 -- 172.21.15.32:0/559268793 wait complete. 2021-04-09T18:32:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.513 7f99476f7700 1 Processor -- start 2021-04-09T18:32:11.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.513 7f99476f7700 1 -- start start 2021-04-09T18:32:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.513 7f99476f7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99401aca20 0x7f99401ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.513 7f99476f7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994018bd40 0x7f994018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:11.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f99476f7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f994008fba0 0x7f994008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:11.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f99476f7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994018c7e0 con 0x7f994018bd40 2021-04-09T18:32:11.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f99476f7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994018ca30 con 0x7f994008fba0 2021-04-09T18:32:11.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f99476f7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99400905f0 con 0x7f99401aca20 2021-04-09T18:32:11.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9944d12700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994018bd40 0x7f994018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9944d12700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994018bd40 0x7f994018c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57368/0 (socket says 172.21.15.32:57368) 2021-04-09T18:32:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9944d12700 1 -- 172.21.15.32:0/3662583096 learned_addr learned my addr 172.21.15.32:0/3662583096 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9945513700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99401aca20 0x7f99401ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9945d14700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f994008fba0 0x7f994008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:11.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9944d12700 1 -- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99401aca20 msgr2=0x7f99401ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9944d12700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99401aca20 0x7f99401ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9944d12700 1 -- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f994008fba0 msgr2=0x7f994008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9944d12700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f994008fba0 0x7f994008ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9944d12700 1 -- 172.21.15.32:0/3662583096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400b4300 con 0x7f994018bd40 2021-04-09T18:32:11.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9945d14700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f994008fba0 0x7f994008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9944d12700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994018bd40 0x7f994018c140 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 rx=0x7f993000b650 tx=0x7f993000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9935ffb700 1 -- 172.21.15.32:0/3662583096 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f993000bde0 con 0x7f994018bd40 2021-04-09T18:32:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9935ffb700 1 -- 172.21.15.32:0/3662583096 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9930006570 con 0x7f994018bd40 2021-04-09T18:32:11.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f9935ffb700 1 -- 172.21.15.32:0/3662583096 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9930002220 con 0x7f994018bd40 2021-04-09T18:32:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f99476f7700 1 -- 172.21.15.32:0/3662583096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99400b4300 con 0x7f994018bd40 2021-04-09T18:32:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.514 7f99476f7700 1 -- 172.21.15.32:0/3662583096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99400b4300 con 0x7f994018bd40 2021-04-09T18:32:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.515 7f9935ffb700 1 -- 172.21.15.32:0/3662583096 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9930009f90 con 0x7f994018bd40 2021-04-09T18:32:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.515 7f9935ffb700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9920018930 0x7f992001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:11.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.515 7f9935ffb700 1 -- 172.21.15.32:0/3662583096 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f9930032e10 con 0x7f994018bd40 2021-04-09T18:32:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.515 7f9945513700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9920018930 0x7f992001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.516 7f9945513700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9920018930 0x7f992001adc0 crc :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:11.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.535 7f99476f7700 1 -- 172.21.15.32:0/3662583096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9940182d70 con 0x7f994018bd40 2021-04-09T18:32:11.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.538 7f9935ffb700 1 -- 172.21.15.32:0/3662583096 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9930000c90 con 0x7f994018bd40 2021-04-09T18:32:11.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.730 7f99476f7700 1 -- 172.21.15.32:0/3662583096 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f994018c760 con 0x7f994018bd40 2021-04-09T18:32:11.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.731 7f9935ffb700 1 -- 172.21.15.32:0/3662583096 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 368 v368) v1 ==== 95+0+3399 (secure 0 0 0) 0x7f9930006570 con 0x7f994018bd40 2021-04-09T18:32:11.748 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 368 2021-04-09T18:32:11.748 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:11.748 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":368,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:32:00.449180","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1],"up":{"mds_0":11697,"mds_1":59147},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59147":{"gid":59147,"name":"b","rank":1,"incarnation":363,"state":"up:active","state_seq":6,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":368,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:32:09.536281","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11700},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:32:11.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 -- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9920018930 msgr2=0x7f992001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:11.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9920018930 0x7f992001adc0 crc :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 -- 172.21.15.32:0/3662583096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994018bd40 msgr2=0x7f994018c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:11.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994018bd40 0x7f994018c140 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 rx=0x7f993000b650 tx=0x7f993000b9f0).stop 2021-04-09T18:32:11.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 -- 172.21.15.32:0/3662583096 shutdown_connections 2021-04-09T18:32:11.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9920018930 0x7f992001adc0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f994018bd40 0x7f994018c140 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f994008fba0 0x7f994008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 --2- 172.21.15.32:0/3662583096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99401aca20 0x7f99401ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:11.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 -- 172.21.15.32:0/3662583096 >> 172.21.15.32:0/3662583096 conn(0x7f994019b460 msgr2=0x7f994019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:11.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 -- 172.21.15.32:0/3662583096 shutdown_connections 2021-04-09T18:32:11.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:11.733 7f99476f7700 1 -- 172.21.15.32:0/3662583096 wait complete. 2021-04-09T18:32:11.760 DEBUG:tasks.ceph_test_case:wait_until_equal: 2 != 3, waiting... 2021-04-09T18:32:16.761 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:16.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.903 7f440640e700 1 Processor -- start 2021-04-09T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.903 7f440640e700 1 -- start start 2021-04-09T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f440640e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440010d010 0x7f44000d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f440640e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44000d8490 0x7f440010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:16.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f440640e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440010d480 con 0x7f44000d1da0 2021-04-09T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f440640e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f440010e780 con 0x7f44000d8490 2021-04-09T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f440640e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44000d8d30 con 0x7f440010d010 2021-04-09T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f4404a2b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44000d8490 0x7f440010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:16.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f4404a2b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f44000d8490 0x7f440010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57380/0 (socket says 172.21.15.32:57380) 2021-04-09T18:32:16.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f4404a2b700 1 -- 172.21.15.32:0/1164738107 learned_addr learned my addr 172.21.15.32:0/1164738107 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:16.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f43fffff700 1 --2- 172.21.15.32:0/1164738107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440010d010 0x7f44000d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:16.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f4404a2b700 1 -- 172.21.15.32:0/1164738107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440010d010 msgr2=0x7f44000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:16.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f4404a2b700 1 --2- 172.21.15.32:0/1164738107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440010d010 0x7f44000d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:16.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f4404a2b700 1 -- 172.21.15.32:0/1164738107 >> v1:172.21.15.86:6789/0 conn(0x7f44000d1da0 legacy=0x7f44000d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:16.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f43ff7fe700 1 -- 172.21.15.32:0/1164738107 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378577820 0 0) 0x7f43e8000ff0 con 0x7f44000d1da0 2021-04-09T18:32:16.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f4404a2b700 1 -- 172.21.15.32:0/1164738107 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000d9220 con 0x7f44000d8490 2021-04-09T18:32:16.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.904 7f43fffff700 1 --2- 172.21.15.32:0/1164738107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440010d010 0x7f44000d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:16.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f4404a2b700 1 --2- 172.21.15.32:0/1164738107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44000d8490 0x7f440010da00 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 rx=0x7f43f8006f00 tx=0x7f43f8009f10).ready entity=mon.0 client_cookie=63a422f44b56a4fb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:16.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f43ff7fe700 1 -- 172.21.15.32:0/1164738107 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f800b3b0 con 0x7f44000d8490 2021-04-09T18:32:16.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f43ff7fe700 1 -- 172.21.15.32:0/1164738107 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f800b730 con 0x7f44000d8490 2021-04-09T18:32:16.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f43ff7fe700 1 -- 172.21.15.32:0/1164738107 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f800bc10 con 0x7f44000d8490 2021-04-09T18:32:16.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f440640e700 1 -- 172.21.15.32:0/1164738107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44000d8490 msgr2=0x7f440010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:16.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f440640e700 1 --2- 172.21.15.32:0/1164738107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44000d8490 0x7f440010da00 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 rx=0x7f43f8006f00 tx=0x7f43f8009f10).stop 2021-04-09T18:32:16.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f440640e700 1 -- 172.21.15.32:0/1164738107 shutdown_connections 2021-04-09T18:32:16.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f440640e700 1 --2- 172.21.15.32:0/1164738107 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44000d8490 0x7f440010da00 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:16.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f440640e700 1 --2- 172.21.15.32:0/1164738107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440010d010 0x7f44000d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:16.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.905 7f440640e700 1 -- 172.21.15.32:0/1164738107 >> 172.21.15.32:0/1164738107 conn(0x7f44000ce5f0 msgr2=0x7f44000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:16.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.906 7f440640e700 1 -- 172.21.15.32:0/1164738107 shutdown_connections 2021-04-09T18:32:16.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.906 7f440640e700 1 -- 172.21.15.32:0/1164738107 wait complete. 2021-04-09T18:32:16.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f440640e700 1 Processor -- start 2021-04-09T18:32:16.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f440640e700 1 -- start start 2021-04-09T18:32:16.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f440640e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44001aca20 0x7f44001ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:16.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f440640e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f440018bd40 0x7f440018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:16.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f440640e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440008fba0 0x7f440008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:16.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f440640e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440018c7e0 con 0x7f440008fba0 2021-04-09T18:32:16.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f440640e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440018ca30 con 0x7f440018bd40 2021-04-09T18:32:16.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f440640e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44000905f0 con 0x7f44001aca20 2021-04-09T18:32:16.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f4404a2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440008fba0 0x7f440008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:16.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.926 7f4404a2b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440008fba0 0x7f440008ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57388/0 (socket says 172.21.15.32:57388) 2021-04-09T18:32:16.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f4404a2b700 1 -- 172.21.15.32:0/2793681974 learned_addr learned my addr 172.21.15.32:0/2793681974 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:16.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fffff700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44001aca20 0x7f44001ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:16.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43f77fe700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f440018bd40 0x7f440018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:16.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fffff700 1 -- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f440018bd40 msgr2=0x7f440018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:16.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fffff700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f440018bd40 0x7f440018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:16.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fffff700 1 -- 172.21.15.32:0/2793681974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440008fba0 msgr2=0x7f440008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:16.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fffff700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440008fba0 0x7f440008ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:16.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fffff700 1 -- 172.21.15.32:0/2793681974 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000b4300 con 0x7f44001aca20 2021-04-09T18:32:16.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f4404a2b700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440008fba0 0x7f440008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:16.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fffff700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44001aca20 0x7f44001ace20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f43f0009540 tx=0x7f43f000b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:16.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fd7fa700 1 -- 172.21.15.32:0/2793681974 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f000bcd0 con 0x7f44001aca20 2021-04-09T18:32:16.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fd7fa700 1 -- 172.21.15.32:0/2793681974 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f00063e0 con 0x7f44001aca20 2021-04-09T18:32:16.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f440640e700 1 -- 172.21.15.32:0/2793681974 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44000b4300 con 0x7f44001aca20 2021-04-09T18:32:16.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f43fd7fa700 1 -- 172.21.15.32:0/2793681974 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f00022b0 con 0x7f44001aca20 2021-04-09T18:32:16.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.927 7f440640e700 1 -- 172.21.15.32:0/2793681974 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44000b4300 con 0x7f44001aca20 2021-04-09T18:32:16.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.928 7f43fd7fa700 1 -- 172.21.15.32:0/2793681974 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f43f0009fe0 con 0x7f44001aca20 2021-04-09T18:32:16.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.928 7f43fd7fa700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43e4018930 0x7f43e401adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:16.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.928 7f43fd7fa700 1 -- 172.21.15.32:0/2793681974 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f43f0032d40 con 0x7f44001aca20 2021-04-09T18:32:16.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.928 7f43f77fe700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43e4018930 0x7f43e401adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:16.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.929 7f43f77fe700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43e4018930 0x7f43e401adf0 crc :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:16.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.948 7f440640e700 1 -- 172.21.15.32:0/2793681974 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4400182d70 con 0x7f44001aca20 2021-04-09T18:32:16.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:16.951 7f43fd7fa700 1 -- 172.21.15.32:0/2793681974 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f43f0000c90 con 0x7f44001aca20 2021-04-09T18:32:17.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.142 7f440640e700 1 -- 172.21.15.32:0/2793681974 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f440018c760 con 0x7f44001aca20 2021-04-09T18:32:17.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.143 7f43fd7fa700 1 -- 172.21.15.32:0/2793681974 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 371 v371) v1 ==== 95+0+3754 (secure 0 0 0) 0x7f43f0000c90 con 0x7f44001aca20 2021-04-09T18:32:17.160 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 371 2021-04-09T18:32:17.160 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:17.161 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":371,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:32:14.790358","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":11697,"mds_1":59147,"mds_2":59246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59147":{"gid":59147,"name":"b","rank":1,"incarnation":363,"state":"up:active","state_seq":6,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59246":{"gid":59246,"name":"d","rank":2,"incarnation":370,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/2540922065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2540922065},{"type":"v1","addr":"172.21.15.86:6836","nonce":2540922065}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":368,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:32:09.536281","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11700},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:32:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 -- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43e4018930 msgr2=0x7f43e401adf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:17.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43e4018930 0x7f43e401adf0 crc :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 -- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44001aca20 msgr2=0x7f44001ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:17.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44001aca20 0x7f44001ace20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f43f0009540 tx=0x7f43f000b8e0).stop 2021-04-09T18:32:17.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 -- 172.21.15.32:0/2793681974 shutdown_connections 2021-04-09T18:32:17.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43e4018930 0x7f43e401adf0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440008fba0 0x7f440008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f440018bd40 0x7f440018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 --2- 172.21.15.32:0/2793681974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44001aca20 0x7f44001ace20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.145 7f440640e700 1 -- 172.21.15.32:0/2793681974 >> 172.21.15.32:0/2793681974 conn(0x7f440019b460 msgr2=0x7f440019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:17.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.146 7f440640e700 1 -- 172.21.15.32:0/2793681974 shutdown_connections 2021-04-09T18:32:17.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.146 7f440640e700 1 -- 172.21.15.32:0/2793681974 wait complete. 2021-04-09T18:32:17.172 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:32:17.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:32:17.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:32:17.288 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:32:17.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:32:17.289 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:32:17.290 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:32:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:32:17.334 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:32:17.335 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:32:17.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:32:17.336 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:32:17.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:32:17.419 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:32:17.419 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:32:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:32:17.462 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:32:17.462 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:32:17.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:32:17.463 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:32:17.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:32:17.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.675 7efe2ffcf700 1 Processor -- start 2021-04-09T18:32:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.675 7efe2ffcf700 1 -- start start 2021-04-09T18:32:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.675 7efe2ffcf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe2810d010 0x7efe280d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ffcf700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efe280d8490 0x7efe2810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ffcf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2810d480 con 0x7efe280d1da0 2021-04-09T18:32:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ffcf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe280d8d30 con 0x7efe2810d010 2021-04-09T18:32:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ffcf700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe280d9220 con 0x7efe280d8490 2021-04-09T18:32:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ddeb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe2810d010 0x7efe280d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2d5ea700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efe280d1da0 0x7efe280d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38752/0 (socket says 172.21.15.32:38752) 2021-04-09T18:32:17.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2d5ea700 1 -- 172.21.15.32:0/4293916170 learned_addr learned my addr 172.21.15.32:0/4293916170 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:17.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2e5ec700 1 --2- 172.21.15.32:0/4293916170 >> v2:172.21.15.86:3300/0 conn(0x7efe280d8490 0x7efe2810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:17.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ddeb700 1 -- 172.21.15.32:0/4293916170 >> v2:172.21.15.86:3300/0 conn(0x7efe280d8490 msgr2=0x7efe2810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:17.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ddeb700 1 --2- 172.21.15.32:0/4293916170 >> v2:172.21.15.86:3300/0 conn(0x7efe280d8490 0x7efe2810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ddeb700 1 -- 172.21.15.32:0/4293916170 >> v1:172.21.15.32:6789/0 conn(0x7efe280d1da0 legacy=0x7efe280d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:17.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ddeb700 1 -- 172.21.15.32:0/4293916170 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2810e020 con 0x7efe2810d010 2021-04-09T18:32:17.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2cde9700 1 -- 172.21.15.32:0/4293916170 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133232145 0 0) 0x7efe18000ce0 con 0x7efe280d1da0 2021-04-09T18:32:17.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ddeb700 1 --2- 172.21.15.32:0/4293916170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2810d010 0x7efe280d1780 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7efe240069d0 tx=0x7efe24009dc0).ready entity=mon.0 client_cookie=aba33d39a586dc5e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:17.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2cde9700 1 -- 172.21.15.32:0/4293916170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe2400b510 con 0x7efe2810d010 2021-04-09T18:32:17.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2cde9700 1 -- 172.21.15.32:0/4293916170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe24006ac0 con 0x7efe2810d010 2021-04-09T18:32:17.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2cde9700 1 -- 172.21.15.32:0/4293916170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe2400ba80 con 0x7efe2810d010 2021-04-09T18:32:17.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ffcf700 1 -- 172.21.15.32:0/4293916170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2810d010 msgr2=0x7efe280d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:17.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.676 7efe2ffcf700 1 --2- 172.21.15.32:0/4293916170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2810d010 0x7efe280d1780 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7efe240069d0 tx=0x7efe24009dc0).stop 2021-04-09T18:32:17.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.677 7efe2ffcf700 1 -- 172.21.15.32:0/4293916170 shutdown_connections 2021-04-09T18:32:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.677 7efe2ffcf700 1 --2- 172.21.15.32:0/4293916170 >> v2:172.21.15.86:3300/0 conn(0x7efe280d8490 0x7efe2810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.677 7efe2ffcf700 1 --2- 172.21.15.32:0/4293916170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2810d010 0x7efe280d1780 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.677 7efe2ffcf700 1 -- 172.21.15.32:0/4293916170 >> 172.21.15.32:0/4293916170 conn(0x7efe280ce5f0 msgr2=0x7efe280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:17.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.677 7efe2ffcf700 1 -- 172.21.15.32:0/4293916170 shutdown_connections 2021-04-09T18:32:17.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.677 7efe2ffcf700 1 -- 172.21.15.32:0/4293916170 wait complete. 2021-04-09T18:32:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.697 7efe2ffcf700 1 Processor -- start 2021-04-09T18:32:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.697 7efe2ffcf700 1 -- start start 2021-04-09T18:32:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ffcf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2809a5e0 0x7efe281884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ffcf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a3350 0x7efe280a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ffcf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280a15b0 0x7efe280a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ffcf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe280a40d0 con 0x7efe2809a5e0 2021-04-09T18:32:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ffcf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe280a2dd0 con 0x7efe280a3350 2021-04-09T18:32:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ffcf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe28188ad0 con 0x7efe280a15b0 2021-04-09T18:32:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ddeb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2809a5e0 0x7efe281884b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ddeb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2809a5e0 0x7efe281884b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57398/0 (socket says 172.21.15.32:57398) 2021-04-09T18:32:17.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ddeb700 1 -- 172.21.15.32:0/2527879299 learned_addr learned my addr 172.21.15.32:0/2527879299 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:17.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2e5ec700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280a15b0 0x7efe280a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:17.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2d5ea700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a3350 0x7efe280a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:17.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ddeb700 1 -- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280a15b0 msgr2=0x7efe280a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:17.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ddeb700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280a15b0 0x7efe280a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ddeb700 1 -- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a3350 msgr2=0x7efe280a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:17.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.698 7efe2ddeb700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a3350 0x7efe280a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:17.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.699 7efe2ddeb700 1 -- 172.21.15.32:0/2527879299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe280a3d70 con 0x7efe2809a5e0 2021-04-09T18:32:17.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.699 7efe2ddeb700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2809a5e0 0x7efe281884b0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 rx=0x7efe2400b930 tx=0x7efe2400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:17.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.699 7efe1e7fc700 1 -- 172.21.15.32:0/2527879299 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe24006030 con 0x7efe2809a5e0 2021-04-09T18:32:17.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.699 7efe1e7fc700 1 -- 172.21.15.32:0/2527879299 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe240063b0 con 0x7efe2809a5e0 2021-04-09T18:32:17.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.699 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe280a3d70 con 0x7efe2809a5e0 2021-04-09T18:32:17.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.699 7efe1e7fc700 1 -- 172.21.15.32:0/2527879299 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe24005660 con 0x7efe2809a5e0 2021-04-09T18:32:17.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.699 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe280a3d70 con 0x7efe2809a5e0 2021-04-09T18:32:17.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.700 7efe1e7fc700 1 -- 172.21.15.32:0/2527879299 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efe24005660 con 0x7efe2809a5e0 2021-04-09T18:32:17.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.700 7efe1e7fc700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe10018900 0x7efe1001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:17.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.700 7efe1e7fc700 1 -- 172.21.15.32:0/2527879299 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7efe24033180 con 0x7efe2809a5e0 2021-04-09T18:32:17.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.700 7efe2d5ea700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe10018900 0x7efe1001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:17.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.700 7efe2d5ea700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe10018900 0x7efe1001adc0 crc :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:17.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.720 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2810d480 con 0x7efe2809a5e0 2021-04-09T18:32:17.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.723 7efe1e7fc700 1 -- 172.21.15.32:0/2527879299 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efe240063b0 con 0x7efe2809a5e0 2021-04-09T18:32:17.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.910 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7efe2810d480 con 0x7efe2809a5e0 2021-04-09T18:32:17.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.912 7efe1e7fc700 1 -- 172.21.15.32:0/2527879299 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v211) v1 ==== 81+0+35618 (secure 0 0 0) 0x7efe240063b0 con 0x7efe2809a5e0 2021-04-09T18:32:17.930 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:17.931 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:32:17.931 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 211, 2021-04-09T18:32:17.931 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:32:17.931 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:32:17.932 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:31:38.197386", 2021-04-09T18:32:17.932 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:32:17.932 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:32:17.932 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:32:17.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:32:17.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:32:17.933 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:32:17.934 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:32:17.934 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:32:17.934 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:32:17.935 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:32:17.935 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:32:17.936 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:32:17.936 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:32:17.936 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:32:17.937 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:32:17.937 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 37, 2021-04-09T18:32:17.938 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:32:17.938 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:32:17.939 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:32:17.940 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:32:17.940 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:32:17.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:17.941 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:32:17.941 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:32:17.941 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:32:17.942 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:32:17.942 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:32:17.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:32:17.943 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:32:17.943 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:32:17.944 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:32:17.944 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:32:17.945 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:32:17.945 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:32:17.945 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:32:17.946 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:32:17.946 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:32:17.947 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:32:17.947 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:32:17.947 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:32:17.948 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:32:17.948 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:32:17.949 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:32:17.949 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:32:17.950 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:32:17.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:17.950 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:32:17.951 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:32:17.951 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:32:17.952 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:32:17.952 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:32:17.952 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:32:17.953 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:32:17.953 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:32:17.954 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:32:17.954 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:32:17.955 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:32:17.955 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:32:17.955 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:32:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:32:17.956 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:32:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:32:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:32:17.957 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:32:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:32:17.958 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:32:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:32:17.959 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:32:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:32:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:32:17.960 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:32:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:32:17.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:32:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:17.962 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:32:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:32:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:32:17.963 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:32:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:32:17.964 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:32:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:32:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:32:17.965 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:32:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:32:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:32:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:32:17.966 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:32:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:32:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:17.967 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 34, 2021-04-09T18:32:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "alpha_metadata", 2021-04-09T18:32:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:31:21.254184", 2021-04-09T18:32:17.968 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:32:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:32:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:32:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:32:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:32:17.969 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:32:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:32:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:32:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:32:17.970 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:32:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:32:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:32:17.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:32:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:32:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:32:17.972 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:32:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:32:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:32:17.973 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:32:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:32:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:17.974 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "202", 2021-04-09T18:32:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:32:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:32:17.975 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:32:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:32:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:32:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:32:17.976 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:32:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:32:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:32:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:32:17.977 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:32:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:32:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:32:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:32:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:32:17.978 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:32:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:32:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:32:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:32:17.979 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:32:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:32:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:32:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:32:17.980 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:32:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:32:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:32:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:32:17.981 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:32:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:32:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:32:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:32:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:32:17.982 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:32:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:32:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:32:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:32:17.983 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:32:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:32:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:32:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:32:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:32:17.984 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:32:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:32:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "alpha" 2021-04-09T18:32:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:17.985 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 35, 2021-04-09T18:32:17.986 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "alpha_data", 2021-04-09T18:32:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:31:22.120246", 2021-04-09T18:32:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:32:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:32:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:32:17.987 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:32:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:32:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:32:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:32:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:32:17.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:32:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:32:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:32:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:32:17.989 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:32:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:32:17.990 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:32:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:32:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:32:17.991 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:32:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:32:17.992 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:32:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "203", 2021-04-09T18:32:17.993 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:32:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:32:17.994 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:32:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:32:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:32:17.995 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:32:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:32:17.996 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:32:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:32:17.997 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:32:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:32:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:32:17.998 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:32:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:32:17.999 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:32:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:32:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:32:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:32:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:32:18.000 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:32:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:32:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:32:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:32:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:32:18.001 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:32:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:32:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:32:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:32:18.002 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:32:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:32:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:32:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:32:18.003 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:32:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:32:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:32:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:32:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:32:18.004 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:32:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:32:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:32:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "alpha" 2021-04-09T18:32:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.005 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 36, 2021-04-09T18:32:18.006 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "bravo_metadata", 2021-04-09T18:32:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:31:28.904296", 2021-04-09T18:32:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:32:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:32:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:32:18.007 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:32:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:32:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:32:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:32:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:32:18.008 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:32:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:32:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:32:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:32:18.009 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:32:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:32:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:32:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:32:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:32:18.010 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:32:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:32:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:32:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "206", 2021-04-09T18:32:18.011 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:32:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:32:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:32:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:32:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:32:18.012 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:32:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:32:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:32:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:32:18.013 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:32:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:32:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:32:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:32:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:32:18.014 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:32:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:32:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:32:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:32:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:32:18.015 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:32:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:32:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:32:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:32:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:32:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:32:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:32:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:32:18.017 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:32:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:32:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:32:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:32:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:32:18.018 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:32:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:32:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:32:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:32:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:32:18.019 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:32:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:32:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:32:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:32:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.020 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:32:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:32:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "bravo" 2021-04-09T18:32:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 37, 2021-04-09T18:32:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "bravo_data", 2021-04-09T18:32:18.022 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:31:30.123936", 2021-04-09T18:32:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:32:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:32:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:32:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:32:18.023 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:32:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:32:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:32:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:32:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:32:18.024 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:32:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:32:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:32:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:32:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:32:18.025 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:32:18.026 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:32:18.026 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:32:18.026 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:32:18.026 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:32:18.026 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:32:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "207", 2021-04-09T18:32:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:32:18.027 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:32:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:32:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:32:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:32:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:32:18.028 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:32:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:32:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:32:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:32:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:32:18.029 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:32:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:32:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:32:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:32:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:32:18.030 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:32:18.031 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:32:18.031 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:32:18.031 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:32:18.031 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:32:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:32:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:32:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:32:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:32:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:32:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:32:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:32:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:32:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:32:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:32:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:32:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:32:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:32:18.034 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:32:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:32:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:32:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:32:18.035 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:32:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:32:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "bravo" 2021-04-09T18:32:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:32:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:32:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:32:18.037 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:32:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:18.038 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:18.039 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:32:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.041 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:32:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.043 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:32:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.044 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:32:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.046 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:32:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.047 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:32:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.048 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:32:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:32:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:18.051 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:32:18.052 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:32:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:32:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:32:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:18.053 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.054 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:32:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:32:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:18.055 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:18.056 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:18.057 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.058 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:32:18.058 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:18.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.059 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.059 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.059 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:32:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.061 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:32:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:32:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.063 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:32:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:32:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:18.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:32:18.068 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.069 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:32:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.070 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:32:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:32:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:32:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:32:18.071 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:32:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:32:18.073 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:18.074 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:18.075 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.076 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:32:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:32:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:18.078 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:18.079 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:32:18.080 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:32:18.081 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.082 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.083 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:32:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.084 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:32:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.087 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:32:18.087 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:18.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.088 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.088 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:32:18.088 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:18.088 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.089 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.089 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:32:18.089 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:32:18.090 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:32:18.090 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:32:18.090 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:18.090 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:18.090 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:18.091 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.091 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:32:18.092 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:32:18.092 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:18.092 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:18.092 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:18.093 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:18.093 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:18.093 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:18.093 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:18.094 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:18.094 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:18.094 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:18.094 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:18.094 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.095 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.095 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:32:18.095 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:18.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.096 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.096 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:32:18.097 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:18.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.097 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.098 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:18.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.098 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:32:18.099 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:18.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.100 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:32:18.100 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:18.100 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.100 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.101 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:18.101 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.102 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:32:18.102 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:18.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.103 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:32:18.103 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:18.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.104 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.104 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:18.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.105 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:32:18.105 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:18.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.106 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.106 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.106 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:32:18.107 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:18.107 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.107 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.108 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:32:18.108 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:32:18.108 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:32:18.108 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:32:18.109 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:18.109 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:18.109 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:18.109 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.110 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:32:18.110 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:32:18.110 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:18.111 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:18.111 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:18.111 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:18.111 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:18.112 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:18.112 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:18.112 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:18.112 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:18.113 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:18.113 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:18.113 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.115 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.115 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:32:18.115 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:18.115 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.116 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.116 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:18.116 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:32:18.117 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:18.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.118 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:32:18.118 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:18.119 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.119 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.119 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:18.120 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.120 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.120 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.120 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:32:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:32:18.122 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:18.122 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.122 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.123 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:18.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.123 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.124 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:32:18.124 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:18.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:32:18.125 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:18.125 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.126 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.126 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:32:18.126 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:32:18.126 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:32:18.127 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:32:18.127 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:18.127 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:18.127 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:18.128 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.128 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:32:18.129 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:32:18.129 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:18.129 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:18.129 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:18.131 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:18.131 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:18.131 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:18.131 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.132 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.132 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.132 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:32:18.132 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:32:18.133 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:18.134 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.134 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.134 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:18.135 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.135 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.135 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.135 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:32:18.136 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:18.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.136 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.137 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:32:18.137 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:18.137 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.137 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.138 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:18.138 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.138 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.139 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:32:18.139 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:18.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.139 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.140 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:32:18.140 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:18.140 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.141 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.141 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:18.141 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.142 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:32:18.142 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:18.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.143 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.143 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.143 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:32:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.144 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:32:18.145 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:32:18.145 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:32:18.145 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:32:18.145 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:32:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:32:18.147 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:18.148 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:18.149 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:18.150 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:18.150 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.151 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:32:18.151 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:18.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.151 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:32:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:18.152 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.153 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.153 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:18.153 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:32:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:18.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.155 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:32:18.155 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:18.156 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.156 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.156 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.156 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:18.157 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.157 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:32:18.158 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:18.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.158 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.158 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.159 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:32:18.159 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:18.159 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.159 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.160 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:18.160 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.160 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.161 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.161 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:32:18.161 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:18.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.162 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.162 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:32:18.162 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:18.163 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.163 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.163 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:32:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:32:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:32:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:32:18.164 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:32:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:32:18.166 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:18.167 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:18.168 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:18.168 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 204, 2021-04-09T18:32:18.168 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:18.168 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:18.169 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:18.169 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.169 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.170 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:32:18.170 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:18.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.170 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.171 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.171 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:32:18.171 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:18.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:32:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:18.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.174 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:32:18.174 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:18.175 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.175 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.175 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:18.175 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.176 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.176 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:32:18.176 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:18.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.177 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.177 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:32:18.178 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:18.178 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.178 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.179 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:18.179 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:18.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.179 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:32:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.180 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:32:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:32:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:32:18.182 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:32:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:32:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:18.183 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:32:18.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:32:18.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:32:18.185 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:18.185 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:18.186 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:18.186 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:18.186 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:18.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.187 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.187 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:32:18.187 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:18.187 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:18.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:32:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:18.189 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:32:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:18.191 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:18.192 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:18.192 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:18.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.192 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.193 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:32:18.193 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:18.193 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:18.193 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:18.194 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:18.194 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:18.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.195 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:32:18.195 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:18.195 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:18.195 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:18.195 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:18.196 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:18.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.196 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.196 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:32:18.197 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:18.197 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:18.197 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:18.197 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:18.198 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:18.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.198 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:18.198 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:32:18.199 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:18.199 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:18.199 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:18.199 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:32:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:32:18.200 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:32:18.201 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:32:18.201 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:32:18.201 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:32:18.201 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/1129605860": "2021-04-10 18:31:38.197351", 2021-04-09T18:32:18.202 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/1129605860": "2021-04-10 18:31:38.197351", 2021-04-09T18:32:18.202 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6837/4017196651": "2021-04-10 18:31:38.192391", 2021-04-09T18:32:18.202 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/4017196651": "2021-04-10 18:31:38.192391" 2021-04-09T18:32:18.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.203 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:32:18.203 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:32:18.203 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:32:18.203 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:32:18.204 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:32:18.204 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:32:18.204 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:32:18.204 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:32:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:32:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:32:18.205 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:32:18.206 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:32:18.206 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:32:18.206 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:32:18.208 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe10018900 msgr2=0x7efe1001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe10018900 0x7efe1001adc0 crc :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2809a5e0 msgr2=0x7efe281884b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2809a5e0 0x7efe281884b0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 rx=0x7efe2400b930 tx=0x7efe2400bcd0).stop 2021-04-09T18:32:18.209 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 shutdown_connections 2021-04-09T18:32:18.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe10018900 0x7efe1001adc0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe280a15b0 0x7efe280a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe280a3350 0x7efe280a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.210 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 --2- 172.21.15.32:0/2527879299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe2809a5e0 0x7efe281884b0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 >> 172.21.15.32:0/2527879299 conn(0x7efe281a0db0 msgr2=0x7efe281a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:18.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 shutdown_connections 2021-04-09T18:32:18.211 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:17.916 7efe2ffcf700 1 -- 172.21.15.32:0/2527879299 wait complete. 2021-04-09T18:32:18.212 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:18.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.346 7f546e981700 1 Processor -- start 2021-04-09T18:32:18.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.346 7f546e981700 1 -- start start 2021-04-09T18:32:18.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f546e981700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f546810d010 0x7f54680d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f546e981700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54680d1da0 0x7f54680d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f546e981700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546810d480 con 0x7f54680d82c0 2021-04-09T18:32:18.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f546e981700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54680d8d30 con 0x7f54680d1da0 2021-04-09T18:32:18.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f546e981700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54680d9220 con 0x7f546810d010 2021-04-09T18:32:18.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f545ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54680d1da0 0x7f54680d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f545ffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f54680d1da0 0x7f54680d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57406/0 (socket says 172.21.15.32:57406) 2021-04-09T18:32:18.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f545ffff700 1 -- 172.21.15.32:0/557869993 learned_addr learned my addr 172.21.15.32:0/557869993 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:18.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f5467fff700 1 --2- 172.21.15.32:0/557869993 >> v2:172.21.15.86:3300/0 conn(0x7f546810d010 0x7f54680d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f545ffff700 1 -- 172.21.15.32:0/557869993 >> v2:172.21.15.86:3300/0 conn(0x7f546810d010 msgr2=0x7f54680d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:18.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f545ffff700 1 --2- 172.21.15.32:0/557869993 >> v2:172.21.15.86:3300/0 conn(0x7f546810d010 0x7f54680d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f545ffff700 1 -- 172.21.15.32:0/557869993 >> v1:172.21.15.86:6789/0 conn(0x7f54680d82c0 legacy=0x7f546810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f545ffff700 1 -- 172.21.15.32:0/557869993 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54680d86f0 con 0x7f54680d1da0 2021-04-09T18:32:18.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.347 7f545ffff700 1 --2- 172.21.15.32:0/557869993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54680d1da0 0x7f54680d7ca0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 rx=0x7f5450006700 tx=0x7f5450009b80).ready entity=mon.0 client_cookie=9b8e3160ba615425 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:18.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f54677fe700 1 -- 172.21.15.32:0/557869993 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f545000c0c0 con 0x7f54680d1da0 2021-04-09T18:32:18.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f54677fe700 1 -- 172.21.15.32:0/557869993 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545000c440 con 0x7f54680d1da0 2021-04-09T18:32:18.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f54677fe700 1 -- 172.21.15.32:0/557869993 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f545000c920 con 0x7f54680d1da0 2021-04-09T18:32:18.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f546e981700 1 -- 172.21.15.32:0/557869993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54680d1da0 msgr2=0x7f54680d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f546e981700 1 --2- 172.21.15.32:0/557869993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54680d1da0 0x7f54680d7ca0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 rx=0x7f5450006700 tx=0x7f5450009b80).stop 2021-04-09T18:32:18.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f546e981700 1 -- 172.21.15.32:0/557869993 shutdown_connections 2021-04-09T18:32:18.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f546e981700 1 --2- 172.21.15.32:0/557869993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54680d1da0 0x7f54680d7ca0 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f546e981700 1 --2- 172.21.15.32:0/557869993 >> v2:172.21.15.86:3300/0 conn(0x7f546810d010 0x7f54680d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f546e981700 1 -- 172.21.15.32:0/557869993 >> 172.21.15.32:0/557869993 conn(0x7f54680ce5f0 msgr2=0x7f54680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:18.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.348 7f546e981700 1 -- 172.21.15.32:0/557869993 shutdown_connections 2021-04-09T18:32:18.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.349 7f546e981700 1 -- 172.21.15.32:0/557869993 wait complete. 2021-04-09T18:32:18.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f546e981700 1 Processor -- start 2021-04-09T18:32:18.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f546e981700 1 -- start start 2021-04-09T18:32:18.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f546e981700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f546818c5c0 0x7f54681a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f546e981700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5468191880 0x7f546818cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f546e981700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5468191c80 0x7f546818d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f546e981700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546818c040 con 0x7f546818c5c0 2021-04-09T18:32:18.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f546e981700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54681aa5c0 con 0x7f5468191880 2021-04-09T18:32:18.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f546e981700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546818bd20 con 0x7f5468191c80 2021-04-09T18:32:18.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f5467fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f546818c5c0 0x7f54681a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f5467fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f546818c5c0 0x7f54681a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57410/0 (socket says 172.21.15.32:57410) 2021-04-09T18:32:18.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f5467fff700 1 -- 172.21.15.32:0/2202629582 learned_addr learned my addr 172.21.15.32:0/2202629582 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:18.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f545ffff700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5468191880 0x7f546818cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.369 7f546cf9e700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5468191c80 0x7f546818d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f5467fff700 1 -- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5468191c80 msgr2=0x7f546818d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f5467fff700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5468191c80 0x7f546818d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f5467fff700 1 -- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5468191880 msgr2=0x7f546818cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f5467fff700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5468191880 0x7f546818cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f5467fff700 1 -- 172.21.15.32:0/2202629582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546818da30 con 0x7f546818c5c0 2021-04-09T18:32:18.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f5467fff700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f546818c5c0 0x7f54681a9fa0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 rx=0x7f545800b560 tx=0x7f545800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:18.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f54657fa700 1 -- 172.21.15.32:0/2202629582 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f545800bcf0 con 0x7f546818c5c0 2021-04-09T18:32:18.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f54657fa700 1 -- 172.21.15.32:0/2202629582 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5458000c90 con 0x7f546818c5c0 2021-04-09T18:32:18.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f546e981700 1 -- 172.21.15.32:0/2202629582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546818da30 con 0x7f546818c5c0 2021-04-09T18:32:18.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f54657fa700 1 -- 172.21.15.32:0/2202629582 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5458002220 con 0x7f546818c5c0 2021-04-09T18:32:18.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.370 7f546e981700 1 -- 172.21.15.32:0/2202629582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546818da30 con 0x7f546818c5c0 2021-04-09T18:32:18.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.371 7f54657fa700 1 -- 172.21.15.32:0/2202629582 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f54580052e0 con 0x7f546818c5c0 2021-04-09T18:32:18.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.371 7f54657fa700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f544c0188c0 0x7f544c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.371 7f54657fa700 1 -- 172.21.15.32:0/2202629582 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f5458032c30 con 0x7f546818c5c0 2021-04-09T18:32:18.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.371 7f545ffff700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f544c0188c0 0x7f544c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.371 7f545ffff700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f544c0188c0 0x7f544c01adc0 crc :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:18.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.391 7f546e981700 1 -- 172.21.15.32:0/2202629582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546818dfc0 con 0x7f546818c5c0 2021-04-09T18:32:18.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.394 7f54657fa700 1 -- 172.21.15.32:0/2202629582 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5458006b00 con 0x7f546818c5c0 2021-04-09T18:32:18.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.587 7f546e981700 1 -- 172.21.15.32:0/2202629582 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f546818dfc0 con 0x7f546818c5c0 2021-04-09T18:32:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.587 7f54657fa700 1 -- 172.21.15.32:0/2202629582 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 371 v371) v1 ==== 95+0+3754 (secure 0 0 0) 0x7f54580052e0 con 0x7f546818c5c0 2021-04-09T18:32:18.604 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 371 2021-04-09T18:32:18.605 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:18.605 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":371,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:32:14.790358","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":211,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":11697,"mds_1":59147,"mds_2":59246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11697":{"gid":11697,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6834/358744459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":358744459},{"type":"v1","addr":"172.21.15.32:6834","nonce":358744459}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59147":{"gid":59147,"name":"b","rank":1,"incarnation":363,"state":"up:active","state_seq":6,"addr":"172.21.15.86:6837/1650816476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1650816476},{"type":"v1","addr":"172.21.15.86:6837","nonce":1650816476}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59246":{"gid":59246,"name":"d","rank":2,"incarnation":370,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/2540922065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2540922065},{"type":"v1","addr":"172.21.15.86:6836","nonce":2540922065}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":368,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:32:09.536281","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":210,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11700},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_11700":{"gid":11700,"name":"c","rank":0,"incarnation":355,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6835/3816325339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3816325339},{"type":"v1","addr":"172.21.15.32:6835","nonce":3816325339}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:32:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 -- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f544c0188c0 msgr2=0x7f544c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f544c0188c0 0x7f544c01adc0 crc :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 -- 172.21.15.32:0/2202629582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f546818c5c0 msgr2=0x7f54681a9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f546818c5c0 0x7f54681a9fa0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 rx=0x7f545800b560 tx=0x7f545800b900).stop 2021-04-09T18:32:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 -- 172.21.15.32:0/2202629582 shutdown_connections 2021-04-09T18:32:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f544c0188c0 0x7f544c01adc0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5468191c80 0x7f546818d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5468191880 0x7f546818cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 --2- 172.21.15.32:0/2202629582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f546818c5c0 0x7f54681a9fa0 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 -- 172.21.15.32:0/2202629582 >> 172.21.15.32:0/2202629582 conn(0x7f54681837d0 msgr2=0x7f5468195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:18.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 -- 172.21.15.32:0/2202629582 shutdown_connections 2021-04-09T18:32:18.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.590 7f546e981700 1 -- 172.21.15.32:0/2202629582 wait complete. 2021-04-09T18:32:18.617 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail alpha 2021-04-09T18:32:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.754 7f951eb23700 1 Processor -- start 2021-04-09T18:32:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.754 7f951eb23700 1 -- start start 2021-04-09T18:32:18.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951eb23700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f951810e7d0 0x7f95180d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951eb23700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95180d7280 con 0x7f95180cffa0 2021-04-09T18:32:18.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951eb23700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95180d2440 con 0x7f95180d6c80 2021-04-09T18:32:18.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951eb23700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95180d7080 con 0x7f951810e7d0 2021-04-09T18:32:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951d140700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f951810e7d0 0x7f95180d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951c93f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f95180cffa0 0x7f95180d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38776/0 (socket says 172.21.15.32:38776) 2021-04-09T18:32:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951c93f700 1 -- 172.21.15.32:0/420936782 learned_addr learned my addr 172.21.15.32:0/420936782 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951d140700 1 -- 172.21.15.32:0/420936782 >> v1:172.21.15.86:6789/0 conn(0x7f95180d6c80 legacy=0x7f951810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951d140700 1 -- 172.21.15.32:0/420936782 >> v1:172.21.15.32:6789/0 conn(0x7f95180cffa0 legacy=0x7f95180d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f95177fe700 1 -- 172.21.15.32:0/420936782 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692801010 0 0) 0x7f9508000fe0 con 0x7f95180d6c80 2021-04-09T18:32:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951d140700 1 -- 172.21.15.32:0/420936782 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95180d91e0 con 0x7f951810e7d0 2021-04-09T18:32:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.755 7f951d140700 1 --2- 172.21.15.32:0/420936782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951810e7d0 0x7f95180d7ff0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 rx=0x7f9510006ab0 tx=0x7f9510009b40).ready entity=mon.0 client_cookie=a9cdcd83e5aaaf86 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f95177fe700 1 -- 172.21.15.32:0/420936782 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95100022e0 con 0x7f951810e7d0 2021-04-09T18:32:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f95177fe700 1 -- 172.21.15.32:0/420936782 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9510002660 con 0x7f951810e7d0 2021-04-09T18:32:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f95177fe700 1 -- 172.21.15.32:0/420936782 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95100022e0 con 0x7f951810e7d0 2021-04-09T18:32:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f951eb23700 1 -- 172.21.15.32:0/420936782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951810e7d0 msgr2=0x7f95180d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f951eb23700 1 --2- 172.21.15.32:0/420936782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951810e7d0 0x7f95180d7ff0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 rx=0x7f9510006ab0 tx=0x7f9510009b40).stop 2021-04-09T18:32:18.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f951eb23700 1 -- 172.21.15.32:0/420936782 shutdown_connections 2021-04-09T18:32:18.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f951eb23700 1 --2- 172.21.15.32:0/420936782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951810e7d0 0x7f95180d7ff0 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f951eb23700 1 -- 172.21.15.32:0/420936782 >> 172.21.15.32:0/420936782 conn(0x7f95180ce5c0 msgr2=0x7f95180cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f951eb23700 1 -- 172.21.15.32:0/420936782 shutdown_connections 2021-04-09T18:32:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.756 7f951eb23700 1 -- 172.21.15.32:0/420936782 wait complete. 2021-04-09T18:32:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.776 7f951eb23700 1 Processor -- start 2021-04-09T18:32:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f951eb23700 1 -- start start 2021-04-09T18:32:18.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f951eb23700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95181a8b00 0x7f95181a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f951eb23700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95181a8f00 0x7f95181a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f951eb23700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f951817da40 0x7f951817de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f951eb23700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95181acff0 con 0x7f95181a8f00 2021-04-09T18:32:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f951eb23700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95181a46c0 con 0x7f95181a8b00 2021-04-09T18:32:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f951eb23700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95181a33f0 con 0x7f951817da40 2021-04-09T18:32:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f9517fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95181a8f00 0x7f95181a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f9517fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95181a8f00 0x7f95181a3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57424/0 (socket says 172.21.15.32:57424) 2021-04-09T18:32:18.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f9517fff700 1 -- 172.21.15.32:0/495633307 learned_addr learned my addr 172.21.15.32:0/495633307 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:18.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f951c93f700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95181a8b00 0x7f95181a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.777 7f951d140700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f951817da40 0x7f951817de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f9517fff700 1 -- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f951817da40 msgr2=0x7f951817de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f9517fff700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f951817da40 0x7f951817de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f9517fff700 1 -- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95181a8b00 msgr2=0x7f95181a3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f9517fff700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95181a8b00 0x7f95181a3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f9517fff700 1 -- 172.21.15.32:0/495633307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951817e490 con 0x7f95181a8f00 2021-04-09T18:32:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f9517fff700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95181a8f00 0x7f95181a3f20 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 rx=0x7f950800b8c0 tx=0x7f950800bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f95157fa700 1 -- 172.21.15.32:0/495633307 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9508003030 con 0x7f95181a8f00 2021-04-09T18:32:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f95157fa700 1 -- 172.21.15.32:0/495633307 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9508001400 con 0x7f95181a8f00 2021-04-09T18:32:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f951eb23700 1 -- 172.21.15.32:0/495633307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951817e490 con 0x7f95181a8f00 2021-04-09T18:32:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f95157fa700 1 -- 172.21.15.32:0/495633307 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9508001850 con 0x7f95181a8f00 2021-04-09T18:32:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.778 7f951eb23700 1 -- 172.21.15.32:0/495633307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951817e490 con 0x7f95181a8f00 2021-04-09T18:32:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.779 7f95157fa700 1 -- 172.21.15.32:0/495633307 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9508001850 con 0x7f95181a8f00 2021-04-09T18:32:18.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.779 7f95157fa700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94fc0188d0 0x7f94fc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:18.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.779 7f95157fa700 1 -- 172.21.15.32:0/495633307 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f9508033140 con 0x7f95181a8f00 2021-04-09T18:32:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.779 7f951c93f700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94fc0188d0 0x7f94fc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.779 7f951c93f700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94fc0188d0 0x7f94fc01add0 crc :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:18.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.799 7f951eb23700 1 -- 172.21.15.32:0/495633307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951817cb30 con 0x7f95181a8f00 2021-04-09T18:32:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.802 7f95157fa700 1 -- 172.21.15.32:0/495633307 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9508001400 con 0x7f95181a8f00 2021-04-09T18:32:19.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:18.984 7f951eb23700 1 -- 172.21.15.32:0/495633307 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "alpha"} v 0) v1 -- 0x7f95180d7280 con 0x7f95181a8f00 2021-04-09T18:32:19.618 INFO:tasks.ceph.mds.d.smithi086.stderr:2021-04-09 18:32:19.609 7f56d450e700 -1 MDSIOContextBase: blacklisted! Restarting... 2021-04-09T18:32:19.619 INFO:tasks.ceph.mds.d.smithi086.stderr: -1> 2021-04-09 18:32:19.609 7f56d450e700 -1 MDSIOContextBase: blacklisted! Restarting... 2021-04-09T18:32:19.672 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:32:19.819 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:32:19.803 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:32:19.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.811 7f95157fa700 1 -- 172.21.15.32:0/495633307 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "alpha"}]=0 alpha marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v372) v1 ==== 159+0+0 (secure 0 0 0) 0x7f9508001610 con 0x7f95181a8f00 2021-04-09T18:32:19.865 INFO:teuthology.orchestra.run.smithi032.stderr:alpha marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:32:19.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.851 7f951eb23700 1 -- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94fc0188d0 msgr2=0x7f94fc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:19.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.851 7f951eb23700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94fc0188d0 0x7f94fc01add0 crc :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:19.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.851 7f951eb23700 1 -- 172.21.15.32:0/495633307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95181a8f00 msgr2=0x7f95181a3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.851 7f951eb23700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95181a8f00 0x7f95181a3f20 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 rx=0x7f950800b8c0 tx=0x7f950800bc60).stop 2021-04-09T18:32:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.852 7f951eb23700 1 -- 172.21.15.32:0/495633307 shutdown_connections 2021-04-09T18:32:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.852 7f951eb23700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94fc0188d0 0x7f94fc01add0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.852 7f951eb23700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95181a8f00 0x7f95181a3f20 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.852 7f951eb23700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f951817da40 0x7f951817de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.852 7f951eb23700 1 --2- 172.21.15.32:0/495633307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95181a8b00 0x7f95181a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.852 7f951eb23700 1 -- 172.21.15.32:0/495633307 >> 172.21.15.32:0/495633307 conn(0x7f95181a5680 msgr2=0x7f9518181d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.852 7f951eb23700 1 -- 172.21.15.32:0/495633307 shutdown_connections 2021-04-09T18:32:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:19.852 7f951eb23700 1 -- 172.21.15.32:0/495633307 wait complete. 2021-04-09T18:32:19.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail bravo 2021-04-09T18:32:19.882 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:32:19.886 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:32:20.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.014 7f5660941700 1 Processor -- start 2021-04-09T18:32:20.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.014 7f5660941700 1 -- start start 2021-04-09T18:32:20.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.015 7f5660941700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f565c0cffa0 0x7f565c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:20.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.015 7f5660941700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f565c0d6c80 0x7f565c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:20.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.015 7f5660941700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c0d7280 con 0x7f565c0d7800 2021-04-09T18:32:20.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.015 7f5660941700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c0d7080 con 0x7f565c0cffa0 2021-04-09T18:32:20.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.015 7f5660941700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c0d91e0 con 0x7f565c0d6c80 2021-04-09T18:32:20.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.015 7f565a61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f565c0cffa0 0x7f565c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:20.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.015 7f565a61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f565c0cffa0 0x7f565c0d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57440/0 (socket says 172.21.15.32:57440) 2021-04-09T18:32:20.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.015 7f565a61c700 1 -- 172.21.15.32:0/3655216770 learned_addr learned my addr 172.21.15.32:0/3655216770 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:20.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.015 7f565ae1d700 1 --2- 172.21.15.32:0/3655216770 >> v2:172.21.15.86:3300/0 conn(0x7f565c0d6c80 0x7f565c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:20.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f565a61c700 1 -- 172.21.15.32:0/3655216770 >> v2:172.21.15.86:3300/0 conn(0x7f565c0d6c80 msgr2=0x7f565c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:20.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f565a61c700 1 --2- 172.21.15.32:0/3655216770 >> v2:172.21.15.86:3300/0 conn(0x7f565c0d6c80 0x7f565c10cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f565a61c700 1 -- 172.21.15.32:0/3655216770 >> v1:172.21.15.86:6789/0 conn(0x7f565c0d7800 legacy=0x7f565c0d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:20.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f565a61c700 1 -- 172.21.15.32:0/3655216770 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c0d83f0 con 0x7f565c0cffa0 2021-04-09T18:32:20.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f565a61c700 1 --2- 172.21.15.32:0/3655216770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0cffa0 0x7f565c0d2440 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 rx=0x7f564c006700 tx=0x7f564c009b00).ready entity=mon.0 client_cookie=bdf20a29432cfc86 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:20.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f565961a700 1 -- 172.21.15.32:0/3655216770 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f564c00c0c0 con 0x7f565c0cffa0 2021-04-09T18:32:20.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f565961a700 1 -- 172.21.15.32:0/3655216770 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564c00c400 con 0x7f565c0cffa0 2021-04-09T18:32:20.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f565961a700 1 -- 172.21.15.32:0/3655216770 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f564c00c8e0 con 0x7f565c0cffa0 2021-04-09T18:32:20.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f5660941700 1 -- 172.21.15.32:0/3655216770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0cffa0 msgr2=0x7f565c0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:20.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.016 7f5660941700 1 --2- 172.21.15.32:0/3655216770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0cffa0 0x7f565c0d2440 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 rx=0x7f564c006700 tx=0x7f564c009b00).stop 2021-04-09T18:32:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.017 7f5660941700 1 -- 172.21.15.32:0/3655216770 shutdown_connections 2021-04-09T18:32:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.017 7f5660941700 1 --2- 172.21.15.32:0/3655216770 >> v2:172.21.15.86:3300/0 conn(0x7f565c0d6c80 0x7f565c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.017 7f5660941700 1 --2- 172.21.15.32:0/3655216770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0cffa0 0x7f565c0d2440 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.017 7f5660941700 1 -- 172.21.15.32:0/3655216770 >> 172.21.15.32:0/3655216770 conn(0x7f565c0ce5c0 msgr2=0x7f565c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.017 7f5660941700 1 -- 172.21.15.32:0/3655216770 shutdown_connections 2021-04-09T18:32:20.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.017 7f5660941700 1 -- 172.21.15.32:0/3655216770 wait complete. 2021-04-09T18:32:20.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.037 7f5660941700 1 Processor -- start 2021-04-09T18:32:20.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.037 7f5660941700 1 -- start start 2021-04-09T18:32:20.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.037 7f5660941700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f565c08d210 0x7f565c08fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:20.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.037 7f5660941700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0903f0 0x7f565c1a5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:20.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5660941700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f565c1a6070 0x7f565c1a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:20.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5660941700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c08d690 con 0x7f565c0903f0 2021-04-09T18:32:20.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5660941700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0b0fd0 con 0x7f565c1a6070 2021-04-09T18:32:20.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5660941700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c1a6470 con 0x7f565c08d210 2021-04-09T18:32:20.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5659e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0903f0 0x7f565c1a5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:20.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5659e1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0903f0 0x7f565c1a5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57448/0 (socket says 172.21.15.32:57448) 2021-04-09T18:32:20.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5659e1b700 1 -- 172.21.15.32:0/367770625 learned_addr learned my addr 172.21.15.32:0/367770625 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:20.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f565a61c700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f565c08d210 0x7f565c08fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:20.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f565ae1d700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f565c1a6070 0x7f565c1a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:20.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5659e1b700 1 -- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f565c08d210 msgr2=0x7f565c08fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:20.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5659e1b700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f565c08d210 0x7f565c08fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5659e1b700 1 -- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f565c1a6070 msgr2=0x7f565c1a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:20.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5659e1b700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f565c1a6070 0x7f565c1a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5659e1b700 1 -- 172.21.15.32:0/367770625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c08d8b0 con 0x7f565c0903f0 2021-04-09T18:32:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5659e1b700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0903f0 0x7f565c1a5a50 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 rx=0x7f565000b8e0 tx=0x7f565000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f564affd700 1 -- 172.21.15.32:0/367770625 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f565000c2f0 con 0x7f565c0903f0 2021-04-09T18:32:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f564affd700 1 -- 172.21.15.32:0/367770625 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565000c570 con 0x7f565c0903f0 2021-04-09T18:32:20.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5660941700 1 -- 172.21.15.32:0/367770625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c08d8b0 con 0x7f565c0903f0 2021-04-09T18:32:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f564affd700 1 -- 172.21.15.32:0/367770625 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f565000c9d0 con 0x7f565c0903f0 2021-04-09T18:32:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.038 7f5660941700 1 -- 172.21.15.32:0/367770625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c08d8b0 con 0x7f565c0903f0 2021-04-09T18:32:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.039 7f564affd700 1 -- 172.21.15.32:0/367770625 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5650003080 con 0x7f565c0903f0 2021-04-09T18:32:20.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.039 7f564affd700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5640018930 0x7f564001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:20.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.039 7f564affd700 1 -- 172.21.15.32:0/367770625 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f56500332c0 con 0x7f565c0903f0 2021-04-09T18:32:20.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.039 7f565a61c700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5640018930 0x7f564001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:20.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.039 7f565a61c700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5640018930 0x7f564001adc0 crc :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:20.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.059 7f5660941700 1 -- 172.21.15.32:0/367770625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c1a52a0 con 0x7f565c0903f0 2021-04-09T18:32:20.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.063 7f564affd700 1 -- 172.21.15.32:0/367770625 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f565000bdc0 con 0x7f565c0903f0 2021-04-09T18:32:20.260 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.244 7f5660941700 1 -- 172.21.15.32:0/367770625 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "bravo"} v 0) v1 -- 0x7f565c1a52a0 con 0x7f565c0903f0 2021-04-09T18:32:20.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.812 7f564affd700 1 -- 172.21.15.32:0/367770625 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "bravo"}]=0 bravo marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v373) v1 ==== 159+0+0 (secure 0 0 0) 0x7f565001fb20 con 0x7f565c0903f0 2021-04-09T18:32:20.875 INFO:teuthology.orchestra.run.smithi032.stderr:bravo marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:32:20.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 -- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5640018930 msgr2=0x7f564001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5640018930 0x7f564001adc0 crc :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 -- 172.21.15.32:0/367770625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0903f0 msgr2=0x7f565c1a5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0903f0 0x7f565c1a5a50 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 rx=0x7f565000b8e0 tx=0x7f565000bc80).stop 2021-04-09T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 -- 172.21.15.32:0/367770625 shutdown_connections 2021-04-09T18:32:20.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5640018930 0x7f564001adc0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f565c0903f0 0x7f565c1a5a50 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f565c1a6070 0x7f565c1a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 --2- 172.21.15.32:0/367770625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f565c08d210 0x7f565c08fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 -- 172.21.15.32:0/367770625 >> 172.21.15.32:0/367770625 conn(0x7f565c1a8f40 msgr2=0x7f565c188050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:20.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 -- 172.21.15.32:0/367770625 shutdown_connections 2021-04-09T18:32:20.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:20.862 7f5660941700 1 -- 172.21.15.32:0/367770625 wait complete. 2021-04-09T18:32:20.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:20.894 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.026 7f08a6d1b700 1 Processor -- start 2021-04-09T18:32:21.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.026 7f08a6d1b700 1 -- start start 2021-04-09T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.026 7f08a6d1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08a010d010 0x7f08a00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a6d1b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a010d480 con 0x7f08a00d8490 2021-04-09T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a6d1b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a010e1f0 con 0x7f08a00d1da0 2021-04-09T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a6d1b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a00d8d30 con 0x7f08a010d010 2021-04-09T18:32:21.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a4b37700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f08a010d010 0x7f08a00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a5338700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f08a00d8490 0x7f08a010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38826/0 (socket says 172.21.15.32:38826) 2021-04-09T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a5338700 1 -- 172.21.15.32:0/153860312 learned_addr learned my addr 172.21.15.32:0/153860312 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a4b37700 1 -- 172.21.15.32:0/153860312 >> v1:172.21.15.86:6789/0 conn(0x7f08a00d1da0 legacy=0x7f08a00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a4b37700 1 -- 172.21.15.32:0/153860312 >> v1:172.21.15.32:6789/0 conn(0x7f08a00d8490 legacy=0x7f08a010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f089f7fe700 1 -- 172.21.15.32:0/153860312 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285010806 0 0) 0x7f0898000d30 con 0x7f08a00d8490 2021-04-09T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a4b37700 1 -- 172.21.15.32:0/153860312 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a00d9220 con 0x7f08a010d010 2021-04-09T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a4b37700 1 --2- 172.21.15.32:0/153860312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a010d010 0x7f08a00d1780 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7f0890006650 tx=0x7f0890009a00).ready entity=mon.0 client_cookie=14a66b0b20e1e903 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:21.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f089f7fe700 1 -- 172.21.15.32:0/153860312 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0890009df0 con 0x7f08a010d010 2021-04-09T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f089f7fe700 1 -- 172.21.15.32:0/153860312 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08900009b0 con 0x7f08a010d010 2021-04-09T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f089f7fe700 1 -- 172.21.15.32:0/153860312 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f089000c2b0 con 0x7f08a010d010 2021-04-09T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a6d1b700 1 -- 172.21.15.32:0/153860312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a010d010 msgr2=0x7f08a00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.027 7f08a6d1b700 1 --2- 172.21.15.32:0/153860312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a010d010 0x7f08a00d1780 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7f0890006650 tx=0x7f0890009a00).stop 2021-04-09T18:32:21.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.028 7f08a6d1b700 1 -- 172.21.15.32:0/153860312 shutdown_connections 2021-04-09T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.028 7f08a6d1b700 1 --2- 172.21.15.32:0/153860312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a010d010 0x7f08a00d1780 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.028 7f08a6d1b700 1 -- 172.21.15.32:0/153860312 >> 172.21.15.32:0/153860312 conn(0x7f08a00ce5f0 msgr2=0x7f08a00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.028 7f08a6d1b700 1 -- 172.21.15.32:0/153860312 shutdown_connections 2021-04-09T18:32:21.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.028 7f08a6d1b700 1 -- 172.21.15.32:0/153860312 wait complete. 2021-04-09T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.048 7f08a6d1b700 1 Processor -- start 2021-04-09T18:32:21.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.048 7f08a6d1b700 1 -- start start 2021-04-09T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a6d1b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08a00c10d0 0x7f08a01bffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a6d1b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08a01c05d0 0x7f08a00bdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a6d1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00be590 0x7f08a01a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a6d1b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a01abd30 con 0x7f08a00be590 2021-04-09T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a6d1b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a00bd9f0 con 0x7f08a00c10d0 2021-04-09T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a6d1b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a00c14d0 con 0x7f08a01c05d0 2021-04-09T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a5338700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00be590 0x7f08a01a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a5338700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00be590 0x7f08a01a5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57476/0 (socket says 172.21.15.32:57476) 2021-04-09T18:32:21.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a5338700 1 -- 172.21.15.32:0/2384947441 learned_addr learned my addr 172.21.15.32:0/2384947441 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a4b37700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08a00c10d0 0x7f08a01bffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f089ffff700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08a01c05d0 0x7f08a00bdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a5338700 1 -- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08a01c05d0 msgr2=0x7f08a00bdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a5338700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08a01c05d0 0x7f08a00bdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a5338700 1 -- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08a00c10d0 msgr2=0x7f08a01bffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a5338700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08a00c10d0 0x7f08a01bffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a5338700 1 -- 172.21.15.32:0/2384947441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a00be990 con 0x7f08a00be590 2021-04-09T18:32:21.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.049 7f08a4b37700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08a00c10d0 0x7f08a01bffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.050 7f08a5338700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00be590 0x7f08a01a5680 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 rx=0x7f0898007f90 tx=0x7f089800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.050 7f089d7fa700 1 -- 172.21.15.32:0/2384947441 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0898000cc0 con 0x7f08a00be590 2021-04-09T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.050 7f089d7fa700 1 -- 172.21.15.32:0/2384947441 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0898007020 con 0x7f08a00be590 2021-04-09T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.050 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a00be990 con 0x7f08a00be590 2021-04-09T18:32:21.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.050 7f089d7fa700 1 -- 172.21.15.32:0/2384947441 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0898007540 con 0x7f08a00be590 2021-04-09T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.050 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a00be990 con 0x7f08a00be590 2021-04-09T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.050 7f089d7fa700 1 -- 172.21.15.32:0/2384947441 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0898007200 con 0x7f08a00be590 2021-04-09T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.050 7f089d7fa700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0880018930 0x7f088001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.051 7f089d7fa700 1 -- 172.21.15.32:0/2384947441 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6905+0+0 (secure 0 0 0) 0x7f0898033270 con 0x7f08a00be590 2021-04-09T18:32:21.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.051 7f08a4b37700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0880018930 0x7f088001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.051 7f08a4b37700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0880018930 0x7f088001ae10 crc :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:21.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.071 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a01abd30 con 0x7f08a00be590 2021-04-09T18:32:21.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.074 7f089d7fa700 1 -- 172.21.15.32:0/2384947441 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0898007d00 con 0x7f08a00be590 2021-04-09T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.265 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08a01abd30 con 0x7f08a00be590 2021-04-09T18:32:21.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.266 7f089d7fa700 1 -- 172.21.15.32:0/2384947441 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 373 v373) v1 ==== 95+0+2355 (secure 0 0 0) 0x7f0898007020 con 0x7f08a00be590 2021-04-09T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 373 2021-04-09T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:21.283 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":373,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":372,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:23.513548","modified":"2021-04-09 18:32:19.804836","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{},"failed":[0,1,2],"damaged":[],"stopped":[],"info":{},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":17},{"mdsmap":{"epoch":373,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:31:31.113459","modified":"2021-04-09 18:32:20.811809","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[1],"info":{},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":18}]} 2021-04-09T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0880018930 msgr2=0x7f088001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0880018930 0x7f088001ae10 crc :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00be590 msgr2=0x7f08a01a5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00be590 0x7f08a01a5680 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 rx=0x7f0898007f90 tx=0x7f089800ac50).stop 2021-04-09T18:32:21.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 shutdown_connections 2021-04-09T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0880018930 0x7f088001ae10 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00be590 0x7f08a01a5680 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08a01c05d0 0x7f08a00bdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 --2- 172.21.15.32:0/2384947441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08a00c10d0 0x7f08a01bffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.268 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 >> 172.21.15.32:0/2384947441 conn(0x7f08a01acbe0 msgr2=0x7f08a008dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.269 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 shutdown_connections 2021-04-09T18:32:21.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.269 7f08a6d1b700 1 -- 172.21.15.32:0/2384947441 wait complete. 2021-04-09T18:32:21.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm alpha --yes-i-really-mean-it 2021-04-09T18:32:21.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.431 7ff78ff9c700 1 Processor -- start 2021-04-09T18:32:21.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.431 7ff78ff9c700 1 -- start start 2021-04-09T18:32:21.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78ff9c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff7880d6c80 0x7ff7880d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78ff9c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff7880d2440 0x7ff78810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78ff9c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7880d7280 con 0x7ff7880cffa0 2021-04-09T18:32:21.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78ff9c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7880d7080 con 0x7ff7880d2440 2021-04-09T18:32:21.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78ff9c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7880d91e0 con 0x7ff7880d6c80 2021-04-09T18:32:21.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78e5b9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff7880d2440 0x7ff78810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78ddb8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff7880cffa0 0x7ff7880d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38838/0 (socket says 172.21.15.32:38838) 2021-04-09T18:32:21.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78ddb8700 1 -- 172.21.15.32:0/3319439247 learned_addr learned my addr 172.21.15.32:0/3319439247 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:21.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78d5b7700 1 --2- 172.21.15.32:0/3319439247 >> v2:172.21.15.86:3300/0 conn(0x7ff7880d6c80 0x7ff7880d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78e5b9700 1 -- 172.21.15.32:0/3319439247 >> v2:172.21.15.86:3300/0 conn(0x7ff7880d6c80 msgr2=0x7ff7880d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:21.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78e5b9700 1 --2- 172.21.15.32:0/3319439247 >> v2:172.21.15.86:3300/0 conn(0x7ff7880d6c80 0x7ff7880d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78e5b9700 1 -- 172.21.15.32:0/3319439247 >> v1:172.21.15.32:6789/0 conn(0x7ff7880cffa0 legacy=0x7ff7880d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78cdb6700 1 -- 172.21.15.32:0/3319439247 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182779908 0 0) 0x7ff784000ce0 con 0x7ff7880cffa0 2021-04-09T18:32:21.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78e5b9700 1 -- 172.21.15.32:0/3319439247 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7880d2840 con 0x7ff7880d2440 2021-04-09T18:32:21.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.432 7ff78e5b9700 1 --2- 172.21.15.32:0/3319439247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7880d2440 0x7ff78810cfd0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 rx=0x7ff780006e30 tx=0x7ff780009de0).ready entity=mon.0 client_cookie=5a371b68c6f60a7d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:21.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.433 7ff78cdb6700 1 -- 172.21.15.32:0/3319439247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff78000b300 con 0x7ff7880d2440 2021-04-09T18:32:21.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.433 7ff78cdb6700 1 -- 172.21.15.32:0/3319439247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78000b680 con 0x7ff7880d2440 2021-04-09T18:32:21.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.433 7ff78cdb6700 1 -- 172.21.15.32:0/3319439247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff78000bb60 con 0x7ff7880d2440 2021-04-09T18:32:21.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.433 7ff78ff9c700 1 -- 172.21.15.32:0/3319439247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7880d2440 msgr2=0x7ff78810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.433 7ff78ff9c700 1 --2- 172.21.15.32:0/3319439247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7880d2440 0x7ff78810cfd0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 rx=0x7ff780006e30 tx=0x7ff780009de0).stop 2021-04-09T18:32:21.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.433 7ff78ff9c700 1 -- 172.21.15.32:0/3319439247 shutdown_connections 2021-04-09T18:32:21.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.433 7ff78ff9c700 1 --2- 172.21.15.32:0/3319439247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7880d2440 0x7ff78810cfd0 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.433 7ff78ff9c700 1 --2- 172.21.15.32:0/3319439247 >> v2:172.21.15.86:3300/0 conn(0x7ff7880d6c80 0x7ff7880d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.433 7ff78ff9c700 1 -- 172.21.15.32:0/3319439247 >> 172.21.15.32:0/3319439247 conn(0x7ff7880ce5c0 msgr2=0x7ff7880cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:21.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.434 7ff78ff9c700 1 -- 172.21.15.32:0/3319439247 shutdown_connections 2021-04-09T18:32:21.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.434 7ff78ff9c700 1 -- 172.21.15.32:0/3319439247 wait complete. 2021-04-09T18:32:21.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ff9c700 1 Processor -- start 2021-04-09T18:32:21.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ff9c700 1 -- start start 2021-04-09T18:32:21.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ff9c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff78808a620 0x7ff78808b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ff9c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff78808b6d0 0x7ff788180620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ff9c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff788180c40 0x7ff788181040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ff9c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78808e660 con 0x7ff78808a620 2021-04-09T18:32:21.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ff9c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78808ab30 con 0x7ff788180c40 2021-04-09T18:32:21.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ff9c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff78808bad0 con 0x7ff78808b6d0 2021-04-09T18:32:21.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ddb8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff78808a620 0x7ff78808b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ddb8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff78808a620 0x7ff78808b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57486/0 (socket says 172.21.15.32:57486) 2021-04-09T18:32:21.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78ddb8700 1 -- 172.21.15.32:0/2534503712 learned_addr learned my addr 172.21.15.32:0/2534503712 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:21.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.454 7ff78d5b7700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff78808b6d0 0x7ff788180620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78e5b9700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff788180c40 0x7ff788181040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78ddb8700 1 -- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff78808b6d0 msgr2=0x7ff788180620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78ddb8700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff78808b6d0 0x7ff788180620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78ddb8700 1 -- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff788180c40 msgr2=0x7ff788181040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:21.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78ddb8700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff788180c40 0x7ff788181040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:21.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78ddb8700 1 -- 172.21.15.32:0/2534503712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78808e1f0 con 0x7ff78808a620 2021-04-09T18:32:21.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78e5b9700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff788180c40 0x7ff788181040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:21.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78d5b7700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff78808b6d0 0x7ff788180620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:21.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78ddb8700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff78808a620 0x7ff78808b0b0 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 rx=0x7ff784006f00 tx=0x7ff78400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:21.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff77e7fc700 1 -- 172.21.15.32:0/2534503712 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff78400bd30 con 0x7ff78808a620 2021-04-09T18:32:21.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff77e7fc700 1 -- 172.21.15.32:0/2534503712 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7840060d0 con 0x7ff78808a620 2021-04-09T18:32:21.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78808e1f0 con 0x7ff78808a620 2021-04-09T18:32:21.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff77e7fc700 1 -- 172.21.15.32:0/2534503712 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff784004620 con 0x7ff78808a620 2021-04-09T18:32:21.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.455 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78808e1f0 con 0x7ff78808a620 2021-04-09T18:32:21.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.456 7ff77e7fc700 1 -- 172.21.15.32:0/2534503712 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff7840062b0 con 0x7ff78808a620 2021-04-09T18:32:21.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.456 7ff77e7fc700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff770018930 0x7ff77001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:21.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.456 7ff77e7fc700 1 -- 172.21.15.32:0/2534503712 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6905+0+0 (secure 0 0 0) 0x7ff784032f20 con 0x7ff78808a620 2021-04-09T18:32:21.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.456 7ff78d5b7700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff770018930 0x7ff77001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:21.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.456 7ff78d5b7700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff770018930 0x7ff77001ae10 crc :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:21.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.476 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7881b7170 con 0x7ff78808a620 2021-04-09T18:32:21.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.479 7ff77e7fc700 1 -- 172.21.15.32:0/2534503712 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff784006b90 con 0x7ff78808a620 2021-04-09T18:32:21.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:21.672 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "alpha", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff7881b7170 con 0x7ff78808a620 2021-04-09T18:32:22.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.249 7ff77e7fc700 1 -- 172.21.15.32:0/2534503712 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "alpha", "yes_i_really_mean_it": true}]=0 v374) v1 ==== 103+0+0 (secure 0 0 0) 0x7ff7840060d0 con 0x7ff78808a620 2021-04-09T18:32:22.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff770018930 msgr2=0x7ff77001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:22.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff770018930 0x7ff77001ae10 crc :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff78808a620 msgr2=0x7ff78808b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:22.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff78808a620 0x7ff78808b0b0 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 rx=0x7ff784006f00 tx=0x7ff78400b940).stop 2021-04-09T18:32:22.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 shutdown_connections 2021-04-09T18:32:22.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff770018930 0x7ff77001ae10 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff788180c40 0x7ff788181040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff78808b6d0 0x7ff788180620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 --2- 172.21.15.32:0/2534503712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff78808a620 0x7ff78808b0b0 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 >> 172.21.15.32:0/2534503712 conn(0x7ff7881897f0 msgr2=0x7ff7881b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:22.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 shutdown_connections 2021-04-09T18:32:22.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.289 7ff78ff9c700 1 -- 172.21.15.32:0/2534503712 wait complete. 2021-04-09T18:32:22.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete alpha_metadata alpha_metadata --yes-i-really-really-mean-it 2021-04-09T18:32:22.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.461 7f76c6c04700 1 Processor -- start 2021-04-09T18:32:22.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.461 7f76c6c04700 1 -- start start 2021-04-09T18:32:22.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c6c04700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76c010d050 0x7f76c00cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:22.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c6c04700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00d0160 0x7f76c00d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:22.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c6c04700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f76c00d82e0 0x7f76c010da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:22.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c6c04700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c00d8d70 con 0x7f76c010d050 2021-04-09T18:32:22.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c6c04700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c00d9260 con 0x7f76c00d82e0 2021-04-09T18:32:22.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c6c04700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c010d500 con 0x7f76c00d0160 2021-04-09T18:32:22.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c4a20700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76c010d050 0x7f76c00cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:22.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c4a20700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f76c010d050 0x7f76c00cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57494/0 (socket says 172.21.15.32:57494) 2021-04-09T18:32:22.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c4a20700 1 -- 172.21.15.32:0/3512229169 learned_addr learned my addr 172.21.15.32:0/3512229169 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:22.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76bffff700 1 --2- 172.21.15.32:0/3512229169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00d0160 0x7f76c00d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:22.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c5221700 1 --2- 172.21.15.32:0/3512229169 >> v2:172.21.15.86:3300/0 conn(0x7f76c00d82e0 0x7f76c010da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c4a20700 1 -- 172.21.15.32:0/3512229169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00d0160 msgr2=0x7f76c00d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c4a20700 1 --2- 172.21.15.32:0/3512229169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00d0160 0x7f76c00d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c4a20700 1 -- 172.21.15.32:0/3512229169 >> v2:172.21.15.86:3300/0 conn(0x7f76c00d82e0 msgr2=0x7f76c010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:22.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c4a20700 1 --2- 172.21.15.32:0/3512229169 >> v2:172.21.15.86:3300/0 conn(0x7f76c00d82e0 0x7f76c010da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c4a20700 1 -- 172.21.15.32:0/3512229169 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c00d86e0 con 0x7f76c010d050 2021-04-09T18:32:22.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.462 7f76c4a20700 1 --2- 172.21.15.32:0/3512229169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c010d050 0x7f76c00cfb40 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 rx=0x7f76a8006700 tx=0x7f76a8009b00).ready entity=mon.0 client_cookie=5db883ba02bfa87f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:22.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76bf7fe700 1 -- 172.21.15.32:0/3512229169 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76a800c0c0 con 0x7f76c010d050 2021-04-09T18:32:22.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76bf7fe700 1 -- 172.21.15.32:0/3512229169 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a800c400 con 0x7f76c010d050 2021-04-09T18:32:22.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76bf7fe700 1 -- 172.21.15.32:0/3512229169 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76a800c8e0 con 0x7f76c010d050 2021-04-09T18:32:22.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76c6c04700 1 -- 172.21.15.32:0/3512229169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c010d050 msgr2=0x7f76c00cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:22.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76c6c04700 1 --2- 172.21.15.32:0/3512229169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c010d050 0x7f76c00cfb40 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 rx=0x7f76a8006700 tx=0x7f76a8009b00).stop 2021-04-09T18:32:22.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76c6c04700 1 -- 172.21.15.32:0/3512229169 shutdown_connections 2021-04-09T18:32:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76c6c04700 1 --2- 172.21.15.32:0/3512229169 >> v2:172.21.15.86:3300/0 conn(0x7f76c00d82e0 0x7f76c010da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76c6c04700 1 --2- 172.21.15.32:0/3512229169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00d0160 0x7f76c00d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76c6c04700 1 --2- 172.21.15.32:0/3512229169 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c010d050 0x7f76c00cfb40 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.463 7f76c6c04700 1 -- 172.21.15.32:0/3512229169 >> 172.21.15.32:0/3512229169 conn(0x7f76c00ce620 msgr2=0x7f76c00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:22.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.464 7f76c6c04700 1 -- 172.21.15.32:0/3512229169 shutdown_connections 2021-04-09T18:32:22.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.464 7f76c6c04700 1 -- 172.21.15.32:0/3512229169 wait complete. 2021-04-09T18:32:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.484 7f76c6c04700 1 Processor -- start 2021-04-09T18:32:22.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.484 7f76c6c04700 1 -- start start 2021-04-09T18:32:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.484 7f76c6c04700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c00ad600 0x7f76c009ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:22.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.484 7f76c6c04700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76c009b1b0 0x7f76c00ba220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.484 7f76c6c04700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00ba840 0x7f76c00a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.484 7f76c6c04700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c009fa80 con 0x7f76c00ad600 2021-04-09T18:32:22.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c6c04700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c009b5b0 con 0x7f76c009b1b0 2021-04-09T18:32:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c6c04700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76c00bac40 con 0x7f76c00ba840 2021-04-09T18:32:22.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c4a20700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c00ad600 0x7f76c009ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c4a20700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c00ad600 0x7f76c009ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57498/0 (socket says 172.21.15.32:57498) 2021-04-09T18:32:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c4a20700 1 -- 172.21.15.32:0/2715344723 learned_addr learned my addr 172.21.15.32:0/2715344723 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:22.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76bffff700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76c009b1b0 0x7f76c00ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:22.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c5221700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00ba840 0x7f76c00a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:22.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c4a20700 1 -- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00ba840 msgr2=0x7f76c00a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:22.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c4a20700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00ba840 0x7f76c00a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c4a20700 1 -- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76c009b1b0 msgr2=0x7f76c00ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:22.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c4a20700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76c009b1b0 0x7f76c00ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:22.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c4a20700 1 -- 172.21.15.32:0/2715344723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c00a97c0 con 0x7f76c00ad600 2021-04-09T18:32:22.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c4a20700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c00ad600 0x7f76c009ab90 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 rx=0x7f76a8006f20 tx=0x7f76a800b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:22.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76bd7fa700 1 -- 172.21.15.32:0/2715344723 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76a80090d0 con 0x7f76c00ad600 2021-04-09T18:32:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76bd7fa700 1 -- 172.21.15.32:0/2715344723 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a800bd00 con 0x7f76c00ad600 2021-04-09T18:32:22.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c00a97c0 con 0x7f76c00ad600 2021-04-09T18:32:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76bd7fa700 1 -- 172.21.15.32:0/2715344723 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76a8005560 con 0x7f76c00ad600 2021-04-09T18:32:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.485 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c00a97c0 con 0x7f76c00ad600 2021-04-09T18:32:22.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.486 7f76bd7fa700 1 -- 172.21.15.32:0/2715344723 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f76a800a000 con 0x7f76c00ad600 2021-04-09T18:32:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.486 7f76bd7fa700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76a4018930 0x7f76a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.486 7f76bd7fa700 1 -- 172.21.15.32:0/2715344723 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6905+0+0 (secure 0 0 0) 0x7f76a8032f20 con 0x7f76c00ad600 2021-04-09T18:32:22.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.486 7f76bffff700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76a4018930 0x7f76a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:22.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.487 7f76bffff700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76a4018930 0x7f76a401ae10 crc :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.506 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c00aa330 con 0x7f76c00ad600 2021-04-09T18:32:22.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.509 7f76bd7fa700 1 -- 172.21.15.32:0/2715344723 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f76a80090d0 con 0x7f76c00ad600 2021-04-09T18:32:22.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:22.692 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "alpha_metadata", "prefix": "osd pool delete", "pool": "alpha_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f76c00aa330 con 0x7f76c00ad600 2021-04-09T18:32:23.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.253 7f76bd7fa700 1 -- 172.21.15.32:0/2715344723 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "alpha_metadata", "prefix": "osd pool delete", "pool": "alpha_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'alpha_metadata' removed v214) v1 ==== 182+0+0 (secure 0 0 0) 0x7f76a800a360 con 0x7f76c00ad600 2021-04-09T18:32:23.272 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_metadata' removed 2021-04-09T18:32:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76a4018930 msgr2=0x7f76a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:23.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76a4018930 0x7f76a401ae10 crc :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c00ad600 msgr2=0x7f76c009ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c00ad600 0x7f76c009ab90 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 rx=0x7f76a8006f20 tx=0x7f76a800b910).stop 2021-04-09T18:32:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 shutdown_connections 2021-04-09T18:32:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f76a4018930 0x7f76a401ae10 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76c00ba840 0x7f76c00a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76c009b1b0 0x7f76c00ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 --2- 172.21.15.32:0/2715344723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76c00ad600 0x7f76c009ab90 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 >> 172.21.15.32:0/2715344723 conn(0x7f76c00ae6f0 msgr2=0x7f76c0097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:23.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 shutdown_connections 2021-04-09T18:32:23.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.258 7f76c6c04700 1 -- 172.21.15.32:0/2715344723 wait complete. 2021-04-09T18:32:23.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete alpha_data alpha_data --yes-i-really-really-mean-it 2021-04-09T18:32:23.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.430 7f58baa45700 1 Processor -- start 2021-04-09T18:32:23.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.430 7f58baa45700 1 -- start start 2021-04-09T18:32:23.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58baa45700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58b410d020 0x7f58b40d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:23.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58baa45700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b40d84a0 0x7f58b410da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:23.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58baa45700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b410d490 con 0x7f58b40d1db0 2021-04-09T18:32:23.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58baa45700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b40d8d40 con 0x7f58b410d020 2021-04-09T18:32:23.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58baa45700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b40d9230 con 0x7f58b40d84a0 2021-04-09T18:32:23.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58b8861700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f58b410d020 0x7f58b40d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:23.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58b3fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f58b40d1db0 0x7f58b40d7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38864/0 (socket says 172.21.15.32:38864) 2021-04-09T18:32:23.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58b3fff700 1 -- 172.21.15.32:0/1428307744 learned_addr learned my addr 172.21.15.32:0/1428307744 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:23.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58b9062700 1 --2- 172.21.15.32:0/1428307744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b40d84a0 0x7f58b410da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:23.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58b8861700 1 -- 172.21.15.32:0/1428307744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b40d84a0 msgr2=0x7f58b410da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:23.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58b8861700 1 --2- 172.21.15.32:0/1428307744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b40d84a0 0x7f58b410da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58b8861700 1 -- 172.21.15.32:0/1428307744 >> v1:172.21.15.32:6789/0 conn(0x7f58b40d1db0 legacy=0x7f58b40d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:23.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58b8861700 1 -- 172.21.15.32:0/1428307744 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b410e030 con 0x7f58b410d020 2021-04-09T18:32:23.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.431 7f58b37fe700 1 -- 172.21.15.32:0/1428307744 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174000024 0 0) 0x7f58a4001060 con 0x7f58b40d1db0 2021-04-09T18:32:23.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58b8861700 1 --2- 172.21.15.32:0/1428307744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b410d020 0x7f58b40d1790 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 rx=0x7f589c006650 tx=0x7f589c009a00).ready entity=mon.0 client_cookie=9406b916baaaa908 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:23.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58b37fe700 1 -- 172.21.15.32:0/1428307744 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c009df0 con 0x7f58b410d020 2021-04-09T18:32:23.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58b37fe700 1 -- 172.21.15.32:0/1428307744 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c0009b0 con 0x7f58b410d020 2021-04-09T18:32:23.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58b37fe700 1 -- 172.21.15.32:0/1428307744 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c009df0 con 0x7f58b410d020 2021-04-09T18:32:23.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58baa45700 1 -- 172.21.15.32:0/1428307744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b410d020 msgr2=0x7f58b40d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:23.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58baa45700 1 --2- 172.21.15.32:0/1428307744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b410d020 0x7f58b40d1790 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 rx=0x7f589c006650 tx=0x7f589c009a00).stop 2021-04-09T18:32:23.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58baa45700 1 -- 172.21.15.32:0/1428307744 shutdown_connections 2021-04-09T18:32:23.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58baa45700 1 --2- 172.21.15.32:0/1428307744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b40d84a0 0x7f58b410da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58baa45700 1 --2- 172.21.15.32:0/1428307744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b410d020 0x7f58b40d1790 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.432 7f58baa45700 1 -- 172.21.15.32:0/1428307744 >> 172.21.15.32:0/1428307744 conn(0x7f58b40ce600 msgr2=0x7f58b40d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:23.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.433 7f58baa45700 1 -- 172.21.15.32:0/1428307744 shutdown_connections 2021-04-09T18:32:23.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.433 7f58baa45700 1 -- 172.21.15.32:0/1428307744 wait complete. 2021-04-09T18:32:23.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.453 7f58baa45700 1 Processor -- start 2021-04-09T18:32:23.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.453 7f58baa45700 1 -- start start 2021-04-09T18:32:23.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58baa45700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b4095190 0x7f58b40948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:23.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58baa45700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58b4092f30 0x7f58b409a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:23.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58baa45700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b4093330 0x7f58b409acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:23.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58baa45700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b409b360 con 0x7f58b4095190 2021-04-09T18:32:23.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58baa45700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b409b570 con 0x7f58b4092f30 2021-04-09T18:32:23.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58baa45700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b40c0bc0 con 0x7f58b4093330 2021-04-09T18:32:23.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b8861700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b4095190 0x7f58b40948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:23.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b8861700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b4095190 0x7f58b40948b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57510/0 (socket says 172.21.15.32:57510) 2021-04-09T18:32:23.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b8861700 1 -- 172.21.15.32:0/1300545916 learned_addr learned my addr 172.21.15.32:0/1300545916 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:23.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b3fff700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58b4092f30 0x7f58b409a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:23.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b9062700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b4093330 0x7f58b409acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:23.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b8861700 1 -- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b4093330 msgr2=0x7f58b409acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:23.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b8861700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b4093330 0x7f58b409acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b8861700 1 -- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58b4092f30 msgr2=0x7f58b409a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:23.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b8861700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58b4092f30 0x7f58b409a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:23.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b8861700 1 -- 172.21.15.32:0/1300545916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b409bac0 con 0x7f58b4095190 2021-04-09T18:32:23.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b8861700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b4095190 0x7f58b40948b0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7f589c0096a0 tx=0x7f589c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:23.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b17fa700 1 -- 172.21.15.32:0/1300545916 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c000c90 con 0x7f58b4095190 2021-04-09T18:32:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b17fa700 1 -- 172.21.15.32:0/1300545916 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c002030 con 0x7f58b4095190 2021-04-09T18:32:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58baa45700 1 -- 172.21.15.32:0/1300545916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b409bac0 con 0x7f58b4095190 2021-04-09T18:32:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58b17fa700 1 -- 172.21.15.32:0/1300545916 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589c002550 con 0x7f58b4095190 2021-04-09T18:32:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.454 7f58baa45700 1 -- 172.21.15.32:0/1300545916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b409bac0 con 0x7f58b4095190 2021-04-09T18:32:23.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.455 7f58b17fa700 1 -- 172.21.15.32:0/1300545916 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f589c002210 con 0x7f58b4095190 2021-04-09T18:32:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.455 7f58b17fa700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5898018930 0x7f589801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.455 7f58b17fa700 1 -- 172.21.15.32:0/1300545916 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f589c032dc0 con 0x7f58b4095190 2021-04-09T18:32:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.455 7f58b3fff700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5898018930 0x7f589801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:23.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.456 7f58b3fff700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5898018930 0x7f589801ae10 crc :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:23.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.475 7f58baa45700 1 -- 172.21.15.32:0/1300545916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b41b7120 con 0x7f58b4095190 2021-04-09T18:32:23.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.479 7f58b17fa700 1 -- 172.21.15.32:0/1300545916 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f589c0009c0 con 0x7f58b4095190 2021-04-09T18:32:23.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:23.663 7f58baa45700 1 -- 172.21.15.32:0/1300545916 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "alpha_data", "prefix": "osd pool delete", "pool": "alpha_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f58b410d490 con 0x7f58b4095190 2021-04-09T18:32:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.665 7f58b17fa700 1 -- 172.21.15.32:0/1300545916 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "alpha_data", "prefix": "osd pool delete", "pool": "alpha_data", "yes_i_really_really_mean_it": true}]=0 pool 'alpha_data' removed v215) v1 ==== 170+0+0 (secure 0 0 0) 0x7f589c002030 con 0x7f58b4095190 2021-04-09T18:32:24.694 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_data' removed 2021-04-09T18:32:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 -- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5898018930 msgr2=0x7f589801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5898018930 0x7f589801ae10 crc :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 -- 172.21.15.32:0/1300545916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b4095190 msgr2=0x7f58b40948b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b4095190 0x7f58b40948b0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7f589c0096a0 tx=0x7f589c00b990).stop 2021-04-09T18:32:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 -- 172.21.15.32:0/1300545916 shutdown_connections 2021-04-09T18:32:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5898018930 0x7f589801ae10 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f58b4093330 0x7f58b409acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58b4092f30 0x7f58b409a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 --2- 172.21.15.32:0/1300545916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f58b4095190 0x7f58b40948b0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 -- 172.21.15.32:0/1300545916 >> 172.21.15.32:0/1300545916 conn(0x7f58b408c120 msgr2=0x7f58b4099e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 -- 172.21.15.32:0/1300545916 shutdown_connections 2021-04-09T18:32:24.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.681 7f58baa45700 1 -- 172.21.15.32:0/1300545916 wait complete. 2021-04-09T18:32:24.708 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm bravo --yes-i-really-mean-it 2021-04-09T18:32:24.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.850 7f6a47b86700 1 Processor -- start 2021-04-09T18:32:24.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.850 7f6a47b86700 1 -- start start 2021-04-09T18:32:24.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a47b86700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a400cffb0 0x7f6a400d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:24.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a47b86700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a400d7810 0x7f6a400d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:24.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a47b86700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400d8230 0x7f6a400d5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:24.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a47b86700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400d91f0 con 0x7f6a400cffb0 2021-04-09T18:32:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a47b86700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400d7290 con 0x7f6a400d7810 2021-04-09T18:32:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a47b86700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400d62a0 con 0x7f6a400d8230 2021-04-09T18:32:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a459a2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a400cffb0 0x7f6a400d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:24.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a459a2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6a400cffb0 0x7f6a400d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57520/0 (socket says 172.21.15.32:57520) 2021-04-09T18:32:24.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a459a2700 1 -- 172.21.15.32:0/2033529956 learned_addr learned my addr 172.21.15.32:0/2033529956 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:24.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a461a3700 1 --2- 172.21.15.32:0/2033529956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400d8230 0x7f6a400d5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:24.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a451a1700 1 --2- 172.21.15.32:0/2033529956 >> v2:172.21.15.86:3300/0 conn(0x7f6a400d7810 0x7f6a400d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a459a2700 1 -- 172.21.15.32:0/2033529956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400d8230 msgr2=0x7f6a400d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a459a2700 1 --2- 172.21.15.32:0/2033529956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400d8230 0x7f6a400d5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a459a2700 1 -- 172.21.15.32:0/2033529956 >> v2:172.21.15.86:3300/0 conn(0x7f6a400d7810 msgr2=0x7f6a400d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:24.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a459a2700 1 --2- 172.21.15.32:0/2033529956 >> v2:172.21.15.86:3300/0 conn(0x7f6a400d7810 0x7f6a400d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a459a2700 1 -- 172.21.15.32:0/2033529956 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400d6490 con 0x7f6a400cffb0 2021-04-09T18:32:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a461a3700 1 --2- 172.21.15.32:0/2033529956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400d8230 0x7f6a400d5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a451a1700 1 --2- 172.21.15.32:0/2033529956 >> v2:172.21.15.86:3300/0 conn(0x7f6a400d7810 0x7f6a400d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a459a2700 1 --2- 172.21.15.32:0/2033529956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a400cffb0 0x7f6a400d2450 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 rx=0x7f6a3c006700 tx=0x7f6a3c009b80).ready entity=mon.0 client_cookie=d3b982ce2192a336 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:24.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.851 7f6a449a0700 1 -- 172.21.15.32:0/2033529956 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a3c00c0c0 con 0x7f6a400cffb0 2021-04-09T18:32:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a449a0700 1 -- 172.21.15.32:0/2033529956 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3c00c440 con 0x7f6a400cffb0 2021-04-09T18:32:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a449a0700 1 -- 172.21.15.32:0/2033529956 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a3c00c920 con 0x7f6a400cffb0 2021-04-09T18:32:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a47b86700 1 -- 172.21.15.32:0/2033529956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a400cffb0 msgr2=0x7f6a400d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:24.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a47b86700 1 --2- 172.21.15.32:0/2033529956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a400cffb0 0x7f6a400d2450 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 rx=0x7f6a3c006700 tx=0x7f6a3c009b80).stop 2021-04-09T18:32:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a47b86700 1 -- 172.21.15.32:0/2033529956 shutdown_connections 2021-04-09T18:32:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a47b86700 1 --2- 172.21.15.32:0/2033529956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400d8230 0x7f6a400d5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a47b86700 1 --2- 172.21.15.32:0/2033529956 >> v2:172.21.15.86:3300/0 conn(0x7f6a400d7810 0x7f6a400d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a47b86700 1 --2- 172.21.15.32:0/2033529956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a400cffb0 0x7f6a400d2450 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a47b86700 1 -- 172.21.15.32:0/2033529956 >> 172.21.15.32:0/2033529956 conn(0x7f6a400ce5d0 msgr2=0x7f6a400cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a47b86700 1 -- 172.21.15.32:0/2033529956 shutdown_connections 2021-04-09T18:32:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.852 7f6a47b86700 1 -- 172.21.15.32:0/2033529956 wait complete. 2021-04-09T18:32:24.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a47b86700 1 Processor -- start 2021-04-09T18:32:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a47b86700 1 -- start start 2021-04-09T18:32:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a47b86700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400b8470 0x7f6a400b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a47b86700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4018f050 0x7f6a4018f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a47b86700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a40190150 0x7f6a40190550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a47b86700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a40193110 con 0x7f6a40190150 2021-04-09T18:32:24.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a47b86700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4018fbd0 con 0x7f6a4018f050 2021-04-09T18:32:24.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a47b86700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a40190b70 con 0x7f6a400b8470 2021-04-09T18:32:24.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a461a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a40190150 0x7f6a40190550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:24.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a461a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a40190150 0x7f6a40190550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57530/0 (socket says 172.21.15.32:57530) 2021-04-09T18:32:24.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.873 7f6a461a3700 1 -- 172.21.15.32:0/1757941133 learned_addr learned my addr 172.21.15.32:0/1757941133 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a451a1700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4018f050 0x7f6a4018f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a459a2700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400b8470 0x7f6a400b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a461a3700 1 -- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400b8470 msgr2=0x7f6a400b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a461a3700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400b8470 0x7f6a400b8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a461a3700 1 -- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4018f050 msgr2=0x7f6a4018f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a461a3700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4018f050 0x7f6a4018f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a461a3700 1 -- 172.21.15.32:0/1757941133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a401a9e60 con 0x7f6a40190150 2021-04-09T18:32:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a459a2700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400b8470 0x7f6a400b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:32:24.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a451a1700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4018f050 0x7f6a4018f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:32:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a461a3700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a40190150 0x7f6a40190550 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 rx=0x7f6a3800b970 tx=0x7f6a3800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a367fc700 1 -- 172.21.15.32:0/1757941133 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a38000f90 con 0x7f6a40190150 2021-04-09T18:32:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a367fc700 1 -- 172.21.15.32:0/1757941133 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a38006790 con 0x7f6a40190150 2021-04-09T18:32:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a367fc700 1 -- 172.21.15.32:0/1757941133 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a380015f0 con 0x7f6a40190150 2021-04-09T18:32:24.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a401a9e60 con 0x7f6a40190150 2021-04-09T18:32:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.874 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a401a9e60 con 0x7f6a40190150 2021-04-09T18:32:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.875 7f6a367fc700 1 -- 172.21.15.32:0/1757941133 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6a380011f0 con 0x7f6a40190150 2021-04-09T18:32:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.875 7f6a367fc700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a28018930 0x7f6a2801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:24.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.875 7f6a367fc700 1 -- 172.21.15.32:0/1757941133 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6089+0+0 (secure 0 0 0) 0x7f6a38032f80 con 0x7f6a40190150 2021-04-09T18:32:24.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.875 7f6a459a2700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a28018930 0x7f6a2801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:24.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.875 7f6a459a2700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a28018930 0x7f6a2801ae10 crc :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:24.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.895 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a40182e10 con 0x7f6a40190150 2021-04-09T18:32:24.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:24.898 7f6a367fc700 1 -- 172.21.15.32:0/1757941133 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6a38000df0 con 0x7f6a40190150 2021-04-09T18:32:25.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.092 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "bravo", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6a40182e10 con 0x7f6a40190150 2021-04-09T18:32:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.666 7f6a367fc700 1 -- 172.21.15.32:0/1757941133 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "bravo", "yes_i_really_mean_it": true}]=0 v376) v1 ==== 103+0+0 (secure 0 0 0) 0x7f6a380011f0 con 0x7f6a40190150 2021-04-09T18:32:25.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a28018930 msgr2=0x7f6a2801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:25.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a28018930 0x7f6a2801ae10 crc :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a40190150 msgr2=0x7f6a40190550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a40190150 0x7f6a40190550 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 rx=0x7f6a3800b970 tx=0x7f6a3800bd10).stop 2021-04-09T18:32:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 shutdown_connections 2021-04-09T18:32:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a28018930 0x7f6a2801ae10 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a40190150 0x7f6a40190550 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4018f050 0x7f6a4018f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 --2- 172.21.15.32:0/1757941133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a400b8470 0x7f6a400b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 >> 172.21.15.32:0/1757941133 conn(0x7f6a40183890 msgr2=0x7f6a4019b900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:25.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 shutdown_connections 2021-04-09T18:32:25.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.709 7f6a47b86700 1 -- 172.21.15.32:0/1757941133 wait complete. 2021-04-09T18:32:25.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete bravo_metadata bravo_metadata --yes-i-really-really-mean-it 2021-04-09T18:32:25.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.882 7f178e4fb700 1 Processor -- start 2021-04-09T18:32:25.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.882 7f178e4fb700 1 -- start start 2021-04-09T18:32:25.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.882 7f178e4fb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f178810d050 0x7f17880cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:25.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.882 7f178e4fb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f17880d84b0 0x7f178810da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:25.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.882 7f178e4fb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178810d500 con 0x7f17880d0160 2021-04-09T18:32:25.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.882 7f178e4fb700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17880d8d70 con 0x7f178810d050 2021-04-09T18:32:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.882 7f178e4fb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17880d9260 con 0x7f17880d84b0 2021-04-09T18:32:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.882 7f1787fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f178810d050 0x7f17880cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f1787fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f178810d050 0x7f17880cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57538/0 (socket says 172.21.15.32:57538) 2021-04-09T18:32:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f1787fff700 1 -- 172.21.15.32:0/1523190931 learned_addr learned my addr 172.21.15.32:0/1523190931 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f178cb18700 1 --2- 172.21.15.32:0/1523190931 >> v2:172.21.15.86:3300/0 conn(0x7f17880d84b0 0x7f178810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f1787fff700 1 -- 172.21.15.32:0/1523190931 >> v2:172.21.15.86:3300/0 conn(0x7f17880d84b0 msgr2=0x7f178810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f1787fff700 1 --2- 172.21.15.32:0/1523190931 >> v2:172.21.15.86:3300/0 conn(0x7f17880d84b0 0x7f178810da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f1787fff700 1 -- 172.21.15.32:0/1523190931 >> v1:172.21.15.86:6789/0 conn(0x7f17880d0160 legacy=0x7f17880d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f1787fff700 1 -- 172.21.15.32:0/1523190931 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178810e0a0 con 0x7f178810d050 2021-04-09T18:32:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f1787fff700 1 --2- 172.21.15.32:0/1523190931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178810d050 0x7f17880cfb40 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 rx=0x7f1778006700 tx=0x7f1778009b80).ready entity=mon.0 client_cookie=82da2beff78ed524 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f17877fe700 1 -- 172.21.15.32:0/1523190931 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f177800c0c0 con 0x7f178810d050 2021-04-09T18:32:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f17877fe700 1 -- 172.21.15.32:0/1523190931 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177800c440 con 0x7f178810d050 2021-04-09T18:32:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f17877fe700 1 -- 172.21.15.32:0/1523190931 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f177800c920 con 0x7f178810d050 2021-04-09T18:32:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f178e4fb700 1 -- 172.21.15.32:0/1523190931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178810d050 msgr2=0x7f17880cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.883 7f178e4fb700 1 --2- 172.21.15.32:0/1523190931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178810d050 0x7f17880cfb40 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 rx=0x7f1778006700 tx=0x7f1778009b80).stop 2021-04-09T18:32:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.884 7f178e4fb700 1 -- 172.21.15.32:0/1523190931 shutdown_connections 2021-04-09T18:32:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.884 7f178e4fb700 1 --2- 172.21.15.32:0/1523190931 >> v2:172.21.15.86:3300/0 conn(0x7f17880d84b0 0x7f178810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.884 7f178e4fb700 1 --2- 172.21.15.32:0/1523190931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178810d050 0x7f17880cfb40 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.884 7f178e4fb700 1 -- 172.21.15.32:0/1523190931 >> 172.21.15.32:0/1523190931 conn(0x7f17880ce620 msgr2=0x7f17880d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.884 7f178e4fb700 1 -- 172.21.15.32:0/1523190931 shutdown_connections 2021-04-09T18:32:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.884 7f178e4fb700 1 -- 172.21.15.32:0/1523190931 wait complete. 2021-04-09T18:32:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.904 7f178e4fb700 1 Processor -- start 2021-04-09T18:32:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.904 7f178e4fb700 1 -- start start 2021-04-09T18:32:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f178e4fb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17880ad600 0x7f178809ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f178e4fb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178809b1b0 0x7f17880ba220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f178e4fb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17880ba840 0x7f17880a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f178e4fb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178809fa80 con 0x7f178809b1b0 2021-04-09T18:32:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f178e4fb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178809b5b0 con 0x7f17880ad600 2021-04-09T18:32:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f178e4fb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17880bac40 con 0x7f17880ba840 2021-04-09T18:32:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f177ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178809b1b0 0x7f17880ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f177ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178809b1b0 0x7f17880ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57546/0 (socket says 172.21.15.32:57546) 2021-04-09T18:32:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f177ffff700 1 -- 172.21.15.32:0/917409972 learned_addr learned my addr 172.21.15.32:0/917409972 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f178cb18700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17880ba840 0x7f17880a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f1787fff700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17880ad600 0x7f178809ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f177ffff700 1 -- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17880ba840 msgr2=0x7f17880a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f177ffff700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17880ba840 0x7f17880a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f177ffff700 1 -- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17880ad600 msgr2=0x7f178809ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f177ffff700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17880ad600 0x7f178809ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f177ffff700 1 -- 172.21.15.32:0/917409972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17880a97c0 con 0x7f178809b1b0 2021-04-09T18:32:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f177ffff700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178809b1b0 0x7f17880ba220 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 rx=0x7f177000b5b0 tx=0x7f177000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.905 7f17857fa700 1 -- 172.21.15.32:0/917409972 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f177000bd40 con 0x7f178809b1b0 2021-04-09T18:32:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.906 7f17857fa700 1 -- 172.21.15.32:0/917409972 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1770006240 con 0x7f178809b1b0 2021-04-09T18:32:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.906 7f178e4fb700 1 -- 172.21.15.32:0/917409972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17880a97c0 con 0x7f178809b1b0 2021-04-09T18:32:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.906 7f17857fa700 1 -- 172.21.15.32:0/917409972 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1770005690 con 0x7f178809b1b0 2021-04-09T18:32:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.906 7f178e4fb700 1 -- 172.21.15.32:0/917409972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17880a97c0 con 0x7f178809b1b0 2021-04-09T18:32:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.906 7f17857fa700 1 -- 172.21.15.32:0/917409972 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f17700054a0 con 0x7f178809b1b0 2021-04-09T18:32:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.906 7f17857fa700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f176c0188d0 0x7f176c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.906 7f17857fa700 1 -- 172.21.15.32:0/917409972 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6089+0+0 (secure 0 0 0) 0x7f1770032c30 con 0x7f178809b1b0 2021-04-09T18:32:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.907 7f1787fff700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f176c0188d0 0x7f176c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.907 7f1787fff700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f176c0188d0 0x7f176c01add0 crc :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:25.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.926 7f178e4fb700 1 -- 172.21.15.32:0/917409972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17880aa330 con 0x7f178809b1b0 2021-04-09T18:32:25.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:25.930 7f17857fa700 1 -- 172.21.15.32:0/917409972 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1770000c60 con 0x7f178809b1b0 2021-04-09T18:32:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.113 7f178e4fb700 1 -- 172.21.15.32:0/917409972 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "bravo_metadata", "prefix": "osd pool delete", "pool": "bravo_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f17880aa330 con 0x7f178809b1b0 2021-04-09T18:32:26.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.667 7f17857fa700 1 -- 172.21.15.32:0/917409972 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "bravo_metadata", "prefix": "osd pool delete", "pool": "bravo_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'bravo_metadata' removed v216) v1 ==== 182+0+0 (secure 0 0 0) 0x7f1770006240 con 0x7f178809b1b0 2021-04-09T18:32:26.693 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_metadata' removed 2021-04-09T18:32:26.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 -- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f176c0188d0 msgr2=0x7f176c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:26.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f176c0188d0 0x7f176c01add0 crc :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 -- 172.21.15.32:0/917409972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178809b1b0 msgr2=0x7f17880ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178809b1b0 0x7f17880ba220 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 rx=0x7f177000b5b0 tx=0x7f177000b950).stop 2021-04-09T18:32:26.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 -- 172.21.15.32:0/917409972 shutdown_connections 2021-04-09T18:32:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f176c0188d0 0x7f176c01add0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f178809b1b0 0x7f17880ba220 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17880ba840 0x7f17880a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 --2- 172.21.15.32:0/917409972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17880ad600 0x7f178809ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 -- 172.21.15.32:0/917409972 >> 172.21.15.32:0/917409972 conn(0x7f17880ae6f0 msgr2=0x7f1788097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:26.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 -- 172.21.15.32:0/917409972 shutdown_connections 2021-04-09T18:32:26.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.680 7f178e4fb700 1 -- 172.21.15.32:0/917409972 wait complete. 2021-04-09T18:32:26.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete bravo_data bravo_data --yes-i-really-really-mean-it 2021-04-09T18:32:26.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a71d72700 1 Processor -- start 2021-04-09T18:32:26.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a71d72700 1 -- start start 2021-04-09T18:32:26.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a71d72700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0d7cc0 0x7f0a6c0d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:26.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a71d72700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a6c10da80 0x7f0a6c10de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a71d72700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c10d500 con 0x7f0a6c10d050 2021-04-09T18:32:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a71d72700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c0d9260 con 0x7f0a6c10da80 2021-04-09T18:32:26.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a71d72700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0d0330 con 0x7f0a6c0d7cc0 2021-04-09T18:32:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a6bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a6c10da80 0x7f0a6c10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:26.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a6bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0a6c10da80 0x7f0a6c10de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57554/0 (socket says 172.21.15.32:57554) 2021-04-09T18:32:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.857 7f0a6bfff700 1 -- 172.21.15.32:0/2443886717 learned_addr learned my addr 172.21.15.32:0/2443886717 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:26.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6affd700 1 --2- 172.21.15.32:0/2443886717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0d7cc0 0x7f0a6c0d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6bfff700 1 -- 172.21.15.32:0/2443886717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0d7cc0 msgr2=0x7f0a6c0d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6bfff700 1 --2- 172.21.15.32:0/2443886717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0d7cc0 0x7f0a6c0d80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6bfff700 1 -- 172.21.15.32:0/2443886717 >> v1:172.21.15.86:6789/0 conn(0x7f0a6c10d050 legacy=0x7f0a6c0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6bfff700 1 -- 172.21.15.32:0/2443886717 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0d86e0 con 0x7f0a6c10da80 2021-04-09T18:32:26.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6affd700 1 --2- 172.21.15.32:0/2443886717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0d7cc0 0x7f0a6c0d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6bfff700 1 --2- 172.21.15.32:0/2443886717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c10da80 0x7f0a6c10de80 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 rx=0x7f0a64006f00 tx=0x7f0a64009f10).ready entity=mon.0 client_cookie=53e171df9c71a5be server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:26.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6a7fc700 1 -- 172.21.15.32:0/2443886717 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a6400b3b0 con 0x7f0a6c10da80 2021-04-09T18:32:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6a7fc700 1 -- 172.21.15.32:0/2443886717 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6400b730 con 0x7f0a6c10da80 2021-04-09T18:32:26.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a6a7fc700 1 -- 172.21.15.32:0/2443886717 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a6400bc10 con 0x7f0a6c10da80 2021-04-09T18:32:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a71d72700 1 -- 172.21.15.32:0/2443886717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c10da80 msgr2=0x7f0a6c10de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.858 7f0a71d72700 1 --2- 172.21.15.32:0/2443886717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c10da80 0x7f0a6c10de80 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 rx=0x7f0a64006f00 tx=0x7f0a64009f10).stop 2021-04-09T18:32:26.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.859 7f0a71d72700 1 -- 172.21.15.32:0/2443886717 shutdown_connections 2021-04-09T18:32:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.859 7f0a71d72700 1 --2- 172.21.15.32:0/2443886717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c10da80 0x7f0a6c10de80 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.859 7f0a71d72700 1 --2- 172.21.15.32:0/2443886717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0d7cc0 0x7f0a6c0d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.859 7f0a71d72700 1 -- 172.21.15.32:0/2443886717 >> 172.21.15.32:0/2443886717 conn(0x7f0a6c0ce620 msgr2=0x7f0a6c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:26.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.859 7f0a71d72700 1 -- 172.21.15.32:0/2443886717 shutdown_connections 2021-04-09T18:32:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.859 7f0a71d72700 1 -- 172.21.15.32:0/2443886717 wait complete. 2021-04-09T18:32:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.879 7f0a71d72700 1 Processor -- start 2021-04-09T18:32:26.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.879 7f0a71d72700 1 -- start start 2021-04-09T18:32:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a71d72700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c0b0920 0x7f0a6c0b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:26.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a71d72700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0a7690 0x7f0a6c0a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:26.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a71d72700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c0c2510 0x7f0a6c0c2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:26.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a71d72700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0b1340 con 0x7f0a6c0b0920 2021-04-09T18:32:26.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a71d72700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0a82e0 con 0x7f0a6c0c2510 2021-04-09T18:32:26.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a71d72700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0a8820 con 0x7f0a6c0a7690 2021-04-09T18:32:26.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c0b0920 0x7f0a6c0b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:26.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c0b0920 0x7f0a6c0b0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57560/0 (socket says 172.21.15.32:57560) 2021-04-09T18:32:26.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6b7fe700 1 -- 172.21.15.32:0/1445933071 learned_addr learned my addr 172.21.15.32:0/1445933071 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:26.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6affd700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0a7690 0x7f0a6c0a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:26.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6bfff700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c0c2510 0x7f0a6c0c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6b7fe700 1 -- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0a7690 msgr2=0x7f0a6c0a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6b7fe700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0a7690 0x7f0a6c0a7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6b7fe700 1 -- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c0c2510 msgr2=0x7f0a6c0c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:26.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6b7fe700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c0c2510 0x7f0a6c0c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:26.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6b7fe700 1 -- 172.21.15.32:0/1445933071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c08fba0 con 0x7f0a6c0b0920 2021-04-09T18:32:26.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a6b7fe700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c0b0920 0x7f0a6c0b0d20 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7f0a5c00b5b0 tx=0x7f0a5c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:26.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.880 7f0a53fff700 1 -- 172.21.15.32:0/1445933071 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a5c00bd40 con 0x7f0a6c0b0920 2021-04-09T18:32:26.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.881 7f0a53fff700 1 -- 172.21.15.32:0/1445933071 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c006240 con 0x7f0a6c0b0920 2021-04-09T18:32:26.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.881 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c08fba0 con 0x7f0a6c0b0920 2021-04-09T18:32:26.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.881 7f0a53fff700 1 -- 172.21.15.32:0/1445933071 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a5c005690 con 0x7f0a6c0b0920 2021-04-09T18:32:26.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.881 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c08fba0 con 0x7f0a6c0b0920 2021-04-09T18:32:26.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.881 7f0a53fff700 1 -- 172.21.15.32:0/1445933071 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0a5c0054a0 con 0x7f0a6c0b0920 2021-04-09T18:32:26.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.881 7f0a53fff700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a4c018900 0x7f0a4c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:26.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.881 7f0a53fff700 1 -- 172.21.15.32:0/1445933071 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f0a5c032a80 con 0x7f0a6c0b0920 2021-04-09T18:32:26.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.882 7f0a6affd700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a4c018900 0x7f0a4c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:26.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.882 7f0a6affd700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a4c018900 0x7f0a4c01adc0 crc :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.901 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c1b7180 con 0x7f0a6c0b0920 2021-04-09T18:32:26.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:26.905 7f0a53fff700 1 -- 172.21.15.32:0/1445933071 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0a5c000c60 con 0x7f0a6c0b0920 2021-04-09T18:32:27.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.089 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "bravo_data", "prefix": "osd pool delete", "pool": "bravo_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0a6c10d500 con 0x7f0a6c0b0920 2021-04-09T18:32:27.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.669 7f0a53fff700 1 -- 172.21.15.32:0/1445933071 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "bravo_data", "prefix": "osd pool delete", "pool": "bravo_data", "yes_i_really_really_mean_it": true}]=0 pool 'bravo_data' removed v217) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0a5c0054a0 con 0x7f0a6c0b0920 2021-04-09T18:32:27.720 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_data' removed 2021-04-09T18:32:27.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a4c018900 msgr2=0x7f0a4c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:27.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a4c018900 0x7f0a4c01adc0 crc :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:27.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c0b0920 msgr2=0x7f0a6c0b0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:27.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c0b0920 0x7f0a6c0b0d20 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7f0a5c00b5b0 tx=0x7f0a5c00b950).stop 2021-04-09T18:32:27.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 shutdown_connections 2021-04-09T18:32:27.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0a4c018900 0x7f0a4c01adc0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:27.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c0c2510 0x7f0a6c0c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:27.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c0a7690 0x7f0a6c0a7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:27.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 --2- 172.21.15.32:0/1445933071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a6c0b0920 0x7f0a6c0b0d20 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:27.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 >> 172.21.15.32:0/1445933071 conn(0x7f0a6c0b6b70 msgr2=0x7f0a6c0b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:27.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 shutdown_connections 2021-04-09T18:32:27.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.707 7f0a71d72700 1 -- 172.21.15.32:0/1445933071 wait complete. 2021-04-09T18:32:27.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:32:27.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.871 7f08dafdd700 1 Processor -- start 2021-04-09T18:32:27.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.871 7f08dafdd700 1 -- start start 2021-04-09T18:32:27.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.871 7f08dafdd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d40d6c90 0x7f08d40d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:27.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.871 7f08dafdd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d40d7290 con 0x7f08d40cffb0 2021-04-09T18:32:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.871 7f08dafdd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d40d2850 con 0x7f08d40d2450 2021-04-09T18:32:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.871 7f08dafdd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40d7090 con 0x7f08d40d6c90 2021-04-09T18:32:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.871 7f08d8df9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f08d40cffb0 0x7f08d40d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38926/0 (socket says 172.21.15.32:38926) 2021-04-09T18:32:27.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.871 7f08d8df9700 1 -- 172.21.15.32:0/2671997869 learned_addr learned my addr 172.21.15.32:0/2671997869 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:27.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cbfff700 1 --2- 172.21.15.32:0/2671997869 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d40d6c90 0x7f08d40d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:27.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cb7fe700 1 -- 172.21.15.32:0/2671997869 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948649895 0 0) 0x7f08d0001060 con 0x7f08d40cffb0 2021-04-09T18:32:27.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cb7fe700 1 -- 172.21.15.32:0/2671997869 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08bc002fe0 con 0x7f08d40cffb0 2021-04-09T18:32:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cb7fe700 1 -- 172.21.15.32:0/2671997869 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675841943 0 0) 0x7f08cc000c10 con 0x7f08d40d2450 2021-04-09T18:32:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cb7fe700 1 -- 172.21.15.32:0/2671997869 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08bc003210 con 0x7f08d40d2450 2021-04-09T18:32:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cb7fe700 1 -- 172.21.15.32:0/2671997869 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3582308202 0 0) 0x7f08d0001250 con 0x7f08d40cffb0 2021-04-09T18:32:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cbfff700 1 -- 172.21.15.32:0/2671997869 >> v1:172.21.15.86:6789/0 conn(0x7f08d40d2450 legacy=0x7f08d410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:27.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cbfff700 1 -- 172.21.15.32:0/2671997869 >> v1:172.21.15.32:6789/0 conn(0x7f08d40cffb0 legacy=0x7f08d40d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cbfff700 1 -- 172.21.15.32:0/2671997869 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d40d91f0 con 0x7f08d40d6c90 2021-04-09T18:32:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cbfff700 1 --2- 172.21.15.32:0/2671997869 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d40d6c90 0x7f08d40d8000 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f08c4006650 tx=0x7f08c4009a80).ready entity=mon.2 client_cookie=3d633c6786992eaf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cb7fe700 1 -- 172.21.15.32:0/2671997869 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08c40009b0 con 0x7f08d40d6c90 2021-04-09T18:32:27.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cb7fe700 1 -- 172.21.15.32:0/2671997869 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c400c0c0 con 0x7f08d40d6c90 2021-04-09T18:32:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08cb7fe700 1 -- 172.21.15.32:0/2671997869 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08c400c510 con 0x7f08d40d6c90 2021-04-09T18:32:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08dafdd700 1 -- 172.21.15.32:0/2671997869 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d40d6c90 msgr2=0x7f08d40d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.872 7f08dafdd700 1 --2- 172.21.15.32:0/2671997869 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d40d6c90 0x7f08d40d8000 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f08c4006650 tx=0x7f08c4009a80).stop 2021-04-09T18:32:27.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.873 7f08dafdd700 1 -- 172.21.15.32:0/2671997869 shutdown_connections 2021-04-09T18:32:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.873 7f08dafdd700 1 --2- 172.21.15.32:0/2671997869 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d40d6c90 0x7f08d40d8000 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.873 7f08dafdd700 1 -- 172.21.15.32:0/2671997869 >> 172.21.15.32:0/2671997869 conn(0x7f08d40ce5d0 msgr2=0x7f08d40cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:27.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.873 7f08dafdd700 1 -- 172.21.15.32:0/2671997869 shutdown_connections 2021-04-09T18:32:27.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.873 7f08dafdd700 1 -- 172.21.15.32:0/2671997869 wait complete. 2021-04-09T18:32:27.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.893 7f08dafdd700 1 Processor -- start 2021-04-09T18:32:27.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.893 7f08dafdd700 1 -- start start 2021-04-09T18:32:27.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08dafdd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d4184080 0x7f08d41955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:27.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08dafdd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d4195c00 0x7f08d408ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08dafdd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d408f550 0x7f08d408f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08dafdd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4090570 con 0x7f08d408f550 2021-04-09T18:32:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08dafdd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4184480 con 0x7f08d4184080 2021-04-09T18:32:27.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08dafdd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4196000 con 0x7f08d4195c00 2021-04-09T18:32:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08d95fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d408f550 0x7f08d408f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08d95fa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d408f550 0x7f08d408f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57574/0 (socket says 172.21.15.32:57574) 2021-04-09T18:32:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08d95fa700 1 -- 172.21.15.32:0/3051466749 learned_addr learned my addr 172.21.15.32:0/3051466749 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:27.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08cbfff700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d4195c00 0x7f08d408ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08d8df9700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d4184080 0x7f08d41955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08d95fa700 1 -- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d4195c00 msgr2=0x7f08d408ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08d95fa700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d4195c00 0x7f08d408ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08d95fa700 1 -- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d4184080 msgr2=0x7f08d41955e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:27.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08d95fa700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d4184080 0x7f08d41955e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.894 7f08d95fa700 1 -- 172.21.15.32:0/3051466749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d41847b0 con 0x7f08d408f550 2021-04-09T18:32:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.895 7f08d95fa700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d408f550 0x7f08d408f950 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7f08cc00b580 tx=0x7f08cc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:27.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.895 7f08c97fa700 1 -- 172.21.15.32:0/3051466749 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08cc00bd10 con 0x7f08d408f550 2021-04-09T18:32:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.895 7f08c97fa700 1 -- 172.21.15.32:0/3051466749 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc006240 con 0x7f08d408f550 2021-04-09T18:32:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.895 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d41847b0 con 0x7f08d408f550 2021-04-09T18:32:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.895 7f08c97fa700 1 -- 172.21.15.32:0/3051466749 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08cc005710 con 0x7f08d408f550 2021-04-09T18:32:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.895 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d41847b0 con 0x7f08d408f550 2021-04-09T18:32:27.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.895 7f08c97fa700 1 -- 172.21.15.32:0/3051466749 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f08cc005520 con 0x7f08d408f550 2021-04-09T18:32:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.896 7f08c97fa700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b4018930 0x7f08b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.896 7f08c97fa700 1 -- 172.21.15.32:0/3051466749 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f08cc0328e0 con 0x7f08d408f550 2021-04-09T18:32:27.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.896 7f08d8df9700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b4018930 0x7f08b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:27.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.896 7f08d8df9700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b4018930 0x7f08b401ae10 crc :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:27.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.915 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d4182e80 con 0x7f08d408f550 2021-04-09T18:32:27.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:27.919 7f08c97fa700 1 -- 172.21.15.32:0/3051466749 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f08cc006030 con 0x7f08d408f550 2021-04-09T18:32:28.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.130 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f08d40d7290 con 0x7f08d408f550 2021-04-09T18:32:28.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.133 7f08c97fa700 1 -- 172.21.15.32:0/3051466749 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f08cc005520 con 0x7f08d408f550 2021-04-09T18:32:28.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b4018930 msgr2=0x7f08b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b4018930 0x7f08b401ae10 crc :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d408f550 msgr2=0x7f08d408f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d408f550 0x7f08d408f950 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7f08cc00b580 tx=0x7f08cc00b920).stop 2021-04-09T18:32:28.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 shutdown_connections 2021-04-09T18:32:28.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f08b4018930 0x7f08b401ae10 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08d408f550 0x7f08d408f950 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08d4195c00 0x7f08d408ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 --2- 172.21.15.32:0/3051466749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08d4184080 0x7f08d41955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 >> 172.21.15.32:0/3051466749 conn(0x7f08d41a8f50 msgr2=0x7f08d4188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:28.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 shutdown_connections 2021-04-09T18:32:28.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.139 7f08dafdd700 1 -- 172.21.15.32:0/3051466749 wait complete. 2021-04-09T18:32:28.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestMultiFilesystems.test_grow_shrink' 2021-04-09T18:32:28.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.298 7f07bd691700 1 Processor -- start 2021-04-09T18:32:28.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.298 7f07bd691700 1 -- start start 2021-04-09T18:32:28.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.298 7f07bd691700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f07b80d7ca0 0x7f07b80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.298 7f07bd691700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07b810da00 0x7f07b810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07bd691700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b810d480 con 0x7f07b810d010 2021-04-09T18:32:28.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07bd691700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b80d9220 con 0x7f07b810da00 2021-04-09T18:32:28.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07bd691700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b80d86c0 con 0x7f07b80d7ca0 2021-04-09T18:32:28.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b7ac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07b810da00 0x7f07b810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b72bf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f07b810d010 0x7f07b80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38940/0 (socket says 172.21.15.32:38940) 2021-04-09T18:32:28.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b72bf700 1 -- 172.21.15.32:0/4290591066 learned_addr learned my addr 172.21.15.32:0/4290591066 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:28.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b6abe700 1 --2- 172.21.15.32:0/4290591066 >> v2:172.21.15.86:3300/0 conn(0x7f07b80d7ca0 0x7f07b80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b7ac0700 1 -- 172.21.15.32:0/4290591066 >> v2:172.21.15.86:3300/0 conn(0x7f07b80d7ca0 msgr2=0x7f07b80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:28.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b7ac0700 1 --2- 172.21.15.32:0/4290591066 >> v2:172.21.15.86:3300/0 conn(0x7f07b80d7ca0 0x7f07b80d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b7ac0700 1 -- 172.21.15.32:0/4290591066 >> v1:172.21.15.32:6789/0 conn(0x7f07b810d010 legacy=0x7f07b80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b62bd700 1 -- 172.21.15.32:0/4290591066 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880254056 0 0) 0x7f07a8000ce0 con 0x7f07b810d010 2021-04-09T18:32:28.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b7ac0700 1 -- 172.21.15.32:0/4290591066 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b80d1f70 con 0x7f07b810da00 2021-04-09T18:32:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b7ac0700 1 --2- 172.21.15.32:0/4290591066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810da00 0x7f07b810de00 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 rx=0x7f07b0006e30 tx=0x7f07b0009de0).ready entity=mon.0 client_cookie=62ff910a6519b8ae server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b62bd700 1 -- 172.21.15.32:0/4290591066 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07b000b300 con 0x7f07b810da00 2021-04-09T18:32:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b62bd700 1 -- 172.21.15.32:0/4290591066 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b000b680 con 0x7f07b810da00 2021-04-09T18:32:28.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07b62bd700 1 -- 172.21.15.32:0/4290591066 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07b000bb60 con 0x7f07b810da00 2021-04-09T18:32:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07bd691700 1 -- 172.21.15.32:0/4290591066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810da00 msgr2=0x7f07b810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.299 7f07bd691700 1 --2- 172.21.15.32:0/4290591066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810da00 0x7f07b810de00 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 rx=0x7f07b0006e30 tx=0x7f07b0009de0).stop 2021-04-09T18:32:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.300 7f07bd691700 1 -- 172.21.15.32:0/4290591066 shutdown_connections 2021-04-09T18:32:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.300 7f07bd691700 1 --2- 172.21.15.32:0/4290591066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b810da00 0x7f07b810de00 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.300 7f07bd691700 1 --2- 172.21.15.32:0/4290591066 >> v2:172.21.15.86:3300/0 conn(0x7f07b80d7ca0 0x7f07b80d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.300 7f07bd691700 1 -- 172.21.15.32:0/4290591066 >> 172.21.15.32:0/4290591066 conn(0x7f07b80ce5f0 msgr2=0x7f07b80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.300 7f07bd691700 1 -- 172.21.15.32:0/4290591066 shutdown_connections 2021-04-09T18:32:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.300 7f07bd691700 1 -- 172.21.15.32:0/4290591066 wait complete. 2021-04-09T18:32:28.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.320 7f07bd691700 1 Processor -- start 2021-04-09T18:32:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.320 7f07bd691700 1 -- start start 2021-04-09T18:32:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07bd691700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b81889c0 0x7f07b809a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07bd691700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b818e2c0 0x7f07b81aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07bd691700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07b81aaad0 0x7f07b80a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07bd691700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b81aaed0 con 0x7f07b818e2c0 2021-04-09T18:32:28.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07bd691700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b81ab0f0 con 0x7f07b81889c0 2021-04-09T18:32:28.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07bd691700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b80a3ed0 con 0x7f07b81aaad0 2021-04-09T18:32:28.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b6abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b818e2c0 0x7f07b81aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b6abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b818e2c0 0x7f07b81aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57590/0 (socket says 172.21.15.32:57590) 2021-04-09T18:32:28.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b6abe700 1 -- 172.21.15.32:0/3265039585 learned_addr learned my addr 172.21.15.32:0/3265039585 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:28.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b7ac0700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07b81aaad0 0x7f07b80a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b72bf700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b81889c0 0x7f07b809a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b6abe700 1 -- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07b81aaad0 msgr2=0x7f07b80a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b6abe700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07b81aaad0 0x7f07b80a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b6abe700 1 -- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b81889c0 msgr2=0x7f07b809a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b6abe700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b81889c0 0x7f07b809a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b6abe700 1 -- 172.21.15.32:0/3265039585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b80a40f0 con 0x7f07b818e2c0 2021-04-09T18:32:28.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b7ac0700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07b81aaad0 0x7f07b80a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b72bf700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b81889c0 0x7f07b809a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.321 7f07b6abe700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b818e2c0 0x7f07b81aa4b0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 rx=0x7f07ac006f00 tx=0x7f07ac00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.322 7f079ffff700 1 -- 172.21.15.32:0/3265039585 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07ac00bd30 con 0x7f07b818e2c0 2021-04-09T18:32:28.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.322 7f079ffff700 1 -- 172.21.15.32:0/3265039585 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac002030 con 0x7f07b818e2c0 2021-04-09T18:32:28.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.322 7f079ffff700 1 -- 172.21.15.32:0/3265039585 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07ac0024c0 con 0x7f07b818e2c0 2021-04-09T18:32:28.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.322 7f07bd691700 1 -- 172.21.15.32:0/3265039585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b80a40f0 con 0x7f07b818e2c0 2021-04-09T18:32:28.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.322 7f07bd691700 1 -- 172.21.15.32:0/3265039585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b80a40f0 con 0x7f07b818e2c0 2021-04-09T18:32:28.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.322 7f079ffff700 1 -- 172.21.15.32:0/3265039585 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f07ac000c90 con 0x7f07b818e2c0 2021-04-09T18:32:28.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.322 7f079ffff700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07980188d0 0x7f079801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.322 7f079ffff700 1 -- 172.21.15.32:0/3265039585 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f07ac032910 con 0x7f07b818e2c0 2021-04-09T18:32:28.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.323 7f07b72bf700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07980188d0 0x7f079801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.323 7f07b72bf700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07980188d0 0x7f079801add0 crc :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:28.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.343 7f07bd691700 1 -- 172.21.15.32:0/3265039585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b810d480 con 0x7f07b818e2c0 2021-04-09T18:32:28.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.346 7f079ffff700 1 -- 172.21.15.32:0/3265039585 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f07ac01f780 con 0x7f07b818e2c0 2021-04-09T18:32:28.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.537 7f07bd691700 1 -- 172.21.15.32:0/3265039585 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestMultiFilesystems.test_grow_shrink"]} v 0) v1 -- 0x7f07b810d480 con 0x7f07b818e2c0 2021-04-09T18:32:28.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.674 7f079ffff700 1 -- 172.21.15.32:0/3265039585 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestMultiFilesystems.test_grow_shrink"]}]=0 v880) v1 ==== 143+0+0 (secure 0 0 0) 0x7f07ac000c90 con 0x7f07b818e2c0 2021-04-09T18:32:28.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 -- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07980188d0 msgr2=0x7f079801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07980188d0 0x7f079801add0 crc :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 -- 172.21.15.32:0/3265039585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b818e2c0 msgr2=0x7f07b81aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b818e2c0 0x7f07b81aa4b0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 rx=0x7f07ac006f00 tx=0x7f07ac00b940).stop 2021-04-09T18:32:28.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 -- 172.21.15.32:0/3265039585 shutdown_connections 2021-04-09T18:32:28.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f07980188d0 0x7f079801add0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07b818e2c0 0x7f07b81aa4b0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07b81aaad0 0x7f07b80a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 --2- 172.21.15.32:0/3265039585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b81889c0 0x7f07b809a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 -- 172.21.15.32:0/3265039585 >> 172.21.15.32:0/3265039585 conn(0x7f07b81a0db0 msgr2=0x7f07b81a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:28.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 -- 172.21.15.32:0/3265039585 shutdown_connections 2021-04-09T18:32:28.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.703 7f07bd691700 1 -- 172.21.15.32:0/3265039585 wait complete. 2021-04-09T18:32:28.729 INFO:tasks.cephfs_test_runner:test_grow_shrink (tasks.cephfs.test_failover.TestMultiFilesystems) ... ok 2021-04-09T18:32:28.729 INFO:tasks.cephfs_test_runner:Starting test: test_standby (tasks.cephfs.test_failover.TestMultiFilesystems) 2021-04-09T18:32:28.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestMultiFilesystems.test_standby' 2021-04-09T18:32:28.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.862 7efe2a76b700 1 Processor -- start 2021-04-09T18:32:28.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.862 7efe2a76b700 1 -- start start 2021-04-09T18:32:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.863 7efe2a76b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efe2410d010 0x7efe240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.863 7efe2a76b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240d8490 0x7efe2410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.863 7efe2a76b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2410d480 con 0x7efe240d1da0 2021-04-09T18:32:28.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.863 7efe2a76b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe240d8d30 con 0x7efe2410d010 2021-04-09T18:32:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.863 7efe2a76b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe240d9220 con 0x7efe240d8490 2021-04-09T18:32:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.863 7efe23fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efe2410d010 0x7efe240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.863 7efe28d88700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240d8490 0x7efe2410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.863 7efe1bfff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efe240d1da0 0x7efe240d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:36884/0 (socket says 172.21.15.32:36884) 2021-04-09T18:32:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.863 7efe1bfff700 1 -- 172.21.15.32:0/754014883 learned_addr learned my addr 172.21.15.32:0/754014883 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe28d88700 1 -- 172.21.15.32:0/754014883 >> v2:172.21.15.86:3300/0 conn(0x7efe2410d010 msgr2=0x7efe240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe28d88700 1 --2- 172.21.15.32:0/754014883 >> v2:172.21.15.86:3300/0 conn(0x7efe2410d010 0x7efe240d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe28d88700 1 -- 172.21.15.32:0/754014883 >> v1:172.21.15.86:6789/0 conn(0x7efe240d1da0 legacy=0x7efe240d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe237fe700 1 -- 172.21.15.32:0/754014883 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434874580 0 0) 0x7efe0c0010c0 con 0x7efe240d1da0 2021-04-09T18:32:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe28d88700 1 -- 172.21.15.32:0/754014883 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2410e020 con 0x7efe240d8490 2021-04-09T18:32:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe28d88700 1 --2- 172.21.15.32:0/754014883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240d8490 0x7efe2410da00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7efe1c006e50 tx=0x7efe1c009e70).ready entity=mon.2 client_cookie=9b0984732f89ce76 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe237fe700 1 -- 172.21.15.32:0/754014883 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe1c006a20 con 0x7efe240d8490 2021-04-09T18:32:28.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe237fe700 1 -- 172.21.15.32:0/754014883 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c006790 con 0x7efe240d8490 2021-04-09T18:32:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe237fe700 1 -- 172.21.15.32:0/754014883 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe1c00b700 con 0x7efe240d8490 2021-04-09T18:32:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe2a76b700 1 -- 172.21.15.32:0/754014883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240d8490 msgr2=0x7efe2410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.864 7efe2a76b700 1 --2- 172.21.15.32:0/754014883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240d8490 0x7efe2410da00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7efe1c006e50 tx=0x7efe1c009e70).stop 2021-04-09T18:32:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.865 7efe2a76b700 1 -- 172.21.15.32:0/754014883 shutdown_connections 2021-04-09T18:32:28.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.865 7efe2a76b700 1 --2- 172.21.15.32:0/754014883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240d8490 0x7efe2410da00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.865 7efe2a76b700 1 --2- 172.21.15.32:0/754014883 >> v2:172.21.15.86:3300/0 conn(0x7efe2410d010 0x7efe240d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.865 7efe2a76b700 1 -- 172.21.15.32:0/754014883 >> 172.21.15.32:0/754014883 conn(0x7efe240ce5f0 msgr2=0x7efe240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:28.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.865 7efe2a76b700 1 -- 172.21.15.32:0/754014883 shutdown_connections 2021-04-09T18:32:28.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.865 7efe2a76b700 1 -- 172.21.15.32:0/754014883 wait complete. 2021-04-09T18:32:28.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.885 7efe2a76b700 1 Processor -- start 2021-04-09T18:32:28.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.885 7efe2a76b700 1 -- start start 2021-04-09T18:32:28.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe2a76b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe241a5210 0x7efe241a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe2a76b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe241a6340 0x7efe241a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe2a76b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240abaa0 0x7efe240abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe2a76b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe241a6d60 con 0x7efe241a5210 2021-04-09T18:32:28.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe2a76b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe240ac4c0 con 0x7efe241a6340 2021-04-09T18:32:28.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe2a76b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe240ac6e0 con 0x7efe240abaa0 2021-04-09T18:32:28.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe23fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe241a5210 0x7efe241a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe23fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe241a5210 0x7efe241a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57602/0 (socket says 172.21.15.32:57602) 2021-04-09T18:32:28.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe23fff700 1 -- 172.21.15.32:0/210141236 learned_addr learned my addr 172.21.15.32:0/210141236 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:28.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe28d88700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240abaa0 0x7efe240abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe1bfff700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe241a6340 0x7efe241a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe23fff700 1 -- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240abaa0 msgr2=0x7efe240abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe23fff700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240abaa0 0x7efe240abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe23fff700 1 -- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe241a6340 msgr2=0x7efe241a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:28.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe23fff700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe241a6340 0x7efe241a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:28.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe23fff700 1 -- 172.21.15.32:0/210141236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2417a720 con 0x7efe241a5210 2021-04-09T18:32:28.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe1bfff700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe241a6340 0x7efe241a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:32:28.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe23fff700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe241a5210 0x7efe241a56c0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 rx=0x7efe14006f90 tx=0x7efe1400b8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:28.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe217fa700 1 -- 172.21.15.32:0/210141236 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe140009b0 con 0x7efe241a5210 2021-04-09T18:32:28.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe217fa700 1 -- 172.21.15.32:0/210141236 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1400bc90 con 0x7efe241a5210 2021-04-09T18:32:28.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe2a76b700 1 -- 172.21.15.32:0/210141236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2417a9c0 con 0x7efe241a5210 2021-04-09T18:32:28.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe217fa700 1 -- 172.21.15.32:0/210141236 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe14002030 con 0x7efe241a5210 2021-04-09T18:32:28.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.886 7efe2a76b700 1 -- 172.21.15.32:0/210141236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2417a9c0 con 0x7efe241a5210 2021-04-09T18:32:28.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.887 7efe217fa700 1 -- 172.21.15.32:0/210141236 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efe14006b00 con 0x7efe241a5210 2021-04-09T18:32:28.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.887 7efe217fa700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe08018930 0x7efe0801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:28.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.887 7efe217fa700 1 -- 172.21.15.32:0/210141236 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5273+0+0 (secure 0 0 0) 0x7efe14034aa0 con 0x7efe241a5210 2021-04-09T18:32:28.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.887 7efe1bfff700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe08018930 0x7efe0801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:28.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.888 7efe1bfff700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe08018930 0x7efe0801adc0 crc :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:28.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.907 7efe2a76b700 1 -- 172.21.15.32:0/210141236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe241a6d60 con 0x7efe241a5210 2021-04-09T18:32:28.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:28.910 7efe217fa700 1 -- 172.21.15.32:0/210141236 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efe14006b00 con 0x7efe241a5210 2021-04-09T18:32:29.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.097 7efe2a76b700 1 -- 172.21.15.32:0/210141236 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestMultiFilesystems.test_standby"]} v 0) v1 -- 0x7efe241a6d60 con 0x7efe241a5210 2021-04-09T18:32:29.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.675 7efe217fa700 1 -- 172.21.15.32:0/210141236 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestMultiFilesystems.test_standby"]}]=0 v881) v1 ==== 142+0+0 (secure 0 0 0) 0x7efe1400bc90 con 0x7efe241a5210 2021-04-09T18:32:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 -- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe08018930 msgr2=0x7efe0801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:29.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe08018930 0x7efe0801adc0 crc :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 -- 172.21.15.32:0/210141236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe241a5210 msgr2=0x7efe241a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe241a5210 0x7efe241a56c0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 rx=0x7efe14006f90 tx=0x7efe1400b8a0).stop 2021-04-09T18:32:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 -- 172.21.15.32:0/210141236 shutdown_connections 2021-04-09T18:32:29.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efe08018930 0x7efe0801adc0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe240abaa0 0x7efe240abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe241a6340 0x7efe241a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 --2- 172.21.15.32:0/210141236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe241a5210 0x7efe241a56c0 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 -- 172.21.15.32:0/210141236 >> 172.21.15.32:0/210141236 conn(0x7efe241a99c0 msgr2=0x7efe2417dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:29.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 -- 172.21.15.32:0/210141236 shutdown_connections 2021-04-09T18:32:29.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.714 7efe2a76b700 1 -- 172.21.15.32:0/210141236 wait complete. 2021-04-09T18:32:29.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:32:29.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.878 7f3b936d7700 1 Processor -- start 2021-04-09T18:32:29.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.878 7f3b936d7700 1 -- start start 2021-04-09T18:32:29.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b936d7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b8c10d010 0x7f3b8c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:29.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b936d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c0d8490 0x7f3b8c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:29.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b936d7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8c10d480 con 0x7f3b8c0d1da0 2021-04-09T18:32:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b936d7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8c0d8d30 con 0x7f3b8c10d010 2021-04-09T18:32:29.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b936d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c0d9220 con 0x7f3b8c0d8490 2021-04-09T18:32:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b914f3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3b8c10d010 0x7f3b8c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b90cf2700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3b8c0d1da0 0x7f3b8c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38972/0 (socket says 172.21.15.32:38972) 2021-04-09T18:32:29.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b90cf2700 1 -- 172.21.15.32:0/682041704 learned_addr learned my addr 172.21.15.32:0/682041704 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b91cf4700 1 --2- 172.21.15.32:0/682041704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c0d8490 0x7f3b8c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:29.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b83fff700 1 -- 172.21.15.32:0/682041704 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842699944 0 0) 0x7f3b7c000ce0 con 0x7f3b8c0d1da0 2021-04-09T18:32:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b83fff700 1 -- 172.21.15.32:0/682041704 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b78002fe0 con 0x7f3b8c0d1da0 2021-04-09T18:32:29.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b914f3700 1 -- 172.21.15.32:0/682041704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c0d8490 msgr2=0x7f3b8c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:29.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b914f3700 1 --2- 172.21.15.32:0/682041704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c0d8490 0x7f3b8c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b914f3700 1 -- 172.21.15.32:0/682041704 >> v1:172.21.15.32:6789/0 conn(0x7f3b8c0d1da0 legacy=0x7f3b8c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:29.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b914f3700 1 -- 172.21.15.32:0/682041704 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c10e020 con 0x7f3b8c10d010 2021-04-09T18:32:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b914f3700 1 --2- 172.21.15.32:0/682041704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c10d010 0x7f3b8c0d1780 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 rx=0x7f3b88006650 tx=0x7f3b88009a80).ready entity=mon.0 client_cookie=f25227939603b5d5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:29.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.879 7f3b83fff700 1 -- 172.21.15.32:0/682041704 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b880009b0 con 0x7f3b8c10d010 2021-04-09T18:32:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.880 7f3b83fff700 1 -- 172.21.15.32:0/682041704 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8800c0c0 con 0x7f3b8c10d010 2021-04-09T18:32:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.880 7f3b83fff700 1 -- 172.21.15.32:0/682041704 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b8800c510 con 0x7f3b8c10d010 2021-04-09T18:32:29.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.880 7f3b936d7700 1 -- 172.21.15.32:0/682041704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c10d010 msgr2=0x7f3b8c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.880 7f3b936d7700 1 --2- 172.21.15.32:0/682041704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c10d010 0x7f3b8c0d1780 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 rx=0x7f3b88006650 tx=0x7f3b88009a80).stop 2021-04-09T18:32:29.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.880 7f3b936d7700 1 -- 172.21.15.32:0/682041704 shutdown_connections 2021-04-09T18:32:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.880 7f3b936d7700 1 --2- 172.21.15.32:0/682041704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c0d8490 0x7f3b8c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.880 7f3b936d7700 1 --2- 172.21.15.32:0/682041704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c10d010 0x7f3b8c0d1780 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.880 7f3b936d7700 1 -- 172.21.15.32:0/682041704 >> 172.21.15.32:0/682041704 conn(0x7f3b8c0ce5f0 msgr2=0x7f3b8c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:29.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.881 7f3b936d7700 1 -- 172.21.15.32:0/682041704 shutdown_connections 2021-04-09T18:32:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.881 7f3b936d7700 1 -- 172.21.15.32:0/682041704 wait complete. 2021-04-09T18:32:29.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b936d7700 1 Processor -- start 2021-04-09T18:32:29.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b936d7700 1 -- start start 2021-04-09T18:32:29.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b936d7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c09e530 0x7f3b8c09b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:29.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b936d7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c09e930 0x7f3b8c09f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:29.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b936d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c09f990 0x7f3b8c094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:29.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b936d7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c09edf0 con 0x7f3b8c09e530 2021-04-09T18:32:29.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b936d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c09fd90 con 0x7f3b8c09f990 2021-04-09T18:32:29.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b936d7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c09bd40 con 0x7f3b8c09e930 2021-04-09T18:32:29.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b914f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c09e530 0x7f3b8c09b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:29.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b914f3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c09e530 0x7f3b8c09b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57616/0 (socket says 172.21.15.32:57616) 2021-04-09T18:32:29.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.901 7f3b914f3700 1 -- 172.21.15.32:0/4095683409 learned_addr learned my addr 172.21.15.32:0/4095683409 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:29.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b90cf2700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c09e930 0x7f3b8c09f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:29.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b91cf4700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c09f990 0x7f3b8c094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:29.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b914f3700 1 -- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c09e930 msgr2=0x7f3b8c09f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:29.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b914f3700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c09e930 0x7f3b8c09f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b914f3700 1 -- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c09f990 msgr2=0x7f3b8c094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:29.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b914f3700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c09f990 0x7f3b8c094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:29.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b914f3700 1 -- 172.21.15.32:0/4095683409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c095290 con 0x7f3b8c09e530 2021-04-09T18:32:29.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b914f3700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c09e530 0x7f3b8c09b720 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 rx=0x7f3b88009630 tx=0x7f3b8800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:29.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b81ffb700 1 -- 172.21.15.32:0/4095683409 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b880093a0 con 0x7f3b8c09e530 2021-04-09T18:32:29.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b81ffb700 1 -- 172.21.15.32:0/4095683409 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8800bcf0 con 0x7f3b8c09e530 2021-04-09T18:32:29.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b81ffb700 1 -- 172.21.15.32:0/4095683409 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b880054d0 con 0x7f3b8c09e530 2021-04-09T18:32:29.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8c095290 con 0x7f3b8c09e530 2021-04-09T18:32:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.902 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8c095290 con 0x7f3b8c09e530 2021-04-09T18:32:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.903 7f3b81ffb700 1 -- 172.21.15.32:0/4095683409 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3b880054f0 con 0x7f3b8c09e530 2021-04-09T18:32:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.903 7f3b81ffb700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3b6c0188c0 0x7f3b6c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:29.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.903 7f3b81ffb700 1 -- 172.21.15.32:0/4095683409 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f3b880328e0 con 0x7f3b8c09e530 2021-04-09T18:32:29.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.903 7f3b90cf2700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3b6c0188c0 0x7f3b6c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:29.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.903 7f3b90cf2700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3b6c0188c0 0x7f3b6c01adc0 crc :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:29.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.923 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8c095b10 con 0x7f3b8c09e530 2021-04-09T18:32:29.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:29.926 7f3b81ffb700 1 -- 172.21.15.32:0/4095683409 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3b880052e0 con 0x7f3b8c09e530 2021-04-09T18:32:30.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.143 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3b8c095b10 con 0x7f3b8c09e530 2021-04-09T18:32:30.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.146 7f3b81ffb700 1 -- 172.21.15.32:0/4095683409 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3b880054f0 con 0x7f3b8c09e530 2021-04-09T18:32:30.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3b6c0188c0 msgr2=0x7f3b6c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:30.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3b6c0188c0 0x7f3b6c01adc0 crc :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c09e530 msgr2=0x7f3b8c09b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:30.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c09e530 0x7f3b8c09b720 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 rx=0x7f3b88009630 tx=0x7f3b8800b900).stop 2021-04-09T18:32:30.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 shutdown_connections 2021-04-09T18:32:30.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3b6c0188c0 0x7f3b6c01adc0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8c09f990 0x7f3b8c094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b8c09e930 0x7f3b8c09f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 --2- 172.21.15.32:0/4095683409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b8c09e530 0x7f3b8c09b720 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 >> 172.21.15.32:0/4095683409 conn(0x7f3b8c0c4290 msgr2=0x7f3b8c0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:30.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 shutdown_connections 2021-04-09T18:32:30.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.152 7f3b936d7700 1 -- 172.21.15.32:0/4095683409 wait complete. 2021-04-09T18:32:30.178 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:32:30.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:32:30.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:32:30.298 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:32:30.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:32:30.300 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:32:30.300 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:32:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:32:30.345 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:32:30.345 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:32:30.346 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:32:30.346 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:32:30.347 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:32:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:32:30.389 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:32:30.390 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:32:30.390 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:32:30.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.523 7ff6f034c700 1 Processor -- start 2021-04-09T18:32:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.524 7ff6f034c700 1 -- start start 2021-04-09T18:32:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.524 7ff6f034c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff6e810d010 0x7ff6e80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.524 7ff6f034c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e80d1da0 0x7ff6e80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:30.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.524 7ff6f034c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e810d480 con 0x7ff6e80d82c0 2021-04-09T18:32:30.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.524 7ff6f034c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e80d9220 con 0x7ff6e810d010 2021-04-09T18:32:30.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.524 7ff6f034c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e80d86c0 con 0x7ff6e80d1da0 2021-04-09T18:32:30.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.524 7ff6ee168700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff6e810d010 0x7ff6e80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:30.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ee168700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff6e810d010 0x7ff6e80d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57624/0 (socket says 172.21.15.32:57624) 2021-04-09T18:32:30.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ee168700 1 -- 172.21.15.32:0/2932256603 learned_addr learned my addr 172.21.15.32:0/2932256603 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:30.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ed967700 1 --2- 172.21.15.32:0/2932256603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e80d1da0 0x7ff6e80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:30.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ee168700 1 -- 172.21.15.32:0/2932256603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e80d1da0 msgr2=0x7ff6e80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:30.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ee168700 1 --2- 172.21.15.32:0/2932256603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e80d1da0 0x7ff6e80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ee168700 1 -- 172.21.15.32:0/2932256603 >> v1:172.21.15.86:6789/0 conn(0x7ff6e80d82c0 legacy=0x7ff6e810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ee168700 1 -- 172.21.15.32:0/2932256603 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e810e1f0 con 0x7ff6e810d010 2021-04-09T18:32:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ee168700 1 --2- 172.21.15.32:0/2932256603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e810d010 0x7ff6e80d1780 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 rx=0x7ff6e4006700 tx=0x7ff6e4009b80).ready entity=mon.0 client_cookie=2423ea6b53572c4c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ed166700 1 -- 172.21.15.32:0/2932256603 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6e400c0c0 con 0x7ff6e810d010 2021-04-09T18:32:30.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ed166700 1 -- 172.21.15.32:0/2932256603 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e400c440 con 0x7ff6e810d010 2021-04-09T18:32:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6ed166700 1 -- 172.21.15.32:0/2932256603 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6e400c920 con 0x7ff6e810d010 2021-04-09T18:32:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6f034c700 1 -- 172.21.15.32:0/2932256603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e810d010 msgr2=0x7ff6e80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.525 7ff6f034c700 1 --2- 172.21.15.32:0/2932256603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e810d010 0x7ff6e80d1780 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 rx=0x7ff6e4006700 tx=0x7ff6e4009b80).stop 2021-04-09T18:32:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.526 7ff6f034c700 1 -- 172.21.15.32:0/2932256603 shutdown_connections 2021-04-09T18:32:30.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.526 7ff6f034c700 1 --2- 172.21.15.32:0/2932256603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e80d1da0 0x7ff6e80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.526 7ff6f034c700 1 --2- 172.21.15.32:0/2932256603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e810d010 0x7ff6e80d1780 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.526 7ff6f034c700 1 -- 172.21.15.32:0/2932256603 >> 172.21.15.32:0/2932256603 conn(0x7ff6e80ce5f0 msgr2=0x7ff6e80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.526 7ff6f034c700 1 -- 172.21.15.32:0/2932256603 shutdown_connections 2021-04-09T18:32:30.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.526 7ff6f034c700 1 -- 172.21.15.32:0/2932256603 wait complete. 2021-04-09T18:32:30.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.546 7ff6f034c700 1 Processor -- start 2021-04-09T18:32:30.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.546 7ff6f034c700 1 -- start start 2021-04-09T18:32:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6f034c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e80b42d0 0x7ff6e8183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6f034c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e81ac090 0x7ff6e81ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6f034c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e8184d30 0x7ff6e8185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:30.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6f034c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e80a1a90 con 0x7ff6e80b42d0 2021-04-09T18:32:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6f034c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e81acab0 con 0x7ff6e8184d30 2021-04-09T18:32:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6f034c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e8185a00 con 0x7ff6e81ac090 2021-04-09T18:32:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ee168700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e80b42d0 0x7ff6e8183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ee168700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e80b42d0 0x7ff6e8183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57630/0 (socket says 172.21.15.32:57630) 2021-04-09T18:32:30.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ee168700 1 -- 172.21.15.32:0/4250066375 learned_addr learned my addr 172.21.15.32:0/4250066375 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ed967700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e81ac090 0x7ff6e81ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ee969700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e8184d30 0x7ff6e8185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ee168700 1 -- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e81ac090 msgr2=0x7ff6e81ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:30.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ee168700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e81ac090 0x7ff6e81ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ee168700 1 -- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e8184d30 msgr2=0x7ff6e8185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ee168700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e8184d30 0x7ff6e8185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.547 7ff6ee168700 1 -- 172.21.15.32:0/4250066375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e8185750 con 0x7ff6e80b42d0 2021-04-09T18:32:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.548 7ff6ee168700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e80b42d0 0x7ff6e8183520 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 rx=0x7ff6e400b5e0 tx=0x7ff6e400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.548 7ff6daffd700 1 -- 172.21.15.32:0/4250066375 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6e400bd70 con 0x7ff6e80b42d0 2021-04-09T18:32:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.548 7ff6daffd700 1 -- 172.21.15.32:0/4250066375 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e4002030 con 0x7ff6e80b42d0 2021-04-09T18:32:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.548 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e8185750 con 0x7ff6e80b42d0 2021-04-09T18:32:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.548 7ff6daffd700 1 -- 172.21.15.32:0/4250066375 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6e4002550 con 0x7ff6e80b42d0 2021-04-09T18:32:30.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.548 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e8185750 con 0x7ff6e80b42d0 2021-04-09T18:32:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.548 7ff6daffd700 1 -- 172.21.15.32:0/4250066375 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff6e4000c90 con 0x7ff6e80b42d0 2021-04-09T18:32:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.548 7ff6daffd700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6d00188d0 0x7ff6d001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.549 7ff6daffd700 1 -- 172.21.15.32:0/4250066375 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5273+0+0 (secure 0 0 0) 0x7ff6e4032930 con 0x7ff6e80b42d0 2021-04-09T18:32:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.549 7ff6ed967700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6d00188d0 0x7ff6d001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:30.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.549 7ff6ed967700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6d00188d0 0x7ff6d001add0 crc :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:30.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.569 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e8182d10 con 0x7ff6e80b42d0 2021-04-09T18:32:30.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.572 7ff6daffd700 1 -- 172.21.15.32:0/4250066375 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff6e401f7a0 con 0x7ff6e80b42d0 2021-04-09T18:32:30.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.762 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff6e810d480 con 0x7ff6e80b42d0 2021-04-09T18:32:30.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.763 7ff6daffd700 1 -- 172.21.15.32:0/4250066375 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v217) v1 ==== 81+0+26248 (secure 0 0 0) 0x7ff6e401f7a0 con 0x7ff6e80b42d0 2021-04-09T18:32:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:30.780 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:32:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 217, 2021-04-09T18:32:30.780 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:32:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:32:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:32:27.666485", 2021-04-09T18:32:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:32:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:32:30.781 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:32:30.782 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:32:30.782 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:32:30.782 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:32:30.782 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:32:30.783 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:32:30.783 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:32:30.783 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:32:30.783 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:32:30.783 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:32:30.784 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:32:30.784 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:32:30.784 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:32:30.784 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 37, 2021-04-09T18:32:30.784 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:32:30.785 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:32:30.785 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:32:30.785 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:32:30.785 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:32:30.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.786 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:32:30.786 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:32:30.786 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:32:30.786 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:32:30.787 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:32:30.787 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:32:30.787 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:32:30.787 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:32:30.788 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:32:30.788 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:32:30.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:32:30.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:32:30.788 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:32:30.789 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:32:30.789 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:32:30.789 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:32:30.789 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:32:30.790 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:32:30.790 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:32:30.790 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:32:30.790 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:32:30.790 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:32:30.791 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:32:30.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.791 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:32:30.791 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:32:30.791 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:32:30.792 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:32:30.792 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:32:30.792 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:32:30.792 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:32:30.793 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:32:30.793 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:32:30.793 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:32:30.793 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:32:30.793 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:32:30.794 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:32:30.794 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:32:30.794 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:32:30.794 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:32:30.795 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:32:30.795 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:32:30.795 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:32:30.795 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:32:30.796 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:32:30.796 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:32:30.796 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:32:30.796 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:32:30.797 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:32:30.797 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:32:30.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:32:30.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.797 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:32:30.798 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:32:30.798 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:32:30.798 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:32:30.798 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:32:30.799 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:32:30.799 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:32:30.799 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:32:30.799 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:32:30.800 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:32:30.800 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:32:30.800 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:32:30.800 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:32:30.801 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:32:30.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.801 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:32:30.801 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:32:30.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.802 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:32:30.802 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:32:30.802 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:30.802 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:30.803 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:30.803 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:30.803 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:30.803 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:30.804 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:30.804 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:30.804 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:30.804 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:30.804 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:30.805 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.805 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.805 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:32:30.805 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:30.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.806 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.806 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:32:30.807 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:30.807 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.807 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.807 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:30.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.808 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.808 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:32:30.808 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:30.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.809 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:32:30.810 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:30.810 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.810 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.810 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:30.811 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.811 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.811 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:32:30.811 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:30.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.812 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.812 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.812 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:32:30.812 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:30.813 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.813 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.813 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:30.813 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.814 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.814 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:32:30.814 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:30.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.815 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.815 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:32:30.815 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:32:30.816 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.816 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.816 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:32:30.816 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:32:30.817 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:32:30.817 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:32:30.817 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:30.817 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:30.817 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:30.818 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:32:30.818 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:32:30.819 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:30.819 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:30.819 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:30.819 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:30.820 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:30.821 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:30.821 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:30.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.822 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:32:30.822 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:30.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.823 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:32:30.823 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:30.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.823 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.824 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:30.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:32:30.825 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:30.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:32:30.826 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:30.826 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.826 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.827 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:30.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:32:30.828 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:30.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.828 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:32:30.829 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:30.829 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.829 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.830 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:30.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:32:30.831 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:30.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.831 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:32:30.832 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:32:30.832 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.832 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.833 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:32:30.833 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:32:30.833 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:32:30.833 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:32:30.833 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:30.834 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:30.834 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:30.834 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.835 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:32:30.835 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:32:30.835 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:30.835 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:30.836 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:30.836 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:30.836 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:30.836 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:30.837 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:30.837 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:30.837 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:30.837 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:30.837 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:30.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:32:30.838 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:30.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.839 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:32:30.840 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:30.840 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.840 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.840 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:30.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:32:30.841 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:30.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.842 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.842 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:32:30.842 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:30.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.843 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.843 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:30.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:32:30.844 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:32:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:30.846 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.847 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.847 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:32:30.847 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:30.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:32:30.848 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:32:30.849 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.849 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.849 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:32:30.850 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:32:30.850 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:32:30.850 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:32:30.850 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:30.850 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:30.851 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:30.851 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.851 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.851 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:32:30.852 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:32:30.852 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:30.852 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:30.852 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:30.853 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:30.853 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:30.853 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:30.853 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:30.853 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:30.854 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:30.854 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:30.854 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:30.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:32:30.855 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:30.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.856 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:32:30.856 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:30.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.857 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.857 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:30.857 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.858 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:32:30.858 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:30.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.859 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.859 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:32:30.859 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:30.859 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.860 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.860 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:30.860 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.861 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.861 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.861 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:32:30.861 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:30.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.862 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:32:30.862 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:30.862 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.863 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.863 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:30.863 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.864 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:32:30.864 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:30.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.865 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:32:30.865 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:32:30.866 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.866 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.866 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:32:30.866 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:32:30.867 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:32:30.867 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:32:30.867 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:30.867 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:30.868 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:30.868 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.868 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:32:30.869 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:32:30.869 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:30.869 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:30.869 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:30.869 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:30.870 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:30.870 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:30.870 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:30.870 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:30.870 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:30.871 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:30.871 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:30.871 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.872 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:32:30.872 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:30.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.873 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.873 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:32:30.873 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:30.873 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.873 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.874 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:30.874 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.875 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.875 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:32:30.875 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:30.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.875 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:32:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.876 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:32:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.878 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:32:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.879 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:32:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:32:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:32:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.882 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:32:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:32:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:32:30.883 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:32:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:30.884 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:32:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:32:30.885 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:30.887 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.888 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:32:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.889 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:32:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.890 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.891 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:32:30.892 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:30.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.892 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.892 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.892 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:32:30.893 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:30.893 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.893 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.894 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:30.894 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.894 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.894 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:32:30.895 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:30.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.895 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:32:30.896 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:30.896 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.896 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.897 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:30.897 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.897 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.897 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:32:30.898 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:30.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.899 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:32:30.899 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:32:30.899 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.899 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.900 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:32:30.900 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:32:30.900 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:32:30.900 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:32:30.901 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:30.901 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:30.901 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:30.901 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.902 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:32:30.902 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:32:30.902 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:30.903 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:30.903 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:30.903 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:30.903 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:30.904 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:30.904 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:30.904 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 205, 2021-04-09T18:32:30.904 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:30.905 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:30.905 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:30.905 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.905 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:32:30.906 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:30.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.907 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:32:30.907 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:30.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.908 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:30.908 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:32:30.909 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:30.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.910 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.910 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:32:30.910 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:30.910 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.910 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.911 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:30.911 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.911 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.912 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:32:30.912 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:30.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.913 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.913 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:32:30.913 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:30.913 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.914 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.914 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:30.914 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.915 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.915 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:32:30.915 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:30.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.916 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:32:30.916 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:32:30.916 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.917 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.917 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:32:30.917 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:32:30.917 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:32:30.918 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:32:30.918 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:30.918 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:30.918 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:30.919 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.919 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:32:30.920 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:32:30.920 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:32:30.920 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:32:30.920 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:32:30.920 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:32:30.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:32:30.921 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:32:30.921 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:32:30.921 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 204, 2021-04-09T18:32:30.921 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:32:30.922 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:32:30.922 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:32:30.922 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.922 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.923 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:32:30.923 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:30.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.924 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:32:30.924 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:30.924 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.925 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.925 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:32:30.925 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.926 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:32:30.926 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:30.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.926 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.927 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:32:30.927 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:30.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.928 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.928 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:32:30.928 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.929 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.929 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:32:30.929 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:30.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.930 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.930 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.930 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:32:30.930 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:30.930 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.931 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.931 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:32:30.931 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:32:30.931 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.932 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:32:30.932 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:32:30.932 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:30.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:32:30.933 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:32:30.933 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:32:30.933 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.933 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:32:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:32:30.934 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:32:30.935 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:32:30.935 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:32:30.935 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:32:30.935 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:32:30.935 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:30.936 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.936 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:32:30.936 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:32:30.936 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.937 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:32:30.937 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:30.937 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:30.937 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:30.937 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:30.938 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:30.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.938 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.938 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:32:30.938 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:30.939 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:30.939 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:30.939 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:30.939 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:30.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.940 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:32:30.940 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:30.940 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:30.941 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:30.941 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:30.941 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:30.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.942 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:32:30.942 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:30.942 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:30.942 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:30.942 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:30.943 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:30.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.943 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:32:30.943 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:30.944 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:30.944 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:30.944 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:30.944 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:30.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.945 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:32:30.945 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:30.945 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:30.946 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:30.946 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:30.946 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:30.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.946 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.947 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:32:30.947 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:30.947 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:30.947 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:30.947 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:30.948 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:30.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:30.948 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:32:30.949 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:32:30.949 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:32:30.949 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:32:30.949 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:32:30.949 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:32:30.950 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.950 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:32:30.950 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:32:30.950 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:32:30.951 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:32:30.951 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:32:30.951 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:32:30.951 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/2540922065": "2021-04-10 18:32:18.986238", 2021-04-09T18:32:30.951 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/3816325339": "2021-04-10 18:32:20.246148", 2021-04-09T18:32:30.952 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/2540922065": "2021-04-10 18:32:18.986238", 2021-04-09T18:32:30.952 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/1650816476": "2021-04-10 18:32:18.986227", 2021-04-09T18:32:30.952 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/358744459": "2021-04-10 18:32:18.986207", 2021-04-09T18:32:30.952 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/358744459": "2021-04-10 18:32:18.986207", 2021-04-09T18:32:30.953 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/3816325339": "2021-04-10 18:32:20.246148", 2021-04-09T18:32:30.953 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6837/1650816476": "2021-04-10 18:32:18.986227", 2021-04-09T18:32:30.953 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/1129605860": "2021-04-10 18:31:38.197351", 2021-04-09T18:32:30.953 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/1129605860": "2021-04-10 18:31:38.197351", 2021-04-09T18:32:30.953 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6837/4017196651": "2021-04-10 18:31:38.192391", 2021-04-09T18:32:30.954 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/4017196651": "2021-04-10 18:31:38.192391" 2021-04-09T18:32:30.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.954 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:32:30.954 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:32:30.954 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:32:30.955 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:32:30.955 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:32:30.955 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:32:30.955 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:32:30.956 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:32:30.956 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:30.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:30.956 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:32:30.956 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:32:30.957 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:32:30.957 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:32:30.957 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:32:30.957 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:32:30.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6d00188d0 msgr2=0x7ff6d001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:30.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6d00188d0 0x7ff6d001add0 crc :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e80b42d0 msgr2=0x7ff6e8183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:30.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e80b42d0 0x7ff6e8183520 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 rx=0x7ff6e400b5e0 tx=0x7ff6e400b980).stop 2021-04-09T18:32:30.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 shutdown_connections 2021-04-09T18:32:30.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff6d00188d0 0x7ff6d001add0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6e8184d30 0x7ff6e8185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6e81ac090 0x7ff6e81ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 --2- 172.21.15.32:0/4250066375 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6e80b42d0 0x7ff6e8183520 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 >> 172.21.15.32:0/4250066375 conn(0x7ff6e819b890 msgr2=0x7ff6e81a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:30.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 shutdown_connections 2021-04-09T18:32:30.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:30.766 7ff6f034c700 1 -- 172.21.15.32:0/4250066375 wait complete. 2021-04-09T18:32:30.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:31.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.096 7fb07470d700 1 Processor -- start 2021-04-09T18:32:31.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.096 7fb07470d700 1 -- start start 2021-04-09T18:32:31.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb07470d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb06c0d6c80 0x7fb06c0d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb07470d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb06c0d2440 0x7fb06c10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb07470d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c0d7280 con 0x7fb06c0cffa0 2021-04-09T18:32:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb07470d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c0d7080 con 0x7fb06c0d6c80 2021-04-09T18:32:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb07470d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c0d91e0 con 0x7fb06c0d2440 2021-04-09T18:32:31.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb071d28700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb06c0d6c80 0x7fb06c0d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb072529700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb06c0cffa0 0x7fb06c0d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:38996/0 (socket says 172.21.15.32:38996) 2021-04-09T18:32:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb072529700 1 -- 172.21.15.32:0/2831317820 learned_addr learned my addr 172.21.15.32:0/2831317820 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb072d2a700 1 --2- 172.21.15.32:0/2831317820 >> v2:172.21.15.86:3300/0 conn(0x7fb06c0d2440 0x7fb06c10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb071d28700 1 -- 172.21.15.32:0/2831317820 >> v2:172.21.15.86:3300/0 conn(0x7fb06c0d2440 msgr2=0x7fb06c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:31.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb071d28700 1 --2- 172.21.15.32:0/2831317820 >> v2:172.21.15.86:3300/0 conn(0x7fb06c0d2440 0x7fb06c10cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb071d28700 1 -- 172.21.15.32:0/2831317820 >> v1:172.21.15.32:6789/0 conn(0x7fb06c0cffa0 legacy=0x7fb06c0d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb071d28700 1 -- 172.21.15.32:0/2831317820 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c0d2840 con 0x7fb06c0d6c80 2021-04-09T18:32:31.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb071527700 1 -- 172.21.15.32:0/2831317820 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781150405 0 0) 0x7fb068000ce0 con 0x7fb06c0cffa0 2021-04-09T18:32:31.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.097 7fb072d2a700 1 --2- 172.21.15.32:0/2831317820 >> v2:172.21.15.86:3300/0 conn(0x7fb06c0d2440 0x7fb06c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:32:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.098 7fb071d28700 1 --2- 172.21.15.32:0/2831317820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0d6c80 0x7fb06c0d7ff0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7fb05c006650 tx=0x7fb05c009a80).ready entity=mon.0 client_cookie=4369540ce275fcdb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.098 7fb071527700 1 -- 172.21.15.32:0/2831317820 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05c0009b0 con 0x7fb06c0d6c80 2021-04-09T18:32:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.098 7fb071527700 1 -- 172.21.15.32:0/2831317820 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c00c0c0 con 0x7fb06c0d6c80 2021-04-09T18:32:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.098 7fb071527700 1 -- 172.21.15.32:0/2831317820 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05c00c510 con 0x7fb06c0d6c80 2021-04-09T18:32:31.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.098 7fb07470d700 1 -- 172.21.15.32:0/2831317820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0d6c80 msgr2=0x7fb06c0d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.098 7fb07470d700 1 --2- 172.21.15.32:0/2831317820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0d6c80 0x7fb06c0d7ff0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7fb05c006650 tx=0x7fb05c009a80).stop 2021-04-09T18:32:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.099 7fb07470d700 1 -- 172.21.15.32:0/2831317820 shutdown_connections 2021-04-09T18:32:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.099 7fb07470d700 1 --2- 172.21.15.32:0/2831317820 >> v2:172.21.15.86:3300/0 conn(0x7fb06c0d2440 0x7fb06c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.099 7fb07470d700 1 --2- 172.21.15.32:0/2831317820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c0d6c80 0x7fb06c0d7ff0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.099 7fb07470d700 1 -- 172.21.15.32:0/2831317820 >> 172.21.15.32:0/2831317820 conn(0x7fb06c0ce5c0 msgr2=0x7fb06c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:31.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.099 7fb07470d700 1 -- 172.21.15.32:0/2831317820 shutdown_connections 2021-04-09T18:32:31.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.099 7fb07470d700 1 -- 172.21.15.32:0/2831317820 wait complete. 2021-04-09T18:32:31.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb07470d700 1 Processor -- start 2021-04-09T18:32:31.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb07470d700 1 -- start start 2021-04-09T18:32:31.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb07470d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb06c1929f0 0x7fb06c192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb07470d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c08ddc0 0x7fb06c08e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb07470d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb06c08e7e0 0x7fb06c08c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb07470d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c193410 con 0x7fb06c08ddc0 2021-04-09T18:32:31.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb07470d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0a9140 con 0x7fb06c08e7e0 2021-04-09T18:32:31.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb07470d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c08ce80 con 0x7fb06c1929f0 2021-04-09T18:32:31.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb071d28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c08ddc0 0x7fb06c08e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb071d28700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c08ddc0 0x7fb06c08e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57646/0 (socket says 172.21.15.32:57646) 2021-04-09T18:32:31.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.119 7fb071d28700 1 -- 172.21.15.32:0/2756422532 learned_addr learned my addr 172.21.15.32:0/2756422532 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:31.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb072d2a700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb06c08e7e0 0x7fb06c08c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb072529700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb06c1929f0 0x7fb06c192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb071d28700 1 -- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb06c1929f0 msgr2=0x7fb06c192df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb071d28700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb06c1929f0 0x7fb06c192df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb071d28700 1 -- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb06c08e7e0 msgr2=0x7fb06c08c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb071d28700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb06c08e7e0 0x7fb06c08c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb071d28700 1 -- 172.21.15.32:0/2756422532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c08d0a0 con 0x7fb06c08ddc0 2021-04-09T18:32:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb072d2a700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb06c08e7e0 0x7fb06c08c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:32:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb072529700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb06c1929f0 0x7fb06c192df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:32:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb071d28700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c08ddc0 0x7fb06c08e1c0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 rx=0x7fb05c009630 tx=0x7fb05c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:31.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb062ffd700 1 -- 172.21.15.32:0/2756422532 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05c0093a0 con 0x7fb06c08ddc0 2021-04-09T18:32:31.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb062ffd700 1 -- 172.21.15.32:0/2756422532 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c00bd70 con 0x7fb06c08ddc0 2021-04-09T18:32:31.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb07470d700 1 -- 172.21.15.32:0/2756422532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06c08d0a0 con 0x7fb06c08ddc0 2021-04-09T18:32:31.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb062ffd700 1 -- 172.21.15.32:0/2756422532 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05c005560 con 0x7fb06c08ddc0 2021-04-09T18:32:31.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.120 7fb07470d700 1 -- 172.21.15.32:0/2756422532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c08d0a0 con 0x7fb06c08ddc0 2021-04-09T18:32:31.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.121 7fb062ffd700 1 -- 172.21.15.32:0/2756422532 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb05c005700 con 0x7fb06c08ddc0 2021-04-09T18:32:31.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.121 7fb062ffd700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb054018930 0x7fb05401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.121 7fb062ffd700 1 -- 172.21.15.32:0/2756422532 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5273+0+0 (secure 0 0 0) 0x7fb05c0328a0 con 0x7fb06c08ddc0 2021-04-09T18:32:31.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.121 7fb072529700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb054018930 0x7fb05401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.121 7fb072529700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb054018930 0x7fb05401adc0 crc :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:31.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.141 7fb07470d700 1 -- 172.21.15.32:0/2756422532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c1b70c0 con 0x7fb06c08ddc0 2021-04-09T18:32:31.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.145 7fb062ffd700 1 -- 172.21.15.32:0/2756422532 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb05c0054f0 con 0x7fb06c08ddc0 2021-04-09T18:32:31.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.335 7fb07470d700 1 -- 172.21.15.32:0/2756422532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb06c0d7280 con 0x7fb06c08ddc0 2021-04-09T18:32:31.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.336 7fb062ffd700 1 -- 172.21.15.32:0/2756422532 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 376 v376) v1 ==== 95+0+1852 (secure 0 0 0) 0x7fb05c005700 con 0x7fb06c08ddc0 2021-04-09T18:32:31.352 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 376 2021-04-09T18:32:31.352 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:31.353 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":376,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11380,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3605741309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3605741309},{"type":"v1","addr":"172.21.15.86:6836","nonce":3605741309}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":375},{"gid":11395,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/915301141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":915301141},{"type":"v1","addr":"172.21.15.86:6837","nonce":915301141}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":375},{"gid":11883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3641032290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3641032290},{"type":"v1","addr":"172.21.15.32:6834","nonce":3641032290}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":375},{"gid":11895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/590896619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":590896619},{"type":"v1","addr":"172.21.15.32:6835","nonce":590896619}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":376}],"filesystems":[]} 2021-04-09T18:32:31.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 -- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb054018930 msgr2=0x7fb05401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb054018930 0x7fb05401adc0 crc :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 -- 172.21.15.32:0/2756422532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c08ddc0 msgr2=0x7fb06c08e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c08ddc0 0x7fb06c08e1c0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 rx=0x7fb05c009630 tx=0x7fb05c00b980).stop 2021-04-09T18:32:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 -- 172.21.15.32:0/2756422532 shutdown_connections 2021-04-09T18:32:31.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb054018930 0x7fb05401adc0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb06c08ddc0 0x7fb06c08e1c0 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb06c08e7e0 0x7fb06c08c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 --2- 172.21.15.32:0/2756422532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb06c1929f0 0x7fb06c192df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 -- 172.21.15.32:0/2756422532 >> 172.21.15.32:0/2756422532 conn(0x7fb06c189b00 msgr2=0x7fb06c1921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:31.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 -- 172.21.15.32:0/2756422532 shutdown_connections 2021-04-09T18:32:31.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.338 7fb07470d700 1 -- 172.21.15.32:0/2756422532 wait complete. 2021-04-09T18:32:31.364 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:31.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.497 7f6a4b50d700 1 Processor -- start 2021-04-09T18:32:31.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.497 7f6a4b50d700 1 -- start start 2021-04-09T18:32:31.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.497 7f6a4b50d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a4410d010 0x7f6a440d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.497 7f6a4b50d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a440d8490 0x7f6a4410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.497 7f6a4b50d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4410d480 con 0x7f6a440d1da0 2021-04-09T18:32:31.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.497 7f6a4b50d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a440d8d30 con 0x7f6a4410d010 2021-04-09T18:32:31.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.497 7f6a4b50d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a440d9220 con 0x7f6a440d8490 2021-04-09T18:32:31.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a48b28700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6a440d1da0 0x7f6a440d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39012/0 (socket says 172.21.15.32:39012) 2021-04-09T18:32:31.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a48b28700 1 -- 172.21.15.32:0/4192580480 learned_addr learned my addr 172.21.15.32:0/4192580480 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:31.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a49b2a700 1 --2- 172.21.15.32:0/4192580480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a440d8490 0x7f6a4410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a49329700 1 --2- 172.21.15.32:0/4192580480 >> v2:172.21.15.86:3300/0 conn(0x7f6a4410d010 0x7f6a440d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a3bfff700 1 -- 172.21.15.32:0/4192580480 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158700850 0 0) 0x7f6a34000ce0 con 0x7f6a440d1da0 2021-04-09T18:32:31.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a3bfff700 1 -- 172.21.15.32:0/4192580480 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a30002fe0 con 0x7f6a440d1da0 2021-04-09T18:32:31.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a3bfff700 1 -- 172.21.15.32:0/4192580480 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3155206630 0 0) 0x7f6a34000ed0 con 0x7f6a440d1da0 2021-04-09T18:32:31.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a49329700 1 -- 172.21.15.32:0/4192580480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a440d8490 msgr2=0x7f6a4410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:31.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a49329700 1 --2- 172.21.15.32:0/4192580480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a440d8490 0x7f6a4410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a49329700 1 -- 172.21.15.32:0/4192580480 >> v1:172.21.15.32:6789/0 conn(0x7f6a440d1da0 legacy=0x7f6a440d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a49329700 1 -- 172.21.15.32:0/4192580480 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4410e020 con 0x7f6a4410d010 2021-04-09T18:32:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a49b2a700 1 --2- 172.21.15.32:0/4192580480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a440d8490 0x7f6a4410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:31.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a49329700 1 --2- 172.21.15.32:0/4192580480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4410d010 0x7f6a440d1780 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f6a40006650 tx=0x7f6a40009a80).ready entity=mon.1 client_cookie=223fef839e0c2cf1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:31.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a3bfff700 1 -- 172.21.15.32:0/4192580480 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a400009b0 con 0x7f6a4410d010 2021-04-09T18:32:31.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.498 7f6a3bfff700 1 -- 172.21.15.32:0/4192580480 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a40006740 con 0x7f6a4410d010 2021-04-09T18:32:31.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.499 7f6a3bfff700 1 -- 172.21.15.32:0/4192580480 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a4000c0c0 con 0x7f6a4410d010 2021-04-09T18:32:31.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.499 7f6a4b50d700 1 -- 172.21.15.32:0/4192580480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4410d010 msgr2=0x7f6a440d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.499 7f6a4b50d700 1 --2- 172.21.15.32:0/4192580480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4410d010 0x7f6a440d1780 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f6a40006650 tx=0x7f6a40009a80).stop 2021-04-09T18:32:31.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.499 7f6a4b50d700 1 -- 172.21.15.32:0/4192580480 shutdown_connections 2021-04-09T18:32:31.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.499 7f6a4b50d700 1 --2- 172.21.15.32:0/4192580480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a440d8490 0x7f6a4410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.499 7f6a4b50d700 1 --2- 172.21.15.32:0/4192580480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a4410d010 0x7f6a440d1780 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.499 7f6a4b50d700 1 -- 172.21.15.32:0/4192580480 >> 172.21.15.32:0/4192580480 conn(0x7f6a440ce5f0 msgr2=0x7f6a440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:31.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.499 7f6a4b50d700 1 -- 172.21.15.32:0/4192580480 shutdown_connections 2021-04-09T18:32:31.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.499 7f6a4b50d700 1 -- 172.21.15.32:0/4192580480 wait complete. 2021-04-09T18:32:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.519 7f6a4b50d700 1 Processor -- start 2021-04-09T18:32:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.519 7f6a4b50d700 1 -- start start 2021-04-09T18:32:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a4b50d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a441a91e0 0x7f6a440a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a4b50d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a441a95e0 0x7f6a440a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a4b50d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a441ac850 0x7f6a441acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a4b50d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a441ad270 con 0x7f6a441a91e0 2021-04-09T18:32:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a4b50d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a440b7640 con 0x7f6a441ac850 2021-04-09T18:32:31.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a4b50d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a44183c80 con 0x7f6a441a95e0 2021-04-09T18:32:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a49329700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a441a91e0 0x7f6a440a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a49329700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a441a91e0 0x7f6a440a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57658/0 (socket says 172.21.15.32:57658) 2021-04-09T18:32:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a49329700 1 -- 172.21.15.32:0/278678547 learned_addr learned my addr 172.21.15.32:0/278678547 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:31.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a48b28700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a441a95e0 0x7f6a440a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.520 7f6a49b2a700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a441ac850 0x7f6a441acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a49329700 1 -- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a441a95e0 msgr2=0x7f6a440a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a49329700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a441a95e0 0x7f6a440a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a49329700 1 -- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a441ac850 msgr2=0x7f6a441acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a49329700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a441ac850 0x7f6a441acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a49329700 1 -- 172.21.15.32:0/278678547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a44183ea0 con 0x7f6a441a91e0 2021-04-09T18:32:31.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a49329700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a441a91e0 0x7f6a440a4a60 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 rx=0x7f6a40009630 tx=0x7f6a4000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:31.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a39ffb700 1 -- 172.21.15.32:0/278678547 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a400093a0 con 0x7f6a441a91e0 2021-04-09T18:32:31.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a39ffb700 1 -- 172.21.15.32:0/278678547 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4000bd70 con 0x7f6a441a91e0 2021-04-09T18:32:31.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a44183ea0 con 0x7f6a441a91e0 2021-04-09T18:32:31.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a39ffb700 1 -- 172.21.15.32:0/278678547 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a400054d0 con 0x7f6a441a91e0 2021-04-09T18:32:31.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.521 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a44183ea0 con 0x7f6a441a91e0 2021-04-09T18:32:31.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.522 7f6a39ffb700 1 -- 172.21.15.32:0/278678547 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6a400054f0 con 0x7f6a441a91e0 2021-04-09T18:32:31.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.522 7f6a39ffb700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a240188c0 0x7f6a2401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:31.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.522 7f6a39ffb700 1 -- 172.21.15.32:0/278678547 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f6a40032680 con 0x7f6a441a91e0 2021-04-09T18:32:31.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.522 7f6a48b28700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a240188c0 0x7f6a2401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:31.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.523 7f6a48b28700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a240188c0 0x7f6a2401adc0 crc :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:31.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.541 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a44182d70 con 0x7f6a441a91e0 2021-04-09T18:32:31.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.545 7f6a39ffb700 1 -- 172.21.15.32:0/278678547 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6a400052e0 con 0x7f6a441a91e0 2021-04-09T18:32:31.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.736 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a4410d480 con 0x7f6a441a91e0 2021-04-09T18:32:31.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.737 7f6a39ffb700 1 -- 172.21.15.32:0/278678547 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 376 v376) v1 ==== 95+0+1852 (secure 0 0 0) 0x7f6a4000bd70 con 0x7f6a441a91e0 2021-04-09T18:32:31.754 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 376 2021-04-09T18:32:31.754 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:31.755 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":376,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11380,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3605741309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3605741309},{"type":"v1","addr":"172.21.15.86:6836","nonce":3605741309}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":375},{"gid":11395,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/915301141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":915301141},{"type":"v1","addr":"172.21.15.86:6837","nonce":915301141}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":375},{"gid":11883,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3641032290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3641032290},{"type":"v1","addr":"172.21.15.32:6834","nonce":3641032290}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":375},{"gid":11895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/590896619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":590896619},{"type":"v1","addr":"172.21.15.32:6835","nonce":590896619}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":376}],"filesystems":[]} 2021-04-09T18:32:31.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.739 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a240188c0 msgr2=0x7f6a2401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.739 7f6a4b50d700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a240188c0 0x7f6a2401adc0 crc :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.739 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a441a91e0 msgr2=0x7f6a440a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:31.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.739 7f6a4b50d700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a441a91e0 0x7f6a440a4a60 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 rx=0x7f6a40009630 tx=0x7f6a4000b980).stop 2021-04-09T18:32:31.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.740 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 shutdown_connections 2021-04-09T18:32:31.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.740 7f6a4b50d700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6a240188c0 0x7f6a2401adc0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.740 7f6a4b50d700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a441ac850 0x7f6a441acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.740 7f6a4b50d700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a441a95e0 0x7f6a440a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.740 7f6a4b50d700 1 --2- 172.21.15.32:0/278678547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a441a91e0 0x7f6a440a4a60 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:31.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.740 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 >> 172.21.15.32:0/278678547 conn(0x7f6a4419b460 msgr2=0x7f6a4419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.740 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 shutdown_connections 2021-04-09T18:32:31.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.740 7f6a4b50d700 1 -- 172.21.15.32:0/278678547 wait complete. 2021-04-09T18:32:31.766 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:32:31.766 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:32:31.767 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:32:31.767 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:32:31.768 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:32:31.768 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:32:31.769 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:32:31.769 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:32:31.769 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:32:31.770 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:32:31.770 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:32:31.771 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:32:31.771 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:32:31.771 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:32:31.772 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:32:31.773 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:32:31.775 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:32:31.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:32:31.777 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:32:31.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:32:31.780 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:32:31.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:32:31.782 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:32:31.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:32:31.819 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:32:31.819 DEBUG:teuthology.parallel:result is None 2021-04-09T18:32:31.821 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:32:31.821 DEBUG:teuthology.parallel:result is None 2021-04-09T18:32:31.822 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:32:31.822 DEBUG:teuthology.parallel:result is None 2021-04-09T18:32:31.823 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:32:31.824 DEBUG:teuthology.parallel:result is None 2021-04-09T18:32:31.824 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:32:31.966 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:32:31.967 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:32:31.987 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:32:31.988 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:32:32.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.963 7fef820f4700 1 Processor -- start 2021-04-09T18:32:32.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.963 7fef820f4700 1 -- start start 2021-04-09T18:32:32.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.963 7fef820f4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fef7c0cffb0 0x7fef7c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.963 7fef820f4700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c0d7290 con 0x7fef7c0d7810 2021-04-09T18:32:32.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.963 7fef820f4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c0d8400 con 0x7fef7c0d6c90 2021-04-09T18:32:32.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.963 7fef820f4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c0d7090 con 0x7fef7c0cffb0 2021-04-09T18:32:32.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.963 7fef7affd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fef7c0d7810 0x7fef7c0d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39038/0 (socket says 172.21.15.32:39038) 2021-04-09T18:32:32.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.963 7fef7affd700 1 -- 172.21.15.32:0/3711329211 learned_addr learned my addr 172.21.15.32:0/3711329211 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:32.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.963 7fef7b7fe700 1 --2- 172.21.15.32:0/3711329211 >> v2:172.21.15.86:3300/0 conn(0x7fef7c0cffb0 0x7fef7c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7a7fc700 1 -- 172.21.15.32:0/3711329211 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915116933 0 0) 0x7fef6c000ce0 con 0x7fef7c0d7810 2021-04-09T18:32:32.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7a7fc700 1 -- 172.21.15.32:0/3711329211 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef68002fe0 con 0x7fef7c0d7810 2021-04-09T18:32:32.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7a7fc700 1 -- 172.21.15.32:0/3711329211 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853206123 0 0) 0x7fef74000d30 con 0x7fef7c0d6c90 2021-04-09T18:32:32.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7a7fc700 1 -- 172.21.15.32:0/3711329211 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef680042c0 con 0x7fef7c0d6c90 2021-04-09T18:32:32.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7b7fe700 1 -- 172.21.15.32:0/3711329211 >> v1:172.21.15.86:6789/0 conn(0x7fef7c0d6c90 legacy=0x7fef7c10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7b7fe700 1 -- 172.21.15.32:0/3711329211 >> v1:172.21.15.32:6789/0 conn(0x7fef7c0d7810 legacy=0x7fef7c0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7b7fe700 1 -- 172.21.15.32:0/3711329211 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c0d91f0 con 0x7fef7c0cffb0 2021-04-09T18:32:32.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7b7fe700 1 --2- 172.21.15.32:0/3711329211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef7c0cffb0 0x7fef7c0d2450 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7fef64006650 tx=0x7fef64009a80).ready entity=mon.1 client_cookie=3d5f8cf122bf1490 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:32.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7a7fc700 1 -- 172.21.15.32:0/3711329211 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef640009b0 con 0x7fef7c0cffb0 2021-04-09T18:32:32.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7a7fc700 1 -- 172.21.15.32:0/3711329211 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6400c0c0 con 0x7fef7c0cffb0 2021-04-09T18:32:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef7a7fc700 1 -- 172.21.15.32:0/3711329211 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef6400c510 con 0x7fef7c0cffb0 2021-04-09T18:32:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef820f4700 1 -- 172.21.15.32:0/3711329211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef7c0cffb0 msgr2=0x7fef7c0d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.964 7fef820f4700 1 --2- 172.21.15.32:0/3711329211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef7c0cffb0 0x7fef7c0d2450 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7fef64006650 tx=0x7fef64009a80).stop 2021-04-09T18:32:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.965 7fef820f4700 1 -- 172.21.15.32:0/3711329211 shutdown_connections 2021-04-09T18:32:32.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.965 7fef820f4700 1 --2- 172.21.15.32:0/3711329211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef7c0cffb0 0x7fef7c0d2450 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.965 7fef820f4700 1 -- 172.21.15.32:0/3711329211 >> 172.21.15.32:0/3711329211 conn(0x7fef7c0ce5d0 msgr2=0x7fef7c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:32.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.965 7fef820f4700 1 -- 172.21.15.32:0/3711329211 shutdown_connections 2021-04-09T18:32:32.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.965 7fef820f4700 1 -- 172.21.15.32:0/3711329211 wait complete. 2021-04-09T18:32:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.986 7fef820f4700 1 Processor -- start 2021-04-09T18:32:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.986 7fef820f4700 1 -- start start 2021-04-09T18:32:32.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef820f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef7c1b8380 0x7fef7c1b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef820f4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef7c08c7b0 0x7fef7c08cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef820f4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef7c08d1d0 0x7fef7c1805b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef820f4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c181a30 con 0x7fef7c08d1d0 2021-04-09T18:32:32.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef820f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c08c230 con 0x7fef7c1b8380 2021-04-09T18:32:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef820f4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c1b8da0 con 0x7fef7c08c7b0 2021-04-09T18:32:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef7c08d1d0 0x7fef7c1805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef7c08d1d0 0x7fef7c1805b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57700/0 (socket says 172.21.15.32:57700) 2021-04-09T18:32:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7bfff700 1 -- 172.21.15.32:0/2635117413 learned_addr learned my addr 172.21.15.32:0/2635117413 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:32.012 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7b7fe700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef7c1b8380 0x7fef7c1b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7affd700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef7c08c7b0 0x7fef7c08cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7bfff700 1 -- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef7c08c7b0 msgr2=0x7fef7c08cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7bfff700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef7c08c7b0 0x7fef7c08cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7bfff700 1 -- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef7c1b8380 msgr2=0x7fef7c1b8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7bfff700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef7c1b8380 0x7fef7c1b8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7bfff700 1 -- 172.21.15.32:0/2635117413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c1b8f90 con 0x7fef7c08d1d0 2021-04-09T18:32:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.987 7fef7bfff700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef7c08d1d0 0x7fef7c1805b0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7fef7400b8c0 tx=0x7fef7400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.988 7fef73fff700 1 -- 172.21.15.32:0/2635117413 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef74005870 con 0x7fef7c08d1d0 2021-04-09T18:32:32.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.988 7fef73fff700 1 -- 172.21.15.32:0/2635117413 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef74005b70 con 0x7fef7c08d1d0 2021-04-09T18:32:32.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.988 7fef73fff700 1 -- 172.21.15.32:0/2635117413 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef74006030 con 0x7fef7c08d1d0 2021-04-09T18:32:32.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.988 7fef820f4700 1 -- 172.21.15.32:0/2635117413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c1b8f90 con 0x7fef7c08d1d0 2021-04-09T18:32:32.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.988 7fef820f4700 1 -- 172.21.15.32:0/2635117413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c1b8f90 con 0x7fef7c08d1d0 2021-04-09T18:32:32.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.988 7fef73fff700 1 -- 172.21.15.32:0/2635117413 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fef74006030 con 0x7fef7c08d1d0 2021-04-09T18:32:32.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.988 7fef73fff700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef60018930 0x7fef6001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.988 7fef73fff700 1 -- 172.21.15.32:0/2635117413 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5273+0+0 (secure 0 0 0) 0x7fef74032ca0 con 0x7fef7c08d1d0 2021-04-09T18:32:32.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.989 7fef7b7fe700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef60018930 0x7fef6001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:31.989 7fef7b7fe700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef60018930 0x7fef6001ae10 crc :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:32.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.007 7fef820f4700 1 -- 172.21.15.32:0/2635117413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c1b7180 con 0x7fef7c08d1d0 2021-04-09T18:32:32.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.011 7fef73fff700 1 -- 172.21.15.32:0/2635117413 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fef7400bda0 con 0x7fef7c08d1d0 2021-04-09T18:32:32.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.215 7fef820f4700 1 -- 172.21.15.32:0/2635117413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fef7c08c230 con 0x7fef7c08d1d0 2021-04-09T18:32:32.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.681 7fef73fff700 1 -- 172.21.15.32:0/2635117413 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v218) v1 ==== 97+0+0 (secure 0 0 0) 0x7fef7400bda0 con 0x7fef7c08d1d0 2021-04-09T18:32:32.745 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:32:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 -- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef60018930 msgr2=0x7fef6001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef60018930 0x7fef6001ae10 crc :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 -- 172.21.15.32:0/2635117413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef7c08d1d0 msgr2=0x7fef7c1805b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef7c08d1d0 0x7fef7c1805b0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7fef7400b8c0 tx=0x7fef7400bc60).stop 2021-04-09T18:32:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 -- 172.21.15.32:0/2635117413 shutdown_connections 2021-04-09T18:32:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fef60018930 0x7fef6001ae10 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef7c08d1d0 0x7fef7c1805b0 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef7c08c7b0 0x7fef7c08cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 --2- 172.21.15.32:0/2635117413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef7c1b8380 0x7fef7c1b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 -- 172.21.15.32:0/2635117413 >> 172.21.15.32:0/2635117413 conn(0x7fef7c189800 msgr2=0x7fef7c1b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:32.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 -- 172.21.15.32:0/2635117413 shutdown_connections 2021-04-09T18:32:32.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.732 7fef820f4700 1 -- 172.21.15.32:0/2635117413 wait complete. 2021-04-09T18:32:32.758 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:32:32.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:32:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.894 7f637fa0e700 1 Processor -- start 2021-04-09T18:32:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.894 7f637fa0e700 1 -- start start 2021-04-09T18:32:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637fa0e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f637810d010 0x7f63780d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637fa0e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63780d8490 0x7f637810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637fa0e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637810d480 con 0x7f63780d1da0 2021-04-09T18:32:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637fa0e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63780d8d30 con 0x7f637810d010 2021-04-09T18:32:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637fa0e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63780d9220 con 0x7f63780d8490 2021-04-09T18:32:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637e02b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63780d8490 0x7f637810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637d82a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f637810d010 0x7f63780d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637d029700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f63780d1da0 0x7f63780d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:36990/0 (socket says 172.21.15.32:36990) 2021-04-09T18:32:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637d029700 1 -- 172.21.15.32:0/1151592351 learned_addr learned my addr 172.21.15.32:0/1151592351 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637c828700 1 -- 172.21.15.32:0/1151592351 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222431518 0 0) 0x7f63680010c0 con 0x7f63780d1da0 2021-04-09T18:32:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.895 7f637c828700 1 -- 172.21.15.32:0/1151592351 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6364002fe0 con 0x7f63780d1da0 2021-04-09T18:32:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637e02b700 1 -- 172.21.15.32:0/1151592351 >> v2:172.21.15.86:3300/0 conn(0x7f637810d010 msgr2=0x7f63780d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:32.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637e02b700 1 --2- 172.21.15.32:0/1151592351 >> v2:172.21.15.86:3300/0 conn(0x7f637810d010 0x7f63780d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637e02b700 1 -- 172.21.15.32:0/1151592351 >> v1:172.21.15.86:6789/0 conn(0x7f63780d1da0 legacy=0x7f63780d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637e02b700 1 -- 172.21.15.32:0/1151592351 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637810e020 con 0x7f63780d8490 2021-04-09T18:32:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637d82a700 1 --2- 172.21.15.32:0/1151592351 >> v2:172.21.15.86:3300/0 conn(0x7f637810d010 0x7f63780d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637e02b700 1 --2- 172.21.15.32:0/1151592351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63780d8490 0x7f637810da00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f6370006e50 tx=0x7f6370009e70).ready entity=mon.2 client_cookie=a00f7da331c76b7d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:32.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637c828700 1 -- 172.21.15.32:0/1151592351 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6370006a20 con 0x7f63780d8490 2021-04-09T18:32:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637c828700 1 -- 172.21.15.32:0/1151592351 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6370006790 con 0x7f63780d8490 2021-04-09T18:32:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637c828700 1 -- 172.21.15.32:0/1151592351 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637000b700 con 0x7f63780d8490 2021-04-09T18:32:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637fa0e700 1 -- 172.21.15.32:0/1151592351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63780d8490 msgr2=0x7f637810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.896 7f637fa0e700 1 --2- 172.21.15.32:0/1151592351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63780d8490 0x7f637810da00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f6370006e50 tx=0x7f6370009e70).stop 2021-04-09T18:32:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.897 7f637fa0e700 1 -- 172.21.15.32:0/1151592351 shutdown_connections 2021-04-09T18:32:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.897 7f637fa0e700 1 --2- 172.21.15.32:0/1151592351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63780d8490 0x7f637810da00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.897 7f637fa0e700 1 --2- 172.21.15.32:0/1151592351 >> v2:172.21.15.86:3300/0 conn(0x7f637810d010 0x7f63780d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.897 7f637fa0e700 1 -- 172.21.15.32:0/1151592351 >> 172.21.15.32:0/1151592351 conn(0x7f63780ce5f0 msgr2=0x7f63780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:32.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.897 7f637fa0e700 1 -- 172.21.15.32:0/1151592351 shutdown_connections 2021-04-09T18:32:32.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.897 7f637fa0e700 1 -- 172.21.15.32:0/1151592351 wait complete. 2021-04-09T18:32:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.917 7f637fa0e700 1 Processor -- start 2021-04-09T18:32:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.917 7f637fa0e700 1 -- start start 2021-04-09T18:32:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.917 7f637fa0e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63781aca20 0x7f63781ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.917 7f637fa0e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f637818bd40 0x7f637818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.917 7f637fa0e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637808fba0 0x7f637808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.917 7f637fa0e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637818c7e0 con 0x7f63781aca20 2021-04-09T18:32:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.917 7f637fa0e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637818ca30 con 0x7f637808fba0 2021-04-09T18:32:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.917 7f637fa0e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63780905f0 con 0x7f637818bd40 2021-04-09T18:32:32.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d82a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63781aca20 0x7f63781ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d029700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f637818bd40 0x7f637818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637e02b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637808fba0 0x7f637808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d82a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63781aca20 0x7f63781ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57710/0 (socket says 172.21.15.32:57710) 2021-04-09T18:32:32.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d82a700 1 -- 172.21.15.32:0/1795760801 learned_addr learned my addr 172.21.15.32:0/1795760801 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d82a700 1 -- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f637818bd40 msgr2=0x7f637818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d82a700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f637818bd40 0x7f637818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d82a700 1 -- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637808fba0 msgr2=0x7f637808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d82a700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637808fba0 0x7f637808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:32.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d82a700 1 -- 172.21.15.32:0/1795760801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63780b4300 con 0x7f63781aca20 2021-04-09T18:32:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637e02b700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637808fba0 0x7f637808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637d82a700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63781aca20 0x7f63781ace20 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 rx=0x7f637400b580 tx=0x7f637400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f636e7fc700 1 -- 172.21.15.32:0/1795760801 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637400bd10 con 0x7f63781aca20 2021-04-09T18:32:32.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f636e7fc700 1 -- 172.21.15.32:0/1795760801 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6374000c90 con 0x7f63781aca20 2021-04-09T18:32:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f636e7fc700 1 -- 172.21.15.32:0/1795760801 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6374002220 con 0x7f63781aca20 2021-04-09T18:32:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63780b4300 con 0x7f63781aca20 2021-04-09T18:32:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.918 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63780b4300 con 0x7f63781aca20 2021-04-09T18:32:32.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.919 7f636e7fc700 1 -- 172.21.15.32:0/1795760801 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6374006b00 con 0x7f63781aca20 2021-04-09T18:32:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.919 7f636e7fc700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63600188d0 0x7f636001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.919 7f636e7fc700 1 -- 172.21.15.32:0/1795760801 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f63740326c0 con 0x7f63781aca20 2021-04-09T18:32:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.919 7f637d029700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63600188d0 0x7f636001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:32.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.919 7f637d029700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63600188d0 0x7f636001add0 crc :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:32.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.939 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6378182d70 con 0x7f63781aca20 2021-04-09T18:32:32.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:32.942 7f636e7fc700 1 -- 172.21.15.32:0/1795760801 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f637400bd10 con 0x7f63781aca20 2021-04-09T18:32:33.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.128 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f637818c760 con 0x7f63781aca20 2021-04-09T18:32:33.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.128 7f636e7fc700 1 -- 172.21.15.32:0/1795760801 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v44) v1 ==== 80+0+4589 (secure 0 0 0) 0x7f6374000c90 con 0x7f63781aca20 2021-04-09T18:32:33.144 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:33.144 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:32:33.145 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:32:33.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.145 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:32:33.145 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:32:33.146 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.146 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:32:33.146 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.146 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.147 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.147 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.147 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.148 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:32:33.148 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:32:33.148 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.148 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:32:33.149 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.149 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.149 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.150 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:32:33.150 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:32:33.150 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.151 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:32:33.151 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.151 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.151 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.152 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.152 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.152 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:32:33.153 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:32:33.153 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.153 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:32:33.153 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.154 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.154 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.155 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.155 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:32:33.155 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:32:33.155 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.156 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.156 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.156 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.156 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.157 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:32:33.157 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:32:33.157 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.158 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.158 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.158 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.158 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.159 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:32:33.159 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:32:33.159 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.160 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.160 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.160 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.160 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.161 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:32:33.161 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:32:33.161 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.162 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.162 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.162 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.162 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.163 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.163 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:32:33.163 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:32:33.163 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.163 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.164 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.164 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.165 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.165 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:32:33.165 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:32:33.165 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.165 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.166 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.166 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.166 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.167 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.167 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:32:33.167 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:32:33.167 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.168 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.168 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.168 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.168 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.169 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:32:33.169 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:32:33.169 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.169 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.170 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.170 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.170 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.171 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:32:33.171 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:32:33.171 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.171 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:32:33.172 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:32:33.172 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=alpha_data, allow rw pool=bravo_data" 2021-04-09T18:32:33.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.173 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.173 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:32:33.173 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:32:33.173 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:32:33.174 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:32:33.174 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=alpha_data, allow rw pool=bravo_data" 2021-04-09T18:32:33.174 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.175 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.175 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:32:33.175 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:32:33.176 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:32:33.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:32:33.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:32:33.177 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.177 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.177 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.178 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:32:33.178 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:32:33.178 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:32:33.178 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:32:33.179 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.179 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:33.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:33.180 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:32:33.180 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:32:33.180 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:32:33.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:32:33.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:32:33.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:32:33.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:33.181 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:32:33.182 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:32:33.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63600188d0 msgr2=0x7f636001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63600188d0 0x7f636001add0 crc :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63781aca20 msgr2=0x7f63781ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63781aca20 0x7f63781ace20 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 rx=0x7f637400b580 tx=0x7f637400b920).stop 2021-04-09T18:32:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 shutdown_connections 2021-04-09T18:32:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63600188d0 0x7f636001add0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f637808fba0 0x7f637808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f637818bd40 0x7f637818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 --2- 172.21.15.32:0/1795760801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63781aca20 0x7f63781ace20 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 >> 172.21.15.32:0/1795760801 conn(0x7f637819b460 msgr2=0x7f637819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:33.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 shutdown_connections 2021-04-09T18:32:33.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.131 7f637fa0e700 1 -- 172.21.15.32:0/1795760801 wait complete. 2021-04-09T18:32:33.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-04-09T18:32:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.331 7f27b1e1a700 1 Processor -- start 2021-04-09T18:32:33.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27b1e1a700 1 -- start start 2021-04-09T18:32:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27b1e1a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f27ac10d050 0x7f27ac0cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27b1e1a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27ac0d84b0 0x7f27ac10da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27b1e1a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ac10d500 con 0x7f27ac0d0160 2021-04-09T18:32:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27b1e1a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27ac0d8d70 con 0x7f27ac0d84b0 2021-04-09T18:32:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27b1e1a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27ac0d9260 con 0x7f27ac10d050 2021-04-09T18:32:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27abfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27ac0d84b0 0x7f27ac10da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27abfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f27ac0d84b0 0x7f27ac10da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57720/0 (socket says 172.21.15.32:57720) 2021-04-09T18:32:33.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27abfff700 1 -- 172.21.15.32:0/2443124769 learned_addr learned my addr 172.21.15.32:0/2443124769 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.332 7f27ab7fe700 1 --2- 172.21.15.32:0/2443124769 >> v2:172.21.15.86:3300/0 conn(0x7f27ac10d050 0x7f27ac0cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27abfff700 1 -- 172.21.15.32:0/2443124769 >> v2:172.21.15.86:3300/0 conn(0x7f27ac10d050 msgr2=0x7f27ac0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27abfff700 1 --2- 172.21.15.32:0/2443124769 >> v2:172.21.15.86:3300/0 conn(0x7f27ac10d050 0x7f27ac0cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27abfff700 1 -- 172.21.15.32:0/2443124769 >> v1:172.21.15.86:6789/0 conn(0x7f27ac0d0160 legacy=0x7f27ac0d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27abfff700 1 -- 172.21.15.32:0/2443124769 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ac10e0a0 con 0x7f27ac0d84b0 2021-04-09T18:32:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27abfff700 1 --2- 172.21.15.32:0/2443124769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac0d84b0 0x7f27ac10da80 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 rx=0x7f27a4006f00 tx=0x7f27a4009f10).ready entity=mon.0 client_cookie=b9c3cbbb63dca270 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27aa7fc700 1 -- 172.21.15.32:0/2443124769 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a400b3b0 con 0x7f27ac0d84b0 2021-04-09T18:32:33.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27aa7fc700 1 -- 172.21.15.32:0/2443124769 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a400b730 con 0x7f27ac0d84b0 2021-04-09T18:32:33.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27aa7fc700 1 -- 172.21.15.32:0/2443124769 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a400bc10 con 0x7f27ac0d84b0 2021-04-09T18:32:33.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27b1e1a700 1 -- 172.21.15.32:0/2443124769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac0d84b0 msgr2=0x7f27ac10da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.333 7f27b1e1a700 1 --2- 172.21.15.32:0/2443124769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac0d84b0 0x7f27ac10da80 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 rx=0x7f27a4006f00 tx=0x7f27a4009f10).stop 2021-04-09T18:32:33.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.334 7f27b1e1a700 1 -- 172.21.15.32:0/2443124769 shutdown_connections 2021-04-09T18:32:33.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.334 7f27b1e1a700 1 --2- 172.21.15.32:0/2443124769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac0d84b0 0x7f27ac10da80 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.334 7f27b1e1a700 1 --2- 172.21.15.32:0/2443124769 >> v2:172.21.15.86:3300/0 conn(0x7f27ac10d050 0x7f27ac0cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.334 7f27b1e1a700 1 -- 172.21.15.32:0/2443124769 >> 172.21.15.32:0/2443124769 conn(0x7f27ac0ce620 msgr2=0x7f27ac0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:33.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.334 7f27b1e1a700 1 -- 172.21.15.32:0/2443124769 shutdown_connections 2021-04-09T18:32:33.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.334 7f27b1e1a700 1 -- 172.21.15.32:0/2443124769 wait complete. 2021-04-09T18:32:33.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.354 7f27b1e1a700 1 Processor -- start 2021-04-09T18:32:33.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.354 7f27b1e1a700 1 -- start start 2021-04-09T18:32:33.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27b1e1a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27ac09b2f0 0x7f27ac09b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27b1e1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac09a8b0 0x7f27ac09e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27b1e1a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27ac09e9c0 0x7f27ac09edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27b1e1a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac0bde80 con 0x7f27ac09a8b0 2021-04-09T18:32:33.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27b1e1a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac09ad70 con 0x7f27ac09e9c0 2021-04-09T18:32:33.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27b1e1a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac09bd10 con 0x7f27ac09b2f0 2021-04-09T18:32:33.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac09a8b0 0x7f27ac09e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27aaffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac09a8b0 0x7f27ac09e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57726/0 (socket says 172.21.15.32:57726) 2021-04-09T18:32:33.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27aaffd700 1 -- 172.21.15.32:0/3574350429 learned_addr learned my addr 172.21.15.32:0/3574350429 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:33.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27ab7fe700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27ac09b2f0 0x7f27ac09b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27abfff700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27ac09e9c0 0x7f27ac09edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27aaffd700 1 -- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27ac09b2f0 msgr2=0x7f27ac09b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27aaffd700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27ac09b2f0 0x7f27ac09b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27aaffd700 1 -- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27ac09e9c0 msgr2=0x7f27ac09edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27aaffd700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27ac09e9c0 0x7f27ac09edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27aaffd700 1 -- 172.21.15.32:0/3574350429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ac09fb80 con 0x7f27ac09a8b0 2021-04-09T18:32:33.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27ab7fe700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27ac09b2f0 0x7f27ac09b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:33.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f27aaffd700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac09a8b0 0x7f27ac09e3a0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7f27a000b5b0 tx=0x7f27a000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:33.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f2793fff700 1 -- 172.21.15.32:0/3574350429 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a000bd40 con 0x7f27ac09a8b0 2021-04-09T18:32:33.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.355 7f2793fff700 1 -- 172.21.15.32:0/3574350429 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a0006240 con 0x7f27ac09a8b0 2021-04-09T18:32:33.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.356 7f2793fff700 1 -- 172.21.15.32:0/3574350429 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27a0005690 con 0x7f27ac09a8b0 2021-04-09T18:32:33.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.356 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ac09fb80 con 0x7f27ac09a8b0 2021-04-09T18:32:33.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.356 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ac09fb80 con 0x7f27ac09a8b0 2021-04-09T18:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.356 7f2793fff700 1 -- 172.21.15.32:0/3574350429 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f27a00054a0 con 0x7f27ac09a8b0 2021-04-09T18:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.356 7f2793fff700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.356 7f2793fff700 1 -- 172.21.15.32:0/3574350429 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f27a00326d0 con 0x7f27ac09a8b0 2021-04-09T18:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.356 7f27ab7fe700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.357 7f27ab7fe700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 crc :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:33.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.376 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ac0b9040 con 0x7f27ac09a8b0 2021-04-09T18:32:33.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.380 7f2793fff700 1 -- 172.21.15.32:0/3574350429 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f27a0000c60 con 0x7f27ac09a8b0 2021-04-09T18:32:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.572 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f27ac10d500 con 0x7f27ac09a8b0 2021-04-09T18:32:33.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.682 7f2793fff700 1 -- 172.21.15.32:0/3574350429 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v377) v1 ==== 136+0+0 (secure 0 0 0) 0x7f27a00054a0 con 0x7f27ac09a8b0 2021-04-09T18:32:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 msgr2=0x7f278c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 crc :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac09a8b0 msgr2=0x7f27ac09e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac09a8b0 0x7f27ac09e3a0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7f27a000b5b0 tx=0x7f27a000b950).stop 2021-04-09T18:32:33.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 shutdown_connections 2021-04-09T18:32:33.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f278c018930 0x7f278c01ae10 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27ac09a8b0 0x7f27ac09e3a0 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27ac09e9c0 0x7f27ac09edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 --2- 172.21.15.32:0/3574350429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27ac09b2f0 0x7f27ac09b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 >> 172.21.15.32:0/3574350429 conn(0x7f27ac0ae6f0 msgr2=0x7f27ac097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 shutdown_connections 2021-04-09T18:32:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.688 7f27b1e1a700 1 -- 172.21.15.32:0/3574350429 wait complete. 2021-04-09T18:32:33.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:32:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.847 7f9564fd2700 1 Processor -- start 2021-04-09T18:32:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.847 7f9564fd2700 1 -- start start 2021-04-09T18:32:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f9564fd2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f95600d7ca0 0x7f95600d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f9564fd2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956010da00 0x7f956010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f9564fd2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956010d480 con 0x7f956010d010 2021-04-09T18:32:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f9564fd2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95600d9220 con 0x7f95600d7ca0 2021-04-09T18:32:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f9564fd2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95600d86c0 con 0x7f956010da00 2021-04-09T18:32:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955f59e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956010da00 0x7f956010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955e59c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f95600d7ca0 0x7f95600d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955ed9d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f956010d010 0x7f95600d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37014/0 (socket says 172.21.15.32:37014) 2021-04-09T18:32:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955ed9d700 1 -- 172.21.15.32:0/2578830311 learned_addr learned my addr 172.21.15.32:0/2578830311 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:33.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955f59e700 1 -- 172.21.15.32:0/2578830311 >> v2:172.21.15.86:3300/0 conn(0x7f95600d7ca0 msgr2=0x7f95600d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955f59e700 1 --2- 172.21.15.32:0/2578830311 >> v2:172.21.15.86:3300/0 conn(0x7f95600d7ca0 0x7f95600d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955f59e700 1 -- 172.21.15.32:0/2578830311 >> v1:172.21.15.86:6789/0 conn(0x7f956010d010 legacy=0x7f95600d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955dd9b700 1 -- 172.21.15.32:0/2578830311 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058880856 0 0) 0x7f95500010c0 con 0x7f956010d010 2021-04-09T18:32:33.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955f59e700 1 -- 172.21.15.32:0/2578830311 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600d1f70 con 0x7f956010da00 2021-04-09T18:32:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955e59c700 1 --2- 172.21.15.32:0/2578830311 >> v2:172.21.15.86:3300/0 conn(0x7f95600d7ca0 0x7f95600d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:33.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.848 7f955f59e700 1 --2- 172.21.15.32:0/2578830311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956010da00 0x7f956010de00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f9558006e50 tx=0x7f9558009e70).ready entity=mon.2 client_cookie=2da8a41bfeba763a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.849 7f955dd9b700 1 -- 172.21.15.32:0/2578830311 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9558006a20 con 0x7f956010da00 2021-04-09T18:32:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.849 7f955dd9b700 1 -- 172.21.15.32:0/2578830311 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9558006790 con 0x7f956010da00 2021-04-09T18:32:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.849 7f955dd9b700 1 -- 172.21.15.32:0/2578830311 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f955800b700 con 0x7f956010da00 2021-04-09T18:32:33.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.849 7f9564fd2700 1 -- 172.21.15.32:0/2578830311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956010da00 msgr2=0x7f956010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.849 7f9564fd2700 1 --2- 172.21.15.32:0/2578830311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956010da00 0x7f956010de00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f9558006e50 tx=0x7f9558009e70).stop 2021-04-09T18:32:33.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.850 7f9564fd2700 1 -- 172.21.15.32:0/2578830311 shutdown_connections 2021-04-09T18:32:33.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.850 7f9564fd2700 1 --2- 172.21.15.32:0/2578830311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956010da00 0x7f956010de00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.850 7f9564fd2700 1 --2- 172.21.15.32:0/2578830311 >> v2:172.21.15.86:3300/0 conn(0x7f95600d7ca0 0x7f95600d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.850 7f9564fd2700 1 -- 172.21.15.32:0/2578830311 >> 172.21.15.32:0/2578830311 conn(0x7f95600ce5f0 msgr2=0x7f95600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.850 7f9564fd2700 1 -- 172.21.15.32:0/2578830311 shutdown_connections 2021-04-09T18:32:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.850 7f9564fd2700 1 -- 172.21.15.32:0/2578830311 wait complete. 2021-04-09T18:32:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f9564fd2700 1 Processor -- start 2021-04-09T18:32:33.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f9564fd2700 1 -- start start 2021-04-09T18:32:33.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f9564fd2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9560185060 0x7f9560185d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f9564fd2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9560186c00 0x7f9560187000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f9564fd2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9560191360 0x7f9560191760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f9564fd2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95601863a0 con 0x7f9560191360 2021-04-09T18:32:33.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f9564fd2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9560185a60 con 0x7f9560186c00 2021-04-09T18:32:33.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f9564fd2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9560187620 con 0x7f9560185060 2021-04-09T18:32:33.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f955f59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9560191360 0x7f9560191760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f955f59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9560191360 0x7f9560191760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57740/0 (socket says 172.21.15.32:57740) 2021-04-09T18:32:33.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f955f59e700 1 -- 172.21.15.32:0/750828501 learned_addr learned my addr 172.21.15.32:0/750828501 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:33.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f955e59c700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9560186c00 0x7f9560187000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.870 7f955ed9d700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9560185060 0x7f9560185d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f955f59e700 1 -- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9560185060 msgr2=0x7f9560185d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f955f59e700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9560185060 0x7f9560185d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f955f59e700 1 -- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9560186c00 msgr2=0x7f9560187000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:33.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f955f59e700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9560186c00 0x7f9560187000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:33.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f955f59e700 1 -- 172.21.15.32:0/750828501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9560185460 con 0x7f9560191360 2021-04-09T18:32:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f955f59e700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9560191360 0x7f9560191760 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 rx=0x7f9558006e60 tx=0x7f955800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f95577fe700 1 -- 172.21.15.32:0/750828501 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9558009130 con 0x7f9560191360 2021-04-09T18:32:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f95577fe700 1 -- 172.21.15.32:0/750828501 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9558009430 con 0x7f9560191360 2021-04-09T18:32:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f95577fe700 1 -- 172.21.15.32:0/750828501 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9558001390 con 0x7f9560191360 2021-04-09T18:32:33.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f9564fd2700 1 -- 172.21.15.32:0/750828501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9560185460 con 0x7f9560191360 2021-04-09T18:32:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.871 7f9564fd2700 1 -- 172.21.15.32:0/750828501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9560185460 con 0x7f9560191360 2021-04-09T18:32:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.872 7f95577fe700 1 -- 172.21.15.32:0/750828501 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9558009130 con 0x7f9560191360 2021-04-09T18:32:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.872 7f95577fe700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f953c0188c0 0x7f953c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:33.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.872 7f95577fe700 1 -- 172.21.15.32:0/750828501 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f95580327b0 con 0x7f9560191360 2021-04-09T18:32:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.872 7f955ed9d700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f953c0188c0 0x7f953c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:33.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.872 7f955ed9d700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f953c0188c0 0x7f953c01adc0 crc :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:33.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.892 7f9564fd2700 1 -- 172.21.15.32:0/750828501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95601866d0 con 0x7f9560191360 2021-04-09T18:32:33.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:33.895 7f95577fe700 1 -- 172.21.15.32:0/750828501 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f955800be30 con 0x7f9560191360 2021-04-09T18:32:34.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.082 7f9564fd2700 1 -- 172.21.15.32:0/750828501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95601866d0 con 0x7f9560191360 2021-04-09T18:32:34.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.083 7f95577fe700 1 -- 172.21.15.32:0/750828501 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f9558009130 con 0x7f9560191360 2021-04-09T18:32:34.099 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:34.100 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":218,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:32:32.677290","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":205,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":204,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:32:34.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 -- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f953c0188c0 msgr2=0x7f953c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f953c0188c0 0x7f953c01adc0 crc :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 -- 172.21.15.32:0/750828501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9560191360 msgr2=0x7f9560191760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9560191360 0x7f9560191760 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 rx=0x7f9558006e60 tx=0x7f955800bcf0).stop 2021-04-09T18:32:34.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 -- 172.21.15.32:0/750828501 shutdown_connections 2021-04-09T18:32:34.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f953c0188c0 0x7f953c01adc0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9560191360 0x7f9560191760 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9560186c00 0x7f9560187000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 --2- 172.21.15.32:0/750828501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9560185060 0x7f9560185d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 -- 172.21.15.32:0/750828501 >> 172.21.15.32:0/750828501 conn(0x7f95601837d0 msgr2=0x7f9560195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:34.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.085 7f9564fd2700 1 -- 172.21.15.32:0/750828501 shutdown_connections 2021-04-09T18:32:34.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.086 7f9564fd2700 1 -- 172.21.15.32:0/750828501 wait complete. 2021-04-09T18:32:34.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:32:34.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:32:34.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.249 7f4e7aaa1700 1 Processor -- start 2021-04-09T18:32:34.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.249 7f4e7aaa1700 1 -- start start 2021-04-09T18:32:34.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.250 7f4e7aaa1700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4e7410d010 0x7f4e740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.250 7f4e7aaa1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e740d1da0 0x7f4e740d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.250 7f4e7aaa1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7410d480 con 0x7f4e740d82c0 2021-04-09T18:32:34.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.250 7f4e7aaa1700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e740d9220 con 0x7f4e7410d010 2021-04-09T18:32:34.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.250 7f4e7aaa1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e740d86c0 con 0x7f4e740d1da0 2021-04-09T18:32:34.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.250 7f4e73fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e740d1da0 0x7f4e740d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.250 7f4e788bd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4e7410d010 0x7f4e740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.250 7f4e790be700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4e740d82c0 0x7f4e7410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37028/0 (socket says 172.21.15.32:37028) 2021-04-09T18:32:34.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.250 7f4e790be700 1 -- 172.21.15.32:0/3971960368 learned_addr learned my addr 172.21.15.32:0/3971960368 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:34.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e737fe700 1 -- 172.21.15.32:0/3971960368 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514700616 0 0) 0x7f4e6c001440 con 0x7f4e740d82c0 2021-04-09T18:32:34.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e737fe700 1 -- 172.21.15.32:0/3971960368 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e60002fe0 con 0x7f4e740d82c0 2021-04-09T18:32:34.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e73fff700 1 -- 172.21.15.32:0/3971960368 >> v2:172.21.15.86:3300/0 conn(0x7f4e7410d010 msgr2=0x7f4e740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:34.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e73fff700 1 --2- 172.21.15.32:0/3971960368 >> v2:172.21.15.86:3300/0 conn(0x7f4e7410d010 0x7f4e740d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e73fff700 1 -- 172.21.15.32:0/3971960368 >> v1:172.21.15.86:6789/0 conn(0x7f4e740d82c0 legacy=0x7f4e7410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e73fff700 1 -- 172.21.15.32:0/3971960368 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7410e1f0 con 0x7f4e740d1da0 2021-04-09T18:32:34.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e788bd700 1 --2- 172.21.15.32:0/3971960368 >> v2:172.21.15.86:3300/0 conn(0x7f4e7410d010 0x7f4e740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:34.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e73fff700 1 --2- 172.21.15.32:0/3971960368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e740d1da0 0x7f4e740d7ca0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f4e64006650 tx=0x7f4e64009a80).ready entity=mon.2 client_cookie=574086e23dae4512 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:34.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e737fe700 1 -- 172.21.15.32:0/3971960368 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e640009b0 con 0x7f4e740d1da0 2021-04-09T18:32:34.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e737fe700 1 -- 172.21.15.32:0/3971960368 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6400c0c0 con 0x7f4e740d1da0 2021-04-09T18:32:34.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e737fe700 1 -- 172.21.15.32:0/3971960368 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e6400c510 con 0x7f4e740d1da0 2021-04-09T18:32:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e7aaa1700 1 -- 172.21.15.32:0/3971960368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e740d1da0 msgr2=0x7f4e740d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.251 7f4e7aaa1700 1 --2- 172.21.15.32:0/3971960368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e740d1da0 0x7f4e740d7ca0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f4e64006650 tx=0x7f4e64009a80).stop 2021-04-09T18:32:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.252 7f4e7aaa1700 1 -- 172.21.15.32:0/3971960368 shutdown_connections 2021-04-09T18:32:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.252 7f4e7aaa1700 1 --2- 172.21.15.32:0/3971960368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e740d1da0 0x7f4e740d7ca0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.252 7f4e7aaa1700 1 --2- 172.21.15.32:0/3971960368 >> v2:172.21.15.86:3300/0 conn(0x7f4e7410d010 0x7f4e740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.252 7f4e7aaa1700 1 -- 172.21.15.32:0/3971960368 >> 172.21.15.32:0/3971960368 conn(0x7f4e740ce5f0 msgr2=0x7f4e740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.252 7f4e7aaa1700 1 -- 172.21.15.32:0/3971960368 shutdown_connections 2021-04-09T18:32:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.252 7f4e7aaa1700 1 -- 172.21.15.32:0/3971960368 wait complete. 2021-04-09T18:32:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.272 7f4e7aaa1700 1 Processor -- start 2021-04-09T18:32:34.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.272 7f4e7aaa1700 1 -- start start 2021-04-09T18:32:34.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e7aaa1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7409e530 0x7f4e7409b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e7aaa1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e7409e930 0x7f4e7409f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e7aaa1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7409f990 0x7f4e74094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e7aaa1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7409edf0 con 0x7f4e7409e930 2021-04-09T18:32:34.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e7aaa1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7409fd90 con 0x7f4e7409f990 2021-04-09T18:32:34.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e7aaa1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7409bd40 con 0x7f4e7409e530 2021-04-09T18:32:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e73fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e7409e930 0x7f4e7409f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e73fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e7409e930 0x7f4e7409f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57748/0 (socket says 172.21.15.32:57748) 2021-04-09T18:32:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e73fff700 1 -- 172.21.15.32:0/653798296 learned_addr learned my addr 172.21.15.32:0/653798296 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:34.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e788bd700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7409e530 0x7f4e7409b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e790be700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7409f990 0x7f4e74094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e73fff700 1 -- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7409e530 msgr2=0x7f4e7409b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e73fff700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7409e530 0x7f4e7409b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e73fff700 1 -- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7409f990 msgr2=0x7f4e74094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e73fff700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7409f990 0x7f4e74094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e73fff700 1 -- 172.21.15.32:0/653798296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e74095290 con 0x7f4e7409e930 2021-04-09T18:32:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e73fff700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e7409e930 0x7f4e7409f370 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 rx=0x7f4e64009630 tx=0x7f4e6400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:34.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.273 7f4e717fa700 1 -- 172.21.15.32:0/653798296 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e640093a0 con 0x7f4e7409e930 2021-04-09T18:32:34.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.274 7f4e717fa700 1 -- 172.21.15.32:0/653798296 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6400bd70 con 0x7f4e7409e930 2021-04-09T18:32:34.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.274 7f4e717fa700 1 -- 172.21.15.32:0/653798296 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e640054d0 con 0x7f4e7409e930 2021-04-09T18:32:34.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.274 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e74095290 con 0x7f4e7409e930 2021-04-09T18:32:34.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.274 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e74095290 con 0x7f4e7409e930 2021-04-09T18:32:34.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.274 7f4e717fa700 1 -- 172.21.15.32:0/653798296 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4e640054f0 con 0x7f4e7409e930 2021-04-09T18:32:34.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.274 7f4e717fa700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e58018930 0x7f4e5801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.274 7f4e717fa700 1 -- 172.21.15.32:0/653798296 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f4e640326d0 con 0x7f4e7409e930 2021-04-09T18:32:34.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.275 7f4e788bd700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e58018930 0x7f4e5801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.275 7f4e788bd700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e58018930 0x7f4e5801adc0 crc :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:34.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.294 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74095b10 con 0x7f4e7409e930 2021-04-09T18:32:34.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.298 7f4e717fa700 1 -- 172.21.15.32:0/653798296 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4e640052e0 con 0x7f4e7409e930 2021-04-09T18:32:34.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.475 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f4e74095b10 con 0x7f4e7409e930 2021-04-09T18:32:34.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.476 7f4e717fa700 1 -- 172.21.15.32:0/653798296 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v218) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4e6400bd70 con 0x7f4e7409e930 2021-04-09T18:32:34.492 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:32:34.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e58018930 msgr2=0x7f4e5801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e58018930 0x7f4e5801adc0 crc :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e7409e930 msgr2=0x7f4e7409f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e7409e930 0x7f4e7409f370 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 rx=0x7f4e64009630 tx=0x7f4e6400b980).stop 2021-04-09T18:32:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 shutdown_connections 2021-04-09T18:32:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4e58018930 0x7f4e5801adc0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e7409e930 0x7f4e7409f370 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7409f990 0x7f4e74094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 --2- 172.21.15.32:0/653798296 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7409e530 0x7f4e7409b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 >> 172.21.15.32:0/653798296 conn(0x7f4e740c4290 msgr2=0x7f4e740c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 shutdown_connections 2021-04-09T18:32:34.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.478 7f4e7aaa1700 1 -- 172.21.15.32:0/653798296 wait complete. 2021-04-09T18:32:34.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:32:34.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.637 7f76512e6700 1 Processor -- start 2021-04-09T18:32:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.637 7f76512e6700 1 -- start start 2021-04-09T18:32:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f76512e6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f764c10da00 0x7f764c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f76512e6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764c10d480 con 0x7f764c10d010 2021-04-09T18:32:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f76512e6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764c0d1f70 con 0x7f764c0d7ca0 2021-04-09T18:32:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f76512e6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764c0d8d30 con 0x7f764c10da00 2021-04-09T18:32:34.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764b07d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f764c10d010 0x7f764c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39114/0 (socket says 172.21.15.32:39114) 2021-04-09T18:32:34.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764b07d700 1 -- 172.21.15.32:0/2299715723 learned_addr learned my addr 172.21.15.32:0/2299715723 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:34.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764b87e700 1 --2- 172.21.15.32:0/2299715723 >> v2:172.21.15.86:3300/0 conn(0x7f764c10da00 0x7f764c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764a07b700 1 -- 172.21.15.32:0/2299715723 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634547079 0 0) 0x7f763c000ce0 con 0x7f764c10d010 2021-04-09T18:32:34.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764a07b700 1 -- 172.21.15.32:0/2299715723 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7634002fe0 con 0x7f764c10d010 2021-04-09T18:32:34.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764a07b700 1 -- 172.21.15.32:0/2299715723 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914162223 0 0) 0x7f7640000c60 con 0x7f764c0d7ca0 2021-04-09T18:32:34.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764a07b700 1 -- 172.21.15.32:0/2299715723 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7634003210 con 0x7f764c0d7ca0 2021-04-09T18:32:34.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764b87e700 1 -- 172.21.15.32:0/2299715723 >> v1:172.21.15.86:6789/0 conn(0x7f764c0d7ca0 legacy=0x7f764c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764b87e700 1 -- 172.21.15.32:0/2299715723 >> v1:172.21.15.32:6789/0 conn(0x7f764c10d010 legacy=0x7f764c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764b87e700 1 -- 172.21.15.32:0/2299715723 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c0d9220 con 0x7f764c10da00 2021-04-09T18:32:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.638 7f764b87e700 1 --2- 172.21.15.32:0/2299715723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c10da00 0x7f764c10de00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7f7644006e50 tx=0x7f7644009e70).ready entity=mon.1 client_cookie=b2784121a019bc70 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f764a07b700 1 -- 172.21.15.32:0/2299715723 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7644006a20 con 0x7f764c10da00 2021-04-09T18:32:34.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f764a07b700 1 -- 172.21.15.32:0/2299715723 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644006790 con 0x7f764c10da00 2021-04-09T18:32:34.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f764a07b700 1 -- 172.21.15.32:0/2299715723 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f764400b700 con 0x7f764c10da00 2021-04-09T18:32:34.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f76512e6700 1 -- 172.21.15.32:0/2299715723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c10da00 msgr2=0x7f764c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f76512e6700 1 --2- 172.21.15.32:0/2299715723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c10da00 0x7f764c10de00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7f7644006e50 tx=0x7f7644009e70).stop 2021-04-09T18:32:34.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f76512e6700 1 -- 172.21.15.32:0/2299715723 shutdown_connections 2021-04-09T18:32:34.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f76512e6700 1 --2- 172.21.15.32:0/2299715723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c10da00 0x7f764c10de00 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f76512e6700 1 -- 172.21.15.32:0/2299715723 >> 172.21.15.32:0/2299715723 conn(0x7f764c0ce5f0 msgr2=0x7f764c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:34.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f76512e6700 1 -- 172.21.15.32:0/2299715723 shutdown_connections 2021-04-09T18:32:34.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.639 7f76512e6700 1 -- 172.21.15.32:0/2299715723 wait complete. 2021-04-09T18:32:34.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.659 7f76512e6700 1 Processor -- start 2021-04-09T18:32:34.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f76512e6700 1 -- start start 2021-04-09T18:32:34.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f76512e6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f764c185a00 0x7f764c097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f76512e6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c0983b0 0x7f764c17abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f76512e6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f764c17b1d0 0x7f764c179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f76512e6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764c0987b0 con 0x7f764c17b1d0 2021-04-09T18:32:34.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f76512e6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764c17b5d0 con 0x7f764c0983b0 2021-04-09T18:32:34.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f76512e6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764c179c30 con 0x7f764c185a00 2021-04-09T18:32:34.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f764b87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f764c17b1d0 0x7f764c179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f764b87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f764c17b1d0 0x7f764c179610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57766/0 (socket says 172.21.15.32:57766) 2021-04-09T18:32:34.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f764b87e700 1 -- 172.21.15.32:0/2697076852 learned_addr learned my addr 172.21.15.32:0/2697076852 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:34.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f764b07d700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f764c185a00 0x7f764c097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.660 7f764a87c700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c0983b0 0x7f764c17abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f764b87e700 1 -- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f764c185a00 msgr2=0x7f764c097d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f764b87e700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f764c185a00 0x7f764c097d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f764b87e700 1 -- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c0983b0 msgr2=0x7f764c17abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f764b87e700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c0983b0 0x7f764c17abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f764b87e700 1 -- 172.21.15.32:0/2697076852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c179e50 con 0x7f764c17b1d0 2021-04-09T18:32:34.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f764a87c700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c0983b0 0x7f764c17abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:34.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f764b87e700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f764c17b1d0 0x7f764c179610 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 rx=0x7f76440099f0 tx=0x7f764400bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:34.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f7633fff700 1 -- 172.21.15.32:0/2697076852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76440090c0 con 0x7f764c17b1d0 2021-04-09T18:32:34.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f7633fff700 1 -- 172.21.15.32:0/2697076852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7644000df0 con 0x7f764c17b1d0 2021-04-09T18:32:34.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f76512e6700 1 -- 172.21.15.32:0/2697076852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764c179e50 con 0x7f764c17b1d0 2021-04-09T18:32:34.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f7633fff700 1 -- 172.21.15.32:0/2697076852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7644001390 con 0x7f764c17b1d0 2021-04-09T18:32:34.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.661 7f76512e6700 1 -- 172.21.15.32:0/2697076852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f764c179e50 con 0x7f764c17b1d0 2021-04-09T18:32:34.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.662 7f7633fff700 1 -- 172.21.15.32:0/2697076852 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f76440056b0 con 0x7f764c17b1d0 2021-04-09T18:32:34.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.662 7f7633fff700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f762c0188d0 0x7f762c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:34.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.662 7f7633fff700 1 -- 172.21.15.32:0/2697076852 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f7644032a40 con 0x7f764c17b1d0 2021-04-09T18:32:34.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.662 7f764b07d700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f762c0188d0 0x7f762c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:34.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.662 7f764b07d700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f762c0188d0 0x7f762c01add0 crc :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:34.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.682 7f76512e6700 1 -- 172.21.15.32:0/2697076852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764c17a3e0 con 0x7f764c17b1d0 2021-04-09T18:32:34.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.685 7f7633fff700 1 -- 172.21.15.32:0/2697076852 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f764400bdd0 con 0x7f764c17b1d0 2021-04-09T18:32:34.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.864 7f76512e6700 1 -- 172.21.15.32:0/2697076852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f764c17a3e0 con 0x7f764c17b1d0 2021-04-09T18:32:34.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.865 7f7633fff700 1 -- 172.21.15.32:0/2697076852 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v218) v1 ==== 84+0+64 (secure 0 0 0) 0x7f7644000df0 con 0x7f764c17b1d0 2021-04-09T18:32:34.882 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:34.882 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:32:34.882 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:34.883 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:32:34.883 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:32:34.884 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:34.884 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:32:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 -- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f762c0188d0 msgr2=0x7f762c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f762c0188d0 0x7f762c01add0 crc :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 -- 172.21.15.32:0/2697076852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f764c17b1d0 msgr2=0x7f764c179610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:34.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f764c17b1d0 0x7f764c179610 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 rx=0x7f76440099f0 tx=0x7f764400bc90).stop 2021-04-09T18:32:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 -- 172.21.15.32:0/2697076852 shutdown_connections 2021-04-09T18:32:34.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f762c0188d0 0x7f762c01add0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f764c17b1d0 0x7f764c179610 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f764c0983b0 0x7f764c17abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 --2- 172.21.15.32:0/2697076852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f764c185a00 0x7f764c097d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.867 7f76512e6700 1 -- 172.21.15.32:0/2697076852 >> 172.21.15.32:0/2697076852 conn(0x7f764c190550 msgr2=0x7f764c1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:34.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.868 7f76512e6700 1 -- 172.21.15.32:0/2697076852 shutdown_connections 2021-04-09T18:32:34.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:34.868 7f76512e6700 1 -- 172.21.15.32:0/2697076852 wait complete. 2021-04-09T18:32:34.894 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'alpha' 2021-04-09T18:32:34.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create alpha_metadata 8 2021-04-09T18:32:35.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.031 7f01ac56a700 1 Processor -- start 2021-04-09T18:32:35.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.031 7f01ac56a700 1 -- start start 2021-04-09T18:32:35.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.032 7f01ac56a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f01a410d010 0x7f01a40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.032 7f01ac56a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a40d8490 0x7f01a410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.032 7f01ac56a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a410d480 con 0x7f01a40d1da0 2021-04-09T18:32:35.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.032 7f01ac56a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a40d8d30 con 0x7f01a410d010 2021-04-09T18:32:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.032 7f01ac56a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a40d9220 con 0x7f01a40d8490 2021-04-09T18:32:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.032 7f01a9b85700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f01a40d1da0 0x7f01a40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39132/0 (socket says 172.21.15.32:39132) 2021-04-09T18:32:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.032 7f01a9b85700 1 -- 172.21.15.32:0/539947574 learned_addr learned my addr 172.21.15.32:0/539947574 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.032 7f01aa386700 1 --2- 172.21.15.32:0/539947574 >> v2:172.21.15.86:3300/0 conn(0x7f01a410d010 0x7f01a40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.032 7f01aab87700 1 --2- 172.21.15.32:0/539947574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a40d8490 0x7f01a410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01a9384700 1 -- 172.21.15.32:0/539947574 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267153859 0 0) 0x7f01940009b0 con 0x7f01a40d1da0 2021-04-09T18:32:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01a9384700 1 -- 172.21.15.32:0/539947574 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0190002fe0 con 0x7f01a40d1da0 2021-04-09T18:32:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01aa386700 1 -- 172.21.15.32:0/539947574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a40d8490 msgr2=0x7f01a410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:35.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01aa386700 1 --2- 172.21.15.32:0/539947574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a40d8490 0x7f01a410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01aa386700 1 -- 172.21.15.32:0/539947574 >> v1:172.21.15.32:6789/0 conn(0x7f01a40d1da0 legacy=0x7f01a40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01aa386700 1 -- 172.21.15.32:0/539947574 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a410e020 con 0x7f01a410d010 2021-04-09T18:32:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01aab87700 1 --2- 172.21.15.32:0/539947574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a40d8490 0x7f01a410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01aa386700 1 --2- 172.21.15.32:0/539947574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a410d010 0x7f01a40d1780 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7f01a0006650 tx=0x7f01a0009a80).ready entity=mon.1 client_cookie=9d72d005c3a5a0c8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:35.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01a9384700 1 -- 172.21.15.32:0/539947574 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01a00009b0 con 0x7f01a410d010 2021-04-09T18:32:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01a9384700 1 -- 172.21.15.32:0/539947574 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a000c0c0 con 0x7f01a410d010 2021-04-09T18:32:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01a9384700 1 -- 172.21.15.32:0/539947574 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01a000c510 con 0x7f01a410d010 2021-04-09T18:32:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01ac56a700 1 -- 172.21.15.32:0/539947574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a410d010 msgr2=0x7f01a40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.033 7f01ac56a700 1 --2- 172.21.15.32:0/539947574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a410d010 0x7f01a40d1780 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7f01a0006650 tx=0x7f01a0009a80).stop 2021-04-09T18:32:35.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.034 7f01ac56a700 1 -- 172.21.15.32:0/539947574 shutdown_connections 2021-04-09T18:32:35.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.034 7f01ac56a700 1 --2- 172.21.15.32:0/539947574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a40d8490 0x7f01a410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.034 7f01ac56a700 1 --2- 172.21.15.32:0/539947574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a410d010 0x7f01a40d1780 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.034 7f01ac56a700 1 -- 172.21.15.32:0/539947574 >> 172.21.15.32:0/539947574 conn(0x7f01a40ce5f0 msgr2=0x7f01a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:35.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.034 7f01ac56a700 1 -- 172.21.15.32:0/539947574 shutdown_connections 2021-04-09T18:32:35.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.034 7f01ac56a700 1 -- 172.21.15.32:0/539947574 wait complete. 2021-04-09T18:32:35.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.054 7f01ac56a700 1 Processor -- start 2021-04-09T18:32:35.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.054 7f01ac56a700 1 -- start start 2021-04-09T18:32:35.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01ac56a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a40b2140 0x7f01a409d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01ac56a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a409e010 0x7f01a4194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01ac56a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a41950a0 0x7f01a40a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01ac56a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a409d470 con 0x7f01a41950a0 2021-04-09T18:32:35.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01ac56a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a409e410 con 0x7f01a40b2140 2021-04-09T18:32:35.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01ac56a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a41954a0 con 0x7f01a409e010 2021-04-09T18:32:35.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aab87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a41950a0 0x7f01a40a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aab87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a41950a0 0x7f01a40a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57780/0 (socket says 172.21.15.32:57780) 2021-04-09T18:32:35.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aab87700 1 -- 172.21.15.32:0/1908343063 learned_addr learned my addr 172.21.15.32:0/1908343063 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:35.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01a9b85700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a409e010 0x7f01a4194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aa386700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a40b2140 0x7f01a409d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aab87700 1 -- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a409e010 msgr2=0x7f01a4194a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aab87700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a409e010 0x7f01a4194a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aab87700 1 -- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a40b2140 msgr2=0x7f01a409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aab87700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a40b2140 0x7f01a409d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aab87700 1 -- 172.21.15.32:0/1908343063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a41956c0 con 0x7f01a41950a0 2021-04-09T18:32:35.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.055 7f01aab87700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a41950a0 0x7f01a40a0460 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7f019c00b910 tx=0x7f019c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:35.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.056 7f019affd700 1 -- 172.21.15.32:0/1908343063 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f019c006aa0 con 0x7f01a41950a0 2021-04-09T18:32:35.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.056 7f019affd700 1 -- 172.21.15.32:0/1908343063 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c0011f0 con 0x7f01a41950a0 2021-04-09T18:32:35.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.056 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a41956c0 con 0x7f01a41950a0 2021-04-09T18:32:35.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.056 7f019affd700 1 -- 172.21.15.32:0/1908343063 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f019c001640 con 0x7f01a41950a0 2021-04-09T18:32:35.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.056 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a41956c0 con 0x7f01a41950a0 2021-04-09T18:32:35.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.056 7f019affd700 1 -- 172.21.15.32:0/1908343063 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f019c0056b0 con 0x7f01a41950a0 2021-04-09T18:32:35.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.056 7f019affd700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f018c018930 0x7f018c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.056 7f019affd700 1 -- 172.21.15.32:0/1908343063 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f019c032a70 con 0x7f01a41950a0 2021-04-09T18:32:35.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.057 7f01aa386700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f018c018930 0x7f018c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.057 7f01aa386700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f018c018930 0x7f018c01ae10 crc :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.076 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a41b7040 con 0x7f01a41950a0 2021-04-09T18:32:35.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.080 7f019affd700 1 -- 172.21.15.32:0/1908343063 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f019c00bdf0 con 0x7f01a41950a0 2021-04-09T18:32:35.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.262 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_metadata"} v 0) v1 -- 0x7f01a41b7040 con 0x7f01a41950a0 2021-04-09T18:32:35.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.692 7f019affd700 1 -- 172.21.15.32:0/1908343063 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_metadata"}]=0 pool 'alpha_metadata' created v219) v1 ==== 131+0+0 (secure 0 0 0) 0x7f019c0056b0 con 0x7f01a41950a0 2021-04-09T18:32:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_metadata' created 2021-04-09T18:32:35.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f018c018930 msgr2=0x7f018c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f018c018930 0x7f018c01ae10 crc :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a41950a0 msgr2=0x7f01a40a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a41950a0 0x7f01a40a0460 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 rx=0x7f019c00b910 tx=0x7f019c00bcb0).stop 2021-04-09T18:32:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 shutdown_connections 2021-04-09T18:32:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f018c018930 0x7f018c01ae10 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01a41950a0 0x7f01a40a0460 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01a409e010 0x7f01a4194a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 --2- 172.21.15.32:0/1908343063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01a40b2140 0x7f01a409d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 >> 172.21.15.32:0/1908343063 conn(0x7f01a40b27e0 msgr2=0x7f01a41b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 shutdown_connections 2021-04-09T18:32:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.729 7f01ac56a700 1 -- 172.21.15.32:0/1908343063 wait complete. 2021-04-09T18:32:35.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create alpha_data 8 2021-04-09T18:32:35.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.893 7f73e52d2700 1 Processor -- start 2021-04-09T18:32:35.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.893 7f73e52d2700 1 -- start start 2021-04-09T18:32:35.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.893 7f73e52d2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f73e00cffc0 0x7f73e00d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.893 7f73e52d2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f73e00d6ca0 0x7f73e010cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.893 7f73e52d2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00d72a0 con 0x7f73e00d7820 2021-04-09T18:32:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.893 7f73e52d2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e00d70a0 con 0x7f73e00d6ca0 2021-04-09T18:32:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.893 7f73e52d2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e00d9200 con 0x7f73e00cffc0 2021-04-09T18:32:35.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.893 7f73df87e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f73e00d6ca0 0x7f73e010cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.893 7f73de87c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f73e00d7820 0x7f73e00d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39146/0 (socket says 172.21.15.32:39146) 2021-04-09T18:32:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73de87c700 1 -- 172.21.15.32:0/922376254 learned_addr learned my addr 172.21.15.32:0/922376254 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73df07d700 1 --2- 172.21.15.32:0/922376254 >> v2:172.21.15.86:3300/0 conn(0x7f73e00cffc0 0x7f73e00d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73df87e700 1 -- 172.21.15.32:0/922376254 >> v2:172.21.15.86:3300/0 conn(0x7f73e00cffc0 msgr2=0x7f73e00d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:35.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73df87e700 1 --2- 172.21.15.32:0/922376254 >> v2:172.21.15.86:3300/0 conn(0x7f73e00cffc0 0x7f73e00d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73df87e700 1 -- 172.21.15.32:0/922376254 >> v1:172.21.15.32:6789/0 conn(0x7f73e00d7820 legacy=0x7f73e00d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73de07b700 1 -- 172.21.15.32:0/922376254 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374045087 0 0) 0x7f73d40009b0 con 0x7f73e00d7820 2021-04-09T18:32:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73df87e700 1 -- 172.21.15.32:0/922376254 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00d8410 con 0x7f73e00d6ca0 2021-04-09T18:32:35.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73df87e700 1 --2- 172.21.15.32:0/922376254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e00d6ca0 0x7f73e010cff0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 rx=0x7f73d8006e30 tx=0x7f73d8009de0).ready entity=mon.0 client_cookie=e3a77bae82b7c798 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73de07b700 1 -- 172.21.15.32:0/922376254 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d800b300 con 0x7f73e00d6ca0 2021-04-09T18:32:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73de07b700 1 -- 172.21.15.32:0/922376254 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d800b680 con 0x7f73e00d6ca0 2021-04-09T18:32:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73de07b700 1 -- 172.21.15.32:0/922376254 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d800bb60 con 0x7f73e00d6ca0 2021-04-09T18:32:35.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73e52d2700 1 -- 172.21.15.32:0/922376254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e00d6ca0 msgr2=0x7f73e010cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.894 7f73e52d2700 1 --2- 172.21.15.32:0/922376254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e00d6ca0 0x7f73e010cff0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 rx=0x7f73d8006e30 tx=0x7f73d8009de0).stop 2021-04-09T18:32:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.895 7f73e52d2700 1 -- 172.21.15.32:0/922376254 shutdown_connections 2021-04-09T18:32:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.895 7f73e52d2700 1 --2- 172.21.15.32:0/922376254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e00d6ca0 0x7f73e010cff0 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.895 7f73e52d2700 1 --2- 172.21.15.32:0/922376254 >> v2:172.21.15.86:3300/0 conn(0x7f73e00cffc0 0x7f73e00d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.895 7f73e52d2700 1 -- 172.21.15.32:0/922376254 >> 172.21.15.32:0/922376254 conn(0x7f73e00ce5e0 msgr2=0x7f73e00cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:35.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.895 7f73e52d2700 1 -- 172.21.15.32:0/922376254 shutdown_connections 2021-04-09T18:32:35.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.895 7f73e52d2700 1 -- 172.21.15.32:0/922376254 wait complete. 2021-04-09T18:32:35.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.915 7f73e52d2700 1 Processor -- start 2021-04-09T18:32:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.915 7f73e52d2700 1 -- start start 2021-04-09T18:32:35.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73e52d2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e01b8ae0 0x7f73e00a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73e52d2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e01b8300 0x7f73e00b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73e52d2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e00b9080 0x7f73e00b9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73e52d2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00b9e00 con 0x7f73e01b8ae0 2021-04-09T18:32:35.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73e52d2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00b8b40 con 0x7f73e00b9080 2021-04-09T18:32:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73e52d2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00b1490 con 0x7f73e01b8300 2021-04-09T18:32:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e01b8ae0 0x7f73e00a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df07d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e01b8ae0 0x7f73e00a3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57792/0 (socket says 172.21.15.32:57792) 2021-04-09T18:32:35.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df07d700 1 -- 172.21.15.32:0/2221204870 learned_addr learned my addr 172.21.15.32:0/2221204870 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df87e700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e00b9080 0x7f73e00b9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73de87c700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e01b8300 0x7f73e00b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df07d700 1 -- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e01b8300 msgr2=0x7f73e00b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df07d700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e01b8300 0x7f73e00b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df07d700 1 -- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e00b9080 msgr2=0x7f73e00b9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df07d700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e00b9080 0x7f73e00b9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df07d700 1 -- 172.21.15.32:0/2221204870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00b7ad0 con 0x7f73e01b8ae0 2021-04-09T18:32:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73df07d700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e01b8ae0 0x7f73e00a3a80 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 rx=0x7f73d0006f60 tx=0x7f73d000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:35.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.916 7f73c7fff700 1 -- 172.21.15.32:0/2221204870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d000bd60 con 0x7f73e01b8ae0 2021-04-09T18:32:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.917 7f73c7fff700 1 -- 172.21.15.32:0/2221204870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d0000c90 con 0x7f73e01b8ae0 2021-04-09T18:32:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.917 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e00b7ad0 con 0x7f73e01b8ae0 2021-04-09T18:32:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.917 7f73c7fff700 1 -- 172.21.15.32:0/2221204870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73d00009b0 con 0x7f73e01b8ae0 2021-04-09T18:32:35.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.917 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e00b7ad0 con 0x7f73e01b8ae0 2021-04-09T18:32:35.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.917 7f73c7fff700 1 -- 172.21.15.32:0/2221204870 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f73d0006b00 con 0x7f73e01b8ae0 2021-04-09T18:32:35.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.918 7f73c7fff700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73c00188c0 0x7f73c001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:35.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.918 7f73c7fff700 1 -- 172.21.15.32:0/2221204870 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5110+0+0 (secure 0 0 0) 0x7f73d00325b0 con 0x7f73e01b8ae0 2021-04-09T18:32:35.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.918 7f73de87c700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73c00188c0 0x7f73c001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:35.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.918 7f73de87c700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73c00188c0 0x7f73c001adc0 crc :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:35.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.938 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e01b8f10 con 0x7f73e01b8ae0 2021-04-09T18:32:35.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:35.941 7f73c7fff700 1 -- 172.21.15.32:0/2221204870 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f73d00009b0 con 0x7f73e01b8ae0 2021-04-09T18:32:36.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:36.134 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_data"} v 0) v1 -- 0x7f73e01b8f10 con 0x7f73e01b8ae0 2021-04-09T18:32:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:36.988 7f73c7fff700 1 -- 172.21.15.32:0/2221204870 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "alpha_data"}]=0 pool 'alpha_data' created v224) v1 ==== 123+0+0 (secure 0 0 0) 0x7f73d0006b00 con 0x7f73e01b8ae0 2021-04-09T18:32:37.015 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_data' created 2021-04-09T18:32:37.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73c00188c0 msgr2=0x7f73c001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73c00188c0 0x7f73c001adc0 crc :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e01b8ae0 msgr2=0x7f73e00a3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e01b8ae0 0x7f73e00a3a80 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 rx=0x7f73d0006f60 tx=0x7f73d000b970).stop 2021-04-09T18:32:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 shutdown_connections 2021-04-09T18:32:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f73c00188c0 0x7f73c001adc0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e00b9080 0x7f73e00b9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e01b8300 0x7f73e00b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 --2- 172.21.15.32:0/2221204870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73e01b8ae0 0x7f73e00a3a80 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 >> 172.21.15.32:0/2221204870 conn(0x7f73e00a1640 msgr2=0x7f73e01b9120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:37.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 shutdown_connections 2021-04-09T18:32:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.002 7f73e52d2700 1 -- 172.21.15.32:0/2221204870 wait complete. 2021-04-09T18:32:37.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new alpha alpha_metadata alpha_data --force 2021-04-09T18:32:37.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.173 7f8b978d8700 1 Processor -- start 2021-04-09T18:32:37.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b978d8700 1 -- start start 2021-04-09T18:32:37.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b978d8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8b9010d020 0x7f8b900d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.218 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b978d8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b900d1db0 0x7f8b900d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b978d8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b900d82d0 0x7f8b9010da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b978d8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b900d8d40 con 0x7f8b900d1db0 2021-04-09T18:32:37.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b978d8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b900d9230 con 0x7f8b9010d020 2021-04-09T18:32:37.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b978d8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9010d4c0 con 0x7f8b900d82d0 2021-04-09T18:32:37.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b94ef3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b900d1db0 0x7f8b900d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b94ef3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b900d1db0 0x7f8b900d7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57806/0 (socket says 172.21.15.32:57806) 2021-04-09T18:32:37.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b94ef3700 1 -- 172.21.15.32:0/2582297811 learned_addr learned my addr 172.21.15.32:0/2582297811 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:37.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b95ef5700 1 --2- 172.21.15.32:0/2582297811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b900d82d0 0x7f8b9010da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.174 7f8b956f4700 1 --2- 172.21.15.32:0/2582297811 >> v2:172.21.15.86:3300/0 conn(0x7f8b9010d020 0x7f8b900d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b94ef3700 1 -- 172.21.15.32:0/2582297811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b900d82d0 msgr2=0x7f8b9010da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:37.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b94ef3700 1 --2- 172.21.15.32:0/2582297811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b900d82d0 0x7f8b9010da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b94ef3700 1 -- 172.21.15.32:0/2582297811 >> v2:172.21.15.86:3300/0 conn(0x7f8b9010d020 msgr2=0x7f8b900d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:37.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b94ef3700 1 --2- 172.21.15.32:0/2582297811 >> v2:172.21.15.86:3300/0 conn(0x7f8b9010d020 0x7f8b900d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b94ef3700 1 -- 172.21.15.32:0/2582297811 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b900d86d0 con 0x7f8b900d1db0 2021-04-09T18:32:37.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b94ef3700 1 --2- 172.21.15.32:0/2582297811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900d1db0 0x7f8b900d7cb0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f8b84006700 tx=0x7f8b84009b00).ready entity=mon.0 client_cookie=8ae287b090ece486 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:37.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b83fff700 1 -- 172.21.15.32:0/2582297811 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8400c0c0 con 0x7f8b900d1db0 2021-04-09T18:32:37.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b83fff700 1 -- 172.21.15.32:0/2582297811 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8400c400 con 0x7f8b900d1db0 2021-04-09T18:32:37.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b83fff700 1 -- 172.21.15.32:0/2582297811 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8400c8e0 con 0x7f8b900d1db0 2021-04-09T18:32:37.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b978d8700 1 -- 172.21.15.32:0/2582297811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900d1db0 msgr2=0x7f8b900d7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.175 7f8b978d8700 1 --2- 172.21.15.32:0/2582297811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900d1db0 0x7f8b900d7cb0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f8b84006700 tx=0x7f8b84009b00).stop 2021-04-09T18:32:37.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.176 7f8b978d8700 1 -- 172.21.15.32:0/2582297811 shutdown_connections 2021-04-09T18:32:37.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.176 7f8b978d8700 1 --2- 172.21.15.32:0/2582297811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b900d82d0 0x7f8b9010da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.176 7f8b978d8700 1 --2- 172.21.15.32:0/2582297811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b900d1db0 0x7f8b900d7cb0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.176 7f8b978d8700 1 --2- 172.21.15.32:0/2582297811 >> v2:172.21.15.86:3300/0 conn(0x7f8b9010d020 0x7f8b900d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.176 7f8b978d8700 1 -- 172.21.15.32:0/2582297811 >> 172.21.15.32:0/2582297811 conn(0x7f8b900ce600 msgr2=0x7f8b900d2320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:37.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.176 7f8b978d8700 1 -- 172.21.15.32:0/2582297811 shutdown_connections 2021-04-09T18:32:37.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.176 7f8b978d8700 1 -- 172.21.15.32:0/2582297811 wait complete. 2021-04-09T18:32:37.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.196 7f8b978d8700 1 Processor -- start 2021-04-09T18:32:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.196 7f8b978d8700 1 -- start start 2021-04-09T18:32:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b978d8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b901a1310 0x7f8b901a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b978d8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b90189ca0 0x7f8b9018a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b978d8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9008d0c0 0x7f8b9008d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b978d8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9008ef30 con 0x7f8b90189ca0 2021-04-09T18:32:37.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b978d8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9018a6c0 con 0x7f8b9008d0c0 2021-04-09T18:32:37.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b978d8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b901893a0 con 0x7f8b901a1310 2021-04-09T18:32:37.233 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b94ef3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b90189ca0 0x7f8b9018a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b94ef3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b90189ca0 0x7f8b9018a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57810/0 (socket says 172.21.15.32:57810) 2021-04-09T18:32:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b94ef3700 1 -- 172.21.15.32:0/322346291 learned_addr learned my addr 172.21.15.32:0/322346291 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b95ef5700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9008d0c0 0x7f8b9008d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b956f4700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b901a1310 0x7f8b901a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b94ef3700 1 -- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b901a1310 msgr2=0x7f8b901a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b94ef3700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b901a1310 0x7f8b901a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b94ef3700 1 -- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9008d0c0 msgr2=0x7f8b9008d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b94ef3700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9008d0c0 0x7f8b9008d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b94ef3700 1 -- 172.21.15.32:0/322346291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9008dae0 con 0x7f8b90189ca0 2021-04-09T18:32:37.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b956f4700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b901a1310 0x7f8b901a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b95ef5700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9008d0c0 0x7f8b9008d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.197 7f8b94ef3700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b90189ca0 0x7f8b9018a0a0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 rx=0x7f8b84006f20 tx=0x7f8b8400b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.198 7f8b81ffb700 1 -- 172.21.15.32:0/322346291 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8400bdd0 con 0x7f8b90189ca0 2021-04-09T18:32:37.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.198 7f8b81ffb700 1 -- 172.21.15.32:0/322346291 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b84006440 con 0x7f8b90189ca0 2021-04-09T18:32:37.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.198 7f8b978d8700 1 -- 172.21.15.32:0/322346291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9008dae0 con 0x7f8b90189ca0 2021-04-09T18:32:37.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.198 7f8b81ffb700 1 -- 172.21.15.32:0/322346291 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b840052e0 con 0x7f8b90189ca0 2021-04-09T18:32:37.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.198 7f8b978d8700 1 -- 172.21.15.32:0/322346291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9008dae0 con 0x7f8b90189ca0 2021-04-09T18:32:37.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.198 7f8b81ffb700 1 -- 172.21.15.32:0/322346291 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8b840052e0 con 0x7f8b90189ca0 2021-04-09T18:32:37.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.199 7f8b81ffb700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b700188c0 0x7f8b7001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.199 7f8b81ffb700 1 -- 172.21.15.32:0/322346291 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 5459+0+0 (secure 0 0 0) 0x7f8b84032970 con 0x7f8b90189ca0 2021-04-09T18:32:37.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.199 7f8b956f4700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b700188c0 0x7f8b7001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.199 7f8b956f4700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b700188c0 0x7f8b7001adc0 crc :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:37.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.218 7f8b978d8700 1 -- 172.21.15.32:0/322346291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b90189770 con 0x7f8b90189ca0 2021-04-09T18:32:37.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.222 7f8b81ffb700 1 -- 172.21.15.32:0/322346291 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8b84006030 con 0x7f8b90189ca0 2021-04-09T18:32:37.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.405 7f8b978d8700 1 -- 172.21.15.32:0/322346291 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "alpha_data", "prefix": "fs new", "force": true, "fs_name": "alpha", "metadata": "alpha_metadata"} v 0) v1 -- 0x7f8b90189770 con 0x7f8b90189ca0 2021-04-09T18:32:37.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.409 7f8b81ffb700 1 -- 172.21.15.32:0/322346291 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "alpha_data", "prefix": "fs new", "force": true, "fs_name": "alpha", "metadata": "alpha_metadata"}]=0 new fs with metadata pool 38 and data pool 39 v379) v1 ==== 186+0+0 (secure 0 0 0) 0x7f8b840052e0 con 0x7f8b90189ca0 2021-04-09T18:32:37.428 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 38 and data pool 39 2021-04-09T18:32:37.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.415 7f8b978d8700 1 -- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b700188c0 msgr2=0x7f8b7001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.415 7f8b978d8700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b700188c0 0x7f8b7001adc0 crc :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.415 7f8b978d8700 1 -- 172.21.15.32:0/322346291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b90189ca0 msgr2=0x7f8b9018a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.415 7f8b978d8700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b90189ca0 0x7f8b9018a0a0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 rx=0x7f8b84006f20 tx=0x7f8b8400b9e0).stop 2021-04-09T18:32:37.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.416 7f8b978d8700 1 -- 172.21.15.32:0/322346291 shutdown_connections 2021-04-09T18:32:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.416 7f8b978d8700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b700188c0 0x7f8b7001adc0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.416 7f8b978d8700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b90189ca0 0x7f8b9018a0a0 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.416 7f8b978d8700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b9008d0c0 0x7f8b9008d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.416 7f8b978d8700 1 --2- 172.21.15.32:0/322346291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b901a1310 0x7f8b901a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.416 7f8b978d8700 1 -- 172.21.15.32:0/322346291 >> 172.21.15.32:0/322346291 conn(0x7f8b9008b600 msgr2=0x7f8b9008a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.417 7f8b978d8700 1 -- 172.21.15.32:0/322346291 shutdown_connections 2021-04-09T18:32:37.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.417 7f8b978d8700 1 -- 172.21.15.32:0/322346291 wait complete. 2021-04-09T18:32:37.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:32:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.575 7f7f1299b700 1 Processor -- start 2021-04-09T18:32:37.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.575 7f7f1299b700 1 -- start start 2021-04-09T18:32:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.576 7f7f1299b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7f0c10d010 0x7f7f0c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.576 7f7f1299b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c0d8490 0x7f7f0c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.576 7f7f1299b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f0c10d480 con 0x7f7f0c0d1da0 2021-04-09T18:32:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.576 7f7f1299b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0c0d8d30 con 0x7f7f0c10d010 2021-04-09T18:32:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.576 7f7f1299b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0d9220 con 0x7f7f0c0d8490 2021-04-09T18:32:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.576 7f7f03fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7f0c0d1da0 0x7f7f0c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39178/0 (socket says 172.21.15.32:39178) 2021-04-09T18:32:37.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.576 7f7f03fff700 1 -- 172.21.15.32:0/722214999 learned_addr learned my addr 172.21.15.32:0/722214999 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.576 7f7f0bfff700 1 --2- 172.21.15.32:0/722214999 >> v2:172.21.15.86:3300/0 conn(0x7f7f0c10d010 0x7f7f0c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.576 7f7f10fb8700 1 --2- 172.21.15.32:0/722214999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c0d8490 0x7f7f0c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f0b7fe700 1 -- 172.21.15.32:0/722214999 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031662965 0 0) 0x7f7ef4000ce0 con 0x7f7f0c0d1da0 2021-04-09T18:32:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f0b7fe700 1 -- 172.21.15.32:0/722214999 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef8002fe0 con 0x7f7f0c0d1da0 2021-04-09T18:32:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f10fb8700 1 -- 172.21.15.32:0/722214999 >> v2:172.21.15.86:3300/0 conn(0x7f7f0c10d010 msgr2=0x7f7f0c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:37.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f10fb8700 1 --2- 172.21.15.32:0/722214999 >> v2:172.21.15.86:3300/0 conn(0x7f7f0c10d010 0x7f7f0c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f10fb8700 1 -- 172.21.15.32:0/722214999 >> v1:172.21.15.32:6789/0 conn(0x7f7f0c0d1da0 legacy=0x7f7f0c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f10fb8700 1 -- 172.21.15.32:0/722214999 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c10e020 con 0x7f7f0c0d8490 2021-04-09T18:32:37.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f0bfff700 1 --2- 172.21.15.32:0/722214999 >> v2:172.21.15.86:3300/0 conn(0x7f7f0c10d010 0x7f7f0c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f10fb8700 1 --2- 172.21.15.32:0/722214999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c0d8490 0x7f7f0c10da00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7f04006e50 tx=0x7f7f04009e70).ready entity=mon.2 client_cookie=51d0c0b878f7795a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f0b7fe700 1 -- 172.21.15.32:0/722214999 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f04006a20 con 0x7f7f0c0d8490 2021-04-09T18:32:37.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f0b7fe700 1 -- 172.21.15.32:0/722214999 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f04006790 con 0x7f7f0c0d8490 2021-04-09T18:32:37.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f0b7fe700 1 -- 172.21.15.32:0/722214999 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f04000f90 con 0x7f7f0c0d8490 2021-04-09T18:32:37.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f1299b700 1 -- 172.21.15.32:0/722214999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c0d8490 msgr2=0x7f7f0c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.577 7f7f1299b700 1 --2- 172.21.15.32:0/722214999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c0d8490 0x7f7f0c10da00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7f04006e50 tx=0x7f7f04009e70).stop 2021-04-09T18:32:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.578 7f7f1299b700 1 -- 172.21.15.32:0/722214999 shutdown_connections 2021-04-09T18:32:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.578 7f7f1299b700 1 --2- 172.21.15.32:0/722214999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c0d8490 0x7f7f0c10da00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.578 7f7f1299b700 1 --2- 172.21.15.32:0/722214999 >> v2:172.21.15.86:3300/0 conn(0x7f7f0c10d010 0x7f7f0c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.578 7f7f1299b700 1 -- 172.21.15.32:0/722214999 >> 172.21.15.32:0/722214999 conn(0x7f7f0c0ce5f0 msgr2=0x7f7f0c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:37.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.578 7f7f1299b700 1 -- 172.21.15.32:0/722214999 shutdown_connections 2021-04-09T18:32:37.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.578 7f7f1299b700 1 -- 172.21.15.32:0/722214999 wait complete. 2021-04-09T18:32:37.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.598 7f7f1299b700 1 Processor -- start 2021-04-09T18:32:37.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.598 7f7f1299b700 1 -- start start 2021-04-09T18:32:37.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f1299b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c1a5210 0x7f7f0c185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f1299b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c185a70 0x7f7f0c0acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f1299b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f0c185e70 0x7f7f0c0ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f1299b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c18f7b0 con 0x7f7f0c185e70 2021-04-09T18:32:37.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f1299b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0ad7f0 con 0x7f7f0c1a5210 2021-04-09T18:32:37.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f1299b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0ada10 con 0x7f7f0c185a70 2021-04-09T18:32:37.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f10fb8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f0c185e70 0x7f7f0c0ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f10fb8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f0c185e70 0x7f7f0c0ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57826/0 (socket says 172.21.15.32:57826) 2021-04-09T18:32:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f10fb8700 1 -- 172.21.15.32:0/1555300754 learned_addr learned my addr 172.21.15.32:0/1555300754 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:37.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f03fff700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c185a70 0x7f7f0c0acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f0bfff700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c1a5210 0x7f7f0c185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f10fb8700 1 -- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c185a70 msgr2=0x7f7f0c0acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f10fb8700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c185a70 0x7f7f0c0acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f10fb8700 1 -- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c1a5210 msgr2=0x7f7f0c185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f10fb8700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c1a5210 0x7f7f0c185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.599 7f7f10fb8700 1 -- 172.21.15.32:0/1555300754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c179610 con 0x7f7f0c185e70 2021-04-09T18:32:37.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.600 7f7f10fb8700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f0c185e70 0x7f7f0c0ad1d0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 rx=0x7f7f040099f0 tx=0x7f7f0400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.600 7f7f097fa700 1 -- 172.21.15.32:0/1555300754 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f040093d0 con 0x7f7f0c185e70 2021-04-09T18:32:37.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.600 7f7f097fa700 1 -- 172.21.15.32:0/1555300754 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f040090c0 con 0x7f7f0c185e70 2021-04-09T18:32:37.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.600 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0c179610 con 0x7f7f0c185e70 2021-04-09T18:32:37.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.600 7f7f097fa700 1 -- 172.21.15.32:0/1555300754 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f04001390 con 0x7f7f0c185e70 2021-04-09T18:32:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.600 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0c179610 con 0x7f7f0c185e70 2021-04-09T18:32:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.600 7f7f097fa700 1 -- 172.21.15.32:0/1555300754 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7f040011a0 con 0x7f7f0c185e70 2021-04-09T18:32:37.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.601 7f7f097fa700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ef00188d0 0x7f7ef001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:37.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.601 7f7f097fa700 1 -- 172.21.15.32:0/1555300754 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7f04032d60 con 0x7f7f0c185e70 2021-04-09T18:32:37.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.601 7f7f0bfff700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ef00188d0 0x7f7ef001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:37.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.601 7f7f0bfff700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ef00188d0 0x7f7ef001add0 crc :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.620 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0c18f7b0 con 0x7f7f0c185e70 2021-04-09T18:32:37.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.624 7f7f097fa700 1 -- 172.21.15.32:0/1555300754 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7f04002f30 con 0x7f7f0c185e70 2021-04-09T18:32:37.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.814 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f0c18f7b0 con 0x7f7f0c185e70 2021-04-09T18:32:37.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.815 7f7f097fa700 1 -- 172.21.15.32:0/1555300754 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) v1 ==== 74+0+14016 (secure 0 0 0) 0x7f7f040011a0 con 0x7f7f0c185e70 2021-04-09T18:32:37.832 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:37.833 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":225,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:32:37.406384","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:32:35.297513","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":39,"pool_name":"alpha_data","create_time":"2021-04-09 18:32:36.169493","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:32:37.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.817 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ef00188d0 msgr2=0x7f7ef001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.817 7f7f1299b700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ef00188d0 0x7f7ef001add0 crc :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.817 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f0c185e70 msgr2=0x7f7f0c0ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:37.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.817 7f7f1299b700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f0c185e70 0x7f7f0c0ad1d0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 rx=0x7f7f040099f0 tx=0x7f7f0400bd10).stop 2021-04-09T18:32:37.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.817 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 shutdown_connections 2021-04-09T18:32:37.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.818 7f7f1299b700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ef00188d0 0x7f7ef001add0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.818 7f7f1299b700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f0c185e70 0x7f7f0c0ad1d0 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.818 7f7f1299b700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f0c185a70 0x7f7f0c0acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.818 7f7f1299b700 1 --2- 172.21.15.32:0/1555300754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c1a5210 0x7f7f0c185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:37.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.818 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 >> 172.21.15.32:0/1555300754 conn(0x7f7f0c1a99c0 msgr2=0x7f7f0c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:37.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.818 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 shutdown_connections 2021-04-09T18:32:37.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.818 7f7f1299b700 1 -- 172.21.15.32:0/1555300754 wait complete. 2021-04-09T18:32:37.844 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:32:38.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.977 7f44bfa17700 1 Processor -- start 2021-04-09T18:32:38.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.977 7f44bfa17700 1 -- start start 2021-04-09T18:32:38.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bfa17700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f44b80cffa0 0x7f44b80d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bfa17700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b80d7280 con 0x7f44b80d7800 2021-04-09T18:32:38.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bfa17700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b80d83f0 con 0x7f44b80d6c80 2021-04-09T18:32:38.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bfa17700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b80d7080 con 0x7f44b80cffa0 2021-04-09T18:32:38.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bd032700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f44b80d7800 0x7f44b80d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39192/0 (socket says 172.21.15.32:39192) 2021-04-09T18:32:38.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bd032700 1 -- 172.21.15.32:0/475107456 learned_addr learned my addr 172.21.15.32:0/475107456 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:38.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bd833700 1 --2- 172.21.15.32:0/475107456 >> v2:172.21.15.86:3300/0 conn(0x7f44b80cffa0 0x7f44b80d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bc831700 1 -- 172.21.15.32:0/475107456 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174387248 0 0) 0x7f44ac0009b0 con 0x7f44b80d7800 2021-04-09T18:32:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bc831700 1 -- 172.21.15.32:0/475107456 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a4002fe0 con 0x7f44b80d7800 2021-04-09T18:32:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bc831700 1 -- 172.21.15.32:0/475107456 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765171967 0 0) 0x7f44b0000d30 con 0x7f44b80d6c80 2021-04-09T18:32:38.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bc831700 1 -- 172.21.15.32:0/475107456 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a4003210 con 0x7f44b80d6c80 2021-04-09T18:32:38.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bd833700 1 -- 172.21.15.32:0/475107456 >> v1:172.21.15.86:6789/0 conn(0x7f44b80d6c80 legacy=0x7f44b810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.978 7f44bd833700 1 -- 172.21.15.32:0/475107456 >> v1:172.21.15.32:6789/0 conn(0x7f44b80d7800 legacy=0x7f44b80d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.979 7f44bd833700 1 -- 172.21.15.32:0/475107456 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b80d91e0 con 0x7f44b80cffa0 2021-04-09T18:32:38.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.979 7f44bd833700 1 --2- 172.21.15.32:0/475107456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b80cffa0 0x7f44b80d2440 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f44b4006650 tx=0x7f44b4009a00).ready entity=mon.1 client_cookie=62caf123e8108c57 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:38.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.979 7f44bc831700 1 -- 172.21.15.32:0/475107456 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b4009df0 con 0x7f44b80cffa0 2021-04-09T18:32:38.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.979 7f44bc831700 1 -- 172.21.15.32:0/475107456 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b40009b0 con 0x7f44b80cffa0 2021-04-09T18:32:38.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.979 7f44bc831700 1 -- 172.21.15.32:0/475107456 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b400c2b0 con 0x7f44b80cffa0 2021-04-09T18:32:38.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.979 7f44bfa17700 1 -- 172.21.15.32:0/475107456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b80cffa0 msgr2=0x7f44b80d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.979 7f44bfa17700 1 --2- 172.21.15.32:0/475107456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b80cffa0 0x7f44b80d2440 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f44b4006650 tx=0x7f44b4009a00).stop 2021-04-09T18:32:38.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.980 7f44bfa17700 1 -- 172.21.15.32:0/475107456 shutdown_connections 2021-04-09T18:32:38.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.980 7f44bfa17700 1 --2- 172.21.15.32:0/475107456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b80cffa0 0x7f44b80d2440 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.980 7f44bfa17700 1 -- 172.21.15.32:0/475107456 >> 172.21.15.32:0/475107456 conn(0x7f44b80ce5c0 msgr2=0x7f44b80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:38.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.980 7f44bfa17700 1 -- 172.21.15.32:0/475107456 shutdown_connections 2021-04-09T18:32:38.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:37.980 7f44bfa17700 1 -- 172.21.15.32:0/475107456 wait complete. 2021-04-09T18:32:38.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44bfa17700 1 Processor -- start 2021-04-09T18:32:38.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44bfa17700 1 -- start start 2021-04-09T18:32:38.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44bfa17700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b819c910 0x7f44b8093030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44bfa17700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44b8093650 0x7f44b819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44bfa17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b819cd10 0x7f44b819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44bfa17700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b8093c20 con 0x7f44b819cd10 2021-04-09T18:32:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44bfa17700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80941a0 con 0x7f44b819c910 2021-04-09T18:32:38.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44bfa17700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80943c0 con 0x7f44b8093650 2021-04-09T18:32:38.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44be034700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b819cd10 0x7f44b819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44be034700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b819cd10 0x7f44b819fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57840/0 (socket says 172.21.15.32:57840) 2021-04-09T18:32:38.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.000 7f44be034700 1 -- 172.21.15.32:0/2895457404 learned_addr learned my addr 172.21.15.32:0/2895457404 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:38.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44bd833700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b819c910 0x7f44b8093030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44bd032700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44b8093650 0x7f44b819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44be034700 1 -- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44b8093650 msgr2=0x7f44b819f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44be034700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44b8093650 0x7f44b819f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44be034700 1 -- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b819c910 msgr2=0x7f44b8093030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44be034700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b819c910 0x7f44b8093030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44be034700 1 -- 172.21.15.32:0/2895457404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b80945e0 con 0x7f44b819cd10 2021-04-09T18:32:38.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44bd833700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b819c910 0x7f44b8093030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:38.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44be034700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b819cd10 0x7f44b819fa80 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 rx=0x7f44b0009980 tx=0x7f44b000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:38.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44aa7fc700 1 -- 172.21.15.32:0/2895457404 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b000be10 con 0x7f44b819cd10 2021-04-09T18:32:38.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44aa7fc700 1 -- 172.21.15.32:0/2895457404 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b0005870 con 0x7f44b819cd10 2021-04-09T18:32:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b80945e0 con 0x7f44b819cd10 2021-04-09T18:32:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44aa7fc700 1 -- 172.21.15.32:0/2895457404 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b0005d50 con 0x7f44b819cd10 2021-04-09T18:32:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.001 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b80945e0 con 0x7f44b819cd10 2021-04-09T18:32:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.002 7f44aa7fc700 1 -- 172.21.15.32:0/2895457404 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f44b0005a50 con 0x7f44b819cd10 2021-04-09T18:32:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.002 7f44aa7fc700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a0018930 0x7f44a001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.002 7f44aa7fc700 1 -- 172.21.15.32:0/2895457404 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f44b0032d80 con 0x7f44b819cd10 2021-04-09T18:32:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.002 7f44bd833700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a0018930 0x7f44a001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.002 7f44bd833700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a0018930 0x7f44a001adc0 crc :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:38.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.022 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b81b70c0 con 0x7f44b819cd10 2021-04-09T18:32:38.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.025 7f44aa7fc700 1 -- 172.21.15.32:0/2895457404 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f44b000be10 con 0x7f44b819cd10 2021-04-09T18:32:38.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.213 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44b80d7280 con 0x7f44b819cd10 2021-04-09T18:32:38.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.213 7f44aa7fc700 1 -- 172.21.15.32:0/2895457404 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) v1 ==== 74+0+14016 (secure 0 0 0) 0x7f44b0005870 con 0x7f44b819cd10 2021-04-09T18:32:38.231 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:38.231 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":225,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:32:37.406384","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:32:35.297513","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":39,"pool_name":"alpha_data","create_time":"2021-04-09 18:32:36.169493","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:32:38.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a0018930 msgr2=0x7f44a001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a0018930 0x7f44a001adc0 crc :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b819cd10 msgr2=0x7f44b819fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b819cd10 0x7f44b819fa80 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 rx=0x7f44b0009980 tx=0x7f44b000bcd0).stop 2021-04-09T18:32:38.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 shutdown_connections 2021-04-09T18:32:38.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44a0018930 0x7f44a001adc0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f44b819cd10 0x7f44b819fa80 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44b8093650 0x7f44b819f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 --2- 172.21.15.32:0/2895457404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44b819c910 0x7f44b8093030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 >> 172.21.15.32:0/2895457404 conn(0x7f44b81864e0 msgr2=0x7f44b81b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:38.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 shutdown_connections 2021-04-09T18:32:38.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.216 7f44bfa17700 1 -- 172.21.15.32:0/2895457404 wait complete. 2021-04-09T18:32:38.243 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set alpha standby_count_wanted 0 2021-04-09T18:32:38.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.378 7fb8550ac700 1 Processor -- start 2021-04-09T18:32:38.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.378 7fb8550ac700 1 -- start start 2021-04-09T18:32:38.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.379 7fb8550ac700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb85010d010 0x7fb8500d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.379 7fb8550ac700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb8500d1da0 0x7fb8500d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.379 7fb8550ac700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85010d480 con 0x7fb8500d82c0 2021-04-09T18:32:38.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.379 7fb8550ac700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8500d8d30 con 0x7fb8500d1da0 2021-04-09T18:32:38.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.379 7fb8550ac700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8500d9220 con 0x7fb85010d010 2021-04-09T18:32:38.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.379 7fb84e59c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb8500d1da0 0x7fb8500d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.379 7fb84f59e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb8500d82c0 0x7fb85010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39206/0 (socket says 172.21.15.32:39206) 2021-04-09T18:32:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.379 7fb84f59e700 1 -- 172.21.15.32:0/399036966 learned_addr learned my addr 172.21.15.32:0/399036966 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.379 7fb84ed9d700 1 --2- 172.21.15.32:0/399036966 >> v2:172.21.15.86:3300/0 conn(0x7fb85010d010 0x7fb8500d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb84ed9d700 1 -- 172.21.15.32:0/399036966 >> v2:172.21.15.32:3300/0 conn(0x7fb8500d1da0 msgr2=0x7fb8500d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:38.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb84ed9d700 1 --2- 172.21.15.32:0/399036966 >> v2:172.21.15.32:3300/0 conn(0x7fb8500d1da0 0x7fb8500d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb84ed9d700 1 -- 172.21.15.32:0/399036966 >> v1:172.21.15.32:6789/0 conn(0x7fb8500d82c0 legacy=0x7fb85010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb84ed9d700 1 -- 172.21.15.32:0/399036966 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8500d86f0 con 0x7fb85010d010 2021-04-09T18:32:38.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb84e59c700 1 --2- 172.21.15.32:0/399036966 >> v2:172.21.15.32:3300/0 conn(0x7fb8500d1da0 0x7fb8500d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:38.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb84ed9d700 1 --2- 172.21.15.32:0/399036966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb85010d010 0x7fb8500d1780 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7fb840006650 tx=0x7fb840009a80).ready entity=mon.1 client_cookie=52455282c1443ddb server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:38.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb84dd9b700 1 -- 172.21.15.32:0/399036966 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8400009b0 con 0x7fb85010d010 2021-04-09T18:32:38.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb84dd9b700 1 -- 172.21.15.32:0/399036966 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84000c0c0 con 0x7fb85010d010 2021-04-09T18:32:38.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb84dd9b700 1 -- 172.21.15.32:0/399036966 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb84000c510 con 0x7fb85010d010 2021-04-09T18:32:38.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb8550ac700 1 -- 172.21.15.32:0/399036966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb85010d010 msgr2=0x7fb8500d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.380 7fb8550ac700 1 --2- 172.21.15.32:0/399036966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb85010d010 0x7fb8500d1780 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7fb840006650 tx=0x7fb840009a80).stop 2021-04-09T18:32:38.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.381 7fb8550ac700 1 -- 172.21.15.32:0/399036966 shutdown_connections 2021-04-09T18:32:38.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.381 7fb8550ac700 1 --2- 172.21.15.32:0/399036966 >> v2:172.21.15.32:3300/0 conn(0x7fb8500d1da0 0x7fb8500d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.381 7fb8550ac700 1 --2- 172.21.15.32:0/399036966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb85010d010 0x7fb8500d1780 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.381 7fb8550ac700 1 -- 172.21.15.32:0/399036966 >> 172.21.15.32:0/399036966 conn(0x7fb8500ce5f0 msgr2=0x7fb8500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:38.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.381 7fb8550ac700 1 -- 172.21.15.32:0/399036966 shutdown_connections 2021-04-09T18:32:38.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.381 7fb8550ac700 1 -- 172.21.15.32:0/399036966 wait complete. 2021-04-09T18:32:38.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.401 7fb8550ac700 1 Processor -- start 2021-04-09T18:32:38.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.401 7fb8550ac700 1 -- start start 2021-04-09T18:32:38.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb8550ac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8500b1e50 0x7fb8500b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb8550ac700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8500b9fd0 0x7fb8500b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb8550ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8500b5510 0x7fb8500b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb8550ac700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8500b2250 con 0x7fb8500b5510 2021-04-09T18:32:38.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb8550ac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8500a1220 con 0x7fb8500b1e50 2021-04-09T18:32:38.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb8550ac700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8500b6b30 con 0x7fb8500b9fd0 2021-04-09T18:32:38.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84f59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8500b5510 0x7fb8500b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84f59e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8500b5510 0x7fb8500b5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57854/0 (socket says 172.21.15.32:57854) 2021-04-09T18:32:38.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84ed9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8500b1e50 0x7fb8500b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84e59c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8500b9fd0 0x7fb8500b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84f59e700 1 -- 172.21.15.32:0/1564498941 learned_addr learned my addr 172.21.15.32:0/1564498941 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:38.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84e59c700 1 -- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8500b1e50 msgr2=0x7fb8500b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84e59c700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8500b1e50 0x7fb8500b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84e59c700 1 -- 172.21.15.32:0/1564498941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8500b5510 msgr2=0x7fb8500b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:38.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84e59c700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8500b5510 0x7fb8500b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:38.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84e59c700 1 -- 172.21.15.32:0/1564498941 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8500b5f30 con 0x7fb8500b9fd0 2021-04-09T18:32:38.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84f59e700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8500b5510 0x7fb8500b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.402 7fb84e59c700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8500b9fd0 0x7fb8500b4ef0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fb84400b4e0 tx=0x7fb84400b880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.403 7fb83f7fe700 1 -- 172.21.15.32:0/1564498941 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb84400bc70 con 0x7fb8500b9fd0 2021-04-09T18:32:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.403 7fb83f7fe700 1 -- 172.21.15.32:0/1564498941 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb844002030 con 0x7fb8500b9fd0 2021-04-09T18:32:38.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.403 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8500b5f30 con 0x7fb8500b9fd0 2021-04-09T18:32:38.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.403 7fb83f7fe700 1 -- 172.21.15.32:0/1564498941 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb844002510 con 0x7fb8500b9fd0 2021-04-09T18:32:38.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.403 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8500b5f30 con 0x7fb8500b9fd0 2021-04-09T18:32:38.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.403 7fb83f7fe700 1 -- 172.21.15.32:0/1564498941 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb844000c90 con 0x7fb8500b9fd0 2021-04-09T18:32:38.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.403 7fb83f7fe700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb834018930 0x7fb83401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.403 7fb83f7fe700 1 -- 172.21.15.32:0/1564498941 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb844032970 con 0x7fb8500b9fd0 2021-04-09T18:32:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.403 7fb84ed9d700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb834018930 0x7fb83401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:38.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.404 7fb84ed9d700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb834018930 0x7fb83401ae10 crc :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:38.440 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.423 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8501b7040 con 0x7fb8500b9fd0 2021-04-09T18:32:38.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.427 7fb83f7fe700 1 -- 172.21.15.32:0/1564498941 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb84400bc70 con 0x7fb8500b9fd0 2021-04-09T18:32:38.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:32:38.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:32:38.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:38.624 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "alpha", "val": "0"} v 0) v1 -- 0x7fb85010d480 con 0x7fb8500b9fd0 2021-04-09T18:32:39.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.410 7fb83f7fe700 1 -- 172.21.15.32:0/1564498941 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "alpha", "val": "0"}]=0 v381) v1 ==== 117+0+0 (secure 0 0 0) 0x7fb844000c90 con 0x7fb8500b9fd0 2021-04-09T18:32:39.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb834018930 msgr2=0x7fb83401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:39.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb834018930 0x7fb83401ae10 crc :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8500b9fd0 msgr2=0x7fb8500b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:39.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8500b9fd0 0x7fb8500b4ef0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fb84400b4e0 tx=0x7fb84400b880).stop 2021-04-09T18:32:39.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 shutdown_connections 2021-04-09T18:32:39.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb834018930 0x7fb83401ae10 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8500b5510 0x7fb8500b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8500b9fd0 0x7fb8500b4ef0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 --2- 172.21.15.32:0/1564498941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8500b1e50 0x7fb8500b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 >> 172.21.15.32:0/1564498941 conn(0x7fb8500b27e0 msgr2=0x7fb8501b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:39.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 shutdown_connections 2021-04-09T18:32:39.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.442 7fb8550ac700 1 -- 172.21.15.32:0/1564498941 wait complete. 2021-04-09T18:32:39.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:39.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.616 7fbb033ec700 1 Processor -- start 2021-04-09T18:32:39.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb033ec700 1 -- start start 2021-04-09T18:32:39.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb033ec700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc0cffa0 0x7fbafc0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:39.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb033ec700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbafc0d7800 0x7fbafc0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:39.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb033ec700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbafc0d5c70 0x7fbafc0d6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:39.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb033ec700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbafc0d8580 con 0x7fbafc0d7800 2021-04-09T18:32:39.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb033ec700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbafc0d91e0 con 0x7fbafc0d5c70 2021-04-09T18:32:39.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb033ec700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0d7280 con 0x7fbafc0cffa0 2021-04-09T18:32:39.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb00a07700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbafc0d7800 0x7fbafc0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:39.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb00a07700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbafc0d7800 0x7fbafc0d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57862/0 (socket says 172.21.15.32:57862) 2021-04-09T18:32:39.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb00a07700 1 -- 172.21.15.32:0/3551708085 learned_addr learned my addr 172.21.15.32:0/3551708085 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:39.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb01a09700 1 --2- 172.21.15.32:0/3551708085 >> v2:172.21.15.86:3300/0 conn(0x7fbafc0d5c70 0x7fbafc0d6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:39.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.617 7fbb01208700 1 --2- 172.21.15.32:0/3551708085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc0cffa0 0x7fbafc0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:39.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbb00a07700 1 -- 172.21.15.32:0/3551708085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc0cffa0 msgr2=0x7fbafc0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:39.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbb00a07700 1 --2- 172.21.15.32:0/3551708085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc0cffa0 0x7fbafc0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbb00a07700 1 -- 172.21.15.32:0/3551708085 >> v2:172.21.15.86:3300/0 conn(0x7fbafc0d5c70 msgr2=0x7fbafc0d6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:39.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbb00a07700 1 --2- 172.21.15.32:0/3551708085 >> v2:172.21.15.86:3300/0 conn(0x7fbafc0d5c70 0x7fbafc0d6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbb00a07700 1 -- 172.21.15.32:0/3551708085 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0d8220 con 0x7fbafc0d7800 2021-04-09T18:32:39.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbb01a09700 1 --2- 172.21.15.32:0/3551708085 >> v2:172.21.15.86:3300/0 conn(0x7fbafc0d5c70 0x7fbafc0d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:39.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbb00a07700 1 --2- 172.21.15.32:0/3551708085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc0d7800 0x7fbafc0d7c00 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 rx=0x7fbaec006700 tx=0x7fbaec009b00).ready entity=mon.0 client_cookie=86bc17b73283d5a5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:39.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbaf3fff700 1 -- 172.21.15.32:0/3551708085 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaec00c0c0 con 0x7fbafc0d7800 2021-04-09T18:32:39.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbaf3fff700 1 -- 172.21.15.32:0/3551708085 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec00c400 con 0x7fbafc0d7800 2021-04-09T18:32:39.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbaf3fff700 1 -- 172.21.15.32:0/3551708085 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaec00c8e0 con 0x7fbafc0d7800 2021-04-09T18:32:39.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbb033ec700 1 -- 172.21.15.32:0/3551708085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc0d7800 msgr2=0x7fbafc0d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:39.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.618 7fbb033ec700 1 --2- 172.21.15.32:0/3551708085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc0d7800 0x7fbafc0d7c00 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 rx=0x7fbaec006700 tx=0x7fbaec009b00).stop 2021-04-09T18:32:39.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.619 7fbb033ec700 1 -- 172.21.15.32:0/3551708085 shutdown_connections 2021-04-09T18:32:39.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.619 7fbb033ec700 1 --2- 172.21.15.32:0/3551708085 >> v2:172.21.15.86:3300/0 conn(0x7fbafc0d5c70 0x7fbafc0d6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.619 7fbb033ec700 1 --2- 172.21.15.32:0/3551708085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc0d7800 0x7fbafc0d7c00 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.619 7fbb033ec700 1 --2- 172.21.15.32:0/3551708085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc0cffa0 0x7fbafc0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.619 7fbb033ec700 1 -- 172.21.15.32:0/3551708085 >> 172.21.15.32:0/3551708085 conn(0x7fbafc0ce5c0 msgr2=0x7fbafc0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:39.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.619 7fbb033ec700 1 -- 172.21.15.32:0/3551708085 shutdown_connections 2021-04-09T18:32:39.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.619 7fbb033ec700 1 -- 172.21.15.32:0/3551708085 wait complete. 2021-04-09T18:32:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.639 7fbb033ec700 1 Processor -- start 2021-04-09T18:32:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.639 7fbb033ec700 1 -- start start 2021-04-09T18:32:39.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb033ec700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc1acff0 0x7fbafc08fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:39.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb033ec700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc090150 0x7fbafc08c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:39.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb033ec700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbafc08cca0 0x7fbafc08e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:39.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb033ec700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc08c100 con 0x7fbafc1acff0 2021-04-09T18:32:39.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb033ec700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc08d0a0 con 0x7fbafc08cca0 2021-04-09T18:32:39.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb033ec700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc08e960 con 0x7fbafc090150 2021-04-09T18:32:39.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01208700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc1acff0 0x7fbafc08fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:39.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01208700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc1acff0 0x7fbafc08fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57864/0 (socket says 172.21.15.32:57864) 2021-04-09T18:32:39.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01208700 1 -- 172.21.15.32:0/2857559757 learned_addr learned my addr 172.21.15.32:0/2857559757 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:39.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01a09700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbafc08cca0 0x7fbafc08e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:39.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb00a07700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc090150 0x7fbafc08c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:39.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01208700 1 -- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc090150 msgr2=0x7fbafc08c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:39.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01208700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc090150 0x7fbafc08c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01208700 1 -- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbafc08cca0 msgr2=0x7fbafc08e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01208700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbafc08cca0 0x7fbafc08e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01208700 1 -- 172.21.15.32:0/2857559757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc08eb80 con 0x7fbafc1acff0 2021-04-09T18:32:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbb01208700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc1acff0 0x7fbafc08fb30 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 rx=0x7fbaf800b560 tx=0x7fbaf800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.640 7fbaf1ffb700 1 -- 172.21.15.32:0/2857559757 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaf800bcf0 con 0x7fbafc1acff0 2021-04-09T18:32:39.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.641 7fbaf1ffb700 1 -- 172.21.15.32:0/2857559757 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf8000c90 con 0x7fbafc1acff0 2021-04-09T18:32:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.641 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc08eb80 con 0x7fbafc1acff0 2021-04-09T18:32:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.641 7fbaf1ffb700 1 -- 172.21.15.32:0/2857559757 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaf8002220 con 0x7fbafc1acff0 2021-04-09T18:32:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.641 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc08eb80 con 0x7fbafc1acff0 2021-04-09T18:32:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.641 7fbaf1ffb700 1 -- 172.21.15.32:0/2857559757 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbaf80052e0 con 0x7fbafc1acff0 2021-04-09T18:32:39.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.641 7fbaf1ffb700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc01cd40 0x7fbadc01f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.641 7fbaf1ffb700 1 -- 172.21.15.32:0/2857559757 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbaf8032a20 con 0x7fbafc1acff0 2021-04-09T18:32:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.641 7fbb00a07700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc01cd40 0x7fbadc01f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:39.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.642 7fbb00a07700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc01cd40 0x7fbadc01f1d0 crc :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:39.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.661 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbafc182dc0 con 0x7fbafc1acff0 2021-04-09T18:32:39.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.665 7fbaf1ffb700 1 -- 172.21.15.32:0/2857559757 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbaf8000c90 con 0x7fbafc1acff0 2021-04-09T18:32:39.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.856 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbafc0d7280 con 0x7fbafc1acff0 2021-04-09T18:32:39.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.857 7fbaf1ffb700 1 -- 172.21.15.32:0/2857559757 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 381 v381) v1 ==== 95+0+2788 (secure 0 0 0) 0x7fbaf8006b00 con 0x7fbafc1acff0 2021-04-09T18:32:39.874 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 381 2021-04-09T18:32:39.874 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:39.875 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":381,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2051228504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2051228504},{"type":"v1","addr":"172.21.15.86:6837","nonce":2051228504}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378},{"gid":11512,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3237972186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3237972186},{"type":"v1","addr":"172.21.15.32:6834","nonce":3237972186}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378},{"gid":11515,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4276774592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4276774592},{"type":"v1","addr":"172.21.15.32:6835","nonce":4276774592}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378}],"filesystems":[{"mdsmap":{"epoch":381,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:39.408909","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11500":{"gid":11500,"name":"b","rank":0,"incarnation":379,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3107210651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3107210651},{"type":"v1","addr":"172.21.15.86:6836","nonce":3107210651}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19}]} 2021-04-09T18:32:39.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.859 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc01cd40 msgr2=0x7fbadc01f1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:39.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.859 7fbb033ec700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc01cd40 0x7fbadc01f1d0 crc :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.859 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc1acff0 msgr2=0x7fbafc08fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:39.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.859 7fbb033ec700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc1acff0 0x7fbafc08fb30 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 rx=0x7fbaf800b560 tx=0x7fbaf800b900).stop 2021-04-09T18:32:39.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.860 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 shutdown_connections 2021-04-09T18:32:39.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.860 7fbb033ec700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbadc01cd40 0x7fbadc01f1d0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.860 7fbb033ec700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbafc08cca0 0x7fbafc08e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.860 7fbb033ec700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbafc090150 0x7fbafc08c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.860 7fbb033ec700 1 --2- 172.21.15.32:0/2857559757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbafc1acff0 0x7fbafc08fb30 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:39.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.860 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 >> 172.21.15.32:0/2857559757 conn(0x7fbafc09abb0 msgr2=0x7fbafc0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:39.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.860 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 shutdown_connections 2021-04-09T18:32:39.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:39.860 7fbb033ec700 1 -- 172.21.15.32:0/2857559757 wait complete. 2021-04-09T18:32:39.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:32:40.062 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.018 7f2988080700 1 Processor -- start 2021-04-09T18:32:40.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.019 7f2988080700 1 -- start start 2021-04-09T18:32:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.019 7f2988080700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f29800d7ca0 0x7f29800d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.019 7f2988080700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298010d480 con 0x7f298010d010 2021-04-09T18:32:40.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.019 7f2988080700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29800d86c0 con 0x7f298010da00 2021-04-09T18:32:40.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.019 7f2988080700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29800d9220 con 0x7f29800d7ca0 2021-04-09T18:32:40.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2985e9c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f298010d010 0x7f29800d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39230/0 (socket says 172.21.15.32:39230) 2021-04-09T18:32:40.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2985e9c700 1 -- 172.21.15.32:0/3647345290 learned_addr learned my addr 172.21.15.32:0/3647345290 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:40.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f298569b700 1 --2- 172.21.15.32:0/3647345290 >> v2:172.21.15.86:3300/0 conn(0x7f29800d7ca0 0x7f29800d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2984e9a700 1 -- 172.21.15.32:0/3647345290 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348642378 0 0) 0x7f297c000ce0 con 0x7f298010d010 2021-04-09T18:32:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2984e9a700 1 -- 172.21.15.32:0/3647345290 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296c002fe0 con 0x7f298010d010 2021-04-09T18:32:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2984e9a700 1 -- 172.21.15.32:0/3647345290 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568222467 0 0) 0x7f2978000d30 con 0x7f298010da00 2021-04-09T18:32:40.066 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2984e9a700 1 -- 172.21.15.32:0/3647345290 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296c003020 con 0x7f298010da00 2021-04-09T18:32:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f298569b700 1 -- 172.21.15.32:0/3647345290 >> v1:172.21.15.86:6789/0 conn(0x7f298010da00 legacy=0x7f298010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f298569b700 1 -- 172.21.15.32:0/3647345290 >> v1:172.21.15.32:6789/0 conn(0x7f298010d010 legacy=0x7f29800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f298569b700 1 -- 172.21.15.32:0/3647345290 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29800d1f70 con 0x7f29800d7ca0 2021-04-09T18:32:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f298569b700 1 --2- 172.21.15.32:0/3647345290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29800d7ca0 0x7f29800d80a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f2970006650 tx=0x7f2970009a00).ready entity=mon.1 client_cookie=cb97cbb70722e939 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2984e9a700 1 -- 172.21.15.32:0/3647345290 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2970009df0 con 0x7f29800d7ca0 2021-04-09T18:32:40.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2984e9a700 1 -- 172.21.15.32:0/3647345290 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29700009b0 con 0x7f29800d7ca0 2021-04-09T18:32:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2984e9a700 1 -- 172.21.15.32:0/3647345290 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f297000c2b0 con 0x7f29800d7ca0 2021-04-09T18:32:40.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2988080700 1 -- 172.21.15.32:0/3647345290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29800d7ca0 msgr2=0x7f29800d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.020 7f2988080700 1 --2- 172.21.15.32:0/3647345290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29800d7ca0 0x7f29800d80a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f2970006650 tx=0x7f2970009a00).stop 2021-04-09T18:32:40.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.021 7f2988080700 1 -- 172.21.15.32:0/3647345290 shutdown_connections 2021-04-09T18:32:40.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.021 7f2988080700 1 --2- 172.21.15.32:0/3647345290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29800d7ca0 0x7f29800d80a0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.021 7f2988080700 1 -- 172.21.15.32:0/3647345290 >> 172.21.15.32:0/3647345290 conn(0x7f29800ce5f0 msgr2=0x7f29800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:40.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.021 7f2988080700 1 -- 172.21.15.32:0/3647345290 shutdown_connections 2021-04-09T18:32:40.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.021 7f2988080700 1 -- 172.21.15.32:0/3647345290 wait complete. 2021-04-09T18:32:40.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.041 7f2988080700 1 Processor -- start 2021-04-09T18:32:40.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.041 7f2988080700 1 -- start start 2021-04-09T18:32:40.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2988080700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2980185a00 0x7f2980097d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2988080700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29800983b0 0x7f298017abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2988080700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f298017b1d0 0x7f2980179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2988080700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29800987b0 con 0x7f2980185a00 2021-04-09T18:32:40.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2988080700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298017b5d0 con 0x7f298017b1d0 2021-04-09T18:32:40.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2988080700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2980179c30 con 0x7f29800983b0 2021-04-09T18:32:40.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2985e9c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2980185a00 0x7f2980097d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2985e9c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2980185a00 0x7f2980097d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57878/0 (socket says 172.21.15.32:57878) 2021-04-09T18:32:40.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2985e9c700 1 -- 172.21.15.32:0/4261822214 learned_addr learned my addr 172.21.15.32:0/4261822214 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:40.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f298569b700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29800983b0 0x7f298017abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f298669d700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f298017b1d0 0x7f2980179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2985e9c700 1 -- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29800983b0 msgr2=0x7f298017abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2985e9c700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29800983b0 0x7f298017abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2985e9c700 1 -- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f298017b1d0 msgr2=0x7f2980179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2985e9c700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f298017b1d0 0x7f2980179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2985e9c700 1 -- 172.21.15.32:0/4261822214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2980179e50 con 0x7f2980185a00 2021-04-09T18:32:40.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.042 7f2985e9c700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2980185a00 0x7f2980097d90 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7f297c00a5d0 tx=0x7f297c00a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:40.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.043 7f29767fc700 1 -- 172.21.15.32:0/4261822214 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f297c000ea0 con 0x7f2980185a00 2021-04-09T18:32:40.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.043 7f29767fc700 1 -- 172.21.15.32:0/4261822214 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297c00ad60 con 0x7f2980185a00 2021-04-09T18:32:40.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.043 7f2988080700 1 -- 172.21.15.32:0/4261822214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2980179e50 con 0x7f2980185a00 2021-04-09T18:32:40.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.043 7f29767fc700 1 -- 172.21.15.32:0/4261822214 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f297c004590 con 0x7f2980185a00 2021-04-09T18:32:40.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.043 7f2988080700 1 -- 172.21.15.32:0/4261822214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2980179e50 con 0x7f2980185a00 2021-04-09T18:32:40.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.043 7f29767fc700 1 -- 172.21.15.32:0/4261822214 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f297c0043a0 con 0x7f2980185a00 2021-04-09T18:32:40.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.043 7f29767fc700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2968018930 0x7f296801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.043 7f29767fc700 1 -- 172.21.15.32:0/4261822214 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f297c032a00 con 0x7f2980185a00 2021-04-09T18:32:40.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.044 7f298569b700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2968018930 0x7f296801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.044 7f298569b700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2968018930 0x7f296801ae10 crc :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:40.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.064 7f2988080700 1 -- 172.21.15.32:0/4261822214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29801b7010 con 0x7f2980185a00 2021-04-09T18:32:40.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.067 7f29767fc700 1 -- 172.21.15.32:0/4261822214 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f297c007620 con 0x7f2980185a00 2021-04-09T18:32:40.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.255 7f2988080700 1 -- 172.21.15.32:0/4261822214 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f298010d480 con 0x7f2980185a00 2021-04-09T18:32:40.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.256 7f29767fc700 1 -- 172.21.15.32:0/4261822214 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+14003 (secure 0 0 0) 0x7f297c0043a0 con 0x7f2980185a00 2021-04-09T18:32:40.273 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:40.274 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":226,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:32:38.407876","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:32:35.297513","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":39,"pool_name":"alpha_data","create_time":"2021-04-09 18:32:36.169493","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:32:40.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 -- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2968018930 msgr2=0x7f296801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2968018930 0x7f296801ae10 crc :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 -- 172.21.15.32:0/4261822214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2980185a00 msgr2=0x7f2980097d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2980185a00 0x7f2980097d90 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7f297c00a5d0 tx=0x7f297c00a970).stop 2021-04-09T18:32:40.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 -- 172.21.15.32:0/4261822214 shutdown_connections 2021-04-09T18:32:40.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2968018930 0x7f296801ae10 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f298017b1d0 0x7f2980179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29800983b0 0x7f298017abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 --2- 172.21.15.32:0/4261822214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2980185a00 0x7f2980097d90 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.258 7f2988080700 1 -- 172.21.15.32:0/4261822214 >> 172.21.15.32:0/4261822214 conn(0x7f2980190550 msgr2=0x7f29801b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:40.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.259 7f2988080700 1 -- 172.21.15.32:0/4261822214 shutdown_connections 2021-04-09T18:32:40.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.259 7f2988080700 1 -- 172.21.15.32:0/4261822214 wait complete. 2021-04-09T18:32:40.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:32:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.417 7fd53c7cd700 1 Processor -- start 2021-04-09T18:32:40.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.417 7fd53c7cd700 1 -- start start 2021-04-09T18:32:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.418 7fd53c7cd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd53410d010 0x7fd5340d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.418 7fd53c7cd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5340d1da0 0x7fd5340d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.418 7fd53c7cd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53410d480 con 0x7fd5340d82c0 2021-04-09T18:32:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.418 7fd53c7cd700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53410e780 con 0x7fd5340d1da0 2021-04-09T18:32:40.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.418 7fd53c7cd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5340d8d30 con 0x7fd53410d010 2021-04-09T18:32:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.418 7fd53adea700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd5340d82c0 0x7fd53410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39246/0 (socket says 172.21.15.32:39246) 2021-04-09T18:32:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.418 7fd53adea700 1 -- 172.21.15.32:0/2516325055 learned_addr learned my addr 172.21.15.32:0/2516325055 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.418 7fd539de8700 1 --2- 172.21.15.32:0/2516325055 >> v2:172.21.15.86:3300/0 conn(0x7fd5340d1da0 0x7fd5340d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.418 7fd53a5e9700 1 --2- 172.21.15.32:0/2516325055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd53410d010 0x7fd5340d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd5395e7700 1 -- 172.21.15.32:0/2516325055 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225575477 0 0) 0x7fd524000d30 con 0x7fd5340d82c0 2021-04-09T18:32:40.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd5395e7700 1 -- 172.21.15.32:0/2516325055 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c002fe0 con 0x7fd5340d82c0 2021-04-09T18:32:40.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd5395e7700 1 -- 172.21.15.32:0/2516325055 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2251443854 0 0) 0x7fd524001270 con 0x7fd5340d82c0 2021-04-09T18:32:40.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd539de8700 1 -- 172.21.15.32:0/2516325055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd53410d010 msgr2=0x7fd5340d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd539de8700 1 --2- 172.21.15.32:0/2516325055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd53410d010 0x7fd5340d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd539de8700 1 -- 172.21.15.32:0/2516325055 >> v1:172.21.15.32:6789/0 conn(0x7fd5340d82c0 legacy=0x7fd53410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd539de8700 1 -- 172.21.15.32:0/2516325055 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5340d9220 con 0x7fd5340d1da0 2021-04-09T18:32:40.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd53a5e9700 1 --2- 172.21.15.32:0/2516325055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd53410d010 0x7fd5340d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd539de8700 1 --2- 172.21.15.32:0/2516325055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5340d1da0 0x7fd5340d7ca0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7fd528006650 tx=0x7fd528009a80).ready entity=mon.1 client_cookie=3a76f47fc580ec45 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd5395e7700 1 -- 172.21.15.32:0/2516325055 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5280009b0 con 0x7fd5340d1da0 2021-04-09T18:32:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd5395e7700 1 -- 172.21.15.32:0/2516325055 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52800c0c0 con 0x7fd5340d1da0 2021-04-09T18:32:40.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd5395e7700 1 -- 172.21.15.32:0/2516325055 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd52800c510 con 0x7fd5340d1da0 2021-04-09T18:32:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd53c7cd700 1 -- 172.21.15.32:0/2516325055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5340d1da0 msgr2=0x7fd5340d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.419 7fd53c7cd700 1 --2- 172.21.15.32:0/2516325055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5340d1da0 0x7fd5340d7ca0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7fd528006650 tx=0x7fd528009a80).stop 2021-04-09T18:32:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.420 7fd53c7cd700 1 -- 172.21.15.32:0/2516325055 shutdown_connections 2021-04-09T18:32:40.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.420 7fd53c7cd700 1 --2- 172.21.15.32:0/2516325055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5340d1da0 0x7fd5340d7ca0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.420 7fd53c7cd700 1 --2- 172.21.15.32:0/2516325055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd53410d010 0x7fd5340d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.420 7fd53c7cd700 1 -- 172.21.15.32:0/2516325055 >> 172.21.15.32:0/2516325055 conn(0x7fd5340ce5f0 msgr2=0x7fd5340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:40.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.420 7fd53c7cd700 1 -- 172.21.15.32:0/2516325055 shutdown_connections 2021-04-09T18:32:40.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.420 7fd53c7cd700 1 -- 172.21.15.32:0/2516325055 wait complete. 2021-04-09T18:32:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.440 7fd53c7cd700 1 Processor -- start 2021-04-09T18:32:40.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53c7cd700 1 -- start start 2021-04-09T18:32:40.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53c7cd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd534184a30 0x7fd5341acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53c7cd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5341a6b90 0x7fd5341a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53c7cd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5341a5150 0x7fd5341a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53c7cd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5341a75b0 con 0x7fd534184a30 2021-04-09T18:32:40.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53c7cd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd534184590 con 0x7fd5341a5150 2021-04-09T18:32:40.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53c7cd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd534184740 con 0x7fd5341a6b90 2021-04-09T18:32:40.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53a5e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd534184a30 0x7fd5341acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53a5e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd534184a30 0x7fd5341acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57894/0 (socket says 172.21.15.32:57894) 2021-04-09T18:32:40.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53a5e9700 1 -- 172.21.15.32:0/2974421755 learned_addr learned my addr 172.21.15.32:0/2974421755 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:40.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd539de8700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5341a6b90 0x7fd5341a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53adea700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5341a5150 0x7fd5341a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53a5e9700 1 -- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5341a6b90 msgr2=0x7fd5341a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53a5e9700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5341a6b90 0x7fd5341a6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53a5e9700 1 -- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5341a5150 msgr2=0x7fd5341a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53a5e9700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5341a5150 0x7fd5341a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53a5e9700 1 -- 172.21.15.32:0/2974421755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5341a5b70 con 0x7fd534184a30 2021-04-09T18:32:40.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.441 7fd53adea700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5341a5150 0x7fd5341a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:40.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.442 7fd53a5e9700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd534184a30 0x7fd5341acb80 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7fd52c006f40 tx=0x7fd52c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:40.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.442 7fd532ffd700 1 -- 172.21.15.32:0/2974421755 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd52c00bd80 con 0x7fd534184a30 2021-04-09T18:32:40.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.442 7fd532ffd700 1 -- 172.21.15.32:0/2974421755 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c000c90 con 0x7fd534184a30 2021-04-09T18:32:40.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.442 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5341a5b70 con 0x7fd534184a30 2021-04-09T18:32:40.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.442 7fd532ffd700 1 -- 172.21.15.32:0/2974421755 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd52c002220 con 0x7fd534184a30 2021-04-09T18:32:40.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.442 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5341a5b70 con 0x7fd534184a30 2021-04-09T18:32:40.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.442 7fd532ffd700 1 -- 172.21.15.32:0/2974421755 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd52c00a000 con 0x7fd534184a30 2021-04-09T18:32:40.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.442 7fd532ffd700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd514018930 0x7fd51401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.443 7fd532ffd700 1 -- 172.21.15.32:0/2974421755 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd52c0329e0 con 0x7fd534184a30 2021-04-09T18:32:40.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.443 7fd539de8700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd514018930 0x7fd51401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.443 7fd539de8700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd514018930 0x7fd51401ae10 crc :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:40.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.462 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5341b7080 con 0x7fd534184a30 2021-04-09T18:32:40.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.466 7fd532ffd700 1 -- 172.21.15.32:0/2974421755 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd52c000c90 con 0x7fd534184a30 2021-04-09T18:32:40.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.655 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd53410d480 con 0x7fd534184a30 2021-04-09T18:32:40.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.656 7fd532ffd700 1 -- 172.21.15.32:0/2974421755 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+14003 (secure 0 0 0) 0x7fd52c006b00 con 0x7fd534184a30 2021-04-09T18:32:40.673 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:40.673 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":226,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:32:38.407876","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:32:35.297513","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":39,"pool_name":"alpha_data","create_time":"2021-04-09 18:32:36.169493","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:32:40.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd514018930 msgr2=0x7fd51401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd514018930 0x7fd51401ae10 crc :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd534184a30 msgr2=0x7fd5341acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd534184a30 0x7fd5341acb80 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7fd52c006f40 tx=0x7fd52c00b990).stop 2021-04-09T18:32:40.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 shutdown_connections 2021-04-09T18:32:40.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd514018930 0x7fd51401ae10 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5341a5150 0x7fd5341a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5341a6b90 0x7fd5341a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 --2- 172.21.15.32:0/2974421755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd534184a30 0x7fd5341acb80 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 >> 172.21.15.32:0/2974421755 conn(0x7fd534191560 msgr2=0x7fd5341b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:40.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 shutdown_connections 2021-04-09T18:32:40.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.658 7fd53c7cd700 1 -- 172.21.15.32:0/2974421755 wait complete. 2021-04-09T18:32:40.685 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 38, 'pool_name': 'alpha_metadata', 'create_time': '2021-04-09 18:32:35.297513', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'alpha'}}}, {'pool': 39, 'pool_name': 'alpha_data', 'create_time': '2021-04-09 18:32:36.169493', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'alpha'}}}] 2021-04-09T18:32:40.686 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:32:40.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.821 7f6e63b91700 1 Processor -- start 2021-04-09T18:32:40.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.821 7f6e63b91700 1 -- start start 2021-04-09T18:32:40.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.822 7f6e63b91700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0d7ca0 0x7f6e5c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.822 7f6e63b91700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c10d480 con 0x7f6e5c10da00 2021-04-09T18:32:40.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.822 7f6e63b91700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0d86c0 con 0x7f6e5c10d010 2021-04-09T18:32:40.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.822 7f6e63b91700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0d9220 con 0x7f6e5c0d7ca0 2021-04-09T18:32:40.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.822 7f6e621ae700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6e5c10da00 0x7f6e5c10de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39260/0 (socket says 172.21.15.32:39260) 2021-04-09T18:32:40.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.822 7f6e621ae700 1 -- 172.21.15.32:0/4100406911 learned_addr learned my addr 172.21.15.32:0/4100406911 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:40.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.822 7f6e611ac700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0d7ca0 0x7f6e5c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.822 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984973509 0 0) 0x7f6e54000d30 con 0x7f6e5c10da00 2021-04-09T18:32:40.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.822 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48002fe0 con 0x7f6e5c10da00 2021-04-09T18:32:40.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720596068 0 0) 0x7f6e58000c60 con 0x7f6e5c10d010 2021-04-09T18:32:40.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48003210 con 0x7f6e5c10d010 2021-04-09T18:32:40.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 423357480 0 0) 0x7f6e54001270 con 0x7f6e5c10da00 2021-04-09T18:32:40.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0d7ca0 msgr2=0x7f6e5c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:40.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0d7ca0 0x7f6e5c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 >> v1:172.21.15.86:6789/0 conn(0x7f6e5c10d010 legacy=0x7f6e5c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c112100 con 0x7f6e5c10da00 2021-04-09T18:32:40.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f6e54001980 con 0x7f6e5c10da00 2021-04-09T18:32:40.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e611ac700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0d7ca0 0x7f6e5c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:40.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 >> v1:172.21.15.32:6789/0 conn(0x7f6e5c10da00 legacy=0x7f6e5c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e48003b90 0x7f6e48005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e48006770 0x7f6e48008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e48009510 0x7f6e4800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e611ac700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e48006770 0x7f6e48008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e480502f0 con 0x7f6e48006770 2021-04-09T18:32:40.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e48050510 con 0x7f6e48009510 2021-04-09T18:32:40.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e48050730 con 0x7f6e48003b90 2021-04-09T18:32:40.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e619ad700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e48003b90 0x7f6e48005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e621ae700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e48009510 0x7f6e4800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e611ac700 1 -- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e48003b90 msgr2=0x7f6e48005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e611ac700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e48003b90 0x7f6e48005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e611ac700 1 -- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e48009510 msgr2=0x7f6e4800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e611ac700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e48009510 0x7f6e4800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e611ac700 1 -- 172.21.15.32:0/4100406911 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480509d0 con 0x7f6e48006770 2021-04-09T18:32:40.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e611ac700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e48006770 0x7f6e48008c70 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 rx=0x7f6e50006aa0 tx=0x7f6e50009ad0).ready entity=mon.0 client_cookie=d9395e3b25d11f02 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:40.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e500009b0 con 0x7f6e48006770 2021-04-09T18:32:40.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e50006770 con 0x7f6e48006770 2021-04-09T18:32:40.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.823 7f6e609ab700 1 -- 172.21.15.32:0/4100406911 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e5000c2b0 con 0x7f6e48006770 2021-04-09T18:32:40.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.824 7f6e63b91700 1 -- 172.21.15.32:0/4100406911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e48006770 msgr2=0x7f6e48008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.824 7f6e63b91700 1 --2- 172.21.15.32:0/4100406911 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e48006770 0x7f6e48008c70 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 rx=0x7f6e50006aa0 tx=0x7f6e50009ad0).stop 2021-04-09T18:32:40.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.824 7f6e621ae700 1 -- 172.21.15.32:0/4100406911 reap_dead start 2021-04-09T18:32:40.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.824 7f6e63b91700 1 -- 172.21.15.32:0/4100406911 shutdown_connections 2021-04-09T18:32:40.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.824 7f6e63b91700 1 -- 172.21.15.32:0/4100406911 >> 172.21.15.32:0/4100406911 conn(0x7f6e5c0ce5f0 msgr2=0x7f6e5c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:40.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.824 7f6e63b91700 1 -- 172.21.15.32:0/4100406911 shutdown_connections 2021-04-09T18:32:40.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.824 7f6e63b91700 1 -- 172.21.15.32:0/4100406911 wait complete. 2021-04-09T18:32:40.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e63b91700 1 Processor -- start 2021-04-09T18:32:40.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e63b91700 1 -- start start 2021-04-09T18:32:40.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e63b91700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c1a04d0 0x7f6e5c0a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e63b91700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0a6e00 0x7f6e5c0a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e63b91700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0a4a20 0x7f6e5c0a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e63b91700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c098720 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e63b91700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0a6290 con 0x7f6e5c0a4a20 2021-04-09T18:32:40.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e63b91700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0a7200 con 0x7f6e5c0a6e00 2021-04-09T18:32:40.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e619ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c1a04d0 0x7f6e5c0a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e619ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c1a04d0 0x7f6e5c0a67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57912/0 (socket says 172.21.15.32:57912) 2021-04-09T18:32:40.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e619ad700 1 -- 172.21.15.32:0/2254067709 learned_addr learned my addr 172.21.15.32:0/2254067709 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:40.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e621ae700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0a4a20 0x7f6e5c0a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.845 7f6e611ac700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0a6e00 0x7f6e5c0a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e619ad700 1 -- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0a6e00 msgr2=0x7f6e5c0a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e619ad700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0a6e00 0x7f6e5c0a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e619ad700 1 -- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0a4a20 msgr2=0x7f6e5c0a4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:40.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e619ad700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0a4a20 0x7f6e5c0a4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:40.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e619ad700 1 -- 172.21.15.32:0/2254067709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c0a1820 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e619ad700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c1a04d0 0x7f6e5c0a67e0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 rx=0x7f6e580059f0 tx=0x7f6e5800a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:40.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e4e7fc700 1 -- 172.21.15.32:0/2254067709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e5800ac90 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e4e7fc700 1 -- 172.21.15.32:0/2254067709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e58000c10 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e4e7fc700 1 -- 172.21.15.32:0/2254067709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e580022b0 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c0a1820 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.846 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c0a1820 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.847 7f6e4e7fc700 1 -- 172.21.15.32:0/2254067709 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6e58006760 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.847 7f6e4e7fc700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e44018930 0x7f6e4401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:40.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.847 7f6e4e7fc700 1 -- 172.21.15.32:0/2254067709 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6e58032a10 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.847 7f6e611ac700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e44018930 0x7f6e4401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:40.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.847 7f6e611ac700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e44018930 0x7f6e4401ae10 crc :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:40.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.866 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c098720 con 0x7f6e5c1a04d0 2021-04-09T18:32:40.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:40.870 7f6e4e7fc700 1 -- 172.21.15.32:0/2254067709 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6e58000c10 con 0x7f6e5c1a04d0 2021-04-09T18:32:41.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.047 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f6e5c098720 con 0x7f6e5c1a04d0 2021-04-09T18:32:41.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.048 7f6e4e7fc700 1 -- 172.21.15.32:0/2254067709 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v226) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6e58000e20 con 0x7f6e5c1a04d0 2021-04-09T18:32:41.065 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:32:41.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.050 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e44018930 msgr2=0x7f6e4401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.050 7f6e63b91700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e44018930 0x7f6e4401ae10 crc :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.050 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c1a04d0 msgr2=0x7f6e5c0a67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.050 7f6e63b91700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c1a04d0 0x7f6e5c0a67e0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 rx=0x7f6e580059f0 tx=0x7f6e5800a8a0).stop 2021-04-09T18:32:41.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.050 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 shutdown_connections 2021-04-09T18:32:41.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.050 7f6e63b91700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6e44018930 0x7f6e4401ae10 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.051 7f6e63b91700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e5c0a4a20 0x7f6e5c0a4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.051 7f6e63b91700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e5c0a6e00 0x7f6e5c0a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.051 7f6e63b91700 1 --2- 172.21.15.32:0/2254067709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e5c1a04d0 0x7f6e5c0a67e0 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.051 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 >> 172.21.15.32:0/2254067709 conn(0x7f6e5c0a22e0 msgr2=0x7f6e5c094d60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:41.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.051 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 shutdown_connections 2021-04-09T18:32:41.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.051 7f6e63b91700 1 -- 172.21.15.32:0/2254067709 wait complete. 2021-04-09T18:32:41.078 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_metadata pg_num 2021-04-09T18:32:41.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.214 7f3ae56b0700 1 Processor -- start 2021-04-09T18:32:41.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.214 7f3ae56b0700 1 -- start start 2021-04-09T18:32:41.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3ae56b0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae010d010 0x7f3ae00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3ae56b0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ae00d1da0 0x7f3ae00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3ae56b0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae010d480 con 0x7f3ae00d82c0 2021-04-09T18:32:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3ae56b0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae010e780 con 0x7f3ae00d1da0 2021-04-09T18:32:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3ae56b0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00d8d30 con 0x7f3ae010d010 2021-04-09T18:32:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adf2bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae010d010 0x7f3ae00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adeabe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ae00d1da0 0x7f3ae00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adfac0700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3ae00d82c0 0x7f3ae010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37206/0 (socket says 172.21.15.32:37206) 2021-04-09T18:32:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adfac0700 1 -- 172.21.15.32:0/2693954997 learned_addr learned my addr 172.21.15.32:0/2693954997 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3ade2bd700 1 -- 172.21.15.32:0/2693954997 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409458171 0 0) 0x7f3ad8001440 con 0x7f3ae00d82c0 2021-04-09T18:32:41.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adf2bf700 1 -- 172.21.15.32:0/2693954997 >> v2:172.21.15.86:3300/0 conn(0x7f3ae00d1da0 msgr2=0x7f3ae00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adf2bf700 1 --2- 172.21.15.32:0/2693954997 >> v2:172.21.15.86:3300/0 conn(0x7f3ae00d1da0 0x7f3ae00d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adf2bf700 1 -- 172.21.15.32:0/2693954997 >> v1:172.21.15.86:6789/0 conn(0x7f3ae00d82c0 legacy=0x7f3ae010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adf2bf700 1 -- 172.21.15.32:0/2693954997 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00d9220 con 0x7f3ae010d010 2021-04-09T18:32:41.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adeabe700 1 --2- 172.21.15.32:0/2693954997 >> v2:172.21.15.86:3300/0 conn(0x7f3ae00d1da0 0x7f3ae00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.215 7f3adf2bf700 1 --2- 172.21.15.32:0/2693954997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae010d010 0x7f3ae00d1780 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f3ad0006650 tx=0x7f3ad0009a80).ready entity=mon.2 client_cookie=b5aa84e6c54f674f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ade2bd700 1 -- 172.21.15.32:0/2693954997 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ad00009b0 con 0x7f3ae010d010 2021-04-09T18:32:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ade2bd700 1 -- 172.21.15.32:0/2693954997 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad000c0c0 con 0x7f3ae010d010 2021-04-09T18:32:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ade2bd700 1 -- 172.21.15.32:0/2693954997 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ad000c510 con 0x7f3ae010d010 2021-04-09T18:32:41.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ae56b0700 1 -- 172.21.15.32:0/2693954997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae010d010 msgr2=0x7f3ae00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ae56b0700 1 --2- 172.21.15.32:0/2693954997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae010d010 0x7f3ae00d1780 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f3ad0006650 tx=0x7f3ad0009a80).stop 2021-04-09T18:32:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ae56b0700 1 -- 172.21.15.32:0/2693954997 shutdown_connections 2021-04-09T18:32:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ae56b0700 1 --2- 172.21.15.32:0/2693954997 >> v2:172.21.15.86:3300/0 conn(0x7f3ae00d1da0 0x7f3ae00d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ae56b0700 1 --2- 172.21.15.32:0/2693954997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae010d010 0x7f3ae00d1780 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ae56b0700 1 -- 172.21.15.32:0/2693954997 >> 172.21.15.32:0/2693954997 conn(0x7f3ae00ce5f0 msgr2=0x7f3ae00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ae56b0700 1 -- 172.21.15.32:0/2693954997 shutdown_connections 2021-04-09T18:32:41.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.216 7f3ae56b0700 1 -- 172.21.15.32:0/2693954997 wait complete. 2021-04-09T18:32:41.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.236 7f3ae56b0700 1 Processor -- start 2021-04-09T18:32:41.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3ae56b0700 1 -- start start 2021-04-09T18:32:41.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3ae56b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ae01a0050 0x7f3ae01a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3ae56b0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae00bdcd0 0x7f3ae00bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3ae56b0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae00bf080 0x7f3ae00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3ae56b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00bf500 con 0x7f3ae01a0050 2021-04-09T18:32:41.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3ae56b0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00bb7b0 con 0x7f3ae00bdcd0 2021-04-09T18:32:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3ae56b0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00be0d0 con 0x7f3ae00bf080 2021-04-09T18:32:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3adf2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ae01a0050 0x7f3ae01a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3adf2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ae01a0050 0x7f3ae01a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57926/0 (socket says 172.21.15.32:57926) 2021-04-09T18:32:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3adf2bf700 1 -- 172.21.15.32:0/293617380 learned_addr learned my addr 172.21.15.32:0/293617380 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:41.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3adeabe700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae00bdcd0 0x7f3ae00bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.237 7f3adfac0700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae00bf080 0x7f3ae00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3adf2bf700 1 -- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae00bf080 msgr2=0x7f3ae00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3adf2bf700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae00bf080 0x7f3ae00b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3adf2bf700 1 -- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae00bdcd0 msgr2=0x7f3ae00bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3adf2bf700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae00bdcd0 0x7f3ae00bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3adf2bf700 1 -- 172.21.15.32:0/293617380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00b1310 con 0x7f3ae01a0050 2021-04-09T18:32:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3adeabe700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae00bdcd0 0x7f3ae00bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3adf2bf700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ae01a0050 0x7f3ae01a0450 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 rx=0x7f3ad0009650 tx=0x7f3ad000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:41.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3ac7fff700 1 -- 172.21.15.32:0/293617380 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ad000bd50 con 0x7f3ae01a0050 2021-04-09T18:32:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3ac7fff700 1 -- 172.21.15.32:0/293617380 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad0006030 con 0x7f3ae01a0050 2021-04-09T18:32:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae00b1310 con 0x7f3ae01a0050 2021-04-09T18:32:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3ac7fff700 1 -- 172.21.15.32:0/293617380 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ad00054d0 con 0x7f3ae01a0050 2021-04-09T18:32:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.238 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae00b1310 con 0x7f3ae01a0050 2021-04-09T18:32:41.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.239 7f3ac7fff700 1 -- 172.21.15.32:0/293617380 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3ad0009300 con 0x7f3ae01a0050 2021-04-09T18:32:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.239 7f3ac7fff700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ac00188c0 0x7f3ac001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.239 7f3ac7fff700 1 -- 172.21.15.32:0/293617380 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ad0032770 con 0x7f3ae01a0050 2021-04-09T18:32:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.239 7f3adeabe700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ac00188c0 0x7f3ac001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.240 7f3adeabe700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ac00188c0 0x7f3ac001adc0 crc :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:41.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.259 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae00bb7b0 con 0x7f3ae01a0050 2021-04-09T18:32:41.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.262 7f3ac7fff700 1 -- 172.21.15.32:0/293617380 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3ad0000940 con 0x7f3ae01a0050 2021-04-09T18:32:41.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.442 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"} v 0) v1 -- 0x7f3ae00bb7b0 con 0x7f3ae01a0050 2021-04-09T18:32:41.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.442 7f3ac7fff700 1 -- 172.21.15.32:0/293617380 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_metadata"}]=0 v226) v1 ==== 103+0+10 (secure 0 0 0) 0x7f3ad000bd50 con 0x7f3ae01a0050 2021-04-09T18:32:41.459 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:32:41.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ac00188c0 msgr2=0x7f3ac001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ac00188c0 0x7f3ac001adc0 crc :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ae01a0050 msgr2=0x7f3ae01a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ae01a0050 0x7f3ae01a0450 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 rx=0x7f3ad0009650 tx=0x7f3ad000b960).stop 2021-04-09T18:32:41.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 shutdown_connections 2021-04-09T18:32:41.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3ac00188c0 0x7f3ac001adc0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae00bf080 0x7f3ae00b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae00bdcd0 0x7f3ae00bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 --2- 172.21.15.32:0/293617380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ae01a0050 0x7f3ae01a0450 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 >> 172.21.15.32:0/293617380 conn(0x7f3ae00975f0 msgr2=0x7f3ae0096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:41.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 shutdown_connections 2021-04-09T18:32:41.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.445 7f3ae56b0700 1 -- 172.21.15.32:0/293617380 wait complete. 2021-04-09T18:32:41.472 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get alpha_data pg_num 2021-04-09T18:32:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.608 7fed3f1eb700 1 Processor -- start 2021-04-09T18:32:41.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.608 7fed3f1eb700 1 -- start start 2021-04-09T18:32:41.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3f1eb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed3810da00 0x7fed3810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3f1eb700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3810d480 con 0x7fed380d7ca0 2021-04-09T18:32:41.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3f1eb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed380d1f70 con 0x7fed3810d010 2021-04-09T18:32:41.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3f1eb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed380d9220 con 0x7fed3810da00 2021-04-09T18:32:41.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3c806700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fed380d7ca0 0x7fed380d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39294/0 (socket says 172.21.15.32:39294) 2021-04-09T18:32:41.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3c806700 1 -- 172.21.15.32:0/2731196517 learned_addr learned my addr 172.21.15.32:0/2731196517 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:41.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3d808700 1 --2- 172.21.15.32:0/2731196517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed3810da00 0x7fed3810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed2ffff700 1 -- 172.21.15.32:0/2731196517 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269474052 0 0) 0x7fed28000ce0 con 0x7fed380d7ca0 2021-04-09T18:32:41.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed2ffff700 1 -- 172.21.15.32:0/2731196517 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed24002fe0 con 0x7fed380d7ca0 2021-04-09T18:32:41.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed2ffff700 1 -- 172.21.15.32:0/2731196517 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548726936 0 0) 0x7fed34000fe0 con 0x7fed3810d010 2021-04-09T18:32:41.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed2ffff700 1 -- 172.21.15.32:0/2731196517 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed240042c0 con 0x7fed3810d010 2021-04-09T18:32:41.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3d808700 1 -- 172.21.15.32:0/2731196517 >> v1:172.21.15.86:6789/0 conn(0x7fed3810d010 legacy=0x7fed380d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3d808700 1 -- 172.21.15.32:0/2731196517 >> v1:172.21.15.32:6789/0 conn(0x7fed380d7ca0 legacy=0x7fed380d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3d808700 1 -- 172.21.15.32:0/2731196517 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3810e420 con 0x7fed3810da00 2021-04-09T18:32:41.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3d808700 1 --2- 172.21.15.32:0/2731196517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed3810da00 0x7fed3810de00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fed30006ad0 tx=0x7fed30009ad0).ready entity=mon.2 client_cookie=c61c39143561177b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:41.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed2ffff700 1 -- 172.21.15.32:0/2731196517 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed300022e0 con 0x7fed3810da00 2021-04-09T18:32:41.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed2ffff700 1 -- 172.21.15.32:0/2731196517 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed30006410 con 0x7fed3810da00 2021-04-09T18:32:41.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed2ffff700 1 -- 172.21.15.32:0/2731196517 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed30000c10 con 0x7fed3810da00 2021-04-09T18:32:41.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3f1eb700 1 -- 172.21.15.32:0/2731196517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed3810da00 msgr2=0x7fed3810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.609 7fed3f1eb700 1 --2- 172.21.15.32:0/2731196517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed3810da00 0x7fed3810de00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fed30006ad0 tx=0x7fed30009ad0).stop 2021-04-09T18:32:41.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.610 7fed3f1eb700 1 -- 172.21.15.32:0/2731196517 shutdown_connections 2021-04-09T18:32:41.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.610 7fed3f1eb700 1 --2- 172.21.15.32:0/2731196517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed3810da00 0x7fed3810de00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.610 7fed3f1eb700 1 -- 172.21.15.32:0/2731196517 >> 172.21.15.32:0/2731196517 conn(0x7fed380ce5f0 msgr2=0x7fed380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:41.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.610 7fed3f1eb700 1 -- 172.21.15.32:0/2731196517 shutdown_connections 2021-04-09T18:32:41.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.610 7fed3f1eb700 1 -- 172.21.15.32:0/2731196517 wait complete. 2021-04-09T18:32:41.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.630 7fed3f1eb700 1 Processor -- start 2021-04-09T18:32:41.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.630 7fed3f1eb700 1 -- start start 2021-04-09T18:32:41.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3f1eb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed38179610 0x7fed38179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3f1eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed3817a820 0x7fed3817ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3f1eb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed381949f0 0x7fed38194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3f1eb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed381b8d50 con 0x7fed3817a820 2021-04-09T18:32:41.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3f1eb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3817b240 con 0x7fed381949f0 2021-04-09T18:32:41.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3f1eb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed38195410 con 0x7fed38179610 2021-04-09T18:32:41.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3c806700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed3817a820 0x7fed3817ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3c806700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed3817a820 0x7fed3817ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57942/0 (socket says 172.21.15.32:57942) 2021-04-09T18:32:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3c806700 1 -- 172.21.15.32:0/1968169387 learned_addr learned my addr 172.21.15.32:0/1968169387 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3d007700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed38179610 0x7fed38179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.631 7fed3d808700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed381949f0 0x7fed38194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3c806700 1 -- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed38179610 msgr2=0x7fed38179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3c806700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed38179610 0x7fed38179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3c806700 1 -- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed381949f0 msgr2=0x7fed38194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3c806700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed381949f0 0x7fed38194df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3c806700 1 -- 172.21.15.32:0/1968169387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed38195630 con 0x7fed3817a820 2021-04-09T18:32:41.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3d007700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed38179610 0x7fed38179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:41.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3d808700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed381949f0 0x7fed38194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:41.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3c806700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed3817a820 0x7fed3817ac20 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 rx=0x7fed28006f00 tx=0x7fed2800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:41.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed2dffb700 1 -- 172.21.15.32:0/1968169387 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed2800bd30 con 0x7fed3817a820 2021-04-09T18:32:41.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed2dffb700 1 -- 172.21.15.32:0/1968169387 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed28006030 con 0x7fed3817a820 2021-04-09T18:32:41.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed2dffb700 1 -- 172.21.15.32:0/1968169387 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed28004590 con 0x7fed3817a820 2021-04-09T18:32:41.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed38195630 con 0x7fed3817a820 2021-04-09T18:32:41.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.632 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed38195630 con 0x7fed3817a820 2021-04-09T18:32:41.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.633 7fed2dffb700 1 -- 172.21.15.32:0/1968169387 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fed280043a0 con 0x7fed3817a820 2021-04-09T18:32:41.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.633 7fed2dffb700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed180188c0 0x7fed1801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:41.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.633 7fed2dffb700 1 -- 172.21.15.32:0/1968169387 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fed28032920 con 0x7fed3817a820 2021-04-09T18:32:41.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.633 7fed3d007700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed180188c0 0x7fed1801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:41.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.633 7fed3d007700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed180188c0 0x7fed1801adc0 crc :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:41.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.652 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3817a2f0 con 0x7fed3817a820 2021-04-09T18:32:41.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.656 7fed2dffb700 1 -- 172.21.15.32:0/1968169387 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fed280009b0 con 0x7fed3817a820 2021-04-09T18:32:41.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.838 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"} v 0) v1 -- 0x7fed3817a2f0 con 0x7fed3817a820 2021-04-09T18:32:41.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.838 7fed2dffb700 1 -- 172.21.15.32:0/1968169387 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "alpha_data"}]=0 v226) v1 ==== 99+0+10 (secure 0 0 0) 0x7fed280043a0 con 0x7fed3817a820 2021-04-09T18:32:41.855 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:32:41.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.840 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed180188c0 msgr2=0x7fed1801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.840 7fed3f1eb700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed180188c0 0x7fed1801adc0 crc :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.840 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed3817a820 msgr2=0x7fed3817ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.840 7fed3f1eb700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed3817a820 0x7fed3817ac20 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 rx=0x7fed28006f00 tx=0x7fed2800b940).stop 2021-04-09T18:32:41.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.841 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 shutdown_connections 2021-04-09T18:32:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.841 7fed3f1eb700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed180188c0 0x7fed1801adc0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.841 7fed3f1eb700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed3817a820 0x7fed3817ac20 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.841 7fed3f1eb700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed381949f0 0x7fed38194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.841 7fed3f1eb700 1 --2- 172.21.15.32:0/1968169387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed38179610 0x7fed38179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:41.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.841 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 >> 172.21.15.32:0/1968169387 conn(0x7fed38092a00 msgr2=0x7fed381b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.841 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 shutdown_connections 2021-04-09T18:32:41.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.841 7fed3f1eb700 1 -- 172.21.15.32:0/1968169387 wait complete. 2021-04-09T18:32:41.868 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:32:42.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.999 7f2264f4f700 1 Processor -- start 2021-04-09T18:32:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:41.999 7f2264f4f700 1 -- start start 2021-04-09T18:32:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f2264f4f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f226010d010 0x7f22600d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f2264f4f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22600d8490 0x7f226010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f2264f4f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226010d480 con 0x7f22600d1da0 2021-04-09T18:32:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f2264f4f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22600d8d30 con 0x7f226010d010 2021-04-09T18:32:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f2264f4f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22600d9220 con 0x7f22600d8490 2021-04-09T18:32:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225e312700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f22600d1da0 0x7f22600d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39306/0 (socket says 172.21.15.32:39306) 2021-04-09T18:32:42.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225e312700 1 -- 172.21.15.32:0/94434257 learned_addr learned my addr 172.21.15.32:0/94434257 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225f314700 1 --2- 172.21.15.32:0/94434257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22600d8490 0x7f226010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225eb13700 1 --2- 172.21.15.32:0/94434257 >> v2:172.21.15.86:3300/0 conn(0x7f226010d010 0x7f22600d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225db11700 1 -- 172.21.15.32:0/94434257 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301106045 0 0) 0x7f2250001060 con 0x7f22600d1da0 2021-04-09T18:32:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225db11700 1 -- 172.21.15.32:0/94434257 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f224c002fe0 con 0x7f22600d1da0 2021-04-09T18:32:42.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225eb13700 1 -- 172.21.15.32:0/94434257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22600d8490 msgr2=0x7f226010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225eb13700 1 --2- 172.21.15.32:0/94434257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22600d8490 0x7f226010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225eb13700 1 -- 172.21.15.32:0/94434257 >> v1:172.21.15.32:6789/0 conn(0x7f22600d1da0 legacy=0x7f22600d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225eb13700 1 -- 172.21.15.32:0/94434257 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226010e020 con 0x7f226010d010 2021-04-09T18:32:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.000 7f225f314700 1 --2- 172.21.15.32:0/94434257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22600d8490 0x7f226010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:42.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.001 7f225eb13700 1 --2- 172.21.15.32:0/94434257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226010d010 0x7f22600d1780 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f2248006650 tx=0x7f2248009a80).ready entity=mon.1 client_cookie=1c201a838ed95303 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:42.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.001 7f225db11700 1 -- 172.21.15.32:0/94434257 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22480009b0 con 0x7f226010d010 2021-04-09T18:32:42.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.001 7f225db11700 1 -- 172.21.15.32:0/94434257 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224800c0c0 con 0x7f226010d010 2021-04-09T18:32:42.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.001 7f225db11700 1 -- 172.21.15.32:0/94434257 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f224800c510 con 0x7f226010d010 2021-04-09T18:32:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.001 7f2264f4f700 1 -- 172.21.15.32:0/94434257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226010d010 msgr2=0x7f22600d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.001 7f2264f4f700 1 --2- 172.21.15.32:0/94434257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226010d010 0x7f22600d1780 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f2248006650 tx=0x7f2248009a80).stop 2021-04-09T18:32:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.002 7f2264f4f700 1 -- 172.21.15.32:0/94434257 shutdown_connections 2021-04-09T18:32:42.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.002 7f2264f4f700 1 --2- 172.21.15.32:0/94434257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22600d8490 0x7f226010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.002 7f2264f4f700 1 --2- 172.21.15.32:0/94434257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f226010d010 0x7f22600d1780 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.002 7f2264f4f700 1 -- 172.21.15.32:0/94434257 >> 172.21.15.32:0/94434257 conn(0x7f22600ce5f0 msgr2=0x7f22600d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.002 7f2264f4f700 1 -- 172.21.15.32:0/94434257 shutdown_connections 2021-04-09T18:32:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.002 7f2264f4f700 1 -- 172.21.15.32:0/94434257 wait complete. 2021-04-09T18:32:42.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.022 7f2264f4f700 1 Processor -- start 2021-04-09T18:32:42.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.022 7f2264f4f700 1 -- start start 2021-04-09T18:32:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.022 7f2264f4f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22601a5210 0x7f2260185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.022 7f2264f4f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2260185a70 0x7f22600acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.022 7f2264f4f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260185e70 0x7f22600ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.022 7f2264f4f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226018f7b0 con 0x7f2260185e70 2021-04-09T18:32:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.022 7f2264f4f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22600ad7f0 con 0x7f22601a5210 2021-04-09T18:32:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.022 7f2264f4f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22600ada10 con 0x7f2260185a70 2021-04-09T18:32:42.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225f314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260185e70 0x7f22600ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225f314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260185e70 0x7f22600ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57958/0 (socket says 172.21.15.32:57958) 2021-04-09T18:32:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225f314700 1 -- 172.21.15.32:0/781392826 learned_addr learned my addr 172.21.15.32:0/781392826 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225eb13700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22601a5210 0x7f2260185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225e312700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2260185a70 0x7f22600acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225f314700 1 -- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2260185a70 msgr2=0x7f22600acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225f314700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2260185a70 0x7f22600acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225f314700 1 -- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22601a5210 msgr2=0x7f2260185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225f314700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22601a5210 0x7f2260185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225f314700 1 -- 172.21.15.32:0/781392826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2260179610 con 0x7f2260185e70 2021-04-09T18:32:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225eb13700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22601a5210 0x7f2260185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f225f314700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260185e70 0x7f22600ad1d0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 rx=0x7f225800b5e0 tx=0x7f225800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f22577fe700 1 -- 172.21.15.32:0/781392826 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2258000c10 con 0x7f2260185e70 2021-04-09T18:32:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f22577fe700 1 -- 172.21.15.32:0/781392826 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225800bd70 con 0x7f2260185e70 2021-04-09T18:32:42.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f2264f4f700 1 -- 172.21.15.32:0/781392826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2260179610 con 0x7f2260185e70 2021-04-09T18:32:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f22577fe700 1 -- 172.21.15.32:0/781392826 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22580024d0 con 0x7f2260185e70 2021-04-09T18:32:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.023 7f2264f4f700 1 -- 172.21.15.32:0/781392826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2260179610 con 0x7f2260185e70 2021-04-09T18:32:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.024 7f22577fe700 1 -- 172.21.15.32:0/781392826 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2258006720 con 0x7f2260185e70 2021-04-09T18:32:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.024 7f22577fe700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f22440188c0 0x7f224401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.025 7f22577fe700 1 -- 172.21.15.32:0/781392826 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2258032910 con 0x7f2260185e70 2021-04-09T18:32:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.025 7f225eb13700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f22440188c0 0x7f224401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.025 7f225eb13700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f22440188c0 0x7f224401adc0 crc :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:42.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.044 7f2264f4f700 1 -- 172.21.15.32:0/781392826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226018f7b0 con 0x7f2260185e70 2021-04-09T18:32:42.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.047 7f22577fe700 1 -- 172.21.15.32:0/781392826 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2258006370 con 0x7f2260185e70 2021-04-09T18:32:42.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.225 7f2264f4f700 1 -- 172.21.15.32:0/781392826 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f226018f7b0 con 0x7f2260185e70 2021-04-09T18:32:42.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.225 7f22577fe700 1 -- 172.21.15.32:0/781392826 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v226) v1 ==== 84+0+206 (secure 0 0 0) 0x7f2258006720 con 0x7f2260185e70 2021-04-09T18:32:42.242 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:42.243 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:32:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:32:42.243 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:32:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 38, 2021-04-09T18:32:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "alpha_metadata" 2021-04-09T18:32:42.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:32:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:32:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 39, 2021-04-09T18:32:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "alpha_data" 2021-04-09T18:32:42.245 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:32:42.246 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:32:42.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 -- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f22440188c0 msgr2=0x7f224401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f22440188c0 0x7f224401adc0 crc :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 -- 172.21.15.32:0/781392826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260185e70 msgr2=0x7f22600ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260185e70 0x7f22600ad1d0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 rx=0x7f225800b5e0 tx=0x7f225800b980).stop 2021-04-09T18:32:42.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 -- 172.21.15.32:0/781392826 shutdown_connections 2021-04-09T18:32:42.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f22440188c0 0x7f224401adc0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2260185e70 0x7f22600ad1d0 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2260185a70 0x7f22600acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 --2- 172.21.15.32:0/781392826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22601a5210 0x7f2260185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 -- 172.21.15.32:0/781392826 >> 172.21.15.32:0/781392826 conn(0x7f22601a99c0 msgr2=0x7f226017dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:42.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 -- 172.21.15.32:0/781392826 shutdown_connections 2021-04-09T18:32:42.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.228 7f2264f4f700 1 -- 172.21.15.32:0/781392826 wait complete. 2021-04-09T18:32:42.255 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'bravo' 2021-04-09T18:32:42.255 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create bravo_metadata 8 2021-04-09T18:32:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.391 7f29a86b1700 1 Processor -- start 2021-04-09T18:32:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.391 7f29a86b1700 1 -- start start 2021-04-09T18:32:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a86b1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00d7ca0 0x7f29a00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a86b1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a010d480 con 0x7f29a010d010 2021-04-09T18:32:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a86b1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a00d86c0 con 0x7f29a010da00 2021-04-09T18:32:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a86b1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a00d9220 con 0x7f29a00d7ca0 2021-04-09T18:32:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a64cd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f29a010d010 0x7f29a00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39320/0 (socket says 172.21.15.32:39320) 2021-04-09T18:32:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a64cd700 1 -- 172.21.15.32:0/3870988649 learned_addr learned my addr 172.21.15.32:0/3870988649 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a5ccc700 1 --2- 172.21.15.32:0/3870988649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00d7ca0 0x7f29a00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a54cb700 1 -- 172.21.15.32:0/3870988649 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330142867 0 0) 0x7f2998001060 con 0x7f29a010d010 2021-04-09T18:32:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a54cb700 1 -- 172.21.15.32:0/3870988649 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2988002fe0 con 0x7f29a010d010 2021-04-09T18:32:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a54cb700 1 -- 172.21.15.32:0/3870988649 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307646336 0 0) 0x7f29900009b0 con 0x7f29a010da00 2021-04-09T18:32:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.392 7f29a54cb700 1 -- 172.21.15.32:0/3870988649 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2988003210 con 0x7f29a010da00 2021-04-09T18:32:42.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a5ccc700 1 -- 172.21.15.32:0/3870988649 >> v1:172.21.15.86:6789/0 conn(0x7f29a010da00 legacy=0x7f29a010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a5ccc700 1 -- 172.21.15.32:0/3870988649 >> v1:172.21.15.32:6789/0 conn(0x7f29a010d010 legacy=0x7f29a00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a54cb700 1 -- 172.21.15.32:0/3870988649 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2710211452 0 0) 0x7f2998001460 con 0x7f29a010d010 2021-04-09T18:32:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a5ccc700 1 -- 172.21.15.32:0/3870988649 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a00d1f70 con 0x7f29a00d7ca0 2021-04-09T18:32:42.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a5ccc700 1 --2- 172.21.15.32:0/3870988649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00d7ca0 0x7f29a00d80a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f2994006650 tx=0x7f2994009a00).ready entity=mon.2 client_cookie=3c8527c7a1d99931 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a54cb700 1 -- 172.21.15.32:0/3870988649 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2994009df0 con 0x7f29a00d7ca0 2021-04-09T18:32:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a54cb700 1 -- 172.21.15.32:0/3870988649 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29940009b0 con 0x7f29a00d7ca0 2021-04-09T18:32:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a54cb700 1 -- 172.21.15.32:0/3870988649 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f299400c2b0 con 0x7f29a00d7ca0 2021-04-09T18:32:42.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a86b1700 1 -- 172.21.15.32:0/3870988649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00d7ca0 msgr2=0x7f29a00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.393 7f29a86b1700 1 --2- 172.21.15.32:0/3870988649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00d7ca0 0x7f29a00d80a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f2994006650 tx=0x7f2994009a00).stop 2021-04-09T18:32:42.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.394 7f29a86b1700 1 -- 172.21.15.32:0/3870988649 shutdown_connections 2021-04-09T18:32:42.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.394 7f29a86b1700 1 --2- 172.21.15.32:0/3870988649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00d7ca0 0x7f29a00d80a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.394 7f29a86b1700 1 -- 172.21.15.32:0/3870988649 >> 172.21.15.32:0/3870988649 conn(0x7f29a00ce5f0 msgr2=0x7f29a00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:42.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.394 7f29a86b1700 1 -- 172.21.15.32:0/3870988649 shutdown_connections 2021-04-09T18:32:42.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.394 7f29a86b1700 1 -- 172.21.15.32:0/3870988649 wait complete. 2021-04-09T18:32:42.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.414 7f29a86b1700 1 Processor -- start 2021-04-09T18:32:42.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.414 7f29a86b1700 1 -- start start 2021-04-09T18:32:42.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a86b1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29a00b1680 0x7f29a00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a86b1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a009e330 0x7f29a00b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a86b1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00b2190 0x7f29a009f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a86b1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a009e730 con 0x7f29a00b1680 2021-04-09T18:32:42.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a86b1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a009f6c0 con 0x7f29a009e330 2021-04-09T18:32:42.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a86b1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a009f8e0 con 0x7f29a00b2190 2021-04-09T18:32:42.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a64cd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29a00b1680 0x7f29a00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a64cd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29a00b1680 0x7f29a00b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57970/0 (socket says 172.21.15.32:57970) 2021-04-09T18:32:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a64cd700 1 -- 172.21.15.32:0/14778029 learned_addr learned my addr 172.21.15.32:0/14778029 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a5ccc700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a009e330 0x7f29a00b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a6cce700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00b2190 0x7f29a009f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a64cd700 1 -- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00b2190 msgr2=0x7f29a009f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a64cd700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00b2190 0x7f29a009f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a64cd700 1 -- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a009e330 msgr2=0x7f29a00b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a64cd700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a009e330 0x7f29a00b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a64cd700 1 -- 172.21.15.32:0/14778029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a009fb00 con 0x7f29a00b1680 2021-04-09T18:32:42.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a5ccc700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a009e330 0x7f29a00b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:42.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a6cce700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00b2190 0x7f29a009f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:42.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a64cd700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29a00b1680 0x7f29a00b0e30 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 rx=0x7f2998007f90 tx=0x7f299800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:42.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f299effd700 1 -- 172.21.15.32:0/14778029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2998004930 con 0x7f29a00b1680 2021-04-09T18:32:42.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a86b1700 1 -- 172.21.15.32:0/14778029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a009fb00 con 0x7f29a00b1680 2021-04-09T18:32:42.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.415 7f29a86b1700 1 -- 172.21.15.32:0/14778029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a009fb00 con 0x7f29a00b1680 2021-04-09T18:32:42.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.416 7f299effd700 1 -- 172.21.15.32:0/14778029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2998001270 con 0x7f29a00b1680 2021-04-09T18:32:42.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.416 7f299effd700 1 -- 172.21.15.32:0/14778029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29980072c0 con 0x7f29a00b1680 2021-04-09T18:32:42.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.416 7f299effd700 1 -- 172.21.15.32:0/14778029 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f299800ad90 con 0x7f29a00b1680 2021-04-09T18:32:42.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.416 7f299effd700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2980018930 0x7f298001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:42.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.416 7f299effd700 1 -- 172.21.15.32:0/14778029 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2998032db0 con 0x7f29a00b1680 2021-04-09T18:32:42.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.417 7f29a5ccc700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2980018930 0x7f298001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:42.449 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.417 7f29a5ccc700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2980018930 0x7f298001ae10 crc :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:42.452 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.436 7f29a86b1700 1 -- 172.21.15.32:0/14778029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297c002fd0 con 0x7f29a00b1680 2021-04-09T18:32:42.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.439 7f299effd700 1 -- 172.21.15.32:0/14778029 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2998007d00 con 0x7f29a00b1680 2021-04-09T18:32:42.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:42.622 7f29a86b1700 1 -- 172.21.15.32:0/14778029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_metadata"} v 0) v1 -- 0x7f297c002ef0 con 0x7f29a00b1680 2021-04-09T18:32:43.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.422 7f299effd700 1 -- 172.21.15.32:0/14778029 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_metadata"}]=0 pool 'bravo_metadata' created v227) v1 ==== 131+0+0 (secure 0 0 0) 0x7f2998001270 con 0x7f29a00b1680 2021-04-09T18:32:43.453 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_metadata' created 2021-04-09T18:32:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.439 7f29a86b1700 1 -- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2980018930 msgr2=0x7f298001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:43.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.439 7f29a86b1700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2980018930 0x7f298001ae10 crc :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.439 7f29a86b1700 1 -- 172.21.15.32:0/14778029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29a00b1680 msgr2=0x7f29a00b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:43.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.439 7f29a86b1700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29a00b1680 0x7f29a00b0e30 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 rx=0x7f2998007f90 tx=0x7f299800ac50).stop 2021-04-09T18:32:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.439 7f29a86b1700 1 -- 172.21.15.32:0/14778029 shutdown_connections 2021-04-09T18:32:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.439 7f29a86b1700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2980018930 0x7f298001ae10 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.439 7f29a86b1700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a00b2190 0x7f29a009f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.439 7f29a86b1700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a009e330 0x7f29a00b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.439 7f29a86b1700 1 --2- 172.21.15.32:0/14778029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29a00b1680 0x7f29a00b0e30 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.440 7f29a86b1700 1 -- 172.21.15.32:0/14778029 >> 172.21.15.32:0/14778029 conn(0x7f29a00c4290 msgr2=0x7f29a00c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.440 7f29a86b1700 1 -- 172.21.15.32:0/14778029 shutdown_connections 2021-04-09T18:32:43.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.440 7f29a86b1700 1 -- 172.21.15.32:0/14778029 wait complete. 2021-04-09T18:32:43.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create bravo_data 8 2021-04-09T18:32:43.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.604 7fb7585ac700 1 Processor -- start 2021-04-09T18:32:43.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.604 7fb7585ac700 1 -- start start 2021-04-09T18:32:43.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb7585ac700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7500d7ca0 0x7fb7500d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:43.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb7585ac700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb75010da00 0x7fb75010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb7585ac700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75010d480 con 0x7fb75010d010 2021-04-09T18:32:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb7585ac700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7500d9220 con 0x7fb75010da00 2021-04-09T18:32:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb7585ac700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb750112100 con 0x7fb7500d7ca0 2021-04-09T18:32:43.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb756bc9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb75010da00 0x7fb75010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb7563c8700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb75010d010 0x7fb7500d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37260/0 (socket says 172.21.15.32:37260) 2021-04-09T18:32:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb7563c8700 1 -- 172.21.15.32:0/3594614963 learned_addr learned my addr 172.21.15.32:0/3594614963 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb755bc7700 1 --2- 172.21.15.32:0/3594614963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7500d7ca0 0x7fb7500d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb7553c6700 1 -- 172.21.15.32:0/3594614963 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135278279 0 0) 0x7fb74c0010c0 con 0x7fb75010d010 2021-04-09T18:32:43.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb7553c6700 1 -- 172.21.15.32:0/3594614963 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73c002fe0 con 0x7fb75010d010 2021-04-09T18:32:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb755bc7700 1 -- 172.21.15.32:0/3594614963 >> v2:172.21.15.86:3300/0 conn(0x7fb75010da00 msgr2=0x7fb75010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb755bc7700 1 --2- 172.21.15.32:0/3594614963 >> v2:172.21.15.86:3300/0 conn(0x7fb75010da00 0x7fb75010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb755bc7700 1 -- 172.21.15.32:0/3594614963 >> v1:172.21.15.86:6789/0 conn(0x7fb75010d010 legacy=0x7fb7500d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:43.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb755bc7700 1 -- 172.21.15.32:0/3594614963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7500d86c0 con 0x7fb7500d7ca0 2021-04-09T18:32:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.605 7fb756bc9700 1 --2- 172.21.15.32:0/3594614963 >> v2:172.21.15.86:3300/0 conn(0x7fb75010da00 0x7fb75010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.606 7fb755bc7700 1 --2- 172.21.15.32:0/3594614963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7500d7ca0 0x7fb7500d80a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fb740006650 tx=0x7fb740009a80).ready entity=mon.2 client_cookie=365d5951fbb182c4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.606 7fb7553c6700 1 -- 172.21.15.32:0/3594614963 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7400009b0 con 0x7fb7500d7ca0 2021-04-09T18:32:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.606 7fb7553c6700 1 -- 172.21.15.32:0/3594614963 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74000c0c0 con 0x7fb7500d7ca0 2021-04-09T18:32:43.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.606 7fb7553c6700 1 -- 172.21.15.32:0/3594614963 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb74000c510 con 0x7fb7500d7ca0 2021-04-09T18:32:43.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.606 7fb7585ac700 1 -- 172.21.15.32:0/3594614963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7500d7ca0 msgr2=0x7fb7500d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:43.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.606 7fb7585ac700 1 --2- 172.21.15.32:0/3594614963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7500d7ca0 0x7fb7500d80a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fb740006650 tx=0x7fb740009a80).stop 2021-04-09T18:32:43.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.607 7fb7585ac700 1 -- 172.21.15.32:0/3594614963 shutdown_connections 2021-04-09T18:32:43.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.607 7fb7585ac700 1 --2- 172.21.15.32:0/3594614963 >> v2:172.21.15.86:3300/0 conn(0x7fb75010da00 0x7fb75010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.607 7fb7585ac700 1 --2- 172.21.15.32:0/3594614963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7500d7ca0 0x7fb7500d80a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.607 7fb7585ac700 1 -- 172.21.15.32:0/3594614963 >> 172.21.15.32:0/3594614963 conn(0x7fb7500ce5f0 msgr2=0x7fb7500d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.607 7fb7585ac700 1 -- 172.21.15.32:0/3594614963 shutdown_connections 2021-04-09T18:32:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.607 7fb7585ac700 1 -- 172.21.15.32:0/3594614963 wait complete. 2021-04-09T18:32:43.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.627 7fb7585ac700 1 Processor -- start 2021-04-09T18:32:43.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.627 7fb7585ac700 1 -- start start 2021-04-09T18:32:43.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.627 7fb7585ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75009c5b0 0x7fb75009d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:43.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.627 7fb7585ac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75009c9b0 0x7fb75009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:43.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.627 7fb7585ac700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb750090390 0x7fb750090790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:43.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.627 7fb7585ac700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75009e290 con 0x7fb75009c5b0 2021-04-09T18:32:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.627 7fb7585ac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75009cf90 con 0x7fb75009c9b0 2021-04-09T18:32:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.627 7fb7585ac700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb750090db0 con 0x7fb750090390 2021-04-09T18:32:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7563c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75009c5b0 0x7fb75009d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7563c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75009c5b0 0x7fb75009d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57982/0 (socket says 172.21.15.32:57982) 2021-04-09T18:32:43.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7563c8700 1 -- 172.21.15.32:0/2566280094 learned_addr learned my addr 172.21.15.32:0/2566280094 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb755bc7700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75009c9b0 0x7fb75009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb756bc9700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb750090390 0x7fb750090790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7563c8700 1 -- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb750090390 msgr2=0x7fb750090790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:43.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7563c8700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb750090390 0x7fb750090790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7563c8700 1 -- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75009c9b0 msgr2=0x7fb75009db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7563c8700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75009c9b0 0x7fb75009db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7563c8700 1 -- 172.21.15.32:0/2566280094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb750090f90 con 0x7fb75009c5b0 2021-04-09T18:32:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb755bc7700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75009c9b0 0x7fb75009db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:43.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7563c8700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75009c5b0 0x7fb75009d510 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 rx=0x7fb74c00a550 tx=0x7fb74c00a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb746ffd700 1 -- 172.21.15.32:0/2566280094 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb74c000e70 con 0x7fb75009c5b0 2021-04-09T18:32:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb746ffd700 1 -- 172.21.15.32:0/2566280094 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c00ace0 con 0x7fb75009c5b0 2021-04-09T18:32:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb750090f90 con 0x7fb75009c5b0 2021-04-09T18:32:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb746ffd700 1 -- 172.21.15.32:0/2566280094 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb74c004510 con 0x7fb75009c5b0 2021-04-09T18:32:43.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.628 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb750090f90 con 0x7fb75009c5b0 2021-04-09T18:32:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.629 7fb746ffd700 1 -- 172.21.15.32:0/2566280094 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb74c004320 con 0x7fb75009c5b0 2021-04-09T18:32:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.629 7fb746ffd700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb7380188d0 0x7fb73801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.629 7fb746ffd700 1 -- 172.21.15.32:0/2566280094 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5926+0+0 (secure 0 0 0) 0x7fb74c032b60 con 0x7fb75009c5b0 2021-04-09T18:32:43.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.629 7fb755bc7700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb7380188d0 0x7fb73801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:43.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.630 7fb755bc7700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb7380188d0 0x7fb73801add0 crc :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:43.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.649 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75010d480 con 0x7fb75009c5b0 2021-04-09T18:32:43.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.653 7fb746ffd700 1 -- 172.21.15.32:0/2566280094 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb74c000c10 con 0x7fb75009c5b0 2021-04-09T18:32:43.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:43.835 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_data"} v 0) v1 -- 0x7fb75010d480 con 0x7fb75009c5b0 2021-04-09T18:32:44.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.429 7fb746ffd700 1 -- 172.21.15.32:0/2566280094 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "bravo_data"}]=0 pool 'bravo_data' created v228) v1 ==== 123+0+0 (secure 0 0 0) 0x7fb74c004320 con 0x7fb75009c5b0 2021-04-09T18:32:44.467 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_data' created 2021-04-09T18:32:44.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.452 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb7380188d0 msgr2=0x7fb73801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:44.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb7380188d0 0x7fb73801add0 crc :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75009c5b0 msgr2=0x7fb75009d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75009c5b0 0x7fb75009d510 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 rx=0x7fb74c00a550 tx=0x7fb74c00a8f0).stop 2021-04-09T18:32:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 shutdown_connections 2021-04-09T18:32:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb7380188d0 0x7fb73801add0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb750090390 0x7fb750090790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb75009c9b0 0x7fb75009db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 --2- 172.21.15.32:0/2566280094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb75009c5b0 0x7fb75009d510 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 >> 172.21.15.32:0/2566280094 conn(0x7fb7500b2fa0 msgr2=0x7fb7500a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 shutdown_connections 2021-04-09T18:32:44.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.453 7fb7585ac700 1 -- 172.21.15.32:0/2566280094 wait complete. 2021-04-09T18:32:44.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new bravo bravo_metadata bravo_data --force 2021-04-09T18:32:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.626 7efd6d8b2700 1 Processor -- start 2021-04-09T18:32:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.626 7efd6d8b2700 1 -- start start 2021-04-09T18:32:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd6d8b2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efd680d7cc0 0x7efd680d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd6d8b2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd6810da80 0x7efd6810de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd6d8b2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6810d500 con 0x7efd6810d050 2021-04-09T18:32:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd6d8b2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd680d9260 con 0x7efd6810da80 2021-04-09T18:32:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd6d8b2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd680d0330 con 0x7efd680d7cc0 2021-04-09T18:32:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd67d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd6810da80 0x7efd6810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd67d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efd6810da80 0x7efd6810de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57992/0 (socket says 172.21.15.32:57992) 2021-04-09T18:32:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd67d75700 1 -- 172.21.15.32:0/873451567 learned_addr learned my addr 172.21.15.32:0/873451567 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd66d73700 1 --2- 172.21.15.32:0/873451567 >> v2:172.21.15.86:3300/0 conn(0x7efd680d7cc0 0x7efd680d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd67d75700 1 -- 172.21.15.32:0/873451567 >> v2:172.21.15.86:3300/0 conn(0x7efd680d7cc0 msgr2=0x7efd680d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd67d75700 1 --2- 172.21.15.32:0/873451567 >> v2:172.21.15.86:3300/0 conn(0x7efd680d7cc0 0x7efd680d80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd67d75700 1 -- 172.21.15.32:0/873451567 >> v1:172.21.15.86:6789/0 conn(0x7efd6810d050 legacy=0x7efd680cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd66572700 1 -- 172.21.15.32:0/873451567 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606434104 0 0) 0x7efd58000c60 con 0x7efd6810d050 2021-04-09T18:32:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd67d75700 1 -- 172.21.15.32:0/873451567 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd680d86e0 con 0x7efd6810da80 2021-04-09T18:32:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd67d75700 1 --2- 172.21.15.32:0/873451567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6810da80 0x7efd6810de80 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 rx=0x7efd60006f00 tx=0x7efd60009f10).ready entity=mon.0 client_cookie=458642114c9f19d3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd66572700 1 -- 172.21.15.32:0/873451567 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd6000b3b0 con 0x7efd6810da80 2021-04-09T18:32:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd66572700 1 -- 172.21.15.32:0/873451567 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6000b730 con 0x7efd6810da80 2021-04-09T18:32:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.627 7efd66572700 1 -- 172.21.15.32:0/873451567 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd6000bc10 con 0x7efd6810da80 2021-04-09T18:32:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.628 7efd6d8b2700 1 -- 172.21.15.32:0/873451567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6810da80 msgr2=0x7efd6810de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.628 7efd6d8b2700 1 --2- 172.21.15.32:0/873451567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6810da80 0x7efd6810de80 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 rx=0x7efd60006f00 tx=0x7efd60009f10).stop 2021-04-09T18:32:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.628 7efd6d8b2700 1 -- 172.21.15.32:0/873451567 shutdown_connections 2021-04-09T18:32:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.628 7efd6d8b2700 1 --2- 172.21.15.32:0/873451567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6810da80 0x7efd6810de80 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.628 7efd6d8b2700 1 --2- 172.21.15.32:0/873451567 >> v2:172.21.15.86:3300/0 conn(0x7efd680d7cc0 0x7efd680d80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.628 7efd6d8b2700 1 -- 172.21.15.32:0/873451567 >> 172.21.15.32:0/873451567 conn(0x7efd680ce620 msgr2=0x7efd680d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.628 7efd6d8b2700 1 -- 172.21.15.32:0/873451567 shutdown_connections 2021-04-09T18:32:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.628 7efd6d8b2700 1 -- 172.21.15.32:0/873451567 wait complete. 2021-04-09T18:32:44.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.648 7efd6d8b2700 1 Processor -- start 2021-04-09T18:32:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.648 7efd6d8b2700 1 -- start start 2021-04-09T18:32:44.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd6d8b2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd680ad600 0x7efd6809ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:44.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd6d8b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6809b1b0 0x7efd680ba220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:44.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd6d8b2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd680ba840 0x7efd680a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:44.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd6d8b2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6809fa80 con 0x7efd6809b1b0 2021-04-09T18:32:44.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd6d8b2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6809b5b0 con 0x7efd680ad600 2021-04-09T18:32:44.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd6d8b2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd680bac40 con 0x7efd680ba840 2021-04-09T18:32:44.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd66d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6809b1b0 0x7efd680ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:44.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd66d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6809b1b0 0x7efd680ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57998/0 (socket says 172.21.15.32:57998) 2021-04-09T18:32:44.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd66d73700 1 -- 172.21.15.32:0/2038362135 learned_addr learned my addr 172.21.15.32:0/2038362135 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:44.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd67574700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd680ad600 0x7efd6809ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:44.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.649 7efd67d75700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd680ba840 0x7efd680a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:44.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd66d73700 1 -- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd680ba840 msgr2=0x7efd680a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:44.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd66d73700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd680ba840 0x7efd680a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd66d73700 1 -- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd680ad600 msgr2=0x7efd6809ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:44.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd66d73700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd680ad600 0x7efd6809ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd66d73700 1 -- 172.21.15.32:0/2038362135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd680a97c0 con 0x7efd6809b1b0 2021-04-09T18:32:44.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd67574700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd680ad600 0x7efd6809ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:44.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd66d73700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6809b1b0 0x7efd680ba220 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 rx=0x7efd5c00b560 tx=0x7efd5c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:44.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd4ffff700 1 -- 172.21.15.32:0/2038362135 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd5c00bcf0 con 0x7efd6809b1b0 2021-04-09T18:32:44.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd4ffff700 1 -- 172.21.15.32:0/2038362135 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c000c90 con 0x7efd6809b1b0 2021-04-09T18:32:44.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd680a97c0 con 0x7efd6809b1b0 2021-04-09T18:32:44.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd4ffff700 1 -- 172.21.15.32:0/2038362135 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd5c002220 con 0x7efd6809b1b0 2021-04-09T18:32:44.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.650 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd680a97c0 con 0x7efd6809b1b0 2021-04-09T18:32:44.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.651 7efd4ffff700 1 -- 172.21.15.32:0/2038362135 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efd5c0052e0 con 0x7efd6809b1b0 2021-04-09T18:32:44.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.651 7efd4ffff700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd480188d0 0x7efd4801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:44.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.651 7efd4ffff700 1 -- 172.21.15.32:0/2038362135 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 6275+0+0 (secure 0 0 0) 0x7efd5c032cb0 con 0x7efd6809b1b0 2021-04-09T18:32:44.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.651 7efd67574700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd480188d0 0x7efd4801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:44.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.651 7efd67574700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd480188d0 0x7efd4801add0 crc :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:44.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.671 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd680aa330 con 0x7efd6809b1b0 2021-04-09T18:32:44.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.675 7efd4ffff700 1 -- 172.21.15.32:0/2038362135 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efd5c000c90 con 0x7efd6809b1b0 2021-04-09T18:32:44.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.857 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "bravo_data", "prefix": "fs new", "force": true, "fs_name": "bravo", "metadata": "bravo_metadata"} v 0) v1 -- 0x7efd680aa330 con 0x7efd6809b1b0 2021-04-09T18:32:44.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.861 7efd4ffff700 1 -- 172.21.15.32:0/2038362135 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "bravo_data", "prefix": "fs new", "force": true, "fs_name": "bravo", "metadata": "bravo_metadata"}]=0 new fs with metadata pool 40 and data pool 41 v382) v1 ==== 186+0+0 (secure 0 0 0) 0x7efd5c006b00 con 0x7efd6809b1b0 2021-04-09T18:32:44.881 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 40 and data pool 41 2021-04-09T18:32:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.867 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd480188d0 msgr2=0x7efd4801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:44.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.867 7efd6d8b2700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd480188d0 0x7efd4801add0 crc :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.867 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6809b1b0 msgr2=0x7efd680ba220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:44.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.867 7efd6d8b2700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6809b1b0 0x7efd680ba220 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 rx=0x7efd5c00b560 tx=0x7efd5c00b900).stop 2021-04-09T18:32:44.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.868 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 shutdown_connections 2021-04-09T18:32:44.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.868 7efd6d8b2700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd480188d0 0x7efd4801add0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.868 7efd6d8b2700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd6809b1b0 0x7efd680ba220 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.868 7efd6d8b2700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd680ba840 0x7efd680a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.868 7efd6d8b2700 1 --2- 172.21.15.32:0/2038362135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd680ad600 0x7efd6809ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:44.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.868 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 >> 172.21.15.32:0/2038362135 conn(0x7efd680ae6f0 msgr2=0x7efd68097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:44.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.868 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 shutdown_connections 2021-04-09T18:32:44.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:44.868 7efd6d8b2700 1 -- 172.21.15.32:0/2038362135 wait complete. 2021-04-09T18:32:44.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:32:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.026 7ff3cc984700 1 Processor -- start 2021-04-09T18:32:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.026 7ff3cc984700 1 -- start start 2021-04-09T18:32:45.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.027 7ff3cc984700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff3c810e7d0 0x7ff3c80d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.027 7ff3cc984700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c80d7280 con 0x7ff3c80d6c80 2021-04-09T18:32:45.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.027 7ff3cc984700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c80d2440 con 0x7ff3c80cffa0 2021-04-09T18:32:45.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.027 7ff3cc984700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c80d7080 con 0x7ff3c810e7d0 2021-04-09T18:32:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.027 7ff3c5e1b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff3c80d6c80 0x7ff3c810cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39364/0 (socket says 172.21.15.32:39364) 2021-04-09T18:32:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.027 7ff3c5e1b700 1 -- 172.21.15.32:0/680779442 learned_addr learned my addr 172.21.15.32:0/680779442 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:45.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c6e1d700 1 --2- 172.21.15.32:0/680779442 >> v2:172.21.15.86:3300/0 conn(0x7ff3c810e7d0 0x7ff3c80d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c561a700 1 -- 172.21.15.32:0/680779442 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385048017 0 0) 0x7ff3bc000ce0 con 0x7ff3c80d6c80 2021-04-09T18:32:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c561a700 1 -- 172.21.15.32:0/680779442 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b0002fe0 con 0x7ff3c80d6c80 2021-04-09T18:32:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c561a700 1 -- 172.21.15.32:0/680779442 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457114147 0 0) 0x7ff3b8000c60 con 0x7ff3c80cffa0 2021-04-09T18:32:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c561a700 1 -- 172.21.15.32:0/680779442 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b00042c0 con 0x7ff3c80cffa0 2021-04-09T18:32:45.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c6e1d700 1 -- 172.21.15.32:0/680779442 >> v1:172.21.15.86:6789/0 conn(0x7ff3c80cffa0 legacy=0x7ff3c80d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c6e1d700 1 -- 172.21.15.32:0/680779442 >> v1:172.21.15.32:6789/0 conn(0x7ff3c80d6c80 legacy=0x7ff3c810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c6e1d700 1 -- 172.21.15.32:0/680779442 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c80d91e0 con 0x7ff3c810e7d0 2021-04-09T18:32:45.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c6e1d700 1 --2- 172.21.15.32:0/680779442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3c810e7d0 0x7ff3c80d7ff0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7ff3c0006e50 tx=0x7ff3c0009df0).ready entity=mon.1 client_cookie=83390c6a1fda26c9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:45.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c561a700 1 -- 172.21.15.32:0/680779442 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3c000b510 con 0x7ff3c810e7d0 2021-04-09T18:32:45.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c561a700 1 -- 172.21.15.32:0/680779442 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c0003030 con 0x7ff3c810e7d0 2021-04-09T18:32:45.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3c561a700 1 -- 172.21.15.32:0/680779442 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3c000b7d0 con 0x7ff3c810e7d0 2021-04-09T18:32:45.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3cc984700 1 -- 172.21.15.32:0/680779442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3c810e7d0 msgr2=0x7ff3c80d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.028 7ff3cc984700 1 --2- 172.21.15.32:0/680779442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3c810e7d0 0x7ff3c80d7ff0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7ff3c0006e50 tx=0x7ff3c0009df0).stop 2021-04-09T18:32:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.029 7ff3cc984700 1 -- 172.21.15.32:0/680779442 shutdown_connections 2021-04-09T18:32:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.029 7ff3cc984700 1 --2- 172.21.15.32:0/680779442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3c810e7d0 0x7ff3c80d7ff0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.029 7ff3cc984700 1 -- 172.21.15.32:0/680779442 >> 172.21.15.32:0/680779442 conn(0x7ff3c80ce5c0 msgr2=0x7ff3c80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:45.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.029 7ff3cc984700 1 -- 172.21.15.32:0/680779442 shutdown_connections 2021-04-09T18:32:45.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.029 7ff3cc984700 1 -- 172.21.15.32:0/680779442 wait complete. 2021-04-09T18:32:45.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.049 7ff3cc984700 1 Processor -- start 2021-04-09T18:32:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.049 7ff3cc984700 1 -- start start 2021-04-09T18:32:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3cc984700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3c81b85b0 0x7ff3c81b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3cc984700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3c808c210 0x7ff3c808c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3cc984700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c80ab9a0 0x7ff3c80abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3cc984700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c808cc30 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3cc984700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c819a630 con 0x7ff3c81b85b0 2021-04-09T18:32:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3cc984700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c80ac3c0 con 0x7ff3c808c210 2021-04-09T18:32:45.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c6e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c80ab9a0 0x7ff3c80abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c6e1d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c80ab9a0 0x7ff3c80abda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58014/0 (socket says 172.21.15.32:58014) 2021-04-09T18:32:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c6e1d700 1 -- 172.21.15.32:0/2878441852 learned_addr learned my addr 172.21.15.32:0/2878441852 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c661c700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3c81b85b0 0x7ff3c81b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c5e1b700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3c808c210 0x7ff3c808c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c6e1d700 1 -- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3c808c210 msgr2=0x7ff3c808c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c6e1d700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3c808c210 0x7ff3c808c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c6e1d700 1 -- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3c81b85b0 msgr2=0x7ff3c81b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c6e1d700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3c81b85b0 0x7ff3c81b89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.050 7ff3c6e1d700 1 -- 172.21.15.32:0/2878441852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c80ac5e0 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.051 7ff3c6e1d700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c80ab9a0 0x7ff3c80abda0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 rx=0x7ff3c0009f80 tx=0x7ff3c000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:45.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.051 7ff3b6ffd700 1 -- 172.21.15.32:0/2878441852 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3c00097c0 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.051 7ff3b6ffd700 1 -- 172.21.15.32:0/2878441852 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c00093d0 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.051 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c80ac5e0 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.051 7ff3b6ffd700 1 -- 172.21.15.32:0/2878441852 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3c0007430 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.051 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c80ac5e0 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.052 7ff3b6ffd700 1 -- 172.21.15.32:0/2878441852 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff3c0007030 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.052 7ff3b6ffd700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3ac0188c0 0x7ff3ac01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.052 7ff3b6ffd700 1 -- 172.21.15.32:0/2878441852 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ff3c00330d0 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.052 7ff3c661c700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3ac0188c0 0x7ff3ac01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.052 7ff3c661c700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3ac0188c0 0x7ff3ac01adc0 crc :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:45.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.071 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c808cc30 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.075 7ff3b6ffd700 1 -- 172.21.15.32:0/2878441852 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff3c00093b0 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.263 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3c808cc30 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.264 7ff3b6ffd700 1 -- 172.21.15.32:0/2878441852 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+16865 (secure 0 0 0) 0x7ff3c00093b0 con 0x7ff3c80ab9a0 2021-04-09T18:32:45.281 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:45.282 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":229,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:32:44.859103","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:32:35.297513","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":39,"pool_name":"alpha_data","create_time":"2021-04-09 18:32:36.169493","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":40,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:32:42.658486","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":41,"pool_name":"bravo_data","create_time":"2021-04-09 18:32:43.872170","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:32:45.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3ac0188c0 msgr2=0x7ff3ac01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3ac0188c0 0x7ff3ac01adc0 crc :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c80ab9a0 msgr2=0x7ff3c80abda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c80ab9a0 0x7ff3c80abda0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 rx=0x7ff3c0009f80 tx=0x7ff3c000bd10).stop 2021-04-09T18:32:45.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 shutdown_connections 2021-04-09T18:32:45.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3ac0188c0 0x7ff3ac01adc0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c80ab9a0 0x7ff3c80abda0 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3c808c210 0x7ff3c808c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 --2- 172.21.15.32:0/2878441852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3c81b85b0 0x7ff3c81b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 >> 172.21.15.32:0/2878441852 conn(0x7ff3c81864e0 msgr2=0x7ff3c81b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:45.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 shutdown_connections 2021-04-09T18:32:45.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.266 7ff3cc984700 1 -- 172.21.15.32:0/2878441852 wait complete. 2021-04-09T18:32:45.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:32:45.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.425 7f25b8077700 1 Processor -- start 2021-04-09T18:32:45.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.426 7f25b8077700 1 -- start start 2021-04-09T18:32:45.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.426 7f25b8077700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f25b010da00 0x7f25b010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.426 7f25b8077700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b010d480 con 0x7f25b010d010 2021-04-09T18:32:45.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.426 7f25b8077700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b00d1f70 con 0x7f25b00d7ca0 2021-04-09T18:32:45.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.426 7f25b8077700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b00d8d30 con 0x7f25b010da00 2021-04-09T18:32:45.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.426 7f25b5e93700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f25b010d010 0x7f25b00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39376/0 (socket says 172.21.15.32:39376) 2021-04-09T18:32:45.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.426 7f25b5e93700 1 -- 172.21.15.32:0/4879521 learned_addr learned my addr 172.21.15.32:0/4879521 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:45.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b6694700 1 --2- 172.21.15.32:0/4879521 >> v2:172.21.15.86:3300/0 conn(0x7f25b010da00 0x7f25b010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b4e91700 1 -- 172.21.15.32:0/4879521 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886547422 0 0) 0x7f25ac000ce0 con 0x7f25b010d010 2021-04-09T18:32:45.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b4e91700 1 -- 172.21.15.32:0/4879521 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259c002fe0 con 0x7f25b010d010 2021-04-09T18:32:45.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b4e91700 1 -- 172.21.15.32:0/4879521 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578620613 0 0) 0x7f25a0001080 con 0x7f25b00d7ca0 2021-04-09T18:32:45.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b4e91700 1 -- 172.21.15.32:0/4879521 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259c003210 con 0x7f25b00d7ca0 2021-04-09T18:32:45.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b4e91700 1 -- 172.21.15.32:0/4879521 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4011746703 0 0) 0x7f25ac0010e0 con 0x7f25b010d010 2021-04-09T18:32:45.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b6694700 1 -- 172.21.15.32:0/4879521 >> v1:172.21.15.86:6789/0 conn(0x7f25b00d7ca0 legacy=0x7f25b00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b6694700 1 -- 172.21.15.32:0/4879521 >> v1:172.21.15.32:6789/0 conn(0x7f25b010d010 legacy=0x7f25b00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b6694700 1 -- 172.21.15.32:0/4879521 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00d9220 con 0x7f25b010da00 2021-04-09T18:32:45.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b6694700 1 --2- 172.21.15.32:0/4879521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25b010da00 0x7f25b010de00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f25a8006e50 tx=0x7f25a8009df0).ready entity=mon.1 client_cookie=ecc95ded7d49742c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:45.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b4e91700 1 -- 172.21.15.32:0/4879521 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25a800b510 con 0x7f25b010da00 2021-04-09T18:32:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b4e91700 1 -- 172.21.15.32:0/4879521 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a8003030 con 0x7f25b010da00 2021-04-09T18:32:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b4e91700 1 -- 172.21.15.32:0/4879521 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25a800b7d0 con 0x7f25b010da00 2021-04-09T18:32:45.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b8077700 1 -- 172.21.15.32:0/4879521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25b010da00 msgr2=0x7f25b010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.427 7f25b8077700 1 --2- 172.21.15.32:0/4879521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25b010da00 0x7f25b010de00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f25a8006e50 tx=0x7f25a8009df0).stop 2021-04-09T18:32:45.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.428 7f25b8077700 1 -- 172.21.15.32:0/4879521 shutdown_connections 2021-04-09T18:32:45.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.428 7f25b8077700 1 --2- 172.21.15.32:0/4879521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25b010da00 0x7f25b010de00 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.428 7f25b8077700 1 -- 172.21.15.32:0/4879521 >> 172.21.15.32:0/4879521 conn(0x7f25b00ce5f0 msgr2=0x7f25b00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:45.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.428 7f25b8077700 1 -- 172.21.15.32:0/4879521 shutdown_connections 2021-04-09T18:32:45.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.428 7f25b8077700 1 -- 172.21.15.32:0/4879521 wait complete. 2021-04-09T18:32:45.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.448 7f25b8077700 1 Processor -- start 2021-04-09T18:32:45.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.448 7f25b8077700 1 -- start start 2021-04-09T18:32:45.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b8077700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25b01aca20 0x7f25b01ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b8077700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b018bd40 0x7f25b018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b8077700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25b008fba0 0x7f25b008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b8077700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b018c7e0 con 0x7f25b018bd40 2021-04-09T18:32:45.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b8077700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b018ca30 con 0x7f25b01aca20 2021-04-09T18:32:45.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b8077700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00905f0 con 0x7f25b008fba0 2021-04-09T18:32:45.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b5692700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b018bd40 0x7f25b018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b5692700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b018bd40 0x7f25b018c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58026/0 (socket says 172.21.15.32:58026) 2021-04-09T18:32:45.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b5692700 1 -- 172.21.15.32:0/2752552436 learned_addr learned my addr 172.21.15.32:0/2752552436 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:45.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b6694700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25b008fba0 0x7f25b008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.449 7f25b5e93700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25b01aca20 0x7f25b01ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25b5692700 1 -- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25b008fba0 msgr2=0x7f25b008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25b5692700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25b008fba0 0x7f25b008ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25b5692700 1 -- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25b01aca20 msgr2=0x7f25b01ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25b5692700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25b01aca20 0x7f25b01ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25b5692700 1 -- 172.21.15.32:0/2752552436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00b4300 con 0x7f25b018bd40 2021-04-09T18:32:45.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25b5692700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b018bd40 0x7f25b018c140 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 rx=0x7f25a000b530 tx=0x7f25a000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:45.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25a67fc700 1 -- 172.21.15.32:0/2752552436 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25a000bcc0 con 0x7f25b018bd40 2021-04-09T18:32:45.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25a67fc700 1 -- 172.21.15.32:0/2752552436 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a0000e20 con 0x7f25b018bd40 2021-04-09T18:32:45.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25a67fc700 1 -- 172.21.15.32:0/2752552436 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25a00054f0 con 0x7f25b018bd40 2021-04-09T18:32:45.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25b8077700 1 -- 172.21.15.32:0/2752552436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b00b4300 con 0x7f25b018bd40 2021-04-09T18:32:45.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.450 7f25b8077700 1 -- 172.21.15.32:0/2752552436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b00b4300 con 0x7f25b018bd40 2021-04-09T18:32:45.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.451 7f25a67fc700 1 -- 172.21.15.32:0/2752552436 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f25a0006240 con 0x7f25b018bd40 2021-04-09T18:32:45.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.451 7f25a67fc700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2598018930 0x7f259801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.451 7f25a67fc700 1 -- 172.21.15.32:0/2752552436 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f25a0032d80 con 0x7f25b018bd40 2021-04-09T18:32:45.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.451 7f25b5e93700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2598018930 0x7f259801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.451 7f25b5e93700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2598018930 0x7f259801ae10 crc :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:45.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.471 7f25b8077700 1 -- 172.21.15.32:0/2752552436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b0182d70 con 0x7f25b018bd40 2021-04-09T18:32:45.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.474 7f25a67fc700 1 -- 172.21.15.32:0/2752552436 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f25a0006030 con 0x7f25b018bd40 2021-04-09T18:32:45.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.663 7f25b8077700 1 -- 172.21.15.32:0/2752552436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25b018c760 con 0x7f25b018bd40 2021-04-09T18:32:45.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.664 7f25a67fc700 1 -- 172.21.15.32:0/2752552436 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+16865 (secure 0 0 0) 0x7f25a0006240 con 0x7f25b018bd40 2021-04-09T18:32:45.681 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:45.681 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":229,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:32:44.859103","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:32:35.297513","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":39,"pool_name":"alpha_data","create_time":"2021-04-09 18:32:36.169493","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":40,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:32:42.658486","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":41,"pool_name":"bravo_data","create_time":"2021-04-09 18:32:43.872170","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:32:45.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 -- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2598018930 msgr2=0x7f259801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2598018930 0x7f259801ae10 crc :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 -- 172.21.15.32:0/2752552436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b018bd40 msgr2=0x7f25b018c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b018bd40 0x7f25b018c140 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 rx=0x7f25a000b530 tx=0x7f25a000b8d0).stop 2021-04-09T18:32:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 -- 172.21.15.32:0/2752552436 shutdown_connections 2021-04-09T18:32:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2598018930 0x7f259801ae10 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25b018bd40 0x7f25b018c140 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25b008fba0 0x7f25b008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 --2- 172.21.15.32:0/2752552436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25b01aca20 0x7f25b01ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 -- 172.21.15.32:0/2752552436 >> 172.21.15.32:0/2752552436 conn(0x7f25b019b460 msgr2=0x7f25b019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:45.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 -- 172.21.15.32:0/2752552436 shutdown_connections 2021-04-09T18:32:45.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.666 7f25b8077700 1 -- 172.21.15.32:0/2752552436 wait complete. 2021-04-09T18:32:45.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set bravo standby_count_wanted 0 2021-04-09T18:32:45.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.829 7f1ebbc4b700 1 Processor -- start 2021-04-09T18:32:45.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.829 7f1ebbc4b700 1 -- start start 2021-04-09T18:32:45.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1ebbc4b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1eb410d010 0x7f1eb40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1ebbc4b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40d1da0 0x7f1eb40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1ebbc4b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb410d480 con 0x7f1eb40d82c0 2021-04-09T18:32:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1ebbc4b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb40d9220 con 0x7f1eb410d010 2021-04-09T18:32:45.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1ebbc4b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40d86c0 con 0x7f1eb40d1da0 2021-04-09T18:32:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eba268700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1eb40d82c0 0x7f1eb410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37318/0 (socket says 172.21.15.32:37318) 2021-04-09T18:32:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eba268700 1 -- 172.21.15.32:0/2207940471 learned_addr learned my addr 172.21.15.32:0/2207940471 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eb9266700 1 --2- 172.21.15.32:0/2207940471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40d1da0 0x7f1eb40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eb9a67700 1 --2- 172.21.15.32:0/2207940471 >> v2:172.21.15.86:3300/0 conn(0x7f1eb410d010 0x7f1eb40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eb8a65700 1 -- 172.21.15.32:0/2207940471 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639859502 0 0) 0x7f1eac001440 con 0x7f1eb40d82c0 2021-04-09T18:32:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eb8a65700 1 -- 172.21.15.32:0/2207940471 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0002fe0 con 0x7f1eb40d82c0 2021-04-09T18:32:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eb9a67700 1 -- 172.21.15.32:0/2207940471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40d1da0 msgr2=0x7f1eb40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:45.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eb9a67700 1 --2- 172.21.15.32:0/2207940471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40d1da0 0x7f1eb40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eb9a67700 1 -- 172.21.15.32:0/2207940471 >> v1:172.21.15.86:6789/0 conn(0x7f1eb40d82c0 legacy=0x7f1eb410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eb9a67700 1 -- 172.21.15.32:0/2207940471 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb410e1f0 con 0x7f1eb410d010 2021-04-09T18:32:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.830 7f1eb9266700 1 --2- 172.21.15.32:0/2207940471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40d1da0 0x7f1eb40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:45.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.831 7f1eb9a67700 1 --2- 172.21.15.32:0/2207940471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb410d010 0x7f1eb40d1780 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f1eb0006650 tx=0x7f1eb0009a80).ready entity=mon.1 client_cookie=44cd193d5734cfcd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.831 7f1eb8a65700 1 -- 172.21.15.32:0/2207940471 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1eb00009b0 con 0x7f1eb410d010 2021-04-09T18:32:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.831 7f1eb8a65700 1 -- 172.21.15.32:0/2207940471 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb000c0c0 con 0x7f1eb410d010 2021-04-09T18:32:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.831 7f1eb8a65700 1 -- 172.21.15.32:0/2207940471 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1eb000c510 con 0x7f1eb410d010 2021-04-09T18:32:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.831 7f1ebbc4b700 1 -- 172.21.15.32:0/2207940471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb410d010 msgr2=0x7f1eb40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.831 7f1ebbc4b700 1 --2- 172.21.15.32:0/2207940471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb410d010 0x7f1eb40d1780 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f1eb0006650 tx=0x7f1eb0009a80).stop 2021-04-09T18:32:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.832 7f1ebbc4b700 1 -- 172.21.15.32:0/2207940471 shutdown_connections 2021-04-09T18:32:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.832 7f1ebbc4b700 1 --2- 172.21.15.32:0/2207940471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40d1da0 0x7f1eb40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.832 7f1ebbc4b700 1 --2- 172.21.15.32:0/2207940471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb410d010 0x7f1eb40d1780 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.832 7f1ebbc4b700 1 -- 172.21.15.32:0/2207940471 >> 172.21.15.32:0/2207940471 conn(0x7f1eb40ce5f0 msgr2=0x7f1eb40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.832 7f1ebbc4b700 1 -- 172.21.15.32:0/2207940471 shutdown_connections 2021-04-09T18:32:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.832 7f1ebbc4b700 1 -- 172.21.15.32:0/2207940471 wait complete. 2021-04-09T18:32:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.852 7f1ebbc4b700 1 Processor -- start 2021-04-09T18:32:45.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.852 7f1ebbc4b700 1 -- start start 2021-04-09T18:32:45.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.852 7f1ebbc4b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40b2140 0x7f1eb409d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.852 7f1ebbc4b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb409e010 0x7f1eb4194a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1ebbc4b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb41950a0 0x7f1eb40a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1ebbc4b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb409d470 con 0x7f1eb409e010 2021-04-09T18:32:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1ebbc4b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb409e410 con 0x7f1eb41950a0 2021-04-09T18:32:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1ebbc4b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb41954a0 con 0x7f1eb40b2140 2021-04-09T18:32:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9266700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb409e010 0x7f1eb4194a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9266700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb409e010 0x7f1eb4194a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58038/0 (socket says 172.21.15.32:58038) 2021-04-09T18:32:45.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9266700 1 -- 172.21.15.32:0/904853405 learned_addr learned my addr 172.21.15.32:0/904853405 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eba268700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb41950a0 0x7f1eb40a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9a67700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40b2140 0x7f1eb409d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9266700 1 -- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40b2140 msgr2=0x7f1eb409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9266700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40b2140 0x7f1eb409d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9266700 1 -- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb41950a0 msgr2=0x7f1eb40a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9266700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb41950a0 0x7f1eb40a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9266700 1 -- 172.21.15.32:0/904853405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb41956c0 con 0x7f1eb409e010 2021-04-09T18:32:45.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eb9266700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb409e010 0x7f1eb4194a80 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 rx=0x7f1ea400b580 tx=0x7f1ea400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eaa7fc700 1 -- 172.21.15.32:0/904853405 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea400bd10 con 0x7f1eb409e010 2021-04-09T18:32:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eaa7fc700 1 -- 172.21.15.32:0/904853405 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea4000c90 con 0x7f1eb409e010 2021-04-09T18:32:45.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb41956c0 con 0x7f1eb409e010 2021-04-09T18:32:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.853 7f1eaa7fc700 1 -- 172.21.15.32:0/904853405 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea4006b00 con 0x7f1eb409e010 2021-04-09T18:32:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.854 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb41956c0 con 0x7f1eb409e010 2021-04-09T18:32:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.854 7f1eaa7fc700 1 -- 172.21.15.32:0/904853405 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1ea4006b00 con 0x7f1eb409e010 2021-04-09T18:32:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.854 7f1eaa7fc700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e9c018930 0x7f1e9c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:45.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.854 7f1eaa7fc700 1 -- 172.21.15.32:0/904853405 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f1ea4032d20 con 0x7f1eb409e010 2021-04-09T18:32:45.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.854 7f1eb9a67700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e9c018930 0x7f1e9c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:45.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.855 7f1eb9a67700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e9c018930 0x7f1e9c01ae10 crc :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:45.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.874 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb41b7040 con 0x7f1eb409e010 2021-04-09T18:32:45.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:45.877 7f1eaa7fc700 1 -- 172.21.15.32:0/904853405 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1ea4000c90 con 0x7f1eb409e010 2021-04-09T18:32:46.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.071 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "bravo", "val": "0"} v 0) v1 -- 0x7f1eb410d480 con 0x7f1eb409e010 2021-04-09T18:32:46.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.865 7f1eaa7fc700 1 -- 172.21.15.32:0/904853405 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "bravo", "val": "0"}]=0 v385) v1 ==== 117+0+0 (secure 0 0 0) 0x7f1ea4006b00 con 0x7f1eb409e010 2021-04-09T18:32:46.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e9c018930 msgr2=0x7f1e9c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:46.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e9c018930 0x7f1e9c01ae10 crc :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:46.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb409e010 msgr2=0x7f1eb4194a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:46.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb409e010 0x7f1eb4194a80 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 rx=0x7f1ea400b580 tx=0x7f1ea400b920).stop 2021-04-09T18:32:46.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 shutdown_connections 2021-04-09T18:32:46.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e9c018930 0x7f1e9c01ae10 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:46.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb409e010 0x7f1eb4194a80 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:46.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb41950a0 0x7f1eb40a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:46.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 --2- 172.21.15.32:0/904853405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eb40b2140 0x7f1eb409d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 >> 172.21.15.32:0/904853405 conn(0x7f1eb40b27e0 msgr2=0x7f1eb41b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 shutdown_connections 2021-04-09T18:32:46.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:46.889 7f1ebbc4b700 1 -- 172.21.15.32:0/904853405 wait complete. 2021-04-09T18:32:46.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:47.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.051 7fed7f783700 1 Processor -- start 2021-04-09T18:32:47.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.052 7fed7f783700 1 -- start start 2021-04-09T18:32:47.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.052 7fed7f783700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fed7810d010 0x7fed780d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.052 7fed7f783700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fed780d1da0 0x7fed780d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.052 7fed7f783700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7810d480 con 0x7fed780d82c0 2021-04-09T18:32:47.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.052 7fed7f783700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed780d8d30 con 0x7fed7810d010 2021-04-09T18:32:47.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.052 7fed7f783700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed780d9220 con 0x7fed780d1da0 2021-04-09T18:32:47.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.052 7fed7d59f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fed7810d010 0x7fed780d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.052 7fed7dda0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fed780d82c0 0x7fed7810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39406/0 (socket says 172.21.15.32:39406) 2021-04-09T18:32:47.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7dda0700 1 -- 172.21.15.32:0/2268498447 learned_addr learned my addr 172.21.15.32:0/2268498447 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:47.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7cd9e700 1 --2- 172.21.15.32:0/2268498447 >> v2:172.21.15.86:3300/0 conn(0x7fed780d1da0 0x7fed780d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed6ffff700 1 -- 172.21.15.32:0/2268498447 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460721600 0 0) 0x7fed70000d30 con 0x7fed780d82c0 2021-04-09T18:32:47.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed6ffff700 1 -- 172.21.15.32:0/2268498447 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed64002fe0 con 0x7fed780d82c0 2021-04-09T18:32:47.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7d59f700 1 -- 172.21.15.32:0/2268498447 >> v2:172.21.15.86:3300/0 conn(0x7fed780d1da0 msgr2=0x7fed780d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:47.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7d59f700 1 --2- 172.21.15.32:0/2268498447 >> v2:172.21.15.86:3300/0 conn(0x7fed780d1da0 0x7fed780d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7d59f700 1 -- 172.21.15.32:0/2268498447 >> v1:172.21.15.32:6789/0 conn(0x7fed780d82c0 legacy=0x7fed7810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7d59f700 1 -- 172.21.15.32:0/2268498447 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed780d86f0 con 0x7fed7810d010 2021-04-09T18:32:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7cd9e700 1 --2- 172.21.15.32:0/2268498447 >> v2:172.21.15.86:3300/0 conn(0x7fed780d1da0 0x7fed780d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7d59f700 1 --2- 172.21.15.32:0/2268498447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7810d010 0x7fed780d1780 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7fed74006650 tx=0x7fed74009a80).ready entity=mon.0 client_cookie=9cead61d077527e2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:47.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed6ffff700 1 -- 172.21.15.32:0/2268498447 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed740009b0 con 0x7fed7810d010 2021-04-09T18:32:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed6ffff700 1 -- 172.21.15.32:0/2268498447 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7400c0c0 con 0x7fed7810d010 2021-04-09T18:32:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed6ffff700 1 -- 172.21.15.32:0/2268498447 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed740009b0 con 0x7fed7810d010 2021-04-09T18:32:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7f783700 1 -- 172.21.15.32:0/2268498447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7810d010 msgr2=0x7fed780d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.053 7fed7f783700 1 --2- 172.21.15.32:0/2268498447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7810d010 0x7fed780d1780 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7fed74006650 tx=0x7fed74009a80).stop 2021-04-09T18:32:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.054 7fed7f783700 1 -- 172.21.15.32:0/2268498447 shutdown_connections 2021-04-09T18:32:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.054 7fed7f783700 1 --2- 172.21.15.32:0/2268498447 >> v2:172.21.15.86:3300/0 conn(0x7fed780d1da0 0x7fed780d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.054 7fed7f783700 1 --2- 172.21.15.32:0/2268498447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7810d010 0x7fed780d1780 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.054 7fed7f783700 1 -- 172.21.15.32:0/2268498447 >> 172.21.15.32:0/2268498447 conn(0x7fed780ce5f0 msgr2=0x7fed780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:47.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.054 7fed7f783700 1 -- 172.21.15.32:0/2268498447 shutdown_connections 2021-04-09T18:32:47.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.054 7fed7f783700 1 -- 172.21.15.32:0/2268498447 wait complete. 2021-04-09T18:32:47.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.074 7fed7f783700 1 Processor -- start 2021-04-09T18:32:47.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.074 7fed7f783700 1 -- start start 2021-04-09T18:32:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7f783700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed78179bf0 0x7fed78179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7f783700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7817c540 0x7fed7817c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7f783700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed780aca60 0x7fed780ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7f783700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7817cf60 con 0x7fed7817c540 2021-04-09T18:32:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7f783700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed780ad480 con 0x7fed780aca60 2021-04-09T18:32:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7f783700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed780ad6a0 con 0x7fed78179bf0 2021-04-09T18:32:47.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7cd9e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7817c540 0x7fed7817c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7cd9e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7817c540 0x7fed7817c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58052/0 (socket says 172.21.15.32:58052) 2021-04-09T18:32:47.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7cd9e700 1 -- 172.21.15.32:0/785749233 learned_addr learned my addr 172.21.15.32:0/785749233 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:47.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7dda0700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed780aca60 0x7fed780ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7d59f700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed78179bf0 0x7fed78179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7cd9e700 1 -- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed78179bf0 msgr2=0x7fed78179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7cd9e700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed78179bf0 0x7fed78179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7cd9e700 1 -- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed780aca60 msgr2=0x7fed780ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.075 7fed7cd9e700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed780aca60 0x7fed780ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.076 7fed7cd9e700 1 -- 172.21.15.32:0/785749233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed780ad8c0 con 0x7fed7817c540 2021-04-09T18:32:47.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.076 7fed7cd9e700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7817c540 0x7fed7817c940 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 rx=0x7fed68006f40 tx=0x7fed6800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:47.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.076 7fed6dffb700 1 -- 172.21.15.32:0/785749233 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed6800bd80 con 0x7fed7817c540 2021-04-09T18:32:47.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.076 7fed6dffb700 1 -- 172.21.15.32:0/785749233 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68000c90 con 0x7fed7817c540 2021-04-09T18:32:47.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.076 7fed7f783700 1 -- 172.21.15.32:0/785749233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed780ad8c0 con 0x7fed7817c540 2021-04-09T18:32:47.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.076 7fed6dffb700 1 -- 172.21.15.32:0/785749233 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed68002220 con 0x7fed7817c540 2021-04-09T18:32:47.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.076 7fed7f783700 1 -- 172.21.15.32:0/785749233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed780ad8c0 con 0x7fed7817c540 2021-04-09T18:32:47.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.076 7fed6dffb700 1 -- 172.21.15.32:0/785749233 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fed6800a000 con 0x7fed7817c540 2021-04-09T18:32:47.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.077 7fed6dffb700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed58018930 0x7fed5801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.077 7fed6dffb700 1 -- 172.21.15.32:0/785749233 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fed68032d10 con 0x7fed7817c540 2021-04-09T18:32:47.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.077 7fed7d59f700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed58018930 0x7fed5801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.077 7fed7d59f700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed58018930 0x7fed5801ae10 crc :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:47.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.096 7fed7f783700 1 -- 172.21.15.32:0/785749233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7817cf60 con 0x7fed7817c540 2021-04-09T18:32:47.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.100 7fed6dffb700 1 -- 172.21.15.32:0/785749233 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fed68000c90 con 0x7fed7817c540 2021-04-09T18:32:47.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.292 7fed7f783700 1 -- 172.21.15.32:0/785749233 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed7817cf60 con 0x7fed7817c540 2021-04-09T18:32:47.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.292 7fed6dffb700 1 -- 172.21.15.32:0/785749233 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 385 v385) v1 ==== 95+0+3719 (secure 0 0 0) 0x7fed68006b00 con 0x7fed7817c540 2021-04-09T18:32:47.309 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 385 2021-04-09T18:32:47.310 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:47.310 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":385,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11512,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3237972186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3237972186},{"type":"v1","addr":"172.21.15.32:6834","nonce":3237972186}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378},{"gid":11515,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4276774592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4276774592},{"type":"v1","addr":"172.21.15.32:6835","nonce":4276774592}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378}],"filesystems":[{"mdsmap":{"epoch":381,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:39.408909","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11500":{"gid":11500,"name":"b","rank":0,"incarnation":379,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3107210651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3107210651},{"type":"v1","addr":"172.21.15.86:6836","nonce":3107210651}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":385,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:46.863593","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11503":{"gid":11503,"name":"d","rank":0,"incarnation":382,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/2051228504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2051228504},{"type":"v1","addr":"172.21.15.86:6837","nonce":2051228504}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:47.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.294 7fed7f783700 1 -- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed58018930 msgr2=0x7fed5801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.294 7fed7f783700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed58018930 0x7fed5801ae10 crc :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.294 7fed7f783700 1 -- 172.21.15.32:0/785749233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7817c540 msgr2=0x7fed7817c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.294 7fed7f783700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7817c540 0x7fed7817c940 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 rx=0x7fed68006f40 tx=0x7fed6800b990).stop 2021-04-09T18:32:47.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.295 7fed7f783700 1 -- 172.21.15.32:0/785749233 shutdown_connections 2021-04-09T18:32:47.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.295 7fed7f783700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed58018930 0x7fed5801ae10 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.295 7fed7f783700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed7817c540 0x7fed7817c940 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.295 7fed7f783700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed780aca60 0x7fed780ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.295 7fed7f783700 1 --2- 172.21.15.32:0/785749233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed78179bf0 0x7fed78179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.295 7fed7f783700 1 -- 172.21.15.32:0/785749233 >> 172.21.15.32:0/785749233 conn(0x7fed781acae0 msgr2=0x7fed7817ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:47.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.295 7fed7f783700 1 -- 172.21.15.32:0/785749233 shutdown_connections 2021-04-09T18:32:47.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.295 7fed7f783700 1 -- 172.21.15.32:0/785749233 wait complete. 2021-04-09T18:32:47.322 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:32:47.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.455 7ffaee83a700 1 Processor -- start 2021-04-09T18:32:47.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffaee83a700 1 -- start start 2021-04-09T18:32:47.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffaee83a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae80d7ca0 0x7ffae80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffaee83a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffae810da00 0x7ffae810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffaee83a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae810d480 con 0x7ffae810d010 2021-04-09T18:32:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffaee83a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae80d9220 con 0x7ffae810da00 2021-04-09T18:32:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffaee83a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae80d86c0 con 0x7ffae80d7ca0 2021-04-09T18:32:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffaece57700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ffae810da00 0x7ffae810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffae7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffae810d010 0x7ffae80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39418/0 (socket says 172.21.15.32:39418) 2021-04-09T18:32:47.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffae7fff700 1 -- 172.21.15.32:0/3696468336 learned_addr learned my addr 172.21.15.32:0/3696468336 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:47.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.456 7ffadffff700 1 --2- 172.21.15.32:0/3696468336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae80d7ca0 0x7ffae80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffae77fe700 1 -- 172.21.15.32:0/3696468336 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989503021 0 0) 0x7ffad8000ce0 con 0x7ffae810d010 2021-04-09T18:32:47.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffae77fe700 1 -- 172.21.15.32:0/3696468336 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad4002fe0 con 0x7ffae810d010 2021-04-09T18:32:47.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffaece57700 1 -- 172.21.15.32:0/3696468336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae80d7ca0 msgr2=0x7ffae80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:47.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffaece57700 1 --2- 172.21.15.32:0/3696468336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae80d7ca0 0x7ffae80d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffaece57700 1 -- 172.21.15.32:0/3696468336 >> v1:172.21.15.32:6789/0 conn(0x7ffae810d010 legacy=0x7ffae80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffaece57700 1 -- 172.21.15.32:0/3696468336 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae80d1f70 con 0x7ffae810da00 2021-04-09T18:32:47.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffaece57700 1 --2- 172.21.15.32:0/3696468336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae810da00 0x7ffae810de00 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 rx=0x7ffae0006e30 tx=0x7ffae0009e60).ready entity=mon.0 client_cookie=c566a13a142ea3bd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffae77fe700 1 -- 172.21.15.32:0/3696468336 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffae0006a50 con 0x7ffae810da00 2021-04-09T18:32:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffae77fe700 1 -- 172.21.15.32:0/3696468336 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae000b300 con 0x7ffae810da00 2021-04-09T18:32:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffae77fe700 1 -- 172.21.15.32:0/3696468336 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffae000b750 con 0x7ffae810da00 2021-04-09T18:32:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffaee83a700 1 -- 172.21.15.32:0/3696468336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae810da00 msgr2=0x7ffae810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.457 7ffaee83a700 1 --2- 172.21.15.32:0/3696468336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae810da00 0x7ffae810de00 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 rx=0x7ffae0006e30 tx=0x7ffae0009e60).stop 2021-04-09T18:32:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.458 7ffaee83a700 1 -- 172.21.15.32:0/3696468336 shutdown_connections 2021-04-09T18:32:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.458 7ffaee83a700 1 --2- 172.21.15.32:0/3696468336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae810da00 0x7ffae810de00 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.458 7ffaee83a700 1 --2- 172.21.15.32:0/3696468336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae80d7ca0 0x7ffae80d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.458 7ffaee83a700 1 -- 172.21.15.32:0/3696468336 >> 172.21.15.32:0/3696468336 conn(0x7ffae80ce5f0 msgr2=0x7ffae80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:47.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.458 7ffaee83a700 1 -- 172.21.15.32:0/3696468336 shutdown_connections 2021-04-09T18:32:47.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.458 7ffaee83a700 1 -- 172.21.15.32:0/3696468336 wait complete. 2021-04-09T18:32:47.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.478 7ffaee83a700 1 Processor -- start 2021-04-09T18:32:47.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.478 7ffaee83a700 1 -- start start 2021-04-09T18:32:47.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffaee83a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae81ac380 0x7ffae818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffaee83a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffae81ac780 0x7ffae818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffaee83a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae8183520 0x7ffae8183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffaee83a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae81ace10 con 0x7ffae81ac380 2021-04-09T18:32:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffaee83a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8189cf0 con 0x7ffae81ac780 2021-04-09T18:32:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffaee83a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8183f40 con 0x7ffae8183520 2021-04-09T18:32:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae81ac380 0x7ffae818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae81ac380 0x7ffae818a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58066/0 (socket says 172.21.15.32:58066) 2021-04-09T18:32:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae7fff700 1 -- 172.21.15.32:0/4261889722 learned_addr learned my addr 172.21.15.32:0/4261889722 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffaece57700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae8183520 0x7ffae8183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffadffff700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffae81ac780 0x7ffae818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae7fff700 1 -- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae8183520 msgr2=0x7ffae8183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae7fff700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae8183520 0x7ffae8183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae7fff700 1 -- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffae81ac780 msgr2=0x7ffae818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae7fff700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffae81ac780 0x7ffae818a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae7fff700 1 -- 172.21.15.32:0/4261889722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8184160 con 0x7ffae81ac380 2021-04-09T18:32:47.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae7fff700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae81ac380 0x7ffae818a230 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 rx=0x7ffad8006f00 tx=0x7ffad800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:47.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae57fa700 1 -- 172.21.15.32:0/4261889722 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffad800bd30 con 0x7ffae81ac380 2021-04-09T18:32:47.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae57fa700 1 -- 172.21.15.32:0/4261889722 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad8006030 con 0x7ffae81ac380 2021-04-09T18:32:47.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae8184160 con 0x7ffae81ac380 2021-04-09T18:32:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.479 7ffae57fa700 1 -- 172.21.15.32:0/4261889722 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffad8004620 con 0x7ffae81ac380 2021-04-09T18:32:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.480 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae8184160 con 0x7ffae81ac380 2021-04-09T18:32:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.480 7ffae57fa700 1 -- 172.21.15.32:0/4261889722 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ffad80043a0 con 0x7ffae81ac380 2021-04-09T18:32:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.480 7ffae57fa700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffacc018930 0x7ffacc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.480 7ffae57fa700 1 -- 172.21.15.32:0/4261889722 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ffad8032d10 con 0x7ffae81ac380 2021-04-09T18:32:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.481 7ffadffff700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffacc018930 0x7ffacc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.481 7ffadffff700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffacc018930 0x7ffacc01ae10 crc :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:47.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.500 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae8182d10 con 0x7ffae81ac380 2021-04-09T18:32:47.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.504 7ffae57fa700 1 -- 172.21.15.32:0/4261889722 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ffad80009b0 con 0x7ffae81ac380 2021-04-09T18:32:47.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.692 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffae8182d10 con 0x7ffae81ac380 2021-04-09T18:32:47.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.693 7ffae57fa700 1 -- 172.21.15.32:0/4261889722 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) v1 ==== 74+0+16852 (secure 0 0 0) 0x7ffad8006010 con 0x7ffae81ac380 2021-04-09T18:32:47.710 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:47.711 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":230,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:32:45.860593","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"alpha_metadata","create_time":"2021-04-09 18:32:35.297513","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"alpha"}}},{"pool":39,"pool_name":"alpha_data","create_time":"2021-04-09 18:32:36.169493","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"alpha"}}},{"pool":40,"pool_name":"bravo_metadata","create_time":"2021-04-09 18:32:42.658486","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bravo"}}},{"pool":41,"pool_name":"bravo_data","create_time":"2021-04-09 18:32:43.872170","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bravo"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:32:47.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffacc018930 msgr2=0x7ffacc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffacc018930 0x7ffacc01ae10 crc :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae81ac380 msgr2=0x7ffae818a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae81ac380 0x7ffae818a230 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 rx=0x7ffad8006f00 tx=0x7ffad800b940).stop 2021-04-09T18:32:47.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 shutdown_connections 2021-04-09T18:32:47.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffacc018930 0x7ffacc01ae10 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffae8183520 0x7ffae8183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffae81ac780 0x7ffae818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 --2- 172.21.15.32:0/4261889722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffae81ac380 0x7ffae818a230 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 >> 172.21.15.32:0/4261889722 conn(0x7ffae819b890 msgr2=0x7ffae81a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:47.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 shutdown_connections 2021-04-09T18:32:47.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.695 7ffaee83a700 1 -- 172.21.15.32:0/4261889722 wait complete. 2021-04-09T18:32:47.723 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:32:47.724 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:32:47.724 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:32:47.724 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:32:47.725 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:32:47.725 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:32:47.725 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:32:47.726 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:32:47.726 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:32:47.726 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:32:47.727 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:32:47.727 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:32:47.727 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:32:47.728 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:32:47.728 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:32:47.728 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:32:47.733 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:32:47.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:32:47.734 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:32:47.735 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:32:47.736 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:32:47.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:32:47.738 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:32:47.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:32:47.774 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:32:47.775 DEBUG:teuthology.parallel:result is None 2021-04-09T18:32:47.775 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:32:47.776 DEBUG:teuthology.parallel:result is None 2021-04-09T18:32:47.779 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:32:47.780 DEBUG:teuthology.parallel:result is None 2021-04-09T18:32:47.781 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:32:47.782 DEBUG:teuthology.parallel:result is None 2021-04-09T18:32:47.782 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:47.923 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:32:47.923 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:32:47.926 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:32:47.927 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:32:47.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.915 7f34dafda700 1 Processor -- start 2021-04-09T18:32:47.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34dafda700 1 -- start start 2021-04-09T18:32:47.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34dafda700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34d410d010 0x7f34d40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34dafda700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f34d40d8490 0x7f34d410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34dafda700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d410d480 con 0x7f34d40d1da0 2021-04-09T18:32:47.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34dafda700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d40d8d30 con 0x7f34d410d010 2021-04-09T18:32:47.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34dafda700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d40d9220 con 0x7f34d40d8490 2021-04-09T18:32:47.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34d8df6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34d410d010 0x7f34d40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34cbfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f34d40d1da0 0x7f34d40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39460/0 (socket says 172.21.15.32:39460) 2021-04-09T18:32:47.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34d8df6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34d410d010 0x7f34d40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58098/0 (socket says 172.21.15.32:58098) 2021-04-09T18:32:47.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34cbfff700 1 -- 172.21.15.32:0/3380445069 learned_addr learned my addr 172.21.15.32:0/3380445069 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:47.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.916 7f34d95f7700 1 --2- 172.21.15.32:0/3380445069 >> v2:172.21.15.86:3300/0 conn(0x7f34d40d8490 0x7f34d410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34d8df6700 1 -- 172.21.15.32:0/3380445069 >> v2:172.21.15.86:3300/0 conn(0x7f34d40d8490 msgr2=0x7f34d410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34d8df6700 1 --2- 172.21.15.32:0/3380445069 >> v2:172.21.15.86:3300/0 conn(0x7f34d40d8490 0x7f34d410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34d8df6700 1 -- 172.21.15.32:0/3380445069 >> v1:172.21.15.32:6789/0 conn(0x7f34d40d1da0 legacy=0x7f34d40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34d8df6700 1 -- 172.21.15.32:0/3380445069 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d410e020 con 0x7f34d410d010 2021-04-09T18:32:47.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34cb7fe700 1 -- 172.21.15.32:0/3380445069 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036530092 0 0) 0x7f34c4000ce0 con 0x7f34d40d1da0 2021-04-09T18:32:47.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34d8df6700 1 --2- 172.21.15.32:0/3380445069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d410d010 0x7f34d40d1780 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 rx=0x7f34d0006650 tx=0x7f34d0009a00).ready entity=mon.0 client_cookie=f2ad006bad3ceca2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:47.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34cb7fe700 1 -- 172.21.15.32:0/3380445069 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34d0009df0 con 0x7f34d410d010 2021-04-09T18:32:47.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34cb7fe700 1 -- 172.21.15.32:0/3380445069 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d00009b0 con 0x7f34d410d010 2021-04-09T18:32:47.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34cb7fe700 1 -- 172.21.15.32:0/3380445069 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34d000c2b0 con 0x7f34d410d010 2021-04-09T18:32:47.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34dafda700 1 -- 172.21.15.32:0/3380445069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d410d010 msgr2=0x7f34d40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.917 7f34dafda700 1 --2- 172.21.15.32:0/3380445069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d410d010 0x7f34d40d1780 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 rx=0x7f34d0006650 tx=0x7f34d0009a00).stop 2021-04-09T18:32:47.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.918 7f34dafda700 1 -- 172.21.15.32:0/3380445069 shutdown_connections 2021-04-09T18:32:47.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.918 7f34dafda700 1 --2- 172.21.15.32:0/3380445069 >> v2:172.21.15.86:3300/0 conn(0x7f34d40d8490 0x7f34d410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.918 7f34dafda700 1 --2- 172.21.15.32:0/3380445069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d410d010 0x7f34d40d1780 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.918 7f34dafda700 1 -- 172.21.15.32:0/3380445069 >> 172.21.15.32:0/3380445069 conn(0x7f34d40ce5f0 msgr2=0x7f34d40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.918 7f34dafda700 1 -- 172.21.15.32:0/3380445069 shutdown_connections 2021-04-09T18:32:47.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.918 7f34dafda700 1 -- 172.21.15.32:0/3380445069 wait complete. 2021-04-09T18:32:47.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.938 7f34dafda700 1 Processor -- start 2021-04-09T18:32:47.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.938 7f34dafda700 1 -- start start 2021-04-09T18:32:47.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34dafda700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d41aca20 0x7f34d41ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34dafda700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34d418bd40 0x7f34d418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34dafda700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d408fba0 0x7f34d408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34dafda700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d418c7e0 con 0x7f34d408fba0 2021-04-09T18:32:47.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34dafda700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d418ca30 con 0x7f34d41aca20 2021-04-09T18:32:47.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34dafda700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40905f0 con 0x7f34d418bd40 2021-04-09T18:32:47.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d95f7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d408fba0 0x7f34d408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d95f7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d408fba0 0x7f34d408ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58106/0 (socket says 172.21.15.32:58106) 2021-04-09T18:32:47.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d95f7700 1 -- 172.21.15.32:0/3129082247 learned_addr learned my addr 172.21.15.32:0/3129082247 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:47.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34cbfff700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34d418bd40 0x7f34d418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d8df6700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d41aca20 0x7f34d41ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d95f7700 1 -- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34d418bd40 msgr2=0x7f34d418c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d95f7700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34d418bd40 0x7f34d418c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d95f7700 1 -- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d41aca20 msgr2=0x7f34d41ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:47.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d95f7700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d41aca20 0x7f34d41ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:47.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d95f7700 1 -- 172.21.15.32:0/3129082247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d40b4300 con 0x7f34d408fba0 2021-04-09T18:32:47.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34d95f7700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d408fba0 0x7f34d408ffa0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 rx=0x7f34cc00b910 tx=0x7f34cc00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:47.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34c97fa700 1 -- 172.21.15.32:0/3129082247 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34cc006aa0 con 0x7f34d408fba0 2021-04-09T18:32:47.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34c97fa700 1 -- 172.21.15.32:0/3129082247 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc0011f0 con 0x7f34d408fba0 2021-04-09T18:32:47.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34c97fa700 1 -- 172.21.15.32:0/3129082247 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34cc001640 con 0x7f34d408fba0 2021-04-09T18:32:47.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34dafda700 1 -- 172.21.15.32:0/3129082247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d40b4300 con 0x7f34d408fba0 2021-04-09T18:32:47.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.939 7f34dafda700 1 -- 172.21.15.32:0/3129082247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d40b4300 con 0x7f34d408fba0 2021-04-09T18:32:47.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.940 7f34c97fa700 1 -- 172.21.15.32:0/3129082247 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f34cc0013d0 con 0x7f34d408fba0 2021-04-09T18:32:47.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.940 7f34c97fa700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34b4018930 0x7f34b401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:47.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.940 7f34c97fa700 1 -- 172.21.15.32:0/3129082247 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f34cc033100 con 0x7f34d408fba0 2021-04-09T18:32:47.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.940 7f34d8df6700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34b4018930 0x7f34b401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:47.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.941 7f34d8df6700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34b4018930 0x7f34b401adc0 crc :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:47.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.960 7f34dafda700 1 -- 172.21.15.32:0/3129082247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d4182d70 con 0x7f34d408fba0 2021-04-09T18:32:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:47.963 7f34c97fa700 1 -- 172.21.15.32:0/3129082247 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f34cc00bdf0 con 0x7f34d408fba0 2021-04-09T18:32:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.156 7f34dafda700 1 -- 172.21.15.32:0/3129082247 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34d418c760 con 0x7f34d408fba0 2021-04-09T18:32:48.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.156 7f34c97fa700 1 -- 172.21.15.32:0/3129082247 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 385 v385) v1 ==== 95+0+3719 (secure 0 0 0) 0x7f34cc0011f0 con 0x7f34d408fba0 2021-04-09T18:32:48.174 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 385 2021-04-09T18:32:48.174 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:48.174 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":385,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11512,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3237972186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3237972186},{"type":"v1","addr":"172.21.15.32:6834","nonce":3237972186}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378},{"gid":11515,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4276774592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4276774592},{"type":"v1","addr":"172.21.15.32:6835","nonce":4276774592}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378}],"filesystems":[{"mdsmap":{"epoch":381,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:39.408909","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11500":{"gid":11500,"name":"b","rank":0,"incarnation":379,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3107210651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3107210651},{"type":"v1","addr":"172.21.15.86:6836","nonce":3107210651}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":385,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:46.863593","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11503":{"gid":11503,"name":"d","rank":0,"incarnation":382,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/2051228504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2051228504},{"type":"v1","addr":"172.21.15.86:6837","nonce":2051228504}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:48.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.158 7f34dafda700 1 -- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34b4018930 msgr2=0x7f34b401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:48.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.158 7f34dafda700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34b4018930 0x7f34b401adc0 crc :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:48.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.158 7f34dafda700 1 -- 172.21.15.32:0/3129082247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d408fba0 msgr2=0x7f34d408ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:48.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.158 7f34dafda700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d408fba0 0x7f34d408ffa0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 rx=0x7f34cc00b910 tx=0x7f34cc00bcb0).stop 2021-04-09T18:32:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.159 7f34dafda700 1 -- 172.21.15.32:0/3129082247 shutdown_connections 2021-04-09T18:32:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.159 7f34dafda700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f34b4018930 0x7f34b401adc0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.159 7f34dafda700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34d408fba0 0x7f34d408ffa0 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.159 7f34dafda700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34d418bd40 0x7f34d418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.159 7f34dafda700 1 --2- 172.21.15.32:0/3129082247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d41aca20 0x7f34d41ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.159 7f34dafda700 1 -- 172.21.15.32:0/3129082247 >> 172.21.15.32:0/3129082247 conn(0x7f34d419b460 msgr2=0x7f34d419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.159 7f34dafda700 1 -- 172.21.15.32:0/3129082247 shutdown_connections 2021-04-09T18:32:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:48.159 7f34dafda700 1 -- 172.21.15.32:0/3129082247 wait complete. 2021-04-09T18:32:48.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 381, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:32:37.406526', 'modified': '2021-04-09 18:32:39.408909', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11500': {'gid': 11500, 'name': 'b', 'rank': 0, 'incarnation': 379, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3107210651', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3107210651}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3107210651}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:32:48.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:32:48.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:32:48.721 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:32:48.721 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:32:48.721 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": -1, 2021-04-09T18:32:48.721 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 11728, 2021-04-09T18:32:48.722 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:boot", 2021-04-09T18:32:48.722 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "null", 2021-04-09T18:32:48.722 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 385, 2021-04-09T18:32:48.722 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 0, 2021-04-09T18:32:48.723 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 0, 2021-04-09T18:32:48.723 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 0.79661650900000003 2021-04-09T18:32:48.723 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:32:48.726 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": -1, "id": 11728, "want_state": "up:boot", "state": "null", "mdsmap_epoch": 385, "osdmap_epoch": 0, "osdmap_epoch_barrier": 0, "uptime": 0.79661650900000003 } 2021-04-09T18:32:49.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.872 7f46019c1700 1 Processor -- start 2021-04-09T18:32:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.872 7f46019c1700 1 -- start start 2021-04-09T18:32:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.872 7f46019c1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f45fc10da00 0x7f45fc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.872 7f46019c1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc10d480 con 0x7f45fc0d7ca0 2021-04-09T18:32:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.872 7f46019c1700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc0d1f70 con 0x7f45fc10d010 2021-04-09T18:32:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.872 7f46019c1700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45fc0d8d30 con 0x7f45fc10da00 2021-04-09T18:32:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fbd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f45fc10da00 0x7f45fc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fad73700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f45fc0d7ca0 0x7f45fc0d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39472/0 (socket says 172.21.15.32:39472) 2021-04-09T18:32:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fbd75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f45fc10da00 0x7f45fc10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58116/0 (socket says 172.21.15.32:58116) 2021-04-09T18:32:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fad73700 1 -- 172.21.15.32:0/130216050 learned_addr learned my addr 172.21.15.32:0/130216050 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fbd75700 1 -- 172.21.15.32:0/130216050 >> v1:172.21.15.86:6789/0 conn(0x7f45fc10d010 legacy=0x7f45fc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fbd75700 1 -- 172.21.15.32:0/130216050 >> v1:172.21.15.32:6789/0 conn(0x7f45fc0d7ca0 legacy=0x7f45fc0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fa572700 1 -- 172.21.15.32:0/130216050 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333789298 0 0) 0x7f45ec000c60 con 0x7f45fc10d010 2021-04-09T18:32:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fbd75700 1 -- 172.21.15.32:0/130216050 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc0d9220 con 0x7f45fc10da00 2021-04-09T18:32:49.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fbd75700 1 --2- 172.21.15.32:0/130216050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc10da00 0x7f45fc10de00 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 rx=0x7f45f4006e30 tx=0x7f45f4009de0).ready entity=mon.0 client_cookie=5ed6e6fae2529d6f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:49.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fa572700 1 -- 172.21.15.32:0/130216050 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45f400b300 con 0x7f45fc10da00 2021-04-09T18:32:49.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fa572700 1 -- 172.21.15.32:0/130216050 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f400b680 con 0x7f45fc10da00 2021-04-09T18:32:49.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f45fa572700 1 -- 172.21.15.32:0/130216050 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45f400bb60 con 0x7f45fc10da00 2021-04-09T18:32:49.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f46019c1700 1 -- 172.21.15.32:0/130216050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc10da00 msgr2=0x7f45fc10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:49.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.873 7f46019c1700 1 --2- 172.21.15.32:0/130216050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc10da00 0x7f45fc10de00 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 rx=0x7f45f4006e30 tx=0x7f45f4009de0).stop 2021-04-09T18:32:49.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.874 7f46019c1700 1 -- 172.21.15.32:0/130216050 shutdown_connections 2021-04-09T18:32:49.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.874 7f46019c1700 1 --2- 172.21.15.32:0/130216050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc10da00 0x7f45fc10de00 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:49.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.874 7f46019c1700 1 -- 172.21.15.32:0/130216050 >> 172.21.15.32:0/130216050 conn(0x7f45fc0ce5f0 msgr2=0x7f45fc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:49.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.874 7f46019c1700 1 -- 172.21.15.32:0/130216050 shutdown_connections 2021-04-09T18:32:49.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.874 7f46019c1700 1 -- 172.21.15.32:0/130216050 wait complete. 2021-04-09T18:32:49.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.894 7f46019c1700 1 Processor -- start 2021-04-09T18:32:49.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.894 7f46019c1700 1 -- start start 2021-04-09T18:32:49.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f46019c1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45fc0af100 0x7f45fc090b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:49.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f46019c1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc1868f0 0x7f45fc185ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:49.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f46019c1700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45fc1a9470 0x7f45fc1a9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:49.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f46019c1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc1860d0 con 0x7f45fc1868f0 2021-04-09T18:32:49.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f46019c1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc186cf0 con 0x7f45fc0af100 2021-04-09T18:32:49.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f46019c1700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc195280 con 0x7f45fc1a9470 2021-04-09T18:32:49.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc1868f0 0x7f45fc185ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:49.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc1868f0 0x7f45fc185ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58118/0 (socket says 172.21.15.32:58118) 2021-04-09T18:32:49.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fad73700 1 -- 172.21.15.32:0/1845034922 learned_addr learned my addr 172.21.15.32:0/1845034922 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:49.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fb574700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45fc0af100 0x7f45fc090b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:49.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fbd75700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45fc1a9470 0x7f45fc1a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:49.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fad73700 1 -- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45fc1a9470 msgr2=0x7f45fc1a9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:49.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fad73700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45fc1a9470 0x7f45fc1a9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:49.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fad73700 1 -- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45fc0af100 msgr2=0x7f45fc090b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:49.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fad73700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45fc0af100 0x7f45fc090b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:49.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fad73700 1 -- 172.21.15.32:0/1845034922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc09d9d0 con 0x7f45fc1868f0 2021-04-09T18:32:49.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fb574700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45fc0af100 0x7f45fc090b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:32:49.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.895 7f45fad73700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc1868f0 0x7f45fc185ab0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 rx=0x7f45f000a5a0 tx=0x7f45f000a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:49.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.896 7f45e3fff700 1 -- 172.21.15.32:0/1845034922 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45f000ad30 con 0x7f45fc1868f0 2021-04-09T18:32:49.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.896 7f45e3fff700 1 -- 172.21.15.32:0/1845034922 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f0007030 con 0x7f45fc1868f0 2021-04-09T18:32:49.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.896 7f46019c1700 1 -- 172.21.15.32:0/1845034922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc09d9d0 con 0x7f45fc1868f0 2021-04-09T18:32:49.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.896 7f45e3fff700 1 -- 172.21.15.32:0/1845034922 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45f0004620 con 0x7f45fc1868f0 2021-04-09T18:32:49.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.896 7f46019c1700 1 -- 172.21.15.32:0/1845034922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc09d9d0 con 0x7f45fc1868f0 2021-04-09T18:32:49.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.897 7f45e3fff700 1 -- 172.21.15.32:0/1845034922 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f45f00043a0 con 0x7f45fc1868f0 2021-04-09T18:32:49.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.897 7f45e3fff700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45dc0188c0 0x7f45dc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:49.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.897 7f45e3fff700 1 -- 172.21.15.32:0/1845034922 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f45f0032a60 con 0x7f45fc1868f0 2021-04-09T18:32:49.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.897 7f45fb574700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45dc0188c0 0x7f45dc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:49.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.898 7f45fb574700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45dc0188c0 0x7f45dc01adc0 crc :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:49.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.916 7f46019c1700 1 -- 172.21.15.32:0/1845034922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc10d480 con 0x7f45fc1868f0 2021-04-09T18:32:49.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:49.920 7f45e3fff700 1 -- 172.21.15.32:0/1845034922 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f45f0000c90 con 0x7f45fc1868f0 2021-04-09T18:32:50.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.111 7f46019c1700 1 -- 172.21.15.32:0/1845034922 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45fc10d480 con 0x7f45fc1868f0 2021-04-09T18:32:50.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.111 7f45e3fff700 1 -- 172.21.15.32:0/1845034922 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 385 v385) v1 ==== 95+0+3719 (secure 0 0 0) 0x7f45f00043a0 con 0x7f45fc1868f0 2021-04-09T18:32:50.129 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 385 2021-04-09T18:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:50.129 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":385,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11512,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3237972186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3237972186},{"type":"v1","addr":"172.21.15.32:6834","nonce":3237972186}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378},{"gid":11515,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4276774592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4276774592},{"type":"v1","addr":"172.21.15.32:6835","nonce":4276774592}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378}],"filesystems":[{"mdsmap":{"epoch":381,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:39.408909","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11500":{"gid":11500,"name":"b","rank":0,"incarnation":379,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3107210651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3107210651},{"type":"v1","addr":"172.21.15.86:6836","nonce":3107210651}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":385,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:46.863593","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11503":{"gid":11503,"name":"d","rank":0,"incarnation":382,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/2051228504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2051228504},{"type":"v1","addr":"172.21.15.86:6837","nonce":2051228504}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.113 7f46019c1700 1 -- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45dc0188c0 msgr2=0x7f45dc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45dc0188c0 0x7f45dc01adc0 crc :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 -- 172.21.15.32:0/1845034922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc1868f0 msgr2=0x7f45fc185ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:50.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc1868f0 0x7f45fc185ab0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 rx=0x7f45f000a5a0 tx=0x7f45f000a940).stop 2021-04-09T18:32:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 -- 172.21.15.32:0/1845034922 shutdown_connections 2021-04-09T18:32:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f45dc0188c0 0x7f45dc01adc0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45fc1868f0 0x7f45fc185ab0 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:50.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45fc1a9470 0x7f45fc1a9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:50.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 --2- 172.21.15.32:0/1845034922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45fc0af100 0x7f45fc090b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:50.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 -- 172.21.15.32:0/1845034922 >> 172.21.15.32:0/1845034922 conn(0x7f45fc1965c0 msgr2=0x7f45fc1ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:50.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 -- 172.21.15.32:0/1845034922 shutdown_connections 2021-04-09T18:32:50.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:50.114 7f46019c1700 1 -- 172.21.15.32:0/1845034922 wait complete. 2021-04-09T18:32:50.543 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 381, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:32:37.406526', 'modified': '2021-04-09 18:32:39.408909', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11500': {'gid': 11500, 'name': 'b', 'rank': 0, 'incarnation': 379, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3107210651', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3107210651}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3107210651}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:32:50.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:32:50.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:32:50.671 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:32:50.672 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:32:50.672 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": -1, 2021-04-09T18:32:50.672 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 11728, 2021-04-09T18:32:50.672 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:boot", 2021-04-09T18:32:50.672 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "null", 2021-04-09T18:32:50.673 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 385, 2021-04-09T18:32:50.673 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 0, 2021-04-09T18:32:50.673 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 0, 2021-04-09T18:32:50.673 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 2.7471537779999999 2021-04-09T18:32:50.674 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:32:50.677 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": -1, "id": 11728, "want_state": "up:boot", "state": "null", "mdsmap_epoch": 385, "osdmap_epoch": 0, "osdmap_epoch_barrier": 0, "uptime": 2.7471537779999999 } 2021-04-09T18:32:51.678 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:51.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.824 7f1a043ad700 1 Processor -- start 2021-04-09T18:32:51.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.824 7f1a043ad700 1 -- start start 2021-04-09T18:32:51.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a043ad700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19fc0d7ca0 0x7f19fc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:51.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a043ad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc10da00 0x7f19fc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a043ad700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19fc10d480 con 0x7f19fc10d010 2021-04-09T18:32:51.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a043ad700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19fc0d9220 con 0x7f19fc0d7ca0 2021-04-09T18:32:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a043ad700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc112100 con 0x7f19fc10da00 2021-04-09T18:32:51.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a019c8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19fc0d7ca0 0x7f19fc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:51.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a029ca700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc10da00 0x7f19fc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:51.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a021c9700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f19fc10d010 0x7f19fc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37410/0 (socket says 172.21.15.32:37410) 2021-04-09T18:32:51.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a021c9700 1 -- 172.21.15.32:0/4244153222 learned_addr learned my addr 172.21.15.32:0/4244153222 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:51.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a029ca700 1 -- 172.21.15.32:0/4244153222 >> v2:172.21.15.86:3300/0 conn(0x7f19fc0d7ca0 msgr2=0x7f19fc0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:51.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a029ca700 1 --2- 172.21.15.32:0/4244153222 >> v2:172.21.15.86:3300/0 conn(0x7f19fc0d7ca0 0x7f19fc0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:51.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a029ca700 1 -- 172.21.15.32:0/4244153222 >> v1:172.21.15.86:6789/0 conn(0x7f19fc10d010 legacy=0x7f19fc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:51.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a029ca700 1 -- 172.21.15.32:0/4244153222 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc0d86c0 con 0x7f19fc10da00 2021-04-09T18:32:51.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a011c7700 1 -- 172.21.15.32:0/4244153222 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910667334 0 0) 0x7f19f4001400 con 0x7f19fc10d010 2021-04-09T18:32:51.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a019c8700 1 --2- 172.21.15.32:0/4244153222 >> v2:172.21.15.86:3300/0 conn(0x7f19fc0d7ca0 0x7f19fc0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:51.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.825 7f1a029ca700 1 --2- 172.21.15.32:0/4244153222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc10da00 0x7f19fc10de00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f19ec006ad0 tx=0x7f19ec009ad0).ready entity=mon.2 client_cookie=c609f00e54b9e9ac server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:51.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.826 7f1a011c7700 1 -- 172.21.15.32:0/4244153222 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ec0022e0 con 0x7f19fc10da00 2021-04-09T18:32:51.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.826 7f1a011c7700 1 -- 172.21.15.32:0/4244153222 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec006410 con 0x7f19fc10da00 2021-04-09T18:32:51.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.826 7f1a011c7700 1 -- 172.21.15.32:0/4244153222 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ec002620 con 0x7f19fc10da00 2021-04-09T18:32:51.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.826 7f1a043ad700 1 -- 172.21.15.32:0/4244153222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc10da00 msgr2=0x7f19fc10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:51.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.826 7f1a043ad700 1 --2- 172.21.15.32:0/4244153222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc10da00 0x7f19fc10de00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f19ec006ad0 tx=0x7f19ec009ad0).stop 2021-04-09T18:32:51.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.826 7f1a043ad700 1 -- 172.21.15.32:0/4244153222 shutdown_connections 2021-04-09T18:32:51.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.826 7f1a043ad700 1 --2- 172.21.15.32:0/4244153222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc10da00 0x7f19fc10de00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:51.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.826 7f1a043ad700 1 --2- 172.21.15.32:0/4244153222 >> v2:172.21.15.86:3300/0 conn(0x7f19fc0d7ca0 0x7f19fc0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:51.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.826 7f1a043ad700 1 -- 172.21.15.32:0/4244153222 >> 172.21.15.32:0/4244153222 conn(0x7f19fc0ce5f0 msgr2=0x7f19fc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:51.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.827 7f1a043ad700 1 -- 172.21.15.32:0/4244153222 shutdown_connections 2021-04-09T18:32:51.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.827 7f1a043ad700 1 -- 172.21.15.32:0/4244153222 wait complete. 2021-04-09T18:32:51.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a043ad700 1 Processor -- start 2021-04-09T18:32:51.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a043ad700 1 -- start start 2021-04-09T18:32:51.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a043ad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc09a670 0x7f19fc093610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:51.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a043ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc1a9fb0 0x7f19fc1aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:51.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a043ad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0a3ab0 0x7f19fc0a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:51.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a043ad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc094060 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a043ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc094710 con 0x7f19fc1a9fb0 2021-04-09T18:32:51.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a043ad700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc093c30 con 0x7f19fc09a670 2021-04-09T18:32:51.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a029ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0a3ab0 0x7f19fc0a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:51.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a029ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0a3ab0 0x7f19fc0a3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58134/0 (socket says 172.21.15.32:58134) 2021-04-09T18:32:51.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a029ca700 1 -- 172.21.15.32:0/3719392491 learned_addr learned my addr 172.21.15.32:0/3719392491 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a021c9700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc09a670 0x7f19fc093610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.847 7f1a019c8700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc1a9fb0 0x7f19fc1aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:51.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f1a029ca700 1 -- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc09a670 msgr2=0x7f19fc093610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f1a029ca700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc09a670 0x7f19fc093610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:51.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f1a029ca700 1 -- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc1a9fb0 msgr2=0x7f19fc1aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:51.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f1a029ca700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc1a9fb0 0x7f19fc1aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:51.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f1a029ca700 1 -- 172.21.15.32:0/3719392491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc1aa9d0 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f1a029ca700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0a3ab0 0x7f19fc0a3eb0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 rx=0x7f19ec009670 tx=0x7f19ec00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f19faffd700 1 -- 172.21.15.32:0/3719392491 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ec00bd80 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f19faffd700 1 -- 172.21.15.32:0/3719392491 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec000a70 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f19faffd700 1 -- 172.21.15.32:0/3719392491 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ec002270 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19fc1aa9d0 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.848 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19fc1aa9d0 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.849 7f19faffd700 1 -- 172.21.15.32:0/3719392491 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f19ec002080 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.849 7f19faffd700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19dc0188c0 0x7f19dc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:51.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.849 7f19faffd700 1 -- 172.21.15.32:0/3719392491 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f19ec032b10 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.849 7f1a021c9700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19dc0188c0 0x7f19dc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:51.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.849 7f1a021c9700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19dc0188c0 0x7f19dc01adc0 crc :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:51.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.869 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc10d480 con 0x7f19fc0a3ab0 2021-04-09T18:32:51.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:51.872 7f19faffd700 1 -- 172.21.15.32:0/3719392491 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f19ec030490 con 0x7f19fc0a3ab0 2021-04-09T18:32:52.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.067 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19fc0946d0 con 0x7f19fc0a3ab0 2021-04-09T18:32:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.068 7f19faffd700 1 -- 172.21.15.32:0/3719392491 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 385 v385) v1 ==== 95+0+3719 (secure 0 0 0) 0x7f19ec030490 con 0x7f19fc0a3ab0 2021-04-09T18:32:52.085 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 385 2021-04-09T18:32:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:52.086 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":385,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11512,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3237972186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3237972186},{"type":"v1","addr":"172.21.15.32:6834","nonce":3237972186}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378},{"gid":11515,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/4276774592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":4276774592},{"type":"v1","addr":"172.21.15.32:6835","nonce":4276774592}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":378}],"filesystems":[{"mdsmap":{"epoch":381,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:39.408909","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11500":{"gid":11500,"name":"b","rank":0,"incarnation":379,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3107210651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3107210651},{"type":"v1","addr":"172.21.15.86:6836","nonce":3107210651}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":385,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:46.863593","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11503":{"gid":11503,"name":"d","rank":0,"incarnation":382,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/2051228504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2051228504},{"type":"v1","addr":"172.21.15.86:6837","nonce":2051228504}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:52.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.070 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19dc0188c0 msgr2=0x7f19dc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:52.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.070 7f1a043ad700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19dc0188c0 0x7f19dc01adc0 crc :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:52.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.070 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0a3ab0 msgr2=0x7f19fc0a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.070 7f1a043ad700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0a3ab0 0x7f19fc0a3eb0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 rx=0x7f19ec009670 tx=0x7f19ec00b990).stop 2021-04-09T18:32:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.070 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 shutdown_connections 2021-04-09T18:32:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.071 7f1a043ad700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f19dc0188c0 0x7f19dc01adc0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.071 7f1a043ad700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f19fc0a3ab0 0x7f19fc0a3eb0 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:52.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.071 7f1a043ad700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19fc1a9fb0 0x7f19fc1aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.071 7f1a043ad700 1 --2- 172.21.15.32:0/3719392491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19fc09a670 0x7f19fc093610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.071 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 >> 172.21.15.32:0/3719392491 conn(0x7f19fc1a0db0 msgr2=0x7f19fc1a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.071 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 shutdown_connections 2021-04-09T18:32:52.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:52.071 7f1a043ad700 1 -- 172.21.15.32:0/3719392491 wait complete. 2021-04-09T18:32:52.182 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:32:52.165 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 2 filesystems are offline (MDS_ALL_DOWN) 2021-04-09T18:32:52.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 381, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:32:37.406526', 'modified': '2021-04-09 18:32:39.408909', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11500': {'gid': 11500, 'name': 'b', 'rank': 0, 'incarnation': 379, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3107210651', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3107210651}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3107210651}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:32:52.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:32:52.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:32:52.627 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:32:52.628 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:32:52.628 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:32:52.628 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 11728, 2021-04-09T18:32:52.629 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:reconnect", 2021-04-09T18:32:52.629 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:replay", 2021-04-09T18:32:52.629 INFO:teuthology.orchestra.run.smithi086.stdout: "replay_status": { 2021-04-09T18:32:52.629 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_read_pos": 4195196, 2021-04-09T18:32:52.629 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_write_pos": 4195196, 2021-04-09T18:32:52.630 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_expire_pos": 4194304, 2021-04-09T18:32:52.630 INFO:teuthology.orchestra.run.smithi086.stdout: "num_events": 1, 2021-04-09T18:32:52.630 INFO:teuthology.orchestra.run.smithi086.stdout: "num_segments": 1 2021-04-09T18:32:52.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2021-04-09T18:32:52.631 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 0.43819517400000002, 2021-04-09T18:32:52.631 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 387, 2021-04-09T18:32:52.631 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 234, 2021-04-09T18:32:52.631 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 0, 2021-04-09T18:32:52.631 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 4.7034451639999997 2021-04-09T18:32:52.632 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:32:52.633 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 11728, "want_state": "up:reconnect", "state": "up:replay", "replay_status": { "journal_read_pos": 4195196, "journal_write_pos": 4195196, "journal_expire_pos": 4194304, "num_events": 1, "num_segments": 1 }, "rank_uptime": 0.43819517400000002, "mdsmap_epoch": 387, "osdmap_epoch": 234, "osdmap_epoch_barrier": 0, "uptime": 4.7034451639999997 } 2021-04-09T18:32:53.635 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:53.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.771 7f280e86d700 1 Processor -- start 2021-04-09T18:32:53.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.771 7f280e86d700 1 -- start start 2021-04-09T18:32:53.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.772 7f280e86d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28080d7ca0 0x7f28080d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:53.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.772 7f280e86d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280810d480 con 0x7f280810da00 2021-04-09T18:32:53.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.772 7f280e86d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28080d86c0 con 0x7f280810d010 2021-04-09T18:32:53.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.772 7f280e86d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28080d9220 con 0x7f28080d7ca0 2021-04-09T18:32:53.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.772 7f27fffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f28080d7ca0 0x7f28080d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:53.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.772 7f280ce8a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f280810da00 0x7f280810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39504/0 (socket says 172.21.15.32:39504) 2021-04-09T18:32:53.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.772 7f280ce8a700 1 -- 172.21.15.32:0/2469505958 learned_addr learned my addr 172.21.15.32:0/2469505958 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:53.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f27fffff700 1 -- 172.21.15.32:0/2469505958 >> v1:172.21.15.86:6789/0 conn(0x7f280810d010 legacy=0x7f28080d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:53.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f27fffff700 1 -- 172.21.15.32:0/2469505958 >> v1:172.21.15.32:6789/0 conn(0x7f280810da00 legacy=0x7f280810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:53.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f28077fe700 1 -- 172.21.15.32:0/2469505958 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262935194 0 0) 0x7f27f8000c60 con 0x7f280810d010 2021-04-09T18:32:53.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f27fffff700 1 -- 172.21.15.32:0/2469505958 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2808112100 con 0x7f28080d7ca0 2021-04-09T18:32:53.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f27fffff700 1 --2- 172.21.15.32:0/2469505958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d7ca0 0x7f28080d80a0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7f27f0006650 tx=0x7f27f0009a80).ready entity=mon.0 client_cookie=b0f93b80b180b75d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:53.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f28077fe700 1 -- 172.21.15.32:0/2469505958 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27f00009b0 con 0x7f28080d7ca0 2021-04-09T18:32:53.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f28077fe700 1 -- 172.21.15.32:0/2469505958 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f000c0c0 con 0x7f28080d7ca0 2021-04-09T18:32:53.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f28077fe700 1 -- 172.21.15.32:0/2469505958 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27f000c510 con 0x7f28080d7ca0 2021-04-09T18:32:53.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f280e86d700 1 -- 172.21.15.32:0/2469505958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d7ca0 msgr2=0x7f28080d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:53.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.773 7f280e86d700 1 --2- 172.21.15.32:0/2469505958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d7ca0 0x7f28080d80a0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7f27f0006650 tx=0x7f27f0009a80).stop 2021-04-09T18:32:53.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.774 7f280e86d700 1 -- 172.21.15.32:0/2469505958 shutdown_connections 2021-04-09T18:32:53.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.774 7f280e86d700 1 --2- 172.21.15.32:0/2469505958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28080d7ca0 0x7f28080d80a0 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:53.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.774 7f280e86d700 1 -- 172.21.15.32:0/2469505958 >> 172.21.15.32:0/2469505958 conn(0x7f28080ce5f0 msgr2=0x7f28080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:53.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.774 7f280e86d700 1 -- 172.21.15.32:0/2469505958 shutdown_connections 2021-04-09T18:32:53.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.774 7f280e86d700 1 -- 172.21.15.32:0/2469505958 wait complete. 2021-04-09T18:32:53.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.794 7f280e86d700 1 Processor -- start 2021-04-09T18:32:53.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.794 7f280e86d700 1 -- start start 2021-04-09T18:32:53.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.794 7f280e86d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28081a9a50 0x7f280808a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:53.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.794 7f280e86d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f280808afb0 0x7f280809d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:53.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f280e86d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f280809ddf0 0x7f2808183b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:53.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f280e86d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28081a9e50 con 0x7f280808afb0 2021-04-09T18:32:53.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f280e86d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280809e410 con 0x7f280809ddf0 2021-04-09T18:32:53.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f280e86d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28081841a0 con 0x7f28081a9a50 2021-04-09T18:32:53.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f27fffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f280808afb0 0x7f280809d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:53.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f27fffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f280808afb0 0x7f280809d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58150/0 (socket says 172.21.15.32:58150) 2021-04-09T18:32:53.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f27fffff700 1 -- 172.21.15.32:0/1916335052 learned_addr learned my addr 172.21.15.32:0/1916335052 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:53.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f280ce8a700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f280809ddf0 0x7f2808183b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:53.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f2807fff700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28081a9a50 0x7f280808a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:53.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f27fffff700 1 -- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28081a9a50 msgr2=0x7f280808a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f27fffff700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28081a9a50 0x7f280808a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f27fffff700 1 -- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f280809ddf0 msgr2=0x7f2808183b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f27fffff700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f280809ddf0 0x7f2808183b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f27fffff700 1 -- 172.21.15.32:0/1916335052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2808184380 con 0x7f280808afb0 2021-04-09T18:32:53.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f27fffff700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f280808afb0 0x7f280809d7d0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 rx=0x7f27f0009650 tx=0x7f27f000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:53.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f28057fa700 1 -- 172.21.15.32:0/1916335052 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27f00093a0 con 0x7f280808afb0 2021-04-09T18:32:53.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f28057fa700 1 -- 172.21.15.32:0/1916335052 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f000bd50 con 0x7f280808afb0 2021-04-09T18:32:53.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f28057fa700 1 -- 172.21.15.32:0/1916335052 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27f00052e0 con 0x7f280808afb0 2021-04-09T18:32:53.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f280e86d700 1 -- 172.21.15.32:0/1916335052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2808184380 con 0x7f280808afb0 2021-04-09T18:32:53.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.795 7f280e86d700 1 -- 172.21.15.32:0/1916335052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2808184380 con 0x7f280808afb0 2021-04-09T18:32:53.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.796 7f28057fa700 1 -- 172.21.15.32:0/1916335052 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f27f0006030 con 0x7f280808afb0 2021-04-09T18:32:53.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.796 7f28057fa700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27ec0188c0 0x7f27ec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:53.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.796 7f28057fa700 1 -- 172.21.15.32:0/1916335052 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f27f0032ba0 con 0x7f280808afb0 2021-04-09T18:32:53.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.796 7f2807fff700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27ec0188c0 0x7f27ec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:53.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.797 7f2807fff700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27ec0188c0 0x7f27ec01adc0 crc :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.816 7f280e86d700 1 -- 172.21.15.32:0/1916335052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280810d480 con 0x7f280808afb0 2021-04-09T18:32:53.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:53.819 7f28057fa700 1 -- 172.21.15.32:0/1916335052 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f27f00009c0 con 0x7f280808afb0 2021-04-09T18:32:54.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.012 7f280e86d700 1 -- 172.21.15.32:0/1916335052 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f280810d480 con 0x7f280808afb0 2021-04-09T18:32:54.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.013 7f28057fa700 1 -- 172.21.15.32:0/1916335052 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 388 v388) v1 ==== 95+0+3726 (secure 0 0 0) 0x7f27f0006030 con 0x7f280808afb0 2021-04-09T18:32:54.031 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 388 2021-04-09T18:32:54.032 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:54.032 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":388,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386},{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:53.172981","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":232,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11728":{"gid":11728,"name":"b","rank":0,"incarnation":387,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.86:6836/2382296066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2382296066},{"type":"v1","addr":"172.21.15.86:6836","nonce":2382296066}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:53.172981","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:54.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 -- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27ec0188c0 msgr2=0x7f27ec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:54.035 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27ec0188c0 0x7f27ec01adc0 crc :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:54.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 -- 172.21.15.32:0/1916335052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f280808afb0 msgr2=0x7f280809d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:54.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f280808afb0 0x7f280809d7d0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 rx=0x7f27f0009650 tx=0x7f27f000b960).stop 2021-04-09T18:32:54.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 -- 172.21.15.32:0/1916335052 shutdown_connections 2021-04-09T18:32:54.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27ec0188c0 0x7f27ec01adc0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:54.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f280808afb0 0x7f280809d7d0 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:54.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f280809ddf0 0x7f2808183b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:54.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 --2- 172.21.15.32:0/1916335052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28081a9a50 0x7f280808a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:54.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 -- 172.21.15.32:0/1916335052 >> 172.21.15.32:0/1916335052 conn(0x7f28081965c0 msgr2=0x7f28081ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:54.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 -- 172.21.15.32:0/1916335052 shutdown_connections 2021-04-09T18:32:54.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:54.015 7f280e86d700 1 -- 172.21.15.32:0/1916335052 wait complete. 2021-04-09T18:32:54.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 388, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:32:37.406526', 'modified': '2021-04-09 18:32:53.172981', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 232, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11728}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11728': {'gid': 11728, 'name': 'b', 'rank': 0, 'incarnation': 387, 'state': 'up:reconnect', 'state_seq': 3, 'addr': '172.21.15.86:6836/2382296066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2382296066}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 2382296066}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:32:54.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_11728:up:reconnect 2021-04-09T18:32:55.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:55.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.583 7f0166cc2700 1 Processor -- start 2021-04-09T18:32:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.583 7f0166cc2700 1 -- start start 2021-04-09T18:32:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0166cc2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f01600cffa0 0x7f01600d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0166cc2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01600d7800 0x7f01600d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:55.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0166cc2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01600d7280 con 0x7f01600d8220 2021-04-09T18:32:55.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0166cc2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01600d91e0 con 0x7f01600cffa0 2021-04-09T18:32:55.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0166cc2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016010dfe0 con 0x7f01600d7800 2021-04-09T18:32:55.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0164ade700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f01600cffa0 0x7f01600d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0164ade700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f01600cffa0 0x7f01600d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58158/0 (socket says 172.21.15.32:58158) 2021-04-09T18:32:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0164ade700 1 -- 172.21.15.32:0/3977409135 learned_addr learned my addr 172.21.15.32:0/3977409135 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0164ade700 1 -- 172.21.15.32:0/3977409135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01600d7800 msgr2=0x7f01600d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0164ade700 1 --2- 172.21.15.32:0/3977409135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01600d7800 0x7f01600d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-04-09T18:32:55.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0164ade700 1 -- 172.21.15.32:0/3977409135 >> v1:172.21.15.86:6789/0 conn(0x7f01600d8220 legacy=0x7f01600d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:55.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0164ade700 1 -- 172.21.15.32:0/3977409135 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016010e200 con 0x7f01600cffa0 2021-04-09T18:32:55.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.584 7f0164ade700 1 --2- 172.21.15.32:0/3977409135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01600cffa0 0x7f01600d2440 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 rx=0x7f0148006700 tx=0x7f0148009b00).ready entity=mon.0 client_cookie=9f0f2025b7d52baf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:55.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f015f7fe700 1 -- 172.21.15.32:0/3977409135 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f014800c0c0 con 0x7f01600cffa0 2021-04-09T18:32:55.625 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f015f7fe700 1 -- 172.21.15.32:0/3977409135 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014800c400 con 0x7f01600cffa0 2021-04-09T18:32:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f015f7fe700 1 -- 172.21.15.32:0/3977409135 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f014800c8e0 con 0x7f01600cffa0 2021-04-09T18:32:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f0166cc2700 1 -- 172.21.15.32:0/3977409135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01600cffa0 msgr2=0x7f01600d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f0166cc2700 1 --2- 172.21.15.32:0/3977409135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01600cffa0 0x7f01600d2440 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 rx=0x7f0148006700 tx=0x7f0148009b00).stop 2021-04-09T18:32:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f0166cc2700 1 -- 172.21.15.32:0/3977409135 shutdown_connections 2021-04-09T18:32:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f0166cc2700 1 --2- 172.21.15.32:0/3977409135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01600d7800 0x7f01600d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-04-09T18:32:55.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f0166cc2700 1 --2- 172.21.15.32:0/3977409135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01600cffa0 0x7f01600d2440 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:55.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f0166cc2700 1 -- 172.21.15.32:0/3977409135 >> 172.21.15.32:0/3977409135 conn(0x7f01600ce5c0 msgr2=0x7f01600cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:55.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f0166cc2700 1 -- 172.21.15.32:0/3977409135 shutdown_connections 2021-04-09T18:32:55.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.585 7f0166cc2700 1 -- 172.21.15.32:0/3977409135 wait complete. 2021-04-09T18:32:55.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.605 7f0166cc2700 1 Processor -- start 2021-04-09T18:32:55.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f0166cc2700 1 -- start start 2021-04-09T18:32:55.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f0166cc2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f016017dcf0 0x7f016017e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:55.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f0166cc2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01601a0820 0x7f01601a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:55.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f0166cc2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016019f050 0x7f016019f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:55.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f0166cc2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01601a02a0 con 0x7f01601a0820 2021-04-09T18:32:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f0166cc2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01601a1240 con 0x7f016019f050 2021-04-09T18:32:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f0166cc2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01600a1aa0 con 0x7f016017dcf0 2021-04-09T18:32:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f015ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01601a0820 0x7f01601a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f015ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01601a0820 0x7f01601a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58162/0 (socket says 172.21.15.32:58162) 2021-04-09T18:32:55.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.606 7f015ffff700 1 -- 172.21.15.32:0/4098044841 learned_addr learned my addr 172.21.15.32:0/4098044841 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f0164ade700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f016017dcf0 0x7f016017e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f01652df700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016019f050 0x7f016019f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f015ffff700 1 -- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f016017dcf0 msgr2=0x7f016017e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:55.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f015ffff700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f016017dcf0 0x7f016017e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f015ffff700 1 -- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016019f050 msgr2=0x7f016019f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f015ffff700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016019f050 0x7f016019f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f015ffff700 1 -- 172.21.15.32:0/4098044841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016019fa70 con 0x7f01601a0820 2021-04-09T18:32:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f015ffff700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01601a0820 0x7f01601a0c20 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 rx=0x7f0150009980 tx=0x7f015000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:55.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f015d7fa700 1 -- 172.21.15.32:0/4098044841 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f015000c2f0 con 0x7f01601a0820 2021-04-09T18:32:55.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f015d7fa700 1 -- 172.21.15.32:0/4098044841 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015000c570 con 0x7f01601a0820 2021-04-09T18:32:55.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016019fa70 con 0x7f01601a0820 2021-04-09T18:32:55.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f015d7fa700 1 -- 172.21.15.32:0/4098044841 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f015000c9d0 con 0x7f01601a0820 2021-04-09T18:32:55.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.607 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016019fa70 con 0x7f01601a0820 2021-04-09T18:32:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.608 7f015d7fa700 1 -- 172.21.15.32:0/4098044841 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0150001010 con 0x7f01601a0820 2021-04-09T18:32:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.608 7f015d7fa700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0144018930 0x7f014401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.608 7f015d7fa700 1 -- 172.21.15.32:0/4098044841 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f0150033170 con 0x7f01601a0820 2021-04-09T18:32:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.608 7f0164ade700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0144018930 0x7f014401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:55.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.608 7f0164ade700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0144018930 0x7f014401ae10 crc :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:55.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.628 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016017caf0 con 0x7f01601a0820 2021-04-09T18:32:55.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.632 7f015d7fa700 1 -- 172.21.15.32:0/4098044841 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f015000be10 con 0x7f01601a0820 2021-04-09T18:32:55.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.822 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f016017caf0 con 0x7f01601a0820 2021-04-09T18:32:55.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.822 7f015d7fa700 1 -- 172.21.15.32:0/4098044841 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+3720 (secure 0 0 0) 0x7f0150001010 con 0x7f01601a0820 2021-04-09T18:32:55.839 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 390 2021-04-09T18:32:55.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:55.839 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":390,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386},{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":232,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11728":{"gid":11728,"name":"b","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2382296066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2382296066},{"type":"v1","addr":"172.21.15.86:6836","nonce":2382296066}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:55.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0144018930 msgr2=0x7f014401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:55.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0144018930 0x7f014401ae10 crc :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:55.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01601a0820 msgr2=0x7f01601a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:55.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01601a0820 0x7f01601a0c20 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 rx=0x7f0150009980 tx=0x7f015000bcd0).stop 2021-04-09T18:32:55.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 shutdown_connections 2021-04-09T18:32:55.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0144018930 0x7f014401ae10 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:55.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f01601a0820 0x7f01601a0c20 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:55.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016019f050 0x7f016019f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:55.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 --2- 172.21.15.32:0/4098044841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f016017dcf0 0x7f016017e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:55.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 >> 172.21.15.32:0/4098044841 conn(0x7f0160099f20 msgr2=0x7f0160094920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:55.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 shutdown_connections 2021-04-09T18:32:55.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:55.825 7f0166cc2700 1 -- 172.21.15.32:0/4098044841 wait complete. 2021-04-09T18:32:56.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 390, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:32:37.406526', 'modified': '2021-04-09 18:32:55.176447', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 232, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11728}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11728': {'gid': 11728, 'name': 'b', 'rank': 0, 'incarnation': 387, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6836/2382296066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2382296066}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 2382296066}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'alpha', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:32:56.254 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:32:56.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:32:56.391 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:32:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:32:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:32:56.391 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 11728, 2021-04-09T18:32:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:32:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:32:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.2013875900000004, 2021-04-09T18:32:56.392 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 390, 2021-04-09T18:32:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 234, 2021-04-09T18:32:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 234, 2021-04-09T18:32:56.393 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 8.4666333829999996 2021-04-09T18:32:56.393 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:32:56.397 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 11728, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.2013875900000004, "mdsmap_epoch": 390, "osdmap_epoch": 234, "osdmap_epoch_barrier": 234, "uptime": 8.4666333829999996 } 2021-04-09T18:32:56.397 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:56.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.536 7fac0a15a700 1 Processor -- start 2021-04-09T18:32:56.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.536 7fac0a15a700 1 -- start start 2021-04-09T18:32:56.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac0a15a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0410d010 0x7fac040d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:56.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac0a15a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac040d8490 0x7fac0410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:56.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac0a15a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0410d480 con 0x7fac040d1da0 2021-04-09T18:32:56.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac0a15a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0410e780 con 0x7fac040d8490 2021-04-09T18:32:56.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac0a15a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac040d8d30 con 0x7fac0410d010 2021-04-09T18:32:56.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac02ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fac040d1da0 0x7fac040d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39532/0 (socket says 172.21.15.32:39532) 2021-04-09T18:32:56.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac02ffd700 1 -- 172.21.15.32:0/2850249833 learned_addr learned my addr 172.21.15.32:0/2850249833 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:56.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac03fff700 1 --2- 172.21.15.32:0/2850249833 >> v2:172.21.15.86:3300/0 conn(0x7fac040d8490 0x7fac0410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:56.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac037fe700 1 --2- 172.21.15.32:0/2850249833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0410d010 0x7fac040d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.537 7fac027fc700 1 -- 172.21.15.32:0/2850249833 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160160302 0 0) 0x7fabf8000ce0 con 0x7fac040d1da0 2021-04-09T18:32:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac027fc700 1 -- 172.21.15.32:0/2850249833 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabec002fe0 con 0x7fac040d1da0 2021-04-09T18:32:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac027fc700 1 -- 172.21.15.32:0/2850249833 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3401428512 0 0) 0x7fabf80010e0 con 0x7fac040d1da0 2021-04-09T18:32:56.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac03fff700 1 -- 172.21.15.32:0/2850249833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0410d010 msgr2=0x7fac040d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:56.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac03fff700 1 --2- 172.21.15.32:0/2850249833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0410d010 0x7fac040d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac03fff700 1 -- 172.21.15.32:0/2850249833 >> v1:172.21.15.32:6789/0 conn(0x7fac040d1da0 legacy=0x7fac040d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:56.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac03fff700 1 -- 172.21.15.32:0/2850249833 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac040d9220 con 0x7fac040d8490 2021-04-09T18:32:56.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac037fe700 1 --2- 172.21.15.32:0/2850249833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0410d010 0x7fac040d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:56.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac03fff700 1 --2- 172.21.15.32:0/2850249833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac040d8490 0x7fac0410da00 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7fabfc006e50 tx=0x7fabfc009e70).ready entity=mon.1 client_cookie=b0858cc5dda6c57e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:56.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac027fc700 1 -- 172.21.15.32:0/2850249833 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabfc006a20 con 0x7fac040d8490 2021-04-09T18:32:56.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac027fc700 1 -- 172.21.15.32:0/2850249833 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc006790 con 0x7fac040d8490 2021-04-09T18:32:56.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac027fc700 1 -- 172.21.15.32:0/2850249833 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabfc00b700 con 0x7fac040d8490 2021-04-09T18:32:56.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac0a15a700 1 -- 172.21.15.32:0/2850249833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac040d8490 msgr2=0x7fac0410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:56.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.538 7fac0a15a700 1 --2- 172.21.15.32:0/2850249833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac040d8490 0x7fac0410da00 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7fabfc006e50 tx=0x7fabfc009e70).stop 2021-04-09T18:32:56.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.539 7fac0a15a700 1 -- 172.21.15.32:0/2850249833 shutdown_connections 2021-04-09T18:32:56.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.539 7fac0a15a700 1 --2- 172.21.15.32:0/2850249833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac040d8490 0x7fac0410da00 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.539 7fac0a15a700 1 --2- 172.21.15.32:0/2850249833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0410d010 0x7fac040d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.539 7fac0a15a700 1 -- 172.21.15.32:0/2850249833 >> 172.21.15.32:0/2850249833 conn(0x7fac040ce5f0 msgr2=0x7fac040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:56.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.539 7fac0a15a700 1 -- 172.21.15.32:0/2850249833 shutdown_connections 2021-04-09T18:32:56.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.539 7fac0a15a700 1 -- 172.21.15.32:0/2850249833 wait complete. 2021-04-09T18:32:56.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.559 7fac0a15a700 1 Processor -- start 2021-04-09T18:32:56.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.559 7fac0a15a700 1 -- start start 2021-04-09T18:32:56.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac0a15a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac041aca20 0x7fac041ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:56.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac0a15a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0418bd40 0x7fac0418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:56.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac0a15a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0408fba0 0x7fac0408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:56.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac0a15a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0418c7e0 con 0x7fac0418bd40 2021-04-09T18:32:56.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac0a15a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0418ca30 con 0x7fac041aca20 2021-04-09T18:32:56.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac0a15a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac040905f0 con 0x7fac0408fba0 2021-04-09T18:32:56.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac02ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0418bd40 0x7fac0418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:56.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac02ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0418bd40 0x7fac0418c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58176/0 (socket says 172.21.15.32:58176) 2021-04-09T18:32:56.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac02ffd700 1 -- 172.21.15.32:0/1813016136 learned_addr learned my addr 172.21.15.32:0/1813016136 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:56.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac03fff700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0408fba0 0x7fac0408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:56.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac037fe700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac041aca20 0x7fac041ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:56.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac02ffd700 1 -- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0408fba0 msgr2=0x7fac0408ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:56.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac02ffd700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0408fba0 0x7fac0408ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac02ffd700 1 -- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac041aca20 msgr2=0x7fac041ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac02ffd700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac041aca20 0x7fac041ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.560 7fac02ffd700 1 -- 172.21.15.32:0/1813016136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac040b4300 con 0x7fac0418bd40 2021-04-09T18:32:56.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.561 7fac02ffd700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0418bd40 0x7fac0418c140 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 rx=0x7fabf8000f90 tx=0x7fabf800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:56.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.561 7fabebfff700 1 -- 172.21.15.32:0/1813016136 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabf8001a30 con 0x7fac0418bd40 2021-04-09T18:32:56.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.561 7fabebfff700 1 -- 172.21.15.32:0/1813016136 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf80043a0 con 0x7fac0418bd40 2021-04-09T18:32:56.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.561 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac040b4300 con 0x7fac0418bd40 2021-04-09T18:32:56.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.561 7fabebfff700 1 -- 172.21.15.32:0/1813016136 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabf80047f0 con 0x7fac0418bd40 2021-04-09T18:32:56.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.561 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac040b4300 con 0x7fac0418bd40 2021-04-09T18:32:56.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.561 7fabebfff700 1 -- 172.21.15.32:0/1813016136 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fabf8004580 con 0x7fac0418bd40 2021-04-09T18:32:56.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.562 7fabebfff700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe4018930 0x7fabe401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:56.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.562 7fabebfff700 1 -- 172.21.15.32:0/1813016136 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fabf8032dc0 con 0x7fac0418bd40 2021-04-09T18:32:56.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.562 7fac037fe700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe4018930 0x7fabe401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.562 7fac037fe700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe4018930 0x7fabe401ae10 crc :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:56.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.581 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac04182d70 con 0x7fac0418bd40 2021-04-09T18:32:56.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.585 7fabebfff700 1 -- 172.21.15.32:0/1813016136 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fabf8001a10 con 0x7fac0418bd40 2021-04-09T18:32:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.775 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac0418c760 con 0x7fac0418bd40 2021-04-09T18:32:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.775 7fabebfff700 1 -- 172.21.15.32:0/1813016136 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+3720 (secure 0 0 0) 0x7fabf800ad30 con 0x7fac0418bd40 2021-04-09T18:32:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 390 2021-04-09T18:32:56.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:56.793 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":390,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386},{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":232,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11728":{"gid":11728,"name":"b","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2382296066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2382296066},{"type":"v1","addr":"172.21.15.86:6836","nonce":2382296066}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:56.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe4018930 msgr2=0x7fabe401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:56.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe4018930 0x7fabe401ae10 crc :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0418bd40 msgr2=0x7fac0418c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:56.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0418bd40 0x7fac0418c140 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 rx=0x7fabf8000f90 tx=0x7fabf800a940).stop 2021-04-09T18:32:56.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 shutdown_connections 2021-04-09T18:32:56.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fabe4018930 0x7fabe401ae10 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac0418bd40 0x7fac0418c140 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0408fba0 0x7fac0408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 --2- 172.21.15.32:0/1813016136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac041aca20 0x7fac041ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:56.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 >> 172.21.15.32:0/1813016136 conn(0x7fac0419b460 msgr2=0x7fac0419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:56.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 shutdown_connections 2021-04-09T18:32:56.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:56.778 7fac0a15a700 1 -- 172.21.15.32:0/1813016136 wait complete. 2021-04-09T18:32:57.206 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 390, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:32:44.859244', 'modified': '2021-04-09 18:32:55.176447', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 231, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11731}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11731': {'gid': 11731, 'name': 'd', 'rank': 0, 'incarnation': 387, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.86:6837/1293489350', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1293489350}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1293489350}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'bravo', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:32:57.207 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:32:57.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:32:57.336 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:32:57.337 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:32:57.337 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:32:57.337 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 11731, 2021-04-09T18:32:57.338 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:32:57.338 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:32:57.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 5.1469698670000001, 2021-04-09T18:32:57.339 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 390, 2021-04-09T18:32:57.339 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 234, 2021-04-09T18:32:57.339 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 234, 2021-04-09T18:32:57.339 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 9.4120503939999995 2021-04-09T18:32:57.340 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:32:57.342 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 11731, "want_state": "up:active", "state": "up:active", "rank_uptime": 5.1469698670000001, "mdsmap_epoch": 390, "osdmap_epoch": 234, "osdmap_epoch_barrier": 234, "uptime": 9.4120503939999995 } 2021-04-09T18:32:57.343 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=alpha_data, allow rw pool=bravo_data' 2021-04-09T18:32:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63fd740700 1 Processor -- start 2021-04-09T18:32:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63fd740700 1 -- start start 2021-04-09T18:32:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63fd740700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f63f80d7c50 0x7f63f81b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63fd740700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63f80d8050 0x7f63f810d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63fd740700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f80d8620 con 0x7f63f80cffb0 2021-04-09T18:32:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63fd740700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f80d91f0 con 0x7f63f80d7c50 2021-04-09T18:32:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63fd740700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f80d2800 con 0x7f63f80d8050 2021-04-09T18:32:57.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63f6d48700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f63f80d7c50 0x7f63f81b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63f7549700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f63f80cffb0 0x7f63f8087e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39542/0 (socket says 172.21.15.32:39542) 2021-04-09T18:32:57.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.490 7f63f7549700 1 -- 172.21.15.32:0/3978890971 learned_addr learned my addr 172.21.15.32:0/3978890971 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:57.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f7d4a700 1 --2- 172.21.15.32:0/3978890971 >> v2:172.21.15.86:3300/0 conn(0x7f63f80d8050 0x7f63f810d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6547700 1 -- 172.21.15.32:0/3978890971 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806407191 0 0) 0x7f63e8000ce0 con 0x7f63f80cffb0 2021-04-09T18:32:57.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6547700 1 -- 172.21.15.32:0/3978890971 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e0002fe0 con 0x7f63f80cffb0 2021-04-09T18:32:57.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6d48700 1 -- 172.21.15.32:0/3978890971 >> v2:172.21.15.86:3300/0 conn(0x7f63f80d8050 msgr2=0x7f63f810d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:57.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6d48700 1 --2- 172.21.15.32:0/3978890971 >> v2:172.21.15.86:3300/0 conn(0x7f63f80d8050 0x7f63f810d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6d48700 1 -- 172.21.15.32:0/3978890971 >> v1:172.21.15.32:6789/0 conn(0x7f63f80cffb0 legacy=0x7f63f8087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6d48700 1 -- 172.21.15.32:0/3978890971 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f81b0ab0 con 0x7f63f80d7c50 2021-04-09T18:32:57.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6d48700 1 --2- 172.21.15.32:0/3978890971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f80d7c50 0x7f63f81b0490 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 rx=0x7f63ec006650 tx=0x7f63ec009a80).ready entity=mon.0 client_cookie=dde0cbf92b5f16b4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:57.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6547700 1 -- 172.21.15.32:0/3978890971 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63ec0009b0 con 0x7f63f80d7c50 2021-04-09T18:32:57.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6547700 1 -- 172.21.15.32:0/3978890971 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ec00c0c0 con 0x7f63f80d7c50 2021-04-09T18:32:57.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63f6547700 1 -- 172.21.15.32:0/3978890971 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63ec00c510 con 0x7f63f80d7c50 2021-04-09T18:32:57.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63fd740700 1 -- 172.21.15.32:0/3978890971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f80d7c50 msgr2=0x7f63f81b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.491 7f63fd740700 1 --2- 172.21.15.32:0/3978890971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f80d7c50 0x7f63f81b0490 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 rx=0x7f63ec006650 tx=0x7f63ec009a80).stop 2021-04-09T18:32:57.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.492 7f63fd740700 1 -- 172.21.15.32:0/3978890971 shutdown_connections 2021-04-09T18:32:57.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.492 7f63fd740700 1 --2- 172.21.15.32:0/3978890971 >> v2:172.21.15.86:3300/0 conn(0x7f63f80d8050 0x7f63f810d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.492 7f63fd740700 1 --2- 172.21.15.32:0/3978890971 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f80d7c50 0x7f63f81b0490 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.492 7f63fd740700 1 -- 172.21.15.32:0/3978890971 >> 172.21.15.32:0/3978890971 conn(0x7f63f80ce5b0 msgr2=0x7f63f80cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:57.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.492 7f63fd740700 1 -- 172.21.15.32:0/3978890971 shutdown_connections 2021-04-09T18:32:57.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.492 7f63fd740700 1 -- 172.21.15.32:0/3978890971 wait complete. 2021-04-09T18:32:57.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.512 7f63fd740700 1 Processor -- start 2021-04-09T18:32:57.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.512 7f63fd740700 1 -- start start 2021-04-09T18:32:57.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63fd740700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f809d680 0x7f63f809afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63fd740700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f809da80 0x7f63f8191dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63fd740700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f81923f0 0x7f63f809be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63fd740700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f809cbe0 con 0x7f63f809da80 2021-04-09T18:32:57.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63fd740700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f809b8e0 con 0x7f63f809d680 2021-04-09T18:32:57.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63fd740700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f809b5c0 con 0x7f63f81923f0 2021-04-09T18:32:57.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f6d48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f809da80 0x7f63f8191dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f6d48700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f809da80 0x7f63f8191dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58188/0 (socket says 172.21.15.32:58188) 2021-04-09T18:32:57.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f6d48700 1 -- 172.21.15.32:0/1690679037 learned_addr learned my addr 172.21.15.32:0/1690679037 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:57.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f7549700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f809d680 0x7f63f809afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f7d4a700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f81923f0 0x7f63f809be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f6d48700 1 -- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f81923f0 msgr2=0x7f63f809be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f6d48700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f81923f0 0x7f63f809be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f6d48700 1 -- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f809d680 msgr2=0x7f63f809afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f6d48700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f809d680 0x7f63f809afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f6d48700 1 -- 172.21.15.32:0/1690679037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f809c480 con 0x7f63f809da80 2021-04-09T18:32:57.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63f6d48700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f809da80 0x7f63f8191dd0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 rx=0x7f63ec009650 tx=0x7f63ec00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:57.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63dffff700 1 -- 172.21.15.32:0/1690679037 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63ec0093a0 con 0x7f63f809da80 2021-04-09T18:32:57.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63dffff700 1 -- 172.21.15.32:0/1690679037 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ec006070 con 0x7f63f809da80 2021-04-09T18:32:57.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63dffff700 1 -- 172.21.15.32:0/1690679037 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63ec0052e0 con 0x7f63f809da80 2021-04-09T18:32:57.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.513 7f63fd740700 1 -- 172.21.15.32:0/1690679037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f809c480 con 0x7f63f809da80 2021-04-09T18:32:57.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.514 7f63fd740700 1 -- 172.21.15.32:0/1690679037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f809c480 con 0x7f63f809da80 2021-04-09T18:32:57.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.514 7f63dffff700 1 -- 172.21.15.32:0/1690679037 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f63ec009300 con 0x7f63f809da80 2021-04-09T18:32:57.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.514 7f63dffff700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63d80188c0 0x7f63d801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.514 7f63dffff700 1 -- 172.21.15.32:0/1690679037 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f63ec032de0 con 0x7f63f809da80 2021-04-09T18:32:57.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.514 7f63f7549700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63d80188c0 0x7f63d801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.515 7f63f7549700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63d80188c0 0x7f63d801adc0 crc :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:57.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.534 7f63fd740700 1 -- 172.21.15.32:0/1690679037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f8093fc0 con 0x7f63f809da80 2021-04-09T18:32:57.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.538 7f63dffff700 1 -- 172.21.15.32:0/1690679037 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f63ec006030 con 0x7f63f809da80 2021-04-09T18:32:57.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.723 7f63fd740700 1 -- 172.21.15.32:0/1690679037 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]} v 0) v1 -- 0x7f63f8093fc0 con 0x7f63f809da80 2021-04-09T18:32:57.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.727 7f63dffff700 1 -- 172.21.15.32:0/1690679037 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]}]=0 updated caps for client.0 v45) v1 ==== 209+0+0 (secure 0 0 0) 0x7f63ec009300 con 0x7f63f809da80 2021-04-09T18:32:57.746 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:32:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 -- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63d80188c0 msgr2=0x7f63d801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63d80188c0 0x7f63d801adc0 crc :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 -- 172.21.15.32:0/1690679037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f809da80 msgr2=0x7f63f8191dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f809da80 0x7f63f8191dd0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 rx=0x7f63ec009650 tx=0x7f63ec00b8e0).stop 2021-04-09T18:32:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 -- 172.21.15.32:0/1690679037 shutdown_connections 2021-04-09T18:32:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f63d80188c0 0x7f63d801adc0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63f809da80 0x7f63f8191dd0 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f81923f0 0x7f63f809be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 --2- 172.21.15.32:0/1690679037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f809d680 0x7f63f809afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 -- 172.21.15.32:0/1690679037 >> 172.21.15.32:0/1690679037 conn(0x7f63f80adaf0 msgr2=0x7f63f80bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 -- 172.21.15.32:0/1690679037 shutdown_connections 2021-04-09T18:32:57.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.733 7f63fd740700 1 -- 172.21.15.32:0/1690679037 wait complete. 2021-04-09T18:32:57.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=alpha_data, allow rw pool=bravo_data' 2021-04-09T18:32:57.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.904 7ff730be6700 1 Processor -- start 2021-04-09T18:32:57.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.904 7ff730be6700 1 -- start start 2021-04-09T18:32:57.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.904 7ff730be6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff72c0d8050 0x7ff72c10d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.904 7ff730be6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c0d8620 con 0x7ff72c0cffb0 2021-04-09T18:32:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.904 7ff730be6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c0d2800 con 0x7ff72c0d7c50 2021-04-09T18:32:57.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.904 7ff730be6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72c1b0c80 con 0x7ff72c0d8050 2021-04-09T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72a85e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff72c0cffb0 0x7ff72c087e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39554/0 (socket says 172.21.15.32:39554) 2021-04-09T18:32:57.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72a85e700 1 -- 172.21.15.32:0/1969291804 learned_addr learned my addr 172.21.15.32:0/1969291804 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:57.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72b05f700 1 --2- 172.21.15.32:0/1969291804 >> v2:172.21.15.86:3300/0 conn(0x7ff72c0d8050 0x7ff72c10d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845857400 0 0) 0x7ff71c001060 con 0x7ff72c0cffb0 2021-04-09T18:32:57.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff714002fe0 con 0x7ff72c0cffb0 2021-04-09T18:32:57.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2162765483 0 0) 0x7ff71c001460 con 0x7ff72c0cffb0 2021-04-09T18:32:57.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 >> v2:172.21.15.86:3300/0 conn(0x7ff72c0d8050 msgr2=0x7ff72c10d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:57.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 --2- 172.21.15.32:0/1969291804 >> v2:172.21.15.86:3300/0 conn(0x7ff72c0d8050 0x7ff72c10d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.955 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 >> v1:172.21.15.86:6789/0 conn(0x7ff72c0d7c50 legacy=0x7ff72c1b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c0d91f0 con 0x7ff72c0cffb0 2021-04-09T18:32:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7ff71c001980 con 0x7ff72c0cffb0 2021-04-09T18:32:57.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 >> v1:172.21.15.32:6789/0 conn(0x7ff72c0cffb0 legacy=0x7ff72c087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff714003a70 0x7ff714005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.957 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7140067a0 0x7ff714008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7140095a0 0x7ff71400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71400c310 con 0x7ff7140095a0 2021-04-09T18:32:57.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71400c530 con 0x7ff714003a70 2021-04-09T18:32:57.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72985c700 1 -- 172.21.15.32:0/1969291804 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff714050760 con 0x7ff7140067a0 2021-04-09T18:32:57.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72a85e700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff714003a70 0x7ff714005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72b05f700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7140095a0 0x7ff71400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.905 7ff72a05d700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7140067a0 0x7ff714008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72b05f700 1 -- 172.21.15.32:0/1969291804 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7140067a0 msgr2=0x7ff714008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:57.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72b05f700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7140067a0 0x7ff714008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72b05f700 1 -- 172.21.15.32:0/1969291804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff714003a70 msgr2=0x7ff714005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:57.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72b05f700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff714003a70 0x7ff714005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72b05f700 1 -- 172.21.15.32:0/1969291804 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff714050990 con 0x7ff7140095a0 2021-04-09T18:32:57.960 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72a85e700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff714003a70 0x7ff714005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:57.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72b05f700 1 -- 172.21.15.32:0/1969291804 reap_dead start 2021-04-09T18:32:57.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72b05f700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7140095a0 0x7ff71400bac0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 rx=0x7ff72400c940 tx=0x7ff72400cce0).ready entity=mon.0 client_cookie=7e694b896c18fc28 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:57.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72985c700 1 -- 172.21.15.32:0/1969291804 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7240067d0 con 0x7ff7140095a0 2021-04-09T18:32:57.961 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72985c700 1 -- 172.21.15.32:0/1969291804 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff724009310 con 0x7ff7140095a0 2021-04-09T18:32:57.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff72985c700 1 -- 172.21.15.32:0/1969291804 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff724009830 con 0x7ff7140095a0 2021-04-09T18:32:57.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff730be6700 1 -- 172.21.15.32:0/1969291804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7140095a0 msgr2=0x7ff71400bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.906 7ff730be6700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7140095a0 0x7ff71400bac0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 rx=0x7ff72400c940 tx=0x7ff72400cce0).stop 2021-04-09T18:32:57.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.907 7ff730be6700 1 -- 172.21.15.32:0/1969291804 shutdown_connections 2021-04-09T18:32:57.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.907 7ff730be6700 1 --2- 172.21.15.32:0/1969291804 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7140095a0 0x7ff71400bac0 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.907 7ff730be6700 1 -- 172.21.15.32:0/1969291804 >> 172.21.15.32:0/1969291804 conn(0x7ff72c0ce5b0 msgr2=0x7ff72c0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:57.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.907 7ff730be6700 1 -- 172.21.15.32:0/1969291804 shutdown_connections 2021-04-09T18:32:57.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.907 7ff730be6700 1 -- 172.21.15.32:0/1969291804 wait complete. 2021-04-09T18:32:57.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.927 7ff730be6700 1 Processor -- start 2021-04-09T18:32:57.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.927 7ff730be6700 1 -- start start 2021-04-09T18:32:57.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff730be6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c094ea0 0x7ff72c0952a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff730be6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0af670 0x7ff72c0afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff730be6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c0adf40 0x7ff72c0ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff730be6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0958c0 con 0x7ff72c0af670 2021-04-09T18:32:57.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff730be6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c095cb0 con 0x7ff72c094ea0 2021-04-09T18:32:57.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff730be6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0b0090 con 0x7ff72c0adf40 2021-04-09T18:32:57.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0af670 0x7ff72c0afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a05d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0af670 0x7ff72c0afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58210/0 (socket says 172.21.15.32:58210) 2021-04-09T18:32:57.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a05d700 1 -- 172.21.15.32:0/3320541403 learned_addr learned my addr 172.21.15.32:0/3320541403 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:57.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72b05f700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c0adf40 0x7ff72c0ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a85e700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c094ea0 0x7ff72c0952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a05d700 1 -- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c0adf40 msgr2=0x7ff72c0ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a05d700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c0adf40 0x7ff72c0ae340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a05d700 1 -- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c094ea0 msgr2=0x7ff72c0952a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:57.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a05d700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c094ea0 0x7ff72c0952a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:57.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a05d700 1 -- 172.21.15.32:0/3320541403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c19e4d0 con 0x7ff72c0af670 2021-04-09T18:32:57.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff72a05d700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0af670 0x7ff72c0afa70 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 rx=0x7ff720007f70 tx=0x7ff72000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:57.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff71b7fe700 1 -- 172.21.15.32:0/3320541403 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff72000acf0 con 0x7ff72c0af670 2021-04-09T18:32:57.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff71b7fe700 1 -- 172.21.15.32:0/3320541403 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72000b070 con 0x7ff72c0af670 2021-04-09T18:32:57.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff730be6700 1 -- 172.21.15.32:0/3320541403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72c19e4d0 con 0x7ff72c0af670 2021-04-09T18:32:57.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.928 7ff71b7fe700 1 -- 172.21.15.32:0/3320541403 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff72000b500 con 0x7ff72c0af670 2021-04-09T18:32:57.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.929 7ff730be6700 1 -- 172.21.15.32:0/3320541403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c19e4d0 con 0x7ff72c0af670 2021-04-09T18:32:57.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.929 7ff71b7fe700 1 -- 172.21.15.32:0/3320541403 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff720006700 con 0x7ff72c0af670 2021-04-09T18:32:57.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.929 7ff71b7fe700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff710018930 0x7ff71001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:57.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.929 7ff71b7fe700 1 -- 172.21.15.32:0/3320541403 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7ff720032d90 con 0x7ff72c0af670 2021-04-09T18:32:57.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.929 7ff72a85e700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff710018930 0x7ff71001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:57.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.930 7ff72a85e700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff710018930 0x7ff71001ae10 crc :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:57.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.949 7ff730be6700 1 -- 172.21.15.32:0/3320541403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c093f90 con 0x7ff72c0af670 2021-04-09T18:32:57.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:57.952 7ff71b7fe700 1 -- 172.21.15.32:0/3320541403 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff72000acf0 con 0x7ff72c0af670 2021-04-09T18:32:58.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.138 7ff730be6700 1 -- 172.21.15.32:0/3320541403 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]} v 0) v1 -- 0x7ff72c0d8620 con 0x7ff72c0af670 2021-04-09T18:32:58.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.141 7ff71b7fe700 1 -- 172.21.15.32:0/3320541403 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=alpha_data, allow rw pool=bravo_data"]}]=0 updated caps for client.1 v46) v1 ==== 209+0+0 (secure 0 0 0) 0x7ff72001f6f0 con 0x7ff72c0af670 2021-04-09T18:32:58.161 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:32:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 -- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff710018930 msgr2=0x7ff71001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff710018930 0x7ff71001ae10 crc :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 -- 172.21.15.32:0/3320541403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0af670 msgr2=0x7ff72c0afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0af670 0x7ff72c0afa70 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 rx=0x7ff720007f70 tx=0x7ff72000a900).stop 2021-04-09T18:32:58.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 -- 172.21.15.32:0/3320541403 shutdown_connections 2021-04-09T18:32:58.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff710018930 0x7ff71001ae10 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff72c0af670 0x7ff72c0afa70 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff72c0adf40 0x7ff72c0ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 --2- 172.21.15.32:0/3320541403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff72c094ea0 0x7ff72c0952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 -- 172.21.15.32:0/3320541403 >> 172.21.15.32:0/3320541403 conn(0x7ff72c0adaf0 msgr2=0x7ff72c0c3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:58.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 -- 172.21.15.32:0/3320541403 shutdown_connections 2021-04-09T18:32:58.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.147 7ff730be6700 1 -- 172.21.15.32:0/3320541403 wait complete. 2021-04-09T18:32:58.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:58.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.306 7f7fe6a44700 1 Processor -- start 2021-04-09T18:32:58.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.307 7f7fe6a44700 1 -- start start 2021-04-09T18:32:58.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.307 7f7fe6a44700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe010d010 0x7f7fe00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.307 7f7fe6a44700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe010d480 con 0x7f7fe00d1da0 2021-04-09T18:32:58.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.307 7f7fe6a44700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe010e1f0 con 0x7f7fe00d8490 2021-04-09T18:32:58.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.307 7f7fe6a44700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe010e780 con 0x7f7fe010d010 2021-04-09T18:32:58.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.307 7f7fdffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7fe00d1da0 0x7f7fe00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39574/0 (socket says 172.21.15.32:39574) 2021-04-09T18:32:58.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.307 7f7fdffff700 1 -- 172.21.15.32:0/3702227516 learned_addr learned my addr 172.21.15.32:0/3702227516 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:58.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.307 7f7fe4860700 1 --2- 172.21.15.32:0/3702227516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe010d010 0x7f7fe00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fdf7fe700 1 -- 172.21.15.32:0/3702227516 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107871470 0 0) 0x7f7fd80009b0 con 0x7f7fe00d8490 2021-04-09T18:32:58.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fdf7fe700 1 -- 172.21.15.32:0/3702227516 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fcc002fe0 con 0x7f7fe00d8490 2021-04-09T18:32:58.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fdf7fe700 1 -- 172.21.15.32:0/3702227516 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242538290 0 0) 0x7f7fd0001060 con 0x7f7fe00d1da0 2021-04-09T18:32:58.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fdf7fe700 1 -- 172.21.15.32:0/3702227516 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fcc0042c0 con 0x7f7fe00d1da0 2021-04-09T18:32:58.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fe4860700 1 -- 172.21.15.32:0/3702227516 >> v1:172.21.15.86:6789/0 conn(0x7f7fe00d8490 legacy=0x7f7fe010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fe4860700 1 -- 172.21.15.32:0/3702227516 >> v1:172.21.15.32:6789/0 conn(0x7f7fe00d1da0 legacy=0x7f7fe00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fe4860700 1 -- 172.21.15.32:0/3702227516 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00d8d30 con 0x7f7fe010d010 2021-04-09T18:32:58.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fe4860700 1 --2- 172.21.15.32:0/3702227516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe010d010 0x7f7fe00d1780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f7fc8006650 tx=0x7f7fc8009a80).ready entity=mon.2 client_cookie=9c8c8ba92c1a879f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:58.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fdf7fe700 1 -- 172.21.15.32:0/3702227516 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fc80009b0 con 0x7f7fe010d010 2021-04-09T18:32:58.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fdf7fe700 1 -- 172.21.15.32:0/3702227516 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc800c0c0 con 0x7f7fe010d010 2021-04-09T18:32:58.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fdf7fe700 1 -- 172.21.15.32:0/3702227516 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fc800c510 con 0x7f7fe010d010 2021-04-09T18:32:58.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fe6a44700 1 -- 172.21.15.32:0/3702227516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe010d010 msgr2=0x7f7fe00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.308 7f7fe6a44700 1 --2- 172.21.15.32:0/3702227516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe010d010 0x7f7fe00d1780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f7fc8006650 tx=0x7f7fc8009a80).stop 2021-04-09T18:32:58.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.309 7f7fe6a44700 1 -- 172.21.15.32:0/3702227516 shutdown_connections 2021-04-09T18:32:58.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.309 7f7fe6a44700 1 --2- 172.21.15.32:0/3702227516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe010d010 0x7f7fe00d1780 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.309 7f7fe6a44700 1 -- 172.21.15.32:0/3702227516 >> 172.21.15.32:0/3702227516 conn(0x7f7fe00ce5f0 msgr2=0x7f7fe00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:58.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.309 7f7fe6a44700 1 -- 172.21.15.32:0/3702227516 shutdown_connections 2021-04-09T18:32:58.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.309 7f7fe6a44700 1 -- 172.21.15.32:0/3702227516 wait complete. 2021-04-09T18:32:58.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.329 7f7fe6a44700 1 Processor -- start 2021-04-09T18:32:58.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.329 7f7fe6a44700 1 -- start start 2021-04-09T18:32:58.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fe6a44700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe00a7f50 0x7f7fe0180be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fe6a44700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe0181200 0x7f7fe008ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fe6a44700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe008b3b0 0x7f7fe01a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fe6a44700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe008a890 con 0x7f7fe0181200 2021-04-09T18:32:58.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fe6a44700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0181640 con 0x7f7fe008b3b0 2021-04-09T18:32:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fe6a44700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe01819f0 con 0x7f7fe00a7f50 2021-04-09T18:32:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fdffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe0181200 0x7f7fe008ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fdffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe0181200 0x7f7fe008ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58222/0 (socket says 172.21.15.32:58222) 2021-04-09T18:32:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fdffff700 1 -- 172.21.15.32:0/4245172257 learned_addr learned my addr 172.21.15.32:0/4245172257 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:58.358 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fe4860700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe00a7f50 0x7f7fe0180be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fe5061700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe008b3b0 0x7f7fe01a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fdffff700 1 -- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe00a7f50 msgr2=0x7f7fe0180be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fdffff700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe00a7f50 0x7f7fe0180be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.359 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fdffff700 1 -- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe008b3b0 msgr2=0x7f7fe01a32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fdffff700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe008b3b0 0x7f7fe01a32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fdffff700 1 -- 172.21.15.32:0/4245172257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe008b7b0 con 0x7f7fe0181200 2021-04-09T18:32:58.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.330 7f7fe4860700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe00a7f50 0x7f7fe0180be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:32:58.360 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.331 7f7fdffff700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe0181200 0x7f7fe008ad90 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 rx=0x7f7fd0009960 tx=0x7f7fd000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:58.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.331 7f7fdd7fa700 1 -- 172.21.15.32:0/4245172257 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd00062b0 con 0x7f7fe0181200 2021-04-09T18:32:58.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.331 7f7fdd7fa700 1 -- 172.21.15.32:0/4245172257 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0004720 con 0x7f7fe0181200 2021-04-09T18:32:58.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.331 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe008b7b0 con 0x7f7fe0181200 2021-04-09T18:32:58.361 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.331 7f7fdd7fa700 1 -- 172.21.15.32:0/4245172257 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd0004b70 con 0x7f7fe0181200 2021-04-09T18:32:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.331 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe008b7b0 con 0x7f7fe0181200 2021-04-09T18:32:58.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.331 7f7fdd7fa700 1 -- 172.21.15.32:0/4245172257 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7fd0006570 con 0x7f7fe0181200 2021-04-09T18:32:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.331 7f7fdd7fa700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fc4018930 0x7f7fc401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.363 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.332 7f7fdd7fa700 1 -- 172.21.15.32:0/4245172257 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f7fd0033180 con 0x7f7fe0181200 2021-04-09T18:32:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.332 7f7fe4860700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fc4018930 0x7f7fc401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.332 7f7fe4860700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fc4018930 0x7f7fc401ae10 crc :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:58.367 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.351 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe01b7070 con 0x7f7fe0181200 2021-04-09T18:32:58.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.355 7f7fdd7fa700 1 -- 172.21.15.32:0/4245172257 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7fd000be30 con 0x7f7fe0181200 2021-04-09T18:32:58.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.546 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fe01b7070 con 0x7f7fe0181200 2021-04-09T18:32:58.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.547 7f7fdd7fa700 1 -- 172.21.15.32:0/4245172257 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+3720 (secure 0 0 0) 0x7f7fd0004720 con 0x7f7fe0181200 2021-04-09T18:32:58.564 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 390 2021-04-09T18:32:58.565 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:58.565 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":390,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386},{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":232,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11728":{"gid":11728,"name":"b","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2382296066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2382296066},{"type":"v1","addr":"172.21.15.86:6836","nonce":2382296066}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fc4018930 msgr2=0x7f7fc401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fc4018930 0x7f7fc401ae10 crc :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe0181200 msgr2=0x7f7fe008ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe0181200 0x7f7fe008ad90 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 rx=0x7f7fd0009960 tx=0x7f7fd000bcf0).stop 2021-04-09T18:32:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 shutdown_connections 2021-04-09T18:32:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7fc4018930 0x7f7fc401ae10 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7fe0181200 0x7f7fe008ad90 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe008b3b0 0x7f7fe01a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 --2- 172.21.15.32:0/4245172257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe00a7f50 0x7f7fe0180be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 >> 172.21.15.32:0/4245172257 conn(0x7f7fe01aa2b0 msgr2=0x7f7fe01a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:58.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 shutdown_connections 2021-04-09T18:32:58.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.549 7f7fe6a44700 1 -- 172.21.15.32:0/4245172257 wait complete. 2021-04-09T18:32:58.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.708 7f061e7a9700 1 Processor -- start 2021-04-09T18:32:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.708 7f061e7a9700 1 -- start start 2021-04-09T18:32:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f061e7a9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f061810da00 0x7f061810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f061e7a9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061810d480 con 0x7f061810d010 2021-04-09T18:32:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f061e7a9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180d1f70 con 0x7f06180d7ca0 2021-04-09T18:32:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f061e7a9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06180d8d30 con 0x7f061810da00 2021-04-09T18:32:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f0617fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f061810d010 0x7f06180d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39586/0 (socket says 172.21.15.32:39586) 2021-04-09T18:32:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f0617fff700 1 -- 172.21.15.32:0/634567899 learned_addr learned my addr 172.21.15.32:0/634567899 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f061cdc6700 1 --2- 172.21.15.32:0/634567899 >> v2:172.21.15.86:3300/0 conn(0x7f061810da00 0x7f061810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f06177fe700 1 -- 172.21.15.32:0/634567899 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079805582 0 0) 0x7f0608000ce0 con 0x7f061810d010 2021-04-09T18:32:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f06177fe700 1 -- 172.21.15.32:0/634567899 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0604002fe0 con 0x7f061810d010 2021-04-09T18:32:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f06177fe700 1 -- 172.21.15.32:0/634567899 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237320865 0 0) 0x7f0600000c60 con 0x7f06180d7ca0 2021-04-09T18:32:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f06177fe700 1 -- 172.21.15.32:0/634567899 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06040042c0 con 0x7f06180d7ca0 2021-04-09T18:32:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.709 7f061cdc6700 1 -- 172.21.15.32:0/634567899 >> v1:172.21.15.86:6789/0 conn(0x7f06180d7ca0 legacy=0x7f06180d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.710 7f061cdc6700 1 -- 172.21.15.32:0/634567899 >> v1:172.21.15.32:6789/0 conn(0x7f061810d010 legacy=0x7f06180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.710 7f061cdc6700 1 -- 172.21.15.32:0/634567899 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06180d9220 con 0x7f061810da00 2021-04-09T18:32:58.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.710 7f061cdc6700 1 --2- 172.21.15.32:0/634567899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061810da00 0x7f061810de00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f0610006e50 tx=0x7f0610009df0).ready entity=mon.1 client_cookie=b820c49bfe177327 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:58.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.710 7f06177fe700 1 -- 172.21.15.32:0/634567899 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f061000b510 con 0x7f061810da00 2021-04-09T18:32:58.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.710 7f06177fe700 1 -- 172.21.15.32:0/634567899 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0610003030 con 0x7f061810da00 2021-04-09T18:32:58.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.710 7f06177fe700 1 -- 172.21.15.32:0/634567899 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f061000b7d0 con 0x7f061810da00 2021-04-09T18:32:58.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.710 7f061e7a9700 1 -- 172.21.15.32:0/634567899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061810da00 msgr2=0x7f061810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.710 7f061e7a9700 1 --2- 172.21.15.32:0/634567899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061810da00 0x7f061810de00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f0610006e50 tx=0x7f0610009df0).stop 2021-04-09T18:32:58.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.711 7f061e7a9700 1 -- 172.21.15.32:0/634567899 shutdown_connections 2021-04-09T18:32:58.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.711 7f061e7a9700 1 --2- 172.21.15.32:0/634567899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f061810da00 0x7f061810de00 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.711 7f061e7a9700 1 -- 172.21.15.32:0/634567899 >> 172.21.15.32:0/634567899 conn(0x7f06180ce5f0 msgr2=0x7f06180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:58.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.711 7f061e7a9700 1 -- 172.21.15.32:0/634567899 shutdown_connections 2021-04-09T18:32:58.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.711 7f061e7a9700 1 -- 172.21.15.32:0/634567899 wait complete. 2021-04-09T18:32:58.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.731 7f061e7a9700 1 Processor -- start 2021-04-09T18:32:58.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.731 7f061e7a9700 1 -- start start 2021-04-09T18:32:58.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061e7a9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06181ac380 0x7f06181ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061e7a9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0618186c90 0x7f0618187090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061e7a9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0618185530 0x7f0618185930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061e7a9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0618186790 con 0x7f0618185530 2021-04-09T18:32:58.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061e7a9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06181acda0 con 0x7f0618186c90 2021-04-09T18:32:58.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061e7a9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061818e100 con 0x7f06181ac380 2021-04-09T18:32:58.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061cdc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0618185530 0x7f0618185930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061cdc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0618185530 0x7f0618185930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58238/0 (socket says 172.21.15.32:58238) 2021-04-09T18:32:58.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061cdc6700 1 -- 172.21.15.32:0/4247878146 learned_addr learned my addr 172.21.15.32:0/4247878146 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:58.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f060ffff700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0618186c90 0x7f0618187090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f0617fff700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06181ac380 0x7f06181ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061cdc6700 1 -- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06181ac380 msgr2=0x7f06181ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061cdc6700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06181ac380 0x7f06181ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061cdc6700 1 -- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0618186c90 msgr2=0x7f0618187090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061cdc6700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0618186c90 0x7f0618187090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061cdc6700 1 -- 172.21.15.32:0/4247878146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06181876b0 con 0x7f0618185530 2021-04-09T18:32:58.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061cdc6700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0618185530 0x7f0618185930 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f0610009f80 tx=0x7f061000bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:58.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f06157fa700 1 -- 172.21.15.32:0/4247878146 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06100093d0 con 0x7f0618185530 2021-04-09T18:32:58.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f06157fa700 1 -- 172.21.15.32:0/4247878146 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0610002080 con 0x7f0618185530 2021-04-09T18:32:58.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06181876b0 con 0x7f0618185530 2021-04-09T18:32:58.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f06157fa700 1 -- 172.21.15.32:0/4247878146 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06100097c0 con 0x7f0618185530 2021-04-09T18:32:58.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.732 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06181876b0 con 0x7f0618185530 2021-04-09T18:32:58.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.734 7f06157fa700 1 -- 172.21.15.32:0/4247878146 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f06100093b0 con 0x7f0618185530 2021-04-09T18:32:58.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.734 7f06157fa700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc0188c0 0x7f05fc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:58.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.734 7f06157fa700 1 -- 172.21.15.32:0/4247878146 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f0610032ec0 con 0x7f0618185530 2021-04-09T18:32:58.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.734 7f0617fff700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc0188c0 0x7f05fc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:58.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.734 7f0617fff700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc0188c0 0x7f05fc01adc0 crc :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:58.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.753 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06181acda0 con 0x7f0618185530 2021-04-09T18:32:58.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.756 7f06157fa700 1 -- 172.21.15.32:0/4247878146 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f061000bdd0 con 0x7f0618185530 2021-04-09T18:32:58.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.950 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06181acda0 con 0x7f0618185530 2021-04-09T18:32:58.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.950 7f06157fa700 1 -- 172.21.15.32:0/4247878146 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+3720 (secure 0 0 0) 0x7f06100093b0 con 0x7f0618185530 2021-04-09T18:32:58.967 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 390 2021-04-09T18:32:58.967 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:58.967 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":390,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386},{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":232,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11728":{"gid":11728,"name":"b","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2382296066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2382296066},{"type":"v1","addr":"172.21.15.86:6836","nonce":2382296066}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.952 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc0188c0 msgr2=0x7f05fc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.952 7f061e7a9700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc0188c0 0x7f05fc01adc0 crc :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.969 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.952 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0618185530 msgr2=0x7f0618185930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:58.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.952 7f061e7a9700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0618185530 0x7f0618185930 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f0610009f80 tx=0x7f061000bc90).stop 2021-04-09T18:32:58.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.953 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 shutdown_connections 2021-04-09T18:32:58.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.953 7f061e7a9700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f05fc0188c0 0x7f05fc01adc0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.970 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.953 7f061e7a9700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0618185530 0x7f0618185930 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.953 7f061e7a9700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0618186c90 0x7f0618187090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.953 7f061e7a9700 1 --2- 172.21.15.32:0/4247878146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06181ac380 0x7f06181ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:58.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.953 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 >> 172.21.15.32:0/4247878146 conn(0x7f061819b890 msgr2=0x7f06181a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:58.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.953 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 shutdown_connections 2021-04-09T18:32:58.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:58.953 7f061e7a9700 1 -- 172.21.15.32:0/4247878146 wait complete. 2021-04-09T18:32:58.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:59.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.112 7f305c363700 1 Processor -- start 2021-04-09T18:32:59.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.112 7f305c363700 1 -- start start 2021-04-09T18:32:59.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.113 7f305c363700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30540cffa0 0x7f30540d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.113 7f305c363700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30540d6c80 0x7f305410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.113 7f305c363700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30540d7280 con 0x7f30540d7800 2021-04-09T18:32:59.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.113 7f305c363700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30540d7080 con 0x7f30540d6c80 2021-04-09T18:32:59.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.113 7f305c363700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30540d8580 con 0x7f30540cffa0 2021-04-09T18:32:59.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.113 7f305a980700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f30540d6c80 0x7f305410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.113 7f305997e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f30540d7800 0x7f30540d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39604/0 (socket says 172.21.15.32:39604) 2021-04-09T18:32:59.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.113 7f305997e700 1 -- 172.21.15.32:0/184107616 learned_addr learned my addr 172.21.15.32:0/184107616 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:59.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.113 7f305a17f700 1 --2- 172.21.15.32:0/184107616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30540cffa0 0x7f30540d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305a980700 1 -- 172.21.15.32:0/184107616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30540cffa0 msgr2=0x7f30540d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:59.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305a980700 1 --2- 172.21.15.32:0/184107616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30540cffa0 0x7f30540d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305a980700 1 -- 172.21.15.32:0/184107616 >> v1:172.21.15.32:6789/0 conn(0x7f30540d7800 legacy=0x7f30540d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305a980700 1 -- 172.21.15.32:0/184107616 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30540d91e0 con 0x7f30540d6c80 2021-04-09T18:32:59.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305917d700 1 -- 172.21.15.32:0/184107616 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034776870 0 0) 0x7f30440009b0 con 0x7f30540d7800 2021-04-09T18:32:59.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305a980700 1 --2- 172.21.15.32:0/184107616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30540d6c80 0x7f305410cfd0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 rx=0x7f304c006ab0 tx=0x7f304c009b40).ready entity=mon.0 client_cookie=39e2caffc97403d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:59.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305917d700 1 -- 172.21.15.32:0/184107616 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f304c0022e0 con 0x7f30540d6c80 2021-04-09T18:32:59.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305917d700 1 -- 172.21.15.32:0/184107616 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c002660 con 0x7f30540d6c80 2021-04-09T18:32:59.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305917d700 1 -- 172.21.15.32:0/184107616 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f304c002b40 con 0x7f30540d6c80 2021-04-09T18:32:59.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305c363700 1 -- 172.21.15.32:0/184107616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30540d6c80 msgr2=0x7f305410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.114 7f305c363700 1 --2- 172.21.15.32:0/184107616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30540d6c80 0x7f305410cfd0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 rx=0x7f304c006ab0 tx=0x7f304c009b40).stop 2021-04-09T18:32:59.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.115 7f305c363700 1 -- 172.21.15.32:0/184107616 shutdown_connections 2021-04-09T18:32:59.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.115 7f305c363700 1 --2- 172.21.15.32:0/184107616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30540d6c80 0x7f305410cfd0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.115 7f305c363700 1 --2- 172.21.15.32:0/184107616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30540cffa0 0x7f30540d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.115 7f305c363700 1 -- 172.21.15.32:0/184107616 >> 172.21.15.32:0/184107616 conn(0x7f30540ce5c0 msgr2=0x7f30540cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:59.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.115 7f305c363700 1 -- 172.21.15.32:0/184107616 shutdown_connections 2021-04-09T18:32:59.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.115 7f305c363700 1 -- 172.21.15.32:0/184107616 wait complete. 2021-04-09T18:32:59.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.135 7f305c363700 1 Processor -- start 2021-04-09T18:32:59.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.135 7f305c363700 1 -- start start 2021-04-09T18:32:59.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305c363700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30541ab8a0 0x7f30541981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305c363700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30541abca0 0x7f30541a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305c363700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30541ac0a0 0x7f30541aa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305c363700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30541ac620 con 0x7f30541ab8a0 2021-04-09T18:32:59.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305c363700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3054185300 con 0x7f30541abca0 2021-04-09T18:32:59.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305c363700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30541ab320 con 0x7f30541ac0a0 2021-04-09T18:32:59.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a17f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30541ab8a0 0x7f30541981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a17f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30541ab8a0 0x7f30541981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58248/0 (socket says 172.21.15.32:58248) 2021-04-09T18:32:59.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a17f700 1 -- 172.21.15.32:0/2040509406 learned_addr learned my addr 172.21.15.32:0/2040509406 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305997e700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30541abca0 0x7f30541a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a980700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30541ac0a0 0x7f30541aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a17f700 1 -- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30541ac0a0 msgr2=0x7f30541aa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a17f700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30541ac0a0 0x7f30541aa500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a17f700 1 -- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30541abca0 msgr2=0x7f30541a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a17f700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30541abca0 0x7f30541a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a17f700 1 -- 172.21.15.32:0/2040509406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3054184e90 con 0x7f30541ab8a0 2021-04-09T18:32:59.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305997e700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30541abca0 0x7f30541a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:59.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305a17f700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30541ab8a0 0x7f30541981b0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 rx=0x7f3050009980 tx=0x7f305000bc50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:59.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f304affd700 1 -- 172.21.15.32:0/2040509406 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3050006030 con 0x7f30541ab8a0 2021-04-09T18:32:59.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f304affd700 1 -- 172.21.15.32:0/2040509406 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3050006370 con 0x7f30541ab8a0 2021-04-09T18:32:59.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305c363700 1 -- 172.21.15.32:0/2040509406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3054184e90 con 0x7f30541ab8a0 2021-04-09T18:32:59.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f304affd700 1 -- 172.21.15.32:0/2040509406 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3050005660 con 0x7f30541ab8a0 2021-04-09T18:32:59.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.136 7f305c363700 1 -- 172.21.15.32:0/2040509406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3054184e90 con 0x7f30541ab8a0 2021-04-09T18:32:59.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.137 7f304affd700 1 -- 172.21.15.32:0/2040509406 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3050006550 con 0x7f30541ab8a0 2021-04-09T18:32:59.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.137 7f304affd700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f303c0188d0 0x7f303c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.137 7f304affd700 1 -- 172.21.15.32:0/2040509406 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f3050035320 con 0x7f30541ab8a0 2021-04-09T18:32:59.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.137 7f305997e700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f303c0188d0 0x7f303c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.138 7f305997e700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f303c0188d0 0x7f303c01add0 crc :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:59.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.157 7f305c363700 1 -- 172.21.15.32:0/2040509406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3054182df0 con 0x7f30541ab8a0 2021-04-09T18:32:59.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.161 7f304affd700 1 -- 172.21.15.32:0/2040509406 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3050006030 con 0x7f30541ab8a0 2021-04-09T18:32:59.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.354 7f305c363700 1 -- 172.21.15.32:0/2040509406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3054182df0 con 0x7f30541ab8a0 2021-04-09T18:32:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.355 7f304affd700 1 -- 172.21.15.32:0/2040509406 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+3720 (secure 0 0 0) 0x7f3050006370 con 0x7f30541ab8a0 2021-04-09T18:32:59.371 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 390 2021-04-09T18:32:59.372 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:59.372 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":390,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386},{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":232,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11728":{"gid":11728,"name":"b","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2382296066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2382296066},{"type":"v1","addr":"172.21.15.86:6836","nonce":2382296066}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:59.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 -- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f303c0188d0 msgr2=0x7f303c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f303c0188d0 0x7f303c01add0 crc :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 -- 172.21.15.32:0/2040509406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30541ab8a0 msgr2=0x7f30541981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30541ab8a0 0x7f30541981b0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 rx=0x7f3050009980 tx=0x7f305000bc50).stop 2021-04-09T18:32:59.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 -- 172.21.15.32:0/2040509406 shutdown_connections 2021-04-09T18:32:59.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f303c0188d0 0x7f303c01add0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30541ac0a0 0x7f30541aa500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30541abca0 0x7f30541a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 --2- 172.21.15.32:0/2040509406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30541ab8a0 0x7f30541981b0 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 -- 172.21.15.32:0/2040509406 >> 172.21.15.32:0/2040509406 conn(0x7f3054183870 msgr2=0x7f30541a9650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 -- 172.21.15.32:0/2040509406 shutdown_connections 2021-04-09T18:32:59.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.357 7f305c363700 1 -- 172.21.15.32:0/2040509406 wait complete. 2021-04-09T18:32:59.384 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s 2021-04-09T18:32:59.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:32:59.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.516 7fa38ccb7700 1 Processor -- start 2021-04-09T18:32:59.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.516 7fa38ccb7700 1 -- start start 2021-04-09T18:32:59.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.517 7fa38ccb7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38810d010 0x7fa3880d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.517 7fa38ccb7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa3880d8490 0x7fa38810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.517 7fa38ccb7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38810d480 con 0x7fa3880d1da0 2021-04-09T18:32:59.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.517 7fa38ccb7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38810e780 con 0x7fa3880d8490 2021-04-09T18:32:59.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.517 7fa38ccb7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880d8d30 con 0x7fa38810d010 2021-04-09T18:32:59.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.517 7fa38705f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa3880d8490 0x7fa38810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.517 7fa38605d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa3880d1da0 0x7fa3880d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39618/0 (socket says 172.21.15.32:39618) 2021-04-09T18:32:59.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.517 7fa38605d700 1 -- 172.21.15.32:0/736674483 learned_addr learned my addr 172.21.15.32:0/736674483 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:32:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38685e700 1 --2- 172.21.15.32:0/736674483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38810d010 0x7fa3880d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38705f700 1 -- 172.21.15.32:0/736674483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38810d010 msgr2=0x7fa3880d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:32:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38705f700 1 --2- 172.21.15.32:0/736674483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38810d010 0x7fa3880d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38705f700 1 -- 172.21.15.32:0/736674483 >> v1:172.21.15.32:6789/0 conn(0x7fa3880d1da0 legacy=0x7fa3880d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38705f700 1 -- 172.21.15.32:0/736674483 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880d9220 con 0x7fa3880d8490 2021-04-09T18:32:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38585c700 1 -- 172.21.15.32:0/736674483 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666453253 0 0) 0x7fa37c000ce0 con 0x7fa3880d1da0 2021-04-09T18:32:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38705f700 1 --2- 172.21.15.32:0/736674483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3880d8490 0x7fa38810da00 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 rx=0x7fa380006e30 tx=0x7fa380009e60).ready entity=mon.0 client_cookie=15ab4cc1ebcbcdf5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38585c700 1 -- 172.21.15.32:0/736674483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa380006a50 con 0x7fa3880d8490 2021-04-09T18:32:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38585c700 1 -- 172.21.15.32:0/736674483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38000b300 con 0x7fa3880d8490 2021-04-09T18:32:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38585c700 1 -- 172.21.15.32:0/736674483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa38000b750 con 0x7fa3880d8490 2021-04-09T18:32:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38ccb7700 1 -- 172.21.15.32:0/736674483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3880d8490 msgr2=0x7fa38810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.518 7fa38ccb7700 1 --2- 172.21.15.32:0/736674483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3880d8490 0x7fa38810da00 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 rx=0x7fa380006e30 tx=0x7fa380009e60).stop 2021-04-09T18:32:59.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.519 7fa38ccb7700 1 -- 172.21.15.32:0/736674483 shutdown_connections 2021-04-09T18:32:59.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.519 7fa38ccb7700 1 --2- 172.21.15.32:0/736674483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3880d8490 0x7fa38810da00 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.519 7fa38ccb7700 1 --2- 172.21.15.32:0/736674483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38810d010 0x7fa3880d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.519 7fa38ccb7700 1 -- 172.21.15.32:0/736674483 >> 172.21.15.32:0/736674483 conn(0x7fa3880ce5f0 msgr2=0x7fa3880d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:59.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.519 7fa38ccb7700 1 -- 172.21.15.32:0/736674483 shutdown_connections 2021-04-09T18:32:59.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.519 7fa38ccb7700 1 -- 172.21.15.32:0/736674483 wait complete. 2021-04-09T18:32:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.539 7fa38ccb7700 1 Processor -- start 2021-04-09T18:32:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.539 7fa38ccb7700 1 -- start start 2021-04-09T18:32:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38ccb7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3881aca20 0x7fa3881ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38ccb7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38818bd40 0x7fa38818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38ccb7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38808fba0 0x7fa38808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38ccb7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38818c7e0 con 0x7fa3881aca20 2021-04-09T18:32:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38ccb7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38818ca30 con 0x7fa38818bd40 2021-04-09T18:32:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38ccb7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880905f0 con 0x7fa38808fba0 2021-04-09T18:32:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3881aca20 0x7fa3881ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3881aca20 0x7fa3881ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58262/0 (socket says 172.21.15.32:58262) 2021-04-09T18:32:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38685e700 1 -- 172.21.15.32:0/2844275962 learned_addr learned my addr 172.21.15.32:0/2844275962 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:32:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38605d700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38818bd40 0x7fa38818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38705f700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38808fba0 0x7fa38808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38685e700 1 -- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38808fba0 msgr2=0x7fa38808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38685e700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38808fba0 0x7fa38808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38685e700 1 -- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38818bd40 msgr2=0x7fa38818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38685e700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38818bd40 0x7fa38818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38685e700 1 -- 172.21.15.32:0/2844275962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880b4300 con 0x7fa3881aca20 2021-04-09T18:32:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38605d700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38818bd40 0x7fa38818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:32:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38685e700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3881aca20 0x7fa3881ace20 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 rx=0x7fa37800b580 tx=0x7fa37800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa3777fe700 1 -- 172.21.15.32:0/2844275962 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3780064d0 con 0x7fa3881aca20 2021-04-09T18:32:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa3777fe700 1 -- 172.21.15.32:0/2844275962 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37800bd10 con 0x7fa3881aca20 2021-04-09T18:32:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.540 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3880b4300 con 0x7fa3881aca20 2021-04-09T18:32:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.541 7fa3777fe700 1 -- 172.21.15.32:0/2844275962 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa378002030 con 0x7fa3881aca20 2021-04-09T18:32:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.541 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3880b4300 con 0x7fa3881aca20 2021-04-09T18:32:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.541 7fa3777fe700 1 -- 172.21.15.32:0/2844275962 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa3780052e0 con 0x7fa3881aca20 2021-04-09T18:32:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.541 7fa3777fe700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa36c018900 0x7fa36c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:32:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.541 7fa3777fe700 1 -- 172.21.15.32:0/2844275962 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fa378032db0 con 0x7fa3881aca20 2021-04-09T18:32:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.541 7fa38605d700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa36c018900 0x7fa36c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:32:59.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.542 7fa38605d700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa36c018900 0x7fa36c01adc0 crc :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:32:59.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.561 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa388182d70 con 0x7fa3881aca20 2021-04-09T18:32:59.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.565 7fa3777fe700 1 -- 172.21.15.32:0/2844275962 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa378006820 con 0x7fa3881aca20 2021-04-09T18:32:59.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.758 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa38818c760 con 0x7fa3881aca20 2021-04-09T18:32:59.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.758 7fa3777fe700 1 -- 172.21.15.32:0/2844275962 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+3720 (secure 0 0 0) 0x7fa37800bd10 con 0x7fa3881aca20 2021-04-09T18:32:59.865 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 390 2021-04-09T18:32:59.866 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:32:59.866 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":390,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386},{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":232,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11728":{"gid":11728,"name":"b","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6836/2382296066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2382296066},{"type":"v1","addr":"172.21.15.86:6836","nonce":2382296066}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:32:59.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa36c018900 msgr2=0x7fa36c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa36c018900 0x7fa36c01adc0 crc :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3881aca20 msgr2=0x7fa3881ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:32:59.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3881aca20 0x7fa3881ace20 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 rx=0x7fa37800b580 tx=0x7fa37800b920).stop 2021-04-09T18:32:59.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 shutdown_connections 2021-04-09T18:32:59.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa36c018900 0x7fa36c01adc0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38808fba0 0x7fa38808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38818bd40 0x7fa38818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 --2- 172.21.15.32:0/2844275962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3881aca20 0x7fa3881ace20 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:32:59.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 >> 172.21.15.32:0/2844275962 conn(0x7fa38819b460 msgr2=0x7fa38819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:32:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 shutdown_connections 2021-04-09T18:32:59.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:32:59.761 7fa38ccb7700 1 -- 172.21.15.32:0/2844275962 wait complete. 2021-04-09T18:32:59.873 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:32:59.873 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:32:59.882 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:32:59.883 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2021-04-09T18:33:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.039 7fd48435e700 1 Processor -- start 2021-04-09T18:33:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.039 7fd48435e700 1 -- start start 2021-04-09T18:33:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd48435e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd47c0d6c90 0x7fd47c0d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd48435e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd47c0d2450 0x7fd47c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd48435e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c0d7290 con 0x7fd47c0cffb0 2021-04-09T18:33:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd48435e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47c0d7090 con 0x7fd47c0d6c90 2021-04-09T18:33:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd48435e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c0d91f0 con 0x7fd47c0d2450 2021-04-09T18:33:00.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd48217a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd47c0cffb0 0x7fd47c0d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39628/0 (socket says 172.21.15.32:39628) 2021-04-09T18:33:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd48217a700 1 -- 172.21.15.32:0/2609738144 learned_addr learned my addr 172.21.15.32:0/2609738144 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:00.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd48297b700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd47c0d2450 0x7fd47c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd481979700 1 --2- 172.21.15.32:0/2609738144 >> v2:172.21.15.86:3300/0 conn(0x7fd47c0d6c90 0x7fd47c0d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd481178700 1 -- 172.21.15.32:0/2609738144 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68392126 0 0) 0x7fd478000ce0 con 0x7fd47c0cffb0 2021-04-09T18:33:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.040 7fd481178700 1 -- 172.21.15.32:0/2609738144 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd468002fe0 con 0x7fd47c0cffb0 2021-04-09T18:33:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1616808459 0 0) 0x7fd4780010e0 con 0x7fd47c0cffb0 2021-04-09T18:33:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd47c0d2450 msgr2=0x7fd47c10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd47c0d2450 0x7fd47c10cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 >> v2:172.21.15.86:3300/0 conn(0x7fd47c0d6c90 msgr2=0x7fd47c0d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 --2- 172.21.15.32:0/2609738144 >> v2:172.21.15.86:3300/0 conn(0x7fd47c0d6c90 0x7fd47c0d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c0d2850 con 0x7fd47c0cffb0 2021-04-09T18:33:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fd478001600 con 0x7fd47c0cffb0 2021-04-09T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 >> v1:172.21.15.32:6789/0 conn(0x7fd47c0cffb0 legacy=0x7fd47c0d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd468003a70 0x7fd468005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd46804a7b0 0x7fd46804ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd46804d5b0 0x7fd46804faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4680502f0 con 0x7fd468003a70 2021-04-09T18:33:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd468050510 con 0x7fd46804a7b0 2021-04-09T18:33:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd468050730 con 0x7fd46804d5b0 2021-04-09T18:33:00.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48217a700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd468003a70 0x7fd468005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:00.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481979700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd46804a7b0 0x7fd46804ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:00.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48297b700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd46804d5b0 0x7fd46804faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48217a700 1 -- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd46804d5b0 msgr2=0x7fd46804faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:00.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48217a700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd46804d5b0 0x7fd46804faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48217a700 1 -- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd46804a7b0 msgr2=0x7fd46804ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48217a700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd46804a7b0 0x7fd46804ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:00.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48217a700 1 -- 172.21.15.32:0/2609738144 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4680509d0 con 0x7fd468003a70 2021-04-09T18:33:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481979700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd46804a7b0 0x7fd46804ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:00.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48297b700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd46804d5b0 0x7fd46804faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48217a700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd468003a70 0x7fd468005f00 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 rx=0x7fd4780014c0 tx=0x7fd478007d20).ready entity=mon.0 client_cookie=dca5e458abd2063 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4780070c0 con 0x7fd468003a70 2021-04-09T18:33:00.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4780043a0 con 0x7fd468003a70 2021-04-09T18:33:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd481178700 1 -- 172.21.15.32:0/2609738144 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd478004900 con 0x7fd468003a70 2021-04-09T18:33:00.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48435e700 1 -- 172.21.15.32:0/2609738144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd468003a70 msgr2=0x7fd468005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:00.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48435e700 1 --2- 172.21.15.32:0/2609738144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd468003a70 0x7fd468005f00 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 rx=0x7fd4780014c0 tx=0x7fd478007d20).stop 2021-04-09T18:33:00.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.041 7fd48297b700 1 -- 172.21.15.32:0/2609738144 reap_dead start 2021-04-09T18:33:00.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.042 7fd48435e700 1 -- 172.21.15.32:0/2609738144 shutdown_connections 2021-04-09T18:33:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.042 7fd48435e700 1 -- 172.21.15.32:0/2609738144 >> 172.21.15.32:0/2609738144 conn(0x7fd47c0ce5d0 msgr2=0x7fd47c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:00.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.042 7fd48435e700 1 -- 172.21.15.32:0/2609738144 shutdown_connections 2021-04-09T18:33:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.042 7fd48435e700 1 -- 172.21.15.32:0/2609738144 wait complete. 2021-04-09T18:33:00.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.062 7fd48435e700 1 Processor -- start 2021-04-09T18:33:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.062 7fd48435e700 1 -- start start 2021-04-09T18:33:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48435e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c08d210 0x7fd47c08fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:00.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48435e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd47c0903f0 0x7fd47c1a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:00.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48435e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd47c1a6080 0x7fd47c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:00.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48435e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c08d690 con 0x7fd47c08d210 2021-04-09T18:33:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48435e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c0b0fd0 con 0x7fd47c0903f0 2021-04-09T18:33:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48435e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c1a6480 con 0x7fd47c1a6080 2021-04-09T18:33:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48217a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c08d210 0x7fd47c08fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48217a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c08d210 0x7fd47c08fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58282/0 (socket says 172.21.15.32:58282) 2021-04-09T18:33:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48217a700 1 -- 172.21.15.32:0/566187251 learned_addr learned my addr 172.21.15.32:0/566187251 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd481979700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd47c0903f0 0x7fd47c1a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48297b700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd47c1a6080 0x7fd47c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48217a700 1 -- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd47c1a6080 msgr2=0x7fd47c1a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48217a700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd47c1a6080 0x7fd47c1a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48217a700 1 -- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd47c0903f0 msgr2=0x7fd47c1a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48217a700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd47c0903f0 0x7fd47c1a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48217a700 1 -- 172.21.15.32:0/566187251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c08d8b0 con 0x7fd47c08d210 2021-04-09T18:33:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.063 7fd48217a700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c08d210 0x7fd47c08fdd0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7fd4780013f0 tx=0x7fd47800a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.064 7fd472ffd700 1 -- 172.21.15.32:0/566187251 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd47800acf0 con 0x7fd47c08d210 2021-04-09T18:33:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.064 7fd472ffd700 1 -- 172.21.15.32:0/566187251 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4780070d0 con 0x7fd47c08d210 2021-04-09T18:33:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.064 7fd48435e700 1 -- 172.21.15.32:0/566187251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47c08d8b0 con 0x7fd47c08d210 2021-04-09T18:33:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.064 7fd472ffd700 1 -- 172.21.15.32:0/566187251 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4780043a0 con 0x7fd47c08d210 2021-04-09T18:33:00.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.064 7fd48435e700 1 -- 172.21.15.32:0/566187251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47c08d8b0 con 0x7fd47c08d210 2021-04-09T18:33:00.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.064 7fd472ffd700 1 -- 172.21.15.32:0/566187251 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd4780072b0 con 0x7fd47c08d210 2021-04-09T18:33:00.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.064 7fd472ffd700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4640188c0 0x7fd46401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:00.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.064 7fd472ffd700 1 -- 172.21.15.32:0/566187251 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fd478032df0 con 0x7fd47c08d210 2021-04-09T18:33:00.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.065 7fd481979700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4640188c0 0x7fd46401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:00.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.065 7fd481979700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4640188c0 0x7fd46401adc0 crc :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:00.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.084 7fd48435e700 1 -- 172.21.15.32:0/566187251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47c1a52b0 con 0x7fd47c08d210 2021-04-09T18:33:00.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.087 7fd472ffd700 1 -- 172.21.15.32:0/566187251 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd4780073e0 con 0x7fd47c08d210 2021-04-09T18:33:00.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:00.282 7fd48435e700 1 -- 172.21.15.32:0/566187251 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fd47c1a52b0 con 0x7fd47c08d210 2021-04-09T18:33:01.215 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:33:01.199 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:33:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.204 7fd472ffd700 1 -- 172.21.15.32:0/566187251 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 11728 v391) v1 ==== 96+0+0 (secure 0 0 0) 0x7fd4780070d0 con 0x7fd47c08d210 2021-04-09T18:33:01.264 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 11728 2021-04-09T18:33:01.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 -- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4640188c0 msgr2=0x7fd46401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:01.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4640188c0 0x7fd46401adc0 crc :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 -- 172.21.15.32:0/566187251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c08d210 msgr2=0x7fd47c08fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:01.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c08d210 0x7fd47c08fdd0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7fd4780013f0 tx=0x7fd47800a900).stop 2021-04-09T18:33:01.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 -- 172.21.15.32:0/566187251 shutdown_connections 2021-04-09T18:33:01.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd4640188c0 0x7fd46401adc0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd47c1a6080 0x7fd47c1a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd47c0903f0 0x7fd47c1a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 --2- 172.21.15.32:0/566187251 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c08d210 0x7fd47c08fdd0 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 -- 172.21.15.32:0/566187251 >> 172.21.15.32:0/566187251 conn(0x7fd47c1a8f50 msgr2=0x7fd47c188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:01.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 -- 172.21.15.32:0/566187251 shutdown_connections 2021-04-09T18:33:01.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.252 7fd48435e700 1 -- 172.21.15.32:0/566187251 wait complete. 2021-04-09T18:33:01.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:01.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.420 7fa0a40db700 1 Processor -- start 2021-04-09T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.421 7fa0a40db700 1 -- start start 2021-04-09T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.421 7fa0a40db700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c10d010 0x7fa09c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.421 7fa0a40db700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09c10d480 con 0x7fa09c0d8490 2021-04-09T18:33:01.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.421 7fa0a40db700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09c10e1f0 con 0x7fa09c0d1da0 2021-04-09T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.421 7fa0a40db700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c10e780 con 0x7fa09c10d010 2021-04-09T18:33:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.421 7fa0a26f8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa09c0d8490 0x7fa09c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39658/0 (socket says 172.21.15.32:39658) 2021-04-09T18:33:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.421 7fa0a26f8700 1 -- 172.21.15.32:0/3354952741 learned_addr learned my addr 172.21.15.32:0/3354952741 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a1ef7700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c10d010 0x7fa09c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718431972 0 0) 0x7fa094000d30 con 0x7fa09c0d8490 2021-04-09T18:33:01.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa088002fe0 con 0x7fa09c0d8490 2021-04-09T18:33:01.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060406145 0 0) 0x7fa08c000c60 con 0x7fa09c0d1da0 2021-04-09T18:33:01.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa088003210 con 0x7fa09c0d1da0 2021-04-09T18:33:01.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 147325462 0 0) 0x7fa094001460 con 0x7fa09c0d8490 2021-04-09T18:33:01.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c10d010 msgr2=0x7fa09c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:01.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c10d010 0x7fa09c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 >> v1:172.21.15.86:6789/0 conn(0x7fa09c0d1da0 legacy=0x7fa09c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:01.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c0d8d30 con 0x7fa09c0d8490 2021-04-09T18:33:01.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fa094001980 con 0x7fa09c0d8490 2021-04-09T18:33:01.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 >> v1:172.21.15.32:6789/0 conn(0x7fa09c0d8490 legacy=0x7fa09c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:01.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa088003b90 0x7fa088005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:01.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0880067a0 0x7fa088008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:01.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa088009510 0x7fa08800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:01.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0880502f0 con 0x7fa088009510 2021-04-09T18:33:01.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0880504d0 con 0x7fa088003b90 2021-04-09T18:33:01.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a26f8700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa088009510 0x7fa08800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:01.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0880506f0 con 0x7fa0880067a0 2021-04-09T18:33:01.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a1ef7700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa088003b90 0x7fa088005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:01.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.422 7fa0a16f6700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0880067a0 0x7fa088008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:01.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a26f8700 1 -- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0880067a0 msgr2=0x7fa088008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:01.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a26f8700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0880067a0 0x7fa088008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a26f8700 1 -- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa088003b90 msgr2=0x7fa088005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:01.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a26f8700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa088003b90 0x7fa088005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a26f8700 1 -- 172.21.15.32:0/3354952741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa088050990 con 0x7fa088009510 2021-04-09T18:33:01.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a1ef7700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa088003b90 0x7fa088005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:01.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a26f8700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa088009510 0x7fa08800ba30 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 rx=0x7fa094001af0 tx=0x7fa09400a060).ready entity=mon.0 client_cookie=a7f7694ba868cbce server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:01.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0940016e0 con 0x7fa088009510 2021-04-09T18:33:01.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0940070c0 con 0x7fa088009510 2021-04-09T18:33:01.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a0ef5700 1 -- 172.21.15.32:0/3354952741 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa094007550 con 0x7fa088009510 2021-04-09T18:33:01.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a40db700 1 -- 172.21.15.32:0/3354952741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa088009510 msgr2=0x7fa08800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:01.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a40db700 1 --2- 172.21.15.32:0/3354952741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa088009510 0x7fa08800ba30 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 rx=0x7fa094001af0 tx=0x7fa09400a060).stop 2021-04-09T18:33:01.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.423 7fa0a26f8700 1 -- 172.21.15.32:0/3354952741 reap_dead start 2021-04-09T18:33:01.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.424 7fa0a40db700 1 -- 172.21.15.32:0/3354952741 shutdown_connections 2021-04-09T18:33:01.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.424 7fa0a40db700 1 -- 172.21.15.32:0/3354952741 >> 172.21.15.32:0/3354952741 conn(0x7fa09c0ce5f0 msgr2=0x7fa09c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:01.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.424 7fa0a40db700 1 -- 172.21.15.32:0/3354952741 shutdown_connections 2021-04-09T18:33:01.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.424 7fa0a40db700 1 -- 172.21.15.32:0/3354952741 wait complete. 2021-04-09T18:33:01.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.444 7fa0a40db700 1 Processor -- start 2021-04-09T18:33:01.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.444 7fa0a40db700 1 -- start start 2021-04-09T18:33:01.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a40db700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c0b5360 0x7fa09c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:01.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a40db700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c18e140 0x7fa09c1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:01.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a40db700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c1ad040 0x7fa09c08fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:01.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a40db700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c1a5360 con 0x7fa09c18e140 2021-04-09T18:33:01.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a40db700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0902c0 con 0x7fa09c1ad040 2021-04-09T18:33:01.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a40db700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0904e0 con 0x7fa09c0b5360 2021-04-09T18:33:01.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a16f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c18e140 0x7fa09c1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:01.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a16f6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c18e140 0x7fa09c1aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58310/0 (socket says 172.21.15.32:58310) 2021-04-09T18:33:01.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a16f6700 1 -- 172.21.15.32:0/1075366007 learned_addr learned my addr 172.21.15.32:0/1075366007 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:01.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a1ef7700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c0b5360 0x7fa09c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:01.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a26f8700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c1ad040 0x7fa09c08fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:01.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a16f6700 1 -- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c0b5360 msgr2=0x7fa09c1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:01.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a16f6700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c0b5360 0x7fa09c1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a16f6700 1 -- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c1ad040 msgr2=0x7fa09c08fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:01.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a16f6700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c1ad040 0x7fa09c08fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a16f6700 1 -- 172.21.15.32:0/1075366007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c183c80 con 0x7fa09c18e140 2021-04-09T18:33:01.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a16f6700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c18e140 0x7fa09c1aca20 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7fa08c0059f0 tx=0x7fa08c00a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:01.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0927fc700 1 -- 172.21.15.32:0/1075366007 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c00ac90 con 0x7fa09c18e140 2021-04-09T18:33:01.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0927fc700 1 -- 172.21.15.32:0/1075366007 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c000c10 con 0x7fa09c18e140 2021-04-09T18:33:01.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c183c80 con 0x7fa09c18e140 2021-04-09T18:33:01.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.445 7fa0927fc700 1 -- 172.21.15.32:0/1075366007 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa08c0022b0 con 0x7fa09c18e140 2021-04-09T18:33:01.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.446 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c183c80 con 0x7fa09c18e140 2021-04-09T18:33:01.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.446 7fa0927fc700 1 -- 172.21.15.32:0/1075366007 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa08c006760 con 0x7fa09c18e140 2021-04-09T18:33:01.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.446 7fa0927fc700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa084018930 0x7fa08401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:01.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.446 7fa0927fc700 1 -- 172.21.15.32:0/1075366007 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6647+0+0 (secure 0 0 0) 0x7fa08c032e40 con 0x7fa09c18e140 2021-04-09T18:33:01.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.446 7fa0a1ef7700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa084018930 0x7fa08401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:01.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.447 7fa0a1ef7700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa084018930 0x7fa08401ae10 crc :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:01.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.465 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09c182d70 con 0x7fa09c18e140 2021-04-09T18:33:01.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.469 7fa0927fc700 1 -- 172.21.15.32:0/1075366007 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa08c000c10 con 0x7fa09c18e140 2021-04-09T18:33:01.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.660 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa09c10d480 con 0x7fa09c18e140 2021-04-09T18:33:01.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.661 7fa0927fc700 1 -- 172.21.15.32:0/1075366007 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 392 v392) v1 ==== 95+0+3381 (secure 0 0 0) 0x7fa08c000e20 con 0x7fa09c18e140 2021-04-09T18:33:01.677 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 392 2021-04-09T18:33:01.678 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:01.678 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":392,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":392,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:01.205098","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:replay","state_seq":2,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:01.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa084018930 msgr2=0x7fa08401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:01.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa084018930 0x7fa08401ae10 crc :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c18e140 msgr2=0x7fa09c1aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:01.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c18e140 0x7fa09c1aca20 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7fa08c0059f0 tx=0x7fa08c00a8a0).stop 2021-04-09T18:33:01.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 shutdown_connections 2021-04-09T18:33:01.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa084018930 0x7fa08401ae10 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa09c18e140 0x7fa09c1aca20 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa09c1ad040 0x7fa09c08fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 --2- 172.21.15.32:0/1075366007 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa09c0b5360 0x7fa09c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:01.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 >> 172.21.15.32:0/1075366007 conn(0x7fa09c19b460 msgr2=0x7fa09c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:01.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 shutdown_connections 2021-04-09T18:33:01.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:01.663 7fa0a40db700 1 -- 172.21.15.32:0/1075366007 wait complete. 2021-04-09T18:33:01.689 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting... 2021-04-09T18:33:06.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.836 7fecbcc80700 1 Processor -- start 2021-04-09T18:33:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.836 7fecbcc80700 1 -- start start 2021-04-09T18:33:06.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.836 7fecbcc80700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fecb810d010 0x7fecb80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:06.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.836 7fecbcc80700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fecb80d1da0 0x7fecb80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:06.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.836 7fecbcc80700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80d82c0 0x7fecb810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:06.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.836 7fecbcc80700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb80d8d30 con 0x7fecb810d010 2021-04-09T18:33:06.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.836 7fecbcc80700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb80d9220 con 0x7fecb80d1da0 2021-04-09T18:33:06.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.836 7fecbcc80700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb810d4b0 con 0x7fecb80d82c0 2021-04-09T18:33:06.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb685e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fecb810d010 0x7fecb80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:06.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb685e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fecb810d010 0x7fecb80d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58320/0 (socket says 172.21.15.32:58320) 2021-04-09T18:33:06.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb685e700 1 -- 172.21.15.32:0/2959212739 learned_addr learned my addr 172.21.15.32:0/2959212739 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:06.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb605d700 1 --2- 172.21.15.32:0/2959212739 >> v2:172.21.15.86:3300/0 conn(0x7fecb80d1da0 0x7fecb80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:06.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb705f700 1 --2- 172.21.15.32:0/2959212739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80d82c0 0x7fecb810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:06.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb685e700 1 -- 172.21.15.32:0/2959212739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80d82c0 msgr2=0x7fecb810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:06.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb685e700 1 --2- 172.21.15.32:0/2959212739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80d82c0 0x7fecb810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:06.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb685e700 1 -- 172.21.15.32:0/2959212739 >> v2:172.21.15.86:3300/0 conn(0x7fecb80d1da0 msgr2=0x7fecb80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb685e700 1 --2- 172.21.15.32:0/2959212739 >> v2:172.21.15.86:3300/0 conn(0x7fecb80d1da0 0x7fecb80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb685e700 1 -- 172.21.15.32:0/2959212739 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb80d86c0 con 0x7fecb810d010 2021-04-09T18:33:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb605d700 1 --2- 172.21.15.32:0/2959212739 >> v2:172.21.15.86:3300/0 conn(0x7fecb80d1da0 0x7fecb80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb685e700 1 --2- 172.21.15.32:0/2959212739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb810d010 0x7fecb80d1780 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 rx=0x7feca8006700 tx=0x7feca8009b00).ready entity=mon.0 client_cookie=92ecb9749b569b6d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:06.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb585c700 1 -- 172.21.15.32:0/2959212739 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feca800c0c0 con 0x7fecb810d010 2021-04-09T18:33:06.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb585c700 1 -- 172.21.15.32:0/2959212739 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca800c400 con 0x7fecb810d010 2021-04-09T18:33:06.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecb585c700 1 -- 172.21.15.32:0/2959212739 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feca800c8e0 con 0x7fecb810d010 2021-04-09T18:33:06.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecbcc80700 1 -- 172.21.15.32:0/2959212739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb810d010 msgr2=0x7fecb80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.837 7fecbcc80700 1 --2- 172.21.15.32:0/2959212739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb810d010 0x7fecb80d1780 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 rx=0x7feca8006700 tx=0x7feca8009b00).stop 2021-04-09T18:33:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.838 7fecbcc80700 1 -- 172.21.15.32:0/2959212739 shutdown_connections 2021-04-09T18:33:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.838 7fecbcc80700 1 --2- 172.21.15.32:0/2959212739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80d82c0 0x7fecb810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.838 7fecbcc80700 1 --2- 172.21.15.32:0/2959212739 >> v2:172.21.15.86:3300/0 conn(0x7fecb80d1da0 0x7fecb80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.838 7fecbcc80700 1 --2- 172.21.15.32:0/2959212739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb810d010 0x7fecb80d1780 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:06.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.838 7fecbcc80700 1 -- 172.21.15.32:0/2959212739 >> 172.21.15.32:0/2959212739 conn(0x7fecb80ce5f0 msgr2=0x7fecb80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:06.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.838 7fecbcc80700 1 -- 172.21.15.32:0/2959212739 shutdown_connections 2021-04-09T18:33:06.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.838 7fecbcc80700 1 -- 172.21.15.32:0/2959212739 wait complete. 2021-04-09T18:33:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.858 7fecbcc80700 1 Processor -- start 2021-04-09T18:33:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.858 7fecbcc80700 1 -- start start 2021-04-09T18:33:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecbcc80700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb8180f10 0x7fecb8181310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecbcc80700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb817a190 0x7fecb817a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:06.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecbcc80700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb817abb0 0x7fecb817fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:06.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecbcc80700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb808fb80 con 0x7fecb8180f10 2021-04-09T18:33:06.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecbcc80700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb8181930 con 0x7fecb817abb0 2021-04-09T18:33:06.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecbcc80700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb8184020 con 0x7fecb817a190 2021-04-09T18:33:06.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecb685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb8180f10 0x7fecb8181310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:06.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecb685e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb8180f10 0x7fecb8181310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58328/0 (socket says 172.21.15.32:58328) 2021-04-09T18:33:06.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecb685e700 1 -- 172.21.15.32:0/3354605340 learned_addr learned my addr 172.21.15.32:0/3354605340 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:06.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecb705f700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb817abb0 0x7fecb817fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:06.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.859 7fecb605d700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb817a190 0x7fecb817a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:06.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecb685e700 1 -- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb817a190 msgr2=0x7fecb817a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:06.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecb685e700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb817a190 0x7fecb817a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:06.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecb685e700 1 -- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb817abb0 msgr2=0x7fecb817fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:06.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecb685e700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb817abb0 0x7fecb817fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:06.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecb685e700 1 -- 172.21.15.32:0/3354605340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb8180070 con 0x7fecb8180f10 2021-04-09T18:33:06.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecb685e700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb8180f10 0x7fecb8181310 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 rx=0x7feca8006f20 tx=0x7feca800b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:06.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecaf7fe700 1 -- 172.21.15.32:0/3354605340 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feca800bdd0 con 0x7fecb8180f10 2021-04-09T18:33:06.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecaf7fe700 1 -- 172.21.15.32:0/3354605340 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca8006440 con 0x7fecb8180f10 2021-04-09T18:33:06.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb8180070 con 0x7fecb8180f10 2021-04-09T18:33:06.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecaf7fe700 1 -- 172.21.15.32:0/3354605340 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feca80052e0 con 0x7fecb8180f10 2021-04-09T18:33:06.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.860 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb8180070 con 0x7fecb8180f10 2021-04-09T18:33:06.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.861 7fecaf7fe700 1 -- 172.21.15.32:0/3354605340 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7feca80052e0 con 0x7fecb8180f10 2021-04-09T18:33:06.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.861 7fecaf7fe700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c0188d0 0x7fec9c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:06.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.861 7fecaf7fe700 1 -- 172.21.15.32:0/3354605340 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6647+0+0 (secure 0 0 0) 0x7feca8032e30 con 0x7fecb8180f10 2021-04-09T18:33:06.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.861 7fecb605d700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c0188d0 0x7fec9c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:06.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.861 7fecb605d700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c0188d0 0x7fec9c01add0 crc :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:06.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.880 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb8181930 con 0x7fecb8180f10 2021-04-09T18:33:06.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:06.884 7fecaf7fe700 1 -- 172.21.15.32:0/3354605340 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7feca8006440 con 0x7fecb8180f10 2021-04-09T18:33:07.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.075 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecb8181930 con 0x7fecb8180f10 2021-04-09T18:33:07.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.075 7fecaf7fe700 1 -- 172.21.15.32:0/3354605340 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 395 v395) v1 ==== 95+0+3381 (secure 0 0 0) 0x7feca800bdd0 con 0x7fecb8180f10 2021-04-09T18:33:07.092 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 395 2021-04-09T18:33:07.092 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:07.092 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":395,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:04.212398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:07.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.077 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c0188d0 msgr2=0x7fec9c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.077 7fecbcc80700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c0188d0 0x7fec9c01add0 crc :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.077 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb8180f10 msgr2=0x7fecb8181310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.077 7fecbcc80700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb8180f10 0x7fecb8181310 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 rx=0x7feca8006f20 tx=0x7feca800b9e0).stop 2021-04-09T18:33:07.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.078 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 shutdown_connections 2021-04-09T18:33:07.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.078 7fecbcc80700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec9c0188d0 0x7fec9c01add0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.078 7fecbcc80700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb817abb0 0x7fecb817fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.078 7fecbcc80700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb817a190 0x7fecb817a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.078 7fecbcc80700 1 --2- 172.21.15.32:0/3354605340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecb8180f10 0x7fecb8181310 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.078 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 >> 172.21.15.32:0/3354605340 conn(0x7fecb8192570 msgr2=0x7fecb81917c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:07.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.078 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 shutdown_connections 2021-04-09T18:33:07.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.078 7fecbcc80700 1 -- 172.21.15.32:0/3354605340 wait complete. 2021-04-09T18:33:07.104 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:07.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.237 7fb349f79700 1 Processor -- start 2021-04-09T18:33:07.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.237 7fb349f79700 1 -- start start 2021-04-09T18:33:07.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.237 7fb349f79700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3440d6c80 0x7fb3440d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.237 7fb349f79700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3440d2440 0x7fb34410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.237 7fb349f79700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440d7280 con 0x7fb3440cffa0 2021-04-09T18:33:07.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.237 7fb349f79700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3440d7080 con 0x7fb3440d2440 2021-04-09T18:33:07.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.237 7fb349f79700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440d91e0 con 0x7fb3440d6c80 2021-04-09T18:33:07.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb343fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3440d2440 0x7fb34410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb342ffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3440d6c80 0x7fb3440d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb343fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3440d2440 0x7fb34410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:50094/0 (socket says 172.21.15.32:50094) 2021-04-09T18:33:07.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb3437fe700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb3440cffa0 0x7fb3440d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37618/0 (socket says 172.21.15.32:37618) 2021-04-09T18:33:07.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb343fff700 1 -- 172.21.15.32:0/655147445 learned_addr learned my addr 172.21.15.32:0/655147445 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:07.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb343fff700 1 -- 172.21.15.32:0/655147445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3440d6c80 msgr2=0x7fb3440d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:07.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb343fff700 1 --2- 172.21.15.32:0/655147445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3440d6c80 0x7fb3440d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb343fff700 1 -- 172.21.15.32:0/655147445 >> v1:172.21.15.86:6789/0 conn(0x7fb3440cffa0 legacy=0x7fb3440d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb343fff700 1 -- 172.21.15.32:0/655147445 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3440d2840 con 0x7fb3440d2440 2021-04-09T18:33:07.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb342ffd700 1 --2- 172.21.15.32:0/655147445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3440d6c80 0x7fb3440d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:07.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb343fff700 1 --2- 172.21.15.32:0/655147445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3440d2440 0x7fb34410cfd0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7fb33c006720 tx=0x7fb33c009ba0).ready entity=mon.1 client_cookie=1687c5017d0f0fb2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:07.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb3427fc700 1 -- 172.21.15.32:0/655147445 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb33c0022e0 con 0x7fb3440d2440 2021-04-09T18:33:07.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb3427fc700 1 -- 172.21.15.32:0/655147445 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c002660 con 0x7fb3440d2440 2021-04-09T18:33:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.238 7fb3427fc700 1 -- 172.21.15.32:0/655147445 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb33c002b40 con 0x7fb3440d2440 2021-04-09T18:33:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.239 7fb349f79700 1 -- 172.21.15.32:0/655147445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3440d2440 msgr2=0x7fb34410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.239 7fb349f79700 1 --2- 172.21.15.32:0/655147445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3440d2440 0x7fb34410cfd0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7fb33c006720 tx=0x7fb33c009ba0).stop 2021-04-09T18:33:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.239 7fb349f79700 1 -- 172.21.15.32:0/655147445 shutdown_connections 2021-04-09T18:33:07.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.239 7fb349f79700 1 --2- 172.21.15.32:0/655147445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3440d2440 0x7fb34410cfd0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.239 7fb349f79700 1 --2- 172.21.15.32:0/655147445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3440d6c80 0x7fb3440d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.239 7fb349f79700 1 -- 172.21.15.32:0/655147445 >> 172.21.15.32:0/655147445 conn(0x7fb3440ce5c0 msgr2=0x7fb3440cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:07.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.239 7fb349f79700 1 -- 172.21.15.32:0/655147445 shutdown_connections 2021-04-09T18:33:07.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.239 7fb349f79700 1 -- 172.21.15.32:0/655147445 wait complete. 2021-04-09T18:33:07.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.259 7fb349f79700 1 Processor -- start 2021-04-09T18:33:07.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb349f79700 1 -- start start 2021-04-09T18:33:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb349f79700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34408ec80 0x7fb34408f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb349f79700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34408c950 0x7fb34408cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb349f79700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34408d560 0x7fb34408d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb349f79700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34408e700 con 0x7fb34408c950 2021-04-09T18:33:07.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb349f79700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34408f6a0 con 0x7fb34408d560 2021-04-09T18:33:07.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb349f79700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3441acff0 con 0x7fb34408ec80 2021-04-09T18:33:07.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb342ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34408c950 0x7fb34408cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb342ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34408c950 0x7fb34408cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58340/0 (socket says 172.21.15.32:58340) 2021-04-09T18:33:07.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb342ffd700 1 -- 172.21.15.32:0/1957538191 learned_addr learned my addr 172.21.15.32:0/1957538191 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:07.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb3437fe700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34408ec80 0x7fb34408f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.260 7fb343fff700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34408d560 0x7fb34408d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb342ffd700 1 -- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34408ec80 msgr2=0x7fb34408f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb342ffd700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34408ec80 0x7fb34408f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb342ffd700 1 -- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34408d560 msgr2=0x7fb34408d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb342ffd700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34408d560 0x7fb34408d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb342ffd700 1 -- 172.21.15.32:0/1957538191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34408df80 con 0x7fb34408c950 2021-04-09T18:33:07.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb342ffd700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34408c950 0x7fb34408cf40 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 rx=0x7fb32c006f40 tx=0x7fb32c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:07.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb33bfff700 1 -- 172.21.15.32:0/1957538191 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb32c00bd80 con 0x7fb34408c950 2021-04-09T18:33:07.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb33bfff700 1 -- 172.21.15.32:0/1957538191 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c000c90 con 0x7fb34408c950 2021-04-09T18:33:07.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb33bfff700 1 -- 172.21.15.32:0/1957538191 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb32c002220 con 0x7fb34408c950 2021-04-09T18:33:07.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb349f79700 1 -- 172.21.15.32:0/1957538191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34408df80 con 0x7fb34408c950 2021-04-09T18:33:07.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.261 7fb349f79700 1 -- 172.21.15.32:0/1957538191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34408df80 con 0x7fb34408c950 2021-04-09T18:33:07.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.262 7fb33bfff700 1 -- 172.21.15.32:0/1957538191 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb32c00a000 con 0x7fb34408c950 2021-04-09T18:33:07.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.262 7fb33bfff700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb328018930 0x7fb32801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.262 7fb33bfff700 1 -- 172.21.15.32:0/1957538191 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6647+0+0 (secure 0 0 0) 0x7fb32c032e40 con 0x7fb34408c950 2021-04-09T18:33:07.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.262 7fb3437fe700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb328018930 0x7fb32801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.262 7fb3437fe700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb328018930 0x7fb32801ae10 crc :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:07.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.281 7fb349f79700 1 -- 172.21.15.32:0/1957538191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb344182dc0 con 0x7fb34408c950 2021-04-09T18:33:07.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.284 7fb33bfff700 1 -- 172.21.15.32:0/1957538191 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb32c006b00 con 0x7fb34408c950 2021-04-09T18:33:07.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.475 7fb349f79700 1 -- 172.21.15.32:0/1957538191 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3440d7280 con 0x7fb34408c950 2021-04-09T18:33:07.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.476 7fb33bfff700 1 -- 172.21.15.32:0/1957538191 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 395 v395) v1 ==== 95+0+3381 (secure 0 0 0) 0x7fb32c00a000 con 0x7fb34408c950 2021-04-09T18:33:07.493 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 395 2021-04-09T18:33:07.493 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:07.493 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":395,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59510,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":386}],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:04.212398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:32:55.176447","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":231,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11731":{"gid":11731,"name":"d","rank":0,"incarnation":387,"state":"up:active","state_seq":5,"addr":"172.21.15.86:6837/1293489350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1293489350},{"type":"v1","addr":"172.21.15.86:6837","nonce":1293489350}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:07.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 -- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb328018930 msgr2=0x7fb32801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb328018930 0x7fb32801ae10 crc :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 -- 172.21.15.32:0/1957538191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34408c950 msgr2=0x7fb34408cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34408c950 0x7fb34408cf40 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 rx=0x7fb32c006f40 tx=0x7fb32c00b990).stop 2021-04-09T18:33:07.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 -- 172.21.15.32:0/1957538191 shutdown_connections 2021-04-09T18:33:07.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb328018930 0x7fb32801ae10 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb34408c950 0x7fb34408cf40 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb34408d560 0x7fb34408d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 --2- 172.21.15.32:0/1957538191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb34408ec80 0x7fb34408f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 -- 172.21.15.32:0/1957538191 >> 172.21.15.32:0/1957538191 conn(0x7fb34409abb0 msgr2=0x7fb3440a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:07.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.478 7fb349f79700 1 -- 172.21.15.32:0/1957538191 shutdown_connections 2021-04-09T18:33:07.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.479 7fb349f79700 1 -- 172.21.15.32:0/1957538191 wait complete. 2021-04-09T18:33:07.505 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:33:07.505 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:33:07.513 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:33:07.514 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2021-04-09T18:33:07.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.651 7f92226b2700 1 Processor -- start 2021-04-09T18:33:07.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.651 7f92226b2700 1 -- start start 2021-04-09T18:33:07.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.652 7f92226b2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c0cffb0 0x7f921c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.652 7f92226b2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921c0d7290 con 0x7f921c0d7810 2021-04-09T18:33:07.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.652 7f92226b2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921c10e7e0 con 0x7f921c0d6c90 2021-04-09T18:33:07.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.652 7f92226b2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921c0d7090 con 0x7f921c0cffb0 2021-04-09T18:33:07.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.652 7f9213fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f921c0d7810 0x7f921c0d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39706/0 (socket says 172.21.15.32:39706) 2021-04-09T18:33:07.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.652 7f9213fff700 1 -- 172.21.15.32:0/343889463 learned_addr learned my addr 172.21.15.32:0/343889463 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:07.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265215171 0 0) 0x7f9204000ce0 con 0x7f921c0d7810 2021-04-09T18:33:07.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921bfff700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c0cffb0 0x7f921c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9208002fe0 con 0x7f921c0d7810 2021-04-09T18:33:07.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1132768125 0 0) 0x7f9204000ed0 con 0x7f921c0d7810 2021-04-09T18:33:07.696 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c0cffb0 msgr2=0x7f921c0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:07.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c0cffb0 0x7f921c0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 >> v1:172.21.15.86:6789/0 conn(0x7f921c0d6c90 legacy=0x7f921c10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c0d8590 con 0x7f921c0d7810 2021-04-09T18:33:07.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f9204001600 con 0x7f921c0d7810 2021-04-09T18:33:07.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 >> v1:172.21.15.32:6789/0 conn(0x7f921c0d7810 legacy=0x7f921c0d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9208003a40 0x7f9208005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9208006790 0x7f9208008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9208009590 0x7f920800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920800c2d0 con 0x7f9208003a40 2021-04-09T18:33:07.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920800c4f0 con 0x7f9208006790 2021-04-09T18:33:07.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921bfff700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9208003a40 0x7f9208005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921b7fe700 1 -- 172.21.15.32:0/343889463 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9208050720 con 0x7f9208009590 2021-04-09T18:33:07.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f9213fff700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9208006790 0x7f9208008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f9220ccf700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9208009590 0x7f920800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921bfff700 1 -- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9208009590 msgr2=0x7f920800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:07.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921bfff700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9208009590 0x7f920800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921bfff700 1 -- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9208006790 msgr2=0x7f9208008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:07.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921bfff700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9208006790 0x7f9208008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921bfff700 1 -- 172.21.15.32:0/343889463 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9208050980 con 0x7f9208003a40 2021-04-09T18:33:07.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f9220ccf700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9208009590 0x7f920800ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:07.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.653 7f921bfff700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9208003a40 0x7f9208005ef0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7f920c006ac0 tx=0x7f920c009ab0).ready entity=mon.0 client_cookie=bd16395e07ca7e51 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:07.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f921b7fe700 1 -- 172.21.15.32:0/343889463 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920c00c0c0 con 0x7f9208003a40 2021-04-09T18:33:07.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f921b7fe700 1 -- 172.21.15.32:0/343889463 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920c00c3c0 con 0x7f9208003a40 2021-04-09T18:33:07.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f921b7fe700 1 -- 172.21.15.32:0/343889463 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920c00c8a0 con 0x7f9208003a40 2021-04-09T18:33:07.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f92226b2700 1 -- 172.21.15.32:0/343889463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9208003a40 msgr2=0x7f9208005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f92226b2700 1 --2- 172.21.15.32:0/343889463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9208003a40 0x7f9208005ef0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7f920c006ac0 tx=0x7f920c009ab0).stop 2021-04-09T18:33:07.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f9220ccf700 1 -- 172.21.15.32:0/343889463 reap_dead start 2021-04-09T18:33:07.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f92226b2700 1 -- 172.21.15.32:0/343889463 shutdown_connections 2021-04-09T18:33:07.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f92226b2700 1 -- 172.21.15.32:0/343889463 >> 172.21.15.32:0/343889463 conn(0x7f921c0ce5d0 msgr2=0x7f921c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:07.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f92226b2700 1 -- 172.21.15.32:0/343889463 shutdown_connections 2021-04-09T18:33:07.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.654 7f92226b2700 1 -- 172.21.15.32:0/343889463 wait complete. 2021-04-09T18:33:07.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.675 7f92226b2700 1 Processor -- start 2021-04-09T18:33:07.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.675 7f92226b2700 1 -- start start 2021-04-09T18:33:07.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.675 7f92226b2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f921c1aced0 0x7f921c0a2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.675 7f92226b2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c184e60 0x7f921c1ab590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.675 7f92226b2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c185260 0x7f921c1abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.675 7f92226b2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c1ad2d0 con 0x7f921c1aced0 2021-04-09T18:33:07.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.675 7f92226b2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c198590 con 0x7f921c184e60 2021-04-09T18:33:07.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.675 7f92226b2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921c198a50 con 0x7f921c185260 2021-04-09T18:33:07.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f921bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f921c1aced0 0x7f921c0a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f921bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f921c1aced0 0x7f921c0a2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58358/0 (socket says 172.21.15.32:58358) 2021-04-09T18:33:07.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f921bfff700 1 -- 172.21.15.32:0/1479925610 learned_addr learned my addr 172.21.15.32:0/1479925610 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:07.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f9213fff700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c184e60 0x7f921c1ab590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f9220ccf700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c185260 0x7f921c1abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f921bfff700 1 -- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c185260 msgr2=0x7f921c1abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f921bfff700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c185260 0x7f921c1abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f921bfff700 1 -- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c184e60 msgr2=0x7f921c1ab590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:07.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f921bfff700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c184e60 0x7f921c1ab590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:07.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f921bfff700 1 -- 172.21.15.32:0/1479925610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c1ac1d0 con 0x7f921c1aced0 2021-04-09T18:33:07.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f9220ccf700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c185260 0x7f921c1abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:07.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f921bfff700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f921c1aced0 0x7f921c0a2800 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 rx=0x7f920c0097a0 tx=0x7f920c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:07.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f92197fa700 1 -- 172.21.15.32:0/1479925610 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920c00bd60 con 0x7f921c1aced0 2021-04-09T18:33:07.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f92197fa700 1 -- 172.21.15.32:0/1479925610 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920c000c90 con 0x7f921c1aced0 2021-04-09T18:33:07.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f92197fa700 1 -- 172.21.15.32:0/1479925610 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920c002220 con 0x7f921c1aced0 2021-04-09T18:33:07.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f92226b2700 1 -- 172.21.15.32:0/1479925610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921c1ac1d0 con 0x7f921c1aced0 2021-04-09T18:33:07.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.676 7f92226b2700 1 -- 172.21.15.32:0/1479925610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921c1ac1d0 con 0x7f921c1aced0 2021-04-09T18:33:07.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.677 7f92197fa700 1 -- 172.21.15.32:0/1479925610 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f920c0052e0 con 0x7f921c1aced0 2021-04-09T18:33:07.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.677 7f92197fa700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9200018930 0x7f920001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.677 7f92197fa700 1 -- 172.21.15.32:0/1479925610 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f920c032e00 con 0x7f921c1aced0 2021-04-09T18:33:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.677 7f9213fff700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9200018930 0x7f920001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:07.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.677 7f9213fff700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9200018930 0x7f920001adc0 crc :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:07.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.696 7f92226b2700 1 -- 172.21.15.32:0/1479925610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921c182e40 con 0x7f921c1aced0 2021-04-09T18:33:07.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.700 7f92197fa700 1 -- 172.21.15.32:0/1479925610 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f920c000c90 con 0x7f921c1aced0 2021-04-09T18:33:07.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:07.890 7f92226b2700 1 -- 172.21.15.32:0/1479925610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f921c182e40 con 0x7f921c1aced0 2021-04-09T18:33:08.242 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:33:08.226 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:33:08.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.236 7f92197fa700 1 -- 172.21.15.32:0/1479925610 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 11731 v396) v1 ==== 96+0+0 (secure 0 0 0) 0x7f920c009140 con 0x7f921c1aced0 2021-04-09T18:33:08.270 INFO:teuthology.orchestra.run.smithi032.stderr:failed mds gid 11731 2021-04-09T18:33:08.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.256 7f92226b2700 1 -- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9200018930 msgr2=0x7f920001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.256 7f92226b2700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9200018930 0x7f920001adc0 crc :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.256 7f92226b2700 1 -- 172.21.15.32:0/1479925610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f921c1aced0 msgr2=0x7f921c0a2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.256 7f92226b2700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f921c1aced0 0x7f921c0a2800 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 rx=0x7f920c0097a0 tx=0x7f920c00b970).stop 2021-04-09T18:33:08.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.256 7f92226b2700 1 -- 172.21.15.32:0/1479925610 shutdown_connections 2021-04-09T18:33:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.257 7f92226b2700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9200018930 0x7f920001adc0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.257 7f92226b2700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f921c185260 0x7f921c1abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.257 7f92226b2700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f921c184e60 0x7f921c1ab590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.257 7f92226b2700 1 --2- 172.21.15.32:0/1479925610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f921c1aced0 0x7f921c0a2800 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.257 7f92226b2700 1 -- 172.21.15.32:0/1479925610 >> 172.21.15.32:0/1479925610 conn(0x7f921c1a8fa0 msgr2=0x7f921c093b90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:08.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.257 7f92226b2700 1 -- 172.21.15.32:0/1479925610 shutdown_connections 2021-04-09T18:33:08.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.257 7f92226b2700 1 -- 172.21.15.32:0/1479925610 wait complete. 2021-04-09T18:33:08.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.417 7fa02e3d0700 1 Processor -- start 2021-04-09T18:33:08.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.417 7fa02e3d0700 1 -- start start 2021-04-09T18:33:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.417 7fa02e3d0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa02810d010 0x7fa0280d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.417 7fa02e3d0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02810d480 con 0x7fa0280d8490 2021-04-09T18:33:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.417 7fa02e3d0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02810e1f0 con 0x7fa0280d1da0 2021-04-09T18:33:08.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.417 7fa02e3d0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0280d8d30 con 0x7fa02810d010 2021-04-09T18:33:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.417 7fa02c9ed700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0280d8490 0x7fa02810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39736/0 (socket says 172.21.15.32:39736) 2021-04-09T18:33:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.417 7fa02c9ed700 1 -- 172.21.15.32:0/3800007761 learned_addr learned my addr 172.21.15.32:0/3800007761 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa027fff700 1 --2- 172.21.15.32:0/3800007761 >> v2:172.21.15.86:3300/0 conn(0x7fa02810d010 0x7fa0280d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa0277fe700 1 -- 172.21.15.32:0/3800007761 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472450979 0 0) 0x7fa020000d30 con 0x7fa0280d8490 2021-04-09T18:33:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa0277fe700 1 -- 172.21.15.32:0/3800007761 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa014002fe0 con 0x7fa0280d8490 2021-04-09T18:33:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa0277fe700 1 -- 172.21.15.32:0/3800007761 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752947145 0 0) 0x7fa010000c60 con 0x7fa0280d1da0 2021-04-09T18:33:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa0277fe700 1 -- 172.21.15.32:0/3800007761 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa014003210 con 0x7fa0280d1da0 2021-04-09T18:33:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa0277fe700 1 -- 172.21.15.32:0/3800007761 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1450678771 0 0) 0x7fa020001460 con 0x7fa0280d8490 2021-04-09T18:33:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa027fff700 1 -- 172.21.15.32:0/3800007761 >> v1:172.21.15.86:6789/0 conn(0x7fa0280d1da0 legacy=0x7fa0280d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:08.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa027fff700 1 -- 172.21.15.32:0/3800007761 >> v1:172.21.15.32:6789/0 conn(0x7fa0280d8490 legacy=0x7fa02810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:08.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa027fff700 1 -- 172.21.15.32:0/3800007761 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0280d9220 con 0x7fa02810d010 2021-04-09T18:33:08.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa027fff700 1 --2- 172.21.15.32:0/3800007761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa02810d010 0x7fa0280d1780 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7fa018006650 tx=0x7fa018009a80).ready entity=mon.1 client_cookie=7b98aeee5ba86377 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:08.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa0277fe700 1 -- 172.21.15.32:0/3800007761 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0180009b0 con 0x7fa02810d010 2021-04-09T18:33:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa0277fe700 1 -- 172.21.15.32:0/3800007761 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01800c0c0 con 0x7fa02810d010 2021-04-09T18:33:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa0277fe700 1 -- 172.21.15.32:0/3800007761 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa01800c510 con 0x7fa02810d010 2021-04-09T18:33:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa02e3d0700 1 -- 172.21.15.32:0/3800007761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa02810d010 msgr2=0x7fa0280d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.418 7fa02e3d0700 1 --2- 172.21.15.32:0/3800007761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa02810d010 0x7fa0280d1780 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7fa018006650 tx=0x7fa018009a80).stop 2021-04-09T18:33:08.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.419 7fa02e3d0700 1 -- 172.21.15.32:0/3800007761 shutdown_connections 2021-04-09T18:33:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.419 7fa02e3d0700 1 --2- 172.21.15.32:0/3800007761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa02810d010 0x7fa0280d1780 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.419 7fa02e3d0700 1 -- 172.21.15.32:0/3800007761 >> 172.21.15.32:0/3800007761 conn(0x7fa0280ce5f0 msgr2=0x7fa0280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.419 7fa02e3d0700 1 -- 172.21.15.32:0/3800007761 shutdown_connections 2021-04-09T18:33:08.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.419 7fa02e3d0700 1 -- 172.21.15.32:0/3800007761 wait complete. 2021-04-09T18:33:08.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.439 7fa02e3d0700 1 Processor -- start 2021-04-09T18:33:08.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.439 7fa02e3d0700 1 -- start start 2021-04-09T18:33:08.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa02e3d0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0281a91e0 0x7fa0280a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:08.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa02e3d0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0281a95e0 0x7fa0280a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:08.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa02e3d0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0281ac850 0x7fa0281acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:08.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa02e3d0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0281ad270 con 0x7fa0281a91e0 2021-04-09T18:33:08.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa02e3d0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0280b7640 con 0x7fa0281ac850 2021-04-09T18:33:08.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa02e3d0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa028183c80 con 0x7fa0281a95e0 2021-04-09T18:33:08.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa027fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0281a91e0 0x7fa0280a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:08.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa027fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0281a91e0 0x7fa0280a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58382/0 (socket says 172.21.15.32:58382) 2021-04-09T18:33:08.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa027fff700 1 -- 172.21.15.32:0/1179374921 learned_addr learned my addr 172.21.15.32:0/1179374921 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:08.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa02c9ed700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0281ac850 0x7fa0281acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:08.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa01ffff700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0281a95e0 0x7fa0280a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:08.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa027fff700 1 -- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0281a95e0 msgr2=0x7fa0280a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:08.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa027fff700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0281a95e0 0x7fa0280a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa027fff700 1 -- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0281ac850 msgr2=0x7fa0281acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa027fff700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0281ac850 0x7fa0281acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa027fff700 1 -- 172.21.15.32:0/1179374921 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa028183ea0 con 0x7fa0281a91e0 2021-04-09T18:33:08.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.440 7fa027fff700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0281a91e0 0x7fa0280a4a60 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 rx=0x7fa018009630 tx=0x7fa01800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.441 7fa0257fa700 1 -- 172.21.15.32:0/1179374921 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0180093a0 con 0x7fa0281a91e0 2021-04-09T18:33:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.441 7fa0257fa700 1 -- 172.21.15.32:0/1179374921 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01800bd70 con 0x7fa0281a91e0 2021-04-09T18:33:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.441 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa028183ea0 con 0x7fa0281a91e0 2021-04-09T18:33:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.441 7fa0257fa700 1 -- 172.21.15.32:0/1179374921 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa018005560 con 0x7fa0281a91e0 2021-04-09T18:33:08.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.441 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa028183ea0 con 0x7fa0281a91e0 2021-04-09T18:33:08.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.441 7fa0257fa700 1 -- 172.21.15.32:0/1179374921 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa0180052e0 con 0x7fa0281a91e0 2021-04-09T18:33:08.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.442 7fa0257fa700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa00c0188d0 0x7fa00c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:08.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.442 7fa0257fa700 1 -- 172.21.15.32:0/1179374921 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6733+0+0 (secure 0 0 0) 0x7fa018032eb0 con 0x7fa0281a91e0 2021-04-09T18:33:08.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.442 7fa01ffff700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa00c0188d0 0x7fa00c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:08.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.442 7fa01ffff700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa00c0188d0 0x7fa00c01add0 crc :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:08.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.461 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa028182d70 con 0x7fa0281a91e0 2021-04-09T18:33:08.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.465 7fa0257fa700 1 -- 172.21.15.32:0/1179374921 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa01801f770 con 0x7fa0281a91e0 2021-04-09T18:33:08.559 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:33:08.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:33:08.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.657 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa02810d480 con 0x7fa0281a91e0 2021-04-09T18:33:08.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.658 7fa0257fa700 1 -- 172.21.15.32:0/1179374921 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 397 v397) v1 ==== 95+0+3043 (secure 0 0 0) 0x7fa01800bd70 con 0x7fa0281a91e0 2021-04-09T18:33:08.675 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 397 2021-04-09T18:33:08.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:08.676 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":397,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:04.212398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":397,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:33:08.237330","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":236,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":59510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_59510":{"gid":59510,"name":"c","rank":0,"incarnation":397,"state":"up:replay","state_seq":2,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:08.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa00c0188d0 msgr2=0x7fa00c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:08.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa00c0188d0 0x7fa00c01add0 crc :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0281a91e0 msgr2=0x7fa0280a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:08.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0281a91e0 0x7fa0280a4a60 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 rx=0x7fa018009630 tx=0x7fa01800b980).stop 2021-04-09T18:33:08.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 shutdown_connections 2021-04-09T18:33:08.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa00c0188d0 0x7fa00c01add0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0281ac850 0x7fa0281acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0281a95e0 0x7fa0280a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 --2- 172.21.15.32:0/1179374921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0281a91e0 0x7fa0280a4a60 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:08.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.660 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 >> 172.21.15.32:0/1179374921 conn(0x7fa02819b460 msgr2=0x7fa02819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:08.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.661 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 shutdown_connections 2021-04-09T18:33:08.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:08.661 7fa02e3d0700 1 -- 172.21.15.32:0/1179374921 wait complete. 2021-04-09T18:33:08.687 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting... 2021-04-09T18:33:13.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:13.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.833 7f1cd24e5700 1 Processor -- start 2021-04-09T18:33:13.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.833 7f1cd24e5700 1 -- start start 2021-04-09T18:33:13.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.833 7f1cd24e5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc10d010 0x7f1ccc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:13.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cd24e5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1ccc0d1da0 0x7f1ccc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:13.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cd24e5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ccc10d480 con 0x7f1ccc0d82c0 2021-04-09T18:33:13.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cd24e5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ccc10e780 con 0x7f1ccc0d1da0 2021-04-09T18:33:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cd24e5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc0d8d30 con 0x7f1ccc10d010 2021-04-09T18:33:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cc37fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1ccc0d1da0 0x7f1ccc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cd0b02700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1ccc0d82c0 0x7f1ccc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39750/0 (socket says 172.21.15.32:39750) 2021-04-09T18:33:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cd0b02700 1 -- 172.21.15.32:0/1688682478 learned_addr learned my addr 172.21.15.32:0/1688682478 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:13.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1ccbfff700 1 --2- 172.21.15.32:0/1688682478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc10d010 0x7f1ccc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:13.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1ccb7fe700 1 -- 172.21.15.32:0/1688682478 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857800467 0 0) 0x7f1cc4000d30 con 0x7f1ccc0d82c0 2021-04-09T18:33:13.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1ccb7fe700 1 -- 172.21.15.32:0/1688682478 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb8002fe0 con 0x7f1ccc0d82c0 2021-04-09T18:33:13.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cc37fe700 1 -- 172.21.15.32:0/1688682478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc10d010 msgr2=0x7f1ccc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:13.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cc37fe700 1 --2- 172.21.15.32:0/1688682478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc10d010 0x7f1ccc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cc37fe700 1 -- 172.21.15.32:0/1688682478 >> v1:172.21.15.32:6789/0 conn(0x7f1ccc0d82c0 legacy=0x7f1ccc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cc37fe700 1 -- 172.21.15.32:0/1688682478 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc0d9220 con 0x7f1ccc0d1da0 2021-04-09T18:33:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1ccbfff700 1 --2- 172.21.15.32:0/1688682478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc10d010 0x7f1ccc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cc37fe700 1 --2- 172.21.15.32:0/1688682478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0d1da0 0x7f1ccc0d7ca0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7f1cb4006650 tx=0x7f1cb4009a80).ready entity=mon.0 client_cookie=31d142327b5db07f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:13.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1ccb7fe700 1 -- 172.21.15.32:0/1688682478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cb40009b0 con 0x7f1ccc0d1da0 2021-04-09T18:33:13.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1ccb7fe700 1 -- 172.21.15.32:0/1688682478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb400c0c0 con 0x7f1ccc0d1da0 2021-04-09T18:33:13.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1ccb7fe700 1 -- 172.21.15.32:0/1688682478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cb400c510 con 0x7f1ccc0d1da0 2021-04-09T18:33:13.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.834 7f1cd24e5700 1 -- 172.21.15.32:0/1688682478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0d1da0 msgr2=0x7f1ccc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:13.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.835 7f1cd24e5700 1 --2- 172.21.15.32:0/1688682478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0d1da0 0x7f1ccc0d7ca0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7f1cb4006650 tx=0x7f1cb4009a80).stop 2021-04-09T18:33:13.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.835 7f1cd24e5700 1 -- 172.21.15.32:0/1688682478 shutdown_connections 2021-04-09T18:33:13.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.835 7f1cd24e5700 1 --2- 172.21.15.32:0/1688682478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc0d1da0 0x7f1ccc0d7ca0 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:13.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.835 7f1cd24e5700 1 --2- 172.21.15.32:0/1688682478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc10d010 0x7f1ccc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:13.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.835 7f1cd24e5700 1 -- 172.21.15.32:0/1688682478 >> 172.21.15.32:0/1688682478 conn(0x7f1ccc0ce5f0 msgr2=0x7f1ccc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:13.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.835 7f1cd24e5700 1 -- 172.21.15.32:0/1688682478 shutdown_connections 2021-04-09T18:33:13.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.835 7f1cd24e5700 1 -- 172.21.15.32:0/1688682478 wait complete. 2021-04-09T18:33:13.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.855 7f1cd24e5700 1 Processor -- start 2021-04-09T18:33:13.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.855 7f1cd24e5700 1 -- start start 2021-04-09T18:33:13.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1cd24e5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc1a5210 0x7f1ccc1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:13.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1cd24e5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ccc1a6340 0x7f1ccc1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:13.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1cd24e5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc0abaa0 0x7f1ccc0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:13.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1cd24e5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc1a6d60 con 0x7f1ccc1a5210 2021-04-09T18:33:13.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1cd24e5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc0ac4c0 con 0x7f1ccc1a6340 2021-04-09T18:33:13.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1cd24e5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc0ac6e0 con 0x7f1ccc0abaa0 2021-04-09T18:33:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1ccbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc1a5210 0x7f1ccc1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1ccbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc1a5210 0x7f1ccc1a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58396/0 (socket says 172.21.15.32:58396) 2021-04-09T18:33:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1ccbfff700 1 -- 172.21.15.32:0/3452039478 learned_addr learned my addr 172.21.15.32:0/3452039478 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:13.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1cc37fe700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ccc1a6340 0x7f1ccc1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.856 7f1cd0b02700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc0abaa0 0x7f1ccc0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1ccbfff700 1 -- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc0abaa0 msgr2=0x7f1ccc0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1ccbfff700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc0abaa0 0x7f1ccc0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1ccbfff700 1 -- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ccc1a6340 msgr2=0x7f1ccc1a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:13.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1ccbfff700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ccc1a6340 0x7f1ccc1a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1ccbfff700 1 -- 172.21.15.32:0/3452039478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc17a720 con 0x7f1ccc1a5210 2021-04-09T18:33:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1ccbfff700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc1a5210 0x7f1ccc1a56c0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 rx=0x7f1cbc006f20 tx=0x7f1cbc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:13.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1cc97fa700 1 -- 172.21.15.32:0/3452039478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cbc00bd80 con 0x7f1ccc1a5210 2021-04-09T18:33:13.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1cc97fa700 1 -- 172.21.15.32:0/3452039478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cbc000c90 con 0x7f1ccc1a5210 2021-04-09T18:33:13.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1cc97fa700 1 -- 172.21.15.32:0/3452039478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cbc0022b0 con 0x7f1ccc1a5210 2021-04-09T18:33:13.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ccc17a720 con 0x7f1ccc1a5210 2021-04-09T18:33:13.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.857 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ccc17a720 con 0x7f1ccc1a5210 2021-04-09T18:33:13.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.858 7f1cc97fa700 1 -- 172.21.15.32:0/3452039478 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1cbc00a000 con 0x7f1ccc1a5210 2021-04-09T18:33:13.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.858 7f1cc97fa700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cb00188c0 0x7f1cb001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:13.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.858 7f1cc97fa700 1 -- 172.21.15.32:0/3452039478 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6733+0+0 (secure 0 0 0) 0x7f1cbc032c90 con 0x7f1ccc1a5210 2021-04-09T18:33:13.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.858 7f1cc37fe700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cb00188c0 0x7f1cb001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:13.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.858 7f1cc37fe700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cb00188c0 0x7f1cb001adc0 crc :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:13.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.878 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ccc1a6d60 con 0x7f1ccc1a5210 2021-04-09T18:33:13.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:13.881 7f1cc97fa700 1 -- 172.21.15.32:0/3452039478 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1cbc006bb0 con 0x7f1ccc1a5210 2021-04-09T18:33:14.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.071 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ccc1a6d60 con 0x7f1ccc1a5210 2021-04-09T18:33:14.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.072 7f1cc97fa700 1 -- 172.21.15.32:0/3452039478 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 400 v400) v1 ==== 95+0+3043 (secure 0 0 0) 0x7f1cbc00bd80 con 0x7f1ccc1a5210 2021-04-09T18:33:14.089 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 400 2021-04-09T18:33:14.090 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:14.090 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":400,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:04.212398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:33:11.245312","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":236,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":59510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_59510":{"gid":59510,"name":"c","rank":0,"incarnation":397,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:14.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cb00188c0 msgr2=0x7f1cb001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cb00188c0 0x7f1cb001adc0 crc :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc1a5210 msgr2=0x7f1ccc1a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc1a5210 0x7f1ccc1a56c0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 rx=0x7f1cbc006f20 tx=0x7f1cbc00b990).stop 2021-04-09T18:33:14.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 shutdown_connections 2021-04-09T18:33:14.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1cb00188c0 0x7f1cb001adc0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ccc0abaa0 0x7f1ccc0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ccc1a6340 0x7f1ccc1a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 --2- 172.21.15.32:0/3452039478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ccc1a5210 0x7f1ccc1a56c0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 >> 172.21.15.32:0/3452039478 conn(0x7f1ccc1a99c0 msgr2=0x7f1ccc17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:14.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 shutdown_connections 2021-04-09T18:33:14.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.074 7f1cd24e5700 1 -- 172.21.15.32:0/3452039478 wait complete. 2021-04-09T18:33:14.101 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.233 7fa41b0de700 1 Processor -- start 2021-04-09T18:33:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.233 7fa41b0de700 1 -- start start 2021-04-09T18:33:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa41b0de700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa41410d010 0x7fa4140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa41b0de700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4140d8490 0x7fa41410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa41b0de700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41410d480 con 0x7fa4140d1da0 2021-04-09T18:33:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa41b0de700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4140d8d30 con 0x7fa41410d010 2021-04-09T18:33:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa41b0de700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4140d9220 con 0x7fa4140d8490 2021-04-09T18:33:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa413fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa4140d1da0 0x7fa4140d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39764/0 (socket says 172.21.15.32:39764) 2021-04-09T18:33:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa413fff700 1 -- 172.21.15.32:0/1079037430 learned_addr learned my addr 172.21.15.32:0/1079037430 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa4196fb700 1 --2- 172.21.15.32:0/1079037430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4140d8490 0x7fa41410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa418efa700 1 --2- 172.21.15.32:0/1079037430 >> v2:172.21.15.86:3300/0 conn(0x7fa41410d010 0x7fa4140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa4137fe700 1 -- 172.21.15.32:0/1079037430 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703140847 0 0) 0x7fa408000ce0 con 0x7fa4140d1da0 2021-04-09T18:33:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa4137fe700 1 -- 172.21.15.32:0/1079037430 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3fc002fe0 con 0x7fa4140d1da0 2021-04-09T18:33:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa418efa700 1 -- 172.21.15.32:0/1079037430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4140d8490 msgr2=0x7fa41410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa418efa700 1 --2- 172.21.15.32:0/1079037430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4140d8490 0x7fa41410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa418efa700 1 -- 172.21.15.32:0/1079037430 >> v1:172.21.15.32:6789/0 conn(0x7fa4140d1da0 legacy=0x7fa4140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.234 7fa418efa700 1 -- 172.21.15.32:0/1079037430 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41410e020 con 0x7fa41410d010 2021-04-09T18:33:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.235 7fa418efa700 1 --2- 172.21.15.32:0/1079037430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa41410d010 0x7fa4140d1780 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7fa404006650 tx=0x7fa404009a00).ready entity=mon.1 client_cookie=adb5f3bcd254eb02 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:14.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.235 7fa4137fe700 1 -- 172.21.15.32:0/1079037430 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa404009df0 con 0x7fa41410d010 2021-04-09T18:33:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.235 7fa4137fe700 1 -- 172.21.15.32:0/1079037430 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4040009b0 con 0x7fa41410d010 2021-04-09T18:33:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.235 7fa4137fe700 1 -- 172.21.15.32:0/1079037430 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa40400c2b0 con 0x7fa41410d010 2021-04-09T18:33:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.235 7fa41b0de700 1 -- 172.21.15.32:0/1079037430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa41410d010 msgr2=0x7fa4140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.235 7fa41b0de700 1 --2- 172.21.15.32:0/1079037430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa41410d010 0x7fa4140d1780 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7fa404006650 tx=0x7fa404009a00).stop 2021-04-09T18:33:14.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.236 7fa41b0de700 1 -- 172.21.15.32:0/1079037430 shutdown_connections 2021-04-09T18:33:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.236 7fa41b0de700 1 --2- 172.21.15.32:0/1079037430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4140d8490 0x7fa41410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.236 7fa41b0de700 1 --2- 172.21.15.32:0/1079037430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa41410d010 0x7fa4140d1780 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.236 7fa41b0de700 1 -- 172.21.15.32:0/1079037430 >> 172.21.15.32:0/1079037430 conn(0x7fa4140ce5f0 msgr2=0x7fa4140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.236 7fa41b0de700 1 -- 172.21.15.32:0/1079037430 shutdown_connections 2021-04-09T18:33:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.236 7fa41b0de700 1 -- 172.21.15.32:0/1079037430 wait complete. 2021-04-09T18:33:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.256 7fa41b0de700 1 Processor -- start 2021-04-09T18:33:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.256 7fa41b0de700 1 -- start start 2021-04-09T18:33:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.256 7fa41b0de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4141aca20 0x7fa4141ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.256 7fa41b0de700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa41418bd40 0x7fa41418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa41b0de700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa41408fba0 0x7fa41408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa41b0de700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41418c7e0 con 0x7fa4141aca20 2021-04-09T18:33:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa41b0de700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41418ca30 con 0x7fa41408fba0 2021-04-09T18:33:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa41b0de700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4140905f0 con 0x7fa41418bd40 2021-04-09T18:33:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa418efa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4141aca20 0x7fa4141ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa418efa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4141aca20 0x7fa4141ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58408/0 (socket says 172.21.15.32:58408) 2021-04-09T18:33:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa418efa700 1 -- 172.21.15.32:0/3626744800 learned_addr learned my addr 172.21.15.32:0/3626744800 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa4196fb700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa41408fba0 0x7fa41408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa413fff700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa41418bd40 0x7fa41418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa418efa700 1 -- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa41418bd40 msgr2=0x7fa41418c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa418efa700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa41418bd40 0x7fa41418c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa418efa700 1 -- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa41408fba0 msgr2=0x7fa41408ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa418efa700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa41408fba0 0x7fa41408ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa418efa700 1 -- 172.21.15.32:0/3626744800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4140b4300 con 0x7fa4141aca20 2021-04-09T18:33:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa418efa700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4141aca20 0x7fa4141ace20 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 rx=0x7fa4040096a0 tx=0x7fa40400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa4117fa700 1 -- 172.21.15.32:0/3626744800 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4040093a0 con 0x7fa4141aca20 2021-04-09T18:33:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa4117fa700 1 -- 172.21.15.32:0/3626744800 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4040067f0 con 0x7fa4141aca20 2021-04-09T18:33:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4140b4300 con 0x7fa4141aca20 2021-04-09T18:33:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa4117fa700 1 -- 172.21.15.32:0/3626744800 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa404002220 con 0x7fa4141aca20 2021-04-09T18:33:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.257 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4140b4300 con 0x7fa4141aca20 2021-04-09T18:33:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.258 7fa4117fa700 1 -- 172.21.15.32:0/3626744800 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa404000c90 con 0x7fa4141aca20 2021-04-09T18:33:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.259 7fa4117fa700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa3f40188c0 0x7fa3f401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.259 7fa4117fa700 1 -- 172.21.15.32:0/3626744800 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6733+0+0 (secure 0 0 0) 0x7fa4040093a0 con 0x7fa4141aca20 2021-04-09T18:33:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.259 7fa413fff700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa3f40188c0 0x7fa3f401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.259 7fa413fff700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa3f40188c0 0x7fa3f401adc0 crc :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:14.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.278 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa414182d70 con 0x7fa4141aca20 2021-04-09T18:33:14.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.281 7fa4117fa700 1 -- 172.21.15.32:0/3626744800 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa404000c90 con 0x7fa4141aca20 2021-04-09T18:33:14.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.474 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa41418c760 con 0x7fa4141aca20 2021-04-09T18:33:14.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.475 7fa4117fa700 1 -- 172.21.15.32:0/3626744800 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 400 v400) v1 ==== 95+0+3043 (secure 0 0 0) 0x7fa40400bda0 con 0x7fa4141aca20 2021-04-09T18:33:14.493 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 400 2021-04-09T18:33:14.493 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:14.493 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":400,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:04.212398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:33:11.245312","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":236,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":59510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_59510":{"gid":59510,"name":"c","rank":0,"incarnation":397,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.477 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa3f40188c0 msgr2=0x7fa3f401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.477 7fa41b0de700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa3f40188c0 0x7fa3f401adc0 crc :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.477 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4141aca20 msgr2=0x7fa4141ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.477 7fa41b0de700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4141aca20 0x7fa4141ace20 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 rx=0x7fa4040096a0 tx=0x7fa40400b9b0).stop 2021-04-09T18:33:14.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.478 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 shutdown_connections 2021-04-09T18:33:14.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.478 7fa41b0de700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa3f40188c0 0x7fa3f401adc0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.478 7fa41b0de700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa41408fba0 0x7fa41408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.478 7fa41b0de700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa41418bd40 0x7fa41418c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.478 7fa41b0de700 1 --2- 172.21.15.32:0/3626744800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4141aca20 0x7fa4141ace20 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.478 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 >> 172.21.15.32:0/3626744800 conn(0x7fa41419b460 msgr2=0x7fa41419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:14.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.478 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 shutdown_connections 2021-04-09T18:33:14.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.478 7fa41b0de700 1 -- 172.21.15.32:0/3626744800 wait complete. 2021-04-09T18:33:14.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:14.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.637 7fa719431700 1 Processor -- start 2021-04-09T18:33:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.637 7fa719431700 1 -- start start 2021-04-09T18:33:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.637 7fa719431700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa71410d010 0x7fa7140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.637 7fa719431700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71410d480 con 0x7fa7140d8490 2021-04-09T18:33:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.637 7fa719431700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71410e1f0 con 0x7fa7140d1da0 2021-04-09T18:33:14.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.637 7fa719431700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7140d8d30 con 0x7fa71410d010 2021-04-09T18:33:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71387e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa7140d8490 0x7fa71410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39774/0 (socket says 172.21.15.32:39774) 2021-04-09T18:33:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71387e700 1 -- 172.21.15.32:0/2112140751 learned_addr learned my addr 172.21.15.32:0/2112140751 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71307d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa71410d010 0x7fa7140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71207b700 1 -- 172.21.15.32:0/2112140751 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187043791 0 0) 0x7fa70c000d30 con 0x7fa7140d8490 2021-04-09T18:33:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71207b700 1 -- 172.21.15.32:0/2112140751 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6fc002fe0 con 0x7fa7140d8490 2021-04-09T18:33:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71307d700 1 -- 172.21.15.32:0/2112140751 >> v1:172.21.15.86:6789/0 conn(0x7fa7140d1da0 legacy=0x7fa7140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71307d700 1 -- 172.21.15.32:0/2112140751 >> v1:172.21.15.32:6789/0 conn(0x7fa7140d8490 legacy=0x7fa71410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71307d700 1 -- 172.21.15.32:0/2112140751 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7140d9220 con 0x7fa71410d010 2021-04-09T18:33:14.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71307d700 1 --2- 172.21.15.32:0/2112140751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71410d010 0x7fa7140d1780 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 rx=0x7fa704006650 tx=0x7fa704009a80).ready entity=mon.0 client_cookie=b2c91dcc1864c58c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:14.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71207b700 1 -- 172.21.15.32:0/2112140751 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7040009b0 con 0x7fa71410d010 2021-04-09T18:33:14.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71207b700 1 -- 172.21.15.32:0/2112140751 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70400c0c0 con 0x7fa71410d010 2021-04-09T18:33:14.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa71207b700 1 -- 172.21.15.32:0/2112140751 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa70400c510 con 0x7fa71410d010 2021-04-09T18:33:14.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa719431700 1 -- 172.21.15.32:0/2112140751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71410d010 msgr2=0x7fa7140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.638 7fa719431700 1 --2- 172.21.15.32:0/2112140751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71410d010 0x7fa7140d1780 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 rx=0x7fa704006650 tx=0x7fa704009a80).stop 2021-04-09T18:33:14.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.639 7fa719431700 1 -- 172.21.15.32:0/2112140751 shutdown_connections 2021-04-09T18:33:14.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.639 7fa719431700 1 --2- 172.21.15.32:0/2112140751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71410d010 0x7fa7140d1780 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.639 7fa719431700 1 -- 172.21.15.32:0/2112140751 >> 172.21.15.32:0/2112140751 conn(0x7fa7140ce5f0 msgr2=0x7fa7140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:14.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.639 7fa719431700 1 -- 172.21.15.32:0/2112140751 shutdown_connections 2021-04-09T18:33:14.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.639 7fa719431700 1 -- 172.21.15.32:0/2112140751 wait complete. 2021-04-09T18:33:14.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.659 7fa719431700 1 Processor -- start 2021-04-09T18:33:14.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.659 7fa719431700 1 -- start start 2021-04-09T18:33:14.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa719431700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7141a7a20 0x7fa71417eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa719431700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71417e690 0x7fa714183440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa719431700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa714183a60 0x7fa71417fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa719431700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71417f500 con 0x7fa71417e690 2021-04-09T18:33:14.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa719431700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71417f160 con 0x7fa7141a7a20 2021-04-09T18:33:14.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa719431700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa714180020 con 0x7fa714183a60 2021-04-09T18:33:14.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71287c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71417e690 0x7fa714183440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71287c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71417e690 0x7fa714183440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58422/0 (socket says 172.21.15.32:58422) 2021-04-09T18:33:14.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71287c700 1 -- 172.21.15.32:0/1306232744 learned_addr learned my addr 172.21.15.32:0/1306232744 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:14.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71307d700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7141a7a20 0x7fa71417eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71387e700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa714183a60 0x7fa71417fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71287c700 1 -- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa714183a60 msgr2=0x7fa71417fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71287c700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa714183a60 0x7fa71417fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71287c700 1 -- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7141a7a20 msgr2=0x7fa71417eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71287c700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7141a7a20 0x7fa71417eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71287c700 1 -- 172.21.15.32:0/1306232744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa714180240 con 0x7fa71417e690 2021-04-09T18:33:14.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71307d700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7141a7a20 0x7fa71417eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:14.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71387e700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa714183a60 0x7fa71417fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:14.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa71287c700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71417e690 0x7fa714183440 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 rx=0x7fa70800b5b0 tx=0x7fa70800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:14.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.660 7fa6fbfff700 1 -- 172.21.15.32:0/1306232744 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa70800bd40 con 0x7fa71417e690 2021-04-09T18:33:14.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.661 7fa6fbfff700 1 -- 172.21.15.32:0/1306232744 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa708006240 con 0x7fa71417e690 2021-04-09T18:33:14.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.661 7fa6fbfff700 1 -- 172.21.15.32:0/1306232744 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa708005690 con 0x7fa71417e690 2021-04-09T18:33:14.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.661 7fa719431700 1 -- 172.21.15.32:0/1306232744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa714180240 con 0x7fa71417e690 2021-04-09T18:33:14.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.661 7fa719431700 1 -- 172.21.15.32:0/1306232744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa714180240 con 0x7fa71417e690 2021-04-09T18:33:14.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.661 7fa6fbfff700 1 -- 172.21.15.32:0/1306232744 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa7080054a0 con 0x7fa71417e690 2021-04-09T18:33:14.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.661 7fa6fbfff700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:14.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.661 7fa6fbfff700 1 -- 172.21.15.32:0/1306232744 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6733+0+0 (secure 0 0 0) 0x7fa708032eb0 con 0x7fa71417e690 2021-04-09T18:33:14.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.662 7fa71307d700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:14.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.662 7fa71307d700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 crc :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:14.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.682 7fa719431700 1 -- 172.21.15.32:0/1306232744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa71417f160 con 0x7fa71417e690 2021-04-09T18:33:14.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.685 7fa6fbfff700 1 -- 172.21.15.32:0/1306232744 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa708000c60 con 0x7fa71417e690 2021-04-09T18:33:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.878 7fa719431700 1 -- 172.21.15.32:0/1306232744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa71417f160 con 0x7fa71417e690 2021-04-09T18:33:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.878 7fa6fbfff700 1 -- 172.21.15.32:0/1306232744 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 400 v400) v1 ==== 95+0+3043 (secure 0 0 0) 0x7fa7080054a0 con 0x7fa71417e690 2021-04-09T18:33:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 400 2021-04-09T18:33:14.897 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:14.897 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":400,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:04.212398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:33:11.245312","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":236,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":59510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_59510":{"gid":59510,"name":"c","rank":0,"incarnation":397,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 -- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 msgr2=0x7fa6f401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 crc :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 -- 172.21.15.32:0/1306232744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71417e690 msgr2=0x7fa714183440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:14.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71417e690 0x7fa714183440 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 rx=0x7fa70800b5b0 tx=0x7fa70800b950).stop 2021-04-09T18:33:14.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 -- 172.21.15.32:0/1306232744 shutdown_connections 2021-04-09T18:33:14.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa6f4018930 0x7fa6f401ae10 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa71417e690 0x7fa714183440 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa714183a60 0x7fa71417fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 --2- 172.21.15.32:0/1306232744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7141a7a20 0x7fa71417eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:14.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 -- 172.21.15.32:0/1306232744 >> 172.21.15.32:0/1306232744 conn(0x7fa7141a7f10 msgr2=0x7fa7141ac280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:14.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 -- 172.21.15.32:0/1306232744 shutdown_connections 2021-04-09T18:33:14.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:14.881 7fa719431700 1 -- 172.21.15.32:0/1306232744 wait complete. 2021-04-09T18:33:14.908 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:33:14.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:33:14.912 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:33:14.912 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:33:14.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:33:14.915 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:33:14.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:15.061 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:33:15.062 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:33:15.062 INFO:tasks.ceph.mds.d.smithi086.stderr:2021-04-09 18:33:15.053 7fc81915c700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2021-04-09T18:33:15.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.048 7f1b8c068700 1 Processor -- start 2021-04-09T18:33:15.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.048 7f1b8c068700 1 -- start start 2021-04-09T18:33:15.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.048 7f1b8c068700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b840cffa0 0x7f1b840d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:15.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.048 7f1b8c068700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b840d7800 0x7f1b840d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:15.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.048 7f1b8c068700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1b840d8220 0x7f1b840d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.048 7f1b8c068700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b840d91e0 con 0x7f1b840cffa0 2021-04-09T18:33:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.048 7f1b8c068700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b840d7280 con 0x7f1b840d8220 2021-04-09T18:33:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.048 7f1b8c068700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840d6290 con 0x7f1b840d7800 2021-04-09T18:33:15.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89e84700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b840cffa0 0x7f1b840d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89e84700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1b840cffa0 0x7f1b840d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58428/0 (socket says 172.21.15.32:58428) 2021-04-09T18:33:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89e84700 1 -- 172.21.15.32:0/1929407563 learned_addr learned my addr 172.21.15.32:0/1929407563 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89683700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b840d7800 0x7f1b840d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b8a685700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1b840d8220 0x7f1b840d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89e84700 1 -- 172.21.15.32:0/1929407563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b840d7800 msgr2=0x7f1b840d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89e84700 1 --2- 172.21.15.32:0/1929407563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b840d7800 0x7f1b840d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89e84700 1 -- 172.21.15.32:0/1929407563 >> v2:172.21.15.86:3300/0 conn(0x7f1b840d8220 msgr2=0x7f1b840d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89e84700 1 --2- 172.21.15.32:0/1929407563 >> v2:172.21.15.86:3300/0 conn(0x7f1b840d8220 0x7f1b840d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89e84700 1 -- 172.21.15.32:0/1929407563 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840d64b0 con 0x7f1b840cffa0 2021-04-09T18:33:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b89e84700 1 --2- 172.21.15.32:0/1929407563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b840cffa0 0x7f1b840d2440 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 rx=0x7f1b80006700 tx=0x7f1b80009b00).ready entity=mon.0 client_cookie=2ac92556cc40bdb0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b88e82700 1 -- 172.21.15.32:0/1929407563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b8000c0c0 con 0x7f1b840cffa0 2021-04-09T18:33:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b88e82700 1 -- 172.21.15.32:0/1929407563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8000c400 con 0x7f1b840cffa0 2021-04-09T18:33:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b88e82700 1 -- 172.21.15.32:0/1929407563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b8000c8e0 con 0x7f1b840cffa0 2021-04-09T18:33:15.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b8c068700 1 -- 172.21.15.32:0/1929407563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b840cffa0 msgr2=0x7f1b840d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:15.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.049 7f1b8c068700 1 --2- 172.21.15.32:0/1929407563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b840cffa0 0x7f1b840d2440 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 rx=0x7f1b80006700 tx=0x7f1b80009b00).stop 2021-04-09T18:33:15.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.050 7f1b8c068700 1 -- 172.21.15.32:0/1929407563 shutdown_connections 2021-04-09T18:33:15.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.050 7f1b8c068700 1 --2- 172.21.15.32:0/1929407563 >> v2:172.21.15.86:3300/0 conn(0x7f1b840d8220 0x7f1b840d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.050 7f1b8c068700 1 --2- 172.21.15.32:0/1929407563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b840d7800 0x7f1b840d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.050 7f1b8c068700 1 --2- 172.21.15.32:0/1929407563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b840cffa0 0x7f1b840d2440 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.050 7f1b8c068700 1 -- 172.21.15.32:0/1929407563 >> 172.21.15.32:0/1929407563 conn(0x7f1b840ce5c0 msgr2=0x7f1b840cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:15.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.050 7f1b8c068700 1 -- 172.21.15.32:0/1929407563 shutdown_connections 2021-04-09T18:33:15.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.050 7f1b8c068700 1 -- 172.21.15.32:0/1929407563 wait complete. 2021-04-09T18:33:15.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.070 7f1b8c068700 1 Processor -- start 2021-04-09T18:33:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.070 7f1b8c068700 1 -- start start 2021-04-09T18:33:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b8c068700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b841acff0 0x7f1b841847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b8c068700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b84183fc0 0x7f1b841a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b8c068700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b84186ff0 0x7f1b841873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b8c068700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b84188c30 con 0x7f1b84183fc0 2021-04-09T18:33:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b8c068700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b841843c0 con 0x7f1b841acff0 2021-04-09T18:33:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b8c068700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b841a5c20 con 0x7f1b84186ff0 2021-04-09T18:33:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89683700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b84183fc0 0x7f1b841a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89683700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b84183fc0 0x7f1b841a5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58434/0 (socket says 172.21.15.32:58434) 2021-04-09T18:33:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89683700 1 -- 172.21.15.32:0/3715352114 learned_addr learned my addr 172.21.15.32:0/3715352114 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89e84700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b841acff0 0x7f1b841847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b8a685700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b84186ff0 0x7f1b841873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89683700 1 -- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b84186ff0 msgr2=0x7f1b841873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89683700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b84186ff0 0x7f1b841873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89683700 1 -- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b841acff0 msgr2=0x7f1b841847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:15.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89683700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b841acff0 0x7f1b841847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89683700 1 -- 172.21.15.32:0/3715352114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b84187a10 con 0x7f1b84183fc0 2021-04-09T18:33:15.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.071 7f1b89683700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b84183fc0 0x7f1b841a5600 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 rx=0x7f1b74006f60 tx=0x7f1b7400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:15.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.072 7f1b7a7fc700 1 -- 172.21.15.32:0/3715352114 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b7400bd60 con 0x7f1b84183fc0 2021-04-09T18:33:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.072 7f1b7a7fc700 1 -- 172.21.15.32:0/3715352114 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b74000c90 con 0x7f1b84183fc0 2021-04-09T18:33:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.072 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b84187a10 con 0x7f1b84183fc0 2021-04-09T18:33:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.072 7f1b7a7fc700 1 -- 172.21.15.32:0/3715352114 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b74002220 con 0x7f1b84183fc0 2021-04-09T18:33:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.072 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b84187a10 con 0x7f1b84183fc0 2021-04-09T18:33:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.072 7f1b7a7fc700 1 -- 172.21.15.32:0/3715352114 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1b74006b00 con 0x7f1b84183fc0 2021-04-09T18:33:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.072 7f1b7a7fc700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b6c018930 0x7f1b6c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.073 7f1b7a7fc700 1 -- 172.21.15.32:0/3715352114 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6733+0+0 (secure 0 0 0) 0x7f1b74000940 con 0x7f1b84183fc0 2021-04-09T18:33:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.073 7f1b89e84700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b6c018930 0x7f1b6c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.073 7f1b89e84700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b6c018930 0x7f1b6c01ae10 crc :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:15.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.093 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b84182dc0 con 0x7f1b84183fc0 2021-04-09T18:33:15.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.096 7f1b7a7fc700 1 -- 172.21.15.32:0/3715352114 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1b74000940 con 0x7f1b84183fc0 2021-04-09T18:33:15.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.291 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b840d7280 con 0x7f1b84183fc0 2021-04-09T18:33:15.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.291 7f1b7a7fc700 1 -- 172.21.15.32:0/3715352114 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 400 v400) v1 ==== 95+0+3043 (secure 0 0 0) 0x7f1b74006b00 con 0x7f1b84183fc0 2021-04-09T18:33:15.309 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 400 2021-04-09T18:33:15.310 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:15.310 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":400,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:04.212398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:33:11.245312","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":236,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":59510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_59510":{"gid":59510,"name":"c","rank":0,"incarnation":397,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:15.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b6c018930 msgr2=0x7f1b6c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:15.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b6c018930 0x7f1b6c01ae10 crc :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b84183fc0 msgr2=0x7f1b841a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:15.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b84183fc0 0x7f1b841a5600 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 rx=0x7f1b74006f60 tx=0x7f1b7400b970).stop 2021-04-09T18:33:15.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 shutdown_connections 2021-04-09T18:33:15.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1b6c018930 0x7f1b6c01ae10 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b84183fc0 0x7f1b841a5600 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b84186ff0 0x7f1b841873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 --2- 172.21.15.32:0/3715352114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b841acff0 0x7f1b841847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:15.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 >> 172.21.15.32:0/3715352114 conn(0x7f1b8409abb0 msgr2=0x7f1b840a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:15.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 shutdown_connections 2021-04-09T18:33:15.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:15.294 7f1b8c068700 1 -- 172.21.15.32:0/3715352114 wait complete. 2021-04-09T18:33:15.322 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-04-09T18:33:20.324 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.469 7fae1c28c700 1 Processor -- start 2021-04-09T18:33:20.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.469 7fae1c28c700 1 -- start start 2021-04-09T18:33:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.469 7fae1c28c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae140d7ca0 0x7fae140d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1c28c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae1410da00 0x7fae1410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1c28c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1410d480 con 0x7fae1410d010 2021-04-09T18:33:20.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1c28c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae140d9220 con 0x7fae1410da00 2021-04-09T18:33:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1c28c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae14112100 con 0x7fae140d7ca0 2021-04-09T18:33:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1a8a9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae1410da00 0x7fae1410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1a0a8700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fae1410d010 0x7fae140d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37724/0 (socket says 172.21.15.32:37724) 2021-04-09T18:33:20.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae198a7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae140d7ca0 0x7fae140d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:20.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1a0a8700 1 -- 172.21.15.32:0/1816489685 learned_addr learned my addr 172.21.15.32:0/1816489685 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:20.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1a8a9700 1 -- 172.21.15.32:0/1816489685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae140d7ca0 msgr2=0x7fae140d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:20.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1a8a9700 1 --2- 172.21.15.32:0/1816489685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae140d7ca0 0x7fae140d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1a8a9700 1 -- 172.21.15.32:0/1816489685 >> v1:172.21.15.86:6789/0 conn(0x7fae1410d010 legacy=0x7fae140d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae1a8a9700 1 -- 172.21.15.32:0/1816489685 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae140d86c0 con 0x7fae1410da00 2021-04-09T18:33:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.470 7fae198a7700 1 --2- 172.21.15.32:0/1816489685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae140d7ca0 0x7fae140d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.471 7fae1a8a9700 1 --2- 172.21.15.32:0/1816489685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae1410da00 0x7fae1410de00 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7fae0c006e50 tx=0x7fae0c009e70).ready entity=mon.1 client_cookie=9c896eeb937f4055 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.471 7fae190a6700 1 -- 172.21.15.32:0/1816489685 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae0c006a20 con 0x7fae1410da00 2021-04-09T18:33:20.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.471 7fae190a6700 1 -- 172.21.15.32:0/1816489685 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0c006790 con 0x7fae1410da00 2021-04-09T18:33:20.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.471 7fae190a6700 1 -- 172.21.15.32:0/1816489685 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae0c00b700 con 0x7fae1410da00 2021-04-09T18:33:20.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.471 7fae1c28c700 1 -- 172.21.15.32:0/1816489685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae1410da00 msgr2=0x7fae1410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:20.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.471 7fae1c28c700 1 --2- 172.21.15.32:0/1816489685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae1410da00 0x7fae1410de00 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7fae0c006e50 tx=0x7fae0c009e70).stop 2021-04-09T18:33:20.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.472 7fae1c28c700 1 -- 172.21.15.32:0/1816489685 shutdown_connections 2021-04-09T18:33:20.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.472 7fae1c28c700 1 --2- 172.21.15.32:0/1816489685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae1410da00 0x7fae1410de00 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.472 7fae1c28c700 1 --2- 172.21.15.32:0/1816489685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae140d7ca0 0x7fae140d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.472 7fae1c28c700 1 -- 172.21.15.32:0/1816489685 >> 172.21.15.32:0/1816489685 conn(0x7fae140ce5f0 msgr2=0x7fae140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.472 7fae1c28c700 1 -- 172.21.15.32:0/1816489685 shutdown_connections 2021-04-09T18:33:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.472 7fae1c28c700 1 -- 172.21.15.32:0/1816489685 wait complete. 2021-04-09T18:33:20.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1c28c700 1 Processor -- start 2021-04-09T18:33:20.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1c28c700 1 -- start start 2021-04-09T18:33:20.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1c28c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae140b42d0 0x7fae14183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:20.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1c28c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae141ac090 0x7fae141ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:20.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1c28c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae14184d30 0x7fae14185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:20.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1c28c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae140a1a90 con 0x7fae140b42d0 2021-04-09T18:33:20.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1c28c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae141acab0 con 0x7fae14184d30 2021-04-09T18:33:20.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1c28c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae14185a00 con 0x7fae141ac090 2021-04-09T18:33:20.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1a0a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae140b42d0 0x7fae14183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:20.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1a0a8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae140b42d0 0x7fae14183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58446/0 (socket says 172.21.15.32:58446) 2021-04-09T18:33:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae1a0a8700 1 -- 172.21.15.32:0/353792537 learned_addr learned my addr 172.21.15.32:0/353792537 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.492 7fae198a7700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae141ac090 0x7fae141ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae1a8a9700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae14184d30 0x7fae14185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae1a0a8700 1 -- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae141ac090 msgr2=0x7fae141ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:20.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae1a0a8700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae141ac090 0x7fae141ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae1a0a8700 1 -- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae14184d30 msgr2=0x7fae14185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:20.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae1a0a8700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae14184d30 0x7fae14185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae1a0a8700 1 -- 172.21.15.32:0/353792537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae14185750 con 0x7fae140b42d0 2021-04-09T18:33:20.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae1a0a8700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae140b42d0 0x7fae14183520 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 rx=0x7fae1000a540 tx=0x7fae1000a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:20.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae0affd700 1 -- 172.21.15.32:0/353792537 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae100070d0 con 0x7fae140b42d0 2021-04-09T18:33:20.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae0affd700 1 -- 172.21.15.32:0/353792537 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1000acd0 con 0x7fae140b42d0 2021-04-09T18:33:20.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae1c28c700 1 -- 172.21.15.32:0/353792537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae14185750 con 0x7fae140b42d0 2021-04-09T18:33:20.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae0affd700 1 -- 172.21.15.32:0/353792537 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae100045a0 con 0x7fae140b42d0 2021-04-09T18:33:20.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.493 7fae1c28c700 1 -- 172.21.15.32:0/353792537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae14185750 con 0x7fae140b42d0 2021-04-09T18:33:20.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.494 7fae0affd700 1 -- 172.21.15.32:0/353792537 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fae10007fe0 con 0x7fae140b42d0 2021-04-09T18:33:20.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.494 7fae0affd700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadfc0188d0 0x7fadfc01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:20.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.494 7fae0affd700 1 -- 172.21.15.32:0/353792537 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6733+0+0 (secure 0 0 0) 0x7fae10032e90 con 0x7fae140b42d0 2021-04-09T18:33:20.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.494 7fae198a7700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadfc0188d0 0x7fadfc01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:20.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.495 7fae198a7700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadfc0188d0 0x7fadfc01add0 crc :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:20.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.514 7fae1c28c700 1 -- 172.21.15.32:0/353792537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae14182d10 con 0x7fae140b42d0 2021-04-09T18:33:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.517 7fae0affd700 1 -- 172.21.15.32:0/353792537 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fae10000c10 con 0x7fae140b42d0 2021-04-09T18:33:20.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.711 7fae1c28c700 1 -- 172.21.15.32:0/353792537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae1410d480 con 0x7fae140b42d0 2021-04-09T18:33:20.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.711 7fae0affd700 1 -- 172.21.15.32:0/353792537 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 401 v401) v1 ==== 95+0+3720 (secure 0 0 0) 0x7fae10007fe0 con 0x7fae140b42d0 2021-04-09T18:33:20.730 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 401 2021-04-09T18:33:20.730 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:20.731 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":401,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2768006990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2768006990},{"type":"v1","addr":"172.21.15.86:6837","nonce":2768006990}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401},{"gid":59606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3794784835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3794784835},{"type":"v1","addr":"172.21.15.86:6836","nonce":3794784835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401}],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:04.212398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:33:11.245312","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":236,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":59510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_59510":{"gid":59510,"name":"c","rank":0,"incarnation":397,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:20.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 -- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadfc0188d0 msgr2=0x7fadfc01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:20.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadfc0188d0 0x7fadfc01add0 crc :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 -- 172.21.15.32:0/353792537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae140b42d0 msgr2=0x7fae14183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:20.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae140b42d0 0x7fae14183520 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 rx=0x7fae1000a540 tx=0x7fae1000a8e0).stop 2021-04-09T18:33:20.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 -- 172.21.15.32:0/353792537 shutdown_connections 2021-04-09T18:33:20.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fadfc0188d0 0x7fadfc01add0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae14184d30 0x7fae14185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae141ac090 0x7fae141ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 --2- 172.21.15.32:0/353792537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fae140b42d0 0x7fae14183520 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:20.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 -- 172.21.15.32:0/353792537 >> 172.21.15.32:0/353792537 conn(0x7fae1419b890 msgr2=0x7fae141a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:20.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 -- 172.21.15.32:0/353792537 shutdown_connections 2021-04-09T18:33:20.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:20.714 7fae1c28c700 1 -- 172.21.15.32:0/353792537 wait complete. 2021-04-09T18:33:20.741 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2021-04-09T18:33:20.742 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:33:20.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:33:20.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:33:20.865 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:33:20.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:20.866 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:33:20.867 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:33:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:33:20.911 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:33:20.911 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:33:20.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:33:20.912 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:33:20.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:20.994 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:33:20.994 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:33:21.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:33:21.038 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:33:21.039 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:33:21.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:33:21.040 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:33:21.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:33:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.252 7f440e02e700 1 Processor -- start 2021-04-09T18:33:21.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.252 7f440e02e700 1 -- start start 2021-04-09T18:33:21.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.252 7f440e02e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f440810d010 0x7f44080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.252 7f440e02e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d8490 0x7f440810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.252 7f440e02e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440810d480 con 0x7f44080d1da0 2021-04-09T18:33:21.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.252 7f440e02e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44080d8d30 con 0x7f440810d010 2021-04-09T18:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.252 7f440e02e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44080d9220 con 0x7f44080d8490 2021-04-09T18:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f44077fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f440810d010 0x7f44080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f43feffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f44080d1da0 0x7f44080d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39816/0 (socket says 172.21.15.32:39816) 2021-04-09T18:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f43feffd700 1 -- 172.21.15.32:0/4101610060 learned_addr learned my addr 172.21.15.32:0/4101610060 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:21.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f4407fff700 1 --2- 172.21.15.32:0/4101610060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d8490 0x7f440810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f44077fe700 1 -- 172.21.15.32:0/4101610060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d8490 msgr2=0x7f440810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f44077fe700 1 --2- 172.21.15.32:0/4101610060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d8490 0x7f440810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f44077fe700 1 -- 172.21.15.32:0/4101610060 >> v1:172.21.15.32:6789/0 conn(0x7f44080d1da0 legacy=0x7f44080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f44077fe700 1 -- 172.21.15.32:0/4101610060 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440810e020 con 0x7f440810d010 2021-04-09T18:33:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f4406ffd700 1 -- 172.21.15.32:0/4101610060 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568592559 0 0) 0x7f43f0000ce0 con 0x7f44080d1da0 2021-04-09T18:33:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f4407fff700 1 --2- 172.21.15.32:0/4101610060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d8490 0x7f440810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f44077fe700 1 --2- 172.21.15.32:0/4101610060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440810d010 0x7f44080d1780 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 rx=0x7f43f8006650 tx=0x7f43f8009a80).ready entity=mon.0 client_cookie=a229ad63b3c8bcca server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f4406ffd700 1 -- 172.21.15.32:0/4101610060 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f80009b0 con 0x7f440810d010 2021-04-09T18:33:21.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f4406ffd700 1 -- 172.21.15.32:0/4101610060 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f800c0c0 con 0x7f440810d010 2021-04-09T18:33:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f4406ffd700 1 -- 172.21.15.32:0/4101610060 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f800c510 con 0x7f440810d010 2021-04-09T18:33:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f440e02e700 1 -- 172.21.15.32:0/4101610060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440810d010 msgr2=0x7f44080d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.253 7f440e02e700 1 --2- 172.21.15.32:0/4101610060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440810d010 0x7f44080d1780 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 rx=0x7f43f8006650 tx=0x7f43f8009a80).stop 2021-04-09T18:33:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.254 7f440e02e700 1 -- 172.21.15.32:0/4101610060 shutdown_connections 2021-04-09T18:33:21.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.254 7f440e02e700 1 --2- 172.21.15.32:0/4101610060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44080d8490 0x7f440810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.254 7f440e02e700 1 --2- 172.21.15.32:0/4101610060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f440810d010 0x7f44080d1780 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.254 7f440e02e700 1 -- 172.21.15.32:0/4101610060 >> 172.21.15.32:0/4101610060 conn(0x7f44080ce5f0 msgr2=0x7f44080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.254 7f440e02e700 1 -- 172.21.15.32:0/4101610060 shutdown_connections 2021-04-09T18:33:21.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.254 7f440e02e700 1 -- 172.21.15.32:0/4101610060 wait complete. 2021-04-09T18:33:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.274 7f440e02e700 1 Processor -- start 2021-04-09T18:33:21.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f440e02e700 1 -- start start 2021-04-09T18:33:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f440e02e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44081a5210 0x7f4408185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f440e02e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408185a70 0x7f44080acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f440e02e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4408185e70 0x7f44080ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f440e02e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440818f7b0 con 0x7f4408185e70 2021-04-09T18:33:21.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f440e02e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44080ad7f0 con 0x7f44081a5210 2021-04-09T18:33:21.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f440e02e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44080ada10 con 0x7f4408185a70 2021-04-09T18:33:21.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f4407fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4408185e70 0x7f44080ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f44077fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44081a5210 0x7f4408185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f43feffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408185a70 0x7f44080acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f4407fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4408185e70 0x7f44080ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58462/0 (socket says 172.21.15.32:58462) 2021-04-09T18:33:21.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.275 7f4407fff700 1 -- 172.21.15.32:0/1369823128 learned_addr learned my addr 172.21.15.32:0/1369823128 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:21.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f4407fff700 1 -- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408185a70 msgr2=0x7f44080acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f4407fff700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408185a70 0x7f44080acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f4407fff700 1 -- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44081a5210 msgr2=0x7f4408185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f4407fff700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44081a5210 0x7f4408185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f4407fff700 1 -- 172.21.15.32:0/1369823128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4408179610 con 0x7f4408185e70 2021-04-09T18:33:21.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f44077fe700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44081a5210 0x7f4408185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:21.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f43feffd700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408185a70 0x7f44080acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:21.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f4407fff700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4408185e70 0x7f44080ad1d0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 rx=0x7f440000b970 tx=0x7f440000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:21.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f4404ff9700 1 -- 172.21.15.32:0/1369823128 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4400000f90 con 0x7f4408185e70 2021-04-09T18:33:21.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f4404ff9700 1 -- 172.21.15.32:0/1369823128 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4400003030 con 0x7f4408185e70 2021-04-09T18:33:21.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f440e02e700 1 -- 172.21.15.32:0/1369823128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4408179610 con 0x7f4408185e70 2021-04-09T18:33:21.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f4404ff9700 1 -- 172.21.15.32:0/1369823128 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4400001400 con 0x7f4408185e70 2021-04-09T18:33:21.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.276 7f440e02e700 1 -- 172.21.15.32:0/1369823128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4408179610 con 0x7f4408185e70 2021-04-09T18:33:21.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.277 7f4404ff9700 1 -- 172.21.15.32:0/1369823128 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4400000e70 con 0x7f4408185e70 2021-04-09T18:33:21.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.277 7f4404ff9700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.277 7f4404ff9700 1 -- 172.21.15.32:0/1369823128 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6733+0+0 (secure 0 0 0) 0x7f44000331f0 con 0x7f4408185e70 2021-04-09T18:33:21.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.277 7f44077fe700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.277 7f44077fe700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01ae10 crc :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:21.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.296 7f440e02e700 1 -- 172.21.15.32:0/1369823128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440818f7b0 con 0x7f4408185e70 2021-04-09T18:33:21.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.300 7f4404ff9700 1 -- 172.21.15.32:0/1369823128 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f44000066f0 con 0x7f4408185e70 2021-04-09T18:33:21.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.489 7f440e02e700 1 -- 172.21.15.32:0/1369823128 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f440818f7b0 con 0x7f4408185e70 2021-04-09T18:33:21.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.490 7f4404ff9700 1 -- 172.21.15.32:0/1369823128 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v236) v1 ==== 81+0+35898 (secure 0 0 0) 0x7f4400000df0 con 0x7f4408185e70 2021-04-09T18:33:21.510 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:21.510 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:33:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 236, 2021-04-09T18:33:21.511 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:33:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:33:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:33:07.891096", 2021-04-09T18:33:21.512 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:33:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:33:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:33:21.513 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:33:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:33:21.514 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:33:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:33:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:33:21.515 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:33:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:33:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:33:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:33:21.516 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:33:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:33:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:33:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 41, 2021-04-09T18:33:21.517 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:33:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:33:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:33:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:33:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:33:21.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:33:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:33:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:33:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:33:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:33:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:33:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:33:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:33:21.520 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:33:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:33:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:33:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:33:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:33:21.521 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:33:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:33:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:33:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:33:21.522 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:33:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:33:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:33:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:33:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:33:21.523 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:33:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:33:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:33:21.524 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:33:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:33:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:33:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:33:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:33:21.525 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:33:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:33:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:33:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:33:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:33:21.526 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:33:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:33:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:33:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:33:21.527 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:33:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:33:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:33:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:33:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:33:21.528 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:33:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:33:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:33:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:33:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:33:21.529 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:33:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:33:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:33:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:33:21.530 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:33:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:33:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:33:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:33:21.531 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:33:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:33:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:33:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:33:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:33:21.532 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:33:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:33:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.533 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 38, 2021-04-09T18:33:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "alpha_metadata", 2021-04-09T18:33:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:32:35.297513", 2021-04-09T18:33:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:33:21.534 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:33:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:33:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:33:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:33:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:33:21.535 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:33:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:33:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:33:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:33:21.536 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:33:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:33:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:33:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:33:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:33:21.537 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:33:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:33:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:33:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:33:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:33:21.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "225", 2021-04-09T18:33:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:33:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:33:21.539 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:33:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:33:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:33:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:33:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:33:21.540 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:33:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:33:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:33:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:33:21.541 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:33:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:33:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:33:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:33:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:33:21.542 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:33:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:33:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:33:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:33:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:33:21.543 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:33:21.544 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:33:21.544 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:33:21.544 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:33:21.544 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:33:21.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.545 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:33:21.545 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:33:21.545 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:33:21.545 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:33:21.546 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:33:21.546 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:33:21.546 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:33:21.546 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:33:21.546 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:33:21.547 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:33:21.547 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:33:21.547 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:33:21.547 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:33:21.548 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:33:21.548 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:33:21.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.548 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:33:21.548 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:33:21.549 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "alpha" 2021-04-09T18:33:21.549 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.549 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.549 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.550 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 39, 2021-04-09T18:33:21.550 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "alpha_data", 2021-04-09T18:33:21.550 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:32:36.169493", 2021-04-09T18:33:21.550 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:33:21.551 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:33:21.551 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:33:21.551 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:33:21.551 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:33:21.551 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:33:21.552 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:33:21.552 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:33:21.552 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:33:21.552 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:33:21.552 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:33:21.553 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:33:21.553 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:33:21.553 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:33:21.553 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:33:21.554 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:33:21.554 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:33:21.554 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:33:21.554 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:33:21.554 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:33:21.555 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.555 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "226", 2021-04-09T18:33:21.555 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:33:21.555 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:33:21.556 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:33:21.556 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:33:21.556 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:33:21.556 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:33:21.556 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:33:21.557 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:33:21.557 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:33:21.557 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:33:21.557 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:33:21.557 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:33:21.558 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:33:21.558 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:33:21.558 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:33:21.558 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:33:21.558 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:33:21.559 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:33:21.559 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:33:21.559 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:33:21.559 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:33:21.560 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:33:21.560 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:33:21.560 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:33:21.560 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:33:21.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:33:21.561 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.561 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:33:21.561 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:33:21.561 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:33:21.562 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:33:21.562 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:33:21.562 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:33:21.562 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:33:21.562 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:33:21.563 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:33:21.563 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:33:21.563 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:33:21.563 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:33:21.563 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:33:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:33:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "alpha" 2021-04-09T18:33:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.565 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 40, 2021-04-09T18:33:21.565 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "bravo_metadata", 2021-04-09T18:33:21.566 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:32:42.658486", 2021-04-09T18:33:21.566 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:33:21.566 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:33:21.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:33:21.566 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:33:21.567 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:33:21.567 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:33:21.567 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:33:21.567 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:33:21.567 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:33:21.568 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:33:21.568 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:33:21.568 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:33:21.568 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:33:21.569 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:33:21.569 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:33:21.569 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:33:21.569 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:33:21.569 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:33:21.570 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:33:21.570 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:33:21.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.570 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "229", 2021-04-09T18:33:21.570 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:33:21.571 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:33:21.571 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:33:21.571 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:33:21.571 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:33:21.572 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:33:21.572 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:33:21.572 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:33:21.572 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:33:21.572 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:33:21.573 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:33:21.573 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:33:21.573 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:33:21.573 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:33:21.573 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:33:21.574 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:33:21.574 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:33:21.574 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:33:21.574 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:33:21.575 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:33:21.575 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:33:21.575 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:33:21.575 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:33:21.575 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:33:21.576 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:33:21.576 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:33:21.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.576 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:33:21.577 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:33:21.577 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:33:21.577 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:33:21.577 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:33:21.577 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:33:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:33:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:33:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:33:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:33:21.578 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:33:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:33:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:33:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:33:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:33:21.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.580 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:33:21.580 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:33:21.580 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "bravo" 2021-04-09T18:33:21.580 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.581 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.581 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 41, 2021-04-09T18:33:21.581 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "bravo_data", 2021-04-09T18:33:21.582 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:32:43.872170", 2021-04-09T18:33:21.582 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:33:21.582 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:33:21.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:33:21.583 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:33:21.583 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:33:21.583 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:33:21.583 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:33:21.583 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:33:21.584 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:33:21.584 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:33:21.584 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:33:21.584 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:33:21.585 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:33:21.585 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:33:21.585 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:33:21.585 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:33:21.585 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:33:21.586 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:33:21.586 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:33:21.586 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:33:21.586 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.586 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "230", 2021-04-09T18:33:21.587 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:33:21.587 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:33:21.587 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:33:21.587 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:33:21.588 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:33:21.588 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:33:21.588 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:33:21.588 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:33:21.588 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:33:21.589 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:33:21.589 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:33:21.589 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:33:21.589 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:33:21.589 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:33:21.590 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:33:21.590 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:33:21.590 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:33:21.590 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:33:21.591 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:33:21.591 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:33:21.591 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:33:21.591 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:33:21.591 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:33:21.592 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:33:21.592 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:33:21.592 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:33:21.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.592 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:33:21.593 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:33:21.593 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:33:21.593 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:33:21.593 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:33:21.594 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:33:21.594 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:33:21.594 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:33:21.594 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:33:21.594 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:33:21.595 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:33:21.595 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:33:21.595 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:33:21.595 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:33:21.595 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "bravo" 2021-04-09T18:33:21.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.596 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:33:21.597 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:33:21.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.597 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:33:21.597 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:33:21.597 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:21.598 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:21.598 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:21.598 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:21.598 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:21.598 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:21.599 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:21.599 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 227, 2021-04-09T18:33:21.599 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:21.599 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:21.600 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:21.600 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.600 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:33:21.601 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:21.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.601 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.602 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:33:21.602 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:21.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.602 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.603 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:21.603 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.603 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.604 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:33:21.604 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:21.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.605 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:33:21.605 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:21.605 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.605 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.606 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:21.606 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.606 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.607 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:33:21.607 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:21.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.608 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.608 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:33:21.608 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:21.608 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.608 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.609 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:21.609 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.610 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:33:21.610 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:21.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.611 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:33:21.611 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:21.611 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.611 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.612 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:33:21.612 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:33:21.612 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:33:21.613 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:33:21.613 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:21.613 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:21.613 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:21.613 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.614 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:33:21.614 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:33:21.614 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:21.615 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:21.615 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:21.615 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:21.615 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:21.615 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:21.616 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:21.616 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 227, 2021-04-09T18:33:21.616 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:21.616 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:21.617 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:21.617 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.617 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:33:21.618 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:21.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.618 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.618 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:33:21.619 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:21.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.619 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.620 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:21.620 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.620 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.620 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.620 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:33:21.621 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:21.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.621 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.622 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:33:21.622 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:21.622 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.622 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.623 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:21.623 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.623 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.624 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:33:21.624 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:21.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.624 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.625 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:33:21.625 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:21.625 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.625 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.626 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:21.626 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.627 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:33:21.627 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:21.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.627 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.631 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:33:21.631 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:21.632 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.632 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.632 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:33:21.633 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:33:21.633 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:33:21.633 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:33:21.633 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:21.633 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:21.634 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:21.634 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.634 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.634 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:33:21.635 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:33:21.635 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:21.635 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:21.635 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:21.636 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:21.636 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:21.636 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:21.636 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:21.636 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:21.637 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:21.637 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:21.637 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:21.637 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.637 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.638 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:33:21.638 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:21.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:21.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:21.640 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:21.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:21.642 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:21.643 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:21.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:21.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:33:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.648 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.648 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:33:21.648 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:21.648 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.649 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.649 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.649 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:33:21.649 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:33:21.650 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:33:21.650 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:33:21.650 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:21.650 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:21.650 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:21.651 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.651 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.651 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:33:21.652 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:33:21.652 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:21.652 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:21.652 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:21.652 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:21.653 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:21.653 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:21.653 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:21.653 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:21.653 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:21.654 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:21.654 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:21.654 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.654 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:33:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.655 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.656 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:33:21.656 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:21.656 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.657 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.657 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:21.657 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.657 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.658 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:33:21.658 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:21.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.659 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.659 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:33:21.659 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:21.660 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.660 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.660 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:21.661 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.661 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.661 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.661 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:33:21.661 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:21.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.662 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.662 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.662 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:33:21.663 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:21.663 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.663 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.663 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:21.664 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.664 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.664 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:33:21.665 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:21.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.665 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.665 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.665 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:33:21.666 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:21.666 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.666 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.667 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:33:21.667 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:33:21.667 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:33:21.667 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:33:21.667 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:21.668 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:21.668 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:21.668 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.668 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.669 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:33:21.669 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:33:21.669 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:21.669 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:21.669 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:21.670 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:21.670 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:21.670 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:21.670 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:21.671 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:21.671 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:21.671 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:21.671 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:21.672 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.672 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:33:21.672 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:21.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.673 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.673 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.673 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:33:21.674 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:21.674 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.674 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.675 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:21.675 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.675 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.675 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.675 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:33:21.676 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:21.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.676 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.677 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:33:21.677 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:21.677 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.677 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.678 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:21.678 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.678 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.678 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.678 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:33:21.679 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:21.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.679 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.680 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:33:21.680 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:21.680 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.680 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.681 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:21.681 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.681 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.682 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:33:21.682 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:21.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.682 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.683 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:33:21.683 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:21.683 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.683 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.684 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:33:21.684 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:33:21.684 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:33:21.685 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:33:21.685 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:21.685 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:21.685 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:21.686 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.686 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.686 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:33:21.686 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:33:21.687 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:21.687 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:21.687 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:21.687 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:21.687 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:21.688 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:21.688 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:21.688 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 224, 2021-04-09T18:33:21.688 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:21.689 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:21.689 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:21.689 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.689 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.690 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:33:21.690 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:21.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.690 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.691 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:33:21.691 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:21.691 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.691 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.693 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:33:21.693 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:21.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.693 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.693 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.694 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:33:21.694 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:21.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.694 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.695 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:21.695 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.695 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.695 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.696 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:33:21.696 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:21.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.696 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.696 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.697 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:33:21.697 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:21.697 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.697 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.698 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:21.698 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.698 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.698 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.699 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:33:21.699 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:21.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.699 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.699 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.700 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:33:21.700 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:21.700 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.700 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.701 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:33:21.701 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:33:21.701 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:33:21.701 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:33:21.702 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:21.702 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:21.702 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:21.702 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.703 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.703 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:33:21.703 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:33:21.703 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:21.703 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:21.704 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:21.704 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:21.704 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:21.704 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:21.704 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:21.705 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:21.705 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:21.705 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:21.705 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:21.706 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.706 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.706 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.706 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:33:21.706 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:21.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.707 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.707 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:33:21.707 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:21.708 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.708 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.708 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:21.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.709 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.709 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:33:21.709 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:21.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.710 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.710 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:33:21.711 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:21.711 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.711 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.711 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:21.712 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.712 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.712 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.712 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:33:21.713 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:21.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.713 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.713 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.713 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:33:21.714 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:21.714 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.714 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.714 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:21.714 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.715 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.715 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.715 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:33:21.715 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:21.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.716 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.716 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.716 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:33:21.716 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:21.716 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.717 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.717 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:33:21.717 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:33:21.717 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:33:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:33:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:21.718 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:33:21.719 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:33:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:21.720 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:21.721 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:21.721 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:21.721 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:21.721 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:21.721 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:21.722 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:21.722 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:21.722 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.722 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.722 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.723 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:33:21.723 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:21.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.723 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.724 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.724 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:33:21.724 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:21.724 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.724 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.725 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.725 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:21.725 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.725 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.726 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:33:21.726 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:21.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.726 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.727 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:33:21.727 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:21.727 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.727 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.728 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.728 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:21.728 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.728 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.729 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:33:21.729 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:21.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.729 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.730 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:33:21.730 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:21.730 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.730 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.731 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:21.731 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:21.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.731 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:21.732 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:33:21.732 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:21.732 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.732 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.733 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:21.733 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:33:21.733 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:21.733 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.733 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.734 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:33:21.734 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:33:21.734 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:33:21.735 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:33:21.735 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:21.735 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:21.735 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:21.735 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:21.736 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.736 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:33:21.736 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:33:21.736 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.737 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:33:21.737 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:21.737 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:21.737 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:33:21.739 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:21.739 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:21.739 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:21.739 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:21.740 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:21.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.740 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:33:21.740 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:21.741 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:21.741 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:21.741 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:21.741 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:21.742 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.742 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.742 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:33:21.742 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:21.742 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:21.743 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:21.743 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:21.743 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:21.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.744 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.744 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:33:21.744 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:21.744 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:21.744 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:21.745 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:21.745 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:21.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.745 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.746 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:33:21.746 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:21.746 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:21.746 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:21.746 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:21.747 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:21.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.747 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.747 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:33:21.747 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:21.748 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:21.748 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:21.748 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:21.748 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:21.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.749 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:21.749 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:33:21.749 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:21.749 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:21.750 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:21.750 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:21.750 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:21.750 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.750 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:33:21.751 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:33:21.751 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:33:21.751 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:33:21.751 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:33:21.751 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:33:21.752 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6837/1293489350": "2021-04-10 18:33:07.891065", 2021-04-09T18:33:21.752 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/1293489350": "2021-04-10 18:33:07.891065", 2021-04-09T18:33:21.752 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/2382296066": "2021-04-10 18:33:00.283626", 2021-04-09T18:33:21.752 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/2382296066": "2021-04-10 18:33:00.283626", 2021-04-09T18:33:21.753 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/3107210651": "2021-04-10 18:32:51.914410", 2021-04-09T18:33:21.753 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/3107210651": "2021-04-10 18:32:51.914410", 2021-04-09T18:33:21.753 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6837/2051228504": "2021-04-10 18:32:51.911792", 2021-04-09T18:33:21.753 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/2051228504": "2021-04-10 18:32:51.911792" 2021-04-09T18:33:21.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.754 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:33:21.754 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:33:21.754 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:33:21.754 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:33:21.754 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:33:21.755 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:33:21.755 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:33:21.755 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:33:21.755 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:21.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:21.756 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:33:21.756 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:33:21.756 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:33:21.757 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:33:21.757 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:33:21.757 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:33:21.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 -- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 msgr2=0x7f43ec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01ae10 crc :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 -- 172.21.15.32:0/1369823128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4408185e70 msgr2=0x7f44080ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4408185e70 0x7f44080ad1d0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 rx=0x7f440000b970 tx=0x7f440000bd10).stop 2021-04-09T18:33:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 -- 172.21.15.32:0/1369823128 shutdown_connections 2021-04-09T18:33:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43ec018930 0x7f43ec01ae10 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4408185e70 0x7f44080ad1d0 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408185a70 0x7f44080acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 --2- 172.21.15.32:0/1369823128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44081a5210 0x7f4408185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 -- 172.21.15.32:0/1369823128 >> 172.21.15.32:0/1369823128 conn(0x7f44081a99c0 msgr2=0x7f440817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:21.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 -- 172.21.15.32:0/1369823128 shutdown_connections 2021-04-09T18:33:21.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.494 7f440e02e700 1 -- 172.21.15.32:0/1369823128 wait complete. 2021-04-09T18:33:21.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:21.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.894 7fdfaac21700 1 Processor -- start 2021-04-09T18:33:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.894 7fdfaac21700 1 -- start start 2021-04-09T18:33:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfaac21700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa410d010 0x7fdfa40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfaac21700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa410d480 con 0x7fdfa40d8490 2021-04-09T18:33:21.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfaac21700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa410e1f0 con 0x7fdfa40d1da0 2021-04-09T18:33:21.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfaac21700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa410e780 con 0x7fdfa410d010 2021-04-09T18:33:21.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa923e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdfa40d8490 0x7fdfa410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39826/0 (socket says 172.21.15.32:39826) 2021-04-09T18:33:21.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa923e700 1 -- 172.21.15.32:0/3701868501 learned_addr learned my addr 172.21.15.32:0/3701868501 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:21.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa8a3d700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa410d010 0x7fdfa40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916339731 0 0) 0x7fdf9c000d30 con 0x7fdfa40d8490 2021-04-09T18:33:21.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf8c002fe0 con 0x7fdfa40d8490 2021-04-09T18:33:21.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1168537704 0 0) 0x7fdf9c001270 con 0x7fdfa40d8490 2021-04-09T18:33:21.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa410d010 msgr2=0x7fdfa40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:21.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa410d010 0x7fdfa40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 >> v1:172.21.15.86:6789/0 conn(0x7fdfa40d1da0 legacy=0x7fdfa40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa40d8d30 con 0x7fdfa40d8490 2021-04-09T18:33:21.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fdf9c001980 con 0x7fdfa40d8490 2021-04-09T18:33:21.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 >> v1:172.21.15.32:6789/0 conn(0x7fdfa40d8490 legacy=0x7fdfa410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8c003af0 0x7fdf8c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf8c006820 0x7fdf8c008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf8c0095b0 0x7fdf8c00bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c00c380 con 0x7fdf8c003af0 2021-04-09T18:33:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.895 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c00c5a0 con 0x7fdf8c0095b0 2021-04-09T18:33:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c0507d0 con 0x7fdf8c006820 2021-04-09T18:33:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa8a3d700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa410d010 0x7fdfa40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa3fff700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf8c006820 0x7fdf8c008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa923e700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf8c0095b0 0x7fdf8c00bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa8a3d700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8c003af0 0x7fdf8c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa3fff700 1 -- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf8c0095b0 msgr2=0x7fdf8c00bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:21.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa3fff700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf8c0095b0 0x7fdf8c00bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa3fff700 1 -- 172.21.15.32:0/3701868501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8c003af0 msgr2=0x7fdf8c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:21.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa3fff700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf8c003af0 0x7fdf8c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa3fff700 1 -- 172.21.15.32:0/3701868501 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c050a30 con 0x7fdf8c006820 2021-04-09T18:33:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa3fff700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf8c006820 0x7fdf8c008d10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fdf98000e30 tx=0x7fdf98009aa0).ready entity=mon.2 client_cookie=d425bccaa2879c22 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf980066e0 con 0x7fdf8c006820 2021-04-09T18:33:21.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9800c0c0 con 0x7fdf8c006820 2021-04-09T18:33:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa37fe700 1 -- 172.21.15.32:0/3701868501 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf980066e0 con 0x7fdf8c006820 2021-04-09T18:33:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfaac21700 1 -- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf8c006820 msgr2=0x7fdf8c008d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfaac21700 1 --2- 172.21.15.32:0/3701868501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf8c006820 0x7fdf8c008d10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fdf98000e30 tx=0x7fdf98009aa0).stop 2021-04-09T18:33:21.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.896 7fdfa923e700 1 -- 172.21.15.32:0/3701868501 reap_dead start 2021-04-09T18:33:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.897 7fdfaac21700 1 -- 172.21.15.32:0/3701868501 shutdown_connections 2021-04-09T18:33:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.897 7fdfaac21700 1 -- 172.21.15.32:0/3701868501 >> 172.21.15.32:0/3701868501 conn(0x7fdfa40ce5f0 msgr2=0x7fdfa40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.897 7fdfaac21700 1 -- 172.21.15.32:0/3701868501 shutdown_connections 2021-04-09T18:33:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.897 7fdfaac21700 1 -- 172.21.15.32:0/3701868501 wait complete. 2021-04-09T18:33:21.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.917 7fdfaac21700 1 Processor -- start 2021-04-09T18:33:21.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.917 7fdfaac21700 1 -- start start 2021-04-09T18:33:21.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfaac21700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa409a670 0x7fdfa4093610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfaac21700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa41a9fb0 0x7fdfa41aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfaac21700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfa40a3ab0 0x7fdfa40a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfaac21700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4094060 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfaac21700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4094710 con 0x7fdfa409a670 2021-04-09T18:33:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfaac21700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4093c30 con 0x7fdfa41a9fb0 2021-04-09T18:33:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa923e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfa40a3ab0 0x7fdfa40a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa923e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfa40a3ab0 0x7fdfa40a3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58480/0 (socket says 172.21.15.32:58480) 2021-04-09T18:33:21.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa923e700 1 -- 172.21.15.32:0/761252499 learned_addr learned my addr 172.21.15.32:0/761252499 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:21.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa8a3d700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa409a670 0x7fdfa4093610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa3fff700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa41a9fb0 0x7fdfa41aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa923e700 1 -- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa41a9fb0 msgr2=0x7fdfa41aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa923e700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa41a9fb0 0x7fdfa41aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa923e700 1 -- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa409a670 msgr2=0x7fdfa4093610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:21.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa923e700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa409a670 0x7fdfa4093610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:21.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.918 7fdfa923e700 1 -- 172.21.15.32:0/761252499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa41aa9d0 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.919 7fdfa923e700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfa40a3ab0 0x7fdfa40a3eb0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 rx=0x7fdf9c000ee0 tx=0x7fdf9c00ac30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:21.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.919 7fdfa17fa700 1 -- 172.21.15.32:0/761252499 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf9c007020 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.919 7fdfa17fa700 1 -- 172.21.15.32:0/761252499 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9c0072e0 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.919 7fdfaac21700 1 -- 172.21.15.32:0/761252499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa41aa9d0 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.919 7fdfa17fa700 1 -- 172.21.15.32:0/761252499 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf9c004720 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.919 7fdfaac21700 1 -- 172.21.15.32:0/761252499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa41aa9d0 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.919 7fdfa17fa700 1 -- 172.21.15.32:0/761252499 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdf9c001d70 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.919 7fdfa17fa700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf84018930 0x7fdf8401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:21.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.920 7fdfa17fa700 1 -- 172.21.15.32:0/761252499 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6733+0+0 (secure 0 0 0) 0x7fdf9c033200 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.920 7fdfa8a3d700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf84018930 0x7fdf8401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:21.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.920 7fdfa8a3d700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf84018930 0x7fdf8401ae10 crc :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:21.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.939 7fdfaac21700 1 -- 172.21.15.32:0/761252499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa410d480 con 0x7fdfa40a3ab0 2021-04-09T18:33:21.959 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:21.942 7fdfa17fa700 1 -- 172.21.15.32:0/761252499 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdf9c0013e0 con 0x7fdfa40a3ab0 2021-04-09T18:33:22.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.133 7fdfaac21700 1 -- 172.21.15.32:0/761252499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfa40946d0 con 0x7fdfa40a3ab0 2021-04-09T18:33:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.133 7fdfa17fa700 1 -- 172.21.15.32:0/761252499 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 401 v401) v1 ==== 95+0+3720 (secure 0 0 0) 0x7fdf9c001d50 con 0x7fdfa40a3ab0 2021-04-09T18:33:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 401 2021-04-09T18:33:22.152 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:22.152 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":401,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2768006990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2768006990},{"type":"v1","addr":"172.21.15.86:6837","nonce":2768006990}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401},{"gid":59606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3794784835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3794784835},{"type":"v1","addr":"172.21.15.86:6836","nonce":3794784835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401}],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:04.212398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":235,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12195},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12195":{"gid":12195,"name":"a","rank":0,"incarnation":392,"state":"up:active","state_seq":7,"addr":"172.21.15.32:6835/2918355247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2918355247},{"type":"v1","addr":"172.21.15.32:6835","nonce":2918355247}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:33:11.245312","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":236,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":59510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_59510":{"gid":59510,"name":"c","rank":0,"incarnation":397,"state":"up:active","state_seq":9,"addr":"172.21.15.32:6834/222606831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":222606831},{"type":"v1","addr":"172.21.15.32:6834","nonce":222606831}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 -- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf84018930 msgr2=0x7fdf8401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf84018930 0x7fdf8401ae10 crc :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 -- 172.21.15.32:0/761252499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfa40a3ab0 msgr2=0x7fdfa40a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfa40a3ab0 0x7fdfa40a3eb0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 rx=0x7fdf9c000ee0 tx=0x7fdf9c00ac30).stop 2021-04-09T18:33:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 -- 172.21.15.32:0/761252499 shutdown_connections 2021-04-09T18:33:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf84018930 0x7fdf8401ae10 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfa40a3ab0 0x7fdfa40a3eb0 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa41a9fb0 0x7fdfa41aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 --2- 172.21.15.32:0/761252499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa409a670 0x7fdfa4093610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 -- 172.21.15.32:0/761252499 >> 172.21.15.32:0/761252499 conn(0x7fdfa41a0db0 msgr2=0x7fdfa41a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 -- 172.21.15.32:0/761252499 shutdown_connections 2021-04-09T18:33:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.136 7fdfaac21700 1 -- 172.21.15.32:0/761252499 wait complete. 2021-04-09T18:33:22.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail alpha 2021-04-09T18:33:22.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.300 7f15616a0700 1 Processor -- start 2021-04-09T18:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.300 7f15616a0700 1 -- start start 2021-04-09T18:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.300 7f15616a0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f155c0d6c90 0x7f155c0d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.300 7f15616a0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c0d2450 0x7f155c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.300 7f15616a0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c0d7290 con 0x7f155c0cffb0 2021-04-09T18:33:22.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.300 7f15616a0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c0d7090 con 0x7f155c0d6c90 2021-04-09T18:33:22.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.300 7f15616a0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0d91f0 con 0x7f155c0d2450 2021-04-09T18:33:22.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155bac0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c0d2450 0x7f155c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155aabe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f155c0d6c90 0x7f155c0d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155bac0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c0d2450 0x7f155c10cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:37480/0 (socket says 172.21.15.32:37480) 2021-04-09T18:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155bac0700 1 -- 172.21.15.32:0/3611835518 learned_addr learned my addr 172.21.15.32:0/3611835518 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155bac0700 1 -- 172.21.15.32:0/3611835518 >> v2:172.21.15.86:3300/0 conn(0x7f155c0d6c90 msgr2=0x7f155c0d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:22.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155bac0700 1 --2- 172.21.15.32:0/3611835518 >> v2:172.21.15.86:3300/0 conn(0x7f155c0d6c90 0x7f155c0d8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155bac0700 1 -- 172.21.15.32:0/3611835518 >> v1:172.21.15.86:6789/0 conn(0x7f155c0cffb0 legacy=0x7f155c0d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155a2bd700 1 -- 172.21.15.32:0/3611835518 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852934432 0 0) 0x7f154c000c60 con 0x7f155c0cffb0 2021-04-09T18:33:22.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155bac0700 1 -- 172.21.15.32:0/3611835518 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0d2850 con 0x7f155c0d2450 2021-04-09T18:33:22.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155aabe700 1 --2- 172.21.15.32:0/3611835518 >> v2:172.21.15.86:3300/0 conn(0x7f155c0d6c90 0x7f155c0d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:22.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155bac0700 1 --2- 172.21.15.32:0/3611835518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c0d2450 0x7f155c10cfe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1554006aa0 tx=0x7f1554009f20).ready entity=mon.2 client_cookie=deb10242f3566548 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155a2bd700 1 -- 172.21.15.32:0/3611835518 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155400b5c0 con 0x7f155c0d2450 2021-04-09T18:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.301 7f155a2bd700 1 -- 172.21.15.32:0/3611835518 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1554003030 con 0x7f155c0d2450 2021-04-09T18:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.302 7f155a2bd700 1 -- 172.21.15.32:0/3611835518 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155400bb40 con 0x7f155c0d2450 2021-04-09T18:33:22.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.302 7f15616a0700 1 -- 172.21.15.32:0/3611835518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c0d2450 msgr2=0x7f155c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.302 7f15616a0700 1 --2- 172.21.15.32:0/3611835518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c0d2450 0x7f155c10cfe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1554006aa0 tx=0x7f1554009f20).stop 2021-04-09T18:33:22.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.302 7f15616a0700 1 -- 172.21.15.32:0/3611835518 shutdown_connections 2021-04-09T18:33:22.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.302 7f15616a0700 1 --2- 172.21.15.32:0/3611835518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c0d2450 0x7f155c10cfe0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.302 7f15616a0700 1 --2- 172.21.15.32:0/3611835518 >> v2:172.21.15.86:3300/0 conn(0x7f155c0d6c90 0x7f155c0d8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.302 7f15616a0700 1 -- 172.21.15.32:0/3611835518 >> 172.21.15.32:0/3611835518 conn(0x7f155c0ce5d0 msgr2=0x7f155c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.302 7f15616a0700 1 -- 172.21.15.32:0/3611835518 shutdown_connections 2021-04-09T18:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.302 7f15616a0700 1 -- 172.21.15.32:0/3611835518 wait complete. 2021-04-09T18:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.322 7f15616a0700 1 Processor -- start 2021-04-09T18:33:22.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.322 7f15616a0700 1 -- start start 2021-04-09T18:33:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f15616a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f155c1a7dc0 0x7f155c1a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f15616a0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c184080 0x7f155c184480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f15616a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f155c184aa0 0x7f155c08f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f15616a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c1a8b40 con 0x7f155c1a7dc0 2021-04-09T18:33:22.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f15616a0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c1a7870 con 0x7f155c184aa0 2021-04-09T18:33:22.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f15616a0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c1a87e0 con 0x7f155c184080 2021-04-09T18:33:22.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f155b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f155c1a7dc0 0x7f155c1a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f155b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f155c1a7dc0 0x7f155c1a81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58492/0 (socket says 172.21.15.32:58492) 2021-04-09T18:33:22.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f155b2bf700 1 -- 172.21.15.32:0/3001758597 learned_addr learned my addr 172.21.15.32:0/3001758597 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:22.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f155bac0700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f155c184aa0 0x7f155c08f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f155aabe700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c184080 0x7f155c184480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f155b2bf700 1 -- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c184080 msgr2=0x7f155c184480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.323 7f155b2bf700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c184080 0x7f155c184480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f155b2bf700 1 -- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f155c184aa0 msgr2=0x7f155c08f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f155b2bf700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f155c184aa0 0x7f155c08f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f155b2bf700 1 -- 172.21.15.32:0/3001758597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c08f810 con 0x7f155c1a7dc0 2021-04-09T18:33:22.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f155b2bf700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f155c1a7dc0 0x7f155c1a81c0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f154c006f20 tx=0x7f154c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:22.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f1543fff700 1 -- 172.21.15.32:0/3001758597 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f154c00bd80 con 0x7f155c1a7dc0 2021-04-09T18:33:22.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f1543fff700 1 -- 172.21.15.32:0/3001758597 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154c001080 con 0x7f155c1a7dc0 2021-04-09T18:33:22.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f1543fff700 1 -- 172.21.15.32:0/3001758597 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f154c001550 con 0x7f155c1a7dc0 2021-04-09T18:33:22.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f15616a0700 1 -- 172.21.15.32:0/3001758597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c08f810 con 0x7f155c1a7dc0 2021-04-09T18:33:22.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f15616a0700 1 -- 172.21.15.32:0/3001758597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c08f810 con 0x7f155c1a7dc0 2021-04-09T18:33:22.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.324 7f1543fff700 1 -- 172.21.15.32:0/3001758597 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f154c001260 con 0x7f155c1a7dc0 2021-04-09T18:33:22.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.325 7f1543fff700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f153c018930 0x7f153c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.325 7f1543fff700 1 -- 172.21.15.32:0/3001758597 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6733+0+0 (secure 0 0 0) 0x7f154c032ec0 con 0x7f155c1a7dc0 2021-04-09T18:33:22.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.325 7f155aabe700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f153c018930 0x7f153c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.325 7f155aabe700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f153c018930 0x7f153c01ae10 crc :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:22.362 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.344 7f15616a0700 1 -- 172.21.15.32:0/3001758597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c182e80 con 0x7f155c1a7dc0 2021-04-09T18:33:22.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.348 7f1543fff700 1 -- 172.21.15.32:0/3001758597 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f154c006bf0 con 0x7f155c1a7dc0 2021-04-09T18:33:22.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.529 7f15616a0700 1 -- 172.21.15.32:0/3001758597 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "alpha"} v 0) v1 -- 0x7f155c0d7290 con 0x7f155c1a7dc0 2021-04-09T18:33:22.574 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:33:22.557 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:33:22.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.564 7f1543fff700 1 -- 172.21.15.32:0/3001758597 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "alpha"}]=0 alpha marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v402) v1 ==== 159+0+0 (secure 0 0 0) 0x7f154c001080 con 0x7f155c1a7dc0 2021-04-09T18:33:22.610 INFO:teuthology.orchestra.run.smithi032.stderr:alpha marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:33:22.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 -- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f153c018930 msgr2=0x7f153c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f153c018930 0x7f153c01ae10 crc :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 -- 172.21.15.32:0/3001758597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f155c1a7dc0 msgr2=0x7f155c1a81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f155c1a7dc0 0x7f155c1a81c0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f154c006f20 tx=0x7f154c00b990).stop 2021-04-09T18:33:22.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 -- 172.21.15.32:0/3001758597 shutdown_connections 2021-04-09T18:33:22.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f153c018930 0x7f153c01ae10 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f155c184aa0 0x7f155c08f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f155c184080 0x7f155c184480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 --2- 172.21.15.32:0/3001758597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f155c1a7dc0 0x7f155c1a81c0 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.595 7f15616a0700 1 -- 172.21.15.32:0/3001758597 >> 172.21.15.32:0/3001758597 conn(0x7f155c1a8f50 msgr2=0x7f155c188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:22.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.596 7f15616a0700 1 -- 172.21.15.32:0/3001758597 shutdown_connections 2021-04-09T18:33:22.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.596 7f15616a0700 1 -- 172.21.15.32:0/3001758597 wait complete. 2021-04-09T18:33:22.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail bravo 2021-04-09T18:33:22.639 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:33:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.759 7f876aaa8700 1 Processor -- start 2021-04-09T18:33:22.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.759 7f876aaa8700 1 -- start start 2021-04-09T18:33:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.759 7f876aaa8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87640cffb0 0x7f87640d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.759 7f876aaa8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f87640d6c90 0x7f876410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.759 7f876aaa8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87640d7290 con 0x7f87640d7810 2021-04-09T18:33:22.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.759 7f876aaa8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87640d7090 con 0x7f87640d6c90 2021-04-09T18:33:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.759 7f876aaa8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87640d8590 con 0x7f87640cffb0 2021-04-09T18:33:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f87690c5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f87640d6c90 0x7f876410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f87688c4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87640cffb0 0x7f87640d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f8763fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f87640d7810 0x7f87640d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37794/0 (socket says 172.21.15.32:37794) 2021-04-09T18:33:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f8763fff700 1 -- 172.21.15.32:0/1252328801 learned_addr learned my addr 172.21.15.32:0/1252328801 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f87688c4700 1 -- 172.21.15.32:0/1252328801 >> v2:172.21.15.86:3300/0 conn(0x7f87640d6c90 msgr2=0x7f876410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f87688c4700 1 --2- 172.21.15.32:0/1252328801 >> v2:172.21.15.86:3300/0 conn(0x7f87640d6c90 0x7f876410cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f87688c4700 1 -- 172.21.15.32:0/1252328801 >> v1:172.21.15.86:6789/0 conn(0x7f87640d7810 legacy=0x7f87640d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f87688c4700 1 -- 172.21.15.32:0/1252328801 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87640d91f0 con 0x7f87640cffb0 2021-04-09T18:33:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f87688c4700 1 --2- 172.21.15.32:0/1252328801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87640cffb0 0x7f87640d2450 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f874c006650 tx=0x7f874c009a80).ready entity=mon.2 client_cookie=3c5865322d56d94c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.760 7f87637fe700 1 -- 172.21.15.32:0/1252328801 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f874c0009b0 con 0x7f87640cffb0 2021-04-09T18:33:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f87637fe700 1 -- 172.21.15.32:0/1252328801 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874c00c0c0 con 0x7f87640cffb0 2021-04-09T18:33:22.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f87637fe700 1 -- 172.21.15.32:0/1252328801 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f874c00c510 con 0x7f87640cffb0 2021-04-09T18:33:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f876aaa8700 1 -- 172.21.15.32:0/1252328801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87640cffb0 msgr2=0x7f87640d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f876aaa8700 1 --2- 172.21.15.32:0/1252328801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87640cffb0 0x7f87640d2450 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f874c006650 tx=0x7f874c009a80).stop 2021-04-09T18:33:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f876aaa8700 1 -- 172.21.15.32:0/1252328801 shutdown_connections 2021-04-09T18:33:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f876aaa8700 1 --2- 172.21.15.32:0/1252328801 >> v2:172.21.15.86:3300/0 conn(0x7f87640d6c90 0x7f876410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f876aaa8700 1 --2- 172.21.15.32:0/1252328801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87640cffb0 0x7f87640d2450 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f876aaa8700 1 -- 172.21.15.32:0/1252328801 >> 172.21.15.32:0/1252328801 conn(0x7f87640ce5d0 msgr2=0x7f87640cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f876aaa8700 1 -- 172.21.15.32:0/1252328801 shutdown_connections 2021-04-09T18:33:22.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.761 7f876aaa8700 1 -- 172.21.15.32:0/1252328801 wait complete. 2021-04-09T18:33:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.781 7f876aaa8700 1 Processor -- start 2021-04-09T18:33:22.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.781 7f876aaa8700 1 -- start start 2021-04-09T18:33:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f876aaa8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8764184370 0x7f8764184770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f876aaa8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87641a7ea0 0x7f87641a82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f876aaa8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87640b7e00 0x7f87640b8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f876aaa8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87641a7920 con 0x7f87640b7e00 2021-04-09T18:33:22.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f876aaa8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87641a88c0 con 0x7f8764184370 2021-04-09T18:33:22.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f876aaa8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87640b8820 con 0x7f87641a7ea0 2021-04-09T18:33:22.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f87690c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87640b7e00 0x7f87640b8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f87690c5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87640b7e00 0x7f87640b8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58518/0 (socket says 172.21.15.32:58518) 2021-04-09T18:33:22.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f87690c5700 1 -- 172.21.15.32:0/1326439292 learned_addr learned my addr 172.21.15.32:0/1326439292 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f8763fff700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87641a7ea0 0x7f87641a82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.782 7f87688c4700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8764184370 0x7f8764184770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87690c5700 1 -- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87641a7ea0 msgr2=0x7f87641a82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87690c5700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87641a7ea0 0x7f87641a82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87690c5700 1 -- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8764184370 msgr2=0x7f8764184770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:22.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87690c5700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8764184370 0x7f8764184770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:22.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87690c5700 1 -- 172.21.15.32:0/1326439292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87640b8a40 con 0x7f87640b7e00 2021-04-09T18:33:22.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87688c4700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8764184370 0x7f8764184770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:22.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87690c5700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87640b7e00 0x7f87640b8200 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7f875c00b910 tx=0x7f875c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:22.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87617fa700 1 -- 172.21.15.32:0/1326439292 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f875c0066f0 con 0x7f87640b7e00 2021-04-09T18:33:22.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87617fa700 1 -- 172.21.15.32:0/1326439292 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c003030 con 0x7f87640b7e00 2021-04-09T18:33:22.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87640b8a40 con 0x7f87640b7e00 2021-04-09T18:33:22.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f87617fa700 1 -- 172.21.15.32:0/1326439292 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f875c001470 con 0x7f87640b7e00 2021-04-09T18:33:22.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.783 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87640b8a40 con 0x7f87640b7e00 2021-04-09T18:33:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.784 7f87617fa700 1 -- 172.21.15.32:0/1326439292 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f875c0011f0 con 0x7f87640b7e00 2021-04-09T18:33:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.784 7f87617fa700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87480188c0 0x7f874801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.784 7f87617fa700 1 -- 172.21.15.32:0/1326439292 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f875c033090 con 0x7f87640b7e00 2021-04-09T18:33:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.784 7f87688c4700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87480188c0 0x7f874801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:22.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.784 7f87688c4700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87480188c0 0x7f874801adc0 crc :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:22.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.803 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8764182e80 con 0x7f87640b7e00 2021-04-09T18:33:22.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.807 7f87617fa700 1 -- 172.21.15.32:0/1326439292 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f875c02df80 con 0x7f87640b7e00 2021-04-09T18:33:23.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:22.992 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "bravo"} v 0) v1 -- 0x7f8764182e80 con 0x7f87640b7e00 2021-04-09T18:33:23.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.565 7f87617fa700 1 -- 172.21.15.32:0/1326439292 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "bravo"}]=0 bravo marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v403) v1 ==== 159+0+0 (secure 0 0 0) 0x7f875c02df80 con 0x7f87640b7e00 2021-04-09T18:33:23.624 INFO:teuthology.orchestra.run.smithi032.stderr:bravo marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:33:23.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87480188c0 msgr2=0x7f874801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:23.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87480188c0 0x7f874801adc0 crc :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:23.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87640b7e00 msgr2=0x7f87640b8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87640b7e00 0x7f87640b8200 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7f875c00b910 tx=0x7f875c00bcb0).stop 2021-04-09T18:33:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 shutdown_connections 2021-04-09T18:33:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87480188c0 0x7f874801adc0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87640b7e00 0x7f87640b8200 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:23.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87641a7ea0 0x7f87641a82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:23.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 --2- 172.21.15.32:0/1326439292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8764184370 0x7f8764184770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:23.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.609 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 >> 172.21.15.32:0/1326439292 conn(0x7f87641a8f50 msgr2=0x7f8764188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:23.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.610 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 shutdown_connections 2021-04-09T18:33:23.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.610 7f876aaa8700 1 -- 172.21.15.32:0/1326439292 wait complete. 2021-04-09T18:33:23.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:23.648 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:33:23.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.779 7fedb2bf6700 1 Processor -- start 2021-04-09T18:33:23.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.779 7fedb2bf6700 1 -- start start 2021-04-09T18:33:23.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb2bf6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fedac10da00 0x7fedac10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:23.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb2bf6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac10d480 con 0x7fedac10d010 2021-04-09T18:33:23.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb2bf6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac0d1f70 con 0x7fedac0d7ca0 2021-04-09T18:33:23.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb2bf6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedac0d8d30 con 0x7fedac10da00 2021-04-09T18:33:23.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb0a12700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fedac10d010 0x7fedac0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39890/0 (socket says 172.21.15.32:39890) 2021-04-09T18:33:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb0a12700 1 -- 172.21.15.32:0/2660788654 learned_addr learned my addr 172.21.15.32:0/2660788654 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb1213700 1 --2- 172.21.15.32:0/2660788654 >> v2:172.21.15.86:3300/0 conn(0x7fedac10da00 0x7fedac10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedab7fe700 1 -- 172.21.15.32:0/2660788654 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390774530 0 0) 0x7fed94000ce0 con 0x7fedac10d010 2021-04-09T18:33:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedab7fe700 1 -- 172.21.15.32:0/2660788654 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed98002fe0 con 0x7fedac10d010 2021-04-09T18:33:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedab7fe700 1 -- 172.21.15.32:0/2660788654 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304710773 0 0) 0x7fed9c000c60 con 0x7fedac0d7ca0 2021-04-09T18:33:23.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedab7fe700 1 -- 172.21.15.32:0/2660788654 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed98003210 con 0x7fedac0d7ca0 2021-04-09T18:33:23.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb1213700 1 -- 172.21.15.32:0/2660788654 >> v1:172.21.15.86:6789/0 conn(0x7fedac0d7ca0 legacy=0x7fedac0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:23.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb1213700 1 -- 172.21.15.32:0/2660788654 >> v1:172.21.15.32:6789/0 conn(0x7fedac10d010 legacy=0x7fedac0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:23.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.780 7fedb1213700 1 -- 172.21.15.32:0/2660788654 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac0d9220 con 0x7fedac10da00 2021-04-09T18:33:23.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedb1213700 1 --2- 172.21.15.32:0/2660788654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedac10da00 0x7fedac10de00 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7feda4006e50 tx=0x7feda4009e70).ready entity=mon.1 client_cookie=c906628fe04b8c51 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:23.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedab7fe700 1 -- 172.21.15.32:0/2660788654 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feda4006a20 con 0x7fedac10da00 2021-04-09T18:33:23.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedab7fe700 1 -- 172.21.15.32:0/2660788654 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda4006790 con 0x7fedac10da00 2021-04-09T18:33:23.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedab7fe700 1 -- 172.21.15.32:0/2660788654 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feda400b700 con 0x7fedac10da00 2021-04-09T18:33:23.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedb2bf6700 1 -- 172.21.15.32:0/2660788654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedac10da00 msgr2=0x7fedac10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:23.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedb2bf6700 1 --2- 172.21.15.32:0/2660788654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedac10da00 0x7fedac10de00 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7feda4006e50 tx=0x7feda4009e70).stop 2021-04-09T18:33:23.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedb2bf6700 1 -- 172.21.15.32:0/2660788654 shutdown_connections 2021-04-09T18:33:23.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedb2bf6700 1 --2- 172.21.15.32:0/2660788654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedac10da00 0x7fedac10de00 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:23.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedb2bf6700 1 -- 172.21.15.32:0/2660788654 >> 172.21.15.32:0/2660788654 conn(0x7fedac0ce5f0 msgr2=0x7fedac0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.781 7fedb2bf6700 1 -- 172.21.15.32:0/2660788654 shutdown_connections 2021-04-09T18:33:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.782 7fedb2bf6700 1 -- 172.21.15.32:0/2660788654 wait complete. 2021-04-09T18:33:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.801 7fedb2bf6700 1 Processor -- start 2021-04-09T18:33:23.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb2bf6700 1 -- start start 2021-04-09T18:33:23.828 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb2bf6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedac179610 0x7fedac179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:23.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb2bf6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedac17a6e0 0x7fedac17aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:23.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb2bf6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedac17b100 0x7fedac08db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:23.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb2bf6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac17d720 con 0x7fedac17b100 2021-04-09T18:33:23.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb2bf6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac185ba0 con 0x7fedac17a6e0 2021-04-09T18:33:23.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb2bf6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedac08e1b0 con 0x7fedac179610 2021-04-09T18:33:23.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb1213700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedac17b100 0x7fedac08db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:23.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb1213700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedac17b100 0x7fedac08db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58542/0 (socket says 172.21.15.32:58542) 2021-04-09T18:33:23.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.802 7fedb1213700 1 -- 172.21.15.32:0/3623552556 learned_addr learned my addr 172.21.15.32:0/3623552556 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:23.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedabfff700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedac17a6e0 0x7fedac17aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb0a12700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedac179610 0x7fedac179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb1213700 1 -- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedac179610 msgr2=0x7fedac179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb1213700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedac179610 0x7fedac179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb1213700 1 -- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedac17a6e0 msgr2=0x7fedac17aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:23.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb1213700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedac17a6e0 0x7fedac17aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:23.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb1213700 1 -- 172.21.15.32:0/3623552556 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac08e3d0 con 0x7fedac17b100 2021-04-09T18:33:23.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb0a12700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedac179610 0x7fedac179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:23.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb1213700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedac17b100 0x7fedac08db90 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7feda40099f0 tx=0x7feda400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:23.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7feda97fa700 1 -- 172.21.15.32:0/3623552556 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feda40093d0 con 0x7fedac17b100 2021-04-09T18:33:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7feda97fa700 1 -- 172.21.15.32:0/3623552556 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda40090c0 con 0x7fedac17b100 2021-04-09T18:33:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac08e3d0 con 0x7fedac17b100 2021-04-09T18:33:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7feda97fa700 1 -- 172.21.15.32:0/3623552556 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feda4001420 con 0x7fedac17b100 2021-04-09T18:33:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.803 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac08e3d0 con 0x7fedac17b100 2021-04-09T18:33:23.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.804 7feda97fa700 1 -- 172.21.15.32:0/3623552556 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7feda40011a0 con 0x7fedac17b100 2021-04-09T18:33:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.804 7feda97fa700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed90018930 0x7fed9001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.804 7feda97fa700 1 -- 172.21.15.32:0/3623552556 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6905+0+0 (secure 0 0 0) 0x7feda4033300 con 0x7fedac17b100 2021-04-09T18:33:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.804 7fedb0a12700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed90018930 0x7fed9001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:23.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.804 7fedb0a12700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed90018930 0x7fed9001ae10 crc :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:23.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.824 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac17a1b0 con 0x7fedac17b100 2021-04-09T18:33:23.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:23.827 7feda97fa700 1 -- 172.21.15.32:0/3623552556 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7feda4000df0 con 0x7fedac17b100 2021-04-09T18:33:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.019 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedac17a1b0 con 0x7fedac17b100 2021-04-09T18:33:24.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.019 7feda97fa700 1 -- 172.21.15.32:0/3623552556 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 403 v403) v1 ==== 95+0+3023 (secure 0 0 0) 0x7feda40090a0 con 0x7fedac17b100 2021-04-09T18:33:24.037 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 403 2021-04-09T18:33:24.037 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:24.037 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":403,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2768006990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2768006990},{"type":"v1","addr":"172.21.15.86:6837","nonce":2768006990}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401},{"gid":59606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3794784835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3794784835},{"type":"v1","addr":"172.21.15.86:6836","nonce":3794784835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401}],"filesystems":[{"mdsmap":{"epoch":402,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:37.406526","modified":"2021-04-09 18:33:22.558439","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":237,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"alpha","balancer":"","standby_count_wanted":0},"id":19},{"mdsmap":{"epoch":403,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:32:44.859244","modified":"2021-04-09 18:33:23.564499","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":238,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"bravo","balancer":"","standby_count_wanted":0},"id":20}]} 2021-04-09T18:33:24.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.021 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed90018930 msgr2=0x7fed9001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.021 7fedb2bf6700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed90018930 0x7fed9001ae10 crc :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.021 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedac17b100 msgr2=0x7fedac08db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.021 7fedb2bf6700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedac17b100 0x7fedac08db90 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7feda40099f0 tx=0x7feda400bd10).stop 2021-04-09T18:33:24.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.022 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 shutdown_connections 2021-04-09T18:33:24.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.022 7fedb2bf6700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed90018930 0x7fed9001ae10 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.022 7fedb2bf6700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedac17b100 0x7fedac08db90 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.022 7fedb2bf6700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedac17a6e0 0x7fedac17aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.022 7fedb2bf6700 1 --2- 172.21.15.32:0/3623552556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedac179610 0x7fedac179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.022 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 >> 172.21.15.32:0/3623552556 conn(0x7fedac190550 msgr2=0x7fedac1b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:24.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.022 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 shutdown_connections 2021-04-09T18:33:24.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.022 7fedb2bf6700 1 -- 172.21.15.32:0/3623552556 wait complete. 2021-04-09T18:33:24.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm alpha --yes-i-really-mean-it 2021-04-09T18:33:24.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.185 7fe041df3700 1 Processor -- start 2021-04-09T18:33:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.185 7fe041df3700 1 -- start start 2021-04-09T18:33:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe041df3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe03c0cffb0 0x7fe03c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe041df3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe03c0d7810 0x7fe03c0d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe041df3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0d8230 0x7fe03c0d5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe041df3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe03c0d91f0 con 0x7fe03c0d7810 2021-04-09T18:33:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe041df3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe03c0d7290 con 0x7fe03c0cffb0 2021-04-09T18:33:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe041df3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c0d62a0 con 0x7fe03c0d8230 2021-04-09T18:33:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03affd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe03c0d7810 0x7fe03c0d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03affd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe03c0d7810 0x7fe03c0d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58550/0 (socket says 172.21.15.32:58550) 2021-04-09T18:33:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03affd700 1 -- 172.21.15.32:0/1392535346 learned_addr learned my addr 172.21.15.32:0/1392535346 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03bfff700 1 --2- 172.21.15.32:0/1392535346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0d8230 0x7fe03c0d5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03b7fe700 1 --2- 172.21.15.32:0/1392535346 >> v2:172.21.15.86:3300/0 conn(0x7fe03c0cffb0 0x7fe03c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.227 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03affd700 1 -- 172.21.15.32:0/1392535346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0d8230 msgr2=0x7fe03c0d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03affd700 1 --2- 172.21.15.32:0/1392535346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0d8230 0x7fe03c0d5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03affd700 1 -- 172.21.15.32:0/1392535346 >> v2:172.21.15.86:3300/0 conn(0x7fe03c0cffb0 msgr2=0x7fe03c0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03affd700 1 --2- 172.21.15.32:0/1392535346 >> v2:172.21.15.86:3300/0 conn(0x7fe03c0cffb0 0x7fe03c0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03affd700 1 -- 172.21.15.32:0/1392535346 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c0d6490 con 0x7fe03c0d7810 2021-04-09T18:33:24.228 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03bfff700 1 --2- 172.21.15.32:0/1392535346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0d8230 0x7fe03c0d5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03b7fe700 1 --2- 172.21.15.32:0/1392535346 >> v2:172.21.15.86:3300/0 conn(0x7fe03c0cffb0 0x7fe03c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03affd700 1 --2- 172.21.15.32:0/1392535346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0d7810 0x7fe03c0d7c10 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 rx=0x7fe024006700 tx=0x7fe024009b80).ready entity=mon.0 client_cookie=a3d7fdcf63530016 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03a7fc700 1 -- 172.21.15.32:0/1392535346 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe02400c0c0 con 0x7fe03c0d7810 2021-04-09T18:33:24.229 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03a7fc700 1 -- 172.21.15.32:0/1392535346 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02400c440 con 0x7fe03c0d7810 2021-04-09T18:33:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.186 7fe03a7fc700 1 -- 172.21.15.32:0/1392535346 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe02400c920 con 0x7fe03c0d7810 2021-04-09T18:33:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.187 7fe041df3700 1 -- 172.21.15.32:0/1392535346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0d7810 msgr2=0x7fe03c0d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.187 7fe041df3700 1 --2- 172.21.15.32:0/1392535346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0d7810 0x7fe03c0d7c10 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 rx=0x7fe024006700 tx=0x7fe024009b80).stop 2021-04-09T18:33:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.187 7fe041df3700 1 -- 172.21.15.32:0/1392535346 shutdown_connections 2021-04-09T18:33:24.230 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.187 7fe041df3700 1 --2- 172.21.15.32:0/1392535346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0d8230 0x7fe03c0d5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.187 7fe041df3700 1 --2- 172.21.15.32:0/1392535346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c0d7810 0x7fe03c0d7c10 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.187 7fe041df3700 1 --2- 172.21.15.32:0/1392535346 >> v2:172.21.15.86:3300/0 conn(0x7fe03c0cffb0 0x7fe03c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.187 7fe041df3700 1 -- 172.21.15.32:0/1392535346 >> 172.21.15.32:0/1392535346 conn(0x7fe03c0ce5d0 msgr2=0x7fe03c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:24.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.187 7fe041df3700 1 -- 172.21.15.32:0/1392535346 shutdown_connections 2021-04-09T18:33:24.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.187 7fe041df3700 1 -- 172.21.15.32:0/1392535346 wait complete. 2021-04-09T18:33:24.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.207 7fe041df3700 1 Processor -- start 2021-04-09T18:33:24.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe041df3700 1 -- start start 2021-04-09T18:33:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe041df3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c1b8670 0x7fe03c1b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe041df3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe03c08c720 0x7fe03c08cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe041df3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0ab9c0 0x7fe03c0abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe041df3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c08c1a0 con 0x7fe03c1b8670 2021-04-09T18:33:24.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe041df3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c1819c0 con 0x7fe03c08c720 2021-04-09T18:33:24.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe041df3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c08d140 con 0x7fe03c0ab9c0 2021-04-09T18:33:24.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe03b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c1b8670 0x7fe03c1b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe03b7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c1b8670 0x7fe03c1b8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58552/0 (socket says 172.21.15.32:58552) 2021-04-09T18:33:24.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe03b7fe700 1 -- 172.21.15.32:0/2633349360 learned_addr learned my addr 172.21.15.32:0/2633349360 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:24.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe03bfff700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0ab9c0 0x7fe03c0abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.208 7fe03affd700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe03c08c720 0x7fe03c08cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe03b7fe700 1 -- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0ab9c0 msgr2=0x7fe03c0abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe03b7fe700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0ab9c0 0x7fe03c0abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe03b7fe700 1 -- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe03c08c720 msgr2=0x7fe03c08cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe03b7fe700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe03c08c720 0x7fe03c08cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe03b7fe700 1 -- 172.21.15.32:0/2633349360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c0ac3e0 con 0x7fe03c1b8670 2021-04-09T18:33:24.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe03b7fe700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c1b8670 0x7fe03c1b8a70 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7fe034006f20 tx=0x7fe03400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:24.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe033fff700 1 -- 172.21.15.32:0/2633349360 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe03400bd80 con 0x7fe03c1b8670 2021-04-09T18:33:24.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe033fff700 1 -- 172.21.15.32:0/2633349360 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe034000c90 con 0x7fe03c1b8670 2021-04-09T18:33:24.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe033fff700 1 -- 172.21.15.32:0/2633349360 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0340022b0 con 0x7fe03c1b8670 2021-04-09T18:33:24.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe041df3700 1 -- 172.21.15.32:0/2633349360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03c0ac3e0 con 0x7fe03c1b8670 2021-04-09T18:33:24.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.209 7fe041df3700 1 -- 172.21.15.32:0/2633349360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03c0ac3e0 con 0x7fe03c1b8670 2021-04-09T18:33:24.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.210 7fe033fff700 1 -- 172.21.15.32:0/2633349360 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe03400a000 con 0x7fe03c1b8670 2021-04-09T18:33:24.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.210 7fe033fff700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0200188c0 0x7fe02001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.210 7fe033fff700 1 -- 172.21.15.32:0/2633349360 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6905+0+0 (secure 0 0 0) 0x7fe034032da0 con 0x7fe03c1b8670 2021-04-09T18:33:24.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.210 7fe03affd700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0200188c0 0x7fe02001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.210 7fe03affd700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0200188c0 0x7fe02001adc0 crc :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:24.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.230 7fe041df3700 1 -- 172.21.15.32:0/2633349360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03c1b7180 con 0x7fe03c1b8670 2021-04-09T18:33:24.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.233 7fe033fff700 1 -- 172.21.15.32:0/2633349360 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe034006bb0 con 0x7fe03c1b8670 2021-04-09T18:33:24.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.427 7fe041df3700 1 -- 172.21.15.32:0/2633349360 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "alpha", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe03c0d7290 con 0x7fe03c1b8670 2021-04-09T18:33:24.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.572 7fe033fff700 1 -- 172.21.15.32:0/2633349360 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "alpha", "yes_i_really_mean_it": true}]=0 v404) v1 ==== 103+0+0 (secure 0 0 0) 0x7fe03400a000 con 0x7fe03c1b8670 2021-04-09T18:33:24.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 -- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0200188c0 msgr2=0x7fe02001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0200188c0 0x7fe02001adc0 crc :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 -- 172.21.15.32:0/2633349360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c1b8670 msgr2=0x7fe03c1b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c1b8670 0x7fe03c1b8a70 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7fe034006f20 tx=0x7fe03400b990).stop 2021-04-09T18:33:24.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 -- 172.21.15.32:0/2633349360 shutdown_connections 2021-04-09T18:33:24.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0200188c0 0x7fe02001adc0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe03c0ab9c0 0x7fe03c0abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe03c08c720 0x7fe03c08cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 --2- 172.21.15.32:0/2633349360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe03c1b8670 0x7fe03c1b8a70 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 -- 172.21.15.32:0/2633349360 >> 172.21.15.32:0/2633349360 conn(0x7fe03c189800 msgr2=0x7fe03c1b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:24.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 -- 172.21.15.32:0/2633349360 shutdown_connections 2021-04-09T18:33:24.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.593 7fe041df3700 1 -- 172.21.15.32:0/2633349360 wait complete. 2021-04-09T18:33:24.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete alpha_metadata alpha_metadata --yes-i-really-really-mean-it 2021-04-09T18:33:24.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.764 7fc50eba6700 1 Processor -- start 2021-04-09T18:33:24.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.764 7fc50eba6700 1 -- start start 2021-04-09T18:33:24.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc50eba6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc50810d050 0x7fc5080cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc50eba6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc5080d0160 0x7fc5080d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc50eba6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5080d82e0 0x7fc50810da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc50eba6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5080d8d70 con 0x7fc5080d0160 2021-04-09T18:33:24.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc50eba6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5080d9260 con 0x7fc50810d050 2021-04-09T18:33:24.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc50eba6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50810d530 con 0x7fc5080d82e0 2021-04-09T18:33:24.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc507fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc5080d0160 0x7fc5080d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc507fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc5080d0160 0x7fc5080d7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58560/0 (socket says 172.21.15.32:58560) 2021-04-09T18:33:24.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc507fff700 1 -- 172.21.15.32:0/396462979 learned_addr learned my addr 172.21.15.32:0/396462979 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:24.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc50d1c3700 1 --2- 172.21.15.32:0/396462979 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5080d82e0 0x7fc50810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc50c9c2700 1 --2- 172.21.15.32:0/396462979 >> v2:172.21.15.86:3300/0 conn(0x7fc50810d050 0x7fc5080cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc507fff700 1 -- 172.21.15.32:0/396462979 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5080d82e0 msgr2=0x7fc50810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:24.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc507fff700 1 --2- 172.21.15.32:0/396462979 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5080d82e0 0x7fc50810da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc507fff700 1 -- 172.21.15.32:0/396462979 >> v2:172.21.15.86:3300/0 conn(0x7fc50810d050 msgr2=0x7fc5080cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:24.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc507fff700 1 --2- 172.21.15.32:0/396462979 >> v2:172.21.15.86:3300/0 conn(0x7fc50810d050 0x7fc5080cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc507fff700 1 -- 172.21.15.32:0/396462979 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080d86e0 con 0x7fc5080d0160 2021-04-09T18:33:24.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.765 7fc50c9c2700 1 --2- 172.21.15.32:0/396462979 >> v2:172.21.15.86:3300/0 conn(0x7fc50810d050 0x7fc5080cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:24.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.766 7fc507fff700 1 --2- 172.21.15.32:0/396462979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080d0160 0x7fc5080d7cc0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 rx=0x7fc4f8006a80 tx=0x7fc4f8009e80).ready entity=mon.0 client_cookie=8a304182237acede server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:24.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.766 7fc5077fe700 1 -- 172.21.15.32:0/396462979 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4f800b510 con 0x7fc5080d0160 2021-04-09T18:33:24.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.766 7fc5077fe700 1 -- 172.21.15.32:0/396462979 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f800b7d0 con 0x7fc5080d0160 2021-04-09T18:33:24.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.766 7fc5077fe700 1 -- 172.21.15.32:0/396462979 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4f800bcb0 con 0x7fc5080d0160 2021-04-09T18:33:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.766 7fc50eba6700 1 -- 172.21.15.32:0/396462979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080d0160 msgr2=0x7fc5080d7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.766 7fc50eba6700 1 --2- 172.21.15.32:0/396462979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080d0160 0x7fc5080d7cc0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 rx=0x7fc4f8006a80 tx=0x7fc4f8009e80).stop 2021-04-09T18:33:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.767 7fc50eba6700 1 -- 172.21.15.32:0/396462979 shutdown_connections 2021-04-09T18:33:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.767 7fc50eba6700 1 --2- 172.21.15.32:0/396462979 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5080d82e0 0x7fc50810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.767 7fc50eba6700 1 --2- 172.21.15.32:0/396462979 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080d0160 0x7fc5080d7cc0 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.767 7fc50eba6700 1 --2- 172.21.15.32:0/396462979 >> v2:172.21.15.86:3300/0 conn(0x7fc50810d050 0x7fc5080cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.767 7fc50eba6700 1 -- 172.21.15.32:0/396462979 >> 172.21.15.32:0/396462979 conn(0x7fc5080ce620 msgr2=0x7fc5080d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:24.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.767 7fc50eba6700 1 -- 172.21.15.32:0/396462979 shutdown_connections 2021-04-09T18:33:24.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.767 7fc50eba6700 1 -- 172.21.15.32:0/396462979 wait complete. 2021-04-09T18:33:24.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.787 7fc50eba6700 1 Processor -- start 2021-04-09T18:33:24.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.787 7fc50eba6700 1 -- start start 2021-04-09T18:33:24.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.787 7fc50eba6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5081799a0 0x7fc5080a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.787 7fc50eba6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080a1360 0x7fc5080bb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.787 7fc50eba6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc508179da0 0x7fc5080bbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.787 7fc50eba6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50817a2e0 con 0x7fc5080a1360 2021-04-09T18:33:24.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.787 7fc50eba6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50817a860 con 0x7fc508179da0 2021-04-09T18:33:24.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.787 7fc50eba6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50817aa80 con 0x7fc5081799a0 2021-04-09T18:33:24.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.787 7fc507fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080a1360 0x7fc5080bb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc507fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080a1360 0x7fc5080bb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58568/0 (socket says 172.21.15.32:58568) 2021-04-09T18:33:24.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc507fff700 1 -- 172.21.15.32:0/922654475 learned_addr learned my addr 172.21.15.32:0/922654475 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:24.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc50c9c2700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5081799a0 0x7fc5080a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc50d1c3700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc508179da0 0x7fc5080bbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc507fff700 1 -- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5081799a0 msgr2=0x7fc5080a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc507fff700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5081799a0 0x7fc5080a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc507fff700 1 -- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc508179da0 msgr2=0x7fc5080bbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:24.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc507fff700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc508179da0 0x7fc5080bbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:24.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc507fff700 1 -- 172.21.15.32:0/922654475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50817aca0 con 0x7fc5080a1360 2021-04-09T18:33:24.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc50c9c2700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5081799a0 0x7fc5080a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:24.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc50d1c3700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc508179da0 0x7fc5080bbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:24.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc507fff700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080a1360 0x7fc5080bb500 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 rx=0x7fc4f800b970 tx=0x7fc4f800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:24.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc5057fa700 1 -- 172.21.15.32:0/922654475 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4f8000d30 con 0x7fc5080a1360 2021-04-09T18:33:24.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc5057fa700 1 -- 172.21.15.32:0/922654475 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8006240 con 0x7fc5080a1360 2021-04-09T18:33:24.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc50eba6700 1 -- 172.21.15.32:0/922654475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50817aca0 con 0x7fc5080a1360 2021-04-09T18:33:24.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc5057fa700 1 -- 172.21.15.32:0/922654475 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4f8005660 con 0x7fc5080a1360 2021-04-09T18:33:24.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.788 7fc50eba6700 1 -- 172.21.15.32:0/922654475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50817aca0 con 0x7fc5080a1360 2021-04-09T18:33:24.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.789 7fc5057fa700 1 -- 172.21.15.32:0/922654475 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc4f8006420 con 0x7fc5080a1360 2021-04-09T18:33:24.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.789 7fc5057fa700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4ec0188c0 0x7fc4ec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:24.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.789 7fc5057fa700 1 -- 172.21.15.32:0/922654475 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6905+0+0 (secure 0 0 0) 0x7fc4f80331c0 con 0x7fc5080a1360 2021-04-09T18:33:24.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.789 7fc50c9c2700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4ec0188c0 0x7fc4ec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:24.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.790 7fc50c9c2700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4ec0188c0 0x7fc4ec01adc0 crc :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:24.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.809 7fc50eba6700 1 -- 172.21.15.32:0/922654475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5080f1950 con 0x7fc5080a1360 2021-04-09T18:33:24.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.812 7fc5057fa700 1 -- 172.21.15.32:0/922654475 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc4f8006030 con 0x7fc5080a1360 2021-04-09T18:33:25.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:24.997 7fc50eba6700 1 -- 172.21.15.32:0/922654475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "alpha_metadata", "prefix": "osd pool delete", "pool": "alpha_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc50810d500 con 0x7fc5080a1360 2021-04-09T18:33:25.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.577 7fc5057fa700 1 -- 172.21.15.32:0/922654475 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "alpha_metadata", "prefix": "osd pool delete", "pool": "alpha_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'alpha_metadata' removed v239) v1 ==== 182+0+0 (secure 0 0 0) 0x7fc4f801f850 con 0x7fc5080a1360 2021-04-09T18:33:25.629 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_metadata' removed 2021-04-09T18:33:25.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 -- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4ec0188c0 msgr2=0x7fc4ec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:25.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4ec0188c0 0x7fc4ec01adc0 crc :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 -- 172.21.15.32:0/922654475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080a1360 msgr2=0x7fc5080bb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:25.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080a1360 0x7fc5080bb500 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 rx=0x7fc4f800b970 tx=0x7fc4f800bd10).stop 2021-04-09T18:33:25.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 -- 172.21.15.32:0/922654475 shutdown_connections 2021-04-09T18:33:25.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc4ec0188c0 0x7fc4ec01adc0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5080a1360 0x7fc5080bb500 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc508179da0 0x7fc5080bbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 --2- 172.21.15.32:0/922654475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5081799a0 0x7fc5080a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.614 7fc50eba6700 1 -- 172.21.15.32:0/922654475 >> 172.21.15.32:0/922654475 conn(0x7fc5080c3890 msgr2=0x7fc50817fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:25.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.615 7fc50eba6700 1 -- 172.21.15.32:0/922654475 shutdown_connections 2021-04-09T18:33:25.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.615 7fc50eba6700 1 -- 172.21.15.32:0/922654475 wait complete. 2021-04-09T18:33:25.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete alpha_data alpha_data --yes-i-really-really-mean-it 2021-04-09T18:33:25.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.787 7f0767c12700 1 Processor -- start 2021-04-09T18:33:25.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.788 7f0767c12700 1 -- start start 2021-04-09T18:33:25.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.788 7f0767c12700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f076010d050 0x7f07600cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:25.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.788 7f0767c12700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07600d0160 0x7f07600d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:25.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.788 7f0767c12700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076010d500 con 0x7f07600d82e0 2021-04-09T18:33:25.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.788 7f0767c12700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07600d9260 con 0x7f076010d050 2021-04-09T18:33:25.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.788 7f0767c12700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07600d86e0 con 0x7f07600d0160 2021-04-09T18:33:25.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.788 7f076622f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f07600d82e0 0x7f076010da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39934/0 (socket says 172.21.15.32:39934) 2021-04-09T18:33:25.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.788 7f076622f700 1 -- 172.21.15.32:0/1599667632 learned_addr learned my addr 172.21.15.32:0/1599667632 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:25.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f076522d700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07600d0160 0x7f07600d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:25.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0765a2e700 1 --2- 172.21.15.32:0/1599667632 >> v2:172.21.15.86:3300/0 conn(0x7f076010d050 0x7f07600cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:25.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393146819 0 0) 0x7f0758000d30 con 0x7f07600d82e0 2021-04-09T18:33:25.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f074c002fe0 con 0x7f07600d82e0 2021-04-09T18:33:25.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2597906061 0 0) 0x7f0758001460 con 0x7f07600d82e0 2021-04-09T18:33:25.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07600d0160 msgr2=0x7f07600d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:25.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07600d0160 0x7f07600d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 >> v2:172.21.15.86:3300/0 conn(0x7f076010d050 msgr2=0x7f07600cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:25.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 --2- 172.21.15.32:0/1599667632 >> v2:172.21.15.86:3300/0 conn(0x7f076010d050 0x7f07600cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076010e270 con 0x7f07600d82e0 2021-04-09T18:33:25.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f0758001980 con 0x7f07600d82e0 2021-04-09T18:33:25.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 >> v1:172.21.15.32:6789/0 conn(0x7f07600d82e0 legacy=0x7f076010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:25.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f074c003af0 0x7f074c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:25.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f074c04a850 0x7f074c04cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:25.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f074c04d5c0 0x7f074c04fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:25.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c050390 con 0x7f074c04d5c0 2021-04-09T18:33:25.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0505b0 con 0x7f074c003af0 2021-04-09T18:33:25.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0507d0 con 0x7f074c04a850 2021-04-09T18:33:25.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0765a2e700 1 --2- 172.21.15.32:0/1599667632 >> v2:172.21.15.86:3300/0 conn(0x7f076010d050 0x7f07600cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:25.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f076622f700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f074c04d5c0 0x7f074c04fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:25.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f076522d700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f074c04a850 0x7f074c04cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:25.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0765a2e700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f074c003af0 0x7f074c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:25.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f076622f700 1 -- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f074c04a850 msgr2=0x7f074c04cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:25.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f076622f700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f074c04a850 0x7f074c04cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f076622f700 1 -- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f074c003af0 msgr2=0x7f074c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f076622f700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f074c003af0 0x7f074c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f076622f700 1 -- 172.21.15.32:0/1599667632 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c050a70 con 0x7f074c04d5c0 2021-04-09T18:33:25.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.789 7f0765a2e700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f074c003af0 0x7f074c005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:25.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.790 7f076622f700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f074c04d5c0 0x7f074c04fae0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 rx=0x7f0758001c80 tx=0x7f075800a060).ready entity=mon.0 client_cookie=7792600ff779b785 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:25.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.790 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07580070c0 con 0x7f074c04d5c0 2021-04-09T18:33:25.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.790 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07580017f0 con 0x7f074c04d5c0 2021-04-09T18:33:25.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.790 7f0764a2c700 1 -- 172.21.15.32:0/1599667632 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0758004720 con 0x7f074c04d5c0 2021-04-09T18:33:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.790 7f0767c12700 1 -- 172.21.15.32:0/1599667632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f074c04d5c0 msgr2=0x7f074c04fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.790 7f0767c12700 1 --2- 172.21.15.32:0/1599667632 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f074c04d5c0 0x7f074c04fae0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 rx=0x7f0758001c80 tx=0x7f075800a060).stop 2021-04-09T18:33:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.790 7f076622f700 1 -- 172.21.15.32:0/1599667632 reap_dead start 2021-04-09T18:33:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.790 7f0767c12700 1 -- 172.21.15.32:0/1599667632 shutdown_connections 2021-04-09T18:33:25.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.790 7f0767c12700 1 -- 172.21.15.32:0/1599667632 >> 172.21.15.32:0/1599667632 conn(0x7f07600ce620 msgr2=0x7f07600d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:25.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.791 7f0767c12700 1 -- 172.21.15.32:0/1599667632 shutdown_connections 2021-04-09T18:33:25.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.791 7f0767c12700 1 -- 172.21.15.32:0/1599667632 wait complete. 2021-04-09T18:33:25.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.811 7f0767c12700 1 Processor -- start 2021-04-09T18:33:25.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.811 7f0767c12700 1 -- start start 2021-04-09T18:33:25.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.811 7f0767c12700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f076009b2f0 0x7f076009b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.811 7f0767c12700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f076009a8b0 0x7f076009e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.811 7f0767c12700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076009e9c0 0x7f076009edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.811 7f0767c12700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07600bde80 con 0x7f076009e9c0 2021-04-09T18:33:25.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.811 7f0767c12700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076009ad70 con 0x7f076009a8b0 2021-04-09T18:33:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.811 7f0767c12700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076009bd10 con 0x7f076009b2f0 2021-04-09T18:33:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076622f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076009e9c0 0x7f076009edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076622f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076009e9c0 0x7f076009edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58590/0 (socket says 172.21.15.32:58590) 2021-04-09T18:33:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076622f700 1 -- 172.21.15.32:0/3708104092 learned_addr learned my addr 172.21.15.32:0/3708104092 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:25.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f0765a2e700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f076009b2f0 0x7f076009b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076522d700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f076009a8b0 0x7f076009e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076622f700 1 -- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f076009b2f0 msgr2=0x7f076009b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076622f700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f076009b2f0 0x7f076009b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076622f700 1 -- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f076009a8b0 msgr2=0x7f076009e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076622f700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f076009a8b0 0x7f076009e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076622f700 1 -- 172.21.15.32:0/3708104092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076009fbd0 con 0x7f076009e9c0 2021-04-09T18:33:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f076622f700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076009e9c0 0x7f076009edc0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f075800a880 tx=0x7f075800ac20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:25.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f07567fc700 1 -- 172.21.15.32:0/3708104092 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0758007880 con 0x7f076009e9c0 2021-04-09T18:33:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f07567fc700 1 -- 172.21.15.32:0/3708104092 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0758004930 con 0x7f076009e9c0 2021-04-09T18:33:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f0767c12700 1 -- 172.21.15.32:0/3708104092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076009fbd0 con 0x7f076009e9c0 2021-04-09T18:33:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f07567fc700 1 -- 172.21.15.32:0/3708104092 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0758004d80 con 0x7f076009e9c0 2021-04-09T18:33:25.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.812 7f0767c12700 1 -- 172.21.15.32:0/3708104092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076009fbd0 con 0x7f076009e9c0 2021-04-09T18:33:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.813 7f07567fc700 1 -- 172.21.15.32:0/3708104092 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f075800c300 con 0x7f076009e9c0 2021-04-09T18:33:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.813 7f07567fc700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0748018930 0x7f074801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.813 7f07567fc700 1 -- 172.21.15.32:0/3708104092 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6469+0+0 (secure 0 0 0) 0x7f0758035310 con 0x7f076009e9c0 2021-04-09T18:33:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.813 7f0765a2e700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0748018930 0x7f074801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:25.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.814 7f0765a2e700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0748018930 0x7f074801ae10 crc :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:25.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.832 7f0767c12700 1 -- 172.21.15.32:0/3708104092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07600bde80 con 0x7f076009e9c0 2021-04-09T18:33:25.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:25.836 7f07567fc700 1 -- 172.21.15.32:0/3708104092 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0758004720 con 0x7f076009e9c0 2021-04-09T18:33:26.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.020 7f0767c12700 1 -- 172.21.15.32:0/3708104092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "alpha_data", "prefix": "osd pool delete", "pool": "alpha_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f07600bde80 con 0x7f076009e9c0 2021-04-09T18:33:26.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.581 7f07567fc700 1 -- 172.21.15.32:0/3708104092 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "alpha_data", "prefix": "osd pool delete", "pool": "alpha_data", "yes_i_really_really_mean_it": true}]=0 pool 'alpha_data' removed v240) v1 ==== 170+0+0 (secure 0 0 0) 0x7f075800c300 con 0x7f076009e9c0 2021-04-09T18:33:26.602 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'alpha_data' removed 2021-04-09T18:33:26.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 -- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0748018930 msgr2=0x7f074801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0748018930 0x7f074801ae10 crc :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 -- 172.21.15.32:0/3708104092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076009e9c0 msgr2=0x7f076009edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076009e9c0 0x7f076009edc0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f075800a880 tx=0x7f075800ac20).stop 2021-04-09T18:33:26.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 -- 172.21.15.32:0/3708104092 shutdown_connections 2021-04-09T18:33:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0748018930 0x7f074801ae10 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f076009e9c0 0x7f076009edc0 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f076009a8b0 0x7f076009e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 --2- 172.21.15.32:0/3708104092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f076009b2f0 0x7f076009b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 -- 172.21.15.32:0/3708104092 >> 172.21.15.32:0/3708104092 conn(0x7f07600ae6f0 msgr2=0x7f0760097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 -- 172.21.15.32:0/3708104092 shutdown_connections 2021-04-09T18:33:26.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.588 7f0767c12700 1 -- 172.21.15.32:0/3708104092 wait complete. 2021-04-09T18:33:26.619 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm bravo --yes-i-really-mean-it 2021-04-09T18:33:26.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.757 7fd498633700 1 Processor -- start 2021-04-09T18:33:26.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.757 7fd498633700 1 -- start start 2021-04-09T18:33:26.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd498633700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd4900cffb0 0x7fd4900d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:26.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd498633700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4900d6c90 0x7fd49010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:26.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd498633700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4900d7290 con 0x7fd4900d7810 2021-04-09T18:33:26.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd498633700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4900d7090 con 0x7fd4900cffb0 2021-04-09T18:33:26.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd498633700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4900d91f0 con 0x7fd4900d6c90 2021-04-09T18:33:26.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd495c4e700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd4900d7810 0x7fd4900d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:37880/0 (socket says 172.21.15.32:37880) 2021-04-09T18:33:26.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd495c4e700 1 -- 172.21.15.32:0/1708268734 learned_addr learned my addr 172.21.15.32:0/1708268734 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:26.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd49644f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd4900cffb0 0x7fd4900d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:26.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd496c50700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4900d6c90 0x7fd49010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:26.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd49544d700 1 -- 172.21.15.32:0/1708268734 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508421418 0 0) 0x7fd4800010c0 con 0x7fd4900d7810 2021-04-09T18:33:26.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.758 7fd49544d700 1 -- 172.21.15.32:0/1708268734 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47c002fe0 con 0x7fd4900d7810 2021-04-09T18:33:26.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd496c50700 1 -- 172.21.15.32:0/1708268734 >> v2:172.21.15.86:3300/0 conn(0x7fd4900cffb0 msgr2=0x7fd4900d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:26.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd496c50700 1 --2- 172.21.15.32:0/1708268734 >> v2:172.21.15.86:3300/0 conn(0x7fd4900cffb0 0x7fd4900d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd496c50700 1 -- 172.21.15.32:0/1708268734 >> v1:172.21.15.86:6789/0 conn(0x7fd4900d7810 legacy=0x7fd4900d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:26.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd496c50700 1 -- 172.21.15.32:0/1708268734 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4900d8400 con 0x7fd4900d6c90 2021-04-09T18:33:26.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd49644f700 1 --2- 172.21.15.32:0/1708268734 >> v2:172.21.15.86:3300/0 conn(0x7fd4900cffb0 0x7fd4900d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:26.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd496c50700 1 --2- 172.21.15.32:0/1708268734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4900d6c90 0x7fd49010cfe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fd488006e50 tx=0x7fd488009e70).ready entity=mon.2 client_cookie=73f49ae42ff93117 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd49544d700 1 -- 172.21.15.32:0/1708268734 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd488006a20 con 0x7fd4900d6c90 2021-04-09T18:33:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd49544d700 1 -- 172.21.15.32:0/1708268734 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd488006790 con 0x7fd4900d6c90 2021-04-09T18:33:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd49544d700 1 -- 172.21.15.32:0/1708268734 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd48800b700 con 0x7fd4900d6c90 2021-04-09T18:33:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd498633700 1 -- 172.21.15.32:0/1708268734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4900d6c90 msgr2=0x7fd49010cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:26.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.759 7fd498633700 1 --2- 172.21.15.32:0/1708268734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4900d6c90 0x7fd49010cfe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fd488006e50 tx=0x7fd488009e70).stop 2021-04-09T18:33:26.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.760 7fd498633700 1 -- 172.21.15.32:0/1708268734 shutdown_connections 2021-04-09T18:33:26.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.760 7fd498633700 1 --2- 172.21.15.32:0/1708268734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4900d6c90 0x7fd49010cfe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.760 7fd498633700 1 --2- 172.21.15.32:0/1708268734 >> v2:172.21.15.86:3300/0 conn(0x7fd4900cffb0 0x7fd4900d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.760 7fd498633700 1 -- 172.21.15.32:0/1708268734 >> 172.21.15.32:0/1708268734 conn(0x7fd4900ce5d0 msgr2=0x7fd4900cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:26.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.760 7fd498633700 1 -- 172.21.15.32:0/1708268734 shutdown_connections 2021-04-09T18:33:26.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.760 7fd498633700 1 -- 172.21.15.32:0/1708268734 wait complete. 2021-04-09T18:33:26.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.780 7fd498633700 1 Processor -- start 2021-04-09T18:33:26.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.780 7fd498633700 1 -- start start 2021-04-09T18:33:26.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.780 7fd498633700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49017d2b0 0x7fd490181640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:26.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.780 7fd498633700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd49017d6b0 0x7fd490181c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:26.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.780 7fd498633700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49017ff40 0x7fd490180340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:26.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.780 7fd498633700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4901810c0 con 0x7fd49017d6b0 2021-04-09T18:33:26.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd498633700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd490180960 con 0x7fd49017d2b0 2021-04-09T18:33:26.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd498633700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd490180b40 con 0x7fd49017ff40 2021-04-09T18:33:26.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd495c4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd49017d6b0 0x7fd490181c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:26.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd495c4e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd49017d6b0 0x7fd490181c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58600/0 (socket says 172.21.15.32:58600) 2021-04-09T18:33:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd495c4e700 1 -- 172.21.15.32:0/1810524954 learned_addr learned my addr 172.21.15.32:0/1810524954 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd496c50700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49017ff40 0x7fd490180340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd49644f700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49017d2b0 0x7fd490181640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd495c4e700 1 -- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49017ff40 msgr2=0x7fd490180340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd495c4e700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49017ff40 0x7fd490180340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd495c4e700 1 -- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49017d2b0 msgr2=0x7fd490181640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd495c4e700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49017d2b0 0x7fd490181640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd495c4e700 1 -- 172.21.15.32:0/1810524954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd490180d60 con 0x7fd49017d6b0 2021-04-09T18:33:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd495c4e700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd49017d6b0 0x7fd490181c60 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 rx=0x7fd48000a550 tx=0x7fd48000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd486ffd700 1 -- 172.21.15.32:0/1810524954 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd480000e70 con 0x7fd49017d6b0 2021-04-09T18:33:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd486ffd700 1 -- 172.21.15.32:0/1810524954 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48000ace0 con 0x7fd49017d6b0 2021-04-09T18:33:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd486ffd700 1 -- 172.21.15.32:0/1810524954 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd480004510 con 0x7fd49017d6b0 2021-04-09T18:33:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd498633700 1 -- 172.21.15.32:0/1810524954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd490180d60 con 0x7fd49017d6b0 2021-04-09T18:33:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.781 7fd498633700 1 -- 172.21.15.32:0/1810524954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd490180d60 con 0x7fd49017d6b0 2021-04-09T18:33:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.782 7fd486ffd700 1 -- 172.21.15.32:0/1810524954 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd480004320 con 0x7fd49017d6b0 2021-04-09T18:33:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.782 7fd486ffd700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd478018930 0x7fd47801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.782 7fd486ffd700 1 -- 172.21.15.32:0/1810524954 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6089+0+0 (secure 0 0 0) 0x7fd480032d40 con 0x7fd49017d6b0 2021-04-09T18:33:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.782 7fd49644f700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd478018930 0x7fd47801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:26.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.783 7fd49644f700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd478018930 0x7fd47801ae10 crc :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:26.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.802 7fd498633700 1 -- 172.21.15.32:0/1810524954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4900d7290 con 0x7fd49017d6b0 2021-04-09T18:33:26.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.805 7fd486ffd700 1 -- 172.21.15.32:0/1810524954 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd480000c10 con 0x7fd49017d6b0 2021-04-09T18:33:27.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:26.999 7fd498633700 1 -- 172.21.15.32:0/1810524954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "bravo", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd4900d7290 con 0x7fd49017d6b0 2021-04-09T18:33:27.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.588 7fd486ffd700 1 -- 172.21.15.32:0/1810524954 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "bravo", "yes_i_really_mean_it": true}]=0 v405) v1 ==== 103+0+0 (secure 0 0 0) 0x7fd480004320 con 0x7fd49017d6b0 2021-04-09T18:33:27.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 -- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd478018930 msgr2=0x7fd47801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:27.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd478018930 0x7fd47801ae10 crc :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 -- 172.21.15.32:0/1810524954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd49017d6b0 msgr2=0x7fd490181c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:27.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd49017d6b0 0x7fd490181c60 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 rx=0x7fd48000a550 tx=0x7fd48000a8f0).stop 2021-04-09T18:33:27.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 -- 172.21.15.32:0/1810524954 shutdown_connections 2021-04-09T18:33:27.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd478018930 0x7fd47801ae10 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd49017d6b0 0x7fd490181c60 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49017ff40 0x7fd490180340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 --2- 172.21.15.32:0/1810524954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49017d2b0 0x7fd490181640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 -- 172.21.15.32:0/1810524954 >> 172.21.15.32:0/1810524954 conn(0x7fd49008e920 msgr2=0x7fd490192cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:27.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 -- 172.21.15.32:0/1810524954 shutdown_connections 2021-04-09T18:33:27.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.618 7fd498633700 1 -- 172.21.15.32:0/1810524954 wait complete. 2021-04-09T18:33:27.645 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete bravo_metadata bravo_metadata --yes-i-really-really-mean-it 2021-04-09T18:33:27.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff291b36700 1 Processor -- start 2021-04-09T18:33:27.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff291b36700 1 -- start start 2021-04-09T18:33:27.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff291b36700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff28c10d050 0x7ff28c0cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:27.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff291b36700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff28c0d0160 0x7ff28c0d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:27.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff291b36700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff28c10d500 con 0x7ff28c0d82e0 2021-04-09T18:33:27.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff291b36700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff28c0d9260 con 0x7ff28c10d050 2021-04-09T18:33:27.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff291b36700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff28c0d86e0 con 0x7ff28c0d0160 2021-04-09T18:33:27.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff28b7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff28c10d050 0x7ff28c0cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:27.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff28bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff28c0d82e0 0x7ff28c10da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39970/0 (socket says 172.21.15.32:39970) 2021-04-09T18:33:27.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff28b7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff28c10d050 0x7ff28c0cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58610/0 (socket says 172.21.15.32:58610) 2021-04-09T18:33:27.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.793 7ff28bfff700 1 -- 172.21.15.32:0/3924288550 learned_addr learned my addr 172.21.15.32:0/3924288550 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:27.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28affd700 1 --2- 172.21.15.32:0/3924288550 >> v2:172.21.15.86:3300/0 conn(0x7ff28c0d0160 0x7ff28c0d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:27.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28b7fe700 1 -- 172.21.15.32:0/3924288550 >> v2:172.21.15.86:3300/0 conn(0x7ff28c0d0160 msgr2=0x7ff28c0d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:27.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28b7fe700 1 --2- 172.21.15.32:0/3924288550 >> v2:172.21.15.86:3300/0 conn(0x7ff28c0d0160 0x7ff28c0d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28b7fe700 1 -- 172.21.15.32:0/3924288550 >> v1:172.21.15.32:6789/0 conn(0x7ff28c0d82e0 legacy=0x7ff28c10da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:27.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28b7fe700 1 -- 172.21.15.32:0/3924288550 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c10e270 con 0x7ff28c10d050 2021-04-09T18:33:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28a7fc700 1 -- 172.21.15.32:0/3924288550 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835924638 0 0) 0x7ff284000d30 con 0x7ff28c0d82e0 2021-04-09T18:33:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28b7fe700 1 --2- 172.21.15.32:0/3924288550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c10d050 0x7ff28c0cfb40 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 rx=0x7ff280006650 tx=0x7ff280009a00).ready entity=mon.0 client_cookie=4aaccb4381f79f35 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28a7fc700 1 -- 172.21.15.32:0/3924288550 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff280009df0 con 0x7ff28c10d050 2021-04-09T18:33:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28a7fc700 1 -- 172.21.15.32:0/3924288550 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2800009b0 con 0x7ff28c10d050 2021-04-09T18:33:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff28a7fc700 1 -- 172.21.15.32:0/3924288550 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff28000c2b0 con 0x7ff28c10d050 2021-04-09T18:33:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff291b36700 1 -- 172.21.15.32:0/3924288550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c10d050 msgr2=0x7ff28c0cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.794 7ff291b36700 1 --2- 172.21.15.32:0/3924288550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c10d050 0x7ff28c0cfb40 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 rx=0x7ff280006650 tx=0x7ff280009a00).stop 2021-04-09T18:33:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.795 7ff291b36700 1 -- 172.21.15.32:0/3924288550 shutdown_connections 2021-04-09T18:33:27.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.795 7ff291b36700 1 --2- 172.21.15.32:0/3924288550 >> v2:172.21.15.86:3300/0 conn(0x7ff28c0d0160 0x7ff28c0d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.795 7ff291b36700 1 --2- 172.21.15.32:0/3924288550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c10d050 0x7ff28c0cfb40 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.795 7ff291b36700 1 -- 172.21.15.32:0/3924288550 >> 172.21.15.32:0/3924288550 conn(0x7ff28c0ce620 msgr2=0x7ff28c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.795 7ff291b36700 1 -- 172.21.15.32:0/3924288550 shutdown_connections 2021-04-09T18:33:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.795 7ff291b36700 1 -- 172.21.15.32:0/3924288550 wait complete. 2021-04-09T18:33:27.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.815 7ff291b36700 1 Processor -- start 2021-04-09T18:33:27.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.815 7ff291b36700 1 -- start start 2021-04-09T18:33:27.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff291b36700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff28c09b2f0 0x7ff28c09b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:27.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff291b36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c09a8b0 0x7ff28c09e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:27.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff291b36700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff28c09e9c0 0x7ff28c09edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:27.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff291b36700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c0bde80 con 0x7ff28c09a8b0 2021-04-09T18:33:27.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff291b36700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c09ad70 con 0x7ff28c09b2f0 2021-04-09T18:33:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff291b36700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c09bd10 con 0x7ff28c09e9c0 2021-04-09T18:33:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c09a8b0 0x7ff28c09e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c09a8b0 0x7ff28c09e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58618/0 (socket says 172.21.15.32:58618) 2021-04-09T18:33:27.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28affd700 1 -- 172.21.15.32:0/1851853841 learned_addr learned my addr 172.21.15.32:0/1851853841 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:27.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28b7fe700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff28c09b2f0 0x7ff28c09b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:27.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28bfff700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff28c09e9c0 0x7ff28c09edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:27.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28affd700 1 -- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff28c09e9c0 msgr2=0x7ff28c09edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:27.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28affd700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff28c09e9c0 0x7ff28c09edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28affd700 1 -- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff28c09b2f0 msgr2=0x7ff28c09b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28affd700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff28c09b2f0 0x7ff28c09b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28affd700 1 -- 172.21.15.32:0/1851853841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c09fbd0 con 0x7ff28c09a8b0 2021-04-09T18:33:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28b7fe700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff28c09b2f0 0x7ff28c09b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:27.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff28affd700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c09a8b0 0x7ff28c09e3a0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 rx=0x7ff27c006f60 tx=0x7ff27c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:27.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.816 7ff273fff700 1 -- 172.21.15.32:0/1851853841 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff27c0009b0 con 0x7ff28c09a8b0 2021-04-09T18:33:27.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.817 7ff273fff700 1 -- 172.21.15.32:0/1851853841 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27c00bd60 con 0x7ff28c09a8b0 2021-04-09T18:33:27.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.817 7ff291b36700 1 -- 172.21.15.32:0/1851853841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c09fbd0 con 0x7ff28c09a8b0 2021-04-09T18:33:27.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.817 7ff273fff700 1 -- 172.21.15.32:0/1851853841 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff27c002030 con 0x7ff28c09a8b0 2021-04-09T18:33:27.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.817 7ff291b36700 1 -- 172.21.15.32:0/1851853841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c09fbd0 con 0x7ff28c09a8b0 2021-04-09T18:33:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.817 7ff273fff700 1 -- 172.21.15.32:0/1851853841 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff27c000c90 con 0x7ff28c09a8b0 2021-04-09T18:33:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.817 7ff273fff700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff26c018930 0x7ff26c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:27.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.817 7ff273fff700 1 -- 172.21.15.32:0/1851853841 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6089+0+0 (secure 0 0 0) 0x7ff27c032c00 con 0x7ff28c09a8b0 2021-04-09T18:33:27.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.818 7ff28b7fe700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff26c018930 0x7ff26c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:27.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.818 7ff28b7fe700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff26c018930 0x7ff26c01adc0 crc :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:27.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.837 7ff291b36700 1 -- 172.21.15.32:0/1851853841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28c0b9040 con 0x7ff28c09a8b0 2021-04-09T18:33:27.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:27.841 7ff273fff700 1 -- 172.21.15.32:0/1851853841 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff27c006b00 con 0x7ff28c09a8b0 2021-04-09T18:33:28.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.025 7ff291b36700 1 -- 172.21.15.32:0/1851853841 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "bravo_metadata", "prefix": "osd pool delete", "pool": "bravo_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff28c10d500 con 0x7ff28c09a8b0 2021-04-09T18:33:28.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.638 7ff273fff700 1 -- 172.21.15.32:0/1851853841 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "bravo_metadata", "prefix": "osd pool delete", "pool": "bravo_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'bravo_metadata' removed v241) v1 ==== 182+0+0 (secure 0 0 0) 0x7ff27c000c90 con 0x7ff28c09a8b0 2021-04-09T18:33:28.656 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_metadata' removed 2021-04-09T18:33:28.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.642 7ff291b36700 1 -- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff26c018930 msgr2=0x7ff26c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:28.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.642 7ff291b36700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff26c018930 0x7ff26c01adc0 crc :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.642 7ff291b36700 1 -- 172.21.15.32:0/1851853841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c09a8b0 msgr2=0x7ff28c09e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:28.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.642 7ff291b36700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c09a8b0 0x7ff28c09e3a0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 rx=0x7ff27c006f60 tx=0x7ff27c00b970).stop 2021-04-09T18:33:28.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.643 7ff291b36700 1 -- 172.21.15.32:0/1851853841 shutdown_connections 2021-04-09T18:33:28.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.643 7ff291b36700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff26c018930 0x7ff26c01adc0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.643 7ff291b36700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff28c09a8b0 0x7ff28c09e3a0 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.643 7ff291b36700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff28c09e9c0 0x7ff28c09edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.643 7ff291b36700 1 --2- 172.21.15.32:0/1851853841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff28c09b2f0 0x7ff28c09b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.643 7ff291b36700 1 -- 172.21.15.32:0/1851853841 >> 172.21.15.32:0/1851853841 conn(0x7ff28c0ae6f0 msgr2=0x7ff28c097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.643 7ff291b36700 1 -- 172.21.15.32:0/1851853841 shutdown_connections 2021-04-09T18:33:28.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.643 7ff291b36700 1 -- 172.21.15.32:0/1851853841 wait complete. 2021-04-09T18:33:28.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete bravo_data bravo_data --yes-i-really-really-mean-it 2021-04-09T18:33:28.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.814 7f75fafe9700 1 Processor -- start 2021-04-09T18:33:28.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.814 7f75fafe9700 1 -- start start 2021-04-09T18:33:28.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.815 7f75fafe9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f75f410d050 0x7f75f40cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:28.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.815 7f75fafe9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f410d500 con 0x7f75f40d84b0 2021-04-09T18:33:28.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.815 7f75fafe9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f410e270 con 0x7f75f40d0160 2021-04-09T18:33:28.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.815 7f75fafe9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f40d8d70 con 0x7f75f410d050 2021-04-09T18:33:28.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.815 7f75f9606700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f75f40d84b0 0x7f75f410da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39986/0 (socket says 172.21.15.32:39986) 2021-04-09T18:33:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.815 7f75f9606700 1 -- 172.21.15.32:0/1704876423 learned_addr learned my addr 172.21.15.32:0/1704876423 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.815 7f75f8e05700 1 --2- 172.21.15.32:0/1704876423 >> v2:172.21.15.86:3300/0 conn(0x7f75f410d050 0x7f75f40cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:28.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634195648 0 0) 0x7f75ec000d30 con 0x7f75f40d84b0 2021-04-09T18:33:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc002fe0 con 0x7f75f40d84b0 2021-04-09T18:33:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770578888 0 0) 0x7f75e8000c60 con 0x7f75f40d0160 2021-04-09T18:33:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc003210 con 0x7f75f40d0160 2021-04-09T18:33:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1258105878 0 0) 0x7f75ec001460 con 0x7f75f40d84b0 2021-04-09T18:33:28.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 >> v2:172.21.15.86:3300/0 conn(0x7f75f410d050 msgr2=0x7f75f40cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 --2- 172.21.15.32:0/1704876423 >> v2:172.21.15.86:3300/0 conn(0x7f75f410d050 0x7f75f40cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 >> v1:172.21.15.86:6789/0 conn(0x7f75f40d0160 legacy=0x7f75f40d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40d9260 con 0x7f75f40d84b0 2021-04-09T18:33:28.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f75ec001980 con 0x7f75f40d84b0 2021-04-09T18:33:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 >> v1:172.21.15.32:6789/0 conn(0x7f75f40d84b0 legacy=0x7f75f410da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc003b10 0x7f75dc005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:28.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75dc006780 0x7f75dc008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc009580 0x7f75dc00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f3fff700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75dc006780 0x7f75dc008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc050300 con 0x7f75dc006780 2021-04-09T18:33:28.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0504e0 con 0x7f75dc003b10 2021-04-09T18:33:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc050700 con 0x7f75dc009580 2021-04-09T18:33:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f8e05700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc003b10 0x7f75dc005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f9606700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc009580 0x7f75dc00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f3fff700 1 -- 172.21.15.32:0/1704876423 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc009580 msgr2=0x7f75dc00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:28.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f3fff700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc009580 0x7f75dc00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f3fff700 1 -- 172.21.15.32:0/1704876423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc003b10 msgr2=0x7f75dc005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f3fff700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc003b10 0x7f75dc005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f9606700 1 -- 172.21.15.32:0/1704876423 reap_dead start 2021-04-09T18:33:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f3fff700 1 -- 172.21.15.32:0/1704876423 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc0509a0 con 0x7f75dc006780 2021-04-09T18:33:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.816 7f75f8e05700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc003b10 0x7f75dc005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.817 7f75f3fff700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75dc006780 0x7f75dc008c80 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 rx=0x7f75e8006ab0 tx=0x7f75e80099d0).ready entity=mon.0 client_cookie=f0ae780959df5843 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.817 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75e8009dc0 con 0x7f75dc006780 2021-04-09T18:33:28.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.817 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e80066d0 con 0x7f75dc006780 2021-04-09T18:33:28.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.817 7f75f37fe700 1 -- 172.21.15.32:0/1704876423 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75e80084d0 con 0x7f75dc006780 2021-04-09T18:33:28.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.817 7f75fafe9700 1 -- 172.21.15.32:0/1704876423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75dc006780 msgr2=0x7f75dc008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:28.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.817 7f75fafe9700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75dc006780 0x7f75dc008c80 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 rx=0x7f75e8006ab0 tx=0x7f75e80099d0).stop 2021-04-09T18:33:28.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.817 7f75fafe9700 1 -- 172.21.15.32:0/1704876423 shutdown_connections 2021-04-09T18:33:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.817 7f75fafe9700 1 --2- 172.21.15.32:0/1704876423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75dc006780 0x7f75dc008c80 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.817 7f75fafe9700 1 -- 172.21.15.32:0/1704876423 >> 172.21.15.32:0/1704876423 conn(0x7f75f40ce620 msgr2=0x7f75f40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.818 7f75fafe9700 1 -- 172.21.15.32:0/1704876423 shutdown_connections 2021-04-09T18:33:28.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.818 7f75fafe9700 1 -- 172.21.15.32:0/1704876423 wait complete. 2021-04-09T18:33:28.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.838 7f75fafe9700 1 Processor -- start 2021-04-09T18:33:28.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.838 7f75fafe9700 1 -- start start 2021-04-09T18:33:28.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.838 7f75fafe9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f40ad5b0 0x7f75f41926b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:28.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.838 7f75fafe9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75f40990f0 0x7f75f40994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:28.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.838 7f75fafe9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f41a30b0 0x7f75f41a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:28.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.838 7f75fafe9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4098b70 con 0x7f75f41a30b0 2021-04-09T18:33:28.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.838 7f75fafe9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40988a0 con 0x7f75f40ad5b0 2021-04-09T18:33:28.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.838 7f75fafe9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4099b10 con 0x7f75f40990f0 2021-04-09T18:33:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f9606700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f41a30b0 0x7f75f41a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f9606700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f41a30b0 0x7f75f41a34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58640/0 (socket says 172.21.15.32:58640) 2021-04-09T18:33:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f9606700 1 -- 172.21.15.32:0/2893900722 learned_addr learned my addr 172.21.15.32:0/2893900722 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:28.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f8e05700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f40ad5b0 0x7f75f41926b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:28.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f3fff700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75f40990f0 0x7f75f40994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:28.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f9606700 1 -- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75f40990f0 msgr2=0x7f75f40994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:28.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f9606700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75f40990f0 0x7f75f40994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f9606700 1 -- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f40ad5b0 msgr2=0x7f75f41926b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:28.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f9606700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f40ad5b0 0x7f75f41926b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:28.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f9606700 1 -- 172.21.15.32:0/2893900722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f41a3ad0 con 0x7f75f41a30b0 2021-04-09T18:33:28.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f9606700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f41a30b0 0x7f75f41a34b0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 rx=0x7f75ec0015e0 tx=0x7f75ec00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:28.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f17fa700 1 -- 172.21.15.32:0/2893900722 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75ec0070d0 con 0x7f75f41a30b0 2021-04-09T18:33:28.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f17fa700 1 -- 172.21.15.32:0/2893900722 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec007450 con 0x7f75f41a30b0 2021-04-09T18:33:28.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75f17fa700 1 -- 172.21.15.32:0/2893900722 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75ec004b20 con 0x7f75f41a30b0 2021-04-09T18:33:28.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f41a3ad0 con 0x7f75f41a30b0 2021-04-09T18:33:28.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.839 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f41a3ad0 con 0x7f75f41a30b0 2021-04-09T18:33:28.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.840 7f75f17fa700 1 -- 172.21.15.32:0/2893900722 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f75ec001d90 con 0x7f75f41a30b0 2021-04-09T18:33:28.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.840 7f75f17fa700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d40188c0 0x7f75d401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:28.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.840 7f75f17fa700 1 -- 172.21.15.32:0/2893900722 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f75ec032c50 con 0x7f75f41a30b0 2021-04-09T18:33:28.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.840 7f75f8e05700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d40188c0 0x7f75d401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:28.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.840 7f75f8e05700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d40188c0 0x7f75d401adc0 crc :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:28.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.859 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f40b9040 con 0x7f75f41a30b0 2021-04-09T18:33:28.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:28.863 7f75f17fa700 1 -- 172.21.15.32:0/2893900722 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f75ec007030 con 0x7f75f41a30b0 2021-04-09T18:33:29.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.045 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "bravo_data", "prefix": "osd pool delete", "pool": "bravo_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f75f410d500 con 0x7f75f41a30b0 2021-04-09T18:33:29.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.638 7f75f17fa700 1 -- 172.21.15.32:0/2893900722 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "bravo_data", "prefix": "osd pool delete", "pool": "bravo_data", "yes_i_really_really_mean_it": true}]=0 pool 'bravo_data' removed v242) v1 ==== 170+0+0 (secure 0 0 0) 0x7f75ec001d90 con 0x7f75f41a30b0 2021-04-09T18:33:29.676 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'bravo_data' removed 2021-04-09T18:33:29.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d40188c0 msgr2=0x7f75d401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:29.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d40188c0 0x7f75d401adc0 crc :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f41a30b0 msgr2=0x7f75f41a34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:29.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f41a30b0 0x7f75f41a34b0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 rx=0x7f75ec0015e0 tx=0x7f75ec00ac80).stop 2021-04-09T18:33:29.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 shutdown_connections 2021-04-09T18:33:29.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75d40188c0 0x7f75d401adc0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75f41a30b0 0x7f75f41a34b0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75f40990f0 0x7f75f40994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 --2- 172.21.15.32:0/2893900722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75f40ad5b0 0x7f75f41926b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 >> 172.21.15.32:0/2893900722 conn(0x7f75f40ae6f0 msgr2=0x7f75f4097890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:29.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 shutdown_connections 2021-04-09T18:33:29.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.662 7f75fafe9700 1 -- 172.21.15.32:0/2893900722 wait complete. 2021-04-09T18:33:29.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:33:29.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.825 7fbd95be8700 1 Processor -- start 2021-04-09T18:33:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.825 7fbd95be8700 1 -- start start 2021-04-09T18:33:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd95be8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd900cffa0 0x7fbd900d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:29.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd95be8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd900d6c80 0x7fbd9010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd95be8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd900d7280 con 0x7fbd900d7800 2021-04-09T18:33:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd95be8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd900d7080 con 0x7fbd900cffa0 2021-04-09T18:33:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd95be8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd900d91e0 con 0x7fbd900d6c80 2021-04-09T18:33:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fbd900cffa0 0x7fbd900d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:29.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8effd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbd900d7800 0x7fbd900d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40008/0 (socket says 172.21.15.32:40008) 2021-04-09T18:33:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8effd700 1 -- 172.21.15.32:0/2984751313 learned_addr learned my addr 172.21.15.32:0/2984751313 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8ffff700 1 --2- 172.21.15.32:0/2984751313 >> v2:172.21.15.86:3300/0 conn(0x7fbd900d6c80 0x7fbd9010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8f7fe700 1 -- 172.21.15.32:0/2984751313 >> v2:172.21.15.86:3300/0 conn(0x7fbd900d6c80 msgr2=0x7fbd9010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:29.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8f7fe700 1 --2- 172.21.15.32:0/2984751313 >> v2:172.21.15.86:3300/0 conn(0x7fbd900d6c80 0x7fbd9010cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8f7fe700 1 -- 172.21.15.32:0/2984751313 >> v1:172.21.15.32:6789/0 conn(0x7fbd900d7800 legacy=0x7fbd900d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:29.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8e7fc700 1 -- 172.21.15.32:0/2984751313 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454810287 0 0) 0x7fbd84000d30 con 0x7fbd900d7800 2021-04-09T18:33:29.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8f7fe700 1 -- 172.21.15.32:0/2984751313 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd900d83f0 con 0x7fbd900cffa0 2021-04-09T18:33:29.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8ffff700 1 --2- 172.21.15.32:0/2984751313 >> v2:172.21.15.86:3300/0 conn(0x7fbd900d6c80 0x7fbd9010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:29.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.826 7fbd8f7fe700 1 --2- 172.21.15.32:0/2984751313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd900cffa0 0x7fbd900d2440 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7fbd80006650 tx=0x7fbd80009a80).ready entity=mon.0 client_cookie=e5bb32aeb30fafb9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:29.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd8e7fc700 1 -- 172.21.15.32:0/2984751313 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd800009b0 con 0x7fbd900cffa0 2021-04-09T18:33:29.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd8e7fc700 1 -- 172.21.15.32:0/2984751313 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd80006740 con 0x7fbd900cffa0 2021-04-09T18:33:29.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd8e7fc700 1 -- 172.21.15.32:0/2984751313 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd8000c0c0 con 0x7fbd900cffa0 2021-04-09T18:33:29.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd95be8700 1 -- 172.21.15.32:0/2984751313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd900cffa0 msgr2=0x7fbd900d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:29.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd95be8700 1 --2- 172.21.15.32:0/2984751313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd900cffa0 0x7fbd900d2440 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7fbd80006650 tx=0x7fbd80009a80).stop 2021-04-09T18:33:29.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd95be8700 1 -- 172.21.15.32:0/2984751313 shutdown_connections 2021-04-09T18:33:29.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd95be8700 1 --2- 172.21.15.32:0/2984751313 >> v2:172.21.15.86:3300/0 conn(0x7fbd900d6c80 0x7fbd9010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd95be8700 1 --2- 172.21.15.32:0/2984751313 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd900cffa0 0x7fbd900d2440 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd95be8700 1 -- 172.21.15.32:0/2984751313 >> 172.21.15.32:0/2984751313 conn(0x7fbd900ce5c0 msgr2=0x7fbd900cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:29.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd95be8700 1 -- 172.21.15.32:0/2984751313 shutdown_connections 2021-04-09T18:33:29.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.827 7fbd95be8700 1 -- 172.21.15.32:0/2984751313 wait complete. 2021-04-09T18:33:29.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.847 7fbd95be8700 1 Processor -- start 2021-04-09T18:33:29.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd95be8700 1 -- start start 2021-04-09T18:33:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd95be8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9017afd0 0x7fbd901b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd95be8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd901814c0 0x7fbd900aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:29.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd95be8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd900ab3b0 0x7fbd9017bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:29.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd95be8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd90192a60 con 0x7fbd901814c0 2021-04-09T18:33:29.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd95be8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9017b590 con 0x7fbd9017afd0 2021-04-09T18:33:29.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd95be8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd901b8600 con 0x7fbd900ab3b0 2021-04-09T18:33:29.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd8effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd901814c0 0x7fbd900aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:29.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd8effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd901814c0 0x7fbd900aad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58654/0 (socket says 172.21.15.32:58654) 2021-04-09T18:33:29.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd8effd700 1 -- 172.21.15.32:0/4193904475 learned_addr learned my addr 172.21.15.32:0/4193904475 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:29.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd8f7fe700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9017afd0 0x7fbd901b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:29.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.848 7fbd8ffff700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd900ab3b0 0x7fbd9017bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd8effd700 1 -- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd900ab3b0 msgr2=0x7fbd9017bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd8effd700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd900ab3b0 0x7fbd9017bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd8effd700 1 -- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9017afd0 msgr2=0x7fbd901b89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd8effd700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9017afd0 0x7fbd901b89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:29.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd8effd700 1 -- 172.21.15.32:0/4193904475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd901a0920 con 0x7fbd901814c0 2021-04-09T18:33:29.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd8effd700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd901814c0 0x7fbd900aad90 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7fbd8400a920 tx=0x7fbd8400acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:29.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd77fff700 1 -- 172.21.15.32:0/4193904475 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd840070d0 con 0x7fbd901814c0 2021-04-09T18:33:29.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd77fff700 1 -- 172.21.15.32:0/4193904475 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd840044c0 con 0x7fbd901814c0 2021-04-09T18:33:29.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd901a0920 con 0x7fbd901814c0 2021-04-09T18:33:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd77fff700 1 -- 172.21.15.32:0/4193904475 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd840011a0 con 0x7fbd901814c0 2021-04-09T18:33:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.849 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd901a0920 con 0x7fbd901814c0 2021-04-09T18:33:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.850 7fbd77fff700 1 -- 172.21.15.32:0/4193904475 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbd8400ae00 con 0x7fbd901814c0 2021-04-09T18:33:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.850 7fbd77fff700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd70018930 0x7fbd7001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:29.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.850 7fbd77fff700 1 -- 172.21.15.32:0/4193904475 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 5273+0+0 (secure 0 0 0) 0x7fbd84032c50 con 0x7fbd901814c0 2021-04-09T18:33:29.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.850 7fbd8f7fe700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd70018930 0x7fbd7001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:29.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.850 7fbd8f7fe700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd70018930 0x7fbd7001ae10 crc :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:29.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.870 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd901b7110 con 0x7fbd901814c0 2021-04-09T18:33:29.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:29.873 7fbd77fff700 1 -- 172.21.15.32:0/4193904475 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbd840011a0 con 0x7fbd901814c0 2021-04-09T18:33:30.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.086 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbd900d7280 con 0x7fbd901814c0 2021-04-09T18:33:30.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.089 7fbd77fff700 1 -- 172.21.15.32:0/4193904475 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbd8400ae00 con 0x7fbd901814c0 2021-04-09T18:33:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd70018930 msgr2=0x7fbd7001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd70018930 0x7fbd7001ae10 crc :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd901814c0 msgr2=0x7fbd900aad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd901814c0 0x7fbd900aad90 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7fbd8400a920 tx=0x7fbd8400acc0).stop 2021-04-09T18:33:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 shutdown_connections 2021-04-09T18:33:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbd70018930 0x7fbd7001ae10 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd901814c0 0x7fbd900aad90 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd900ab3b0 0x7fbd9017bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 --2- 172.21.15.32:0/4193904475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9017afd0 0x7fbd901b89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 >> 172.21.15.32:0/4193904475 conn(0x7fbd901905d0 msgr2=0x7fbd901b7340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:30.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 shutdown_connections 2021-04-09T18:33:30.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.095 7fbd95be8700 1 -- 172.21.15.32:0/4193904475 wait complete. 2021-04-09T18:33:30.122 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestMultiFilesystems.test_standby' 2021-04-09T18:33:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.253 7f47ca502700 1 Processor -- start 2021-04-09T18:33:30.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47ca502700 1 -- start start 2021-04-09T18:33:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47ca502700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47c410d010 0x7f47c40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47ca502700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c40d8490 0x7f47c410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47ca502700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c410d480 con 0x7f47c40d1da0 2021-04-09T18:33:30.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47ca502700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c40d8d30 con 0x7f47c410d010 2021-04-09T18:33:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47ca502700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c40d9220 con 0x7f47c40d8490 2021-04-09T18:33:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47c3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f47c410d010 0x7f47c40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47bbfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f47c40d1da0 0x7f47c40d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40018/0 (socket says 172.21.15.32:40018) 2021-04-09T18:33:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47bbfff700 1 -- 172.21.15.32:0/1407472829 learned_addr learned my addr 172.21.15.32:0/1407472829 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:30.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.254 7f47c8b1f700 1 --2- 172.21.15.32:0/1407472829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c40d8490 0x7f47c410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47c3fff700 1 -- 172.21.15.32:0/1407472829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c40d8490 msgr2=0x7f47c410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47c3fff700 1 --2- 172.21.15.32:0/1407472829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c40d8490 0x7f47c410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47c3fff700 1 -- 172.21.15.32:0/1407472829 >> v1:172.21.15.32:6789/0 conn(0x7f47c40d1da0 legacy=0x7f47c40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47c3fff700 1 -- 172.21.15.32:0/1407472829 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c410e020 con 0x7f47c410d010 2021-04-09T18:33:30.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47c37fe700 1 -- 172.21.15.32:0/1407472829 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642091223 0 0) 0x7f47ac000ce0 con 0x7f47c40d1da0 2021-04-09T18:33:30.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47c3fff700 1 --2- 172.21.15.32:0/1407472829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c410d010 0x7f47c40d1780 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 rx=0x7f47b4006650 tx=0x7f47b4009a00).ready entity=mon.0 client_cookie=d056f15d36a7c213 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:30.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47c37fe700 1 -- 172.21.15.32:0/1407472829 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47b4009df0 con 0x7f47c410d010 2021-04-09T18:33:30.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47c37fe700 1 -- 172.21.15.32:0/1407472829 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b40009b0 con 0x7f47c410d010 2021-04-09T18:33:30.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47c37fe700 1 -- 172.21.15.32:0/1407472829 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47b400c2b0 con 0x7f47c410d010 2021-04-09T18:33:30.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47ca502700 1 -- 172.21.15.32:0/1407472829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c410d010 msgr2=0x7f47c40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.255 7f47ca502700 1 --2- 172.21.15.32:0/1407472829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c410d010 0x7f47c40d1780 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 rx=0x7f47b4006650 tx=0x7f47b4009a00).stop 2021-04-09T18:33:30.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.256 7f47ca502700 1 -- 172.21.15.32:0/1407472829 shutdown_connections 2021-04-09T18:33:30.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.256 7f47ca502700 1 --2- 172.21.15.32:0/1407472829 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c40d8490 0x7f47c410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.256 7f47ca502700 1 --2- 172.21.15.32:0/1407472829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c410d010 0x7f47c40d1780 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.256 7f47ca502700 1 -- 172.21.15.32:0/1407472829 >> 172.21.15.32:0/1407472829 conn(0x7f47c40ce5f0 msgr2=0x7f47c40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:30.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.256 7f47ca502700 1 -- 172.21.15.32:0/1407472829 shutdown_connections 2021-04-09T18:33:30.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.256 7f47ca502700 1 -- 172.21.15.32:0/1407472829 wait complete. 2021-04-09T18:33:30.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.276 7f47ca502700 1 Processor -- start 2021-04-09T18:33:30.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.276 7f47ca502700 1 -- start start 2021-04-09T18:33:30.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47ca502700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4179fd0 0x7f47c40976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47ca502700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c4097cc0 0x7f47c41a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47ca502700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c41a4b10 0x7f47c41a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47ca502700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c4098140 con 0x7f47c4097cc0 2021-04-09T18:33:30.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47ca502700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c417a3d0 con 0x7f47c41a4b10 2021-04-09T18:33:30.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47ca502700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c417c870 con 0x7f47c4179fd0 2021-04-09T18:33:30.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47bbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c4097cc0 0x7f47c41a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47bbfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c4097cc0 0x7f47c41a44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58668/0 (socket says 172.21.15.32:58668) 2021-04-09T18:33:30.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47bbfff700 1 -- 172.21.15.32:0/3991797154 learned_addr learned my addr 172.21.15.32:0/3991797154 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:30.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47c3fff700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4179fd0 0x7f47c40976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47c8b1f700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c41a4b10 0x7f47c41a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47bbfff700 1 -- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4179fd0 msgr2=0x7f47c40976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47bbfff700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4179fd0 0x7f47c40976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47bbfff700 1 -- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c41a4b10 msgr2=0x7f47c41a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47bbfff700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c41a4b10 0x7f47c41a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47bbfff700 1 -- 172.21.15.32:0/3991797154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c4179bf0 con 0x7f47c4097cc0 2021-04-09T18:33:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47c3fff700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4179fd0 0x7f47c40976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47c8b1f700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c41a4b10 0x7f47c41a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47bbfff700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c4097cc0 0x7f47c41a44f0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 rx=0x7f47ac006f00 tx=0x7f47ac00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:30.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47c17fa700 1 -- 172.21.15.32:0/3991797154 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47ac006020 con 0x7f47c4097cc0 2021-04-09T18:33:30.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47c17fa700 1 -- 172.21.15.32:0/3991797154 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ac0063a0 con 0x7f47c4097cc0 2021-04-09T18:33:30.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47ca502700 1 -- 172.21.15.32:0/3991797154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c4179bf0 con 0x7f47c4097cc0 2021-04-09T18:33:30.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47c17fa700 1 -- 172.21.15.32:0/3991797154 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47ac004620 con 0x7f47c4097cc0 2021-04-09T18:33:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.277 7f47ca502700 1 -- 172.21.15.32:0/3991797154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c4179bf0 con 0x7f47c4097cc0 2021-04-09T18:33:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.278 7f47c17fa700 1 -- 172.21.15.32:0/3991797154 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f47ac00a000 con 0x7f47c4097cc0 2021-04-09T18:33:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.278 7f47c17fa700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47a80188c0 0x7f47a801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.278 7f47c17fa700 1 -- 172.21.15.32:0/3991797154 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f47ac032680 con 0x7f47c4097cc0 2021-04-09T18:33:30.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.279 7f47c3fff700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47a80188c0 0x7f47a801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.279 7f47c3fff700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47a80188c0 0x7f47a801adc0 crc :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:30.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.298 7f47ca502700 1 -- 172.21.15.32:0/3991797154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c41b7050 con 0x7f47c4097cc0 2021-04-09T18:33:30.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.302 7f47c17fa700 1 -- 172.21.15.32:0/3991797154 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f47ac006b90 con 0x7f47c4097cc0 2021-04-09T18:33:30.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.491 7f47ca502700 1 -- 172.21.15.32:0/3991797154 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestMultiFilesystems.test_standby"]} v 0) v1 -- 0x7f47c410d480 con 0x7f47c4097cc0 2021-04-09T18:33:30.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.640 7f47c17fa700 1 -- 172.21.15.32:0/3991797154 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestMultiFilesystems.test_standby"]}]=0 v935) v1 ==== 139+0+0 (secure 0 0 0) 0x7f47ac00a000 con 0x7f47c4097cc0 2021-04-09T18:33:30.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 -- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47a80188c0 msgr2=0x7f47a801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47a80188c0 0x7f47a801adc0 crc :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 -- 172.21.15.32:0/3991797154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c4097cc0 msgr2=0x7f47c41a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c4097cc0 0x7f47c41a44f0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 rx=0x7f47ac006f00 tx=0x7f47ac00b940).stop 2021-04-09T18:33:30.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 -- 172.21.15.32:0/3991797154 shutdown_connections 2021-04-09T18:33:30.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f47a80188c0 0x7f47a801adc0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47c4097cc0 0x7f47c41a44f0 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c41a4b10 0x7f47c41a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 --2- 172.21.15.32:0/3991797154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4179fd0 0x7f47c40976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 -- 172.21.15.32:0/3991797154 >> 172.21.15.32:0/3991797154 conn(0x7f47c4199680 msgr2=0x7f47c41a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:30.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 -- 172.21.15.32:0/3991797154 shutdown_connections 2021-04-09T18:33:30.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.657 7f47ca502700 1 -- 172.21.15.32:0/3991797154 wait complete. 2021-04-09T18:33:30.683 INFO:tasks.cephfs_test_runner:test_standby (tasks.cephfs.test_failover.TestMultiFilesystems) ... ok 2021-04-09T18:33:30.684 INFO:tasks.cephfs_test_runner:Starting test: test_rank_stopped (tasks.cephfs.test_failover.TestStandbyReplay) 2021-04-09T18:33:30.684 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestStandbyReplay.test_rank_stopped' 2021-04-09T18:33:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.816 7fe4e3779700 1 Processor -- start 2021-04-09T18:33:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.816 7fe4e3779700 1 -- start start 2021-04-09T18:33:30.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4e3779700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc10d010 0x7fe4dc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4e3779700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe4dc0d8490 0x7fe4dc10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4e3779700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4dc10d480 con 0x7fe4dc0d1da0 2021-04-09T18:33:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4e3779700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc10e780 con 0x7fe4dc0d8490 2021-04-09T18:33:30.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4e3779700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0d8d30 con 0x7fe4dc10d010 2021-04-09T18:33:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4e0d94700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe4dc0d1da0 0x7fe4dc0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40036/0 (socket says 172.21.15.32:40036) 2021-04-09T18:33:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4e0d94700 1 -- 172.21.15.32:0/1175446562 learned_addr learned my addr 172.21.15.32:0/1175446562 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4e1595700 1 --2- 172.21.15.32:0/1175446562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc10d010 0x7fe4dc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4e1d96700 1 --2- 172.21.15.32:0/1175446562 >> v2:172.21.15.86:3300/0 conn(0x7fe4dc0d8490 0x7fe4dc10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4d3fff700 1 -- 172.21.15.32:0/1175446562 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092777710 0 0) 0x7fe4cc0009b0 con 0x7fe4dc0d1da0 2021-04-09T18:33:30.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.817 7fe4d3fff700 1 -- 172.21.15.32:0/1175446562 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c8002fe0 con 0x7fe4dc0d1da0 2021-04-09T18:33:30.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4e1d96700 1 -- 172.21.15.32:0/1175446562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc10d010 msgr2=0x7fe4dc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:30.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4e1d96700 1 --2- 172.21.15.32:0/1175446562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc10d010 0x7fe4dc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4e1d96700 1 -- 172.21.15.32:0/1175446562 >> v1:172.21.15.32:6789/0 conn(0x7fe4dc0d1da0 legacy=0x7fe4dc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4e1d96700 1 -- 172.21.15.32:0/1175446562 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc0d9220 con 0x7fe4dc0d8490 2021-04-09T18:33:30.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4e1595700 1 --2- 172.21.15.32:0/1175446562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc10d010 0x7fe4dc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:30.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4e1d96700 1 --2- 172.21.15.32:0/1175446562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4dc0d8490 0x7fe4dc10da00 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7fe4d4006e50 tx=0x7fe4d4009e70).ready entity=mon.1 client_cookie=e8b00a74d84063fd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:30.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4d3fff700 1 -- 172.21.15.32:0/1175446562 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4d4006a20 con 0x7fe4dc0d8490 2021-04-09T18:33:30.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4d3fff700 1 -- 172.21.15.32:0/1175446562 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d4006790 con 0x7fe4dc0d8490 2021-04-09T18:33:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4d3fff700 1 -- 172.21.15.32:0/1175446562 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4d400b700 con 0x7fe4dc0d8490 2021-04-09T18:33:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4e3779700 1 -- 172.21.15.32:0/1175446562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4dc0d8490 msgr2=0x7fe4dc10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.818 7fe4e3779700 1 --2- 172.21.15.32:0/1175446562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4dc0d8490 0x7fe4dc10da00 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7fe4d4006e50 tx=0x7fe4d4009e70).stop 2021-04-09T18:33:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.819 7fe4e3779700 1 -- 172.21.15.32:0/1175446562 shutdown_connections 2021-04-09T18:33:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.819 7fe4e3779700 1 --2- 172.21.15.32:0/1175446562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4dc0d8490 0x7fe4dc10da00 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.819 7fe4e3779700 1 --2- 172.21.15.32:0/1175446562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc10d010 0x7fe4dc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.819 7fe4e3779700 1 -- 172.21.15.32:0/1175446562 >> 172.21.15.32:0/1175446562 conn(0x7fe4dc0ce5f0 msgr2=0x7fe4dc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:30.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.819 7fe4e3779700 1 -- 172.21.15.32:0/1175446562 shutdown_connections 2021-04-09T18:33:30.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.819 7fe4e3779700 1 -- 172.21.15.32:0/1175446562 wait complete. 2021-04-09T18:33:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.839 7fe4e3779700 1 Processor -- start 2021-04-09T18:33:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.839 7fe4e3779700 1 -- start start 2021-04-09T18:33:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e3779700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc1a5210 0x7fe4dc185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e3779700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4dc185a70 0x7fe4dc0acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e3779700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4dc185e70 0x7fe4dc0ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e3779700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc18f7b0 con 0x7fe4dc185e70 2021-04-09T18:33:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e3779700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0ad7f0 con 0x7fe4dc185a70 2021-04-09T18:33:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e3779700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0ada10 con 0x7fe4dc1a5210 2021-04-09T18:33:30.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1d96700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4dc185e70 0x7fe4dc0ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1d96700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4dc185e70 0x7fe4dc0ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58684/0 (socket says 172.21.15.32:58684) 2021-04-09T18:33:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1d96700 1 -- 172.21.15.32:0/3585056812 learned_addr learned my addr 172.21.15.32:0/3585056812 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1595700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc1a5210 0x7fe4dc185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1d96700 1 -- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc1a5210 msgr2=0x7fe4dc185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e0d94700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4dc185a70 0x7fe4dc0acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1d96700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc1a5210 0x7fe4dc185450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1d96700 1 -- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4dc185a70 msgr2=0x7fe4dc0acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:30.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1d96700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4dc185a70 0x7fe4dc0acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:30.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1d96700 1 -- 172.21.15.32:0/3585056812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc179610 con 0x7fe4dc185e70 2021-04-09T18:33:30.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4e1d96700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4dc185e70 0x7fe4dc0ad1d0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 rx=0x7fe4d40099f0 tx=0x7fe4d400bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:30.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.840 7fe4d1ffb700 1 -- 172.21.15.32:0/3585056812 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4d40090c0 con 0x7fe4dc185e70 2021-04-09T18:33:30.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.841 7fe4d1ffb700 1 -- 172.21.15.32:0/3585056812 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d4000df0 con 0x7fe4dc185e70 2021-04-09T18:33:30.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.841 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc179610 con 0x7fe4dc185e70 2021-04-09T18:33:30.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.841 7fe4d1ffb700 1 -- 172.21.15.32:0/3585056812 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4d4001390 con 0x7fe4dc185e70 2021-04-09T18:33:30.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.841 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc179610 con 0x7fe4dc185e70 2021-04-09T18:33:30.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.841 7fe4d1ffb700 1 -- 172.21.15.32:0/3585056812 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe4d40011a0 con 0x7fe4dc185e70 2021-04-09T18:33:30.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.841 7fe4d1ffb700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4bc018900 0x7fe4bc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:30.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.841 7fe4d1ffb700 1 -- 172.21.15.32:0/3585056812 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 5273+0+0 (secure 0 0 0) 0x7fe4d4032c70 con 0x7fe4dc185e70 2021-04-09T18:33:30.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.842 7fe4e1595700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4bc018900 0x7fe4bc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:30.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.842 7fe4e1595700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4bc018900 0x7fe4bc01adc0 crc :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:30.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.861 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4dc18f7b0 con 0x7fe4dc185e70 2021-04-09T18:33:30.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:30.864 7fe4d1ffb700 1 -- 172.21.15.32:0/3585056812 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe4d4001ee0 con 0x7fe4dc185e70 2021-04-09T18:33:31.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.052 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestStandbyReplay.test_rank_stopped"]} v 0) v1 -- 0x7fe4dc18f7b0 con 0x7fe4dc185e70 2021-04-09T18:33:31.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.641 7fe4d1ffb700 1 -- 172.21.15.32:0/3585056812 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestStandbyReplay.test_rank_stopped"]}]=0 v936) v1 ==== 144+0+0 (secure 0 0 0) 0x7fe4d40011a0 con 0x7fe4dc185e70 2021-04-09T18:33:31.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4bc018900 msgr2=0x7fe4bc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:31.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4bc018900 0x7fe4bc01adc0 crc :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4dc185e70 msgr2=0x7fe4dc0ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:31.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4dc185e70 0x7fe4dc0ad1d0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 rx=0x7fe4d40099f0 tx=0x7fe4d400bc90).stop 2021-04-09T18:33:31.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 shutdown_connections 2021-04-09T18:33:31.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4bc018900 0x7fe4bc01adc0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4dc185e70 0x7fe4dc0ad1d0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4dc185a70 0x7fe4dc0acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 --2- 172.21.15.32:0/3585056812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4dc1a5210 0x7fe4dc185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 >> 172.21.15.32:0/3585056812 conn(0x7fe4dc1a99c0 msgr2=0x7fe4dc17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:31.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 shutdown_connections 2021-04-09T18:33:31.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.670 7fe4e3779700 1 -- 172.21.15.32:0/3585056812 wait complete. 2021-04-09T18:33:31.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:33:31.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.836 7f002280e700 1 Processor -- start 2021-04-09T18:33:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.836 7f002280e700 1 -- start start 2021-04-09T18:33:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f002280e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f001c0d7ca0 0x7f001c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:31.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f002280e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f001c10da00 0x7f001c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f002280e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c10d480 con 0x7f001c10d010 2021-04-09T18:33:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f002280e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001c0d9220 con 0x7f001c10da00 2021-04-09T18:33:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f002280e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001c0d86c0 con 0x7f001c0d7ca0 2021-04-09T18:33:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f0020e2b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f001c10da00 0x7f001c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:31.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f001bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f001c10d010 0x7f001c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40046/0 (socket says 172.21.15.32:40046) 2021-04-09T18:33:31.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f001bfff700 1 -- 172.21.15.32:0/2422020119 learned_addr learned my addr 172.21.15.32:0/2422020119 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:31.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f0013fff700 1 --2- 172.21.15.32:0/2422020119 >> v2:172.21.15.86:3300/0 conn(0x7f001c0d7ca0 0x7f001c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:31.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f0020e2b700 1 -- 172.21.15.32:0/2422020119 >> v2:172.21.15.86:3300/0 conn(0x7f001c0d7ca0 msgr2=0x7f001c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:31.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f0020e2b700 1 --2- 172.21.15.32:0/2422020119 >> v2:172.21.15.86:3300/0 conn(0x7f001c0d7ca0 0x7f001c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f0020e2b700 1 -- 172.21.15.32:0/2422020119 >> v1:172.21.15.32:6789/0 conn(0x7f001c10d010 legacy=0x7f001c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f0020e2b700 1 -- 172.21.15.32:0/2422020119 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c0d1f70 con 0x7f001c10da00 2021-04-09T18:33:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.837 7f001b7fe700 1 -- 172.21.15.32:0/2422020119 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910052 0 0) 0x7f000c000ce0 con 0x7f001c10d010 2021-04-09T18:33:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.838 7f0020e2b700 1 --2- 172.21.15.32:0/2422020119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c10da00 0x7f001c10de00 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 rx=0x7f0014006e30 tx=0x7f0014009de0).ready entity=mon.0 client_cookie=8b9f9c1a0906c847 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:31.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.838 7f001b7fe700 1 -- 172.21.15.32:0/2422020119 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f001400b300 con 0x7f001c10da00 2021-04-09T18:33:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.838 7f001b7fe700 1 -- 172.21.15.32:0/2422020119 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001400b680 con 0x7f001c10da00 2021-04-09T18:33:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.838 7f001b7fe700 1 -- 172.21.15.32:0/2422020119 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f001400bb60 con 0x7f001c10da00 2021-04-09T18:33:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.838 7f002280e700 1 -- 172.21.15.32:0/2422020119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c10da00 msgr2=0x7f001c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:31.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.838 7f002280e700 1 --2- 172.21.15.32:0/2422020119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c10da00 0x7f001c10de00 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 rx=0x7f0014006e30 tx=0x7f0014009de0).stop 2021-04-09T18:33:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.839 7f002280e700 1 -- 172.21.15.32:0/2422020119 shutdown_connections 2021-04-09T18:33:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.839 7f002280e700 1 --2- 172.21.15.32:0/2422020119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c10da00 0x7f001c10de00 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.839 7f002280e700 1 --2- 172.21.15.32:0/2422020119 >> v2:172.21.15.86:3300/0 conn(0x7f001c0d7ca0 0x7f001c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.839 7f002280e700 1 -- 172.21.15.32:0/2422020119 >> 172.21.15.32:0/2422020119 conn(0x7f001c0ce5f0 msgr2=0x7f001c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:31.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.839 7f002280e700 1 -- 172.21.15.32:0/2422020119 shutdown_connections 2021-04-09T18:33:31.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.839 7f002280e700 1 -- 172.21.15.32:0/2422020119 wait complete. 2021-04-09T18:33:31.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.859 7f002280e700 1 Processor -- start 2021-04-09T18:33:31.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.859 7f002280e700 1 -- start start 2021-04-09T18:33:31.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.859 7f002280e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f001c08a910 0x7f001c09e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:31.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.859 7f002280e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c08ad10 0x7f001c09ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:31.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.859 7f002280e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f001c092350 0x7f001c092750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:31.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.859 7f002280e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c09e0d0 con 0x7f001c08ad10 2021-04-09T18:33:31.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.859 7f002280e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c09f290 con 0x7f001c08a910 2021-04-09T18:33:31.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.859 7f002280e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001c08b430 con 0x7f001c092350 2021-04-09T18:33:31.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.859 7f0013fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c08ad10 0x7f001c09ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:31.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f0013fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c08ad10 0x7f001c09ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58696/0 (socket says 172.21.15.32:58696) 2021-04-09T18:33:31.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f0013fff700 1 -- 172.21.15.32:0/2390217499 learned_addr learned my addr 172.21.15.32:0/2390217499 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:31.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f0020e2b700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f001c092350 0x7f001c092750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:31.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f001bfff700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f001c08a910 0x7f001c09e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:31.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f0013fff700 1 -- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f001c092350 msgr2=0x7f001c092750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:31.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f0013fff700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f001c092350 0x7f001c092750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f0013fff700 1 -- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f001c08a910 msgr2=0x7f001c09e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:31.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f0013fff700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f001c08a910 0x7f001c09e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:31.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f0013fff700 1 -- 172.21.15.32:0/2390217499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c092d70 con 0x7f001c08ad10 2021-04-09T18:33:31.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f0013fff700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c08ad10 0x7f001c09ec70 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 rx=0x7f000400b560 tx=0x7f000400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:31.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f00197fa700 1 -- 172.21.15.32:0/2390217499 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f000400bcf0 con 0x7f001c08ad10 2021-04-09T18:33:31.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f00197fa700 1 -- 172.21.15.32:0/2390217499 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0004000c90 con 0x7f001c08ad10 2021-04-09T18:33:31.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f002280e700 1 -- 172.21.15.32:0/2390217499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001c092d70 con 0x7f001c08ad10 2021-04-09T18:33:31.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f00197fa700 1 -- 172.21.15.32:0/2390217499 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0004002220 con 0x7f001c08ad10 2021-04-09T18:33:31.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.860 7f002280e700 1 -- 172.21.15.32:0/2390217499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001c092d70 con 0x7f001c08ad10 2021-04-09T18:33:31.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.861 7f00197fa700 1 -- 172.21.15.32:0/2390217499 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f00040052e0 con 0x7f001c08ad10 2021-04-09T18:33:31.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.861 7f00197fa700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00000188d0 0x7f000001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:31.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.861 7f00197fa700 1 -- 172.21.15.32:0/2390217499 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f00040328f0 con 0x7f001c08ad10 2021-04-09T18:33:31.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.861 7f001bfff700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00000188d0 0x7f000001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:31.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.861 7f001bfff700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00000188d0 0x7f000001add0 crc :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:31.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.881 7f002280e700 1 -- 172.21.15.32:0/2390217499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001c10d480 con 0x7f001c08ad10 2021-04-09T18:33:31.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:31.884 7f00197fa700 1 -- 172.21.15.32:0/2390217499 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0004000c90 con 0x7f001c08ad10 2021-04-09T18:33:32.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.098 7f002280e700 1 -- 172.21.15.32:0/2390217499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f001c10d480 con 0x7f001c08ad10 2021-04-09T18:33:32.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.101 7f00197fa700 1 -- 172.21.15.32:0/2390217499 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f00040052e0 con 0x7f001c08ad10 2021-04-09T18:33:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 -- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00000188d0 msgr2=0x7f000001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:32.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00000188d0 0x7f000001add0 crc :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 -- 172.21.15.32:0/2390217499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c08ad10 msgr2=0x7f001c09ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:32.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c08ad10 0x7f001c09ec70 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 rx=0x7f000400b560 tx=0x7f000400b900).stop 2021-04-09T18:33:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 -- 172.21.15.32:0/2390217499 shutdown_connections 2021-04-09T18:33:32.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f00000188d0 0x7f000001add0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f001c08ad10 0x7f001c09ec70 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f001c092350 0x7f001c092750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 --2- 172.21.15.32:0/2390217499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f001c08a910 0x7f001c09e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 -- 172.21.15.32:0/2390217499 >> 172.21.15.32:0/2390217499 conn(0x7f001c0b2fa0 msgr2=0x7f001c0a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:32.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 -- 172.21.15.32:0/2390217499 shutdown_connections 2021-04-09T18:33:32.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.107 7f002280e700 1 -- 172.21.15.32:0/2390217499 wait complete. 2021-04-09T18:33:32.134 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:33:32.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:33:32.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:33:32.251 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:33:32.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:32.252 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:33:32.253 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:33:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:33:32.298 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:33:32.299 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:33:32.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:32.301 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:33:32.302 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:33:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:33:32.345 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:33:32.345 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:33:32.346 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:33:32.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.477 7f88f2156700 1 Processor -- start 2021-04-09T18:33:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.477 7f88f2156700 1 -- start start 2021-04-09T18:33:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88f2156700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88ec0d6c80 0x7f88ec0d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88f2156700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec0d2440 0x7f88ec10cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:32.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88f2156700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec0d7280 con 0x7f88ec0cffa0 2021-04-09T18:33:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88f2156700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec0d7080 con 0x7f88ec0d6c80 2021-04-09T18:33:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88f2156700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0d91e0 con 0x7f88ec0d2440 2021-04-09T18:33:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88eaffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f88ec0d6c80 0x7f88ec0d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88eb7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f88ec0cffa0 0x7f88ec0d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40060/0 (socket says 172.21.15.32:40060) 2021-04-09T18:33:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88eb7fe700 1 -- 172.21.15.32:0/2758588253 learned_addr learned my addr 172.21.15.32:0/2758588253 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88ebfff700 1 --2- 172.21.15.32:0/2758588253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec0d2440 0x7f88ec10cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88eaffd700 1 -- 172.21.15.32:0/2758588253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec0d2440 msgr2=0x7f88ec10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88eaffd700 1 --2- 172.21.15.32:0/2758588253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec0d2440 0x7f88ec10cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.478 7f88eaffd700 1 -- 172.21.15.32:0/2758588253 >> v1:172.21.15.32:6789/0 conn(0x7f88ec0cffa0 legacy=0x7f88ec0d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.479 7f88ea7fc700 1 -- 172.21.15.32:0/2758588253 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953629113 0 0) 0x7f88dc000ce0 con 0x7f88ec0cffa0 2021-04-09T18:33:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.479 7f88eaffd700 1 -- 172.21.15.32:0/2758588253 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec0d2840 con 0x7f88ec0d6c80 2021-04-09T18:33:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.479 7f88eaffd700 1 --2- 172.21.15.32:0/2758588253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0d6c80 0x7f88ec0d7ff0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 rx=0x7f88e0006650 tx=0x7f88e0009a00).ready entity=mon.0 client_cookie=f151ac088ca7dda0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.479 7f88ea7fc700 1 -- 172.21.15.32:0/2758588253 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88e0009df0 con 0x7f88ec0d6c80 2021-04-09T18:33:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.479 7f88ea7fc700 1 -- 172.21.15.32:0/2758588253 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e00009b0 con 0x7f88ec0d6c80 2021-04-09T18:33:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.479 7f88ea7fc700 1 -- 172.21.15.32:0/2758588253 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88e000c2b0 con 0x7f88ec0d6c80 2021-04-09T18:33:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.479 7f88f2156700 1 -- 172.21.15.32:0/2758588253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0d6c80 msgr2=0x7f88ec0d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.479 7f88f2156700 1 --2- 172.21.15.32:0/2758588253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0d6c80 0x7f88ec0d7ff0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 rx=0x7f88e0006650 tx=0x7f88e0009a00).stop 2021-04-09T18:33:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.480 7f88f2156700 1 -- 172.21.15.32:0/2758588253 shutdown_connections 2021-04-09T18:33:32.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.480 7f88f2156700 1 --2- 172.21.15.32:0/2758588253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec0d2440 0x7f88ec10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.480 7f88f2156700 1 --2- 172.21.15.32:0/2758588253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0d6c80 0x7f88ec0d7ff0 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.480 7f88f2156700 1 -- 172.21.15.32:0/2758588253 >> 172.21.15.32:0/2758588253 conn(0x7f88ec0ce5c0 msgr2=0x7f88ec0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:32.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.480 7f88f2156700 1 -- 172.21.15.32:0/2758588253 shutdown_connections 2021-04-09T18:33:32.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.480 7f88f2156700 1 -- 172.21.15.32:0/2758588253 wait complete. 2021-04-09T18:33:32.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.500 7f88f2156700 1 Processor -- start 2021-04-09T18:33:32.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.500 7f88f2156700 1 -- start start 2021-04-09T18:33:32.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.500 7f88f2156700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88ec0aa220 0x7f88ec0a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:32.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.500 7f88f2156700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec193440 0x7f88ec0b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:32.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.500 7f88f2156700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0b5530 0x7f88ec197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:32.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.500 7f88f2156700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec198b40 con 0x7f88ec0b5530 2021-04-09T18:33:32.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.500 7f88f2156700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0aa6a0 con 0x7f88ec0aa220 2021-04-09T18:33:32.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.500 7f88f2156700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec198510 con 0x7f88ec193440 2021-04-09T18:33:32.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0b5530 0x7f88ec197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:32.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88ebfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0b5530 0x7f88ec197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58712/0 (socket says 172.21.15.32:58712) 2021-04-09T18:33:32.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88ebfff700 1 -- 172.21.15.32:0/3102432847 learned_addr learned my addr 172.21.15.32:0/3102432847 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:32.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88eaffd700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec193440 0x7f88ec0b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:32.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88eb7fe700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88ec0aa220 0x7f88ec0a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:32.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88ebfff700 1 -- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec193440 msgr2=0x7f88ec0b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:32.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88ebfff700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec193440 0x7f88ec0b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88ebfff700 1 -- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88ec0aa220 msgr2=0x7f88ec0a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:32.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88ebfff700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88ec0aa220 0x7f88ec0a99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88ebfff700 1 -- 172.21.15.32:0/3102432847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec198730 con 0x7f88ec0b5530 2021-04-09T18:33:32.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88ebfff700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0b5530 0x7f88ec197ef0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7f88e40099a0 tx=0x7f88e400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:32.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88d3fff700 1 -- 172.21.15.32:0/3102432847 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88e40066f0 con 0x7f88ec0b5530 2021-04-09T18:33:32.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88d3fff700 1 -- 172.21.15.32:0/3102432847 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e4003030 con 0x7f88ec0b5530 2021-04-09T18:33:32.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88f2156700 1 -- 172.21.15.32:0/3102432847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec198730 con 0x7f88ec0b5530 2021-04-09T18:33:32.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88d3fff700 1 -- 172.21.15.32:0/3102432847 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88e4001470 con 0x7f88ec0b5530 2021-04-09T18:33:32.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.501 7f88f2156700 1 -- 172.21.15.32:0/3102432847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec198730 con 0x7f88ec0b5530 2021-04-09T18:33:32.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.502 7f88d3fff700 1 -- 172.21.15.32:0/3102432847 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f88e40011f0 con 0x7f88ec0b5530 2021-04-09T18:33:32.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.502 7f88d3fff700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88cc018930 0x7f88cc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:32.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.502 7f88d3fff700 1 -- 172.21.15.32:0/3102432847 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f88e4032c70 con 0x7f88ec0b5530 2021-04-09T18:33:32.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.502 7f88eb7fe700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88cc018930 0x7f88cc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:32.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.502 7f88eb7fe700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88cc018930 0x7f88cc01ae10 crc :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:32.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.522 7f88f2156700 1 -- 172.21.15.32:0/3102432847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec182e10 con 0x7f88ec0b5530 2021-04-09T18:33:32.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.526 7f88d3fff700 1 -- 172.21.15.32:0/3102432847 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f88e400be30 con 0x7f88ec0b5530 2021-04-09T18:33:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.715 7f88f2156700 1 -- 172.21.15.32:0/3102432847 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f88ec0d7280 con 0x7f88ec0b5530 2021-04-09T18:33:32.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.716 7f88d3fff700 1 -- 172.21.15.32:0/3102432847 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v242) v1 ==== 81+0+26248 (secure 0 0 0) 0x7f88e401fae0 con 0x7f88ec0b5530 2021-04-09T18:33:32.733 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:32.734 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:33:32.734 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 242, 2021-04-09T18:33:32.734 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:33:32.734 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:33:32.735 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:33:29.636918", 2021-04-09T18:33:32.735 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:33:32.735 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:33:32.735 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:33:32.736 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:33:32.736 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:33:32.736 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:33:32.736 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:33:32.737 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:33:32.737 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:33:32.737 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:33:32.737 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:33:32.738 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:33:32.738 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:33:32.738 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:33:32.738 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:33:32.739 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 41, 2021-04-09T18:33:32.739 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:33:32.739 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:33:32.739 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:33:32.739 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:33:32.740 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:33:32.740 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.740 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:33:32.741 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:33:32.741 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:33:32.741 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:33:32.741 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:33:32.741 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:33:32.742 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:33:32.742 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:33:32.742 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:33:32.742 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:33:32.742 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:33:32.743 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:33:32.743 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:33:32.743 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:33:32.743 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:33:32.744 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:33:32.744 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:33:32.744 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:33:32.745 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:33:32.745 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:33:32.745 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:33:32.745 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:33:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:33:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:33:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:33:32.746 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:33:32.747 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:33:32.747 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:33:32.747 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:33:32.747 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:33:32.748 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:33:32.748 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:33:32.748 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:33:32.748 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:33:32.748 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:33:32.749 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:33:32.749 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:33:32.749 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:33:32.749 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:33:32.750 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:33:32.750 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:33:32.750 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:33:32.750 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:33:32.750 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:33:32.751 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:33:32.751 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:33:32.751 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:33:32.751 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:33:32.752 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:33:32.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:33:32.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.752 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:33:32.752 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:33:32.753 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:33:32.753 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:33:32.753 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:33:32.753 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:33:32.754 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:33:32.754 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:33:32.754 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:33:32.754 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:33:32.755 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:33:32.755 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:33:32.755 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:33:32.755 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:33:32.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.756 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.756 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:33:32.756 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:33:32.756 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.757 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:33:32.757 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:33:32.757 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:32.757 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:32.758 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:32.758 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:32.758 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:32.758 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:32.759 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:32.759 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 227, 2021-04-09T18:33:32.759 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:32.760 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:32.760 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:32.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.761 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:33:32.761 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:32.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.762 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.762 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.762 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:33:32.762 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:32.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.763 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.763 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:32.764 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.764 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.764 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.764 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:33:32.765 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:32.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.765 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.766 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:33:32.766 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:32.766 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.766 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.767 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:32.767 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.768 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.768 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:33:32.769 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:32.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.770 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.770 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.771 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:33:32.771 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:32.771 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.772 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.773 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:32.773 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.773 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.774 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.774 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:33:32.774 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:32.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.775 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.775 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.775 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:33:32.775 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:33:32.775 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.776 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.776 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:33:32.776 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:33:32.777 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:33:32.777 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:33:32.777 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:32.777 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:32.777 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:32.778 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.778 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.778 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:33:32.779 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:33:32.779 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:32.779 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:32.779 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:32.779 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:32.780 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:32.780 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:32.780 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:32.780 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 227, 2021-04-09T18:33:32.781 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:32.781 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:32.781 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:32.781 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.781 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.782 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.782 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:33:32.782 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:32.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.783 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.783 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.783 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:33:32.783 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:32.784 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.784 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.785 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:32.785 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.785 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.785 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.786 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:33:32.786 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:32.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.786 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.787 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.787 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:33:32.787 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:32.787 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.787 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.788 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:32.788 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.788 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.789 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:33:32.789 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:32.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.789 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.790 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.790 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:33:32.790 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:32.790 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.791 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.791 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:32.791 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.792 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.792 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:33:32.792 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:32.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.793 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.793 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.793 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:33:32.793 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:33:32.793 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.794 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.794 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:33:32.794 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:33:32.794 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:33:32.795 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:33:32.795 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:32.795 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:32.795 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:32.796 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.796 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.796 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:33:32.797 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:33:32.797 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:32.797 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:32.797 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:32.797 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:32.798 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:32.798 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:32.798 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:32.798 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:32.799 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:32.799 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:32.799 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:32.799 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.799 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.800 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.800 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:33:32.800 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:32.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.801 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.801 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.801 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:33:32.801 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:32.801 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.802 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.802 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:32.802 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.803 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:33:32.803 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:32.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.804 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.804 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:33:32.804 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:32.804 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.805 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.805 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:32.805 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.806 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.806 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:33:32.806 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:32.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.807 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.807 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:33:32.807 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:32.808 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.808 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.808 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:32.808 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.809 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.809 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:33:32.810 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:32.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.810 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.810 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.811 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:33:32.811 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:33:32.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.811 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.812 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:33:32.812 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:33:32.813 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:33:32.813 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:33:32.813 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:32.813 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:32.814 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:32.814 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.814 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:33:32.815 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:33:32.815 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:32.815 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:32.815 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:32.816 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:32.816 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:32.816 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:32.816 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:32.817 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:32.817 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:32.817 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:32.817 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:32.817 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.818 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.818 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:33:32.818 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:32.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.819 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.819 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:33:32.819 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:32.820 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.820 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.820 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:32.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.821 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.821 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:33:32.821 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:32.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.822 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:33:32.822 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:32.823 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.823 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.823 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:32.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.824 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.824 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:33:32.824 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:32.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.825 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.825 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:33:32.826 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:32.826 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.826 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.826 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:32.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.827 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:33:32.828 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:32.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.828 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.828 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:33:32.829 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:33:32.829 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.829 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.830 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:33:32.830 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:33:32.830 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:33:32.830 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:33:32.830 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:32.831 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:32.831 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:32.831 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.832 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:33:32.832 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:33:32.832 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:32.832 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:32.833 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:32.833 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:32.833 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:32.833 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:32.833 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:32.834 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:32.834 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:32.834 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:32.834 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:32.835 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.835 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.835 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.835 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:33:32.836 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:32.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.836 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:33:32.837 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:32.837 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.837 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.838 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:32.838 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.838 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.839 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:33:32.839 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:32.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.839 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.840 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:33:32.840 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:32.840 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.841 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.841 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:32.841 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.842 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.842 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.842 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:33:32.842 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:32.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.843 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.843 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:33:32.843 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:32.844 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.844 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.844 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:32.845 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.845 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:33:32.845 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:32.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.846 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.846 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.846 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:33:32.847 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:33:32.847 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.847 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:33:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:33:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:33:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:33:32.848 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:32.849 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:32.849 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:32.849 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.850 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:33:32.850 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:33:32.850 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:32.850 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:32.851 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:32.851 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:32.851 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:32.851 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:32.851 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:32.852 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 224, 2021-04-09T18:33:32.852 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:32.852 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:32.852 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:32.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.853 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.853 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:33:32.853 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:32.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:33:32.854 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:32.855 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.855 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.856 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:33:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.857 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.858 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:33:32.858 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:32.858 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.858 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.859 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:32.859 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.859 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.860 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:33:32.860 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:32.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.860 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.861 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.861 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:33:32.861 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:32.861 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.862 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.862 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:32.862 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.863 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.863 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:33:32.863 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:32.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.864 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.864 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:33:32.864 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:33:32.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.865 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.865 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:33:32.865 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:33:32.866 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:33:32.866 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:33:32.866 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:32.866 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:32.866 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:32.867 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.867 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.867 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:33:32.868 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:33:32.868 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:32.868 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:32.868 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:32.868 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:32.869 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:32.869 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:32.869 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:32.869 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:32.869 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:32.870 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:32.870 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:32.870 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.871 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:33:32.871 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:32.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.871 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.872 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:33:32.872 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:32.872 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.872 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.873 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:32.873 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.874 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.874 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:33:32.874 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:32.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.874 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.875 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.875 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:33:32.875 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:32.875 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.875 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.876 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:32.876 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.877 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.877 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:33:32.877 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:32.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.877 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.878 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.878 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:33:32.878 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:32.878 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.879 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.879 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:32.879 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.879 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.880 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:33:32.880 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:32.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.881 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:33:32.881 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:33:32.881 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:33:32.882 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:33:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:33:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:33:32.883 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:32.884 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.885 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:33:32.886 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:33:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:33:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:33:32.887 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:33:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:33:32.888 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:33:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:33:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:33:32.889 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 228, 2021-04-09T18:33:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:33:32.890 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:33:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:33:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:33:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:32.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:33:32.893 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:33:32.894 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:33:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:32.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:33:32.896 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.897 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:33:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:33:32.898 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.899 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:33:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.900 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:33:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:33:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:33:32.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:33:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:32.902 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:33:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:33:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:33:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:33:32.904 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:33:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:33:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:33:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:33:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:33:32.905 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:33:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:33:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.906 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:33:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:32.907 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.908 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:33:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:32.909 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:33:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:32.910 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:32.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:33:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:32.912 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:32.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:33:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:32.914 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:32.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:33:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:32.916 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:33:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:32.918 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:32.919 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:33:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:33:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:33:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:33:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:33:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:33:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:33:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:33:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:33:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:33:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:33:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:33:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6834/222606831": "2021-04-10 18:33:22.994232", 2021-04-09T18:33:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/2918355247": "2021-04-10 18:33:22.531505", 2021-04-09T18:33:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/2918355247": "2021-04-10 18:33:22.531505", 2021-04-09T18:33:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6837/1293489350": "2021-04-10 18:33:07.891065", 2021-04-09T18:33:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/1293489350": "2021-04-10 18:33:07.891065", 2021-04-09T18:33:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/2382296066": "2021-04-10 18:33:00.283626", 2021-04-09T18:33:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/2382296066": "2021-04-10 18:33:00.283626", 2021-04-09T18:33:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6832/222606831": "2021-04-10 18:33:22.994232", 2021-04-09T18:33:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/3107210651": "2021-04-10 18:32:51.914410", 2021-04-09T18:33:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/3107210651": "2021-04-10 18:32:51.914410", 2021-04-09T18:33:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6837/2051228504": "2021-04-10 18:32:51.911792", 2021-04-09T18:33:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6835/2051228504": "2021-04-10 18:32:51.911792" 2021-04-09T18:33:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.925 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:33:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:33:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:33:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:33:32.926 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:33:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:33:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:33:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:33:32.927 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:33:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:33:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:33:32.928 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:33:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:33:32.929 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:33:32.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.718 7f88f2156700 1 -- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88cc018930 msgr2=0x7f88cc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:32.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.718 7f88f2156700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88cc018930 0x7f88cc01ae10 crc :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.718 7f88f2156700 1 -- 172.21.15.32:0/3102432847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0b5530 msgr2=0x7f88ec197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.718 7f88f2156700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0b5530 0x7f88ec197ef0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7f88e40099a0 tx=0x7f88e400bcf0).stop 2021-04-09T18:33:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.719 7f88f2156700 1 -- 172.21.15.32:0/3102432847 shutdown_connections 2021-04-09T18:33:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.719 7f88f2156700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f88cc018930 0x7f88cc01ae10 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.719 7f88f2156700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88ec0b5530 0x7f88ec197ef0 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.719 7f88f2156700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88ec193440 0x7f88ec0b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.719 7f88f2156700 1 --2- 172.21.15.32:0/3102432847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88ec0aa220 0x7f88ec0a99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.719 7f88f2156700 1 -- 172.21.15.32:0/3102432847 >> 172.21.15.32:0/3102432847 conn(0x7f88ec1a0320 msgr2=0x7f88ec19ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:32.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.719 7f88f2156700 1 -- 172.21.15.32:0/3102432847 shutdown_connections 2021-04-09T18:33:32.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:32.719 7f88f2156700 1 -- 172.21.15.32:0/3102432847 wait complete. 2021-04-09T18:33:32.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:33.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.072 7f88a76c9700 1 Processor -- start 2021-04-09T18:33:33.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.072 7f88a76c9700 1 -- start start 2021-04-09T18:33:33.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.073 7f88a76c9700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88a010d010 0x7f88a00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.073 7f88a76c9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a010d480 con 0x7f88a00d1da0 2021-04-09T18:33:33.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.073 7f88a76c9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a010e1f0 con 0x7f88a00d8490 2021-04-09T18:33:33.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.073 7f88a76c9700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a00d8d30 con 0x7f88a010d010 2021-04-09T18:33:33.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.073 7f88a4ce4700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f88a00d1da0 0x7f88a00d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40078/0 (socket says 172.21.15.32:40078) 2021-04-09T18:33:33.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.073 7f88a4ce4700 1 -- 172.21.15.32:0/3729972195 learned_addr learned my addr 172.21.15.32:0/3729972195 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:33.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f88a54e5700 1 --2- 172.21.15.32:0/3729972195 >> v2:172.21.15.86:3300/0 conn(0x7f88a010d010 0x7f88a00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f8897fff700 1 -- 172.21.15.32:0/3729972195 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623094461 0 0) 0x7f8890000ce0 con 0x7f88a00d1da0 2021-04-09T18:33:33.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f8897fff700 1 -- 172.21.15.32:0/3729972195 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888c002fe0 con 0x7f88a00d1da0 2021-04-09T18:33:33.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f8897fff700 1 -- 172.21.15.32:0/3729972195 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2809959007 0 0) 0x7f88900010e0 con 0x7f88a00d1da0 2021-04-09T18:33:33.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f88a54e5700 1 -- 172.21.15.32:0/3729972195 >> v1:172.21.15.86:6789/0 conn(0x7f88a00d8490 legacy=0x7f88a010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f88a54e5700 1 -- 172.21.15.32:0/3729972195 >> v1:172.21.15.32:6789/0 conn(0x7f88a00d1da0 legacy=0x7f88a00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f88a54e5700 1 -- 172.21.15.32:0/3729972195 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a00d9220 con 0x7f88a010d010 2021-04-09T18:33:33.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f88a54e5700 1 --2- 172.21.15.32:0/3729972195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88a010d010 0x7f88a00d1780 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f889c0069d0 tx=0x7f889c009dc0).ready entity=mon.1 client_cookie=f4931590f6913bec server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:33.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f8897fff700 1 -- 172.21.15.32:0/3729972195 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f889c00b510 con 0x7f88a010d010 2021-04-09T18:33:33.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f8897fff700 1 -- 172.21.15.32:0/3729972195 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c006ac0 con 0x7f88a010d010 2021-04-09T18:33:33.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f8897fff700 1 -- 172.21.15.32:0/3729972195 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f889c00ba80 con 0x7f88a010d010 2021-04-09T18:33:33.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f88a76c9700 1 -- 172.21.15.32:0/3729972195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88a010d010 msgr2=0x7f88a00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.074 7f88a76c9700 1 --2- 172.21.15.32:0/3729972195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88a010d010 0x7f88a00d1780 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f889c0069d0 tx=0x7f889c009dc0).stop 2021-04-09T18:33:33.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.075 7f88a76c9700 1 -- 172.21.15.32:0/3729972195 shutdown_connections 2021-04-09T18:33:33.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.075 7f88a76c9700 1 --2- 172.21.15.32:0/3729972195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88a010d010 0x7f88a00d1780 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.075 7f88a76c9700 1 -- 172.21.15.32:0/3729972195 >> 172.21.15.32:0/3729972195 conn(0x7f88a00ce5f0 msgr2=0x7f88a00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:33.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.075 7f88a76c9700 1 -- 172.21.15.32:0/3729972195 shutdown_connections 2021-04-09T18:33:33.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.075 7f88a76c9700 1 -- 172.21.15.32:0/3729972195 wait complete. 2021-04-09T18:33:33.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.095 7f88a76c9700 1 Processor -- start 2021-04-09T18:33:33.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.095 7f88a76c9700 1 -- start start 2021-04-09T18:33:33.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a76c9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88a00a8b10 0x7f88a01b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a76c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88a01b8b60 0x7f88a01ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a76c9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88a00a8f10 0x7f88a01abc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a76c9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a01ac220 con 0x7f88a01b8b60 2021-04-09T18:33:33.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a76c9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a008cf40 con 0x7f88a00a8b10 2021-04-09T18:33:33.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a76c9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a008d160 con 0x7f88a00a8f10 2021-04-09T18:33:33.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a4ce4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88a01b8b60 0x7f88a01ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a4ce4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88a01b8b60 0x7f88a01ab5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58722/0 (socket says 172.21.15.32:58722) 2021-04-09T18:33:33.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a4ce4700 1 -- 172.21.15.32:0/1819800145 learned_addr learned my addr 172.21.15.32:0/1819800145 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:33.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a54e5700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88a00a8b10 0x7f88a01b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.096 7f88a5ce6700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88a00a8f10 0x7f88a01abc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f88a4ce4700 1 -- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88a00a8f10 msgr2=0x7f88a01abc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f88a4ce4700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88a00a8f10 0x7f88a01abc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f88a4ce4700 1 -- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88a00a8b10 msgr2=0x7f88a01b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f88a4ce4700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88a00a8b10 0x7f88a01b8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f88a4ce4700 1 -- 172.21.15.32:0/1819800145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a008d380 con 0x7f88a01b8b60 2021-04-09T18:33:33.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f88a4ce4700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88a01b8b60 0x7f88a01ab5e0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f8890000f90 tx=0x7f889000a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:33.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f8895ffb700 1 -- 172.21.15.32:0/1819800145 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8890001a30 con 0x7f88a01b8b60 2021-04-09T18:33:33.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f8895ffb700 1 -- 172.21.15.32:0/1819800145 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88900043a0 con 0x7f88a01b8b60 2021-04-09T18:33:33.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a008d380 con 0x7f88a01b8b60 2021-04-09T18:33:33.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f8895ffb700 1 -- 172.21.15.32:0/1819800145 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88900047f0 con 0x7f88a01b8b60 2021-04-09T18:33:33.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.097 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a008d380 con 0x7f88a01b8b60 2021-04-09T18:33:33.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.098 7f8895ffb700 1 -- 172.21.15.32:0/1819800145 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8890004580 con 0x7f88a01b8b60 2021-04-09T18:33:33.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.098 7f8895ffb700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8880018930 0x7f888001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.098 7f8895ffb700 1 -- 172.21.15.32:0/1819800145 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f88900328b0 con 0x7f88a01b8b60 2021-04-09T18:33:33.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.098 7f88a54e5700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8880018930 0x7f888001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.098 7f88a54e5700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8880018930 0x7f888001adc0 crc :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:33.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.117 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a01ab0b0 con 0x7f88a01b8b60 2021-04-09T18:33:33.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.121 7f8895ffb700 1 -- 172.21.15.32:0/1819800145 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8890001a10 con 0x7f88a01b8b60 2021-04-09T18:33:33.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.311 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88a01ab0b0 con 0x7f88a01b8b60 2021-04-09T18:33:33.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.312 7f8895ffb700 1 -- 172.21.15.32:0/1819800145 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 406 v406) v1 ==== 95+0+1858 (secure 0 0 0) 0x7f889000ad30 con 0x7f88a01b8b60 2021-04-09T18:33:33.329 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 406 2021-04-09T18:33:33.329 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:33.329 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":406,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2768006990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2768006990},{"type":"v1","addr":"172.21.15.86:6837","nonce":2768006990}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401},{"gid":12384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3001291771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3001291771},{"type":"v1","addr":"172.21.15.32:6835","nonce":3001291771}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":405},{"gid":12396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2820215844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2820215844},{"type":"v1","addr":"172.21.15.32:6834","nonce":2820215844}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":406},{"gid":59606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3794784835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3794784835},{"type":"v1","addr":"172.21.15.86:6836","nonce":3794784835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401}],"filesystems":[]} 2021-04-09T18:33:33.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8880018930 msgr2=0x7f888001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8880018930 0x7f888001adc0 crc :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88a01b8b60 msgr2=0x7f88a01ab5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88a01b8b60 0x7f88a01ab5e0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f8890000f90 tx=0x7f889000a940).stop 2021-04-09T18:33:33.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 shutdown_connections 2021-04-09T18:33:33.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8880018930 0x7f888001adc0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88a01b8b60 0x7f88a01ab5e0 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88a00a8f10 0x7f88a01abc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 --2- 172.21.15.32:0/1819800145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88a00a8b10 0x7f88a01b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 >> 172.21.15.32:0/1819800145 conn(0x7f88a0190570 msgr2=0x7f88a018ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:33.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 shutdown_connections 2021-04-09T18:33:33.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.314 7f88a76c9700 1 -- 172.21.15.32:0/1819800145 wait complete. 2021-04-09T18:33:33.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:33.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.474 7f60c6268700 1 Processor -- start 2021-04-09T18:33:33.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.474 7f60c6268700 1 -- start start 2021-04-09T18:33:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c6268700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f60c00d7ca0 0x7f60c00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c6268700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60c010da00 0x7f60c010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c6268700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c010d480 con 0x7f60c010d010 2021-04-09T18:33:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c6268700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c00d9220 con 0x7f60c010da00 2021-04-09T18:33:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c6268700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c00d86c0 con 0x7f60c00d7ca0 2021-04-09T18:33:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c4885700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60c010da00 0x7f60c010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60bffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f60c010d010 0x7f60c00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40088/0 (socket says 172.21.15.32:40088) 2021-04-09T18:33:33.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c4885700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f60c010da00 0x7f60c010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58734/0 (socket says 172.21.15.32:58734) 2021-04-09T18:33:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60b7fff700 1 --2- 172.21.15.32:0/115312926 >> v2:172.21.15.86:3300/0 conn(0x7f60c00d7ca0 0x7f60c00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60bffff700 1 -- 172.21.15.32:0/115312926 learned_addr learned my addr 172.21.15.32:0/115312926 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c4885700 1 -- 172.21.15.32:0/115312926 >> v2:172.21.15.86:3300/0 conn(0x7f60c00d7ca0 msgr2=0x7f60c00d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c4885700 1 --2- 172.21.15.32:0/115312926 >> v2:172.21.15.86:3300/0 conn(0x7f60c00d7ca0 0x7f60c00d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c4885700 1 -- 172.21.15.32:0/115312926 >> v1:172.21.15.32:6789/0 conn(0x7f60c010d010 legacy=0x7f60c00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c4885700 1 -- 172.21.15.32:0/115312926 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c00d1f70 con 0x7f60c010da00 2021-04-09T18:33:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60bf7fe700 1 -- 172.21.15.32:0/115312926 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068147417 0 0) 0x7f60b0000ce0 con 0x7f60c010d010 2021-04-09T18:33:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60b7fff700 1 --2- 172.21.15.32:0/115312926 >> v2:172.21.15.86:3300/0 conn(0x7f60c00d7ca0 0x7f60c00d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60c4885700 1 --2- 172.21.15.32:0/115312926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c010da00 0x7f60c010de00 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f60b8006e30 tx=0x7f60b8009e60).ready entity=mon.0 client_cookie=73dbf31d8afe1e1d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60bf7fe700 1 -- 172.21.15.32:0/115312926 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b8006a50 con 0x7f60c010da00 2021-04-09T18:33:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60bf7fe700 1 -- 172.21.15.32:0/115312926 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b800b300 con 0x7f60c010da00 2021-04-09T18:33:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.475 7f60bf7fe700 1 -- 172.21.15.32:0/115312926 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b800b750 con 0x7f60c010da00 2021-04-09T18:33:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.476 7f60c6268700 1 -- 172.21.15.32:0/115312926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c010da00 msgr2=0x7f60c010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.476 7f60c6268700 1 --2- 172.21.15.32:0/115312926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c010da00 0x7f60c010de00 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f60b8006e30 tx=0x7f60b8009e60).stop 2021-04-09T18:33:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.476 7f60c6268700 1 -- 172.21.15.32:0/115312926 shutdown_connections 2021-04-09T18:33:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.476 7f60c6268700 1 --2- 172.21.15.32:0/115312926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c010da00 0x7f60c010de00 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.476 7f60c6268700 1 --2- 172.21.15.32:0/115312926 >> v2:172.21.15.86:3300/0 conn(0x7f60c00d7ca0 0x7f60c00d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.476 7f60c6268700 1 -- 172.21.15.32:0/115312926 >> 172.21.15.32:0/115312926 conn(0x7f60c00ce5f0 msgr2=0x7f60c00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.476 7f60c6268700 1 -- 172.21.15.32:0/115312926 shutdown_connections 2021-04-09T18:33:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.476 7f60c6268700 1 -- 172.21.15.32:0/115312926 wait complete. 2021-04-09T18:33:33.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.496 7f60c6268700 1 Processor -- start 2021-04-09T18:33:33.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.496 7f60c6268700 1 -- start start 2021-04-09T18:33:33.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c6268700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c01acea0 0x7f60c01ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c6268700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c019b2c0 0x7f60c0184da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c6268700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c01853c0 0x7f60c0185fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c6268700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c01857c0 con 0x7f60c01853c0 2021-04-09T18:33:33.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c6268700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c0185a40 con 0x7f60c01acea0 2021-04-09T18:33:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c6268700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c01865e0 con 0x7f60c019b2c0 2021-04-09T18:33:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c4885700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c01853c0 0x7f60c0185fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c4885700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c01853c0 0x7f60c0185fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58740/0 (socket says 172.21.15.32:58740) 2021-04-09T18:33:33.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c4885700 1 -- 172.21.15.32:0/3920135664 learned_addr learned my addr 172.21.15.32:0/3920135664 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60b7fff700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c019b2c0 0x7f60c0184da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60bffff700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c01acea0 0x7f60c01ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c4885700 1 -- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c019b2c0 msgr2=0x7f60c0184da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.497 7f60c4885700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c019b2c0 0x7f60c0184da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60c4885700 1 -- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c01acea0 msgr2=0x7f60c01ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60c4885700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c01acea0 0x7f60c01ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60c4885700 1 -- 172.21.15.32:0/3920135664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c0186800 con 0x7f60c01853c0 2021-04-09T18:33:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60c4885700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c01853c0 0x7f60c0185fc0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 rx=0x7f60b800b970 tx=0x7f60b800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:33.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60bd7fa700 1 -- 172.21.15.32:0/3920135664 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b8009380 con 0x7f60c01853c0 2021-04-09T18:33:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60bd7fa700 1 -- 172.21.15.32:0/3920135664 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b8000f60 con 0x7f60c01853c0 2021-04-09T18:33:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60c6268700 1 -- 172.21.15.32:0/3920135664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c0186800 con 0x7f60c01853c0 2021-04-09T18:33:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60bd7fa700 1 -- 172.21.15.32:0/3920135664 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b8005aa0 con 0x7f60c01853c0 2021-04-09T18:33:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60c6268700 1 -- 172.21.15.32:0/3920135664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c0186800 con 0x7f60c01853c0 2021-04-09T18:33:33.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.498 7f60bd7fa700 1 -- 172.21.15.32:0/3920135664 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f60b8005820 con 0x7f60c01853c0 2021-04-09T18:33:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.499 7f60bd7fa700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60a40188d0 0x7f60a401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.499 7f60bd7fa700 1 -- 172.21.15.32:0/3920135664 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f60b8032cd0 con 0x7f60c01853c0 2021-04-09T18:33:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.499 7f60bffff700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60a40188d0 0x7f60a401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.499 7f60bffff700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60a40188d0 0x7f60a401add0 crc :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:33.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.519 7f60c6268700 1 -- 172.21.15.32:0/3920135664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c0182d80 con 0x7f60c01853c0 2021-04-09T18:33:33.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.522 7f60bd7fa700 1 -- 172.21.15.32:0/3920135664 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f60b8009380 con 0x7f60c01853c0 2021-04-09T18:33:33.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.714 7f60c6268700 1 -- 172.21.15.32:0/3920135664 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60c010d480 con 0x7f60c01853c0 2021-04-09T18:33:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.714 7f60bd7fa700 1 -- 172.21.15.32:0/3920135664 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 406 v406) v1 ==== 95+0+1858 (secure 0 0 0) 0x7f60b8006ee0 con 0x7f60c01853c0 2021-04-09T18:33:33.732 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 406 2021-04-09T18:33:33.733 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:33.733 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":406,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/2768006990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2768006990},{"type":"v1","addr":"172.21.15.86:6837","nonce":2768006990}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401},{"gid":12384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3001291771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":3001291771},{"type":"v1","addr":"172.21.15.32:6835","nonce":3001291771}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":405},{"gid":12396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2820215844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2820215844},{"type":"v1","addr":"172.21.15.32:6834","nonce":2820215844}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":406},{"gid":59606,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/3794784835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3794784835},{"type":"v1","addr":"172.21.15.86:6836","nonce":3794784835}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":401}],"filesystems":[]} 2021-04-09T18:33:33.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 -- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60a40188d0 msgr2=0x7f60a401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60a40188d0 0x7f60a401add0 crc :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 -- 172.21.15.32:0/3920135664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c01853c0 msgr2=0x7f60c0185fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c01853c0 0x7f60c0185fc0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 rx=0x7f60b800b970 tx=0x7f60b800bd10).stop 2021-04-09T18:33:33.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 -- 172.21.15.32:0/3920135664 shutdown_connections 2021-04-09T18:33:33.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f60a40188d0 0x7f60a401add0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c01853c0 0x7f60c0185fc0 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c019b2c0 0x7f60c0184da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 --2- 172.21.15.32:0/3920135664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c01acea0 0x7f60c01ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 -- 172.21.15.32:0/3920135664 >> 172.21.15.32:0/3920135664 conn(0x7f60c019b890 msgr2=0x7f60c01a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:33.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 -- 172.21.15.32:0/3920135664 shutdown_connections 2021-04-09T18:33:33.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.717 7f60c6268700 1 -- 172.21.15.32:0/3920135664 wait complete. 2021-04-09T18:33:33.743 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:33:33.744 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:33:33.744 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:33:33.744 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:33:33.745 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:33:33.745 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:33:33.745 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:33:33.745 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:33:33.746 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:33:33.746 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:33:33.747 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:33:33.747 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:33:33.747 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:33:33.748 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:33:33.748 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:33:33.748 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:33:33.752 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:33:33.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:33:33.754 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:33:33.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:33:33.756 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:33:33.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:33:33.758 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:33:33.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:33:33.795 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:33:33.795 DEBUG:teuthology.parallel:result is None 2021-04-09T18:33:33.797 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:33:33.797 DEBUG:teuthology.parallel:result is None 2021-04-09T18:33:33.798 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:33:33.798 DEBUG:teuthology.parallel:result is None 2021-04-09T18:33:33.799 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:33:33.800 DEBUG:teuthology.parallel:result is None 2021-04-09T18:33:33.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:33:33.939 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:33:33.940 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:33:33.948 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:33:33.950 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:33:33.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346d2e3700 1 Processor -- start 2021-04-09T18:33:33.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346d2e3700 1 -- start start 2021-04-09T18:33:33.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346d2e3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34680cffb0 0x7f34680d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346d2e3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f34680d6c90 0x7f346810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346d2e3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34680d7290 con 0x7f34680d7810 2021-04-09T18:33:33.984 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346d2e3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34680d7090 con 0x7f34680cffb0 2021-04-09T18:33:33.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346d2e3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34680d91f0 con 0x7f34680d6c90 2021-04-09T18:33:33.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346707d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34680cffb0 0x7f34680d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346707d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f34680cffb0 0x7f34680d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58768/0 (socket says 172.21.15.32:58768) 2021-04-09T18:33:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.939 7f346707d700 1 -- 172.21.15.32:0/3334292046 learned_addr learned my addr 172.21.15.32:0/3334292046 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:33.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346787e700 1 --2- 172.21.15.32:0/3334292046 >> v2:172.21.15.86:3300/0 conn(0x7f34680d6c90 0x7f346810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346707d700 1 -- 172.21.15.32:0/3334292046 >> v2:172.21.15.86:3300/0 conn(0x7f34680d6c90 msgr2=0x7f346810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:33.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346707d700 1 --2- 172.21.15.32:0/3334292046 >> v2:172.21.15.86:3300/0 conn(0x7f34680d6c90 0x7f346810cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346707d700 1 -- 172.21.15.32:0/3334292046 >> v1:172.21.15.86:6789/0 conn(0x7f34680d7810 legacy=0x7f34680d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346707d700 1 -- 172.21.15.32:0/3334292046 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680d8400 con 0x7f34680cffb0 2021-04-09T18:33:33.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346707d700 1 --2- 172.21.15.32:0/3334292046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34680cffb0 0x7f34680d2450 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 rx=0x7f3458006a80 tx=0x7f3458009f00).ready entity=mon.0 client_cookie=51b543625fe23c62 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:33.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346687c700 1 -- 172.21.15.32:0/3334292046 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f345800b5c0 con 0x7f34680cffb0 2021-04-09T18:33:33.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346687c700 1 -- 172.21.15.32:0/3334292046 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345800b8c0 con 0x7f34680cffb0 2021-04-09T18:33:33.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346687c700 1 -- 172.21.15.32:0/3334292046 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f345800bd20 con 0x7f34680cffb0 2021-04-09T18:33:33.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346d2e3700 1 -- 172.21.15.32:0/3334292046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34680cffb0 msgr2=0x7f34680d2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.940 7f346d2e3700 1 --2- 172.21.15.32:0/3334292046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34680cffb0 0x7f34680d2450 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 rx=0x7f3458006a80 tx=0x7f3458009f00).stop 2021-04-09T18:33:33.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.941 7f346d2e3700 1 -- 172.21.15.32:0/3334292046 shutdown_connections 2021-04-09T18:33:33.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.941 7f346d2e3700 1 --2- 172.21.15.32:0/3334292046 >> v2:172.21.15.86:3300/0 conn(0x7f34680d6c90 0x7f346810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.941 7f346d2e3700 1 --2- 172.21.15.32:0/3334292046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34680cffb0 0x7f34680d2450 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.941 7f346d2e3700 1 -- 172.21.15.32:0/3334292046 >> 172.21.15.32:0/3334292046 conn(0x7f34680ce5d0 msgr2=0x7f34680cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:33.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.941 7f346d2e3700 1 -- 172.21.15.32:0/3334292046 shutdown_connections 2021-04-09T18:33:33.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.941 7f346d2e3700 1 -- 172.21.15.32:0/3334292046 wait complete. 2021-04-09T18:33:33.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.961 7f346d2e3700 1 Processor -- start 2021-04-09T18:33:33.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.961 7f346d2e3700 1 -- start start 2021-04-09T18:33:33.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346d2e3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3468184ec0 0x7f3468198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346d2e3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34681abf00 0x7f34681accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346d2e3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34681ac300 0x7f3468184020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:33.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346d2e3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34681ac750 con 0x7f3468184ec0 2021-04-09T18:33:33.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346d2e3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34681ad2f0 con 0x7f34681abf00 2021-04-09T18:33:33.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346d2e3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3468188cb0 con 0x7f34681ac300 2021-04-09T18:33:33.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346707d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3468184ec0 0x7f3468198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346707d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3468184ec0 0x7f3468198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58774/0 (socket says 172.21.15.32:58774) 2021-04-09T18:33:33.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346707d700 1 -- 172.21.15.32:0/3888344618 learned_addr learned my addr 172.21.15.32:0/3888344618 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346787e700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34681ac300 0x7f3468184020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f345e87c700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34681abf00 0x7f34681accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346707d700 1 -- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34681ac300 msgr2=0x7f3468184020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:33.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346707d700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34681ac300 0x7f3468184020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:33.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346707d700 1 -- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34681abf00 msgr2=0x7f34681accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:34.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346707d700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34681abf00 0x7f34681accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:34.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346707d700 1 -- 172.21.15.32:0/3888344618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3468184640 con 0x7f3468184ec0 2021-04-09T18:33:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f346707d700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3468184ec0 0x7f3468198170 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 rx=0x7f3458009450 tx=0x7f345800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.962 7f3464878700 1 -- 172.21.15.32:0/3888344618 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3458006e40 con 0x7f3468184ec0 2021-04-09T18:33:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.963 7f3464878700 1 -- 172.21.15.32:0/3888344618 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345800be60 con 0x7f3468184ec0 2021-04-09T18:33:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.963 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3468184640 con 0x7f3468184ec0 2021-04-09T18:33:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.963 7f3464878700 1 -- 172.21.15.32:0/3888344618 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f345800c4e0 con 0x7f3468184ec0 2021-04-09T18:33:34.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.963 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3468184640 con 0x7f3468184ec0 2021-04-09T18:33:34.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.963 7f3464878700 1 -- 172.21.15.32:0/3888344618 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3458005660 con 0x7f3468184ec0 2021-04-09T18:33:34.003 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.963 7f3464878700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f344c018930 0x7f344c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:34.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.963 7f3464878700 1 -- 172.21.15.32:0/3888344618 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 5273+0+0 (secure 0 0 0) 0x7f3458032c80 con 0x7f3468184ec0 2021-04-09T18:33:34.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.963 7f345e87c700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f344c018930 0x7f344c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:34.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.964 7f345e87c700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f344c018930 0x7f344c01ae10 crc :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:34.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.983 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3468182e20 con 0x7f3468184ec0 2021-04-09T18:33:34.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:33.986 7f3464878700 1 -- 172.21.15.32:0/3888344618 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f345801faf0 con 0x7f3468184ec0 2021-04-09T18:33:34.207 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.191 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f34680d7290 con 0x7f3468184ec0 2021-04-09T18:33:34.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.648 7f3464878700 1 -- 172.21.15.32:0/3888344618 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v243) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3458005660 con 0x7f3468184ec0 2021-04-09T18:33:34.671 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:33:34.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f344c018930 msgr2=0x7f344c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:34.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f344c018930 0x7f344c01ae10 crc :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:34.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3468184ec0 msgr2=0x7f3468198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:34.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3468184ec0 0x7f3468198170 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 rx=0x7f3458009450 tx=0x7f345800bd20).stop 2021-04-09T18:33:34.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 shutdown_connections 2021-04-09T18:33:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f344c018930 0x7f344c01ae10 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34681ac300 0x7f3468184020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34681abf00 0x7f34681accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:34.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 --2- 172.21.15.32:0/3888344618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3468184ec0 0x7f3468198170 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:34.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 >> 172.21.15.32:0/3888344618 conn(0x7f34681838a0 msgr2=0x7f34681a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:34.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 shutdown_connections 2021-04-09T18:33:34.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.658 7f346d2e3700 1 -- 172.21.15.32:0/3888344618 wait complete. 2021-04-09T18:33:34.685 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:33:34.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:33:34.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.820 7ff0879c2700 1 Processor -- start 2021-04-09T18:33:34.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.821 7ff0879c2700 1 -- start start 2021-04-09T18:33:34.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.821 7ff0879c2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0800d7ca0 0x7ff0800d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.821 7ff0879c2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08010d480 con 0x7ff08010d010 2021-04-09T18:33:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.821 7ff0879c2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0800d86c0 con 0x7ff08010da00 2021-04-09T18:33:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.821 7ff0879c2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0800d9220 con 0x7ff0800d7ca0 2021-04-09T18:33:34.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.821 7ff084fdd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0800d7ca0 0x7ff0800d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.821 7ff0857de700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff08010d010 0x7ff0800d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40140/0 (socket says 172.21.15.32:40140) 2021-04-09T18:33:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.821 7ff084fdd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff0800d7ca0 0x7ff0800d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58786/0 (socket says 172.21.15.32:58786) 2021-04-09T18:33:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.821 7ff0857de700 1 -- 172.21.15.32:0/893807958 learned_addr learned my addr 172.21.15.32:0/893807958 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:34.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff084fdd700 1 -- 172.21.15.32:0/893807958 >> v1:172.21.15.86:6789/0 conn(0x7ff08010da00 legacy=0x7ff08010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff084fdd700 1 -- 172.21.15.32:0/893807958 >> v1:172.21.15.32:6789/0 conn(0x7ff08010d010 legacy=0x7ff0800d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff084fdd700 1 -- 172.21.15.32:0/893807958 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0800d1f70 con 0x7ff0800d7ca0 2021-04-09T18:33:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff077fff700 1 -- 172.21.15.32:0/893807958 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265511844 0 0) 0x7ff078000d30 con 0x7ff08010da00 2021-04-09T18:33:34.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff084fdd700 1 --2- 172.21.15.32:0/893807958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0800d7ca0 0x7ff0800d80a0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 rx=0x7ff070006650 tx=0x7ff070009a80).ready entity=mon.0 client_cookie=df488563273beec9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff077fff700 1 -- 172.21.15.32:0/893807958 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0700009b0 con 0x7ff0800d7ca0 2021-04-09T18:33:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff077fff700 1 -- 172.21.15.32:0/893807958 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07000c0c0 con 0x7ff0800d7ca0 2021-04-09T18:33:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff077fff700 1 -- 172.21.15.32:0/893807958 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff07000c510 con 0x7ff0800d7ca0 2021-04-09T18:33:34.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff0879c2700 1 -- 172.21.15.32:0/893807958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0800d7ca0 msgr2=0x7ff0800d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:34.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.822 7ff0879c2700 1 --2- 172.21.15.32:0/893807958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0800d7ca0 0x7ff0800d80a0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 rx=0x7ff070006650 tx=0x7ff070009a80).stop 2021-04-09T18:33:34.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.823 7ff0879c2700 1 -- 172.21.15.32:0/893807958 shutdown_connections 2021-04-09T18:33:34.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.823 7ff0879c2700 1 --2- 172.21.15.32:0/893807958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0800d7ca0 0x7ff0800d80a0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.823 7ff0879c2700 1 -- 172.21.15.32:0/893807958 >> 172.21.15.32:0/893807958 conn(0x7ff0800ce5f0 msgr2=0x7ff0800d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.823 7ff0879c2700 1 -- 172.21.15.32:0/893807958 shutdown_connections 2021-04-09T18:33:34.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.823 7ff0879c2700 1 -- 172.21.15.32:0/893807958 wait complete. 2021-04-09T18:33:34.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.843 7ff0879c2700 1 Processor -- start 2021-04-09T18:33:34.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.843 7ff0879c2700 1 -- start start 2021-04-09T18:33:34.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0879c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08017dca0 0x7ff08017e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0879c2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801a3320 0x7ff0801a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0879c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080180cb0 0x7ff0801810b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0879c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801a2da0 con 0x7ff08017dca0 2021-04-09T18:33:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0879c2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801a3d40 con 0x7ff080180cb0 2021-04-09T18:33:34.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0879c2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0801816d0 con 0x7ff0801a3320 2021-04-09T18:33:34.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0857de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08017dca0 0x7ff08017e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:34.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0857de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08017dca0 0x7ff08017e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58788/0 (socket says 172.21.15.32:58788) 2021-04-09T18:33:34.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0857de700 1 -- 172.21.15.32:0/4155269782 learned_addr learned my addr 172.21.15.32:0/4155269782 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff084fdd700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801a3320 0x7ff0801a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff085fdf700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080180cb0 0x7ff0801810b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0857de700 1 -- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801a3320 msgr2=0x7ff0801a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0857de700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801a3320 0x7ff0801a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:34.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0857de700 1 -- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080180cb0 msgr2=0x7ff0801810b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:34.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0857de700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080180cb0 0x7ff0801810b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:34.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0857de700 1 -- 172.21.15.32:0/4155269782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0801818f0 con 0x7ff08017dca0 2021-04-09T18:33:34.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff084fdd700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801a3320 0x7ff0801a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:34.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff085fdf700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080180cb0 0x7ff0801810b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.844 7ff0857de700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08017dca0 0x7ff08017e0a0 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 rx=0x7ff07c006f00 tx=0x7ff07c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.845 7ff075ffb700 1 -- 172.21.15.32:0/4155269782 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff07c006020 con 0x7ff08017dca0 2021-04-09T18:33:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.845 7ff075ffb700 1 -- 172.21.15.32:0/4155269782 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c006360 con 0x7ff08017dca0 2021-04-09T18:33:34.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.845 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0801818f0 con 0x7ff08017dca0 2021-04-09T18:33:34.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.845 7ff075ffb700 1 -- 172.21.15.32:0/4155269782 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff07c004620 con 0x7ff08017dca0 2021-04-09T18:33:34.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.845 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0801818f0 con 0x7ff08017dca0 2021-04-09T18:33:34.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.845 7ff075ffb700 1 -- 172.21.15.32:0/4155269782 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff07c00a000 con 0x7ff08017dca0 2021-04-09T18:33:34.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.845 7ff075ffb700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0600188d0 0x7ff06001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:34.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.845 7ff075ffb700 1 -- 172.21.15.32:0/4155269782 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 4757+0+0 (secure 0 0 0) 0x7ff07c032680 con 0x7ff08017dca0 2021-04-09T18:33:34.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.846 7ff084fdd700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0600188d0 0x7ff06001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:34.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.846 7ff084fdd700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0600188d0 0x7ff06001add0 crc :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:34.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.865 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08017caa0 con 0x7ff08017dca0 2021-04-09T18:33:34.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:34.869 7ff075ffb700 1 -- 172.21.15.32:0/4155269782 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff07c006340 con 0x7ff08017dca0 2021-04-09T18:33:35.069 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.053 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff08010d480 con 0x7ff08017dca0 2021-04-09T18:33:35.070 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.053 7ff075ffb700 1 -- 172.21.15.32:0/4155269782 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v46) v1 ==== 80+0+4589 (secure 0 0 0) 0x7ff07c006b90 con 0x7ff08017dca0 2021-04-09T18:33:35.071 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:35.072 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:33:35.072 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:33:35.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.073 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:33:35.073 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:33:35.074 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.074 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:33:35.075 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.075 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.075 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.076 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.077 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:33:35.077 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:33:35.078 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.078 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:33:35.079 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.079 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.080 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.080 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.081 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:33:35.082 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:33:35.082 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.082 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:33:35.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.083 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.084 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.085 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:33:35.085 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:33:35.085 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:33:35.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.086 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.087 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.087 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:33:35.088 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:33:35.088 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.088 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.088 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.088 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.089 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.089 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:33:35.090 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:33:35.090 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.090 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.091 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.091 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:33:35.092 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:33:35.092 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.092 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.092 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.092 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.093 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:33:35.094 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:33:35.094 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.094 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.094 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.094 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.095 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.095 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.095 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:33:35.096 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:33:35.096 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.096 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.096 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.097 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.097 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:33:35.098 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:33:35.098 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.098 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.098 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.098 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.099 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.100 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:33:35.100 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:33:35.100 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.100 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.100 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.101 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.101 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.101 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:33:35.102 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:33:35.102 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.102 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.102 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.103 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.104 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:33:35.104 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:33:35.104 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.104 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:33:35.104 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:33:35.105 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=alpha_data, allow rw pool=bravo_data" 2021-04-09T18:33:35.105 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.105 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:33:35.106 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:33:35.106 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.106 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:33:35.107 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:33:35.107 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=alpha_data, allow rw pool=bravo_data" 2021-04-09T18:33:35.107 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.108 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:33:35.108 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:33:35.108 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.108 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:33:35.109 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:33:35.109 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:33:35.109 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.109 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.110 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.110 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:33:35.110 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:33:35.110 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.110 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:33:35.111 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:33:35.111 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.111 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:35.112 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:35.112 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:33:35.112 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:33:35.112 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:33:35.112 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:33:35.113 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:33:35.113 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:33:35.113 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.113 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:35.113 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:33:35.114 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:33:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0600188d0 msgr2=0x7ff06001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0600188d0 0x7ff06001add0 crc :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08017dca0 msgr2=0x7ff08017e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08017dca0 0x7ff08017e0a0 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 rx=0x7ff07c006f00 tx=0x7ff07c00b940).stop 2021-04-09T18:33:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 shutdown_connections 2021-04-09T18:33:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff0600188d0 0x7ff06001add0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080180cb0 0x7ff0801810b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801a3320 0x7ff0801a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 --2- 172.21.15.32:0/4155269782 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff08017dca0 0x7ff08017e0a0 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 >> 172.21.15.32:0/4155269782 conn(0x7ff0801a98f0 msgr2=0x7ff080093990 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:35.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 shutdown_connections 2021-04-09T18:33:35.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.056 7ff0879c2700 1 -- 172.21.15.32:0/4155269782 wait complete. 2021-04-09T18:33:35.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:33:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.252 7fecdafd8700 1 Processor -- start 2021-04-09T18:33:35.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.252 7fecdafd8700 1 -- start start 2021-04-09T18:33:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.252 7fecdafd8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fecd410d010 0x7fecd40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.252 7fecdafd8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd410d480 con 0x7fecd40d8490 2021-04-09T18:33:35.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.252 7fecdafd8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd410e1f0 con 0x7fecd40d1da0 2021-04-09T18:33:35.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.252 7fecdafd8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd40d8d30 con 0x7fecd410d010 2021-04-09T18:33:35.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd95f5700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fecd40d8490 0x7fecd410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40154/0 (socket says 172.21.15.32:40154) 2021-04-09T18:33:35.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd95f5700 1 -- 172.21.15.32:0/217777679 learned_addr learned my addr 172.21.15.32:0/217777679 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:35.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd8df4700 1 --2- 172.21.15.32:0/217777679 >> v2:172.21.15.86:3300/0 conn(0x7fecd410d010 0x7fecd40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd37fe700 1 -- 172.21.15.32:0/217777679 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834658822 0 0) 0x7feccc0009b0 con 0x7fecd40d8490 2021-04-09T18:33:35.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd37fe700 1 -- 172.21.15.32:0/217777679 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc0002fe0 con 0x7fecd40d8490 2021-04-09T18:33:35.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd37fe700 1 -- 172.21.15.32:0/217777679 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928465728 0 0) 0x7fecc4000fe0 con 0x7fecd40d1da0 2021-04-09T18:33:35.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd37fe700 1 -- 172.21.15.32:0/217777679 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc0003210 con 0x7fecd40d1da0 2021-04-09T18:33:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd37fe700 1 -- 172.21.15.32:0/217777679 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 653159397 0 0) 0x7feccc0010e0 con 0x7fecd40d8490 2021-04-09T18:33:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd8df4700 1 -- 172.21.15.32:0/217777679 >> v1:172.21.15.86:6789/0 conn(0x7fecd40d1da0 legacy=0x7fecd40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd8df4700 1 -- 172.21.15.32:0/217777679 >> v1:172.21.15.32:6789/0 conn(0x7fecd40d8490 legacy=0x7fecd410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd8df4700 1 -- 172.21.15.32:0/217777679 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd40d9220 con 0x7fecd410d010 2021-04-09T18:33:35.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd8df4700 1 --2- 172.21.15.32:0/217777679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd410d010 0x7fecd40d1780 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7fecbc006650 tx=0x7fecbc009a80).ready entity=mon.1 client_cookie=8044ff82e2b3f256 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:35.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.253 7fecd37fe700 1 -- 172.21.15.32:0/217777679 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecbc0009b0 con 0x7fecd410d010 2021-04-09T18:33:35.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.254 7fecd37fe700 1 -- 172.21.15.32:0/217777679 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecbc00c0c0 con 0x7fecd410d010 2021-04-09T18:33:35.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.254 7fecd37fe700 1 -- 172.21.15.32:0/217777679 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecbc00c510 con 0x7fecd410d010 2021-04-09T18:33:35.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.254 7fecdafd8700 1 -- 172.21.15.32:0/217777679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd410d010 msgr2=0x7fecd40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.254 7fecdafd8700 1 --2- 172.21.15.32:0/217777679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd410d010 0x7fecd40d1780 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7fecbc006650 tx=0x7fecbc009a80).stop 2021-04-09T18:33:35.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.254 7fecdafd8700 1 -- 172.21.15.32:0/217777679 shutdown_connections 2021-04-09T18:33:35.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.254 7fecdafd8700 1 --2- 172.21.15.32:0/217777679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd410d010 0x7fecd40d1780 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.254 7fecdafd8700 1 -- 172.21.15.32:0/217777679 >> 172.21.15.32:0/217777679 conn(0x7fecd40ce5f0 msgr2=0x7fecd40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.254 7fecdafd8700 1 -- 172.21.15.32:0/217777679 shutdown_connections 2021-04-09T18:33:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.254 7fecdafd8700 1 -- 172.21.15.32:0/217777679 wait complete. 2021-04-09T18:33:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.274 7fecdafd8700 1 Processor -- start 2021-04-09T18:33:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.274 7fecdafd8700 1 -- start start 2021-04-09T18:33:35.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecdafd8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd41abcb0 0x7fecd41ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecdafd8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4191140 0x7fecd4191540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecdafd8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecd4191b60 0x7fecd41a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecdafd8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd4191f60 con 0x7fecd41abcb0 2021-04-09T18:33:35.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecdafd8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd41ac6d0 con 0x7fecd4191140 2021-04-09T18:33:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecdafd8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd41aa5c0 con 0x7fecd4191b60 2021-04-09T18:33:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecd8df4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd41abcb0 0x7fecd41ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecd8df4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd41abcb0 0x7fecd41ac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58800/0 (socket says 172.21.15.32:58800) 2021-04-09T18:33:35.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecd8df4700 1 -- 172.21.15.32:0/1710408198 learned_addr learned my addr 172.21.15.32:0/1710408198 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecd3fff700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4191140 0x7fecd4191540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecd95f5700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecd4191b60 0x7fecd41a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.275 7fecd8df4700 1 -- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecd4191b60 msgr2=0x7fecd41a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecd8df4700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecd4191b60 0x7fecd41a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecd8df4700 1 -- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4191140 msgr2=0x7fecd4191540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecd8df4700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4191140 0x7fecd4191540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecd8df4700 1 -- 172.21.15.32:0/1710408198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd41aa7e0 con 0x7fecd41abcb0 2021-04-09T18:33:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecd3fff700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4191140 0x7fecd4191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:35.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecd8df4700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd41abcb0 0x7fecd41ac0b0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7fecbc009650 tx=0x7fecbc00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:35.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecd17fa700 1 -- 172.21.15.32:0/1710408198 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecbc0093a0 con 0x7fecd41abcb0 2021-04-09T18:33:35.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecd17fa700 1 -- 172.21.15.32:0/1710408198 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecbc006070 con 0x7fecd41abcb0 2021-04-09T18:33:35.306 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd41aa7e0 con 0x7fecd41abcb0 2021-04-09T18:33:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecd17fa700 1 -- 172.21.15.32:0/1710408198 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecbc0052e0 con 0x7fecd41abcb0 2021-04-09T18:33:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.276 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd41aa7e0 con 0x7fecd41abcb0 2021-04-09T18:33:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.277 7fecd17fa700 1 -- 172.21.15.32:0/1710408198 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fecbc009300 con 0x7fecd41abcb0 2021-04-09T18:33:35.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.277 7fecd17fa700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecb80188c0 0x7fecb801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.277 7fecd17fa700 1 -- 172.21.15.32:0/1710408198 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fecbc032480 con 0x7fecd41abcb0 2021-04-09T18:33:35.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.277 7fecd3fff700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecb80188c0 0x7fecb801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.278 7fecd3fff700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecb80188c0 0x7fecb801adc0 crc :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:35.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.296 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd4191f60 con 0x7fecd41abcb0 2021-04-09T18:33:35.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.300 7fecd17fa700 1 -- 172.21.15.32:0/1710408198 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fecbc000940 con 0x7fecd41abcb0 2021-04-09T18:33:35.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.492 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecd4191f60 con 0x7fecd41abcb0 2021-04-09T18:33:35.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.493 7fecd17fa700 1 -- 172.21.15.32:0/1710408198 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+11154 (secure 0 0 0) 0x7fecbc009300 con 0x7fecd41abcb0 2021-04-09T18:33:35.510 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:35.511 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":243,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:33:34.643425","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:33:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecb80188c0 msgr2=0x7fecb801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecb80188c0 0x7fecb801adc0 crc :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd41abcb0 msgr2=0x7fecd41ac0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd41abcb0 0x7fecd41ac0b0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7fecbc009650 tx=0x7fecbc00b8e0).stop 2021-04-09T18:33:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 shutdown_connections 2021-04-09T18:33:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fecb80188c0 0x7fecb801adc0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecd4191b60 0x7fecd41a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecd4191140 0x7fecd4191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 --2- 172.21.15.32:0/1710408198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecd41abcb0 0x7fecd41ac0b0 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 >> 172.21.15.32:0/1710408198 conn(0x7fecd41837d0 msgr2=0x7fecd4195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 shutdown_connections 2021-04-09T18:33:35.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.495 7fecdafd8700 1 -- 172.21.15.32:0/1710408198 wait complete. 2021-04-09T18:33:35.522 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:33:35.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:33:35.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.660 7f6f2f380700 1 Processor -- start 2021-04-09T18:33:35.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.660 7f6f2f380700 1 -- start start 2021-04-09T18:33:35.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.660 7f6f2f380700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6f2810d010 0x7f6f280d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.660 7f6f2f380700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280d1da0 0x7f6f280d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.660 7f6f2f380700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2810d480 con 0x7f6f280d82c0 2021-04-09T18:33:35.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.660 7f6f2f380700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f280d9220 con 0x7f6f2810d010 2021-04-09T18:33:35.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.660 7f6f2f380700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f280d86c0 con 0x7f6f280d1da0 2021-04-09T18:33:35.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2c99b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280d1da0 0x7f6f280d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.700 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2d19c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6f2810d010 0x7f6f280d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2d99d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6f280d82c0 0x7f6f2810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38094/0 (socket says 172.21.15.32:38094) 2021-04-09T18:33:35.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2d99d700 1 -- 172.21.15.32:0/3824771266 learned_addr learned my addr 172.21.15.32:0/3824771266 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:35.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2c99b700 1 -- 172.21.15.32:0/3824771266 >> v2:172.21.15.86:3300/0 conn(0x7f6f2810d010 msgr2=0x7f6f280d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:35.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2c99b700 1 --2- 172.21.15.32:0/3824771266 >> v2:172.21.15.86:3300/0 conn(0x7f6f2810d010 0x7f6f280d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2c99b700 1 -- 172.21.15.32:0/3824771266 >> v1:172.21.15.86:6789/0 conn(0x7f6f280d82c0 legacy=0x7f6f2810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2c99b700 1 -- 172.21.15.32:0/3824771266 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2810e1f0 con 0x7f6f280d1da0 2021-04-09T18:33:35.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2d19c700 1 --2- 172.21.15.32:0/3824771266 >> v2:172.21.15.86:3300/0 conn(0x7f6f2810d010 0x7f6f280d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:35.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f1ffff700 1 -- 172.21.15.32:0/3824771266 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552351363 0 0) 0x7f6f20001440 con 0x7f6f280d82c0 2021-04-09T18:33:35.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f2c99b700 1 --2- 172.21.15.32:0/3824771266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280d1da0 0x7f6f280d7ca0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6f18006650 tx=0x7f6f18009a80).ready entity=mon.2 client_cookie=2d5de4a2a923f04e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:35.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f1ffff700 1 -- 172.21.15.32:0/3824771266 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f180009b0 con 0x7f6f280d1da0 2021-04-09T18:33:35.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f1ffff700 1 -- 172.21.15.32:0/3824771266 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1800c0c0 con 0x7f6f280d1da0 2021-04-09T18:33:35.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.661 7f6f1ffff700 1 -- 172.21.15.32:0/3824771266 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f1800c510 con 0x7f6f280d1da0 2021-04-09T18:33:35.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.662 7f6f2f380700 1 -- 172.21.15.32:0/3824771266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280d1da0 msgr2=0x7f6f280d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.662 7f6f2f380700 1 --2- 172.21.15.32:0/3824771266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280d1da0 0x7f6f280d7ca0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6f18006650 tx=0x7f6f18009a80).stop 2021-04-09T18:33:35.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.662 7f6f2f380700 1 -- 172.21.15.32:0/3824771266 shutdown_connections 2021-04-09T18:33:35.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.662 7f6f2f380700 1 --2- 172.21.15.32:0/3824771266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280d1da0 0x7f6f280d7ca0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.662 7f6f2f380700 1 --2- 172.21.15.32:0/3824771266 >> v2:172.21.15.86:3300/0 conn(0x7f6f2810d010 0x7f6f280d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.662 7f6f2f380700 1 -- 172.21.15.32:0/3824771266 >> 172.21.15.32:0/3824771266 conn(0x7f6f280ce5f0 msgr2=0x7f6f280d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:35.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.662 7f6f2f380700 1 -- 172.21.15.32:0/3824771266 shutdown_connections 2021-04-09T18:33:35.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.662 7f6f2f380700 1 -- 172.21.15.32:0/3824771266 wait complete. 2021-04-09T18:33:35.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2f380700 1 Processor -- start 2021-04-09T18:33:35.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2f380700 1 -- start start 2021-04-09T18:33:35.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2f380700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280c3c00 0x7f6f2809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2f380700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f28095240 0x7f6f281a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2f380700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f281a1270 0x7f6f280b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2f380700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2809cda0 con 0x7f6f28095240 2021-04-09T18:33:35.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2f380700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f280c3680 con 0x7f6f281a1270 2021-04-09T18:33:35.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2f380700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f28095640 con 0x7f6f280c3c00 2021-04-09T18:33:35.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2c99b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f28095240 0x7f6f281a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2c99b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f28095240 0x7f6f281a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58814/0 (socket says 172.21.15.32:58814) 2021-04-09T18:33:35.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.683 7f6f2c99b700 1 -- 172.21.15.32:0/1457174991 learned_addr learned my addr 172.21.15.32:0/1457174991 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:35.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2d99d700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f281a1270 0x7f6f280b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2d19c700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280c3c00 0x7f6f2809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2c99b700 1 -- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280c3c00 msgr2=0x7f6f2809c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2c99b700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280c3c00 0x7f6f2809c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2c99b700 1 -- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f281a1270 msgr2=0x7f6f280b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2c99b700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f281a1270 0x7f6f280b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2c99b700 1 -- 172.21.15.32:0/1457174991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f281a1670 con 0x7f6f28095240 2021-04-09T18:33:35.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2c99b700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f28095240 0x7f6f281a0c50 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 rx=0x7f6f18009650 tx=0x7f6f1800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:35.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f1dffb700 1 -- 172.21.15.32:0/1457174991 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f180093a0 con 0x7f6f28095240 2021-04-09T18:33:35.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f1dffb700 1 -- 172.21.15.32:0/1457174991 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f18006070 con 0x7f6f28095240 2021-04-09T18:33:35.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f281a1670 con 0x7f6f28095240 2021-04-09T18:33:35.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f1dffb700 1 -- 172.21.15.32:0/1457174991 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f180052e0 con 0x7f6f28095240 2021-04-09T18:33:35.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.684 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f281a1670 con 0x7f6f28095240 2021-04-09T18:33:35.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.685 7f6f1dffb700 1 -- 172.21.15.32:0/1457174991 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6f18009300 con 0x7f6f28095240 2021-04-09T18:33:35.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.685 7f6f1dffb700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f08018930 0x7f6f0801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:35.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.685 7f6f1dffb700 1 -- 172.21.15.32:0/1457174991 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f6f180326e0 con 0x7f6f28095240 2021-04-09T18:33:35.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.685 7f6f2d19c700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f08018930 0x7f6f0801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:35.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.685 7f6f2d19c700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f08018930 0x7f6f0801adc0 crc :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:35.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.704 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f28093d50 con 0x7f6f28095240 2021-04-09T18:33:35.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.708 7f6f1dffb700 1 -- 172.21.15.32:0/1457174991 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6f18006030 con 0x7f6f28095240 2021-04-09T18:33:35.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.887 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f6f2810d480 con 0x7f6f28095240 2021-04-09T18:33:35.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.887 7f6f1dffb700 1 -- 172.21.15.32:0/1457174991 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v243) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6f18000940 con 0x7f6f28095240 2021-04-09T18:33:35.904 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:33:35.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f08018930 msgr2=0x7f6f0801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f08018930 0x7f6f0801adc0 crc :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f28095240 msgr2=0x7f6f281a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:35.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f28095240 0x7f6f281a0c50 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 rx=0x7f6f18009650 tx=0x7f6f1800b8e0).stop 2021-04-09T18:33:35.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 shutdown_connections 2021-04-09T18:33:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6f08018930 0x7f6f0801adc0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f28095240 0x7f6f281a0c50 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f281a1270 0x7f6f280b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 --2- 172.21.15.32:0/1457174991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f280c3c00 0x7f6f2809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:35.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 >> 172.21.15.32:0/1457174991 conn(0x7f6f280975f0 msgr2=0x7f6f28096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 shutdown_connections 2021-04-09T18:33:35.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:35.890 7f6f2f380700 1 -- 172.21.15.32:0/1457174991 wait complete. 2021-04-09T18:33:35.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:33:36.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.049 7fec50edc700 1 Processor -- start 2021-04-09T18:33:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.049 7fec50edc700 1 -- start start 2021-04-09T18:33:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec50edc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec4c10d010 0x7fec4c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec50edc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c0d8490 0x7fec4c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec50edc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c10d480 con 0x7fec4c0d1da0 2021-04-09T18:33:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec50edc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4c0d8d30 con 0x7fec4c10d010 2021-04-09T18:33:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec50edc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c0d9220 con 0x7fec4c0d8490 2021-04-09T18:33:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4a312700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fec4c0d1da0 0x7fec4c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40184/0 (socket says 172.21.15.32:40184) 2021-04-09T18:33:36.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4a312700 1 -- 172.21.15.32:0/3476210855 learned_addr learned my addr 172.21.15.32:0/3476210855 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:36.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4ab13700 1 --2- 172.21.15.32:0/3476210855 >> v2:172.21.15.86:3300/0 conn(0x7fec4c10d010 0x7fec4c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4b314700 1 --2- 172.21.15.32:0/3476210855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c0d8490 0x7fec4c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec49b11700 1 -- 172.21.15.32:0/3476210855 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086134506 0 0) 0x7fec40000ce0 con 0x7fec4c0d1da0 2021-04-09T18:33:36.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec49b11700 1 -- 172.21.15.32:0/3476210855 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec34002fe0 con 0x7fec4c0d1da0 2021-04-09T18:33:36.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4ab13700 1 -- 172.21.15.32:0/3476210855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c0d8490 msgr2=0x7fec4c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:36.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4ab13700 1 --2- 172.21.15.32:0/3476210855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c0d8490 0x7fec4c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4ab13700 1 -- 172.21.15.32:0/3476210855 >> v1:172.21.15.32:6789/0 conn(0x7fec4c0d1da0 legacy=0x7fec4c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4ab13700 1 -- 172.21.15.32:0/3476210855 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c10e020 con 0x7fec4c10d010 2021-04-09T18:33:36.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4b314700 1 --2- 172.21.15.32:0/3476210855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c0d8490 0x7fec4c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:36.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.050 7fec4ab13700 1 --2- 172.21.15.32:0/3476210855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec4c10d010 0x7fec4c0d1780 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7fec3c006650 tx=0x7fec3c009a80).ready entity=mon.1 client_cookie=219630d890698dd5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:36.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.051 7fec49b11700 1 -- 172.21.15.32:0/3476210855 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec3c0009b0 con 0x7fec4c10d010 2021-04-09T18:33:36.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.051 7fec49b11700 1 -- 172.21.15.32:0/3476210855 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3c00c0c0 con 0x7fec4c10d010 2021-04-09T18:33:36.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.051 7fec49b11700 1 -- 172.21.15.32:0/3476210855 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec3c00c510 con 0x7fec4c10d010 2021-04-09T18:33:36.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.051 7fec50edc700 1 -- 172.21.15.32:0/3476210855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec4c10d010 msgr2=0x7fec4c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.051 7fec50edc700 1 --2- 172.21.15.32:0/3476210855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec4c10d010 0x7fec4c0d1780 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7fec3c006650 tx=0x7fec3c009a80).stop 2021-04-09T18:33:36.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.052 7fec50edc700 1 -- 172.21.15.32:0/3476210855 shutdown_connections 2021-04-09T18:33:36.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.052 7fec50edc700 1 --2- 172.21.15.32:0/3476210855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c0d8490 0x7fec4c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.052 7fec50edc700 1 --2- 172.21.15.32:0/3476210855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec4c10d010 0x7fec4c0d1780 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.052 7fec50edc700 1 -- 172.21.15.32:0/3476210855 >> 172.21.15.32:0/3476210855 conn(0x7fec4c0ce5f0 msgr2=0x7fec4c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:36.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.052 7fec50edc700 1 -- 172.21.15.32:0/3476210855 shutdown_connections 2021-04-09T18:33:36.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.052 7fec50edc700 1 -- 172.21.15.32:0/3476210855 wait complete. 2021-04-09T18:33:36.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec50edc700 1 Processor -- start 2021-04-09T18:33:36.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec50edc700 1 -- start start 2021-04-09T18:33:36.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec50edc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c1a5210 0x7fec4c185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec50edc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec4c185a70 0x7fec4c0acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec50edc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c185e70 0x7fec4c0ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec50edc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c18f7b0 con 0x7fec4c1a5210 2021-04-09T18:33:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec50edc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c0ad7f0 con 0x7fec4c185a70 2021-04-09T18:33:36.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec50edc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c0ada10 con 0x7fec4c185e70 2021-04-09T18:33:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec4ab13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c1a5210 0x7fec4c185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec4ab13700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c1a5210 0x7fec4c185450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58828/0 (socket says 172.21.15.32:58828) 2021-04-09T18:33:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.072 7fec4ab13700 1 -- 172.21.15.32:0/3289241491 learned_addr learned my addr 172.21.15.32:0/3289241491 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec4a312700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec4c185a70 0x7fec4c0acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec4b314700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c185e70 0x7fec4c0ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec4ab13700 1 -- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c185e70 msgr2=0x7fec4c0ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec4ab13700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c185e70 0x7fec4c0ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec4ab13700 1 -- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec4c185a70 msgr2=0x7fec4c0acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec4ab13700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec4c185a70 0x7fec4c0acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec4ab13700 1 -- 172.21.15.32:0/3289241491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c179610 con 0x7fec4c1a5210 2021-04-09T18:33:36.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec4ab13700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c1a5210 0x7fec4c185450 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7fec3c009650 tx=0x7fec3c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:36.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec3b7fe700 1 -- 172.21.15.32:0/3289241491 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec3c0093a0 con 0x7fec4c1a5210 2021-04-09T18:33:36.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec3b7fe700 1 -- 172.21.15.32:0/3289241491 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3c006070 con 0x7fec4c1a5210 2021-04-09T18:33:36.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec50edc700 1 -- 172.21.15.32:0/3289241491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4c179610 con 0x7fec4c1a5210 2021-04-09T18:33:36.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec3b7fe700 1 -- 172.21.15.32:0/3289241491 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec3c0052e0 con 0x7fec4c1a5210 2021-04-09T18:33:36.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.073 7fec50edc700 1 -- 172.21.15.32:0/3289241491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4c179610 con 0x7fec4c1a5210 2021-04-09T18:33:36.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.074 7fec3b7fe700 1 -- 172.21.15.32:0/3289241491 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fec3c0052e0 con 0x7fec4c1a5210 2021-04-09T18:33:36.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.074 7fec3b7fe700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec30018930 0x7fec3001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.074 7fec3b7fe700 1 -- 172.21.15.32:0/3289241491 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fec3c0326e0 con 0x7fec4c1a5210 2021-04-09T18:33:36.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.074 7fec4a312700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec30018930 0x7fec3001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.074 7fec4a312700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec30018930 0x7fec3001ae10 crc :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:36.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.094 7fec50edc700 1 -- 172.21.15.32:0/3289241491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4c18f7b0 con 0x7fec4c1a5210 2021-04-09T18:33:36.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.097 7fec3b7fe700 1 -- 172.21.15.32:0/3289241491 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fec3c006030 con 0x7fec4c1a5210 2021-04-09T18:33:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.276 7fec50edc700 1 -- 172.21.15.32:0/3289241491 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fec4c18f7b0 con 0x7fec4c1a5210 2021-04-09T18:33:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.276 7fec3b7fe700 1 -- 172.21.15.32:0/3289241491 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v243) v1 ==== 84+0+64 (secure 0 0 0) 0x7fec3c000940 con 0x7fec4c1a5210 2021-04-09T18:33:36.293 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:36.293 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:33:36.294 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:33:36.294 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:33:36.294 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:33:36.294 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:36.295 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:33:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.278 7fec50edc700 1 -- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec30018930 msgr2=0x7fec3001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.278 7fec50edc700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec30018930 0x7fec3001ae10 crc :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.278 7fec50edc700 1 -- 172.21.15.32:0/3289241491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c1a5210 msgr2=0x7fec4c185450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.278 7fec50edc700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c1a5210 0x7fec4c185450 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7fec3c009650 tx=0x7fec3c00b8e0).stop 2021-04-09T18:33:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.279 7fec50edc700 1 -- 172.21.15.32:0/3289241491 shutdown_connections 2021-04-09T18:33:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.279 7fec50edc700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fec30018930 0x7fec3001ae10 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.279 7fec50edc700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec4c185e70 0x7fec4c0ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.279 7fec50edc700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec4c185a70 0x7fec4c0acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.279 7fec50edc700 1 --2- 172.21.15.32:0/3289241491 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec4c1a5210 0x7fec4c185450 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.279 7fec50edc700 1 -- 172.21.15.32:0/3289241491 >> 172.21.15.32:0/3289241491 conn(0x7fec4c1a99c0 msgr2=0x7fec4c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.279 7fec50edc700 1 -- 172.21.15.32:0/3289241491 shutdown_connections 2021-04-09T18:33:36.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.279 7fec50edc700 1 -- 172.21.15.32:0/3289241491 wait complete. 2021-04-09T18:33:36.305 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:33:36.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:33:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.442 7fdfa0ce0700 1 Processor -- start 2021-04-09T18:33:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.442 7fdfa0ce0700 1 -- start start 2021-04-09T18:33:36.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.443 7fdfa0ce0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c10da00 0x7fdf9c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.443 7fdfa0ce0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9c10d480 con 0x7fdf9c10d010 2021-04-09T18:33:36.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.443 7fdfa0ce0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9c0d1f70 con 0x7fdf9c0d7ca0 2021-04-09T18:33:36.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.443 7fdfa0ce0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0d9220 con 0x7fdf9c10da00 2021-04-09T18:33:36.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.443 7fdf9aae8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdf9c10d010 0x7fdf9c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40194/0 (socket says 172.21.15.32:40194) 2021-04-09T18:33:36.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.443 7fdf9aae8700 1 -- 172.21.15.32:0/2792344527 learned_addr learned my addr 172.21.15.32:0/2792344527 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:36.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.443 7fdf9b2e9700 1 --2- 172.21.15.32:0/2792344527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c10da00 0x7fdf9c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf99ae6700 1 -- 172.21.15.32:0/2792344527 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617729574 0 0) 0x7fdf94001060 con 0x7fdf9c10d010 2021-04-09T18:33:36.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf99ae6700 1 -- 172.21.15.32:0/2792344527 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf88002fe0 con 0x7fdf9c10d010 2021-04-09T18:33:36.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf99ae6700 1 -- 172.21.15.32:0/2792344527 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684250920 0 0) 0x7fdf84000c60 con 0x7fdf9c0d7ca0 2021-04-09T18:33:36.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf99ae6700 1 -- 172.21.15.32:0/2792344527 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf880042c0 con 0x7fdf9c0d7ca0 2021-04-09T18:33:36.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf9b2e9700 1 -- 172.21.15.32:0/2792344527 >> v1:172.21.15.86:6789/0 conn(0x7fdf9c0d7ca0 legacy=0x7fdf9c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf9b2e9700 1 -- 172.21.15.32:0/2792344527 >> v1:172.21.15.32:6789/0 conn(0x7fdf9c10d010 legacy=0x7fdf9c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf9b2e9700 1 -- 172.21.15.32:0/2792344527 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c10e420 con 0x7fdf9c10da00 2021-04-09T18:33:36.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf9b2e9700 1 --2- 172.21.15.32:0/2792344527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c10da00 0x7fdf9c10de00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fdf8c006ad0 tx=0x7fdf8c009ad0).ready entity=mon.2 client_cookie=730ebe734db9e627 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:36.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf99ae6700 1 -- 172.21.15.32:0/2792344527 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf8c0022e0 con 0x7fdf9c10da00 2021-04-09T18:33:36.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf99ae6700 1 -- 172.21.15.32:0/2792344527 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c006410 con 0x7fdf9c10da00 2021-04-09T18:33:36.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdf99ae6700 1 -- 172.21.15.32:0/2792344527 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf8c002620 con 0x7fdf9c10da00 2021-04-09T18:33:36.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdfa0ce0700 1 -- 172.21.15.32:0/2792344527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c10da00 msgr2=0x7fdf9c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.444 7fdfa0ce0700 1 --2- 172.21.15.32:0/2792344527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c10da00 0x7fdf9c10de00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fdf8c006ad0 tx=0x7fdf8c009ad0).stop 2021-04-09T18:33:36.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.445 7fdfa0ce0700 1 -- 172.21.15.32:0/2792344527 shutdown_connections 2021-04-09T18:33:36.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.445 7fdfa0ce0700 1 --2- 172.21.15.32:0/2792344527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c10da00 0x7fdf9c10de00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.445 7fdfa0ce0700 1 -- 172.21.15.32:0/2792344527 >> 172.21.15.32:0/2792344527 conn(0x7fdf9c0ce5f0 msgr2=0x7fdf9c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:36.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.445 7fdfa0ce0700 1 -- 172.21.15.32:0/2792344527 shutdown_connections 2021-04-09T18:33:36.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.445 7fdfa0ce0700 1 -- 172.21.15.32:0/2792344527 wait complete. 2021-04-09T18:33:36.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.465 7fdfa0ce0700 1 Processor -- start 2021-04-09T18:33:36.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.465 7fdfa0ce0700 1 -- start start 2021-04-09T18:33:36.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.465 7fdfa0ce0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c193c00 0x7fdf9c179610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdfa0ce0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf9c179c30 0x7fdf9c1b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdfa0ce0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf9c1b8b60 0x7fdf9c17a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdfa0ce0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c17a060 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdfa0ce0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c0c0de0 con 0x7fdf9c179c30 2021-04-09T18:33:36.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdfa0ce0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c194000 con 0x7fdf9c193c00 2021-04-09T18:33:36.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9b2e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf9c1b8b60 0x7fdf9c17a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9b2e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf9c1b8b60 0x7fdf9c17a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58842/0 (socket says 172.21.15.32:58842) 2021-04-09T18:33:36.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9b2e9700 1 -- 172.21.15.32:0/1962128978 learned_addr learned my addr 172.21.15.32:0/1962128978 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:36.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9aae8700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c193c00 0x7fdf9c179610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9a2e7700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf9c179c30 0x7fdf9c1b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9b2e9700 1 -- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c193c00 msgr2=0x7fdf9c179610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9b2e9700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c193c00 0x7fdf9c179610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9b2e9700 1 -- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf9c179c30 msgr2=0x7fdf9c1b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9b2e9700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf9c179c30 0x7fdf9c1b8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:36.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9b2e9700 1 -- 172.21.15.32:0/1962128978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c194220 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf9b2e9700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf9c1b8b60 0x7fdf9c17a5e0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 rx=0x7fdf8c00b610 tx=0x7fdf8c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:36.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf937fe700 1 -- 172.21.15.32:0/1962128978 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf8c002030 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf937fe700 1 -- 172.21.15.32:0/1962128978 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c002330 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdf937fe700 1 -- 172.21.15.32:0/1962128978 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf8c002780 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9c194220 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.466 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9c194220 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.467 7fdf937fe700 1 -- 172.21.15.32:0/1962128978 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fdf8c000a70 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.467 7fdf937fe700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf80018930 0x7fdf8001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:36.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.467 7fdf937fe700 1 -- 172.21.15.32:0/1962128978 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fdf8c032730 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.467 7fdf9aae8700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf80018930 0x7fdf8001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:36.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.467 7fdf9aae8700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf80018930 0x7fdf8001adc0 crc :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.487 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9c10d480 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.491 7fdf937fe700 1 -- 172.21.15.32:0/1962128978 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fdf8c01f7a0 con 0x7fdf9c1b8b60 2021-04-09T18:33:36.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:36.673 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fdf9c10d480 con 0x7fdf9c1b8b60 2021-04-09T18:33:37.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.656 7fdf937fe700 1 -- 172.21.15.32:0/1962128978 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v244) v1 ==== 133+0+0 (secure 0 0 0) 0x7fdf8c000a70 con 0x7fdf9c1b8b60 2021-04-09T18:33:37.705 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:33:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf80018930 msgr2=0x7fdf8001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:37.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf80018930 0x7fdf8001adc0 crc :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf9c1b8b60 msgr2=0x7fdf9c17a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf9c1b8b60 0x7fdf9c17a5e0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 rx=0x7fdf8c00b610 tx=0x7fdf8c00b9b0).stop 2021-04-09T18:33:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 shutdown_connections 2021-04-09T18:33:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdf80018930 0x7fdf8001adc0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:37.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf9c1b8b60 0x7fdf9c17a5e0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:37.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf9c179c30 0x7fdf9c1b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:37.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 --2- 172.21.15.32:0/1962128978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf9c193c00 0x7fdf9c179610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:37.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 >> 172.21.15.32:0/1962128978 conn(0x7fdf9c0bbda0 msgr2=0x7fdf9c1b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:37.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 shutdown_connections 2021-04-09T18:33:37.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.692 7fdfa0ce0700 1 -- 172.21.15.32:0/1962128978 wait complete. 2021-04-09T18:33:37.720 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:33:37.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.861 7f5483079700 1 Processor -- start 2021-04-09T18:33:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.861 7f5483079700 1 -- start start 2021-04-09T18:33:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.862 7f5483079700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f547c10d010 0x7f547c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:37.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.862 7f5483079700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547c10d480 con 0x7f547c0d8490 2021-04-09T18:33:37.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.862 7f5483079700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547c10e1f0 con 0x7f547c0d1da0 2021-04-09T18:33:37.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.862 7f5483079700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f547c0d8d30 con 0x7f547c10d010 2021-04-09T18:33:37.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.862 7f5481696700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f547c0d8490 0x7f547c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40210/0 (socket says 172.21.15.32:40210) 2021-04-09T18:33:37.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.862 7f5480e95700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f547c10d010 0x7f547c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:37.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.862 7f5481696700 1 -- 172.21.15.32:0/476149611 learned_addr learned my addr 172.21.15.32:0/476149611 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:37.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.862 7f547b7fe700 1 -- 172.21.15.32:0/476149611 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219669157 0 0) 0x7f5474001100 con 0x7f547c0d8490 2021-04-09T18:33:37.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.862 7f547b7fe700 1 -- 172.21.15.32:0/476149611 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5468002fe0 con 0x7f547c0d8490 2021-04-09T18:33:37.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f547b7fe700 1 -- 172.21.15.32:0/476149611 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251866897 0 0) 0x7f5464000c60 con 0x7f547c0d1da0 2021-04-09T18:33:37.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f547b7fe700 1 -- 172.21.15.32:0/476149611 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5468003210 con 0x7f547c0d1da0 2021-04-09T18:33:37.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f5480e95700 1 -- 172.21.15.32:0/476149611 >> v1:172.21.15.86:6789/0 conn(0x7f547c0d1da0 legacy=0x7f547c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:37.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f5480e95700 1 -- 172.21.15.32:0/476149611 >> v1:172.21.15.32:6789/0 conn(0x7f547c0d8490 legacy=0x7f547c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:37.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f5480e95700 1 -- 172.21.15.32:0/476149611 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c0d9220 con 0x7f547c10d010 2021-04-09T18:33:37.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f5480e95700 1 --2- 172.21.15.32:0/476149611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c10d010 0x7f547c0d1780 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 rx=0x7f546c0069d0 tx=0x7f546c009e40).ready entity=mon.0 client_cookie=cab13410c4e0deab server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:37.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f547b7fe700 1 -- 172.21.15.32:0/476149611 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546c000d30 con 0x7f547c10d010 2021-04-09T18:33:37.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f547b7fe700 1 -- 172.21.15.32:0/476149611 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c0066c0 con 0x7f547c10d010 2021-04-09T18:33:37.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f547b7fe700 1 -- 172.21.15.32:0/476149611 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546c00b700 con 0x7f547c10d010 2021-04-09T18:33:37.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f5483079700 1 -- 172.21.15.32:0/476149611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c10d010 msgr2=0x7f547c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:37.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.863 7f5483079700 1 --2- 172.21.15.32:0/476149611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c10d010 0x7f547c0d1780 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 rx=0x7f546c0069d0 tx=0x7f546c009e40).stop 2021-04-09T18:33:37.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.864 7f5483079700 1 -- 172.21.15.32:0/476149611 shutdown_connections 2021-04-09T18:33:37.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.864 7f5483079700 1 --2- 172.21.15.32:0/476149611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c10d010 0x7f547c0d1780 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:37.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.864 7f5483079700 1 -- 172.21.15.32:0/476149611 >> 172.21.15.32:0/476149611 conn(0x7f547c0ce5f0 msgr2=0x7f547c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:37.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.864 7f5483079700 1 -- 172.21.15.32:0/476149611 shutdown_connections 2021-04-09T18:33:37.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.864 7f5483079700 1 -- 172.21.15.32:0/476149611 wait complete. 2021-04-09T18:33:37.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.884 7f5483079700 1 Processor -- start 2021-04-09T18:33:37.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.884 7f5483079700 1 -- start start 2021-04-09T18:33:37.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.884 7f5483079700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c191ee0 0x7f547c0a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:37.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5483079700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f547c0a6c30 0x7f547c0b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:37.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5483079700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f547c0b4330 0x7f547c1b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:37.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5483079700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0b3790 con 0x7f547c191ee0 2021-04-09T18:33:37.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5483079700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0a7030 con 0x7f547c0a6c30 2021-04-09T18:33:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5483079700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0b4730 con 0x7f547c0b4330 2021-04-09T18:33:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5480e95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c191ee0 0x7f547c0a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:37.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5480e95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c191ee0 0x7f547c0a6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58854/0 (socket says 172.21.15.32:58854) 2021-04-09T18:33:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5480e95700 1 -- 172.21.15.32:0/1501225954 learned_addr learned my addr 172.21.15.32:0/1501225954 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:37.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5481696700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f547c0b4330 0x7f547c1b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:37.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f547bfff700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f547c0a6c30 0x7f547c0b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:37.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5480e95700 1 -- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f547c0b4330 msgr2=0x7f547c1b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5480e95700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f547c0b4330 0x7f547c1b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:37.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5480e95700 1 -- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f547c0a6c30 msgr2=0x7f547c0b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:37.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5480e95700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f547c0a6c30 0x7f547c0b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:37.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5480e95700 1 -- 172.21.15.32:0/1501225954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c1b8580 con 0x7f547c191ee0 2021-04-09T18:33:37.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5480e95700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c191ee0 0x7f547c0a6610 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 rx=0x7f546c009f40 tx=0x7f546c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f54797fa700 1 -- 172.21.15.32:0/1501225954 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546c006240 con 0x7f547c191ee0 2021-04-09T18:33:37.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f54797fa700 1 -- 172.21.15.32:0/1501225954 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c006e20 con 0x7f547c191ee0 2021-04-09T18:33:37.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5483079700 1 -- 172.21.15.32:0/1501225954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547c1b8580 con 0x7f547c191ee0 2021-04-09T18:33:37.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f54797fa700 1 -- 172.21.15.32:0/1501225954 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f546c005660 con 0x7f547c191ee0 2021-04-09T18:33:37.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.885 7f5483079700 1 -- 172.21.15.32:0/1501225954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547c1b8580 con 0x7f547c191ee0 2021-04-09T18:33:37.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.886 7f54797fa700 1 -- 172.21.15.32:0/1501225954 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f546c006500 con 0x7f547c191ee0 2021-04-09T18:33:37.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.886 7f54797fa700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54600188d0 0x7f546001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:37.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.886 7f54797fa700 1 -- 172.21.15.32:0/1501225954 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 5111+0+0 (secure 0 0 0) 0x7f546c032d10 con 0x7f547c191ee0 2021-04-09T18:33:37.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.886 7f547bfff700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54600188d0 0x7f546001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:37.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.887 7f547bfff700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54600188d0 0x7f546001add0 crc :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:37.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.906 7f5483079700 1 -- 172.21.15.32:0/1501225954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f547c10d480 con 0x7f547c191ee0 2021-04-09T18:33:37.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:37.910 7f54797fa700 1 -- 172.21.15.32:0/1501225954 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f546c006030 con 0x7f547c191ee0 2021-04-09T18:33:38.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.095 7f5483079700 1 -- 172.21.15.32:0/1501225954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f547c10d480 con 0x7f547c191ee0 2021-04-09T18:33:38.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:33:38.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:33:38.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.946 7f54797fa700 1 -- 172.21.15.32:0/1501225954 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v249) v1 ==== 125+0+0 (secure 0 0 0) 0x7f546c006500 con 0x7f547c191ee0 2021-04-09T18:33:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:33:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 -- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54600188d0 msgr2=0x7f546001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54600188d0 0x7f546001add0 crc :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 -- 172.21.15.32:0/1501225954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c191ee0 msgr2=0x7f547c0a6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c191ee0 0x7f547c0a6610 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 rx=0x7f546c009f40 tx=0x7f546c00bd40).stop 2021-04-09T18:33:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 -- 172.21.15.32:0/1501225954 shutdown_connections 2021-04-09T18:33:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f54600188d0 0x7f546001add0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f547c0b4330 0x7f547c1b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f547c0a6c30 0x7f547c0b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 --2- 172.21.15.32:0/1501225954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f547c191ee0 0x7f547c0a6610 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 -- 172.21.15.32:0/1501225954 >> 172.21.15.32:0/1501225954 conn(0x7f547c0bbda0 msgr2=0x7f547c1b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 -- 172.21.15.32:0/1501225954 shutdown_connections 2021-04-09T18:33:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:38.962 7f5483079700 1 -- 172.21.15.32:0/1501225954 wait complete. 2021-04-09T18:33:38.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:33:39.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.136 7f1df0cad700 1 Processor -- start 2021-04-09T18:33:39.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.136 7f1df0cad700 1 -- start start 2021-04-09T18:33:39.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1df0cad700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1dec0d7cc0 0x7f1dec0d80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1df0cad700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec10d500 con 0x7f1dec10da80 2021-04-09T18:33:39.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1df0cad700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec0d0330 con 0x7f1dec10d050 2021-04-09T18:33:39.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1df0cad700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dec0d9260 con 0x7f1dec0d7cc0 2021-04-09T18:33:39.176 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1dea05d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1dec0d7cc0 0x7f1dec0d80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1deb05f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1dec10da80 0x7f1dec10de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40226/0 (socket says 172.21.15.32:40226) 2021-04-09T18:33:39.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1deb05f700 1 -- 172.21.15.32:0/766819672 learned_addr learned my addr 172.21.15.32:0/766819672 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:39.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1dea05d700 1 -- 172.21.15.32:0/766819672 >> v1:172.21.15.86:6789/0 conn(0x7f1dec10d050 legacy=0x7f1dec0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1dea05d700 1 -- 172.21.15.32:0/766819672 >> v1:172.21.15.32:6789/0 conn(0x7f1dec10da80 legacy=0x7f1dec10de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1dea05d700 1 -- 172.21.15.32:0/766819672 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec112140 con 0x7f1dec0d7cc0 2021-04-09T18:33:39.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1dea05d700 1 --2- 172.21.15.32:0/766819672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0d7cc0 0x7f1dec0d80c0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 rx=0x7f1de0006650 tx=0x7f1de0009a80).ready entity=mon.0 client_cookie=8433d9b53f66e078 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:39.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1de985c700 1 -- 172.21.15.32:0/766819672 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1de00009b0 con 0x7f1dec0d7cc0 2021-04-09T18:33:39.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1de985c700 1 -- 172.21.15.32:0/766819672 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de000c0c0 con 0x7f1dec0d7cc0 2021-04-09T18:33:39.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1de985c700 1 -- 172.21.15.32:0/766819672 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1de000c510 con 0x7f1dec0d7cc0 2021-04-09T18:33:39.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.137 7f1df0cad700 1 -- 172.21.15.32:0/766819672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0d7cc0 msgr2=0x7f1dec0d80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.138 7f1df0cad700 1 --2- 172.21.15.32:0/766819672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0d7cc0 0x7f1dec0d80c0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 rx=0x7f1de0006650 tx=0x7f1de0009a80).stop 2021-04-09T18:33:39.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.138 7f1df0cad700 1 -- 172.21.15.32:0/766819672 shutdown_connections 2021-04-09T18:33:39.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.138 7f1df0cad700 1 --2- 172.21.15.32:0/766819672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0d7cc0 0x7f1dec0d80c0 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.138 7f1df0cad700 1 -- 172.21.15.32:0/766819672 >> 172.21.15.32:0/766819672 conn(0x7f1dec0ce620 msgr2=0x7f1dec0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.138 7f1df0cad700 1 -- 172.21.15.32:0/766819672 shutdown_connections 2021-04-09T18:33:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.138 7f1df0cad700 1 -- 172.21.15.32:0/766819672 wait complete. 2021-04-09T18:33:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.158 7f1df0cad700 1 Processor -- start 2021-04-09T18:33:39.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.158 7f1df0cad700 1 -- start start 2021-04-09T18:33:39.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1df0cad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dec0adcb0 0x7f1dec0c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1df0cad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dec0c36d0 0x7f1dec1c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1df0cad700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0c3ad0 0x7f1dec1c22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1df0cad700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0ae280 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1df0cad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec1c2900 con 0x7f1dec0adcb0 2021-04-09T18:33:39.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1df0cad700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec1c2b20 con 0x7f1dec0c36d0 2021-04-09T18:33:39.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1deb05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0c3ad0 0x7f1dec1c22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1deb05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0c3ad0 0x7f1dec1c22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58876/0 (socket says 172.21.15.32:58876) 2021-04-09T18:33:39.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1deb05f700 1 -- 172.21.15.32:0/2276834618 learned_addr learned my addr 172.21.15.32:0/2276834618 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:39.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1dea85e700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dec0adcb0 0x7f1dec0c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1dea05d700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dec0c36d0 0x7f1dec1c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1deb05f700 1 -- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dec0c36d0 msgr2=0x7f1dec1c1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1deb05f700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dec0c36d0 0x7f1dec1c1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1deb05f700 1 -- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dec0adcb0 msgr2=0x7f1dec0c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1deb05f700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dec0adcb0 0x7f1dec0c30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1deb05f700 1 -- 172.21.15.32:0/2276834618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec1a8950 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.159 7f1dea05d700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dec0c36d0 0x7f1dec1c1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:39.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.160 7f1deb05f700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0c3ad0 0x7f1dec1c22e0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7f1de400a900 tx=0x7f1de400aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:39.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.160 7f1ddb7fe700 1 -- 172.21.15.32:0/2276834618 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1de40070d0 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.160 7f1ddb7fe700 1 -- 172.21.15.32:0/2276834618 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de4000f90 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.160 7f1ddb7fe700 1 -- 172.21.15.32:0/2276834618 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1de4004720 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.160 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec1a8950 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.160 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec1a8950 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.160 7f1ddb7fe700 1 -- 172.21.15.32:0/2276834618 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1de4007390 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.161 7f1ddb7fe700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1dd00188d0 0x7f1dd001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.161 7f1ddb7fe700 1 -- 172.21.15.32:0/2276834618 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f1de4032d10 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.161 7f1dea85e700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1dd00188d0 0x7f1dd001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.161 7f1dea85e700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1dd00188d0 0x7f1dd001add0 crc :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:39.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.181 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dec1120b0 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.184 7f1ddb7fe700 1 -- 172.21.15.32:0/2276834618 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1de4007cb0 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.366 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f1dec1120b0 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.370 7f1ddb7fe700 1 -- 172.21.15.32:0/2276834618 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 42 and data pool 43 v408) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1de4007350 con 0x7f1dec0c3ad0 2021-04-09T18:33:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 42 and data pool 43 2021-04-09T18:33:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.376 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1dd00188d0 msgr2=0x7f1dd001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.376 7f1df0cad700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1dd00188d0 0x7f1dd001add0 crc :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.376 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0c3ad0 msgr2=0x7f1dec1c22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.376 7f1df0cad700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0c3ad0 0x7f1dec1c22e0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7f1de400a900 tx=0x7f1de400aca0).stop 2021-04-09T18:33:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.377 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 shutdown_connections 2021-04-09T18:33:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.377 7f1df0cad700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1dd00188d0 0x7f1dd001add0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.377 7f1df0cad700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1dec0c3ad0 0x7f1dec1c22e0 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.377 7f1df0cad700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dec0c36d0 0x7f1dec1c1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.377 7f1df0cad700 1 --2- 172.21.15.32:0/2276834618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dec0adcb0 0x7f1dec0c30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.377 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 >> 172.21.15.32:0/2276834618 conn(0x7f1dec0a7cb0 msgr2=0x7f1dec0bf930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:39.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.377 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 shutdown_connections 2021-04-09T18:33:39.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.377 7f1df0cad700 1 -- 172.21.15.32:0/2276834618 wait complete. 2021-04-09T18:33:39.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:33:39.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.535 7f4d0ae4f700 1 Processor -- start 2021-04-09T18:33:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.535 7f4d0ae4f700 1 -- start start 2021-04-09T18:33:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d0ae4f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4d0410d010 0x7f4d040d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d0ae4f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d040d1da0 0x7f4d040d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d0ae4f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0410d480 con 0x7f4d040d82c0 2021-04-09T18:33:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d0ae4f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d040d9220 con 0x7f4d0410d010 2021-04-09T18:33:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d0ae4f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d04112100 con 0x7f4d040d1da0 2021-04-09T18:33:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d03fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d040d1da0 0x7f4d040d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d08c6b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4d0410d010 0x7f4d040d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d03fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d040d1da0 0x7f4d040d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:37876/0 (socket says 172.21.15.32:37876) 2021-04-09T18:33:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d08c6b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4d0410d010 0x7f4d040d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:50638/0 (socket says 172.21.15.32:50638) 2021-04-09T18:33:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d03fff700 1 -- 172.21.15.32:0/2450437277 learned_addr learned my addr 172.21.15.32:0/2450437277 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.536 7f4d0946c700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4d040d82c0 0x7f4d0410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38168/0 (socket says 172.21.15.32:38168) 2021-04-09T18:33:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d03fff700 1 -- 172.21.15.32:0/2450437277 >> v2:172.21.15.86:3300/0 conn(0x7f4d0410d010 msgr2=0x7f4d040d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d03fff700 1 --2- 172.21.15.32:0/2450437277 >> v2:172.21.15.86:3300/0 conn(0x7f4d0410d010 0x7f4d040d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d03fff700 1 -- 172.21.15.32:0/2450437277 >> v1:172.21.15.86:6789/0 conn(0x7f4d040d82c0 legacy=0x7f4d0410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d03fff700 1 -- 172.21.15.32:0/2450437277 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d040d09c0 con 0x7f4d040d1da0 2021-04-09T18:33:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d037fe700 1 -- 172.21.15.32:0/2450437277 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758468675 0 0) 0x7f4cfc000d30 con 0x7f4d040d82c0 2021-04-09T18:33:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d08c6b700 1 --2- 172.21.15.32:0/2450437277 >> v2:172.21.15.86:3300/0 conn(0x7f4d0410d010 0x7f4d040d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d03fff700 1 --2- 172.21.15.32:0/2450437277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d040d1da0 0x7f4d040d7ca0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f4cf8006700 tx=0x7f4cf8009b80).ready entity=mon.2 client_cookie=532b336125c8d8f2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d037fe700 1 -- 172.21.15.32:0/2450437277 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cf800c0c0 con 0x7f4d040d1da0 2021-04-09T18:33:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d037fe700 1 -- 172.21.15.32:0/2450437277 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf800c440 con 0x7f4d040d1da0 2021-04-09T18:33:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d037fe700 1 -- 172.21.15.32:0/2450437277 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cf800c920 con 0x7f4d040d1da0 2021-04-09T18:33:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d0ae4f700 1 -- 172.21.15.32:0/2450437277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d040d1da0 msgr2=0x7f4d040d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.537 7f4d0ae4f700 1 --2- 172.21.15.32:0/2450437277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d040d1da0 0x7f4d040d7ca0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f4cf8006700 tx=0x7f4cf8009b80).stop 2021-04-09T18:33:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.538 7f4d0ae4f700 1 -- 172.21.15.32:0/2450437277 shutdown_connections 2021-04-09T18:33:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.538 7f4d0ae4f700 1 --2- 172.21.15.32:0/2450437277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d040d1da0 0x7f4d040d7ca0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.538 7f4d0ae4f700 1 --2- 172.21.15.32:0/2450437277 >> v2:172.21.15.86:3300/0 conn(0x7f4d0410d010 0x7f4d040d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.538 7f4d0ae4f700 1 -- 172.21.15.32:0/2450437277 >> 172.21.15.32:0/2450437277 conn(0x7f4d040ce5f0 msgr2=0x7f4d040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.538 7f4d0ae4f700 1 -- 172.21.15.32:0/2450437277 shutdown_connections 2021-04-09T18:33:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.538 7f4d0ae4f700 1 -- 172.21.15.32:0/2450437277 wait complete. 2021-04-09T18:33:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.558 7f4d0ae4f700 1 Processor -- start 2021-04-09T18:33:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.558 7f4d0ae4f700 1 -- start start 2021-04-09T18:33:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0ae4f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d040b42d0 0x7f4d04183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0ae4f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d041ac090 0x7f4d041ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0ae4f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d04184d30 0x7f4d04185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0ae4f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d040a1a90 con 0x7f4d04184d30 2021-04-09T18:33:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0ae4f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d041acab0 con 0x7f4d040b42d0 2021-04-09T18:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0ae4f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d04185a00 con 0x7f4d041ac090 2021-04-09T18:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0946c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d04184d30 0x7f4d04185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0946c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d04184d30 0x7f4d04185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58890/0 (socket says 172.21.15.32:58890) 2021-04-09T18:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0946c700 1 -- 172.21.15.32:0/1969387747 learned_addr learned my addr 172.21.15.32:0/1969387747 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d08c6b700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d040b42d0 0x7f4d04183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d03fff700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d041ac090 0x7f4d041ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d08c6b700 1 -- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d041ac090 msgr2=0x7f4d041ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d08c6b700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d041ac090 0x7f4d041ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d08c6b700 1 -- 172.21.15.32:0/1969387747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d04184d30 msgr2=0x7f4d04185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d08c6b700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d04184d30 0x7f4d04185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d08c6b700 1 -- 172.21.15.32:0/1969387747 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d04185750 con 0x7f4d040b42d0 2021-04-09T18:33:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.559 7f4d0946c700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d04184d30 0x7f4d04185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.560 7f4d08c6b700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d040b42d0 0x7f4d04183520 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f4cf4009540 tx=0x7f4cf400b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:39.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.560 7f4d017fa700 1 -- 172.21.15.32:0/1969387747 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cf400bcd0 con 0x7f4d040b42d0 2021-04-09T18:33:39.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.560 7f4d017fa700 1 -- 172.21.15.32:0/1969387747 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf40063e0 con 0x7f4d040b42d0 2021-04-09T18:33:39.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.560 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d04185750 con 0x7f4d040b42d0 2021-04-09T18:33:39.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.560 7f4d017fa700 1 -- 172.21.15.32:0/1969387747 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cf4002030 con 0x7f4d040b42d0 2021-04-09T18:33:39.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.560 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d04185750 con 0x7f4d040b42d0 2021-04-09T18:33:39.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.560 7f4d017fa700 1 -- 172.21.15.32:0/1969387747 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4cf4000c90 con 0x7f4d040b42d0 2021-04-09T18:33:39.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.561 7f4d017fa700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4ce4018930 0x7f4ce401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.561 7f4d017fa700 1 -- 172.21.15.32:0/1969387747 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4cf4032910 con 0x7f4d040b42d0 2021-04-09T18:33:39.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.561 7f4d03fff700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4ce4018930 0x7f4ce401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.561 7f4d03fff700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4ce4018930 0x7f4ce401ae10 crc :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:39.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.580 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d04182d10 con 0x7f4d040b42d0 2021-04-09T18:33:39.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.584 7f4d017fa700 1 -- 172.21.15.32:0/1969387747 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4cf400bcd0 con 0x7f4d040b42d0 2021-04-09T18:33:39.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.774 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d0410d480 con 0x7f4d040b42d0 2021-04-09T18:33:39.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.775 7f4d017fa700 1 -- 172.21.15.32:0/1969387747 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f4cf4000c90 con 0x7f4d040b42d0 2021-04-09T18:33:39.792 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:39.792 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":250,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:33:39.367763","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:33:36.709256","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-04-09 18:33:38.131023","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:33:39.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4ce4018930 msgr2=0x7f4ce401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4ce4018930 0x7f4ce401ae10 crc :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d040b42d0 msgr2=0x7f4d04183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d040b42d0 0x7f4d04183520 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f4cf4009540 tx=0x7f4cf400b8e0).stop 2021-04-09T18:33:39.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 shutdown_connections 2021-04-09T18:33:39.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4ce4018930 0x7f4ce401ae10 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d04184d30 0x7f4d04185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d041ac090 0x7f4d041ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 --2- 172.21.15.32:0/1969387747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d040b42d0 0x7f4d04183520 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 >> 172.21.15.32:0/1969387747 conn(0x7f4d0419b890 msgr2=0x7f4d041a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:39.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 shutdown_connections 2021-04-09T18:33:39.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.777 7f4d0ae4f700 1 -- 172.21.15.32:0/1969387747 wait complete. 2021-04-09T18:33:39.804 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:33:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.936 7fee51993700 1 Processor -- start 2021-04-09T18:33:39.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.936 7fee51993700 1 -- start start 2021-04-09T18:33:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee51993700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee4c10d010 0x7fee4c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee51993700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c0d8490 0x7fee4c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee51993700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c10d480 con 0x7fee4c0d1da0 2021-04-09T18:33:39.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee51993700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c0d8d30 con 0x7fee4c10d010 2021-04-09T18:33:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee51993700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0d9220 con 0x7fee4c0d8490 2021-04-09T18:33:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee4bd75700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c0d8490 0x7fee4c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee4b574700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee4c10d010 0x7fee4c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee43fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fee4c0d1da0 0x7fee4c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38178/0 (socket says 172.21.15.32:38178) 2021-04-09T18:33:39.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee43fff700 1 -- 172.21.15.32:0/1390953568 learned_addr learned my addr 172.21.15.32:0/1390953568 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee4ad73700 1 -- 172.21.15.32:0/1390953568 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631512099 0 0) 0x7fee340010c0 con 0x7fee4c0d1da0 2021-04-09T18:33:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee4ad73700 1 -- 172.21.15.32:0/1390953568 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee38002fe0 con 0x7fee4c0d1da0 2021-04-09T18:33:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee4bd75700 1 -- 172.21.15.32:0/1390953568 >> v2:172.21.15.86:3300/0 conn(0x7fee4c10d010 msgr2=0x7fee4c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:39.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.937 7fee4bd75700 1 --2- 172.21.15.32:0/1390953568 >> v2:172.21.15.86:3300/0 conn(0x7fee4c10d010 0x7fee4c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.938 7fee4bd75700 1 -- 172.21.15.32:0/1390953568 >> v1:172.21.15.86:6789/0 conn(0x7fee4c0d1da0 legacy=0x7fee4c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.938 7fee4bd75700 1 -- 172.21.15.32:0/1390953568 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0d09c0 con 0x7fee4c0d8490 2021-04-09T18:33:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.938 7fee4b574700 1 --2- 172.21.15.32:0/1390953568 >> v2:172.21.15.86:3300/0 conn(0x7fee4c10d010 0x7fee4c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.938 7fee4bd75700 1 --2- 172.21.15.32:0/1390953568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c0d8490 0x7fee4c10da00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fee44006e50 tx=0x7fee44009e70).ready entity=mon.2 client_cookie=1f8710563a675e7f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:39.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.938 7fee4ad73700 1 -- 172.21.15.32:0/1390953568 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee44006a20 con 0x7fee4c0d8490 2021-04-09T18:33:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.938 7fee4ad73700 1 -- 172.21.15.32:0/1390953568 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44006790 con 0x7fee4c0d8490 2021-04-09T18:33:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.938 7fee4ad73700 1 -- 172.21.15.32:0/1390953568 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee4400b700 con 0x7fee4c0d8490 2021-04-09T18:33:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.938 7fee51993700 1 -- 172.21.15.32:0/1390953568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c0d8490 msgr2=0x7fee4c10da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.938 7fee51993700 1 --2- 172.21.15.32:0/1390953568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c0d8490 0x7fee4c10da00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fee44006e50 tx=0x7fee44009e70).stop 2021-04-09T18:33:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.939 7fee51993700 1 -- 172.21.15.32:0/1390953568 shutdown_connections 2021-04-09T18:33:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.939 7fee51993700 1 --2- 172.21.15.32:0/1390953568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c0d8490 0x7fee4c10da00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.939 7fee51993700 1 --2- 172.21.15.32:0/1390953568 >> v2:172.21.15.86:3300/0 conn(0x7fee4c10d010 0x7fee4c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.939 7fee51993700 1 -- 172.21.15.32:0/1390953568 >> 172.21.15.32:0/1390953568 conn(0x7fee4c0ce5f0 msgr2=0x7fee4c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:39.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.939 7fee51993700 1 -- 172.21.15.32:0/1390953568 shutdown_connections 2021-04-09T18:33:39.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.939 7fee51993700 1 -- 172.21.15.32:0/1390953568 wait complete. 2021-04-09T18:33:39.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee51993700 1 Processor -- start 2021-04-09T18:33:39.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee51993700 1 -- start start 2021-04-09T18:33:39.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee51993700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c179bf0 0x7fee4c0abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee51993700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4c0ac400 0x7fee4c1a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.985 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee51993700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee4c17a110 0x7fee4c1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee51993700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c17a510 con 0x7fee4c0ac400 2021-04-09T18:33:39.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee51993700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c1a68b0 con 0x7fee4c17a110 2021-04-09T18:33:39.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee51993700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c182e30 con 0x7fee4c179bf0 2021-04-09T18:33:39.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee43fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4c0ac400 0x7fee4c1a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee4b574700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c179bf0 0x7fee4c0abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee4bd75700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee4c17a110 0x7fee4c1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee43fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4c0ac400 0x7fee4c1a6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58902/0 (socket says 172.21.15.32:58902) 2021-04-09T18:33:39.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.959 7fee43fff700 1 -- 172.21.15.32:0/1326573248 learned_addr learned my addr 172.21.15.32:0/1326573248 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:39.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee4b574700 1 -- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee4c17a110 msgr2=0x7fee4c1a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee4b574700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee4c17a110 0x7fee4c1a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee4b574700 1 -- 172.21.15.32:0/1326573248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4c0ac400 msgr2=0x7fee4c1a6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:39.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee4b574700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4c0ac400 0x7fee4c1a6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:39.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee4b574700 1 -- 172.21.15.32:0/1326573248 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c1a56c0 con 0x7fee4c179bf0 2021-04-09T18:33:39.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee4bd75700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee4c17a110 0x7fee4c1a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:39.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee43fff700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4c0ac400 0x7fee4c1a6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:39.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee4b574700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c179bf0 0x7fee4c0abde0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fee3c009540 tx=0x7fee3c00b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee48d6f700 1 -- 172.21.15.32:0/1326573248 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee3c00bcd0 con 0x7fee4c179bf0 2021-04-09T18:33:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee48d6f700 1 -- 172.21.15.32:0/1326573248 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3c002030 con 0x7fee4c179bf0 2021-04-09T18:33:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee51993700 1 -- 172.21.15.32:0/1326573248 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c1a56c0 con 0x7fee4c179bf0 2021-04-09T18:33:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee48d6f700 1 -- 172.21.15.32:0/1326573248 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee3c0024c0 con 0x7fee4c179bf0 2021-04-09T18:33:39.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.960 7fee51993700 1 -- 172.21.15.32:0/1326573248 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c1a56c0 con 0x7fee4c179bf0 2021-04-09T18:33:39.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.961 7fee48d6f700 1 -- 172.21.15.32:0/1326573248 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fee3c000c90 con 0x7fee4c179bf0 2021-04-09T18:33:39.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.961 7fee48d6f700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee300188d0 0x7fee3001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:39.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.961 7fee48d6f700 1 -- 172.21.15.32:0/1326573248 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fee3c032aa0 con 0x7fee4c179bf0 2021-04-09T18:33:39.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.961 7fee43fff700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee300188d0 0x7fee3001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:39.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.961 7fee43fff700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee300188d0 0x7fee3001add0 crc :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:39.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.981 7fee51993700 1 -- 172.21.15.32:0/1326573248 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c17a510 con 0x7fee4c179bf0 2021-04-09T18:33:40.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:39.985 7fee48d6f700 1 -- 172.21.15.32:0/1326573248 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fee3c00bcd0 con 0x7fee4c179bf0 2021-04-09T18:33:40.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.174 7fee51993700 1 -- 172.21.15.32:0/1326573248 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee4c17a510 con 0x7fee4c179bf0 2021-04-09T18:33:40.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.175 7fee48d6f700 1 -- 172.21.15.32:0/1326573248 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fee3c000c90 con 0x7fee4c179bf0 2021-04-09T18:33:40.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:40.194 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":250,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:33:39.367763","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:33:36.709256","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-04-09 18:33:38.131023","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:33:40.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 -- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee300188d0 msgr2=0x7fee3001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:40.196 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee300188d0 0x7fee3001add0 crc :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 -- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c179bf0 msgr2=0x7fee4c0abde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:40.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c179bf0 0x7fee4c0abde0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fee3c009540 tx=0x7fee3c00b8e0).stop 2021-04-09T18:33:40.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 -- 172.21.15.32:0/1326573248 shutdown_connections 2021-04-09T18:33:40.197 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fee300188d0 0x7fee3001add0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fee4c0ac400 0x7fee4c1a6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee4c17a110 0x7fee4c1a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.198 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 --2- 172.21.15.32:0/1326573248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee4c179bf0 0x7fee4c0abde0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 -- 172.21.15.32:0/1326573248 >> 172.21.15.32:0/1326573248 conn(0x7fee4c1acae0 msgr2=0x7fee4c17ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:40.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 -- 172.21.15.32:0/1326573248 shutdown_connections 2021-04-09T18:33:40.199 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.179 7fee51993700 1 -- 172.21.15.32:0/1326573248 wait complete. 2021-04-09T18:33:40.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:33:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.343 7fcffad1a700 1 Processor -- start 2021-04-09T18:33:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.343 7fcffad1a700 1 -- start start 2021-04-09T18:33:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcffad1a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcff410d010 0x7fcff40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:40.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcffad1a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff40d1da0 0x7fcff40d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:40.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcffad1a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff410d480 con 0x7fcff40d82c0 2021-04-09T18:33:40.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcffad1a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcff40d9220 con 0x7fcff410d010 2021-04-09T18:33:40.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcffad1a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff40d86c0 con 0x7fcff40d1da0 2021-04-09T18:33:40.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff8b36700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcff410d010 0x7fcff40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:40.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff8b36700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcff410d010 0x7fcff40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58908/0 (socket says 172.21.15.32:58908) 2021-04-09T18:33:40.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff8b36700 1 -- 172.21.15.32:0/4030707090 learned_addr learned my addr 172.21.15.32:0/4030707090 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:40.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff3fff700 1 --2- 172.21.15.32:0/4030707090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff40d1da0 0x7fcff40d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:40.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff8b36700 1 -- 172.21.15.32:0/4030707090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff40d1da0 msgr2=0x7fcff40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:40.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff8b36700 1 --2- 172.21.15.32:0/4030707090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff40d1da0 0x7fcff40d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff8b36700 1 -- 172.21.15.32:0/4030707090 >> v1:172.21.15.86:6789/0 conn(0x7fcff40d82c0 legacy=0x7fcff410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:40.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff8b36700 1 -- 172.21.15.32:0/4030707090 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff410e1f0 con 0x7fcff410d010 2021-04-09T18:33:40.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff3fff700 1 --2- 172.21.15.32:0/4030707090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff40d1da0 0x7fcff40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:40.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff8b36700 1 --2- 172.21.15.32:0/4030707090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff410d010 0x7fcff40d1780 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 rx=0x7fcfe4006700 tx=0x7fcfe4009b80).ready entity=mon.0 client_cookie=bada1fa2a1405af9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:40.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff37fe700 1 -- 172.21.15.32:0/4030707090 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfe400c0c0 con 0x7fcff410d010 2021-04-09T18:33:40.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff37fe700 1 -- 172.21.15.32:0/4030707090 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe400c440 con 0x7fcff410d010 2021-04-09T18:33:40.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.344 7fcff37fe700 1 -- 172.21.15.32:0/4030707090 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfe400c920 con 0x7fcff410d010 2021-04-09T18:33:40.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.345 7fcffad1a700 1 -- 172.21.15.32:0/4030707090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff410d010 msgr2=0x7fcff40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:40.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.345 7fcffad1a700 1 --2- 172.21.15.32:0/4030707090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff410d010 0x7fcff40d1780 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 rx=0x7fcfe4006700 tx=0x7fcfe4009b80).stop 2021-04-09T18:33:40.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.345 7fcffad1a700 1 -- 172.21.15.32:0/4030707090 shutdown_connections 2021-04-09T18:33:40.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.345 7fcffad1a700 1 --2- 172.21.15.32:0/4030707090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff40d1da0 0x7fcff40d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.345 7fcffad1a700 1 --2- 172.21.15.32:0/4030707090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff410d010 0x7fcff40d1780 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.345 7fcffad1a700 1 -- 172.21.15.32:0/4030707090 >> 172.21.15.32:0/4030707090 conn(0x7fcff40ce5f0 msgr2=0x7fcff40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.345 7fcffad1a700 1 -- 172.21.15.32:0/4030707090 shutdown_connections 2021-04-09T18:33:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.345 7fcffad1a700 1 -- 172.21.15.32:0/4030707090 wait complete. 2021-04-09T18:33:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.365 7fcffad1a700 1 Processor -- start 2021-04-09T18:33:40.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.365 7fcffad1a700 1 -- start start 2021-04-09T18:33:40.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcffad1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff408a910 0x7fcff409e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:40.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcffad1a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff408ad10 0x7fcff409ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:40.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcffad1a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcff4092380 0x7fcff4092780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:40.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcffad1a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff408b110 con 0x7fcff408a910 2021-04-09T18:33:40.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcffad1a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff409e0d0 con 0x7fcff4092380 2021-04-09T18:33:40.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcffad1a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff408b3f0 con 0x7fcff408ad10 2021-04-09T18:33:40.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcff8b36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff408a910 0x7fcff409e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:40.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcff8b36700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff408a910 0x7fcff409e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58914/0 (socket says 172.21.15.32:58914) 2021-04-09T18:33:40.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcff8b36700 1 -- 172.21.15.32:0/3065689821 learned_addr learned my addr 172.21.15.32:0/3065689821 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcff3fff700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff408ad10 0x7fcff409ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.366 7fcff9337700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcff4092380 0x7fcff4092780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcff8b36700 1 -- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff408ad10 msgr2=0x7fcff409ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcff8b36700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff408ad10 0x7fcff409ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcff8b36700 1 -- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcff4092380 msgr2=0x7fcff4092780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcff8b36700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcff4092380 0x7fcff4092780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcff8b36700 1 -- 172.21.15.32:0/3065689821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff4092da0 con 0x7fcff408a910 2021-04-09T18:33:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcff8b36700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff408a910 0x7fcff409e650 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7fcfe400b580 tx=0x7fcfe400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcff17fa700 1 -- 172.21.15.32:0/3065689821 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfe400bd10 con 0x7fcff408a910 2021-04-09T18:33:40.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcff17fa700 1 -- 172.21.15.32:0/3065689821 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe4002030 con 0x7fcff408a910 2021-04-09T18:33:40.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff4092da0 con 0x7fcff408a910 2021-04-09T18:33:40.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcff17fa700 1 -- 172.21.15.32:0/3065689821 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfe4002550 con 0x7fcff408a910 2021-04-09T18:33:40.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.367 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff4092da0 con 0x7fcff408a910 2021-04-09T18:33:40.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.368 7fcff17fa700 1 -- 172.21.15.32:0/3065689821 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcfe4002210 con 0x7fcff408a910 2021-04-09T18:33:40.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.368 7fcff17fa700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfd4018930 0x7fcfd401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:40.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.368 7fcff17fa700 1 -- 172.21.15.32:0/3065689821 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcfe4032a30 con 0x7fcff408a910 2021-04-09T18:33:40.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.368 7fcff3fff700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfd4018930 0x7fcfd401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:40.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.368 7fcff3fff700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfd4018930 0x7fcfd401ae10 crc :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:40.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.388 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff418fc90 con 0x7fcff408a910 2021-04-09T18:33:40.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.391 7fcff17fa700 1 -- 172.21.15.32:0/3065689821 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcfe4002300 con 0x7fcff408a910 2021-04-09T18:33:40.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:40.587 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fcff418fc90 con 0x7fcff408a910 2021-04-09T18:33:41.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.376 7fcff17fa700 1 -- 172.21.15.32:0/3065689821 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v411) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcfe4002030 con 0x7fcff408a910 2021-04-09T18:33:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfd4018930 msgr2=0x7fcfd401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:41.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfd4018930 0x7fcfd401ae10 crc :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff408a910 msgr2=0x7fcff409e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff408a910 0x7fcff409e650 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7fcfe400b580 tx=0x7fcfe400b920).stop 2021-04-09T18:33:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 shutdown_connections 2021-04-09T18:33:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcfd4018930 0x7fcfd401ae10 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcff4092380 0x7fcff4092780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcff408ad10 0x7fcff409ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 --2- 172.21.15.32:0/3065689821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcff408a910 0x7fcff409e650 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 >> 172.21.15.32:0/3065689821 conn(0x7fcff40b2fa0 msgr2=0x7fcff40a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:41.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 shutdown_connections 2021-04-09T18:33:41.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.404 7fcffad1a700 1 -- 172.21.15.32:0/3065689821 wait complete. 2021-04-09T18:33:41.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:41.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.564 7f845347f700 1 Processor -- start 2021-04-09T18:33:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.564 7f845347f700 1 -- start start 2021-04-09T18:33:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f845347f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f844c10d010 0x7f844c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f845347f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0d1da0 0x7f844c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:41.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f845347f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c10d480 con 0x7f844c0d82c0 2021-04-09T18:33:41.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f845347f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844c0d9220 con 0x7f844c10d010 2021-04-09T18:33:41.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f845347f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0d86c0 con 0x7f844c0d1da0 2021-04-09T18:33:41.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8451a9c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f844c0d82c0 0x7f844c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40282/0 (socket says 172.21.15.32:40282) 2021-04-09T18:33:41.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8451a9c700 1 -- 172.21.15.32:0/3897880092 learned_addr learned my addr 172.21.15.32:0/3897880092 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:41.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f845129b700 1 --2- 172.21.15.32:0/3897880092 >> v2:172.21.15.86:3300/0 conn(0x7f844c10d010 0x7f844c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:41.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8450a9a700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0d1da0 0x7f844c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:41.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637643269 0 0) 0x7f8444000d30 con 0x7f844c0d82c0 2021-04-09T18:33:41.609 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8438002fe0 con 0x7f844c0d82c0 2021-04-09T18:33:41.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 415043876 0 0) 0x7f8444001460 con 0x7f844c0d82c0 2021-04-09T18:33:41.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0d1da0 msgr2=0x7f844c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:41.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0d1da0 0x7f844c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 >> v2:172.21.15.86:3300/0 conn(0x7f844c10d010 msgr2=0x7f844c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:41.610 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 --2- 172.21.15.32:0/3897880092 >> v2:172.21.15.86:3300/0 conn(0x7f844c10d010 0x7f844c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c10e1f0 con 0x7f844c0d82c0 2021-04-09T18:33:41.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f8444001980 con 0x7f844c0d82c0 2021-04-09T18:33:41.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 >> v1:172.21.15.32:6789/0 conn(0x7f844c0d82c0 legacy=0x7f844c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:41.611 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8438003b90 0x7f8438006080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:41.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f843804a8b0 0x7f843804cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:41.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f843804d650 0x7f843804fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:41.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8438050420 con 0x7f843804a8b0 2021-04-09T18:33:41.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8438050640 con 0x7f843804d650 2021-04-09T18:33:41.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8443fff700 1 -- 172.21.15.32:0/3897880092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8438050860 con 0x7f8438003b90 2021-04-09T18:33:41.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.565 7f8450a9a700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c0d1da0 0x7f844c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:41.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f845129b700 1 --2- 172.21.15.32:0/3897880092 >> v2:172.21.15.86:3300/0 conn(0x7f844c10d010 0x7f844c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:41.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8450a9a700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f843804a8b0 0x7f843804cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:41.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8451a9c700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f843804d650 0x7f843804fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:41.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f845129b700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8438003b90 0x7f8438006080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:41.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8450a9a700 1 -- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8438003b90 msgr2=0x7f8438006080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:41.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8450a9a700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8438003b90 0x7f8438006080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8450a9a700 1 -- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f843804d650 msgr2=0x7f843804fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:41.614 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8450a9a700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f843804d650 0x7f843804fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8450a9a700 1 -- 172.21.15.32:0/3897880092 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8438050b00 con 0x7f843804a8b0 2021-04-09T18:33:41.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8450a9a700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f843804a8b0 0x7f843804cdb0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 rx=0x7f843c0063a0 tx=0x7f843c009a20).ready entity=mon.0 client_cookie=c5590a469c0266cc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:41.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8443fff700 1 -- 172.21.15.32:0/3897880092 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f843c00c0c0 con 0x7f843804a8b0 2021-04-09T18:33:41.615 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8443fff700 1 -- 172.21.15.32:0/3897880092 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c00c400 con 0x7f843804a8b0 2021-04-09T18:33:41.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8443fff700 1 -- 172.21.15.32:0/3897880092 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f843c00c8e0 con 0x7f843804a8b0 2021-04-09T18:33:41.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f845347f700 1 -- 172.21.15.32:0/3897880092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f843804a8b0 msgr2=0x7f843804cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:41.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f845347f700 1 --2- 172.21.15.32:0/3897880092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f843804a8b0 0x7f843804cdb0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 rx=0x7f843c0063a0 tx=0x7f843c009a20).stop 2021-04-09T18:33:41.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.566 7f8451a9c700 1 -- 172.21.15.32:0/3897880092 reap_dead start 2021-04-09T18:33:41.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.567 7f845347f700 1 -- 172.21.15.32:0/3897880092 shutdown_connections 2021-04-09T18:33:41.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.567 7f845347f700 1 -- 172.21.15.32:0/3897880092 >> 172.21.15.32:0/3897880092 conn(0x7f844c0ce5f0 msgr2=0x7f844c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:41.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.567 7f845347f700 1 -- 172.21.15.32:0/3897880092 shutdown_connections 2021-04-09T18:33:41.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.567 7f845347f700 1 -- 172.21.15.32:0/3897880092 wait complete. 2021-04-09T18:33:41.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.587 7f845347f700 1 Processor -- start 2021-04-09T18:33:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.587 7f845347f700 1 -- start start 2021-04-09T18:33:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845347f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c1aca20 0x7f844c1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845347f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c18bd40 0x7f844c18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:41.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845347f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c08fba0 0x7f844c08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845347f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c18c7e0 con 0x7f844c1aca20 2021-04-09T18:33:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845347f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c18ca30 con 0x7f844c18bd40 2021-04-09T18:33:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845347f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0905f0 con 0x7f844c08fba0 2021-04-09T18:33:41.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845129b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c1aca20 0x7f844c1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:41.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845129b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c1aca20 0x7f844c1ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58934/0 (socket says 172.21.15.32:58934) 2021-04-09T18:33:41.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845129b700 1 -- 172.21.15.32:0/3181727268 learned_addr learned my addr 172.21.15.32:0/3181727268 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:41.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f8451a9c700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c08fba0 0x7f844c08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:41.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f8450a9a700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c18bd40 0x7f844c18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:41.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845129b700 1 -- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c08fba0 msgr2=0x7f844c08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:41.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845129b700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c08fba0 0x7f844c08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845129b700 1 -- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c18bd40 msgr2=0x7f844c18c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:41.621 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845129b700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c18bd40 0x7f844c18c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f845129b700 1 -- 172.21.15.32:0/3181727268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c0b4300 con 0x7f844c1aca20 2021-04-09T18:33:41.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.588 7f8451a9c700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c08fba0 0x7f844c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:41.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.589 7f845129b700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c1aca20 0x7f844c1ace20 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7f8448006af0 tx=0x7f844800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:41.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.589 7f8441ffb700 1 -- 172.21.15.32:0/3181727268 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f844800bd50 con 0x7f844c1aca20 2021-04-09T18:33:41.622 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.589 7f8441ffb700 1 -- 172.21.15.32:0/3181727268 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8448002030 con 0x7f844c1aca20 2021-04-09T18:33:41.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.589 7f845347f700 1 -- 172.21.15.32:0/3181727268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c0b4300 con 0x7f844c1aca20 2021-04-09T18:33:41.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.589 7f8441ffb700 1 -- 172.21.15.32:0/3181727268 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8448002550 con 0x7f844c1aca20 2021-04-09T18:33:41.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.589 7f845347f700 1 -- 172.21.15.32:0/3181727268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c0b4300 con 0x7f844c1aca20 2021-04-09T18:33:41.623 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.589 7f8441ffb700 1 -- 172.21.15.32:0/3181727268 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8448000c90 con 0x7f844c1aca20 2021-04-09T18:33:41.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.590 7f8441ffb700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:41.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.590 7f8441ffb700 1 -- 172.21.15.32:0/3181727268 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8448032a40 con 0x7f844c1aca20 2021-04-09T18:33:41.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.590 7f8450a9a700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:41.624 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.590 7f8450a9a700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 crc :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:41.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.609 7f845347f700 1 -- 172.21.15.32:0/3181727268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c182d70 con 0x7f844c1aca20 2021-04-09T18:33:41.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.612 7f8441ffb700 1 -- 172.21.15.32:0/3181727268 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f844801f780 con 0x7f844c1aca20 2021-04-09T18:33:41.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.802 7f845347f700 1 -- 172.21.15.32:0/3181727268 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f844c18c760 con 0x7f844c1aca20 2021-04-09T18:33:41.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.803 7f8441ffb700 1 -- 172.21.15.32:0/3181727268 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 411 v411) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f844800bd50 con 0x7f844c1aca20 2021-04-09T18:33:41.820 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 411 2021-04-09T18:33:41.820 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:41.821 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":411,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12519,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407},{"gid":12522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407},{"gid":59714,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407}],"filesystems":[{"mdsmap":{"epoch":411,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:41.372484","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 -- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 msgr2=0x7f842c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 crc :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 -- 172.21.15.32:0/3181727268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c1aca20 msgr2=0x7f844c1ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:41.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c1aca20 0x7f844c1ace20 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7f8448006af0 tx=0x7f844800b960).stop 2021-04-09T18:33:41.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 -- 172.21.15.32:0/3181727268 shutdown_connections 2021-04-09T18:33:41.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f842c018930 0x7f842c01ae10 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f844c08fba0 0x7f844c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844c18bd40 0x7f844c18c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 --2- 172.21.15.32:0/3181727268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f844c1aca20 0x7f844c1ace20 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.805 7f845347f700 1 -- 172.21.15.32:0/3181727268 >> 172.21.15.32:0/3181727268 conn(0x7f844c19b460 msgr2=0x7f844c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:41.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.806 7f845347f700 1 -- 172.21.15.32:0/3181727268 shutdown_connections 2021-04-09T18:33:41.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.806 7f845347f700 1 -- 172.21.15.32:0/3181727268 wait complete. 2021-04-09T18:33:41.832 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:33:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.963 7f0844444700 1 Processor -- start 2021-04-09T18:33:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.963 7f0844444700 1 -- start start 2021-04-09T18:33:42.004 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.964 7f0844444700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f083c0cffa0 0x7f083c0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.964 7f0844444700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c0d7800 0x7f083c0d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.964 7f0844444700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c0d7280 con 0x7f083c0d8220 2021-04-09T18:33:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.964 7f0844444700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083c0d91e0 con 0x7f083c0cffa0 2021-04-09T18:33:42.005 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.964 7f0844444700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c10dfe0 con 0x7f083c0d7800 2021-04-09T18:33:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0841a5f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c0d7800 0x7f083c0d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0842260700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f083c0cffa0 0x7f083c0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0842a61700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f083c0d8220 0x7f083c0d5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38226/0 (socket says 172.21.15.32:38226) 2021-04-09T18:33:42.006 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0842a61700 1 -- 172.21.15.32:0/1714212459 learned_addr learned my addr 172.21.15.32:0/1714212459 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0841a5f700 1 -- 172.21.15.32:0/1714212459 >> v2:172.21.15.86:3300/0 conn(0x7f083c0cffa0 msgr2=0x7f083c0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0841a5f700 1 --2- 172.21.15.32:0/1714212459 >> v2:172.21.15.86:3300/0 conn(0x7f083c0cffa0 0x7f083c0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0841a5f700 1 -- 172.21.15.32:0/1714212459 >> v1:172.21.15.86:6789/0 conn(0x7f083c0d8220 legacy=0x7f083c0d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.007 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0841a5f700 1 -- 172.21.15.32:0/1714212459 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c10e200 con 0x7f083c0d7800 2021-04-09T18:33:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0842260700 1 --2- 172.21.15.32:0/1714212459 >> v2:172.21.15.86:3300/0 conn(0x7f083c0cffa0 0x7f083c0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f0841a5f700 1 --2- 172.21.15.32:0/1714212459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c0d7800 0x7f083c0d7c00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f0834006650 tx=0x7f0834009a80).ready entity=mon.2 client_cookie=45adf31227abc746 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f084125e700 1 -- 172.21.15.32:0/1714212459 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08340009b0 con 0x7f083c0d7800 2021-04-09T18:33:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f084125e700 1 -- 172.21.15.32:0/1714212459 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083400c0c0 con 0x7f083c0d7800 2021-04-09T18:33:42.008 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.965 7f084125e700 1 -- 172.21.15.32:0/1714212459 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f083400c510 con 0x7f083c0d7800 2021-04-09T18:33:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.966 7f0844444700 1 -- 172.21.15.32:0/1714212459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c0d7800 msgr2=0x7f083c0d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.966 7f0844444700 1 --2- 172.21.15.32:0/1714212459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c0d7800 0x7f083c0d7c00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f0834006650 tx=0x7f0834009a80).stop 2021-04-09T18:33:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.966 7f0844444700 1 -- 172.21.15.32:0/1714212459 shutdown_connections 2021-04-09T18:33:42.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.966 7f0844444700 1 --2- 172.21.15.32:0/1714212459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c0d7800 0x7f083c0d7c00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.966 7f0844444700 1 --2- 172.21.15.32:0/1714212459 >> v2:172.21.15.86:3300/0 conn(0x7f083c0cffa0 0x7f083c0d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.966 7f0844444700 1 -- 172.21.15.32:0/1714212459 >> 172.21.15.32:0/1714212459 conn(0x7f083c0ce5c0 msgr2=0x7f083c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.966 7f0844444700 1 -- 172.21.15.32:0/1714212459 shutdown_connections 2021-04-09T18:33:42.010 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.966 7f0844444700 1 -- 172.21.15.32:0/1714212459 wait complete. 2021-04-09T18:33:42.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0844444700 1 Processor -- start 2021-04-09T18:33:42.011 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0844444700 1 -- start start 2021-04-09T18:33:42.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0844444700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1acff0 0x7f083c08fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0844444700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f083c090150 0x7f083c08c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0844444700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c08cca0 0x7f083c08e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0844444700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c08c100 con 0x7f083c090150 2021-04-09T18:33:42.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0844444700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c08d0a0 con 0x7f083c08cca0 2021-04-09T18:33:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0844444700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c08e960 con 0x7f083c1acff0 2021-04-09T18:33:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0841a5f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f083c090150 0x7f083c08c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0841a5f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f083c090150 0x7f083c08c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58946/0 (socket says 172.21.15.32:58946) 2021-04-09T18:33:42.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0841a5f700 1 -- 172.21.15.32:0/1321579866 learned_addr learned my addr 172.21.15.32:0/1321579866 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0842a61700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c08cca0 0x7f083c08e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.987 7f0842260700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1acff0 0x7f083c08fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f0841a5f700 1 -- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1acff0 msgr2=0x7f083c08fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f0841a5f700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1acff0 0x7f083c08fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f0841a5f700 1 -- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c08cca0 msgr2=0x7f083c08e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f0841a5f700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c08cca0 0x7f083c08e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f0841a5f700 1 -- 172.21.15.32:0/1321579866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c08eb80 con 0x7f083c090150 2021-04-09T18:33:42.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f0841a5f700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f083c090150 0x7f083c08c680 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 rx=0x7f0834009630 tx=0x7f083400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f083affd700 1 -- 172.21.15.32:0/1321579866 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08340093a0 con 0x7f083c090150 2021-04-09T18:33:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f083affd700 1 -- 172.21.15.32:0/1321579866 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083400bd70 con 0x7f083c090150 2021-04-09T18:33:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f0844444700 1 -- 172.21.15.32:0/1321579866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c08eb80 con 0x7f083c090150 2021-04-09T18:33:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f083affd700 1 -- 172.21.15.32:0/1321579866 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08340054d0 con 0x7f083c090150 2021-04-09T18:33:42.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.988 7f0844444700 1 -- 172.21.15.32:0/1321579866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c08eb80 con 0x7f083c090150 2021-04-09T18:33:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.989 7f083affd700 1 -- 172.21.15.32:0/1321579866 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f08340054f0 con 0x7f083c090150 2021-04-09T18:33:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.989 7f083affd700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f081c018930 0x7f081c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.989 7f083affd700 1 -- 172.21.15.32:0/1321579866 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0834032a30 con 0x7f083c090150 2021-04-09T18:33:42.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.989 7f0842260700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f081c018930 0x7f081c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:41.989 7f0842260700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f081c018930 0x7f081c01ae10 crc :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:42.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.008 7f0844444700 1 -- 172.21.15.32:0/1321579866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083c182dc0 con 0x7f083c090150 2021-04-09T18:33:42.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.012 7f083affd700 1 -- 172.21.15.32:0/1321579866 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f08340052e0 con 0x7f083c090150 2021-04-09T18:33:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.202 7f0844444700 1 -- 172.21.15.32:0/1321579866 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f083c0d7280 con 0x7f083c090150 2021-04-09T18:33:42.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.203 7f083affd700 1 -- 172.21.15.32:0/1321579866 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f08340054f0 con 0x7f083c090150 2021-04-09T18:33:42.222 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:42.222 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":251,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:33:40.369286","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:33:36.709256","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-04-09 18:33:38.131023","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:33:42.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.205 7f0844444700 1 -- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f081c018930 msgr2=0x7f081c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.205 7f0844444700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f081c018930 0x7f081c01ae10 crc :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.205 7f0844444700 1 -- 172.21.15.32:0/1321579866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f083c090150 msgr2=0x7f083c08c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.205 7f0844444700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f083c090150 0x7f083c08c680 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 rx=0x7f0834009630 tx=0x7f083400b980).stop 2021-04-09T18:33:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.206 7f0844444700 1 -- 172.21.15.32:0/1321579866 shutdown_connections 2021-04-09T18:33:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.206 7f0844444700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f081c018930 0x7f081c01ae10 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.206 7f0844444700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f083c090150 0x7f083c08c680 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.206 7f0844444700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c08cca0 0x7f083c08e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.206 7f0844444700 1 --2- 172.21.15.32:0/1321579866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1acff0 0x7f083c08fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.206 7f0844444700 1 -- 172.21.15.32:0/1321579866 >> 172.21.15.32:0/1321579866 conn(0x7f083c09abb0 msgr2=0x7f083c0a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:42.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.206 7f0844444700 1 -- 172.21.15.32:0/1321579866 shutdown_connections 2021-04-09T18:33:42.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.206 7f0844444700 1 -- 172.21.15.32:0/1321579866 wait complete. 2021-04-09T18:33:42.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:33:42.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.377 7fb3e9462700 1 Processor -- start 2021-04-09T18:33:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.377 7fb3e9462700 1 -- start start 2021-04-09T18:33:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e9462700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e40cffb0 0x7fb3e4087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e9462700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3e40d7c50 0x7fb3e41b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e9462700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3e40d8050 0x7fb3e410d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e9462700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e40d91f0 con 0x7fb3e40d8050 2021-04-09T18:33:42.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e9462700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e4088460 con 0x7fb3e40d7c50 2021-04-09T18:33:42.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e9462700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40d8620 con 0x7fb3e40cffb0 2021-04-09T18:33:42.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e387e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3e40d8050 0x7fb3e410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e287c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3e40d7c50 0x7fb3e41b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e387e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3e40d8050 0x7fb3e410d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58954/0 (socket says 172.21.15.32:58954) 2021-04-09T18:33:42.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e387e700 1 -- 172.21.15.32:0/499241768 learned_addr learned my addr 172.21.15.32:0/499241768 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:42.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e307d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e40cffb0 0x7fb3e4087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e287c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3e40d7c50 0x7fb3e41b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:50712/0 (socket says 172.21.15.32:50712) 2021-04-09T18:33:42.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e387e700 1 -- 172.21.15.32:0/499241768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e40cffb0 msgr2=0x7fb3e4087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:42.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e387e700 1 --2- 172.21.15.32:0/499241768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e40cffb0 0x7fb3e4087e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e387e700 1 -- 172.21.15.32:0/499241768 >> v2:172.21.15.86:3300/0 conn(0x7fb3e40d7c50 msgr2=0x7fb3e41b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:42.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e387e700 1 --2- 172.21.15.32:0/499241768 >> v2:172.21.15.86:3300/0 conn(0x7fb3e40d7c50 0x7fb3e41b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e387e700 1 -- 172.21.15.32:0/499241768 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e40d2800 con 0x7fb3e40d8050 2021-04-09T18:33:42.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e387e700 1 --2- 172.21.15.32:0/499241768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e40d8050 0x7fb3e410d200 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7fb3dc006b80 tx=0x7fb3dc009b10).ready entity=mon.0 client_cookie=c4dc864fc07cdbe6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:42.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e207b700 1 -- 172.21.15.32:0/499241768 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3dc0022e0 con 0x7fb3e40d8050 2021-04-09T18:33:42.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e207b700 1 -- 172.21.15.32:0/499241768 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc002620 con 0x7fb3e40d8050 2021-04-09T18:33:42.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.378 7fb3e207b700 1 -- 172.21.15.32:0/499241768 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3dc002b00 con 0x7fb3e40d8050 2021-04-09T18:33:42.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.379 7fb3e9462700 1 -- 172.21.15.32:0/499241768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e40d8050 msgr2=0x7fb3e410d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.379 7fb3e9462700 1 --2- 172.21.15.32:0/499241768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e40d8050 0x7fb3e410d200 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7fb3dc006b80 tx=0x7fb3dc009b10).stop 2021-04-09T18:33:42.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.379 7fb3e9462700 1 -- 172.21.15.32:0/499241768 shutdown_connections 2021-04-09T18:33:42.422 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.379 7fb3e9462700 1 --2- 172.21.15.32:0/499241768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e40d8050 0x7fb3e410d200 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.379 7fb3e9462700 1 --2- 172.21.15.32:0/499241768 >> v2:172.21.15.86:3300/0 conn(0x7fb3e40d7c50 0x7fb3e41b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.379 7fb3e9462700 1 --2- 172.21.15.32:0/499241768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e40cffb0 0x7fb3e4087e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.379 7fb3e9462700 1 -- 172.21.15.32:0/499241768 >> 172.21.15.32:0/499241768 conn(0x7fb3e40ce5b0 msgr2=0x7fb3e40cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:42.423 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.379 7fb3e9462700 1 -- 172.21.15.32:0/499241768 shutdown_connections 2021-04-09T18:33:42.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.379 7fb3e9462700 1 -- 172.21.15.32:0/499241768 wait complete. 2021-04-09T18:33:42.424 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.399 7fb3e9462700 1 Processor -- start 2021-04-09T18:33:42.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e9462700 1 -- start start 2021-04-09T18:33:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e9462700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3e409d680 0x7fb3e409afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e9462700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e409da80 0x7fb3e4191dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e9462700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e41923f0 0x7fb3e409be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.427 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e9462700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e409cbe0 con 0x7fb3e41923f0 2021-04-09T18:33:42.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e9462700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e409b8e0 con 0x7fb3e409d680 2021-04-09T18:33:42.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e9462700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e409b5c0 con 0x7fb3e409da80 2021-04-09T18:33:42.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e387e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e41923f0 0x7fb3e409be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.428 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e387e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e41923f0 0x7fb3e409be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58960/0 (socket says 172.21.15.32:58960) 2021-04-09T18:33:42.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e387e700 1 -- 172.21.15.32:0/300623760 learned_addr learned my addr 172.21.15.32:0/300623760 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:42.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e307d700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3e409d680 0x7fb3e409afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.400 7fb3e287c700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e409da80 0x7fb3e4191dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3e387e700 1 -- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e409da80 msgr2=0x7fb3e4191dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.429 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3e387e700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e409da80 0x7fb3e4191dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3e387e700 1 -- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3e409d680 msgr2=0x7fb3e409afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3e387e700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3e409d680 0x7fb3e409afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3e387e700 1 -- 172.21.15.32:0/300623760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e409c480 con 0x7fb3e41923f0 2021-04-09T18:33:42.430 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3e387e700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e41923f0 0x7fb3e409be60 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 rx=0x7fb3dc006ec0 tx=0x7fb3dc00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3dbfff700 1 -- 172.21.15.32:0/300623760 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3dc00bdd0 con 0x7fb3e41923f0 2021-04-09T18:33:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3dbfff700 1 -- 172.21.15.32:0/300623760 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc006240 con 0x7fb3e41923f0 2021-04-09T18:33:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3dbfff700 1 -- 172.21.15.32:0/300623760 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3dc005720 con 0x7fb3e41923f0 2021-04-09T18:33:42.431 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3e9462700 1 -- 172.21.15.32:0/300623760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e409c480 con 0x7fb3e41923f0 2021-04-09T18:33:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.401 7fb3e9462700 1 -- 172.21.15.32:0/300623760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e409c480 con 0x7fb3e41923f0 2021-04-09T18:33:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.402 7fb3dbfff700 1 -- 172.21.15.32:0/300623760 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb3dc006a30 con 0x7fb3e41923f0 2021-04-09T18:33:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.402 7fb3dbfff700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3c8018930 0x7fb3c801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.402 7fb3dbfff700 1 -- 172.21.15.32:0/300623760 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb3dc032a40 con 0x7fb3e41923f0 2021-04-09T18:33:42.432 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.402 7fb3e307d700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3c8018930 0x7fb3c801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.402 7fb3e307d700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3c8018930 0x7fb3c801ae10 crc :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:42.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.422 7fb3e9462700 1 -- 172.21.15.32:0/300623760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e4093fc0 con 0x7fb3e41923f0 2021-04-09T18:33:42.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.425 7fb3dbfff700 1 -- 172.21.15.32:0/300623760 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb3dc006030 con 0x7fb3e41923f0 2021-04-09T18:33:42.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.613 7fb3e9462700 1 -- 172.21.15.32:0/300623760 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb3e4093fc0 con 0x7fb3e41923f0 2021-04-09T18:33:42.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.616 7fb3dbfff700 1 -- 172.21.15.32:0/300623760 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v47) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb3dc006240 con 0x7fb3e41923f0 2021-04-09T18:33:42.633 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:33:42.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 -- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3c8018930 msgr2=0x7fb3c801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3c8018930 0x7fb3c801ae10 crc :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 -- 172.21.15.32:0/300623760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e41923f0 msgr2=0x7fb3e409be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e41923f0 0x7fb3e409be60 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 rx=0x7fb3dc006ec0 tx=0x7fb3dc00b9e0).stop 2021-04-09T18:33:42.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 -- 172.21.15.32:0/300623760 shutdown_connections 2021-04-09T18:33:42.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3c8018930 0x7fb3c801ae10 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3e41923f0 0x7fb3e409be60 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3e409da80 0x7fb3e4191dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 --2- 172.21.15.32:0/300623760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3e409d680 0x7fb3e409afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 -- 172.21.15.32:0/300623760 >> 172.21.15.32:0/300623760 conn(0x7fb3e40adaf0 msgr2=0x7fb3e40bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:42.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 -- 172.21.15.32:0/300623760 shutdown_connections 2021-04-09T18:33:42.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.618 7fb3e9462700 1 -- 172.21.15.32:0/300623760 wait complete. 2021-04-09T18:33:42.646 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:33:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.789 7f3da2295700 1 Processor -- start 2021-04-09T18:33:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.789 7f3da2295700 1 -- start start 2021-04-09T18:33:42.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3da2295700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d9c0d8050 0x7f3d9c10d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3da2295700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0d8620 con 0x7f3d9c0cffb0 2021-04-09T18:33:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3da2295700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0d2800 con 0x7f3d9c0d7c50 2021-04-09T18:33:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3da2295700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c1b0c80 con 0x7f3d9c0d8050 2021-04-09T18:33:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3d9bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d9c0cffb0 0x7f3d9c087e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40322/0 (socket says 172.21.15.32:40322) 2021-04-09T18:33:42.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3d9bfff700 1 -- 172.21.15.32:0/3836746237 learned_addr learned my addr 172.21.15.32:0/3836746237 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:42.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3da08b2700 1 --2- 172.21.15.32:0/3836746237 >> v2:172.21.15.86:3300/0 conn(0x7f3d9c0d8050 0x7f3d9c10d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3d9b7fe700 1 -- 172.21.15.32:0/3836746237 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968726466 0 0) 0x7f3d8c000ce0 con 0x7f3d9c0cffb0 2021-04-09T18:33:42.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3d9b7fe700 1 -- 172.21.15.32:0/3836746237 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d88002fe0 con 0x7f3d9c0cffb0 2021-04-09T18:33:42.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3d9b7fe700 1 -- 172.21.15.32:0/3836746237 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290262001 0 0) 0x7f3d84000c60 con 0x7f3d9c0d7c50 2021-04-09T18:33:42.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3d9b7fe700 1 -- 172.21.15.32:0/3836746237 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d88003210 con 0x7f3d9c0d7c50 2021-04-09T18:33:42.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3d9b7fe700 1 -- 172.21.15.32:0/3836746237 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 343898599 0 0) 0x7f3d8c000ed0 con 0x7f3d9c0cffb0 2021-04-09T18:33:42.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3da08b2700 1 -- 172.21.15.32:0/3836746237 >> v1:172.21.15.86:6789/0 conn(0x7f3d9c0d7c50 legacy=0x7f3d9c1b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3da08b2700 1 -- 172.21.15.32:0/3836746237 >> v1:172.21.15.32:6789/0 conn(0x7f3d9c0cffb0 legacy=0x7f3d9c087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.790 7f3da08b2700 1 -- 172.21.15.32:0/3836746237 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c0d91f0 con 0x7f3d9c0d8050 2021-04-09T18:33:42.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.791 7f3da08b2700 1 --2- 172.21.15.32:0/3836746237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d9c0d8050 0x7f3d9c10d200 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7f3d94006e50 tx=0x7f3d94009e70).ready entity=mon.1 client_cookie=a4daf70233b422de server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:42.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.791 7f3d9b7fe700 1 -- 172.21.15.32:0/3836746237 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d94006a20 con 0x7f3d9c0d8050 2021-04-09T18:33:42.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.791 7f3d9b7fe700 1 -- 172.21.15.32:0/3836746237 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d94006790 con 0x7f3d9c0d8050 2021-04-09T18:33:42.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.791 7f3d9b7fe700 1 -- 172.21.15.32:0/3836746237 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d9400b700 con 0x7f3d9c0d8050 2021-04-09T18:33:42.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.791 7f3da2295700 1 -- 172.21.15.32:0/3836746237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d9c0d8050 msgr2=0x7f3d9c10d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.791 7f3da2295700 1 --2- 172.21.15.32:0/3836746237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d9c0d8050 0x7f3d9c10d200 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7f3d94006e50 tx=0x7f3d94009e70).stop 2021-04-09T18:33:42.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.792 7f3da2295700 1 -- 172.21.15.32:0/3836746237 shutdown_connections 2021-04-09T18:33:42.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.792 7f3da2295700 1 --2- 172.21.15.32:0/3836746237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d9c0d8050 0x7f3d9c10d200 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.792 7f3da2295700 1 -- 172.21.15.32:0/3836746237 >> 172.21.15.32:0/3836746237 conn(0x7f3d9c0ce5b0 msgr2=0x7f3d9c0cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:42.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.792 7f3da2295700 1 -- 172.21.15.32:0/3836746237 shutdown_connections 2021-04-09T18:33:42.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.792 7f3da2295700 1 -- 172.21.15.32:0/3836746237 wait complete. 2021-04-09T18:33:42.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3da2295700 1 Processor -- start 2021-04-09T18:33:42.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3da2295700 1 -- start start 2021-04-09T18:33:42.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3da2295700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d9c1a0840 0x7f3d9c191ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3da2295700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c192100 0x7f3d9c0babe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3da2295700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d9c0bb200 0x7f3d9c0c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3da2295700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0ba660 con 0x7f3d9c192100 2021-04-09T18:33:42.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3da2295700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c192500 con 0x7f3d9c0bb200 2021-04-09T18:33:42.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3da2295700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0bb600 con 0x7f3d9c1a0840 2021-04-09T18:33:42.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3d93fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c192100 0x7f3d9c0babe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.812 7f3d93fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c192100 0x7f3d9c0babe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58972/0 (socket says 172.21.15.32:58972) 2021-04-09T18:33:42.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d93fff700 1 -- 172.21.15.32:0/2742997564 learned_addr learned my addr 172.21.15.32:0/2742997564 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:42.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3da08b2700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d9c0bb200 0x7f3d9c0c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d9bfff700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d9c1a0840 0x7f3d9c191ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d93fff700 1 -- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d9c1a0840 msgr2=0x7f3d9c191ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d93fff700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d9c1a0840 0x7f3d9c191ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d93fff700 1 -- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d9c0bb200 msgr2=0x7f3d9c0c0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:42.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d93fff700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d9c0bb200 0x7f3d9c0c0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:42.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d93fff700 1 -- 172.21.15.32:0/2742997564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c0c11a0 con 0x7f3d9c192100 2021-04-09T18:33:42.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d9bfff700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d9c1a0840 0x7f3d9c191ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:42.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d93fff700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c192100 0x7f3d9c0babe0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 rx=0x7f3d84006f10 tx=0x7f3d8400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:42.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d997fa700 1 -- 172.21.15.32:0/2742997564 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d8400bd80 con 0x7f3d9c192100 2021-04-09T18:33:42.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d997fa700 1 -- 172.21.15.32:0/2742997564 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d84000e20 con 0x7f3d9c192100 2021-04-09T18:33:42.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3da2295700 1 -- 172.21.15.32:0/2742997564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c0c11a0 con 0x7f3d9c192100 2021-04-09T18:33:42.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3d997fa700 1 -- 172.21.15.32:0/2742997564 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d84000bc0 con 0x7f3d9c192100 2021-04-09T18:33:42.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.813 7f3da2295700 1 -- 172.21.15.32:0/2742997564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c0c11a0 con 0x7f3d9c192100 2021-04-09T18:33:42.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.814 7f3d997fa700 1 -- 172.21.15.32:0/2742997564 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3d84009fe0 con 0x7f3d9c192100 2021-04-09T18:33:42.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.814 7f3d997fa700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d800188c0 0x7f3d8001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:42.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.814 7f3d997fa700 1 -- 172.21.15.32:0/2742997564 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3d84032790 con 0x7f3d9c192100 2021-04-09T18:33:42.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.814 7f3d9bfff700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d800188c0 0x7f3d8001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:42.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.814 7f3d9bfff700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d800188c0 0x7f3d8001adc0 crc :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:42.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.834 7f3da2295700 1 -- 172.21.15.32:0/2742997564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c0bb600 con 0x7f3d9c192100 2021-04-09T18:33:42.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:42.837 7f3d997fa700 1 -- 172.21.15.32:0/2742997564 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3d84000b80 con 0x7f3d9c192100 2021-04-09T18:33:43.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.022 7f3da2295700 1 -- 172.21.15.32:0/2742997564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3d9c0bb600 con 0x7f3d9c192100 2021-04-09T18:33:43.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.026 7f3d997fa700 1 -- 172.21.15.32:0/2742997564 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v48) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3d8401f6f0 con 0x7f3d9c192100 2021-04-09T18:33:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:33:43.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 -- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d800188c0 msgr2=0x7f3d8001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:43.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d800188c0 0x7f3d8001adc0 crc :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 -- 172.21.15.32:0/2742997564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c192100 msgr2=0x7f3d9c0babe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:43.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c192100 0x7f3d9c0babe0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 rx=0x7f3d84006f10 tx=0x7f3d8400b990).stop 2021-04-09T18:33:43.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 -- 172.21.15.32:0/2742997564 shutdown_connections 2021-04-09T18:33:43.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d800188c0 0x7f3d8001adc0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d9c192100 0x7f3d9c0babe0 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d9c0bb200 0x7f3d9c0c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 --2- 172.21.15.32:0/2742997564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d9c1a0840 0x7f3d9c191ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 -- 172.21.15.32:0/2742997564 >> 172.21.15.32:0/2742997564 conn(0x7f3d9c0adaf0 msgr2=0x7f3d9c0bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:43.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 -- 172.21.15.32:0/2742997564 shutdown_connections 2021-04-09T18:33:43.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.032 7f3da2295700 1 -- 172.21.15.32:0/2742997564 wait complete. 2021-04-09T18:33:43.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:43.231 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.192 7fda71604700 1 Processor -- start 2021-04-09T18:33:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.192 7fda71604700 1 -- start start 2021-04-09T18:33:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.192 7fda71604700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda6c10d010 0x7fda6c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:43.232 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.192 7fda71604700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fda6c0d1da0 0x7fda6c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.192 7fda71604700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c10d480 con 0x7fda6c0d82c0 2021-04-09T18:33:43.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.192 7fda71604700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6c0d8d30 con 0x7fda6c10d010 2021-04-09T18:33:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.192 7fda71604700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6c0d9220 con 0x7fda6c0d1da0 2021-04-09T18:33:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6b2bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda6c10d010 0x7fda6c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6b2bf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fda6c10d010 0x7fda6c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58978/0 (socket says 172.21.15.32:58978) 2021-04-09T18:33:43.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6b2bf700 1 -- 172.21.15.32:0/3042628316 learned_addr learned my addr 172.21.15.32:0/3042628316 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6aabe700 1 --2- 172.21.15.32:0/3042628316 >> v2:172.21.15.86:3300/0 conn(0x7fda6c0d1da0 0x7fda6c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6b2bf700 1 -- 172.21.15.32:0/3042628316 >> v2:172.21.15.86:3300/0 conn(0x7fda6c0d1da0 msgr2=0x7fda6c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6b2bf700 1 --2- 172.21.15.32:0/3042628316 >> v2:172.21.15.86:3300/0 conn(0x7fda6c0d1da0 0x7fda6c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6b2bf700 1 -- 172.21.15.32:0/3042628316 >> v1:172.21.15.86:6789/0 conn(0x7fda6c0d82c0 legacy=0x7fda6c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:43.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6b2bf700 1 -- 172.21.15.32:0/3042628316 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0d86f0 con 0x7fda6c10d010 2021-04-09T18:33:43.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6a2bd700 1 -- 172.21.15.32:0/3042628316 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373117391 0 0) 0x7fda64000d30 con 0x7fda6c0d82c0 2021-04-09T18:33:43.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6aabe700 1 --2- 172.21.15.32:0/3042628316 >> v2:172.21.15.86:3300/0 conn(0x7fda6c0d1da0 0x7fda6c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:43.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6b2bf700 1 --2- 172.21.15.32:0/3042628316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c10d010 0x7fda6c0d1780 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7fda5c006700 tx=0x7fda5c009b80).ready entity=mon.0 client_cookie=c1521405e1fbf0e0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:43.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6a2bd700 1 -- 172.21.15.32:0/3042628316 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5c00c0c0 con 0x7fda6c10d010 2021-04-09T18:33:43.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6a2bd700 1 -- 172.21.15.32:0/3042628316 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c00c440 con 0x7fda6c10d010 2021-04-09T18:33:43.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda6a2bd700 1 -- 172.21.15.32:0/3042628316 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5c00c920 con 0x7fda6c10d010 2021-04-09T18:33:43.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda71604700 1 -- 172.21.15.32:0/3042628316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c10d010 msgr2=0x7fda6c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:43.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.193 7fda71604700 1 --2- 172.21.15.32:0/3042628316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c10d010 0x7fda6c0d1780 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7fda5c006700 tx=0x7fda5c009b80).stop 2021-04-09T18:33:43.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.194 7fda71604700 1 -- 172.21.15.32:0/3042628316 shutdown_connections 2021-04-09T18:33:43.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.194 7fda71604700 1 --2- 172.21.15.32:0/3042628316 >> v2:172.21.15.86:3300/0 conn(0x7fda6c0d1da0 0x7fda6c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.194 7fda71604700 1 --2- 172.21.15.32:0/3042628316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c10d010 0x7fda6c0d1780 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.194 7fda71604700 1 -- 172.21.15.32:0/3042628316 >> 172.21.15.32:0/3042628316 conn(0x7fda6c0ce5f0 msgr2=0x7fda6c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:43.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.194 7fda71604700 1 -- 172.21.15.32:0/3042628316 shutdown_connections 2021-04-09T18:33:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.194 7fda71604700 1 -- 172.21.15.32:0/3042628316 wait complete. 2021-04-09T18:33:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.214 7fda71604700 1 Processor -- start 2021-04-09T18:33:43.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.214 7fda71604700 1 -- start start 2021-04-09T18:33:43.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda71604700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c1889c0 0x7fda6c09a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:43.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda71604700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c18e2c0 0x7fda6c1aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:43.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda71604700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1aaad0 0x7fda6c0a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:43.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda71604700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c1aaed0 con 0x7fda6c1889c0 2021-04-09T18:33:43.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda71604700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c1ab0f0 con 0x7fda6c1aaad0 2021-04-09T18:33:43.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda71604700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c0a3ed0 con 0x7fda6c18e2c0 2021-04-09T18:33:43.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c1889c0 0x7fda6c09a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:43.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6b2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c1889c0 0x7fda6c09a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58988/0 (socket says 172.21.15.32:58988) 2021-04-09T18:33:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6b2bf700 1 -- 172.21.15.32:0/2165715976 learned_addr learned my addr 172.21.15.32:0/2165715976 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6bac0700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1aaad0 0x7fda6c0a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6aabe700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c18e2c0 0x7fda6c1aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6b2bf700 1 -- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c18e2c0 msgr2=0x7fda6c1aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6b2bf700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c18e2c0 0x7fda6c1aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6b2bf700 1 -- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1aaad0 msgr2=0x7fda6c0a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:43.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6b2bf700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1aaad0 0x7fda6c0a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6b2bf700 1 -- 172.21.15.32:0/2165715976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0a40f0 con 0x7fda6c1889c0 2021-04-09T18:33:43.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.215 7fda6b2bf700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c1889c0 0x7fda6c09a3f0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 rx=0x7fda5c00b580 tx=0x7fda5c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:43.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.216 7fda53fff700 1 -- 172.21.15.32:0/2165715976 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5c00bd10 con 0x7fda6c1889c0 2021-04-09T18:33:43.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.216 7fda53fff700 1 -- 172.21.15.32:0/2165715976 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c002030 con 0x7fda6c1889c0 2021-04-09T18:33:43.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.216 7fda71604700 1 -- 172.21.15.32:0/2165715976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c0a40f0 con 0x7fda6c1889c0 2021-04-09T18:33:43.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.216 7fda53fff700 1 -- 172.21.15.32:0/2165715976 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5c002550 con 0x7fda6c1889c0 2021-04-09T18:33:43.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.216 7fda71604700 1 -- 172.21.15.32:0/2165715976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c0a40f0 con 0x7fda6c1889c0 2021-04-09T18:33:43.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.216 7fda53fff700 1 -- 172.21.15.32:0/2165715976 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fda5c002210 con 0x7fda6c1889c0 2021-04-09T18:33:43.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.216 7fda53fff700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda4c018930 0x7fda4c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:43.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.216 7fda53fff700 1 -- 172.21.15.32:0/2165715976 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fda5c032a30 con 0x7fda6c1889c0 2021-04-09T18:33:43.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.217 7fda6aabe700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda4c018930 0x7fda4c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:43.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.217 7fda6aabe700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda4c018930 0x7fda4c01adc0 crc :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:43.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.237 7fda71604700 1 -- 172.21.15.32:0/2165715976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c10d480 con 0x7fda6c1889c0 2021-04-09T18:33:43.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.240 7fda53fff700 1 -- 172.21.15.32:0/2165715976 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fda5c002300 con 0x7fda6c1889c0 2021-04-09T18:33:43.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.433 7fda71604700 1 -- 172.21.15.32:0/2165715976 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda6c10d480 con 0x7fda6c1889c0 2021-04-09T18:33:43.451 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.433 7fda53fff700 1 -- 172.21.15.32:0/2165715976 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 411 v411) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fda5c002030 con 0x7fda6c1889c0 2021-04-09T18:33:43.451 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 411 2021-04-09T18:33:43.452 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:43.452 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":411,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12519,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407},{"gid":12522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407},{"gid":59714,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407}],"filesystems":[{"mdsmap":{"epoch":411,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:41.372484","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 -- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda4c018930 msgr2=0x7fda4c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda4c018930 0x7fda4c01adc0 crc :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 -- 172.21.15.32:0/2165715976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c1889c0 msgr2=0x7fda6c09a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:43.454 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c1889c0 0x7fda6c09a3f0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 rx=0x7fda5c00b580 tx=0x7fda5c00b920).stop 2021-04-09T18:33:43.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 -- 172.21.15.32:0/2165715976 shutdown_connections 2021-04-09T18:33:43.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fda4c018930 0x7fda4c01adc0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda6c1aaad0 0x7fda6c0a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda6c18e2c0 0x7fda6c1aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 --2- 172.21.15.32:0/2165715976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda6c1889c0 0x7fda6c09a3f0 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 -- 172.21.15.32:0/2165715976 >> 172.21.15.32:0/2165715976 conn(0x7fda6c1a0db0 msgr2=0x7fda6c1a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 -- 172.21.15.32:0/2165715976 shutdown_connections 2021-04-09T18:33:43.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:43.436 7fda71604700 1 -- 172.21.15.32:0/2165715976 wait complete. 2021-04-09T18:33:43.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 411, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:33:39.367894', 'modified': '2021-04-09 18:33:41.372484', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12022}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12022': {'gid': 12022, 'name': 'b', 'rank': 0, 'incarnation': 408, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/154900016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 154900016}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 154900016}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:33:43.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:33:43.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:33:43.998 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:33:43.998 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:33:43.998 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:33:43.999 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 12022, 2021-04-09T18:33:43.999 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:33:44.000 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:33:44.000 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.6112792679999997, 2021-04-09T18:33:44.000 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 411, 2021-04-09T18:33:44.000 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 251, 2021-04-09T18:33:44.001 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 251, 2021-04-09T18:33:44.001 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10.056599502999999 2021-04-09T18:33:44.001 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:33:44.004 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12022, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.6112792679999997, "mdsmap_epoch": 411, "osdmap_epoch": 251, "osdmap_epoch_barrier": 251, "uptime": 10.056599502999999 } 2021-04-09T18:33:44.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:33:44.177 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.138 7f5e1ed0d700 1 Processor -- start 2021-04-09T18:33:44.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.138 7f5e1ed0d700 1 -- start start 2021-04-09T18:33:44.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1ed0d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5e180d6c80 0x7f5e180d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1ed0d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e180d2440 0x7f5e1810cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1ed0d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e180d7280 con 0x7f5e180cffa0 2021-04-09T18:33:44.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1ed0d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e180d7080 con 0x7f5e180d2440 2021-04-09T18:33:44.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1ed0d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e180d91e0 con 0x7f5e180d6c80 2021-04-09T18:33:44.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1d32a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e180d2440 0x7f5e1810cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1d32a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5e180d2440 0x7f5e1810cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58994/0 (socket says 172.21.15.32:58994) 2021-04-09T18:33:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1d32a700 1 -- 172.21.15.32:0/3945335890 learned_addr learned my addr 172.21.15.32:0/3945335890 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e17fff700 1 --2- 172.21.15.32:0/3945335890 >> v2:172.21.15.86:3300/0 conn(0x7f5e180d6c80 0x7f5e180d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1d32a700 1 -- 172.21.15.32:0/3945335890 >> v2:172.21.15.86:3300/0 conn(0x7f5e180d6c80 msgr2=0x7f5e180d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1d32a700 1 --2- 172.21.15.32:0/3945335890 >> v2:172.21.15.86:3300/0 conn(0x7f5e180d6c80 0x7f5e180d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1d32a700 1 -- 172.21.15.32:0/3945335890 >> v1:172.21.15.86:6789/0 conn(0x7f5e180cffa0 legacy=0x7f5e180d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:44.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1d32a700 1 -- 172.21.15.32:0/3945335890 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e180d2840 con 0x7f5e180d2440 2021-04-09T18:33:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.139 7f5e1d32a700 1 --2- 172.21.15.32:0/3945335890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e180d2440 0x7f5e1810cfd0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 rx=0x7f5e10006f00 tx=0x7f5e10009f10).ready entity=mon.0 client_cookie=726b79a41251d304 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e177fe700 1 -- 172.21.15.32:0/3945335890 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e1000b3b0 con 0x7f5e180d2440 2021-04-09T18:33:44.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e177fe700 1 -- 172.21.15.32:0/3945335890 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1000b730 con 0x7f5e180d2440 2021-04-09T18:33:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e177fe700 1 -- 172.21.15.32:0/3945335890 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e1000bc10 con 0x7f5e180d2440 2021-04-09T18:33:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e1ed0d700 1 -- 172.21.15.32:0/3945335890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e180d2440 msgr2=0x7f5e1810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e1ed0d700 1 --2- 172.21.15.32:0/3945335890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e180d2440 0x7f5e1810cfd0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 rx=0x7f5e10006f00 tx=0x7f5e10009f10).stop 2021-04-09T18:33:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e1ed0d700 1 -- 172.21.15.32:0/3945335890 shutdown_connections 2021-04-09T18:33:44.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e1ed0d700 1 --2- 172.21.15.32:0/3945335890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e180d2440 0x7f5e1810cfd0 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e1ed0d700 1 --2- 172.21.15.32:0/3945335890 >> v2:172.21.15.86:3300/0 conn(0x7f5e180d6c80 0x7f5e180d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e1ed0d700 1 -- 172.21.15.32:0/3945335890 >> 172.21.15.32:0/3945335890 conn(0x7f5e180ce5c0 msgr2=0x7f5e180cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e1ed0d700 1 -- 172.21.15.32:0/3945335890 shutdown_connections 2021-04-09T18:33:44.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.140 7f5e1ed0d700 1 -- 172.21.15.32:0/3945335890 wait complete. 2021-04-09T18:33:44.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.160 7f5e1ed0d700 1 Processor -- start 2021-04-09T18:33:44.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.160 7f5e1ed0d700 1 -- start start 2021-04-09T18:33:44.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1ed0d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e181929f0 0x7f5e18192df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1ed0d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e1808ddc0 0x7f5e1808e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1ed0d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e1808e7e0 0x7f5e1808c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1ed0d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e18193410 con 0x7f5e181929f0 2021-04-09T18:33:44.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1ed0d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e180a9140 con 0x7f5e1808ddc0 2021-04-09T18:33:44.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1ed0d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1808ce80 con 0x7f5e1808e7e0 2021-04-09T18:33:44.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1cb29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e181929f0 0x7f5e18192df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1cb29700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e181929f0 0x7f5e18192df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58998/0 (socket says 172.21.15.32:58998) 2021-04-09T18:33:44.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1cb29700 1 -- 172.21.15.32:0/4288521290 learned_addr learned my addr 172.21.15.32:0/4288521290 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:44.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e17fff700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e1808ddc0 0x7f5e1808e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.191 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.161 7f5e1d32a700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e1808e7e0 0x7f5e1808c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e1cb29700 1 -- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e1808e7e0 msgr2=0x7f5e1808c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e1cb29700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e1808e7e0 0x7f5e1808c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e1cb29700 1 -- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e1808ddc0 msgr2=0x7f5e1808e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e1cb29700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e1808ddc0 0x7f5e1808e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.192 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e1cb29700 1 -- 172.21.15.32:0/4288521290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1808d0a0 con 0x7f5e181929f0 2021-04-09T18:33:44.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e1cb29700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e181929f0 0x7f5e18192df0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7f5e0800b530 tx=0x7f5e0800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:44.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e157fa700 1 -- 172.21.15.32:0/4288521290 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e0800bcc0 con 0x7f5e181929f0 2021-04-09T18:33:44.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e157fa700 1 -- 172.21.15.32:0/4288521290 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e08006020 con 0x7f5e181929f0 2021-04-09T18:33:44.193 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1808d0a0 con 0x7f5e181929f0 2021-04-09T18:33:44.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e157fa700 1 -- 172.21.15.32:0/4288521290 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e080054a0 con 0x7f5e181929f0 2021-04-09T18:33:44.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.162 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1808d0a0 con 0x7f5e181929f0 2021-04-09T18:33:44.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.163 7f5e157fa700 1 -- 172.21.15.32:0/4288521290 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5e08006200 con 0x7f5e181929f0 2021-04-09T18:33:44.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.163 7f5e157fa700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5df80188d0 0x7f5df801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.163 7f5e157fa700 1 -- 172.21.15.32:0/4288521290 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5e080329f0 con 0x7f5e181929f0 2021-04-09T18:33:44.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.163 7f5e17fff700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5df80188d0 0x7f5df801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.195 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.163 7f5e17fff700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5df80188d0 0x7f5df801add0 crc :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:44.200 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.183 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e181b70c0 con 0x7f5e181929f0 2021-04-09T18:33:44.204 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.186 7f5e157fa700 1 -- 172.21.15.32:0/4288521290 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5e0800bcc0 con 0x7f5e181929f0 2021-04-09T18:33:44.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.375 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e180d7280 con 0x7f5e181929f0 2021-04-09T18:33:44.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.376 7f5e157fa700 1 -- 172.21.15.32:0/4288521290 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f5e08006020 con 0x7f5e181929f0 2021-04-09T18:33:44.394 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:44.394 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":251,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:33:40.369286","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:33:36.709256","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-04-09 18:33:38.131023","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"251","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:33:44.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5df80188d0 msgr2=0x7f5df801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5df80188d0 0x7f5df801add0 crc :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e181929f0 msgr2=0x7f5e18192df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e181929f0 0x7f5e18192df0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7f5e0800b530 tx=0x7f5e0800b8d0).stop 2021-04-09T18:33:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 shutdown_connections 2021-04-09T18:33:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5df80188d0 0x7f5df801add0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e1808e7e0 0x7f5e1808c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e1808ddc0 0x7f5e1808e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 --2- 172.21.15.32:0/4288521290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5e181929f0 0x7f5e18192df0 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 >> 172.21.15.32:0/4288521290 conn(0x7f5e18189b00 msgr2=0x7f5e181921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:44.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 shutdown_connections 2021-04-09T18:33:44.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.378 7f5e1ed0d700 1 -- 172.21.15.32:0/4288521290 wait complete. 2021-04-09T18:33:44.406 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 42, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:33:36.709256', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 43, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:33:38.131023', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '251', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:33:44.407 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:33:44.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.542 7f2e19965700 1 Processor -- start 2021-04-09T18:33:44.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.542 7f2e19965700 1 -- start start 2021-04-09T18:33:44.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e19965700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e1410d010 0x7f2e140d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e19965700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e140d1da0 0x7f2e140d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e19965700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1410d480 con 0x7f2e140d82c0 2021-04-09T18:33:44.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e19965700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1410e780 con 0x7f2e140d1da0 2021-04-09T18:33:44.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e19965700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140d8d30 con 0x7f2e1410d010 2021-04-09T18:33:44.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e12d73700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e140d1da0 0x7f2e140d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e13574700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e1410d010 0x7f2e140d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e13d75700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2e140d82c0 0x7f2e1410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38292/0 (socket says 172.21.15.32:38292) 2021-04-09T18:33:44.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e13d75700 1 -- 172.21.15.32:0/504718964 learned_addr learned my addr 172.21.15.32:0/504718964 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:44.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e13574700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e1410d010 0x7f2e140d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:38000/0 (socket says 172.21.15.32:38000) 2021-04-09T18:33:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.543 7f2e12d73700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e140d1da0 0x7f2e140d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:50768/0 (socket says 172.21.15.32:50768) 2021-04-09T18:33:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e12572700 1 -- 172.21.15.32:0/504718964 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153302019 0 0) 0x7f2e0c001440 con 0x7f2e140d82c0 2021-04-09T18:33:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e13574700 1 -- 172.21.15.32:0/504718964 >> v2:172.21.15.86:3300/0 conn(0x7f2e140d1da0 msgr2=0x7f2e140d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:44.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e13574700 1 --2- 172.21.15.32:0/504718964 >> v2:172.21.15.86:3300/0 conn(0x7f2e140d1da0 0x7f2e140d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e13574700 1 -- 172.21.15.32:0/504718964 >> v1:172.21.15.86:6789/0 conn(0x7f2e140d82c0 legacy=0x7f2e1410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e13574700 1 -- 172.21.15.32:0/504718964 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e140d9220 con 0x7f2e1410d010 2021-04-09T18:33:44.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e12d73700 1 --2- 172.21.15.32:0/504718964 >> v2:172.21.15.86:3300/0 conn(0x7f2e140d1da0 0x7f2e140d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:44.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e13574700 1 --2- 172.21.15.32:0/504718964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e1410d010 0x7f2e140d1780 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f2e04006650 tx=0x7f2e04009a80).ready entity=mon.2 client_cookie=3032a1915750dd4c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:44.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e12572700 1 -- 172.21.15.32:0/504718964 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e040009b0 con 0x7f2e1410d010 2021-04-09T18:33:44.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e12572700 1 -- 172.21.15.32:0/504718964 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0400c0c0 con 0x7f2e1410d010 2021-04-09T18:33:44.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e12572700 1 -- 172.21.15.32:0/504718964 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e0400c510 con 0x7f2e1410d010 2021-04-09T18:33:44.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e19965700 1 -- 172.21.15.32:0/504718964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e1410d010 msgr2=0x7f2e140d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.544 7f2e19965700 1 --2- 172.21.15.32:0/504718964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e1410d010 0x7f2e140d1780 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f2e04006650 tx=0x7f2e04009a80).stop 2021-04-09T18:33:44.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.545 7f2e19965700 1 -- 172.21.15.32:0/504718964 shutdown_connections 2021-04-09T18:33:44.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.545 7f2e19965700 1 --2- 172.21.15.32:0/504718964 >> v2:172.21.15.86:3300/0 conn(0x7f2e140d1da0 0x7f2e140d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.545 7f2e19965700 1 --2- 172.21.15.32:0/504718964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e1410d010 0x7f2e140d1780 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.545 7f2e19965700 1 -- 172.21.15.32:0/504718964 >> 172.21.15.32:0/504718964 conn(0x7f2e140ce5f0 msgr2=0x7f2e140d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:44.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.545 7f2e19965700 1 -- 172.21.15.32:0/504718964 shutdown_connections 2021-04-09T18:33:44.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.545 7f2e19965700 1 -- 172.21.15.32:0/504718964 wait complete. 2021-04-09T18:33:44.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.565 7f2e19965700 1 Processor -- start 2021-04-09T18:33:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.565 7f2e19965700 1 -- start start 2021-04-09T18:33:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e19965700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e140a0a10 0x7f2e140bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e19965700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1418ad00 0x7f2e1419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e19965700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e141a03f0 0x7f2e14095240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e19965700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1418b100 con 0x7f2e140a0a10 2021-04-09T18:33:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e19965700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14095860 con 0x7f2e1418ad00 2021-04-09T18:33:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e19965700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14095a80 con 0x7f2e141a03f0 2021-04-09T18:33:44.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e140a0a10 0x7f2e140bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e140a0a10 0x7f2e140bf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59012/0 (socket says 172.21.15.32:59012) 2021-04-09T18:33:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13574700 1 -- 172.21.15.32:0/2313019303 learned_addr learned my addr 172.21.15.32:0/2313019303 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e12d73700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1418ad00 0x7f2e1419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13d75700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e141a03f0 0x7f2e14095240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13574700 1 -- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e141a03f0 msgr2=0x7f2e14095240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13574700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e141a03f0 0x7f2e14095240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13574700 1 -- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1418ad00 msgr2=0x7f2e1419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13574700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1418ad00 0x7f2e1419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13574700 1 -- 172.21.15.32:0/2313019303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e14189250 con 0x7f2e140a0a10 2021-04-09T18:33:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2e13574700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e140a0a10 0x7f2e140bf870 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7f2e04009650 tx=0x7f2e0400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2dfbfff700 1 -- 172.21.15.32:0/2313019303 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e040093a0 con 0x7f2e140a0a10 2021-04-09T18:33:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.566 7f2dfbfff700 1 -- 172.21.15.32:0/2313019303 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0400bd50 con 0x7f2e140a0a10 2021-04-09T18:33:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.567 7f2e19965700 1 -- 172.21.15.32:0/2313019303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e14189250 con 0x7f2e140a0a10 2021-04-09T18:33:44.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.567 7f2dfbfff700 1 -- 172.21.15.32:0/2313019303 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e040052e0 con 0x7f2e140a0a10 2021-04-09T18:33:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.567 7f2e19965700 1 -- 172.21.15.32:0/2313019303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e14189250 con 0x7f2e140a0a10 2021-04-09T18:33:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.567 7f2dfbfff700 1 -- 172.21.15.32:0/2313019303 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2e04006030 con 0x7f2e140a0a10 2021-04-09T18:33:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.567 7f2dfbfff700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2df4018930 0x7f2df401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.567 7f2dfbfff700 1 -- 172.21.15.32:0/2313019303 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2e040329e0 con 0x7f2e140a0a10 2021-04-09T18:33:44.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.567 7f2e12d73700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2df4018930 0x7f2df401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.568 7f2e12d73700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2df4018930 0x7f2df401adc0 crc :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:44.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.587 7f2e19965700 1 -- 172.21.15.32:0/2313019303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e14093d50 con 0x7f2e140a0a10 2021-04-09T18:33:44.608 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.591 7f2dfbfff700 1 -- 172.21.15.32:0/2313019303 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2e040009c0 con 0x7f2e140a0a10 2021-04-09T18:33:44.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.771 7f2e19965700 1 -- 172.21.15.32:0/2313019303 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2e1410d480 con 0x7f2e140a0a10 2021-04-09T18:33:44.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.771 7f2dfbfff700 1 -- 172.21.15.32:0/2313019303 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v251) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2e0400bd50 con 0x7f2e140a0a10 2021-04-09T18:33:44.789 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:33:44.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 -- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2df4018930 msgr2=0x7f2df401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2df4018930 0x7f2df401adc0 crc :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 -- 172.21.15.32:0/2313019303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e140a0a10 msgr2=0x7f2e140bf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e140a0a10 0x7f2e140bf870 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7f2e04009650 tx=0x7f2e0400b960).stop 2021-04-09T18:33:44.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 -- 172.21.15.32:0/2313019303 shutdown_connections 2021-04-09T18:33:44.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2df4018930 0x7f2df401adc0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e141a03f0 0x7f2e14095240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1418ad00 0x7f2e1419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 --2- 172.21.15.32:0/2313019303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e140a0a10 0x7f2e140bf870 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 -- 172.21.15.32:0/2313019303 >> 172.21.15.32:0/2313019303 conn(0x7f2e140975f0 msgr2=0x7f2e14096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 -- 172.21.15.32:0/2313019303 shutdown_connections 2021-04-09T18:33:44.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.774 7f2e19965700 1 -- 172.21.15.32:0/2313019303 wait complete. 2021-04-09T18:33:44.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:33:44.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.936 7fcd77c63700 1 Processor -- start 2021-04-09T18:33:44.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.936 7fcd77c63700 1 -- start start 2021-04-09T18:33:44.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd77c63700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcd7010d010 0x7fcd700d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd77c63700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700d1da0 0x7fcd700d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd77c63700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcd700d82c0 0x7fcd7010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd77c63700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd700d8d30 con 0x7fcd700d82c0 2021-04-09T18:33:44.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd77c63700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd700d9220 con 0x7fcd7010d010 2021-04-09T18:33:44.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd77c63700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7010d480 con 0x7fcd700d1da0 2021-04-09T18:33:44.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd76280700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcd700d82c0 0x7fcd7010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd76280700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fcd700d82c0 0x7fcd7010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59022/0 (socket says 172.21.15.32:59022) 2021-04-09T18:33:44.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd76280700 1 -- 172.21.15.32:0/1870392549 learned_addr learned my addr 172.21.15.32:0/1870392549 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:44.978 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd75a7f700 1 --2- 172.21.15.32:0/1870392549 >> v2:172.21.15.86:3300/0 conn(0x7fcd7010d010 0x7fcd700d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.937 7fcd7527e700 1 --2- 172.21.15.32:0/1870392549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700d1da0 0x7fcd700d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd76280700 1 -- 172.21.15.32:0/1870392549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700d1da0 msgr2=0x7fcd700d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:44.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd76280700 1 --2- 172.21.15.32:0/1870392549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700d1da0 0x7fcd700d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd76280700 1 -- 172.21.15.32:0/1870392549 >> v2:172.21.15.86:3300/0 conn(0x7fcd7010d010 msgr2=0x7fcd700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:44.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd76280700 1 --2- 172.21.15.32:0/1870392549 >> v2:172.21.15.86:3300/0 conn(0x7fcd7010d010 0x7fcd700d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd76280700 1 -- 172.21.15.32:0/1870392549 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd700d86c0 con 0x7fcd700d82c0 2021-04-09T18:33:44.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd76280700 1 --2- 172.21.15.32:0/1870392549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd700d82c0 0x7fcd7010da00 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7fcd68006b80 tx=0x7fcd68009b10).ready entity=mon.0 client_cookie=7d91de967113240d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:44.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd74a7d700 1 -- 172.21.15.32:0/1870392549 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd680022e0 con 0x7fcd700d82c0 2021-04-09T18:33:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd74a7d700 1 -- 172.21.15.32:0/1870392549 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd68002620 con 0x7fcd700d82c0 2021-04-09T18:33:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd74a7d700 1 -- 172.21.15.32:0/1870392549 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd68002b00 con 0x7fcd700d82c0 2021-04-09T18:33:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd77c63700 1 -- 172.21.15.32:0/1870392549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd700d82c0 msgr2=0x7fcd7010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.938 7fcd77c63700 1 --2- 172.21.15.32:0/1870392549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd700d82c0 0x7fcd7010da00 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7fcd68006b80 tx=0x7fcd68009b10).stop 2021-04-09T18:33:44.981 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.939 7fcd77c63700 1 -- 172.21.15.32:0/1870392549 shutdown_connections 2021-04-09T18:33:44.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.939 7fcd77c63700 1 --2- 172.21.15.32:0/1870392549 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd700d82c0 0x7fcd7010da00 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.939 7fcd77c63700 1 --2- 172.21.15.32:0/1870392549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700d1da0 0x7fcd700d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.939 7fcd77c63700 1 --2- 172.21.15.32:0/1870392549 >> v2:172.21.15.86:3300/0 conn(0x7fcd7010d010 0x7fcd700d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.939 7fcd77c63700 1 -- 172.21.15.32:0/1870392549 >> 172.21.15.32:0/1870392549 conn(0x7fcd700ce5f0 msgr2=0x7fcd700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:44.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.939 7fcd77c63700 1 -- 172.21.15.32:0/1870392549 shutdown_connections 2021-04-09T18:33:44.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.939 7fcd77c63700 1 -- 172.21.15.32:0/1870392549 wait complete. 2021-04-09T18:33:44.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.959 7fcd77c63700 1 Processor -- start 2021-04-09T18:33:44.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.959 7fcd77c63700 1 -- start start 2021-04-09T18:33:44.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd77c63700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700c3c00 0x7fcd7009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd77c63700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd70095240 0x7fcd701a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd77c63700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd701a1270 0x7fcd700b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd77c63700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7009cda0 con 0x7fcd701a1270 2021-04-09T18:33:44.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd77c63700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd700c3680 con 0x7fcd70095240 2021-04-09T18:33:44.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd77c63700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd70095640 con 0x7fcd700c3c00 2021-04-09T18:33:44.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd76280700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd701a1270 0x7fcd700b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd76280700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd701a1270 0x7fcd700b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59026/0 (socket says 172.21.15.32:59026) 2021-04-09T18:33:44.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd76280700 1 -- 172.21.15.32:0/2377530320 learned_addr learned my addr 172.21.15.32:0/2377530320 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:44.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd7527e700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd70095240 0x7fcd701a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd75a7f700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700c3c00 0x7fcd7009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd76280700 1 -- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700c3c00 msgr2=0x7fcd7009c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd76280700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700c3c00 0x7fcd7009c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd76280700 1 -- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd70095240 msgr2=0x7fcd701a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:44.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd76280700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd70095240 0x7fcd701a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:44.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd76280700 1 -- 172.21.15.32:0/2377530320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd701a1670 con 0x7fcd701a1270 2021-04-09T18:33:44.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd76280700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd701a1270 0x7fcd700b0cf0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 rx=0x7fcd68006f30 tx=0x7fcd6800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:44.990 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd667fc700 1 -- 172.21.15.32:0/2377530320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd6800bcd0 con 0x7fcd701a1270 2021-04-09T18:33:44.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd667fc700 1 -- 172.21.15.32:0/2377530320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd68000940 con 0x7fcd701a1270 2021-04-09T18:33:44.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd701a1670 con 0x7fcd701a1270 2021-04-09T18:33:44.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.960 7fcd667fc700 1 -- 172.21.15.32:0/2377530320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd680054a0 con 0x7fcd701a1270 2021-04-09T18:33:44.991 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.961 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd701a1670 con 0x7fcd701a1270 2021-04-09T18:33:44.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.961 7fcd667fc700 1 -- 172.21.15.32:0/2377530320 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcd68000b20 con 0x7fcd701a1270 2021-04-09T18:33:44.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.961 7fcd667fc700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd58018930 0x7fcd5801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:44.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.961 7fcd667fc700 1 -- 172.21.15.32:0/2377530320 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcd68032a00 con 0x7fcd701a1270 2021-04-09T18:33:44.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.961 7fcd75a7f700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd58018930 0x7fcd5801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:44.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.962 7fcd75a7f700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd58018930 0x7fcd5801ae10 crc :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:44.998 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.981 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd70093d50 con 0x7fcd701a1270 2021-04-09T18:33:45.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:44.985 7fcd667fc700 1 -- 172.21.15.32:0/2377530320 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcd6800bcd0 con 0x7fcd701a1270 2021-04-09T18:33:45.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.165 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fcd7010d480 con 0x7fcd701a1270 2021-04-09T18:33:45.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.165 7fcd667fc700 1 -- 172.21.15.32:0/2377530320 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v251) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcd68000940 con 0x7fcd701a1270 2021-04-09T18:33:45.183 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:33:45.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.167 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd58018930 msgr2=0x7fcd5801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.167 7fcd77c63700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd58018930 0x7fcd5801ae10 crc :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.167 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd701a1270 msgr2=0x7fcd700b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.167 7fcd77c63700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd701a1270 0x7fcd700b0cf0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 rx=0x7fcd68006f30 tx=0x7fcd6800b8e0).stop 2021-04-09T18:33:45.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.167 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 shutdown_connections 2021-04-09T18:33:45.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.167 7fcd77c63700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcd58018930 0x7fcd5801ae10 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.168 7fcd77c63700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcd701a1270 0x7fcd700b0cf0 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.168 7fcd77c63700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd70095240 0x7fcd701a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.168 7fcd77c63700 1 --2- 172.21.15.32:0/2377530320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd700c3c00 0x7fcd7009c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.189 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.168 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 >> 172.21.15.32:0/2377530320 conn(0x7fcd700975f0 msgr2=0x7fcd70096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:45.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.168 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 shutdown_connections 2021-04-09T18:33:45.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.168 7fcd77c63700 1 -- 172.21.15.32:0/2377530320 wait complete. 2021-04-09T18:33:45.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:33:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.329 7f1a63acf700 1 Processor -- start 2021-04-09T18:33:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.329 7f1a63acf700 1 -- start start 2021-04-09T18:33:45.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.330 7f1a63acf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a5c0cffc0 0x7f1a5c0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.330 7f1a63acf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0d6ca0 0x7f1a5c10cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.330 7f1a63acf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c0d72a0 con 0x7f1a5c0d7820 2021-04-09T18:33:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.330 7f1a63acf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0d70a0 con 0x7f1a5c0cffc0 2021-04-09T18:33:45.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.330 7f1a63acf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0d9200 con 0x7f1a5c0d6ca0 2021-04-09T18:33:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.330 7f1a618eb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1a5c0cffc0 0x7f1a5c0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.330 7f1a610ea700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1a5c0d7820 0x7f1a5c0d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40392/0 (socket says 172.21.15.32:40392) 2021-04-09T18:33:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.330 7f1a610ea700 1 -- 172.21.15.32:0/2481187731 learned_addr learned my addr 172.21.15.32:0/2481187731 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.330 7f1a620ec700 1 --2- 172.21.15.32:0/2481187731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0d6ca0 0x7f1a5c10cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a608e9700 1 -- 172.21.15.32:0/2481187731 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802930740 0 0) 0x7f1a4c0009b0 con 0x7f1a5c0d7820 2021-04-09T18:33:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a608e9700 1 -- 172.21.15.32:0/2481187731 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a48002fe0 con 0x7f1a5c0d7820 2021-04-09T18:33:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a618eb700 1 -- 172.21.15.32:0/2481187731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0d6ca0 msgr2=0x7f1a5c10cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a618eb700 1 --2- 172.21.15.32:0/2481187731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0d6ca0 0x7f1a5c10cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.372 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a618eb700 1 -- 172.21.15.32:0/2481187731 >> v1:172.21.15.32:6789/0 conn(0x7f1a5c0d7820 legacy=0x7f1a5c0d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a618eb700 1 -- 172.21.15.32:0/2481187731 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0d8410 con 0x7f1a5c0cffc0 2021-04-09T18:33:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a618eb700 1 --2- 172.21.15.32:0/2481187731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c0cffc0 0x7f1a5c0d2460 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7f1a58006650 tx=0x7f1a58009a80).ready entity=mon.0 client_cookie=aa51984b7c876438 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a608e9700 1 -- 172.21.15.32:0/2481187731 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a580009b0 con 0x7f1a5c0cffc0 2021-04-09T18:33:45.373 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a608e9700 1 -- 172.21.15.32:0/2481187731 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5800c0c0 con 0x7f1a5c0cffc0 2021-04-09T18:33:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a608e9700 1 -- 172.21.15.32:0/2481187731 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a5800c510 con 0x7f1a5c0cffc0 2021-04-09T18:33:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a63acf700 1 -- 172.21.15.32:0/2481187731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c0cffc0 msgr2=0x7f1a5c0d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.331 7f1a63acf700 1 --2- 172.21.15.32:0/2481187731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c0cffc0 0x7f1a5c0d2460 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7f1a58006650 tx=0x7f1a58009a80).stop 2021-04-09T18:33:45.374 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.332 7f1a63acf700 1 -- 172.21.15.32:0/2481187731 shutdown_connections 2021-04-09T18:33:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.332 7f1a63acf700 1 --2- 172.21.15.32:0/2481187731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0d6ca0 0x7f1a5c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.332 7f1a63acf700 1 --2- 172.21.15.32:0/2481187731 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c0cffc0 0x7f1a5c0d2460 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.332 7f1a63acf700 1 -- 172.21.15.32:0/2481187731 >> 172.21.15.32:0/2481187731 conn(0x7f1a5c0ce5e0 msgr2=0x7f1a5c0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.332 7f1a63acf700 1 -- 172.21.15.32:0/2481187731 shutdown_connections 2021-04-09T18:33:45.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.332 7f1a63acf700 1 -- 172.21.15.32:0/2481187731 wait complete. 2021-04-09T18:33:45.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.352 7f1a63acf700 1 Processor -- start 2021-04-09T18:33:45.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.352 7f1a63acf700 1 -- start start 2021-04-09T18:33:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a63acf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c1b82c0 0x7f1a5c0b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a63acf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a5c1b86c0 0x7f1a5c0b9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a63acf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0b9f60 0x7f1a5c0c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a63acf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0b93c0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a63acf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0ba360 con 0x7f1a5c1b86c0 2021-04-09T18:33:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a63acf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c1b8db0 con 0x7f1a5c0b9f60 2021-04-09T18:33:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a618eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c1b82c0 0x7f1a5c0b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.381 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a618eb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c1b82c0 0x7f1a5c0b89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59038/0 (socket says 172.21.15.32:59038) 2021-04-09T18:33:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a618eb700 1 -- 172.21.15.32:0/805277377 learned_addr learned my addr 172.21.15.32:0/805277377 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a610ea700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a5c1b86c0 0x7f1a5c0b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a620ec700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0b9f60 0x7f1a5c0c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a618eb700 1 -- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0b9f60 msgr2=0x7f1a5c0c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.382 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a618eb700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0b9f60 0x7f1a5c0c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a618eb700 1 -- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a5c1b86c0 msgr2=0x7f1a5c0b9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a618eb700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a5c1b86c0 0x7f1a5c0b9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a618eb700 1 -- 172.21.15.32:0/805277377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0b8fc0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a618eb700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c1b82c0 0x7f1a5c0b89a0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 rx=0x7f1a58009630 tx=0x7f1a5800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a527fc700 1 -- 172.21.15.32:0/805277377 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a580093a0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a527fc700 1 -- 172.21.15.32:0/805277377 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5800bd70 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a63acf700 1 -- 172.21.15.32:0/805277377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c0b8fc0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.353 7f1a527fc700 1 -- 172.21.15.32:0/805277377 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a580054d0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.354 7f1a63acf700 1 -- 172.21.15.32:0/805277377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c0b8fc0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.354 7f1a527fc700 1 -- 172.21.15.32:0/805277377 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1a580054f0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.354 7f1a527fc700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a44018930 0x7f1a4401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.354 7f1a527fc700 1 -- 172.21.15.32:0/805277377 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1a58032a00 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.354 7f1a610ea700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a44018930 0x7f1a4401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.355 7f1a610ea700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a44018930 0x7f1a4401ae10 crc :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:45.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.374 7f1a63acf700 1 -- 172.21.15.32:0/805277377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c1b70c0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.378 7f1a527fc700 1 -- 172.21.15.32:0/805277377 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1a580052e0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.560 7f1a63acf700 1 -- 172.21.15.32:0/805277377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f1a5c0d72a0 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.560 7f1a527fc700 1 -- 172.21.15.32:0/805277377 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v251) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1a5800bd70 con 0x7f1a5c1b82c0 2021-04-09T18:33:45.578 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:33:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 -- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a44018930 msgr2=0x7f1a4401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a44018930 0x7f1a4401ae10 crc :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 -- 172.21.15.32:0/805277377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c1b82c0 msgr2=0x7f1a5c0b89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c1b82c0 0x7f1a5c0b89a0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 rx=0x7f1a58009630 tx=0x7f1a5800b980).stop 2021-04-09T18:33:45.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 -- 172.21.15.32:0/805277377 shutdown_connections 2021-04-09T18:33:45.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1a44018930 0x7f1a4401ae10 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a5c0b9f60 0x7f1a5c0c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a5c1b86c0 0x7f1a5c0b9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 --2- 172.21.15.32:0/805277377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a5c1b82c0 0x7f1a5c0b89a0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 -- 172.21.15.32:0/805277377 >> 172.21.15.32:0/805277377 conn(0x7f1a5c0a06a0 msgr2=0x7f1a5c08fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 -- 172.21.15.32:0/805277377 shutdown_connections 2021-04-09T18:33:45.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.563 7f1a63acf700 1 -- 172.21.15.32:0/805277377 wait complete. 2021-04-09T18:33:45.591 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:33:45.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:45.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.723 7f8b784d4700 1 Processor -- start 2021-04-09T18:33:45.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.723 7f8b784d4700 1 -- start start 2021-04-09T18:33:45.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b784d4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b700d7ca0 0x7f8b700d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b784d4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b7010da00 0x7f8b7010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b784d4700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b7010d480 con 0x7f8b7010d010 2021-04-09T18:33:45.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b784d4700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b700d9220 con 0x7f8b7010da00 2021-04-09T18:33:45.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b784d4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b700d86c0 con 0x7f8b700d7ca0 2021-04-09T18:33:45.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b76af1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b7010da00 0x7f8b7010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b75aef700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b700d7ca0 0x7f8b700d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b76af1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8b7010da00 0x7f8b7010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59046/0 (socket says 172.21.15.32:59046) 2021-04-09T18:33:45.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b76af1700 1 -- 172.21.15.32:0/1904022479 learned_addr learned my addr 172.21.15.32:0/1904022479 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:45.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b762f0700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8b7010d010 0x7f8b700d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38328/0 (socket says 172.21.15.32:38328) 2021-04-09T18:33:45.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b76af1700 1 -- 172.21.15.32:0/1904022479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b700d7ca0 msgr2=0x7f8b700d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:45.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b752ee700 1 -- 172.21.15.32:0/1904022479 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402150741 0 0) 0x7f8b6c000c60 con 0x7f8b7010d010 2021-04-09T18:33:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b76af1700 1 --2- 172.21.15.32:0/1904022479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b700d7ca0 0x7f8b700d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b76af1700 1 -- 172.21.15.32:0/1904022479 >> v1:172.21.15.86:6789/0 conn(0x7f8b7010d010 legacy=0x7f8b700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b76af1700 1 -- 172.21.15.32:0/1904022479 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b70112100 con 0x7f8b7010da00 2021-04-09T18:33:45.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b75aef700 1 --2- 172.21.15.32:0/1904022479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b700d7ca0 0x7f8b700d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.724 7f8b76af1700 1 --2- 172.21.15.32:0/1904022479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7010da00 0x7f8b7010de00 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 rx=0x7f8b68006f00 tx=0x7f8b68009f10).ready entity=mon.0 client_cookie=4f7174f885a8516b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.725 7f8b752ee700 1 -- 172.21.15.32:0/1904022479 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b6800b3b0 con 0x7f8b7010da00 2021-04-09T18:33:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.725 7f8b752ee700 1 -- 172.21.15.32:0/1904022479 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6800b730 con 0x7f8b7010da00 2021-04-09T18:33:45.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.725 7f8b752ee700 1 -- 172.21.15.32:0/1904022479 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b6800bc10 con 0x7f8b7010da00 2021-04-09T18:33:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.725 7f8b784d4700 1 -- 172.21.15.32:0/1904022479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7010da00 msgr2=0x7f8b7010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.725 7f8b784d4700 1 --2- 172.21.15.32:0/1904022479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7010da00 0x7f8b7010de00 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 rx=0x7f8b68006f00 tx=0x7f8b68009f10).stop 2021-04-09T18:33:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.725 7f8b784d4700 1 -- 172.21.15.32:0/1904022479 shutdown_connections 2021-04-09T18:33:45.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.725 7f8b784d4700 1 --2- 172.21.15.32:0/1904022479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7010da00 0x7f8b7010de00 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.725 7f8b784d4700 1 --2- 172.21.15.32:0/1904022479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b700d7ca0 0x7f8b700d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.725 7f8b784d4700 1 -- 172.21.15.32:0/1904022479 >> 172.21.15.32:0/1904022479 conn(0x7f8b700ce5f0 msgr2=0x7f8b700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.726 7f8b784d4700 1 -- 172.21.15.32:0/1904022479 shutdown_connections 2021-04-09T18:33:45.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.726 7f8b784d4700 1 -- 172.21.15.32:0/1904022479 wait complete. 2021-04-09T18:33:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b784d4700 1 Processor -- start 2021-04-09T18:33:45.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b784d4700 1 -- start start 2021-04-09T18:33:45.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b784d4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b700b42d0 0x7f8b70183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b784d4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b701ac090 0x7f8b701ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b784d4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b70184d30 0x7f8b70185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b784d4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b700a1a90 con 0x7f8b70184d30 2021-04-09T18:33:45.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b784d4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b701acab0 con 0x7f8b700b42d0 2021-04-09T18:33:45.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b784d4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70185a00 con 0x7f8b701ac090 2021-04-09T18:33:45.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b76af1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b70184d30 0x7f8b70185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b76af1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b70184d30 0x7f8b70185130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59052/0 (socket says 172.21.15.32:59052) 2021-04-09T18:33:45.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b76af1700 1 -- 172.21.15.32:0/3054030200 learned_addr learned my addr 172.21.15.32:0/3054030200 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b762f0700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b700b42d0 0x7f8b70183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.746 7f8b75aef700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b701ac090 0x7f8b701ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b76af1700 1 -- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b701ac090 msgr2=0x7f8b701ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b76af1700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b701ac090 0x7f8b701ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b76af1700 1 -- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b700b42d0 msgr2=0x7f8b70183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b76af1700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b700b42d0 0x7f8b70183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b76af1700 1 -- 172.21.15.32:0/3054030200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b70185750 con 0x7f8b70184d30 2021-04-09T18:33:45.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b76af1700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b70184d30 0x7f8b70185130 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f8b6800b9b0 tx=0x7f8b6800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b66ffd700 1 -- 172.21.15.32:0/3054030200 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b68009450 con 0x7f8b70184d30 2021-04-09T18:33:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b66ffd700 1 -- 172.21.15.32:0/3054030200 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6800c2f0 con 0x7f8b70184d30 2021-04-09T18:33:45.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b66ffd700 1 -- 172.21.15.32:0/3054030200 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b6800c700 con 0x7f8b70184d30 2021-04-09T18:33:45.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b70185750 con 0x7f8b70184d30 2021-04-09T18:33:45.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.747 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b70185750 con 0x7f8b70184d30 2021-04-09T18:33:45.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.748 7f8b66ffd700 1 -- 172.21.15.32:0/3054030200 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8b68001270 con 0x7f8b70184d30 2021-04-09T18:33:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.748 7f8b66ffd700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b580188d0 0x7f8b5801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:45.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.748 7f8b66ffd700 1 -- 172.21.15.32:0/3054030200 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8b68032d80 con 0x7f8b70184d30 2021-04-09T18:33:45.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.748 7f8b762f0700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b580188d0 0x7f8b5801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:45.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.748 7f8b762f0700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b580188d0 0x7f8b5801add0 crc :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:45.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.768 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b70182d10 con 0x7f8b70184d30 2021-04-09T18:33:45.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.771 7f8b66ffd700 1 -- 172.21.15.32:0/3054030200 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8b68000f90 con 0x7f8b70184d30 2021-04-09T18:33:45.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.964 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b7010d480 con 0x7f8b70184d30 2021-04-09T18:33:45.982 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.965 7f8b66ffd700 1 -- 172.21.15.32:0/3054030200 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 411 v411) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f8b68001270 con 0x7f8b70184d30 2021-04-09T18:33:45.983 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 411 2021-04-09T18:33:45.984 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:45.984 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":411,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12519,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407},{"gid":12522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407},{"gid":59714,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407}],"filesystems":[{"mdsmap":{"epoch":411,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:41.372484","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:45.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b580188d0 msgr2=0x7f8b5801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b580188d0 0x7f8b5801add0 crc :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b70184d30 msgr2=0x7f8b70185130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:45.987 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b70184d30 0x7f8b70185130 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f8b6800b9b0 tx=0x7f8b6800bd50).stop 2021-04-09T18:33:45.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 shutdown_connections 2021-04-09T18:33:45.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8b580188d0 0x7f8b5801add0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b70184d30 0x7f8b70185130 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.988 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b701ac090 0x7f8b701ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 --2- 172.21.15.32:0/3054030200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b700b42d0 0x7f8b70183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 >> 172.21.15.32:0/3054030200 conn(0x7f8b7019b890 msgr2=0x7f8b701a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 shutdown_connections 2021-04-09T18:33:45.989 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:45.967 7f8b784d4700 1 -- 172.21.15.32:0/3054030200 wait complete. 2021-04-09T18:33:46.398 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 411, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:33:39.367894', 'modified': '2021-04-09 18:33:41.372484', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12022}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12022': {'gid': 12022, 'name': 'b', 'rank': 0, 'incarnation': 408, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/154900016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 154900016}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 154900016}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:33:46.398 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:33:46.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:33:46.526 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:33:46.526 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:33:46.526 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:33:46.527 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 12022, 2021-04-09T18:33:46.527 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:33:46.527 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:33:46.527 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 7.1395757880000001, 2021-04-09T18:33:46.528 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 411, 2021-04-09T18:33:46.528 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 251, 2021-04-09T18:33:46.528 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 251, 2021-04-09T18:33:46.528 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12.5848947 2021-04-09T18:33:46.528 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:33:46.532 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12022, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.1395757880000001, "mdsmap_epoch": 411, "osdmap_epoch": 251, "osdmap_epoch_barrier": 251, "uptime": 12.5848947 } 2021-04-09T18:33:46.533 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:33:46.533 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:33:46.533 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:33:46.534 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:46.534 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:33:46.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:33:46.639 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:33:46.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:33:46.697 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:33:46.698 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:33:46.698 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:33:46.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:46.700 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:33:46.743 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:33:46.744 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:33:46.747 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:33:46.808 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:33:46.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:33:46.868 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:33:46.868 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:33:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:33:46.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:46.870 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:33:46.897 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:33:46.879 7f28741bbf80 -1 init, newargv = 0x55b1171d92b0 newargc=9ceph-fuse[ 2021-04-09T18:33:46.897 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:105578]: starting ceph client 2021-04-09T18:33:46.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[105578]: starting fuse 2021-04-09T18:33:46.916 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:33:46.917 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:33:46.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:46.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:33:46.919 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:33:46.920 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:33:46.920 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:33:46.921 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:33:46.922 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:33:46.923 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:33:46.923 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:33:46.924 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:33:46.924 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:33:46.924 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:33:46.925 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:33:46.925 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:33:46.926 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:33:46.926 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:33:46.926 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:33:46.927 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:33:46.927 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:33:46.928 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:33:46.928 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:33:46.928 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:33:46.929 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:33:46.929 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:33:47.038 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.105578.asok 2021-04-09T18:33:47.042 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.105578.asok 2021-04-09T18:33:47.043 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:47.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.105578.asok status 2021-04-09T18:33:47.167 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:33:47.168 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:33:47.168 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:33:47.168 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:33:47.169 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:33:47.169 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:33:47.170 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:33:47.170 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "105578", 2021-04-09T18:33:47.171 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:33:47.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:47.171 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:33:47.172 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:33:47.172 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12651, 2021-04-09T18:33:47.173 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:33:47.173 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:33:47.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:33:47.174 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 12651 2021-04-09T18:33:47.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:47.175 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:33:47.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:47.176 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:33:47.177 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 4008937023 2021-04-09T18:33:47.177 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:33:47.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:47.178 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:33:47.178 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:33:47.179 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:33:47.179 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 4008937023 2021-04-09T18:33:47.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:33:47.179 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.12651 172.21.15.32:0/4008937023", 2021-04-09T18:33:47.180 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/4008937023", 2021-04-09T18:33:47.180 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:33:47.180 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 411, 2021-04-09T18:33:47.180 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 251, 2021-04-09T18:33:47.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:33:47.181 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:33:47.181 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:33:47.181 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:33:47.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:47.183 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:33:47.184 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:33:47.229 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:33:47.230 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:33:47.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:33:47.231 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:33:47.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:47.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.460 7fe0fda92700 1 Processor -- start 2021-04-09T18:33:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.460 7fe0fda92700 1 -- start start 2021-04-09T18:33:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.460 7fe0fda92700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f810d010 0x7fe0f80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.460 7fe0fda92700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0f80d8490 0x7fe0f810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.460 7fe0fda92700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f810d480 con 0x7fe0f80d1da0 2021-04-09T18:33:47.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.460 7fe0fda92700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f810e780 con 0x7fe0f80d8490 2021-04-09T18:33:47.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.460 7fe0fda92700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80d8d30 con 0x7fe0f810d010 2021-04-09T18:33:47.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe0f80d8490 0x7fe0f810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f6ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe0f80d1da0 0x7fe0f80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40434/0 (socket says 172.21.15.32:40434) 2021-04-09T18:33:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f6ffd700 1 -- 172.21.15.32:0/3668230828 learned_addr learned my addr 172.21.15.32:0/3668230828 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:47.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f77fe700 1 --2- 172.21.15.32:0/3668230828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f810d010 0x7fe0f80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f7fff700 1 -- 172.21.15.32:0/3668230828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f810d010 msgr2=0x7fe0f80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:47.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f7fff700 1 --2- 172.21.15.32:0/3668230828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f810d010 0x7fe0f80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f7fff700 1 -- 172.21.15.32:0/3668230828 >> v1:172.21.15.32:6789/0 conn(0x7fe0f80d1da0 legacy=0x7fe0f80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f7fff700 1 -- 172.21.15.32:0/3668230828 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f80d9220 con 0x7fe0f80d8490 2021-04-09T18:33:47.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f67fc700 1 -- 172.21.15.32:0/3668230828 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952214230 0 0) 0x7fe0e0000ce0 con 0x7fe0f80d1da0 2021-04-09T18:33:47.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f7fff700 1 --2- 172.21.15.32:0/3668230828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f80d8490 0x7fe0f810da00 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 rx=0x7fe0e8006ab0 tx=0x7fe0e8009ac0).ready entity=mon.0 client_cookie=78bca466eea9fedd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:47.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f67fc700 1 -- 172.21.15.32:0/3668230828 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e80066d0 con 0x7fe0f80d8490 2021-04-09T18:33:47.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f67fc700 1 -- 172.21.15.32:0/3668230828 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e80022e0 con 0x7fe0f80d8490 2021-04-09T18:33:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0f67fc700 1 -- 172.21.15.32:0/3668230828 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e80027c0 con 0x7fe0f80d8490 2021-04-09T18:33:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0fda92700 1 -- 172.21.15.32:0/3668230828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f80d8490 msgr2=0x7fe0f810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.461 7fe0fda92700 1 --2- 172.21.15.32:0/3668230828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f80d8490 0x7fe0f810da00 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 rx=0x7fe0e8006ab0 tx=0x7fe0e8009ac0).stop 2021-04-09T18:33:47.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.462 7fe0fda92700 1 -- 172.21.15.32:0/3668230828 shutdown_connections 2021-04-09T18:33:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.462 7fe0fda92700 1 --2- 172.21.15.32:0/3668230828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f80d8490 0x7fe0f810da00 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.462 7fe0fda92700 1 --2- 172.21.15.32:0/3668230828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f810d010 0x7fe0f80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.462 7fe0fda92700 1 -- 172.21.15.32:0/3668230828 >> 172.21.15.32:0/3668230828 conn(0x7fe0f80ce5f0 msgr2=0x7fe0f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:47.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.462 7fe0fda92700 1 -- 172.21.15.32:0/3668230828 shutdown_connections 2021-04-09T18:33:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.462 7fe0fda92700 1 -- 172.21.15.32:0/3668230828 wait complete. 2021-04-09T18:33:47.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.482 7fe0fda92700 1 Processor -- start 2021-04-09T18:33:47.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.482 7fe0fda92700 1 -- start start 2021-04-09T18:33:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0fda92700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f81a5210 0x7fe0f8185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0fda92700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0f8185a70 0x7fe0f80acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0fda92700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8185e70 0x7fe0f80ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0fda92700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f818f7b0 con 0x7fe0f8185e70 2021-04-09T18:33:47.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0fda92700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80ad7f0 con 0x7fe0f8185a70 2021-04-09T18:33:47.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0fda92700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80ada10 con 0x7fe0f81a5210 2021-04-09T18:33:47.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8185e70 0x7fe0f80ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8185e70 0x7fe0f80ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59082/0 (socket says 172.21.15.32:59082) 2021-04-09T18:33:47.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f7fff700 1 -- 172.21.15.32:0/735757276 learned_addr learned my addr 172.21.15.32:0/735757276 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:47.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f77fe700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f81a5210 0x7fe0f8185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f6ffd700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0f8185a70 0x7fe0f80acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f7fff700 1 -- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f81a5210 msgr2=0x7fe0f8185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f7fff700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f81a5210 0x7fe0f8185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f7fff700 1 -- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0f8185a70 msgr2=0x7fe0f80acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f7fff700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0f8185a70 0x7fe0f80acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.483 7fe0f7fff700 1 -- 172.21.15.32:0/735757276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f8179610 con 0x7fe0f8185e70 2021-04-09T18:33:47.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.484 7fe0f7fff700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8185e70 0x7fe0f80ad1d0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7fe0e800afc0 tx=0x7fe0e8002850).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:47.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.484 7fe0effff700 1 -- 172.21.15.32:0/735757276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e800a070 con 0x7fe0f8185e70 2021-04-09T18:33:47.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.484 7fe0effff700 1 -- 172.21.15.32:0/735757276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e800a330 con 0x7fe0f8185e70 2021-04-09T18:33:47.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.484 7fe0effff700 1 -- 172.21.15.32:0/735757276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e8009610 con 0x7fe0f8185e70 2021-04-09T18:33:47.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.484 7fe0fda92700 1 -- 172.21.15.32:0/735757276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f8179610 con 0x7fe0f8185e70 2021-04-09T18:33:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.484 7fe0fda92700 1 -- 172.21.15.32:0/735757276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f8179610 con 0x7fe0f8185e70 2021-04-09T18:33:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.485 7fe0effff700 1 -- 172.21.15.32:0/735757276 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe0e8002c40 con 0x7fe0f8185e70 2021-04-09T18:33:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.485 7fe0effff700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0dc0188c0 0x7fe0dc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.485 7fe0effff700 1 -- 172.21.15.32:0/735757276 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe0e8032900 con 0x7fe0f8185e70 2021-04-09T18:33:47.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.485 7fe0f77fe700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0dc0188c0 0x7fe0dc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.486 7fe0f77fe700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0dc0188c0 0x7fe0dc01adc0 crc :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:47.522 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.505 7fe0fda92700 1 -- 172.21.15.32:0/735757276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f81b7070 con 0x7fe0f8185e70 2021-04-09T18:33:47.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.508 7fe0effff700 1 -- 172.21.15.32:0/735757276 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe0e80009b0 con 0x7fe0f8185e70 2021-04-09T18:33:47.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.699 7fe0fda92700 1 -- 172.21.15.32:0/735757276 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0f810d480 con 0x7fe0f8185e70 2021-04-09T18:33:47.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.699 7fe0effff700 1 -- 172.21.15.32:0/735757276 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 411 v411) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fe0e8002c40 con 0x7fe0f8185e70 2021-04-09T18:33:47.718 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 411 2021-04-09T18:33:47.718 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:47.718 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":411,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12519,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407},{"gid":12522,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407},{"gid":59714,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":407}],"filesystems":[{"mdsmap":{"epoch":411,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:41.372484","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 -- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0dc0188c0 msgr2=0x7fe0dc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0dc0188c0 0x7fe0dc01adc0 crc :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 -- 172.21.15.32:0/735757276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8185e70 msgr2=0x7fe0f80ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8185e70 0x7fe0f80ad1d0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7fe0e800afc0 tx=0x7fe0e8002850).stop 2021-04-09T18:33:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 -- 172.21.15.32:0/735757276 shutdown_connections 2021-04-09T18:33:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe0dc0188c0 0x7fe0dc01adc0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0f8185e70 0x7fe0f80ad1d0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0f8185a70 0x7fe0f80acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 --2- 172.21.15.32:0/735757276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0f81a5210 0x7fe0f8185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 -- 172.21.15.32:0/735757276 >> 172.21.15.32:0/735757276 conn(0x7fe0f81a99c0 msgr2=0x7fe0f817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:47.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 -- 172.21.15.32:0/735757276 shutdown_connections 2021-04-09T18:33:47.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.702 7fe0fda92700 1 -- 172.21.15.32:0/735757276 wait complete. 2021-04-09T18:33:47.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2021-04-09T18:33:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.864 7efc0f081700 1 Processor -- start 2021-04-09T18:33:47.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.864 7efc0f081700 1 -- start start 2021-04-09T18:33:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.865 7efc0f081700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efc0810d010 0x7efc080d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.865 7efc0f081700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc080d1da0 0x7efc080d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.865 7efc0f081700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0810d480 con 0x7efc080d82c0 2021-04-09T18:33:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.865 7efc0f081700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc080d8d30 con 0x7efc080d1da0 2021-04-09T18:33:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.865 7efc0f081700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc080d9220 con 0x7efc0810d010 2021-04-09T18:33:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.865 7efc07fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc080d1da0 0x7efc080d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.865 7efc07fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efc080d1da0 0x7efc080d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59086/0 (socket says 172.21.15.32:59086) 2021-04-09T18:33:47.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.865 7efc07fff700 1 -- 172.21.15.32:0/781953817 learned_addr learned my addr 172.21.15.32:0/781953817 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:47.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.865 7efc0ce9d700 1 --2- 172.21.15.32:0/781953817 >> v2:172.21.15.86:3300/0 conn(0x7efc0810d010 0x7efc080d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc0ce9d700 1 -- 172.21.15.32:0/781953817 >> v2:172.21.15.32:3300/0 conn(0x7efc080d1da0 msgr2=0x7efc080d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:47.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc0ce9d700 1 --2- 172.21.15.32:0/781953817 >> v2:172.21.15.32:3300/0 conn(0x7efc080d1da0 0x7efc080d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc0ce9d700 1 -- 172.21.15.32:0/781953817 >> v1:172.21.15.86:6789/0 conn(0x7efc080d82c0 legacy=0x7efc0810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc0ce9d700 1 -- 172.21.15.32:0/781953817 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080d86f0 con 0x7efc0810d010 2021-04-09T18:33:47.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc077fe700 1 -- 172.21.15.32:0/781953817 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375479834 0 0) 0x7efc00000d30 con 0x7efc080d82c0 2021-04-09T18:33:47.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc07fff700 1 --2- 172.21.15.32:0/781953817 >> v2:172.21.15.32:3300/0 conn(0x7efc080d1da0 0x7efc080d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:47.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc0ce9d700 1 --2- 172.21.15.32:0/781953817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0810d010 0x7efc080d1780 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7efbf0006650 tx=0x7efbf0009a80).ready entity=mon.1 client_cookie=9edff79d8875dc2f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:47.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc077fe700 1 -- 172.21.15.32:0/781953817 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbf000c0c0 con 0x7efc0810d010 2021-04-09T18:33:47.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc077fe700 1 -- 172.21.15.32:0/781953817 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf000c3c0 con 0x7efc0810d010 2021-04-09T18:33:47.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc077fe700 1 -- 172.21.15.32:0/781953817 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbf000c810 con 0x7efc0810d010 2021-04-09T18:33:47.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc0f081700 1 -- 172.21.15.32:0/781953817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0810d010 msgr2=0x7efc080d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.866 7efc0f081700 1 --2- 172.21.15.32:0/781953817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0810d010 0x7efc080d1780 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7efbf0006650 tx=0x7efbf0009a80).stop 2021-04-09T18:33:47.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.867 7efc0f081700 1 -- 172.21.15.32:0/781953817 shutdown_connections 2021-04-09T18:33:47.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.867 7efc0f081700 1 --2- 172.21.15.32:0/781953817 >> v2:172.21.15.32:3300/0 conn(0x7efc080d1da0 0x7efc080d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.867 7efc0f081700 1 --2- 172.21.15.32:0/781953817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0810d010 0x7efc080d1780 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.867 7efc0f081700 1 -- 172.21.15.32:0/781953817 >> 172.21.15.32:0/781953817 conn(0x7efc080ce5f0 msgr2=0x7efc080d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:47.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.867 7efc0f081700 1 -- 172.21.15.32:0/781953817 shutdown_connections 2021-04-09T18:33:47.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.867 7efc0f081700 1 -- 172.21.15.32:0/781953817 wait complete. 2021-04-09T18:33:47.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.887 7efc0f081700 1 Processor -- start 2021-04-09T18:33:47.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.887 7efc0f081700 1 -- start start 2021-04-09T18:33:47.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0f081700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc081a0050 0x7efc081a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0f081700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc080bdcd0 0x7efc080bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0f081700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080bf080 0x7efc080b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0f081700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080bf500 con 0x7efc080bf080 2021-04-09T18:33:47.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0f081700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080bb7b0 con 0x7efc080bdcd0 2021-04-09T18:33:47.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0f081700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080bb960 con 0x7efc081a0050 2021-04-09T18:33:47.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0d69e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080bf080 0x7efc080b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0d69e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080bf080 0x7efc080b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59090/0 (socket says 172.21.15.32:59090) 2021-04-09T18:33:47.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0d69e700 1 -- 172.21.15.32:0/2271811023 learned_addr learned my addr 172.21.15.32:0/2271811023 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:47.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc07fff700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc080bdcd0 0x7efc080bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0ce9d700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc081a0050 0x7efc081a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0d69e700 1 -- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc081a0050 msgr2=0x7efc081a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0d69e700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc081a0050 0x7efc081a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0d69e700 1 -- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc080bdcd0 msgr2=0x7efc080bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:47.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0d69e700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc080bdcd0 0x7efc080bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:47.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0d69e700 1 -- 172.21.15.32:0/2271811023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080be0d0 con 0x7efc080bf080 2021-04-09T18:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc07fff700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc080bdcd0 0x7efc080bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.888 7efc0d69e700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080bf080 0x7efc080b0cf0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 rx=0x7efc00009980 tx=0x7efc0000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.889 7efc057fa700 1 -- 172.21.15.32:0/2271811023 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc00005870 con 0x7efc080bf080 2021-04-09T18:33:47.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.889 7efc057fa700 1 -- 172.21.15.32:0/2271811023 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc00005af0 con 0x7efc080bf080 2021-04-09T18:33:47.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.889 7efc0f081700 1 -- 172.21.15.32:0/2271811023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc080be0d0 con 0x7efc080bf080 2021-04-09T18:33:47.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.889 7efc057fa700 1 -- 172.21.15.32:0/2271811023 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc00006030 con 0x7efc080bf080 2021-04-09T18:33:47.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.889 7efc0f081700 1 -- 172.21.15.32:0/2271811023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc080be0d0 con 0x7efc080bf080 2021-04-09T18:33:47.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.889 7efc057fa700 1 -- 172.21.15.32:0/2271811023 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efc00006030 con 0x7efc080bf080 2021-04-09T18:33:47.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.889 7efc057fa700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbec018930 0x7efbec01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:47.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.889 7efc057fa700 1 -- 172.21.15.32:0/2271811023 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efc00032db0 con 0x7efc080bf080 2021-04-09T18:33:47.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.890 7efc0ce9d700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbec018930 0x7efbec01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:47.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.890 7efc0ce9d700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbec018930 0x7efbec01adc0 crc :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:47.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.909 7efc0f081700 1 -- 172.21.15.32:0/2271811023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc08093d50 con 0x7efc080bf080 2021-04-09T18:33:47.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:47.913 7efc057fa700 1 -- 172.21.15.32:0/2271811023 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efc0000be10 con 0x7efc080bf080 2021-04-09T18:33:48.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.108 7efc0f081700 1 -- 172.21.15.32:0/2271811023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"} v 0) v1 -- 0x7efc08093d50 con 0x7efc080bf080 2021-04-09T18:33:48.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.398 7efc057fa700 1 -- 172.21.15.32:0/2271811023 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "2"}]=0 v412) v1 ==== 105+0+0 (secure 0 0 0) 0x7efc00006030 con 0x7efc080bf080 2021-04-09T18:33:48.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 -- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbec018930 msgr2=0x7efbec01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:48.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbec018930 0x7efbec01adc0 crc :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 -- 172.21.15.32:0/2271811023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080bf080 msgr2=0x7efc080b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:48.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080bf080 0x7efc080b0cf0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 rx=0x7efc00009980 tx=0x7efc0000bcd0).stop 2021-04-09T18:33:48.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 -- 172.21.15.32:0/2271811023 shutdown_connections 2021-04-09T18:33:48.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efbec018930 0x7efbec01adc0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc080bf080 0x7efc080b0cf0 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc080bdcd0 0x7efc080bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 --2- 172.21.15.32:0/2271811023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc081a0050 0x7efc081a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 -- 172.21.15.32:0/2271811023 >> 172.21.15.32:0/2271811023 conn(0x7efc080975f0 msgr2=0x7efc08096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:48.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 -- 172.21.15.32:0/2271811023 shutdown_connections 2021-04-09T18:33:48.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.424 7efc0f081700 1 -- 172.21.15.32:0/2271811023 wait complete. 2021-04-09T18:33:48.452 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2021-04-09T18:33:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.589 7f39a91c7700 1 Processor -- start 2021-04-09T18:33:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.589 7f39a91c7700 1 -- start start 2021-04-09T18:33:48.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.589 7f39a91c7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a40cffc0 0x7f39a40d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:48.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.589 7f39a91c7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39a40d6ca0 0x7f39a410cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:48.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.589 7f39a91c7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a40d72a0 con 0x7f39a40d7820 2021-04-09T18:33:48.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.589 7f39a91c7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a40d70a0 con 0x7f39a40d6ca0 2021-04-09T18:33:48.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.589 7f39a91c7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a40d85a0 con 0x7f39a40cffc0 2021-04-09T18:33:48.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a359e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f39a40d6ca0 0x7f39a410cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:48.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a259c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f39a40d7820 0x7f39a40d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40470/0 (socket says 172.21.15.32:40470) 2021-04-09T18:33:48.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a259c700 1 -- 172.21.15.32:0/2048549969 learned_addr learned my addr 172.21.15.32:0/2048549969 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:48.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a2d9d700 1 --2- 172.21.15.32:0/2048549969 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a40cffc0 0x7f39a40d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a359e700 1 -- 172.21.15.32:0/2048549969 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a40cffc0 msgr2=0x7f39a40d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a359e700 1 --2- 172.21.15.32:0/2048549969 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a40cffc0 0x7f39a40d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a359e700 1 -- 172.21.15.32:0/2048549969 >> v1:172.21.15.32:6789/0 conn(0x7f39a40d7820 legacy=0x7f39a40d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a1d9b700 1 -- 172.21.15.32:0/2048549969 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096864768 0 0) 0x7f398c000d30 con 0x7f39a40d7820 2021-04-09T18:33:48.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a359e700 1 -- 172.21.15.32:0/2048549969 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a40d9200 con 0x7f39a40d6ca0 2021-04-09T18:33:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a359e700 1 --2- 172.21.15.32:0/2048549969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a40d6ca0 0x7f39a410cff0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 rx=0x7f399c006ab0 tx=0x7f399c009ac0).ready entity=mon.0 client_cookie=71992a2ab6dbac8d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a1d9b700 1 -- 172.21.15.32:0/2048549969 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f399c0066d0 con 0x7f39a40d6ca0 2021-04-09T18:33:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a1d9b700 1 -- 172.21.15.32:0/2048549969 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c0022e0 con 0x7f39a40d6ca0 2021-04-09T18:33:48.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a1d9b700 1 -- 172.21.15.32:0/2048549969 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f399c0027c0 con 0x7f39a40d6ca0 2021-04-09T18:33:48.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a91c7700 1 -- 172.21.15.32:0/2048549969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a40d6ca0 msgr2=0x7f39a410cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:48.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.590 7f39a91c7700 1 --2- 172.21.15.32:0/2048549969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a40d6ca0 0x7f39a410cff0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 rx=0x7f399c006ab0 tx=0x7f399c009ac0).stop 2021-04-09T18:33:48.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.591 7f39a91c7700 1 -- 172.21.15.32:0/2048549969 shutdown_connections 2021-04-09T18:33:48.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.591 7f39a91c7700 1 --2- 172.21.15.32:0/2048549969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a40d6ca0 0x7f39a410cff0 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.634 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.591 7f39a91c7700 1 --2- 172.21.15.32:0/2048549969 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a40cffc0 0x7f39a40d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.591 7f39a91c7700 1 -- 172.21.15.32:0/2048549969 >> 172.21.15.32:0/2048549969 conn(0x7f39a40ce5e0 msgr2=0x7f39a40cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:48.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.591 7f39a91c7700 1 -- 172.21.15.32:0/2048549969 shutdown_connections 2021-04-09T18:33:48.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.591 7f39a91c7700 1 -- 172.21.15.32:0/2048549969 wait complete. 2021-04-09T18:33:48.635 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a91c7700 1 Processor -- start 2021-04-09T18:33:48.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a91c7700 1 -- start start 2021-04-09T18:33:48.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a91c7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39a41a0580 0x7f39a4185d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:48.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a91c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a4180790 0x7f39a4186380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:48.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a91c7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a4181260 0x7f39a4181660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a91c7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a40a33f0 con 0x7f39a4180790 2021-04-09T18:33:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a91c7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4181c80 con 0x7f39a41a0580 2021-04-09T18:33:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a91c7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4181ea0 con 0x7f39a4181260 2021-04-09T18:33:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a4180790 0x7f39a4186380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a259c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a4180790 0x7f39a4186380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59118/0 (socket says 172.21.15.32:59118) 2021-04-09T18:33:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a259c700 1 -- 172.21.15.32:0/1675887273 learned_addr learned my addr 172.21.15.32:0/1675887273 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a2d9d700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39a41a0580 0x7f39a4185d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.612 7f39a359e700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a4181260 0x7f39a4181660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:48.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f39a259c700 1 -- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a4181260 msgr2=0x7f39a4181660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f39a259c700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a4181260 0x7f39a4181660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f39a259c700 1 -- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39a41a0580 msgr2=0x7f39a4185d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f39a259c700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39a41a0580 0x7f39a4185d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f39a259c700 1 -- 172.21.15.32:0/1675887273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a4094d90 con 0x7f39a4180790 2021-04-09T18:33:48.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f39a259c700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a4180790 0x7f39a4186380 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 rx=0x7f398c00a880 tx=0x7f398c00ac20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:48.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f399b7fe700 1 -- 172.21.15.32:0/1675887273 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f398c0070d0 con 0x7f39a4180790 2021-04-09T18:33:48.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f399b7fe700 1 -- 172.21.15.32:0/1675887273 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c007390 con 0x7f39a4180790 2021-04-09T18:33:48.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a4094d90 con 0x7f39a4180790 2021-04-09T18:33:48.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f399b7fe700 1 -- 172.21.15.32:0/1675887273 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f398c004740 con 0x7f39a4180790 2021-04-09T18:33:48.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.613 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a4094d90 con 0x7f39a4180790 2021-04-09T18:33:48.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.614 7f399b7fe700 1 -- 172.21.15.32:0/1675887273 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f398c0044c0 con 0x7f39a4180790 2021-04-09T18:33:48.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.614 7f399b7fe700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3988018930 0x7f398801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:48.644 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.614 7f399b7fe700 1 -- 172.21.15.32:0/1675887273 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f398c032d90 con 0x7f39a4180790 2021-04-09T18:33:48.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.614 7f39a2d9d700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3988018930 0x7f398801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:48.645 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.614 7f39a2d9d700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3988018930 0x7f398801ae10 crc :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:48.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.633 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a40a33f0 con 0x7f39a4180790 2021-04-09T18:33:48.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.637 7f399b7fe700 1 -- 172.21.15.32:0/1675887273 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f398c007b80 con 0x7f39a4180790 2021-04-09T18:33:48.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:48.832 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "allow_standby_replay", "prefix": "fs set", "fs_name": "cephfs", "val": "true"} v 0) v1 -- 0x7f39a40a33f0 con 0x7f39a4180790 2021-04-09T18:33:49.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.403 7f399b7fe700 1 -- 172.21.15.32:0/1675887273 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "allow_standby_replay", "prefix": "fs set", "fs_name": "cephfs", "val": "true"}]=0 v414) v1 ==== 121+0+0 (secure 0 0 0) 0x7f398c0044c0 con 0x7f39a4180790 2021-04-09T18:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3988018930 msgr2=0x7f398801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:49.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3988018930 0x7f398801ae10 crc :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a4180790 msgr2=0x7f39a4186380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a4180790 0x7f39a4186380 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 rx=0x7f398c00a880 tx=0x7f398c00ac20).stop 2021-04-09T18:33:49.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 shutdown_connections 2021-04-09T18:33:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3988018930 0x7f398801ae10 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f39a4180790 0x7f39a4186380 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39a4181260 0x7f39a4181660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 --2- 172.21.15.32:0/1675887273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39a41a0580 0x7f39a4185d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 >> 172.21.15.32:0/1675887273 conn(0x7f39a409faa0 msgr2=0x7f39a4188d70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:49.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 shutdown_connections 2021-04-09T18:33:49.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.449 7f39a91c7700 1 -- 172.21.15.32:0/1675887273 wait complete. 2021-04-09T18:33:49.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:49.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.611 7f1892b45700 1 Processor -- start 2021-04-09T18:33:49.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.611 7f1892b45700 1 -- start start 2021-04-09T18:33:49.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1892b45700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f188c10d010 0x7f188c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:49.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1892b45700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c0d8490 0x7f188c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:49.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1892b45700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f188c10d480 con 0x7f188c0d1da0 2021-04-09T18:33:49.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1892b45700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f188c0d8d30 con 0x7f188c10d010 2021-04-09T18:33:49.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1892b45700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188c0d9220 con 0x7f188c0d8490 2021-04-09T18:33:49.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1891162700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c0d8490 0x7f188c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1890961700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f188c10d010 0x7f188c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f188bfff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f188c0d1da0 0x7f188c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38416/0 (socket says 172.21.15.32:38416) 2021-04-09T18:33:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1891162700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c0d8490 0x7f188c10da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:38128/0 (socket says 172.21.15.32:38128) 2021-04-09T18:33:49.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f188bfff700 1 -- 172.21.15.32:0/696662824 learned_addr learned my addr 172.21.15.32:0/696662824 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1890961700 1 -- 172.21.15.32:0/696662824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c0d8490 msgr2=0x7f188c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1890961700 1 --2- 172.21.15.32:0/696662824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c0d8490 0x7f188c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1890961700 1 -- 172.21.15.32:0/696662824 >> v1:172.21.15.86:6789/0 conn(0x7f188c0d1da0 legacy=0x7f188c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:49.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f1890961700 1 -- 172.21.15.32:0/696662824 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188c10e020 con 0x7f188c10d010 2021-04-09T18:33:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.612 7f188b7fe700 1 -- 172.21.15.32:0/696662824 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27053265 0 0) 0x7f18740010c0 con 0x7f188c0d1da0 2021-04-09T18:33:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.613 7f1891162700 1 --2- 172.21.15.32:0/696662824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c0d8490 0x7f188c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:33:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.613 7f1890961700 1 --2- 172.21.15.32:0/696662824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c10d010 0x7f188c0d1780 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7f187c0069d0 tx=0x7f187c009e40).ready entity=mon.1 client_cookie=60576c5f95bbdd2c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.613 7f188b7fe700 1 -- 172.21.15.32:0/696662824 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f187c00b510 con 0x7f188c10d010 2021-04-09T18:33:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.613 7f188b7fe700 1 -- 172.21.15.32:0/696662824 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187c00b890 con 0x7f188c10d010 2021-04-09T18:33:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.613 7f188b7fe700 1 -- 172.21.15.32:0/696662824 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f187c00bd70 con 0x7f188c10d010 2021-04-09T18:33:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.613 7f1892b45700 1 -- 172.21.15.32:0/696662824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c10d010 msgr2=0x7f188c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.613 7f1892b45700 1 --2- 172.21.15.32:0/696662824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c10d010 0x7f188c0d1780 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7f187c0069d0 tx=0x7f187c009e40).stop 2021-04-09T18:33:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.614 7f1892b45700 1 -- 172.21.15.32:0/696662824 shutdown_connections 2021-04-09T18:33:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.614 7f1892b45700 1 --2- 172.21.15.32:0/696662824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c0d8490 0x7f188c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.614 7f1892b45700 1 --2- 172.21.15.32:0/696662824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c10d010 0x7f188c0d1780 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.614 7f1892b45700 1 -- 172.21.15.32:0/696662824 >> 172.21.15.32:0/696662824 conn(0x7f188c0ce5f0 msgr2=0x7f188c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.614 7f1892b45700 1 -- 172.21.15.32:0/696662824 shutdown_connections 2021-04-09T18:33:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.614 7f1892b45700 1 -- 172.21.15.32:0/696662824 wait complete. 2021-04-09T18:33:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.634 7f1892b45700 1 Processor -- start 2021-04-09T18:33:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.634 7f1892b45700 1 -- start start 2021-04-09T18:33:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.634 7f1892b45700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c184e20 0x7f188c1857e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.634 7f1892b45700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c187ac0 0x7f188c185e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.634 7f1892b45700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c08ea60 0x7f188c08ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.634 7f1892b45700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c185260 con 0x7f188c08ea60 2021-04-09T18:33:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.634 7f1892b45700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c0ac330 con 0x7f188c184e20 2021-04-09T18:33:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.634 7f1892b45700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188c0ac510 con 0x7f188c187ac0 2021-04-09T18:33:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1891162700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c08ea60 0x7f188c08ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1891162700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c08ea60 0x7f188c08ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59138/0 (socket says 172.21.15.32:59138) 2021-04-09T18:33:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1891162700 1 -- 172.21.15.32:0/693649598 learned_addr learned my addr 172.21.15.32:0/693649598 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1890961700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c184e20 0x7f188c1857e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f188bfff700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c187ac0 0x7f188c185e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1891162700 1 -- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c187ac0 msgr2=0x7f188c185e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1891162700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c187ac0 0x7f188c185e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1891162700 1 -- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c184e20 msgr2=0x7f188c1857e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1891162700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c184e20 0x7f188c1857e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1891162700 1 -- 172.21.15.32:0/693649598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188c08f4c0 con 0x7f188c08ea60 2021-04-09T18:33:49.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1890961700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c184e20 0x7f188c1857e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1891162700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c08ea60 0x7f188c08ee60 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f188400b5e0 tx=0x7f188400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f18897fa700 1 -- 172.21.15.32:0/693649598 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1884000c10 con 0x7f188c08ea60 2021-04-09T18:33:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f18897fa700 1 -- 172.21.15.32:0/693649598 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188400bd70 con 0x7f188c08ea60 2021-04-09T18:33:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f18897fa700 1 -- 172.21.15.32:0/693649598 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18840024d0 con 0x7f188c08ea60 2021-04-09T18:33:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1892b45700 1 -- 172.21.15.32:0/693649598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188c08f4c0 con 0x7f188c08ea60 2021-04-09T18:33:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.635 7f1892b45700 1 -- 172.21.15.32:0/693649598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188c08f4c0 con 0x7f188c08ea60 2021-04-09T18:33:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.636 7f18897fa700 1 -- 172.21.15.32:0/693649598 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1884006720 con 0x7f188c08ea60 2021-04-09T18:33:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.636 7f18897fa700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18700188c0 0x7f187001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:49.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.636 7f18897fa700 1 -- 172.21.15.32:0/693649598 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1884032a00 con 0x7f188c08ea60 2021-04-09T18:33:49.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.636 7f1890961700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18700188c0 0x7f187001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:49.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.637 7f1890961700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18700188c0 0x7f187001adc0 crc :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:49.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.656 7f1892b45700 1 -- 172.21.15.32:0/693649598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f188c10d480 con 0x7f188c08ea60 2021-04-09T18:33:49.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.659 7f18897fa700 1 -- 172.21.15.32:0/693649598 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1884006370 con 0x7f188c08ea60 2021-04-09T18:33:49.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.851 7f1892b45700 1 -- 172.21.15.32:0/693649598 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f188c10d480 con 0x7f188c08ea60 2021-04-09T18:33:49.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.851 7f18897fa700 1 -- 172.21.15.32:0/693649598 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 415 v415) v1 ==== 95+0+2817 (secure 0 0 0) 0x7f188400bd70 con 0x7f188c08ea60 2021-04-09T18:33:49.870 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 415 2021-04-09T18:33:49.871 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:49.871 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":415,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":415,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:49.404512","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12022,"mds_1":12519},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12519":{"gid":12519,"name":"a","rank":1,"incarnation":413,"state":"up:active","state_seq":5,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12522":{"gid":12522,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59714":{"gid":59714,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 -- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18700188c0 msgr2=0x7f187001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18700188c0 0x7f187001adc0 crc :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 -- 172.21.15.32:0/693649598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c08ea60 msgr2=0x7f188c08ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c08ea60 0x7f188c08ee60 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f188400b5e0 tx=0x7f188400b980).stop 2021-04-09T18:33:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 -- 172.21.15.32:0/693649598 shutdown_connections 2021-04-09T18:33:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f18700188c0 0x7f187001adc0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f188c08ea60 0x7f188c08ee60 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f188c187ac0 0x7f188c185e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 --2- 172.21.15.32:0/693649598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f188c184e20 0x7f188c1857e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 -- 172.21.15.32:0/693649598 >> 172.21.15.32:0/693649598 conn(0x7f188c1a9360 msgr2=0x7f188c1918a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:49.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 -- 172.21.15.32:0/693649598 shutdown_connections 2021-04-09T18:33:49.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:49.854 7f1892b45700 1 -- 172.21.15.32:0/693649598 wait complete. 2021-04-09T18:33:50.284 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 415, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:33:39.367894', 'modified': '2021-04-09 18:33:49.404512', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 12022, 'mds_1': 12519}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12022': {'gid': 12022, 'name': 'b', 'rank': 0, 'incarnation': 408, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/154900016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 154900016}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 154900016}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12519': {'gid': 12519, 'name': 'a', 'rank': 1, 'incarnation': 413, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.32:6833/2947726765', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2947726765}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2947726765}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12522': {'gid': 12522, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6835/3605541389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3605541389}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3605541389}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_59714': {'gid': 59714, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.86:6837/1243122766', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1243122766}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1243122766}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:33:50.285 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-04-09T18:33:50.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:33:50.424 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:33:50.424 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:33:50.425 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:33:50.425 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 12022, 2021-04-09T18:33:50.425 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:33:50.425 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:33:50.425 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 11.037558908999999, 2021-04-09T18:33:50.426 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 415, 2021-04-09T18:33:50.426 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 251, 2021-04-09T18:33:50.426 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 251, 2021-04-09T18:33:50.426 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 16.482878153000001 2021-04-09T18:33:50.427 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:33:50.431 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12022, "want_state": "up:active", "state": "up:active", "rank_uptime": 11.037558908999999, "mdsmap_epoch": 415, "osdmap_epoch": 251, "osdmap_epoch_barrier": 251, "uptime": 16.482878153000001 } 2021-04-09T18:33:50.431 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:33:50.574 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:33:50.575 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:33:50.575 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2021-04-09T18:33:50.575 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12519, 2021-04-09T18:33:50.576 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:33:50.576 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:33:50.576 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 2.156375315, 2021-04-09T18:33:50.576 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 415, 2021-04-09T18:33:50.576 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 251, 2021-04-09T18:33:50.577 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 251, 2021-04-09T18:33:50.577 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 16.623873726999999 2021-04-09T18:33:50.577 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:33:50.580 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 1, "id": 12519, "want_state": "up:active", "state": "up:active", "rank_uptime": 2.156375315, "mdsmap_epoch": 415, "osdmap_epoch": 251, "osdmap_epoch_barrier": 251, "uptime": 16.623873726999999 } 2021-04-09T18:33:50.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:33:50.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.716 7f2a2adc5700 1 Processor -- start 2021-04-09T18:33:50.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.716 7f2a2adc5700 1 -- start start 2021-04-09T18:33:50.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.717 7f2a2adc5700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a240d7ca0 0x7f2a240d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.717 7f2a2adc5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a2410da00 0x7f2a2410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.717 7f2a2adc5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2410d480 con 0x7f2a2410d010 2021-04-09T18:33:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.717 7f2a2adc5700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a240d9220 con 0x7f2a240d7ca0 2021-04-09T18:33:50.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.717 7f2a2adc5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240d86c0 con 0x7f2a2410da00 2021-04-09T18:33:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.717 7f2a23fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2a240d7ca0 0x7f2a240d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.717 7f2a28be1700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2a2410d010 0x7f2a240d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40502/0 (socket says 172.21.15.32:40502) 2021-04-09T18:33:50.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.717 7f2a28be1700 1 -- 172.21.15.32:0/1972252794 learned_addr learned my addr 172.21.15.32:0/1972252794 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.717 7f2a293e2700 1 --2- 172.21.15.32:0/1972252794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a2410da00 0x7f2a2410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a23fff700 1 -- 172.21.15.32:0/1972252794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a2410da00 msgr2=0x7f2a2410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:50.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a23fff700 1 --2- 172.21.15.32:0/1972252794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a2410da00 0x7f2a2410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a23fff700 1 -- 172.21.15.32:0/1972252794 >> v1:172.21.15.32:6789/0 conn(0x7f2a2410d010 legacy=0x7f2a240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a237fe700 1 -- 172.21.15.32:0/1972252794 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081929256 0 0) 0x7f2a14000ce0 con 0x7f2a2410d010 2021-04-09T18:33:50.761 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a23fff700 1 -- 172.21.15.32:0/1972252794 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240d1f70 con 0x7f2a240d7ca0 2021-04-09T18:33:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a293e2700 1 --2- 172.21.15.32:0/1972252794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a2410da00 0x7f2a2410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a23fff700 1 --2- 172.21.15.32:0/1972252794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240d7ca0 0x7f2a240d80a0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 rx=0x7f2a18006650 tx=0x7f2a18009a80).ready entity=mon.0 client_cookie=9226da1320d523f3 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:50.762 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a237fe700 1 -- 172.21.15.32:0/1972252794 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a180009b0 con 0x7f2a240d7ca0 2021-04-09T18:33:50.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a237fe700 1 -- 172.21.15.32:0/1972252794 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1800c0c0 con 0x7f2a240d7ca0 2021-04-09T18:33:50.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a237fe700 1 -- 172.21.15.32:0/1972252794 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a180009b0 con 0x7f2a240d7ca0 2021-04-09T18:33:50.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a2adc5700 1 -- 172.21.15.32:0/1972252794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240d7ca0 msgr2=0x7f2a240d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:50.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.718 7f2a2adc5700 1 --2- 172.21.15.32:0/1972252794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240d7ca0 0x7f2a240d80a0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 rx=0x7f2a18006650 tx=0x7f2a18009a80).stop 2021-04-09T18:33:50.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.719 7f2a2adc5700 1 -- 172.21.15.32:0/1972252794 shutdown_connections 2021-04-09T18:33:50.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.719 7f2a2adc5700 1 --2- 172.21.15.32:0/1972252794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a2410da00 0x7f2a2410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.719 7f2a2adc5700 1 --2- 172.21.15.32:0/1972252794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240d7ca0 0x7f2a240d80a0 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.719 7f2a2adc5700 1 -- 172.21.15.32:0/1972252794 >> 172.21.15.32:0/1972252794 conn(0x7f2a240ce5f0 msgr2=0x7f2a240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:50.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.719 7f2a2adc5700 1 -- 172.21.15.32:0/1972252794 shutdown_connections 2021-04-09T18:33:50.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.719 7f2a2adc5700 1 -- 172.21.15.32:0/1972252794 wait complete. 2021-04-09T18:33:50.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.739 7f2a2adc5700 1 Processor -- start 2021-04-09T18:33:50.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.739 7f2a2adc5700 1 -- start start 2021-04-09T18:33:50.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a2adc5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a240b1680 0x7f2a240b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:50.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a2adc5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a2409e330 0x7f2a240b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:50.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a2adc5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240b2190 0x7f2a2409f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:50.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a2adc5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2409e730 con 0x7f2a240b2190 2021-04-09T18:33:50.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a2adc5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2409f6c0 con 0x7f2a2409e330 2021-04-09T18:33:50.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a2adc5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2409f8e0 con 0x7f2a240b1680 2021-04-09T18:33:50.774 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a293e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240b2190 0x7f2a2409f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:50.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a293e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240b2190 0x7f2a2409f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59154/0 (socket says 172.21.15.32:59154) 2021-04-09T18:33:50.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a293e2700 1 -- 172.21.15.32:0/169803158 learned_addr learned my addr 172.21.15.32:0/169803158 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:50.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a28be1700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a240b1680 0x7f2a240b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:50.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a23fff700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a2409e330 0x7f2a240b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:50.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a293e2700 1 -- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a240b1680 msgr2=0x7f2a240b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:50.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a293e2700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a240b1680 0x7f2a240b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:50.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a293e2700 1 -- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a2409e330 msgr2=0x7f2a240b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:50.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a293e2700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a2409e330 0x7f2a240b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:50.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a293e2700 1 -- 172.21.15.32:0/169803158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2409fb00 con 0x7f2a240b2190 2021-04-09T18:33:50.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a293e2700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240b2190 0x7f2a2409f0a0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 rx=0x7f2a1c00b8c0 tx=0x7f2a1c00bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:50.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a217fa700 1 -- 172.21.15.32:0/169803158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a1c00c2f0 con 0x7f2a240b2190 2021-04-09T18:33:50.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a217fa700 1 -- 172.21.15.32:0/169803158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1c006790 con 0x7f2a240b2190 2021-04-09T18:33:50.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2409fb00 con 0x7f2a240b2190 2021-04-09T18:33:50.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a217fa700 1 -- 172.21.15.32:0/169803158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a1c00c7e0 con 0x7f2a240b2190 2021-04-09T18:33:50.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.740 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2409fb00 con 0x7f2a240b2190 2021-04-09T18:33:50.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.741 7f2a217fa700 1 -- 172.21.15.32:0/169803158 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2a1c0011f0 con 0x7f2a240b2190 2021-04-09T18:33:50.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.741 7f2a217fa700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a04018930 0x7f2a0401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:50.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.741 7f2a217fa700 1 -- 172.21.15.32:0/169803158 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2a1c032d70 con 0x7f2a240b2190 2021-04-09T18:33:50.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.741 7f2a28be1700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a04018930 0x7f2a0401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:50.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.742 7f2a28be1700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a04018930 0x7f2a0401ae10 crc :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:50.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.761 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2410d480 con 0x7f2a240b2190 2021-04-09T18:33:50.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.765 7f2a217fa700 1 -- 172.21.15.32:0/169803158 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2a1c00bda0 con 0x7f2a240b2190 2021-04-09T18:33:50.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:50.957 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f2a2410d480 con 0x7f2a240b2190 2021-04-09T18:33:51.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.408 7f2a217fa700 1 -- 172.21.15.32:0/169803158 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v416) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2a1c0011f0 con 0x7f2a240b2190 2021-04-09T18:33:51.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a04018930 msgr2=0x7f2a0401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:51.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a04018930 0x7f2a0401ae10 crc :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240b2190 msgr2=0x7f2a2409f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:51.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240b2190 0x7f2a2409f0a0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 rx=0x7f2a1c00b8c0 tx=0x7f2a1c00bc60).stop 2021-04-09T18:33:51.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 shutdown_connections 2021-04-09T18:33:51.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2a04018930 0x7f2a0401ae10 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2a240b2190 0x7f2a2409f0a0 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a2409e330 0x7f2a240b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 --2- 172.21.15.32:0/169803158 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a240b1680 0x7f2a240b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 >> 172.21.15.32:0/169803158 conn(0x7f2a240c4290 msgr2=0x7f2a240c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 shutdown_connections 2021-04-09T18:33:51.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.424 7f2a2adc5700 1 -- 172.21.15.32:0/169803158 wait complete. 2021-04-09T18:33:51.453 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:51.626 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.586 7f879a789700 1 Processor -- start 2021-04-09T18:33:51.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.586 7f879a789700 1 -- start start 2021-04-09T18:33:51.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.587 7f879a789700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f879410d010 0x7f87940d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:51.627 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.587 7f879a789700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87940d1da0 0x7f87940d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.587 7f879a789700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879410d480 con 0x7f87940d82c0 2021-04-09T18:33:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.587 7f879a789700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87940d8d30 con 0x7f87940d1da0 2021-04-09T18:33:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.587 7f879a789700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87940d9220 con 0x7f879410d010 2021-04-09T18:33:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.587 7f878bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87940d1da0 0x7f87940d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:51.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.587 7f878bfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87940d1da0 0x7f87940d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59162/0 (socket says 172.21.15.32:59162) 2021-04-09T18:33:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.587 7f878bfff700 1 -- 172.21.15.32:0/2040666977 learned_addr learned my addr 172.21.15.32:0/2040666977 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.587 7f8793fff700 1 --2- 172.21.15.32:0/2040666977 >> v2:172.21.15.86:3300/0 conn(0x7f879410d010 0x7f87940d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f878bfff700 1 -- 172.21.15.32:0/2040666977 >> v2:172.21.15.86:3300/0 conn(0x7f879410d010 msgr2=0x7f87940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:51.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f878bfff700 1 --2- 172.21.15.32:0/2040666977 >> v2:172.21.15.86:3300/0 conn(0x7f879410d010 0x7f87940d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f878bfff700 1 -- 172.21.15.32:0/2040666977 >> v1:172.21.15.86:6789/0 conn(0x7f87940d82c0 legacy=0x7f879410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:51.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f878bfff700 1 -- 172.21.15.32:0/2040666977 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87940d86f0 con 0x7f87940d1da0 2021-04-09T18:33:51.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f8793fff700 1 --2- 172.21.15.32:0/2040666977 >> v2:172.21.15.86:3300/0 conn(0x7f879410d010 0x7f87940d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:51.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f878bfff700 1 --2- 172.21.15.32:0/2040666977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87940d1da0 0x7f87940d7ca0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 rx=0x7f877c006700 tx=0x7f877c009b80).ready entity=mon.0 client_cookie=43422abd4d005d38 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:51.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f87937fe700 1 -- 172.21.15.32:0/2040666977 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f877c00c0c0 con 0x7f87940d1da0 2021-04-09T18:33:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f87937fe700 1 -- 172.21.15.32:0/2040666977 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c00c440 con 0x7f87940d1da0 2021-04-09T18:33:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f87937fe700 1 -- 172.21.15.32:0/2040666977 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f877c00c920 con 0x7f87940d1da0 2021-04-09T18:33:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f879a789700 1 -- 172.21.15.32:0/2040666977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87940d1da0 msgr2=0x7f87940d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.588 7f879a789700 1 --2- 172.21.15.32:0/2040666977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87940d1da0 0x7f87940d7ca0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 rx=0x7f877c006700 tx=0x7f877c009b80).stop 2021-04-09T18:33:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.589 7f879a789700 1 -- 172.21.15.32:0/2040666977 shutdown_connections 2021-04-09T18:33:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.589 7f879a789700 1 --2- 172.21.15.32:0/2040666977 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87940d1da0 0x7f87940d7ca0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.589 7f879a789700 1 --2- 172.21.15.32:0/2040666977 >> v2:172.21.15.86:3300/0 conn(0x7f879410d010 0x7f87940d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.589 7f879a789700 1 -- 172.21.15.32:0/2040666977 >> 172.21.15.32:0/2040666977 conn(0x7f87940ce5f0 msgr2=0x7f87940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.589 7f879a789700 1 -- 172.21.15.32:0/2040666977 shutdown_connections 2021-04-09T18:33:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.589 7f879a789700 1 -- 172.21.15.32:0/2040666977 wait complete. 2021-04-09T18:33:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.609 7f879a789700 1 Processor -- start 2021-04-09T18:33:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.609 7f879a789700 1 -- start start 2021-04-09T18:33:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.609 7f879a789700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87940b8480 0x7f87940b8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.609 7f879a789700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879418d800 0x7f879418dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:51.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.609 7f879a789700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87941a9500 0x7f87941a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:51.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.609 7f879a789700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879418d280 con 0x7f879418d800 2021-04-09T18:33:51.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.609 7f879a789700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879418e220 con 0x7f87940b8480 2021-04-09T18:33:51.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.609 7f879a789700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87941a9f20 con 0x7f87941a9500 2021-04-09T18:33:51.637 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f878bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879418d800 0x7f879418dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:51.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f878bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879418d800 0x7f879418dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59168/0 (socket says 172.21.15.32:59168) 2021-04-09T18:33:51.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f878bfff700 1 -- 172.21.15.32:0/3060654419 learned_addr learned my addr 172.21.15.32:0/3060654419 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:51.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f8793fff700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87940b8480 0x7f87940b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:51.638 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f8798da6700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87941a9500 0x7f87941a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f878bfff700 1 -- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87941a9500 msgr2=0x7f87941a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f878bfff700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87941a9500 0x7f87941a9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f878bfff700 1 -- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87940b8480 msgr2=0x7f87940b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f878bfff700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87940b8480 0x7f87940b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.639 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f878bfff700 1 -- 172.21.15.32:0/3060654419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87941aa140 con 0x7f879418d800 2021-04-09T18:33:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f878bfff700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879418d800 0x7f879418dc00 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7f877c00b580 tx=0x7f877c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f87917fa700 1 -- 172.21.15.32:0/3060654419 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f877c00bd10 con 0x7f879418d800 2021-04-09T18:33:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f87917fa700 1 -- 172.21.15.32:0/3060654419 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c000c90 con 0x7f879418d800 2021-04-09T18:33:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f879a789700 1 -- 172.21.15.32:0/3060654419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87941aa140 con 0x7f879418d800 2021-04-09T18:33:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f87917fa700 1 -- 172.21.15.32:0/3060654419 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f877c002220 con 0x7f879418d800 2021-04-09T18:33:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.610 7f879a789700 1 -- 172.21.15.32:0/3060654419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87941aa140 con 0x7f879418d800 2021-04-09T18:33:51.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.611 7f87917fa700 1 -- 172.21.15.32:0/3060654419 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f877c009f90 con 0x7f879418d800 2021-04-09T18:33:51.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.611 7f87917fa700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8778018900 0x7f877801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:51.642 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.611 7f87917fa700 1 -- 172.21.15.32:0/3060654419 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f877c0329f0 con 0x7f879418d800 2021-04-09T18:33:51.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.611 7f8793fff700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8778018900 0x7f877801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:51.643 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.612 7f8793fff700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8778018900 0x7f877801adc0 crc :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:51.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.631 7f879a789700 1 -- 172.21.15.32:0/3060654419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8794182d70 con 0x7f879418d800 2021-04-09T18:33:51.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.635 7f87917fa700 1 -- 172.21.15.32:0/3060654419 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f877c0091e0 con 0x7f879418d800 2021-04-09T18:33:51.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.825 7f879a789700 1 -- 172.21.15.32:0/3060654419 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f879410d480 con 0x7f879418d800 2021-04-09T18:33:51.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.826 7f87917fa700 1 -- 172.21.15.32:0/3060654419 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 417 v417) v1 ==== 95+0+2819 (secure 0 0 0) 0x7f877c000c90 con 0x7f879418d800 2021-04-09T18:33:51.844 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 417 2021-04-09T18:33:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:51.844 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":417,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":417,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:51.410433","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":12022,"mds_1":12519},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12519":{"gid":12519,"name":"a","rank":1,"incarnation":413,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12522":{"gid":12522,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59714":{"gid":59714,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:51.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 -- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8778018900 msgr2=0x7f877801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:51.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8778018900 0x7f877801adc0 crc :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 -- 172.21.15.32:0/3060654419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879418d800 msgr2=0x7f879418dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:51.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879418d800 0x7f879418dc00 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7f877c00b580 tx=0x7f877c00b920).stop 2021-04-09T18:33:51.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 -- 172.21.15.32:0/3060654419 shutdown_connections 2021-04-09T18:33:51.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8778018900 0x7f877801adc0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f879418d800 0x7f879418dc00 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87941a9500 0x7f87941a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 --2- 172.21.15.32:0/3060654419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87940b8480 0x7f87940b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:51.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 -- 172.21.15.32:0/3060654419 >> 172.21.15.32:0/3060654419 conn(0x7f879408fb70 msgr2=0x7f879419ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:51.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 -- 172.21.15.32:0/3060654419 shutdown_connections 2021-04-09T18:33:51.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:51.828 7f879a789700 1 -- 172.21.15.32:0/3060654419 wait complete. 2021-04-09T18:33:52.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 417, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:33:39.367894', 'modified': '2021-04-09 18:33:51.410433', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 12022, 'mds_1': 12519}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12022': {'gid': 12022, 'name': 'b', 'rank': 0, 'incarnation': 408, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/154900016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 154900016}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 154900016}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12519': {'gid': 12519, 'name': 'a', 'rank': 1, 'incarnation': 413, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6833/2947726765', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2947726765}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2947726765}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12522': {'gid': 12522, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6835/3605541389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3605541389}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3605541389}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_59714': {'gid': 59714, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.86:6837/1243122766', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1243122766}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1243122766}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:33:52.258 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_12519:up:stopping 2021-04-09T18:33:53.260 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:53.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.393 7f259ecf2700 1 Processor -- start 2021-04-09T18:33:53.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.393 7f259ecf2700 1 -- start start 2021-04-09T18:33:53.433 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.393 7f259ecf2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25980d7ca0 0x7f25980d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:53.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.393 7f259ecf2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f259810da00 0x7f259810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:53.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.393 7f259ecf2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259810d480 con 0x7f259810d010 2021-04-09T18:33:53.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.393 7f259ecf2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25980d9220 con 0x7f25980d7ca0 2021-04-09T18:33:53.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.393 7f259ecf2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25980d86c0 con 0x7f259810da00 2021-04-09T18:33:53.434 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f2597fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25980d7ca0 0x7f25980d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:53.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f2597fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f25980d7ca0 0x7f25980d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59172/0 (socket says 172.21.15.32:59172) 2021-04-09T18:33:53.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f2597fff700 1 -- 172.21.15.32:0/3646519134 learned_addr learned my addr 172.21.15.32:0/3646519134 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:53.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f259d30f700 1 --2- 172.21.15.32:0/3646519134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f259810da00 0x7f259810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:53.435 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f2597fff700 1 -- 172.21.15.32:0/3646519134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f259810da00 msgr2=0x7f259810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:53.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f2597fff700 1 --2- 172.21.15.32:0/3646519134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f259810da00 0x7f259810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f2597fff700 1 -- 172.21.15.32:0/3646519134 >> v1:172.21.15.86:6789/0 conn(0x7f259810d010 legacy=0x7f25980d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:53.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f2597fff700 1 -- 172.21.15.32:0/3646519134 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25980d1f70 con 0x7f25980d7ca0 2021-04-09T18:33:53.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f2597fff700 1 --2- 172.21.15.32:0/3646519134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25980d7ca0 0x7f25980d80a0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7f2588006a80 tx=0x7f2588009e80).ready entity=mon.0 client_cookie=7a8feb8813227346 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:53.436 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f25977fe700 1 -- 172.21.15.32:0/3646519134 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f258800b510 con 0x7f25980d7ca0 2021-04-09T18:33:53.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f25977fe700 1 -- 172.21.15.32:0/3646519134 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258800b7d0 con 0x7f25980d7ca0 2021-04-09T18:33:53.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f25977fe700 1 -- 172.21.15.32:0/3646519134 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f258800bcb0 con 0x7f25980d7ca0 2021-04-09T18:33:53.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f259ecf2700 1 -- 172.21.15.32:0/3646519134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25980d7ca0 msgr2=0x7f25980d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:53.437 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.394 7f259ecf2700 1 --2- 172.21.15.32:0/3646519134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25980d7ca0 0x7f25980d80a0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7f2588006a80 tx=0x7f2588009e80).stop 2021-04-09T18:33:53.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.395 7f259ecf2700 1 -- 172.21.15.32:0/3646519134 shutdown_connections 2021-04-09T18:33:53.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.395 7f259ecf2700 1 --2- 172.21.15.32:0/3646519134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f259810da00 0x7f259810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.395 7f259ecf2700 1 --2- 172.21.15.32:0/3646519134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25980d7ca0 0x7f25980d80a0 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.395 7f259ecf2700 1 -- 172.21.15.32:0/3646519134 >> 172.21.15.32:0/3646519134 conn(0x7f25980ce5f0 msgr2=0x7f25980d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:53.438 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.395 7f259ecf2700 1 -- 172.21.15.32:0/3646519134 shutdown_connections 2021-04-09T18:33:53.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.395 7f259ecf2700 1 -- 172.21.15.32:0/3646519134 wait complete. 2021-04-09T18:33:53.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.415 7f259ecf2700 1 Processor -- start 2021-04-09T18:33:53.439 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.415 7f259ecf2700 1 -- start start 2021-04-09T18:33:53.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259ecf2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25981a91e0 0x7f25980a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:53.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259ecf2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25981a95e0 0x7f25980a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:53.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259ecf2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25981ac850 0x7f25981acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:53.442 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259ecf2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25981ad270 con 0x7f25981a91e0 2021-04-09T18:33:53.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259ecf2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25980b7640 con 0x7f25981a95e0 2021-04-09T18:33:53.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259ecf2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2598183c80 con 0x7f25981ac850 2021-04-09T18:33:53.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259cb0e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25981a91e0 0x7f25980a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:53.443 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259cb0e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25981a91e0 0x7f25980a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59174/0 (socket says 172.21.15.32:59174) 2021-04-09T18:33:53.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259cb0e700 1 -- 172.21.15.32:0/2344152274 learned_addr learned my addr 172.21.15.32:0/2344152274 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:53.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f2597fff700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25981a95e0 0x7f25980a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:53.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.416 7f259d30f700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25981ac850 0x7f25981acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:53.444 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f259cb0e700 1 -- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25981ac850 msgr2=0x7f25981acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:53.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f259cb0e700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25981ac850 0x7f25981acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f259cb0e700 1 -- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25981a95e0 msgr2=0x7f25980a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:53.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f259cb0e700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25981a95e0 0x7f25980a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f259cb0e700 1 -- 172.21.15.32:0/2344152274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598183ea0 con 0x7f25981a91e0 2021-04-09T18:33:53.445 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f259cb0e700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25981a91e0 0x7f25980a4a60 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f258000b530 tx=0x7f258000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:53.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f25957fa700 1 -- 172.21.15.32:0/2344152274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f258000bcc0 con 0x7f25981a91e0 2021-04-09T18:33:53.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f25957fa700 1 -- 172.21.15.32:0/2344152274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2580006020 con 0x7f25981a91e0 2021-04-09T18:33:53.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2598183ea0 con 0x7f25981a91e0 2021-04-09T18:33:53.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f25957fa700 1 -- 172.21.15.32:0/2344152274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25800054a0 con 0x7f25981a91e0 2021-04-09T18:33:53.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.417 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2598183ea0 con 0x7f25981a91e0 2021-04-09T18:33:53.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.418 7f25957fa700 1 -- 172.21.15.32:0/2344152274 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2580006200 con 0x7f25981a91e0 2021-04-09T18:33:53.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.418 7f25957fa700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f257c018930 0x7f257c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:53.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.418 7f25957fa700 1 -- 172.21.15.32:0/2344152274 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f2580032a30 con 0x7f25981a91e0 2021-04-09T18:33:53.447 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.418 7f2597fff700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f257c018930 0x7f257c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:53.448 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.418 7f2597fff700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f257c018930 0x7f257c01ae10 crc :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:53.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.437 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2598182d70 con 0x7f25981a91e0 2021-04-09T18:33:53.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.441 7f25957fa700 1 -- 172.21.15.32:0/2344152274 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f258000bcc0 con 0x7f25981a91e0 2021-04-09T18:33:53.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.633 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f259810d480 con 0x7f25981a91e0 2021-04-09T18:33:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.633 7f25957fa700 1 -- 172.21.15.32:0/2344152274 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 417 v417) v1 ==== 95+0+2819 (secure 0 0 0) 0x7f2580006020 con 0x7f25981a91e0 2021-04-09T18:33:53.651 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 417 2021-04-09T18:33:53.652 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:53.652 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":417,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":417,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:51.410433","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":12022,"mds_1":12519},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12519":{"gid":12519,"name":"a","rank":1,"incarnation":413,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12522":{"gid":12522,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59714":{"gid":59714,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:53.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.635 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f257c018930 msgr2=0x7f257c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.635 7f259ecf2700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f257c018930 0x7f257c01ae10 crc :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.635 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25981a91e0 msgr2=0x7f25980a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.635 7f259ecf2700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25981a91e0 0x7f25980a4a60 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f258000b530 tx=0x7f258000b8d0).stop 2021-04-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.636 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 shutdown_connections 2021-04-09T18:33:53.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.636 7f259ecf2700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f257c018930 0x7f257c01ae10 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.636 7f259ecf2700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25981ac850 0x7f25981acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.636 7f259ecf2700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f25981a95e0 0x7f25980a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.636 7f259ecf2700 1 --2- 172.21.15.32:0/2344152274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25981a91e0 0x7f25980a4a60 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:53.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.636 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 >> 172.21.15.32:0/2344152274 conn(0x7f259819b460 msgr2=0x7f259819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.636 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 shutdown_connections 2021-04-09T18:33:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:53.636 7f259ecf2700 1 -- 172.21.15.32:0/2344152274 wait complete. 2021-04-09T18:33:54.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 417, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:33:39.367894', 'modified': '2021-04-09 18:33:51.410433', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 12022, 'mds_1': 12519}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12022': {'gid': 12022, 'name': 'b', 'rank': 0, 'incarnation': 408, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/154900016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 154900016}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 154900016}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12519': {'gid': 12519, 'name': 'a', 'rank': 1, 'incarnation': 413, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6833/2947726765', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2947726765}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2947726765}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12522': {'gid': 12522, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6835/3605541389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3605541389}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3605541389}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_59714': {'gid': 59714, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.86:6837/1243122766', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1243122766}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1243122766}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:33:54.066 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_12519:up:stopping 2021-04-09T18:33:55.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:55.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.211 7f8d7805f700 1 Processor -- start 2021-04-09T18:33:55.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.212 7f8d7805f700 1 -- start start 2021-04-09T18:33:55.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.212 7f8d7805f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8d7010d010 0x7f8d700d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:55.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.212 7f8d7805f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8d700d1da0 0x7f8d700d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:55.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.212 7f8d7805f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7010d480 con 0x7f8d700d82c0 2021-04-09T18:33:55.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.212 7f8d7805f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d700d8d30 con 0x7f8d700d1da0 2021-04-09T18:33:55.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.212 7f8d7805f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d700d9220 con 0x7f8d7010d010 2021-04-09T18:33:55.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.212 7f8d7567a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8d700d1da0 0x7f8d700d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:55.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.212 7f8d7667c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8d700d82c0 0x7f8d7010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40542/0 (socket says 172.21.15.32:40542) 2021-04-09T18:33:55.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.212 7f8d7667c700 1 -- 172.21.15.32:0/490201320 learned_addr learned my addr 172.21.15.32:0/490201320 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:55.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d75e7b700 1 --2- 172.21.15.32:0/490201320 >> v2:172.21.15.86:3300/0 conn(0x7f8d7010d010 0x7f8d700d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:55.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d74e79700 1 -- 172.21.15.32:0/490201320 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046224849 0 0) 0x7f8d68000d30 con 0x7f8d700d82c0 2021-04-09T18:33:55.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d74e79700 1 -- 172.21.15.32:0/490201320 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c002fe0 con 0x7f8d700d82c0 2021-04-09T18:33:55.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d7567a700 1 -- 172.21.15.32:0/490201320 >> v2:172.21.15.86:3300/0 conn(0x7f8d7010d010 msgr2=0x7f8d700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:55.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d7567a700 1 --2- 172.21.15.32:0/490201320 >> v2:172.21.15.86:3300/0 conn(0x7f8d7010d010 0x7f8d700d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d7567a700 1 -- 172.21.15.32:0/490201320 >> v1:172.21.15.32:6789/0 conn(0x7f8d700d82c0 legacy=0x7f8d7010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d7567a700 1 -- 172.21.15.32:0/490201320 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d700d86f0 con 0x7f8d700d1da0 2021-04-09T18:33:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d7567a700 1 --2- 172.21.15.32:0/490201320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d700d1da0 0x7f8d700d7ca0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 rx=0x7f8d60006650 tx=0x7f8d60009a80).ready entity=mon.0 client_cookie=41d38574447039a6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:55.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d74e79700 1 -- 172.21.15.32:0/490201320 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d600009b0 con 0x7f8d700d1da0 2021-04-09T18:33:55.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d74e79700 1 -- 172.21.15.32:0/490201320 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6000c0c0 con 0x7f8d700d1da0 2021-04-09T18:33:55.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d74e79700 1 -- 172.21.15.32:0/490201320 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d6000c510 con 0x7f8d700d1da0 2021-04-09T18:33:55.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d7805f700 1 -- 172.21.15.32:0/490201320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d700d1da0 msgr2=0x7f8d700d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:55.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.213 7f8d7805f700 1 --2- 172.21.15.32:0/490201320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d700d1da0 0x7f8d700d7ca0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 rx=0x7f8d60006650 tx=0x7f8d60009a80).stop 2021-04-09T18:33:55.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.214 7f8d7805f700 1 -- 172.21.15.32:0/490201320 shutdown_connections 2021-04-09T18:33:55.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.214 7f8d7805f700 1 --2- 172.21.15.32:0/490201320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d700d1da0 0x7f8d700d7ca0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.214 7f8d7805f700 1 --2- 172.21.15.32:0/490201320 >> v2:172.21.15.86:3300/0 conn(0x7f8d7010d010 0x7f8d700d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.214 7f8d7805f700 1 -- 172.21.15.32:0/490201320 >> 172.21.15.32:0/490201320 conn(0x7f8d700ce5f0 msgr2=0x7f8d700d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.214 7f8d7805f700 1 -- 172.21.15.32:0/490201320 shutdown_connections 2021-04-09T18:33:55.259 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.214 7f8d7805f700 1 -- 172.21.15.32:0/490201320 wait complete. 2021-04-09T18:33:55.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7805f700 1 Processor -- start 2021-04-09T18:33:55.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7805f700 1 -- start start 2021-04-09T18:33:55.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7805f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d700b5360 0x7f8d701a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:55.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7805f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d7018e140 0x7f8d701aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:55.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7805f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d701ad040 0x7f8d7008fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:55.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7805f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d701a5360 con 0x7f8d7018e140 2021-04-09T18:33:55.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7805f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700902c0 con 0x7f8d700b5360 2021-04-09T18:33:55.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7805f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d700904e0 con 0x7f8d701ad040 2021-04-09T18:33:55.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7567a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d7018e140 0x7f8d701aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:55.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7567a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d7018e140 0x7f8d701aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59190/0 (socket says 172.21.15.32:59190) 2021-04-09T18:33:55.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d7567a700 1 -- 172.21.15.32:0/1633281739 learned_addr learned my addr 172.21.15.32:0/1633281739 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:55.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.235 7f8d75e7b700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d700b5360 0x7f8d701a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:55.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7667c700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d701ad040 0x7f8d7008fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:55.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7567a700 1 -- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d701ad040 msgr2=0x7f8d7008fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:55.266 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7567a700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d701ad040 0x7f8d7008fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7567a700 1 -- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d700b5360 msgr2=0x7f8d701a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:55.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7567a700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d700b5360 0x7f8d701a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7567a700 1 -- 172.21.15.32:0/1633281739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70183c80 con 0x7f8d7018e140 2021-04-09T18:33:55.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7667c700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d701ad040 0x7f8d7008fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:33:55.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7567a700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d7018e140 0x7f8d701aca20 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 rx=0x7f8d60009650 tx=0x7f8d6000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:55.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d667fc700 1 -- 172.21.15.32:0/1633281739 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d6000bd50 con 0x7f8d7018e140 2021-04-09T18:33:55.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d667fc700 1 -- 172.21.15.32:0/1633281739 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d60006030 con 0x7f8d7018e140 2021-04-09T18:33:55.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d667fc700 1 -- 172.21.15.32:0/1633281739 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d60005560 con 0x7f8d7018e140 2021-04-09T18:33:55.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d70183c80 con 0x7f8d7018e140 2021-04-09T18:33:55.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.236 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d70183c80 con 0x7f8d7018e140 2021-04-09T18:33:55.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.237 7f8d667fc700 1 -- 172.21.15.32:0/1633281739 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8d60005700 con 0x7f8d7018e140 2021-04-09T18:33:55.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.237 7f8d667fc700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d58018930 0x7f8d5801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.237 7f8d667fc700 1 -- 172.21.15.32:0/1633281739 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8d60032a00 con 0x7f8d7018e140 2021-04-09T18:33:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.237 7f8d75e7b700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d58018930 0x7f8d5801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:55.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.237 7f8d75e7b700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d58018930 0x7f8d5801ae10 crc :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:55.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.256 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d70182d70 con 0x7f8d7018e140 2021-04-09T18:33:55.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.260 7f8d667fc700 1 -- 172.21.15.32:0/1633281739 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8d600054f0 con 0x7f8d7018e140 2021-04-09T18:33:55.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.452 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d7010d480 con 0x7f8d7018e140 2021-04-09T18:33:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.453 7f8d667fc700 1 -- 172.21.15.32:0/1633281739 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 417 v417) v1 ==== 95+0+2819 (secure 0 0 0) 0x7f8d6000bd50 con 0x7f8d7018e140 2021-04-09T18:33:55.471 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 417 2021-04-09T18:33:55.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:55.472 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":417,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":417,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:51.410433","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":12022,"mds_1":12519},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12519":{"gid":12519,"name":"a","rank":1,"incarnation":413,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12522":{"gid":12522,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59714":{"gid":59714,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:55.473 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d58018930 msgr2=0x7f8d5801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d58018930 0x7f8d5801ae10 crc :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d7018e140 msgr2=0x7f8d701aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d7018e140 0x7f8d701aca20 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 rx=0x7f8d60009650 tx=0x7f8d6000b960).stop 2021-04-09T18:33:55.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 shutdown_connections 2021-04-09T18:33:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d58018930 0x7f8d5801ae10 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d7018e140 0x7f8d701aca20 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d701ad040 0x7f8d7008fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 --2- 172.21.15.32:0/1633281739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d700b5360 0x7f8d701a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 >> 172.21.15.32:0/1633281739 conn(0x7f8d7019b460 msgr2=0x7f8d7019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 shutdown_connections 2021-04-09T18:33:55.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:55.456 7f8d7805f700 1 -- 172.21.15.32:0/1633281739 wait complete. 2021-04-09T18:33:55.885 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 417, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:33:39.367894', 'modified': '2021-04-09 18:33:51.410433', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 12022, 'mds_1': 12519}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12022': {'gid': 12022, 'name': 'b', 'rank': 0, 'incarnation': 408, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/154900016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 154900016}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 154900016}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12519': {'gid': 12519, 'name': 'a', 'rank': 1, 'incarnation': 413, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6833/2947726765', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2947726765}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2947726765}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12522': {'gid': 12522, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6835/3605541389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3605541389}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3605541389}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_59714': {'gid': 59714, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.86:6837/1243122766', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1243122766}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1243122766}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:33:55.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_12519:up:stopping 2021-04-09T18:33:56.888 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.031 7f57c77e0700 1 Processor -- start 2021-04-09T18:33:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.031 7f57c77e0700 1 -- start start 2021-04-09T18:33:57.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c77e0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57c00cffa0 0x7f57c00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c77e0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c00d7280 con 0x7f57c00d7800 2021-04-09T18:33:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c77e0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c00d83f0 con 0x7f57c00d6c80 2021-04-09T18:33:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c77e0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c00d7080 con 0x7f57c00cffa0 2021-04-09T18:33:57.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c55fc700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f57c00cffa0 0x7f57c00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c4dfb700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f57c00d7800 0x7f57c00d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40558/0 (socket says 172.21.15.32:40558) 2021-04-09T18:33:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c4dfb700 1 -- 172.21.15.32:0/1599705885 learned_addr learned my addr 172.21.15.32:0/1599705885 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c55fc700 1 -- 172.21.15.32:0/1599705885 >> v1:172.21.15.86:6789/0 conn(0x7f57c00d6c80 legacy=0x7f57c010cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:57.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c55fc700 1 -- 172.21.15.32:0/1599705885 >> v1:172.21.15.32:6789/0 conn(0x7f57c00d7800 legacy=0x7f57c00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:57.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c55fc700 1 -- 172.21.15.32:0/1599705885 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c00d91e0 con 0x7f57c00cffa0 2021-04-09T18:33:57.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57b7fff700 1 -- 172.21.15.32:0/1599705885 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372609229 0 0) 0x7f57b00009b0 con 0x7f57c00d7800 2021-04-09T18:33:57.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.032 7f57c55fc700 1 --2- 172.21.15.32:0/1599705885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c00cffa0 0x7f57c00d2440 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 rx=0x7f57bc006650 tx=0x7f57bc009a00).ready entity=mon.0 client_cookie=97da128f7c8b3349 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:57.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57b7fff700 1 -- 172.21.15.32:0/1599705885 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57bc009df0 con 0x7f57c00cffa0 2021-04-09T18:33:57.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57b7fff700 1 -- 172.21.15.32:0/1599705885 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc0009b0 con 0x7f57c00cffa0 2021-04-09T18:33:57.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57b7fff700 1 -- 172.21.15.32:0/1599705885 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57bc00c2b0 con 0x7f57c00cffa0 2021-04-09T18:33:57.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57c77e0700 1 -- 172.21.15.32:0/1599705885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c00cffa0 msgr2=0x7f57c00d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:57.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57c77e0700 1 --2- 172.21.15.32:0/1599705885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c00cffa0 0x7f57c00d2440 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 rx=0x7f57bc006650 tx=0x7f57bc009a00).stop 2021-04-09T18:33:57.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57c77e0700 1 -- 172.21.15.32:0/1599705885 shutdown_connections 2021-04-09T18:33:57.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57c77e0700 1 --2- 172.21.15.32:0/1599705885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c00cffa0 0x7f57c00d2440 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:57.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57c77e0700 1 -- 172.21.15.32:0/1599705885 >> 172.21.15.32:0/1599705885 conn(0x7f57c00ce5c0 msgr2=0x7f57c00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:57.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57c77e0700 1 -- 172.21.15.32:0/1599705885 shutdown_connections 2021-04-09T18:33:57.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.033 7f57c77e0700 1 -- 172.21.15.32:0/1599705885 wait complete. 2021-04-09T18:33:57.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c77e0700 1 Processor -- start 2021-04-09T18:33:57.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c77e0700 1 -- start start 2021-04-09T18:33:57.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c77e0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01a6250 0x7f57c01a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:57.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c77e0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c01a7470 0x7f57c01a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:57.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c77e0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01aba60 0x7f57c01abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c77e0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c01a6c70 con 0x7f57c01a7470 2021-04-09T18:33:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c77e0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c01a6ef0 con 0x7f57c01a6250 2021-04-09T18:33:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c77e0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c01a7e90 con 0x7f57c01aba60 2021-04-09T18:33:57.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c4dfb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c01a7470 0x7f57c01a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:57.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c4dfb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c01a7470 0x7f57c01a7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59204/0 (socket says 172.21.15.32:59204) 2021-04-09T18:33:57.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.054 7f57c4dfb700 1 -- 172.21.15.32:0/2306915554 learned_addr learned my addr 172.21.15.32:0/2306915554 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:57.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c55fc700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01a6250 0x7f57c01a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:57.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c5dfd700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01aba60 0x7f57c01abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:57.082 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c4dfb700 1 -- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01aba60 msgr2=0x7f57c01abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:57.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c4dfb700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01aba60 0x7f57c01abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:57.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c4dfb700 1 -- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01a6250 msgr2=0x7f57c01a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:57.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c4dfb700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01a6250 0x7f57c01a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:57.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c4dfb700 1 -- 172.21.15.32:0/2306915554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0092900 con 0x7f57c01a7470 2021-04-09T18:33:57.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c4dfb700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c01a7470 0x7f57c01a7870 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f57b0001c60 tx=0x7f57b000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:57.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57b5ffb700 1 -- 172.21.15.32:0/2306915554 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57b000acc0 con 0x7f57c01a7470 2021-04-09T18:33:57.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57b5ffb700 1 -- 172.21.15.32:0/2306915554 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b0004140 con 0x7f57c01a7470 2021-04-09T18:33:57.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c0092900 con 0x7f57c01a7470 2021-04-09T18:33:57.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57b5ffb700 1 -- 172.21.15.32:0/2306915554 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57b0004620 con 0x7f57c01a7470 2021-04-09T18:33:57.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.055 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c0092900 con 0x7f57c01a7470 2021-04-09T18:33:57.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.056 7f57b5ffb700 1 -- 172.21.15.32:0/2306915554 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f57b0000dd0 con 0x7f57c01a7470 2021-04-09T18:33:57.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.056 7f57b5ffb700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57a0018930 0x7f57a001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:57.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.056 7f57b5ffb700 1 -- 172.21.15.32:0/2306915554 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f57b0032a20 con 0x7f57c01a7470 2021-04-09T18:33:57.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.056 7f57c55fc700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57a0018930 0x7f57a001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:57.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.056 7f57c55fc700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57a0018930 0x7f57a001ae10 crc :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:57.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.076 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c017cae0 con 0x7f57c01a7470 2021-04-09T18:33:57.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.079 7f57b5ffb700 1 -- 172.21.15.32:0/2306915554 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f57b00009b0 con 0x7f57c01a7470 2021-04-09T18:33:57.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.272 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57c00d7280 con 0x7f57c01a7470 2021-04-09T18:33:57.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.273 7f57b5ffb700 1 -- 172.21.15.32:0/2306915554 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 417 v417) v1 ==== 95+0+2819 (secure 0 0 0) 0x7f57b0000dd0 con 0x7f57c01a7470 2021-04-09T18:33:57.291 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 417 2021-04-09T18:33:57.291 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:57.292 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":417,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":417,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:51.410433","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":12022,"mds_1":12519},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12519":{"gid":12519,"name":"a","rank":1,"incarnation":413,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12522":{"gid":12522,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59714":{"gid":59714,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:57.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57a0018930 msgr2=0x7f57a001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57a0018930 0x7f57a001ae10 crc :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c01a7470 msgr2=0x7f57c01a7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c01a7470 0x7f57c01a7870 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f57b0001c60 tx=0x7f57b000a8d0).stop 2021-04-09T18:33:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 shutdown_connections 2021-04-09T18:33:57.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f57a0018930 0x7f57a001ae10 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57c01a7470 0x7f57c01a7870 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01aba60 0x7f57c01abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 --2- 172.21.15.32:0/2306915554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01a6250 0x7f57c01a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:57.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.275 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 >> 172.21.15.32:0/2306915554 conn(0x7f57c018ce60 msgr2=0x7f57c0094920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:57.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.276 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 shutdown_connections 2021-04-09T18:33:57.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:57.276 7f57c77e0700 1 -- 172.21.15.32:0/2306915554 wait complete. 2021-04-09T18:33:57.705 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 417, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:33:39.367894', 'modified': '2021-04-09 18:33:51.410433', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 12022, 'mds_1': 12519}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12022': {'gid': 12022, 'name': 'b', 'rank': 0, 'incarnation': 408, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/154900016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 154900016}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 154900016}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12519': {'gid': 12519, 'name': 'a', 'rank': 1, 'incarnation': 413, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6833/2947726765', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2947726765}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2947726765}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12522': {'gid': 12522, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6835/3605541389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3605541389}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3605541389}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_59714': {'gid': 59714, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.86:6837/1243122766', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1243122766}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1243122766}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:33:57.706 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_12519:up:stopping 2021-04-09T18:33:58.706 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:33:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.850 7f5d1f390700 1 Processor -- start 2021-04-09T18:33:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.850 7f5d1f390700 1 -- start start 2021-04-09T18:33:58.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.851 7f5d1f390700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d180d7ca0 0x7f5d180d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.851 7f5d1f390700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5d1810da00 0x7f5d1810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.851 7f5d1f390700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1810d480 con 0x7f5d1810d010 2021-04-09T18:33:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.851 7f5d1f390700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d180d9220 con 0x7f5d1810da00 2021-04-09T18:33:58.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.851 7f5d1f390700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d180d86c0 con 0x7f5d180d7ca0 2021-04-09T18:33:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.851 7f5d1d9ad700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5d1810da00 0x7f5d1810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.851 7f5d1d1ac700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5d1810d010 0x7f5d180d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40568/0 (socket says 172.21.15.32:40568) 2021-04-09T18:33:58.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.851 7f5d1d1ac700 1 -- 172.21.15.32:0/3769147885 learned_addr learned my addr 172.21.15.32:0/3769147885 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:33:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.851 7f5d1c9ab700 1 --2- 172.21.15.32:0/3769147885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d180d7ca0 0x7f5d180d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d1d9ad700 1 -- 172.21.15.32:0/3769147885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d180d7ca0 msgr2=0x7f5d180d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:33:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d1d9ad700 1 --2- 172.21.15.32:0/3769147885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d180d7ca0 0x7f5d180d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:58.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d1d9ad700 1 -- 172.21.15.32:0/3769147885 >> v1:172.21.15.32:6789/0 conn(0x7f5d1810d010 legacy=0x7f5d180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d1d9ad700 1 -- 172.21.15.32:0/3769147885 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d180d1f70 con 0x7f5d1810da00 2021-04-09T18:33:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d0ffff700 1 -- 172.21.15.32:0/3769147885 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439212733 0 0) 0x7f5d14000ce0 con 0x7f5d1810d010 2021-04-09T18:33:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d1d9ad700 1 --2- 172.21.15.32:0/3769147885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d1810da00 0x7f5d1810de00 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7f5d10006e30 tx=0x7f5d10009de0).ready entity=mon.0 client_cookie=7083be4ff079aec8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:58.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d0ffff700 1 -- 172.21.15.32:0/3769147885 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d1000b300 con 0x7f5d1810da00 2021-04-09T18:33:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d0ffff700 1 -- 172.21.15.32:0/3769147885 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1000b680 con 0x7f5d1810da00 2021-04-09T18:33:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d0ffff700 1 -- 172.21.15.32:0/3769147885 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d1000bb60 con 0x7f5d1810da00 2021-04-09T18:33:58.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d1f390700 1 -- 172.21.15.32:0/3769147885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d1810da00 msgr2=0x7f5d1810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.852 7f5d1f390700 1 --2- 172.21.15.32:0/3769147885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d1810da00 0x7f5d1810de00 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7f5d10006e30 tx=0x7f5d10009de0).stop 2021-04-09T18:33:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.853 7f5d1f390700 1 -- 172.21.15.32:0/3769147885 shutdown_connections 2021-04-09T18:33:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.853 7f5d1f390700 1 --2- 172.21.15.32:0/3769147885 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d1810da00 0x7f5d1810de00 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:58.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.853 7f5d1f390700 1 --2- 172.21.15.32:0/3769147885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d180d7ca0 0x7f5d180d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:58.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.853 7f5d1f390700 1 -- 172.21.15.32:0/3769147885 >> 172.21.15.32:0/3769147885 conn(0x7f5d180ce5f0 msgr2=0x7f5d180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:58.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.853 7f5d1f390700 1 -- 172.21.15.32:0/3769147885 shutdown_connections 2021-04-09T18:33:58.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.853 7f5d1f390700 1 -- 172.21.15.32:0/3769147885 wait complete. 2021-04-09T18:33:58.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.873 7f5d1f390700 1 Processor -- start 2021-04-09T18:33:58.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.873 7f5d1f390700 1 -- start start 2021-04-09T18:33:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1f390700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d181ac380 0x7f5d1818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:58.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1f390700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d181ac780 0x7f5d1818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:58.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1f390700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d18183520 0x7f5d18183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:58.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1f390700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d181ace10 con 0x7f5d181ac380 2021-04-09T18:33:58.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1f390700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d18189cf0 con 0x7f5d181ac780 2021-04-09T18:33:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1f390700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d18183f40 con 0x7f5d18183520 2021-04-09T18:33:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d1ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d181ac380 0x7f5d1818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d1ac700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d181ac380 0x7f5d1818a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59216/0 (socket says 172.21.15.32:59216) 2021-04-09T18:33:58.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d1ac700 1 -- 172.21.15.32:0/4249632155 learned_addr learned my addr 172.21.15.32:0/4249632155 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:33:58.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d9ad700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d18183520 0x7f5d18183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:58.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1c9ab700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d181ac780 0x7f5d1818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:58.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d1ac700 1 -- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d18183520 msgr2=0x7f5d18183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:58.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d1ac700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d18183520 0x7f5d18183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:58.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d1ac700 1 -- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d181ac780 msgr2=0x7f5d1818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:58.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d1ac700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d181ac780 0x7f5d1818a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:58.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d1ac700 1 -- 172.21.15.32:0/4249632155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d18184160 con 0x7f5d181ac380 2021-04-09T18:33:58.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1c9ab700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d181ac780 0x7f5d1818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:33:58.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.874 7f5d1d1ac700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d181ac380 0x7f5d1818a230 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7f5d1400b530 tx=0x7f5d1400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:58.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.875 7f5d0dffb700 1 -- 172.21.15.32:0/4249632155 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d14006020 con 0x7f5d181ac380 2021-04-09T18:33:58.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.875 7f5d0dffb700 1 -- 172.21.15.32:0/4249632155 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d140063a0 con 0x7f5d181ac380 2021-04-09T18:33:58.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.875 7f5d0dffb700 1 -- 172.21.15.32:0/4249632155 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d14004590 con 0x7f5d181ac380 2021-04-09T18:33:58.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.875 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d18184160 con 0x7f5d181ac380 2021-04-09T18:33:58.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.875 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d18184160 con 0x7f5d181ac380 2021-04-09T18:33:58.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.875 7f5d0dffb700 1 -- 172.21.15.32:0/4249632155 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5d1400a000 con 0x7f5d181ac380 2021-04-09T18:33:58.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.875 7f5d0dffb700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5cf80188c0 0x7f5cf801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:33:58.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.875 7f5d0dffb700 1 -- 172.21.15.32:0/4249632155 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f5d14032a00 con 0x7f5d181ac380 2021-04-09T18:33:58.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.876 7f5d1c9ab700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5cf80188c0 0x7f5cf801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:33:58.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.876 7f5d1c9ab700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5cf80188c0 0x7f5cf801adc0 crc :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:33:58.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.895 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d18182d10 con 0x7f5d181ac380 2021-04-09T18:33:58.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:58.899 7f5d0dffb700 1 -- 172.21.15.32:0/4249632155 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5d14006b90 con 0x7f5d181ac380 2021-04-09T18:33:59.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.088 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d18182d10 con 0x7f5d181ac380 2021-04-09T18:33:59.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.088 7f5d0dffb700 1 -- 172.21.15.32:0/4249632155 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 417 v417) v1 ==== 95+0+2819 (secure 0 0 0) 0x7f5d1400a000 con 0x7f5d181ac380 2021-04-09T18:33:59.106 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 417 2021-04-09T18:33:59.106 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:33:59.106 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":417,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":417,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:51.410433","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0,1],"up":{"mds_0":12022,"mds_1":12519},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12519":{"gid":12519,"name":"a","rank":1,"incarnation":413,"state":"up:stopping","state_seq":5,"addr":"172.21.15.32:6833/2947726765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2947726765},{"type":"v1","addr":"172.21.15.32:6833","nonce":2947726765}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12522":{"gid":12522,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_59714":{"gid":59714,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.86:6837/1243122766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1243122766},{"type":"v1","addr":"172.21.15.86:6837","nonce":1243122766}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:33:59.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5cf80188c0 msgr2=0x7f5cf801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:59.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5cf80188c0 0x7f5cf801adc0 crc :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:59.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d181ac380 msgr2=0x7f5d1818a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:33:59.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d181ac380 0x7f5d1818a230 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7f5d1400b530 tx=0x7f5d1400b8d0).stop 2021-04-09T18:33:59.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 shutdown_connections 2021-04-09T18:33:59.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5cf80188c0 0x7f5cf801adc0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:59.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d18183520 0x7f5d18183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:59.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d181ac780 0x7f5d1818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:59.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 --2- 172.21.15.32:0/4249632155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d181ac380 0x7f5d1818a230 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:33:59.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 >> 172.21.15.32:0/4249632155 conn(0x7f5d1819b890 msgr2=0x7f5d181a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:33:59.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 shutdown_connections 2021-04-09T18:33:59.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:33:59.091 7f5d1f390700 1 -- 172.21.15.32:0/4249632155 wait complete. 2021-04-09T18:33:59.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 417, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:33:39.367894', 'modified': '2021-04-09 18:33:51.410433', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 12022, 'mds_1': 12519}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12022': {'gid': 12022, 'name': 'b', 'rank': 0, 'incarnation': 408, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/154900016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 154900016}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 154900016}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12519': {'gid': 12519, 'name': 'a', 'rank': 1, 'incarnation': 413, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.32:6833/2947726765', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2947726765}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 2947726765}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12522': {'gid': 12522, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6835/3605541389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3605541389}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3605541389}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_59714': {'gid': 59714, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.86:6837/1243122766', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 1243122766}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 1243122766}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:33:59.520 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_12519:up:stopping 2021-04-09T18:33:59.636 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:33:59.639 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:34:00.521 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:00.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.666 7fab02bf6700 1 Processor -- start 2021-04-09T18:34:00.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.666 7fab02bf6700 1 -- start start 2021-04-09T18:34:00.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.666 7fab02bf6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faafc10d010 0x7faafc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.666 7fab02bf6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faafc10d480 con 0x7faafc0d8490 2021-04-09T18:34:00.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.666 7fab02bf6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faafc10e1f0 con 0x7faafc0d1da0 2021-04-09T18:34:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.666 7fab02bf6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faafc0d8d30 con 0x7faafc10d010 2021-04-09T18:34:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.666 7fab00a12700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7faafc10d010 0x7faafc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:00.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.666 7fab01213700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7faafc0d8490 0x7faafc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40594/0 (socket says 172.21.15.32:40594) 2021-04-09T18:34:00.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.666 7fab01213700 1 -- 172.21.15.32:0/1189942355 learned_addr learned my addr 172.21.15.32:0/1189942355 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:00.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7fab00a12700 1 -- 172.21.15.32:0/1189942355 >> v1:172.21.15.86:6789/0 conn(0x7faafc0d1da0 legacy=0x7faafc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:00.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7fab00a12700 1 -- 172.21.15.32:0/1189942355 >> v1:172.21.15.32:6789/0 conn(0x7faafc0d8490 legacy=0x7faafc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:00.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7faafb7fe700 1 -- 172.21.15.32:0/1189942355 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096432052 0 0) 0x7faaec000fe0 con 0x7faafc0d1da0 2021-04-09T18:34:00.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7fab00a12700 1 -- 172.21.15.32:0/1189942355 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc0d9220 con 0x7faafc10d010 2021-04-09T18:34:00.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7fab00a12700 1 --2- 172.21.15.32:0/1189942355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc10d010 0x7faafc0d1780 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 rx=0x7faae4006650 tx=0x7faae4009a00).ready entity=mon.0 client_cookie=33dd0b6f1abfd395 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:00.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7faafb7fe700 1 -- 172.21.15.32:0/1189942355 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faae4009df0 con 0x7faafc10d010 2021-04-09T18:34:00.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7faafb7fe700 1 -- 172.21.15.32:0/1189942355 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae40009b0 con 0x7faafc10d010 2021-04-09T18:34:00.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7faafb7fe700 1 -- 172.21.15.32:0/1189942355 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faae400c2b0 con 0x7faafc10d010 2021-04-09T18:34:00.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7fab02bf6700 1 -- 172.21.15.32:0/1189942355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc10d010 msgr2=0x7faafc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:00.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.667 7fab02bf6700 1 --2- 172.21.15.32:0/1189942355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc10d010 0x7faafc0d1780 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 rx=0x7faae4006650 tx=0x7faae4009a00).stop 2021-04-09T18:34:00.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.668 7fab02bf6700 1 -- 172.21.15.32:0/1189942355 shutdown_connections 2021-04-09T18:34:00.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.668 7fab02bf6700 1 --2- 172.21.15.32:0/1189942355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc10d010 0x7faafc0d1780 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:00.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.668 7fab02bf6700 1 -- 172.21.15.32:0/1189942355 >> 172.21.15.32:0/1189942355 conn(0x7faafc0ce5f0 msgr2=0x7faafc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:00.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.668 7fab02bf6700 1 -- 172.21.15.32:0/1189942355 shutdown_connections 2021-04-09T18:34:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.668 7fab02bf6700 1 -- 172.21.15.32:0/1189942355 wait complete. 2021-04-09T18:34:00.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.688 7fab02bf6700 1 Processor -- start 2021-04-09T18:34:00.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.688 7fab02bf6700 1 -- start start 2021-04-09T18:34:00.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab02bf6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faafc0af100 0x7faafc090b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:00.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab02bf6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faafc1868f0 0x7faafc185ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:00.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab02bf6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc1a9470 0x7faafc1a9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:00.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab02bf6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc1860d0 con 0x7faafc1a9470 2021-04-09T18:34:00.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab02bf6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc186cf0 con 0x7faafc1868f0 2021-04-09T18:34:00.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab02bf6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faafc195280 con 0x7faafc0af100 2021-04-09T18:34:00.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab01213700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc1a9470 0x7faafc1a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:00.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab01213700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc1a9470 0x7faafc1a9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59242/0 (socket says 172.21.15.32:59242) 2021-04-09T18:34:00.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab01213700 1 -- 172.21.15.32:0/2693061225 learned_addr learned my addr 172.21.15.32:0/2693061225 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:00.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab00a12700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faafc0af100 0x7faafc090b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:00.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7faafbfff700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faafc1868f0 0x7faafc185ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:00.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab01213700 1 -- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faafc0af100 msgr2=0x7faafc090b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab01213700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faafc0af100 0x7faafc090b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab01213700 1 -- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faafc1868f0 msgr2=0x7faafc185ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:00.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab01213700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faafc1868f0 0x7faafc185ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab01213700 1 -- 172.21.15.32:0/2693061225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc09d9d0 con 0x7faafc1a9470 2021-04-09T18:34:00.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.689 7fab01213700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc1a9470 0x7faafc1a9870 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7faaf40014b0 tx=0x7faaf400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.690 7faaf97fa700 1 -- 172.21.15.32:0/2693061225 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaf4000ef0 con 0x7faafc1a9470 2021-04-09T18:34:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.690 7faaf97fa700 1 -- 172.21.15.32:0/2693061225 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf400acc0 con 0x7faafc1a9470 2021-04-09T18:34:00.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.690 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faafc09d9d0 con 0x7faafc1a9470 2021-04-09T18:34:00.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.690 7faaf97fa700 1 -- 172.21.15.32:0/2693061225 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaf4004620 con 0x7faafc1a9470 2021-04-09T18:34:00.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.690 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faafc09d9d0 con 0x7faafc1a9470 2021-04-09T18:34:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.690 7faaf97fa700 1 -- 172.21.15.32:0/2693061225 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7faaf40043a0 con 0x7faafc1a9470 2021-04-09T18:34:00.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.690 7faaf97fa700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faae0018930 0x7faae001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:00.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.690 7faaf97fa700 1 -- 172.21.15.32:0/2693061225 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7faaf40329d0 con 0x7faafc1a9470 2021-04-09T18:34:00.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.690 7fab00a12700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faae0018930 0x7faae001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:00.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.691 7fab00a12700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faae0018930 0x7faae001ae10 crc :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:00.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.710 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faafc10d480 con 0x7faafc1a9470 2021-04-09T18:34:00.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.714 7faaf97fa700 1 -- 172.21.15.32:0/2693061225 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7faaf4000940 con 0x7faafc1a9470 2021-04-09T18:34:00.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.904 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faafc10d480 con 0x7faafc1a9470 2021-04-09T18:34:00.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.905 7faaf97fa700 1 -- 172.21.15.32:0/2693061225 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 418 v418) v1 ==== 95+0+2118 (secure 0 0 0) 0x7faaf40043a0 con 0x7faafc1a9470 2021-04-09T18:34:00.922 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 418 2021-04-09T18:34:00.923 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:00.923 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":418,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":418,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:59.558064","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12022},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12522":{"gid":12522,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:34:00.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faae0018930 msgr2=0x7faae001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faae0018930 0x7faae001ae10 crc :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc1a9470 msgr2=0x7faafc1a9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc1a9470 0x7faafc1a9870 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7faaf40014b0 tx=0x7faaf400a8d0).stop 2021-04-09T18:34:00.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 shutdown_connections 2021-04-09T18:34:00.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7faae0018930 0x7faae001ae10 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:00.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faafc1a9470 0x7faafc1a9870 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:00.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faafc1868f0 0x7faafc185ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:00.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 --2- 172.21.15.32:0/2693061225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faafc0af100 0x7faafc090b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 >> 172.21.15.32:0/2693061225 conn(0x7faafc1965c0 msgr2=0x7faafc1ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 shutdown_connections 2021-04-09T18:34:00.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:00.907 7fab02bf6700 1 -- 172.21.15.32:0/2693061225 wait complete. 2021-04-09T18:34:01.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 418, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:33:39.367894', 'modified': '2021-04-09 18:33:59.558064', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12022}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_12022': {'gid': 12022, 'name': 'b', 'rank': 0, 'incarnation': 408, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/154900016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 154900016}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 154900016}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12522': {'gid': 12522, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6835/3605541389', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 3605541389}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 3605541389}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:01.337 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:01.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-04-09T18:34:01.474 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:34:01.475 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:01.475 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:34:01.476 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 12022, 2021-04-09T18:34:01.476 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:34:01.477 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:34:01.477 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 22.087795652000001, 2021-04-09T18:34:01.477 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 418, 2021-04-09T18:34:01.478 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 251, 2021-04-09T18:34:01.478 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 251, 2021-04-09T18:34:01.479 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 27.533119391 2021-04-09T18:34:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:34:01.481 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12022, "want_state": "up:active", "state": "up:active", "rank_uptime": 22.087795652000001, "mdsmap_epoch": 418, "osdmap_epoch": 251, "osdmap_epoch_barrier": 251, "uptime": 27.533119391 } 2021-04-09T18:34:01.482 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:34:01.482 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:34:01.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:34:01.607 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:34:01.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:01.609 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:34:01.610 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:34:01.657 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:34:01.657 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:34:01.658 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:34:01.658 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:34:01.659 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:34:01.727 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[105578]: fuse finished with error 0 and tester_r 0 2021-04-09T18:34:01.729 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:01.730 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:34:01.731 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:34:01.776 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:34:01.776 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:34:01.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:34:01.868 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:34:01.912 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:01.912 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:34:01.913 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:34:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:34:01.957 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:34:01.958 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:34:01.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:34:01.959 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:34:02.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:34:02.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.173 7f4dff8cc700 1 Processor -- start 2021-04-09T18:34:02.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.173 7f4dff8cc700 1 -- start start 2021-04-09T18:34:02.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.173 7f4dff8cc700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4df80d6c80 0x7f4df80d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.212 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dff8cc700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80d7280 con 0x7f4df80cffa0 2021-04-09T18:34:02.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dff8cc700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80d2840 con 0x7f4df80d2440 2021-04-09T18:34:02.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dff8cc700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df80d7080 con 0x7f4df80d6c80 2021-04-09T18:34:02.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dfd6e8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4df80cffa0 0x7f4df80d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40604/0 (socket says 172.21.15.32:40604) 2021-04-09T18:34:02.213 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dfd6e8700 1 -- 172.21.15.32:0/1588348664 learned_addr learned my addr 172.21.15.32:0/1588348664 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dfcee7700 1 --2- 172.21.15.32:0/1588348664 >> v2:172.21.15.86:3300/0 conn(0x7f4df80d6c80 0x7f4df80d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4deffff700 1 -- 172.21.15.32:0/1588348664 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170674605 0 0) 0x7f4df4000ce0 con 0x7f4df80cffa0 2021-04-09T18:34:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4deffff700 1 -- 172.21.15.32:0/1588348664 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de4002fe0 con 0x7f4df80cffa0 2021-04-09T18:34:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4deffff700 1 -- 172.21.15.32:0/1588348664 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123479266 0 0) 0x7f4df0000d30 con 0x7f4df80d2440 2021-04-09T18:34:02.214 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4deffff700 1 -- 172.21.15.32:0/1588348664 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de4003210 con 0x7f4df80d2440 2021-04-09T18:34:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dfcee7700 1 -- 172.21.15.32:0/1588348664 >> v1:172.21.15.86:6789/0 conn(0x7f4df80d2440 legacy=0x7f4df810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dfcee7700 1 -- 172.21.15.32:0/1588348664 >> v1:172.21.15.32:6789/0 conn(0x7f4df80cffa0 legacy=0x7f4df80d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dfcee7700 1 -- 172.21.15.32:0/1588348664 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df80d91e0 con 0x7f4df80d6c80 2021-04-09T18:34:02.215 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.174 7f4dfcee7700 1 --2- 172.21.15.32:0/1588348664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4df80d6c80 0x7f4df80d7ff0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f4de8006650 tx=0x7f4de8009a80).ready entity=mon.1 client_cookie=7483041e38fc8c13 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:02.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4deffff700 1 -- 172.21.15.32:0/1588348664 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4de80009b0 con 0x7f4df80d6c80 2021-04-09T18:34:02.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4deffff700 1 -- 172.21.15.32:0/1588348664 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de800c0c0 con 0x7f4df80d6c80 2021-04-09T18:34:02.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4deffff700 1 -- 172.21.15.32:0/1588348664 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4de800c510 con 0x7f4df80d6c80 2021-04-09T18:34:02.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4dff8cc700 1 -- 172.21.15.32:0/1588348664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4df80d6c80 msgr2=0x7f4df80d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.216 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4dff8cc700 1 --2- 172.21.15.32:0/1588348664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4df80d6c80 0x7f4df80d7ff0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f4de8006650 tx=0x7f4de8009a80).stop 2021-04-09T18:34:02.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4dff8cc700 1 -- 172.21.15.32:0/1588348664 shutdown_connections 2021-04-09T18:34:02.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4dff8cc700 1 --2- 172.21.15.32:0/1588348664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4df80d6c80 0x7f4df80d7ff0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4dff8cc700 1 -- 172.21.15.32:0/1588348664 >> 172.21.15.32:0/1588348664 conn(0x7f4df80ce5c0 msgr2=0x7f4df80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:02.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4dff8cc700 1 -- 172.21.15.32:0/1588348664 shutdown_connections 2021-04-09T18:34:02.217 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.175 7f4dff8cc700 1 -- 172.21.15.32:0/1588348664 wait complete. 2021-04-09T18:34:02.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dff8cc700 1 Processor -- start 2021-04-09T18:34:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dff8cc700 1 -- start start 2021-04-09T18:34:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dff8cc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df81b90a0 0x7f4df808e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dff8cc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4df81b94a0 0x7f4df808ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dff8cc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4df808b880 0x7f4df808bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dff8cc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df808f370 con 0x7f4df81b90a0 2021-04-09T18:34:02.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dff8cc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df808c2a0 con 0x7f4df81b94a0 2021-04-09T18:34:02.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dff8cc700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df808c4c0 con 0x7f4df808b880 2021-04-09T18:34:02.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dfd6e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df81b90a0 0x7f4df808e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dfd6e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df81b90a0 0x7f4df808e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59252/0 (socket says 172.21.15.32:59252) 2021-04-09T18:34:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.196 7f4dfd6e8700 1 -- 172.21.15.32:0/3095159734 learned_addr learned my addr 172.21.15.32:0/3095159734 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dfdee9700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4df808b880 0x7f4df808bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dfcee7700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4df81b94a0 0x7f4df808ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dfd6e8700 1 -- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4df808b880 msgr2=0x7f4df808bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.223 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dfd6e8700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4df808b880 0x7f4df808bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dfd6e8700 1 -- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4df81b94a0 msgr2=0x7f4df808ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dfd6e8700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4df81b94a0 0x7f4df808ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dfd6e8700 1 -- 172.21.15.32:0/3095159734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df808a650 con 0x7f4df81b90a0 2021-04-09T18:34:02.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dfd6e8700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df81b90a0 0x7f4df808e6b0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 rx=0x7f4df4006f00 tx=0x7f4df400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:02.224 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dedffb700 1 -- 172.21.15.32:0/3095159734 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4df400bd30 con 0x7f4df81b90a0 2021-04-09T18:34:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dedffb700 1 -- 172.21.15.32:0/3095159734 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df4006030 con 0x7f4df81b90a0 2021-04-09T18:34:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df808a650 con 0x7f4df81b90a0 2021-04-09T18:34:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dedffb700 1 -- 172.21.15.32:0/3095159734 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4df4004590 con 0x7f4df81b90a0 2021-04-09T18:34:02.225 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.197 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df808a650 con 0x7f4df81b90a0 2021-04-09T18:34:02.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.198 7f4dedffb700 1 -- 172.21.15.32:0/3095159734 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4df40043a0 con 0x7f4df81b90a0 2021-04-09T18:34:02.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.198 7f4dedffb700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4dd8018930 0x7f4dd801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.198 7f4dedffb700 1 -- 172.21.15.32:0/3095159734 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4df40329c0 con 0x7f4df81b90a0 2021-04-09T18:34:02.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.198 7f4dfcee7700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4dd8018930 0x7f4dd801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.226 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.198 7f4dfcee7700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4dd8018930 0x7f4dd801ae10 crc :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:02.234 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.218 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df808c4c0 con 0x7f4df81b90a0 2021-04-09T18:34:02.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.221 7f4dedffb700 1 -- 172.21.15.32:0/3095159734 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4df40009b0 con 0x7f4df81b90a0 2021-04-09T18:34:02.425 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.409 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f4df808c4c0 con 0x7f4df81b90a0 2021-04-09T18:34:02.426 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.410 7f4dedffb700 1 -- 172.21.15.32:0/3095159734 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v251) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f4df4006030 con 0x7f4df81b90a0 2021-04-09T18:34:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:02.429 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 251, 2021-04-09T18:34:02.429 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:34:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:33:40.369286", 2021-04-09T18:34:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:34:02.430 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:34:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:34:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:34:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:34:02.431 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:34:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:34:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:34:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:34:02.432 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:34:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:34:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:34:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:34:02.433 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:34:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:34:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 43, 2021-04-09T18:34:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:34:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:34:02.434 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:34:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:34:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:34:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.435 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:34:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:34:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:34:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:34:02.436 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:34:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:34:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:34:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:34:02.437 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:34:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:34:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:34:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:34:02.438 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:34:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:34:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:34:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:34:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:34:02.439 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:34:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:34:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:34:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:34:02.440 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:34:02.441 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:34:02.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.441 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:34:02.441 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:34:02.442 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:34:02.442 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:34:02.442 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:34:02.442 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:34:02.443 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:34:02.443 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:34:02.443 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:34:02.443 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:34:02.444 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:34:02.444 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:34:02.444 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:34:02.444 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:34:02.444 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:34:02.445 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:34:02.445 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:34:02.445 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:34:02.445 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:34:02.446 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:34:02.446 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:34:02.446 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:34:02.446 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:34:02.446 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:34:02.447 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:34:02.447 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:34:02.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:34:02.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.448 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:34:02.448 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:34:02.448 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:34:02.448 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:34:02.449 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:34:02.449 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:34:02.449 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:34:02.449 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:34:02.449 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:34:02.450 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:34:02.450 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:34:02.450 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:34:02.450 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:34:02.451 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:34:02.451 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.452 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 42, 2021-04-09T18:34:02.452 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:34:02.452 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:33:36.709256", 2021-04-09T18:34:02.452 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:34:02.453 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:34:02.453 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:34:02.453 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:34:02.453 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:34:02.453 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:34:02.454 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:34:02.454 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:34:02.454 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:34:02.454 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:34:02.455 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:34:02.455 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:34:02.455 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:34:02.455 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:34:02.456 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:34:02.456 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:34:02.456 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:34:02.456 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:34:02.457 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:34:02.457 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:34:02.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.457 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "250", 2021-04-09T18:34:02.457 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:34:02.458 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:34:02.458 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:34:02.458 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:34:02.458 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:34:02.459 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:34:02.459 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:34:02.459 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:34:02.459 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:34:02.460 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:34:02.460 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:34:02.460 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:34:02.460 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:34:02.461 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:34:02.461 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:34:02.461 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:34:02.461 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:34:02.462 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:34:02.462 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:34:02.462 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:34:02.462 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:34:02.463 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:34:02.463 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:34:02.463 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:34:02.463 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:34:02.464 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:34:02.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.464 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:34:02.464 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:34:02.465 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:34:02.465 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:34:02.465 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:34:02.465 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:34:02.466 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:34:02.466 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:34:02.466 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:34:02.466 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:34:02.467 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:34:02.467 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:34:02.467 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:34:02.468 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:34:02.468 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:34:02.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.468 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:34:02.469 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:34:02.469 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:34:02.469 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.469 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.470 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.470 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 43, 2021-04-09T18:34:02.470 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:34:02.471 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:33:38.131023", 2021-04-09T18:34:02.471 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:34:02.471 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:34:02.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:34:02.472 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:34:02.472 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:34:02.472 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:34:02.472 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:34:02.473 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:34:02.473 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:34:02.473 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:34:02.474 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:34:02.474 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:34:02.474 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:34:02.474 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:34:02.475 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:34:02.475 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:34:02.475 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:34:02.475 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:34:02.476 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:34:02.476 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:34:02.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.476 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "251", 2021-04-09T18:34:02.477 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:34:02.477 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:34:02.477 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:34:02.477 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:34:02.478 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:34:02.478 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:34:02.478 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:34:02.478 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:34:02.479 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:34:02.479 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:34:02.479 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:34:02.479 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:34:02.479 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:34:02.480 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:34:02.480 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:34:02.480 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:34:02.480 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:34:02.481 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:34:02.481 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:34:02.481 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:34:02.481 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:34:02.482 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:34:02.482 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:34:02.482 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:34:02.482 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:34:02.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:34:02.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.483 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:34:02.483 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:34:02.483 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:34:02.484 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:34:02.484 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:34:02.484 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:34:02.484 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:34:02.485 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:34:02.485 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:34:02.485 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:34:02.485 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:34:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:34:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:34:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:34:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:34:02.486 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.487 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.487 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:34:02.488 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:34:02.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.488 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:34:02.488 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:34:02.489 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:02.489 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:02.489 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:02.489 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:02.490 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:02.490 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:02.490 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:02.490 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 227, 2021-04-09T18:34:02.490 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:02.491 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:02.491 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:02.491 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.492 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.492 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.492 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:34:02.492 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:02.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.493 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.493 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:34:02.494 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:02.494 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.494 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.495 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:02.495 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.495 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.496 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.496 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:34:02.496 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:02.496 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.497 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.497 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:34:02.497 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:02.498 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.498 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.498 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:02.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.499 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.499 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.499 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:34:02.500 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:02.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.500 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.500 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.501 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:34:02.501 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:02.501 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.501 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.502 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:02.502 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.502 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.503 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:34:02.503 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:02.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.506 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:34:02.506 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:02.506 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.507 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.508 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:34:02.508 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:34:02.508 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:34:02.509 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:34:02.509 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:02.509 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:02.510 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:02.510 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.512 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:34:02.512 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:34:02.513 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:02.513 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:02.513 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:02.514 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:02.514 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:02.515 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:02.515 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:02.516 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:02.516 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:02.516 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:02.516 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:02.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.517 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.517 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:34:02.518 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:02.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.518 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.518 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:34:02.519 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:02.519 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.519 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.520 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:02.520 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.520 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.521 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.521 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:34:02.521 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:02.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.522 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.522 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.522 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:34:02.522 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:02.522 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.523 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.523 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:02.523 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.523 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.524 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.524 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:34:02.524 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:02.524 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.525 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.525 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:34:02.525 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:02.525 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.526 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.526 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:02.526 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.526 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.527 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.527 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:34:02.527 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:02.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.528 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.528 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.528 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:34:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.529 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:34:02.530 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:34:02.530 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:34:02.530 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:34:02.530 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:02.531 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:02.531 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:02.531 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.531 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.532 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:34:02.532 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:34:02.532 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:02.532 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:02.533 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:02.533 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:02.533 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:02.533 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:02.534 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:02.534 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:02.535 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:02.535 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:02.535 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:02.535 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.536 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.536 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:34:02.536 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:02.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.537 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.537 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.537 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:34:02.537 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:02.537 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.538 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.538 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:02.539 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.539 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:34:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.540 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:34:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:02.541 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.543 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:02.543 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.543 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:34:02.544 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:02.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.545 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:34:02.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:02.546 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.546 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.546 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:02.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.547 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:34:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.548 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:34:02.549 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:02.549 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.549 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.550 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:34:02.550 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:34:02.550 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:34:02.550 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:34:02.550 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:02.551 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:02.551 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:02.551 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.551 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.552 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.552 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:34:02.552 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:34:02.552 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:02.552 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:02.553 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:02.553 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:02.553 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:02.553 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:02.553 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:02.554 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:02.554 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:02.554 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:02.554 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:02.555 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.555 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.555 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.556 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:34:02.556 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:02.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.556 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.557 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:34:02.557 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:02.558 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.558 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.559 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:02.559 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.559 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.560 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:34:02.560 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:02.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.560 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.561 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.561 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:34:02.561 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:02.561 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.561 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.562 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:02.562 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.562 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.563 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:34:02.563 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:02.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.563 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.564 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.564 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:34:02.564 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:02.564 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.564 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.565 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.565 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:02.565 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.565 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.566 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:34:02.566 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:02.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:34:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:02.567 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:34:02.568 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:34:02.569 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:34:02.569 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:34:02.569 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:02.569 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:02.569 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:02.570 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.570 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.570 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.570 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:34:02.570 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:34:02.571 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:02.571 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:02.571 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:02.571 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:02.572 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:02.572 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:02.572 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:02.572 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 247, 2021-04-09T18:34:02.572 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:02.573 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:02.573 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:02.573 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.573 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.573 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.574 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:34:02.574 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:02.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.574 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.575 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:34:02.575 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:02.575 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.575 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.576 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:02.576 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.576 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.577 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:34:02.577 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:02.577 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.577 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.578 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:34:02.578 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:02.578 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.578 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.579 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:02.579 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.580 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.580 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.580 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:34:02.580 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:02.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.581 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.581 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.581 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:34:02.581 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:02.582 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.582 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.582 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.582 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:02.582 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.583 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.583 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:34:02.583 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:02.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.584 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.584 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.584 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:34:02.584 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:02.585 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.585 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.585 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:34:02.586 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:34:02.586 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:34:02.586 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:34:02.586 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:02.586 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:02.587 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:02.587 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.587 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.588 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:34:02.588 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:34:02.588 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:02.588 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:02.588 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:02.589 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:02.589 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:02.589 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:02.589 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:02.590 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:34:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:02.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:34:02.592 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:02.593 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:34:02.594 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.595 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:34:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.596 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:34:02.597 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.598 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:34:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.599 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:02.600 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.600 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.600 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:34:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.601 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:34:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.602 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:34:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:34:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:34:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:34:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:02.603 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.604 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:34:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:34:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:02.605 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:02.606 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.607 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:34:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:02.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:34:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:02.609 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.610 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:34:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:02.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:34:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:02.612 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.613 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:34:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.614 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:34:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:02.615 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.616 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:34:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.617 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:34:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:02.618 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:34:02.619 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:34:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:34:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:34:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:02.620 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.621 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:34:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:34:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:02.622 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:02.623 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 244, 2021-04-09T18:34:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:02.624 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:34:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:02.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:34:02.626 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:02.627 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:34:02.628 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:34:02.629 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:02.630 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:34:02.631 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:34:02.632 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:02.633 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:34:02.634 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:34:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:34:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:34:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:34:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:34:02.637 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:02.638 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:34:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:34:02.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:34:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:02.640 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.641 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:34:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:02.642 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:34:02.643 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:02.644 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:34:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:02.645 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:02.646 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:34:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:02.647 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.648 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:34:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:02.649 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.650 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:34:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:02.651 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:34:02.652 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:02.653 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:34:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:34:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:34:02.654 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:34:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:34:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:34:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:34:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:34:02.655 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:34:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:34:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:34:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:34:02.656 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:34:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:34:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:34:02.657 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:34:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:34:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:34:02.658 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:34:02.658 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:02.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4dd8018930 msgr2=0x7f4dd801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4dd8018930 0x7f4dd801ae10 crc :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df81b90a0 msgr2=0x7f4df808e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df81b90a0 0x7f4df808e6b0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 rx=0x7f4df4006f00 tx=0x7f4df400b940).stop 2021-04-09T18:34:02.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 shutdown_connections 2021-04-09T18:34:02.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4dd8018930 0x7f4dd801ae10 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4df808b880 0x7f4df808bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4df81b94a0 0x7f4df808ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 --2- 172.21.15.32:0/3095159734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4df81b90a0 0x7f4df808e6b0 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 >> 172.21.15.32:0/3095159734 conn(0x7f4df81a9590 msgr2=0x7f4df818ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 shutdown_connections 2021-04-09T18:34:02.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.414 7f4dff8cc700 1 -- 172.21.15.32:0/3095159734 wait complete. 2021-04-09T18:34:02.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:02.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.797 7f8d7181f700 1 Processor -- start 2021-04-09T18:34:02.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.797 7f8d7181f700 1 -- start start 2021-04-09T18:34:02.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d7181f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8d6c0d6c80 0x7f8d6c0d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d7181f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6c0d7280 con 0x7f8d6c0d2440 2021-04-09T18:34:02.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d7181f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6c0d2840 con 0x7f8d6c0cffa0 2021-04-09T18:34:02.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d7181f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d6c0d7080 con 0x7f8d6c0d6c80 2021-04-09T18:34:02.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6bd75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8d6c0d2440 0x7f8d6c10cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40620/0 (socket says 172.21.15.32:40620) 2021-04-09T18:34:02.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6bd75700 1 -- 172.21.15.32:0/253502814 learned_addr learned my addr 172.21.15.32:0/253502814 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:02.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6ad73700 1 --2- 172.21.15.32:0/253502814 >> v2:172.21.15.86:3300/0 conn(0x7f8d6c0d6c80 0x7f8d6c0d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6a572700 1 -- 172.21.15.32:0/253502814 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964207411 0 0) 0x7f8d640009b0 con 0x7f8d6c0d2440 2021-04-09T18:34:02.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6a572700 1 -- 172.21.15.32:0/253502814 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d54002fe0 con 0x7f8d6c0d2440 2021-04-09T18:34:02.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6a572700 1 -- 172.21.15.32:0/253502814 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529696190 0 0) 0x7f8d5c000c60 con 0x7f8d6c0cffa0 2021-04-09T18:34:02.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6a572700 1 -- 172.21.15.32:0/253502814 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d54003090 con 0x7f8d6c0cffa0 2021-04-09T18:34:02.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6ad73700 1 -- 172.21.15.32:0/253502814 >> v1:172.21.15.86:6789/0 conn(0x7f8d6c0cffa0 legacy=0x7f8d6c0d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6ad73700 1 -- 172.21.15.32:0/253502814 >> v1:172.21.15.32:6789/0 conn(0x7f8d6c0d2440 legacy=0x7f8d6c10cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6ad73700 1 -- 172.21.15.32:0/253502814 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c0d91e0 con 0x7f8d6c0d6c80 2021-04-09T18:34:02.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6ad73700 1 --2- 172.21.15.32:0/253502814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d6c0d6c80 0x7f8d6c0d7ff0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f8d600069d0 tx=0x7f8d60009e40).ready entity=mon.1 client_cookie=8f609337dd44e3b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:02.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.798 7f8d6a572700 1 -- 172.21.15.32:0/253502814 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d6000b510 con 0x7f8d6c0d6c80 2021-04-09T18:34:02.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.799 7f8d6a572700 1 -- 172.21.15.32:0/253502814 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6000b890 con 0x7f8d6c0d6c80 2021-04-09T18:34:02.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.799 7f8d6a572700 1 -- 172.21.15.32:0/253502814 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d6000bd70 con 0x7f8d6c0d6c80 2021-04-09T18:34:02.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.799 7f8d7181f700 1 -- 172.21.15.32:0/253502814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d6c0d6c80 msgr2=0x7f8d6c0d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.799 7f8d7181f700 1 --2- 172.21.15.32:0/253502814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d6c0d6c80 0x7f8d6c0d7ff0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f8d600069d0 tx=0x7f8d60009e40).stop 2021-04-09T18:34:02.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.799 7f8d7181f700 1 -- 172.21.15.32:0/253502814 shutdown_connections 2021-04-09T18:34:02.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.799 7f8d7181f700 1 --2- 172.21.15.32:0/253502814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d6c0d6c80 0x7f8d6c0d7ff0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.799 7f8d7181f700 1 -- 172.21.15.32:0/253502814 >> 172.21.15.32:0/253502814 conn(0x7f8d6c0ce5c0 msgr2=0x7f8d6c0cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.799 7f8d7181f700 1 -- 172.21.15.32:0/253502814 shutdown_connections 2021-04-09T18:34:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.799 7f8d7181f700 1 -- 172.21.15.32:0/253502814 wait complete. 2021-04-09T18:34:02.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.819 7f8d7181f700 1 Processor -- start 2021-04-09T18:34:02.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.819 7f8d7181f700 1 -- start start 2021-04-09T18:34:02.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d7181f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d6c18e990 0x7f8d6c18ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d7181f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d6c08bd20 0x7f8d6c08c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d7181f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d6c08a650 0x7f8d6c08aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d7181f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c08c740 con 0x7f8d6c18e990 2021-04-09T18:34:02.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d7181f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c08b070 con 0x7f8d6c08a650 2021-04-09T18:34:02.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d7181f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c08b290 con 0x7f8d6c08bd20 2021-04-09T18:34:02.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d6c18e990 0x7f8d6c18ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6b574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d6c18e990 0x7f8d6c18ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59270/0 (socket says 172.21.15.32:59270) 2021-04-09T18:34:02.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6b574700 1 -- 172.21.15.32:0/1967493708 learned_addr learned my addr 172.21.15.32:0/1967493708 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6ad73700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d6c08bd20 0x7f8d6c08c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6bd75700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d6c08a650 0x7f8d6c08aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6b574700 1 -- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d6c08bd20 msgr2=0x7f8d6c08c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6b574700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d6c08bd20 0x7f8d6c08c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6b574700 1 -- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d6c08a650 msgr2=0x7f8d6c08aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:02.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6b574700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d6c08a650 0x7f8d6c08aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:02.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6b574700 1 -- 172.21.15.32:0/1967493708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c08b4b0 con 0x7f8d6c18e990 2021-04-09T18:34:02.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.820 7f8d6b574700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d6c18e990 0x7f8d6c18ed90 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f8d5c006f10 tx=0x7f8d5c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:02.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.821 7f8d53fff700 1 -- 172.21.15.32:0/1967493708 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d5c00bd80 con 0x7f8d6c18e990 2021-04-09T18:34:02.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.821 7f8d53fff700 1 -- 172.21.15.32:0/1967493708 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c000e20 con 0x7f8d6c18e990 2021-04-09T18:34:02.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.821 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6c08b4b0 con 0x7f8d6c18e990 2021-04-09T18:34:02.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.821 7f8d53fff700 1 -- 172.21.15.32:0/1967493708 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d5c0054f0 con 0x7f8d6c18e990 2021-04-09T18:34:02.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.821 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6c08b4b0 con 0x7f8d6c18e990 2021-04-09T18:34:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.821 7f8d53fff700 1 -- 172.21.15.32:0/1967493708 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8d5c009fe0 con 0x7f8d6c18e990 2021-04-09T18:34:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.821 7f8d53fff700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d4c018930 0x7f8d4c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.822 7f8d53fff700 1 -- 172.21.15.32:0/1967493708 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f8d5c032a50 con 0x7f8d6c18e990 2021-04-09T18:34:02.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.822 7f8d6ad73700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d4c018930 0x7f8d4c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:02.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.822 7f8d6ad73700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d4c018930 0x7f8d4c01ae10 crc :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:02.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.842 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6c1b70c0 con 0x7f8d6c18e990 2021-04-09T18:34:02.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:02.845 7f8d53fff700 1 -- 172.21.15.32:0/1967493708 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8d5c000e20 con 0x7f8d6c18e990 2021-04-09T18:34:03.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.035 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d6c1b70c0 con 0x7f8d6c18e990 2021-04-09T18:34:03.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.036 7f8d53fff700 1 -- 172.21.15.32:0/1967493708 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 418 v418) v1 ==== 95+0+2118 (secure 0 0 0) 0x7f8d5c009fe0 con 0x7f8d6c18e990 2021-04-09T18:34:03.053 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 418 2021-04-09T18:34:03.053 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:03.054 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":418,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":418,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:33:59.558064","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12022},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_12022":{"gid":12022,"name":"b","rank":0,"incarnation":408,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/154900016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":154900016},{"type":"v1","addr":"172.21.15.86:6836","nonce":154900016}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12522":{"gid":12522,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6835/3605541389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3605541389},{"type":"v1","addr":"172.21.15.32:6835","nonce":3605541389}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:34:03.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d4c018930 msgr2=0x7f8d4c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d4c018930 0x7f8d4c01ae10 crc :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d6c18e990 msgr2=0x7f8d6c18ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d6c18e990 0x7f8d6c18ed90 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f8d5c006f10 tx=0x7f8d5c00b990).stop 2021-04-09T18:34:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 shutdown_connections 2021-04-09T18:34:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8d4c018930 0x7f8d4c01ae10 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d6c08a650 0x7f8d6c08aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d6c08bd20 0x7f8d6c08c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 --2- 172.21.15.32:0/1967493708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d6c18e990 0x7f8d6c18ed90 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 >> 172.21.15.32:0/1967493708 conn(0x7f8d6c189b00 msgr2=0x7f8d6c1921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:03.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 shutdown_connections 2021-04-09T18:34:03.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.038 7f8d7181f700 1 -- 172.21.15.32:0/1967493708 wait complete. 2021-04-09T18:34:03.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:34:03.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.202 7f33efa6e700 1 Processor -- start 2021-04-09T18:34:03.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.202 7f33efa6e700 1 -- start start 2021-04-09T18:34:03.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33efa6e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f33e80d6c90 0x7f33e80d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33efa6e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33e80d2450 0x7f33e810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33efa6e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e80d7290 con 0x7f33e80cffb0 2021-04-09T18:34:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33efa6e700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e80d7090 con 0x7f33e80d6c90 2021-04-09T18:34:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33efa6e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e80d91f0 con 0x7f33e80d2450 2021-04-09T18:34:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ed089700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f33e80d6c90 0x7f33e80d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ed089700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f33e80d6c90 0x7f33e80d8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59278/0 (socket says 172.21.15.32:59278) 2021-04-09T18:34:03.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ed089700 1 -- 172.21.15.32:0/982474324 learned_addr learned my addr 172.21.15.32:0/982474324 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:03.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ee08b700 1 --2- 172.21.15.32:0/982474324 >> v2:172.21.15.86:3300/0 conn(0x7f33e80d2450 0x7f33e810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ed089700 1 -- 172.21.15.32:0/982474324 >> v2:172.21.15.86:3300/0 conn(0x7f33e80d2450 msgr2=0x7f33e810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:03.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ed089700 1 --2- 172.21.15.32:0/982474324 >> v2:172.21.15.86:3300/0 conn(0x7f33e80d2450 0x7f33e810cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ed089700 1 -- 172.21.15.32:0/982474324 >> v1:172.21.15.86:6789/0 conn(0x7f33e80cffb0 legacy=0x7f33e80d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ed089700 1 -- 172.21.15.32:0/982474324 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e80d2850 con 0x7f33e80d6c90 2021-04-09T18:34:03.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ec888700 1 -- 172.21.15.32:0/982474324 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366880665 0 0) 0x7f33e4000c60 con 0x7f33e80cffb0 2021-04-09T18:34:03.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.203 7f33ed089700 1 --2- 172.21.15.32:0/982474324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80d6c90 0x7f33e80d8000 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7f33d8006700 tx=0x7f33d8009b00).ready entity=mon.0 client_cookie=f8942fee2f60b78f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.204 7f33ec888700 1 -- 172.21.15.32:0/982474324 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33d800c0c0 con 0x7f33e80d6c90 2021-04-09T18:34:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.204 7f33ec888700 1 -- 172.21.15.32:0/982474324 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d800c400 con 0x7f33e80d6c90 2021-04-09T18:34:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.204 7f33ec888700 1 -- 172.21.15.32:0/982474324 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33d800c8e0 con 0x7f33e80d6c90 2021-04-09T18:34:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.204 7f33efa6e700 1 -- 172.21.15.32:0/982474324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80d6c90 msgr2=0x7f33e80d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.204 7f33efa6e700 1 --2- 172.21.15.32:0/982474324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80d6c90 0x7f33e80d8000 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7f33d8006700 tx=0x7f33d8009b00).stop 2021-04-09T18:34:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.204 7f33efa6e700 1 -- 172.21.15.32:0/982474324 shutdown_connections 2021-04-09T18:34:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.204 7f33efa6e700 1 --2- 172.21.15.32:0/982474324 >> v2:172.21.15.86:3300/0 conn(0x7f33e80d2450 0x7f33e810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.204 7f33efa6e700 1 --2- 172.21.15.32:0/982474324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80d6c90 0x7f33e80d8000 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.204 7f33efa6e700 1 -- 172.21.15.32:0/982474324 >> 172.21.15.32:0/982474324 conn(0x7f33e80ce5d0 msgr2=0x7f33e80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:03.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.205 7f33efa6e700 1 -- 172.21.15.32:0/982474324 shutdown_connections 2021-04-09T18:34:03.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.205 7f33efa6e700 1 -- 172.21.15.32:0/982474324 wait complete. 2021-04-09T18:34:03.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33efa6e700 1 Processor -- start 2021-04-09T18:34:03.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33efa6e700 1 -- start start 2021-04-09T18:34:03.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33efa6e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e817afe0 0x7f33e81b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33efa6e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e81814d0 0x7f33e80aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33efa6e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80ab3b0 0x7f33e817bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33efa6e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e8192a70 con 0x7f33e80ab3b0 2021-04-09T18:34:03.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33efa6e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e817b5a0 con 0x7f33e81814d0 2021-04-09T18:34:03.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33efa6e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e81b8610 con 0x7f33e817afe0 2021-04-09T18:34:03.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33ee08b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80ab3b0 0x7f33e817bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.225 7f33ee08b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80ab3b0 0x7f33e817bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59284/0 (socket says 172.21.15.32:59284) 2021-04-09T18:34:03.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ee08b700 1 -- 172.21.15.32:0/1338219098 learned_addr learned my addr 172.21.15.32:0/1338219098 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:03.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ed88a700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e817afe0 0x7f33e81b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ed089700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e81814d0 0x7f33e80aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ee08b700 1 -- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e817afe0 msgr2=0x7f33e81b89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ee08b700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e817afe0 0x7f33e81b89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ee08b700 1 -- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e81814d0 msgr2=0x7f33e80aad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ee08b700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e81814d0 0x7f33e80aad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ee08b700 1 -- 172.21.15.32:0/1338219098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e81a0930 con 0x7f33e80ab3b0 2021-04-09T18:34:03.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ed089700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e81814d0 0x7f33e80aad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:03.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33ee08b700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80ab3b0 0x7f33e817bae0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 rx=0x7f33e000b910 tx=0x7f33e000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:03.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33de7fc700 1 -- 172.21.15.32:0/1338219098 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33e0006aa0 con 0x7f33e80ab3b0 2021-04-09T18:34:03.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33de7fc700 1 -- 172.21.15.32:0/1338219098 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e0003030 con 0x7f33e80ab3b0 2021-04-09T18:34:03.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e81a0930 con 0x7f33e80ab3b0 2021-04-09T18:34:03.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33de7fc700 1 -- 172.21.15.32:0/1338219098 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33e0001470 con 0x7f33e80ab3b0 2021-04-09T18:34:03.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.226 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e81a0930 con 0x7f33e80ab3b0 2021-04-09T18:34:03.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.227 7f33de7fc700 1 -- 172.21.15.32:0/1338219098 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f33e0000ec0 con 0x7f33e80ab3b0 2021-04-09T18:34:03.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.227 7f33de7fc700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33d0018930 0x7f33d001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.227 7f33de7fc700 1 -- 172.21.15.32:0/1338219098 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f33e0032de0 con 0x7f33e80ab3b0 2021-04-09T18:34:03.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.227 7f33ed88a700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33d0018930 0x7f33d001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.227 7f33ed88a700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33d0018930 0x7f33d001ae10 crc :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:03.263 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.247 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e81b7120 con 0x7f33e80ab3b0 2021-04-09T18:34:03.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.250 7f33de7fc700 1 -- 172.21.15.32:0/1338219098 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f33e000bdf0 con 0x7f33e80ab3b0 2021-04-09T18:34:03.450 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.433 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f33e80d7290 con 0x7f33e80ab3b0 2021-04-09T18:34:03.592 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:34:03.576 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:34:03.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.578 7f33de7fc700 1 -- 172.21.15.32:0/1338219098 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v419) v1 ==== 161+0+0 (secure 0 0 0) 0x7f33e0000ec0 con 0x7f33e80ab3b0 2021-04-09T18:34:03.614 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:34:03.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33d0018930 msgr2=0x7f33d001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33d0018930 0x7f33d001ae10 crc :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80ab3b0 msgr2=0x7f33e817bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80ab3b0 0x7f33e817bae0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 rx=0x7f33e000b910 tx=0x7f33e000bcb0).stop 2021-04-09T18:34:03.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 shutdown_connections 2021-04-09T18:34:03.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f33d0018930 0x7f33d001ae10 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f33e80ab3b0 0x7f33e817bae0 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e81814d0 0x7f33e80aad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 --2- 172.21.15.32:0/1338219098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e817afe0 0x7f33e81b89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.619 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 >> 172.21.15.32:0/1338219098 conn(0x7f33e81905e0 msgr2=0x7f33e81b7350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:03.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 shutdown_connections 2021-04-09T18:34:03.620 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.600 7f33efa6e700 1 -- 172.21.15.32:0/1338219098 wait complete. 2021-04-09T18:34:03.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:03.653 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:03.655 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:34:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.760 7f8c6dd17700 1 Processor -- start 2021-04-09T18:34:03.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.760 7f8c6dd17700 1 -- start start 2021-04-09T18:34:03.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c6dd17700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8c680d7ca0 0x7f8c680d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c6dd17700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6810da00 0x7f8c6810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c6dd17700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6810d480 con 0x7f8c6810d010 2021-04-09T18:34:03.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c6dd17700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c680d9220 con 0x7f8c680d7ca0 2021-04-09T18:34:03.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c6dd17700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c680d86c0 con 0x7f8c6810da00 2021-04-09T18:34:03.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c66ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8c680d7ca0 0x7f8c680d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c677fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8c6810d010 0x7f8c680d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40658/0 (socket says 172.21.15.32:40658) 2021-04-09T18:34:03.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c677fe700 1 -- 172.21.15.32:0/1515553411 learned_addr learned my addr 172.21.15.32:0/1515553411 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:03.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c67fff700 1 --2- 172.21.15.32:0/1515553411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6810da00 0x7f8c6810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c66ffd700 1 -- 172.21.15.32:0/1515553411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6810da00 msgr2=0x7f8c6810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:03.802 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c66ffd700 1 --2- 172.21.15.32:0/1515553411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6810da00 0x7f8c6810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c66ffd700 1 -- 172.21.15.32:0/1515553411 >> v1:172.21.15.32:6789/0 conn(0x7f8c6810d010 legacy=0x7f8c680d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c66ffd700 1 -- 172.21.15.32:0/1515553411 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c680d1f70 con 0x7f8c680d7ca0 2021-04-09T18:34:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c667fc700 1 -- 172.21.15.32:0/1515553411 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006993238 0 0) 0x7f8c58000ce0 con 0x7f8c6810d010 2021-04-09T18:34:03.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c66ffd700 1 --2- 172.21.15.32:0/1515553411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c680d7ca0 0x7f8c680d80a0 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 rx=0x7f8c5c006650 tx=0x7f8c5c009a00).ready entity=mon.0 client_cookie=d42117cbc0c1af0d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:03.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c667fc700 1 -- 172.21.15.32:0/1515553411 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c5c009df0 con 0x7f8c680d7ca0 2021-04-09T18:34:03.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c667fc700 1 -- 172.21.15.32:0/1515553411 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5c0009b0 con 0x7f8c680d7ca0 2021-04-09T18:34:03.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.761 7f8c667fc700 1 -- 172.21.15.32:0/1515553411 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c5c00c2b0 con 0x7f8c680d7ca0 2021-04-09T18:34:03.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.762 7f8c6dd17700 1 -- 172.21.15.32:0/1515553411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c680d7ca0 msgr2=0x7f8c680d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.762 7f8c6dd17700 1 --2- 172.21.15.32:0/1515553411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c680d7ca0 0x7f8c680d80a0 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 rx=0x7f8c5c006650 tx=0x7f8c5c009a00).stop 2021-04-09T18:34:03.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.762 7f8c6dd17700 1 -- 172.21.15.32:0/1515553411 shutdown_connections 2021-04-09T18:34:03.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.762 7f8c6dd17700 1 --2- 172.21.15.32:0/1515553411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6810da00 0x7f8c6810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.762 7f8c6dd17700 1 --2- 172.21.15.32:0/1515553411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c680d7ca0 0x7f8c680d80a0 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.762 7f8c6dd17700 1 -- 172.21.15.32:0/1515553411 >> 172.21.15.32:0/1515553411 conn(0x7f8c680ce5f0 msgr2=0x7f8c680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:03.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.762 7f8c6dd17700 1 -- 172.21.15.32:0/1515553411 shutdown_connections 2021-04-09T18:34:03.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.762 7f8c6dd17700 1 -- 172.21.15.32:0/1515553411 wait complete. 2021-04-09T18:34:03.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.782 7f8c6dd17700 1 Processor -- start 2021-04-09T18:34:03.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.782 7f8c6dd17700 1 -- start start 2021-04-09T18:34:03.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c6dd17700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c68186770 0x7f8c68181690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c6dd17700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c68181cb0 0x7f8c6808eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c6dd17700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6808f0d0 0x7f8c681a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c6dd17700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c681820b0 con 0x7f8c6808f0d0 2021-04-09T18:34:03.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c6dd17700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c68186b70 con 0x7f8c68186770 2021-04-09T18:34:03.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c6dd17700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6808f4d0 con 0x7f8c68181cb0 2021-04-09T18:34:03.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c67fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6808f0d0 0x7f8c681a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c67fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6808f0d0 0x7f8c681a56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59308/0 (socket says 172.21.15.32:59308) 2021-04-09T18:34:03.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c67fff700 1 -- 172.21.15.32:0/286043451 learned_addr learned my addr 172.21.15.32:0/286043451 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:03.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c677fe700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c68186770 0x7f8c68181690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c66ffd700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c68181cb0 0x7f8c6808eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c67fff700 1 -- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c68181cb0 msgr2=0x7f8c6808eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c67fff700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c68181cb0 0x7f8c6808eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c67fff700 1 -- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c68186770 msgr2=0x7f8c68181690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:03.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c67fff700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c68186770 0x7f8c68181690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.783 7f8c67fff700 1 -- 172.21.15.32:0/286043451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c681a5d00 con 0x7f8c6808f0d0 2021-04-09T18:34:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.784 7f8c67fff700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6808f0d0 0x7f8c681a56e0 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 rx=0x7f8c6000b970 tx=0x7f8c6000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.784 7f8c4ffff700 1 -- 172.21.15.32:0/286043451 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c60000f90 con 0x7f8c6808f0d0 2021-04-09T18:34:03.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.784 7f8c4ffff700 1 -- 172.21.15.32:0/286043451 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c60006aa0 con 0x7f8c6808f0d0 2021-04-09T18:34:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.784 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c681a5d00 con 0x7f8c6808f0d0 2021-04-09T18:34:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.784 7f8c4ffff700 1 -- 172.21.15.32:0/286043451 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c600015f0 con 0x7f8c6808f0d0 2021-04-09T18:34:03.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.784 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c681a5d00 con 0x7f8c6808f0d0 2021-04-09T18:34:03.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.784 7f8c4ffff700 1 -- 172.21.15.32:0/286043451 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8c60000e70 con 0x7f8c6808f0d0 2021-04-09T18:34:03.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.785 7f8c4ffff700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c480188d0 0x7f8c4801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:03.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.785 7f8c4ffff700 1 -- 172.21.15.32:0/286043451 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f8c60032dc0 con 0x7f8c6808f0d0 2021-04-09T18:34:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.785 7f8c677fe700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c480188d0 0x7f8c4801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:03.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.785 7f8c677fe700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c480188d0 0x7f8c4801add0 crc :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:03.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.805 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c68181160 con 0x7f8c6808f0d0 2021-04-09T18:34:03.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.808 7f8c4ffff700 1 -- 172.21.15.32:0/286043451 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8c600066f0 con 0x7f8c6808f0d0 2021-04-09T18:34:04.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.998 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c68181160 con 0x7f8c6808f0d0 2021-04-09T18:34:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:03.998 7f8c4ffff700 1 -- 172.21.15.32:0/286043451 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 419 v419) v1 ==== 95+0+1426 (secure 0 0 0) 0x7f8c60006aa0 con 0x7f8c6808f0d0 2021-04-09T18:34:04.016 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 419 2021-04-09T18:34:04.017 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:04.017 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":419,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":419,"flags":51,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:33:39.367894","modified":"2021-04-09 18:34:03.576558","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[1],"info":{},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-04-09T18:34:04.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c480188d0 msgr2=0x7f8c4801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c480188d0 0x7f8c4801add0 crc :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6808f0d0 msgr2=0x7f8c681a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6808f0d0 0x7f8c681a56e0 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 rx=0x7f8c6000b970 tx=0x7f8c6000bd10).stop 2021-04-09T18:34:04.019 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 shutdown_connections 2021-04-09T18:34:04.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8c480188d0 0x7f8c4801add0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c6808f0d0 0x7f8c681a56e0 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c68181cb0 0x7f8c6808eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 --2- 172.21.15.32:0/286043451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c68186770 0x7f8c68181690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.020 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 >> 172.21.15.32:0/286043451 conn(0x7f8c68195e70 msgr2=0x7f8c681a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:04.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 shutdown_connections 2021-04-09T18:34:04.021 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.001 7f8c6dd17700 1 -- 172.21.15.32:0/286043451 wait complete. 2021-04-09T18:34:04.028 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:34:04.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.203 7f7bff0cf700 1 Processor -- start 2021-04-09T18:34:04.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.203 7f7bff0cf700 1 -- start start 2021-04-09T18:34:04.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.204 7f7bff0cf700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7bf80d6c90 0x7f7bf80d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.204 7f7bff0cf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80d2450 0x7f7bf810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.204 7f7bff0cf700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf80d7290 con 0x7f7bf80cffb0 2021-04-09T18:34:04.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.204 7f7bff0cf700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf80d7090 con 0x7f7bf80d6c90 2021-04-09T18:34:04.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.204 7f7bff0cf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf80d91f0 con 0x7f7bf80d2450 2021-04-09T18:34:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.204 7f7bf7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7bf80d6c90 0x7f7bf80d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.204 7f7bf7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7bf80d6c90 0x7f7bf80d8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59314/0 (socket says 172.21.15.32:59314) 2021-04-09T18:34:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.204 7f7bf7fff700 1 -- 172.21.15.32:0/706877606 learned_addr learned my addr 172.21.15.32:0/706877606 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.204 7f7bfd6ec700 1 --2- 172.21.15.32:0/706877606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80d2450 0x7f7bf810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bf7fff700 1 -- 172.21.15.32:0/706877606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80d2450 msgr2=0x7f7bf810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bf7fff700 1 --2- 172.21.15.32:0/706877606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80d2450 0x7f7bf810cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bf7fff700 1 -- 172.21.15.32:0/706877606 >> v1:172.21.15.86:6789/0 conn(0x7f7bf80cffb0 legacy=0x7f7bf80d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bf7fff700 1 -- 172.21.15.32:0/706877606 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf80d2850 con 0x7f7bf80d6c90 2021-04-09T18:34:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bf7fff700 1 --2- 172.21.15.32:0/706877606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf80d6c90 0x7f7bf80d8000 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f7be8006700 tx=0x7f7be8009b00).ready entity=mon.0 client_cookie=4d2f2f4af4d86273 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:04.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bf77fe700 1 -- 172.21.15.32:0/706877606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7be800c0c0 con 0x7f7bf80d6c90 2021-04-09T18:34:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bf77fe700 1 -- 172.21.15.32:0/706877606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be800c400 con 0x7f7bf80d6c90 2021-04-09T18:34:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bf77fe700 1 -- 172.21.15.32:0/706877606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7be800c8e0 con 0x7f7bf80d6c90 2021-04-09T18:34:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bff0cf700 1 -- 172.21.15.32:0/706877606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf80d6c90 msgr2=0x7f7bf80d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.205 7f7bff0cf700 1 --2- 172.21.15.32:0/706877606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf80d6c90 0x7f7bf80d8000 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f7be8006700 tx=0x7f7be8009b00).stop 2021-04-09T18:34:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.206 7f7bff0cf700 1 -- 172.21.15.32:0/706877606 shutdown_connections 2021-04-09T18:34:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.206 7f7bff0cf700 1 --2- 172.21.15.32:0/706877606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80d2450 0x7f7bf810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.206 7f7bff0cf700 1 --2- 172.21.15.32:0/706877606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf80d6c90 0x7f7bf80d8000 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.206 7f7bff0cf700 1 -- 172.21.15.32:0/706877606 >> 172.21.15.32:0/706877606 conn(0x7f7bf80ce5d0 msgr2=0x7f7bf80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:04.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.206 7f7bff0cf700 1 -- 172.21.15.32:0/706877606 shutdown_connections 2021-04-09T18:34:04.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.206 7f7bff0cf700 1 -- 172.21.15.32:0/706877606 wait complete. 2021-04-09T18:34:04.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.226 7f7bff0cf700 1 Processor -- start 2021-04-09T18:34:04.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.226 7f7bff0cf700 1 -- start start 2021-04-09T18:34:04.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.226 7f7bff0cf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf81b8380 0x7f7bf81b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.226 7f7bff0cf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf808c7b0 0x7f7bf808cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.226 7f7bff0cf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf808d1d0 0x7f7bf81805b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.226 7f7bff0cf700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf8181a30 con 0x7f7bf808c7b0 2021-04-09T18:34:04.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.226 7f7bff0cf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf808c230 con 0x7f7bf81b8380 2021-04-09T18:34:04.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.226 7f7bff0cf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf81b8da0 con 0x7f7bf808d1d0 2021-04-09T18:34:04.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf808c7b0 0x7f7bf808cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf808c7b0 0x7f7bf808cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59318/0 (socket says 172.21.15.32:59318) 2021-04-09T18:34:04.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf7fff700 1 -- 172.21.15.32:0/1407910140 learned_addr learned my addr 172.21.15.32:0/1407910140 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:04.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bfd6ec700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf808d1d0 0x7f7bf81805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bfceeb700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf81b8380 0x7f7bf81b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf7fff700 1 -- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf808d1d0 msgr2=0x7f7bf81805b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf7fff700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf808d1d0 0x7f7bf81805b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf7fff700 1 -- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf81b8380 msgr2=0x7f7bf81b8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf7fff700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf81b8380 0x7f7bf81b8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf7fff700 1 -- 172.21.15.32:0/1407910140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf81b8f90 con 0x7f7bf808c7b0 2021-04-09T18:34:04.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf7fff700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf808c7b0 0x7f7bf808cbb0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7f7be800b660 tx=0x7f7be800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:04.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf57fa700 1 -- 172.21.15.32:0/1407910140 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7be8009440 con 0x7f7bf808c7b0 2021-04-09T18:34:04.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf57fa700 1 -- 172.21.15.32:0/1407910140 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8006390 con 0x7f7bf808c7b0 2021-04-09T18:34:04.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf81b8f90 con 0x7f7bf808c7b0 2021-04-09T18:34:04.256 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bf57fa700 1 -- 172.21.15.32:0/1407910140 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7be8002030 con 0x7f7bf808c7b0 2021-04-09T18:34:04.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.227 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf81b8f90 con 0x7f7bf808c7b0 2021-04-09T18:34:04.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.228 7f7bf57fa700 1 -- 172.21.15.32:0/1407910140 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7be80093e0 con 0x7f7bf808c7b0 2021-04-09T18:34:04.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.228 7f7bf57fa700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bdc018930 0x7f7bdc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.228 7f7bf57fa700 1 -- 172.21.15.32:0/1407910140 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f7be8032a90 con 0x7f7bf808c7b0 2021-04-09T18:34:04.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.228 7f7bfceeb700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bdc018930 0x7f7bdc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.228 7f7bfceeb700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bdc018930 0x7f7bdc01adc0 crc :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:04.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.248 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf81b7180 con 0x7f7bf808c7b0 2021-04-09T18:34:04.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.252 7f7bf57fa700 1 -- 172.21.15.32:0/1407910140 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7be8006370 con 0x7f7bf808c7b0 2021-04-09T18:34:04.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.445 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7bf808c230 con 0x7f7bf808c7b0 2021-04-09T18:34:04.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.590 7f7bf57fa700 1 -- 172.21.15.32:0/1407910140 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v420) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7be8006370 con 0x7f7bf808c7b0 2021-04-09T18:34:04.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bdc018930 msgr2=0x7f7bdc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.628 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bdc018930 0x7f7bdc01adc0 crc :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf808c7b0 msgr2=0x7f7bf808cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf808c7b0 0x7f7bf808cbb0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7f7be800b660 tx=0x7f7be800ba00).stop 2021-04-09T18:34:04.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 shutdown_connections 2021-04-09T18:34:04.629 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7bdc018930 0x7f7bdc01adc0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7bf808c7b0 0x7f7bf808cbb0 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf808d1d0 0x7f7bf81805b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 --2- 172.21.15.32:0/1407910140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf81b8380 0x7f7bf81b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 >> 172.21.15.32:0/1407910140 conn(0x7f7bf8189800 msgr2=0x7f7bf81b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:04.630 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 shutdown_connections 2021-04-09T18:34:04.631 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.611 7f7bff0cf700 1 -- 172.21.15.32:0/1407910140 wait complete. 2021-04-09T18:34:04.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:34:04.829 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.785 7f865e752700 1 Processor -- start 2021-04-09T18:34:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.785 7f865e752700 1 -- start start 2021-04-09T18:34:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.785 7f865e752700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f865810d050 0x7f86580cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.785 7f865e752700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865810d500 con 0x7f86580d0160 2021-04-09T18:34:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.785 7f865e752700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865810e270 con 0x7f86580d84b0 2021-04-09T18:34:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.785 7f865e752700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86580d8d70 con 0x7f865810d050 2021-04-09T18:34:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.785 7f8657fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f865810d050 0x7f86580cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.785 7f864ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f86580d0160 0x7f86580d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40690/0 (socket says 172.21.15.32:40690) 2021-04-09T18:34:04.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.785 7f864ffff700 1 -- 172.21.15.32:0/3295411836 learned_addr learned my addr 172.21.15.32:0/3295411836 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:04.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f8657fff700 1 -- 172.21.15.32:0/3295411836 >> v1:172.21.15.86:6789/0 conn(0x7f86580d84b0 legacy=0x7f865810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f8657fff700 1 -- 172.21.15.32:0/3295411836 >> v1:172.21.15.32:6789/0 conn(0x7f86580d0160 legacy=0x7f86580d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f8657fff700 1 -- 172.21.15.32:0/3295411836 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580d9260 con 0x7f865810d050 2021-04-09T18:34:04.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f86577fe700 1 -- 172.21.15.32:0/3295411836 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759861899 0 0) 0x7f8640000ce0 con 0x7f86580d0160 2021-04-09T18:34:04.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f8657fff700 1 --2- 172.21.15.32:0/3295411836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f865810d050 0x7f86580cfb40 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f8648006650 tx=0x7f8648009a00).ready entity=mon.0 client_cookie=99c62ef35d52962a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:04.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f86577fe700 1 -- 172.21.15.32:0/3295411836 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8648009df0 con 0x7f865810d050 2021-04-09T18:34:04.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f86577fe700 1 -- 172.21.15.32:0/3295411836 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86480009b0 con 0x7f865810d050 2021-04-09T18:34:04.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f86577fe700 1 -- 172.21.15.32:0/3295411836 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f864800c2b0 con 0x7f865810d050 2021-04-09T18:34:04.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f865e752700 1 -- 172.21.15.32:0/3295411836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f865810d050 msgr2=0x7f86580cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.786 7f865e752700 1 --2- 172.21.15.32:0/3295411836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f865810d050 0x7f86580cfb40 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f8648006650 tx=0x7f8648009a00).stop 2021-04-09T18:34:04.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.787 7f865e752700 1 -- 172.21.15.32:0/3295411836 shutdown_connections 2021-04-09T18:34:04.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.787 7f865e752700 1 --2- 172.21.15.32:0/3295411836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f865810d050 0x7f86580cfb40 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.787 7f865e752700 1 -- 172.21.15.32:0/3295411836 >> 172.21.15.32:0/3295411836 conn(0x7f86580ce620 msgr2=0x7f86580d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:04.839 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.787 7f865e752700 1 -- 172.21.15.32:0/3295411836 shutdown_connections 2021-04-09T18:34:04.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.787 7f865e752700 1 -- 172.21.15.32:0/3295411836 wait complete. 2021-04-09T18:34:04.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.807 7f865e752700 1 Processor -- start 2021-04-09T18:34:04.840 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.807 7f865e752700 1 -- start start 2021-04-09T18:34:04.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865e752700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86580bc0d0 0x7f86580bb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865e752700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86580bd3b0 0x7f86580bd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865e752700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86580abed0 0x7f86580ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865e752700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86580bad30 con 0x7f86580bd3b0 2021-04-09T18:34:04.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865e752700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86580bddd0 con 0x7f86580bc0d0 2021-04-09T18:34:04.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865e752700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86580ac8f0 con 0x7f86580abed0 2021-04-09T18:34:04.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f864ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86580bd3b0 0x7f86580bd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f864ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86580bd3b0 0x7f86580bd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59336/0 (socket says 172.21.15.32:59336) 2021-04-09T18:34:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f864ffff700 1 -- 172.21.15.32:0/3901695924 learned_addr learned my addr 172.21.15.32:0/3901695924 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f8657fff700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86580bc0d0 0x7f86580bb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865cd6f700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86580abed0 0x7f86580ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f864ffff700 1 -- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86580abed0 msgr2=0x7f86580ac2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f864ffff700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86580abed0 0x7f86580ac2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f864ffff700 1 -- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86580bc0d0 msgr2=0x7f86580bb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f864ffff700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86580bc0d0 0x7f86580bb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f864ffff700 1 -- 172.21.15.32:0/3901695924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580bc4d0 con 0x7f86580bd3b0 2021-04-09T18:34:04.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f8657fff700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86580bc0d0 0x7f86580bb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:34:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865cd6f700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86580abed0 0x7f86580ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:34:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f864ffff700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86580bd3b0 0x7f86580bd7b0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 rx=0x7f8640006f00 tx=0x7f864000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f86557fa700 1 -- 172.21.15.32:0/3901695924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8640006020 con 0x7f86580bd3b0 2021-04-09T18:34:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f86557fa700 1 -- 172.21.15.32:0/3901695924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86400063a0 con 0x7f86580bd3b0 2021-04-09T18:34:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865e752700 1 -- 172.21.15.32:0/3901695924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86580bc4d0 con 0x7f86580bd3b0 2021-04-09T18:34:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f86557fa700 1 -- 172.21.15.32:0/3901695924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8640004620 con 0x7f86580bd3b0 2021-04-09T18:34:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.808 7f865e752700 1 -- 172.21.15.32:0/3901695924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86580bc4d0 con 0x7f86580bd3b0 2021-04-09T18:34:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.809 7f86557fa700 1 -- 172.21.15.32:0/3901695924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f864000a000 con 0x7f86580bd3b0 2021-04-09T18:34:04.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.809 7f86557fa700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f863c018930 0x7f863c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.809 7f86557fa700 1 -- 172.21.15.32:0/3901695924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f8640032a30 con 0x7f86580bd3b0 2021-04-09T18:34:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.809 7f8657fff700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f863c018930 0x7f863c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:04.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.810 7f8657fff700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f863c018930 0x7f863c01adc0 crc :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.829 7f865e752700 1 -- 172.21.15.32:0/3901695924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86580bad30 con 0x7f86580bd3b0 2021-04-09T18:34:04.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:04.833 7f86557fa700 1 -- 172.21.15.32:0/3901695924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8640006b90 con 0x7f86580bd3b0 2021-04-09T18:34:05.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.017 7f865e752700 1 -- 172.21.15.32:0/3901695924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f86580bad30 con 0x7f86580bd3b0 2021-04-09T18:34:05.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.630 7f86557fa700 1 -- 172.21.15.32:0/3901695924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v253) v1 ==== 185+0+0 (secure 0 0 0) 0x7f864000a000 con 0x7f86580bd3b0 2021-04-09T18:34:05.648 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:34:05.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 -- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f863c018930 msgr2=0x7f863c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:05.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f863c018930 0x7f863c01adc0 crc :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 -- 172.21.15.32:0/3901695924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86580bd3b0 msgr2=0x7f86580bd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:05.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86580bd3b0 0x7f86580bd7b0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 rx=0x7f8640006f00 tx=0x7f864000b940).stop 2021-04-09T18:34:05.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 -- 172.21.15.32:0/3901695924 shutdown_connections 2021-04-09T18:34:05.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f863c018930 0x7f863c01adc0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86580bd3b0 0x7f86580bd7b0 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86580abed0 0x7f86580ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 --2- 172.21.15.32:0/3901695924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86580bc0d0 0x7f86580bb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 -- 172.21.15.32:0/3901695924 >> 172.21.15.32:0/3901695924 conn(0x7f86580c1850 msgr2=0x7f86581aa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:05.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 -- 172.21.15.32:0/3901695924 shutdown_connections 2021-04-09T18:34:05.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.634 7f865e752700 1 -- 172.21.15.32:0/3901695924 wait complete. 2021-04-09T18:34:05.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:34:05.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.807 7f2dff449700 1 Processor -- start 2021-04-09T18:34:05.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.807 7f2dff449700 1 -- start start 2021-04-09T18:34:05.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dff449700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df810d050 0x7f2df80cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:05.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dff449700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2df80d0160 0x7f2df80d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:05.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dff449700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df810d500 con 0x7f2df80d82e0 2021-04-09T18:34:05.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dff449700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df80d8d70 con 0x7f2df80d0160 2021-04-09T18:34:05.848 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dff449700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80d9260 con 0x7f2df810d050 2021-04-09T18:34:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dfca64700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2df80d0160 0x7f2df80d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dfd265700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df810d050 0x7f2df80cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dfda66700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2df80d82e0 0x7f2df810da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38628/0 (socket says 172.21.15.32:38628) 2021-04-09T18:34:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dfda66700 1 -- 172.21.15.32:0/2176611958 learned_addr learned my addr 172.21.15.32:0/2176611958 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2deffff700 1 -- 172.21.15.32:0/2176611958 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654775004 0 0) 0x7f2df0001440 con 0x7f2df80d82e0 2021-04-09T18:34:05.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2deffff700 1 -- 172.21.15.32:0/2176611958 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de4002fe0 con 0x7f2df80d82e0 2021-04-09T18:34:05.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dfd265700 1 -- 172.21.15.32:0/2176611958 >> v2:172.21.15.86:3300/0 conn(0x7f2df80d0160 msgr2=0x7f2df80d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:05.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dfd265700 1 --2- 172.21.15.32:0/2176611958 >> v2:172.21.15.86:3300/0 conn(0x7f2df80d0160 0x7f2df80d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.850 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.808 7f2dfd265700 1 -- 172.21.15.32:0/2176611958 >> v1:172.21.15.86:6789/0 conn(0x7f2df80d82e0 legacy=0x7f2df810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:05.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.809 7f2dfd265700 1 -- 172.21.15.32:0/2176611958 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df80d86e0 con 0x7f2df810d050 2021-04-09T18:34:05.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.809 7f2dfca64700 1 --2- 172.21.15.32:0/2176611958 >> v2:172.21.15.86:3300/0 conn(0x7f2df80d0160 0x7f2df80d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:05.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.809 7f2dfd265700 1 --2- 172.21.15.32:0/2176611958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df810d050 0x7f2df80cfb40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f2df4006650 tx=0x7f2df4009a80).ready entity=mon.2 client_cookie=a13668607125fe4d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:05.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.809 7f2deffff700 1 -- 172.21.15.32:0/2176611958 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df40009b0 con 0x7f2df810d050 2021-04-09T18:34:05.851 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.809 7f2deffff700 1 -- 172.21.15.32:0/2176611958 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df400c0c0 con 0x7f2df810d050 2021-04-09T18:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.809 7f2deffff700 1 -- 172.21.15.32:0/2176611958 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df400c510 con 0x7f2df810d050 2021-04-09T18:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.809 7f2dff449700 1 -- 172.21.15.32:0/2176611958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df810d050 msgr2=0x7f2df80cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.809 7f2dff449700 1 --2- 172.21.15.32:0/2176611958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df810d050 0x7f2df80cfb40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f2df4006650 tx=0x7f2df4009a80).stop 2021-04-09T18:34:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.810 7f2dff449700 1 -- 172.21.15.32:0/2176611958 shutdown_connections 2021-04-09T18:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.810 7f2dff449700 1 --2- 172.21.15.32:0/2176611958 >> v2:172.21.15.86:3300/0 conn(0x7f2df80d0160 0x7f2df80d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.810 7f2dff449700 1 --2- 172.21.15.32:0/2176611958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df810d050 0x7f2df80cfb40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.810 7f2dff449700 1 -- 172.21.15.32:0/2176611958 >> 172.21.15.32:0/2176611958 conn(0x7f2df80ce620 msgr2=0x7f2df80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.810 7f2dff449700 1 -- 172.21.15.32:0/2176611958 shutdown_connections 2021-04-09T18:34:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.810 7f2dff449700 1 -- 172.21.15.32:0/2176611958 wait complete. 2021-04-09T18:34:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.830 7f2dff449700 1 Processor -- start 2021-04-09T18:34:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.830 7f2dff449700 1 -- start start 2021-04-09T18:34:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dff449700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2df80b0920 0x7f2df80b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dff449700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2df80a7690 0x7f2df80a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:05.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dff449700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df80c2510 0x7f2df80c2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dff449700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80b1340 con 0x7f2df80a7690 2021-04-09T18:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dff449700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80a82e0 con 0x7f2df80b0920 2021-04-09T18:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dff449700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80a8820 con 0x7f2df80c2510 2021-04-09T18:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfca64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2df80a7690 0x7f2df80a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:05.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfd265700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2df80b0920 0x7f2df80b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:05.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfca64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2df80a7690 0x7f2df80a7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59350/0 (socket says 172.21.15.32:59350) 2021-04-09T18:34:05.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfca64700 1 -- 172.21.15.32:0/3123042792 learned_addr learned my addr 172.21.15.32:0/3123042792 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:05.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfda66700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df80c2510 0x7f2df80c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:05.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfd265700 1 -- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df80c2510 msgr2=0x7f2df80c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfd265700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df80c2510 0x7f2df80c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfd265700 1 -- 172.21.15.32:0/3123042792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2df80a7690 msgr2=0x7f2df80a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfd265700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2df80a7690 0x7f2df80a7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfd265700 1 -- 172.21.15.32:0/3123042792 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df808fba0 con 0x7f2df80b0920 2021-04-09T18:34:05.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.831 7f2dfca64700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2df80a7690 0x7f2df80a7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.832 7f2dfd265700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2df80b0920 0x7f2df80b0d20 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f2df400b4b0 tx=0x7f2df400b850).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.832 7f2dedffb700 1 -- 172.21.15.32:0/3123042792 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df4009280 con 0x7f2df80b0920 2021-04-09T18:34:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.832 7f2dedffb700 1 -- 172.21.15.32:0/3123042792 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df400bc40 con 0x7f2df80b0920 2021-04-09T18:34:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.832 7f2dff449700 1 -- 172.21.15.32:0/3123042792 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df808fba0 con 0x7f2df80b0920 2021-04-09T18:34:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.832 7f2dedffb700 1 -- 172.21.15.32:0/3123042792 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df4002220 con 0x7f2df80b0920 2021-04-09T18:34:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.832 7f2dff449700 1 -- 172.21.15.32:0/3123042792 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df808fba0 con 0x7f2df80b0920 2021-04-09T18:34:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.832 7f2dedffb700 1 -- 172.21.15.32:0/3123042792 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2df4000c90 con 0x7f2df80b0920 2021-04-09T18:34:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.832 7f2dedffb700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dd8018930 0x7f2dd801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:05.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.833 7f2dedffb700 1 -- 172.21.15.32:0/3123042792 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 5225+0+0 (secure 0 0 0) 0x7f2df4032950 con 0x7f2df80b0920 2021-04-09T18:34:05.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.833 7f2dfca64700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dd8018930 0x7f2dd801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:05.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.833 7f2dfca64700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dd8018930 0x7f2dd801ae10 crc :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:05.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.852 7f2dff449700 1 -- 172.21.15.32:0/3123042792 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df81b7180 con 0x7f2df80b0920 2021-04-09T18:34:05.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:05.855 7f2dedffb700 1 -- 172.21.15.32:0/3123042792 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2df4009280 con 0x7f2df80b0920 2021-04-09T18:34:06.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.038 7f2dff449700 1 -- 172.21.15.32:0/3123042792 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2df81b7180 con 0x7f2df80b0920 2021-04-09T18:34:06.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.637 7f2dedffb700 1 -- 172.21.15.32:0/3123042792 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v254) v1 ==== 173+0+0 (secure 0 0 0) 0x7f2df4000c90 con 0x7f2df80b0920 2021-04-09T18:34:06.670 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:34:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 -- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dd8018930 msgr2=0x7f2dd801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:06.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dd8018930 0x7f2dd801ae10 crc :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 -- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2df80b0920 msgr2=0x7f2df80b0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:06.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2df80b0920 0x7f2df80b0d20 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f2df400b4b0 tx=0x7f2df400b850).stop 2021-04-09T18:34:06.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 -- 172.21.15.32:0/3123042792 shutdown_connections 2021-04-09T18:34:06.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2dd8018930 0x7f2dd801ae10 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2df80a7690 0x7f2df80a7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2df80c2510 0x7f2df80c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 --2- 172.21.15.32:0/3123042792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2df80b0920 0x7f2df80b0d20 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.656 7f2dff449700 1 -- 172.21.15.32:0/3123042792 >> 172.21.15.32:0/3123042792 conn(0x7f2df80b6b70 msgr2=0x7f2df80b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:06.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.657 7f2dff449700 1 -- 172.21.15.32:0/3123042792 shutdown_connections 2021-04-09T18:34:06.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.657 7f2dff449700 1 -- 172.21.15.32:0/3123042792 wait complete. 2021-04-09T18:34:06.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:34:06.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.836 7f4510d30700 1 Processor -- start 2021-04-09T18:34:06.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.836 7f4510d30700 1 -- start start 2021-04-09T18:34:06.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f4510d30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c0cffb0 0x7f450c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:06.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f4510d30700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f450c0d7810 0x7f450c0d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:06.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f4510d30700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c0d7290 con 0x7f450c0d5c80 2021-04-09T18:34:06.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f4510d30700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c0d8590 con 0x7f450c0d7810 2021-04-09T18:34:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f4510d30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c0d91f0 con 0x7f450c0cffb0 2021-04-09T18:34:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f450a312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f450c0d7810 0x7f450c0d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:06.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f450b314700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f450c0d5c80 0x7f450c0d6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40716/0 (socket says 172.21.15.32:40716) 2021-04-09T18:34:06.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f450b314700 1 -- 172.21.15.32:0/3254494388 learned_addr learned my addr 172.21.15.32:0/3254494388 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:06.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f450ab13700 1 --2- 172.21.15.32:0/3254494388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c0cffb0 0x7f450c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:06.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f450a312700 1 -- 172.21.15.32:0/3254494388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c0cffb0 msgr2=0x7f450c0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:06.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f450a312700 1 --2- 172.21.15.32:0/3254494388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c0cffb0 0x7f450c0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f450a312700 1 -- 172.21.15.32:0/3254494388 >> v1:172.21.15.32:6789/0 conn(0x7f450c0d5c80 legacy=0x7f450c0d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:06.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f450a312700 1 -- 172.21.15.32:0/3254494388 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c0d8230 con 0x7f450c0d7810 2021-04-09T18:34:06.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.837 7f4509b11700 1 -- 172.21.15.32:0/3254494388 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748712247 0 0) 0x7f4504000d30 con 0x7f450c0d5c80 2021-04-09T18:34:06.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.838 7f450a312700 1 --2- 172.21.15.32:0/3254494388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c0d7810 0x7f450c0d7c10 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 rx=0x7f4500006650 tx=0x7f4500009a00).ready entity=mon.0 client_cookie=5f7628748edcf957 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:06.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.838 7f4509b11700 1 -- 172.21.15.32:0/3254494388 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4500009df0 con 0x7f450c0d7810 2021-04-09T18:34:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.838 7f4509b11700 1 -- 172.21.15.32:0/3254494388 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45000009b0 con 0x7f450c0d7810 2021-04-09T18:34:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.838 7f4509b11700 1 -- 172.21.15.32:0/3254494388 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f450000c2b0 con 0x7f450c0d7810 2021-04-09T18:34:06.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.838 7f4510d30700 1 -- 172.21.15.32:0/3254494388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c0d7810 msgr2=0x7f450c0d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:06.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.838 7f4510d30700 1 --2- 172.21.15.32:0/3254494388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c0d7810 0x7f450c0d7c10 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 rx=0x7f4500006650 tx=0x7f4500009a00).stop 2021-04-09T18:34:06.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.839 7f4510d30700 1 -- 172.21.15.32:0/3254494388 shutdown_connections 2021-04-09T18:34:06.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.839 7f4510d30700 1 --2- 172.21.15.32:0/3254494388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c0d7810 0x7f450c0d7c10 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.839 7f4510d30700 1 --2- 172.21.15.32:0/3254494388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c0cffb0 0x7f450c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.839 7f4510d30700 1 -- 172.21.15.32:0/3254494388 >> 172.21.15.32:0/3254494388 conn(0x7f450c0ce5d0 msgr2=0x7f450c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:06.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.839 7f4510d30700 1 -- 172.21.15.32:0/3254494388 shutdown_connections 2021-04-09T18:34:06.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.839 7f4510d30700 1 -- 172.21.15.32:0/3254494388 wait complete. 2021-04-09T18:34:06.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.859 7f4510d30700 1 Processor -- start 2021-04-09T18:34:06.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.859 7f4510d30700 1 -- start start 2021-04-09T18:34:06.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.859 7f4510d30700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c0a8b80 0x7f450c0a8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:06.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.859 7f4510d30700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c1b8610 0x7f450c1b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:06.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.859 7f4510d30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c08d9d0 0x7f450c08ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:06.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.859 7f4510d30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c17d400 con 0x7f450c08d9d0 2021-04-09T18:34:06.891 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.859 7f4510d30700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c08e3f0 con 0x7f450c0a8b80 2021-04-09T18:34:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.859 7f4510d30700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c1a0a80 con 0x7f450c1b8610 2021-04-09T18:34:06.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.859 7f450b314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c08d9d0 0x7f450c08ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450b314700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c08d9d0 0x7f450c08ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59362/0 (socket says 172.21.15.32:59362) 2021-04-09T18:34:06.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450b314700 1 -- 172.21.15.32:0/628327192 learned_addr learned my addr 172.21.15.32:0/628327192 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:06.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450a312700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c1b8610 0x7f450c1b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:06.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450ab13700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c0a8b80 0x7f450c0a8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:06.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450b314700 1 -- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c1b8610 msgr2=0x7f450c1b8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:06.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450b314700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c1b8610 0x7f450c1b8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450b314700 1 -- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c0a8b80 msgr2=0x7f450c0a8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:06.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450b314700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c0a8b80 0x7f450c0a8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:06.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450b314700 1 -- 172.21.15.32:0/628327192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c08c630 con 0x7f450c08d9d0 2021-04-09T18:34:06.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f450b314700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c08d9d0 0x7f450c08ddd0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 rx=0x7f4504007f90 tx=0x7f450400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:06.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f44fb7fe700 1 -- 172.21.15.32:0/628327192 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4504000cc0 con 0x7f450c08d9d0 2021-04-09T18:34:06.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f44fb7fe700 1 -- 172.21.15.32:0/628327192 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4504007020 con 0x7f450c08d9d0 2021-04-09T18:34:06.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f44fb7fe700 1 -- 172.21.15.32:0/628327192 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4504007500 con 0x7f450c08d9d0 2021-04-09T18:34:06.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f4510d30700 1 -- 172.21.15.32:0/628327192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c08c630 con 0x7f450c08d9d0 2021-04-09T18:34:06.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.860 7f4510d30700 1 -- 172.21.15.32:0/628327192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c08c630 con 0x7f450c08d9d0 2021-04-09T18:34:06.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.861 7f44fb7fe700 1 -- 172.21.15.32:0/628327192 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4504007200 con 0x7f450c08d9d0 2021-04-09T18:34:06.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.861 7f44fb7fe700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44f00188c0 0x7f44f001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:06.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.861 7f44fb7fe700 1 -- 172.21.15.32:0/628327192 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f4504032840 con 0x7f450c08d9d0 2021-04-09T18:34:06.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.861 7f450ab13700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44f00188c0 0x7f44f001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:06.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.862 7f450ab13700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44f00188c0 0x7f44f001adc0 crc :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:06.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.881 7f4510d30700 1 -- 172.21.15.32:0/628327192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c08e3f0 con 0x7f450c08d9d0 2021-04-09T18:34:06.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:06.885 7f44fb7fe700 1 -- 172.21.15.32:0/628327192 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4504007d00 con 0x7f450c08d9d0 2021-04-09T18:34:07.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.096 7f4510d30700 1 -- 172.21.15.32:0/628327192 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f450c08e3f0 con 0x7f450c08d9d0 2021-04-09T18:34:07.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.099 7f44fb7fe700 1 -- 172.21.15.32:0/628327192 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4504007200 con 0x7f450c08d9d0 2021-04-09T18:34:07.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 -- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44f00188c0 msgr2=0x7f44f001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44f00188c0 0x7f44f001adc0 crc :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 -- 172.21.15.32:0/628327192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c08d9d0 msgr2=0x7f450c08ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c08d9d0 0x7f450c08ddd0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 rx=0x7f4504007f90 tx=0x7f450400ac50).stop 2021-04-09T18:34:07.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 -- 172.21.15.32:0/628327192 shutdown_connections 2021-04-09T18:34:07.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f44f00188c0 0x7f44f001adc0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f450c08d9d0 0x7f450c08ddd0 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f450c1b8610 0x7f450c1b8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 --2- 172.21.15.32:0/628327192 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f450c0a8b80 0x7f450c0a8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 -- 172.21.15.32:0/628327192 >> 172.21.15.32:0/628327192 conn(0x7f450c1905e0 msgr2=0x7f450c1b7350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 -- 172.21.15.32:0/628327192 shutdown_connections 2021-04-09T18:34:07.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.106 7f4510d30700 1 -- 172.21.15.32:0/628327192 wait complete. 2021-04-09T18:34:07.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestStandbyReplay.test_rank_stopped' 2021-04-09T18:34:07.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.268 7f2769c72700 1 Processor -- start 2021-04-09T18:34:07.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f2769c72700 1 -- start start 2021-04-09T18:34:07.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f2769c72700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f276410d010 0x7f27640d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f2769c72700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f27640d1da0 0x7f27640d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f2769c72700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276410d480 con 0x7f27640d82c0 2021-04-09T18:34:07.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f2769c72700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27640d8d30 con 0x7f276410d010 2021-04-09T18:34:07.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f2769c72700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27640d9220 con 0x7f27640d1da0 2021-04-09T18:34:07.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f27637fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f276410d010 0x7f27640d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f2763fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f27640d82c0 0x7f276410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40730/0 (socket says 172.21.15.32:40730) 2021-04-09T18:34:07.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f2763fff700 1 -- 172.21.15.32:0/2605513393 learned_addr learned my addr 172.21.15.32:0/2605513393 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:07.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.269 7f2762ffd700 1 --2- 172.21.15.32:0/2605513393 >> v2:172.21.15.86:3300/0 conn(0x7f27640d1da0 0x7f27640d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f27637fe700 1 -- 172.21.15.32:0/2605513393 >> v2:172.21.15.86:3300/0 conn(0x7f27640d1da0 msgr2=0x7f27640d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:07.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f27637fe700 1 --2- 172.21.15.32:0/2605513393 >> v2:172.21.15.86:3300/0 conn(0x7f27640d1da0 0x7f27640d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f27637fe700 1 -- 172.21.15.32:0/2605513393 >> v1:172.21.15.32:6789/0 conn(0x7f27640d82c0 legacy=0x7f276410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f27637fe700 1 -- 172.21.15.32:0/2605513393 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27640d86f0 con 0x7f276410d010 2021-04-09T18:34:07.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f27627fc700 1 -- 172.21.15.32:0/2605513393 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269906303 0 0) 0x7f275c000d30 con 0x7f27640d82c0 2021-04-09T18:34:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f27637fe700 1 --2- 172.21.15.32:0/2605513393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276410d010 0x7f27640d1780 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 rx=0x7f2754006650 tx=0x7f2754009a00).ready entity=mon.0 client_cookie=13380df3bd5839e6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:07.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f27627fc700 1 -- 172.21.15.32:0/2605513393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2754009df0 con 0x7f276410d010 2021-04-09T18:34:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f27627fc700 1 -- 172.21.15.32:0/2605513393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27540009b0 con 0x7f276410d010 2021-04-09T18:34:07.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f27627fc700 1 -- 172.21.15.32:0/2605513393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f275400c2b0 con 0x7f276410d010 2021-04-09T18:34:07.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f2769c72700 1 -- 172.21.15.32:0/2605513393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276410d010 msgr2=0x7f27640d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.270 7f2769c72700 1 --2- 172.21.15.32:0/2605513393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276410d010 0x7f27640d1780 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 rx=0x7f2754006650 tx=0x7f2754009a00).stop 2021-04-09T18:34:07.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.271 7f2769c72700 1 -- 172.21.15.32:0/2605513393 shutdown_connections 2021-04-09T18:34:07.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.271 7f2769c72700 1 --2- 172.21.15.32:0/2605513393 >> v2:172.21.15.86:3300/0 conn(0x7f27640d1da0 0x7f27640d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.271 7f2769c72700 1 --2- 172.21.15.32:0/2605513393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276410d010 0x7f27640d1780 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.271 7f2769c72700 1 -- 172.21.15.32:0/2605513393 >> 172.21.15.32:0/2605513393 conn(0x7f27640ce5f0 msgr2=0x7f27640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:07.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.271 7f2769c72700 1 -- 172.21.15.32:0/2605513393 shutdown_connections 2021-04-09T18:34:07.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.271 7f2769c72700 1 -- 172.21.15.32:0/2605513393 wait complete. 2021-04-09T18:34:07.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.291 7f2769c72700 1 Processor -- start 2021-04-09T18:34:07.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.291 7f2769c72700 1 -- start start 2021-04-09T18:34:07.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2769c72700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764179bf0 0x7f2764179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2769c72700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f276417c540 0x7f276417c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2769c72700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27640aca60 0x7f27640ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2769c72700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276417cf60 con 0x7f27640aca60 2021-04-09T18:34:07.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2769c72700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27640ad480 con 0x7f2764179bf0 2021-04-09T18:34:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2769c72700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27640ad6a0 con 0x7f276417c540 2021-04-09T18:34:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2763fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27640aca60 0x7f27640ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2763fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27640aca60 0x7f27640ace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59378/0 (socket says 172.21.15.32:59378) 2021-04-09T18:34:07.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2763fff700 1 -- 172.21.15.32:0/3198850170 learned_addr learned my addr 172.21.15.32:0/3198850170 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:07.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f27637fe700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764179bf0 0x7f2764179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2762ffd700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f276417c540 0x7f276417c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2763fff700 1 -- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f276417c540 msgr2=0x7f276417c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2763fff700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f276417c540 0x7f276417c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2763fff700 1 -- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764179bf0 msgr2=0x7f2764179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2763fff700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764179bf0 0x7f2764179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2763fff700 1 -- 172.21.15.32:0/3198850170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27640ad8c0 con 0x7f27640aca60 2021-04-09T18:34:07.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2762ffd700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f276417c540 0x7f276417c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:34:07.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f2763fff700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27640aca60 0x7f27640ace60 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7f275c007f90 tx=0x7f275c00ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:07.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.292 7f274bfff700 1 -- 172.21.15.32:0/3198850170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f275c000cc0 con 0x7f27640aca60 2021-04-09T18:34:07.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.293 7f274bfff700 1 -- 172.21.15.32:0/3198850170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275c007020 con 0x7f27640aca60 2021-04-09T18:34:07.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.293 7f2769c72700 1 -- 172.21.15.32:0/3198850170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27640ad8c0 con 0x7f27640aca60 2021-04-09T18:34:07.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.293 7f274bfff700 1 -- 172.21.15.32:0/3198850170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f275c007500 con 0x7f27640aca60 2021-04-09T18:34:07.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.293 7f2769c72700 1 -- 172.21.15.32:0/3198850170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27640ad8c0 con 0x7f27640aca60 2021-04-09T18:34:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.294 7f274bfff700 1 -- 172.21.15.32:0/3198850170 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f275c00ad90 con 0x7f27640aca60 2021-04-09T18:34:07.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.294 7f274bfff700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27440188c0 0x7f274401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.294 7f274bfff700 1 -- 172.21.15.32:0/3198850170 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f275c032b60 con 0x7f27640aca60 2021-04-09T18:34:07.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.294 7f27637fe700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27440188c0 0x7f274401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.294 7f27637fe700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27440188c0 0x7f274401adc0 crc :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:07.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.313 7f2769c72700 1 -- 172.21.15.32:0/3198850170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276417cf60 con 0x7f27640aca60 2021-04-09T18:34:07.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.317 7f274bfff700 1 -- 172.21.15.32:0/3198850170 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f275c00ad90 con 0x7f27640aca60 2021-04-09T18:34:07.521 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.504 7f2769c72700 1 -- 172.21.15.32:0/3198850170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestStandbyReplay.test_rank_stopped"]} v 0) v1 -- 0x7f276417cf60 con 0x7f27640aca60 2021-04-09T18:34:07.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.636 7f274bfff700 1 -- 172.21.15.32:0/3198850170 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestStandbyReplay.test_rank_stopped"]}]=0 v967) v1 ==== 141+0+0 (secure 0 0 0) 0x7f275c001270 con 0x7f27640aca60 2021-04-09T18:34:07.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 -- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27440188c0 msgr2=0x7f274401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27440188c0 0x7f274401adc0 crc :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 -- 172.21.15.32:0/3198850170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27640aca60 msgr2=0x7f27640ace60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27640aca60 0x7f27640ace60 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7f275c007f90 tx=0x7f275c00ac50).stop 2021-04-09T18:34:07.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 -- 172.21.15.32:0/3198850170 shutdown_connections 2021-04-09T18:34:07.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f27440188c0 0x7f274401adc0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27640aca60 0x7f27640ace60 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f276417c540 0x7f276417c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 --2- 172.21.15.32:0/3198850170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764179bf0 0x7f2764179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.670 7f2769c72700 1 -- 172.21.15.32:0/3198850170 >> 172.21.15.32:0/3198850170 conn(0x7f27641acae0 msgr2=0x7f276417ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:07.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.671 7f2769c72700 1 -- 172.21.15.32:0/3198850170 shutdown_connections 2021-04-09T18:34:07.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.671 7f2769c72700 1 -- 172.21.15.32:0/3198850170 wait complete. 2021-04-09T18:34:07.698 INFO:tasks.cephfs_test_runner:test_rank_stopped (tasks.cephfs.test_failover.TestStandbyReplay) ... ok 2021-04-09T18:34:07.698 INFO:tasks.cephfs_test_runner:Starting test: test_standby_replay_failure (tasks.cephfs.test_failover.TestStandbyReplay) 2021-04-09T18:34:07.699 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestStandbyReplay.test_standby_replay_failure' 2021-04-09T18:34:07.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.830 7f43594b5700 1 Processor -- start 2021-04-09T18:34:07.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.830 7f43594b5700 1 -- start start 2021-04-09T18:34:07.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.831 7f43594b5700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f435410da00 0x7f435410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.831 7f43594b5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435410d480 con 0x7f435410d010 2021-04-09T18:34:07.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.831 7f43594b5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43540d1f70 con 0x7f43540d7ca0 2021-04-09T18:34:07.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.831 7f43594b5700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43540d8d30 con 0x7f435410da00 2021-04-09T18:34:07.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.831 7f43532bf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f435410d010 0x7f43540d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40742/0 (socket says 172.21.15.32:40742) 2021-04-09T18:34:07.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.831 7f43532bf700 1 -- 172.21.15.32:0/964912506 learned_addr learned my addr 172.21.15.32:0/964912506 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:07.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.831 7f4353ac0700 1 --2- 172.21.15.32:0/964912506 >> v2:172.21.15.86:3300/0 conn(0x7f435410da00 0x7f435410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.831 7f43522bd700 1 -- 172.21.15.32:0/964912506 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774633530 0 0) 0x7f4344000ce0 con 0x7f435410d010 2021-04-09T18:34:07.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f43522bd700 1 -- 172.21.15.32:0/964912506 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433c002fe0 con 0x7f435410d010 2021-04-09T18:34:07.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f43522bd700 1 -- 172.21.15.32:0/964912506 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180101382 0 0) 0x7f4348001080 con 0x7f43540d7ca0 2021-04-09T18:34:07.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f43522bd700 1 -- 172.21.15.32:0/964912506 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433c003210 con 0x7f43540d7ca0 2021-04-09T18:34:07.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f4353ac0700 1 -- 172.21.15.32:0/964912506 >> v1:172.21.15.86:6789/0 conn(0x7f43540d7ca0 legacy=0x7f43540d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f4353ac0700 1 -- 172.21.15.32:0/964912506 >> v1:172.21.15.32:6789/0 conn(0x7f435410d010 legacy=0x7f43540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f43522bd700 1 -- 172.21.15.32:0/964912506 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2872185015 0 0) 0x7f4344000ed0 con 0x7f435410d010 2021-04-09T18:34:07.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f4353ac0700 1 -- 172.21.15.32:0/964912506 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43540d9220 con 0x7f435410da00 2021-04-09T18:34:07.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f4353ac0700 1 --2- 172.21.15.32:0/964912506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f435410da00 0x7f435410de00 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f434c006e50 tx=0x7f434c009e70).ready entity=mon.1 client_cookie=ec86317d0a23dc1 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:07.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f43522bd700 1 -- 172.21.15.32:0/964912506 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f434c006a20 con 0x7f435410da00 2021-04-09T18:34:07.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f43522bd700 1 -- 172.21.15.32:0/964912506 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c006790 con 0x7f435410da00 2021-04-09T18:34:07.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f43522bd700 1 -- 172.21.15.32:0/964912506 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f434c00b700 con 0x7f435410da00 2021-04-09T18:34:07.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f43594b5700 1 -- 172.21.15.32:0/964912506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f435410da00 msgr2=0x7f435410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.832 7f43594b5700 1 --2- 172.21.15.32:0/964912506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f435410da00 0x7f435410de00 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f434c006e50 tx=0x7f434c009e70).stop 2021-04-09T18:34:07.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.833 7f43594b5700 1 -- 172.21.15.32:0/964912506 shutdown_connections 2021-04-09T18:34:07.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.833 7f43594b5700 1 --2- 172.21.15.32:0/964912506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f435410da00 0x7f435410de00 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.833 7f43594b5700 1 -- 172.21.15.32:0/964912506 >> 172.21.15.32:0/964912506 conn(0x7f43540ce5f0 msgr2=0x7f43540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:07.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.833 7f43594b5700 1 -- 172.21.15.32:0/964912506 shutdown_connections 2021-04-09T18:34:07.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.833 7f43594b5700 1 -- 172.21.15.32:0/964912506 wait complete. 2021-04-09T18:34:07.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.853 7f43594b5700 1 Processor -- start 2021-04-09T18:34:07.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.853 7f43594b5700 1 -- start start 2021-04-09T18:34:07.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f43594b5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43540b42d0 0x7f4354183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f43594b5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43541ac090 0x7f43541ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f43594b5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4354184d30 0x7f4354185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f43594b5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43540a1a90 con 0x7f43541ac090 2021-04-09T18:34:07.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f43594b5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43541acab0 con 0x7f4354184d30 2021-04-09T18:34:07.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f43594b5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4354185a00 con 0x7f43540b42d0 2021-04-09T18:34:07.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4352abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43541ac090 0x7f43541ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4352abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43541ac090 0x7f43541ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59394/0 (socket says 172.21.15.32:59394) 2021-04-09T18:34:07.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4352abe700 1 -- 172.21.15.32:0/3226029330 learned_addr learned my addr 172.21.15.32:0/3226029330 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:07.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f43532bf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43540b42d0 0x7f4354183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4353ac0700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4354184d30 0x7f4354185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4352abe700 1 -- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43540b42d0 msgr2=0x7f4354183520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4352abe700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43540b42d0 0x7f4354183520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4352abe700 1 -- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4354184d30 msgr2=0x7f4354185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:07.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4352abe700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4354184d30 0x7f4354185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:07.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4352abe700 1 -- 172.21.15.32:0/3226029330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4354185750 con 0x7f43541ac090 2021-04-09T18:34:07.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.854 7f4352abe700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43541ac090 0x7f43541ac490 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 rx=0x7f4348006f00 tx=0x7f434800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:07.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.855 7f433bfff700 1 -- 172.21.15.32:0/3226029330 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f434800bd30 con 0x7f43541ac090 2021-04-09T18:34:07.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.855 7f433bfff700 1 -- 172.21.15.32:0/3226029330 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4348000e20 con 0x7f43541ac090 2021-04-09T18:34:07.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.855 7f433bfff700 1 -- 172.21.15.32:0/3226029330 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43480054f0 con 0x7f43541ac090 2021-04-09T18:34:07.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.855 7f43594b5700 1 -- 172.21.15.32:0/3226029330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4354185750 con 0x7f43541ac090 2021-04-09T18:34:07.887 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.855 7f43594b5700 1 -- 172.21.15.32:0/3226029330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4354185750 con 0x7f43541ac090 2021-04-09T18:34:07.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.855 7f433bfff700 1 -- 172.21.15.32:0/3226029330 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4348006240 con 0x7f43541ac090 2021-04-09T18:34:07.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.855 7f433bfff700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43340188c0 0x7f433401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:07.888 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.855 7f433bfff700 1 -- 172.21.15.32:0/3226029330 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f4348032780 con 0x7f43541ac090 2021-04-09T18:34:07.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.856 7f43532bf700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43340188c0 0x7f433401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:07.889 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.856 7f43532bf700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43340188c0 0x7f433401adc0 crc :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:07.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.875 7f43594b5700 1 -- 172.21.15.32:0/3226029330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4354182d10 con 0x7f43541ac090 2021-04-09T18:34:07.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:07.879 7f433bfff700 1 -- 172.21.15.32:0/3226029330 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4348006030 con 0x7f43541ac090 2021-04-09T18:34:08.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.066 7f43594b5700 1 -- 172.21.15.32:0/3226029330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestStandbyReplay.test_standby_replay_failure"]} v 0) v1 -- 0x7f43540a1a90 con 0x7f43541ac090 2021-04-09T18:34:08.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.641 7f433bfff700 1 -- 172.21.15.32:0/3226029330 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestStandbyReplay.test_standby_replay_failure"]}]=0 v968) v1 ==== 154+0+0 (secure 0 0 0) 0x7f4348006240 con 0x7f43541ac090 2021-04-09T18:34:08.701 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 -- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43340188c0 msgr2=0x7f433401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43340188c0 0x7f433401adc0 crc :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 -- 172.21.15.32:0/3226029330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43541ac090 msgr2=0x7f43541ac490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43541ac090 0x7f43541ac490 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 rx=0x7f4348006f00 tx=0x7f434800b940).stop 2021-04-09T18:34:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 -- 172.21.15.32:0/3226029330 shutdown_connections 2021-04-09T18:34:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f43340188c0 0x7f433401adc0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43541ac090 0x7f43541ac490 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4354184d30 0x7f4354185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 --2- 172.21.15.32:0/3226029330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43540b42d0 0x7f4354183520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.684 7f43594b5700 1 -- 172.21.15.32:0/3226029330 >> 172.21.15.32:0/3226029330 conn(0x7f435419b890 msgr2=0x7f43541a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:08.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.685 7f43594b5700 1 -- 172.21.15.32:0/3226029330 shutdown_connections 2021-04-09T18:34:08.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.685 7f43594b5700 1 -- 172.21.15.32:0/3226029330 wait complete. 2021-04-09T18:34:08.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:34:08.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:34:08.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:34:08.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.857 7fc171846700 1 Processor -- start 2021-04-09T18:34:08.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.857 7fc171846700 1 -- start start 2021-04-09T18:34:08.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc171846700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc16c10d010 0x7fc16c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:08.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc171846700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc16c0d1da0 0x7fc16c0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:08.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc171846700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16c10d480 con 0x7fc16c0d82c0 2021-04-09T18:34:08.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc171846700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c0d8d30 con 0x7fc16c10d010 2021-04-09T18:34:08.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc171846700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c0d9220 con 0x7fc16c0d1da0 2021-04-09T18:34:08.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc16b574700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc16c10d010 0x7fc16c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:08.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc16bd75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc16c0d82c0 0x7fc16c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40760/0 (socket says 172.21.15.32:40760) 2021-04-09T18:34:08.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc16bd75700 1 -- 172.21.15.32:0/3683265289 learned_addr learned my addr 172.21.15.32:0/3683265289 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:08.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc16ad73700 1 --2- 172.21.15.32:0/3683265289 >> v2:172.21.15.86:3300/0 conn(0x7fc16c0d1da0 0x7fc16c0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:08.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc16b574700 1 -- 172.21.15.32:0/3683265289 >> v2:172.21.15.86:3300/0 conn(0x7fc16c0d1da0 msgr2=0x7fc16c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:08.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc16b574700 1 --2- 172.21.15.32:0/3683265289 >> v2:172.21.15.86:3300/0 conn(0x7fc16c0d1da0 0x7fc16c0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.858 7fc16b574700 1 -- 172.21.15.32:0/3683265289 >> v1:172.21.15.32:6789/0 conn(0x7fc16c0d82c0 legacy=0x7fc16c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:08.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.859 7fc16b574700 1 -- 172.21.15.32:0/3683265289 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c0d86f0 con 0x7fc16c10d010 2021-04-09T18:34:08.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.859 7fc16a572700 1 -- 172.21.15.32:0/3683265289 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047368213 0 0) 0x7fc164000d30 con 0x7fc16c0d82c0 2021-04-09T18:34:08.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.859 7fc16b574700 1 --2- 172.21.15.32:0/3683265289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c10d010 0x7fc16c0d1780 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 rx=0x7fc15c006650 tx=0x7fc15c009a00).ready entity=mon.0 client_cookie=8b35250a6f536edf server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:08.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.859 7fc16a572700 1 -- 172.21.15.32:0/3683265289 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc15c009df0 con 0x7fc16c10d010 2021-04-09T18:34:08.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.859 7fc16a572700 1 -- 172.21.15.32:0/3683265289 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c0009b0 con 0x7fc16c10d010 2021-04-09T18:34:08.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.859 7fc16a572700 1 -- 172.21.15.32:0/3683265289 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc15c00c2b0 con 0x7fc16c10d010 2021-04-09T18:34:08.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.859 7fc171846700 1 -- 172.21.15.32:0/3683265289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c10d010 msgr2=0x7fc16c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:08.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.859 7fc171846700 1 --2- 172.21.15.32:0/3683265289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c10d010 0x7fc16c0d1780 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 rx=0x7fc15c006650 tx=0x7fc15c009a00).stop 2021-04-09T18:34:08.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.860 7fc171846700 1 -- 172.21.15.32:0/3683265289 shutdown_connections 2021-04-09T18:34:08.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.860 7fc171846700 1 --2- 172.21.15.32:0/3683265289 >> v2:172.21.15.86:3300/0 conn(0x7fc16c0d1da0 0x7fc16c0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.860 7fc171846700 1 --2- 172.21.15.32:0/3683265289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c10d010 0x7fc16c0d1780 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.860 7fc171846700 1 -- 172.21.15.32:0/3683265289 >> 172.21.15.32:0/3683265289 conn(0x7fc16c0ce5f0 msgr2=0x7fc16c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:08.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.860 7fc171846700 1 -- 172.21.15.32:0/3683265289 shutdown_connections 2021-04-09T18:34:08.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.860 7fc171846700 1 -- 172.21.15.32:0/3683265289 wait complete. 2021-04-09T18:34:08.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.880 7fc171846700 1 Processor -- start 2021-04-09T18:34:08.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.880 7fc171846700 1 -- start start 2021-04-09T18:34:08.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc171846700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc16c09e530 0x7fc16c09b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:08.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc171846700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c09e930 0x7fc16c09f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:08.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc171846700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc16c09f990 0x7fc16c094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:08.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc171846700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c09edf0 con 0x7fc16c09e930 2021-04-09T18:34:08.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc171846700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c09fd90 con 0x7fc16c09e530 2021-04-09T18:34:08.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc171846700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c09bd40 con 0x7fc16c09f990 2021-04-09T18:34:08.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16ad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c09e930 0x7fc16c09f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:08.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16ad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c09e930 0x7fc16c09f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59408/0 (socket says 172.21.15.32:59408) 2021-04-09T18:34:08.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16ad73700 1 -- 172.21.15.32:0/290747638 learned_addr learned my addr 172.21.15.32:0/290747638 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:08.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16bd75700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc16c09f990 0x7fc16c094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:08.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16b574700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc16c09e530 0x7fc16c09b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:08.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16ad73700 1 -- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc16c09f990 msgr2=0x7fc16c094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:08.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16ad73700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc16c09f990 0x7fc16c094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16ad73700 1 -- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc16c09e530 msgr2=0x7fc16c09b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:08.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16ad73700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc16c09e530 0x7fc16c09b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:08.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16ad73700 1 -- 172.21.15.32:0/290747638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c095290 con 0x7fc16c09e930 2021-04-09T18:34:08.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc16ad73700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c09e930 0x7fc16c09f370 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7fc160006f00 tx=0x7fc16000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:08.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.881 7fc153fff700 1 -- 172.21.15.32:0/290747638 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc16000bd30 con 0x7fc16c09e930 2021-04-09T18:34:08.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.882 7fc153fff700 1 -- 172.21.15.32:0/290747638 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc160002030 con 0x7fc16c09e930 2021-04-09T18:34:08.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.882 7fc171846700 1 -- 172.21.15.32:0/290747638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16c095290 con 0x7fc16c09e930 2021-04-09T18:34:08.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.882 7fc153fff700 1 -- 172.21.15.32:0/290747638 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc160002590 con 0x7fc16c09e930 2021-04-09T18:34:08.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.882 7fc171846700 1 -- 172.21.15.32:0/290747638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16c095290 con 0x7fc16c09e930 2021-04-09T18:34:08.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.882 7fc153fff700 1 -- 172.21.15.32:0/290747638 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc160000c90 con 0x7fc16c09e930 2021-04-09T18:34:08.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.882 7fc153fff700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc14c018930 0x7fc14c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:08.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.882 7fc153fff700 1 -- 172.21.15.32:0/290747638 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fc160032760 con 0x7fc16c09e930 2021-04-09T18:34:08.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.882 7fc16b574700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc14c018930 0x7fc14c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:08.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.883 7fc16b574700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc14c018930 0x7fc14c01ae10 crc :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:08.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.902 7fc171846700 1 -- 172.21.15.32:0/290747638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16c095b10 con 0x7fc16c09e930 2021-04-09T18:34:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:08.906 7fc153fff700 1 -- 172.21.15.32:0/290747638 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc16001f780 con 0x7fc16c09e930 2021-04-09T18:34:09.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.118 7fc171846700 1 -- 172.21.15.32:0/290747638 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc16c095b10 con 0x7fc16c09e930 2021-04-09T18:34:09.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.122 7fc153fff700 1 -- 172.21.15.32:0/290747638 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc160000c90 con 0x7fc16c09e930 2021-04-09T18:34:09.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 -- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc14c018930 msgr2=0x7fc14c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc14c018930 0x7fc14c01ae10 crc :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 -- 172.21.15.32:0/290747638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c09e930 msgr2=0x7fc16c09f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c09e930 0x7fc16c09f370 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7fc160006f00 tx=0x7fc16000b940).stop 2021-04-09T18:34:09.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 -- 172.21.15.32:0/290747638 shutdown_connections 2021-04-09T18:34:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc14c018930 0x7fc14c01ae10 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc16c09e930 0x7fc16c09f370 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc16c09f990 0x7fc16c094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 --2- 172.21.15.32:0/290747638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc16c09e530 0x7fc16c09b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.127 7fc171846700 1 -- 172.21.15.32:0/290747638 >> 172.21.15.32:0/290747638 conn(0x7fc16c0c4290 msgr2=0x7fc16c0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.128 7fc171846700 1 -- 172.21.15.32:0/290747638 shutdown_connections 2021-04-09T18:34:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.128 7fc171846700 1 -- 172.21.15.32:0/290747638 wait complete. 2021-04-09T18:34:09.154 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:34:09.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:34:09.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:34:09.317 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:34:09.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:09.318 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:34:09.318 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:34:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:34:09.363 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:34:09.363 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:34:09.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:09.364 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:34:09.365 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:34:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:34:09.409 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:34:09.410 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:34:09.410 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:34:09.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.542 7feccd8fe700 1 Processor -- start 2021-04-09T18:34:09.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.542 7feccd8fe700 1 -- start start 2021-04-09T18:34:09.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7feccd8fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc810d010 0x7fecc80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:09.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7feccd8fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fecc80d1da0 0x7fecc80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:09.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7feccd8fe700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fecc80d82c0 0x7fecc810da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:09.582 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7feccd8fe700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc810e780 con 0x7fecc80d1da0 2021-04-09T18:34:09.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7feccd8fe700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc80d8d30 con 0x7fecc80d82c0 2021-04-09T18:34:09.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7feccd8fe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc80d9220 con 0x7fecc810d010 2021-04-09T18:34:09.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6d73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fecc80d1da0 0x7fecc80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:09.583 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6d73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fecc80d1da0 0x7fecc80d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59418/0 (socket says 172.21.15.32:59418) 2021-04-09T18:34:09.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6d73700 1 -- 172.21.15.32:0/3501160260 learned_addr learned my addr 172.21.15.32:0/3501160260 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:09.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc7574700 1 --2- 172.21.15.32:0/3501160260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc810d010 0x7fecc80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:09.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc7d75700 1 --2- 172.21.15.32:0/3501160260 >> v2:172.21.15.86:3300/0 conn(0x7fecc80d82c0 0x7fecc810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:09.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6d73700 1 -- 172.21.15.32:0/3501160260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc810d010 msgr2=0x7fecc80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:09.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6d73700 1 --2- 172.21.15.32:0/3501160260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc810d010 0x7fecc80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6d73700 1 -- 172.21.15.32:0/3501160260 >> v2:172.21.15.86:3300/0 conn(0x7fecc80d82c0 msgr2=0x7fecc810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:09.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6d73700 1 --2- 172.21.15.32:0/3501160260 >> v2:172.21.15.86:3300/0 conn(0x7fecc80d82c0 0x7fecc810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6d73700 1 -- 172.21.15.32:0/3501160260 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc810d480 con 0x7fecc80d1da0 2021-04-09T18:34:09.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6d73700 1 --2- 172.21.15.32:0/3501160260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc80d1da0 0x7fecc80d7ca0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7fecbc006700 tx=0x7fecbc009b00).ready entity=mon.0 client_cookie=3372a6840a5a663a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:09.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6572700 1 -- 172.21.15.32:0/3501160260 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecbc00c0c0 con 0x7fecc80d1da0 2021-04-09T18:34:09.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6572700 1 -- 172.21.15.32:0/3501160260 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecbc00c400 con 0x7fecc80d1da0 2021-04-09T18:34:09.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.543 7fecc6572700 1 -- 172.21.15.32:0/3501160260 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecbc00c8e0 con 0x7fecc80d1da0 2021-04-09T18:34:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.544 7feccd8fe700 1 -- 172.21.15.32:0/3501160260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc80d1da0 msgr2=0x7fecc80d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.544 7feccd8fe700 1 --2- 172.21.15.32:0/3501160260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc80d1da0 0x7fecc80d7ca0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7fecbc006700 tx=0x7fecbc009b00).stop 2021-04-09T18:34:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.544 7feccd8fe700 1 -- 172.21.15.32:0/3501160260 shutdown_connections 2021-04-09T18:34:09.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.544 7feccd8fe700 1 --2- 172.21.15.32:0/3501160260 >> v2:172.21.15.86:3300/0 conn(0x7fecc80d82c0 0x7fecc810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.544 7feccd8fe700 1 --2- 172.21.15.32:0/3501160260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc80d1da0 0x7fecc80d7ca0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.544 7feccd8fe700 1 --2- 172.21.15.32:0/3501160260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc810d010 0x7fecc80d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.544 7feccd8fe700 1 -- 172.21.15.32:0/3501160260 >> 172.21.15.32:0/3501160260 conn(0x7fecc80ce5f0 msgr2=0x7fecc80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.544 7feccd8fe700 1 -- 172.21.15.32:0/3501160260 shutdown_connections 2021-04-09T18:34:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.544 7feccd8fe700 1 -- 172.21.15.32:0/3501160260 wait complete. 2021-04-09T18:34:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.564 7feccd8fe700 1 Processor -- start 2021-04-09T18:34:09.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.564 7feccd8fe700 1 -- start start 2021-04-09T18:34:09.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7feccd8fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc81aca20 0x7fecc81ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:09.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7feccd8fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc818bd40 0x7fecc818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:09.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7feccd8fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc808fba0 0x7fecc808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:09.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7feccd8fe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc818c7e0 con 0x7fecc818bd40 2021-04-09T18:34:09.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7feccd8fe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc818ca30 con 0x7fecc808fba0 2021-04-09T18:34:09.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7feccd8fe700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc80905f0 con 0x7fecc81aca20 2021-04-09T18:34:09.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc6d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc818bd40 0x7fecc818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:09.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc6d73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc818bd40 0x7fecc818c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59420/0 (socket says 172.21.15.32:59420) 2021-04-09T18:34:09.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc6d73700 1 -- 172.21.15.32:0/481167663 learned_addr learned my addr 172.21.15.32:0/481167663 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:09.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc7d75700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc808fba0 0x7fecc808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:09.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc7574700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc81aca20 0x7fecc81ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:09.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc6d73700 1 -- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc81aca20 msgr2=0x7fecc81ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:09.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc6d73700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc81aca20 0x7fecc81ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc6d73700 1 -- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc808fba0 msgr2=0x7fecc808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:09.596 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc6d73700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc808fba0 0x7fecc808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.565 7fecc6d73700 1 -- 172.21.15.32:0/481167663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc80b4300 con 0x7fecc818bd40 2021-04-09T18:34:09.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.566 7fecc6d73700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc818bd40 0x7fecc818c140 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 rx=0x7fecbc00b660 tx=0x7fecbc00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:09.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.566 7fecaffff700 1 -- 172.21.15.32:0/481167663 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecbc009440 con 0x7fecc818bd40 2021-04-09T18:34:09.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.566 7fecaffff700 1 -- 172.21.15.32:0/481167663 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecbc006390 con 0x7fecc818bd40 2021-04-09T18:34:09.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.566 7fecaffff700 1 -- 172.21.15.32:0/481167663 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecbc002030 con 0x7fecc818bd40 2021-04-09T18:34:09.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.566 7feccd8fe700 1 -- 172.21.15.32:0/481167663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc80b4300 con 0x7fecc818bd40 2021-04-09T18:34:09.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.566 7feccd8fe700 1 -- 172.21.15.32:0/481167663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc80b4300 con 0x7fecc818bd40 2021-04-09T18:34:09.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.567 7fecaffff700 1 -- 172.21.15.32:0/481167663 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fecbc0093e0 con 0x7fecc818bd40 2021-04-09T18:34:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.567 7fecaffff700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feca80188c0 0x7feca801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.567 7fecaffff700 1 -- 172.21.15.32:0/481167663 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fecbc0324d0 con 0x7fecc818bd40 2021-04-09T18:34:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.567 7fecc7574700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feca80188c0 0x7feca801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:09.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.567 7fecc7574700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feca80188c0 0x7feca801adc0 crc :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:09.603 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.587 7feccd8fe700 1 -- 172.21.15.32:0/481167663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc8182d70 con 0x7fecc818bd40 2021-04-09T18:34:09.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.590 7fecaffff700 1 -- 172.21.15.32:0/481167663 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fecbc0090c0 con 0x7fecc818bd40 2021-04-09T18:34:09.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.779 7feccd8fe700 1 -- 172.21.15.32:0/481167663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fecc818c760 con 0x7fecc818bd40 2021-04-09T18:34:09.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.780 7fecaffff700 1 -- 172.21.15.32:0/481167663 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v254) v1 ==== 81+0+25548 (secure 0 0 0) 0x7fecbc0093e0 con 0x7fecc818bd40 2021-04-09T18:34:09.799 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:09.799 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:09.800 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 254, 2021-04-09T18:34:09.800 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:09.800 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:34:09.800 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:34:06.630329", 2021-04-09T18:34:09.801 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:34:09.801 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:34:09.801 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:34:09.801 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:34:09.802 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:34:09.802 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:34:09.802 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:34:09.802 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:34:09.803 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:34:09.803 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:34:09.803 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:34:09.803 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:34:09.804 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:34:09.804 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:34:09.804 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:34:09.804 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 43, 2021-04-09T18:34:09.805 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:34:09.805 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:34:09.805 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:34:09.805 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:34:09.805 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:34:09.806 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.806 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:34:09.806 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:34:09.806 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:34:09.807 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:34:09.807 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:34:09.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:34:09.807 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:34:09.808 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:34:09.808 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:34:09.808 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:34:09.808 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:34:09.809 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:34:09.809 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:34:09.809 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:34:09.809 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:34:09.810 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:34:09.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:34:09.810 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:34:09.810 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:34:09.810 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:34:09.811 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:34:09.811 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:34:09.811 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:34:09.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.812 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:34:09.812 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:34:09.812 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:34:09.812 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:34:09.813 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:34:09.813 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:34:09.813 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:34:09.813 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:34:09.813 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:34:09.814 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:34:09.814 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:34:09.814 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:34:09.814 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:34:09.815 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:34:09.815 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:34:09.815 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:34:09.815 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:34:09.816 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:34:09.816 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:34:09.816 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:34:09.816 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:34:09.817 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:34:09.817 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:34:09.817 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:34:09.817 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:34:09.818 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:34:09.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:34:09.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.818 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:34:09.819 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:34:09.819 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:34:09.819 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:34:09.819 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:34:09.820 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:34:09.820 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:34:09.820 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:34:09.820 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:34:09.820 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:34:09.821 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:34:09.821 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:34:09.821 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:34:09.821 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:34:09.822 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.822 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.822 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:34:09.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:34:09.823 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.823 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:34:09.823 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:34:09.823 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:09.823 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:09.824 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:09.824 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:09.824 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:09.824 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:09.825 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:09.825 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 227, 2021-04-09T18:34:09.825 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:09.825 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:09.826 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:09.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.826 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.826 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:34:09.827 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:09.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.827 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.828 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:34:09.828 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:09.828 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.828 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.829 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:09.829 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.829 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.829 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.830 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:34:09.830 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:09.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.830 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.831 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:34:09.831 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:09.831 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.832 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.832 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:09.832 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.832 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.833 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.833 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:34:09.833 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:09.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.834 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.834 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.834 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:34:09.834 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:09.834 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.835 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.835 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:09.835 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.836 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.836 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:34:09.836 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:09.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.837 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.837 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:34:09.837 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:34:09.838 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.838 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.838 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:34:09.839 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:34:09.839 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:34:09.839 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:34:09.839 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:09.840 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:09.840 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:09.840 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.841 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.841 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:34:09.841 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:34:09.841 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:09.841 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:09.842 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:09.842 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:09.842 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:09.842 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:09.843 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.844 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:34:09.845 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:09.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.845 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.845 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:34:09.846 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:09.846 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.846 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.847 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:09.847 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.847 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.847 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.848 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:34:09.848 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:09.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.848 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.849 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.850 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:09.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:34:09.852 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:09.853 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:34:09.854 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.855 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.856 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:34:09.857 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:09.858 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:09.858 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:09.858 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.858 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.859 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:34:09.859 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:34:09.859 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:09.859 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:09.860 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:09.860 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:09.860 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:09.860 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:09.861 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:09.861 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:09.861 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:09.861 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:09.861 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:09.862 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.862 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.862 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:34:09.863 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:09.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.863 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.863 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.864 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:34:09.864 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:09.864 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.864 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.865 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:09.865 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.865 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.866 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:34:09.866 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:09.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.867 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.867 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:34:09.867 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:09.867 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.868 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.868 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:09.868 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.868 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.869 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:34:09.869 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:09.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.870 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.870 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:34:09.870 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:09.870 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.871 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.871 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:09.871 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.872 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.872 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:34:09.872 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:09.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.873 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.873 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.873 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:34:09.873 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:34:09.874 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.874 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.874 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:34:09.875 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:34:09.875 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:34:09.875 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:34:09.875 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:09.876 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:09.876 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:09.876 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.876 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.877 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:34:09.877 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:34:09.877 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:09.877 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:09.878 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:09.878 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:09.878 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:09.878 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:09.879 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:09.879 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:09.879 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:09.879 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:09.879 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:09.880 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.880 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.880 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:34:09.881 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:09.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.881 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.882 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:34:09.882 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:09.882 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.882 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.883 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:09.883 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.883 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.883 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.884 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:34:09.884 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:09.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.884 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.885 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:34:09.885 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:09.885 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.885 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.886 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:09.886 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.886 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:34:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:09.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:34:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:09.888 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.889 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.889 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:09.889 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.890 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.890 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:34:09.890 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:09.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.891 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.891 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:34:09.891 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:34:09.892 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.892 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.892 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:34:09.893 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:34:09.893 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:34:09.893 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:34:09.893 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:09.893 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:09.894 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:09.894 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.894 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.895 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:34:09.895 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:34:09.895 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:09.895 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:09.896 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:09.896 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:09.896 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:09.896 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:09.896 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:09.897 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 247, 2021-04-09T18:34:09.897 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:09.897 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:09.897 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:09.898 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.898 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.898 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:34:09.899 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:09.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.899 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.900 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:34:09.900 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:09.900 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.900 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.901 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:09.901 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.901 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.902 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:34:09.902 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:09.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.902 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.903 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.903 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:34:09.903 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:09.903 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.904 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.904 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:09.904 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.904 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.905 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:34:09.905 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:09.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.906 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.906 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:34:09.906 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:09.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.907 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.907 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:09.907 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.908 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:34:09.908 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:09.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.909 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.909 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:34:09.910 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:34:09.910 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.910 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.911 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:34:09.911 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:34:09.911 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:34:09.911 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:34:09.912 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:09.912 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:09.912 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:09.912 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.913 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.913 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:34:09.913 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:34:09.913 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:09.914 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:09.914 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:09.914 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:09.914 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:09.914 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:09.915 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:09.915 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:09.915 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:09.915 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:09.916 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:09.916 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.916 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.917 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:34:09.917 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:09.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.917 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.917 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.918 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:34:09.918 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:09.919 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.919 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.920 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:09.920 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.920 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.920 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.920 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:34:09.921 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:09.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.921 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.921 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.922 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:34:09.922 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:09.922 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.922 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.923 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:09.923 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.923 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.924 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:34:09.924 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:09.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.925 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.925 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:34:09.925 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:09.926 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.926 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.926 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:09.926 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.927 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.927 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:34:09.927 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:09.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.928 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.928 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.928 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:34:09.929 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:34:09.929 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.929 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.930 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:34:09.930 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:34:09.930 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:34:09.930 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:34:09.930 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:09.931 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:09.931 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:09.931 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.932 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.932 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:34:09.932 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:34:09.932 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:09.933 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:09.933 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:09.933 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:09.933 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:09.933 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:09.934 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:09.934 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 249, 2021-04-09T18:34:09.934 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:09.934 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:09.935 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:09.935 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.935 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.936 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:34:09.936 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:09.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.937 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:34:09.937 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:09.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.938 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.938 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:09.938 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.939 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.939 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:34:09.939 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:09.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.940 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.940 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:34:09.941 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:09.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.941 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.942 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:09.942 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.942 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.943 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:34:09.943 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:09.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.943 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.944 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:34:09.944 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:09.944 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.944 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.945 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:09.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.946 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:34:09.946 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:09.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.947 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:34:09.947 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:34:09.948 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.948 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.948 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:34:09.949 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:34:09.949 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:34:09.949 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:34:09.949 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:09.950 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:09.950 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:09.950 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.951 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:34:09.951 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:34:09.951 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:34:09.951 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:34:09.952 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:34:09.952 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:34:09.952 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:34:09.953 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:34:09.953 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:34:09.953 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 244, 2021-04-09T18:34:09.953 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:34:09.954 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:34:09.954 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:34:09.954 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.955 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.955 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:34:09.955 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:09.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.956 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.956 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.956 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:34:09.957 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:09.957 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.957 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.957 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:34:09.958 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.958 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.958 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:34:09.959 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:09.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.959 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.960 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:34:09.960 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:09.960 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.960 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.961 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:34:09.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:34:09.962 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:09.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.963 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:34:09.963 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:09.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.964 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.964 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:34:09.964 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:34:09.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:34:09.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:34:09.965 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:09.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.966 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.966 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:09.966 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:34:09.967 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:34:09.967 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.967 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.967 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:34:09.968 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:34:09.968 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:34:09.968 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:34:09.968 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:34:09.969 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:34:09.969 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:34:09.969 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:09.969 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.970 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:34:09.970 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:34:09.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.970 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:34:09.970 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:09.971 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:09.971 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:09.971 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:09.971 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:09.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.972 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.972 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:34:09.972 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:09.973 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:09.973 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:09.973 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:09.973 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:09.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.974 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.974 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:34:09.974 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:09.974 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:09.975 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:09.975 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:09.975 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:09.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.976 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.976 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:34:09.976 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:09.976 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:09.977 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:09.977 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:09.977 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:09.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.977 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.978 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:34:09.978 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:09.978 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:09.978 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:09.979 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:09.979 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:09.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.979 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:34:09.980 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:09.980 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:09.980 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:09.980 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:09.981 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:09.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.981 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:34:09.982 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:09.982 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:09.982 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:09.982 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:09.983 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:09.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.983 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:09.983 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:34:09.984 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:34:09.984 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:34:09.984 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:34:09.984 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:34:09.984 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:34:09.985 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.985 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:34:09.985 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:34:09.985 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:34:09.986 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:34:09.986 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:34:09.986 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:34:09.986 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6836/154900016": "2021-04-10 18:34:03.434712", 2021-04-09T18:34:09.986 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.86:6834/154900016": "2021-04-10 18:34:03.434712" 2021-04-09T18:34:09.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.987 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:34:09.987 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:34:09.987 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:34:09.988 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:34:09.988 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:34:09.988 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:34:09.988 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:34:09.989 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:34:09.989 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:09.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:09.989 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:34:09.989 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:34:09.990 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:34:09.990 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:34:09.990 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:34:09.990 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 -- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feca80188c0 msgr2=0x7feca801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:09.992 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feca80188c0 0x7feca801adc0 crc :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 -- 172.21.15.32:0/481167663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc818bd40 msgr2=0x7fecc818c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc818bd40 0x7fecc818c140 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 rx=0x7fecbc00b660 tx=0x7fecbc00ba00).stop 2021-04-09T18:34:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 -- 172.21.15.32:0/481167663 shutdown_connections 2021-04-09T18:34:09.993 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7feca80188c0 0x7feca801adc0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc818bd40 0x7fecc818c140 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecc808fba0 0x7fecc808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 --2- 172.21.15.32:0/481167663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecc81aca20 0x7fecc81ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:09.994 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 -- 172.21.15.32:0/481167663 >> 172.21.15.32:0/481167663 conn(0x7fecc819b460 msgr2=0x7fecc819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 -- 172.21.15.32:0/481167663 shutdown_connections 2021-04-09T18:34:09.995 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:09.784 7feccd8fe700 1 -- 172.21.15.32:0/481167663 wait complete. 2021-04-09T18:34:09.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:10.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.129 7f1cb79c3700 1 Processor -- start 2021-04-09T18:34:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.129 7f1cb79c3700 1 -- start start 2021-04-09T18:34:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb79c3700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1cb010d010 0x7f1cb00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb79c3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb010d480 con 0x7f1cb00d8490 2021-04-09T18:34:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb79c3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb010e1f0 con 0x7f1cb00d1da0 2021-04-09T18:34:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb79c3700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb00d8d30 con 0x7f1cb010d010 2021-04-09T18:34:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb57df700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1cb010d010 0x7f1cb00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb5fe0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1cb00d8490 0x7f1cb010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40788/0 (socket says 172.21.15.32:40788) 2021-04-09T18:34:10.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb5fe0700 1 -- 172.21.15.32:0/2948193422 learned_addr learned my addr 172.21.15.32:0/2948193422 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb57df700 1 -- 172.21.15.32:0/2948193422 >> v1:172.21.15.86:6789/0 conn(0x7f1cb00d1da0 legacy=0x7f1cb00d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb57df700 1 -- 172.21.15.32:0/2948193422 >> v1:172.21.15.32:6789/0 conn(0x7f1cb00d8490 legacy=0x7f1cb010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb57df700 1 -- 172.21.15.32:0/2948193422 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb00d9220 con 0x7f1cb010d010 2021-04-09T18:34:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1ca7fff700 1 -- 172.21.15.32:0/2948193422 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639516855 0 0) 0x7f1ca8000d30 con 0x7f1cb00d8490 2021-04-09T18:34:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.130 7f1cb57df700 1 --2- 172.21.15.32:0/2948193422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb010d010 0x7f1cb00d1780 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 rx=0x7f1cac006650 tx=0x7f1cac009a00).ready entity=mon.0 client_cookie=2f62a584ad3759b4 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:10.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.131 7f1ca7fff700 1 -- 172.21.15.32:0/2948193422 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cac009df0 con 0x7f1cb010d010 2021-04-09T18:34:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.131 7f1ca7fff700 1 -- 172.21.15.32:0/2948193422 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac0009b0 con 0x7f1cb010d010 2021-04-09T18:34:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.131 7f1ca7fff700 1 -- 172.21.15.32:0/2948193422 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cac00c2b0 con 0x7f1cb010d010 2021-04-09T18:34:10.173 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.131 7f1cb79c3700 1 -- 172.21.15.32:0/2948193422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb010d010 msgr2=0x7f1cb00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.131 7f1cb79c3700 1 --2- 172.21.15.32:0/2948193422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb010d010 0x7f1cb00d1780 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 rx=0x7f1cac006650 tx=0x7f1cac009a00).stop 2021-04-09T18:34:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.131 7f1cb79c3700 1 -- 172.21.15.32:0/2948193422 shutdown_connections 2021-04-09T18:34:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.131 7f1cb79c3700 1 --2- 172.21.15.32:0/2948193422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb010d010 0x7f1cb00d1780 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.131 7f1cb79c3700 1 -- 172.21.15.32:0/2948193422 >> 172.21.15.32:0/2948193422 conn(0x7f1cb00ce5f0 msgr2=0x7f1cb00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:10.174 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.131 7f1cb79c3700 1 -- 172.21.15.32:0/2948193422 shutdown_connections 2021-04-09T18:34:10.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.132 7f1cb79c3700 1 -- 172.21.15.32:0/2948193422 wait complete. 2021-04-09T18:34:10.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb79c3700 1 Processor -- start 2021-04-09T18:34:10.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb79c3700 1 -- start start 2021-04-09T18:34:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb79c3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cb00a75d0 0x7f1cb01bef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb79c3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb00bc870 0x7f1cb00b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.178 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb79c3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cb01bf520 0x7f1cb00b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb79c3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb01abd30 con 0x7f1cb00bc870 2021-04-09T18:34:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb79c3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb00b4290 con 0x7f1cb01bf520 2021-04-09T18:34:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb79c3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb00a7b00 con 0x7f1cb00a75d0 2021-04-09T18:34:10.179 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb4fde700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb00bc870 0x7f1cb00b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.152 7f1cb4fde700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb00bc870 0x7f1cb00b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59434/0 (socket says 172.21.15.32:59434) 2021-04-09T18:34:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb4fde700 1 -- 172.21.15.32:0/4230485469 learned_addr learned my addr 172.21.15.32:0/4230485469 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:10.180 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb57df700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cb00a75d0 0x7f1cb01bef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb5fe0700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cb01bf520 0x7f1cb00b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb4fde700 1 -- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cb00a75d0 msgr2=0x7f1cb01bef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.181 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb4fde700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cb00a75d0 0x7f1cb01bef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb4fde700 1 -- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cb01bf520 msgr2=0x7f1cb00b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb4fde700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cb01bf520 0x7f1cb00b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb4fde700 1 -- 172.21.15.32:0/4230485469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb00a7d20 con 0x7f1cb00bc870 2021-04-09T18:34:10.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb5fe0700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cb01bf520 0x7f1cb00b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:34:10.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb4fde700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb00bc870 0x7f1cb00b3560 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7f1ca000b530 tx=0x7f1ca000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:10.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1ca5ffb700 1 -- 172.21.15.32:0/4230485469 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ca000bcc0 con 0x7f1cb00bc870 2021-04-09T18:34:10.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1ca5ffb700 1 -- 172.21.15.32:0/4230485469 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca0006020 con 0x7f1cb00bc870 2021-04-09T18:34:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb00a7d20 con 0x7f1cb00bc870 2021-04-09T18:34:10.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1ca5ffb700 1 -- 172.21.15.32:0/4230485469 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ca00054a0 con 0x7f1cb00bc870 2021-04-09T18:34:10.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.153 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb00a7d20 con 0x7f1cb00bc870 2021-04-09T18:34:10.186 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.154 7f1ca5ffb700 1 -- 172.21.15.32:0/4230485469 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1ca0006200 con 0x7f1cb00bc870 2021-04-09T18:34:10.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.154 7f1ca5ffb700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c900188d0 0x7f1c9001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.154 7f1ca5ffb700 1 -- 172.21.15.32:0/4230485469 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f1ca0032750 con 0x7f1cb00bc870 2021-04-09T18:34:10.187 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.154 7f1cb57df700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c900188d0 0x7f1c9001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.188 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.154 7f1cb57df700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c900188d0 0x7f1c9001add0 crc :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:10.190 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.174 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb01a4770 con 0x7f1cb00bc870 2021-04-09T18:34:10.194 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.177 7f1ca5ffb700 1 -- 172.21.15.32:0/4230485469 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1ca000bcc0 con 0x7f1cb00bc870 2021-04-09T18:34:10.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.369 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cb0112070 con 0x7f1cb00bc870 2021-04-09T18:34:10.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.370 7f1ca5ffb700 1 -- 172.21.15.32:0/4230485469 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 421 v421) v1 ==== 95+0+1852 (secure 0 0 0) 0x7f1ca0006020 con 0x7f1cb00bc870 2021-04-09T18:34:10.388 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 421 2021-04-09T18:34:10.389 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:10.389 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":421,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/953246178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":953246178},{"type":"v1","addr":"172.21.15.86:6837","nonce":953246178}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":420},{"gid":12292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/359877522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":359877522},{"type":"v1","addr":"172.21.15.86:6836","nonce":359877522}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":421},{"gid":12744,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1046408571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1046408571},{"type":"v1","addr":"172.21.15.32:6833","nonce":1046408571}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":420},{"gid":12783,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3925204342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3925204342},{"type":"v1","addr":"172.21.15.32:6835","nonce":3925204342}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":421}],"filesystems":[]} 2021-04-09T18:34:10.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c900188d0 msgr2=0x7f1c9001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c900188d0 0x7f1c9001add0 crc :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb00bc870 msgr2=0x7f1cb00b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb00bc870 0x7f1cb00b3560 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7f1ca000b530 tx=0x7f1ca000b8d0).stop 2021-04-09T18:34:10.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 shutdown_connections 2021-04-09T18:34:10.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1c900188d0 0x7f1c9001add0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cb00bc870 0x7f1cb00b3560 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cb01bf520 0x7f1cb00b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 --2- 172.21.15.32:0/4230485469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cb00a75d0 0x7f1cb01bef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 >> 172.21.15.32:0/4230485469 conn(0x7f1cb01acbe0 msgr2=0x7f1cb008dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:10.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 shutdown_connections 2021-04-09T18:34:10.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.372 7f1cb79c3700 1 -- 172.21.15.32:0/4230485469 wait complete. 2021-04-09T18:34:10.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.533 7f3c99f6b700 1 Processor -- start 2021-04-09T18:34:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.533 7f3c99f6b700 1 -- start start 2021-04-09T18:34:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c99f6b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c9410d010 0x7f3c940d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c99f6b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c940d1da0 0x7f3c940d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c99f6b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9410d480 con 0x7f3c940d82c0 2021-04-09T18:34:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c99f6b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9410e780 con 0x7f3c940d1da0 2021-04-09T18:34:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c99f6b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c940d8d30 con 0x7f3c9410d010 2021-04-09T18:34:10.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c92ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c940d1da0 0x7f3c940d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c93fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3c940d82c0 0x7f3c9410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40802/0 (socket says 172.21.15.32:40802) 2021-04-09T18:34:10.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c93fff700 1 -- 172.21.15.32:0/4219703540 learned_addr learned my addr 172.21.15.32:0/4219703540 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:10.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c937fe700 1 --2- 172.21.15.32:0/4219703540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c9410d010 0x7f3c940d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c92ffd700 1 -- 172.21.15.32:0/4219703540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c9410d010 msgr2=0x7f3c940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:10.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c92ffd700 1 --2- 172.21.15.32:0/4219703540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c9410d010 0x7f3c940d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c92ffd700 1 -- 172.21.15.32:0/4219703540 >> v1:172.21.15.32:6789/0 conn(0x7f3c940d82c0 legacy=0x7f3c9410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c92ffd700 1 -- 172.21.15.32:0/4219703540 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c940d9220 con 0x7f3c940d1da0 2021-04-09T18:34:10.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c927fc700 1 -- 172.21.15.32:0/4219703540 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351374999 0 0) 0x7f3c8c000d30 con 0x7f3c940d82c0 2021-04-09T18:34:10.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.534 7f3c92ffd700 1 --2- 172.21.15.32:0/4219703540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c940d1da0 0x7f3c940d7ca0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7f3c88006650 tx=0x7f3c88009a00).ready entity=mon.0 client_cookie=70214534f009db95 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:10.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c927fc700 1 -- 172.21.15.32:0/4219703540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c88009df0 con 0x7f3c940d1da0 2021-04-09T18:34:10.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c927fc700 1 -- 172.21.15.32:0/4219703540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c880009b0 con 0x7f3c940d1da0 2021-04-09T18:34:10.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c927fc700 1 -- 172.21.15.32:0/4219703540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c8800c2b0 con 0x7f3c940d1da0 2021-04-09T18:34:10.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c99f6b700 1 -- 172.21.15.32:0/4219703540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c940d1da0 msgr2=0x7f3c940d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c99f6b700 1 --2- 172.21.15.32:0/4219703540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c940d1da0 0x7f3c940d7ca0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7f3c88006650 tx=0x7f3c88009a00).stop 2021-04-09T18:34:10.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c99f6b700 1 -- 172.21.15.32:0/4219703540 shutdown_connections 2021-04-09T18:34:10.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c99f6b700 1 --2- 172.21.15.32:0/4219703540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c940d1da0 0x7f3c940d7ca0 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c99f6b700 1 --2- 172.21.15.32:0/4219703540 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c9410d010 0x7f3c940d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c99f6b700 1 -- 172.21.15.32:0/4219703540 >> 172.21.15.32:0/4219703540 conn(0x7f3c940ce5f0 msgr2=0x7f3c940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:10.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c99f6b700 1 -- 172.21.15.32:0/4219703540 shutdown_connections 2021-04-09T18:34:10.578 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.535 7f3c99f6b700 1 -- 172.21.15.32:0/4219703540 wait complete. 2021-04-09T18:34:10.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.555 7f3c99f6b700 1 Processor -- start 2021-04-09T18:34:10.581 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c99f6b700 1 -- start start 2021-04-09T18:34:10.584 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c99f6b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94179bf0 0x7f3c94179ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c99f6b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c9417c540 0x7f3c9417c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c99f6b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c940aca60 0x7f3c940ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c99f6b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9417cf60 con 0x7f3c9417c540 2021-04-09T18:34:10.585 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c99f6b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c940ad480 con 0x7f3c94179bf0 2021-04-09T18:34:10.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c99f6b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c940ad6a0 con 0x7f3c940aca60 2021-04-09T18:34:10.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c92ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c9417c540 0x7f3c9417c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.586 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c92ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c9417c540 0x7f3c9417c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59448/0 (socket says 172.21.15.32:59448) 2021-04-09T18:34:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c92ffd700 1 -- 172.21.15.32:0/3955575577 learned_addr learned my addr 172.21.15.32:0/3955575577 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.556 7f3c93fff700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c940aca60 0x7f3c940ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c937fe700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94179bf0 0x7f3c94179ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c92ffd700 1 -- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c940aca60 msgr2=0x7f3c940ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.588 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c92ffd700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c940aca60 0x7f3c940ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c92ffd700 1 -- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94179bf0 msgr2=0x7f3c94179ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c92ffd700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94179bf0 0x7f3c94179ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.589 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c92ffd700 1 -- 172.21.15.32:0/3955575577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c940ad8c0 con 0x7f3c9417c540 2021-04-09T18:34:10.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c92ffd700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c9417c540 0x7f3c9417c940 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 rx=0x7f3c880096a0 tx=0x7f3c8800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:10.591 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c7bfff700 1 -- 172.21.15.32:0/3955575577 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c88000c90 con 0x7f3c9417c540 2021-04-09T18:34:10.592 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c7bfff700 1 -- 172.21.15.32:0/3955575577 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c88002030 con 0x7f3c9417c540 2021-04-09T18:34:10.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c7bfff700 1 -- 172.21.15.32:0/3955575577 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c88002550 con 0x7f3c9417c540 2021-04-09T18:34:10.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c940ad8c0 con 0x7f3c9417c540 2021-04-09T18:34:10.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.557 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c940ad8c0 con 0x7f3c9417c540 2021-04-09T18:34:10.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.558 7f3c7bfff700 1 -- 172.21.15.32:0/3955575577 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3c88002210 con 0x7f3c9417c540 2021-04-09T18:34:10.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.558 7f3c7bfff700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c740188d0 0x7f3c7401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:10.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.558 7f3c7bfff700 1 -- 172.21.15.32:0/3955575577 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f3c88032790 con 0x7f3c9417c540 2021-04-09T18:34:10.594 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.558 7f3c937fe700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c740188d0 0x7f3c7401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:10.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.558 7f3c937fe700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c740188d0 0x7f3c7401add0 crc :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:10.595 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.578 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c941b7010 con 0x7f3c9417c540 2021-04-09T18:34:10.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.581 7f3c7bfff700 1 -- 172.21.15.32:0/3955575577 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3c880009c0 con 0x7f3c9417c540 2021-04-09T18:34:10.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.772 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c9410d480 con 0x7f3c9417c540 2021-04-09T18:34:10.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.773 7f3c7bfff700 1 -- 172.21.15.32:0/3955575577 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 421 v421) v1 ==== 95+0+1852 (secure 0 0 0) 0x7f3c8800bd80 con 0x7f3c9417c540 2021-04-09T18:34:10.790 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 421 2021-04-09T18:34:10.790 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:10.791 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":421,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/953246178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":953246178},{"type":"v1","addr":"172.21.15.86:6837","nonce":953246178}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":420},{"gid":12292,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/359877522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":359877522},{"type":"v1","addr":"172.21.15.86:6836","nonce":359877522}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":421},{"gid":12744,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6833/1046408571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1046408571},{"type":"v1","addr":"172.21.15.32:6833","nonce":1046408571}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":420},{"gid":12783,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/3925204342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":3925204342},{"type":"v1","addr":"172.21.15.32:6835","nonce":3925204342}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":421}],"filesystems":[]} 2021-04-09T18:34:10.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c740188d0 msgr2=0x7f3c7401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c740188d0 0x7f3c7401add0 crc :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c9417c540 msgr2=0x7f3c9417c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:10.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c9417c540 0x7f3c9417c940 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 rx=0x7f3c880096a0 tx=0x7f3c8800b990).stop 2021-04-09T18:34:10.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 shutdown_connections 2021-04-09T18:34:10.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c740188d0 0x7f3c7401add0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c9417c540 0x7f3c9417c940 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c940aca60 0x7f3c940ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 --2- 172.21.15.32:0/3955575577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94179bf0 0x7f3c94179ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:10.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.775 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 >> 172.21.15.32:0/3955575577 conn(0x7f3c941acae0 msgr2=0x7f3c9417ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:10.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.776 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 shutdown_connections 2021-04-09T18:34:10.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:10.776 7f3c99f6b700 1 -- 172.21.15.32:0/3955575577 wait complete. 2021-04-09T18:34:10.803 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:34:10.804 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:34:10.805 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:34:10.805 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:10.806 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:34:10.806 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:34:10.807 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:34:10.807 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:10.808 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:34:10.808 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:34:10.808 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:34:10.809 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:10.809 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:34:10.809 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:34:10.810 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:34:10.810 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:10.813 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:34:10.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:34:10.815 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:34:10.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:34:10.816 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:34:10.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:34:10.818 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:34:10.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:34:10.856 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:34:10.856 DEBUG:teuthology.parallel:result is None 2021-04-09T18:34:10.858 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:34:10.858 DEBUG:teuthology.parallel:result is None 2021-04-09T18:34:10.860 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:34:10.861 DEBUG:teuthology.parallel:result is None 2021-04-09T18:34:10.862 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:34:10.863 DEBUG:teuthology.parallel:result is None 2021-04-09T18:34:10.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:34:10.998 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:34:10.999 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:34:10.999 INFO:tasks.ceph.mds.d.smithi086.stderr:2021-04-09 18:34:10.990 7f486e1d1700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2021-04-09T18:34:11.005 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:11.005 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:34:11.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafd638c700 1 Processor -- start 2021-04-09T18:34:11.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafd638c700 1 -- start start 2021-04-09T18:34:11.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafd638c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd00cffb0 0x7fafd00d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafd638c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafd00d7810 0x7fafd00d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafd638c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00d7290 con 0x7fafd00d5c80 2021-04-09T18:34:11.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafd638c700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd00d8590 con 0x7fafd00d7810 2021-04-09T18:34:11.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafd638c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00d91f0 con 0x7fafd00cffb0 2021-04-09T18:34:11.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafc7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafd00d7810 0x7fafd00d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafc7fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fafd00d7810 0x7fafd00d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59480/0 (socket says 172.21.15.32:59480) 2021-04-09T18:34:11.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.000 7fafc7fff700 1 -- 172.21.15.32:0/3136820280 learned_addr learned my addr 172.21.15.32:0/3136820280 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:11.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafcffff700 1 --2- 172.21.15.32:0/3136820280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd00cffb0 0x7fafd00d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafc7fff700 1 -- 172.21.15.32:0/3136820280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd00cffb0 msgr2=0x7fafd00d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:11.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafc7fff700 1 --2- 172.21.15.32:0/3136820280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd00cffb0 0x7fafd00d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafc7fff700 1 -- 172.21.15.32:0/3136820280 >> v1:172.21.15.86:6789/0 conn(0x7fafd00d5c80 legacy=0x7fafd00d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:11.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafc7fff700 1 -- 172.21.15.32:0/3136820280 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00d8230 con 0x7fafd00d7810 2021-04-09T18:34:11.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafc7fff700 1 --2- 172.21.15.32:0/3136820280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00d7810 0x7fafd00d7c10 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7fafb8006700 tx=0x7fafb8009b80).ready entity=mon.0 client_cookie=8d8d6d96c9f84f64 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:11.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafcf7fe700 1 -- 172.21.15.32:0/3136820280 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafb800c0c0 con 0x7fafd00d7810 2021-04-09T18:34:11.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafcf7fe700 1 -- 172.21.15.32:0/3136820280 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb800c440 con 0x7fafd00d7810 2021-04-09T18:34:11.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafcf7fe700 1 -- 172.21.15.32:0/3136820280 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafb800c920 con 0x7fafd00d7810 2021-04-09T18:34:11.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafd638c700 1 -- 172.21.15.32:0/3136820280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00d7810 msgr2=0x7fafd00d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:11.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.001 7fafd638c700 1 --2- 172.21.15.32:0/3136820280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00d7810 0x7fafd00d7c10 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7fafb8006700 tx=0x7fafb8009b80).stop 2021-04-09T18:34:11.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.002 7fafd638c700 1 -- 172.21.15.32:0/3136820280 shutdown_connections 2021-04-09T18:34:11.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.002 7fafd638c700 1 --2- 172.21.15.32:0/3136820280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00d7810 0x7fafd00d7c10 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.002 7fafd638c700 1 --2- 172.21.15.32:0/3136820280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd00cffb0 0x7fafd00d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.002 7fafd638c700 1 -- 172.21.15.32:0/3136820280 >> 172.21.15.32:0/3136820280 conn(0x7fafd00ce5d0 msgr2=0x7fafd00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:11.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.002 7fafd638c700 1 -- 172.21.15.32:0/3136820280 shutdown_connections 2021-04-09T18:34:11.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.002 7fafd638c700 1 -- 172.21.15.32:0/3136820280 wait complete. 2021-04-09T18:34:11.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.022 7fafd638c700 1 Processor -- start 2021-04-09T18:34:11.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.022 7fafd638c700 1 -- start start 2021-04-09T18:34:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafd638c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00a8b80 0x7fafd00a8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafd638c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd01b8610 0x7fafd01b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafd638c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd008d9d0 0x7fafd008ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafd638c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd017d400 con 0x7fafd00a8b80 2021-04-09T18:34:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafd638c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd008e3f0 con 0x7fafd008d9d0 2021-04-09T18:34:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafd638c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd01a0a80 con 0x7fafd01b8610 2021-04-09T18:34:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00a8b80 0x7fafd00a8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00a8b80 0x7fafd00a8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59488/0 (socket says 172.21.15.32:59488) 2021-04-09T18:34:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcffff700 1 -- 172.21.15.32:0/3825229142 learned_addr learned my addr 172.21.15.32:0/3825229142 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafc7fff700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd01b8610 0x7fafd01b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafd49a9700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd008d9d0 0x7fafd008ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcffff700 1 -- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd01b8610 msgr2=0x7fafd01b8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcffff700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd01b8610 0x7fafd01b8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcffff700 1 -- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd008d9d0 msgr2=0x7fafd008ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcffff700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd008d9d0 0x7fafd008ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcffff700 1 -- 172.21.15.32:0/3825229142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd008c630 con 0x7fafd00a8b80 2021-04-09T18:34:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcffff700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00a8b80 0x7fafd00a8f80 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7fafc0006f00 tx=0x7fafc000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.023 7fafcd7fa700 1 -- 172.21.15.32:0/3825229142 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafc000bd30 con 0x7fafd00a8b80 2021-04-09T18:34:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.024 7fafcd7fa700 1 -- 172.21.15.32:0/3825229142 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc0002030 con 0x7fafd00a8b80 2021-04-09T18:34:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.024 7fafcd7fa700 1 -- 172.21.15.32:0/3825229142 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafc0002590 con 0x7fafd00a8b80 2021-04-09T18:34:11.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.024 7fafd638c700 1 -- 172.21.15.32:0/3825229142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd008c630 con 0x7fafd00a8b80 2021-04-09T18:34:11.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.024 7fafd638c700 1 -- 172.21.15.32:0/3825229142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd008c630 con 0x7fafd00a8b80 2021-04-09T18:34:11.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.024 7fafcd7fa700 1 -- 172.21.15.32:0/3825229142 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fafc0000c90 con 0x7fafd00a8b80 2021-04-09T18:34:11.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.024 7fafcd7fa700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fafb40188d0 0x7fafb401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.024 7fafcd7fa700 1 -- 172.21.15.32:0/3825229142 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fafc0032760 con 0x7fafd00a8b80 2021-04-09T18:34:11.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.024 7fafc7fff700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fafb40188d0 0x7fafb401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.025 7fafc7fff700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fafb40188d0 0x7fafb401add0 crc :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:11.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.044 7fafd638c700 1 -- 172.21.15.32:0/3825229142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd008e3f0 con 0x7fafd00a8b80 2021-04-09T18:34:11.065 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.048 7fafcd7fa700 1 -- 172.21.15.32:0/3825229142 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fafc001f780 con 0x7fafd00a8b80 2021-04-09T18:34:11.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.252 7fafd638c700 1 -- 172.21.15.32:0/3825229142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fafd008e3f0 con 0x7fafd00a8b80 2021-04-09T18:34:11.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.650 7fafcd7fa700 1 -- 172.21.15.32:0/3825229142 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v255) v1 ==== 97+0+0 (secure 0 0 0) 0x7fafc0000c90 con 0x7fafd00a8b80 2021-04-09T18:34:11.682 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:34:11.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 -- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fafb40188d0 msgr2=0x7fafb401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:11.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fafb40188d0 0x7fafb401add0 crc :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 -- 172.21.15.32:0/3825229142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00a8b80 msgr2=0x7fafd00a8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:11.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00a8b80 0x7fafd00a8f80 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7fafc0006f00 tx=0x7fafc000b940).stop 2021-04-09T18:34:11.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 -- 172.21.15.32:0/3825229142 shutdown_connections 2021-04-09T18:34:11.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fafb40188d0 0x7fafb401add0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd008d9d0 0x7fafd008ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd01b8610 0x7fafd01b8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 --2- 172.21.15.32:0/3825229142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafd00a8b80 0x7fafd00a8f80 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 -- 172.21.15.32:0/3825229142 >> 172.21.15.32:0/3825229142 conn(0x7fafd01905e0 msgr2=0x7fafd01b7350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:11.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 -- 172.21.15.32:0/3825229142 shutdown_connections 2021-04-09T18:34:11.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.668 7fafd638c700 1 -- 172.21.15.32:0/3825229142 wait complete. 2021-04-09T18:34:11.696 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:34:11.697 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:34:11.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.829 7f4246065700 1 Processor -- start 2021-04-09T18:34:11.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.829 7f4246065700 1 -- start start 2021-04-09T18:34:11.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.829 7f4246065700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f424010d010 0x7f42400d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.829 7f4246065700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f42400d1da0 0x7f42400d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f4246065700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424010d480 con 0x7f42400d82c0 2021-04-09T18:34:11.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f4246065700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42400d8d30 con 0x7f424010d010 2021-04-09T18:34:11.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f4246065700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42400d9220 con 0x7f42400d1da0 2021-04-09T18:34:11.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423f7fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f424010d010 0x7f42400d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423ffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f42400d82c0 0x7f424010da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40854/0 (socket says 172.21.15.32:40854) 2021-04-09T18:34:11.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423ffff700 1 -- 172.21.15.32:0/1969868936 learned_addr learned my addr 172.21.15.32:0/1969868936 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:11.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423effd700 1 --2- 172.21.15.32:0/1969868936 >> v2:172.21.15.86:3300/0 conn(0x7f42400d1da0 0x7f42400d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423f7fe700 1 -- 172.21.15.32:0/1969868936 >> v2:172.21.15.86:3300/0 conn(0x7f42400d1da0 msgr2=0x7f42400d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:11.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423f7fe700 1 --2- 172.21.15.32:0/1969868936 >> v2:172.21.15.86:3300/0 conn(0x7f42400d1da0 0x7f42400d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423f7fe700 1 -- 172.21.15.32:0/1969868936 >> v1:172.21.15.32:6789/0 conn(0x7f42400d82c0 legacy=0x7f424010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:11.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423f7fe700 1 -- 172.21.15.32:0/1969868936 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400d86f0 con 0x7f424010d010 2021-04-09T18:34:11.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423e7fc700 1 -- 172.21.15.32:0/1969868936 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312999776 0 0) 0x7f4238000d30 con 0x7f42400d82c0 2021-04-09T18:34:11.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423effd700 1 --2- 172.21.15.32:0/1969868936 >> v2:172.21.15.86:3300/0 conn(0x7f42400d1da0 0x7f42400d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:11.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423f7fe700 1 --2- 172.21.15.32:0/1969868936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f424010d010 0x7f42400d1780 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 rx=0x7f4230006650 tx=0x7f4230009a80).ready entity=mon.0 client_cookie=b4c34f1b31f2b70f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:11.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423e7fc700 1 -- 172.21.15.32:0/1969868936 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42300009b0 con 0x7f424010d010 2021-04-09T18:34:11.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423e7fc700 1 -- 172.21.15.32:0/1969868936 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423000c0c0 con 0x7f424010d010 2021-04-09T18:34:11.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f423e7fc700 1 -- 172.21.15.32:0/1969868936 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f423000c510 con 0x7f424010d010 2021-04-09T18:34:11.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f4246065700 1 -- 172.21.15.32:0/1969868936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f424010d010 msgr2=0x7f42400d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:11.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.830 7f4246065700 1 --2- 172.21.15.32:0/1969868936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f424010d010 0x7f42400d1780 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 rx=0x7f4230006650 tx=0x7f4230009a80).stop 2021-04-09T18:34:11.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.831 7f4246065700 1 -- 172.21.15.32:0/1969868936 shutdown_connections 2021-04-09T18:34:11.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.831 7f4246065700 1 --2- 172.21.15.32:0/1969868936 >> v2:172.21.15.86:3300/0 conn(0x7f42400d1da0 0x7f42400d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.831 7f4246065700 1 --2- 172.21.15.32:0/1969868936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f424010d010 0x7f42400d1780 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.831 7f4246065700 1 -- 172.21.15.32:0/1969868936 >> 172.21.15.32:0/1969868936 conn(0x7f42400ce5f0 msgr2=0x7f42400d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:11.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.831 7f4246065700 1 -- 172.21.15.32:0/1969868936 shutdown_connections 2021-04-09T18:34:11.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.831 7f4246065700 1 -- 172.21.15.32:0/1969868936 wait complete. 2021-04-09T18:34:11.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.851 7f4246065700 1 Processor -- start 2021-04-09T18:34:11.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.851 7f4246065700 1 -- start start 2021-04-09T18:34:11.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f4246065700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f424009a5e0 0x7f42401884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f4246065700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42400a3350 0x7f42400a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f4246065700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400a15b0 0x7f42400a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f4246065700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42400a40d0 con 0x7f42400a15b0 2021-04-09T18:34:11.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f4246065700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42400a2dd0 con 0x7f424009a5e0 2021-04-09T18:34:11.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f4246065700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4240188ad0 con 0x7f42400a3350 2021-04-09T18:34:11.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400a15b0 0x7f42400a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400a15b0 0x7f42400a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59504/0 (socket says 172.21.15.32:59504) 2021-04-09T18:34:11.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423ffff700 1 -- 172.21.15.32:0/1148120683 learned_addr learned my addr 172.21.15.32:0/1148120683 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:11.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423f7fe700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f424009a5e0 0x7f42401884b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423effd700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42400a3350 0x7f42400a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423ffff700 1 -- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42400a3350 msgr2=0x7f42400a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:11.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423ffff700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42400a3350 0x7f42400a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423ffff700 1 -- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f424009a5e0 msgr2=0x7f42401884b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:11.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423ffff700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f424009a5e0 0x7f42401884b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:11.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423ffff700 1 -- 172.21.15.32:0/1148120683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400a3d70 con 0x7f42400a15b0 2021-04-09T18:34:11.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.852 7f423f7fe700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f424009a5e0 0x7f42401884b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:11.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.853 7f423ffff700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400a15b0 0x7f42400a19b0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7f4238001cf0 tx=0x7f4238002090).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:11.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.853 7f4227fff700 1 -- 172.21.15.32:0/1148120683 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4238000cc0 con 0x7f42400a15b0 2021-04-09T18:34:11.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.853 7f4227fff700 1 -- 172.21.15.32:0/1148120683 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42380070d0 con 0x7f42400a15b0 2021-04-09T18:34:11.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.853 7f4227fff700 1 -- 172.21.15.32:0/1148120683 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4238004720 con 0x7f42400a15b0 2021-04-09T18:34:11.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.853 7f4246065700 1 -- 172.21.15.32:0/1148120683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42400a3d70 con 0x7f42400a15b0 2021-04-09T18:34:11.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.853 7f4246065700 1 -- 172.21.15.32:0/1148120683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42400a3d70 con 0x7f42400a15b0 2021-04-09T18:34:11.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.853 7f4227fff700 1 -- 172.21.15.32:0/1148120683 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f42380072b0 con 0x7f42400a15b0 2021-04-09T18:34:11.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.853 7f4227fff700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42200188d0 0x7f422001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:11.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.854 7f4227fff700 1 -- 172.21.15.32:0/1148120683 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f4238034c40 con 0x7f42400a15b0 2021-04-09T18:34:11.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.854 7f423f7fe700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42200188d0 0x7f422001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:11.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.854 7f423f7fe700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42200188d0 0x7f422001add0 crc :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:11.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.874 7f4246065700 1 -- 172.21.15.32:0/1148120683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424010d480 con 0x7f42400a15b0 2021-04-09T18:34:11.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:11.877 7f4227fff700 1 -- 172.21.15.32:0/1148120683 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4238007d00 con 0x7f42400a15b0 2021-04-09T18:34:12.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.063 7f4246065700 1 -- 172.21.15.32:0/1148120683 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f424010d480 con 0x7f42400a15b0 2021-04-09T18:34:12.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.063 7f4227fff700 1 -- 172.21.15.32:0/1148120683 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v48) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f42380070d0 con 0x7f42400a15b0 2021-04-09T18:34:12.081 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:12.081 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:34:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.082 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:34:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:34:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.083 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:34:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.084 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:34:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:34:12.085 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:34:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.087 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:34:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:34:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:34:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:34:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:34:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:34:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:34:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:34:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.092 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:34:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:34:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.094 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:34:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:34:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.096 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.097 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.097 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.097 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.099 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:34:12.099 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:34:12.099 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.099 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.099 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.100 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.100 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.101 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:34:12.101 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:34:12.101 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.101 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.102 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.102 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.102 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.103 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:34:12.103 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:34:12.104 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.104 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.104 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.104 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.105 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.105 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:34:12.105 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:34:12.106 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.106 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.106 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.106 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.106 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.107 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.107 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:34:12.107 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:34:12.108 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.108 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.108 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.109 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.109 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.110 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:34:12.110 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:34:12.110 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.110 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:34:12.111 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:34:12.111 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:34:12.111 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.111 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.112 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:34:12.112 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:34:12.112 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.112 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:34:12.113 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:34:12.113 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:34:12.113 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.114 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:34:12.114 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:34:12.114 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.114 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:34:12.114 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:34:12.115 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:34:12.115 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.115 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.116 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.116 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:34:12.116 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:34:12.116 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.116 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:34:12.117 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:34:12.117 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.117 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:12.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:12.118 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:34:12.118 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:34:12.119 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:34:12.119 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:34:12.119 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:34:12.120 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:34:12.120 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.120 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:12.121 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:34:12.121 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.065 7f4246065700 1 -- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42200188d0 msgr2=0x7f422001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.065 7f4246065700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42200188d0 0x7f422001add0 crc :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.065 7f4246065700 1 -- 172.21.15.32:0/1148120683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400a15b0 msgr2=0x7f42400a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.065 7f4246065700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400a15b0 0x7f42400a19b0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7f4238001cf0 tx=0x7f4238002090).stop 2021-04-09T18:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.066 7f4246065700 1 -- 172.21.15.32:0/1148120683 shutdown_connections 2021-04-09T18:34:12.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.066 7f4246065700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f42200188d0 0x7f422001add0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.066 7f4246065700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42400a15b0 0x7f42400a19b0 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.066 7f4246065700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42400a3350 0x7f42400a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.066 7f4246065700 1 --2- 172.21.15.32:0/1148120683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f424009a5e0 0x7f42401884b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.066 7f4246065700 1 -- 172.21.15.32:0/1148120683 >> 172.21.15.32:0/1148120683 conn(0x7f42401a0db0 msgr2=0x7f42401a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:12.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.066 7f4246065700 1 -- 172.21.15.32:0/1148120683 shutdown_connections 2021-04-09T18:34:12.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.066 7f4246065700 1 -- 172.21.15.32:0/1148120683 wait complete. 2021-04-09T18:34:12.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:34:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.259 7f1f68f70700 1 Processor -- start 2021-04-09T18:34:12.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.259 7f1f68f70700 1 -- start start 2021-04-09T18:34:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.259 7f1f68f70700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1f640d7ca0 0x7f1f640d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.259 7f1f68f70700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f6410da00 0x7f1f6410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.259 7f1f68f70700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6410d480 con 0x7f1f6410d010 2021-04-09T18:34:12.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.259 7f1f68f70700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f640d9220 con 0x7f1f640d7ca0 2021-04-09T18:34:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.259 7f1f68f70700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640d86c0 con 0x7f1f6410da00 2021-04-09T18:34:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f62312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1f640d7ca0 0x7f1f640d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f62312700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1f640d7ca0 0x7f1f640d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59512/0 (socket says 172.21.15.32:59512) 2021-04-09T18:34:12.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f62312700 1 -- 172.21.15.32:0/1452799210 learned_addr learned my addr 172.21.15.32:0/1452799210 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f63314700 1 --2- 172.21.15.32:0/1452799210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f6410da00 0x7f1f6410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f62312700 1 -- 172.21.15.32:0/1452799210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f6410da00 msgr2=0x7f1f6410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f62312700 1 --2- 172.21.15.32:0/1452799210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f6410da00 0x7f1f6410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f62312700 1 -- 172.21.15.32:0/1452799210 >> v1:172.21.15.86:6789/0 conn(0x7f1f6410d010 legacy=0x7f1f640d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f62312700 1 -- 172.21.15.32:0/1452799210 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f640d1f70 con 0x7f1f640d7ca0 2021-04-09T18:34:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f62312700 1 --2- 172.21.15.32:0/1452799210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f640d7ca0 0x7f1f640d80a0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7f1f58006700 tx=0x7f1f58009b80).ready entity=mon.0 client_cookie=2c58347a745a8bb2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f61b11700 1 -- 172.21.15.32:0/1452799210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f5800c0c0 con 0x7f1f640d7ca0 2021-04-09T18:34:12.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f61b11700 1 -- 172.21.15.32:0/1452799210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5800c440 con 0x7f1f640d7ca0 2021-04-09T18:34:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f61b11700 1 -- 172.21.15.32:0/1452799210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f5800c920 con 0x7f1f640d7ca0 2021-04-09T18:34:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f68f70700 1 -- 172.21.15.32:0/1452799210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f640d7ca0 msgr2=0x7f1f640d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.260 7f1f68f70700 1 --2- 172.21.15.32:0/1452799210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f640d7ca0 0x7f1f640d80a0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7f1f58006700 tx=0x7f1f58009b80).stop 2021-04-09T18:34:12.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.261 7f1f68f70700 1 -- 172.21.15.32:0/1452799210 shutdown_connections 2021-04-09T18:34:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.261 7f1f68f70700 1 --2- 172.21.15.32:0/1452799210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f6410da00 0x7f1f6410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.261 7f1f68f70700 1 --2- 172.21.15.32:0/1452799210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f640d7ca0 0x7f1f640d80a0 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.261 7f1f68f70700 1 -- 172.21.15.32:0/1452799210 >> 172.21.15.32:0/1452799210 conn(0x7f1f640ce5f0 msgr2=0x7f1f640d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.261 7f1f68f70700 1 -- 172.21.15.32:0/1452799210 shutdown_connections 2021-04-09T18:34:12.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.261 7f1f68f70700 1 -- 172.21.15.32:0/1452799210 wait complete. 2021-04-09T18:34:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.281 7f1f68f70700 1 Processor -- start 2021-04-09T18:34:12.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.281 7f1f68f70700 1 -- start start 2021-04-09T18:34:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f68f70700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f64191920 0x7f1f6419d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f68f70700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f641ad220 0x7f1f6419dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f68f70700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f64191d20 0x7f1f641a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f68f70700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6419d0f0 con 0x7f1f641ad220 2021-04-09T18:34:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f68f70700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6419cce0 con 0x7f1f64191920 2021-04-09T18:34:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f68f70700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6419cec0 con 0x7f1f64191d20 2021-04-09T18:34:12.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f62312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f641ad220 0x7f1f6419dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f62312700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f641ad220 0x7f1f6419dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59516/0 (socket says 172.21.15.32:59516) 2021-04-09T18:34:12.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f62312700 1 -- 172.21.15.32:0/1766648697 learned_addr learned my addr 172.21.15.32:0/1766648697 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:12.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f62b13700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f64191920 0x7f1f6419d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f63314700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f64191d20 0x7f1f641a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f62312700 1 -- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f64191d20 msgr2=0x7f1f641a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f62312700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f64191d20 0x7f1f641a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f62312700 1 -- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f64191920 msgr2=0x7f1f6419d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f62312700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f64191920 0x7f1f6419d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.282 7f1f62312700 1 -- 172.21.15.32:0/1766648697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f641aa5c0 con 0x7f1f641ad220 2021-04-09T18:34:12.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.283 7f1f62312700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f641ad220 0x7f1f6419dc90 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7f1f5800b580 tx=0x7f1f5800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:12.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.283 7f1f537fe700 1 -- 172.21.15.32:0/1766648697 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f5800bd10 con 0x7f1f641ad220 2021-04-09T18:34:12.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.283 7f1f537fe700 1 -- 172.21.15.32:0/1766648697 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58000c90 con 0x7f1f641ad220 2021-04-09T18:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.283 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f641aa5c0 con 0x7f1f641ad220 2021-04-09T18:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.283 7f1f537fe700 1 -- 172.21.15.32:0/1766648697 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f58002220 con 0x7f1f641ad220 2021-04-09T18:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.283 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f641aa5c0 con 0x7f1f641ad220 2021-04-09T18:34:12.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.284 7f1f537fe700 1 -- 172.21.15.32:0/1766648697 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1f58009f90 con 0x7f1f641ad220 2021-04-09T18:34:12.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.284 7f1f537fe700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f480188c0 0x7f1f4801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.284 7f1f537fe700 1 -- 172.21.15.32:0/1766648697 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f1f58032470 con 0x7f1f641ad220 2021-04-09T18:34:12.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.284 7f1f62b13700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f480188c0 0x7f1f4801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.284 7f1f62b13700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f480188c0 0x7f1f4801adc0 crc :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:12.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.304 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f64182d50 con 0x7f1f641ad220 2021-04-09T18:34:12.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.307 7f1f537fe700 1 -- 172.21.15.32:0/1766648697 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1f580091e0 con 0x7f1f641ad220 2021-04-09T18:34:12.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.497 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f6410d480 con 0x7f1f641ad220 2021-04-09T18:34:12.514 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.498 7f1f537fe700 1 -- 172.21.15.32:0/1766648697 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f1f58009f90 con 0x7f1f641ad220 2021-04-09T18:34:12.515 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:12.515 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":255,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:34:11.646596","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":247,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:34:12.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f480188c0 msgr2=0x7f1f4801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.517 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f480188c0 0x7f1f4801adc0 crc :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f641ad220 msgr2=0x7f1f6419dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f641ad220 0x7f1f6419dc90 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7f1f5800b580 tx=0x7f1f5800b920).stop 2021-04-09T18:34:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 shutdown_connections 2021-04-09T18:34:12.518 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1f480188c0 0x7f1f4801adc0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f641ad220 0x7f1f6419dc90 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f64191d20 0x7f1f641a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 --2- 172.21.15.32:0/1766648697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f64191920 0x7f1f6419d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.519 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 >> 172.21.15.32:0/1766648697 conn(0x7f1f641837d0 msgr2=0x7f1f64195430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 shutdown_connections 2021-04-09T18:34:12.520 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.500 7f1f68f70700 1 -- 172.21.15.32:0/1766648697 wait complete. 2021-04-09T18:34:12.527 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:34:12.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:34:12.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.664 7fd93ae55700 1 Processor -- start 2021-04-09T18:34:12.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.664 7fd93ae55700 1 -- start start 2021-04-09T18:34:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd93ae55700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9340d7ca0 0x7fd9340d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd93ae55700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93410da00 0x7fd93410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd93ae55700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93410d480 con 0x7fd93410d010 2021-04-09T18:34:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd93ae55700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9340d9220 con 0x7fd9340d7ca0 2021-04-09T18:34:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd93ae55700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9340d86c0 con 0x7fd93410da00 2021-04-09T18:34:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd933fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd9340d7ca0 0x7fd9340d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd938c71700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd93410d010 0x7fd9340d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40880/0 (socket says 172.21.15.32:40880) 2021-04-09T18:34:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd938c71700 1 -- 172.21.15.32:0/2982796370 learned_addr learned my addr 172.21.15.32:0/2982796370 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:12.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd939472700 1 --2- 172.21.15.32:0/2982796370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93410da00 0x7fd93410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd933fff700 1 -- 172.21.15.32:0/2982796370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93410da00 msgr2=0x7fd93410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:12.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd933fff700 1 --2- 172.21.15.32:0/2982796370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93410da00 0x7fd93410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd933fff700 1 -- 172.21.15.32:0/2982796370 >> v1:172.21.15.32:6789/0 conn(0x7fd93410d010 legacy=0x7fd9340d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd933fff700 1 -- 172.21.15.32:0/2982796370 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9340d1f70 con 0x7fd9340d7ca0 2021-04-09T18:34:12.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.665 7fd933fff700 1 --2- 172.21.15.32:0/2982796370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9340d7ca0 0x7fd9340d80a0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 rx=0x7fd91c006650 tx=0x7fd91c009a80).ready entity=mon.0 client_cookie=389122731e95cd36 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:12.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd9337fe700 1 -- 172.21.15.32:0/2982796370 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd91c0009b0 con 0x7fd9340d7ca0 2021-04-09T18:34:12.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd9337fe700 1 -- 172.21.15.32:0/2982796370 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91c00c0c0 con 0x7fd9340d7ca0 2021-04-09T18:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd9337fe700 1 -- 172.21.15.32:0/2982796370 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd91c00c510 con 0x7fd9340d7ca0 2021-04-09T18:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd93ae55700 1 -- 172.21.15.32:0/2982796370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9340d7ca0 msgr2=0x7fd9340d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd93ae55700 1 --2- 172.21.15.32:0/2982796370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9340d7ca0 0x7fd9340d80a0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 rx=0x7fd91c006650 tx=0x7fd91c009a80).stop 2021-04-09T18:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd93ae55700 1 -- 172.21.15.32:0/2982796370 shutdown_connections 2021-04-09T18:34:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd93ae55700 1 --2- 172.21.15.32:0/2982796370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93410da00 0x7fd93410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd93ae55700 1 --2- 172.21.15.32:0/2982796370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9340d7ca0 0x7fd9340d80a0 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd93ae55700 1 -- 172.21.15.32:0/2982796370 >> 172.21.15.32:0/2982796370 conn(0x7fd9340ce5f0 msgr2=0x7fd9340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.666 7fd93ae55700 1 -- 172.21.15.32:0/2982796370 shutdown_connections 2021-04-09T18:34:12.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.667 7fd93ae55700 1 -- 172.21.15.32:0/2982796370 wait complete. 2021-04-09T18:34:12.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.687 7fd93ae55700 1 Processor -- start 2021-04-09T18:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.687 7fd93ae55700 1 -- start start 2021-04-09T18:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd93ae55700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93409e530 0x7fd93409b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd93ae55700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd93409e930 0x7fd93409f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd93ae55700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd93409f990 0x7fd934094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd93ae55700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93409edf0 con 0x7fd93409e930 2021-04-09T18:34:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd93ae55700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93409fd90 con 0x7fd93409f990 2021-04-09T18:34:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd93ae55700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93409bd40 con 0x7fd93409e530 2021-04-09T18:34:12.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd933fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd93409e930 0x7fd93409f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd933fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd93409e930 0x7fd93409f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59528/0 (socket says 172.21.15.32:59528) 2021-04-09T18:34:12.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd933fff700 1 -- 172.21.15.32:0/324662957 learned_addr learned my addr 172.21.15.32:0/324662957 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:12.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd939472700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd93409f990 0x7fd934094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd938c71700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93409e530 0x7fd93409b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd933fff700 1 -- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93409e530 msgr2=0x7fd93409b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd933fff700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93409e530 0x7fd93409b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd933fff700 1 -- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd93409f990 msgr2=0x7fd934094c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd933fff700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd93409f990 0x7fd934094c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd933fff700 1 -- 172.21.15.32:0/324662957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd934095290 con 0x7fd93409e930 2021-04-09T18:34:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd933fff700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd93409e930 0x7fd93409f370 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7fd91c009650 tx=0x7fd91c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd9317fa700 1 -- 172.21.15.32:0/324662957 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd91c0093a0 con 0x7fd93409e930 2021-04-09T18:34:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd9317fa700 1 -- 172.21.15.32:0/324662957 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91c00bd50 con 0x7fd93409e930 2021-04-09T18:34:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd93ae55700 1 -- 172.21.15.32:0/324662957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd934095290 con 0x7fd93409e930 2021-04-09T18:34:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd9317fa700 1 -- 172.21.15.32:0/324662957 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd91c0052e0 con 0x7fd93409e930 2021-04-09T18:34:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.688 7fd93ae55700 1 -- 172.21.15.32:0/324662957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd934095290 con 0x7fd93409e930 2021-04-09T18:34:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.689 7fd9317fa700 1 -- 172.21.15.32:0/324662957 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd91c006030 con 0x7fd93409e930 2021-04-09T18:34:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.689 7fd9317fa700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd918018930 0x7fd91801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:12.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.689 7fd9317fa700 1 -- 172.21.15.32:0/324662957 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fd91c0326b0 con 0x7fd93409e930 2021-04-09T18:34:12.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.689 7fd938c71700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd918018930 0x7fd91801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:12.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.689 7fd938c71700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd918018930 0x7fd91801ae10 crc :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:12.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.709 7fd93ae55700 1 -- 172.21.15.32:0/324662957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd934095b10 con 0x7fd93409e930 2021-04-09T18:34:12.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.712 7fd9317fa700 1 -- 172.21.15.32:0/324662957 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd91c0009c0 con 0x7fd93409e930 2021-04-09T18:34:12.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.892 7fd93ae55700 1 -- 172.21.15.32:0/324662957 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd934095b10 con 0x7fd93409e930 2021-04-09T18:34:12.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.893 7fd9317fa700 1 -- 172.21.15.32:0/324662957 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v255) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd91c00bd50 con 0x7fd93409e930 2021-04-09T18:34:12.911 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:34:12.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 -- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd918018930 msgr2=0x7fd91801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd918018930 0x7fd91801ae10 crc :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 -- 172.21.15.32:0/324662957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd93409e930 msgr2=0x7fd93409f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd93409e930 0x7fd93409f370 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7fd91c009650 tx=0x7fd91c00b960).stop 2021-04-09T18:34:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 -- 172.21.15.32:0/324662957 shutdown_connections 2021-04-09T18:34:12.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd918018930 0x7fd91801ae10 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd93409e930 0x7fd93409f370 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd93409f990 0x7fd934094c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 --2- 172.21.15.32:0/324662957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd93409e530 0x7fd93409b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 -- 172.21.15.32:0/324662957 >> 172.21.15.32:0/324662957 conn(0x7fd9340c4290 msgr2=0x7fd9340c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:12.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 -- 172.21.15.32:0/324662957 shutdown_connections 2021-04-09T18:34:12.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:12.895 7fd93ae55700 1 -- 172.21.15.32:0/324662957 wait complete. 2021-04-09T18:34:12.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:34:13.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.056 7f3c49c6a700 1 Processor -- start 2021-04-09T18:34:13.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.056 7f3c49c6a700 1 -- start start 2021-04-09T18:34:13.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.056 7f3c49c6a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3c440d7ca0 0x7f3c440d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.056 7f3c49c6a700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c4410da00 0x7f3c4410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.056 7f3c49c6a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4410d480 con 0x7f3c4410d010 2021-04-09T18:34:13.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.056 7f3c49c6a700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c440d9220 con 0x7f3c4410da00 2021-04-09T18:34:13.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.056 7f3c49c6a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c440d86c0 con 0x7f3c440d7ca0 2021-04-09T18:34:13.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c43fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c4410da00 0x7f3c4410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c43fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c4410da00 0x7f3c4410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59538/0 (socket says 172.21.15.32:59538) 2021-04-09T18:34:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c43fff700 1 -- 172.21.15.32:0/3676265446 learned_addr learned my addr 172.21.15.32:0/3676265446 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c42ffd700 1 --2- 172.21.15.32:0/3676265446 >> v2:172.21.15.86:3300/0 conn(0x7f3c440d7ca0 0x7f3c440d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c43fff700 1 -- 172.21.15.32:0/3676265446 >> v2:172.21.15.86:3300/0 conn(0x7f3c440d7ca0 msgr2=0x7f3c440d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:13.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c43fff700 1 --2- 172.21.15.32:0/3676265446 >> v2:172.21.15.86:3300/0 conn(0x7f3c440d7ca0 0x7f3c440d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c43fff700 1 -- 172.21.15.32:0/3676265446 >> v1:172.21.15.86:6789/0 conn(0x7f3c4410d010 legacy=0x7f3c440d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:13.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c43fff700 1 -- 172.21.15.32:0/3676265446 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c440d1f70 con 0x7f3c4410da00 2021-04-09T18:34:13.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c43fff700 1 --2- 172.21.15.32:0/3676265446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4410da00 0x7f3c4410de00 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f3c3c006f00 tx=0x7f3c3c009f10).ready entity=mon.0 client_cookie=d096cd6176b3a922 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:13.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c427fc700 1 -- 172.21.15.32:0/3676265446 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c3c00b3b0 con 0x7f3c4410da00 2021-04-09T18:34:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c427fc700 1 -- 172.21.15.32:0/3676265446 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c00b730 con 0x7f3c4410da00 2021-04-09T18:34:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c427fc700 1 -- 172.21.15.32:0/3676265446 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c3c00bc10 con 0x7f3c4410da00 2021-04-09T18:34:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c49c6a700 1 -- 172.21.15.32:0/3676265446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4410da00 msgr2=0x7f3c4410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.057 7f3c49c6a700 1 --2- 172.21.15.32:0/3676265446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4410da00 0x7f3c4410de00 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f3c3c006f00 tx=0x7f3c3c009f10).stop 2021-04-09T18:34:13.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.058 7f3c49c6a700 1 -- 172.21.15.32:0/3676265446 shutdown_connections 2021-04-09T18:34:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.058 7f3c49c6a700 1 --2- 172.21.15.32:0/3676265446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4410da00 0x7f3c4410de00 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.058 7f3c49c6a700 1 --2- 172.21.15.32:0/3676265446 >> v2:172.21.15.86:3300/0 conn(0x7f3c440d7ca0 0x7f3c440d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.058 7f3c49c6a700 1 -- 172.21.15.32:0/3676265446 >> 172.21.15.32:0/3676265446 conn(0x7f3c440ce5f0 msgr2=0x7f3c440d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:13.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.058 7f3c49c6a700 1 -- 172.21.15.32:0/3676265446 shutdown_connections 2021-04-09T18:34:13.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.058 7f3c49c6a700 1 -- 172.21.15.32:0/3676265446 wait complete. 2021-04-09T18:34:13.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.078 7f3c49c6a700 1 Processor -- start 2021-04-09T18:34:13.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.078 7f3c49c6a700 1 -- start start 2021-04-09T18:34:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c49c6a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c441aca20 0x7f3c441ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c49c6a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4418bd40 0x7f3c4418c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c49c6a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c4408fba0 0x7f3c4408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c49c6a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4418c7e0 con 0x7f3c4418bd40 2021-04-09T18:34:13.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c49c6a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4418ca30 con 0x7f3c4408fba0 2021-04-09T18:34:13.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c49c6a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440905f0 con 0x7f3c441aca20 2021-04-09T18:34:13.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c42ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4418bd40 0x7f3c4418c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c42ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4418bd40 0x7f3c4418c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59544/0 (socket says 172.21.15.32:59544) 2021-04-09T18:34:13.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c42ffd700 1 -- 172.21.15.32:0/1620179669 learned_addr learned my addr 172.21.15.32:0/1620179669 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c43fff700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c4408fba0 0x7f3c4408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c437fe700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c441aca20 0x7f3c441ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c42ffd700 1 -- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c441aca20 msgr2=0x7f3c441ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:13.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c42ffd700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c441aca20 0x7f3c441ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c42ffd700 1 -- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c4408fba0 msgr2=0x7f3c4408ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:13.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c42ffd700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c4408fba0 0x7f3c4408ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.079 7f3c42ffd700 1 -- 172.21.15.32:0/1620179669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c440b4300 con 0x7f3c4418bd40 2021-04-09T18:34:13.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.080 7f3c42ffd700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4418bd40 0x7f3c4418c140 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7f3c38006f60 tx=0x7f3c3800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.080 7f3c2bfff700 1 -- 172.21.15.32:0/1620179669 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c3800bd60 con 0x7f3c4418bd40 2021-04-09T18:34:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.080 7f3c2bfff700 1 -- 172.21.15.32:0/1620179669 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c38000c90 con 0x7f3c4418bd40 2021-04-09T18:34:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.080 7f3c2bfff700 1 -- 172.21.15.32:0/1620179669 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c38002220 con 0x7f3c4418bd40 2021-04-09T18:34:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.080 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c440b4300 con 0x7f3c4418bd40 2021-04-09T18:34:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.080 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c440b4300 con 0x7f3c4418bd40 2021-04-09T18:34:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.081 7f3c2bfff700 1 -- 172.21.15.32:0/1620179669 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3c380052e0 con 0x7f3c4418bd40 2021-04-09T18:34:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.081 7f3c2bfff700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c240188c0 0x7f3c2401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.081 7f3c2bfff700 1 -- 172.21.15.32:0/1620179669 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f3c380326c0 con 0x7f3c4418bd40 2021-04-09T18:34:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.081 7f3c437fe700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c240188c0 0x7f3c2401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.081 7f3c437fe700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c240188c0 0x7f3c2401adc0 crc :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:13.117 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.100 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c44182d70 con 0x7f3c4418bd40 2021-04-09T18:34:13.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.104 7f3c2bfff700 1 -- 172.21.15.32:0/1620179669 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3c38000c90 con 0x7f3c4418bd40 2021-04-09T18:34:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.280 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3c4418c760 con 0x7f3c4418bd40 2021-04-09T18:34:13.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.281 7f3c2bfff700 1 -- 172.21.15.32:0/1620179669 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v255) v1 ==== 84+0+64 (secure 0 0 0) 0x7f3c38006b00 con 0x7f3c4418bd40 2021-04-09T18:34:13.298 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:13.298 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:34:13.299 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:34:13.299 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:34:13.299 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:34:13.299 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:13.299 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c240188c0 msgr2=0x7f3c2401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c240188c0 0x7f3c2401adc0 crc :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4418bd40 msgr2=0x7f3c4418c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4418bd40 0x7f3c4418c140 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7f3c38006f60 tx=0x7f3c3800b970).stop 2021-04-09T18:34:13.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 shutdown_connections 2021-04-09T18:34:13.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c240188c0 0x7f3c2401adc0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c4418bd40 0x7f3c4418c140 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c4408fba0 0x7f3c4408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 --2- 172.21.15.32:0/1620179669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c441aca20 0x7f3c441ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 >> 172.21.15.32:0/1620179669 conn(0x7f3c4419b460 msgr2=0x7f3c4419bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:13.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 shutdown_connections 2021-04-09T18:34:13.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.283 7f3c49c6a700 1 -- 172.21.15.32:0/1620179669 wait complete. 2021-04-09T18:34:13.310 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:34:13.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:34:13.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.446 7fd8d3f6d700 1 Processor -- start 2021-04-09T18:34:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.447 7fd8d3f6d700 1 -- start start 2021-04-09T18:34:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.447 7fd8d3f6d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8cc10d010 0x7fd8cc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.447 7fd8d3f6d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd8cc0d1da0 0x7fd8cc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.447 7fd8d3f6d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc10d480 con 0x7fd8cc0d82c0 2021-04-09T18:34:13.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.447 7fd8d3f6d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc0d8d30 con 0x7fd8cc10d010 2021-04-09T18:34:13.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.447 7fd8d3f6d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc0d9220 con 0x7fd8cc0d1da0 2021-04-09T18:34:13.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d1d89700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8cc10d010 0x7fd8cc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d1d89700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd8cc10d010 0x7fd8cc0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59550/0 (socket says 172.21.15.32:59550) 2021-04-09T18:34:13.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d1d89700 1 -- 172.21.15.32:0/2035423531 learned_addr learned my addr 172.21.15.32:0/2035423531 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:13.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d1588700 1 --2- 172.21.15.32:0/2035423531 >> v2:172.21.15.86:3300/0 conn(0x7fd8cc0d1da0 0x7fd8cc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d1d89700 1 -- 172.21.15.32:0/2035423531 >> v2:172.21.15.86:3300/0 conn(0x7fd8cc0d1da0 msgr2=0x7fd8cc0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:13.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d1d89700 1 --2- 172.21.15.32:0/2035423531 >> v2:172.21.15.86:3300/0 conn(0x7fd8cc0d1da0 0x7fd8cc0d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d1d89700 1 -- 172.21.15.32:0/2035423531 >> v1:172.21.15.86:6789/0 conn(0x7fd8cc0d82c0 legacy=0x7fd8cc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d1d89700 1 -- 172.21.15.32:0/2035423531 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc0d86f0 con 0x7fd8cc10d010 2021-04-09T18:34:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d1d89700 1 --2- 172.21.15.32:0/2035423531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc10d010 0x7fd8cc0d1780 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7fd8c8006700 tx=0x7fd8c8009b80).ready entity=mon.0 client_cookie=1ae4a303c5828865 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:13.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d0d87700 1 -- 172.21.15.32:0/2035423531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8c800c0c0 con 0x7fd8cc10d010 2021-04-09T18:34:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d0d87700 1 -- 172.21.15.32:0/2035423531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c800c440 con 0x7fd8cc10d010 2021-04-09T18:34:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d0d87700 1 -- 172.21.15.32:0/2035423531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8c800c920 con 0x7fd8cc10d010 2021-04-09T18:34:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d3f6d700 1 -- 172.21.15.32:0/2035423531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc10d010 msgr2=0x7fd8cc0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:13.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.448 7fd8d3f6d700 1 --2- 172.21.15.32:0/2035423531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc10d010 0x7fd8cc0d1780 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7fd8c8006700 tx=0x7fd8c8009b80).stop 2021-04-09T18:34:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.449 7fd8d3f6d700 1 -- 172.21.15.32:0/2035423531 shutdown_connections 2021-04-09T18:34:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.449 7fd8d3f6d700 1 --2- 172.21.15.32:0/2035423531 >> v2:172.21.15.86:3300/0 conn(0x7fd8cc0d1da0 0x7fd8cc0d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.449 7fd8d3f6d700 1 --2- 172.21.15.32:0/2035423531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc10d010 0x7fd8cc0d1780 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.449 7fd8d3f6d700 1 -- 172.21.15.32:0/2035423531 >> 172.21.15.32:0/2035423531 conn(0x7fd8cc0ce5f0 msgr2=0x7fd8cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:13.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.449 7fd8d3f6d700 1 -- 172.21.15.32:0/2035423531 shutdown_connections 2021-04-09T18:34:13.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.449 7fd8d3f6d700 1 -- 172.21.15.32:0/2035423531 wait complete. 2021-04-09T18:34:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.469 7fd8d3f6d700 1 Processor -- start 2021-04-09T18:34:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.469 7fd8d3f6d700 1 -- start start 2021-04-09T18:34:13.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d3f6d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8cc09c5b0 0x7fd8cc09d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d3f6d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc09c9b0 0x7fd8cc09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d3f6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8cc090390 0x7fd8cc090790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d3f6d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc09e290 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d3f6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc09cf90 con 0x7fd8cc090390 2021-04-09T18:34:13.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d3f6d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc090db0 con 0x7fd8cc09c5b0 2021-04-09T18:34:13.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1588700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc09c9b0 0x7fd8cc09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1588700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc09c9b0 0x7fd8cc09db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59556/0 (socket says 172.21.15.32:59556) 2021-04-09T18:34:13.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1588700 1 -- 172.21.15.32:0/3603029245 learned_addr learned my addr 172.21.15.32:0/3603029245 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:13.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1d89700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8cc09c5b0 0x7fd8cc09d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d258a700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8cc090390 0x7fd8cc090790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1588700 1 -- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8cc09c5b0 msgr2=0x7fd8cc09d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:13.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1588700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8cc09c5b0 0x7fd8cc09d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1588700 1 -- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8cc090390 msgr2=0x7fd8cc090790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:13.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1588700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8cc090390 0x7fd8cc090790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:13.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1588700 1 -- 172.21.15.32:0/3603029245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc090f90 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d1588700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc09c9b0 0x7fd8cc09db30 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7fd8bc00b580 tx=0x7fd8bc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:13.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8c27fc700 1 -- 172.21.15.32:0/3603029245 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8bc006900 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8c27fc700 1 -- 172.21.15.32:0/3603029245 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8bc00bd10 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.470 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8cc090f90 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.471 7fd8c27fc700 1 -- 172.21.15.32:0/3603029245 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8bc002030 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.471 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc090f90 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.471 7fd8c27fc700 1 -- 172.21.15.32:0/3603029245 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd8bc0068b0 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.471 7fd8c27fc700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8b4018930 0x7fd8b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:13.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.471 7fd8c27fc700 1 -- 172.21.15.32:0/3603029245 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fd8bc032730 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.472 7fd8d1d89700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8b4018930 0x7fd8b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:13.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.472 7fd8d1d89700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8b4018930 0x7fd8b401ae10 crc :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:13.508 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.491 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc091b80 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.495 7fd8c27fc700 1 -- 172.21.15.32:0/3603029245 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd8bc000940 con 0x7fd8cc09c9b0 2021-04-09T18:34:13.695 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:13.677 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd8cc10d480 con 0x7fd8cc09c9b0 2021-04-09T18:34:14.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.660 7fd8c27fc700 1 -- 172.21.15.32:0/3603029245 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v256) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd8bc0068b0 con 0x7fd8cc09c9b0 2021-04-09T18:34:14.711 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:34:14.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8b4018930 msgr2=0x7fd8b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:14.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8b4018930 0x7fd8b401ae10 crc :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc09c9b0 msgr2=0x7fd8cc09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:14.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc09c9b0 0x7fd8cc09db30 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7fd8bc00b580 tx=0x7fd8bc00b920).stop 2021-04-09T18:34:14.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 shutdown_connections 2021-04-09T18:34:14.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd8b4018930 0x7fd8b401ae10 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd8cc09c9b0 0x7fd8cc09db30 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8cc090390 0x7fd8cc090790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 --2- 172.21.15.32:0/3603029245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8cc09c5b0 0x7fd8cc09d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 >> 172.21.15.32:0/3603029245 conn(0x7fd8cc0b2fa0 msgr2=0x7fd8cc0a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:14.716 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 shutdown_connections 2021-04-09T18:34:14.717 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.696 7fd8d3f6d700 1 -- 172.21.15.32:0/3603029245 wait complete. 2021-04-09T18:34:14.725 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:34:14.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.866 7f5dd6073700 1 Processor -- start 2021-04-09T18:34:14.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.866 7f5dd6073700 1 -- start start 2021-04-09T18:34:14.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.866 7f5dd6073700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5dd010d010 0x7f5dd00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.866 7f5dd6073700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5dd00d1da0 0x7f5dd00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.866 7f5dd6073700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd010d480 con 0x7f5dd00d82c0 2021-04-09T18:34:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.866 7f5dd6073700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd00d8d30 con 0x7f5dd00d1da0 2021-04-09T18:34:14.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dd6073700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd00d9220 con 0x7f5dd010d010 2021-04-09T18:34:14.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dceffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5dd00d1da0 0x7f5dd00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:14.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dceffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f5dd00d1da0 0x7f5dd00d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59568/0 (socket says 172.21.15.32:59568) 2021-04-09T18:34:14.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dceffd700 1 -- 172.21.15.32:0/203893862 learned_addr learned my addr 172.21.15.32:0/203893862 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:14.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dcf7fe700 1 --2- 172.21.15.32:0/203893862 >> v2:172.21.15.86:3300/0 conn(0x7f5dd010d010 0x7f5dd00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:14.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dceffd700 1 -- 172.21.15.32:0/203893862 >> v2:172.21.15.86:3300/0 conn(0x7f5dd010d010 msgr2=0x7f5dd00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:14.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dceffd700 1 --2- 172.21.15.32:0/203893862 >> v2:172.21.15.86:3300/0 conn(0x7f5dd010d010 0x7f5dd00d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dceffd700 1 -- 172.21.15.32:0/203893862 >> v1:172.21.15.86:6789/0 conn(0x7f5dd00d82c0 legacy=0x7f5dd010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:14.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dceffd700 1 -- 172.21.15.32:0/203893862 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd00d86f0 con 0x7f5dd00d1da0 2021-04-09T18:34:14.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dceffd700 1 --2- 172.21.15.32:0/203893862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00d1da0 0x7f5dd00d7ca0 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7f5dc4006700 tx=0x7f5dc4009b80).ready entity=mon.0 client_cookie=d3388e83ff4a274e server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:14.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dce7fc700 1 -- 172.21.15.32:0/203893862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dc400c0c0 con 0x7f5dd00d1da0 2021-04-09T18:34:14.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dce7fc700 1 -- 172.21.15.32:0/203893862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc400c440 con 0x7f5dd00d1da0 2021-04-09T18:34:14.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dce7fc700 1 -- 172.21.15.32:0/203893862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dc400c920 con 0x7f5dd00d1da0 2021-04-09T18:34:14.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dd6073700 1 -- 172.21.15.32:0/203893862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00d1da0 msgr2=0x7f5dd00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:14.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.867 7f5dd6073700 1 --2- 172.21.15.32:0/203893862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00d1da0 0x7f5dd00d7ca0 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7f5dc4006700 tx=0x7f5dc4009b80).stop 2021-04-09T18:34:14.911 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.868 7f5dd6073700 1 -- 172.21.15.32:0/203893862 shutdown_connections 2021-04-09T18:34:14.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.868 7f5dd6073700 1 --2- 172.21.15.32:0/203893862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00d1da0 0x7f5dd00d7ca0 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.868 7f5dd6073700 1 --2- 172.21.15.32:0/203893862 >> v2:172.21.15.86:3300/0 conn(0x7f5dd010d010 0x7f5dd00d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.868 7f5dd6073700 1 -- 172.21.15.32:0/203893862 >> 172.21.15.32:0/203893862 conn(0x7f5dd00ce5f0 msgr2=0x7f5dd00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:14.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.868 7f5dd6073700 1 -- 172.21.15.32:0/203893862 shutdown_connections 2021-04-09T18:34:14.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.868 7f5dd6073700 1 -- 172.21.15.32:0/203893862 wait complete. 2021-04-09T18:34:14.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.888 7f5dd6073700 1 Processor -- start 2021-04-09T18:34:14.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dd6073700 1 -- start start 2021-04-09T18:34:14.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dd6073700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00c3c00 0x7f5dd009c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:14.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dd6073700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dd0095240 0x7f5dd01a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:14.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dd6073700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dd01a1270 0x7f5dd00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:14.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dd6073700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd009cda0 con 0x7f5dd00c3c00 2021-04-09T18:34:14.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dd6073700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00c3680 con 0x7f5dd0095240 2021-04-09T18:34:14.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dd6073700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd0095640 con 0x7f5dd01a1270 2021-04-09T18:34:14.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dcf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00c3c00 0x7f5dd009c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:14.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dcf7fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00c3c00 0x7f5dd009c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59570/0 (socket says 172.21.15.32:59570) 2021-04-09T18:34:14.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dcf7fe700 1 -- 172.21.15.32:0/4227941377 learned_addr learned my addr 172.21.15.32:0/4227941377 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:14.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dcffff700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dd01a1270 0x7f5dd00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:14.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.889 7f5dceffd700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dd0095240 0x7f5dd01a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:14.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5dcf7fe700 1 -- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dd01a1270 msgr2=0x7f5dd00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:14.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5dcf7fe700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dd01a1270 0x7f5dd00b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5dcf7fe700 1 -- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dd0095240 msgr2=0x7f5dd01a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:14.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5dcf7fe700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dd0095240 0x7f5dd01a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:14.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5dcf7fe700 1 -- 172.21.15.32:0/4227941377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd01a1670 con 0x7f5dd00c3c00 2021-04-09T18:34:14.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5dceffd700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dd0095240 0x7f5dd01a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:14.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5dcf7fe700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00c3c00 0x7f5dd009c600 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f5dc000b580 tx=0x7f5dc000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:14.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5db7fff700 1 -- 172.21.15.32:0/4227941377 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dc00064d0 con 0x7f5dd00c3c00 2021-04-09T18:34:14.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5db7fff700 1 -- 172.21.15.32:0/4227941377 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc000bd10 con 0x7f5dd00c3c00 2021-04-09T18:34:14.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5db7fff700 1 -- 172.21.15.32:0/4227941377 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dc0006d00 con 0x7f5dd00c3c00 2021-04-09T18:34:14.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd01a1670 con 0x7f5dd00c3c00 2021-04-09T18:34:14.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.890 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd01a1670 con 0x7f5dd00c3c00 2021-04-09T18:34:14.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.891 7f5db7fff700 1 -- 172.21.15.32:0/4227941377 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f5dc0006c90 con 0x7f5dd00c3c00 2021-04-09T18:34:14.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.891 7f5db7fff700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5db00188d0 0x7f5db001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:14.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.891 7f5db7fff700 1 -- 172.21.15.32:0/4227941377 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 5111+0+0 (secure 0 0 0) 0x7f5dc0032830 con 0x7f5dd00c3c00 2021-04-09T18:34:14.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.891 7f5dceffd700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5db00188d0 0x7f5db001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:14.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.891 7f5dceffd700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5db00188d0 0x7f5db001add0 crc :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:14.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.910 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd0093d50 con 0x7f5dd00c3c00 2021-04-09T18:34:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:14.914 7f5db7fff700 1 -- 172.21.15.32:0/4227941377 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f5dc0006820 con 0x7f5dd00c3c00 2021-04-09T18:34:15.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:15.105 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f5dd010d480 con 0x7f5dd00c3c00 2021-04-09T18:34:16.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.009 7f5db7fff700 1 -- 172.21.15.32:0/4227941377 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v261) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5dc0006c90 con 0x7f5dd00c3c00 2021-04-09T18:34:16.038 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:34:16.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5db00188d0 msgr2=0x7f5db001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5db00188d0 0x7f5db001add0 crc :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00c3c00 msgr2=0x7f5dd009c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00c3c00 0x7f5dd009c600 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f5dc000b580 tx=0x7f5dc000b920).stop 2021-04-09T18:34:16.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 shutdown_connections 2021-04-09T18:34:16.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f5db00188d0 0x7f5db001add0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dd01a1270 0x7f5dd00b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dd0095240 0x7f5dd01a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 --2- 172.21.15.32:0/4227941377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dd00c3c00 0x7f5dd009c600 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 >> 172.21.15.32:0/4227941377 conn(0x7f5dd00975f0 msgr2=0x7f5dd0096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 shutdown_connections 2021-04-09T18:34:16.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.023 7f5dd6073700 1 -- 172.21.15.32:0/4227941377 wait complete. 2021-04-09T18:34:16.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:34:16.235 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.195 7fe48ddeb700 1 Processor -- start 2021-04-09T18:34:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.195 7fe48ddeb700 1 -- start start 2021-04-09T18:34:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe48ddeb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48810d050 0x7fe4880cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe48ddeb700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe4880d0160 0x7fe4880d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe48ddeb700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48810d500 con 0x7fe4880d82e0 2021-04-09T18:34:16.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe48ddeb700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4880d8d70 con 0x7fe4880d0160 2021-04-09T18:34:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe48ddeb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880d9260 con 0x7fe48810d050 2021-04-09T18:34:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe4877fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48810d050 0x7fe4880cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe487fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe4880d82e0 0x7fe48810da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38868/0 (socket says 172.21.15.32:38868) 2021-04-09T18:34:16.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe486ffd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe4880d0160 0x7fe4880d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe487fff700 1 -- 172.21.15.32:0/2035607480 learned_addr learned my addr 172.21.15.32:0/2035607480 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe4867fc700 1 -- 172.21.15.32:0/2035607480 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200784374 0 0) 0x7fe480001440 con 0x7fe4880d82e0 2021-04-09T18:34:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe4867fc700 1 -- 172.21.15.32:0/2035607480 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe46c002fe0 con 0x7fe4880d82e0 2021-04-09T18:34:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe486ffd700 1 -- 172.21.15.32:0/2035607480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48810d050 msgr2=0x7fe4880cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:16.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.196 7fe486ffd700 1 --2- 172.21.15.32:0/2035607480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48810d050 0x7fe4880cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.197 7fe486ffd700 1 -- 172.21.15.32:0/2035607480 >> v1:172.21.15.86:6789/0 conn(0x7fe4880d82e0 legacy=0x7fe48810da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.197 7fe486ffd700 1 -- 172.21.15.32:0/2035607480 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4880d86e0 con 0x7fe4880d0160 2021-04-09T18:34:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.197 7fe4877fe700 1 --2- 172.21.15.32:0/2035607480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48810d050 0x7fe4880cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:16.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.197 7fe486ffd700 1 --2- 172.21.15.32:0/2035607480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880d0160 0x7fe4880d7cc0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7fe474006650 tx=0x7fe474009a80).ready entity=mon.1 client_cookie=7a6de0599d94797 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.197 7fe4867fc700 1 -- 172.21.15.32:0/2035607480 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4740009b0 con 0x7fe4880d0160 2021-04-09T18:34:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.197 7fe4867fc700 1 -- 172.21.15.32:0/2035607480 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47400c0c0 con 0x7fe4880d0160 2021-04-09T18:34:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.197 7fe4867fc700 1 -- 172.21.15.32:0/2035607480 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe47400c510 con 0x7fe4880d0160 2021-04-09T18:34:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.197 7fe48ddeb700 1 -- 172.21.15.32:0/2035607480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880d0160 msgr2=0x7fe4880d7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.197 7fe48ddeb700 1 --2- 172.21.15.32:0/2035607480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880d0160 0x7fe4880d7cc0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7fe474006650 tx=0x7fe474009a80).stop 2021-04-09T18:34:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.198 7fe48ddeb700 1 -- 172.21.15.32:0/2035607480 shutdown_connections 2021-04-09T18:34:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.198 7fe48ddeb700 1 --2- 172.21.15.32:0/2035607480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880d0160 0x7fe4880d7cc0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.198 7fe48ddeb700 1 --2- 172.21.15.32:0/2035607480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48810d050 0x7fe4880cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.198 7fe48ddeb700 1 -- 172.21.15.32:0/2035607480 >> 172.21.15.32:0/2035607480 conn(0x7fe4880ce620 msgr2=0x7fe4880d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.198 7fe48ddeb700 1 -- 172.21.15.32:0/2035607480 shutdown_connections 2021-04-09T18:34:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.198 7fe48ddeb700 1 -- 172.21.15.32:0/2035607480 wait complete. 2021-04-09T18:34:16.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.218 7fe48ddeb700 1 Processor -- start 2021-04-09T18:34:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.218 7fe48ddeb700 1 -- start start 2021-04-09T18:34:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.218 7fe48ddeb700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4880b0920 0x7fe4880b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.245 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe48ddeb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880a7690 0x7fe4880a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe48ddeb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c2510 0x7fe4880c2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe48ddeb700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880b1340 con 0x7fe4880c2510 2021-04-09T18:34:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe48ddeb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880a82e0 con 0x7fe4880a7690 2021-04-09T18:34:16.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe48ddeb700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880a8820 con 0x7fe4880b0920 2021-04-09T18:34:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe487fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c2510 0x7fe4880c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe487fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c2510 0x7fe4880c2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59592/0 (socket says 172.21.15.32:59592) 2021-04-09T18:34:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe487fff700 1 -- 172.21.15.32:0/3122312009 learned_addr learned my addr 172.21.15.32:0/3122312009 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:16.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe486ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880a7690 0x7fe4880a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe4877fe700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4880b0920 0x7fe4880b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe486ffd700 1 -- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4880b0920 msgr2=0x7fe4880b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe486ffd700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4880b0920 0x7fe4880b0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe486ffd700 1 -- 172.21.15.32:0/3122312009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c2510 msgr2=0x7fe4880c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe486ffd700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c2510 0x7fe4880c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe486ffd700 1 -- 172.21.15.32:0/3122312009 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48808fba0 con 0x7fe4880a7690 2021-04-09T18:34:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe487fff700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c2510 0x7fe4880c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe486ffd700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880a7690 0x7fe4880a7a90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7fe47400b530 tx=0x7fe47400b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:16.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe47ffff700 1 -- 172.21.15.32:0/3122312009 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe474006740 con 0x7fe4880a7690 2021-04-09T18:34:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe47ffff700 1 -- 172.21.15.32:0/3122312009 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe474002030 con 0x7fe4880a7690 2021-04-09T18:34:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48808fba0 con 0x7fe4880a7690 2021-04-09T18:34:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe47ffff700 1 -- 172.21.15.32:0/3122312009 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe474002590 con 0x7fe4880a7690 2021-04-09T18:34:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.219 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48808fba0 con 0x7fe4880a7690 2021-04-09T18:34:16.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.220 7fe47ffff700 1 -- 172.21.15.32:0/3122312009 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe474002210 con 0x7fe4880a7690 2021-04-09T18:34:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.220 7fe47ffff700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe464018900 0x7fe46401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.220 7fe47ffff700 1 -- 172.21.15.32:0/3122312009 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5461+0+0 (secure 0 0 0) 0x7fe4740328a0 con 0x7fe4880a7690 2021-04-09T18:34:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.220 7fe4877fe700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe464018900 0x7fe46401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.221 7fe4877fe700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe464018900 0x7fe46401adc0 crc :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:16.257 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.240 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4881b7180 con 0x7fe4880a7690 2021-04-09T18:34:16.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.243 7fe47ffff700 1 -- 172.21.15.32:0/3122312009 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe474006740 con 0x7fe4880a7690 2021-04-09T18:34:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.428 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fe48810d500 con 0x7fe4880a7690 2021-04-09T18:34:16.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.438 7fe47ffff700 1 -- 172.21.15.32:0/3122312009 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 44 and data pool 45 v423) v1 ==== 189+0+0 (secure 0 0 0) 0x7fe474002030 con 0x7fe4880a7690 2021-04-09T18:34:16.461 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 44 and data pool 45 2021-04-09T18:34:16.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe464018900 msgr2=0x7fe46401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe464018900 0x7fe46401adc0 crc :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880a7690 msgr2=0x7fe4880a7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880a7690 0x7fe4880a7a90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7fe47400b530 tx=0x7fe47400b8d0).stop 2021-04-09T18:34:16.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 shutdown_connections 2021-04-09T18:34:16.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe464018900 0x7fe46401adc0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4880c2510 0x7fe4880c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4880a7690 0x7fe4880a7a90 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 --2- 172.21.15.32:0/3122312009 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4880b0920 0x7fe4880b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 >> 172.21.15.32:0/3122312009 conn(0x7fe4880b6b70 msgr2=0x7fe4880b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:16.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 shutdown_connections 2021-04-09T18:34:16.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.447 7fe48ddeb700 1 -- 172.21.15.32:0/3122312009 wait complete. 2021-04-09T18:34:16.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:34:16.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.606 7fa4b7954700 1 Processor -- start 2021-04-09T18:34:16.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.606 7fa4b7954700 1 -- start start 2021-04-09T18:34:16.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b7954700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4b00cffa0 0x7fa4b00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b7954700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b00d7800 0x7fa4b00d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b7954700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4b00d8220 0x7fa4b00d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b7954700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b00d91e0 con 0x7fa4b00cffa0 2021-04-09T18:34:16.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b7954700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b00d7280 con 0x7fa4b00d8220 2021-04-09T18:34:16.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b7954700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00d6290 con 0x7fa4b00d7800 2021-04-09T18:34:16.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b5770700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4b00cffa0 0x7fa4b00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b5770700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fa4b00cffa0 0x7fa4b00d2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59608/0 (socket says 172.21.15.32:59608) 2021-04-09T18:34:16.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b5770700 1 -- 172.21.15.32:0/360623431 learned_addr learned my addr 172.21.15.32:0/360623431 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:16.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b4f6f700 1 --2- 172.21.15.32:0/360623431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b00d7800 0x7fa4b00d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b5f71700 1 --2- 172.21.15.32:0/360623431 >> v2:172.21.15.86:3300/0 conn(0x7fa4b00d8220 0x7fa4b00d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b5770700 1 -- 172.21.15.32:0/360623431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b00d7800 msgr2=0x7fa4b00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:16.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b5770700 1 --2- 172.21.15.32:0/360623431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b00d7800 0x7fa4b00d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b5770700 1 -- 172.21.15.32:0/360623431 >> v2:172.21.15.86:3300/0 conn(0x7fa4b00d8220 msgr2=0x7fa4b00d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b5770700 1 --2- 172.21.15.32:0/360623431 >> v2:172.21.15.86:3300/0 conn(0x7fa4b00d8220 0x7fa4b00d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.607 7fa4b5770700 1 -- 172.21.15.32:0/360623431 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00d64b0 con 0x7fa4b00cffa0 2021-04-09T18:34:16.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.608 7fa4b5770700 1 --2- 172.21.15.32:0/360623431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b00cffa0 0x7fa4b00d2440 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7fa4ac006700 tx=0x7fa4ac009b00).ready entity=mon.0 client_cookie=dbb405b0a0f6aa04 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:16.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.608 7fa4a7fff700 1 -- 172.21.15.32:0/360623431 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ac00c0c0 con 0x7fa4b00cffa0 2021-04-09T18:34:16.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.608 7fa4a7fff700 1 -- 172.21.15.32:0/360623431 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac00c400 con 0x7fa4b00cffa0 2021-04-09T18:34:16.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.608 7fa4a7fff700 1 -- 172.21.15.32:0/360623431 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ac00c8e0 con 0x7fa4b00cffa0 2021-04-09T18:34:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.608 7fa4b7954700 1 -- 172.21.15.32:0/360623431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b00cffa0 msgr2=0x7fa4b00d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.608 7fa4b7954700 1 --2- 172.21.15.32:0/360623431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b00cffa0 0x7fa4b00d2440 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7fa4ac006700 tx=0x7fa4ac009b00).stop 2021-04-09T18:34:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.609 7fa4b7954700 1 -- 172.21.15.32:0/360623431 shutdown_connections 2021-04-09T18:34:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.609 7fa4b7954700 1 --2- 172.21.15.32:0/360623431 >> v2:172.21.15.86:3300/0 conn(0x7fa4b00d8220 0x7fa4b00d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.609 7fa4b7954700 1 --2- 172.21.15.32:0/360623431 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b00d7800 0x7fa4b00d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.609 7fa4b7954700 1 --2- 172.21.15.32:0/360623431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b00cffa0 0x7fa4b00d2440 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.609 7fa4b7954700 1 -- 172.21.15.32:0/360623431 >> 172.21.15.32:0/360623431 conn(0x7fa4b00ce5c0 msgr2=0x7fa4b00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.609 7fa4b7954700 1 -- 172.21.15.32:0/360623431 shutdown_connections 2021-04-09T18:34:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.609 7fa4b7954700 1 -- 172.21.15.32:0/360623431 wait complete. 2021-04-09T18:34:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.629 7fa4b7954700 1 Processor -- start 2021-04-09T18:34:16.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.629 7fa4b7954700 1 -- start start 2021-04-09T18:34:16.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.629 7fa4b7954700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b01acff0 0x7fa4b01847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.629 7fa4b7954700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b0183fc0 0x7fa4b01a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.629 7fa4b7954700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b0186ff0 0x7fa4b01873f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.629 7fa4b7954700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b0188c30 con 0x7fa4b01acff0 2021-04-09T18:34:16.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b7954700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01843c0 con 0x7fa4b0186ff0 2021-04-09T18:34:16.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b7954700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01a5c20 con 0x7fa4b0183fc0 2021-04-09T18:34:16.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5770700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b01acff0 0x7fa4b01847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5770700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b01acff0 0x7fa4b01847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59614/0 (socket says 172.21.15.32:59614) 2021-04-09T18:34:16.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5770700 1 -- 172.21.15.32:0/34849391 learned_addr learned my addr 172.21.15.32:0/34849391 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:16.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b4f6f700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b0183fc0 0x7fa4b01a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5f71700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b0186ff0 0x7fa4b01873f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5770700 1 -- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b0183fc0 msgr2=0x7fa4b01a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5770700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b0183fc0 0x7fa4b01a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5770700 1 -- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b0186ff0 msgr2=0x7fa4b01873f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5770700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b0186ff0 0x7fa4b01873f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5770700 1 -- 172.21.15.32:0/34849391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0187a10 con 0x7fa4b01acff0 2021-04-09T18:34:16.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b5770700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b01acff0 0x7fa4b01847a0 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 rx=0x7fa4ac00b660 tx=0x7fa4ac00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:16.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4a5ffb700 1 -- 172.21.15.32:0/34849391 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ac009440 con 0x7fa4b01acff0 2021-04-09T18:34:16.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4a5ffb700 1 -- 172.21.15.32:0/34849391 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac006390 con 0x7fa4b01acff0 2021-04-09T18:34:16.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4a5ffb700 1 -- 172.21.15.32:0/34849391 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ac002030 con 0x7fa4b01acff0 2021-04-09T18:34:16.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b7954700 1 -- 172.21.15.32:0/34849391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b0187a10 con 0x7fa4b01acff0 2021-04-09T18:34:16.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.630 7fa4b7954700 1 -- 172.21.15.32:0/34849391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b0187a10 con 0x7fa4b01acff0 2021-04-09T18:34:16.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.631 7fa4a5ffb700 1 -- 172.21.15.32:0/34849391 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fa4ac0093e0 con 0x7fa4b01acff0 2021-04-09T18:34:16.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.631 7fa4a5ffb700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4900188c0 0x7fa49001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:16.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.631 7fa4a5ffb700 1 -- 172.21.15.32:0/34849391 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fa4ac0327d0 con 0x7fa4b01acff0 2021-04-09T18:34:16.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.631 7fa4b4f6f700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4900188c0 0x7fa49001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:16.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.632 7fa4b4f6f700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4900188c0 0x7fa49001adc0 crc :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:16.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.651 7fa4b7954700 1 -- 172.21.15.32:0/34849391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b0182dc0 con 0x7fa4b01acff0 2021-04-09T18:34:16.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.655 7fa4a5ffb700 1 -- 172.21.15.32:0/34849391 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fa4ac02dfc0 con 0x7fa4b01acff0 2021-04-09T18:34:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.845 7fa4b7954700 1 -- 172.21.15.32:0/34849391 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4b00d7280 con 0x7fa4b01acff0 2021-04-09T18:34:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.846 7fa4a5ffb700 1 -- 172.21.15.32:0/34849391 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fa4ac02dfc0 con 0x7fa4b01acff0 2021-04-09T18:34:16.867 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:16.867 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":262,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:34:16.431130","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":45,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":44,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:34:13.713911","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":45,"pool_name":"cephfs_data","create_time":"2021-04-09 18:34:15.142274","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:34:16.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 -- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4900188c0 msgr2=0x7fa49001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4900188c0 0x7fa49001adc0 crc :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 -- 172.21.15.32:0/34849391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b01acff0 msgr2=0x7fa4b01847a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:16.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b01acff0 0x7fa4b01847a0 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 rx=0x7fa4ac00b660 tx=0x7fa4ac00ba00).stop 2021-04-09T18:34:16.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 -- 172.21.15.32:0/34849391 shutdown_connections 2021-04-09T18:34:16.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fa4900188c0 0x7fa49001adc0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b0186ff0 0x7fa4b01873f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b0183fc0 0x7fa4b01a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 --2- 172.21.15.32:0/34849391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4b01acff0 0x7fa4b01847a0 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:16.878 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 -- 172.21.15.32:0/34849391 >> 172.21.15.32:0/34849391 conn(0x7fa4b009abb0 msgr2=0x7fa4b00a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:16.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 -- 172.21.15.32:0/34849391 shutdown_connections 2021-04-09T18:34:16.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:16.848 7fa4b7954700 1 -- 172.21.15.32:0/34849391 wait complete. 2021-04-09T18:34:16.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:34:17.073 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.033 7f438d13b700 1 Processor -- start 2021-04-09T18:34:17.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.033 7f438d13b700 1 -- start start 2021-04-09T18:34:17.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.033 7f438d13b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f438810e7d0 0x7f43880d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.074 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438d13b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43880d7280 con 0x7f43880d6c80 2021-04-09T18:34:17.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438d13b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43880d2440 con 0x7f43880cffa0 2021-04-09T18:34:17.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438d13b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43880d7080 con 0x7f438810e7d0 2021-04-09T18:34:17.075 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438759e700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f438810e7d0 0x7f43880d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438659c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f43880d6c80 0x7f438810cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40982/0 (socket says 172.21.15.32:40982) 2021-04-09T18:34:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438659c700 1 -- 172.21.15.32:0/1272699181 learned_addr learned my addr 172.21.15.32:0/1272699181 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:17.076 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438759e700 1 -- 172.21.15.32:0/1272699181 >> v1:172.21.15.86:6789/0 conn(0x7f43880cffa0 legacy=0x7f43880d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f4385d9b700 1 -- 172.21.15.32:0/1272699181 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706719341 0 0) 0x7f437c000ce0 con 0x7f43880d6c80 2021-04-09T18:34:17.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438759e700 1 -- 172.21.15.32:0/1272699181 >> v1:172.21.15.32:6789/0 conn(0x7f43880d6c80 legacy=0x7f438810cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.077 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438759e700 1 -- 172.21.15.32:0/1272699181 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43880d91e0 con 0x7f438810e7d0 2021-04-09T18:34:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438759e700 1 --2- 172.21.15.32:0/1272699181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438810e7d0 0x7f43880d7ff0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 rx=0x7f4380006ab0 tx=0x7f4380009ac0).ready entity=mon.0 client_cookie=334e747dfa4ade59 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f4385d9b700 1 -- 172.21.15.32:0/1272699181 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43800066d0 con 0x7f438810e7d0 2021-04-09T18:34:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f4385d9b700 1 -- 172.21.15.32:0/1272699181 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43800022e0 con 0x7f438810e7d0 2021-04-09T18:34:17.078 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f4385d9b700 1 -- 172.21.15.32:0/1272699181 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43800027c0 con 0x7f438810e7d0 2021-04-09T18:34:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438d13b700 1 -- 172.21.15.32:0/1272699181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438810e7d0 msgr2=0x7f43880d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.034 7f438d13b700 1 --2- 172.21.15.32:0/1272699181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438810e7d0 0x7f43880d7ff0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 rx=0x7f4380006ab0 tx=0x7f4380009ac0).stop 2021-04-09T18:34:17.079 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.035 7f438d13b700 1 -- 172.21.15.32:0/1272699181 shutdown_connections 2021-04-09T18:34:17.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.035 7f438d13b700 1 --2- 172.21.15.32:0/1272699181 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438810e7d0 0x7f43880d7ff0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.035 7f438d13b700 1 -- 172.21.15.32:0/1272699181 >> 172.21.15.32:0/1272699181 conn(0x7f43880ce5c0 msgr2=0x7f43880cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:17.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.035 7f438d13b700 1 -- 172.21.15.32:0/1272699181 shutdown_connections 2021-04-09T18:34:17.080 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.035 7f438d13b700 1 -- 172.21.15.32:0/1272699181 wait complete. 2021-04-09T18:34:17.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.055 7f438d13b700 1 Processor -- start 2021-04-09T18:34:17.081 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.055 7f438d13b700 1 -- start start 2021-04-09T18:34:17.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438d13b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f438817dce0 0x7f438818d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.084 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438d13b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f438817e0e0 0x7f438818dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438d13b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438819ae80 0x7f438819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438d13b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43881964f0 con 0x7f438819ae80 2021-04-09T18:34:17.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438d13b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438819c820 con 0x7f438817dce0 2021-04-09T18:34:17.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438d13b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438817e820 con 0x7f438817e0e0 2021-04-09T18:34:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438759e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438819ae80 0x7f438819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438759e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438819ae80 0x7f438819b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59630/0 (socket says 172.21.15.32:59630) 2021-04-09T18:34:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438759e700 1 -- 172.21.15.32:0/345329748 learned_addr learned my addr 172.21.15.32:0/345329748 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:17.086 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438659c700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f438817e0e0 0x7f438818dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f4386d9d700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f438817dce0 0x7f438818d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438759e700 1 -- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f438817e0e0 msgr2=0x7f438818dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.087 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438759e700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f438817e0e0 0x7f438818dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438759e700 1 -- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f438817dce0 msgr2=0x7f438818d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438759e700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f438817dce0 0x7f438818d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.056 7f438759e700 1 -- 172.21.15.32:0/345329748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438819b8a0 con 0x7f438819ae80 2021-04-09T18:34:17.088 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.057 7f438759e700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438819ae80 0x7f438819b280 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7f438000afc0 tx=0x7f4380002850).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:17.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.057 7f43777fe700 1 -- 172.21.15.32:0/345329748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f438000a070 con 0x7f438819ae80 2021-04-09T18:34:17.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.057 7f43777fe700 1 -- 172.21.15.32:0/345329748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438000a330 con 0x7f438819ae80 2021-04-09T18:34:17.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.057 7f438d13b700 1 -- 172.21.15.32:0/345329748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438819b8a0 con 0x7f438819ae80 2021-04-09T18:34:17.089 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.057 7f43777fe700 1 -- 172.21.15.32:0/345329748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4380009610 con 0x7f438819ae80 2021-04-09T18:34:17.090 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.057 7f438d13b700 1 -- 172.21.15.32:0/345329748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438819b8a0 con 0x7f438819ae80 2021-04-09T18:34:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.057 7f43777fe700 1 -- 172.21.15.32:0/345329748 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4380002c40 con 0x7f438819ae80 2021-04-09T18:34:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.058 7f43777fe700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f436c018930 0x7f436c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.091 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.058 7f43777fe700 1 -- 172.21.15.32:0/345329748 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4380032a00 con 0x7f438819ae80 2021-04-09T18:34:17.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.058 7f4386d9d700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f436c018930 0x7f436c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.092 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.058 7f4386d9d700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f436c018930 0x7f436c01adc0 crc :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:17.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.078 7f438d13b700 1 -- 172.21.15.32:0/345329748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43881964b0 con 0x7f438819ae80 2021-04-09T18:34:17.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.081 7f43777fe700 1 -- 172.21.15.32:0/345329748 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f43800009b0 con 0x7f438819ae80 2021-04-09T18:34:17.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.270 7f438d13b700 1 -- 172.21.15.32:0/345329748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43881964b0 con 0x7f438819ae80 2021-04-09T18:34:17.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.271 7f43777fe700 1 -- 172.21.15.32:0/345329748 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f4380002c40 con 0x7f438819ae80 2021-04-09T18:34:17.289 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:17.290 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":262,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:34:16.431130","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":45,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":44,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:34:13.713911","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":45,"pool_name":"cephfs_data","create_time":"2021-04-09 18:34:15.142274","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:34:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 -- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f436c018930 msgr2=0x7f436c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f436c018930 0x7f436c01adc0 crc :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 -- 172.21.15.32:0/345329748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438819ae80 msgr2=0x7f438819b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438819ae80 0x7f438819b280 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7f438000afc0 tx=0x7f4380002850).stop 2021-04-09T18:34:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 -- 172.21.15.32:0/345329748 shutdown_connections 2021-04-09T18:34:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f436c018930 0x7f436c01adc0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f438819ae80 0x7f438819b280 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f438817e0e0 0x7f438818dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 --2- 172.21.15.32:0/345329748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f438817dce0 0x7f438818d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 -- 172.21.15.32:0/345329748 >> 172.21.15.32:0/345329748 conn(0x7f438818ce60 msgr2=0x7f4388094920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 -- 172.21.15.32:0/345329748 shutdown_connections 2021-04-09T18:34:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.273 7f438d13b700 1 -- 172.21.15.32:0/345329748 wait complete. 2021-04-09T18:34:17.302 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:34:17.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.438 7f147d1a3700 1 Processor -- start 2021-04-09T18:34:17.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.438 7f147d1a3700 1 -- start start 2021-04-09T18:34:17.482 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f147d1a3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f14780d7ca0 0x7f14780d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f147d1a3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147810d480 con 0x7f147810da00 2021-04-09T18:34:17.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f147d1a3700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14780d86c0 con 0x7f147810d010 2021-04-09T18:34:17.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f147d1a3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14780d9220 con 0x7f14780d7ca0 2021-04-09T18:34:17.483 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f147759e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f147810da00 0x7f147810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40998/0 (socket says 172.21.15.32:40998) 2021-04-09T18:34:17.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f147759e700 1 -- 172.21.15.32:0/142471928 learned_addr learned my addr 172.21.15.32:0/142471928 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:17.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f147659c700 1 --2- 172.21.15.32:0/142471928 >> v2:172.21.15.86:3300/0 conn(0x7f14780d7ca0 0x7f14780d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605905064 0 0) 0x7f1470000d30 con 0x7f147810da00 2021-04-09T18:34:17.484 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1460002fe0 con 0x7f147810da00 2021-04-09T18:34:17.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3771278841 0 0) 0x7f1470001460 con 0x7f147810da00 2021-04-09T18:34:17.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 >> v2:172.21.15.86:3300/0 conn(0x7f14780d7ca0 msgr2=0x7f14780d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:17.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 --2- 172.21.15.32:0/142471928 >> v2:172.21.15.86:3300/0 conn(0x7f14780d7ca0 0x7f14780d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.485 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 >> v1:172.21.15.86:6789/0 conn(0x7f147810d010 legacy=0x7f14780d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478112100 con 0x7f147810da00 2021-04-09T18:34:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f1470001980 con 0x7f147810da00 2021-04-09T18:34:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f147659c700 1 --2- 172.21.15.32:0/142471928 >> v2:172.21.15.86:3300/0 conn(0x7f14780d7ca0 0x7f14780d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:17.486 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 >> v1:172.21.15.32:6789/0 conn(0x7f147810da00 legacy=0x7f147810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1460003a60 0x7f1460005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1460006790 0x7f1460008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1460009590 0x7f146000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146000c300 con 0x7f1460009590 2021-04-09T18:34:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146000c520 con 0x7f1460006790 2021-04-09T18:34:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f1475d9b700 1 -- 172.21.15.32:0/142471928 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1460050750 con 0x7f1460003a60 2021-04-09T18:34:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.439 7f147759e700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1460009590 0x7f146000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147659c700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1460006790 0x7f1460008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f1476d9d700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1460003a60 0x7f1460005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147759e700 1 -- 172.21.15.32:0/142471928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1460003a60 msgr2=0x7f1460005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147759e700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1460003a60 0x7f1460005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147759e700 1 -- 172.21.15.32:0/142471928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1460006790 msgr2=0x7f1460008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:17.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147759e700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1460006790 0x7f1460008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147759e700 1 -- 172.21.15.32:0/142471928 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1460050980 con 0x7f1460009590 2021-04-09T18:34:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147659c700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1460006790 0x7f1460008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147759e700 1 -- 172.21.15.32:0/142471928 reap_dead start 2021-04-09T18:34:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147759e700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1460009590 0x7f146000bab0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7f1470007cf0 tx=0x7f147000a310).ready entity=mon.0 client_cookie=7d5e1fbf6775f4b2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:17.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f1475d9b700 1 -- 172.21.15.32:0/142471928 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1470007340 con 0x7f1460009590 2021-04-09T18:34:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f1475d9b700 1 -- 172.21.15.32:0/142471928 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1470004720 con 0x7f1460009590 2021-04-09T18:34:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f1475d9b700 1 -- 172.21.15.32:0/142471928 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1470004b80 con 0x7f1460009590 2021-04-09T18:34:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147d1a3700 1 -- 172.21.15.32:0/142471928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1460009590 msgr2=0x7f146000bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.440 7f147d1a3700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1460009590 0x7f146000bab0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7f1470007cf0 tx=0x7f147000a310).stop 2021-04-09T18:34:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.441 7f147d1a3700 1 -- 172.21.15.32:0/142471928 shutdown_connections 2021-04-09T18:34:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.441 7f147d1a3700 1 --2- 172.21.15.32:0/142471928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1460009590 0x7f146000bab0 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.441 7f147d1a3700 1 -- 172.21.15.32:0/142471928 >> 172.21.15.32:0/142471928 conn(0x7f14780ce5f0 msgr2=0x7f14780d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.441 7f147d1a3700 1 -- 172.21.15.32:0/142471928 shutdown_connections 2021-04-09T18:34:17.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.441 7f147d1a3700 1 -- 172.21.15.32:0/142471928 wait complete. 2021-04-09T18:34:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147d1a3700 1 Processor -- start 2021-04-09T18:34:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147d1a3700 1 -- start start 2021-04-09T18:34:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147d1a3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14780c3c00 0x7f147809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147d1a3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1478095260 0x7f14781a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147d1a3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14781a1270 0x7f14780b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147d1a3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147809cda0 con 0x7f1478095260 2021-04-09T18:34:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147d1a3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14780c3680 con 0x7f14781a1270 2021-04-09T18:34:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147d1a3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1478095660 con 0x7f14780c3c00 2021-04-09T18:34:17.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1478095260 0x7f14781a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147659c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1478095260 0x7f14781a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59650/0 (socket says 172.21.15.32:59650) 2021-04-09T18:34:17.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147659c700 1 -- 172.21.15.32:0/3230103938 learned_addr learned my addr 172.21.15.32:0/3230103938 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:17.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f1476d9d700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14780c3c00 0x7f147809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.462 7f147759e700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14781a1270 0x7f14780b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f147659c700 1 -- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14780c3c00 msgr2=0x7f147809c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f147659c700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14780c3c00 0x7f147809c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f147659c700 1 -- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14781a1270 msgr2=0x7f14780b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f147659c700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14781a1270 0x7f14780b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:17.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f147659c700 1 -- 172.21.15.32:0/3230103938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14781a1670 con 0x7f1478095260 2021-04-09T18:34:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f147659c700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1478095260 0x7f14781a0c50 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7f146c00b5e0 tx=0x7f146c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f14677fe700 1 -- 172.21.15.32:0/3230103938 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f146c00bd70 con 0x7f1478095260 2021-04-09T18:34:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f14677fe700 1 -- 172.21.15.32:0/3230103938 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c006740 con 0x7f1478095260 2021-04-09T18:34:17.497 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14781a1670 con 0x7f1478095260 2021-04-09T18:34:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f14677fe700 1 -- 172.21.15.32:0/3230103938 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f146c0022b0 con 0x7f1478095260 2021-04-09T18:34:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.463 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14781a1670 con 0x7f1478095260 2021-04-09T18:34:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.464 7f14677fe700 1 -- 172.21.15.32:0/3230103938 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f146c00a000 con 0x7f1478095260 2021-04-09T18:34:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.464 7f14677fe700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f145c0188c0 0x7f145c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:17.498 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.464 7f14677fe700 1 -- 172.21.15.32:0/3230103938 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f146c032970 con 0x7f1478095260 2021-04-09T18:34:17.499 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.464 7f1476d9d700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f145c0188c0 0x7f145c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.464 7f1476d9d700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f145c0188c0 0x7f145c01adc0 crc :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:17.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.482 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147810d480 con 0x7f1478095260 2021-04-09T18:34:17.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.486 7f14677fe700 1 -- 172.21.15.32:0/3230103938 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f146c000c90 con 0x7f1478095260 2021-04-09T18:34:17.698 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:17.680 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f147810d480 con 0x7f1478095260 2021-04-09T18:34:18.697 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.679 7f14677fe700 1 -- 172.21.15.32:0/3230103938 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v426) v1 ==== 118+0+0 (secure 0 0 0) 0x7f146c00a000 con 0x7f1478095260 2021-04-09T18:34:18.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f145c0188c0 msgr2=0x7f145c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:18.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f145c0188c0 0x7f145c01adc0 crc :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1478095260 msgr2=0x7f14781a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:18.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1478095260 0x7f14781a0c50 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7f146c00b5e0 tx=0x7f146c00b980).stop 2021-04-09T18:34:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 shutdown_connections 2021-04-09T18:34:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f145c0188c0 0x7f145c01adc0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1478095260 0x7f14781a0c50 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14781a1270 0x7f14780b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 --2- 172.21.15.32:0/3230103938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14780c3c00 0x7f147809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 >> 172.21.15.32:0/3230103938 conn(0x7f14780975f0 msgr2=0x7f1478096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:18.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 shutdown_connections 2021-04-09T18:34:18.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.700 7f147d1a3700 1 -- 172.21.15.32:0/3230103938 wait complete. 2021-04-09T18:34:18.730 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:18.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.867 7f7d59eaa700 1 Processor -- start 2021-04-09T18:34:18.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.867 7f7d59eaa700 1 -- start start 2021-04-09T18:34:18.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.867 7f7d59eaa700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7d540d7ca0 0x7f7d540d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:18.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d59eaa700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d5410da00 0x7f7d5410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:18.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d59eaa700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5410d480 con 0x7f7d5410d010 2021-04-09T18:34:18.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d59eaa700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d540d9220 con 0x7f7d5410da00 2021-04-09T18:34:18.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d59eaa700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d540d86c0 con 0x7f7d540d7ca0 2021-04-09T18:34:18.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d53fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7d5410da00 0x7f7d5410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:18.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d537fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7d5410d010 0x7f7d540d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41014/0 (socket says 172.21.15.32:41014) 2021-04-09T18:34:18.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d537fe700 1 -- 172.21.15.32:0/3925821075 learned_addr learned my addr 172.21.15.32:0/3925821075 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:18.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d52ffd700 1 --2- 172.21.15.32:0/3925821075 >> v2:172.21.15.86:3300/0 conn(0x7f7d540d7ca0 0x7f7d540d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:18.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d53fff700 1 -- 172.21.15.32:0/3925821075 >> v2:172.21.15.86:3300/0 conn(0x7f7d540d7ca0 msgr2=0x7f7d540d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:18.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d53fff700 1 --2- 172.21.15.32:0/3925821075 >> v2:172.21.15.86:3300/0 conn(0x7f7d540d7ca0 0x7f7d540d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d53fff700 1 -- 172.21.15.32:0/3925821075 >> v1:172.21.15.32:6789/0 conn(0x7f7d5410d010 legacy=0x7f7d540d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:18.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d53fff700 1 -- 172.21.15.32:0/3925821075 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d540d1f70 con 0x7f7d5410da00 2021-04-09T18:34:18.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d527fc700 1 -- 172.21.15.32:0/3925821075 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619094665 0 0) 0x7f7d44000ce0 con 0x7f7d5410d010 2021-04-09T18:34:18.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d53fff700 1 --2- 172.21.15.32:0/3925821075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d5410da00 0x7f7d5410de00 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f7d4c006e30 tx=0x7f7d4c009e60).ready entity=mon.0 client_cookie=bc6694aa73bd073a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:18.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d527fc700 1 -- 172.21.15.32:0/3925821075 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4c006a50 con 0x7f7d5410da00 2021-04-09T18:34:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d527fc700 1 -- 172.21.15.32:0/3925821075 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c00b300 con 0x7f7d5410da00 2021-04-09T18:34:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d527fc700 1 -- 172.21.15.32:0/3925821075 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4c00b750 con 0x7f7d5410da00 2021-04-09T18:34:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d59eaa700 1 -- 172.21.15.32:0/3925821075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d5410da00 msgr2=0x7f7d5410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:18.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.868 7f7d59eaa700 1 --2- 172.21.15.32:0/3925821075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d5410da00 0x7f7d5410de00 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f7d4c006e30 tx=0x7f7d4c009e60).stop 2021-04-09T18:34:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.869 7f7d59eaa700 1 -- 172.21.15.32:0/3925821075 shutdown_connections 2021-04-09T18:34:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.869 7f7d59eaa700 1 --2- 172.21.15.32:0/3925821075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d5410da00 0x7f7d5410de00 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.869 7f7d59eaa700 1 --2- 172.21.15.32:0/3925821075 >> v2:172.21.15.86:3300/0 conn(0x7f7d540d7ca0 0x7f7d540d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.869 7f7d59eaa700 1 -- 172.21.15.32:0/3925821075 >> 172.21.15.32:0/3925821075 conn(0x7f7d540ce5f0 msgr2=0x7f7d540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:18.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.869 7f7d59eaa700 1 -- 172.21.15.32:0/3925821075 shutdown_connections 2021-04-09T18:34:18.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.869 7f7d59eaa700 1 -- 172.21.15.32:0/3925821075 wait complete. 2021-04-09T18:34:18.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.889 7f7d59eaa700 1 Processor -- start 2021-04-09T18:34:18.922 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.889 7f7d59eaa700 1 -- start start 2021-04-09T18:34:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d59eaa700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d54184e20 0x7f7d541857e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d59eaa700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d54187ac0 0x7f7d54185e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d59eaa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d5408ea60 0x7f7d5408ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:18.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d59eaa700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d54185260 con 0x7f7d54187ac0 2021-04-09T18:34:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d59eaa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540ac330 con 0x7f7d5408ea60 2021-04-09T18:34:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d59eaa700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540ac510 con 0x7f7d54184e20 2021-04-09T18:34:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d52ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d54187ac0 0x7f7d54185e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d52ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d54187ac0 0x7f7d54185e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59662/0 (socket says 172.21.15.32:59662) 2021-04-09T18:34:18.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d52ffd700 1 -- 172.21.15.32:0/3461610259 learned_addr learned my addr 172.21.15.32:0/3461610259 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d53fff700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d5408ea60 0x7f7d5408ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d537fe700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d54184e20 0x7f7d541857e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d52ffd700 1 -- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d54184e20 msgr2=0x7f7d541857e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:18.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d52ffd700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d54184e20 0x7f7d541857e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d52ffd700 1 -- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d5408ea60 msgr2=0x7f7d5408ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:18.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d52ffd700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d5408ea60 0x7f7d5408ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:18.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d52ffd700 1 -- 172.21.15.32:0/3461610259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5408f4c0 con 0x7f7d54187ac0 2021-04-09T18:34:18.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.890 7f7d52ffd700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d54187ac0 0x7f7d54185e00 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f7d3c006f60 tx=0x7f7d3c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:18.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.891 7f7d4bfff700 1 -- 172.21.15.32:0/3461610259 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3c00bd60 con 0x7f7d54187ac0 2021-04-09T18:34:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.891 7f7d4bfff700 1 -- 172.21.15.32:0/3461610259 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c000c90 con 0x7f7d54187ac0 2021-04-09T18:34:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.891 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5408f4c0 con 0x7f7d54187ac0 2021-04-09T18:34:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.891 7f7d4bfff700 1 -- 172.21.15.32:0/3461610259 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3c002220 con 0x7f7d54187ac0 2021-04-09T18:34:18.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.891 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5408f4c0 con 0x7f7d54187ac0 2021-04-09T18:34:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.891 7f7d4bfff700 1 -- 172.21.15.32:0/3461610259 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7d3c0052e0 con 0x7f7d54187ac0 2021-04-09T18:34:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.891 7f7d4bfff700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d38018930 0x7f7d3801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.891 7f7d4bfff700 1 -- 172.21.15.32:0/3461610259 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7d3c0329f0 con 0x7f7d54187ac0 2021-04-09T18:34:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.892 7f7d537fe700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d38018930 0x7f7d3801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:18.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.892 7f7d537fe700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d38018930 0x7f7d3801ae10 crc :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:18.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.912 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5410d480 con 0x7f7d54187ac0 2021-04-09T18:34:18.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:18.915 7f7d4bfff700 1 -- 172.21.15.32:0/3461610259 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7d3c000c90 con 0x7f7d54187ac0 2021-04-09T18:34:19.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.107 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d5410d480 con 0x7f7d54187ac0 2021-04-09T18:34:19.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.107 7f7d4bfff700 1 -- 172.21.15.32:0/3461610259 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 426 v426) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f7d3c006b00 con 0x7f7d54187ac0 2021-04-09T18:34:19.126 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 426 2021-04-09T18:34:19.126 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:19.126 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":426,"default_fscid":22,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/368655383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":368655383},{"type":"v1","addr":"172.21.15.32:6834","nonce":368655383}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/4251960201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":4251960201},{"type":"v1","addr":"172.21.15.86:6836","nonce":4251960201}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59951,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1668354365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1668354365},{"type":"v1","addr":"172.21.15.86:6837","nonce":1668354365}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422}],"filesystems":[{"mdsmap":{"epoch":426,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:34:16.431259","modified":"2021-04-09 18:34:18.673793","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12379":{"gid":12379,"name":"a","rank":0,"incarnation":423,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2610523978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2610523978},{"type":"v1","addr":"172.21.15.32:6835","nonce":2610523978}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[45],"metadata_pool":44,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":22}]} 2021-04-09T18:34:19.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d38018930 msgr2=0x7f7d3801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d38018930 0x7f7d3801ae10 crc :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d54187ac0 msgr2=0x7f7d54185e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d54187ac0 0x7f7d54185e00 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f7d3c006f60 tx=0x7f7d3c00b970).stop 2021-04-09T18:34:19.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 shutdown_connections 2021-04-09T18:34:19.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7d38018930 0x7f7d3801ae10 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d54187ac0 0x7f7d54185e00 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d5408ea60 0x7f7d5408ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 --2- 172.21.15.32:0/3461610259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d54184e20 0x7f7d541857e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 >> 172.21.15.32:0/3461610259 conn(0x7f7d541a9360 msgr2=0x7f7d541918a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:19.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 shutdown_connections 2021-04-09T18:34:19.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.110 7f7d59eaa700 1 -- 172.21.15.32:0/3461610259 wait complete. 2021-04-09T18:34:19.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:34:19.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.269 7fd6d181d700 1 Processor -- start 2021-04-09T18:34:19.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.269 7fd6d181d700 1 -- start start 2021-04-09T18:34:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6d181d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc10d010 0x7fd6cc0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6d181d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd6cc0d1da0 0x7fd6cc0d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6d181d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc10d480 con 0x7fd6cc0d82c0 2021-04-09T18:34:19.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6d181d700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc10e780 con 0x7fd6cc0d1da0 2021-04-09T18:34:19.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6d181d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc0d8d30 con 0x7fd6cc10d010 2021-04-09T18:34:19.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cad73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd6cc0d1da0 0x7fd6cc0d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cbd75700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd6cc0d82c0 0x7fd6cc10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41028/0 (socket says 172.21.15.32:41028) 2021-04-09T18:34:19.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cbd75700 1 -- 172.21.15.32:0/3281344534 learned_addr learned my addr 172.21.15.32:0/3281344534 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:19.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cb574700 1 --2- 172.21.15.32:0/3281344534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc10d010 0x7fd6cc0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cad73700 1 -- 172.21.15.32:0/3281344534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc10d010 msgr2=0x7fd6cc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:19.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cad73700 1 --2- 172.21.15.32:0/3281344534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc10d010 0x7fd6cc0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cad73700 1 -- 172.21.15.32:0/3281344534 >> v1:172.21.15.32:6789/0 conn(0x7fd6cc0d82c0 legacy=0x7fd6cc10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6ca572700 1 -- 172.21.15.32:0/3281344534 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624967896 0 0) 0x7fd6c4000d30 con 0x7fd6cc0d82c0 2021-04-09T18:34:19.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cad73700 1 -- 172.21.15.32:0/3281344534 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc0d9220 con 0x7fd6cc0d1da0 2021-04-09T18:34:19.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cb574700 1 --2- 172.21.15.32:0/3281344534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc10d010 0x7fd6cc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:19.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6cad73700 1 --2- 172.21.15.32:0/3281344534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc0d1da0 0x7fd6cc0d7ca0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7fd6c0006650 tx=0x7fd6c0009a80).ready entity=mon.0 client_cookie=9840ea3fbaa4de38 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:19.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6ca572700 1 -- 172.21.15.32:0/3281344534 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c00009b0 con 0x7fd6cc0d1da0 2021-04-09T18:34:19.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6ca572700 1 -- 172.21.15.32:0/3281344534 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c000c0c0 con 0x7fd6cc0d1da0 2021-04-09T18:34:19.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.270 7fd6ca572700 1 -- 172.21.15.32:0/3281344534 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c000c510 con 0x7fd6cc0d1da0 2021-04-09T18:34:19.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.271 7fd6d181d700 1 -- 172.21.15.32:0/3281344534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc0d1da0 msgr2=0x7fd6cc0d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.271 7fd6d181d700 1 --2- 172.21.15.32:0/3281344534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc0d1da0 0x7fd6cc0d7ca0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7fd6c0006650 tx=0x7fd6c0009a80).stop 2021-04-09T18:34:19.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.271 7fd6d181d700 1 -- 172.21.15.32:0/3281344534 shutdown_connections 2021-04-09T18:34:19.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.271 7fd6d181d700 1 --2- 172.21.15.32:0/3281344534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc0d1da0 0x7fd6cc0d7ca0 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.271 7fd6d181d700 1 --2- 172.21.15.32:0/3281344534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc10d010 0x7fd6cc0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.271 7fd6d181d700 1 -- 172.21.15.32:0/3281344534 >> 172.21.15.32:0/3281344534 conn(0x7fd6cc0ce5f0 msgr2=0x7fd6cc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:19.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.271 7fd6d181d700 1 -- 172.21.15.32:0/3281344534 shutdown_connections 2021-04-09T18:34:19.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.271 7fd6d181d700 1 -- 172.21.15.32:0/3281344534 wait complete. 2021-04-09T18:34:19.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.291 7fd6d181d700 1 Processor -- start 2021-04-09T18:34:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.291 7fd6d181d700 1 -- start start 2021-04-09T18:34:19.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6d181d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1a91e0 0x7fd6cc0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6d181d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc1a95e0 0x7fd6cc0a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6d181d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac850 0x7fd6cc1acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6d181d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc1ad270 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6d181d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc0b7640 con 0x7fd6cc1a91e0 2021-04-09T18:34:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6d181d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc183c80 con 0x7fd6cc1ac850 2021-04-09T18:34:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc1a95e0 0x7fd6cc0a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cad73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc1a95e0 0x7fd6cc0a5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59676/0 (socket says 172.21.15.32:59676) 2021-04-09T18:34:19.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cad73700 1 -- 172.21.15.32:0/1195598444 learned_addr learned my addr 172.21.15.32:0/1195598444 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cbd75700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac850 0x7fd6cc1acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cb574700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1a91e0 0x7fd6cc0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cad73700 1 -- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac850 msgr2=0x7fd6cc1acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cad73700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac850 0x7fd6cc1acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cad73700 1 -- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1a91e0 msgr2=0x7fd6cc0a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cad73700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1a91e0 0x7fd6cc0a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.292 7fd6cad73700 1 -- 172.21.15.32:0/1195598444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc183ea0 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.293 7fd6cad73700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc1a95e0 0x7fd6cc0a5080 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 rx=0x7fd6c0009650 tx=0x7fd6c000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.293 7fd6b3fff700 1 -- 172.21.15.32:0/1195598444 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c00093a0 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.293 7fd6b3fff700 1 -- 172.21.15.32:0/1195598444 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c0006070 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.293 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc183ea0 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.293 7fd6b3fff700 1 -- 172.21.15.32:0/1195598444 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6c00052e0 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.293 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc183ea0 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.293 7fd6b3fff700 1 -- 172.21.15.32:0/1195598444 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd6c0009300 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.294 7fd6b3fff700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6ac018930 0x7fd6ac01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.294 7fd6b3fff700 1 -- 172.21.15.32:0/1195598444 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd6c0032a10 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.294 7fd6cb574700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6ac018930 0x7fd6ac01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.333 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.294 7fd6cb574700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6ac018930 0x7fd6ac01ae10 crc :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:19.334 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.314 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc182d70 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.317 7fd6b3fff700 1 -- 172.21.15.32:0/1195598444 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd6c0006030 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.507 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6cc10d480 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.508 7fd6b3fff700 1 -- 172.21.15.32:0/1195598444 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v263) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fd6c0000940 con 0x7fd6cc1a95e0 2021-04-09T18:34:19.528 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:19.528 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":263,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:34:17.432629","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":45,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":44,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:34:13.713911","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":45,"pool_name":"cephfs_data","create_time":"2021-04-09 18:34:15.142274","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:34:19.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6ac018930 msgr2=0x7fd6ac01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6ac018930 0x7fd6ac01ae10 crc :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc1a95e0 msgr2=0x7fd6cc0a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc1a95e0 0x7fd6cc0a5080 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 rx=0x7fd6c0009650 tx=0x7fd6c000b8e0).stop 2021-04-09T18:34:19.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 shutdown_connections 2021-04-09T18:34:19.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd6ac018930 0x7fd6ac01ae10 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6cc1a95e0 0x7fd6cc0a5080 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6cc1ac850 0x7fd6cc1acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 --2- 172.21.15.32:0/1195598444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6cc1a91e0 0x7fd6cc0a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 >> 172.21.15.32:0/1195598444 conn(0x7fd6cc19b460 msgr2=0x7fd6cc19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 shutdown_connections 2021-04-09T18:34:19.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.510 7fd6d181d700 1 -- 172.21.15.32:0/1195598444 wait complete. 2021-04-09T18:34:19.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:34:19.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.682 7fb98f3e2700 1 Processor -- start 2021-04-09T18:34:19.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.683 7fb98f3e2700 1 -- start start 2021-04-09T18:34:19.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.683 7fb98f3e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880d2700 0x7fb9880cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.683 7fb98f3e2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb9880f28c0 0x7fb9880d7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.683 7fb98f3e2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9880d8700 con 0x7fb9880d8270 2021-04-09T18:34:19.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.683 7fb98f3e2700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9880d91b0 con 0x7fb9880f28c0 2021-04-09T18:34:19.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.683 7fb98f3e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881b0ce0 con 0x7fb9880d2700 2021-04-09T18:34:19.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98d9ff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb9880d8270 0x7fb9881b04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:38968/0 (socket says 172.21.15.32:38968) 2021-04-09T18:34:19.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98d1fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880d2700 0x7fb9880cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98d9ff700 1 -- 172.21.15.32:0/1222496615 learned_addr learned my addr 172.21.15.32:0/1222496615 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.683 7fb98c9fd700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb9880f28c0 0x7fb9880d7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb97ffff700 1 -- 172.21.15.32:0/1222496615 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149573399 0 0) 0x7fb980001440 con 0x7fb9880d8270 2021-04-09T18:34:19.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb97ffff700 1 -- 172.21.15.32:0/1222496615 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb974002fe0 con 0x7fb9880d8270 2021-04-09T18:34:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98d1fe700 1 -- 172.21.15.32:0/1222496615 >> v2:172.21.15.86:3300/0 conn(0x7fb9880f28c0 msgr2=0x7fb9880d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98d1fe700 1 --2- 172.21.15.32:0/1222496615 >> v2:172.21.15.86:3300/0 conn(0x7fb9880f28c0 0x7fb9880d7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98d1fe700 1 -- 172.21.15.32:0/1222496615 >> v1:172.21.15.86:6789/0 conn(0x7fb9880d8270 legacy=0x7fb9881b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98d1fe700 1 -- 172.21.15.32:0/1222496615 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb988112090 con 0x7fb9880d2700 2021-04-09T18:34:19.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98c9fd700 1 --2- 172.21.15.32:0/1222496615 >> v2:172.21.15.86:3300/0 conn(0x7fb9880f28c0 0x7fb9880d7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:19.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98d1fe700 1 --2- 172.21.15.32:0/1222496615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880d2700 0x7fb9880cfeb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb984006650 tx=0x7fb984009a80).ready entity=mon.2 client_cookie=bb2d2e1239a59609 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:19.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb97ffff700 1 -- 172.21.15.32:0/1222496615 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9840009b0 con 0x7fb9880d2700 2021-04-09T18:34:19.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb97ffff700 1 -- 172.21.15.32:0/1222496615 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98400c0c0 con 0x7fb9880d2700 2021-04-09T18:34:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb97ffff700 1 -- 172.21.15.32:0/1222496615 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb98400c510 con 0x7fb9880d2700 2021-04-09T18:34:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98f3e2700 1 -- 172.21.15.32:0/1222496615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880d2700 msgr2=0x7fb9880cfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.684 7fb98f3e2700 1 --2- 172.21.15.32:0/1222496615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880d2700 0x7fb9880cfeb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb984006650 tx=0x7fb984009a80).stop 2021-04-09T18:34:19.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.685 7fb98f3e2700 1 -- 172.21.15.32:0/1222496615 shutdown_connections 2021-04-09T18:34:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.685 7fb98f3e2700 1 --2- 172.21.15.32:0/1222496615 >> v2:172.21.15.86:3300/0 conn(0x7fb9880f28c0 0x7fb9880d7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.685 7fb98f3e2700 1 --2- 172.21.15.32:0/1222496615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880d2700 0x7fb9880cfeb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.685 7fb98f3e2700 1 -- 172.21.15.32:0/1222496615 >> 172.21.15.32:0/1222496615 conn(0x7fb9880ce590 msgr2=0x7fb9880cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:19.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.685 7fb98f3e2700 1 -- 172.21.15.32:0/1222496615 shutdown_connections 2021-04-09T18:34:19.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.685 7fb98f3e2700 1 -- 172.21.15.32:0/1222496615 wait complete. 2021-04-09T18:34:19.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.705 7fb98f3e2700 1 Processor -- start 2021-04-09T18:34:19.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98f3e2700 1 -- start start 2021-04-09T18:34:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98f3e2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9880912e0 0x7fb9881adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98f3e2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a87f0 0x7fb9881ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98f3e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9881986f0 0x7fb9881abc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98f3e2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881aea60 con 0x7fb9880912e0 2021-04-09T18:34:19.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98f3e2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881ac220 con 0x7fb9881986f0 2021-04-09T18:34:19.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98f3e2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881ac400 con 0x7fb9880a87f0 2021-04-09T18:34:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98d1fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9880912e0 0x7fb9881adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98d1fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9880912e0 0x7fb9881adda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59692/0 (socket says 172.21.15.32:59692) 2021-04-09T18:34:19.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98d1fe700 1 -- 172.21.15.32:0/3414300518 learned_addr learned my addr 172.21.15.32:0/3414300518 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98c9fd700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a87f0 0x7fb9881ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.706 7fb98d9ff700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9881986f0 0x7fb9881abc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb98d1fe700 1 -- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a87f0 msgr2=0x7fb9881ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb98d1fe700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a87f0 0x7fb9881ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb98d1fe700 1 -- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9881986f0 msgr2=0x7fb9881abc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb98d1fe700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9881986f0 0x7fb9881abc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb98d1fe700 1 -- 172.21.15.32:0/3414300518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9881ac620 con 0x7fb9880912e0 2021-04-09T18:34:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb98c9fd700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a87f0 0x7fb9881ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb98d1fe700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9880912e0 0x7fb9881adda0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7fb984009650 tx=0x7fb98400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb97dffb700 1 -- 172.21.15.32:0/3414300518 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb98400bd50 con 0x7fb9880912e0 2021-04-09T18:34:19.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb97dffb700 1 -- 172.21.15.32:0/3414300518 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb984006030 con 0x7fb9880912e0 2021-04-09T18:34:19.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb97dffb700 1 -- 172.21.15.32:0/3414300518 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb984005560 con 0x7fb9880912e0 2021-04-09T18:34:19.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9881ac620 con 0x7fb9880912e0 2021-04-09T18:34:19.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.707 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9881ac620 con 0x7fb9880912e0 2021-04-09T18:34:19.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.708 7fb97dffb700 1 -- 172.21.15.32:0/3414300518 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb984005700 con 0x7fb9880912e0 2021-04-09T18:34:19.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.708 7fb97dffb700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:19.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.708 7fb97dffb700 1 -- 172.21.15.32:0/3414300518 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb984032a00 con 0x7fb9880912e0 2021-04-09T18:34:19.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.708 7fb98c9fd700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:19.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.708 7fb98c9fd700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 crc :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:19.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.727 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb988103970 con 0x7fb9880912e0 2021-04-09T18:34:19.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.730 7fb97dffb700 1 -- 172.21.15.32:0/3414300518 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb9840054f0 con 0x7fb9880912e0 2021-04-09T18:34:19.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.916 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb988103970 con 0x7fb9880912e0 2021-04-09T18:34:19.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.919 7fb97dffb700 1 -- 172.21.15.32:0/3414300518 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v49) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb984006030 con 0x7fb9880912e0 2021-04-09T18:34:19.937 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:34:19.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 msgr2=0x7fb96801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 crc :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9880912e0 msgr2=0x7fb9881adda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:19.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9880912e0 0x7fb9881adda0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7fb984009650 tx=0x7fb98400b960).stop 2021-04-09T18:34:19.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 shutdown_connections 2021-04-09T18:34:19.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb968018930 0x7fb96801ae10 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9881986f0 0x7fb9881abc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a87f0 0x7fb9881ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 --2- 172.21.15.32:0/3414300518 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb9880912e0 0x7fb9881adda0 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:19.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.921 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 >> 172.21.15.32:0/3414300518 conn(0x7fb988092120 msgr2=0x7fb988103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:19.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.922 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 shutdown_connections 2021-04-09T18:34:19.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:19.922 7fb98f3e2700 1 -- 172.21.15.32:0/3414300518 wait complete. 2021-04-09T18:34:19.949 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:34:20.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.093 7f87ea1e7700 1 Processor -- start 2021-04-09T18:34:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.093 7f87ea1e7700 1 -- start start 2021-04-09T18:34:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.093 7f87ea1e7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40cffb0 0x7f87e4087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.093 7f87ea1e7700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87e40d7c50 0x7f87e41b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.093 7f87ea1e7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e4088460 con 0x7f87e40d8050 2021-04-09T18:34:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.093 7f87ea1e7700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e40d91f0 con 0x7f87e40d7c50 2021-04-09T18:34:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.093 7f87ea1e7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40d8620 con 0x7f87e40cffb0 2021-04-09T18:34:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.093 7f87dbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87e40d7c50 0x7f87e41b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87dbfff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f87e40d7c50 0x7f87e41b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59696/0 (socket says 172.21.15.32:59696) 2021-04-09T18:34:20.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87dbfff700 1 -- 172.21.15.32:0/2444532987 learned_addr learned my addr 172.21.15.32:0/2444532987 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:20.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87e3fff700 1 --2- 172.21.15.32:0/2444532987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40cffb0 0x7f87e4087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87dbfff700 1 -- 172.21.15.32:0/2444532987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40cffb0 msgr2=0x7f87e4087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:20.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87dbfff700 1 --2- 172.21.15.32:0/2444532987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40cffb0 0x7f87e4087e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87dbfff700 1 -- 172.21.15.32:0/2444532987 >> v1:172.21.15.32:6789/0 conn(0x7f87e40d8050 legacy=0x7f87e410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87e37fe700 1 -- 172.21.15.32:0/2444532987 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483279389 0 0) 0x7f87dc000f90 con 0x7f87e40d8050 2021-04-09T18:34:20.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87dbfff700 1 -- 172.21.15.32:0/2444532987 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e40d2800 con 0x7f87e40d7c50 2021-04-09T18:34:20.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87e3fff700 1 --2- 172.21.15.32:0/2444532987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40cffb0 0x7f87e4087e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:34:20.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87dbfff700 1 --2- 172.21.15.32:0/2444532987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e40d7c50 0x7f87e41b0490 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 rx=0x7f87cc006700 tx=0x7f87cc009b80).ready entity=mon.0 client_cookie=f9adc79619618740 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:20.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87e37fe700 1 -- 172.21.15.32:0/2444532987 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87cc00c0c0 con 0x7f87e40d7c50 2021-04-09T18:34:20.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87e37fe700 1 -- 172.21.15.32:0/2444532987 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc00c440 con 0x7f87e40d7c50 2021-04-09T18:34:20.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87e37fe700 1 -- 172.21.15.32:0/2444532987 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87cc00c920 con 0x7f87e40d7c50 2021-04-09T18:34:20.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87ea1e7700 1 -- 172.21.15.32:0/2444532987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e40d7c50 msgr2=0x7f87e41b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.094 7f87ea1e7700 1 --2- 172.21.15.32:0/2444532987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e40d7c50 0x7f87e41b0490 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 rx=0x7f87cc006700 tx=0x7f87cc009b80).stop 2021-04-09T18:34:20.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.095 7f87ea1e7700 1 -- 172.21.15.32:0/2444532987 shutdown_connections 2021-04-09T18:34:20.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.095 7f87ea1e7700 1 --2- 172.21.15.32:0/2444532987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e40d7c50 0x7f87e41b0490 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.095 7f87ea1e7700 1 --2- 172.21.15.32:0/2444532987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40cffb0 0x7f87e4087e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.095 7f87ea1e7700 1 -- 172.21.15.32:0/2444532987 >> 172.21.15.32:0/2444532987 conn(0x7f87e40ce5b0 msgr2=0x7f87e40cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:20.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.095 7f87ea1e7700 1 -- 172.21.15.32:0/2444532987 shutdown_connections 2021-04-09T18:34:20.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.095 7f87ea1e7700 1 -- 172.21.15.32:0/2444532987 wait complete. 2021-04-09T18:34:20.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.115 7f87ea1e7700 1 Processor -- start 2021-04-09T18:34:20.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.115 7f87ea1e7700 1 -- start start 2021-04-09T18:34:20.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87ea1e7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40b1950 0x7f87e4183e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87ea1e7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e41795e0 0x7f87e4094eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87ea1e7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e41799e0 0x7f87e40954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87ea1e7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40b13d0 con 0x7f87e41799e0 2021-04-09T18:34:20.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87ea1e7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40b1d50 con 0x7f87e41795e0 2021-04-09T18:34:20.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87ea1e7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e419e4a0 con 0x7f87e40b1950 2021-04-09T18:34:20.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e8804700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e41799e0 0x7f87e40954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e8804700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e41799e0 0x7f87e40954d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59700/0 (socket says 172.21.15.32:59700) 2021-04-09T18:34:20.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e8804700 1 -- 172.21.15.32:0/1392871785 learned_addr learned my addr 172.21.15.32:0/1392871785 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:20.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87dbfff700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e41795e0 0x7f87e4094eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e3fff700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40b1950 0x7f87e4183e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e8804700 1 -- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40b1950 msgr2=0x7f87e4183e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e8804700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40b1950 0x7f87e4183e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e8804700 1 -- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e41795e0 msgr2=0x7f87e4094eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e8804700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e41795e0 0x7f87e4094eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e8804700 1 -- 172.21.15.32:0/1392871785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e40b0180 con 0x7f87e41799e0 2021-04-09T18:34:20.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e8804700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e41799e0 0x7f87e40954d0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f87dc009980 tx=0x7f87dc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:20.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.116 7f87e17fa700 1 -- 172.21.15.32:0/1392871785 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87dc005870 con 0x7f87e41799e0 2021-04-09T18:34:20.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.117 7f87e17fa700 1 -- 172.21.15.32:0/1392871785 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc005a70 con 0x7f87e41799e0 2021-04-09T18:34:20.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.117 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e40b0180 con 0x7f87e41799e0 2021-04-09T18:34:20.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.117 7f87e17fa700 1 -- 172.21.15.32:0/1392871785 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87dc006030 con 0x7f87e41799e0 2021-04-09T18:34:20.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.117 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e40b0180 con 0x7f87e41799e0 2021-04-09T18:34:20.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.117 7f87e17fa700 1 -- 172.21.15.32:0/1392871785 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f87dc005c50 con 0x7f87e41799e0 2021-04-09T18:34:20.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.117 7f87e17fa700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87c80188d0 0x7f87c801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.118 7f87e17fa700 1 -- 172.21.15.32:0/1392871785 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f87dc032d70 con 0x7f87e41799e0 2021-04-09T18:34:20.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.118 7f87e3fff700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87c80188d0 0x7f87c801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.118 7f87e3fff700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87c80188d0 0x7f87c801add0 crc :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:20.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.137 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e4093fc0 con 0x7f87e41799e0 2021-04-09T18:34:20.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.141 7f87e17fa700 1 -- 172.21.15.32:0/1392871785 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f87dc00be10 con 0x7f87e41799e0 2021-04-09T18:34:20.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.325 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f87e40d8620 con 0x7f87e41799e0 2021-04-09T18:34:20.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.328 7f87e17fa700 1 -- 172.21.15.32:0/1392871785 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v50) v1 ==== 184+0+0 (secure 0 0 0) 0x7f87dc005a70 con 0x7f87e41799e0 2021-04-09T18:34:20.350 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:34:20.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87c80188d0 msgr2=0x7f87c801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87c80188d0 0x7f87c801add0 crc :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e41799e0 msgr2=0x7f87e40954d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e41799e0 0x7f87e40954d0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f87dc009980 tx=0x7f87dc00bcd0).stop 2021-04-09T18:34:20.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 shutdown_connections 2021-04-09T18:34:20.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f87c80188d0 0x7f87c801add0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e41799e0 0x7f87e40954d0 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e41795e0 0x7f87e4094eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 --2- 172.21.15.32:0/1392871785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e40b1950 0x7f87e4183e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 >> 172.21.15.32:0/1392871785 conn(0x7f87e41889b0 msgr2=0x7f87e4189210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:20.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 shutdown_connections 2021-04-09T18:34:20.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.334 7f87ea1e7700 1 -- 172.21.15.32:0/1392871785 wait complete. 2021-04-09T18:34:20.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:20.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.492 7fddcd9ea700 1 Processor -- start 2021-04-09T18:34:20.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.492 7fddcd9ea700 1 -- start start 2021-04-09T18:34:20.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.493 7fddcd9ea700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fddc810da00 0x7fddc810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.493 7fddcd9ea700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc810d480 con 0x7fddc810d010 2021-04-09T18:34:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.493 7fddcd9ea700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc80d1f70 con 0x7fddc80d7ca0 2021-04-09T18:34:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.493 7fddcd9ea700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc80d8d30 con 0x7fddc810da00 2021-04-09T18:34:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.493 7fddc77fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fddc810d010 0x7fddc80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41066/0 (socket says 172.21.15.32:41066) 2021-04-09T18:34:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.493 7fddc77fe700 1 -- 172.21.15.32:0/2952953835 learned_addr learned my addr 172.21.15.32:0/2952953835 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.493 7fddc7fff700 1 --2- 172.21.15.32:0/2952953835 >> v2:172.21.15.86:3300/0 conn(0x7fddc810da00 0x7fddc810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.493 7fddc67fc700 1 -- 172.21.15.32:0/2952953835 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178255404 0 0) 0x7fddb8000ce0 con 0x7fddc810d010 2021-04-09T18:34:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.493 7fddc67fc700 1 -- 172.21.15.32:0/2952953835 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb0002fe0 con 0x7fddc810d010 2021-04-09T18:34:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc67fc700 1 -- 172.21.15.32:0/2952953835 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024672171 0 0) 0x7fddbc001080 con 0x7fddc80d7ca0 2021-04-09T18:34:20.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc67fc700 1 -- 172.21.15.32:0/2952953835 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb0003210 con 0x7fddc80d7ca0 2021-04-09T18:34:20.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc67fc700 1 -- 172.21.15.32:0/2952953835 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 504339829 0 0) 0x7fddb80010e0 con 0x7fddc810d010 2021-04-09T18:34:20.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc7fff700 1 -- 172.21.15.32:0/2952953835 >> v1:172.21.15.86:6789/0 conn(0x7fddc80d7ca0 legacy=0x7fddc80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc7fff700 1 -- 172.21.15.32:0/2952953835 >> v1:172.21.15.32:6789/0 conn(0x7fddc810d010 legacy=0x7fddc80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc7fff700 1 -- 172.21.15.32:0/2952953835 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc80d9220 con 0x7fddc810da00 2021-04-09T18:34:20.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc7fff700 1 --2- 172.21.15.32:0/2952953835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc810da00 0x7fddc810de00 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7fddc0006e50 tx=0x7fddc0009df0).ready entity=mon.1 client_cookie=3c63569cca0a21a0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:20.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc67fc700 1 -- 172.21.15.32:0/2952953835 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddc000b510 con 0x7fddc810da00 2021-04-09T18:34:20.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc67fc700 1 -- 172.21.15.32:0/2952953835 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc0003030 con 0x7fddc810da00 2021-04-09T18:34:20.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddc67fc700 1 -- 172.21.15.32:0/2952953835 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddc000b7d0 con 0x7fddc810da00 2021-04-09T18:34:20.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddcd9ea700 1 -- 172.21.15.32:0/2952953835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc810da00 msgr2=0x7fddc810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.494 7fddcd9ea700 1 --2- 172.21.15.32:0/2952953835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc810da00 0x7fddc810de00 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7fddc0006e50 tx=0x7fddc0009df0).stop 2021-04-09T18:34:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.495 7fddcd9ea700 1 -- 172.21.15.32:0/2952953835 shutdown_connections 2021-04-09T18:34:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.495 7fddcd9ea700 1 --2- 172.21.15.32:0/2952953835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc810da00 0x7fddc810de00 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.495 7fddcd9ea700 1 -- 172.21.15.32:0/2952953835 >> 172.21.15.32:0/2952953835 conn(0x7fddc80ce5f0 msgr2=0x7fddc80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:20.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.495 7fddcd9ea700 1 -- 172.21.15.32:0/2952953835 shutdown_connections 2021-04-09T18:34:20.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.495 7fddcd9ea700 1 -- 172.21.15.32:0/2952953835 wait complete. 2021-04-09T18:34:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.515 7fddcd9ea700 1 Processor -- start 2021-04-09T18:34:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.515 7fddcd9ea700 1 -- start start 2021-04-09T18:34:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddcd9ea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddc80b42d0 0x7fddc8183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddcd9ea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc81ac090 0x7fddc81ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddcd9ea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddc8184d30 0x7fddc8185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddcd9ea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80a1a90 con 0x7fddc80b42d0 2021-04-09T18:34:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddcd9ea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc81acab0 con 0x7fddc81ac090 2021-04-09T18:34:20.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddcd9ea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc8185a00 con 0x7fddc8184d30 2021-04-09T18:34:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddc80b42d0 0x7fddc8183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc77fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddc80b42d0 0x7fddc8183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59714/0 (socket says 172.21.15.32:59714) 2021-04-09T18:34:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc77fe700 1 -- 172.21.15.32:0/834516563 learned_addr learned my addr 172.21.15.32:0/834516563 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:20.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc6ffd700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc81ac090 0x7fddc81ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc7fff700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddc8184d30 0x7fddc8185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc77fe700 1 -- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddc8184d30 msgr2=0x7fddc8185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc77fe700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddc8184d30 0x7fddc8185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc77fe700 1 -- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc81ac090 msgr2=0x7fddc81ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc77fe700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc81ac090 0x7fddc81ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc77fe700 1 -- 172.21.15.32:0/834516563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc8185750 con 0x7fddc80b42d0 2021-04-09T18:34:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc6ffd700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc81ac090 0x7fddc81ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:34:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc7fff700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddc8184d30 0x7fddc8185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:34:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddc77fe700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddc80b42d0 0x7fddc8183520 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7fddb8000f90 tx=0x7fddb800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.516 7fddaffff700 1 -- 172.21.15.32:0/834516563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddb8001a30 con 0x7fddc80b42d0 2021-04-09T18:34:20.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.517 7fddaffff700 1 -- 172.21.15.32:0/834516563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb80043a0 con 0x7fddc80b42d0 2021-04-09T18:34:20.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.517 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc8185750 con 0x7fddc80b42d0 2021-04-09T18:34:20.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.517 7fddaffff700 1 -- 172.21.15.32:0/834516563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddb8004880 con 0x7fddc80b42d0 2021-04-09T18:34:20.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.517 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc8185750 con 0x7fddc80b42d0 2021-04-09T18:34:20.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.517 7fddaffff700 1 -- 172.21.15.32:0/834516563 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fddb8004580 con 0x7fddc80b42d0 2021-04-09T18:34:20.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.517 7fddaffff700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdda8018930 0x7fdda801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:20.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.517 7fddaffff700 1 -- 172.21.15.32:0/834516563 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fddb80329f0 con 0x7fddc80b42d0 2021-04-09T18:34:20.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.518 7fddc6ffd700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdda8018930 0x7fdda801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:20.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.518 7fddc6ffd700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdda8018930 0x7fdda801ae10 crc :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:20.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.537 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc8182d10 con 0x7fddc80b42d0 2021-04-09T18:34:20.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.541 7fddaffff700 1 -- 172.21.15.32:0/834516563 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fddb80045b0 con 0x7fddc80b42d0 2021-04-09T18:34:20.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.733 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddc810d480 con 0x7fddc80b42d0 2021-04-09T18:34:20.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.733 7fddaffff700 1 -- 172.21.15.32:0/834516563 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 426 v426) v1 ==== 95+0+2786 (secure 0 0 0) 0x7fddb80043a0 con 0x7fddc80b42d0 2021-04-09T18:34:20.752 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 426 2021-04-09T18:34:20.752 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:20.753 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":426,"default_fscid":22,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/368655383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":368655383},{"type":"v1","addr":"172.21.15.32:6834","nonce":368655383}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/4251960201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":4251960201},{"type":"v1","addr":"172.21.15.86:6836","nonce":4251960201}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59951,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1668354365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1668354365},{"type":"v1","addr":"172.21.15.86:6837","nonce":1668354365}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422}],"filesystems":[{"mdsmap":{"epoch":426,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:34:16.431259","modified":"2021-04-09 18:34:18.673793","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12379":{"gid":12379,"name":"a","rank":0,"incarnation":423,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2610523978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2610523978},{"type":"v1","addr":"172.21.15.32:6835","nonce":2610523978}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[45],"metadata_pool":44,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":22}]} 2021-04-09T18:34:20.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdda8018930 msgr2=0x7fdda801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdda8018930 0x7fdda801ae10 crc :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddc80b42d0 msgr2=0x7fddc8183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:20.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddc80b42d0 0x7fddc8183520 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7fddb8000f90 tx=0x7fddb800a940).stop 2021-04-09T18:34:20.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 shutdown_connections 2021-04-09T18:34:20.755 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fdda8018930 0x7fdda801ae10 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddc8184d30 0x7fddc8185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddc81ac090 0x7fddc81ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 --2- 172.21.15.32:0/834516563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddc80b42d0 0x7fddc8183520 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:20.756 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 >> 172.21.15.32:0/834516563 conn(0x7fddc819b890 msgr2=0x7fddc81a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:20.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 shutdown_connections 2021-04-09T18:34:20.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:20.736 7fddcd9ea700 1 -- 172.21.15.32:0/834516563 wait complete. 2021-04-09T18:34:21.167 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 426, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:18.673793', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:21.167 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:21.168 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:21.308 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:21.308 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:21.309 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 4.8589621709999999, 2021-04-09T18:34:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 426, 2021-04-09T18:34:21.310 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 263, 2021-04-09T18:34:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 263, 2021-04-09T18:34:21.311 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 10.301076594 2021-04-09T18:34:21.311 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:21.314 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.8589621709999999, "mdsmap_epoch": 426, "osdmap_epoch": 263, "osdmap_epoch_barrier": 263, "uptime": 10.301076594 } 2021-04-09T18:34:21.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:34:21.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.447 7f6f1dbe8700 1 Processor -- start 2021-04-09T18:34:21.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.447 7f6f1dbe8700 1 -- start start 2021-04-09T18:34:21.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f1dbe8700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6f1810d010 0x7f6f180d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f1dbe8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f180d1da0 0x7f6f180d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f1dbe8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f1810d480 con 0x7f6f180d82c0 2021-04-09T18:34:21.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f1dbe8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f180d8d30 con 0x7f6f180d1da0 2021-04-09T18:34:21.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f1dbe8700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f180d9220 con 0x7f6f1810d010 2021-04-09T18:34:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f16ffd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f6f180d1da0 0x7f6f180d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f17fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6f180d82c0 0x7f6f1810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41080/0 (socket says 172.21.15.32:41080) 2021-04-09T18:34:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f17fff700 1 -- 172.21.15.32:0/2182691165 learned_addr learned my addr 172.21.15.32:0/2182691165 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f177fe700 1 --2- 172.21.15.32:0/2182691165 >> v2:172.21.15.86:3300/0 conn(0x7f6f1810d010 0x7f6f180d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f16ffd700 1 -- 172.21.15.32:0/2182691165 >> v2:172.21.15.86:3300/0 conn(0x7f6f1810d010 msgr2=0x7f6f180d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:21.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f16ffd700 1 --2- 172.21.15.32:0/2182691165 >> v2:172.21.15.86:3300/0 conn(0x7f6f1810d010 0x7f6f180d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f16ffd700 1 -- 172.21.15.32:0/2182691165 >> v1:172.21.15.32:6789/0 conn(0x7f6f180d82c0 legacy=0x7f6f1810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:21.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f167fc700 1 -- 172.21.15.32:0/2182691165 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214002806 0 0) 0x7f6f10000d30 con 0x7f6f180d82c0 2021-04-09T18:34:21.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f16ffd700 1 -- 172.21.15.32:0/2182691165 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f180d86f0 con 0x7f6f180d1da0 2021-04-09T18:34:21.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f16ffd700 1 --2- 172.21.15.32:0/2182691165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f180d1da0 0x7f6f180d7ca0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7f6f0c006650 tx=0x7f6f0c009a00).ready entity=mon.0 client_cookie=7c019067255a7bdd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:21.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f167fc700 1 -- 172.21.15.32:0/2182691165 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f0c009df0 con 0x7f6f180d1da0 2021-04-09T18:34:21.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f167fc700 1 -- 172.21.15.32:0/2182691165 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0c0009b0 con 0x7f6f180d1da0 2021-04-09T18:34:21.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f167fc700 1 -- 172.21.15.32:0/2182691165 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f0c00c2b0 con 0x7f6f180d1da0 2021-04-09T18:34:21.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.448 7f6f1dbe8700 1 -- 172.21.15.32:0/2182691165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f180d1da0 msgr2=0x7f6f180d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:21.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.449 7f6f1dbe8700 1 --2- 172.21.15.32:0/2182691165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f180d1da0 0x7f6f180d7ca0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7f6f0c006650 tx=0x7f6f0c009a00).stop 2021-04-09T18:34:21.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.449 7f6f1dbe8700 1 -- 172.21.15.32:0/2182691165 shutdown_connections 2021-04-09T18:34:21.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.449 7f6f1dbe8700 1 --2- 172.21.15.32:0/2182691165 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f180d1da0 0x7f6f180d7ca0 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.449 7f6f1dbe8700 1 --2- 172.21.15.32:0/2182691165 >> v2:172.21.15.86:3300/0 conn(0x7f6f1810d010 0x7f6f180d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.449 7f6f1dbe8700 1 -- 172.21.15.32:0/2182691165 >> 172.21.15.32:0/2182691165 conn(0x7f6f180ce5f0 msgr2=0x7f6f180d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:21.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.449 7f6f1dbe8700 1 -- 172.21.15.32:0/2182691165 shutdown_connections 2021-04-09T18:34:21.495 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.449 7f6f1dbe8700 1 -- 172.21.15.32:0/2182691165 wait complete. 2021-04-09T18:34:21.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.469 7f6f1dbe8700 1 Processor -- start 2021-04-09T18:34:21.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f1dbe8700 1 -- start start 2021-04-09T18:34:21.500 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f1dbe8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f181a5210 0x7f6f18185450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f1dbe8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f18185a70 0x7f6f180acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f1dbe8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f18185e70 0x7f6f180ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f1dbe8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1818f7b0 con 0x7f6f18185e70 2021-04-09T18:34:21.501 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f1dbe8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f180ad7f0 con 0x7f6f18185a70 2021-04-09T18:34:21.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f1dbe8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f180ada10 con 0x7f6f181a5210 2021-04-09T18:34:21.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f18185e70 0x7f6f180ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f17fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f18185e70 0x7f6f180ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59726/0 (socket says 172.21.15.32:59726) 2021-04-09T18:34:21.502 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f17fff700 1 -- 172.21.15.32:0/1439713603 learned_addr learned my addr 172.21.15.32:0/1439713603 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:21.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f16ffd700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f18185a70 0x7f6f180acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.470 7f6f177fe700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f181a5210 0x7f6f18185450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.503 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6f17fff700 1 -- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f181a5210 msgr2=0x7f6f18185450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:21.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6f17fff700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f181a5210 0x7f6f18185450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6f17fff700 1 -- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f18185a70 msgr2=0x7f6f180acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:21.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6f17fff700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f18185a70 0x7f6f180acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.504 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6f17fff700 1 -- 172.21.15.32:0/1439713603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f18179610 con 0x7f6f18185e70 2021-04-09T18:34:21.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6f17fff700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f18185e70 0x7f6f180ad1d0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 rx=0x7f6f10007f90 tx=0x7f6f1000abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:21.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6efffff700 1 -- 172.21.15.32:0/1439713603 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f10000cc0 con 0x7f6f18185e70 2021-04-09T18:34:21.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6efffff700 1 -- 172.21.15.32:0/1439713603 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f100070d0 con 0x7f6f18185e70 2021-04-09T18:34:21.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f18179610 con 0x7f6f18185e70 2021-04-09T18:34:21.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6efffff700 1 -- 172.21.15.32:0/1439713603 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f10007520 con 0x7f6f18185e70 2021-04-09T18:34:21.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.471 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f18179610 con 0x7f6f18185e70 2021-04-09T18:34:21.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.472 7f6efffff700 1 -- 172.21.15.32:0/1439713603 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6f100072b0 con 0x7f6f18185e70 2021-04-09T18:34:21.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.472 7f6efffff700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ef8018930 0x7f6ef801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.506 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.472 7f6efffff700 1 -- 172.21.15.32:0/1439713603 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6f10032d30 con 0x7f6f18185e70 2021-04-09T18:34:21.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.472 7f6f177fe700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ef8018930 0x7f6ef801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.507 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.472 7f6f177fe700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ef8018930 0x7f6ef801adc0 crc :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:21.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.492 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1818f7b0 con 0x7f6f18185e70 2021-04-09T18:34:21.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.495 7f6efffff700 1 -- 172.21.15.32:0/1439713603 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6f10007d00 con 0x7f6f18185e70 2021-04-09T18:34:21.702 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.685 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f1818f7b0 con 0x7f6f18185e70 2021-04-09T18:34:21.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.685 7f6efffff700 1 -- 172.21.15.32:0/1439713603 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v263) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f6f100070d0 con 0x7f6f18185e70 2021-04-09T18:34:21.704 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:21.704 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":263,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:34:17.432629","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":45,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":44,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:34:13.713911","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":45,"pool_name":"cephfs_data","create_time":"2021-04-09 18:34:15.142274","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:34:21.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ef8018930 msgr2=0x7f6ef801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:21.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ef8018930 0x7f6ef801adc0 crc :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f18185e70 msgr2=0x7f6f180ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:21.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f18185e70 0x7f6f180ad1d0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 rx=0x7f6f10007f90 tx=0x7f6f1000abd0).stop 2021-04-09T18:34:21.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 shutdown_connections 2021-04-09T18:34:21.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6ef8018930 0x7f6ef801adc0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f18185e70 0x7f6f180ad1d0 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f18185a70 0x7f6f180acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 --2- 172.21.15.32:0/1439713603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f181a5210 0x7f6f18185450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 >> 172.21.15.32:0/1439713603 conn(0x7f6f181a99c0 msgr2=0x7f6f1817dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:21.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 shutdown_connections 2021-04-09T18:34:21.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.688 7f6f1dbe8700 1 -- 172.21.15.32:0/1439713603 wait complete. 2021-04-09T18:34:21.716 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 44, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:34:13.713911', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 45, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:34:15.142274', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '263', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:34:21.717 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:34:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.851 7f43277c6700 1 Processor -- start 2021-04-09T18:34:21.892 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.852 7f43277c6700 1 -- start start 2021-04-09T18:34:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.852 7f43277c6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f432010d010 0x7f43200d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.852 7f43277c6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f43200d1da0 0x7f43200d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.852 7f43277c6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432010d480 con 0x7f43200d82c0 2021-04-09T18:34:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.852 7f43277c6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43200d8d30 con 0x7f432010d010 2021-04-09T18:34:21.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.852 7f43277c6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43200d9220 con 0x7f43200d1da0 2021-04-09T18:34:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.852 7f43255e2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f432010d010 0x7f43200d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f43255e2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f432010d010 0x7f43200d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59734/0 (socket says 172.21.15.32:59734) 2021-04-09T18:34:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f43255e2700 1 -- 172.21.15.32:0/2960483658 learned_addr learned my addr 172.21.15.32:0/2960483658 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:21.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f4324de1700 1 --2- 172.21.15.32:0/2960483658 >> v2:172.21.15.86:3300/0 conn(0x7f43200d1da0 0x7f43200d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f43255e2700 1 -- 172.21.15.32:0/2960483658 >> v2:172.21.15.86:3300/0 conn(0x7f43200d1da0 msgr2=0x7f43200d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f43255e2700 1 --2- 172.21.15.32:0/2960483658 >> v2:172.21.15.86:3300/0 conn(0x7f43200d1da0 0x7f43200d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f43255e2700 1 -- 172.21.15.32:0/2960483658 >> v1:172.21.15.86:6789/0 conn(0x7f43200d82c0 legacy=0x7f432010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:21.895 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f43255e2700 1 -- 172.21.15.32:0/2960483658 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43200d86f0 con 0x7f432010d010 2021-04-09T18:34:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f4324de1700 1 --2- 172.21.15.32:0/2960483658 >> v2:172.21.15.86:3300/0 conn(0x7f43200d1da0 0x7f43200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:34:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f43255e2700 1 --2- 172.21.15.32:0/2960483658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f432010d010 0x7f43200d1780 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7f431c006700 tx=0x7f431c009b80).ready entity=mon.0 client_cookie=1278dd7db0479360 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f4317fff700 1 -- 172.21.15.32:0/2960483658 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f431c00c0c0 con 0x7f432010d010 2021-04-09T18:34:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f4317fff700 1 -- 172.21.15.32:0/2960483658 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c00c440 con 0x7f432010d010 2021-04-09T18:34:21.896 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f4317fff700 1 -- 172.21.15.32:0/2960483658 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f431c00c920 con 0x7f432010d010 2021-04-09T18:34:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f43277c6700 1 -- 172.21.15.32:0/2960483658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f432010d010 msgr2=0x7f43200d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.853 7f43277c6700 1 --2- 172.21.15.32:0/2960483658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f432010d010 0x7f43200d1780 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7f431c006700 tx=0x7f431c009b80).stop 2021-04-09T18:34:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.854 7f43277c6700 1 -- 172.21.15.32:0/2960483658 shutdown_connections 2021-04-09T18:34:21.897 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.854 7f43277c6700 1 --2- 172.21.15.32:0/2960483658 >> v2:172.21.15.86:3300/0 conn(0x7f43200d1da0 0x7f43200d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.854 7f43277c6700 1 --2- 172.21.15.32:0/2960483658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f432010d010 0x7f43200d1780 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.854 7f43277c6700 1 -- 172.21.15.32:0/2960483658 >> 172.21.15.32:0/2960483658 conn(0x7f43200ce5f0 msgr2=0x7f43200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:21.898 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.854 7f43277c6700 1 -- 172.21.15.32:0/2960483658 shutdown_connections 2021-04-09T18:34:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.854 7f43277c6700 1 -- 172.21.15.32:0/2960483658 wait complete. 2021-04-09T18:34:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.874 7f43277c6700 1 Processor -- start 2021-04-09T18:34:21.899 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.874 7f43277c6700 1 -- start start 2021-04-09T18:34:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f43277c6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43200911c0 0x7f432009e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.902 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f43277c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43200915c0 0x7f432009e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f43277c6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432008d940 0x7f432008dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f43277c6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432009dce0 con 0x7f43200915c0 2021-04-09T18:34:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f43277c6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432009eea0 con 0x7f432008d940 2021-04-09T18:34:21.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f43277c6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432008e360 con 0x7f43200911c0 2021-04-09T18:34:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f4324de1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43200915c0 0x7f432009e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f4324de1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43200915c0 0x7f432009e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59740/0 (socket says 172.21.15.32:59740) 2021-04-09T18:34:21.904 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f4324de1700 1 -- 172.21.15.32:0/502080117 learned_addr learned my addr 172.21.15.32:0/502080117 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f43255e2700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43200911c0 0x7f432009e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.875 7f4325de3700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432008d940 0x7f432008dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f4324de1700 1 -- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43200911c0 msgr2=0x7f432009e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:21.905 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f4324de1700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43200911c0 0x7f432009e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f4324de1700 1 -- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432008d940 msgr2=0x7f432008dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:21.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f4324de1700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432008d940 0x7f432008dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:21.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f4324de1700 1 -- 172.21.15.32:0/502080117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432008e580 con 0x7f43200915c0 2021-04-09T18:34:21.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f4324de1700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43200915c0 0x7f432009e880 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 rx=0x7f4310006f20 tx=0x7f431000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:21.906 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f4315ffb700 1 -- 172.21.15.32:0/502080117 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f431000bd80 con 0x7f43200915c0 2021-04-09T18:34:21.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f4315ffb700 1 -- 172.21.15.32:0/502080117 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4310000c90 con 0x7f43200915c0 2021-04-09T18:34:21.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f43277c6700 1 -- 172.21.15.32:0/502080117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432008e580 con 0x7f43200915c0 2021-04-09T18:34:21.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f4315ffb700 1 -- 172.21.15.32:0/502080117 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43100022b0 con 0x7f43200915c0 2021-04-09T18:34:21.907 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.876 7f43277c6700 1 -- 172.21.15.32:0/502080117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432008e580 con 0x7f43200915c0 2021-04-09T18:34:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.877 7f4315ffb700 1 -- 172.21.15.32:0/502080117 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f431000a000 con 0x7f43200915c0 2021-04-09T18:34:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.877 7f4315ffb700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4300018930 0x7f430001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.877 7f4315ffb700 1 -- 172.21.15.32:0/502080117 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4310032a30 con 0x7f43200915c0 2021-04-09T18:34:21.908 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.877 7f43255e2700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4300018930 0x7f430001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:21.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.877 7f43255e2700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4300018930 0x7f430001ae10 crc :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:21.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.897 7f43277c6700 1 -- 172.21.15.32:0/502080117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432010d480 con 0x7f43200915c0 2021-04-09T18:34:21.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:21.900 7f4315ffb700 1 -- 172.21.15.32:0/502080117 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4310006bb0 con 0x7f43200915c0 2021-04-09T18:34:22.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.080 7f43277c6700 1 -- 172.21.15.32:0/502080117 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f432010d480 con 0x7f43200915c0 2021-04-09T18:34:22.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.080 7f4315ffb700 1 -- 172.21.15.32:0/502080117 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v263) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4310006bb0 con 0x7f43200915c0 2021-04-09T18:34:22.099 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:34:22.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.082 7f43277c6700 1 -- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4300018930 msgr2=0x7f430001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.082 7f43277c6700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4300018930 0x7f430001ae10 crc :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.082 7f43277c6700 1 -- 172.21.15.32:0/502080117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43200915c0 msgr2=0x7f432009e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.082 7f43277c6700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43200915c0 0x7f432009e880 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 rx=0x7f4310006f20 tx=0x7f431000b990).stop 2021-04-09T18:34:22.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.083 7f43277c6700 1 -- 172.21.15.32:0/502080117 shutdown_connections 2021-04-09T18:34:22.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.083 7f43277c6700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4300018930 0x7f430001ae10 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.083 7f43277c6700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43200915c0 0x7f432009e880 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.083 7f43277c6700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432008d940 0x7f432008dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.083 7f43277c6700 1 --2- 172.21.15.32:0/502080117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43200911c0 0x7f432009e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.083 7f43277c6700 1 -- 172.21.15.32:0/502080117 >> 172.21.15.32:0/502080117 conn(0x7f43200b2fa0 msgr2=0x7f43200a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:22.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.083 7f43277c6700 1 -- 172.21.15.32:0/502080117 shutdown_connections 2021-04-09T18:34:22.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.083 7f43277c6700 1 -- 172.21.15.32:0/502080117 wait complete. 2021-04-09T18:34:22.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:34:22.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.258 7fbf6e412700 1 Processor -- start 2021-04-09T18:34:22.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.258 7fbf6e412700 1 -- start start 2021-04-09T18:34:22.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.259 7fbf6e412700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6810d010 0x7fbf680d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.259 7fbf6e412700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbf680d1da0 0x7fbf680d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.259 7fbf6e412700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6810d480 con 0x7fbf680d82c0 2021-04-09T18:34:22.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.259 7fbf6e412700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6810e780 con 0x7fbf680d1da0 2021-04-09T18:34:22.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.259 7fbf6e412700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680d8d30 con 0x7fbf6810d010 2021-04-09T18:34:22.312 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.259 7fbf67fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6810d010 0x7fbf680d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.259 7fbf5ffff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbf680d1da0 0x7fbf680d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.259 7fbf67fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6810d010 0x7fbf680d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:38742/0 (socket says 172.21.15.32:38742) 2021-04-09T18:34:22.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.259 7fbf67fff700 1 -- 172.21.15.32:0/338231851 learned_addr learned my addr 172.21.15.32:0/338231851 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:22.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf67fff700 1 -- 172.21.15.32:0/338231851 >> v2:172.21.15.86:3300/0 conn(0x7fbf680d1da0 msgr2=0x7fbf680d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:22.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf67fff700 1 --2- 172.21.15.32:0/338231851 >> v2:172.21.15.86:3300/0 conn(0x7fbf680d1da0 0x7fbf680d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf67fff700 1 -- 172.21.15.32:0/338231851 >> v1:172.21.15.86:6789/0 conn(0x7fbf680d82c0 legacy=0x7fbf6810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf67fff700 1 -- 172.21.15.32:0/338231851 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680d9220 con 0x7fbf6810d010 2021-04-09T18:34:22.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf677fe700 1 -- 172.21.15.32:0/338231851 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161809228 0 0) 0x7fbf60000d30 con 0x7fbf680d82c0 2021-04-09T18:34:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf5ffff700 1 --2- 172.21.15.32:0/338231851 >> v2:172.21.15.86:3300/0 conn(0x7fbf680d1da0 0x7fbf680d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf67fff700 1 --2- 172.21.15.32:0/338231851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6810d010 0x7fbf680d1780 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fbf58006700 tx=0x7fbf58009b80).ready entity=mon.2 client_cookie=eb23496c86c98231 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:22.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf677fe700 1 -- 172.21.15.32:0/338231851 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf5800c0c0 con 0x7fbf6810d010 2021-04-09T18:34:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf677fe700 1 -- 172.21.15.32:0/338231851 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5800c440 con 0x7fbf6810d010 2021-04-09T18:34:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf677fe700 1 -- 172.21.15.32:0/338231851 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf580009b0 con 0x7fbf6810d010 2021-04-09T18:34:22.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf6e412700 1 -- 172.21.15.32:0/338231851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6810d010 msgr2=0x7fbf680d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.260 7fbf6e412700 1 --2- 172.21.15.32:0/338231851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6810d010 0x7fbf680d1780 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fbf58006700 tx=0x7fbf58009b80).stop 2021-04-09T18:34:22.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.261 7fbf6e412700 1 -- 172.21.15.32:0/338231851 shutdown_connections 2021-04-09T18:34:22.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.261 7fbf6e412700 1 --2- 172.21.15.32:0/338231851 >> v2:172.21.15.86:3300/0 conn(0x7fbf680d1da0 0x7fbf680d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.261 7fbf6e412700 1 --2- 172.21.15.32:0/338231851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6810d010 0x7fbf680d1780 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.261 7fbf6e412700 1 -- 172.21.15.32:0/338231851 >> 172.21.15.32:0/338231851 conn(0x7fbf680ce5f0 msgr2=0x7fbf680d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:22.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.261 7fbf6e412700 1 -- 172.21.15.32:0/338231851 shutdown_connections 2021-04-09T18:34:22.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.261 7fbf6e412700 1 -- 172.21.15.32:0/338231851 wait complete. 2021-04-09T18:34:22.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.281 7fbf6e412700 1 Processor -- start 2021-04-09T18:34:22.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.281 7fbf6e412700 1 -- start start 2021-04-09T18:34:22.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6e412700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6809e530 0x7fbf6809b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6e412700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf6809e930 0x7fbf6809f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6e412700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf6809f990 0x7fbf68094c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6e412700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6809edf0 con 0x7fbf6809f990 2021-04-09T18:34:22.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6e412700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6809fd90 con 0x7fbf6809e930 2021-04-09T18:34:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6e412700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6809bd40 con 0x7fbf6809e530 2021-04-09T18:34:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf67fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6809e530 0x7fbf6809b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6ca2f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf6809f990 0x7fbf68094c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf67fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6809e530 0x7fbf6809b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:38748/0 (socket says 172.21.15.32:38748) 2021-04-09T18:34:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf67fff700 1 -- 172.21.15.32:0/2663452744 learned_addr learned my addr 172.21.15.32:0/2663452744 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:22.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf5ffff700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf6809e930 0x7fbf6809f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6ca2f700 1 -- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6809e530 msgr2=0x7fbf6809b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6ca2f700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6809e530 0x7fbf6809b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6ca2f700 1 -- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf6809e930 msgr2=0x7fbf6809f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6ca2f700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf6809e930 0x7fbf6809f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6ca2f700 1 -- 172.21.15.32:0/2663452744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf68095290 con 0x7fbf6809f990 2021-04-09T18:34:22.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf67fff700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6809e530 0x7fbf6809b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6ca2f700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf6809f990 0x7fbf68094c70 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 rx=0x7fbf6000b8a0 tx=0x7fbf6000bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf657fa700 1 -- 172.21.15.32:0/2663452744 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf60003080 con 0x7fbf6809f990 2021-04-09T18:34:22.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf657fa700 1 -- 172.21.15.32:0/2663452744 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6000c2f0 con 0x7fbf6809f990 2021-04-09T18:34:22.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf68095290 con 0x7fbf6809f990 2021-04-09T18:34:22.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.282 7fbf657fa700 1 -- 172.21.15.32:0/2663452744 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf6000c740 con 0x7fbf6809f990 2021-04-09T18:34:22.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.283 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf68095290 con 0x7fbf6809f990 2021-04-09T18:34:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.283 7fbf657fa700 1 -- 172.21.15.32:0/2663452744 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fbf60003020 con 0x7fbf6809f990 2021-04-09T18:34:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.284 7fbf657fa700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf4c0188c0 0x7fbf4c01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.284 7fbf657fa700 1 -- 172.21.15.32:0/2663452744 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fbf60032e00 con 0x7fbf6809f990 2021-04-09T18:34:22.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.284 7fbf67fff700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf4c0188c0 0x7fbf4c01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.284 7fbf67fff700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf4c0188c0 0x7fbf4c01adc0 crc :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:22.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.303 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf68095b10 con 0x7fbf6809f990 2021-04-09T18:34:22.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.306 7fbf657fa700 1 -- 172.21.15.32:0/2663452744 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fbf60002fe0 con 0x7fbf6809f990 2021-04-09T18:34:22.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.487 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fbf68095b10 con 0x7fbf6809f990 2021-04-09T18:34:22.505 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.487 7fbf657fa700 1 -- 172.21.15.32:0/2663452744 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v263) v1 ==== 104+0+10 (secure 0 0 0) 0x7fbf6000bd80 con 0x7fbf6809f990 2021-04-09T18:34:22.506 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:34:22.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf4c0188c0 msgr2=0x7fbf4c01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf4c0188c0 0x7fbf4c01adc0 crc :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.509 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf6809f990 msgr2=0x7fbf68094c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf6809f990 0x7fbf68094c70 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 rx=0x7fbf6000b8a0 tx=0x7fbf6000bc40).stop 2021-04-09T18:34:22.510 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 shutdown_connections 2021-04-09T18:34:22.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fbf4c0188c0 0x7fbf4c01adc0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf6809f990 0x7fbf68094c70 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.511 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf6809e930 0x7fbf6809f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 --2- 172.21.15.32:0/2663452744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf6809e530 0x7fbf6809b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.512 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 >> 172.21.15.32:0/2663452744 conn(0x7fbf680c4290 msgr2=0x7fbf680c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 shutdown_connections 2021-04-09T18:34:22.513 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.490 7fbf6e412700 1 -- 172.21.15.32:0/2663452744 wait complete. 2021-04-09T18:34:22.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:34:22.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.662 7fd27bae9700 1 Processor -- start 2021-04-09T18:34:22.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd27bae9700 1 -- start start 2021-04-09T18:34:22.704 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd27bae9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd27410d010 0x7fd2740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd27bae9700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2740d1da0 0x7fd2740d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd27bae9700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27410d480 con 0x7fd2740d82c0 2021-04-09T18:34:22.705 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd27bae9700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd27410e780 con 0x7fd2740d1da0 2021-04-09T18:34:22.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd27bae9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740d8d30 con 0x7fd27410d010 2021-04-09T18:34:22.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd279104700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2740d1da0 0x7fd2740d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.706 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd279104700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd2740d1da0 0x7fd2740d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59766/0 (socket says 172.21.15.32:59766) 2021-04-09T18:34:22.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd279104700 1 -- 172.21.15.32:0/4154531568 learned_addr learned my addr 172.21.15.32:0/4154531568 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:22.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.663 7fd279905700 1 --2- 172.21.15.32:0/4154531568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd27410d010 0x7fd2740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.707 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd279104700 1 -- 172.21.15.32:0/4154531568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd27410d010 msgr2=0x7fd2740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:22.708 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd279104700 1 --2- 172.21.15.32:0/4154531568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd27410d010 0x7fd2740d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd278903700 1 -- 172.21.15.32:0/4154531568 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899362364 0 0) 0x7fd26c000d30 con 0x7fd2740d82c0 2021-04-09T18:34:22.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd279104700 1 -- 172.21.15.32:0/4154531568 >> v1:172.21.15.86:6789/0 conn(0x7fd2740d82c0 legacy=0x7fd27410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd279104700 1 -- 172.21.15.32:0/4154531568 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2740d9220 con 0x7fd2740d1da0 2021-04-09T18:34:22.710 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd279104700 1 --2- 172.21.15.32:0/4154531568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740d1da0 0x7fd2740d7ca0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7fd264006700 tx=0x7fd264009b80).ready entity=mon.0 client_cookie=498aa218330674f8 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:22.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd278903700 1 -- 172.21.15.32:0/4154531568 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd26400c0c0 con 0x7fd2740d1da0 2021-04-09T18:34:22.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd278903700 1 -- 172.21.15.32:0/4154531568 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26400c440 con 0x7fd2740d1da0 2021-04-09T18:34:22.711 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd278903700 1 -- 172.21.15.32:0/4154531568 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd26400c920 con 0x7fd2740d1da0 2021-04-09T18:34:22.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd27bae9700 1 -- 172.21.15.32:0/4154531568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740d1da0 msgr2=0x7fd2740d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.664 7fd27bae9700 1 --2- 172.21.15.32:0/4154531568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740d1da0 0x7fd2740d7ca0 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7fd264006700 tx=0x7fd264009b80).stop 2021-04-09T18:34:22.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.665 7fd27bae9700 1 -- 172.21.15.32:0/4154531568 shutdown_connections 2021-04-09T18:34:22.712 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.665 7fd27bae9700 1 --2- 172.21.15.32:0/4154531568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740d1da0 0x7fd2740d7ca0 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.665 7fd27bae9700 1 --2- 172.21.15.32:0/4154531568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd27410d010 0x7fd2740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.665 7fd27bae9700 1 -- 172.21.15.32:0/4154531568 >> 172.21.15.32:0/4154531568 conn(0x7fd2740ce5f0 msgr2=0x7fd2740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:22.713 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.665 7fd27bae9700 1 -- 172.21.15.32:0/4154531568 shutdown_connections 2021-04-09T18:34:22.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.665 7fd27bae9700 1 -- 172.21.15.32:0/4154531568 wait complete. 2021-04-09T18:34:22.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.685 7fd27bae9700 1 Processor -- start 2021-04-09T18:34:22.714 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.685 7fd27bae9700 1 -- start start 2021-04-09T18:34:22.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd27bae9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740b1e50 0x7fd2740b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.718 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd27bae9700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2740b9fd0 0x7fd2740b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd27bae9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2740b5510 0x7fd2740b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd27bae9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740b2250 con 0x7fd2740b1e50 2021-04-09T18:34:22.719 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd27bae9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740a1220 con 0x7fd2740b5510 2021-04-09T18:34:22.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd27bae9700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2740b6b30 con 0x7fd2740b9fd0 2021-04-09T18:34:22.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279905700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740b1e50 0x7fd2740b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.720 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279905700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740b1e50 0x7fd2740b99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59768/0 (socket says 172.21.15.32:59768) 2021-04-09T18:34:22.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279905700 1 -- 172.21.15.32:0/3643030030 learned_addr learned my addr 172.21.15.32:0/3643030030 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:22.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279104700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2740b9fd0 0x7fd2740b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.721 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd27a106700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2740b5510 0x7fd2740b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279905700 1 -- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2740b9fd0 msgr2=0x7fd2740b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279905700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2740b9fd0 0x7fd2740b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279905700 1 -- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2740b5510 msgr2=0x7fd2740b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279905700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2740b5510 0x7fd2740b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279905700 1 -- 172.21.15.32:0/3643030030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2740b5f30 con 0x7fd2740b1e50 2021-04-09T18:34:22.723 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.686 7fd279905700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740b1e50 0x7fd2740b99b0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 rx=0x7fd270006f60 tx=0x7fd27000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.687 7fd26a7fc700 1 -- 172.21.15.32:0/3643030030 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd27000bd60 con 0x7fd2740b1e50 2021-04-09T18:34:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.687 7fd26a7fc700 1 -- 172.21.15.32:0/3643030030 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd270000c90 con 0x7fd2740b1e50 2021-04-09T18:34:22.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.687 7fd26a7fc700 1 -- 172.21.15.32:0/3643030030 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2700009b0 con 0x7fd2740b1e50 2021-04-09T18:34:22.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.687 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2740b5f30 con 0x7fd2740b1e50 2021-04-09T18:34:22.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.688 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2740b5f30 con 0x7fd2740b1e50 2021-04-09T18:34:22.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.688 7fd26a7fc700 1 -- 172.21.15.32:0/3643030030 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd270006b00 con 0x7fd2740b1e50 2021-04-09T18:34:22.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.688 7fd26a7fc700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd25c018930 0x7fd25c01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:22.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.688 7fd26a7fc700 1 -- 172.21.15.32:0/3643030030 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fd270032a00 con 0x7fd2740b1e50 2021-04-09T18:34:22.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.688 7fd279104700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd25c018930 0x7fd25c01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:22.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.689 7fd279104700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd25c018930 0x7fd25c01ae10 crc :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:22.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.707 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2741b7040 con 0x7fd2740b1e50 2021-04-09T18:34:22.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.711 7fd26a7fc700 1 -- 172.21.15.32:0/3643030030 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd2700009b0 con 0x7fd2740b1e50 2021-04-09T18:34:22.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.891 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fd2741b7040 con 0x7fd2740b1e50 2021-04-09T18:34:22.910 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.891 7fd26a7fc700 1 -- 172.21.15.32:0/3643030030 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v263) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd270006b00 con 0x7fd2740b1e50 2021-04-09T18:34:22.910 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:34:22.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd25c018930 msgr2=0x7fd25c01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd25c018930 0x7fd25c01ae10 crc :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740b1e50 msgr2=0x7fd2740b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:22.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740b1e50 0x7fd2740b99b0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 rx=0x7fd270006f60 tx=0x7fd27000b970).stop 2021-04-09T18:34:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 shutdown_connections 2021-04-09T18:34:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd25c018930 0x7fd25c01ae10 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2740b5510 0x7fd2740b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2740b9fd0 0x7fd2740b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 --2- 172.21.15.32:0/3643030030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd2740b1e50 0x7fd2740b99b0 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 >> 172.21.15.32:0/3643030030 conn(0x7fd2740b27e0 msgr2=0x7fd2741b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:22.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 shutdown_connections 2021-04-09T18:34:22.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:22.894 7fd27bae9700 1 -- 172.21.15.32:0/3643030030 wait complete. 2021-04-09T18:34:22.923 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:34:22.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:23.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.053 7f3c23b35700 1 Processor -- start 2021-04-09T18:34:23.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.053 7f3c23b35700 1 -- start start 2021-04-09T18:34:23.093 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c23b35700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c1c0d7ca0 0x7f3c1c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:23.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c23b35700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c10d480 con 0x7f3c1c10d010 2021-04-09T18:34:23.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c23b35700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c0d86c0 con 0x7f3c1c10da00 2021-04-09T18:34:23.094 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c23b35700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c0d9220 con 0x7f3c1c0d7ca0 2021-04-09T18:34:23.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c21150700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c1c0d7ca0 0x7f3c1c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:23.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c21951700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3c1c10d010 0x7f3c1c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41134/0 (socket says 172.21.15.32:41134) 2021-04-09T18:34:23.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c21951700 1 -- 172.21.15.32:0/3845237099 learned_addr learned my addr 172.21.15.32:0/3845237099 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:23.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c2094f700 1 -- 172.21.15.32:0/3845237099 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092231481 0 0) 0x7f3c14000d30 con 0x7f3c1c10da00 2021-04-09T18:34:23.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c21150700 1 -- 172.21.15.32:0/3845237099 >> v1:172.21.15.86:6789/0 conn(0x7f3c1c10da00 legacy=0x7f3c1c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:23.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c21150700 1 -- 172.21.15.32:0/3845237099 >> v1:172.21.15.32:6789/0 conn(0x7f3c1c10d010 legacy=0x7f3c1c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:23.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c21150700 1 -- 172.21.15.32:0/3845237099 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c0d1f70 con 0x7f3c1c0d7ca0 2021-04-09T18:34:23.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.054 7f3c21150700 1 --2- 172.21.15.32:0/3845237099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0d7ca0 0x7f3c1c0d80a0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f3c10006650 tx=0x7f3c10009a00).ready entity=mon.0 client_cookie=8dab3fb7951dae67 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:23.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c2094f700 1 -- 172.21.15.32:0/3845237099 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c10009df0 con 0x7f3c1c0d7ca0 2021-04-09T18:34:23.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c2094f700 1 -- 172.21.15.32:0/3845237099 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c100009b0 con 0x7f3c1c0d7ca0 2021-04-09T18:34:23.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c2094f700 1 -- 172.21.15.32:0/3845237099 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c1000c2b0 con 0x7f3c1c0d7ca0 2021-04-09T18:34:23.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c23b35700 1 -- 172.21.15.32:0/3845237099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0d7ca0 msgr2=0x7f3c1c0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:23.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c23b35700 1 --2- 172.21.15.32:0/3845237099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0d7ca0 0x7f3c1c0d80a0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f3c10006650 tx=0x7f3c10009a00).stop 2021-04-09T18:34:23.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c23b35700 1 -- 172.21.15.32:0/3845237099 shutdown_connections 2021-04-09T18:34:23.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c23b35700 1 --2- 172.21.15.32:0/3845237099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0d7ca0 0x7f3c1c0d80a0 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:23.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c23b35700 1 -- 172.21.15.32:0/3845237099 >> 172.21.15.32:0/3845237099 conn(0x7f3c1c0ce5f0 msgr2=0x7f3c1c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:23.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c23b35700 1 -- 172.21.15.32:0/3845237099 shutdown_connections 2021-04-09T18:34:23.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.055 7f3c23b35700 1 -- 172.21.15.32:0/3845237099 wait complete. 2021-04-09T18:34:23.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.075 7f3c23b35700 1 Processor -- start 2021-04-09T18:34:23.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.075 7f3c23b35700 1 -- start start 2021-04-09T18:34:23.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c23b35700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c1c1c2230 0x7f3c1c1c2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:23.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c23b35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0bbba0 0x7f3c1c0bbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:23.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c23b35700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c1c0ba830 0x7f3c1c0bac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:23.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c23b35700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c1abeb0 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c23b35700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0bc5c0 con 0x7f3c1c0ba830 2021-04-09T18:34:23.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c23b35700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0b7970 con 0x7f3c1c1c2230 2021-04-09T18:34:23.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c21150700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0bbba0 0x7f3c1c0bbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:23.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c21150700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0bbba0 0x7f3c1c0bbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59786/0 (socket says 172.21.15.32:59786) 2021-04-09T18:34:23.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c21150700 1 -- 172.21.15.32:0/3377042475 learned_addr learned my addr 172.21.15.32:0/3377042475 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:23.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c21951700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c1c1c2230 0x7f3c1c1c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:23.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.076 7f3c22152700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c1c0ba830 0x7f3c1c0bac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:23.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c21150700 1 -- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c1c1c2230 msgr2=0x7f3c1c1c2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:23.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c21150700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c1c1c2230 0x7f3c1c1c2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:23.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c21150700 1 -- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c1c0ba830 msgr2=0x7f3c1c0bac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:23.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c21150700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c1c0ba830 0x7f3c1c0bac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:23.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c21150700 1 -- 172.21.15.32:0/3377042475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c0bde70 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c21150700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0bbba0 0x7f3c1c0bbfa0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f3c1000b580 tx=0x7f3c1000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:23.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c0e7fc700 1 -- 172.21.15.32:0/3377042475 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c10000c90 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c0e7fc700 1 -- 172.21.15.32:0/3377042475 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c10002030 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c0e7fc700 1 -- 172.21.15.32:0/3377042475 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c10002480 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c0bde70 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.077 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c0bde70 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.078 7f3c0e7fc700 1 -- 172.21.15.32:0/3377042475 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3c10002210 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.078 7f3c0e7fc700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c040188c0 0x7f3c0401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:23.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.078 7f3c0e7fc700 1 -- 172.21.15.32:0/3377042475 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3c10032800 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.078 7f3c21951700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c040188c0 0x7f3c0401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:23.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.078 7f3c21951700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c040188c0 0x7f3c0401adc0 crc :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:23.116 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.098 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c1a47d0 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.101 7f3c0e7fc700 1 -- 172.21.15.32:0/3377042475 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3c100009c0 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.292 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c1c1a47d0 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.293 7f3c0e7fc700 1 -- 172.21.15.32:0/3377042475 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 426 v426) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f3c1000bd10 con 0x7f3c1c0bbba0 2021-04-09T18:34:23.311 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 426 2021-04-09T18:34:23.312 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:23.312 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":426,"default_fscid":22,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/368655383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":368655383},{"type":"v1","addr":"172.21.15.32:6834","nonce":368655383}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/4251960201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":4251960201},{"type":"v1","addr":"172.21.15.86:6836","nonce":4251960201}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59951,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1668354365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1668354365},{"type":"v1","addr":"172.21.15.86:6837","nonce":1668354365}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422}],"filesystems":[{"mdsmap":{"epoch":426,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:34:16.431259","modified":"2021-04-09 18:34:18.673793","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12379":{"gid":12379,"name":"a","rank":0,"incarnation":423,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2610523978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2610523978},{"type":"v1","addr":"172.21.15.32:6835","nonce":2610523978}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[45],"metadata_pool":44,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":22}]} 2021-04-09T18:34:23.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c040188c0 msgr2=0x7f3c0401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:23.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c040188c0 0x7f3c0401adc0 crc :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:23.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0bbba0 msgr2=0x7f3c1c0bbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:23.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0bbba0 0x7f3c1c0bbfa0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f3c1000b580 tx=0x7f3c1000b920).stop 2021-04-09T18:34:23.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 shutdown_connections 2021-04-09T18:34:23.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c040188c0 0x7f3c0401adc0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:23.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c1c0bbba0 0x7f3c1c0bbfa0 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:23.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c1c0ba830 0x7f3c1c0bac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:23.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 --2- 172.21.15.32:0/3377042475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c1c1c2230 0x7f3c1c1c2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:23.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 >> 172.21.15.32:0/3377042475 conn(0x7f3c1c0bcd80 msgr2=0x7f3c1c08dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:23.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 shutdown_connections 2021-04-09T18:34:23.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:23.295 7f3c23b35700 1 -- 172.21.15.32:0/3377042475 wait complete. 2021-04-09T18:34:23.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 426, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:18.673793', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:23.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:23.727 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:23.864 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:23.864 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:23.864 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:23.865 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:23.865 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:23.865 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:23.865 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 7.4147730489999999, 2021-04-09T18:34:23.865 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 426, 2021-04-09T18:34:23.866 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 263, 2021-04-09T18:34:23.866 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 263, 2021-04-09T18:34:23.866 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 12.856889439 2021-04-09T18:34:23.866 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:23.870 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.4147730489999999, "mdsmap_epoch": 426, "osdmap_epoch": 263, "osdmap_epoch_barrier": 263, "uptime": 12.856889439 } 2021-04-09T18:34:23.871 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:34:23.871 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:34:23.871 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:34:23.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:23.872 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:34:23.916 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:34:23.975 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:34:23.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:34:24.034 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:34:24.034 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:34:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:34:24.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:24.036 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:34:24.080 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:34:24.080 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:34:24.121 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:34:24.182 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:34:24.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:34:24.241 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:34:24.242 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:34:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:34:24.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:24.244 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:34:24.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:34:24.254 7fd76f579f80 -1 init, newargv = 0x55db2f60f2b0 newargc=9 2021-04-09T18:34:24.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[109035]: starting ceph client 2021-04-09T18:34:24.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[109035]: starting fuse 2021-04-09T18:34:24.289 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:34:24.289 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:34:24.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:24.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:34:24.290 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:34:24.291 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:34:24.291 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:34:24.292 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:34:24.292 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:34:24.293 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:34:24.293 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:34:24.293 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:34:24.294 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:34:24.294 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:34:24.295 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:34:24.295 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:34:24.295 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:34:24.296 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:34:24.296 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:34:24.297 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:34:24.297 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:34:24.297 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:34:24.298 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:34:24.298 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:34:24.299 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:34:24.299 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:34:24.411 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.109035.asok 2021-04-09T18:34:24.414 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.109035.asok 2021-04-09T18:34:24.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:24.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.109035.asok status 2021-04-09T18:34:24.540 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:24.540 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:34:24.541 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:34:24.541 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:34:24.541 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:34:24.541 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:34:24.541 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:34:24.542 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "109035", 2021-04-09T18:34:24.542 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:34:24.542 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:24.542 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:34:24.542 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:34:24.543 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 60050, 2021-04-09T18:34:24.543 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:34:24.543 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:34:24.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:34:24.544 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 60050 2021-04-09T18:34:24.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:24.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:34:24.544 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:24.544 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:34:24.545 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1285738709 2021-04-09T18:34:24.545 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:34:24.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:24.545 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:34:24.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:34:24.546 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:34:24.546 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1285738709 2021-04-09T18:34:24.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:34:24.546 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.60050 172.21.15.32:0/1285738709", 2021-04-09T18:34:24.547 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/1285738709", 2021-04-09T18:34:24.547 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:34:24.548 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 426, 2021-04-09T18:34:24.548 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 263, 2021-04-09T18:34:24.548 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:34:24.549 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:34:24.549 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:34:24.549 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:24.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:24.552 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:34:24.553 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:34:24.598 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:34:24.599 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:34:24.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:34:24.600 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:34:24.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:24.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.792 7f6ffa7dd700 1 Processor -- start 2021-04-09T18:34:24.832 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.792 7f6ffa7dd700 1 -- start start 2021-04-09T18:34:24.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.792 7f6ffa7dd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff40cffa0 0x7f6ff40d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:24.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.792 7f6ffa7dd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40d7280 con 0x7f6ff40d6c80 2021-04-09T18:34:24.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ffa7dd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff410e7d0 con 0x7f6ff40d7800 2021-04-09T18:34:24.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ffa7dd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40d7080 con 0x7f6ff40cffa0 2021-04-09T18:34:24.833 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff8dfa700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6ff40d6c80 0x7f6ff410cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41164/0 (socket says 172.21.15.32:41164) 2021-04-09T18:34:24.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff8dfa700 1 -- 172.21.15.32:0/3886492619 learned_addr learned my addr 172.21.15.32:0/3886492619 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:24.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff3fff700 1 --2- 172.21.15.32:0/3886492619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff40cffa0 0x7f6ff40d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:24.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff37fe700 1 -- 172.21.15.32:0/3886492619 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250990485 0 0) 0x7f6fec000d30 con 0x7f6ff40d6c80 2021-04-09T18:34:24.834 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff37fe700 1 -- 172.21.15.32:0/3886492619 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd4002fe0 con 0x7f6ff40d6c80 2021-04-09T18:34:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff37fe700 1 -- 172.21.15.32:0/3886492619 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871032721 0 0) 0x7f6fdc001080 con 0x7f6ff40d7800 2021-04-09T18:34:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff37fe700 1 -- 172.21.15.32:0/3886492619 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd4003210 con 0x7f6ff40d7800 2021-04-09T18:34:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff3fff700 1 -- 172.21.15.32:0/3886492619 >> v1:172.21.15.86:6789/0 conn(0x7f6ff40d7800 legacy=0x7f6ff40d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff3fff700 1 -- 172.21.15.32:0/3886492619 >> v1:172.21.15.32:6789/0 conn(0x7f6ff40d6c80 legacy=0x7f6ff410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:24.835 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff37fe700 1 -- 172.21.15.32:0/3886492619 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3131202443 0 0) 0x7f6fec001270 con 0x7f6ff40d6c80 2021-04-09T18:34:24.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff3fff700 1 -- 172.21.15.32:0/3886492619 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40d8580 con 0x7f6ff40cffa0 2021-04-09T18:34:24.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.793 7f6ff3fff700 1 --2- 172.21.15.32:0/3886492619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff40cffa0 0x7f6ff40d2440 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f6fe4006650 tx=0x7f6fe4009a00).ready entity=mon.2 client_cookie=e7b4f2d31293e9cd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:24.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ff37fe700 1 -- 172.21.15.32:0/3886492619 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fe4009df0 con 0x7f6ff40cffa0 2021-04-09T18:34:24.836 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ff37fe700 1 -- 172.21.15.32:0/3886492619 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe40009b0 con 0x7f6ff40cffa0 2021-04-09T18:34:24.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ff37fe700 1 -- 172.21.15.32:0/3886492619 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fe400c2b0 con 0x7f6ff40cffa0 2021-04-09T18:34:24.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ffa7dd700 1 -- 172.21.15.32:0/3886492619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff40cffa0 msgr2=0x7f6ff40d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:24.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ffa7dd700 1 --2- 172.21.15.32:0/3886492619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff40cffa0 0x7f6ff40d2440 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f6fe4006650 tx=0x7f6fe4009a00).stop 2021-04-09T18:34:24.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ffa7dd700 1 -- 172.21.15.32:0/3886492619 shutdown_connections 2021-04-09T18:34:24.837 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ffa7dd700 1 --2- 172.21.15.32:0/3886492619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff40cffa0 0x7f6ff40d2440 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ffa7dd700 1 -- 172.21.15.32:0/3886492619 >> 172.21.15.32:0/3886492619 conn(0x7f6ff40ce5c0 msgr2=0x7f6ff40cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ffa7dd700 1 -- 172.21.15.32:0/3886492619 shutdown_connections 2021-04-09T18:34:24.838 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.794 7f6ffa7dd700 1 -- 172.21.15.32:0/3886492619 wait complete. 2021-04-09T18:34:24.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ffa7dd700 1 Processor -- start 2021-04-09T18:34:24.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ffa7dd700 1 -- start start 2021-04-09T18:34:24.841 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ffa7dd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff41ab8a0 0x7f6ff41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:24.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ffa7dd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff41abca0 0x7f6ff41a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:24.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ffa7dd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff41ac0a0 0x7f6ff41aa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:24.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ffa7dd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41ac620 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ffa7dd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff4185300 con 0x7f6ff41ac0a0 2021-04-09T18:34:24.842 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ffa7dd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41ab320 con 0x7f6ff41abca0 2021-04-09T18:34:24.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ff3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff41ab8a0 0x7f6ff41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:24.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ff3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff41ab8a0 0x7f6ff41981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59814/0 (socket says 172.21.15.32:59814) 2021-04-09T18:34:24.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ff3fff700 1 -- 172.21.15.32:0/1214814814 learned_addr learned my addr 172.21.15.32:0/1214814814 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:24.843 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6febfff700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff41abca0 0x7f6ff41a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:24.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.815 7f6ff8dfa700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff41ac0a0 0x7f6ff41aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:24.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ff3fff700 1 -- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff41abca0 msgr2=0x7f6ff41a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:24.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ff3fff700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff41abca0 0x7f6ff41a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:24.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ff3fff700 1 -- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff41ac0a0 msgr2=0x7f6ff41aa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:24.844 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ff3fff700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff41ac0a0 0x7f6ff41aa500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:24.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ff3fff700 1 -- 172.21.15.32:0/1214814814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff4184e90 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ff3fff700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff41ab8a0 0x7f6ff41981b0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 rx=0x7f6fe40096a0 tx=0x7f6fe400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:24.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ff17fa700 1 -- 172.21.15.32:0/1214814814 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fe4000c90 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ff17fa700 1 -- 172.21.15.32:0/1214814814 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4002030 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.845 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff4184e90 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ff17fa700 1 -- 172.21.15.32:0/1214814814 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fe4002550 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.816 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff4184e90 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.817 7f6ff17fa700 1 -- 172.21.15.32:0/1214814814 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6fe4000c90 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.817 7f6ff17fa700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fcc0188c0 0x7f6fcc01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:24.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.817 7f6ff17fa700 1 -- 172.21.15.32:0/1214814814 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f6fe4032830 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.817 7f6febfff700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fcc0188c0 0x7f6fcc01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:24.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.818 7f6febfff700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fcc0188c0 0x7f6fcc01adc0 crc :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:24.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.836 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff4182df0 con 0x7f6ff41ab8a0 2021-04-09T18:34:24.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:24.840 7f6ff17fa700 1 -- 172.21.15.32:0/1214814814 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f6fe40009a0 con 0x7f6ff41ab8a0 2021-04-09T18:34:25.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.032 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ff4182df0 con 0x7f6ff41ab8a0 2021-04-09T18:34:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.032 7f6ff17fa700 1 -- 172.21.15.32:0/1214814814 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 426 v426) v1 ==== 95+0+2786 (secure 0 0 0) 0x7f6fe400bd80 con 0x7f6ff41ab8a0 2021-04-09T18:34:25.051 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 426 2021-04-09T18:34:25.051 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:25.052 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":426,"default_fscid":22,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/368655383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":368655383},{"type":"v1","addr":"172.21.15.32:6834","nonce":368655383}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/4251960201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":4251960201},{"type":"v1","addr":"172.21.15.86:6836","nonce":4251960201}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59951,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1668354365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1668354365},{"type":"v1","addr":"172.21.15.86:6837","nonce":1668354365}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422}],"filesystems":[{"mdsmap":{"epoch":426,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:34:16.431259","modified":"2021-04-09 18:34:18.673793","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12379":{"gid":12379,"name":"a","rank":0,"incarnation":423,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2610523978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2610523978},{"type":"v1","addr":"172.21.15.32:6835","nonce":2610523978}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[45],"metadata_pool":44,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":22}]} 2021-04-09T18:34:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fcc0188c0 msgr2=0x7f6fcc01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fcc0188c0 0x7f6fcc01adc0 crc :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff41ab8a0 msgr2=0x7f6ff41981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff41ab8a0 0x7f6ff41981b0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 rx=0x7f6fe40096a0 tx=0x7f6fe400b990).stop 2021-04-09T18:34:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 shutdown_connections 2021-04-09T18:34:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6fcc0188c0 0x7f6fcc01adc0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff41ac0a0 0x7f6ff41aa500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff41abca0 0x7f6ff41a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 --2- 172.21.15.32:0/1214814814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ff41ab8a0 0x7f6ff41981b0 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 >> 172.21.15.32:0/1214814814 conn(0x7f6ff4183870 msgr2=0x7f6ff41a9650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:25.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 shutdown_connections 2021-04-09T18:34:25.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.035 7f6ffa7dd700 1 -- 172.21.15.32:0/1214814814 wait complete. 2021-04-09T18:34:25.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2021-04-09T18:34:25.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.198 7f0c0bbae700 1 Processor -- start 2021-04-09T18:34:25.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.198 7f0c0bbae700 1 -- start start 2021-04-09T18:34:25.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c0bbae700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c0410d010 0x7f0c040d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c0bbae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040d1da0 0x7f0c040d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c0bbae700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0410d480 con 0x7f0c040d82c0 2021-04-09T18:34:25.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c0bbae700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c040d9220 con 0x7f0c0410d010 2021-04-09T18:34:25.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c0bbae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040d86c0 con 0x7f0c040d1da0 2021-04-09T18:34:25.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c099ca700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0c0410d010 0x7f0c040d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c0a1cb700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0c040d82c0 0x7f0c0410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41178/0 (socket says 172.21.15.32:41178) 2021-04-09T18:34:25.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c0a1cb700 1 -- 172.21.15.32:0/2736010120 learned_addr learned my addr 172.21.15.32:0/2736010120 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:25.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c091c9700 1 --2- 172.21.15.32:0/2736010120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040d1da0 0x7f0c040d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c099ca700 1 -- 172.21.15.32:0/2736010120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040d1da0 msgr2=0x7f0c040d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:25.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c099ca700 1 --2- 172.21.15.32:0/2736010120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040d1da0 0x7f0c040d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c099ca700 1 -- 172.21.15.32:0/2736010120 >> v1:172.21.15.32:6789/0 conn(0x7f0c040d82c0 legacy=0x7f0c0410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c099ca700 1 -- 172.21.15.32:0/2736010120 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0410e1f0 con 0x7f0c0410d010 2021-04-09T18:34:25.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c089c8700 1 -- 172.21.15.32:0/2736010120 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425451497 0 0) 0x7f0bf40009b0 con 0x7f0c040d82c0 2021-04-09T18:34:25.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.199 7f0c099ca700 1 --2- 172.21.15.32:0/2736010120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0410d010 0x7f0c040d1780 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 rx=0x7f0bfc0069d0 tx=0x7f0bfc009dc0).ready entity=mon.0 client_cookie=758f1def37b1f690 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:25.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.200 7f0c089c8700 1 -- 172.21.15.32:0/2736010120 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bfc00b510 con 0x7f0c0410d010 2021-04-09T18:34:25.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.200 7f0c089c8700 1 -- 172.21.15.32:0/2736010120 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc006ac0 con 0x7f0c0410d010 2021-04-09T18:34:25.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.200 7f0c089c8700 1 -- 172.21.15.32:0/2736010120 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bfc00ba80 con 0x7f0c0410d010 2021-04-09T18:34:25.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.200 7f0c0bbae700 1 -- 172.21.15.32:0/2736010120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0410d010 msgr2=0x7f0c040d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.200 7f0c0bbae700 1 --2- 172.21.15.32:0/2736010120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0410d010 0x7f0c040d1780 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 rx=0x7f0bfc0069d0 tx=0x7f0bfc009dc0).stop 2021-04-09T18:34:25.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.200 7f0c0bbae700 1 -- 172.21.15.32:0/2736010120 shutdown_connections 2021-04-09T18:34:25.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.200 7f0c0bbae700 1 --2- 172.21.15.32:0/2736010120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040d1da0 0x7f0c040d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.200 7f0c0bbae700 1 --2- 172.21.15.32:0/2736010120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0410d010 0x7f0c040d1780 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.200 7f0c0bbae700 1 -- 172.21.15.32:0/2736010120 >> 172.21.15.32:0/2736010120 conn(0x7f0c040ce5f0 msgr2=0x7f0c040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:25.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.201 7f0c0bbae700 1 -- 172.21.15.32:0/2736010120 shutdown_connections 2021-04-09T18:34:25.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.201 7f0c0bbae700 1 -- 172.21.15.32:0/2736010120 wait complete. 2021-04-09T18:34:25.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c0bbae700 1 Processor -- start 2021-04-09T18:34:25.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c0bbae700 1 -- start start 2021-04-09T18:34:25.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c0bbae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c0408a910 0x7f0c0409e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c0bbae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0408ad10 0x7f0c0409ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c0bbae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040922e0 0x7f0c040926e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c0bbae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0418fc90 con 0x7f0c0408ad10 2021-04-09T18:34:25.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c0bbae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0409e090 con 0x7f0c0408a910 2021-04-09T18:34:25.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c0bbae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0408b400 con 0x7f0c040922e0 2021-04-09T18:34:25.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c091c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0408ad10 0x7f0c0409ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c091c9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0408ad10 0x7f0c0409ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59826/0 (socket says 172.21.15.32:59826) 2021-04-09T18:34:25.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.221 7f0c091c9700 1 -- 172.21.15.32:0/395582941 learned_addr learned my addr 172.21.15.32:0/395582941 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:25.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c0a1cb700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040922e0 0x7f0c040926e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c091c9700 1 -- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040922e0 msgr2=0x7f0c040926e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c099ca700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c0408a910 0x7f0c0409e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c091c9700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040922e0 0x7f0c040926e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c091c9700 1 -- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c0408a910 msgr2=0x7f0c0409e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c091c9700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c0408a910 0x7f0c0409e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c091c9700 1 -- 172.21.15.32:0/395582941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c04092d00 con 0x7f0c0408ad10 2021-04-09T18:34:25.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c091c9700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0408ad10 0x7f0c0409ec30 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f0bf800b560 tx=0x7f0bf800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:25.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c027fc700 1 -- 172.21.15.32:0/395582941 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bf800bcf0 con 0x7f0c0408ad10 2021-04-09T18:34:25.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c027fc700 1 -- 172.21.15.32:0/395582941 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf8000c90 con 0x7f0c0408ad10 2021-04-09T18:34:25.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c027fc700 1 -- 172.21.15.32:0/395582941 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bf8002220 con 0x7f0c0408ad10 2021-04-09T18:34:25.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c04092d00 con 0x7f0c0408ad10 2021-04-09T18:34:25.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.222 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c04092d00 con 0x7f0c0408ad10 2021-04-09T18:34:25.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.223 7f0c027fc700 1 -- 172.21.15.32:0/395582941 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0bf80052e0 con 0x7f0c0408ad10 2021-04-09T18:34:25.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.223 7f0c027fc700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0be40188c0 0x7f0be401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.223 7f0c027fc700 1 -- 172.21.15.32:0/395582941 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f0bf8032a20 con 0x7f0c0408ad10 2021-04-09T18:34:25.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.223 7f0c099ca700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0be40188c0 0x7f0be401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.223 7f0c099ca700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0be40188c0 0x7f0be401adc0 crc :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:25.261 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.243 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0418fc90 con 0x7f0c0408ad10 2021-04-09T18:34:25.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.246 7f0c027fc700 1 -- 172.21.15.32:0/395582941 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0bf8006b00 con 0x7f0c0408ad10 2021-04-09T18:34:25.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.441 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"} v 0) v1 -- 0x7f0c0418fc90 con 0x7f0c0408ad10 2021-04-09T18:34:25.709 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.691 7f0c027fc700 1 -- 172.21.15.32:0/395582941 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "max_mds", "prefix": "fs set", "fs_name": "cephfs", "val": "1"}]=0 v427) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0bf80052e0 con 0x7f0c0408ad10 2021-04-09T18:34:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0be40188c0 msgr2=0x7f0be401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0be40188c0 0x7f0be401adc0 crc :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0408ad10 msgr2=0x7f0c0409ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0408ad10 0x7f0c0409ec30 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f0bf800b560 tx=0x7f0bf800b900).stop 2021-04-09T18:34:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 shutdown_connections 2021-04-09T18:34:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0be40188c0 0x7f0be401adc0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0c0408ad10 0x7f0c0409ec30 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c040922e0 0x7f0c040926e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 --2- 172.21.15.32:0/395582941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c0408a910 0x7f0c0409e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 >> 172.21.15.32:0/395582941 conn(0x7f0c040b2fa0 msgr2=0x7f0c040a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:25.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 shutdown_connections 2021-04-09T18:34:25.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.707 7f0c0bbae700 1 -- 172.21.15.32:0/395582941 wait complete. 2021-04-09T18:34:25.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2021-04-09T18:34:25.912 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.872 7f12aa4b0700 1 Processor -- start 2021-04-09T18:34:25.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.872 7f12aa4b0700 1 -- start start 2021-04-09T18:34:25.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12aa4b0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12a410d010 0x7f12a40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12aa4b0700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f12a40d8490 0x7f12a410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.913 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12aa4b0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a410d480 con 0x7f12a40d1da0 2021-04-09T18:34:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12aa4b0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a40d8d30 con 0x7f12a410d010 2021-04-09T18:34:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12aa4b0700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a40d9220 con 0x7f12a40d8490 2021-04-09T18:34:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12a3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12a410d010 0x7f12a40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12a3fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f12a410d010 0x7f12a40d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59832/0 (socket says 172.21.15.32:59832) 2021-04-09T18:34:25.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12a3fff700 1 -- 172.21.15.32:0/3292221603 learned_addr learned my addr 172.21.15.32:0/3292221603 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:25.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12a8acd700 1 --2- 172.21.15.32:0/3292221603 >> v2:172.21.15.86:3300/0 conn(0x7f12a40d8490 0x7f12a410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12a3fff700 1 -- 172.21.15.32:0/3292221603 >> v2:172.21.15.86:3300/0 conn(0x7f12a40d8490 msgr2=0x7f12a410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:25.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12a3fff700 1 --2- 172.21.15.32:0/3292221603 >> v2:172.21.15.86:3300/0 conn(0x7f12a40d8490 0x7f12a410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12a3fff700 1 -- 172.21.15.32:0/3292221603 >> v1:172.21.15.86:6789/0 conn(0x7f12a40d1da0 legacy=0x7f12a40d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.873 7f12a3fff700 1 -- 172.21.15.32:0/3292221603 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a410e020 con 0x7f12a410d010 2021-04-09T18:34:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.874 7f12a3fff700 1 --2- 172.21.15.32:0/3292221603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a410d010 0x7f12a40d1780 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7f1294006700 tx=0x7f1294009b00).ready entity=mon.0 client_cookie=b1b040aa8e0cb43d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.874 7f12a37fe700 1 -- 172.21.15.32:0/3292221603 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f129400c0c0 con 0x7f12a410d010 2021-04-09T18:34:25.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.874 7f12a37fe700 1 -- 172.21.15.32:0/3292221603 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129400c400 con 0x7f12a410d010 2021-04-09T18:34:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.874 7f12a37fe700 1 -- 172.21.15.32:0/3292221603 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f129400c8e0 con 0x7f12a410d010 2021-04-09T18:34:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.874 7f12aa4b0700 1 -- 172.21.15.32:0/3292221603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a410d010 msgr2=0x7f12a40d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.874 7f12aa4b0700 1 --2- 172.21.15.32:0/3292221603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a410d010 0x7f12a40d1780 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7f1294006700 tx=0x7f1294009b00).stop 2021-04-09T18:34:25.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.875 7f12aa4b0700 1 -- 172.21.15.32:0/3292221603 shutdown_connections 2021-04-09T18:34:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.875 7f12aa4b0700 1 --2- 172.21.15.32:0/3292221603 >> v2:172.21.15.86:3300/0 conn(0x7f12a40d8490 0x7f12a410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.875 7f12aa4b0700 1 --2- 172.21.15.32:0/3292221603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a410d010 0x7f12a40d1780 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.875 7f12aa4b0700 1 -- 172.21.15.32:0/3292221603 >> 172.21.15.32:0/3292221603 conn(0x7f12a40ce5f0 msgr2=0x7f12a40d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.875 7f12aa4b0700 1 -- 172.21.15.32:0/3292221603 shutdown_connections 2021-04-09T18:34:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.875 7f12aa4b0700 1 -- 172.21.15.32:0/3292221603 wait complete. 2021-04-09T18:34:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12aa4b0700 1 Processor -- start 2021-04-09T18:34:25.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12aa4b0700 1 -- start start 2021-04-09T18:34:25.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12aa4b0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a417c460 0x7f12a40a2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12aa4b0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12a417c860 0x7f12a40a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12aa4b0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12a417a990 0x7f12a417ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12aa4b0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a40a2090 con 0x7f12a417c460 2021-04-09T18:34:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12aa4b0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a417b3b0 con 0x7f12a417c860 2021-04-09T18:34:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12aa4b0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a417b5d0 con 0x7f12a417a990 2021-04-09T18:34:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12a3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a417c460 0x7f12a40a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12a3fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a417c460 0x7f12a40a2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59836/0 (socket says 172.21.15.32:59836) 2021-04-09T18:34:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12a3fff700 1 -- 172.21.15.32:0/3829020286 learned_addr learned my addr 172.21.15.32:0/3829020286 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f12a8acd700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12a417a990 0x7f12a417ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.895 7f129bfff700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12a417c860 0x7f12a40a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12a3fff700 1 -- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12a417a990 msgr2=0x7f12a417ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12a3fff700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12a417a990 0x7f12a417ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12a3fff700 1 -- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12a417c860 msgr2=0x7f12a40a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12a3fff700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12a417c860 0x7f12a40a2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12a3fff700 1 -- 172.21.15.32:0/3829020286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a417b7f0 con 0x7f12a417c460 2021-04-09T18:34:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12a3fff700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a417c460 0x7f12a40a2610 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 rx=0x7f129400b590 tx=0x7f129400b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12a17fa700 1 -- 172.21.15.32:0/3829020286 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1294006390 con 0x7f12a417c460 2021-04-09T18:34:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12a17fa700 1 -- 172.21.15.32:0/3829020286 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129400be20 con 0x7f12a417c460 2021-04-09T18:34:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a417b7f0 con 0x7f12a417c460 2021-04-09T18:34:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12a17fa700 1 -- 172.21.15.32:0/3829020286 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1294002030 con 0x7f12a417c460 2021-04-09T18:34:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.896 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a417b7f0 con 0x7f12a417c460 2021-04-09T18:34:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.897 7f12a17fa700 1 -- 172.21.15.32:0/3829020286 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f12940093e0 con 0x7f12a417c460 2021-04-09T18:34:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.897 7f12a17fa700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f127c0188d0 0x7f127c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.897 7f12a17fa700 1 -- 172.21.15.32:0/3829020286 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f1294032a10 con 0x7f12a417c460 2021-04-09T18:34:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.897 7f129bfff700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f127c0188d0 0x7f127c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.897 7f129bfff700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f127c0188d0 0x7f127c01add0 crc :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.917 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a410d480 con 0x7f12a417c460 2021-04-09T18:34:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:25.920 7f12a17fa700 1 -- 172.21.15.32:0/3829020286 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f12940090c0 con 0x7f12a417c460 2021-04-09T18:34:26.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.114 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "allow_standby_replay", "prefix": "fs set", "fs_name": "cephfs", "val": "true"} v 0) v1 -- 0x7f12a410d480 con 0x7f12a417c460 2021-04-09T18:34:26.715 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.698 7f12a17fa700 1 -- 172.21.15.32:0/3829020286 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "allow_standby_replay", "prefix": "fs set", "fs_name": "cephfs", "val": "true"}]=0 v428) v1 ==== 121+0+0 (secure 0 0 0) 0x7f12940093e0 con 0x7f12a417c460 2021-04-09T18:34:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f127c0188d0 msgr2=0x7f127c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:26.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f127c0188d0 0x7f127c01add0 crc :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a417c460 msgr2=0x7f12a40a2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a417c460 0x7f12a40a2610 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 rx=0x7f129400b590 tx=0x7f129400b930).stop 2021-04-09T18:34:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 shutdown_connections 2021-04-09T18:34:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f127c0188d0 0x7f127c01add0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.751 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12a417a990 0x7f12a417ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12a417c860 0x7f12a40a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 --2- 172.21.15.32:0/3829020286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12a417c460 0x7f12a40a2610 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.752 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 >> 172.21.15.32:0/3829020286 conn(0x7f12a40a3ec0 msgr2=0x7f12a40bc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:26.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 shutdown_connections 2021-04-09T18:34:26.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.731 7f12aa4b0700 1 -- 172.21.15.32:0/3829020286 wait complete. 2021-04-09T18:34:26.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:34:26.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.890 7f86137c6700 1 Processor -- start 2021-04-09T18:34:26.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.890 7f86137c6700 1 -- start start 2021-04-09T18:34:26.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.891 7f86137c6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f860c10d010 0x7f860c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:26.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.891 7f86137c6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0d8490 0x7f860c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:26.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.891 7f86137c6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c10d480 con 0x7f860c0d1da0 2021-04-09T18:34:26.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.891 7f86137c6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c0d8d30 con 0x7f860c10d010 2021-04-09T18:34:26.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.891 7f86137c6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0d9220 con 0x7f860c0d8490 2021-04-09T18:34:26.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.891 7f8611de3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0d8490 0x7f860c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:26.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.891 7f86115e2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f860c10d010 0x7f860c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:26.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.891 7f8610de1700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f860c0d1da0 0x7f860c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:39136/0 (socket says 172.21.15.32:39136) 2021-04-09T18:34:26.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.891 7f8610de1700 1 -- 172.21.15.32:0/2136866645 learned_addr learned my addr 172.21.15.32:0/2136866645 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:34:26.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f8603fff700 1 -- 172.21.15.32:0/2136866645 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955828773 0 0) 0x7f85fc0010c0 con 0x7f860c0d1da0 2021-04-09T18:34:26.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f8603fff700 1 -- 172.21.15.32:0/2136866645 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f8002fe0 con 0x7f860c0d1da0 2021-04-09T18:34:26.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f86115e2700 1 -- 172.21.15.32:0/2136866645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0d8490 msgr2=0x7f860c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:34:26.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f86115e2700 1 --2- 172.21.15.32:0/2136866645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0d8490 0x7f860c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f86115e2700 1 -- 172.21.15.32:0/2136866645 >> v1:172.21.15.86:6789/0 conn(0x7f860c0d1da0 legacy=0x7f860c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:26.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f86115e2700 1 -- 172.21.15.32:0/2136866645 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c10e020 con 0x7f860c10d010 2021-04-09T18:34:26.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f8611de3700 1 --2- 172.21.15.32:0/2136866645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0d8490 0x7f860c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:26.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f86115e2700 1 --2- 172.21.15.32:0/2136866645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c10d010 0x7f860c0d1780 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f8608006650 tx=0x7f8608009a80).ready entity=mon.1 client_cookie=bb8c845b9e5c242a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:26.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f8603fff700 1 -- 172.21.15.32:0/2136866645 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86080009b0 con 0x7f860c10d010 2021-04-09T18:34:26.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f8603fff700 1 -- 172.21.15.32:0/2136866645 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860800c0c0 con 0x7f860c10d010 2021-04-09T18:34:26.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f8603fff700 1 -- 172.21.15.32:0/2136866645 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f860800c510 con 0x7f860c10d010 2021-04-09T18:34:26.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f86137c6700 1 -- 172.21.15.32:0/2136866645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c10d010 msgr2=0x7f860c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:26.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.892 7f86137c6700 1 --2- 172.21.15.32:0/2136866645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c10d010 0x7f860c0d1780 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f8608006650 tx=0x7f8608009a80).stop 2021-04-09T18:34:26.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.893 7f86137c6700 1 -- 172.21.15.32:0/2136866645 shutdown_connections 2021-04-09T18:34:26.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.893 7f86137c6700 1 --2- 172.21.15.32:0/2136866645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0d8490 0x7f860c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.893 7f86137c6700 1 --2- 172.21.15.32:0/2136866645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c10d010 0x7f860c0d1780 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.893 7f86137c6700 1 -- 172.21.15.32:0/2136866645 >> 172.21.15.32:0/2136866645 conn(0x7f860c0ce5f0 msgr2=0x7f860c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.893 7f86137c6700 1 -- 172.21.15.32:0/2136866645 shutdown_connections 2021-04-09T18:34:26.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.893 7f86137c6700 1 -- 172.21.15.32:0/2136866645 wait complete. 2021-04-09T18:34:26.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.913 7f86137c6700 1 Processor -- start 2021-04-09T18:34:26.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.913 7f86137c6700 1 -- start start 2021-04-09T18:34:26.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86137c6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c1a5210 0x7f860c1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:26.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86137c6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c1a6340 0x7f860c1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:26.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86137c6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0abaa0 0x7f860c0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:26.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86137c6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c1a6d60 con 0x7f860c1a6340 2021-04-09T18:34:26.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86137c6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0ac4c0 con 0x7f860c1a5210 2021-04-09T18:34:26.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86137c6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0ac670 con 0x7f860c0abaa0 2021-04-09T18:34:26.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8610de1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c1a6340 0x7f860c1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:26.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86115e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c1a5210 0x7f860c1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:26.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8611de3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0abaa0 0x7f860c0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:26.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86115e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c1a5210 0x7f860c1a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:51616/0 (socket says 172.21.15.32:51616) 2021-04-09T18:34:26.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86115e2700 1 -- 172.21.15.32:0/1537854275 learned_addr learned my addr 172.21.15.32:0/1537854275 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:34:26.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8611de3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0abaa0 0x7f860c0abea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:38856/0 (socket says 172.21.15.32:38856) 2021-04-09T18:34:26.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8610de1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c1a6340 0x7f860c1a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59860/0 (socket says 172.21.15.32:59860) 2021-04-09T18:34:26.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8610de1700 1 -- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0abaa0 msgr2=0x7f860c0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:26.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8610de1700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0abaa0 0x7f860c0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8610de1700 1 -- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c1a5210 msgr2=0x7f860c1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:26.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8610de1700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c1a5210 0x7f860c1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:26.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8610de1700 1 -- 172.21.15.32:0/1537854275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c0ac890 con 0x7f860c1a6340 2021-04-09T18:34:26.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8611de3700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0abaa0 0x7f860c0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:26.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f86115e2700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c1a5210 0x7f860c1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:34:26.947 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.914 7f8610de1700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c1a6340 0x7f860c1a6740 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 rx=0x7f85fc00a490 tx=0x7f85fc00a830).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:26.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.915 7f8601ffb700 1 -- 172.21.15.32:0/1537854275 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85fc00ac20 con 0x7f860c1a6340 2021-04-09T18:34:26.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.915 7f8601ffb700 1 -- 172.21.15.32:0/1537854275 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc004320 con 0x7f860c1a6340 2021-04-09T18:34:26.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.915 7f86137c6700 1 -- 172.21.15.32:0/1537854275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c0ac890 con 0x7f860c1a6340 2021-04-09T18:34:26.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.915 7f8601ffb700 1 -- 172.21.15.32:0/1537854275 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85fc004800 con 0x7f860c1a6340 2021-04-09T18:34:26.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.915 7f86137c6700 1 -- 172.21.15.32:0/1537854275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c0ac890 con 0x7f860c1a6340 2021-04-09T18:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.915 7f8601ffb700 1 -- 172.21.15.32:0/1537854275 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f85fc004500 con 0x7f860c1a6340 2021-04-09T18:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.915 7f8601ffb700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85ec018930 0x7f85ec01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.915 7f8601ffb700 1 -- 172.21.15.32:0/1537854275 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f85fc032960 con 0x7f860c1a6340 2021-04-09T18:34:26.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.916 7f86115e2700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85ec018930 0x7f85ec01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:34:26.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.916 7f86115e2700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85ec018930 0x7f85ec01ae10 crc :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:34:26.953 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.935 7f86137c6700 1 -- 172.21.15.32:0/1537854275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c1b7070 con 0x7f860c1a6340 2021-04-09T18:34:26.956 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:26.938 7f8601ffb700 1 -- 172.21.15.32:0/1537854275 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f85fc00ac20 con 0x7f860c1a6340 2021-04-09T18:34:27.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.129 7f86137c6700 1 -- 172.21.15.32:0/1537854275 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f860c1b7070 con 0x7f860c1a6340 2021-04-09T18:34:27.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.130 7f8601ffb700 1 -- 172.21.15.32:0/1537854275 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 429 v429) v1 ==== 95+0+2794 (secure 0 0 0) 0x7f85fc004320 con 0x7f860c1a6340 2021-04-09T18:34:27.149 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 429 2021-04-09T18:34:27.149 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:34:27.149 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":429,"default_fscid":22,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/4251960201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":4251960201},{"type":"v1","addr":"172.21.15.86:6836","nonce":4251960201}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59951,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1668354365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1668354365},{"type":"v1","addr":"172.21.15.86:6837","nonce":1668354365}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422}],"filesystems":[{"mdsmap":{"epoch":429,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:34:16.431259","modified":"2021-04-09 18:34:26.699543","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12379":{"gid":12379,"name":"a","rank":0,"incarnation":423,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2610523978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2610523978},{"type":"v1","addr":"172.21.15.32:6835","nonce":2610523978}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_12894":{"gid":12894,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6834/368655383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":368655383},{"type":"v1","addr":"172.21.15.32:6834","nonce":368655383}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[45],"metadata_pool":44,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":22}]} 2021-04-09T18:34:27.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 -- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85ec018930 msgr2=0x7f85ec01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:27.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85ec018930 0x7f85ec01ae10 crc :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:27.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 -- 172.21.15.32:0/1537854275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c1a6340 msgr2=0x7f860c1a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:34:27.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c1a6340 0x7f860c1a6740 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 rx=0x7f85fc00a490 tx=0x7f85fc00a830).stop 2021-04-09T18:34:27.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 -- 172.21.15.32:0/1537854275 shutdown_connections 2021-04-09T18:34:27.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f85ec018930 0x7f85ec01ae10 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:27.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f860c1a6340 0x7f860c1a6740 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:27.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f860c0abaa0 0x7f860c0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:27.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 --2- 172.21.15.32:0/1537854275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c1a5210 0x7f860c1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:34:27.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 -- 172.21.15.32:0/1537854275 >> 172.21.15.32:0/1537854275 conn(0x7f860c1a99c0 msgr2=0x7f860c17dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:34:27.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 -- 172.21.15.32:0/1537854275 shutdown_connections 2021-04-09T18:34:27.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:34:27.132 7f86137c6700 1 -- 172.21.15.32:0/1537854275 wait complete. 2021-04-09T18:34:27.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:27.563 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:27.563 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:27.706 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:27.706 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:27.706 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:27.707 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:27.707 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:27.708 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:27.708 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 11.256463321, 2021-04-09T18:34:27.709 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 429, 2021-04-09T18:34:27.709 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 263, 2021-04-09T18:34:27.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 263, 2021-04-09T18:34:27.710 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 16.698578939000001 2021-04-09T18:34:27.710 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:27.712 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 11.256463321, "mdsmap_epoch": 429, "osdmap_epoch": 263, "osdmap_epoch_barrier": 263, "uptime": 16.698578939000001 } 2021-04-09T18:34:31.714 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:34:31.714 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:34:31.715 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:34:31.715 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:31.725 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:34:31.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:34:31.767 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:34:31.923 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:32.170 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:32.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:32.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:32.313 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:32.313 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:32.313 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:32.314 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:32.314 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:32.314 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:32.314 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 15.861605437, 2021-04-09T18:34:32.314 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 429, 2021-04-09T18:34:32.315 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 263, 2021-04-09T18:34:32.315 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 263, 2021-04-09T18:34:32.315 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 21.303723093999999 2021-04-09T18:34:32.316 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:32.317 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 15.861605437, "mdsmap_epoch": 429, "osdmap_epoch": 263, "osdmap_epoch_barrier": 263, "uptime": 21.303723093999999 } 2021-04-09T18:34:34.318 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:34:34.319 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:34:34.320 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:34:34.320 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:34.330 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:34:34.331 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:34:34.373 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:34:34.530 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:34.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:34.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:34.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:34.915 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:34.916 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:34.916 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:34.916 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:34.917 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:34.917 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:34.918 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 18.465773453000001, 2021-04-09T18:34:34.918 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 429, 2021-04-09T18:34:34.919 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 263, 2021-04-09T18:34:34.919 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 263, 2021-04-09T18:34:34.919 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 23.907891330999998 2021-04-09T18:34:34.920 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:34.922 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.465773453000001, "mdsmap_epoch": 429, "osdmap_epoch": 263, "osdmap_epoch_barrier": 263, "uptime": 23.907891330999998 } 2021-04-09T18:34:38.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:34:38.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:34:38.923 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:34:38.924 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:34:38.924 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:34:38.925 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:38.938 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:34:38.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:34:38.981 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:34:39.126 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:39.384 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:39.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:39.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:39.523 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:39.524 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:39.524 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:39.524 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:39.524 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:39.525 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:39.525 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 23.073987536000001, 2021-04-09T18:34:39.525 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 431, 2021-04-09T18:34:39.525 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 264, 2021-04-09T18:34:39.526 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 264, 2021-04-09T18:34:39.526 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 28.516102996000001 2021-04-09T18:34:39.526 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:39.529 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 23.073987536000001, "mdsmap_epoch": 431, "osdmap_epoch": 264, "osdmap_epoch_barrier": 264, "uptime": 28.516102996000001 } 2021-04-09T18:34:41.530 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:34:41.531 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:34:41.531 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:34:41.532 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:41.542 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:34:41.543 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:34:41.585 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:34:41.741 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:41.988 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:41.988 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:41.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:42.126 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:42.127 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:42.127 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:42.127 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:42.128 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:42.128 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:42.128 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 25.677351088000002, 2021-04-09T18:34:42.128 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 431, 2021-04-09T18:34:42.129 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 264, 2021-04-09T18:34:42.129 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 264, 2021-04-09T18:34:42.129 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 31.119468422000001 2021-04-09T18:34:42.129 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:42.133 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 25.677351088000002, "mdsmap_epoch": 431, "osdmap_epoch": 264, "osdmap_epoch_barrier": 264, "uptime": 31.119468422000001 } 2021-04-09T18:34:44.134 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:34:44.135 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:34:44.136 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:34:44.136 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:44.144 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:34:44.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:34:44.187 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:34:44.336 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:44.590 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:44.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:44.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:44.729 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:44.729 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:44.729 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:44.730 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:44.730 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:44.730 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:44.730 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 28.279326914999999, 2021-04-09T18:34:44.731 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 431, 2021-04-09T18:34:44.731 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 264, 2021-04-09T18:34:44.731 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 264, 2021-04-09T18:34:44.731 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 33.721444595999998 2021-04-09T18:34:44.732 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:44.735 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 28.279326914999999, "mdsmap_epoch": 431, "osdmap_epoch": 264, "osdmap_epoch_barrier": 264, "uptime": 33.721444595999998 } 2021-04-09T18:34:48.736 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:34:48.737 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:34:48.738 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:34:48.738 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:48.747 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:34:48.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:34:48.789 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:34:48.936 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:49.194 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:49.195 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:49.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:49.334 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:49.334 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:49.335 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:49.335 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:49.335 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:49.336 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:49.336 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 32.884194747999999, 2021-04-09T18:34:49.337 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 433, 2021-04-09T18:34:49.337 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 265, 2021-04-09T18:34:49.338 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 265, 2021-04-09T18:34:49.338 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 38.326312188000003 2021-04-09T18:34:49.339 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:49.340 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 32.884194747999999, "mdsmap_epoch": 433, "osdmap_epoch": 265, "osdmap_epoch_barrier": 265, "uptime": 38.326312188000003 } 2021-04-09T18:34:52.342 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:34:52.343 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:34:52.344 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:34:52.344 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:52.353 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:34:52.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:34:52.395 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:34:52.545 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:52.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:52.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:52.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:52.937 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 36.487245063000003, 2021-04-09T18:34:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 433, 2021-04-09T18:34:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 265, 2021-04-09T18:34:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 265, 2021-04-09T18:34:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 41.929360655000004 2021-04-09T18:34:53.070 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:53.072 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 36.487245063000003, "mdsmap_epoch": 433, "osdmap_epoch": 265, "osdmap_epoch_barrier": 265, "uptime": 41.929360655000004 } 2021-04-09T18:34:55.074 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:34:55.074 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:34:55.075 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:34:55.075 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:34:55.086 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:34:55.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:34:55.128 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:34:55.285 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:34:55.532 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:34:55.533 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:34:55.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:34:55.675 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:34:55.676 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:34:55.676 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:34:55.676 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:34:55.677 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:34:55.677 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:34:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 39.225692721999998, 2021-04-09T18:34:55.678 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 433, 2021-04-09T18:34:55.679 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 265, 2021-04-09T18:34:55.679 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 265, 2021-04-09T18:34:55.679 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 44.667807875999998 2021-04-09T18:34:55.680 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:34:55.682 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 39.225692721999998, "mdsmap_epoch": 433, "osdmap_epoch": 265, "osdmap_epoch_barrier": 265, "uptime": 44.667807875999998 } 2021-04-09T18:35:00.683 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:35:00.684 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:35:00.684 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:35:00.685 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:35:00.694 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:35:00.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:35:00.736 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:35:00.889 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:35:01.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:35:01.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:35:01.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:35:01.277 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:35:01.277 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:35:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:35:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:35:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:35:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:35:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 44.827701658999999, 2021-04-09T18:35:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 435, 2021-04-09T18:35:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 266, 2021-04-09T18:35:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 266, 2021-04-09T18:35:01.280 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 50.269819097000003 2021-04-09T18:35:01.280 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:35:01.283 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 44.827701658999999, "mdsmap_epoch": 435, "osdmap_epoch": 266, "osdmap_epoch_barrier": 266, "uptime": 50.269819097000003 } 2021-04-09T18:35:03.285 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:35:03.286 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:35:03.286 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:35:03.287 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:35:03.296 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:35:03.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:35:03.339 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:35:03.484 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:35:03.742 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 429, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:34:16.431259', 'modified': '2021-04-09 18:34:26.699543', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12379': {'gid': 12379, 'name': 'a', 'rank': 0, 'incarnation': 423, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6835/2610523978', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6833', 'nonce': 2610523978}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 2610523978}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_12894': {'gid': 12894, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/368655383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 368655383}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 368655383}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [45], 'metadata_pool': 44, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:35:03.742 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:35:03.743 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-04-09T18:35:03.885 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:35:03.885 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:35:03.885 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2021-04-09T18:35:03.886 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 12379, 2021-04-09T18:35:03.886 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2021-04-09T18:35:03.887 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2021-04-09T18:35:03.887 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 47.435524434999998, 2021-04-09T18:35:03.887 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 435, 2021-04-09T18:35:03.887 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 266, 2021-04-09T18:35:03.888 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 266, 2021-04-09T18:35:03.888 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 52.877641236999999 2021-04-09T18:35:03.888 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:35:03.891 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12379, "want_state": "up:active", "state": "up:active", "rank_uptime": 47.435524434999998, "mdsmap_epoch": 435, "osdmap_epoch": 266, "osdmap_epoch_barrier": 266, "uptime": 52.877641236999999 } 2021-04-09T18:35:03.892 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:35:03.892 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:35:03.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:35:04.010 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:35:04.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:04.011 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:35:04.012 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:35:04.057 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:35:04.058 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:35:04.058 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:35:04.059 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:35:04.059 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:35:04.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[109035]: fuse finished with error 0 and tester_r 0 2021-04-09T18:35:04.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:04.127 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:35:04.127 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:35:04.171 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:35:04.172 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:35:04.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:35:04.279 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:35:04.362 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:04.363 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:35:04.364 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:35:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:35:04.408 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:35:04.409 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:35:04.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:35:04.410 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:35:04.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:35:04.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.621 7efe0498f700 1 Processor -- start 2021-04-09T18:35:04.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.621 7efe0498f700 1 -- start start 2021-04-09T18:35:04.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efe0498f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0010d010 0x7efe000d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:04.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efe0498f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe000d1da0 0x7efe000d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:04.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efe0498f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efe000d82c0 0x7efe0010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:04.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efe0498f700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0010e780 con 0x7efe000d1da0 2021-04-09T18:35:04.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efe0498f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe000d8d30 con 0x7efe000d82c0 2021-04-09T18:35:04.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efe0498f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe000d9220 con 0x7efe0010d010 2021-04-09T18:35:04.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfde1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe000d1da0 0x7efe000d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:04.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfe61c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0010d010 0x7efe000d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:04.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfee1d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efe000d82c0 0x7efe0010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:04.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfde1b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efe000d1da0 0x7efe000d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60022/0 (socket says 172.21.15.32:60022) 2021-04-09T18:35:04.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfde1b700 1 -- 172.21.15.32:0/2036814673 learned_addr learned my addr 172.21.15.32:0/2036814673 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:04.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfe61c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0010d010 0x7efe000d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:39012/0 (socket says 172.21.15.32:39012) 2021-04-09T18:35:04.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfde1b700 1 -- 172.21.15.32:0/2036814673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0010d010 msgr2=0x7efe000d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:04.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfde1b700 1 --2- 172.21.15.32:0/2036814673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0010d010 0x7efe000d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:04.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfde1b700 1 -- 172.21.15.32:0/2036814673 >> v2:172.21.15.86:3300/0 conn(0x7efe000d82c0 msgr2=0x7efe0010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:04.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfde1b700 1 --2- 172.21.15.32:0/2036814673 >> v2:172.21.15.86:3300/0 conn(0x7efe000d82c0 0x7efe0010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:04.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfde1b700 1 -- 172.21.15.32:0/2036814673 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0010d480 con 0x7efe000d1da0 2021-04-09T18:35:04.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfe61c700 1 --2- 172.21.15.32:0/2036814673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0010d010 0x7efe000d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:04.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfee1d700 1 --2- 172.21.15.32:0/2036814673 >> v2:172.21.15.86:3300/0 conn(0x7efe000d82c0 0x7efe0010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:04.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfde1b700 1 --2- 172.21.15.32:0/2036814673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe000d1da0 0x7efe000d7ca0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7efde8006700 tx=0x7efde8009b80).ready entity=mon.0 client_cookie=659a34a6c7861c17 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:04.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.622 7efdfd61a700 1 -- 172.21.15.32:0/2036814673 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efde800c0c0 con 0x7efe000d1da0 2021-04-09T18:35:04.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efdfd61a700 1 -- 172.21.15.32:0/2036814673 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde800c440 con 0x7efe000d1da0 2021-04-09T18:35:04.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efdfd61a700 1 -- 172.21.15.32:0/2036814673 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efde800c920 con 0x7efe000d1da0 2021-04-09T18:35:04.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efe0498f700 1 -- 172.21.15.32:0/2036814673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe000d1da0 msgr2=0x7efe000d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:04.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efe0498f700 1 --2- 172.21.15.32:0/2036814673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe000d1da0 0x7efe000d7ca0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7efde8006700 tx=0x7efde8009b80).stop 2021-04-09T18:35:04.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efe0498f700 1 -- 172.21.15.32:0/2036814673 shutdown_connections 2021-04-09T18:35:04.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efe0498f700 1 --2- 172.21.15.32:0/2036814673 >> v2:172.21.15.86:3300/0 conn(0x7efe000d82c0 0x7efe0010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:04.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efe0498f700 1 --2- 172.21.15.32:0/2036814673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe000d1da0 0x7efe000d7ca0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:04.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efe0498f700 1 --2- 172.21.15.32:0/2036814673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0010d010 0x7efe000d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:04.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efe0498f700 1 -- 172.21.15.32:0/2036814673 >> 172.21.15.32:0/2036814673 conn(0x7efe000ce5f0 msgr2=0x7efe000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:04.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.623 7efe0498f700 1 -- 172.21.15.32:0/2036814673 shutdown_connections 2021-04-09T18:35:04.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.624 7efe0498f700 1 -- 172.21.15.32:0/2036814673 wait complete. 2021-04-09T18:35:04.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efe0498f700 1 Processor -- start 2021-04-09T18:35:04.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efe0498f700 1 -- start start 2021-04-09T18:35:04.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efe0498f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe001aca20 0x7efe001ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:04.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efe0498f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0018bd40 0x7efe0018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:04.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efe0498f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe0008fba0 0x7efe0008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:04.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efe0498f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0018c7e0 con 0x7efe001aca20 2021-04-09T18:35:04.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efe0498f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0018ca30 con 0x7efe0008fba0 2021-04-09T18:35:04.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efe0498f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe000905f0 con 0x7efe0018bd40 2021-04-09T18:35:04.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efdfe61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe001aca20 0x7efe001ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:04.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efdfe61c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe001aca20 0x7efe001ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60024/0 (socket says 172.21.15.32:60024) 2021-04-09T18:35:04.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efdfe61c700 1 -- 172.21.15.32:0/1524937407 learned_addr learned my addr 172.21.15.32:0/1524937407 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:04.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efdfee1d700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe0008fba0 0x7efe0008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:04.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.644 7efdfde1b700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0018bd40 0x7efe0018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:04.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdfe61c700 1 -- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0018bd40 msgr2=0x7efe0018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:04.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdfe61c700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0018bd40 0x7efe0018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:04.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdfe61c700 1 -- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe0008fba0 msgr2=0x7efe0008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:04.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdfe61c700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe0008fba0 0x7efe0008ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:04.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdfe61c700 1 -- 172.21.15.32:0/1524937407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe000b4300 con 0x7efe001aca20 2021-04-09T18:35:04.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdfde1b700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0018bd40 0x7efe0018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:04.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdfe61c700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe001aca20 0x7efe001ace20 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7efdf800b970 tx=0x7efdf800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:04.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdf6ffd700 1 -- 172.21.15.32:0/1524937407 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdf8000d30 con 0x7efe001aca20 2021-04-09T18:35:04.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdf6ffd700 1 -- 172.21.15.32:0/1524937407 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf8006020 con 0x7efe001aca20 2021-04-09T18:35:04.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efe0498f700 1 -- 172.21.15.32:0/1524937407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe000b4300 con 0x7efe001aca20 2021-04-09T18:35:04.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efdf6ffd700 1 -- 172.21.15.32:0/1524937407 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdf80064f0 con 0x7efe001aca20 2021-04-09T18:35:04.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.645 7efe0498f700 1 -- 172.21.15.32:0/1524937407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe000b4300 con 0x7efe001aca20 2021-04-09T18:35:04.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.646 7efdf6ffd700 1 -- 172.21.15.32:0/1524937407 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efdf8006410 con 0x7efe001aca20 2021-04-09T18:35:04.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.646 7efdf6ffd700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efde40188c0 0x7efde401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:04.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.646 7efdf6ffd700 1 -- 172.21.15.32:0/1524937407 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efdf8032d10 con 0x7efe001aca20 2021-04-09T18:35:04.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.646 7efdfde1b700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efde40188c0 0x7efde401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:04.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.646 7efdfde1b700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efde40188c0 0x7efde401adc0 crc :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:04.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.666 7efe0498f700 1 -- 172.21.15.32:0/1524937407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe00182d70 con 0x7efe001aca20 2021-04-09T18:35:04.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.669 7efdf6ffd700 1 -- 172.21.15.32:0/1524937407 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efdf8006000 con 0x7efe001aca20 2021-04-09T18:35:04.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.859 7efe0498f700 1 -- 172.21.15.32:0/1524937407 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7efe0018c760 con 0x7efe001aca20 2021-04-09T18:35:04.877 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.860 7efdf6ffd700 1 -- 172.21.15.32:0/1524937407 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v266) v1 ==== 81+0+30374 (secure 0 0 0) 0x7efdf8006210 con 0x7efe001aca20 2021-04-09T18:35:04.880 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:04.880 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:35:04.881 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 266, 2021-04-09T18:35:04.881 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:35:04.881 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:35:04.881 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:34:59.272215", 2021-04-09T18:35:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:35:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:35:04.882 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:35:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:35:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:35:04.883 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:35:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:35:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:35:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:35:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:35:04.884 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:35:04.885 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:35:04.885 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:35:04.885 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:35:04.885 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:35:04.886 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 45, 2021-04-09T18:35:04.886 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:35:04.886 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:35:04.886 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:35:04.886 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:35:04.887 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:35:04.887 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.887 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:35:04.887 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:35:04.888 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:35:04.888 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:35:04.889 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:35:04.889 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:35:04.889 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:35:04.890 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:35:04.890 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:35:04.891 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:35:04.891 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:35:04.891 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:35:04.892 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:35:04.892 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:35:04.893 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:35:04.893 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:35:04.893 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:35:04.894 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:35:04.894 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:35:04.895 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:35:04.896 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:35:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:35:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:35:04.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:35:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:35:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:35:04.898 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:35:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:35:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:35:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:35:04.899 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:35:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:35:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:35:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:35:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:35:04.900 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:35:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:35:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:35:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:35:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:35:04.901 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:35:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:35:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:35:04.902 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:35:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:35:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:35:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:35:04.903 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:35:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:35:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:35:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:35:04.904 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:35:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:35:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:35:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:35:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:35:04.905 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:35:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:35:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:35:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:35:04.906 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:35:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:35:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:35:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:35:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 44, 2021-04-09T18:35:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:35:04.908 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:34:13.713911", 2021-04-09T18:35:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:35:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:35:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:35:04.909 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:35:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:35:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:35:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:35:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:35:04.910 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:35:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:35:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:35:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:35:04.911 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:35:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:35:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:35:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:35:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:35:04.912 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:35:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:35:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:35:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "262", 2021-04-09T18:35:04.913 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:35:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:35:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:35:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:35:04.914 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:35:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:35:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:35:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:35:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:35:04.915 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:35:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:35:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:35:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:35:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:35:04.916 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:35:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:35:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:35:04.918 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:35:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:35:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:35:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:35:04.919 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:35:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:35:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:35:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:35:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:35:04.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:35:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:35:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:35:04.921 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:35:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:35:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:35:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:35:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:35:04.922 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:35:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:35:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:35:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:35:04.923 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:35:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:35:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:35:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:35:04.924 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:35:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:35:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.925 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 45, 2021-04-09T18:35:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:35:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:34:15.142274", 2021-04-09T18:35:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:35:04.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:35:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:35:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:35:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:35:04.927 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:35:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:35:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:35:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:35:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:35:04.928 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:35:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:35:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:35:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:35:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:35:04.929 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:35:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:35:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:35:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:35:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:35:04.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "263", 2021-04-09T18:35:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:35:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:35:04.931 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:35:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:35:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:35:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:35:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:35:04.932 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:35:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:35:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:35:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:35:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:35:04.933 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:35:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:35:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:35:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:35:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:35:04.934 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:35:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:35:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:35:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:35:04.935 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:35:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:35:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:35:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:35:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:35:04.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:35:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:35:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:35:04.937 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:35:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:35:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:35:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:35:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:35:04.938 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:35:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:35:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:35:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:35:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:35:04.939 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:35:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:35:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.940 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:35:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:35:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.941 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:35:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:35:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:04.942 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:04.943 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.944 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:35:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.945 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:35:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.946 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.947 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:35:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.948 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:35:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:04.949 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.950 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.951 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:04.952 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.953 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.955 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:35:04.956 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:04.957 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:35:04.958 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:04.959 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 256, 2021-04-09T18:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:04.960 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.961 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.962 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.963 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:35:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.965 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:35:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.966 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:35:04.967 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.968 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:35:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.969 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:35:04.970 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:35:04.971 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.972 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:35:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:35:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:35:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:35:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:04.973 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:35:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:35:04.975 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:04.976 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:04.977 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:35:04.978 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.979 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:35:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.980 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:35:04.981 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:35:04.982 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.983 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:35:04.984 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:35:04.985 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:04.986 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:35:04.987 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:35:04.988 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.989 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:35:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:35:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:35:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:35:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:04.990 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:04.991 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:04.991 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.991 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.992 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:35:04.992 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:35:04.992 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:04.992 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:04.993 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:04.993 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:04.993 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:04.993 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:04.993 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:04.994 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:04.994 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:04.994 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:04.994 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:04.995 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.995 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.995 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.995 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:35:04.996 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:04.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.996 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.996 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:04.996 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:35:04.997 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:04.997 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:04.997 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:04.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.998 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:04.998 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:04.998 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:04.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:04.999 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:35:04.999 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:04.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:04.999 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.000 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:35:05.000 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:05.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.001 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.001 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:05.001 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.001 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.002 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.002 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:35:05.002 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:05.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.003 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.003 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:35:05.003 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:05.003 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.004 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.004 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:05.004 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.004 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.005 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:35:05.005 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:05.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.006 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.006 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.006 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:35:05.006 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:05.007 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.007 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.007 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:35:05.007 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:35:05.008 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:35:05.008 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:35:05.008 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:05.008 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:05.008 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:05.009 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.009 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.009 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:35:05.009 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:35:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:05.011 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:05.011 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:05.011 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:05.011 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:05.011 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:05.012 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:05.012 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:05.012 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.012 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.013 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:35:05.013 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:05.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.013 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.014 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:35:05.014 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:05.014 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.014 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.015 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:05.015 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.015 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.016 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:35:05.016 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:05.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.016 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.016 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.017 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:35:05.017 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:05.017 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.017 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.018 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:05.018 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.018 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.018 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.019 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:35:05.019 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:05.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.019 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.019 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.020 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:35:05.020 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:05.020 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.020 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:35:05.022 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:05.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.022 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.022 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:35:05.023 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:05.023 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.023 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.024 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:35:05.024 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:35:05.024 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:35:05.024 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:35:05.024 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:05.025 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:05.025 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:05.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.025 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.026 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:35:05.026 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:35:05.026 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:05.026 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:05.026 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:05.027 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:05.027 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:05.027 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:05.027 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 256, 2021-04-09T18:35:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:05.028 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.029 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.029 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:35:05.029 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:05.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.030 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.030 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.030 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:35:05.030 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:05.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.031 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.031 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:05.031 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.032 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.032 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:35:05.032 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:05.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:35:05.033 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:05.033 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.034 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.035 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:35:05.035 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:05.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.036 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.036 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:35:05.036 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:05.036 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.037 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.038 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:35:05.038 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:05.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.038 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:35:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.039 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:35:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:35:05.040 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:35:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:35:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:05.041 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:35:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:35:05.042 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:05.043 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:05.043 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:05.043 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:05.043 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:05.043 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:05.044 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:05.044 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:05.044 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:05.044 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:05.044 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:05.045 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.045 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.045 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.045 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:35:05.046 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:05.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.046 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.046 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.046 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:35:05.047 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:05.047 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.047 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.047 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:05.048 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.048 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.048 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:35:05.049 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:05.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.049 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.049 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:35:05.050 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:05.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.050 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.050 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:05.051 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.051 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.051 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:35:05.051 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:05.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.052 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.052 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:35:05.053 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:05.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.053 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.053 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:05.054 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.054 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.054 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.054 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:35:05.054 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:05.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.055 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.055 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:35:05.055 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:05.056 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.056 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.056 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:35:05.056 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:35:05.057 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:35:05.057 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:35:05.057 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:05.057 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:05.058 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:05.058 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.058 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.058 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:35:05.059 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:35:05.059 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:05.059 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:05.059 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:05.059 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:05.060 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:05.060 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:05.060 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:05.060 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:05.061 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:05.061 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:05.061 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:05.061 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.062 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:35:05.062 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:05.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.063 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:35:05.063 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:05.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.063 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.064 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:05.064 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.064 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.064 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.065 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:35:05.065 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:05.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.066 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:35:05.066 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:05.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.066 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.067 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:05.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.067 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.068 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:35:05.068 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:05.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.068 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.069 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:35:05.069 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:05.069 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.069 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.069 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.070 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:05.070 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:05.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.070 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:05.070 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:35:05.071 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:05.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.071 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.071 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:05.072 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:35:05.072 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:05.072 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.072 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.073 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:35:05.073 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:35:05.073 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:35:05.073 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:35:05.073 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:05.074 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:05.074 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:05.074 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:05.074 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.075 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:35:05.076 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:35:05.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.076 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:35:05.076 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:05.077 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:05.077 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:05.077 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:05.077 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:05.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.078 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.078 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:35:05.078 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:05.078 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:05.078 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:05.079 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:05.079 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:05.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.079 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.080 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:35:05.080 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:05.080 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:05.080 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:05.080 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:05.081 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:05.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.081 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:35:05.081 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:05.082 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:05.082 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:05.082 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:05.082 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:05.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.083 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:35:05.083 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:05.083 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:05.084 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:05.084 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:05.084 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:05.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.085 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:35:05.085 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:05.085 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:05.085 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:05.086 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:05.086 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:05.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.086 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.086 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:35:05.087 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:05.087 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:05.087 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:05.087 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:05.088 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:05.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.088 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:05.088 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:35:05.088 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:05.089 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:05.089 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:05.089 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:05.089 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:05.090 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.090 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:35:05.090 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:35:05.090 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:35:05.090 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:35:05.091 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:35:05.091 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:35:05.091 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:35:05.091 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:35:05.091 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:35:05.092 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:35:05.092 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:35:05.092 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:35:05.092 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:35:05.092 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:35:05.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:05.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:05.093 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:35:05.093 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:35:05.094 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:35:05.094 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:35:05.094 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:35:05.094 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:35:05.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.864 7efe0498f700 1 -- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efde40188c0 msgr2=0x7efde401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.864 7efe0498f700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efde40188c0 0x7efde401adc0 crc :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.864 7efe0498f700 1 -- 172.21.15.32:0/1524937407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe001aca20 msgr2=0x7efe001ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.864 7efe0498f700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe001aca20 0x7efe001ace20 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7efdf800b970 tx=0x7efdf800bd10).stop 2021-04-09T18:35:05.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.865 7efe0498f700 1 -- 172.21.15.32:0/1524937407 shutdown_connections 2021-04-09T18:35:05.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.865 7efe0498f700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efde40188c0 0x7efde401adc0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.865 7efe0498f700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe0008fba0 0x7efe0008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.865 7efe0498f700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe0018bd40 0x7efe0018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.865 7efe0498f700 1 --2- 172.21.15.32:0/1524937407 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe001aca20 0x7efe001ace20 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.865 7efe0498f700 1 -- 172.21.15.32:0/1524937407 >> 172.21.15.32:0/1524937407 conn(0x7efe0019b460 msgr2=0x7efe0019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:05.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.865 7efe0498f700 1 -- 172.21.15.32:0/1524937407 shutdown_connections 2021-04-09T18:35:05.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:04.865 7efe0498f700 1 -- 172.21.15.32:0/1524937407 wait complete. 2021-04-09T18:35:05.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:35:05.277 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.232 7f4f403b7700 1 Processor -- start 2021-04-09T18:35:05.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.232 7f4f403b7700 1 -- start start 2021-04-09T18:35:05.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f403b7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3810d010 0x7f4f380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f403b7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4f380d1da0 0x7f4f380d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f403b7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3810d480 con 0x7f4f380d82c0 2021-04-09T18:35:05.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f403b7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3810e780 con 0x7f4f380d1da0 2021-04-09T18:35:05.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f403b7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f380d8d30 con 0x7f4f3810d010 2021-04-09T18:35:05.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f3d9d2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4f380d1da0 0x7f4f380d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f3e1d3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3810d010 0x7f4f380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f3d9d2700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4f380d1da0 0x7f4f380d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:51794/0 (socket says 172.21.15.32:51794) 2021-04-09T18:35:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f3d9d2700 1 -- 172.21.15.32:0/1884057786 learned_addr learned my addr 172.21.15.32:0/1884057786 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f3e9d4700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4f380d82c0 0x7f4f3810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:39318/0 (socket says 172.21.15.32:39318) 2021-04-09T18:35:05.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f3e1d3700 1 -- 172.21.15.32:0/1884057786 >> v2:172.21.15.86:3300/0 conn(0x7f4f380d1da0 msgr2=0x7f4f380d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.233 7f4f3e1d3700 1 --2- 172.21.15.32:0/1884057786 >> v2:172.21.15.86:3300/0 conn(0x7f4f380d1da0 0x7f4f380d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f3e1d3700 1 -- 172.21.15.32:0/1884057786 >> v1:172.21.15.86:6789/0 conn(0x7f4f380d82c0 legacy=0x7f4f3810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f3e1d3700 1 -- 172.21.15.32:0/1884057786 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f380d9220 con 0x7f4f3810d010 2021-04-09T18:35:05.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f3d1d1700 1 -- 172.21.15.32:0/1884057786 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628762081 0 0) 0x7f4f30000d30 con 0x7f4f380d82c0 2021-04-09T18:35:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f3d9d2700 1 --2- 172.21.15.32:0/1884057786 >> v2:172.21.15.86:3300/0 conn(0x7f4f380d1da0 0x7f4f380d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f3e1d3700 1 --2- 172.21.15.32:0/1884057786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3810d010 0x7f4f380d1780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f4f34006650 tx=0x7f4f34009a80).ready entity=mon.2 client_cookie=d22c988212f203a7 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:05.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f3d1d1700 1 -- 172.21.15.32:0/1884057786 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f340009b0 con 0x7f4f3810d010 2021-04-09T18:35:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f3d1d1700 1 -- 172.21.15.32:0/1884057786 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3400c0c0 con 0x7f4f3810d010 2021-04-09T18:35:05.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f3d1d1700 1 -- 172.21.15.32:0/1884057786 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f3400c510 con 0x7f4f3810d010 2021-04-09T18:35:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f403b7700 1 -- 172.21.15.32:0/1884057786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3810d010 msgr2=0x7f4f380d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.287 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.234 7f4f403b7700 1 --2- 172.21.15.32:0/1884057786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3810d010 0x7f4f380d1780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f4f34006650 tx=0x7f4f34009a80).stop 2021-04-09T18:35:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.235 7f4f403b7700 1 -- 172.21.15.32:0/1884057786 shutdown_connections 2021-04-09T18:35:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.235 7f4f403b7700 1 --2- 172.21.15.32:0/1884057786 >> v2:172.21.15.86:3300/0 conn(0x7f4f380d1da0 0x7f4f380d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.288 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.235 7f4f403b7700 1 --2- 172.21.15.32:0/1884057786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3810d010 0x7f4f380d1780 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.235 7f4f403b7700 1 -- 172.21.15.32:0/1884057786 >> 172.21.15.32:0/1884057786 conn(0x7f4f380ce5f0 msgr2=0x7f4f380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:05.289 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.235 7f4f403b7700 1 -- 172.21.15.32:0/1884057786 shutdown_connections 2021-04-09T18:35:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.235 7f4f403b7700 1 -- 172.21.15.32:0/1884057786 wait complete. 2021-04-09T18:35:05.290 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.255 7f4f403b7700 1 Processor -- start 2021-04-09T18:35:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.255 7f4f403b7700 1 -- start start 2021-04-09T18:35:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.255 7f4f403b7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f381aca20 0x7f4f381ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.255 7f4f403b7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3818bd40 0x7f4f3818c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.255 7f4f403b7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3808fba0 0x7f4f3808ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.292 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.255 7f4f403b7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3818c7e0 con 0x7f4f381aca20 2021-04-09T18:35:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.255 7f4f403b7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3818ca30 con 0x7f4f3808fba0 2021-04-09T18:35:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.255 7f4f403b7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f380905f0 con 0x7f4f3818bd40 2021-04-09T18:35:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e1d3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f381aca20 0x7f4f381ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e1d3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f381aca20 0x7f4f381ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60042/0 (socket says 172.21.15.32:60042) 2021-04-09T18:35:05.294 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e1d3700 1 -- 172.21.15.32:0/2239261201 learned_addr learned my addr 172.21.15.32:0/2239261201 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3d9d2700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3818bd40 0x7f4f3818c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e9d4700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3808fba0 0x7f4f3808ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e1d3700 1 -- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3818bd40 msgr2=0x7f4f3818c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e1d3700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3818bd40 0x7f4f3818c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e1d3700 1 -- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3808fba0 msgr2=0x7f4f3808ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e1d3700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3808fba0 0x7f4f3808ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e1d3700 1 -- 172.21.15.32:0/2239261201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f380b4300 con 0x7f4f381aca20 2021-04-09T18:35:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3d9d2700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3818bd40 0x7f4f3818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e9d4700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3808fba0 0x7f4f3808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:05.298 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f3e1d3700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f381aca20 0x7f4f381ace20 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 rx=0x7f4f34009650 tx=0x7f4f3400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f2effd700 1 -- 172.21.15.32:0/2239261201 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f340093a0 con 0x7f4f381aca20 2021-04-09T18:35:05.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f2effd700 1 -- 172.21.15.32:0/2239261201 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3400bd50 con 0x7f4f381aca20 2021-04-09T18:35:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f380b4300 con 0x7f4f381aca20 2021-04-09T18:35:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f2effd700 1 -- 172.21.15.32:0/2239261201 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f340052e0 con 0x7f4f381aca20 2021-04-09T18:35:05.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.256 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f380b4300 con 0x7f4f381aca20 2021-04-09T18:35:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.257 7f4f2effd700 1 -- 172.21.15.32:0/2239261201 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f4f34006030 con 0x7f4f381aca20 2021-04-09T18:35:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.257 7f4f2effd700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f20018930 0x7f4f2001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.257 7f4f2effd700 1 -- 172.21.15.32:0/2239261201 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f4f34032a10 con 0x7f4f381aca20 2021-04-09T18:35:05.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.257 7f4f3d9d2700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f20018930 0x7f4f2001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.257 7f4f3d9d2700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f20018930 0x7f4f2001ae10 crc :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:05.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.277 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38182d70 con 0x7f4f381aca20 2021-04-09T18:35:05.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.280 7f4f2effd700 1 -- 172.21.15.32:0/2239261201 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f4f34000940 con 0x7f4f381aca20 2021-04-09T18:35:05.488 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.471 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f3818c760 con 0x7f4f381aca20 2021-04-09T18:35:05.489 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.471 7f4f2effd700 1 -- 172.21.15.32:0/2239261201 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 435 v435) v1 ==== 95+0+2797 (secure 0 0 0) 0x7f4f3400bd50 con 0x7f4f381aca20 2021-04-09T18:35:05.489 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 435 2021-04-09T18:35:05.489 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:05.490 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":435,"default_fscid":22,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/4251960201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":4251960201},{"type":"v1","addr":"172.21.15.86:6836","nonce":4251960201}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59951,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1668354365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1668354365},{"type":"v1","addr":"172.21.15.86:6837","nonce":1668354365}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422}],"filesystems":[{"mdsmap":{"epoch":435,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:34:16.431259","modified":"2021-04-09 18:34:59.578203","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12379":{"gid":12379,"name":"a","rank":0,"incarnation":423,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6835/2610523978","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":2610523978},{"type":"v1","addr":"172.21.15.32:6835","nonce":2610523978}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_13113":{"gid":13113,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6834/1308180432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1308180432},{"type":"v1","addr":"172.21.15.32:6834","nonce":1308180432}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[45],"metadata_pool":44,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":22}]} 2021-04-09T18:35:05.491 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.473 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f20018930 msgr2=0x7f4f2001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.473 7f4f403b7700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f20018930 0x7f4f2001ae10 crc :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f381aca20 msgr2=0x7f4f381ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f381aca20 0x7f4f381ace20 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 rx=0x7f4f34009650 tx=0x7f4f3400b960).stop 2021-04-09T18:35:05.492 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 shutdown_connections 2021-04-09T18:35:05.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f4f20018930 0x7f4f2001ae10 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3808fba0 0x7f4f3808ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f3818bd40 0x7f4f3818c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.493 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 --2- 172.21.15.32:0/2239261201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f381aca20 0x7f4f381ace20 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 >> 172.21.15.32:0/2239261201 conn(0x7f4f3819b460 msgr2=0x7f4f3819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:05.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 shutdown_connections 2021-04-09T18:35:05.494 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.474 7f4f403b7700 1 -- 172.21.15.32:0/2239261201 wait complete. 2021-04-09T18:35:05.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:35:05.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.637 7fe5ce4ae700 1 Processor -- start 2021-04-09T18:35:05.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.637 7fe5ce4ae700 1 -- start start 2021-04-09T18:35:05.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5ce4ae700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe5c80cffb0 0x7fe5c80d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5ce4ae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c80d7810 0x7fe5c80d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5ce4ae700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5c80d8230 0x7fe5c80d5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5ce4ae700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c80d91f0 con 0x7fe5c80d8230 2021-04-09T18:35:05.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5ce4ae700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c80d7290 con 0x7fe5c80cffb0 2021-04-09T18:35:05.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5ce4ae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c80d62a0 con 0x7fe5c80d7810 2021-04-09T18:35:05.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5ccacb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5c80d8230 0x7fe5c80d5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5ccacb700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5c80d8230 0x7fe5c80d5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60048/0 (socket says 172.21.15.32:60048) 2021-04-09T18:35:05.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5ccacb700 1 -- 172.21.15.32:0/2523360502 learned_addr learned my addr 172.21.15.32:0/2523360502 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:05.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5bffff700 1 --2- 172.21.15.32:0/2523360502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c80d7810 0x7fe5c80d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.638 7fe5c7fff700 1 --2- 172.21.15.32:0/2523360502 >> v2:172.21.15.86:3300/0 conn(0x7fe5c80cffb0 0x7fe5c80d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5ccacb700 1 -- 172.21.15.32:0/2523360502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c80d7810 msgr2=0x7fe5c80d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:05.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5ccacb700 1 --2- 172.21.15.32:0/2523360502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c80d7810 0x7fe5c80d7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5ccacb700 1 -- 172.21.15.32:0/2523360502 >> v2:172.21.15.86:3300/0 conn(0x7fe5c80cffb0 msgr2=0x7fe5c80d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5ccacb700 1 --2- 172.21.15.32:0/2523360502 >> v2:172.21.15.86:3300/0 conn(0x7fe5c80cffb0 0x7fe5c80d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5ccacb700 1 -- 172.21.15.32:0/2523360502 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c80d64c0 con 0x7fe5c80d8230 2021-04-09T18:35:05.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5c7fff700 1 --2- 172.21.15.32:0/2523360502 >> v2:172.21.15.86:3300/0 conn(0x7fe5c80cffb0 0x7fe5c80d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:05.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5bffff700 1 --2- 172.21.15.32:0/2523360502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c80d7810 0x7fe5c80d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:05.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5ccacb700 1 --2- 172.21.15.32:0/2523360502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c80d8230 0x7fe5c80d5c80 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7fe5c0006f00 tx=0x7fe5c0009f10).ready entity=mon.0 client_cookie=3fd7427d9b40fa9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:05.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5c77fe700 1 -- 172.21.15.32:0/2523360502 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5c000b3b0 con 0x7fe5c80d8230 2021-04-09T18:35:05.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5c77fe700 1 -- 172.21.15.32:0/2523360502 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c000b730 con 0x7fe5c80d8230 2021-04-09T18:35:05.682 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5c77fe700 1 -- 172.21.15.32:0/2523360502 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5c000bc10 con 0x7fe5c80d8230 2021-04-09T18:35:05.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5ce4ae700 1 -- 172.21.15.32:0/2523360502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c80d8230 msgr2=0x7fe5c80d5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.639 7fe5ce4ae700 1 --2- 172.21.15.32:0/2523360502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c80d8230 0x7fe5c80d5c80 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7fe5c0006f00 tx=0x7fe5c0009f10).stop 2021-04-09T18:35:05.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.640 7fe5ce4ae700 1 -- 172.21.15.32:0/2523360502 shutdown_connections 2021-04-09T18:35:05.683 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.640 7fe5ce4ae700 1 --2- 172.21.15.32:0/2523360502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c80d8230 0x7fe5c80d5c80 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.640 7fe5ce4ae700 1 --2- 172.21.15.32:0/2523360502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c80d7810 0x7fe5c80d7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.640 7fe5ce4ae700 1 --2- 172.21.15.32:0/2523360502 >> v2:172.21.15.86:3300/0 conn(0x7fe5c80cffb0 0x7fe5c80d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.640 7fe5ce4ae700 1 -- 172.21.15.32:0/2523360502 >> 172.21.15.32:0/2523360502 conn(0x7fe5c80ce5d0 msgr2=0x7fe5c80cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.640 7fe5ce4ae700 1 -- 172.21.15.32:0/2523360502 shutdown_connections 2021-04-09T18:35:05.684 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.640 7fe5ce4ae700 1 -- 172.21.15.32:0/2523360502 wait complete. 2021-04-09T18:35:05.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.660 7fe5ce4ae700 1 Processor -- start 2021-04-09T18:35:05.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.660 7fe5ce4ae700 1 -- start start 2021-04-09T18:35:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ce4ae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5c8184ec0 0x7fe5c8198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ce4ae700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c81abf00 0x7fe5c81accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ce4ae700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c81ac300 0x7fe5c8184020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ce4ae700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c81ac750 con 0x7fe5c81ac300 2021-04-09T18:35:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ce4ae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c81ad2f0 con 0x7fe5c8184ec0 2021-04-09T18:35:05.689 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ce4ae700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c8188cb0 con 0x7fe5c81abf00 2021-04-09T18:35:05.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ccacb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c81ac300 0x7fe5c8184020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ccacb700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c81ac300 0x7fe5c8184020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60052/0 (socket says 172.21.15.32:60052) 2021-04-09T18:35:05.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ccacb700 1 -- 172.21.15.32:0/1228097723 learned_addr learned my addr 172.21.15.32:0/1228097723 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:05.690 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5bffff700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c81abf00 0x7fe5c81accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5c7fff700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5c8184ec0 0x7fe5c8198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ccacb700 1 -- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c81abf00 msgr2=0x7fe5c81accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ccacb700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c81abf00 0x7fe5c81accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ccacb700 1 -- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5c8184ec0 msgr2=0x7fe5c8198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ccacb700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5c8184ec0 0x7fe5c8198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ccacb700 1 -- 172.21.15.32:0/1228097723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c8184640 con 0x7fe5c81ac300 2021-04-09T18:35:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5c7fff700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5c8184ec0 0x7fe5c8198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:35:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ccacb700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c81ac300 0x7fe5c8184020 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7fe5c000b940 tx=0x7fe5c000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5c57fa700 1 -- 172.21.15.32:0/1228097723 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5c0000f90 con 0x7fe5c81ac300 2021-04-09T18:35:05.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5c57fa700 1 -- 172.21.15.32:0/1228097723 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c0000cc0 con 0x7fe5c81ac300 2021-04-09T18:35:05.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c8184640 con 0x7fe5c81ac300 2021-04-09T18:35:05.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.661 7fe5c57fa700 1 -- 172.21.15.32:0/1228097723 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5c0001460 con 0x7fe5c81ac300 2021-04-09T18:35:05.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.662 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c8184640 con 0x7fe5c81ac300 2021-04-09T18:35:05.693 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.662 7fe5c57fa700 1 -- 172.21.15.32:0/1228097723 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe5c0005820 con 0x7fe5c81ac300 2021-04-09T18:35:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.662 7fe5c57fa700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5ac018930 0x7fe5ac01adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.662 7fe5c57fa700 1 -- 172.21.15.32:0/1228097723 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe5c0032da0 con 0x7fe5c81ac300 2021-04-09T18:35:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.662 7fe5c7fff700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5ac018930 0x7fe5ac01adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:05.694 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.663 7fe5c7fff700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5ac018930 0x7fe5ac01adc0 crc :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:05.699 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.682 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c8182e20 con 0x7fe5c81ac300 2021-04-09T18:35:05.703 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.686 7fe5c57fa700 1 -- 172.21.15.32:0/1228097723 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe5c000be20 con 0x7fe5c81ac300 2021-04-09T18:35:05.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:05.867 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe5c80d7290 con 0x7fe5c81ac300 2021-04-09T18:35:06.589 INFO:tasks.ceph.mon.a.smithi032.stderr:2021-04-09 18:35:06.571 7f5117900700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-04-09T18:35:06.593 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.576 7fe5c57fa700 1 -- 172.21.15.32:0/1228097723 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v436) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe5c0000cc0 con 0x7fe5c81ac300 2021-04-09T18:35:06.600 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-04-09T18:35:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5ac018930 msgr2=0x7fe5ac01adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5ac018930 0x7fe5ac01adc0 crc :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:06.604 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c81ac300 msgr2=0x7fe5c8184020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c81ac300 0x7fe5c8184020 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7fe5c000b940 tx=0x7fe5c000bce0).stop 2021-04-09T18:35:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 shutdown_connections 2021-04-09T18:35:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe5ac018930 0x7fe5ac01adc0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:06.605 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5c81ac300 0x7fe5c8184020 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5c81abf00 0x7fe5c81accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 --2- 172.21.15.32:0/1228097723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5c8184ec0 0x7fe5c8198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:06.606 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 >> 172.21.15.32:0/1228097723 conn(0x7fe5c81838a0 msgr2=0x7fe5c81a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 shutdown_connections 2021-04-09T18:35:06.607 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.586 7fe5ce4ae700 1 -- 172.21.15.32:0/1228097723 wait complete. 2021-04-09T18:35:06.618 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:35:06.660 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:35:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.752 7fd1cee57700 1 Processor -- start 2021-04-09T18:35:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.752 7fd1cee57700 1 -- start start 2021-04-09T18:35:06.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.752 7fd1cee57700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c810d010 0x7fd1c80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.752 7fd1cee57700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c810d480 con 0x7fd1c80d1da0 2021-04-09T18:35:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.752 7fd1cee57700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c810e1f0 con 0x7fd1c80d8490 2021-04-09T18:35:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.752 7fd1cee57700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c810e780 con 0x7fd1c810d010 2021-04-09T18:35:06.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.752 7fd1c7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd1c80d1da0 0x7fd1c80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41428/0 (socket says 172.21.15.32:41428) 2021-04-09T18:35:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.752 7fd1c7fff700 1 -- 172.21.15.32:0/842390989 learned_addr learned my addr 172.21.15.32:0/842390989 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1ccc73700 1 --2- 172.21.15.32:0/842390989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c810d010 0x7fd1c80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:06.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1c77fe700 1 -- 172.21.15.32:0/842390989 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247076621 0 0) 0x7fd1b8000ce0 con 0x7fd1c80d1da0 2021-04-09T18:35:06.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1c77fe700 1 -- 172.21.15.32:0/842390989 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b4002fe0 con 0x7fd1c80d1da0 2021-04-09T18:35:06.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1c77fe700 1 -- 172.21.15.32:0/842390989 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317409972 0 0) 0x7fd1c0001400 con 0x7fd1c80d8490 2021-04-09T18:35:06.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1c77fe700 1 -- 172.21.15.32:0/842390989 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b4003210 con 0x7fd1c80d8490 2021-04-09T18:35:06.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1c77fe700 1 -- 172.21.15.32:0/842390989 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3023627371 0 0) 0x7fd1b80010e0 con 0x7fd1c80d1da0 2021-04-09T18:35:06.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1ccc73700 1 -- 172.21.15.32:0/842390989 >> v1:172.21.15.86:6789/0 conn(0x7fd1c80d8490 legacy=0x7fd1c810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:06.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1ccc73700 1 -- 172.21.15.32:0/842390989 >> v1:172.21.15.32:6789/0 conn(0x7fd1c80d1da0 legacy=0x7fd1c80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:06.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1ccc73700 1 -- 172.21.15.32:0/842390989 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c80d8d30 con 0x7fd1c810d010 2021-04-09T18:35:06.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1ccc73700 1 --2- 172.21.15.32:0/842390989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c810d010 0x7fd1c80d1780 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fd1b0006650 tx=0x7fd1b0009a80).ready entity=mon.2 client_cookie=e5298172fb410171 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:06.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1c77fe700 1 -- 172.21.15.32:0/842390989 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1b00009b0 con 0x7fd1c810d010 2021-04-09T18:35:06.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1c77fe700 1 -- 172.21.15.32:0/842390989 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b000c0c0 con 0x7fd1c810d010 2021-04-09T18:35:06.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1c77fe700 1 -- 172.21.15.32:0/842390989 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1b000c510 con 0x7fd1c810d010 2021-04-09T18:35:06.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1cee57700 1 -- 172.21.15.32:0/842390989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c810d010 msgr2=0x7fd1c80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:06.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.753 7fd1cee57700 1 --2- 172.21.15.32:0/842390989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c810d010 0x7fd1c80d1780 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fd1b0006650 tx=0x7fd1b0009a80).stop 2021-04-09T18:35:06.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.754 7fd1cee57700 1 -- 172.21.15.32:0/842390989 shutdown_connections 2021-04-09T18:35:06.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.754 7fd1cee57700 1 --2- 172.21.15.32:0/842390989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c810d010 0x7fd1c80d1780 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:06.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.754 7fd1cee57700 1 -- 172.21.15.32:0/842390989 >> 172.21.15.32:0/842390989 conn(0x7fd1c80ce5f0 msgr2=0x7fd1c80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:06.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.754 7fd1cee57700 1 -- 172.21.15.32:0/842390989 shutdown_connections 2021-04-09T18:35:06.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.754 7fd1cee57700 1 -- 172.21.15.32:0/842390989 wait complete. 2021-04-09T18:35:06.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.774 7fd1cee57700 1 Processor -- start 2021-04-09T18:35:06.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.774 7fd1cee57700 1 -- start start 2021-04-09T18:35:06.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1cee57700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c80b5360 0x7fd1c81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:06.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1cee57700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c818e140 0x7fd1c81aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:06.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1cee57700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c81ad040 0x7fd1c808fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:06.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1cee57700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c81a5360 con 0x7fd1c80b5360 2021-04-09T18:35:06.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1cee57700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c80902c0 con 0x7fd1c81ad040 2021-04-09T18:35:06.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1cee57700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c80904e0 con 0x7fd1c818e140 2021-04-09T18:35:06.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1ccc73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c80b5360 0x7fd1c81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:06.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1ccc73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c80b5360 0x7fd1c81a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60076/0 (socket says 172.21.15.32:60076) 2021-04-09T18:35:06.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1ccc73700 1 -- 172.21.15.32:0/1936885023 learned_addr learned my addr 172.21.15.32:0/1936885023 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:06.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1cd474700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c81ad040 0x7fd1c808fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:06.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.775 7fd1c7fff700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c818e140 0x7fd1c81aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:06.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1ccc73700 1 -- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c818e140 msgr2=0x7fd1c81aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:06.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1ccc73700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c818e140 0x7fd1c81aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:06.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1ccc73700 1 -- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c81ad040 msgr2=0x7fd1c808fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:06.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1ccc73700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c81ad040 0x7fd1c808fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:06.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1ccc73700 1 -- 172.21.15.32:0/1936885023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c8183c80 con 0x7fd1c80b5360 2021-04-09T18:35:06.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1ccc73700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c80b5360 0x7fd1c81a4d40 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7fd1b0009650 tx=0x7fd1b000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:06.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1c57fa700 1 -- 172.21.15.32:0/1936885023 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1b00093a0 con 0x7fd1c80b5360 2021-04-09T18:35:06.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1c57fa700 1 -- 172.21.15.32:0/1936885023 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b000bd50 con 0x7fd1c80b5360 2021-04-09T18:35:06.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c8183c80 con 0x7fd1c80b5360 2021-04-09T18:35:06.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1c57fa700 1 -- 172.21.15.32:0/1936885023 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1b00052e0 con 0x7fd1c80b5360 2021-04-09T18:35:06.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c8183c80 con 0x7fd1c80b5360 2021-04-09T18:35:06.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.776 7fd1c57fa700 1 -- 172.21.15.32:0/1936885023 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd1b0006030 con 0x7fd1c80b5360 2021-04-09T18:35:06.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.777 7fd1c57fa700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1ac018930 0x7fd1ac01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:06.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.777 7fd1c57fa700 1 -- 172.21.15.32:0/1936885023 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 5663+0+0 (secure 0 0 0) 0x7fd1b0032a30 con 0x7fd1c80b5360 2021-04-09T18:35:06.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.777 7fd1c7fff700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1ac018930 0x7fd1ac01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:06.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.777 7fd1c7fff700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1ac018930 0x7fd1ac01ae10 crc :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:06.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.796 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c8182d70 con 0x7fd1c80b5360 2021-04-09T18:35:06.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.800 7fd1c57fa700 1 -- 172.21.15.32:0/1936885023 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd1b00009c0 con 0x7fd1c80b5360 2021-04-09T18:35:07.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.991 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1c8182d70 con 0x7fd1c80b5360 2021-04-09T18:35:07.009 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.991 7fd1c57fa700 1 -- 172.21.15.32:0/1936885023 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 436 v436) v1 ==== 95+0+2102 (secure 0 0 0) 0x7fd1b000bd50 con 0x7fd1c80b5360 2021-04-09T18:35:07.010 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 436 2021-04-09T18:35:07.011 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:07.011 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":436,"default_fscid":22,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":59948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/4251960201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":4251960201},{"type":"v1","addr":"172.21.15.86:6836","nonce":4251960201}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59951,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1668354365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1668354365},{"type":"v1","addr":"172.21.15.86:6837","nonce":1668354365}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422}],"filesystems":[{"mdsmap":{"epoch":436,"flags":51,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:34:16.431259","modified":"2021-04-09 18:35:06.572088","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[45],"metadata_pool":44,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":22}]} 2021-04-09T18:35:07.013 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1ac018930 msgr2=0x7fd1ac01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:07.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1ac018930 0x7fd1ac01ae10 crc :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c80b5360 msgr2=0x7fd1c81a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:07.014 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c80b5360 0x7fd1c81a4d40 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7fd1b0009650 tx=0x7fd1b000b960).stop 2021-04-09T18:35:07.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 shutdown_connections 2021-04-09T18:35:07.015 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd1ac018930 0x7fd1ac01ae10 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c81ad040 0x7fd1c808fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1c818e140 0x7fd1c81aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.016 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 --2- 172.21.15.32:0/1936885023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1c80b5360 0x7fd1c81a4d40 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 >> 172.21.15.32:0/1936885023 conn(0x7fd1c819b460 msgr2=0x7fd1c819bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:07.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 shutdown_connections 2021-04-09T18:35:07.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:06.994 7fd1cee57700 1 -- 172.21.15.32:0/1936885023 wait complete. 2021-04-09T18:35:07.022 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-04-09T18:35:07.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.196 7f9510bb0700 1 Processor -- start 2021-04-09T18:35:07.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.196 7f9510bb0700 1 -- start start 2021-04-09T18:35:07.236 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f9510bb0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c0cffb0 0x7f950c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:07.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f9510bb0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f950c0d7810 0x7f950c0d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:07.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f9510bb0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c0d7290 con 0x7f950c0d5c80 2021-04-09T18:35:07.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f9510bb0700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950c0d8590 con 0x7f950c0d7810 2021-04-09T18:35:07.237 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f9510bb0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0d91f0 con 0x7f950c0cffb0 2021-04-09T18:35:07.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950a05d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f950c0d7810 0x7f950c0d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:07.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950b05f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f950c0d5c80 0x7f950c0d6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41444/0 (socket says 172.21.15.32:41444) 2021-04-09T18:35:07.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950b05f700 1 -- 172.21.15.32:0/196977076 learned_addr learned my addr 172.21.15.32:0/196977076 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:07.238 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950a05d700 1 --2- 172.21.15.32:0/196977076 >> v2:172.21.15.32:3300/0 conn(0x7f950c0d7810 0x7f950c0d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60088/0 (socket says 172.21.15.32:60088) 2021-04-09T18:35:07.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950a85e700 1 --2- 172.21.15.32:0/196977076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c0cffb0 0x7f950c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:07.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950a05d700 1 -- 172.21.15.32:0/196977076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c0cffb0 msgr2=0x7f950c0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:07.239 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950a05d700 1 --2- 172.21.15.32:0/196977076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c0cffb0 0x7f950c0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950a05d700 1 -- 172.21.15.32:0/196977076 >> v1:172.21.15.32:6789/0 conn(0x7f950c0d5c80 legacy=0x7f950c0d6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:07.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950a05d700 1 -- 172.21.15.32:0/196977076 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c0d8230 con 0x7f950c0d7810 2021-04-09T18:35:07.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950985c700 1 -- 172.21.15.32:0/196977076 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178652217 0 0) 0x7f9504000d30 con 0x7f950c0d5c80 2021-04-09T18:35:07.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950a05d700 1 --2- 172.21.15.32:0/196977076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c0d7810 0x7f950c0d7c10 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 rx=0x7f9500006650 tx=0x7f9500009a00).ready entity=mon.0 client_cookie=2eb91e193bea3173 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:07.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950985c700 1 -- 172.21.15.32:0/196977076 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9500009df0 con 0x7f950c0d7810 2021-04-09T18:35:07.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950985c700 1 -- 172.21.15.32:0/196977076 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95000009b0 con 0x7f950c0d7810 2021-04-09T18:35:07.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f950985c700 1 -- 172.21.15.32:0/196977076 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f950000c2b0 con 0x7f950c0d7810 2021-04-09T18:35:07.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.197 7f9510bb0700 1 -- 172.21.15.32:0/196977076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c0d7810 msgr2=0x7f950c0d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:07.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.198 7f9510bb0700 1 --2- 172.21.15.32:0/196977076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c0d7810 0x7f950c0d7c10 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 rx=0x7f9500006650 tx=0x7f9500009a00).stop 2021-04-09T18:35:07.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.198 7f9510bb0700 1 -- 172.21.15.32:0/196977076 shutdown_connections 2021-04-09T18:35:07.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.198 7f9510bb0700 1 --2- 172.21.15.32:0/196977076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c0d7810 0x7f950c0d7c10 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.242 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.198 7f9510bb0700 1 --2- 172.21.15.32:0/196977076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c0cffb0 0x7f950c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.198 7f9510bb0700 1 -- 172.21.15.32:0/196977076 >> 172.21.15.32:0/196977076 conn(0x7f950c0ce5d0 msgr2=0x7f950c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:07.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.198 7f9510bb0700 1 -- 172.21.15.32:0/196977076 shutdown_connections 2021-04-09T18:35:07.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.198 7f9510bb0700 1 -- 172.21.15.32:0/196977076 wait complete. 2021-04-09T18:35:07.243 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.218 7f9510bb0700 1 Processor -- start 2021-04-09T18:35:07.244 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.218 7f9510bb0700 1 -- start start 2021-04-09T18:35:07.246 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f9510bb0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c184080 0x7f950c1955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f9510bb0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f950c195c00 0x7f950c08ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f9510bb0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c08f550 0x7f950c08f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f9510bb0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c090570 con 0x7f950c08f550 2021-04-09T18:35:07.247 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f9510bb0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c184480 con 0x7f950c195c00 2021-04-09T18:35:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f9510bb0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c196000 con 0x7f950c184080 2021-04-09T18:35:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f950b05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c08f550 0x7f950c08f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f950a85e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c184080 0x7f950c1955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:07.248 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f950a05d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f950c195c00 0x7f950c08ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:07.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f950b05f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c08f550 0x7f950c08f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60094/0 (socket says 172.21.15.32:60094) 2021-04-09T18:35:07.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f950b05f700 1 -- 172.21.15.32:0/3049345878 learned_addr learned my addr 172.21.15.32:0/3049345878 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:07.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.219 7f950a85e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c184080 0x7f950c1955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:39084/0 (socket says 172.21.15.32:39084) 2021-04-09T18:35:07.249 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f950b05f700 1 -- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c184080 msgr2=0x7f950c1955e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:07.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f950b05f700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c184080 0x7f950c1955e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f950b05f700 1 -- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f950c195c00 msgr2=0x7f950c08ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:07.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f950b05f700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f950c195c00 0x7f950c08ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.250 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f950b05f700 1 -- 172.21.15.32:0/3049345878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c1847b0 con 0x7f950c08f550 2021-04-09T18:35:07.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f950b05f700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c08f550 0x7f950c08f950 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f9504007f90 tx=0x7f950400abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:07.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f94fb7fe700 1 -- 172.21.15.32:0/3049345878 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9504000cc0 con 0x7f950c08f550 2021-04-09T18:35:07.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f94fb7fe700 1 -- 172.21.15.32:0/3049345878 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95040070d0 con 0x7f950c08f550 2021-04-09T18:35:07.251 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f94fb7fe700 1 -- 172.21.15.32:0/3049345878 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9504007520 con 0x7f950c08f550 2021-04-09T18:35:07.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c1847b0 con 0x7f950c08f550 2021-04-09T18:35:07.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.220 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c1847b0 con 0x7f950c08f550 2021-04-09T18:35:07.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.221 7f94fb7fe700 1 -- 172.21.15.32:0/3049345878 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f95040072b0 con 0x7f950c08f550 2021-04-09T18:35:07.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.221 7f94fb7fe700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94f0018930 0x7f94f001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:07.252 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.221 7f94fb7fe700 1 -- 172.21.15.32:0/3049345878 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f9504032d80 con 0x7f950c08f550 2021-04-09T18:35:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.221 7f950a85e700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94f0018930 0x7f94f001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:07.253 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.221 7f950a85e700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94f0018930 0x7f94f001ae10 crc :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:07.258 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.241 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c182e80 con 0x7f950c08f550 2021-04-09T18:35:07.262 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.244 7f94fb7fe700 1 -- 172.21.15.32:0/3049345878 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9504007d00 con 0x7f950c08f550 2021-04-09T18:35:07.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.438 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f950c0d7290 con 0x7f950c08f550 2021-04-09T18:35:07.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.872 7f94fb7fe700 1 -- 172.21.15.32:0/3049345878 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v437) v1 ==== 104+0+0 (secure 0 0 0) 0x7f95040070d0 con 0x7f950c08f550 2021-04-09T18:35:07.923 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94f0018930 msgr2=0x7f94f001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:07.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94f0018930 0x7f94f001ae10 crc :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c08f550 msgr2=0x7f950c08f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:07.924 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c08f550 0x7f950c08f950 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f9504007f90 tx=0x7f950400abd0).stop 2021-04-09T18:35:07.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 shutdown_connections 2021-04-09T18:35:07.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f94f0018930 0x7f94f001ae10 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f950c08f550 0x7f950c08f950 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.925 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f950c195c00 0x7f950c08ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 --2- 172.21.15.32:0/3049345878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f950c184080 0x7f950c1955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:07.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.905 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 >> 172.21.15.32:0/3049345878 conn(0x7f950c1a8f50 msgr2=0x7f950c188060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:07.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.906 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 shutdown_connections 2021-04-09T18:35:07.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:07.906 7f9510bb0700 1 -- 172.21.15.32:0/3049345878 wait complete. 2021-04-09T18:35:07.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-04-09T18:35:08.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.078 7f2e0dcc8700 1 Processor -- start 2021-04-09T18:35:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.078 7f2e0dcc8700 1 -- start start 2021-04-09T18:35:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e0dcc8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e0810da80 0x7f2e0810de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:08.119 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e0dcc8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0810d500 con 0x7f2e0810d050 2021-04-09T18:35:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e0dcc8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e080d0330 con 0x7f2e080d7cc0 2021-04-09T18:35:08.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e0dcc8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e080d8d70 con 0x7f2e0810da80 2021-04-09T18:35:08.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e07fff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2e0810da80 0x7f2e0810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:08.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e077fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2e0810d050 0x7f2e080cfb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41458/0 (socket says 172.21.15.32:41458) 2021-04-09T18:35:08.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e077fe700 1 -- 172.21.15.32:0/2198226656 learned_addr learned my addr 172.21.15.32:0/2198226656 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e067fc700 1 -- 172.21.15.32:0/2198226656 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527842720 0 0) 0x7f2df8001060 con 0x7f2e0810d050 2021-04-09T18:35:08.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e067fc700 1 -- 172.21.15.32:0/2198226656 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df0002fe0 con 0x7f2e0810d050 2021-04-09T18:35:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e07fff700 1 -- 172.21.15.32:0/2198226656 >> v1:172.21.15.86:6789/0 conn(0x7f2e080d7cc0 legacy=0x7f2e080d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:08.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e07fff700 1 -- 172.21.15.32:0/2198226656 >> v1:172.21.15.32:6789/0 conn(0x7f2e0810d050 legacy=0x7f2e080cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.079 7f2e07fff700 1 -- 172.21.15.32:0/2198226656 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e080d9260 con 0x7f2e0810da80 2021-04-09T18:35:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.080 7f2e07fff700 1 --2- 172.21.15.32:0/2198226656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0810da80 0x7f2e0810de80 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7f2e00006ab0 tx=0x7f2e00009b40).ready entity=mon.0 client_cookie=db7817cf64e98217 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:08.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.080 7f2e067fc700 1 -- 172.21.15.32:0/2198226656 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e000022e0 con 0x7f2e0810da80 2021-04-09T18:35:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.080 7f2e067fc700 1 -- 172.21.15.32:0/2198226656 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e00002660 con 0x7f2e0810da80 2021-04-09T18:35:08.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.080 7f2e067fc700 1 -- 172.21.15.32:0/2198226656 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e00002b40 con 0x7f2e0810da80 2021-04-09T18:35:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.080 7f2e0dcc8700 1 -- 172.21.15.32:0/2198226656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0810da80 msgr2=0x7f2e0810de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.080 7f2e0dcc8700 1 --2- 172.21.15.32:0/2198226656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0810da80 0x7f2e0810de80 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7f2e00006ab0 tx=0x7f2e00009b40).stop 2021-04-09T18:35:08.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.080 7f2e0dcc8700 1 -- 172.21.15.32:0/2198226656 shutdown_connections 2021-04-09T18:35:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.080 7f2e0dcc8700 1 --2- 172.21.15.32:0/2198226656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e0810da80 0x7f2e0810de80 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:08.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.080 7f2e0dcc8700 1 -- 172.21.15.32:0/2198226656 >> 172.21.15.32:0/2198226656 conn(0x7f2e080ce620 msgr2=0x7f2e080d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:08.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.081 7f2e0dcc8700 1 -- 172.21.15.32:0/2198226656 shutdown_connections 2021-04-09T18:35:08.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.081 7f2e0dcc8700 1 -- 172.21.15.32:0/2198226656 wait complete. 2021-04-09T18:35:08.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.101 7f2e0dcc8700 1 Processor -- start 2021-04-09T18:35:08.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.101 7f2e0dcc8700 1 -- start start 2021-04-09T18:35:08.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.101 7f2e0dcc8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080a2220 0x7f2e080a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.101 7f2e0dcc8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e080c0d20 0x7f2e080c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:08.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.101 7f2e0dcc8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0819ce90 0x7f2e0819d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:08.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.101 7f2e0dcc8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080bca10 con 0x7f2e080a2220 2021-04-09T18:35:08.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.101 7f2e0dcc8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080ae5e0 con 0x7f2e0819ce90 2021-04-09T18:35:08.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.101 7f2e0dcc8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0819d8b0 con 0x7f2e080c0d20 2021-04-09T18:35:08.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080a2220 0x7f2e080a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:08.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e077fe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080a2220 0x7f2e080a2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60106/0 (socket says 172.21.15.32:60106) 2021-04-09T18:35:08.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e077fe700 1 -- 172.21.15.32:0/329380682 learned_addr learned my addr 172.21.15.32:0/329380682 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:08.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e07fff700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0819ce90 0x7f2e0819d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:08.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e06ffd700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e080c0d20 0x7f2e080c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e077fe700 1 -- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e080c0d20 msgr2=0x7f2e080c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e077fe700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e080c0d20 0x7f2e080c1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:08.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e077fe700 1 -- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0819ce90 msgr2=0x7f2e0819d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:08.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e077fe700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0819ce90 0x7f2e0819d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:08.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e077fe700 1 -- 172.21.15.32:0/329380682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0819dad0 con 0x7f2e080a2220 2021-04-09T18:35:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e077fe700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080a2220 0x7f2e080a2770 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7f2df8009980 tx=0x7f2df800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2deffff700 1 -- 172.21.15.32:0/329380682 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df8006080 con 0x7f2e080a2220 2021-04-09T18:35:08.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2deffff700 1 -- 172.21.15.32:0/329380682 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df8006380 con 0x7f2e080a2220 2021-04-09T18:35:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2deffff700 1 -- 172.21.15.32:0/329380682 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df80049a0 con 0x7f2e080a2220 2021-04-09T18:35:08.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0819dad0 con 0x7f2e080a2220 2021-04-09T18:35:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.102 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0819dad0 con 0x7f2e080a2220 2021-04-09T18:35:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.103 7f2deffff700 1 -- 172.21.15.32:0/329380682 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f2df8006560 con 0x7f2e080a2220 2021-04-09T18:35:08.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.103 7f2deffff700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de80188d0 0x7f2de801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.103 7f2deffff700 1 -- 172.21.15.32:0/329380682 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f2df8032dd0 con 0x7f2e080a2220 2021-04-09T18:35:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.103 7f2e06ffd700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de80188d0 0x7f2de801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:08.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.103 7f2e06ffd700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de80188d0 0x7f2de801add0 crc :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:08.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.123 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e080b2d30 con 0x7f2e080a2220 2021-04-09T18:35:08.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.126 7f2deffff700 1 -- 172.21.15.32:0/329380682 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f2df8000d30 con 0x7f2e080a2220 2021-04-09T18:35:08.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.308 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2e0810d500 con 0x7f2e080a2220 2021-04-09T18:35:08.893 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.876 7f2deffff700 1 -- 172.21.15.32:0/329380682 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v268) v1 ==== 185+0+0 (secure 0 0 0) 0x7f2df8006260 con 0x7f2e080a2220 2021-04-09T18:35:08.947 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2021-04-09T18:35:08.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.925 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de80188d0 msgr2=0x7f2de801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:08.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.925 7f2e0dcc8700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de80188d0 0x7f2de801add0 crc :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:08.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.925 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080a2220 msgr2=0x7f2e080a2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:08.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.925 7f2e0dcc8700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080a2220 0x7f2e080a2770 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7f2df8009980 tx=0x7f2df800bcd0).stop 2021-04-09T18:35:08.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.925 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 shutdown_connections 2021-04-09T18:35:08.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.925 7f2e0dcc8700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f2de80188d0 0x7f2de801add0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:08.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.926 7f2e0dcc8700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0819ce90 0x7f2e0819d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:08.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.926 7f2e0dcc8700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e080c0d20 0x7f2e080c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:08.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.926 7f2e0dcc8700 1 --2- 172.21.15.32:0/329380682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e080a2220 0x7f2e080a2770 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:08.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.926 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 >> 172.21.15.32:0/329380682 conn(0x7f2e080920d0 msgr2=0x7f2e080a56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:08.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.926 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 shutdown_connections 2021-04-09T18:35:08.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:08.926 7f2e0dcc8700 1 -- 172.21.15.32:0/329380682 wait complete. 2021-04-09T18:35:08.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-04-09T18:35:09.024 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:35:09.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:35:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.102 7f1208537700 1 Processor -- start 2021-04-09T18:35:09.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.102 7f1208537700 1 -- start start 2021-04-09T18:35:09.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.103 7f1208537700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f120010da80 0x7f120010de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:09.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.103 7f1208537700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120010d500 con 0x7f12000d7cc0 2021-04-09T18:35:09.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.103 7f1208537700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12000d0330 con 0x7f120010d050 2021-04-09T18:35:09.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.103 7f1208537700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12000d8d70 con 0x7f120010da80 2021-04-09T18:35:09.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.103 7f1205b52700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f12000d7cc0 0x7f12000d80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41474/0 (socket says 172.21.15.32:41474) 2021-04-09T18:35:09.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.103 7f1205b52700 1 -- 172.21.15.32:0/348753757 learned_addr learned my addr 172.21.15.32:0/348753757 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:09.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.103 7f1205351700 1 -- 172.21.15.32:0/348753757 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072161627 0 0) 0x7f11f0000ce0 con 0x7f12000d7cc0 2021-04-09T18:35:09.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.103 7f1206b54700 1 --2- 172.21.15.32:0/348753757 >> v2:172.21.15.86:3300/0 conn(0x7f120010da80 0x7f120010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:09.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.103 7f1205351700 1 -- 172.21.15.32:0/348753757 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11ec002fe0 con 0x7f12000d7cc0 2021-04-09T18:35:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639724666 0 0) 0x7f11fc000c60 con 0x7f120010d050 2021-04-09T18:35:09.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11ec003210 con 0x7f120010d050 2021-04-09T18:35:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 134854804 0 0) 0x7f11f0000ed0 con 0x7f12000d7cc0 2021-04-09T18:35:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 >> v2:172.21.15.86:3300/0 conn(0x7f120010da80 msgr2=0x7f120010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:09.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 --2- 172.21.15.32:0/348753757 >> v2:172.21.15.86:3300/0 conn(0x7f120010da80 0x7f120010de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 >> v1:172.21.15.86:6789/0 conn(0x7f120010d050 legacy=0x7f12000cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:09.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12000d9260 con 0x7f12000d7cc0 2021-04-09T18:35:09.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f11f0001600 con 0x7f12000d7cc0 2021-04-09T18:35:09.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 >> v1:172.21.15.32:6789/0 conn(0x7f12000d7cc0 legacy=0x7f12000d80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:09.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ec003ba0 0x7f11ec005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:09.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11ec0067b0 0x7f11ec008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:09.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11ec0095b0 0x7f11ec00bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:09.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec050330 con 0x7f11ec003ba0 2021-04-09T18:35:09.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec0504e0 con 0x7f11ec0095b0 2021-04-09T18:35:09.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec050700 con 0x7f11ec0067b0 2021-04-09T18:35:09.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1206353700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ec003ba0 0x7f11ec005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:09.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1206b54700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11ec0095b0 0x7f11ec00bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:09.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205b52700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11ec0067b0 0x7f11ec008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:09.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1206353700 1 -- 172.21.15.32:0/348753757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11ec0067b0 msgr2=0x7f11ec008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:09.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1206353700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11ec0067b0 0x7f11ec008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1206353700 1 -- 172.21.15.32:0/348753757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11ec0095b0 msgr2=0x7f11ec00bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1206353700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11ec0095b0 0x7f11ec00bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1206b54700 1 -- 172.21.15.32:0/348753757 reap_dead start 2021-04-09T18:35:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1206353700 1 -- 172.21.15.32:0/348753757 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec0509a0 con 0x7f11ec003ba0 2021-04-09T18:35:09.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205b52700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11ec0067b0 0x7f11ec008cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:09.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1206353700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ec003ba0 0x7f11ec005fb0 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7f11fc001040 tx=0x7f11fc009a70).ready entity=mon.0 client_cookie=66083f2a36b6710b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:09.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.104 7f1205351700 1 -- 172.21.15.32:0/348753757 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11fc0066e0 con 0x7f11ec003ba0 2021-04-09T18:35:09.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.105 7f1205351700 1 -- 172.21.15.32:0/348753757 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc00c0c0 con 0x7f11ec003ba0 2021-04-09T18:35:09.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.105 7f1205351700 1 -- 172.21.15.32:0/348753757 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11fc00c5e0 con 0x7f11ec003ba0 2021-04-09T18:35:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.105 7f1208537700 1 -- 172.21.15.32:0/348753757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ec003ba0 msgr2=0x7f11ec005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.105 7f1208537700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ec003ba0 0x7f11ec005fb0 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7f11fc001040 tx=0x7f11fc009a70).stop 2021-04-09T18:35:09.159 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.105 7f1208537700 1 -- 172.21.15.32:0/348753757 shutdown_connections 2021-04-09T18:35:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.105 7f1208537700 1 --2- 172.21.15.32:0/348753757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11ec003ba0 0x7f11ec005fb0 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.105 7f1208537700 1 -- 172.21.15.32:0/348753757 >> 172.21.15.32:0/348753757 conn(0x7f12000ce620 msgr2=0x7f12000d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:09.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.105 7f1208537700 1 -- 172.21.15.32:0/348753757 shutdown_connections 2021-04-09T18:35:09.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.105 7f1208537700 1 -- 172.21.15.32:0/348753757 wait complete. 2021-04-09T18:35:09.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1208537700 1 Processor -- start 2021-04-09T18:35:09.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1208537700 1 -- start start 2021-04-09T18:35:09.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1208537700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12000abf00 0x7f12000a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:09.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1208537700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000a7360 0x7f12001a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:09.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1208537700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000aa6c0 0x7f12001a35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:09.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1208537700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12000c3fe0 con 0x7f12000a7360 2021-04-09T18:35:09.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1208537700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12000c4560 con 0x7f12000aa6c0 2021-04-09T18:35:09.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1208537700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12000c3cd0 con 0x7f12000abf00 2021-04-09T18:35:09.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1205b52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000a7360 0x7f12001a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:09.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1205b52700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000a7360 0x7f12001a2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60132/0 (socket says 172.21.15.32:60132) 2021-04-09T18:35:09.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1205b52700 1 -- 172.21.15.32:0/3464008778 learned_addr learned my addr 172.21.15.32:0/3464008778 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:09.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1206353700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12000abf00 0x7f12000a6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:09.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.126 7f1206b54700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000aa6c0 0x7f12001a35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:09.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f1205b52700 1 -- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12000abf00 msgr2=0x7f12000a6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:09.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f1205b52700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12000abf00 0x7f12000a6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f1205b52700 1 -- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000aa6c0 msgr2=0x7f12001a35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:09.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f1205b52700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000aa6c0 0x7f12001a35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f1205b52700 1 -- 172.21.15.32:0/3464008778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12001a3bd0 con 0x7f12000a7360 2021-04-09T18:35:09.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f1206b54700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000aa6c0 0x7f12001a35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:35:09.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f1205b52700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000a7360 0x7f12001a2f90 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 rx=0x7f11f0007680 tx=0x7f11f000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f11f6ffd700 1 -- 172.21.15.32:0/3464008778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f0001930 con 0x7f12000a7360 2021-04-09T18:35:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f11f6ffd700 1 -- 172.21.15.32:0/3464008778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f000ace0 con 0x7f12000a7360 2021-04-09T18:35:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f11f6ffd700 1 -- 172.21.15.32:0/3464008778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f00043a0 con 0x7f12000a7360 2021-04-09T18:35:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f1208537700 1 -- 172.21.15.32:0/3464008778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12001a3bd0 con 0x7f12000a7360 2021-04-09T18:35:09.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.127 7f1208537700 1 -- 172.21.15.32:0/3464008778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12001a3bd0 con 0x7f12000a7360 2021-04-09T18:35:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.128 7f11f6ffd700 1 -- 172.21.15.32:0/3464008778 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f11f00043a0 con 0x7f12000a7360 2021-04-09T18:35:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.128 7f11f6ffd700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e8018900 0x7f11e801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.128 7f11f6ffd700 1 -- 172.21.15.32:0/3464008778 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 5225+0+0 (secure 0 0 0) 0x7f11f0032860 con 0x7f12000a7360 2021-04-09T18:35:09.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.128 7f1206353700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e8018900 0x7f11e801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:09.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.128 7f1206353700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e8018900 0x7f11e801adc0 crc :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:09.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.147 7f1208537700 1 -- 172.21.15.32:0/3464008778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12000b9040 con 0x7f12000a7360 2021-04-09T18:35:09.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.150 7f11f6ffd700 1 -- 172.21.15.32:0/3464008778 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f11f0001cc0 con 0x7f12000a7360 2021-04-09T18:35:09.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.333 7f1208537700 1 -- 172.21.15.32:0/3464008778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f120010d500 con 0x7f12000a7360 2021-04-09T18:35:09.900 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.883 7f11f6ffd700 1 -- 172.21.15.32:0/3464008778 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v269) v1 ==== 173+0+0 (secure 0 0 0) 0x7f11f00043a0 con 0x7f12000a7360 2021-04-09T18:35:09.915 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2021-04-09T18:35:09.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 -- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e8018900 msgr2=0x7f11e801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:09.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e8018900 0x7f11e801adc0 crc :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 -- 172.21.15.32:0/3464008778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000a7360 msgr2=0x7f12001a2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:09.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000a7360 0x7f12001a2f90 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 rx=0x7f11f0007680 tx=0x7f11f000a8f0).stop 2021-04-09T18:35:09.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 -- 172.21.15.32:0/3464008778 shutdown_connections 2021-04-09T18:35:09.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f11e8018900 0x7f11e801adc0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12000a7360 0x7f12001a2f90 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12000aa6c0 0x7f12001a35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 --2- 172.21.15.32:0/3464008778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f12000abf00 0x7f12000a6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:09.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 -- 172.21.15.32:0/3464008778 >> 172.21.15.32:0/3464008778 conn(0x7f120017f870 msgr2=0x7f12000aea90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:09.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 -- 172.21.15.32:0/3464008778 shutdown_connections 2021-04-09T18:35:09.921 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:09.900 7f1208537700 1 -- 172.21.15.32:0/3464008778 wait complete. 2021-04-09T18:35:09.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-04-09T18:35:10.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.066 7fd183bd6700 1 Processor -- start 2021-04-09T18:35:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.066 7fd183bd6700 1 -- start start 2021-04-09T18:35:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd183bd6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd17c0cffb0 0x7fd17c0d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd183bd6700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd17c0d7810 0x7fd17c0d7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd183bd6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c0d7290 con 0x7fd17c0d8230 2021-04-09T18:35:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd183bd6700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c0d91f0 con 0x7fd17c0d7810 2021-04-09T18:35:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd183bd6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c10dff0 con 0x7fd17c0cffb0 2021-04-09T18:35:10.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1811f1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd17c0d7810 0x7fd17c0d7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1811f1700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fd17c0d7810 0x7fd17c0d7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60140/0 (socket says 172.21.15.32:60140) 2021-04-09T18:35:10.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1811f1700 1 -- 172.21.15.32:0/3214099906 learned_addr learned my addr 172.21.15.32:0/3214099906 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:10.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1819f2700 1 --2- 172.21.15.32:0/3214099906 >> v2:172.21.15.86:3300/0 conn(0x7fd17c0cffb0 0x7fd17c0d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1811f1700 1 -- 172.21.15.32:0/3214099906 >> v2:172.21.15.86:3300/0 conn(0x7fd17c0cffb0 msgr2=0x7fd17c0d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:10.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1811f1700 1 --2- 172.21.15.32:0/3214099906 >> v2:172.21.15.86:3300/0 conn(0x7fd17c0cffb0 0x7fd17c0d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1811f1700 1 -- 172.21.15.32:0/3214099906 >> v1:172.21.15.86:6789/0 conn(0x7fd17c0d8230 legacy=0x7fd17c0d5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1811f1700 1 -- 172.21.15.32:0/3214099906 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c10e210 con 0x7fd17c0d7810 2021-04-09T18:35:10.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1809f0700 1 -- 172.21.15.32:0/3214099906 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173285400 0 0) 0x7fd174000d30 con 0x7fd17c0d8230 2021-04-09T18:35:10.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.067 7fd1811f1700 1 --2- 172.21.15.32:0/3214099906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c0d7810 0x7fd17c0d7c10 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7fd16c006700 tx=0x7fd16c009b80).ready entity=mon.0 client_cookie=e0fa6af4ad21c115 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:10.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.068 7fd1809f0700 1 -- 172.21.15.32:0/3214099906 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd16c00c0c0 con 0x7fd17c0d7810 2021-04-09T18:35:10.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.068 7fd1809f0700 1 -- 172.21.15.32:0/3214099906 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c00c440 con 0x7fd17c0d7810 2021-04-09T18:35:10.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.068 7fd1809f0700 1 -- 172.21.15.32:0/3214099906 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd16c00c0c0 con 0x7fd17c0d7810 2021-04-09T18:35:10.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.068 7fd183bd6700 1 -- 172.21.15.32:0/3214099906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c0d7810 msgr2=0x7fd17c0d7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.068 7fd183bd6700 1 --2- 172.21.15.32:0/3214099906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c0d7810 0x7fd17c0d7c10 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7fd16c006700 tx=0x7fd16c009b80).stop 2021-04-09T18:35:10.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.068 7fd183bd6700 1 -- 172.21.15.32:0/3214099906 shutdown_connections 2021-04-09T18:35:10.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.068 7fd183bd6700 1 --2- 172.21.15.32:0/3214099906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c0d7810 0x7fd17c0d7c10 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.068 7fd183bd6700 1 --2- 172.21.15.32:0/3214099906 >> v2:172.21.15.86:3300/0 conn(0x7fd17c0cffb0 0x7fd17c0d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.068 7fd183bd6700 1 -- 172.21.15.32:0/3214099906 >> 172.21.15.32:0/3214099906 conn(0x7fd17c0ce5d0 msgr2=0x7fd17c0cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:10.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.069 7fd183bd6700 1 -- 172.21.15.32:0/3214099906 shutdown_connections 2021-04-09T18:35:10.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.069 7fd183bd6700 1 -- 172.21.15.32:0/3214099906 wait complete. 2021-04-09T18:35:10.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd183bd6700 1 Processor -- start 2021-04-09T18:35:10.120 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd183bd6700 1 -- start start 2021-04-09T18:35:10.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd183bd6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c08a620 0x7fd17c08b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd183bd6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd17c08b6d0 0x7fd17c180630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd183bd6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd17c180c50 0x7fd17c181050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.122 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd183bd6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c08e660 con 0x7fd17c08a620 2021-04-09T18:35:10.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd183bd6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c08ab30 con 0x7fd17c180c50 2021-04-09T18:35:10.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd183bd6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c08bad0 con 0x7fd17c08b6d0 2021-04-09T18:35:10.123 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd1819f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c08a620 0x7fd17c08b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd1819f2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c08a620 0x7fd17c08b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60142/0 (socket says 172.21.15.32:60142) 2021-04-09T18:35:10.124 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd1819f2700 1 -- 172.21.15.32:0/1039602833 learned_addr learned my addr 172.21.15.32:0/1039602833 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:10.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd1821f3700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd17c180c50 0x7fd17c181050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.089 7fd1811f1700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd17c08b6d0 0x7fd17c180630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1819f2700 1 -- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd17c08b6d0 msgr2=0x7fd17c180630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1819f2700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd17c08b6d0 0x7fd17c180630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.126 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1819f2700 1 -- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd17c180c50 msgr2=0x7fd17c181050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.127 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1819f2700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd17c180c50 0x7fd17c181050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1819f2700 1 -- 172.21.15.32:0/1039602833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c08e1f0 con 0x7fd17c08a620 2021-04-09T18:35:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1811f1700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd17c08b6d0 0x7fd17c180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:10.128 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1819f2700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c08a620 0x7fd17c08b0b0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7fd178006f60 tx=0x7fd17800dba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1727fc700 1 -- 172.21.15.32:0/1039602833 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1780009b0 con 0x7fd17c08a620 2021-04-09T18:35:10.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1727fc700 1 -- 172.21.15.32:0/1039602833 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17800b030 con 0x7fd17c08a620 2021-04-09T18:35:10.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c08e1f0 con 0x7fd17c08a620 2021-04-09T18:35:10.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1727fc700 1 -- 172.21.15.32:0/1039602833 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd178002020 con 0x7fd17c08a620 2021-04-09T18:35:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c08e1f0 con 0x7fd17c08a620 2021-04-09T18:35:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.090 7fd1727fc700 1 -- 172.21.15.32:0/1039602833 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd178009250 con 0x7fd17c08a620 2021-04-09T18:35:10.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.091 7fd1727fc700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd164018930 0x7fd16401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.091 7fd1727fc700 1 -- 172.21.15.32:0/1039602833 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fd178034960 con 0x7fd17c08a620 2021-04-09T18:35:10.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.091 7fd1811f1700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd164018930 0x7fd16401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.091 7fd1811f1700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd164018930 0x7fd16401ae10 crc :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:10.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.111 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c1b7180 con 0x7fd17c08a620 2021-04-09T18:35:10.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.114 7fd1727fc700 1 -- 172.21.15.32:0/1039602833 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd178006b50 con 0x7fd17c08a620 2021-04-09T18:35:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.324 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd17c0d7290 con 0x7fd17c08a620 2021-04-09T18:35:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.328 7fd1727fc700 1 -- 172.21.15.32:0/1039602833 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd178009250 con 0x7fd17c08a620 2021-04-09T18:35:10.352 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.333 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd164018930 msgr2=0x7fd16401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.333 7fd183bd6700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd164018930 0x7fd16401ae10 crc :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.353 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.333 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c08a620 msgr2=0x7fd17c08b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.333 7fd183bd6700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c08a620 0x7fd17c08b0b0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7fd178006f60 tx=0x7fd17800dba0).stop 2021-04-09T18:35:10.354 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.334 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 shutdown_connections 2021-04-09T18:35:10.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.334 7fd183bd6700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd164018930 0x7fd16401ae10 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.355 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.334 7fd183bd6700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd17c180c50 0x7fd17c181050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.334 7fd183bd6700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd17c08b6d0 0x7fd17c180630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.334 7fd183bd6700 1 --2- 172.21.15.32:0/1039602833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd17c08a620 0x7fd17c08b0b0 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.356 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.334 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 >> 172.21.15.32:0/1039602833 conn(0x7fd17c189800 msgr2=0x7fd17c1b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.334 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 shutdown_connections 2021-04-09T18:35:10.357 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.334 7fd183bd6700 1 -- 172.21.15.32:0/1039602833 wait complete. 2021-04-09T18:35:10.362 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestStandbyReplay.test_standby_replay_failure' 2021-04-09T18:35:10.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.498 7fe51281e700 1 Processor -- start 2021-04-09T18:35:10.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.498 7fe51281e700 1 -- start start 2021-04-09T18:35:10.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe51281e700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe50c0d7ca0 0x7fe50c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe51281e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe50c10d480 con 0x7fe50c10d010 2021-04-09T18:35:10.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe51281e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe50c0d86c0 con 0x7fe50c10da00 2021-04-09T18:35:10.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe51281e700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe50c0d9220 con 0x7fe50c0d7ca0 2021-04-09T18:35:10.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe50c10d010 0x7fe50c0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41508/0 (socket says 172.21.15.32:41508) 2021-04-09T18:35:10.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50bfff700 1 -- 172.21.15.32:0/3969680029 learned_addr learned my addr 172.21.15.32:0/3969680029 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:10.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe503fff700 1 --2- 172.21.15.32:0/3969680029 >> v2:172.21.15.86:3300/0 conn(0x7fe50c0d7ca0 0x7fe50c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349997459 0 0) 0x7fe4fc000ce0 con 0x7fe50c10d010 2021-04-09T18:35:10.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f8002fe0 con 0x7fe50c10d010 2021-04-09T18:35:10.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 743075754 0 0) 0x7fe4fc000ed0 con 0x7fe50c10d010 2021-04-09T18:35:10.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 >> v2:172.21.15.86:3300/0 conn(0x7fe50c0d7ca0 msgr2=0x7fe50c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:10.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 --2- 172.21.15.32:0/3969680029 >> v2:172.21.15.86:3300/0 conn(0x7fe50c0d7ca0 0x7fe50c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 >> v1:172.21.15.86:6789/0 conn(0x7fe50c10da00 legacy=0x7fe50c10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c0d1f70 con 0x7fe50c10d010 2021-04-09T18:35:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fe4fc001600 con 0x7fe50c10d010 2021-04-09T18:35:10.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 >> v1:172.21.15.32:6789/0 conn(0x7fe50c10d010 legacy=0x7fe50c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.499 7fe50b7fe700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4f8003af0 0x7fe4f8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe50b7fe700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4f8006840 0x7fe4f8008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe50b7fe700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4f8009640 0x7fe4f800bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f800c3b0 con 0x7fe4f8009640 2021-04-09T18:35:10.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f800c5d0 con 0x7fe4f8006840 2021-04-09T18:35:10.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f8050800 con 0x7fe4f8003af0 2021-04-09T18:35:10.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe510e3b700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4f8009640 0x7fe4f800bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe503fff700 1 --2- 172.21.15.32:0/3969680029 >> v2:172.21.15.86:3300/0 conn(0x7fe50c0d7ca0 0x7fe50c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:10.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe50bfff700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4f8003af0 0x7fe4f8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe503fff700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4f8006840 0x7fe4f8008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe510e3b700 1 -- 172.21.15.32:0/3969680029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4f8003af0 msgr2=0x7fe4f8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:10.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe510e3b700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4f8003af0 0x7fe4f8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe510e3b700 1 -- 172.21.15.32:0/3969680029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4f8006840 msgr2=0x7fe4f8008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe510e3b700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4f8006840 0x7fe4f8008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe510e3b700 1 -- 172.21.15.32:0/3969680029 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f8050a60 con 0x7fe4f8009640 2021-04-09T18:35:10.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe503fff700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4f8006840 0x7fe4f8008d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:35:10.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe510e3b700 1 -- 172.21.15.32:0/3969680029 reap_dead start 2021-04-09T18:35:10.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe510e3b700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4f8009640 0x7fe4f800bb60 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7fe504000f50 tx=0x7fe504009e20).ready entity=mon.0 client_cookie=72f0bdf76ace4e30 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:10.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe50400b510 con 0x7fe4f8009640 2021-04-09T18:35:10.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe504003210 con 0x7fe4f8009640 2021-04-09T18:35:10.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe50b7fe700 1 -- 172.21.15.32:0/3969680029 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe50400bb10 con 0x7fe4f8009640 2021-04-09T18:35:10.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe51281e700 1 -- 172.21.15.32:0/3969680029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4f8009640 msgr2=0x7fe4f800bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.500 7fe51281e700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4f8009640 0x7fe4f800bb60 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7fe504000f50 tx=0x7fe504009e20).stop 2021-04-09T18:35:10.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.501 7fe51281e700 1 -- 172.21.15.32:0/3969680029 shutdown_connections 2021-04-09T18:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.501 7fe51281e700 1 --2- 172.21.15.32:0/3969680029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4f8009640 0x7fe4f800bb60 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.501 7fe51281e700 1 -- 172.21.15.32:0/3969680029 >> 172.21.15.32:0/3969680029 conn(0x7fe50c0ce5f0 msgr2=0x7fe50c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.501 7fe51281e700 1 -- 172.21.15.32:0/3969680029 shutdown_connections 2021-04-09T18:35:10.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.501 7fe51281e700 1 -- 172.21.15.32:0/3969680029 wait complete. 2021-04-09T18:35:10.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe51281e700 1 Processor -- start 2021-04-09T18:35:10.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe51281e700 1 -- start start 2021-04-09T18:35:10.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe51281e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c09a5e0 0x7fe50c1884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe51281e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c0a3350 0x7fe50c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe51281e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe50c0a15b0 0x7fe50c0a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe51281e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c0a40d0 con 0x7fe50c0a3350 2021-04-09T18:35:10.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe51281e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c0a2dd0 con 0x7fe50c09a5e0 2021-04-09T18:35:10.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe51281e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c188ad0 con 0x7fe50c0a15b0 2021-04-09T18:35:10.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe503fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c0a3350 0x7fe50c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe503fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c0a3350 0x7fe50c0a3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60164/0 (socket says 172.21.15.32:60164) 2021-04-09T18:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe503fff700 1 -- 172.21.15.32:0/2794547437 learned_addr learned my addr 172.21.15.32:0/2794547437 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe50bfff700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c09a5e0 0x7fe50c1884b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.522 7fe510e3b700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe50c0a15b0 0x7fe50c0a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe503fff700 1 -- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe50c0a15b0 msgr2=0x7fe50c0a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe503fff700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe50c0a15b0 0x7fe50c0a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe503fff700 1 -- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c09a5e0 msgr2=0x7fe50c1884b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe503fff700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c09a5e0 0x7fe50c1884b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe503fff700 1 -- 172.21.15.32:0/2794547437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c0a3d70 con 0x7fe50c0a3350 2021-04-09T18:35:10.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe503fff700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c0a3350 0x7fe50c0a3750 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7fe4f400b5e0 tx=0x7fe4f400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe5097fa700 1 -- 172.21.15.32:0/2794547437 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4f4006c20 con 0x7fe50c0a3350 2021-04-09T18:35:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe5097fa700 1 -- 172.21.15.32:0/2794547437 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f40052e0 con 0x7fe50c0a3350 2021-04-09T18:35:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe5097fa700 1 -- 172.21.15.32:0/2794547437 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4f4005800 con 0x7fe50c0a3350 2021-04-09T18:35:10.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe51281e700 1 -- 172.21.15.32:0/2794547437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50c0a3d70 con 0x7fe50c0a3350 2021-04-09T18:35:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.523 7fe51281e700 1 -- 172.21.15.32:0/2794547437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50c0a3d70 con 0x7fe50c0a3350 2021-04-09T18:35:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.524 7fe5097fa700 1 -- 172.21.15.32:0/2794547437 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe4f4006030 con 0x7fe50c0a3350 2021-04-09T18:35:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.524 7fe5097fa700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4f0018930 0x7fe4f001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.524 7fe5097fa700 1 -- 172.21.15.32:0/2794547437 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fe4f4032770 con 0x7fe50c0a3350 2021-04-09T18:35:10.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.524 7fe50bfff700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4f0018930 0x7fe4f001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:10.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.524 7fe50bfff700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4f0018930 0x7fe4f001ae10 crc :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:10.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.543 7fe51281e700 1 -- 172.21.15.32:0/2794547437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe50c10d480 con 0x7fe50c0a3350 2021-04-09T18:35:10.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.546 7fe5097fa700 1 -- 172.21.15.32:0/2794547437 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe4f401f790 con 0x7fe50c0a3350 2021-04-09T18:35:10.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.733 7fe51281e700 1 -- 172.21.15.32:0/2794547437 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestStandbyReplay.test_standby_replay_failure"]} v 0) v1 -- 0x7fe50c10d480 con 0x7fe50c0a3350 2021-04-09T18:35:10.901 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.884 7fe5097fa700 1 -- 172.21.15.32:0/2794547437 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestStandbyReplay.test_standby_replay_failure"]}]=0 v1001) v1 ==== 151+0+0 (secure 0 0 0) 0x7fe4f4006030 con 0x7fe50c0a3350 2021-04-09T18:35:10.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 -- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4f0018930 msgr2=0x7fe4f001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4f0018930 0x7fe4f001ae10 crc :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 -- 172.21.15.32:0/2794547437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c0a3350 msgr2=0x7fe50c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:10.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c0a3350 0x7fe50c0a3750 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7fe4f400b5e0 tx=0x7fe4f400b980).stop 2021-04-09T18:35:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 -- 172.21.15.32:0/2794547437 shutdown_connections 2021-04-09T18:35:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe4f0018930 0x7fe4f001ae10 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe50c0a3350 0x7fe50c0a3750 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.918 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe50c0a15b0 0x7fe50c0a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 --2- 172.21.15.32:0/2794547437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c09a5e0 0x7fe50c1884b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:10.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 -- 172.21.15.32:0/2794547437 >> 172.21.15.32:0/2794547437 conn(0x7fe50c1a0db0 msgr2=0x7fe50c1a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:10.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 -- 172.21.15.32:0/2794547437 shutdown_connections 2021-04-09T18:35:10.919 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:10.899 7fe51281e700 1 -- 172.21.15.32:0/2794547437 wait complete. 2021-04-09T18:35:10.927 INFO:tasks.cephfs_test_runner:test_standby_replay_failure (tasks.cephfs.test_failover.TestStandbyReplay) ... ok 2021-04-09T18:35:10.927 INFO:tasks.cephfs_test_runner:Starting test: test_standby_replay_singleton (tasks.cephfs.test_failover.TestStandbyReplay) 2021-04-09T18:35:10.928 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestStandbyReplay.test_standby_replay_singleton' 2021-04-09T18:35:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.059 7f6309775700 1 Processor -- start 2021-04-09T18:35:11.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.059 7f6309775700 1 -- start start 2021-04-09T18:35:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.060 7f6309775700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63040d7ca0 0x7f63040d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.060 7f6309775700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f630410da00 0x7f630410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:11.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.060 7f6309775700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630410d480 con 0x7f630410d010 2021-04-09T18:35:11.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.060 7f6309775700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63040d9220 con 0x7f630410da00 2021-04-09T18:35:11.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.060 7f6309775700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63040d86c0 con 0x7f63040d7ca0 2021-04-09T18:35:11.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.060 7f6303d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f630410da00 0x7f630410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:11.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.060 7f6303d75700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f630410da00 0x7f630410de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60172/0 (socket says 172.21.15.32:60172) 2021-04-09T18:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.060 7f6303d75700 1 -- 172.21.15.32:0/401402291 learned_addr learned my addr 172.21.15.32:0/401402291 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.060 7f6302d73700 1 --2- 172.21.15.32:0/401402291 >> v2:172.21.15.86:3300/0 conn(0x7f63040d7ca0 0x7f63040d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6303d75700 1 -- 172.21.15.32:0/401402291 >> v2:172.21.15.86:3300/0 conn(0x7f63040d7ca0 msgr2=0x7f63040d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:11.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6303d75700 1 --2- 172.21.15.32:0/401402291 >> v2:172.21.15.86:3300/0 conn(0x7f63040d7ca0 0x7f63040d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6303d75700 1 -- 172.21.15.32:0/401402291 >> v1:172.21.15.86:6789/0 conn(0x7f630410d010 legacy=0x7f63040d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:11.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6303d75700 1 -- 172.21.15.32:0/401402291 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304112100 con 0x7f630410da00 2021-04-09T18:35:11.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6303d75700 1 --2- 172.21.15.32:0/401402291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f630410da00 0x7f630410de00 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7f62fc006f00 tx=0x7f62fc009e90).ready entity=mon.0 client_cookie=9e22e26a0ff4600a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:11.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6302572700 1 -- 172.21.15.32:0/401402291 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62fc00b300 con 0x7f630410da00 2021-04-09T18:35:11.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6302572700 1 -- 172.21.15.32:0/401402291 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc00b640 con 0x7f630410da00 2021-04-09T18:35:11.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6302572700 1 -- 172.21.15.32:0/401402291 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62fc00bb20 con 0x7f630410da00 2021-04-09T18:35:11.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6309775700 1 -- 172.21.15.32:0/401402291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f630410da00 msgr2=0x7f630410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:11.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.061 7f6309775700 1 --2- 172.21.15.32:0/401402291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f630410da00 0x7f630410de00 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7f62fc006f00 tx=0x7f62fc009e90).stop 2021-04-09T18:35:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.062 7f6309775700 1 -- 172.21.15.32:0/401402291 shutdown_connections 2021-04-09T18:35:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.062 7f6309775700 1 --2- 172.21.15.32:0/401402291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f630410da00 0x7f630410de00 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.062 7f6309775700 1 --2- 172.21.15.32:0/401402291 >> v2:172.21.15.86:3300/0 conn(0x7f63040d7ca0 0x7f63040d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.062 7f6309775700 1 -- 172.21.15.32:0/401402291 >> 172.21.15.32:0/401402291 conn(0x7f63040ce5f0 msgr2=0x7f63040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.062 7f6309775700 1 -- 172.21.15.32:0/401402291 shutdown_connections 2021-04-09T18:35:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.062 7f6309775700 1 -- 172.21.15.32:0/401402291 wait complete. 2021-04-09T18:35:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.082 7f6309775700 1 Processor -- start 2021-04-09T18:35:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.082 7f6309775700 1 -- start start 2021-04-09T18:35:11.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6309775700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63040b42d0 0x7f6304183520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6309775700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63041ac090 0x7f63041ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6309775700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6304184d30 0x7f6304185130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6309775700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040a1a90 con 0x7f63040b42d0 2021-04-09T18:35:11.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6309775700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63041acab0 con 0x7f6304184d30 2021-04-09T18:35:11.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6309775700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6304185a00 con 0x7f63041ac090 2021-04-09T18:35:11.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63040b42d0 0x7f6304183520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:11.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303574700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63040b42d0 0x7f6304183520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60176/0 (socket says 172.21.15.32:60176) 2021-04-09T18:35:11.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303574700 1 -- 172.21.15.32:0/1809050924 learned_addr learned my addr 172.21.15.32:0/1809050924 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:11.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6302d73700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63041ac090 0x7f63041ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:11.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303d75700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6304184d30 0x7f6304185130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:11.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303574700 1 -- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63041ac090 msgr2=0x7f63041ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:11.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303574700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63041ac090 0x7f63041ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303574700 1 -- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6304184d30 msgr2=0x7f6304185130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:11.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303574700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6304184d30 0x7f6304185130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303574700 1 -- 172.21.15.32:0/1809050924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304185750 con 0x7f63040b42d0 2021-04-09T18:35:11.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6303574700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63040b42d0 0x7f6304183520 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7f62f400b5b0 tx=0x7f62f400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:11.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f62fbfff700 1 -- 172.21.15.32:0/1809050924 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62f400bd40 con 0x7f63040b42d0 2021-04-09T18:35:11.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f62fbfff700 1 -- 172.21.15.32:0/1809050924 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4006240 con 0x7f63040b42d0 2021-04-09T18:35:11.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6309775700 1 -- 172.21.15.32:0/1809050924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6304185750 con 0x7f63040b42d0 2021-04-09T18:35:11.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f62fbfff700 1 -- 172.21.15.32:0/1809050924 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62f4005690 con 0x7f63040b42d0 2021-04-09T18:35:11.114 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.083 7f6309775700 1 -- 172.21.15.32:0/1809050924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6304185750 con 0x7f63040b42d0 2021-04-09T18:35:11.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.084 7f62fbfff700 1 -- 172.21.15.32:0/1809050924 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f62f40054a0 con 0x7f63040b42d0 2021-04-09T18:35:11.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.084 7f62fbfff700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62e0018930 0x7f62e001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:11.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.084 7f62fbfff700 1 -- 172.21.15.32:0/1809050924 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f62f4032750 con 0x7f63040b42d0 2021-04-09T18:35:11.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.084 7f6302d73700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62e0018930 0x7f62e001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:11.115 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.085 7f6302d73700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62e0018930 0x7f62e001ae10 crc :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:11.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.104 7f6309775700 1 -- 172.21.15.32:0/1809050924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6304182d10 con 0x7f63040b42d0 2021-04-09T18:35:11.125 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.108 7f62fbfff700 1 -- 172.21.15.32:0/1809050924 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f62f4000c60 con 0x7f63040b42d0 2021-04-09T18:35:11.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.295 7f6309775700 1 -- 172.21.15.32:0/1809050924 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestStandbyReplay.test_standby_replay_singleton"]} v 0) v1 -- 0x7f630410d480 con 0x7f63040b42d0 2021-04-09T18:35:11.903 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.886 7f62fbfff700 1 -- 172.21.15.32:0/1809050924 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestStandbyReplay.test_standby_replay_singleton"]}]=0 v1002) v1 ==== 156+0+0 (secure 0 0 0) 0x7f62f40054a0 con 0x7f63040b42d0 2021-04-09T18:35:11.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 -- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62e0018930 msgr2=0x7f62e001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:11.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62e0018930 0x7f62e001ae10 crc :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 -- 172.21.15.32:0/1809050924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63040b42d0 msgr2=0x7f6304183520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:11.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63040b42d0 0x7f6304183520 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7f62f400b5b0 tx=0x7f62f400b950).stop 2021-04-09T18:35:11.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 -- 172.21.15.32:0/1809050924 shutdown_connections 2021-04-09T18:35:11.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f62e0018930 0x7f62e001ae10 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6304184d30 0x7f6304185130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63041ac090 0x7f63041ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 --2- 172.21.15.32:0/1809050924 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63040b42d0 0x7f6304183520 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:11.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.913 7f6309775700 1 -- 172.21.15.32:0/1809050924 >> 172.21.15.32:0/1809050924 conn(0x7f630419b890 msgr2=0x7f63041a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:11.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.914 7f6309775700 1 -- 172.21.15.32:0/1809050924 shutdown_connections 2021-04-09T18:35:11.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:11.914 7f6309775700 1 -- 172.21.15.32:0/1809050924 wait complete. 2021-04-09T18:35:11.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-04-09T18:35:12.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.085 7f0449680700 1 Processor -- start 2021-04-09T18:35:12.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.085 7f0449680700 1 -- start start 2021-04-09T18:35:12.131 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0449680700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f04440cffc0 0x7f04440d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0449680700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f04440d6ca0 0x7f044410cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0449680700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04440d72a0 con 0x7f04440d7820 2021-04-09T18:35:12.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0449680700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04440d70a0 con 0x7f04440d6ca0 2021-04-09T18:35:12.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0449680700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04440d9200 con 0x7f04440cffc0 2021-04-09T18:35:12.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0443ac0700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f04440d6ca0 0x7f044410cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0442abe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f04440d7820 0x7f04440d7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41544/0 (socket says 172.21.15.32:41544) 2021-04-09T18:35:12.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0442abe700 1 -- 172.21.15.32:0/1775836898 learned_addr learned my addr 172.21.15.32:0/1775836898 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:12.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f04432bf700 1 --2- 172.21.15.32:0/1775836898 >> v2:172.21.15.86:3300/0 conn(0x7f04440cffc0 0x7f04440d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0443ac0700 1 -- 172.21.15.32:0/1775836898 >> v2:172.21.15.86:3300/0 conn(0x7f04440cffc0 msgr2=0x7f04440d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:12.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0443ac0700 1 --2- 172.21.15.32:0/1775836898 >> v2:172.21.15.86:3300/0 conn(0x7f04440cffc0 0x7f04440d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0443ac0700 1 -- 172.21.15.32:0/1775836898 >> v1:172.21.15.32:6789/0 conn(0x7f04440d7820 legacy=0x7f04440d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f0443ac0700 1 -- 172.21.15.32:0/1775836898 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04440d8410 con 0x7f04440d6ca0 2021-04-09T18:35:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f04422bd700 1 -- 172.21.15.32:0/1775836898 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189069859 0 0) 0x7f0438001060 con 0x7f04440d7820 2021-04-09T18:35:12.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.086 7f04432bf700 1 --2- 172.21.15.32:0/1775836898 >> v2:172.21.15.86:3300/0 conn(0x7f04440cffc0 0x7f04440d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:12.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.087 7f0443ac0700 1 --2- 172.21.15.32:0/1775836898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04440d6ca0 0x7f044410cff0 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f043c006ab0 tx=0x7f043c009b40).ready entity=mon.0 client_cookie=35318f96bcce8083 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:12.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.087 7f04422bd700 1 -- 172.21.15.32:0/1775836898 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f043c0022e0 con 0x7f04440d6ca0 2021-04-09T18:35:12.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.087 7f04422bd700 1 -- 172.21.15.32:0/1775836898 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c002660 con 0x7f04440d6ca0 2021-04-09T18:35:12.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.087 7f04422bd700 1 -- 172.21.15.32:0/1775836898 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f043c000c10 con 0x7f04440d6ca0 2021-04-09T18:35:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.087 7f0449680700 1 -- 172.21.15.32:0/1775836898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04440d6ca0 msgr2=0x7f044410cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.087 7f0449680700 1 --2- 172.21.15.32:0/1775836898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04440d6ca0 0x7f044410cff0 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f043c006ab0 tx=0x7f043c009b40).stop 2021-04-09T18:35:12.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.088 7f0449680700 1 -- 172.21.15.32:0/1775836898 shutdown_connections 2021-04-09T18:35:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.088 7f0449680700 1 --2- 172.21.15.32:0/1775836898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04440d6ca0 0x7f044410cff0 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.088 7f0449680700 1 --2- 172.21.15.32:0/1775836898 >> v2:172.21.15.86:3300/0 conn(0x7f04440cffc0 0x7f04440d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.088 7f0449680700 1 -- 172.21.15.32:0/1775836898 >> 172.21.15.32:0/1775836898 conn(0x7f04440ce5e0 msgr2=0x7f04440cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:12.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.088 7f0449680700 1 -- 172.21.15.32:0/1775836898 shutdown_connections 2021-04-09T18:35:12.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.088 7f0449680700 1 -- 172.21.15.32:0/1775836898 wait complete. 2021-04-09T18:35:12.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0449680700 1 Processor -- start 2021-04-09T18:35:12.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0449680700 1 -- start start 2021-04-09T18:35:12.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0449680700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04441833a0 0x7f04440a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0449680700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04440a2550 0x7f04440a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0449680700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0444094d00 0x7f0444095100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0449680700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04440a1fd0 con 0x7f0444094d00 2021-04-09T18:35:12.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0449680700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04440a2f70 con 0x7f04441833a0 2021-04-09T18:35:12.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0449680700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0444095720 con 0x7f04440a2550 2021-04-09T18:35:12.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0443ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0444094d00 0x7f0444095100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0443ac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0444094d00 0x7f0444095100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60192/0 (socket says 172.21.15.32:60192) 2021-04-09T18:35:12.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0443ac0700 1 -- 172.21.15.32:0/407766571 learned_addr learned my addr 172.21.15.32:0/407766571 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:12.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.108 7f0442abe700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04440a2550 0x7f04440a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f04432bf700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04441833a0 0x7f04440a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f0443ac0700 1 -- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04440a2550 msgr2=0x7f04440a2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f0443ac0700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04440a2550 0x7f04440a2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f0443ac0700 1 -- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04441833a0 msgr2=0x7f04440a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f0443ac0700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04441833a0 0x7f04440a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f0443ac0700 1 -- 172.21.15.32:0/407766571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0444095940 con 0x7f0444094d00 2021-04-09T18:35:12.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f0443ac0700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0444094d00 0x7f0444095100 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 rx=0x7f043c00b5f0 tx=0x7f043c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:12.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f042bfff700 1 -- 172.21.15.32:0/407766571 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f043c00bd80 con 0x7f0444094d00 2021-04-09T18:35:12.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f042bfff700 1 -- 172.21.15.32:0/407766571 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c006020 con 0x7f0444094d00 2021-04-09T18:35:12.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f0449680700 1 -- 172.21.15.32:0/407766571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0444095940 con 0x7f0444094d00 2021-04-09T18:35:12.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f042bfff700 1 -- 172.21.15.32:0/407766571 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f043c0054a0 con 0x7f0444094d00 2021-04-09T18:35:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.109 7f0449680700 1 -- 172.21.15.32:0/407766571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0444095940 con 0x7f0444094d00 2021-04-09T18:35:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.110 7f042bfff700 1 -- 172.21.15.32:0/407766571 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f043c0054a0 con 0x7f0444094d00 2021-04-09T18:35:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.110 7f042bfff700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0424018930 0x7f042401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.110 7f042bfff700 1 -- 172.21.15.32:0/407766571 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f043c032730 con 0x7f0444094d00 2021-04-09T18:35:12.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.110 7f04432bf700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0424018930 0x7f042401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.110 7f04432bf700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0424018930 0x7f042401adc0 crc :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:12.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.130 7f0449680700 1 -- 172.21.15.32:0/407766571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0444095be0 con 0x7f0444094d00 2021-04-09T18:35:12.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.134 7f042bfff700 1 -- 172.21.15.32:0/407766571 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f043c01f750 con 0x7f0444094d00 2021-04-09T18:35:12.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.347 7f0449680700 1 -- 172.21.15.32:0/407766571 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0444095be0 con 0x7f0444094d00 2021-04-09T18:35:12.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.350 7f042bfff700 1 -- 172.21.15.32:0/407766571 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7f043c0054a0 con 0x7f0444094d00 2021-04-09T18:35:12.375 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 -- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0424018930 msgr2=0x7f042401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0424018930 0x7f042401adc0 crc :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.376 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 -- 172.21.15.32:0/407766571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0444094d00 msgr2=0x7f0444095100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0444094d00 0x7f0444095100 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 rx=0x7f043c00b5f0 tx=0x7f043c00b990).stop 2021-04-09T18:35:12.377 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 -- 172.21.15.32:0/407766571 shutdown_connections 2021-04-09T18:35:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0424018930 0x7f042401adc0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0444094d00 0x7f0444095100 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.378 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04440a2550 0x7f04440a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 --2- 172.21.15.32:0/407766571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04441833a0 0x7f04440a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.379 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 -- 172.21.15.32:0/407766571 >> 172.21.15.32:0/407766571 conn(0x7f044409fa00 msgr2=0x7f04440b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:12.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 -- 172.21.15.32:0/407766571 shutdown_connections 2021-04-09T18:35:12.380 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.356 7f0449680700 1 -- 172.21.15.32:0/407766571 wait complete. 2021-04-09T18:35:12.384 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:35:12.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:35:12.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:35:12.500 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:35:12.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:12.501 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:35:12.502 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:35:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:35:12.546 INFO:teuthology.orchestra.run.smithi032.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-04-09T18:35:12.546 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-04-09T18:35:12.547 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:12.548 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:35:12.548 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:35:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:35:12.591 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:35:12.591 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:35:12.592 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:35:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.724 7f65045a0700 1 Processor -- start 2021-04-09T18:35:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.724 7f65045a0700 1 -- start start 2021-04-09T18:35:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.725 7f65045a0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc0d7ca0 0x7f64fc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.725 7f65045a0700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc10d480 con 0x7f64fc10d010 2021-04-09T18:35:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.725 7f65045a0700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc0d86c0 con 0x7f64fc10da00 2021-04-09T18:35:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.725 7f65045a0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0d9220 con 0x7f64fc0d7ca0 2021-04-09T18:35:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.725 7f65023bc700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f64fc10d010 0x7f64fc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41556/0 (socket says 172.21.15.32:41556) 2021-04-09T18:35:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.725 7f65023bc700 1 -- 172.21.15.32:0/56052272 learned_addr learned my addr 172.21.15.32:0/56052272 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f6501bbb700 1 --2- 172.21.15.32:0/56052272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc0d7ca0 0x7f64fc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65013ba700 1 -- 172.21.15.32:0/56052272 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261552798 0 0) 0x7f64f8000ce0 con 0x7f64fc10d010 2021-04-09T18:35:12.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65013ba700 1 -- 172.21.15.32:0/56052272 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e8002fe0 con 0x7f64fc10d010 2021-04-09T18:35:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65013ba700 1 -- 172.21.15.32:0/56052272 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302036028 0 0) 0x7f64f4000d30 con 0x7f64fc10da00 2021-04-09T18:35:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65013ba700 1 -- 172.21.15.32:0/56052272 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e8003210 con 0x7f64fc10da00 2021-04-09T18:35:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f6501bbb700 1 -- 172.21.15.32:0/56052272 >> v1:172.21.15.86:6789/0 conn(0x7f64fc10da00 legacy=0x7f64fc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f6501bbb700 1 -- 172.21.15.32:0/56052272 >> v1:172.21.15.32:6789/0 conn(0x7f64fc10d010 legacy=0x7f64fc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65013ba700 1 -- 172.21.15.32:0/56052272 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2397821422 0 0) 0x7f64f80010e0 con 0x7f64fc10d010 2021-04-09T18:35:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f6501bbb700 1 -- 172.21.15.32:0/56052272 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc0d1f70 con 0x7f64fc0d7ca0 2021-04-09T18:35:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f6501bbb700 1 --2- 172.21.15.32:0/56052272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc0d7ca0 0x7f64fc0d80a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f64ec006650 tx=0x7f64ec009a00).ready entity=mon.2 client_cookie=dcce7a94fc6d210d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:12.769 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65013ba700 1 -- 172.21.15.32:0/56052272 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64ec009df0 con 0x7f64fc0d7ca0 2021-04-09T18:35:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65013ba700 1 -- 172.21.15.32:0/56052272 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec0009b0 con 0x7f64fc0d7ca0 2021-04-09T18:35:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65013ba700 1 -- 172.21.15.32:0/56052272 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64ec00c2b0 con 0x7f64fc0d7ca0 2021-04-09T18:35:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65045a0700 1 -- 172.21.15.32:0/56052272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc0d7ca0 msgr2=0x7f64fc0d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.726 7f65045a0700 1 --2- 172.21.15.32:0/56052272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc0d7ca0 0x7f64fc0d80a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f64ec006650 tx=0x7f64ec009a00).stop 2021-04-09T18:35:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.727 7f65045a0700 1 -- 172.21.15.32:0/56052272 shutdown_connections 2021-04-09T18:35:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.727 7f65045a0700 1 --2- 172.21.15.32:0/56052272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc0d7ca0 0x7f64fc0d80a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.727 7f65045a0700 1 -- 172.21.15.32:0/56052272 >> 172.21.15.32:0/56052272 conn(0x7f64fc0ce5f0 msgr2=0x7f64fc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:12.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.727 7f65045a0700 1 -- 172.21.15.32:0/56052272 shutdown_connections 2021-04-09T18:35:12.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.727 7f65045a0700 1 -- 172.21.15.32:0/56052272 wait complete. 2021-04-09T18:35:12.772 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f65045a0700 1 Processor -- start 2021-04-09T18:35:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f65045a0700 1 -- start start 2021-04-09T18:35:12.775 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f65045a0700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc1ac380 0x7f64fc18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f65045a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64fc1ac780 0x7f64fc18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f65045a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc183520 0x7f64fc183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f65045a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc1ace10 con 0x7f64fc183520 2021-04-09T18:35:12.776 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f65045a0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc189cf0 con 0x7f64fc1ac780 2021-04-09T18:35:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f65045a0700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc183f40 con 0x7f64fc1ac380 2021-04-09T18:35:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f6502bbd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc183520 0x7f64fc183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f6502bbd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc183520 0x7f64fc183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60208/0 (socket says 172.21.15.32:60208) 2021-04-09T18:35:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f6502bbd700 1 -- 172.21.15.32:0/1385930461 learned_addr learned my addr 172.21.15.32:0/1385930461 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:12.777 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f65023bc700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc1ac380 0x7f64fc18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.748 7f6501bbb700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64fc1ac780 0x7f64fc18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f6502bbd700 1 -- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc1ac380 msgr2=0x7f64fc18a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f6502bbd700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc1ac380 0x7f64fc18a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.778 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f6502bbd700 1 -- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64fc1ac780 msgr2=0x7f64fc18a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:12.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f6502bbd700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64fc1ac780 0x7f64fc18a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:12.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f6502bbd700 1 -- 172.21.15.32:0/1385930461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc184160 con 0x7f64fc183520 2021-04-09T18:35:12.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f6502bbd700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc183520 0x7f64fc183920 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7f64f4009980 tx=0x7f64f400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:12.779 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f64f2ffd700 1 -- 172.21.15.32:0/1385930461 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64f400be10 con 0x7f64fc183520 2021-04-09T18:35:12.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f64f2ffd700 1 -- 172.21.15.32:0/1385930461 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f4005870 con 0x7f64fc183520 2021-04-09T18:35:12.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f65045a0700 1 -- 172.21.15.32:0/1385930461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64fc184160 con 0x7f64fc183520 2021-04-09T18:35:12.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f64f2ffd700 1 -- 172.21.15.32:0/1385930461 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64f4005d50 con 0x7f64fc183520 2021-04-09T18:35:12.780 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.749 7f65045a0700 1 -- 172.21.15.32:0/1385930461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64fc1846f0 con 0x7f64fc183520 2021-04-09T18:35:12.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.750 7f64f2ffd700 1 -- 172.21.15.32:0/1385930461 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f64f4005a50 con 0x7f64fc183520 2021-04-09T18:35:12.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.750 7f64f2ffd700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64e4018930 0x7f64e401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:12.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.750 7f64f2ffd700 1 -- 172.21.15.32:0/1385930461 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f64f4032aa0 con 0x7f64fc183520 2021-04-09T18:35:12.781 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.750 7f65023bc700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64e4018930 0x7f64e401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.750 7f65023bc700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64e4018930 0x7f64e401ae10 crc :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.769 7f65045a0700 1 -- 172.21.15.32:0/1385930461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64fc189cf0 con 0x7f64fc183520 2021-04-09T18:35:12.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.773 7f64f2ffd700 1 -- 172.21.15.32:0/1385930461 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f64f400be10 con 0x7f64fc183520 2021-04-09T18:35:12.979 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.962 7f65045a0700 1 -- 172.21.15.32:0/1385930461 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f64fc189cf0 con 0x7f64fc183520 2021-04-09T18:35:12.980 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.963 7f64f2ffd700 1 -- 172.21.15.32:0/1385930461 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v269) v1 ==== 81+0+25550 (secure 0 0 0) 0x7f64f4005870 con 0x7f64fc183520 2021-04-09T18:35:12.981 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:12.982 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:35:12.982 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 269, 2021-04-09T18:35:12.982 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:35:12.982 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:35:12.983 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:35:09.875648", 2021-04-09T18:35:12.983 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:35:12.983 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:35:12.984 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:35:12.984 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:35:12.984 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:35:12.984 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:35:12.984 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:35:12.985 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:35:12.985 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:35:12.985 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:35:12.985 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:35:12.986 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:35:12.986 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:35:12.986 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:35:12.986 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:35:12.986 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 45, 2021-04-09T18:35:12.987 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:35:12.987 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:35:12.987 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:35:12.987 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:35:12.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:35:12.988 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:12.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:35:12.988 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:35:12.989 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:35:12.989 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:35:12.989 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:35:12.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:35:12.989 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:35:12.990 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:35:12.990 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:35:12.990 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:35:12.990 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:35:12.991 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:35:12.991 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:35:12.991 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:35:12.991 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:35:12.991 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:35:12.992 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:35:12.992 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:35:12.992 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:35:12.992 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:35:12.993 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:35:12.993 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:35:12.993 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:35:12.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:12.994 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:35:12.994 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:35:12.995 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:35:12.995 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:35:12.995 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:35:12.996 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:35:12.997 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:35:12.997 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:35:12.997 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:35:12.998 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:35:12.998 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:35:12.999 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:35:12.999 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:35:12.999 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:35:13.000 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:35:13.000 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:35:13.001 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:35:13.002 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:35:13.002 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:35:13.003 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:35:13.003 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:35:13.003 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:35:13.004 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:35:13.004 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:35:13.005 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:35:13.005 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:35:13.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:35:13.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.006 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:35:13.007 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:35:13.007 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:35:13.008 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:35:13.008 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:35:13.008 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:35:13.009 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:35:13.009 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:35:13.010 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:35:13.010 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:35:13.011 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:35:13.011 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:35:13.011 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:35:13.012 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:35:13.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.013 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.013 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:35:13.013 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:35:13.014 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.014 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:35:13.015 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:35:13.015 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:13.016 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:13.016 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:13.016 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:13.017 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:13.017 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:13.018 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:13.018 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:13.019 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:13.019 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:13.020 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:13.020 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.021 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.021 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:35:13.022 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:13.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.023 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.023 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.024 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:35:13.024 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:13.024 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.025 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.026 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:13.026 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.026 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.027 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.027 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:35:13.028 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:13.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.029 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:35:13.029 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:13.030 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.030 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.031 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:13.031 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.031 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.032 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:35:13.032 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:13.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.033 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.033 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:35:13.033 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:13.034 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.034 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.034 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:13.034 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.035 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:35:13.036 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:13.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.036 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.037 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.037 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:35:13.038 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:35:13.038 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.038 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.039 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:35:13.040 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:35:13.040 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:35:13.040 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:35:13.041 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:13.041 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:13.042 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:13.042 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.043 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.043 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:35:13.044 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:35:13.044 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:13.045 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:13.045 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:13.045 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:13.046 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:13.046 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:13.047 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:13.047 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 256, 2021-04-09T18:35:13.048 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:13.048 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:13.048 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:13.049 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.049 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.050 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.050 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:35:13.050 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:13.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.051 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.052 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:35:13.053 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:13.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.053 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.054 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:13.055 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.055 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.056 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.056 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:35:13.056 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:13.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.058 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:35:13.059 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:13.059 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.059 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.060 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:13.060 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:35:13.061 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:13.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.062 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.063 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:35:13.063 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:13.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.064 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.064 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:13.064 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.065 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.065 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:35:13.066 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:13.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.067 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:35:13.067 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:35:13.068 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.068 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.069 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:35:13.069 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:35:13.069 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:35:13.070 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:35:13.070 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:13.070 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.072 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.072 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:35:13.072 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:35:13.073 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:13.073 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:13.073 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:13.073 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:13.074 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:13.074 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:13.074 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:13.075 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:13.075 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:13.075 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:13.075 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:13.076 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.076 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.077 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:35:13.077 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:13.077 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.077 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.078 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:35:13.078 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:13.079 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.079 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.079 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:13.080 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.080 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.080 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.081 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:35:13.081 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:13.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.081 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.082 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.082 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:35:13.082 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:13.082 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.083 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.083 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:13.083 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.084 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.084 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.084 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:35:13.085 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:13.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.085 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.086 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:35:13.086 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:13.086 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.086 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.087 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.088 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.088 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:35:13.088 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:13.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.089 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.089 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.089 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:35:13.089 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:35:13.090 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.090 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.090 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:35:13.091 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:35:13.091 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:35:13.091 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:35:13.091 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:13.092 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:13.092 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:13.092 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.093 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.093 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:35:13.093 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:35:13.093 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:13.094 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:13.094 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:13.094 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:13.094 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:13.095 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:13.095 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:13.095 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:13.096 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:13.096 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:13.096 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:13.096 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.097 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.097 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:35:13.097 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:13.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.098 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.098 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.098 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:35:13.098 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:13.099 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.099 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.100 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:13.100 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.100 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.100 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.101 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:35:13.101 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:13.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.101 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.102 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:35:13.102 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:13.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.103 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.103 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:13.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.104 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.104 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:35:13.105 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:13.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.106 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:35:13.106 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:13.106 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.106 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.107 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:13.107 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.108 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.108 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:35:13.108 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:13.109 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.109 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.109 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.109 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:35:13.110 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:35:13.110 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.110 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.111 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:35:13.111 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:35:13.111 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:35:13.111 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:35:13.112 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:13.112 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:13.112 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:13.113 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.113 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.113 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:35:13.114 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:35:13.114 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:13.114 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:13.114 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:13.115 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:13.115 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:13.115 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:13.116 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:13.116 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:13.116 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:13.116 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:13.117 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:13.117 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.117 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.118 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:35:13.118 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:13.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.118 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.119 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:35:13.119 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:13.120 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.120 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.120 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:13.121 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.121 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:35:13.122 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:13.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.122 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.122 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.123 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:35:13.123 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:13.123 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.123 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.124 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:13.124 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.125 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:35:13.125 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:13.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.126 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.126 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:35:13.126 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:13.127 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.127 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.127 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:13.128 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.128 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.128 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.128 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:35:13.129 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:13.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.130 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:35:13.130 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:35:13.130 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.130 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.131 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:35:13.131 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:35:13.131 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:35:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:35:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:13.132 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:13.133 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:13.133 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.133 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.133 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:35:13.134 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:35:13.134 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:13.134 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:13.134 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:13.135 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:13.135 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:13.135 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:13.136 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:13.136 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 256, 2021-04-09T18:35:13.136 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:13.136 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:13.137 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:13.137 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.137 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.138 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:35:13.138 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:13.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.138 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.139 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:35:13.139 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:13.139 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.140 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.140 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:13.140 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.141 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.141 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.141 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:35:13.141 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:13.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.142 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.142 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:35:13.143 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:13.143 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.143 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.144 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:13.144 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.144 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.145 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:35:13.145 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:13.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.145 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.146 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:35:13.146 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:13.146 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:35:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:35:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:35:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:35:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:35:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:35:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:35:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:35:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:35:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:35:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:35:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:35:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:35:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:35:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:35:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:35:13.169 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:13.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.169 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:35:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:35:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:35:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:35:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:35:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:35:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:35:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:35:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:35:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:35:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:35:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:35:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:35:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:35:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:35:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:35:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:35:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:35:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:35:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:35:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:35:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:35:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:35:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:35:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:35:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:35:13.186 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:13.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.186 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:35:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:35:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:35:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:35:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:35:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:35:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:35:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.192 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:35:13.192 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:35:13.192 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:35:13.192 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:35:13.193 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:35:13.193 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:35:13.193 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:35:13.193 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:35:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:35:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:35:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:35:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:35:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:13.200 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:13.200 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:13.200 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:13.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.201 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.201 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:35:13.201 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:13.201 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:13.202 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:13.202 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:13.202 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:13.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.203 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.203 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:35:13.203 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:13.204 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:13.204 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:13.204 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:13.204 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:13.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.205 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.205 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:35:13.205 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:13.206 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:13.206 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:13.206 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:13.206 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:13.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.207 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.207 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:35:13.207 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:13.208 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:13.208 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:13.208 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:13.209 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:13.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.209 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:13.209 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:35:13.210 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:35:13.210 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:35:13.210 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:35:13.210 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:35:13.211 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:35:13.211 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.211 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:35:13.211 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:35:13.212 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:35:13.212 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:35:13.212 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:35:13.212 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": { 2021-04-09T18:35:13.213 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6835/2610523978": "2021-04-10 18:35:05.868775", 2021-04-09T18:35:13.213 INFO:teuthology.orchestra.run.smithi032.stdout: "172.21.15.32:6833/2610523978": "2021-04-10 18:35:05.868775" 2021-04-09T18:35:13.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.213 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:35:13.214 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:35:13.214 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:35:13.214 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:35:13.214 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:35:13.215 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:35:13.215 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:35:13.215 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:35:13.215 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:13.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:13.216 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:35:13.216 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:35:13.216 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:35:13.217 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:35:13.217 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:35:13.217 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:35:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 -- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64e4018930 msgr2=0x7f64e401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.219 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64e4018930 0x7f64e401ae10 crc :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 -- 172.21.15.32:0/1385930461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc183520 msgr2=0x7f64fc183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc183520 0x7f64fc183920 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7f64f4009980 tx=0x7f64f400bcd0).stop 2021-04-09T18:35:13.220 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 -- 172.21.15.32:0/1385930461 shutdown_connections 2021-04-09T18:35:13.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f64e4018930 0x7f64e401ae10 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc183520 0x7f64fc183920 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64fc1ac780 0x7f64fc18a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.221 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 --2- 172.21.15.32:0/1385930461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64fc1ac380 0x7f64fc18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 -- 172.21.15.32:0/1385930461 >> 172.21.15.32:0/1385930461 conn(0x7f64fc19b890 msgr2=0x7f64fc1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:13.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 -- 172.21.15.32:0/1385930461 shutdown_connections 2021-04-09T18:35:13.222 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:12.965 7f65045a0700 1 -- 172.21.15.32:0/1385930461 wait complete. 2021-04-09T18:35:13.223 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.356 7f1e3fe70700 1 Processor -- start 2021-04-09T18:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.356 7f1e3fe70700 1 -- start start 2021-04-09T18:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3fe70700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1e3810d010 0x7f1e380d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3fe70700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1e380d1da0 0x7f1e380d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3fe70700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3810d480 con 0x7f1e380d82c0 2021-04-09T18:35:13.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3fe70700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e380d8d30 con 0x7f1e3810d010 2021-04-09T18:35:13.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3fe70700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e380d9220 con 0x7f1e380d1da0 2021-04-09T18:35:13.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3dc8c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f1e3810d010 0x7f1e380d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3e48d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1e380d82c0 0x7f1e3810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41574/0 (socket says 172.21.15.32:41574) 2021-04-09T18:35:13.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3e48d700 1 -- 172.21.15.32:0/4104909225 learned_addr learned my addr 172.21.15.32:0/4104909225 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:13.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3d48b700 1 --2- 172.21.15.32:0/4104909225 >> v2:172.21.15.86:3300/0 conn(0x7f1e380d1da0 0x7f1e380d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3dc8c700 1 -- 172.21.15.32:0/4104909225 >> v2:172.21.15.86:3300/0 conn(0x7f1e380d1da0 msgr2=0x7f1e380d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:13.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3dc8c700 1 --2- 172.21.15.32:0/4104909225 >> v2:172.21.15.86:3300/0 conn(0x7f1e380d1da0 0x7f1e380d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3dc8c700 1 -- 172.21.15.32:0/4104909225 >> v1:172.21.15.32:6789/0 conn(0x7f1e380d82c0 legacy=0x7f1e3810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3cc8a700 1 -- 172.21.15.32:0/4104909225 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260847022 0 0) 0x7f1e340009b0 con 0x7f1e380d82c0 2021-04-09T18:35:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3dc8c700 1 -- 172.21.15.32:0/4104909225 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e380d86f0 con 0x7f1e3810d010 2021-04-09T18:35:13.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3dc8c700 1 --2- 172.21.15.32:0/4104909225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3810d010 0x7f1e380d1780 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7f1e300069d0 tx=0x7f1e30009dc0).ready entity=mon.0 client_cookie=421e11b17fc03c3a server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:13.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3cc8a700 1 -- 172.21.15.32:0/4104909225 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e3000b510 con 0x7f1e3810d010 2021-04-09T18:35:13.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3cc8a700 1 -- 172.21.15.32:0/4104909225 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30006ac0 con 0x7f1e3810d010 2021-04-09T18:35:13.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.357 7f1e3cc8a700 1 -- 172.21.15.32:0/4104909225 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e3000ba80 con 0x7f1e3810d010 2021-04-09T18:35:13.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.358 7f1e3fe70700 1 -- 172.21.15.32:0/4104909225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3810d010 msgr2=0x7f1e380d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.358 7f1e3fe70700 1 --2- 172.21.15.32:0/4104909225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3810d010 0x7f1e380d1780 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7f1e300069d0 tx=0x7f1e30009dc0).stop 2021-04-09T18:35:13.404 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.358 7f1e3fe70700 1 -- 172.21.15.32:0/4104909225 shutdown_connections 2021-04-09T18:35:13.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.358 7f1e3fe70700 1 --2- 172.21.15.32:0/4104909225 >> v2:172.21.15.86:3300/0 conn(0x7f1e380d1da0 0x7f1e380d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.358 7f1e3fe70700 1 --2- 172.21.15.32:0/4104909225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e3810d010 0x7f1e380d1780 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.358 7f1e3fe70700 1 -- 172.21.15.32:0/4104909225 >> 172.21.15.32:0/4104909225 conn(0x7f1e380ce5f0 msgr2=0x7f1e380d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:13.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.358 7f1e3fe70700 1 -- 172.21.15.32:0/4104909225 shutdown_connections 2021-04-09T18:35:13.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.358 7f1e3fe70700 1 -- 172.21.15.32:0/4104909225 wait complete. 2021-04-09T18:35:13.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3fe70700 1 Processor -- start 2021-04-09T18:35:13.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3fe70700 1 -- start start 2021-04-09T18:35:13.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3fe70700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e381ac380 0x7f1e3818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3fe70700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e381ac780 0x7f1e3818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3fe70700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e38183520 0x7f1e38183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3fe70700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e381ace10 con 0x7f1e38183520 2021-04-09T18:35:13.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3fe70700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e38189cf0 con 0x7f1e381ac780 2021-04-09T18:35:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3fe70700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e38183f40 con 0x7f1e381ac380 2021-04-09T18:35:13.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3e48d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e38183520 0x7f1e38183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3e48d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e38183520 0x7f1e38183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60222/0 (socket says 172.21.15.32:60222) 2021-04-09T18:35:13.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3e48d700 1 -- 172.21.15.32:0/2584301817 learned_addr learned my addr 172.21.15.32:0/2584301817 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:13.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3dc8c700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e381ac380 0x7f1e3818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.379 7f1e3d48b700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e381ac780 0x7f1e3818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e3e48d700 1 -- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e381ac380 msgr2=0x7f1e3818a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e3e48d700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e381ac380 0x7f1e3818a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e3e48d700 1 -- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e381ac780 msgr2=0x7f1e3818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e3e48d700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e381ac780 0x7f1e3818a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e3e48d700 1 -- 172.21.15.32:0/2584301817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e38184160 con 0x7f1e38183520 2021-04-09T18:35:13.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e3dc8c700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e381ac380 0x7f1e3818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:13.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e3e48d700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e38183520 0x7f1e38183920 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7f1e3400a590 tx=0x7f1e3400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:13.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e2e7fc700 1 -- 172.21.15.32:0/2584301817 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e3400ad20 con 0x7f1e38183520 2021-04-09T18:35:13.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e2e7fc700 1 -- 172.21.15.32:0/2584301817 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e340043a0 con 0x7f1e38183520 2021-04-09T18:35:13.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e38184160 con 0x7f1e38183520 2021-04-09T18:35:13.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e2e7fc700 1 -- 172.21.15.32:0/2584301817 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e34004880 con 0x7f1e38183520 2021-04-09T18:35:13.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.380 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e38184160 con 0x7f1e38183520 2021-04-09T18:35:13.418 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.381 7f1e2e7fc700 1 -- 172.21.15.32:0/2584301817 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f1e34004580 con 0x7f1e38183520 2021-04-09T18:35:13.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.381 7f1e2e7fc700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e20018930 0x7f1e2001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.381 7f1e2e7fc700 1 -- 172.21.15.32:0/2584301817 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f1e34032730 con 0x7f1e38183520 2021-04-09T18:35:13.419 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.381 7f1e3dc8c700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e20018930 0x7f1e2001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.420 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.381 7f1e3dc8c700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e20018930 0x7f1e2001ae10 crc :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:13.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.401 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e38189cf0 con 0x7f1e38183520 2021-04-09T18:35:13.421 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.404 7f1e2e7fc700 1 -- 172.21.15.32:0/2584301817 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f1e340045b0 con 0x7f1e38183520 2021-04-09T18:35:13.612 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.595 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e38189cf0 con 0x7f1e38183520 2021-04-09T18:35:13.613 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.595 7f1e2e7fc700 1 -- 172.21.15.32:0/2584301817 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 438 v438) v1 ==== 95+0+1858 (secure 0 0 0) 0x7f1e340043a0 con 0x7f1e38183520 2021-04-09T18:35:13.614 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 438 2021-04-09T18:35:13.614 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:13.614 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":438,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13128,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3252718041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3252718041},{"type":"v1","addr":"172.21.15.32:6834","nonce":3252718041}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":437},{"gid":13149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1660114813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1660114813},{"type":"v1","addr":"172.21.15.32:6835","nonce":1660114813}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":438},{"gid":59948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/4251960201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":4251960201},{"type":"v1","addr":"172.21.15.86:6836","nonce":4251960201}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59951,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1668354365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1668354365},{"type":"v1","addr":"172.21.15.86:6837","nonce":1668354365}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422}],"filesystems":[]} 2021-04-09T18:35:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e20018930 msgr2=0x7f1e2001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e20018930 0x7f1e2001ae10 crc :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.616 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e38183520 msgr2=0x7f1e38183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e38183520 0x7f1e38183920 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7f1e3400a590 tx=0x7f1e3400a930).stop 2021-04-09T18:35:13.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 shutdown_connections 2021-04-09T18:35:13.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f1e20018930 0x7f1e2001ae10 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.617 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e38183520 0x7f1e38183920 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e381ac780 0x7f1e3818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 --2- 172.21.15.32:0/2584301817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e381ac380 0x7f1e3818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 >> 172.21.15.32:0/2584301817 conn(0x7f1e3819b890 msgr2=0x7f1e381a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:13.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 shutdown_connections 2021-04-09T18:35:13.618 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.598 7f1e3fe70700 1 -- 172.21.15.32:0/2584301817 wait complete. 2021-04-09T18:35:13.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:35:13.803 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.758 7f0b0487b700 1 Processor -- start 2021-04-09T18:35:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.758 7f0b0487b700 1 -- start start 2021-04-09T18:35:13.804 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0b0487b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0b0010d010 0x7f0b000d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0b0487b700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b000d8490 0x7f0b0010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0b0487b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0010d480 con 0x7f0b000d1da0 2021-04-09T18:35:13.805 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0b0487b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b000d8d30 con 0x7f0b0010d010 2021-04-09T18:35:13.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0b0487b700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b000d9220 con 0x7f0b000d8490 2021-04-09T18:35:13.806 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afe61c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0b0010d010 0x7f0b000d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afde1b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0b000d1da0 0x7f0b000d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41584/0 (socket says 172.21.15.32:41584) 2021-04-09T18:35:13.807 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afde1b700 1 -- 172.21.15.32:0/2517896212 learned_addr learned my addr 172.21.15.32:0/2517896212 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afee1d700 1 --2- 172.21.15.32:0/2517896212 >> v2:172.21.15.86:3300/0 conn(0x7f0b000d8490 0x7f0b0010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afe61c700 1 -- 172.21.15.32:0/2517896212 >> v2:172.21.15.86:3300/0 conn(0x7f0b000d8490 msgr2=0x7f0b0010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:13.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afe61c700 1 --2- 172.21.15.32:0/2517896212 >> v2:172.21.15.86:3300/0 conn(0x7f0b000d8490 0x7f0b0010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afe61c700 1 -- 172.21.15.32:0/2517896212 >> v1:172.21.15.32:6789/0 conn(0x7f0b000d1da0 legacy=0x7f0b000d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.809 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afe61c700 1 -- 172.21.15.32:0/2517896212 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0010e020 con 0x7f0b0010d010 2021-04-09T18:35:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afd61a700 1 -- 172.21.15.32:0/2517896212 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816995270 0 0) 0x7f0af4000ce0 con 0x7f0b000d1da0 2021-04-09T18:35:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afee1d700 1 --2- 172.21.15.32:0/2517896212 >> v2:172.21.15.86:3300/0 conn(0x7f0b000d8490 0x7f0b0010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:13.810 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afe61c700 1 --2- 172.21.15.32:0/2517896212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0010d010 0x7f0b000d1780 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 rx=0x7f0af0006650 tx=0x7f0af0009a80).ready entity=mon.0 client_cookie=f4f0d0fa089023a6 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afd61a700 1 -- 172.21.15.32:0/2517896212 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0af00009b0 con 0x7f0b0010d010 2021-04-09T18:35:13.811 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afd61a700 1 -- 172.21.15.32:0/2517896212 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af000c0c0 con 0x7f0b0010d010 2021-04-09T18:35:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.759 7f0afd61a700 1 -- 172.21.15.32:0/2517896212 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0af000c510 con 0x7f0b0010d010 2021-04-09T18:35:13.812 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.760 7f0b0487b700 1 -- 172.21.15.32:0/2517896212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0010d010 msgr2=0x7f0b000d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.760 7f0b0487b700 1 --2- 172.21.15.32:0/2517896212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0010d010 0x7f0b000d1780 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 rx=0x7f0af0006650 tx=0x7f0af0009a80).stop 2021-04-09T18:35:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.760 7f0b0487b700 1 -- 172.21.15.32:0/2517896212 shutdown_connections 2021-04-09T18:35:13.813 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.760 7f0b0487b700 1 --2- 172.21.15.32:0/2517896212 >> v2:172.21.15.86:3300/0 conn(0x7f0b000d8490 0x7f0b0010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.760 7f0b0487b700 1 --2- 172.21.15.32:0/2517896212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0010d010 0x7f0b000d1780 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.814 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.760 7f0b0487b700 1 -- 172.21.15.32:0/2517896212 >> 172.21.15.32:0/2517896212 conn(0x7f0b000ce5f0 msgr2=0x7f0b000d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:13.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.760 7f0b0487b700 1 -- 172.21.15.32:0/2517896212 shutdown_connections 2021-04-09T18:35:13.815 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.760 7f0b0487b700 1 -- 172.21.15.32:0/2517896212 wait complete. 2021-04-09T18:35:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0b0487b700 1 Processor -- start 2021-04-09T18:35:13.816 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0b0487b700 1 -- start start 2021-04-09T18:35:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0b0487b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b000b8320 0x7f0b000b8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0b0487b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0018c540 0x7f0b0018c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.817 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0b0487b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b0018a350 0x7f0b0018a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0b0487b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b00193080 con 0x7f0b0018c540 2021-04-09T18:35:13.818 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0b0487b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0018bfc0 con 0x7f0b0018a350 2021-04-09T18:35:13.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0b0487b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0018cf60 con 0x7f0b000b8320 2021-04-09T18:35:13.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0afde1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0018c540 0x7f0b0018c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.819 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0afde1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0018c540 0x7f0b0018c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60236/0 (socket says 172.21.15.32:60236) 2021-04-09T18:35:13.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0afde1b700 1 -- 172.21.15.32:0/499668839 learned_addr learned my addr 172.21.15.32:0/499668839 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:13.820 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0afe61c700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b000b8320 0x7f0b000b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.781 7f0afee1d700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b0018a350 0x7f0b0018a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0afde1b700 1 -- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b000b8320 msgr2=0x7f0b000b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.821 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0afde1b700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b000b8320 0x7f0b000b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0afde1b700 1 -- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b0018a350 msgr2=0x7f0b0018a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:13.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0afde1b700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b0018a350 0x7f0b0018a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:13.822 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0afde1b700 1 -- 172.21.15.32:0/499668839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0008a650 con 0x7f0b0018c540 2021-04-09T18:35:13.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0afe61c700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b000b8320 0x7f0b000b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:13.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0afde1b700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0018c540 0x7f0b0018c940 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 rx=0x7f0af4006f00 tx=0x7f0af400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:13.823 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0aeeffd700 1 -- 172.21.15.32:0/499668839 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0af4006020 con 0x7f0b0018c540 2021-04-09T18:35:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0aeeffd700 1 -- 172.21.15.32:0/499668839 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af40063a0 con 0x7f0b0018c540 2021-04-09T18:35:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0aeeffd700 1 -- 172.21.15.32:0/499668839 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0af4004620 con 0x7f0b0018c540 2021-04-09T18:35:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0b0487b700 1 -- 172.21.15.32:0/499668839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0008a650 con 0x7f0b0018c540 2021-04-09T18:35:13.824 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0b0487b700 1 -- 172.21.15.32:0/499668839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0008a650 con 0x7f0b0018c540 2021-04-09T18:35:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.782 7f0aeeffd700 1 -- 172.21.15.32:0/499668839 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f0af400a000 con 0x7f0b0018c540 2021-04-09T18:35:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.783 7f0aeeffd700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ae40188d0 0x7f0ae401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.783 7f0aeeffd700 1 -- 172.21.15.32:0/499668839 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 4843+0+0 (secure 0 0 0) 0x7f0af4032730 con 0x7f0b0018c540 2021-04-09T18:35:13.825 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.783 7f0afe61c700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ae40188d0 0x7f0ae401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:13.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.783 7f0afe61c700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ae40188d0 0x7f0ae401add0 crc :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:13.826 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.803 7f0b0487b700 1 -- 172.21.15.32:0/499668839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b00193080 con 0x7f0b0018c540 2021-04-09T18:35:13.827 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:13.806 7f0aeeffd700 1 -- 172.21.15.32:0/499668839 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f0af4006b90 con 0x7f0b0018c540 2021-04-09T18:35:14.017 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.000 7f0b0487b700 1 -- 172.21.15.32:0/499668839 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b00193080 con 0x7f0b0018c540 2021-04-09T18:35:14.018 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.000 7f0aeeffd700 1 -- 172.21.15.32:0/499668839 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 438 v438) v1 ==== 95+0+1858 (secure 0 0 0) 0x7f0af400a000 con 0x7f0b0018c540 2021-04-09T18:35:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 438 2021-04-09T18:35:14.019 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:14.020 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":438,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13128,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/3252718041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":3252718041},{"type":"v1","addr":"172.21.15.32:6834","nonce":3252718041}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":437},{"gid":13149,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1660114813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1660114813},{"type":"v1","addr":"172.21.15.32:6835","nonce":1660114813}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":438},{"gid":59948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6836/4251960201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":4251960201},{"type":"v1","addr":"172.21.15.86:6836","nonce":4251960201}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422},{"gid":59951,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1668354365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1668354365},{"type":"v1","addr":"172.21.15.86:6837","nonce":1668354365}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":422}],"filesystems":[]} 2021-04-09T18:35:14.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 -- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ae40188d0 msgr2=0x7f0ae401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:14.022 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ae40188d0 0x7f0ae401add0 crc :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 -- 172.21.15.32:0/499668839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0018c540 msgr2=0x7f0b0018c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:14.023 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0018c540 0x7f0b0018c940 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 rx=0x7f0af4006f00 tx=0x7f0af400b940).stop 2021-04-09T18:35:14.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 -- 172.21.15.32:0/499668839 shutdown_connections 2021-04-09T18:35:14.024 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f0ae40188d0 0x7f0ae401add0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b0018c540 0x7f0b0018c940 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b0018a350 0x7f0b0018a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.025 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 --2- 172.21.15.32:0/499668839 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b000b8320 0x7f0b000b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 -- 172.21.15.32:0/499668839 >> 172.21.15.32:0/499668839 conn(0x7f0b00183b00 msgr2=0x7f0b00193300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:14.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 -- 172.21.15.32:0/499668839 shutdown_connections 2021-04-09T18:35:14.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.003 7f0b0487b700 1 -- 172.21.15.32:0/499668839 wait complete. 2021-04-09T18:35:14.032 INFO:tasks.ceph.mds.a:Restarting daemon 2021-04-09T18:35:14.032 INFO:tasks.ceph.mds.a:Stopping old one... 2021-04-09T18:35:14.033 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-04-09T18:35:14.033 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:35:14.034 INFO:tasks.ceph.mds.c:Restarting daemon 2021-04-09T18:35:14.034 INFO:tasks.ceph.mds.c:Stopping old one... 2021-04-09T18:35:14.035 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2021-04-09T18:35:14.035 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:35:14.036 INFO:tasks.ceph.mds.b:Restarting daemon 2021-04-09T18:35:14.036 INFO:tasks.ceph.mds.b:Stopping old one... 2021-04-09T18:35:14.037 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-04-09T18:35:14.037 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:35:14.038 INFO:tasks.ceph.mds.d:Restarting daemon 2021-04-09T18:35:14.038 INFO:tasks.ceph.mds.d:Stopping old one... 2021-04-09T18:35:14.039 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2021-04-09T18:35:14.039 INFO:teuthology.orchestra.run:waiting for 300 2021-04-09T18:35:14.042 INFO:tasks.ceph.mds.a:Stopped 2021-04-09T18:35:14.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-04-09T18:35:14.045 INFO:tasks.ceph.mds.c:Stopped 2021-04-09T18:35:14.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2021-04-09T18:35:14.047 INFO:tasks.ceph.mds.b:Stopped 2021-04-09T18:35:14.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-04-09T18:35:14.049 INFO:tasks.ceph.mds.d:Stopped 2021-04-09T18:35:14.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2021-04-09T18:35:14.086 INFO:tasks.ceph.mds.a:Started 2021-04-09T18:35:14.087 DEBUG:teuthology.parallel:result is None 2021-04-09T18:35:14.089 INFO:tasks.ceph.mds.c:Started 2021-04-09T18:35:14.090 DEBUG:teuthology.parallel:result is None 2021-04-09T18:35:14.091 INFO:tasks.ceph.mds.b:Started 2021-04-09T18:35:14.092 INFO:tasks.ceph.mds.d:Started 2021-04-09T18:35:14.093 DEBUG:teuthology.parallel:result is None 2021-04-09T18:35:14.094 DEBUG:teuthology.parallel:result is None 2021-04-09T18:35:14.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2021-04-09T18:35:14.231 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2021-04-09T18:35:14.232 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2021-04-09T18:35:14.235 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2021-04-09T18:35:14.235 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2021-04-09T18:35:14.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.230 7fd968cc6700 1 Processor -- start 2021-04-09T18:35:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.230 7fd968cc6700 1 -- start start 2021-04-09T18:35:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd968cc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640d6c90 0x7fd9640d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd968cc6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9640d7290 con 0x7fd9640d2450 2021-04-09T18:35:14.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd968cc6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9640d2850 con 0x7fd9640cffb0 2021-04-09T18:35:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd968cc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9640d7090 con 0x7fd9640d6c90 2021-04-09T18:35:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96305f700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd9640d2450 0x7fd96410cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41624/0 (socket says 172.21.15.32:41624) 2021-04-09T18:35:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96305f700 1 -- 172.21.15.32:0/2753103442 learned_addr learned my addr 172.21.15.32:0/2753103442 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:14.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96205d700 1 --2- 172.21.15.32:0/2753103442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640d6c90 0x7fd9640d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96185c700 1 -- 172.21.15.32:0/2753103442 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272567378 0 0) 0x7fd95c000d30 con 0x7fd9640d2450 2021-04-09T18:35:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96185c700 1 -- 172.21.15.32:0/2753103442 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd94c002fe0 con 0x7fd9640d2450 2021-04-09T18:35:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96185c700 1 -- 172.21.15.32:0/2753103442 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191255538 0 0) 0x7fd954000c60 con 0x7fd9640cffb0 2021-04-09T18:35:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96185c700 1 -- 172.21.15.32:0/2753103442 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd94c0042c0 con 0x7fd9640cffb0 2021-04-09T18:35:14.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96205d700 1 -- 172.21.15.32:0/2753103442 >> v1:172.21.15.86:6789/0 conn(0x7fd9640cffb0 legacy=0x7fd9640d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96205d700 1 -- 172.21.15.32:0/2753103442 >> v1:172.21.15.32:6789/0 conn(0x7fd9640d2450 legacy=0x7fd96410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96205d700 1 -- 172.21.15.32:0/2753103442 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9640d91f0 con 0x7fd9640d6c90 2021-04-09T18:35:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96205d700 1 --2- 172.21.15.32:0/2753103442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640d6c90 0x7fd9640d8000 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fd958006650 tx=0x7fd958009a80).ready entity=mon.2 client_cookie=31c47e21ef58922d server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:14.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96185c700 1 -- 172.21.15.32:0/2753103442 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9580009b0 con 0x7fd9640d6c90 2021-04-09T18:35:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96185c700 1 -- 172.21.15.32:0/2753103442 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95800c0c0 con 0x7fd9640d6c90 2021-04-09T18:35:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd96185c700 1 -- 172.21.15.32:0/2753103442 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd95800c510 con 0x7fd9640d6c90 2021-04-09T18:35:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.231 7fd968cc6700 1 -- 172.21.15.32:0/2753103442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640d6c90 msgr2=0x7fd9640d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.232 7fd968cc6700 1 --2- 172.21.15.32:0/2753103442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640d6c90 0x7fd9640d8000 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fd958006650 tx=0x7fd958009a80).stop 2021-04-09T18:35:14.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.232 7fd968cc6700 1 -- 172.21.15.32:0/2753103442 shutdown_connections 2021-04-09T18:35:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.232 7fd968cc6700 1 --2- 172.21.15.32:0/2753103442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640d6c90 0x7fd9640d8000 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.232 7fd968cc6700 1 -- 172.21.15.32:0/2753103442 >> 172.21.15.32:0/2753103442 conn(0x7fd9640ce5d0 msgr2=0x7fd9640cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.232 7fd968cc6700 1 -- 172.21.15.32:0/2753103442 shutdown_connections 2021-04-09T18:35:14.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.232 7fd968cc6700 1 -- 172.21.15.32:0/2753103442 wait complete. 2021-04-09T18:35:14.278 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd968cc6700 1 Processor -- start 2021-04-09T18:35:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd968cc6700 1 -- start start 2021-04-09T18:35:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd968cc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640a80a0 0x7fd96417cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd968cc6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd96417d0e0 0x7fd96408c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:14.279 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd968cc6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd96408ce00 0x7fd96417afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd968cc6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96408c2e0 con 0x7fd96408ce00 2021-04-09T18:35:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd968cc6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96408d200 con 0x7fd96417d0e0 2021-04-09T18:35:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd968cc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96418f280 con 0x7fd9640a80a0 2021-04-09T18:35:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd96305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd96408ce00 0x7fd96417afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:14.280 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd96305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd96408ce00 0x7fd96417afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60274/0 (socket says 172.21.15.32:60274) 2021-04-09T18:35:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd96305f700 1 -- 172.21.15.32:0/2184004686 learned_addr learned my addr 172.21.15.32:0/2184004686 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd96205d700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd96417d0e0 0x7fd96408c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.253 7fd96285e700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640a80a0 0x7fd96417cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:14.281 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd96305f700 1 -- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640a80a0 msgr2=0x7fd96417cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd96305f700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640a80a0 0x7fd96417cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd96305f700 1 -- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd96417d0e0 msgr2=0x7fd96408c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd96305f700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd96417d0e0 0x7fd96408c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd96305f700 1 -- 172.21.15.32:0/2184004686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96417b600 con 0x7fd96408ce00 2021-04-09T18:35:14.282 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd96205d700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd96417d0e0 0x7fd96408c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd96305f700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd96408ce00 0x7fd96417afe0 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7fd95c00b8c0 tx=0x7fd95c00bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd9537fe700 1 -- 172.21.15.32:0/2184004686 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd95c00bda0 con 0x7fd96408ce00 2021-04-09T18:35:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd9537fe700 1 -- 172.21.15.32:0/2184004686 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c006030 con 0x7fd96408ce00 2021-04-09T18:35:14.283 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd9537fe700 1 -- 172.21.15.32:0/2184004686 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd95c0064c0 con 0x7fd96408ce00 2021-04-09T18:35:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96417b600 con 0x7fd96408ce00 2021-04-09T18:35:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.254 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96417b600 con 0x7fd96408ce00 2021-04-09T18:35:14.284 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.255 7fd9537fe700 1 -- 172.21.15.32:0/2184004686 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fd95c006210 con 0x7fd96408ce00 2021-04-09T18:35:14.285 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.255 7fd9537fe700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd948018900 0x7fd94801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.255 7fd9537fe700 1 -- 172.21.15.32:0/2184004686 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 4843+0+0 (secure 0 0 0) 0x7fd95c032c40 con 0x7fd96408ce00 2021-04-09T18:35:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.255 7fd96285e700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd948018900 0x7fd94801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:14.286 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.255 7fd96285e700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd948018900 0x7fd94801adc0 crc :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:14.291 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.274 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96408d200 con 0x7fd96408ce00 2021-04-09T18:35:14.295 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.277 7fd9537fe700 1 -- 172.21.15.32:0/2184004686 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fd95c000f90 con 0x7fd96408ce00 2021-04-09T18:35:14.496 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.479 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fd96408d200 con 0x7fd96408ce00 2021-04-09T18:35:14.909 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.892 7fd9537fe700 1 -- 172.21.15.32:0/2184004686 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v270) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd95c006030 con 0x7fd96408ce00 2021-04-09T18:35:14.911 INFO:teuthology.orchestra.run.smithi032.stderr: removed all blacklist entries 2021-04-09T18:35:14.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd948018900 msgr2=0x7fd94801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:14.914 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd948018900 0x7fd94801adc0 crc :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd96408ce00 msgr2=0x7fd96417afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:14.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd96408ce00 0x7fd96417afe0 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7fd95c00b8c0 tx=0x7fd95c00bc60).stop 2021-04-09T18:35:14.915 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 shutdown_connections 2021-04-09T18:35:14.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fd948018900 0x7fd94801adc0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd96408ce00 0x7fd96417afe0 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.916 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd96417d0e0 0x7fd96408c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 --2- 172.21.15.32:0/2184004686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9640a80a0 0x7fd96417cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:14.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 >> 172.21.15.32:0/2184004686 conn(0x7fd9641905e0 msgr2=0x7fd9641b7350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:14.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 shutdown_connections 2021-04-09T18:35:14.917 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:14.896 7fd968cc6700 1 -- 172.21.15.32:0/2184004686 wait complete. 2021-04-09T18:35:14.924 INFO:tasks.cephfs.cephfs_test_case:['0', '1'] 2021-04-09T18:35:14.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2021-04-09T18:35:15.095 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.056 7f9f73b50700 1 Processor -- start 2021-04-09T18:35:15.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.056 7f9f73b50700 1 -- start start 2021-04-09T18:35:15.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.056 7f9f73b50700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9f6c10d010 0x7f9f6c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f73b50700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c0d8490 0x7f9f6c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.096 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f73b50700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6c10d480 con 0x7f9f6c0d1da0 2021-04-09T18:35:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f73b50700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6c0d8d30 con 0x7f9f6c10d010 2021-04-09T18:35:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f73b50700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c0d9220 con 0x7f9f6c0d8490 2021-04-09T18:35:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7196c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9f6c10d010 0x7f9f6c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7196c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f9f6c10d010 0x7f9f6c0d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60278/0 (socket says 172.21.15.32:60278) 2021-04-09T18:35:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7196c700 1 -- 172.21.15.32:0/3051570581 learned_addr learned my addr 172.21.15.32:0/3051570581 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7216d700 1 --2- 172.21.15.32:0/3051570581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c0d8490 0x7f9f6c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7196c700 1 -- 172.21.15.32:0/3051570581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c0d8490 msgr2=0x7f9f6c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:15.098 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7196c700 1 --2- 172.21.15.32:0/3051570581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c0d8490 0x7f9f6c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7196c700 1 -- 172.21.15.32:0/3051570581 >> v1:172.21.15.86:6789/0 conn(0x7f9f6c0d1da0 legacy=0x7f9f6c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7196c700 1 -- 172.21.15.32:0/3051570581 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c10e020 con 0x7f9f6c10d010 2021-04-09T18:35:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7216d700 1 --2- 172.21.15.32:0/3051570581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c0d8490 0x7f9f6c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7196c700 1 --2- 172.21.15.32:0/3051570581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c10d010 0x7f9f6c0d1780 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7f9f64006a80 tx=0x7f9f64009f00).ready entity=mon.0 client_cookie=9570750bbe69a00 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7096a700 1 -- 172.21.15.32:0/3051570581 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f6400b5c0 con 0x7f9f6c10d010 2021-04-09T18:35:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7096a700 1 -- 172.21.15.32:0/3051570581 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6400b8c0 con 0x7f9f6c10d010 2021-04-09T18:35:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f7096a700 1 -- 172.21.15.32:0/3051570581 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f6400bda0 con 0x7f9f6c10d010 2021-04-09T18:35:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f73b50700 1 -- 172.21.15.32:0/3051570581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c10d010 msgr2=0x7f9f6c0d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.057 7f9f73b50700 1 --2- 172.21.15.32:0/3051570581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c10d010 0x7f9f6c0d1780 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7f9f64006a80 tx=0x7f9f64009f00).stop 2021-04-09T18:35:15.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.058 7f9f73b50700 1 -- 172.21.15.32:0/3051570581 shutdown_connections 2021-04-09T18:35:15.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.058 7f9f73b50700 1 --2- 172.21.15.32:0/3051570581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c0d8490 0x7f9f6c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.058 7f9f73b50700 1 --2- 172.21.15.32:0/3051570581 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c10d010 0x7f9f6c0d1780 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.058 7f9f73b50700 1 -- 172.21.15.32:0/3051570581 >> 172.21.15.32:0/3051570581 conn(0x7f9f6c0ce5f0 msgr2=0x7f9f6c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:15.101 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.058 7f9f73b50700 1 -- 172.21.15.32:0/3051570581 shutdown_connections 2021-04-09T18:35:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.058 7f9f73b50700 1 -- 172.21.15.32:0/3051570581 wait complete. 2021-04-09T18:35:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.078 7f9f73b50700 1 Processor -- start 2021-04-09T18:35:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.078 7f9f73b50700 1 -- start start 2021-04-09T18:35:15.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f73b50700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c0a90c0 0x7f9f6c1a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f73b50700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1a76d0 0x7f9f6c1a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f73b50700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c17bce0 0x7f9f6c17c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f73b50700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1a7c60 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f73b50700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1a8e20 con 0x7f9f6c0a90c0 2021-04-09T18:35:15.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f73b50700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c17c700 con 0x7f9f6c17bce0 2021-04-09T18:35:15.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7116b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1a76d0 0x7f9f6c1a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7116b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1a76d0 0x7f9f6c1a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60284/0 (socket says 172.21.15.32:60284) 2021-04-09T18:35:15.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7116b700 1 -- 172.21.15.32:0/3523153774 learned_addr learned my addr 172.21.15.32:0/3523153774 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:15.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7196c700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c0a90c0 0x7f9f6c1a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.107 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7216d700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c17bce0 0x7f9f6c17c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7116b700 1 -- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c17bce0 msgr2=0x7f9f6c17c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7116b700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c17bce0 0x7f9f6c17c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7116b700 1 -- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c0a90c0 msgr2=0x7f9f6c1a81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7116b700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c0a90c0 0x7f9f6c1a81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7116b700 1 -- 172.21.15.32:0/3523153774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c17c920 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.079 7f9f7116b700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1a76d0 0x7f9f6c1a8800 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7f9f6000b5b0 tx=0x7f9f6000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:15.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.080 7f9f6a7fc700 1 -- 172.21.15.32:0/3523153774 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f6000bd40 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.080 7f9f6a7fc700 1 -- 172.21.15.32:0/3523153774 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f60006240 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.080 7f9f6a7fc700 1 -- 172.21.15.32:0/3523153774 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f60005690 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.080 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c17c920 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.080 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c17c920 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.080 7f9f6a7fc700 1 -- 172.21.15.32:0/3523153774 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9f600054a0 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.080 7f9f6a7fc700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f4c0188d0 0x7f9f4c01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.080 7f9f6a7fc700 1 -- 172.21.15.32:0/3523153774 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f9f60032700 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.081 7f9f7196c700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f4c0188d0 0x7f9f4c01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.081 7f9f7196c700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f4c0188d0 0x7f9f4c01add0 crc :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:15.118 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.101 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c1b7010 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.121 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.104 7f9f6a7fc700 1 -- 172.21.15.32:0/3523153774 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9f60000c60 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.289 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9f6c10d480 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.307 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.290 7f9f6a7fc700 1 -- 172.21.15.32:0/3523153774 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+4539 (secure 0 0 0) 0x7f9f600054a0 con 0x7f9f6c1a76d0 2021-04-09T18:35:15.308 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:15.308 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:35:15.308 INFO:teuthology.orchestra.run.smithi032.stdout: "auth_dump": [ 2021-04-09T18:35:15.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.309 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.a", 2021-04-09T18:35:15.309 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBg4nUQJRAAUUcE7IDMg7fwcTkQKFYb5w==", 2021-04-09T18:35:15.309 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.310 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:35:15.310 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.310 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.310 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.311 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.311 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.b", 2021-04-09T18:35:15.311 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgSeG9OBAAyFYthIiLH7lqUWA8Ck26GQ==", 2021-04-09T18:35:15.312 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.312 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:35:15.312 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.312 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.312 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.313 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.314 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.314 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.c", 2021-04-09T18:35:15.314 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgDKgwLxAAAgZuxHOoV4PABHOCMYmViQ==", 2021-04-09T18:35:15.314 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.315 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:35:15.315 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.315 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.315 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.315 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.316 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mds.d", 2021-04-09T18:35:15.316 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgCg5cBhAAGg7DxPwfZytJb0wsE7o0Zg==", 2021-04-09T18:35:15.317 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.317 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:35:15.317 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.317 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.317 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.318 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.0", 2021-04-09T18:35:15.320 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDTmHBgqFXkKRAAVZsoGsw2sgg2jEGSZBF+dQ==", 2021-04-09T18:35:15.320 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.320 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.320 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.321 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.321 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.321 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.321 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.1", 2021-04-09T18:35:15.322 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDWmHBguSH+JxAA7CXRldeYGYTwTcXFAEqYFg==", 2021-04-09T18:35:15.322 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.322 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.322 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.323 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.323 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.324 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.2", 2021-04-09T18:35:15.324 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDZmHBg4vufJRAABL2aKLeKGvavwtFh5+paLw==", 2021-04-09T18:35:15.324 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.325 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.325 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.325 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.325 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.326 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.3", 2021-04-09T18:35:15.327 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDcmHBgVMiaIhAAT0vh5kVH+dGs/flBg21mSw==", 2021-04-09T18:35:15.327 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.327 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.327 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.327 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.328 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.328 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.4", 2021-04-09T18:35:15.329 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDimHBgeDcRKhAAI2BKrBi3h8iIEwC+Pjg5Bg==", 2021-04-09T18:35:15.329 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.329 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.329 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.329 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.330 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.330 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.5", 2021-04-09T18:35:15.330 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDlmHBg0qa4JhAAKUMqN60PEUfUS3TzNTAj4Q==", 2021-04-09T18:35:15.331 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.331 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.331 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.331 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.332 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.332 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.332 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.6", 2021-04-09T18:35:15.332 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDomHBgHhaLIxAAXD/IEhSwrClaU8itK/Nwrg==", 2021-04-09T18:35:15.333 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.333 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.333 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.333 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.333 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.334 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "osd.7", 2021-04-09T18:35:15.334 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDrmHBgcdVFIBAAFqgcvx2NGXLCChedymL5Qg==", 2021-04-09T18:35:15.335 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.335 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.335 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.335 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.336 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.0", 2021-04-09T18:35:15.336 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgZOdHDxAACfg4XhgSrfUXRCjkBYhsUw==", 2021-04-09T18:35:15.336 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.337 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:35:15.337 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:35:15.337 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:35:15.337 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.338 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.338 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.1", 2021-04-09T18:35:15.338 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDSmHBgs2XgFRAAsC9q/7DwfAMiPYwk7rvdag==", 2021-04-09T18:35:15.338 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.339 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow", 2021-04-09T18:35:15.339 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow r", 2021-04-09T18:35:15.339 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow rw pool=cephfs_data" 2021-04-09T18:35:15.339 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.340 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "client.admin", 2021-04-09T18:35:15.340 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDQmHBg4wfNKxAAYh2qMTz/6VxywyXjXq9L8A==", 2021-04-09T18:35:15.340 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.340 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:35:15.341 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr": "allow *", 2021-04-09T18:35:15.341 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow *", 2021-04-09T18:35:15.341 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.341 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.342 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.342 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.x", 2021-04-09T18:35:15.342 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgrgzSHRAADD0DkkVwbBy+WhtzXYTpbA==", 2021-04-09T18:35:15.342 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.343 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:35:15.343 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:35:15.343 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.343 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:15.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:15.344 INFO:teuthology.orchestra.run.smithi032.stdout: "entity": "mgr.y", 2021-04-09T18:35:15.344 INFO:teuthology.orchestra.run.smithi032.stdout: "key": "AQDRmHBgqvpWFhAAcOdX3ouSMDZBmxrg6EtJOg==", 2021-04-09T18:35:15.344 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": { 2021-04-09T18:35:15.345 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": "allow *", 2021-04-09T18:35:15.345 INFO:teuthology.orchestra.run.smithi032.stdout: "mon": "allow profile mgr", 2021-04-09T18:35:15.345 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": "allow *" 2021-04-09T18:35:15.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.345 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:15.346 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:35:15.346 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:35:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f4c0188d0 msgr2=0x7f9f4c01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f4c0188d0 0x7f9f4c01add0 crc :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1a76d0 msgr2=0x7f9f6c1a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1a76d0 0x7f9f6c1a8800 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7f9f6000b5b0 tx=0x7f9f6000b950).stop 2021-04-09T18:35:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 shutdown_connections 2021-04-09T18:35:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9f4c0188d0 0x7f9f4c01add0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9f6c1a76d0 0x7f9f6c1a8800 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c17bce0 0x7f9f6c17c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 --2- 172.21.15.32:0/3523153774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c0a90c0 0x7f9f6c1a81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 >> 172.21.15.32:0/3523153774 conn(0x7f9f6c1acae0 msgr2=0x7f9f6c17ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:15.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 shutdown_connections 2021-04-09T18:35:15.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.292 7f9f73b50700 1 -- 172.21.15.32:0/3523153774 wait complete. 2021-04-09T18:35:15.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.483 7ffa3b358700 1 Processor -- start 2021-04-09T18:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.483 7ffa3b358700 1 -- start start 2021-04-09T18:35:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa3b358700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3410da00 0x7ffa3410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa3b358700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa3410d480 con 0x7ffa340d7ca0 2021-04-09T18:35:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa3b358700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa340d1f70 con 0x7ffa3410d010 2021-04-09T18:35:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa3b358700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa340d9220 con 0x7ffa3410da00 2021-04-09T18:35:15.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa38973700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffa340d7ca0 0x7ffa340d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41652/0 (socket says 172.21.15.32:41652) 2021-04-09T18:35:15.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa38973700 1 -- 172.21.15.32:0/1669297809 learned_addr learned my addr 172.21.15.32:0/1669297809 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:15.525 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa39975700 1 --2- 172.21.15.32:0/1669297809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3410da00 0x7ffa3410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa2bfff700 1 -- 172.21.15.32:0/1669297809 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579398366 0 0) 0x7ffa24000ce0 con 0x7ffa340d7ca0 2021-04-09T18:35:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa2bfff700 1 -- 172.21.15.32:0/1669297809 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa20002fe0 con 0x7ffa340d7ca0 2021-04-09T18:35:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa2bfff700 1 -- 172.21.15.32:0/1669297809 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762847826 0 0) 0x7ffa30000c60 con 0x7ffa3410d010 2021-04-09T18:35:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa2bfff700 1 -- 172.21.15.32:0/1669297809 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa20002fe0 con 0x7ffa3410d010 2021-04-09T18:35:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa39975700 1 -- 172.21.15.32:0/1669297809 >> v1:172.21.15.86:6789/0 conn(0x7ffa3410d010 legacy=0x7ffa340d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa39975700 1 -- 172.21.15.32:0/1669297809 >> v1:172.21.15.32:6789/0 conn(0x7ffa340d7ca0 legacy=0x7ffa340d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa2bfff700 1 -- 172.21.15.32:0/1669297809 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 911988056 0 0) 0x7ffa240010e0 con 0x7ffa340d7ca0 2021-04-09T18:35:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa39975700 1 -- 172.21.15.32:0/1669297809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa34112100 con 0x7ffa3410da00 2021-04-09T18:35:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.484 7ffa39975700 1 --2- 172.21.15.32:0/1669297809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3410da00 0x7ffa3410de00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7ffa2c006e50 tx=0x7ffa2c009df0).ready entity=mon.2 client_cookie=eeb1d905017a116c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.485 7ffa2bfff700 1 -- 172.21.15.32:0/1669297809 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa2c00b510 con 0x7ffa3410da00 2021-04-09T18:35:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.485 7ffa2bfff700 1 -- 172.21.15.32:0/1669297809 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2c003030 con 0x7ffa3410da00 2021-04-09T18:35:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.485 7ffa2bfff700 1 -- 172.21.15.32:0/1669297809 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa2c00b7d0 con 0x7ffa3410da00 2021-04-09T18:35:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.485 7ffa3b358700 1 -- 172.21.15.32:0/1669297809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3410da00 msgr2=0x7ffa3410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.485 7ffa3b358700 1 --2- 172.21.15.32:0/1669297809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3410da00 0x7ffa3410de00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7ffa2c006e50 tx=0x7ffa2c009df0).stop 2021-04-09T18:35:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.485 7ffa3b358700 1 -- 172.21.15.32:0/1669297809 shutdown_connections 2021-04-09T18:35:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.485 7ffa3b358700 1 --2- 172.21.15.32:0/1669297809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3410da00 0x7ffa3410de00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.485 7ffa3b358700 1 -- 172.21.15.32:0/1669297809 >> 172.21.15.32:0/1669297809 conn(0x7ffa340ce5f0 msgr2=0x7ffa340d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:15.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.486 7ffa3b358700 1 -- 172.21.15.32:0/1669297809 shutdown_connections 2021-04-09T18:35:15.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.486 7ffa3b358700 1 -- 172.21.15.32:0/1669297809 wait complete. 2021-04-09T18:35:15.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa3b358700 1 Processor -- start 2021-04-09T18:35:15.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa3b358700 1 -- start start 2021-04-09T18:35:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa3b358700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa341ac670 0x7ffa341a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa3b358700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa341a5dc0 0x7ffa340ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa3b358700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa341aca70 0x7ffa340ba800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa3b358700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa341a5220 con 0x7ffa341aca70 2021-04-09T18:35:15.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa3b358700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3418e100 con 0x7ffa341a5dc0 2021-04-09T18:35:15.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa3b358700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3419ccc0 con 0x7ffa341ac670 2021-04-09T18:35:15.538 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa39975700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa341aca70 0x7ffa340ba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.506 7ffa39975700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa341aca70 0x7ffa340ba800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60302/0 (socket says 172.21.15.32:60302) 2021-04-09T18:35:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa39975700 1 -- 172.21.15.32:0/2690485234 learned_addr learned my addr 172.21.15.32:0/2690485234 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:15.539 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa38973700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa341a5dc0 0x7ffa340ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa39174700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa341ac670 0x7ffa341a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa39975700 1 -- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa341ac670 msgr2=0x7ffa341a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa39975700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa341ac670 0x7ffa341a57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa39975700 1 -- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa341a5dc0 msgr2=0x7ffa340ba1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa39975700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa341a5dc0 0x7ffa340ba1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa39975700 1 -- 172.21.15.32:0/2690485234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa340bae20 con 0x7ffa341aca70 2021-04-09T18:35:15.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa38973700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa341a5dc0 0x7ffa340ba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:35:15.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa39975700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa341aca70 0x7ffa340ba800 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7ffa2c009f80 tx=0x7ffa2c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:15.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa29ffb700 1 -- 172.21.15.32:0/2690485234 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa2c0097c0 con 0x7ffa341aca70 2021-04-09T18:35:15.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa29ffb700 1 -- 172.21.15.32:0/2690485234 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2c0093d0 con 0x7ffa341aca70 2021-04-09T18:35:15.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa340bae20 con 0x7ffa341aca70 2021-04-09T18:35:15.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa29ffb700 1 -- 172.21.15.32:0/2690485234 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa2c001430 con 0x7ffa341aca70 2021-04-09T18:35:15.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.507 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa340bae20 con 0x7ffa341aca70 2021-04-09T18:35:15.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.508 7ffa29ffb700 1 -- 172.21.15.32:0/2690485234 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ffa2c001240 con 0x7ffa341aca70 2021-04-09T18:35:15.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.508 7ffa29ffb700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa140188d0 0x7ffa1401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.508 7ffa29ffb700 1 -- 172.21.15.32:0/2690485234 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 4757+0+0 (secure 0 0 0) 0x7ffa2c032bf0 con 0x7ffa341aca70 2021-04-09T18:35:15.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.508 7ffa39174700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa140188d0 0x7ffa1401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.508 7ffa39174700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa140188d0 0x7ffa1401add0 crc :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:15.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.528 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa34182d10 con 0x7ffa341aca70 2021-04-09T18:35:15.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.531 7ffa29ffb700 1 -- 172.21.15.32:0/2690485234 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ffa2c0093b0 con 0x7ffa341aca70 2021-04-09T18:35:15.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.722 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa34182d10 con 0x7ffa341aca70 2021-04-09T18:35:15.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.723 7ffa29ffb700 1 -- 172.21.15.32:0/2690485234 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+11154 (secure 0 0 0) 0x7ffa2c0093b0 con 0x7ffa341aca70 2021-04-09T18:35:15.742 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:15.742 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":270,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:35:14.887582","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":45,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":256,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:35:15.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa140188d0 msgr2=0x7ffa1401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa140188d0 0x7ffa1401add0 crc :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa341aca70 msgr2=0x7ffa340ba800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa341aca70 0x7ffa340ba800 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7ffa2c009f80 tx=0x7ffa2c00bd10).stop 2021-04-09T18:35:15.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 shutdown_connections 2021-04-09T18:35:15.747 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ffa140188d0 0x7ffa1401add0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa341aca70 0x7ffa340ba800 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa341a5dc0 0x7ffa340ba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.748 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 --2- 172.21.15.32:0/2690485234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa341ac670 0x7ffa341a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 >> 172.21.15.32:0/2690485234 conn(0x7ffa3419b890 msgr2=0x7ffa341a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:15.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 shutdown_connections 2021-04-09T18:35:15.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.725 7ffa3b358700 1 -- 172.21.15.32:0/2690485234 wait complete. 2021-04-09T18:35:15.754 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-09T18:35:15.755 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:35:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.892 7fc02b484700 1 Processor -- start 2021-04-09T18:35:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.892 7fc02b484700 1 -- start start 2021-04-09T18:35:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.892 7fc02b484700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc0240d7ca0 0x7fc0240d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.892 7fc02b484700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02410d480 con 0x7fc02410d010 2021-04-09T18:35:15.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.892 7fc02b484700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0240d86c0 con 0x7fc02410da00 2021-04-09T18:35:15.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.892 7fc02b484700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0240d9220 con 0x7fc0240d7ca0 2021-04-09T18:35:15.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc0292a0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc02410d010 0x7fc0240d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41664/0 (socket says 172.21.15.32:41664) 2021-04-09T18:35:15.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc0292a0700 1 -- 172.21.15.32:0/2693057484 learned_addr learned my addr 172.21.15.32:0/2693057484 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:15.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc028a9f700 1 --2- 172.21.15.32:0/2693057484 >> v2:172.21.15.32:3300/0 conn(0x7fc0240d7ca0 0x7fc0240d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc028a9f700 1 -- 172.21.15.32:0/2693057484 >> v1:172.21.15.86:6789/0 conn(0x7fc02410da00 legacy=0x7fc02410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc028a9f700 1 -- 172.21.15.32:0/2693057484 >> v1:172.21.15.32:6789/0 conn(0x7fc02410d010 legacy=0x7fc0240d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc028a9f700 1 -- 172.21.15.32:0/2693057484 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0240d1f70 con 0x7fc0240d7ca0 2021-04-09T18:35:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc028a9f700 1 --2- 172.21.15.32:0/2693057484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0240d7ca0 0x7fc0240d80a0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7fc018006650 tx=0x7fc018009a80).ready entity=mon.0 client_cookie=fc1bd3721d428b56 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:15.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc023fff700 1 -- 172.21.15.32:0/2693057484 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0180009b0 con 0x7fc0240d7ca0 2021-04-09T18:35:15.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc023fff700 1 -- 172.21.15.32:0/2693057484 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01800c0c0 con 0x7fc0240d7ca0 2021-04-09T18:35:15.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc023fff700 1 -- 172.21.15.32:0/2693057484 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc01800c510 con 0x7fc0240d7ca0 2021-04-09T18:35:15.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc02b484700 1 -- 172.21.15.32:0/2693057484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0240d7ca0 msgr2=0x7fc0240d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.893 7fc02b484700 1 --2- 172.21.15.32:0/2693057484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0240d7ca0 0x7fc0240d80a0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7fc018006650 tx=0x7fc018009a80).stop 2021-04-09T18:35:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.894 7fc02b484700 1 -- 172.21.15.32:0/2693057484 shutdown_connections 2021-04-09T18:35:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.894 7fc02b484700 1 --2- 172.21.15.32:0/2693057484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0240d7ca0 0x7fc0240d80a0 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.894 7fc02b484700 1 -- 172.21.15.32:0/2693057484 >> 172.21.15.32:0/2693057484 conn(0x7fc0240ce5f0 msgr2=0x7fc0240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.894 7fc02b484700 1 -- 172.21.15.32:0/2693057484 shutdown_connections 2021-04-09T18:35:15.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.894 7fc02b484700 1 -- 172.21.15.32:0/2693057484 wait complete. 2021-04-09T18:35:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.914 7fc02b484700 1 Processor -- start 2021-04-09T18:35:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.914 7fc02b484700 1 -- start start 2021-04-09T18:35:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc02b484700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0241c2850 0x7fc0240bc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc02b484700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc02408cc40 0x7fc02408d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc02b484700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0241a5680 0x7fc0241a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc02b484700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240bc8a0 con 0x7fc02408cc40 2021-04-09T18:35:15.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc02b484700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02408d660 con 0x7fc0241a5680 2021-04-09T18:35:15.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc02b484700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0241a60a0 con 0x7fc0241c2850 2021-04-09T18:35:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc028a9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc02408cc40 0x7fc02408d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc028a9f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc02408cc40 0x7fc02408d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60314/0 (socket says 172.21.15.32:60314) 2021-04-09T18:35:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc028a9f700 1 -- 172.21.15.32:0/1333049572 learned_addr learned my addr 172.21.15.32:0/1333049572 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc0292a0700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0241c2850 0x7fc0240bc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.915 7fc029aa1700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0241a5680 0x7fc0241a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc028a9f700 1 -- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0241c2850 msgr2=0x7fc0240bc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc028a9f700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0241c2850 0x7fc0240bc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc028a9f700 1 -- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0241a5680 msgr2=0x7fc0241a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc028a9f700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0241a5680 0x7fc0241a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:15.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc028a9f700 1 -- 172.21.15.32:0/1333049572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0241a6490 con 0x7fc02408cc40 2021-04-09T18:35:15.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc028a9f700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc02408cc40 0x7fc02408d040 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7fc018009650 tx=0x7fc01800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:15.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc021ffb700 1 -- 172.21.15.32:0/1333049572 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0180093a0 con 0x7fc02408cc40 2021-04-09T18:35:15.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc021ffb700 1 -- 172.21.15.32:0/1333049572 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01800bd50 con 0x7fc02408cc40 2021-04-09T18:35:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc02b484700 1 -- 172.21.15.32:0/1333049572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0241a6490 con 0x7fc02408cc40 2021-04-09T18:35:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc021ffb700 1 -- 172.21.15.32:0/1333049572 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0180052e0 con 0x7fc02408cc40 2021-04-09T18:35:15.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.916 7fc02b484700 1 -- 172.21.15.32:0/1333049572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0241a6490 con 0x7fc02408cc40 2021-04-09T18:35:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.917 7fc021ffb700 1 -- 172.21.15.32:0/1333049572 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc018006030 con 0x7fc02408cc40 2021-04-09T18:35:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.917 7fc021ffb700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc004018930 0x7fc00401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.917 7fc021ffb700 1 -- 172.21.15.32:0/1333049572 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 4757+0+0 (secure 0 0 0) 0x7fc0180326b0 con 0x7fc02408cc40 2021-04-09T18:35:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.917 7fc0292a0700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc004018930 0x7fc00401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:15.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.917 7fc0292a0700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc004018930 0x7fc00401ae10 crc :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:15.954 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.937 7fc02b484700 1 -- 172.21.15.32:0/1333049572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0241a4770 con 0x7fc02408cc40 2021-04-09T18:35:15.958 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:15.940 7fc021ffb700 1 -- 172.21.15.32:0/1333049572 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc0180009c0 con 0x7fc02408cc40 2021-04-09T18:35:16.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.121 7fc02b484700 1 -- 172.21.15.32:0/1333049572 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc024112070 con 0x7fc02408cc40 2021-04-09T18:35:16.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.122 7fc021ffb700 1 -- 172.21.15.32:0/1333049572 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v270) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc01800bd50 con 0x7fc02408cc40 2021-04-09T18:35:16.140 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:35:16.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 -- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc004018930 msgr2=0x7fc00401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc004018930 0x7fc00401ae10 crc :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 -- 172.21.15.32:0/1333049572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc02408cc40 msgr2=0x7fc02408d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc02408cc40 0x7fc02408d040 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7fc018009650 tx=0x7fc01800b960).stop 2021-04-09T18:35:16.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 -- 172.21.15.32:0/1333049572 shutdown_connections 2021-04-09T18:35:16.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc004018930 0x7fc00401ae10 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc02408cc40 0x7fc02408d040 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0241a5680 0x7fc0241a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 --2- 172.21.15.32:0/1333049572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0241c2850 0x7fc0240bc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 -- 172.21.15.32:0/1333049572 >> 172.21.15.32:0/1333049572 conn(0x7fc0240bca90 msgr2=0x7fc0240b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:16.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 -- 172.21.15.32:0/1333049572 shutdown_connections 2021-04-09T18:35:16.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.124 7fc02b484700 1 -- 172.21.15.32:0/1333049572 wait complete. 2021-04-09T18:35:16.152 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2021-04-09T18:35:16.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.285 7f9811e5d700 1 Processor -- start 2021-04-09T18:35:16.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.285 7f9811e5d700 1 -- start start 2021-04-09T18:35:16.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f9811e5d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0d7ca0 0x7f980c0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f9811e5d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f980c10da00 0x7f980c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f9811e5d700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c10d480 con 0x7f980c10d010 2021-04-09T18:35:16.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f9811e5d700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c0d9220 con 0x7f980c10da00 2021-04-09T18:35:16.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f9811e5d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c0d86c0 con 0x7f980c0d7ca0 2021-04-09T18:35:16.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980affd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0d7ca0 0x7f980c0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980bfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f980c10da00 0x7f980c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980bfff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f980c10da00 0x7f980c10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.32:52080/0 (socket says 172.21.15.32:52080) 2021-04-09T18:35:16.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980affd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0d7ca0 0x7f980c0d80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:39318/0 (socket says 172.21.15.32:39318) 2021-04-09T18:35:16.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980bfff700 1 -- 172.21.15.32:0/656110772 learned_addr learned my addr 172.21.15.32:0/656110772 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:16.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980bfff700 1 -- 172.21.15.32:0/656110772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0d7ca0 msgr2=0x7f980c0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:16.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980bfff700 1 --2- 172.21.15.32:0/656110772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0d7ca0 0x7f980c0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980bfff700 1 -- 172.21.15.32:0/656110772 >> v1:172.21.15.86:6789/0 conn(0x7f980c10d010 legacy=0x7f980c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980bfff700 1 -- 172.21.15.32:0/656110772 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c112100 con 0x7f980c10da00 2021-04-09T18:35:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980a7fc700 1 -- 172.21.15.32:0/656110772 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643190606 0 0) 0x7f97fc000c60 con 0x7f980c10d010 2021-04-09T18:35:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.286 7f980bfff700 1 --2- 172.21.15.32:0/656110772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c10da00 0x7f980c10de00 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f9804006aa0 tx=0x7f9804009ea0).ready entity=mon.1 client_cookie=4579dbb4f0a86f2f server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:16.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.287 7f980a7fc700 1 -- 172.21.15.32:0/656110772 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980400b510 con 0x7f980c10da00 2021-04-09T18:35:16.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.287 7f980a7fc700 1 -- 172.21.15.32:0/656110772 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9804003030 con 0x7f980c10da00 2021-04-09T18:35:16.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.287 7f980a7fc700 1 -- 172.21.15.32:0/656110772 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980400ba10 con 0x7f980c10da00 2021-04-09T18:35:16.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.287 7f9811e5d700 1 -- 172.21.15.32:0/656110772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c10da00 msgr2=0x7f980c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.287 7f9811e5d700 1 --2- 172.21.15.32:0/656110772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c10da00 0x7f980c10de00 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f9804006aa0 tx=0x7f9804009ea0).stop 2021-04-09T18:35:16.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.288 7f9811e5d700 1 -- 172.21.15.32:0/656110772 shutdown_connections 2021-04-09T18:35:16.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.288 7f9811e5d700 1 --2- 172.21.15.32:0/656110772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c10da00 0x7f980c10de00 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.288 7f9811e5d700 1 --2- 172.21.15.32:0/656110772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0d7ca0 0x7f980c0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.288 7f9811e5d700 1 -- 172.21.15.32:0/656110772 >> 172.21.15.32:0/656110772 conn(0x7f980c0ce5f0 msgr2=0x7f980c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:16.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.288 7f9811e5d700 1 -- 172.21.15.32:0/656110772 shutdown_connections 2021-04-09T18:35:16.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.288 7f9811e5d700 1 -- 172.21.15.32:0/656110772 wait complete. 2021-04-09T18:35:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.308 7f9811e5d700 1 Processor -- start 2021-04-09T18:35:16.335 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.308 7f9811e5d700 1 -- start start 2021-04-09T18:35:16.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f9811e5d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1ac380 0x7f980c18a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f9811e5d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c1ac780 0x7f980c18a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f9811e5d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c183520 0x7f980c183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f9811e5d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c1ace10 con 0x7f980c1ac780 2021-04-09T18:35:16.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f9811e5d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c189cf0 con 0x7f980c1ac380 2021-04-09T18:35:16.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f9811e5d700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c183f40 con 0x7f980c183520 2021-04-09T18:35:16.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c1ac780 0x7f980c18a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.337 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c1ac780 0x7f980c18a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60328/0 (socket says 172.21.15.32:60328) 2021-04-09T18:35:16.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980bfff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c183520 0x7f980c183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980affd700 1 -- 172.21.15.32:0/3161620114 learned_addr learned my addr 172.21.15.32:0/3161620114 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:16.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980b7fe700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1ac380 0x7f980c18a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.338 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980affd700 1 -- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c183520 msgr2=0x7f980c183920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980affd700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c183520 0x7f980c183920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980affd700 1 -- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1ac380 msgr2=0x7f980c18a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980affd700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1ac380 0x7f980c18a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.339 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980affd700 1 -- 172.21.15.32:0/3161620114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c184160 con 0x7f980c1ac780 2021-04-09T18:35:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980bfff700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c183520 0x7f980c183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980b7fe700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1ac380 0x7f980c18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f980affd700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c1ac780 0x7f980c18a850 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f9800006f60 tx=0x7f980000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:16.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f97f3fff700 1 -- 172.21.15.32:0/3161620114 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980000bd60 con 0x7f980c1ac780 2021-04-09T18:35:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f97f3fff700 1 -- 172.21.15.32:0/3161620114 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9800000c90 con 0x7f980c1ac780 2021-04-09T18:35:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c184160 con 0x7f980c1ac780 2021-04-09T18:35:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f97f3fff700 1 -- 172.21.15.32:0/3161620114 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9800002220 con 0x7f980c1ac780 2021-04-09T18:35:16.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.309 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c184160 con 0x7f980c1ac780 2021-04-09T18:35:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.310 7f97f3fff700 1 -- 172.21.15.32:0/3161620114 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f98000052e0 con 0x7f980c1ac780 2021-04-09T18:35:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.310 7f97f3fff700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97ec0188d0 0x7f97ec01add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.310 7f97f3fff700 1 -- 172.21.15.32:0/3161620114 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f98000326c0 con 0x7f980c1ac780 2021-04-09T18:35:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.310 7f980b7fe700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97ec0188d0 0x7f97ec01add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.311 7f980b7fe700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97ec0188d0 0x7f97ec01add0 crc :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:16.347 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.330 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c189cf0 con 0x7f980c1ac780 2021-04-09T18:35:16.350 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.333 7f97f3fff700 1 -- 172.21.15.32:0/3161620114 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9800000c90 con 0x7f980c1ac780 2021-04-09T18:35:16.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.513 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f980c189cf0 con 0x7f980c1ac780 2021-04-09T18:35:16.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.513 7f97f3fff700 1 -- 172.21.15.32:0/3161620114 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v270) v1 ==== 84+0+64 (secure 0 0 0) 0x7f9800006b00 con 0x7f980c1ac780 2021-04-09T18:35:16.531 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:16.532 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2021-04-09T18:35:16.532 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:35:16.532 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2021-04-09T18:35:16.532 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": "rbd" 2021-04-09T18:35:16.533 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:16.533 INFO:teuthology.orchestra.run.smithi032.stdout:] 2021-04-09T18:35:16.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97ec0188d0 msgr2=0x7f97ec01add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97ec0188d0 0x7f97ec01add0 crc :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c1ac780 msgr2=0x7f980c18a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c1ac780 0x7f980c18a850 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f9800006f60 tx=0x7f980000b970).stop 2021-04-09T18:35:16.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 shutdown_connections 2021-04-09T18:35:16.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f97ec0188d0 0x7f97ec01add0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f980c1ac780 0x7f980c18a850 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c183520 0x7f980c183920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 --2- 172.21.15.32:0/3161620114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c1ac380 0x7f980c18a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 >> 172.21.15.32:0/3161620114 conn(0x7f980c19b890 msgr2=0x7f980c1a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:16.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 shutdown_connections 2021-04-09T18:35:16.537 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.516 7f9811e5d700 1 -- 172.21.15.32:0/3161620114 wait complete. 2021-04-09T18:35:16.544 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-04-09T18:35:16.545 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-04-09T18:35:16.722 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.680 7f16ff3c2700 1 Processor -- start 2021-04-09T18:35:16.846 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.680 7f16ff3c2700 1 -- start start 2021-04-09T18:35:16.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.681 7f16ff3c2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f810da00 0x7f16f810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.847 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.681 7f16ff3c2700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f810d480 con 0x7f16f810d010 2021-04-09T18:35:16.854 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.681 7f16ff3c2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80d1f70 con 0x7f16f80d7ca0 2021-04-09T18:35:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.681 7f16ff3c2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80d9220 con 0x7f16f810da00 2021-04-09T18:35:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.681 7f16fd1de700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f16f810d010 0x7f16f80d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41692/0 (socket says 172.21.15.32:41692) 2021-04-09T18:35:16.855 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.681 7f16fd1de700 1 -- 172.21.15.32:0/4123958062 learned_addr learned my addr 172.21.15.32:0/4123958062 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.681 7f16fd9df700 1 --2- 172.21.15.32:0/4123958062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f810da00 0x7f16f810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16effff700 1 -- 172.21.15.32:0/4123958062 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296679150 0 0) 0x7f16f4000ce0 con 0x7f16f810d010 2021-04-09T18:35:16.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16effff700 1 -- 172.21.15.32:0/4123958062 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e4002fe0 con 0x7f16f810d010 2021-04-09T18:35:16.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16effff700 1 -- 172.21.15.32:0/4123958062 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110709719 0 0) 0x7f16e8001080 con 0x7f16f80d7ca0 2021-04-09T18:35:16.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16effff700 1 -- 172.21.15.32:0/4123958062 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e4003210 con 0x7f16f80d7ca0 2021-04-09T18:35:16.858 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16effff700 1 -- 172.21.15.32:0/4123958062 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3200591994 0 0) 0x7f16f40010e0 con 0x7f16f810d010 2021-04-09T18:35:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16fd9df700 1 -- 172.21.15.32:0/4123958062 >> v1:172.21.15.86:6789/0 conn(0x7f16f80d7ca0 legacy=0x7f16f80d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16fd9df700 1 -- 172.21.15.32:0/4123958062 >> v1:172.21.15.32:6789/0 conn(0x7f16f810d010 legacy=0x7f16f80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16fd9df700 1 -- 172.21.15.32:0/4123958062 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f810e420 con 0x7f16f810da00 2021-04-09T18:35:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16fd9df700 1 --2- 172.21.15.32:0/4123958062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f810da00 0x7f16f810de00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f16f0006e50 tx=0x7f16f0009df0).ready entity=mon.2 client_cookie=f6794bc550d9c68c server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:16.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16effff700 1 -- 172.21.15.32:0/4123958062 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16f000b510 con 0x7f16f810da00 2021-04-09T18:35:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16effff700 1 -- 172.21.15.32:0/4123958062 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f0003030 con 0x7f16f810da00 2021-04-09T18:35:16.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16effff700 1 -- 172.21.15.32:0/4123958062 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16f000b7d0 con 0x7f16f810da00 2021-04-09T18:35:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16ff3c2700 1 -- 172.21.15.32:0/4123958062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f810da00 msgr2=0x7f16f810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.682 7f16ff3c2700 1 --2- 172.21.15.32:0/4123958062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f810da00 0x7f16f810de00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f16f0006e50 tx=0x7f16f0009df0).stop 2021-04-09T18:35:16.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.683 7f16ff3c2700 1 -- 172.21.15.32:0/4123958062 shutdown_connections 2021-04-09T18:35:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.683 7f16ff3c2700 1 --2- 172.21.15.32:0/4123958062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f810da00 0x7f16f810de00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.863 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.683 7f16ff3c2700 1 -- 172.21.15.32:0/4123958062 >> 172.21.15.32:0/4123958062 conn(0x7f16f80ce5f0 msgr2=0x7f16f80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:16.864 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.683 7f16ff3c2700 1 -- 172.21.15.32:0/4123958062 shutdown_connections 2021-04-09T18:35:16.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.683 7f16ff3c2700 1 -- 172.21.15.32:0/4123958062 wait complete. 2021-04-09T18:35:16.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.703 7f16ff3c2700 1 Processor -- start 2021-04-09T18:35:16.865 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.703 7f16ff3c2700 1 -- start start 2021-04-09T18:35:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16ff3c2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80c3c00 0x7f16f809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16ff3c2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f8095260 0x7f16f81a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.866 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16ff3c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f81a1270 0x7f16f80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16ff3c2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f809cda0 con 0x7f16f80c3c00 2021-04-09T18:35:16.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16ff3c2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80c3680 con 0x7f16f81a1270 2021-04-09T18:35:16.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16ff3c2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8095660 con 0x7f16f8095260 2021-04-09T18:35:16.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16fd1de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80c3c00 0x7f16f809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16fd1de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80c3c00 0x7f16f809c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60342/0 (socket says 172.21.15.32:60342) 2021-04-09T18:35:16.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16fd1de700 1 -- 172.21.15.32:0/1862034883 learned_addr learned my addr 172.21.15.32:0/1862034883 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:16.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16fc9dd700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f8095260 0x7f16f81a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.704 7f16fd9df700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f81a1270 0x7f16f80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16fd1de700 1 -- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f8095260 msgr2=0x7f16f81a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16fd1de700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f8095260 0x7f16f81a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16fd1de700 1 -- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f81a1270 msgr2=0x7f16f80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:16.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16fd1de700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f81a1270 0x7f16f80b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:16.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16fd1de700 1 -- 172.21.15.32:0/1862034883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f81a1670 con 0x7f16f80c3c00 2021-04-09T18:35:16.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16fd1de700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80c3c00 0x7f16f809c600 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 rx=0x7f16f40010b0 tx=0x7f16f400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:16.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16edffb700 1 -- 172.21.15.32:0/1862034883 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16f40070d0 con 0x7f16f80c3c00 2021-04-09T18:35:16.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16edffb700 1 -- 172.21.15.32:0/1862034883 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f400ace0 con 0x7f16f80c3c00 2021-04-09T18:35:16.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f81a1670 con 0x7f16f80c3c00 2021-04-09T18:35:16.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16edffb700 1 -- 172.21.15.32:0/1862034883 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16f4004830 con 0x7f16f80c3c00 2021-04-09T18:35:16.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.705 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f81a1670 con 0x7f16f80c3c00 2021-04-09T18:35:16.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.706 7f16edffb700 1 -- 172.21.15.32:0/1862034883 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f16f4001af0 con 0x7f16f80c3c00 2021-04-09T18:35:16.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.706 7f16edffb700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d80188d0 0x7f16d801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:16.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.706 7f16edffb700 1 -- 172.21.15.32:0/1862034883 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 4757+0+0 (secure 0 0 0) 0x7f16f40326b0 con 0x7f16f80c3c00 2021-04-09T18:35:16.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.706 7f16fc9dd700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d80188d0 0x7f16d801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:16.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.706 7f16fc9dd700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d80188d0 0x7f16d801add0 crc :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:16.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.725 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f810d480 con 0x7f16f80c3c00 2021-04-09T18:35:16.876 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.729 7f16edffb700 1 -- 172.21.15.32:0/1862034883 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f16f4004ef0 con 0x7f16f80c3c00 2021-04-09T18:35:16.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:16.913 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f16f810d480 con 0x7f16f80c3c00 2021-04-09T18:35:17.920 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.902 7f16edffb700 1 -- 172.21.15.32:0/1862034883 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v271) v1 ==== 133+0+0 (secure 0 0 0) 0x7f16f4001af0 con 0x7f16f80c3c00 2021-04-09T18:35:17.946 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2021-04-09T18:35:17.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.930 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d80188d0 msgr2=0x7f16d801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:17.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.930 7f16ff3c2700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d80188d0 0x7f16d801add0 crc :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:17.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.930 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80c3c00 msgr2=0x7f16f809c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:17.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.930 7f16ff3c2700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80c3c00 0x7f16f809c600 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 rx=0x7f16f40010b0 tx=0x7f16f400a8f0).stop 2021-04-09T18:35:17.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.931 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 shutdown_connections 2021-04-09T18:35:17.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.931 7f16ff3c2700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f16d80188d0 0x7f16d801add0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:17.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.931 7f16ff3c2700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16f81a1270 0x7f16f80b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:17.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.931 7f16ff3c2700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f16f8095260 0x7f16f81a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:17.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.931 7f16ff3c2700 1 --2- 172.21.15.32:0/1862034883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16f80c3c00 0x7f16f809c600 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:17.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.931 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 >> 172.21.15.32:0/1862034883 conn(0x7f16f80975f0 msgr2=0x7f16f8096b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:17.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.931 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 shutdown_connections 2021-04-09T18:35:17.952 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:17.931 7f16ff3c2700 1 -- 172.21.15.32:0/1862034883 wait complete. 2021-04-09T18:35:17.960 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2021-04-09T18:35:18.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.096 7f8f78a73700 1 Processor -- start 2021-04-09T18:35:18.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.096 7f8f78a73700 1 -- start start 2021-04-09T18:35:18.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.097 7f8f78a73700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f7410d010 0x7f8f740d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:18.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.097 7f8f78a73700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f740d1da0 0x7f8f740d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.097 7f8f78a73700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8f740d82c0 0x7f8f7410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.097 7f8f78a73700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7410e780 con 0x7f8f740d1da0 2021-04-09T18:35:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.097 7f8f78a73700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f740d8d30 con 0x7f8f740d82c0 2021-04-09T18:35:18.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.097 7f8f78a73700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740d9220 con 0x7f8f7410d010 2021-04-09T18:35:18.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.097 7f8f7205d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f740d1da0 0x7f8f740d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:18.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.097 7f8f7205d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f8f740d1da0 0x7f8f740d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60352/0 (socket says 172.21.15.32:60352) 2021-04-09T18:35:18.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.097 7f8f7205d700 1 -- 172.21.15.32:0/2241120354 learned_addr learned my addr 172.21.15.32:0/2241120354 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:18.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7305f700 1 --2- 172.21.15.32:0/2241120354 >> v2:172.21.15.86:3300/0 conn(0x7f8f740d82c0 0x7f8f7410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:18.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7285e700 1 --2- 172.21.15.32:0/2241120354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f7410d010 0x7f8f740d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:18.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7205d700 1 -- 172.21.15.32:0/2241120354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f7410d010 msgr2=0x7f8f740d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:18.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7205d700 1 --2- 172.21.15.32:0/2241120354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f7410d010 0x7f8f740d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:18.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7205d700 1 -- 172.21.15.32:0/2241120354 >> v2:172.21.15.86:3300/0 conn(0x7f8f740d82c0 msgr2=0x7f8f7410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:18.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7205d700 1 --2- 172.21.15.32:0/2241120354 >> v2:172.21.15.86:3300/0 conn(0x7f8f740d82c0 0x7f8f7410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:18.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7205d700 1 -- 172.21.15.32:0/2241120354 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7410d480 con 0x7f8f740d1da0 2021-04-09T18:35:18.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7285e700 1 --2- 172.21.15.32:0/2241120354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f7410d010 0x7f8f740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:18.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7205d700 1 --2- 172.21.15.32:0/2241120354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740d1da0 0x7f8f740d7ca0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7f8f68006700 tx=0x7f8f68009b80).ready entity=mon.0 client_cookie=175702d8459bf5ad server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:18.142 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7185c700 1 -- 172.21.15.32:0/2241120354 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f6800c0c0 con 0x7f8f740d1da0 2021-04-09T18:35:18.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7185c700 1 -- 172.21.15.32:0/2241120354 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6800c440 con 0x7f8f740d1da0 2021-04-09T18:35:18.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f7185c700 1 -- 172.21.15.32:0/2241120354 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f6800c960 con 0x7f8f740d1da0 2021-04-09T18:35:18.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f78a73700 1 -- 172.21.15.32:0/2241120354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740d1da0 msgr2=0x7f8f740d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:18.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.098 7f8f78a73700 1 --2- 172.21.15.32:0/2241120354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740d1da0 0x7f8f740d7ca0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7f8f68006700 tx=0x7f8f68009b80).stop 2021-04-09T18:35:18.143 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.099 7f8f78a73700 1 -- 172.21.15.32:0/2241120354 shutdown_connections 2021-04-09T18:35:18.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.099 7f8f78a73700 1 --2- 172.21.15.32:0/2241120354 >> v2:172.21.15.86:3300/0 conn(0x7f8f740d82c0 0x7f8f7410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:18.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.099 7f8f78a73700 1 --2- 172.21.15.32:0/2241120354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740d1da0 0x7f8f740d7ca0 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:18.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.099 7f8f78a73700 1 --2- 172.21.15.32:0/2241120354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f7410d010 0x7f8f740d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:18.144 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.099 7f8f78a73700 1 -- 172.21.15.32:0/2241120354 >> 172.21.15.32:0/2241120354 conn(0x7f8f740ce5f0 msgr2=0x7f8f740d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:18.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.099 7f8f78a73700 1 -- 172.21.15.32:0/2241120354 shutdown_connections 2021-04-09T18:35:18.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.099 7f8f78a73700 1 -- 172.21.15.32:0/2241120354 wait complete. 2021-04-09T18:35:18.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.119 7f8f78a73700 1 Processor -- start 2021-04-09T18:35:18.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.119 7f8f78a73700 1 -- start start 2021-04-09T18:35:18.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f78a73700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f7408e350 0x7f8f740bd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:18.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f78a73700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f74193ef0 0x7f8f740b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:18.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f78a73700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740b9180 0x7f8f7408ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:18.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f78a73700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7408e810 con 0x7f8f740b9180 2021-04-09T18:35:18.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f78a73700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7408f330 con 0x7f8f7408e350 2021-04-09T18:35:18.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f78a73700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7408f550 con 0x7f8f74193ef0 2021-04-09T18:35:18.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740b9180 0x7f8f7408ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:18.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7305f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740b9180 0x7f8f7408ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60358/0 (socket says 172.21.15.32:60358) 2021-04-09T18:35:18.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7305f700 1 -- 172.21.15.32:0/1922429425 learned_addr learned my addr 172.21.15.32:0/1922429425 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:18.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7285e700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f7408e350 0x7f8f740bd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:18.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7205d700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f74193ef0 0x7f8f740b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:18.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7305f700 1 -- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f74193ef0 msgr2=0x7f8f740b8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:18.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7305f700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f74193ef0 0x7f8f740b8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:18.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7305f700 1 -- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f7408e350 msgr2=0x7f8f740bd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:18.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7305f700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f7408e350 0x7f8f740bd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:18.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7305f700 1 -- 172.21.15.32:0/1922429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7408f770 con 0x7f8f740b9180 2021-04-09T18:35:18.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7285e700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f7408e350 0x7f8f740bd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:18.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.120 7f8f7305f700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740b9180 0x7f8f7408ed10 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 rx=0x7f8f6c00a940 tx=0x7f8f6c00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:18.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.121 7f8f637fe700 1 -- 172.21.15.32:0/1922429425 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f6c001400 con 0x7f8f740b9180 2021-04-09T18:35:18.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.121 7f8f637fe700 1 -- 172.21.15.32:0/1922429425 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c00ae20 con 0x7f8f740b9180 2021-04-09T18:35:18.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.121 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7408f770 con 0x7f8f740b9180 2021-04-09T18:35:18.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.121 7f8f637fe700 1 -- 172.21.15.32:0/1922429425 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f6c0018b0 con 0x7f8f740b9180 2021-04-09T18:35:18.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.121 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7408f770 con 0x7f8f740b9180 2021-04-09T18:35:18.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.121 7f8f637fe700 1 -- 172.21.15.32:0/1922429425 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f8f6c001680 con 0x7f8f740b9180 2021-04-09T18:35:18.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.122 7f8f637fe700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f580188c0 0x7f8f5801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.122 7f8f637fe700 1 -- 172.21.15.32:0/1922429425 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5111+0+0 (secure 0 0 0) 0x7f8f6c032960 con 0x7f8f740b9180 2021-04-09T18:35:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.122 7f8f7285e700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f580188c0 0x7f8f5801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:18.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.122 7f8f7285e700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f580188c0 0x7f8f5801adc0 crc :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:18.160 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.141 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7410d480 con 0x7f8f740b9180 2021-04-09T18:35:18.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.145 7f8f637fe700 1 -- 172.21.15.32:0/1922429425 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f8f6c0011f0 con 0x7f8f740b9180 2021-04-09T18:35:18.351 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:18.333 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f8f7410d480 con 0x7f8f740b9180 2021-04-09T18:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.236 7f8f637fe700 1 -- 172.21.15.32:0/1922429425 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v276) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8f6c001680 con 0x7f8f740b9180 2021-04-09T18:35:19.267 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2021-04-09T18:35:19.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f580188c0 msgr2=0x7f8f5801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.271 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f580188c0 0x7f8f5801adc0 crc :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740b9180 msgr2=0x7f8f7408ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.272 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740b9180 0x7f8f7408ed10 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 rx=0x7f8f6c00a940 tx=0x7f8f6c00ace0).stop 2021-04-09T18:35:19.273 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 shutdown_connections 2021-04-09T18:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f8f580188c0 0x7f8f5801adc0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f740b9180 0x7f8f7408ed10 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f74193ef0 0x7f8f740b8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 --2- 172.21.15.32:0/1922429425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f7408e350 0x7f8f740bd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 >> 172.21.15.32:0/1922429425 conn(0x7f8f740950b0 msgr2=0x7f8f740b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 shutdown_connections 2021-04-09T18:35:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.251 7f8f78a73700 1 -- 172.21.15.32:0/1922429425 wait complete. 2021-04-09T18:35:19.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-04-09T18:35:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.423 7f730548b700 1 Processor -- start 2021-04-09T18:35:19.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.423 7f730548b700 1 -- start start 2021-04-09T18:35:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f730548b700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f730010d050 0x7f73000cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f730548b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73000d0160 0x7f73000d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f730548b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730010d500 con 0x7f73000d82e0 2021-04-09T18:35:19.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f730548b700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73000d9260 con 0x7f730010d050 2021-04-09T18:35:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f730548b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73000d86e0 con 0x7f73000d0160 2021-04-09T18:35:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f72ff07d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f730010d050 0x7f73000cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f72ff87e700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f73000d82e0 0x7f730010da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41726/0 (socket says 172.21.15.32:41726) 2021-04-09T18:35:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f72ff87e700 1 -- 172.21.15.32:0/4190040873 learned_addr learned my addr 172.21.15.32:0/4190040873 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:19.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f72fe87c700 1 --2- 172.21.15.32:0/4190040873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73000d0160 0x7f73000d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f72ff07d700 1 -- 172.21.15.32:0/4190040873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73000d0160 msgr2=0x7f73000d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:19.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f72ff07d700 1 --2- 172.21.15.32:0/4190040873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73000d0160 0x7f73000d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f72ff07d700 1 -- 172.21.15.32:0/4190040873 >> v1:172.21.15.32:6789/0 conn(0x7f73000d82e0 legacy=0x7f730010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.424 7f72fe07b700 1 -- 172.21.15.32:0/4190040873 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179675301 0 0) 0x7f72f8000d30 con 0x7f73000d82e0 2021-04-09T18:35:19.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.425 7f72ff07d700 1 -- 172.21.15.32:0/4190040873 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730010e270 con 0x7f730010d050 2021-04-09T18:35:19.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.425 7f72ff07d700 1 --2- 172.21.15.32:0/4190040873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730010d050 0x7f73000cfb40 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f72f0006650 tx=0x7f72f0009a00).ready entity=mon.0 client_cookie=756e6f99750b9cd server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:19.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.425 7f72fe07b700 1 -- 172.21.15.32:0/4190040873 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72f0009df0 con 0x7f730010d050 2021-04-09T18:35:19.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.425 7f72fe07b700 1 -- 172.21.15.32:0/4190040873 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f00009b0 con 0x7f730010d050 2021-04-09T18:35:19.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.425 7f72fe07b700 1 -- 172.21.15.32:0/4190040873 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72f000c2b0 con 0x7f730010d050 2021-04-09T18:35:19.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.425 7f730548b700 1 -- 172.21.15.32:0/4190040873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730010d050 msgr2=0x7f73000cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.425 7f730548b700 1 --2- 172.21.15.32:0/4190040873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730010d050 0x7f73000cfb40 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f72f0006650 tx=0x7f72f0009a00).stop 2021-04-09T18:35:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.426 7f730548b700 1 -- 172.21.15.32:0/4190040873 shutdown_connections 2021-04-09T18:35:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.426 7f730548b700 1 --2- 172.21.15.32:0/4190040873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73000d0160 0x7f73000d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.426 7f730548b700 1 --2- 172.21.15.32:0/4190040873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f730010d050 0x7f73000cfb40 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.426 7f730548b700 1 -- 172.21.15.32:0/4190040873 >> 172.21.15.32:0/4190040873 conn(0x7f73000ce620 msgr2=0x7f73000d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.426 7f730548b700 1 -- 172.21.15.32:0/4190040873 shutdown_connections 2021-04-09T18:35:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.426 7f730548b700 1 -- 172.21.15.32:0/4190040873 wait complete. 2021-04-09T18:35:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.446 7f730548b700 1 Processor -- start 2021-04-09T18:35:19.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.446 7f730548b700 1 -- start start 2021-04-09T18:35:19.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.446 7f730548b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730009a890 0x7f73000a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.446 7f730548b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730009ac90 0x7f73000a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.446 7f730548b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73000c2220 0x7f73000c2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.446 7f730548b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73000b86e0 con 0x7f73000c2220 2021-04-09T18:35:19.475 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.446 7f730548b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73000a2920 con 0x7f730009ac90 2021-04-09T18:35:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.446 7f730548b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73000c2c40 con 0x7f730009a890 2021-04-09T18:35:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73000c2220 0x7f73000c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff87e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73000c2220 0x7f73000c2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60376/0 (socket says 172.21.15.32:60376) 2021-04-09T18:35:19.476 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff87e700 1 -- 172.21.15.32:0/3311129355 learned_addr learned my addr 172.21.15.32:0/3311129355 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:19.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72fe87c700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730009ac90 0x7f73000a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff07d700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730009a890 0x7f73000a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff87e700 1 -- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730009a890 msgr2=0x7f73000a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff87e700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730009a890 0x7f73000a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff87e700 1 -- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730009ac90 msgr2=0x7f73000a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff87e700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730009ac90 0x7f73000a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff87e700 1 -- 172.21.15.32:0/3311129355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73000c2e20 con 0x7f73000c2220 2021-04-09T18:35:19.478 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72ff87e700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73000c2220 0x7f73000c2620 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7f72f8007f90 tx=0x7f72f800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:19.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72e7fff700 1 -- 172.21.15.32:0/3311129355 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72f8000cc0 con 0x7f73000c2220 2021-04-09T18:35:19.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72e7fff700 1 -- 172.21.15.32:0/3311129355 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f8007020 con 0x7f73000c2220 2021-04-09T18:35:19.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f730548b700 1 -- 172.21.15.32:0/3311129355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73000c30c0 con 0x7f73000c2220 2021-04-09T18:35:19.479 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f72e7fff700 1 -- 172.21.15.32:0/3311129355 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72f8007500 con 0x7f73000c2220 2021-04-09T18:35:19.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.447 7f730548b700 1 -- 172.21.15.32:0/3311129355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73000c2e20 con 0x7f73000c2220 2021-04-09T18:35:19.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.448 7f72e7fff700 1 -- 172.21.15.32:0/3311129355 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f72f8007200 con 0x7f73000c2220 2021-04-09T18:35:19.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.448 7f72e7fff700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72e00188d0 0x7f72e001add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.448 7f72e7fff700 1 -- 172.21.15.32:0/3311129355 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5461+0+0 (secure 0 0 0) 0x7f72f8032cd0 con 0x7f73000c2220 2021-04-09T18:35:19.480 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.448 7f72ff07d700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72e00188d0 0x7f72e001add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.481 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.448 7f72ff07d700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72e00188d0 0x7f72e001add0 crc :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:19.487 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.468 7f730548b700 1 -- 172.21.15.32:0/3311129355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73001b7180 con 0x7f73000c2220 2021-04-09T18:35:19.490 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.472 7f72e7fff700 1 -- 172.21.15.32:0/3311129355 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f72f8007d00 con 0x7f73000c2220 2021-04-09T18:35:19.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.655 7f730548b700 1 -- 172.21.15.32:0/3311129355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f730010d500 con 0x7f73000c2220 2021-04-09T18:35:19.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.659 7f72e7fff700 1 -- 172.21.15.32:0/3311129355 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 46 and data pool 47 v440) v1 ==== 189+0+0 (secure 0 0 0) 0x7f72f8007020 con 0x7f73000c2220 2021-04-09T18:35:19.684 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 46 and data pool 47 2021-04-09T18:35:19.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 -- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72e00188d0 msgr2=0x7f72e001add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72e00188d0 0x7f72e001add0 crc :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 -- 172.21.15.32:0/3311129355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73000c2220 msgr2=0x7f73000c2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.685 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73000c2220 0x7f73000c2620 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7f72f8007f90 tx=0x7f72f800ac50).stop 2021-04-09T18:35:19.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 -- 172.21.15.32:0/3311129355 shutdown_connections 2021-04-09T18:35:19.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f72e00188d0 0x7f72e001add0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f73000c2220 0x7f73000c2620 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730009ac90 0x7f73000a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.686 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 --2- 172.21.15.32:0/3311129355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730009a890 0x7f73000a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 -- 172.21.15.32:0/3311129355 >> 172.21.15.32:0/3311129355 conn(0x7f73000b6b70 msgr2=0x7f73000b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:19.687 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 -- 172.21.15.32:0/3311129355 shutdown_connections 2021-04-09T18:35:19.688 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.665 7f730548b700 1 -- 172.21.15.32:0/3311129355 wait complete. 2021-04-09T18:35:19.698 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:35:19.867 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.827 7f9e0abc7700 1 Processor -- start 2021-04-09T18:35:19.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.827 7f9e0abc7700 1 -- start start 2021-04-09T18:35:19.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e0abc7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0410da00 0x7f9e0410de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.868 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e0abc7700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0410d480 con 0x7f9e040d7ca0 2021-04-09T18:35:19.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e0abc7700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e040d1f70 con 0x7f9e0410d010 2021-04-09T18:35:19.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e0abc7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040d9220 con 0x7f9e0410da00 2021-04-09T18:35:19.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e03fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9e040d7ca0 0x7f9e040d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41740/0 (socket says 172.21.15.32:41740) 2021-04-09T18:35:19.869 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e03fff700 1 -- 172.21.15.32:0/759562954 learned_addr learned my addr 172.21.15.32:0/759562954 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e091e4700 1 --2- 172.21.15.32:0/759562954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0410da00 0x7f9e0410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e037fe700 1 -- 172.21.15.32:0/759562954 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497814283 0 0) 0x7f9df8000ce0 con 0x7f9e040d7ca0 2021-04-09T18:35:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e037fe700 1 -- 172.21.15.32:0/759562954 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec002fe0 con 0x7f9e040d7ca0 2021-04-09T18:35:19.870 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e037fe700 1 -- 172.21.15.32:0/759562954 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188155862 0 0) 0x7f9df4000c60 con 0x7f9e0410d010 2021-04-09T18:35:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e037fe700 1 -- 172.21.15.32:0/759562954 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec0042c0 con 0x7f9e0410d010 2021-04-09T18:35:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e091e4700 1 -- 172.21.15.32:0/759562954 >> v1:172.21.15.86:6789/0 conn(0x7f9e0410d010 legacy=0x7f9e040d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e091e4700 1 -- 172.21.15.32:0/759562954 >> v1:172.21.15.32:6789/0 conn(0x7f9e040d7ca0 legacy=0x7f9e040d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.871 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.828 7f9e091e4700 1 -- 172.21.15.32:0/759562954 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04112100 con 0x7f9e0410da00 2021-04-09T18:35:19.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e091e4700 1 --2- 172.21.15.32:0/759562954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0410da00 0x7f9e0410de00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f9dfc006e50 tx=0x7f9dfc009e70).ready entity=mon.2 client_cookie=c70658d2474c7f57 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:19.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e037fe700 1 -- 172.21.15.32:0/759562954 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dfc006a20 con 0x7f9e0410da00 2021-04-09T18:35:19.872 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e037fe700 1 -- 172.21.15.32:0/759562954 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc006790 con 0x7f9e0410da00 2021-04-09T18:35:19.873 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e037fe700 1 -- 172.21.15.32:0/759562954 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dfc00b700 con 0x7f9e0410da00 2021-04-09T18:35:19.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e0abc7700 1 -- 172.21.15.32:0/759562954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0410da00 msgr2=0x7f9e0410de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e0abc7700 1 --2- 172.21.15.32:0/759562954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0410da00 0x7f9e0410de00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f9dfc006e50 tx=0x7f9dfc009e70).stop 2021-04-09T18:35:19.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e0abc7700 1 -- 172.21.15.32:0/759562954 shutdown_connections 2021-04-09T18:35:19.874 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e0abc7700 1 --2- 172.21.15.32:0/759562954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0410da00 0x7f9e0410de00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e0abc7700 1 -- 172.21.15.32:0/759562954 >> 172.21.15.32:0/759562954 conn(0x7f9e040ce5f0 msgr2=0x7f9e040d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e0abc7700 1 -- 172.21.15.32:0/759562954 shutdown_connections 2021-04-09T18:35:19.875 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.829 7f9e0abc7700 1 -- 172.21.15.32:0/759562954 wait complete. 2021-04-09T18:35:19.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e0abc7700 1 Processor -- start 2021-04-09T18:35:19.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e0abc7700 1 -- start start 2021-04-09T18:35:19.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e0abc7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e04179610 0x7f9e04179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.879 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e0abc7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0417a6e0 0x7f9e0417aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e0abc7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0417b100 0x7f9e0408db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e0abc7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0417d720 con 0x7f9e0417a6e0 2021-04-09T18:35:19.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e0abc7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e04185ba0 con 0x7f9e04179610 2021-04-09T18:35:19.880 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e0abc7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e04185d50 con 0x7f9e0417b100 2021-04-09T18:35:19.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e03fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0417a6e0 0x7f9e0417aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e03fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0417a6e0 0x7f9e0417aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60384/0 (socket says 172.21.15.32:60384) 2021-04-09T18:35:19.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.850 7f9e03fff700 1 -- 172.21.15.32:0/586096960 learned_addr learned my addr 172.21.15.32:0/586096960 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:19.881 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e091e4700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0417b100 0x7f9e0408db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e089e3700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e04179610 0x7f9e04179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e03fff700 1 -- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0417b100 msgr2=0x7f9e0408db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e03fff700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0417b100 0x7f9e0408db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.882 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e03fff700 1 -- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e04179610 msgr2=0x7f9e04179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:19.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e03fff700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e04179610 0x7f9e04179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:19.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e03fff700 1 -- 172.21.15.32:0/586096960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0408e1b0 con 0x7f9e0417a6e0 2021-04-09T18:35:19.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e091e4700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0417b100 0x7f9e0408db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:35:19.883 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e089e3700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e04179610 0x7f9e04179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:35:19.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e03fff700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0417a6e0 0x7f9e0417aae0 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7f9df800b5c0 tx=0x7f9df800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:19.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e017fa700 1 -- 172.21.15.32:0/586096960 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df800bd50 con 0x7f9e0417a6e0 2021-04-09T18:35:19.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e017fa700 1 -- 172.21.15.32:0/586096960 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df80060d0 con 0x7f9e0417a6e0 2021-04-09T18:35:19.884 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0408e1b0 con 0x7f9e0417a6e0 2021-04-09T18:35:19.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e017fa700 1 -- 172.21.15.32:0/586096960 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df80043a0 con 0x7f9e0417a6e0 2021-04-09T18:35:19.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.851 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0408e1b0 con 0x7f9e0417a6e0 2021-04-09T18:35:19.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.852 7f9e017fa700 1 -- 172.21.15.32:0/586096960 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f9df80062b0 con 0x7f9e0417a6e0 2021-04-09T18:35:19.885 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.852 7f9e017fa700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:19.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.852 7f9e017fa700 1 -- 172.21.15.32:0/586096960 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(277..277 src has 1..277) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f9df8032a70 con 0x7f9e0417a6e0 2021-04-09T18:35:19.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.852 7f9e089e3700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:19.886 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.852 7f9e089e3700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 crc :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:19.890 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.872 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0417a1b0 con 0x7f9e0417a6e0 2021-04-09T18:35:19.894 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:19.875 7f9e017fa700 1 -- 172.21.15.32:0/586096960 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f9df80062c0 con 0x7f9e0417a6e0 2021-04-09T18:35:20.083 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.065 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e0417a1b0 con 0x7f9e0417a6e0 2021-04-09T18:35:20.085 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.065 7f9e017fa700 1 -- 172.21.15.32:0/586096960 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v277) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f9df80060b0 con 0x7f9e0417a6e0 2021-04-09T18:35:20.097 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:20.098 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":277,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:35:19.657611","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":47,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":46,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:35:16.949280","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":47,"pool_name":"cephfs_data","create_time":"2021-04-09 18:35:18.370170","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":275,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:35:20.103 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.067 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 msgr2=0x7f9de401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.104 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.067 7f9e0abc7700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 crc :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.105 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.067 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0417a6e0 msgr2=0x7f9e0417aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.067 7f9e0abc7700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0417a6e0 0x7f9e0417aae0 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7f9df800b5c0 tx=0x7f9df800b960).stop 2021-04-09T18:35:20.106 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.068 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 shutdown_connections 2021-04-09T18:35:20.108 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.068 7f9e0abc7700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9de40188d0 0x7f9de401add0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.109 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.068 7f9e0abc7700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0417a6e0 0x7f9e0417aae0 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.110 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.068 7f9e0abc7700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e0417b100 0x7f9e0408db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.111 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.068 7f9e0abc7700 1 --2- 172.21.15.32:0/586096960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e04179610 0x7f9e04179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.068 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 >> 172.21.15.32:0/586096960 conn(0x7f9e04190550 msgr2=0x7f9e041b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:20.112 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.068 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 shutdown_connections 2021-04-09T18:35:20.113 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.068 7f9e0abc7700 1 -- 172.21.15.32:0/586096960 wait complete. 2021-04-09T18:35:20.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:35:20.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.258 7fb399285700 1 Processor -- start 2021-04-09T18:35:20.299 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.258 7fb399285700 1 -- start start 2021-04-09T18:35:20.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb399285700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb39410d010 0x7fb3940d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb399285700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3940d1da0 0x7fb3940d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb399285700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3940d82c0 0x7fb39410da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb399285700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3940d8d30 con 0x7fb3940d1da0 2021-04-09T18:35:20.300 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb399285700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3940d9220 con 0x7fb39410d010 2021-04-09T18:35:20.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb399285700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39410d4b0 con 0x7fb3940d82c0 2021-04-09T18:35:20.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39287c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3940d1da0 0x7fb3940d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39287c700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb3940d1da0 0x7fb3940d7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60396/0 (socket says 172.21.15.32:60396) 2021-04-09T18:35:20.301 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39287c700 1 -- 172.21.15.32:0/1125920411 learned_addr learned my addr 172.21.15.32:0/1125920411 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:20.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39387e700 1 --2- 172.21.15.32:0/1125920411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3940d82c0 0x7fb39410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39307d700 1 --2- 172.21.15.32:0/1125920411 >> v2:172.21.15.86:3300/0 conn(0x7fb39410d010 0x7fb3940d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39287c700 1 -- 172.21.15.32:0/1125920411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3940d82c0 msgr2=0x7fb39410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:20.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39287c700 1 --2- 172.21.15.32:0/1125920411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3940d82c0 0x7fb39410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.302 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39287c700 1 -- 172.21.15.32:0/1125920411 >> v2:172.21.15.86:3300/0 conn(0x7fb39410d010 msgr2=0x7fb3940d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:20.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39287c700 1 --2- 172.21.15.32:0/1125920411 >> v2:172.21.15.86:3300/0 conn(0x7fb39410d010 0x7fb3940d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39287c700 1 -- 172.21.15.32:0/1125920411 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3940d86c0 con 0x7fb3940d1da0 2021-04-09T18:35:20.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39307d700 1 --2- 172.21.15.32:0/1125920411 >> v2:172.21.15.86:3300/0 conn(0x7fb39410d010 0x7fb3940d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:20.303 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39287c700 1 --2- 172.21.15.32:0/1125920411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3940d1da0 0x7fb3940d7ca0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 rx=0x7fb388006700 tx=0x7fb388009b80).ready entity=mon.0 client_cookie=c544019dd27f6adc server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:20.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39207b700 1 -- 172.21.15.32:0/1125920411 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb38800c0c0 con 0x7fb3940d1da0 2021-04-09T18:35:20.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39207b700 1 -- 172.21.15.32:0/1125920411 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38800c440 con 0x7fb3940d1da0 2021-04-09T18:35:20.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.259 7fb39207b700 1 -- 172.21.15.32:0/1125920411 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb38800c920 con 0x7fb3940d1da0 2021-04-09T18:35:20.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.260 7fb399285700 1 -- 172.21.15.32:0/1125920411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3940d1da0 msgr2=0x7fb3940d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.304 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.260 7fb399285700 1 --2- 172.21.15.32:0/1125920411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3940d1da0 0x7fb3940d7ca0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 rx=0x7fb388006700 tx=0x7fb388009b80).stop 2021-04-09T18:35:20.305 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.260 7fb399285700 1 -- 172.21.15.32:0/1125920411 shutdown_connections 2021-04-09T18:35:20.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.260 7fb399285700 1 --2- 172.21.15.32:0/1125920411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3940d82c0 0x7fb39410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.308 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.260 7fb399285700 1 --2- 172.21.15.32:0/1125920411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3940d1da0 0x7fb3940d7ca0 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.260 7fb399285700 1 --2- 172.21.15.32:0/1125920411 >> v2:172.21.15.86:3300/0 conn(0x7fb39410d010 0x7fb3940d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.260 7fb399285700 1 -- 172.21.15.32:0/1125920411 >> 172.21.15.32:0/1125920411 conn(0x7fb3940ce5f0 msgr2=0x7fb3940d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:20.309 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.260 7fb399285700 1 -- 172.21.15.32:0/1125920411 shutdown_connections 2021-04-09T18:35:20.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.260 7fb399285700 1 -- 172.21.15.32:0/1125920411 wait complete. 2021-04-09T18:35:20.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.280 7fb399285700 1 Processor -- start 2021-04-09T18:35:20.310 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.280 7fb399285700 1 -- start start 2021-04-09T18:35:20.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb399285700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3941a2e80 0x7fb39408d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.313 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb399285700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb39408dfc0 0x7fb394090920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb399285700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb394090f40 0x7fb39408c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb399285700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb394096ce0 con 0x7fb3941a2e80 2021-04-09T18:35:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb399285700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb394091340 con 0x7fb394090f40 2021-04-09T18:35:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb399285700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb394091560 con 0x7fb39408dfc0 2021-04-09T18:35:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb39307d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3941a2e80 0x7fb39408d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb39307d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3941a2e80 0x7fb39408d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60398/0 (socket says 172.21.15.32:60398) 2021-04-09T18:35:20.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb39307d700 1 -- 172.21.15.32:0/2403402140 learned_addr learned my addr 172.21.15.32:0/2403402140 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:20.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb39387e700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb394090f40 0x7fb39408c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb39287c700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb39408dfc0 0x7fb394090920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.281 7fb39307d700 1 -- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb39408dfc0 msgr2=0x7fb394090920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb39307d700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb39408dfc0 0x7fb394090920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb39307d700 1 -- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb394090f40 msgr2=0x7fb39408c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb39307d700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb394090f40 0x7fb39408c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb39307d700 1 -- 172.21.15.32:0/2403402140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39408ce80 con 0x7fb3941a2e80 2021-04-09T18:35:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb39307d700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3941a2e80 0x7fb39408d9a0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 rx=0x7fb384006f20 tx=0x7fb38400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:20.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb37bfff700 1 -- 172.21.15.32:0/2403402140 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb38400bd80 con 0x7fb3941a2e80 2021-04-09T18:35:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb37bfff700 1 -- 172.21.15.32:0/2403402140 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb384000c90 con 0x7fb3941a2e80 2021-04-09T18:35:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb399285700 1 -- 172.21.15.32:0/2403402140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39408ce80 con 0x7fb3941a2e80 2021-04-09T18:35:20.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb37bfff700 1 -- 172.21.15.32:0/2403402140 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3840022b0 con 0x7fb3941a2e80 2021-04-09T18:35:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.282 7fb399285700 1 -- 172.21.15.32:0/2403402140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39408ce80 con 0x7fb3941a2e80 2021-04-09T18:35:20.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.283 7fb37bfff700 1 -- 172.21.15.32:0/2403402140 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fb38400a000 con 0x7fb3941a2e80 2021-04-09T18:35:20.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.283 7fb37bfff700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3740188c0 0x7fb37401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.322 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.283 7fb37bfff700 1 -- 172.21.15.32:0/2403402140 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(277..277 src has 1..277) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fb384032a30 con 0x7fb3941a2e80 2021-04-09T18:35:20.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.283 7fb39287c700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3740188c0 0x7fb37401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.323 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.283 7fb39287c700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3740188c0 0x7fb37401adc0 crc :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:20.324 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.303 7fb399285700 1 -- 172.21.15.32:0/2403402140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3940f1820 con 0x7fb3941a2e80 2021-04-09T18:35:20.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.306 7fb37bfff700 1 -- 172.21.15.32:0/2403402140 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fb384006bb0 con 0x7fb3941a2e80 2021-04-09T18:35:20.515 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.496 7fb399285700 1 -- 172.21.15.32:0/2403402140 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb39410d480 con 0x7fb3941a2e80 2021-04-09T18:35:20.516 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.497 7fb37bfff700 1 -- 172.21.15.32:0/2403402140 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v277) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fb384006bb0 con 0x7fb3941a2e80 2021-04-09T18:35:20.525 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:20.526 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":277,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:35:19.657611","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":47,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":46,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:35:16.949280","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":47,"pool_name":"cephfs_data","create_time":"2021-04-09 18:35:18.370170","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":275,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:35:20.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 -- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3740188c0 msgr2=0x7fb37401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3740188c0 0x7fb37401adc0 crc :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 -- 172.21.15.32:0/2403402140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3941a2e80 msgr2=0x7fb39408d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3941a2e80 0x7fb39408d9a0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 rx=0x7fb384006f20 tx=0x7fb38400b990).stop 2021-04-09T18:35:20.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 -- 172.21.15.32:0/2403402140 shutdown_connections 2021-04-09T18:35:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fb3740188c0 0x7fb37401adc0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb394090f40 0x7fb39408c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb39408dfc0 0x7fb394090920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 --2- 172.21.15.32:0/2403402140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3941a2e80 0x7fb39408d9a0 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 -- 172.21.15.32:0/2403402140 >> 172.21.15.32:0/2403402140 conn(0x7fb394192580 msgr2=0x7fb3940f1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:20.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 -- 172.21.15.32:0/2403402140 shutdown_connections 2021-04-09T18:35:20.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.499 7fb399285700 1 -- 172.21.15.32:0/2403402140 wait complete. 2021-04-09T18:35:20.537 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-04-09T18:35:20.724 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.683 7f78d5e4f700 1 Processor -- start 2021-04-09T18:35:20.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.683 7f78d5e4f700 1 -- start start 2021-04-09T18:35:20.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78d5e4f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00d7ca0 0x7f78d00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.725 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78d5e4f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d010d480 con 0x7f78d010da00 2021-04-09T18:35:20.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78d5e4f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d00d86c0 con 0x7f78d010d010 2021-04-09T18:35:20.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78d5e4f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00d9220 con 0x7f78d00d7ca0 2021-04-09T18:35:20.726 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78cffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f78d010da00 0x7f78d010de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41766/0 (socket says 172.21.15.32:41766) 2021-04-09T18:35:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78cffff700 1 -- 172.21.15.32:0/1257400210 learned_addr learned my addr 172.21.15.32:0/1257400210 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78ceffd700 1 --2- 172.21.15.32:0/1257400210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00d7ca0 0x7f78d00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.727 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78ce7fc700 1 -- 172.21.15.32:0/1257400210 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886997199 0 0) 0x7f78c0000d30 con 0x7f78d010da00 2021-04-09T18:35:20.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78ce7fc700 1 -- 172.21.15.32:0/1257400210 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78bc002fe0 con 0x7f78d010da00 2021-04-09T18:35:20.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78ce7fc700 1 -- 172.21.15.32:0/1257400210 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230752293 0 0) 0x7f78c8000c60 con 0x7f78d010d010 2021-04-09T18:35:20.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78ce7fc700 1 -- 172.21.15.32:0/1257400210 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78bc0042c0 con 0x7f78d010d010 2021-04-09T18:35:20.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78ceffd700 1 -- 172.21.15.32:0/1257400210 >> v1:172.21.15.86:6789/0 conn(0x7f78d010d010 legacy=0x7f78d00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78ceffd700 1 -- 172.21.15.32:0/1257400210 >> v1:172.21.15.32:6789/0 conn(0x7f78d010da00 legacy=0x7f78d010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78ceffd700 1 -- 172.21.15.32:0/1257400210 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0112100 con 0x7f78d00d7ca0 2021-04-09T18:35:20.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.684 7f78ceffd700 1 --2- 172.21.15.32:0/1257400210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00d7ca0 0x7f78d00d80a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f78b8006650 tx=0x7f78b8009a80).ready entity=mon.2 client_cookie=1f06152b94980432 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:20.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.685 7f78ce7fc700 1 -- 172.21.15.32:0/1257400210 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78b80009b0 con 0x7f78d00d7ca0 2021-04-09T18:35:20.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.685 7f78ce7fc700 1 -- 172.21.15.32:0/1257400210 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b800c0c0 con 0x7f78d00d7ca0 2021-04-09T18:35:20.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.685 7f78ce7fc700 1 -- 172.21.15.32:0/1257400210 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78b800c510 con 0x7f78d00d7ca0 2021-04-09T18:35:20.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.685 7f78d5e4f700 1 -- 172.21.15.32:0/1257400210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00d7ca0 msgr2=0x7f78d00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.685 7f78d5e4f700 1 --2- 172.21.15.32:0/1257400210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00d7ca0 0x7f78d00d80a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f78b8006650 tx=0x7f78b8009a80).stop 2021-04-09T18:35:20.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.685 7f78d5e4f700 1 -- 172.21.15.32:0/1257400210 shutdown_connections 2021-04-09T18:35:20.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.685 7f78d5e4f700 1 --2- 172.21.15.32:0/1257400210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d00d7ca0 0x7f78d00d80a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.685 7f78d5e4f700 1 -- 172.21.15.32:0/1257400210 >> 172.21.15.32:0/1257400210 conn(0x7f78d00ce5f0 msgr2=0x7f78d00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:20.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.685 7f78d5e4f700 1 -- 172.21.15.32:0/1257400210 shutdown_connections 2021-04-09T18:35:20.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.686 7f78d5e4f700 1 -- 172.21.15.32:0/1257400210 wait complete. 2021-04-09T18:35:20.736 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.706 7f78d5e4f700 1 Processor -- start 2021-04-09T18:35:20.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.706 7f78d5e4f700 1 -- start start 2021-04-09T18:35:20.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78d5e4f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d0191ee0 0x7f78d00a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.737 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78d5e4f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d00a6c30 0x7f78d00b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78d5e4f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d00b4330 0x7f78d01b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78d5e4f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00b3790 con 0x7f78d00a6c30 2021-04-09T18:35:20.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78d5e4f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00a7030 con 0x7f78d00b4330 2021-04-09T18:35:20.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78d5e4f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00b4730 con 0x7f78d0191ee0 2021-04-09T18:35:20.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78ceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d00a6c30 0x7f78d00b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78ceffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d00a6c30 0x7f78d00b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60412/0 (socket says 172.21.15.32:60412) 2021-04-09T18:35:20.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78ceffd700 1 -- 172.21.15.32:0/2730569503 learned_addr learned my addr 172.21.15.32:0/2730569503 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:20.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78cffff700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d00b4330 0x7f78d01b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78cf7fe700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d0191ee0 0x7f78d00a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78ceffd700 1 -- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d0191ee0 msgr2=0x7f78d00a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78ceffd700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d0191ee0 0x7f78d00a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78ceffd700 1 -- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d00b4330 msgr2=0x7f78d01b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:20.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78ceffd700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d00b4330 0x7f78d01b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:20.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78ceffd700 1 -- 172.21.15.32:0/2730569503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d01b8580 con 0x7f78d00a6c30 2021-04-09T18:35:20.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78ceffd700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d00a6c30 0x7f78d00b3d10 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 rx=0x7f78b8009650 tx=0x7f78b800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:20.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78c7fff700 1 -- 172.21.15.32:0/2730569503 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78b80093a0 con 0x7f78d00a6c30 2021-04-09T18:35:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78c7fff700 1 -- 172.21.15.32:0/2730569503 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b800bd50 con 0x7f78d00a6c30 2021-04-09T18:35:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d01b8580 con 0x7f78d00a6c30 2021-04-09T18:35:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78c7fff700 1 -- 172.21.15.32:0/2730569503 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78b80052e0 con 0x7f78d00a6c30 2021-04-09T18:35:20.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.707 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d01b8580 con 0x7f78d00a6c30 2021-04-09T18:35:20.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.708 7f78c7fff700 1 -- 172.21.15.32:0/2730569503 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f78b8006030 con 0x7f78d00a6c30 2021-04-09T18:35:20.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.708 7f78c7fff700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b4018930 0x7f78b401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:20.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.708 7f78c7fff700 1 -- 172.21.15.32:0/2730569503 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f78b80329e0 con 0x7f78d00a6c30 2021-04-09T18:35:20.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.708 7f78cf7fe700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b4018930 0x7f78b401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:20.745 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.708 7f78cf7fe700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b4018930 0x7f78b401ae10 crc :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:20.746 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.728 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d010d480 con 0x7f78d00a6c30 2021-04-09T18:35:20.749 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.731 7f78c7fff700 1 -- 172.21.15.32:0/2730569503 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f78b80009c0 con 0x7f78d00a6c30 2021-04-09T18:35:20.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:20.925 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f78d010d480 con 0x7f78d00a6c30 2021-04-09T18:35:21.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.921 7f78c7fff700 1 -- 172.21.15.32:0/2730569503 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v443) v1 ==== 118+0+0 (secure 0 0 0) 0x7f78b8006030 con 0x7f78d00a6c30 2021-04-09T18:35:21.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b4018930 msgr2=0x7f78b401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:21.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b4018930 0x7f78b401ae10 crc :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:21.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d00a6c30 msgr2=0x7f78d00b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:21.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d00a6c30 0x7f78d00b3d10 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 rx=0x7f78b8009650 tx=0x7f78b800b960).stop 2021-04-09T18:35:21.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 shutdown_connections 2021-04-09T18:35:21.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f78b4018930 0x7f78b401ae10 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:21.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78d00a6c30 0x7f78d00b3d10 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:21.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d00b4330 0x7f78d01b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:21.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 --2- 172.21.15.32:0/2730569503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d0191ee0 0x7f78d00a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:21.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 >> 172.21.15.32:0/2730569503 conn(0x7f78d00bbda0 msgr2=0x7f78d01b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:21.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 shutdown_connections 2021-04-09T18:35:21.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:21.943 7f78d5e4f700 1 -- 172.21.15.32:0/2730569503 wait complete. 2021-04-09T18:35:21.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:35:22.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.106 7f61367ff700 1 Processor -- start 2021-04-09T18:35:22.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.106 7f61367ff700 1 -- start start 2021-04-09T18:35:22.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.106 7f61367ff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f613010d010 0x7f61300d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f61367ff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61300d8490 0x7f613010da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f61367ff700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613010d480 con 0x7f61300d1da0 2021-04-09T18:35:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f61367ff700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61300d8d30 con 0x7f613010d010 2021-04-09T18:35:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f61367ff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61300d9220 con 0x7f61300d8490 2021-04-09T18:35:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f6134e1c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61300d8490 0x7f613010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f612ffff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f613010d010 0x7f61300d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f6134e1c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61300d8490 0x7f613010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.32:39416/0 (socket says 172.21.15.32:39416) 2021-04-09T18:35:22.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f6134e1c700 1 -- 172.21.15.32:0/673106718 learned_addr learned my addr 172.21.15.32:0/673106718 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:22.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f612ffff700 1 -- 172.21.15.32:0/673106718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61300d8490 msgr2=0x7f613010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:22.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f612ffff700 1 --2- 172.21.15.32:0/673106718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61300d8490 0x7f613010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f612ffff700 1 -- 172.21.15.32:0/673106718 >> v1:172.21.15.86:6789/0 conn(0x7f61300d1da0 legacy=0x7f61300d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f612ffff700 1 -- 172.21.15.32:0/673106718 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613010e020 con 0x7f613010d010 2021-04-09T18:35:22.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f612f7fe700 1 -- 172.21.15.32:0/673106718 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148445968 0 0) 0x7f6118000c60 con 0x7f61300d1da0 2021-04-09T18:35:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f6134e1c700 1 --2- 172.21.15.32:0/673106718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61300d8490 0x7f613010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:22.154 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.107 7f612ffff700 1 --2- 172.21.15.32:0/673106718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f613010d010 0x7f61300d1780 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f6120006650 tx=0x7f6120009a80).ready entity=mon.1 client_cookie=96e2fdfac59741df server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.108 7f612f7fe700 1 -- 172.21.15.32:0/673106718 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61200009b0 con 0x7f613010d010 2021-04-09T18:35:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.108 7f612f7fe700 1 -- 172.21.15.32:0/673106718 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612000c0c0 con 0x7f613010d010 2021-04-09T18:35:22.155 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.108 7f612f7fe700 1 -- 172.21.15.32:0/673106718 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f612000c510 con 0x7f613010d010 2021-04-09T18:35:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.108 7f61367ff700 1 -- 172.21.15.32:0/673106718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f613010d010 msgr2=0x7f61300d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.108 7f61367ff700 1 --2- 172.21.15.32:0/673106718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f613010d010 0x7f61300d1780 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f6120006650 tx=0x7f6120009a80).stop 2021-04-09T18:35:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.108 7f61367ff700 1 -- 172.21.15.32:0/673106718 shutdown_connections 2021-04-09T18:35:22.156 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.108 7f61367ff700 1 --2- 172.21.15.32:0/673106718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61300d8490 0x7f613010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.108 7f61367ff700 1 --2- 172.21.15.32:0/673106718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f613010d010 0x7f61300d1780 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.108 7f61367ff700 1 -- 172.21.15.32:0/673106718 >> 172.21.15.32:0/673106718 conn(0x7f61300ce5f0 msgr2=0x7f61300d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.109 7f61367ff700 1 -- 172.21.15.32:0/673106718 shutdown_connections 2021-04-09T18:35:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.109 7f61367ff700 1 -- 172.21.15.32:0/673106718 wait complete. 2021-04-09T18:35:22.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f61367ff700 1 Processor -- start 2021-04-09T18:35:22.158 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f61367ff700 1 -- start start 2021-04-09T18:35:22.162 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f61367ff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61301aca20 0x7f61301ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f61367ff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613018bd40 0x7f613018c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f61367ff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f613008fba0 0x7f613008ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.163 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f61367ff700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613018c7e0 con 0x7f61301aca20 2021-04-09T18:35:22.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f61367ff700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613018ca30 con 0x7f613008fba0 2021-04-09T18:35:22.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f61367ff700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61300905f0 con 0x7f613018bd40 2021-04-09T18:35:22.164 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f612ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61301aca20 0x7f61301ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f612ffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61301aca20 0x7f61301ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60426/0 (socket says 172.21.15.32:60426) 2021-04-09T18:35:22.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.129 7f612ffff700 1 -- 172.21.15.32:0/4230495876 learned_addr learned my addr 172.21.15.32:0/4230495876 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:22.165 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f6134e1c700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f613008fba0 0x7f613008ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f6127fff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613018bd40 0x7f613018c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f612ffff700 1 -- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613018bd40 msgr2=0x7f613018c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f612ffff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613018bd40 0x7f613018c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.166 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f612ffff700 1 -- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f613008fba0 msgr2=0x7f613008ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f612ffff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f613008fba0 0x7f613008ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f612ffff700 1 -- 172.21.15.32:0/4230495876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61300b4300 con 0x7f61301aca20 2021-04-09T18:35:22.167 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f612ffff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61301aca20 0x7f61301ace20 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f6120009650 tx=0x7f612000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:22.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f612d7fa700 1 -- 172.21.15.32:0/4230495876 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61200093a0 con 0x7f61301aca20 2021-04-09T18:35:22.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f612d7fa700 1 -- 172.21.15.32:0/4230495876 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612000bd50 con 0x7f61301aca20 2021-04-09T18:35:22.168 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f61367ff700 1 -- 172.21.15.32:0/4230495876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61300b4300 con 0x7f61301aca20 2021-04-09T18:35:22.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f612d7fa700 1 -- 172.21.15.32:0/4230495876 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61200052e0 con 0x7f61301aca20 2021-04-09T18:35:22.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.130 7f61367ff700 1 -- 172.21.15.32:0/4230495876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61300b4300 con 0x7f61301aca20 2021-04-09T18:35:22.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.131 7f612d7fa700 1 -- 172.21.15.32:0/4230495876 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6120006030 con 0x7f61301aca20 2021-04-09T18:35:22.169 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.131 7f612d7fa700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61140188d0 0x7f611401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.131 7f612d7fa700 1 -- 172.21.15.32:0/4230495876 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f61200329e0 con 0x7f61301aca20 2021-04-09T18:35:22.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.131 7f6127fff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61140188d0 0x7f611401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.170 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.131 7f6127fff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61140188d0 0x7f611401add0 crc :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:22.171 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.150 7f61367ff700 1 -- 172.21.15.32:0/4230495876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6130182d70 con 0x7f61301aca20 2021-04-09T18:35:22.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.154 7f612d7fa700 1 -- 172.21.15.32:0/4230495876 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f61200009c0 con 0x7f61301aca20 2021-04-09T18:35:22.364 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.346 7f61367ff700 1 -- 172.21.15.32:0/4230495876 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f613018c760 con 0x7f61301aca20 2021-04-09T18:35:22.365 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.346 7f612d7fa700 1 -- 172.21.15.32:0/4230495876 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 443 v443) v1 ==== 95+0+2789 (secure 0 0 0) 0x7f612000bd50 con 0x7f61301aca20 2021-04-09T18:35:22.366 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 443 2021-04-09T18:35:22.366 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:22.367 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":443,"default_fscid":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2260217511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2260217511},{"type":"v1","addr":"172.21.15.32:6834","nonce":2260217511}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":13257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1133511394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1133511394},{"type":"v1","addr":"172.21.15.32:6835","nonce":1133511394}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":60194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1457882085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1457882085},{"type":"v1","addr":"172.21.15.86:6837","nonce":1457882085}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439}],"filesystems":[{"mdsmap":{"epoch":443,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:35:19.657739","modified":"2021-04-09 18:35:21.914126","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12706":{"gid":12706,"name":"d","rank":0,"incarnation":440,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3693986652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3693986652},{"type":"v1","addr":"172.21.15.86:6836","nonce":3693986652}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[47],"metadata_pool":46,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":23}]} 2021-04-09T18:35:22.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 -- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61140188d0 msgr2=0x7f611401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.368 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61140188d0 0x7f611401add0 crc :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 -- 172.21.15.32:0/4230495876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61301aca20 msgr2=0x7f61301ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.369 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61301aca20 0x7f61301ace20 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f6120009650 tx=0x7f612000b960).stop 2021-04-09T18:35:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 -- 172.21.15.32:0/4230495876 shutdown_connections 2021-04-09T18:35:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f61140188d0 0x7f611401add0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f613008fba0 0x7f613008ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f613018bd40 0x7f613018c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.370 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 --2- 172.21.15.32:0/4230495876 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61301aca20 0x7f61301ace20 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 -- 172.21.15.32:0/4230495876 >> 172.21.15.32:0/4230495876 conn(0x7f613019b460 msgr2=0x7f613019bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:22.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 -- 172.21.15.32:0/4230495876 shutdown_connections 2021-04-09T18:35:22.371 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.349 7f61367ff700 1 -- 172.21.15.32:0/4230495876 wait complete. 2021-04-09T18:35:22.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:35:22.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.507 7f92bd587700 1 Processor -- start 2021-04-09T18:35:22.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.507 7f92bd587700 1 -- start start 2021-04-09T18:35:22.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92bd587700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f92b80d7ca0 0x7f92b80d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92bd587700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b810d480 con 0x7f92b810da00 2021-04-09T18:35:22.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92bd587700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b80d86c0 con 0x7f92b810d010 2021-04-09T18:35:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92bd587700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b80d9220 con 0x7f92b80d7ca0 2021-04-09T18:35:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92b6abe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f92b80d7ca0 0x7f92b80d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92b7ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f92b810da00 0x7f92b810de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41792/0 (socket says 172.21.15.32:41792) 2021-04-09T18:35:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92b7ac0700 1 -- 172.21.15.32:0/3115886544 learned_addr learned my addr 172.21.15.32:0/3115886544 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:22.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92b62bd700 1 -- 172.21.15.32:0/3115886544 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325135611 0 0) 0x7f92b0000d30 con 0x7f92b810da00 2021-04-09T18:35:22.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92b62bd700 1 -- 172.21.15.32:0/3115886544 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a0002fe0 con 0x7f92b810da00 2021-04-09T18:35:22.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92b6abe700 1 -- 172.21.15.32:0/3115886544 >> v1:172.21.15.86:6789/0 conn(0x7f92b810d010 legacy=0x7f92b80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92b6abe700 1 -- 172.21.15.32:0/3115886544 >> v1:172.21.15.32:6789/0 conn(0x7f92b810da00 legacy=0x7f92b810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.508 7f92b6abe700 1 -- 172.21.15.32:0/3115886544 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b80d1f70 con 0x7f92b80d7ca0 2021-04-09T18:35:22.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92b6abe700 1 --2- 172.21.15.32:0/3115886544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b80d7ca0 0x7f92b80d80a0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7f92ac006650 tx=0x7f92ac009a80).ready entity=mon.0 client_cookie=62c80e4a6d25acf5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92b62bd700 1 -- 172.21.15.32:0/3115886544 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ac0009b0 con 0x7f92b80d7ca0 2021-04-09T18:35:22.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92b62bd700 1 -- 172.21.15.32:0/3115886544 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac00c0c0 con 0x7f92b80d7ca0 2021-04-09T18:35:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92b62bd700 1 -- 172.21.15.32:0/3115886544 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ac00c510 con 0x7f92b80d7ca0 2021-04-09T18:35:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92bd587700 1 -- 172.21.15.32:0/3115886544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b80d7ca0 msgr2=0x7f92b80d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92bd587700 1 --2- 172.21.15.32:0/3115886544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b80d7ca0 0x7f92b80d80a0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7f92ac006650 tx=0x7f92ac009a80).stop 2021-04-09T18:35:22.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92bd587700 1 -- 172.21.15.32:0/3115886544 shutdown_connections 2021-04-09T18:35:22.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92bd587700 1 --2- 172.21.15.32:0/3115886544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b80d7ca0 0x7f92b80d80a0 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92bd587700 1 -- 172.21.15.32:0/3115886544 >> 172.21.15.32:0/3115886544 conn(0x7f92b80ce5f0 msgr2=0x7f92b80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92bd587700 1 -- 172.21.15.32:0/3115886544 shutdown_connections 2021-04-09T18:35:22.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.509 7f92bd587700 1 -- 172.21.15.32:0/3115886544 wait complete. 2021-04-09T18:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.530 7f92bd587700 1 Processor -- start 2021-04-09T18:35:22.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.530 7f92bd587700 1 -- start start 2021-04-09T18:35:22.564 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92bd587700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b80af100 0x7f92b8090b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92bd587700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b81868f0 0x7f92b8185ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.565 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92bd587700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92b81a9470 0x7f92b81a9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.566 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92bd587700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b81860d0 con 0x7f92b81868f0 2021-04-09T18:35:22.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92bd587700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b8186cf0 con 0x7f92b80af100 2021-04-09T18:35:22.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92bd587700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b8195280 con 0x7f92b81a9470 2021-04-09T18:35:22.567 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b6abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b81868f0 0x7f92b8185ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b6abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b81868f0 0x7f92b8185ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60440/0 (socket says 172.21.15.32:60440) 2021-04-09T18:35:22.568 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b6abe700 1 -- 172.21.15.32:0/3678749709 learned_addr learned my addr 172.21.15.32:0/3678749709 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:22.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b72bf700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b80af100 0x7f92b8090b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.569 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b7ac0700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92b81a9470 0x7f92b81a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b6abe700 1 -- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92b81a9470 msgr2=0x7f92b81a9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.570 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b6abe700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92b81a9470 0x7f92b81a9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b6abe700 1 -- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b80af100 msgr2=0x7f92b8090b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.571 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b6abe700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b80af100 0x7f92b8090b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b6abe700 1 -- 172.21.15.32:0/3678749709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b809d9d0 con 0x7f92b81868f0 2021-04-09T18:35:22.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92b6abe700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b81868f0 0x7f92b8185ab0 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7f92ac009650 tx=0x7f92ac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:22.572 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f929ffff700 1 -- 172.21.15.32:0/3678749709 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ac0093a0 con 0x7f92b81868f0 2021-04-09T18:35:22.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f929ffff700 1 -- 172.21.15.32:0/3678749709 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac00bd50 con 0x7f92b81868f0 2021-04-09T18:35:22.573 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92bd587700 1 -- 172.21.15.32:0/3678749709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b809d9d0 con 0x7f92b81868f0 2021-04-09T18:35:22.574 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f929ffff700 1 -- 172.21.15.32:0/3678749709 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ac0052e0 con 0x7f92b81868f0 2021-04-09T18:35:22.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.531 7f92bd587700 1 -- 172.21.15.32:0/3678749709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b809d9d0 con 0x7f92b81868f0 2021-04-09T18:35:22.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.532 7f929ffff700 1 -- 172.21.15.32:0/3678749709 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f92ac006030 con 0x7f92b81868f0 2021-04-09T18:35:22.575 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.532 7f929ffff700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9298018930 0x7f929801adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.532 7f929ffff700 1 -- 172.21.15.32:0/3678749709 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f92ac0329e0 con 0x7f92b81868f0 2021-04-09T18:35:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.532 7f92b72bf700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9298018930 0x7f929801adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.576 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.533 7f92b72bf700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9298018930 0x7f929801adc0 crc :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.552 7f92bd587700 1 -- 172.21.15.32:0/3678749709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b810d480 con 0x7f92b81868f0 2021-04-09T18:35:22.577 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.556 7f929ffff700 1 -- 172.21.15.32:0/3678749709 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f92ac0009c0 con 0x7f92b81868f0 2021-04-09T18:35:22.763 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.744 7f92bd587700 1 -- 172.21.15.32:0/3678749709 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92b810d480 con 0x7f92b81868f0 2021-04-09T18:35:22.764 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.745 7f929ffff700 1 -- 172.21.15.32:0/3678749709 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v278) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f92ac006030 con 0x7f92b81868f0 2021-04-09T18:35:22.765 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:22.765 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":278,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:35:20.659017","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":47,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":46,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:35:16.949280","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":47,"pool_name":"cephfs_data","create_time":"2021-04-09 18:35:18.370170","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":275,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:35:22.766 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 -- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9298018930 msgr2=0x7f929801adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9298018930 0x7f929801adc0 crc :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 -- 172.21.15.32:0/3678749709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b81868f0 msgr2=0x7f92b8185ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b81868f0 0x7f92b8185ab0 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7f92ac009650 tx=0x7f92ac00b960).stop 2021-04-09T18:35:22.767 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 -- 172.21.15.32:0/3678749709 shutdown_connections 2021-04-09T18:35:22.768 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f9298018930 0x7f929801adc0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92b81868f0 0x7f92b8185ab0 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92b81a9470 0x7f92b81a9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.770 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 --2- 172.21.15.32:0/3678749709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b80af100 0x7f92b8090b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.747 7f92bd587700 1 -- 172.21.15.32:0/3678749709 >> 172.21.15.32:0/3678749709 conn(0x7f92b81965c0 msgr2=0x7f92b81ac720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.748 7f92bd587700 1 -- 172.21.15.32:0/3678749709 shutdown_connections 2021-04-09T18:35:22.771 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.748 7f92bd587700 1 -- 172.21.15.32:0/3678749709 wait complete. 2021-04-09T18:35:22.777 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:35:22.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.920 7fc1cbac8700 1 Processor -- start 2021-04-09T18:35:22.962 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.920 7fc1cbac8700 1 -- start start 2021-04-09T18:35:22.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.920 7fc1cbac8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1c40cffb0 0x7fc1c4087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1cbac8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c4088460 con 0x7fc1c40d7c50 2021-04-09T18:35:22.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1cbac8700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c40d8620 con 0x7fc1c40d8050 2021-04-09T18:35:22.963 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1cbac8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c41b0c80 con 0x7fc1c40cffb0 2021-04-09T18:35:22.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1c98e4700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fc1c40cffb0 0x7fc1c4087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1c90e3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc1c40d7c50 0x7fc1c41b0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41806/0 (socket says 172.21.15.32:41806) 2021-04-09T18:35:22.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1c90e3700 1 -- 172.21.15.32:0/3612169800 learned_addr learned my addr 172.21.15.32:0/3612169800 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:22.964 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1c88e2700 1 -- 172.21.15.32:0/3612169800 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406406550 0 0) 0x7fc1b8000ce0 con 0x7fc1c40d7c50 2021-04-09T18:35:22.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1c98e4700 1 -- 172.21.15.32:0/3612169800 >> v1:172.21.15.86:6789/0 conn(0x7fc1c40d8050 legacy=0x7fc1c410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1c98e4700 1 -- 172.21.15.32:0/3612169800 >> v1:172.21.15.32:6789/0 conn(0x7fc1c40d7c50 legacy=0x7fc1c41b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1c98e4700 1 -- 172.21.15.32:0/3612169800 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40d91f0 con 0x7fc1c40cffb0 2021-04-09T18:35:22.965 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.921 7fc1c98e4700 1 --2- 172.21.15.32:0/3612169800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40cffb0 0x7fc1c4087e40 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 rx=0x7fc1b4006650 tx=0x7fc1b4009a80).ready entity=mon.0 client_cookie=f5c87cc723d1b4c5 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:22.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.923 7fc1c88e2700 1 -- 172.21.15.32:0/3612169800 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1b40009b0 con 0x7fc1c40cffb0 2021-04-09T18:35:22.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.923 7fc1c88e2700 1 -- 172.21.15.32:0/3612169800 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b400c0c0 con 0x7fc1c40cffb0 2021-04-09T18:35:22.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.923 7fc1c88e2700 1 -- 172.21.15.32:0/3612169800 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1b400c510 con 0x7fc1c40cffb0 2021-04-09T18:35:22.966 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.923 7fc1cbac8700 1 -- 172.21.15.32:0/3612169800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40cffb0 msgr2=0x7fc1c4087e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.923 7fc1cbac8700 1 --2- 172.21.15.32:0/3612169800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40cffb0 0x7fc1c4087e40 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 rx=0x7fc1b4006650 tx=0x7fc1b4009a80).stop 2021-04-09T18:35:22.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.924 7fc1cbac8700 1 -- 172.21.15.32:0/3612169800 shutdown_connections 2021-04-09T18:35:22.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.924 7fc1cbac8700 1 --2- 172.21.15.32:0/3612169800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40cffb0 0x7fc1c4087e40 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.924 7fc1cbac8700 1 -- 172.21.15.32:0/3612169800 >> 172.21.15.32:0/3612169800 conn(0x7fc1c40ce5b0 msgr2=0x7fc1c40cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:22.967 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.924 7fc1cbac8700 1 -- 172.21.15.32:0/3612169800 shutdown_connections 2021-04-09T18:35:22.968 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.924 7fc1cbac8700 1 -- 172.21.15.32:0/3612169800 wait complete. 2021-04-09T18:35:22.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.944 7fc1cbac8700 1 Processor -- start 2021-04-09T18:35:22.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.944 7fc1cbac8700 1 -- start start 2021-04-09T18:35:22.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1cbac8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40c0f30 0x7fc1c40a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.971 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1cbac8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c40a9f20 0x7fc1c40a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1cbac8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1c40a1b50 0x7fc1c40a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1cbac8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40a9380 con 0x7fc1c40c0f30 2021-04-09T18:35:22.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1cbac8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40aa320 con 0x7fc1c40a9f20 2021-04-09T18:35:22.972 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1cbac8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40c1330 con 0x7fc1c40a1b50 2021-04-09T18:35:22.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c98e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40c0f30 0x7fc1c40a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c98e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40c0f30 0x7fc1c40a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60452/0 (socket says 172.21.15.32:60452) 2021-04-09T18:35:22.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c98e4700 1 -- 172.21.15.32:0/564641055 learned_addr learned my addr 172.21.15.32:0/564641055 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:22.973 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c90e3700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c40a9f20 0x7fc1c40a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1ca0e5700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1c40a1b50 0x7fc1c40a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c98e4700 1 -- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1c40a1b50 msgr2=0x7fc1c40a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c98e4700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1c40a1b50 0x7fc1c40a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.974 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c98e4700 1 -- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c40a9f20 msgr2=0x7fc1c40a1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:22.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c98e4700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c40a9f20 0x7fc1c40a1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:22.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c98e4700 1 -- 172.21.15.32:0/564641055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40a1f50 con 0x7fc1c40c0f30 2021-04-09T18:35:22.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c98e4700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40c0f30 0x7fc1c40a9900 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7fc1b4009650 tx=0x7fc1b400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:22.975 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c27fc700 1 -- 172.21.15.32:0/564641055 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1b40093a0 con 0x7fc1c40c0f30 2021-04-09T18:35:22.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.945 7fc1c27fc700 1 -- 172.21.15.32:0/564641055 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b400bd50 con 0x7fc1c40c0f30 2021-04-09T18:35:22.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.946 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c40a1f50 con 0x7fc1c40c0f30 2021-04-09T18:35:22.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.946 7fc1c27fc700 1 -- 172.21.15.32:0/564641055 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1b40052e0 con 0x7fc1c40c0f30 2021-04-09T18:35:22.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.946 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c40a1f50 con 0x7fc1c40c0f30 2021-04-09T18:35:22.976 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.946 7fc1c27fc700 1 -- 172.21.15.32:0/564641055 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc1b4006030 con 0x7fc1c40c0f30 2021-04-09T18:35:22.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.946 7fc1c27fc700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1a4018930 0x7fc1a401ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:22.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.946 7fc1c27fc700 1 -- 172.21.15.32:0/564641055 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc1b40329e0 con 0x7fc1c40c0f30 2021-04-09T18:35:22.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.946 7fc1c90e3700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1a4018930 0x7fc1a401ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:22.977 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.947 7fc1c90e3700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1a4018930 0x7fc1a401ae10 crc :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:22.983 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.964 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c4093ff0 con 0x7fc1c40c0f30 2021-04-09T18:35:22.986 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:22.968 7fc1c27fc700 1 -- 172.21.15.32:0/564641055 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc1b40009c0 con 0x7fc1c40c0f30 2021-04-09T18:35:23.172 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.154 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc1c40d8cd0 con 0x7fc1c40c0f30 2021-04-09T18:35:23.175 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.157 7fc1c27fc700 1 -- 172.21.15.32:0/564641055 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v51) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc1b4006030 con 0x7fc1c40c0f30 2021-04-09T18:35:23.180 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.0 2021-04-09T18:35:23.182 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.163 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1a4018930 msgr2=0x7fc1a401ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.163 7fc1cbac8700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1a4018930 0x7fc1a401ae10 crc :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.163 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40c0f30 msgr2=0x7fc1c40a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.163 7fc1cbac8700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40c0f30 0x7fc1c40a9900 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7fc1b4009650 tx=0x7fc1b400b960).stop 2021-04-09T18:35:23.183 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.164 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 shutdown_connections 2021-04-09T18:35:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.164 7fc1cbac8700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1a4018930 0x7fc1a401ae10 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.164 7fc1cbac8700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1c40a1b50 0x7fc1c40a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.164 7fc1cbac8700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c40a9f20 0x7fc1c40a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.164 7fc1cbac8700 1 --2- 172.21.15.32:0/564641055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1c40c0f30 0x7fc1c40a9900 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.184 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.164 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 >> 172.21.15.32:0/564641055 conn(0x7fc1c40adb50 msgr2=0x7fc1c40ab280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:23.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.164 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 shutdown_connections 2021-04-09T18:35:23.185 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.164 7fc1cbac8700 1 -- 172.21.15.32:0/564641055 wait complete. 2021-04-09T18:35:23.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-04-09T18:35:23.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.335 7f3d64a1a700 1 Processor -- start 2021-04-09T18:35:23.383 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.335 7f3d64a1a700 1 -- start start 2021-04-09T18:35:23.384 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.335 7f3d64a1a700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d600cffb0 0x7f3d60087e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.335 7f3d64a1a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d60088460 con 0x7f3d600d7c50 2021-04-09T18:35:23.385 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.335 7f3d64a1a700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d600d8620 con 0x7f3d600d8050 2021-04-09T18:35:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d64a1a700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d601b0c80 con 0x7f3d600cffb0 2021-04-09T18:35:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5de1b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d600d7c50 0x7f3d601b0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41822/0 (socket says 172.21.15.32:41822) 2021-04-09T18:35:23.386 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5de1b700 1 -- 172.21.15.32:0/4206471343 learned_addr learned my addr 172.21.15.32:0/4206471343 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:23.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5e61c700 1 --2- 172.21.15.32:0/4206471343 >> v2:172.21.15.86:3300/0 conn(0x7f3d600cffb0 0x7f3d60087e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.387 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395051150 0 0) 0x7f3d54000d30 con 0x7f3d600d7c50 2021-04-09T18:35:23.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48002fe0 con 0x7f3d600d7c50 2021-04-09T18:35:23.388 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 575479003 0 0) 0x7f3d54001390 con 0x7f3d600d7c50 2021-04-09T18:35:23.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 >> v2:172.21.15.86:3300/0 conn(0x7f3d600cffb0 msgr2=0x7f3d60087e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:23.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 --2- 172.21.15.32:0/4206471343 >> v2:172.21.15.86:3300/0 conn(0x7f3d600cffb0 0x7f3d60087e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.389 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 >> v1:172.21.15.86:6789/0 conn(0x7f3d600d8050 legacy=0x7f3d6010d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d600d91f0 con 0x7f3d600d7c50 2021-04-09T18:35:23.390 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7f3d540018b0 con 0x7f3d600d7c50 2021-04-09T18:35:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5e61c700 1 --2- 172.21.15.32:0/4206471343 >> v2:172.21.15.86:3300/0 conn(0x7f3d600cffb0 0x7f3d60087e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 >> v1:172.21.15.32:6789/0 conn(0x7f3d600d7c50 legacy=0x7f3d601b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.391 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d48003aa0 0x7f3d48005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d48006770 0x7f3d48008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.392 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d48009590 0x7f3d4800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4800c300 con 0x7f3d48009590 2021-04-09T18:35:23.393 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4800c520 con 0x7f3d48006770 2021-04-09T18:35:23.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.336 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d480506e0 con 0x7f3d48003aa0 2021-04-09T18:35:23.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5ee1d700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d48009590 0x7f3d4800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.394 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5e61c700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d48003aa0 0x7f3d48005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.395 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5de1b700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d48006770 0x7f3d48008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5ee1d700 1 -- 172.21.15.32:0/4206471343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d48003aa0 msgr2=0x7f3d48005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:23.396 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5ee1d700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d48003aa0 0x7f3d48005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5ee1d700 1 -- 172.21.15.32:0/4206471343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d48006770 msgr2=0x7f3d48008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:23.397 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5ee1d700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d48006770 0x7f3d48008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5ee1d700 1 -- 172.21.15.32:0/4206471343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d48050980 con 0x7f3d48009590 2021-04-09T18:35:23.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5ee1d700 1 -- 172.21.15.32:0/4206471343 reap_dead start 2021-04-09T18:35:23.398 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5ee1d700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d48009590 0x7f3d4800bab0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7f3d58000bd0 tx=0x7f3d58009a20).ready entity=mon.0 client_cookie=2e4af5583d37a638 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:23.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d580066e0 con 0x7f3d48009590 2021-04-09T18:35:23.399 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5800c0c0 con 0x7f3d48009590 2021-04-09T18:35:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d5d61a700 1 -- 172.21.15.32:0/4206471343 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d5800c620 con 0x7f3d48009590 2021-04-09T18:35:23.400 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d64a1a700 1 -- 172.21.15.32:0/4206471343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d48009590 msgr2=0x7f3d4800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.337 7f3d64a1a700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d48009590 0x7f3d4800bab0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7f3d58000bd0 tx=0x7f3d58009a20).stop 2021-04-09T18:35:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.338 7f3d64a1a700 1 -- 172.21.15.32:0/4206471343 shutdown_connections 2021-04-09T18:35:23.401 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.338 7f3d64a1a700 1 --2- 172.21.15.32:0/4206471343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d48009590 0x7f3d4800bab0 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.338 7f3d64a1a700 1 -- 172.21.15.32:0/4206471343 >> 172.21.15.32:0/4206471343 conn(0x7f3d600ce5b0 msgr2=0x7f3d600cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:23.402 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.338 7f3d64a1a700 1 -- 172.21.15.32:0/4206471343 shutdown_connections 2021-04-09T18:35:23.403 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.338 7f3d64a1a700 1 -- 172.21.15.32:0/4206471343 wait complete. 2021-04-09T18:35:23.405 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.358 7f3d64a1a700 1 Processor -- start 2021-04-09T18:35:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.358 7f3d64a1a700 1 -- start start 2021-04-09T18:35:23.406 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d64a1a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6009d680 0x7f3d6009afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d64a1a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6009da80 0x7f3d60191dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.407 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d64a1a700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d601923f0 0x7f3d6009be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d64a1a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6009cbe0 con 0x7f3d6009da80 2021-04-09T18:35:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d64a1a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6009b8e0 con 0x7f3d6009d680 2021-04-09T18:35:23.408 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d64a1a700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6009b5c0 con 0x7f3d601923f0 2021-04-09T18:35:23.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6009da80 0x7f3d60191dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6009da80 0x7f3d60191dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60476/0 (socket says 172.21.15.32:60476) 2021-04-09T18:35:23.409 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5ee1d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d601923f0 0x7f3d6009be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5de1b700 1 -- 172.21.15.32:0/2930080937 learned_addr learned my addr 172.21.15.32:0/2930080937 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:23.410 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5e61c700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6009d680 0x7f3d6009afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5de1b700 1 -- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d601923f0 msgr2=0x7f3d6009be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.411 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5de1b700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d601923f0 0x7f3d6009be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5de1b700 1 -- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6009d680 msgr2=0x7f3d6009afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5de1b700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6009d680 0x7f3d6009afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5de1b700 1 -- 172.21.15.32:0/2930080937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6009c480 con 0x7f3d6009da80 2021-04-09T18:35:23.412 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5e61c700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6009d680 0x7f3d6009afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:23.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5ee1d700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d601923f0 0x7f3d6009be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:23.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d5de1b700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6009da80 0x7f3d60191dd0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f3d54001fe0 tx=0x7f3d5400ad00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:23.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d4effd700 1 -- 172.21.15.32:0/2930080937 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d540011a0 con 0x7f3d6009da80 2021-04-09T18:35:23.413 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.359 7f3d4effd700 1 -- 172.21.15.32:0/2930080937 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d540070d0 con 0x7f3d6009da80 2021-04-09T18:35:23.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.360 7f3d4effd700 1 -- 172.21.15.32:0/2930080937 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d54004740 con 0x7f3d6009da80 2021-04-09T18:35:23.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.360 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6009c480 con 0x7f3d6009da80 2021-04-09T18:35:23.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.360 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6009c480 con 0x7f3d6009da80 2021-04-09T18:35:23.414 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.360 7f3d4effd700 1 -- 172.21.15.32:0/2930080937 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3d5400ae40 con 0x7f3d6009da80 2021-04-09T18:35:23.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.360 7f3d4effd700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d440188c0 0x7f3d4401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.360 7f3d4effd700 1 -- 172.21.15.32:0/2930080937 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3d54032b60 con 0x7f3d6009da80 2021-04-09T18:35:23.415 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.361 7f3d5e61c700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d440188c0 0x7f3d4401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.416 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.361 7f3d5e61c700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d440188c0 0x7f3d4401adc0 crc :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:23.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.380 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6009b8e0 con 0x7f3d6009da80 2021-04-09T18:35:23.417 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.383 7f3d4effd700 1 -- 172.21.15.32:0/2930080937 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3d54030f40 con 0x7f3d6009da80 2021-04-09T18:35:23.587 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.569 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3d6009b8e0 con 0x7f3d6009da80 2021-04-09T18:35:23.590 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.572 7f3d4effd700 1 -- 172.21.15.32:0/2930080937 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v52) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3d54030f40 con 0x7f3d6009da80 2021-04-09T18:35:23.594 INFO:teuthology.orchestra.run.smithi032.stderr:updated caps for client.1 2021-04-09T18:35:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d440188c0 msgr2=0x7f3d4401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.597 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d440188c0 0x7f3d4401adc0 crc :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6009da80 msgr2=0x7f3d60191dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6009da80 0x7f3d60191dd0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f3d54001fe0 tx=0x7f3d5400ad00).stop 2021-04-09T18:35:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 shutdown_connections 2021-04-09T18:35:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3d440188c0 0x7f3d4401adc0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.598 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d6009da80 0x7f3d60191dd0 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d601923f0 0x7f3d6009be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 --2- 172.21.15.32:0/2930080937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d6009d680 0x7f3d6009afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 >> 172.21.15.32:0/2930080937 conn(0x7f3d600adaf0 msgr2=0x7f3d600bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:23.599 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 shutdown_connections 2021-04-09T18:35:23.600 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.578 7f3d64a1a700 1 -- 172.21.15.32:0/2930080937 wait complete. 2021-04-09T18:35:23.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:35:23.782 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.737 7fca29ae2700 1 Processor -- start 2021-04-09T18:35:23.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.737 7fca29ae2700 1 -- start start 2021-04-09T18:35:23.783 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca29ae2700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca2410d010 0x7fca240d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca29ae2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca240d1da0 0x7fca240d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.784 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca29ae2700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2410d480 con 0x7fca240d82c0 2021-04-09T18:35:23.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca29ae2700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca240d9220 con 0x7fca2410d010 2021-04-09T18:35:23.785 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca29ae2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca240d86c0 con 0x7fca240d1da0 2021-04-09T18:35:23.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca237fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca2410d010 0x7fca240d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca237fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fca2410d010 0x7fca240d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60480/0 (socket says 172.21.15.32:60480) 2021-04-09T18:35:23.786 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca23fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fca240d82c0 0x7fca2410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.32:39766/0 (socket says 172.21.15.32:39766) 2021-04-09T18:35:23.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca237fe700 1 -- 172.21.15.32:0/280478120 learned_addr learned my addr 172.21.15.32:0/280478120 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:23.787 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca22ffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca240d1da0 0x7fca240d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca237fe700 1 -- 172.21.15.32:0/280478120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca240d1da0 msgr2=0x7fca240d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:23.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca237fe700 1 --2- 172.21.15.32:0/280478120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca240d1da0 0x7fca240d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.788 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca227fc700 1 -- 172.21.15.32:0/280478120 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394933507 0 0) 0x7fca1c000d30 con 0x7fca240d82c0 2021-04-09T18:35:23.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca237fe700 1 -- 172.21.15.32:0/280478120 >> v1:172.21.15.86:6789/0 conn(0x7fca240d82c0 legacy=0x7fca2410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.789 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca237fe700 1 -- 172.21.15.32:0/280478120 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2410e1f0 con 0x7fca2410d010 2021-04-09T18:35:23.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca22ffd700 1 --2- 172.21.15.32:0/280478120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca240d1da0 0x7fca240d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:23.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.738 7fca237fe700 1 --2- 172.21.15.32:0/280478120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca2410d010 0x7fca240d1780 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7fca14006700 tx=0x7fca14009b80).ready entity=mon.0 client_cookie=360985267de3eb5b server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:23.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca227fc700 1 -- 172.21.15.32:0/280478120 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca1400c0c0 con 0x7fca2410d010 2021-04-09T18:35:23.790 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca227fc700 1 -- 172.21.15.32:0/280478120 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1400c440 con 0x7fca2410d010 2021-04-09T18:35:23.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca227fc700 1 -- 172.21.15.32:0/280478120 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca1400c920 con 0x7fca2410d010 2021-04-09T18:35:23.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca29ae2700 1 -- 172.21.15.32:0/280478120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca2410d010 msgr2=0x7fca240d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca29ae2700 1 --2- 172.21.15.32:0/280478120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca2410d010 0x7fca240d1780 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7fca14006700 tx=0x7fca14009b80).stop 2021-04-09T18:35:23.791 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca29ae2700 1 -- 172.21.15.32:0/280478120 shutdown_connections 2021-04-09T18:35:23.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca29ae2700 1 --2- 172.21.15.32:0/280478120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca240d1da0 0x7fca240d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca29ae2700 1 --2- 172.21.15.32:0/280478120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca2410d010 0x7fca240d1780 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca29ae2700 1 -- 172.21.15.32:0/280478120 >> 172.21.15.32:0/280478120 conn(0x7fca240ce5f0 msgr2=0x7fca240d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:23.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca29ae2700 1 -- 172.21.15.32:0/280478120 shutdown_connections 2021-04-09T18:35:23.792 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.739 7fca29ae2700 1 -- 172.21.15.32:0/280478120 wait complete. 2021-04-09T18:35:23.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.759 7fca29ae2700 1 Processor -- start 2021-04-09T18:35:23.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca29ae2700 1 -- start start 2021-04-09T18:35:23.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca29ae2700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca24186770 0x7fca24181690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.793 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca29ae2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca24181cb0 0x7fca2408eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca29ae2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2408f0d0 0x7fca241a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca29ae2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca241820b0 con 0x7fca24181cb0 2021-04-09T18:35:23.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca29ae2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca24186b70 con 0x7fca2408f0d0 2021-04-09T18:35:23.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca29ae2700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2408f4d0 con 0x7fca24186770 2021-04-09T18:35:23.794 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca22ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca24181cb0 0x7fca2408eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca22ffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca24181cb0 0x7fca2408eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60488/0 (socket says 172.21.15.32:60488) 2021-04-09T18:35:23.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca22ffd700 1 -- 172.21.15.32:0/2322341249 learned_addr learned my addr 172.21.15.32:0/2322341249 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:23.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca237fe700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca24186770 0x7fca24181690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.795 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.760 7fca23fff700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2408f0d0 0x7fca241a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca22ffd700 1 -- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca24186770 msgr2=0x7fca24181690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca22ffd700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca24186770 0x7fca24181690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca22ffd700 1 -- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2408f0d0 msgr2=0x7fca241a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:23.796 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca22ffd700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2408f0d0 0x7fca241a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:23.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca22ffd700 1 -- 172.21.15.32:0/2322341249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca241a5d00 con 0x7fca24181cb0 2021-04-09T18:35:23.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca22ffd700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca24181cb0 0x7fca2408eab0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7fca18006f40 tx=0x7fca1800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:23.797 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca0bfff700 1 -- 172.21.15.32:0/2322341249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca1800bd80 con 0x7fca24181cb0 2021-04-09T18:35:23.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca0bfff700 1 -- 172.21.15.32:0/2322341249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca18000c90 con 0x7fca24181cb0 2021-04-09T18:35:23.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca0bfff700 1 -- 172.21.15.32:0/2322341249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca18002220 con 0x7fca24181cb0 2021-04-09T18:35:23.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca241a5d00 con 0x7fca24181cb0 2021-04-09T18:35:23.798 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.761 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca241a5d00 con 0x7fca24181cb0 2021-04-09T18:35:23.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.762 7fca0bfff700 1 -- 172.21.15.32:0/2322341249 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fca1800a000 con 0x7fca24181cb0 2021-04-09T18:35:23.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.762 7fca0bfff700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca040188d0 0x7fca0401add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:23.799 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.762 7fca0bfff700 1 -- 172.21.15.32:0/2322341249 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fca18032a10 con 0x7fca24181cb0 2021-04-09T18:35:23.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.762 7fca237fe700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca040188d0 0x7fca0401add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:23.800 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.762 7fca237fe700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca040188d0 0x7fca0401add0 crc :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:23.801 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.782 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca24181160 con 0x7fca24181cb0 2021-04-09T18:35:23.808 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.785 7fca0bfff700 1 -- 172.21.15.32:0/2322341249 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fca18006b00 con 0x7fca24181cb0 2021-04-09T18:35:23.996 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.977 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca24181160 con 0x7fca24181cb0 2021-04-09T18:35:23.997 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.978 7fca0bfff700 1 -- 172.21.15.32:0/2322341249 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 443 v443) v1 ==== 95+0+2789 (secure 0 0 0) 0x7fca1800a000 con 0x7fca24181cb0 2021-04-09T18:35:23.997 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 443 2021-04-09T18:35:23.998 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:23.998 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":443,"default_fscid":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2260217511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2260217511},{"type":"v1","addr":"172.21.15.32:6834","nonce":2260217511}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":13257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1133511394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1133511394},{"type":"v1","addr":"172.21.15.32:6835","nonce":1133511394}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":60194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1457882085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1457882085},{"type":"v1","addr":"172.21.15.86:6837","nonce":1457882085}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439}],"filesystems":[{"mdsmap":{"epoch":443,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:35:19.657739","modified":"2021-04-09 18:35:21.914126","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12706":{"gid":12706,"name":"d","rank":0,"incarnation":440,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3693986652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3693986652},{"type":"v1","addr":"172.21.15.86:6836","nonce":3693986652}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[47],"metadata_pool":46,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":23}]} 2021-04-09T18:35:23.999 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.980 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca040188d0 msgr2=0x7fca0401add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:24.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.980 7fca29ae2700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca040188d0 0x7fca0401add0 crc :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.980 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca24181cb0 msgr2=0x7fca2408eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:24.000 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.980 7fca29ae2700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca24181cb0 0x7fca2408eab0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7fca18006f40 tx=0x7fca1800b990).stop 2021-04-09T18:35:24.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.981 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 shutdown_connections 2021-04-09T18:35:24.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.981 7fca29ae2700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fca040188d0 0x7fca0401add0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.981 7fca29ae2700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca24181cb0 0x7fca2408eab0 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.001 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.981 7fca29ae2700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2408f0d0 0x7fca241a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.981 7fca29ae2700 1 --2- 172.21.15.32:0/2322341249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca24186770 0x7fca24181690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.981 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 >> 172.21.15.32:0/2322341249 conn(0x7fca24195e70 msgr2=0x7fca241a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:24.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.981 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 shutdown_connections 2021-04-09T18:35:24.002 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:23.981 7fca29ae2700 1 -- 172.21.15.32:0/2322341249 wait complete. 2021-04-09T18:35:24.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 443, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:35:19.657739', 'modified': '2021-04-09 18:35:21.914126', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12706}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12706': {'gid': 12706, 'name': 'd', 'rank': 0, 'incarnation': 440, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3693986652', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3693986652}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3693986652}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [47], 'metadata_pool': 46, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:35:24.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:35:24.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:35:24.542 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:35:24.542 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:35:24.543 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:35:24.543 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 12706, 2021-04-09T18:35:24.543 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:35:24.543 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:35:24.544 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 4.8643482809999998, 2021-04-09T18:35:24.544 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 443, 2021-04-09T18:35:24.544 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 278, 2021-04-09T18:35:24.545 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 278, 2021-04-09T18:35:24.545 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10.309286343 2021-04-09T18:35:24.545 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:35:24.548 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12706, "want_state": "up:active", "state": "up:active", "rank_uptime": 4.8643482809999998, "mdsmap_epoch": 443, "osdmap_epoch": 278, "osdmap_epoch_barrier": 278, "uptime": 10.309286343 } 2021-04-09T18:35:24.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-04-09T18:35:24.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.687 7f75ed9e8700 1 Processor -- start 2021-04-09T18:35:24.728 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.687 7f75ed9e8700 1 -- start start 2021-04-09T18:35:24.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75ed9e8700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75e810d010 0x7f75e80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:24.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75ed9e8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80d1da0 0x7f75e80d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:24.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75ed9e8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e810d480 con 0x7f75e80d82c0 2021-04-09T18:35:24.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75ed9e8700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e80d9220 con 0x7f75e810d010 2021-04-09T18:35:24.729 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75ed9e8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e80d86c0 con 0x7f75e80d1da0 2021-04-09T18:35:24.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75e77fe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f75e810d010 0x7f75e80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:24.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75e7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f75e80d82c0 0x7f75e810da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41854/0 (socket says 172.21.15.32:41854) 2021-04-09T18:35:24.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75e7fff700 1 -- 172.21.15.32:0/2152962263 learned_addr learned my addr 172.21.15.32:0/2152962263 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:24.730 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75e6ffd700 1 --2- 172.21.15.32:0/2152962263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80d1da0 0x7f75e80d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:24.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75e67fc700 1 -- 172.21.15.32:0/2152962263 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719731852 0 0) 0x7f75e00009b0 con 0x7f75e80d82c0 2021-04-09T18:35:24.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.688 7f75e67fc700 1 -- 172.21.15.32:0/2152962263 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d0002fe0 con 0x7f75e80d82c0 2021-04-09T18:35:24.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75e77fe700 1 -- 172.21.15.32:0/2152962263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80d1da0 msgr2=0x7f75e80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:24.731 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75e77fe700 1 --2- 172.21.15.32:0/2152962263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80d1da0 0x7f75e80d7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75e77fe700 1 -- 172.21.15.32:0/2152962263 >> v1:172.21.15.32:6789/0 conn(0x7f75e80d82c0 legacy=0x7f75e810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:24.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75e77fe700 1 -- 172.21.15.32:0/2152962263 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e810e1f0 con 0x7f75e810d010 2021-04-09T18:35:24.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75e77fe700 1 --2- 172.21.15.32:0/2152962263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e810d010 0x7f75e80d1780 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f75d80069d0 tx=0x7f75d8009dc0).ready entity=mon.0 client_cookie=b4be4d0ba071ee69 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:24.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75e67fc700 1 -- 172.21.15.32:0/2152962263 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75d800b510 con 0x7f75e810d010 2021-04-09T18:35:24.732 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75e67fc700 1 -- 172.21.15.32:0/2152962263 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d8006ac0 con 0x7f75e810d010 2021-04-09T18:35:24.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75e67fc700 1 -- 172.21.15.32:0/2152962263 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75d800ba80 con 0x7f75e810d010 2021-04-09T18:35:24.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75ed9e8700 1 -- 172.21.15.32:0/2152962263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e810d010 msgr2=0x7f75e80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:24.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.689 7f75ed9e8700 1 --2- 172.21.15.32:0/2152962263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e810d010 0x7f75e80d1780 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f75d80069d0 tx=0x7f75d8009dc0).stop 2021-04-09T18:35:24.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.690 7f75ed9e8700 1 -- 172.21.15.32:0/2152962263 shutdown_connections 2021-04-09T18:35:24.733 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.690 7f75ed9e8700 1 --2- 172.21.15.32:0/2152962263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e80d1da0 0x7f75e80d7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.690 7f75ed9e8700 1 --2- 172.21.15.32:0/2152962263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e810d010 0x7f75e80d1780 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.690 7f75ed9e8700 1 -- 172.21.15.32:0/2152962263 >> 172.21.15.32:0/2152962263 conn(0x7f75e80ce5f0 msgr2=0x7f75e80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:24.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.690 7f75ed9e8700 1 -- 172.21.15.32:0/2152962263 shutdown_connections 2021-04-09T18:35:24.734 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.690 7f75ed9e8700 1 -- 172.21.15.32:0/2152962263 wait complete. 2021-04-09T18:35:24.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.710 7f75ed9e8700 1 Processor -- start 2021-04-09T18:35:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.710 7f75ed9e8700 1 -- start start 2021-04-09T18:35:24.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.710 7f75ed9e8700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81ac380 0x7f75e818a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.710 7f75ed9e8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e81ac780 0x7f75e818a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.710 7f75ed9e8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8183520 0x7f75e8183920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.710 7f75ed9e8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e81ace10 con 0x7f75e8183520 2021-04-09T18:35:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.710 7f75ed9e8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e8189cf0 con 0x7f75e81ac780 2021-04-09T18:35:24.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75ed9e8700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e8183f40 con 0x7f75e81ac380 2021-04-09T18:35:24.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8183520 0x7f75e8183920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:24.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e7fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8183520 0x7f75e8183920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60504/0 (socket says 172.21.15.32:60504) 2021-04-09T18:35:24.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e7fff700 1 -- 172.21.15.32:0/158447862 learned_addr learned my addr 172.21.15.32:0/158447862 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:24.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e77fe700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81ac380 0x7f75e818a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e6ffd700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e81ac780 0x7f75e818a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e7fff700 1 -- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81ac380 msgr2=0x7f75e818a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e7fff700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81ac380 0x7f75e818a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e7fff700 1 -- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e81ac780 msgr2=0x7f75e818a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e7fff700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e81ac780 0x7f75e818a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e7fff700 1 -- 172.21.15.32:0/158447862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e8184160 con 0x7f75e8183520 2021-04-09T18:35:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75e7fff700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8183520 0x7f75e8183920 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f75e00014f0 tx=0x7f75e000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75cffff700 1 -- 172.21.15.32:0/158447862 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75e000ace0 con 0x7f75e8183520 2021-04-09T18:35:24.742 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75cffff700 1 -- 172.21.15.32:0/158447862 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e00043a0 con 0x7f75e8183520 2021-04-09T18:35:24.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e8184160 con 0x7f75e8183520 2021-04-09T18:35:24.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75cffff700 1 -- 172.21.15.32:0/158447862 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75e0004830 con 0x7f75e8183520 2021-04-09T18:35:24.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.711 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e8184160 con 0x7f75e8183520 2021-04-09T18:35:24.743 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.712 7f75cffff700 1 -- 172.21.15.32:0/158447862 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f75e0000ef0 con 0x7f75e8183520 2021-04-09T18:35:24.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.712 7f75cffff700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75c8018930 0x7f75c801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:24.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.712 7f75cffff700 1 -- 172.21.15.32:0/158447862 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f75e00329c0 con 0x7f75e8183520 2021-04-09T18:35:24.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.712 7f75e77fe700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75c8018930 0x7f75c801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:24.744 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.712 7f75e77fe700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75c8018930 0x7f75c801ae10 crc :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:24.750 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.732 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e8189cf0 con 0x7f75e8183520 2021-04-09T18:35:24.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.735 7f75cffff700 1 -- 172.21.15.32:0/158447862 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f75e0004f00 con 0x7f75e8183520 2021-04-09T18:35:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.926 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75e8189cf0 con 0x7f75e8183520 2021-04-09T18:35:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.927 7f75cffff700 1 -- 172.21.15.32:0/158447862 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v278) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f75e0000ef0 con 0x7f75e8183520 2021-04-09T18:35:24.945 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:24.946 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":278,"fsid":"cf69fbbc-37e8-4833-99b4-5ecbe4444cfb","created":"2021-04-09 18:12:12.512919","modified":"2021-04-09 18:35:20.659017","last_up_change":"2021-04-09 18:12:24.480980","last_in_change":"2021-04-09 18:12:24.480980","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":47,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-04-09 18:12:28.923594","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":46,"pool_name":"cephfs_metadata","create_time":"2021-04-09 18:35:16.949280","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"277","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":47,"pool_name":"cephfs_data","create_time":"2021-04-09 18:35:18.370170","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"278","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f892062-dd3f-413b-bbd2-6cfd9d640f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":24886},{"type":"v1","addr":"172.21.15.32:6801","nonce":24886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":24886},{"type":"v1","addr":"172.21.15.32:6803","nonce":24886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":24886},{"type":"v1","addr":"172.21.15.32:6807","nonce":24886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":24886},{"type":"v1","addr":"172.21.15.32:6805","nonce":24886}]},"public_addr":"172.21.15.32:6801/24886","cluster_addr":"172.21.15.32:6803/24886","heartbeat_back_addr":"172.21.15.32:6807/24886","heartbeat_front_addr":"172.21.15.32:6805/24886","state":["exists","up"]},{"osd":1,"uuid":"5f094ec4-4fc7-4ed9-a278-f51fab1f1727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":24901},{"type":"v1","addr":"172.21.15.32:6809","nonce":24901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":24901},{"type":"v1","addr":"172.21.15.32:6811","nonce":24901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":24901},{"type":"v1","addr":"172.21.15.32:6815","nonce":24901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":24901},{"type":"v1","addr":"172.21.15.32:6813","nonce":24901}]},"public_addr":"172.21.15.32:6809/24901","cluster_addr":"172.21.15.32:6811/24901","heartbeat_back_addr":"172.21.15.32:6815/24901","heartbeat_front_addr":"172.21.15.32:6813/24901","state":["exists","up"]},{"osd":2,"uuid":"7a25e453-ac85-4d4e-8a06-5a18f990be19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":24934},{"type":"v1","addr":"172.21.15.32:6817","nonce":24934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":24934},{"type":"v1","addr":"172.21.15.32:6819","nonce":24934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":24934},{"type":"v1","addr":"172.21.15.32:6823","nonce":24934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":24934},{"type":"v1","addr":"172.21.15.32:6821","nonce":24934}]},"public_addr":"172.21.15.32:6817/24934","cluster_addr":"172.21.15.32:6819/24934","heartbeat_back_addr":"172.21.15.32:6823/24934","heartbeat_front_addr":"172.21.15.32:6821/24934","state":["exists","up"]},{"osd":3,"uuid":"fc7ef9cf-5cd2-4fd2-8112-245a5141a327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":24942},{"type":"v1","addr":"172.21.15.32:6825","nonce":24942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":24942},{"type":"v1","addr":"172.21.15.32:6827","nonce":24942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":24942},{"type":"v1","addr":"172.21.15.32:6831","nonce":24942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":24942},{"type":"v1","addr":"172.21.15.32:6829","nonce":24942}]},"public_addr":"172.21.15.32:6825/24942","cluster_addr":"172.21.15.32:6827/24942","heartbeat_back_addr":"172.21.15.32:6831/24942","heartbeat_front_addr":"172.21.15.32:6829/24942","state":["exists","up"]},{"osd":4,"uuid":"122503dc-3cb6-4959-b287-cbae5ce53e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":25232},{"type":"v1","addr":"172.21.15.86:6803","nonce":25232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":25232},{"type":"v1","addr":"172.21.15.86:6805","nonce":25232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":25232},{"type":"v1","addr":"172.21.15.86:6809","nonce":25232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":25232},{"type":"v1","addr":"172.21.15.86:6807","nonce":25232}]},"public_addr":"172.21.15.86:6803/25232","cluster_addr":"172.21.15.86:6805/25232","heartbeat_back_addr":"172.21.15.86:6809/25232","heartbeat_front_addr":"172.21.15.86:6807/25232","state":["exists","up"]},{"osd":5,"uuid":"097269c3-c880-4660-a0c2-12cd6fc7bcf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":25237},{"type":"v1","addr":"172.21.15.86:6811","nonce":25237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":25237},{"type":"v1","addr":"172.21.15.86:6813","nonce":25237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":25237},{"type":"v1","addr":"172.21.15.86:6817","nonce":25237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":25237},{"type":"v1","addr":"172.21.15.86:6815","nonce":25237}]},"public_addr":"172.21.15.86:6811/25237","cluster_addr":"172.21.15.86:6813/25237","heartbeat_back_addr":"172.21.15.86:6817/25237","heartbeat_front_addr":"172.21.15.86:6815/25237","state":["exists","up"]},{"osd":6,"uuid":"2a3e3f80-b45d-4425-8527-c07a38cea770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":271,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":25238},{"type":"v1","addr":"172.21.15.86:6819","nonce":25238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":25238},{"type":"v1","addr":"172.21.15.86:6821","nonce":25238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":25238},{"type":"v1","addr":"172.21.15.86:6825","nonce":25238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":25238},{"type":"v1","addr":"172.21.15.86:6823","nonce":25238}]},"public_addr":"172.21.15.86:6819/25238","cluster_addr":"172.21.15.86:6821/25238","heartbeat_back_addr":"172.21.15.86:6825/25238","heartbeat_front_addr":"172.21.15.86:6823/25238","state":["exists","up"]},{"osd":7,"uuid":"526aa5a2-5c73-4473-a78b-240711b2015c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":275,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":25248},{"type":"v1","addr":"172.21.15.86:6827","nonce":25248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":25248},{"type":"v1","addr":"172.21.15.86:6829","nonce":25248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":25248},{"type":"v1","addr":"172.21.15.86:6833","nonce":25248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":25248},{"type":"v1","addr":"172.21.15.86:6831","nonce":25248}]},"public_addr":"172.21.15.86:6827/25248","cluster_addr":"172.21.15.86:6829/25248","heartbeat_back_addr":"172.21.15.86:6833/25248","heartbeat_front_addr":"172.21.15.86:6831/25248","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-04-09T18:35:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75c8018930 msgr2=0x7f75c801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75c8018930 0x7f75c801ae10 crc :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8183520 msgr2=0x7f75e8183920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8183520 0x7f75e8183920 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f75e00014f0 tx=0x7f75e000a8f0).stop 2021-04-09T18:35:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 shutdown_connections 2021-04-09T18:35:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f75c8018930 0x7f75c801ae10 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75e8183520 0x7f75e8183920 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e81ac780 0x7f75e818a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 --2- 172.21.15.32:0/158447862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e81ac380 0x7f75e818a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 >> 172.21.15.32:0/158447862 conn(0x7f75e819b890 msgr2=0x7f75e81a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 shutdown_connections 2021-04-09T18:35:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:24.929 7f75ed9e8700 1 -- 172.21.15.32:0/158447862 wait complete. 2021-04-09T18:35:24.958 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-04-09 18:12:28.923594', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 46, 'pool_name': 'cephfs_metadata', 'create_time': '2021-04-09 18:35:16.949280', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '277', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 47, 'pool_name': 'cephfs_data', 'create_time': '2021-04-09 18:35:18.370170', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '278', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-04-09T18:35:24.959 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-04-09T18:35:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.094 7fed89714700 1 Processor -- start 2021-04-09T18:35:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.094 7fed89714700 1 -- start start 2021-04-09T18:35:25.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed89714700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed8410d010 0x7fed840d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed89714700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8410d480 con 0x7fed840d8490 2021-04-09T18:35:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed89714700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8410e1f0 con 0x7fed840d1da0 2021-04-09T18:35:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed89714700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8410e780 con 0x7fed8410d010 2021-04-09T18:35:25.136 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed83ac0700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fed840d8490 0x7fed8410da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41868/0 (socket says 172.21.15.32:41868) 2021-04-09T18:35:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed83ac0700 1 -- 172.21.15.32:0/176773302 learned_addr learned my addr 172.21.15.32:0/176773302 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed832bf700 1 --2- 172.21.15.32:0/176773302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed8410d010 0x7fed840d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed822bd700 1 -- 172.21.15.32:0/176773302 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191577362 0 0) 0x7fed7c000d30 con 0x7fed840d8490 2021-04-09T18:35:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed822bd700 1 -- 172.21.15.32:0/176773302 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed6c002fe0 con 0x7fed840d8490 2021-04-09T18:35:25.137 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed822bd700 1 -- 172.21.15.32:0/176773302 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230463775 0 0) 0x7fed78000c60 con 0x7fed840d1da0 2021-04-09T18:35:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed822bd700 1 -- 172.21.15.32:0/176773302 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed6c0042c0 con 0x7fed840d1da0 2021-04-09T18:35:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed832bf700 1 -- 172.21.15.32:0/176773302 >> v1:172.21.15.86:6789/0 conn(0x7fed840d1da0 legacy=0x7fed840d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.095 7fed832bf700 1 -- 172.21.15.32:0/176773302 >> v1:172.21.15.32:6789/0 conn(0x7fed840d8490 legacy=0x7fed8410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.096 7fed832bf700 1 -- 172.21.15.32:0/176773302 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed840d8d30 con 0x7fed8410d010 2021-04-09T18:35:25.138 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.096 7fed832bf700 1 --2- 172.21.15.32:0/176773302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed8410d010 0x7fed840d1780 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fed74006650 tx=0x7fed74009a80).ready entity=mon.2 client_cookie=d5358d3308bf54ee server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:25.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.096 7fed822bd700 1 -- 172.21.15.32:0/176773302 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed740009b0 con 0x7fed8410d010 2021-04-09T18:35:25.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.096 7fed822bd700 1 -- 172.21.15.32:0/176773302 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7400c0c0 con 0x7fed8410d010 2021-04-09T18:35:25.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.096 7fed822bd700 1 -- 172.21.15.32:0/176773302 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed7400c510 con 0x7fed8410d010 2021-04-09T18:35:25.139 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.096 7fed89714700 1 -- 172.21.15.32:0/176773302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed8410d010 msgr2=0x7fed840d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.096 7fed89714700 1 --2- 172.21.15.32:0/176773302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed8410d010 0x7fed840d1780 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fed74006650 tx=0x7fed74009a80).stop 2021-04-09T18:35:25.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.097 7fed89714700 1 -- 172.21.15.32:0/176773302 shutdown_connections 2021-04-09T18:35:25.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.097 7fed89714700 1 --2- 172.21.15.32:0/176773302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed8410d010 0x7fed840d1780 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.097 7fed89714700 1 -- 172.21.15.32:0/176773302 >> 172.21.15.32:0/176773302 conn(0x7fed840ce5f0 msgr2=0x7fed840d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:25.140 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.097 7fed89714700 1 -- 172.21.15.32:0/176773302 shutdown_connections 2021-04-09T18:35:25.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.097 7fed89714700 1 -- 172.21.15.32:0/176773302 wait complete. 2021-04-09T18:35:25.141 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed89714700 1 Processor -- start 2021-04-09T18:35:25.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed89714700 1 -- start start 2021-04-09T18:35:25.145 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed89714700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed84179610 0x7fed84179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed89714700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed8417a820 0x7fed8417ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed89714700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed841949f0 0x7fed84194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed89714700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed841b8d50 con 0x7fed8417a820 2021-04-09T18:35:25.146 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed89714700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8417b240 con 0x7fed841949f0 2021-04-09T18:35:25.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed89714700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed84195410 con 0x7fed84179610 2021-04-09T18:35:25.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed82abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed8417a820 0x7fed8417ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed82abe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed8417a820 0x7fed8417ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60514/0 (socket says 172.21.15.32:60514) 2021-04-09T18:35:25.147 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.117 7fed82abe700 1 -- 172.21.15.32:0/504047499 learned_addr learned my addr 172.21.15.32:0/504047499 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:25.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed83ac0700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed841949f0 0x7fed84194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed832bf700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed84179610 0x7fed84179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed82abe700 1 -- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed84179610 msgr2=0x7fed84179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.148 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed82abe700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed84179610 0x7fed84179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed82abe700 1 -- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed841949f0 msgr2=0x7fed84194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed82abe700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed841949f0 0x7fed84194df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed82abe700 1 -- 172.21.15.32:0/504047499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84195630 con 0x7fed8417a820 2021-04-09T18:35:25.149 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed82abe700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed8417a820 0x7fed8417ac20 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7fed78006f10 tx=0x7fed7800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:25.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed6bfff700 1 -- 172.21.15.32:0/504047499 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed7800bd80 con 0x7fed8417a820 2021-04-09T18:35:25.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed6bfff700 1 -- 172.21.15.32:0/504047499 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed78000e20 con 0x7fed8417a820 2021-04-09T18:35:25.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed89714700 1 -- 172.21.15.32:0/504047499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed84195630 con 0x7fed8417a820 2021-04-09T18:35:25.150 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed6bfff700 1 -- 172.21.15.32:0/504047499 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed780054f0 con 0x7fed8417a820 2021-04-09T18:35:25.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.118 7fed89714700 1 -- 172.21.15.32:0/504047499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed84195630 con 0x7fed8417a820 2021-04-09T18:35:25.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.119 7fed6bfff700 1 -- 172.21.15.32:0/504047499 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fed78009fe0 con 0x7fed8417a820 2021-04-09T18:35:25.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.119 7fed6bfff700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.151 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.119 7fed6bfff700 1 -- 172.21.15.32:0/504047499 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fed78032a10 con 0x7fed8417a820 2021-04-09T18:35:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.119 7fed832bf700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.152 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.120 7fed832bf700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401adc0 crc :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:25.157 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.139 7fed89714700 1 -- 172.21.15.32:0/504047499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8417a2f0 con 0x7fed8417a820 2021-04-09T18:35:25.161 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.143 7fed6bfff700 1 -- 172.21.15.32:0/504047499 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fed78000e20 con 0x7fed8417a820 2021-04-09T18:35:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.321 7fed89714700 1 -- 172.21.15.32:0/504047499 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fed8417a2f0 con 0x7fed8417a820 2021-04-09T18:35:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.322 7fed6bfff700 1 -- 172.21.15.32:0/504047499 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v278) v1 ==== 92+0+10 (secure 0 0 0) 0x7fed78009fe0 con 0x7fed8417a820 2021-04-09T18:35:25.341 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:35:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 -- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 msgr2=0x7fed6401adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401adc0 crc :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 -- 172.21.15.32:0/504047499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed8417a820 msgr2=0x7fed8417ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed8417a820 0x7fed8417ac20 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7fed78006f10 tx=0x7fed7800b990).stop 2021-04-09T18:35:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 -- 172.21.15.32:0/504047499 shutdown_connections 2021-04-09T18:35:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fed64018930 0x7fed6401adc0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed8417a820 0x7fed8417ac20 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed841949f0 0x7fed84194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 --2- 172.21.15.32:0/504047499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed84179610 0x7fed84179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.324 7fed89714700 1 -- 172.21.15.32:0/504047499 >> 172.21.15.32:0/504047499 conn(0x7fed84092a00 msgr2=0x7fed841b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.325 7fed89714700 1 -- 172.21.15.32:0/504047499 shutdown_connections 2021-04-09T18:35:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.325 7fed89714700 1 -- 172.21.15.32:0/504047499 wait complete. 2021-04-09T18:35:25.354 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-04-09T18:35:25.528 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.488 7fcbf0325700 1 Processor -- start 2021-04-09T18:35:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.488 7fcbf0325700 1 -- start start 2021-04-09T18:35:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.488 7fcbf0325700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcbe810d010 0x7fcbe80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbf0325700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe810d480 con 0x7fcbe80d1da0 2021-04-09T18:35:25.529 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbf0325700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe810e1f0 con 0x7fcbe80d8490 2021-04-09T18:35:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbf0325700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe80d8d30 con 0x7fcbe810d010 2021-04-09T18:35:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbed940700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcbe80d1da0 0x7fcbe80d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41884/0 (socket says 172.21.15.32:41884) 2021-04-09T18:35:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbed940700 1 -- 172.21.15.32:0/2343533001 learned_addr learned my addr 172.21.15.32:0/2343533001 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:25.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbee141700 1 --2- 172.21.15.32:0/2343533001 >> v2:172.21.15.86:3300/0 conn(0x7fcbe810d010 0x7fcbe80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbed13f700 1 -- 172.21.15.32:0/2343533001 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024899307 0 0) 0x7fcbd80009b0 con 0x7fcbe80d1da0 2021-04-09T18:35:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbed13f700 1 -- 172.21.15.32:0/2343533001 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd4002fe0 con 0x7fcbe80d1da0 2021-04-09T18:35:25.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbed13f700 1 -- 172.21.15.32:0/2343533001 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610177749 0 0) 0x7fcbe0000d30 con 0x7fcbe80d8490 2021-04-09T18:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbed13f700 1 -- 172.21.15.32:0/2343533001 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd4002fe0 con 0x7fcbe80d8490 2021-04-09T18:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbee141700 1 -- 172.21.15.32:0/2343533001 >> v1:172.21.15.86:6789/0 conn(0x7fcbe80d8490 legacy=0x7fcbe810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbee141700 1 -- 172.21.15.32:0/2343533001 >> v1:172.21.15.32:6789/0 conn(0x7fcbe80d1da0 legacy=0x7fcbe80d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.532 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbee141700 1 -- 172.21.15.32:0/2343533001 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80d9220 con 0x7fcbe810d010 2021-04-09T18:35:25.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.489 7fcbee141700 1 --2- 172.21.15.32:0/2343533001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe810d010 0x7fcbe80d1780 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7fcbe4006650 tx=0x7fcbe4009a80).ready entity=mon.1 client_cookie=8650d33eaf7860e2 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:25.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbed13f700 1 -- 172.21.15.32:0/2343533001 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbe40009b0 con 0x7fcbe810d010 2021-04-09T18:35:25.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbed13f700 1 -- 172.21.15.32:0/2343533001 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe400c0c0 con 0x7fcbe810d010 2021-04-09T18:35:25.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbed13f700 1 -- 172.21.15.32:0/2343533001 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbe400c510 con 0x7fcbe810d010 2021-04-09T18:35:25.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbf0325700 1 -- 172.21.15.32:0/2343533001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe810d010 msgr2=0x7fcbe80d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbf0325700 1 --2- 172.21.15.32:0/2343533001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe810d010 0x7fcbe80d1780 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7fcbe4006650 tx=0x7fcbe4009a80).stop 2021-04-09T18:35:25.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbf0325700 1 -- 172.21.15.32:0/2343533001 shutdown_connections 2021-04-09T18:35:25.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbf0325700 1 --2- 172.21.15.32:0/2343533001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe810d010 0x7fcbe80d1780 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbf0325700 1 -- 172.21.15.32:0/2343533001 >> 172.21.15.32:0/2343533001 conn(0x7fcbe80ce5f0 msgr2=0x7fcbe80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:25.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbf0325700 1 -- 172.21.15.32:0/2343533001 shutdown_connections 2021-04-09T18:35:25.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.490 7fcbf0325700 1 -- 172.21.15.32:0/2343533001 wait complete. 2021-04-09T18:35:25.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.510 7fcbf0325700 1 Processor -- start 2021-04-09T18:35:25.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbf0325700 1 -- start start 2021-04-09T18:35:25.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbf0325700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe80b8de0 0x7fcbe80c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbf0325700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe80c0640 0x7fcbe81936d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbf0325700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbe8193cf0 0x7fcbe81940f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbf0325700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80c0a40 con 0x7fcbe8193cf0 2021-04-09T18:35:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbf0325700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80b89a0 con 0x7fcbe80c0640 2021-04-09T18:35:25.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbf0325700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81b7b70 con 0x7fcbe80b8de0 2021-04-09T18:35:25.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbee942700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbe8193cf0 0x7fcbe81940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbee942700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbe8193cf0 0x7fcbe81940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60532/0 (socket says 172.21.15.32:60532) 2021-04-09T18:35:25.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbee942700 1 -- 172.21.15.32:0/13015623 learned_addr learned my addr 172.21.15.32:0/13015623 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:25.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbed940700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe80c0640 0x7fcbe81936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.511 7fcbee141700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe80b8de0 0x7fcbe80c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbee942700 1 -- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe80b8de0 msgr2=0x7fcbe80c0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbee942700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe80b8de0 0x7fcbe80c0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbee942700 1 -- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe80c0640 msgr2=0x7fcbe81936d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbee942700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe80c0640 0x7fcbe81936d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbee942700 1 -- 172.21.15.32:0/13015623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80bb2d0 con 0x7fcbe8193cf0 2021-04-09T18:35:25.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbee942700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbe8193cf0 0x7fcbe81940f0 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7fcbe0009980 tx=0x7fcbe000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:25.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbdeffd700 1 -- 172.21.15.32:0/13015623 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbe000be10 con 0x7fcbe8193cf0 2021-04-09T18:35:25.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbdeffd700 1 -- 172.21.15.32:0/13015623 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe0005870 con 0x7fcbe8193cf0 2021-04-09T18:35:25.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbf0325700 1 -- 172.21.15.32:0/13015623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe80bb2d0 con 0x7fcbe8193cf0 2021-04-09T18:35:25.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbdeffd700 1 -- 172.21.15.32:0/13015623 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbe0005d50 con 0x7fcbe8193cf0 2021-04-09T18:35:25.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbf0325700 1 -- 172.21.15.32:0/13015623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe80bb2d0 con 0x7fcbe8193cf0 2021-04-09T18:35:25.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.512 7fcbdeffd700 1 -- 172.21.15.32:0/13015623 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fcbe0005a50 con 0x7fcbe8193cf0 2021-04-09T18:35:25.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.513 7fcbdeffd700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbd0018930 0x7fcbd001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.513 7fcbdeffd700 1 -- 172.21.15.32:0/13015623 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fcbe0032d80 con 0x7fcbe8193cf0 2021-04-09T18:35:25.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.513 7fcbee141700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbd0018930 0x7fcbd001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.513 7fcbee141700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbd0018930 0x7fcbd001ae10 crc :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:25.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.533 7fcbf0325700 1 -- 172.21.15.32:0/13015623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe80b89a0 con 0x7fcbe8193cf0 2021-04-09T18:35:25.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.536 7fcbdeffd700 1 -- 172.21.15.32:0/13015623 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fcbe000be10 con 0x7fcbe8193cf0 2021-04-09T18:35:25.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.716 7fcbf0325700 1 -- 172.21.15.32:0/13015623 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fcbe80b89a0 con 0x7fcbe8193cf0 2021-04-09T18:35:25.735 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.717 7fcbdeffd700 1 -- 172.21.15.32:0/13015623 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v278) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcbe0006f20 con 0x7fcbe8193cf0 2021-04-09T18:35:25.736 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:35:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 -- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbd0018930 msgr2=0x7fcbd001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.738 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbd0018930 0x7fcbd001ae10 crc :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 -- 172.21.15.32:0/13015623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbe8193cf0 msgr2=0x7fcbe81940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbe8193cf0 0x7fcbe81940f0 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7fcbe0009980 tx=0x7fcbe000bcd0).stop 2021-04-09T18:35:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 -- 172.21.15.32:0/13015623 shutdown_connections 2021-04-09T18:35:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fcbd0018930 0x7fcbd001ae10 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcbe8193cf0 0x7fcbe81940f0 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe80c0640 0x7fcbe81936d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 --2- 172.21.15.32:0/13015623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe80b8de0 0x7fcbe80c0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 -- 172.21.15.32:0/13015623 >> 172.21.15.32:0/13015623 conn(0x7fcbe8092a00 msgr2=0x7fcbe81b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 -- 172.21.15.32:0/13015623 shutdown_connections 2021-04-09T18:35:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.719 7fcbf0325700 1 -- 172.21.15.32:0/13015623 wait complete. 2021-04-09T18:35:25.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-04-09T18:35:25.926 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.882 7f7ad1260700 1 Processor -- start 2021-04-09T18:35:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.882 7f7ad1260700 1 -- start start 2021-04-09T18:35:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.883 7f7ad1260700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7acc0d6ca0 0x7f7acc0d8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.883 7f7ad1260700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc0d2460 0x7f7acc10cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.927 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.883 7f7ad1260700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc0d72a0 con 0x7f7acc0cffc0 2021-04-09T18:35:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.883 7f7ad1260700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7acc0d70a0 con 0x7f7acc0d6ca0 2021-04-09T18:35:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.883 7f7ad1260700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0d9200 con 0x7f7acc0d2460 2021-04-09T18:35:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.883 7f7acad9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7acc0cffc0 0x7f7acc0d7820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41894/0 (socket says 172.21.15.32:41894) 2021-04-09T18:35:25.928 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.883 7f7acad9d700 1 -- 172.21.15.32:0/3597596369 learned_addr learned my addr 172.21.15.32:0/3597596369 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.883 7f7aca59c700 1 --2- 172.21.15.32:0/3597596369 >> v2:172.21.15.86:3300/0 conn(0x7f7acc0d6ca0 0x7f7acc0d8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7acb59e700 1 --2- 172.21.15.32:0/3597596369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc0d2460 0x7f7acc10cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7ac9d9b700 1 -- 172.21.15.32:0/3597596369 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816424933 0 0) 0x7f7abc000ce0 con 0x7f7acc0cffc0 2021-04-09T18:35:25.929 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7ac9d9b700 1 -- 172.21.15.32:0/3597596369 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab4002fe0 con 0x7f7acc0cffc0 2021-04-09T18:35:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7ac9d9b700 1 -- 172.21.15.32:0/3597596369 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3095956153 0 0) 0x7f7abc0010e0 con 0x7f7acc0cffc0 2021-04-09T18:35:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7aca59c700 1 -- 172.21.15.32:0/3597596369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc0d2460 msgr2=0x7f7acc10cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7aca59c700 1 --2- 172.21.15.32:0/3597596369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc0d2460 0x7f7acc10cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.930 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7aca59c700 1 -- 172.21.15.32:0/3597596369 >> v1:172.21.15.32:6789/0 conn(0x7f7acc0cffc0 legacy=0x7f7acc0d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7aca59c700 1 -- 172.21.15.32:0/3597596369 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0d2860 con 0x7f7acc0d6ca0 2021-04-09T18:35:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7acb59e700 1 --2- 172.21.15.32:0/3597596369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc0d2460 0x7f7acc10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7aca59c700 1 --2- 172.21.15.32:0/3597596369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7acc0d6ca0 0x7f7acc0d8010 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f7ac00069d0 tx=0x7f7ac0009e40).ready entity=mon.1 client_cookie=669df721ff218755 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:25.931 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7ac9d9b700 1 -- 172.21.15.32:0/3597596369 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac000b510 con 0x7f7acc0d6ca0 2021-04-09T18:35:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7ac9d9b700 1 -- 172.21.15.32:0/3597596369 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac000b890 con 0x7f7acc0d6ca0 2021-04-09T18:35:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7ac9d9b700 1 -- 172.21.15.32:0/3597596369 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac000bd70 con 0x7f7acc0d6ca0 2021-04-09T18:35:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7ad1260700 1 -- 172.21.15.32:0/3597596369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7acc0d6ca0 msgr2=0x7f7acc0d8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.932 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.884 7f7ad1260700 1 --2- 172.21.15.32:0/3597596369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7acc0d6ca0 0x7f7acc0d8010 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f7ac00069d0 tx=0x7f7ac0009e40).stop 2021-04-09T18:35:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.885 7f7ad1260700 1 -- 172.21.15.32:0/3597596369 shutdown_connections 2021-04-09T18:35:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.885 7f7ad1260700 1 --2- 172.21.15.32:0/3597596369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc0d2460 0x7f7acc10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.933 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.885 7f7ad1260700 1 --2- 172.21.15.32:0/3597596369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7acc0d6ca0 0x7f7acc0d8010 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.885 7f7ad1260700 1 -- 172.21.15.32:0/3597596369 >> 172.21.15.32:0/3597596369 conn(0x7f7acc0ce5e0 msgr2=0x7f7acc0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.885 7f7ad1260700 1 -- 172.21.15.32:0/3597596369 shutdown_connections 2021-04-09T18:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.885 7f7ad1260700 1 -- 172.21.15.32:0/3597596369 wait complete. 2021-04-09T18:35:25.934 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.905 7f7ad1260700 1 Processor -- start 2021-04-09T18:35:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.905 7f7ad1260700 1 -- start start 2021-04-09T18:35:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7ad1260700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc09ebf0 0x7f7acc1883d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7ad1260700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7acc09adf0 0x7f7acc09b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.935 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7ad1260700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc09bee0 0x7f7acc09c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7ad1260700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc187fc0 con 0x7f7acc09ebf0 2021-04-09T18:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7ad1260700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc1889f0 con 0x7f7acc09adf0 2021-04-09T18:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7ad1260700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc09c900 con 0x7f7acc09bee0 2021-04-09T18:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7acad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc09ebf0 0x7f7acc1883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.936 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7acad9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc09ebf0 0x7f7acc1883d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60542/0 (socket says 172.21.15.32:60542) 2021-04-09T18:35:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7acad9d700 1 -- 172.21.15.32:0/3180603831 learned_addr learned my addr 172.21.15.32:0/3180603831 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7aca59c700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7acc09adf0 0x7f7acc09b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7acb59e700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc09bee0 0x7f7acc09c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.937 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7acad9d700 1 -- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc09bee0 msgr2=0x7f7acc09c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7acad9d700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc09bee0 0x7f7acc09c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7acad9d700 1 -- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7acc09adf0 msgr2=0x7f7acc09b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7acad9d700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7acc09adf0 0x7f7acc09b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:25.938 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.906 7f7acad9d700 1 -- 172.21.15.32:0/3180603831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc09cb20 con 0x7f7acc09ebf0 2021-04-09T18:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.907 7f7acad9d700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc09ebf0 0x7f7acc1883d0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7f7abc000f90 tx=0x7f7abc00a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.907 7f7abb7fe700 1 -- 172.21.15.32:0/3180603831 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7abc001a30 con 0x7f7acc09ebf0 2021-04-09T18:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.907 7f7abb7fe700 1 -- 172.21.15.32:0/3180603831 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc0043a0 con 0x7f7acc09ebf0 2021-04-09T18:35:25.939 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.907 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc09cb20 con 0x7f7acc09ebf0 2021-04-09T18:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.907 7f7abb7fe700 1 -- 172.21.15.32:0/3180603831 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7abc0047f0 con 0x7f7acc09ebf0 2021-04-09T18:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.907 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc09cb20 con 0x7f7acc09ebf0 2021-04-09T18:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.907 7f7abb7fe700 1 -- 172.21.15.32:0/3180603831 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f7abc004580 con 0x7f7acc09ebf0 2021-04-09T18:35:25.940 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.908 7f7abb7fe700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab00188c0 0x7f7ab001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:25.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.908 7f7abb7fe700 1 -- 172.21.15.32:0/3180603831 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f7abc0329e0 con 0x7f7acc09ebf0 2021-04-09T18:35:25.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.908 7f7aca59c700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab00188c0 0x7f7ab001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:25.941 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.908 7f7aca59c700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab00188c0 0x7f7ab001adc0 crc :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:25.946 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.927 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc1889f0 con 0x7f7acc09ebf0 2021-04-09T18:35:25.949 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:25.931 7f7abb7fe700 1 -- 172.21.15.32:0/3180603831 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f7abc001a10 con 0x7f7acc09ebf0 2021-04-09T18:35:26.129 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.110 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f7acc1889f0 con 0x7f7acc09ebf0 2021-04-09T18:35:26.130 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.111 7f7abb7fe700 1 -- 172.21.15.32:0/3180603831 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v278) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7abc00ad30 con 0x7f7acc09ebf0 2021-04-09T18:35:26.130 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 8 2021-04-09T18:35:26.132 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab00188c0 msgr2=0x7f7ab001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:26.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab00188c0 0x7f7ab001adc0 crc :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc09ebf0 msgr2=0x7f7acc1883d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:26.133 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc09ebf0 0x7f7acc1883d0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7f7abc000f90 tx=0x7f7abc00a940).stop 2021-04-09T18:35:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 shutdown_connections 2021-04-09T18:35:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f7ab00188c0 0x7f7ab001adc0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7acc09bee0 0x7f7acc09c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7acc09adf0 0x7f7acc09b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 --2- 172.21.15.32:0/3180603831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7acc09ebf0 0x7f7acc1883d0 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.113 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 >> 172.21.15.32:0/3180603831 conn(0x7f7acc09fa00 msgr2=0x7f7acc0c3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.114 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 shutdown_connections 2021-04-09T18:35:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.114 7f7ad1260700 1 -- 172.21.15.32:0/3180603831 wait complete. 2021-04-09T18:35:26.143 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-04-09T18:35:26.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:35:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.273 7efcd5a25700 1 Processor -- start 2021-04-09T18:35:26.314 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.273 7efcd5a25700 1 -- start start 2021-04-09T18:35:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efcd5a25700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcd00cffa0 0x7efcd00d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efcd5a25700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcd00d6c80 0x7efcd010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:26.315 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efcd5a25700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd00d7280 con 0x7efcd00d7800 2021-04-09T18:35:26.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efcd5a25700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd00d7080 con 0x7efcd00d6c80 2021-04-09T18:35:26.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efcd5a25700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd00d91e0 con 0x7efcd00cffa0 2021-04-09T18:35:26.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efccffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcd00d6c80 0x7efcd010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:26.316 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efccffff700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7efcd00d6c80 0x7efcd010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60552/0 (socket says 172.21.15.32:60552) 2021-04-09T18:35:26.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efccffff700 1 -- 172.21.15.32:0/2067357949 learned_addr learned my addr 172.21.15.32:0/2067357949 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:26.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efccf7fe700 1 --2- 172.21.15.32:0/2067357949 >> v2:172.21.15.86:3300/0 conn(0x7efcd00cffa0 0x7efcd00d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:26.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efccffff700 1 -- 172.21.15.32:0/2067357949 >> v2:172.21.15.86:3300/0 conn(0x7efcd00cffa0 msgr2=0x7efcd00d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:26.317 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efccffff700 1 --2- 172.21.15.32:0/2067357949 >> v2:172.21.15.86:3300/0 conn(0x7efcd00cffa0 0x7efcd00d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efccffff700 1 -- 172.21.15.32:0/2067357949 >> v1:172.21.15.86:6789/0 conn(0x7efcd00d7800 legacy=0x7efcd00d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:26.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.274 7efccffff700 1 -- 172.21.15.32:0/2067357949 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd00d83f0 con 0x7efcd00d6c80 2021-04-09T18:35:26.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.275 7efccffff700 1 --2- 172.21.15.32:0/2067357949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd00d6c80 0x7efcd010cfd0 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7efcc8006f00 tx=0x7efcc8009f10).ready entity=mon.0 client_cookie=13f49f2572268776 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:26.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.275 7efcce7fc700 1 -- 172.21.15.32:0/2067357949 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcc800b3b0 con 0x7efcd00d6c80 2021-04-09T18:35:26.318 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.275 7efcce7fc700 1 -- 172.21.15.32:0/2067357949 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc800b730 con 0x7efcd00d6c80 2021-04-09T18:35:26.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.275 7efcce7fc700 1 -- 172.21.15.32:0/2067357949 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcc800bc10 con 0x7efcd00d6c80 2021-04-09T18:35:26.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.275 7efcd5a25700 1 -- 172.21.15.32:0/2067357949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd00d6c80 msgr2=0x7efcd010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:26.319 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.275 7efcd5a25700 1 --2- 172.21.15.32:0/2067357949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd00d6c80 0x7efcd010cfd0 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7efcc8006f00 tx=0x7efcc8009f10).stop 2021-04-09T18:35:26.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.276 7efcd5a25700 1 -- 172.21.15.32:0/2067357949 shutdown_connections 2021-04-09T18:35:26.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.276 7efcd5a25700 1 --2- 172.21.15.32:0/2067357949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd00d6c80 0x7efcd010cfd0 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.276 7efcd5a25700 1 --2- 172.21.15.32:0/2067357949 >> v2:172.21.15.86:3300/0 conn(0x7efcd00cffa0 0x7efcd00d2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.276 7efcd5a25700 1 -- 172.21.15.32:0/2067357949 >> 172.21.15.32:0/2067357949 conn(0x7efcd00ce5c0 msgr2=0x7efcd00cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.276 7efcd5a25700 1 -- 172.21.15.32:0/2067357949 shutdown_connections 2021-04-09T18:35:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.276 7efcd5a25700 1 -- 172.21.15.32:0/2067357949 wait complete. 2021-04-09T18:35:26.321 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.296 7efcd5a25700 1 Processor -- start 2021-04-09T18:35:26.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.296 7efcd5a25700 1 -- start start 2021-04-09T18:35:26.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.296 7efcd5a25700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd0091d10 0x7efcd0092110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:26.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.296 7efcd5a25700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcd0183fc0 0x7efcd01843c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:26.325 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.296 7efcd5a25700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd0090140 0x7efcd0090540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:26.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.296 7efcd5a25700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd0091790 con 0x7efcd0090140 2021-04-09T18:35:26.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efcd5a25700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd0092730 con 0x7efcd0091d10 2021-04-09T18:35:26.326 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efcd5a25700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd01acf90 con 0x7efcd0183fc0 2021-04-09T18:35:26.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd0090140 0x7efcd0090540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:26.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccffff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd0090140 0x7efcd0090540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60558/0 (socket says 172.21.15.32:60558) 2021-04-09T18:35:26.327 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccffff700 1 -- 172.21.15.32:0/3415690539 learned_addr learned my addr 172.21.15.32:0/3415690539 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccf7fe700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd0091d10 0x7efcd0092110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efcceffd700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcd0183fc0 0x7efcd01843c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccffff700 1 -- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcd0183fc0 msgr2=0x7efcd01843c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:26.328 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccffff700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcd0183fc0 0x7efcd01843c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccffff700 1 -- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd0091d10 msgr2=0x7efcd0092110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccffff700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd0091d10 0x7efcd0092110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccffff700 1 -- 172.21.15.32:0/3415690539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd01849e0 con 0x7efcd0090140 2021-04-09T18:35:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efccffff700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd0090140 0x7efcd0090540 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7efcc800b8e0 tx=0x7efcc800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:26.329 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efcb7fff700 1 -- 172.21.15.32:0/3415690539 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcc8009450 con 0x7efcd0090140 2021-04-09T18:35:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efcb7fff700 1 -- 172.21.15.32:0/3415690539 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc8005820 con 0x7efcd0090140 2021-04-09T18:35:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd01849e0 con 0x7efcd0090140 2021-04-09T18:35:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efcb7fff700 1 -- 172.21.15.32:0/3415690539 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcc8005c70 con 0x7efcd0090140 2021-04-09T18:35:26.330 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.297 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd01849e0 con 0x7efcd0090140 2021-04-09T18:35:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.298 7efcb7fff700 1 -- 172.21.15.32:0/3415690539 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efcc8005a00 con 0x7efcd0090140 2021-04-09T18:35:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.298 7efcb7fff700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcb0018930 0x7efcb001ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.298 7efcb7fff700 1 -- 172.21.15.32:0/3415690539 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efcc8032d70 con 0x7efcd0090140 2021-04-09T18:35:26.331 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.298 7efccf7fe700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcb0018930 0x7efcb001ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:26.332 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.299 7efccf7fe700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcb0018930 0x7efcb001ae10 crc :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:26.336 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.318 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd0182dc0 con 0x7efcd0090140 2021-04-09T18:35:26.340 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.322 7efcb7fff700 1 -- 172.21.15.32:0/3415690539 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efcc8000f90 con 0x7efcd0090140 2021-04-09T18:35:26.530 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.511 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcd00d7280 con 0x7efcd0090140 2021-04-09T18:35:26.531 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.512 7efcb7fff700 1 -- 172.21.15.32:0/3415690539 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 443 v443) v1 ==== 95+0+2789 (secure 0 0 0) 0x7efcc8005820 con 0x7efcd0090140 2021-04-09T18:35:26.531 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 443 2021-04-09T18:35:26.532 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:26.532 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":443,"default_fscid":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2260217511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2260217511},{"type":"v1","addr":"172.21.15.32:6834","nonce":2260217511}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":13257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1133511394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1133511394},{"type":"v1","addr":"172.21.15.32:6835","nonce":1133511394}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":60194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1457882085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1457882085},{"type":"v1","addr":"172.21.15.86:6837","nonce":1457882085}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439}],"filesystems":[{"mdsmap":{"epoch":443,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:35:19.657739","modified":"2021-04-09 18:35:21.914126","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12706":{"gid":12706,"name":"d","rank":0,"incarnation":440,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3693986652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3693986652},{"type":"v1","addr":"172.21.15.86:6836","nonce":3693986652}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[47],"metadata_pool":46,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":23}]} 2021-04-09T18:35:26.533 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcb0018930 msgr2=0x7efcb001ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcb0018930 0x7efcb001ae10 crc :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd0090140 msgr2=0x7efcd0090540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd0090140 0x7efcd0090540 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7efcc800b8e0 tx=0x7efcc800bc80).stop 2021-04-09T18:35:26.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 shutdown_connections 2021-04-09T18:35:26.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efcb0018930 0x7efcb001ae10 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efcd0090140 0x7efcd0090540 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.535 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcd0183fc0 0x7efcd01843c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 --2- 172.21.15.32:0/3415690539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd0091d10 0x7efcd0092110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 >> 172.21.15.32:0/3415690539 conn(0x7efcd0183840 msgr2=0x7efcd019b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 shutdown_connections 2021-04-09T18:35:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:26.514 7efcd5a25700 1 -- 172.21.15.32:0/3415690539 wait complete. 2021-04-09T18:35:26.946 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 443, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-04-09 18:35:19.657739', 'modified': '2021-04-09 18:35:21.914126', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12706}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12706': {'gid': 12706, 'name': 'd', 'rank': 0, 'incarnation': 440, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3693986652', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3693986652}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3693986652}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [47], 'metadata_pool': 46, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:35:26.946 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:35:26.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:35:27.075 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:35:27.076 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:35:27.076 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:35:27.076 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 12706, 2021-04-09T18:35:27.077 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:35:27.077 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:35:27.077 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 7.3976675529999998, 2021-04-09T18:35:27.077 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 443, 2021-04-09T18:35:27.078 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 278, 2021-04-09T18:35:27.078 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 278, 2021-04-09T18:35:27.078 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12.842608577 2021-04-09T18:35:27.078 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:35:27.081 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12706, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.3976675529999998, "mdsmap_epoch": 443, "osdmap_epoch": 278, "osdmap_epoch_barrier": 278, "uptime": 12.842608577 } 2021-04-09T18:35:27.082 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-04-09T18:35:27.082 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-04-09T18:35:27.083 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi032.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-04-09T18:35:27.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:27.083 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:35:27.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:35:27.188 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:35:27.189 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:35:27.247 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:35:27.247 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:35:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:35:27.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:27.249 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:35:27.293 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-04-09T18:35:27.294 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:35:27.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2021-04-09T18:35:27.395 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-04-09T18:35:27.396 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-04-09T18:35:27.455 INFO:teuthology.orchestra.run.smithi032.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-04-09T18:35:27.456 INFO:teuthology.orchestra.run.smithi032.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-04-09T18:35:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-04-09T18:35:27.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:27.458 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2021-04-09T18:35:27.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:2021-04-09 18:35:27.466 7f186e7e4f80 -1 init, newargv = 0x55bbc65f5270 newargc=9 2021-04-09T18:35:27.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[112843]: starting ceph client 2021-04-09T18:35:27.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[112843]: starting fuse 2021-04-09T18:35:27.502 INFO:teuthology.orchestra.run.smithi032.stdout:42 2021-04-09T18:35:27.503 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-04-09T18:35:27.503 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:27.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo python3 -c ' 2021-04-09T18:35:27.505 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2021-04-09T18:35:27.505 DEBUG:teuthology.orchestra.run.smithi032:> import re 2021-04-09T18:35:27.505 DEBUG:teuthology.orchestra.run.smithi032:> import os 2021-04-09T18:35:27.506 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2021-04-09T18:35:27.506 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:35:27.507 DEBUG:teuthology.orchestra.run.smithi032:> def find_socket(client_name): 2021-04-09T18:35:27.507 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-04-09T18:35:27.508 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2021-04-09T18:35:27.508 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:35:27.509 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2021-04-09T18:35:27.509 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2021-04-09T18:35:27.510 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2021-04-09T18:35:27.510 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2021-04-09T18:35:27.511 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:35:27.511 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2021-04-09T18:35:27.512 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-04-09T18:35:27.512 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2021-04-09T18:35:27.513 DEBUG:teuthology.orchestra.run.smithi032:> return f 2021-04-09T18:35:27.513 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-04-09T18:35:27.513 DEBUG:teuthology.orchestra.run.smithi032:> 2021-04-09T18:35:27.514 DEBUG:teuthology.orchestra.run.smithi032:> print(find_socket("client.0")) 2021-04-09T18:35:27.514 DEBUG:teuthology.orchestra.run.smithi032:> ' 2021-04-09T18:35:27.624 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.0.112843.asok 2021-04-09T18:35:27.629 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.112843.asok 2021-04-09T18:35:27.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:27.629 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.112843.asok status 2021-04-09T18:35:27.754 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:35:27.754 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2021-04-09T18:35:27.755 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "d23cd2d5f486487b0b80655c4e33344f0f97b82b", 2021-04-09T18:35:27.755 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 14.2.19-116-gd23cd2d5f48 (d23cd2d5f486487b0b80655c4e33344f0f97b82b) nautilus (stable)", 2021-04-09T18:35:27.755 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "0", 2021-04-09T18:35:27.755 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2021-04-09T18:35:27.756 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-04-09T18:35:27.756 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "112843", 2021-04-09T18:35:27.756 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2021-04-09T18:35:27.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:27.757 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2021-04-09T18:35:27.757 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2021-04-09T18:35:27.757 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 13398, 2021-04-09T18:35:27.757 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2021-04-09T18:35:27.758 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2021-04-09T18:35:27.758 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2021-04-09T18:35:27.758 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 13398 2021-04-09T18:35:27.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:27.759 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:35:27.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:27.760 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:35:27.760 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1511398703 2021-04-09T18:35:27.760 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:35:27.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:27.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2021-04-09T18:35:27.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:35:27.761 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2021-04-09T18:35:27.761 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1511398703 2021-04-09T18:35:27.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:35:27.762 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.13398 172.21.15.32:0/1511398703", 2021-04-09T18:35:27.762 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/1511398703", 2021-04-09T18:35:27.763 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2021-04-09T18:35:27.763 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 443, 2021-04-09T18:35:27.763 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 278, 2021-04-09T18:35:27.763 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2021-04-09T18:35:27.763 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklisted": false, 2021-04-09T18:35:27.764 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2021-04-09T18:35:27.764 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:35:27.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:27.766 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:35:27.766 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:35:27.812 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:35:27.812 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:35:27.813 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:35:27.814 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:35:27.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:35:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.005 7ff3e5684700 1 Processor -- start 2021-04-09T18:35:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.005 7ff3e5684700 1 -- start start 2021-04-09T18:35:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3e5684700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff3e00d7ca0 0x7ff3e00d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3e5684700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e010da00 0x7ff3e010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3e5684700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e010d480 con 0x7ff3e010d010 2021-04-09T18:35:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3e5684700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e00d9220 con 0x7ff3e00d7ca0 2021-04-09T18:35:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3e5684700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e00d86c0 con 0x7ff3e010da00 2021-04-09T18:35:28.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3deabe700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff3e00d7ca0 0x7ff3e00d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3df2bf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff3e010d010 0x7ff3e00d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41936/0 (socket says 172.21.15.32:41936) 2021-04-09T18:35:28.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3df2bf700 1 -- 172.21.15.32:0/500152400 learned_addr learned my addr 172.21.15.32:0/500152400 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:28.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3dfac0700 1 --2- 172.21.15.32:0/500152400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e010da00 0x7ff3e010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3deabe700 1 -- 172.21.15.32:0/500152400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e010da00 msgr2=0x7ff3e010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:28.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3deabe700 1 --2- 172.21.15.32:0/500152400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e010da00 0x7ff3e010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3deabe700 1 -- 172.21.15.32:0/500152400 >> v1:172.21.15.32:6789/0 conn(0x7ff3e010d010 legacy=0x7ff3e00d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3deabe700 1 -- 172.21.15.32:0/500152400 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e00d1f70 con 0x7ff3e00d7ca0 2021-04-09T18:35:28.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.006 7ff3de2bd700 1 -- 172.21.15.32:0/500152400 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048824444 0 0) 0x7ff3d0000ce0 con 0x7ff3e010d010 2021-04-09T18:35:28.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.007 7ff3deabe700 1 --2- 172.21.15.32:0/500152400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e00d7ca0 0x7ff3e00d80a0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7ff3d4006650 tx=0x7ff3d4009a00).ready entity=mon.0 client_cookie=5ba25dbe12e8f929 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:28.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.007 7ff3de2bd700 1 -- 172.21.15.32:0/500152400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d4009df0 con 0x7ff3e00d7ca0 2021-04-09T18:35:28.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.007 7ff3de2bd700 1 -- 172.21.15.32:0/500152400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d40009b0 con 0x7ff3e00d7ca0 2021-04-09T18:35:28.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.007 7ff3de2bd700 1 -- 172.21.15.32:0/500152400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d400c2b0 con 0x7ff3e00d7ca0 2021-04-09T18:35:28.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.007 7ff3e5684700 1 -- 172.21.15.32:0/500152400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e00d7ca0 msgr2=0x7ff3e00d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.007 7ff3e5684700 1 --2- 172.21.15.32:0/500152400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e00d7ca0 0x7ff3e00d80a0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7ff3d4006650 tx=0x7ff3d4009a00).stop 2021-04-09T18:35:28.051 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.008 7ff3e5684700 1 -- 172.21.15.32:0/500152400 shutdown_connections 2021-04-09T18:35:28.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.008 7ff3e5684700 1 --2- 172.21.15.32:0/500152400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e010da00 0x7ff3e010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.008 7ff3e5684700 1 --2- 172.21.15.32:0/500152400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e00d7ca0 0x7ff3e00d80a0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.008 7ff3e5684700 1 -- 172.21.15.32:0/500152400 >> 172.21.15.32:0/500152400 conn(0x7ff3e00ce5f0 msgr2=0x7ff3e00d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:28.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.008 7ff3e5684700 1 -- 172.21.15.32:0/500152400 shutdown_connections 2021-04-09T18:35:28.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.008 7ff3e5684700 1 -- 172.21.15.32:0/500152400 wait complete. 2021-04-09T18:35:28.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.028 7ff3e5684700 1 Processor -- start 2021-04-09T18:35:28.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.028 7ff3e5684700 1 -- start start 2021-04-09T18:35:28.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.028 7ff3e5684700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e017ba60 0x7ff3e017be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.028 7ff3e5684700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3e0179610 0x7ff3e0179a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.028 7ff3e5684700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e017a030 0x7ff3e017a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.028 7ff3e5684700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e017c480 con 0x7ff3e017a030 2021-04-09T18:35:28.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.028 7ff3e5684700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e0186860 con 0x7ff3e0179610 2021-04-09T18:35:28.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.028 7ff3e5684700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e017aa50 con 0x7ff3e017ba60 2021-04-09T18:35:28.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3dfac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e017a030 0x7ff3e017a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3dfac0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e017a030 0x7ff3e017a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60588/0 (socket says 172.21.15.32:60588) 2021-04-09T18:35:28.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3dfac0700 1 -- 172.21.15.32:0/2178706574 learned_addr learned my addr 172.21.15.32:0/2178706574 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:28.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3deabe700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3e0179610 0x7ff3e0179a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3df2bf700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e017ba60 0x7ff3e017be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3dfac0700 1 -- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e017ba60 msgr2=0x7ff3e017be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3dfac0700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e017ba60 0x7ff3e017be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3dfac0700 1 -- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3e0179610 msgr2=0x7ff3e0179a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3dfac0700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3e0179610 0x7ff3e0179a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3dfac0700 1 -- 172.21.15.32:0/2178706574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e017ac70 con 0x7ff3e017a030 2021-04-09T18:35:28.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3dfac0700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e017a030 0x7ff3e017a430 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 rx=0x7ff3d800b910 tx=0x7ff3d800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3c7fff700 1 -- 172.21.15.32:0/2178706574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d8006aa0 con 0x7ff3e017a030 2021-04-09T18:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3c7fff700 1 -- 172.21.15.32:0/2178706574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d80011f0 con 0x7ff3e017a030 2021-04-09T18:35:28.061 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e017ac70 con 0x7ff3e017a030 2021-04-09T18:35:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3c7fff700 1 -- 172.21.15.32:0/2178706574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d8001640 con 0x7ff3e017a030 2021-04-09T18:35:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.029 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e017ac70 con 0x7ff3e017a030 2021-04-09T18:35:28.063 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.030 7ff3c7fff700 1 -- 172.21.15.32:0/2178706574 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7ff3d80013d0 con 0x7ff3e017a030 2021-04-09T18:35:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.030 7ff3c7fff700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3c0018900 0x7ff3c001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.030 7ff3c7fff700 1 -- 172.21.15.32:0/2178706574 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7ff3d8032dd0 con 0x7ff3e017a030 2021-04-09T18:35:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.030 7ff3df2bf700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3c0018900 0x7ff3c001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.064 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.031 7ff3df2bf700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3c0018900 0x7ff3c001adc0 crc :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:28.068 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.050 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e017b530 con 0x7ff3e017a030 2021-04-09T18:35:28.072 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.053 7ff3c7fff700 1 -- 172.21.15.32:0/2178706574 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7ff3d800bdf0 con 0x7ff3e017a030 2021-04-09T18:35:28.264 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.245 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3e017b530 con 0x7ff3e017a030 2021-04-09T18:35:28.265 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.246 7ff3c7fff700 1 -- 172.21.15.32:0/2178706574 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 443 v443) v1 ==== 95+0+2789 (secure 0 0 0) 0x7ff3d80011f0 con 0x7ff3e017a030 2021-04-09T18:35:28.265 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 443 2021-04-09T18:35:28.265 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:28.266 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":443,"default_fscid":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6834/2260217511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2260217511},{"type":"v1","addr":"172.21.15.32:6834","nonce":2260217511}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":13257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1133511394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1133511394},{"type":"v1","addr":"172.21.15.32:6835","nonce":1133511394}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":60194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1457882085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1457882085},{"type":"v1","addr":"172.21.15.86:6837","nonce":1457882085}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439}],"filesystems":[{"mdsmap":{"epoch":443,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-04-09 18:35:19.657739","modified":"2021-04-09 18:35:21.914126","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12706":{"gid":12706,"name":"d","rank":0,"incarnation":440,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3693986652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3693986652},{"type":"v1","addr":"172.21.15.86:6836","nonce":3693986652}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[47],"metadata_pool":46,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":23}]} 2021-04-09T18:35:28.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3c0018900 msgr2=0x7ff3c001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.267 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3c0018900 0x7ff3c001adc0 crc :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e017a030 msgr2=0x7ff3e017a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e017a030 0x7ff3e017a430 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 rx=0x7ff3d800b910 tx=0x7ff3d800bcb0).stop 2021-04-09T18:35:28.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 shutdown_connections 2021-04-09T18:35:28.268 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7ff3c0018900 0x7ff3c001adc0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3e017a030 0x7ff3e017a430 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3e0179610 0x7ff3e0179a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 --2- 172.21.15.32:0/2178706574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3e017ba60 0x7ff3e017be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.269 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 >> 172.21.15.32:0/2178706574 conn(0x7ff3e0191560 msgr2=0x7ff3e01b7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:28.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.248 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 shutdown_connections 2021-04-09T18:35:28.270 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.249 7ff3e5684700 1 -- 172.21.15.32:0/2178706574 wait complete. 2021-04-09T18:35:28.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2021-04-09T18:35:28.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.411 7fe901578700 1 Processor -- start 2021-04-09T18:35:28.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.411 7fe901578700 1 -- start start 2021-04-09T18:35:28.455 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.411 7fe901578700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe8fc0d7ca0 0x7fe8fc0d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.411 7fe901578700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc10d480 con 0x7fe8fc10d010 2021-04-09T18:35:28.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.411 7fe901578700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc0d86c0 con 0x7fe8fc10da00 2021-04-09T18:35:28.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.411 7fe901578700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8fc0d9220 con 0x7fe8fc0d7ca0 2021-04-09T18:35:28.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fb2bf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe8fc10d010 0x7fe8fc0d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41950/0 (socket says 172.21.15.32:41950) 2021-04-09T18:35:28.456 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fb2bf700 1 -- 172.21.15.32:0/1851174272 learned_addr learned my addr 172.21.15.32:0/1851174272 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:28.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8faabe700 1 --2- 172.21.15.32:0/1851174272 >> v2:172.21.15.86:3300/0 conn(0x7fe8fc0d7ca0 0x7fe8fc0d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334480020 0 0) 0x7fe8ec000ce0 con 0x7fe8fc10d010 2021-04-09T18:35:28.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e4002fe0 con 0x7fe8fc10d010 2021-04-09T18:35:28.457 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202017642 0 0) 0x7fe8f4001400 con 0x7fe8fc10da00 2021-04-09T18:35:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e4003210 con 0x7fe8fc10da00 2021-04-09T18:35:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1133676093 0 0) 0x7fe8ec0010e0 con 0x7fe8fc10d010 2021-04-09T18:35:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 >> v2:172.21.15.86:3300/0 conn(0x7fe8fc0d7ca0 msgr2=0x7fe8fc0d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 --2- 172.21.15.32:0/1851174272 >> v2:172.21.15.86:3300/0 conn(0x7fe8fc0d7ca0 0x7fe8fc0d80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.458 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 >> v1:172.21.15.86:6789/0 conn(0x7fe8fc10da00 legacy=0x7fe8fc10de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc0d1f70 con 0x7fe8fc10d010 2021-04-09T18:35:28.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7fe8ec001600 con 0x7fe8fc10d010 2021-04-09T18:35:28.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 >> v1:172.21.15.32:6789/0 conn(0x7fe8fc10d010 legacy=0x7fe8fc0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.459 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8e4003b90 0x7fe8e4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e4006840 0x7fe8e4008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8e4009640 0x7fe8e400bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.460 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40503c0 con 0x7fe8e4006840 2021-04-09T18:35:28.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40505a0 con 0x7fe8e4009640 2021-04-09T18:35:28.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40507c0 con 0x7fe8e4003b90 2021-04-09T18:35:28.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8faabe700 1 --2- 172.21.15.32:0/1851174272 >> v2:172.21.15.86:3300/0 conn(0x7fe8fc0d7ca0 0x7fe8fc0d80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-04-09T18:35:28.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fb2bf700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8e4003b90 0x7fe8e4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.461 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.412 7fe8fbac0700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8e4009640 0x7fe8e400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8faabe700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e4006840 0x7fe8e4008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8faabe700 1 -- 172.21.15.32:0/1851174272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8e4003b90 msgr2=0x7fe8e4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8faabe700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8e4003b90 0x7fe8e4005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8faabe700 1 -- 172.21.15.32:0/1851174272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8e4009640 msgr2=0x7fe8e400bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:35:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8faabe700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8e4009640 0x7fe8e400bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8faabe700 1 -- 172.21.15.32:0/1851174272 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e4050a60 con 0x7fe8e4006840 2021-04-09T18:35:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8fbac0700 1 -- 172.21.15.32:0/1851174272 reap_dead start 2021-04-09T18:35:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8faabe700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e4006840 0x7fe8e4008d40 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7fe8f0000b80 tx=0x7fe8f0009e80).ready entity=mon.0 client_cookie=4226240f237b63c9 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8f0006b40 con 0x7fe8e4006840 2021-04-09T18:35:28.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f0006e00 con 0x7fe8e4006840 2021-04-09T18:35:28.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe8fa2bd700 1 -- 172.21.15.32:0/1851174272 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8f0007260 con 0x7fe8e4006840 2021-04-09T18:35:28.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe901578700 1 -- 172.21.15.32:0/1851174272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e4006840 msgr2=0x7fe8e4008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.413 7fe901578700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e4006840 0x7fe8e4008d40 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7fe8f0000b80 tx=0x7fe8f0009e80).stop 2021-04-09T18:35:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.414 7fe901578700 1 -- 172.21.15.32:0/1851174272 shutdown_connections 2021-04-09T18:35:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.414 7fe901578700 1 --2- 172.21.15.32:0/1851174272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8e4006840 0x7fe8e4008d40 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.414 7fe901578700 1 -- 172.21.15.32:0/1851174272 >> 172.21.15.32:0/1851174272 conn(0x7fe8fc0ce5f0 msgr2=0x7fe8fc0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.414 7fe901578700 1 -- 172.21.15.32:0/1851174272 shutdown_connections 2021-04-09T18:35:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.414 7fe901578700 1 -- 172.21.15.32:0/1851174272 wait complete. 2021-04-09T18:35:28.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.434 7fe901578700 1 Processor -- start 2021-04-09T18:35:28.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.434 7fe901578700 1 -- start start 2021-04-09T18:35:28.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe901578700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8fc0b1680 0x7fe8fc0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe901578700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8fc09e330 0x7fe8fc0b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe901578700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc0b2190 0x7fe8fc09f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe901578700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc09e730 con 0x7fe8fc0b1680 2021-04-09T18:35:28.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe901578700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc09f6c0 con 0x7fe8fc0b2190 2021-04-09T18:35:28.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe901578700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc09f8e0 con 0x7fe8fc09e330 2021-04-09T18:35:28.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fb2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8fc0b1680 0x7fe8fc0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fb2bf700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8fc0b1680 0x7fe8fc0b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60604/0 (socket says 172.21.15.32:60604) 2021-04-09T18:35:28.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fb2bf700 1 -- 172.21.15.32:0/3602753560 learned_addr learned my addr 172.21.15.32:0/3602753560 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:28.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8faabe700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8fc09e330 0x7fe8fc0b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.468 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fbac0700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc0b2190 0x7fe8fc09f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fb2bf700 1 -- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8fc09e330 msgr2=0x7fe8fc0b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fb2bf700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8fc09e330 0x7fe8fc0b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fb2bf700 1 -- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc0b2190 msgr2=0x7fe8fc09f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:28.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fb2bf700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc0b2190 0x7fe8fc09f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:28.469 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fb2bf700 1 -- 172.21.15.32:0/3602753560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc09fb00 con 0x7fe8fc0b1680 2021-04-09T18:35:28.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fbac0700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc0b2190 0x7fe8fc09f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:35:28.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8fb2bf700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8fc0b1680 0x7fe8fc0b0e30 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7fe8ec001790 tx=0x7fe8ec00a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:28.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8e3fff700 1 -- 172.21.15.32:0/3602753560 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8ec0070d0 con 0x7fe8fc0b1680 2021-04-09T18:35:28.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8e3fff700 1 -- 172.21.15.32:0/3602753560 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec00ac70 con 0x7fe8fc0b1680 2021-04-09T18:35:28.470 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe901578700 1 -- 172.21.15.32:0/3602753560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8fc09fb00 con 0x7fe8fc0b1680 2021-04-09T18:35:28.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe8e3fff700 1 -- 172.21.15.32:0/3602753560 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8ec004620 con 0x7fe8fc0b1680 2021-04-09T18:35:28.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.435 7fe901578700 1 -- 172.21.15.32:0/3602753560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc09fb00 con 0x7fe8fc0b1680 2021-04-09T18:35:28.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.436 7fe8e3fff700 1 -- 172.21.15.32:0/3602753560 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fe8ec001c10 con 0x7fe8fc0b1680 2021-04-09T18:35:28.471 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.436 7fe8e3fff700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8dc018930 0x7fe8dc01ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:28.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.436 7fe8e3fff700 1 -- 172.21.15.32:0/3602753560 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fe8ec032a10 con 0x7fe8fc0b1680 2021-04-09T18:35:28.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.436 7fe8faabe700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8dc018930 0x7fe8dc01ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:28.472 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.437 7fe8faabe700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8dc018930 0x7fe8dc01ae10 crc :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:28.474 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.456 7fe901578700 1 -- 172.21.15.32:0/3602753560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc10d480 con 0x7fe8fc0b1680 2021-04-09T18:35:28.477 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.459 7fe8e3fff700 1 -- 172.21.15.32:0/3602753560 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fe8ec000940 con 0x7fe8fc0b1680 2021-04-09T18:35:28.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:28.653 7fe901578700 1 -- 172.21.15.32:0/3602753560 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"var": "allow_standby_replay", "prefix": "fs set", "fs_name": "cephfs", "val": "true"} v 0) v1 -- 0x7fe8fc10d480 con 0x7fe8fc0b1680 2021-04-09T18:35:29.320 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.302 7fe8e3fff700 1 -- 172.21.15.32:0/3602753560 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"var": "allow_standby_replay", "prefix": "fs set", "fs_name": "cephfs", "val": "true"}]=0 v444) v1 ==== 121+0+0 (secure 0 0 0) 0x7fe8ec001b70 con 0x7fe8fc0b1680 2021-04-09T18:35:29.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 -- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8dc018930 msgr2=0x7fe8dc01ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:29.341 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8dc018930 0x7fe8dc01ae10 crc :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:29.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 -- 172.21.15.32:0/3602753560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8fc0b1680 msgr2=0x7fe8fc0b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:29.342 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8fc0b1680 0x7fe8fc0b0e30 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7fe8ec001790 tx=0x7fe8ec00a880).stop 2021-04-09T18:35:29.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 -- 172.21.15.32:0/3602753560 shutdown_connections 2021-04-09T18:35:29.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fe8dc018930 0x7fe8dc01ae10 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:29.343 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc0b2190 0x7fe8fc09f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:29.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8fc09e330 0x7fe8fc0b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:29.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 --2- 172.21.15.32:0/3602753560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8fc0b1680 0x7fe8fc0b0e30 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:29.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 -- 172.21.15.32:0/3602753560 >> 172.21.15.32:0/3602753560 conn(0x7fe8fc0c4290 msgr2=0x7fe8fc0c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:29.344 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 -- 172.21.15.32:0/3602753560 shutdown_connections 2021-04-09T18:35:29.345 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:29.321 7fe901578700 1 -- 172.21.15.32:0/3602753560 wait complete. 2021-04-09T18:35:39.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:35:39.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-09T18:35:59.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:35:59.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.493 7fc149115700 1 Processor -- start 2021-04-09T18:35:59.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.493 7fc149115700 1 -- start start 2021-04-09T18:35:59.540 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc149115700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440d6c80 0x7fc1440d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:59.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc149115700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1440d7280 con 0x7fc1440cffa0 2021-04-09T18:35:59.541 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc149115700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1440d2840 con 0x7fc1440d2440 2021-04-09T18:35:59.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc149115700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1440d7080 con 0x7fc1440d6c80 2021-04-09T18:35:59.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc142d9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc1440cffa0 0x7fc1440d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41978/0 (socket says 172.21.15.32:41978) 2021-04-09T18:35:59.542 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc142d9d700 1 -- 172.21.15.32:0/256234751 learned_addr learned my addr 172.21.15.32:0/256234751 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:35:59.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc14259c700 1 --2- 172.21.15.32:0/256234751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440d6c80 0x7fc1440d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:59.543 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc141d9b700 1 -- 172.21.15.32:0/256234751 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519690391 0 0) 0x7fc134001060 con 0x7fc1440cffa0 2021-04-09T18:35:59.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc141d9b700 1 -- 172.21.15.32:0/256234751 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc130002fe0 con 0x7fc1440cffa0 2021-04-09T18:35:59.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc141d9b700 1 -- 172.21.15.32:0/256234751 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323572077 0 0) 0x7fc13c0009b0 con 0x7fc1440d2440 2021-04-09T18:35:59.544 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc141d9b700 1 -- 172.21.15.32:0/256234751 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1300042c0 con 0x7fc1440d2440 2021-04-09T18:35:59.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc14259c700 1 -- 172.21.15.32:0/256234751 >> v1:172.21.15.86:6789/0 conn(0x7fc1440d2440 legacy=0x7fc14410cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:59.545 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc14259c700 1 -- 172.21.15.32:0/256234751 >> v1:172.21.15.32:6789/0 conn(0x7fc1440cffa0 legacy=0x7fc1440d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:59.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.494 7fc14259c700 1 -- 172.21.15.32:0/256234751 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1440d91e0 con 0x7fc1440d6c80 2021-04-09T18:35:59.546 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.495 7fc14259c700 1 --2- 172.21.15.32:0/256234751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440d6c80 0x7fc1440d7ff0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fc12c006650 tx=0x7fc12c009a80).ready entity=mon.2 client_cookie=ff5764a387c6db76 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:59.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.495 7fc141d9b700 1 -- 172.21.15.32:0/256234751 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc12c0009b0 con 0x7fc1440d6c80 2021-04-09T18:35:59.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.495 7fc141d9b700 1 -- 172.21.15.32:0/256234751 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c00c0c0 con 0x7fc1440d6c80 2021-04-09T18:35:59.547 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.495 7fc141d9b700 1 -- 172.21.15.32:0/256234751 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc12c00c510 con 0x7fc1440d6c80 2021-04-09T18:35:59.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.495 7fc149115700 1 -- 172.21.15.32:0/256234751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440d6c80 msgr2=0x7fc1440d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:59.548 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.495 7fc149115700 1 --2- 172.21.15.32:0/256234751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440d6c80 0x7fc1440d7ff0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fc12c006650 tx=0x7fc12c009a80).stop 2021-04-09T18:35:59.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.496 7fc149115700 1 -- 172.21.15.32:0/256234751 shutdown_connections 2021-04-09T18:35:59.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.496 7fc149115700 1 --2- 172.21.15.32:0/256234751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440d6c80 0x7fc1440d7ff0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:59.549 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.496 7fc149115700 1 -- 172.21.15.32:0/256234751 >> 172.21.15.32:0/256234751 conn(0x7fc1440ce5c0 msgr2=0x7fc1440cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.496 7fc149115700 1 -- 172.21.15.32:0/256234751 shutdown_connections 2021-04-09T18:35:59.550 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.496 7fc149115700 1 -- 172.21.15.32:0/256234751 wait complete. 2021-04-09T18:35:59.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.516 7fc149115700 1 Processor -- start 2021-04-09T18:35:59.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.516 7fc149115700 1 -- start start 2021-04-09T18:35:59.551 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.516 7fc149115700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1441b85b0 0x7fc14417dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:59.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.516 7fc149115700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14417e0e0 0x7fc1440adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:59.552 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.516 7fc149115700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440ae220 0x7fc1440ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:59.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.516 7fc149115700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1440964d0 con 0x7fc1441b85b0 2021-04-09T18:35:59.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.516 7fc149115700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1441b8e00 con 0x7fc14417e0e0 2021-04-09T18:35:59.553 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.516 7fc149115700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1441b89b0 con 0x7fc1440ae220 2021-04-09T18:35:59.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.516 7fc142d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1441b85b0 0x7fc14417dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:59.554 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc142d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1441b85b0 0x7fc14417dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60626/0 (socket says 172.21.15.32:60626) 2021-04-09T18:35:59.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc142d9d700 1 -- 172.21.15.32:0/2165618301 learned_addr learned my addr 172.21.15.32:0/2165618301 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:35:59.555 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc14259c700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14417e0e0 0x7fc1440adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:59.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc14359e700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440ae220 0x7fc1440ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:59.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc142d9d700 1 -- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440ae220 msgr2=0x7fc1440ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:59.556 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc142d9d700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440ae220 0x7fc1440ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:59.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc142d9d700 1 -- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14417e0e0 msgr2=0x7fc1440adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:59.557 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc142d9d700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14417e0e0 0x7fc1440adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc142d9d700 1 -- 172.21.15.32:0/2165618301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144095040 con 0x7fc1441b85b0 2021-04-09T18:35:59.558 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc142d9d700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1441b85b0 0x7fc14417dac0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7fc134009980 tx=0x7fc13400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc13b7fe700 1 -- 172.21.15.32:0/2165618301 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc134006030 con 0x7fc1441b85b0 2021-04-09T18:35:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc13b7fe700 1 -- 172.21.15.32:0/2165618301 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc134006370 con 0x7fc1441b85b0 2021-04-09T18:35:59.559 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc149115700 1 -- 172.21.15.32:0/2165618301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc144095040 con 0x7fc1441b85b0 2021-04-09T18:35:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc13b7fe700 1 -- 172.21.15.32:0/2165618301 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc134004910 con 0x7fc1441b85b0 2021-04-09T18:35:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.517 7fc149115700 1 -- 172.21.15.32:0/2165618301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc144095040 con 0x7fc1441b85b0 2021-04-09T18:35:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.518 7fc13b7fe700 1 -- 172.21.15.32:0/2165618301 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7fc134004720 con 0x7fc1441b85b0 2021-04-09T18:35:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.518 7fc13b7fe700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1280188d0 0x7fc12801add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:35:59.560 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.518 7fc13b7fe700 1 -- 172.21.15.32:0/2165618301 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7fc134032db0 con 0x7fc1441b85b0 2021-04-09T18:35:59.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.518 7fc14259c700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1280188d0 0x7fc12801add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:35:59.561 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.519 7fc14259c700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1280188d0 0x7fc12801add0 crc :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:35:59.562 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.538 7fc149115700 1 -- 172.21.15.32:0/2165618301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1441b89b0 con 0x7fc1441b85b0 2021-04-09T18:35:59.563 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.541 7fc13b7fe700 1 -- 172.21.15.32:0/2165618301 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7fc134000d30 con 0x7fc1441b85b0 2021-04-09T18:35:59.753 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.734 7fc149115700 1 -- 172.21.15.32:0/2165618301 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1441b89b0 con 0x7fc1441b85b0 2021-04-09T18:35:59.754 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.735 7fc13b7fe700 1 -- 172.21.15.32:0/2165618301 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 445 v445) v1 ==== 95+0+2797 (secure 0 0 0) 0x7fc134006350 con 0x7fc1441b85b0 2021-04-09T18:35:59.755 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 445 2021-04-09T18:35:59.755 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:35:59.756 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":445,"default_fscid":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1133511394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1133511394},{"type":"v1","addr":"172.21.15.32:6835","nonce":1133511394}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":60194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1457882085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1457882085},{"type":"v1","addr":"172.21.15.86:6837","nonce":1457882085}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439}],"filesystems":[{"mdsmap":{"epoch":445,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:35:19.657739","modified":"2021-04-09 18:35:29.303676","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12706":{"gid":12706,"name":"d","rank":0,"incarnation":440,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3693986652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3693986652},{"type":"v1","addr":"172.21.15.86:6836","nonce":3693986652}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_13254":{"gid":13254,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6834/2260217511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2260217511},{"type":"v1","addr":"172.21.15.32:6834","nonce":2260217511}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[47],"metadata_pool":46,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":23}]} 2021-04-09T18:35:59.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 -- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1280188d0 msgr2=0x7fc12801add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:59.757 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1280188d0 0x7fc12801add0 crc :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:59.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 -- 172.21.15.32:0/2165618301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1441b85b0 msgr2=0x7fc14417dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:35:59.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1441b85b0 0x7fc14417dac0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7fc134009980 tx=0x7fc13400bcd0).stop 2021-04-09T18:35:59.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 -- 172.21.15.32:0/2165618301 shutdown_connections 2021-04-09T18:35:59.758 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7fc1280188d0 0x7fc12801add0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:59.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1440ae220 0x7fc1440ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:59.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc14417e0e0 0x7fc1440adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:59.759 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 --2- 172.21.15.32:0/2165618301 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1441b85b0 0x7fc14417dac0 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:35:59.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 -- 172.21.15.32:0/2165618301 >> 172.21.15.32:0/2165618301 conn(0x7fc1441864e0 msgr2=0x7fc1441b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:35:59.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 -- 172.21.15.32:0/2165618301 shutdown_connections 2021-04-09T18:35:59.760 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:35:59.737 7fc149115700 1 -- 172.21.15.32:0/2165618301 wait complete. 2021-04-09T18:36:00.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 445, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-04-09 18:35:19.657739', 'modified': '2021-04-09 18:35:29.303676', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12706}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12706': {'gid': 12706, 'name': 'd', 'rank': 0, 'incarnation': 440, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/3693986652', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 3693986652}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 3693986652}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}, 'gid_13254': {'gid': 13254, 'name': 'a', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.32:6834/2260217511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 2260217511}, {'type': 'v1', 'addr': '172.21.15.32:6834', 'nonce': 2260217511}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [47], 'metadata_pool': 46, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-04-09T18:36:00.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-04-09T18:36:00.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok status 2021-04-09T18:36:00.312 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2021-04-09T18:36:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:36:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "whoami": 0, 2021-04-09T18:36:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "id": 12706, 2021-04-09T18:36:00.313 INFO:teuthology.orchestra.run.smithi086.stdout: "want_state": "up:active", 2021-04-09T18:36:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "up:active", 2021-04-09T18:36:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rank_uptime": 40.634392226999999, 2021-04-09T18:36:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "mdsmap_epoch": 445, 2021-04-09T18:36:00.314 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch": 278, 2021-04-09T18:36:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_epoch_barrier": 278, 2021-04-09T18:36:00.315 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 46.079333740999999 2021-04-09T18:36:00.315 INFO:teuthology.orchestra.run.smithi086.stdout:} 2021-04-09T18:36:00.319 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", "whoami": 0, "id": 12706, "want_state": "up:active", "state": "up:active", "rank_uptime": 40.634392226999999, "mdsmap_epoch": 445, "osdmap_epoch": 278, "osdmap_epoch_barrier": 278, "uptime": 46.079333740999999 } 2021-04-09T18:36:00.320 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-04-09T18:36:00.321 DEBUG:teuthology.orchestra.run.smithi032:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:36:00.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-04-09T18:36:00.440 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-04-09T18:36:00.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:36:00.441 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:36:00.442 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:36:00.488 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2021-04-09T18:36:00.488 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-04-09T18:36:00.489 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2021-04-09T18:36:00.489 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-04-09T18:36:00.489 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:36:00.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi032.stderr:ceph-fuse[112843]: fuse finished with error 0 and tester_r 0 2021-04-09T18:36:00.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:36:00.557 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:36:00.558 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:36:00.602 INFO:teuthology.orchestra.run.smithi032.stdout:ext2/ext3 2021-04-09T18:36:00.603 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-04-09T18:36:00.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:36:00.685 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-04-09T18:36:00.729 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-04-09T18:36:00.730 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-04-09T18:36:00.730 DEBUG:teuthology.orchestra.run.smithi086:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:36:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-09T18:36:00.773 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-04-09T18:36:00.774 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-04-09T18:36:00.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-04-09T18:36:00.775 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-04-09T18:36:00.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2021-04-09T18:36:01.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.985 7f6628034700 1 Processor -- start 2021-04-09T18:36:01.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.985 7f6628034700 1 -- start start 2021-04-09T18:36:01.026 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6628034700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66200d7ca0 0x7f66200d80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6628034700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662010da00 0x7f662010de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6628034700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662010d480 con 0x7f662010d010 2021-04-09T18:36:01.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6628034700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66200d9220 con 0x7f66200d7ca0 2021-04-09T18:36:01.027 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6628034700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66200d86c0 con 0x7f662010da00 2021-04-09T18:36:01.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f662564f700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66200d7ca0 0x7f66200d80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6625e50700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f662010d010 0x7f66200d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41992/0 (socket says 172.21.15.32:41992) 2021-04-09T18:36:01.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6625e50700 1 -- 172.21.15.32:0/2144699668 learned_addr learned my addr 172.21.15.32:0/2144699668 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:36:01.028 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6626651700 1 --2- 172.21.15.32:0/2144699668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662010da00 0x7f662010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6624e4e700 1 -- 172.21.15.32:0/2144699668 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906101668 0 0) 0x7f661c000ce0 con 0x7f662010d010 2021-04-09T18:36:01.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6624e4e700 1 -- 172.21.15.32:0/2144699668 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f660c002fe0 con 0x7f662010d010 2021-04-09T18:36:01.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f662564f700 1 -- 172.21.15.32:0/2144699668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662010da00 msgr2=0x7f662010de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:36:01.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f662564f700 1 --2- 172.21.15.32:0/2144699668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662010da00 0x7f662010de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.029 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f662564f700 1 -- 172.21.15.32:0/2144699668 >> v1:172.21.15.32:6789/0 conn(0x7f662010d010 legacy=0x7f66200d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f662564f700 1 -- 172.21.15.32:0/2144699668 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66200d1f70 con 0x7f66200d7ca0 2021-04-09T18:36:01.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f662564f700 1 --2- 172.21.15.32:0/2144699668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66200d7ca0 0x7f66200d80a0 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f6610006650 tx=0x7f6610009a80).ready entity=mon.0 client_cookie=ba7c451b7f4c1aa server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:36:01.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6624e4e700 1 -- 172.21.15.32:0/2144699668 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66100009b0 con 0x7f66200d7ca0 2021-04-09T18:36:01.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6624e4e700 1 -- 172.21.15.32:0/2144699668 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661000c0c0 con 0x7f66200d7ca0 2021-04-09T18:36:01.030 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6624e4e700 1 -- 172.21.15.32:0/2144699668 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f661000c510 con 0x7f66200d7ca0 2021-04-09T18:36:01.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6628034700 1 -- 172.21.15.32:0/2144699668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66200d7ca0 msgr2=0x7f66200d80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.986 7f6628034700 1 --2- 172.21.15.32:0/2144699668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66200d7ca0 0x7f66200d80a0 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f6610006650 tx=0x7f6610009a80).stop 2021-04-09T18:36:01.031 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.987 7f6628034700 1 -- 172.21.15.32:0/2144699668 shutdown_connections 2021-04-09T18:36:01.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.987 7f6628034700 1 --2- 172.21.15.32:0/2144699668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662010da00 0x7f662010de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.987 7f6628034700 1 --2- 172.21.15.32:0/2144699668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66200d7ca0 0x7f66200d80a0 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.032 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.987 7f6628034700 1 -- 172.21.15.32:0/2144699668 >> 172.21.15.32:0/2144699668 conn(0x7f66200ce5f0 msgr2=0x7f66200d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:36:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.987 7f6628034700 1 -- 172.21.15.32:0/2144699668 shutdown_connections 2021-04-09T18:36:01.033 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:00.987 7f6628034700 1 -- 172.21.15.32:0/2144699668 wait complete. 2021-04-09T18:36:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.007 7f6628034700 1 Processor -- start 2021-04-09T18:36:01.036 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.007 7f6628034700 1 -- start start 2021-04-09T18:36:01.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f6628034700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017daa0 0x7f6620180850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f6628034700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620180e70 0x7f662017ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f6628034700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6620181270 0x7f662017f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.037 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f6628034700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662017dea0 con 0x7f6620180e70 2021-04-09T18:36:01.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f6628034700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662017e190 con 0x7f6620181270 2021-04-09T18:36:01.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f6628034700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662017f680 con 0x7f662017daa0 2021-04-09T18:36:01.038 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f662564f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620180e70 0x7f662017ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f662564f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620180e70 0x7f662017ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60642/0 (socket says 172.21.15.32:60642) 2021-04-09T18:36:01.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f662564f700 1 -- 172.21.15.32:0/679981295 learned_addr learned my addr 172.21.15.32:0/679981295 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:36:01.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f6625e50700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017daa0 0x7f6620180850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f6626651700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6620181270 0x7f662017f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.039 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f662564f700 1 -- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017daa0 msgr2=0x7f6620180850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.008 7f662564f700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017daa0 0x7f6620180850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f662564f700 1 -- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6620181270 msgr2=0x7f662017f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f662564f700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6620181270 0x7f662017f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.040 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f662564f700 1 -- 172.21.15.32:0/679981295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662017f8a0 con 0x7f6620180e70 2021-04-09T18:36:01.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f662564f700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620180e70 0x7f662017ea40 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f6610009650 tx=0x7f661000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:36:01.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f66167fc700 1 -- 172.21.15.32:0/679981295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66100093a0 con 0x7f6620180e70 2021-04-09T18:36:01.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f66167fc700 1 -- 172.21.15.32:0/679981295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661000bd50 con 0x7f6620180e70 2021-04-09T18:36:01.041 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f6628034700 1 -- 172.21.15.32:0/679981295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662017f8a0 con 0x7f6620180e70 2021-04-09T18:36:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f66167fc700 1 -- 172.21.15.32:0/679981295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66100052e0 con 0x7f6620180e70 2021-04-09T18:36:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f6628034700 1 -- 172.21.15.32:0/679981295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662017f8a0 con 0x7f6620180e70 2021-04-09T18:36:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.009 7f66167fc700 1 -- 172.21.15.32:0/679981295 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f6610006030 con 0x7f6620180e70 2021-04-09T18:36:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.010 7f66167fc700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6608018930 0x7f660801ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.010 7f66167fc700 1 -- 172.21.15.32:0/679981295 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f66100329e0 con 0x7f6620180e70 2021-04-09T18:36:01.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.010 7f6625e50700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6608018930 0x7f660801ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.043 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.010 7f6625e50700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6608018930 0x7f660801ae10 crc :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:36:01.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.030 7f6628034700 1 -- 172.21.15.32:0/679981295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66201b7080 con 0x7f6620180e70 2021-04-09T18:36:01.052 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.033 7f66167fc700 1 -- 172.21.15.32:0/679981295 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f66100009c0 con 0x7f6620180e70 2021-04-09T18:36:01.240 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.221 7f6628034700 1 -- 172.21.15.32:0/679981295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f662010d480 con 0x7f6620180e70 2021-04-09T18:36:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.223 7f66167fc700 1 -- 172.21.15.32:0/679981295 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v278) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f6610006030 con 0x7f6620180e70 2021-04-09T18:36:01.244 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:36:01.244 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2021-04-09T18:36:01.244 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 278, 2021-04-09T18:36:01.244 INFO:teuthology.orchestra.run.smithi032.stdout: "fsid": "cf69fbbc-37e8-4833-99b4-5ecbe4444cfb", 2021-04-09T18:36:01.245 INFO:teuthology.orchestra.run.smithi032.stdout: "created": "2021-04-09 18:12:12.512919", 2021-04-09T18:36:01.245 INFO:teuthology.orchestra.run.smithi032.stdout: "modified": "2021-04-09 18:35:20.659017", 2021-04-09T18:36:01.245 INFO:teuthology.orchestra.run.smithi032.stdout: "last_up_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:36:01.245 INFO:teuthology.orchestra.run.smithi032.stdout: "last_in_change": "2021-04-09 18:12:24.480980", 2021-04-09T18:36:01.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-04-09T18:36:01.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_num": 5799936, 2021-04-09T18:36:01.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_set": [ 2021-04-09T18:36:01.246 INFO:teuthology.orchestra.run.smithi032.stdout: "pglog_hardlimit", 2021-04-09T18:36:01.246 INFO:teuthology.orchestra.run.smithi032.stdout: "purged_snapdirs", 2021-04-09T18:36:01.247 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_deletes", 2021-04-09T18:36:01.247 INFO:teuthology.orchestra.run.smithi032.stdout: "sortbitwise" 2021-04-09T18:36:01.247 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:36:01.247 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_version": 4, 2021-04-09T18:36:01.248 INFO:teuthology.orchestra.run.smithi032.stdout: "full_ratio": 0.94999998807907104, 2021-04-09T18:36:01.248 INFO:teuthology.orchestra.run.smithi032.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-04-09T18:36:01.248 INFO:teuthology.orchestra.run.smithi032.stdout: "nearfull_ratio": 0.85000002384185791, 2021-04-09T18:36:01.248 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_snapshot": "", 2021-04-09T18:36:01.248 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_max": 47, 2021-04-09T18:36:01.249 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd": 8, 2021-04-09T18:36:01.249 INFO:teuthology.orchestra.run.smithi032.stdout: "require_min_compat_client": "jewel", 2021-04-09T18:36:01.249 INFO:teuthology.orchestra.run.smithi032.stdout: "min_compat_client": "jewel", 2021-04-09T18:36:01.249 INFO:teuthology.orchestra.run.smithi032.stdout: "require_osd_release": "nautilus", 2021-04-09T18:36:01.250 INFO:teuthology.orchestra.run.smithi032.stdout: "pools": [ 2021-04-09T18:36:01.250 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.250 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 1, 2021-04-09T18:36:01.250 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "rbd", 2021-04-09T18:36:01.250 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:12:28.923594", 2021-04-09T18:36:01.251 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:36:01.251 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:36:01.251 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:36:01.251 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:36:01.252 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:36:01.252 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:36:01.252 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:36:01.252 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:36:01.252 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:36:01.253 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:36:01.253 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:36:01.253 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:36:01.253 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:36:01.253 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:36:01.254 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:36:01.254 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:36:01.254 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:36:01.254 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:36:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:36:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:36:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "16", 2021-04-09T18:36:01.255 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:36:01.256 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:36:01.256 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:36:01.256 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:36:01.256 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:36:01.257 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:36:01.257 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:36:01.257 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:36:01.257 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:36:01.257 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:36:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:36:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:36:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:36:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:36:01.258 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:36:01.259 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:36:01.259 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:36:01.259 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:36:01.259 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:36:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:36:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:36:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:36:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:36:01.260 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:36:01.261 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:36:01.261 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:36:01.261 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.261 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:36:01.261 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:36:01.262 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:36:01.262 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:36:01.262 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:36:01.262 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:36:01.263 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:36:01.263 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:36:01.263 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:36:01.263 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:36:01.264 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:36:01.264 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:36:01.264 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:36:01.264 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd": {} 2021-04-09T18:36:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 46, 2021-04-09T18:36:01.265 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_metadata", 2021-04-09T18:36:01.266 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:35:16.949280", 2021-04-09T18:36:01.266 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:36:01.266 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:36:01.266 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:36:01.267 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:36:01.267 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:36:01.267 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:36:01.267 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:36:01.267 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:36:01.268 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:36:01.268 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:36:01.268 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:36:01.268 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:36:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:36:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:36:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:36:01.272 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:36:01.273 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:36:01.273 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:36:01.273 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:36:01.273 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:36:01.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.274 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "277", 2021-04-09T18:36:01.274 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:36:01.274 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:36:01.274 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:36:01.275 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:36:01.275 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:36:01.275 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:36:01.275 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:36:01.275 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:36:01.276 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:36:01.276 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:36:01.276 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:36:01.276 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:36:01.277 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:36:01.277 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:36:01.277 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:36:01.277 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:36:01.277 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:36:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:36:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:36:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:36:01.278 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:36:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:36:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:36:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:36:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:36:01.279 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:36:01.280 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.280 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:36:01.281 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:36:01.281 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:36:01.282 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:36:01.282 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:36:01.282 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:36:01.283 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:36:01.283 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:36:01.283 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:36:01.283 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:36:01.284 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:36:01.284 INFO:teuthology.orchestra.run.smithi032.stdout: "options": { 2021-04-09T18:36:01.284 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_bias": 4, 2021-04-09T18:36:01.284 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_min": 16, 2021-04-09T18:36:01.285 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_priority": 5 2021-04-09T18:36:01.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.285 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:36:01.285 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:36:01.285 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": "cephfs" 2021-04-09T18:36:01.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.287 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.288 INFO:teuthology.orchestra.run.smithi032.stdout: "pool": 47, 2021-04-09T18:36:01.289 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": "cephfs_data", 2021-04-09T18:36:01.289 INFO:teuthology.orchestra.run.smithi032.stdout: "create_time": "2021-04-09 18:35:18.370170", 2021-04-09T18:36:01.289 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2021-04-09T18:36:01.290 INFO:teuthology.orchestra.run.smithi032.stdout: "flags_names": "hashpspool", 2021-04-09T18:36:01.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": 1, 2021-04-09T18:36:01.290 INFO:teuthology.orchestra.run.smithi032.stdout: "size": 2, 2021-04-09T18:36:01.290 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": 1, 2021-04-09T18:36:01.290 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_rule": 0, 2021-04-09T18:36:01.291 INFO:teuthology.orchestra.run.smithi032.stdout: "object_hash": 2, 2021-04-09T18:36:01.291 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscale_mode": "warn", 2021-04-09T18:36:01.291 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": 8, 2021-04-09T18:36:01.291 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num": 8, 2021-04-09T18:36:01.292 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_placement_num_target": 8, 2021-04-09T18:36:01.292 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_target": 8, 2021-04-09T18:36:01.292 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num_pending": 8, 2021-04-09T18:36:01.292 INFO:teuthology.orchestra.run.smithi032.stdout: "last_pg_merge_meta": { 2021-04-09T18:36:01.293 INFO:teuthology.orchestra.run.smithi032.stdout: "source_pgid": "0.0", 2021-04-09T18:36:01.293 INFO:teuthology.orchestra.run.smithi032.stdout: "ready_epoch": 0, 2021-04-09T18:36:01.293 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_started": 0, 2021-04-09T18:36:01.293 INFO:teuthology.orchestra.run.smithi032.stdout: "last_epoch_clean": 0, 2021-04-09T18:36:01.293 INFO:teuthology.orchestra.run.smithi032.stdout: "source_version": "0'0", 2021-04-09T18:36:01.294 INFO:teuthology.orchestra.run.smithi032.stdout: "target_version": "0'0" 2021-04-09T18:36:01.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.294 INFO:teuthology.orchestra.run.smithi032.stdout: "last_change": "278", 2021-04-09T18:36:01.294 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend": "0", 2021-04-09T18:36:01.295 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_prenautilus": "0", 2021-04-09T18:36:01.295 INFO:teuthology.orchestra.run.smithi032.stdout: "last_force_op_resend_preluminous": "0", 2021-04-09T18:36:01.295 INFO:teuthology.orchestra.run.smithi032.stdout: "auid": 0, 2021-04-09T18:36:01.295 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_mode": "selfmanaged", 2021-04-09T18:36:01.296 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_seq": 0, 2021-04-09T18:36:01.296 INFO:teuthology.orchestra.run.smithi032.stdout: "snap_epoch": 0, 2021-04-09T18:36:01.296 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_snaps": [], 2021-04-09T18:36:01.296 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps": "[]", 2021-04-09T18:36:01.296 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_bytes": 0, 2021-04-09T18:36:01.297 INFO:teuthology.orchestra.run.smithi032.stdout: "quota_max_objects": 0, 2021-04-09T18:36:01.297 INFO:teuthology.orchestra.run.smithi032.stdout: "tiers": [], 2021-04-09T18:36:01.297 INFO:teuthology.orchestra.run.smithi032.stdout: "tier_of": -1, 2021-04-09T18:36:01.297 INFO:teuthology.orchestra.run.smithi032.stdout: "read_tier": -1, 2021-04-09T18:36:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: "write_tier": -1, 2021-04-09T18:36:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_mode": "none", 2021-04-09T18:36:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_bytes": 0, 2021-04-09T18:36:01.298 INFO:teuthology.orchestra.run.smithi032.stdout: "target_max_objects": 0, 2021-04-09T18:36:01.299 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-04-09T18:36:01.299 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-04-09T18:36:01.299 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_target_full_ratio_micro": 800000, 2021-04-09T18:36:01.299 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_flush_age": 0, 2021-04-09T18:36:01.300 INFO:teuthology.orchestra.run.smithi032.stdout: "cache_min_evict_age": 0, 2021-04-09T18:36:01.300 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profile": "", 2021-04-09T18:36:01.300 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_params": { 2021-04-09T18:36:01.300 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "none" 2021-04-09T18:36:01.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.301 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_period": 0, 2021-04-09T18:36:01.301 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_count": 0, 2021-04-09T18:36:01.301 INFO:teuthology.orchestra.run.smithi032.stdout: "use_gmt_hitset": true, 2021-04-09T18:36:01.301 INFO:teuthology.orchestra.run.smithi032.stdout: "min_read_recency_for_promote": 0, 2021-04-09T18:36:01.302 INFO:teuthology.orchestra.run.smithi032.stdout: "min_write_recency_for_promote": 0, 2021-04-09T18:36:01.302 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_grade_decay_rate": 0, 2021-04-09T18:36:01.302 INFO:teuthology.orchestra.run.smithi032.stdout: "hit_set_search_last_n": 0, 2021-04-09T18:36:01.302 INFO:teuthology.orchestra.run.smithi032.stdout: "grade_table": [], 2021-04-09T18:36:01.302 INFO:teuthology.orchestra.run.smithi032.stdout: "stripe_width": 0, 2021-04-09T18:36:01.303 INFO:teuthology.orchestra.run.smithi032.stdout: "expected_num_objects": 0, 2021-04-09T18:36:01.303 INFO:teuthology.orchestra.run.smithi032.stdout: "fast_read": false, 2021-04-09T18:36:01.303 INFO:teuthology.orchestra.run.smithi032.stdout: "options": {}, 2021-04-09T18:36:01.303 INFO:teuthology.orchestra.run.smithi032.stdout: "application_metadata": { 2021-04-09T18:36:01.304 INFO:teuthology.orchestra.run.smithi032.stdout: "cephfs": { 2021-04-09T18:36:01.304 INFO:teuthology.orchestra.run.smithi032.stdout: "data": "cephfs" 2021-04-09T18:36:01.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.304 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.305 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:36:01.305 INFO:teuthology.orchestra.run.smithi032.stdout: "osds": [ 2021-04-09T18:36:01.305 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.305 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:36:01.306 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "8f892062-dd3f-413b-bbd2-6cfd9d640f69", 2021-04-09T18:36:01.306 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:36:01.306 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:36:01.306 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:36:01.307 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:36:01.307 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:36:01.307 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:36:01.307 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:36:01.307 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 276, 2021-04-09T18:36:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:36:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:36:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:36:01.308 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2021-04-09T18:36:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:36:01.309 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.310 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.310 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2021-04-09T18:36:01.310 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:36:01.310 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.311 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.311 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:36:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6802", 2021-04-09T18:36:01.312 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:36:01.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.313 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.313 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.313 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6803", 2021-04-09T18:36:01.314 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:36:01.314 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.314 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.315 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:36:01.315 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.315 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.315 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.315 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6806", 2021-04-09T18:36:01.316 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:36:01.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.316 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.316 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.316 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6807", 2021-04-09T18:36:01.317 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:36:01.317 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.317 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.318 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:36:01.318 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.318 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.318 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6804", 2021-04-09T18:36:01.319 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:36:01.319 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.319 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.319 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.320 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6805", 2021-04-09T18:36:01.320 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24886 2021-04-09T18:36:01.320 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.320 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.321 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6801/24886", 2021-04-09T18:36:01.321 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6803/24886", 2021-04-09T18:36:01.321 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6807/24886", 2021-04-09T18:36:01.322 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6805/24886", 2021-04-09T18:36:01.322 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:36:01.322 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:36:01.322 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:36:01.322 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.323 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.323 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:36:01.323 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "5f094ec4-4fc7-4ed9-a278-f51fab1f1727", 2021-04-09T18:36:01.324 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:36:01.324 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:36:01.324 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:36:01.324 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:36:01.324 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:36:01.325 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:36:01.325 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:36:01.325 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 276, 2021-04-09T18:36:01.325 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:36:01.325 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:36:01.326 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:36:01.326 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.326 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.326 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.326 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6808", 2021-04-09T18:36:01.327 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:36:01.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.327 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.328 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.328 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6809", 2021-04-09T18:36:01.328 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:36:01.328 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.328 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.329 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:36:01.329 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.329 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.330 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.330 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6810", 2021-04-09T18:36:01.330 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:36:01.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.330 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.331 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.331 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6811", 2021-04-09T18:36:01.331 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:36:01.331 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.332 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.332 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:36:01.332 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.333 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.333 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6814", 2021-04-09T18:36:01.333 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:36:01.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.334 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.334 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.334 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6815", 2021-04-09T18:36:01.334 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:36:01.335 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.335 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.335 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:36:01.336 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.336 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.336 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.336 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6812", 2021-04-09T18:36:01.336 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:36:01.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.337 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.337 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6813", 2021-04-09T18:36:01.338 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24901 2021-04-09T18:36:01.338 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.338 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.338 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.338 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6809/24901", 2021-04-09T18:36:01.339 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6811/24901", 2021-04-09T18:36:01.339 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6815/24901", 2021-04-09T18:36:01.339 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6813/24901", 2021-04-09T18:36:01.339 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:36:01.339 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:36:01.340 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:36:01.340 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.340 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.341 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:36:01.341 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "7a25e453-ac85-4d4e-8a06-5a18f990be19", 2021-04-09T18:36:01.341 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:36:01.341 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:36:01.341 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:36:01.342 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:36:01.342 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:36:01.342 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:36:01.342 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:36:01.343 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 271, 2021-04-09T18:36:01.343 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:36:01.343 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:36:01.343 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:36:01.343 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.344 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.344 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.344 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6816", 2021-04-09T18:36:01.344 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:36:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6817", 2021-04-09T18:36:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:36:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:36:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6818", 2021-04-09T18:36:01.347 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:36:01.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.348 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.348 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6819", 2021-04-09T18:36:01.348 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:36:01.349 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.349 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.349 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:36:01.350 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.350 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.350 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.350 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6822", 2021-04-09T18:36:01.350 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:36:01.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.351 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.351 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.351 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6823", 2021-04-09T18:36:01.351 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:36:01.352 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.352 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.352 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:36:01.353 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.353 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.353 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.353 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6820", 2021-04-09T18:36:01.353 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:36:01.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.354 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.354 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.354 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6821", 2021-04-09T18:36:01.354 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24934 2021-04-09T18:36:01.355 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.355 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.355 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6817/24934", 2021-04-09T18:36:01.356 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6819/24934", 2021-04-09T18:36:01.356 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6823/24934", 2021-04-09T18:36:01.356 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6821/24934", 2021-04-09T18:36:01.356 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:36:01.356 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:36:01.357 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:36:01.357 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.357 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.357 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:36:01.358 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "fc7ef9cf-5cd2-4fd2-8112-245a5141a327", 2021-04-09T18:36:01.358 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:36:01.358 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:36:01.358 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:36:01.359 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:36:01.359 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:36:01.359 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:36:01.359 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:36:01.359 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 276, 2021-04-09T18:36:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:36:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:36:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:36:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.360 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6824", 2021-04-09T18:36:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:36:01.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6825", 2021-04-09T18:36:01.362 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:36:01.363 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:36:01.364 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6826", 2021-04-09T18:36:01.365 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:36:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6827", 2021-04-09T18:36:01.366 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:36:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.367 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:36:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6830", 2021-04-09T18:36:01.368 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:36:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.369 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6831", 2021-04-09T18:36:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:36:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.370 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:36:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6828", 2021-04-09T18:36:01.371 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:36:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.372 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6829", 2021-04-09T18:36:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 24942 2021-04-09T18:36:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.373 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.32:6825/24942", 2021-04-09T18:36:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.32:6827/24942", 2021-04-09T18:36:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.32:6831/24942", 2021-04-09T18:36:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.32:6829/24942", 2021-04-09T18:36:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:36:01.374 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:36:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:36:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.375 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:36:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "122503dc-3cb6-4959-b287-cbae5ce53e7d", 2021-04-09T18:36:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:36:01.376 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:36:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:36:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:36:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:36:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:36:01.377 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:36:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 261, 2021-04-09T18:36:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:36:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:36:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:36:01.378 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6802", 2021-04-09T18:36:01.379 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:36:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6803", 2021-04-09T18:36:01.380 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:36:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:36:01.381 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6804", 2021-04-09T18:36:01.382 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:36:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6805", 2021-04-09T18:36:01.383 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:36:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:36:01.384 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6808", 2021-04-09T18:36:01.385 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:36:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6809", 2021-04-09T18:36:01.386 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:36:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.387 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:36:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6806", 2021-04-09T18:36:01.388 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:36:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.389 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6807", 2021-04-09T18:36:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25232 2021-04-09T18:36:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.390 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6803/25232", 2021-04-09T18:36:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6805/25232", 2021-04-09T18:36:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6809/25232", 2021-04-09T18:36:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6807/25232", 2021-04-09T18:36:01.391 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:36:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:36:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:36:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.392 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:36:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "097269c3-c880-4660-a0c2-12cd6fc7bcf9", 2021-04-09T18:36:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:36:01.393 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:36:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:36:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:36:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:36:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:36:01.394 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:36:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 276, 2021-04-09T18:36:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:36:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:36:01.395 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:36:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6810", 2021-04-09T18:36:01.396 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:36:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.397 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6811", 2021-04-09T18:36:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:36:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.398 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:36:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.399 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6812", 2021-04-09T18:36:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:36:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.400 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6813", 2021-04-09T18:36:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:36:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:36:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.402 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6816", 2021-04-09T18:36:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:36:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.403 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6817", 2021-04-09T18:36:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:36:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:36:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.405 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6814", 2021-04-09T18:36:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:36:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6815", 2021-04-09T18:36:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25237 2021-04-09T18:36:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6811/25237", 2021-04-09T18:36:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6813/25237", 2021-04-09T18:36:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6817/25237", 2021-04-09T18:36:01.408 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6815/25237", 2021-04-09T18:36:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:36:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:36:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:36:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:36:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "2a3e3f80-b45d-4425-8527-c07a38cea770", 2021-04-09T18:36:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:36:01.410 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:36:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:36:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:36:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:36:01.411 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:36:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:36:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 271, 2021-04-09T18:36:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:36:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:36:01.412 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:36:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6818", 2021-04-09T18:36:01.413 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:36:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.414 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6819", 2021-04-09T18:36:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:36:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.415 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:36:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.416 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6820", 2021-04-09T18:36:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:36:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.417 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6821", 2021-04-09T18:36:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:36:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.418 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:36:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.419 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6824", 2021-04-09T18:36:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:36:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.420 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6825", 2021-04-09T18:36:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:36:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.421 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:36:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.422 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6822", 2021-04-09T18:36:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:36:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.423 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6823", 2021-04-09T18:36:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25238 2021-04-09T18:36:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6819/25238", 2021-04-09T18:36:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6821/25238", 2021-04-09T18:36:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6825/25238", 2021-04-09T18:36:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6823/25238", 2021-04-09T18:36:01.425 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:36:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:36:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:36:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.426 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:36:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "uuid": "526aa5a2-5c73-4473-a78b-240711b2015c", 2021-04-09T18:36:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "up": 1, 2021-04-09T18:36:01.427 INFO:teuthology.orchestra.run.smithi032.stdout: "in": 1, 2021-04-09T18:36:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "weight": 1, 2021-04-09T18:36:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_affinity": 1, 2021-04-09T18:36:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_begin": 0, 2021-04-09T18:36:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "last_clean_end": 0, 2021-04-09T18:36:01.428 INFO:teuthology.orchestra.run.smithi032.stdout: "up_from": 13, 2021-04-09T18:36:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "up_thru": 275, 2021-04-09T18:36:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "down_at": 0, 2021-04-09T18:36:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "lost_at": 0, 2021-04-09T18:36:01.429 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addrs": { 2021-04-09T18:36:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6826", 2021-04-09T18:36:01.430 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:36:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6827", 2021-04-09T18:36:01.431 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:36:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.432 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addrs": { 2021-04-09T18:36:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6828", 2021-04-09T18:36:01.433 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:36:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6829", 2021-04-09T18:36:01.434 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:36:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.435 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addrs": { 2021-04-09T18:36:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.436 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6832", 2021-04-09T18:36:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:36:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.437 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6833", 2021-04-09T18:36:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:36:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.438 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addrs": { 2021-04-09T18:36:01.439 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2021-04-09T18:36:01.439 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.439 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2021-04-09T18:36:01.439 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6830", 2021-04-09T18:36:01.440 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:36:01.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.440 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.440 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2021-04-09T18:36:01.440 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.86:6831", 2021-04-09T18:36:01.441 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 25248 2021-04-09T18:36:01.441 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.441 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.442 INFO:teuthology.orchestra.run.smithi032.stdout: "public_addr": "172.21.15.86:6827/25248", 2021-04-09T18:36:01.442 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_addr": "172.21.15.86:6829/25248", 2021-04-09T18:36:01.442 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_back_addr": "172.21.15.86:6833/25248", 2021-04-09T18:36:01.442 INFO:teuthology.orchestra.run.smithi032.stdout: "heartbeat_front_addr": "172.21.15.86:6831/25248", 2021-04-09T18:36:01.442 INFO:teuthology.orchestra.run.smithi032.stdout: "state": [ 2021-04-09T18:36:01.443 INFO:teuthology.orchestra.run.smithi032.stdout: "exists", 2021-04-09T18:36:01.443 INFO:teuthology.orchestra.run.smithi032.stdout: "up" 2021-04-09T18:36:01.443 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2021-04-09T18:36:01.443 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.444 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:36:01.444 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_xinfo": [ 2021-04-09T18:36:01.444 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.444 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 0, 2021-04-09T18:36:01.444 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:36:01.445 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:36:01.445 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:36:01.445 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:36:01.445 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:36:01.446 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.446 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.446 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 1, 2021-04-09T18:36:01.446 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:36:01.446 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:36:01.447 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:36:01.447 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:36:01.447 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:36:01.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.447 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.448 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 2, 2021-04-09T18:36:01.448 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:36:01.448 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:36:01.448 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:36:01.449 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:36:01.449 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:36:01.449 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.449 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.450 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 3, 2021-04-09T18:36:01.450 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:36:01.450 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:36:01.450 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:36:01.450 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:36:01.451 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:36:01.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.451 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 4, 2021-04-09T18:36:01.452 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:36:01.452 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:36:01.452 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:36:01.452 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:36:01.452 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:36:01.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.453 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.453 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 5, 2021-04-09T18:36:01.453 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:36:01.453 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:36:01.454 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:36:01.454 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:36:01.454 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:36:01.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.455 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.455 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 6, 2021-04-09T18:36:01.455 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:36:01.455 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:36:01.455 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:36:01.456 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:36:01.456 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:36:01.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.456 INFO:teuthology.orchestra.run.smithi032.stdout: { 2021-04-09T18:36:01.456 INFO:teuthology.orchestra.run.smithi032.stdout: "osd": 7, 2021-04-09T18:36:01.457 INFO:teuthology.orchestra.run.smithi032.stdout: "down_stamp": "0.000000", 2021-04-09T18:36:01.457 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_probability": 0, 2021-04-09T18:36:01.457 INFO:teuthology.orchestra.run.smithi032.stdout: "laggy_interval": 0, 2021-04-09T18:36:01.457 INFO:teuthology.orchestra.run.smithi032.stdout: "features": 4611087854035861503, 2021-04-09T18:36:01.458 INFO:teuthology.orchestra.run.smithi032.stdout: "old_weight": 0 2021-04-09T18:36:01.458 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.458 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2021-04-09T18:36:01.458 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap": [], 2021-04-09T18:36:01.458 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_upmap_items": [], 2021-04-09T18:36:01.459 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_temp": [], 2021-04-09T18:36:01.459 INFO:teuthology.orchestra.run.smithi032.stdout: "primary_temp": [], 2021-04-09T18:36:01.459 INFO:teuthology.orchestra.run.smithi032.stdout: "blacklist": {}, 2021-04-09T18:36:01.459 INFO:teuthology.orchestra.run.smithi032.stdout: "erasure_code_profiles": { 2021-04-09T18:36:01.459 INFO:teuthology.orchestra.run.smithi032.stdout: "default": { 2021-04-09T18:36:01.460 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-failure-domain": "osd", 2021-04-09T18:36:01.460 INFO:teuthology.orchestra.run.smithi032.stdout: "k": "2", 2021-04-09T18:36:01.460 INFO:teuthology.orchestra.run.smithi032.stdout: "m": "1", 2021-04-09T18:36:01.460 INFO:teuthology.orchestra.run.smithi032.stdout: "plugin": "jerasure", 2021-04-09T18:36:01.461 INFO:teuthology.orchestra.run.smithi032.stdout: "ruleset-failure-domain": "osd", 2021-04-09T18:36:01.461 INFO:teuthology.orchestra.run.smithi032.stdout: "technique": "reed_sol_van" 2021-04-09T18:36:01.461 INFO:teuthology.orchestra.run.smithi032.stdout: } 2021-04-09T18:36:01.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2021-04-09T18:36:01.461 INFO:teuthology.orchestra.run.smithi032.stdout: "removed_snaps_queue": [], 2021-04-09T18:36:01.462 INFO:teuthology.orchestra.run.smithi032.stdout: "new_removed_snaps": [], 2021-04-09T18:36:01.462 INFO:teuthology.orchestra.run.smithi032.stdout: "new_purged_snaps": [], 2021-04-09T18:36:01.462 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_node_flags": {}, 2021-04-09T18:36:01.462 INFO:teuthology.orchestra.run.smithi032.stdout: "device_class_flags": {} 2021-04-09T18:36:01.462 INFO:teuthology.orchestra.run.smithi032.stdout:} 2021-04-09T18:36:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.226 7f6628034700 1 -- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6608018930 msgr2=0x7f660801ae10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.464 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6608018930 0x7f660801ae10 crc :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 -- 172.21.15.32:0/679981295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620180e70 msgr2=0x7f662017ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620180e70 0x7f662017ea40 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f6610009650 tx=0x7f661000b960).stop 2021-04-09T18:36:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 -- 172.21.15.32:0/679981295 shutdown_connections 2021-04-09T18:36:01.465 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f6608018930 0x7f660801ae10 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6620180e70 0x7f662017ea40 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6620181270 0x7f662017f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 --2- 172.21.15.32:0/679981295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f662017daa0 0x7f6620180850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 -- 172.21.15.32:0/679981295 >> 172.21.15.32:0/679981295 conn(0x7f66201acf80 msgr2=0x7f6620187e90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:36:01.466 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 -- 172.21.15.32:0/679981295 shutdown_connections 2021-04-09T18:36:01.467 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.227 7f6628034700 1 -- 172.21.15.32:0/679981295 wait complete. 2021-04-09T18:36:01.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-04-09T18:36:01.646 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.597 7efd62b8c700 1 Processor -- start 2021-04-09T18:36:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.597 7efd62b8c700 1 -- start start 2021-04-09T18:36:01.647 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd62b8c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efd5c10d010 0x7efd5c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd62b8c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c10d480 con 0x7efd5c0d8490 2021-04-09T18:36:01.648 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd62b8c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c10e1f0 con 0x7efd5c0d1da0 2021-04-09T18:36:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd62b8c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5c0d8d30 con 0x7efd5c10d010 2021-04-09T18:36:01.649 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd611a9700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7efd5c0d8490 0x7efd5c10da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42006/0 (socket says 172.21.15.32:42006) 2021-04-09T18:36:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd611a9700 1 -- 172.21.15.32:0/1624340153 learned_addr learned my addr 172.21.15.32:0/1624340153 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:36:01.650 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd609a8700 1 --2- 172.21.15.32:0/1624340153 >> v2:172.21.15.86:3300/0 conn(0x7efd5c10d010 0x7efd5c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413705992 0 0) 0x7efd540009b0 con 0x7efd5c0d8490 2021-04-09T18:36:01.651 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd48002fe0 con 0x7efd5c0d8490 2021-04-09T18:36:01.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3337311932 0 0) 0x7efd540010e0 con 0x7efd5c0d8490 2021-04-09T18:36:01.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 >> v2:172.21.15.86:3300/0 conn(0x7efd5c10d010 msgr2=0x7efd5c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:36:01.652 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd5b7fe700 1 --2- 172.21.15.32:0/1624340153 >> v2:172.21.15.86:3300/0 conn(0x7efd5c10d010 0x7efd5c0d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.598 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 >> v1:172.21.15.86:6789/0 conn(0x7efd5c0d1da0 legacy=0x7efd5c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.653 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c0d9220 con 0x7efd5c0d8490 2021-04-09T18:36:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 601680711 0 0) 0x7efd54001600 con 0x7efd5c0d8490 2021-04-09T18:36:01.654 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 >> v1:172.21.15.32:6789/0 conn(0x7efd5c0d8490 legacy=0x7efd5c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd48003a60 0x7efd48005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.655 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd48006790 0x7efd48008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd48009590 0x7efd4800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.656 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4800c300 con 0x7efd48003a60 2021-04-09T18:36:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4800c520 con 0x7efd48006790 2021-04-09T18:36:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd48050750 con 0x7efd48009590 2021-04-09T18:36:01.657 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd609a8700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd48003a60 0x7efd48005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd611a9700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd48009590 0x7efd4800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.658 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5bfff700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd48006790 0x7efd48008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd609a8700 1 -- 172.21.15.32:0/1624340153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd48009590 msgr2=0x7efd4800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:36:01.659 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd609a8700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd48009590 0x7efd4800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd609a8700 1 -- 172.21.15.32:0/1624340153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd48006790 msgr2=0x7efd48008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-04-09T18:36:01.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd609a8700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd48006790 0x7efd48008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.660 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd609a8700 1 -- 172.21.15.32:0/1624340153 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd480509b0 con 0x7efd48003a60 2021-04-09T18:36:01.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd611a9700 1 -- 172.21.15.32:0/1624340153 reap_dead start 2021-04-09T18:36:01.661 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5bfff700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd48006790 0x7efd48008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-04-09T18:36:01.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd609a8700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd48003a60 0x7efd48005ef0 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 rx=0x7efd4400c470 tx=0x7efd44006c20).ready entity=mon.0 client_cookie=10997f7fd89dadae server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:36:01.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd440090d0 con 0x7efd48003a60 2021-04-09T18:36:01.662 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd44009450 con 0x7efd48003a60 2021-04-09T18:36:01.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd5b7fe700 1 -- 172.21.15.32:0/1624340153 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd44009930 con 0x7efd48003a60 2021-04-09T18:36:01.663 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd62b8c700 1 -- 172.21.15.32:0/1624340153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd48003a60 msgr2=0x7efd48005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.599 7efd62b8c700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd48003a60 0x7efd48005ef0 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 rx=0x7efd4400c470 tx=0x7efd44006c20).stop 2021-04-09T18:36:01.664 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.600 7efd62b8c700 1 -- 172.21.15.32:0/1624340153 shutdown_connections 2021-04-09T18:36:01.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.600 7efd62b8c700 1 --2- 172.21.15.32:0/1624340153 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd48003a60 0x7efd48005ef0 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.600 7efd62b8c700 1 -- 172.21.15.32:0/1624340153 >> 172.21.15.32:0/1624340153 conn(0x7efd5c0ce5f0 msgr2=0x7efd5c0d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:36:01.665 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.600 7efd62b8c700 1 -- 172.21.15.32:0/1624340153 shutdown_connections 2021-04-09T18:36:01.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.600 7efd62b8c700 1 -- 172.21.15.32:0/1624340153 wait complete. 2021-04-09T18:36:01.666 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.620 7efd62b8c700 1 Processor -- start 2021-04-09T18:36:01.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.620 7efd62b8c700 1 -- start start 2021-04-09T18:36:01.667 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd62b8c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd5c1aca20 0x7efd5c1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd62b8c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c18bd40 0x7efd5c18c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd62b8c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd5c08fba0 0x7efd5c08ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.668 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd62b8c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c18c7e0 con 0x7efd5c18bd40 2021-04-09T18:36:01.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd62b8c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c18ca30 con 0x7efd5c08fba0 2021-04-09T18:36:01.669 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd62b8c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0905f0 con 0x7efd5c1aca20 2021-04-09T18:36:01.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd5bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c18bd40 0x7efd5c18c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd5bfff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c18bd40 0x7efd5c18c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60658/0 (socket says 172.21.15.32:60658) 2021-04-09T18:36:01.670 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd5bfff700 1 -- 172.21.15.32:0/1946241346 learned_addr learned my addr 172.21.15.32:0/1946241346 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:36:01.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd609a8700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd5c1aca20 0x7efd5c1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.671 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd611a9700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd5c08fba0 0x7efd5c08ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd5bfff700 1 -- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd5c1aca20 msgr2=0x7efd5c1ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.672 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd5bfff700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd5c1aca20 0x7efd5c1ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd5bfff700 1 -- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd5c08fba0 msgr2=0x7efd5c08ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd5bfff700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd5c08fba0 0x7efd5c08ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.673 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd5bfff700 1 -- 172.21.15.32:0/1946241346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c0b4300 con 0x7efd5c18bd40 2021-04-09T18:36:01.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd609a8700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd5c1aca20 0x7efd5c1ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:36:01.674 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd611a9700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd5c08fba0 0x7efd5c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-04-09T18:36:01.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.621 7efd5bfff700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c18bd40 0x7efd5c18c140 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 rx=0x7efd4c0099c0 tx=0x7efd4c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:36:01.675 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.622 7efd597fa700 1 -- 172.21.15.32:0/1946241346 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd4c001240 con 0x7efd5c18bd40 2021-04-09T18:36:01.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.622 7efd597fa700 1 -- 172.21.15.32:0/1946241346 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4c001540 con 0x7efd5c18bd40 2021-04-09T18:36:01.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.622 7efd597fa700 1 -- 172.21.15.32:0/1946241346 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd4c001a20 con 0x7efd5c18bd40 2021-04-09T18:36:01.676 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.622 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c0b4300 con 0x7efd5c18bd40 2021-04-09T18:36:01.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.622 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c0b4300 con 0x7efd5c18bd40 2021-04-09T18:36:01.677 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.622 7efd597fa700 1 -- 172.21.15.32:0/1946241346 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7efd4c001720 con 0x7efd5c18bd40 2021-04-09T18:36:01.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.622 7efd597fa700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd40018900 0x7efd4001adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:01.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.622 7efd597fa700 1 -- 172.21.15.32:0/1946241346 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7efd4c032de0 con 0x7efd5c18bd40 2021-04-09T18:36:01.678 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.623 7efd609a8700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd40018900 0x7efd4001adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:01.679 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.623 7efd609a8700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd40018900 0x7efd4001adc0 crc :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:36:01.680 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.642 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c182d70 con 0x7efd5c18bd40 2021-04-09T18:36:01.681 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.645 7efd597fa700 1 -- 172.21.15.32:0/1946241346 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7efd4c001750 con 0x7efd5c18bd40 2021-04-09T18:36:01.856 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.837 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd5c18c760 con 0x7efd5c18bd40 2021-04-09T18:36:01.857 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.837 7efd597fa700 1 -- 172.21.15.32:0/1946241346 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 445 v445) v1 ==== 95+0+2797 (secure 0 0 0) 0x7efd4c001520 con 0x7efd5c18bd40 2021-04-09T18:36:01.857 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 445 2021-04-09T18:36:01.857 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-04-09T18:36:01.858 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":445,"default_fscid":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.32:6835/1133511394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6833","nonce":1133511394},{"type":"v1","addr":"172.21.15.32:6835","nonce":1133511394}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439},{"gid":60194,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6837/1457882085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1457882085},{"type":"v1","addr":"172.21.15.86:6837","nonce":1457882085}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":439}],"filesystems":[{"mdsmap":{"epoch":445,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-04-09 18:35:19.657739","modified":"2021-04-09 18:35:29.303676","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12706},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12706":{"gid":12706,"name":"d","rank":0,"incarnation":440,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/3693986652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3693986652},{"type":"v1","addr":"172.21.15.86:6836","nonce":3693986652}]},"export_targets":[],"features":4611087854035861503,"flags":0},"gid_13254":{"gid":13254,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.32:6834/2260217511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":2260217511},{"type":"v1","addr":"172.21.15.32:6834","nonce":2260217511}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[47],"metadata_pool":46,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":23}]} 2021-04-09T18:36:01.859 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd40018900 msgr2=0x7efd4001adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd40018900 0x7efd4001adc0 crc :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c18bd40 msgr2=0x7efd5c18c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:01.860 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c18bd40 0x7efd5c18c140 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 rx=0x7efd4c0099c0 tx=0x7efd4c00bd40).stop 2021-04-09T18:36:01.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 shutdown_connections 2021-04-09T18:36:01.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7efd40018900 0x7efd4001adc0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efd5c18bd40 0x7efd5c18c140 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd5c08fba0 0x7efd5c08ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.861 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 --2- 172.21.15.32:0/1946241346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd5c1aca20 0x7efd5c1ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:01.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 >> 172.21.15.32:0/1946241346 conn(0x7efd5c19b460 msgr2=0x7efd5c19bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:36:01.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 shutdown_connections 2021-04-09T18:36:01.862 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:01.840 7efd62b8c700 1 -- 172.21.15.32:0/1946241346 wait complete. 2021-04-09T18:36:01.870 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2021-04-09T18:36:02.044 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.003 7f3c649cd700 1 Processor -- start 2021-04-09T18:36:02.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.004 7f3c649cd700 1 -- start start 2021-04-09T18:36:02.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.004 7f3c649cd700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c6010e7e0 0x7f3c600d8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:02.045 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.004 7f3c649cd700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c600d7290 con 0x7f3c600cffb0 2021-04-09T18:36:02.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.004 7f3c649cd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c600d2450 con 0x7f3c600d6c90 2021-04-09T18:36:02.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.004 7f3c649cd700 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c600d7090 con 0x7f3c6010e7e0 2021-04-09T18:36:02.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.004 7f3c5ee1d700 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f3c6010e7e0 0x7f3c600d8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:02.046 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.004 7f3c5e61c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3c600cffb0 0x7f3c600d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42024/0 (socket says 172.21.15.32:42024) 2021-04-09T18:36:02.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.004 7f3c5e61c700 1 -- 172.21.15.32:0/1299923552 learned_addr learned my addr 172.21.15.32:0/1299923552 (peer_addr_for_me v1:172.21.15.32:0/0) 2021-04-09T18:36:02.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c5ee1d700 1 -- 172.21.15.32:0/1299923552 >> v1:172.21.15.86:6789/0 conn(0x7f3c600d6c90 legacy=0x7f3c6010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:02.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c5ee1d700 1 -- 172.21.15.32:0/1299923552 >> v1:172.21.15.32:6789/0 conn(0x7f3c600cffb0 legacy=0x7f3c600d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:02.047 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c5d61a700 1 -- 172.21.15.32:0/1299923552 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409351774 0 0) 0x7f3c58001060 con 0x7f3c600cffb0 2021-04-09T18:36:02.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c5ee1d700 1 -- 172.21.15.32:0/1299923552 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c600d91f0 con 0x7f3c6010e7e0 2021-04-09T18:36:02.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c5ee1d700 1 --2- 172.21.15.32:0/1299923552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6010e7e0 0x7f3c600d8000 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 rx=0x7f3c50006ab0 tx=0x7f3c50009ac0).ready entity=mon.0 client_cookie=bcd75969c903e589 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:36:02.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c5d61a700 1 -- 172.21.15.32:0/1299923552 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c500066d0 con 0x7f3c6010e7e0 2021-04-09T18:36:02.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c5d61a700 1 -- 172.21.15.32:0/1299923552 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c500022e0 con 0x7f3c6010e7e0 2021-04-09T18:36:02.048 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c5d61a700 1 -- 172.21.15.32:0/1299923552 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c500027c0 con 0x7f3c6010e7e0 2021-04-09T18:36:02.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c649cd700 1 -- 172.21.15.32:0/1299923552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6010e7e0 msgr2=0x7f3c600d8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:02.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.005 7f3c649cd700 1 --2- 172.21.15.32:0/1299923552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6010e7e0 0x7f3c600d8000 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 rx=0x7f3c50006ab0 tx=0x7f3c50009ac0).stop 2021-04-09T18:36:02.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.006 7f3c649cd700 1 -- 172.21.15.32:0/1299923552 shutdown_connections 2021-04-09T18:36:02.049 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.006 7f3c649cd700 1 --2- 172.21.15.32:0/1299923552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c6010e7e0 0x7f3c600d8000 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:02.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.006 7f3c649cd700 1 -- 172.21.15.32:0/1299923552 >> 172.21.15.32:0/1299923552 conn(0x7f3c600ce5d0 msgr2=0x7f3c600cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-04-09T18:36:02.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.006 7f3c649cd700 1 -- 172.21.15.32:0/1299923552 shutdown_connections 2021-04-09T18:36:02.050 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.006 7f3c649cd700 1 -- 172.21.15.32:0/1299923552 wait complete. 2021-04-09T18:36:02.053 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.026 7f3c649cd700 1 Processor -- start 2021-04-09T18:36:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.026 7f3c649cd700 1 -- start start 2021-04-09T18:36:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c649cd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c601a38b0 0x7f3c601a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c649cd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c601a8390 0x7f3c601a8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:02.054 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c649cd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c601a8db0 0x7f3c601a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:02.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c649cd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601a4630 con 0x7f3c601a8390 2021-04-09T18:36:02.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c649cd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601a3360 con 0x7f3c601a8db0 2021-04-09T18:36:02.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c649cd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601a42d0 con 0x7f3c601a38b0 2021-04-09T18:36:02.055 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c601a8390 0x7f3c601a8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:02.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5de1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c601a8390 0x7f3c601a8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60674/0 (socket says 172.21.15.32:60674) 2021-04-09T18:36:02.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5de1b700 1 -- 172.21.15.32:0/3349742531 learned_addr learned my addr 172.21.15.32:0/3349742531 (peer_addr_for_me v2:172.21.15.32:0/0) 2021-04-09T18:36:02.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5e61c700 1 --2- 172.21.15.32:0/3349742531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c601a38b0 0x7f3c601a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:02.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5ee1d700 1 --2- 172.21.15.32:0/3349742531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c601a8db0 0x7f3c601a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:02.056 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5de1b700 1 -- 172.21.15.32:0/3349742531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c601a38b0 msgr2=0x7f3c601a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:02.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5de1b700 1 --2- 172.21.15.32:0/3349742531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c601a38b0 0x7f3c601a3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:02.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5de1b700 1 -- 172.21.15.32:0/3349742531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c601a8db0 msgr2=0x7f3c601a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-04-09T18:36:02.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5de1b700 1 --2- 172.21.15.32:0/3349742531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c601a8db0 0x7f3c601a1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-04-09T18:36:02.057 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5de1b700 1 -- 172.21.15.32:0/3349742531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c601a91b0 con 0x7f3c601a8390 2021-04-09T18:36:02.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c5de1b700 1 --2- 172.21.15.32:0/3349742531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3c601a8390 0x7f3c601a8790 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 rx=0x7f3c48006f20 tx=0x7f3c4800dbc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:36:02.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.027 7f3c56ffd700 1 -- 172.21.15.32:0/3349742531 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c48006030 con 0x7f3c601a8390 2021-04-09T18:36:02.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.028 7f3c56ffd700 1 -- 172.21.15.32:0/3349742531 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c480062b0 con 0x7f3c601a8390 2021-04-09T18:36:02.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.028 7f3c649cd700 1 -- 172.21.15.32:0/3349742531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c601a91b0 con 0x7f3c601a8390 2021-04-09T18:36:02.058 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.028 7f3c56ffd700 1 -- 172.21.15.32:0/3349742531 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c480057a0 con 0x7f3c601a8390 2021-04-09T18:36:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.028 7f3c649cd700 1 -- 172.21.15.32:0/3349742531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c601a91b0 con 0x7f3c601a8390 2021-04-09T18:36:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.028 7f3c56ffd700 1 -- 172.21.15.32:0/3349742531 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) v1 ==== 33430+0+0 (secure 0 0 0) 0x7f3c48000b50 con 0x7f3c601a8390 2021-04-09T18:36:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.028 7f3c56ffd700 1 --2- 172.21.15.32:0/3349742531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c44018900 0x7f3c4401adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-04-09T18:36:02.059 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.028 7f3c56ffd700 1 -- 172.21.15.32:0/3349742531 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 5577+0+0 (secure 0 0 0) 0x7f3c48034da0 con 0x7f3c601a8390 2021-04-09T18:36:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.029 7f3c5e61c700 1 --2- 172.21.15.32:0/3349742531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c44018900 0x7f3c4401adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-04-09T18:36:02.060 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.029 7f3c5e61c700 1 --2- 172.21.15.32:0/3349742531 >> [v2:172.21.15.86:6800/23869,v1:172.21.15.86:6801/23869] conn(0x7f3c44018900 0x7f3c4401adc0 crc :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-04-09T18:36:02.067 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.048 7f3c649cd700 1 -- 172.21.15.32:0/3349742531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6017cb40 con 0x7f3c601a8390 2021-04-09T18:36:02.071 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.052 7f3c56ffd700 1 -- 172.21.15.32:0/3349742531 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111559 (secure 0 0 0) 0x7f3c48006bb0 con 0x7f3c601a8390 2021-04-09T18:36:02.254 INFO:teuthology.orchestra.run.smithi032.stderr:2021-04-09 18:36:02.234 7f3c649cd700 1 -- 172.21.15.32:0/3349742531 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3c600d7290 con 0x7f3c601a8390 2021-04-09T18:36:02.743 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-04-09T18:36:02.744 DEBUG:teuthology.task.console_log:Killing console logger for smithi032 2021-04-09T18:36:02.745 DEBUG:teuthology.task.console_log:Killing console logger for smithi086 2021-04-09T18:36:02.745 DEBUG:teuthology.exit:Finished running handlers